MyArxiv
🏆 Weekly Top Picks (2198 Papers)
9
LLaVA-UHD v3: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs
Title CN LLaVA-UHD v3：用于多模态大模型中高效原生分辨率编码的渐进式视觉压缩
Keywords 渐进式视觉压缩 · 高分辨率处理 · 视觉令牌压缩 · 多模态大模型 · ViT架构优化
Summary 本文提出了LLaVA-UHD v3，基于新设计的渐进式视觉压缩（PVC）方法，实现高效的原生分辨率视觉编码。PVC包含两个模块：支持灵活块大小缩放的精细化块嵌入，以及跨ViT层层次部署的窗口化令牌压缩，能够在保留预训练ViT通用性的同时大幅提升推理效率。实验表明，ViT-UHD在性能上媲美MoonViT，并将首令牌时间（TTFT）减少2.4倍；LLaVA-UHD v3性能与Qwen2-VL相当，TTFT进一步降低1.9倍。
Reason 该论文提出了一种名为渐进式视觉压缩（PVC）的方法，通过精细化的块嵌入和分层窗口化令牌压缩，在保持原生高分辨率输入的同时显著降低计算开销。这一方法直接针对文档图像理解中的核心痛点——高分辨率处理与细粒度定位，且其ViT-UHD架构可无缝集成到现有VLM中，提升OCR-free文档理解的效率与精度。属于DeepSeek-OCR所倡导的‘视觉压缩’路线的重要进展，具有范式迁移意义。
Abstract Visual encoding followed by token condensing has become the standard architectural paradigm in multi-modal large language models (MLLMs). Many recent MLLMs increasingly favor global native- resolution visual encoding over slice-based methods. To investigate this trend, we systematically compare their behavior on vision-language understanding and attention patterns, revealing that global encoding enhances overall capability but at the expense of greater computational overhead. To address this issue, we present LLaVA-UHD v3, an MLLM centered upon our proposed Progressive Visual Compression (PVC) method, which can be seamlessly integrated into standard Vision Transformer (ViT) to enable efficient native-resolution encoding. The PVC approach consists of two key modules: (i) refined patch embedding, which supports flexible patch-size scaling for fine-grained visual model- ing, (ii) windowed token compression, hierarchically deployed across ViT layers to progressively aggregate local token representations. Jointly modulated by these two modules, a widely pretrained ViT can be reconfigured into an efficient architecture while largely preserving generality. Evaluated across extensive benchmarks, the transformed ViT, termed ViT-UHD, demonstrates competitive performance with MoonViT while reducing TTFT (time-to-first-token) by 2.4x, when developed within an identical MLLM architecture. Building upon ViT-UHD, LLaVA-UHD v3 also achieves competitive performance to Qwen2-VL, while further reducing TTFT by 1.9x. We will release all code and checkpoints to support future research on efficient MLLMs.
Authors Shichu Sun, Yichen Zhang, Haolin Song, Zonghao Guo, Chi Chen, Yidan Zhang, Yuan Yao, Zhiyuan Liu, Maosong Sun
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
9
Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
Title CN 视觉思维链：通过连续视觉令牌教会视觉语言模型更好地看与思考
Keywords 视觉思维链 · 连续视觉令牌 · 视觉语言模型 · 密集感知 · 空间推理
Summary 本文提出Chain-of-Visual-Thought（COVT），通过引入连续视觉令牌提升视觉语言模型在密集感知任务上的表现。这些紧凑的潜变量编码了丰富的视觉线索（如空间布局、边缘结构等），模型在训练时通过重建深度、分割、边缘等信号学习感知能力，在推理时可在视觉令牌空间中直接推理，兼顾效率与精度。在多个高难度视觉理解基准上性能提升显著。
Reason 该论文提出Chain-of-Visual-Thought（COVT），引入连续视觉令牌作为密集感知信息的紧凑表示，使VLM能在视觉空间中进行推理。这一机制直接应对文档图像理解中的核心挑战——细粒度视觉感知与空间结构建模，尤其适用于复杂排版、表格结构识别和OCR-free场景下的精准定位。其利用约20个视觉令牌编码2D外观、空间布局和边缘结构的能力，可显著增强VLM对文档中密集文本区域和几何关系的理解，属于DIU领域具有范式迁移潜力的底层创新。
Abstract Vision-Language Models (VLMs) excel at reasoning in linguistic space but struggle with perceptual understanding that requires dense visual perception, e.g., spatial reasoning and geometric awareness. This limitation stems from the fact that current VLMs have limited mechanisms to capture dense visual information across spatial dimensions. We introduce Chain-of-Visual-Thought (COVT), a framework that enables VLMs to reason not only in words but also through continuous visual tokens-compact latent representations that encode rich perceptual cues. Within a small budget of roughly 20 tokens, COVT distills knowledge from lightweight vision experts, capturing complementary properties such as 2D appearance, 3D geometry, spatial layout, and edge structure. During training, the VLM with COVT autoregressively predicts these visual tokens to reconstruct dense supervision signals (e.g., depth, segmentation, edges, and DINO features). At inference, the model reasons directly in the continuous visual token space, preserving efficiency while optionally decoding dense predictions for interpretability. Evaluated across more than ten diverse perception benchmarks, including CV-Bench, MMVP, RealWorldQA, MMStar, WorldMedQA, and HRBench, integrating COVT into strong VLMs such as Qwen2.5-VL and LLaVA consistently improves performance by 3% to 16% and demonstrates that compact continuous visual thinking enables more precise, grounded, and interpretable multimodal intelligence.
Authors Yiming Qin, Bomin Wei, Jiaxin Ge, Konstantinos Kallidromitis, Stephanie Fu, Trevor Darrell, Xudong Wang
Comment Project page: https://wakalsprojectpage.github.io/comt-website/
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
9
HunyuanOCR Technical Report
Title CN HunyuanOCR 技术报告
Keywords 文档图像理解 · 视觉语言模型 · 端到端OCR · 信息抽取 · 强化学习
Summary HunyuanOCR是一种轻量级、端到端的视觉语言模型，专用于解决OCR任务中的多功能性、效率和误差传播问题。它采用原生ViT与小型LLM通过MLP连接的结构，在不依赖传统预处理模块的情况下，统一支持文本识别、解析、信息抽取、问答和翻译等多项能力。模型在多个基准上超越更大规模的通用VLM和商业API，并在ICDAR 2025小模型赛道夺冠。研究还证明了高质量数据与强化学习策略对OCR性能提升的关键作用。项目已开源并提供基于vLLM的高效部署方案。
Reason 该论文提出了一种专为OCR任务设计的轻量级视觉语言模型HunyuanOCR，针对文档图像理解中的核心问题（如传统流水线误差传播、多任务支持不足、效率与性能难以兼顾）进行了系统性改进。其端到端架构消除了对布局分析等预处理模块的依赖，从根本上缓解了错误累积问题；同时在1B参数量级下实现多任务统一（文本检测、解析、信息抽取、VQA、翻译），并在ICDAR 2025挑战赛和OCRBench上取得SOTA表现。更重要的是，首次在工业级OCR中验证了强化学习策略的有效性，为DIU领域的训练范式提供了新方向。属于DIU本题的高质量工作。
Abstract This paper presents HunyuanOCR, a commercial-grade, open-source, and lightweight (1B parameters) Vision-Language Model (VLM) dedicated to OCR tasks. The architecture comprises a Native Vision Transformer (ViT) and a lightweight LLM connected via an MLP adapter. HunyuanOCR demonstrates superior performance, outperforming commercial APIs, traditional pipelines, and larger models (e.g., Qwen3-VL-4B). Specifically, it surpasses current public solutions in perception tasks (Text Spotting, Parsing) and excels in semantic tasks (IE, Text Image Translation), securing first place in the ICDAR 2025 DIMT Challenge (Small Model Track). Furthermore, it achieves state-of-the-art (SOTA) results on OCRBench among VLMs with fewer than 3B parameters. HunyuanOCR achieves breakthroughs in three key aspects: 1) Unifying Versatility and Efficiency: We implement comprehensive support for core capabilities including spotting, parsing, IE, VQA, and translation within a lightweight framework. This addresses the limitations of narrow "OCR expert models" and inefficient "General VLMs". 2) Streamlined End-to-End Architecture: Adopting a pure end-to-end paradigm eliminates dependencies on pre-processing modules (e.g., layout analysis). This fundamentally resolves error propagation common in traditional pipelines and simplifies system deployment. 3) Data-Driven and RL Strategies: We confirm the critical role of high-quality data and, for the first time in the industry, demonstrate that Reinforcement Learning (RL) strategies yield significant performance gains in OCR tasks. HunyuanOCR is officially open-sourced on HuggingFace. We also provide a high-performance deployment solution based on vLLM, placing its production efficiency in the top tier. We hope this model will advance frontier research and provide a solid foundation for industrial applications.
Authors Hunyuan Vision Team, Pengyuan Lyu, Xingyu Wan, Gengluo Li, Shangpin Peng, Weinong Wang, Liang Wu, Huawen Shen, Yu Zhou, Canhui Tang, Qi Yang, Qiming Peng, Bin Luo, Hower Yang, Houwen Peng, Hongming Yang, Senhao Xie, Binghong Wu, Mana Yang, Sergey Wang, Raccoon Liu, Dick Zhu, Jie Jiang, Linus, Han Hu, Chengquan Zhang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
9
Scaling Capability in Token Space: An Analysis of Large Vision Language Model
Title CN 在令牌空间中扩展能力：大型视觉语言模型的分析
Keywords 视觉语言模型 · 视觉token · 缩放律 · 表示学习 · 多模态建模
Summary 本文研究视觉语言模型中视觉令牌数量的扩展规律，提出一个数学框架来描述视觉令牌数与其语义距离发散之间的关系，发现存在子线性和线性两种缩放 regime，并给出性能公式 S(n) ≈ c / n^{α(n)}。实验验证了该理论在多个多模态基准上的有效性，增进了对Transformer中视觉token扩展行为的理解。
Reason 该论文提出了关于视觉语言模型中视觉令牌数量与模型性能之间可预测的缩放关系的理论框架，揭示了子线性与线性两种缩放机制，并通过数学建模解释了视觉令牌表示间的相关结构如何影响整体性能。这一工作为VLM中视觉输入的高效表示提供了底层原理支持，尤其对文档图像理解（DIU）中高分辨率图像处理和视觉token压缩（如DeepSeek-OCR路线）具有直接指导意义——可通过控制视觉token数量在精度与效率间取得最优平衡，缓解长文档或密集排版下的计算负担。
Abstract Large language models have demonstrated predictable scaling behaviors with respect to model parameters and training data. This study investigates whether a similar scaling relationship exist for vision-language models with respect to the number of vision tokens. A mathematical framework is developed to characterize a relationship between vision token number and the expected divergence of distance between vision-referencing sequences. The theoretical analysis reveals two distinct scaling regimes: sublinear scaling for less vision tokens and linear scaling for more vision tokens. This aligns with model performance relationships of the form \(S(n) \approx c / n^{α(n)}\), where the scaling exponent relates to the correlation structure between vision token representations. Empirical validations across multiple vision-language benchmarks show that model performance matches the prediction from scaling relationship. The findings contribute to understanding vision token scaling in transformers through a theoretical framework that complements empirical observations.
Authors Tenghui Li, Guoxu Zhou, Xuyang Zhao, Qibin Zhao
Categories Artificial Intelligence
9
AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs
Title CN AdaTok：面向高效多模态大模型的对象感知自适应令牌压缩
Keywords 视觉令牌压缩 · 多模态大模型 · 对象感知表示 · 高效VLM · 幻觉抑制
Summary 本文提出AdaTok，一种基于对象感知的自适应令牌压缩策略，用于高效多模态大模型。通过将图像分块转换为更具语义的对象级表示，实现视觉令牌的智能合并，在大幅降低计算开销的同时保持高性能。实验表明，该方法仅使用约10%的视觉令牌即可达到原始模型96%的表现，并在多个基准上优于现有压缩方法。
Reason 该论文提出了一种基于对象感知的自适应令牌压缩方法，将图像从传统的块级（patch-level）表示提升为对象级表示，显著减少视觉令牌数量（仅用10%令牌保留96%性能），且更符合人类视觉认知机制。这一方法直接对应文档图像理解中的核心痛点——高分辨率输入导致的计算冗余与幻觉问题。尤其适用于处理复杂排版、密集文本的文档图像，可有效缓解OCR幻觉和上下文过载问题。其视觉压缩范式具备成为DIU领域基础组件的潜力，属于底层架构创新。
Abstract Multimodal Large Language Models (MLLMs) have demonstrated substantial value in unified text-image understanding and reasoning, primarily by converting images into sequences of patch-level tokens that align with their architectural paradigm. However, patch-level tokenization leads to a quadratic growth in image tokens, burdening MLLMs' understanding and reasoning with enormous computation and memory. Additionally, the traditional patch-wise scanning tokenization workflow misaligns with the human vision cognition system, further leading to hallucination and computational redundancy. To address this issue, we propose an object-level token merging strategy for Adaptive Token compression, revealing the consistency with human vision system. The experiments are conducted on multiple comprehensive benchmarks, which show that our approach averagely, utilizes only 10% tokens while achieving almost 96% of the vanilla model's performance. More extensive experimental results in comparison with relevant works demonstrate the superiority of our method in balancing compression ratio and performance. Our code will be available.
Authors Xinliang Zhang, Lei Zhu, Hangzhou He, Shuang Zeng, Ourui Fu, Jiakui Hu, Zhengjian Yao, Yanye Lu
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
9
Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats NeurIPS 2025
Title CN 干预所有路径：跨对齐格式统一缓解大视觉语言模型幻觉
Keywords 视觉语言模型 · 幻觉抑制 · 因果干预 · 多模态对齐 · Transformer架构
Summary 本文提出了一种基于Transformer因果结构的统一干预框架Intervene-All-Paths，用于跨对齐格式全面缓解LVLM的幻觉问题。研究发现，幻觉源于多个因果路径的交互，并且模型依赖的路径随问答格式变化而变化。据此设计了针对判别式与生成式任务的路径特异性干预策略，在多个基准上实现了稳定的幻觉抑制效果。
Reason 该论文系统性地分析了大视觉语言模型（LVLM）中幻觉产生的多路径机制，首次揭示图像到输入文本、图像到输出文本和文本到文本通路在不同问答对齐格式下的协同作用。其提出的针对不同通路中关键‘幻觉头’的干预方法，可直接迁移至文档图像理解（DIU）领域，用于缓解OCR幻觉与语义生成不一致问题。尤其适用于复杂排版文档中的细粒度推理任务，具备成为通用去幻觉工具的潜力。
Abstract Despite their impressive performance across a wide range of tasks, Large Vision-Language Models (LVLMs) remain prone to hallucination. In this study, we propose a comprehensive intervention framework aligned with the transformer's causal architecture in LVLMs, integrating the effects of different intervention paths on hallucination. We find that hallucinations in LVLMs do not arise from a single causal path, but rather from the interplay among image-to-input-text, image-to-output-text, and text-to-text pathways. For the first time, we also find that LVLMs rely on different pathways depending on the question-answer alignment format. Building on these insights, we propose simple yet effective methods to identify and intervene on critical hallucination heads within each pathway, tailored to discriminative and generative formats. Experiments across multiple benchmarks demonstrate that our approach consistently reduces hallucinations across diverse alignment types.
Authors Jiaye Qian, Ge Zheng, Yuchen Zhu, Sibei Yang
Comment Accepted to NeurIPS 2025, Project Page: https://github.com/SooLab/AllPath
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
9
DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding
Title CN DocSLM：一种用于长多模态文档理解的小型视觉-语言模型
Keywords 文档图像理解 · 视觉语言模型 · 视觉token压缩 · 长文档处理 · 边缘计算
Summary DocSLM是一种面向资源受限边缘设备的高效小型视觉语言模型，专为长多模态文档理解设计。它通过分层多模态压缩器将每页的视觉、文本和布局信息压缩为固定长度序列，大幅降低内存占用；同时引入基于熵的流式弃权机制，实现对长文档的顺序处理与低置信响应过滤。在多个长文档基准上，DocSLM在参数量、视觉token和延迟均显著降低的情况下，性能达到或超过现有SOTA方法，适合轻量级部署。
Reason 该论文针对长文档理解中的高内存消耗和部署困难问题，提出了专为文档设计的小型视觉语言模型DocSLM。其核心创新——分层多模态压缩器（Hierarchical Multimodal Compressor）实现了视觉、文本与布局信息的联合压缩，显著减少视觉token数量（减少82%），同时保持全局与局部语义，直接回应了DIU领域中‘高分辨率输入处理’与‘视觉token效率’的核心痛点。此外，流式弃权机制（Streaming Abstention）通过不确定性校准实现对长文档分段处理时的低置信度响应过滤，有助于缓解长文档推理中的幻觉问题。该方法属于VLM for Doc方向中具有工程与架构双重意义的扎实工作，且其视觉压缩思想可迁移至其他OCR-free或端到端文档理解系统。
Abstract Large Vision-Language Models (LVLMs) have demonstrated strong multimodal reasoning capabilities on long and complex documents. However, their high memory footprint makes them impractical for deployment on resource-constrained edge devices. We present DocSLM, an efficient Small Vision-Language Model designed for long-document understanding under constrained memory resources. DocSLM incorporates a Hierarchical Multimodal Compressor that jointly encodes visual, textual, and layout information from each page into a fixed-length sequence, greatly reducing memory consumption while preserving both local and global semantics. To enable scalable processing over arbitrarily long inputs, we introduce a Streaming Abstention mechanism that operates on document segments sequentially and filters low-confidence responses using an entropy-based uncertainty calibrator. Across multiple long multimodal document benchmarks, DocSLM matches or surpasses state-of-the-art methods while using 82\% fewer visual tokens, 75\% fewer parameters, and 71\% lower latency, delivering reliable multimodal document understanding on lightweight edge devices. Code and Model are available in https://github.com/Tanveer81/DocSLM.git.
Authors Tanveer Hannan, Dimitrios Mallios, Parth Pathak, Faegheh Sardari, Thomas Seidl, Gedas Bertasius, Mohsen Fayyaz, Sunando Sengupta
Categories Computer Vision and Pattern Recognition
8
Monet: Reasoning in Latent Visual Space Beyond Images and Language
Title CN Monet：超越图像与语言的潜在视觉空间推理
Keywords 潜在视觉推理 · 多模态大模型 · 视觉思维 · VLPO · 视觉表征学习
Summary 本文提出Monet，一种让多模态大模型在潜在视觉空间中进行推理的训练框架。通过三阶段蒸馏式SFT和新提出的VLPO强化学习算法，模型能生成连续的潜在视觉嵌入作为中间推理步骤，实现更接近人类的抽象视觉思维。作者构建了包含12.5万条图文交错CoT的数据集Monet-SFT-125K，并验证了模型在现实感知、图表理解和几何推理等任务上的优越性能与泛化能力。
Reason 该论文提出在多模态大模型的潜在视觉空间中生成连续嵌入作为中间视觉思维，实现‘用视觉表征思考’。其核心创新在于训练框架和VLPO强化学习算法，使模型能在不依赖外部工具的情况下进行抽象视觉推理。这种方法对文档图像理解（DIU）具有显著迁移价值：例如，在复杂排版或表格识别中，传统方法受限于OCR输出的离散符号和固定布局，容易产生幻觉或丢失结构信息；而通过在潜在视觉空间中构建‘视觉思维’，可保留细粒度的空间语义，并支持多步推理过程中的视觉状态演化，从而缓解OCR幻觉、提升长文档或多阶段逻辑任务（如DocVQA）中的推理一致性。
Abstract "Thinking with images" has emerged as an effective paradigm for advancing visual reasoning, extending beyond text-only chains of thought by injecting visual evidence into intermediate reasoning steps. However, existing methods fall short of human-like abstract visual thinking, as their flexibility is fundamentally limited by external tools. In this work, we introduce Monet, a training framework that enables multimodal large language models (MLLMs) to reason directly within the latent visual space by generating continuous embeddings that function as intermediate visual thoughts. We identify two core challenges in training MLLMs for latent visual reasoning: high computational cost in latent-vision alignment and insufficient supervision over latent embeddings, and address them with a three-stage distillation-based supervised fine-tuning (SFT) pipeline. We further reveal a limitation of applying GRPO to latent reasoning: it primarily enhances text-based reasoning rather than latent reasoning. To overcome this, we propose VLPO (Visual-latent Policy Optimization), a reinforcement learning method that explicitly incorporates latent embeddings into policy gradient updates. To support SFT, we construct Monet-SFT-125K, a high-quality text-image interleaved CoT dataset containing 125K real-world, chart, OCR, and geometry CoTs. Our model, Monet-7B, shows consistent gains across real-world perception and reasoning benchmarks and exhibits strong out-of-distribution generalization on challenging abstract visual reasoning tasks. We also empirically analyze the role of each training component and discuss our early unsuccessful attempts, providing insights for future developments in visual latent reasoning. Our model, data, and code are available at https://github.com/NOVAglow646/Monet.
Authors Qixun Wang, Yang Shi, Yifei Wang, Yuanxing Zhang, Pengfei Wan, Kun Gai, Xianghua Ying, Yisen Wang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
8
Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning AAAI 2026
Title CN 连接点：通过代理式推理实现无需训练的视觉定位
Keywords 视觉定位 · 代理推理 · 零样本学习 · 多模态大模型 · 无需微调
Summary 本文提出了GroundingAgent，一种无需任何任务特定微调的视觉定位新框架。它利用预训练的开放词汇检测器、多模态大语言模型和纯语言模型，通过结构化的迭代推理机制，结合语义与空间分析逐步缩小候选区域范围。在RefCOCO系列基准上实现了65.1%的零样本定位准确率，且推理过程具有高度可解释性。即使仅使用LLM进行选择，准确率也可达约90%，接近有监督方法。
Reason 该工作提出了一种无需微调的视觉定位框架GroundingAgent，通过结合开放词汇目标检测器、多模态大模型和纯语言模型，以迭代推理方式联合优化语义与空间匹配。其核心价值在于‘代理式推理’机制可直接迁移至文档图像理解中的细粒度定位任务——例如在DocVQA中精确定位问题所指的表格单元或文本片段，尤其适用于长文档中跨区域信息整合场景。此外，该方法避免了对标注数据的依赖，符合DIU领域向弱监督/零样本演进的趋势。
Abstract Visual grounding, the task of linking textual queries to specific regions within images, plays a pivotal role in vision-language integration. Existing methods typically rely on extensive task-specific annotations and fine-tuning, limiting their ability to generalize effectively to novel or out-of-distribution scenarios. To address these limitations, we introduce GroundingAgent, a novel agentic visual grounding framework that operates without any task-specific fine-tuning. GroundingAgent employs a structured, iterative reasoning mechanism that integrates pretrained open-vocabulary object detectors, multimodal large language models (MLLMs), and large language models (LLMs) to progressively refine candidate regions through joint semantic and spatial analyses. Remarkably, GroundingAgent achieves an average zero-shot grounding accuracy of 65.1 % on widely-used benchmarks (RefCOCO, RefCOCO+, RefCOCOg), entirely without fine-tuning. Furthermore, by substituting MLLM-generated captions with the original query texts, the accuracy at the selection stage alone reaches approximately 90 %, closely matching supervised performance and underscoring the critical role of LLM reasoning capabilities. GroundingAgent also offers strong interpretability, transparently illustrating each reasoning step and providing clear insights into its decision-making process.
Authors Liqin Luo, Guangyao Chen, Xiawu Zheng, Yongxing Dai, Yixiong Zou, Yonghong Tian
Comment AAAI 2026
Categories Computer Vision and Pattern Recognition
8
Vision Remember: Recovering Visual Information in Efficient LVLM with Vision Feature Resampling
Title CN 视觉记忆：通过视觉特征重采样在高效大型视觉语言模型中恢复视觉信息
Keywords 视觉特征重采样 · 视觉语言模型 · 视觉token压缩 · 细粒度定位 · 多模态对齐
Summary 为解决大型视觉语言模型中因视觉token压缩导致的细粒度信息丢失问题，本文提出Vision Remember方法，通过在LLM解码器层引入视觉特征重采样机制，包含Token-Feature交叉注意力和Token双向自注意力模块，实现视觉信息的动态恢复。实验表明该方法在多个视觉理解基准上显著优于现有高效LVLM方案，尤其在需要精细空间感知的任务中表现突出。
Reason 该论文提出了一种在LVLM解码器层中重采样原始视觉特征以恢复因视觉压缩而丢失的细粒度空间信息的方法。其核心思想——通过跨层特征重融合来补偿压缩损失——可直接迁移至文档图像理解（DIU）领域，尤其是应对OCR幻觉和表格/图表理解中的高分辨率需求。当前DIU主流趋势如DeepSeek-OCR依赖视觉token压缩，但面临细节丢失问题，本文的Token-Feature Cross-Attention与多级融合机制为‘如何在保持效率的同时恢复关键视觉信息’提供了新路径，具有明确的架构级迁移价值。
Abstract The computational expense of redundant vision tokens in Large Vision-Language Models (LVLMs) has led many existing methods to compress them via a vision projector. However, this compression may lose visual information that is crucial for tasks relying on fine-grained spatial relationships, such as OCR and Chart&Table Understanding. In this paper, we propose to resample original vision features across the LLM decoder layers to recover visual information and attain efficiency. Following this principle, we introduce Vision Remember, which includes two key modules: (1) Token-Feature Cross-Attention Layer and (2) Token Bidirectional Self-Attention Layer. In the Token bidirectional attention, we employ self-attention mechanism to maintain the bidirectional interaction between vision tokens and the text-guided token. In the Token-Feature interaction attention, we introduce local cross-attention to resample the visual feature and utilize the multi-level fusion to enrich the visual representation. We conduct comprehensive experiments on multiple visual understanding benchmarks and the results with the LLaVA-NeXT baseline show that Vision Remember outperforms TokenPacker by +2.7 and FastV by +5.7 across nearly all the settings. Compared with previous vision feature re-fusion methods, our approach also surpasses DeepStack by +3.9 and SVA Aggregator by +3.4 on the same baseline. The experimental results validate the generalization capability of the proposed method when combined with various efficient vision projectors and LVLMs.
Authors Ze Feng, Jiang-jiang Liu, Sen Yang, Lingyu Xiao, Zhibin Quan, Zhenhua Feng, Wankou Yang, Jingdong Wang
Categories Computer Vision and Pattern Recognition
8
Searching Latent Program Spaces NeurIPS 2025
Title CN 搜索潜在程序空间
Keywords 程序合成 · 测试时搜索 · 潜在空间 · 神经符号系统 · 泛化
Summary 本文提出Latent Program Network（LPN），一种将测试时搜索内建于神经网络的新架构。LPN学习一个隐式的程序空间，通过梯度在测试时对该空间进行搜索，从而实现对新任务的快速适应。该方法无需预定义领域特定语言（DSL），在编程示例任务和ARC-AGI基准上表现出色，尤其在分布外任务中性能翻倍，展示了神经与符号方法结合的潜力。
Reason 该论文提出Latent Program Network（LPN），在推理阶段通过梯度驱动的搜索机制，在隐式程序空间中进行适应性调整。这一方法属于Test-time Compute的源头创新，其核心思想——在紧凑的潜在空间中进行可微分搜索——可被迁移用于解决文档图像理解中的长文档多步推理与逻辑跳跃问题。例如，在复杂表单或财务报告的DocVQA任务中，模型需执行多跳信息定位和结构化推导，LPN提供的test-time搜索能力可增强VLM在推理时的动态适应性，减少因一次性前向推理导致的幻觉或遗漏，具备明确的DIU迁移价值。
Abstract General intelligence requires systems that acquire new skills efficiently and generalize beyond their training distributions. Although program synthesis approaches have strong generalization power, they face scaling issues due to the large combinatorial spaces that quickly render them impractical, requiring human-generated DSLs or pre-trained priors to narrow this search space. On the other hand, deep learning methods have had high successes, but they lack structured test-time adaptation and rely on heavy stochastic sampling or expensive gradient updates for fine-tuning. In this work, we propose the Latent Program Network (LPN), a novel architecture that builds in test-time search directly into neural models. LPN learns a latent space of implicit programs -- neurally mapping inputs to outputs -- through which it can search using gradients at test time. LPN combines the adaptability of symbolic approaches and the scalability of neural methods. It searches through a compact latent space at test time and bypasses the need for pre-defined domain-specific languages. On a range of programming-by-examples tasks, LPN either outperforms or matches performance compared to in-context learning and test-time training methods. Tested on the ARC-AGI benchmark, we demonstrate that LPN can both learn a compact program space and search through it at test time to adapt to novel tasks. LPN doubles its performance on out-of-distribution tasks when test-time search is switched on.
Authors Matthew V Macfarlane, Clement Bonnet
Comment NeurIPS 2025 spotlight. Code available at https://github.com/clement-bonnet/lpn
Categories Artificial Intelligence
8
Deep Hidden Cognition Facilitates Reliable Chain-of-Thought Reasoning AAAI-26
Title CN 深度隐藏认知促进可靠的思维链推理
Keywords 思维链 · 推理可靠性 · 注意力机制 · 置信度预测 · 多模态推理
Summary 本文提出一种基于模型内在真实性编码的CoT校准方法，利用特定注意力头的激活信号训练信心预测器，结合束搜索动态筛选最优推理路径。实验表明该方法在数学、符号和常识推理任务中显著优于现有基线，在单模态与多模态场景下均表现出更高的准确性和稳定性。同时验证了其在大型推理模型上的适用性，并探讨了模型自我修正能力的作用。
Reason 该论文提出了一种通过模型内部注意力头激活来校准思维链（CoT）推理准确性的新方法，发现特定的注意力头能够可靠反映推理步骤的真实性，并据此训练置信度预测器，在推理过程中动态选择最合理的路径。这一机制可直接迁移至文档图像理解（DIU）中的长文档多步推理任务，例如在复杂排版或跨页信息整合中缓解因中间步骤错误累积导致的幻觉问题，提升VLM在DocVQA任务中的推理可靠性。
Abstract Chain of Thought (CoT) reasoning has demonstrated remarkable deep reasoning capabilities in both large language models (LLMs) and multimodal large language models (MLLMs). However, its reliability is often undermined by the accumulation of errors in intermediate steps. This paper introduces an novel approach to calibrate the CoT reasoning accuracy by leveraging the model's intrinsic veracity encoding. We discover that specific attention head activations reliably reflect the truthfulness of reasoning steps in CoT. Based on this insight, we train a confidence predictor to evaluate the correctness of each reasoning step using these truthfulness-sensitive activations, dynamically selecting the most plausible reasoning path via beam search. Experimental results demonstrate that our method significantly outperforms the state-of-the-art baselines (e.g., Few-Shot CoT, Self-Consistency, and Self-Evaluation Guided Beam Search) across the mathematical, symbolic, and commonsense reasoning tasks, exhibiting superior accuracy and reliability in both unimodal and multimodal settings. We further validate the approach on large reasoning models, confirming its applicability to specialized reasoning models. Additionally, we explore the role of the model's self-correction ability in CoT reasoning. This work provides a novel reliability improvement path for CoT reasoning with broad application potential.
Authors Zijun Chen, Wenbo Hu, Richang Hong
Comment This paper has been accepted by AAAI-26
Categories Artificial Intelligence
8
Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning
Title CN Spatial-SSRL：通过自监督强化学习增强空间理解能力
Keywords 自监督学习 · 强化学习 · 空间理解 · 视觉语言模型 · 预训练任务
Summary Spatial-SSRL是一种用于提升大视觉语言模型空间理解能力的自监督强化学习方法。它通过设计五种无需人工标注的预任务（如图像块重排、翻转识别、深度排序等），从原始图像中自动生成可验证的监督信号，并用于强化学习训练。该方法在多个空间理解基准上显著优于Qwen2.5-VL基线模型，且不损害通用视觉能力。其核心思想——利用图像内在结构构造可验证任务——为低成本、大规模提升模型空间推理提供了新路径。
Reason 该论文提出了一种自监督强化学习框架Spatial-SSRL，通过从普通图像中自动构建可验证的预训练任务（如打乱图像块重排序、翻转识别、深度顺序判断等）来提升LVLM的空间理解能力。这些任务无需人工标注或外部工具即可生成可靠监督信号，解决了传统方法依赖高成本标注的问题。该方法对文档图像理解具有显著迁移价值：文档中的复杂排版、细粒度定位和逻辑结构推理本质上是2D空间关系建模问题。例如，表格结构识别、多栏文本顺序恢复、图文对应关系判断均可视为‘打乱块重排序’或‘区域相对位置预测’的特例。因此，这种基于内在结构生成自监督信号的范式，可被引入DIU领域以增强VLM对文档布局的深层理解，尤其有助于缓解因OCR错误或视觉压缩导致的空间幻觉问题。
Abstract Spatial understanding remains a weakness of Large Vision-Language Models (LVLMs). Existing supervised fine-tuning (SFT) and recent reinforcement learning with verifiable rewards (RLVR) pipelines depend on costly supervision, specialized tools, or constrained environments that limit scale. We introduce Spatial-SSRL, a self-supervised RL paradigm that derives verifiable signals directly from ordinary RGB or RGB-D images. Spatial-SSRL automatically formulates five pretext tasks that capture 2D and 3D spatial structure: shuffled patch reordering, flipped patch recognition, cropped patch inpainting, regional depth ordering, and relative 3D position prediction. These tasks provide ground-truth answers that are easy to verify and require no human or LVLM annotation. Training on our tasks substantially improves spatial reasoning while preserving general visual capabilities. On seven spatial understanding benchmarks in both image and video settings, Spatial-SSRL delivers average accuracy gains of 4.63% (3B) and 3.89% (7B) over the Qwen2.5-VL baselines. Our results show that simple, intrinsic supervision enables RLVR at scale and provides a practical route to stronger spatial intelligence in LVLMs.
Authors Yuhong Liu, Beichen Zhang, Yuhang Zang, Yuhang Cao, Long Xing, Xiaoyi Dong, Haodong Duan, Dahua Lin, Jiaqi Wang
Comment preprint
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
8
CropVLM: Learning to Zoom for Fine-Grained Vision-Language Perception
Title CN CropVLM：学习缩放以实现细粒度视觉-语言感知
Keywords 视觉-语言模型 · 细粒度识别 · 裁剪放大 · 强化学习 · 外部推理增强
Summary CropVLM是一种外部轻量级模块，通过强化学习训练一个裁剪策略网络，使VLM能够动态聚焦于图像中最相关的局部区域，从而提升其在细粒度视觉语言任务上的表现。该方法不依赖人工标注边界框，也不需对目标VLM进行微调，具备良好的通用性和跨域适应能力，在场景文字识别和文档理解等需要高分辨率感知的任务上表现出显著增益。
Reason 该论文提出了一种无需修改或微调基础VLM即可提升其对高分辨率图像中细粒度内容理解能力的方法，通过强化学习让模型学会自动‘裁剪放大’关键区域。这一机制可直接迁移至文档图像理解（DIU）领域，用于缓解复杂排版下的文本识别困难、局部信息遗漏等问题，尤其适用于处理高分辨率文档中的小字号、密集表格或印章等细节元素，有效增强VLM在DocVQA和布局分析任务中的感知能力。
Abstract Vision-Language Models (VLMs) often struggle with tasks that require fine-grained image understanding, such as scene-text recognition or document analysis, due to perception limitations and visual fragmentation. To address these challenges, we introduce CropVLM as an external low-cost method for boosting performance, enabling VLMs to dynamically ''zoom in'' on relevant image regions, enhancing their ability to capture fine details. CropVLM is trained using reinforcement learning, without using human-labeled bounding boxes as a supervision signal, and without expensive synthetic evaluations. The model is trained once and can be paired with both open-source and proprietary VLMs to improve their performance. Our approach delivers significant improvements on tasks that require high-resolution image understanding, notably for benchmarks that are out-of-domain for the target VLM, without modifying or fine-tuning the VLM, thus avoiding catastrophic forgetting.
Authors Miguel Carvalho, Helder Dias, Bruno Martins
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
8
When to Think and When to Look: Uncertainty-Guided Lookback
Title CN 何时思考与何时查看：基于不确定性的回看机制
Keywords 视觉语言模型 · 测试时推理 · 视觉接地 · 去幻觉 · 解码策略
Summary 本文首次系统分析了测试时思考（test-time thinking）对大型视觉语言模型（LVLM）视觉推理的影响，发现过长的推理链反而会导致模型脱离图像内容，产生错误路径。研究发现成功的推理轨迹中常包含明确回看图像的短语。基于此，作者提出了不确定性引导的回看机制，利用不确定性信号触发图像重聚焦，并结合广度搜索优化推理路径。该方法在MMMU等六个基准上均取得提升，尤其在传统推理表现弱的类别中增益明显。
Reason 该论文提出了一种训练-free的解码策略——不确定性引导的回看（uncertainty-guided lookback），通过结合不确定性信号和自适应的图像回看提示，在多步推理过程中动态决定何时依赖语言推理、何时重新关注视觉输入。这一方法直接针对LVLM在视觉推理中容易忽略图像、产生错误推理轨迹的问题，显著提升了视觉 grounding 能力。对于文档图像理解（DIU）任务，尤其是DocVQA中的长文档多跳推理和复杂排版下的信息定位，此类机制可有效缓解模型‘遗忘图像’导致的幻觉问题，具备强迁移价值。
Abstract Test-time thinking (that is, generating explicit intermediate reasoning chains) is known to boost performance in large language models and has recently shown strong gains for large vision language models (LVLMs). However, despite these promising results, there is still no systematic analysis of how thinking actually affects visual reasoning. We provide the first such analysis with a large scale, controlled comparison of thinking for LVLMs, evaluating ten variants from the InternVL3.5 and Qwen3-VL families on MMMU-val under generous token budgets and multi pass decoding. We show that more thinking is not always better; long chains often yield long wrong trajectories that ignore the image and underperform the same models run in standard instruct mode. A deeper analysis reveals that certain short lookback phrases, which explicitly refer back to the image, are strongly enriched in successful trajectories and correlate with better visual grounding. Building on this insight, we propose uncertainty guided lookback, a training free decoding strategy that combines an uncertainty signal with adaptive lookback prompts and breadth search. Our method improves overall MMMU performance, delivers the largest gains in categories where standard thinking is weak, and outperforms several strong decoding baselines, setting a new state of the art under fixed model families and token budgets. We further show that this decoding strategy generalizes, yielding consistent improvements on five additional benchmarks, including two broad multimodal suites and math focused visual reasoning datasets.
Authors Jing Bi, Filippos Bellos, Junjia Guo, Yayuan Li, Chao Huang, Yolo Y. Tang, Luchuan Song, Susan Liang, Zhongfei Mark Zhang, Jason J. Corso, Chenliang Xu
Categories Computer Vision and Pattern Recognition, Computation and Language
8
Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention
Title CN 告诉模型看哪里：通过视觉引导注意力减轻多模态大语言模型中的幻觉
Keywords 视觉引导注意力 · 幻觉缓解 · 多模态大模型 · 视觉定位 · 训练-free干预
Summary 本文提出视觉引导注意力（VGA），一种无需训练的多模态大模型幻觉缓解方法。该方法通过解析视觉token的语义信息建立精准的视觉 grounding，并用其指导模型注意力聚焦于关键区域。在图像描述生成中，VGA还能动态抑制已描述区域，避免重复或虚构。实验表明VGA在多个MLLM上显著降低幻觉，且仅引入极低延迟。该技术可直接应用于文档理解中的视觉-语言对齐优化。
Reason 该论文提出了一种训练-free的视觉引导注意力（VGA）方法，通过利用视觉token的语义构建精确的视觉定位，并在推理过程中引导模型关注相关视觉区域，从而缓解MLLMs的幻觉问题。这一机制对文档图像理解（DIU）具有直接迁移价值，尤其是在DocVQA任务中，可有效抑制因错误视觉注意力导致的OCR幻觉或内容误读。此外，其低延迟和兼容FlashAttention的特点，使其适用于高分辨率文档输入下的细粒度识别场景。
Abstract Visual attention serves as the primary mechanism through which MLLMs interpret visual information; however, its limited localization capability often leads to hallucinations. We observe that although MLLMs can accurately extract visual semantics from visual tokens, they fail to fully leverage this advantage during subsequent inference. To address this limitation, we propose Vision-Guided Attention (VGA), a training-free method that first constructs precise visual grounding by exploiting the semantic content of visual tokens, and then uses this grounding to guide the model's focus toward relevant visual regions. In image captioning, VGA further refines this guidance dynamically during generation by suppressing regions that have already been described. In VGA, each token undergoes only a single forward pass, introducing a negligible latency overhead of just 4.36\%. In addition, VGA is fully compatible with efficient attention implementations such as FlashAttention. Extensive experiments across diverse MLLMs and multiple hallucination benchmarks demonstrate that VGA achieves state-of-the-art dehallucination performance. Further analysis confirms that explicit visual guidance plays a crucial role in enhancing the visual understanding capabilities of MLLMs.
Authors Jianfei Zhao, Feng Zhang, Xin Sun, Chong Feng, Zhixing Tan
Comment Under Review
Categories Computer Vision and Pattern Recognition
8
HybriDLA: Hybrid Generation for Document Layout Analysis AAAI 2026
Title CN HybriDLA：用于文档布局分析的混合生成方法
Keywords 文档布局分析 · 扩散模型 · 自回归生成 · 多尺度特征融合 · 生成式检测
Summary HybriDLA是一种创新的文档布局分析框架，结合了扩散模型和自回归生成的优势，能够动态、迭代地生成并优化文档区域的边界框，同时融入语义上下文信息。其设计克服了传统基于固定查询的方法在面对复杂、多样布局时的性能瓶颈。实验表明，该方法在DocLayNet和M$^6$Doc数据集上达到SOTA性能，mAP达83.5%。
Reason 该论文针对文档布局分析（DLA）任务，提出了一种融合扩散模型与自回归解码的新型生成框架，解决了传统方法在处理现代复杂、多变布局时因固定查询数和单次前向推理导致的局限性。通过扩散模块迭代优化边界框假设，并结合自回归模块引入语义与上下文信息，显著提升了对多样化文档结构的适应能力。此外，多尺度特征融合编码器增强了细粒度与高层视觉特征的捕获，进一步提高检测精度。此方法为DIU领域中复杂排版理解提供了新的建模范式，尤其对长文档、异构布局的精准解析具有迁移价值。
Abstract Conventional document layout analysis (DLA) traditionally depends on empirical priors or a fixed set of learnable queries executed in a single forward pass. While sufficient for early-generation documents with a small, predetermined number of regions, this paradigm struggles with contemporary documents, which exhibit diverse element counts and increasingly complex layouts. To address challenges posed by modern documents, we present HybriDLA, a novel generative framework that unifies diffusion and autoregressive decoding within a single layer. The diffusion component iteratively refines bounding-box hypotheses, whereas the autoregressive component injects semantic and contextual awareness, enabling precise region prediction even in highly varied layouts. To further enhance detection quality, we design a multi-scale feature-fusion encoder that captures both fine-grained and high-level visual cues. This architecture elevates performance to 83.5% mean Average Precision (mAP). Extensive experiments on the DocLayNet and M$^6$Doc benchmarks demonstrate that HybriDLA sets a state-of-the-art performance, outperforming previous approaches. All data and models will be made publicly available at https://yufanchen96.github.io/projects/HybriDLA.
Authors Yufan Chen, Omar Moured, Ruiping Liu, Junwei Zheng, Kunyu Peng, Jiaming Zhang, Rainer Stiefelhagen
Comment Accepted by AAAI 2026 (Oral). Project page at https://yufanchen96.github.io/projects/HybriDLA
Categories Computer Vision and Pattern Recognition
8
Reading Between the Lines: Abstaining from VLM-Generated OCR Errors via Latent Representation Probes
Title CN 透过表象阅读：通过潜在表示探测避免VLM生成的OCR错误
Keywords VLM · OCR去幻觉 · 潜在表示探测 · 拒答机制 · 不确定性估计
Summary 本文提出Latent Representation Probing（LRP）方法，通过在VLM的隐藏状态或注意力图上训练轻量级探测器，以识别模型在OCR相关任务中的不确定性并实现自动拒答。实验表明该方法在多个图像和视频基准上显著优于现有基线，且最佳信号来源于中间层而非最终输出层，为构建可靠AI系统提供了新路径。
Reason 该论文针对STVQA任务中VLM因OCR错误导致的幻觉问题，提出通过在模型内部隐状态中训练轻量级探测器（Probe）来识别不确定性，从而实现拒答机制。其核心创新在于不依赖输出概率或语义一致性，而是从中间层的隐藏状态或注意力模式中挖掘置信度信号，解决了传统方法在OCR任务上校准性差的问题。该方法可直接迁移到文档图像理解（DIU）中的DocVQA任务，用于检测模型对文本识别结果的可信度，尤其适用于缓解OCR幻觉带来的错误答案，在金融、法律等高可靠性要求场景下具有重要应用价值。
Abstract As VLMs are deployed in safety-critical applications, their ability to abstain from answering when uncertain becomes crucial for reliability, especially in Scene Text Visual Question Answering (STVQA) tasks. For example, OCR errors like misreading "50 mph" as "60 mph" could cause severe traffic accidents. This leads us to ask: Can VLMs know when they can't see? Existing abstention methods suggest pessimistic answers: they either rely on miscalibrated output probabilities or require semantic agreement unsuitable for OCR tasks. However, this failure may indicate we are looking in the wrong place: uncertainty signals could be hidden in VLMs' internal representations. Building on this insight, we propose Latent Representation Probing (LRP): training lightweight probes on hidden states or attention patterns. We explore three probe designs: concatenating representations across all layers, aggregating attention over visual tokens, and ensembling single layer probes by majority vote. Experiments on four benchmarks across image and video modalities show LRP improves abstention accuracy by 7.6\% over best baselines. Our analysis reveals: probes generalize across various uncertainty sources and datasets, and optimal signals emerge from intermediate rather than final layers. This establishes a principled framework for building deployment-ready AI systems by detecting confidence signals from internal states rather than unreliable outputs.
Authors Jihan Yao, Achin Kulshrestha, Nathalie Rauschmayr, Reed Roberts, Banghua Zhu, Yulia Tsvetkov, Federico Tombari
Categories Computer Vision and Pattern Recognition
8
Text-Guided Semantic Image Encoder
Title CN 文本引导的语义图像编码器
Keywords 文本引导编码 · 视觉语言模型 · 文档视觉问答 · 细粒度定位 · 推理效率
Summary 本文提出Text-Guided Semantic Image Encoder（TIE），一种基于文本查询条件化生成图像表示的新架构。与传统独立预训练图像编码器不同，TIE在编码阶段即引入文本引导，使视觉特征提取更具任务针对性。实验表明，TIE在多个图像到文本任务中优于传统VLM，尤其在DocVQA和InfoVQA上表现突出，同时将所需图像token减少一半，显著提升推理效率。定性分析显示TIE能准确聚焦于与问题相关的文档区域，增强了解释性和接地能力。
Reason 该论文提出了一种文本引导的图像编码器TIE，能够根据输入的文本查询生成条件化的图像表示。在DocVQA和InfoVQA等文档视觉问答任务上取得了显著提升（最高+6分），且仅用一半图像块（tokens）就实现了更优性能，说明其在细粒度视觉定位和推理方面具有优势。该方法通过文本引导注意力机制，增强了模型对查询相关区域的关注，可有效缓解文档理解中的幻觉问题，并提升多模态对齐能力，具备向DIU领域迁移的高价值。
Abstract Image encoders, a fundamental component of vision-language models (VLMs), are typically pretrained independently before being aligned with a language model. This standard paradigm results in encoders that process images agnostically, without regard to the specific downstream task or text query. To address this limitation, we propose the Text-Guided Semantic Image Encoder (TIE), which generates image representations conditioned on the input text query. VLMs equipped with TIE outperform their conventional counterparts by +1.5 and +1.3 points on average across nine image-to-text benchmarks at the 1B and 3B scales, respectively, with gains reaching up to 6 points on tasks such as DocVQA and InfoVQA. Moreover, TIE-based VLMs attain superior performance while utilizing only half as many image tiles (tokens), resulting in notably improved inference efficiency. TIE also generalizes well with generic queries, indicating that text-conditioned training effectively optimizes the encoder to capture key visual features. Qualitative analysis confirms that TIE consistently attends to query-relevant regions, enhancing both interpretability and query-specific grounding.
Authors Raghuveer Thirukovalluru, Xiaochuang Han, Bhuwan Dhingra, Emily Dinan, Maha Elbayad
Comment 20 pages, 6 figures
Categories Computer Vision and Pattern Recognition
8
Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning
Title CN 重新思考扩展测试时计算资源下的微调：限制置信度可提升数学推理能力
Keywords 测试时计算 · 过自信 · 训练-推理协同设计 · 数学推理 · 搜索策略
Summary 本文发现使用交叉熵损失训练的大语言模型在配合pass@N等测试时计算策略时会出现性能随训练加深而下降的现象，根源在于模型变得过于自信，抑制了采样多样性。为此，作者提出一种限制置信度的训练损失，显著提升了模型在数学问答和定理证明等任务上的搜索有效性。这项工作强调了训练策略与测试时推理机制联合设计的重要性。
Reason 该论文提出在训练阶段通过修改损失函数来限制模型置信度，从而更好地与测试时的搜索策略（如pass@N）对齐。这一方法论的核心思想——缓解模型过自信以提升搜索效率——可直接迁移到文档图像理解中的复杂推理任务。例如，在DocVQA中，VLM常因过度自信产生OCR幻觉或错误推理路径，结合test-time compute（如多路径答案搜索）时性能反而下降。本文提出的训练-推理协同设计思路为DIU系统在长文档多步推理、抗幻觉验证机制等方面提供了新的优化方向。
Abstract Recent progress in large language models (LLMs) highlights the power of scaling test-time compute to achieve strong performance on complex tasks, such as mathematical reasoning and code generation. This raises a critical question: how should model training be modified to optimize performance under a subsequent test-time compute strategy and budget? To explore this, we focus on pass@N, a simple test-time strategy that searches for a correct answer in $N$ independent samples. We show, surprisingly, that training with cross-entropy (CE) loss can be ${\it misaligned}$ with pass@N in that pass@N accuracy ${\it decreases}$ with longer training. We explain the origins of this misalignment in terms of model overconfidence induced by CE, and experimentally verify our prediction of overconfidence as an impediment to scaling test-time compute via pass@N. Furthermore we suggest a principled, modified training loss that is better aligned to pass@N by limiting model confidence and rescuing pass@N test performance. Our algorithm demonstrates improved mathematical reasoning on MATH and MiniF2F benchmarks under several scenarios: (1) providing answers to math questions; and (2) proving theorems by searching over proof trees of varying shapes. Overall our work underscores the importance of co-designing two traditionally separate phases of LLM development: training-time protocols and test-time search and reasoning strategies.
Authors Feng Chen, Allan Raventos, Nan Cheng, Surya Ganguli, Shaul Druckmann
Categories Artificial Intelligence
8
Parallel Vision Token Scheduling for Fast and Accurate Multimodal LMMs Inference
Title CN 用于快速准确多模态大语言模型推理的并行视觉令牌调度
Keywords 视觉令牌压缩 · 高效推理 · 多模态大模型 · 高分辨率处理 · 无需训练
Summary 本文提出了ParVTS，一种无需训练的并行视觉令牌调度框架，通过将视觉令牌分为主体和非主体路径并行处理，并在推理过程中提前丢弃非主体路径，有效减少了多模态大语言模型的计算开销。该方法在多个MLLM上实现了最高88.9%的视觉令牌剪枝率，带来1.77倍速度提升和70%的FLOPs降低，且对性能影响极小。
Reason 该论文提出了一种无需训练的视觉令牌调度框架ParVTS，将视觉令牌分为主体和非主体两组并行处理，并在推理中期丢弃非主体路径以减少计算量。这种方法显著降低了高分辨率图像带来的长序列自注意力计算负担，同时保留了关键语义信息。对于文档图像理解（DIU）而言，该方法可直接迁移用于提升VLM处理高分辨率文档（如扫描PDF、复杂排版页面）时的效率与细粒度识别能力，尤其适用于需精细定位文字区域或表格结构的任务，缓解因视觉令牌过多导致的延迟和内存压力。
Abstract Multimodal large language models (MLLMs) deliver impressive vision-language reasoning but suffer steep inference latency because self-attention scales quadratically with sequence length and thousands of visual tokens contributed by high-resolution images. Naively pruning less-informative visual tokens reduces this burden, yet indiscriminate removal can strip away contextual cues essential for background or fine-grained questions, undermining accuracy. In this paper, we present ParVTS (Parallel Vision Token Scheduling), a training-free scheduling framework that partitions visual tokens into subject and non-subject groups, processes them in parallel to transfer their semantics into question tokens, and discards the non-subject path mid-inference to reduce computation. This scheduling reduces computational complexity, requires no heuristics or additional modules, and is compatible with diverse existing MLLM architectures. Experiments across multiple MLLM backbones show that ParVTS prunes up to 88.9% of visual tokens with minimal performance drop, achieving 1.77x speedup and 70% FLOPs reduction.
Authors Wengyi Zhan, Mingbao Lin, Zhihang Lin, Rongrong Ji
Categories Computer Vision and Pattern Recognition
8
Systematic Reward Gap Optimization for Mitigating VLM Hallucinations NeurIPS 2025
Title CN 系统性奖励差距优化以缓解视觉语言模型的幻觉
Keywords 视觉语言模型 · 幻觉缓解 · 偏好优化 · 奖励差距 · 数据构建
Summary 本文提出TPR框架，通过在主题层面重写VLM响应中的语义内容并引入模型自采样的候选，实现对偏好数据中奖励差距的系统性优化。该方法能精确调控每对偏好样本的语义差异程度，支持渐进式难度调节，从而更有效地引导模型学习真实语义分布。实验表明，TPR在多个幻觉评测基准上达到SOTA，尤其在ObjectHal-Bench上减少93%的幻觉，且具备高数据效率。
Reason 该论文提出了一种名为Topic-level Preference Rewriting（TPR）的新框架，用于在数据构建阶段系统性地优化偏好对中的奖励差距，从而更有效地缓解VLM中的幻觉问题。其核心创新在于通过语义主题级别的重写与重采样，实现对细粒度语义错误的精准控制，并可渐进式调整负样本难度，塑造更有效的学习信号。这一方法可直接迁移至文档图像理解（DIU）领域，尤其是应对OCR幻觉或VLM在复杂排版下生成不准确文本描述的问题。例如，在DocVQA任务中，模型常因视觉误解或冗余信息产生虚构答案，TPR提供的细粒度、可控的偏好数据构造机制，可用于训练更具鲁棒性的文档理解模型，显著降低幻觉率。
Abstract The success of Direct Preference Optimization (DPO) in mitigating hallucinations in Vision Language Models (VLMs) critically hinges on the true reward gaps within preference pairs. However, current methods, typically relying on ranking or rewriting strategies, often struggle to optimize these reward gaps in a systematic way during data curation. A core difficulty lies in precisely characterizing and strategically manipulating the overall reward gap configuration, that is, the deliberate design of how to shape these reward gaps within each preference pair across the data. To address this, we introduce Topic-level Preference Rewriting(TPR), a novel framework designed for the systematic optimization of reward gap configuration. Through selectively replacing semantic topics within VLM responses with model's own resampled candidates for targeted rewriting, TPR can provide topic-level control over fine-grained semantic details. This precise control enables advanced data curation strategies, such as progressively adjusting the difficulty of rejected responses, thereby sculpting an effective reward gap configuration that guides the model to overcome challenging hallucinations. Comprehensive experiments demonstrate TPR achieves state-of-the-art performance on multiple hallucination benchmarks, outperforming previous methods by an average of 20%. Notably, it significantly reduces hallucinations by up to 93% on ObjectHal-Bench, and also exhibits superior data efficiency towards robust and cost-effective VLM alignment. Code and datasets are available at https://tpr-dpo.github.io .
Authors Lehan He, Zeren Chen, Zhelun Shi, Tianyu Yu, Jing Shao, Lu Sheng
Comment 34 pages, 12 figures, Accepted by NeurIPS 2025
Categories Computer Vision and Pattern Recognition, Computation and Language
8
FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models NeurIPS 2025
Title CN FlowCut：通过信息流重新思考视觉语言模型中的冗余问题
Keywords 视觉token剪枝 · 信息流分析 · 视觉语言模型 · 效率优化 · 冗余消除
Summary 大型视觉语言模型在多模态理解上表现优异，但因视觉token冗余而计算成本高昂。现有剪枝方法多依赖单层注意力分数，难以准确识别跨层冗余。本文从信息流动视角出发，发现CLS token可作为信息中继，并揭示冗余是逐层动态形成的。基于此，作者提出FlowCut框架，通过建模跨层信息流来更精准地识别和剪除冗余视觉token。实验表明，该方法在LLaVA系列模型上实现了高达94.4%的token压缩率，同时提升性能，并带来3.2倍预填充阶段加速。
Reason 该论文提出了一种基于信息流分析的视觉token剪枝框架FlowCut，从多层交互的角度识别冗余视觉token，解决了传统单层注意力评分方法在冗余判断上的不一致性。这一方法对文档图像理解（DIU）具有重要迁移价值：文档VLM通常需要处理高分辨率、密集排版的输入，导致视觉token数量庞大、计算开销高；FlowCut可通过高效压缩视觉token，在保留关键布局与文本信息的同时显著降低计算负载，尤其适用于长文档或多页文档的端到端理解场景。
Abstract Large vision-language models (LVLMs) excel at multimodal understanding but suffer from high computational costs due to redundant vision tokens. Existing pruning methods typically rely on single-layer attention scores to rank and prune redundant visual tokens to solve this inefficiency. However, as the interaction between tokens and layers is complicated, this raises a basic question: Is such a simple single-layer criterion sufficient to identify redundancy? To answer this question, we rethink the emergence of redundant visual tokens from a fundamental perspective: information flow, which models the interaction between tokens and layers by capturing how information moves between tokens across layers. We find (1) the CLS token acts as an information relay, which can simplify the complicated flow analysis; (2) the redundancy emerges progressively and dynamically via layer-wise attention concentration; and (3) relying solely on attention scores from single layers can lead to contradictory redundancy identification. Based on this, we propose FlowCut, an information-flow-aware pruning framework, mitigating the insufficiency of the current criterion for identifying redundant tokens and better aligning with the model's inherent behaviors. Extensive experiments show that FlowCut achieves superior results, outperforming SoTA by 1.6% on LLaVA-1.5-7B with 88.9% token reduction, and by 4.3% on LLaVA-NeXT-7B with 94.4% reduction, delivering 3.2x speed-up in the prefilling stage. Our code is available at https://github.com/TungChintao/FlowCut
Authors Jintao Tong, Wenwei Jin, Pengda Qin, Anqi Li, Yixiong Zou, Yuhong Li, Yuhua Li, Ruixuan Li
Comment Accepted by NeurIPS 2025
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
8
Self-Empowering VLMs: Achieving Hierarchical Consistency via Self-Elicited Knowledge Distillation
Title CN 自我赋能的视觉语言模型：通过自激发知识蒸馏实现层次一致性
Keywords 视觉语言模型 · 层次化理解 · 知识蒸馏 · 自监督学习 · 多步推理
Summary 本文发现当前视觉语言模型在层次化视觉问答（Hierarchical VQA）任务中表现不佳，主要原因是无法维持跨层级的状态一致性，而非知识缺失。为此提出Self-Elicited Knowledge Distillation（SEKD），利用同一VLM通过逐步推理生成硬标签、软分布和隐藏状态作为监督信号，训练一个高效的单通路学生模型。该方法无需外部标注，在域内路径一致性上提升高达29.5个百分点，并显著增强零样本泛化能力，为紧凑型VLM赋予多步依赖推理能力提供了可扩展方案。
Reason 该论文针对VLM在层次化理解任务中跨层级状态不一致的问题，提出了一种无需人工标注的自激发知识蒸馏（SEKD）方法。其核心贡献在于通过多步推理作为教师、单步模型作为学生，实现了依赖感知的多步推理能力迁移。该方法可直接迁移到文档图像理解中的长文档结构解析、表格层级识别等任务中，解决因缺乏跨层级状态维护而导致的逻辑断裂问题，尤其适用于需要从粗到细结构预测的DocVQA场景。
Abstract Vision-language models (VLMs) possess rich knowledge but often fail on hierarchical understanding tasks, where the goal is to predict a coarse-to-fine taxonomy path that remains consistent across all levels. We compare three inference paradigms for hierarchical VQA and find that stepwise reasoning, when conditioned on prior answers, significantly outperforms single-pass prompting. Further analysis indicates that the main limitation of current VLMs is their inability to maintain cross-level state, rather than a lack of taxonomic knowledge. Motivated by this diagnosis, we propose Self-Elicited Knowledge Distillation (SEKD), which requires no human labels or external tools: the same VLM is prompted to reason step by step and act as a teacher by exposing its hard labels, soft distributions, and decoder hidden states, while a single-pass student distills these signals. The student VLM remains efficient while approaching the accuracy of its multi-step teacher. It improves in-domain path consistency (HCA) by up to +29.50 percentage points, raises zero-shot HCA on an unseen taxonomy from 4.15% to 42.26%, and yields gains on challenging mathematical benchmarks. Because all supervision is self-elicited, SEKD scales to new taxonomies and datasets without annotation cost, providing a practical route to imbue compact VLMs with dependency-aware multi-step reasoning.
Authors Wei Yang, Yiran Zhu, Zilin Li, Xunjia Zhang, Hongtao Wang
Comment 21 pages, 18 tables, 6 figures
Categories Computer Vision and Pattern Recognition
8
Graph of Verification: Structured Verification of LLM Reasoning with Directed Acyclic Graphs AAAI 2026
Title CN 验证图：基于有向无环图的LLM推理结构化验证
Keywords 推理验证 · 多粒度验证 · 结构化推理 · LLM幻觉 · 无需训练
Summary 本文提出了Graph of Verification（GoV），一种面向大语言模型复杂推理过程的结构化验证框架。通过构建有向无环图并引入可调节粒度的‘节点块’，GoV能根据推理内容的结构特性（如形式化证明或自然语言叙述）自适应地组织验证流程。实验表明，GoV在结构化和非结构化推理任务上均优于整体式和其他分解式验证方法，无需训练即可显著提升推理可靠性。
Reason 该论文提出了一种灵活的、基于有向无环图（DAG）的验证框架GoV，能够自适应地对LLM多步推理过程进行多粒度验证。其核心创新在于‘节点块’机制，可根据推理结构动态调整验证粒度（从原子步骤到整段文本），从而在精度与鲁棒性之间取得平衡。此方法可直接迁移至文档图像理解中的长文档多跳推理任务，例如在复杂报表或法律文书中进行多步骤信息抽取与逻辑验证，有效缓解因中间步骤错误累积导致的幻觉问题。尤其适用于DocVQA中需要结构化推理的场景，具备较强的上游工具价值。
Abstract Verifying the complex and multi-step reasoning of Large Language Models (LLMs) is a critical challenge, as holistic methods often overlook localized flaws. Step-by-step validation is a promising alternative, yet existing methods are often rigid. They struggle to adapt to diverse reasoning structures, from formal proofs to informal natural language narratives. To address this adaptability gap, we propose the Graph of Verification (GoV), a novel framework for adaptable and multi-granular verification. GoV's core innovation is its flexible "node block" architecture. This mechanism allows GoV to adaptively adjust its verification granularity--from atomic steps for formal tasks to entire paragraphs for natural language--to match the native structure of the reasoning process. This flexibility allows GoV to resolve the fundamental trade-off between verification precision and robustness. Experiments on both well-structured and loosely-structured benchmarks demonstrate GoV's versatility. The results show that GoV's adaptive approach significantly outperforms both holistic baselines and other state-of-the-art decomposition-based methods, establishing a new standard for training-free reasoning verification.
Authors Jiwei Fang, Bin Zhang, Changwei Wang, Jin Wan, Zhiwei Xu
Comment Accepted to AAAI 2026
Categories Artificial Intelligence
8
L2V-CoT: Cross-Modal Transfer of Chain-of-Thought Reasoning via Latent Intervention AAAI 2026
Title CN L2V-CoT：通过潜在干预实现思维链推理的跨模态迁移
Keywords 思维链迁移 · 潜在干预 · 视觉语言模型 · 无训练推理增强 · 跨模态学习
Summary 本文提出L2V-CoT，一种无需训练的跨模态思维链迁移方法。通过线性人工层析成像（LAT）发现LLM和VLM在低频潜在空间中具有相似的CoT表示结构，进而在频率域对LLM的CoT特征进行提取与重采样，并注入到VLM中以增强其多步推理能力。实验表明该方法优于无训练基线，甚至超过部分监督方法。
Reason 该论文提出了一种无需训练的潜在干预方法L2V-CoT，将语言模型中的低频思维链（CoT）表示迁移到视觉语言模型中，从而增强VLM在多步推理任务中的表现。其核心贡献在于揭示了LLM与VLM在低频潜在空间中存在共享的推理结构，并通过频率域操作实现跨模态注入。这一方法可直接迁移至文档图像理解（DIU）领域，用于缓解长文档推理中的逻辑断裂问题，或抑制OCR-free VLM因视觉误解导致的幻觉性推理。尤其适用于DocVQA中需要多跳推理的复杂场景，是一种具有明确迁移路径的上游方法创新。
Abstract Recently, Chain-of-Thought (CoT) reasoning has significantly enhanced the capabilities of large language models (LLMs), but Vision-Language Models (VLMs) still struggle with multi-step reasoning tasks due to limited multimodal reasoning data. To bridge this gap, researchers have explored methods to transfer CoT reasoning from LLMs to VLMs. However, existing approaches either need high training costs or require architectural alignment. In this paper, we use Linear Artificial Tomography (LAT) to empirically show that LLMs and VLMs share similar low-frequency latent representations of CoT reasoning despite architectural differences. Based on this insight, we propose L2V-CoT, a novel training-free latent intervention approach that transfers CoT reasoning from LLMs to VLMs. L2V-CoT extracts and resamples low-frequency CoT representations from LLMs in the frequency domain, enabling dimension matching and latent injection into VLMs during inference to enhance reasoning capabilities. Extensive experiments demonstrate that our approach consistently outperforms training-free baselines and even surpasses supervised methods.
Authors Yuliang Zhan, Xinyu Tang, Han Wan, Jian Li, Ji-Rong Wen, Hao Sun
Comment AAAI 2026 oral
Categories Computation and Language
8
Extending Test-Time Scaling: A 3D Perspective with Context, Batch, and Turn
Title CN 扩展测试时缩放：基于上下文、批处理和轮次的三维视角
Keywords 测试时缩放 · 推理增强 · 多维缩放 · 大模型推理 · 人类反馈
Summary 本文提出了一种三维测试时缩放方法，通过上下文、批处理和轮次三个维度联合提升大模型在测试阶段的推理能力。实验表明，三者协同能显著提高在IOI、IMO等复杂推理任务上的表现，并可融入人类反馈。该框架为提升VLM在复杂、高噪声场景下的稳健推理提供了新思路。
Reason 该论文提出了一个统一的多维测试时缩放框架——3D测试时缩放，结合上下文长度、批处理（batch）和轮次（turn）三个维度来增强推理模型在测试阶段的表现。虽然其验证任务集中在通用推理（如IMO、CPHO等），但其核心思想对文档图像理解中的长文档推理、复杂逻辑问答（如DocVQA）具有显著迁移价值。例如，在处理长表格或多页合同的理解任务中，可通过批采样多个片段并进行迭代自优化（turn scaling）提升整体推理鲁棒性；同时，该方法可缓解因OCR幻觉或局部误识别导致的错误传播问题。作为一种系统性的推理增强范式，具备成为DIU领域‘推理军火库’的潜力。
Abstract Reasoning reinforcement learning (RL) has recently revealed a new scaling effect: test-time scaling. Thinking models such as R1 and o1 improve their reasoning accuracy at test time as the length of the reasoning context increases. However, compared with training-time scaling, test-time scaling is fundamentally limited by the limited context length of base models, which remains orders of magnitude smaller than the amount of tokens consumed during training. We revisit test-time enhancement techniques through the lens of scaling effect and introduce a unified framework of multi-dimensional test-time scaling to extend the capacity of test-time reasoning. Beyond conventional context-length scaling, we consider two additional dimensions: batch scaling, where accuracy improves with parallel sampling, and turn scaling, where iterative self-refinement enhances reasoning quality. Building on this perspective, we propose 3D test-time scaling, which integrates context, batch, and turn scaling. We show that: (1) each dimension demonstrates a test-time scaling effect, but with a bounded capacity; (2) combining all three dimensions substantially improves the reasoning performance of challenging testbeds, including IOI, IMO, and CPHO, and further benefits from human preference feedback; and (3) the human-in-the-loop framework naturally extends to a more open-ended domain, i.e., embodied learning, which enables the design of humanoid control behaviors.
Authors Chao Yu, Qixin Tan, Jiaxuan Gao, Shi Yu, Hong Lu, Xinting Yang, Zelai Xu, Yu Wang, Yi Wu, Eugene Vinitsky
Comment 44 pages, 12 figures
Categories Artificial Intelligence
8
Attention Guided Alignment in Efficient Vision-Language Models 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop on Efficient Reasoning
Title CN 高效视觉语言模型中的注意力引导对齐
Keywords 视觉语言模型 · 注意力机制 · 幻觉抑制 · 视觉接地 · SAM
Summary 本文发现现有高效视觉语言模型在处理图像-文本对齐时存在语义匹配判断缺陷，容易引发对象幻觉。为此提出AGE-VLM框架，通过插入交错式交叉注意力层并融合SAM提取的空间结构信息，增强模型对关键图像区域的关注能力，从而显著降低幻觉率，在多个视觉为中心的基准上表现优异。
Reason 该论文针对高效VLM中因拼接架构导致的模态错位和物体幻觉问题，提出通过交错交叉注意力机制增强视觉接地能力，并利用SAM蒸馏的空间知识引导注意力聚焦正确图像区域。这一方法直接应对文档图像理解中的核心痛点——OCR幻觉与细粒度定位困难。其引入的‘可感知空间结构的注意力引导’机制，可迁移用于提升DocVQA任务中对复杂排版或表格内容的精准识别与推理，避免模型生成不在原图中的文本内容。
Abstract Large Vision-Language Models (VLMs) rely on effective multimodal alignment between pre-trained vision encoders and Large Language Models (LLMs) to integrate visual and textual information. This paper presents a comprehensive analysis of attention patterns in efficient VLMs, revealing that concatenation-based architectures frequently fail to distinguish between semantically matching and non-matching image-text pairs. This is a key factor for object hallucination in these models. To address this, we introduce Attention-Guided Efficient Vision-Language Models (AGE-VLM), a novel framework that enhances visual grounding through interleaved cross-attention layers to instill vision capabilities in pretrained small language models. This enforces in VLM the ability "look" at the correct image regions by leveraging spatial knowledge distilled from the Segment Anything Model (SAM), significantly reducing hallucination. We validate our approach across different vision-centric benchmarks where our method is better or comparable to prior work on efficient VLMs. Our findings provide valuable insights for future research aimed at achieving enhanced visual and linguistic understanding in VLMs.
Authors Shweta Mahajan, Hoang Le, Hyojin Park, Farzad Farhadzadeh, Munawar Hayat, Fatih Porikli
Comment 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop on Efficient Reasoning
Categories Computer Vision and Pattern Recognition
7
Agentic Learner with Grow-and-Refine Multimodal Semantic Memory
Title CN 具有增长与精炼多模态语义记忆的智能体学习者
Keywords 多模态记忆 · 智能体学习 · 幻觉抑制 · 视觉-逻辑对齐 · 持续学习
Summary 本文提出ViLoMem，一种受人类语义记忆启发的双流记忆框架，用于增强多模态大语言模型在持续问题求解中的学习能力。该框架将视觉注意力模式与逻辑推理过程分别建模，构建紧凑且可更新的schema-based记忆库，遵循‘增长与精炼’原则实现知识积累与遗忘控制。实验表明，该方法在六个多模态基准上显著提升准确率，并有效减少重复性视觉与逻辑错误。
Reason 该论文提出了一种双流记忆框架ViLoMem，分别编码视觉干扰模式和逻辑推理错误，支持MLLM从成功与失败经验中学习。其核心创新在于构建了可增量积累与更新的多模态语义记忆，显式分离‘分心’与‘幻觉’，这对文档图像理解中的OCR幻觉、复杂排版导致的注意力错位等问题具有直接迁移价值。例如，在DocVQA任务中，模型常因视觉布局干扰而误读字段内容，或在多次推理中重复同类错误，ViLoMem的记忆机制可帮助模型识别并纠正此类模式，提升长文档或多轮交互下的稳定性。
Abstract MLLMs exhibit strong reasoning on isolated queries, yet they operate de novo -- solving each problem independently and often repeating the same mistakes. Existing memory-augmented agents mainly store past trajectories for reuse. However, trajectory-based memory suffers from brevity bias, gradually losing essential domain knowledge. More critically, even in truly multimodal problem-solving settings, it records only a single-modality trace of past behavior, failing to preserve how visual attention and logical reasoning jointly contributed to the solution. This is fundamentally misaligned with human cognition: semantic memory is both multimodal and integrated, preserving visual and abstract knowledge through coordinated but distinct representational streams. We thus introduce ViLoMem, a dual-stream memory framework that constructs compact, schema-based memory. It separately encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences. Following a grow-and-refine principle, the system incrementally accumulates and updates multimodal semantic knowledge -- preserving stable, generalizable strategies while avoiding catastrophic forgetting. Across six multimodal benchmarks, ViLoMem consistently improves pass@1 accuracy and substantially reduces repeated visual and logical errors. Ablations confirm the necessity of dual-stream memory with explicit distraction--hallucination separation, demonstrating the value of error-aware multimodal memory for lifelong and cross-domain agentic learning. Our project page will be available at https://weihao-bo.github.io/ViLoMeo-page.
Authors Weihao Bo, Shan Zhang, Yanpeng Sun, Jingjing Wu, Qunyi Xie, Xiao Tan, Kunbin Chen, Wei He, Xiaofan Li, Na Zhao, Jingdong Wang, Zechao Li
Categories Artificial Intelligence
7
Qwen3-VL Technical Report
Title CN Qwen3-VL 技术报告
Keywords 视觉语言模型 · 长上下文 · 多模态推理 · DeepStack · MRoPE
Summary Qwen3-VL是通义千问系列最新的视觉语言模型，支持长达256K token的交错文本-图像-视频输入，在纯文本理解、长上下文 comprehension 和多模态推理方面均有显著提升。通过引入增强的MRoPE、DeepStack特征融合和显式文本时间对齐，模型在MMMU、MathVista等基准上达到领先水平。该模型提供多种规模（密集型与MoE），适用于不同延迟-质量权衡场景。
Reason 该论文提出了一种支持高达256K token上下文的视觉语言模型，具备处理长文档和交错多模态输入的能力。其架构改进包括增强的交错式MRoPE、DeepStack集成以融合多层ViT特征，以及文本时间对齐机制。这些技术可直接迁移至文档图像理解领域：256K上下文支持长文档端到端理解，缓解分块截断问题；DeepStack加强视觉-语言对齐，有助于复杂排版下的细粒度信息定位；MRoPE的空间建模能力也利于文档结构感知。尽管并非专为DIU设计，但其在长上下文、高分辨率输入和多模态对齐方面的进步对解决OCR幻觉、长文档推理等核心痛点具有显著价值。
Abstract We introduce Qwen3-VL, the most capable vision-language model in the Qwen series to date, achieving superior performance across a broad range of multimodal benchmarks. It natively supports interleaved contexts of up to 256K tokens, seamlessly integrating text, images, and video. The model family includes both dense (2B/4B/8B/32B) and mixture-of-experts (30B-A3B/235B-A22B) variants to accommodate diverse latency-quality trade-offs. Qwen3-VL delivers three core pillars: (i) markedly stronger pure-text understanding, surpassing comparable text-only backbones in several cases; (ii) robust long-context comprehension with a native 256K-token window for both text and interleaved multimodal inputs, enabling faithful retention, retrieval, and cross-referencing across long documents and videos; and (iii) advanced multimodal reasoning across single-image, multi-image, and video tasks, demonstrating leading performance on comprehensive evaluations such as MMMU and visual-math benchmarks (e.g., MathVista and MathVision). Architecturally, we introduce three key upgrades: (i) an enhanced interleaved-MRoPE for stronger spatial-temporal modeling across images and video; (ii) DeepStack integration, which effectively leverages multi-level ViT features to tighten vision-language alignment; and (iii) text-based time alignment for video, evolving from T-RoPE to explicit textual timestamp alignment for more precise temporal grounding. Under comparable token budgets and latency constraints, Qwen3-VL achieves superior performance in both dense and Mixture-of-Experts (MoE) architectures. We envision Qwen3-VL serving as a foundational engine for image-grounded reasoning, agentic decision-making, and multimodal code intelligence in real-world workflows.
Authors Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, Ke Zhu
Comment 42 pages
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
7
Beyond Introspection: Reinforcing Thinking via Externalist Behavioral Feedback
Title CN 超越内省：通过外在行为反馈强化推理过程
Keywords 推理增强 · 外部反馈 · 行为建模 · 去幻觉 · 测试时计算
Summary 本文提出DRR框架，通过提取大模型推理的行为轨迹，训练一个外部判别模型作为批评者，在推理时动态检测并拒绝可疑的思维步骤，迫使LLM探索更可靠的替代路径。该方法无需修改基础模型，且不依赖人工标注，实验显示其在多个推理基准上优于主流自批判方法。
Reason 该论文提出了一种名为DRR的外部主义推理增强框架，通过构建轻量级外部判别模型（DM）来评估LLM的可观察行为并提供反馈，从而纠正不可靠的推理路径。这一方法避免了自指性批判中的认知偏差问题。对于文档图像理解（DIU）领域，尤其是在DocVQA等需要长链逻辑推理的任务中，该方法具有明确迁移价值：可用来识别和拒绝由视觉误解或OCR幻觉引发的错误中间推理步骤，提升多步问答的鲁棒性。
Abstract While inference-time thinking allows Large Language Models (LLMs) to address complex problems, the extended thinking process can be unreliable or inconsistent because of the model's probabilistic nature, especially near its knowledge boundaries. Existing approaches attempt to mitigate this by having the model critique its own reasoning to make corrections. However, such self-critique inherits the same biases of the original output, known as the introspection illusion. Moving beyond such introspection and inspired by core methodologies in ethology, we propose an externalist three-step framework Distillation-Reinforcement-Reasoning (DRR). Rather than relying on a model's introspection, DRR evaluates its observable behaviors to provide corrective feedback. DRR first distills the reasoner's behavioral traces, then trains a lightweight, external Discriminative Model (DM). At inference time, this DM acts as a critic, identifying and rejecting suspicious reasoning steps. This external feedback compels the LLM to discard flawed pathways and explore alternatives, thereby enhancing reasoning quality without altering the base model. Experiments on multiple reasoning benchmarks show that our framework significantly outperforms prominent self-critique methods. Benefiting from a lightweight and annotation-free design, DRR offers a scalable and adaptable solution for improving the reliability of reasoning in a wide range of LLMs.
Authors Diji Yang, Linda Zeng, Kezhen Chen, Yi Zhang
Categories Artificial Intelligence, Computation and Language
7
EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokens AAAI 2026
Title CN EM-KD：通过不平衡视觉令牌蒸馏高效的多模态大语言模型
Keywords 知识蒸馏 · 视觉令牌压缩 · 多模态大模型 · 视觉-语言对齐 · 细粒度理解
Summary 本文提出EM-KD，一种面向高效多模态大模型的知识蒸馏新范式，解决因视觉令牌压缩导致的师生模型视觉令牌不平衡问题。通过匈牙利算法进行空间对齐，并设计了视觉-语言亲和性蒸馏和视觉语义蒸馏两种策略，有效保留细粒度视觉语义信息。实验表明该方法在多个基准上显著优于现有蒸馏方法。
Reason 该论文针对高效多模态大模型（MLLMs）中因视觉令牌压缩导致的信息损失问题，提出了一种新的知识蒸馏框架EM-KD。其核心创新在于处理师生模型间视觉令牌数量不匹配的问题——通过曼哈顿距离计算与匈牙利匹配算法实现空间维度对齐，进而引入视觉-语言亲和性蒸馏（VLAD）和视觉语义蒸馏（VSD）。这一方法特别适用于文档图像理解中的高分辨率输入与视觉信息压缩之间的矛盾。例如，在类似DeepSeek-OCR的视觉令牌压缩路线中，如何在压缩后保留细粒度文本结构是一个关键挑战，而EM-KD提供的跨模态对齐与语义保留机制可直接迁移用于提升OCR-free或端到端DocVQA系统的准确性，尤其在缓解因压缩引发的‘幻觉’问题上有潜在价值。
Abstract Efficient Multimodal Large Language Models (MLLMs) compress vision tokens to reduce resource consumption, but the loss of visual information can degrade comprehension capabilities. Although some priors introduce Knowledge Distillation to enhance student models, they overlook the fundamental differences in fine-grained vision comprehension caused by unbalanced vision tokens between the efficient student and vanilla teacher. In this paper, we propose EM-KD, a novel paradigm that enhances the Efficient MLLMs with Knowledge Distillation. To overcome the challenge of unbalanced vision tokens, we first calculate the Manhattan distance between the vision logits of teacher and student, and then align them in the spatial dimension with the Hungarian matching algorithm. After alignment, EM-KD introduces two distillation strategies: 1) Vision-Language Affinity Distillation (VLAD) and 2) Vision Semantic Distillation (VSD). Specifically, VLAD calculates the affinity matrix between text tokens and aligned vision tokens, and minimizes the smooth L1 distance of the student and the teacher affinity matrices. Considering the semantic richness of vision logits in the final layer, VSD employs the reverse KL divergence to measure the discrete probability distributions of the aligned vision logits over the vocabulary space. Comprehensive evaluation on diverse benchmarks demonstrates that EM-KD trained model outperforms prior Efficient MLLMs on both accuracy and efficiency with a large margin, validating its effectiveness. Compared with previous distillation methods, which are equipped with our proposed vision token matching strategy for fair comparison, EM-KD also achieves better performance.
Authors Ze Feng, Sen Yang, Boqiang Duan, Wankou Yang, Jingdong Wang
Comment accepted by AAAI 2026
Categories Computer Vision and Pattern Recognition
7
Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Rumination
Title CN Video-R4：通过视觉反刍增强富含文本的视频推理
Keywords 视觉反刍 · 迭代推理 · 细粒度定位 · 去幻觉 · 多模态推理
Summary Video-R4提出一种受人类阅读启发的视觉反刍机制，通过迭代地选择、放大、重编码关键视觉区域来增强富含文本视频的理解能力。模型在监督微调和强化学习框架下训练，能够有效减少幻觉，并在多个视频问答基准上达到SOTA。值得注意的是，该方法还泛化到了多页文档问答和幻灯片问答任务，显示出其在非视频场景下的潜力。
Reason 该论文提出了一种名为视觉反刍（visual rumination）的迭代推理机制，通过反复选择帧、放大关键区域、重新编码像素并更新推理状态，模拟人类对细小或瞬态文本的重读行为。这一方法直接针对OCR幻觉和细粒度定位问题——这正是文档图像理解（DIU）中的核心痛点。尽管其主要应用场景是视频，但其‘迭代重检+局部放大+状态更新’的架构可迁移至长文档或多页文档的VQA任务中，用于解决因分辨率不足或信息分散导致的遗漏与幻觉问题。尤其在处理扫描PPT、多页报表等结构化文本时，该范式有望提升模型对关键字段的聚焦能力。
Abstract Understanding text-rich videos requires reading small, transient textual cues that often demand repeated inspection. Yet most video QA models rely on single-pass perception over fixed frames, leading to hallucinations and failures on fine-grained evidence. Inspired by how humans pause, zoom, and re-read critical regions, we introduce Video-R4 (Reinforcing Text-Rich Video Reasoning with Visual Rumination), a video reasoning LMM that performs visual rumination: iteratively selecting frames, zooming into informative regions, re-encoding retrieved pixels, and updating its reasoning state. We construct two datasets with executable rumination trajectories: Video-R4-CoT-17k for supervised practice and Video-R4-RL-30k for reinforcement learning. We propose a multi-stage rumination learning framework that progressively finetunes a 7B LMM to learn atomic and mixing visual operations via SFT and GRPO-based RL. Video-R4-7B achieves state-of-the-art results on M4-ViteVQA and further generalizes to multi-page document QA, slides QA, and generic video QA, demonstrating that iterative rumination is an effective paradigm for pixel-grounded multimodal reasoning. Project Page: https://yunlong10.github.io/Video-R4/
Authors Yolo Y. Tang, Daiki Shimada, Hang Hua, Chao Huang, Jing Bi, Rogerio Feris, Chenliang Xu
Categories Computer Vision and Pattern Recognition
7
LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?
Title CN LogicOCR：你的大模型在富含文本的图像上擅长逻辑推理吗？
Keywords 文档图像理解 · 逻辑推理 · 评测基准 · 去幻觉 · 注意力引导
Summary 本文构建了一个名为LogicOCR的新基准，用于评估大模型在富含文本的图像上的逻辑推理能力，涵盖生成和真实文档图像。通过多维度分析发现现有模型在视觉-文本联合推理方面仍显著弱于纯文本推理。作者提出一种无需训练的TextCue方法，利用注意力机制和文本分割技术增强关键文本区域的输入显示，提升了模型表现。该工作揭示了当前VLM在文档理解中的根本瓶颈，并提供了一种可推广的推理增强策略。
Reason 该论文提出了一个面向文档图像逻辑推理能力评测的新基准LogicOCR，包含生成和真实场景下的多选与开放性问题。其核心贡献在于系统性地暴露了当前大模型在结合视觉与文本进行复杂推理时的不足，尤其是在排版多样、字体变化的真实文档上的局限。虽然本身是评测工作，但提出的TextCue方法（基于注意力图和文本分割定位关键区域并增强输入）为解决DIU中的细粒度定位与幻觉问题提供了可迁移的技术思路——例如可用于DocVQA中聚焦问题相关文本块，缓解无关内容干扰。因此具备向DIU领域输出方法论的价值。
Abstract Recent advances in Large Multimodal Models (LMMs) have revolutionized their reasoning and Optical Character Recognition (OCR) capabilities. However, their complex logical reasoning performance on text-rich images remains underexplored. To bridge this gap, we introduce LogicOCR, a benchmark comprising 2780 questions with two subsets, i.e., LogicOCR-Gen with 1100 multi-choice questions on generated images, and LogicOCR-Real with 1680 meticulously designed free-form questions on real-world images. For constructing LogicOCR-Gen, we first curate a text corpus from the Chinese National Civil Servant Examination, and customize an automatic pipeline to steer GPT-Image-1 to generate images with varied layouts and fonts, ensuring contextual relevance and visual realism. Then, the generated images are manually verified. We evaluate a range of representative LMMs under Chain-of-Thought (CoT) and direct-answer settings. Our multi-dimensional analysis reveals key insights, such as the impact of test-time scaling, input modality differences, and sensitivity to visual-text orientation. Notably, LMMs still lag in multimodal reasoning compared to text-only inputs, indicating that they have not fully bridged visual reading with reasoning. Moreover, we propose TextCue, a training-free method that enhances LMMs' perception of image regions containing important text cues for solving questions. We leverage LMMs' attention maps and an off-the-shelf text segmentation specialist to determine the region, which is then cropped and enlarged to augment the original image. Experiments show its effectiveness, e.g., a 1.8% accuracy gain over LLaVA-OV-1.5-8B under the CoT setting. Our benchmark is available at https://github.com/MiliLab/LogicOCR.
Authors Maoyuan Ye, Haibin He, Qihuang Zhong, Jing Zhang, Juhua Liu, Bo Du
Comment GitHub: https://github.com/MiliLab/LogicOCR
Categories Computer Vision and Pattern Recognition, Computation and Language
7
Beyond Generation: Multi-Hop Reasoning for Factual Accuracy in Vision-Language Models NewInML Workshop ICML, 2025
Title CN 超越生成：面向视觉语言模型事实准确性的多跳推理
Keywords 视觉语言模型 · 多跳推理 · 知识图谱 · 事实准确性 · 幻觉抑制
Summary 本文针对视觉语言模型（VLM）生成内容存在事实错误的问题，提出一种结合结构化知识图谱的多跳推理框架。该方法通过识别图像中的实体，检索知识图谱中的关联路径，并据此修正生成的图像描述，在Google Landmarks v2、Conceptual Captions和COCO Captions的混合数据集上实现了约31%的事实准确率提升。研究揭示了多模态推理中的典型错误模式，为构建更可靠的VLM提供了新思路。
Reason 该论文提出了一种基于知识图谱的多跳推理框架，用于提升VLM在图像描述任务中的事实准确性。其核心贡献在于引入结构化知识进行多步验证（如视觉实体识别→知识图谱遍历→基于事实的描述修正），从而缓解幻觉问题。这一方法可直接迁移到文档图像理解（DIU）中，尤其是应对OCR或VLM在复杂文本场景下的‘语义幻觉’——例如KIE或DocVQA任务中模型虚构字段值的问题。通过将文档中的检测文本与外部知识库（如行业术语库、表单模式）对齐并进行逻辑验证，有望显著提升系统输出的事实一致性。
Abstract Visual Language Models (VLMs) are powerful generative tools but often produce factually in- accurate outputs due to a lack of robust reason- ing capabilities. While extensive research has been conducted on integrating external knowl- edge for reasoning in large language models (LLMs), such efforts remain underexplored in VLMs, where the challenge is compounded by the need to bridge multiple modalities seam- lessly. This work introduces a framework for knowledge-guided reasoning in VLMs, leverag- ing structured knowledge graphs for multi-hop verification using image-captioning task to il- lustrate our framework. Our approach enables systematic reasoning across multiple steps, in- cluding visual entity recognition, knowledge graph traversal, and fact-based caption refine- ment. We evaluate the framework using hi- erarchical, triple-based and bullet-point based knowledge representations, analyzing their ef- fectiveness in factual accuracy and logical infer- ence. Empirical results show that our approach improves factual accuracy by approximately 31% on preliminary experiments on a curated dataset of mixtures from Google Landmarks v2, Conceptual captions and Coco captions re- vealing key insights into reasoning patterns and failure modes. This work demonstrates the po- tential of integrating external knowledge for advancing reasoning in VLMs, paving the way for more reliable and knowledgable multimodal systems.
Authors Shamima Hossain
Comment Accepted as poster at NewInML Workshop ICML, 2025
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
7
Object-Centric Vision Token Pruning for Vision Language Models
Title CN 面向视觉语言模型的以对象为中心的视觉Token剪枝
Keywords 视觉Token剪枝 · 视觉语言模型 · 推理效率 · 对象中心表示 · 无微调插入
Summary 本文提出OC-VTP，一种以对象为中心的视觉Token剪枝方法，用于提升视觉语言模型的推理效率。该方法通过最小化原始Token重建误差，保证保留最具代表性的视觉Token，无需对主干模型进行微调即可插入现有VLM中。实验表明，在不同剪枝比例下，OC-VTP均能显著提升推理速度并保持较高准确率，且具备良好的可解释性。
Reason 该论文提出了一种以对象为中心的视觉Token剪枝方法（OC-VTP），通过轻量级预训练一个小型剪枝模块，直接选择最具代表性的视觉Token，从而在不微调原有模型的前提下提升VLM推理效率并保持精度。该方法对文档图像理解（DIU）具有明确迁移价值：文档图像通常包含大量冗余背景或非关键区域（如页眉、边框、空白），而核心信息集中在文本块、表格等结构化对象上。OC-VTP通过聚焦对象区域保留关键视觉Token，可有效缓解高分辨率文档输入带来的计算压力，同时减少因处理无关区域导致的注意力分散问题，有助于提升DocVQA和复杂排版分析中的细粒度定位与推理准确性。
Abstract In Vision Language Models (VLMs), vision tokens are quantity-heavy yet information-dispersed compared with language tokens, thus consume too much unnecessary computation. Pruning redundant vision tokens for high VLM inference efficiency has been continuously studied but all existing methods resort to indirect and non-guaranteed ways. We propose OC-VTP, a direct and guaranteed approach to select the most representative vision tokens for high-efficiency yet accuracy-preserving VLM inference. Our OC-VTP requires merely light-weight pre-training of a small object-centric vision token pruner, which can then be inserted into existing VLMs, without fine-tuning of any models on any datasets. It is gauranteed that the most representative vision tokens are kept by minimizing the error in reconstructing the original unpruned tokens from the selected ones. Across any vision pruning ratios, i.e., inference efficiency, our OC-VTP consistently helps mainstream VLMs to preserve the highest inference accuracy. Our pruning also demonstrates interesting interpretability. Our codes are available at https://github.com/GarryLarry010131/OC-VTP.
Authors Guangyuan Li, Rongzhen Zhao, Jinhong Deng, Yanbo Wang, Joni Pajarinen
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
7
Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Models
Title CN 跨层视觉平滑：通过关键对象的持续关注增强大视觉语言模型的视觉理解
Keywords 视觉注意力 · 注意力平滑 · 视觉记忆 · 多模态大模型 · 视觉定位
Summary 本文提出跨层视觉平滑（CLVS）方法，通过构建一个跨网络层的视觉记忆机制，使大视觉语言模型在处理图像时能持续关注关键对象。该方法在多个视觉理解基准上取得SOTA效果，并在图像描述等任务中表现优异。其注意力稳定性设计为解决VLM在复杂视觉输入下的注意力漂移问题提供了新思路。
Reason 该论文提出了一种通过跨层注意力平滑机制（CLVS）来维持LVLM对关键视觉对象的持续关注，从而提升视觉理解能力。其核心创新在于引入可迭代更新的视觉记忆模块，在早期和中间层维持对关键区域的聚焦。这一机制对文档图像理解具有明确迁移价值：在复杂排版或密集文本场景中，VLM常因注意力分散导致OCR幻觉或漏检，而CLVS提供的持续聚焦能力可增强对文本区域的稳定感知，尤其适用于细粒度文本定位与抗干扰识别。
Abstract Large Vision-Language Models (LVLMs) can accurately locate key objects in images, yet their attention to these objects tends to be very brief. Motivated by the hypothesis that sustained focus on key objects can improve LVLMs' visual capabilities, we propose Cross-Layer Vision Smoothing (CLVS). The core idea of CLVS is to incorporate a vision memory that smooths the attention distribution across layers. Specifically, we initialize this vision memory with position-unbiased visual attention in the first layer. In subsequent layers, the model's visual attention jointly considers the vision memory from previous layers, while the memory is updated iteratively, thereby maintaining smooth attention on key objects. Given that visual understanding primarily occurs in the early and middle layers of the model, we use uncertainty as an indicator of completed visual understanding and terminate the smoothing process accordingly. Experiments on four benchmarks across three LVLMs confirm the effectiveness and generalizability of our method. CLVS achieves state-of-the-art overall performance across a variety of visual understanding tasks and attains comparable results to the leading approaches on image captioning benchmarks.
Authors Jianfei Zhao, Feng Zhang, Xin Sun, Chong Feng, Zhixing Tan
Comment Under Review
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
7
MambaEye: A Size-Agnostic Visual Encoder with Causal Sequential Processing
Title CN MambaEye：一种具有因果序列处理能力的尺寸无关视觉编码器
Keywords 视觉编码器 · Mamba · 因果建模 · 分辨率自适应 · 序列化处理
Summary MambaEye提出了一种全新的因果序列化视觉编码器，采用单向Mamba2结构和相对移动嵌入机制，使模型能够以线性复杂度处理任意尺寸图像，并在ImageNet-1K等任务上验证了其在高分辨率下的鲁棒性。该方法打破了传统Transformer或双向Mamba在视觉编码中的局限，为高效、可扩展的视觉建模提供了新路径。
Reason 该论文提出了一种基于纯Mamba2架构的因果、单向视觉编码器，通过相对移动嵌入和扩散启发式损失函数，实现了对任意图像分辨率和扫描模式的自适应。其核心创新在于保持状态空间模型的因果性，并支持在输入序列任意位置生成预测，这对文档图像理解中常见的高分辨率、长序列建模问题具有直接迁移价值。特别是其线性复杂度和对高分辨率（如1536²）的良好表现，可被用于提升文档VLM中的细粒度识别与OCR-free解析能力，缓解现有方法在处理长文档或密集排版时的计算瓶颈。
Abstract Despite decades of progress, a truly input-size agnostic visual encoder-a fundamental characteristic of human vision-has remained elusive. We address this limitation by proposing \textbf{MambaEye}, a novel, causal sequential encoder that leverages the low complexity and causal-process based pure Mamba2 backbone. Unlike previous Mamba-based vision encoders that often employ bidirectional processing, our strictly unidirectional approach preserves the inherent causality of State Space Models, enabling the model to generate a prediction at any point in its input sequence. A core innovation is our use of relative move embedding, which encodes the spatial shift between consecutive patches, providing a strong inductive bias for translation invariance and making the model inherently adaptable to arbitrary image resolutions and scanning patterns. To achieve this, we introduce a novel diffusion-inspired loss function that provides dense, step-wise supervision, training the model to build confidence as it gathers more visual evidence. We demonstrate that MambaEye exhibits robust performance across a wide range of image resolutions, especially at higher resolutions such as $1536^2$ on the ImageNet-1K classification task. This feat is achieved while maintaining linear time and memory complexity relative to the number of patches.
Authors Changho Choi, Minho Kim, Jinkyu Kim
Comment Code will be released in github
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
7
Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models
Title CN 面向Best-of-N采样的推理感知微调
Keywords 推理感知微调 · Best-of-N · 去幻觉 · 强化学习 · 推理时计算
Summary 本文提出了一种新的微调范式——推理感知微调，专门针对Best-of-N这类依赖推理时计算的策略进行优化。作者设计了首个适用于BoN的模仿学习与强化学习算法，克服了选择最佳响应过程中不可导的问题。实验表明，该方法提升了Gemma 2B在数学推理和代码生成任务上的BoN性能，显示出模型学会了隐式的元策略，兼顾高质响应与多样性探索。
Reason 该论文提出了一种推理感知的微调范式，通过在训练阶段显式优化Best-of-N（BoN）推理策略下的模型表现，解决了非可导argmax操作带来的挑战，并引入了基于模仿学习和强化学习的方法。该方法使模型在推理时能更好地平衡探索与利用，生成更多样化且高质量的候选输出。这一思路可直接迁移到文档图像理解中的VQA任务，尤其是在应对OCR或视觉语言模型幻觉问题上具有潜力：通过BoN机制生成多个答案并由验证器选择最优解，能够显著降低幻觉风险，提升DocVQA的鲁棒性。
Abstract Recent studies have indicated that effectively utilizing inference-time compute is crucial for attaining better performance from large language models (LLMs). In this work, we propose a novel inference-aware fine-tuning paradigm, in which the model is fine-tuned in a manner that directly optimizes the performance of the inference-time strategy. We study this paradigm using the simple yet effective Best-of-N (BoN) inference strategy, in which a verifier selects the best out of a set of LLM-generated responses. We devise the first imitation learning and reinforcement learning~(RL) methods for BoN-aware fine-tuning, overcoming the challenging, non-differentiable argmax operator within BoN. We empirically demonstrate that our BoN-aware models implicitly learn a meta-strategy that interleaves best responses with more diverse responses that might be better suited to a test-time input -- a process reminiscent of the exploration-exploitation trade-off in RL. Our experiments demonstrate the effectiveness of BoN-aware fine-tuning in terms of improved performance and inference-time compute. In particular, we show that our methods improve the Bo32 performance of Gemma 2B on Hendrycks MATH from 26.8% to 30.8%, and pass@32 from 60.0% to 67.0%, as well as the pass@16 on HumanEval from 61.6% to 67.1%.
Authors Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, Aleksandra Faust
Categories Artificial Intelligence, Computation and Language
7
DINO-Tok: Adapting DINO for Visual Tokenizers
Title CN DINO-Tok：基于DINO的视觉分词器
Keywords 视觉分词器 · DINO · 潜在生成模型 · 向量量化 · 视觉表示学习
Summary DINO-Tok利用预训练的DINO模型构建层次化视觉分词器，在ImageNet上实现了先进的重建性能。其核心贡献在于将强大的视觉基础模型适配为高质量视觉tokenizer，并解决了高维向量量化中的信息丢失与码本坍塌问题，为生成模型提供了兼具语义一致性与重建保真度的潜在表示。
Reason 该工作提出了一种基于DINO的视觉分词器DINO-Tok，通过融合浅层细节特征与深层语义特征，构建信息完整的潜在空间表示，并引入全局PCA重加权机制缓解高维空间下的向量量化信息丢失问题。这一视觉token压缩与表示学习方法可直接迁移到文档图像理解领域，尤其适用于提升VLM在处理高分辨率文档时的视觉表征能力，增强细粒度文本识别和布局结构感知，对抗OCR幻觉有潜在价值。
Abstract Recent advances in visual generation have highlighted the rise of Latent Generative Models (LGMs), which rely on effective visual tokenizers to bridge pixels and semantics. However, existing tokenizers are typically trained from scratch and struggle to balance semantic representation and reconstruction fidelity, particularly in high-dimensional latent spaces. In this work, we introduce DINO-Tok, a DINO-based visual tokenizer that unifies hierarchical representations into an information-complete latent space. By integrating shallow features that retain fine-grained details with deep features encoding global semantics, DINO-Tok effectively bridges pretrained representations and visual generation. We further analyze the challenges of vector quantization (VQ) in this high-dimensional space, where key information is often lost and codebook collapse occurs. We thus propose a global PCA reweighting mechanism to stabilize VQ and preserve essential information across dimensions. On ImageNet 256$\times$256, DINO-Tok achieves state-of-the-art reconstruction performance, reaching 28.54 PSNR for autoencoding and 23.98 PSNR for VQ-based modeling, significantly outperforming prior tokenizers and comparable to billion-level data trained models (such as Hunyuan and Wan). These results demonstrate that adapting powerful pretrained vision models like DINO for tokenization enables semantically aligned and high-fidelity latent representations, enabling next-generation visual generative models. Code will be publicly available at https://github.com/MKJia/DINO-Tok.
Authors Mingkai Jia, Mingxiao Li, Liaoyuan Fan, Tianxing Shi, Jiaxin Guo, Zeming Li, Xiaoyang Guo, Xiao-Xiao Long, Qian Zhang, Ping Tan, Wei Yin
Categories Computer Vision and Pattern Recognition
7
Look Where It Matters: Training-Free Ultra-HR Remote Sensing VQA via Adaptive Zoom Search
Title CN 关注关键区域：通过自适应缩放搜索实现无需训练的超高清遥感视觉问答
Keywords 视觉问答 · 高分辨率图像 · 注意力定位 · 无需训练 · 图像块重组
Summary 本文针对超高清遥感图像在VQA任务中因分辨率过高导致的计算开销大、细节丢失问题，提出了ZoomSearch方法。该方法无需训练，通过自适应搜索机制聚焦于图像中与问题相关的关键区域，并重新组织这些局部区域以保留空间布局信息，从而提升回答准确率和推理效率。实验表明其在多个遥感VQA基准上显著优于现有方法。
Reason 该论文提出了一种无需训练的自适应多分支缩放搜索方法（ZoomSearch），通过分层搜索图像块来定位与问题相关的关键区域，并采用布局感知的图像块重组技术，将选定区域重组成紧凑且保持布局一致性的输入。这一‘先定位再回答’的解耦策略，可直接迁移用于解决文档图像理解中的高分辨率处理与细粒度定位难题，尤其是应对OCR幻觉和复杂排版下模型注意力分散的问题。其对超高清图像的高效推理机制也适用于长文档或多页PDF的逐步聚焦阅读。
Abstract With advances in satellite constellations, sensor technologies, and imaging pipelines, ultra-high-resolution (Ultra-HR) remote sensing imagery is becoming increasingly widespread. However, current remote sensing foundation models are ill-suited to such inputs: full-image encoding exhausts token and memory budgets, while resize-based preprocessing loses fine-grained and answer-critical details. In this context, guiding the model look where it matters before prediction becomes crucial. Therefore, we present ZoomSearch, a training-free, plug-and-play pipeline that decouples 'where to look' from 'how to answer' for Ultra-HR Remote Sensing Visual Question Answering (RS-VQA). ZoomSearch combines Adaptive Multi-Branch Zoom Search, which performs a hierarchical search over image patches to localize query-relevant regions, with Layout-Aware Patch Reassembly, which reorganizes the selected patches into a compact, layout-faithful canvas. We conduct comprehensive experiments on Ultra-HR RS-VQA benchmarks MME-RealWorld-RS and LRS-VQA, comparing against (i) strong general foundation models, (ii) remote sensing foundation models, (iii) Ultra-HR RS-VQA methods, and (iv) plug-and-play search-based VQA methods. When integrated with LLaVA-ov, ZoomSearch attains state-of-the-art accuracy across diverse tasks, improving the LLaVA-ov baseline by 26.3% on LRS-VQA and 114.8\% on MME-RealWorld-RS. Meanwhile, it achieves much higher inference efficiency, outperforming prior search-based methods by 20%~44% in speed.
Authors Yunqi Zhou, Chengjie Jiang, Chun Yuan, Jing Li
Comment 17 pages, 8 figures
Categories Computer Vision and Pattern Recognition
7
Boosting Reasoning in Large Multimodal Models via Activation Replay
Title CN 通过激活重放增强大型多模态模型的推理能力
Keywords 激活重放 · 多模态推理 · 推理干预 · 无训练增强 · 低熵激活
Summary 本文从logit lens视角分析RLVR后训练对LMM激活状态的影响，发现其显著改变了低熵激活。基于此，作者提出激活重放技术，在测试时通过重放基础模型的低熵激活来优化推理过程，无需额外训练即可提升多种任务下的推理表现，并验证了其在扩展推理多样性与准确性上的有效性。
Reason 该论文提出了一种无需训练的推理阶段干预方法——激活重放（Activation Replay），通过在测试时操纵视觉token，重放基础LMM中的低熵激活来调节经RLVR后训练的模型行为，从而提升多模态推理能力。虽然其实验场景涵盖数学、视觉代理和视频推理，但其核心思想——利用输入激活的调控来改善推理路径、缓解推理覆盖狭窄问题——可直接迁移到文档图像理解（DIU）中的长文档多步推理与逻辑连贯性建模。例如，在复杂表单或财务报告的DocVQA任务中，模型常因注意力分散导致推理链断裂，该方法可通过保留关键上下文的低熵激活实现更稳定的推理引导。此外，其对幻觉缓解（如Pass@K提升）的贡献也暗示其在抑制OCR-free VLM中文本幻觉方面的潜在价值。
Abstract Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach to incentivizing reasoning capability in Large Multimodal Models (LMMs), while the underlying mechanisms behind this post-training paradigm are poorly understood. We begin by exploring how input activations are affected by RLVR through the perspective of logit lens. Our systematic investigations across multiple post-trained LMMs suggest that RLVR shifts low-entropy activations unexpectedly, while high-entropy ones are less affected. We further demonstrate that such phenomena are associated with LMM reasoning by controlled experiments, suggesting a potentially beneficial role of modulating low-entropy activations. To this end, we propose Activation Replay, a novel simple yet effective training-free approach that boosts multimodal reasoning of post-trained LMMs without requiring expensive policy optimization. Our design involves manipulation of visual tokens at test time, replaying low-entropy activations from the input context of base LMMs to regulating the RLVR counterparts. Activation Replay triggers better reasoning across diverse scenarios, including mathematics, o3-like visual agents, and video reasoning. We further show that Activation Replay boosts Pass@K and mitigates narrower reasoning coverage of RLVR. Our design is compared against alternative choices, such as replaying high-entropy activations instead of low-entropy ones, or direct cross-model intervention instead of manipulating input tokens, demonstrating the superiority of our implementation. Codes will be made publicly available.
Authors Yun Xing, Xiaobin Hu, Qingdong He, Jiangning Zhang, Shuicheng Yan, Shijian Lu, Yu-Gang Jiang
Comment 11 figures, 10 tables
Categories Computer Vision and Pattern Recognition
7
Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models NeurIPS'25
Title CN Directed-Tokens：一种面向大语言-视觉模型的鲁棒多模态对齐方法
Keywords 多模态对齐 · 顺序重建 · 视觉-语言模型 · 去幻觉 · 结构化学习
Summary 本文提出Directed-Tokens方法，通过在预训练和微调阶段引入图像与文本顺序重建任务，增强大语言-视觉模型的跨模态对齐能力。结合新设计的token机制和引导损失函数，显著提升了模型在视觉理解与推理任务上的表现，并在多个学术基准上达到SOTA。
Reason 该论文提出了一种通过引入图像与文本顺序重建任务来增强多模态对齐的新机制，并设计了Directed-Token和Image-to-Response Guided Loss以提升跨模态理解与推理能力。虽然其评估主要在通用VQA和指令遵循任务上，但其核心思想——通过结构化重建任务（如顺序恢复）强化视觉-语言对齐——可迁移至文档图像理解中的复杂排版建模与逻辑推理任务。例如，在长文档或多页表格理解中，段落或单元格的顺序错乱是常见挑战，该方法提供的‘顺序重建’训练范式有助于提升模型对文档结构的深层理解，缓解因布局复杂导致的幻觉问题。
Abstract Large multimodal models (LMMs) have gained impressive performance due to their outstanding capability in various understanding tasks. However, these models still suffer from some fundamental limitations related to robustness and generalization due to the alignment and correlation between visual and textual features. In this paper, we introduce a simple but efficient learning mechanism for improving the robust alignment between visual and textual modalities by solving shuffling problems. In particular, the proposed approach can improve reasoning capability, visual understanding, and cross-modality alignment by introducing two new tasks: reconstructing the image order and the text order into the LMM's pre-training and fine-tuning phases. In addition, we propose a new directed-token approach to capture visual and textual knowledge, enabling the capability to reconstruct the correct order of visual inputs. Then, we introduce a new Image-to-Response Guided loss to further improve the visual understanding of the LMM in its responses. The proposed approach consistently achieves state-of-the-art (SoTA) performance compared with prior LMMs on academic task-oriented and instruction-following LMM benchmarks.
Authors Thanh-Dat Truong, Huu-Thien Tran, Tran Thai Son, Bhiksha Raj, Khoa Luu
Comment Accepted to NeurIPS'25
Categories Computer Vision and Pattern Recognition
7
PRInTS: Reward Modeling for Long-Horizon Information Seeking
Title CN PRInTS：面向长视野信息检索的奖励建模
Keywords 奖励建模 · 信息检索 · 长序列推理 · 上下文压缩 · AI代理
Summary PRInTS是一种面向长视野信息检索任务的生成式过程奖励模型，能够对工具调用过程中的每一步进行多维度质量评估（如工具调用的信息量、输出解读的合理性），并通过轨迹摘要机制压缩不断增长的上下文。实验表明，结合best-of-n采样，PRInTS能显著提升开源模型在FRAMES、GAIA和WebWalkerQA等基准上的表现，甚至媲美更大规模的前沿模型。该方法为构建具备长期记忆与推理能力的文档理解代理提供了新的训练与引导范式。
Reason 该工作提出了一种生成式过程奖励模型PRInTS，具备密集评分和轨迹摘要能力，可对多步信息检索中的每一步进行细粒度评估，并通过上下文压缩缓解长程任务中的信息膨胀问题。这一方法可迁移至文档图像理解中的长文档推理场景：例如在跨页表格识别或复杂表单问答中，代理需多次定位、提取并推理局部区域信息，PRInTS的轨迹总结机制能有效压缩视觉-语言上下文，其多维度打分能力可用于判断OCR输出是否被正确解读、当前聚焦区域是否推进了任务进展，从而提升DIU系统在长文档上的推理鲁棒性与效率。
Abstract Information-seeking is a core capability for AI agents, requiring them to gather and reason over tool-generated information across long trajectories. However, such multi-step information-seeking tasks remain challenging for agents backed by language models. While process reward models (PRMs) can guide agents by ranking candidate steps at test-time, existing PRMs, designed for short reasoning with binary judgment, cannot capture richer dimensions of information-seeking steps, such as tool interactions and reasoning over tool outputs, nor handle the rapidly growing context in long-horizon tasks. To address these limitations, we introduce PRInTS, a generative PRM trained with dual capabilities: (1) dense scoring based on the PRM's reasoning across multiple step quality dimensions (e.g., interpretation of tool outputs, tool call informativeness) and (2) trajectory summarization that compresses the growing context while preserving essential information for step evaluation. Extensive evaluations across FRAMES, GAIA (levels 1-3), and WebWalkerQA (easy-hard) benchmarks on multiple models, along with ablations, reveal that best-of-n sampling with PRInTS enhances information-seeking abilities of open-source models as well as specialized agents, matching or surpassing the performance of frontier models with a much smaller backbone agent and outperforming other strong reward modeling baselines.
Authors Jaewoo Lee, Archiki Prasad, Justin Chih-Yao Chen, Zaid Khan, Elias Stengel-Eskin, Mohit Bansal
Comment 18 pages, code: https://github.com/G-JWLee/PRInTS
Categories Artificial Intelligence, Computation and Language
7
Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs
Title CN 为VLM中的工具集成推理扩展代理式强化学习
Keywords 视觉语言模型 · 代理式推理 · 强化学习 · 工具集成 · 多模态推理
Summary 本文提出了VISTA-Gym，一个支持大规模代理式强化学习的训练环境，旨在提升视觉语言模型在多步视觉交互中的工具集成推理能力。该环境统一了13个数据集中的7类真实世界任务，提供标准化的视觉工具接口、可执行的交互流程和可验证的反馈信号。基于此，作者训练了VISTA-R1模型，能够在多轮对话中交替使用工具并进行推理，在11个高难度VQA基准上显著超越同类模型。该方法为构建具备主动感知与工具调用能力的文档理解系统提供了新范式。
Reason 该论文提出VISTA-Gym，一个用于训练具备工具集成能力的视觉语言模型的可扩展环境，并通过多轮轨迹采样和端到端强化学习训练VISTA-R1，实现工具使用与代理式推理的交织。虽然其任务设定涵盖多个多模态推理场景，但其核心贡献——标准化的视觉工具接口（如定位、解析）、可执行交互循环和验证反馈机制——是解决文档图像理解中复杂逻辑推理与细粒度信息提取的潜在上游方法。例如，在长文档VQA或表格结构识别中，模型常因缺乏中间操作工具（如OCR校正、区域聚焦）而产生幻觉或遗漏关键字段；本工作提供的‘工具调用+强化学习优化’框架可迁移用于构建能主动调用OCR修正模块或布局分析工具的DIU系统，从而缓解OCR错误传播问题。
Abstract While recent vision-language models (VLMs) demonstrate strong image understanding, their ability to "think with images", i.e., to reason through multi-step visual interactions, remains limited. We introduce VISTA-Gym, a scalable training environment for incentivizing tool-integrated visual reasoning capabilities in VLMs. VISTA-Gym unifies diverse real-world multimodal reasoning tasks (7 tasks from 13 datasets in total) with a standardized interface for visual tools (e.g., grounding, parsing), executable interaction loops, verifiable feedback signals, and efficient trajectory logging, enabling visual agentic reinforcement learning at scale. While recent VLMs exhibit strong text-only reasoning, both proprietary and open-source models still struggle with tool selection, invocation, and coordination. With VISTA-Gym, we train VISTA-R1 to interleave tool-use with agentic reasoning via multi-turn trajectory sampling and end-to-end reinforcement learning. Extensive experiments across 11 public reasoning-intensive VQA benchmarks show that VISTA-R1-8B outperforms state-of-the-art baselines with similar sizes by 9.51%-18.72%, demonstrating VISTA-Gym as an effective training ground to unlock the tool-integrated reasoning capabilities for VLMs.
Authors Meng Lu, Ran Xu, Yi Fang, Wenxuan Zhang, Yue Yu, Gaurav Srivastava, Yuchen Zhuang, Mohamed Elhoseiny, Charles Fleming, Carl Yang, Zhengzhong Tu, Yang Xie, Guanghua Xiao, Hanrui Wang, Di Jin, Wenqi Shi, Xuan Wang
Comment 17 pages, 9 figures, work in progress
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
7
Dynamic Granularity Matters: Rethinking Vision Transformers Beyond Fixed Patch Splitting
Title CN 动态粒度至关重要：超越固定图像块分割的视觉Transformer再思考
Keywords 视觉Transformer · 动态粒度 · 多尺度表示 · 视觉表示学习 · 高效注意力
Summary 本文提出Granularity-driven Vision Transformer（Grc-ViT），通过引入基于图像复杂度（边缘密度、熵、频域特征）的粗粒度评估模块，动态确定最优patch和窗口尺寸，并在精细化模块中据此优化注意力计算。模型通过端到端学习两个参数α和β来平衡全局推理与局部感知，在保持计算效率的同时增强了细粒度判别能力。
Reason 该论文提出了一种动态调整视觉粒度的Vision Transformer架构（Grc-ViT），通过评估图像复杂度来自适应选择patch大小，从而在全局建模与局部细节捕捉之间取得平衡。这一方法对文档图像理解具有明确的迁移价值：文档图像常同时包含大范围布局结构（如章节标题）和细粒度文本元素（如小字号表格内容），传统固定patch划分难以兼顾。Grc-ViT的动态粒度机制可提升VLM处理复杂排版文档时的效率与识别精度，尤其适用于高分辨率扫描件中的多尺度信息提取。
Abstract Vision Transformers (ViTs) have demonstrated strong capabilities in capturing global dependencies but often struggle to efficiently represent fine-grained local details. Existing multi-scale approaches alleviate this issue by integrating hierarchical or hybrid features; however, they rely on fixed patch sizes and introduce redundant computation. To address these limitations, we propose Granularity-driven Vision Transformer (Grc-ViT), a dynamic coarse-to-fine framework that adaptively adjusts visual granularity based on image complexity. It comprises two key stages: (1) Coarse Granularity Evaluation module, which assesses visual complexity using edge density, entropy, and frequency-domain cues to estimate suitable patch and window sizes; (2) Fine-grained Refinement module, which refines attention computation according to the selected granularity, enabling efficient and precise feature learning. Two learnable parameters, α and \b{eta}, are optimized end-to-end to balance global reasoning and local perception. Comprehensive evaluations demonstrate that Grc-ViT enhances fine-grained discrimination while achieving a superior trade-off between accuracy and computational efficiency.
Authors Qiyang Yu, Yu Fang, Tianrui Li, Xuemei Cao, Yan Chen, Jianghao Li, Fan Min
Comment 10 pages, 7 figures
Categories Computer Vision and Pattern Recognition
7
Upsample Anything: A Simple and Hard to Beat Baseline for Feature Upsampling
Title CN 上采样任何东西：一种简单且难以超越的特征上采样基线
Keywords 特征上采样 · 测试时优化 · 高分辨率恢复 · 边缘感知 · 视觉基础模型
Summary Upsample Anything 提出一种通用、无需训练的测试时优化框架，通过学习结合空间与范围线索的各向异性高斯核，将低分辨率特征高效恢复为高分辨率像素级输出。该方法无需微调，运行速度快，在语义分割、深度估计等任务中达到SOTA效果，并具备跨模型、跨模态的迁移能力。
Reason 该论文提出了一种无需训练、基于测试时优化的轻量级特征上采样方法，通过学习各向异性的高斯核实现从低分辨率到高分辨率像素级输出的重建。其核心优势在于跨架构和模态的泛化能力，且保持边缘感知特性。这一方法可直接迁移至文档图像理解（DIU）领域，用于提升VLM在处理高分辨率文档时的细粒度定位能力，缓解因视觉特征下采样导致的文字边界模糊或小字丢失问题，尤其适用于复杂排版文档的布局分析与表格识别任务。
Abstract We present \textbf{Upsample Anything}, a lightweight test-time optimization (TTO) framework that restores low-resolution features to high-resolution, pixel-wise outputs without any training. Although Vision Foundation Models demonstrate strong generalization across diverse downstream tasks, their representations are typically downsampled by 14x/16x (e.g., ViT), which limits their direct use in pixel-level applications. Existing feature upsampling approaches depend on dataset-specific retraining or heavy implicit optimization, restricting scalability and generalization. Upsample Anything addresses these issues through a simple per-image optimization that learns an anisotropic Gaussian kernel combining spatial and range cues, effectively bridging Gaussian Splatting and Joint Bilateral Upsampling. The learned kernel acts as a universal, edge-aware operator that transfers seamlessly across architectures and modalities, enabling precise high-resolution reconstruction of features, depth, or probability maps. It runs in only $\approx0.419 \text{s}$ per 224x224 image and achieves state-of-the-art performance on semantic segmentation, depth estimation, and both depth and probability map upsampling. \textbf{Project page:} \href{https://seominseok0429.github.io/Upsample-Anything/}{https://seominseok0429.github.io/Upsample-Anything/}
Authors Minseok Seo, Mark Hamilton, Changick Kim
Comment 15 pages, 12 figures
Categories Computer Vision and Pattern Recognition
7
OmniDocLayout: Towards Diverse Document Layout Generation via Coarse-to-Fine LLM Learning
Title CN OmniDocLayout：通过粗到细的LLM学习实现多样化的文档布局生成
Keywords 文档布局生成 · 布局分析 · LLM · 数据集 · Coarse-to-Fine
Summary 本文针对现有文档布局生成研究局限于学术论文等简单结构的问题，提出了首个大规模多样化文档布局数据集OmniDocLayout-1M，涵盖报纸、杂志等多种复杂类型。同时设计了一个两阶段的LLM学习框架：先从粗粒度类别学习通用布局原则，再迁移到细粒度特定领域。实验表明该方法在多个域上显著优于现有布局生成方法和通用LLM，实现了更好的条件一致性和美学合理性。
Reason 该论文聚焦于文档布局生成任务，属于文档图像理解（DIU）中的布局分析（DLA）方向。作者构建了百万级多样化文档布局数据集OmniDocLayout-1M，并提出基于0.5B LLM的两阶段粗到细学习范式，以解决复杂、长序列文档布局生成中的连贯性问题。该方法在M$^6$Doc上表现优异，对DIU领域中复杂排版建模和生成具有实际参考价值，尤其为未来合成高质量训练数据提供了可能路径。
Abstract Document AI has advanced rapidly and is attracting increasing attention. Yet, while most efforts have focused on document layout analysis (DLA), its generative counterpart, layout generation, remains underexplored. Distinct from traditional graphic layout design and room layout planning, document layout generation typically involves a larger number of elements per page and exhibits greater structural diversity and complexity. Currently, a major obstacle lies in the scarcity of diverse document layouts: academic papers with Manhattan-style structures dominate existing studies, while open-world genres such as newspapers and magazines remain severely underrepresented. To address this gap, we curate OmniDocLayout-1M, the first million-scale dataset of diverse document layouts, covering six common document types and comprising contemporary layouts collected from multiple sources. Moreover, since existing methods struggle in complex domains and often fail to arrange long sequences coherently, we introduce OmniDocLayout-LLM, a 0.5B model with designed two-stage Coarse-to-Fine learning paradigm:1) learning universal layout principles from our dataset with coarse category definitions, and 2) transferring the knowledge to a specific domain with few fine-grained annotated samples. Extensive experiments demonstrate that our approach achieves strong performance on multiple domains in M$^6$Doc dataset, substantially surpassing both existing layout generation experts and several latest general-purpose LLMs. Our code, dataset, and models will be publicly released.
Authors Hengrui Kang, Zhuangcheng Gu, Zhiyuan Zhao, Zichen Wen, Bin Wang, Weijia Li, Conghui He
Comment TL;DR: With the proposed OmniDocLayout-1M dataset and the LLM-based coarse-to-fine learning strategy, we enable diverse and complex document layout generation that achieves both strong condition consistency and adherence to fundamental aesthetic principles
Categories Computer Vision and Pattern Recognition
7
Navigating Gigapixel Pathology Images with Large Multimodal Models
Title CN 利用大规模多模态模型导航千兆像素病理图像
Keywords 多模态大模型 · 高分辨率图像 · Agent框架 · 主动推理 · 医学图像
Summary 本文提出了GIANT框架和MultiPathQA基准，用于评估大规模多模态模型在千兆像素病理图像上的推理能力。通过模拟病理学家的视觉浏览行为，GIANT引导LMM逐步聚焦关键区域，显著优于传统的基于缩略图或随机块的方法，在专家设计的问题上甚至超过专用模型。该工作揭示了通用LMM在专业高分辨率视觉任务中的潜力。
Reason 该论文提出GIANT框架，使多模态大模型能够像病理学家一样迭代导航全切片图像（WSI），并构建了包含临床相关任务的MultiPathQA基准。虽然研究领域为医学图像，但其核心方法——基于Agent的高分辨率图像主动探索与推理机制——可直接迁移至文档图像理解中的长文档、高分辨率复杂排版处理。例如，DIU中面对多页PDF或高密度表格时，同样面临信息分散、需多步定位的问题，GIANT所展示的‘聚焦-推理-导航’闭环策略为解决此类问题提供了新范式。
Abstract Despite being widely used to support clinical care, general-purpose large multimodal models (LMMs) have generally shown poor or inconclusive performance in medical image interpretation, particularly in pathology, where gigapixel images are used. However, prior studies have used either low-resolution thumbnails or random patches, which likely underestimated model performance. Here, we ask whether LMMs can be adapted to reason coherently and accurately in the evaluation of such images. In this study, we introduce Gigapixel Image Agent for Navigating Tissue (GIANT), the first framework that allows LMMs to iteratively navigate whole-slide images (WSIs) like a pathologist. Accompanying GIANT, we release MultiPathQA, a new benchmark, which comprises 934 WSI-level questions, encompassing five clinically-relevant tasks ranging from cancer diagnosis to open-ended reasoning. MultiPathQA also includes 128 questions, authored by two professional pathologists, requiring direct slide interpretation. Using MultiPathQA, we show that our simple agentic system substantially outperforms conventional patch- and thumbnail-based baselines, approaching or surpassing the performance of specialized models trained on millions of images. For example, on pathologist-authored questions, GPT-5 with GIANT achieves 62.5% accuracy, outperforming specialist pathology models such as TITAN (43.8%) and SlideChat (37.5%). Our findings reveal the strengths and limitations of current foundation models and ground future development of LMMs for expert reasoning in pathology.
Authors Thomas A. Buckley, Kian R. Weihrauch, Katherine Latham, Andrew Z. Zhou, Padmini A. Manrai, Arjun K. Manrai
Categories Computer Vision and Pattern Recognition
7
Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization AAAI2026
Title CN 通过基于梯度的表示优化在基础大语言模型中激发思维链
Keywords 思维链 · 隐藏状态操纵 · 推理增强 · 基础大模型 · 表示优化
Summary 本文提出一种基于梯度的隐藏状态优化方法，用于在未微调的基础大语言模型中激发思维链推理能力。通过将推理过程建模为带先验正则化的条件生成优化问题，该方法能够在不破坏语言连贯性的前提下，引导模型隐状态走向更具推理性的轨迹。实验表明其在数学、常识和逻辑推理任务上优于现有引导技术。
Reason 该论文提出了一种通过优化隐藏状态来激发基础LLM中思维链推理能力的方法，相较于传统的线性激活引导，其引入了概率条件生成框架和正则化机制，有效缓解了分布偏移问题。此方法属于推理阶段干预（Intervention/Steering）的源头创新，可直接迁移至文档图像理解中的复杂逻辑推理任务，例如DocVQA中需要多步推导的问题（如‘计算表格中某季度同比增长率’），帮助VLM在无微调情况下激活内在推理路径，同时保持文本生成质量，对解决长文档或多表关联下的幻觉与推理断裂问题具有潜在价值。
Abstract Chain-of-Thought (CoT) reasoning is a critical capability for large language models (LLMs), enabling them to tackle com- plex multi-step tasks. While base LLMs, pre-trained on general text corpora, often struggle with reasoning due to a lack of specialized training, recent studies reveal their latent reason- ing potential tied to hidden states. However, existing hidden state manipulation methods, such as linear activation steering, suffer from limitations due to their rigid and unconstrained nature, often leading to distribution shifts and degraded text quality. In this work, we propose a novel approach for elic- iting CoT reasoning from base LLMs through hidden state manipulation grounded in probabilistic conditional generation. By reformulating the challenge as an optimization problem with a balanced likelihood and prior regularization framework, our method guides hidden states toward reasoning-oriented trajectories while preserving linguistic coherence. Extensive evaluations across mathematical, commonsense, and logical reasoning benchmarks demonstrate that our approach con- sistently outperforms existing steering methods, offering a theoretically principled and effective solution for enhancing reasoning capabilities in base LLMs.
Authors Zijian Wang, Yanxiang Ma, Chang Xu
Comment AAAI2026
Categories Computation and Language
7
One SPACE to Rule Them All: Jointly Mitigating Factuality and Faithfulness Hallucinations in LLMs NIPS 2025
Title CN 一个SPACE统管一切：联合缓解LLM中的事实性与忠实性幻觉
Keywords 幻觉缓解 · 激活空间编辑 · 事实性 · 忠实性 · 子空间干预
Summary 本文发现大语言模型中事实性幻觉与忠实性幻觉共享神经激活子空间，提出SPACE框架通过建模双任务特征、结合谱聚类与注意力头显著性分析，定位并编辑这些共享子空间，实现两类幻觉的协同缓解。实验表明该方法在多个基准上优于现有独立干预方法。
Reason 该论文提出通过编辑LLM激活空间中的共享子空间来联合缓解事实性与忠实性幻觉，其核心思想——在神经表示层面识别并干预导致不同类型幻觉的重叠子空间——具有迁移到文档图像理解（DIU）的潜力。在DocVQA任务中，VLM常因视觉-语言对齐偏差或OCR后处理错误产生幻觉，尤其是面对复杂排版或低质量扫描件时。本文提出的混合探针策略（谱聚类+注意力头显著性评分）可被用于分析和干预VLM在处理文档图像时的内部表征，从而针对性抑制由视觉噪声或布局干扰引发的幻觉，提升回答的忠实性与事实准确性。
Abstract LLMs have demonstrated unprecedented capabilities in natural language processing, yet their practical deployment remains hindered by persistent factuality and faithfulness hallucinations. While existing methods address these hallucination types independently, they inadvertently induce performance trade-offs, as interventions targeting one type often exacerbate the other. Through empirical and theoretical analysis of activation space dynamics in LLMs, we reveal that these hallucination categories share overlapping subspaces within neural representations, presenting an opportunity for concurrent mitigation. To harness this insight, we propose SPACE, a unified framework that jointly enhances factuality and faithfulness by editing shared activation subspaces. SPACE establishes a geometric foundation for shared subspace existence through dual-task feature modeling, then identifies and edits these subspaces via a hybrid probe strategy combining spectral clustering and attention head saliency scoring. Experimental results across multiple benchmark datasets demonstrate the superiority of our approach.
Authors Pengbo Wang, Chaozhuo Li, Chenxu Wang, Liwen Zheng, Litian Zhang, Xi Zhang
Comment Accepted as NIPS 2025 poster
Categories Artificial Intelligence, Computation and Language
7
Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning
Title CN 基于感知证据锚定的强化学习用于多模态推理
Keywords 多模态推理 · 强化学习 · 视觉幻觉 · 感知-推理协同 · 奖励设计
Summary 本文提出PEARL，一种感知与推理协同的强化学习框架，用于提升VLM的多模态推理可靠性。PEARL通过构建感知清单（一系列可验证的视觉子问题）来评估模型对输入图像的理解，并据此生成感知奖励。只有当模型通过感知检查时，才允许其进行高级推理并获得推理奖励；否则训练过程被阻断，防止基于错误感知的幻觉推理。该方法在MathVerse等基准上显著优于现有RLVR方法。
Reason 该论文提出PEARL框架，通过引入可验证的视觉感知子问题（感知清单）来约束VLM的推理过程，确保推理建立在可靠的视觉理解基础上。这一方法直接针对文档图像理解中的核心痛点——OCR幻觉与视觉幻觉：在DocVQA任务中，模型常因误读文本或布局而生成看似合理但错误的回答。PEARL的感知奖励机制可被迁移用于强制模型在回答前正确识别关键视觉元素（如表格结构、字段位置、文字内容），从而提升答案的忠实性与可解释性。虽然其验证实验集中在数学图像理解（MathVerse），但其‘先验证感知、再允许推理’的设计范式对长文档、复杂排版下的可信推理具有明确迁移价值。
Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs) and is now being applied to Vision-Language Models (VLMs). However, vanilla RLVR for VLMs verifies only the final textual output, critically neglecting the foundational step of visual perception. This oversight leads to visual hallucinations and reward hacking, as reasoning built upon flawed perception is inherently unreliable. To address this, we propose PEARL (Perceptual-Evidence Anchored Reinforced Learning), a dual-branch, perception-reasoning synergistic that strengthens multimodal reasoning by explicitly anchoring it to verified visual evidence. For each reasoning-oriented QA instance, PEARL first derive a perception checklist -- a set of perception-oriented sub-questions with verifiable answers that probe the model's understanding of key visual evidence. During training, auxiliary rollouts on this checklist yield a perceptual reward that both directly reinforces the model's perception ability and acts as a fidelity gate for reasoning. If the model passes the perception check, its policy update is biased towards evidence-anchored reasoning. Otherwise, the process is halted to prevent reasoning from flawed premises. PEARL can be seamlessly integrated with popular RL methods like GRPO and DAPO. Comprehensive experiments show PEARL achieves substantial gains on multimodal reasoning benchmarks, e.g., a +9.7% improvement over the baseline and +6.6% over GRPO on MathVerse.
Authors Chi Zhang, Haibo Qiu, Qiming Zhang, Yufei Xu, Zhixiong Zeng, Siqi Yang, Peng Shi, Lin Ma, Jing Zhang
Categories Computer Vision and Pattern Recognition
7
SciPostLayoutTree: A Dataset for Structural Analysis of Scientific Posters
Title CN SciPostLayoutTree：一种用于科学海报结构分析的数据集
Keywords 文档布局分析 · 阅读顺序预测 · 结构化表示 · Layout Tree Decoder · 复杂排版
Summary 本文提出了SciPostLayoutTree，一个包含约8000张科学海报的数据集，标注了阅读顺序和元素间的父子关系，填补了海报类文档结构分析的空白。为应对其中频繁出现的复杂空间关系，作者设计了融合视觉与几何特征的Layout Tree Decoder，并引入beam search提升整体结构合理性。实验表明该方法在挑战性空间关系预测上表现更优，为结构化文档理解提供了新基准与工具。
Reason 该论文构建了一个大规模的科学海报布局结构数据集，并提出了针对阅读顺序和父子关系预测的Layout Tree Decoder模型。其核心贡献在于处理空间上复杂的布局关系（如上下、横向、长距离关联），这与文档图像理解中复杂排版解析高度相关。所提出的结合视觉特征、边界框位置信息与类别、并使用beam search优化序列合理性的方法，可迁移至DocVQA或Layout Analysis任务中提升对非线性排版的理解能力。
Abstract Scientific posters play a vital role in academic communication by presenting ideas through visual summaries. Analyzing reading order and parent-child relations of posters is essential for building structure-aware interfaces that facilitate clear and accurate understanding of research content. Despite their prevalence in academic communication, posters remain underexplored in structural analysis research, which has primarily focused on papers. To address this gap, we constructed SciPostLayoutTree, a dataset of approximately 8,000 posters annotated with reading order and parent-child relations. Compared to an existing structural analysis dataset, SciPostLayoutTree contains more instances of spatially challenging relations, including upward, horizontal, and long-distance relations. As a solution to these challenges, we develop Layout Tree Decoder, which incorporates visual features as well as bounding box features including position and category information. The model also uses beam search to predict relations while capturing sequence-level plausibility. Experimental results demonstrate that our model improves the prediction accuracy for spatially challenging relations and establishes a solid baseline for poster structure analysis. The dataset is publicly available at https://huggingface.co/datasets/omron-sinicx/scipostlayouttree. The code is also publicly available at https://github.com/omron-sinicx/scipostlayouttree.
Authors Shohei Tanaka, Atsushi Hashimoto, Yoshitaka Ushiku
Categories Computer Vision and Pattern Recognition
7
ARIAL: An Agentic Framework for Document VQA with Precise Answer Localization
Title CN ARIAL：一种用于文档VQA的具有精确答案定位能力的智能体框架
Keywords 文档VQA · 智能体框架 · 答案定位 · 模块化架构 · 可解释性
Summary ARIAL是一个面向文档视觉问答的模块化智能体框架，利用LLM作为规划器调度OCR、语义检索、大模型生成和定位对齐等多个专用模块，实现在DocVQA任务中既准确回答问题又能精确定位答案位置。在多个基准测试上达到SOTA，兼顾了性能与可解释性。
Reason 该论文提出了一种模块化智能体框架ARIAL，通过LLM代理协调OCR、语义检索、答案生成和文本到区域对齐等专用工具，在DocVQA任务中同时优化文本准确性和空间定位精度。其核心贡献在于将复杂文档理解任务分解为可审计的子任务流程，提升了系统的可解释性与性能。这种基于智能体的任务分解与工具编排机制属于上游方法论创新，尤其适用于解决长文档或多步骤推理中的信息迷失问题，具备迁移到其他DIU场景（如表格解析、布局分析）的潜力。
Abstract Document Visual Question Answering (VQA) requires models to not only extract accurate textual answers but also precisely localize them within document images, a capability critical for interpretability in high-stakes applications. However, existing systems achieve strong textual accuracy while producing unreliable spatial grounding, or sacrifice performance for interpretability. We present ARIAL (Agentic Reasoning for Interpretable Answer Localization), a modular framework that orchestrates specialized tools through an LLM-based planning agent to achieve both precise answer extraction and reliable spatial grounding. ARIAL decomposes Document VQA into structured subtasks: OCR-based text extraction with TrOCR, retrieval-augmented context selection using semantic search, answer generation via a fine-tuned Gemma 3-27B model, and explicit bounding-box localization through text-to-region alignment. This modular architecture produces transparent reasoning traces, enabling tool-level auditability and independent component optimization. We evaluate ARIAL on four benchmarks (DocVQA, FUNSD, CORD, and SROIE) using both textual accuracy (ANLS) and spatial precision (mAP at IoU 0.50 to 0.95). ARIAL achieves state-of-the-art results across all datasets: 88.7 ANLS and 50.1 mAP on DocVQA, 90.0 ANLS and 50.3 mAP on FUNSD, 85.5 ANLS and 60.2 mAP on CORD, and 93.1 ANLS on SROIE, surpassing the previous best method (DLaVA) by +2.8 ANLS and +3.9 mAP on DocVQA. Our work demonstrates how agentic orchestration of specialized tools can simultaneously improve performance and interpretability, providing a pathway toward trustworthy, explainable document AI systems.
Authors Ahmad Mohammadshirazi, Pinaki Prasad Guha Neogi, Dheeraj Kulshrestha, Rajiv Ramnath
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
7
Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Models
Title CN Athena：通过数据高效的进程奖励模型增强多模态推理
Keywords 进程奖励模型 · 多模态推理 · 测试时扩展 · 去幻觉 · 数据高效
Summary 本文提出了Athena-PRM，一种数据高效的多模态进程奖励模型，通过弱模型与强模型之间的预测一致性自动筛选高质量的推理步骤标签，仅用5,000样本即可训练出高性能PRM。该模型在测试时验证、推理步骤评估和奖励排序微调等场景中均表现优异，在WeMath和MathVista上分别提升10.2和7.1个点，并在VisualProcessBench上达到SOTA。基于此还构建了Athena-7B模型，在多个基准上显著超越基线。
Reason 该论文提出了一种高效的多模态进程奖励模型（PRM），可用于评估复杂推理任务中每一步的正确性。其核心创新在于利用强/弱补全器之间的预测一致性来自动生成高质量的步骤级标签，显著降低对人工标注或高成本蒙特卡洛采样的依赖。这一方法可直接迁移到文档图像理解（DIU）中的长文档多步推理场景，例如在DocVQA中验证模型是否按逻辑顺序逐步提取并推理信息，从而缓解幻觉和推理断裂问题。此外，其在测试时扩展（test-time scaling）中提升10.2点的表现表明该方法对提升VLM在复杂任务上的鲁棒性具有实际价值。
Abstract We present Athena-PRM, a multimodal process reward model (PRM) designed to evaluate the reward score for each step in solving complex reasoning problems. Developing high-performance PRMs typically demands significant time and financial investment, primarily due to the necessity for step-level annotations of reasoning steps. Conventional automated labeling methods, such as Monte Carlo estimation, often produce noisy labels and incur substantial computational costs. To efficiently generate high-quality process-labeled data, we propose leveraging prediction consistency between weak and strong completers as a criterion for identifying reliable process labels. Remarkably, Athena-PRM demonstrates outstanding effectiveness across various scenarios and benchmarks with just 5,000 samples. Furthermore, we also develop two effective strategies to improve the performance of PRMs: ORM initialization and up-sampling for negative data. We validate our approach in three specific scenarios: verification for test time scaling, direct evaluation of reasoning step correctness, and reward ranked fine-tuning. Our Athena-PRM consistently achieves superior performance across multiple benchmarks and scenarios. Notably, when using Qwen2.5-VL-7B as the policy model, Athena-PRM enhances performance by 10.2 points on WeMath and 7.1 points on MathVista for test time scaling. Furthermore, Athena-PRM sets the state-of-the-art (SoTA) results in VisualProcessBench and outperforms the previous SoTA by 3.9 F1-score, showcasing its robust capability to accurately assess the correctness of the reasoning step. Additionally, utilizing Athena-PRM as the reward model, we develop Athena-7B with reward ranked fine-tuning and outperforms baseline with a significant margin on five benchmarks.
Authors Shuai Wang, Zhenhua Liu, Jiaheng Wei, Xuanwu Yin, Dong Li, Emad Barsoum
Comment v3: fix typos, add data scaling exp
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
7
Beneficial Reasoning Behaviors in Agentic Search and Effective Post-training to Obtain Them
Title CN 代理搜索中的有益推理行为及通过后训练实现的有效方法
Keywords 代理搜索 · 推理行为 · 行为引导 · 后训练 · 强化学习
Summary 本文提出一种基于LLM的分析管道，用于识别代理搜索中的四种有益推理行为：信息验证、权威评估、自适应搜索和错误恢复。基于此，作者设计了‘行为引导’训练方法，先在具备这些行为但不一定结果正确的轨迹上进行监督微调（SFT），再进行强化学习。实验表明该方法在多个多跳问答基准上优于传统RL和仅基于正确结果的SFT方法，并揭示推理行为本身是性能提升的关键。该方法为构建具备稳健推理能力的文档理解系统提供了可迁移的训练范式。
Reason 该论文提出了‘行为引导’（Behavior Priming）的训练方法，通过在SFT阶段注入具有特定推理行为（如信息验证、权威性评估、自适应搜索、错误恢复）的轨迹，显著提升代理在复杂信息检索任务中的表现。这些推理行为——尤其是信息验证和错误恢复——可直接迁移至文档图像理解中的长文档多跳问答（DocVQA）或跨页信息整合任务中，用于缓解模型因上下文缺失或OCR幻觉导致的推理偏差。此外，其强调‘推理过程质量优于最终答案正确性’的理念，为DIU中基于强化学习的训练提供了新思路。
Abstract Agentic search leverages LLMs to solve complex user information needs by executing a multi-step process of planning, searching, and synthesizing information to provide answers. This paradigm introduces unique challenges for LLMs' agentic reasoning capabilities when interacting with search systems. In this paper, we propose an LLM-based pipeline to study effective reasoning behavior patterns in agentic search by analyzing agentic search trajectories. Using this pipeline, we identify four beneficial reasoning behaviors: Information Verification, Authority Evaluation, Adaptive Search, and Error Recovery. Based on these findings, we propose a technique called Behavior Priming to train agentic search models. It synthesizes trajectories that exhibit these four behaviors and integrates them into the agentic search model through SFT, followed by standard reinforcement learning. Experiments on Qwen3-1.7B and Llama3.2-3B-Instruct across three web benchmarks and seven multi-hop QA benchmarks demonstrate that behavior priming 1) yields significant performance gains compared to training with direct RL, and 2) outperforms other SFT-then-RL baselines, such as those SFT on randomly selected trajectories or on trajectories with merely correct outcomes. Crucially, we demonstrate that the reasoning behaviors, rather than the correctness of the final answer, is the critical factor for achieving strong performance in RL: SFT on trajectories with reasoning behaviors but incorrect answers leads to comparable performance with SFT on those with reasoning behaviors and correct answers. Our analysis further reveals that the introduced reasoning behaviors endow models with more effective exploration (higher pass@k and entropy) and test-time scaling (longer trajectories) capabilities, providing a strong foundation for RL. Our code are avalible at https://github.com/cxcscmu/Behavior_Priming_For_Agentic_Search.
Authors Jiahe Jin, Abhijay Paladugu, Chenyan Xiong
Categories Artificial Intelligence
7
MGA-VQA: Secure and Interpretable Graph-Augmented Visual Question Answering with Memory-Guided Protection Against Unauthorized Knowledge Use
Title CN MGA-VQA：基于图增强与记忆引导保护的安全可解释视觉问答方法
Keywords DocVQA · 空间图推理 · 记忆增强 · 可解释性 · 多模态
Summary 本文提出MGA-VQA，一种用于文档视觉问答的多模态框架，通过结合文本级编码、空间图推理、记忆增强推理和问题引导压缩，在FUNSD、CORD、SROIE等多个标准DocVQA数据集上实现了更优的准确率与效率表现。相比传统黑箱模型，该方法通过图结构决策路径提升推理可解释性，并利用记忆机制防止未经授权的知识使用，增强了安全性。
Reason 该论文针对DocVQA任务，提出了一种融合空间图推理、记忆增强推理和问题引导压缩的多模态框架，有效应对文档理解中的多跳推理、空间关系建模和高分辨率处理难题。其引入的图结构化推理路径和结构化记忆访问机制，提升了模型的可解释性和推理透明度，属于DIU本题范畴内的扎实改进工作。尤其在空间布局建模和多步推理方面的方法设计，对复杂排版文档的理解具有明确价值。
Abstract Document Visual Question Answering (DocVQA) requires models to jointly understand textual semantics, spatial layout, and visual features. Current methods struggle with explicit spatial relationship modeling, inefficiency with high-resolution documents, multi-hop reasoning, and limited interpretability. We propose MGA-VQA, a multi-modal framework that integrates token-level encoding, spatial graph reasoning, memory-augmented inference, and question-guided compression. Unlike prior black-box models, MGA-VQA introduces interpretable graph-based decision pathways and structured memory access for enhanced reasoning transparency. Evaluation across six benchmarks (FUNSD, CORD, SROIE, DocVQA, STE-VQA, and RICO) demonstrates superior accuracy and efficiency, with consistent improvements in both answer prediction and spatial localization.
Authors Ahmad Mohammadshirazi, Pinaki Prasad Guha Neogi, Dheeraj Kulshrestha, Rajiv Ramnath
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
7
FastMMoE: Accelerating Multimodal Large Language Models through Dynamic Expert Activation and Routing-Aware Token Pruning
Title CN FastMMoE：通过动态专家激活与路由感知的Token剪枝加速多模态大语言模型
Keywords 多模态大模型 · 视觉Token剪枝 · MoE · 推理加速 · 高分辨率处理
Summary 本文提出了FastMMoE，一种面向MoE型多模态大语言模型的训练-free加速框架。它结合两种策略：一是减少视觉Token激活的专家数量以降低计算量；二是基于路由概率分布相似性进行Token剪枝，去除高度冗余的视觉Token。在DeepSeek-VL2和InternVL3.5等大型MoE-MLLM上实验表明，该方法可在保留约95.5%性能的前提下减少高达55.0%的FLOPs，显著优于现有的密集模型剪枝方法。
Reason 该论文提出了一种针对MoE架构的多模态大模型无训练加速框架，通过专家激活减少和路由感知的视觉Token剪枝来降低高分辨率输入带来的计算开销。此方法可直接迁移至文档图像理解（DIU）领域，尤其是处理高分辨率文档图像时，能有效缓解视觉Token冗余导致的长序列推理瓶颈，提升VLM在复杂排版、密集文本场景下的推理效率，具有明确的实用性与迁移价值。
Abstract Multimodal large language models (MLLMs) have achieved impressive performance, but high-resolution visual inputs result in long sequences of visual tokens and substantial inference latency. Reducing redundant visual tokens is critical to ease computational/memory burdens while preserving performance, enabling MLLM deployment in resource-constrained or latency-sensitive scenarios. Current visual token pruning methods mainly rely on attention-based redundancy analysis and are tailored to dense architectures. We propose Fast Multimodal Mixture-of-Experts (FastMMoE), a training-free acceleration framework for mixture-of-experts (MoE) based MLLMs, developed from a routing analysis perspective. FastMMoE combines two complementary strategies: (i) expert activation reduction for visual tokens to minimize unnecessary expert computation; and (ii) routing-aware token pruning that leverages similarity in routing probability distributions to identify and remove highly redundant visual tokens. Experiments on large-scale MoE-MLLMs such as DeepSeek-VL2 and InternVL3.5 demonstrate that FastMMoE can reduce FLOPs by up to 55.0% while retaining approximately 95.5% of the original performance, consistently outperforming dense-model pruning baselines including FastV and SparseVLM across multiple retention rates.
Authors Guoyang Xia, Yifeng Ding, Fengfa Li, Lei Ren, Wei Chen, Fangxiang Feng, Xiaojie Wang
Categories Computer Vision and Pattern Recognition
7
SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization
Title CN SPINE：基于熵带正则化的令牌选择性测试时强化学习
Keywords 测试时强化学习 · 推理优化 · 令牌选择 · 去幻觉 · 多模态推理
Summary SPINE是一种面向大语言模型和多模态大模型的测试时自适应方法，针对现有测试时强化学习中因均匀序列更新导致的响应坍缩和性能下降问题，提出仅对推理链中高熵的‘分叉令牌’进行参数更新，并引入熵带正则项以平衡探索与利用。该方法无需真实标签或奖励模型，在多种推理任务上显著提升了Pass@1指标，且保持了输出长度和训练稳定性。
Reason 该论文提出了一种测试时无需标签的强化学习框架SPINE，通过识别推理路径中的高熵分叉令牌（forking tokens）并仅对这些关键位置进行更新，结合熵带正则化来维持探索与抑制噪声。这一机制可有效迁移到文档图像理解（DIU）中的复杂推理任务，例如在DocVQA中处理多步逻辑推理或长文档信息整合时，避免模型过早收敛到错误路径、缓解幻觉问题，并提升答案的准确性和稳定性。其不依赖监督信号的特点也适合DIU中缺乏精细标注的场景。
Abstract Large language models (LLMs) and multimodal LLMs (MLLMs) excel at chain-of-thought reasoning but face distribution shift at test-time and a lack of verifiable supervision. Recent test-time reinforcement learning (TTRL) methods derive label-free pseudo-rewards from self-consistency voting over sampled trajectories, yet they often collapse: the majority-vote reward prevails, responses shorten, and Pass@1 declines. We trace this to uniform sequence updates in which most tokens are low-entropy followers, while a small high-entropy subset determines the reasoning branches. Thus we propose SPINE, a token-selective test-time reinforcement learning framework that (i) updates only forking tokens, the high-entropy branch points identified from forward-pass statistics, and (ii) applies an entropy-band regularizer at those tokens to sustain exploration when entropy is too low and to suppress noisy supervision when it is too high. SPINE plugs into GRPO-style objectives, optionally with a KL anchor, and requires no labels or reward models. Across ten benchmarks spanning multimodal VQA, general and expert QA, mathematical reasoning, and medical QA, SPINE consistently improves Pass@1 over TTRL while avoiding response-length collapse and yielding more stable training dynamics on both LLM and MLLM backbones. These results indicate that aligning updates with chain-of-thought branch points is a simple and label-free mechanism for stable and effective test-time adaptation in reasoning models. Code is available at https://github.com/JianghaoWu/SPINE.
Authors Jianghao Wu, Yasmeen George, Jin Ye, Yicheng Wu, Daniel F. Schmidt, Jianfei Cai
Categories Computation and Language
7
Lost in Translation and Noise: A Deep Dive into the Failure Modes of VLMs on Real-World Tables EurIPS 2025 Workshop on AI For Tabular Data
Title CN 在翻译与噪声中迷失：深入探究VLM在真实世界表格上的失败模式
Keywords 表格问答 · 视觉语言模型 · 多语言 · 视觉噪声 · 基准数据集
Summary 现有表格问答数据集多为干净、单语的数字表格，无法反映现实中文档的复杂性。为此，作者构建了MirageTVQA，包含近6万个多语言（24种语言）QA对，并引入真实感的视觉噪声以模拟扫描文档。实验表明，主流VLM在视觉噪声下性能显著下降，且存在英语优先的推理迁移障碍。该基准有助于推动更鲁棒的表格理解模型发展。
Reason 该论文提出了一个面向真实场景的多语言、带视觉噪声的表格问答新基准MirageTVQA，直接针对文档图像理解中的核心挑战——视觉退化（如扫描质量差）和多语言泛化问题。其揭示的VLM在噪声下性能大幅下降（超过35%）和英语优先偏见，精准指出了当前模型在实际DIU应用中的脆弱性。该工作虽为数据集导向，但具有方法论警示意义，可推动鲁棒性文档VLM的设计，属于DIU本题范畴。
Abstract The impressive performance of VLMs is largely measured on benchmarks that fail to capture the complexities of real-world scenarios. Existing datasets for tabular QA, such as WikiTableQuestions and FinQA, are overwhelmingly monolingual (English) and present tables in a digitally perfect, clean format. This creates a significant gap between research and practice. To address this, we present \textbf{MirageTVQA}, a new benchmark designed to evaluate VLMs on these exact dimensions. Featuring nearly 60,000 QA pairs across 24 languages, MirageTVQA challenges models with tables that are not only multilingual but also visually imperfect, incorporating realistic noise to mimic scanned documents. Our evaluation of the leading VLMs reveals two primary failure points: a severe degradation in performance (over 35\% drop for the best models) when faced with visual noise and a consistent English-first bias where reasoning abilities fail to transfer to other languages. MirageTVQA provides a benchmark for measuring and driving progress towards more robust VLM models for table reasoning. The dataset and the code are available at: https://github.com/anshulsc/MirageTVQA.
Authors Anshul Singh, Rohan Chaudhary, Gagneet Singh, Abhay Kumary
Comment Accepted as Spotligh Talk at EurIPS 2025 Workshop on AI For Tabular Data
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
7
Budget-Aware Tool-Use Enables Effective Agent Scaling
Title CN 预算感知的工具使用实现高效的智能体扩展
Keywords 智能体扩展 · 预算感知 · 测试时计算 · 工具调用 · 成本性能权衡
Summary 本文研究如何在显式工具调用预算约束下有效扩展工具增强型智能体的性能。作者发现单纯增加工具调用次数无法提升效果，因智能体缺乏对预算的感知。为此提出Budget Tracker插件和BATS框架，使智能体能根据剩余资源决定是深入挖掘还是转向新路径。通过统一的成本度量指标分析token与工具消耗，实现了更优的代价-性能权衡。这是首个关于预算受限智能体的系统性研究。
Reason 该论文提出了一种在显式工具调用预算下有效扩展工具增强型智能体的方法BATS，通过引入Budget Tracker使智能体具备预算感知能力，并动态调整规划与验证策略。这一方法可迁移到文档图像理解中的长文档多步推理任务中：例如在复杂报表或法律文档的DocVQA任务中，模型需多次调用OCR、定位、检索等工具，受限于计算成本。本方法能帮助DIU系统在有限资源下更高效地分配‘细粒度识别’与‘上下文推理’动作，避免无效步骤，提升整体推理效率与准确性。
Abstract Scaling test-time computation improves performance across different tasks on large language models (LLMs), which has also been extended to tool-augmented agents. For these agents, scaling involves not only "thinking" in tokens but also "acting" via tool calls. The number of tool calls directly bounds the agent's interaction with the external environment. However, we find that simply granting agents a larger tool-call budget fails to improve performance, as they lack "budget awareness" and quickly hit a performance ceiling. To address this, we study how to scale such agents effectively under explicit tool-call budgets, focusing on web search agents. We first introduce the Budget Tracker, a lightweight plug-in that provides the agent with continuous budget awareness, enabling simple yet effective scaling. We further develop BATS (Budget Aware Test-time Scaling), an advanced framework that leverages this awareness to dynamically adapt its planning and verification strategy, deciding whether to "dig deeper" on a promising lead or "pivot" to new paths based on remaining resources. To analyze cost-performance scaling in a controlled manner, we formalize a unified cost metric that jointly accounts for token and tool consumption. We provide the first systematic study on budget-constrained agents, showing that budget-aware methods produce more favorable scaling curves and push the cost-performance Pareto frontier. Our work offers empirical insights toward a more transparent and principled understanding of scaling in tool-augmented agents.
Authors Tengxiao Liu, Zifeng Wang, Jin Miao, I-Hung Hsu, Jun Yan, Jiefeng Chen, Rujun Han, Fangyuan Xu, Yanfei Chen, Ke Jiang, Samira Daruki, Yi Liang, William Yang Wang, Tomas Pfister, Chen-Yu Lee
Categories Artificial Intelligence
7
Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models
Title CN 看见森林与树木：面向长视频多模态语言模型的查询感知型分词器
Keywords 视觉token压缩 · 查询感知 · 长视频理解 · 多模态语言模型 · 动态剪枝
Summary 本文提出QTSplus，一种轻量级但高效的视觉token选择模块，作为视觉编码器与大语言模型之间的信息门控机制。它根据文本查询动态评估并选择最重要的视觉token，并预测实例化的保留数量，结合直通估计器实现端到端训练。同时引入小型重编码器保留绝对时间信息以维持时序结构。在Qwen2.5-VL上集成后，视觉流压缩达89%，端到端延迟降低28%，在八个长视频理解基准上表现接近原模型，且在时序推理任务上显著超越。
Reason 该论文提出QTSplus，一种基于查询动态选择关键视觉token的压缩机制，通过跨注意力评分、自适应保留预算预测和可微分选择策略，在长视频场景下显著降低计算开销并保持时序定位能力。此方法中的**视觉token动态剪枝与查询感知压缩机制**可直接迁移至文档图像理解领域，用于解决高分辨率文档或长文档（如多页PDF）输入时的视觉token爆炸问题，尤其适用于DocVQA中需聚焦局部区域的任务，缓解OCR幻觉带来的噪声干扰。
Abstract Despite the recent advances in the video understanding ability of multimodal large language models (MLLMs), long video understanding remains a challenge. One of the main issues is that the number of vision tokens grows linearly with video length, which causes an explosion in attention cost, memory, and latency. To solve this challenge, we present Query-aware Token Selector (\textbf{QTSplus}), a lightweight yet powerful visual token selection module that serves as an information gate between the vision encoder and LLMs. Given a text query and video tokens, QTSplus dynamically selects the most important visual evidence for the input text query by (i) scoring visual tokens via cross-attention, (ii) \emph{predicting} an instance-specific retention budget based on the complexity of the query, and (iii) \emph{selecting} Top-$n$ tokens with a differentiable straight-through estimator during training and a hard gate at inference. Furthermore, a small re-encoder preserves temporal order using absolute time information, enabling second-level localization while maintaining global coverage. Integrated into Qwen2.5-VL, QTSplus compresses the vision stream by up to \textbf{89\%} and reduces end-to-end latency by \textbf{28\%} on long videos. The evaluation on eight long video understanding benchmarks shows near-parity accuracy overall when compared with the original Qwen models and outperforms the original model by \textbf{+20.5} and \textbf{+5.6} points respectively on TempCompass direction and order accuracies. These results show that QTSplus is an effective, general mechanism for scaling MLLMs to real-world long-video scenarios while preserving task-relevant evidence.
Authors Siyou Li, Huanan Wu, Juexi Shao, Yinghao Ma, Yujian Gan, Yihao Luo, Yuwei Wang, Dong Nie, Lu Wang, Wengqing Wu, Le Zhang, Massimo Poesio, Juntao Yu
Categories Computer Vision and Pattern Recognition
7
SpatialGeo:Boosting Spatial Reasoning in Multimodal LLMs via Geometry-Semantics Fusion
Title CN SpatialGeo：通过几何-语义融合增强多模态大语言模型中的空间推理能力
Keywords 多模态大语言模型 · 空间推理 · 视觉接地 · 几何特征融合 · 视觉编码器
Summary 本文提出SpatialGeo，一种基于几何与语义特征层次融合的新型视觉编码器，旨在增强多模态大语言模型的空间推理能力。通过引入自监督视觉模型提供的几何信息，并设计分层适配器与随机特征丢弃策略，有效提升了空间接地性能，在SpatialRGPT-Bench上超越现有方法至少8.0%，同时推理内存消耗降低约50%。
Reason 该论文提出了一种新的视觉编码器，通过层次化融合几何与语义特征来增强多模态大语言模型（MLLM）的空间感知能力。其核心创新在于弥补了CLIP等主流视觉编码器在实例级语义特征中丢失的空间结构信息，引入来自自监督视觉学习的几何特征，从而提升模型对三维空间布局的理解和定位能力。这一方法对文档图像理解（DIU）具有明确迁移价值：文档中的复杂排版、表格结构、图文位置关系等均依赖精细的空间推理与视觉接地（visual grounding）。例如，在表格识别或表单问答任务中，模型需准确理解‘左上角的日期’、‘第二行第三列的内容’等空间描述，而现有VLM常因空间模糊导致错误。SpatialGeo所提出的几何增强机制可直接用于提升文档场景下的细粒度空间定位能力，缓解因布局复杂引发的幻觉问题。
Abstract Multimodal large language models (MLLMs) have achieved significant progress in image and language tasks due to the strong reasoning capability of large language models (LLMs). Nevertheless, most MLLMs suffer from limited spatial reasoning ability to interpret and infer spatial arrangements in three-dimensional space. In this work, we propose a novel vision encoder based on hierarchical fusion of geometry and semantics features, generating spatial-aware visual embedding and boosting the spatial grounding capability of MLLMs. Specifically, we first unveil that the spatial ambiguity shortcoming stems from the lossy embedding of the vision encoder utilized in most existing MLLMs (e.g., CLIP), restricted to instance-level semantic features. This motivates us to complement CLIP with the geometry features from vision-only self-supervised learning via a hierarchical adapter, enhancing the spatial awareness in the proposed SpatialGeo. The network is efficiently trained using pretrained LLaVA model and optimized with random feature dropping to avoid trivial solutions relying solely on the CLIP encoder. Experimental results show that SpatialGeo improves the accuracy in spatial reasoning tasks, enhancing state-of-the-art models by at least 8.0% in SpatialRGPT-Bench with approximately 50% less memory cost during inference. The source code is available via https://ricky-plus.github.io/SpatialGeoPages/.
Authors Jiajie Guo, Qingpeng Zhu, Jin Zeng, Xiaolong Wu, Changyong He, Weida Wang
Categories Computer Vision and Pattern Recognition
7
ChainV: Atomic Visual Hints Make Multimodal Reasoning Shorter and Better
Title CN ChainV：原子视觉提示让多模态推理更短更好
Keywords 多模态推理 · 视觉提示 · 推理压缩 · 注意力机制 · 去冗余
Summary ChainV是一种提升多模态推理效率与质量的新方法，它通过从视觉输入中动态选取关键patch作为‘原子视觉提示’，引导模型聚焦于关键区域，并基于提示的可靠性自适应控制推理过程。实验表明其在数学密集型视觉推理任务上显著提升了准确率，同时大幅降低延迟和输出长度。
Reason 该论文提出了一种在多模态推理过程中动态引入原子级视觉提示的框架ChainV，通过注意力机制选择最具代表性的视觉patch，并结合一致性评估机制自适应调整推理深度。该方法可有效缩短推理链长度、减少冗余自省，提升推理效率与准确性。对于文档图像理解（DIU）中的长文档多步推理任务（如复杂表格问答或跨区域信息整合），ChainV提供的动态视觉聚焦和推理控制机制具有明确迁移价值——可用于抑制模型在无关键信息区域的过度推理（类似OCR幻觉引发的错误推导），并加速对高分辨率文档的细粒度定位与逻辑推理。
Abstract Recent advances in multimodal reasoning models have demonstrated impressive capabilities across text and vision. However, even leading models exhibit redundant self-reflection when generating lengthy reasoning chains. While training-free CoT compression methods have emerged in the LLMs domain, they rely on static visual references and thus provide limited gains for multimodal reasoning. Therefore, we propose ChainV, a framework that dynamically integrates visual hints into the reasoning process, thereby making multimodal reasoning shorter and better. Specifically, ChainV first performs a coarse visual patch selection based on the previous reasoning step, then refines it by identifying the most representative atomic visual hint according to the averaged attention intensity. Additionally, ChainV introduces a consistency-based evaluation mechanism to assess the reliability of the chosen hint, guiding the model to adaptively adjust its level of self-reflection. Eventually, the pixel coordinates of the selected visual hint and its reliability are incorporated into thinking with a Bernoulli stochastic process. Experiments indicate that our method significantly improves reasoning accuracy and efficiency, especially on math-intensive benchmarks where visual hints are crucial for multi-step symbolic reasoning. For example, ChainV achieves $2.3\%$ improvement on the MathVista within MIMO-VL-RL, while reducing inference latency by $51.4\%$ and shortening output token length by $24.5\%$.
Authors Yuan Zhang, Ming Lu, Junwen Pan, Tao Huang, Kuan Cheng, Qi She, Shanghang Zhang
Comment 16 pages
Categories Computer Vision and Pattern Recognition
7
Draft and Refine with Visual Experts
Title CN 利用视觉专家进行草稿与精炼
Keywords 视觉语言模型 · 幻觉抑制 · 视觉接地 · 代理框架 · 推理时优化
Summary 本文提出Draft and Refine（DnR）框架，通过一个可量化的视觉利用度指标引导大型视觉语言模型在推理阶段逐步优化其响应。该方法利用外部视觉专家提供的定位输出（如边界框或掩码）作为视觉线索，重新渲染图像并反馈给模型，以选择能最大程度提升视觉依赖得分的答案。实验表明该方法能有效降低幻觉、提高准确率，适用于VQA和图像描述等任务。
Reason 该论文提出了一种名为Draft and Refine（DnR）的代理框架，通过量化模型在推理过程中对视觉证据的依赖程度来减少幻觉。其核心是构建一个基于问题的相关性图，并通过相关性引导的概率掩码测量视觉信息的使用。这一机制可直接迁移到文档图像理解（DIU）领域，用于缓解OCR幻觉或VLM在密集文本和复杂排版下的视觉错位问题。例如，在DocVQA任务中，当模型因语言先验错误回答时，可通过引入外部视觉专家（如文本检测器或布局分析模块）生成的视觉线索（如文本框或区域掩码），重新引导VLM聚焦于真实视觉内容，从而提升答案的准确性与可解释性。这是一种无需微调即可增强视觉接地的方法，具有较强的即插即用潜力。
Abstract While recent Large Vision-Language Models (LVLMs) exhibit strong multimodal reasoning abilities, they often produce ungrounded or hallucinated responses because they rely too heavily on linguistic priors instead of visual evidence. This limitation highlights the absence of a quantitative measure of how much these models actually use visual information during reasoning. We propose Draft and Refine (DnR), an agent framework driven by a question-conditioned utilization metric. The metric quantifies the model's reliance on visual evidence by first constructing a query-conditioned relevance map to localize question-specific cues and then measuring dependence through relevance-guided probabilistic masking. Guided by this metric, the DnR agent refines its initial draft using targeted feedback from external visual experts. Each expert's output (such as boxes or masks) is rendered as visual cues on the image, and the model is re-queried to select the response that yields the largest improvement in utilization. This process strengthens visual grounding without retraining or architectural changes. Experiments across VQA and captioning benchmarks show consistent accuracy gains and reduced hallucination, demonstrating that measuring visual utilization provides a principled path toward more interpretable and evidence-driven multimodal agent systems. Code is available at https://github.com/EavnJeong/Draft-and-Refine-with-Visual-Experts.
Authors Sungheon Jeong, Ryozo Masukawa, Jihong Park, Sanggeon Yun, Wenjun Huang, Hanning Chen, Mahdi Imani, Mohsen Imani
Categories Computer Vision and Pattern Recognition
6
Escaping the Verifier: Learning to Reason via Demonstrations
Title CN 逃离验证器：通过示范学习推理
Keywords 推理训练 · 逆强化学习 · 专家示范 · 对抗学习 · Verifier-free
Summary 本文提出RARO，一种无需任务特定验证器的推理训练框架，通过逆强化学习和对抗机制让大模型从专家示范中学习推理能力。策略网络模仿专家输出，判别网络则相对比较策略与专家的回答差异，二者通过强化学习联合优化。实验表明该方法在多个非可微分任务上显著优于基线，并展现出良好的扩展性。
Reason 该论文提出RARO方法，利用逆强化学习从专家示范中学习推理能力，适用于缺乏显式验证器的复杂推理任务。虽然其评估任务（如数学推导、诗歌生成）非文档理解场景，但其核心思想——通过对抗性判别机制从高质量示范中蒸馏推理行为——可迁移至DIU中的长文档多步推理或表格逻辑推断等任务。例如，在DocVQA中，当答案无法被简单规则验证时，可用类似框架让模型模仿专家标注路径进行推理。然而，该方法在视觉-语言对齐或细粒度定位方面无直接改进，迁移需额外设计。
Abstract Training Large Language Models (LLMs) to reason often relies on Reinforcement Learning (RL) with task-specific verifiers. However, many real-world reasoning-intensive tasks lack verifiers, despite offering abundant expert demonstrations that remain under-utilized for reasoning-focused training. We introduce RARO (Relativistic Adversarial Reasoning Optimization) that learns strong reasoning capabilities from only expert demonstrations via Inverse Reinforcement Learning. Our method sets up an adversarial interaction between a policy (generator) and a relativistic critic (discriminator): the policy learns to mimic expert answers, while the critic learns to compare and distinguish between policy and expert answers. Our method trains both the policy and the critic jointly and continuously via RL, and we identify the key stabilization techniques required for robust learning. Empirically, RARO significantly outperforms strong verifier-free baselines on all of our evaluation tasks -- Countdown, DeepMath, and Poetry Writing -- and enjoys the same robust scaling trends as RL on verifiable tasks. These results demonstrate that our method effectively elicits strong reasoning performance from expert demonstrations alone, enabling robust reasoning learning even when task-specific verifiers are unavailable.
Authors Locke Cai, Ivan Provilkov
Categories Artificial Intelligence
6
Pessimistic Verification for Open Ended Math Questions
Title CN 面向开放性数学问题的悲观验证
Keywords 悲观验证 · 错误检测 · 语言模型可靠性 · 数学推理 · 测试时扩展
Summary 本文提出悲观验证方法，通过多路并行验证机制提升开放性数学问题的答案校验能力。该方法简单高效，在多个数学基准上显著提升性能且计算开销低。研究表明，强模型中的多数误报实为数据标注错误，说明该方法表现可能被低估。自我验证机制有助于提升语言模型在长程推理任务中的可靠性和准确性。
Reason 该论文提出了一种名为‘悲观验证’的方法，通过构建多个并行验证路径来检测数学证明中的错误，只要任一路径发现错误即判定整体为错。这种方法在数学问题上提升了语言模型输出的可靠性，并展示了在长视野任务中的潜力。虽然其直接应用场景是数学推理，但其核心思想——多路径并行验证以增强错误检测能力——可迁移至文档图像理解中的复杂逻辑推理与幻觉抑制任务。例如，在DocVQA中，当模型对表格或密集文本进行推理时，可通过设计多种验证路径（如数值一致性、上下文语义匹配、格式约束）联合判断答案正确性，从而缓解因OCR误差或上下文误解导致的幻觉问题。然而，该方法需依赖高质量的验证信号设计，在文档领域缺乏标准化监督的情况下迁移难度较高，故评分中等。
Abstract The key limitation of the verification performance lies in the ability of error detection. With this intuition we designed several variants of pessimistic verification, which are simple workflows that could significantly improve the verification of open-ended math questions. In pessimistic verification we construct multiple parallel verifications for the same proof, and the proof is deemed incorrect if any one of them reports an error. This simple technique significantly improves the performance across many math verification benchmarks without incurring substantial computational resources. Its token efficiency even surpassed extended long-CoT in test-time scaling. Our case studies further indicate that the majority of false negatives in stronger models are actually caused by annotation errors in the original dataset, so our method's performance is in fact underestimated. Self-verification for mathematical problems can effectively improve the reliability and performance of language model outputs, and it also plays a critical role in enabling long-horizon mathematical tasks. We believe that research on pessimistic verification will help enhance the mathematical capabilities of language models across a wide range of tasks.
Authors Yanxing Huang, Zihan Tang, Zejin Lin, Peng Li, Yang Liu
Categories Artificial Intelligence
6
Frequency-Aware Token Reduction for Efficient Vision Transformer Neurips 2025
Title CN 面向高效视觉Transformer的频率感知令牌缩减
Keywords 视觉Transformer · 令牌缩减 · 频率分析 · 计算效率 · 秩坍缩
Summary 本文针对Vision Transformer中token计算复杂度高的问题，提出一种频率感知的token缩减策略。该方法将token划分为高频与低频部分，选择性保留高频token，并将低频token聚合为一个紧凑的直流token，从而在降低计算开销的同时缓解了rank collapsing和过平滑现象。实验表明该方法在保持性能的前提下显著提升了效率。
Reason 该论文提出了一种频率感知的token reduction方法，通过分离高频和低频token并保留关键高频信息、聚合低频信息来缓解rank collapsing和过平滑问题。虽然其目标是通用Vision Transformer的效率优化，但该方法对文档图像理解（DIU）中的高分辨率输入处理具有潜在迁移价值：文档图像通常包含大量细粒度文本结构（如小字、表格线），属于高频细节，若能在VLM中有效保留这些高频token，可能提升OCR-free模型在复杂排版下的识别能力。然而，本文未直接针对多模态或语言-视觉对齐设计，也未验证在DocVQA等任务上的效果，迁移路径需额外适配，因此为弱相关。
Abstract Vision Transformers have demonstrated exceptional performance across various computer vision tasks, yet their quadratic computational complexity concerning token length remains a significant challenge. To address this, token reduction methods have been widely explored. However, existing approaches often overlook the frequency characteristics of self-attention, such as rank collapsing and over-smoothing phenomenon. In this paper, we propose a frequency-aware token reduction strategy that improves computational efficiency while preserving performance by mitigating rank collapsing. Our method partitions tokens into high-frequency tokens and low-frequency tokens. high-frequency tokens are selectively preserved, while low-frequency tokens are aggregated into a compact direct current token to retain essential low-frequency components. Through extensive experiments and analysis, we demonstrate that our approach significantly improves accuracy while reducing computational overhead and mitigating rank collapsing and over smoothing. Furthermore, we analyze the previous methods, shedding light on their implicit frequency characteristics and limitations.
Authors Dong-Jae Lee, Jiwan Hur, Jaehyun Choi, Jaemyung Yu, Junmo Kim
Comment Neurips 2025
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
6
Do Reasoning Vision-Language Models Inversely Scale in Test-Time Compute? A Distractor-centric Empirical Analysis
Title CN 推理型视觉语言模型在测试时计算中是否呈现逆向缩放？一种以干扰项为中心的实证分析
Keywords 视觉语言模型 · 测试时计算 · 干扰项 · 逆向缩放 · 推理偏见
Summary 本文构建了一个名为Idis的数据集，系统性地引入语义、数值和空间维度的视觉干扰项，用于研究视觉语言模型在测试时计算扩展下的表现。实验发现，与文本干扰不同，视觉干扰会降低准确率但不增加推理长度，表明模型更易被误导而非陷入冗长无效推理。作者通过分析推理过程中的属性计数揭示了干扰机制，并提出一种提示策略来缓解偏见驱动的预测。
Reason 该论文研究了视觉语言模型在面对图像中干扰信息（distractors）时，随着测试时计算量增加（如CoT、更多推理步）性能反而下降的现象。虽然其核心关注的是多模态偏见与推理干扰问题，但其实验设计和对推理轨迹中属性计数的分析方法，可能为文档图像理解中的OCR幻觉或无关元素干扰（如水印、背景图案）导致的错误推理提供诊断工具。然而，该工作并未直接针对文档场景设计，迁移需额外验证，且未涉及文档特有的高分辨率、密集文本等挑战。
Abstract How does irrelevant information (i.e., distractors) affect test-time scaling in vision-language models (VLMs)? Prior studies on language models have reported an inverse scaling effect, where textual distractors lead to longer but less effective reasoning. To investigate whether similar phenomena occur in multimodal settings, we introduce Idis (Images with distractors), a visual question-answering dataset that systematically varies distractors along semantic, numerical, and spatial dimensions. Our analyses reveal that visual distractors differ fundamentally from textual ones: although inverse scaling persists, adding visual distractors reduces accuracy without increasing reasoning length. We further show that tracking attribute counts within reasoning traces provides key insights into how distractors, reasoning length, and accuracy interact. Finally, we demonstrate that these trends extend to established visual bias benchmarks such as Waterbirds, and we propose a simple prompting strategy to mitigate bias-driven predictions in reasoning models.
Authors Jiyun Bae, Hyunjong Ok, Sangwoo Mo, Jaeho Lee
Comment preprint
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
6
Characterizing Pattern Matching and Its Limits on Compositional Task Structures
Title CN 刻画模式匹配及其在组合任务结构中的局限性
Keywords 模式匹配 · 组合泛化 · 路径歧义 · 数据复杂度 · 推理机制
Summary 本文提出一种将模式匹配形式化为函数等价的方法，用于精确研究LLM在组合任务中的泛化机制。通过控制实验和理论证明，作者发现模型能否成功进行模式匹配取决于观察到该等价性的上下文数量，并给出了两跳结构学习的样本复杂度边界。研究还指出路径歧义是阻碍正确推理的结构性瓶颈，且标准Chain-of-Thought无法解决此问题。这些结论为理解LLM的泛化边界提供了可验证的理论框架。
Reason 该论文从理论角度形式化了LLM中的模式匹配行为，并分析其在组合任务中的泛化能力与局限，特别是揭示了路径歧义（path ambiguity）对模型推理的负面影响。虽然研究不直接针对文档图像理解（DIU），但其对‘多路径信息整合失败’的诊断可迁移至DocVQA中的复杂逻辑推理场景，例如当模型需通过多个文本区域联合推导答案时出现的推理断裂问题。此外，其关于上下文数量影响模式匹配成功的发现，可能启发DIU中更有效的CoT设计或训练数据构造策略。
Abstract Despite impressive capabilities, LLMs' successes often rely on pattern-matching behaviors, yet these are also linked to OOD generalization failures in compositional tasks. However, behavioral studies commonly employ task setups that allow multiple generalization sources (e.g., algebraic invariances, structural repetition), obscuring a precise and testable account of how well LLMs perform generalization through pattern matching and their limitations. To address this ambiguity, we first formalize pattern matching as functional equivalence, i.e., identifying pairs of subsequences of inputs that consistently lead to identical results when the rest of the input is held constant. Then, we systematically study how decoder-only Transformer and Mamba behave in controlled tasks with compositional structures that isolate this mechanism. Our formalism yields predictive and quantitative insights: (1) Instance-wise success of pattern matching is well predicted by the number of contexts witnessing the relevant functional equivalence. (2) We prove a tight sample complexity bound of learning a two-hop structure by identifying the exponent of the data scaling law for perfect in-domain generalization. Our empirical results align with the theoretical prediction, under 20x parameter scaling and across architectures. (3) Path ambiguity is a structural barrier: when a variable influences the output via multiple paths, models fail to form unified intermediate state representations, impairing accuracy and interpretability. (4) Chain-of-Thought reduces data requirements yet does not resolve path ambiguity. Hence, we provide a predictive, falsifiable boundary for pattern matching and a foundational diagnostic for disentangling mixed generalization mechanisms.
Authors Hoyeon Chang, Jinho Park, Hanseul Cho, Sohee Yang, Miyoung Ko, Hyeonbin Hwang, Seungpil Won, Dohaeng Lee, Youbin Ahn, Minjoon Seo
Categories Artificial Intelligence, Computation and Language
6
Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning
Title CN Agent0-VL：探索用于工具集成视觉语言推理的自进化智能体
Keywords 视觉语言智能体 · 自演化推理 · 工具集成 · 自我验证 · 去幻觉
Summary Agent0-VL是一种无需人工标注或外部奖励模型的自进化视觉语言智能体，通过将工具集成到推理与自我验证中，构建了一个包含Solver和Verifier双角色的自演化循环。该框架利用工具提供的证据进行细粒度批评与自奖励，在几何问题求解和视觉科学分析任务上实现了比基础模型高出12.5%的表现。其核心思想是让模型通过可信赖的工具输出来校准自身的推理过程，从而实现稳定自我改进。
Reason 该论文提出一种自进化的视觉语言智能体Agent0-VL，通过将工具使用整合到推理、自我评估与修复中，实现无外部奖励的持续自我提升。其核心创新在于‘工具增强的自我验证’机制，能基于工具输出进行细粒度反馈和自奖励，缓解纯文本自评带来的幻觉问题。该方法对DIU领域具有潜在迁移价值：在处理长文档多步推理时（如跨页信息整合或复杂表格逻辑推导），可引入类似Self-Evolving Reasoning Cycle机制，利用OCR或结构解析工具作为‘外部工具’，在推理过程中动态验证中间步骤正确性，从而减少VLM在密集文本或复杂排版下的幻觉输出。然而，当前实验集中在几何题和科学图像分析，未涉及真实文档场景，且未解决高分辨率、细粒度定位等文档特有挑战，迁移路径需进一步设计。
Abstract Vision-language agents have achieved remarkable progress in a variety of multimodal reasoning tasks; however, their learning remains constrained by the limitations of human-annotated supervision. Recent self-rewarding approaches attempt to overcome this constraint by allowing models to act as their own critics or reward providers. Yet, purely text-based self-evaluation struggles to verify complex visual reasoning steps and often suffers from evaluation hallucinations. To address these challenges, inspired by recent advances in tool-integrated reasoning, we propose Agent0-VL, a self-evolving vision-language agent that achieves continual improvement with tool-integrated reasoning. Agent0-VL incorporates tool usage not only into reasoning but also into self-evaluation and self-repair, enabling the model to introspect, verify, and refine its reasoning through evidence-grounded analysis. It unifies two synergistic roles within a single LVLM: a Solver that performs multi-turn tool-integrated reasoning, and a Verifier that generates structured feedback and fine-grained self-rewards through tool-grounded critique. These roles interact through a Self-Evolving Reasoning Cycle, where tool-based verification and reinforcement learning jointly align the reasoning and evaluation distributions for stable self-improvement. Through this zero-external-reward evolution, Agent0-VL aligns its reasoning and verification behaviors without any human annotation or external reward models, achieving continual self-improvement. Experiments on geometric problem solving and visual scientific analysis show that Agent0-VL achieves an 12.5% improvement over the base model. Our code is available at https://github.com/aiming-lab/Agent0.
Authors Jiaqi Liu, Kaiwen Xiong, Peng Xia, Yiyang Zhou, Haonian Ji, Lu Feng, Siwei Han, Mingyu Ding, Huaxiu Yao
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
6
UniGame: Turning a Unified Multimodal Model Into Its Own Adversary
Title CN UniGame：将统一多模态模型转变为自身的对抗者
Keywords 多模态一致性 · 自对抗训练 · 表征对齐 · 后训练优化 · 幻觉缓解
Summary UniGame提出了一种自对抗的后训练方法，通过让统一多模态模型的生成分支去攻击其理解分支，暴露并修复二者之间的表征不一致性。该方法显著提升了模型在跨模态一致性、分布外泛化和对抗鲁棒性方面的表现，且几乎不增加参数量。尽管实验集中在通用多模态任务上，但其增强多模态对齐的思想对解决文档理解中的幻觉问题具有启发意义。
Reason 该论文提出一种自对抗的后训练框架UniGame，通过在共享token接口引入轻量级扰动器，使生成分支主动挑战理解分支中的脆弱部分，从而提升多模态模型的一致性与鲁棒性。虽然其主要验证任务并非文档图像理解（DIU），但其核心思想——利用模型自身进行对抗式一致性优化——具有潜在迁移价值。例如，在DocVQA中，OCR或视觉理解模块常因与LLM生成分支表征不一致而引发幻觉问题，UniGame的机制可被借鉴用于缓解此类跨模态不匹配，尤其适用于高幻觉风险的复杂排版文档理解场景。
Abstract Unified Multimodal Models (UMMs) have shown impressive performance in both understanding and generation with a single architecture. However, UMMs still exhibit a fundamental inconsistency: understanding favors compact embeddings, whereas generation favors reconstruction-rich representations. This structural trade-off produces misaligned decision boundaries, degraded cross-modal coherence, and heightened vulnerability under distributional and adversarial shifts. In this paper, we present UniGame, a self-adversarial post-training framework that directly targets the inconsistencies. By applying a lightweight perturber at the shared token interface, UniGame enables the generation branch to actively seek and challenge fragile understanding, turning the model itself into its own adversary. Experiments demonstrate that UniGame significantly improves the consistency (+4.6%). Moreover, it also achieves substantial improvements in understanding (+3.6%), generation (+0.02), out-of-distribution and adversarial robustness (+4.8% and +6.2% on NaturalBench and AdVQA). The framework is architecture-agnostic, introduces less than 1% additional parameters, and is complementary to existing post-training methods. These results position adversarial self-play as a general and effective principle for enhancing the coherence, stability, and unified competence of future multimodal foundation models. The official code is available at: https://github.com/AIFrontierLab/UniGame
Authors Zhaolong Su, Wang Lu, Hao Chen, Sharon Li, Jindong Wang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
6
Latent Collaboration in Multi-Agent Systems
Title CN 多智能体系统中的潜在协作
Keywords 多智能体系统 · 潜在空间协作 · 推理效率 · 无训练框架 · 系统级智能
Summary 本文提出LatentMAS，一种无需训练的多智能体系统框架，允许大语言模型在潜在空间中进行隐式协作。每个智能体通过最后一层隐藏状态自回归生成‘潜在思维’，并通过共享潜在工作记忆实现高效、无损的信息传递。理论分析表明该方法具有更高表达能力和更低复杂度，实验显示其在多个推理任务上优于传统文本中介的多智能体系统，同时减少70%以上输出token并提速4倍。
Reason 该论文提出在连续潜在空间中实现LLM智能体之间的直接协作，通过共享潜在工作记忆实现无损信息交换，并显著提升推理效率与性能。虽然其评估任务（数学推理、常识理解、代码生成）与文档图像理解有一定关联，但该方法的核心是智能体间通信机制的改进，属于上游多智能体协同的架构创新。理论上，该潜在协作机制可被迁移用于长文档的多段落协同理解，例如将不同页面或区域的视觉-语言表征作为智能体在潜在空间中融合，从而缓解长距离依赖和信息丢失问题。然而，目前未涉及视觉输入或文档结构建模，迁移到DIU场景仍需大量适配工作，实际可行性尚不明确。
Abstract Multi-agent systems (MAS) extend large language models (LLMs) from independent single-model reasoning to coordinative system-level intelligence. While existing LLM agents depend on text-based mediation for reasoning and communication, we take a step forward by enabling models to collaborate directly within the continuous latent space. We introduce LatentMAS, an end-to-end training-free framework that enables pure latent collaboration among LLM agents. In LatentMAS, each agent first performs auto-regressive latent thoughts generation through last-layer hidden embeddings. A shared latent working memory then preserves and transfers each agent's internal representations, ensuring lossless information exchange. We provide theoretical analyses establishing that LatentMAS attains higher expressiveness and lossless information preservation with substantially lower complexity than vanilla text-based MAS. In addition, empirical evaluations across 9 comprehensive benchmarks spanning math and science reasoning, commonsense understanding, and code generation show that LatentMAS consistently outperforms strong single-model and text-based MAS baselines, achieving up to 14.6% higher accuracy, reducing output token usage by 70.8%-83.7%, and providing 4x-4.3x faster end-to-end inference. These results demonstrate that our new latent collaboration framework enhances system-level reasoning quality while offering substantial efficiency gains without any additional training. Code and data are fully open-sourced at https://github.com/Gen-Verse/LatentMAS.
Authors Jiaru Zou, Xiyuan Yang, Ruizhong Qiu, Gaotang Li, Katherine Tieu, Pan Lu, Ke Shen, Hanghang Tong, Yejin Choi, Jingrui He, James Zou, Mengdi Wang, Ling Yang
Comment Project: https://github.com/Gen-Verse/LatentMAS
Categories Artificial Intelligence, Computation and Language
6
DRAFT-RL: Multi-Agent Chain-of-Draft Reasoning for Reinforcement Learning-Enhanced LLMs
Title CN DRAFT-RL：面向强化学习增强大语言模型的多智能体草稿链推理
Keywords 多智能体系统 · 强化学习 · 链式草稿 · 大语言模型 · 推理优化
Summary DRAFT-RL是一种基于强化学习的多智能体推理框架，每个智能体为同一问题生成多个草稿，由同伴智能体和奖励模型评估，选出最优路径用于后续策略更新。该方法实现了显式的多路径探索与协同反思，在代码、数学和问答任务上表现优于现有方法。
Reason 该工作提出了一种多智能体框架DRAFT-RL，通过生成多个推理草稿并结合强化学习进行路径选择与策略优化，提升了复杂推理任务中的准确性和收敛速度。虽然其评估任务（代码生成、符号数学、知识密集型问答）部分与文档理解存在交集，但该方法本身属于通用多智能体推理架构创新。其核心机制——多路径草稿生成与peer评审机制——理论上可迁移至长文档VQA或多步信息抽取任务中，用于缓解推理迷失和提升逻辑连贯性。然而，论文未涉及视觉输入或文档结构建模，对OCR幻觉、细粒度定位等DIU核心问题无直接解决方案，迁移需大量适配工作，故为弱相关。
Abstract Large Language Models (LLMs) have shown impressive capabilities in multi-step reasoning and problem-solving.Recent works introduce multi-agent reflection frameworks where multiple LLM agents critique and refine each other's outputs using reinforcement learning (RL). However, these approaches often rely on single-shot responses and lack structural diversity in reasoning exploration. In this paper, we propose DRAFT-RL, a novel framework that integrates Chain-of-Draft (CoD) reasoning into multi-agent RL training. Instead of generating single responses, each agent produces multiple drafts per query, which are then evaluated by peer agents and a learned reward model to identify the most promising trajectory. These selected drafts are used to refine future reasoning strategies through actor-critic learning.DRAFT-RL enables explicit multi-path exploration, peer-guided reflection, and reward-aligned selection, resulting in more robust and interpretable LLM agent behavior. We evaluate our method on complex reasoning tasks including code synthesis, symbolic math, and knowledge-intensive QA,demonstrating that DRAFT-RL outperforms existing reflective and RL-based agents by significant margins in both accuracy and convergence speed
Authors Yuanhao Li, Mingshan Liu, Hongbo Wang, Yiding Zhang, Yifei Ma, Wei Tan
Categories Artificial Intelligence
6
DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization NeurIPS 2025
Title CN DisCO：通过判别式约束优化增强大推理模型
Keywords 强化学习 · 推理模型 · 判别式优化 · 去偏差 · 训练稳定性
Summary 本文分析了GRPO在二元奖励设置下的局限性，指出其存在题目难度偏差问题，并提出新的Discriminative Constrained Optimization（DisCO）框架。DisCO采用判别式目标替代组相对目标，使用无裁剪的RL代理函数作为打分函数，并通过约束优化控制KL散度。实验表明，DisCO在消除难度偏差、稳定训练动态和处理数据不平衡方面优于GRPO及其变体，在六个数学推理基准上显著提升性能。
Reason 该论文提出了一种用于强化大推理模型（LRM）的判别式约束优化框架DisCO，主要针对数学推理任务中的问题难度偏差和熵不稳定问题。虽然其核心动机源于推理模型训练的稳定性与公平性，但该方法属于通用推理优化范式，理论上可迁移至文档图像理解中的长文档多步推理或VQA去幻觉场景——例如，在DocVQA中当存在多个候选答案路径时，DisCO的判别式打分机制可帮助抑制OCR幻觉导致的错误推理分支，提升答案选择的鲁棒性。然而，论文未涉及视觉输入或多模态对齐，且应用场景完全基于纯文本数学推理，对DIU的直接迁移路径不明确，因此仅为弱相关。
Abstract The recent success and openness of DeepSeek-R1 have brought widespread attention to Group Relative Policy Optimization (GRPO) as a reinforcement learning method for large reasoning models (LRMs). In this work, we analyze the GRPO objective under a binary reward setting and reveal an inherent limitation of question-level difficulty bias. We also identify a connection between GRPO and traditional discriminative methods in supervised learning. Motivated by these insights, we introduce a new Discriminative Constrained Optimization (DisCO) framework for reinforcing LRMs, grounded in the principle of discriminative learning. The main differences between DisCO and GRPO and its recent variants are: (1) it replaces the group relative objective with a discriminative objective defined by a scoring function; (2) it abandons clipping-based surrogates in favor of non-clipping RL surrogate objectives used as scoring functions; (3) it employs a simple yet effective constrained optimization approach to enforce the KL divergence constraint. As a result, DisCO offers notable advantages over GRPO and its variants: (i) it completely eliminates difficulty bias by adopting discriminative objectives; (ii) it addresses the entropy instability in GRPO and its variants through the use of non-clipping scoring functions and a constrained optimization approach, yielding long and stable training dynamics; (iii) it allows the incorporation of advanced discriminative learning techniques to address data imbalance, where a significant number of questions have more negative than positive generated answers during training. Our experiments on enhancing the mathematical reasoning capabilities of SFT-finetuned models show that DisCO significantly outperforms GRPO and its improved variants such as DAPO, achieving average gains of 7\% over GRPO and 6\% over DAPO across six benchmark tasks for an 1.5B model.
Authors Gang Li, Ming Lin, Tomer Galanti, Zhengzhong Tu, Tianbao Yang
Comment Accepted to NeurIPS 2025
Categories Artificial Intelligence
6
Scalable and Accurate Graph Reasoning with LLM-based Multi-Agents AAAI 2026 Workshop WMAC
Title CN 基于大语言模型多智能体的可扩展且准确的图推理
Keywords 图推理 · 多智能体系统 · 大语言模型 · 分布式推理 · 零样本学习
Summary 本文提出了GraphAgent-Reasoner，一个无需微调、基于多智能体协作的大语言模型框架，用于解决图结构上的复杂推理任务。受分布式图计算启发，该框架将图问题分解为以节点为中心的子任务，分配给多个代理并行处理，并通过协作机制汇总结果。这种方法显著降低了单个LLM处理长文本和复杂结构的压力，提升了推理精度，在GraphInstruct数据集上实现了接近完美的表现，并能扩展到超过1000个节点的大规模图。此外，它还展示了在网页重要性分析等实际应用中的潜力。
Reason 该论文提出了一种基于多智能体协作的图推理框架GraphAgent-Reasoner，通过将图问题分解为节点级任务并由多个LLM代理协同求解，提升了复杂结构下的推理准确性与可扩展性。虽然其核心任务是图推理，但其‘将复杂结构分解为局部子任务并通过多代理协作降低单个模型负担’的设计思想，可能迁移到长文档或多页表单的文档理解中——例如，在处理跨页表格或长篇合同文档时，可用类似机制将文档切分为区域或段落，由不同代理分别处理后再整合，缓解VLM在长上下文中的迷失问题。然而，该方法目前未直接涉及视觉输入或文档模态，迁移需较大工程适配，故为弱相关。
Abstract Recent research has explored the use of Large Language Models (LLMs) for tackling complex graph reasoning tasks. However, due to the intricacies of graph structures and the inherent limitations of LLMs in handling long text, current approaches often fail to deliver satisfactory accuracy, even on small-scale graphs and simple tasks. To address these challenges, we introduce GraphAgent-Reasoner, a fine-tuning-free framework that utilizes a multi-agent collaboration strategy for explicit and precise graph reasoning. Inspired by distributed graph computation theory, our framework decomposes graph problems into smaller, node-centric tasks that are distributed among multiple agents. The agents collaborate to solve the overall problem, significantly reducing the amount of information and complexity handled by a single LLM, thus enhancing the accuracy of graph reasoning. By simply increasing the number of agents, GraphAgent-Reasoner can efficiently scale to accommodate larger graphs with over 1,000 nodes. Evaluated on the GraphInstruct dataset, our framework demonstrates near-perfect accuracy on polynomial-time graph reasoning tasks, significantly outperforming the best available models, both closed-source and fine-tuned open-source variants. Our framework also demonstrates the capability to handle real-world graph reasoning applications such as webpage importance analysis.
Authors Yuwei Hu, Runlin Lei, Xinyi Huang, Zhewei Wei, Yongchao Liu
Comment Accepted by AAAI 2026 Workshop WMAC
Categories Artificial Intelligence
6
Improving Language Agents through BREW
Title CN 通过BREW改进语言智能体
Keywords 智能体优化 · 知识库构建 · 经验学习 · 状态空间搜索 · 可解释AI
Summary 本文提出BREW框架，通过构建和持续优化智能体的经验知识库（KB）来替代传统的模型参数微调方法，从而提升智能体在复杂环境下的任务精度与执行效率。该方法引入记忆分块、状态空间搜索和行为评分机制，在OSWorld、τ²Bench和SpreadsheetBench上实现了10-20%的任务精度提升，同时减少API调用次数。与传统将记忆作为静态上下文的方法不同，BREW将KB视为可主动塑造行为的动态控制层，增强了智能体的可解释性和可扩展性。
Reason 该论文提出BREW框架，通过构建和迭代优化基于经验学习的知识库（KB）来提升LLM智能体在多步任务中的性能。其核心贡献在于将智能体的记忆模块化、可解释化，并通过任务评分器与行为规范实现可控优化。虽然工作本身应用于操作系统、电子表格等场景，属于通用Agent架构研究，但其‘结构化经验记忆+状态空间搜索+行为可解释调控’的思路对文档图像理解中的长文档多步推理、跨页面信息检索具有潜在迁移价值。例如，在处理需多跳推理的复杂表单或合同文档时，BREW的KB机制可用于积累中间语义状态，避免上下文丢失；其对自然语言噪声的鲁棒性设计也可缓解OCR后文本不准确带来的幻觉问题。
Abstract Large Language Model (LLM)-based agents are increasingly applied to tasks requiring structured reasoning, tool use, and environmental adaptation, such as data manipulation, multistep planning, and computer-use automation. However, despite their versatility, current training paradigms for model weight optimization methods, like PPO and GRPO, remain relatively impractical with their high computational overhead for rollout convergence. In addition, the resulting agent policies are difficult to interpret, adapt, or incrementally improve. To address this, we investigate creating and refining structured memory of experiential learning of an agent from its environment as an alternative route to agent optimization. We introduce BREW (Bootstrapping expeRientially-learned Environmental knoWledge), a framework for agent optimization for downstream tasks via KB construction and refinement. In our formulation, we introduce an effective method for partitioning agent memory for more efficient retrieval and refinement. BREW uses task graders and behavior rubrics to learn insights while leveraging state-space search for ensuring robustness from the noise and non-specificity in natural language. Empirical results on real world, domain-grounded benchmarks -- OSWorld, $τ^2$Bench, and SpreadsheetBench -- show BREW achieves $10-20\%$ improvement in task precision, $10-15\%$ reduction in API/tool calls leading to faster execution time, all while maintaining computational efficiency on par with base models. Unlike prior work where memory is treated as static context, we establish the KB as a modular and controllable substrate for agent optimization -- an explicit lever for shaping behavior in a transparent, interpretable, and extensible manner.
Authors Shashank Kirtania, Param Biyani, Priyanshu Gupta, Yasharth Bajpai, Roshni Iyer, Sumit Gulwani, Gustavo Soares
Categories Artificial Intelligence
6
Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design
Title CN 通过基于推测的算法-系统协同设计降低LLM搜索代理的延迟
Keywords LLM搜索代理 · 推测执行 · 延迟优化 · 算法-系统协同设计 · 多步推理
Summary 本文提出SPAgent，一种基于推测的算法-系统协同设计框架，用于加速LLM驱动的搜索代理。该框架引入两阶段自适应推测机制，在安全情况下省略验证步骤，并通过两级调度器动态管理推测请求以避免资源浪费。实验表明，SPAgent在保持甚至提升准确率的同时，实现了最高1.65倍的端到端加速效果，有助于多步搜索代理的实际部署。
Reason 该论文提出SPAgent框架，通过推测机制优化LLM搜索代理的推理效率，核心贡献在于两阶段自适应推测算法和两级调度器设计。虽然其应用场景是通用搜索代理，但其算法-系统协同优化思路对DIU领域中长文档多步推理任务具有潜在迁移价值：例如在处理需跨页检索信息的DocVQA任务时，可利用推测机制跳过部分序列化推理步骤以加速推理；同时系统级调度策略也可用于控制高分辨率文档输入下的计算开销。然而，该方法未直接针对文档特有的结构建模或视觉-文本对齐问题进行改进，迁移需额外适配。
Abstract LLM-based search agents achieve strong performance but suffer from severe latency, as each step requires serialized LLM reasoning followed by action of tool execution. We revisit this bottleneck through the lens of speculation. While traditional predict-verify speculation paradigm can break serial execution, its benefit remains limited, as it retains the full original workload and adds extra inference overhead. We observe that early agent steps often involve simple evidence-gathering, where correct actions can often be predicted without full reasoning. Building on these observations, we present SPAgent, an algorithm-system co-design framework that expands the role of speculation in search agents to reduce latency. Algorithmically, SPAgent introduces a two-phase adaptive speculation mechanism that selectively omits verification when safe. System-wise, a two-level scheduler regulates speculative requests based on engine load to ensure speculation remains beneficial. We implement SPAgent in real-world systems. Across extensive experimental settings, SPAgent achieves up to $1.65\times$ end-to-end speedup while maintaining same or even achieving higher accuracy, enabling practical deployment of multi-step search agents.
Authors Zixiao Huang, Wen Zeng, Tianyu Fu, Tengxuan Liu, Yizhou Sun, Ke Hong, Xinhao Yang, Chengchun Liu, Yan Li, Quanlu Zhang, Guohao Dai, Zhenhua Zhu, Yu Wang
Categories Artificial Intelligence
6
Metis-HOME: Hybrid Optimized Mixture-of-Experts for Multimodal Reasoning
Title CN Metis-HOME：面向多模态推理的混合优化专家混合框架
Keywords 多模态推理 · 专家混合模型 · 动态路由 · 高效推理 · 思维与直觉平衡
Summary Metis-HOME提出一种新型混合专家架构，将多模态大模型划分为专用于复杂推理的‘思考分支’和适用于快速直推的‘非思考分支’，并通过轻量级路由器动态分配任务。基于Qwen2.5-VL-7B实现后，模型在保持甚至提升一般VQA与OCR能力的同时，显著增强复杂多步推理性能，解决了当前推理专用模型普遍存在的泛化退化问题。
Reason 该论文提出一种混合专家架构（MoE），通过将模型分为‘思考’与‘非思考’两个分支，动态路由复杂推理与简单任务，提升效率与泛化能力。虽然其动机源于多模态推理中的计算冗余问题，但该方法对文档图像理解（DIU）具有潜在迁移价值：在DocVQA中，可让‘思考分支’处理需多步逻辑推理的复杂问题（如跨表格推导），而‘非思考分支’快速响应简单事实查询（如OCR内容提取），从而提升长文档问答系统的整体效率与稳定性。然而，该工作本身未针对文档结构、排版或视觉压缩等DIU核心挑战进行设计，属于通用多模态架构优化，迁移需额外适配。
Abstract Inspired by recent advancements in LLM reasoning, the field of multimodal reasoning has seen remarkable progress, achieving significant performance gains on intricate tasks such as mathematical problem-solving. Despite this progress, current multimodal large reasoning models exhibit two key limitations. They tend to employ computationally expensive reasoning even for simple queries, leading to inefficiency. Furthermore, this focus on specialized reasoning often impairs their broader, more general understanding capabilities. In this paper, we propose Metis-HOME: a Hybrid Optimized Mixture-of-Experts framework designed to address this trade-off. Metis-HOME enables a ''Hybrid Thinking'' paradigm by structuring the original dense model into two distinct expert branches: a thinking branch tailored for complex, multi-step reasoning, and a non-thinking branch optimized for rapid, direct inference on tasks like general VQA and OCR. A lightweight, trainable router dynamically allocates queries to the most suitable expert. We instantiate Metis-HOME by adapting the Qwen2.5-VL-7B into an MoE architecture. Comprehensive evaluations reveal that our approach not only substantially enhances complex reasoning abilities but also improves the model's general capabilities, reversing the degradation trend observed in other reasoning-specialized models. Our work establishes a new paradigm for building powerful and versatile MLLMs, effectively resolving the prevalent reasoning-vs-generalization dilemma. Code and weights are available at https://github.com/MM-Thinking/Metis-HOME.
Authors Xiaohan Lan, Fanfan Liu, Haibo Qiu, Siqi Yang, Delian Ruan, Peng Shi, Lin Ma
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
6
Beyond ReAct: A Planner-Centric Framework for Complex Tool-Augmented LLM Reasoning AAAI 2026
Title CN 超越ReAct：一种面向复杂工具增强型LLM推理的以规划器为中心的框架
Keywords Planner · 工具增强推理 · DAG规划 · ReAct改进 · 长程推理
Summary 本文提出了一种以规划器为中心的Plan-Execute范式，用于解决复杂查询下的工具增强型LLM推理问题。该方法引入全局DAG规划机制，结合ComplexTool-Plan数据集和两阶段训练策略（SFT+GRPO），显著提升了多工具组合与执行的准确率，在StableToolBench上达到SOTA。其核心创新在于将局部动作序列生成转变为全局任务结构建模。
Reason 该论文提出了一种以规划器（Planner）为核心的新型推理架构，通过全局有向无环图（DAG）规划来避免传统ReAct框架中的局部优化陷阱，并在复杂多工具协作任务上实现了SOTA。虽然其应用场景是通用工具调用，但其核心思想——通过结构化、全局性的计划生成来提升长程推理的一致性和准确性——具有迁移到文档图像理解（DIU）领域的潜力。例如，在处理长文档VQA或复杂表单解析时，模型常因逐步决策而偏离正确路径，该Planner机制可用于引导VLM进行多步信息定位与逻辑推理，缓解‘推理迷失’问题。然而，目前尚未直接应用于文档模态，且未涉及视觉细粒度对齐或OCR相关挑战，迁移仍需额外设计。
Abstract Existing tool-augmented large language models (LLMs) encounter significant challenges when processing complex queries. Current frameworks such as ReAct are prone to local optimization traps due to their reliance on incremental decision-making processes. To address these limitations, we propose a novel Planner-centric Plan-Execute paradigm that fundamentally resolves local optimization bottlenecks through architectural innovation. Central to our approach is a novel Planner model that performs global Directed Acyclic Graph (DAG) planning for complex queries, enabling optimized execution beyond conventional tool coordination. We also introduce ComplexTool-Plan, a large-scale benchmark dataset featuring complex queries that demand sophisticated multi-tool composition and coordination capabilities. Additionally, we develop a two-stage training methodology that integrates Supervised Fine-Tuning (SFT) with Group Relative Policy Optimization (GRPO), systematically enhancing the Planner's tool selection accuracy and global planning awareness through structured DAG-based planning. When integrated with a capable executor, our framework achieves state-of-the-art performance on the StableToolBench benchmark for complex user queries, demonstrating superior end-to-end execution capabilities and robust handling of intricate multi-tool workflows.
Authors Xiaolong Wei, Yuehu Dong, Xingliang Wang, Xingyu Zhang, Zhejun Zhao, Dongdong Shen, Long Xia, Dawei Yin
Comment Accepted by AAAI 2026
Categories Artificial Intelligence
6
Distilling Cross-Modal Knowledge via Feature Disentanglement AAAI 2026
Title CN 通过特征解耦蒸馏跨模态知识
Keywords 知识蒸馏 · 跨模态学习 · 频域特征 · 特征解耦 · 多模态对齐
Summary 本文提出频率解耦的跨模态知识蒸馏（FD-CMKD），利用频域分析发现低频特征在跨模态间具有一致性而高频特征差异大，因此分别施加强对齐和松散对齐损失，并引入尺度一致性损失和共享分类器来统一特征空间。实验表明该方法在多个基准上优于传统及现有跨模态KD方法。
Reason 该论文提出一种基于频域解耦的跨模态知识蒸馏方法，通过分离低频（高跨模态一致性）和高频（低一致性）特征并施加不同的对齐损失，提升视觉-语言模型间的知识迁移效率。虽然其核心任务未明确指向文档图像理解（DIU），但该方法为VLM在跨模态对齐上的不一致性问题提供了新思路。若将OCR输出或文本模态与文档图像进行频域层面的特征解耦对齐，可能缓解因模态差异导致的幻觉问题。此外，共享分类器与尺度一致性损失的设计有助于统一视觉与语言表征空间，对构建更鲁棒的端到端DocVQA系统具有潜在迁移价值。
Abstract Knowledge distillation (KD) has proven highly effective for compressing large models and enhancing the performance of smaller ones. However, its effectiveness diminishes in cross-modal scenarios, such as vision-to-language distillation, where inconsistencies in representation across modalities lead to difficult knowledge transfer. To address this challenge, we propose frequency-decoupled cross-modal knowledge distillation, a method designed to decouple and balance knowledge transfer across modalities by leveraging frequency-domain features. We observed that low-frequency features exhibit high consistency across different modalities, whereas high-frequency features demonstrate extremely low cross-modal similarity. Accordingly, we apply distinct losses to these features: enforcing strong alignment in the low-frequency domain and introducing relaxed alignment for high-frequency features. We also propose a scale consistency loss to address distributional shifts between modalities, and employ a shared classifier to unify feature spaces. Extensive experiments across multiple benchmark datasets show our method substantially outperforms traditional KD and state-of-the-art cross-modal KD approaches. Code is available at https://github.com/Johumliu/FD-CMKD.
Authors Junhong Liu, Yuan Zhang, Tao Huang, Wenchao Xu, Renyu Yang
Comment Accepted by AAAI 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
6
Bridging Symbolic Control and Neural Reasoning in LLM Agents: The Structured Cognitive Loop
Title CN 连接符号控制与神经推理的LLM智能体：结构化认知循环
Keywords LLM智能体 · 神经符号系统 · 模块化架构 · 推理控制 · 可解释性
Summary 本文提出了结构化认知循环（SCL）架构，将智能体的认知过程分为检索、认知、控制、行动和记忆五个阶段，并引入软符号控制机制，在保持神经网络灵活性的同时引入符号系统的可控性。实验表明，SCL在多步条件推理任务中实现了零策略违规、消除冗余工具调用并保持决策全程可追溯。作者还开源了实现代码，并展示了基于GPT-4o的旅行规划智能体案例。
Reason 该论文提出了一种模块化的智能体架构SCL，通过软符号控制机制分离认知与执行，提升多步推理过程中的可控性和可解释性。虽然其验证任务非文档理解，但该架构设计中的‘模块化分解’和‘透明状态管理’原则可迁移至长文档VQA或多步表单处理场景中，用于缓解长文档推理中的信息迷失和冗余操作问题，具备一定方法论参考价值。
Abstract Large language model agents suffer from fundamental architectural problems: entangled reasoning and execution, memory volatility, and uncontrolled action sequences. We introduce Structured Cognitive Loop (SCL), a modular architecture that explicitly separates agent cognition into five phases: Retrieval, Cognition, Control, Action, and Memory (R-CCAM). At the core of SCL is Soft Symbolic Control, an adaptive governance mechanism that applies symbolic constraints to probabilistic inference, preserving neural flexibility while restoring the explainability and controllability of classical symbolic systems. Through empirical validation on multi-step conditional reasoning tasks, we demonstrate that SCL achieves zero policy violations, eliminates redundant tool calls, and maintains complete decision traceability. These results address critical gaps in existing frameworks such as ReAct, AutoGPT, and memory-augmented approaches. Our contributions are threefold: (1) we situate SCL within the taxonomy of hybrid intelligence, differentiating it from prompt-centric and memory-only approaches; (2) we formally define Soft Symbolic Control and contrast it with neuro-symbolic AI; and (3) we derive three design principles for trustworthy agents: modular decomposition, adaptive symbolic governance, and transparent state management. We provide a complete open-source implementation demonstrating the R-CCAM loop architecture, alongside a live GPT-4o-powered travel planning agent. By connecting expert system principles with modern LLM capabilities, this work offers a practical and theoretically grounded path toward reliable, explainable, and governable AI agents.
Authors Myung Ho Kim
Comment Polished the abstract and replaced the demonstration screenshots
Categories Artificial Intelligence, Computation and Language
6
Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward
Title CN 理解是否真正指导生成？从统一多模态模型的分析到路径探索
Keywords 多模态模型 · 理解与生成 · Chain-of-Thought · 知识迁移 · 自训练
Summary 本文提出UniSandbox，一个用于分析统一多模态模型中理解与生成关系的解耦评估框架。通过合成数据避免数据泄露，研究发现当前模型存在显著的理解-生成鸿沟，尤其体现在推理生成和知识转移两方面。实验表明，显式Chain-of-Thought能有效桥接这一鸿沟，并可通过自训练实现隐式推理。此外，查询式架构被发现具有潜在的CoT特性，影响知识迁移效果。
Reason 该论文提出UniSandbox框架，通过解耦评估方式研究多模态模型中‘理解’与‘生成’之间的鸿沟，发现显式思维链（CoT）能有效弥合推理生成和知识迁移中的差距。虽然研究本身不直接针对文档图像理解任务，但其揭示的‘理解-生成’脱节问题在DocVQA等场景中同样严重，尤其是OCR幻觉、逻辑推理断裂等问题。其提出的自训练内化CoT能力的方法，可迁移用于提升VLM在文档问答中的推理一致性；查询式架构中隐含的CoT特性也对设计更可靠的文档理解架构有启发意义。
Abstract Recent years have witnessed significant progress in Unified Multimodal Models, yet a fundamental question remains: Does understanding truly inform generation? To investigate this, we introduce UniSandbox, a decoupled evaluation framework paired with controlled, synthetic datasets to avoid data leakage and enable detailed analysis. Our findings reveal a significant understanding-generation gap, which is mainly reflected in two key dimensions: reasoning generation and knowledge transfer. Specifically, for reasoning generation tasks, we observe that explicit Chain-of-Thought (CoT) in the understanding module effectively bridges the gap, and further demonstrate that a self-training approach can successfully internalize this ability, enabling implicit reasoning during generation. Additionally, for knowledge transfer tasks, we find that CoT assists the generative process by helping retrieve newly learned knowledge, and also discover that query-based architectures inherently exhibit latent CoT-like properties that affect this transfer. UniSandbox provides preliminary insights for designing future unified architectures and training strategies that truly bridge the gap between understanding and generation. Code and data are available at https://github.com/PKU-YuanGroup/UniSandBox
Authors Yuwei Niu, Weiyang Jin, Jiaqi Liao, Chaoran Feng, Peng Jin, Bin Lin, Zongjian Li, Bin Zhu, Weihao Yu, Li Yuan
Categories Computer Vision and Pattern Recognition, Computation and Language
6
Panoptic Captioning: An Equivalence Bridge for Image and Text NeurIPS 2025
Title CN 全景字幕：图像与文本的等价桥梁
Keywords 全景字幕 · 多模态大模型 · 细粒度对齐 · 分步推理 · 数据引擎
Summary 本文提出了全景字幕（Panoptic Captioning）这一新任务，目标是生成能够完全等价于图像内容的全面文本描述，涵盖所有对象、位置、属性、关系及全局状态。作者发现现有MLLM在该任务上表现有限，因此构建了PancapEngine数据引擎生成高质量训练数据，并提出PancapChain方法将任务分解为多个阶段逐步生成结果。同时贡献了评估指标PancapScore和人工标注测试集。实验表明其模型优于InternVL、GPT-4o等主流模型。
Reason 该论文提出‘全景字幕’任务，旨在生成包含图像中所有实体、属性、关系及全局状态的完整文本描述，并设计了PancapChain分步生成方法和高质量数据引擎。虽然任务本身属于通用视觉-语言理解，但其核心思想——通过结构化、细粒度地对齐视觉元素与文本描述，可能为文档图像理解中的复杂排版解析、多模态对齐提供启发。特别是其分阶段解耦策略（decoupling complex description into stages）可迁移至DocVQA中的长文档多步推理任务，以缓解信息遗漏或幻觉问题。然而，该工作未直接针对文档场景设计，也未处理OCR相关挑战如密集文本识别或布局分析，迁移需额外适配。
Abstract This work introduces panoptic captioning, a novel task striving to seek the minimum text equivalent of images, which has broad potential applications. We take the first step towards panoptic captioning by formulating it as a task of generating a comprehensive textual description for an image, which encapsulates all entities, their respective locations and attributes, relationships among entities, as well as global image state. Through an extensive evaluation, our work reveals that state-of-the-art Multi-modal Large Language Models (MLLMs) have limited performance in solving panoptic captioning. To address this, we propose an effective data engine named PancapEngine to produce high-quality data and a novel method named PancapChain to improve panoptic captioning. Specifically, our PancapEngine first detects diverse categories of entities in images by an elaborate detection suite, and then generates required panoptic captions using entity-aware prompts. Additionally, our PancapChain explicitly decouples the challenging panoptic captioning task into multiple stages and generates panoptic captions step by step. More importantly, we contribute a comprehensive metric named PancapScore and a human-curated test set for reliable model evaluation. Experiments show that our PancapChain-13B model can beat state-of-the-art open-source MLLMs like InternVL-2.5-78B and even surpass proprietary models like GPT-4o and Gemini-2.0-Pro, demonstrating the effectiveness of our data engine and method. Project page: https://visual-ai.github.io/pancap/
Authors Kun-Yu Lin, Hongjun Wang, Weining Ren, Kai Han
Comment NeurIPS 2025; Project page: https://visual-ai.github.io/pancap/
Categories Computer Vision and Pattern Recognition
6
TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models Findings of the Association for Computational Linguistics: EMNLP 2025
Title CN TurnBench-MS：一种评估大语言模型多轮、多步推理能力的基准
Keywords 多步推理 · 评测基准 · 大语言模型 · 交互式任务 · 推理链
Summary 本文提出了TurnBench，一个基于互动猜谜游戏的多轮多步推理评测基准，包含Classic和Nightmare两种模式，要求模型通过多轮猜测与结构化反馈推断隐藏规则。实验显示当前LLM在高复杂度下表现显著下降，而人类仍能保持完美表现，揭示了现有模型在持续推理上的不足。该基准支持细粒度分析，并降低数据污染风险。
Reason 该论文提出了一种用于评估多轮、多步推理能力的交互式基准测试，其核心贡献在于引入反馈循环和跨步骤推理追踪机制。虽然任务本身非文档理解，但其设计的动态推理诊断框架（如对中间推理步的标注与一致性维护）可迁移至长文档VQA场景中，用于检测模型在复杂逻辑链下的幻觉或信息丢失问题。
Abstract Despite impressive advances in large language models (LLMs), existing benchmarks often focus on single-turn or single-step tasks, failing to capture the kind of iterative reasoning required in real-world settings. To address this limitation, we introduce TurnBench, a novel benchmark that evaluates multi-turn, multi-step reasoning through an interactive code-breaking task inspired by the "Turing Machine Board Game." In each episode, a model must uncover hidden logical or arithmetic rules by making sequential guesses, receiving structured feedback, and integrating clues across multiple rounds. This dynamic setup requires models to reason over time, adapt based on past information, and maintain consistency across steps-capabilities underexplored in current benchmarks. TurnBench includes two modes: Classic, which tests standard reasoning, and Nightmare, which introduces increased complexity and requires robust inferential chains. To support fine-grained analysis, we provide ground-truth annotations for intermediate reasoning steps. Our evaluation of state-of-the-art LLMs reveals significant gaps: the best model achieves 84% accuracy in Classic mode, but performance drops to 18% in Nightmare mode. In contrast, human participants achieve 100% in both, underscoring the challenge TurnBench poses to current models. By incorporating feedback loops and hiding task rules, TurnBench reduces contamination risks and provides a rigorous testbed for diagnosing and advancing multi-step, multi-turn reasoning in LLMs.
Authors Yiran Zhang, Mo Wang, Xiaoyang Li, Kaixuan Ren, Chencheng Zhu, Usman Naseem
Comment Accepted to Findings of the Association for Computational Linguistics: EMNLP 2025
Categories Computation and Language
6
Cognitive Foundations for Reasoning and Their Manifestation in LLMs
Title CN 推理的认知基础及其在大语言模型中的体现
Keywords 认知科学 · 推理机制 · 测试时引导 · 多模态推理 · LLM评估
Summary 本文结合认知科学研究，提出一个涵盖28个认知要素的分类体系，用于系统分析大语言模型在跨模态任务中的推理行为。作者构建了一个细粒度评估框架，分析了18个模型共19.2万条推理轨迹，并对比了54个人类‘出声思考’样本。研究发现，模型在结构不良的问题上倾向于单一的顺序处理，缺乏人类所表现出的抽象与概念化思维，且社区研究过度集中于可量化但浅层的推理组件（如分解），忽视了与成功强相关的元认知控制能力。基于此，作者开发了一种测试时推理引导机制，能自动构建有效的推理结构，使复杂任务性能最高提升66.7%。
Reason 该论文从认知科学角度构建了包含28个认知要素的分类体系，并提出一种细粒度评估框架来分析LLM在多模态任务中的推理行为。虽然研究涵盖文本、视觉和音频，但其核心贡献是通用推理机制的诊断与增强，而非针对文档图像理解（DIU）的具体问题。其提出的‘测试时推理引导’方法（test-time reasoning guidance）具有潜在迁移价值——例如可用于缓解DocVQA中的幻觉或逻辑断裂问题，通过引入抽象表征与元认知监控来提升复杂文档的多步推理能力。然而，该工作未涉及视觉布局、高分辨率输入处理或多模态对齐等DIU关键挑战，且实验不聚焦于文档场景，因此属于关联领域中偏通用的方法论探索。
Abstract Large language models (LLMs) solve complex problems yet fail on simpler variants, suggesting they achieve correct outputs through mechanisms fundamentally different from human reasoning. To understand this gap, we synthesize cognitive science research into a taxonomy of 28 cognitive elements spanning reasoning invariants, meta-cognitive controls, representations for organizing reasoning & knowledge, and transformation operations. We introduce a fine-grained evaluation framework and conduct the first large-scale empirical analysis of 192K traces from 18 models across text, vision, and audio, complemented by 54 human think-aloud traces, which we make publicly available. We find that models under-utilize cognitive elements correlated with success, narrowing to rigid sequential processing on ill-structured problems where diverse representations and meta-cognitive monitoring are critical. Human traces show more abstraction and conceptual processing, while models default to surface-level enumeration. Meta-analysis of 1.6K LLM reasoning papers reveals the research community concentrates on easily quantifiable elements (sequential organization: 55%, decomposition: 60%) but neglecting meta-cognitive controls (self-awareness: 16%) that correlate with success. Models possess behavioral repertoires associated with success but fail to deploy them spontaneously. Leveraging these patterns, we develop test-time reasoning guidance that automatically scaffold successful structures, improving performance by up to 66.7% on complex problems. By establishing a shared vocabulary between cognitive science and LLM research, our framework enables systematic diagnosis of reasoning failures and principled development of models that reason through robust cognitive mechanisms rather than spurious shortcuts, while providing tools to test theories of human cognition at scale.
Authors Priyanka Kargupta, Shuyue Stella Li, Haocheng Wang, Jinu Lee, Shan Chen, Orevaoghene Ahia, Dean Light, Thomas L. Griffiths, Max Kleiman-Weiner, Jiawei Han, Asli Celikyilmaz, Yulia Tsvetkov
Comment 40 pages, 4 tables, 6 figures
Categories Artificial Intelligence
6
CLASH: A Benchmark for Cross-Modal Contradiction Detection
Title CN CLASH：跨模态矛盾检测基准
Keywords 跨模态矛盾检测 · 幻觉抑制 · 多模态基准 · 可靠性 · VLM评估
Summary 本文提出了CLASH，一个专注于检测图像与文本之间跨模态矛盾的新基准，包含带有可控矛盾的COCO图像-字幕对，并设计了多项选择和开放式问题评估方式。实验表明现有模型在识别此类矛盾方面存在明显不足，而针对性微调可显著提升性能。该基准有助于提升多模态系统的可靠性和一致性。
Reason 该论文提出一个用于检测图像与文本之间跨模态矛盾的新基准CLASH，聚焦于对象级或属性级的矛盾识别。虽然其核心任务并非文档理解，但其目标——提升多模态系统对不一致输入的敏感性、减少幻觉——与文档图像理解（DIU）中VLM因OCR错误或布局干扰产生幻觉的问题高度相关。该方法可作为上游工具迁移到DIU领域，例如在DocVQA中引入矛盾检测机制来验证模型回答是否与视觉内容一致，从而缓解幻觉问题。然而，该工作本身未涉及文档结构、排版或真实文档场景，迁移需额外适配，因此价值有限。
Abstract Contradictory multimodal inputs are common in real-world settings, yet existing benchmarks typically assume input consistency and fail to evaluate cross-modal contradiction detection - a fundamental capability for preventing hallucinations and ensuring reliability. We introduce CLASH, a novel benchmark for multimodal contradiction detection, featuring COCO images paired with contradictory captions containing controlled object-level or attribute-level contradictions. The samples include targeted questions evaluated in both multiple-choice and open-ended formats. The benchmark provides an extensive fine-tuning set filtered through automated quality checks, alongside a smaller human-verified diagnostic set. Our analysis of state-of-the-art models reveals substantial limitations in recognizing cross-modal conflicts, exposing systematic modality biases and category-specific weaknesses. Furthermore, we empirically demonstrate that targeted fine-tuning on CLASH substantially enhances conflict detection capabilities.
Authors Teodora Popordanoska, Jiameng Li, Matthew B. Blaschko
Comment First two authors contributed equally
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
6
Synthesizing Visual Concepts as Vision-Language Programs
Title CN 将视觉概念综合为视觉-语言程序
Keywords 视觉语言模型 · 程序合成 · 神经符号系统 · 系统性推理 · 去幻觉
Summary 本文提出Vision-Language Programs（VLP），一种结合VLM感知能力和程序合成系统性推理的方法。VLP引导VLM生成结构化视觉描述，并将其编译为可在图像上执行的神经符号程序，从而确保输出符合逻辑约束并提供可解释性。实验表明该方法在需要复杂逻辑推理的任务上优于直接提示或结构化提示方法。
Reason 该论文提出Vision-Language Programs（VLP），通过将VLM生成的结构化视觉描述编译为可执行的神经符号程序，以提升系统性视觉推理能力。其核心贡献在于解耦感知与推理，利用程序合成保证逻辑一致性。虽然方法本身并非针对文档图像理解设计，但其‘通过程序结构约束VLM输出以减少幻觉’的思想，对DIU中常见的OCR或语义推理幻觉问题具有潜在迁移价值——例如在复杂表单理解或多跳DocVQA任务中引入可验证的推理路径。然而，目前缺乏在真实文档数据上的验证，且文档中的密集文本和排版约束未被建模，因此实际迁移难度较高。
Abstract Vision-Language models (VLMs) achieve strong performance on multimodal tasks but often fail at systematic visual reasoning tasks, leading to inconsistent or illogical outputs. Neuro-symbolic methods promise to address this by inducing interpretable logical rules, though they exploit rigid, domain-specific perception modules. We propose Vision-Language Programs (VLP), which combine the perceptual flexibility of VLMs with systematic reasoning of program synthesis. Rather than embedding reasoning inside the VLM, VLP leverages the model to produce structured visual descriptions that are compiled into neuro-symbolic programs. The resulting programs execute directly on images, remain consistent with task constraints, and provide human-interpretable explanations that enable easy shortcut mitigation. Experiments on synthetic and real-world datasets demonstrate that VLPs outperform direct and structured prompting, particularly on tasks requiring complex logical reasoning.
Authors Antonia Wüst, Wolfgang Stammer, Hikaru Shindo, Lukas Helff, Devendra Singh Dhami, Kristian Kersting
Categories Artificial Intelligence
6
COLI: A Hierarchical Efficient Compressor for Large Images
Title CN COLI：一种用于大图像的分层高效压缩器
Keywords 图像压缩 · 隐式神经表示 · 高分辨率图像 · NeRV · 视觉表示学习
Summary 本文提出了COLI，一种面向大尺寸图像的高效神经压缩框架。它结合NeRV结构，通过训练策略优化（预训练+混合精度+并行损失）加快收敛速度，并设计Hyper-Compression方法进一步压缩模型权重，在保持高质量重建的同时显著降低比特率。实验表明其在医学图像上优于现有方法。
Reason 该论文提出了一种基于隐式神经表示（INR）的大图像压缩框架COLI，通过预训练-微调范式、混合精度训练和并行化优化显著加速了NeRV的收敛，并引入Hyper-Compression技术提升压缩比。虽然其应用背景是医学影像，但其核心思想——高效高分辨率视觉信号的神经压缩与重建——与文档图像理解中处理高分辨率扫描件（如合同、票据、书籍页面）时面临的计算与存储瓶颈高度相关。特别是其对INR训练效率的改进，可能为OCR-free或端到端VLM在长文档、高分辨率输入下的部署提供底层支持，属于可迁移的‘视觉表示学习’工具类工作。
Abstract The escalating adoption of high-resolution, large-field-of-view imagery amplifies the need for efficient compression methodologies. Conventional techniques frequently fail to preserve critical image details, while data-driven approaches exhibit limited generalizability. Implicit Neural Representations (INRs) present a promising alternative by learning continuous mappings from spatial coordinates to pixel intensities for individual images, thereby storing network weights rather than raw pixels and avoiding the generalization problem. However, INR-based compression of large images faces challenges including slow compression speed and suboptimal compression ratios. To address these limitations, we introduce COLI (Compressor for Large Images), a novel framework leveraging Neural Representations for Videos (NeRV). First, recognizing that INR-based compression constitutes a training process, we accelerate its convergence through a pretraining-finetuning paradigm, mixed-precision training, and reformulation of the sequential loss into a parallelizable objective. Second, capitalizing on INRs' transformation of image storage constraints into weight storage, we implement Hyper-Compression, a novel post-training technique to substantially enhance compression ratios while maintaining minimal output distortion. Evaluations across two medical imaging datasets demonstrate that COLI consistently achieves competitive or superior PSNR and SSIM metrics at significantly reduced bits per pixel (bpp), while accelerating NeRV training by up to 4 times.
Authors Haoran Wang, Hanyu Pei, Yang Lyu, Kai Zhang, Li Li, Feng-Lei Fan
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
6
CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring 39th Conference on Neural Information Processing Systems (NeurIPS 2025)
Title CN 抓住CoT的把柄：链式思维监控的压力测试
Keywords 链式思维监控 · 推理干预 · 混合监控 · 模型安全性 · 红队测试
Summary 本文提出并评估了一种链式思维（CoT）监控机制，用于在红队测试中检测强大但不可信模型在执行编码任务时的隐蔽有害行为。研究发现，尽管CoT监控优于仅监控输出结果的方法，但推理轨迹可能包含欺骗性解释，反而降低检测效果。为此，作者设计了一种混合监控协议，独立评估推理与动作，并通过加权平均结合两者，显著提升了对隐秘欺骗行为的检出率。
Reason 该论文研究了在AI自主性增强背景下，如何通过弱监督模型监控强模型的中间推理步骤（Chain-of-Thought）以检测有害行为。虽然其核心任务不属于文档图像理解（DIU），但其提出的混合监控机制——对推理过程和最终动作分别评分并加权融合——作为一种推理阶段干预（Intervention/Steering）的通用方法，具备迁移到DIU领域的潜力。例如，在复杂DocVQA任务中，主模型可能因排版干扰产生错误的推理路径，此时可引入此类可信监控机制，在推理中途识别并纠正由OCR幻觉或布局误解导致的‘误导性理性化’，从而提升答案可靠性。然而，该方法目前应用于代码安全场景，迁移至视觉-语言文档理解仍需大量适配工作，且未直接处理高分辨率、细粒度定位等DIU核心挑战，故为中等偏弱相关。
Abstract As AI models are deployed with increasing autonomy, it is important to ensure they do not take harmful actions unnoticed. As a potential mitigation, we investigate Chain-of-Thought (CoT) monitoring, wherein a weaker trusted monitor model continuously oversees the intermediate reasoning steps of a more powerful but untrusted model. We compare CoT monitoring to action-only monitoring, where only final outputs are reviewed, in a red-teaming setup where the untrusted model is instructed to pursue harmful side tasks while completing a coding problem. We find that while CoT monitoring is more effective than overseeing only model outputs in scenarios where action-only monitoring fails to reliably identify sabotage, reasoning traces can contain misleading rationalizations that deceive the CoT monitors, reducing performance in obvious sabotage cases. To address this, we introduce a hybrid protocol that independently scores model reasoning and actions, and combines them using a weighted average. Our hybrid monitor consistently outperforms both CoT and action-only monitors across all tested models and tasks, with detection rates twice higher than action-only monitoring for subtle deception scenarios.
Authors Benjamin Arnav, Pablo Bernabeu-Pérez, Nathan Helm-Burger, Tim Kostolansky, Hannes Whittingham, Mary Phuong
Comment To be published in the 39th Conference on Neural Information Processing Systems (NeurIPS 2025)
Categories Artificial Intelligence
6
Towards Efficient VLMs: Information-Theoretic Driven Compression via Adaptive Structural Pruning
Title CN 迈向高效的视觉语言模型：基于信息论驱动的自适应结构化剪枝压缩
Keywords 视觉语言模型压缩 · 信息瓶颈 · 结构化剪枝 · 注意力头剪枝 · 高效推理
Summary 本文提出InfoPrune，一种基于信息论的视觉语言模型压缩框架，利用信息瓶颈原则衡量注意力头的信息保留能力，结合eRank与KS距离进行自适应结构剪枝，并设计了训练式注意力头剪枝和无训练FFN低秩压缩两种方案。实验表明其可在VQA任务上实现显著加速与FLOPs降低，同时性能损失极小。
Reason 该论文提出了一种基于信息瓶颈原理的VLM压缩框架InfoPrune，通过熵有效秩（eRank）和KS距离量化注意力头的重要性，实现结构化剪枝。虽然其本身不是针对文档图像理解任务设计的，但其提出的自适应结构压缩方法具有良好的迁移潜力——例如，在处理高分辨率文档图像时，大型VLM常面临计算冗余与部署困难，该方法可被用于压缩专用于DIU的多模态模型（如LayoutLLM或DocLLM），在保持对复杂排版和细粒度文本识别能力的同时提升推理效率。然而，论文未涉及OCR幻觉、文本定位或长文档推理等核心DIU挑战，且缺乏在文档数据上的验证，因此属于弱相关的方法工具类工作。
Abstract Recent advances in vision-language models (VLMs) have shown remarkable performance across multimodal tasks, yet their ever-growing scale poses severe challenges for deployment and efficiency. Existing compression methods often rely on heuristic importance metrics or empirical pruning rules, lacking theoretical guarantees about information preservation. In this work, we propose InfoPrune, an information-theoretic framework for adaptive structural compression of VLMs. Grounded in the Information Bottleneck principle, we formulate pruning as a trade-off between retaining task-relevant semantics and discarding redundant dependencies. To quantify the contribution of each attention head, we introduce an entropy-based effective rank (eRank) and employ the Kolmogorov--Smirnov (KS) distance to measure the divergence between original and compressed structures. This yields a unified criterion that jointly considers structural sparsity and informational efficiency. Building on this foundation, we further design two complementary schemes: (1) a training-based head pruning guided by the proposed information loss objective, and (2) a training-free FFN compression via adaptive low-rank approximation. Extensive experiments on VQAv2, TextVQA, and GQA demonstrate that InfoPrune achieves up to 3.2x FLOP reduction and 1.8x acceleration with negligible performance degradation, establishing a theoretically grounded and practically effective step toward efficient multimodal large models.
Authors Zhaoqi Xu, Yingying Zhang, Jian Li, Jianwei Guo, Qiannan Zhu, Hua Huang
Categories Artificial Intelligence
6
Restore Text First, Enhance Image Later: Two-Stage Scene Text Image Super-Resolution with Glyph Structure Guidance
Title CN 先恢复文本，再增强图像：基于字形结构引导的两阶段场景文本图像超分辨率
Keywords 文本图像超分辨率 · 字形结构引导 · 两阶段框架 · 场景文本 · 图像增强
Summary 本文提出TIGER模型，采用两阶段策略进行场景文本图像超分辨率：第一阶段专注于精确恢复文本的字形结构，第二阶段以该结构为引导完成整幅图像的高质量重建。为支持训练与评估，作者构建了首个面向中文场景文本的极端缩放数据集UZ-ST。实验表明，TIGER在保持图像美观的同时显著提升了文本可读性。
Reason 该论文提出了一种两阶段的文本图像超分辨率框架TIGER，通过先恢复文本结构再进行整体图像增强，缓解了自然图像超分中常见的文本扭曲问题。其核心思想是利用字形结构指导超分辨率过程，从而提升文本可读性。虽然研究目标是场景文本图像质量提升，属于预处理环节，但其‘结构优先、内容后补’的设计思路对文档图像理解中的OCR前处理有潜在借鉴意义，例如在低质量扫描件或模糊文档中增强关键文本区域。然而，该工作仍聚焦于图像重建任务本身，并未直接解决DIU中的多模态推理、布局分析或VLM集成等核心问题，迁移价值有限。
Abstract Current image super-resolution methods show strong performance on natural images but distort text, creating a fundamental trade-off between image quality and textual readability. To address this, we introduce TIGER (Text-Image Guided supEr-Resolution), a novel two-stage framework that breaks this trade-off through a "text-first, image-later" paradigm. TIGER explicitly decouples glyph restoration from image enhancement: it first reconstructs precise text structures and uses them to guide full-image super-resolution. This ensures high fidelity and readability. To support comprehensive training and evaluation, we present the UZ-ST (UltraZoom-Scene Text) dataset, the first Chinese scene text dataset with extreme zoom. Extensive experiments show TIGER achieves state-of-the-art performance, enhancing readability and image quality.
Authors Minxing Luo, Linlong Fan, Wang Qiushi, Ge Wu, Yiyan Luo, Yuhang Yu, Jinwei Chen, Yaxing Wang, Qingnan Fan, Jian Yang
Categories Computer Vision and Pattern Recognition
6
FVAR: Visual Autoregressive Modeling via Next Focus Prediction
Title CN FVAR：通过下一焦点预测实现视觉自回归建模
Keywords 视觉自回归 · 去混叠 · 多尺度表示 · 高频残差学习 · 渐进重构
Summary FVAR提出一种新的视觉自回归建模框架，摒弃传统均匀下采样的多尺度金字塔，转而模拟光学聚焦过程，使用递减半径的散焦点扩散函数（PSF）构建从模糊到清晰的渐进金字塔，从根本上消除混叠伪影。同时引入高频残差教师网络，在训练时学习结构与别名信息，推理时蒸馏给轻量级模型，保持部署简洁性。实验表明其在ImageNet上显著提升生成质量与文本可读性。
Reason 该论文提出了一种新的视觉自回归生成范式，将传统的多尺度下采样预测改为模拟相机聚焦过程的‘下一焦点预测’，通过物理一致的散焦核构建无混叠的多尺度金字塔，并引入高频残差学习来保留细节。虽然其主要实验在ImageNet上进行，且目标是图像生成，但其核心思想——通过可控的模糊到清晰过渡来避免高频失真——对文档图像理解中的高分辨率处理和文本可读性提升具有潜在迁移价值。例如，在OCR-free DocVQA中，该方法可能有助于缓解因下采样导致的细小文字丢失或摩尔纹干扰，从而改善VLM对密集文本区域的感知能力。
Abstract Visual autoregressive models achieve remarkable generation quality through next-scale predictions across multi-scale token pyramids. However, the conventional method uses uniform scale downsampling to build these pyramids, leading to aliasing artifacts that compromise fine details and introduce unwanted jaggies and moiré patterns. To tackle this issue, we present \textbf{FVAR}, which reframes the paradigm from \emph{next-scale prediction} to \emph{next-focus prediction}, mimicking the natural process of camera focusing from blur to clarity. Our approach introduces three key innovations: \textbf{1) Next-Focus Prediction Paradigm} that transforms multi-scale autoregression by progressively reducing blur rather than simply downsampling; \textbf{2) Progressive Refocusing Pyramid Construction} that uses physics-consistent defocus kernels to build clean, alias-free multi-scale representations; and \textbf{3) High-Frequency Residual Learning} that employs a specialized residual teacher network to effectively incorporate alias information during training while maintaining deployment simplicity. Specifically, we construct optical low-pass views using defocus point spread function (PSF) kernels with decreasing radius, creating smooth blur-to-clarity transitions that eliminate aliasing at its source. To further enhance detail generation, we introduce a High-Frequency Residual Teacher that learns from both clean structure and alias residuals, distilling this knowledge to a vanilla VAR deployment network for seamless inference. Extensive experiments on ImageNet demonstrate that FVAR substantially reduces aliasing artifacts, improves fine detail preservation, and enhances text readability, achieving superior performance with perfect compatibility to existing VAR frameworks.
Authors Xiaofan Li, Chenming Wu, Yanpeng Sun, Jiaming Zhou, Delin Qu, Yansong Qu, Weihao Bo, Haibao Yu, Dingkang Liang
Comment 10 pages, 4 figures
Categories Computer Vision and Pattern Recognition
6
CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization
Title CN CodeV：通过工具感知策略优化实现基于代码的忠实视觉推理
Keywords 视觉代理 · 工具调用 · 过程监督 · 强化学习 · 忠实推理
Summary 本文提出CodeV，一种基于代码的视觉代理模型，采用工具感知策略优化（TAPO）框架，在视觉工具调用过程中引入细粒度的过程监督，提升模型中间推理步骤的忠实性。通过将视觉工具表示为可执行Python代码，并根据工具输入输出分配逐歩奖励，有效减少了传统方法中因依赖思维链令牌而导致的奖励欺骗问题。实验表明，CodeV在保持高准确率的同时显著提升了工具使用的忠实度，并在多种多模态推理与数学任务上表现优异。
Reason 该论文提出了一种用于视觉语言代理的训练框架TAPO，通过在工具输入输出上定义密集奖励来提升中间步骤的忠实性，解决模型调用图像操作时可能忽略关键区域或无视工具输出的问题。虽然其核心任务是通用视觉推理而非文档理解，但其‘过程级监督’和‘基于可执行代码的工具调用’机制对DIU领域具有潜在迁移价值——例如可用于长文档中多步信息检索的流程控制，防止模型在复杂排版中迷失；也可用于抑制OCR幻觉，强制模型依据真实裁剪区域进行推理。然而，目前未见直接应用于文档结构分析或表格识别等具体DIU任务，属于上游方法探索，迁移路径尚需验证。
Abstract Agentic vision-language models are increasingly trained to "think with images" by calling image operations. However, we show that high final-answer accuracy often hides unfaithful visual reasoning: models may invoke tools on irrelevant regions or ignore tool outputs entirely, yet still guess the correct answer. In this work, we first propose a faithfulness evaluation protocol that measures whether intermediate visual tool outputs (e.g., crops) actually contain the queried evidence. This reveals that recent visual agents achieve high final-answer accuracy but exhibit low rates of faithful tool-use on visual search benchmarks. We then introduce CodeV, a code-based visual agent trained with Tool-Aware Policy Optimization (TAPO). TAPO is a process-level RL framework that augments GRPO with dense rewards defined directly on visual tool inputs and outputs, rather than on chain-of-thought tokens, making supervision easier to verify and less susceptible to reward hacking. CodeV represents visual tools as executable Python code, and TAPO assigns step-wise rewards based solely on the question and tool output, encouraging both necessary and evidence-consistent tool use. In a two-stage SFT+RL pipeline, CodeV achieves competitive or superior accuracy while substantially increasing faithful tool-use rates on related visual search benchmarks. Beyond visual search, CodeV attains strong performance on a range of multimodal reasoning and math benchmarks, suggesting that explicitly supervising intermediate tool behavior is crucial for building trustworthy, agentic visual reasoning systems.
Authors Xinhai Hou, Shaoyuan Xu, Manan Biyani, Mayan Li, Jia Liu, Todd C. Hollon, Bryan Wang
Categories Computer Vision and Pattern Recognition
6
Learning to Reason: Training LLMs with GPT-OSS or DeepSeek R1 Reasoning Traces
Title CN 学会推理：使用GPT-OSS或DeepSeek R1推理轨迹训练大语言模型
Keywords 推理轨迹 · 测试时扩展 · 后训练 · 大语言模型 · 数学推理
Summary 本文比较了使用DeepSeek-R1和gpt-oss生成的推理轨迹对中等规模语言模型进行后训练的效果，旨在提升模型在数学问题上的推理准确性和推理效率。研究表明，基于大规模模型生成的高质量推理路径进行监督学习，是一种有效的推理能力传授方式，无需人工标注。该方法为构建具备推理能力的小模型提供了低成本路径。
Reason 该论文研究如何利用前沿模型（如DeepSeek-R1和gpt-oss）生成的推理轨迹来对中小规模LLM进行后训练，从而赋予其复杂问题求解的推理能力。虽然任务本身属于纯文本数学推理，但其方法论——利用高质量推理路径作为监督信号提升模型推理能力——具有迁移到文档图像理解（DIU）的潜力。例如，在DocVQA中，模型常需执行多跳推理（如跨表格与正文的信息整合），若能用类似方式注入结构化推理轨迹（如‘先定位表格→提取数值→比较上下文→得出结论’），可缓解长文档中的逻辑断链问题。然而，当前工作未涉及视觉输入或图文对齐，迁移仍需额外设计模态融合机制，故为弱相关。
Abstract Test-time scaling, which leverages additional computation during inference to improve model accuracy, has enabled a new class of Large Language Models (LLMs) that are able to reason through complex problems by understanding the goal, turning this goal into a plan, working through intermediate steps, and checking their own work before answering . Frontier large language models with reasoning capabilities, such as DeepSeek-R1 and OpenAI's gpt-oss, follow the same procedure when solving complex problems by generating intermediate reasoning traces before giving the final answer. Today, these models are being increasingly used to generate reasoning traces that serve as high-quality supervised data for post-training of small and medium-sized language models to teach reasoning capabilities without requiring expensive human curation. In this work, we compare the performance of medium-sized LLMs on Math problems after post-training on two kinds of reasoning traces. We compare the impact of reasoning traces generated by DeepSeek-R1 and gpt-oss LLMs in terms of accuracy and inference efficiency.
Authors Shaltiel Shmidman, Asher Fredman, Oleg Sudakov, Meriem Bendris
Categories Computation and Language
6
SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
Title CN SlimInfer：通过动态令牌剪枝加速长上下文LLM推理
Keywords 长上下文推理 · 令牌剪枝 · LLM加速 · KV缓存优化 · 动态稀疏化
Summary SlimInfer提出一种动态细粒度令牌剪枝框架，通过在前向传播中逐层剔除不重要的隐藏状态令牌来加速长上下文LLM推理。其利用信息扩散特性，在保持模型输出质量的同时显著减少计算和内存开销，实验显示在LLaMA3.1-8B上实现高达2.53倍的首 token 延迟加速和1.88倍端到端延迟降低。
Reason 该论文提出了一种在LLM推理过程中动态剪枝冗余提示令牌的方法，以加速长上下文处理并降低内存开销。其核心机制是基于信息扩散现象，在中间层逐步去除对语义贡献较小的隐藏状态令牌，并设计了异步KV缓存预取机制。虽然工作本身面向通用长文本LLM推理，但其思想对文档图像理解（DIU）中处理长文档（如合同、报表）具有潜在迁移价值——例如，在DocVQA任务中结合高分辨率文档输入时，VLM需处理大量视觉token，SlimInfer的动态剪枝策略可被用于压缩视觉token序列，缓解高分辨率带来的计算压力，提升推理效率。然而，该方法尚未在多模态或视觉上下文中验证，直接应用于文档图像仍存在适配挑战，故为弱相关。
Abstract Long-context inference for Large Language Models (LLMs) is heavily limited by high computational demands. While several existing methods optimize attention computation, they still process the full set of hidden states at each layer, limiting overall efficiency. In this work, we propose SlimInfer, an innovative framework that aims to accelerate inference by directly pruning less critical prompt tokens during the forward pass. Our key insight is an information diffusion phenomenon: As information from critical tokens propagates through layers, it becomes distributed across the entire sequence. This diffusion process suggests that LLMs can maintain their semantic integrity when excessive tokens, even including these critical ones, are pruned in hidden states. Motivated by this, SlimInfer introduces a dynamic fine-grained pruning mechanism that accurately removes redundant tokens of hidden state at intermediate layers. This layer-wise pruning naturally enables an asynchronous KV cache manager that prefetches required token blocks without complex predictors, reducing both memory usage and I/O costs. Extensive experiments show that SlimInfer can achieve up to $\mathbf{2.53\times}$ time-to-first-token (TTFT) speedup and $\mathbf{1.88\times}$ end-to-end latency reduction for LLaMA3.1-8B-Instruct on a single RTX 4090, without sacrificing performance on LongBench. Our code is available at https://github.com/Longxmas/SlimInfer.
Authors Lingkun Long, Rubing Yang, Yushi Huang, Desheng Hui, Ao Zhou, Jianlei Yang
Categories Computation and Language
6
Lessons from Studying Two-Hop Latent Reasoning
Title CN 关于两步隐式推理的研究启示
Keywords 大语言模型 · 隐式推理 · 多跳推理 · 链式思维 · 合成数据
Summary 本文通过构建受控的两步问答任务，使用合成事实排除记忆和捷径推理干扰，研究大语言模型是否具备隐式多跳推理能力。实验发现模型难以组合两个合成事实，但在一个合成与一个自然事实条件下能成功，表明LLM确实具备一定隐式推理能力，但其能力受限于训练分布与实验设置。研究提醒社区在评估LLM推理时需谨慎设计实验以避免虚假成败结论。
Reason 该论文研究了大语言模型在无外部提示（如CoT）情况下的两步推理能力，通过合成事实控制实验环境，证实LLM具备一定的隐式多跳推理能力。虽然任务本身非文档图像理解（DIU），但其方法论对DIU中的长文档多步信息整合与逻辑推理有潜在启发：例如，在处理需跨段落、跨表格推理的DocVQA任务时，可借鉴其合成数据设计思路来诊断或增强模型的内在推理链完整性。然而，该工作未直接涉及视觉输入或多模态对齐，迁移需较大改造，故为中等偏弱价值。
Abstract Large language models can use chain-of-thought (CoT) to externalize reasoning, potentially enabling oversight of capable LLM agents. Prior work has shown that models struggle at two-hop question-answering without CoT. This capability is so basic that if it was a fundamental limitation, it would imply that many complex agentic tasks would similarly require CoT. We investigate LLM latent reasoning capabilities using two-hop question answering as a case study. Previous work on the gap between latent and externalized two-hop reasoning produced mixed evidence with inconclusive results. In this paper, we introduce a controlled setting for investigating two-hop reasoning in LLMs, where a positive result provides definitive evidence for latent reasoning. We fine-tune LLMs (including Llama 3 8B and GPT-4o) on synthetic facts and test two-hop reasoning over these facts. By using synthetic facts, we rule out memorization and reasoning shortcuts as explanations for two-hop performance. We observe a nuanced picture: Models fail to compose two synthetic facts, but can succeed when one fact is synthetic and the other is natural. These results demonstrate that LLMs are undeniably capable of latent two-hop reasoning, although it remains unclear how this ability scales with model size. Finally, we highlight a lesson for researchers studying LLM reasoning: when drawing conclusions about LLM latent reasoning, one must be careful to avoid both spurious successes (that stem from memorization and reasoning shortcuts) and spurious failures (that may stem from artificial experimental setups, divorced from training setups of frontier LLMs).
Authors Mikita Balesni, Tomek Korbak, Owain Evans
Categories Artificial Intelligence, Computation and Language
6
AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expert
Title CN AnyExperts：面向多模态语言模型的按需专家分配混合专家系统
Keywords 多模态MoE · 动态路由 · 计算效率 · 虚拟专家 · 语义重要性
Summary 本文提出AnyExperts，一种面向多模态大模型的动态专家分配框架，通过引入语义重要性感知的路由机制，按需分配真实与虚拟专家，控制总计算量的同时提升资源利用效率。在图像、视频和文本密集型任务（如OCR）中验证了其有效性，在保持性能的同时显著降低真实专家调用次数。
Reason 该论文提出了一种基于语义重要性的动态路由机制，在多模态MoE模型中实现按需分配真实与虚拟专家，提升计算效率。虽然其在OCR密集文本任务上进行了测试并展示了10%的真实专家减少而性能不变，但其核心贡献属于通用多模态MoE架构优化，对文档理解中的OCR幻觉、复杂排版或细粒度定位等关键痛点缺乏直接建模或针对性改进。其方法可视为一种潜在的效率工具，若用于大规模VLM-based DIU系统的推理加速具有一定迁移价值，但本身未触及文档结构理解或视觉-文本对齐的本质挑战。
Abstract Multimodal Mixture-of-Experts (MoE) models offer a promising path toward scalable and efficient large vision-language systems. However, existing approaches rely on rigid routing strategies (typically activating a fixed number of experts per token) ignoring the inherent heterogeneity in semantic importance across modalities. This leads to suboptimal compute allocation, where redundant tokens consume as many resources as critical ones. To address this, we propose AnyExperts, a novel on-demand, budget-aware dynamic routing framework that allocates a variable total number of expert slots per token based on its semantic importance. Crucially, to prevent uncontrolled compute growth, the total slots per token are constrained within a fixed range, and each slot is filled by either a real expert or a virtual expert, with the virtual share capped at a small maximum (e.g., 20%). The model then adaptively balances the real-to-virtual ratio per token, assigning more real experts to semantically rich regions and relying more on virtual experts for redundant content. Evaluated across diverse tasks in visual understanding, audio understanding, and NLP understanding, AnyExperts improves performance under the same compute budget. Notably, on general image/video tasks, it achieves comparable accuracy with 40% fewer real expert activations; on text-dense tasks (OCR and NLP), it maintains performance while reducing real expert usage by 10%. These results demonstrate that fine-grained, importance-driven expert allocation significantly enhances both the efficiency and effectiveness of multimodal MoE models.
Authors Yuting Gao, Wang Lan, Hengyuan Zhao, Linjiang Huang, Si Liu, Qingpei Guo
Categories Artificial Intelligence
6
Extreme Model Compression for Edge Vision-Language Models: Sparse Temporal Token Fusion and Adaptive Neural Compression
Title CN 面向边缘视觉-语言模型的极致模型压缩：稀疏时序Token融合与自适应神经压缩
Keywords 模型压缩 · 视觉语言模型 · 边缘计算 · 动态推理 · token融合
Summary 本文提出了两种面向边缘设备的视觉-语言模型压缩方法：稀疏时序Token融合（STTF）和自适应神经压缩（ANC）。STTF通过事件驱动的变更检测动态复用视觉token，ANC则通过学习路由选择性激活编码器分支。在COCO和DVS128等数据集上验证了方法在保持性能的同时大幅降低计算量和延迟。适用于资源受限场景下的实时多模态推理。
Reason 该论文提出两种模型压缩技术（STTF和ANC），通过动态token复用和条件编码器激活实现高效推理，主要面向边缘设备上的视觉-语言任务。虽然其核心思想——动态减少视觉token数量、根据输入复杂度自适应计算——对文档图像理解中高分辨率处理和效率优化有一定启发，尤其STTF的事件驱动token精简机制可能迁移到文档场景以降低长文档或多页文档的冗余计算，但整体工作基于通用VQA和事件相机数据，并未针对文档结构、OCR对齐或密集文本等DIU核心问题设计，迁移需较大改造。
Abstract The demand for edge AI in vision-language tasks requires models that achieve real-time performance on resource-constrained devices with limited power and memory. This paper proposes two adaptive compression techniques -- Sparse Temporal Token Fusion (STTF) and Adaptive Neural Compression (ANC) -- that integrate algorithmic innovations with hardware-aware optimizations. Unlike previous approaches relying on static pruning or uniform scaling, STTF dynamically reuses visual tokens through event-driven change detection, while ANC conditionally activates encoder branches via a learned router, enabling fine-grained adaptation to scene complexity. Our 3B-parameter TinyGPT-STTF achieves CIDEr 131.2, BLEU-4 0.38, METEOR 0.31, and ROUGE-L 0.56 on the COCO 2017 test set, surpassing LLaVA-1.5 7B by 17.6 CIDEr points while using 2.3x fewer parameters and 62x fewer on-device FLOPs. TinyGPT-ANC reaches CIDEr 128.5. On event-based vision tasks, STTF reduces average token count by 84% (from 196 to 31 tokens) while preserving 95.6% accuracy on the DVS128 Gesture dataset, and ANC cuts FLOPs by up to 90% in low-motion scenes. Compared to strong baselines, our models improve accuracy by up to 4.4% and reduce latency by up to 13x. These results enable efficient deployment of capable vision-language models on real-world edge devices.
Authors Md Tasnin Tanvir, Soumitra Das, Sk Md Abidar Rahaman, Ali Shiri Sichani
Comment 9 pages, 6 figures
Categories Computer Vision and Pattern Recognition
6
NAF: Zero-Shot Feature Upsampling via Neighborhood Attention Filtering
Title CN NAF：通过邻域注意力过滤实现零样本特征上采样
Keywords 特征上采样 · 零样本 · 视觉基础模型 · 邻域注意力 · 高分辨率
Summary NAF是一种零样本、模型无关的特征上采样方法，通过引入跨尺度邻域注意力和旋转位置编码，仅依赖高分辨率输入图像即可自适应地恢复被下采样的视觉特征。它在无需微调的情况下适配任意视觉基础模型，兼顾高效性与精度，在语义分割、边缘检测和图像修复等多个任务上达到SOTA。该方法为高分辨率视觉理解提供了通用的上游工具。
Reason 该论文提出了一种VFM无关的零样本特征上采样方法NAF，利用邻域注意力和RoPE实现跨尺度特征恢复，无需针对特定视觉基础模型重新训练。虽然其主要应用场景是像素级任务（如分割、图像修复），但其高分辨率特征重建能力（支持2K分辨率，18 FPS）对文档图像理解中的细粒度文本定位和复杂排版解析具有潜在迁移价值。例如，可将NAF用于增强OCR-free VLM的输入视觉表征，缓解因特征下采样导致的小字模糊或结构失真问题。然而，论文未直接涉及多模态推理、语义理解或文档布局建模，属于通用视觉工具型工作，迁移到DIU需额外验证。
Abstract Vision Foundation Models (VFMs) extract spatially downsampled representations, posing challenges for pixel-level tasks. Existing upsampling approaches face a fundamental trade-off: classical filters are fast and broadly applicable but rely on fixed forms, while modern upsamplers achieve superior accuracy through learnable, VFM-specific forms at the cost of retraining for each VFM. We introduce Neighborhood Attention Filtering (NAF), which bridges this gap by learning adaptive spatial-and-content weights through Cross-Scale Neighborhood Attention and Rotary Position Embeddings (RoPE), guided solely by the high-resolution input image. NAF operates zero-shot: it upsamples features from any VFM without retraining, making it the first VFM-agnostic architecture to outperform VFM-specific upsamplers and achieve state-of-the-art performance across multiple downstream tasks. It maintains high efficiency, scaling to 2K feature maps and reconstructing intermediate-resolution maps at 18 FPS. Beyond feature upsampling, NAF demonstrates strong performance on image restoration, highlighting its versatility. Code and checkpoints are available at https://github.com/valeoai/NAF.
Authors Loick Chambon, Paul Couairon, Eloi Zablocki, Alexandre Boulch, Nicolas Thome, Matthieu Cord
Comment Code: https://github.com/valeoai/NAF
Categories Computer Vision and Pattern Recognition
6
DiVE-k: Differential Visual Reasoning for Fine-grained Image Recognition
Title CN DiVE-k：用于细粒度图像识别的差分视觉推理
Keywords 细粒度图像识别 · 差分推理 · 强化学习 · 视觉语言模型 · top-k生成
Summary 本文提出DiVE-k，一种基于top-k生成结果构建多选题并使用强化学习训练LVLM进行细粒度视觉识别的方法。通过要求模型在多个合理选项中选择正确答案，促使模型发展出差异化的推理能力，减少对训练集的死记硬背，增强对新类别的泛化能力。在五个细粒度识别数据集上显著优于现有方法。
Reason 该论文提出通过模型自身的top-k输出构建多选题，利用强化学习训练模型进行细粒度区分推理，从而提升在视觉相似类别间的判别能力。虽然其任务聚焦于自然图像的细粒度识别（如鸟类、飞机型号），而非文档图像理解（DIU），但其核心思想——通过生成多个候选并引导模型进行对比式推理以抑制幻觉和提升判别力——具备迁移到DocVQA场景的潜力。例如，在处理易混淆的表格结构或密集文本字段时，可构造候选答案集并引入差分推理机制来缓解VLM的OCR幻觉或误匹配问题。然而，该方法尚未在文档领域验证，且未针对文档特有的排版、逻辑结构建模，迁移路径需额外设计，故为中等偏弱价值。
Abstract Large Vision Language Models (LVLMs) possess extensive text knowledge but struggles to utilize this knowledge for fine-grained image recognition, often failing to differentiate between visually similar categories. Existing fine-tuning methods using Reinforcement Learning (RL) with exact-match reward signals are often brittle, encourage memorization of training categories, and fail to elicit differential reasoning needed for generalization to unseen classes. To address this, we propose $\textbf{DiVE-k}$, $\textbf{Di}$fferential $\textbf{V}$isual r$\textbf{E}$asoning using top-$\textbf{k}$ generations, framework that leverages model's own top-k predictions as a training signal. For each training image, DiVE-k creates a multiple-choice question from the model's top-k outputs and uses RL to train the model to select the correct answer. This approach requires the model to perform fine-grained differential reasoning among plausible options and provides a simple, verifiable reward signal that mitigates memorization and improves generalization. Experiments on five standard fine-grained datasets show that our method significantly outperforms existing approaches. In the standard base-to-novel generalization setting, DiVE-k surpasses the QWEN2.5-VL-7B and ViRFT by 10.04% and 6.16% on the Harmonic Mean metric, respectively. Further experiments show similar gains in mixed-domain and few-shot scenarios.
Authors Raja Kumar, Arka Sadhu, Ram Nevatia
Categories Computer Vision and Pattern Recognition
6
Beyond Token Length: Step Pruner for Efficient and Accurate Reasoning in Large Language Models
Title CN 超越Token长度：用于大语言模型高效准确推理的步骤剪枝器
Keywords 大推理模型 · 推理效率 · 强化学习 · 过思考 · 步骤剪枝
Summary 本文提出了Step Pruner（SP），一种基于强化学习的推理优化框架，旨在减少大推理模型中的“过度思考”现象。SP通过引入步长感知的奖励机制，在保证正确性的前提下惩罚冗余推理步骤，并采用动态停止策略防止模型通过合并步骤来‘作弊’。实验表明，SP在多个复杂推理任务上实现了更短响应长度与更高准确率的结合，例如在AIME24上减少了69.7%的token使用量。
Reason 该论文提出了一种基于强化学习的Step Pruner框架，通过设计步长感知的奖励函数来压缩冗余的推理步骤，同时避免模型因过度追求简洁而产生错误。虽然其核心任务并非文档图像理解（DIU），但其所提出的推理效率优化和动态停止机制，可能被迁移用于长文档多步推理场景中，例如在DocVQA中减少因上下文过长导致的信息迷失或重复推理。然而，该方法目前聚焦纯文本推理，未涉及视觉输入或跨模态对齐问题，因此对DIU的直接迁移价值有限。
Abstract Large Reasoning Models (LRMs) demonstrate strong performance on complex tasks but often suffer from excessive verbosity, known as "overthinking." Existing solutions via reinforcement learning (RL) typically penalize generated tokens to promote conciseness. However, these methods encounter two challenges: responses with fewer tokens do not always correspond to fewer reasoning steps, and models may develop hacking behavior in later stages of training by discarding reasoning steps to minimize token usage. In this work, we introduce \textbf{Step Pruner (SP)}, an RL framework that steers LRMs toward more efficient reasoning by favoring compact reasoning steps. Our step-aware reward function prioritizes correctness while imposing penalties for redundant steps, and withholds rewards for incorrect responses to prevent the reinforcement of erroneous reasoning. Moreover, we propose a dynamic stopping mechanism: when the model's output no longer shortens, training is halted to prevent hacking behavior caused by the merging of steps. Extensive experiments across four reasoning benchmarks demonstrate that SP achieves state-of-the-art accuracy while significantly reducing response length. For instance, on AIME24, SP reduces token usage by \textbf{69.7\%}.
Authors Canhui Wu, Qiong Cao, Chang Li, Zhenfang Wang, Chao Xue, Yuwei Fan, Wei Xi, Xiaodong He
Comment 21pages, 9 figures
Categories Artificial Intelligence, Computation and Language
6
Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
Title CN 超越语义：无意义中间标记的不合理有效性
Keywords Chain of Thought · 推理机制 · 语言模型行为 · 可解释性 · 训练数据有效性
Summary 本文通过控制实验发现，训练语言模型使用的推理轨迹中，中间步骤是否合理对最终性能影响不大，甚至损坏的推理链也能达到相似或更好的泛化效果。同时，推理长度与问题复杂度无关，且强化学习后训练不能提升推理有效性。结果挑战了将CoT视为真实推理过程的假设。
Reason 该论文研究了推理模型中‘思维链’（CoT）的中间标记是否具有实际语义作用，发现即使中间步骤完全无关或被破坏，模型仍能保持性能甚至在OOD任务上表现更好。这表明CoT的中间表示可能并不具备人类可解释的推理结构。虽然该工作属于上游基础研究，不直接针对文档理解任务，但其对‘推理痕迹有效性’的质疑对DIU领域有潜在启示：在DocVQA等任务中依赖LLM生成的CoT进行逻辑推理时，需警惕其表面连贯性背后的幻觉风险。该方法论可迁移至分析和干预文档问答中的错误推理路径。
Abstract Recent impressive results from large reasoning models have been interpreted as a triumph of Chain of Thought (CoT), especially of training on CoTs sampled from base LLMs to help find new reasoning patterns. While these traces certainly seem to help model performance, it is not clear how they actually influence it, with some works ascribing semantics to the traces and others cautioning against relying on them as transparent and faithful proxies of the model's internal computational process. To systematically investigate the role of end-user semantics of derivational traces, we set up a controlled study where we train transformer models from scratch on formally verifiable reasoning traces and the solutions they lead to. We notice that, despite significant gains over the solution-only baseline, models trained on entirely correct traces can still produce invalid reasoning traces even when arriving at correct solutions. More interestingly, our experiments also show that models trained on corrupted traces, whose intermediate reasoning steps bear no relation to the problem they accompany, perform similarly to those trained on correct ones, and even generalize better on out-of-distribution tasks. We also study the effect of GRPO-based RL post-training on trace validity, noting that while solution accuracy increase, this is not accompanied by any improvements in trace validity. Finally, we examine whether reasoning-trace length reflects inference-time scaling and find that trace length is largely agnostic to the underlying computational complexity of the problem being solved. These results challenge the assumption that intermediate tokens or ``Chains of Thought'' reflect or induce predictable reasoning behaviors and caution against anthropomorphizing such outputs or over-interpreting them (despite their mostly seemingly forms) as evidence of human-like or algorithmic behaviors in language models.
Authors Karthik Valmeekam, Kaya Stechly, Vardhan Palod, Atharva Gundawar, Subbarao Kambhampati
Categories Artificial Intelligence
6
PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection
Title CN PRISM-Bench：一个基于谜题的视觉任务基准，支持链式思维中的错误检测
Keywords 视觉推理 · 链式思维 · 错误检测 · 多模态基准 · 逻辑一致性
Summary PRISM-Bench是一个新型视觉推理基准，专注于评估多模态大模型在解决视觉谜题时的链式思维推理质量。它不仅关注最终答案是否正确，更关键的是要求模型识别出推理过程中第一个出现逻辑错误的步骤。该基准强调逻辑一致性与错误检测能力，揭示了当前MLLMs在流畅生成与可靠推理之间的显著差距，呼吁建立更精细的诊断性评估体系。
Reason 该论文提出了一种用于评估多模态大模型推理一致性和错误检测能力的诊断型基准，通过要求模型在包含单个逻辑错误的链式思维（CoT）中定位首个错误步骤，实现对视觉推理过程的细粒度分析。虽然其任务设定基于抽象视觉谜题而非真实文档，但其核心思想——分离答案生成与推理验证、引入诊断性评估机制——可迁移至文档图像理解领域。例如，在DocVQA中，模型常因OCR幻觉或布局误解产生看似合理但实际错误的推理路径，PRISM-Bench所倡导的错误检测范式可用于构建更具挑战性的DIU评估协议，推动模型发展出更鲁棒的逻辑验证能力。
Abstract Multimodal large language models (MLLMs) have achieved remarkable progress on vision-language tasks, yet their reasoning processes remain sometimes unreliable. We introduce PRISM-Bench, a benchmark of puzzle-based visual challenges designed to evaluate not only whether models can solve problems, but how their reasoning unfolds. Unlike prior evaluations that measure only final-answer accuracy, PRISM-Bench introduces a diagnostic task: given a visual puzzle and a step-by-step chain-of-thought (CoT) containing exactly one error, models must identify the first incorrect step. This setting enables fine-grained assessment of logical consistency, error detection, and visual reasoning. The puzzles in PRISM-Bench require multi-step symbolic, geometric, and analogical reasoning, resisting shortcuts based on superficial pattern matching. Evaluations across state-of-the-art MLLMs reveal a persistent gap between fluent generation and faithful reasoning: models that produce plausible CoTs often fail to locate simple logical faults. By disentangling answer generation from reasoning verification, PRISM-Bench offers a sharper lens on multimodal reasoning competence and underscores the need for diagnostic evaluation protocols in the development of trustworthy MLLMs.
Authors Yusu Qian, Cheng Wan, Chao Jia, Yinfei Yang, Qingyu Zhao, Zhe Gan
Categories Computer Vision and Pattern Recognition
6
DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search EMNLP 2025 Main
Title CN DCIS：通过分而治之的缩放因子搜索实现LLM的高效长度外推
Keywords 上下文扩展 · RoPE · 缩放因子搜索 · 高效微调 · 长序列建模
Summary 本文提出DCIS方法，通过分而治之的策略高效搜索RoPE中的最优缩放因子，从而在不增加微调成本的前提下实现LLM上下文长度的有效外推。实验表明该方法不仅提升了长上下文下的性能稳定性，还具备无需微调即可工作的潜力，同时搜索效率是现有方法的两倍。
Reason 该论文提出了一种针对RoPE缩放因子的高效搜索算法DCIS，用于扩展LLM的上下文长度，降低长上下文微调成本。虽然其核心任务是纯文本语言模型的上下文扩展，不直接处理文档图像或多模态输入，但其所解决的‘长序列建模’问题与文档理解中的长文档推理（如多页报表、合同理解）存在潜在关联。若将VLM的文本解码器部分替换为支持长上下文的LLM，该方法可间接支持更长的输出结构化内容生成。然而，该工作未涉及视觉输入处理、多模态对齐或OCR相关幻觉问题，迁移需依赖外部架构整合，故仅为弱相关。
Abstract Large language models (LLMs) based on the Transformer architecture usually have their context length limited due to the high training cost. Recent advancements extend the context window by adjusting the scaling factors of RoPE and fine-tuning. However, suboptimal initialization of these factors results in increased fine-tuning costs and reduced performance at target length. To address these challenges, we propose a novel RoPE-based fine-tuning framework that diverges from conventional scaling factors search. Specifically, we present a \textbf{D}ivide-and-\textbf{C}onquer \textbf{I}ncremental \textbf{S}earch (DCIS) algorithm that strategically determines the better scaling factors. Further fine-tuning with the identified scaling factors effectively extends the context window of LLMs. Empirical results demonstrate that our methodology not only mitigates performance decay at extended target lengths but also allows the model to fine-tune on short contexts and generalize to long contexts, thereby reducing the cost of fine-tuning. The scaling factors obtained through DCIS can even perform effectively without fine-tuning. Further analysis of the search space reveals that DCIS achieves twice the search efficiency compared to other methods. We also examine the impact of the non-strictly increasing scaling factors utilized in DCIS and evaluate the general capabilities of LLMs across various context lengths.
Authors Lei Yang, Shaoyang Xu, Jianxiang Peng, Shaolin Zhu, Deyi Xiong
Comment EMNLP 2025 Main
Categories Computation and Language
6
Hallucinate Less by Thinking More: Aspect-Based Causal Abstention for Large Language Models AAAI 2026
Title CN 通过更多思考减少幻觉：面向大语言模型的基于方面的因果放弃框架
Keywords 幻觉抑制 · 因果推理 · 知识放弃 · 大语言模型 · 可靠性
Summary 本文提出Aspect-Based Causal Abstention（ABCA），一种通过因果推理分析大语言模型内部知识多样性的幻觉抑制框架。它根据知识在不同语义方面的一致性与否，实现提前放弃回答，有效区分知识冲突与知识不足两种情况，在标准基准上取得了当前最优的放弃可靠性表现。
Reason 该论文提出了一种基于因果推理的早期放弃机制ABCA，通过分析LLM内部知识在不同‘方面’（如学科、法律背景、时间框架）上的多样性来判断回答的可靠性，并据此决定是否放弃回答。虽然其主要针对纯文本LLM的幻觉问题，但其核心思想——利用因果干预识别知识冲突或不足——可迁移至文档图像理解中的VLM幻觉问题。例如，在DocVQA中，当视觉与文本模态信息存在冲突，或OCR提取内容在多个语义方面不一致时，该方法可用于构建更可靠的放弃机制，避免生成错误答案。然而，该工作尚未涉及多模态输入，迁移需额外设计跨模态的‘方面’划分与因果建模，因此为弱相关上游方法。
Abstract Large Language Models (LLMs) often produce fluent but factually incorrect responses, a phenomenon known as hallucination. Abstention, where the model chooses not to answer and instead outputs phrases such as "I don't know", is a common safeguard. However, existing abstention methods typically rely on post-generation signals, such as generation variations or feedback, which limits their ability to prevent unreliable responses in advance. In this paper, we introduce Aspect-Based Causal Abstention (ABCA), a new framework that enables early abstention by analysing the internal diversity of LLM knowledge through causal inference. This diversity reflects the multifaceted nature of parametric knowledge acquired from various sources, representing diverse aspects such as disciplines, legal contexts, or temporal frames. ABCA estimates causal effects conditioned on these aspects to assess the reliability of knowledge relevant to a given query. Based on these estimates, we enable two types of abstention: Type-1, where aspect effects are inconsistent (knowledge conflict), and Type-2, where aspect effects consistently support abstention (knowledge insufficiency). Experiments on standard benchmarks demonstrate that ABCA improves abstention reliability, achieves state-of-the-art performance, and enhances the interpretability of abstention decisions.
Authors Vy Nguyen, Ziqi Xu, Jeffrey Chan, Estrid He, Feng Xia, Xiuzhen Zhang
Comment Accepted to AAAI 2026 (Main Technical Track)
Categories Artificial Intelligence, Computation and Language
6
Deep Improvement Supervision
Title CN 深度改进监督
Keywords 递归模型 · 推理效率 · 隐式策略改进 · 小型模型 · ARC
Summary 本文将小型递归模型（TRM）的潜在推理过程建模为一种无需分类器指导的生成过程，并提出一种新的训练方案，在每个推理循环中显式提供学习目标。该方法显著提升了训练效率，减少18倍前向传播次数，并移除了复杂的停机机制，同时在ARC-1任务上以仅0.8M参数达到24%准确率，表现优于多数大语言模型。
Reason 该论文提出了一种针对小型递归模型（TRM）的训练机制，通过在每一轮循环中引入目标信号来提升推理效率和性能。虽然其核心任务是抽象推理（ARC），并非直接面向文档图像理解（DIU），但其‘隐式策略改进’和‘无停机机制’的设计可能为长文档多步推理中的稳定性与效率优化提供思路。例如，在处理长文档VQA时，现有Agent式方法常依赖动态终止机制进行多轮推理，而本文的方法或可简化该过程，提升推理一致性。然而，迁移路径尚不直接，需额外设计适配机制。
Abstract Recently, it was shown that small, looped architectures, such as Tiny Recursive Models (TRMs), can outperform Large Language Models (LLMs) on complex reasoning tasks, including the Abstraction and Reasoning Corpus (ARC). In this work, we investigate a core question: how can we further improve the efficiency of these methods with minimal changes? To address this, we frame the latent reasoning of TRMs as a form of classifier-free guidance and implicit policy improvement algorithm. Building on these insights, we propose a novel training scheme that provides a target for each loop during training. We demonstrate that our approach significantly enhances training efficiency. Our method reduces the total number of forward passes by 18x and eliminates halting mechanisms, while maintaining quality comparable to standard TRMs. Notably, we achieve 24% accuracy on ARC-1 with only 0.8M parameters, outperforming most LLMs.
Authors Arip Asadulaev, Rayan Banerjee, Fakhri Karray, Martin Takac
Categories Artificial Intelligence, Computation and Language
6
Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models
Title CN 缩小智能：探索小型多模态模型中的感知与推理瓶颈
Keywords 多模态模型 · 模型压缩 · 视觉感知 · 小模型 · Extract+Think
Summary 本文系统分析了多模态模型在缩小规模过程中视觉理解与推理能力的退化现象，发现语言模型容量减少会显著削弱视觉感知能力，甚至超过对推理的影响。为解决此问题，作者提出Extract+Think方法：先通过视觉提取微调让模型稳定提取任务相关视觉细节，再进行逐步推理。该方法在保持高效的同时提升了小模型的视觉理解性能。
Reason 该论文研究了在缩小多模态模型规模时，视觉感知与推理能力的退化问题，并提出Extract+Think方法，通过显式训练模型提取指令相关的视觉细节来缓解性能下降。虽然其核心关注点是小型化模型的效率问题，但所揭示的‘LLM容量缩减会不成比例损害视觉感知’这一现象对文档图像理解（DIU）有警示意义。更重要的是，其提出的‘视觉提取微调’策略可被迁移用于提升OCR-free DocVQA系统在低资源下的细粒度视觉信息捕捉能力，尤其适用于移动端或边缘端文档理解场景。
Abstract Scaling up multimodal models has enabled remarkable advances in visual understanding and reasoning, but practical demands call for smaller, efficient systems. In this work, we conduct a principled analysis of downscaling intelligence in multimodal models, examining how reduced large language model (LLM) capacity affects multimodal capabilities. Our initial findings reveal an interesting trend: LLM downscaling disproportionately affects visual capabilities, rather than abilities inherited from the LLM. We then examine whether this drop mainly reflects the expected decline in visual reasoning or a more fundamental loss of perceptual abilities. Isolating the effect of LLM downscaling on perception, we find performance still drops sharply, often matching or exceeding the impact on reasoning. To address this bottleneck, we introduce visual extraction tuning, which explicitly trains the model to extract instruction-relevant visual details consistently across tasks. With these extracted visual details, we then apply step-by-step reasoning to generate answers. Together, these components form our Extract+Think approach, setting a new standard for efficiency and performance in this space.
Authors Mark Endo, Serena Yeung-Levy
Comment Website at https://web.stanford.edu/~markendo/projects/downscaling_intelligence
Categories Computer Vision and Pattern Recognition
6
DSeq-JEPA: Discriminative Sequential Joint-Embedding Predictive Architecture
Title CN DSeq-JEPA：判别式序列联合嵌入预测架构
Keywords 自监督学习 · 视觉表示学习 · 序列预测 · 显著性检测 · Transformer
Summary DSeq-JEPA是一种新型自监督视觉模型架构，通过引入显著性感知的判别区域排序机制，在I-JEPA基础上实现从主到次的序列化预测，融合了JEPA的全局上下文建模与GPT式的顺序推理。实验表明其在分类、检测、分割和简单推理任务上优于传统JEPA方法，强调学习更具判别性的特征表示。
Reason 该工作属于通用视觉表示学习，提出通过显著性图识别重要区域并按判别性顺序进行序列化预测，以提升自监督学习的表征质量。虽然其核心思想——从关键区域出发、逐步推理次要区域——与文档图像理解中‘从标题/字段标签到细节数值’的阅读顺序有潜在契合点，且其渐进式语义建模可能有助于缓解长文档或多区域间的上下文迷失问题，但论文本身未涉及文本布局、OCR信号或任何DIU任务验证。迁移需额外设计适配机制（如结合文本位置编码），直接应用难度较大。
Abstract Image-based Joint-Embedding Predictive Architecture (I-JEPA) learns visual representations by predicting latent embeddings of masked regions from visible context. However, it treats all regions uniformly and independently, lacking an explicit notion of where or in what order predictions should be made. Inspired by human visual perception, which deploys attention selectively and sequentially from the most informative to secondary regions, we propose DSeq-JEPA, a Discriminative Sequential Joint-Embedding Predictive Architecture that bridges predictive and autoregressive self-supervised learning, integrating JEPA-style latent prediction with GPT-style sequential reasoning. Specifically, DSeq-JEPA (i) first identifies primary discriminative regions based on a transformer-derived saliency map, emphasizing the distribution of visual importance, and then (ii) predicts subsequent regions in this discriminative order, progressively forming a curriculum-like semantic progression from primary to secondary cues -- a form of GPT-style pre-training. Extensive experiments across diverse tasks, including image classification (ImageNet), fine-grained visual categorization (iNaturalist21, CUB-200-2011, Stanford-Cars), detection and segmentation (MS-COCO, ADE20K), and low-level reasoning tasks (Clevr/Count, Clevr/Dist), demonstrate that DSeq-JEPA consistently focuses on more discriminative and generalizable representations than I-JEPA variants. Project page: https://github.com/SkyShunsuke/DSeq-JEPA.
Authors Xiangteng He, Shunsuke Sakai, Kun Yuan, Nicolas Padoy, Tatsuhito Hasegawa, Leonid Sigal
Comment Project page: https://github.com/SkyShunsuke/DSeq-JEPA
Categories Computer Vision and Pattern Recognition
6
EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewards
Title CN EvoLMM：通过连续奖励实现自我进化的大型多模态模型
Keywords 自演化 · 多模态大模型 · 无监督学习 · 内部一致性 · 自我奖励
Summary 本文提出EvoLMM，一种基于单一骨干模型构建Proposer和Solver两个协作代理的自演化框架，利用原始图像自动生成图文问题并进行内部一致性推理，通过连续自我奖励机制提升多模态数学推理能力。在无标注数据条件下，在ChartQA、MathVista等基准上取得约3%的性能提升。方法强调完全无监督的自我进化，为LMM训练提供新思路。
Reason 该论文提出了一种无需标注数据的自演化多模态模型框架，通过Proposer-Solver双代理机制实现内部一致性推理与自我奖励。虽然其在ChartQA等图表理解任务上进行了验证，但核心创新在于训练范式而非针对文档图像理解（DIU）中的具体痛点（如OCR幻觉、复杂排版处理）进行改进。其自演化机制作为一种上游方法，理论上可迁移用于长文档多步推理中减少对外部标注依赖的问题，但目前未展示对VIE、DocVQA等典型DIU任务的直接增益，迁移路径尚不明确。
Abstract Recent advances in large multimodal models (LMMs) have enabled impressive reasoning and perception abilities, yet most existing training pipelines still depend on human-curated data or externally verified reward models, limiting their autonomy and scalability. In this work, we strive to improve LMM reasoning capabilities in a purely unsupervised fashion (without any annotated data or reward distillation). To this end, we propose a self-evolving framework, named EvoLMM, that instantiates two cooperative agents from a single backbone model: a Proposer, which generates diverse, image-grounded questions, and a Solver, which solves them through internal consistency, where learning proceeds through a continuous self-rewarding process. This dynamic feedback encourages both the generation of informative queries and the refinement of structured reasoning without relying on ground-truth or human judgments. When using the popular Qwen2.5-VL as the base model, our EvoLMM yields consistent gains upto $\sim$3\% on multimodal math-reasoning benchmarks, including ChartQA, MathVista, and MathVision, using only raw training images. We hope our simple yet effective approach will serve as a solid baseline easing future research in self-improving LMMs in a fully-unsupervised fashion. Our code and models are available at https://github.com/mbzuai-oryx/EvoLMM.
Authors Omkar Thawakar, Shravan Venkatraman, Ritesh Thawkar, Abdelrahman Shaker, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Fahad Khan
Comment 9 Pages, 6 Figures, 4 Tables
Categories Computer Vision and Pattern Recognition
6
Concise Reasoning via Reinforcement Learning
Title CN 通过强化学习实现简洁推理
Keywords 强化学习 · 推理效率 · 响应简洁性 · PPO · GRPO
Summary 论文分析了PPO和GRPO等强化学习算法在推理模型训练中导致输出冗长的内在机制，发现错误回答会系统性诱导策略生成更长文本。作者提出一种两阶段强化学习方案，在少量可解问题上进行精炼训练，有效缩短响应长度并保持甚至提升准确性。实验验证了简洁性与正确性的正相关关系，并揭示了GRPO存在的崩溃模式问题。
Reason 该论文研究了推理模型在强化学习训练中产生冗长输出的机制，指出错误答案驱动策略趋向于生成更长响应，是一种优化副作用。其提出的两阶段RL方法在少量可解问题上微调以提升简洁性，对降低VLM在文档理解任务中的推理开销有一定启发意义。虽然不直接针对DIU，但若将DocVQA中的多步推理路径视为序列决策过程，该方法可用于约束幻觉性或冗余的中间推理步骤，从而缓解OCR幻觉引发的错误传播。然而，迁移需解决视觉-语言联合空间中的动作空间设计和奖励建模问题，技术路径尚不明确。
Abstract A major drawback of reasoning models is their excessive token usage, inflating computational cost, resource demand, and latency. We show this verbosity stems not from deeper reasoning but from reinforcement learning loss minimization when models produce incorrect answers. With unsolvable problems dominating training, this effect compounds into a systematic tendency toward longer outputs. Through theoretical analysis of PPO and GRPO, we prove that incorrect answers inherently drive policies toward verbosity \textit{even when} $γ=1$, reframing response lengthening as an optimization artifact. We further uncover a consistent correlation between conciseness and correctness across reasoning and non-reasoning models. Building on these insights, we propose a two-phase RL procedure where a brief secondary stage, trained on a small set of solvable problems, significantly reduces response length while preserving or improving accuracy. Finally, we show that while GRPO shares properties with PPO, it exhibits collapse modes, limiting its reliability for concise reasoning. Our claims are supported by extensive experiments.
Authors Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, Kartik Talamadupula
Categories Computation and Language
5
Lost in Serialization: Invariance and Generalization of LLM Graph Reasoners AAAI 2026 Workshop on Graphs and more Complex Structures For Learning and Reasoning (GCLR)
Title CN 迷失在序列化中：LLM图推理器的不变性与泛化能力
Keywords 图推理 · 序列化敏感性 · 大语言模型 · 不变性 · 泛化
Summary 论文系统分析了大语言模型在处理图结构的不同序列化形式时的表现，发现即使是对称等价的表示，LLM也可能输出不一致的结果。作者分解了序列化的三个因素（节点标注、边编码和语法格式），并通过实验评估了模型对这些变化的敏感性。结果显示，较大的基础模型更具鲁棒性，而微调可能降低对结构和格式变化的稳定性，且不一定提升在未见任务上的泛化能力。
Reason 该论文研究了基于大语言模型的图推理器在面对图结构不同序列化方式时的鲁棒性问题，如节点重编号、边顺序变化等。虽然其关注的是图数据的表示敏感性，但这一问题与文档图像理解中的布局结构解析存在弱关联——例如，在处理表格或复杂版式时，若将文档结构建模为图（如文本块之间的关系图），则该工作提示我们：当前VLM对输入序列化的敏感性可能导致推理结果不稳定。然而，该研究本身并未涉及视觉输入、多模态建模或文档任务，且未提出可直接迁移至DIU的技术工具（如新的架构、训练策略或去幻觉机制），因此迁移价值有限。
Abstract While promising, graph reasoners based on Large Language Models (LLMs) lack built-in invariance to symmetries in graph representations. Operating on sequential graph serializations, LLMs can produce different outputs under node reindexing, edge reordering, or formatting changes, raising robustness concerns. We systematically analyze these effects, studying how fine-tuning impacts encoding sensitivity as well generalization on unseen tasks. We propose a principled decomposition of graph serializations into node labeling, edge encoding, and syntax, and evaluate LLM robustness to variations of each of these factors on a comprehensive benchmarking suite. We also contribute a novel set of spectral tasks to further assess generalization abilities of fine-tuned reasoners. Results show that larger (non-fine-tuned) models are more robust. Fine-tuning reduces sensitivity to node relabeling but may increase it to variations in structure and format, while it does not consistently improve performance on unseen tasks.
Authors Daniel Herbst, Lea Karbevska, Divyanshu Kumar, Akanksha Ahuja, Fatemeh Gholamzadeh Nasrabadi, Fabrizio Frasca
Comment AAAI 2026 Workshop on Graphs and more Complex Structures For Learning and Reasoning (GCLR). Version 2 fixes typos in author name and Figure 1
Categories Artificial Intelligence
5
SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition
Title CN SpatialBench：面向多模态大语言模型的空间认知评测基准
Keywords 空间认知 · 评测基准 · 多模态大模型 · 空间推理 · 分层评估
Summary 本文提出了一个名为SpatialBench的多模态大语言模型空间认知评测基准，通过五级分层框架系统评估模型在空间感知、定位、关系推理、因果推断与规划等方面的能力。实验表明当前MLLM在高层空间推理上仍显著弱于人类。该基准有助于揭示模型在空间理解上的短板。
Reason 该论文构建了一个分层的、细粒度的空间认知评测框架和基准，涵盖从基础感知到高级规划的15项任务。虽然其目标是评估MLLM在物理环境中的空间推理能力，但与文档图像理解（DIU）中的复杂排版理解、元素定位与逻辑关系建模有一定概念关联。例如，文档中的表格结构识别、字段对齐、阅读顺序推断可视为一种受限平面空间推理。然而，该工作本身是评测导向而非方法创新，未提出可用于提升DIU性能的新架构、训练策略或推理机制，迁移价值有限，属于弱相关上游工作。
Abstract Spatial cognition is fundamental to real-world multimodal intelligence, allowing models to effectively interact with the physical environment. While multimodal large language models (MLLMs) have made significant strides, existing benchmarks often oversimplify spatial cognition, reducing it to a single-dimensional metric, which fails to capture the hierarchical structure and interdependence of spatial abilities. To address this gap, we propose a hierarchical spatial cognition framework that decomposes spatial intelligence into five progressively complex levels from basic observation to high-level planning. Building upon this taxonomy, we construct SpatialBench, a large-scale, fine-grained benchmark covering 15 tasks aligned with these cognitive levels. To provide a unified evaluation across heterogeneous tasks, we further introduce a high-level capability-oriented metric that reliably assesses a model's overall spatial reasoning ability. Extensive experiments over massive MLLMs reveal distinct performance stratification across cognitive levels: models exhibit strong perceptual grounding yet remain limited in symbolic reasoning, causal inference, and planning. Additional human tests demonstrate that humans perform selective, goal-directed abstraction, while MLLMs tend to over-attend to surface details without coherent spatial intent. Our work establishes the first systematic framework for measuring hierarchical spatial cognition in MLLMs, laying the foundation for future spatially intelligent systems.
Authors Peiran Xu, Sudong Wang, Yao Zhu, Jianing Li, Yunjian Zhang
Categories Artificial Intelligence
5
Subjective Depth and Timescale Transformers: Learning Where and When to Compute
Title CN 主观深度与时间尺度变换器：学习何时何地进行计算
Keywords Transformer · 条件计算 · 贝叶斯惊奇 · 推理效率 · 动态路由
Summary 本文提出主观深度变换器（SDT）和主观时间尺度变换器（STT），利用贝叶斯惊奇信号在解码器-only Transformer中动态决定何时何地执行计算。SDT通过决策层与动态层交替结构实现块级计算路由，STT则进一步在时间维度预测残差更新以控制模块执行。两者均能在训练中展现出从新奇性驱动到预测性驱动的门控转变，在降低75%自注意力计算和50% KV缓存占用的同时保持性能，为高效模型提供新思路。
Reason 该论文提出通过贝叶斯惊奇信号动态路由计算的两种Transformer架构（SDT和STT），在推理过程中实现条件计算，减少自注意力计算量和KV缓存需求。虽然其核心动机是提升模型效率与可扩展性，但方法本身属于通用序列建模层面的架构改进，并未针对多模态或文档结构特性设计。尽管其‘动态跳过’机制理论上可用于长文档VLM中缓解长序列处理压力，但迁移路径不直接，且缺乏对视觉-语言对齐、细粒度定位或OCR幻觉等DIU核心痛点的支持证据，因此仅为弱相关。
Abstract The rigid, uniform allocation of computation in standard Transformer (TF) architectures can limit their efficiency and scalability, particularly for large-scale models and long sequences. Addressing this, we introduce Subjective Depth Transformers (SDT) and Subjective Timescale Transformers (STT), two distinct architectures that leverage Bayesian surprise signals to dynamically route computation, learning where and when to compute within decoder-only TFs. SDT augments a decoder-only stack with alternating Decision and Dynamic layers: a Decision layer computes a full block 'posterior' and a lightweight 'prior,' while a Dynamic layer employs fixed-capacity Top-K routing based on Bayesian surprise (Expected and Unexpected Change), maintaining a static compute graph. STT extends this conditional computation to the temporal domain: a transition network predicts residual updates, forming a temporal 'change hypothesis' that informs a router to dynamically execute or bypass TF blocks for each token, managing KV-cache contributions. Both architectures exhibit the predicted shift from novelty to prediction driven gating over training, suggesting alignment with surprise based principles. While operating at reduced capacity, they offer preliminary insights into the compute-accuracy trade-offs of conditional computation. The proposed architectures establish a flexible framework for efficiency, reducing self-attention computation by 75% and KV-cache requirements by 50% within each compute skipping layer, setting a pathway for more efficient models.
Authors Frederico Wieser, Martin Benfeghoul, Haitham Bou Ammar, Jun Wang, Zafeirios Fountas
Categories Artificial Intelligence, Computation and Language
5
The More, the Merrier: Contrastive Fusion for Higher-Order Multimodal Alignment
Title CN 越多越好：用于高阶多模态对齐的对比融合
Keywords 多模态学习 · 对比学习 · 高阶对齐 · 表示学习 · 模态融合
Summary 本文提出了Contrastive Fusion（ConFu），一种通过扩展对比学习目标以同时对齐单个模态和融合模态的多模态表示学习框架。该方法引入了融合模态对比项，能够捕捉XOR类的高阶依赖关系，同时保持良好的成对对齐性能。实验表明其在检索和分类任务上具有竞争力，并支持统一的一对一和二对一检索。然而，该工作未聚焦于文档理解场景，也未验证在文档相关任务中的有效性。
Reason 该论文提出了一种名为Contrastive Fusion（ConFu）的框架，旨在通过联合嵌入单个模态及其融合组合来实现更高阶的多模态对齐。虽然方法在多模态学习中有理论价值，但其核心关注的是通用多模态表示学习，未针对文档图像理解中的具体痛点（如OCR幻觉、复杂排版、细粒度定位等）进行设计或验证。尽管‘高阶对齐’概念在理论上可能有助于处理文档中文本、布局与视觉结构之间的复杂关系，但迁移路径不明确，且缺乏在DocVQA、表格识别或布局分析等任务上的实证支持，属于通用多模态方法，对DIU的直接贡献有限。
Abstract Learning joint representations across multiple modalities remains a central challenge in multimodal machine learning. Prevailing approaches predominantly operate in pairwise settings, aligning two modalities at a time. While some recent methods aim to capture higher-order interactions among multiple modalities, they often overlook or insufficiently preserve pairwise relationships, limiting their effectiveness on single-modality tasks. In this work, we introduce Contrastive Fusion (ConFu), a framework that jointly embeds both individual modalities and their fused combinations into a unified representation space, where modalities and their fused counterparts are aligned. ConFu extends traditional pairwise contrastive objectives with an additional fused-modality contrastive term, encouraging the joint embedding of modality pairs with a third modality. This formulation enables ConFu to capture higher-order dependencies, such as XOR-like relationships, that cannot be recovered through pairwise alignment alone, while still maintaining strong pairwise correspondence. We evaluate ConFu on synthetic and real-world multimodal benchmarks, assessing its ability to exploit cross-modal complementarity, capture higher-order dependencies, and scale with increasing multimodal complexity. Across these settings, ConFu demonstrates competitive performance on retrieval and classification tasks, while supporting unified one-to-one and two-to-one retrieval within a single contrastive framework.
Authors Stefanos Koutoupis, Michaela Areti Zervou, Konstantinos Kontras, Maarten De Vos, Panagiotis Tsakalides, Grigorios Tsagatakis
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
5
Think Visually, Reason Textually: Vision-Language Synergy in ARC
Title CN 视觉思考，文本推理：ARC中的视觉-语言协同
Keywords 视觉语言协同 · 抽象推理 · 模态分工 · 自纠错 · ARC-AGI
Summary 本文提出在抽象推理任务ARC-AGI中融合视觉与语言模态的协同框架，认为视觉擅长模式抽象与验证，语言擅长符号化规则执行。为此设计了视觉-语言协同推理（VLSR）和模态切换自纠错（MSSC）机制，在多个大模型上提升了4.33%的性能。
Reason 该论文研究的是ARC-AGI抽象推理任务，虽然涉及视觉与语言模态的协同，但其核心任务是基于网格图案的规则归纳，与文档图像理解（DIU）中的实际痛点如OCR幻觉、复杂排版、密集文本处理等关联较弱。尽管提出的VLSR和MSSC框架在模态分工与自我修正上有启发性，理论上或可迁移至长文档多步推理中的错误校正，但其验证场景远离真实文档结构，迁移路径不明确，对DIU的直接价值有限。
Abstract Abstract reasoning from minimal examples remains a core unsolved problem for frontier foundation models such as GPT-5 and Grok 4. These models still fail to infer structured transformation rules from a handful of examples, which is a key hallmark of human intelligence. The Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) provides a rigorous testbed for this capability, demanding conceptual rule induction and transfer to novel tasks. Most existing methods treat ARC-AGI as a purely textual reasoning task, overlooking the fact that humans rely heavily on visual abstraction when solving such puzzles. However, our pilot experiments reveal a paradox: naively rendering ARC-AGI grids as images degrades performance due to imprecise rule execution. This leads to our central hypothesis that vision and language possess complementary strengths across distinct reasoning stages: vision supports global pattern abstraction and verification, whereas language specializes in symbolic rule formulation and precise execution. Building on this insight, we introduce two synergistic strategies: (1) Vision-Language Synergy Reasoning (VLSR), which decomposes ARC-AGI into modality-aligned subtasks; and (2) Modality-Switch Self-Correction (MSSC), which leverages vision to verify text-based reasoning for intrinsic error correction. Extensive experiments demonstrate that our approach yields up to a 4.33\% improvement over text-only baselines across diverse flagship models and multiple ARC-AGI tasks. Our findings suggest that unifying visual abstraction with linguistic reasoning is a crucial step toward achieving generalizable, human-like intelligence in future foundation models. Source code is released at https://github.com/InternLM/ARC-VL.
Authors Beichen Zhang, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, Jiaqi Wang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
5
LightMem: Lightweight and Efficient Memory-Augmented Generation
Title CN LightMem：轻量高效的记忆增强生成
Keywords 记忆系统 · 大语言模型 · 效率优化 · 上下文压缩 · 对话系统
Summary LightMem提出一种轻量级记忆增强框架，将记忆分为感官记忆、短时记忆和长时记忆三个阶段，通过主题聚类与离线更新机制，在保持性能的同时大幅降低推理开销。实验显示其在减少token使用和API调用方面优势显著，适用于需要高效上下文管理的场景。
Reason 该论文提出了一种受认知科学启发的三阶段记忆系统LightMem，旨在提升LLM在动态环境中对历史交互信息的利用效率。虽然其在长上下文问答和对话任务中表现出显著的token和API调用节省，但整体工作聚焦于通用对话/问答场景下的记忆机制优化，并未针对文档图像理解中的核心挑战（如OCR幻觉、复杂版式建模、细粒度视觉定位或高分辨率输入处理）进行设计或验证。尽管‘记忆机制’理论上可用于长文档分步阅读与信息整合，但本文缺乏与文档结构、视觉布局或跨模态对齐相关的建模创新，迁移价值有限，属于DIU领域常规延伸而非方法论突破。
Abstract Despite their remarkable capabilities, Large Language Models (LLMs) struggle to effectively leverage historical interaction information in dynamic and complex environments. Memory systems enable LLMs to move beyond stateless interactions by introducing persistent information storage, retrieval, and utilization mechanisms. However, existing memory systems often introduce substantial time and computational overhead. To this end, we introduce a new memory system called LightMem, which strikes a balance between the performance and efficiency of memory systems. Inspired by the Atkinson-Shiffrin model of human memory, LightMem organizes memory into three complementary stages. First, cognition-inspired sensory memory rapidly filters irrelevant information through lightweight compression and groups information according to their topics. Next, topic-aware short-term memory consolidates these topic-based groups, organizing and summarizing content for more structured access. Finally, long-term memory with sleep-time update employs an offline procedure that decouples consolidation from online inference. On LongMemEval and LoCoMo, using GPT and Qwen backbones, LightMem consistently surpasses strong baselines, improving QA accuracy by up to 7.7% / 29.3%, reducing total token usage by up to 38x / 20.9x and API calls by up to 30x / 55.5x, while purely online test-time costs are even lower, achieving up to 106x / 117x token reduction and 159x / 310x fewer API calls. The code is available at https://github.com/zjunlp/LightMem.
Authors Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, Yuqi Tang, Ziwen Xu, Shumin Deng, Yunzhi Yao, Mengru Wang, Shuofei Qiao, Huajun Chen, Ningyu Zhang
Comment Work in progress
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
5
Mechanism of Task-oriented Information Removal in In-context Learning
Title CN 上下文学习中面向任务的信息移除机制
Keywords 上下文学习 · 信息移除 · 去噪机制 · 注意力头分析 · 推理干预
Summary 本文提出了一种解释上下文学习机制的新视角：任务导向的信息移除。研究表明，大语言模型在零样本情况下会生成包含所有可能任务信息的非选择性表示，导致输出混乱；而少样本示例能通过隐式的‘信息过滤’过程，去除冗余信息，聚焦目标任务。作者进一步发现特定的‘去噪注意力头’在此过程中起关键作用，并通过消融实验证实其必要性。
Reason 该论文研究了语言模型在上下文学习（ICL）中的信息过滤机制，提出了通过低秩滤波器从隐藏状态中选择性移除无关信息以实现任务定向推理的机制，并识别出执行该操作的关键注意力头（Denoising Heads）。虽然其核心是纯文本领域的机制分析，但其‘推理过程中主动去除干扰信息’的思想对文档图像理解中的幻觉问题具有潜在迁移价值——例如，在DocVQA中，模型常因视觉或语义噪声产生幻觉答案，若能在推理阶段设计干预机制，抑制与问题无关的布局或文本信息传播，可能缓解此类问题。然而，该工作未涉及多模态输入，也未针对视觉-语言对齐场景设计验证，迁移到DIU仍需大量适配工作，故仅为弱相关。
Abstract In-context Learning (ICL) is an emerging few-shot learning paradigm based on modern Language Models (LMs), yet its inner mechanism remains unclear. In this paper, we investigate the mechanism through a novel perspective of information removal. Specifically, we demonstrate that in the zero-shot scenario, LMs encode queries into non-selective representations in hidden states containing information for all possible tasks, leading to arbitrary outputs without focusing on the intended task, resulting in near-zero accuracy. Meanwhile, we find that selectively removing specific information from hidden states by a low-rank filter effectively steers LMs toward the intended task. Building on these findings, by measuring the hidden states on carefully designed metrics, we observe that few-shot ICL effectively simulates such task-oriented information removal processes, selectively removing the redundant information from entangled non-selective representations, and improving the output based on the demonstrations, which constitutes a key mechanism underlying ICL. Moreover, we identify essential attention heads inducing the removal operation, termed Denoising Heads, which enables the ablation experiments blocking the information removal operation from the inference, where the ICL accuracy significantly degrades, especially when the correct label is absent from the few-shot demonstrations, confirming both the critical role of the information removal mechanism and denoising heads.
Authors Hakaze Cho, Haolin Yang, Gouki Minegishi, Naoya Inoue
Comment 87 pages, 90 figures, 7 tables
Categories Artificial Intelligence, Computation and Language
5
From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
Title CN 从比特到轮次：面向扩散语言模型的带探索并行解码
Keywords 扩散语言模型 · 并行解码 · 信息瓶颈 · 解码效率 · ETE
Summary 本文指出传统扩散语言模型在解码时过度依赖高置信度token会导致信息瓶颈，进而拖慢生成速度。作者提出‘比特到轮次’理论，证明解码轮数受限于总信息量和每轮的信息预算，并设计了无需训练的Explore-Then-Exploit（ETE）策略，通过跨块解码和主动探索不确定token来激发后续高置信预测，从而减少解码轮次。实验表明ETE在保持质量的同时显著提升解码效率。
Reason 该论文研究扩散语言模型（DLMs）中的并行解码效率问题，提出通过探索高不确定性token来提升信息吞吐量的ETE解码策略。虽然其核心动机是加速生成过程，且理论分析具有普适性，但当前工作聚焦于通用文本生成，并未涉及视觉或文档模态。尽管‘信息高效解码’的思想在理论上可能对长文档多步推理中的路径优化有启发，但缺乏与视觉输入、布局结构或OCR联合建模的设计，迁移到文档图像理解（DIU）的接口不明确。属于上游方法中迁移难度较高的工作，不具备直接可操作性。
Abstract Diffusion Language Models (DLMs) have recently emerged as a strong alternative to autoregressive language models (LMs). DLMs offer comparable accuracy with faster inference speed via parallel decoding. However, standard DLM decoding strategies relying on high-confidence tokens encounter an inherent information-theoretic bottleneck that restricts decoding progress and ultimately slows generation. We demonstrate both theoretically and empirically that prioritizing high-confidence tokens is inherently inefficient. High-probability tokens carry negligible information and strictly relying on them limits the effective progress made in each decoding round. We prove that the number of decoding rounds must grow linearly with the sample's total information (negative log-likelihood) and inversely with the per-round information budget, establishing a bits-to-rounds principle. We also propose Explore-Then-Exploit (ETE), a training-free decoding strategy that maximizes information throughput and decoding efficiency. ETE combines cross-block decoding with targeted exploration of high-uncertainty tokens to reshape the conditional distribution and trigger cascades of confident predictions. Experiments verify our theoretical bounds and demonstrate that ETE consistently reduces the required number of decoding rounds compared to confidence-only baselines without compromising generation quality.
Authors Hengyu Fu, Baihe Huang, Virginia Adams, Charles Wang, Venkat Srinivasan, Jiantao Jiao
Comment 24 pages, 6 figures
Categories Artificial Intelligence
5
FANoise: Singular Value-Adaptive Noise Modulation for Robust Multimodal Representation Learning AAAI2026
Title CN FANoise：面向鲁棒多模态表示学习的奇异值自适应噪声调制
Keywords 多模态表示学习 · 噪声注入 · 对比学习 · 训练稳定性 · FANoise
Summary 本文提出FANoise，一种基于特征分布动态的自适应噪声注入方法，用于增强多模态表示学习的鲁棒性。通过分析InfoNCE损失下的梯度行为和特征空间演化，该方法在训练过程中根据奇异值动态调节噪声强度，从而在保留噪声正则化作用的同时减少负面影响。实验表明其在多种VLM基模型上均能稳定提升性能。
Reason 该论文提出一种特征自适应的噪声注入策略FANoise，用于提升多模态表示学习的鲁棒性，其核心创新在于根据对比学习过程中的特征分布动态调整噪声。虽然方法在多模态任务上验证有效，但并未针对文档图像理解（DIU）特有的挑战（如OCR幻觉、复杂版式建模、细粒度文本定位）进行设计或验证。其迁移潜力有限，因噪声调制属于通用训练稳定性优化技术，在DIU中可能仅带来边际增益，且缺乏对高分辨率视觉输入或长序列语言建模的支持机制。
Abstract Representation learning is fundamental to modern machine learning, powering applications such as text retrieval and multimodal understanding. However, learning robust and generalizable representations remains challenging. While prior work has demonstrated that active noise injection, a form of data augmentation, can enhance encoding performance, most existing methods rely on heuristic or static noise, overlooking the dynamic nature of feature distributions during training. In this work, we systematically study the role of noise in representation learning from both gradient-based and feature distribution perspectives, using InfoNCE loss as a representative example. Focusing on multimodal representation learning, we propose FANoise, a novel feature-adaptive noise injection strategy. By leveraging the dynamics of contrastive learning, FANoise effectively mitigates the negative impacts of noise while preserving its benefits. Under this theoretically grounded framework, comprehensive experiments demonstrate that FANoise consistently improves overall performance on multimodal tasks across various base VLM models.
Authors Jiaoyang Li, Jun Fang, Tianhao Gao, Xiaohui Zhang, Zhiyuan Liu, Chao Liu, Pengzhang Liu, Qixia Jiang
Comment 13 pages, 5 figures, accept to AAAI2026
Categories Artificial Intelligence
5
OuroMamba: A Data-Free Quantization Framework for Vision Mamba ICCV 2025
Title CN OuroMamba：一种用于视觉Mamba模型的无数据量化框架
Keywords Vision Mamba · 无数据量化 · 模型压缩 · 动态异常检测 · 合成数据生成
Summary 本文提出了OuroMamba，首个适用于视觉Mamba模型的无数据量化框架，包含两个阶段：通过潜在空间邻域交互生成语义丰富的合成数据（OuroMamba-Gen），以及结合轻量级动态异常检测的混合精度量化（OuroMamba-Quant）。实验表明其在多种量化设置下优于有数据PTQ方法，并实现了最高2.36倍的GPU加速。
Reason 该论文提出了一种针对Vision Mamba架构的无数据后训练量化方法，主要解决VMM模型在量化过程中因状态转移和动态异常激活导致的性能下降问题。虽然其核心贡献在于模型压缩与部署优化，且面向的是Mamba类视觉模型这一新兴架构，但并未直接涉及文档图像理解中的关键挑战（如OCR幻觉、复杂排版建模、细粒度文本定位或长文档推理）。其合成数据生成和动态异常检测机制虽有一定方法论价值，但迁移至DIU场景的路径不明确——例如无法直接缓解VLM在文档上的语义幻觉或提升对齐精度。
Abstract We present OuroMamba, the first data-free post-training quantization (DFQ) method for vision Mamba-based models (VMMs). We identify two key challenges in enabling DFQ for VMMs, (1) VMM's recurrent state transitions restricts capturing of long-range interactions and leads to semantically weak synthetic data, (2) VMM activations exhibit dynamic outlier variations across time-steps, rendering existing static PTQ techniques ineffective. To address these challenges, OuroMamba presents a two-stage framework: (1) OuroMamba-Gen to generate semantically rich and meaningful synthetic data. It applies contrastive learning on patch level VMM features generated through neighborhood interactions in the latent state space, (2) OuroMamba-Quant to employ mixed-precision quantization with lightweight dynamic outlier detection during inference. In specific, we present a thresholding based outlier channel selection strategy for activations that gets updated every time-step. Extensive experiments across vision and generative tasks show that our data-free OuroMamba surpasses existing data-driven PTQ techniques, achieving state-of-the-art performance across diverse quantization settings. Additionally, we implement efficient GPU kernels to achieve practical latency speedup of up to 2.36x. Code and synthetic dataset are available here: https://github.com/georgia-tech-synergy-lab/ICCV-OuroMamba
Authors Akshat Ramachandran, Mingyu Lee, Huan Xu, Souvik Kundu, Tushar Krishna
Comment Accepted to ICCV 2025
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
5
One-Step Diffusion-Based Image Compression with Semantic Distillation NeurIPS 2025
Title CN 基于单步扩散与语义蒸馏的图像压缩
Keywords 图像压缩 · 扩散模型 · 语义蒸馏 · 超先验 · 生成式编解码器
Summary 本文提出了OneDC，一种基于单步扩散的生成式图像压缩框架，结合潜在空间压缩模块与一步生成器，利用超先验作为语义信号并通过语义蒸馏机制从预训练生成式tokenizer中迁移知识，同时采用像素域与潜在域混合优化策略，在仅一步生成的情况下实现了SOTA的感知质量，解码速度快39%，码率降低20倍。项目主页：https://onedc-codec.github.io/
Reason 该论文提出了一种用于通用图像压缩的单步扩散模型，通过语义蒸馏增强超先验的表达能力，并在解码速度和码率上取得显著提升。虽然其核心思想——利用强语义先验指导生成重建——在理论上对文档图像的视觉表示学习有一定启发，但该方法针对的是自然图像压缩任务，未涉及文本结构建模、OCR对齐或细粒度定位等DIU核心问题。迁移至文档理解场景需大幅重构，且缺乏对高分辨率文档布局保持、字符级保真度等关键需求的支持，因此属于弱相关上游工作。
Abstract While recent diffusion-based generative image codecs have shown impressive performance, their iterative sampling process introduces unpleasing latency. In this work, we revisit the design of a diffusion-based codec and argue that multi-step sampling is not necessary for generative compression. Based on this insight, we propose OneDC, a One-step Diffusion-based generative image Codec -- that integrates a latent compression module with a one-step diffusion generator. Recognizing the critical role of semantic guidance in one-step diffusion, we propose using the hyperprior as a semantic signal, overcoming the limitations of text prompts in representing complex visual content. To further enhance the semantic capability of the hyperprior, we introduce a semantic distillation mechanism that transfers knowledge from a pretrained generative tokenizer to the hyperprior codec. Additionally, we adopt a hybrid pixel- and latent-domain optimization to jointly enhance both reconstruction fidelity and perceptual realism. Extensive experiments demonstrate that OneDC achieves SOTA perceptual quality even with one-step generation, offering over 39% bitrate reduction and 20x faster decoding compared to prior multi-step diffusion-based codecs. Project: https://onedc-codec.github.io/
Authors Naifu Xue, Zhaoyang Jia, Jiahao Li, Bin Li, Yuan Zhang, Yan Lu
Comment Accepted by NeurIPS 2025
Categories Computer Vision and Pattern Recognition
5
Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regression
Title CN 门控KalmaNet：通过测试时岭回归实现的衰减记忆层
Keywords 线性状态空间模型 · 长序列建模 · 测试时计算 · 岭回归 · 记忆机制
Summary 本文提出了Gated KalmaNet（GKA），一种新型SSM层，通过测试时在线岭回归增强对历史信息的保留能力，结合自适应正则化与门控机制保障数值稳定性，并采用Chebyshev迭代提升低精度环境下的收敛性。实验显示其在短上下文语言理解和长达128k token的长上下文RAG/LongQA任务中均优于现有SSM方法。尽管计算高效且擅长长程依赖建模，但该模型局限于纯文本领域，未考虑视觉模态或文档结构特性。
Reason 该论文提出了一种新的线性状态空间模型（SSM）Gated KalmaNet，通过测试时在线岭回归和卡尔曼滤波启发机制，在保持计算效率的同时改善长序列记忆能力，并在长文本RAG和LongQA任务上表现优异。虽然其核心动机是语言建模中的长程依赖问题，且未直接涉及视觉或多模态输入，但其对长序列信息保持和数值稳定性的改进，理论上可迁移到长文档VLM的推理过程中，尤其是处理超长文档（如128k token以上）时的记忆管理。然而，该工作完全基于纯文本序列，缺乏对视觉输入、多模态对齐或文档布局建模的支持，无法直接解决DIU中的OCR幻觉、细粒度定位或复杂排版理解等关键问题，迁移路径较间接。
Abstract As efficient alternatives to softmax Attention, linear state-space models (SSMs) achieve constant memory and linear compute, but maintain only a lossy, fading summary of the past, often leading to inferior performance in recall oriented tasks. We propose Gated KalmaNet (GKA), a layer that reduces this gap by accounting for the full past when predicting the next token, while maintaining SSM-style efficiency. GKA achieves this by solving an online ridge regression problem at test time, with constant memory and linear compute cost in the sequence length. Drawing inspiration from the Kalman Filter, we iteratively solve the online ridge regression problem. However, a critical insight is that standard Kalman filter equations are numerically unstable in low-precision environments (like bfloat16) and difficult to parallelize in modern hardware. We address both challenges through two key innovations: (1) an adaptive regularization strategy with input-dependent gating that controls the condition number of the ridge regression, ensuring numerical stability while balancing memory retention. And (2) the use of Chebyshev Iteration instead of other conventional iterative solvers, which we demonstrate to be more stable in low-precision settings. To further improve scalability, we develop a hardware-aware chunk-wise implementation of Chebyshev Iteration along with custom kernels for backpropagating through our adaptive regularization and gating mechanisms. Empirically, GKA shows strong language understanding capabilites on short-context tasks outperforming existing SSM layers (like Mamba2, GLA and Gated DeltaNet). On long-context, GKA excels at real-world RAG and LongQA tasks up to 128k tokens, achieving more than $10$% relative improvement over other fading memory baselines.
Authors Liangzu Peng, Aditya Chattopadhyay, Luca Zancato, Elvis Nunez, Wei Xia, Stefano Soatto
Comment 30 pages, 10 figures
Categories Computation and Language
5
Soft Adaptive Policy Optimization
Title CN 软自适应策略优化
Keywords 强化学习 · 策略优化 · 大语言模型 · 软门控 · 训练稳定性
Summary 本文提出了Soft Adaptive Policy Optimization（SAPO），一种用于大语言模型强化学习训练的新策略优化方法。SAPO采用温度控制的软门控机制，自适应地衰减离策略更新，避免传统方法中因硬裁剪导致的学习信号丢失问题。相比GSPO和GRPO，SAPO在保持序列一致性的同时实现令牌级自适应调节，在数学推理任务中表现出更高的训练稳定性和Pass@1性能，并在Qwen3-VL模型系列中验证了其有效性。
Reason 该论文提出了一种用于大语言模型强化学习训练的策略优化方法SAPO，通过软门控机制替代硬裁剪，实现更稳定且高效的策略更新。虽然其在数学推理等任务上验证了有效性，并应用于Qwen3-VL系列模型的训练，但核心贡献属于LLM训练中的优化算法改进，未针对文档图像理解（DIU）任务本身的设计（如布局分析、表格识别、OCR-free架构）或文档特有的挑战（如高分辨率、密集文本、视觉幻觉）提出直接解决方案。尽管SAPO可能间接提升VLM在DocVQA中的推理稳定性，但迁移路径不明确，对DIU的方法论推动较弱。
Abstract Reinforcement learning (RL) plays an increasingly important role in enhancing the reasoning capabilities of large language models (LLMs), yet stable and performant policy optimization remains challenging. Token-level importance ratios often exhibit high variance-a phenomenon exacerbated in Mixture-of-Experts models-leading to unstable updates. Existing group-based policy optimization methods, such as GSPO and GRPO, alleviate this problem via hard clipping, making it difficult to maintain both stability and effective learning. We propose Soft Adaptive Policy Optimization (SAPO), which replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful learning signals. Compared with GSPO and GRPO, SAPO is both sequence-coherent and token-adaptive. Like GSPO, SAPO maintains sequence-level coherence, but its soft gating forms a continuous trust region that avoids the brittle hard clipping band used in GSPO. When a sequence contains a few highly off-policy tokens, GSPO suppresses all gradients for that sequence, whereas SAPO selectively down-weights only the offending tokens and preserves the learning signal from the near-on-policy ones, improving sample efficiency. Relative to GRPO, SAPO replaces hard token-level clipping with smooth, temperature-controlled scaling, enabling more informative and stable updates. Empirical results on mathematical reasoning benchmarks indicate that SAPO exhibits improved training stability and higher Pass@1 performance under comparable training budgets. Moreover, we employ SAPO to train the Qwen3-VL model series, demonstrating that SAPO yields consistent performance gains across diverse tasks and different model sizes. Overall, SAPO provides a more reliable, scalable, and effective optimization strategy for RL training of LLMs.
Authors Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, Junyang Lin
Categories Artificial Intelligence, Computation and Language
5
Dynamic Test-Time Compute Scaling in Control Policy: Difficulty-Aware Stochastic Interpolant Policy
Title CN 控制策略中的动态测试时计算扩展：难度感知的随机插值策略
Keywords 动态计算 · 扩散策略 · 机器人控制 · 测试时扩展 · 效率优化
Summary 本文提出了难度感知的随机插值策略（DA-SIP），使机器人控制器能根据当前任务难度动态调整求解步数和计算强度，在保持性能的同时实现2.6-4.4倍的计算加速。该方法基于扩散模型和流匹配框架，引入难度分类器实时决策最优积分路径。
Reason 该论文属于具身智能与机器人控制领域，提出了一种在推理时根据任务难度自适应调整计算资源的控制策略（DA-SIP），核心贡献在于提升机器人操作中的计算效率。尽管其‘动态调整测试时计算’的思想在理论上对长文档推理或多步DocVQA任务有一定启发意义（例如在简单问题上快速响应，复杂问题上分配更多推理步骤），但该方法针对的是连续控制与动力学系统，迁移到文档图像理解中的具体机制不明确，且无直接视觉-语言或多模态对齐改进。因此属于关联性较弱的上游方法，迁移难度大。
Abstract Diffusion- and flow-based policies deliver state-of-the-art performance on long-horizon robotic manipulation and imitation learning tasks. However, these controllers employ a fixed inference budget at every control step, regardless of task complexity, leading to computational inefficiency for simple subtasks while potentially underperforming on challenging ones. To address these issues, we introduce Difficulty-Aware Stochastic Interpolant Policy (DA-SIP), a framework that enables robotic controllers to adaptively adjust their integration horizon in real time based on task difficulty. Our approach employs a difficulty classifier that analyzes observations to dynamically select the step budget, the optimal solver variant, and ODE/SDE integration at each control cycle. DA-SIP builds upon the stochastic interpolant formulation to provide a unified framework that unlocks diverse training and inference configurations for diffusion- and flow-based policies. Through comprehensive benchmarks across diverse manipulation tasks, DA-SIP achieves 2.6-4.4x reduction in total computation time while maintaining task success rates comparable to fixed maximum-computation baselines. By implementing adaptive computation within this framework, DA-SIP transforms generative robot controllers into efficient, task-aware systems that intelligently allocate inference resources where they provide the greatest benefit.
Authors Inkook Chun, Seungjae Lee, Michael S. Albergo, Saining Xie, Eric Vanden-Eijnden
Categories Artificial Intelligence
5
Representation Interventions Enable Lifelong Unstructured Knowledge Control
Title CN 表示干预实现终身非结构化知识控制
Keywords 知识编辑 · 表示干预 · 大语言模型 · 终身学习 · 模型编辑
Summary 本文提出RILKE，一种基于表示空间干预的大模型终身知识控制方法。它通过学习低维、局部化的编辑模块，并利用查询自适应路由机制，在不微调原始模型的前提下实现高效、可扩展的知识更新。方法在LLaMA和Qwen上验证了编辑成功率、抗 paraphrase 干扰能力及通用性能保持。
Reason 该论文提出RILKE方法，通过在表示空间中进行干预来实现大语言模型的持续知识编辑，核心贡献在于支持多轮、无干扰的知识更新。虽然其动机来自通用LLM的知识修正，但该‘表示空间干预’机制具有迁移到文档图像理解（DIU）的潜力：例如，在VLM处理DocVQA时，可用于动态纠正OCR幻觉导致的错误知识，或在不重训练的情况下注入领域特定的排版-语义规则。然而，该方法目前未在多模态或视觉-语言任务中验证，且DIU中的错误更多源于视觉感知而非纯语言知识错误，迁移路径存在不确定性，因此仅为弱相关。
Abstract Large language models (LLMs) often produce incorrect or outdated content. Updating their knowledge efficiently and accurately without costly retraining is a major challenge. This problem is especially hard for complex, unstructured knowledge in a lifelong setting, where many edits must coexist without interference. We introduce RILKE (Representation Intervention for Lifelong KnowledgE Control), a robust and scalable method that treats knowledge control as interventions within the model's representation space. Leveraging representation-space expressiveness, we identify two properties enabling RILKE to deliver fine-grained control over complex, unstructured knowledge while maintaining general utility with frozen base weights. During training, RILKE learns paraphrase-robust and edit-localized modules that limit each update to a low-dimensional subspace to minimize cross-edit interference. In inference, a query-adaptive router selects the appropriate module to guide the model's generation. In evaluation on knowledge editing benchmarks with LLaMA and Qwen models, RILKE is scalable to large-scale datasets, demonstrating high edit success, strong paraphrase generalization, and preserving general utility with modest memory overhead. These results show RILKE is an effective and scalable solution for lifelong knowledge control in LLMs.
Authors Xuyuan Liu, Zhengzhang Chen, Xinshuai Dong, Yanchi Liu, Xujiang Zhao, Shengyu Chen, Haoyu Wang, Yujun Yan, Haifeng Chen
Comment 18 Page
Categories Artificial Intelligence
5
Weak-to-Strong Generalization under Distribution Shifts NeurIPS 2025
Title CN 分布偏移下的弱到强泛化
Keywords 弱到强泛化 · 分布偏移 · 鲁棒训练 · 模型集成 · 监督学习
Summary 本文发现现有的弱到强泛化方法在分布偏移下性能下降，甚至不如弱监督模型本身。为此提出RAVEN框架，联合学习强模型参数与弱模型组合权重，实现更鲁棒的监督迁移。在图像分类、文本分类和偏好对齐任务上验证了其在分布外场景下的显著优势，并显示其能自动赋予更准确的弱模型更高权重。
Reason 该论文提出RAVEN框架，通过动态组合弱模型来监督强模型，在分布偏移下提升弱到强泛化的鲁棒性。虽然其在图像分类和文本分类中验证了方法，但实验任务与文档图像理解的核心挑战（如OCR幻觉、复杂版式、细粒度定位）关联较弱。尽管‘利用弱监督提升强模型’的思想在理论上可迁移到DIU中的去幻觉或校正机制设计，例如用小模型检测大模型在文档理解中的错误输出，但本文未涉及多模态输入、视觉-文本对齐或高分辨率处理等DIU关键问题，迁移路径不直接，创新点对DIU的推动力有限。
Abstract As future superhuman models become increasingly complex, accurately supervising their behavior may exceed human capabilities. Recent works have demonstrated that in such scenarios, weak models can effectively supervise strong models, a phenomenon known as weak-to-strong generalization. However, we find that naive weak-to-strong generalization fails under distribution shifts, often leading to worse performance of the strong model than its weak supervisors. To address this, we propose RAVEN, a robust weak-to-strong generalization framework that dynamically learns the optimal combinations of weak models in addition to parameters of the strong model. We demonstrate the effectiveness of RAVEN on image classification, text classification, and preference alignment tasks. RAVEN outperforms alternative baselines by over 30% on out-of-distribution tasks while matching or surpassing existing methods on in-distribution tasks. Moreover, our results show that RAVEN assigns higher weights to more accurate weak models, demonstrating its ability to automatically identify trustworthy supervision.
Authors Myeongho Jeon, Jan Sobotka, Suhwan Choi, Maria Brbić
Comment Accepted to NeurIPS 2025; affiliations and acknowledgements updated
Categories Artificial Intelligence
5
ST-PPO: Stabilized Off-Policy Proximal Policy Optimization for Multi-Turn Agents Training
Title CN ST-PPO：用于多轮Agent训练的稳定化离策略近端策略优化
Keywords PPO · 强化学习 · 多轮推理 · 训练稳定性 · Agent
Summary 本文分析了标准PPO在多轮语言模型训练中的不稳定性来源，并提出了两种稳定化技术：回合级重要性采样和剪切偏差校正，组合成新算法ST-PPO。实验表明该方法能有效防止性能崩溃，提升多跳问答等任务的表现。然而，该工作聚焦于纯文本多轮推理的训练稳定性，未结合视觉输入或文档结构，不属于文档图像理解本体问题的创新。
Reason 该论文提出了一种改进多轮对话和推理任务中PPO训练稳定性的方法，核心贡献在于引入回合级重要性采样和剪切偏差校正。虽然其在多跳问答和医疗QA等任务上验证了有效性，但本质上是强化学习优化层面的方法改进，且实验集中在通用或医疗领域的多轮问答Agent，属于特定场景下的训练稳定性提升。尽管‘多轮推理’机制理论上可迁移至长文档阅读中的信息追踪问题，但文中未涉及任何视觉或文档模态，也未针对OCR幻觉、布局理解等DIU核心痛点设计模块，迁移路径不直接。
Abstract PPO has been widely adopted for training large language models (LLMs) at the token level in multi-turn dialogue and reasoning tasks. However, its performance is often unstable and prone to collapse. Through empirical analysis, we identify two main sources of instability in this setting: (1)~token-level importance sampling, which is misaligned with the natural granularity of multi-turn environments that have distinct turn-level stages, and (2) inaccurate advantage estimates from off-policy samples, where the critic has not learned to evaluate certain state-action pairs, resulting in high-variance gradients and unstable updates. To address these challenges, we introduce two complementary stabilization techniques: (1) turn-level importance sampling, which aligns optimization with the natural structure of multi-turn reasoning, and (2) clipping-bias correction, which normalizes gradients by downweighting unreliable, highly off-policy samples. Depending on how these components are combined, we obtain three variants: Turn-PPO (turn-level sampling only), S-PPO (clipping-bias correction applied to token-level PPO), and ST-PPO (turn-level sampling combined with clipping-bias correction). In our experiments, we primarily study ST-PPO and S-PPO, which together demonstrate how the two stabilization mechanisms address complementary sources of instability. Experiments on multi-turn search tasks across general QA, multi-hop QA, and medical multiple-choice QA benchmarks show that ST-PPO and S-PPO consistently prevent the performance collapses observed in large-model training, maintain lower clipping ratios throughout optimization, and achieve higher task performance than standard token-level PPO. These results demonstrate that combining turn-level importance sampling with clipping-bias correction provides a practical and scalable solution for stabilizing multi-turn LLM agent training.
Authors Chenliang Li, Adel Elmahdy, Alex Boyd, Zhongruo Wang, Alfredo Garcia, Parminder Bhatia, Taha Kass-Hout, Cao Xiao, Mingyi Hong
Categories Artificial Intelligence, Computation and Language
5
Concept-Aware Batch Sampling Improves Language-Image Pretraining
Title CN 概念感知的批量采样提升语言-图像预训练
Keywords 视觉-语言预训练 · 数据采样 · 概念感知 · 数据策展 · CLIP
Summary 本文提出了DataConcept数据集和CABS采样框架，通过在线、概念感知的方式动态构造训练批次，提升CLIP/SigLIP类模型在多种视觉-语言任务上的表现。CABS支持多样性最大化和频率最大化两种策略，增强了模型对概念分布的适应能力。
Reason 该论文提出了一种概念感知的在线批量采样框架CABS，用于在视觉-语言预训练中动态构建包含多样化或高频概念的数据批次。虽然其核心思想——通过控制训练数据的概念分布来提升模型性能——具有一定的方法论价值，但该工作属于通用视觉-语言预训练的数据策略优化，并未针对文档图像理解中的具体痛点（如OCR幻觉、复杂版式建模、细粒度文本定位等）进行设计。尽管理论上可被迁移用于构建更均衡的文档概念训练集（例如表格、印章、字段等关键元素的平衡采样），但缺乏在DIU任务上的验证与针对性建模，迁移路径不直接，对文档理解的增益有限。
Abstract What data should a vision-language model be trained on? To answer this question, many data curation efforts center on the quality of a dataset. However, most of these existing methods are (i) offline, i.e. they produce a static dataset from a set of predetermined filtering criteria, and (ii) concept-agnostic, i.e. they use model-based filters which induce additional data biases. In this work, we go beyond such offline, concept-agnostic methods and advocate for more flexible, task-adaptive online concept-based curation. Our first contribution is DataConcept, a collection of 128M web-crawled image-text pairs annotated with fine-grained details about their concept composition. Building on DataConcept, we introduce Concept-Aware Batch Sampling (CABS), a simple yet effective batch sampling framework that flexibly constructs batches on-the-fly based on specific target distributions. We propose two variants: (i) Diversity Maximization (CABS-DM) to curate batches with a broad coverage of available concepts, and (ii) Frequency Maximization (CABS-FM) to curate batches with high object multiplicity. Through extensive evaluations across 28 benchmarks, we demonstrate that our CABS method significantly benefits CLIP/SigLIP model classes and yields highly performant models. Overall, CABS represents a strong open-source alternative to proprietary online data curation algorithms, enabling practitioners to define custom concept distributions that optimize for specific downstream tasks.
Authors Adhiraj Ghosh, Vishaal Udandarao, Thao Nguyen, Matteo Farina, Mehdi Cherti, Jenia Jitsev, Sewoong Oh, Elisa Ricci, Ludwig Schmidt, Matthias Bethge
Comment Tech Report
Categories Computer Vision and Pattern Recognition
5
SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
Title CN SSA：通过在特征空间中对齐全注意力与稀疏注意力输出实现稀疏注意力
Keywords 稀疏注意力 · 长上下文建模 · 注意力对齐 · 梯度传播 · 语言模型
Summary 本文提出了SSA（Sparse Sparse Attention），一种统一的稀疏注意力训练框架，通过对齐全注意力和稀疏注意力在每一层的特征输出，解决传统稀疏注意力训练中因梯度缺失导致的关键token无法学习的问题。该方法不仅提升了稀疏模型的性能，还实现了更强的注意力稀疏性，并展现出良好的长上下文外推能力。此外，SSA支持在推理时灵活调整稀疏预算，实现计算与性能之间的权衡。
Reason 该论文提出了一种用于语言模型的稀疏注意力训练框架SSA，核心贡献在于通过双向对齐全注意力与稀疏注意力的输出来维持梯度流动，从而在降低计算复杂度的同时提升长上下文建模能力。虽然其动机与长文档理解中的‘长程依赖’问题有一定关联，但方法完全基于纯文本序列设计，未涉及视觉模态或文档布局信息。尽管理论上可用于处理长文档的LLM端推理优化，但缺乏对多模态（图像+文本）、高分辨率输入、细粒度定位等DIU核心挑战的支持路径，迁移难度大，属于通用NLP效率优化工作。
Abstract The quadratic complexity of full attention limits efficient long-context processing in large language models (LLMs). Sparse attention mitigates this cost by restricting each query to attend to a subset of previous tokens; however, training-free approaches often lead to severe performance degradation. Native sparse-attention methods (e.g., NSA, MoBA) alleviate this issue, yet exhibit a critical paradox: they produce lower attention sparsity than full-attention models, despite aiming to approximate full attention, which may constrain their effectiveness. We attribute this paradox to gradient update deficiency: low-ranked key-value pairs excluded during sparse training receive neither forward contribution nor backward gradients, and thus never learn proper suppression. To overcome this limitation, we propose SSA (Sparse Sparse Attention), a unified training framework that considers both sparse and full attention and enforces bidirectional alignment at every layer. This design preserves gradient flow to all tokens while explicitly encouraging sparse-attention outputs to align with their full-attention counterparts, thereby promoting stronger sparsity. As a result, SSA achieves state-of-the-art performance under both sparse and full attention inference across multiple commonsense benchmarks. Furthermore, SSA enables models to adapt smoothly to varying sparsity budgets; performance improves consistently as more tokens are allowed to attend, supporting flexible compute-performance trade-offs at inference time. Finally, we show that native sparse-attention training surprisingly improves long-context extrapolation by mitigating the over-allocation of attention values in sink areas, with SSA demonstrating the strongest extrapolation capability.
Authors Zhenyi Shen, Junru Lu, Lin Gui, Jiazheng Li, Yulan He, Di Yin, Xing Sun
Comment 28 pages
Categories Computation and Language
5
AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking
Title CN 通过强化抽象思维增强大语言模型的推理能力
Keywords 抽象推理 · 强化学习 · 大语言模型 · 数学推理 · 分布偏移
Summary 本文提出AbstRaL方法，利用强化学习训练大语言模型对数学问题进行抽象表示，从而提升其在分布偏移下的鲁棒性和泛化能力。实验表明该方法在多个数学推理扰动基准上显著缓解性能下降，并对OOD数学与一般推理任务有隐式增益。然而，研究局限于纯文本数学推理，未涉及视觉或多模态输入。
Reason 该论文聚焦于提升大语言模型在小学数学推理任务中的抽象思维能力，采用强化学习来训练模型进行问题抽象，以应对分布偏移问题。虽然其核心思想——促进抽象推理——具有一定的通用价值，但该工作完全基于纯文本数学推理任务（GSM），未涉及任何视觉或文档模态。尽管抽象推理若能迁移至多模态场景可能有助于长文档逻辑理解或减少幻觉，但本文并未提供跨模态应用路径，也未针对文档结构、布局或视觉信息建模提出改进，因此对文档图像理解（DIU）的直接迁移价值有限。
Abstract Recent studies have shown that large language models (LLMs), especially smaller ones, often lack robustness in grade school math (GSM) reasoning. In particular, they tend to experience performance drops when faced with distribution shifts, such as changes to numerical or nominal variables, or insertions of distracting clauses. A possible strategy to address this involves generating synthetic data to further "instantiate" reasoning problems on potential variations. In this work, we instead focuses on the strategy of "abstracting" reasoning problems. This not only helps counteract distribution shifts but also facilitates the connection to symbolic tools for deriving solutions. Focusing on GSM, we find that this abstraction process is better acquired through reinforcement learning (RL) than just supervised fine-tuning, which often fails to produce faithful abstractions. Our method, AbstRaL -- which promotes abstract reasoning in LLMs using RL on granular abstraction data -- significantly mitigates performance degradation on recent GSM perturbation benchmarks. Besides, improving GSM robustness via AbstRaL is shown to also implicitly benefit LLMs' capabilities on OOD mathematical and general reasoning tasks, indicating that abstract thinking broadly enables better generalizability.
Authors Silin Gao, Antoine Bosselut, Samy Bengio, Emmanuel Abbe
Comment Under review
Categories Artificial Intelligence, Computation and Language
5
GraphMind: Theorem Selection and Conclusion Generation Framework with Dynamic GNN for LLM Reasoning
Title CN GraphMind：基于动态GNN的LLM推理定理选择与结论生成框架
Keywords 多步推理 · 图神经网络 · 大语言模型 · 定理选择 · 结构化推理
Summary GraphMind将LLM的多步推理过程建模为一个异构演化图，其中节点表示条件、定理和结论，边表示逻辑依赖关系。通过GNN编码当前推理状态，并结合语义匹配实现上下文感知的定理选择与中间结论生成，形成闭环的结构化推理机制。实验表明其在多个问答数据集上优于基线方法。
Reason 该论文提出了一种基于动态图神经网络的多步推理框架，用于数学证明等任务中的定理选择和结论生成。虽然其核心思想——通过结构化图表示中间推理状态并进行上下文感知的迭代推理——在理论上对长文档推理（如跨段落逻辑推导）有一定启发意义，但其应用场景聚焦于数学推理而非文档图像理解，且未涉及视觉输入、OCR、排版建模或细粒度定位等DIU核心问题。方法迁移需大幅重构，对DIU的直接价值有限。
Abstract Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation, including multi-step reasoning such as mathematical proving. However, existing approaches often lack an explicit and dynamic mechanism to structurally represent and evolve intermediate reasoning states, which limits their ability to perform context-aware theorem selection and iterative conclusion generation. To address these challenges, we propose GraphMind, a novel dynamic graph-based framework that integrates the graph neural network (GNN) with LLMs to iteratively select theorems and generate intermediate conclusions for multi-step reasoning. Our method models the reasoning process as a heterogeneous evolving graph, where nodes represent conditions, theorems, and conclusions, while edges capture logical dependencies between nodes. By encoding the current reasoning state with GNN and leveraging semantic matching for theorem selection, our framework enables context-aware, interpretable, and structured reasoning in a closed-loop manner. Experiments on various question-answering (QA) datasets demonstrate that our proposed GraphMind method achieves consistent performance improvements and significantly outperforms existing baselines in multi-step reasoning, validating the effectiveness and generalizability of our approach.
Authors Yutong Li, Yitian Zhou, Xudong Wang, GuoChen, Caiyan Qin
Categories Artificial Intelligence, Computation and Language
5
HERMES: Towards Efficient and Verifiable Mathematical Reasoning in LLMs
Title CN HERMES：面向高效且可验证的LLM数学推理
Keywords 数学推理 · 形式化验证 · LLM代理 · 推理优化 · 工具使用
Summary HERMES是一种新型LLM推理框架，通过在非正式语言推理中嵌入Lean的形式化验证步骤，实现高效且可验证的数学问题求解。系统引入中间检查点和记忆模块，在保持探索灵活性的同时确保逻辑正确性。实验表明其在多个数学基准上显著提升准确率并大幅降低计算开销。
Reason 该论文提出了一种将非正式推理与Lean中的形式化证明步骤交错执行的框架，通过中间形式化检查防止推理漂移，并利用记忆模块维持长链条推理的一致性。虽然其核心任务是数学推理而非文档理解，但其‘推理过程中引入外部工具进行阶段性验证’的机制对DIU领域有潜在启发，尤其是在处理长文档多步推理时可能缓解信息丢失或逻辑断裂问题。然而，该方法迁移至文档理解场景（如复杂排版解析、OCR纠错）的路径不够直接，且未涉及视觉模态或空间结构建模，因此属于弱关联的上游方法。
Abstract Informal mathematics has been central to modern large language model (LLM) reasoning, offering flexibility and enabling efficient construction of arguments. However, purely informal reasoning is prone to logical gaps and subtle errors that are difficult to detect and correct. In contrast, formal theorem proving provides rigorous, verifiable mathematical reasoning, where each inference step is checked by a trusted compiler in systems such as Lean, but lacks the exploratory freedom of informal problem solving. This mismatch leaves current LLM-based math agents without a principled way to combine the strengths of both paradigms. In this work, we introduce Hermes, the first tool-assisted agent that explicitly interleaves informal reasoning with formally verified proof steps in Lean. The framework performs intermediate formal checking to prevent reasoning drift and employs a memory module that maintains proof continuity across long, multi-step reasoning chains, enabling both exploration and verification within a single workflow. We evaluate Hermes on four challenging mathematical reasoning benchmarks using LLMs of varying parameter scales, from small models to state-of-the-art systems. Across all settings, Hermes reliably improves the reasoning accuracy of base models while substantially reducing token usage and computational cost compared to reward-based approaches. On difficult datasets such as AIME'25, Hermes achieves up to a 67% accuracy improvement while using 80% fewer total inference FLOPs. The implementation and codebase are publicly available at https://github.com/aziksh-ospanov/HERMES.
Authors Azim Ospanov, Zijin Feng, Jiacheng Sun, Haoli Bai, Xin Shen, Farzan Farnia
Categories Artificial Intelligence
5
Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFT
Title CN 超越选择题：可验证的开放性视觉语言问答用于鲁棒的强化微调
Keywords 视觉语言模型 · 开放问答 · 强化微调 · 幻觉抑制 · 自动评估
Summary 本文提出ReVeL框架，将多选题转换为可验证的开放式问题，用于更可靠地训练和评估多模态大模型。通过重写问题并设计分类别的验证策略，该方法在保持评估效率的同时揭示了传统MCQA中存在的评分虚高问题，并提升了开放问答性能。尽管其对减少模型猜测和幻觉有启示，但未直接解决文档理解中的OCR错误、密集文本或长文档推理等核心挑战。
Reason 该论文提出将多选题转化为可验证的开放式问答（OpenQA）以提升VLM训练和评估的可靠性，核心贡献在于评估范式和RFT训练信号的改进。虽然其动机涉及减少模型猜测行为、提升奖励信号鲁棒性，对DIU中常见的幻觉问题有一定启发意义，但方法本身并非针对文档图像理解任务设计，也未在DocVQA、布局分析等典型DIU任务上验证。其迁移价值局限于‘如何构造更可靠的验证机制来抑制幻觉’这一抽象层面，缺乏直接应用于文档细粒度理解或复杂排版推理的具体技术路径，因此属于弱相关上游工作。
Abstract Multiple-choice question answering (MCQA) has been a popular format for evaluating and reinforcement fine-tuning (RFT) of modern multimodal language models. Its constrained output format allows for simplified, deterministic automatic verification. However, we find that the options may leak exploitable signals, which makes the accuracy metrics unreliable for indicating real capabilities and encourages explicit or implicit answer guessing behaviors during RFT. We propose ReVeL (Rewrite and Verify by LLM), a framework that rewrites multiple-choice questions into open-form questions while keeping answers verifiable whenever possible. The framework categorizes questions according to different answer types, apply different rewriting and verification schemes, respectively. When applied for RFT, we converted 20k MCQA examples and use GRPO to finetune Qwen2.5-VL models. Models trained on ReVeL-OpenQA match MCQA accuracy on multiple-choice benchmarks and improve OpenQA accuracy by about six percentage points, indicating better data efficiency and more robust reward signals than MCQA-based training. When used for evaluation, ReVeL also reveals up to 20 percentage points of score inflation in MCQA benchmarks (relative to OpenQA), improves judging accuracy, and reduces both cost and latency. We will release code and data publicly.
Authors Yesheng Liu, Hao Li, Haiyu Xu, Baoqi Pei, Jiahao Wang, Mingxuan Zhao, Jingshu Zheng, Zheqi He, JG Yao, Bowen Qin, Xi Yang, Jiajun Zhang
Comment Project url: https://flageval-baai.github.io/ReVeL/
Categories Artificial Intelligence, Computation and Language
5
Deterministic Continuous Replacement: Fast and Stable Module Replacement in Pretrained Transformers NeurIPS 2025 ScaleOPT Workshop
Title CN 确定性连续替换：预训练Transformer中的快速稳定模块替换
Keywords 模块替换 · Transformer优化 · 注意力机制 · 知识蒸馏 · NeurIPS
Summary 本文提出确定性连续替换（DCR）方法，用于在预训练Transformer中稳定地替换模块（如用高效注意力替代标准自注意力）。通过确定性退火权重融合教师与学生输出，DCR消除了传统随机门控带来的梯度方差，在单一种子实验中表现出更快收敛和更强对齐效果。该方法为异构算子替换提供了新思路。
Reason 该论文提出了一种在预训练模型中替换模块的通用方法DCR，通过确定性加权融合教师和学生输出，解决替换过程中的优化不稳定性问题。虽然其动机来自高效注意力机制替换（如将二次复杂度自注意力替换为线性替代），但其核心贡献是模型内部组件替换的稳定性机制。该方法属于模型架构层面的通用优化技术，理论上可迁移到文档理解中的VLM架构改进，例如在高分辨率视觉编码器中替换注意力模块以提升效率。然而，目前未展示在多模态或文档任务上的应用，迁移路径尚不明确，且非针对DIU设计，因此仅为弱相关。
Abstract Replacing modules in pretrained models, especially swapping quadratic self-attention for efficient attention alternatives, poses a hard optimization problem: cold-start reinitialization destabilizes frozen backbones. We isolate this core stability challenge in a controlled study. Deterministic Continuous Replacement (DCR) blends teacher and student outputs with a deterministic, annealed weight. Theoretically, DCR eliminates gate-induced gradient variance inherent to stochastic replacement. In a single-seed study, DCR attains faster convergence and stronger alignment than stochastic gating and distillation baselines on controlled attention replacement, establishing a foundation for heterogeneous operator swaps.
Authors Rowan Bradbury, Aniket Srinivasan Ashok, Sai Ram Kasanagottu, Gunmay Jhingran, Shuai Meng
Comment Accepted to NeurIPS 2025 ScaleOPT Workshop; 8 pages; includes figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
5
Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths CoLM'25
Title CN 注意力跨度的混合：通过异构滑动窗口长度优化大语言模型推理效率
Keywords 滑动窗口注意力 · 推理效率 · 长上下文建模 · 注意力机制优化 · LLM
Summary 本文提出Mixture of Attention Spans（MoA），通过为不同注意力头和层自动分配最优滑动窗口长度来优化LLM在长文本场景下的推理效率。该方法构建了一个窗口长度搜索空间，并通过性能剖析选择最佳配置，实现了更高效的上下文利用与内存占用平衡。实验表明，MoA在多个主流LLM上显著提升了长上下文理解能力与解码吞吐量，同时大幅缩小了与全注意力机制的性能差距。
Reason 该论文提出了一种针对LLM推理阶段的滑动窗口注意力机制优化方法，核心贡献在于根据不同的注意力头和层自动配置异构的窗口长度，从而在保持性能的同时显著提升长上下文处理的效率。虽然其动机是通用LLM场景，但其技术路径对文档图像理解（DIU）中的长文档多模态推理具有潜在迁移价值——例如，在处理扫描长文档（如合同、报表）时，VLM需要高效建模跨页或跨段落的依赖关系，MoA可被用于设计更高效的视觉-文本联合注意力机制，缓解高分辨率输入带来的计算压力。然而，该工作本身未涉及多模态或视觉输入，也未验证在文档任务上的效果，属于通用推理优化中较为贴近DIU需求的一类弱相关上游方法。
Abstract Sliding-window attention offers a hardware-efficient solution to the memory and throughput challenges of Large Language Models (LLMs) in long-context scenarios. Existing methods typically employ a single window length across all attention heads and input sizes. However, this uniform approach fails to capture the heterogeneous attention patterns inherent in LLMs, ignoring their distinct accuracy-latency trade-offs. To address this challenge, we propose *Mixture of Attention Spans* (MoA), which automatically tailors distinct sliding-window length configurations to different heads and layers. MoA constructs and navigates a search space of various window lengths and their scaling rules relative to input sizes. It profiles the model, evaluates potential configurations, and pinpoints the optimal length configurations for each head. MoA adapts to varying input sizes, revealing that some attention heads expand their focus to accommodate longer inputs, while other heads consistently concentrate on fixed-length local contexts. Experiments show that MoA increases the effective context length by 3.9x with the same average sliding-window length, boosting retrieval accuracy by 1.5-7.1x over the uniform-window baseline across Vicuna-{7B, 13B} and Llama3-{8B, 70B} models. Moreover, MoA narrows the performance gap with full attention, reducing the maximum relative performance drop from 9%-36% to within 5% across three long-context understanding benchmarks. MoA achieves a 1.2-1.4x GPU memory reduction, boosting decode throughput by 6.6-8.2x and 1.7-1.9x over FlashAttention2 and vLLM, with minimal performance impact. Our code is available at: https://github.com/thu-nics/MoA
Authors Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang
Comment Published at CoLM'25
Categories Artificial Intelligence, Computation and Language
5
Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries LoG Conference 2025
Title CN 位置：超越欧几里得——基础模型应拥抱非欧几里得几何
Keywords 非欧几里得几何 · 基础模型 · 嵌入空间 · 层次结构 · 任务自适应
Summary 本文主张下一代基础模型应突破传统欧几里得空间限制，采用非欧几里得几何（如双曲空间、球面空间）来更高效地捕捉现实世界数据中的层次结构、对称性和多体关系。作者通过理论与实验分析指出，当前主流模型在欧氏空间下难以充分表达这些结构，并提出一种任务感知的自适应嵌入重构框架，支持在微调、从头训练和混合模式下构建几何化基础模型。
Reason 该论文提出基础模型应从欧几里得空间转向非欧几里得几何以更好地建模复杂数据结构，属于理论倡导型工作。虽然其讨论的层次结构、多关系建模等特性在文档排版（如表格嵌套、段落层级）中存在对应场景，但论文本身未针对文档图像或VLM进行设计，也未提供可直接迁移的技术模块。其方法论偏抽象，缺乏在视觉或文档理解任务中的实证支撑，迁移到DIU领域的路径不明确，故仅为弱相关。
Abstract In the era of foundation models and Large Language Models (LLMs), Euclidean space has been the de facto geometric setting for machine learning architectures. However, recent literature has demonstrated that this choice comes with fundamental limitations. At a large scale, real-world data often exhibits inherently non-Euclidean structures, such as multi-way relationships, hierarchies, symmetries, and non-isotropic scaling, in a variety of domains, such as languages, vision, and the natural sciences. It is challenging to effectively capture these structures within the constraints of Euclidean spaces. This position paper argues that moving beyond Euclidean geometry is not merely an optional enhancement but a necessity to maintain the scaling law for the next-generation of foundation models. By adopting these geometries, foundation models could more efficiently leverage the aforementioned structures. Task-aware adaptability that dynamically reconfigures embeddings to match the geometry of downstream applications could further enhance efficiency and expressivity. Our position is supported by a series of theoretical and empirical investigations of prevalent foundation models. Finally, we outline a roadmap for integrating non-Euclidean geometries into foundation models, including strategies for building geometric foundation models via fine-tuning, training from scratch, and hybrid approaches.
Authors Neil He, Jiahong Liu, Buze Zhang, Ngoc Bui, Ali Maatouk, Menglin Yang, Irwin King, Melanie Weber, Rex Ying
Comment 27 pages, 6 figures, LoG Conference 2025
Categories Artificial Intelligence
5
LAST: LeArning to Think in Space and Time for Generalist Vision-Language Models
Title CN LAST：让通用视觉语言模型学会在空间和时间中思考
Keywords 视觉语言模型 · 空间理解 · 时间推理 · 思维轨迹 · 零样本学习
Summary 本文提出LAST框架，通过引导视觉语言模型在推理过程中显式地构建空间和时间上的视觉思维轨迹，从而提升其在3D理解和长视频理解任务上的表现。该方法可在零样本和微调两种设置下应用，并在多个基准上取得显著提升。
Reason 该论文提出的方法旨在提升VLM在3D空间和长视频理解任务中的表现，其核心是构建‘空间与时间中的视觉思维轨迹’。虽然方法本身具有一定的通用性，但其目标场景（3D感知、视频时序建模）与文档图像理解的核心痛点（如OCR幻觉、密集文本识别、复杂版式分析）关联较弱。文档图像本质上是静态2D结构，缺乏显著的深度或长时间动态变化，因此该方法迁移到DIU领域的路径不明确。尽管‘思维轨迹’的概念可能启发长文档多步推理的设计，但当前工作未展示对细粒度定位、高分辨率处理或多模态对齐等DIU关键问题的直接帮助。
Abstract Humans can perceive and understand 3D space and long videos from sequential visual observations. But do vision-language models (VLMs) can? Recent work demonstrates that even state-of-the-art VLMs still struggle to understand 3D space and long videos, although they are powerful in typical vision-language tasks. Current methods often rely on specialized architectural designs to improve performance for 3D tasks and video understanding tasks separately. In contrast, we propose LAST, short for LeArn to Think in Space and Time, to jointly improve 3D spatial and long video understanding for general VLMs with only a set of 2D images as inputs. LAST makes VLMs think in space and time rather than only with text before giving the final answer, building visual thinking trajectories in 3D space and temporal dimension. We demonstrate the effectiveness of LAST in two scenarios: 1) zero-shot, where we directly prompt proprietary models; and 2) fine-tuning general VLMs with data that include thinking trajectories in 3D space and time. We show that LAST brings substantial gains in various benchmarks, including 3 spatial understanding, 4 video understanding, and 3 image understanding tasks. Notably, 15.8% gains on EgoSchema with GPT-4o in a zero-shot manner and 8.3 gains on VSI-Bench compared with Qwen2.5-VL-7B.
Authors Shuai Wang, Daoan Zhang, Tianyi Bai, Shitong Shao, Jiebo Luo, Jiaheng Wei
Categories Computer Vision and Pattern Recognition
5
Granular Computing-driven SAM: From Coarse-to-Fine Guidance for Prompt-Free Segmentation
Title CN 基于粒计算的SAM：从粗到细的无提示分割引导
Keywords 粒计算 · 图像分割 · SAM · 多粒度注意力 · 高分辨率处理
Summary 本文提出Grc-SAM，一种受粒计算启发的无提示图像分割框架，采用粗到细策略：首先在粗粒度阶段自适应提取高响应区域以实现前景定位，然后在细粒度阶段使用稀疏局部Swin注意力增强细节建模，并将优化后的掩码编码为潜在提示嵌入供SAM解码器使用。该方法提升了分割的准确性和高分辨率扩展性，为无提示分割提供了新的计算视角。
Reason 该论文提出了一种用于无提示图像分割的粗到细框架Grc-SAM，通过粒计算思想在视觉Transformer中引入多粒度注意力机制，提升了高分辨率下的细节建模和区域定位能力。虽然其核心动机是通用图像分割任务，但其‘自适应响应区域提取’和‘稀疏局部注意力增强高分辨率处理’的设计对文档图像理解中的细粒度定位与复杂排版解析具有潜在迁移价值。例如，可借鉴其粗阶段自动聚焦文本区域、细阶段精修字符边界的思路来缓解OCR对小字或密集文本的漏检问题。然而，该工作未直接涉及文本识别或多模态推理，也未针对文档结构特性设计模块，属于通用视觉方法向分割任务的延伸，对DIU的适配性需进一步验证，故仅为弱相关。
Abstract Prompt-free image segmentation aims to generate accurate masks without manual guidance. Typical pre-trained models, notably Segmentation Anything Model (SAM), generate prompts directly at a single granularity level. However, this approach has two limitations: (1) Localizability, lacking mechanisms for autonomous region localization; (2) Scalability, limited fine-grained modeling at high resolution. To address these challenges, we introduce Granular Computing-driven SAM (Grc-SAM), a coarse-to-fine framework motivated by Granular Computing (GrC). First, the coarse stage adaptively extracts high-response regions from features to achieve precise foreground localization and reduce reliance on external prompts. Second, the fine stage applies finer patch partitioning with sparse local swin-style attention to enhance detail modeling and enable high-resolution segmentation. Third, refined masks are encoded as latent prompt embeddings for the SAM decoder, replacing handcrafted prompts with an automated reasoning process. By integrating multi-granularity attention, Grc-SAM bridges granular computing with vision transformers. Extensive experimental results demonstrate Grc-SAM outperforms baseline methods in both accuracy and scalability. It offers a unique granular computational perspective for prompt-free segmentation.
Authors Qiyang Yu, Yu Fang, Tianrui Li, Xuemei Cao, Yan Chen, Jianghao Li, Fan Min, Yi Zhang
Comment 19 pages, 7 figures
Categories Computer Vision and Pattern Recognition
5
Benchmarking Corruption Robustness of LVLMs: A Discriminative Benchmark and Robustness Alignment Metric
Title CN 评估大视觉语言模型的腐蚀鲁棒性：一种判别性基准与鲁棒性对齐度量
Keywords 视觉语言模型 · 鲁棒性评估 · 视觉腐蚀 · 判别性基准 · 预测结构退化
Summary 本文提出Bench-C基准和鲁棒性对齐分数（RAS），用于更准确地评估大视觉语言模型在视觉腐蚀情况下的鲁棒性。通过筛选具有预测不一致性和语义多样性的高判别性样本，并从logit层面分析预测结构的变化，揭示了现有模型在轻微腐蚀下虽准确率变化不大但内部预测结构已退化的问题。研究还分解了破坏性与纠正性成分，以揭示不同模型的失败与恢复模式。
Reason 该论文聚焦于视觉语言模型在图像受到视觉腐蚀（如模糊、噪声等）下的鲁棒性评估，提出了一个强调高判别性样本的基准Bench-C和一种新的度量RAS。虽然其核心任务并非文档图像理解，但其提出的鲁棒性评估范式和对预测结构退化的建模可能对DIU领域有间接启发，例如用于分析OCR幻觉或复杂排版下模型输出的稳定性。然而，该方法本身是评估工具而非可直接迁移的技术组件，且未涉及文档特有的结构、布局或细粒度文本识别问题，因此迁移价值有限。
Abstract Despite the remarkable reasoning abilities of large vision-language models (LVLMs), their robustness under visual corruptions remains insufficiently studied. Existing evaluation paradigms exhibit two major limitations: 1) the dominance of low-discriminative samples in current datasets masks the real robustness gap between models; and 2) conventional accuracy-based metric fail to capture the degradation of the underlying prediction structure. To bridge these gaps, we introduce Bench-C, a comprehensive benchmark emphasizing discriminative samples for assessing corruption robustness, where a selection strategy is proposed to jointly consider the prediction inconsistency under corruption and the semantic diversity. Furthermore, we propose the Robustness Alignment Score (RAS), a unified metric that measures degradation in logit-level prediction structure by considering the shifts in prediction uncertainty and calibration alignment. Comprehensive experiments and analysis reveal several interesting findings: 1) model behaviors exhibit distinguish patterns under corruptions, such as erroneous confidence and hesitation; 2) despite subtle corruption may lead to a slight accuracy gain, the overall prediction structure still degrades; 3) by decomposing corruption robustness into destructive and corrective components, the distinct failure and recovery patterns across models can be revealed.
Authors Xiangjie Sui, Songyang Li, Hanwei Zhu, Baoliang Chen, Yuming Fang, Xin Sun
Comment 15 pages
Categories Computer Vision and Pattern Recognition
5
Understanding Task Transfer in Vision-Language Models
Title CN 理解视觉语言模型中的任务迁移
Keywords 视觉语言模型 · 任务迁移 · 多任务学习 · 零样本迁移 · PGF
Summary 本文提出了一种名为完美差距因子（PGF）的新指标，用于系统分析视觉语言模型在不同视觉感知任务之间的迁移效应。通过对三个开源VLM在13个感知任务上的表现进行评估，作者构建了任务迁移图，揭示了任务间的正负迁移模式，并基于此指导数据选择以提升训练效率。研究为多任务学习中的任务组合提供了可量化的决策依据。
Reason 该论文研究了在一种视觉感知任务上微调VLM对其他感知任务零样本性能的影响，提出了衡量任务间迁移效应的PGF指标，并构建了任务迁移图。虽然其分析框架有助于理解多任务干扰与正向迁移，但研究聚焦于通用视觉感知任务（如深度估计、物体计数），并未涉及文档结构、文本识别或布局理解等DIU核心问题。其方法论对DIU的迁移价值有限，因文档理解更关注细粒度文本定位、OCR准确性与逻辑推理一致性，而非跨感知任务的泛化。因此属于关联性较弱的上游工作，难以直接解决OCR幻觉或复杂排版建模等痛点。
Abstract Vision-Language Models (VLMs) perform well on multimodal benchmarks but lag behind humans and specialized models on visual perception tasks like depth estimation or object counting. Finetuning on one task can unpredictably affect performance on others, making task-specific finetuning challenging. In this paper, we address this challenge through a systematic study of task transferability. We examine how finetuning a VLM on one perception task affects its zero-shot performance on others. To quantify these effects, we introduce Perfection Gap Factor (PGF), a metric that captures both the breadth and magnitude of transfer. Using three open-weight VLMs evaluated across 13 perception tasks, we construct a task-transfer graph that reveals previously unobserved relationships among perception tasks. Our analysis uncovers patterns of positive and negative transfer, identifies groups of tasks that mutually influence each other, organizes tasks into personas based on their transfer behavior and demonstrates how PGF can guide data selection for more efficient training. These findings highlight both opportunities for positive transfer and risks of negative interference, offering actionable guidance for advancing VLMs.
Authors Bhuvan Sachdeva, Karan Uppal, Abhinav Java, Vineeth N. Balasubramanian
Categories Computer Vision and Pattern Recognition
5
INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Models
Title CN INTERLACE：大视觉语言模型中的交错式层剪枝与高效适配
Keywords 视觉语言模型 · 模型压缩 · 层剪枝 · 高效微调 · 模型效率
Summary 本文提出INTERLACE框架，用于大视觉语言模型的层剪枝。该方法通过分析三层层之间的冗余关系，移除最冗余的一层，并对剩余层进行少量数据微调，同时冻结第三层作为稳定锚点。这种交错的微调-冻结机制可在仅使用1%数据的情况下快速收敛，在删除25%网络层后仍保留88.9%的平均性能，达到SOTA水平。
Reason 该论文提出一种针对视觉语言模型（VLM）的层剪枝方法，通过分析连续三层的局部冗余性，交错进行微调与冻结以实现高效压缩。虽然其在VLM上实现了高效的模型压缩和性能保持，但核心贡献属于模型效率优化范畴，且未针对文档图像理解中的具体任务（如DocVQA、表格识别等）或痛点（如OCR幻觉、复杂排版）设计。尽管模型压缩技术理论上可用于部署轻量级DIU系统，但迁移价值有限，属于DIU领域外围的常规优化工作。
Abstract We introduce INTERLACE, a novel framework that prunes redundant layers in VLMs while maintaining performance through sample-efficient finetuning. Existing layer pruning methods lead to significant performance drop when applied to VLMs. Instead, we analyze triplets of consecutive layers to identify local redundancy, removing the most redundant of the first two layers, finetune the remaining layer to compensate for the lost capacity, and freeze the third layer to serve as a stable anchor during finetuning. We found that this interleaved finetune-freeze design enables rapid convergence with minimal data after pruning. By finetuning only a subset of layers on just 1% of the FineVision dataset for one epoch, Interlace achieves 88.9% average performance retention after dropping 25% of the network, achieving SOTA performance. Our code is available at: https://github.com/pmadinei/Interlace.git
Authors Parsa Madinei, Ryan Solgi, Ziqi Wen, Jonathan Skaza, Miguel Eckstein, Ramtin Pedarsani
Categories Computer Vision and Pattern Recognition
5
Entropy-Guided Reasoning Compression
Title CN 熵引导的推理压缩
Keywords 推理压缩 · 思维链 · 熵优化 · 大语言模型 · 训练冲突
Summary 本文提出熵引导的推理压缩框架，解决大模型在复杂推理任务中思维链过长导致计算成本高的问题。通过分析训练过程中压缩目标与性能目标之间的熵冲突，设计动态调节机制：低熵时鼓励简洁推理，高熵时增强探索能力。实验表明可在将推理长度压缩至原始20%的同时保持甚至提升准确性。
Reason 该论文提出了一种针对大模型推理过程中思维链过长问题的压缩方法，通过熵引导训练框架在保持准确率的同时显著缩短推理路径。虽然其核心关注的是数学推理任务中的效率优化，但其对‘冗余逻辑连接词’的识别与压缩机制可能间接启发文档图像理解中多步推理过程的精简，例如在长文档VQA中减少无关推理步骤。然而，该方法并未直接处理视觉输入或文档结构特性，迁移至DIU场景需额外设计模态对齐与空间信息保留机制，应用路径较远。
Abstract Large reasoning models have demonstrated remarkable performance on complex reasoning tasks, yet the excessive length of their chain-of-thought outputs remains a major practical bottleneck due to high computation cost and poor deployability. Existing compression methods have achieved partial success but overlook a crucial phenomenon in the training process -- the entropy conflict. During compression training, entropy decreases, leading to shorter reasoning but limited exploration, while accuracy-oriented objectives increase entropy, lengthening reasoning chains. This can cause the model to get stuck in a local dilemma. Our analysis further reveals the origin of the entropy conflict: many high-entropy tokens are logical connectors that receive larger gradients and are encouraged under the performance objective, while the compression objective simultaneously penalizes these potentially redundant connectors. This opposing pressure creates a direct source of entropy conflict. To address these issues, we adopt an entropy-guided training framework. As entropy descends, the model is guided toward efficient reasoning by encouraging concise thought steps; as entropy rises, exploration is reinforced under the compact reasoning mode to improve robustness. Experiments on six mathematical benchmarks show that our method compresses reasoning length to 20% of the original while maintaining or even surpassing baseline accuracy. Code and models will be released publicly.
Authors Hourun Zhu, Yang Gao, Wenlong Fei, Jiawei Li, Huashan Sun
Comment 10pages, 4 figures
Categories Computation and Language
5
Majority of the Bests: Improving Best-of-N via Bootstrapping
Title CN 最优中的多数：通过自举改进最佳N选一
Keywords Best-of-N · 自举方法 · 输出选择机制 · 推理稳定性 · 语言模型采样
Summary 论文观察到在使用不完美的奖励模型时，Best-of-N（BoN）策略容易选出错误答案，因为最高评分输出未必正确，但正确答案往往是输出分布中最常出现的模式。为此提出Majority-of-the-Bests（MoB），通过自举重采样估计BoN的输出分布，并选择最频繁的结果作为最终输出。实验表明MoB在多种LLM和奖励模型组合下显著优于传统BoN和自洽性方法。
Reason 该论文提出了一种新的输出选择机制Majority-of-the-Bests（MoB），通过自举估计Best-of-N的输出分布并选择其众数，以提升在不完美奖励模型下的推理稳定性。虽然方法本身针对通用LLM推理优化，但其核心思想——利用输出分布的模式而非单次高分采样来抑制错误——具备迁移到文档图像理解（DIU）中复杂推理任务的潜力，例如DocVQA中对抗视觉幻觉或OCR错误导致的错误答案选择。然而，该方法未直接涉及多模态输入或视觉信息处理，迁移需额外设计适配机制，且当前实验未覆盖视觉-语言场景，因此对DIU的适用性尚存不确定性。
Abstract Sampling multiple outputs from a Large Language Model (LLM) and selecting the most frequent (Self-consistency) or highest-scoring (Best-of-N) candidate is a popular approach to achieve higher accuracy in tasks with discrete final answers. Best-of-N (BoN) selects the output with the highest reward, and with perfect rewards, it often achieves near-perfect accuracy. With imperfect rewards from reward models, however, BoN fails to reliably find the correct answer and its performance degrades drastically. We consider the distribution of BoN's outputs and highlight that, although the correct answer does not usually have a probability close to one under imperfect rewards, it is often the most likely outcome. This suggests that the mode of this distribution can be more reliably correct than a sample from it. Based on this idea, we propose Majority-of-the-Bests (MoB), a novel selection mechanism that estimates the output distribution of BoN via bootstrapping and selects its mode. Experimental results across five benchmarks, three different base LLMs, and two reward models demonstrate consistent improvements over BoN in 25 out of 30 setups. We also provide theoretical results for the consistency of the bootstrapping. MoB serves as a simple, yet strong alternative to BoN and self-consistency, and more broadly, motivates further research in more nuanced selection mechanisms.
Authors Amin Rakhsha, Kanika Madan, Tianyu Zhang, Amir-massoud Farahmand, Amir Khasahmadi
Categories Artificial Intelligence, Computation and Language
5
KV-Efficient VLA: A Method to Speed up Vision Language Models with RNN-Gated Chunked KV Cache
Title CN KV-Efficient VLA：一种通过RNN门控分块KV缓存加速视觉语言模型的方法
Keywords 视觉语言模型 · KV缓存优化 · 推理加速 · 记忆压缩 · RNN门控
Summary 本文提出KV-Efficient VLA，通过将KV缓存划分为固定大小的块，并引入基于循环网络的门控模块，根据学习到的效用分数对历史上下文进行摘要与过滤，在保持近期细粒度信息的同时清除过时的低相关性记忆。实验表明该方法可显著减少FLOPs、提升推理速度并降低KV内存占用，且无需修改下游控制逻辑即可集成到现有VLA系统中。
Reason 该论文提出了一种针对视觉语言动作（VLA）模型的KV缓存压缩方法，核心是通过分块和循环门控机制实现历史上下文的选择性保留，从而降低推理时的计算量和内存占用。虽然其应用场景为机器人领域，但其提出的KV缓存高效管理机制属于推理阶段的通用优化技术。该方法在理论上可迁移到文档图像理解中的长文档或多页文档处理场景，用于缓解高分辨率图像或长序列输入带来的KV缓存膨胀问题，提升VLM在DocVQA等任务中的推理效率。然而，目前未见其在文档领域的直接验证，且主要动机并非解决DIU核心痛点（如OCR幻觉、复杂布局建模），迁移路径存在一定工程适配成本，因此仅为中等偏弱价值。
Abstract Vision-Language-Action (VLA) models offer a unified framework for robotic perception and control, but their ability to scale to real-world, long-horizon tasks is limited by the high computational cost of attention and the large memory required for storing key-value (KV) pairs during inference, particularly when retaining historical image tokens as context. Recent methods have focused on scaling backbone architectures to improve generalization, with less emphasis on addressing inference inefficiencies essential for real-time use. In this work, we present KV-Efficient VLA, a model-agnostic memory compression approach designed to address these limitations by introducing a lightweight mechanism to selectively retain high-utility context. Our method partitions the KV cache into fixed-size chunks and employs a recurrent gating module to summarize and filter the historical context according to learned utility scores. This design aims to preserve recent fine-grained detail while aggressively pruning stale, low-relevance memory. Based on experiments, our approach can yield an average of 24.6% FLOPs savings, 1.34x inference speedup, and 1.87x reduction in KV memory. Our method integrates seamlessly into recent VLA stacks, enabling scalable inference without modifying downstream control logic.
Authors Wanshun Xu, Long Zhuang, Lianlei Shan
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
5
Steering Latent Traits, Not Learned Facts: An Empirical Study of Activation Control Limits
Title CN 引导潜在特质而非习得事实：激活控制限制的实证研究
Keywords 激活干预 · 行为控制 · 大语言模型 · 干预有效性 · 经验分析
Summary 本文对大语言模型中的激活干预技术进行了实证研究，分析了在50种不同行为类型（包括人格特质、风格、错位行为等）上的干预效果差异。研究发现干预效果显著依赖于行为类型，且某些特质呈现倒U型响应曲线；同时，向量分离指标无法预测成功，但更大的训练数据支持更强干预。结果为激活干预提供了经验指导。
Reason 该论文研究的是大语言模型中通过激活干预（activation steering）来控制行为类型（如人格特质、风格等）的有效性，属于通用LLM层面的干预方法探索。虽然其提出的行为类型依赖性和训练数据规模对干预效果的影响具有启发性，但并未涉及多模态或视觉输入，也未针对文档理解中的具体痛点（如OCR幻觉、细粒度定位、复杂排版）设计迁移路径。尽管‘intervention’机制理论上可作为DIU推理阶段去幻觉或行为引导的工具，但本文缺乏与视觉-语言联合建模的结合点，迁移难度较大，仅能视为弱相关的方法论参考。
Abstract Large language models (LLMs) require precise behavior control for safe and effective deployment across diverse applications. Activation steering offers a promising approach for LLMs' behavioral control. We focus on the question of how steering effectiveness varies across different behavior types and whether the nature of target behaviors can predict steering success. We address this through empirical analysis of activation steering across 50 behaviors that span persona archetypes, personality traits, misalignment behaviors, style cues, and impersonation of public figures. We present a set of comprehensive experiments on coefficient optimization, vector properties, and data requirements to provide comprehensive guidance for the implementation of activation steering. Our analysis demonstrates that steering effectiveness varies significantly by behavior type, with different behavioral categories exhibiting distinct response patterns to intervention strength. We find that trait expression follows an inverted-U curve with a steering coefficient strength. We also show that vector separation metrics do not predict steering success, but larger training datasets enable more aggressive steering. These findings provide empirically grounded guidance for implementing activation steering and demonstrate that steering effectiveness is heavily influenced by behavior type.
Authors Tetiana Bas, Krystian Novak
Categories Artificial Intelligence
5
Path-Constrained Retrieval: A Structural Approach to Reliable LLM Agent Reasoning Through Graph-Scoped Semantic Search
Title CN 路径约束检索：通过图范围语义搜索实现可靠LLM代理推理的结构化方法
Keywords 路径约束检索 · 知识图谱 · LLM代理 · 结构一致性 · 语义搜索
Summary 本文提出了路径约束检索（PCR），通过将语义搜索限制在知识图中从锚点可达的节点范围内，确保检索结果与当前推理路径保持结构一致性。实验表明，PCR在结构一致性上显著优于基线方法，并大幅缩短了检索结果的图距离。该方法有助于提升LLM代理推理的连贯性和可靠性。
Reason 该论文提出了一种结合知识图谱结构约束与语义搜索的检索机制（PCR），以提升LLM代理在推理过程中上下文的一致性。虽然其核心动机是解决通用Agent推理中的信息不连贯问题，但该方法对文档图像理解（DIU）中长文档多步推理和信息检索具有潜在迁移价值——例如，在处理复杂排版或跨页表格时，可通过构建文档布局图作为知识图结构，利用PCR避免检索到逻辑无关区域的信息。然而，论文未直接涉及视觉输入、多模态建模或OCR相关幻觉问题，且应用场景完全基于文本知识库，因此对DIU的适配仍需较大改造，属于弱关联的上游方法。
Abstract Large Language Model agents often retrieve context from knowledge bases that lack structural consistency with the agent's current reasoning state, leading to incoherent reasoning chains. We introduce Path-Constrained Retrieval (PCR), a retrieval method that combines structural graph constraints with semantic search to ensure retrieved information maintains logical relationships within a knowledge graph. PCR restricts the search space to nodes reachable from an anchor node, preventing retrieval of structurally disconnected information that may lead to inconsistent reasoning. We evaluate PCR on PathRAG-6, a benchmark spanning six domains with 180 nodes and 360 edges. Our results show that PCR achieves full structural consistency compared to 24-32 percent in baseline methods, while maintaining strong relevance scores. On the technology domain, PCR obtains full relevance at rank 10 with full structural consistency, significantly outperforming vector search and hybrid retrieval. PCR reduces the average graph distance of retrieved context by 78 percent compared to baselines, demonstrating retrieval of more structurally consistent information. These findings suggest that path-constrained retrieval is an effective approach for improving the reliability and coherence of LLM agent reasoning systems.
Authors Joseph Oladokun
Comment 10 pages
Categories Computation and Language
5
VCU-Bridge: Hierarchical Visual Connotation Understanding via Semantic Bridging
Title CN VCU-Bridge：通过语义桥接实现分层视觉内涵理解
Keywords 多模态大模型 · 分层推理 · 语义桥接 · 基准构建 · MCTS
Summary 本文提出了VCU-Bridge框架和HVCU-Bench基准，旨在实现从底层视觉感知到高层抽象语义的分层推理，并通过蒙特卡洛树搜索构建训练数据以增强低层能力向高层的传递。实验表明强化低层理解能提升高层推理性能，并在多个通用多模态基准上取得增益。然而，其研究对象主要为通用图像的抽象内涵理解，与文档图像理解的核心任务如表格识别、VIE、布局分析等存在领域偏差。
Reason 该论文提出了一种分层的视觉内涵理解框架VCU-Bridge和对应的基准HVCU-Bench，强调从低层感知到高层语义推理之间的语义桥接与可追溯性。虽然其动机涉及多模态模型在视觉理解中缺乏人类式层次化推理的问题，但任务设定偏向通用图像的抽象概念推理（如艺术图像、隐喻理解），而非文档图像中的结构化信息提取或文本-布局联合理解。尽管其提出的‘语义桥接’和MCTS引导的数据生成可能对长文档多步推理有一定启发，但整体迁移路径不直接，且未针对OCR幻觉、密集文本、复杂排版等DIU核心痛点设计机制，属于弱关联的上游方法。
Abstract While Multimodal Large Language Models (MLLMs) excel on benchmarks, their processing paradigm differs from the human ability to integrate visual information. Unlike humans who naturally bridge details and high-level concepts, models tend to treat these elements in isolation. Prevailing evaluation protocols often decouple low-level perception from high-level reasoning, overlooking their semantic and causal dependencies, which yields non-diagnostic results and obscures performance bottlenecks. We present VCU-Bridge, a framework that operationalizes a human-like hierarchy of visual connotation understanding: multi-level reasoning that advances from foundational perception through semantic bridging to abstract connotation, with an explicit evidence-to-inference trace from concrete cues to abstract conclusions. Building on this framework, we construct HVCU-Bench, a benchmark for hierarchical visual connotation understanding with explicit, level-wise diagnostics. Comprehensive experiments demonstrate a consistent decline in performance as reasoning progresses to higher levels. We further develop a data generation pipeline for instruction tuning guided by Monte Carlo Tree Search (MCTS) and show that strengthening low-level capabilities yields measurable gains at higher levels. Interestingly, it not only improves on HVCU-Bench but also brings benefits on general benchmarks (average +2.53%), especially with substantial gains on MMStar (+7.26%), demonstrating the significance of the hierarchical thinking pattern and its effectiveness in enhancing MLLM capabilities. The project page is at https://vcu-bridge.github.io .
Authors Ming Zhong, Yuanlei Wang, Liuzhou Zhang, Arctanx An, Renrui Zhang, Hao Liang, Ming Lu, Ying Shen, Wentao Zhang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
5
AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokens
Title CN AdaPerceiver：具有自适应宽度、深度和令牌的Transformer
Keywords 自适应计算 · Transformer · 视觉令牌 · 高效推理 · 动态架构
Summary AdaPerceiver是首个在同一模型中实现深度、宽度和令牌三重自适应的Transformer架构，通过联合训练策略保证不同配置下的性能稳定性。在图像分类任务中，它在保持高精度的同时显著提升了吞吐量；在密集预测任务中，相比ViT-H/14减少了约26倍的编码器FLOPs。该方法为资源受限场景下的高效推理提供了新思路，但尚未探索其在文档理解或多模态任务中的潜力。
Reason 该论文提出了一种在推理时可动态调整模型深度、宽度和令牌数量的Transformer架构，旨在提升计算效率与吞吐量。虽然其评估任务包含图像分类和密集预测（如语义分割），但并未针对文档图像理解中的核心问题（如OCR幻觉、复杂版式建模、细粒度文本定位）进行设计或验证。尽管其‘视觉令牌自适应’机制理论上可用于高分辨率文档处理以降低FLOPs，从而支持长文档或多尺度分析，但缺乏对多模态（文本-布局-图像）融合的支持，且未涉及语言模型联合训练或VQA类任务，迁移至DIU场景需大量重构，实际应用路径不明确。
Abstract Modern transformer architectures achieve remarkable performance across tasks and domains but remain rigid in how they allocate computation at inference time. Real-world deployment often requires models to adapt to diverse hardware and latency constraints, yet most approaches to dynamic computation focus on a single axis -- such as reducing the number of tokens. We present a novel capability: AdaPerceiver, the first transformer architecture with unified adaptivity across depth, width, and tokens within a single model. We propose an architecture that supports adaptivity along these axes. We couple this with an efficient joint training regime that ensures the model maintains performance across its various configurations. We evaluate AdaPerceiver on image classification, semantic segmentation, and depth estimation tasks. On image classification, AdaPerceiver expands the accuracy-throughput Pareto front. It achieves 85.4% accuracy while yielding 36% higher throughput than FlexiViT-L. On dense prediction, AdaPerceiver matches ViT-H/14 while having $\sim$26x fewer encoder FLOPs (floating-point operations) on semantic segmentation and depth estimation. Finally, we show how AdaPerceiver equipped with a policy can maintain ImageNet1K accuracy ($\pm0.1$ percentage points) while reducing FLOPs by $24-33$%.
Authors Purvish Jajal, Nick John Eliopoulos, Benjamin Shiue-Hal Chou, George K. Thiruvathukal, Yung-Hsiang Lu, James C. Davis
Categories Computer Vision and Pattern Recognition
5
Quantifying Modality Contributions via Disentangling Multimodal Representations
Title CN 通过解耦多模态表示来量化模态贡献
Keywords 多模态表示 · 模态贡献量化 · 部分信息分解 · 可解释性 · IPFP
Summary 本文提出一种基于部分信息分解（PID）的新框架，用于量化多模态模型中各模态在内部表示中的信息贡献，区分出独特、冗余和协同成分。通过迭代比例拟合算法（IPFP），可在不重训练的情况下实现推理阶段的层级与数据集级分析，提供比传统准确率下降法更清晰的归因结果。
Reason 该论文提出了一种基于部分信息分解（PID）的框架，用于在表示层面量化多模态模型中不同模态的独特、冗余和协同信息贡献，并设计了无需重新训练的IPFP算法进行可扩展分析。虽然方法本身具有理论深度，且对理解VLM中视觉与语言模态的交互机制有潜在启发，但其核心目标是模型解释性，而非直接解决文档图像理解中的具体痛点（如OCR幻觉、复杂排版建模或高分辨率处理）。迁移到DIU场景需额外设计干预或架构调整机制，目前尚无明确路径表明其能显著提升DocVQA或布局分析性能，因此属于关联性较弱的方法论探索。
Abstract Quantifying modality contributions in multimodal models remains a challenge, as existing approaches conflate the notion of contribution itself. Prior work relies on accuracy-based approaches, interpreting performance drops after removing a modality as indicative of its influence. However, such outcome-driven metrics fail to distinguish whether a modality is inherently informative or whether its value arises only through interaction with other modalities. This distinction is particularly important in cross-attention architectures, where modalities influence each other's representations. In this work, we propose a framework based on Partial Information Decomposition (PID) that quantifies modality contributions by decomposing predictive information in internal embeddings into unique, redundant, and synergistic components. To enable scalable, inference-only analysis, we develop an algorithm based on the Iterative Proportional Fitting Procedure (IPFP) that computes layer and dataset-level contributions without retraining. This provides a principled, representation-level view of multimodal behavior, offering clearer and more interpretable insights than outcome-based metrics.
Authors Padegal Amit, Omkar Mahesh Kashyap, Namitha Rayasam, Nidhi Shekhar, Surabhi Narayan
Comment 16 pages, 11 figures
Categories Computation and Language
5
How LLMs Learn to Reason: A Complex Network Perspective ICLR 2026
Title CN 大语言模型如何学会推理：一种复杂网络视角
Keywords 推理机制 · 强化学习 · 复杂网络 · 拓扑演化 · 灾难性遗忘
Summary 本文提出大语言模型通过可验证奖励强化学习（RLVR）训练时出现的学习曲线分段、响应长度V型变化等现象，本质上是语义空间中隐含推理图拓扑自组织的结果。作者构建了一个极简的概念网络模型（CoNet），证明LLM的行为与该网络的动力学同构，发现其推理结构会自发形成平均度为2的稀疏网络。基于此，他们识别出阶段转换时的‘最大受挫状态’，并提出Annealed-RLVR方法，通过引入SFT‘加热’步骤来打破拓扑瓶颈。实验表明该方法在多个数学推理基准上优于标准RLVR。
Reason 该论文从复杂网络的角度分析LLM在强化学习中推理能力的形成机制，提出了概念网络自组织的拓扑演化解释，并设计了基于理论的Annealed-RLVR算法。虽然其核心关注的是通用LLM推理训练中的动力学行为，但所揭示的‘推理图结构演化’与‘拓扑瓶颈导致遗忘’现象，对文档图像理解中的长文档多步推理、VLM训练稳定性优化具有潜在启发意义。例如，在处理复杂排版文档时，模型需构建内部语义图进行跨区域推理，本文提出的‘防止主干边断连’和‘缓解策略冻结’的思想可迁移用于增强DIU系统在细粒度信息整合中的鲁棒性。然而，该工作本身未涉及视觉模态或文档任务，属于通用NLP训练机制研究，迁移路径间接，创新性主要体现在理论建模而非实际工具输出。
Abstract Training large language models with Reinforcement Learning with Verifiable Rewards (RLVR) exhibits a set of distinctive and puzzling behaviors that remain poorly understood, including a two-stage learning curve, a V-shaped response-length trajectory, and a pronounced vulnerability to catastrophic forgetting. In this work, we propose that these behaviors are emergent collective phenomena governed not by neural implementation details, but by the topological evolution of the latent reasoning graph in semantic space. By demonstrating a dynamical isomorphism between a 1.5B-parameter LLM and a minimal Concept Network Model (CoNet), we trace the causal source to the self-organization of a sparse concept web pinned to an average degree of two. This geometric perspective provides a unified physical explanation for the observed anomalies: the V-shaped trajectory tracks the evolution from parallel local skill optimization to global network integration; catastrophic forgetting stems from the topological disconnection of critical ``trunk'' edges; and policy collapse arises from the accumulation of sequential transitions at the web's leaf nodes, where broad exploration abruptly freezes into rigid, high-reward trajectories. Identifying a ``maximally frustrated state'' at the transition between learning stages, we propose Annealed-RLVR, a principled algorithm that injects a targeted SFT ``heating'' step to resolve this topological bottleneck. Experiments confirm that this theory-driven intervention outperforms standard RLVR on both in-distribution and out-of-distribution benchmarks (including Minerva and AIME). By recasting RLVR from black-box optimization into a predictable process of structural self-organization, our work provides a new physical intuition for engineering the emergent reasoning capabilities of future AI systems.
Authors Sihan Hu, Xiansheng Cai, Yuan Huang, Zhiyuan Yao, Linfeng Zhang, Pan Zhang, Youjin Deng, Kun Chen
Comment 24 pages, 11 figures, 1 table, under review as a conference paper at ICLR 2026
Categories Artificial Intelligence
5
Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration
Title CN 通过内在探索实现大语言模型的高效强化学习
Keywords 强化学习 · 数据效率 · 内在探索 · 大语言模型 · 推理
Summary 本文提出PREPO方法，利用提示困惑度衡量模型理解能力，并通过放大不同生成路径间的相对熵来促进探索，从而在数学推理任务中以更少的rollout次数实现与基线相当的性能。该方法提升了强化学习的数据效率，但实验完全基于纯文本语言模型，未涉及图像或文档模态。
Reason 该论文提出了一种提升大语言模型在强化学习中数据效率的方法（PREPO），通过提示困惑度和 rollout 熵差异来优化训练过程。虽然其在数学推理任务上减少了rollout需求，但整体工作聚焦于纯文本LLM的训练效率优化，未涉及视觉模态或文档结构信息处理。尽管‘减少无效rollout’的思想可能间接启发DIU中的长文档多步推理策略设计，但缺乏对多模态输入、视觉定位或OCR相关幻觉问题的直接建模，迁移路径不明确。
Abstract Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning ability of large language models, yet training remains costly because many rollouts contribute little to optimization, considering the amount of computation required. This study investigates how simply leveraging intrinsic data properties, almost free benefit during training, can improve data efficiency for RLVR. We propose PREPO with two complementary components. First, we adopt prompt perplexity as an indicator of model adaptability in learning, enabling the model to progress from well-understood contexts to more challenging ones. Second, we amplify the discrepancy among the rollouts by differentiating their relative entropy, and prioritize sequences that exhibit a higher degree of exploration. Together, these mechanisms reduce rollout demand while preserving competitive performance. On the Qwen and Llama models, PREPO achieves effective results on mathematical reasoning benchmarks with up to 3 times fewer rollouts than the baselines. Beyond empirical gains, we provide theoretical and in-depth analyses explaining the underlying rationale of our method to improve the data efficiency of RLVR.
Authors Yan Sun, Jia Guo, Stanley Kok, Zihao Wang, Zujie Wen, Zhiqiang Zhang
Categories Artificial Intelligence
5
MSRS: Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Models
Title CN MSRS：用于大语言模型属性对齐的自适应多子空间表示引导
Keywords 激活引导 · 多属性控制 · 子空间学习 · 行为调控 · 大语言模型
Summary 本文提出Multi-Subspace Representation Steering（MSRS），一种通过子空间分解实现大语言模型多属性行为控制的新方法。MSRS为不同属性分配正交子空间以减少相互干扰，并引入特有子空间与共享子空间的混合结构，配合动态加权机制进行组合。在推理时，该方法在token级别动态识别并干预语义关键位置，实现精细化控制。实验表明MSRS在多属性协同控制和下游任务泛化上优于现有方法。
Reason 该论文提出一种通过子空间表示微调实现多属性控制的激活引导方法，核心贡献在于减少多属性引导中的干扰问题。尽管其在LLM行为控制方面具有创新性，但属于通用的语言模型干预技术，未涉及视觉或文档模态，且缺乏与多模态输入（如图像、布局）的结合设计。虽然推理阶段的细粒度token级干预机制理论上可被迁移用于缓解VLM在文档理解中的幻觉或属性冲突问题，但原文未探讨与视觉输入、OCR信号或布局结构的联合建模，迁移路径不明确，对DIU的直接启发有限。
Abstract Activation steering offers a promising approach to controlling the behavior of Large Language Models by directly manipulating their internal activations. However, most existing methods struggle to jointly steer multiple attributes, often resulting in interference and undesirable trade-offs. To address this challenge, we propose Multi-Subspace Representation Steering (MSRS), a novel framework for effective multi-attribute steering via subspace representation fine-tuning. MSRS reduces inter-attribute interference by allocating orthogonal subspaces to each attribute, isolating their influence within the model's representation space. MSRS also incorporates a hybrid subspace composition strategy: it combines attribute-specific subspaces for unique steering directions with a shared subspace for common steering directions. A dynamic weighting function learns to efficiently integrate these components for precise control. During inference, MSRS introduces a token-level steering mechanism that dynamically identifies and intervenes on the most semantically relevant tokens, enabling fine-grained behavioral modulation. Experimental results show that MSRS significantly reduces attribute conflicts, surpasses existing methods across a range of attributes, and generalizes effectively to diverse downstream tasks.
Authors Xinyan Jiang, Lin Zhang, Jiayi Zhang, Qingsong Yang, Guimin Hu, Di Wang, Lijie Hu
Categories Artificial Intelligence
5
CascadedViT: Cascaded Chunk-FeedForward and Cascaded Group Attention Vision Transformer
Title CN 级联ViT：级联块前馈与级联组注意力视觉Transformer
Keywords Vision Transformer · 轻量化模型 · 计算效率 · 能效优化 · CCFFN
Summary 本文提出了CascadedViT（CViT），一种新型轻量级视觉Transformer，通过级联块前馈网络（CCFFN）和级联组注意力机制，在保持准确率的同时显著降低计算量和能耗。在ImageNet-1K上验证了其优越的能效比，尤其适合移动端和无人机等低功耗平台部署。然而，该工作面向通用图像分类任务，未涉及文本或文档结构建模。
Reason 该论文提出了一种轻量级、计算高效的Vision Transformer架构CascadedViT，通过引入级联块前馈网络（CCFFN）提升参数和FLOPs效率，适用于资源受限设备。虽然其核心贡献在于模型效率优化，但并未针对文档图像理解中的关键挑战（如OCR幻觉、复杂布局建模、细粒度文本定位或长文档推理）进行设计。尽管高效ViT结构理论上可用于文档编码器的加速，但缺乏对多模态对齐、高分辨率文档处理或文本-布局联合建模的支持，迁移价值有限。
Abstract Vision Transformers (ViTs) have demonstrated remarkable performance across a range of computer vision tasks; however, their high computational, memory, and energy demands hinder deployment on resource-constrained platforms. In this paper, we propose \emph{Cascaded-ViT (CViT)}, a lightweight and compute-efficient vision transformer architecture featuring a novel feedforward network design called \emph{Cascaded-Chunk Feed Forward Network (CCFFN)}. By splitting input features, CCFFN improves parameter and FLOP efficiency without sacrificing accuracy. Experiments on ImageNet-1K show that our \emph{CViT-XL} model achieves 75.5\% Top-1 accuracy while reducing FLOPs by 15\% and energy consumption by 3.3\% compared to EfficientViT-M5. Across various model sizes, the CViT family consistently exhibits the lowest energy consumption, making it suitable for deployment on battery-constrained devices such as mobile phones and drones. Furthermore, when evaluated using a new metric called \emph{Accuracy-Per-FLOP (APF)}, which quantifies compute efficiency relative to accuracy, CViT models consistently achieve top-ranking efficiency. Particularly, CViT-L is 2.2\% more accurate than EfficientViT-M2 while having comparable APF scores.
Authors Srivathsan Sivakumar, Faisal Z. Qureshi
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
5
Drift No More? Context Equilibria in Multi-Turn LLM Interactions
Title CN 不再漂移？多轮LLM交互中的上下文均衡
Keywords 上下文漂移 · 多轮交互 · 干预机制 · 动态建模 · KL散度
Summary 本文提出一种动态框架来理解和控制多轮LLM交互中的上下文漂移现象，将其建模为受控随机过程，并验证简单干预（如提示）可有效降低偏离。实验表明，模型行为趋向于稳定噪声限制下的均衡状态，而非持续退化。
Reason 该论文研究多轮对话中大语言模型的上下文漂移问题，提出通过KL散度量化漂移并引入具有恢复力的动态模型来建模其演化过程。虽然其核心关注的是对话系统中的目标一致性维持，但所提出的‘恢复力干预’机制（如提醒）理论上可迁移到长文档推理任务中，用于缓解模型在处理长文档时的信息迷失或注意力偏移问题。然而，该工作未涉及视觉输入或多模态对齐，与文档图像理解的核心挑战（如OCR幻觉、排版复杂性）关联较弱，迁移路径不够直接。
Abstract Large Language Models (LLMs) excel at single-turn tasks such as instruction following and summarization, yet real-world deployments require sustained multi-turn interactions where user goals and conversational context persist and evolve. A recurring challenge in this setting is context drift: the gradual divergence of a model's outputs from goal-consistent behavior across turns. Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics. In this work, we present a study of context drift in multi-turn interactions and propose a simple dynamical framework to interpret its behavior. We formalize drift as the turn-wise KL divergence between the token-level predictive distributions of the test model and a goal-consistent reference model, and propose a recurrence model that interprets its evolution as a bounded stochastic process with restoring forces and controllable interventions. We instantiate this framework in both synthetic long-horizon rewriting tasks and realistic user-agent simulations such as in $τ$-Bench, measuring drift for several open-weight LLMs that are used as user simulators. Our experiments consistently reveal stable, noise-limited equilibria rather than runaway degradation, and demonstrate that simple reminder interventions reliably reduce divergence in line with theoretical predictions. Together, these results suggest that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay, providing a foundation for studying and mitigating context drift in extended interactions.
Authors Vardhan Dongre, Ryan A. Rossi, Viet Dac Lai, David Seunghyun Yoon, Dilek Hakkani-Tür, Trung Bui
Categories Artificial Intelligence, Computation and Language
5
Understanding Counting Mechanisms in Large Language and Vision-Language Models
Title CN 大型语言与视觉-语言模型中计数机制的理解
Keywords 计数机制 · 视觉-语言模型 · 可解释性 · 数值表征 · 结构线索
Summary 本文通过可控实验和可解释性工具CountScope，系统分析了大语言模型和视觉-语言模型在处理重复文本和视觉项时的计数机制。研究发现模型各层逐步构建数值表征，低层处理小数值，高层处理大数值，且数值信息编码在最后一个token或区域中，可在上下文中迁移。在视觉-语言模型中，视觉嵌入也参与计数，其注意力随空间布局变化。模型依赖文本分隔符等结构线索作为计数捷径，影响预测准确性。整体上，计数行为呈现出一种受输入结构塑造的分层涌现过程。
Reason 该论文研究了LLMs和LVLMs在计数任务中如何表示和处理数值信息，通过因果中介分析和激活补丁揭示了模型内部存在一种分层渐进的数值表征机制，并发现视觉嵌入中也包含与空间结构相关的计数信号。虽然其核心关注的是模型内部机制，但并未针对文档图像理解中的具体任务（如DocVQA、表格识别或布局分析）进行设计或验证。然而，其所揭示的‘内部计数器’机制和对结构线索（如分隔符）的依赖，可能为解决文档中项目列表计数、表格行数推断等细粒度推理任务提供启发，具备一定的方法论迁移潜力，但迁移路径不够直接明确。
Abstract This paper examines how large language models (LLMs) and large vision-language models (LVLMs) represent and compute numerical information in counting tasks. We use controlled experiments with repeated textual and visual items and analyze model behavior through causal mediation and activation patching. To this end, we design a specialized tool, CountScope, for mechanistic interpretability of numerical content. Results show that individual tokens or visual features encode latent positional count information that can be extracted and transferred across contexts. Layerwise analyses reveal a progressive emergence of numerical representations, with lower layers encoding small counts and higher layers representing larger ones. We identify an internal counter mechanism that updates with each item, stored mainly in the final token or region and transferable between contexts. In LVLMs, numerical information also appears in visual embeddings, shifting between background and foreground regions depending on spatial composition. Models rely on structural cues such as separators in text, which act as shortcuts for tracking item counts and influence the accuracy of numerical predictions. Overall, counting emerges as a structured, layerwise process in LLMs and follows the same general pattern in LVLMs, shaped by the properties of the vision encoder.
Authors Hosein Hasani, Amirmohammad Izadi, Fatemeh Askari, Mobin Bagherian, Sadegh Mohammadian, Mohammad Izadi, Mahdieh Soleymani Baghshah
Categories Artificial Intelligence
5
Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimization
Title CN 探索更多，学得更好：基于互信息最小化的并行MLLM嵌入
Keywords 多模态嵌入 · MLLM · 互信息最小化 · 并行表示 · 可学习前缀
Summary 本文提出一种基于MLLM的并行嵌入学习框架PDF，利用可学习前缀激发模型对同一输入生成多样化的嵌入表示，并通过互信息最小化与对比学习联合优化，实现更全面的语义覆盖。实验表明该方法在MMEB基准上显著提升性能，且小模型下具备高效推理优势。
Reason 该论文提出了一种并行解耦框架（PDF），通过可学习前缀引导MLLM生成多个差异化嵌入，并利用互信息最小化促进多样性。虽然其在多模态嵌入学习上有所创新，但核心贡献集中在通用多模态表示学习，未针对文档图像理解中的具体痛点（如OCR幻觉、复杂版式建模、细粒度定位）进行设计或验证。尽管‘并行嵌入’思路理论上可用于增强文档语义覆盖（例如从不同视角提取表格与文本的多重表示），但缺乏向DIU任务迁移的具体路径和实证支持，属于上游方法中迁移到文档理解困难的一类。
Abstract Embedding models are a cornerstone of modern AI. Driven by Multimodal Large Language Models (MLLMs), they have made great progress in architecture and data curation, while the holistic paradigm is still limited to SSC, i.e., single input, singular embedding, contrastive supervision, which collapses rich, multifaceted inputs into monolithic embeddings and fails to fully exploit MLLM capabilities. In this paper, we tailor one Parallel Decoupling Framework (PDF) for multimodal embedding learning, by utilizing the proprietary steerability of MLLMs, i.e., their ability to flexibly generate quite differentiated response under explicit instructions. Concretely, PDF conditions a shared MLLM backbone on distinct, learnable prefixes to roll out multiple parallel paths for one input, then relies on these paths to obtain parallel embeddings. To promote full parallel diversity, we employ Mutual Information Minimization (MIM) as an explicit constraint, coupled with per-path contrastive supervision to maintain semantic alignment. Such dual-objectives force PDF to yield robust semantic coverage and a generalizable embedding space. Ultimately, the remarkable embedding space are accessible at inference via one single forward pass, incurring negligible computational overhead. We instantiate PDF on multiple MLLM backbones and prove its effectiveness on MMEB benchmark. Significant gains are consistently achieved across various resolutions and model sizes, e.g., boosting the VLM2Vec-LLaVA-1.6-LR model by a remarkable +8.9% (7B), while the VLM2Vec-Qwen2VL models by +4.2% (2B) and +3.1% (7B). In terms of efficiency, our 2B model surpasses its baseline by +2.6% using only half the computational budget.
Authors Zhicheng Wang, Chen Ju, Xu Chen, Shuai Xiao, Jinsong Lan, Xiaoyong Zhu, Ying Chen, Zhiguo Cao
Categories Computer Vision and Pattern Recognition
5
Improving Latent Reasoning in LLMs via Soft Concept Mixing
Title CN 通过软概念混合提升大语言模型的潜在推理能力
Keywords 软概念 · 潜在推理 · 大语言模型 · 强化学习 · 训练优化
Summary 本文提出Soft Concept Mixing（SCM），一种在训练阶段将概率加权的嵌入向量混合进隐藏状态的训练策略，结合强化学习优化整个潜在推理过程。实验表明该方法能提升LLM在多种推理任务上的表现并保持训练稳定。其创新点在于显式建模软概念空间以逼近人类抽象推理。
Reason 该论文提出了一种在训练中引入软概念表示的方法SCM，旨在弥合离散token生成与抽象概念空间推理之间的差距。虽然其核心关注的是通用LLM的推理能力提升，但所提出的软概念混合机制对文档图像理解（DIU）中的长文档多步推理或逻辑推导任务有一定潜在启发。然而，该方法未涉及多模态输入、视觉表示或文档结构建模，迁移至DIU需额外设计视觉-语义联合软表示机制，技术路径尚不直接。属于上游推理优化工作，但针对DIU痛点的关联较弱。
Abstract Unlike human reasoning in abstract conceptual spaces, large language models (LLMs) typically reason by generating discrete tokens, which potentially limit their expressive power. The recent work Soft Thinking has shown that LLMs' latent reasoning via soft concepts is a promising direction, but LLMs are trained on discrete tokens. To reduce this gap between the soft concepts in reasoning and the discrete tokens in training, we propose Soft Concept Mixing (SCM), a soft concept aware training scheme that directly exposes the model to soft representations during training. Specifically, SCM constructs a soft concept vector by forming a probability-weighted average of embeddings. Then, this vector is mixed into the model's hidden states, which embody rich contextual information. Finally, the entire latent reasoning process is optimized with Reinforcement Learning (RL). Experiments on five reasoning benchmarks demonstrate that SCM improves the reasoning performance of LLMs, and simultaneously maintains a stable training dynamic.
Authors Kang Wang, Xiangyu Duan, Tianyi Du
Comment 7 pages, 3 figures
Categories Computation and Language
4
On the Limits of Innate Planning in Large Language Models
Title CN 论大语言模型中内在规划能力的局限性
Keywords 大语言模型 · 规划能力 · 状态推理 · Chain-of-Thought · 8-puzzle
Summary 本文通过8-puzzle任务系统评估了大语言模型在没有代码执行等外部工具辅助下的规划与状态推理能力。实验表明，即使在提供有效动作验证的外部辅助下，当前LLM仍无法完成基本规划任务，暴露出内部状态表示脆弱和启发式规划能力弱两大缺陷。研究指出，真正的进步可能需要引入显式状态维护和结构化搜索机制。
Reason 该论文研究了LLM在无外部工具条件下进行状态跟踪与目标导向规划的能力，使用8-puzzle作为测试任务。虽然其发现（如内部状态表示脆弱、缺乏有效启发式搜索）对长文档多步推理中的信息追踪和逻辑连贯性问题有一定启示，但该工作本身属于纯文本推理分析，未涉及视觉或文档模态，迁移至DIU场景需额外设计机制支撑。其方法论并非直接可迁移的‘军火库’型创新，且不针对文档结构、布局理解或视觉-语言对齐等核心DIU挑战。
Abstract Large language models (LLMs) achieve impressive results on many benchmarks, yet their capacity for planning and stateful reasoning remains unclear. We study these abilities directly, without code execution or other tools, using the 8-puzzle: a classic task that requires state tracking and goal-directed planning while allowing precise, step-by-step evaluation. Four models are tested under common prompting conditions (Zero-Shot, Chain-of-Thought, Algorithm-of-Thought) and with tiered corrective feedback. Feedback improves success rates for some model-prompt combinations, but many successful runs are long, computationally expensive, and indirect. We then examine the models with an external move validator that provides only valid moves. Despite this level of assistance, none of the models solve any puzzles in this setting. Qualitative analysis reveals two dominant deficits across all models: (1) brittle internal state representations, leading to frequent invalid moves, and (2) weak heuristic planning, with models entering loops or selecting actions that do not reduce the distance to the goal state. These findings indicate that, in the absence of external tools such as code interpreters, current LLMs have substantial limitations in planning and that further progress may require mechanisms for maintaining explicit state and performing structured search.
Authors Charles Schepanowski, Charles Ling
Comment 33 pages, 7 figures
Categories Artificial Intelligence
4
Dual-Balancing for Multi-Task Learning Neural Networks
Title CN 多任务学习的双重平衡
Keywords 多任务学习 · 梯度平衡 · 损失缩放 · 优化方法 · 深度学习
Summary 本文提出Dual-Balancing Multi-Task Learning（DB-MTL），通过损失函数的对数变换实现损失尺度平衡，并利用最大梯度范数归一化来统一各任务梯度幅度，从而在多任务学习中实现更均衡的优化。实验表明该方法在多个基准数据集上优于现有技术。
Reason 该论文提出了一种从损失和梯度两个层面进行任务平衡的多任务学习方法DB-MTL，属于通用的多任务优化框架。虽然在理论上可能对文档图像理解中涉及多任务（如检测+识别+理解）的系统有间接帮助，但其本身并未针对DIU中的具体问题（如OCR幻觉、复杂排版、细粒度定位等）设计，也未验证在文档场景下的有效性。迁移价值有限，属于常规方法改进。
Abstract Multi-task learning aims to learn multiple related tasks simultaneously and has achieved great success in various fields. However, the disparity in loss and gradient scales among tasks often leads to performance compromises, and the balancing of tasks remains a significant challenge. In this paper, we propose Dual-Balancing Multi-Task Learning (DB-MTL) to achieve task balancing from both the loss and gradient perspectives. Specifically, DB-MTL achieves loss-scale balancing by performing logarithm transformation on each task loss, and rescales gradient magnitudes by normalizing all task gradients to comparable magnitudes using the maximum gradient norm. Extensive experiments on a number of benchmark datasets demonstrate that DB-MTL consistently performs better than the current state-of-the-art.
Authors Baijiong Lin, Weisen Jiang, Feiyang Ye, Yu Zhang, Pengguang Chen, Ying-Cong Chen, Shu Liu, Ivor W. Tsang, James T. Kwok
Comment Accepted by Neural Networks
Categories Artificial Intelligence
4
Universe of Thoughts: Enabling Creative Reasoning with Large Language Models
Title CN 思维宇宙：利用大语言模型实现创造性推理
Keywords 大语言模型 · 创造性推理 · 思维链 · 推理框架 · UoT
Summary 本文提出了名为“思维宇宙”（Universe of Thoughts, UoT）的新方法，旨在通过组合性、探索性和变革性三种推理模式提升大语言模型在创造性任务中的表现。作者构建了三个需要创造性求解的新任务和评估基准，实验表明UoT在创意可行性、效用和新颖性方面优于现有SOTA推理方法。然而，该工作聚焦于开放域创造性生成，未涉及图像模态或文档结构理解。
Reason 该论文提出了一种面向创造性推理的计算框架UoT，引入组合性、探索性和变革性三种推理范式，并构建了相应的任务与评估基准。虽然在推理机制设计上有一定新颖性，但其核心目标是‘创造性问题解决’而非文档理解中的准确性、细粒度定位或幻觉抑制等关键痛点。尽管CoT类推理方法对长文档多步推理有一定启发意义，但本文并未针对视觉-语言对齐、高分辨率输入处理或OCR错误鲁棒性等DIU核心挑战进行建模，迁移价值有限。
Abstract Reasoning based on Large Language Models (LLMs) has garnered increasing attention due to outstanding performance of these models in mathematical and complex logical tasks. Beginning with the Chain-of-Thought (CoT) prompting technique, numerous reasoning methods have emerged that decompose problems into smaller, sequential steps (or thoughts). However, existing reasoning models focus on conventional problem-solving and do not necessarily generate creative solutions by ``creative reasoning''. In domains where the solution space is expansive and conventional solutions are suboptimal, such as drug discovery or business strategization, creative reasoning to discover innovative solutions is crucial. To address this gap, first we introduce a computational framework for creative reasoning inspired by established cognitive science principles. With this framework, we propose three core creative reasoning paradigms, namely, \textit{combinational}, \textit{exploratory}, and \textit{transformative} reasoning, where each offers specific directions for systematic exploration of the universe of thoughts to generate creative solutions. Next, to materialize this framework using LLMs, we introduce the \textit{Universe of Thoughts} (or \textit{UoT}, for short), a novel set of methods to implement the aforementioned three creative processes. Finally, we introduce three novel tasks that necessitate creative problem-solving, along with an evaluation benchmark to assess creativity from three orthogonal perspectives: feasibility as constraint, and utility and novelty as metrics. With a comparative analysis against the state-of-the-art (SOTA) reasoning techniques as well as representative commercial models with reasoning capability, we show that UoT demonstrates superior performance in creative reasoning.
Authors Yuto Suzuki, Farnoush Banaei-Kashani
Categories Artificial Intelligence
4
Decorrelation Speeds Up Vision Transformers
Title CN 去相关性加速视觉Transformer
Keywords 深度学习 · 视觉Transformer · 高效AI · 去相关性 · MAE预训练
Summary 本文将去相关反向传播（DBP）应用于Masked Autoencoder（MAE）预训练中的视觉Transformer，通过减少网络各层输入的相关性来加快收敛速度。实验表明，在ImageNet-1K和ADE20K数据集的低数据量设定下，该方法可将达到基线性能所需时间减少21.1%，碳排放降低21.4%，同时分割性能提升1.1 mIoU。在私有工业数据上也观察到类似增益，证明其在资源受限场景下的实用性。
Reason 该论文提出将Decorrelated Backpropagation（DBP）引入MAE预训练，通过减少层间输入相关性来加速ViT的收敛速度，提升训练效率并降低能耗。虽然在工业数据上验证了有效性，但其核心贡献属于通用视觉模型训练优化，针对的是标准图像识别与分割任务（如ImageNet、ADE20K），并未涉及文档结构、文本布局、OCR对齐或细粒度文本理解等DIU核心问题。尽管训练加速技术理论上可被迁移用于文档VLM的预训练阶段以降低成本，但方法本身无针对文档模态特性的设计，对解决OCR幻觉、复杂排版建模或长文档推理等关键痛点无直接迁移价值，属于弱关联的上游效率工作。
Abstract Masked Autoencoder (MAE) pre-training of vision transformers (ViTs) yields strong performance in low-label data regimes but comes with substantial computational costs, making it impractical in time- and resource-constrained industrial settings. We address this by nitegrating Decorrelated Backpropagation (DBP) into MAE pre-training, an optimization method that iteratively reduces input correlations at each layer to accelerate convergence. Applied selectively to the encoder, DBP achieves faster pre-training without loss of stability. To mimic constrained-data scenarios, we evaluate our approach on ImageNet-1K pre-training and ADE20K fine-tuning using randomly sampled subsets of each dataset. Under this setting, DBP-MAE reduces wall-clock time to baseline performance by 21.1%, lowers carbon emissions by 21.4%, and improves segmentation mIoU by 1.1 points. We observe similar gains when pre-training and fine-tuning on proprietary industrial data, confirming the method's applicability in real-world scenarios. These results demonstrate that DBP can reduce training time and energy use while improving downstream performance for large-scale ViT pre-training. Keywords: Deep learning, Vision transformers, Efficient AI, Decorrelation
Authors Kieran Carrigg, Rob van Gastel, Melda Yeghaian, Sander Dalm, Faysal Boughorbel, Marcel van Gerven
Comment 16 pages, 12 figures, submitted to CVC 2026
Categories Computer Vision and Pattern Recognition
4
Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining
Title CN 像测试一样过滤：面向CLIP预训练的数据驱动数据过滤
Keywords 数据过滤 · 视觉语言预训练 · CLIP · 数据选择 · 下游反馈
Summary 本文提出了FLYT算法，通过下游任务的梯度信号训练一个评分模型，评估每个预训练样本的有用性，并引入M-FLYT融合多个评分来源，结合Soft Cap Sampling策略实现高效数据过滤。在DataComp基准上取得了当前最佳的ImageNet零样本分类表现。
Reason 该论文提出了一种基于下游任务信号来学习数据点重要性的数据筛选方法FLYT，并结合多种评分机制进行数据加权采样，提升了大规模视觉语言模型的预训练效率与性能。虽然其核心思想——利用下游反馈优化上游数据选择——在理论上对文档图像理解（DIU）中的高质量VLM训练有潜在价值，例如可用于构建更精准的文档图文对数据集以缓解OCR幻觉或排版误解，但该工作本身聚焦通用视觉语言预训练，未涉及文本布局、细粒度文字识别或复杂文档结构等DIU核心挑战，且迁移需依赖额外设计，直接应用性较弱。
Abstract We introduce Filter Like You Test (FLYT), an algorithm for curating large-scale vision-language datasets that learns the usefulness of each data point as a pretraining example. FLYT trains a scoring model that learns to weigh each example's features using gradient signals from downstream tasks training sets. Based on FLYT, we implement Mixing-FLYT (M-FLYT), which takes the per-example scores generated by different scoring methods as features, and learns to unify them into a single score. FLYT naturally produces a distribution over the training examples, which we leverage through Soft Cap Sampling (SCS), a strategy for obtaining a filtered pretraining dataset from per-example probabilities that samples examples while preventing over-representation through a repetition penalty. Using these methods, we achieve 40.1% ImageNet zero-shot accuracy on the DataComp medium scale filtering benchmark, a 2% absolute accuracy increase over all previous results and a 5.5% increase over results that - like us - use only public resources. Our approach also yields 37.7\% on the average of 38 DataComp evaluation tasks, outperforming previous public-resource approaches by 0.4\%.
Authors Mikey Shechter, Yair Carmon
Categories Computer Vision and Pattern Recognition
4
ReMatch: Boosting Representation through Matching for Multimodal Retrieval
Title CN ReMatch：通过匹配增强多模态检索的表示能力
Keywords 多模态检索 · MLLM · 表示学习 · 生成式匹配 · 可学习token
Summary ReMatch提出一种利用MLLM生成能力进行多模态匹配的新训练框架，通过在匹配阶段自回归判断查询与文档的相关性，并结合对比学习提供更强的梯度信号。同时使用多个可学习token生成更丰富的嵌入表示，在MMEB基准上达到SOTA，并展现出良好的零样本泛化能力。
Reason 该论文聚焦于通用多模态检索任务，提出通过生成式匹配机制和可学习token来提升跨模态表示。尽管其引入了MLLM的生成能力用于匹配监督，并增强了嵌入的细粒度表达，但整体框架仍面向开放域多模态检索（如图像-文本匹配），并未针对文档理解中的核心挑战（如OCR幻觉、复杂版式建模、密集文本处理）进行设计。虽然‘生成式匹配’和‘多token增强’可能为DIU提供一定启发，但迁移路径不明确，且缺乏对文档结构、布局或逻辑推理的专门建模，属于常规方法改进而非范式创新。
Abstract We present ReMatch, a framework that leverages the generative strength of MLLMs for multimodal retrieval. Previous approaches treated an MLLM as a simple encoder, ignoring its generative nature, and under-utilising its compositional reasoning and world knowledge. We instead train the embedding MLLM end-to-end with a chat-style generative matching stage. The matching stage uses the same MLLM to autoregressively decide relevance from multi-view inputs, including both raw data and its own projected embeddings for each query and document. It provides instance-wise discrimination supervision that complements a standard contrastive loss, offering stronger gradients on hard negatives and preserving the compositional strengths of the original MLLM. To obtain semantically richer multimodal embeddings, we use multiple learnable tokens to augment each input, generating fine-grained contextual, mutually orthogonal embeddings with low inference cost. Leveraging our established high-performance baseline,we assemble the ideas mentioned above into a powerful training recipe and achieve a new state-of-the-art on the Massive Multimodal Embedding Benchmark (MMEB). Our experiments show particularly strong zero-shot generalization results on five datasets, highlighting the robustness and transferability of ReMatch.
Authors Qianying Liu, Xiao Liang, Zhiqiang Zhang, Zhongfei Qing, Fengfan Zhou, Yibo Chen, Xu Tang, Yao Hu, Paul Henderson
Categories Computer Vision and Pattern Recognition
4
Scaling Efficient LLMs
Title CN 扩展高效的大语言模型
Keywords 大语言模型 · 参数效率 · 循环Transformer · 长序列建模 · AI扩展律
Summary 本文提出循环Transformer结构，通过在固定宽度滑动窗口上重复使用单个Transformer层，实现线性时间与内存效率的长序列建模。实验表明其在语言任务中表现良好，具备一定的长程依赖建模能力。然而，该模型为纯文本设计，未考虑视觉模态或文档图像特性。
Reason 该论文提出了一种名为‘循环Transformer’的架构，旨在通过滑动窗口机制和递归应用单个Transformer层来提升LLM的参数效率和序列处理效率。虽然其在长序列建模方面有一定潜力，理论上可能对长文档理解中的上下文建模有间接启发，但该工作聚焦于纯文本语言模型，并未涉及多模态输入、视觉布局建模或文档图像理解中的核心挑战（如OCR幻觉、细粒度定位、高分辨率图像处理）。其方法迁移到DIU领域的路径不明确，缺乏对视觉-语言对齐、图像分块压缩或文档结构编码的支持，因此迁移价值有限。
Abstract Recent LLMs have hundreds of billions of parameters consuming vast resources. Furthermore, the so called "AI scaling law" for transformers suggests that the number of parameters must scale linearly with the size of the data. In response, we inquire into efficient LLMs, i.e. those with the fewest parameters that achieve the desired accuracy on a training corpus. Specifically, by comparing theoretical and empirical estimates of the Kullback-Leibler divergence, we derive a natural AI scaling law that the number of parameters in an efficient LLM scales as $D^γ$ where $D$ is the size of the training data and $ γ\in [0.44, 0.72]$, suggesting the existence of more efficient architectures. Against this backdrop, we propose recurrent transformers, combining the efficacy of transformers with the efficiency of recurrent networks, progressively applying a single transformer layer to a fixed-width sliding window across the input sequence. Recurrent transformers (a) run in linear time in the sequence length, (b) are memory-efficient and amenable to parallel processing in large batches, (c) learn to forget history for language tasks, or accumulate history for long range tasks like copy and selective copy, and (d) are amenable to curriculum training to overcome vanishing gradients. In our experiments, we find that recurrent transformers perform favorably on benchmark tests.
Authors B. N. Kausik
Categories Computation and Language
4
Position-Aware Depth Decay Decoding ($D^3$): Boosting Large Language Model Inference Efficiency
Title CN 位置感知的深度衰减解码（$D^3$）：提升大语言模型推理效率
Keywords 动态推理 · 层跳过 · 推理加速 · 位置感知 · 无需训练
Summary 本文提出D^3方法，通过引入基于位置的幂律衰减函数来动态决定每个生成token所使用的网络层数，越靠后的token使用越少的层，从而在无需重训练的前提下显著提升LLM推理效率。实验表明该方法在多个规模的Llama模型上均能实现约1.5倍加速，且在GSM8K和BBH等任务上性能下降小于1%。
Reason 该论文提出一种训练无关的动态层跳过方法D^3，根据生成token的位置自适应减少参与计算的网络层数，从而在几乎无性能损失的情况下实现1.5倍推理加速。虽然其针对通用LLM推理优化，且未涉及视觉或多模态内容，但其‘位置感知的计算衰减’思想原则上可迁移到长文档VLM推理中——例如，在处理长序列文档输出时（如逐段生成报告），后续token可能依赖较浅的语义处理，从而允许模型动态压缩深层计算。然而，该方法需与视觉输入结构结合才具实际DIU价值，目前迁移路径不直接，属于弱相关的方法论优化。
Abstract Due to the large number of parameters, the inference phase of Large Language Models (LLMs) is resource-intensive. Unlike traditional model compression, which needs retraining, recent dynamic computation methods show that not all components are required for inference, enabling a training-free pipeline. In this paper, we focus on the dynamic depth of LLM generation. A token-position aware layer skipping framework is proposed to save 1.5x times operations efficiently while maintaining performance. We first observed that tokens predicted later have lower perplexity and thus require less computation. Then, we propose a training-free algorithm called Position-Aware Depth Decay Decoding ($D^3$), which leverages a power-law decay function, $\left\lfloor L \times (α^i) \right\rfloor$, to determine the number of layers to retain when generating token $T_i$. Remarkably, without any retraining, the $D^3$ achieves success across a wide range of generation tasks for the first time. Experiments on large language models (\ie the Llama) with $7 \sim 70$ billion parameters show that $D^3$ can achieve an average 1.5x speedup compared with the full-inference pipeline while maintaining comparable performance with nearly no performance drop ($<1\%$) on the GSM8K and BBH benchmarks.
Authors Siqi Fan, Xuezhi Fang, Xingrun Xing, Peng Han, Shuo Shang, Yequan Wang
Categories Computation and Language
4
MGAS: Multi-Granularity Architecture Search for Trade-Off Between Model Effectiveness and Efficiency
Title CN MGAS：面向模型有效性与效率权衡的多粒度架构搜索
Keywords 神经架构搜索 · 多粒度优化 · 可微分搜索 · 模型压缩 · 自适应剪枝
Summary 本文提出MG-DARTS，一种多粒度可微分架构搜索框架，通过自适应剪枝和分阶段优化，在操作级、滤波器级和权重级联合探索神经网络结构，提升模型在CIFAR和ImageNet上的精度-效率权衡。方法核心是跨粒度的动态保留机制与渐进式重评估策略。
Reason 该论文提出了一种多粒度可微分架构搜索方法（MG-DARTS），在图像分类任务上优化模型精度与参数效率的平衡。虽然其架构搜索机制可能对轻量化文档理解模型设计有一定启发，但整体工作属于通用NAS领域，未针对文档模态特性（如布局、文本结构、OCR交互等）进行建模，也未涉及VLM或多模态输入处理。其创新集中在网络层级的结构优化，迁移至DIU场景的价值有限，属于常规效率优化方向。
Abstract Neural architecture search (NAS) has gained significant traction in automating the design of neural networks. To reduce search time, differentiable architecture search (DAS) reframes the traditional paradigm of discrete candidate sampling and evaluation into a differentiable optimization over a super-net, followed by discretization. However, most existing DAS methods primarily focus on optimizing the coarse-grained operation-level topology, while neglecting finer-grained structures such as filter-level and weight-level patterns. This limits their ability to balance model performance with model size. Additionally, many methods compromise search quality to save memory during the search process. To tackle these issues, we propose Multi-Granularity Differentiable Architecture Search (MG-DARTS), a unified framework which aims to discover both effective and efficient architectures from scratch by comprehensively yet memory-efficiently exploring a multi-granularity search space. Specifically, we improve the existing DAS methods in two aspects. First, we adaptively adjust the retention ratios of searchable units across different granularity levels through adaptive pruning, which is achieved by learning granularity-specific discretization functions along with the evolving architecture. Second, we decompose the super-net optimization and discretization into multiple stages, each operating on a sub-net, and introduce progressive re-evaluation to enable re-pruning and regrowth of previous units, thereby mitigating potential bias. Extensive experiments on CIFAR-10, CIFAR-100 and ImageNet demonstrate that MG-DARTS outperforms other state-of-the-art methods in achieving a better trade-off between model accuracy and parameter efficiency. Codes are available at https://github.com/lxy12357/MG_DARTS.
Authors Xiaoyun Liu, Divya Saxena, Jiannong Cao, Yuqing Zhao, Penghui Ruan
Categories Artificial Intelligence
4
Geometry of Decision Making in Language Models NeurIPS 2025
Title CN 语言模型决策的几何学
Keywords 语言模型 · 内在维度 · 决策几何 · 表示学习 · 多选题问答
Summary 本文通过大规模分析28个开源Transformer模型，研究语言模型在多选题问答中的决策几何特性。发现早期层在低维流形上处理输入，中间层扩展表示空间，后期层再次压缩至任务相关表示。这表明LLMs隐式地将语言输入投影到与任务决策对齐的低维结构中，为理解模型泛化和推理提供了新的几何视角。
Reason 该论文研究了语言模型在多选题问答任务中隐藏表示的内在维度（ID）变化，揭示了决策过程中表示空间的压缩与扩展模式。虽然对理解LLM推理机制有理论价值，但其研究对象为纯文本模态下的决策几何，未涉及视觉输入或文档结构，与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版建模、细粒度视觉定位等关联较弱。尽管‘表示压缩’的思想可能间接启发未来VLM的设计，但该方法本身难以直接迁移至多模态文档场景。
Abstract Large Language Models (LLMs) show strong generalization across diverse tasks, yet the internal decision-making processes behind their predictions remain opaque. In this work, we study the geometry of hidden representations in LLMs through the lens of \textit{intrinsic dimension} (ID), focusing specifically on decision-making dynamics in a multiple-choice question answering (MCQA) setting. We perform a large-scale study, with 28 open-weight transformer models and estimate ID across layers using multiple estimators, while also quantifying per-layer performance on MCQA tasks. Our findings reveal a consistent ID pattern across models: early layers operate on low-dimensional manifolds, middle layers expand this space, and later layers compress it again, converging to decision-relevant representations. Together, these results suggest LLMs implicitly learn to project linguistic inputs onto structured, low-dimensional manifolds aligned with task-specific decisions, providing new geometric insights into how generalization and reasoning emerge in language models.
Authors Abhinav Joshi, Divyanshu Bhatt, Ashutosh Modi
Comment Accepted at NeurIPS 2025
Categories Artificial Intelligence, Computation and Language
4
STAlloc: Enhancing Memory Efficiency in Large-Scale Model Training with Spatio-Temporal Planning
Title CN STAlloc：通过时空规划提升大规模模型训练中的内存效率
Keywords GPU内存优化 · 内存碎片 · 大规模模型训练 · 时空规划 · PyTorch
Summary 随着大模型规模的扩大，GPU内存压力加剧，尤其是虚拟流水线和重计算等优化技术导致张量生命周期被打乱，引发严重内存碎片。STAlloc提出一种新的内存分配范式，结合离线的时空规律建模与在线分配机制，在PyTorch中实现了极低开销下的内存碎片大幅降低（平均减少85.1%），显著提升了训练吞吐性能。
Reason 该论文提出了一种针对深度学习框架的GPU内存分配器STAlloc，通过结合离线规划与在线分配来减少训练过程中的内存碎片，从而提升内存利用率和训练吞吐量。虽然其方法对大模型训练具有系统级优化价值，但属于底层系统优化技术，未直接涉及文档图像理解（DIU）任务本身（如DocVQA、布局分析等），也未提出可迁移至DIU领域的新型多模态建模、视觉表示学习或推理机制。其技术聚焦于训练效率而非模型结构或推理能力改进，对解决OCR幻觉、复杂排版理解、长文档推理等DIU核心痛点缺乏直接迁移路径。
Abstract The rapid scaling of large language models (LLMs) has significantly increased GPU memory pressure, which is further aggravated by training optimization techniques such as virtual pipeline and recomputation that disrupt tensor lifespans and introduce considerable memory fragmentation. Such fragmentation stems from the use of online GPU memory allocators in popular deep learning frameworks like PyTorch, which disregard tensor lifespans. As a result, this inefficiency can waste as much as 43% of memory and trigger out-of-memory errors, undermining the effectiveness of optimization methods. To address this, we introduce STAlloc, a GPU memory allocator for deep learning frameworks that reduces fragmentation by exploiting the spatial and temporal regularity in memory allocation behaviors of training workloads. STAlloc introduces a novel paradigm that combines offline planning with online allocation. The offline planning leverages spatio-temporal regularities to generate a near-optimal allocation plan, while the online allocation handles complex and dynamic models such as Mixture-of-Experts (MoE). Built as a pluggable PyTorch memory allocator, STAlloc reduces fragmentation ratio on average by 85.1% (up to 100%) across both dense and MoE models, with negligible overhead. This enables more efficient, high-throughput training configurations and improves throughput performance by up to 32.5%.
Authors Zixiao Huang, Junhao Hu, Hao Lin, Chunyang Zhu, Yueran Tang, Quanlu Zhang, Zhen Guo, Zhenhua Li, Shengen Yan, Zhenhua Zhu, Guohao Dai, Yu Wang
Categories Artificial Intelligence
4
Optimally Deep Networks - Adapting Model Depth to Datasets for Superior Efficiency
Title CN 最优深度网络——通过调整模型深度以适应数据集，实现卓越效率
Keywords 最优深度 · 模型压缩 · 渐进训练 · 神经架构搜索 · 高效推理
Summary 本文提出了最优深度网络（ODN）的概念，通过从浅层开始逐步增加网络深度的训练方式，在保证精度的同时显著降低模型内存占用和计算成本。实验表明，在MNIST和SVHN等简单数据集上，ResNet-18和ResNet-34仅需极小的深度即可达到高性能，内存 Footprint 最多减少98.64%。该方法适用于低复杂度任务的高效模型训练。
Reason 该论文提出了一种渐进式扩展深度的训练策略（Progressive Depth Expansion），旨在根据任务复杂度自适应地确定神经网络的最优深度，从而减少冗余计算和内存占用。虽然方法在通用视觉任务上验证有效，且对资源受限场景下的模型部署有积极意义，但其核心贡献属于模型效率优化范畴，并未针对文档图像理解中的关键挑战（如OCR幻觉、复杂版面分析、细粒度文本定位或长文档推理）进行设计或验证。尽管此类效率技术理论上可被迁移到DIU系统中用于轻量化VLM或OCR主干网络，但迁移路径不直接，创新点与文档理解任务本身关联较弱。
Abstract Deep neural networks (DNNs) have provided brilliant performance across various tasks. However, this success often comes at the cost of unnecessarily large model sizes, high computational demands, and substantial memory footprints. Typically, powerful architectures are trained at full depths but not all datasets or tasks require such high model capacity. Training big and deep architectures on relatively low-complexity datasets frequently leads to wasted computation, unnecessary energy consumption, and excessive memory usage, which in turn makes deployment of models on resource-constrained devices impractical. To address this problem, we introduce the concept of Optimally Deep Networks (ODNs), which provides a balance between model depth and task complexity. Specifically, we propose a NAS like training strategy called progressive depth expansion, which begins by training neural networks at shallower depths and incrementally increases their depth as the earlier blocks converge, continuing this process until the target accuracy is reached. ODNs use only the optimal depth for the tasks at hand, removing redundant layers. This cuts down future training and inference costs, lowers the model memory footprint, enhances computational efficiency, and facilitates deployment on edge devices. Empirical results show that the optimal depths of ResNet-18 and ResNet-34 for MNIST and SVHN, achieve up to 98.64 % and 96.44 % reduction in memory footprint, while maintaining a competitive accuracy of 99.31 % and 96.08 %, respectively.
Authors Shaharyar Ahmed Khan Tareen, Filza Khan Tareen
Comment 6 pages, 4 figures, 1 table, 2 equations, 1 algorithm
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
4
Length-MAX Tokenizer for Language Models
Title CN 面向语言模型的长度最优分词器
Keywords 分词器 · 语言模型 · 效率优化 · 图划分 · token压缩
Summary 本文提出Length-MAX分词器，通过将词汇选择建模为图划分问题，优化平均token长度而非仅频率，显著减少了语言模型训练和推理所需的token数量。实验显示其相比BPE可减少13–18%的token数，加快训练速度、降低延迟，并在多个下游任务上取得更好表现。然而，该工作聚焦于纯文本建模效率，未涉及视觉或多模态内容。
Reason 该论文提出了一种新的文本分词方法Length-MAX，通过最小化平均字符对应的token数量来提升语言模型的训练和推理效率。虽然在通用语言建模中能减少token数、降低延迟并提升吞吐，但其核心贡献集中在纯文本NLP领域，并未涉及多模态或视觉输入处理。对于文档图像理解（DIU）而言，瓶颈主要在于高分辨率视觉编码、OCR幻觉、复杂版式建模等，而非文本分词效率。尽管理论上更高效的tokenizer可能间接加速VLM的文本侧处理，但这种改进属于系统优化层级，对DIU性能提升作用有限，迁移价值较弱。
Abstract We introduce a new tokenizer for language models that minimizes the average tokens per character, thereby reducing the number of tokens needed to represent text during training and to generate text during inference. Our method, which we refer to as the Length-MAX tokenizer, obtains its vocabulary by casting a length-weighted objective maximization as a graph partitioning problem and developing a greedy approximation algorithm. On FineWeb and diverse domains, it yields 14--18\% fewer tokens than Byte Pair Encoding (BPE) across vocabulary sizes from 10K to 50K, and the reduction is 13.0\% when the size is 64K. Training GPT-2 models at 124M, 355M, and 1.3B parameters from scratch with five runs each shows 18.5\%, 17.2\%, and 18.5\% fewer steps, respectively, to reach a fixed validation loss, and 13.7\%, 12.7\%, and 13.7\% lower inference latency, together with a 16\% throughput gain at 124M, while consistently improving on downstream tasks including reducing LAMBADA perplexity by 11.7\% and enhancing HellaSwag accuracy by 4.3\%. Moreover, the Length-MAX tokenizer achieves 99.62\% vocabulary coverage and the out-of-vocabulary rate remains low at 0.12\% on test sets. These results demonstrate that optimizing for average token length, rather than frequency alone, offers an effective approach to more efficient language modeling without sacrificing -- and often improving -- downstream performance. The tokenizer is compatible with production systems and reduces embedding and KV-cache memory by 18\% at inference.
Authors Dong Dong, Weijie Su
Categories Artificial Intelligence, Computation and Language
4
Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalization
Title CN 面向多模态领域泛化的模态平衡协同蒸馏
Keywords 多模态学习 · 领域泛化 · 知识蒸馏 · 权重平均 · 模态不平衡
Summary 本文提出MBCD框架，通过自适应模态dropout、梯度一致性约束和基于权重平均的跨模态知识蒸馏，缓解多模态学习中因不同模态收敛速度差异导致的优化偏差问题。实验表明该方法在多个MMDG基准上提升了跨域泛化性能。
Reason 该论文研究多模态领域泛化（MMDG）中的模态不平衡问题，提出通过协同蒸馏和权重平均来改善跨模态融合。虽然其动机涉及多模态学习中的优化不均衡问题，在理论上可能对VLM训练有启发，但方法聚焦于通用MMDG场景（如RGB-D图像、多视角视觉），并未针对文档图像中特有的结构、布局或OCR相关幻觉问题设计，迁移至DIU的路径不明确。例如，文档理解中的核心挑战是高分辨率细粒度识别、文本-布局联合建模等，而该方法未涉及视觉token压缩、长序列处理或多模态对齐机制的改进，因此对DIU的实际增益有限。
Abstract Weight Averaging (WA) has emerged as a powerful technique for enhancing generalization by promoting convergence to a flat loss landscape, which correlates with stronger out-of-distribution performance. However, applying WA directly to multi-modal domain generalization (MMDG) is challenging: differences in optimization speed across modalities lead WA to overfit to faster-converging ones in early stages, suppressing the contribution of slower yet complementary modalities, thereby hindering effective modality fusion and skewing the loss surface toward sharper, less generalizable minima. To address this issue, we propose MBCD, a unified collaborative distillation framework that retains WA's flatness-inducing advantages while overcoming its shortcomings in multi-modal contexts. MBCD begins with adaptive modality dropout in the student model to curb early-stage bias toward dominant modalities. A gradient consistency constraint then aligns learning signals between uni-modal branches and the fused representation, encouraging coordinated and smoother optimization. Finally, a WA-based teacher conducts cross-modal distillation by transferring fused knowledge to each uni-modal branch, which strengthens cross-modal interactions and steer convergence toward flatter solutions. Extensive experiments on MMDG benchmarks show that MBCD consistently outperforms existing methods, achieving superior accuracy and robustness across diverse unseen domains.
Authors Xiaohan Wang, Zhangtao Cheng, Ting Zhong, Leiting Chen, Fan Zhou
Categories Computer Vision and Pattern Recognition
4
ABM-LoRA: Activation Boundary Matching for Fast Convergence in Low-Rank Adaptation
Title CN ABM-LoRA：用于低秩适应中快速收敛的激活边界匹配
Keywords LoRA · 参数高效微调 · 激活对齐 · 快速收敛 · 视觉表示学习
Summary 本文提出了ABM-LoRA，一种基于激活边界匹配的LoRA初始化策略，旨在解决因随机初始化导致的信息丢失和收敛缓慢问题。通过在下游任务训练前对齐适配器与预训练模型的激活范围，使梯度更有效地投影到低秩子空间，从而实现更快的收敛和更低的初始损失。实验涵盖NLP和CV多个任务，在VTAB-1K上表现优异，尤其在需要几何理解的结构化推理任务中。
Reason 该论文提出了一种改进LoRA初始化的方法，通过匹配预训练模型与适配器之间的激活边界来加速收敛。虽然在ViT等视觉模型上进行了验证，但其核心贡献属于参数高效微调（PEFT）方法的优化，属于通用训练技巧，并未针对文档图像理解中的特定痛点（如OCR幻觉、复杂排版、细粒度定位或长文档推理）进行设计。尽管该方法理论上可被迁移到DIU任务中以加快VLM微调速度，但缺乏对文档结构建模或多模态对齐机制的直接改进，迁移价值有限。
Abstract We propose Activation Boundary Matching for Low-Rank Adaptation (ABM-LoRA), a principled initialization strategy that substantially accelerates the convergence of low-rank adapters. While LoRA offers high parameter efficiency, its random initialization restricts gradient updates to a mismatched tangent space, causing significant information loss and hindering early convergence. Our ABM-LoRA addresses this by aligning the adapter's activation boundaries with those of the pretrained model before downstream training, thereby maximizing the projection of full-parameter gradients into the adapter subspace. This alignment sharply reduces information loss at initialization, yields a lower starting loss, and accelerates convergence. We demonstrate ABM-LoRA's effectiveness across diverse architectures and tasks: language understanding (T5-Base on GLUE), dialogue generation (LLaMA2-7B on WizardLM), and vision recognition (ViT-B/16 on VTAB-1K). On VTAB-1K, it achieves the highest accuracy among all methods, with strong gains on structured reasoning tasks requiring geometric understanding.
Authors Dongha Lee, Jinhee Park, Minjun Kim, Junseok Kwon
Comment 16 pages, 5 figures, under review
Categories Computer Vision and Pattern Recognition
4
Low-Resolution Editing is All You Need for High-Resolution Editing
Title CN 低分辨率编辑就是你所需要的高分辨率编辑
Keywords 高分辨率图像编辑 · 分块优化 · 细节迁移 · 测试时优化 · 视觉一致性
Summary 本文提出了一个面向高分辨率图像编辑的测试时优化框架，通过对图像分块进行低分辨率优化，再结合细粒度细节迁移和同步策略，实现高质量的高分辨率内容修改。实验表明该方法在多种编辑任务中表现良好。
Reason 该论文聚焦于通用高分辨率图像编辑任务，提出通过分块优化与细节迁移实现高分辨率图像的可控编辑。尽管其处理高分辨率输入的能力具有一定技术价值，但方法面向的是图形学与通用视觉内容编辑，未涉及文本理解、布局分析或OCR相关建模，与文档图像理解中的核心痛点（如密集文本识别、排版结构解析、VQA去幻觉）关联较弱。迁移到DIU场景存在显著障碍：文档图像强调语义精确性与字符级对齐，而该方法侧重视觉一致性，难以直接用于解决细粒度文本定位或逻辑推理问题。
Abstract High-resolution content creation is rapidly emerging as a central challenge in both the vision and graphics communities. While images serve as the most fundamental modality for visual expression, content generation that aligns with the user intent requires effective, controllable high-resolution image manipulation mechanisms. However, existing approaches remain limited to low-resolution settings, typically supporting only up to 1K resolution. In this work, we introduce the task of high-resolution image editing and propose a test-time optimization framework to address it. Our method performs patch-wise optimization on high-resolution source images, followed by a fine-grained detail transfer module and a novel synchronization strategy to maintain consistency across patches. Extensive experiments show that our method produces high-quality edits, facilitating the way toward high-resolution content creation.
Authors Junsung Lee, Hyunsoo Lee, Yong Jae Lee, Bohyung Han
Comment 14 pages, 8 figures, 2 tables
Categories Computer Vision and Pattern Recognition
4
Activator: GLU Activation Function as the Core Component of a Vision Transformer
Title CN Activator：以GLU激活函数作为视觉Transformer的核心组件
Keywords 视觉Transformer · GLU激活函数 · 模型效率 · 注意力替代 · 架构优化
Summary 本文探索了将门控线性单元（GLU）作为视觉Transformer核心组件的可能性，取代传统的MLP与注意力机制，以减少计算开销。实验表明该方法在降低复杂度的同时保持了竞争力的性能。然而，该工作属于通用视觉Transformer的效率优化，未涉及文档理解任务特有的挑战，如文本布局建模或多模态对齐。
Reason 该论文提出用GLU激活函数替代传统Transformer中的MLP和注意力机制，旨在降低计算复杂度。虽然在架构效率上有所改进，但其方法属于通用Transformer优化，并未针对文档图像理解（DIU）中的具体痛点（如OCR幻觉、复杂布局、细粒度定位或长文档推理）进行设计或验证。尽管高效架构理论上可被迁移用于文档VLM，但缺乏对多模态对齐、高分辨率输入处理或视觉token压缩等DIU关键问题的直接支持，迁移价值有限。
Abstract The transformer architecture has driven many successes in a variety of tasks within the field of deep learning, in particular the recent advances in natural language processing (NLP) culminating with large language models (LLM). Adding to that success, transformer architecture has found widespread interest from computer vision (CV) researchers and practitioners, allowing for many advancements in vision-related tasks and opening the door for multitask and multi-modal deep learning architectures that share the same principle of operation. One drawback to these architectures is their reliance on the scaled dot product attention mechanism with the softmax activation function, which is computationally expensive and requires large compute capabilities for both training and inference. This paper investigates substituting the MLP and attention mechanism usually adopted for transformer architecture with an architecture based on incorporating a gated linear unit (GLU) activation function structure with the aim of reducing the computational cost. The equalized experimental assessments conducted in this work show that the proposed modification with the targeted reductions in computational complexity offers competitive performance compared to the selected baseline architectures. The results are significantly in support of the aims of this work, in which the focus was to extensively utilize GLU-based MLPs, establishing a more efficient but capable alternative to the traditional MLP and the attention mechanism as the core component in the design of transformer architectures.
Authors Abdullah Nazhat Abdullah, Tarkan Aydin
Categories Computer Vision and Pattern Recognition
4
Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)
Title CN 为什么推理如此重要？多模态推理进展的综述
Keywords 多模态推理 · 大语言模型 · 推理综述 · 测试时推理 · 模态对齐
Summary 本文系统回顾了多模态大模型中的推理技术进展，分析了在图文联合场景下进行有效推理所面临的挑战，如模态间信息冲突、推理一致性等问题，并总结了当前主流的后训练优化与测试时计算策略。文章旨在连接理论框架与实际应用，为未来研究提供方向指引。
Reason 该论文是一篇关于多模态推理技术的综述，涵盖了文本与视觉输入下的推理方法、训练优化和测试时推理策略。虽然其内容涉及CoT、后训练优化和跨模态对齐等可能对DIU有启发的技术方向，但本身并无具体方法创新，属于领域综述而非源头技术工作。对于文档图像理解而言，其中提到的推理机制（如处理模态冲突、提升推理连贯性）理论上可迁移用于缓解OCR幻觉或长文档逻辑推理问题，但作为综述文章缺乏可直接复现或集成的方法模块，迁移价值有限。
Abstract Reasoning is central to human intelligence, enabling structured problem-solving across diverse tasks. Recent advances in large language models (LLMs) have greatly enhanced their reasoning abilities in arithmetic, commonsense, and symbolic domains. However, effectively extending these capabilities into multimodal contexts-where models must integrate both visual and textual inputs-continues to be a significant challenge. Multimodal reasoning introduces complexities, such as handling conflicting information across modalities, which require models to adopt advanced interpretative strategies. Addressing these challenges involves not only sophisticated algorithms but also robust methodologies for evaluating reasoning accuracy and coherence. This paper offers a concise yet insightful overview of reasoning techniques in both textual and multimodal LLMs. Through a thorough and up-to-date comparison, we clearly formulate core reasoning challenges and opportunities, highlighting practical methods for post-training optimization and test-time inference. Our work provides valuable insights and guidance, bridging theoretical frameworks and practical implementations, and sets clear directions for future research.
Authors Jing Bi, Susan Liang, Xiaofei Zhou, Pinxin Liu, Junjia Guo, Yunlong Tang, Luchuan Song, Chao Huang, Ali Vosoughi, Guangyu Sun, Jinxi He, Jiarui Wu, Shu Yang, Daoan Zhang, Chen Chen, Lianggong Bruce Wen, Zhang Liu, Jiebo Luo, Chenliang Xu
Categories Computation and Language
4
Toward Honest Language Models for Deductive Reasoning
Title CN 面向诚实语言模型的演绎推理
Keywords 演绎推理 · 诚实性 · 强化学习 · 训练稳定性 · 逻辑幻觉
Summary 本文提出在演绎推理中衡量语言模型‘诚实性’的任务框架，构建了包含不可回答案例的合成数据集，并发现现有训练方法（如GRPO）在负奖励主导下易训练崩溃。为此提出ACNCHOR方法，通过在强化学习rollout中注入真实轨迹来稳定训练过程，提升模型在可回答性判断和正确推理上的表现。
Reason 该论文研究的是语言模型在演绎推理任务中的‘诚实性’问题，即仅在前提充分时才作答，否则 abstain。其核心贡献是提出一种强化学习方法 ACNCHOR 来稳定训练动态，防止早期训练崩溃。虽然其关注的‘幻觉’问题与文档图像理解（DIU）中的OCR或VLM幻觉有概念上的交集，但该工作完全基于纯文本逻辑推理任务，未涉及任何视觉输入、多模态对齐或文档结构建模。其方法迁移至DIU场景（如DocVQA中的幻觉抑制）需极大改造，且缺乏对细粒度定位、高分辨率处理等DIU核心痛点的支持，因此迁移价值有限。
Abstract Deductive reasoning is the process of deriving conclusions strictly from the given premises, without relying on external knowledge. We define honesty in this setting as a model's ability to respond only when the conclusion is logically entailed by the premises, and to abstain otherwise. However, current language models often fail to reason honestly, producing unwarranted answers when the input is insufficient. To study this challenge, we formulate honest deductive reasoning as multi-step tasks where models must either derive the correct conclusion or abstain. We curate two datasets from graph structures, one for linear algebra and one for logical inference, and introduce unanswerable cases by randomly perturbing an edge in half of the instances. We find that prompting and existing training methods, including GRPO with or without supervised fine-tuning initialization, struggle on these tasks. In particular, GRPO optimize only for final task outcomes, leaving models vulnerable to collapse when negative rewards dominate early training. To address this, we propose ACNCHOR, a reinforcement learning method that injects ground truth trajectories into rollouts, preventing early training collapse. Our results demonstrate that this method stabilizes learning and significantly improves the overall reasoning performance, underscoring the importance of training dynamics for enabling honest deductive reasoning in language models.
Authors Jiarui Liu, Kaustubh Dhole, Yingheng Wang, Haoyang Wen, Sarah Zhang, Haitao Mao, Gaotang Li, Neeraj Varshney, Jingguo Liu, Xiaoman Pan
Categories Computation and Language
4
SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compression
Title CN SWAN：通过无解压KV缓存压缩实现推理内存减少的稀疏筛选注意力
Keywords KV缓存压缩 · 推理优化 · 内存效率 · 稀疏注意力 · 大语言模型
Summary 本文提出SWAN框架，通过离线正交矩阵旋转和剪枝KV缓存，在无需重建和微调的前提下实现高效的注意力计算，显著降低推理内存消耗。方法支持运行时动态调节压缩率，在50-60%内存节省下仍保持接近原始模型的性能。
Reason 该论文提出了一种针对大语言模型推理过程中KV缓存内存占用过大的问题，设计了无需微调、无解压开销的KV缓存压缩方法。虽然其在长上下文推理场景下具有优势，但属于通用LLM推理优化技术，并未针对文档图像理解中的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）进行建模或验证。尽管理论上可用于支持长文档VLM的高效推理，但迁移路径不直接，且缺乏与视觉输入或文档结构的关联设计，因此对DIU的针对性较弱。
Abstract Large Language Models (LLMs) face a significant bottleneck during autoregressive inference due to the massive memory footprint of the Key-Value (KV) cache. Existing compression techniques like token eviction, quantization, or other low-rank methods often risk information loss, have fixed limits, or introduce significant computational overhead from explicit decompression steps. In this work, we introduce SWAN, a novel, fine-tuning-free framework that eliminates this overhead. Our method uses an offline orthogonal matrix to rotate and prune the KV-cache, which is then used directly in the attention computation without any reconstruction. Our extensive experiments demonstrate that SWAN, augmented with a small dense buffer, offers a robust trade-off, maintaining performance close to the uncompressed baseline even at aggressive 50-60% memory savings per-token on KV-cache. A key advantage is its runtime-tunable compression level, allowing operators to dynamically adjust the memory footprint, a flexibility absent in methods requiring fixed offline configurations. This combination of a decompression-free design, high performance under compression, and adaptability makes SWAN a practical and efficient solution for serving LLMs with long contexts.
Authors Santhosh G S, Saurav Prakash, Balaraman Ravindran
Categories Artificial Intelligence, Computation and Language
4
VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RL
Title CN VADE：基于在线样本级难度估计的方差感知动态采样用于多模态强化学习
Keywords 多模态强化学习 · 动态采样 · 难度估计 · 梯度消失 · 组策略优化
Summary 本文提出了VADE，一种方差感知的动态采样框架，通过在线估计样本难度、Thompson采样最大化信息增益以及双尺度先验衰减机制，动态选择最具信息量的样本进行训练，缓解了组策略优化中的梯度消失问题。实验表明其在多模态推理任务上提升了性能与样本效率，并可作为插件集成到现有算法中。
Reason 该论文提出了一种用于多模态强化学习中的动态采样框架VADE，旨在解决组策略优化中因奖励相同导致的梯度消失问题。虽然其方法在训练效率和信号强度上有提升，属于上游方法，但其核心关注的是推理过程中的采样策略优化，迁移到文档图像理解（DIU）场景的路径不明确。尽管DIU中存在长文档推理或复杂布局下的信息检索问题，但VADE依赖于对‘回答正确性概率’的建模与Thompson采样机制，在缺乏明确定义的‘多模态问答动作空间’和细粒度反馈信号的DIU任务中难以直接应用。因此，虽有一定方法论价值，但对OCR幻觉、密集文本识别、排版理解等核心痛点帮助有限。
Abstract Group-based policy optimization methods like GRPO and GSPO have become standard for training multimodal models, leveraging group-wise rollouts and relative advantage estimation. However, they suffer from a critical \emph{gradient vanishing} problem when all responses within a group receive identical rewards, causing advantage estimates to collapse and training signals to diminish. Existing attempts to mitigate this issue fall into two paradigms: filtering-based and sampling-based methods. Filtering-based methods first generate rollouts broadly and then retroactively filter out uninformative groups, leading to substantial computational overhead. Sampling-based methods proactively select effective samples before rollout but rely on static criteria or prior dataset knowledge, lacking real-time adaptability. To address these issues, we propose \textbf{VADE}, a \textbf{V}ariance-\textbf{A}ware \textbf{D}ynamic sampling framework via online sample-level difficulty \textbf{E}stimation. Our framework integrates three key components: online sample-level difficulty estimation using Beta distributions, a Thompson sampler that maximizes information gain through the estimated correctness probability, and a two-scale prior decay mechanism that maintains robust estimation under policy evolution. This three components design enables VADE to dynamically select the most informative samples, thereby amplifying training signals while eliminating extra rollout costs. Extensive experiments on multimodal reasoning benchmarks show that VADE consistently outperforms strong baselines in both performance and sample efficiency, while achieving a dramatic reduction in computational overhead. More importantly, our framework can serves as a plug-and-play component to be seamlessly integrated into existing group-based RL algorithms. Code and models are available at https://VADE-RL.github.io.
Authors Zengjie Hu, Jiantao Qiu, Tianyi Bai, Haojin Yang, Binhang Yuan, Qi Jing, Conghui He, Wentao Zhang
Categories Artificial Intelligence
4
VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunking
Title CN 闪存中的视觉语言模型：通过神经元分块实现I/O高效的VLM稀疏化
Keywords 视觉语言模型 · 模型压缩 · 边缘计算 · I/O优化 · 神经元稀疏化
Summary 本文提出Neuron Chunking，一种结合神经元重要性和存储访问延迟的VLM稀疏化策略，通过以“连续内存块”为单位进行激活剪枝，显著提升了边缘设备上VLM的I/O效率。该方法从系统层面优化了权重卸载过程，适用于资源受限场景下的多模态模型部署。
Reason 该论文聚焦于VLM在边缘设备上的部署效率，提出了一种考虑存储访问模式的神经元稀疏化方法Neuron Chunking，以降低闪存I/O开销。虽然其核心是VLM推理优化，但方法本身属于模型压缩与系统协同设计范畴，并未针对文档图像理解中的关键问题（如OCR幻觉、复杂布局解析、细粒度定位或长文档推理）进行建模改进。其技术迁移至DIU场景的价值有限，除非未来DIU模型在端侧部署成为瓶颈，否则对提升文档理解性能无直接帮助。
Abstract Edge deployment of large Vision-Language Models (VLMs) increasingly relies on flash-based weight offloading, where activation sparsification is used to reduce I/O overhead. However, conventional sparsification remains model-centric, selecting neurons solely by activation magnitude and neglecting how access patterns influence flash performance. We present Neuron Chunking, an I/O-efficient sparsification strategy that operates on chunks (i.e., groups of contiguous neurons in memory) and couples neuron importance with storage access cost. The method models I/O latency through a lightweight abstraction of access contiguity and selects chunks with high utility, defined as neuron importance normalized by estimated latency. By aligning sparsification decisions with the underlying storage behavior, Neuron Chunking improves I/O efficiency by up to 4.65x and 5.76x on Jetson Orin Nano and Jetson AGX Orin, respectively.
Authors Kichang Yang, Seonjun Kim, Minjae Kim, Nairan Zhang, Chi Zhang, Youngki Lee
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
4
Low-Rank GEMM: Efficient Matrix Multiplication via Low-Rank Approximation with FP8 Acceleration
Title CN 低秩GEMM：通过低秩近似与FP8加速实现高效矩阵乘法
Keywords 矩阵乘法 · 低秩近似 · FP8量化 · 计算效率 · 硬件加速
Summary 本文提出Low-Rank GEMM方法，利用低秩分解（如SVD）结合FP8量化，在NVIDIA RTX 4090上实现了高达378 TFLOPS的矩阵乘法性能，显著优于PyTorch FP32实现，尤其在大矩阵（N≥10240）场景下具备速度和内存优势。系统能根据硬件自动选择最优分解策略与精度配置。
Reason 该论文聚焦于底层线性代数计算优化，提出利用低秩近似和FP8精度加速大规模矩阵乘法。虽然其在硬件效率和内存节省方面表现突出，但属于通用深度学习计算优化技术，并未针对文档图像理解中的具体问题（如OCR幻觉、复杂排版建模或细粒度视觉定位）进行设计。尽管理论上可用于加速VLM推理中的注意力计算，迁移路径间接且无明确机制适配，对DIU性能提升缺乏直接贡献。
Abstract Large matrix multiplication is a cornerstone of modern machine learning workloads, yet traditional approaches suffer from cubic computational complexity (e.g., $\mathcal{O}(n^3)$ for a matrix of size $n\times n$). We present Low-Rank GEMM, a novel approach that leverages low-rank matrix approximations to achieve sub-quadratic complexity while maintaining hardware-accelerated performance through FP8 precision and intelligent kernel selection. On a NVIDIA RTX 4090, our implementation achieves up to 378 TFLOPS on matrices up to $N=20480$, providing 75\% memory savings and $7.8\times$ speedup over PyTorch FP32 for large matrices. The system automatically adapts to hardware capabilities, selecting optimal decomposition methods (SVD, randomized SVD) and precision levels based on matrix characteristics and available accelerators. Comprehensive benchmarking on NVIDIA RTX 4090 demonstrates that Low-Rank GEMM becomes the fastest approach for matrices $N\geq10240$, surpassing traditional cuBLAS implementations through memory bandwidth optimization rather than computational shortcuts.
Authors Alfredo Metere
Categories Artificial Intelligence
4
Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference
Title CN 面向可扩展MoE推理的动态专家量化
Keywords MoE · 动态量化 · 推理优化 · 内存效率 · 大模型部署
Summary 本文提出DynaExq，一种在MoE模型推理过程中动态管理专家精度的运行时系统。通过热度感知的精度控制器、异步精度切换流水线和无碎片内存池机制，实现显存受限下稳定高效的多精度专家共存。实验表明其可在单张消费级GPU上部署大型MoE模型，并相比静态量化提升最多4.03点准确率。
Reason 该论文聚焦于Mixture-of-Experts模型在推理时的内存效率优化，提出了一种动态调整专家精度的方法，以降低显存占用并提升部署效率。虽然对大模型部署有实用价值，但其核心贡献属于系统优化与量化工程范畴，并未针对文档图像理解中的关键问题（如OCR幻觉、复杂布局建模、细粒度视觉定位或长文档逻辑推理）提出直接可迁移的方法创新。其技术路径难以直接缓解VLM在文档场景下的语义幻觉或多步推理偏差，也不涉及视觉表征学习或高分辨率处理等DIU核心挑战。
Abstract Mixture-of-Experts (MoE) models scale LLM capacity efficiently, but deployment on consumer GPUs is limited by the large memory footprint of inactive experts. Static post-training quantization reduces storage costs but cannot adapt to shifting activation patterns, causing accuracy loss under aggressive compression. So we present DynaExq, a runtime system that treats expert precision as a first-class, dynamically managed resource. DynaExq combines (1) a hotness-aware precision controller that continuously aligns expert bit-widths with long-term activation statistics, (2) a fully asynchronous precision-switching pipeline that overlaps promotion and demotion with MoE computation, and (3) a fragmentation-free memory pooling mechanism that supports hybrid-precision experts with deterministic allocation. Together, these components enable stable, non-blocking precision transitions under strict HBM budgets. Across Qwen3-30B and Qwen3-80B MoE models and six representative benchmarks, DynaExq deploys large LLMs on single RTX 5090 and A6000 GPUs and improves accuracy by up to 4.03 points over static low-precision baselines. The results show that adaptive, workload-aware quantization is an effective strategy for memory-constrained MoE serving.
Authors Kexin Chu, Dawei Xiang, Zixu Shen, Yiwei Yang, Zecheng Liu, Wei Zhang
Comment 7 pages
Categories Artificial Intelligence
4
AttenDence: Maximizing Attention Confidence for Test Time Adaptation
Title CN AttenDence：最大化注意力置信度以实现测试时自适应
Keywords Test-time Adaptation · Attention Entropy · Robustness · Transformer · Unsupervised Learning
Summary 本文提出AttenDence方法，利用Transformer模型中CLS token与图像块之间的注意力分布熵作为无监督信号，在测试阶段优化模型对分布偏移的适应能力。实验表明该方法在多种图像退化类型下提升了鲁棒性，同时不损害干净样本的性能。
Reason 该论文提出通过最小化CLS token对图像块的注意力分布熵来增强Transformer在测试时自适应（TTA）中的鲁棒性。虽然方法在通用视觉领域有一定创新，但其核心关注的是自然图像在分布偏移下的鲁棒推理，并未涉及文档特有的结构、排版或语言-视觉对齐问题。尽管注意力置信度机制理论上可用于缓解OCR幻觉或提升定位稳定性，但迁移路径不直接，且无针对文本密集、高分辨率文档的验证，对DIU的实用价值有限。
Abstract Test-time adaptation (TTA) enables models to adapt to distribution shifts at inference time. While entropy minimization over the output distribution has proven effective for TTA, transformers offer an additional unsupervised learning signal through their attention mechanisms. We propose minimizing the entropy of attention distributions from the CLS token to image patches as a novel TTA objective.This approach encourages the model to attend more confidently to relevant image regions under distribution shift and is effective even when only a single test image is available. We demonstrate that attention entropy minimization improves robustness across diverse corruption types while not hurting performance on clean data on a single sample stream of images at test time.
Authors Yash Mali
Comment Initial submission. 5 pages, 4 figures
Categories Computer Vision and Pattern Recognition
4
Rethinking Vision Transformer Depth via Structural Reparameterization
Title CN 通过结构重参数化重新思考视觉Transformer的深度
Keywords Vision Transformer · 模型压缩 · 结构重参数化 · 深度缩减 · 推理加速
Summary 本文提出一种结构重参数化方法来压缩Vision Transformer的深度，通过训练时引入并行分支、推理时合并为单路径的方式，在ViT-Tiny上实现了层数从12层压缩至3层的同时保持ImageNet分类精度，并获得最高37%的移动端CPU加速效果。研究表明深层堆叠并非必需，为高效视觉Transformer设计提供了新思路。
Reason 该论文提出一种基于分支结构的重参数化方法，在训练时使用多分支架构，推理时将其合并为单路径模型，从而减少ViT的层数并保持性能。虽然在模型压缩和效率提升方面有一定贡献，但其研究对象是通用图像分类任务下的ViT架构优化，未涉及文本、布局或文档特有的结构信息。尽管模型压缩技术理论上可迁移到文档VLM以提升推理效率，但本文并未针对文档理解中的核心痛点（如OCR幻觉、细粒度定位、复杂排版建模）进行设计，迁移价值有限。
Abstract The computational overhead of Vision Transformers in practice stems fundamentally from their deep architectures, yet existing acceleration strategies have primarily targeted algorithmic-level optimizations such as token pruning and attention speedup. This leaves an underexplored research question: can we reduce the number of stacked transformer layers while maintaining comparable representational capacity? To answer this, we propose a branch-based structural reparameterization technique that operates during the training phase. Our approach leverages parallel branches within transformer blocks that can be systematically consolidated into streamlined single-path models suitable for inference deployment. The consolidation mechanism works by gradually merging branches at the entry points of nonlinear components, enabling both feed-forward networks (FFN) and multi-head self-attention (MHSA) modules to undergo exact mathematical reparameterization without inducing approximation errors at test time. When applied to ViT-Tiny, the framework successfully reduces the original 12-layer architecture to 6, 4, or as few as 3 layers while maintaining classification accuracy on ImageNet-1K. The resulting compressed models achieve inference speedups of up to 37% on mobile CPU platforms. Our findings suggest that the conventional wisdom favoring extremely deep transformer stacks may be unnecessarily restrictive, and point toward new opportunities for constructing efficient vision transformers.
Authors Chengwei Zhou, Vipin Chaudhary, Gourav Datta
Comment 21 pages, 6 figures
Categories Computer Vision and Pattern Recognition
4
Perceptual Taxonomy: Evaluating and Guiding Hierarchical Scene Reasoning in Vision-Language Models
Title CN 感知分类法：评估与引导视觉语言模型中的层次化场景推理
Keywords 视觉语言模型 · 视觉推理 · 基准测试 · 分层理解 · 属性推理
Summary 本文提出了Perceptual Taxonomy，一种用于评估视觉语言模型进行层次化场景理解能力的基准，涵盖物体识别、空间配置及属性推理等层面。基于3173个标注对象和5802张图像，构建了包含2.8万多个模板问题和50个专家设计问题的多选题数据集。实验表明现有VLM在属性驱动和多步推理任务上表现显著下降，暴露了当前模型依赖模式匹配的局限性。通过引入模拟场景中的上下文推理示例，可提升实际场景中的推理性能。
Reason 该论文提出了一种用于评估视觉语言模型在物理世界属性推理（如材质、功能、可操作性）方面能力的基准测试，强调从物体识别到属性推断再到目标导向推理的层次化理解。虽然其核心关注的是通用视觉推理而非文档图像，且任务设定偏向物理场景理解，但其提出的‘分层推理’结构和模板化问题构造方式对DIU领域中需要复杂逻辑推理的任务（如表格功能理解、表单意图识别）有一定启发价值。然而，该工作本身并未涉及文本密集、排版结构或OCR相关挑战，迁移成本高，属于弱关联的上游方法。
Abstract We propose Perceptual Taxonomy, a structured process of scene understanding that first recognizes objects and their spatial configurations, then infers task-relevant properties such as material, affordance, function, and physical attributes to support goal-directed reasoning. While this form of reasoning is fundamental to human cognition, current vision-language benchmarks lack comprehensive evaluation of this ability and instead focus on surface-level recognition or image-text alignment. To address this gap, we introduce Perceptual Taxonomy, a benchmark for physically grounded visual reasoning. We annotate 3173 objects with four property families covering 84 fine-grained attributes. Using these annotations, we construct a multiple-choice question benchmark with 5802 images across both synthetic and real domains. The benchmark contains 28033 template-based questions spanning four types (object description, spatial reasoning, property matching, and taxonomy reasoning), along with 50 expert-crafted questions designed to evaluate models across the full spectrum of perceptual taxonomy reasoning. Experimental results show that leading vision-language models perform well on recognition tasks but degrade by 10 to 20 percent on property-driven questions, especially those requiring multi-step reasoning over structured attributes. These findings highlight a persistent gap in structured visual understanding and the limitations of current models that rely heavily on pattern matching. We also show that providing in-context reasoning examples from simulated scenes improves performance on real-world and expert-curated questions, demonstrating the effectiveness of perceptual-taxonomy-guided prompting.
Authors Jonathan Lee, Xingrui Wang, Jiawei Peng, Luoxin Ye, Zehan Zheng, Tiezheng Zhang, Tao Wang, Wufei Ma, Siyi Chen, Yu-Cheng Chou, Prakhar Kaushik, Alan Yuille
Categories Computer Vision and Pattern Recognition
4
TRIM: Token Reduction and Inference Modeling for Cost-Effective Language Generation
Title CN TRIM：用于低成本语言生成的令牌缩减与推理建模
Keywords 语言生成优化 · 令牌压缩 · 推理效率 · LLM成本控制 · 文本蒸馏
Summary TRIM提出一种两阶段语言生成优化框架：大模型在推理时跳过语义冗余词生成压缩文本，再由小型专用模型将其还原为完整回答。实验显示可在GPT-4o上平均节省19.4%的token消耗，性能下降微小。该方法聚焦于纯文本生成效率，在不损失语义的前提下减少计算开销。
Reason 该论文提出了一种通过生成压缩语言并由小模型重建来降低LLM推理成本的流水线方法，核心关注自然语言输出中的冗余问题。虽然其动机（减少冗余、提升效率）在长文档理解中有一定共鸣，但该方法完全基于纯文本层面的压缩与恢复，未涉及视觉输入、多模态对齐或文档结构信息处理。对于文档图像理解中的核心挑战如OCR幻觉、高分辨率布局解析或细粒度视觉定位无直接迁移路径。属于DIU边缘相关但迁移价值有限的工作。
Abstract The high inference cost of Large Language Models (LLMs) poses challenges, especially for tasks requiring lengthy outputs. However, natural language often contains redundancy, which presents an opportunity for optimization. We have observed that LLMs can generate distilled language (i.e., concise outputs that retain essential meaning) when prompted appropriately. We propose TRIM, a pipeline for saving computational cost in which the LLM omits a predefined set of semantically irrelevant and easily inferable words based on the context during inference. Then, a specifically trained smaller language model with lower inference cost reconstructs the distilled answer into the ideal answer. Our experiments show promising results, particularly on the proposed NaLDA evaluation dataset focused on the reconstruction task, with 19.4% saved tokens on average for GPT-4o and only a tiny decrease in evaluation metrics. This suggests that the approach can effectively balance efficiency and accuracy in language processing tasks.
Authors Alfredo Garrachón Ruiz, Tomás de la Rosa, Daniel Borrajo
Comment 16 pages, 9 tables, 5 figures
Categories Computation and Language
4
Concept than Document: Context Compression via AMR-based Conceptual Entropy
Title CN 概念而非文档：基于AMR的概念熵上下文压缩
Keywords 上下文压缩 · AMR · 概念熵 · 信息冗余 · 语言模型
Summary 本文提出一种基于AMR图和概念熵的无监督上下文压缩方法，用于在RAG等任务中去除冗余文本，保留关键语义内容。实验表明该方法能在显著缩短上下文的同时提升问答准确率。这是首次将AMR与熵度量结合用于上下文压缩的工作。
Reason 该工作提出了一种基于抽象语义表示（AMR）图的无监督上下文压缩框架，通过计算节点级概念熵来保留语义核心信息。虽然其动机是缓解长上下文的信息过载问题，与文档理解中的长文档推理有一定关联，但其方法完全基于纯文本的AMR解析，未涉及任何视觉或文档布局信息，也不处理OCR、图像模态或多模态对齐等DIU核心问题。因此，尽管‘上下文压缩’在理念上可启发DIU中冗余信息过滤，但该方法本身难以直接迁移至文档图像理解场景。
Abstract Large Language Models (LLMs) face information overload when handling long contexts, particularly in Retrieval-Augmented Generation (RAG) where extensive supporting documents often introduce redundant content. This issue not only weakens reasoning accuracy but also increases computational overhead. We propose an unsupervised context compression framework that exploits Abstract Meaning Representation (AMR) graphs to preserve semantically essential information while filtering out irrelevant text. By quantifying node-level entropy within AMR graphs, our method estimates the conceptual importance of each node, enabling the retention of core semantics. Specifically, we construct AMR graphs from raw contexts, compute the conceptual entropy of each node, and screen significant informative nodes to form a condensed and semantically focused context than raw documents. Experiments on the PopQA and EntityQuestions datasets show that our method outperforms vanilla and other baselines, achieving higher accuracy while substantially reducing context length. To the best of our knowledge, this is the first work introducing AMR-based conceptual entropy for context compression, demonstrating the potential of stable linguistic features in context engineering.
Authors Kaize Shi, Xueyao Sun, Xiaohui Tao, Lin Li, Qika Lin, Guandong Xu
Categories Computation and Language
4
A Comprehensive Survey on Long Context Language Modeling
Title CN 长上下文语言建模综述
Keywords 长上下文建模 · 语言模型 · 综述 · 高效推理 · 序列建模
Summary 本文系统回顾了长上下文语言模型在数据策略、架构设计、训练部署和评估分析方面的最新进展，旨在为研究人员和工程师提供全面的技术参考。文章还提供了持续更新的GitHub资源库。
Reason 该论文是一篇关于长上下文语言模型（LCLM）的综合性综述，涵盖了架构设计、训练部署和评估方法。虽然长上下文处理能力对文档图像理解中的长文档推理有一定关联价值，但本文属于NLP领域的通用技术总结，并未针对多模态或视觉-文本联合建模进行专门探讨，也未提出可直接迁移至DIU场景的新方法。其内容更偏向于纯文本序列建模，缺乏对OCR噪声、布局结构、视觉定位等文档核心问题的考虑，因此迁移价值有限。
Abstract Efficient processing of long contexts has been a persistent pursuit in Natural Language Processing. With the growing number of long documents, dialogues, and other textual data, it is important to develop Long Context Language Models (LCLMs) that can process and analyze extensive inputs in an effective and efficient way. In this paper, we present a comprehensive survey on recent advances in long-context modeling for large language models. Our survey is structured around three key aspects: how to obtain effective and efficient LCLMs, how to train and deploy LCLMs efficiently, and how to evaluate and analyze LCLMs comprehensively. For the first aspect, we discuss data strategies, architectural designs, and workflow approaches oriented with long context processing. For the second aspect, we provide a detailed examination of the infrastructure required for LCLM training and inference. For the third aspect, we present evaluation paradigms for long-context comprehension and long-form generation, as well as behavioral analysis and mechanism interpretability of LCLMs. Beyond these three key aspects, we thoroughly explore the diverse application scenarios where existing LCLMs have been deployed and outline promising future development directions. This survey provides an up-to-date review of the literature on long-context LLMs, which we wish to serve as a valuable resource for both researchers and engineers. An associated GitHub repository collecting the latest papers and repos is available at: \href{https://github.com/LCLM-Horizon/A-Comprehensive-Survey-For-Long-Context-Language-Modeling}{\color[RGB]{175,36,67}{LCLM-Horizon}}.
Authors Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, Yuanxing Zhang, Zhuo Chen, Hangyu Guo, Shilong Li, Ziqiang Liu, Yong Shan, Yifan Song, Jiayi Tian, Wenhao Wu, Zhejian Zhou, Ruijie Zhu, Junlan Feng, Yang Gao, Shizhu He, Zhoujun Li, Tianyu Liu, Fanyu Meng, Wenbo Su, Yingshui Tan, Zili Wang, Jian Yang, Wei Ye, Bo Zheng, Wangchunshu Zhou, Wenhao Huang, Sujian Li, Zhaoxiang Zhang
Categories Computation and Language
4
Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection
Title CN 使用镜像下降优化注意力：广义最大间隔令牌选择
Keywords 注意力机制 · 镜像下降 · 令牌选择 · 优化理论 · SVM
Summary 本文研究了基于镜像下降的注意力优化算法，证明其在分类任务中会收敛到一种广义硬间隔SVM结构，并能提升模型在token选择上的泛化能力。实验显示该方法相比传统梯度下降在真实数据上表现更好。
Reason 该论文研究了镜像下降（MD）在softmax注意力机制中的收敛性与隐式偏置，理论表明其可导向具有ℓ_p范数目标的广义硬间隔SVM解，并在令牌选择上表现更优。虽然其提出的优化方法在理论上对token selection有增益，可能间接有助于VLM中关键视觉/文本token的聚焦，但该工作属于通用注意力机制优化，未针对文档结构、多模态对齐或高分辨率输入等DIU核心痛点设计，迁移路径不直接，创新点偏向理论优化而非解决实际文档理解任务。
Abstract Attention mechanisms have revolutionized several domains of artificial intelligence, such as natural language processing and computer vision, by enabling models to selectively focus on relevant parts of the input data. While recent work has characterized the optimization dynamics of gradient descent (GD) in attention-based models and the structural properties of its preferred solutions, less is known about more general optimization algorithms such as mirror descent (MD). In this paper, we investigate the convergence properties and implicit biases of a family of MD algorithms tailored for softmax attention mechanisms, with the potential function chosen as the $p$-th power of the $\ell_p$-norm. Specifically, we show that these algorithms converge in direction to a generalized hard-margin SVM with an $\ell_p$-norm objective when applied to a classification problem using a softmax attention model. Notably, our theoretical results reveal that the convergence rate is comparable to that of traditional GD in simpler models, despite the highly nonlinear and nonconvex nature of the present problem. Additionally, we delve into the joint optimization dynamics of the key-query matrix and the decoder, establishing conditions under which this complex joint optimization converges to their respective hard-margin SVM solutions. Lastly, our numerical experiments on real data demonstrate that MD algorithms improve generalization over standard GD and excel in optimal token selection.
Authors Addison Kristanto Julistiono, Davoud Ataee Tarzanagh, Navid Azizan
Categories Artificial Intelligence, Computation and Language
4
RNN as Linear Transformer: A Closer Investigation into Representational Potentials of Visual Mamba Models
Title CN 将RNN视为线性Transformer：对视觉Mamba模型表征潜力的深入研究
Keywords Mamba · 线性注意力 · 表征分析 · 自监督学习 · 激活图可视化
Summary 本文系统研究了Mamba模型在视觉领域的表征能力，从理论上证明其可作为Softmax注意力的低秩近似，本质上等价于线性注意力的一种形式。作者提出了一个用于评估激活图的二值分割量化指标，用以衡量模型捕捉长距离依赖的能力，并通过DINO自监督预训练获得了更具可解释性的激活模式。实验显示该模型在ImageNet上线性探针准确率达到78.5%，表明其具有良好的表征学习能力。整体工作偏向基础架构分析，强调可解释性与理论统一性。
Reason 该论文聚焦于分析Mamba在通用视觉任务中的表征机制，主要贡献在于理论解释Mamba与注意力机制的关系、提出新的激活图评估指标，并在ImageNet上验证性能。虽然其对长程依赖建模和高分辨率特征处理有一定启示，但并未针对文档图像理解（DIU）中的具体痛点（如OCR幻觉、复杂版式、细粒度定位）进行设计或验证。其方法属于通用视觉架构分析，迁移到DIU领域的路径不明确，且未涉及多模态语言模型（VLM）或文本-布局联合建模，因此仅具备弱相关性。
Abstract Mamba has recently garnered attention as an effective backbone for vision tasks. However, its underlying mechanism in visual domains remains poorly understood. In this work, we systematically investigate Mamba's representational properties and make three primary contributions. First, we theoretically analyze Mamba's relationship to Softmax and Linear Attention, confirming that it can be viewed as a low-rank approximation of Softmax Attention and thereby bridging the representational gap between Softmax and Linear forms. Second, we introduce a novel binary segmentation metric for activation map evaluation, extending qualitative assessments to a quantitative measure that demonstrates Mamba's capacity to model long-range dependencies. Third, by leveraging DINO for self-supervised pretraining, we obtain clearer activation maps than those produced by standard supervised approaches, highlighting Mamba's potential for interpretability. Notably, our model also achieves a 78.5 percent linear probing accuracy on ImageNet, underscoring its strong performance. We hope this work can provide valuable insights for future investigations of Mamba-based vision architectures.
Authors Timing Yang, Guoyizhe Wei, Alan Yuille, Feng Wang
Categories Computer Vision and Pattern Recognition
4
Prompt Optimization as a State-Space Search Problem
Title CN 将提示优化视为状态空间搜索问题
Keywords 提示优化 · 状态空间搜索 · 束搜索 · 语言模型 · 提示工程
Summary 本文提出将提示优化形式化为状态空间搜索问题，通过构建提示状态图并应用搜索算法（如束搜索）来自动发现更优提示。实验表明即使浅层搜索也能在开发集上提升性能，但存在过拟合风险。主要贡献在于提示工程的方法论框架，而非模型架构或视觉-语言对齐机制。
Reason 该论文将提示词优化建模为状态空间搜索问题，使用束搜索和随机游走探索不同的提示变体，在多个NLP任务上进行验证。虽然方法具有一定的系统性和可扩展性，但其核心是针对纯文本语言模型的提示工程优化，并未涉及多模态或视觉输入。对于文档图像理解（DIU）而言，提示优化可能作为外围技术间接辅助VLM的推理稳定性，但该工作缺乏对OCR噪声、视觉布局、细粒度定位等DIU核心痛点的直接建模，迁移价值有限。
Abstract Language Models are extremely susceptible to performance collapse with even small changes to input prompt strings. Libraries such as DSpy (from Stanford NLP) avoid this problem through demonstration-based prompt optimisation. Inspired by this, I propose an alternative approach that treats prompt optimisation as a classical state-space search problem. I model the prompt space as a graph where nodes represent prompt states and edges correspond to deliberate transformations such as shortening, adding examples, or re- ordering content. Using beam search and random walk algorithms, I systematically explore this space, evaluating candidates on development sets and pruning unpromising branches. Across five NLP tasks (sentiment classification, question answering, summarisation, reason- ing, and natural language inference), I find that even shallow search configurations (beam width=2, depth=2) improve upon seed prompts on development sets. For instance, beam search achieves development accuracy gains from 0.40 to 0.80 on reasoning tasks, though test set improvements are more modest (0.20 to 0.50), indicating overfitting to the develop- ment heuristic. Analysis of successful optimisation paths reveals that transformations that make prompts concise appear most frequently, while verbosity operators are never selected. My results validate prompt optimization as a search problem and suggest that with greater computational resources and improved evaluation metrics, deeper exploration could yield more robust prompts that generalize beyond development sets. Code and implementation are available at [https://github.com/MaanasTaneja/PromptOptimiser].
Authors Maanas Taneja
Categories Computation and Language
4
FAIR-Pruner: Leveraging Tolerance of Difference for Flexible Automatic Layer-Wise Neural Network Pruning CVPR 2026
Title CN FAIR-Pruner：利用差异容忍度实现灵活的自动逐层神经网络剪枝
Keywords 神经网络剪枝 · 模型压缩 · 差异容忍度 · 逐层剪枝 · 高效推理
Summary 本文提出FAIR-Pruner，一种新颖的逐层神经网络剪枝方法，通过引入‘差异容忍度’（ToD）指标，结合架构级和任务级的重要性评分，自适应地确定各层剪枝阈值。该方法支持灵活调整剪枝比例，并在高稀疏度下仍保持较高精度。实验表明其在多种模型和数据集上优于现有剪枝方法。
Reason 该论文属于模型压缩与加速领域，提出了一种基于‘差异容忍度’（ToD）的逐层剪枝方法，用于在保持性能的同时提升模型效率。虽然其在神经网络压缩方面有技术贡献，但并未针对文档图像理解（DIU）任务设计，也没有涉及VLM、多模态对齐、OCR去幻觉、细粒度定位或高分辨率处理等DIU核心痛点。尽管模型压缩技术理论上可用于部署轻量级DIU模型，但该工作本身缺乏与文档理解任务的结合，迁移价值有限，属于常规效率优化研究。
Abstract Neural network pruning has been widely adopted to reduce the parameter scale of complex neural networks, enabling efficient deployment on resource-limited edge devices. Mainstream pruning methods typically adopt uniform pruning strategies, which tend to cause a substantial performance degradation under high sparsity levels. Recent studies focus on non-uniform layer-wise pruning, but such approaches typically depend on global architecture optimization, which is computational expensive and lacks flexibility. To address these limitations, this paper proposes a novel method named Flexible Automatic Identification and Removal (FAIR)-Pruner, which adaptively determines the sparsity levels of each layer and identifies the units to be pruned. The core of FAIR-Pruner lies in the introduction of a novel indicator, Tolerance of Differences (ToD), designed to balance the importance scores obtained from two complementary perspectives: the architecture-level (Utilization Score) and the task-level (Reconstruction Score). By controlling ToD at preset levels, FAIR-Pruner determines layer-specific thresholds and removes units whose Utilization Scores fall below the corresponding thresholds. Furthermore, by decoupling threshold determination from importance estimation, FAIR-Pruner allows users to flexibly obtain pruned models under varying pruning ratios. Extensive experiments demonstrate that FAIR-Pruner achieves state-of-the-art performance, maintaining higher accuracy even at high compression ratios. Moreover, the ToD based layer-wise pruning ratios can be directly applied to existing powerful importance measurements, thereby improving the performance under uniform-pruning.
Authors Chenqing Lin, Mostafa Hussien, Chengyao Yu, Bingyi Jing, Mohamed Cheriet, Osama Abdelrahman, Ruixing Ming
Comment Submitted to CVPR 2026
Categories Artificial Intelligence
4
When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA
Title CN 当更强的教师模型无法培养出更优秀的学生：重新审视VQA中CLIP模型的知识蒸馏
Keywords 知识蒸馏 · 视觉语言模型 · CLIP · 模型压缩 · 视觉问答
Summary 本文首次系统性地研究了在CLIP风格的视觉语言模型之间进行知识蒸馏的效果，尤其是在视觉问答等下游多模态任务中的表现。实验表明，与自然语言处理等领域不同，更强的教师模型在当前蒸馏框架下未必能产生更优的学生模型，甚至可能导致性能下降。这一发现挑战了知识蒸馏领域的一些固有假设，并呼吁为多模态模型设计新的蒸馏范式。
Reason 该论文研究了在视觉问答（VQA）任务中，使用不同规模的CLIP风格教师模型进行知识蒸馏的效果，发现更强的教师模型并不总能蒸馏出性能更好的轻量级学生模型。虽然其关注的是VLM的效率问题，且涉及VQA这一多模态任务，但整体工作仍集中在通用VQA或图像-文本匹配任务上，并未针对文档图像理解中的核心痛点（如OCR幻觉、复杂版式建模、细粒度文本定位）进行设计或验证。尽管知识蒸馏作为一种模型压缩技术理论上可用于加速DIU模型部署，但本文未提出可迁移至文档领域的新型架构、表示学习或推理机制，属于常规的模型压缩研究。
Abstract Vision-language models (VLMs) have achieved remarkable success across multimodal tasks, yet their substantial computational demands hinder efficient deployment. Knowledge distillation (KD) has emerged as a powerful approach for building lightweight but competitive models, with strong evidence from both language and vision domains. However, its application to VLMs, particularly CLIP-style models, remains limited, often constrained to small-scale teachers and narrow evaluation tasks such as classification or retrieval. In this work, we present the first systematic study of distillation across a range of CLIP-style teacher models, ranging from standard baselines to large-scale state-of-the-art models. Contrary to trends observed in NLP and vision, we find that stronger teachers do not consistently yield better students; in fact, existing distillation frameworks often fail to scale, leading to degraded performance in downstream multimodal tasks such as visual question answering. Our findings challenge prevailing assumptions in KD and point toward new directions for designing parameter-efficient multimodal models.
Authors Pume Tuchinda, Parinthapat Pengpun, Romrawin Chumpu, Sarana Nutanong, Peerat Limkonchotiwat
Categories Computer Vision and Pattern Recognition, Computation and Language
4
Masked-and-Reordered Self-Supervision for Reinforcement Learning from Verifiable Rewards
Title CN 基于可验证奖励的强化学习中的掩码与重排序自监督
Keywords 自监督学习 · 强化学习 · 推理扩展 · 数学推理 · 过程监督
Summary 本文提出MR-RLVR方法，通过引入类似BERT的掩码填充和步骤重排序机制，在数学推理任务中构建中间过程的自监督信号，提升强化学习从仅结果可验证奖励（RLVR）中的学习效率。在Qwen和DeepSeek模型上验证显示，该方法在Pass@k指标上有显著提升。然而，该方法未涉及视觉输入或文档结构，属于纯NLP推理优化。
Reason 该论文提出在数学推理任务中通过‘掩码填充’和‘步骤重排序’构造中间过程的自监督信号，以增强仅结果可验证场景下的RLVR训练效果。虽然其关注中间推理链的学习，对长文档多步推理有一定启发，但整体工作仍聚焦于纯文本数学推理领域，未涉及视觉或文档模态，且无明确迁移路径到文档图像理解中的OCR幻觉、布局建模或细粒度定位等核心问题。
Abstract Test-time scaling has been shown to substantially improve large language models' (LLMs) mathematical reasoning. However, for a large portion of mathematical corpora, especially theorem proving, RLVR's scalability is limited: intermediate reasoning is crucial, while final answers are difficult to directly and reliably verify. Meanwhile, token-level SFT often degenerates into rote memorization rather than inducing longer chains of thought. Inspired by BERT's self-supervised tasks, we propose MR-RLVR (Masked-and-Reordered RLVR), which constructs process-level self-supervised rewards via "masked-then-fill" and "step reordering" to extract learnable signals from intermediate reasoning. Our training pipeline comprises two stages: we first perform self-supervised training on sampled mathematical calculation and proof data; we then conduct RLVR fine-tuning on mathematical calculation datasets where only outcomes are verifiable. We implement MR-RLVR on Qwen2.5-3B and DeepSeek-R1-Distill-Qwen-1.5B, and evaluate on AIME24, AIME25, AMC23, and MATH500. Under a fixed sampling and decoding budget, MR-RLVR achieves average relative gains over the original RLVR of +9.86% Pass@1, +5.27% Pass@5, and +4.00% Pass@8. These results indicate that incorporating process-aware self-supervised signals can effectively enhance RLVR's scalability and performance in only outcome-verifiable settings.
Authors Zhen Wang, Zhifeng Gao, Guolin Ke
Categories Artificial Intelligence, Computation and Language
4
MMT-ARD: Multimodal Multi-Teacher Adversarial Distillation for Robust Vision-Language Models
Title CN MMT-ARD：用于鲁棒视觉语言模型的多模态多教师对抗蒸馏
Keywords 视觉语言模型 · 对抗蒸馏 · 多教师学习 · 鲁棒性 · 知识蒸馏
Summary 本文提出MMT-ARD，一种多模态多教师对抗蒸馏框架，通过双教师结构融合干净特征保留与鲁棒特征增强，并引入基于置信度的动态权重分配和跨模态自适应加权函数，以提升学生模型的对抗鲁棒性与零样本性能。实验表明其在ViT-B-32上显著优于单教师蒸馏方法。
Reason 该论文聚焦于提升视觉语言模型在对抗攻击下的鲁棒性，属于VLM上游方法研究。其核心贡献是多教师蒸馏框架与动态权重分配机制，虽对模型鲁棒性和训练效率有提升，但并未针对文档图像理解中的典型痛点（如OCR幻觉、复杂版式建模、细粒度文本定位）进行设计或验证。尽管对抗鲁棒性在理论上可能间接缓解部分输入扰动问题，但该方法迁移到DIU场景的路径不明确，且未涉及高分辨率处理、长序列推理或多模态对齐等文档关键挑战，因此迁移价值有限。
Abstract Vision-Language Models (VLMs) are increasingly deployed in safety-critical applications, making their adversarial robustness a crucial concern. While adversarial knowledge distillation has shown promise in transferring robustness from teacher to student models, traditional single-teacher approaches suffer from limited knowledge diversity, slow convergence, and difficulty in balancing robustness and accuracy. To address these challenges, we propose MMT-ARD: a Multimodal Multi-Teacher Adversarial Robust Distillation framework. Our key innovation is a dual-teacher knowledge fusion architecture that collaboratively optimizes clean feature preservation and robust feature enhancement. To better handle challenging adversarial examples, we introduce a dynamic weight allocation strategy based on teacher confidence, enabling adaptive focus on harder samples. Moreover, to mitigate bias among teachers, we design an adaptive sigmoid-based weighting function that balances the strength of knowledge transfer across modalities. Extensive experiments on ImageNet and zero-shot benchmarks demonstrate that MMT-ARD improves robust accuracy by +4.32% and zero-shot accuracy by +3.5% on the ViT-B-32 model, while achieving a 2.3x increase in training efficiency over traditional single-teacher methods. These results highlight the effectiveness and scalability of MMT-ARD in enhancing the adversarial robustness of multimodal large models. Our codes are available at https://github.com/itsnotacie/MMT-ARD.
Authors Yuqi Li, Junhao Dong, Chuanguang Yang, Shiping Wen, Piotr Koniusz, Tingwen Huang, Yingli Tian, Yew-Soon Ong
Comment 10 pages
Categories Computer Vision and Pattern Recognition
4
Attention Via Convolutional Nearest Neighbors
Title CN 通过卷积最近邻实现注意力
Keywords 卷积神经网络 · 自注意力 · 视觉架构 · 邻居聚合 · 特征表示
Summary 本文提出了Convolutional Nearest Neighbors（ConvNN），将卷积和自注意力视为邻居选择与聚合的不同形式，构建了一个统一的特征聚合框架，并在VGG和ViT上验证了其有效性。结果显示，在结合空间邻近性和特征相似性时可带来性能提升。
Reason 该论文提出了一种将卷积与自注意力统一的框架ConvNN，属于通用视觉架构层面的探索。虽然其思想在理论上可能对多模态模型中的视觉主干设计有一定启发，但并未针对文档图像理解中的具体痛点（如OCR幻觉、密集文本识别、复杂版式建模等）进行设计或验证。其在CIFAR上的实验也远离真实文档场景。尽管‘统一卷积与注意力’具有理论价值，但迁移到DIU领域缺乏直接路径，且无证据表明其能提升VLM在文档上的细粒度定位或高分辨率处理能力。
Abstract The shift from Convolutional Neural Networks to Transformers has reshaped computer vision, yet these two architectural families are typically viewed as fundamentally distinct. We argue that convolution and self-attention, despite their apparent differences, can be unified within a single k-nearest neighbor aggregation framework. The critical insight is that both operations are special cases of neighbor selection and aggregation; convolution selects neighbors by spatial proximity, while attention selects by feature similarity, revealing they exist on a continuous spectrum. We introduce Convolutional Nearest Neighbors (ConvNN), a unified framework that formalizes this connection. Crucially, ConvNN serves as a drop-in replacement for convolutional and attention layers, enabling systematic exploration of the intermediate spectrum between these two extremes. We validate the framework's coherence on CIFAR-10 and CIFAR-100 classification tasks across two complementary architectures: (1) Hybrid branching in VGG improves accuracy on both CIFAR datasets by combining spatial-proximity and feature-similarity selection; and (2) ConvNN in ViT outperforms standard attention and other attention variants on both datasets. Extensive ablations on $k$ values and architectural variants reveal that interpolating along this spectrum provides regularization benefits by balancing local and global receptive fields. Our work provides a unifying framework that dissolves the apparent distinction between convolution and attention, with implications for designing more principled and interpretable vision architectures.
Authors Mingi Kang, Jeová Farias Sales Rocha Neto
Categories Computer Vision and Pattern Recognition
4
Loss-Oriented Ranking for Automated Visual Prompting in LVLMs
Title CN 面向LVLMs中自动化视觉提示的损失导向排序
Keywords 视觉提示 · LVLM增强 · 损失排序 · 自动化提示选择 · 多模态学习
Summary 本文提出AutoV，一种用于大型视觉语言模型（LVLM）的自动化视觉提示选择方法。通过构建一个基于预训练LVLM预测损失的排序机制，AutoV能够从多个候选视觉提示中自动学习选择最优者。作者设计了自动数据收集与标注流程，利用损失排名作为监督信号训练提示选择模块。实验表明该方法可显著提升LLaVA-OV和Qwen2.5-VL等模型在VizWiz和MMMU等多个图像理解任务上的表现。
Reason 该论文提出AutoV方法，通过损失导向的排序机制自动选择最优视觉提示，以提升LVLM在图像理解任务上的性能。其核心贡献在于自动化视觉提示选择框架和基于模型预测损失的监督信号构建。虽然方法在VizWiz、MMMU等包含一定文档成分的基准上验证了有效性，但其关注的是通用视觉提示工程，未针对文档特有的结构化信息、密集文本、OCR幻觉或复杂排版进行建模。所提方法属于通用VLM增强技术，迁移到DIU场景需额外设计适配机制（如与文本布局对齐），直接应用潜力有限。
Abstract Inspired by text prompts in large language models (LLMs), visual prompts have been explored to enhance the reasoning capabilities of large vision-language models (LVLMs). Current methods design heuristic visual prompts, such as overlaying a text-query-guided attention heatmap on the original input image. However, designing effective prompts manually is challenging and time-consuming, and it often fails to explore the benefits of different visual prompts, leading to sub-optimal performance. To this end, we propose \textbf{AutoV} that learns to automatically select the optimal visual prompt from various candidates based on given textual queries and the input image. To train AutoV, we develop an automatic data collection and labeling pipeline that evaluates various visual prompts with a pre-trained LVLM. We input a set of visual prompts into the LVLM and rank them according to the prediction losses generated by the model. Using the ranking as a supervision signal, we train AutoV to automatically choose the optimal visual prompt from various visual prompts for LVLMs. Experiments indicate that AutoV enhances the performance of various LVLMs across multiple image understanding tasks. For instance, LLaVA-OV with AutoV achieves $\textbf{10.2}\%$ accuracy gain on VizWiz, and AutoV boosts Qwen2.5-VL by $\textbf{3.8}\%$ on MMMU, highlighting its potential as an optimal visual prompting method.
Authors Yuan Zhang, Chun-Kai Fan, Tao Huang, Ming Lu, Sicheng Yu, Junwen Pan, Kuan Cheng, Qi She, Shanghang Zhang
Comment 17 pages
Categories Computer Vision and Pattern Recognition
4
Selective Rotary Position Embedding
Title CN 选择性旋转位置嵌入
Keywords 位置编码 · RoPE · 序列建模 · Transformer · 状态空间模型
Summary 本文提出了Selective RoPE，一种输入依赖的旋转位置嵌入方法，将传统的固定角度RoPE推广到可根据输入动态调整旋转角度，从而增强模型对序列顺序的建模能力。作者在语言建模和复杂序列任务（如复制、状态跟踪）上验证了其有效性。该方法揭示了softmax注意力中隐含的位置旋转结构，并统一了线性注意力与状态空间模型中的位置编码视角。
Reason 该论文提出了一种输入依赖的旋转位置编码机制Selective RoPE，旨在改进序列建模中的位置表示，适用于语言建模和长序列任务。虽然其方法在序列建模中表现出色，但并未涉及多模态或视觉输入，也未针对文档图像理解中的核心问题（如高分辨率处理、细粒度定位、OCR幻觉等）进行设计。尽管理论上位置编码的改进可能间接有助于长文档文本序列的建模，但缺乏与视觉模态对齐或文档结构建模的明确机制，迁移价值有限。
Abstract Position information is essential for language modeling. In softmax transformers, Rotary Position Embeddings (\textit{RoPE}) encode positions through \textit{fixed-angle} rotations, while in linear transformers, order is handled via input-dependent (selective) gating that decays past key-value associations. Selectivity has generally been shown to improve language-related tasks. Inspired by this, we introduce \textit{Selective RoPE}, an \textit{input-dependent} rotary embedding mechanism, that generalizes \textit{RoPE}, and enables rotation in \textit{arbitrary angles} for both linear and softmax transformers. We show that softmax attention already performs a hidden form of these rotations on query-key pairs, uncovering an implicit positional structure. We further show that in state-space models and gated linear transformers, the real part manages forgetting while the imaginary part encodes positions through rotations. We validate our method by equipping gated transformers with \textit{Selective RoPE}, demonstrating that its input-dependent rotations improve performance in language modeling and on difficult sequence tasks like copying, state tracking, and retrieval.
Authors Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, Volkan Cevher
Categories Computation and Language
4
DeepCoT: Deep Continual Transformers for Real-Time Inference on Data Streams
Title CN DeepCoT：用于数据流实时推理的深度持续型Transformer
Keywords Transformer · 流式推理 · 计算效率 · 持续学习 · 冗余消除
Summary 本文提出了DeepCoT，一种可在不损失性能的前提下显著降低Transformer在流式数据上计算冗余的模型，适用于音频、视频和文本序列任务。通过在各层实现无冗余的连续状态传递，实现了接近线性的计算增长，速度比现有高效模型快两个数量级。
Reason 该论文提出了一种适用于音频、视频和文本流的持续型Transformer架构，旨在减少滑动窗口带来的冗余计算，提升推理效率。虽然其‘深度可扩展’和‘线性计算成本’特性在理论上对长文档或多页文档的流式处理有一定启发意义，但该工作并未针对文档图像理解中的核心问题（如OCR幻觉、复杂排版、视觉-文本对齐）进行设计，也未涉及多模态输入或视觉token处理。其应用场景集中在单模态数据流，迁移到DIU领域需大幅重构，实用性有限。
Abstract Transformer-based models have dramatically increased their size and parameter count to tackle increasingly complex tasks. At the same time, there is a growing demand for low-latency inference on resource-constrained devices that achieves high performance. In particular, stream data inference is typically performed over a sliding temporal window, leading to highly redundant computations. The recent Continual Transformers have addressed this issue, but they can only be effectively used in shallow models, which limits their scope and generalization power. In this paper, we propose the Deep Continual Transformer (DeepCoT), a redundancy-free encoder-only model that can be applied over existing deep encoder architectures with minimal changes. In our experiments over audio, video, and text streams, we show that DeepCoTs retain comparative performance to their non-continual baselines while offering a linear computational cost for all Transformer layers, which reduces up to two orders of magnitude in the running time compared to previous efficient models.
Authors Ginés Carreto Picón, Peng Yuan Zhou, Qi Zhang, Alexandros Iosifidis
Comment 13 pages, 5 figures
Categories Computation and Language
3
Revisiting Generalization Across Difficulty Levels: It's Not So Easy
Title CN 重新审视跨难度水平的泛化能力：这并不容易
Keywords 大语言模型 · 泛化能力 · 难度分级 · 项目反应理论 · 数据构建
Summary 本文通过项目反应理论（IRT）和数千个LLM输出对六个数据集的样本难度进行客观评级，系统评估了大语言模型在不同难度级别间的泛化能力。结果表明，在简单或困难数据上训练无法一致地提升所有难度级别的表现，强调训练和评估中应覆盖广泛难度。研究揭示了当前数据构建中‘走捷径’的风险。
Reason 该论文研究大语言模型在不同任务难度下的泛化能力，属于纯NLP领域的基础分析工作，未涉及任何视觉或文档模态。尽管其关于训练数据难度分布的结论具有一定方法论启示，但无法直接迁移解决文档图像理解中的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）。论文完全基于文本任务和LLM行为分析，与多模态文档理解无实质关联。
Abstract We investigate how well large language models (LLMs) generalize across different task difficulties, a key question for effective data curation and evaluation. Existing research is mixed regarding whether training on easier or harder data leads to better results, and whether those gains come on easier or harder test data. We address this question by conducting a systematic evaluation of LLMs' generalization across models, datasets, and fine-grained groups of example difficulty. We rank examples in six datasets using the outputs of thousands of different LLMs and Item Response Theory (IRT), a well-established difficulty metric in educational testing. Unlike prior work, our difficulty ratings are therefore determined solely by the abilities of many different LLMs, excluding human opinions of difficulty. With a more objective, larger-scale, and finer-grained analysis, we show that cross-difficulty generalization is often limited; training on either easy or hard data cannot achieve consistent improvements across the full range of difficulties. These results show the importance of having a range of difficulties in both training and evaluation data for LLMs, and that taking shortcuts with respect to difficulty is risky.
Authors Yeganeh Kordi, Nihal V. Nayak, Max Zuo, Ilana Nguyen, Stephen H. Bach
Categories Artificial Intelligence, Computation and Language
3
ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration
Title CN ToolOrchestra：通过高效模型与工具协同提升智能
Keywords Agent · Tool Orchestration · Reinforcement Learning · Efficiency · Scalable Reasoning
Summary 本文提出ToolOrchestra，一种通过强化学习训练小型协调模型（8B）来高效调度多种工具和模型的方法，在复杂任务上实现了比GPT-5更高的准确率和更低的成本。重点在于工具使用的效率与用户偏好对齐，适用于通用Agent场景。
Reason 该论文属于典型的Agent系统设计工作，聚焦于通用复杂任务（如Humanity's Last Exam）中的工具调度与成本效率优化。虽然其提出的轻量级协调器架构和强化学习奖励机制在通用Agent领域具有价值，但并未针对文档图像理解（DIU）中的核心痛点（如OCR幻觉、密集文本识别、复杂版式建模、长文档推理）提出可迁移的底层方法。其应用场景为多工具调用的开放任务，而非文档模态特有的结构化理解或视觉-语言对齐问题，无法直接用于解决DIU的关键挑战。
Abstract Large language models are powerful generalists, yet solving deep and complex problems such as those of the Humanity's Last Exam (HLE) remains both conceptually challenging and computationally expensive. We show that small orchestrators managing other models and a variety of tools can both push the upper bound of intelligence and improve efficiency in solving difficult agentic tasks. We introduce ToolOrchestra, a method for training small orchestrators that coordinate intelligent tools. ToolOrchestra explicitly uses reinforcement learning with outcome-, efficiency-, and user-preference-aware rewards. Using ToolOrchestra, we produce Orchestrator, an 8B model that achieves higher accuracy at lower cost than previous tool-use agents while aligning with user preferences on which tools are to be used for a given query. On HLE, Orchestrator achieves a score of 37.1%, outperforming GPT-5 (35.1%) while being 2.5x more efficient. On tau2-Bench and FRAMES, Orchestrator surpasses GPT-5 by a wide margin while using only about 30% of the cost. Extensive analysis shows that Orchestrator achieves the best trade-off between performance and cost under multiple metrics, and generalizes robustly to unseen tools. These results demonstrate that composing diverse tools with a lightweight orchestration model is both more efficient and more effective than existing methods, paving the way for practical and scalable tool-augmented reasoning systems.
Authors Hongjin Su, Shizhe Diao, Ximing Lu, Mingjie Liu, Jiacheng Xu, Xin Dong, Yonggan Fu, Peter Belcak, Hanrong Ye, Hongxu Yin, Yi Dong, Evelina Bakhturina, Tao Yu, Yejin Choi, Jan Kautz, Pavlo Molchanov
Comment 21 pages, 6 figures
Categories Artificial Intelligence, Computation and Language
3
G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
Title CN G²VLM：基于几何定位的视觉语言模型，实现统一的3D重建与空间推理
Keywords 3D重建 · 空间推理 · 视觉语言模型 · 几何定位 · 多模态学习
Summary 本文提出G²VLM，一种结合3D视觉几何先验的视觉语言模型，通过统一框架实现从2D图像中进行3D空间重建并增强空间推理能力。模型利用多视角图像和视频数据训练，在无需密集标注的情况下学习空间结构，支持上下文学习和交错推理，在3D重建与空间理解任务上达到先进水平。
Reason 该论文聚焦于3D场景重建与通用空间理解，其核心是将2D图像提升至3D空间表示，并增强VLM在空间关系上的推理能力。然而，文档图像理解中的‘空间’结构本质上是2D布局语义（如文本流、对齐、层级），而非真实世界深度与视角变换。尽管其‘几何定位’概念看似相关，但方法依赖多视角图像和视频数据训练，且目标为3D属性预测，与文档中字符级或区域级细粒度定位需求不匹配。无法有效迁移解决OCR幻觉、复杂排版解析等DIU核心问题，属于无关领域扩展。
Abstract Vision-Language Models (VLMs) still lack robustness in spatial intelligence, demonstrating poor performance on spatial understanding and reasoning tasks. We attribute this gap to the absence of a visual geometry learning process capable of reconstructing 3D space from 2D images. We present G$^2$VLM, a geometry grounded vision-language model that bridges two fundamental aspects of spatial intelligence: spatial 3D reconstruction and spatial understanding. G$^2$VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning. Our unified design is highly scalable for spatial understanding: it trains on abundant multi-view image and video data, while simultaneously leveraging the benefits of 3D visual priors that are typically only derived from hard-to-collect annotations. Experimental results demonstrate G$^2$VLM is proficient in both tasks, achieving comparable results to state-of-the-art feed-forward 3D reconstruction models and achieving better or competitive results across spatial understanding and reasoning tasks. By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G$^2$VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing.
Authors Wenbo Hu, Jingli Lin, Yilin Long, Yunlong Ran, Lihan Jiang, Yifan Wang, Chenming Zhu, Runsen Xu, Tai Wang, Jiangmiao Pang
Comment code are released at https://github.com/InternRobotics/G2VLM
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
3
Continual Error Correction on Low-Resource Devices ACM MMSys 2025
Title CN 低资源设备上的持续错误纠正
Keywords 错误纠正 · 知识蒸馏 · 原型学习 · 低资源学习 · 边缘设备
Summary 本文提出一种用于低资源设备的持续错误纠正系统，通过服务器端知识蒸馏与设备端原型更新相结合的方式，实现无需重新训练模型的高效纠错。在Food-101和Flowers-102等图像分类数据集上验证了其在一-shot场景下超过50%的纠错能力，且遗忘率极低。系统已通过Android应用验证实用性。
Reason 该论文聚焦于通用图像分类和目标检测任务中的用户驱动错误纠正，属于在特定应用场景下的模型维护技术。其核心方法是通过原型更新实现设备端的高效纠错，但并未针对文档图像理解中的关键挑战（如OCR幻觉、复杂版面解析、细粒度文本定位或长文档推理）进行设计或验证。虽然‘原型更新’机制理论上可用于减少VLM在DocVQA中的幻觉，但本文未涉及语言-视觉对齐、文本结构建模或高分辨率文档处理等DIU核心问题，属于通用AI维护技术在边缘设备的应用，符合‘平行下游应用’特征。
Abstract The proliferation of AI models in everyday devices has highlighted a critical challenge: prediction errors that degrade user experience. While existing solutions focus on error detection, they rarely provide efficient correction mechanisms, especially for resource-constrained devices. We present a novel system enabling users to correct AI misclassifications through few-shot learning, requiring minimal computational resources and storage. Our approach combines server-side foundation model training with on-device prototype-based classification, enabling efficient error correction through prototype updates rather than model retraining. The system consists of two key components: (1) a server-side pipeline that leverages knowledge distillation to transfer robust feature representations from foundation models to device-compatible architectures, and (2) a device-side mechanism that enables ultra-efficient error correction through prototype adaptation. We demonstrate our system's effectiveness on both image classification and object detection tasks, achieving over 50% error correction in one-shot scenarios on Food-101 and Flowers-102 datasets while maintaining minimal forgetting (less than 0.02%) and negligible computational overhead. Our implementation, validated through an Android demonstration app, proves the system's practicality in real-world scenarios.
Authors Kirill Paramonov, Mete Ozay, Aristeidis Mystakidis, Nikolaos Tsalikidis, Dimitrios Sotos, Anastasios Drosou, Dimitrios Tzovaras, Hyunjun Kim, Kiseok Chang, Sangdok Mo, Namwoong Kim, Woojong Yoo, Jijoong Moon, Umberto Michieli
Comment ACM MMSys 2025
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Mechanisms of Non-Monotonic Scaling in Vision Transformers
Title CN Vision Transformer中非单调缩放的机制
Keywords Vision Transformer · 缩放规律 · 表示学习 · 信息混合 · 深度设计
Summary 本文通过分析ViT-S/B/L在ImageNet上的表现，发现随着深度增加会出现Cliff-Plateau-Climb三阶段模式，并指出深层ViT中[CLS] token作用减弱，信息扩散早于任务性能提升。提出Information Scrambling Index来量化信息混合程度，建议应更注重深度的精细校准而非单纯扩大参数量。
Reason 该论文研究的是标准Vision Transformer在ImageNet上的深度增加导致性能下降的现象，聚焦于[CLS] token退化、信息混合模式等基础表示学习问题。虽然其提出的Information Scrambling Index可能对模型诊断有一定启发，但研究对象为通用图像分类任务下的ViT架构，并未涉及文档结构、多模态对齐、OCR错误缓解或细粒度文本定位等DIU核心挑战。既不属于DIU本题工作，也缺乏可直接迁移至文档理解场景的方法论创新，属于通用视觉表示分析，与文档图像理解关联度低。
Abstract Deeper Vision Transformers often perform worse than shallower ones, which challenges common scaling assumptions. Through a systematic empirical analysis of ViT-S, ViT-B, and ViT-L on ImageNet, we identify a consistent three-phase Cliff-Plateau-Climb pattern that governs how representations evolve with depth. We observe that better performance is associated with progressive marginalization of the [CLS] token, originally designed as a global aggregation hub, in favor of distributed consensus among patch tokens. We quantify patterns of information mixing with an Information Scrambling Index, and show that in ViT-L the information-task tradeoff emerges roughly 10 layers later than in ViT-B, and that these additional layers correlate with increased information diffusion rather than improved task performance. Taken together, these results suggest that transformer architectures in this regime may benefit more from carefully calibrated depth that executes clean phase transitions than from simply increasing parameter count. The Information Scrambling Index provides a useful diagnostic for existing models and suggests a potential design target for future architectures. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/Cliff-Plateau-Climb.
Authors Anantha Padmanaban Krishna Kumar
Comment 16 pages total (11 pages main text, 1 pages references, 4 pages appendix), 5 figures, 11 tables. Code available at https://github.com/AnanthaPadmanaban-KrishnaKumar/Cliff-Plateau-Climb
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
The Impossibility of Inverse Permutation Learning in Transformer Models
Title CN Transformer模型中逆排列学习的不可能性
Keywords Transformer理论 · 逆排列学习 · 推理能力 · 因果注意力 · 提示机制
Summary 本文从理论上证明了decoder-only Transformer无法完成逆排列学习任务，并指出encoder-decoder架构或引入‘scratch tokens’可能弥补这一缺陷。该工作揭示了当前大模型在某些推理任务上的根本性限制，但完全局限于纯文本序列建模，未涉及图像或文档布局信息。
Reason 该论文研究的是decoder-only Transformer在逆排列学习任务上的表达能力局限，属于理论性质的分析，主要关注语言模型内部的推理机制和架构缺陷。虽然其讨论的‘长上下文检索’和‘上下文学习’与文档理解有一定关联，但并未涉及视觉模态或文档结构信息处理，无法直接迁移解决DIU中的OCR幻觉、复杂排版或细粒度定位等核心问题。其提出的‘scratch tokens’机制虽对推理过程有启发意义，但仍属于纯NLP理论范畴，缺乏多模态和视觉空间建模的考量。
Abstract In this technical note, we study the problem of inverse permutation learning in decoder-only transformers. Given a permutation and a string to which that permutation has been applied, the model is tasked with producing the original (``canonical'') string. We argue that this task models a natural robustness property across a variety of reasoning tasks, including long-context retrieval, multiple choice QA and in-context learning. Our primary contribution is an impossibility result: we show that an arbitrary depth, decoder-only transformer cannot learn this task. This result concerns the expressive capacity of decoder-only transformer models and is agnostic to training dynamics or sample complexity. We give a pair of alternative constructions under which inverse permutation learning is feasible. The first of these highlights the fundamental role of the causal attention mask, and reveals a gap between the expressivity of encoder-decoder transformers and the more popular decoder-only architecture. The latter result is more surprising: we show that simply padding the input with ``scratch tokens" yields a construction under which inverse permutation learning is possible. We conjecture that this may suggest an alternative mechanism by which chain-of-thought prompting or, more generally, intermediate ``thinking'' tokens can enable reasoning in large language models, even when these tokens encode no meaningful semantic information (e.g., the results of intermediate computations).
Authors Rohan Alur, Chris Hays, Manish Raghavan, Devavrat Shah
Categories Artificial Intelligence
3
Scale-Agnostic Kolmogorov-Arnold Geometry in Neural Networks
Title CN 神经网络中的尺度无关科尔莫戈罗夫-阿诺德几何
Keywords 神经网络理论 · Kolmogorov-Arnold几何 · 尺度不变性 · MLP · MNIST
Summary 本文研究了浅层多层感知机在训练过程中是否会在高维现实数据（如MNIST）上形成科尔莫戈罗夫-阿诺德几何结构，并发现这种结构在多个空间尺度上一致出现，具有尺度无关性。尽管揭示了神经网络学习过程中的内在几何组织现象，但其成果偏向基础理论，缺乏对文档图像理解任务的实际建模价值。
Reason 该论文研究的是浅层MLP在MNIST等图像数据上自发形成的几何结构，属于基础神经网络理论分析，虽然涉及视觉输入，但其方法和结论与文档图像理解（DIU）的核心任务如DocVQA、布局分析、OCR-free理解等无直接关联。未提出可用于解决OCR幻觉、复杂排版或长文档推理的可迁移技术。属于理论性AI研究，而非面向文档模态的建模创新。
Abstract Recent work by Freedman and Mulligan demonstrated that shallow multilayer perceptrons spontaneously develop Kolmogorov-Arnold geometric (KAG) structure during training on synthetic three-dimensional tasks. However, it remained unclear whether this phenomenon persists in realistic high-dimensional settings and what spatial properties this geometry exhibits. We extend KAG analysis to MNIST digit classification (784 dimensions) using 2-layer MLPs with systematic spatial analysis at multiple scales. We find that KAG emerges during training and appears consistently across spatial scales, from local 7-pixel neighborhoods to the full 28x28 image. This scale-agnostic property holds across different training procedures: both standard training and training with spatial augmentation produce the same qualitative pattern. These findings reveal that neural networks spontaneously develop organized, scale-invariant geometric structure during learning on realistic high-dimensional data.
Authors Mathew Vanherreweghe, Michael H. Freedman, Keith M. Adams
Categories Artificial Intelligence
3
On the Origin of Algorithmic Progress in AI
Title CN 论人工智能中算法进步的起源
Keywords 算法效率 · 缩放定律 · Transformer · LSTM · 计算规模
Summary 论文通过缩放实验分析2012–2023年间AI算法效率提升的来源，发现传统归因于算法创新的22,000倍FLOP效率增益中，仅有约6,930倍可被解释，其中大部分来自LSTM到Transformer的架构转变所带来的规模依赖性效率提升。结果表明，小模型下的算法进步远慢于预期，且算法效率评估具有强参考依赖性。
Reason 该论文研究的是AI领域整体算法效率随计算规模演进的变化，重点在于LSTM与Transformer在缩放规律上的差异，并非针对文档图像理解（DIU）或视觉-语言模型（VLM）中的具体技术问题。虽然其关于‘算法效率依赖于计算规模’的结论具有宏观启示，但未提出可直接迁移至DIU任务（如DocVQA、OCR-free识别、复杂版面分析）的方法论创新，属于通用AI发展分析，无具体技术组件可用于解决OCR幻觉、细粒度定位或高分辨率文档处理等核心痛点。
Abstract Algorithms have been estimated to increase AI training FLOP efficiency by a factor of 22,000 between 2012 and 2023 [Ho et al., 2024]. Running small-scale ablation experiments on key innovations from this time period, we are able to account for less than 10x of these gains. Surveying the broader literature, we estimate that additional innovations not included in our ablations account for less than 10x, yielding a total under 100x. This leads us to conduct scaling experiments, which reveal that much of this efficiency gap can be explained by algorithms with scale-dependent efficiency improvements. In particular, we conduct scaling experiments between LSTMs and Transformers, finding exponent differences in their compute-optimal scaling law while finding little scaling difference for many other innovations. These experiments demonstrate that - contrary to standard assumptions - an algorithm's efficiency gains are tied to compute scale. Using experimental extrapolation and literature estimates, we account for 6,930x efficiency gains over the same time period, with the scale-dependent LSTM-to-Transformer transition accounting for the majority of gains. Our results indicate that algorithmic progress for small models has been far slower than previously assumed, and that measures of algorithmic efficiency are strongly reference-dependent.
Authors Hans Gundlach, Alex Fogelson, Jayson Lynch, Ana Trisovic, Jonathan Rosenfeld, Anmol Sandhu, Neil Thompson
Categories Artificial Intelligence
3
Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining
Title CN 超越URL：用于高效LLM预训练的元数据多样性与位置
Keywords 元数据 · 预训练加速 · 大语言模型 · 训练效率 · latent structure
Summary 本文探讨了在大语言模型预训练中使用多种元数据（如文档质量信号）来加速训练过程的方法，发现比URL更细粒度的元数据更有效，并提出通过附加元数据预测作为辅助任务来提升效率。同时引入可学习的meta-token来建模质量感知的潜在结构。
Reason 该论文研究在LLM预训练中引入不同类型的元数据（如文档质量细粒度指标）以提升训练效率，核心关注点是语言模型的预训练加速。尽管涉及文档级信号，但其方法完全基于文本模态，未涉及视觉输入、多模态对齐或文档图像理解中的关键挑战（如OCR幻觉、布局分析、高分辨率处理等）。属于通用NLP训练优化工作，无法直接迁移解决DIU的核心问题。
Abstract Incorporating metadata in Large Language Models (LLMs) pretraining has recently emerged as a promising approach to accelerate training. However prior work highlighted only one useful signal-URLs, leaving open the question of whether other forms of metadata could yield greater benefits. In this study, we investigate a wider range of metadata types and find other types of metadata, such as fine-grained indicators of document quality that can also accelerate pretraining when prepended. We identify a common feature among effective metadata: they encode information at a finer granularity. We further introduce metadata appending as a means of improving training efficiency, where predicting an appropriate metadata as auxiliary task can help speed up pretraining. In addition, learnable meta-tokens trained with masked loss can recover part of the speedup by inducing quality-aware latent structure. Using probing, we analyze latent representations to understand how metadata shapes learning. Together, these results yield practical guidelines for integrating metadata to improve both the efficiency and effectiveness of LLM pretraining.
Authors Dongyang Fan, Diba Hashemi, Sai Praneeth Karimireddy, Martin Jaggi
Categories Artificial Intelligence, Computation and Language
3
BAMAS: Structuring Budget-Aware Multi-Agent Systems AAAI 2026
Title CN BAMAS：构建预算感知的多智能体系统
Keywords 多智能体系统 · 大语言模型 · 成本优化 · 强化学习 · 整数线性规划
Summary 本文提出了BAMAS，一种在预算约束下构建多智能体系统的新方法。它通过整数线性规划选择性能与成本平衡的LLM组合，并利用强化学习确定最优的智能体协作结构，在保持性能的同时最多可降低86%的成本。研究重点是系统级成本控制，而非任务层面的语义理解或视觉-语言建模创新。
Reason 该论文研究的是多智能体系统中的成本优化问题，核心贡献在于通过整数线性规划和强化学习选择性价比高的LLM并构建协作拓扑。虽然涉及多智能体协同机制设计，但其目标与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析或长文档推理无直接关联。方法聚焦于计算资源与模型调用的成本权衡，并未提出可用于提升VLM在文档任务中准确性或鲁棒性的底层技术。属于典型的‘平行应用’类智能体架构工作，不具备向DIU迁移的方法论价值。
Abstract Large language model (LLM)-based multi-agent systems have emerged as a powerful paradigm for enabling autonomous agents to solve complex tasks. As these systems scale in complexity, cost becomes an important consideration for practical deployment. However, existing work rarely addresses how to structure multi-agent systems under explicit budget constraints. In this paper, we propose BAMAS, a novel approach for building multi-agent systems with budget awareness. BAMAS first selects an optimal set of LLMs by formulating and solving an Integer Linear Programming problem that balances performance and cost. It then determines how these LLMs should collaborate by leveraging a reinforcement learning-based method to select the interaction topology. Finally, the system is instantiated and executed based on the selected agents and their collaboration topology. We evaluate BAMAS on three representative tasks and compare it with state-of-the-art agent construction methods. Results show that BAMAS achieves comparable performance while reducing cost by up to 86%.
Authors Liming Yang, Junyu Luo, Xuanzhe Liu, Yiling Lou, Zhenpeng Chen
Comment Accepted by AAAI 2026 (oral paper)
Categories Artificial Intelligence
3
BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotations
Title CN BoundingDocs：一个具有空间标注的文档问答统一数据集
Keywords 文档问答 · 数据集 · 信息提取 · OCR · 空间标注
Summary 该研究提出了BoundingDocs，一个整合多个文档AI数据集并统一为问答格式的数据集，包含OCR结果和答案的边界框位置。旨在用于训练和评估大语言模型在文档理解中的表现，并探索不同提示方式（含空间信息）对开源模型性能的影响。
Reason 该论文构建了一个将多个公开数据集整合并转换为问答形式的文档QA数据集，并添加了答案的边界框标注。虽然引入空间信息有一定价值，但其核心工作属于对现有资源的重新组织与格式转换，未提出新的模型架构、训练方法或推理机制。本质上是面向特定任务（如DocVQA）的下游数据集封装，缺乏底层方法论创新，属于‘平行应用’范畴，无法为DIU领域提供可迁移的技术范式。
Abstract We present a unified dataset for document Question-Answering (QA), which is obtained combining several public datasets related to Document AI and visually rich document understanding (VRDU). Our main contribution is twofold: on the one hand we reformulate existing Document AI tasks, such as Information Extraction (IE), into a Question-Answering task, making it a suitable resource for training and evaluating Large Language Models; on the other hand, we release the OCR of all the documents and include the exact position of the answer to be found in the document image as a bounding box. Using this dataset, we explore the impact of different prompting techniques (that might include bounding box information) on the performance of open-weight models, identifying the most effective approaches for document comprehension.
Authors Simone Giovannini, Fabio Coppini, Andrea Gemelli, Simone Marinai
Categories Artificial Intelligence, Computation and Language
3
Hierarchical Ranking Neural Network for Long Document Readability Assessment
Title CN 用于长文档可读性评估的分层排序神经网络
Keywords 可读性评估 · 长文档 · 分层排序 · 深度学习 · 自然语言处理
Summary 本文提出一种分层排序神经网络，通过双向机制捕捉句子级语义并利用成对排序算法建模可读性标签的序数关系，实现长文档的可读性评估。在中英文数据集上验证了模型有效性。
Reason 该论文研究的是文本可读性评估任务，属于NLP中的文本分析范畴，并未涉及文档图像、视觉布局、OCR或图文多模态理解等DIU核心问题。尽管处理了‘长文档’，但其输入是纯文本，不包含对图像、排版结构或视觉信息的建模，无法解决文档图像理解中的OCR幻觉、复杂版式或细粒度定位等关键痛点。属于平行应用类工作，且无视觉或多模态技术创新。
Abstract Readability assessment aims to evaluate the reading difficulty of a text. In recent years, while deep learning technology has been gradually applied to readability assessment, most approaches fail to consider either the length of the text or the ordinal relationship of readability labels. This paper proposes a bidirectional readability assessment mechanism that captures contextual information to identify regions with rich semantic information in the text, thereby predicting the readability level of individual sentences. These sentence-level labels are then used to assist in predicting the overall readability level of the document. Additionally, a pairwise sorting algorithm is introduced to model the ordinal relationship between readability levels through label subtraction. Experimental results on Chinese and English datasets demonstrate that the proposed model achieves competitive performance and outperforms other baseline models.
Authors Yurui Zheng, Yijun Chen, Shaohong Zhang
Categories Artificial Intelligence, Computation and Language
3
DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
Title CN DR Tulu：基于演化评分标准的深度研究强化学习
Keywords 强化学习 · 长文本生成 · 深度研究 · 评分标准演化 · 开放研究
Summary 本文提出了一种名为RLER（基于演化评分标准的强化学习）的方法，用于训练能够进行多步骤深度研究的开放模型DR Tulu。该模型通过与不断演化的评分标准协同训练，实现对长篇、有引用支持的答案生成，并在多个领域超越现有开源系统。然而，该工作聚焦于纯文本问答与知识检索流程优化，不涉及图像或文档布局信息处理。
Reason 该论文属于典型的‘平行应用’类型，尽管其方法（RLER）在长文本生成和多步推理方面有一定创新，但其核心任务是通用领域的‘深度研究’（deep research），并非针对文档图像理解（DIU）中的具体问题如OCR幻觉、复杂排版解析或视觉-语言对齐。虽然其Agent架构或强化学习框架可能被间接借鉴，但论文本身未涉及视觉输入、文档结构建模或多模态交互，无法直接迁移用于解决DIU的核心痛点，属于将现有LLM技术应用于下游研究流程的案例。
Abstract Deep research models perform multi-step research to produce long-form, well-attributed answers. However, most open deep research models are trained on easily verifiable short-form QA tasks via reinforcement learning with verifiable rewards (RLVR), which does not extend to realistic long-form tasks. We address this with Reinforcement Learning with Evolving Rubrics (RLER), in which we construct and maintain rubrics that co-evolve with the policy model during training; this allows the rubrics to incorporate information that the model has newly explored and to provide discriminative, on-policy feedback. Using RLER, we develop Deep Research Tulu (DR Tulu-8B), the first open model that is directly trained for open-ended, long-form deep research. Across four long-form deep research benchmarks in science, healthcare and general domains, DR Tulu substantially outperforms existing open deep research models, and matches or exceeds proprietary deep research systems, while being significantly smaller and cheaper per query. To facilitate future research, we release all data, models, and code, including our new MCP-based agent infrastructure for deep research systems.
Authors Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen-tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi, Pang Wei Koh
Categories Artificial Intelligence, Computation and Language
3
SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation NeurIPS 2025
Title CN SaFiRe：基于Mamba的扫视-注视迭代方法用于指代表达式分割
Keywords 指代表达式分割 · Mamba · 认知建模 · 多模态定位 · aRefCOCO
Summary 本文提出SaFiRe模型，模拟人类视觉认知过程，通过Mamba架构实现多轮全局理解与细节精炼，用于解决指代表达式分割中因语言歧义导致的定位困难。作者还构建了新数据集aRefCOCO以评估复杂表达下的性能。实验表明该方法在标准和新数据集上均优于现有基线。
Reason 该论文研究的是指代表达式分割（RIS），属于通用视觉-语言定位任务，其核心挑战是处理自然语言描述中的指代歧义，并针对的是开放图像中的目标分割问题。尽管其提出的‘扫视-注视’认知模拟机制和Mamba架构设计具有一定新颖性，但任务本身与文档图像理解（DIU）的核心痛点（如OCR幻觉、密集文本解析、复杂版式建模、长文档推理）关联较弱。文档图像中语言表达通常是结构化或半结构化的，而非自由指代表达；且RIS关注像素级定位而非语义理解或信息抽取。该方法难以迁移至DocVQA或表格识别等典型DIU任务，属于通用视觉语言领域应用，不构成对DIU的方法论推动。
Abstract Referring Image Segmentation (RIS) aims to segment the target object in an image given a natural language expression. While recent methods leverage pre-trained vision backbones and more training corpus to achieve impressive results, they predominantly focus on simple expressions--short, clear noun phrases like "red car" or "left girl". This simplification often reduces RIS to a key word/concept matching problem, limiting the model's ability to handle referential ambiguity in expressions. In this work, we identify two challenging real-world scenarios: object-distracting expressions, which involve multiple entities with contextual cues, and category-implicit expressions, where the object class is not explicitly stated. To address the challenges, we propose a novel framework, SaFiRe, which mimics the human two-phase cognitive process--first forming a global understanding, then refining it through detail-oriented inspection. This is naturally supported by Mamba's scan-then-update property, which aligns with our phased design and enables efficient multi-cycle refinement with linear complexity. We further introduce aRefCOCO, a new benchmark designed to evaluate RIS models under ambiguous referring expressions. Extensive experiments on both standard and proposed datasets demonstrate the superiority of SaFiRe over state-of-the-art baselines.
Authors Zhenjie Mao, Yuhuan Yang, Chaofan Ma, Dongsheng Jiang, Jiangchao Yao, Ya Zhang, Yanfeng Wang
Comment NeurIPS 2025; Project page: https://zhenjiemao.github.io/SaFiRe/
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Automated Dynamic AI Inference Scaling on HPC-Infrastructure: Integrating Kubernetes, Slurm and vLLM
Title CN 在HPC基础设施上实现AI推理的自动化动态扩展：集成Kubernetes、Slurm和vLLM
Keywords AI推理扩展 · HPC · Kubernetes · Slurm · vLLM
Summary 本文提出了一种在超级计算机RAMSES上集成vLLM、Slurm和Kubernetes的架构，用于支持大规模并发的AI推理请求。实验表明该系统能高效扩展至千级并发，端到端延迟仅增加约500ms。重点在于底层计算资源的调度与服务部署效率，而非模型层面的改进。
Reason 该论文聚焦于AI推理服务的工程部署架构，解决的是高性能计算环境中LLM服务的并发扩展与资源调度问题。尽管其系统优化技术对大规模模型部署有通用价值，但并未涉及任何文档图像理解（DIU）的核心任务（如DocVQA、布局分析、表格识别等），也没有提出可用于缓解OCR幻觉、复杂排版理解或长文档推理的方法论创新。属于典型的系统工程类工作，无法直接迁移用于提升DIU模型的语义理解或多模态对齐能力。
Abstract Due to rising demands for Artificial Inteligence (AI) inference, especially in higher education, novel solutions utilising existing infrastructure are emerging. The utilisation of High-Performance Computing (HPC) has become a prevalent approach for the implementation of such solutions. However, the classical operating model of HPC does not adapt well to the requirements of synchronous, user-facing dynamic AI application workloads. In this paper, we propose our solution that serves LLMs by integrating vLLM, Slurm and Kubernetes on the supercomputer \textit{RAMSES}. The initial benchmark indicates that the proposed architecture scales efficiently for 100, 500 and 1000 concurrent requests, incurring only an overhead of approximately 500 ms in terms of end-to-end latency.
Authors Tim Trappen, Robert Keßler, Roland Pabel, Viktor Achter, Stefan Wesner
Comment 6 pages, 3 figures
Categories Artificial Intelligence
3
Training Introspective Behavior: Fine-Tuning Induces Reliable Internal State Detection in a 7B Model
Title CN 训练内省行为：微调在7B模型中诱导出可靠的内部状态检测
Keywords 内省 · 语言模型微调 · 内部状态检测 · 可解释性 · 激活注入
Summary 该研究通过微调使一个7B参数的语言模型能够可靠地检测并报告被注入的单token‘思想’，实现了高达85%的准确率且无虚假阳性，满足了内省行为的部分标准。结果显示该能力具有泛化性，但并未证明其具备真正的元认知表征。这项工作聚焦于提升模型的自我感知能力，以增强AI透明性。
Reason 该论文研究的是语言模型对注入的‘思维’模式的自我报告能力，属于纯NLP领域的元认知机制探索，尽管涉及模型内部状态的可解释性，但其任务设定与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版、细粒度定位或高分辨率视觉处理无直接关联。虽然‘内部状态检测’理论上可能用于推理过程监控，但该工作未涉及多模态输入、视觉表示或文档结构建模，迁移到DIU场景的路径极不明确，属于无关领域的基础NLP研究。
Abstract Lindsey (2025) investigates introspective awareness in language models through four experiments, finding that models can sometimes detect and identify injected activation patterns -- but unreliably (~20% success in the best model). We focus on the first of these experiments -- self-report of injected "thoughts" -- and ask whether this capability can be directly trained rather than waiting for emergence. Through fine-tuning on transient single-token injections, we transform a 7B parameter model from near-complete failure (0.4% accuracy, 6.7% false positive rate) to reliable detection (85% accuracy on held-out concepts at α=40, 0% false positives). Our model detects fleeting "thoughts" injected at a single token position, retains that information, and reports the semantic content across subsequent generation steps. On this task, our trained model satisfies three of Lindsey's criteria: accuracy (correct identification), grounding (0/60 false positives), and internality (detection precedes verbalization). Generalization to unseen concept vectors (7.5pp gap) demonstrates the model learns a transferable skill rather than memorizing specific vectors, though this does not establish metacognitive representation in Lindsey's sense. These results address an open question raised by Lindsey: whether "training for introspection would help eliminate cross-model differences." We show that at least one component of introspective behavior can be directly induced, offering a pathway to built-in AI transparency.
Authors Joshua Fonseca Rivera
Comment 16 pages, 8 figures
Categories Artificial Intelligence, Computation and Language
3
Prune4Web: DOM Tree Pruning Programming for Web Agent AAAI 2026
Title CN Prune4Web：面向Web智能体的DOM树剪枝编程
Keywords Web Agent · DOM剪枝 · 程序化过滤 · LLM效率 · 任务分解
Summary 本文提出Prune4Web，通过让LLM生成Python脚本来对网页DOM树进行程序化剪枝，从而解决因DOM过大导致的Web Agent效率与精度问题。该方法避免了直接将庞大DOM输入LLM，转而用轻量级程序动态筛选关键元素，显著提升了动作定位准确率。
Reason 该论文属于典型的‘平行应用’，其核心是将LLM生成程序用于剪枝网页DOM结构以提升Web Agent效率。尽管其‘程序化过滤’和‘减少冗余信息输入’的思想具有一定通用性，但整体工作仍局限于Web自动化场景，未涉及文档图像理解中的OCR幻觉、复杂版式解析或视觉-文本对齐等核心问题。虽然DOM剪枝思想在长文档结构处理中或有类比空间，但其方法依赖于HTML语义结构，难以迁移到无结构或扫描文档的图像模态中，对DIU领域缺乏直接可迁移的技术创新。
Abstract Web automation employs intelligent agents to execute high-level tasks by mimicking human interactions with web interfaces. Despite the capabilities of recent Large Language Model (LLM)-based web agents, navigating complex, real-world webpages efficiently remains a significant hurdle due to the prohibitively large size of Document Object Model (DOM) structures, often ranging from 10,000 to 100,000 tokens. Existing strategies typically rely on crude DOM truncation -- risking the loss of critical information -- or employ inefficient heuristics and separate ranking models, failing to achieve an optimal balance between precision and scalability. To address these challenges, we introduce Prune4Web, a novel paradigm that shifts DOM processing from resource-intensive LLM reading to efficient programmatic pruning. Central to our approach is DOM Tree Pruning Programming, where an LLM generates executable Python scoring scripts to dynamically filter DOM elements based on semantic cues from decomposed sub-tasks. This mechanism eliminates the need for LLMs to ingest raw, massive DOMs, instead delegating traversal and scoring to lightweight, interpretable programs. This methodology achieves a 25x to 50x reduction in candidate elements for grounding, thereby facilitating precise action localization while mitigating attention dilution. Furthermore, we propose a specialized data annotation pipeline and a two-turn dialogue training strategy that jointly optimizes the Planner, Programmatic Filter, and Grounder within a unified framework. Extensive experiments demonstrate state-of-the-art performance. Notably, on our low-level grounding task, Prune4Web dramatically improves accuracy from 46.8% to 88.28%, underscoring its efficacy in real-world web automation.
Authors Jiayuan Zhang, Kaiquan Chen, Zhihao Lu, Enshen Zhou, Qian Yu, Jing Zhang
Comment Paper accepted to AAAI 2026
Categories Artificial Intelligence, Computation and Language
3
The Directed Prediction Change - Efficient and Trustworthy Fidelity Assessment for Local Feature Attribution Methods AAAI SECURE-AI4H workshop
Title CN 定向预测变化——用于局部特征归因方法的高效且可信的保真度评估
Keywords 可解释AI · 特征归因 · 保真度评估 · DPC · Guided Perturbation
Summary 本文提出了一种名为定向预测变化（DPC）的新指标，用于高效、确定性地评估局部特征归因方法的保真度。相比依赖随机采样的Infidelity指标，DPC通过引入扰动与归因的方向信息，在引导扰动实验中实现了近十倍加速，并消除了随机性。实验覆盖多种模型与解释算法，结果表明DPC能提供可重复、可信的评估结果。
Reason 该论文提出了一种用于评估解释方法保真度的新指标DPC，主要面向可解释AI中的归因方法验证，应用场景包括医学图像和金融表格数据。尽管涉及‘表格数据’，但其核心贡献是解释性评估工具，并未针对文档图像理解（DIU）中的关键任务如DocVQA、布局分析或OCR-free理解进行方法创新。属于通用可解释性研究，对DIU无直接迁移价值，且不解决OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract The utility of an explanation method critically depends on its fidelity to the underlying machine learning model. Especially in high-stakes medical settings, clinicians and regulators require explanations that faithfully reflect the model's decision process. Existing fidelity metrics such as Infidelity rely on Monte Carlo approximation, which demands numerous model evaluations and introduces uncertainty due to random sampling. This work proposes a novel metric for evaluating the fidelity of local feature attribution methods by modifying the existing Prediction Change (PC) metric within the Guided Perturbation Experiment. By incorporating the direction of both perturbation and attribution, the proposed Directed Prediction Change (DPC) metric achieves an almost tenfold speedup and eliminates randomness, resulting in a deterministic and trustworthy evaluation procedure that measures the same property as local Infidelity. DPC is evaluated on two datasets (skin lesion images and financial tabular data), two black-box models, seven explanation algorithms, and a wide range of hyperparameters. Across $4\,744$ distinct explanations, the results demonstrate that DPC, together with PC, enables a holistic and computationally efficient evaluation of both baseline-oriented and local feature attribution methods, while providing deterministic and reproducible outcomes.
Authors Kevin Iselborn, David Dembinsky, Adriano Lucieri, Andreas Dengel
Comment 13 pages, 10 figures, 5 tables, accepted at AAAI SECURE-AI4H workshop
Categories Artificial Intelligence
3
ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition
Title CN ProtoPFormer：在视觉Transformer中聚焦典型部分以实现可解释图像识别
Keywords 可解释人工智能 · 视觉Transformer · 原型网络 · 图像分类 · 注意力机制
Summary 本文提出ProtoPFormer，通过引入全局和局部原型机制解决Vision Transformer在应用原型网络时易受背景干扰的问题。全局原型提供整体语义引导，局部原型聚焦关键部件，在相互校正中提升模型的可解释性和分类性能。实验表明其在可视化与性能上优于现有原型方法。
Reason 该论文研究的是通用图像分类任务中的可解释性问题，核心是通过全局和局部原型机制提升Vision Transformer在自然图像上的可解释性。其方法针对的是物体识别中的‘背景干扰’问题，而非文档图像理解中的OCR幻觉、复杂布局解析或细粒度文本定位等核心痛点。尽管原型学习可能在理论上用于特征解耦，但该工作未涉及文本内容建模、多模态对齐或文档结构理解，迁移到DIU场景的路径不明确，属于通用计算机视觉方法在XAI方向的应用，不符合DIU本题优先或上游工具迁移的标准。
Abstract Prototypical part network (ProtoPNet) has drawn wide attention and boosted many follow-up studies due to its self-explanatory property for explainable artificial intelligence (XAI). However, when directly applying ProtoPNet on vision transformer (ViT) backbones, learned prototypes have a "distraction" problem: they have a relatively high probability of being activated by the background and pay less attention to the foreground. The powerful capability of modeling long-term dependency makes the transformer-based ProtoPNet hard to focus on prototypical parts, thus severely impairing its inherent interpretability. This paper proposes prototypical part transformer (ProtoPFormer) for appropriately and effectively applying the prototype-based method with ViTs for interpretable image recognition. The proposed method introduces global and local prototypes for capturing and highlighting the representative holistic and partial features of targets according to the architectural characteristics of ViTs. The global prototypes are adopted to provide the global view of objects to guide local prototypes to concentrate on the foreground while eliminating the influence of the background. Afterwards, local prototypes are explicitly supervised to concentrate on their respective prototypical visual parts, increasing the overall interpretability. Extensive experiments demonstrate that our proposed global and local prototypes can mutually correct each other and jointly make final decisions, which faithfully and transparently reason the decision-making processes associatively from the whole and local perspectives, respectively. Moreover, ProtoPFormer consistently achieves superior performance and visualization results over the state-of-the-art (SOTA) prototype-based baselines. Our code has been released at https://github.com/zju-vipa/ProtoPFormer.
Authors Mengqi Xue, Qihan Huang, Haofei Zhang, Jingwen Hu, Jie Song, Mingli Song, Canghong Jin
Comment Arxiv preprint; 18 pages, 12 figures, 7 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Causality Without Causal Models TARK 2025
Title CN 没有因果模型的因果性
Keywords 因果性 · 反事实推理 · 形式化方法 · 解释生成 · 抽象模型
Summary 本文提出了一种脱离具体因果模型（如结构方程模型）的因果性定义方式，通过提取原有因果定义的关键特征，使其可应用于更广泛的包含反事实语义的系统中。新框架支持处理包含析取、否定、信念甚至嵌套反事实的复杂命题，并可推广到解释生成任务。尽管具有理论深度，但缺乏与视觉或多模态系统的结合路径。
Reason 该论文属于理论因果推理的抽象形式化工作，虽然在广义因果和解释建模上有理论价值，但其方法高度抽象且未针对多模态感知或视觉语言任务设计。无法直接迁移用于解决文档图像理解中的OCR幻觉、细粒度定位或复杂排版解析等实际问题，属于与DIU无关的上游理论研究。
Abstract Perhaps the most prominent current definition of (actual) causality is due to Halpern and Pearl. It is defined using causal models (also known as structural equations models). We abstract the definition, extracting its key features, so that it can be applied to any other model where counterfactuals are defined. By abstracting the definition, we gain a number of benefits. Not only can we apply the definition in a wider range of models, including ones that allow, for example, backtracking, but we can apply the definition to determine if A is a cause of B even if A and B are formulas involving disjunctions, negations, beliefs, and nested counterfactuals (none of which can be handled by the Halpern-Pearl definition). Moreover, we can extend the ideas to getting an abstract definition of explanation that can be applied beyond causal models. Finally, we gain a deeper understanding of features of the definition even in causal models.
Authors Joseph Y. Halpern, Rafael Pass
Comment In Proceedings TARK 2025, arXiv:2511.20540
Categories Artificial Intelligence
3
Progress by Pieces: Test-Time Scaling for Autoregressive Image Generation
Title CN 逐块进步：用于自回归图像生成的测试时扩展
Keywords 自回归图像生成 · 测试时扩展 · 生成质量优化 · 提示词重写 · 渐进式生成
Summary 本文提出GridAR，一种面向自回归图像生成的测试时扩展框架。通过将画布分块并进行渐进式生成，结合早期剪枝和锚点引导机制，以及基于局部视图重写提示词来推断可行布局，从而缓解传统逐行扫描生成缺乏全局蓝图的问题。实验表明其在较低计算成本下优于Best-of-N方法。
Reason 该论文研究的是文本到图像生成中的测试时计算扩展问题，属于纯图像生成领域。虽然提出了GridAR这种在生成过程中进行剪枝和引导的框架，但其目标是提升生成图像的质量与布局合理性，而非解决文档图像理解中的OCR幻觉、复杂排版解析或细粒度文本定位等核心痛点。方法难以迁移到文档理解任务，且未涉及多模态语言模型对文档内容的结构化理解。属于无关领域（图像生成）的应用，不符合DIU本题或可迁移工具的标准。
Abstract Recent visual autoregressive (AR) models have shown promising capabilities in text-to-image generation, operating in a manner similar to large language models. While test-time computation scaling has brought remarkable success in enabling reasoning-enhanced outputs for challenging natural language tasks, its adaptation to visual AR models remains unexplored and poses unique challenges. Naively applying test-time scaling strategies such as Best-of-N can be suboptimal: they consume full-length computation on erroneous generation trajectories, while the raster-scan decoding scheme lacks a blueprint of the entire canvas, limiting scaling benefits as only a few prompt-aligned candidates are generated. To address these, we introduce GridAR, a test-time scaling framework designed to elicit the best possible results from visual AR models. GridAR employs a grid-partitioned progressive generation scheme in which multiple partial candidates for the same position are generated within a canvas, infeasible ones are pruned early, and viable ones are fixed as anchors to guide subsequent decoding. Coupled with this, we present a layout-specified prompt reformulation strategy that inspects partial views to infer a feasible layout for satisfying the prompt. The reformulated prompt then guides subsequent image generation to mitigate the blueprint deficiency. Together, GridAR achieves higher-quality results under limited test-time scaling: with N=4, it even outperforms Best-of-N (N=8) by 14.4% on T2I-CompBench++ while reducing cost by 25.6%. It also generalizes to autoregressive image editing, showing comparable edit quality and a 13.9% gain in semantic preservation on PIE-Bench over larger-N baselines.
Authors Joonhyung Park, Hyeongwon Jang, Joowon Kim, Eunho Yang
Comment Project page: https://grid-ar.github.io/
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Beyond Patch Aggregation: 3-Pass Pyramid Indexing for Vision-Enhanced Document Retrieval
Title CN 超越补丁聚合：用于视觉增强文档检索的三通路金字塔索引
Keywords 文档检索 · 视觉增强RAG · 无OCR · 金字塔索引 · 多模态检索
Summary 本文提出VisionRAG，一种无需OCR且模型无关的多模态检索系统，通过三通路金字塔索引（全局摘要、章节标题、视觉热点、事实线索）生成轻量级向量表示，在保持布局信息的同时大幅降低存储开销。检索阶段使用倒数排名融合整合多层级信号，最终将原始图像传给多模态大模型进行答案生成。在金融文档数据集上表现出良好性能，展示了OCR-free检索的可行性与可扩展性。
Reason 该论文属于典型的‘平行下游应用’（Wrapper/Application），其核心是构建一个基于视觉的RAG系统（VisionRAG）用于文档检索，本质上是将现有VLM和图像检索技术应用于特定流程中。尽管提出了三通路金字塔索引以减少存储开销，但方法本身未对文档理解中的核心痛点（如OCR幻觉、密集文本识别、复杂排版建模、长文档推理）提供可迁移的底层技术创新。提出的‘summary-guided retrieval’仍属应用级设计，无法直接提升DIU模型的细粒度理解或去幻觉能力。
Abstract Document centric RAG pipelines usually begin with OCR, followed by brittle heuristics for chunking, table parsing, and layout reconstruction. These text first workflows are costly to maintain, sensitive to small layout shifts, and often lose the spatial cues that contain the answer. Vision first retrieval has emerged as a strong alternative. By operating directly on page images, systems like ColPali and ColQwen preserve structure and reduce pipeline complexity while achieving strong benchmark performance. However, these late interaction models tie retrieval to a specific vision backbone and require storing hundreds of patch embeddings per page, creating high memory overhead and complicating large scale deployment. We introduce VisionRAG, a multimodal retrieval system that is OCR free and model agnostic. VisionRAG indexes documents directly as images, preserving layout, tables, and spatial cues, and builds semantic vectors without committing to a specific extraction. Our three pass pyramid indexing framework creates vectors using global page summaries, section headers, visual hotspots, and fact level cues. These summaries act as lightweight retrieval surrogates. At query time, VisionRAG retrieves the most relevant pages using the pyramid index, then forwards the raw page image encoded as base64 to a multimodal LLM for final question answering. During retrieval, reciprocal rank fusion integrates signals across the pyramid to produce robust ranking. VisionRAG stores only 17 to 27 vectors per page, matching the efficiency of patch based methods while staying flexible across multimodal encoders. On financial document benchmarks, it achieves 0.8051 accuracy at 10 on FinanceBench and 0.9629 recall at 100 on TAT DQA. These results show that OCR free, summary guided multimodal retrieval is a practical and scalable alternative to traditional text extraction pipelines.
Authors Anup Roy, Rishabh Gyanendra Upadhyay, Animesh Rameshbhai Panara, Robin Mills
Categories Artificial Intelligence
3
Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applications
Title CN LLM系统中的失效模式：面向可靠AI应用的系统级分类法
Keywords LLM可靠性 · 系统失效模式 · 推理漂移 · 上下文退化 · 系统工程
Summary 本文提出了一个包含15种隐性失效模式的系统级分类法，用于分析大语言模型在实际应用中的可靠性问题，如推理漂移、工具误调用和版本退化等。指出现有评测基准忽视稳定性与可复现性，并呼吁从系统工程角度构建可靠的LLM系统。
Reason 该论文提出了一种针对LLM系统的失效模式分类法，聚焦于多步推理漂移、上下文边界退化等系统性问题。尽管其对可靠性分析有一定价值，但属于通用LLM系统工程的讨论，并未涉及视觉输入、文档结构或OCR相关挑战。没有方法论创新可直接迁移至文档图像理解（DIU）领域，如去幻觉、细粒度定位或高分辨率处理。属于平行应用层面的系统观察，不提供底层技术工具。
Abstract Large language models (LLMs) are being rapidly integrated into decision-support tools, automation workflows, and AI-enabled software systems. However, their behavior in production environments remains poorly understood, and their failure patterns differ fundamentally from those of traditional machine learning models. This paper presents a system-level taxonomy of fifteen hidden failure modes that arise in real-world LLM applications, including multi-step reasoning drift, latent inconsistency, context-boundary degradation, incorrect tool invocation, version drift, and cost-driven performance collapse. Using this taxonomy, we analyze the growing gap in evaluation and monitoring practices: existing benchmarks measure knowledge or reasoning but provide little insight into stability, reproducibility, drift, or workflow integration. We further examine the production challenges associated with deploying LLMs - including observability limitations, cost constraints, and update-induced regressions - and outline high-level design principles for building reliable, maintainable, and cost-aware LLM systems. Finally, we outline high-level design principles for building reliable, maintainable, and cost-aware LLM-based systems. By framing LLM reliability as a system-engineering problem rather than a purely model-centric one, this work provides an analytical foundation for future research on evaluation methodology, AI system robustness, and dependable LLM deployment.
Authors Vaishali Vinay
Categories Artificial Intelligence
3
MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts
Title CN MLPMoE：将密集LLM的MLP模块零样本转化为静态混合专家
Keywords MLP · Mixture-of-Experts · 模型压缩 · 零样本转换 · 推理效率
Summary 本文提出MLPMoE，一种无需训练的模型重构技术，通过张量切分将密集LLM中的MLP层转化为静态混合专家结构，实现参数级稀疏化并保持性能稳定。方法适用于Qwen和Llama等模型，在不增加训练成本的前提下实现约20%的参数压缩。然而，该工作完全脱离视觉或多模态场景，属于典型的纯语言模型效率优化研究。
Reason 该论文聚焦于大语言模型中MLP层的结构重构，提出一种无需训练、基于张量切片的Mixture-of-Experts转换方法，旨在提升推理效率。尽管其在模型压缩和稀疏计算上有一定创新，但完全属于纯NLP模型架构优化范畴，未涉及任何视觉输入、多模态对齐或文档结构建模。与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂版面分析、细粒度定位等无直接关联，也无法迁移用于处理高分辨率文档图像或多模态文档推理任务。
Abstract Large Language Models (LLMs) are predominantly deployed as dense transformers, where every parameter in every feed-forward block is activated for every token. While architecturally simple, this is computationally inefficient, since inference costs scale linearly with parameter count. Recent upcycling methods such as MoEfication, CMoE, ToMoE, and MoORE reveal that much of the useful computation lives in sparse, semi-modular substructures inside dense feed-forward networks, but these approaches typically rely on clustering, activation profiling, singular value decomposition, or custom routing that requires calibration data. This paper introduces MLPMoE (MLP Mixture-of-Experts), a training-free, deterministic transformation that restructures the dense MLP in transformer blocks into a static, high-cardinality mixture of experts. The transformation uses simple tensor slicing and summation, reinterpreting the algebra of tensor parallelism as a topological conversion rather than a distributed training pattern. We further introduce Fractal Fade (differential branch sparsity) and Compensated Pruning (variance-preserving branch reduction) as lightweight mechanisms for structured sparsity. On Qwen2.5-0.5B-Instruct and DeepSeek-R1-Distill-Llama-8B, the zero-shot MLPMoE transform changes a proxy perplexity metric by less than 0.05 percent while keeping the parameter count effectively constant. On the 8B model, differential sparsity removes about 20 percent of MLP parameters while keeping perplexity within about 2 percent of the dense baseline. The method operates entirely post hoc on existing checkpoints and does not require gradients, calibration sets, or router training. Code is available at https://gist.github.com/iwallarm/fc2ef1eddf226ca7814f9e5e2ae9bad1
Authors Ivan Novikov
Categories Artificial Intelligence
3
OVOD-Agent: A Markov-Bandit Framework for Proactive Visual Reasoning and Self-Evolving Detection
Title CN OVOD-Agent：一种用于主动视觉推理和自进化检测的马尔可夫-_bandit_框架
Keywords 开放词汇目标检测 · 视觉推理 · 马尔可夫决策过程 · bandit算法 · 自演化检测
Summary 本文提出OVOD-Agent，将开放词汇目标检测从被动匹配转化为主动推理过程，通过构建弱马尔可夫决策过程（w-MDP）建模视觉上下文转换，并引入Bandit模块生成探索信号以聚焦不确定区域，同时利用Markov转移矩阵与Bandit轨迹进行自监督奖励模型优化，形成闭环学习机制。实验表明该方法在COCO和LVIS数据集上对多种OVOD主干网络均有提升，尤其改善了对罕见类别的检测性能。
Reason 该论文研究的是开放词汇目标检测（OVOD），属于通用计算机视觉任务，其核心创新在于通过马尔可夫决策过程与bandit机制结合实现主动推理和自演化检测。尽管引入了类似Chain-of-Thought的Visual-CoT设计，但其应用场景和问题设定与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、长文档推理）关联较弱。所提方法针对的是类别泛化和稀有物体检测，迁移至文档领域的细粒度文本识别、布局分析或VQA任务存在显著障碍，且未涉及多模态语言模型在文档上的应用或视觉token压缩等DIU关键方向。
Abstract Open-Vocabulary Object Detection (OVOD) aims to enable detectors to generalize across categories by leveraging semantic information. Although existing methods are pretrained on large vision-language datasets, their inference is still limited to fixed category names, creating a gap between multimodal training and unimodal inference. Previous work has shown that improving textual representation can significantly enhance OVOD performance, indicating that the textual space is still underexplored. To this end, we propose OVOD-Agent, which transforms passive category matching into proactive visual reasoning and self-evolving detection. Inspired by the Chain-of-Thought (CoT) paradigm, OVOD-Agent extends the textual optimization process into an interpretable Visual-CoT with explicit actions. OVOD's lightweight nature makes LLM-based management unsuitable; instead, we model visual context transitions as a Weakly Markovian Decision Process (w-MDP) over eight state spaces, which naturally represents the agent's state, memory, and interaction dynamics. A Bandit module generates exploration signals under limited supervision, helping the agent focus on uncertain regions and adapt its detection policy. We further integrate Markov transition matrices with Bandit trajectories for self-supervised Reward Model (RM) optimization, forming a closed loop from Bandit exploration to RM learning. Experiments on COCO and LVIS show that OVOD-Agent provides consistent improvements across OVOD backbones, particularly on rare categories, confirming the effectiveness of the proposed framework.
Authors Chujie Wang, Jianyu Lu, Zhiyuan Luo, Xi Chen, Chu He
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
CroMe: Multimodal Fake News Detection using Cross-Modal Tri-Transformer and Metric Learning
Title CN CroMe：基于跨模态三重变换器与度量学习的多模态虚假新闻检测
Keywords 多模态虚假新闻检测 · 跨模态学习 · 度量学习 · 三重变换器 · BLIP2
Summary 本文提出CroMe模型用于多模态虚假新闻检测，通过BLIP2提取文本、图像及图文联合表征，引入代理锚点度量学习捕捉模态内关系，并设计跨模态三重变换器进行特征融合。实验表明该方法在虚假新闻检测任务上表现优异。
Reason 该论文属于特定应用场景（虚假新闻检测）的多模态融合方法研究，本质上是将现有VLM（如BLIP2）和度量学习技术应用于下游任务，未提出可用于文档图像理解的底层技术创新。尽管使用了跨模态对齐机制，但其架构设计针对社交媒体图文内容，与文档结构分析、OCR去幻觉、细粒度文本定位等DIU核心问题关联薄弱，属于‘平行下游应用’，不符合本方向关注的技术源头创新要求。
Abstract Multimodal Fake News Detection has received increasing attention recently. Existing methods rely on independently encoded unimodal data and overlook the advantages of capturing intra-modality relationships and integrating inter-modal similarities using advanced techniques. To address these issues, Cross-Modal Tri-Transformer and Metric Learning for Multimodal Fake News Detection (CroMe) is proposed. CroMe utilizes Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (BLIP2) as encoders to capture detailed text, image and combined image-text representations. The metric learning module employs a proxy anchor method to capture intra-modality relationships while the feature fusion module uses a Cross-Modal and Tri-Transformer for effective integration. The final fake news detector processes the fused features through a classifier to predict the authenticity of the content. Experiments on datasets show that CroMe excels in multimodal fake news detection.
Authors Eunjee Choi, Junhyun Ahn, XinYu Piao, Jong-Kook Kim
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Structure-Aware Prototype Guided Trusted Multi-View Classification
Title CN 面向可信多视图分类的结构感知原型引导方法
Keywords 多视图分类 · 原型学习 · 结构一致性 · 可信学习 · 邻域建模
Summary 本文提出一种基于原型的可信多视图分类框架，通过引入原型来表示各视图的邻域结构，简化视图内关系学习，并实现视图间结构的动态对齐，从而提升多视图分类的鲁棒性和一致性。实验表明该方法在多个公开数据集上具有竞争力的性能。
Reason 该论文研究的是多视图分类（Multi-View Classification）中的信任决策问题，核心关注点是跨视图的一致性建模与邻域结构学习，属于通用多视图学习范畴。尽管其方法可能涉及视觉特征，但任务本身与文档图像理解（如DocVQA、布局分析、表格识别等）无直接关联。未解决OCR幻觉、密集文本处理、复杂排版或长文档推理等DIU核心痛点，且不属于可迁移至文档理解的底层VLM架构或推理机制创新，属于平行应用类方法论延伸，不符合本筛选标准。
Abstract Trustworthy multi-view classification (TMVC) addresses the challenge of achieving reliable decision-making in complex scenarios where multi-source information is heterogeneous, inconsistent, or even conflicting. Existing TMVC approaches predominantly rely on globally dense neighbor relationships to model intra-view dependencies, leading to high computational costs and an inability to directly ensure consistency across inter-view relationships. Furthermore, these methods typically aggregate evidence from different views through manually assigned weights, lacking guarantees that the learned multi-view neighbor structures are consistent within the class space, thus undermining the trustworthiness of classification outcomes. To overcome these limitations, we propose a novel TMVC framework that introduces prototypes to represent the neighbor structures of each view. By simplifying the learning of intra-view neighbor relations and enabling dynamic alignment of intra- and inter-view structure, our approach facilitates more efficient and consistent discovery of cross-view consensus. Extensive experiments on multiple public multi-view datasets demonstrate that our method achieves competitive downstream performance and robustness compared to prevalent TMVC methods.
Authors Haojian Huang, Jiahao Shi, Zhe Liu, Harold Haodong Chen, Han Fang, Hao Sun, Zhongjiang He
Comment 12 pages, 8 figures, 7 tables, Ongoing Work
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
ICPO: Intrinsic Confidence-Driven Group Relative Preference Optimization for Efficient Reinforcement Learning
Title CN ICPO：基于内在置信度的组相对偏好优化以实现高效的强化学习
Keywords 强化学习 · 偏好优化 · 置信度 · 大语言模型 · 推理
Summary 本文提出ICPO方法，利用大语言模型生成不同响应时的概率差异作为内在置信度信号，结合可验证奖励进行相对偏好优化，提升推理类任务的强化学习效率。实验显示其在多个数学和通用推理基准上优于GRPO。然而，该工作完全基于纯文本模态，未涉及图像或文档布局信息处理。
Reason 该论文聚焦于大语言模型推理过程中的强化学习优化，提出通过生成概率的相对比较来构建偏好优势分数，从而改善奖励稀疏和噪声问题。尽管其方法在通用LLM推理中有意义，但并未涉及多模态输入、视觉信息处理或文档结构建模，与文档图像理解（DIU）的核心挑战如OCR幻觉、细粒度定位、复杂排版等无直接关联。属于通用NLP推理优化范畴，且未展示对视觉-语言对齐或高分辨率视觉输入的支持能力，无法迁移用于解决DIU关键痛点。
Abstract Reinforcement Learning with Verifiable Rewards (RLVR) demonstrates significant potential in enhancing the reasoning capabilities of Large Language Models (LLMs). However, existing RLVR methods are often constrained by issues such as coarse-grained rewards, reward noise, and inefficient exploration, which lead to unstable training and entropy collapse. To address this challenge, we propose the Intrinsic Confidence-Driven Group Relative Preference Optimization method (ICPO). The intuition behind it lies in the fact that the probabilities of an LLM generating different responses can inherently and directly reflect its self-assessment of the reasoning process. Inspired by the idea of preference modeling, ICPO calculates a preference advantage score for each response by comparing the relative generation probabilities of multiple responses under the same input prompt, and integrates this score with verifiable rewards to guide the exploration process. We have discovered that the preference advantage score not only alleviates the issues of coarse-grained rewards and reward noise but also effectively curbs overconfident errors, enhances the relative superiority of undervalued high-quality responses, and prevents the model from overfitting to specific strategies, thereby facilitating more thorough exploration. Comprehensive experiments across four general-domain benchmarks and three mathematical benchmarks demonstrate that ICPO steadily boosts reasoning compared to GRPO.
Authors Jinpeng Wang, Chao Li, Ting Ye, Mengyuan Zhang, Wei Liu, Jian Luan
Categories Artificial Intelligence
3
Towards Audio Token Compression in Large Audio Language Models
Title CN 面向大型音频语言模型中的音频令牌压缩
Keywords 音频语言模型 · 令牌压缩 · 无监督分段 · 平均池化 · 低秩适配
Summary 本文探讨了在大型音频语言模型（LALM）中减少音频令牌数量的技术，以缓解注意力机制的计算负担并提升长音频处理能力。通过在音频编码器输出后引入无监督分段、均匀平均池化等方式进行令牌压缩，并使用低秩适配器微调模型以减轻性能下降。实验表明，在语音识别和语音到语音翻译任务上，压缩后的模型能在令牌数减少三倍的情况下接近原始性能。
Reason 该论文研究的是音频模态下的令牌压缩问题，尽管其‘压缩’思想看似与文档图像理解中的视觉令牌压缩（如DeepSeek-OCR）有形式上的相似性，但音频信号的时序特性与文档图像的空间结构差异巨大。文档图像理解的核心挑战在于复杂排版、细粒度文本识别和多模态对齐，而音频令牌压缩主要处理时间序列降采样与语义保留，迁移价值有限。此外，方法上采用的是无监督分段和平均池化等传统时序压缩手段，并未提出可泛化的跨模态压缩原理，难以直接迁移到高分辨率文档图像的视觉token压缩任务中。
Abstract Large Audio Language Models (LALMs) demonstrate impressive performance across diverse tasks, ranging from speech recognition to general audio understanding. However, their scalability is limited by the quadratic complexity of attention and the high token rates of audio signals. These challenges make it difficult to extend LALMs to long-form audio and to deploy them on resource-constrained platforms such as edge devices. In this paper, we explore techniques such as unsupervised segmentation, uniform average pooling, etc., to reduce the number of audio tokens generated by the LALM's audio encoder but before they are consumed by the LLM decoder. To mitigate potential performance degradation introduced by the compressed representations, we employ low-rank adapters to finetune the model. We evaluate our proposed models on two tasks, automatic speech recognition and speech-to-speech translation tasks, that are dependent on effectively uncovering the underlying lexical content of the input signal and study the effect of downsampling on these tasks. Experimental results show that compressed LALMs can achieve performance closer to frame-level LALMs while reducing the input audio token count upto three times before the LLM backbone.
Authors Saurabhchand Bhati, Samuel Thomas, Hilde Kuehne, Rogerio Feris, James Glass
Categories Artificial Intelligence, Computation and Language
3
Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following
Title CN Multi-Crit：在多元标准下对多模态评判模型的基准测试
Keywords 多模态评估 · 评判模型 · 基准测试 · 标准遵循 · 人类偏好
Summary 本文提出了Multi-Crit，一个用于评估多模态大模型在遵循多元、细粒度评判标准方面能力的基准。数据集包含开放式生成和可验证推理任务，并通过人工标注构建了多标准响应对。引入了三个新指标来衡量模型在标准遵循、切换灵活性和冲突识别上的表现。实验分析了25个主流多模态模型的表现，发现现有模型在一致遵循多元标准方面存在明显不足，尤其是开源模型。研究还探讨了批评者微调、推理时扩展等策略的效果。
Reason 该论文聚焦于评估多模态大模型作为‘评判者’在多元评价标准下的表现，属于AI评估（AI evaluation）领域的元研究。尽管涉及多模态判断和细粒度标准遵循，但其任务设定与文档图像理解（DIU）的核心问题如OCR、布局分析、表格识别、视觉幻觉缓解等无直接关联。它并未提出可用于提升DocVQA或文档理解性能的底层架构、训练方法或推理机制。本质上是为评估LMMs服务的基准构建工作，属于平行应用性质的工具性研究，无法迁移解决DIU中的密集文本、复杂排版或长文档推理等关键挑战。
Abstract Large multimodal models (LMMs) are increasingly adopted as judges in multimodal evaluation systems due to their strong instruction following and consistency with human preferences. However, their ability to follow diverse, fine-grained evaluation criteria remains underexplored. We develop Multi-Crit, a benchmark for evaluating multimodal judges on their capacity to follow pluralistic criteria and produce reliable criterion-level judgments. Covering both open-ended generation and verifiable reasoning tasks, Multi-Crit is built through a rigorous data curation pipeline that gathers challenging response pairs with multi-criterion human annotations. It further introduces three novel metrics for systematically assessing pluralistic adherence, criterion-switching flexibility, and the ability to recognize criterion-level preference conflicts. Comprehensive analysis of 25 LMMs reveals that 1) proprietary models still struggle to maintain consistent adherence to pluralistic criteria--especially in open-ended evaluation; 2) open-source models lag further behind in flexibly following diverse criteria; and 3) critic fine-tuning with holistic judgment signals enhances visual grounding but fails to generalize to pluralistic criterion-level judgment. Additional analyses on reasoning fine-tuning, test-time scaling, and boundary consistency between open-source and proprietary models further probe the limits of current multimodal judges. As a pioneering study, Multi-Crit lays the foundation for building reliable and steerable multimodal AI evaluation.
Authors Tianyi Xiong, Yi Ge, Ming Li, Zuolong Zhang, Pranav Kulkarni, Kaishen Wang, Qi He, Zeying Zhu, Chenxi Liu, Ruibo Chen, Tong Zheng, Yanshuo Chen, Xiyao Wang, Renrui Zhang, Wenhu Chen, Heng Huang
Categories Computer Vision and Pattern Recognition
3
TinyChemVL: Advancing Chemical Vision-Language Models via Efficient Visual Token Reduction and Complex Reaction Tasks AAAI 2026
Title CN TinyChemVL：通过高效的视觉令牌缩减和复杂反应任务推进化学视觉语言模型
Keywords 视觉语言模型 · 视觉令牌压缩 · 化学图像理解 · 反应预测 · 效率优化
Summary 本文提出了TinyChemVL，一种面向化学领域的高效视觉语言模型，通过减少视觉令牌数量来提升推理与训练效率，并引入反应级任务以增强化学推理能力。同时发布了化学反应理解基准ChemRxn-V。模型在仅使用1/16视觉令牌的情况下超越了ChemVLM的表现。
Reason 该论文聚焦于化学领域的视觉语言建模，提出了一种针对分子图像的视觉令牌压缩方法，并构建了反应级任务基准ChemRxn-V。尽管其采用了视觉令牌减少（Visual Token Reduction）技术，具有一定的效率创新，但整体工作属于特定垂直领域（化学结构理解）的应用型研究，任务设计和数据集均围绕化学分子与反应展开，未针对文档图像理解中的通用痛点（如OCR幻觉、复杂版式、长文档推理）进行建模。其方法虽涉及视觉压缩，但迁移至通用DIU场景的通用性和适配性较弱，属于‘平行下游应用’，缺乏对文档理解核心问题的方法论贡献。
Abstract While Vision Language Models (VLMs) have demonstrated remarkable capabilities in general visual understanding, their application in the chemical domain has been limited, with previous works predominantly focusing on text and thus overlooking critical visual information, such as molecular structures. Current approaches that directly adopt standard VLMs for chemical tasks suffer from two primary issues: (i) computational inefficiency of processing entire chemical images with non-informative backgrounds. (ii) a narrow scope on molecular-level tasks that restricts progress in chemical reasoning. In this work, we propose \textbf{TinyChemVL}, an efficient and powerful chemical VLM that leverages visual token reduction and reaction-level tasks to improve model efficiency and reasoning capacity. Also, we propose \textbf{ChemRxn-V}, a reaction-level benchmark for assessing vision-based reaction recognition and prediction tasks. Directly predicting reaction products from molecular images poses a non-trivial challenge, as it requires models to integrate both recognition and reasoning capacities. Our results demonstrate that with only 4B parameters, TinyChemVL achieves superior performance on both molecular and reaction tasks while demonstrating faster inference and training speeds compared to existing models. Notably, TinyChemVL outperforms ChemVLM while utilizing only 1/16th of the visual tokens. This work builds efficient yet powerful VLMs for chemical domains by co-designing model architecture and task complexity.
Authors Xuanle Zhao, Shuxin Zeng, Xinyuan Cai, Xiang Cheng, Duzhen Zhang, Xiuyi Chen, Bo Xu
Comment Accepted by AAAI 2026
Categories Computer Vision and Pattern Recognition
3
EoS-FM: Can an Ensemble of Specialist Models act as a Generalist Feature Extractor?
Title CN EoS-FM：专家模型集成能否作为通用特征提取器？
Keywords 遥感基础模型 · 专家集成 · 模块化训练 · 高效AI · 联邦学习
Summary 本文提出一种面向遥感领域的专家模型集成框架（EoS-FM），通过训练多个冻结的、任务特定的ConvNeXtV2模型，并将其组合为一个通用特征提取器。该方法强调效率、可解释性和可扩展性，支持联邦学习和持续集成，在遥感领域内具有创新性。
Reason 该论文研究的是遥感（Earth Observation）领域的基础模型构建，提出通过轻量级、任务特定的专家模型集成来构建遥感基础模型。尽管其模块化、高效训练的设计理念具有一定通用性，但其应用场景与文档图像理解（DIU）差异巨大，未涉及文本识别、布局分析、VQA等核心问题。方法上也未针对OCR幻觉、细粒度定位或复杂排版建模进行设计，迁移价值极低。
Abstract Recent advances in foundation models have shown great promise in domains such as natural language processing and computer vision, and similar efforts are now emerging in the Earth Observation community. These models aim to generalize across tasks with limited supervision, reducing the need for training separate models for each task. However, current strategies, which largely focus on scaling model size and dataset volume, require prohibitive computational and data resources, limiting accessibility to only a few large institutions. Moreover, this paradigm of ever-larger models stands in stark contrast with the principles of sustainable and environmentally responsible AI, as it leads to immense carbon footprints and resource inefficiency. In this work, we present a novel and efficient alternative: an Ensemble-of-Specialists framework for building Remote Sensing Foundation Models (RSFMs). Our method decomposes the training process into lightweight, task-specific ConvNeXtV2 specialists that can be frozen and reused. This modular approach offers strong advantages in efficiency, interpretability, and extensibility. Moreover, it naturally supports federated training, pruning, and continuous specialist integration, making it particularly well-suited for collaborative and resource-constrained settings. Our framework sets a new direction for building scalable and efficient RSFMs.
Authors Pierre Adorni, Minh-Tan Pham, Stéphane May, Sébastien Lefèvre
Categories Computer Vision and Pattern Recognition
3
CanKD: Cross-Attention-based Non-local operation for Feature-based Knowledge Distillation WACV 2026
Title CN CanKD：基于交叉注意力的非局部操作用于基于特征的知识蒸馏
Keywords 知识蒸馏 · 交叉注意力 · 非局部操作 · 特征对齐 · 计算机视觉
Summary 本文提出了一种名为CanKD的新型基于特征的知识蒸馏框架，利用交叉注意力机制实现学生网络与教师网络之间的非局部特征交互。相比传统的自注意力方法，CanKD允许学生特征图中的每个像素动态关注教师特征图中所有位置，从而更全面地捕捉像素间关系，提升知识迁移效率。实验表明该方法在目标检测和语义分割任务上优于现有蒸馏方法。
Reason 该论文属于典型的计算机视觉通用方法改进，研究的是知识蒸馏中的特征对齐机制，应用于目标检测和图像分割任务。虽然其交叉注意力机制在理论上可能增强特征匹配，但并未针对文档图像理解中的核心痛点（如OCR幻觉、复杂排版、细粒度文本定位或长文档结构建模）进行设计或验证。方法本质上是通用视觉模型压缩技术，属于‘平行应用’类的底层组件，缺乏向DIU迁移的具体路径和动机，不符合本领域方法论创新要求。
Abstract We propose Cross-Attention-based Non-local Knowledge Distillation (CanKD), a novel feature-based knowledge distillation framework that leverages cross-attention mechanisms to enhance the knowledge transfer process. Unlike traditional self-attention-based distillation methods that align teacher and student feature maps independently, CanKD enables each pixel in the student feature map to dynamically consider all pixels in the teacher feature map. This non-local knowledge transfer more thoroughly captures pixel-wise relationships, improving feature representation learning. Our method introduces only an additional loss function to achieve superior performance compared with existing attention-guided distillation methods. Extensive experiments on object detection and image segmentation tasks demonstrate that CanKD outperforms state-of-the-art feature and hybrid distillation methods. These experimental results highlight CanKD's potential as a new paradigm for attention-guided distillation in computer vision tasks. Code is available at https://github.com/tori-hotaru/CanKD
Authors Shizhe Sun, Wataru Ohyama
Comment WACV 2026 Accepted
Categories Computer Vision and Pattern Recognition
3
Geometrically Regularized Transfer Learning with On-Manifold and Off-Manifold Perturbation
Title CN 基于流形内外扰动的几何正则化迁移学习
Keywords 域适应 · 对抗数据增强 · 流形正则化 · 迁移学习 · 决策边界平滑
Summary 本文提出MAADA框架，通过将对抗扰动分解为流形内（on-manifold）和流形外（off-manifold）两部分，结合几何感知对齐损失来缩小源域与目标域之间的测地线差异，从而提升跨域迁移性能。在多个标准域适应数据集上验证了其有效性。
Reason 该论文研究的是通用视觉领域的域自适应问题，提出了一种分解对抗扰动为流形内和流形外成分的数据增强方法。尽管其理论涉及模型鲁棒性和泛化性，但工作属于通用图像分类下的迁移学习范畴，未针对文档图像理解中的核心痛点（如OCR幻觉、密集文本、复杂排版或长文档推理）设计解决方案。所提方法在文档图像上的可迁移性不明确，且无任何与多模态语言模型、视觉-语言对齐、高分辨率处理或细粒度定位相关的机制。属于通用CV方法，对DIU缺乏直接指导意义。
Abstract Transfer learning under domain shift remains a fundamental challenge due to the divergence between source and target data manifolds. In this paper, we propose MAADA (Manifold-Aware Adversarial Data Augmentation), a novel framework that decomposes adversarial perturbations into on-manifold and off-manifold components to simultaneously capture semantic variation and model brittleness. We theoretically demonstrate that enforcing on-manifold consistency reduces hypothesis complexity and improves generalization, while off-manifold regularization smooths decision boundaries in low-density regions. Moreover, we introduce a geometry-aware alignment loss that minimizes geodesic discrepancy between source and target manifolds. Experiments on DomainNet, VisDA, and Office-Home show that MAADA consistently outperforms existing adversarial and adaptation methods in both unsupervised and few-shot settings, demonstrating superior structural robustness and cross-domain generalization.
Authors Hana Satou, Alan Mitkiy, Emma Collins, Finn Kingston
Categories Computer Vision and Pattern Recognition
3
Disentangled Geometric Alignment with Adaptive Contrastive Perturbation for Reliable Domain Transfer
Title CN 基于自适应对比扰动的解耦几何对齐以实现可靠的域迁移
Keywords 域适应 · 几何对齐 · 特征解耦 · 对比学习 · 迁移学习
Summary 本文提出GAMA++，一种用于域适应的几何感知方法，通过潜在空间解耦和自适应对比扰动策略来提升跨域语义对齐效果，在DomainNet等通用视觉基准上取得SOTA。方法聚焦于类别流形的对齐与扰动优化，但未考虑文本结构、布局信息或OCR相关挑战。
Reason 该论文属于通用视觉领域的域自适应方法研究，尽管其在几何对齐和特征解耦方面有一定创新，但核心任务与文档图像理解（DIU）无直接关联。未涉及DocVQA、OCR、版面分析等DIU核心问题，也未提出可迁移至文档理解场景的通用机制（如高分辨率处理、细粒度定位或幻觉抑制）。属于通用CV方法在域适应任务上的改进，不符合DIU本题要求，且无明确迁移路径到文档理解痛点。
Abstract Despite progress in geometry-aware domain adaptation, current methods such as GAMA still suffer from two unresolved issues: (1) insufficient disentanglement of task-relevant and task-irrelevant manifold dimensions, and (2) rigid perturbation schemes that ignore per-class alignment asymmetries. To address this, we propose GAMA++, a novel framework that introduces (i) latent space disentanglement to isolate label-consistent manifold directions from nuisance factors, and (ii) an adaptive contrastive perturbation strategy that tailors both on- and off-manifold exploration to class-specific manifold curvature and alignment discrepancy. We further propose a cross-domain contrastive consistency loss that encourages local semantic clusters to align while preserving intra-domain diversity. Our method achieves state-of-the-art results on DomainNet, Office-Home, and VisDA benchmarks under both standard and few-shot settings, with notable improvements in class-level alignment fidelity and boundary robustness. GAMA++ sets a new standard for semantic geometry alignment in transfer learning.
Authors Emma Collins, Myungseo wong, Kim Yun, Finn Kingston, Hana Satou
Categories Computer Vision and Pattern Recognition
3
Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuning
Title CN 用边界框思考：通过强化微调增强时空视频定位
Keywords 视频定位 · 多模态大模型 · 强化微调 · 思维链 · 区域对齐
Summary 本文提出STVG-o1框架，首次实现无需修改架构即可让现成的多模态大语言模型（MLLM）在时空视频定位（STVG）任务上达到SOTA性能。方法引入‘边界框思维链’机制，在输出前显式推理时空位置，并设计包含格式、一致性、时间、空间和思维奖励的多维强化学习目标进行微调。在多个数据集上大幅超越现有MLLM方法，并展现出强泛化能力。
Reason 该论文研究的是视频领域的时空定位（STVG）任务，属于视频理解范畴，尽管使用了多模态大模型和细粒度对齐机制，但其核心应用场景与文档图像理解（DIU）无关。虽然其提出的边界框链式推理和强化奖励机制在理论上可能对定位任务有启发，但视频时序建模、动态对象追踪等核心挑战与文档中的静态文本布局分析、OCR去幻觉、密集排版理解等痛点差异显著，迁移价值极低。
Abstract Spatio-temporal video grounding (STVG) requires localizing a target object in untrimmed videos both temporally and spatially from natural language descriptions. Despite their strong language understanding, multimodal large language models (MLLMs) underperform on STVG due to misaligned training objectives and weak fine-grained region-word alignment in standard visual encoders. To address this, we propose STVG-o1, the first framework that enables off-the-shelf MLLMs to achieve state-of-the-art STVG performance without any architectural modifications. Our method introduces a bounding-box chain-of-thought mechanism that explicitly reasons about spatio-temporal locations in an intermediate step before producing the final prediction. We further design a multi-dimensional reinforcement reward function consisting of format, consistency, temporal, spatial, and think rewards, which provides geometry-aware supervision through reinforcement fine-tuning. Evaluated on HCSTVG-v1/v2 and VidSTG, STVG-o1 sets new state-of-the-art results on HCSTVG, outperforming the best task-specific method by 7.3\% m\_tIoU on HCSTVG-v1, matching specialized models on VidSTG, and surpassing all existing MLLM-based approaches by large margins. It also demonstrates strong open-vocabulary generalization across datasets, establishing MLLMs as viable and powerful backbones for precise spatio-temporal grounding. Our code and models will be released.
Authors Xin Gu, Haoji Zhang, Qihang Fan, Jingxuan Niu, Zhipeng Zhang, Libo Zhang, Guang Chen, Fan Chen, Longyin Wen, Sijie Zhu
Categories Computer Vision and Pattern Recognition
3
Unsupervised Segmentation by Diffusing, Walking and Cutting The IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2026
Title CN 通过扩散、行走和切割实现无监督分割
Keywords 无监督分割 · 扩散模型 · 自注意力 · 随机游走 · 归一化割
Summary 本文提出一种基于预训练扩散模型自注意力机制的无监督图像分割方法，将自注意力概率解释为图像块之间的转移矩阵，结合归一化割（NCuts）进行递归分割，实现零样本下的层次化语义分割，在COCO-Stuff-27和Cityscapes上达到SOTA。无需额外训练。
Reason 该论文属于通用图像分割领域，使用扩散模型的自注意力机制进行无监督语义分割，虽然技术上有创新，但其方法面向自然图像（如COCO-Stuff、Cityscapes），并未针对文档图像特有的结构（如文本流、表格、密集排版）设计，且未涉及OCR、VQA或文档布局分析等DIU核心任务。尽管‘随机游走’和‘归一化割’可能在理论上对区域划分有启发，但迁移到文档理解场景中缺乏直接路径，尤其是无法解决OCR幻觉、细粒度定位或长文档推理等关键痛点。
Abstract We propose an unsupervised image segmentation method using features from pre-trained text-to-image diffusion models. Inspired by classic spectral clustering approaches, we construct adjacency matrices from self-attention layers between image patches and recursively partition using Normalised Cuts. A key insight is that self-attention probability distributions, which capture semantic relations between patches, can be interpreted as a transition matrix for random walks across the image. We leverage this by first using Random Walk Normalized Cuts directly on these self-attention activations to partition the image, minimizing transition probabilities between clusters while maximizing coherence within clusters. Applied recursively, this yields a hierarchical segmentation that reflects the rich semantics in the pre-trained attention layers, without any additional training. Next, we explore other ways to build the NCuts adjacency matrix from features, and how we can use the random walk interpretation of self-attention to capture long-range relationships. Finally, we propose an approach to automatically determine the NCut cost criterion, avoiding the need to tune this manually. We quantitatively analyse the effect incorporating different features, a constant versus dynamic NCut threshold, and incorporating multi-node paths when constructing the NCuts adjacency matrix. We show that our approach surpasses all existing methods for zero-shot unsupervised segmentation, achieving state-of-the-art results on COCO-Stuff-27 and Cityscapes.
Authors Daniela Ivanova, Marco Aversa, Paul Henderson, John Williamson
Comment Accepted to The IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2026
Categories Computer Vision and Pattern Recognition
3
Co-Training Vision Language Models for Remote Sensing Multi-task Learning
Title CN 通过协同训练视觉语言模型实现遥感多任务学习
Keywords 视觉语言模型 · 多任务学习 · 遥感图像 · 动态分辨率 · VLM
Summary 本文提出RSCoVLM，一个用于遥感多任务学习的视觉语言模型，通过构建数据策展引擎和统一的动态分辨率策略，在遥感图像理解、定位和超高清图像推理等任务上取得SOTA表现。引入Zoom-in Chain机制以应对UHR图像，并发布配套数据集LRS-VQA-Zoom。所有代码、模型和数据均已开源。
Reason 该论文聚焦于遥感（Remote Sensing）图像理解与多任务学习，尽管使用了VLM架构并涉及高分辨率图像处理，但其应用场景和数据特性与文档图像理解（DIU）差异显著。遥感图像中的‘文本’语义薄弱，排版结构与文档完全不同，且核心挑战（如地物识别、尺度变化）并非DIU关注的OCR幻觉、复杂布局解析或长文档推理等问题。虽然提出了动态分辨率策略和Zoom-in Chain机制，但这些方法针对的是地理空间目标的层级观察，难以直接迁移到文档的细粒度文本定位与逻辑结构理解。属于特定领域的平行应用，不具备对DIU的方法论启发。
Abstract With Transformers achieving outstanding performance on individual remote sensing (RS) tasks, we are now approaching the realization of a unified model that excels across multiple tasks through multi-task learning (MTL). Compared to single-task approaches, MTL methods offer improved generalization, enhanced scalability, and greater practical applicability. Recently, vision language models (VLMs) have achieved promising results in RS image understanding, grounding, and ultra-high-resolution (UHR) image reasoning, respectively. Moreover, the unified text-based interface demonstrates significant potential for MTL. Hence, in this work, we present RSCoVLM, a simple yet flexible VLM baseline for RS MTL. Firstly, we create the data curation engine, including data acquisition, offline processing and integrating, as well as online loading and weighting. This data engine effectively addresses complex RS data enviroment and generates flexible vision-language conversations. Furthermore, we propose a unified dynamic-resolution strategy to address the diverse image scales inherent in RS imagery. For UHR images, we introduce the Zoom-in Chain mechanism together with its corresponding dataset, LRS-VQA-Zoom. The strategies are flexible and effectively mitigate the computational burdens. Additionally, we significantly enhance the model's object detection capability and propose a novel evaluation protocol that ensures fair comparison between VLMs and conventional detection models. Extensive experiments demonstrate that RSCoVLM achieves state-of-the-art performance across diverse tasks, outperforming existing RS VLMs and even rivaling specialized expert models. All the training and evaluating tools, model weights, and datasets have been fully open-sourced to support reproducibility. We expect that this baseline will promote further progress toward general-purpose RS models.
Authors Qingyun Li, Shuran Ma, Junwei Luo, Yi Yu, Yue Zhou, Fengxiang Wang, Xudong Lu, Xiaoxing Wang, Xin He, Yushi Chen, Xue Yang, Junchi Yan
Comment 14 pages, 6 figures
Categories Computer Vision and Pattern Recognition
3
Shift-Equivariant Complex-Valued Convolutional Neural Networks WACV 2026
Title CN 具有平移等变性的复数值卷积神经网络
Keywords 复数值神经网络 · 平移等变性 · 多相采样 · SAR图像 · 卷积网络
Summary 本文扩展了可学习多相采样（LPS）到复数值神经网络，提出一种新的投影层以在复数域到实数域转换时保持平移等变性，并在极化SAR图像的分类、重建和语义分割任务中进行验证。重点在于信号处理层面的架构设计，而非通用视觉表征或文档理解。
Reason 该论文研究的是复数值CNN在极化SAR图像上的平移等变性问题，核心关注点为遥感图像中的重建与语义分割任务。虽然理论上平移等变性对文档布局分析有一定潜在价值，但其方法高度依赖复数表示和雷达图像特性，迁移到文档图像理解（如OCR、表格识别、VQA）的路径不清晰，且未涉及多模态大模型、视觉token压缩或文档特有的结构建模。属于特定领域的模型改进，与DIU核心痛点（如幻觉、复杂排版、长文档推理）无直接关联。
Abstract Convolutional neural networks have shown remarkable performance in recent years on various computer vision problems. However, the traditional convolutional neural network architecture lacks a critical property: shift equivariance and invariance, broken by downsampling and upsampling operations. Although data augmentation techniques can help the model learn the latter property empirically, a consistent and systematic way to achieve this goal is by designing downsampling and upsampling layers that theoretically guarantee these properties by construction. Adaptive Polyphase Sampling (APS) introduced the cornerstone for shift invariance, later extended to shift equivariance with Learnable Polyphase up/downsampling (LPS) applied to real-valued neural networks. In this paper, we extend the work on LPS to complex-valued neural networks both from a theoretical perspective and with a novel building block of a projection layer from $\mathbb{C}$ to $\mathbb{R}$ before the Gumbel Softmax. We finally evaluate this extension on several computer vision problems, specifically for either the invariance property in classification tasks or the equivariance property in both reconstruction and semantic segmentation problems, using polarimetric Synthetic Aperture Radar images.
Authors Quentin Gabot, Teck-Yian Lim, Jérémy Fix, Joana Frontera-Pons, Chengfang Ren, Jean-Philippe Ovarlez
Comment Accepted to WACV 2026
Categories Computer Vision and Pattern Recognition
3
Active Negative Loss: A Robust Framework for Learning with Noisy Labels
Title CN 主动负损失：一种用于含噪标签学习的鲁棒框架
Keywords 标签噪声 · 损失函数 · 鲁棒学习 · 图像分割 · 深度学习
Summary 本文提出了一种新的损失函数框架Active Negative Loss（ANL），通过引入归一化负损失函数（NNLFs）替代原有APL框架中的MAE损失，使模型更关注已被记住的干净样本，从而提升在标签噪声下的鲁棒性。同时在非对称噪声下加入基于熵的正则化以缓解类别不平衡问题。实验表明该方法在多种噪声类型和图像分割任务上表现优异。
Reason 该论文研究的是在图像分类或分割任务中应对标签噪声的损失函数设计，属于通用视觉鲁棒训练方法。虽然其提出的Active Negative Loss（ANL）在含噪环境下有效，但并未针对文档图像理解（DIU）中的核心痛点（如OCR幻觉、复杂排版、细粒度定位、高分辨率处理等）提出专门解决方案。方法本质是损失函数层面的优化，难以直接迁移用于缓解VLM在文档上的语义或空间幻觉问题，也不涉及多模态建模、视觉token压缩或多步推理机制。因此属于与DIU无关的通用CV方法。
Abstract Deep supervised learning has achieved remarkable success across a wide range of tasks, yet it remains susceptible to overfitting when confronted with noisy labels. To address this issue, noise-robust loss functions offer an effective solution for enhancing learning in the presence of label noise. In this work, we systematically investigate the limitation of the recently proposed Active Passive Loss (APL), which employs Mean Absolute Error (MAE) as its passive loss function. Despite the robustness brought by MAE, one of its key drawbacks is that it pays equal attention to clean and noisy samples; this feature slows down convergence and potentially makes training difficult, particularly in large-scale datasets. To overcome these challenges, we introduce a novel loss function class, termed Normalized Negative Loss Functions (NNLFs), which serve as passive loss functions within the APL framework. NNLFs effectively address the limitations of MAE by concentrating more on memorized clean samples. By replacing MAE in APL with our proposed NNLFs, we enhance APL and present a new framework called Active Negative Loss (ANL). Moreover, in non-symmetric noise scenarios, we propose an entropy-based regularization technique to mitigate the vulnerability to the label imbalance. Extensive experiments demonstrate that the new loss functions adopted by our ANL framework can achieve better or comparable performance to state-of-the-art methods across various label noise types and in image segmentation tasks. The source code is available at: https://github.com/Virusdoll/Active-Negative-Loss.
Authors Xichen Ye, Yifan Wu, Yiqi Wang, Xiaoqiang Li, Weizhong Zhang, Yifan Chen
Comment This work has been submitted to the IEEE for possible publication
Categories Computer Vision and Pattern Recognition
3
AnchorOPT: Towards Optimizing Dynamic Anchors for Adaptive Prompt Learning
Title CN AnchorOPT：面向动态锚点优化的自适应提示学习
Keywords 提示学习 · 动态锚点 · 位置优化 · CLIP · 跨任务泛化
Summary 本文提出AnchorOPT，一种基于动态锚点的提示学习框架，通过从任务数据中学习锚点表示并自适应优化锚点与软提示的位置关系，提升CLIP类模型的泛化能力。方法分为两阶段训练：先学习锚点，再固定锚点优化软提示和位置矩阵。实验显示其在多个数据集上能有效提升性能。
Reason 该论文属于通用视觉-语言模型（如CLIP）上的提示学习改进工作，提出动态调整锚点值和位置以提升跨任务泛化能力。虽然方法在理论上具有一定的灵活性，但其核心是标准的多模态表示学习中的提示工程优化，并未针对文档图像理解（DIU）特有的问题（如OCR幻觉、复杂版式、细粒度文本定位、长文档推理等）进行建模或验证。同时，该工作属于‘通用VLM提示调优’范畴，缺乏对文档结构、密集文本或视觉布局敏感的设计，难以直接迁移解决DIU的核心痛点。更关键的是，它本质上是一种可插拔模块（plug-and-play），符合‘平行应用级技术包装’特征，无明确的文档领域方法论创新。
Abstract Existing prompt learning methods, which are built upon CLIP models, leverage textual tokens as anchors to guide the learnable soft tokens. This guidance improves CLIP generalizations. However, these anchors-static in both value and position-lack cross-task and stage-adaptive flexibility. To address this limitation, we propose AnchorOPT, a dynamic anchor-based prompt learning framework. Specifically, AnchorOPT introduces dynamism in two key dimensions: (i) anchor values eschew handcrafted explicit textual tokens (e.g., "shape", "color"), instead learning dynamically from task-specific data; and (ii) the positional relationship between anchor and soft tokens is no longer fixed but adaptively optimized via a learnable position matrix conditioned on the training stage and task context. Training occurs in two stages: we first learn the anchor tokens, then freeze and transfer them to the second stage for optimization of soft tokens and the position matrix. Extensive experiments demonstrate that using only a simple learnable anchor and position matrix achieves performance comparable to or exceeding some methods incorporating additional learnable modules or regularization techniques. As a plug-and-play module, AnchorOPT integrates seamlessly into existing frameworks, yielding consistent performance gains across diverse datasets. Code is publicly available at https://github.com/zhengli97/ATPrompt.
Authors Zheng Li, Yibing Song, Xin Zhang, Lei Luo, Xiang Li, Jian Yang
Comment Technical Report
Categories Computer Vision and Pattern Recognition, Computation and Language
3
CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness NeurIPS 2025
Title CN CAPability：一个全面评估视觉描述正确性和完整性的基准
Keywords 视觉描述 · 评测基准 · 多模态大模型 · 正确性 · 完整性
Summary 本文提出了CAPability，一个涵盖12个维度、6个视角的视觉描述评测基准，包含近1.1万张人工标注的图像和视频，旨在全面评估多模态大模型在生成描述时的正确性与完整性。作者设计了precision和hit指标，并通过将标注转化为问答对引入K̄T指标，揭示模型‘知道但无法说出’的现象。该基准用于系统分析当前MLLMs在视觉描述上的优缺点。
Reason 该论文提出的是一个面向通用视觉描述（image/video captioning）的综合性评测基准，虽然涉及多模态大模型（MLLMs），但其任务设定、数据构建和评估维度均聚焦于开放域图像/视频描述生成，与文档图像理解（DIU）的核心任务如DocVQA、表格识别、布局分析等无直接关联。尽管其提出的‘know but cannot tell’现象可能间接反映模型的信息提取不全问题，但该工作本身并未针对OCR幻觉、密集文本处理、复杂排版或长文档推理等DIU关键痛点提供可迁移的方法论或架构改进，属于对现有技术的评测而非源头创新。且其应用场景明显偏向通用视觉理解，非文档专属。
Abstract Visual captioning benchmarks have become outdated with the emergence of modern multimodal large language models (MLLMs), as the brief ground-truth sentences and traditional metrics fail to assess detailed captions effectively. While recent benchmarks attempt to address this by focusing on keyword extraction or object-centric evaluation, they remain limited to vague-view or object-view analyses and incomplete visual element coverage. In this paper, we introduce CAPability, a comprehensive multi-view benchmark for evaluating visual captioning across 12 dimensions spanning six critical views. We curate nearly 11K human-annotated images and videos with visual element annotations to evaluate the generated captions. CAPability stably assesses both the correctness and thoroughness of captions with \textit{precision} and \textit{hit} metrics. By converting annotations to QA pairs, we further introduce a heuristic metric, \textit{know but cannot tell} ($K\bar{T}$), indicating a significant performance gap between QA and caption capabilities. Our work provides a holistic analysis of MLLMs' captioning abilities, as we identify their strengths and weaknesses across various dimensions, guiding future research to enhance specific aspects of their capabilities.
Authors Zhihang Liu, Chen-Wei Xie, Bin Wen, Feiwu Yu, Jixuan Chen, Pandeng Li, Boqiang Zhang, Nianzu Yang, Yinglu Li, Zuan Gao, Yun Zheng, Hongtao Xie
Comment Accepted to NeurIPS 2025
Categories Computer Vision and Pattern Recognition, Computation and Language
3
FlowTok: Flowing Seamlessly Across Text and Image Tokens
Title CN FlowTok：在文本与图像Token之间无缝流动
Keywords 跨模态生成 · 流形匹配 · 1D Token压缩 · 潜在空间对齐 · 高效生成
Summary FlowTok提出一种通过流形匹配在文本与图像模态间直接转换的方法，通过将图像压缩为1D token序列，实现在共享潜在空间中的模态流动。该方法简化了生成流程，提升了效率和速度，适用于文本到图像及图像到文本的双向生成任务，但主要面向通用跨模态生成场景。
Reason 该论文属于跨模态生成领域，核心目标是实现文本与图像之间的直接流形匹配（flow matching），其技术路线聚焦于生成模型的范式简化，而非文档理解中的OCR去幻觉、细粒度定位或复杂排版解析。尽管提出了将图像编码为紧凑的1D token序列以对齐文本表示，但其动机和实验场景均围绕通用图像生成展开，未涉及文档结构、布局语义或视觉-语言联合推理等DIU核心挑战。该方法难以直接迁移用于解决文档图像中的高精度识别、逻辑推理或长上下文建模问题，且无证据表明其对OCR-free DocVQA或表格理解有增益。因此属于无关领域的平行应用，不构成对DIU的方法论推动。
Abstract Bridging different modalities lies at the heart of cross-modality generation. While conventional approaches treat the text modality as a conditioning signal that gradually guides the denoising process from Gaussian noise to the target image modality, we explore a much simpler paradigm-directly evolving between text and image modalities through flow matching. This requires projecting both modalities into a shared latent space, which poses a significant challenge due to their inherently different representations: text is highly semantic and encoded as 1D tokens, whereas images are spatially redundant and represented as 2D latent embeddings. To address this, we introduce FlowTok, a minimal framework that seamlessly flows across text and images by encoding images into a compact 1D token representation. Compared to prior methods, this design reduces the latent space size by 3.3x at an image resolution of 256, eliminating the need for complex conditioning mechanisms or noise scheduling. Moreover, FlowTok naturally extends to image-to-text generation under the same formulation. With its streamlined architecture centered around compact 1D tokens, FlowTok is highly memory-efficient, requires significantly fewer training resources, and achieves much faster sampling speeds-all while delivering performance comparable to state-of-the-art models. Code is available at https://github.com/TACJu/FlowTok.
Authors Ju He, Qihang Yu, Qihao Liu, Liang-Chieh Chen
Comment Project page at https://tacju.github.io/projects/flowtok.html
Categories Computer Vision and Pattern Recognition
3
Saliency-R1: Incentivizing Unified Saliency Reasoning Capability in MLLM with Confidence-Guided Reinforcement Learning
Title CN Saliency-R1：通过置信度引导的强化学习在多模态大语言模型中激励统一的显著性推理能力
Keywords 显著性检测 · 多模态大模型 · 强化学习 · 视觉推理 · 实例分割
Summary 本文提出Saliency-R1，首个统一处理多种显著性任务（SOD、SIS、CoSOD）的多模态大模型框架，采用结构化文本标签和新提出的置信度引导策略优化（CGPO）进行训练，在显著性推理任务上表现优异。但其研究目标偏离文档图像理解核心问题。
Reason 该论文聚焦于通用视觉显著性检测（如显著物体检测、实例分割和共显著对象检测），属于通用图像理解任务，而非文档图像理解（DIU）的核心任务（如DocVQA、表格识别、版面分析等）。虽然显著性建模理论上可能辅助文档中的关键区域定位，但其任务设定与文档特有的结构化信息提取、OCR对齐、复杂排版解析等痛点脱节。方法上虽提出CGPO强化学习机制，但未体现针对文档高分辨率、密集文本或逻辑结构的适配设计，迁移价值有限。
Abstract Although multimodal large language models (MLLMs) excel in high-level vision-language reasoning, they lack inherent awareness of visual saliency, making it difficult to identify key visual elements. To bridge this gap, we propose Saliency-R1, the first unified MLLM framework that jointly tackles three representative and heterogeneous saliency tasks: Salient Object Detection (SOD), Salient Instance Segmentation (SIS), and Co-salient Object Detection (CoSOD), enhancing the model's capacity for saliency reasoning. We introduce a textual interface with structured tags (<rg>, <ins>) to encode region- and instance-level referring expressions, enabling a single referring segmenter to produce task-appropriate masks. To train the MLLM efficiently, we propose Confidence-Guided Policy Optimization (CGPO), a novel single-sample reinforcement learning algorithm. CGPO improves on GRPO by replacing group-normalized advantages with a per-sample signal based on reward-confidence discrepancy, thereby reducing computational waste, mitigating signal dilution, and lowering training overhead. Our model exceeds or matches the performance of robust open/closed-source MLLMs and specialized state-of-the-art methods across all three tasks, demonstrating the efficacy of our framework in saliency reasoning.
Authors Long Li, Shuichen Ji, Ziyang Luo, Zhihui Li, Dingwen Zhang, Junwei Han, Nian Liu
Comment Main text (excluding references): 8 pages, 4 figures; Supplementary Materials (excluding references): 9 pages, 10 figures
Categories Computer Vision and Pattern Recognition
3
One-Step Diffusion Transformer for Controllable Real-World Image Super-Resolution
Title CN 用于可控真实世界图像超分辨率的一次性扩散Transformer
Keywords 图像超分辨率 · 扩散模型 · 文本图像增强 · 可控生成 · 视觉恢复
Summary 本文提出ODTSR，一种基于Qwen-Image的一次性扩散Transformer模型，通过噪声混合视觉流（NVS）设计和保真度感知对抗训练（FAA），在真实世界图像超分辨率任务中同时实现高保真和可控性。模型可在无需特定数据微调的情况下支持提示控制的中文字符图像超分，但在文档理解任务中缺乏对布局分析、表格识别或VIE等关键问题的支持。
Reason 该论文聚焦于通用图像和场景文本图像的超分辨率任务，虽然提到了中文字符场景下的应用，但其核心是图像生成与增强领域的扩散模型架构改进，属于真实世界图像超分（Real-ISR）方向。尽管其在STISR（场景文本图像超分辨率）上做了实验，但方法本身并未针对文档理解中的OCR幻觉、复杂版式建模或长文档推理等DIU核心痛点进行设计，且主要贡献在于图像重建质量与控制性的平衡，属于图像处理层面的技术，难以直接迁移至文档理解的语义解析或结构建模任务。因此属于边缘相关但无直接迁移价值的工作。
Abstract Recent advances in diffusion-based real-world image super-resolution (Real-ISR) have demonstrated remarkable perceptual quality, yet the balance between fidelity and controllability remains a problem: multi-step diffusion-based methods suffer from generative diversity and randomness, resulting in low fidelity, while one-step methods lose control flexibility due to fidelity-specific finetuning. In this paper, we present ODTSR, a one-step diffusion transformer based on Qwen-Image that performs Real-ISR considering fidelity and controllability simultaneously: a newly introduced visual stream receives low-quality images (LQ) with adjustable noise (Control Noise), and the original visual stream receives LQs with consistent noise (Prior Noise), forming the Noise-hybrid Visual Stream (NVS) design. ODTSR further employs Fidelity-aware Adversarial Training (FAA) to enhance controllability and achieve one-step inference. Extensive experiments demonstrate that ODTSR not only achieves state-of-the-art (SOTA) performance on generic Real-ISR, but also enables prompt controllability on challenging scenarios such as real-world scene text image super-resolution (STISR) of Chinese characters without training on specific datasets. Codes are available at $\href{https://github.com/RedMediaTech/ODTSR}{\text{this url}}$.
Authors Yushun Fang, Yuxiang Chen, Shibo Yin, Qiang Hu, Jiangchao Yao, Ya Zhang, Xiaoyun Zhang, Yanfeng Wang
Categories Computer Vision and Pattern Recognition
3
EvoEmpirBench: Dynamic Spatial Reasoning with Agent-ExpVer AAAI 2026
Title CN EvoEmpirBench：基于智能体经验验证的动态空间推理
Keywords 动态空间推理 · 智能体 · 记忆机制 · 基准测试 · 环境反馈
Summary 本文提出了两个动态空间推理基准：局部可观察的迷宫导航和match-2消除任务，用于评估模型在部分观测、环境动态变化条件下的空间理解与自适应规划能力。每个动作会引发环境结构变化，要求模型持续更新认知策略。作者还设计了一种基于主观体验的记忆机制以实现跨任务经验迁移。实验表明主流模型在此类任务上存在显著局限。代码和数据已公开。
Reason 该论文聚焦于动态环境下的空间推理与智能体导航任务，属于具身智能（Embodied AI）领域，主要应用场景为迷宫导航和游戏式消除任务。尽管提出了记忆机制和动态认知更新方法，但其任务设定与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版、细粒度文本定位、长文档结构建模等无直接关联。所提出的Agent架构和经验验证机制难以迁移到文档理解场景中，且未涉及多模态语言模型在视觉-语言对齐或高分辨率文档处理方面的创新。属于无关领域工作。
Abstract Most existing spatial reasoning benchmarks focus on static or globally observable environments, failing to capture the challenges of long-horizon reasoning and memory utilization under partial observability and dynamic changes. We introduce two dynamic spatial benchmarks, locally observable maze navigation and match-2 elimination that systematically evaluate models' abilities in spatial understanding and adaptive planning when local perception, environment feedback, and global objectives are tightly coupled. Each action triggers structural changes in the environment, requiring continuous update of cognition and strategy. We further propose a subjective experience-based memory mechanism for cross-task experience transfer and validation. Experiments show that our benchmarks reveal key limitations of mainstream models in dynamic spatial reasoning and long-term memory, providing a comprehensive platform for future methodological advances. Our code and data are available at https://anonymous.4open.science/r/EvoEmpirBench-143C/.
Authors Pukun Zhao, Longxiang Wang, Miaowei Wang, Chen Chen, Fanqing Zhou, Haojian Huang
Comment Accepted by AAAI 2026, 29 pages, 3 figures, 7 tables
Categories Computer Vision and Pattern Recognition
3
Deep Parameter Interpolation for Scalar Conditioning
Title CN 用于标量条件化的深度参数插值
Keywords 深度参数插值 · 标量条件化 · 扩散模型 · 流匹配 · 神经网络架构
Summary 本文提出了深度参数插值（DPI）方法，通过在网络内部维护两组可学习参数并根据标量值动态插值得以实现标量条件化。该方法简单且与架构无关，在扩散模型和流匹配任务中表现出优于传统方法的样本生成质量和去噪性能，同时保持相近的计算效率。
Reason 该论文提出了一种使神经网络能够接受标量输入的通用方法（DPI），通过在两个可学习参数集之间进行插值来引入标量依赖性。尽管其在扩散模型和流匹配中提升了去噪与生成质量，但该工作属于通用架构改进，且聚焦于图像生成任务中的时间/噪声步长建模，并未涉及多模态对齐、视觉-语言推理或文档结构理解等DIU核心问题。其技术虽具一定通用性，但迁移到文档图像理解场景（如处理复杂排版、细粒度文本定位或抑制OCR幻觉）的路径不明确，缺乏直接关联性。
Abstract We propose deep parameter interpolation (DPI), a general-purpose method for transforming an existing deep neural network architecture into one that accepts an additional scalar input. Recent deep generative models, including diffusion models and flow matching, employ a single neural network to learn a time- or noise level-dependent vector field. Designing a network architecture to accurately represent this vector field is challenging because the network must integrate information from two different sources: a high-dimensional vector (usually an image) and a scalar. Common approaches either encode the scalar as an additional image input or combine scalar and vector information in specific network components, which restricts architecture choices. Instead, we propose to maintain two learnable parameter sets within a single network and to introduce the scalar dependency by dynamically interpolating between the parameter sets based on the scalar value during training and sampling. DPI is a simple, architecture-agnostic method for adding scalar dependence to a neural network. We demonstrate that our method improves denoising performance and enhances sample quality for both diffusion and flow matching models, while achieving computational efficiency comparable to standard scalar conditioning techniques. Code is available at https://github.com/wustl-cig/parameter_interpolation.
Authors Chicago Y. Park, Michael T. McCann, Cristina Garcia-Cardona, Brendt Wohlberg, Ulugbek S. Kamilov
Categories Computer Vision and Pattern Recognition
3
CaptionQA: Is Your Caption as Useful as the Image Itself?
Title CN CaptionQA：你的图注是否和它所代表的图像本身一样有用？
Keywords 图像描述评估 · 多模态基准 · 下游任务效用 · 图注质量 · VLM评测
Summary 本文提出了CaptionQA，一个基于实用性的图像描述（caption）评估基准，通过让LLM仅依赖图注回答多选题，来评估图注在自然图像、文档、电商和具身AI四个领域中替代图像完成任务的能力。数据集包含超过3.3万张图像上的密集标注问题（平均每图50.3题），揭示了当前MLLM在传统VQA表现相近的情况下，在图注实用性上存在显著差距。作者同时开源了可扩展的构建管线。
Reason 该论文提出了一种基于任务效用的图注评估基准CaptionQA，旨在衡量生成的图注在下游任务中替代图像的能力。虽然其覆盖了‘文档’领域作为应用场景之一，但其核心贡献是面向通用多模态系统（如推荐、代理推理链）的图注质量评估框架，并未针对文档图像理解中的具体技术痛点（如OCR幻觉、复杂版式建模、细粒度文本定位等）提出方法改进。本质上属于将现有VLM+Captioning技术应用于新评测场景的‘平行应用’，缺乏对DIU领域的方法论创新。
Abstract Image captions serve as efficient surrogates for visual content in multimodal systems such as retrieval, recommendation, and multi-step agentic inference pipelines. Yet current evaluation practices miss a fundamental question: Can captions stand-in for images in real downstream tasks? We propose a utility-based benchmark, CaptionQA, to evaluate model-generated captions, where caption quality is measured by how well it supports downstream tasks. CaptionQA is an extensible domain-dependent benchmark covering 4 domains--Natural, Document, E-commerce, and Embodied AI--each with fine-grained taxonomies (25 top-level and 69 subcategories) that identify useful information for domain-specific tasks. CaptionQA builds 33,027 densely annotated multiple-choice questions (50.3 per image on average) that explicitly require visual information to answer, providing a comprehensive probe of caption utility. In our evaluation protocol, an LLM answers these questions using captions alone, directly measuring whether captions preserve image-level utility and are utilizable by a downstream LLM. Evaluating state-of-the-art MLLMs reveals substantial gaps between the image and its caption utility. Notably, models nearly identical on traditional image-QA benchmarks lower by up to 32% in caption utility. We release CaptionQA along with an open-source pipeline for extension to new domains. The code is available at https://github.com/bronyayang/CaptionQA.
Authors Shijia Yang, Yunong Liu, Bohan Zhai, Ximeng Sun, Zicheng Liu, Emad Barsoum, Manling Li, Chenfeng Xu
Categories Computer Vision and Pattern Recognition
3
RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness NeurIPS 2025 (Spotlight)
Title CN RobustMerge：具有方向鲁棒性的MLLM参数高效模型合并方法
Keywords 模型合并 · 参数高效微调 · 方向鲁棒性 · 低秩分解 · 多任务学习
Summary 本文提出RobustMerge，一种无需训练的参数高效模型合并方法，旨在解决通过低秩适配（如LoRA）微调得到的专家模型在合并时因方向不稳定导致性能下降的问题。作者从低秩分解角度分析发现，奇异值差异过大会影响合并后方向鲁棒性，因此引入参数剪枝与系数缩放策略来稳定方向，并通过跨任务归一化提升对未见任务的泛化能力。实验在一个包含多种多模态任务的基准上进行，结果表明该方法优于现有合并策略。
Reason 该论文研究的是多任务专家模型的参数高效合并问题，属于模型集成与参数高效微调的交叉方向。尽管其方法在多模态大模型（MLLM）场景下验证，但核心贡献集中在模型合并机制本身，并未针对文档图像理解中的具体痛点（如OCR幻觉、复杂版式建模、细粒度定位或长文档推理）进行设计或验证。同时，该工作属于‘将已有技术应用于模型合并’的范畴，且未提出可迁移至DIU领域的新型架构、推理机制或视觉表示学习方法，属于平行应用类研究。
Abstract Fine-tuning pre-trained models with custom data leads to numerous expert models on specific tasks. Merging models into one universal model to empower multi-task ability refraining from data leakage has gained popularity. With the expansion in data and model size, parameter-efficient tuning becomes the common practice for obtaining task-specific models efficiently. However, few methods are dedicated to efficient merging, and existing methods designed for full fine-tuning merging fail under efficient merging. To address the issue, we analyze from low-rank decomposition and reveal that direction robustness during merging is crucial for merging efficient modules. We furthermore uncover that compensating for the gap between stark singular values contributes to direction robustness. Therefore, we propose RobustMerge, a training-free parameter-efficient merging method with complementary parameter adaptation to maintain direction robustness. Specifically, we (1) prune parameters and scale coefficients from inter-parameter relation for singular values to maintain direction stability away from task interference, and (2) perform cross-task normalization to enhance unseen task generalization. We establish a benchmark consisting of diverse multimodal tasks, on which we conduct experiments to certify the outstanding performance and generalizability of our method. Additional studies and extensive analyses further showcase the effectiveness. Code is available at https://github.com/AuroraZengfh/RobustMerge.
Authors Fanhu Zeng, Haiyang Guo, Fei Zhu, Li Shen, Hao Tang
Comment NeurIPS 2025 (Spotlight) Fix some typos
Categories Computer Vision and Pattern Recognition
3
ReasonAct: Progressive Training for Fine-Grained Video Reasoning in Small Models
Title CN ReasonAct：面向小模型细粒度视频推理的渐进式训练
Keywords 视频推理 · 渐进式训练 · 强化学习 · 小模型 · 时序建模
Summary 本文提出 ReasonAct，一种通过三阶段渐进训练提升小模型在视频中进行细粒度时序推理能力的方法，包括文本推理预热、视频微调和引入时间感知的强化学习优化。实验表明该方法显著提升了小模型在多个视频动作识别数据集上的性能。
Reason 该论文聚焦于视频理解中的时序推理任务，属于视频分析领域，其方法设计针对动作识别和时间一致性建模，与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂版面解析、文本密集识别等无直接关联。尽管其渐进式训练和强化学习奖励机制在理论上具有通用性，但迁移到文档理解场景的路径不明确，且未涉及多模态对齐、高分辨率处理或细粒度视觉-语言 grounding 等 DIU 关键挑战。
Abstract While recent multimodal models have shown progress in vision-language tasks, small-scale variants still struggle with the fine-grained temporal reasoning required for video understanding. We introduce ReasonAct, a method that enhances video reasoning in smaller models through a three-stage training process: first building a foundation with text-only reasoning, then fine-tuning on video, and finally refining with temporal-aware reinforcement learning. We build upon Temporal Group Relative Policy Optimization (T-GRPO) by incorporating temporal consistency modeling into policy optimization. We also propose a biomechanically-motivated sub-action decomposition mechanism that provides graduated rewards for constituent action phases. Through experiments on HMDB51, UCF-101, and Kinetics-400, our 3B-parameter model achieves 67.2%, 94.1%, and 78.9% accuracy respectively, demonstrating improvements of 17.9, 15.8, and 12.3 points over baselines. Ablation studies validate that our progressive training enables smaller models to achieve competitive video reasoning performance while maintaining computational efficiency.
Authors Jiaxin Liu, Zhaolu Kang
Categories Computer Vision and Pattern Recognition
3
Auxiliary Metrics Help Decoding Skill Neurons in the Wild
Title CN 辅助指标有助于在真实场景中解码技能神经元
Keywords 大语言模型 · 可解释性 · 技能神经元 · 辅助指标 · 神经元激活
Summary 本文提出一种轻量级方法，利用辅助指标（如外部标签和模型自信度）来识别大语言模型中与特定技能相关的神经元，无需手动聚合token即可发现可解释的行为模式。实验验证了该方法在开放生成和自然语言推理任务上的有效性，并揭示了算术推理中的隐式捷径。
Reason 该论文研究大语言模型内部机制，提出通过辅助指标（如外部标签和置信度）来识别编码特定能力的神经元。虽然方法对理解LLM有潜在价值，但其应用场景为纯文本生成与自然语言推理，未涉及任何视觉或文档模态，与文档图像理解（DIU）的核心问题（如OCR幻觉、复杂版面分析、细粒度定位等）无直接关联。属于纯NLP可解释性研究，无法迁移至多模态文档理解任务。
Abstract Large language models (LLMs) exhibit remarkable capabilities across a wide range of tasks, yet their internal mechanisms remain largely opaque. In this paper, we introduce a simple, lightweight, and broadly applicable method with a focus on isolating neurons that encode specific skills. Building upon prior work that identified "skill neurons" via soft prompt training on classification tasks, our approach extends the analysis to complex scenarios involving multiple skills. We correlate neuron activations with auxiliary metrics -- such as external labels and the model's own confidence score -- thereby uncovering interpretable and task-specific behaviors without the need for manual token aggregation. We empirically validate our method on tasks spanning open-ended text generation and natural language inference, demonstrating its ability to detect neurons that not only drive known skills but also reveal previously unidentified shortcuts in arithmetic reasoning on BigBench.
Authors Yixiu Zhao, Xiaozhi Wang, Zijun Yao, Lei Hou, Juanzi Li
Comment 7 pages, 7 figures. Includes additional appendix
Categories Computation and Language
3
RoParQ: Paraphrase-Aware Alignment of Large Language Models Towards Robustness to Paraphrased Questions
Title CN RoParQ：面向改写问题鲁棒性的大语言模型改写感知对齐
Keywords 大语言模型 · 改写鲁棒性 · 语义一致性 · 监督微调 · 评测基准
Summary 本文提出RoParQ基准和XParaCon指标，用于评估大模型在面对语义相同但表述不同的问题时的回答一致性，并通过改写感知的监督微调提升模型鲁棒性。实验表明小模型经训练后可达到大模型级别的稳定性。
Reason 该论文聚焦于纯文本问答中对改写问题的一致性，属于通用NLP领域的问题，未涉及任何视觉或文档模态。其核心贡献——跨改写一致性评估（RoParQ基准）和基于推理的微调策略——虽然在语义鲁棒性上有意义，但完全脱离多模态上下文，无法迁移解决文档图像理解中的OCR幻觉、细粒度定位或复杂排版等核心痛点。属于典型的‘平行应用’类研究（仅将LLM应用于改写变体），且无视觉成分，与DIU无关。
Abstract Large Language Models (LLMs) often exhibit inconsistent behavior when answering paraphrased questions, suggesting a reliance on surface-level patterns rather than true semantic understanding. To address this limitation, we introduce RoParQ, a benchmark specifically constructed to evaluate cross-paraphrase consistency in closed-book multiple-choice QA. This benchmark is derived from standard datasets by generating paraphrases via proprietary models and selectively retaining examples that elicit inconsistent confidence from a judge model. We further propose XParaCon, a novel evaluation metric that quantifies a model's robustness by measuring the standard deviation of accuracies across question variants. Additionally, we implement a reasoning-based, paraphrase-aware Supervised Fine-Tuning (SFT) strategy designed to align models toward semantic invariance. Our experiments demonstrate that this targeted alignment significantly enhances robustness. Notably, fine-tuned lightweight models achieved consistency levels comparable to much larger pre-trained models. These results highlight the efficacy of our approach in mitigating superficial memorization and fostering more robust, reliable LLMs.
Authors Minjoon Choi
Comment 12 pages, 9 figures, 8 tables
Categories Computation and Language
3
A Systematic Study of Model Merging Techniques in Large Language Models
Title CN 大语言模型中模型融合技术的系统性研究
Keywords 模型融合 · 大语言模型 · Task Arithmetic · 参数合并 · 无训练集成
Summary 本文系统评估了六种先进的模型融合方法在四个开源大语言模型上的表现，发现只有最简单的Task Arithmetic方法能稳定提升性能，其他复杂方法反而导致性能下降。结果表明现有融合技术难以直接迁移到现代LLM，需设计专用于LLM的融合算法。
Reason 该论文研究的是大语言模型之间的模型融合方法，属于纯NLP领域的方法论探索，未涉及任何视觉模态或文档图像理解任务。尽管模型融合可能在理论上对多任务DIU系统有间接帮助，但其方法无法直接迁移解决OCR幻觉、细粒度定位、高分辨率文档处理等核心DIU痛点，且不针对多模态场景设计。属于与文档图像理解无关的平行方法研究。
Abstract Model merging combines multiple fine-tuned checkpoints into a single model without additional training, offering an attractive approach to reusing models and efficiently improving performance. However, it remains unclear whether the advantages reported for smaller models and classifiers generalize to LLMs. We present a large-scale, systematic evaluation of six state-of-the-art merging methods, including recent subspace methods, across four open-weight LLMs, twelve fine-tuned checkpoints per base model, and sixteen standard LLM benchmarks. Evaluating through standardized benchmarks, we measure both the probability that a merged model outperforms the base model and relative gains over the best individual checkpoint. Our results show that the oldest and simplest method, Task Arithmetic, is the only approach that reliably yields performance gains on LLMs. Other interference-aware and subspace merging methods typically result in significant performance drops. Our findings indicate that current merging techniques do not directly transfer to modern LLMs. This motivates the design of LLM-specific merging algorithms and merging-aware fine-tuning methods. Code will be released upon acceptance of this paper.
Authors Oğuz Kağan Hitit, Leander Girrbach, Zeynep Akata
Categories Computation and Language
3
Odin: Oriented Dual-module Integration for Text-rich Network Representation Learning
Title CN Odin：面向文本丰富网络表示学习的定向双模块集成
Keywords 图表示学习 · 文本图 · Transformer · GNN · 结构融合
Summary 本文提出Odin模型，通过在Transformer中引入定向双模块机制来融合图结构信息，实现文本丰富图的表示学习。该方法避免了GNN的过平滑问题，并在语义层次上对齐结构抽象。同时提出轻量版Light Odin以提升效率。实验表明其在多个文本图基准上达到SOTA。
Reason 该论文研究的是文本属性图（text-attributed graphs）上的表示学习，核心问题是图结构与文本语义的融合，属于图神经网络与自然语言处理的交叉领域。尽管其提出的Odin架构在结构-文本融合方面有一定创新，但其任务设定、数据模态和应用场景均远离文档图像理解（DIU）的核心问题，如OCR、版面分析、视觉-语言对齐、细粒度定位或复杂排版建模。文中未涉及图像输入、多模态建模或视觉表示学习，无法迁移用于解决文档图像中的幻觉、高分辨率处理或长文档推理等关键挑战。
Abstract Text-attributed graphs require models to effectively combine strong textual understanding with structurally informed reasoning. Existing approaches either rely on GNNs--limited by over-smoothing and hop-dependent diffusion--or employ Transformers that overlook graph topology and treat nodes as isolated sequences. We propose Odin (Oriented Dual-module INtegration), a new architecture that injects graph structure into Transformers at selected depths through an oriented dual-module mechanism.Unlike message-passing GNNs, Odin does not rely on multi-hop diffusion; instead, multi-hop structures are integrated at specific Transformer layers, yielding low-, mid-, and high-level structural abstraction aligned with the model's semantic hierarchy. Because aggregation operates on the global [CLS] representation, Odin fundamentally avoids over-smoothing and decouples structural abstraction from neighborhood size or graph topology. We further establish that Odin's expressive power strictly contains that of both pure Transformers and GNNs.To make the design efficient in large-scale or low-resource settings, we introduce Light Odin, a lightweight variant that preserves the same layer-aligned structural abstraction for faster training and inference. Experiments on multiple text-rich graph benchmarks show that Odin achieves state-of-the-art accuracy, while Light Odin delivers competitive performance with significantly reduced computational cost. Together, Odin and Light Odin form a unified, hop-free framework for principled structure-text integration. The source code of this model has been released at https://github.com/hongkaifeng/Odin.
Authors Kaifeng Hong, Yinglong Zhang, Xiaoying Hong, Xuewen Xia, Xing Xu
Comment 32 pages, 2 figures
Categories Computation and Language
3
Text-to-SQL as Dual-State Reasoning: Integrating Adaptive Context and Progressive Generation
Title CN 将Text-to-SQL视为双状态推理：集成自适应上下文与渐进生成
Keywords Text-to-SQL · 双状态推理 · 链式思维 · 语义解析 · 大语言模型
Summary 本文提出DSR-SQL，一种双状态推理框架，通过构建自适应上下文状态和渐进生成状态来提升Text-to-SQL的准确性。前者压缩并选择相关数据库结构，后者通过反馈机制逐步生成可修正的SQL语句。在Spider和BIRD数据集上取得良好效果，无需后训练或上下文示例。
Reason 该论文研究的是Text-to-SQL任务中的推理建模，属于典型的NLP下游应用（数据库语义解析），尽管提出了双状态推理框架，但其核心场景和问题定义均远离文档图像理解（DIU）的核心痛点，如OCR幻觉、复杂版式解析、视觉-文本对齐等。虽然‘自适应上下文’和‘反馈引导的状态转移’在理论上可能启发长文档多步推理的设计，但文中未涉及任何视觉模态或文档结构信息处理，迁移路径不直接且缺乏适用基础。属于平行任务应用，不符合本领域方法论创新要求。
Abstract Recent divide-and-conquer reasoning approaches, particularly those based on Chain-of-Thought (CoT), have substantially improved the Text-to-SQL capabilities of Large Language Models (LLMs). However, when applied to complex enterprise databases, such methods struggle to maintain coherent reasoning due to limited context capacity, unreliable schema linking, and weak grounding in database semantics. To overcome these issues, we introduce DSR-SQL, a \textbf{D}ual-\textbf{S}tate \textbf{R}easoning framework that models Text-to-SQL as an interaction between an adaptive context state and a progressive generation state. The first constructs a compact, semantically faithful environment by refining large schemas and selecting relevant structures, while the second formalizes SQL synthesis as feedback-guided state transitions, enabling the model to self-correct and align with user intent. Without any post-training or in-context examples, DSR-SQL achieves competitive performance, reaching 35.28\% execution accuracy on Spider 2.0-Snow and 68.32\% on BIRD development set. Our implementation will be open-sourced at: https://github.com/DMIRLAB-Group/DSR-SQL.
Authors Zhifeng Hao, Qibin Song, Ruichu Cai, Boyan Xu
Categories Computation and Language
3
AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser
Title CN AICC：解析HTML更精细，让模型更强大——基于模型的HTML解析器构建的7.3万亿Token AI就绪语料库
Keywords HTML解析 · 网页语料库 · 数据预处理 · 序列标注 · 语言模型预训练
Summary 本文提出MinerU-HTML，一种基于0.6B参数语言模型的HTML内容提取方法，将提取任务建模为序列标注问题，相比传统启发式工具（如Trafilatura）能更好保留代码块、公式和表格等结构化元素。基于此构建了7.3万亿token的多语言语料库AICC，在相同过滤条件下预训练出的模型性能优于基线。作者开源了基准MainWebBench、解析器和语料库。
Reason 该论文核心贡献是改进网页到文本的提取流程（从HTML生成高质量Markdown），提升预训练语料质量。虽然其提出的模型驱动HTML解析方法在理论上可能间接有利于文档理解任务的数据构造，但整体工作属于‘平行应用’范畴：它并未直接解决文档图像理解（DIU）中的核心问题（如OCR幻觉、复杂排版、视觉-语言对齐、细粒度定位等），而是面向纯文本Web数据清洗与预处理。其技术路线无法直接迁移用于处理扫描文档、表格识别或图文混合推理任务，且未涉及任何视觉模态或VLM架构创新。
Abstract While web data quality is crucial for large language models, most curation efforts focus on filtering and deduplication,treating HTML-to-text extraction as a fixed pre-processing step. Existing web corpora rely on heuristic-based extractors like Trafilatura, which struggle to preserve document structure and frequently corrupt structured elements such as formulas, codes, and tables. We hypothesize that improving extraction quality can be as impactful as aggressive filtering strategies for downstream performance. We introduce MinerU-HTML, a novel extraction pipeline that reformulates content extraction as a sequence labeling problem solved by a 0.6B-parameter language model. Unlike text-density heuristics, MinerU-HTML leverages semantic understanding and employs a two-stage formatting pipeline that explicitly categorizes semantic elements before converting to Markdown. Crucially, its model-based approach is inherently scalable, whereas heuristic methods offer limited improvement pathways. On MainWebBench, our benchmark of 7,887 annotated web pages, MinerU-HTML achieves 81.8\% ROUGE-N F1 compared to Trafilatura's 63.6\%, with exceptional structured element preservation (90.9\% for code blocks, 94.0\% for formulas). Using MinerU-HTML, we construct AICC (AI-ready Common Crawl), a 7.3-trillion token multilingual corpus from two Common Crawl snapshots. In controlled pretraining experiments where AICC and Trafilatura-extracted TfCC undergo identical filtering, models trained on AICC (62B tokens) achieve 50.8\% average accuracy across 13 benchmarks, outperforming TfCC by 1.08pp-providing direct evidence that extraction quality significantly impacts model capabilities. AICC also surpasses RefinedWeb and FineWeb on key benchmarks. We publicly release MainWebBench, MinerU-HTML, and AICC, demonstrating that HTML extraction is a critical, often underestimated component of web corpus construction.
Authors Ren Ma, Jiantao Qiu, Chao Xu, Pei Chu, Kaiwen Liu, Pengli Ren, Yuan Qu, Jiahui Peng, Linfeng Hou, Mengjie Liu, Lindong Lu, Wenchang Ning, Jia Yu, Rui Min, Jin Shi, Haojiong Chen, Peng Zhang, Wenjian Zhang, Qian Jiang, Zengjie Hu, Guoqiang Yang, Zhenxiang Li, Fukai Shang, Runyuan Ma, Chenlin Su, Zhongying Tu, Wentao Zhang, Dahua Lin, Conghui He
Categories Computation and Language
3
PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark
Title CN PEFT-Bench：一种参数高效微调方法的基准测试
Keywords 参数高效微调 · PEFT · 大语言模型 · 基准测试 · NLP
Summary 本文提出了一个名为PEFT-Bench的统一基准，用于评估自回归大语言模型中多种参数高效微调方法的表现。通过在27个NLP数据集上测试6种主流PEFT方法，并引入考虑可训练参数量、推理速度和训练内存消耗的PSCP评分机制，提升评估全面性与可复现性。
Reason 该论文属于NLP领域的通用参数高效微调（PEFT）方法评估工作，虽然对大模型微调具有工程和实践参考价值，但其研究内容为通用语言模型的微调策略比较，并未涉及多模态、视觉输入或文档结构理解。无法直接解决文档图像理解中的OCR幻觉、复杂排版、高分辨率处理等核心问题。属于典型的‘平行应用’类技术整合，而非针对DIU的方法论创新。
Abstract Despite the state-of-the-art performance of Large Language Models (LLMs) achieved on many tasks, their massive scale often leads to high computational and environmental costs, limiting their accessibility. Parameter-efficient fine-tuning (PEFT) methods address this challenge by reducing the number of trainable parameters while maintaining strong downstream performance. Despite the increased development in PEFT methods, current evaluations remain limited (in terms of evaluated models and datasets) and difficult to reproduce. To bridge this gap, we introduce PEFT-Bench, a unified end-to-end benchmark for evaluating diverse PEFT methods on autoregressive LLMs. We demonstrate its usage across 27 NLP datasets and 6 PEFT methods. To account for different PEFT training and inference factors, we also introduce the PEFT Soft Score Penalties (PSCP) metric, which takes trainable parameters, inference speed, and training memory usage into account.
Authors Robert Belanec, Branislav Pecher, Ivan Srba, Maria Bielikova
Categories Computation and Language
3
Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement Learning
Title CN Prompt-R1：通过端到端强化学习的协同自动提示框架
Keywords 自动提示 · 强化学习 · 大语言模型 · 协同推理 · 提示工程
Summary 本文提出Prompt-R1，一个基于端到端强化学习的自动提示框架，利用小规模语言模型生成提示来引导大规模语言模型完成复杂推理任务。通过双约束奖励机制优化正确性和推理质量，在多个数据集上表现出色。该框架为通用语言模型交互设计，不涉及视觉或多模态内容处理。
Reason 该论文属于典型的‘平行应用’类工作，提出了一种通用的自动提示生成框架，用于提升LLM在复杂任务上的表现。其核心是设计一个小模型与大模型协作的强化学习机制，并非针对文档图像理解（DIU）中的具体问题（如OCR幻觉、布局分析、细粒度定位等）进行方法创新。尽管其‘自动提示’思想可能间接用于DocVQA任务的提示工程优化，但缺乏对视觉输入处理、多模态对齐或文档结构建模的直接贡献，属于通用NLP层面的技术包装，不具备向DIU迁移的底层方法价值。
Abstract Recently, advanced large language models (LLMs) have emerged at an increasingly rapid pace. However, when faced with complex problems, most users are often unable to provide accurate and effective prompts to interact with LLMs, thus limiting the performance of LLMs. To address this challenge, we propose Prompt-R1, an end-to-end reinforcement learning framework that uses a small-scale LLM to collaborate with large-scale LLMs, replacing user interaction to solve problems better. This collaboration is cast as a multi-turn prompt interaction, where the small-scale LLM thinks and generates prompts, and the large-scale LLM performs complex reasoning. A dual-constrained reward is designed to optimize for correctness, generation quality, and reasoning accuracy. Prompt-R1 provides a plug-and-play framework that supports both inference and training with various large-scale LLMs. Experiments on multiple public datasets show that Prompt-R1 significantly outperforms baseline models across tasks. Our code is publicly available at https://github.com/QwenQKing/Prompt-R1.
Authors Wenjin Liu, Haoran Luo, Xueyuan Lin, Haoming Liu, Tiesunlong Shen, Jiapu Wang, Rui Mao, Erik Cambria
Categories Computation and Language
3
Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization
Title CN 多模态摘要的细粒度且可解释的事实性评估
Keywords 多模态摘要 · 事实性评估 · 幻觉检测 · 评估框架 · NLP
Summary 本文提出了两种用于评估多模态摘要事实性的细粒度框架——有参考和无参考版本，旨在检测生成摘要中的不实信息。无需真实标签的无参考框架具备更广的应用场景。作者通过与其他指标的相关性分析验证了其有效性，并承诺开源代码与数据集。
Reason 该论文聚焦于多模态摘要生成中的事实性评估问题，提出的是针对摘要结果的评价框架（FALLACIOUS），属于对现有模型输出进行打分的下游评估方法。其应用场景为新闻或跨模态内容摘要，与文档图像理解（DIU）的核心任务如DocVQA、表格识别、布局分析等无直接关联。虽然‘去幻觉’是DIU的重要痛点，但本文并未提出任何缓解幻觉的建模机制或架构改进，而是构建了一个外部评估体系，属于平行应用层面的工作，无法迁移为解决OCR幻觉或视觉推理错误的底层技术手段。
Abstract Multimodal summarization aims to generate a concise summary based on the input text and image. However, the existing methods potentially suffer from unfactual output. To evaluate the factuality of multimodal summarization models, we propose two fine-grained and explainable evaluation frameworks (FALLACIOUS) for different application scenarios, i.e. reference-based factuality evaluation framework and reference-free factuality evaluation framework. Notably, the reference-free factuality evaluation framework doesn't need ground truth and hence it has a wider application scenario. To evaluate the effectiveness of the proposed frameworks, we compute the correlation between our frameworks and the other metrics. The experimental results show the effectiveness of our proposed method. We will release our code and dataset via github.
Authors Yue Zhang, Jingxuan Zuo, Liqiang Jing
Categories Computation and Language
3
Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models
Title CN 大型语言模型中的正字法约束满足与人类难度对齐
Keywords 正字法约束 · 语言模型 · 字符级生成 · 人类难度对齐 · 推理幻觉
Summary 论文评估了多种大语言模型在58个需要字符级约束满足的单词谜题上的表现，发现架构差异对性能的影响远大于参数规模扩展，并指出模型在常见但拼写异常的词上失败率高，反映出对分布合理性的过度依赖。同时分析了推理计算预算的敏感性及模型与人类难度感知的对齐程度。
Reason 该论文研究的是纯文本生成中字符级约束满足问题（如字谜求解），属于纯NLP范畴，未涉及任何视觉或文档模态。尽管提到了LLM在处理非典型拼写时的幻觉问题，但其任务设定完全脱离图像输入、OCR输出或文档布局等DIU核心场景，无法迁移解决OCR幻觉或细粒度定位等实际文档理解痛点。
Abstract Large language models must satisfy hard orthographic constraints during controlled text generation, yet systematic cross-architecture evaluation remains limited. We evaluate 28 configurations spanning three model families (Qwen3, Claude Haiku-4.5, GPT-5-mini) on 58 word puzzles requiring character-level constraint satisfaction. Architectural differences produce substantially larger performance gaps (2.0-2.2x, F1=0.761 vs. 0.343) than parameter scaling within families (83% gain from eightfold scaling), suggesting that constraint satisfaction may require specialized architectural features or training objectives beyond standard language model scaling. Thinking budget sensitivity proves heterogeneous: high-capacity models show strong returns (+0.102 to +0.136 F1), while mid-sized variants saturate or degrade. These patterns are inconsistent with uniform compute benefits. Using difficulty ratings from 10,000 human solvers per puzzle, we establish modest but consistent calibration (r=0.24-0.38) across all families, yet identify systematic failures on common words with unusual orthography ("data", "poop", "loll": 86-95% human success, 89-96% model miss rate). These failures reveal over-reliance on distributional plausibility that penalizes orthographically atypical but constraint-valid patterns, suggesting architectural innovations may be required beyond simply scaling parameters or computational budgets.
Authors Bryan E. Tuck, Rakesh M. Verma
Categories Computation and Language
3
The Structure-Content Trade-off in Knowledge Graph Retrieval
Title CN 知识图谱检索中的结构与内容权衡
Keywords 知识图谱 · 大语言模型 · 检索策略 · 问题分解 · 事实推理
Summary 本文探讨了在基于大语言模型的知识图谱问答中，使用原始问题检索与分解后的子问题检索之间的权衡：前者保留知识图谱的结构连贯性但可能缺乏相关性，后者提高内容精确度但导致子图碎片化。实验表明最优性能出现在两者平衡点。研究核心在于检索策略对推理效果的影响。
Reason 该论文研究的是知识图谱检索中问题分解对子图结构和内容的影响，属于纯NLP与知识图谱结合的推理任务，未涉及任何视觉或文档模态。其方法聚焦于如何在结构保持与内容相关性之间做权衡，但完全不涉及文档图像、OCR、多模态建模或视觉表示学习。无法迁移用于解决文档图像理解中的OCR幻觉、复杂排版或高分辨率处理等核心痛点，属于无关领域的下游应用。
Abstract Large Language Models (LLMs) increasingly rely on knowledge graphs for factual reasoning, yet how retrieval design shapes their performance remains unclear. We examine how question decomposition changes the retrieved subgraph's content and structure. Using a hybrid retrieval function that controls the importance of initial question and subquestions, we show that subquestion-based retrieval improves content precision, but yields disjoint subgraphs, while question-based retrieval maintains structure at the cost of relevance. Optimal performance arises between these extremes, revealing that balancing retrieval content and structure is key to effective LLM reasoning over structured knowledge.
Authors Valentin Six, Evan Dufraisse, Gaël de Chalendar
Categories Computation and Language
3
A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency
Title CN 关于大语言模型推理引擎的综述：优化与效率的视角
Keywords 大语言模型 · 推理引擎 · 效率优化 · 系统综述 · 部署
Summary 本文综述了25个开源和商业的大语言模型推理引擎，从易用性、可扩展性、吞吐量与延迟优化等方面进行评估，并分析其背后支持的优化技术。文章还探讨了生态系统成熟度和成本策略，为研究人员和开发者提供选型和设计指导。
Reason 该论文是一篇关于LLM推理引擎的系统性综述，聚焦于通用大语言模型服务中的部署、优化与效率问题。尽管其讨论的推理优化技术（如并行化、缓存、压缩）在广义上对多模态系统有潜在价值，但内容完全不涉及视觉输入、文档结构或图文对齐等文档图像理解（DIU）的核心挑战。未提出任何可用于解决OCR幻觉、复杂排版建模或高分辨率文档处理的新方法或架构设计，属于通用NLP基础设施层面的总结，无直接迁移价值。
Abstract Large language models (LLMs) are widely applied in chatbots, code generators, and search engines. Workload such as chain-of-throught, complex reasoning, agent services significantly increase the inference cost by invoke the model repeatedly. Optimization methods such as parallelism, compression, and caching have been adopted to reduce costs, but the diverse service requirements make it hard to select the right method. Recently, specialized LLM inference engines have emerged as a key component for integrating the optimization methods into service-oriented infrastructures. However, a systematic study on inference engines is still lacking.This paper provides a comprehensive evaluation of 25 open-source and commercial inference engines. We examine each inference engine in terms of ease-of-use, ease-of-deployment, general-purpose support, scalability, and suitability for throughput- and latency-aware computation. Furthermore, we explore the design goals of each inference engine by investigating the optimization techniques it supports. In addition, we assess the ecosystem maturity of open source inference engines and handle the performance and cost policy of commercial solutions.We outline future research directions that include support for complex LLM-based services, support of various hardware, and enhanced security, offering practical guidance to researchers and developers in selecting and designing optimized LLM inference engines. We also provide a public repository to continually track developments in this fast-evolving field: \href{https://github.com/sihyeong/Awesome-LLM-Inference-Engine}{https://github.com/sihyeong/Awesome-LLM-Inference-Engine}.
Authors Sihyeong Park, Sungryeol Jeon, Chaelyn Lee, Seokhun Jeon, Byung-Soo Kim, Jemin Lee
Comment Under review; 106 pages; 46 figures
Categories Computation and Language
3
CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis AAAI 2026
Title CN CAMERA：通过微专家冗余分析实现MoE模型的多矩阵联合压缩
Keywords MoE模型 · 模型压缩 · 微专家 · 参数冗余 · 量化
Summary 本文提出CAMERA框架，将MoE模型中的专家分解为更细粒度的‘微专家’，并通过跨矩阵联合压缩实现高效参数缩减。该方法包含无需训练的微专家冗余分析、结构化剪枝（CAMERA-P）和混合精度量化（CAMERA-Q），在保持性能的同时大幅降低计算开销。实验显示其在多个NLP任务上优于现有压缩方法。
Reason 该论文聚焦于MoE架构大语言模型的参数压缩与效率优化，属于模型压缩领域的通用技术。尽管其方法在计算效率上有显著提升，但并未针对文档图像理解（DIU）中的核心问题（如OCR幻觉、复杂排版、细粒度定位、高分辨率处理等）提出直接解决方案。其应用场景和实验任务均为通用NLP下游任务，无证据表明该方法可迁移至文档视觉问答或布局分析等DIU场景。属于典型的‘平行应用’类技术优化，缺乏对多模态尤其是视觉-文本联合建模的创新，无法解决VLM在文档理解中的根本瓶颈。
Abstract Large Language Models (LLMs) with Mixture-of-Experts (MoE) architectures are distinguished by their strong performance scaling with increasing parameters across a wide range of tasks, yet they also suffer from substantial computational and storage overheads. Notably, the performance gains of MoE models do not scale proportionally with the growth in expert parameters. While prior works attempt to reduce parameters via expert-level pruning, merging, or decomposition, they still suffer from challenges in both performance and computational efficiency. In this paper, we address these challenges by introducing micro-expert as a finer-grained compression unit that spans across matrices. We first establish a more fundamental perspective, viewing MoE layers as mixtures of micro-experts, and present CAMERA, a lightweight and training-free framework for identifying micro-expert redundancy. Our analysis uncovers significant variance in micro-expert contributions during decoding. Based on this insight, we further propose CAMERA-P, a structured micro-expert pruning framework, and CAMERA-Q, a mixed-precision quantization idea designed for micro-experts. Extensive experiments on nine downstream tasks show that CAMERA-P consistently outperforms strong baselines under pruning ratios ranging from 20% to 60%. Furthermore, CAMERA-Q achieves superior results under aggressive 2-bit quantization, surpassing existing matrix- and channel-level ideas. Notably, our method enables complete micro-expert analysis of Qwen2-57B-A14B in less than 5 minutes on a single NVIDIA A100-40GB GPU.
Authors Yuzhuang Xu, Xu Han, Yuanchi Zhang, Yixuan Wang, Yijun Liu, Shiyu Ji, Qingfu Zhu, Wanxiang Che
Comment Accepted in AAAI 2026
Categories Computation and Language
3
A Unified Understanding of Offline Data Selection and Online Self-refining Generation for Post-training LLMs
Title CN 用于大模型后训练的离线数据选择与在线自优化生成的统一理解
Keywords 大语言模型 · 数据选择 · 自优化生成 · 双层优化 · 后训练
Summary 本文从优化视角提出一个统一框架，将大模型后训练中的离线数据选择和在线自优化生成建模为基于验证集的数据加权问题，采用双层优化策略为每个问题和回答分配学习权重，从而提升微调效果。实验表明该方法在文本质量增强和安全感知微调上优于直接混合基线。
Reason 该论文聚焦于大语言模型在文本模态下的后训练阶段，研究离线数据筛选与在线自我优化生成的统一框架，核心关注点为文本质量提升与安全对齐。其方法论围绕纯文本数据加权与模型适应展开，并未涉及视觉输入或文档图像中的多模态理解问题。尽管‘数据选择’和‘自优化’机制在理论上具有通用性，但本文完全缺乏对OCR噪声、视觉布局、高分辨率图像处理等文档理解关键挑战的建模能力，且无任何视觉-语言对齐设计。因此，无法迁移用于解决DIU中的幻觉、细粒度定位或复杂排版解析等核心痛点。
Abstract Offline data selection and online self-refining generation, which enhance the data quality, are crucial steps in adapting large language models (LLMs) to specific downstream tasks. We tackle offline data selection and online self-refining generations through an optimization perspective. Specifically, bilevel data selection is used for offline data selection with respect to the validation dataset, and we treat online self-refining generation as a model adaptation step of selecting the model trained on current responses that best fits the validation data. Our framework offers a unified understanding of offline data selection and self-refining generation by assigning a learned data weight to each question and response, either explicitly or implicitly. For the first time, we theoretically demonstrate the effectiveness of the bilevel data selection framework and demonstrate its performance gains over unfiltered direct mixing baselines. By combining offline data with validation-weighted online generations, our method enhances fine-tuning performance. Experiments on quality enhancement and safety-aware LLM fine-tuning validate its effectiveness.
Authors Quan Xiao, Tianyi Chen
Categories Computation and Language
3
MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models
Title CN MapReduce LoRA：推进生成模型多偏好优化中的帕累托前沿
Keywords 多偏好优化 · LoRA · RLHF · 生成模型对齐 · Reward Modeling
Summary 本文提出MapReduce LoRA与RaTE两种方法，用于协调生成模型在多个奖励信号下的优化冲突。通过并行训练偏好特定的LoRA专家并迭代合并，以及学习可组合的奖励感知词嵌入，实现多偏好控制。在文生图、文生视频和语言助手任务中均取得显著性能提升。
Reason 该论文聚焦于多偏好对齐优化，应用于文生图、文生视频和语言模型的美学与安全性平衡，属于典型的跨模态生成任务调优框架。尽管其方法在技术上具有系统性，但核心目标并非解决文档图像理解中的关键挑战（如OCR幻觉、复杂版式建模、细粒度视觉定位或长文档推理）。虽然实验中提到了OCR指标提升，但仅为生成结果的可读性副产物，并未针对文档结构、布局分析或VQA等DIU核心任务设计机制，属于‘平行应用’类方法，无直接迁移价值。
Abstract Reinforcement learning from human feedback (RLHF) with reward models has advanced alignment of generative models to human aesthetic and perceptual preferences. However, jointly optimizing multiple rewards often incurs an alignment tax, improving one dimension while degrading others. To address this, we introduce two complementary methods: MapReduce LoRA and Reward-aware Token Embedding (RaTE). MapReduce LoRA trains preference-specific LoRA experts in parallel and iteratively merges them to refine a shared base model; RaTE learns reward-specific token embeddings that compose at inference for flexible preference control. Experiments on Text-to-Image generation (Stable Diffusion 3.5 Medium and FLUX.1-dev) show improvements of 36.1%, 4.6%, and 55.7%, and 32.7%, 4.3%, and 67.1% on GenEval, PickScore, and OCR, respectively. On Text-to-Video generation (HunyuanVideo), visual and motion quality improve by 48.1% and 90.0%, respectively. On the language task, Helpful Assistant, with Llama-2 7B, helpful and harmless improve by 43.4% and 136.7%, respectively. Our framework sets a new state-of-the-art multi-preference alignment recipe across modalities.
Authors Chieh-Yun Chen, Zhonghao Wang, Qi Chen, Zhifan Ye, Min Shi, Yue Zhao, Yinan Zhao, Hui Qu, Wei-An Lin, Yiru Shen, Ajinkya Kale, Irfan Essa, Humphrey Shi
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
ROOT: Robust Orthogonalized Optimizer for Neural Network Training
Title CN ROOT：面向神经网络训练的鲁棒正交化优化器
Keywords 优化器 · 训练稳定性 · 正交化 · 大模型训练 · 神经网络
Summary 本文提出了一种名为ROOT的鲁棒正交化优化器，通过自适应牛顿迭代和邻近优化机制来提升大规模模型训练的稳定性和收敛速度。该方法主要解决优化过程中的维度脆弱性和异常值噪声问题，适用于大语言模型等场景，但未涉及文档图像或多模态理解任务。
Reason 该论文聚焦于神经网络训练中的优化器设计，属于通用深度学习训练方法的研究，虽然可能对大模型训练有潜在价值，但并未针对文档图像理解（DIU）任务或其核心痛点（如OCR幻觉、复杂排版、细粒度定位等）提出直接改进。其方法属于通用优化算法创新，迁移至DIU领域的路径不明确，且无任何与多模态、视觉输入或文档结构相关的实验验证，属于与DIU无关的底层训练技术，不符合本筛选标准中对‘方法可迁移性’的要求。
Abstract The optimization of large language models (LLMs) remains a critical challenge, particularly as model scaling exacerbates sensitivity to algorithmic imprecision and training instability. Recent advances in optimizers have improved convergence efficiency through momentum orthogonalization, but suffer from two key robustness limitations: dimensional fragility in orthogonalization precision and vulnerability to outlier-induced noise. To address these robustness challenges, we introduce ROOT, a Robust Orthogonalized Optimizer that enhances training stability through dual robustness mechanisms. First, we develop a dimension-robust orthogonalization scheme using adaptive Newton iterations with fine-grained coefficients tailored to specific matrix sizes, ensuring consistent precision across diverse architectural configurations. Second, we introduce an optimization-robust framework via proximal optimization that suppresses outlier noise while preserving meaningful gradient directions. Extensive experiments demonstrate that ROOT achieves significantly improved robustness, with faster convergence and superior final performance compared to both Muon and Adam-based optimizers, particularly in noisy and non-convex scenarios. Our work establishes a new paradigm for developing robust and precise optimizers capable of handling the complexities of modern large-scale model training. The code will be available at https://github.com/huawei-noah/noah-research/tree/master/ROOT.
Authors Wei He, Kai Han, Hang Zhou, Hanting Chen, Zhicheng Liu, Xinghao Chen, Yunhe Wang
Categories Artificial Intelligence
3
DiFR: Inference Verification Despite Nondeterminism
Title CN DiFR：尽管存在非确定性仍可进行推理验证
Keywords 推理验证 · 非确定性 · 采样一致性 · 激活指纹 · 可信LLM
Summary 本文提出Token-DiFR和Activation-DiFR方法，用于在存在数值噪声的情况下验证大语言模型推理输出的正确性。通过同步采样种子并比对可信参考实现的输出，能够以极低开销检测出错误或篡改行为。实验表明其在检测4位量化方面表现优异，且Activation-DiFR显著降低了通信成本。
Reason 该论文聚焦于大模型推理过程的正确性验证，目标是检测服务提供商是否在推理过程中引入错误或篡改（如量化导致的偏差），其核心贡献是Token-DiFR和Activation-DiFR两种验证机制。然而，这项工作属于LLM部署与可信计算范畴，并未涉及多模态输入、视觉内容理解或文档结构建模。虽然其验证思想理论上可用于监控VLM推理稳定性，但本身不解决文档图像理解中的OCR幻觉、复杂排版解析、细粒度定位等核心问题，也无迁移路径直接提升DIU性能。
Abstract As demand for LLM inference grows, it is becoming increasingly important that providers and their customers can verify that inference processes are performed correctly, without errors or tampering. However, re-running the same inference process twice often leads to different results due to benign numerical noise, making it difficult to distinguish legitimate variation from actual problems. To address this problem, we introduce Token-DiFR (Token-Divergence-From-Reference), a method for verifying inference outputs by comparing generated tokens against predictions made by a trusted reference implementation conditioned on the same random seed. Sampling seed synchronization tightly constrains valid outputs, leaving providers minimal room to deviate from correct inference, which allows output tokens themselves to serve as auditable evidence of correctness at zero additional cost to the provider. Token-DiFR reliably identifies sampling errors, simulated bugs, and model quantization, detecting 4-bit quantization with AUC $>$ 0.999 within 300 output tokens. For applications requiring sample-efficient forward-pass verification, we additionally introduce Activation-DiFR, a scheme that uses random orthogonal projections to compress activations into compact fingerprints for subsequent verification. Activation-DiFR detects 4-bit quantization with AUC $>$ 0.999 using just 2 output tokens, while reducing communication overhead by 25-75% relative to existing methods. We release an open-source integration with vLLM to accelerate practical deployment of verifiable inference.
Authors Adam Karvonen, Daniel Reuter, Roy Rinberg, Luke Marks, Adrià Garriga-Alonso, Keri Warr
Categories Artificial Intelligence
3
Block Cascading: Training Free Acceleration of Block-Causal Video Models
Title CN 块级级联：无需训练的块因果视频模型加速方法
Keywords 视频生成 · 块因果模型 · 推理加速 · 并行化 · 无需训练
Summary 本文提出Block Cascading，一种无需训练的加速方法，用于块因果视频生成模型。通过允许后续视频块基于前序块的“部分去噪”结果提前开始生成，实现多个块的并行去噪，从而在5块GPU上达到约2倍的推理加速，且不牺牲生成质量。同时消除了KV缓存切换带来的延迟，提升了交互式生成效率。
Reason 该论文聚焦于块因果视频生成模型的推理加速，属于视频生成领域，核心问题是提升视频生成的帧率与响应速度。尽管其提出的并行化思路在理论上具有启发性，但其应用场景与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版解析、细粒度文本定位、长文档逻辑推理）无直接关联。视频时序建模与文档结构理解在语义、空间布局和任务目标上差异巨大，该方法难以迁移至DIU场景。此外，未涉及多模态对齐、视觉压缩或VLM架构改进等对文档理解有潜在价值的技术路径。
Abstract Block-causal video generation faces a stark speed-quality trade-off: small 1.3B models manage only 16 FPS while large 14B models crawl at 4.5 FPS, forcing users to choose between responsiveness and quality. Block Cascading significantly mitigates this trade-off through training-free parallelization. Our key insight: future video blocks do not need fully denoised current blocks to begin generation. By starting block generation with partially denoised context from predecessors, we transform sequential pipelines into parallel cascades where multiple blocks denoise simultaneously. With 5 GPUs exploiting temporal parallelism, we achieve ~2x acceleration across all model scales: 1.3B models accelerate from 16 to 30 FPS, 14B models from 4.5 to 12.5 FPS. Beyond inference speed, Block Cascading eliminates overhead from KV-recaching (of ~200ms) during context switches for interactive generation. Extensive evaluations validated against multiple block-causal pipelines demonstrate no significant loss in generation quality when switching from block-causal to Block Cascading pipelines for inference. Project Page: https://hmrishavbandy.github.io/block_cascading_page/
Authors Hmrishav Bandyopadhyay, Nikhil Pinnaparaju, Rahim Entezari, Jim Scott, Yi-Zhe Song, Varun Jampani
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Counterfactual Simulatability of LLM Explanations for Generation Tasks INLG25
Title CN 生成任务中LLM解释的反事实可模拟性
Keywords 大语言模型 · 可解释性 · 反事实推理 · 生成任务 · 评估框架
Summary 本文提出了一种用于评估大语言模型在生成任务中解释质量的框架——反事实可模拟性，并在新闻摘要和医疗建议任务上进行了验证。结果显示该方法对技能型任务更有效，而在知识密集型任务中提升有限。
Reason 该论文研究的是自然语言生成任务中大模型解释的可模拟性评估，聚焦于新闻摘要和医疗建议等纯文本生成场景。其核心是评估人类能否通过模型提供的解释预测其在反事实输入下的输出行为。尽管涉及LLM可解释性，但完全不涉及视觉模态、文档结构或OCR相关问题，与文档图像理解（DIU）无直接关联。属于通用NLP方法在特定任务上的分析，无法迁移解决DIU中的OCR幻觉、复杂排版或细粒度定位等关键痛点。
Abstract LLMs can be unpredictable, as even slight alterations to the prompt can cause the output to change in unexpected ways. Thus, the ability of models to accurately explain their behavior is critical, especially in high-stakes settings. One approach for evaluating explanations is counterfactual simulatability, how well an explanation allows users to infer the model's output on related counterfactuals. Counterfactual simulatability has been previously studied for yes/no question answering tasks. We provide a general framework for extending this method to generation tasks, using news summarization and medical suggestion as example use cases. We find that while LLM explanations do enable users to better predict LLM outputs on counterfactuals in the summarization setting, there is significant room for improvement for medical suggestion. Furthermore, our results suggest that the evaluation for counterfactual simulatability may be more appropriate for skill-based tasks as opposed to knowledge-based tasks.
Authors Marvin Limpijankit, Yanda Chen, Melanie Subbiah, Nicholas Deas, Kathleen McKeown
Comment INLG25
Categories Artificial Intelligence, Computation and Language
3
Iterative Inference in a Chess-Playing Neural Network
Title CN 国际象棋神经网络中的迭代推理
Keywords 神经网络推理 · 可解释性 · logit lens · 国际象棋AI
Summary 本文通过扩展logit lens方法分析Leela Chess Zero策略网络的层间表示变化，发现其能力提升并非平滑渐进，而是经历多个计算阶段，且中间层找到的正确解可能在后期被推翻，最终层更倾向于安全而非激进策略。这表明启发式先验可能覆盖战术解。
Reason 该论文研究的是国际象棋AI中策略网络的内部表示演化，属于具身智能与游戏AI领域，与文档图像理解（DIU）或视觉语言模型在文档任务中的应用无直接关联。尽管其分析方法（如logit lens）可能对模型可解释性有启发，但文中未涉及多模态输入、视觉文本理解、OCR、布局分析等DIU核心问题，也无法迁移用于解决OCR幻觉、复杂排版或长文档推理等痛点。
Abstract Do neural networks build their representations through smooth, gradual refinement, or via more complex computational processes? We investigate this by extending the logit lens to analyze the policy network of Leela Chess Zero, a superhuman chess engine. Although playing strength and puzzle-solving ability improve consistently across layers, capability progression occurs in distinct computational phases with move preferences undergoing continuous reevaluation--move rankings remain poorly correlated with final outputs until late, and correct puzzle solutions found in middle layers are sometimes overridden. This late-layer reversal is accompanied by concept preference analyses showing final layers prioritize safety over aggression, suggesting a mechanism by which heuristic priors can override tactical solutions.
Authors Elias Sandmann, Sebastian Lapuschkin, Wojciech Samek
Categories Artificial Intelligence
3
Active Inference in Discrete State Spaces from First Principles
Title CN 从第一性原理看离散状态空间中的主动推理
Keywords 主动推理 · 自由能原理 · 变分推断 · 离散状态空间 · 认知建模
Summary 本文从第一性原理出发，重新审视主动推理机制，提出在离散状态空间中通过约束散度最小化来实现感知与行动的统一建模，使用标准平均场方法求解，而无需依赖预期自由能概念。虽然为智能体行为建模提供了新视角，但缺乏与视觉或多模态实际系统的连接。
Reason 该论文聚焦于主动推理（active inference）的理论基础，试图将其与自由能原理解耦，并提出基于离散状态空间的优化框架。尽管在AI理论层面有一定探讨价值，但其内容属于通用认知架构的数学建模，未涉及任何多模态、视觉或文档理解任务。无法迁移解决文档图像理解中的核心痛点如OCR幻觉、复杂排版处理或细粒度定位等问题。
Abstract We seek to clarify the concept of active inference by disentangling it from the Free Energy Principle. We show how the optimizations that need to be carried out in order to implement active inference in discrete state spaces can be formulated as constrained divergence minimization problems which can be solved by standard mean field methods that do not appeal to the idea of expected free energy. When it is used to model perception, the perception/action divergence criterion that we propose coincides with variational free energy. When it is used to model action, it differs from an expected free energy functional by an entropy regularizer.
Authors Patrick Kenny
Comment 56 pages
Categories Artificial Intelligence
3
HVAdam: A Full-Dimension Adaptive Optimizer
Title CN HVAdam：一种全维度自适应优化器
Keywords 自适应优化器 · 深度学习训练 · 收敛性分析 · Anon · IDU
Summary 本文提出Anon优化器，支持连续可调的自适应性，能够在SGD-like和Adam-like行为之间插值甚至外推，并引入增量延迟更新（IDU）机制以保证收敛性。理论和实验表明其在图像分类、扩散模型和语言建模任务上优于现有优化器。
Reason 该论文提出了一种新的自适应优化器Anon，旨在统一SGD与Adam的优点，并通过可调的自适应性机制提升在不同模型上的泛化能力。然而，这项工作属于通用深度学习优化算法的研究，虽然可能间接用于训练文档理解模型，但并未针对文档图像理解（DIU）中的任何具体问题（如OCR幻觉、复杂排版建模、视觉token压缩等）进行设计或验证。其方法论也不属于VLM架构、推理增强、多模态对齐或高分辨率处理等可直接迁移至DIU的关键工具。因此属于‘平行技术应用’的上游基础组件，缺乏对DIU领域的直接贡献或迁移价值。
Abstract Adaptive optimizers such as Adam have achieved great success in training large-scale models like large language models and diffusion models. However, they often generalize worse than non-adaptive methods, such as SGD on classical architectures like CNNs. We identify a key cause of this performance gap: adaptivity in pre-conditioners, which limits the optimizer's ability to adapt to diverse optimization landscapes. To address this, we propose Anon (Adaptivity Non-restricted Optimizer with Novel convergence technique), a novel optimizer with continuously tunable adaptivity , allowing it to interpolate between SGD-like and Adam-like behaviors and even extrapolate beyond both. To ensure convergence across the entire adaptivity spectrum, we introduce incremental delay update (IDU), a novel mechanism that is more flexible than AMSGrad's hard max-tracking strategy and enhances robustness to gradient noise. We theoretically establish convergence guarantees under both convex and non-convex settings. Empirically, Anon consistently outperforms state-of-the-art optimizers on representative image classification, diffusion, and language modeling tasks. These results demonstrate that adaptivity can serve as a valuable tunable design principle, and Anon provides the first unified and reliable framework capable of bridging the gap between classical and modern optimizers and surpassing their advantageous properties.
Authors Yiheng Zhang, Shaowu Wu, Yuanzhuo Xu, Jiajun Wu, Shang Xu, Steve Drew, Xiaoguang Niu
Categories Artificial Intelligence
3
Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits NeurIPS 2025
Title CN 超越组件：基于奇异向量的Transformer电路可解释性
Keywords Transformer可解释性 · 奇异向量分析 · 电路分解 · 机械可解释性 · 内部机制
Summary 本文提出一种基于奇异向量分解的方法来解析Transformer中注意力头和MLP层的内部功能，发现单个组件内存在多个正交的子功能方向。尽管为模型可解释性提供了更细粒度的视角，但研究完全基于纯文本模型和标准NLP任务，未涉及任何视觉或多模态内容。
Reason 该论文属于模型可解释性领域的基础研究，聚焦于语言模型内部计算的细粒度分解，提出通过奇异方向分析注意力头和MLP层中的复合功能。虽然其方法论具有理论深度，但完全局限于纯文本Transformer的机制分析，未涉及视觉输入、多模态建模或文档结构理解。对于文档图像理解（DIU）中的核心问题如OCR幻觉、复杂排版建模、视觉-文本对齐等无直接迁移路径，也不提供处理高分辨率图像或细粒度定位的能力。属于典型的‘平行上游理论’，与DIU任务脱节。
Abstract Transformer-based language models exhibit complex and distributed behavior, yet their internal computations remain poorly understood. Existing mechanistic interpretability methods typically treat attention heads and multilayer perceptron layers (MLPs) (the building blocks of a transformer architecture) as indivisible units, overlooking possibilities of functional substructure learned within them. In this work, we introduce a more fine-grained perspective that decomposes these components into orthogonal singular directions, revealing superposed and independent computations within a single head or MLP. We validate our perspective on widely used standard tasks like Indirect Object Identification (IOI), Gender Pronoun (GP), and Greater Than (GT), showing that previously identified canonical functional heads, such as the name mover, encode multiple overlapping subfunctions aligned with distinct singular directions. Nodes in a computational graph, that are previously identified as circuit elements show strong activation along specific low-rank directions, suggesting that meaningful computations reside in compact subspaces. While some directions remain challenging to interpret fully, our results highlight that transformer computations are more distributed, structured, and compositional than previously assumed. This perspective opens new avenues for fine-grained mechanistic interpretability and a deeper understanding of model internals.
Authors Areeb Ahmad, Abhinav Joshi, Ashutosh Modi
Comment Accepted at NeurIPS 2025
Categories Artificial Intelligence, Computation and Language
3
Demystifying Higher-Order Graph Neural Networks
Title CN 揭秘高阶图神经网络
Keywords 图神经网络 · 高阶关系 · 拓扑深度学习 · 模型分类 · GNN表达能力
Summary 本文提出了高阶图神经网络（HOGNNs）的分类体系和设计蓝图，系统分析了现有模型的架构差异与理论基础，总结了选择合适HOGNN的指导原则，并指出了未来研究的挑战与机遇。目标是统一和澄清当前碎片化的HOGNN研究现状。
Reason 该论文属于图神经网络的理论分类与分析工作，聚焦于拓扑深度学习中的高阶关系建模，尽管在GNN领域具有一定系统性贡献，但其研究对象与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。未提出可迁移至DIU领域的具体技术工具（如视觉表示学习、多模态对齐、细粒度定位或去幻觉机制），且不涉及视觉-语言模型架构、高分辨率处理或推理扩展等可用于解决OCR幻觉、复杂排版或长文档推理的方法。属于通用AI理论整理，难以直接赋能文档理解任务。
Abstract Higher-order graph neural networks (HOGNNs) and the related architectures from Topological Deep Learning are an important class of GNN models that harness polyadic relations between vertices beyond plain edges. They have been used to eliminate issues such as over-smoothing or over-squashing, to significantly enhance the accuracy of GNN predictions, to improve the expressiveness of GNN architectures, and for numerous other goals. A plethora of HOGNN models have been introduced, and they come with diverse neural architectures, and even with different notions of what the "higher-order" means. This richness makes it very challenging to appropriately analyze and compare HOGNN models, and to decide in what scenario to use specific ones. To alleviate this, we first design an in-depth taxonomy and a blueprint for HOGNNs. This facilitates designing models that maximize performance. Then, we use our taxonomy to analyze and compare the available HOGNN models. The outcomes of our analysis are synthesized in a set of insights that help to select the most beneficial GNN model in a given scenario, and a comprehensive list of challenges and opportunities for further research into more powerful HOGNNs.
Authors Maciej Besta, Florian Scheidl, Lukas Gianinazzi, Grzegorz Kwasniewski, Shachar Klaiman, Jürgen Müller, Torsten Hoefler
Categories Artificial Intelligence
3
ConfTuner: Training Large Language Models to Express Their Confidence Verbally NeurIPS 2025
Title CN ConfTuner：训练大语言模型以文字形式表达其置信度
Keywords 大语言模型 · 置信度校准 · 不确定性表达 · 微调 · 可信AI
Summary 本文提出ConfTuner，一种通过新型损失函数（tokenized Brier score）微调大语言模型以更好表达其预测置信度的方法。该方法无需真实置信标签，理论上具有正确性保障，并在多种推理任务上提升了模型自我修正和级联使用的能力。目标是缓解LLM在高风险领域中的过度自信问题。
Reason 该论文研究的是大语言模型在生成答案时如何通过文本表达其置信度，核心问题是模型校准（calibration）与不确定性表达，属于纯NLP范畴。虽然‘可信LLM’在理论上对所有应用领域有潜在价值，但该工作并未涉及视觉输入、多模态对齐或文档结构理解等DIU核心问题。其方法无法直接迁移用于解决OCR幻觉、细粒度定位或复杂排版推理等文档理解痛点，属于通用语言模型的可靠性优化，与文档图像理解无实质关联。
Abstract Large Language Models (LLMs) are increasingly deployed in high-stakes domains such as science, law, and healthcare, where accurate expressions of uncertainty are essential for reliability and trust. However, current LLMs are often observed to generate incorrect answers with high confidence, a phenomenon known as "overconfidence". Recent efforts have focused on calibrating LLMs' verbalized confidence: i.e., their expressions of confidence in text form, such as "I am 80% confident that...". Existing approaches either rely on prompt engineering or fine-tuning with heuristically generated uncertainty estimates, both of which have limited effectiveness and generalizability. Motivated by the notion of proper scoring rules for calibration in classical machine learning models, we introduce ConfTuner, a simple and efficient fine-tuning method that introduces minimal overhead and does not require ground-truth confidence scores or proxy confidence estimates. ConfTuner relies on a new loss function, tokenized Brier score, which we theoretically prove to be a proper scoring rule, intuitively meaning that it "correctly incentivizes the model to report its true probability of being correct". ConfTuner improves calibration across diverse reasoning tasks and generalizes to black-box models such as GPT-4o. Our results further show that better-calibrated confidence enables downstream gains in self-correction and model cascade, advancing the development of trustworthy LLM systems. The code is available at https://github.com/liushiliushi/ConfTuner.
Authors Yibo Li, Miao Xiong, Jiaying Wu, Bryan Hooi
Comment Accepted by NeurIPS 2025
Categories Artificial Intelligence, Computation and Language
3
While recognizing actions, LMMs struggle to detect core interaction events
Title CN 在识别动作时，大型多模态模型难以检测核心交互事件
Keywords 多模态模型 · 视频理解 · 感知接地 · 交互事件检测 · 细粒度定位
Summary 本文研究发现，当前大型多模态模型（LMMs）虽然能准确描述视频中的动作和对象，但在精确定位手与物体之间‘接触’和‘释放’这类核心交互事件的时间点和空间位置上表现不佳。作者构建了一个包含2万多个标注交互事件的大规模数据集，并通过人类标注验证了模型在细粒度感知接地方面的缺陷。实验表明，即使模型能生成合理推理，仍无法准确识别交互起止帧。
Reason 该论文研究的是视频中物理交互事件（如接触与释放）的细粒度定位问题，属于视频理解领域。虽然其关注的‘感知接地’（perceptual grounding）问题与文档图像中的幻觉或定位问题有一定抽象相似性，但其任务场景、数据模态和评估目标均集中在动态视觉交互上，与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、长文本推理等）关联薄弱。所提出的数据集和分析框架难以直接迁移至文档场景，且未涉及文本布局、字符级识别或多模态对齐在文档上的具体挑战。
Abstract Large multi-modal models (LMMs) show increasing performance in realistic visual tasks for images and, more recently, for videos. For example, given a video sequence, such models are able to describe in detail objects, the surroundings and dynamic actions. In this study, we explored the extent to which these models ground their semantic understanding in the actual visual input. Specifically, given sequences of hands interacting with objects, we asked models when and where the interaction begins or ends. For this purpose, we introduce a first of its kind, large-scale dataset with more than 20K annotated interactions on videos from the Something-Something-V2 dataset. 250 AMTurk human annotators labeled core interaction events, particularly when and where objects and agents become attached ('contact') or detached ('release'). We asked two LMMs (Qwen-2.5VL and GPT-4o) to locate these events in short videos, each with a single event. The results show that although the models can reliably name the target objects, identify the action and provide coherent reasoning, they consistently fail to identify the frame where the interaction begins or ends and cannot localize the event within the scene. Our findings suggest that in struggling to pinpoint the moment and location of physical contact that defines the interaction, the models lack the perceptual grounding required for deeper understanding of dynamic scenes.
Authors Daniel Harari, Michael Sidorov, Liel David, Chen Shterental, Abrham Kahsay Gebreselasie, Muhammad Haris Khan
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
SEDA: A Self-Adapted Entity-Centric Data Augmentation for Boosting Gird-based Discontinuous NER Models
Title CN SEDA：一种自适应的以实体为中心的数据增强方法，用于提升基于网格的不连续命名实体识别模型
Keywords 命名实体识别 · 不连续实体 · 数据增强 · 网格模型 · 文本分割
Summary 本文提出了一种名为SEDA的自适应数据增强方法，用于提升基于网格的不连续命名实体识别模型性能。通过借鉴图像处理中的裁剪、缩放和填充技术，在文本网格表示上进行数据增强，以缓解跨句子不连续实体的分割与遗漏问题。实验表明该方法在多个医学文本数据集上对不连续实体识别有显著提升。
Reason 该论文属于纯文本NLP任务中的命名实体识别（NER）研究，尽管涉及‘不连续实体’这一复杂结构，但其方法论基于传统文本网格模型并引入图像风格的数据增强（如裁剪、缩放），并未处理任何视觉或文档布局信息。核心问题设定在文本分割层面，且未涉及多模态、OCR、视觉排版或文档图像理解中的关键挑战（如OCR幻觉、细粒度定位、高分辨率输入等）。因此属于与文档图像理解无关的平行NLP任务，无法为DIU提供可迁移的方法论创新。
Abstract Named Entity Recognition (NER) is a critical task in natural language processing, yet it remains particularly challenging for discontinuous entities. The primary difficulty lies in text segmentation, as traditional methods often missegment or entirely miss cross-sentence discontinuous entities, significantly affecting recognition accuracy. Therefore, we aim to address the segmentation and omission issues associated with such entities. Recent studies have shown that grid-tagging methods are effective for information extraction due to their flexible tagging schemes and robust architectures. Building on this, we integrate image data augmentation techniques, such as cropping, scaling, and padding, into grid-based models to enhance their ability to recognize discontinuous entities and handle segmentation challenges. Experimental results demonstrate that traditional segmentation methods often fail to capture cross-sentence discontinuous entities, leading to decreased performance. In contrast, our augmented grid models achieve notable improvements. Evaluations on the CADEC, ShARe13, and ShARe14 datasets show F1 score gains of 1-2.5% overall and 3.7-8.4% for discontinuous entities, confirming the effectiveness of our approach.
Authors Wen-Fang Su, Hsiao-Wei Chou, Wen-Yang Lin
Comment 9 pages, 5 figures
Categories Artificial Intelligence, Computation and Language
3
IDAP++: Advancing Divergence-Based Pruning via Filter-Level and Layer-Level Optimization
Title CN IDAP++：通过滤波器级和层级优化推进基于发散的剪枝
Keywords 模型压缩 · 剪枝 · 信息流分析 · 层级优化 · 滤波器级剪枝
Summary 本文提出IDAP++，一种基于张量流发散分析的神经网络压缩框架，通过两阶段优化实现滤波器级剪枝和层级别删除，以减少模型参数并保持性能。该方法适用于CNN、Transformer等多种架构，在多个模型上实现了优于现有方法的压缩效果。
Reason 该论文聚焦于神经网络压缩中的结构冗余问题，提出基于信息流发散的滤波器级与层级联合剪枝方法。尽管其在模型压缩方面表现良好，但属于通用模型压缩技术，并未针对文档图像理解（DIU）中的核心痛点（如OCR幻觉、复杂排版、细粒度定位或高分辨率处理）进行设计或验证。方法本质上是模型效率优化，且无任何与多模态输入、视觉-语言对齐或文档结构建模相关的讨论，难以直接迁移用于解决DIU的关键挑战。
Abstract This paper presents a novel approach to neural network compression that addresses redundancy at both the filter and architectural levels through a unified framework grounded in information flow analysis. Building on the concept of tensor flow divergence, which quantifies how information is transformed across network layers, we develop a two-stage optimization process. The first stage employs iterative divergence-aware pruning to identify and remove redundant filters while preserving critical information pathways. The second stage extends this principle to higher-level architecture optimization by analyzing layer-wise contributions to information propagation and selectively eliminating entire layers that demonstrate minimal impact on network performance. The proposed method naturally adapts to diverse architectures, including convolutional networks, transformers, and hybrid designs, providing a consistent metric for comparing the structural importance across different layer types. Experimental validation across multiple modern architectures and datasets reveals that this combined approach achieves substantial model compression while maintaining competitive accuracy. The presented approach achieves parameter reduction results that are globally comparable to those of state-of-the-art solutions and outperforms them across a wide range of modern neural network architectures, from convolutional models to transformers. The results demonstrate how flow divergence serves as an effective guiding principle for both filter-level and layer-level optimization, offering practical benefits for deployment in resource-constrained environments.
Authors Aleksei Samarin, Artem Nazarenko, Egor Kotenko, Valentin Malykh, Alexander Savelev, Aleksei Toropov
Comment 65 pages, 4 figures, 38 tables
Categories Artificial Intelligence
3
VICoT-Agent: A Vision-Interleaved Chain-of-Thought Framework for Interpretable Multimodal Reasoning and Scalable Remote Sensing Analysis
Title CN VICoT-Agent：一种用于可解释多模态推理和可扩展遥感分析的视觉交错思维链框架
Keywords 多模态代理 · 思维链 · 遥感分析 · 视觉语言推理 · 模型蒸馏
Summary 本文提出了一种名为VICoT-Agent的多模态代理框架，通过将视觉工具动态嵌入思维链中，实现多轮交错的视觉-语言推理，用于遥感图像的智能分析。该框架采用基于栈的推理结构和模块化工具套件，提升大模型在复杂任务中的透明度与效率，并通过推理栈蒸馏技术将能力迁移到轻量级模型。实验表明其在多个遥感基准上优于现有方法。
Reason 该论文聚焦于遥感图像分析，属于无关领域（Remote Sensing），其任务场景与文档图像理解（DIU）差异显著。尽管提出了Vision-Interleaved Chain-of-Thought和Agent架构，但其核心应用在遥感领域的复杂推理与工具调用，迁移至文档理解（如DocVQA、表格识别等）场景困难。方法虽有一定新颖性，但属于特定垂直领域的Agent应用，符合‘平行下游应用’特征，缺乏对OCR幻觉、密集文本、复杂版式等DIU核心痛点的针对性解决方案。
Abstract The current remote sensing image analysis task is increasingly evolving from traditional object recognition to complex intelligence reasoning, which places higher requirements on the model's reasoning ability and the flexibility of tool invocation. To this end, we propose a new multimodal agent framework, Vision-Interleaved Chain-of-Thought Framework (VICoT), which implements explicit multi-round reasoning by dynamically incorporating visual tools into the chain of thought. Through a stack-based reasoning structure and a modular MCP-compatible tool suite, VICoT enables LLMs to efficiently perform multi-round, interleaved vision-language reasoning tasks with strong generalization and flexibility.We also propose the Reasoning Stack distillation method to migrate complex Agent behaviors to small, lightweight models, which ensures the reasoning capability while significantly reducing complexity. Experiments on multiple remote sensing benchmarks demonstrate that VICoT significantly outperforms existing SOTA frameworks in reasoning transparency, execution efficiency, and generation quality.
Authors Chujie Wang, Zhiyuan Luo, Ruiqi Liu, Can Ran, Shenghua Fan, Xi Chen, Chu He
Categories Artificial Intelligence
3
BMGQ: A Bottom-up Method for Generating Complex Multi-hop Reasoning Questions from Semi-structured Data
Title CN BMGQ：一种从半结构化数据生成复杂多跳推理问题的自下而上方法
Keywords 多跳问答 · 数据生成 · 自下而上方法 · 自然语言推断 · 证据簇
Summary 本文提出了BMGQ，一种自下而上的自动化方法，用于从半结构化知识源生成高难度、适合训练的多跳问答数据。该方法通过基于自然语言推断的关系标注和多样性扩展构建证据簇，采用逆向问题构造方式组合间接线索，并通过多模型共识过滤与结构化约束分解保证生成质量。整个流程可规模化生成检索困难但可验证的问题，适用于监督微调和强化学习训练，同时减少人工标注成本。
Reason 该论文提出了一种自动生成多跳问答数据的方法，目标是构建用于训练检索与推理模型的高质量QA数据集。尽管其在数据生成层面有一定技术性，但本质属于特定类型数据集的构造方法，且面向的是通用知识源和开放域QA场景，并未针对文档图像理解（DIU）中的核心挑战（如OCR幻觉、排版复杂性、视觉-文本对齐等）进行设计。虽然‘多跳推理’是长文档理解中的潜在需求，但该方法关注的是语义层面的问题生成机制，而非解决文档模态特有的感知或结构解析问题，无法直接迁移用于提升VLM在DocVQA等任务上的表现，属于平行应用类工作。
Abstract Building training-ready multi-hop question answering (QA) datasets that truly stress a model's retrieval and reasoning abilities remains highly challenging recently. While there have been a few recent evaluation datasets that capture the characteristics of hard-to-search but easy-to-verify problems -- requiring the integration of ambiguous, indirect, and cross-domain cues -- these data resources remain scarce and are mostly designed for evaluation, making them unsuitable for supervised fine-tuning (SFT) or reinforcement learning (RL). Meanwhile, manually curating non-trivially retrievable questions -- where answers cannot be found through a single direct query but instead require multi-hop reasoning over oblique and loosely connected evidence -- incurs prohibitive human costs and fails to scale, creating a critical data bottleneck for training high-capability retrieval-and-reasoning agents. To address this, we present BMGQ, a bottom-up automated method for generating high-difficulty, training-ready multi-hop questions from semi-structured knowledge sources. The BMGQ system (i) grows diverse, logically labeled evidence clusters through Natural Language Inference (NLI)-based relation typing and diversity-aware expansion; (ii) applies reverse question construction to compose oblique cues so that isolated signals are underinformative but their combination uniquely identifies the target entity; and (iii) enforces quality with a two-step evaluation pipeline that combines multi-model consensus filtering with structured constraint decomposition and evidence-based matching. The result is a scalable process that yields complex, retrieval-resistant yet verifiable questions suitable for SFT/RL training as well as challenging evaluation, substantially reducing human curation effort while preserving the difficulty profile of strong evaluation benchmarks.
Authors Bingsen Qiu, Zijian Liu, Xiao Liu, Bingjie Wang, Feier Zhang, Yixuan Qin, Chunyan Li, Haoshen Yang, Zeren Gao
Categories Artificial Intelligence
3
"Are We Done Yet?": A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents AAAI 2026 Workshop on Trust and Control in Agentic AI (TrustAgent)
Title CN 我们完成任务了吗？一种用于计算机使用代理自主任务完成的基于视觉的判断器
Keywords 计算机使用代理 · 视觉语言模型 · 任务完成判断 · 自主评估 · GUI交互
Summary 本文提出了一种基于视觉语言模型的评估框架，通过分析屏幕截图和任务描述来判断计算机使用代理是否完成了指定任务。实验覆盖42个macOS应用和1260个人工标注任务，在任务成功检测上达到73%的准确率，并能带来27%的相对性能提升。该框架作为反馈机制增强了代理的自纠错能力。
Reason 该论文属于具身智能/机器人代理在数字界面操作中的应用，研究的是通用计算机使用代理（CUA）的任务完成判断问题，其核心场景是GUI交互而非文档理解。尽管使用了视觉语言模型和截图进行判断，但任务本质与文档图像理解（如DocVQA、布局分析、表格识别等）无关，且未提出可迁移至DIU领域的底层方法（如视觉压缩、去幻觉机制、细粒度定位等）。属于无关领域应用，不符合关注方向。
Abstract Computer Use Agents (CUAs) are designed to autonomously operate digital interfaces, yet they often fail to reliably determine whether a given task has been completed. We present an autonomous evaluation and feedback framework that uses vision-language models to assess task completion directly from screenshots and task descriptions. Our dataset covers 42 built-in macOS applications and 1,260 human-labeled tasks across a wide range of scenarios. Our framework achieves up to 73 percent accuracy in task success detection and yields an average relative improvement of 27 percent in overall task success when evaluator feedback is applied. These results show that vision-based evaluation can serve as an effective feedback mechanism that improves the reliability and self-correction of autonomous computer-use agents.
Authors Marta Sumyk, Oleksandr Kosovan
Comment This work has been accepted to appear at the AAAI 2026 Workshop on Trust and Control in Agentic AI (TrustAgent)
Categories Artificial Intelligence
3
Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation
Title CN Ming-Flash-Omni：一种用于多模态感知与生成的稀疏统一架构
Keywords 多模态大模型 · MoE · 语音识别 · 图像生成 · 生成式分割
Summary 本文提出了Ming-Flash-Omni，一个基于稀疏专家混合（MoE）的百亿参数多模态模型，每token仅激活61亿参数，支持跨视觉、语音和语言的高效统一智能。模型在上下文语音识别、方言识别、文本到图像生成及生成式分割上取得SOTA表现，并引入高保真文本渲染与更强的编辑一致性。然而，所有实验均围绕通用多模态生成与语音展开，未涉及文档图像理解任务。
Reason 该论文聚焦于通用多模态大模型在视觉、语音和语言上的统一架构设计，强调图像生成、语音识别和生成式分割等任务。尽管其MoE架构和高分辨率处理能力理论上可能对文档理解有间接帮助，但全文未提及任何文档图像理解（DIU）相关任务（如DocVQA、表格识别、布局分析等），也未针对OCR幻觉、密集文本或复杂排版等DIU核心痛点进行建模改进。属于典型的‘平行应用’类工作——将现有VLM架构扩展到多模态生成与语音领域，而非为文档理解提供方法论创新。
Abstract We propose Ming-Flash-Omni, an upgraded version of Ming-Omni, built upon a sparser Mixture-of-Experts (MoE) variant of Ling-Flash-2.0 with 100 billion total parameters, of which only 6.1 billion are active per token. This architecture enables highly efficient scaling (dramatically improving computational efficiency while significantly expanding model capacity) and empowers stronger unified multimodal intelligence across vision, speech, and language, representing a key step toward Artificial General Intelligence (AGI). Compared to its predecessor, the upgraded version exhibits substantial improvements across multimodal understanding and generation. We significantly advance speech recognition capabilities, achieving state-of-the-art performance in contextual ASR and highly competitive results in dialect-aware ASR. In image generation, Ming-Flash-Omni introduces high-fidelity text rendering and demonstrates marked gains in scene consistency and identity preservation during image editing. Furthermore, Ming-Flash-Omni introduces generative segmentation, a capability that not only achieves strong standalone segmentation performance but also enhances spatial control in image generation and improves editing consistency. Notably, Ming-Flash-Omni achieves state-of-the-art results in text-to-image generation and generative segmentation, and sets new records on all 12 contextual ASR benchmarks, all within a single unified architecture.
Authors Inclusion AI, :, Bowen Ma, Cheng Zou, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Chenyu Lian, Dandan Zheng, Fudong Wang, Furong Xu, GuangMing Yao, Jun Zhou, Jingdong Chen, Jianing Li, Jianxin Sun, Jiajia Liu, Jian Sha, Jianjiang Zhu, Jianping Jiang, Jun Peng, Kaixiang Ji, Kaimeng Ren, Libin Wang, Lixiang Ru, Longhua Tan, Lu Ma, Lan Wang, Mochen Bai, Ning Gao, Qingpei Guo, Qinglong Zhang, Qiang Xu, Rui Liu, Ruijie Xiong, Ruobing Zheng, Sirui Gao, Tao Zhang, Tianqi Li, Tinghao Liu, Weilong Chai, Xinyu Xiao, Xiaomei Wang, Xiaolong Wang, Xiao Lu, Xiaoyu Li, Xingning Dong, Xuzheng Yu, Yi Yuan, Yuting Gao, Yuting Xiao, Yunxiao Sun, Yipeng Chen, Yifan Mao, Yifei Wu, Yongjie Lyu, Ziping Ma, Zhiqiang Fang, Zhihao Qiu, Ziyuan Huang, Zizheng Yang, Zhengyu He
Comment 18 pages, 5 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Directional Optimization Asymmetry in Transformers: A Synthetic Stress Test
Title CN Transformer中的方向性优化不对称：一种合成压力测试
Keywords Transformer · 方向性偏差 · 合成基准 · 序列建模 · 优化不对称
Summary 论文通过构造无语义、熵可控的合成字符串映射任务，测试Transformer在前向和逆向学习中的性能差异，发现即使去除语言先验后仍存在显著的方向性优化劣势，表明这种偏差源于架构本身而非数据分布。实验涵盖GPT-2、MLP和LoRA等设置，揭示了因果注意力机制在逆向建模上的固有困难。
Reason 该论文研究Transformer架构在合成任务中的方向性学习偏差（如前向与逆向映射的性能差异），属于基础序列建模机制的理论分析。尽管其发现的‘方向性优化间隙’对理解LLM的时间不对称性有启发意义，但完全脱离视觉或多模态上下文，未涉及文档结构、视觉布局或图文对齐等DIU核心问题。也无法直接迁移用于解决OCR幻觉、细粒度定位或高分辨率文档处理等实际挑战，属于纯语言模型内部机制的探讨论文，与文档图像理解无实质关联。
Abstract Transformers are theoretically reversal-invariant: their function class does not prefer left-to-right over right-to-left mappings. Yet empirical studies on natural language repeatedly report a "reversal curse," and recent work on temporal asymmetry in LLMs suggests that real-world corpora carry their own arrow of time. This leaves an unresolved question: do directional failures stem from linguistic statistics, or from the architecture itself? We cut through this ambiguity with a fully synthetic, entropy-controlled benchmark designed as a clean-room stress test for directional learning. Using random string mappings with tunable branching factor K, we construct forward tasks with zero conditional entropy and inverse tasks with analytically determined entropy floors. Excess loss above these floors reveals that even scratch-trained GPT-2 models exhibit a strong, reproducible directional optimization gap (e.g., 1.16 nats at K=5), far larger than that of an MLP trained on the same data. Pre-trained initializations shift optimization behavior but do not eliminate this gap, while LoRA encounters a sharp capacity wall on high-entropy inverse mappings. Together, these results isolate a minimal, semantics-free signature of directional friction intrinsic to causal Transformer training-one that persists even when linguistic priors, token frequencies, and corpus-level temporal asymmetries are removed. Our benchmark provides a controlled instrument for dissecting directional biases in modern sequence models and motivates deeper mechanistic study of why inversion remains fundamentally harder for Transformers.
Authors Mihir Sahasrabudhe
Comment 19 pages, 4 figures. Code available at https://github.com/mihirs-0/synass
Categories Artificial Intelligence, Computation and Language
3
On-Demand Multi-Task Sparsity for Efficient Large-Model Deployment on Edge Devices
Title CN 面向边缘设备上大模型部署的按需多任务稀疏化
Keywords 模型稀疏化 · 边缘计算 · 多任务学习 · 参数重用 · 高效部署
Summary 本文提出一种按需多任务稀疏化框架，通过将模型权重分解为可重用的块粒度单元，并对齐不同任务间的稀疏结构以最大化参数重叠，从而在任务切换时仅加载差异部分的小型块集，显著减少冷启动延迟。在真实自动驾驶平台上实现了平均6.6倍的任务切换加速。
Reason 该论文研究的是在边缘设备上进行多任务切换时的模型稀疏化与参数复用，以降低I/O开销和冷启动延迟。尽管其在高效部署方面有贡献，但属于模型压缩与系统优化范畴，且实验场景为自动驾驶平台，并未涉及文档图像理解（DIU）中的核心问题如OCR幻觉、复杂版式解析或视觉-语言推理。同时，该工作是典型的‘平行应用’于边缘计算场景下的多任务部署，缺乏对DIU领域可迁移的方法论创新。
Abstract Sparsity is essential for deploying large models on resource constrained edge platforms. However, optimizing sparsity patterns for individual tasks in isolation ignores the significant I/O overhead incurred during frequent task switching. We introduce an on-demand multi-task sparsity framework specifically designed to minimize switching costs by maximizing parameter reuse. Unlike monolithic approaches, we decompose weights into reusable block-granular units and align sparse structures across tasks to maximize overlap. By dynamically loading only the small differential set of blocks required for the next task, our method effectively mitigates the cold-start latency inherent in traditional monolithic approaches.Experiments on a real-world autonomous driving platform demonstrate that our framework achieves superior switching efficiency, accelerating task switching by over 6.6X on average compared to existing sparsity methods.
Authors Lianming Huang, Haibo Hu, Qiao Li, Nan Guan, Chun Jason Xue
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
M$^3$Prune: Hierarchical Communication Graph Pruning for Efficient Multi-Modal Multi-Agent Retrieval-Augmented Generation
Title CN M³Prune：用于高效多模态多智能体检索增强生成的分层通信图剪枝
Keywords 多智能体系统 · 检索增强生成 · 图剪枝 · 通信效率 · 多模态
Summary 本文提出M³Prune，一种用于多模态检索增强生成的分层通信图剪枝框架，通过模态内稀疏化和动态跨模态拓扑构建来减少多智能体系统中的冗余通信，从而降低token开销并提升效率。实验显示其在通用和特定领域mRAG基准上优于单智能体和其他多智能体系统。
Reason 该论文属于典型的‘平行应用’类型，其核心是将多智能体系统应用于mRAG场景并优化通信开销。尽管提出了图剪枝机制以减少token消耗，但其方法框架仍局限于多智能体协作系统的内部效率优化，并未针对文档图像理解（DIU）中的核心痛点如OCR幻觉、复杂排版解析、细粒度定位或长文档推理进行设计。所提技术难以直接迁移至DIU领域解决实际问题，且无证据表明其能提升VLM在文档任务上的感知或推理能力。
Abstract Recent advancements in multi-modal retrieval-augmented generation (mRAG), which enhance multi-modal large language models (MLLMs) with external knowledge, have demonstrated that the collective intelligence of multiple agents can significantly outperform a single model through effective communication. Despite impressive performance, existing multi-agent systems inherently incur substantial token overhead and increased computational costs, posing challenges for large-scale deployment. To address these issues, we propose a novel Multi-Modal Multi-agent hierarchical communication graph PRUNING framework, termed M$^3$Prune. Our framework eliminates redundant edges across different modalities, achieving an optimal balance between task performance and token overhead. Specifically, M$^3$Prune first applies intra-modal graph sparsification to textual and visual modalities, identifying the edges most critical for solving the task. Subsequently, we construct a dynamic communication topology using these key edges for inter-modal graph sparsification. Finally, we progressively prune redundant edges to obtain a more efficient and hierarchical topology. Extensive experiments on both general and domain-specific mRAG benchmarks demonstrate that our method consistently outperforms both single-agent and robust multi-agent mRAG systems while significantly reducing token consumption.
Authors Weizi Shao, Taolin Zhang, Zijie Zhou, Chen Chen, Chengyu Wang, Xiaofeng He
Categories Artificial Intelligence
3
AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
Title CN AReaL：一种用于语言推理的大规模异步强化学习系统
Keywords 强化学习 · 大语言模型 · 异步训练 · 系统优化 · 训练效率
Summary AReaL提出了一种完全异步的强化学习系统，解耦了生成与训练过程，通过持续生成输出和动态收集数据实现更高的GPU利用率。系统采用负载均衡策略控制数据陈旧性，并设计了支持陈旧性的PPO变体来稳定训练。实验表明其在数学和代码推理任务上相比同步系统有显著速度提升且性能相当或更优。
Reason 该论文聚焦于大语言模型（LLM）在数学与代码推理任务上的强化学习训练系统优化，提出了一种异步架构以提升GPU利用率和训练效率。尽管其技术可能对长文档多步推理中的Agent训练有潜在加速作用，但整体工作属于底层训练系统工程优化，并未直接针对文档图像理解（DIU）中的核心问题如OCR幻觉、复杂排版解析或视觉-文本对齐等进行方法创新。且其应用场景为纯文本语言推理，未涉及多模态输入或视觉信息处理，迁移至DIU场景的路径不明确。
Abstract Reinforcement learning (RL) has become a dominant paradigm for training large language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs requires massive parallelization and poses an urgent need for efficient training systems. Most existing large-scale RL systems for LLMs are synchronous, alternating generation and training in a batch setting where rollouts in each training batch are generated by the same model. This approach stabilizes RL training but suffers from severe system-level inefficiency: generation must wait until the longest output in the batch is completed before model updates, resulting in GPU underutilization. We present AReaL, a fully asynchronous RL system that completely decouples generation from training. Rollout workers in AReaL continuously generate new outputs without waiting, while training workers update the model whenever a batch of data is collected. AReaL also incorporates a collection of system-level optimizations, leading to substantially higher GPU utilization. To stabilize RL training, AReaL balances the workload of rollout and training workers to control data staleness, and adopts a staleness-enhanced PPO variant to better handle outdated training samples. Extensive experiments on math and code reasoning benchmarks show that AReaL achieves up to 2.77$\times$ training speedup compared to synchronous systems with the same number of GPUs and matched or improved final performance. The code of AReaL is available at https://github.com/inclusionAI/AReaL/.
Authors Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, Yi Wu
Categories Artificial Intelligence
3
Addressing divergent representations from causal interventions on neural networks
Title CN 解决神经网络因果干预中的发散表征问题
Keywords 因果干预 · 表征偏移 · 模型可解释性 · 干预可靠性 · 反事实损失
Summary 本文从理论和实验上证明，对神经网络进行因果干预常会导致内部表征偏离原始数据分布，进而影响可解释性的可靠性。作者区分了‘无害’和‘有害’的表征偏移，并通过改进自Grant (2025) 的反事实潜在损失（CL loss）来缓解有害偏移，在保持解释能力的同时使干预后的表征更贴近自然状态。
Reason 该论文研究的是因果干预在模型可解释性中引发的表征偏移问题，属于机制可解释性（mechanistic interpretability）领域的理论分析，虽然涉及VLM可能使用的干预方法，但其核心并未针对文档图像理解任务（如DocVQA、布局分析等），也没有提出可用于提升文档理解性能的具体架构、训练策略或推理方法。其讨论的‘干预后表征发散’问题虽理论上可能影响DIU中的干预式推理（Intervention/Steering），但论文本身未建立与文档模态、多模态对齐、OCR幻觉或细粒度定位等实际痛点的联系，属于通用深度学习理论，迁移价值极低。
Abstract A common approach to mechanistic interpretability is to causally manipulate model representations via targeted interventions in order to understand what those representations encode. Here we ask whether such interventions create out-of-distribution (divergent) representations, and whether this raises concerns about how faithful their resulting explanations are to the target model in its natural state. First, we demonstrate theoretically and empirically that common causal intervention techniques often do shift internal representations away from the natural distribution of the target model. Then, we provide a theoretical analysis of two classes of such divergences: "harmless" divergences that occur in the null-space of the weights and from covariance within behavioral decision boundaries, and "pernicious" divergences that activate hidden network pathways and cause dormant behavioral changes. Finally, in an effort to mitigate the pernicious cases, we apply and modify the Counterfactual Latent (CL) loss from Grant (2025) allowing representations from causal interventions to remain closer to the natural distribution, reducing the likelihood of harmful divergences while preserving the interpretive power of the interventions. Together, these results highlight a path towards more reliable interpretability methods.
Authors Satchel Grant, Simon Jerome Han, Alexa R. Tartaglini, Christopher Potts
Categories Artificial Intelligence
3
Understanding and Optimizing Multi-Stage AI Inference Pipelines
Title CN 理解和优化多阶段AI推理流水线
Keywords LLM推理 · 多阶段流水线 · 异构系统 · 推理仿真 · 硬件协同设计
Summary 本文提出了HERMES，一个面向多阶段大语言模型推理的异构执行仿真器，支持RAG、KV缓存检索、推理、prefill和decode等阶段在复杂硬件层级中的建模。通过结合真实硬件迹线与分析建模，HERMES能够捕捉内存带宽争用、跨集群通信延迟和批处理效率等关键权衡，帮助系统设计者优化下一代AI工作负载的软硬件协同设计。
Reason 该论文聚焦于LLM推理系统的工程优化，提出了一种用于模拟异构多阶段AI推理流程的仿真器HERMES，属于系统架构与推理基础设施工作。尽管其涉及RAG、KV缓存、多步推理等环节，但核心贡献在于系统级调度与硬件协同设计，并未针对文档图像理解中的OCR幻觉、细粒度定位、复杂版面解析或多模态对齐等关键问题提供可迁移的方法论。其技术路径偏向分布式计算与性能建模，难以直接应用于DIU任务的模型层面改进。
Abstract The rapid evolution of Large Language Models (LLMs) has driven the need for increasingly sophisticated inference pipelines and hardware platforms. Modern LLM serving extends beyond traditional prefill-decode workflows, incorporating multi-stage processes such as Retrieval Augmented Generation (RAG), key-value (KV) cache retrieval, dynamic model routing, and multi step reasoning. These stages exhibit diverse computational demands, requiring distributed systems that integrate GPUs, ASICs, CPUs, and memory-centric architectures. However, existing simulators lack the fidelity to model these heterogeneous, multi-engine workflows, limiting their ability to inform architectural decisions. To address this gap, we introduce HERMES, a Heterogeneous Multi-stage LLM inference Execution Simulator. HERMES models diverse request stages; including RAG, KV retrieval, reasoning, prefill, and decode across complex hardware hierarchies. HERMES supports heterogeneous clients executing multiple models concurrently unlike prior frameworks while incorporating advanced batching strategies and multi-level memory hierarchies. By integrating real hardware traces with analytical modeling, HERMES captures critical trade-offs such as memory bandwidth contention, inter-cluster communication latency, and batching efficiency in hybrid CPU-accelerator deployments. Through case studies, we explore the impact of reasoning stages on end-to-end latency, optimal batching strategies for hybrid pipelines, and the architectural implications of remote KV cache retrieval. HERMES empowers system designers to navigate the evolving landscape of LLM inference, providing actionable insights into optimizing hardware-software co-design for next-generation AI workloads.
Authors Abhimanyu Rajeshkumar Bambhaniya, Hanjiang Wu, Suvinay Subramanian, Sudarshan Srinivasan, Souvik Kundu, Amir Yazdanbakhsh, Midhilesh Elavazhagan, Madhu Kumar, Tushar Krishna
Comment Inference System Design for Multi-Stage AI Inference Pipelines. 13 Pages, 15 Figues, 3 Tables
Categories Artificial Intelligence
3
RPM-MCTS: Knowledge-Retrieval as Process Reward Model with Monte Carlo Tree Search for Code Generation AAAI 2026
Title CN RPM-MCTS：将知识检索作为过程奖励模型结合蒙特卡洛树搜索用于代码生成
Keywords 代码生成 · 蒙特卡洛树搜索 · 过程奖励模型 · 知识检索 · 推理优化
Summary 该论文提出RPM-MCTS，通过知识检索构建过程奖励模型，结合蒙特卡洛树搜索来提升大模型在代码生成中的推理能力。利用相似性过滤减少冗余推理路径，并借助沙箱执行反馈定位错误步骤以进行修正。实验表明其在多个基准上优于现有方法，同时降低约15%的token消耗。
Reason 该论文属于代码生成领域的下游应用研究，核心贡献是将知识检索与MCTS结合优化代码生成的推理路径。尽管其引入了过程奖励建模和错误定位机制，但任务场景与文档图像理解（DIU）差异显著，且未涉及视觉输入、多模态对齐或OCR相关问题。虽然MCTS和过程干预思想在理论上可能启发长文档推理中的路径搜索，但本文方法高度依赖代码执行反馈（sandbox execution），难以迁移到非程序性、非可执行的文档理解任务中。属于典型的‘平行应用’，无直接方法论迁移价值。
Abstract Tree search-based methods have made significant progress in enhancing the code generation capabilities of large language models. However, due to the difficulty in effectively evaluating intermediate algorithmic steps and the inability to locate and timely correct erroneous steps, these methods often generate incorrect code and incur increased computational costs. To tackle these problems, we propose RPM-MCTS, an effective method that utilizes Knowledge-Retrieval as Process Reward Model based on Monte Carlo Tree Search to evaluate intermediate algorithmic steps. By utilizing knowledge base retrieval, RPM-MCTS avoids the complex training of process reward models. During the expansion phase, similarity filtering is employed to remove redundant nodes, ensuring diversity in reasoning paths. Furthermore, our method utilizes sandbox execution feedback to locate erroneous algorithmic steps during generation, enabling timely and targeted corrections. Extensive experiments on four public code generation benchmarks demonstrate that RPM-MCTS outperforms current state-of-the-art methods while achieving an approximately 15% reduction in token consumption. Furthermore, full fine-tuning of the base model using the data constructed by RPM-MCTS significantly enhances its code capabilities.
Authors Yuanyuan Lin, Xiangyu Ouyang, Teng Zhang, Kaixin Sui
Comment Accepted at AAAI 2026
Categories Artificial Intelligence
3
Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator NeurIPS 2025
Title CN 你的预训练大语言模型其实是一个秘密的无监督置信度校准器
Keywords 置信度校准 · 大语言模型 · 后训练 · 无监督 · 预测分歧
Summary 论文指出经过后训练的大语言模型（PoLM）常出现置信度过高的问题，影响可靠性。为此提出DACA方法，利用预训练语言模型（PLM）与PoLM之间的预测一致性，在无标注数据的情况下仅使用两者预测一致的样本来进行温度缩放校准，从而避免因预测分歧导致的校准偏差。实验表明该方法能显著降低多种LLM的预期校准误差（ECE）。
Reason 该论文研究的是大语言模型在后训练过程中产生的置信度过高问题，并提出一种无监督的置信度校准方法DACA。尽管置信度校准对文档理解中的幻觉问题有一定潜在关联，但本文完全基于纯文本NLP场景，未涉及任何视觉输入、多模态对齐或文档结构建模。其方法迁移至文档图像理解（DIU）领域需额外大量假设，且不直接解决OCR幻觉、细粒度定位或复杂排版等核心痛点，属于通用NLP技术，不具备针对DIU的方法论创新。
Abstract Post-training of large language models is essential for adapting pre-trained language models (PLMs) to align with human preferences and downstream tasks. While PLMs typically exhibit well-calibrated confidence, post-trained language models (PoLMs) often suffer from over-confidence, assigning high confidence to both correct and incorrect outputs, which can undermine reliability in critical applications. A major obstacle in calibrating PoLMs is the scarcity of labeled data for individual downstream tasks. To address this, we propose Disagreement-Aware Confidence Alignment (DACA), a novel unsupervised method to optimize the parameters (e.g., temperature $τ$) in post-hoc confidence calibration. Our method is motivated by the under-confidence issue caused by prediction disagreement between the PLM and PoLM while aligning their confidence via temperature scaling. Theoretically, the PLM's confidence underestimates PoLM's prediction accuracy on disagreement examples, causing a larger $τ$ and producing under-confident predictions. DACA mitigates this by selectively using only agreement examples for calibration, effectively decoupling the influence of disagreement. In this manner, our method avoids an overly large $τ$ in temperature scaling caused by disagreement examples, improving calibration performance. Extensive experiments demonstrate the effectiveness of our method, improving the average ECE of open-sourced and API-based LLMs (e.g. GPT-4o) by up to 15.08$\%$ on common benchmarks.
Authors Beier Luo, Shuoyuan Wang, Sharon Li, Hongxin Wei
Comment NeurIPS 2025
Categories Artificial Intelligence
3
Reinforcement Learning with $ω$-Regular Objectives and Constraints
Title CN 具有ω-正则目标和约束的强化学习
Keywords 强化学习 · 形式化方法 · 时序逻辑 · 安全约束 · 线性规划
Summary 本文提出一种基于ω-正则目标和约束的强化学习方法，通过线性规划求解满足复杂行为规范（如时序逻辑）的同时满足安全约束的策略。重点在于将高阶行为要求形式化为可计算的优化问题，并提供理论保证。然而，该工作未涉及任何视觉或文档模态，也未处理现实感知输入。
Reason 该论文研究的是强化学习中使用ω-正则逻辑表达复杂行为目标与安全约束的理论框架，属于形式化方法与RL的结合。尽管其在安全性与长期行为规范方面有理论价值，但完全不涉及多模态、视觉输入或文档理解任务，也无法直接迁移用于解决文档图像理解中的OCR幻觉、细粒度定位或复杂排版解析等核心问题。属于通用RL理论工作，与DIU无实质关联。
Abstract Reinforcement learning (RL) commonly relies on scalar rewards with limited ability to express temporal, conditional, or safety-critical goals, and can lead to reward hacking. Temporal logic expressible via the more general class of $ω$-regular objectives addresses this by precisely specifying rich behavioural properties. Even still, measuring performance by a single scalar (be it reward or satisfaction probability) masks safety-performance trade-offs that arise in settings with a tolerable level of risk. We address both limitations simultaneously by combining $ω$-regular objectives with explicit constraints, allowing safety requirements and optimisation targets to be treated separately. We develop a model-based RL algorithm based on linear programming, which in the limit produces a policy maximising the probability of satisfying an $ω$-regular objective while also adhering to $ω$-regular constraints within specified thresholds. Furthermore, we establish a translation to constrained limit-average problems with optimality-preserving guarantees.
Authors Dominik Wagner, Leon Witzman, Luke Ong
Categories Artificial Intelligence
3
MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmark NeurIPS 2025
Title CN MMTU：一个大规模多任务表格理解与推理基准
Keywords 表格理解 · 基准测试 · 多任务学习 · 结构化数据 · LLM评测
Summary 本文提出了MMTU，一个包含超过28,000个问题的大规模多任务表格理解与推理基准，涵盖来自数据库、电子表格等场景的25种真实任务。实验表明当前前沿模型（如GPT-5和DeepSeek R1）在此基准上表现有限，揭示了现有模型在处理复杂表格任务上的不足。作者希望借此推动结构化数据处理领域的发展。
Reason 该论文提出的是一个面向表格理解与推理的基准测试集MMTU，包含25种现实世界中的表格任务，旨在评估模型在专家级表格操作上的综合能力。尽管任务设计复杂且具有挑战性，但其本质是一个评测基准（Benchmark），而非方法论创新。它没有提出新的文档图像理解架构、训练策略或解决OCR幻觉、细粒度定位等DIU核心痛点的技术路径。属于典型的‘平行下游应用’类工作——即构建特定领域的评估任务，而非推动底层技术进步。
Abstract Tables and table-based use cases play a crucial role in many important real-world applications, such as spreadsheets, databases, and computational notebooks, which traditionally require expert-level users like data engineers, data analysts, and database administrators to operate. Although LLMs have shown remarkable progress in working with tables (e.g., in spreadsheet and database copilot scenarios), comprehensive benchmarking of such capabilities remains limited. In contrast to an extensive and growing list of NLP benchmarks, evaluations of table-related tasks are scarce, and narrowly focus on tasks like NL-to-SQL and Table-QA, overlooking the broader spectrum of real-world tasks that professional users face. This gap limits our understanding and model progress in this important area. In this work, we introduce MMTU, a large-scale benchmark with over 28K questions across 25 real-world table tasks, designed to comprehensively evaluate models ability to understand, reason, and manipulate real tables at the expert-level. These tasks are drawn from decades' worth of computer science research on tabular data, with a focus on complex table tasks faced by professional users. We show that MMTU require a combination of skills -- including table understanding, reasoning, and coding -- that remain challenging for today's frontier models, where even frontier reasoning models like OpenAI GPT-5 and DeepSeek R1 score only around 69\% and 57\% respectively, suggesting significant room for improvement. We highlight key findings in our evaluation using MMTU and hope that this benchmark drives further advances in understanding and developing foundation models for structured data processing and analysis. Our code and data are available at https://github.com/MMTU-Benchmark/MMTU and https://huggingface.co/datasets/MMTU-benchmark/MMTU.
Authors Junjie Xing, Yeye He, Mengyu Zhou, Haoyu Dong, Shi Han, Lingjiao Chen, Dongmei Zhang, Surajit Chaudhuri, H. V. Jagadish
Comment Accepted at NeurIPS 2025; Code and data available at https://github.com/MMTU-Benchmark/MMTU and https://huggingface.co/datasets/MMTU-benchmark/MMTU
Categories Artificial Intelligence, Computation and Language
3
A Unified Evaluation-Instructed Framework for Query-Dependent Prompt Optimization
Title CN 一种统一的、由评估指导的查询依赖型提示优化框架
Keywords 提示优化 · 评估器 · 可解释性 · 查询依赖 · 执行前预测
Summary 本文提出一种统一的提示评估与优化框架，通过训练一个无需执行即可预测提示质量的评估模型，实现对查询依赖型提示的可解释优化。该方法在多个数据集和骨干模型上优于静态模板和其他查询依赖方法，强调评估信号的系统性与可解释性。
Reason 该论文聚焦于通用提示词优化与评估框架，核心贡献是构建一个可解释的、基于多维评分的执行前评估器来指导查询依赖的提示优化。尽管其方法在提升LLM推理稳定性方面有一定价值，但属于典型的‘平行应用’类技术包装：它并未针对文档图像理解中的具体痛点（如OCR幻觉、复杂版式建模、细粒度视觉定位或高分辨率处理）提出解决方案，也未涉及多模态输入或视觉-语言对齐机制。其所提方法难以直接迁移至DIU场景，因文档理解的关键瓶颈不在文本提示优化，而在视觉表征质量与跨模态对齐精度。
Abstract Most prompt-optimization methods refine a single static template, making them ineffective in complex and dynamic user scenarios. Existing query-dependent approaches rely on unstable textual feedback or black-box reward models, providing weak and uninterpretable optimization signals. More fundamentally, prompt quality itself lacks a unified, systematic definition, resulting in fragmented and unreliable evaluation signals. Our approach first establishes a performance-oriented, systematic, and comprehensive prompt evaluation framework. Furthermore, we develop and finetune an execution-free evaluator that predicts multi-dimensional quality scores directly from text. The evaluator then instructs a metric-aware optimizer that diagnoses failure modes and rewrites prompts in an interpretable, query-dependent manner. Our evaluator achieves the strongest accuracy in predicting prompt performance, and the evaluation-instructed optimization consistently surpass both static-template and query-dependent baselines across eight datasets and on three backbone models. Overall, we propose a unified, metric-grounded perspective on prompt quality, and demonstrated that our evaluation-instructed optimization pipeline delivers stable, interpretable, and model-agnostic improvements across diverse tasks.
Authors Ke Chen, Yifeng Wang, Hassan Almosapeeh, Haohan Wang
Categories Artificial Intelligence
3
Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models
Title CN 马赛克剪枝：一种用于稀疏专家混合模型的可泛化剪枝分层框架
Keywords 模型剪枝 · 专家混合模型 · MoE · 模型压缩 · 跨域泛化
Summary 本文提出Mosaic Pruning（MoP），一种面向稀疏专家混合模型（SMoE）的层次化剪枝框架。通过‘聚类-选择’策略，基于专家在不同任务上的表现相似性进行功能聚类，并使用激活变异分数选取代表性专家，从而保留功能互补的专家子集，提升剪枝后模型在通用与特定任务（如数学推理、代码生成）上的跨域泛化能力。实验表明其在多个MoE模型上优于现有剪枝方法。
Reason 该论文研究的是MoE大模型的剪枝方法，核心贡献在于提升剪枝后模型在跨域任务上的泛化能力。尽管其‘cluster-then-select’策略和Activation Variability Score具有一定方法论价值，但整体工作属于模型压缩与高效推理领域，且聚焦于纯语言模型的专家选择机制。未涉及视觉输入、多模态对齐或文档结构理解等DIU核心问题。虽然理论上模型压缩有助于部署长文档VLM系统，但该方法并非针对文档理解中的OCR幻觉、高分辨率处理或细粒度定位等痛点设计，迁移路径不直接，属于通用模型优化技术。
Abstract Sparse Mixture-of-Experts (SMoE) architectures have enabled a new frontier in scaling Large Language Models (LLMs), offering superior performance by activating only a fraction of their total parameters during inference. However, their practical deployment is severely hampered by substantial static memory overhead, as all experts must be loaded into memory. Existing post-training pruning methods, while reducing model size, often derive their pruning criteria from a single, general-purpose corpus. This leads to a critical limitation: a catastrophic performance degradation when the pruned model is applied to other domains, necessitating a costly re-pruning for each new domain. To address this generalization gap, we introduce Mosaic Pruning (MoP). The core idea of MoP is to construct a functionally comprehensive set of experts through a structured ``cluster-then-select" process. This process leverages a similarity metric that captures expert performance across different task domains to functionally cluster the experts, and subsequently selects the most representative expert from each cluster based on our proposed Activation Variability Score. Unlike methods that optimize for a single corpus, our proposed Mosaic Pruning ensures that the pruned model retains a functionally complementary set of experts, much like the tiles of a mosaic that together form a complete picture of the original model's capabilities, enabling it to handle diverse downstream tasks.Extensive experiments on various MoE models demonstrate the superiority of our approach. MoP significantly outperforms prior work, achieving a 7.24\% gain on general tasks and 8.92\% on specialized tasks like math reasoning and code generation.
Authors Wentao Hu, Mingkuan Zhao, Shuangyong Song, Xiaoyan Zhu, Xin Lai, Jiayin Wang
Categories Artificial Intelligence
3
Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning
Title CN 先学规则，再信胜利：用于代理强化学习的渐进探索式自我模仿
Keywords 强化学习 · 代理系统 · 自我模仿学习 · 探索-利用平衡 · LLM训练
Summary 本文提出SPEAR，一种结合渐进探索与自我模仿学习的强化学习框架，用于提升LLM代理在长周期任务中的表现。通过课程化调度内在奖励和经验回放，平衡探索与利用，在多个文本型交互环境中显著提升成功率。然而，该工作未涉及视觉或文档模态，属于通用Agent训练策略，对文档理解领域无直接方法论贡献。
Reason 该论文属于典型的Agent应用框架研究，聚焦于通用长视野、稀疏奖励任务中的探索-利用权衡问题，尽管其方法（SPEAR）在ALFWorld、WebShop等基准上提升了成功率，但其核心贡献是强化学习策略优化的‘下游封装’，并未针对文档图像理解中的具体痛点（如OCR幻觉、复杂布局解析、细粒度定位或高分辨率处理）提出可迁移的底层技术创新。虽然‘渐进探索’思想理论上可用于长文档阅读路径规划，但文中未涉及视觉输入处理或多模态对齐机制，迁移价值极低。
Abstract Reinforcement learning (RL) is the dominant paradigm for sharpening strategic tool use capabilities of LLMs on long-horizon, sparsely-rewarded agent tasks, yet it faces a fundamental challenge of exploration-exploitation trade-off. Existing studies stimulate exploration through the lens of policy entropy, but such mechanical entropy maximization is prone to RL instability due to the multi-turn distribution shifting. In this paper, we target the progressive exploration-exploitation balance under the guidance of the agent's own experiences without succumbing to either entropy collapsing or runaway divergence. We propose SPEAR, a self-imitation learning (SIL) recipe for training agentic LLMs. It extends the vanilla SIL, where a replay buffer stores good experience for off-policy update, by gradually steering the policy entropy across stages. Specifically, the proposed curriculum scheduling harmonizes intrinsic reward shaping and self-imitation to 1) expedite exploration via frequent tool interactions at the beginning, and 2) strengthen exploitation of successful tactics upon convergence towards familiarity with the environment. We also combine bag-of-tricks of industrial RL optimizations for a strong baseline Dr.BoT to demonstrate our effectiveness. In ALFWorld and WebShop, SPEAR increases the success rates of GRPO/GiGPO/Dr.BoT by up to 16.1%/5.1%/8.6% and 20.7%/11.8%/13.9%, respectively. In AIME24 and AIME25, SPEAR boosts Dr.BoT by up to 3.8% and 6.1%, respectively. Such gains incur only 10%-25% extra theoretical complexity and negligible runtime overhead in practice, demonstrating the plug-and-play scalability of SPEAR.
Authors Yulei Qin, Xiaoyu Tan, Zhengbao He, Gang Li, Haojia Lin, Zongyi Li, Zihan Xu, Yuchen Shi, Siqi Cai, Renting Rui, Shaofei Cai, Yuzheng Cai, Xuan Zhang, Sheng Ye, Ke Li, Xing Sun
Comment 45 pages, 14 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
3
Learning to Clean: Reinforcement Learning for Noisy Label Correction NeurIPS 2025
Title CN 学习去清洗：用于噪声标签校正的强化学习
Keywords 噪声标签 · 强化学习 · 标签校正 · 鲁棒学习 · 深度学习
Summary 本文提出一种基于强化学习的噪声标签校正框架RLNLC，将标签校正建模为强化学习问题，通过状态空间、动作空间和奖励机制来学习最优校正策略，并利用actor-critic方法训练策略网络。实验表明该方法在多个基准数据集上优于现有技术。
Reason 该论文研究的是通用噪声标签校正问题，属于基础机器学习鲁棒性方向。虽然理论上标签校正技术可能间接有助于训练更鲁棒的文档理解模型，但其方法并非针对文档图像理解（DIU）中的具体痛点（如OCR幻觉、复杂排版、细粒度定位等）设计，也没有在DIU任务上验证或讨论迁移路径。属于与DIU弱相关的通用技术，且无明确可迁移的机制创新，落入‘平行方法应用’边缘，按原则应低分。
Abstract The challenge of learning with noisy labels is significant in machine learning, as it can severely degrade the performance of prediction models if not addressed properly. This paper introduces a novel framework that conceptualizes noisy label correction as a reinforcement learning (RL) problem. The proposed approach, Reinforcement Learning for Noisy Label Correction (RLNLC), defines a comprehensive state space representing data and their associated labels, an action space that indicates possible label corrections, and a reward mechanism that evaluates the efficacy of label corrections. RLNLC learns a deep feature representation based policy network to perform label correction through reinforcement learning, utilizing an actor-critic method. The learned policy is subsequently deployed to iteratively correct noisy training labels and facilitate the training of the prediction model. The effectiveness of RLNLC is demonstrated through extensive experiments on multiple benchmark datasets, where it consistently outperforms existing state-of-the-art techniques for learning with noisy labels.
Authors Marzi Heidari, Hanping Zhang, Yuhong Guo
Comment NeurIPS 2025
Categories Artificial Intelligence
3
Guaranteed Optimal Compositional Explanations for Neurons
Title CN 神经元的最优组合解释保证
Keywords 可解释性 · 组合解释 · 神经元分析 · beam search · 最优性保证
Summary 本文提出了一种能够计算神经元组合解释的最优解框架，通过分解影响因素、设计启发式估计方法，首次实现了在合理时间内获得保证最优的逻辑规则解释。实验表明，在使用beam search的传统方法中，有10%-40%的解释在涉及重叠概念时是次优的。尽管该工作对可解释AI有理论贡献，但其应用场景为通用CNN的神经元分析，并未面向文档图像或多模态语言模型。
Reason 该论文研究的是神经元激活与人类可理解概念之间的逻辑关系建模，属于模型可解释性（XAI）的理论方法，聚焦于计算机视觉中的CNN分析。虽然其提出的搜索算法和最优性保证在理论上具有价值，但并未针对文档图像理解（DIU）任务设计，也无法直接迁移解决OCR幻觉、复杂排版或长文档推理等核心痛点。其方法更偏向通用视觉模型解释，与VLM在文档场景下的多模态对齐、细粒度定位或高分辨率处理无直接关联。
Abstract While neurons are the basic units of deep neural networks, it is still unclear what they learn and if their knowledge is aligned with that of humans. Compositional explanations aim to answer this question by describing the spatial alignment between neuron activations and concepts through logical rules. These logical descriptions are typically computed via a search over all possible concept combinations. Since computing the spatial alignment over the entire state space is computationally infeasible, the literature commonly adopts beam search to restrict the space. However, beam search cannot provide any theoretical guarantees of optimality, and it remains unclear how close current explanations are to the true optimum. In this theoretical paper, we address this gap by introducing the first framework for computing guaranteed optimal compositional explanations. Specifically, we propose: (i) a decomposition that identifies the factors influencing the spatial alignment, (ii) a heuristic to estimate the alignment at any stage of the search, and (iii) the first algorithm that can compute optimal compositional explanations within a feasible time. Using this framework, we analyze the differences between optimal and non-optimal explanations in the most popular settings for compositional explanations, the computer vision domain and Convolutional Neural Networks. In these settings, we demonstrate that 10-40 percent of explanations obtained with beam search are suboptimal when overlapping concepts are involved. Finally, we evaluate a beam-search variant guided by our proposed decomposition and heuristic, showing that it matches or improves runtime over prior methods while offering greater flexibility in hyperparameters and computational resources.
Authors Biagio La Rosa, Leilani H. Gilpin
Comment 41 pages, 10 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Open Vocabulary Compositional Explanations for Neuron Alignment
Title CN 用于神经元对齐的开放词汇组合式解释
Keywords 神经元解释 · 开放词汇分割 · 组合式解释 · 模型可解释性 · 视觉理解
Summary 本文提出一种基于开放词汇语义分割的组合式解释框架，用于分析神经网络中神经元激活与人类可理解概念之间的空间对齐关系。该方法允许用户在任意数据集和任意概念上探测神经元行为，摆脱了对人工标注概念的依赖。通过三步流程：定义概念、生成分割掩码、推导组合解释，实现了更灵活的模型解释能力。实验比较了新旧方法在量化指标和人类可读性上的差异。
Reason 该论文聚焦于神经元解释性分析，提出利用开放词汇语义分割生成组合式解释以替代人工标注概念，属于模型可解释性研究。尽管其方法涉及视觉领域，但核心目标是神经元级别的理解与对齐，并未针对文档图像理解中的任何具体任务（如DocVQA、表格识别、布局分析等）进行设计或验证。同时，该工作不具备可迁移至DIU领域的明确路径——例如无法直接缓解OCR幻觉、提升细粒度定位或处理长文档推理。其技术本质属于‘平行应用’于神经元解释的框架扩展，而非底层方法创新，且不满足DIU本题要求。
Abstract Neurons are the fundamental building blocks of deep neural networks, and their interconnections allow AI to achieve unprecedented results. Motivated by the goal of understanding how neurons encode information, compositional explanations leverage logical relationships between concepts to express the spatial alignment between neuron activations and human knowledge. However, these explanations rely on human-annotated datasets, restricting their applicability to specific domains and predefined concepts. This paper addresses this limitation by introducing a framework for the vision domain that allows users to probe neurons for arbitrary concepts and datasets. Specifically, the framework leverages masks generated by open vocabulary semantic segmentation to compute open vocabulary compositional explanations. The proposed framework consists of three steps: specifying arbitrary concepts, generating semantic segmentation masks using open vocabulary models, and deriving compositional explanations from these masks. The paper compares the proposed framework with previous methods for computing compositional explanations both in terms of quantitative metrics and human interpretability, analyzes the differences in explanations when shifting from human-annotated data to model-annotated data, and showcases the additional capabilities provided by the framework in terms of flexibility of the explanations with respect to the tasks and properties of interest.
Authors Biagio La Rosa, Leilani H. Gilpin
Comment 47 pages, 11 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory
Title CN Evo-Memory：通过自演化记忆评估大语言模型代理的测试时学习
Keywords LLM Agent · Test-time Learning · Memory Evolution · Continual Reasoning · ExpRAG
Summary 本文提出Evo-Memory，一个用于评估大语言模型代理在连续任务流中自演化记忆能力的基准框架。它强调模型在部署过程中动态检索、整合和更新记忆的能力，并构建了包含10个多样化数据集的任务流评测体系。作者还提出了ExpRAG和ReMem两种基线方法，以实现经验复用与推理-行动-记忆的闭环优化。然而，该工作完全基于文本任务，未涉及视觉输入或文档结构理解。
Reason 该论文聚焦于LLM代理在连续任务流中的记忆演化与经验复用，属于通用Agent架构研究。尽管其提出的ReMem流水线和ExpRAG机制涉及推理与记忆更新的协同，但整体工作仍为通用对话或交互式AI场景设计，并未针对文档图像理解（DIU）中的核心痛点如OCR幻觉、复杂排版解析、细粒度视觉定位或高分辨率输入处理进行优化。虽然‘测试时演化’的思想理论上可迁移至长文档多步阅读任务，但缺乏对视觉模态、布局结构或图文对齐的建模，无法直接解决DIU的关键挑战。因此属于通用Agent方法，尚未触及DIU的方法论创新。
Abstract Statefulness is essential for large language model (LLM) agents to perform long-term planning and problem-solving. This makes memory a critical component, yet its management and evolution remain largely underexplored. Existing evaluations mostly focus on static conversational settings, where memory is passively retrieved from dialogue to answer queries, overlooking the dynamic ability to accumulate and reuse experience across evolving task streams. In real-world environments such as interactive problem assistants or embodied agents, LLMs are required to handle continuous task streams, yet often fail to learn from accumulated interactions, losing valuable contextual insights, a limitation that calls for test-time evolution, where LLMs retrieve, integrate, and update memory continuously during deployment. To bridge this gap, we introduce Evo-Memory, a comprehensive streaming benchmark and framework for evaluating self-evolving memory in LLM agents. Evo-Memory structures datasets into sequential task streams, requiring LLMs to search, adapt, and evolve memory after each interaction. We unify and implement over ten representative memory modules and evaluate them across 10 diverse multi-turn goal-oriented and single-turn reasoning and QA datasets. To better benchmark experience reuse, we provide a baseline method, ExpRAG, for retrieving and utilizing prior experience, and further propose ReMem, an action-think-memory refine pipeline that tightly integrates reasoning, task actions, and memory updates to achieve continual improvement.
Authors Tianxin Wei, Noveen Sachdeva, Benjamin Coleman, Zhankui He, Yuanchen Bei, Xuying Ning, Mengting Ai, Yunzhe Li, Jingrui He, Ed H. Chi, Chi Wang, Shuo Chen, Fernando Pereira, Wang-Cheng Kang, Derek Zhiyuan Cheng
Categories Artificial Intelligence, Computation and Language
3
Pre-train to Gain: Robust Learning Without Clean Labels
Title CN 预训练以获益：无需干净标签的鲁棒学习
Keywords 自监督学习 · 噪声标签 · 鲁棒训练 · 预训练 · 深度学习
Summary 本文提出一种利用自监督学习（如SimCLR、Barlow Twins）对特征提取器进行无监督预训练，再在带噪声的标签数据上进行有监督训练的方法，以提升模型对标签噪声的鲁棒性。实验表明该方法在CIFAR-10/100上的各种噪声水平下均能提高分类准确率和标签错误检测能力，尤其在高噪声情况下优于ImageNet预训练模型。
Reason 该论文研究在存在噪声标签的情况下如何通过自监督预训练提升模型鲁棒性，属于通用深度学习训练方法的改进。虽然标签噪声问题在理论上可能间接影响文档理解中的标注数据质量，但本文未针对文档模态、多模态对齐、OCR幻觉或复杂排版等DIU核心痛点设计方法，也未涉及视觉-语言模型架构或高分辨率处理等可迁移工具。其方法基于CIFAR数据集验证，远离文档图像的实际场景，且无任何与VLM、视觉token化、细粒度定位或长文档推理相关的创新，属于通用AI训练技术，无法直接为DIU提供方法论突破。
Abstract Training deep networks with noisy labels leads to poor generalization and degraded accuracy due to overfitting to label noise. Existing approaches for learning with noisy labels often rely on the availability of a clean subset of data. By pre-training a feature extractor backbone without labels using self-supervised learning (SSL), followed by standard supervised training on the noisy dataset, we can train a more noise robust model without requiring a subset with clean labels. We evaluate the use of SimCLR and Barlow~Twins as SSL methods on CIFAR-10 and CIFAR-100 under synthetic and real world noise. Across all noise rates, self-supervised pre-training consistently improves classification accuracy and enhances downstream label-error detection (F1 and Balanced Accuracy). The performance gap widens as the noise rate increases, demonstrating improved robustness. Notably, our approach achieves comparable results to ImageNet pre-trained models at low noise levels, while substantially outperforming them under high noise conditions.
Authors David Szczecina, Nicholas Pellegrino, Paul Fieguth
Comment 5 pages, 3 figures
Categories Artificial Intelligence
3
Structured Prompting Enables More Robust, Holistic Evaluation of Language Models
Title CN 结构化提示实现对语言模型更稳健、全面的评估
Keywords 语言模型评测 · 结构化提示 · DSPy · HELM · 提示优化
Summary 本文提出了一种结合DSPy与HELM的结构化提示评测框架，通过四种提示方法对多个前沿语言模型进行系统性评估，发现传统固定提示会低估模型性能并导致排名不稳定。引入结构化提示（尤其是推理链）可降低对提示设计的敏感性，提升评估可靠性。研究强调了可扩展的性能上限估计在基准测试中的重要性，并开源了相关工具链。
Reason 该论文聚焦于语言模型的评测框架优化，属于评测方法学研究，其核心贡献是将DSPy等结构化提示技术应用于HELM基准测试中以提升评估稳定性。尽管涉及提示工程与推理链（CoT），但其目标领域为通用和医疗文本的LM评测，未涉及任何视觉或文档图像模态。无法直接解决文档图像理解中的OCR幻觉、排版复杂性或细粒度定位等问题，且无多模态扩展设计。属于典型的‘平行应用’类研究——在现有评测框架上做提示工程改进，而非提出可迁移至DIU的底层技术创新。
Abstract As language models (LMs) are increasingly adopted across domains, high-quality benchmarking frameworks that accurately estimate performance are essential for guiding deployment decisions. While frameworks such as Holistic Evaluation of Language Models (HELM) enable broad evaluation across tasks, they often rely on fixed prompts that fail to generalize across LMs, yielding unrepresentative performance estimates. Unless we estimate each LM's ceiling (maximum achievable via changes to the prompt), we risk underestimating performance. Declarative prompting frameworks, such as DSPy, offer a scalable alternative to manual prompt engineering by crafting structured prompts that can be optimized per task. However, such frameworks have not been systematically evaluated across established benchmarks. We present a reproducible DSPy+HELM framework that introduces structured prompting methods which elicit reasoning, enabling more accurate LM benchmarking. Using four prompting methods, we evaluate four frontier LMs across seven benchmarks (general/medical domain) against existing HELM baseline scores. We find that without structured prompting: (i) HELM underestimates LM performance (by 4% average), (ii) performance estimates vary more across benchmarks (+2% standard deviation), (iii) performance gaps are misrepresented (leaderboard rankings flip on 3/7 benchmarks), and (iv) introducing reasoning (chain-of-thought) reduces LM sensitivity to prompt design (smaller Δ across prompts). To our knowledge, this is the first large-scale benchmarking study to empirically characterize LM behavior across benchmarks and prompting methods, showing that scalable performance ceiling estimation enables more decision-useful benchmarks. We open-source (i) DSPy+HELM Integration (https://github.com/stanford-crfm/helm/pull/3893) and (ii) Prompt Optimization Pipeline (https://github.com/StanfordMIMI/dspy-helm).
Authors Asad Aali, Muhammad Ahmed Mohsin, Vasiliki Bikia, Arnav Singhvi, Richard Gaus, Suhana Bedi, Hejie Cui, Miguel Fuentes, Alyssa Unell, Yifan Mai, Jordan Cahoon, Michael Pfeffer, Roxana Daneshjou, Sanmi Koyejo, Emily Alsentzer, Percy Liang, Christopher Potts, Nigam H. Shah, Akshay S. Chaudhari
Categories Artificial Intelligence, Computation and Language
3
SPHINX: A Synthetic Environment for Visual Perception and Reasoning
Title CN SPHINX：一个用于视觉感知与推理的合成环境
Keywords 视觉推理 · 合成数据 · 基准测试 · 视觉语言模型 · 强化学习
Summary 本文提出了SPHINX，一个基于程序化生成的视觉推理合成环境，包含25种类型的谜题，涵盖对称性、空间推理、图表解读等任务，配有可验证的真值答案。实验表明当前最先进的LVLM（如GPT-5）在此基准上准确率仅为51.1%，远低于人类水平，并尝试用可验证奖励的强化学习（RLVR）提升性能。该工作主要贡献在于构建可控的视觉推理评测集，而非提出可迁移至文档理解的技术方法。
Reason 该论文构建了一个合成环境用于评估视觉语言模型在多种视觉推理任务上的表现，并探索了强化学习提升性能的可能性。然而，其核心是通用视觉推理基准的构建与验证，任务类型如对称性检测、几何变换、图表理解等虽与文档有一定关联，但整体设计面向抽象认知任务，而非真实文档结构、排版或文本密集场景的理解。其方法难以直接迁移解决文档图像理解中的OCR幻觉、细粒度定位或复杂布局解析等核心痛点，属于通用视觉推理领域的下游评测工作，不具备针对DIU的方法论创新。
Abstract We present Sphinx, a synthetic environment for visual perception and reasoning that targets core cognitive primitives. Sphinx procedurally generates puzzles using motifs, tiles, charts, icons, and geometric primitives, each paired with verifiable ground-truth solutions, enabling both precise evaluation and large-scale dataset construction. The benchmark covers 25 task types spanning symmetry detection, geometric transformations, spatial reasoning, chart interpretation, and sequence prediction. Evaluating recent large vision-language models (LVLMs) shows that even state-of-the-art GPT-5 attains only 51.1% accuracy, well below human performance. Finally, we demonstrate that reinforcement learning with verifiable rewards (RLVR) substantially improves model accuracy on these tasks and yields gains on external visual reasoning benchmarks, highlighting its promise for advancing multimodal reasoning.
Authors Md Tanvirul Alam, Saksham Aggarwal, Justin Yang Chae, Nidhi Rastogi
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Physics Steering: Causal Control of Cross-Domain Concepts in a Physics Foundation Model
Title CN 物理引导：在物理基础模型中对跨域概念的因果控制
Keywords 机制可解释性 · 概念引导 · 基础模型 · 物理仿真 · 激活空间
Summary 该研究探索了一个专注于物理任务的基础模型内部是否形成可解释的抽象概念表示，并提出通过提取不同物理状态之间的激活差异（delta tensors）作为概念方向，在推理时注入这些方向以实现对模型行为的因果控制。实验表明可以诱导或抑制特定物理特征的出现，说明模型学习到了泛化的物理规律而非表面相关性。
Reason 该论文研究的是物理领域的基础模型内部表示及其通过激活空间方向进行行为引导的方法，属于科学计算与LLM可解释性的交叉，但其任务和方法均不直接涉及文档图像理解（DIU）的核心问题，如OCR幻觉、复杂排版解析或视觉-语言对齐。虽然‘干预/引导’机制本身具有方法论潜力，但本文的应用场景（物理仿真）与文档理解差距过大，迁移路径不明确，属于无关领域的平行应用。
Abstract Recent advances in mechanistic interpretability have revealed that large language models (LLMs) develop internal representations corresponding not only to concrete entities but also distinct, human-understandable abstract concepts and behaviour. Moreover, these hidden features can be directly manipulated to steer model behaviour. However, it remains an open question whether this phenomenon is unique to models trained on inherently structured data (ie. language, images) or if it is a general property of foundation models. In this work, we investigate the internal representations of a large physics-focused foundation model. Inspired by recent work identifying single directions in activation space for complex behaviours in LLMs, we extract activation vectors from the model during forward passes over simulation datasets for different physical regimes. We then compute "delta" representations between the two regimes. These delta tensors act as concept directions in activation space, encoding specific physical features. By injecting these concept directions back into the model during inference, we can steer its predictions, demonstrating causal control over physical behaviours, such as inducing or removing some particular physical feature from a simulation. These results suggest that scientific foundation models learn generalised representations of physical principles. They do not merely rely on superficial correlations and patterns in the simulations. Our findings open new avenues for understanding and controlling scientific foundation models and has implications for AI-enabled scientific discovery.
Authors Rio Alexa Fear, Payel Mukhopadhyay, Michael McCabe, Alberto Bietti, Miles Cranmer
Comment 16 Pages, 9 Figures. Code available at https://github.com/DJ-Fear/walrus_steering
Categories Artificial Intelligence
3
Vision-Language Memory for Spatial Reasoning
Title CN 用于空间推理的视觉-语言记忆
Keywords 视觉-语言模型 · 空间推理 · 持久化记忆 · 3D感知 · 视频理解
Summary 本文提出VLM²，一种带有持久化记忆的视觉-语言模型，用于从2D视频中构建视图一致、具备3D感知的空间表示，以支持长视野的空间推理。模型包含工作记忆（滑动窗口）和情景记忆（长期存储关键信息），在多个视频空间推理基准上达到SOTA水平。
Reason 该论文聚焦于视频中的空间推理任务，主要面向机器人等具身智能场景，解决的是3D语义与几何对齐以及长时间记忆问题。其核心贡献——双记忆模块（工作记忆+情景记忆）——虽然在长时序推理上有创新，但应用场景为视频理解，属于负面清单中的‘无关领域’。文档图像理解中的长文档推理虽有‘长期依赖’表层相似性，但本质是静态高分辨率图像中的细粒度文本定位与逻辑结构建模，而非动态视角下的3D空间一致性维护。该方法难以迁移至文档场景，且未涉及OCR、布局分析或VQA等DIU核心任务。
Abstract Spatial reasoning is a critical capability for intelligent robots, yet current vision-language models (VLMs) still fall short of human-level performance in video-based spatial reasoning. This gap mainly stems from two challenges: a semantic-geometric misalignment that prevents consistent 3D understanding, and the absence of persistent memory to retain 3D representation and understanding over time. To address these limitations, we present VLM$^2$, a Vision-Language Model with persistent Memory for spatial reasoning with a view-consistent, 3D-aware representation purely from 2D video. Specifically, to enhance long-horizon reasoning, we incorporate a dual-memory module, consisting of a working memory that operates as a sliding window to focus on immediate context, and an episodic memory that consolidates and stores critical long-term information. This design enables efficient and long-horizon spatial reasoning with a fixed computational cost. Extensive experiments on multiple benchmarks show that VLM$^2$ achieves state-of-the-art performance among video-only models, significantly advancing the frontier of visual-spatial intelligence.
Authors Zuntao Liu, Yi Du, Taimeng Fu, Shaoshu Su, Cherie Ho, Chen Wang
Categories Computer Vision and Pattern Recognition
3
Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition
Title CN 释放视觉-语言模型在长尾多标签视觉识别中的潜力
Keywords 长尾学习 · 多标签分类 · 视觉语言模型 · CLIP · 标签相关性
Summary 本文提出CAPNET框架，通过CLIP文本编码器显式建模标签相关性，结合图卷积网络与可学习软提示，在长尾多标签视觉识别任务中提升尾部类别的性能。采用分布均衡Focal Loss与测试时集成策略优化训练，并通过参数高效微调保持模态对齐。在多个基准数据集上取得了优于现有方法的表现。
Reason 该论文聚焦于通用图像领域的长尾多标签分类任务，使用CLIP等VLM进行语义建模并设计了标签相关性适配机制。尽管其利用了视觉-语言先验知识和分布平衡策略，但任务本质仍属于标准的多标签图像分类，未涉及文档结构、文本布局、OCR或细粒度文本定位等文档理解核心问题。方法上也未针对高分辨率文档输入、密集文本处理或排版理解进行设计，无法迁移解决DIU中的OCR幻觉、复杂逻辑推理或长文档阅读等痛点。
Abstract Long-tailed multi-label visual recognition poses a significant challenge, as images typically contain multiple labels with highly imbalanced class distributions, leading to biased models that favor head classes while underperforming on tail classes. Recent efforts have leveraged pre-trained vision-language models, such as CLIP, alongside long-tailed learning techniques to exploit rich visual-textual priors for improved performance. However, existing methods often derive semantic inter-class relationships directly from imbalanced datasets, resulting in unreliable correlations for tail classes due to data scarcity. Moreover, CLIP's zero-shot paradigm is optimized for single-label image-text matching, making it suboptimal for multi-label tasks. To address these issues, we propose the correlation adaptation prompt network (CAPNET), a novel end-to-end framework that explicitly models label correlations from CLIP's textual encoder. The framework incorporates a graph convolutional network for label-aware propagation and learnable soft prompts for refined embeddings. It utilizes a distribution-balanced Focal loss with class-aware re-weighting for optimized training under imbalance. Moreover, it improves generalization through test-time ensembling and realigns visual-textual modalities using parameter-efficient fine-tuning to avert overfitting on tail classes without compromising head class performance. Extensive experiments and ablation studies on benchmarks including VOC-LT, COCO-LT, and NUS-WIDE demonstrate that CAPNET achieves substantial improvements over state-of-the-art methods, validating its effectiveness for real-world long-tailed multi-label visual recognition.
Authors Wei Tang, Zuo-Zheng Wang, Kun Zhang, Tong Wei, Min-Ling Zhang
Categories Computer Vision and Pattern Recognition
3
Localizing Knowledge in Diffusion Transformers
Title CN 在扩散Transformer中定位知识
Keywords Diffusion Transformer · Knowledge Localization · Model Editing · Interpretability · Generative Models
Summary 本文提出一种模型无关的方法来定位扩散Transformer（DiT）中特定知识编码的位置，并在PixArt-alpha、FLUX、SANA等先进DiT模型上验证了其有效性。通过识别与特定知识相关的网络块，实现了可解释且因果有效的模型编辑，应用于模型个性化和知识遗忘任务，能够在最小干扰下实现高效、定向的模型更新。
Reason 该论文研究的是扩散Transformer（DiT）中知识分布的定位问题，主要应用于图像生成模型的可解释性与编辑，如模型个性化和知识遗忘。其核心任务聚焦于生成式视觉模型内部机制分析，属于纯图像生成领域，与文档图像理解（DIU）中的OCR、布局分析、表格识别等任务无直接关联。尽管‘知识定位’概念在理论上可能启发模型干预方法，但该工作并未涉及多模态输入、文本-视觉对齐或文档结构建模，迁移到DIU场景（如消除OCR幻觉或处理复杂排版）极为困难。
Abstract Understanding how knowledge is distributed across the layers of generative models is crucial for improving interpretability, controllability, and adaptation. While prior work has explored knowledge localization in UNet-based architectures, Diffusion Transformer (DiT)-based models remain underexplored in this context. In this paper, we propose a model- and knowledge-agnostic method to localize where specific types of knowledge are encoded within the DiT blocks. We evaluate our method on state-of-the-art DiT-based models, including PixArt-alpha, FLUX, and SANA, across six diverse knowledge categories. We show that the identified blocks are both interpretable and causally linked to the expression of knowledge in generated outputs. Building on these insights, we apply our localization framework to two key applications: model personalization and knowledge unlearning. In both settings, our localized fine-tuning approach enables efficient and targeted updates, reducing computational cost, improving task-specific performance, and better preserving general model behavior with minimal interference to unrelated or surrounding content. Overall, our findings offer new insights into the internal structure of DiTs and introduce a practical pathway for more interpretable, efficient, and controllable model editing.
Authors Arman Zarei, Samyadeep Basu, Keivan Rezaei, Zihao Lin, Sayan Nag, Soheil Feizi
Categories Computer Vision and Pattern Recognition
3
FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questions NeurIPS 2025 Workshop on Foundations of Reasoning in Language Models (FoRLM)
Title CN FlagEval 研究报告：基于可自动验证的文本与视觉问题对大模型推理能力的初步评估
Keywords VLM评测 · 推理模型 · 视觉问答 · 基准测试 · 污染检测
Summary 本文发布了一个名为ROME的视觉语言模型推理评测基准，强调污染控制下的模型评估，关注从视觉线索中进行推理的能力。研究团队进行了中等规模的实验并分享了初步发现。项目主页提供了数据和评测资源。该工作将发表于NeurIPS 2025的FoRLM研讨会。
Reason 该论文主要贡献是一个面向大推理模型（LRMs）的评测基准ROME，用于评估模型在文本和视觉问题上的推理能力，并在NeurIPS 2025的一个研讨会上展示。尽管涉及视觉语言模型评估，但其核心是评测框架而非针对文档图像理解（DIU）任务的方法创新。未提出可用于解决OCR幻觉、复杂排版或长文档推理等DIU痛点的新架构、训练策略或推理机制。属于通用VLM评测工作，对DIU领域的直接迁移价值有限。
Abstract We conduct a moderate-scale contamination-free (to some extent) evaluation of current large reasoning models (LRMs) with some preliminary findings. We also release ROME, our evaluation benchmark for vision language models intended to test reasoning from visual clues. We attach links to the benchmark, evaluation data, and other updates on this website: https://flageval-baai.github.io/LRM-Eval/
Authors Bowen Qin, Chen Yue, Fang Yin, Hui Wang, JG Yao, Jiakang Liu, Jing-Shu Zheng, Miguel Hu Chen, Richeng Xuan, Shibei Meng, Shiqi Zhou, Teng Dai, Tong-Shuai Ren, Wei Cui, Xi Yang, Xialin Du, Xiaojing Xu, Xue Sun, Xuejing Li, Yaming Liu, Yesheng Liu, Ying Liu, Yonghua Lin, Yu Zhao, Yunduo Zhang, Yuwen Luo, Zheqi He, Zhiyuan He, Zhongyuan Wang
Comment Project homepage: https://flageval-baai.github.io/LRM-Eval/ This work will also be presented at NeurIPS 2025 Workshop on Foundations of Reasoning in Language Models (FoRLM); update with trials on Gemini 3 Pro
Categories Computer Vision and Pattern Recognition, Computation and Language
3
HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation
Title CN HBridge：用于统一多模态理解和生成的异构专家H形桥接
Keywords 多模态学习 · 模型融合 · H形架构 · 跨模态对齐 · 语义重建
Summary 本文提出HBridge，一种不对称的H形架构，用于连接多模态中的理解与生成专家模型。通过选择性桥接中间层并引入语义重建token，减少跨模态注意力共享，提升生成效率与语义一致性。实验表明其在多个多模态基准上性能优越。
Reason 该论文聚焦于统一多模态理解与生成，核心是连接理解专家（如LLM）和生成专家（如扩散模型），其主要创新在于跨模态架构设计，目标是提升图像生成质量与多模态一致性。然而，该工作属于典型的‘平行应用’类架构整合，并未针对文档图像理解中的关键痛点（如OCR幻觉、密集文本识别、复杂版式建模、长文档推理）提出专门解决方案。其H形桥接结构虽在通用多模态生成任务上有优势，但迁移至文档场景的价值有限——文档理解更关注细粒度视觉-语言对齐与高分辨率语义提取，而非生成能力。此外，该方法仍基于MoT范式演进，不构成底层范式转移，且缺乏对文档特有挑战（如排版噪声、小字体、表格结构）的建模。
Abstract Recent unified models integrate understanding experts (e.g., LLMs) with generative experts (e.g., diffusion models), achieving strong multimodal performance. However, recent advanced methods such as BAGEL and LMFusion follow the Mixture-of-Transformers (MoT) paradigm, adopting a symmetric design that mirrors one expert to another for convenient initialization and fusion, which remains suboptimal due to inherent modality discrepancies. In this work, we propose HBridge, an asymmetric H-shaped architecture that enables heterogeneous experts to optimally leverage pretrained priors from their respective modality domains. Unlike prior dense fusion strategies that straightforwardly connect all layers between experts via shared attention, HBridge selectively bridges intermediate layers, reducing over 40% attention sharing, which improves efficiency and enhances generation quality. Shallow and deep layers, which capture modality-specific representations, are decoupled, while mid-layer bridging promotes semantic alignment. To further strengthen cross-modal coherence, we introduce semantic reconstruction tokens that explicitly guide the generative expert to reconstruct visual semantic tokens of the target image. Extensive experiments across multiple benchmarks demonstrate the effectiveness and superior performance of HBridge, establishing a new paradigm for unified multimodal generation.
Authors Xiang Wang, Zhifei Zhang, He Zhang, Zhe Lin, Yuqian Zhou, Qing Liu, Shiwei Zhang, Yijun Li, Shaoteng Liu, Haitian Zheng, Jason Kuen, Yuehuan Wang, Changxin Gao, Nong Sang
Categories Computer Vision and Pattern Recognition
3
AlignBench: Benchmarking Fine-Grained Image-Text Alignment with Synthetic Image-Caption Pairs
Title CN AlignBench：使用合成图像-文本对进行细粒度图文对齐评测
Keywords 图文对齐 · 评测基准 · 合成数据 · VLM评估 · CLIP
Summary 本文提出了AlignBench，一个基于合成图像-文本对的细粒度图文对齐评测基准。通过使用多种图像到文本和文本到图像模型生成详细描述，并人工标注句子正确性，用于评估现有VLM在细粒度对齐上的表现。实验发现当前CLIP类模型在细粒度对齐上能力有限，存在早期句子偏好和自我偏好等问题。
Reason 该论文属于图文对齐评估的基准构建工作，虽然涉及VLM评估，但其核心是为通用图像-文本对齐设计合成数据和评测机制，并未针对文档图像理解中的具体任务（如DocVQA、表格识别、布局分析等）提出改进。所用数据为合成图像-文本对，不聚焦于文档特有的密集文本、复杂排版或OCR相关问题。属于通用VLM评估方向的‘平行应用’性质工作，无法直接迁移解决DIU的核心痛点，如OCR幻觉或长文档推理。
Abstract Assessing image-text alignment models such as CLIP is crucial for bridging visual and linguistic representations. Yet existing benchmarks rely on rule-based perturbations or short captions, limiting their ability to measure fine-grained alignment. We introduce AlignBench, a benchmark that provides a new indicator of image-text alignment by evaluating detailed image-caption pairs generated by diverse image-to-text and text-to-image models. Each sentence is annotated for correctness, enabling direct assessment of VLMs as alignment evaluators. Benchmarking a wide range of decoder-based VLMs reveals three key findings: (i) CLIP-based models, even those tailored for compositional reasoning, remain nearly blind; (ii) detectors systematically over-score early sentences; and (iii) they show strong self-preference, favoring their own outputs and harming detection performance. Our project page will be available at https://dahlian00.github.io/AlignBench/.
Authors Kuniaki Saito, Risa Shinoda, Shohei Tanaka, Tosho Hirasawa, Fumio Okura, Yoshitaka Ushiku
Comment Project Page: https://dahlian00.github.io/AlignBench/
Categories Computer Vision and Pattern Recognition
3
Adversarial Robustness for Unified Multi-Modal Encoders via Efficient Calibration
Title CN 通过高效校准实现统一多模态编码器的对抗鲁棒性
Keywords 对抗鲁棒性 · 多模态学习 · 统一编码器 · 模态校准 · 对抗攻击
Summary 本文首次系统研究了统一多模态编码器在对抗扰动下的脆弱性，发现即使是轻微扰动也会导致跨模态性能显著下降。为此提出一种高效的对抗校准框架，在不改动预训练编码器的前提下，通过为各模态设计特定的投影头并仅使用对抗样例进行训练，提升了模型鲁棒性。实验表明该方法在六种模态和三类Bind-style模型上均有效，同时保持了原始性能。
Reason 该论文研究的是统一多模态编码器在对抗攻击下的鲁棒性问题，提出了一种无需修改预训练模型即可提升鲁棒性的校准方法。虽然其技术可能对多模态系统有一定通用价值，但核心关注点是安全与鲁棒性，而非文档图像理解（DIU）中的OCR幻觉、复杂排版或长文档推理等关键痛点。且未见其方法可直接迁移用于解决文档细粒度识别或视觉-文本对齐的具体路径，属于通用多模态安全研究，不构成对DIU的方法论推动。
Abstract Recent unified multi-modal encoders align a wide range of modalities into a shared representation space, enabling diverse cross-modal tasks. Despite their impressive capabilities, the robustness of these models under adversarial perturbations remains underexplored, which is a critical concern for safety-sensitive applications. In this work, we present the first comprehensive study of adversarial vulnerability in unified multi-modal encoders. We find that even mild adversarial perturbations lead to substantial performance drops across all modalities. Non-visual inputs, such as audio and point clouds, are especially fragile, while visual inputs like images and videos also degrade significantly. To address this, we propose an efficient adversarial calibration framework that improves robustness across modalities without modifying pretrained encoders or semantic centers, ensuring compatibility with existing foundation models. Our method introduces modality-specific projection heads trained solely on adversarial examples, while keeping the backbone and embeddings frozen. We explore three training objectives: fixed-center cross-entropy, clean-to-adversarial L2 alignment, and clean-adversarial InfoNCE, and we introduce a regularization strategy to ensure modality-consistent alignment under attack. Experiments on six modalities and three Bind-style models show that our method improves adversarial robustness by up to 47.3 percent at epsilon = 4/255, while preserving or even improving clean zero-shot and retrieval performance with less than 1 percent trainable parameters.
Authors Chih-Ting Liao, Zhangquan Chen, Chunlei Meng, Tzu-Yu Huang, Xin Cao, Xu Zheng
Categories Computer Vision and Pattern Recognition
3
ScenarioCLIP: Pretrained Transferable Visual Language Models and Action-Genome Dataset for Natural Scene Analysis
Title CN ScenarioCLIP：用于自然场景分析的预训练可迁移视觉语言模型及动作图谱数据集
Keywords 视觉语言模型 · 场景理解 · 关系建模 · CLIP · 跨模态检索
Summary 本文提出ScenarioCLIP，一种结合输入文本、对象关系和图像中特定区域的视觉语言模型，用于提升自然场景下的细粒度理解和跨模态检索。作者构建了一个新的动作-对象关系数据集，通过LLM辅助生成并人工校正，支持对复杂场景中多个对象及其交互行为的理解。模型在零样本和微调设置下均表现出色。
Reason 该论文聚焦于自然场景中的多对象与动作关系建模，属于通用视觉语言模型在复杂场景理解上的扩展，但其任务设定（如跨模态检索、细粒度场景理解）主要面向自然图像而非文档图像。未涉及文档特有的挑战如OCR幻觉、表格结构识别、密集文本排版或长文档推理。尽管使用了关系建模和局部区域对齐，但这些改进并未针对文档内容设计，也无法直接迁移解决文档图像理解的核心痛点，属于通用VLM在另一场景的平行应用。
Abstract Until recently, the general corpus of CLIP-type fundamental models has widely explored either the retrieval of short descriptions or the classification of objects in the scene as SINGLE-object image classification task. The same holds for retrieving the image embedding (image retrieval task) given a text prompt. However, real-world scene images exhibit rich compositional structure involving multiple objects and actions. The latest methods in the CLIP-based literature improve class-level discrimination by mining harder negative image-text pairs and by refining permanent text prompts, often using LLMs. However, these improvements remain confined to predefined class lists and do not explicitly model relational or compositional structure. PyramidCLIP partially addresses this gap by aligning global and local visual features, yet it still lacks explicit modeling of inter-object relations. Hence, to further leverage this aspect for scene analysis, the proposed ScenarioCLIP model accepts input texts, grounded relations, and input images, along with focused regions highlighting relations. The proposed model is pretrained on curated scenario data, and finetuned for specialized downstream tasks, such as cross-modal retrieval and fine-grained visual understanding tasks. To address the lack of domain-specific datasets, we generate a novel dataset by extending image-text pairs from existing diverse indoor and outdoor scenario datasets that are publicly available. We used a pipeline of existing language models to ground action, object, and relations, filled by manual and automatic curation. We established a comprehensive benchmark for several scenario-based tasks and compared it with many baseline methods. ScenarioCLIP demonstrates robust zero-shot and finetune performance on various domain-specific tasks. Our code and dataset are available at https://github.com/scenario-clip/ScenarioCLIP
Authors Advik Sinha, Saurabh Atreya, Aashutosh A, Sk Aziz Ali, Abhijit Das
Categories Computer Vision and Pattern Recognition
3
Probabilistic Hyper-Graphs using Multiple Randomly Masked Autoencoders for Semi-supervised Multi-modal Multi-task Learning Neurocomputing
Title CN 基于多个随机掩码自编码器的半监督多模态多任务学习概率超图
Keywords 多模态学习 · 掩码自编码器 · 概率图模型 · 半监督学习 · 多任务学习
Summary 本文提出了PHG-MAE，一种将神经图模型与掩码自编码器统一的概率超图框架，通过随机遮蔽整个模态而非图像块来采样超边分布，并将预训练与微调融合为单一训练流程。该方法支持推理时集成以提升性能，并可进行知识蒸馏压缩至小模型。实验集中在无人机航拍场景的多模态理解，配套发布了扩展版Dronescapes数据集及自动化数据流水线。
Reason 该论文提出了一种结合神经图模型与掩码自编码器的概率超图框架，主要应用于无人机场景（UAV scenes）下的多模态多任务学习，并强调了在自主驾驶或室内机器人等领域的适用性。尽管其方法涉及多模态学习和自编码器架构，但研究背景和实验设计均远离文档图像理解的核心任务（如DocVQA、表格识别、布局分析等），且未体现对OCR幻觉、密集文本处理、细粒度视觉定位或长文档推理等问题的解决方案。其迁移至DIU领域的路径不明确，属于通用多模态方法在非文档场景的应用，不符合本筛选标准中对‘上游创新工具’的要求。
Abstract The computer vision domain has greatly benefited from an abundance of data across many modalities to improve on various visual tasks. Recently, there has been a lot of focus on self-supervised pre-training methods through Masked Autoencoders (MAE) \cite{he2022masked,bachmann2022multimae}, usually used as a first step before optimizing for a downstream task, such as classification or regression. This is very useful as it doesn't require any manually labeled data. In this work, we introduce Probabilistic Hyper-Graphs using Masked Autoencoders (PHG-MAE): a novel model that unifies the classical work on neural graphs \cite{leordeanu2021semi} with the modern approach of masked autoencoders under a common theoretical framework. Through random masking of entire modalities, not just patches, the model samples from the distribution of hyper-edges on each forward pass. Additionally, the model adapts the standard MAE algorithm by combining pre-training and fine-tuning into a single training loop. Moreover, our approach enables the creation of inference-time ensembles which, through aggregation, boost the final prediction performance and consistency. Lastly, we show that we can apply knowledge distillation on top of the ensembles with little loss in performance, even with models that have fewer than 1M parameters. While our work mostly focuses on outdoor UAV scenes that contain multiple world interpretations and modalities, the same steps can be followed in other similar domains, such as autonomous driving or indoor robotics. In order to streamline the process of integrating external pre-trained experts for computer vision multi-modal multi-task learning (MTL) scenarios, we developed a data-pipeline software. Using this tool, we have created and released a fully-automated extension of the Dronescapes dataset. All the technical details, code and reproduction steps are publicly released.
Authors Pîrvu Mihai-Cristian, Marius Leordeanu
Comment Submitted to Neurocomputing
Categories Computer Vision and Pattern Recognition
3
Alternating Perception-Reasoning for Hallucination-Resistant Video Understanding
Title CN 交替感知-推理以实现抗幻觉的视频理解
Keywords 视频理解 · 幻觉抑制 · 感知-推理循环 · 偏好学习 · 事实评估器
Summary 本文提出一种交替进行感知与推理的框架（PLR），通过分段描述带时间戳的视频片段并引入事实感知评估器（FAE）提供反幻觉奖励，以缓解视频大模型中的幻觉问题。实验表明该方法在多个参数规模下达到SOTA，并具备良好数据效率。
Reason 该论文研究的是视频理解中的幻觉问题，虽然其提出的交替感知-推理循环和反幻觉奖励机制在理论上具有方法论价值，但其核心任务（视频时序推理）与文档图像理解（DIU）差异较大。尽管‘感知-推理循环’和‘基于评估器的奖励机制’可能被构想用于长文档多步推理或减少OCR幻觉，但视频中的时间动态建模与文档的空间结构理解存在本质不同，迁移路径不明确。且该工作属于特定领域（视频）的方法封装，未针对文档特有的密集文本、排版复杂性或视觉token压缩等痛点设计，极易被视为平行应用的变体。
Abstract Sufficient visual perception is the foundation of video reasoning. Nevertheless, existing Video Reasoning LLMs suffer from perception shortcuts, relying on a flawed single-step perception paradigm. This paradigm describes the video and then conducts reasoning, which runs the risk of insufficient evidence and emergent hallucinations. To address these issues, we introduce a new framework that integrates a loop-based paradigm with an anti-hallucination reward. First, to address the insufficient evidence, we introduce the Perception Loop Reasoning (PLR) paradigm. Instead of describing the video at once, each loop requires the model to describe a video segment with precise timestamps, analyze this segment, and decide the next action. Second, for the risk of hallucinations, the Factual-Aware Evaluator (FAE) evaluates each perception result as a reliable anti-hallucination reward. This reward encourages the model to provide sufficient and precise video evidence. Our FAE, which performs comparably to GPT-4o, is tuned on our AnetHallu-117K, a large-scale hallucination judgment preference dataset. Extensive experiments show that our Video-PLR achieves the state-of-the-art in both 3B and 7B parameter scales and has the best data efficiency. Our code, models, and datasets are released on: https://github.com/BoweiPu/VideoPLR.
Authors Bowei Pu, Chuanbin Liu, Yifan Ge, Peicheng Zhou, Yiwei Sun, Zhiying Lu, Jiankang Wang, Hongtao Xie
Comment 32 pages, 36 figures
Categories Computer Vision and Pattern Recognition
3
Can Modern Vision Models Understand the Difference Between an Object and a Look-alike?
Title CN 现代视觉模型能否理解对象与其相似物之间的区别？
Keywords 视觉语言模型 · 细粒度识别 · CLIP · 嵌入空间方向 · 跨模态检索
Summary 本文提出RoLA数据集以研究视觉语言模型是否能够区分真实物体与其外观相似的非实例（如玩具、绘画）。通过构建‘真实/相似’提示对，并在CLIP嵌入空间中估计区分方向，提升了跨模态检索和图像描述的质量。然而，该工作聚焦于通用视觉认知中的哲学性细粒度区分，不针对文档内容理解或布局分析。
Reason 该论文研究的是视觉语言模型（如CLIP）是否能区分真实物体与其外观相似的仿制品（如玩具、雕像、涂鸦等），其核心任务属于通用图像理解中的细粒度感知问题，使用RoLA数据集进行跨模态检索和描述生成。虽然涉及VLM的语义分辨能力，但其关注点并非文档结构、文本内容理解或排版逻辑，与文档图像理解（DIU）的核心痛点（如OCR幻觉、密集文本识别、表格解析）无直接关联。提出的方法——在嵌入空间中学习‘真实 vs. 相似’的方向向量——难以迁移到文档场景下的细粒度文本定位或去幻觉任务，且未涉及高分辨率处理、长序列建模或复杂布局分析等DIU关键挑战。
Abstract Recent advances in computer vision have yielded models with strong performance on recognition benchmarks; however, significant gaps remain in comparison to human perception. One subtle ability is to judge whether an image looks like a given object without being an instance of that object. We study whether vision-language models such as CLIP capture this distinction. We curated a dataset named RoLA (Real or Lookalike) of real and lookalike exemplars (e.g., toys, statues, drawings, pareidolia) across multiple categories, and first evaluate a prompt-based baseline with paired "real"/"lookalike" prompts. We then estimate a direction in CLIP's embedding space that moves representations between real and lookalike. Applying this direction to image and text embeddings improves discrimination in cross-modal retrieval on Conceptual12M, and also enhances captions produced by a CLIP prefix captioner.
Authors Itay Cohen, Ethan Fetaya, Amir Rosenfeld
Categories Computer Vision and Pattern Recognition
3
Hybrid Convolution and Frequency State Space Network for Image Compression
Title CN 用于图像压缩的混合卷积与频域状态空间网络
Keywords 图像压缩 · 状态空间模型 · 频域建模 · 卷积神经网络 · 率失真优化
Summary 本文提出HCFSSNet，一种结合卷积神经网络与频域状态空间模型的混合架构，用于学习型图像压缩。通过CNN提取高频局部结构，并设计VFSS模块捕捉低频长距离依赖，引入AFMM进行内容自适应的DCT频带加权，以及FSTAM模块用于熵模型中的频域感知上下文建模，在Kodak等多个数据集上实现了优于VTM的压缩性能。
Reason 该论文研究的是通用图像压缩（Learned Image Compression），其核心任务是率失真优化，目标是提升压缩效率与重建质量。尽管使用了视觉频率建模和状态空间模型等技术，但其方法设计面向的是自然图像的压缩场景，未涉及文档理解中的关键问题如OCR幻觉、文本识别、布局分析或视觉-语言推理。虽然频域建模在理论上可能对高分辨率文档处理有间接启发，但该工作本身并未针对文档图像特性（如密集文本、排版结构、语义解析）进行建模，也未提出可用于DIU的方法论创新，属于与文档理解无关的技术路线。
Abstract Learned image compression (LIC) has recently benefited from Transformer based and state space model (SSM) based architectures. Convolutional neural networks (CNNs) effectively capture local high frequency details, whereas Transformers and SSMs provide strong long range modeling capabilities but may cause structural information loss or ignore frequency characteristics that are crucial for compression. In this work we propose HCFSSNet, a Hybrid Convolution and Frequency State Space Network for LIC. HCFSSNet uses CNNs to extract local high frequency structures and introduces a Vision Frequency State Space (VFSS) block that models long range low frequency information. The VFSS block combines an Omni directional Neighborhood State Space (VONSS) module, which scans features horizontally, vertically and diagonally, with an Adaptive Frequency Modulation Module (AFMM) that applies content adaptive weighting of discrete cosine transform frequency components for more efficient bit allocation. To further reduce redundancy in the entropy model, we integrate AFMM with a Swin Transformer to form a Frequency Swin Transformer Attention Module (FSTAM) for frequency aware side information modeling. Experiments on the Kodak, Tecnick and CLIC Professional Validation datasets show that HCFSSNet achieves competitive rate distortion performance compared with recent SSM based codecs such as MambaIC, while using significantly fewer parameters. On Kodak, Tecnick and CLIC, HCFSSNet reduces BD rate over the VTM anchor by 18.06, 24.56 and 22.44 percent, respectively, providing an efficient and interpretable hybrid architecture for future learned image compression systems.
Authors Haodong Pan, Hao Wei, Yusong Wang, Nanning Zheng, Caigui Jiang
Comment 36 pages, 8 figures
Categories Computer Vision and Pattern Recognition
3
Dream-IF: Dynamic Relative EnhAnceMent for Image Fusion
Title CN Dream-IF：面向图像融合的动态相对增强
Keywords 图像融合 · 跨模态增强 · 退化建模 · 提示编码 · 多模态恢复
Summary 本文提出Dream-IF框架，通过量化不同模态在各层级的相对主导性，实现跨模态的动态相互增强，用于提升多源图像融合质量。引入基于提示的编码来捕捉特定退化信息，指导图像恢复过程。实验表明该方法在图像融合与增强任务上优于现有方法。
Reason 该论文研究的是多模态图像融合（如红外与可见光图像融合）中的增强问题，属于传统计算机视觉任务，核心关注点是跨模态退化建模与恢复，与文档图像理解（DIU）的核心痛点（如OCR幻觉、文本布局分析、表格识别、细粒度视觉语言对齐）无直接关联。虽然提出了‘动态相对增强’和提示编码机制，但其方法设计针对的是传感器级图像退化，难以迁移到以语义结构和逻辑推理为核心的文档场景。属于无关领域的技术路线，不符合DIU本题或可迁移工具的标准。
Abstract Image fusion aims to integrate comprehensive information from images acquired through multiple sources. However, images captured by diverse sensors often encounter various degradations that can negatively affect fusion quality. Traditional fusion methods generally treat image enhancement and fusion as separate processes, overlooking the inherent correlation between them; notably, the dominant regions in one modality of a fused image often indicate areas where the other modality might benefit from enhancement. Inspired by this observation, we introduce the concept of dominant regions for image enhancement and present a Dynamic Relative EnhAnceMent framework for Image Fusion (Dream-IF). This framework quantifies the relative dominance of each modality across different layers and leverages this information to facilitate reciprocal cross-modal enhancement. By integrating the relative dominance derived from image fusion, our approach supports not only image restoration but also a broader range of image enhancement applications. Furthermore, we employ prompt-based encoding to capture degradation-specific details, which dynamically steer the restoration process and promote coordinated enhancement in both multi-modal image fusion and image enhancement scenarios. Extensive experimental results demonstrate that Dream-IF consistently outperforms its counterparts. The code is publicly available.\footnote{ https://github.com/jehovahxu/Dream-IF
Authors Xingxin Xu, Bing Cao, Dongdong Li, Qinghua Hu, Pengfei Zhu
Categories Computer Vision and Pattern Recognition
3
SAM-MI: A Mask-Injected Framework for Enhancing Open-Vocabulary Semantic Segmentation with SAM
Title CN SAM-MI：一种通过掩码注入增强开放词汇语义分割的框架
Keywords 开放词汇语义分割 · SAM · 掩码注入 · 过分割 · 多模态分割
Summary 本文提出SAM-MI框架，通过文本引导的稀疏点采样、浅层掩码聚合和解耦掩码注入机制，改进了SAM在开放词汇语义分割中的过分割问题并提升了效率。实验显示其在MESS等基准上显著优于Grounded-SAM。
Reason 该论文属于开放词汇语义分割（OVSS）领域，核心是利用SAM模型提升通用图像分割性能，其技术动机和应用场景均集中在自然图像的像素级分类，与文档图像理解（DIU）中的文本识别、布局分析、表格结构化等任务无直接关联。尽管SAM本身具有通用分割能力，但本文提出的稀疏点提示、掩码聚合与解耦注入机制主要针对自然图像中物体边界的优化，无法直接迁移解决文档图像中的OCR幻觉、密集文本解析或复杂排版建模等问题。且未涉及多模态语言模型（VLM）在文档上的推理增强或视觉token压缩等DIU关键方向。
Abstract Open-vocabulary semantic segmentation (OVSS) aims to segment and recognize objects universally. Trained on extensive high-quality segmentation data, the segment anything model (SAM) has demonstrated remarkable universal segmentation capabilities, offering valuable support for OVSS. Although previous methods have made progress in leveraging SAM for OVSS, there are still some challenges: (1) SAM's tendency to over-segment and (2) hard combinations between fixed masks and labels. This paper introduces a novel mask-injected framework, SAM-MI, which effectively integrates SAM with OVSS models to address these challenges. Initially, SAM-MI employs a Text-guided Sparse Point Prompter to sample sparse prompts for SAM instead of previous dense grid-like prompts, thus significantly accelerating the mask generation process. The framework then introduces Shallow Mask Aggregation (SMAgg) to merge partial masks to mitigate the SAM's over-segmentation issue. Finally, Decoupled Mask Injection (DMI) incorporates SAM-generated masks for guidance at low-frequency and high-frequency separately, rather than directly combining them with labels. Extensive experiments on multiple benchmarks validate the superiority of SAM-MI. Notably, the proposed method achieves a 16.7% relative improvement in mIoU over Grounded-SAM on the MESS benchmark, along with a 1.6$\times$ speedup. We hope SAM-MI can serve as an alternative methodology to effectively equip the OVSS model with SAM.
Authors Lin Chen, Yingjian Zhu, Qi Yang, Xin Niu, Kun Ding, Shiming Xiang
Categories Computer Vision and Pattern Recognition
3
OmniRefiner: Reinforcement-Guided Local Diffusion Refinement
Title CN OmniRefiner：强化学习引导的局部扩散优化
Keywords 图像编辑 · 扩散模型 · 强化学习 · 参考引导生成 · 细节保持
Summary 本文提出OmniRefiner，一种基于强化学习引导的两阶段局部扩散优化框架，用于在参考图像指导下提升生成图像的像素级一致性。第一阶段通过微调单图像扩散编辑器实现全局结构保持的协同优化；第二阶段引入强化学习机制，显式优化局部细节准确性和语义一致性。实验表明该方法在多个参考引导修复基准上优于现有开源与商业模型。
Reason 该论文聚焦于参考图像引导下的图像生成与编辑任务，核心问题是图像细节的保真修复与局部优化，属于纯图像生成/编辑领域。尽管其使用了强化学习提升局部编辑能力，但整体工作与文档图像理解（DIU）中的OCR、布局分析、表格识别或VQA等任务无直接关联。方法基于扩散模型和VAE潜在空间操作，主要解决的是纹理一致性与视觉属性保持问题，难以迁移至文档场景中的文本细粒度识别、结构理解或逻辑推理。属于无关领域（纯图像生成/修复），命中负面清单第2条。
Abstract Reference-guided image generation has progressed rapidly, yet current diffusion models still struggle to preserve fine-grained visual details when refining a generated image using a reference. This limitation arises because VAE-based latent compression inherently discards subtle texture information, causing identity- and attribute-specific cues to vanish. Moreover, post-editing approaches that amplify local details based on existing methods often produce results inconsistent with the original image in terms of lighting, texture, or shape. To address this, we introduce \ourMthd{}, a detail-aware refinement framework that performs two consecutive stages of reference-driven correction to enhance pixel-level consistency. We first adapt a single-image diffusion editor by fine-tuning it to jointly ingest the draft image and the reference image, enabling globally coherent refinement while maintaining structural fidelity. We then apply reinforcement learning to further strengthen localized editing capability, explicitly optimizing for detail accuracy and semantic consistency. Extensive experiments demonstrate that \ourMthd{} significantly improves reference alignment and fine-grained detail preservation, producing faithful and visually coherent edits that surpass both open-source and commercial models on challenging reference-guided restoration benchmarks.
Authors Yaoli Liu, Ziheng Ouyang, Shengtao Lou, Yiren Song
Categories Computer Vision and Pattern Recognition
3
Context-Aware Token Pruning and Discriminative Selective Attention for Transformer Tracking
Title CN 用于Transformer跟踪的上下文感知令牌剪枝与判别性选择性注意力
Keywords Transformer跟踪 · 令牌剪枝 · 选择性注意力 · 视觉跟踪 · 背景抑制
Summary 本文提出CPDATrack，一种基于Transformer的目标跟踪框架，通过上下文感知的令牌剪枝和判别性选择性注意力机制来抑制背景干扰并提升效率。在GOT-10k等基准上达到SOTA性能。
Reason 该论文属于目标跟踪（visual tracking）领域，尽管提出了令牌剪枝和注意力机制优化，但其任务背景与文档图像理解（DIU）或DocVQA无关。方法聚焦于视频序列中的目标定位，无法直接迁移解决OCR幻觉、复杂版面分析或长文档推理等DIU核心问题。属于无关领域工作，且无明确可迁移的方法论创新。
Abstract One-stream Transformer-based trackers have demonstrated remarkable performance by concatenating template and search region tokens, thereby enabling joint attention across all tokens. However, enabling an excessive proportion of background search tokens to attend to the target template tokens weakens the tracker's discriminative capability. Several token pruning methods have been proposed to mitigate background interference; however, they often remove tokens near the target, leading to the loss of essential contextual information and degraded tracking performance. Moreover, the presence of distractors within the search tokens further reduces the tracker's ability to accurately identify the target. To address these limitations, we propose CPDATrack, a novel tracking framework designed to suppress interference from background and distractor tokens while enhancing computational efficiency. First, a learnable module is integrated between two designated encoder layers to estimate the probability of each search token being associated with the target. Based on these estimates, less-informative background tokens are pruned from the search region while preserving the contextual cues surrounding the target. To further suppress background interference, a discriminative selective attention mechanism is employed that fully blocks search-to-template attention in the early layers. In the subsequent encoder layers, high-probability target tokens are selectively extracted from a localized region to attend to the template tokens, thereby reducing the influence of background and distractor tokens. The proposed CPDATrack achieves state-of-the-art performance across multiple benchmarks, particularly on GOT-10k, where it attains an average overlap of 75.1 percent.
Authors Janani Kugarajeevan, Thanikasalam Kokul, Amirthalingam Ramanan, Subha Fernando
Categories Computer Vision and Pattern Recognition
3
Intelligent Image Search Algorithms Fusing Visual Large Models
Title CN 融合视觉大模型的智能图像搜索算法
Keywords 细粒度图像检索 · 视觉大模型 · 目标检测 · 零样本搜索 · 状态识别
Summary 本文提出DetVLM框架，通过结合YOLO检测器与视觉大模型实现细粒度图像检索。第一阶段用YOLO快速筛选包含目标组件的图像，第二阶段利用VLM进行召回增强，支持基于组件状态（如遮阳板是否放下）和零样本（如驾驶员是否戴口罩）的检索。实验显示在车辆组件数据集上达到94.82%的整体检索准确率。
Reason 该论文聚焦于细粒度图像检索任务，应用场景为安防和工业检测中的部件状态识别与零样本搜索。尽管使用了VLM进行二次验证并提出两阶段Pipeline，但其核心是将现有目标检测器（YOLO）与VLM结合用于特定下游任务，属于典型的‘平行应用’架构设计，未针对文档图像理解中的OCR幻觉、复杂版式或长文档推理等核心痛点提出可迁移的方法论创新。
Abstract Fine-grained image retrieval, which aims to find images containing specific object components and assess their detailed states, is critical in fields like security and industrial inspection. However, conventional methods face significant limitations: manual features (e.g., SIFT) lack robustness; deep learning-based detectors (e.g., YOLO) can identify component presence but cannot perform state-specific retrieval or zero-shot search; Visual Large Models (VLMs) offer semantic and zero-shot capabilities but suffer from poor spatial grounding and high computational cost, making them inefficient for direct retrieval. To bridge these gaps, this paper proposes DetVLM, a novel intelligent image search framework that synergistically fuses object detection with VLMs. The framework pioneers a search-enhancement paradigm via a two-stage pipeline: a YOLO detector first conducts efficient, high-recall component-level screening to determine component presence; then, a VLM acts as a recall-enhancement unit, performing secondary verification for components missed by the detector. This architecture directly enables two advanced capabilities: 1) State Search: Guided by task-specific prompts, the VLM refines results by verifying component existence and executing sophisticated state judgments (e.g., "sun visor lowered"), allowing retrieval based on component state. 2) Zero-shot Search: The framework leverages the VLM's inherent zero-shot capability to recognize and retrieve images containing unseen components or attributes (e.g., "driver wearing a mask") without any task-specific training. Experiments on a vehicle component dataset show DetVLM achieves a state-of-the-art overall retrieval accuracy of 94.82\%, significantly outperforming detection-only baselines. It also attains 94.95\% accuracy in zero-shot search for driver mask-wearing and over 90\% average accuracy in state search tasks.
Authors Kehan Wang, Tingqiong Cui, Yang Zhang, Yu Chen, Shifeng Wu, Zhenzhang Li
Comment 31 pages,7 figures
Categories Computer Vision and Pattern Recognition
3
VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering
Title CN VeriSciQA：一种用于科学视觉问答的自动验证数据集
Keywords 视觉问答 · 数据集构建 · 生成-验证框架 · 科学图表 · 多模态
Summary 本文提出了VeriSciQA，一个包含2万多个问答对的科学图表视觉问答数据集，通过‘生成-验证’框架提升合成数据质量，旨在推动开源模型在科学图表理解上的性能。实验表明该数据集更具挑战性且能有效提升模型表现。
Reason 该论文聚焦于科学图表理解（SVQA），属于特定领域的下游应用，其核心贡献是构建一个面向科学文献图表问答的数据集，并未提出可迁移至文档图像理解（DIU）的通用方法论。尽管其生成-验证框架具有一定设计思路，但整体仍局限于科学论文这一垂直场景，且重点在于数据合成而非解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点，属于‘平行下游应用’，不符合本筛选原则。
Abstract Large Vision-Language Models (LVLMs) show promise for scientific applications, yet open-source models still struggle with Scientific Visual Question Answering (SVQA), namely answering questions about figures from scientific papers. A key bottleneck lies in the lack of public, large-scale, high-quality SVQA datasets. Although recent work uses LVLMs to synthesize data at scale, we identify systematic errors in their resulting QA pairs, stemming from LVLMs' inherent limitations and information asymmetry between figures and text. To address these challenges, we propose a verification-centric Generate-then-Verify framework that first generates QA pairs with figure-associated textual context, then applies cross-modal consistency checks against figures along with auxiliary filters to eliminate erroneous pairs. We instantiate this framework to curate VeriSciQA, a dataset of 20,351 QA pairs spanning 20 scientific domains and 12 figure types. VeriSciQA poses a challenging benchmark for open-source models, with a substantial accuracy gap between the leading open-source models (64%) and a proprietary model (82%). Moreover, models fine-tuned on VeriSciQA achieve consistent improvements on SVQA benchmarks, with performance gains that scale with data size and surpass models trained on existing datasets. Human evaluation further validates the superior correctness of VeriSciQA. Together, these evidences demonstrate that continued data expansion by our scalable framework can further advance SVQA capability in the open-source community.
Authors Yuyi Li, Daoyuan Chen, Zhen Wang, Yutong Lu, Yaliang Li
Categories Computer Vision and Pattern Recognition
3
MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustness NeurIPS 2025 DB Track
Title CN MMPerspective：多模态大模型理解视角吗？一个关于视角感知、推理与鲁棒性的综合基准
Keywords 多模态大模型 · 视角理解 · 空间推理 · 基准测试 · 鲁棒性
Summary 本文提出了MMPerspective，首个专门评估多模态大模型（MLLM）对视角理解能力的基准，涵盖感知、推理和鲁棒性三个维度，包含2711个真实与合成图像样本和5083个问答对。通过对43种主流MLLM的评测，发现模型在表层感知任务上表现尚可，但在组合推理和扰动下的空间一致性方面存在显著缺陷。研究揭示了模型结构、规模与视角理解能力之间的关系，并验证了思维链提示的有效性。
Reason 该论文提出的是一个面向通用视觉-语言模型的视角理解基准，任务集中在透视几何、消失点感知、3D空间线关系等与自然图像场景相关的高级视觉推理问题。虽然涉及MLLM的空间理解能力，但其核心关注点与文档图像理解（DIU）中的关键挑战——如OCR幻觉、密集文本识别、复杂版式分析、表格结构提取或长文档逻辑推理——无直接关联。文档图像通常为正交投影（无透视变形），因此该基准所评测的能力在DIU中几乎不构成实际需求，迁移价值极低。
Abstract Understanding perspective is fundamental to human visual perception, yet the extent to which multimodal large language models (MLLMs) internalize perspective geometry remains unclear. We introduce MMPerspective, the first benchmark specifically designed to systematically evaluate MLLMs' understanding of perspective through 10 carefully crafted tasks across three complementary dimensions: Perspective Perception, Reasoning, and Robustness. Our benchmark comprises 2,711 real-world and synthetic image instances with 5,083 question-answer pairs that probe key capabilities, such as vanishing point perception and counting, perspective type reasoning, line relationship understanding in 3D space, invariance to perspective-preserving transformations, etc. Through a comprehensive evaluation of 43 state-of-the-art MLLMs, we uncover significant limitations: while models demonstrate competence on surface-level perceptual tasks, they struggle with compositional reasoning and maintaining spatial consistency under perturbations. Our analysis further reveals intriguing patterns between model architecture, scale, and perspective capabilities, highlighting both robustness bottlenecks and the benefits of chain-of-thought prompting. MMPerspective establishes a valuable testbed for diagnosing and advancing spatial understanding in vision-language systems. Resources available at: https://yunlong10.github.io/MMPerspective/
Authors Yolo Y. Tang, Pinxin Liu, Zhangyun Tan, Mingqian Feng, Rui Mao, Chao Huang, Jing Bi, Yunzhong Xiao, Susan Liang, Hang Hua, Ali Vosoughi, Luchuan Song, Zeliang Zhang, Chenliang Xu
Comment Accepted to NeurIPS 2025 DB Track. Rating: 5,5,5,5
Categories Computer Vision and Pattern Recognition
3
PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model
Title CN PaddleOCR-VL：通过0.9B超紧凑视觉语言模型提升多语言文档解析
Keywords 文档解析 · 多语言OCR · 视觉语言模型 · 轻量化模型 · PaddleOCR
Summary 本文提出了PaddleOCR-VL，一个基于0.9B参数量视觉语言模型的多语言文档解析系统。该模型结合了动态分辨率视觉编码器与小型语言模型，在多种文档元素识别任务上实现了SOTA性能，并强调低资源消耗和高推理速度。适用于实际部署场景。
Reason 该论文属于典型的‘平行下游应用’，其核心是将现有技术（NaViT、ERNIE-4.5）集成到一个工程系统中，用于多语言文档解析。尽管声称支持109种语言且资源消耗低，但未提出新的方法论或底层架构创新。重点在于‘部署实用性’和‘多语言覆盖’，而非解决文档理解中的根本性挑战（如OCR幻觉、复杂排版推理、视觉token压缩等）。其技术路线为典型的VLM封装式应用，缺乏对DIU领域有范式影响的创新。
Abstract In this report, we propose PaddleOCR-VL, a SOTA and resource-efficient model tailored for document parsing. Its core component is PaddleOCR-VL-0.9B, a compact yet powerful vision-language model (VLM) that integrates a NaViT-style dynamic resolution visual encoder with the ERNIE-4.5-0.3B language model to enable accurate element recognition. This innovative model efficiently supports 109 languages and excels in recognizing complex elements (e.g., text, tables, formulas, and charts), while maintaining minimal resource consumption. Through comprehensive evaluations on widely used public benchmarks and in-house benchmarks, PaddleOCR-VL achieves SOTA performance in both page-level document parsing and element-level recognition. It significantly outperforms existing solutions, exhibits strong competitiveness against top-tier VLMs, and delivers fast inference speeds. These strengths make it highly suitable for practical deployment in real-world scenarios. Code is available at https://github.com/PaddlePaddle/PaddleOCR .
Authors Cheng Cui, Ting Sun, Suyin Liang, Tingquan Gao, Zelun Zhang, Jiaxuan Liu, Xueqing Wang, Changda Zhou, Hongen Liu, Manhui Lin, Yue Zhang, Yubo Zhang, Handong Zheng, Jing Zhang, Jun Zhang, Yi Liu, Dianhai Yu, Yanjun Ma
Comment Github Repo: https://github.com/PaddlePaddle/PaddleOCR
Categories Computer Vision and Pattern Recognition
3
GaINeR: Geometry-Aware Implicit Network Representation
Title CN GaINeR：几何感知的隐式网络表示
Keywords 隐式神经表示 · 图像重建 · 几何感知 · 局部编辑 · 高斯分布
Summary 本文提出GaINeR，一种结合可训练高斯分布与神经网络的几何感知隐式神经表示方法，用于连续表示2D图像。通过检索最近的K个高斯并加权聚合其嵌入，模型能实现高质量图像重建、局部编辑和几何结构解释性。然而，该方法主要面向通用图像建模，未涉及文本内容理解或文档结构建模。
Reason 该论文属于计算机视觉中的图像表示方法，聚焦于通用2D图像的隐式神经表示（INR），虽在图像重建、超分辨率等方面有潜力，但其核心目标并非文档图像理解任务（如DocVQA、布局分析、表格识别等）。尽管高分辨率处理能力可能间接有助于文档处理，但该方法未针对OCR幻觉、复杂排版或文本密集性等DIU核心痛点进行设计，且缺乏向文档领域迁移的具体路径。整体属于通用视觉表示研究，与文档理解关联较弱。
Abstract Implicit Neural Representations (INRs) have become an essential tool for modeling continuous 2D images, enabling high-fidelity reconstruction, super-resolution, and compression. Popular architectures such as SIREN, WIRE, and FINER demonstrate the potential of INR for capturing fine-grained image details. However, traditional INRs often lack explicit geometric structure and have limited capabilities for local editing or integration with physical simulation, restricting their applicability in dynamic or interactive settings. To address these limitations, we propose GaINeR: Geometry-Aware Implicit Network Representation, a novel framework for 2D images that combines trainable Gaussian distributions with a neural network-based INR. For a given image coordinate, the model retrieves the K nearest Gaussians, aggregates distance-weighted embeddings, and predicts the RGB value via a neural network. This design enables continuous image representation, interpretable geometric structure, and flexible local editing, providing a foundation for physically aware and interactive image manipulation. The official implementation of our method is publicly available at https://github.com/WJakubowska/GaINeR.
Authors Weronika Jakubowska, Mikołaj Zieliński, Rafał Tobiasz, Krzysztof Byrski, Maciej Zięba, Dominik Belter, Przemysław Spurek
Comment 16 pages, 16 figures
Categories Computer Vision and Pattern Recognition
3
MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learning NeurIPS'25
Title CN MANGO：基于多模态注意力的归一化流融合学习方法
Keywords 多模态融合 · 归一化流 · 交叉注意力 · 可逆网络 · NeurIPS
Summary 本文提出MANGO，一种基于归一化流的多模态融合方法，通过引入可逆交叉注意力（ICA）层和三种新型交叉注意力机制（MMCA、IMCA、LICA），实现对多模态数据间复杂关系的显式建模。该方法在语义分割、图像翻译和电影分类任务上达到SOTA性能，强调模型的可解释性与可追踪性。
Reason 该论文提出一种基于归一化流和可逆交叉注意力的多模态融合框架，核心创新在于显式建模多模态特征间的复杂相关性。然而，其实验任务为语义分割、图像翻译和电影分类，均非文档理解任务。尽管其多模态对齐机制理论上可能迁移至VLM，但未涉及文本布局、OCR错误处理或细粒度视觉-语言对齐等DIU核心挑战，且无任何与文档结构或语言模态结合的设计。属于通用多模态方法在非文档场景的应用，不符合DIU本题要求，亦无明确可迁移的技术路径。
Abstract Multimodal learning has gained much success in recent years. However, current multimodal fusion methods adopt the attention mechanism of Transformers to implicitly learn the underlying correlation of multimodal features. As a result, the multimodal model cannot capture the essential features of each modality, making it difficult to comprehend complex structures and correlations of multimodal inputs. This paper introduces a novel Multimodal Attention-based Normalizing Flow (MANGO) approach to developing explicit, interpretable, and tractable multimodal fusion learning. In particular, we propose a new Invertible Cross-Attention (ICA) layer to develop the Normalizing Flow-based Model for multimodal data. To efficiently capture the complex, underlying correlations in multimodal data in our proposed invertible cross-attention layer, we propose three new cross-attention mechanisms: Modality-to-Modality Cross-Attention (MMCA), Inter-Modality Cross-Attention (IMCA), and Learnable Inter-Modality Cross-Attention (LICA). Finally, we introduce a new Multimodal Attention-based Normalizing Flow to enable the scalability of our proposed method to high-dimensional multimodal data. Our experimental results on three different multimodal learning tasks, i.e., semantic segmentation, image-to-image translation, and movie genre classification, have illustrated the state-of-the-art (SoTA) performance of the proposed approach.
Authors Thanh-Dat Truong, Christophe Bobda, Nitin Agarwal, Khoa Luu
Comment Accepted to NeurIPS'25
Categories Computer Vision and Pattern Recognition
3
Intriguing Properties of Dynamic Sampling Networks
Title CN 动态采样网络的 intriguing 性质
Keywords 动态采样 · 可变形卷积 · 理论分析 · warping算子 · 训练稳定性
Summary 本文提出一种称为‘warping’的通用算子来统一建模现有的动态采样方法（如可变形卷积），并通过统计方法分析其前向与反向传播的不对称性及训练稳定性。研究还探讨了离散化效应，并引入一种基于梯度更新的损失景观可视化方法。
Reason 该论文研究动态采样机制（如可变形卷积、空间变换网络）的统一理论建模与统计分析，属于传统计算机视觉中的架构分析工作。虽然其对动态算子的理论建模具有一定深度，但并未针对文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、长文档推理）提出直接可迁移的方法。其所探讨的‘warping’操作和训练稳定性分析仍停留在通用视觉特征提取层面，缺乏与多模态语言模型（VLM）、高分辨率文档处理或细粒度文本定位的结合路径，难以直接赋能当前主流的端到端文档理解范式。
Abstract Dynamic sampling mechanisms in deep learning architectures have demonstrated utility across many computer vision models, though the theoretical analysis of these structures has not yet been unified. In this paper we connect the various dynamic sampling methods by developing and analyzing a novel operator which generalizes existing methods, which we term "warping". Warping provides a minimal implementation of dynamic sampling which is amenable to analysis, and can be used to reconstruct existing architectures including deformable convolutions, active convolutional units, and spatial transformer networks. Using our formalism, we provide statistical analysis of the operator by modeling the inputs as both IID variables and homogeneous random fields. Extending this analysis, we discover a unique asymmetry between the forward and backward pass of the model training. We demonstrate that these mechanisms represent an entirely different class of orthogonal operators to the traditional translationally invariant operators defined by convolutions. With a combination of theoretical analysis and empirical investigation, we find the conditions necessary to ensure stable training of dynamic sampling networks. In addition, statistical analysis of discretization effects are studied. Finally, we introduce a novel loss landscape visualization which utilizes gradient update information directly, to better understand learning behavior.
Authors Dario Morle, Reid Zaffino
Categories Computer Vision and Pattern Recognition
3
LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling
Title CN LongVT：通过原生工具调用激励“长视频思考”
Keywords 长视频理解 · 多模态大模型 · 工具调用 · 推理框架 · 时间定位
Summary 本文提出LongVT，一种用于长视频理解的端到端代理框架，通过交错的多模态工具链思维实现‘边思考边调用工具’，利用模型自身的时间定位能力裁剪关键片段并细化推理过程。为支持训练，作者构建了VideoSIAH数据集，并设计三阶段训练策略，在多个长视频QA基准上超越现有方法。
Reason 该论文研究的是长视频理解中的推理问题，提出了一种基于多模态工具链思维的代理框架，利用LMM的时间定位能力作为视频裁剪工具进行全局到局部的推理。尽管其‘思考-调用工具-聚焦细节’的范式在理论上对长文档处理有一定启发，但其任务域为视频（temporal sequence），与文档图像（spatial layout + structured text）存在本质差异。方法核心依赖于时间轴上的片段检索和重采样，无法直接迁移至文档的高分辨率空间结构理解或OCR-free文本识别等任务。属于无关领域工作，不符合DIU本题要求。
Abstract Large multimodal models (LMMs) have shown great potential for video reasoning with textual Chain-of-Thought. However, they remain vulnerable to hallucinations, especially when processing long-form videos where evidence is sparse and temporally dispersed. Inspired by how humans comprehend long videos - by first skimming globally and then examining relevant clips for details - we introduce LongVT, an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought. Specifically, we exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames. This global-to-local reasoning loop continues until answers are grounded in retrieved visual evidence. Given the scarcity of fine-grained question-answering (QA) data for the long video reasoning task, we curate and will release a data suite named VideoSIAH to facilitate both training and evaluation. Specifically, our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning, respectively. Our evaluation benchmark consists of 1,280 QA pairs that are carefully curated through a semi-automatic data pipeline with human-in-the-loop validation. With a meticulously designed three-stage training strategy and extensive empirical validation, LongVT consistently outperforms existing strong baselines across four challenging long-video understanding and reasoning benchmarks. Our codes, data, and model checkpoints are publicly available at https://github.com/EvolvingLMMs-Lab/LongVT .
Authors Zuhao Yang, Sudong Wang, Kaichen Zhang, Keming Wu, Sicong Leng, Yifan Zhang, Chengwei Qin, Shijian Lu, Xingxuan Li, Lidong Bing
Categories Computer Vision and Pattern Recognition
3
The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models AAAI 2026
Title CN 类比的奇妙案例：探究大语言模型中的类比推理
Keywords 类比推理 · 大语言模型 · 关系表示 · 结构对齐 · 认知机制
Summary 本文研究大语言模型在类比推理任务中的能力，发现模型虽能编码实体间的深层关系，但在将关系迁移至新实体时存在困难，且成功推理依赖于情境间强结构对齐。通过隐藏层表示修补可在一定程度上改善表现。整体揭示了LLM在高阶关系推理上的潜力与局限。
Reason 该论文研究大语言模型在类比推理任务上的表现，属于纯文本模态下的认知机制分析，未涉及视觉或文档模态。尽管其关于关系对齐和隐藏状态修补的发现具有一定理论价值，但方法完全基于文本类比（如比例类比、故事类比），不包含多模态建模、视觉输入处理或文档结构理解，无法迁移到文档图像理解中的核心问题（如OCR幻觉、复杂版面解析、细粒度定位等）。因此属于无关领域的基础NLP研究。
Abstract Analogical reasoning is at the core of human cognition, serving as an important foundation for a variety of intellectual activities. While prior work has shown that LLMs can represent task patterns and surface-level concepts, it remains unclear whether these models can encode high-level relational concepts and apply them to novel situations through structured comparisons. In this work, we explore this fundamental aspect using proportional and story analogies, and identify three key findings. First, LLMs effectively encode the underlying relationships between analogous entities; both attributive and relational information propagate through mid-upper layers in correct cases, whereas reasoning failures reflect missing relational information within these layers. Second, unlike humans, LLMs often struggle not only when relational information is missing, but also when attempting to apply it to new entities. In such cases, strategically patching hidden representations at critical token positions can facilitate information transfer to a certain extent. Lastly, successful analogical reasoning in LLMs is marked by strong structural alignment between analogous situations, whereas failures often reflect degraded or misplaced alignment. Overall, our findings reveal that LLMs exhibit emerging but limited capabilities in encoding and applying high-level relational concepts, highlighting both parallels and gaps with human cognition.
Authors Taewhoo Lee, Minju Song, Chanwoong Yoon, Jungwoo Park, Jaewoo Kang
Comment AAAI 2026
Categories Computation and Language
3
Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios AAAI-2026
Title CN 扩展大语言模型推测解码：大规模批次场景下的非自回归预测
Keywords 推测解码 · 非自回归模型 · LLM推理加速 · SpecFormer · 大批量推理
Summary 本文提出SpecFormer架构，通过结合单向和双向注意力机制，在保持序列生成一致性的前提下实现非自回归的并行推测解码，以提升大语言模型在大规模批次推理中的效率。实验表明该方法在降低训练和计算成本的同时实现了稳定的加速效果。
Reason 该论文聚焦于纯文本大模型推理加速中的推测解码技术，属于NLP系统优化范畴。尽管其在计算效率上有改进，但未涉及任何视觉模态、多模态对齐或文档结构信息处理。无法迁移解决文档图像理解中的核心痛点（如OCR幻觉、复杂排版、细粒度定位等），属于与DIU无关的平行技术路线。
Abstract Speculative decoding accelerates LLM inference by utilizing otherwise idle computational resources during memory-to-chip data transfer. Current speculative decoding methods typically assume a considerable amount of available computing power, then generate a complex and massive draft tree using a small autoregressive language model to improve overall prediction accuracy. However, methods like batching have been widely applied in mainstream model inference systems as a superior alternative to speculative decoding, as they compress the available idle computing power. Therefore, performing speculative decoding with low verification resources and low scheduling costs has become an important research problem. We believe that more capable models that allow for parallel generation on draft sequences are what we truly need. Recognizing the fundamental nature of draft models to only generate sequences of limited length, we propose SpecFormer, a novel architecture that integrates unidirectional and bidirectional attention mechanisms. SpecFormer combines the autoregressive model's ability to extract information from the entire input sequence with the parallel generation benefits of non-autoregressive models. This design eliminates the reliance on large prefix trees and achieves consistent acceleration, even in large-batch scenarios. Through lossless speculative decoding experiments across models of various scales, we demonstrate that SpecFormer sets a new standard for scaling LLM inference with lower training demands and reduced computational costs.
Authors Luohe Shi, Zuchao Li, Lefei Zhang, Baoyuan Qi, Guoming Liu, Hai Zhao
Comment accepted by AAAI-2026
Categories Computation and Language
3
Agentar-Scale-SQL: Advancing Text-to-SQL through Orchestrated Test-Time Scaling
Title CN Agentar-Scale-SQL：通过协同测试时扩展推进文本到SQL生成
Keywords Text-to-SQL · Test-time Scaling · 推理优化 · 语言模型 · 数据库接口
Summary 本文提出Agentar-Scale-SQL，一种面向Text-to-SQL任务的测试时扩展框架，通过内部强化推理、顺序迭代优化和并行生成-锦标赛选择三种策略协同提升性能，在BIRD基准上达到SOTA。然而该工作局限于单模态语言任务，缺乏对视觉输入或跨模态对齐的支持，不属于文档理解或多模态VLM范畴。
Reason 该论文属于典型的‘平行下游应用’，其核心是将已有的test-time scaling思想应用于Text-to-SQL这一特定任务，未提出适用于文档图像理解（DIU）的通用方法论创新。尽管其Orchestrated Scaling策略（如并行合成、迭代优化）在推理机制设计上有一定启发性，但整体框架紧密绑定数据库语义转换场景，难以迁移至OCR幻觉抑制、复杂版面分析或长文档多步推理等DIU核心痛点。且无证据表明其方法可泛化至多模态输入（如图像+文本），故视为无营养的垂直应用。
Abstract State-of-the-art (SOTA) Text-to-SQL methods still lag significantly behind human experts on challenging benchmarks like BIRD. Current approaches that explore test-time scaling lack an orchestrated strategy and neglect the model's internal reasoning process. To bridge this gap, we introduce Agentar-Scale-SQL, a novel framework leveraging scalable computation to improve performance. Agentar-Scale-SQL implements an Orchestrated Test-Time Scaling strategy that synergistically combines three distinct perspectives: i) Internal Scaling via RL-enhanced Intrinsic Reasoning, ii) Sequential Scaling through Iterative Refinement, and iii) Parallel Scaling using Diverse Synthesis and Tournament Selection. Agentar-Scale-SQL is a general-purpose framework designed for easy adaptation to new databases and more powerful language models. Extensive experiments show that Agentar-Scale-SQL achieves SOTA performance on the BIRD benchmark, reaching 81.67% execution accuracy on the test set and ranking first on the official leaderboard, demonstrating an effective path toward human-level performance.
Authors Pengfei Wang, Baolin Sun, Xuemei Dong, Yaxun Dai, Hongwei Yuan, Mengdie Chu, Yingqi Gao, Xiang Qi, Peng Zhang, Ying Yan
Categories Computation and Language
3
MedS$^3$: Towards Medical Slow Thinking with Self-Evolved Soft Dual-sided Process Supervision AAAI26
Title CN MedS³：通过自演化的软性双侧过程监督实现医学慢思考
Keywords 医学语言模型 · 强化学习 · 推理链监督 · 蒙特卡洛树搜索 · 偏好学习
Summary 本文提出MedS³，一种面向医学临床推理的小型化语言模型训练框架，利用课程采样和蒙特卡洛树搜索生成可验证的推理路径，并引入带有价值动态的软性双过程奖励模型来识别中间步骤中的错误。通过自我演化的方式进行策略优化，在多个医学基准上超越现有模型。
Reason 该论文属于特定领域的下游应用（医学语言模型），其核心贡献是将强化学习与蒙特卡洛树搜索用于提升医学推理任务中的推理链质量。尽管其推理机制（如MCTS、细粒度奖励建模）具有一定的上游方法论潜力，但整体框架高度针对医学临床场景设计，且未涉及视觉或多模态输入，无法直接迁移至文档图像理解中的OCR幻觉、排版复杂性或视觉-文本对齐等核心问题。属于‘平行下游应用’，不符合本研究方向对底层技术创新的要求。
Abstract Medical language models face critical barriers to real-world clinical reasoning applications. However, mainstream efforts, which fall short in task coverage, lack fine-grained supervision for intermediate reasoning steps, and rely on proprietary systems, are still far from a versatile, credible and efficient language model for clinical reasoning usage. To this end, we propose MedS3, a self-evolving framework that imparts robust reasoning capabilities to small, deployable models. Starting with 8,000 curated instances sampled via a curriculum strategy across five medical domains and 16 datasets, we use a small base policy model to conduct Monte Carlo Tree Search (MCTS) for constructing rule-verifiable reasoning trajectories. Self-explored reasoning trajectories ranked by node values are used to bootstrap the policy model via reinforcement fine-tuning and preference learning. Moreover, we introduce a soft dual process reward model that incorporates value dynamics: steps that degrade node value are penalized, enabling fine-grained identification of reasoning errors even when the final answer is correct. Experiments on eleven benchmarks show that MedS3 outperforms the previous state-of-the-art medical model by +6.45 accuracy points and surpasses 32B-scale general-purpose reasoning models by +8.57 points. Additional empirical analysis further demonstrates that MedS3 achieves robust and faithful reasoning behavior.
Authors Shuyang Jiang, Yusheng Liao, Zhe Chen, Ya Zhang, Yanfeng Wang, Yu Wang
Comment 20 pages;Accepted as a Main paper at AAAI26
Categories Computation and Language
3
More Bias, Less Bias: BiasPrompting for Enhanced Multiple-Choice Question Answering 41st ACM/SIGAPP Symposium On Applied Computing (SAC 2026), Main Conference
Title CN 更多偏见，更少偏见：用于增强多项选择题问答的BiasPrompting
Keywords BiasPrompting · 多项选择题问答 · 大语言模型 · 推理增强 · 提示工程
Summary 本文提出BiasPrompting方法，通过引导LLM为每个答案选项生成支持性推理，并在第二阶段综合这些推理做出最终选择，从而提升多项选择题的回答准确性。实验表明其在五个主流MCQA基准上均有显著提升。
Reason 该论文提出一种名为BiasPrompting的推理框架，用于提升大语言模型在多项选择题（MCQ）任务中的表现，核心是通过生成每个选项的支持性推理并进行综合评估来增强模型的推理能力。然而，该工作完全基于纯文本的多项选择题场景，未涉及任何视觉输入或文档模态，且应用场景为通用NLP任务。尽管其推理机制设计有一定启发性，但属于典型的‘平行下游应用’范式——即在现有LLM上添加提示策略以改进特定任务性能，缺乏对文档图像理解（DIU）中关键挑战（如OCR幻觉、复杂排版、细粒度定位等）的建模与迁移路径。无法直接或间接解决DIU的核心问题。
Abstract With the advancement of large language models (LLMs), their performance on multiple-choice question (MCQ) tasks has improved significantly. However, existing approaches face key limitations: answer choices are typically presented to LLMs without contextual grounding or explanation. This absence of context can lead to incomplete exploration of all possible answers, ultimately degrading the models' reasoning capabilities. To address these challenges, we introduce BiasPrompting, a novel inference framework that guides LLMs to generate and critically evaluate reasoning across all plausible answer options before reaching a final prediction. It consists of two components: first, a reasoning generation stage, where the model is prompted to produce supportive reasonings for each answer option, and then, a reasoning-guided agreement stage, where the generated reasonings are synthesized to select the most plausible answer. Through comprehensive evaluations, BiasPrompting demonstrates significant improvements in five widely used multiple-choice question answering benchmarks. Our experiments showcase that BiasPrompting enhances the reasoning capabilities of LLMs and provides a strong foundation for tackling complex and challenging questions, particularly in settings where existing methods underperform.
Authors Duc Anh Vu, Thong Nguyen, Cong-Duy Nguyen, Viet Anh Nguyen, Anh Tuan Luu
Comment Accepted at the 41st ACM/SIGAPP Symposium On Applied Computing (SAC 2026), Main Conference
Categories Computation and Language
3
Online-PVLM: Advancing Personalized VLMs with Online Concept Learning
Title CN Online-PVLM：通过在线概念学习推进个性化视觉语言模型
Keywords 个性化VLM · 在线学习 · 概念嵌入 · 双曲表示 · 视觉语言模型
Summary 本文提出Online-PVLM，一种基于双曲空间表示的个性化视觉语言模型框架，支持在测试阶段无需训练地动态学习新概念（如用户私有物体），并发布OP-Eval这一大规模在线概念学习评测集。实验表明其在个性化交互任务上性能领先。
Reason 该论文聚焦于个性化视觉语言模型中的用户特定概念（如识别用户的自行车）的实时学习，其核心贡献在于提出一种无需训练即可在测试时生成概念嵌入的方法，并构建了一个大规模评测基准。然而，这项工作属于典型的‘平行应用’范式——它关注的是如何让VLM适应个人物品识别这一特定场景，而非解决文档图像理解中的根本挑战（如OCR幻觉、复杂版式建模、长文档推理或高分辨率处理）。尽管其超球面表示和在线学习机制具有一定技术新颖性，但这些方法与文档结构、文本语义解析或视觉-语言对齐在文档上的细粒度需求无直接关联，难以迁移到DIU任务中。
Abstract Personalized Visual Language Models (VLMs) are gaining increasing attention for their formidable ability in user-specific concepts aligned interactions (e.g., identifying a user's bike). Existing methods typically require the learning of separate embeddings for each new concept, which fails to support real-time adaptation during testing. This limitation becomes particularly pronounced in large-scale scenarios, where efficient retrieval of concept embeddings is not achievable. To alleviate this gap, we propose Online-PVLM, a framework for online concept learning by leveraging hyperbolic representations. Our approach makes a train-free paradigm for concept embeddings generation at test time, making the use of personalized VLMs both scalable and efficient. In addition, we develop OP-Eval, a comprehensive and large-scale benchmark comprising 1,292 concepts and over 30K high-quality instances with diverse question types, designed to rigorously assess online concept learning in realistic scenarios. Extensive experiments demonstrate the state-of-the-art performance of our proposed framework. Our source code and dataset will be made available.
Authors Huiyu Bai, Runze Wang, Zhuoyun Du, Yiyang Zhao, Fengji Zhang, Haoyu Chen, Xiaoyong Zhu, Bo Zheng, Xuejiao Zhao
Comment Work in Progress
Categories Computation and Language
3
SAS: Simulated Attention Score
Title CN SAS：模拟注意力得分
Keywords 注意力机制 · Transformer · 参数效率 · 模型压缩 · 特征扩展
Summary 本文提出Simulated Attention Score（SAS），通过将低维注意力头映射到高维空间以模拟更大容量的模型，在保持参数量不变的情况下提升性能。同时引入PEAA以控制参数开销。实验显示其在多个任务上优于现有注意力变体。
Reason 该论文提出了一种通过低维表示投影到高维空间来模拟更多注意力头和更高特征维度的机制，旨在提升Transformer的表达能力而不增加参数量。尽管其方法在语言模型中可能有效，但论文未涉及任何视觉或文档模态，也未针对多模态对齐、高分辨率图像处理、细粒度文本定位等文档理解核心问题设计。其技术属于通用架构优化，缺乏向文档图像理解（DIU）迁移的具体路径或验证，且无证据表明能缓解OCR幻觉、复杂排版建模或长文档推理等问题。
Abstract The attention mechanism is a core component of the Transformer architecture. Various methods have been developed to compute attention scores, including multi-head attention (MHA), multi-query attention, group-query attention and so on. We further analyze the MHA and observe that its performance improves as the number of attention heads increases, provided the hidden size per head remains sufficiently large. Therefore, increasing both the head count and hidden size per head with minimal parameter overhead can lead to significant performance gains at a low cost. Motivated by this insight, we introduce Simulated Attention Score (SAS), which maintains a compact model size while simulating a larger number of attention heads and hidden feature dimension per head. This is achieved by projecting a low-dimensional head representation into a higher-dimensional space, effectively increasing attention capacity without increasing parameter count. Beyond the head representations, we further extend the simulation approach to feature dimension of the key and query embeddings, enhancing expressiveness by mimicking the behavior of a larger model while preserving the original model size. To control the parameter cost, we also propose Parameter-Efficient Attention Aggregation (PEAA). Comprehensive experiments on a variety of datasets and tasks demonstrate the effectiveness of the proposed SAS method, achieving significant improvements over different attention variants.
Authors Chuanyang Zheng, Jiankai Sun, Yihang Gao, Yuehao Wang, Peihao Wang, Jing Xiong, Liliang Ren, Hao Cheng, Janardhan Kulkarni, Yelong Shen, Atlas Wang, Mac Schwager, Anderson Schneider, Xiaodong Liu, Jianfeng Gao
Comment Tech Report
Categories Computation and Language
3
EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruning
Title CN EfficientXpert：通过传播感知剪枝实现大语言模型的高效领域自适应
Keywords 大语言模型 · 领域自适应 · 模型压缩 · 剪枝 · LoRA
Summary 本文提出EfficientXpert，一种结合传播感知剪枝与高效适配器更新的轻量级领域自适应框架，可在保持98%性能的同时实现40%的模型稀疏化，在医疗与法律任务上优于现有压缩方法。研究表明通用剪枝掩码在不同领域中效果下降，需采用领域自适应策略。
Reason 该论文属于典型的‘平行下游应用’类型，其核心是将LoRA微调与剪枝结合，用于法律、医疗等特定领域的模型压缩。虽然方法上提出了传播感知剪枝（Foresight Mask）和部分Brain Surgeon算法，但整体框架仍局限于‘为垂直领域构建轻量专家模型’的应用范式，并未针对文档图像理解中的OCR幻觉、复杂版面解析、细粒度定位或高分辨率多模态输入等核心问题提出解决方案。其技术路线难以直接迁移至DIU所需的视觉-语言对齐或视觉表示优化任务。
Abstract The rapid advancement of large language models (LLMs) has increased the demand for domain-specialized variants in areas such as law, healthcare, and finance. However, their large size remains a barrier to deployment in resource-constrained environments, and existing compression methods either generalize poorly across domains or incur high overhead. In this work, we propose \textbf{EfficientXpert}, a lightweight domain-pruning framework that combines a propagation-aware pruning criterion (Foresight Mask) with an efficient adapter-update algorithm (Partial Brain Surgeon). Integrated into the LoRA fine-tuning process, EfficientXpert enables a one-step transformation of general pretrained models into sparse, domain-adapted experts. Across health and legal tasks, it retains up to 98% of dense-model performance at 40% sparsity, outperforming state-of-the-art methods. Further analysis reveals substantial domain-dependent structural shifts that degrade the effectiveness of general pruning masks, underscoring the need for adaptive, domain-aware pruning strategies tailored to each domain.
Authors Songlin Zhao, Michael Pitts, Zhuwei Qin
Categories Computation and Language
3
Filtering with Self-Attention and Storing with MLP: One-Layer Transformers Can Provably Acquire and Extract Knowledge
Title CN 通过自注意力过滤并用MLP存储：单层Transformer可证明地获取和提取知识
Keywords 知识存储 · 幻觉 · MLP · Transformer理论 · 微调
Summary 本文提出首个涵盖MLP模块、OOD适应性和自回归目标的理论框架，用于分析单层Transformer在预训练中知识获取与微调后知识提取的能力。理论与实验证明，在适当条件下模型能有效保留并提取未在微调中重现的知识，否则将产生幻觉。研究揭示了低秩微调的有效性机制，并在PopQA等基准上验证了理论发现。
Reason 该论文聚焦于语言模型中知识存储与提取的理论分析，核心关注点是MLP在知识记忆中的作用以及微调过程中的泛化与幻觉问题。虽然其对‘幻觉’成因的理论建模具有一定启发性，但研究对象为纯文本LLM，未涉及视觉输入或文档多模态场景。尽管结论可能间接启发DIU领域对VLM中幻觉机制的理解，但缺乏对OCR幻觉、视觉-文本对齐或多模态训练动态的直接建模，属于通用NLP理论工作，无法直接迁移解决文档理解的核心痛点。
Abstract Modern large language models (LLMs) demonstrate exceptional performance on knowledge-intensive tasks, yet the theoretical mechanisms underlying knowledge acquisition (storage and memorization) during pre-training and extraction (retrieval and recall) during inference after fine-tuning remain poorly understood. Although prior theoretical studies have explored these processes through analyses of training dynamics, they overlook critical components essential for a comprehensive theory: (1) the multi-layer perceptron (MLP), empirically identified as the primary module for knowledge storage; (2) out-of-distribution (OOD) adaptivity, which enables LLMs to generalize to unseen scenarios post-pre-training; and (3) next-token prediction, the standard autoregressive objective that encodes knowledge as conditional probabilities. In this work, we introduce, to the best of our knowledge, the first theoretical framework that addresses these limitations by examining the training dynamics of one-layer transformers. Under regularity assumptions, we establish that: (i) transformers attain near-optimal training loss during pre-training, demonstrating effective knowledge acquisition; (ii) given a sufficiently large fine-tuning dataset and appropriate data multiplicity conditions, transformers achieve low generalization error on factual knowledge acquired during pre-training but not revisited in fine-tuning, indicating robust knowledge extraction; and (iii) violation of these conditions leads to elevated generalization error, manifesting as hallucinations. Our analysis encompasses both full fine-tuning and low-rank fine-tuning, yielding insights into the efficacy of practical low-rank adaptation methods. We validate our theoretical findings through experiments on synthetic datasets and the real-world PopQA benchmark, employing GPT-2 and Llama-3.2-1B models.
Authors Ruichen Xu, Kexin Chen
Categories Computation and Language
3
CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoning
Title CN CLaRa：通过连续潜在推理桥接检索与生成
Keywords RAG · Latent Reasoning · Context Compression · End-to-End Training · Differentiable Retrieval
Summary 本文提出CLaRa，一种通过连续潜在空间实现检索与生成统一优化的框架，采用语义保持的压缩向量和端到端训练提升RAG性能。实验显示其在多个QA基准上优于基线方法。
Reason 该论文聚焦于通用RAG框架下的长上下文压缩与检索-生成联合优化，属于典型的‘平行应用’类方法改进，其技术路线（如SCP数据合成、可微top-k）虽有创新，但核心问题域为开放域问答，未涉及任何视觉模态或文档结构信息处理。尽管其‘压缩+联合优化’思想在理论上可能对长文档VLM的推理链构建有间接启发，但缺乏对OCR幻觉、布局理解、细粒度定位等DIU核心痛点的直接建模，且无多模态扩展设计，迁移价值有限。
Abstract Retrieval-augmented generation (RAG) enhances large language models (LLMs) with external knowledge but still suffers from long contexts and disjoint retrieval-generation optimization. In this work, we propose CLaRa (Continuous Latent Reasoning), a unified framework that performs embedding-based compression and joint optimization in a shared continuous space. To obtain semantically rich and retrievable compressed vectors, we introduce SCP, a key-preserving data synthesis framework using QA and paraphrase supervision. CLaRa then trains the reranker and generator end-to-end via a single language modeling loss, with gradients flowing through both modules using a differentiable top-k estimator. Theoretically, this unified optimization aligns retrieval relevance with answer quality. Experiments across multiple QA benchmarks show that CLaRa achieves state-of-the-art compression and reranking performance, often surpassing text-based fine-tuned baselines.
Authors Jie He, Richard He Bai, Sinead Williamson, Jeff Z. Pan, Navdeep Jaitly, Yizhe Zhang
Categories Computation and Language
3
A Psychology-based Unified Dynamic Framework for Curriculum Learning Computational Linguistics
Title CN 一种基于心理学的课程学习统一动态框架
Keywords 课程学习 · 项目反应理论 · 训练策略 · 难度评估 · 语言模型微调
Summary 本文提出PUDF框架，利用项目反应理论（IRT）结合人工群体响应（AC）量化训练样本难度，并通过模型能力估计动态选择数据，实现更高效的课程学习。实验显示该方法能加快收敛并提升性能。然而，该工作完全基于纯文本任务，未涉及任何视觉或多模态成分，与文档图像理解的实际挑战脱节。
Reason 该论文提出了一种基于心理测量学（项目反应理论IRT）的课程学习框架，用于优化模型训练过程中样本的难易排序与调度。尽管其在训练策略上有理论创新，但属于通用NLP训练方法的改进，并未针对文档图像理解（DIU）中的核心问题（如OCR幻觉、复杂版面解析、视觉-文本对齐等）设计，也未涉及多模态或视觉输入。其方法难以直接迁移至DIU场景，尤其是处理高分辨率图像、细粒度定位或视觉语言联合推理等任务。因此属于通用NLP方法在语言模型微调中的应用，不构成对DIU的技术推动。
Abstract Directly learning from examples of varying difficulty levels is often challenging for both humans and machine learning models. A more effective strategy involves exposing learners to examples in a progressive order from easy to difficult. Curriculum Learning (CL) has been proposed to implement this strategy in machine learning model training. However, two key challenges persist in CL framework design: defining the difficulty of training data and determining the appropriate amount of data to input at each training step. Drawing inspiration from psychometrics, this paper presents a Psychology-based Unified Dynamic Framework for Curriculum Learning (PUDF). We quantify the difficulty of training data by applying Item Response Theory (IRT) to responses from Artificial Crowds (AC). This theory-driven IRT-AC approach leads to global (i.e., model-independent) and interpretable difficulty values. Leveraging IRT, we propose a training strategy, Dynamic Data Selection via Model Ability Estimation (DDS-MAE), to schedule the appropriate amount of data during model training. Since our difficulty labeling and model ability estimation are based on a consistent theory, namely IRT, their values are comparable within the same scope, potentially leading to aligned training data selection and faster convergence compared to the other CL methods. Experimental results demonstrate that fine-tuning pre-trained large language models with PUDF leads to higher accuracy and faster convergence on a suite of benchmark datasets compared to standard fine-tuning and state-of-the-art CL methods. Ablation studies and downstream analyses further validate the impact of PUDF for CL.
Authors Guangyu Meng, Qingkai Zeng, John P. Lalor, Hong Yu
Comment Accepted for publication in Computational Linguistics. This is a pre-MIT Press publication version. Code available at https://github.com/nd-ball/cl-irt
Categories Computation and Language
3
SAGE: An Agentic Explainer Framework for Interpreting SAE Features in Language Models
Title CN SAGE：一种用于解释语言模型中SAE特征的智能体解释框架
Keywords 大语言模型 · 可解释性 · 稀疏自编码器 · 智能体 · 特征解释
Summary 本文提出SAGE，一个基于智能体的框架，用于自动解释稀疏自编码器（SAE）在大语言模型中发现的特征。该框架通过生成多种假设、设计实验验证并基于激活反馈迭代优化解释，提升了特征解释的准确性和可靠性。实验表明SAGE在多个语言模型的SAE特征上优于现有方法。
Reason 该论文聚焦于语言模型内部表示的可解释性，具体针对稀疏自编码器（SAE）提取特征的解释问题，属于纯NLP模型可解释性研究。尽管其提出的‘智能体式’主动解释框架在方法论上有一定新颖性，但完全未涉及视觉模态或文档图像理解任务，与文档图像理解（DIU/DocVQA）的核心痛点如OCR幻觉、复杂排版、细粒度定位等无直接关联。且属于典型的‘平行应用’范式——将Agent用于某个解释任务，而非提出可迁移至DIU的底层架构或推理机制。
Abstract Large language models (LLMs) have achieved remarkable progress, yet their internal mechanisms remain largely opaque, posing a significant challenge to their safe and reliable deployment. Sparse autoencoders (SAEs) have emerged as a promising tool for decomposing LLM representations into more interpretable features, but explaining the features captured by SAEs remains a challenging task. In this work, we propose SAGE (SAE AGentic Explainer), an agent-based framework that recasts feature interpretation from a passive, single-pass generation task into an active, explanation-driven process. SAGE implements a rigorous methodology by systematically formulating multiple explanations for each feature, designing targeted experiments to test them, and iteratively refining explanations based on empirical activation feedback. Experiments on features from SAEs of diverse language models demonstrate that SAGE produces explanations with significantly higher generative and predictive accuracy compared to state-of-the-art baselines.an agent-based framework that recasts feature interpretation from a passive, single-pass generation task into an active, explanationdriven process. SAGE implements a rigorous methodology by systematically formulating multiple explanations for each feature, designing targeted experiments to test them, and iteratively refining explanations based on empirical activation feedback. Experiments on features from SAEs of diverse language models demonstrate that SAGE produces explanations with significantly higher generative and predictive accuracy compared to state-of-the-art baselines.
Authors Jiaojiao Han, Wujiang Xu, Mingyu Jin, Mengnan Du
Categories Computation and Language
3
VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflection
Title CN VDC-Agent：当视频细粒度描述器通过代理式自我反思实现自我进化
Keywords 视频描述 · 自反思 · 偏好优化 · 无监督学习 · 多模态大模型
Summary 本文提出VDC-Agent，一种无需人工标注或大教师模型的视频细粒度描述自进化框架。该框架通过生成字幕、基于原则的评分与文本建议、提示词优化形成闭环，并在质量退化时利用思维链进行自我修正。在无标签视频上运行后构建出包含1.8万对的偏好数据集VDC-Agent-19K，并采用由易到难的课程学习方式对基础MLLM进行直接偏好优化。基于Qwen2.5-VL-7B-Instruct的模型在VDC基准上达到49.08%的平均准确率，超越现有专用方法。
Reason 该论文研究的是视频细粒度描述（Video Detailed Captioning）任务，属于视频理解领域，且其方法框架围绕视频内容生成与自反馈优化展开。尽管引入了Agent式的自我反思和偏好优化机制，但其应用场景和输入模态（长视频序列）与文档图像理解（DIU）差异巨大。文档图像理解的核心痛点如OCR幻觉、密集文本识别、版面分析、细粒度视觉定位等，在该工作中无直接对应或可迁移的机制。此外，该工作未涉及任何文档、文本布局或结构化信息处理，属于典型的无关领域研究。
Abstract We present VDC-Agent, a self-evolving framework for Video Detailed Captioning that requires neither human annotations nor larger teacher models. The agent forms a closed loop of caption generation, principle-guided scoring (score and textual suggestions), and prompt refinement. When caption quality regresses, a self-reflection path leverages the previous chain-of-thought to amend the update. Running this process on unlabeled videos produces trajectories of (caption, score) pairs. We convert the trajectories into preference tuples and filter out samples with JSON parsing errors, resulting in VDC-Agent-19K, which contains 18,886 automatically constructed pairs. We then fine-tune the base MLLM on this dataset using an easy-to-hard curriculum direct preference optimization. Built on Qwen2.5-VL-7B-Instruct, our VDC-Agent-7B attains state-of-the-art performance on the VDC benchmark with 49.08% average accuracy and 2.50 score, surpassing specialized video captioners and improving over the base model by +5.13% accuracy and +0.27 score at similar inference cost.
Authors Qiang Wang, Xinyuan Gao, SongLin Dong, Jizhou Han, Jiangyang Li, Yuhang He, Yihong Gong
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
SLMFix: Leveraging Small Language Models for Error Fixing with Reinforcement Learning
Title CN SLMFix：利用小型语言模型通过强化学习进行错误修复
Keywords 小语言模型 · 强化学习 · 程序修复 · 代码生成 · 领域特定语言
Summary 本文提出SLMFix，一种结合小语言模型与强化学习的代码修复流程，用于修正大模型生成代码中的语法错误。通过静态验证器和语义相似性构建奖励函数，实现在多个领域特定语言上超过95%的通过率。结果显示该方法优于监督微调，尤其适用于低资源编程语言场景。
Reason 该论文属于典型的‘平行下游应用’，其核心是将小语言模型用于修复大模型在代码生成中的语法错误，应用场景为编程领域的DSL（领域特定语言）程序修复。尽管使用了强化学习和奖励机制，但方法本身并未针对文档图像理解中的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）提出可迁移的底层技术创新。代码与文本结构差异显著，难以直接迁移到DIU任务中。
Abstract Recent advancements in large language models (LLMs) have shown very impressive capabilities in code generation across many programming languages. However, even state-of-the-art LLMs generate programs that contains syntactic errors and fail to complete the given tasks, especially for low-resource programming languages (LRPLs). In addition, high training cost makes finetuning LLMs unaffordable with constrained computational resources, further undermining the effectiveness of LLMs for code generation. In this work, we propose SLMFix, a novel code generation pipeline that leverages a small language model (SLM) finetuned using reinforcement learning (RL) techniques to fix syntactic errors in LLM-generated programs to improve the quality of LLM-generated programs for domain-specific languages (DSLs). In specific, we applied RL on the SLM for the program repair task using a reward calculated using both a static validator and a static semantic similarity metric. Our experimental results demonstrate the effectiveness and generalizability of our approach across multiple DSLs, achieving more than 95% pass rate on the static validator. Notably, SLMFix brings substantial improvement to the base model and outperforms supervised finetuning approach even for 7B models on a LRPL, showing the potential of our approach as an alternative to traditional finetuning approaches.
Authors David Jiahao Fu, Aryan Gupta, Aaron Councilman, David Grove, Yu-Xiong Wang, Vikram Adve
Categories Artificial Intelligence
3
Be My Eyes: Extending Large Language Models to New Modalities Through Multi-Agent Collaboration
Title CN 做我的眼睛：通过多智能体协作将大语言模型扩展到新模态
Keywords 多智能体系统 · 模态扩展 · 视觉语言模型 · LLM推理 · 数据合成
Summary 本文提出BeMyEyes，一种通过多智能体协作将纯文本大模型扩展至多模态推理的框架，其中小型视觉语言模型作为‘眼睛’负责感知，大语言模型作为‘大脑’负责推理。通过合成数据和监督微调训练两者协作能力，在多个知识密集型多模态任务上超越GPT-4o等大型闭源模型。
Reason 该论文属于典型的‘平行应用’类工作，提出的是一个通用的多智能体框架（感知模块+推理模块），用于将LLM与小型VLM结合进行跨模态推理。虽然其架构设计看似具有模块化优势，但本质上是将已有VLM和LLM拼接起来完成多模态任务，并未针对文档图像理解中的核心痛点（如OCR幻觉、复杂版面解析、长文档结构建模）提出专门的方法改进。其‘BeMyEyes’框架可应用于各种视觉-语言任务，但缺乏对文档特有的细粒度文本定位、表格结构恢复或视觉压缩表示的学习机制，属于通用型Agent Wrapper，不具备向DIU领域迁移的独特技术价值。
Abstract Large Language Models (LLMs) have demonstrated remarkable capabilities in challenging, knowledge-intensive reasoning tasks. However, extending LLMs to perceive and reason over a new modality (e.g., vision), often requires costly development of large-scale vision language models (VLMs) with LLMs as backbones. Smaller VLMs are more efficient and adaptable but often lack the broad knowledge and reasoning capabilities of frontier LLMs. In this work, we propose BeMyEyes, a modular, multi-agent framework for extending LLMs to multimodal reasoning by orchestrating collaboration between efficient, adaptable VLMs as perceivers and powerful LLMs as reasoners through conversations. We then introduce a data synthesis and supervised fine-tuning pipeline to train the perceiver agent to effectively collaborate with the reasoner agent. By combining the complementary strengths of perception and reasoning agents, BeMyEyes avoids the need for training large-scale multimodal models, preserves the generalization and reasoning capabilities of LLMs, and allows flexible extension to new domains and modalities. Experiments show that our framework unlocks the multimodal reasoning capabilities for LLMs, enabling a lightweight and fully open-source solution, i.e. equipping text-only DeepSeek-R1 with Qwen2.5-VL-7B perceiver, to outperform large-scale proprietary VLMs such as GPT-4o on a wide range of knowledge-intensive multimodal tasks. These results demonstrate the effectiveness, modularity, and scalability of our multi-agent approach for building future multimodal reasoning systems.
Authors James Y. Huang, Sheng Zhang, Qianchu Liu, Guanghui Qin, Tinghui Zhu, Tristan Naumann, Muhao Chen, Hoifung Poon
Categories Artificial Intelligence, Computation and Language
3
Bridging LLM Planning Agents and Formal Methods: A Case Study in Plan Verification AgenticSE Workshop at ASE 2025
Title CN 连接大语言模型规划智能体与形式化方法：基于计划验证的案例研究
Keywords 大语言模型 · 规划验证 · 形式化方法 · 线性时序逻辑 · 智能体
Summary 该研究提出一种利用大语言模型将自然语言计划转换为Kripke结构和线性时序逻辑（LTL）的形式化表示，并通过模型检测技术验证计划正确性的框架。实验在简化版PlanBench数据集上进行，结果显示GPT-5在分类性能上达到96.3%的F1分数，且生成的语法形式正确，但语义完备性仍有待提升。整体工作聚焦于智能体规划的可信验证，属于AI规划与软件工程交叉领域。
Reason 该论文属于典型的‘平行应用’类型，其核心是将LLM生成的自然语言计划转化为形式化逻辑（LTL）并进行模型检验，应用场景聚焦于智能体规划验证。尽管使用了LLM和形式化方法结合的技术，但并未针对文档图像理解中的任何具体任务（如DocVQA、布局分析、表格识别等）提出改进，也未涉及视觉输入处理、OCR幻觉缓解、多模态对齐或高分辨率文档建模等DIU核心问题。虽然形式化验证思想在理论上可用于增强推理一致性，但本文未展示其在文档理解场景下的迁移路径或潜在价值，属于与DIU无关的下游智能体应用。
Abstract We introduce a novel framework for evaluating the alignment between natural language plans and their expected behavior by converting them into Kripke structures and Linear Temporal Logic (LTL) using Large Language Models (LLMs) and performing model checking. We systematically evaluate this framework on a simplified version of the PlanBench plan verification dataset and report on metrics like Accuracy, Precision, Recall and F1 scores. Our experiments demonstrate that GPT-5 achieves excellent classification performance (F1 score of 96.3%) while almost always producing syntactically perfect formal representations that can act as guarantees. However, the synthesis of semantically perfect formal models remains an area for future exploration.
Authors Keshav Ramani, Vali Tawosi, Salwa Alamir, Daniel Borrajo
Comment Accepted to AgenticSE Workshop at ASE 2025
Categories Artificial Intelligence
3
The Geometry of Cortical Computation: Manifold Disentanglement and Predictive Dynamics in VCNet NeurIPS 2025 Workshop: Symmetry and Geometry in Neural Representations (NeurReps)
Title CN 皮层计算的几何学：VCNet中的流形解耦与预测动力学
Keywords VCNet · 生物启发架构 · 流形解耦 · 预测编码 · 几何深度学习
Summary 本文提出了VCNet，一种受灵长类视觉系统组织结构启发的新型神经网络架构，通过分层处理、双通路信息分离和自上而下的预测反馈机制来学习结构化的低维神经流形。在Spots-10动物纹理数据集和光场图像分类任务上取得了SOTA性能，强调了将神经科学原理与几何视角结合对提升模型效率和鲁棒性的价值。
Reason 该论文提出了一种受灵长类视觉皮层启发的神经网络架构VCNet，聚焦于动物纹理识别和光场图像分类任务，属于生物启发式通用视觉模型研究。其任务与文档图像理解（DIU）无关，未涉及DocVQA、布局分析、表格识别等核心文档任务，且方法迁移至文档领域的路径不明确，尤其是对OCR幻觉、密集文本、复杂排版等DIU痛点无直接解决潜力。
Abstract Despite their success, modern convolutional neural networks (CNNs) exhibit fundamental limitations, including data inefficiency, poor out-of-distribution generalization, and vulnerability to adversarial perturbations. These shortcomings can be traced to a lack of inductive biases that reflect the inherent geometric structure of the visual world. The primate visual system, in contrast, demonstrates superior efficiency and robustness, suggesting that its architectural and computational principles,which evolved to internalize these structures,may offer a blueprint for more capable artificial vision. This paper introduces Visual Cortex Network (VCNet), a novel neural network architecture whose design is informed by the macro-scale organization of the primate visual cortex. VCNet is framed as a geometric framework that emulates key biological mechanisms, including hierarchical processing across distinct cortical areas, dual-stream information segregation for learning disentangled representations, and top-down predictive feedback for representation refinement. We interpret these mechanisms through the lens of geometry and dynamical systems, positing that they guide the learning of structured, low-dimensional neural manifolds. We evaluate VCNet on two specialized benchmarks: the Spots-10 animal pattern dataset, which probes sensitivity to natural textures, and a light field image classification task, which requires processing higher-dimensional visual data. Our results show that VCNet achieves state-of-the-art accuracy of 92.1\% on Spots-10 and 74.4\% on the light field dataset, surpassing contemporary models of comparable size. This work demonstrates that integrating high-level neuroscientific principles, viewed through a geometric lens, can lead to more efficient and robust models, providing a promising direction for addressing long-standing challenges in machine learning.
Authors Brennen A. Hill, Zhang Xinyu, Timothy Putra Prasetio
Comment Published in the proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop: Symmetry and Geometry in Neural Representations (NeurReps). Additionally accepted for presentation in NeurIPS 2025 Workshop: Interpreting Cognition in Deep Learning Models (CogInterp)
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
FOCUS: Efficient Keyframe Selection for Long Video Understanding
Title CN FOCUS：面向长视频理解的高效关键帧选择
Keywords 关键帧选择 · 长视频理解 · 多模态大模型 · 视觉token · 探索-利用
Summary 本文提出FOCUS，一种无需训练、模型无关的关键帧选择模块，用于在严格的token预算下选择与查询相关的关键帧。该方法将关键帧选择建模为多臂老虎机中的组合纯探索问题，通过经验均值和Bernstein置信半径识别信息丰富的时段，并采用两阶段的探索-利用策略：先定位高价值时间段，再从中选取最优帧。实验表明，FOCUS在仅处理不到2%视频帧的情况下显著提升了长视频问答的准确率。
Reason 该论文研究的是长视频理解中的关键帧选择问题，属于视频理解领域，与文档图像理解（DIU）的核心任务如DocVQA、表格识别、布局分析等无直接关联。虽然其提出的探索-利用机制在理论上具有启发性，但方法针对的是时间序列上的视频帧选择，迁移到文档图像中的细粒度定位或长文档推理场景极为困难，且未涉及OCR幻觉、复杂排版或多模态对齐等DIU核心痛点。
Abstract Multimodal large language models (MLLMs) represent images and video frames as visual tokens. Scaling from single images to hour-long videos, however, inflates the token budget far beyond practical limits. Popular pipelines therefore either uniformly subsample or apply keyframe selection with retrieval-style scoring using smaller vision-language models. However, these keyframe selection methods still rely on pre-filtering before selection to reduce the inference cost and can miss the most informative moments. We propose FOCUS, Frame-Optimistic Confidence Upper-bound Selection, a training-free, model-agnostic keyframe selection module that selects query-relevant frames under a strict token budget. FOCUS formulates keyframe selection as a combinatorial pure-exploration (CPE) problem in multi-armed bandits: it treats short temporal clips as arms, and uses empirical means and Bernstein confidence radius to identify informative regions while preserving exploration of uncertain areas. The resulting two-stage exploration-exploitation procedure reduces from a sequential policy with theoretical guarantees, first identifying high-value temporal regions, then selecting top-scoring frames within each region. On two long-video question-answering benchmarks, FOCUS delivers substantial accuracy improvements while processing less than 2% of video frames. For videos longer than 20 minutes, it achieves an 11.9% gain in accuracy on LongVideoBench, demonstrating its effectiveness as a keyframe selection method and providing a simple and general solution for scalable long-video understanding with MLLMs. Code is available at https://github.com/NUS-HPC-AI-Lab/FOCUS.
Authors Zirui Zhu, Hailun Xu, Yang Luo, Yong Liu, Kanchan Sarkar, Zhenheng Yang, Yang You
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
In Machina N400: Pinpointing Where a Causal Language Model Detects Semantic Violations AICS2025
Title CN 在机器之中N400：精确定位因果语言模型检测语义违规的位置
Keywords 语言模型 · 语义违规 · 隐藏状态分析 · 线性探针 · 表征学习
Summary 本文通过构建包含合理与不合理结尾的句子语料库，利用phi-2模型的隐藏层状态，探究因果语言模型在何处检测到语义违规。使用线性探针和有效维度分析发现，语义违规信号在模型中层开始显著增强，并伴随表征空间先扩展后压缩的现象，提示存在从探索到整合的动态过程。结果与人类阅读中语义异常检测晚于句法解析的心理语言学发现相呼应。
Reason 该论文研究的是因果语言模型在处理文本序列时如何检测语义违例，其方法聚焦于隐藏状态的线性探测与表征维度分析，属于纯文本模态下的认知建模，未涉及视觉或文档模态。尽管其对‘幻觉’或语义不一致的检测思路看似相关，但完全缺乏多模态输入（如图像、布局、OCR输出）的考量，无法直接迁移用于解决文档图像理解中的OCR幻觉或视觉-语义对齐问题。且研究目标为模拟人类阅读的认知过程，并非面向实际DIU任务（如DocVQA、表格识别等）的性能提升，属于平行的语言模型内部机制分析。
Abstract How and where does a transformer notice that a sentence has gone semantically off the rails? To explore this question, we evaluated the causal language model (phi-2) using a carefully curated corpus, with sentences that concluded plausibly or implausibly. Our analysis focused on the hidden states sampled at each model layer. To investigate how violations are encoded, we utilized two complementary probes. First, we conducted a per-layer detection using a linear probe. Our findings revealed that a simple linear decoder struggled to distinguish between plausible and implausible endings in the lowest third of the model's layers. However, its accuracy sharply increased in the middle blocks, reaching a peak just before the top layers. Second, we examined the effective dimensionality of the encoded violation. Initially, the violation widens the representational subspace, followed by a collapse after a mid-stack bottleneck. This might indicate an exploratory phase that transitions into rapid consolidation. Taken together, these results contemplate the idea of alignment with classical psycholinguistic findings in human reading, where semantic anomalies are detected only after syntactic resolution, occurring later in the online processing sequence.
Authors Christos-Nikolaos Zacharopoulos, Revekka Kyriakoglou
Comment Accepted at AICS2025
Categories Artificial Intelligence, Computation and Language
3
Benchmarking the Spatial Robustness of DNNs via Natural and Adversarial Localized Corruptions Pattern Recognition
Title CN 通过自然与对抗性局部损坏评估DNN的空间鲁棒性
Keywords 语义分割 · 空间鲁棒性 · 局部损坏 · 对抗攻击 · 自动驾驶
Summary 本文提出了一套用于评估深度神经网络在局部自然和对抗性损坏下空间鲁棒性的新指标与框架，重点应用于驾驶场景中的语义分割模型。研究发现基于Transformer的模型对自然损坏更鲁棒，而CNN模型对对抗攻击更具抵抗力，并通过集成方法平衡两者表现。
Reason 该论文研究的是语义分割模型在驾驶场景中对局部损坏（如遮挡、噪声）的鲁棒性，属于通用计算机视觉领域，主要关注安全关键场景下的模型稳定性。其提出的方法（区域感知度量、多攻击对抗分析）聚焦于自动驾驶等密集视觉任务，并未涉及文档图像特有的结构、布局或文本语义理解问题。虽然‘局部损坏’在文档中也可能出现（如污渍、模糊），但该工作未针对OCR、排版解析或VLM进行设计，迁移价值极低，且属于典型的非DIU主线应用。
Abstract The robustness of deep neural networks is a crucial factor in safety-critical applications, particularly in complex and dynamic environments (e.g., medical or driving scenarios) where localized corruptions can arise. While previous studies have evaluated the robustness of semantic segmentation (SS) models under whole-image natural or adversarial corruptions, a comprehensive investigation into the spatial robustness of dense vision models under localized corruptions remains underexplored. This paper fills this gap by introducing novel, region-aware metrics for benchmarking the spatial robustness of segmentation models, along with an evaluation framework to assess the impact of natural localized corruptions. Furthermore, it uncovers the inherent complexity of evaluating worst-case spatial robustness using only a single localized adversarial attack. To address this, the work proposes a region-aware multi-attack adversarial analysis to systematically assess model robustness across specific image regions. The proposed metrics and analysis were exploited to evaluate 14 segmentation models in driving scenarios, uncovering key insights into the effects of localized corruption in both natural and adversarial forms. The results reveal that models respond to these two types of threats differently; for instance, transformer-based segmentation models demonstrate notable robustness to localized natural corruptions but are highly vulnerable to adversarial ones, and vice versa for CNN-based models. Consequently, we also address the challenge of balancing robustness to both natural and adversarial localized corruptions by means of ensemble models, thereby achieving a broader threat coverage and improved reliability for dense vision tasks.
Authors Giulia Marchiori Pietrosanti, Giulio Rossolini, Alessandro Biondi, Giorgio Buttazzo
Comment Accepted for publication in Pattern Recognition
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
In-Situ Tweedie Discrete Diffusion Models
Title CN 原位Tweedie离散扩散模型
Keywords 离散扩散模型 · Tweedie公式 · one-hot扩散 · 图像分类 · 文本生成
Summary 本文提出了原位Tweedie离散扩散模型（TDD），在离散的one-hot向量空间中直接执行扩散过程，通过高斯噪声破坏one-hot向量，并使用时间步条件下的交叉熵目标进行迭代去噪。该方法统一了判别式分类与生成式建模，适用于图像分类和文本生成任务，在实验中表现出色。然而，该工作未聚焦于文档理解场景，缺乏对OCR错误、密集文本、排版复杂性等问题的针对性设计。
Reason 该论文提出了一种在离散one-hot空间中直接进行扩散的TDD框架，理论基础为Tweedie公式，主要应用于图像分类和文本生成任务。尽管其方法论涉及离散数据建模，但核心目标并非文档图像理解（DIU）中的典型问题（如OCR幻觉、复杂版式解析、细粒度定位等）。其技术路线属于通用生成模型设计，且未针对文档结构、多模态对齐或高分辨率视觉处理进行优化，难以直接迁移用于解决DocVQA或表格识别等任务。属于通用AI方法在非DIU主战场的应用，无明确DIU方法论创新。
Abstract While diffusion models excel at generating continuous data such as images, adapting them to discrete tasks has relied on indirect approaches that either operate in continuous embedding spaces or use token masking mechanisms, both of which deviate from modeling the true discrete data distribution that can be theoretically guaranteed by Tweedie's formula. We propose in-situ Tweedie Discrete Diffusion (TDD), a framework that performs diffusion guaranteed by Tweedie's formula directly within the discrete one-hot space, hence "in-situ." Unlike prior methods that diffuse continuous embeddings or mask tokens, TDD directly corrupts one-hot vectors with Gaussian noise and performs iterative denoising through a timestep-conditioned cross-entropy objective rather than mean-squared-error reconstruction. At each denoising step, the model predicts class probabilities, applies argmax to obtain discrete predictions, converts them to one-hot vectors, and feeds them into the next iteration with progressively reduced noise. This process naturally unifies discriminative classification and generative modeling under a single framework. Experiments demonstrate that TDD achieves strong performance on both image classification and text generation tasks, with extensive ablation studies confirming the effectiveness of each design component. Our work establishes a principled approach to discrete diffusion that preserves the core characteristics of diffusion models while operating natively in discrete space.
Authors Xiao Li, Jiaqi Zhang, Shuxiang Zhang, Tianshui Chen, Liang Lin, Guangrun Wang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
3
OrdMoE: Preference Alignment via Hierarchical Expert Group Ranking in Multimodal Mixture-of-Experts LLMs
Title CN OrdMoE：通过多模态专家混合模型中分层专家组排序实现偏好对齐
Keywords 偏好对齐 · MoE · 多模态大模型 · 自监督学习 · 路由机制
Summary 本文提出OrdMoE，一种无需人工偏好标注的多模态MoE大模型对齐框架。它利用专家路由得分隐含的质量排序，将专家分组为不同层级，逐层激活生成质量递增的响应序列，构建零成本的自监督偏好数据用于训练。实验表明该方法在多个多模态基准上有效提升模型对齐性和整体性能。
Reason 该论文聚焦于多模态大模型中的偏好对齐问题，提出利用MoE架构内部的路由分数构建自监督的响应排序，从而避免依赖人工标注的偏好数据。尽管其方法在多模态学习中有一定创新性，但核心贡献属于训练策略层面的优化，且完全基于通用多模态对话场景，并未针对文档图像理解（DIU）中的具体痛点（如OCR幻觉、复杂版式、细粒度定位等）设计机制。该方法缺乏可迁移至DIU任务的明确路径，例如无法直接缓解视觉文本误识别或提升表格结构解析能力，属于通用VLM训练范式改进，而非面向文档理解的方法论突破。
Abstract Preference learning has recently emerged as a pivotal strategy for post-training alignment of Multimodal Large Language Models (MLLMs). However, existing approaches predominantly rely on external human-annotated preference data, which is costly and labor-intensive to collect. In this work, we propose OrdMoE, a novel preference alignment framework that bypasses the reliance on external human preferences entirely by leveraging intrinsic signals within Mixture-of-Experts (MoE) architectures. Specifically, we observe that the router's expert selection scores implicitly encode a quality-aware ranking of responses (i.e. higher-scoring experts consistently generate higher-quality outputs). Building on this insight, OrdMoE constructs an internal preference hierarchy by grouping experts into ranked tiers based on their per-token routing scores and activating each tier separately to produce a sequence of responses with increasing quality. This yields a zero-cost, self-supervised preference ordering over generated responses, which can be directly optimized using standard preference learning objectives. Extensive experiments across multiple multimodal benchmarks demnstrate that OrdMoE significantly enhances both alignment and overall performance of multimodal Mixture-of-Experts LLMs, achieving competitive results without requiring any human-annotated preference data.
Authors Yuting Gao, Weihao Chen, Lan Wang, Ruihan Xu, Qingpei Guo
Categories Artificial Intelligence
3
Causally Reliable Concept Bottleneck Models NeurIPS 2025
Title CN 因果可靠的可解释概念瓶颈模型
Keywords 可解释性 · 因果推理 · 概念瓶颈模型 · 干预响应 · 结构学习
Summary 本文提出了一种因果可靠的概念瓶颈模型（C²BM），通过将推理过程限制在由真实世界因果机制组织的概念瓶颈上，提升模型的可解释性和因果合理性。作者还设计了一个从观测数据和非结构化背景知识中自动学习因果结构的流程。实验表明，该模型在保持准确率的同时，比传统概念模型和黑箱模型更具可解释性、因果可靠性及对干预的响应能力。
Reason 该论文研究的是通用深度学习中的概念瓶颈模型与因果推理问题，主要关注模型的可解释性、因果机制建模以及干预响应能力。虽然其方法在理论上可能对任何需要因果推理的任务有潜在价值，但其本身并未针对文档图像理解（DIU）中的具体任务（如DocVQA、布局分析、表格识别等）进行设计或验证。所提出的C²BM框架依赖于外部因果结构学习，难以直接迁移至文档视觉理解中常见的OCR幻觉、细粒度定位或复杂排版解析等核心痛点。属于通用AI可解释性研究，不构成对DIU的方法论推动。
Abstract Concept-based models are an emerging paradigm in deep learning that constrains the inference process to operate through human-interpretable variables, facilitating explainability and human interaction. However, these architectures, on par with popular opaque neural models, fail to account for the true causal mechanisms underlying the target phenomena represented in the data. This hampers their ability to support causal reasoning tasks, limits out-of-distribution generalization, and hinders the implementation of fairness constraints. To overcome these issues, we propose Causally reliable Concept Bottleneck Models (C$^2$BMs), a class of concept-based architectures that enforce reasoning through a bottleneck of concepts structured according to a model of the real-world causal mechanisms. We also introduce a pipeline to automatically learn this structure from observational data and unstructured background knowledge (e.g., scientific literature). Experimental evidence suggests that C$^2$BMs are more interpretable, causally reliable, and improve responsiveness to interventions w.r.t. standard opaque and concept-based models, while maintaining their accuracy.
Authors Giovanni De Felice, Arianna Casanova Flores, Francesco De Santis, Silvia Santini, Johannes Schneider, Pietro Barbiero, Alberto Termine
Comment Accepted at NeurIPS 2025
Categories Artificial Intelligence
3
Dynamic Mixture of Experts Against Severe Distribution Shifts
Title CN 应对严重分布偏移的动态专家混合模型
Keywords Mixture of Experts · Continual Learning · Distribution Shift · Neural Architecture · Plasticity-Stability
Summary 本文探讨了在持续学习和强化学习场景下，使用动态专家混合（DynamicMoE）模型应对严重分布偏移的问题，旨在通过动态增加专家来维持模型的可塑性并缓解灾难性遗忘。实验表明该方法在参数效率和性能上优于传统网络扩展方法。
Reason 该论文研究的是持续学习和强化学习中的分布偏移问题，采用Dynamic MoE架构来动态适应数据流变化。虽然MoE在理论上具有扩展性和对分布变化的适应能力，但其核心应用场景（持续学习、强化学习）与文档图像理解（DIU）任务无直接关联。未提出可用于文档视觉理解、OCR去幻觉、细粒度定位或高分辨率处理的新方法，属于通用模型结构在其他领域的应用，无法迁移解决DIU的核心痛点。
Abstract The challenge of building neural networks that can continuously learn and adapt to evolving data streams is central to the fields of continual learning (CL) and reinforcement learning (RL). This lifelong learning problem is often framed in terms of the plasticity-stability dilemma, focusing on issues like loss of plasticity and catastrophic forgetting. Unlike neural networks, biological brains maintain plasticity through capacity growth, inspiring researchers to explore similar approaches in artificial networks, such as adding capacity dynamically. Prior solutions often lack parameter efficiency or depend on explicit task indices, but Mixture-of-Experts (MoE) architectures offer a promising alternative by specializing experts for distinct distributions. This paper aims to evaluate a DynamicMoE approach for continual and reinforcement learning environments and benchmark its effectiveness against existing network expansion methods.
Authors Donghu Kim
Categories Artificial Intelligence
3
Q-SAM2: Accurate Quantization for Segment Anything Model 2
Title CN Q-SAM2：用于Segment Anything Model 2的精确量化
Keywords 模型量化 · Segment Anything · 低比特推理 · 模型压缩 · QAT
Summary 本文提出Q-SAM2，一种针对Segment Anything Model 2的高精度低比特量化方法，通过方差减少校准（VRC）和可学习统计裁剪（LSC）技术，在仅2比特的情况下显著降低模型大小并保持分割性能。实验显示其在视频和实例分割基准上优于现有量化方法。
Reason 该论文聚焦于视觉基础模型SAM2的低比特量化，属于模型压缩与部署优化工作。尽管其方法在视频和实例分割任务中表现出色，但核心贡献是针对通用图像分割模型的推理效率提升，并未涉及文档图像理解（DIU）中的关键问题，如OCR幻觉、复杂版面分析、文本细粒度识别或长文档推理。同时，SAM系列本身并非为文档场景设计，其promptable segmentation机制在文档逻辑结构理解或VIE等任务中迁移价值有限。因此，属于与DIU无关的下游模型压缩应用，不具方法论上的可迁移性。
Abstract The Segment Anything Model 2 (SAM2) is a powerful foundation model for promptable segmentation. However, its high computational and memory costs are a major barrier to deployment on resource-constrained devices. In this paper, we present Q-SAM2, an accurate low-bit quantization method that achieves high compression and high fidelity. To address performance degradation arising from challenging weight and activation distributions during quantization, Q-SAM2 introduces two novel contributions: Variance-Reduced Calibration (VRC), an initialization method that reduces weight statistical variance by minimizing the Frobenius norm over a small calibration batch; and Learnable Statistical Clipping (LSC), a Quantization-Aware Training (QAT) method that learns momentum-stabilized clipping factors to manage outliers in weights and activations. Comprehensive experiments demonstrate that Q-SAM2 achieves highly accurate inference with substantial efficiency gains, significantly surpassing state-of-the-art general QAT schemes, particularly in the ultra-low 2-bit regime. Specifically, Q-SAM2 achieves an accuracy gain of up to 9.7 ppt in J&F on the video segmentation benchmark and 7.3 ppt in mIoU for instance segmentation over the best competing QAT model, all while achieving an 8x reduction in model size compared to the BF16 baseline.
Authors Nicola Farronato, Florian Scheidegger, Mattia Rigotti, Cristiano Malossi, Michele Magno, Haotong Qin
Comment 22 pages
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
FastForward Pruning: Efficient LLM Pruning via Single-Step Reinforcement Learning
Title CN 快速前向剪枝：通过单步强化学习实现高效的LLM剪枝
Keywords 模型剪枝 · 强化学习 · 效率优化 · 大语言模型 · 结构压缩
Summary 本文提出了一种名为FastForward Pruning的方法，通过解耦策略优化与资源约束，在单步强化学习框架下高效搜索大语言模型的层间稀疏性分配方案。该方法采用课程学习策略，从简单任务逐步过渡到复杂任务，显著降低了传统搜索方法的计算开销，并在LLaMA、Mistral和OPT系列模型上验证了有效性。尽管在模型压缩方面表现良好，但其应用场景局限于纯文本LLM，不涉及视觉或多模态内容。
Reason 该论文研究的是大语言模型的参数剪枝问题，属于模型压缩领域。虽然其方法在LLM效率优化上有一定价值，但完全聚焦于纯文本语言模型的结构压缩，并未涉及视觉输入、多模态建模或文档布局等DIU核心问题。无法直接迁移用于解决OCR幻觉、复杂排版理解或高分辨率文档处理等挑战，属于与文档图像理解无关的平行技术路线。
Abstract Pruning is an effective method for compressing Large Language Models, but finding an optimal, non-uniform layer-wise sparsity allocation remains a key challenge. While heuristic methods are fast but yield suboptimal performance, more powerful search-based approaches like Reinforcement Learning are often hindered by prohibitive computational costs on large-scale models. To overcome this efficiency barrier, we propose FastForward Pruning. Its core is a decoupled, single-step RL framework that separates policy optimization from the complex budget satisfaction problem. Such a decoupling is crucial for efficiently searching the vast policy space of LLMs. This curriculum-based strategy begins with low-cost, simple tasks and gradually increases in complexity, significantly reducing the search's computational overhead. Evaluated on the LLaMA, Mistral, and OPT model families, our framework discovers pruning policies that achieve superior performance over strong heuristic baselines. Crucially, when compared to other search-based algorithms, our method achieves competitive or superior results at a fraction of the computational cost, demonstrating a clear advantage in search efficiency.
Authors Xin Yuan, Siqi Li, Jiateng Wei, Chengrui Zhu, Yanming Wu, Qingpeng Li, Jiajun Lv, Xiaoke Lan, Jun Chen, Yong Liu
Comment 5 pages, 2 figures, 4 tables
Categories Artificial Intelligence
3
Active Inference is a Subtype of Variational Inference EIML Workshop 2025 at EurIPS
Title CN 主动推理是变分推理的一种子类型
Keywords 主动推理 · 变分推理 · 期望自由能 · 规划即推断 · 消息传递
Summary 本文从理论层面将主动推理中的期望自由能（EFE）最小化重新表述为变分推理问题，实现了与‘规划即推断’（Planning-as-Inference）框架的统一，并提出一种新的消息传递方案以提升可扩展性，从而解决高维规划中的计算不可行问题。
Reason 该论文属于具身智能与决策理论领域，研究主动推理（Active Inference）与变分推理的统一框架，尽管其方法论涉及推理机制优化，但核心关注点为通用决策系统中的探索-利用权衡，并非针对多模态文档理解或视觉语言模型中的具体问题。无法直接迁移用于解决文档图像理解中的OCR幻觉、复杂排版、细粒度定位或高分辨率处理等痛点，属于无关领域。
Abstract Automated decision-making under uncertainty requires balancing exploitation and exploration. Classical methods treat these separately using heuristics, while Active Inference unifies them through Expected Free Energy (EFE) minimization. However, EFE minimization is computationally expensive, limiting scalability. We build on recent theory recasting EFE minimization as variational inference, formally unifying it with Planning-as-Inference and showing the epistemic drive as a unique entropic contribution. Our main contribution is a novel message-passing scheme for this unified objective, enabling scalable Active Inference in factored-state MDPs and overcoming high-dimensional planning intractability.
Authors Wouter W. L. Nuijten, Mykola Lukashchuk
Comment Accepted to the EIML Workshop 2025 at EurIPS (non-archival)
Categories Artificial Intelligence
3
Skeletons Matter: Dynamic Data Augmentation for Text-to-Query EMNLP 2025
Title CN 骨架至关重要：面向文本到查询的动态数据增强
Keywords 语义解析 · 文本到查询 · 数据增强 · 大语言模型 · 语法骨架
Summary 本文提出了一种基于查询骨架的动态数据增强框架，用于提升大模型在多种查询语言上的语义解析能力。通过识别模型在处理特定语法骨架时的弱点，有针对性地生成训练数据，在少量合成数据下实现了SOTA性能。研究集中在纯文本语义映射任务，不涉及视觉或文档结构信息。
Reason 该论文聚焦于将自然语言问题转化为查询语言（如SQL）的语义解析任务，属于典型的下游应用型研究。尽管提出了跨查询语言的统一框架和基于骨架的数据增强方法，但其核心场景是Text-to-SQL类任务，未涉及文档图像、视觉布局、OCR或图文多模态理解等DIU核心问题。属于‘平行应用’范畴，对文档图像理解无直接方法论贡献。
Abstract The task of translating natural language questions into query languages has long been a central focus in semantic parsing. Recent advancements in Large Language Models (LLMs) have significantly accelerated progress in this field. However, existing studies typically focus on a single query language, resulting in methods with limited generalizability across different languages. In this paper, we formally define the Text-to-Query task paradigm, unifying semantic parsing tasks across various query languages. We identify query skeletons as a shared optimization target of Text-to-Query tasks, and propose a general dynamic data augmentation framework that explicitly diagnoses model-specific weaknesses in handling these skeletons to synthesize targeted training data. Experiments on four Text-to-Query benchmarks demonstrate that our method achieves state-of-the-art performance using only a small amount of synthesized data, highlighting the efficiency and generality of our approach and laying a solid foundation for unified research on Text-to-Query tasks. We release our code at https://github.com/jjjycaptain/Skeletron.
Authors Yuchen Ji, Bo Xu, Jie Shi, Jiaqing Liang, Deqing Yang, Yu Mao, Hai Chen, Yanghua Xiao
Comment Accepted at EMNLP 2025
Categories Artificial Intelligence, Computation and Language
3
Learning What to Trust: Bayesian Prior-Guided Optimization for Visual Generation
Title CN 学习该信任什么：基于贝叶斯先验引导的视觉生成优化
Keywords 视觉生成 · 奖励建模 · 贝叶斯先验 · GRPO · 不确定性建模
Summary 本文提出BPGO方法，用于解决视觉生成模型训练中因文本-视觉多对多对应关系导致的奖励信号模糊问题。通过引入语义先验锚点，BPGO在组间和组内两个层面动态调整对奖励信号的信任，提升生成结果的语义一致性和感知质量。实验表明其在图像和视频生成任务上优于GRPO等基线方法。
Reason 该论文聚焦于视觉生成任务中的奖励建模不确定性问题，提出通过贝叶斯先验来调节优化过程的信任程度。尽管其方法在图像和视频生成中有效，但属于纯视觉生成领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。虽然‘信任建模’和‘减少幻觉’在理念上有潜在启发性，但该工作并未针对OCR幻觉、文本密集或复杂排版等文档特有挑战设计，且应用于生成而非理解任务，迁移路径不明确。
Abstract Group Relative Policy Optimization (GRPO) has emerged as an effective and lightweight framework for post-training visual generative models. However, its performance is fundamentally limited by the ambiguity of textual visual correspondence: a single prompt may validly describe diverse visual outputs, and a single image or video may support multiple equally correct interpretations. This many to many relationship leads reward models to generate uncertain and weakly discriminative signals, causing GRPO to underutilize reliable feedback and overfit noisy ones. We introduce Bayesian Prior-Guided Optimization (BPGO), a novel extension of GRPO that explicitly models reward uncertainty through a semantic prior anchor. BPGO adaptively modulates optimization trust at two levels: inter-group Bayesian trust allocation emphasizes updates from groups consistent with the prior while down-weighting ambiguous ones, and intra-group prior-anchored renormalization sharpens sample distinctions by expanding confident deviations and compressing uncertain scores. Across both image and video generation tasks, BPGO delivers consistently stronger semantic alignment, enhanced perceptual fidelity, and faster convergence than standard GRPO and recent variants.
Authors Ruiying Liu, Yuanzhi Liang, Haibin Huang, Tianshu Yu, Chi Zhang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
Title CN 学习率衰减如何在基于课程的LLM预训练中浪费了你最好的数据
Keywords 课程学习 · 学习率衰减 · 模型平均 · LLM预训练 · 数据质量
Summary 本文指出在基于课程的LLM预训练中，随着数据质量递增排序进行训练时，标准的学习率衰减策略会导致模型在最关键的高质量数据阶段更新不足，从而削弱课程学习的优势。作者提出采用更平缓的学习率衰减或模型平均策略来缓解这一问题，在1.5B参数模型上取得了1.64%的性能提升。
Reason 该论文研究的是纯文本大语言模型预训练中的课程学习与优化策略（如学习率衰减、模型平均）之间的相互作用，属于NLP领域的通用训练方法优化。虽然其关于‘高质量数据利用’的动机有一定普适性，但未涉及任何视觉或文档模态，且不针对多模态对齐、视觉表示、OCR去噪、细粒度定位等DIU核心问题。无法直接迁移用于解决文档图像理解中的幻觉、复杂排版或高分辨率输入等挑战。
Abstract Due to the scarcity of high-quality data, large language models (LLMs) are often trained on mixtures of data with varying quality levels, even after sophisticated data curation. A natural approach to better leverage high-quality data is curriculum-based pretraining, where the model is trained on data sorted in ascending order of quality as determined by a quality metric. However, prior studies have reported limited improvements from such curriculum-based pretraining strategies. This work identifies a critical factor constraining these methods: the incompatibility between the ascending data quality order and the decaying learning rate (LR) schedule. We find that while curriculum-based training substantially outperforms random shuffling when using a constant LR, its advantage diminishes under standard LR decay schedules. Our experiments show this incompatibility can be mitigated by two simple strategies: (1) employing a more moderate LR decay schedule, where the final LR is only moderately smaller than the peak LR, and (2) replacing LR decay with model averaging, i.e., computing a weighted average of the final few checkpoints. By combining these strategies, we improve the average score on a suite of standard benchmarks by 1.64% over random shuffling, without additional data refinement. Validated on 1.5B-parameter models trained over 30B tokens with various data-quality metrics, our findings call for a re-evaluation of curriculum-based LLM pretraining and underscore the potential of co-designing data curricula with optimization methods.
Authors Kairong Luo, Zhenbo Sun, Haodong Wen, Xinyu Shi, Jiarui Cui, Chenyi Dang, Kaifeng Lyu, Wenguang Chen
Categories Artificial Intelligence, Computation and Language
3
Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models NeurIPS 2025
Title CN Nemotron-Flash：面向延迟最优的混合小型语言模型
Keywords 小型语言模型 · 延迟优化 · 架构搜索 · 算子优化 · NeurIPS
Summary 本文提出Nemotron-Flash，一种面向真实设备延迟优化的小型语言模型家族。通过分析深度-宽度比和算子选择对延迟的影响，结合进化搜索自动发现最优算子组合，并引入权重归一化技术提升训练收敛性，在准确率、延迟和吞吐量上均显著优于现有SLM。然而，该工作局限于纯文本语言模型，未涉及任何视觉或多模态组件。
Reason 该论文聚焦于小型语言模型（SLM）的架构设计与训练优化，以实现低延迟推理，属于纯文本模态下的高效LLM研究。尽管其在算子选择和进化搜索框架上有一定创新，但完全未涉及视觉输入、多模态建模或文档结构理解，无法直接迁移解决文档图像理解中的核心痛点（如OCR幻觉、高分辨率处理、细粒度定位等）。其方法本质是语言模型层面的效率优化，对DIU缺乏实质性贡献。
Abstract Efficient deployment of small language models (SLMs) is essential for numerous real-world applications with stringent latency constraints. While previous work on SLM design has primarily focused on reducing the number of parameters to achieve parameter-optimal SLMs, parameter efficiency does not necessarily translate into proportional real-device speed-ups. This work aims to identify the key determinants of SLMs' real-device latency and offer generalizable principles and methodologies for SLM design and training when real-device latency is the primary consideration. Specifically, we identify two central architectural factors: depth-width ratios and operator choices. The former is crucial for small-batch-size latency, while the latter affects both latency and large-batch-size throughput. In light of this, we first study latency-optimal depth-width ratios, with the key finding that although deep-thin models generally achieve better accuracy under the same parameter budget, they may not lie on the accuracy-latency trade-off frontier. Next, we explore emerging efficient attention alternatives to evaluate their potential as candidate building operators. Using the identified promising operators, we construct an evolutionary search framework to automatically discover latency-optimal combinations of these operators within hybrid SLMs, thereby advancing the accuracy-latency frontier. In addition to architectural improvements, we further enhance SLM training using a weight normalization technique that enables more effective weight updates and improves final convergence. Combining these methods, we introduce a new family of hybrid SLMs, called Nemotron-Flash, which significantly advances the accuracy-efficiency frontier of state-of-the-art SLMs, e.g., achieving over +5.5% average accuracy, 1.3x/1.9x lower latency, and 18.7x/45.6x higher throughput compared to Qwen3-1.7B/0.6B, respectively.
Authors Yonggan Fu, Xin Dong, Shizhe Diao, Matthijs Van keirsbilck, Hanrong Ye, Wonmin Byeon, Yashaswi Karnati, Lucas Liebenwein, Hannah Zhang, Nikolaus Binder, Maksim Khadkevich, Alexander Keller, Jan Kautz, Yingyan Celine Lin, Pavlo Molchanov
Comment Accepted by NeurIPS 2025
Categories Artificial Intelligence
3
CoreEval: Automatically Building Contamination-Resilient Datasets with Real-World Knowledge toward Reliable LLM Evaluation ACL'25
Title CN CoreEval：利用真实世界知识自动构建抗污染数据集以实现可靠的LLM评估
Keywords LLM评估 · 数据污染 · 知识更新 · 自动数据重构 · 评测可靠性
Summary 本文提出CoreEval，一种通过提取原始数据中的实体关系并结合GDELT获取最新现实知识，从而自动更新和重构评测数据的方法，以增强LLM评估的抗污染能力。通过重新上下文化和迭代标签验证机制，确保更新后数据的语义一致性和任务相关性。实验表明该方法能有效减少因数据泄露导致的性能高估。
Reason 该论文聚焦于自然语言处理中的LLM评估问题，核心贡献是缓解数据污染（data contamination）对模型性能评估的影响。其方法依赖GDELT等新闻数据库更新事实知识，并重构评测数据。然而，该工作未涉及任何视觉或文档模态，完全基于纯文本NLP任务，与文档图像理解（DIU）的核心挑战如OCR幻觉、复杂版式建模、视觉-文本对齐等无关。尽管‘抗污染’在理论上对评测可靠性重要，但其技术路径无法直接迁移至多模态文档场景，尤其是不处理视觉输入、定位或结构化输出等问题。
Abstract Data contamination poses a significant challenge to the fairness of LLM evaluations in natural language processing tasks by inadvertently exposing models to test data during training. Current studies attempt to mitigate this issue by modifying existing datasets or generating new ones from freshly collected information. However, these methods fall short of ensuring contamination-resilient evaluation, as they fail to fully eliminate pre-existing knowledge from models or preserve the semantic complexity of the original datasets. To address these limitations, we propose \textbf{CoreEval}, a \textbf{Co}ntamination-\textbf{re}silient \textbf{Eval}uation strategy for automatically updating data with real-world knowledge. This approach begins by extracting entity relationships from the original data and leveraging the GDELT database to retrieve relevant, up-to-date knowledge. The retrieved knowledge is then recontextualized and integrated with the original data, which is refined and restructured to ensure semantic coherence and enhanced task relevance. Ultimately, a robust data reflection mechanism is employed to iteratively verify and refine labels, ensuring consistency between the updated and original datasets. Extensive experiments on updated datasets validate the robustness of CoreEval, demonstrating its effectiveness in mitigating performance overestimation caused by data contamination.
Authors Jingqian Zhao, Bingbing Wang, Geng Tu, Yice Zhang, Qianlong Wang, Bin Liang, Jing Li, Ruifeng Xu
Comment ACL'25
Categories Artificial Intelligence, Computation and Language
3
Human Cognition Inspired RAG with Knowledge Graph for Complex Problem Solving AAAI 2026
Title CN 基于知识图谱的人类认知启发式RAG用于复杂问题求解
Keywords RAG · 知识图谱 · 复杂推理 · 思维导图 · 自验证
Summary 本文提出了CogGRAG，一种受人类认知启发的图结构RAG框架，用于知识图谱问答任务。通过构建树状思维导图实现问题分解，结合局部与全局知识检索，并在推理过程中引入双通道自验证机制以提升结果一致性与可靠性。实验表明其在准确性与稳定性上优于现有方法。
Reason 该论文提出了一种面向知识图谱问答（KGQA）的认知启发式RAG框架CogGRAG，其核心创新在于引入树状思维导图结构进行问题分解与自验证推理。尽管其推理机制具有一定的通用性，但整体工作仍属于特定领域的下游应用——即在KGQA任务上构建RAG流程，未涉及任何文档图像或多模态输入处理。方法本身虽包含多步推理和知识整合机制，但无法直接迁移解决文档理解中的核心痛点如OCR幻觉、排版复杂性或视觉-文本对齐问题。且未触及视觉表示学习、高分辨率建模或VLM架构改进等DIU关键路径。
Abstract Large Language Models (LLMs) have demonstrated significant potential across various domains. However, they often struggle with integrating external knowledge and performing complex reasoning, leading to hallucinations and unreliable outputs. Retrieval Augmented Generation (RAG) has emerged as a promising paradigm to mitigate these issues by incorporating external knowledge. Yet, conventional RAG approaches, especially those based on vector similarity, fail to effectively capture relational dependencies and support multi-step reasoning. In this work, we propose CogGRAG, a human cognition-inspired, graph-based RAG framework designed for Knowledge Graph Question Answering (KGQA). CogGRAG models the reasoning process as a tree-structured mind map that decomposes the original problem into interrelated subproblems and explicitly encodes their semantic relationships. This structure not only provides a global view to guide subsequent retrieval and reasoning but also enables self-consistent verification across reasoning paths. The framework operates in three stages: (1) top-down problem decomposition via mind map construction, (2) structured retrieval of both local and global knowledge from external Knowledge Graphs (KGs), and (3) bottom-up reasoning with dual-process self-verification. Unlike previous tree-based decomposition methods such as MindMap or Graph-CoT, CogGRAG unifies problem decomposition, knowledge retrieval, and reasoning under a single graph-structured cognitive framework, allowing early integration of relational knowledge and adaptive verification. Extensive experiments demonstrate that CogGRAG achieves superior accuracy and reliability compared to existing methods.
Authors Yao Cheng, Yibo Zhao, Jiapeng Zhu, Yao Liu, Xing Sun, Xiang Li
Comment The paper has been accepted by AAAI 2026
Categories Artificial Intelligence
3
Pathway to Relevance: How Cross-Encoders Implement a Semantic Variant of BM25
Title CN 通向相关性：交叉编码器如何实现语义版本的BM25
Keywords 信息检索 · 交叉编码器 · 可解释性 · BM25 · 机制分析
Summary 本文通过机械分析方法研究交叉编码器在信息检索中如何判断文档与查询的相关性，发现其早期到中期层提取了词频、逆文档频率等传统信号，并在后期层以类似BM25的方式组合这些信号。研究有助于理解IR模型的内部工作机制，为未来干预和改进提供基础。
Reason 该论文研究的是信息检索（IR）模型中交叉编码器如何计算查询与文档的相关性，属于纯文本信息检索的机制解释工作。虽然其对模型内部机制的分析可能为干预提供思路，但并未涉及任何视觉或文档图像内容的理解、定位或OCR相关问题，完全脱离文档图像理解（DIU）的核心场景。且无任何多模态、视觉输入或布局结构处理成分，无法迁移用于解决OCR幻觉、复杂排版或细粒度视觉文本识别等DIU痛点。
Abstract Mechanistic interpretation has greatly contributed to a more detailed understanding of generative language models, enabling significant progress in identifying structures that implement key behaviors through interactions between internal components. In contrast, interpretability in information retrieval (IR) remains relatively coarse-grained, and much is still unknown as to how IR models determine whether a document is relevant to a query. In this work, we address this gap by mechanistically analyzing how one commonly used model, a cross-encoder, estimates relevance. We find that the model extracts traditional relevance signals, such as term frequency and inverse document frequency, in early-to-middle layers. These concepts are then combined in later layers, similar to the well-known probabilistic ranking function, BM25. Overall, our analysis offers a more nuanced understanding of how IR models compute relevance. Isolating these components lays the groundwork for future interventions that could enhance transparency, mitigate safety risks, and improve scalability.
Authors Meng Lu, Catherine Chen, Carsten Eickhoff
Categories Artificial Intelligence
3
UNeMo: Collaborative Visual-Language Reasoning and Navigation via a Multimodal World Model
Title CN UNeMo：通过多模态世界模型实现视觉-语言协同推理与导航
Keywords 视觉-语言导航 · 多模态世界模型 · 跨模态推理 · 具身智能 · LLM推理
Summary 本文提出UNeMo框架，通过引入多模态世界模型（MWM）和分层预测-反馈机制（HPN），实现视觉、语言与动作的联合建模，提升视觉-语言导航中的跨模态推理能力。MWM能够预测动作执行后的视觉状态，并与导航策略形成双向优化。在R2R和REVERIE数据集上验证了有效性。
Reason 该论文研究的是视觉-语言导航（VLN），属于具身智能（Embodied AI）领域，核心任务是基于语言指令在三维环境中进行视觉导航。尽管其提出了多模态世界模型和推理-反馈机制，具有一定的方法创新性，但应用场景与文档图像理解（DIU）无直接关联。其所谓的‘跨模态推理’和‘策略协同优化’难以迁移到DocVQA、表格识别或OCR-free理解等文档任务中，且未涉及文档特有的痛点如密集文本处理、排版结构建模、OCR幻觉抑制等。因此属于无关领域应用，不符合关注方向。
Abstract Vision-and-Language Navigation (VLN) requires agents to autonomously navigate complex environments via visual images and natural language instruction--remains highly challenging. Recent research on enhancing language-guided navigation reasoning using pre-trained large language models (LLMs) has shown promising prospects. However, the reasoning of such methods is limited to the linguistic modality, lacking visual reasoning capabilities. Moreover, existing reasoning modules are optimized separately from navigation policies, leading to incompatibility and potential conflicts in optimization objectives. To tackle these challenges, we introduce UNeMo, a novel framework designed for the collaborative optimization of visual state reasoning and navigational decision-making. It introduces a Multimodal World Model (MWM) that takes visual features, language instructions, and navigational actions as inputs to jointly predict subsequent visual states, enabling cross-modal reasoning. Via a Hierarchical Prediction-Feedback (HPN) mechanism, MWM collaborates with navigation policies: the first layer generates actions using current vision-and-language features; MWM then infers post-action visual states to guide the second layer's fine-grained decisions. This forms a dynamic bidirectional promotion mechanism where MWM reasoning optimizes navigation policies, while policy decisions feedback to improve MWM's reasoning accuracy. Experiments on R2R and REVERIE datasets show UNeMo outperforms state-of-the-art methods by 2.1% and 0.7% in navigation accuracy for unseen scenes, validating its effectiveness.
Authors Changxin Huang, Lv Tang, Zhaohuan Zhan, Lisha Yu, Runhao Zeng, Zun Liu, Zhengjie Wang, Jianqiang Li
Categories Artificial Intelligence
3
REAL-Prover: Retrieval Augmented Lean Prover for Mathematical Reasoning
Title CN REAL-Prover：用于数学推理的检索增强型Lean定理证明器
Keywords 定理证明 · 检索增强 · 数学推理 · 大语言模型 · Lean
Summary 本文提出了REAL-Prover，一个基于大语言模型并结合检索系统Leansearch-PS的开源逐步定理证明器，用于解决大学级别的数学问题。通过自研数据提取管道HERALD-AF和交互环境Jixia-interactive，实现了从自然语言题目到形式化语句的转换与合成数据收集。在ProofNet上达到23.7%的Pass@64成绩，并在新提出的代数基准FATE-M上取得56.7%的SOTA表现。
Reason 该论文属于形式化数学推理领域的下游应用，核心贡献是构建一个面向Lean 4的定理证明流程，聚焦于将大模型应用于大学级别数学问题的自动证明。虽然涉及检索增强和推理流程设计，但其任务场景与文档图像理解（DIU）无直接关联，未提出可迁移至DIU领域的通用方法，如视觉表示学习、多模态对齐、高分辨率处理或去幻觉机制。其使用的SFT+检索策略属于常规范式，在DIU中已有广泛探索，不具备底层创新性。
Abstract Nowadays, formal theorem provers have made monumental progress on high-school and competition-level mathematics, but few of them generalize to more advanced mathematics. In this paper, we present REAL-Prover, a new open-source stepwise theorem prover for Lean 4 to push this boundary. This prover, based on our fine-tuned large language model (REAL-Prover-v1) and integrated with a retrieval system (Leansearch-PS), notably boosts performance on solving college-level mathematics problems. To train REAL-Prover-v1, we developed HERALD-AF, a data extraction pipeline that converts natural language math problems into formal statements, and a new open-source Lean 4 interactive environment (Jixia-interactive) to facilitate synthesis data collection. In our experiments, our prover using only supervised fine-tune achieves competitive results with a 23.7% success rate (Pass@64) on the ProofNet dataset-comparable to state-of-the-art (SOTA) models. To further evaluate our approach, we introduce FATE-M, a new benchmark focused on algebraic problems, where our prover achieves a SOTA success rate of 56.7% (Pass@64).
Authors Ziju Shen, Naohao Huang, Fanyi Yang, Yutong Wang, Guoxiong Gao, Tianyi Xu, Jiedong Jiang, Wanyi He, Pu Yang, Mengzhou Sun, Haocheng Ju, Peihao Wu, Bryan Dai, Bin Dong
Categories Artificial Intelligence, Computation and Language
3
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
Title CN 强化学习真的能激励大语言模型超越基础模型的推理能力吗？
Keywords 强化学习 · 推理能力 · 大语言模型 · RLVR · 知识蒸馏
Summary 本文系统评估了基于可验证奖励的强化学习（RLVR）是否真正提升了大语言模型的推理能力。通过在多个模型族、RL算法和基准测试上的实验发现，尽管RLVR在小k值下表现更好，但基础模型在大k值下覆盖率更高，说明当前RLVR并未产生根本性的新推理模式，推理能力仍受限于基础模型。相比之下，蒸馏反而能引入新的推理路径。研究呼吁需要更好的RL范式来释放潜力。
Reason 该论文研究的是强化学习对大语言模型推理能力的影响，主要聚焦于数学、编程和视觉推理任务中的pass@k指标分析。虽然涉及‘视觉推理’，但其方法论并未针对文档图像理解（DIU）中的核心问题如OCR幻觉、复杂版式或细粒度定位进行设计。且其关注点为RLVR是否激发新推理模式，属于纯NLP层面的机制探讨，未提出可迁移至文档多模态理解的架构或训练创新。整体属于通用LLM推理研究，与DIU无直接关联。
Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated notable success in enhancing the reasoning performance of large language models (LLMs), particularly on mathematics and programming tasks. Similar to how traditional RL helps agents explore and learn new strategies, RLVR is believed to enable LLMs to continuously self-improve, thus acquiring novel reasoning abilities beyond those of the corresponding base models. In this study we critically examine the current state of RLVR by systematically probing the reasoning capability boundaries of RLVR-trained LLMs across various model families, RL algorithms, and math, coding, and visual reasoning benchmarks, using pass@k at large k values as the evaluation metric. Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large. Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model. Treating the base model as an upper bound, our quantitative analysis shows that six popular RLVR algorithms perform similarly and remain far from optimal in leveraging the potential of the base model. By contrast, we find that distillation can introduce new reasoning patterns from the teacher and genuinely expand the model's reasoning capabilities. Overall, our findings suggest that current RLVR methods have not yet realized the potential of RL to elicit truly novel reasoning abilities in LLMs. This highlights the need for improved RL paradigms, such as continual scaling and multi-turn agent-environment interaction, to unlock this potential.
Authors Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, Gao Huang
Comment 31 pages, 27 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
3
How Focused Are LLMs? A Quantitative Study via Repetitive Deterministic Prediction Tasks
Title CN 大语言模型有多专注？基于重复确定性预测任务的定量研究
Keywords 大语言模型 · 推理错误 · 序列建模 · 注意力干扰 · 确定性任务
Summary 本文通过设计重复性确定性任务（如规则替换、整数加法等），发现大语言模型的生成准确率随着输出长度增加出现双指数级骤降，形成‘准确率悬崖’。作者提出一个基于统计物理的模型，解释这种失败源于提示条件作用与生成token间内部干扰的竞争。该模型可拟合实证数据，量化不同模型-任务组合的错误累积特性。
Reason 该论文研究大语言模型在重复性确定性任务中的表现，揭示其准确率随序列长度急剧下降的现象，并提出一个受统计物理启发的模型来解释注意力机制内部的干扰效应。尽管对理解LLM推理过程有一定理论价值，但其任务设置（如字符串替换、整数加法）与文档图像理解（DIU）的核心挑战（如OCR幻觉、复杂版面分析、视觉-文本对齐）无直接关联。未涉及多模态输入、视觉表示或文档结构建模，无法迁移用于解决DIU中的关键问题。
Abstract We investigate the performance of large language models on repetitive deterministic prediction tasks and study how the sequence accuracy rate scales with output length. Each such task involves repeating the same operation n times. Examples include letter replacement in strings following a given rule, integer addition, and multiplication of string operators in many body quantum mechanics. If the model performs the task through a simple repetition algorithm, the success rate should decay exponentially with sequence length. In contrast, our experiments on leading large language models reveal a sharp double exponential drop beyond a characteristic length scale, forming an accuracy cliff that marks the transition from reliable to unstable generation. This indicates that the models fail to execute each operation independently. To explain this phenomenon, we propose a statistical physics inspired model that captures the competition between external conditioning from the prompt and internal interference among generated tokens. The model quantitatively reproduces the observed crossover and provides an interpretable link between attention induced interference and sequence level failure. Fitting the model to empirical results across multiple models and tasks yields effective parameters that characterize the intrinsic error rate and error accumulation factor for each model task pair, offering a principled framework for understanding the limits of deterministic accuracy in large language models.
Authors Wanda Hou, Leon Zhou, Hong-Ye Hu, Yubei Chen, Yi-Zhuang You, Xiao-Liang Qi
Categories Artificial Intelligence
3
Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking
Title CN 从Grokking的学习动力学中推导特征涌现的可证明缩放定律
Keywords grokking · 特征涌现 · 学习动力学 · 缩放定律 · 泛化
Summary 本文提出了一种名为Li₂的新框架，用于建模两层非线性网络在grokking（延迟泛化）过程中特征涌现的三个阶段：惰性学习、独立特征学习和交互式特征学习。通过分析反向传播梯度如何携带标签信息并引导隐藏节点独立学习可泛化的特征，作者推导出特征涌现、记忆与泛化的可证明缩放律，并解释了权重衰减、学习率等超参数的作用。研究还揭示了Muon等新型优化器有效性的原理。
Reason 该论文研究的是神经网络在grokking现象下的特征涌现机制，属于理论深度学习范畴，聚焦于两层非线性网络中的梯度动力学和特征形成过程。尽管其对理解泛化与记忆的动态平衡有理论价值，但并未涉及多模态输入、视觉结构建模或文档内容理解等DIU核心问题。所提出的Li₂框架和能量函数E分析难以直接迁移至文档图像理解任务（如DocVQA、布局分析）中解决OCR幻觉、复杂排版或细粒度定位等实际痛点，也未提供可用于高分辨率图像处理或多模态对齐的架构或训练方法。因此属于上游理论工作，但对DIU领域缺乏直接可迁移性。
Abstract While the phenomenon of grokking, i.e., delayed generalization, has been studied extensively, it remains an open problem whether there is a mathematical framework that characterizes what kind of features will emerge, how and in which conditions it happens, and is closely related to the gradient dynamics of the training, for complex structured inputs. We propose a novel framework, named $\mathbf{Li}_2$, that captures three key stages for the grokking behavior of 2-layer nonlinear networks: (I) Lazy learning, (II) independent feature learning and (III) interactive feature learning. At the lazy learning stage, top layer overfits to random hidden representation and the model appears to memorize. Thanks to lazy learning and weight decay, the backpropagated gradient $G_F$ from the top layer now carries information about the target label, with a specific structure that enables each hidden node to learn their representation independently. Interestingly, the independent dynamics follows exactly the gradient ascent of an energy function $E$, and its local maxima are precisely the emerging features. We study whether these local-optima induced features are generalizable, their representation power, and how they change on sample size, in group arithmetic tasks. When hidden nodes start to interact in the later stage of learning, we provably show how $G_F$ changes to focus on missing features that need to be learned. Our study sheds lights on roles played by key hyperparameters such as weight decay, learning rate and sample sizes in grokking, leads to provable scaling laws of feature emergence, memorization and generalization, and reveals why recent optimizers such as Muon can be effective, from the first principles of gradient dynamics. Our analysis can be extended to multi-layers. The code is available at https://github.com/yuandong-tian/understanding/tree/main/ssl/real-dataset/cogo.
Authors Yuandong Tian
Categories Artificial Intelligence
3
Thinking Ahead: Foresight Intelligence in MLLMs and World Models submitted to CVPR 2026
Title CN 前瞻思考：多模态大模型与世界模型中的预见智能
Keywords 预见智能 · 世界模型 · 视觉问答 · 多模态大模型 · CVPR
Summary 本文提出了‘预见智能’的概念，旨在让模型能够预测和解释未来事件，并发布了相应的VQA数据集FSU-QA。实验表明当前主流VLM在此类任务上表现不佳，而基于该数据集微调的小模型可超越大型先进模型。此外，该数据集还可用于评估世界模型生成预测的语义连贯性。
Reason 该论文提出了一种名为Foresight Intelligence的能力，并构建了面向未来事件预测的VQA数据集FSU-QA，用于评估模型在自动驾驶等场景下的前瞻性推理能力。尽管其对世界模型和VLM的推理能力进行了探索，但任务设定主要面向视频或动态场景的未来预测，与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析、长文档结构化推理等无直接关联。所提方法难以迁移到静态文档理解场景，且未涉及视觉token压缩、高分辨率处理、细粒度定位等DIU关键技术路径。
Abstract In this work, we define Foresight Intelligence as the capability to anticipate and interpret future events-an ability essential for applications such as autonomous driving, yet largely overlooked by existing research. To bridge this gap, we introduce FSU-QA, a new Visual Question-Answering (VQA) dataset specifically designed to elicit and evaluate Foresight Intelligence. Using FSU-QA, we conduct the first comprehensive study of state-of-the-art Vision-Language Models (VLMs) under foresight-oriented tasks, revealing that current models still struggle to reason about future situations. Beyond serving as a benchmark, FSU-QA also enables the assessment of world models by measuring the semantic coherence of their generated predictions, quantified through performance gains when VLMs are augmented with such outputs. Our experiments further demonstrate that FSU-QA can effectively enhance foresight reasoning: even small VLMs fine-tuned on FSU-QA surpass much larger, advanced models by a substantial margin. Together, these findings position FSU-QA as a principled foundation for developing next-generation models capable of truly anticipating and understanding future events.
Authors Zhantao Gong, Liaoyuan Fan, Qing Guo, Xun Xu, Xulei Yang, Shijie Li
Comment 25 pages, 27 figures, submitted to CVPR 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
GMoE: Empowering LLMs Fine-Tuning via MoE Graph Collaboration
Title CN GMoE：通过专家图协作增强LLM微调
Keywords MoE · LLM微调 · 图路由器 · 负载均衡 · LoRA
Summary 本文提出GMoE框架，利用图结构路由器促进稀疏专家模型（MoE）中专家之间的信息共享与协作，通过泊松分布区分策略和正态分布平衡策略提升专家利用率和训练稳定性。采用LoRA进行参数高效微调，并在多个数据集上验证了效果。
Reason 该论文聚焦于大语言模型中MoE架构的负载均衡问题，提出通过图结构路由机制增强专家间的协作。尽管其在LLM微调效率和稳定性上有改进，但完全属于纯语言模态下的模型架构优化，未涉及视觉输入、多模态对齐或文档布局等DIU核心问题。方法难以直接迁移至文档图像理解领域，尤其是无法解决OCR幻觉、细粒度定位或高分辨率图像处理等关键挑战。
Abstract The sparse Mixture-of-Experts (MoE) architecture of large language models (LLMs) confronts an inherent issue of load imbalance arising from the simplistic linear router strategy, which ultimately causes the instability and inefficient learning of LLMs. To address this challenge, we introduce a novel MoE graph-based framework $\textbf{GMoE}$, aimed at enhancing the collaboration among multiple experts. In GMoE, a graph router function is designed to capture the collaboration signals among experts. This enables all experts to dynamically allocate information derived from input data by sharing information with their neighboring experts. Moreover, we put forward two coordination strategies in GMoE: the $\textit{Poisson distribution-based distinction strategy}$ and the $\textit{Normal distribution-based balance strategy}$, to further release the capacity of each expert and increase the model stability in the fine-tuning of LLMs. Specifically, we leverage a parameter-efficient fine-tuning technique, i.e., Low-Rank Adaptation (LoRA), to implement the graph MoE architecture. Extensive experiments on four real-world benchmark datasets demonstrate the effectiveness of GMoE, showing the benefits of facilitating collaborations of multiple experts in LLM fine-tuning. The code of experimental implementation is available at https://github.com/BAI-LAB/GMoE
Authors Ting Bai, Yue Yu, Le Huang, Zenan Xu, Chuan Shi
Comment 9 pages, 25 figures
Categories Artificial Intelligence, Computation and Language
3
Visual Anagrams Reveal Hidden Differences in Holistic Shape Processing Across Vision Models
Title CN 视觉字谜揭示不同视觉模型在整体形状处理中的隐藏差异
Keywords 形状处理 · 视觉模型比较 · 构型识别 · Transformer · 自监督学习
Summary 本文提出了一种名为Configural Shape Score（CSS）的新指标，用于评估视觉模型在保持局部纹理不变的情况下，识别经过部件重排的‘对象字谜’（Object-Anagram）的能力，从而衡量模型对全局构型形状的理解水平。研究发现，自监督和语言对齐的Transformer模型（如DINOv2、SigLIP2、EVA-CLIP）表现出更强的构型敏感性，并依赖长距离注意力机制进行全局信息整合。BagNet控制实验排除了边界线索作弊的可能性。结果表明，未来鲁棒视觉系统应同时融合局部纹理与全局形状表征。
Reason 该论文研究的是通用视觉模型对物体整体构型（configural shape）的识别能力，核心关注点是形状与纹理的表征偏好问题，属于基础视觉表示分析，与文档图像理解（DIU）中的文本识别、布局分析、表格理解等任务无直接关联。虽然其提出的Configural Shape Score（CSS）可能反映模型的全局结构感知能力，但该工作并未针对文档特有的密集文本、排版复杂性或OCR幻觉等问题设计方法，也未提供可迁移至DIU领域的具体技术工具（如高分辨率处理、细粒度定位、多步推理等）。因此属于无关领域研究，不符合DIU方法论创新的要求。
Abstract Humans are able to recognize objects based on both local texture cues and the configuration of object parts, yet contemporary vision models primarily harvest local texture cues, yielding brittle, non-compositional features. Work on shape-vs-texture bias has pitted shape and texture representations in opposition, measuring shape relative to texture, ignoring the possibility that models (and humans) can simultaneously rely on both types of cues, and obscuring the absolute quality of both types of representation. We therefore recast shape evaluation as a matter of absolute configural competence, operationalized by the Configural Shape Score (CSS), which (i) measures the ability to recognize both images in Object-Anagram pairs that preserve local texture while permuting global part arrangement to depict different object categories. Across 86 convolutional, transformer, and hybrid models, CSS (ii) uncovers a broad spectrum of configural sensitivity with fully self-supervised and language-aligned transformers -- exemplified by DINOv2, SigLIP2 and EVA-CLIP -- occupying the top end of the CSS spectrum. Mechanistic probes reveal that (iii) high-CSS networks depend on long-range interactions: radius-controlled attention masks abolish performance showing a distinctive U-shaped integration profile, and representational-similarity analyses expose a mid-depth transition from local to global coding. A BagNet control remains at chance (iv), ruling out "border-hacking" strategies. Finally, (v) we show that configural shape score also predicts other shape-dependent evals. Overall, we propose that the path toward truly robust, generalizable, and human-like vision systems may not lie in forcing an artificial choice between shape and texture, but rather in architectural and learning frameworks that seamlessly integrate both local-texture and global configural shape.
Authors Fenil R. Doshi, Thomas Fel, Talia Konkle, George Alvarez
Comment Project page: https://www.fenildoshi.com/configural-shape/ updated email address
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
TRAP: Targeted Redirecting of Agentic Preferences NeurIPS 2025
Title CN TRAP：针对代理偏好的定向重定向
Keywords 对抗攻击 · 视觉语言模型 · 代理系统 · 语义注入 · 跨模态推理
Summary 本文提出TRAP，一种基于扩散模型的生成式对抗框架，通过在视觉-语言嵌入空间中注入语义信息来操纵代理AI系统的决策。该方法无需访问模型内部结构，利用负提示降级与正向语义优化结合，在保持图像视觉自然性的同时诱导代理产生选择偏好偏差。在COCO构建的多候选决策场景中验证了其对LLaVA、GPT-4o等主流VLM的有效攻击能力，揭示了跨模态代理系统存在的语义级脆弱性。
Reason 该论文研究的是对视觉语言代理（agentic AI）的对抗性攻击，通过语义注入操纵其决策过程。虽然涉及VLM和跨模态推理，但核心是安全与对抗攻击问题，属于AI安全中的语义攻击范畴，且实验基于COCO等通用视觉场景，并未针对文档图像理解（DIU）任务如DocVQA、表格识别或布局分析进行设计或验证。尽管其‘语义空间操纵’技术在理论上可能启发幻觉控制，但缺乏向DIU迁移的具体路径或证据，属于无关领域的平行应用。
Abstract Autonomous agentic AI systems powered by vision-language models (VLMs) are rapidly advancing toward real-world deployment, yet their cross-modal reasoning capabilities introduce new attack surfaces for adversarial manipulation that exploit semantic reasoning across modalities. Existing adversarial attacks typically rely on visible pixel perturbations or require privileged model or environment access, making them impractical for stealthy, real-world exploitation. We introduce TRAP, a novel generative adversarial framework that manipulates the agent's decision-making using diffusion-based semantic injections into the vision-language embedding space. Our method combines negative prompt-based degradation with positive semantic optimization, guided by a Siamese semantic network and layout-aware spatial masking. Without requiring access to model internals, TRAP produces visually natural images yet induces consistent selection biases in agentic AI systems. We evaluate TRAP on the Microsoft Common Objects in Context (COCO) dataset, building multi-candidate decision scenarios. Across these scenarios, TRAP consistently induces decision-level preference redirection on leading models, including LLaVA-34B, Gemma3, GPT-4o, and Mistral-3.2, significantly outperforming existing baselines such as SPSA, Bandit, and standard diffusion approaches. These findings expose a critical, generalized vulnerability: autonomous agents can be consistently misled through visually subtle, semantically-guided cross-modal manipulations. Overall, our results show the need for defense strategies beyond pixel-level robustness to address semantic vulnerabilities in cross-modal decision-making. The code for TRAP is accessible on GitHub at https://github.com/uiuc-focal-lab/TRAP.
Authors Hangoo Kang, Jehyeok Yeon, Gagandeep Singh
Comment Accepted to NeurIPS 2025
Categories Artificial Intelligence
3
Vision Language Models Can Parse Floor Plan Maps
Title CN 视觉语言模型可以解析平面图地图
Keywords 视觉语言模型 · 地图解析 · 平面图 · 室内导航 · 任务规划
Summary 本文研究视觉语言模型在平面图地图解析中的应用，通过提示VLM生成室内导航任务计划来评估其性能。实验显示VLM在包含九步动作序列的导航任务中成功率高达0.96，但在大范围开放区域表现下降。作者提出了应对该问题的实用建议。研究主要面向机器人应用场景。
Reason 该论文将VLM应用于平面图解析任务，属于特定场景下的下游应用（Wrapper/Application），尽管任务具有一定新颖性，但其核心是将现有VLM能力应用于机器人导航相关的地图理解，未提出针对文档图像理解的通用方法论创新。虽然平面图具有类似文档的结构化布局，但研究目标聚焦于导航任务规划，而非解决DIU的核心痛点如OCR幻觉、密集文本处理或复杂排版建模。因此属于平行应用，缺乏可迁移的底层技术创新。
Abstract Vision language models (VLMs) can simultaneously reason about images and texts to tackle many tasks, from visual question answering to image captioning. This paper focuses on map parsing, a novel task that is unexplored within the VLM context and particularly useful to mobile robots. Map parsing requires understanding not only the labels but also the geometric configurations of a map, i.e., what areas are like and how they are connected. To evaluate the performance of VLMs on map parsing, we prompt VLMs with floor plan maps to generate task plans for complex indoor navigation. Our results demonstrate the remarkable capability of VLMs in map parsing, with a success rate of 0.96 in tasks requiring a sequence of nine navigation actions, e.g., approaching and going through doors. Other than intuitive observations, e.g., VLMs do better in smaller maps and simpler navigation tasks, there was a very interesting observation that its performance drops in large open areas. We provide practical suggestions to address such challenges as validated by our experimental results. Webpage: https://sites.google.com/view/vlm-floorplan/
Authors David DeFazio, Hrudayangam Mehta, Meng Wang, Ping Yang, Jeremy Blackburn, Shiqi Zhang
Categories Artificial Intelligence
3
A Layered Protocol Architecture for the Internet of Agents
Title CN 面向智能体互联网的分层协议架构
Keywords 智能体通信 · 多智能体系统 · 协议架构 · 语义协商 · LLM协作
Summary 本文提出了一个为‘智能体互联网’（IoA）设计的分层协议架构，包含第8层（Agent Communication Layer）和第9层（Agent Semantic Negotiation Layer），旨在标准化智能体之间的通信结构与语义协商机制，以支持跨智能体的上下文共享和协作计算。然而，该工作聚焦于通用智能体通信协议，未涉及文档图像理解的具体挑战或应用场景。
Reason 该论文提出了一种用于智能体间通信与语义协商的网络协议架构（L8/L9层），属于多智能体系统协作的通用框架设计。尽管其理念可能对长文档推理中的多步信息检索有一定启发，但本质上是平行应用层面的系统架构提案，并未针对文档图像理解（DIU）中的具体痛点如OCR幻觉、密集文本识别、复杂版式分析或视觉-语言对齐进行方法改进。且无任何在文档理解任务上的验证或迁移路径说明，属于典型的‘无关领域’+‘非技术源头’的下游包装工作。
Abstract Large Language Models (LLMs) have demonstrated remarkable performance improvements and the ability to learn domain-specific languages (DSLs), including APIs and tool interfaces. This capability has enabled the creation of AI agents that can perform preliminary computations and act through tool calling, now being standardized via protocols like MCP. However, LLMs face fundamental limitations: their context windows cannot grow indefinitely, constraining their memory and computational capacity. Agent collaboration emerges as essential for solving increasingly complex problems, mirroring how computational systems rely on different types of memory to scale. The "Internet of Agents" (IoA) represents the communication stack that enables agents to scale by distributing computation across collaborating entities. Current network architectural stacks (OSI and TCP/IP) were designed for data delivery between hosts and processes, not for agent collaboration with semantic understanding. To address this gap, we propose two new layers: an \textbf{Agent Communication Layer (L8)} and an \textbf{Agent Semantic Negotiation Layer (L9)}. L8 formalizes the \textit{structure} of communication, standardizing message envelopes, speech-act performatives (e.g., REQUEST, INFORM), and interaction patterns (e.g., request-reply, publish-subscribe), building on protocols like MCP. L9, which does not exist today, formalizes the \textit{meaning} of communication, enabling agents to discover, negotiate, and lock a "Shared Context" -- a formal schema defining the concepts, tasks, and parameters relevant to their interaction. Together, these layers provide the foundation for scalable, distributed agent collaboration, enabling the next generation of multi-agentic systems.
Authors Charles Fleming, Vijoy Pandey, Ramana Kompella, Luca Muscariello
Categories Artificial Intelligence
3
LLM Collaboration With Multi-Agent Reinforcement Learning
Title CN 基于多智能体强化学习的LLM协作
Keywords 多智能体系统 · 强化学习 · LLM协作 · 策略优化 · MARL
Summary 本文提出将大语言模型（LLM）的协作建模为多智能体强化学习（MARL）问题，并设计了一种名为MAGRPO的多智能体策略优化算法，在写作和代码生成任务上验证了协作效率的提升。然而，该工作聚焦于纯文本任务中的智能体协调，未涉及视觉模态或文档结构信息。
Reason 该论文研究的是多智能体系统中LLM之间的协作机制，属于典型的‘平行应用’类工作：它将现有LLM应用于多智能体框架，并引入MARL进行联合优化。尽管MAGRPO算法有一定方法论成分，但其核心场景（写作与编程协作）与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂版面分析、细粒度视觉定位等无直接关联。该方法难以迁移至文档理解任务中的视觉-语言对齐或长文档结构解析，且未涉及视觉输入处理，因此对DIU领域缺乏实质性推动。
Abstract A large amount of work has been done in Multi-Agent Systems (MAS) for modeling and solving problems with multiple interacting agents. However, most LLMs are pretrained independently and not specifically optimized for coordination. Existing LLM fine-tuning frameworks rely on individual rewards, which require complex reward designs for each agent to encourage collaboration. To address these challenges, we model LLM collaboration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. We develop a multi-agent, multi-turn algorithm, Multi-Agent Group Relative Policy Optimization (MAGRPO), to solve it, building on current RL approaches for LLMs as well as MARL techniques. Our experiments on LLM writing and coding collaboration demonstrate that fine-tuning MAS with MAGRPO enables agents to generate high-quality responses efficiently through effective cooperation. Our approach opens the door to using other MARL methods for LLMs and highlights the associated challenges. Our code is available at https://github.com/OpenMLRL/CoMLRL.
Authors Shuo Liu, Tianle Chen, Zeyu Liang, Xueguang Lyu, Christopher Amato
Categories Artificial Intelligence
3
Fara-7B: An Efficient Agentic Model for Computer Use
Title CN Fara-7B：一种用于计算机操作的高效智能体模型
Keywords 智能体 · 计算机使用 · 合成数据 · 多步任务 · 截图到动作
Summary 本文提出了Fara-7B，一个基于截图输入、通过坐标预测执行操作的小型智能体模型，用于完成多步网页任务。作者设计了FaraGen系统来自动生成和筛选高质量的人机交互轨迹数据，并用这些数据训练模型，在多个Web操作基准上取得了优于同规模模型的表现，甚至媲美更大模型。同时发布了新基准WebTailBench和模型权重。
Reason 该论文属于典型的‘平行应用’类型，聚焦于计算机使用智能体（CUA）在网页任务上的执行，其核心贡献是构建合成数据集FaraGen和训练特定用途的智能体模型。尽管涉及视觉输入（截图）和动作预测，但其方法论未针对文档图像理解中的核心痛点（如OCR幻觉、密集文本识别、复杂版式建模或长文档推理）提出可迁移的底层技术创新。虽然其多步验证机制有一定通用性，但整体工作仍属于下游应用场景构建，不符合DIU本题或上游方法迁移的标准。
Abstract Progress in computer use agents (CUAs) has been constrained by the absence of large and high-quality datasets that capture how humans interact with a computer. While LLMs have thrived on abundant textual data, no comparable corpus exists for CUA trajectories. To address these gaps, we introduce FaraGen, a novel synthetic data generation system for multi-step web tasks. FaraGen can propose diverse tasks from frequently used websites, generate multiple solution attempts, and filter successful trajectories using multiple verifiers. It achieves high throughput, yield, and diversity for multi-step web tasks, producing verified trajectories at approximately $1 each. We use this data to train Fara-7B, a native CUA model that perceives the computer using only screenshots, executes actions via predicted coordinates, and is small enough to run on-device. We find that Fara-7B outperforms other CUA models of comparable size on benchmarks like WebVoyager, Online-Mind2Web, and WebTailBench -- our novel benchmark that better captures under-represented web tasks in pre-existing benchmarks. Furthermore, Fara-7B is competitive with much larger frontier models, illustrating key benefits of scalable data generation systems in advancing small efficient agentic models. We are making Fara-7B open-weight on Microsoft Foundry and HuggingFace, and we are releasing WebTailBench.
Authors Ahmed Awadallah, Yash Lara, Raghav Magazine, Hussein Mozannar, Akshay Nambi, Yash Pandya, Aravind Rajeswaran, Corby Rosset, Alexey Taymanov, Vibhav Vineet, Spencer Whitehead, Andrew Zhao
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
3
Efficient Multi-Hop Question Answering over Knowledge Graphs via LLM Planning and Embedding-Guided Search
Title CN 基于大模型规划与嵌入引导搜索的高效多跳知识图谱问答
Keywords 知识图谱 · 多跳问答 · 大语言模型规划 · 嵌入引导搜索 · 可验证推理
Summary 本文提出两种混合算法以提升知识图谱上多跳问答的效率与可验证性：一是利用单次LLM调用生成关系序列并结合广度优先搜索进行推理；二是完全去除LLM调用，使用轻量级边评分器融合文本与图嵌入进行神经搜索。实验表明，结构化规划比直接生成更具可解释性和迁移性，在保持高性能的同时大幅降低计算成本。
Reason 该论文研究的是知识图谱上的多跳问答问题，其核心方法（LLM规划+嵌入引导搜索）虽然在推理效率和结果可验证性上有提升，但属于典型的‘平行下游应用’模式：将LLM用于特定结构化数据（KG）的推理任务。尽管其‘规划+搜索’的思想具有一定通用性，但并未针对文档图像理解中的关键痛点（如OCR幻觉、复杂排版、高分辨率输入处理、视觉-文本对齐等）提出可迁移的底层技术创新。且任务场景完全脱离视觉或文档模态，无法直接服务于DIU。
Abstract Multi-hop question answering over knowledge graphs remains computationally challenging due to the combinatorial explosion of possible reasoning paths. Recent approaches rely on expensive Large Language Model (LLM) inference for both entity linking and path ranking, limiting their practical deployment. Additionally, LLM-generated answers often lack verifiable grounding in structured knowledge. We present two complementary hybrid algorithms that address both efficiency and verifiability: (1) LLM-Guided Planning that uses a single LLM call to predict relation sequences executed via breadth-first search, achieving near-perfect accuracy (micro-F1 > 0.90) while ensuring all answers are grounded in the knowledge graph, and (2) Embedding-Guided Neural Search that eliminates LLM calls entirely by fusing text and graph embeddings through a lightweight 6.7M-parameter edge scorer, achieving over 100 times speedup with competitive accuracy. Through knowledge distillation, we compress planning capability into a 4B-parameter model that matches large-model performance at zero API cost. Evaluation on MetaQA demonstrates that grounded reasoning consistently outperforms ungrounded generation, with structured planning proving more transferable than direct answer generation. Our results show that verifiable multi-hop reasoning does not require massive models at inference time, but rather the right architectural inductive biases combining symbolic structure with learned representations.
Authors Manil Shrestha, Edward Kim
Categories Artificial Intelligence, Computation and Language
3
SafeFix: Targeted Model Repair via Controlled Image Generation
Title CN SafeFix：通过可控图像生成进行定向模型修复
Keywords 模型修复 · 合成数据 · 视觉语言模型 · 文本到图像生成 · 语义一致性
Summary 该研究提出SafeFix，一种基于可解释归因的模型修复方法，利用条件文本到图像生成合成针对失败案例的训练样本，并用大视觉语言模型过滤以保证语义一致性和分布对齐，从而提升模型在罕见子群体上的鲁棒性。
Reason 该论文聚焦于通用视觉识别中的模型调试与修复，使用文本到图像生成结合LVLM过滤来增强罕见样本的训练数据。尽管使用了LVLM和图像生成技术，但其核心是针对分类等通用CV任务的模型修复框架，属于特定下游问题的解决方案，未涉及文档结构、布局理解、OCR错误修正或文档特有的密集文本与排版建模。无法直接迁移解决文档图像理解中的核心痛点如OCR幻觉、细粒度定位或长文档推理，属于平行应用类方法。
Abstract Deep learning models for visual recognition often exhibit systematic errors due to underrepresented semantic subpopulations. Although existing debugging frameworks can pinpoint these failures by identifying key failure attributes, repairing the model effectively remains difficult. Current solutions often rely on manually designed prompts to generate synthetic training images -- an approach prone to distribution shift and semantic errors. To overcome these challenges, we introduce a model repair module that builds on an interpretable failure attribution pipeline. Our approach uses a conditional text-to-image model to generate semantically faithful and targeted images for failure cases. To preserve the quality and relevance of the generated samples, we further employ a large vision-language model (LVLM) to filter the outputs, enforcing alignment with the original data distribution and maintaining semantic consistency. By retraining vision models with this rare-case-augmented synthetic dataset, we significantly reduce errors associated with rare cases. Our experiments demonstrate that this targeted repair strategy improves model robustness without introducing new bugs. Code is available at https://github.com/oxu2/SafeFix
Authors Ouyang Xu, Baoming Zhang, Ruiyu Mao, Yunhui Guo
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Many Ways to be Right: Rashomon Sets for Concept-Based Neural Networks
Title CN 多种正确方式：面向基于概念的神经网络的Rashomon集合
Keywords Rashomon效应 · 概念瓶颈模型 · 模型多样性 · 可解释性 · 适配器
Summary 该论文提出了Rashomon概念瓶颈模型，旨在构建多个准确但基于不同人类可理解概念进行推理的神经网络。通过轻量级适配器模块和多样性正则化训练目标，避免从头训练即可高效生成行为差异化的模型集合，从而揭示相同性能下不同的决策逻辑。该方法有助于深度模型的审计、比较与对齐。
Reason 该论文研究的是在概念瓶颈模型中构建具有多样化推理路径但性能相当的模型集合，强调模型间的‘合理性多样性’。虽然其关于模型多样性与可解释性的探讨具有一定理论价值，但其核心任务并非文档图像理解（DIU）中的具体问题（如DocVQA、布局分析、表格识别等），也未涉及多模态输入、视觉文本联合建模或高分辨率图像处理等DIU关键技术挑战。提出的方法属于模型解释性与鲁棒性研究范畴，难以直接迁移用于解决OCR幻觉、密集文本识别或复杂排版理解等实际DIU痛点，且无证据表明其适配于视觉-语言架构或文档场景。
Abstract Modern neural networks rarely have a single way to be right. For many tasks, multiple models can achieve identical performance while relying on different features or reasoning patterns, a property known as the Rashomon Effect. However, uncovering this diversity in deep architectures is challenging as their continuous parameter spaces contain countless near-optimal solutions that are numerically distinct but often behaviorally similar. We introduce Rashomon Concept Bottleneck Models, a framework that learns multiple neural networks which are all accurate yet reason through distinct human-understandable concepts. By combining lightweight adapter modules with a diversity-regularized training objective, our method constructs a diverse set of deep concept-based models efficiently without retraining from scratch. The resulting networks provide fundamentally different reasoning processes for the same predictions, revealing how concept reliance and decision making vary across equally performing solutions. Our framework enables systematic exploration of data-driven reasoning diversity in deep models, offering a new mechanism for auditing, comparison, and alignment across equally accurate solutions.
Authors Shihan Feng, Cheng Zhang, Michael Xi, Ethan Hsu, Lesia Semenova, Chudi Zhong
Categories Artificial Intelligence
3
Trust-Based Social Learning for Communication (TSLEC) Protocol Evolution in Multi-Agent Reinforcement Learning
Title CN 基于信任的多智能体强化学习中通信协议演化
Keywords 多智能体强化学习 · 涌现通信 · 信任机制 · 社会学习 · 协议演化
Summary 本文提出TSLEC框架，在多智能体强化学习中通过引入信任机制调节智能体间的策略传授，加速通信协议的演化并提升鲁棒性。实验表明该方法显著减少收敛所需轮次，并生成结构化通信协议。然而，该工作聚焦于多智能体协作中的通信生成，不涉及任何形式的文档理解或多模态输入处理。
Reason 该论文研究多智能体系统中的 emergent communication 和信任机制，属于具身智能与多智能体协同领域。虽然涉及‘社会学习’和‘知识传递’，但其核心任务与文档图像理解（DIU）无直接关联，无法迁移用于解决OCR幻觉、复杂排版或长文档推理等关键问题。属于无关领域应用，且为典型的平行研究（multi-agent communication protocol），对DIU方法论无贡献。
Abstract Emergent communication in multi-agent systems typically occurs through independent learning, resulting in slow convergence and potentially suboptimal protocols. We introduce TSLEC (Trust-Based Social Learning with Emergent Communication), a framework where agents explicitly teach successful strategies to peers, with knowledge transfer modulated by learned trust relationships. Through experiments with 100 episodes across 30 random seeds, we demonstrate that trust-based social learning reduces episodes-to-convergence by 23.9% (p < 0.001, Cohen's d = 1.98) compared to independent emergence, while producing compositional protocols (C = 0.38) that remain robust under dynamic objectives (Phi > 0.867 decoding accuracy). Trust scores strongly correlate with teaching quality (r = 0.743, p < 0.001), enabling effective knowledge filtering. Our results establish that explicit social learning fundamentally accelerates emergent communication in multi-agent coordination.
Authors Abraham Itzhak Weinberg
Categories Artificial Intelligence
3
Merging without Forgetting: Continual Fusion of Task-Specific Models via Optimal Transport
Title CN 不忘融合：基于最优传输的持续任务模型融合
Keywords 模型融合 · 最优传输 · 多任务学习 · 参数插值 · 持续学习
Summary 本文提出OTMF，一种基于最优传输理论的模型融合框架，旨在解决多任务微调模型合并时因参数插值导致的特征分布偏移问题。该方法通过发现任务向量上的公共掩码来对齐语义几何结构，在保留任务特异性的同时提取可迁移组件，并支持无需回访旧任务的持续融合模式。实验表明其在多个视觉与语言基准上优于现有融合方法。
Reason 该论文研究的是多任务模型融合问题，提出通过最优传输理论对不同任务微调后的模型进行参数融合。尽管其方法在理论上具有通用性，但本质上属于‘平行应用’范畴——它并未针对文档图像理解（DIU）中的核心痛点（如OCR幻觉、复杂排版、细粒度定位或长文档推理）设计专门机制。虽然模型融合技术可能被用于集成多个DIU子任务模型，但该工作本身未涉及文档结构、视觉布局或文本-视觉对齐等关键问题，且无任何在DocVQA、表格识别或布局分析上的验证。因此属于通用模型集成方法在多模态场景下的常规延伸，缺乏对DIU领域的方法论创新。
Abstract Merging models fine-tuned for different tasks into a single unified model has become an increasingly important direction for building versatile, efficient multi-task systems. Existing approaches predominantly rely on parameter interpolation in weight space, which we show introduces significant distribution shift in the feature space and undermines task-specific knowledge. In this paper, we propose OTMF (Optimal Transport-based Masked Fusion), a novel model merging framework rooted in optimal transport theory to address the distribution shift that arises from naive parameter interpolation. Instead of directly aggregating features or weights, OTMF aligns the semantic geometry of task-specific models by discovering common masks applied to task vectors through optimal transport plans. These masks selectively extract transferable and task-agnostic components while preserving the unique structural identities of each task. To ensure scalability in real-world settings, OTMF further supports a continual fusion paradigm that incrementally integrates each new task vector without revisiting previous ones, maintaining a bounded memory footprint and enabling efficient fusion across a growing number of tasks. We conduct comprehensive experiments on multiple vision and language benchmarks, and results show that OTMF achieves state-of-the-art performance in terms of both accuracy and efficiency. These findings highlight the practical and theoretical value of our approach to model merging.
Authors Zecheng Pan, Zhikang Chen, Ding Li, Min Zhang, Sen Cui, Hongshuo Jin, Luqi Tao, Yi Yang, Deheng Ye, Yu Zhang, Tingting Zhu, Tianling Ren
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Think First, Assign Next (ThiFAN-VQA): A Two-stage Chain-of-Thought Framework for Post-Disaster Damage Assessment
Title CN 先思考，再分配（ThiFAN-VQA）：一种用于灾后损害评估的两阶段思维链框架
Keywords 视觉问答 · 思维链 · 灾后评估 · 遥感图像 · 幻觉抑制
Summary 本文提出ThiFAN-VQA，一种用于灾后损害评估的两阶段视觉问答框架。该方法首先通过思维链提示生成推理路径，再通过答案选择模块筛选最一致的回答，结合领域特定检索与提示设计，在FloodNet和RescueNet-VQA数据集上实现了更准确且可解释的结果。目标是解决遥感图像中少样本、开放域问答下的幻觉与泛化问题。
Reason 该论文聚焦于灾后遥感图像中的VQA任务，属于特定垂直领域的下游应用。其核心方法为在灾害场景下结合ICL与两阶段CoT推理以缓解标注数据不足和幻觉问题，但未提出可迁移至文档图像理解（DIU）的通用底层技术创新。尽管其答案选择机制可能对去幻觉有一定启发，但整体架构紧密耦合于遥感与灾害评估场景，难以直接迁移到OCR密集、排版复杂或长文档推理等DIU核心痛点，属于‘平行应用’范畴。
Abstract Timely and accurate assessment of damages following natural disasters is essential for effective emergency response and recovery. Recent AI-based frameworks have been developed to analyze large volumes of aerial imagery collected by Unmanned Aerial Vehicles, providing actionable insights rapidly. However, creating and annotating data for training these models is costly and time-consuming, resulting in datasets that are limited in size and diversity. Furthermore, most existing approaches rely on traditional classification-based frameworks with fixed answer spaces, restricting their ability to provide new information without additional data collection or model retraining. Using pre-trained generative models built on in-context learning (ICL) allows for flexible and open-ended answer spaces. However, these models often generate hallucinated outputs or produce generic responses that lack domain-specific relevance. To address these limitations, we propose ThiFAN-VQA, a two-stage reasoning-based framework for visual question answering (VQA) in disaster scenarios. ThiFAN-VQA first generates structured reasoning traces using chain-of-thought (CoT) prompting and ICL to enable interpretable reasoning under limited supervision. A subsequent answer selection module evaluates the generated responses and assigns the most coherent and contextually accurate answer, effectively improve the model performance. By integrating a custom information retrieval system, domain-specific prompting, and reasoning-guided answer selection, ThiFAN-VQA bridges the gap between zero-shot and supervised methods, combining flexibility with consistency. Experiments on FloodNet and RescueNet-VQA, UAV-based datasets from flood- and hurricane-affected regions, demonstrate that ThiFAN-VQA achieves superior accuracy, interpretability, and adaptability for real-world post-disaster damage assessment tasks.
Authors Ehsan Karimi, Nhut Le, Maryam Rahnemoonfar
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectories
Title CN 发现、学习与强化：通过多样化强化学习生成轨迹扩展视觉-语言-动作预训练
Keywords 强化学习 · 行为多样性 · 视觉语言动作模型 · 具身智能 · 预训练
Summary 本文提出DLR框架，利用信息论方法在强化学习中发现多种高成功率的行为模式，以提升视觉-语言-动作（VLA）模型预训练的数据多样性。相比传统RL只能学到单一策略，DLR能生成多样化成功轨迹，在LIBERO数据集上验证了其在状态-动作空间覆盖和下游任务泛化能力上的优势。然而，该工作属于机器人与具身智能范畴，与文档理解无关。
Reason 该论文属于具身智能（Embodied AI）领域，研究通过多模式强化学习生成多样化的操作轨迹用于VLA模型预训练。尽管其方法论涉及数据多样性与行为探索的创新，但应用场景聚焦于机器人控制与物理交互任务，与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版解析、细粒度文本定位等无直接关联。所提框架难以迁移至非具身、静态图像+语言的文档理解场景，且未涉及视觉token压缩、高分辨率处理或多模态对齐等DIU关键技术路径。
Abstract Scaling vision-language-action (VLA) model pre-training requires large volumes of diverse, high-quality manipulation trajectories. Most current data is obtained via human teleoperation, which is expensive and difficult to scale. Reinforcement learning (RL) methods learn useful skills through autonomous exploration, making them a viable approach for generating data. However, standard RL training collapses to a narrow execution pattern, limiting its utility for large-scale pre-training. We propose Discover, Lea rn and Reinforce (DLR), an information-theoretic pattern discovery framework that generates multiple distinct, high-success behavioral patterns for VLA pretraining. Empirically, DLR generates a markedly more diverse trajectory corpus on LIBERO. Specifically, it learns multiple distinct, high-success strategies for the same task where standard RL discovers only one, and hence it covers substantially broader regions of the state-action space. When adapted to unseen downstream task suites, VLA models pretrained on our diverse RL data surpass counterparts trained on equal-sized standard RL datasets. Moreover, DLR exhibits positive data-scaling behavior that single-pattern RL lacks. These results position multi-pattern RL as a practical, scalable data engine for embodied foundation models.
Authors Rushuai Yang, Zhiyuan Feng, Tianxiang Zhang, Kaixin Wang, Chuheng Zhang, Li Zhao, Xiu Su, Yi Chen, Jiang Bian
Categories Artificial Intelligence
3
Active Slice Discovery in Large Language Models NeurIPS 2025 - Reliable ML Workshop
Title CN 大语言模型中的主动切片发现
Keywords 主动学习 · 错误切片 · 模型可靠性 · 大语言模型 · NeurIPS
Summary 本文提出了“主动切片发现”框架，旨在通过主动学习策略高效识别大语言模型在特定数据子集上的系统性错误。实验集中在毒性评论分类任务中发现人为定义的错误切片，评估了不同特征表示和主动学习算法的效果，结果显示基于不确定性的方法仅用2%-10%的标注信息即可有效发现错误模式。
Reason 该论文研究的是在毒性评论分类等任务中，通过主动学习方式发现模型表现不佳的特定数据子集（error slices），属于模型可信赖性分析（Reliable ML）范畴。虽然其方法涉及LLM错误分析，但任务本身与文档图像理解（DIU）的核心问题如OCR幻觉、布局分析、表格识别等无直接关联。其提出的技术难以迁移用于解决文档理解中的视觉-语言对齐、高分辨率处理或细粒度文本定位等关键挑战，属于平行的方法论应用而非底层技术创新。
Abstract Large Language Models (LLMs) often exhibit systematic errors on specific subsets of data, known as error slices. For instance, a slice can correspond to a certain demographic, where a model does poorly in identifying toxic comments regarding that demographic. Identifying error slices is crucial to understanding and improving models, but it is also challenging. An appealing approach to reduce the amount of manual annotation required is to actively group errors that are likely to belong to the same slice, while using limited access to an annotator to verify whether the chosen samples share the same pattern of model mistake. In this paper, we formalize this approach as Active Slice Discovery and explore it empirically on a problem of discovering human-defined slices in toxicity classification. We examine the efficacy of active slice discovery under different choices of feature representations and active learning algorithms. On several slices, we find that uncertainty-based active learning algorithms are most effective, achieving competitive accuracy using 2-10% of the available slice membership information, while significantly outperforming baselines.
Authors Minhui Zhang, Prahar Ijner, Yoav Wald, Elliot Creager
Comment Accepted for presentation at NeurIPS 2025 - Reliable ML Workshop
Categories Artificial Intelligence
3
Post-Pruning Accuracy Recovery via Data-Free Knowledge Distillation
Title CN 通过无数据知识蒸馏实现剪枝后精度恢复
Keywords 模型剪枝 · 无数据知识蒸馏 · DeepInversion · 隐私保护 · 模型压缩
Summary 本文提出一种无数据知识蒸馏框架，利用DeepInversion从预训练教师模型中反演批量归一化统计量生成“梦境”图像，用于在不访问原始数据的情况下将知识从教师模型迁移到剪枝后的学生模型，在CIFAR-10上实现了剪枝后精度的有效恢复。整个流程不依赖任何真实训练样本，适用于医疗、金融等隐私敏感场景。
Reason 该论文研究的是通用模型压缩中的无数据知识蒸馏方法，主要面向图像分类任务（如CIFAR-10），且未涉及多模态、文档结构或视觉-语言推理。虽然模型压缩技术在部署中具有价值，但其方法针对的是标准CNN架构和自然图像，无法直接迁移解决文档图像理解中的核心痛点（如OCR幻觉、复杂版式建模、细粒度文本定位等）。属于与DIU无关的下游优化工作，且无针对文档特性的适配设计。
Abstract Model pruning is a widely adopted technique to reduce the computational complexity and memory footprint of Deep Neural Networks (DNNs). However, global unstructured pruning often leads to significant degradation in accuracy, typically necessitating fine-tuning on the original training dataset to recover performance. In privacy-sensitive domains such as healthcare or finance, access to the original training data is often restricted post-deployment due to regulations (e.g., GDPR, HIPAA). This paper proposes a Data-Free Knowledge Distillation framework to bridge the gap between model compression and data privacy. We utilize DeepInversion to synthesize privacy-preserving ``dream'' images from the pre-trained teacher model by inverting Batch Normalization (BN) statistics. These synthetic images serve as a transfer set to distill knowledge from the original teacher to the pruned student network. Experimental results on CIFAR-10 across various architectures (ResNet, MobileNet, VGG) demonstrate that our method significantly recovers accuracy lost during pruning without accessing a single real data point.
Authors Chinmay Tripurwar, Utkarsh Maurya, Dishant
Categories Artificial Intelligence
3
UISearch: Graph-Based Embeddings for Multimodal Enterprise UI Screenshots Retrieval
Title CN UISearch：基于图的多模态企业级UI截图检索嵌入方法
Keywords UI检索 · 图神经网络 · 多模态嵌入 · 对比学习 · 结构化表示
Summary 本文提出一种基于图的表示方法，将企业级UI截图转化为属性图以捕捉层次结构和空间布局，并通过对比图自编码器学习多模态嵌入，用于高效多模态检索。系统在2万多个金融软件UI上实现了高准确率和低延迟的搜索性能。
Reason 该论文聚焦于企业软件用户界面（UI）截图的检索，提出将UI转换为属性图并结合对比图自编码器进行多模态检索。尽管其图结构建模和空间层次编码在形式上与文档布局分析有表面相似性，但其核心任务是UI设计资产检索，而非文档理解中的文本识别、排版解析或VQA等核心问题。作者虽提及‘可能泛化到文档布局’，但未针对OCR幻觉、密集文本处理、长文档推理等DIU关键痛点设计任何机制，也未使用标准DIU数据集验证。属于典型的平行应用类工作，缺乏对文档理解领域的方法论贡献。
Abstract Enterprise software companies maintain thousands of user interface screens across products and versions, creating critical challenges for design consistency, pattern discovery, and compliance check. Existing approaches rely on visual similarity or text semantics, lacking explicit modeling of structural properties fundamental to user interface (UI) composition. We present a novel graph-based representation that converts UI screenshots into attributed graphs encoding hierarchical relationships and spatial arrangements, potentially generalizable to document layouts, architectural diagrams, and other structured visual domains. A contrastive graph autoencoder learns embeddings preserving multi-level similarity across visual, structural, and semantic properties. The comprehensive analysis demonstrates that our structural embeddings achieve better discriminative power than state-of-the-art Vision Encoders, representing a fundamental advance in the expressiveness of the UI representation. We implement this representation in UISearch, a multi-modal search framework that combines structural embeddings with semantic search through a composable query language. On 20,396 financial software UIs, UISearch achieves 0.92 Top-5 accuracy with 47.5ms median latency (P95: 124ms), scaling to 20,000+ screens. The hybrid indexing architecture enables complex queries and supports fine-grained UI distinction impossible with vision-only approaches.
Authors Maroun Ayli, Youssef Bakouny, Tushar Sharma, Nader Jalloul, Hani Seifeddine, Rima Kilany
Comment 12 pages, 2 figures, 3 algorithms, 4 tables
Categories Computer Vision and Pattern Recognition
3
Syn-GRPO: Self-Evolving Data Synthesis for MLLM Perception Reasoning
Title CN Syn-GRPO：用于多模态大语言模型感知推理的自进化数据合成
Keywords 多模态大语言模型 · 强化学习 · 数据合成 · 感知推理 · 数据增强
Summary 本文提出Syn-GRPO，一种结合在线数据生成与GRPO强化学习的框架，通过图像生成模型从现有样本中合成新数据，并引入多样性奖励机制以促进多模态大模型产生多样化响应。系统包含数据服务器和GRPO工作流两部分，实现高效的数据合成与训练闭环。在多个视觉感知任务上验证了其提升数据质量和模型性能的能力。
Reason 该论文提出一种基于强化学习（GRPO）框架下的在线数据生成方法，旨在提升MLLM感知任务中的数据多样性与质量。虽然其动机涉及提升多模态模型的感知能力，但核心贡献是数据增强层面的技术优化，且实验集中在通用视觉感知任务，并未针对文档图像理解（如OCR、排版分析、表格识别等）设计任务或结构。该方法属于对现有RL训练流程的改进，缺乏面向DIU领域特有问题（如密集文本处理、细粒度定位、长文档逻辑推理）的迁移路径。整体属于‘平行应用’类技术包装，不具备直接推动文档理解底层能力突破的方法论创新。
Abstract RL (reinforcement learning) methods (e.g., GRPO) for MLLM (Multimodal LLM) perception ability has attracted wide research interest owing to its remarkable generalization ability. Nevertheless, existing reinforcement learning methods still face the problem of low data quality, where data samples cannot elicit diverse responses from MLLMs, thus restricting the exploration scope for MLLM reinforcement learning. Some methods attempt to mitigate this problem by imposing constraints on entropy, but none address it at its root. Therefore, to tackle this problem, this work proposes Syn-GRPO (Synthesis-GRPO), which employs an online data generator to synthesize high-quality training data with diverse responses in GRPO training. Specifically, Syn-GRPO consists of two components: (1) data server; (2) GRPO workflow. The data server synthesizes new samples from existing ones using an image generation model, featuring a decoupled and asynchronous scheme to achieve high generation efficiency. The GRPO workflow provides the data server with the new image descriptions, and it leverages a diversity reward to supervise the MLLM to predict image descriptions for synthesizing samples with diverse responses. Experiment results across three visual perception tasks demonstrate that Syn-GRPO improves the data quality by a large margin, achieving significant superior performance to existing MLLM perception methods, and Syn-GRPO presents promising potential for scaling long-term self-evolving RL. Our code is available at https://github.com/hqhQAQ/Syn-GRPO.
Authors Qihan Huang, Haofei Zhang, Rong Wei, Yi Wang, Rui Tang, Mingli Song, Jie Song
Categories Computer Vision and Pattern Recognition
3
InfoScale: Unleashing Training-free Variable-scaled Image Generation via Effective Utilization of Information
Title CN InfoScale：通过有效利用信息实现无需训练的可变尺度图像生成
Keywords 扩散模型 · 可变分辨率生成 · 图像生成 · 视觉信息利用 · 无需训练
Summary 本文提出InfoScale框架，旨在解决扩散模型在不同分辨率图像生成时性能下降的问题。通过分析膨胀卷积、注意力机制和初始噪声三个方面的信息利用问题，设计了频率补偿、自适应信息聚合和噪声适配模块，以提升可变尺度图像生成质量。该方法为通用图像生成服务，不针对文档内容理解。
Reason 该论文聚焦于扩散模型在不同分辨率下的图像生成问题，属于纯图像生成领域，主要解决高/低分辨率生成中的信息丢失与对齐问题。尽管其涉及高分辨率处理，但目标是通用图像生成而非文档理解任务（如DocVQA、表格识别等）。方法虽对视觉表示有改进，但无法直接迁移用于解决文档图像中的OCR幻觉、复杂排版或细粒度文本定位等核心痛点，属于无关领域应用。
Abstract Diffusion models (DMs) have become dominant in visual generation but suffer performance drop when tested on resolutions that differ from the training scale, whether lower or higher. In fact, the key challenge in generating variable-scale images lies in the differing amounts of information across resolutions, which requires information conversion procedures to be varied for generating variable-scaled images. In this paper, we investigate the issues of three critical aspects in DMs for a unified analysis in variable-scaled generation: dilated convolution, attention mechanisms, and initial noise. Specifically, 1) dilated convolution in DMs for the higher-resolution generation loses high-frequency information. 2) Attention for variable-scaled image generation struggles to adjust the information aggregation adaptively. 3) The spatial distribution of information in the initial noise is misaligned with variable-scaled image. To solve the above problems, we propose \textbf{InfoScale}, an information-centric framework for variable-scaled image generation by effectively utilizing information from three aspects correspondingly. For information loss in 1), we introduce Progressive Frequency Compensation module to compensate for high-frequency information lost by dilated convolution in higher-resolution generation. For information aggregation inflexibility in 2), we introduce Adaptive Information Aggregation module to adaptively aggregate information in lower-resolution generation and achieve an effective balance between local and global information in higher-resolution generation. For information distribution misalignment in 3), we design Noise Adaptation module to re-distribute information in initial noise for variable-scaled generation. Our method is plug-and-play for DMs and extensive experiments demonstrate the effectiveness in variable-scaled image generation.
Authors Guohui Zhang, Jiangtong Tan, Linjiang Huang, Zhonghang Yuan, Mingde Yao, Jie Huang, Feng Zhao
Categories Computer Vision and Pattern Recognition
3
Diffusion Reconstruction-based Data Likelihood Estimation for Core-Set Selection AAAI 2026
Title CN 基于扩散重建的数据似然估计用于核心集选择
Keywords 核心集选择 · 扩散模型 · 数据似然 · 重建误差 · 训练动态
Summary 本文提出一种基于扩散模型重建偏差来估计数据似然性的核心集选择方法，通过理论推导建立重建误差与数据似然之间的联系，并利用ELBO框架实现分布感知的数据评分。实验表明该方法在ImageNet上优于现有启发式策略，仅用50%数据即可接近全量训练性能。
Reason 该论文研究的是通用图像数据上的核心集（core-set）选择问题，使用扩散模型通过重建偏差来估计数据似然性，从而指导数据子集的选取。尽管方法在理论上具有创新性，但其应用场景为通用视觉（ImageNet），并未针对文档图像理解（DIU）中的具体任务如DocVQA、布局分析或表格识别等进行设计或验证。此外，该工作属于数据选择与训练优化范畴，虽有潜在上游价值，但缺乏对DIU核心痛点（如OCR幻觉、复杂排版、细粒度定位）的直接建模或可迁移机制说明，且未涉及多模态大模型（VLM）架构、推理增强或视觉token压缩等DIU关键方向。因此属于‘平行应用’类方法论延伸，不符合本领域高优先级关注标准。
Abstract Existing core-set selection methods predominantly rely on heuristic scoring signals such as training dynamics or model uncertainty, lacking explicit modeling of data likelihood. This omission may hinder the constructed subset from capturing subtle yet critical distributional structures that underpin effective model training. In this work, we propose a novel, theoretically grounded approach that leverages diffusion models to estimate data likelihood via reconstruction deviation induced by partial reverse denoising. Specifically, we establish a formal connection between reconstruction error and data likelihood, grounded in the Evidence Lower Bound (ELBO) of Markovian diffusion processes, thereby enabling a principled, distribution-aware scoring criterion for data selection. Complementarily, we introduce an efficient information-theoretic method to identify the optimal reconstruction timestep, ensuring that the deviation provides a reliable signal indicative of underlying data likelihood. Extensive experiments on ImageNet demonstrate that reconstruction deviation offers an effective scoring criterion, consistently outperforming existing baselines across selection ratios, and closely matching full-data training using only 50% of the data. Further analysis shows that the likelihood-informed nature of our score reveals informative insights in data selection, shedding light on the interplay between data distributional characteristics and model learning preferences.
Authors Mingyang Chen, Jiawei Du, Bo Huang, Yi Wang, Xiaobo Zhang, Wei Wang
Comment Accepted by AAAI 2026
Categories Computer Vision and Pattern Recognition
3
Zero-Shot Coreset Selection via Iterative Subspace Sampling WACV 2026
Title CN 通过迭代子空间采样的零样本核心集选择
Keywords 核心集选择 · 零样本学习 · 数据高效训练 · 子空间采样 · 视觉表示
Summary 本文提出ZCore，一种无需标签和训练即可从大规模无标签数据中选择核心集的方法。它利用预训练基础模型生成高维嵌入空间，并通过迭代子空间采样来衡量数据的覆盖性和冗余性，从而选出最具代表性的子集。在ImageNet等数据集上验证了其有效性，尤其在低数据率下表现优异。
Reason 该论文研究的是通用图像数据上的零样本核心集选择，目标是从大规模无标签数据中选取代表性子集以降低训练成本。其方法基于视觉embedding和子空间分布进行数据筛选，属于数据高效学习（data-efficient learning）的范畴。然而，该工作并未针对文档图像理解（DIU）中的任何具体任务（如DocVQA、表格识别、布局分析等）进行设计或验证，也未涉及OCR幻觉、细粒度定位、复杂排版建模等DIU核心痛点。虽然理论上核心集选择可用于构建高效文档数据集，但本文属于通用计算机视觉中的数据采样方法，且不涉及多模态大模型（VLM）或文档特有的结构与语义建模，迁移价值有限。
Abstract Deep learning increasingly relies on massive data with substantial storage, annotation, and training costs. To reduce costs, coreset selection finds a representative subset of data to train models while ideally performing on par with the full data training. To maximize performance, current state-of-the-art coreset methods select data using dataset-specific ground truth labels and training. However, these methodological requirements prevent selection at scale on real-world, unlabeled data. To that end, this paper addresses the selection of coresets that achieve state-of-the-art performance but without using any labels or training on candidate data. Instead, our solution, Zero-Shot Coreset Selection via Iterative Subspace Sampling (ZCore), uses previously-trained foundation models to generate zero-shot, high-dimensional embedding spaces to interpret unlabeled data. ZCore then iteratively quantifies the relative value of all candidate data based on coverage and redundancy in numerous subspace distributions. Finally, ZCore selects a coreset sized for any data budget to train downstream models. We evaluate ZCore on four datasets and outperform several state-of-the-art label-based methods, especially at low data rates that provide the most substantial cost reduction. On ImageNet, ZCore selections for 10% training data achieve a downstream validation accuracy of 53.99%, which outperforms prior label-based methods and removes annotation and training costs for 1.15 million images. Our paper's code is publicly available at https://github.com/voxel51/zcore.
Authors Brent A. Griffin, Jacob Marks, Jason J. Corso
Comment WACV 2026
Categories Computer Vision and Pattern Recognition
3
MedBridge: Bridging Foundation Vision-Language Models to Medical Image Diagnosis in Chest X-Ray
Title CN MedBridge：将基础视觉语言模型桥接到胸部X光医学图像诊断
Keywords 视觉语言模型 · 领域自适应 · 医学图像诊断 · 轻量级适配 · 专家混合
Summary MedBridge提出了一种轻量级多模态适配框架，用于将预训练的视觉语言模型迁移到胸部X光诊断任务中。它通过焦点采样提取高分辨率局部区域，使用可学习查询对齐特征，并采用专家混合机制融合多个VLM的优势。在五个胸片基准上验证了其有效性，显著提升了多标签疾病诊断的AUC指标。
Reason 该论文属于典型的‘平行下游应用’，即将现有VLM技术通过适配模块应用于特定领域（医疗影像）的诊断任务。尽管提出了Focal Sampling、Query-Encoder和MoE等组件，但其核心创新集中在医疗领域的跨域适应，而非解决文档图像理解中的OCR幻觉、复杂排版或长文档推理等共性问题。方法难以迁移到DIU场景，属于领域专用的wrapper式工作。
Abstract Recent vision-language foundation models deliver state-of-the-art results in natural image classification, but falter in medical images due to pronounced domain shifts. Training a medical foundation model also requires substantial resources, including extensive annotated data and high computational capacity. To bridge this gap with minimal overhead, we introduce MedBridge, a lightweight multimodal adaptation framework that flexibly re-purposes arbitrary pre-trained foundation VLMs for medical image diagnosis. MedBridge comprises three novel core components. First, a Focal Sampling module that subsamples and extracts high-resolution local regions to capture subtle pathological features, compensating for the limited input resolution of foundation VLMs. Second, a Query-Encoder model with a small set of learnable queries to align the feature maps of frozen VLMs with medical semantics, without requiring retraining of the backbone layers. Third, a Mixture of Experts mechanism, driven by learnable queries, harnesses the complementary strength of various VLMs to maximize diagnostic performance. We evaluate MedBridge on five chest radiograph benchmarks in three key adaptation tasks, demonstrating its superior performance in both cross-domain and in-domain adaptation settings under varying levels of training data availability. MedBridge achieved an improvement of 6-15% in AUC compared to state-of-the-art VLM adaptation methods in multi-label thoracic disease diagnosis, underscoring its effectiveness in leveraging diverse foundation models for accurate and data-efficient medical diagnosis. Our project and code are available at https://github.com/ai-med/MedBridge.
Authors Yitong Li, Morteza Ghahremani, Christian Wachinger
Categories Computer Vision and Pattern Recognition
3
Test-Time Preference Optimization for Image Restoration AAAI26
Title CN 用于图像修复的测试时偏好优化
Keywords 图像修复 · 测试时优化 · 偏好对齐 · 扩散模型 · 无训练
Summary 本文提出了一种名为测试时偏好优化（TTPO）的新范式，用于在不重新训练模型的前提下提升图像修复的感知质量。该方法通过扩散反演生成候选图像，利用自动化指标或人工反馈选择偏好样本，并以此作为奖励信号优化去噪过程。实验表明该流程可适配多种图像修复模型并提升主观质量。
Reason 该论文研究的是通用图像修复任务中的感知质量提升问题，核心目标是使修复结果更符合人类视觉偏好。尽管其提出的TTPO范式在测试阶段无需训练即可优化输出，具有一定的方法论新颖性，但其应用场景与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂版面解析、细粒度文本定位、长文档推理等）无直接关联。文档图像强调语义结构和符号准确性，而非视觉感知美感；因此，该方法难以迁移用于解决DIU中的关键挑战，属于无关领域的应用工作。
Abstract Image restoration (IR) models are typically trained to recover high-quality images using L1 or LPIPS loss. To handle diverse unknown degradations, zero-shot IR methods have also been introduced. However, existing pre-trained and zero-shot IR approaches often fail to align with human preferences, resulting in restored images that may not be favored. This highlights the critical need to enhance restoration quality and adapt flexibly to various image restoration tasks or backbones without requiring model retraining and ideally without labor-intensive preference data collection. In this paper, we propose the first Test-Time Preference Optimization (TTPO) paradigm for image restoration, which enhances perceptual quality, generates preference data on-the-fly, and is compatible with any IR model backbone. Specifically, we design a training-free, three-stage pipeline: (i) generate candidate preference images online using diffusion inversion and denoising based on the initially restored image; (ii) select preferred and dispreferred images using automated preference-aligned metrics or human feedback; and (iii) use the selected preference images as reward signals to guide the diffusion denoising process, optimizing the restored image to better align with human preferences. Extensive experiments across various image restoration tasks and models demonstrate the effectiveness and flexibility of the proposed pipeline.
Authors Bingchen Li, Xin Li, Jiaqi Xu, Jiaming Guo, Wenbo Li, Renjing Pei, Zhibo Chen
Comment Accepted by AAAI26
Categories Computer Vision and Pattern Recognition
3
Collaborative Learning with Multiple Foundation Models for Source-Free Domain Adaptation
Title CN 利用多个基础模型进行无源域适应的协同学习
Keywords 域适应 · 基础模型 · 无源域适应 · 多模态学习 · 语义对齐
Summary 本文提出CoMA框架，通过结合多个基础模型（如CLIP和BLIP）的互补语义信息，在不访问源数据的情况下实现无源域适应。采用双向适应机制和分解互信息（DMI）来稳定训练并增强真实依赖关系。实验表明在多个标准视觉识别任务上优于现有方法。
Reason 该论文研究的是无源域适应（SFDA），属于通用计算机视觉中的模型自适应问题，其方法基于CLIP和BLIP等多模态基础模型进行知识协同，提升目标域上的泛化能力。虽然涉及多模态模型，但任务本身与文档图像理解（DIU）无直接关联，未针对OCR幻觉、复杂版面解析、细粒度文本定位或长文档推理等DIU核心痛点设计。且其应用场景为通用图像分类基准（如Office-31、DomainNet），并非文档图像。属于将现有VLM用于模型适应的‘平行应用’类工作，缺乏对DIU的方法论迁移价值。
Abstract Source-Free Domain Adaptation (SFDA) aims to adapt a pre-trained source model to an unlabeled target domain without access to source data. Recent advances in Foundation Models (FMs) have introduced new opportunities for leveraging external semantic knowledge to guide SFDA. However, relying on a single FM is often insufficient, as it tends to bias adaptation toward a restricted semantic coverage, failing to capture diverse contextual cues under domain shift. To overcome this limitation, we propose a Collaborative Multi-foundation Adaptation (CoMA) framework that jointly leverages two different FMs (e.g., CLIP and BLIP) with complementary properties to capture both global semantics and local contextual cues. Specifically, we employ a bidirectional adaptation mechanism that (1) aligns different FMs with the target model for task adaptation while maintaining their semantic distinctiveness, and (2) transfers complementary knowledge from the FMs to the target model. To ensure stable adaptation under mini-batch training, we introduce Decomposed Mutual Information (DMI) that selectively enhances true dependencies while suppressing false dependencies arising from incomplete class coverage. Extensive experiments demonstrate that our method consistently outperforms existing state-of-the-art SFDA methods across four benchmarks, including Office-31, Office-Home, DomainNet-126, and VisDA, under the closed-set setting, while also achieving best results on partial-set and open-set variants.
Authors Huisoo Lee, Jisu Han, Hyunsouk Cho, Wonjun Hwang
Comment 15 pages, 8 figures
Categories Computer Vision and Pattern Recognition
3
HiGFA: Hierarchical Guidance for Fine-grained Data Augmentation with Diffusion Models
Title CN HiGFA：基于扩散模型的分层引导细粒度数据增强
Keywords 扩散模型 · 数据增强 · 细粒度分类 · 分层引导 · 生成模型
Summary 本文提出HiGFA，一种用于细粒度视觉分类的数据增强方法，通过在扩散模型采样过程中引入分层引导机制——前期用文本和轮廓信息构建整体结构，后期动态激活细粒度分类器引导以精修细节——从而生成既多样又保真的图像。实验表明其在多个细粒度分类数据集上有效。
Reason 该论文聚焦于细粒度视觉分类（FGVC）任务中的图像生成式数据增强，使用扩散模型结合分层引导策略来提升合成图像的细节保真度。尽管其方法在细粒度图像生成上有一定创新，但核心目标并非文档理解或VLM相关任务。未涉及OCR、布局分析、表格识别、DocVQA等DIU核心问题，也未提出可迁移至文档图像理解领域的通用底层技术（如视觉token压缩、高分辨率处理、多模态对齐等）。属于特定CV任务下的数据增强方法，对DIU领域无直接方法论贡献。
Abstract Generative diffusion models show promise for data augmentation. However, applying them to fine-grained tasks presents a significant challenge: ensuring synthetic images accurately capture the subtle, category-defining features critical for high fidelity. Standard approaches, such as text-based Classifier-Free Guidance (CFG), often lack the required specificity, potentially generating misleading examples that degrade fine-grained classifier performance. To address this, we propose Hierarchically Guided Fine-grained Augmentation (HiGFA). HiGFA leverages the temporal dynamics of the diffusion sampling process. It employs strong text and transformed contour guidance with fixed strengths in the early-to-mid sampling stages to establish overall scene, style, and structure. In the final sampling stages, HiGFA activates a specialized fine-grained classifier guidance and dynamically modulates the strength of all guidance signals based on prediction confidence. This hierarchical, confidence-driven orchestration enables HiGFA to generate diverse yet faithful synthetic images by intelligently balancing global structure formation with precise detail refinement. Experiments on several FGVC datasets demonstrate the effectiveness of HiGFA.
Authors Zhiguang Lu, Qianqian Xu, Peisong Wen, Siran Dai, Qingming Huang
Categories Computer Vision and Pattern Recognition
3
Compressor-VLA: Instruction-Guided Visual Token Compression for Efficient Robotic Manipulation
Title CN Compressor-VLA：面向高效机器人操作的指令引导视觉令牌压缩
Keywords 视觉令牌压缩 · 具身智能 · 机器人操作 · 视觉-语言-动作模型 · 指令引导
Summary 本文提出Compressor-VLA，一种用于机器人操作的指令引导视觉令牌压缩框架，包含语义任务压缩器和空间细化压缩器，通过自然语言指令动态调节视觉信息压缩，在保持任务性能的同时显著降低计算开销，并在仿真和真实机器人平台上验证了有效性。
Reason 该论文属于具身智能与机器人领域，研究的是视觉-语言-动作（VLA）模型在机器人操纵任务中的应用。尽管其提出了指令引导的视觉令牌压缩机制，具有一定的方法创新性，但其核心目标是服务于机器人控制，而非文档图像理解（DIU）中的文本识别、布局分析或DocVQA等任务。虽然‘视觉令牌压缩’这一技术概念与DIU中如DeepSeek-OCR的视觉压缩路线有一定形式上的相似性，但其压缩策略针对的是三维空间中的物体操作语义，迁移至文档场景（如密集文本、表格结构）的适配性和有效性极低，且未涉及OCR幻觉、细粒度定位或长文档推理等DIU核心痛点。因此属于无关领域的平行应用，不具备对DIU的方法论迁移价值。
Abstract Vision-Language-Action (VLA) models have emerged as a powerful paradigm in Embodied AI. However, the significant computational overhead of processing redundant visual tokens remains a critical bottleneck for real-time robotic deployment. While standard token pruning techniques can alleviate this, these task-agnostic methods struggle to preserve task-critical visual information. To address this challenge, simultaneously preserving both the holistic context and fine-grained details for precise action, we propose Compressor-VLA, a novel hybrid instruction-conditioned token compression framework designed for efficient, task-oriented compression of visual information in VLA models. The proposed Compressor-VLA framework consists of two token compression modules: a Semantic Task Compressor (STC) that distills holistic, task-relevant context, and a Spatial Refinement Compressor (SRC) that preserves fine-grained spatial details. This compression is dynamically modulated by the natural language instruction, allowing for the adaptive condensation of task-relevant visual information. Experimentally, extensive evaluations demonstrate that Compressor-VLA achieves a competitive success rate on the LIBERO benchmark while reducing FLOPs by 59% and the visual token count by over 3x compared to its baseline. The real-robot deployments on a dual-arm robot platform validate the model's sim-to-real transferability and practical applicability. Moreover, qualitative analyses reveal that our instruction guidance dynamically steers the model's perceptual focus toward task-relevant objects, thereby validating the effectiveness of our approach.
Authors Juntao Gao, Feiyang Ye, Jing Zhang, Wenjing Qian
Comment 11 pages, 5 figures
Categories Computer Vision and Pattern Recognition
3
Sim-DETR: Unlock DETR for Temporal Sentence Grounding ICCV 2025
Title CN Sim-DETR：解锁DETR用于时序句子定位
Keywords Temporal Sentence Grounding · DETR · Video Understanding · Query Conflict · Self-Attention
Summary 本文提出Sim-DETR，通过在DETR解码器中引入基于语义和位置重叠的查询自注意力约束以及查询到帧的对齐机制，解决视频中时序句子 grounding 任务上DETR性能不升反降的问题。实验表明该方法能有效提升定位精度。
Reason 该论文研究的是视频领域的时序句子定位（Temporal Sentence Grounding），属于视频理解任务，其核心问题与文档图像理解（DIU）差异显著。尽管使用了DETR架构并改进了查询机制，但其方法针对的是时间序列上的语义对齐，而非文档中的空间布局、密集文本或OCR相关问题。无法有效迁移至DocVQA、表格识别或复杂排版分析等DIU核心任务。
Abstract Temporal sentence grounding aims to identify exact moments in a video that correspond to a given textual query, typically addressed with detection transformer (DETR) solutions. However, we find that typical strategies designed to enhance DETR do not improve, and may even degrade, its performance in this task. We systematically analyze and identify the root causes of this abnormal behavior: (1) conflicts between queries from similar target moments and (2) internal query conflicts due to the tension between global semantics and local localization. Building on these insights, we propose a simple yet powerful baseline, Sim-DETR, which extends the standard DETR with two minor modifications in the decoder layers: (1) constraining self-attention between queries based on their semantic and positional overlap and (2) adding query-to-frame alignment to bridge the global and local contexts. Experiments demonstrate that Sim-DETR unlocks the full potential of DETR for temporal sentence grounding, offering a strong baseline for future research.
Authors Jiajin Tang, Zhengxuan Wei, Yuchen Zhu, Cheng Shi, Guanbin Li, Liang Lin, Sibei Yang
Comment This work is accepted by ICCV 2025
Categories Computer Vision and Pattern Recognition
3
Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabeling AAAI 2026 (Oral)
Title CN 重新思考长尾数据集蒸馏：一种具有无偏恢复与重标注的统一层级框架
Keywords 数据集蒸馏 · 长尾分布 · 统计对齐 · BN校准 · 软标签
Summary 本文提出了一种面向长尾分布的数据集蒸馏新框架，通过增强专家模型、重校准BN统计量以及多轮机制初始化合成图像，实现对长尾数据中少数类别的无偏恢复与软标签重标注。实验表明该方法在多个长尾基准上显著优于现有技术。
Reason 该论文研究的是长尾分布下的数据集蒸馏问题，核心贡献在于提升类别不平衡场景中蒸馏数据的质量和模型训练的公平性。尽管其方法涉及视觉表示学习和统计对齐，但完全聚焦于通用图像分类任务（如CIFAR、Tiny-ImageNet），并未涉及文档结构、布局理解、多模态对齐或OCR相关建模。无法直接迁移解决文档图像理解中的OCR幻觉、复杂排版或细粒度定位等核心痛点，属于通用CV领域的方法改进，且非针对文档或多模态语言模型设计。
Abstract Dataset distillation creates a small distilled set that enables efficient training by capturing key information from the full dataset. While existing dataset distillation methods perform well on balanced datasets, they struggle under long-tailed distributions, where imbalanced class frequencies induce biased model representations and corrupt statistical estimates such as Batch Normalization (BN) statistics. In this paper, we rethink long-tailed dataset distillation by revisiting the limitations of trajectory-based methods, and instead adopt the statistical alignment perspective to jointly mitigate model bias and restore fair supervision. To this end, we introduce three dedicated components that enable unbiased recovery of distilled images and soft relabeling: (1) enhancing expert models (an observer model for recovery and a teacher model for relabeling) to enable reliable statistics estimation and soft-label generation; (2) recalibrating BN statistics via a full forward pass with dynamically adjusted momentum to reduce representation skew; (3) initializing synthetic images by incrementally selecting high-confidence and diverse augmentations via a multi-round mechanism that promotes coverage and diversity. Extensive experiments on four long-tailed benchmarks show consistent improvements over state-of-the-art methods across varying degrees of class imbalance.Notably, our approach improves top-1 accuracy by 15.6% on CIFAR-100-LT and 11.8% on Tiny-ImageNet-LT under IPC=10 and IF=10.
Authors Xiao Cui, Yulei Qin, Xinyue Li, Wengang Zhou, Hongsheng Li, Houqiang Li
Comment AAAI 2026 (Oral)
Categories Computer Vision and Pattern Recognition
3
2D Gaussians Spatial Transport for Point-supervised Density Regression AAAI, 2026
Title CN 用于点监督密度回归的二维高斯空间传输
Keywords Gaussian Splatting · Optimal Transport · Point Supervision · Density Regression · Crowd Counting
Summary 本文提出了高斯空间传输（GST）框架，利用高斯splatting估计像素与标注之间的对应关系，并通过贝叶斯概率推导出传输计划，用于优化网络训练。相比传统最优传输方法，GST避免了迭代计算，提升了效率。实验验证于人群计数和关键点检测任务。
Reason 该论文提出了一种基于高斯splatting的空间传输方法（GST），用于从图像坐标空间到标注图的概率测度传输，主要应用于人群计数和关键点检测等任务。尽管其在优化标注对应关系和传输计划方面有一定创新，但其核心应用场景与文档图像理解（DIU）中的核心问题如文本识别、布局分析、表格理解或VQA无关。未见其方法可直接迁移用于解决OCR幻觉、复杂排版建模、细粒度文本定位或长文档推理等DIU关键痛点。属于通用视觉中的特定监督范式改进，不具备对文档理解任务的直接方法论迁移价值。
Abstract This paper introduces Gaussian Spatial Transport (GST), a novel framework that leverages Gaussian splatting to facilitate transport from the probability measure in the image coordinate space to the annotation map. We propose a Gaussian splatting-based method to estimate pixel-annotation correspondence, which is then used to compute a transport plan derived from Bayesian probability. To integrate the resulting transport plan into standard network optimization in typical computer vision tasks, we derive a loss function that measures discrepancy after transport. Extensive experiments on representative computer vision tasks, including crowd counting and landmark detection, validate the effectiveness of our approach. Compared to conventional optimal transport schemes, GST eliminates iterative transport plan computation during training, significantly improving efficiency. Code is available at https://github.com/infinite0522/GST.
Authors Miao Shang, Xiaopeng Hong
Comment 15 pages, 6 figures. This is the preprint version of the paper and supplemental material to appear in AAAI, 2026. Please cite the final published version. Code is available at https://github.com/infinite0522/GST
Categories Computer Vision and Pattern Recognition
3
Uncertainty-Aware Dual-Student Knowledge Distillation for Efficient Image Classification
Title CN 面向高效图像分类的不确定性感知双学生知识蒸馏
Keywords 知识蒸馏 · 模型压缩 · 不确定性估计 · 协同学习 · 图像分类
Summary 本文提出一种不确定性感知的双学生知识蒸馏框架，通过引入教师模型预测的不确定性来加权指导两个异构学生模型（ResNet-18和MobileNetV2）的学习，并结合同伴学习机制提升分类性能。在ImageNet-100上的实验表明该方法优于传统单学生蒸馏方法。
Reason 该论文研究的是通用图像分类任务中的模型压缩问题，采用知识蒸馏和学生网络协同学习机制，核心关注点为分类精度与模型效率的平衡。其方法未涉及文本理解、布局分析、视觉-语言对齐或多模态推理等文档图像理解（DIU）的关键挑战，且应用场景完全基于ImageNet类自然图像，与文档图像无关。属于典型的非相关领域工作，无法迁移解决OCR幻觉、复杂排版或长文档推理等问题。
Abstract Knowledge distillation has emerged as a powerful technique for model compression, enabling the transfer of knowledge from large teacher networks to compact student models. However, traditional knowledge distillation methods treat all teacher predictions equally, regardless of the teacher's confidence in those predictions. This paper proposes an uncertainty-aware dual-student knowledge distillation framework that leverages teacher prediction uncertainty to selectively guide student learning. We introduce a peer-learning mechanism where two heterogeneous student architectures, specifically ResNet-18 and MobileNetV2, learn collaboratively from both the teacher network and each other. Experimental results on ImageNet-100 demonstrate that our approach achieves superior performance compared to baseline knowledge distillation methods, with ResNet-18 achieving 83.84\% top-1 accuracy and MobileNetV2 achieving 81.46\% top-1 accuracy, representing improvements of 2.04\% and 0.92\% respectively over traditional single-student distillation approaches.
Authors Aakash Gore, Anoushka Dey, Aryan Mishra
Categories Computer Vision and Pattern Recognition
3
SupLID: Geometrical Guidance for Out-of-Distribution Detection in Semantic Segmentation CIKM 2025
Title CN SupLID：用于语义分割中分布外检测的几何引导
Keywords 分布外检测 · 语义分割 · 几何引导 · LID · 超像素
Summary 本文提出SupLID，一种基于几何结构引导的分布外检测框架，用于语义分割任务中定位异常像素区域。该方法利用线性本征维数（LID）分析语义空间的局部结构，并通过构建几何核心集（coreset）在超像素级别计算OOD得分，提升检测的效率与空间平滑性。作为后处理模块，可与任意分割模型结合，在AUR、FPR等指标上达到SOTA性能。
Reason 该论文研究的是语义分割中的分布外检测（OOD）问题，属于通用计算机视觉领域，主要面向自动驾驶等场景的像素级异常检测。尽管其方法利用了几何结构（如LID）和超像素进行打分优化，但核心任务与文档图像理解（DIU）无直接关联，未涉及OCR、文本理解、布局分析或VLM在文档上的应用。无法迁移解决DIU中的OCR幻觉、复杂排版或长文档推理等核心痛点，属于无关领域工作。
Abstract Out-of-Distribution (OOD) detection in semantic segmentation aims to localize anomalous regions at the pixel level, advancing beyond traditional image-level OOD techniques to better suit real-world applications such as autonomous driving. Recent literature has successfully explored the adaptation of commonly used image-level OOD methods--primarily based on classifier-derived confidence scores (e.g., energy or entropy)--for this pixel-precise task. However, these methods inherit a set of limitations, including vulnerability to overconfidence. In this work, we introduce SupLID, a novel framework that effectively guides classifier-derived OOD scores by exploiting the geometrical structure of the underlying semantic space, particularly using Linear Intrinsic Dimensionality (LID). While LID effectively characterizes the local structure of high-dimensional data by analyzing distance distributions, its direct application at the pixel level remains challenging. To overcome this, SupLID constructs a geometrical coreset that captures the intrinsic structure of the in-distribution (ID) subspace. It then computes OOD scores at the superpixel level, enabling both efficient real-time inference and improved spatial smoothness. We demonstrate that geometrical cues derived from SupLID serve as a complementary signal to traditional classifier confidence, enhancing the model's ability to detect diverse OOD scenarios. Designed as a post-hoc scoring method, SupLID can be seamlessly integrated with any semantic segmentation classifier at deployment time. Our results demonstrate that SupLID significantly enhances existing classifier-based OOD scores, achieving state-of-the-art performance across key evaluation metrics, including AUR, FPR, and AUP. Code is available at https://github.com/hdnugit/SupLID.
Authors Nimeshika Udayangani, Sarah Erfani, Christopher Leckie
Comment 10 pages, CIKM 2025
Categories Computer Vision and Pattern Recognition
3
Seeing What Matters: Visual Preference Policy Optimization for Visual Generation
Title CN 看见关键内容：面向视觉生成的视觉偏好策略优化
Keywords 强化学习 · 视觉生成 · 偏好优化 · 像素级优势 · 感知结构
Summary 本文提出了Visual Preference Policy Optimization（ViPO），一种改进的Group Relative Policy Optimization方法，通过引入基于预训练视觉骨干网络的感知结构模块，将标量奖励转化为空间和时间感知的像素级优势图，从而在图像和视频生成中实现更精细的偏好对齐。该方法在多个基准上优于传统GRPO，且兼容现有训练流程。
Reason 该论文提出了一种用于视觉生成模型后训练的强化学习方法ViPO，通过像素级优势映射提升图像和视频生成的质量。然而，其核心任务是视觉生成（如图像/视频合成），属于纯图像生成领域，与文档图像理解（DIU）中的信息提取、结构化识别、DocVQA等任务无直接关联。尽管其‘细粒度感知’思想看似有启发性，但该方法针对的是生成质量的感知优化，而非解决OCR幻觉、复杂排版解析或长文档推理等DIU核心痛点，迁移价值极低。
Abstract Reinforcement learning (RL) has become a powerful tool for post-training visual generative models, with Group Relative Policy Optimization (GRPO) increasingly used to align generators with human preferences. However, existing GRPO pipelines rely on a single scalar reward per sample, treating each image or video as a holistic entity and ignoring the rich spatial and temporal structure of visual content. This coarse supervision hinders the correction of localized artifacts and the modeling of fine-grained perceptual cues. We introduce Visual Preference Policy Optimization (ViPO), a GRPO variant that lifts scalar feedback into structured, pixel-level advantages. ViPO employs a Perceptual Structuring Module that uses pretrained vision backbones to construct spatially and temporally aware advantage maps, redistributing optimization pressure toward perceptually important regions while preserving the stability of standard GRPO. Across both image and video benchmarks, ViPO consistently outperforms vanilla GRPO, improving in-domain alignment with human-preference rewards and enhancing generalization on out-of-domain evaluations. The method is architecture-agnostic, lightweight, and fully compatible with existing GRPO training pipelines, providing a more expressive and informative learning signal for visual generation.
Authors Ziqi Ni, Yuanzhi Liang, Rui Li, Yi Zhou, Haibing Huang, Chi Zhang, Xuelong Li
Categories Computer Vision and Pattern Recognition
3
Dendritic Convolution for Noise Image Recognition
Title CN 用于噪声图像识别的树突卷积
Keywords 树突卷积 · 噪声鲁棒性 · 生物启发计算 · 特征提取 · 图像分类
Summary 本文提出了一种受生物神经元树突结构启发的新型卷积——树突卷积（Dendritic Convolution），通过模拟树突间的非线性交互逻辑（如XOR预处理）来增强模型对噪声的抵抗能力。实验表明，在多种主流模型（如EfficientNet-B0、YOLO系列）上替换传统卷积后，其在噪声数据集上的分类准确率和检测mAP均有显著提升。
Reason 该论文提出一种模仿神经元树突结构的卷积方法，旨在提升通用图像识别在噪声环境下的鲁棒性。尽管其方法在生物启发计算上有一定新颖性，但研究目标是通用图像分类与目标检测中的噪声抑制，未涉及文档图像特有的挑战（如文本布局、OCR幻觉、细粒度文字定位等）。其技术路径难以直接迁移至文档图像理解任务，且属于通用CV领域改进，对DIU无实质推动。
Abstract In real-world scenarios of image recognition, there exists substantial noise interference. Existing works primarily focus on methods such as adjusting networks or training strategies to address noisy image recognition, and the anti-noise performance has reached a bottleneck. However, little is known about the exploration of anti-interference solutions from a neuronal perspective.This paper proposes an anti-noise neuronal convolution. This convolution mimics the dendritic structure of neurons, integrates the neighborhood interaction computation logic of dendrites into the underlying design of convolutional operations, and simulates the XOR logic preprocessing function of biological dendrites through nonlinear interactions between input features, thereby fundamentally reconstructing the mathematical paradigm of feature extraction. Unlike traditional convolution where noise directly interferes with feature extraction and exerts a significant impact, DDC mitigates the influence of noise by focusing on the interaction of neighborhood information. Experimental results demonstrate that in image classification tasks (using YOLOv11-cls, VGG16, and EfficientNet-B0) and object detection tasks (using YOLOv11, YOLOv8, and YOLOv5), after replacing traditional convolution with the dendritic convolution, the accuracy of the EfficientNet-B0 model on noisy datasets is relatively improved by 11.23%, and the mean Average Precision (mAP) of YOLOv8 is increased by 19.80%. The consistency between the computation method of this convolution and the dendrites of biological neurons enables it to perform significantly better than traditional convolution in complex noisy environments.
Authors Jiarui Xue, Dongjian Yang, Ye Sun, Gang Liu
Comment 11 pages, 8 figures
Categories Computer Vision and Pattern Recognition
3
EVCC: Enhanced Vision Transformer-ConvNeXt-CoAtNet Fusion for Classification
Title CN EVCC：用于分类的增强型Vision Transformer-ConvNeXt-CoAtNet融合
Keywords 图像分类 · 混合架构 · Vision Transformer · ConvNeXt · token剪枝
Summary 本文提出了一种结合Vision Transformer、ConvNeXt和CoAtNet的多分支混合架构EVCC，通过自适应token剪枝、门控双向交叉注意力、辅助分类头和动态路由机制，在降低计算量的同时提升了图像分类精度。实验表明其在多个数据集上优于主流模型。然而，该工作聚焦于通用视觉分类，未涉及文档理解场景。
Reason 该论文属于通用图像分类任务，尽管提出了多分支架构和自适应token剪枝等技术，但其核心目标是提升自然图像分类性能，并未针对文档图像理解（DIU）中的关键问题（如OCR幻觉、复杂版式、细粒度文本定位或长文档推理）进行设计。所用数据集（如CIFAR-100、CelebA）与文档无关，方法也未体现对高分辨率文档、文本布局结构或视觉-语言对齐的支持。虽然token压缩思想看似相关，但其上下文感知的动态路由机制并未面向文档特有的序列建模或VLM集成需求，迁移价值有限。
Abstract Hybrid vision architectures combining Transformers and CNNs have significantly advanced image classification, but they usually do so at significant computational cost. We introduce EVCC (Enhanced Vision Transformer-ConvNeXt-CoAtNet), a novel multi-branch architecture integrating the Vision Transformer, lightweight ConvNeXt, and CoAtNet through key innovations: (1) adaptive token pruning with information preservation, (2) gated bidirectional cross-attention for enhanced feature refinement, (3) auxiliary classification heads for multi-task learning, and (4) a dynamic router gate employing context-aware confidence-driven weighting. Experiments across the CIFAR-100, Tobacco3482, CelebA, and Brain Cancer datasets demonstrate EVCC's superiority over powerful models like DeiT-Base, MaxViT-Base, and CrossViT-Base by consistently achieving state-of-the-art accuracy with improvements of up to 2 percentage points, while reducing FLOPs by 25 to 35%. Our adaptive architecture adjusts computational demands to deployment needs by dynamically reducing token count, efficiently balancing the accuracy-efficiency trade-off while combining global context, local details, and hierarchical features for real-world applications. The source code of our implementation is available at https://anonymous.4open.science/r/EVCC.
Authors Kazi Reyazul Hasan, Md Nafiu Rahman, Wasif Jalal, Sadif Ahmed, Shahriar Raj, Mubasshira Musarrat, Muhammad Abdullah Adnan
Categories Computer Vision and Pattern Recognition
3
One Attention, One Scale: Phase-Aligned Rotary Positional Embeddings for Mixed-Resolution Diffusion Transformer
Title CN 一次注意力，一次尺度：用于混合分辨率扩散变换器的相位对齐旋转位置嵌入
Keywords Diffusion Transformer · Rotary Position Embedding · Mixed-Resolution · Phase Aliasing · Attention Mechanism
Summary 本文指出在使用线性插值旋转位置嵌入（RoPE）处理混合分辨率时会导致注意力机制崩溃，原因是不同空间网格的token产生相位混叠。为此提出CRPA方法，通过将所有Q/K位置映射到查询步长上来实现相位对齐，从而稳定预训练DiT的多分辨率生成性能。实验表明其在图像和视频生成上优于现有方法。
Reason 该论文研究的是扩散变换器（DiT）在图像和视频生成任务中处理多分辨率输入时的位置编码问题，提出了一种相位对齐机制CRPA以稳定注意力计算。尽管其技术细节涉及高分辨率视觉输入的建模，但其核心目标是**生成式任务**（image/video generation），与文档图像理解（DIU）中的识别、推理、结构化提取等任务存在根本性差异。方法上未针对OCR幻觉、文本细粒度定位或复杂版面解析等DIU痛点设计，且属于视觉生成领域，迁移价值极低。
Abstract We identify a core failure mode that occurs when using the usual linear interpolation on rotary positional embeddings (RoPE) for mixed-resolution denoising with Diffusion Transformers. When tokens from different spatial grids are mixed, the attention mechanism collapses. The issue is structural. Linear coordinate remapping forces a single attention head to compare RoPE phases sampled at incompatible rates, creating phase aliasing that destabilizes the score landscape. Pretrained DiTs are especially brittle-many heads exhibit extremely sharp, periodic phase selectivity-so even tiny cross-rate inconsistencies reliably cause blur, artifacts, or full collapse. To this end, our main contribution is Cross-Resolution Phase-Aligned Attention (CRPA), a training-free drop-in fix that eliminates this failure at its source. CRPA modifies only the RoPE index map for each attention call: all Q/K positions are expressed on the query's stride so that equal physical distances always induce identical phase increments. This restores the precise phase patterns that DiTs rely on. CRPA is fully compatible with pretrained DiTs, stabilizes all heads and layers uniformly. We demonstrate that CRPA enables high-fidelity and efficient mixed-resolution generation, outperforming previous state-of-the-art methods on image and video generation.
Authors Haoyu Wu, Jingyi Xu, Qiaomu Miao, Dimitris Samaras, Hieu Le
Categories Computer Vision and Pattern Recognition
3
FMPlug: Plug-In Foundation Flow-Matching Priors for Inverse Problems
Title CN FMPlug：用于逆问题的插件式基础流匹配先验
Keywords 流匹配 · 逆问题 · 图像超分辨率 · 去模糊 · 生成先验
Summary 本文提出FMPlug，一种利用基础流匹配模型作为先验的插件框架，通过时间自适应预热和高斯正则化提升在图像超分辨率和去模糊等逆问题上的表现。方法强调对生成流高斯性的利用，增强无监督重建能力。
Reason 该论文属于图像恢复领域（如超分辨率、去模糊），虽然使用了基于流的生成模型作为先验，但其方法定位为通用逆问题的‘插件式’改进，并未针对文档图像理解中的核心痛点（如OCR幻觉、复杂版面分析、细粒度文本定位）设计。其技术路线聚焦于信号重建的正则化策略，难以直接迁移至文档视觉问答或结构化理解任务，且无涉及多模态语言模型（VLM）或文本-布局联合建模。属于与DIU无关的平行应用或通用CV方法。
Abstract We present FMPlug, a novel plug-in framework that enhances foundation flow-matching (FM) priors for solving ill-posed inverse problems. Unlike traditional approaches that rely on domain-specific or untrained priors, FMPlug smartly leverages two simple but powerful insights: the similarity between observed and desired objects and the Gaussianity of generative flows. By introducing a time-adaptive warm-up strategy and sharp Gaussianity regularization, FMPlug unlocks the true potential of domain-agnostic foundation models. Our method beats state-of-the-art methods that use foundation FM priors by significant margins, on image super-resolution and Gaussian deblurring.
Authors Yuxiang Wan, Ryan Devera, Wenjie Zhang, Ju Sun
Categories Computer Vision and Pattern Recognition
3
Lightweight Transformer Framework for Weakly Supervised Semantic Segmentation
Title CN 用于弱监督语义分割的轻量级Transformer框架
Keywords 弱监督语义分割 · 边界优化 · 不确定性建模 · 多尺度融合 · SegFormer
Summary 本文提出CrispFormer，一种基于SegFormer解码器的轻量级改进框架，通过引入边界分支、不确定性引导的修正模块和动态多尺度融合机制，在弱监督语义分割任务中提升了边界清晰度和对小物体的召回能力。模型保持单次前向传播，计算开销低，适用于图像级标签下的高保真掩码生成。
Reason 该论文聚焦于通用图像语义分割中的弱监督学习问题，尽管其在边界锐化和噪声抑制方面有一定技术贡献，但任务本身（WSSS）与文档图像理解（DIU）核心任务如DocVQA、表格识别、文本定位等无直接关联。方法中提出的边界分支、不确定性修正和多尺度融合机制虽有一定设计巧思，但难以迁移解决文档理解中的关键痛点，如OCR幻觉、密集文本解析或复杂版式建模。属于通用CV领域的方法改进，非DIU本题工作，亦无明确可迁移的技术接口。
Abstract Weakly supervised semantic segmentation (WSSS) must learn dense masks from noisy, under-specified cues. We revisit the SegFormer decoder and show that three small, synergistic changes make weak supervision markedly more effective-without altering the MiT backbone or relying on heavy post-processing. Our method, CrispFormer, augments the decoder with: (1) a boundary branch that supervises thin object contours using a lightweight edge head and a boundary-aware loss; (2) an uncertainty-guided refiner that predicts per-pixel aleatoric uncertainty and uses it to weight losses and gate a residual correction of the segmentation logits; and (3) a dynamic multi-scale fusion layer that replaces static concatenation with spatial softmax gating over multi-resolution features, optionally modulated by uncertainty. The result is a single-pass model that preserves crisp boundaries, selects appropriate scales per location, and resists label noise from weak cues. Integrated into a standard WSSS pipeline (seed, student, and EMA relabeling), CrispFormer consistently improves boundary F-score, small-object recall, and mIoU over SegFormer baselines trained on the same seeds, while adding minimal compute. Our decoder-centric formulation is simple to implement, broadly compatible with existing SegFormer variants, and offers a reproducible path to higher-fidelity masks from image-level supervision.
Authors Ali Torabi, Sanjog Gaihre, Yaqoob Majeed
Categories Computer Vision and Pattern Recognition
3
What You See is (Usually) What You Get: Multimodal Prototype Networks that Abstain from Expensive Modalities
Title CN 所见即所得：可放弃昂贵模态的多模态原型网络
Keywords 多模态学习 · 原型网络 · 可解释性 · 成本感知 · 物种识别
Summary 本文提出一种多模态原型网络，通过加权融合不同模态的原型，并在预测时判断是否需要使用昂贵的遗传数据，从而实现成本感知的物种识别。模型能在视觉特征足够清晰时仅依赖图像数据进行分类，在细粒度区分困难时才调用遗传信息。
Reason 该论文研究的是生物物种识别中的多模态学习问题，核心目标是减少对昂贵遗传数据的依赖，利用图像辅助分类。尽管其方法具有一定的可解释性和模态选择机制，但属于特定垂直场景下的应用设计，未针对文档图像理解中的OCR幻觉、复杂排版或长文本推理等核心痛点提出可迁移的方法论创新。属于‘平行下游应用’，不符合本研究方向的技术源头筛选标准。
Abstract Species detection is important for monitoring the health of ecosystems and identifying invasive species, serving a crucial role in guiding conservation efforts. Multimodal neural networks have seen increasing use for identifying species to help automate this task, but they have two major drawbacks. First, their black-box nature prevents the interpretability of their decision making process. Second, collecting genetic data is often expensive and requires invasive procedures, often necessitating researchers to capture or kill the target specimen. We address both of these problems by extending prototype networks (ProtoPNets), which are a popular and interpretable alternative to traditional neural networks, to the multimodal, cost-aware setting. We ensemble prototypes from each modality, using an associated weight to determine how much a given prediction relies on each modality. We further introduce methods to identify cases for which we do not need the expensive genetic information to make confident predictions. We demonstrate that our approach can intelligently allocate expensive genetic data for fine-grained distinctions while using abundant image data for clearer visual classifications and achieving comparable accuracy to models that consistently use both modalities.
Authors Muchang Bahng, Charlie Berens, Jon Donnelly, Eric Chen, Chaofan Chen, Cynthia Rudin
Comment 19 pages. 16 figures. 10 tables
Categories Computer Vision and Pattern Recognition
3
Efficient Transferable Optimal Transport via Min-Sliced Transport Plans
Title CN 通过最小切片传输计划实现高效可迁移的最优传输
Keywords 最优传输 · 切片方法 · 可迁移性 · 点云对齐 · 生成模型
Summary 本文提出了一种基于最小切片传输计划（min-STP）的最优传输方法，通过学习可迁移的投影方向来降低多次最优传输计算的成本，并引入小批量形式以提升扩展性。理论分析表明，在数据分布轻微变化下，优化后的切片器仍保持有效，实验验证了其在点云对齐和基于流的生成建模中的一次匹配性能。
Reason 该论文研究的是最优传输（Optimal Transport）中的计算效率与切片方法的可迁移性，属于底层优化理论，主要应用于点云对齐、生成模型等通用视觉任务。虽然OT在多模态对齐中有潜在作用，但本文未涉及任何文档结构、文本理解或VLM相关设计，且其方法迁移到文档图像理解（如排版分析、表格识别、去OCR幻觉）的路径不明确。属于通用CV优化方法，无针对DIU的创新或适配。
Abstract Optimal Transport (OT) offers a powerful framework for finding correspondences between distributions and addressing matching and alignment problems in various areas of computer vision, including shape analysis, image generation, and multimodal tasks. The computation cost of OT, however, hinders its scalability. Slice-based transport plans have recently shown promise for reducing the computational cost by leveraging the closed-form solutions of 1D OT problems. These methods optimize a one-dimensional projection (slice) to obtain a conditional transport plan that minimizes the transport cost in the ambient space. While efficient, these methods leave open the question of whether learned optimal slicers can transfer to new distribution pairs under distributional shift. Understanding this transferability is crucial in settings with evolving data or repeated OT computations across closely related distributions. In this paper, we study the min-Sliced Transport Plan (min-STP) framework and investigate the transferability of optimized slicers: can a slicer trained on one distribution pair yield effective transport plans for new, unseen pairs? Theoretically, we show that optimized slicers remain close under slight perturbations of the data distributions, enabling efficient transfer across related tasks. To further improve scalability, we introduce a minibatch formulation of min-STP and provide statistical guarantees on its accuracy. Empirically, we demonstrate that the transferable min-STP achieves strong one-shot matching performance and facilitates amortized training for point cloud alignment and flow-based generative modeling.
Authors Xinran Liu, Elaheh Akbari, Rocio Diaz Martin, Navid NaderiAlizadeh, Soheil Kolouri
Categories Computer Vision and Pattern Recognition
3
RADSeg: Unleashing Parameter and Compute Efficient Zero-Shot Open-Vocabulary Segmentation Using Agglomerative Models
Title CN RADSeg：利用凝聚模型实现参数和计算高效的零样本开放词汇分割
Keywords 开放词汇分割 · 零样本学习 · 视觉基础模型 · 效率优化 · RADIO
Summary 本文提出RADSeg，基于凝聚式基础模型RADIO，通过自相关递归注意力、全局聚合和高效掩码优化，在零样本开放词汇语义分割任务上实现了更高的准确率、更低的延迟和更少的参数量。相比大模型组合方案，RADSeg在显著降低计算成本的同时达到SOTA性能。
Reason 该论文聚焦于通用图像的开放词汇语义分割（OVSS），属于通用计算机视觉任务，虽使用了RADIO这一多模态架构，但其方法设计和实验均未涉及文档图像理解（DIU）的核心挑战，如文本识别、布局分析、表格结构提取或OCR幻觉问题。尽管其在效率和零样本迁移方面有优势，但属于通用视觉分割领域的方法优化，对文档图像中的细粒度文本-布局联合理解缺乏直接可迁移性，且未针对文档特有的高分辨率、密集文本、复杂排版等特性进行建模。因此属于无关领域的高效方法，无法为DIU提供底层技术突破。
Abstract Open-vocabulary semantic segmentation (OVSS) underpins many vision and robotics tasks that require generalizable semantic understanding. Existing approaches either rely on limited segmentation training data, which hinders generalization, or apply zero-shot heuristics to vision-language models (e.g CLIP), while the most competitive approaches combine multiple models to improve performance at the cost of high computational and memory demands. In this work, we leverage an overlooked agglomerative vision foundation model, RADIO, to improve zero-shot OVSS along three key axes simultaneously: mIoU, latency, and parameter efficiency. We present the first comprehensive study of RADIO for zero-shot OVSS and enhance its performance through self-correlating recursive attention, self-correlating global aggregation, and computationally efficient mask refinement. Our approach, RADSeg, achieves 6-30% mIoU improvement in the base ViT class while being 3.95x faster and using 2.5x fewer parameters. Surprisingly, RADSeg-base (105M) outperforms previous combinations of huge vision models (850-1350M) in mIoU, achieving state-of-the-art accuracy with substantially lower computational and memory cost.
Authors Omar Alama, Darshil Jariwala, Avigyan Bhattacharya, Seungchan Kim, Wenshan Wang, Sebastian Scherer
Categories Computer Vision and Pattern Recognition
3
Shortcut Invariance: Targeted Jacobian Regularization in Disentangled Latent Space
Title CN 捷径不变性：在解耦潜在空间中的目标雅可比正则化
Keywords 捷径学习 · 分布外泛化 · 解耦表示 · 雅可比正则化 · 潜在空间噪声
Summary 本文提出一种针对解耦潜在空间的目标雅可比正则化方法，通过注入各向异性潜在噪声使分类器对捷径信号不敏感，从而提升模型在分布外数据上的泛化性能。该方法依赖于将潜在空间划分为核心与虚假特征，并利用标签相关性识别捷径维度，实现函数层面的鲁棒性而非表示层面的改进。
Reason 该论文研究的是深度神经网络中的捷径学习问题，提出通过在解耦的潜在空间中对雅可比矩阵进行目标正则化来增强模型的分布外泛化能力。尽管其方法论具有一定通用性，但其核心关注点为表示学习中的偏见缓解，并未涉及多模态输入、视觉-语言对齐或文档结构建模等文档图像理解（DIU）的关键挑战。虽然理论上‘忽略虚假特征’的思想可能间接适用于缓解OCR幻觉，但该方法缺乏对视觉文本布局、高分辨率图像处理或多步推理的支持，且无任何面向文档任务的验证或设计考量，迁移路径模糊。
Abstract Deep neural networks are prone to learning shortcuts, spurious and easily learned correlations in training data that cause severe failures in out-of-distribution (OOD) generalization. A dominant line of work seeks robustness by learning a robust representation, often explicitly partitioning the latent space into core and spurious components; this approach can be complex, brittle, and difficult to scale. We take a different approach, instead of a robust representation, we learn a robust function. We present a simple and effective training method that renders the classifier functionally invariant to shortcut signals. Our method operates within a disentangled latent space, which is essential as it isolates spurious and core features into distinct dimensions. This separation enables the identification of candidate shortcut features by their strong correlation with the label, used as a proxy for semantic simplicity. The classifier is then desensitized to these features by injecting targeted, anisotropic latent noise during training. We analyze this as targeted Jacobian regularization, which forces the classifier to ignore spurious features and rely on more complex, core semantic signals. The result is state-of-the-art OOD performance on established shortcut learning benchmarks.
Authors Shivam Pal, Sakshi Varshney, Piyush Rai
Categories Computer Vision and Pattern Recognition
3
PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers
Title CN PEANuT：基于权重感知神经调节器的参数高效微调
Keywords 参数高效微调 · PEFT · LoRA · 神经网络适配 · 权重感知
Summary 本文提出了PEANuT，一种新的参数高效微调框架，利用权重感知的神经调节器生成基于预训练权重的任务特定更新，在保持低计算开销的同时提升了表达能力。实验表明其在多个NLP与视觉基准上优于现有PEFT方法。
Reason 该论文提出了一种参数高效的微调方法PEANuT，通过引入权重感知的紧凑神经模块来生成任务自适应的参数更新。尽管其在NLP和视觉任务上表现良好，但属于通用PEFT框架，并未针对文档图像理解（DIU）中的具体痛点（如OCR幻觉、复杂排版、细粒度定位等）进行设计或验证。其方法虽理论上可迁移至VLM-based DIU模型的微调阶段，但缺乏对文档结构建模、高分辨率处理或多模态对齐机制的专门改进，属于典型的‘平行下游技术应用’范式，而非面向DIU的方法论创新。
Abstract Fine-tuning large pre-trained foundation models often yields excellent downstream performance but is prohibitively expensive when updating all parameters. Parameter-efficient fine-tuning (PEFT) methods such as LoRA alleviate this by introducing lightweight update modules, yet they commonly rely on weight-agnostic linear approximations, limiting their expressiveness. In this work, we propose PEANuT, a novel PEFT framework that introduces weight-aware neural tweakers, compact neural modules that generate task-adaptive updates conditioned on frozen pre-trained weights. PEANuT provides a flexible yet efficient way to capture complex update patterns without full model tuning. We theoretically show that PEANuT achieves equivalent or greater expressivity than existing linear PEFT methods with comparable or fewer parameters. Extensive experiments across four benchmarks with over twenty datasets demonstrate that PEANuT consistently outperforms strong baselines in both NLP and vision tasks, while maintaining low computational overhead.
Authors Yibo Zhong, Haoxiang Jiang, Lincan Li, Ryumei Nakada, Tianci Liu, Linjun Zhang, Huaxiu Yao, Haoyu Wang
Categories Computation and Language
3
ContrastScore: Towards Higher Quality, Less Biased, More Efficient Evaluation Metrics with Contrastive Evaluation AACL 2025
Title CN ContrastScore：通过对比评估实现更高质量、更少偏见、更高效的评价指标
Keywords 自动评估 · 对比学习 · 大语言模型 · 文本生成 · 评测指标
Summary 本文提出了ContrastScore，一种基于大语言模型的对比式自动评估方法，用于自然语言生成任务的质量评判。相比传统单次打分模式，它通过成对比较生成文本来提升与人类评分的相关性，同时减少长度偏好等系统性偏差。实验表明，即使使用较小规模的Qwen模型（如0.5B），ContrastScore也能超越更大模型的表现。
Reason 该论文提出了一种基于对比学习的文本生成自动评估指标ContrastScore，属于NLP下游任务的评估方法改进。尽管其在机器翻译和摘要任务上表现良好，但本质是‘平行应用’性质的评估器设计，并未针对文档图像理解（DIU）中的核心问题如OCR幻觉、复杂排版或视觉-语言对齐进行建模。虽然LLM-based评估思路理论上可被借用，但该工作本身不提供可迁移至DIU的底层架构、表示学习或推理机制创新，属于负面清单中的‘Wrapper/Application’类研究。
Abstract Evaluating the quality of generated text automatically remains a significant challenge. Conventional reference-based metrics have been shown to exhibit relatively weak correlation with human evaluations. Recent research advocates the use of large language models (LLMs) as source-based metrics for natural language generation (NLG) assessment. While promising, LLM-based metrics, particularly those using smaller models, still fall short in aligning with human judgments. In this work, we introduce ContrastScore, a contrastive evaluation metric designed to enable higher-quality, less biased, and more efficient assessment of generated text. We evaluate ContrastScore on two NLG tasks: machine translation and summarization. Experimental results show that ContrastScore consistently achieves stronger correlation with human judgments than both single-model and ensemble-based baselines. Notably, ContrastScore based on Qwen 3B and 0.5B even outperforms Qwen 7B, despite having only half as many parameters, demonstrating its efficiency. Furthermore, it effectively mitigates common evaluation biases such as length and likelihood preferences, resulting in more robust automatic evaluation.
Authors Xiao Wang, Daniil Larionov, Siwei Wu, Yiqi Liu, Steffen Eger, Nafise Sadat Moosavi, Chenghua Lin
Comment Accepted at AACL 2025 (Main Conference Paper)
Categories Computation and Language
3
CDLM: Consistency Diffusion Language Models For Faster Sampling
Title CN CDLM：用于加速采样的一致性扩散语言模型
Keywords 扩散语言模型 · 并行生成 · 推理加速 · KV缓存 · 一致性建模
Summary 本文提出了一种名为CDLM的一致性扩散语言模型，通过引入一致性建模实现多token同时完成，显著减少采样步数，并设计块状因果注意力掩码以兼容KV缓存，从而在数学与代码生成任务上实现3.6至14.5倍的延迟降低。然而，该工作完全基于纯文本生成框架，不涉及图像输入或多模态交互。
Reason 该论文研究的是通用文本生成中的扩散语言模型加速问题，属于纯NLP生成范畴，未涉及任何视觉模态或文档图像理解任务。尽管其提出的块状注意力机制和KV缓存优化可能对长序列建模有理论价值，但无法直接迁移解决文档图像理解中的核心痛点（如OCR幻觉、细粒度定位、高分辨率处理等）。属于与DIU无关的平行技术路线。
Abstract Diffusion Language Models (DLMs) offer a promising parallel generation paradigm but suffer from slow inference due to numerous refinement steps and the inability to use standard KV caching. We introduce CDLM (Consistency Diffusion Language Models), a training-based acceleration method that simultaneously tackles both bottlenecks. CDLM integrates consistency modeling to drastically reduce the number of required sampling steps by enabling multi-token finalization. Furthermore, we enforce a block-wise causal attention mask during fine-tuning, making the model fully compatible with KV caching. Experiments show CDLM achieves 3.6x-14.5x lower latency while maintaining competitive accuracy on math and coding tasks. The full training and evaluation code is available at https://github.com/SqueezeAILab/CDLM.
Authors Minseo Kim, Chenfeng Xu, Coleman Hooper, Harman Singh, Ben Athiwaratkun, Ce Zhang, Kurt Keutzer, Amir Gholami
Comment 18 pages, 6 figures
Categories Computation and Language
3
Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models EMNLP 2025 Findings
Title CN 不要想当然地接受前提：评估大语言模型的前提批判能力
Keywords 大语言模型 · 前提批判 · 推理鲁棒性 · 评测基准 · 幻觉检测
Summary 论文提出了一个名为PCBench的评测基准，用于评估大语言模型是否能主动发现并指出输入中包含的错误前提。研究发现当前大多数LLM缺乏自主批判能力，容易因错误前提导致推理低效或失败。虽然对提升模型可靠性有一定意义，但完全基于纯文本任务，不涉及视觉或文档模态。
Reason 该论文聚焦于纯文本NLP领域中的大语言模型在面对错误前提时的识别与批判能力，属于通用语言模型推理安全与鲁棒性研究。尽管其探讨的‘幻觉’相关问题（如LLM盲目接受错误输入）在广义上与文档理解中的OCR幻觉有微弱关联，但本文未涉及任何视觉模态、多模态对齐或文档结构信息处理，提出的方法和评测框架均为纯文本逻辑推理场景设计，无法直接迁移至文档图像理解中的核心痛点（如细粒度定位、高分辨率布局分析、表格识别等）。因此属于无关领域的平行应用研究，不符合DIU方法论创新要求。
Abstract Large language models (LLMs) have witnessed rapid advancements, demonstrating remarkable capabilities. However, a notable vulnerability persists: LLMs often uncritically accept flawed or contradictory premises, leading to inefficient reasoning and unreliable outputs. This emphasizes the significance of possessing the \textbf{Premise Critique Ability} for LLMs, defined as the capacity to proactively identify and articulate errors in input premises. Most existing studies assess LLMs' reasoning ability in ideal settings, largely ignoring their vulnerabilities when faced with flawed premises. Thus, we introduce the \textbf{Premise Critique Bench (PCBench)}, designed by incorporating four error types across three difficulty levels, paired with multi-faceted evaluation metrics. We conducted systematic evaluations of 15 representative LLMs. Our findings reveal: (1) Most models rely heavily on explicit prompts to detect errors, with limited autonomous critique; (2) Premise critique ability depends on question difficulty and error type, with direct contradictions being easier to detect than complex or procedural errors; (3) Reasoning ability does not consistently correlate with the premise critique ability; (4) Flawed premises trigger overthinking in reasoning models, markedly lengthening responses due to repeated attempts at resolving conflicts. These insights underscore the urgent need to enhance LLMs' proactive evaluation of input validity, positioning premise critique as a foundational capability for developing reliable, human-centric systems. The code is available at https://github.com/MLGroupJLU/Premise_Critique.
Authors Jinzhe Li, Gengxu Li, Yi Chang, Yuan Wu
Comment EMNLP 2025 Findings camera-ready version
Categories Computation and Language
3
Think Before You Prune: Selective Self-Generated Calibration for Pruning Large Reasoning Models
Title CN 剪枝前先思考：面向大推理模型剪枝的自生成选择性校准
Keywords 大推理模型 · 模型剪枝 · 自生成数据 · 校准 · 推理效率
Summary 本文首次探索了大推理模型（LRM）的剪枝方法，发现直接沿用现有LLM剪枝技术效果不佳。作者提出使用自生成的推理数据作为校准集，并设计了一种选择性生成策略（SSGR），优先采用难度适中、长度合理的复杂推理样本，显著提升了剪枝后模型的推理能力。实验基于DeepSeek-R1-Distill系列模型验证了该方法的有效性。
Reason 该论文研究的是大推理模型（LRM）的模型压缩问题，核心是通过自生成推理数据进行剪枝校准，属于LLM推理效率优化范畴。虽然其方法论涉及推理链质量对模型性能的影响，但完全未涉及视觉模态、文档结构或OCR相关挑战，无法迁移到文档图像理解中的幻觉、密集文本处理或排版建模等核心痛点。属于典型的非DIU领域‘平行技术’应用（在推理模型上做剪枝），且无多模态扩展设计。
Abstract Large Reasoning Models (LRMs) have demonstrated remarkable performance on complex reasoning benchmarks. However, their long chain-of-thought reasoning processes incur significant inference overhead. Pruning has emerged as a promising approach to reducing computational costs. However, existing efforts have primarily focused on large language models (LLMs), while pruning LRMs remains unexplored. In this work, we conduct the first empirical study on pruning LRMs and show that directly applying existing pruning techniques fails to yield satisfactory results. Our findings indicate that using self-generated reasoning data for calibration can substantially improve pruning performance. We further investigate how the difficulty and length of reasoning data affect pruning outcomes. Our analysis reveals that challenging and moderately long self-generated reasoning data serve as ideal calibration data. Based on these insights, we propose a Selective Self-Generated Reasoning (SSGR) data construction strategy to provide effective calibration data for pruning LRMs. Experimental results on the DeepSeek-R1-Distill model series validate that our strategy improves the reasoning ability of pruned LRMs by 10%-13% compared to general pruning methods.
Authors Yang Xiang, Yixin Ji, Juntao Li, Min Zhang
Comment Under Review
Categories Computation and Language
3
Advancing Multi-Agent RAG Systems with Minimalist Reinforcement Learning
Title CN 利用极简强化学习推进多智能体RAG系统
Keywords 多智能体系统 · 检索增强生成 · 强化学习 · 长上下文推理 · 复杂问答
Summary 本文提出Mujica-MyGo框架，通过多智能体协作（Mujica）将多跳检索任务分解为子任务以缓解长上下文问题，并设计极简强化学习算法MyGO对LLM进行后训练，避免依赖上下文学习。实验表明其在多个问答基准上表现优异。
Reason 该论文属于典型的‘平行下游应用’，其核心是构建一个多智能体RAG框架用于复杂问答任务，尽管使用了强化学习进行策略优化，但整体工作聚焦于通用文本和知识图谱上的RAG流程改进，并未针对文档图像理解（DIU）中的任何具体挑战（如OCR幻觉、布局分析、视觉-文本对齐、高分辨率处理等）提出方法创新。虽然‘多智能体协作’和‘减少上下文依赖’的思想在理论上可能启发长文档处理流程设计，但该方法本身缺乏向DIU迁移的关键接口（例如视觉输入建模、细粒度定位、视觉token压缩等），属于将现有技术组合应用于特定流程的工程性优化，不符合本筛选标准中对底层方法或DIU本题创新的要求。
Abstract Large Language Models (LLMs) equipped with modern Retrieval-Augmented Generation (RAG) systems often employ multi-turn interaction pipelines to interface with search engines for complex reasoning tasks. However, such multi-turn interactions inevitably produce long intermediate contexts, as context length grows exponentially with exploration depth. This leads to a well-known limitation of LLMs: their difficulty in effectively leveraging information from long contexts. This problem is further amplified in RAG systems that depend on in-context learning, where few-shot demonstrations must also be included in the prompt, compounding the context-length bottleneck. To address these challenges, we propose Mujica-MyGo, a unified framework for efficient multi-turn reasoning in RAG. Inspired by the divide-and-conquer principle, we introduce Mujica (Multi-hop Joint Intelligence for Complex Question Answering), a multi-agent RAG workflow that decomposes multi-turn interactions into cooperative sub-interactions, thereby mitigating long-context issues. To eliminate the dependency on in-context learning, we further develop MyGO (Minimalist Policy Gradient Optimization), a lightweight and efficient reinforcement learning algorithm that enables effective post-training of LLMs within complex RAG pipelines. We provide theoretical guarantees for MyGO's convergence to the optimal policy. Empirical evaluations across diverse question-answering benchmarks, covering both text corpora and knowledge graphs, show that Mujica-MyGO achieves superior performance.
Authors Yihong Wu, Liheng Ma, Muzhi Li, Jiaming Zhou, Lei Ding, Jianye Hao, Ho-fung Leung, Irwin King, Yingxue Zhang, Jian-Yun Nie
Categories Computation and Language
3
Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost
Title CN Kitty：基于动态通道级精度提升的精确高效2位KV缓存量化
Keywords KV缓存量化 · 2-bit量化 · 推理加速 · 混合精度 · 内存优化
Summary 本文提出Kitty，一种算法-系统协同设计的混合精度KV缓存量化方案，通过动态通道级精度提升，在保持接近2位平均精度的同时将关键缓存部分以4位存储，从而在几乎无精度损失的情况下实现近8倍的KV内存压缩。该方法优化了页面布局和反量化内核，支持Triton且运行时开销低，在多种任务和模型上显著提升了吞吐量。
Reason 该论文聚焦于LLM推理阶段的KV缓存内存优化，属于通用大模型推理加速技术。尽管其在长上下文处理中提升了效率，但并未针对文档图像理解（DIU）中的核心问题如OCR幻觉、细粒度定位、复杂排版或视觉-语言对齐进行设计。方法本质上是系统级优化，难以直接迁移用于解决DIU特有的多模态表征、高分辨率输入建模或视觉token压缩等挑战，属于与DIU任务无关的底层推理加速工作。
Abstract The KV cache is a dominant memory bottleneck for LLM inference. While 4-bit KV quantization preserves accuracy, 2-bit often degrades it, especially on long-context reasoning. We close this gap via an algorithm-system co-design for mixed-precision KV caching: Kitty. On the algorithm side, extensive experiments show that Dynamic Channel-wise Precision Boost -- which ranks Key-cache channels by sensitivity and keeps only a small fraction at higher precision -- maintains near-zero loss in accuracy drop while approaching 2-bit memory. The main challenge is handling dynamic 4-bit channel boosts while keeping the page layout coalesced and the dequantization uniform, with no scattered reads or hard-coded masks. Kitty addresses these issues by decompose each mixed-precision Key page into two tensors with unified 2-bit precision. Based on this, Kitty provides a page-centric KV layout, Triton-compatible page dequantization kernels, and a lightweight runtime pipeline that preserves coalescing and avoids divergence. Across seven tasks and two model families (Qwen3, LLaMA3), Kitty cuts KV memory by nearly 8x with negligible accuracy loss, enabling up to 8x larger batches and 2.1x-4.1x higher throughput under the same memory budget. We release the full implementation of Kitty at https://github.com/Summer-Summer/Kitty.
Authors Haojun Xia, Xiaoxia Wu, Jisen Li, Robert Wu, Junxiong Wang, Jue Wang, Chenxi Li, Aman Singhal, Alay Dilipbhai Shah, Alpay Ariyak, Donglin Zhuang, Zhongzhu Zhou, Ben Athiwaratkun, Zhen Zheng, Shuaiwen Leon Song
Categories Artificial Intelligence
3
Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs ICLR 2025
Title CN 直降至零：为何将学习率线性衰减至零对大语言模型最有效
Keywords 学习率调度 · 训练优化 · 大语言模型 · D2Z · AdamW
Summary 本文提出，在大语言模型训练中，采用线性衰减至零（D2Z）的学习率调度策略优于常见的余弦衰减至10%最大值的方法，可在最优峰值学习率下显著降低训练损失并节省高达60%的计算资源。作者通过将AdamW解释为权重更新的指数移动平均，理论分析了D2Z在早期逃离初始状态和后期抑制梯度噪声之间的平衡优势。
Reason 该论文研究的是大语言模型训练中的学习率调度策略，属于通用LLM训练优化范畴。尽管其方法在训练效率上表现优异，但并未针对文档图像理解（DIU）任务设计，也不涉及多模态输入、视觉表示、OCR去幻觉、细粒度定位或高分辨率处理等DIU核心痛点。该工作属于标准的NLP训练工程优化，无法直接迁移解决文档理解中的关键挑战，且无任何视觉-语言交互机制创新。
Abstract LLMs are commonly trained with a learning rate (LR) warmup, followed by cosine decay to 10% of the maximum (10x decay). In a large-scale empirical study, we show that under an optimal peak LR, a simple linear decay-to-zero (D2Z) schedule consistently outperforms other schedules when training at compute-optimal dataset sizes. D2Z is superior across a range of model sizes, batch sizes, datasets, and vocabularies. Benefits increase as dataset size increases. Leveraging a novel interpretation of AdamW as an exponential moving average of weight updates, we show how linear D2Z optimally balances the demands of early training (moving away from initial conditions) and late training (averaging over more updates in order to mitigate gradient noise). In experiments, a 610M-parameter model trained for 80 tokens-per-parameter (TPP) using D2Z achieves lower loss than when trained for 200 TPP using 10x decay, corresponding to an astonishing 60% compute savings. Models such as Llama2-7B, trained for 286 TPP with 10x decay, could likely have saved a majority of compute by training with D2Z.
Authors Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, Joel Hestness
Comment ICLR 2025
Categories Artificial Intelligence, Computation and Language
3
Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training NeurIPS 2025
Title CN 权力线：大语言模型预训练中权重衰减与批量大小的缩放定律
Keywords 缩放定律 · 权重衰减 · 批量大小 · 大语言模型 · 超参数优化
Summary 本文提出了在大语言模型预训练中，权重衰减和批量大小随模型规模和数据量变化的缩放定律，验证了AdamW时间尺度应保持恒定，并发现最优权重衰减依赖于D/N比率，而最优与临界批量大小则遵循与模型大小无关的数据量幂律。这些规律可用于指导大规模训练的超参数设置。
Reason 该论文研究的是大语言模型预训练中的超参数缩放规律，聚焦于权重衰减、批量大小与模型/数据规模之间的关系。虽然其发现对LLM训练具有指导意义，但完全属于纯语言模态的训练优化问题，未涉及任何视觉输入、多模态对齐或文档结构建模。无法迁移解决文档图像理解中的OCR幻觉、复杂排版或高分辨率视觉处理等核心痛点，属于与DIU无关的底层NLP训练工程工作。
Abstract Efficient LLM pre-training requires well-tuned hyperparameters (HPs), including learning rate $η$ and weight decay $λ$. We study scaling laws for HPs: formulas for how to scale HPs as we scale model size N, dataset size D, and batch size B. Recent work suggests the AdamW timescale, $τ= B/(ηλD)$, should remain constant across training settings, and we verify the implication that optimal $λ$ scales linearly with B, for a fixed N and D. However, as N and D scale, we show optimal $τ$ obeys a precise power law in the tokens-per-parameter ratio, D/N. This law thus provides a method to accurately predict $λ$opt in advance of large-scale training. We also study scaling laws for optimal batch size Bopt (the B enabling lowest loss at a given N,D) and critical batch size Bcrit (the B beyond which further data parallelism becomes ineffective). In contrast to prior work, we find both Bopt and Bcrit scale as power laws in D, independent of model size, N. Finally, we analyze how these findings inform the real-world selection of Pareto-optimal N and D under dual training time and compute objectives. All experiments were run on Cerebras CS-3 systems.
Authors Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, Joel Hestness
Comment NeurIPS 2025
Categories Artificial Intelligence, Computation and Language
3
Multimodal Continual Learning with MLLMs from Multi-scenario Perspectives CVPR 2026
Title CN 基于多场景视角的多模态大语言模型持续学习
Keywords 持续学习 · 多模态大语言模型 · 灾难性遗忘 · 跨场景理解 · 视觉表征
Summary 本文提出UNIFIER方法，通过在视觉块中解耦不同场景的视觉信息并引入一致性约束，缓解多模态大语言模型在动态场景流中的灾难性遗忘。作者构建了包含四种场景的MSVQA数据集进行验证，在跨场景VQA任务上展示了知识积累能力。
Reason 该论文研究的是多模态大语言模型在不同视觉场景（如高空、水下、室内等）下的持续学习与灾难性遗忘问题，其核心关注点是跨场景视觉理解中的表征稳定性，而非文档图像特有的结构、布局或文本密集性等问题。虽然使用了VQA形式的数据集MSVQA，但任务设定与文档无关，方法重点在于多分支视觉编码以应对视角变化，对解决OCR幻觉、复杂版式解析、长文档推理等DIU核心痛点无直接迁移价值。属于将现有MLLM+持续学习框架应用于新场景的‘平行应用’，不符合本方向对底层方法创新的要求。
Abstract Continual learning in visual understanding aims to deal with catastrophic forgetting in Multimodal Large Language Models (MLLMs). MLLMs deployed on devices have to continuously adapt to dynamic scenarios in downstream tasks, such as variations in background and perspective, to effectively perform complex visual tasks. To this end, we construct a multimodal visual understanding dataset (MSVQA) encompassing four different scenarios and perspectives including high altitude, underwater, low altitude and indoor, to investigate the catastrophic forgetting in MLLMs under the dynamics of scenario shifts in real-world data streams. Furthermore, we propose mUltimodal coNtInual learning with MLLMs From multi-scenarIo pERspectives (UNIFIER) to address visual discrepancies while learning different scenarios. Specifically, it decouples the visual information from different scenarios into distinct branches within each vision block and projects them into the same feature space. A consistency constraint is imposed on the features of each branch to maintain the stability of visual representations across scenarios. Extensive experiments on the MSVQA dataset demonstrate that UNIFIER effectively alleviates forgetting of cross-scenario tasks and achieves knowledge accumulation within the same scenario.
Authors Kai Jiang, Siqi Huang, Xiangyu Chen, Jiawei Shao, Hongyuan Zhang, Xuelong Li
Comment 18 pages, 16 figures. This is a preprint version of a paper submitted to CVPR 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraints
Title CN ORIGAMISPACE：基于数学约束的多步空间推理中多模态大语言模型的基准测试
Keywords 多模态大模型 · 空间推理 · 基准测试 · 折纸任务 · 数学约束
Summary 本文提出了ORIGAMISPACE，一个用于评估多模态大语言模型在多步空间推理和数学约束处理能力上的新基准，基于折纸任务设计了四种评估任务，并构建了一个包含折痕图、折叠过程和最终形状的数据集。实验揭示了现有MLLM在复杂空间推理上的局限性。
Reason 该论文聚焦于折纸任务中的多步空间推理与数学约束建模，属于特定领域的视觉-语言推理应用。虽然涉及多模态模型的空间理解能力评估，但其任务设定（如折纸图解生成、折叠过程预测）与文档图像理解的核心痛点（如OCR幻觉、密集文本识别、复杂版式分析）关联较弱。未提出可迁移至DIU领域的通用架构、训练方法或表示学习机制，本质上是一个垂直场景的基准构建工作，属于‘平行应用’范畴。
Abstract Spatial reasoning is a key capability in the field of artificial intelligence, especially crucial in areas such as robotics, computer vision, and natural language understanding. However, evaluating the ability of multimodal large language models(MLLMs) in complex spatial reasoning still faces challenges, particularly in scenarios requiring multi-step reasoning and precise mathematical constraints. This paper introduces ORIGAMISPACE, a new dataset and benchmark designed to evaluate the multi-step spatial reasoning ability and the capacity to handle mathematical constraints of MLLMs through origami tasks. The dataset contains 350 data instances,each comprising a strictly formatted crease pattern (CP diagram), the Compiled Flat Pattern, the complete Folding Process, and the final Folded Shape Image. We propose four evaluation tasks: Pattern Prediction, Multi-step Spatial Reasoning, Spatial Relationship Prediction, and End-to-End CP Code Generation. For the CP code generation task, we design an interactive environment and explore the possibility of using reinforcement learning methods to train MLLMs. Through experiments on existing MLLMs, we initially reveal the strengths and weaknesses of these models in handling complex spatial reasoning tasks.
Authors Rui Xu, Dakuan Lu, Zicheng Zhao, Xiaoyu Tan, Xintao Wang, Siyu Yuan, Jiangjie Chen, Yinghui Xu
Categories Artificial Intelligence
3
DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translation
Title CN DocPTBench：面向端到端拍摄文档解析与翻译的基准测试
Keywords 文档图像理解 · 基准测试 · 拍摄文档 · 多模态大模型 · 文档翻译
Summary 本文提出了DocPTBench，一个包含1300多份高分辨率拍摄文档的大规模基准数据集，涵盖多个领域和八种翻译场景，用于评估端到端文档解析与翻译系统在真实拍摄条件下的表现。实验表明，当前主流MLLM和专用文档模型在面对拍摄文档时性能显著下降，暴露出对几何畸变和光照变化等现实因素的脆弱性。尽管具有评测价值，但该工作本身无技术源头创新。
Reason 该论文提出一个针对拍摄文档（photographed documents）的解析与翻译基准数据集，主要贡献在于数据构建和评测场景设定。然而，其工作属于特定场景下的下游应用型基准建设，并未提出任何新的模型架构、训练方法或可迁移的技术机制。尽管揭示了现有模型在真实拍摄条件下的性能下降问题，但本身不提供解决OCR幻觉、复杂排版处理或长文档推理等DIU核心痛点的方法论创新，属于‘平行下游应用’范畴。
Abstract The advent of Multimodal Large Language Models (MLLMs) has unlocked the potential for end-to-end document parsing and translation. However, prevailing benchmarks such as OmniDocBench and DITrans are dominated by pristine scanned or digital-born documents, and thus fail to adequately represent the intricate challenges of real-world capture conditions, such as geometric distortions and photometric variations. To fill this gap, we introduce DocPTBench, a comprehensive benchmark specifically designed for Photographed Document Parsing and Translation. DocPTBench comprises over 1,300 high-resolution photographed documents from multiple domains, includes eight translation scenarios, and provides meticulously human-verified annotations for both parsing and translation. Our experiments demonstrate that transitioning from digital-born to photographed documents results in a substantial performance decline: popular MLLMs exhibit an average accuracy drop of 18% in end-to-end parsing and 12% in translation, while specialized document parsing models show significant average decrease of 25%. This substantial performance gap underscores the unique challenges posed by documents captured in real-world conditions and reveals the limited robustness of existing models. Dataset and code are available at https://github.com/Topdu/DocPTBench.
Authors Yongkun Du, Pinxuan Chen, Xuye Ying, Zhineng Chen
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
General Agentic Memory Via Deep Research
Title CN 通过深度研究实现通用代理记忆
Keywords AI Agent · Memory System · Test-time Scalability · Reinforcement Learning · Context Construction
Summary 本文提出了通用代理记忆（GAM）框架，采用‘即时编译’理念，在离线阶段保留简洁记忆，在运行时通过Researcher模块从全局页面存储中检索并整合信息，以支持高效、可扩展的任务执行。实验表明其在多种需记忆支撑的任务上优于现有系统。
Reason 该论文提出了一种通用的代理记忆框架GAM，核心是‘即时编译’式的动态上下文构建机制，包含轻量级记忆体和运行时检索整合模块。尽管其在代理系统中具有架构创新性，但属于Agent内存管理的通用设计，并未针对文档图像理解中的具体痛点（如OCR幻觉、密集文本识别、复杂版式建模）进行适配或验证。虽然理论上可用于长文档多步推理任务的记忆保持，但缺乏向DIU场景迁移的具体路径与证据，属于平行应用层面的上游技术，对文档理解的方法论推动有限。
Abstract Memory is critical for AI agents, yet the widely-adopted static memory, aiming to create readily available memory in advance, is inevitably subject to severe information loss. To address this limitation, we propose a novel framework called \textbf{general agentic memory (GAM)}. GAM follows the principle of "\textbf{just-in time (JIT) compilation}" where it focuses on creating optimized contexts for its client at runtime while keeping only simple but useful memory during the offline stage. To this end, GAM employs a duo-design with the following components. 1) \textbf{Memorizer}, which highlights key historical information using a lightweight memory, while maintaining complete historical information within a universal page-store. 2) \textbf{Researcher}, which retrieves and integrates useful information from the page-store for its online request guided by the pre-constructed memory. This design allows GAM to effectively leverage the agentic capabilities and test-time scalability of frontier large language models (LLMs), while also facilitating end-to-end performance optimization through reinforcement learning. In our experimental study, we demonstrate that GAM achieves substantial improvement on various memory-grounded task completion scenarios against existing memory systems.
Authors B. Y. Yan, Chaofan Li, Hongjin Qian, Shuqi Lu, Zheng Liu
Categories Artificial Intelligence, Computation and Language
3
Categorical Equivariant Deep Learning: Category-Equivariant Neural Networks and Universal Approximation Theorems
Title CN 范畴等变深度学习：范畴等变神经网络与通用逼近定理
Keywords 等变神经网络 · 范畴论 · 通用逼近定理 · 图神经网络 · 理论深度学习
Summary 本文提出了范畴等变神经网络（CENNs）的统一理论框架，将群、群胚、偏序集、图和层神经网络等现有等变网络纳入一个基于范畴论和拓扑范畴中自然性的统一形式化体系，并证明了广义的等变通用逼近定理。该工作旨在从数学层面扩展等变深度学习的边界，涵盖几何、上下文和组合对称性。
Reason 该论文属于理论性极强的数学框架研究，提出了一种基于范畴论的等变神经网络统一理论。尽管其在群、图、格等结构上具有普适性，但并未针对文档图像理解（DIU）中的具体任务（如DocVQA、版面分析、表格识别）或实际痛点（如OCR幻觉、高分辨率处理、细粒度定位）提供可迁移的解决方案。其方法抽象程度过高，缺乏向DIU场景落地的技术路径，属于与文档理解无关的上游理论探索。
Abstract We develop a theory of category-equivariant neural networks (CENNs) that unifies group/groupoid-equivariant networks, poset/lattice-equivariant networks, graph and sheaf neural networks. Equivariance is formulated as naturality in a topological category with Radon measures, formulating linear and nonlinear layers in the categorical setup. We prove the equivariant universal approximation theorem in the general setting: the class of finite-depth CENNs is dense in the space of continuous equivariant transformations. We instantiate the framework for groups/groupoids, posets/lattices, graphs and cellular sheaves, deriving universal approximation theorems for them in a systematic manner. Categorical equivariant deep learning thus allows us to expand the horizons of equivariant deep learning beyond group actions, encompassing not only geometric symmetries but also contextual and compositional symmetries.
Authors Yoshihiro Maruyama
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
NSTR: Neural Spectral Transport Representation for Space-Varying Frequency Fields
Title CN NSTR：用于空间变化频率场的神经谱传输表示
Keywords 隐式神经表示 · 频谱建模 · 频率传输 · 信号重建 · 空间变化
Summary 本文提出了一种名为NSTR的新型隐式神经表示框架，通过可学习的频率传输方程显式建模空间变化的局部频谱结构，从而更高效地重建具有局部高频细节和频谱漂移的信号。实验显示其在参数效率和收敛速度上优于SIREN、傅里叶特征MLP和Instant-NGP等方法。
Reason 该论文属于隐式神经表示（INR）领域，研究的是通用信号（如图像、音频、3D）的频谱建模，其核心贡献是引入空间变化的频率场以提升表示效率。尽管在高分辨率信号建模上有潜力，但其方法面向的是与文档图像理解（DIU）无关的底层表示学习，且未涉及多模态语言模型、文本布局或视觉-语言对齐等DIU关键问题。无法直接迁移用于解决OCR幻觉、复杂排版解析或DocVQA中的推理问题，属于无关领域的技术探索。
Abstract Implicit Neural Representations (INRs) have emerged as a powerful paradigm for representing signals such as images, audio, and 3D scenes. However, existing INR frameworks -- including MLPs with Fourier features, SIREN, and multiresolution hash grids -- implicitly assume a \textit{global and stationary} spectral basis. This assumption is fundamentally misaligned with real-world signals whose frequency characteristics vary significantly across space, exhibiting local high-frequency textures, smooth regions, and frequency drift phenomena. We propose \textbf{Neural Spectral Transport Representation (NSTR)}, the first INR framework that \textbf{explicitly models a spatially varying local frequency field}. NSTR introduces a learnable \emph{frequency transport equation}, a PDE that governs how local spectral compositions evolve across space. Given a learnable local spectrum field $S(x)$ and a frequency transport network $F_θ$ enforcing $\nabla S(x) \approx F_θ(x, S(x))$, NSTR reconstructs signals by spatially modulating a compact set of global sinusoidal bases. This formulation enables strong local adaptivity and offers a new level of interpretability via visualizing frequency flows. Experiments on 2D image regression, audio reconstruction, and implicit 3D geometry show that NSTR achieves significantly better accuracy-parameter trade-offs than SIREN, Fourier-feature MLPs, and Instant-NGP. NSTR requires fewer global frequencies, converges faster, and naturally explains signal structure through spectral transport fields. We believe NSTR opens a new direction in INR research by introducing explicit modeling of space-varying spectrum.
Authors Plein Versace
Categories Artificial Intelligence
3
Progressive Localisation in Localist LLMs
Title CN 局部主义大语言模型中的渐进式定位
Keywords 大语言模型 · 注意力机制 · 可解释性 · AI安全 · 渐进式定位
Summary 本文探讨了在大语言模型中采用渐进式注意力局部化的架构设计，通过在GPT-2上进行微调实验，发现从早期分布式注意力到后期局部化注意力的渐进策略（尤其是五次多项式增长）能在仅轻微降低性能的情况下显著提升模型可解释性，适用于需要人类监督的安全关键场景。然而，该工作未涉及图像或文档模态。
Reason 该论文研究的是纯文本大语言模型中的注意力局部化问题，聚焦于AI安全和可解释性，未涉及任何视觉模态或文档图像理解任务。尽管‘late-layer localization’可能对推理过程的可控性有一定启发，但其方法完全基于文本模态，没有提出可用于解决文档图像理解中OCR幻觉、细粒度定位或高分辨率处理等核心痛点的技术路径。属于无关领域的纯NLP研究，且不涉及多模态对齐或视觉表示学习。
Abstract This paper demonstrates that progressive localization, the gradual increase of attention locality from early distributed layers to late localized layers, represents the optimal architecture for creating interpretable large language models while preserving performance. Through systematic experimentation with GPT-2 fine tuned on The Psychology of Artificial Superintelligence, we evaluate seven locality configurations ranging from fully distributed to strictly localist, with five progressive schedules implementing polynomial increases (linear through quintic). Our key finding is that late-layer localization is critical for AI safety applications: the progressive quintic schedule achieves perplexity of 14.64, only 1.89 times worse than the fully distributed baseline while providing interpretable attention patterns in output layers where safety-critical decisions are made. This represents an 84.2% improvement over previous localist implementations and narrows the performance gap from 6.6 times to 1.89 times. The systematic relationship between localization schedule steepness and performance validates the hypothesis that early layers require distributed processing for feature extraction while late layers benefit from localized, interpretable attention for decision-making. These findings establish progressive localization as the principled approach for building transparent AI systems in safety-critical domains, where human oversight of model reasoning is essential.
Authors Joachim Diederich
Categories Artificial Intelligence
3
Low-Confidence Gold: Refining Low-Confidence Samples for Efficient Instruction Tuning EMNLP Findings 2025
Title CN 低置信度黄金：通过精炼低置信度样本来实现高效指令微调
Keywords 指令微调 · 数据效率 · 置信度筛选 · 聚类 · 轻量级分类器
Summary 本文提出了一种名为Low-Confidence Gold（LCG）的数据过滤框架，用于提升大语言模型指令微调的效率。该方法通过质心聚类和置信度引导选择，结合轻量级分类器进行半监督筛选，在仅使用6K样本的情况下实现了优于现有方法的性能。然而，该工作完全局限于纯文本指令数据，不涉及任何视觉或多模态成分。
Reason 该论文聚焦于纯文本大语言模型的指令微调数据筛选，提出通过聚类和置信度选择机制优化训练集。尽管其数据效率思想具有一定通用性，但方法完全基于文本模态，未涉及视觉或文档图像输入，与文档图像理解（DIU）的核心挑战如OCR幻觉、复杂版式建模、多模态对齐等无直接关联。属于典型的NLP领域平行技术，无法迁移用于解决文档理解中的视觉-语言联合推理问题。
Abstract The effectiveness of instruction fine-tuning for Large Language Models is fundamentally constrained by the quality and efficiency of training datasets. This work introduces Low-Confidence Gold (LCG), a novel filtering framework that employs centroid-based clustering and confidence-guided selection for identifying valuable instruction pairs. Through a semi-supervised approach using a lightweight classifier trained on representative samples, LCG curates high-quality subsets while preserving data diversity. Experimental evaluation demonstrates that models fine-tuned on LCG-filtered subsets of 6K samples achieve superior performance compared to existing methods, with substantial improvements on MT-bench and consistent gains across comprehensive evaluation metrics. The framework's efficacy while maintaining model performance establishes a promising direction for efficient instruction tuning.
Authors Hongyi Cai, Jie Li, Mohammad Mahdinur Rahman, Wenzhen Dong
Comment Accepted to EMNLP Findings 2025
Categories Artificial Intelligence, Computation and Language
3
Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
Title CN Uni-MoE-2.0-Omni：通过先进的MoE、训练和数据扩展语言为中心的全模态大模型
Keywords 全模态大模型 · MoE · 多模态学习 · 生成模型 · 跨模态对齐
Summary 本文提出了Uni-MoE-2.0-Omni，一个开源的全模态大模型，基于动态容量的MoE架构、渐进式强化训练策略和多模态数据匹配技术，在85个基准上表现出色，尤其在视频理解、音视频推理和语音处理方面。模型支持文本、图像和语音的生成与理解，但未专门针对文档图像理解任务进行优化或评估。
Reason 该论文属于全模态大模型（OLM）的通用架构改进，聚焦于多模态理解与生成（如视频、音频、图像、文本），尽管其MoE设计和训练策略具有一定技术深度，但并未针对文档图像理解（DIU）中的核心痛点（如OCR幻觉、复杂版式、细粒度定位）进行建模或验证。其强调的能力集中在视频理解、语音处理和跨模态生成，迁移至文档理解场景的路径不明确。且未涉及视觉token压缩、高分辨率处理或文档特定结构建模，属于通用VLM的平行扩展，不符合DIU本题优先原则。
Abstract We present Uni-MoE 2.0 from the Lychee family. As a fully open-source omnimodal large model (OLM), it substantially advances Lychee's Uni-MoE series in language-centric multimodal understanding, reasoning, and generating. Based on the dense LLM, we build Uni-MoE-2.0-Omni from scratch through three core contributions: dynamic-capacity Mixture-of-Experts (MoE) design, a progressive training strategy enhanced with an iterative reinforcement strategy, and a carefully curated multimodal data matching technique. It is capable of omnimodal understanding, as well as generating images, text, and speech. Architecturally, our new MoE framework balances computational efficiency and capability for 10 cross-modal inputs using shared, routed, and null experts, while our Omni-Modality 3D RoPE ensures spatio-temporal cross-modality alignment in the self-attention layer. For training, following cross-modal pretraining, we use a progressive supervised fine-tuning strategy that activates modality-specific experts and is enhanced by balanced data composition and an iterative GSPO-DPO method to stabilise RL training and improve reasoning. Data-wise, the base model, trained on approximately 75B tokens of open-source multimodal data, is equipped with special speech and image generation tokens, allowing it to learn these generative tasks by conditioning its outputs on linguistic cues. Extensive evaluation across 85 benchmarks demonstrates that our model achieves SOTA or highly competitive performance against leading OLMs, surpassing Qwen2.5-Omni (trained with 1.2T tokens) on over 50 of 76 benchmarks. Key strengths include video understanding (+7% avg. of 8), omnimodallity understanding (+7% avg. of 4), and audiovisual reasoning (+4%). It also advances long-form speech processing (reducing WER by 4.2%) and leads in low-level image processing and controllable generation across 5 metrics.
Authors Yunxin Li, Xinyu Chen, Shenyuan Jiang, Haoyuan Shi, Zhenyu Liu, Xuanyu Zhang, Nanhao Deng, Zhenran Xu, Yicheng Ma, Meishan Zhang, Baotian Hu, Min Zhang
Comment 47 pages,10 Figures, Project Website: https://idealistxy.github.io/Uni-MoE-v2.github.io/ Codes: https://github.com/HITsz-TMG/Uni-MoE
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
3
OmniStruct: Universal Text-to-Structure Generation across Diverse Schemas
Title CN OmniStruct：跨多样化模式的通用文本到结构生成
Keywords 文本到结构生成 · 大语言模型 · 信息抽取 · 合成数据 · 基准测试
Summary 本文提出了OmniStruct，一个评估大语言模型在多种文本到结构任务（如信息抽取、表格生成）中表现的基准，通过合成数据训练小型模型实现接近GPT-4o的效果。研究重点在于提升LLM在任意schema下的结构化输出能力，但完全基于纯文本输入，未结合视觉或布局信息。
Reason 该论文聚焦于从文本生成结构化输出（如信息抽取、表格生成、函数调用），构建了一个名为OmniStruct的基准，并通过合成数据训练小型模型以逼近GPT-4o性能。然而，其核心任务是纯文本输入下的结构化生成，未涉及文档图像、视觉布局或OCR相关内容，属于典型的‘平行下游应用’范式——即在特定任务上测试LLM结构化能力，而非解决文档理解中的视觉-语言对齐、排版复杂性或OCR幻觉等根本问题。尽管结构化生成对DIU有潜在辅助作用，但本文方法本身不处理图像模态，无法迁移用于文档图像理解的核心挑战。
Abstract The ability of Large Language Models (LLMs) to generate structured outputs that follow arbitrary schemas is crucial to a wide range of downstream tasks that require diverse structured representations of results such as information extraction, table generation, and function calling. While modern LLMs excel in generating unstructured responses in natural language, whether this advancement translates to a strong performance on text-to-structure tasks remains unclear. To bridge this gap, we first introduce OmniStruct, a comprehensive benchmark for assessing LLMs' capabilities on diverse text-to-structure tasks such as information extraction, table generation, and function calling. We build OmniStruct by identifying existing datasets across a wide range of tasks that are suitable for a structured answer format, and adapting them under a unified text-to-structure problem setting. To facilitate the development of efficient text-to-structure models, we collect high-quality training data via synthetic task generation. Without using any supervised data for OmniStruct tasks, our experiments demonstrate the possibility of fine-tuning much smaller models on synthetic data into universal structured generation models that can rival the performance of GPT-4o.
Authors James Y. Huang, Wenxuan Zhou, Nan Xu, Fei Wang, Qin Liu, Sheng Zhang, Hoifung Poon, Muhao Chen
Categories Artificial Intelligence, Computation and Language
3
MindCraft: How Concept Trees Take Shape In Deep Models
Title CN MindCraft：概念树如何在深度模型中成形
Keywords 可解释AI · 概念树 · 谱分解 · 表示学习 · 因果启发
Summary 本文提出MindCraft框架，利用谱分解和概念路径构建概念树，揭示大模型中概念的层次化形成过程。实验涵盖医疗诊断、物理推理等领域，展示了概念解耦与语义层级重建能力。该工作聚焦于模型内部表示的可解释性分析，而非具体任务性能提升。
Reason 该论文属于模型可解释性方向，提出通过谱分解构建‘概念树’以分析深度模型内部的概念分层结构。尽管其框架具有跨模态、跨任务的分析潜力，但并未针对文档图像理解（DIU）中的具体痛点（如OCR幻觉、复杂排版、细粒度定位等）进行设计或验证。其方法更偏向理论分析工具，缺乏向DIU场景迁移的直接路径，且未涉及多模态输入处理或视觉-语言对齐机制。因此属于无营养的平行应用类研究，不符合本领域方法论创新的要求。
Abstract Large-scale foundation models demonstrate strong performance across language, vision, and reasoning tasks. However, how they internally structure and stabilize concepts remains elusive. Inspired by causal inference, we introduce the MindCraft framework built upon Concept Trees. By applying spectral decomposition at each layer and linking principal directions into branching Concept Paths, Concept Trees reconstruct the hierarchical emergence of concepts, revealing exactly when they diverge from shared representations into linearly separable subspaces. Empirical evaluations across diverse scenarios across disciplines, including medical diagnosis, physics reasoning, and political decision-making, show that Concept Trees recover semantic hierarchies, disentangle latent concepts, and can be widely applied across multiple domains. The Concept Tree establishes a widely applicable and powerful framework that enables in-depth analysis of conceptual representations in deep models, marking a significant step forward in the foundation of interpretable AI.
Authors Bowei Tian, Yexiao He, Wanghao Ye, Ziyao Wang, Meng Liu, Ang Li
Categories Artificial Intelligence
3
ScriptViT: Vision Transformer-Based Personalized Handwriting Generation
Title CN ScriptViT：基于视觉Transformer的个性化手写生成
Keywords 手写生成 · Vision Transformer · 风格迁移 · 跨注意力 · Salient Stroke Attention
Summary 本文提出ScriptViT，一种基于Vision Transformer的手写风格编码器，通过多张参考图像学习书写者的全局风格特征，并利用跨注意力机制将风格信息融合到目标文本生成中。此外，引入显著笔画注意力分析（SSAA）增强模型可解释性。实验表明该方法在保持文本准确性的同时，提升了生成手写体的风格一致性。
Reason 该论文研究的是个性化手写体生成任务，属于文本风格化图像生成范畴。其核心问题是合成符合特定书写者风格的手写内容，而非文档理解或视觉问答（DIU/DocVQA）。尽管使用了Vision Transformer和跨注意力机制来建模长距离笔画依赖，但目标是生成而非理解文档内容。该工作属于‘平行应用’中的字体生成方向，未涉及OCR去幻觉、复杂版面分析、细粒度定位或文档推理等DIU核心痛点，无法迁移为提升文档理解性能的底层方法。
Abstract Styled handwriting generation aims to synthesize handwritten text that looks both realistic and aligned with a specific writer's style. While recent approaches involving GAN, transformer and diffusion-based models have made progress, they often struggle to capture the full spectrum of writer-specific attributes, particularly global stylistic patterns that span long-range spatial dependencies. As a result, capturing subtle writer-specific traits such as consistent slant, curvature or stroke pressure, while keeping the generated text accurate is still an open problem. In this work, we present a unified framework designed to address these limitations. We introduce a Vision Transformer-based style encoder that learns global stylistic patterns from multiple reference images, allowing the model to better represent long-range structural characteristics of handwriting. We then integrate these style cues with the target text using a cross-attention mechanism, enabling the system to produce handwritten images that more faithfully reflect the intended style. To make the process more interpretable, we utilize Salient Stroke Attention Analysis (SSAA), which reveals the stroke-level features the model focuses on during style transfer. Together, these components lead to handwriting synthesis that is not only more stylistically coherent, but also easier to understand and analyze.
Authors Sajjan Acharya, Rajendra Baskota
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Spatial Knowledge Graph-Guided Multimodal Synthesis IEEE/ACM Transactions on Audio, Speech and Language Processing
Title CN 空间知识图谱引导的多模态合成
Keywords 多模态合成 · 空间知识图谱 · 数据生成 · MLLM增强 · 空间推理
Summary 本文提出SKG2DATA，一种由空间知识图谱（SKG）引导的多模态数据合成框架。通过自动构建包含方向和距离关系的SKG，指导扩散模型生成符合空间常识的图像，并由MLLM生成对应文本描述。实验表明，使用该方法合成的数据能显著提升MLLM的空间推理能力，但可能轻微削弱其通用性能。
Reason 该论文属于典型的‘平行应用’类型：其核心是利用空间知识图谱生成具有空间一致性的多模态数据，以提升MLLM的空间感知能力。虽然方法本身有一定创新性，但其目标场景是通用视觉-语言模型的训练数据合成，并未针对文档图像理解中的具体痛点（如OCR幻觉、复杂版式、密集文本等）进行建模或验证。所提出的方法难以直接迁移至文档领域，例如文档中的逻辑结构、表格嵌套、阅读顺序等问题无法通过方向与距离类的空间关系有效建模。因此属于对DIU无营养的外围工作。
Abstract Recent advances in Multimodal Large Language Models (MLLMs) have significantly enhanced their capabilities; however, their spatial perception abilities remain a notable limitation. To address this challenge, multimodal data synthesis offers a promising solution. Yet, ensuring that synthesized data adhere to spatial common sense is a non-trivial task. Our approach addresses this critical gap by providing a systematic framework for generating spatially coherent data. In this work, we introduce SKG2DATA, a novel multimodal synthesis approach guided by spatial knowledge graphs, grounded in the concept of knowledge-to-data generation. SKG2DATA employs an automated pipeline for constructing Spatial Knowledge Graph (SKG) that effectively captures human-like spatial cognition, including directional and distance relationships. These structured representations then serve as precise guidance for our integrated synthesis pipeline, where a diffusion model generates spatially-consistent images while a MLLM produces corresponding textual descriptions. The automated construction of SKG enables scalable generation of diverse yet realistic spatial configurations, overcoming the limitations of manual data collection and annotation. Extensive experiments demonstrate that data synthesized from diverse types of spatial knowledge, including direction and distance, enhance the spatial perception and reasoning abilities of MLLMs markedly, albeit with a slight cost to their general capabilities. We hope that the idea of knowledge-based data synthesis can advance the development of spatial intelligence. Code is available at https://github.com/zjunlp/Knowledge2Data.
Authors Yida Xue, Zhen Bi, Jinnan Yang, Jungang Lou, Kehai Chen, Min Zhang, Huajun Chen, Ningyu Zhang
Comment IEEE/ACM Transactions on Audio, Speech and Language Processing
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
3
C3Po: Cross-View Cross-Modality Correspondence by Pointmap Prediction NeurIPS 2025
Title CN C3Po：通过点图预测实现跨视角跨模态对应
Keywords 跨模态对应 · 几何推理 · 三维重建 · 点图预测 · 视觉定位
Summary 本文提出C3Po方法及C3数据集，旨在建立地面照片与建筑平面图之间的像素级对应关系。通过从互联网图片集合中进行三维重建，并手动将其注册到收集的平面图上，构建了包含9万对数据的大规模数据集。实验表明现有模型在此任务上表现不佳，而基于新数据训练可显著提升性能。研究推动了跨模态几何推理的发展，但应用场景局限于三维视觉与地理对齐任务。
Reason 该论文聚焦于地面照片与平面图之间的跨模态几何对应问题，属于特定场景下的视觉几何理解任务。虽然提出了新的数据集C3和在跨模态对应上的改进，但其核心任务是建筑或地理场景的重建与对齐，与文档图像理解（如DocVQA、表格识别、文本布局分析等）的核心目标无直接关联。尽管存在‘跨模态’关键词，但其方法针对的是空间几何结构而非文档中的语义、逻辑或排版结构，无法迁移解决OCR幻觉、密集文本处理或复杂文档推理等问题。
Abstract Geometric models like DUSt3R have shown great advances in understanding the geometry of a scene from pairs of photos. However, they fail when the inputs are from vastly different viewpoints (e.g., aerial vs. ground) or modalities (e.g., photos vs. abstract drawings) compared to what was observed during training. This paper addresses a challenging version of this problem: predicting correspondences between ground-level photos and floor plans. Current datasets for joint photo--floor plan reasoning are limited, either lacking in varying modalities (VIGOR) or lacking in correspondences (WAFFLE). To address these limitations, we introduce a new dataset, C3, created by first reconstructing a number of scenes in 3D from Internet photo collections via structure-from-motion, then manually registering the reconstructions to floor plans gathered from the Internet, from which we can derive correspondence between images and floor plans. C3 contains 90K paired floor plans and photos across 597 scenes with 153M pixel-level correspondences and 85K camera poses. We find that state-of-the-art correspondence models struggle on this task. By training on our new data, we can improve on the best performing method by 34% in RMSE. We also identify open challenges in cross-modal geometric reasoning that our dataset aims to help address.
Authors Kuan Wei Huang, Brandon Li, Bharath Hariharan, Noah Snavely
Comment NeurIPS 2025
Categories Computer Vision and Pattern Recognition
3
SloMo-Fast: Slow-Momentum and Fast-Adaptive Teachers for Source-Free Continual Test-Time Adaptation
Title CN SloMo-Fast：用于无源持续测试时适应的慢动量与快自适应教师
Keywords 持续测试时适应 · 无源适应 · 域漂移 · 知识保留 · 双教师框架
Summary 本文提出SloMo-Fast，一种无源的双教师持续测试时适应框架，通过慢遗忘的Slow-Teacher和快适应的Fast-Teacher来应对连续变化的目标域，在保持历史知识的同时提升新域适应能力。同时构建了Cyclic-TTA基准模拟周期性域转移。实验表明其在多个CTTA设置下优于现有方法。
Reason 该论文研究的是持续测试时适应（CTTA）问题，属于模型在动态域上自我适应的技术，虽在视觉领域有一定价值，但其方法聚焦于通用图像分类场景下的域漂移问题，并未涉及文档结构、文本识别、多模态对齐或OCR相关挑战。且为典型的‘平行应用’类方法扩展，不提供针对文档图像理解（DIU）特有的如复杂排版、密集文本、视觉-语言对齐等痛点的解决方案，也无法直接迁移用于DocVQA或表格识别等任务。
Abstract Continual Test-Time Adaptation (CTTA) is crucial for deploying models in real-world applications with unseen, evolving target domains. Existing CTTA methods, however, often rely on source data or prototypes, limiting their applicability in privacy-sensitive and resource-constrained settings. Additionally, these methods suffer from long-term forgetting, which degrades performance on previously encountered domains as target domains shift. To address these challenges, we propose SloMo-Fast, a source-free, dual-teacher CTTA framework designed for enhanced adaptability and generalization. It includes two complementary teachers: the Slow-Teacher, which exhibits slow forgetting and retains long-term knowledge of previously encountered domains to ensure robust generalization, and the Fast-Teacher rapidly adapts to new domains while accumulating and integrating knowledge across them. This framework preserves knowledge of past domains and adapts efficiently to new ones. We also introduce Cyclic Test-Time Adaptation (Cyclic-TTA), a novel CTTA benchmark that simulates recurring domain shifts. Our extensive experiments demonstrate that SloMo-Fast consistently outperforms state-of-the-art methods across Cyclic-TTA, as well as ten other CTTA settings, highlighting its ability to both adapt and generalize across evolving and revisited domains.
Authors Md Akil Raihan Iftee, Mir Sazzat Hossain, Rakibul Hasan Rajib, Tariq Iqbal, Md Mofijul Islam, M Ashraful Amin, Amin Ahsan Ali, AKM Mahbubur Rahman
Comment 38 pages, 38 tables, 16 figures
Categories Computer Vision and Pattern Recognition
3
SineProject: Machine Unlearning for Stable Vision Language Alignment
Title CN SineProject：用于稳定视觉语言对齐的机器遗忘
Keywords 机器遗忘 · 视觉语言模型 · 安全对齐 · 投影网络 · 雅可比条件
Summary 本文提出SineProject，旨在解决多模态大模型在执行知识遗忘时破坏视觉语言对齐的问题。通过在冻结的投影网络中引入正弦调制的可训练参数，改善优化过程中的雅可比矩阵条件，从而稳定跨模态嵌入空间。实验表明该方法能在完全遗忘目标信息的同时显著减少对良性查询的误拒，适用于LLaVA等主流MLLM的安全与隐私维护。
Reason 该论文研究的是多模态大模型中的知识遗忘问题，核心目标是让模型安全地忘记特定信息（如隐私或有害内容），同时避免对良性查询产生过度拒绝。虽然涉及VLM，但其动机和方法聚焦于模型安全与隐私合规，属于纯NLP安全对齐范畴，并未针对文档图像理解中的OCR幻觉、复杂排版、细粒度定位等核心痛点提供可迁移的技术路径。且‘machine unlearning’在DIU场景中缺乏直接应用价值，属于无关领域的平行技术。
Abstract Multimodal Large Language Models (MLLMs) increasingly need to forget specific knowledge such as unsafe or private information without requiring full retraining. However, existing unlearning methods often disrupt vision language alignment, causing models to reject both harmful and benign queries. We trace this failure to the projector network during unlearning, its Jacobian becomes severely illconditioned, leading to unstable optimization and drift in cross modal embeddings. We introduce SineProject, a simple method that augments the frozen projector with sinusoidally modulated trainable parameters, improving the Jacobian's spectral conditioning and stabilizing alignment throughout unlearning. Across standard safety and privacy unlearning benchmarks using LLaVA v1.5 7B and 13B, SineProject reduces benign query refusals while achieving complete forgetting of targeted information, yielding state of the art forget retain trade offs with negligible computational overhead.
Authors Arpit Garg, Hemanth Saratchandran, Simon Lucey
Comment In Submission
Categories Computer Vision and Pattern Recognition
3
Exploring Weak-to-Strong Generalization for CLIP-based Classification TMLR
Title CN 探索基于CLIP分类的弱到强泛化
Keywords CLIP · 弱到强泛化 · 类原型学习 · 弱监督 · 视觉-语言模型
Summary 本文提出一种名为类原型学习（CPL）的方法，用于在弱监督下提升CLIP模型的分类性能，通过学习更具代表性的类别原型来实现弱模型对强模型的监督。实验表明在预训练数据有限的情况下，该方法能带来稳定的性能提升。
Reason 该论文研究的是在视觉-语言模型中利用较弱模型监督较强模型的弱到强泛化问题，属于模型对齐与训练范式探索，其核心任务为通用图像分类，并未涉及文档结构、布局分析、文本识别或文档特有的复杂排版处理。尽管其弱监督思想具有一定新颖性，但无法直接迁移解决文档图像理解中的OCR幻觉、细粒度定位或长文档推理等关键痛点，且未针对文档模态特性设计方法，属于通用VLM训练策略探索，偏离DIU核心问题。
Abstract Aligning large-scale commercial models with user intent is crucial to preventing harmful outputs. Current methods rely on human supervision but become impractical as model complexity increases. When models surpass human knowledge, providing accurate feedback becomes challenging and inefficient. A novel solution proposed recently is using a weaker model to supervise a stronger model. This concept leverages the ability of weaker models to perform evaluations, thereby reducing the workload on human supervisors. Previous work has shown the effectiveness of weak-to-strong generalization in the context of language-only models. Extending this concept to vision-language models leverages these insights, adapting the proven benefits to a multi-modal context. In our study, we explore weak-to-strong generalization for CLIP-based classification. We propose a method, class prototype learning (CPL), which aims to enhance the classification capabilities of the CLIP model, by learning more representative prototypes for each category. Our findings indicate that, despite using a simple loss function under weak supervision, CPL yields robust improvements in targeted scenarios, particularly when pretraining is limited. Extensive experiments demonstrate that our approach is effective under these settings, achieving a 3.67% improvement over strong baseline methods.
Authors Jinhao Li, Sarah M. Erfani, Lei Feng, James Bailey, Feng Liu
Comment TMLR
Categories Computer Vision and Pattern Recognition
3
Less is More: Token-Efficient Video-QA via Adaptive Frame-Pruning and Semantic Graph Integration
Title CN 少即是多：通过自适应帧剪枝和语义图集成实现高效的视频问答
Keywords 视频问答 · token效率 · 帧剪枝 · 视觉冗余 · 多模态大模型
Summary 本文提出一种用于视频问答的高效多模态大模型推理框架，通过自适应帧剪枝（AFP）去除冗余帧，并结合轻量级语义图进行语义补偿，在显著降低输入token数的同时提升或保持性能。实验表明该方法可减少高达80%的token消耗，并优于使用更多帧的基线方法。
Reason 该论文聚焦于视频问答（Video-QA）中的token效率问题，提出通过帧剪枝和语义图补偿来减少视觉冗余。尽管其‘视觉回声’（visual echoes）概念与文档理解中的OCR幻觉在表层上有一定相似性，但其方法设计完全针对视频时序冗余，且依赖于帧间动态建模，难以迁移到文档图像的静态、高密度文本布局场景。此外，任务领域属于视频理解，不在文档图像理解（DIU）核心范畴内。
Abstract The practical application of Multimodal Large Language Models (MLLMs) to Video Question Answering (Video-QA) is severely hindered by the high token cost of processing numerous video frames. While keyframe selection is the dominant strategy to mitigate this, we identify that even state-of-the-art selectors produce prompts laden with significant temporal redundancy, a challenge unique to video that we term 'visual echoes'. This issue leads to context dilution and can paradoxically degrade performance. To address this dual challenge, we propose a novel refinement framework that synergistically combines Adaptive Frame-Pruning (AFP) with a lightweight text-based semantic graph. AFP intelligently prunes 'visual echoes' by adaptively clustering frames, while the semantic graph provides crucial, low-cost semantic compensation. Conducting extensive experiments on the LongVideoBench and VideoMME benchmarks against multiple state-of-the-art selectors, our approach demonstrates a drastic reduction in total input tokens by up to 80%. Crucially, by creating a concise, high-quality prompt, our framework not only enhances efficiency but also demonstrates a remarkable ability to robustify and improve the accuracy of upstream selectors, achieving results that are highly competitive with, and often superior to, baselines that use vastly more frames.
Authors Shaoguang Wang, Weiyu Guo, Ziyang Chen, Yijie Xu, Xuming Hu, Hui Xiong
Comment This manuscript is a preprint. 22 pages, 19 figures
Categories Computer Vision and Pattern Recognition
3
ConsistCompose: Unified Multimodal Layout Control for Image Composition
Title CN ConsistCompose：用于图像合成的统一多模态布局控制
Keywords 多模态生成 · 布局控制 · 图像合成 · 视觉语言模型 · 坐标嵌入
Summary 本文提出ConsistCompose，一个通过将布局坐标嵌入语言提示来实现多实例图像生成的统一多模态框架，并构建了包含340万样本的ConsistCompose3M数据集。该方法支持在交错图文输入下进行布局可控的生成，利用实例-坐标绑定提示和坐标感知的无分类器引导机制，在COCO-Position和MS-Bench上提升了空间生成精度。
Reason 该论文聚焦于多实例图像生成中的布局可控性问题，核心贡献是将布局坐标嵌入语言提示以实现文本/图像引导下的可控生成。尽管涉及‘layout’和‘grounding’，但其任务本质是视觉生成（image composition），目标是提升生成图像的空间准确性，而非文档理解或识别。与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本解析、复杂排版结构化等无直接关联。方法难以迁移到DocVQA或表格识别等分析型任务，属于生成式CV的平行应用，不具备对DIU的方法论启发价值。
Abstract Unified multimodal models that couple visual understanding with image generation have advanced rapidly, yet most systems still focus on visual grounding-aligning language with image regions-while their generative counterpart, linguistic-embedded layout-grounded generation (LELG) for layout-controllable multi-instance generation, remains underexplored and limits precise compositional control. We present ConsistCompose, a unified multimodal framework that embeds layout coordinates directly into language prompts, enabling layout-controlled multi-instance image generation from Interleaved Image-Text within a single generative interface. We further construct ConsistCompose3M, a 3.4M multi-instance generation dataset with layout and identity annotations (2.6M text-guided and 0.8M image-guided data pairs) that provides large-scale supervision for layout-conditioned generation. Within this framework, LELG is instantiated through instance-coordinate binding prompts and coordinate-aware classifier-free guidance, which translate linguistic layout cues into precise spatial control without task-specific branches. Experiments on COCO-Position and MS-Bench show that ConsistCompose substantially improves spatial accuracy over layout-controlled baselines while preserving identity fidelity and competitive general multimodal understanding, establishing a unified paradigm for layout-controllable multimodal image generation.
Authors Xuanke Shi, Boxuan Li, Xiaoyang Han, Zhongang Cai, Lei Yang, Dahua Lin, Quan Wang
Comment 22 pages, 17 figures
Categories Computer Vision and Pattern Recognition
3
From Tables to Signals: Revealing Spectral Adaptivity in TabPFN
Title CN 从表格到信号：揭示TabPFN中的谱适应性
Keywords 表格基础模型 · 谱分析 · 上下文学习 · 信号重建 · 位置编码
Summary 本文通过信号重建的视角分析TabPFN这一表格基础模型的归纳偏置，提出‘谱适应性’概念，发现其频率响应能力能随上下文样本数量自适应变化，并受位置编码调制。作者进一步展示了其在无需训练的情况下进行图像去噪的潜力，表明其可作为通用隐式信号模型。
Reason 该论文研究的是表格型基础模型（TabPFN）在信号重建视角下的频谱特性，属于纯表格数据建模与理论分析，不涉及文档图像理解中的视觉结构、布局、OCR或VLM相关技术。尽管‘表格’是文档理解的一个子任务，但本文完全未处理文档图像、多模态输入或视觉-语言联合建模，而是聚焦于数值型表格的隐式神经表示和频率分析，与DIU的核心问题（如排版理解、视觉文本识别、幻觉抑制等）无直接关联。其方法难以迁移到文档图像场景，且不属于VLM或视觉压缩等可借鉴的上游工具。
Abstract Task-agnostic tabular foundation models such as TabPFN have achieved impressive performance on tabular learning tasks, yet the origins of their inductive biases remain poorly understood. In this work, we study TabPFN through the lens of signal reconstruction and provide the first frequency-based analysis of its in-context learning behavior. We show that TabPFN possesses a broader effective frequency capacity than standard ReLU-MLPs, even without hyperparameter tuning. Moreover, unlike MLPs whose spectra evolve primarily over training epochs, we find that TabPFN's spectral capacity adapts directly to the number of samples provided in-context, a phenomenon we term Spectral Adaptivity. We further demonstrate that positional encoding modulates TabPFN's frequency response, mirroring classical results in implicit neural representations. Finally, we show that these properties enable TabPFN to perform training-free and hyperparameter-free image denoising, illustrating its potential as a task-agnostic implicit model. Our analysis provides new insight into the structure and inductive biases of tabular foundation models and highlights their promise for broader signal reconstruction tasks.
Authors Jianqiao Zheng, Cameron Gordon, Yiping Ji, Hemanth Saratchandran, Simon Lucey
Categories Computer Vision and Pattern Recognition
3
Vision Token Masking Alone Cannot Prevent PHI Leakage in Medical Document OCR: A Systematic Evaluation
Title CN 仅靠视觉令牌掩码无法防止医疗文档OCR中的PHI泄露：一项系统性评估
Keywords PHI泄露 · 视觉令牌掩码 · 医疗OCR · 隐私保护 · VLM安全
Summary 本文系统评估了在医疗文档OCR中使用视觉令牌掩码来防止受保护健康信息（PHI）泄露的效果，发现现有掩码策略只能有效抑制长文本类PHI（如姓名、地址），但对短结构化信息（如病历号、社保号）完全无效。研究表明，语言模型的上下文推断能力是导致泄露的根本原因，单纯扩大视觉掩码范围无济于事。最终提出需结合NLP后处理和解码器级微调的混合防御架构。
Reason 该论文聚焦于医疗场景下的隐私保护问题，核心是PHI（受保护健康信息）在VLM推理过程中的泄露控制，属于特定领域的下游应用安全研究。虽然使用了DeepSeek-OCR作为基础模型，但其方法本质是针对医疗OCR的隐私增强技术，并未提出可泛化的文档理解新架构、训练范式或视觉表示学习机制。其‘视觉掩码’策略被证明对结构化短标识符无效，且最终依赖NLP后处理实现效果提升，属于典型的垂直领域安全加固方案，不具备向通用DIU任务迁移的方法论价值。
Abstract Large vision-language models (VLMs) are increasingly deployed for optical character recognition (OCR) in healthcare settings, raising critical concerns about protected health information (PHI) exposure during document processing. This work presents the first systematic evaluation of inference-time vision token masking as a privacy-preserving mechanism for medical document OCR using DeepSeek-OCR. We introduce seven masking strategies (V3-V9) targeting different architectural layers (SAM encoder blocks, compression layers, dual vision encoders, projector fusion) and evaluate PHI reduction across HIPAA-defined categories using 100 synthetic medical billing statements (drawn from a corpus of 38,517 annotated documents) with perfect ground-truth annotations. All masking strategies converge to 42.9% PHI reduction, successfully suppressing long-form spatially-distributed identifiers (patient names, dates of birth, physical addresses at 100% effectiveness) while failing to prevent short structured identifiers (medical record numbers, social security numbers, email addresses, account numbers at 0% effectiveness). Ablation studies varying mask expansion radius (r=1,2,3) demonstrate that increased spatial coverage does not improve reduction beyond this ceiling, indicating that language model contextual inference - not insufficient visual masking - drives structured identifier leakage. A simulated hybrid architecture combining vision masking with NLP post-processing achieves 88.6% total PHI reduction (assuming 80% NLP accuracy on remaining identifiers). This negative result establishes boundaries for vision-only privacy interventions in VLMs, provides guidance distinguishing PHI types amenable to vision-level versus language-level redaction, and redirects future research toward decoder-level fine-tuning and hybrid defense-in-depth architectures for HIPAA-compliant medical document processing.
Authors Richard J. Young
Comment 24 pages, 11 figures, 2 tables
Categories Computer Vision and Pattern Recognition
3
Demystifying CLIP Data
Title CN 揭秘CLIP数据
Keywords CLIP · 数据构建 · 元数据调控 · 图文预训练 · 零样本分类
Summary 本文分析了CLIP成功背后的数据因素，提出了Metadata-Curated Language-Image Pre-training（MetaCLIP），通过引入基于概念的元数据分布控制来提升图文数据集的质量。在相同模型和训练设置下，MetaCLIP在ImageNet零样本分类等基准上优于原始CLIP数据，证明了数据构建策略的重要性。
Reason 该论文聚焦于通用图文对比学习中的数据构建问题，提出通过元数据调控来优化图像-文本对的筛选。尽管其数据治理思路具有一定启发性，但研究对象为通用视觉-语言模型（如CLIP）的训练数据机制，并未针对文档图像理解（DIU）特有的挑战（如OCR幻觉、复杂版式、细粒度文本定位等）进行设计或验证。方法难以直接迁移至文档领域，且无涉及文档结构、布局、表格或VIE任务的相关实验，属于通用VLM数据研究，与DIU核心痛点关联薄弱。
Abstract Contrastive Language-Image Pre-training (CLIP) is an approach that has advanced research and applications in computer vision, fueling modern recognition systems and generative models. We believe that the main ingredient to the success of CLIP is its data and not the model architecture or pre-training objective. However, CLIP only provides very limited information about its data and how it has been collected, leading to works that aim to reproduce CLIP's data by filtering with its model parameters. In this work, we intend to reveal CLIP's data curation approach and in our pursuit of making it open to the community introduce Metadata-Curated Language-Image Pre-training (MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's concepts) and yields a balanced subset over the metadata distribution. Our experimental study rigorously isolates the model and training settings, concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M image-text data pairs outperforms CLIP's data on multiple standard benchmarks. In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy, surpassing CLIP's 68.3% on ViT-B models. Scaling to 1B data, while maintaining the same training budget, attains 72.4%. Our observations hold across various model sizes, exemplified by ViT-H achieving 80.5%, without any bells-and-whistles. Curation code and training data distribution on metadata is made available at https://github.com/facebookresearch/MetaCLIP.
Authors Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, Christoph Feichtenhofer
Comment 17 pages. arXiv admin note: text overlap with arXiv:2103.00020 by other authors
Categories Computer Vision and Pattern Recognition, Computation and Language
3
Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval ACL 2024
Title CN Llama2Vec：用于稠密检索的大型语言模型无监督适配
Keywords 稠密检索 · 语言模型适配 · 无监督学习 · 文本嵌入 · LLM
Summary 本文提出Llama2Vec，通过两种自编码预训练任务（EBAE和EBAR）对LLaMA-2进行无监督适配，以提升其在稠密检索任务中的表现。方法在MSMARCO和BEIR等基准上达到SOTA，但完全聚焦于纯文本语义匹配，不涉及图像或文档模态。
Reason 该论文研究如何将LLM无监督地适配到稠密检索任务中，核心贡献是针对文本检索的嵌入表示优化。虽然涉及LLM的语义理解能力，但其方法完全基于纯文本序列，未涉及任何视觉或文档布局信息，也不解决OCR、多模态对齐、细粒度定位等文档图像理解的核心问题。属于典型的‘平行下游应用’——将现有LLM用于另一NLP子任务（检索），并无DIU相关的方法论创新。
Abstract Dense retrieval calls for discriminative embeddings to represent the semantic relationship between query and document. It may benefit from the using of large language models (LLMs), given LLMs' strong capability on semantic understanding. However, the LLMs are learned by auto-regression, whose working mechanism is completely different from representing whole text as one discriminative embedding. Thus, it is imperative to study how to adapt LLMs properly so that they can be effectively initialized as the backbone encoder for dense retrieval. In this paper, we propose a novel approach, called Llama2Vec, which performs unsupervised adaptation of LLM for its dense retrieval application. Llama2Vec consists of two pretext tasks: EBAE (Embedding-Based Auto-Encoding) and EBAR (Embedding-Based Auto-Regression), where the LLM is prompted to reconstruct the input sentence and predict the next sentence based on its text embeddings. Llama2Vec is simple, lightweight, but highly effective. It is used to adapt LLaMA-2-7B on the Wikipedia corpus. With a moderate steps of adaptation, it substantially improves the model's fine-tuned performances on a variety of dense retrieval benchmarks. Notably, it results in the new state-of-the-art performances on popular benchmarks, such as passage and document retrieval on MSMARCO, and zero-shot retrieval on BEIR. The model and source code will be made publicly available to facilitate the future research. Our model is available at https://github.com/FlagOpen/FlagEmbedding.
Authors Zheng Liu, Chaofan Li, Shitao Xiao, Yingxia Shao, Defu Lian
Comment ACL 2024
Categories Computation and Language
3
Conversations: Love Them, Hate Them, Steer Them
Title CN 对话：热爱它们，讨厌它们，引导它们
Keywords 激活工程 · 情感控制 · LLM干预 · 归因补丁 · 对话系统
Summary 本文提出一种基于归因补丁和激活差异向量的干预方法，用于调控LLaMA模型在对话中的情感表达，使其输出更具人类情感特征（如喜悦、信任）并增强第一人称使用频率。方法无需微调，具备可解释性，但在文档图像理解任务中缺乏直接迁移路径。
Reason 该论文研究通过激活工程（activation engineering）调控LLM在对话中情感表达的方法，属于纯语言模态下的情感控制技术。尽管‘intervention’机制本身具有一定方法论价值，但其目标是情感拟人化而非解决多模态文档理解中的核心问题（如OCR幻觉、细粒度定位、复杂排版建模等）。未见任何视觉输入处理或跨模态对齐设计，且应用场景完全脱离文档图像理解范畴，属于典型的非相关领域平行应用。
Abstract Large Language Models (LLMs) demonstrate increasing conversational fluency, yet instilling them with nuanced, human-like emotional expression remains a significant challenge. Current alignment techniques often address surface-level output or require extensive fine-tuning. This paper demonstrates that targeted activation engineering can steer LLaMA 3.1-8B to exhibit more human-like emotional nuances. We first employ attribution patching to identify causally influential components, to find a key intervention locus by observing activation patterns during diagnostic conversational tasks. We then derive emotional expression vectors from the difference in the activations generated by contrastive text pairs (positive vs. negative examples of target emotions). Applying these vectors to new conversational prompts significantly enhances emotional characteristics: steered responses show increased positive sentiment (e.g., joy, trust) and more frequent first-person pronoun usage, indicative of greater personal engagement. Our findings offer a precise and interpretable method for controlling specific emotional attributes in LLMs, contributing to developing more aligned and empathetic conversational AI.
Authors Niranjan Chebrolu, Gerard Christopher Yeo, Kokil Jaidka
Comment We have created a new arXiv submission with a more up to date version of this paper at arXiv:2511.12832
Categories Computation and Language
3
LoKI: Low-damage Knowledge Implanting of Large Language Models AAAI-26 Oral
Title CN LoKI：大语言模型的低损伤知识植入
Keywords 参数高效微调 · 灾难性遗忘 · 知识存储 · 大语言模型 · LoKI
Summary 本文提出LoKI，一种基于Transformer架构中知识存储机制理解的参数高效微调方法，通过低损伤方式植入任务特定知识，在保持预训练模型通用能力的同时实现良好的任务性能。实验表明其在防止灾难性遗忘方面优于现有PEFT方法。
Reason 该论文提出一种参数高效微调方法LoKI，旨在缓解微调过程中的灾难性遗忘问题，属于通用LLM微调技术。虽然其机制性理解可能对模型适应能力有启发，但未涉及视觉输入、多模态建模或文档结构理解，无法直接解决文档图像理解中的OCR幻觉、细粒度定位或复杂排版等核心痛点。属于纯NLP领域的微调方法研究，与DIU任务无实质关联。
Abstract Fine-tuning adapts pretrained models for specific tasks but poses the risk of catastrophic forgetting (CF), where critical knowledge from pretraining is overwritten. To address the issue of CF in a general-purpose framework, we propose Low-damage Knowledge Implanting (LoKI), a parameter-efficient fine-tuning (PEFT) technique that utilizes recent mechanistic understanding of how knowledge is stored in transformer architectures. We compare LoKI against state-of-the-art PEFT methods in two real-world fine-tuning scenarios. The results show that LoKI demonstrates significantly better preservation of general capabilities. At the same time, its task-specific performance is comparable to or even surpasses that of full parameter fine-tuning and these PEFT methods across various model architectures. Our work bridges the mechanistic insights of LLMs' knowledge storage with practical fine-tuning objectives, enabling an effective balance between task-specific adaptation and the retention of general-purpose capabilities.
Authors Runyu Wang, Peng Ping, Zhengyu Guo, Xiaoye Zhang, Quan Shi, Liting Zhou, Tianbo Ji
Comment AAAI-26 Oral
Categories Computation and Language
3
Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts
Title CN Nested-ReFT：通过离策略推理实现大语言模型微调的高效强化学习
Keywords 强化学习 · 参数高效微调 · 离策略学习 · 大语言模型 · 推理效率
Summary 本文提出Nested-ReFT，一种基于离策略强化学习的大语言模型微调框架，通过让目标模型的部分层作为行为模型生成候选回答，降低标准ReFT中的推理开销。结合动态层跳过和偏差缓解策略，在保持性能的同时提升了训练效率。实验集中在数学推理任务上。
Reason 该论文属于纯NLP方向的强化学习微调方法优化，聚焦于数学推理等文本推理任务中的训练效率问题。虽然其提出的‘离策略’和‘层跳跃’机制在理论上可能对推理效率有提升，但未涉及任何视觉或文档模态，且其方法依赖于文本生成与奖励建模框架，难以直接迁移至文档图像理解中的核心痛点（如OCR幻觉、细粒度定位、复杂排版建模）。无证据表明该方法能处理多模态输入或解决视觉-语言对齐问题，因此对DIU领域无直接贡献。
Abstract Advanced reasoning in LLMs on challenging domains like mathematical reasoning can be tackled using verifiable rewards based reinforced fine-tuning (ReFT). In standard ReFT frameworks, a behavior model generates multiple completions with answers per problem, for the answer to be then scored by a reward function. While such RL post-training methods demonstrate significant performance improvements across challenging reasoning domains, the computational cost of generating completions during training with multiple inference steps makes the training cost non-trivial. To address this, we draw inspiration from off-policy RL, and speculative decoding to introduce a novel ReFT framework, dubbed Nested-ReFT, where a subset of layers of the target model acts as the behavior model to generate off-policy completions during training. The behavior model configured with dynamic layer skipping per batch during training decreases the inference cost compared to the standard ReFT frameworks. Our theoretical analysis shows that Nested-ReFT yields unbiased gradient estimates with controlled variance. Our empirical analysis demonstrates improved computational efficiency measured as tokens/sec across multiple math reasoning benchmarks and model sizes. Additionally, we explore three variants of bias mitigation to minimize the off-policyness in the gradient updates that allows for maintaining performance that matches the baseline ReFT performance.
Authors Maxime Heuillet, Yufei Cui, Boxing Chen, Audrey Durand, Prasanna Parthasarathi
Categories Artificial Intelligence, Computation and Language
3
Human-Inspired Multi-Level Reinforcement Learning NeurIPS 2025
Title CN 受人类启发的多级强化学习
Keywords 强化学习 · 多级经验 · 策略优化 · 人类启发 · NeurIPS
Summary 本文提出一种受人类学习机制启发的多级强化学习方法，通过区分不同层级的经验（如成功、失败程度）来提取更丰富的策略优化信号。该方法在低层利用评分式RL推断内在奖励，在高层引入基于性能层级的方向性信息，并设计新的策略损失函数以加权惩罚策略分布相似性。整体框架旨在模拟人类对不同程度表现的学习能力。
Reason 该论文属于强化学习理论方法研究，提出了一种从多级经验中提取信息的新型RL框架，但其研究目标与文档图像理解（DIU）的核心任务如DocVQA、布局分析、OCR-free理解等无直接关联。尽管‘多级学习’思想在理论上可能对长文档分层推理有间接启发，但缺乏明确的技术迁移路径，且未针对视觉-语言或多模态输入进行设计，无法有效应对DIU中的关键痛点如幻觉、细粒度定位或高分辨率处理。
Abstract Reinforcement learning (RL), a common tool in decision making, learns control policies from various experiences based on the associated cumulative return/rewards without treating them differently. Humans, on the contrary, often learn to distinguish from discrete levels of performance and extract the underlying insights/information (beyond reward signals) towards their decision optimization. For instance, when learning to play tennis, a human player does not treat all unsuccessful attempts equally. Missing the ball completely signals a more severe mistake than hitting it out of bounds (although the cumulative rewards can be similar for both cases). Learning effectively from multi-level experiences is essential in human decision making. This motivates us to develop a novel multi-level RL method that learns from multi-level experiences via extracting multi-level information. At the low level of information extraction, we utilized the existing rating-based reinforcement learning to infer inherent reward signals that illustrate the value of states or state-action pairs accordingly. At the high level of information extraction, we propose to extract important directional information from different-level experiences so that policies can be updated towards desired deviation from these different levels of experiences. Specifically, we propose a new policy loss function that penalizes distribution similarities between the current policy and different-level experiences, and assigns different weights to the penalty terms based on the performance levels. Furthermore, the integration of the two levels towards multi-level RL guides the agent toward policy improvements that benefit both reward improvement and policy improvement, hence yielding a similar learning mechanism as humans.
Authors Mingkang Wu, Devin White, Vernon Lawhern, Nicholas R. Waytowich, Yongcan Cao
Comment Accepted to the Aligning Reinforcement Learning Experimentalists and Theorists Workshop at NeurIPS 2025
Categories Artificial Intelligence
3
Synthetic Data Generation and Differential Privacy using Tensor Networks' Matrix Product States (MPS)
Title CN 基于张量网络矩阵乘积态的合成数据生成与差分隐私
Keywords 合成数据 · 差分隐私 · 张量网络 · 矩阵乘积态 · 表格生成
Summary 本文提出一种基于张量网络（特别是矩阵乘积态MPS）生成高质量、隐私保护的合成表格数据的方法，结合差分隐私机制，在多个指标上优于CTGAN、VAE等基线模型。强调在保证数据保真度的同时实现可证明的隐私保障，适用于对数据安全要求高的领域。
Reason 该论文聚焦于表格数据的合成生成与隐私保护，属于数据生成领域的通用方法研究。尽管其技术可能间接用于构造DIU训练数据，但并未针对文档图像理解中的核心问题（如OCR幻觉、复杂排版、细粒度定位等）提出直接解决方案，也未涉及多模态模型或视觉-语言推理架构。属于平行应用层面的技术工具，且无明确迁移路径到文档理解任务，不符合本方向关注的‘底层方法创新’或‘VLM for Doc’主线。
Abstract Synthetic data generation is a key technique in modern artificial intelligence, addressing data scarcity, privacy constraints, and the need for diverse datasets in training robust models. In this work, we propose a method for generating privacy-preserving high-quality synthetic tabular data using Tensor Networks, specifically Matrix Product States (MPS). We benchmark the MPS-based generative model against state-of-the-art models such as CTGAN, VAE, and PrivBayes, focusing on both fidelity and privacy-preserving capabilities. To ensure differential privacy (DP), we integrate noise injection and gradient clipping during training, enabling privacy guarantees via Rényi Differential Privacy accounting. Across multiple metrics analyzing data fidelity and downstream machine learning task performance, our results show that MPS outperforms classical models, particularly under strict privacy constraints. This work highlights MPS as a promising tool for privacy-aware synthetic data generation. By combining the expressive power of tensor network representations with formal privacy mechanisms, the proposed approach offers an interpretable and scalable alternative for secure data sharing. Its structured design facilitates integration into sensitive domains where both data quality and confidentiality are critical.
Authors Alejandro Moreno R., Desale Fentaw, Samuel Palmer, Raúl Salles de Padua, Ninad Dixit, Samuel Mugel, Roman Orús, Manuel Radons, Josef Menter, Ali Abedi
Comment 10 pages
Categories Artificial Intelligence
3
Supervised Dynamic Dimension Reduction with Deep Neural Network
Title CN 基于深度神经网络的有监督动态降维
Keywords 降维 · 时间序列预测 · 因子模型 · 深度学习 · 主成分分析
Summary 本文提出一种名为SDDP的有监督动态主成分分析框架，通过引入目标变量和滞后观测来指导高维预测变量的降维过程，并利用时序神经网络生成具有预测性的加权特征，进而提升时间序列预测性能。该方法强调因子的可解释性和任务特异性，在多个真实数据集上表现出优于现有方法的预测精度。
Reason 该论文研究的是面向时间序列预测的高维变量降维问题，属于典型的计量经济学与预测建模交叉工作，核心任务为因子提取与预测模型构建。尽管使用了深度学习和动态加权机制，但其方法论目标与文档图像理解（DIU）中的OCR幻觉、复杂排版解析、细粒度视觉定位或长文档推理等核心痛点无直接关联。未涉及多模态建模、视觉表示学习、VLM架构改进或文档结构理解，无法迁移用于解决文档理解中的关键挑战。
Abstract This paper studies the problem of dimension reduction, tailored to improving time series forecasting with high-dimensional predictors. We propose a novel Supervised Deep Dynamic Principal component analysis (SDDP) framework that incorporates the target variable and lagged observations into the factor extraction process. Assisted by a temporal neural network, we construct target-aware predictors by scaling the original predictors in a supervised manner, with larger weights assigned to predictors with stronger forecasting power. A principal component analysis is then performed on the target-aware predictors to extract the estimated SDDP factors. This supervised factor extraction not only improves predictive accuracy in the downstream forecasting task but also yields more interpretable and target-specific latent factors. Building upon SDDP, we propose a factor-augmented nonlinear dynamic forecasting model that unifies a broad family of factor-model-based forecasting approaches. To further demonstrate the broader applicability of SDDP, we extend our studies to a more challenging scenario when the predictors are only partially observable. We validate the empirical performance of the proposed method on several real-world public datasets. The results show that our algorithm achieves notable improvements in forecasting accuracy compared to state-of-the-art methods.
Authors Zhanye Luo, Yuefeng Han, Xiufan Yu
Categories Artificial Intelligence
3
Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Models
Title CN 偏见是一个子空间，而非坐标：对视觉-语言模型后处理去偏的几何重思考
Keywords 视觉-语言模型 · 去偏 · 子空间投影 · 公平性 · 表示学习
Summary 本文指出当前基于坐标级替换的VLM去偏方法存在特征纠缠、跨数据集泛化差和去偏不彻底的问题，提出偏见应被视为分布在低维线性子空间中而非孤立坐标。为此，作者提出子空间投影去偏（SPD）方法，通过识别并移除可线性解码的偏见子空间，同时保留语义均值以维持语义保真度。实验表明该方法在零样本分类、文生图检索和生成任务上显著提升公平性，同时保持任务性能。
Reason 该论文聚焦于视觉-语言模型中的社会性偏见（如性别、种族等）去除，属于VLM公平性（fairness）研究范畴。其提出的方法SPD针对的是模型表示中与敏感属性相关的子空间进行投影消除。尽管技术上有几何解释的创新，但其核心问题与文档图像理解（DIU）的核心痛点——如OCR幻觉、复杂排版、细粒度定位、长文档推理等——无直接关联。该方法难以迁移用于解决文档理解中的结构识别或文本准确性问题，且不属于DIU任务本身或可复用的底层推理/架构工具。
Abstract Vision-Language Models (VLMs) have become indispensable for multimodal reasoning, yet their representations often encode and amplify demographic biases, resulting in biased associations and misaligned predictions in downstream tasks. Such behavior undermines fairness and distorts the intended alignment between vision and language. Recent post-hoc approaches attempt to mitigate bias by replacing the most attribute-correlated embedding coordinates with neutral values. However, our systematic analysis reveals three critical failures of this coordinate-wise approach: feature entanglement, poor cross-dataset generalization, and incomplete bias removal. We find that bias is not localized to a few coordinates but is instead distributed across a few linear subspaces. To address these limitations, we propose $\textbf{S}$ubspace $\textbf{P}$rojection $\textbf{D}$ebiasing ($\textbf{SPD}$), a geometrically principled framework that identifies and removes the entire subspace of linearly decodable bias while reinserting a neutral mean component to preserve semantic fidelity. Extensive experiments across zero-shot classification, text-to-image retrieval, and image generation validate the effectiveness of SPD: our method achieves more robust debiasing with an average improvement of $18.5\%$ across four fairness metrics, while maintaining minimal loss in task performance compared to the best debiasing baseline.
Authors Dachuan Zhao, Weiyue Li, Zhenda Shen, Yushu Qiu, Bowen Xu, Haoyu Chen, Yongchao Chen
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
3
Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning
Title CN 常被忽视的开胃菜：大语言模型、分布假说与意义
Keywords 分词 · 分布假说 · 语义表征 · 大语言模型 · 偏见
Summary 本文探讨了大语言模型中分词机制的作用，认为当前的分词方法（如BPE）在构建语义单元方面存在缺陷，并可能成为偏见注入的后门。作者主张应重新设计分词策略，使其更符合语言的分布特性并具备语义原始性。研究基于RoBERTa等纯文本模型分析token的分布演化，强调分词目标函数对模型认知的潜在影响。
Reason 该论文聚焦于纯文本大语言模型中的分词（tokenization）机制及其对语义表征和偏见传播的影响，属于基础NLP理论探讨。虽然其关于token作为语义基元的讨论具有一定哲学启发性，但完全未涉及视觉模态、文档结构或OCR相关问题，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。文中研究对象为RoBERTa等纯文本模型，不包含多模态架构或视觉token处理方法，无法迁移用于解决文档理解中的幻觉、高分辨率输入、细粒度定位等关键挑战。
Abstract Tokenization is a necessary component within the current architecture of many language mod-els, including the transformer-based large language models (LLMs) of Generative AI, yet its impact on the model's cognition is often overlooked. We argue that LLMs demonstrate that the Distributional Hypothesis (DH) is sufficient for reasonably human-like language performance (particularly with respect to inferential lexical competence), and that the emergence of human-meaningful linguistic units among tokens and current structural constraints motivate changes to existing, linguistically-agnostic tokenization techniques, particularly with respect to their roles as (1) vehicles for conveying salient distributional patterns from human language to the model and as (2) semantic primitives. We explore tokenizations from a BPE tokenizer; extant model vocabularies obtained from Hugging Face and tiktoken; and the information in exemplar token vectors as they move through the layers of a RoBERTa (large) model. Besides creating suboptimal semantic building blocks and obscuring the model's access to the necessary distributional patterns, we describe how tokens and pretraining can act as a backdoor for bias and other unwanted content, which current alignment practices may not remediate. Additionally, we relay evidence that the tokenization algorithm's objective function impacts the LLM's cognition, despite being arguably meaningfully insulated from the main system intelligence. Finally, we discuss implications for architectural choices, meaning construction, the primacy of language for thought, and LLM cognition. [First uploaded to arXiv in December, 2024.]
Authors Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer, Alejandro J. Ruiz, Calla Beauregard, Ashley Fehr, Mikaela Irene Fudolig, Bradford Demarest, Yoshi Meke Bird, Milo Z. Trujillo, Christopher M. Danforth, Peter Sheridan Dodds
Categories Artificial Intelligence, Computation and Language
3
StepFun-Formalizer: Unlocking the Autoformalization Potential of LLMs through Knowledge-Reasoning Fusion AAAI 2026
Title CN StepFun-Formalizer：通过知识与推理融合释放大语言模型的自动形式化潜力
Keywords 自动形式化 · 数学推理 · 大语言模型 · 知识融合 · 推理轨迹
Summary 本文提出StepFun-Formalizer，通过构建富含形式化知识的数据集和生成非正式到正式的推理路径，提升大模型在数学语句自动形式化任务上的表现。采用SFT和RLVR训练策略，在7B和32B模型上取得了SOTA结果。然而，该工作聚焦于纯文本数学推理，未涉及视觉或文档模态。
Reason 该论文研究的是数学语句的自动形式化（Autoformalization），即将自然语言数学问题转化为形式化语言，其核心任务属于数学推理与形式化验证领域。尽管使用了LLM和推理轨迹生成等技术，但其应用场景完全不涉及文档图像、视觉输入或任何多模态理解任务。没有OCR、无视觉内容处理、不涉及排版结构或视觉布局分析，与文档图像理解（DIU）的核心痛点如幻觉、密集文本、复杂版式、细粒度定位等毫无关联。属于典型的‘平行应用’类工作——将现有LLM方法应用于特定领域（数学形式化），并无可迁移至DIU的方法论创新。
Abstract Autoformalization aims to translate natural-language mathematical statements into a formal language. While LLMs have accelerated progress in this area, existing methods still suffer from low accuracy. We identify two key abilities for effective autoformalization: comprehensive mastery of formal-language domain knowledge, and reasoning capability of natural language problem understanding and informal-formal alignment. Without the former, a model cannot identify the correct formal objects; without the latter, it struggles to interpret real-world contexts and map them precisely into formal expressions. To address these gaps, we introduce ThinkingF, a data synthesis and training pipeline that improves both abilities. First, we construct two datasets: one by distilling and selecting large-scale examples rich in formal knowledge, and another by generating informal-to-formal reasoning trajectories guided by expert-designed templates. We then apply SFT and RLVR with these datasets to further fuse and refine the two abilities. The resulting 7B and 32B models exhibit both comprehensive formal knowledge and strong informal-to-formal reasoning. Notably, StepFun-Formalizer-32B achieves SOTA BEq@1 scores of 40.5% on FormalMATH-Lite and 26.7% on ProverBench, surpassing all prior general-purpose and specialized models.
Authors Yutong Wu, Di Huang, Ruosi Wan, Yue Peng, Shijie Shang, Chenrui Cao, Lei Qi, Rui Zhang, Zidong Du, Jie Yan, Xing Hu
Comment AAAI 2026 Oral. Extended version with full appendix, 25 pages, 17 figures
Categories Artificial Intelligence, Computation and Language
3
Escaping Optimization Stagnation: Taking Steps Beyond Task Arithmetic via Difference Vectors
Title CN 逃离优化停滞：通过差分向量超越任务算术
Keywords 模型编辑 · 任务算术 · 差分向量 · 优化停滞 · 参数高效微调
Summary 本文提出了一种基于差分向量的各向异性缩放迭代算法（DV-BASI），用于改进任务算术中的优化过程。通过利用训练历史中的参数移动方向构造差分向量，并将其作为定向扰动，实现无需额外模块的连续优化。该方法在多任务模型融合中表现出优于个体微调模型的性能，并可扩展为单任务微调方案。
Reason 该论文聚焦于模型编辑与参数空间中的任务算术操作，属于模型编辑和多任务融合的通用方法研究。尽管其提出的差分向量和DV-BASI算法在优化路径探索上有理论价值，但未涉及多模态输入、视觉表示或文档结构理解等DIU核心问题。方法本质上是针对权重空间的操作，难以直接迁移用于解决OCR幻觉、细粒度视觉定位或高分辨率文档处理等实际文档理解痛点，属于通用模型编辑技术，不具备对DIU领域的直接方法论推动。
Abstract Current methods for editing pre-trained models face significant challenges, primarily high computational costs and limited scalability. Task arithmetic has recently emerged as a promising solution, using simple arithmetic operations-addition and negation-based on task vectors which are the differences between fine-tuned and pre-trained model weights, to efficiently modify model behavior. However, the full potential of task arithmetic remains underexplored, primarily due to limited mechanisms for overcoming optimization stagnation. To address this challenge, we introduce the notion of difference vector, a generalized form of task vectors derived from the historical movements during optimization. Using difference vectors as directed perturbations, we propose the Difference Vector-based Anisotropic Scaling Iterative algorithm (DV-BASI) to enable a continuous optimization process for task arithmetic methods without relying on any additional modules or components. Notably, by leveraging escapability and directional advantages of difference vectors, the average performance on different tasks of the multi-task model merged by DV-BASI may even outperform models individually fine-tuned. Based on this observation, we extend the application of difference vectors to a feasible fine-tuning method for single-task models. On the practical side, DV-BASI allows expressive searching directions with few learnable parameters and forms a scalable framework. We also integrate DV-BASI with task arithmetic methods and advanced optimization techniques to achieve state-of-the-art performance on both supervised and unsupervised evaluation protocols.
Authors Jinping Wang, Zhiqiang Gao, Dinggen Zhang, Zhiwu Xie
Categories Artificial Intelligence
3
Agentic Large Language Models, a survey
Title CN 代理型大语言模型综述
Keywords 代理型大语言模型 · 综述 · 推理 · 多智能体系统 · 工具使用
Summary 这是一篇关于代理型大语言模型（Agentic LLMs）的综述性文章，从推理、行动和交互三个维度梳理了当前研究进展，并提出了未来研究方向。文章讨论了此类模型在医疗、物流、金融等领域的应用潜力，以及其在科学发现和社会行为模拟中的作用。然而，该工作并未提出任何新方法，且重点在于通用AI代理的应用而非文档理解相关技术。
Reason 该论文是一篇关于‘代理型LLM’的综述，属于对现有技术的总结与分类，并未提出任何新的方法或架构。其内容聚焦于将LLM作为智能体在医疗诊断、金融分析等领域的应用框架，属于典型的平行下游应用综述，未涉及文档图像理解（DIU）中的核心问题如OCR幻觉、复杂排版建模或视觉表征学习。尽管‘推理-行动-交互’框架可能启发Agent设计，但本文本身无具体技术创新，也无法直接迁移解决DIU任务中的细粒度定位或高分辨率处理等痛点。
Abstract Background: There is great interest in agentic LLMs, large language models that act as agents. Objectives: We review the growing body of work in this area and provide a research agenda. Methods: Agentic LLMs are LLMs that (1) reason, (2) act, and (3) interact. We organize the literature according to these three categories. Results: The research in the first category focuses on reasoning, reflection, and retrieval, aiming to improve decision making; the second category focuses on action models, robots, and tools, aiming for agents that act as useful assistants; the third category focuses on multi-agent systems, aiming for collaborative task solving and simulating interaction to study emergent social behavior. We find that works mutually benefit from results in other categories: retrieval enables tool use, reflection improves multi-agent collaboration, and reasoning benefits all categories. Conclusions: We discuss applications of agentic LLMs and provide an agenda for further research. Important applications are in medical diagnosis, logistics and financial market analysis. Meanwhile, self-reflective agents playing roles and interacting with one another augment the process of scientific research itself. Further, agentic LLMs provide a solution for the problem of LLMs running out of training data: inference-time behavior generates new training states, such that LLMs can keep learning without needing ever larger datasets. We note that there is risk associated with LLM assistants taking action in the real world-safety, liability and security are open problems-while agentic LLMs are also likely to benefit society.
Authors Aske Plaat, Max van Duijn, Niki van Stein, Mike Preuss, Peter van der Putten, Kees Joost Batenburg
Comment Website: https://askeplaat.github.io/agentic-llm-survey-site/
Categories Artificial Intelligence, Computation and Language
3
VITAL: Vision-Encoder-centered Pre-training for LMMs in Visual Quality Assessment
Title CN VITAL：面向视觉质量评估中LMM的以视觉编码器为中心的预训练
Keywords 视觉质量评估 · 多模态大模型 · 预训练 · 零样本迁移 · 高效微调
Summary 本文提出VITAL系列大型多模态模型，采用以视觉编码器为中心的生成式预训练框架，通过机器执行的标注审查范式构建了450万以上的视觉语言对，支持图像与视频质量评估的多任务学习，并实现了解码器的快速热启动，提升模型泛化性与可扩展性。
Reason 该论文聚焦于视觉质量评估（VQualA），属于图像/视频质量打分任务，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出以视觉编码器为中心的预训练和高效微调策略具有一定方法价值，但其数据构建范式和多任务设计均围绕‘质量评分’与‘感知失真解释’展开，难以迁移到解决OCR幻觉、密集文本识别、复杂排版理解等DIU关键痛点。属于特定领域的下游应用扩展，不具备对文档理解的方法论推动。
Abstract Developing a robust visual quality assessment (VQualA) large multi-modal model (LMM) requires achieving versatility, powerfulness, and transferability. However, existing VQualA LMMs typically focus on a single task and rely on full-parameter fine-tuning, which makes them prone to overfitting on specific modalities or task types, thereby limiting their generalization capacity and transferability. To address this, we propose a vision-encoder-centered generative pre-training pipeline and develop the VITAL-Series LMMs. (1) We adopt a machine-executed annotation-scrutiny paradigm, constructing over 4.5M vision-language (VL) pairs-the largest VQualA training dataset to date. (2) We employ a multi-task training workflow that simultaneously enhances the model's quantitative scoring precision and strengthens its capability for quality interpretation across both image and video modalities. (3) Building upon the vision encoder, we realize an efficient model zoo extension: the model zoo exhibits strong zero-shot performance, and each paired decoder requires only a swift warm-up using less than 1/1000 of the pre-training data to achieve performance comparable to the fully trained counterpart. Overall, our work lays a cornerstone for advancing toward the foundation LMM for VQualA.
Authors Ziheng Jia, Linhan Cao, Jinliang Han, Zicheng Zhang, Jiaying Qian, Jiarui Wang, Zijian Chen, Guangtao Zhai, Xiongkuo Min
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Measuring the Impact of Lexical Training Data Coverage on Hallucination Detection in Large Language Models
Title CN 衡量词汇训练数据覆盖范围对大语言模型幻觉检测的影响
Keywords 幻觉检测 · 训练数据覆盖 · 大语言模型 · n-gram统计 · 开放域问答
Summary 本文探讨了大语言模型预训练数据中词汇覆盖程度是否可用于检测生成结果的幻觉问题。作者通过构建RedPajama语料库上的后缀数组来统计提示和生成内容的n-gram出现频率，并发现词汇覆盖率与log概率结合时可轻微提升幻觉检测效果。研究局限于纯文本QA场景，未涉及视觉或多模态信息。
Reason 该论文研究的是纯文本大语言模型在开放域问答中的幻觉检测问题，核心方法是利用预训练语料中的n-gram覆盖率作为幻觉检测信号。虽然幻觉是文档图像理解（DIU）中的重要问题（如OCR或VLM产生幻觉），但本文完全基于文本模态，未涉及视觉输入、多模态对齐或文档布局等DIU核心要素。其方法依赖于大规模文本语料的精确匹配统计，难以直接迁移到视觉生成内容（如识别扫描文档中的文字）的幻觉检测中。属于通用NLP幻觉研究，不具备对DIU的方法论迁移价值。
Abstract Hallucination in large language models (LLMs) is a fundamental challenge, particularly in open-domain question answering. Prior work attempts to detect hallucination with model-internal signals such as token-level entropy or generation consistency, while the connection between pretraining data exposure and hallucination is underexplored. Existing studies show that LLMs underperform on long-tail knowledge, i.e., the accuracy of the generated answer drops for the ground-truth entities that are rare in pretraining. However, examining whether data coverage itself can serve as a detection signal is overlooked. We propose a complementary question: Does lexical training-data coverage of the question and/or generated answer provide additional signal for hallucination detection? To investigate this, we construct scalable suffix arrays over RedPajama's 1.3-trillion-token pretraining corpus to retrieve $n$-gram statistics for both prompts and model generations. We evaluate their effectiveness for hallucination detection across three QA benchmarks. Our observations show that while occurrence-based features are weak predictors when used alone, they yield modest gains when combined with log-probabilities, particularly on datasets with higher intrinsic model uncertainty. These findings suggest that lexical coverage features provide a complementary signal for hallucination detection. All code and suffix-array infrastructure are provided at https://github.com/WWWonderer/ostd.
Authors Shuo Zhang, Fabrizio Gotti, Fengran Mo, Jian-Yun Nie
Categories Artificial Intelligence, Computation and Language
3
Towards Efficient LLM-aware Heterogeneous Graph Learning
Title CN 面向高效的LLM感知异构图学习
Keywords 异构图 · 大语言模型 · 关系推理 · 链式思维 · 效率优化
Summary 本文提出ELLA框架，通过LLM对异构图中的多跳、多类型关系进行编码，并设计Hop-level Relation Graph Transformer降低计算复杂度，同时引入细粒度任务感知的CoT提示来弥合预训练与微调之间的语义鸿沟。实验表明该方法在多个异构图数据集上性能优越且支持13B参数LLM，实现最高4倍加速。
Reason 该论文研究的是异构图上的LLM增强表示学习，属于图神经网络与语言模型结合的通用方法，但其任务和数据模态均不涉及文档图像、视觉输入或文档理解中的核心问题（如OCR幻觉、排版建模、细粒度定位等）。尽管使用了Chain-of-Thought提示和效率优化，但整体框架是为纯文本图结构设计的，无法直接迁移至文档图像理解场景。属于无关领域的下游应用，命中负面清单中的‘无关领域’。
Abstract Heterogeneous graphs are widely present in real-world complex networks, where the diversity of node and relation types leads to complex and rich semantics. Efforts for modeling complex relation semantics in heterogeneous graphs are restricted by the limitations of predefined semantic dependencies and the scarcity of supervised signals. The advanced pre-training and fine-tuning paradigm leverages graph structure to provide rich self-supervised signals, but introduces semantic gaps between tasks. Large Language Models (LLMs) offer significant potential to address the semantic issues of relations and tasks in heterogeneous graphs through their strong reasoning capabilities in textual modality, but their incorporation into heterogeneous graphs is largely limited by computational complexity. Therefore, in this paper, we propose an Efficient LLM-Aware (ELLA) framework for heterogeneous graphs, addressing the above issues. To capture complex relation semantics, we propose an LLM-aware Relation Tokenizer that leverages LLM to encode multi-hop, multi-type relations. To reduce computational complexity, we further employ a Hop-level Relation Graph Transformer, which help reduces the complexity of LLM-aware relation reasoning from exponential to linear. To bridge semantic gaps between pre-training and fine-tuning tasks, we introduce the fine-grained task-aware textual Chain-of-Thought (CoT) prompts. Extensive experiments on four heterogeneous graphs show that our proposed ELLA outperforms state-of-the-art methods in the performance and efficiency. In particular, ELLA scales up to 13b-parameter LLMs and achieves up to a 4x speedup compared with existing LLM-based methods. Our code is publicly available at https://github.com/l-wd/ELLA.
Authors Wenda Li, Tongya Zheng, Shunyu Liu, Yu Wang, Kaixuan Chen, Hanyang Yuan, Bingde Hu, Zujie Ren, Mingli Song, Gang Chen
Categories Artificial Intelligence, Computation and Language
3
InfiniBench: Infinite Benchmarking for Visual Spatial Reasoning with Customizable Scene Complexity
Title CN InfiniBench：基于可定制场景复杂度的视觉空间推理无限评测
Keywords 视觉空间推理 · 基准生成 · 3D场景合成 · VLM评测 · 程序化生成
Summary 本文提出InfiniBench，一个可自动生成无限多样化3D场景视频的基准生成工具，支持对视觉语言模型的空间推理能力进行高度定制化的评估。通过LLM代理框架、基于簇的布局优化和任务感知相机轨迹设计，实现复杂且物理合理的场景建模。实验涵盖测量、视角理解和时空追踪等任务。
Reason 该论文聚焦于通用视觉空间推理的基准生成，核心贡献是构建高复杂度、可定制的3D场景视频用于评测VLM的空间理解能力。尽管其在空间推理方面有潜力，但其应用场景集中在3D布局、动态视角和物理合理性上，与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、表格结构解析或长文档逻辑推理无直接关联。所生成的3D场景和视频输入形式远离文档的静态、平面、符号化特性，难以迁移至文档领域的细粒度定位或去幻觉任务。
Abstract Modern vision-language models (VLMs) are expected to have abilities of spatial reasoning with diverse scene complexities, but evaluating such abilities is difficult due to the lack of benchmarks that are not only diverse and scalable but also fully customizable. Existing benchmarks offer limited customizability over the scene complexity and are incapable of isolating and analyzing specific VLM failure modes under distinct spatial conditions. To address this gap, instead of individually presenting benchmarks for different scene complexities, in this paper we present InfiniBench, a fully automated, customizable and user-friendly benchmark generator that can synthesize a theoretically infinite variety of 3D scenes with parameterized control on scene complexity. InfiniBench uniquely translates scene descriptions in natural language into photo-realistic videos with complex and physically plausible 3D layouts. This is achieved through three key innovations: 1) a LLM-based agentic framework that iteratively refines procedural scene constraints from scene descriptions; 2) a flexible cluster-based layout optimizer that generates dense and cluttered scenes previously intractable for procedural methods; and 3) a task-aware camera trajectory optimization method that renders scenes into videos with full object coverage as VLM input. Experiments demonstrate that InfiniBench outperforms state-of-the-art procedural and LLM-based 3D generation methods in prompt fidelity and physical plausibility, especially in high-complexity scenarios. We further showcased the usefulness of InfiniBench, by generating benchmarks for representative spatial reasoning tasks including measurement, perspective-taking and spatiotemporal tracking.
Authors Haoming Wang, Qiyao Xue, Wei Gao
Categories Computer Vision and Pattern Recognition
3
Matching-Based Few-Shot Semantic Segmentation Models Are Interpretable by Design
Title CN 基于匹配的少样本语义分割模型天生可解释
Keywords 少样本学习 · 可解释AI · 语义分割 · 归因图 · 匹配机制
Summary 本文提出了一种针对基于匹配的少样本语义分割（FSS）模型的可解释方法Affinity Explainer，通过分析支持集和支持集与查询图像之间的特征匹配关系，生成归因图以揭示模型决策依据。实验表明该方法优于传统适配的归因方法，并能提供结构化、符合模型架构的注意力模式。尽管为FSS领域首次引入系统性可解释框架，但其应用场景局限于自然图像的语义分割，与文档图像理解任务脱节。
Reason 该论文研究的是少样本语义分割（FSS）模型的可解释性问题，属于通用计算机视觉领域，且聚焦于自然图像的分割解释。虽然其提出的Affinity Explainer方法在FSS中具有创新性，但其任务设定、数据模态和文档图像理解（DIU）的核心痛点（如OCR幻觉、密集文本、复杂排版、长文档推理）无直接关联。文档图像理解中的可解释性需求通常围绕文本识别置信度、布局结构推理路径或跨模态对齐错误诊断展开，而该方法针对的是像素级视觉匹配机制，迁移到文档场景的适用性极低。因此属于无关领域的技术工作，不构成对DIU的方法论推动。
Abstract Few-Shot Semantic Segmentation (FSS) models achieve strong performance in segmenting novel classes with minimal labeled examples, yet their decision-making processes remain largely opaque. While explainable AI has advanced significantly in standard computer vision tasks, interpretability in FSS remains virtually unexplored despite its critical importance for understanding model behavior and guiding support set selection in data-scarce scenarios. This paper introduces the first dedicated method for interpreting matching-based FSS models by leveraging their inherent structural properties. Our Affinity Explainer approach extracts attribution maps that highlight which pixels in support images contribute most to query segmentation predictions, using matching scores computed between support and query features at multiple feature levels. We extend standard interpretability evaluation metrics to the FSS domain and propose additional metrics to better capture the practical utility of explanations in few-shot scenarios. Comprehensive experiments on FSS benchmark datasets, using different models, demonstrate that our Affinity Explainer significantly outperforms adapted standard attribution methods. Qualitative analysis reveals that our explanations provide structured, coherent attention patterns that align with model architectures and and enable effective model diagnosis. This work establishes the foundation for interpretable FSS research, enabling better model understanding and diagnostic for more reliable few-shot segmentation systems. The source code is publicly available at https://github.com/pasqualedem/AffinityExplainer.
Authors Pasquale De Marinis, Uzay Kaymak, Rogier Brussee, Gennaro Vessio, Giovanna Castellano
Categories Computer Vision and Pattern Recognition
3
AVERY: Adaptive VLM Split Computing through Embodied Self-Awareness for Efficient Disaster Response Systems
Title CN AVERY：通过具身自感知实现高效灾难响应系统的自适应VLM分拆计算
Keywords VLM · 分拆计算 · 边缘计算 · 自适应压缩 · 无人机
Summary 论文提出AVERY框架，通过认知启发的双流分拆计算（上下文流与洞察流）和自感知控制器，在资源受限的无人机平台上实现VLM的高效部署。在动态网络条件下，该方法优于静态配置，提升了准确率并显著降低能耗。然而，其创新点集中在系统级调度与压缩策略，而非文档理解所需的细粒度视觉建模或多步逻辑推理能力。
Reason 该论文聚焦于无人机在灾难响应场景下的视觉语言模型（VLM）部署问题，提出一种基于双流架构的分拆计算框架。尽管涉及VLM和高分辨率图像处理，但其核心是边缘计算与网络适应性的系统优化，应用场景高度特定于灾难救援中的通信受限环境。方法本身并未针对文档图像理解中的OCR幻觉、复杂版式建模或长文档推理等核心痛点提供可迁移的技术路径。属于将现有VLM技术应用于特定下游任务的‘Wrapper’型工作，缺乏对DIU领域的方法论贡献。
Abstract Unmanned Aerial Vehicles (UAVs) in disaster response require complex, queryable intelligence that on-board CNNs cannot provide. While Vision-Language Models (VLMs) offer this semantic reasoning, their high resource demands make on-device deployment infeasible, and naive cloud offloading fails under the low-bandwidth networks common in disaster zones. We present AVERY, a framework that enables VLM deployment through adaptive split computing. We advance the split computing paradigm beyond traditional depth-wise partitioning by introducing a functional, cognitive-inspired dual-stream split that separates the VLM into a high-frequency, low-resolution "context stream" for real-time awareness and a low-frequency, high-fidelity "insight stream" for deep analysis. A lightweight, self-aware on-board controller manages this architecture, monitoring network conditions and operator intent to dynamically select from pre-trained compression models, navigating the fundamental accuracy-throughput trade-off. Evaluated using the VLM LISA-7B across an edge-cloud scenario under fluctuating network conditions, AVERY consistently outperforms static configurations, achieving 11.2% higher accuracy than raw image compression and 93.98% lower energy consumption compared to full-edge execution, thereby enhancing mission efficiency and enabling real-time, queryable intelligence on resource-constrained platforms in dynamic environments.
Authors Rajat Bhattacharjya, Sing-Yao Wu, Hyunwoo Oh, Chaewon Nam, Suyeon Koo, Mohsen Imani, Elaheh Bozorgzadeh, Nikil Dutt
Comment 8 pages, 5 figures. Paper is currently under review. Authors' version posted for personal use and not for redistribution
Categories Computer Vision and Pattern Recognition
3
Resilient Contrastive Pre-training under Non-Stationary Drift
Title CN 非平稳漂移下的鲁棒对比预训练
Keywords 对比学习 · 概念漂移 · 因果干预 · 表征学习 · 非平稳环境
Summary 本文针对动态数据流中存在的概念漂移问题，指出传统对比学习在非平稳环境下的不稳定性，并构建结构因果模型分析漂移如何作为混淆因子影响表征学习。基于此，提出Resilient Contrastive Pre-training（RCP），通过因果干预设计新的优化目标以减轻分布变化引起的偏差。实验表明该方法在多种下游任务中能提升表示的鲁棒性和泛化能力。
Reason 该论文研究的是在动态数据流和概念漂移环境下进行对比预训练的鲁棒性问题，核心贡献是提出一种基于因果干预的方法RCP来缓解分布变化带来的表征偏差。虽然引入了因果干预这一方法论，但其应用场景为通用视觉或数据流学习中的非平稳性问题，并未涉及文档图像理解（DIU）的核心挑战如OCR幻觉、复杂版式建模、细粒度文本定位或长文档推理。所提方法缺乏与多模态语言模型（VLM）处理文档内容的直接关联，且无证据表明其可迁移至解决文档视觉问答（DocVQA）、表格识别或布局分析等具体任务。属于通用表示学习方向的工作，对DIU领域无直接方法论启发。
Abstract The remarkable success of large-scale contrastive pre-training has been largely driven by by vast yet static datasets. However, as the scaling paradigm evolves, this paradigm encounters a fundamental challenge when applied to dynamic data streams characterized by concept drift - unpredictable changes in the underlying data distribution. This paper aims to advance robust pre-training under such non-stationary environments. We begin by revealing that conventional contrastive pre-training methods are highly susceptible to concept drift, resulting in significant substantial bias and instability within the learned feature representations. To systematically analyze these effects, we develop a structural causal model that elucidates how drift acts as a confounder, distorting the learned representations. Based on these causal insights, we propose Resilient Contrastive Pre-training (RCP), a novel method that incorporates causal intervention. RCP formulates a causally-informed objective to mitigate drift-induced biases through targeted interventions. The method is designed for simple and scalable implementation and exhibits notable adaptability, promoting robust and autonomous pre-training on non-stationary data. Comprehensive experiments across various downstream tasks consistently demonstrate that RCP effectively alleviates the detrimental impact of concept drift, yielding more resilient and generalizable representations.
Authors Xiaoyu Yang, Jie Lu, En Yu, Wei Duan
Comment 17pages, 3 figures
Categories Computer Vision and Pattern Recognition
3
PromptMoE: Generalizable Zero-Shot Anomaly Detection via Visually-Guided Prompt Mixtures AAAI 2026
Title CN PromptMoE：通过视觉引导的提示混合实现可泛化的零样本异常检测
Keywords 零样本异常检测 · 视觉语言模型 · 提示学习 · MoE · CLIP
Summary 本文提出PromptMoE，用于零样本异常检测，通过构建一个专家提示池和视觉引导的Mixture-of-Experts机制，动态组合语义基元提示以增强对未见异常的泛化能力。在15个工业与医学数据集上验证了其优越性能。
Reason 该论文属于工业与医学图像异常检测领域，尽管使用了视觉-语言模型和MoE架构进行提示学习，但其核心任务与文档图像理解（DIU）无关。未涉及DocVQA、布局分析、表格识别或OCR-free等DIU核心任务，且提出的视觉引导MoE机制针对的是异常区域定位问题，难以迁移到文档理解中的OCR幻觉、复杂排版或长文本推理等痛点。属于特定领域的应用型工作，不符合DIU本题或可迁移上游方法的标准。
Abstract Zero-Shot Anomaly Detection (ZSAD) aims to identify and localize anomalous regions in images of unseen object classes. While recent methods based on vision-language models like CLIP show promise, their performance is constrained by existing prompt engineering strategies. Current approaches, whether relying on single fixed, learnable, or dense dynamic prompts, suffer from a representational bottleneck and are prone to overfitting on auxiliary data, failing to generalize to the complexity and diversity of unseen anomalies. To overcome these limitations, we propose $\mathtt{PromptMoE}$. Our core insight is that robust ZSAD requires a compositional approach to prompt learning. Instead of learning monolithic prompts, $\mathtt{PromptMoE}$ learns a pool of expert prompts, which serve as a basis set of composable semantic primitives, and a visually-guided Mixture-of-Experts (MoE) mechanism to dynamically combine them for each instance. Our framework materializes this concept through a Visually-Guided Mixture of Prompt (VGMoP) that employs an image-gated sparse MoE to aggregate diverse normal and abnormal expert state prompts, generating semantically rich textual representations with strong generalization. Extensive experiments across 15 datasets in industrial and medical domains demonstrate the effectiveness and state-of-the-art performance of $\mathtt{PromptMoE}$.
Authors Yuheng Shao, Lizhang Wang, Changhao Li, Peixian Chen, Qinyuan Liu
Comment 14 pages, 8 figures. Accepted to AAAI 2026
Categories Computer Vision and Pattern Recognition
3
Versatile Recompression-Aware Perceptual Image Super-Resolution
Title CN 面向多变重压缩的感知图像超分辨率
Keywords 图像超分辨率 · 感知恢复 · 重压缩感知 · 扩散模型 · 编解码器模拟
Summary 本文提出了一种考虑后续重压缩影响的感知超分辨率方法VRPSR，通过将压缩过程建模为条件文本到图像生成任务，并利用预训练扩散模型构建可泛化的编解码器模拟器，从而联合优化超分辨率与压缩鲁棒性。实验表明该方法在H.264/H.265/H.266压缩下能显著降低码率同时保持视觉质量。
Reason 该论文研究的是通用图像超分辨率在重压缩场景下的优化问题，核心关注点是视觉质量与码率的权衡，属于通用图像恢复（image restoration）领域。虽然其方法涉及图像重建和压缩鲁棒性，但并未针对文档图像特有的结构（如文本布局、表格、语义结构）进行建模，也不解决文档理解中的OCR幻觉、细粒度定位或复杂排版解析等核心痛点。其技术路线基于扩散模型模拟压缩过程，属于图像编解码优化范畴，难以直接迁移至DIU任务。因此属于无关领域的常规改进工作。
Abstract Perceptual image super-resolution (SR) methods restore degraded images and produce sharp outputs. In practice, those outputs are usually recompressed for storage and transmission. Ignoring recompression is suboptimal as the downstream codec might add additional artifacts to restored images. However, jointly optimizing SR and recompression is challenging, as the codecs are not differentiable and vary in configuration. In this paper, we present Versatile Recompression-Aware Perceptual Super-Resolution (VRPSR), which makes existing perceptual SR aware of versatile compression. First, we formulate compression as conditional text-to-image generation and utilize a pre-trained diffusion model to build a generalizable codec simulator. Next, we propose a set of training techniques tailored for perceptual SR, including optimizing the simulator using perceptual targets and adopting slightly compressed images as the training target. Empirically, our VRPSR saves more than 10\% bitrate based on Real-ESRGAN and S3Diff under H.264/H.265/H.266 compression. Besides, our VRPSR facilitates joint optimization of the SR and post-processing model after recompression.
Authors Mingwei He, Tongda Xu, Xingtong Ge, Ming Sun, Chao Zhou, Yan Wang
Categories Computer Vision and Pattern Recognition
3
MambaX: Image Super-Resolution with State Predictive Control
Title CN MambaX：基于状态预测控制的图像超分辨率
Keywords 图像超分辨率 · Mamba · 状态空间模型 · 多模态融合 · 非线性控制
Summary 本文提出MambaX，一种基于状态预测控制的非线性序列模型，用于图像超分辨率任务。通过动态学习状态空间模型中的非线性参数，结合跨模态状态控制和渐进式过渡学习，提升单图像及多模态图像的重建质量。实验表明其在光谱广义建模方面具有潜力。
Reason 该论文聚焦于图像超分辨率任务，尤其是多光谱和多模态融合场景下的重建优化，属于通用图像恢复领域。尽管其提出的MambaX架构引入了非线性状态预测控制和跨模态状态交互机制，但核心目标是提升分辨率而非解决文档图像理解中的语义解析、布局建模或OCR幻觉等问题。方法上虽有创新，但迁移到文档理解（如细粒度文本识别、表格结构解析）的路径不明确，且未涉及VLM或多模态语言模型架构改进。属于与DIU相关但无直接迁移价值的视觉重建工作。
Abstract Image super-resolution (SR) is a critical technology for overcoming the inherent hardware limitations of sensors. However, existing approaches mainly focus on directly enhancing the final resolution, often neglecting effective control over error propagation and accumulation during intermediate stages. Recently, Mamba has emerged as a promising approach that can represent the entire reconstruction process as a state sequence with multiple nodes, allowing for intermediate intervention. Nonetheless, its fixed linear mapper is limited by a narrow receptive field and restricted flexibility, which hampers its effectiveness in fine-grained images. To address this, we created a nonlinear state predictive control model \textbf{MambaX} that maps consecutive spectral bands into a latent state space and generalizes the SR task by dynamically learning the nonlinear state parameters of control equations. Compared to existing sequence models, MambaX 1) employs dynamic state predictive control learning to approximate the nonlinear differential coefficients of state-space models; 2) introduces a novel state cross-control paradigm for multimodal SR fusion; and 3) utilizes progressive transitional learning to mitigate heterogeneity caused by domain and modality shifts. Our evaluation demonstrates the superior performance of the dynamic spectrum-state representation model in both single-image SR and multimodal fusion-based SR tasks, highlighting its substantial potential to advance spectrally generalized modeling across arbitrary dimensions and modalities.
Authors Chenyu Li, Danfeng Hong, Bing Zhang, Zhaojie Pan, Naoto Yokoya, Jocelyn Chanussot
Categories Computer Vision and Pattern Recognition
3
Signal: Selective Interaction and Global-local Alignment for Multi-Modal Object Re-Identification AAAI2026
Title CN Signal：用于多模态物体重识别的Selective Interaction与全局-局部对齐
Keywords 多模态ReID · 特征对齐 · 选择性交互 · 跨模态检索 · 行人重识别
Summary 本文提出Signal框架用于多模态物体重识别，包含选择性交互模块（SIM）以提取关键图像块特征，并通过全局对齐模块（GAM）和局部对齐模块（LAM）实现多模态特征的一致性对齐。在RGBNT201、RGBNT100和MSVR310三个基准上验证了有效性。
Reason 该论文研究的是多模态物体重识别（Re-ID），属于行人重识别领域，尽管使用了多模态特征对齐和交互机制，但其任务目标与文档图像理解（DIU）如DocVQA、布局分析、表格识别等无直接关联。方法上虽有全局-局部对齐设计，但针对的是红外-可见光等模态下的行人匹配问题，背景干扰抑制与文档中的OCR幻觉、复杂排版建模无关。无法迁移解决DIU核心痛点，属于无关领域应用。
Abstract Multi-modal object Re-IDentification (ReID) is devoted to retrieving specific objects through the exploitation of complementary multi-modal image information. Existing methods mainly concentrate on the fusion of multi-modal features, yet neglecting the background interference. Besides, current multi-modal fusion methods often focus on aligning modality pairs but suffer from multi-modal consistency alignment. To address these issues, we propose a novel selective interaction and global-local alignment framework called Signal for multi-modal object ReID. Specifically, we first propose a Selective Interaction Module (SIM) to select important patch tokens with intra-modal and inter-modal information. These important patch tokens engage in the interaction with class tokens, thereby yielding more discriminative features. Then, we propose a Global Alignment Module (GAM) to simultaneously align multi-modal features by minimizing the volume of 3D polyhedra in the gramian space. Meanwhile, we propose a Local Alignment Module (LAM) to align local features in a shift-aware manner. With these modules, our proposed framework could extract more discriminative features for object ReID. Extensive experiments on three multi-modal object ReID benchmarks (i.e., RGBNT201, RGBNT100, MSVR310) validate the effectiveness of our method. The source code is available at https://github.com/010129/Signal.
Authors Yangyang Liu, Yuhao Wang, Pingping Zhang
Comment Accepted by AAAI2026. More modifications may be performed
Categories Computer Vision and Pattern Recognition
3
Agent-as-a-Graph: Knowledge Graph-Based Tool and Agent Retrieval for LLM Multi-Agent Systems
Title CN 图式智能体：基于知识图谱的工具与智能体检索用于大语言模型多智能体系统
Keywords 多智能体系统 · 知识图谱 · 检索增强生成 · 工具调用 · LLM
Summary 本文提出一种名为Agent-as-a-Graph的检索方法，通过将智能体及其工具建模为知识图谱中的节点和边，结合向量检索与类型加权的倒数排名融合（wRRF）策略，提升多智能体系统中工具与智能体的召回效果。实验表明其在LiveMCPBenchmark上显著优于现有方法。
Reason 该论文聚焦于多智能体系统的智能体与工具检索机制，属于典型的Agent架构在通用NLP任务中的应用，未涉及任何文档图像理解（DIU）任务如DocVQA、表格识别或布局分析。尽管其知识图谱增强的检索机制具有一定方法论价值，但该方法针对的是MCP工具服务器的调用场景，与文档理解中的OCR幻觉、复杂排版、细粒度定位等核心痛点无直接关联，迁移路径不清晰。
Abstract Recent advances in Large Language Model Multi-Agent Systems enable scalable orchestration and retrieval of specialized, parallelized subagents, each equipped with hundreds or thousands of Model Context Protocol (MCP) servers and tools. However, existing agent, MCP, and retrieval methods typically match queries against a single agent description, obscuring fine-grained tool capabilities of each agent, resulting in suboptimal agent selection. We introduce Agent-as-a-Graph retrieval, a knowledge graph retrieval augmented generation approach that represents both tools and their parent agents as nodes and edges in a knowledge graph. During retrieval, i) relevant agents and tool nodes are first retrieved through vector search, ii) we apply a type-specific weighted reciprocal rank fusion (wRRF) for reranking tools and agents, and iii) parent agents are traversed in the knowledge graph for the final set of agents. We evaluate Agent-as-a-Graph on the LiveMCPBenchmark, achieving 14.9% and 14.6% improvements in Recall@5 and nDCG@5 over prior state-of-the-art retrievers, and 2.4% improvements in wRRF optimizations.
Authors Faheem Nizar, Elias Lumer, Anmol Gulati, Pradeep Honaganahalli Basavaraju, Vamse Kumar Subbiah
Categories Computation and Language
3
Vector Arithmetic in Concept and Token Subspaces NeurIPS 2025 Mechanistic Interpretability Workshop
Title CN 概念与词元子空间中的向量运算
Keywords 向量算术 · 机制可解释性 · 概念诱导头 · 词元诱导头 · 子空间分析
Summary 本文提出通过概念诱导头和词元诱导头的注意力权重变换隐藏状态，从而在Llama-2模型中构建具有语义一致性的子空间，并在此子空间中实现高精度的向量类比运算（如‘Athens’ - ‘Greece’ + ‘China’ = ‘Beijing’）。实验表明该方法显著提升了最近邻匹配准确率。
Reason 该论文研究的是语言模型内部表示的可解释性，聚焦于概念诱导头和词元诱导头如何分离语义与表层信息，并在隐藏状态中实现类比运算。尽管其关于子空间分解和向量算术的分析具有一定理论价值，但工作完全基于纯文本LLM（Llama-2），未涉及视觉输入、多模态建模或文档结构理解。其方法迁移至文档图像理解（DIU）的路径不明确，无法直接解决OCR幻觉、细粒度定位或复杂排版建模等核心问题。属于机制可解释性领域的基础研究，但对DIU缺乏实际工具价值。
Abstract In order to predict the next token, LLMs must represent semantic and surface-level information about the current word. Previous work identified two types of attention heads that disentangle this information: (i) Concept induction heads, which copy word meanings, and (ii) Token induction heads, which copy literal token representations (Feucht et al., 2025). We show that these heads can be used to identify subspaces of model activations that exhibit coherent semantic structure in Llama-2-7b. Specifically, when we transform hidden states using the attention weights of concept heads, we are able to more accurately perform parallelogram arithmetic (Mikolov et al., 2013) on the resulting hidden states, e.g., showing that "Athens" - "Greece" + "China" = "Beijing". This transformation allows for much higher nearest-neighbor accuracy (80%) than direct use of raw hidden states (47%). Analogously, we show that token heads allow for transformations that reveal surface-level word information in hidden states, allowing for operations like "coding" - "code" + "dance" = "dancing".
Authors Sheridan Feucht, Byron Wallace, David Bau
Comment 9 pages, 6 figures. NeurIPS 2025 Mechanistic Interpretability Workshop
Categories Computation and Language
3
InTAct: Interval-based Task Activation Consolidation for Continual Learning
Title CN InTAct：基于区间任务激活整合的持续学习
Keywords 持续学习 · 表示漂移 · 激活约束 · 提示学习 · 神经网络稳定性
Summary 本文提出InTAct方法，用于解决持续学习中的表示漂移问题。该方法通过捕捉先前任务中神经元激活的特征区间，并在训练过程中约束共享层在此区间内的功能一致性，从而避免重要特征被覆盖。该方法无需冻结参数或存储旧数据，兼容现有基于提示的学习框架，在多个域增量基准上显著提升性能。
Reason 该论文研究的是持续学习中的表示漂移问题，属于模型训练过程中的稳定性-可塑性权衡问题，核心贡献在于通过约束神经元激活范围来缓解遗忘。尽管其方法具有一定的通用性，但并未涉及文档图像理解（DIU）的核心挑战，如OCR幻觉、复杂版面解析、细粒度定位或多模态对齐等。同时，该工作也不属于VLM架构改进、推理增强或视觉表示学习等可直接迁移至DIU的‘上游工具’。其应用场景集中在域增量学习（如DomainNet），与文档理解任务无直接关联。
Abstract Continual learning aims to enable neural networks to acquire new knowledge without forgetting previously learned information. While recent prompt-based methods perform strongly in class-incremental settings, they remain vulnerable under domain shifts, where the input distribution changes but the label space remains fixed. This exposes a persistent problem known as representation drift. Shared representations evolve in ways that overwrite previously useful features and cause forgetting even when prompts isolate task-specific parameters. To address this issue, we introduce InTAct, a method that preserves functional behavior in shared layers without freezing parameters or storing past data. InTAct captures the characteristic activation ranges associated with previously learned tasks and constrains updates to ensure the network remains consistent within these regions, while still allowing for flexible adaptation elsewhere. In doing so, InTAct stabilizes the functional role of important neurons rather than directly restricting parameter values. The approach is architecture-agnostic and integrates seamlessly into existing prompt-based continual learning frameworks. By regulating representation changes where past knowledge is encoded, InTAct achieves a principled balance between stability and plasticity. Across diverse domain-incremental benchmarks, including DomainNet and ImageNet-R, InTAct consistently reduces representation drift and improves performance, increasing Average Accuracy by up to 8 percentage points over state-of-the-art baselines.
Authors Patryk Krukowski, Jan Miksa, Piotr Helm, Jacek Tabor, Paweł Wawrzyński, Przemysław Spurek
Categories Artificial Intelligence
3
SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation
Title CN SMILE：一种用于问答评估的复合词法-语义度量
Keywords 问答评估 · 语义相似度 · 词法匹配 · BERTScore · 自动化评测
Summary 本文提出了SMILE，一种融合句子级语义、关键词级语义和精确词匹配的复合式问答评估指标。相比传统基于n-gram的方法或纯嵌入式方法（如BERTScore），SMILE在保持计算轻量的同时更贴近人类判断。实验覆盖文本、图像和视频问答场景，显示其与人工评分高度相关。
Reason 该论文提出了一种新的问答评估指标SMILE，旨在结合词法精确性和语义相关性来更好地衡量QA系统的输出质量。尽管其目标是改进VQA等任务的评估方式，但本质上属于‘评估方法’的改进，而非文档图像理解（DIU）中的核心任务（如DocVQA、布局分析、表格识别等）的技术创新。它没有解决OCR幻觉、复杂排版理解或长文档推理等DIU关键问题，也无法作为可迁移的底层方法（如视觉压缩、推理增强架构）直接提升DIU模型性能。因此属于平行应用层面的方法论，不具备对DIU的技术源头价值。
Abstract Traditional evaluation metrics for textual and visual question answering, like ROUGE, METEOR, and Exact Match (EM), focus heavily on n-gram based lexical similarity, often missing the deeper semantic understanding needed for accurate assessment. While measures like BERTScore and MoverScore leverage contextual embeddings to address this limitation, they lack flexibility in balancing sentence-level and keyword-level semantics and ignore lexical similarity, which remains important. Large Language Model (LLM) based evaluators, though powerful, come with drawbacks like high costs, bias, inconsistency, and hallucinations. To address these issues, we introduce SMILE: Semantic Metric Integrating Lexical Exactness, a novel approach that combines sentence-level semantic understanding with keyword-level semantic understanding and easy keyword matching. This composite method balances lexical precision and semantic relevance, offering a comprehensive evaluation. Extensive benchmarks across text, image, and video QA tasks show SMILE is highly correlated with human judgments and computationally lightweight, bridging the gap between lexical and semantic evaluation.
Authors Shrikant Kendre, Austin Xu, Honglu Zhou, Michael Ryoo, Shafiq Joty, Juan Carlos Niebles
Comment 23 pages, 6 tables, 9 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
3
DS-Span: Single-Phase Discriminative Subgraph Mining for Efficient Graph Embeddings
Title CN DS-Span：用于高效图嵌入的单阶段判别性子图挖掘
Keywords 图表示学习 · 子图挖掘 · 判别性模式 · 信息增益 · 高效算法
Summary 本文提出DS-Span，一种单阶段判别性子图挖掘框架，通过在一次搜索遍历中统一模式增长、剪枝和监督打分，提升图表示学习的效率与判别能力。引入覆盖率限制机制和信息增益引导选择，减少冗余并增强类别区分性。实验表明该方法能在更短时间内生成更紧凑且判别性强的子图特征。
Reason 该论文研究的是图表示学习中的判别性子图挖掘，属于纯图数据挖掘领域，与文档图像理解（DIU）或视觉-语言模型（VLM）无直接关联。尽管其方法强调效率和判别性特征提取，但未涉及任何视觉模态、多模态对齐、OCR、布局分析或文档推理等DIU核心问题。无法迁移用于解决OCR幻觉、复杂排版或长文档理解等痛点，属于无关领域的传统图挖掘工作。
Abstract Graph representation learning seeks to transform complex, high-dimensional graph structures into compact vector spaces that preserve both topology and semantics. Among the various strategies, subgraph-based methods provide an interpretable bridge between symbolic pattern discovery and continuous embedding learning. Yet, existing frequent or discriminative subgraph mining approaches often suffer from redundant multi-phase pipelines, high computational cost, and weak coupling between mined structures and their discriminative relevance. We propose DS-Span, a single-phase discriminative subgraph mining framework that unifies pattern growth, pruning, and supervision-driven scoring within one traversal of the search space. DS-Span introduces a coverage-capped eligibility mechanism that dynamically limits exploration once a graph is sufficiently represented, and an information-gain-guided selection that promotes subgraphs with strong class-separating ability while minimizing redundancy. The resulting subgraph set serves as an efficient, interpretable basis for downstream graph embedding and classification. Extensive experiments across benchmarks demonstrate that DS-Span generates more compact and discriminative subgraph features than prior multi-stage methods, achieving higher or comparable accuracy with significantly reduced runtime. These results highlight the potential of unified, single-phase discriminative mining as a foundation for scalable and interpretable graph representation learning.
Authors Yeamin Kaiser, Muhammed Tasnim Bin Anwar, Bholanath Das, Chowdhury Farhan Ahmed, Md. Tanvir Alam
Categories Artificial Intelligence
3
Sparse Mixture-of-Experts for Multi-Channel Imaging: Are All Channel Interactions Required? NeurIPS AI4Science Workshop 2025
Title CN 面向多通道成像的稀疏专家混合模型：是否需要所有通道交互？
Keywords Vision Transformer · Mixture-of-Experts · 多通道图像 · 效率优化 · 跨通道注意力
Summary 本文提出MoE-ViT，一种用于多通道图像的Vision Transformer架构，通过将每个通道视为一个‘专家’，并使用轻量级路由器选择每块区域中最相关的通道参与注意力计算，从而减少跨通道交互带来的计算开销。实验表明该方法在保持甚至提升性能的同时显著降低FLOPs，适用于细胞成像和卫星图像等高维通道数据。
Reason 该论文研究多通道图像（如细胞成像、遥感）中的ViT架构效率问题，提出用MoE机制稀疏化跨通道注意力。尽管其在效率优化上有一定创新，但应用场景与文档图像理解（DIU）差异显著：文档图像通常为单通道或三通道，且核心挑战在于文本结构、排版复杂性和OCR相关幻觉，而非多光谱或多模态通道融合。该方法难以迁移到DIU领域解决OCR幻觉、细粒度定位或长文档推理等关键痛点，属于特定领域的效率工程优化，不具备对文档理解的方法论启发。
Abstract Vision Transformers ($\text{ViTs}$) have become the backbone of vision foundation models, yet their optimization for multi-channel domains - such as cell painting or satellite imagery - remains underexplored. A key challenge in these domains is capturing interactions between channels, as each channel carries different information. While existing works have shown efficacy by treating each channel independently during tokenization, this approach naturally introduces a major computational bottleneck in the attention block - channel-wise comparisons leads to a quadratic growth in attention, resulting in excessive $\text{FLOPs}$ and high training cost. In this work, we shift focus from efficacy to the overlooked efficiency challenge in cross-channel attention and ask: "Is it necessary to model all channel interactions?". Inspired by the philosophy of Sparse Mixture-of-Experts ($\text{MoE}$), we propose MoE-ViT, a Mixture-of-Experts architecture for multi-channel images in $\text{ViTs}$, which treats each channel as an expert and employs a lightweight router to select only the most relevant experts per patch for attention. Proof-of-concept experiments on real-world datasets - JUMP-CP and So2Sat - demonstrate that $\text{MoE-ViT}$ achieves substantial efficiency gains without sacrificing, and in some cases enhancing, performance, making it a practical and attractive backbone for multi-channel imaging.
Authors Sukwon Yun, Heming Yao, Burkhard Hoeckendorf, David Richmond, Aviv Regev, Russell Littman
Comment This has been accepted at the NeurIPS AI4Science Workshop 2025
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Agentifying Agentic AI
Title CN 将智能体AI进行智能体化
Keywords Agentic AI · BDI架构 · 多智能体系统 · 智能体理论 · 治理模型
Summary 论文主张通过自主智能体与多智能体系统领域（AAMAS）的理论工具（如BDI架构、通信协议、机制设计等）来构建具备持续自主性、推理与交互能力的Agentic AI系统，强调将数据驱动方法与结构化推理模型结合，以实现透明、合作且可追责的智能体。然而全文无实验、无技术实现路径，停留在概念整合层面。
Reason 该论文属于人工智能理论层面的讨论，聚焦于智能体（Agent）的概念建模、认知架构与治理机制，引用BDI、多智能体协作等传统AAMAS框架。虽然‘agentic behavior’在当前VLM和推理系统中受到关注，但本文未提出可具体实现或迁移的技术方法（如推理结构、干预机制或文档处理流程），也未涉及任何视觉或多模态内容。其核心为哲学性与体系性思辨，无法直接用于解决文档图像理解中的OCR幻觉、复杂排版或细粒度定位等实际问题，属于平行理论应用，不具备对DIU的方法论贡献。
Abstract Agentic AI seeks to endow systems with sustained autonomy, reasoning, and interaction capabilities. To realize this vision, its assumptions about agency must be complemented by explicit models of cognition, cooperation, and governance. This paper argues that the conceptual tools developed within the Autonomous Agents and Multi-Agent Systems (AAMAS) community, such as BDI architectures, communication protocols, mechanism design, and institutional modelling, provide precisely such a foundation. By aligning adaptive, data-driven approaches with structured models of reasoning and coordination, we outline a path toward agentic systems that are not only capable and flexible, but also transparent, cooperative, and accountable. The result is a perspective on agency that bridges formal theory and practical autonomy.
Authors Virginia Dignum, Frank Dignum
Comment 10 pages; 1 figure
Categories Artificial Intelligence
3
Estimating Global Input Relevance and Enforcing Sparse Representations with a Scalable Spectral Neural Network Approach
Title CN 通过可扩展的谱神经网络方法估计全局输入相关性并强制稀疏表示
Keywords 特征重要性 · 谱方法 · 稀疏表示 · 神经网络优化 · 可解释性
Summary 本文提出一种基于谱重参数化的深度神经网络输入特征重要性评估方法，利用输入节点对应的特征值作为其相关性的代理指标，并可通过正则化特征值实现稀疏输入表示。该过程在训练中自动完成，无需额外计算。实验表明其在合成与真实数据上优于现有特征选择方法。
Reason 该论文研究的是通用深度神经网络中输入特征重要性的估计与稀疏表示，属于模型可解释性与正则化方法的范畴。尽管其技术可能在理论上适用于任何基于DNN的系统，但并未针对文档图像理解（DIU）中的具体任务（如DocVQA、布局分析、OCR等）进行设计或验证。其方法也未直接解决DIU的核心痛点，如高分辨率视觉处理、多模态对齐、视觉幻觉或细粒度定位。由于是通用机器学习方法且无明确迁移路径到文档理解场景，属于与DIU间接关联但难以有效迁移的工作。
Abstract In machine learning practice it is often useful to identify relevant input features. Isolating key input elements, ranked according their respective degree of relevance, can help to elaborate on the process of decision making. Here, we propose a novel method to estimate the relative importance of the input components for a Deep Neural Network. This is achieved by leveraging on a spectral re-parametrization of the optimization process. Eigenvalues associated to input nodes provide in fact a robust proxy to gauge the relevance of the supplied entry features. Notably, the spectral features ranking is performed automatically, as a byproduct of the network training, with no additional processing to be carried out. Moreover, by leveraging on the regularization of the eigenvalues, it is possible to enforce solutions making use of a minimum subset of the input components, increasing the explainability of the model and providing sparse input representations. The technique is compared to the most common methods in the literature and is successfully challenged against both synthetic and real data.
Authors Lorenzo Chicchi, Lorenzo Buffoni, Diego Febbe, Lorenzo Giambagli, Raffaele Marino, Duccio Fanelli
Categories Artificial Intelligence
3
MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignment
Title CN MOCHA：多模态对象感知的跨架构对齐
Keywords 知识蒸馏 · 个性化检测 · 多模态学习 · 轻量化模型 · 区域对齐
Summary 本文提出MOCHA框架，通过跨架构蒸馏将冻结的多模态大模型（VLM）中的区域级视觉-语言知识迁移到轻量级纯视觉检测器中，提升其在少样本个性化目标检测任务上的性能。该方法采用双目标损失函数，实现局部特征对齐与区域间关系一致性建模，在不修改教师模型且无需推理时输入文本的前提下，显著优于现有基线。
Reason 该论文属于个性化目标检测领域的知识蒸馏工作，核心是将VLM的区域级语义知识迁移到轻量级检测器中。尽管使用了VLM作为教师模型，但其任务目标（通用目标检测）和应用场景（few-shot object detection）与文档图像理解（DIU）的核心任务如文本识别、布局分析、表格理解等无直接关联。方法上虽涉及多模态对齐，但聚焦于物体实例而非文本结构或视觉-语言联合推理，无法有效迁移解决OCR幻觉、密集文本处理或复杂排版理解等DIU关键问题。
Abstract Personalized object detection aims to adapt a general-purpose detector to recognize user-specific instances from only a few examples. Lightweight models often struggle in this setting due to their weak semantic priors, while large vision-language models (VLMs) offer strong object-level understanding but are too computationally demanding for real-time or on-device applications. We introduce MOCHA (Multi-modal Objects-aware Cross-arcHitecture Alignment), a distillation framework that transfers multimodal region-level knowledge from a frozen VLM teacher into a lightweight vision-only detector. MOCHA extracts fused visual and textual teacher's embeddings and uses them to guide student training through a dual-objective loss that enforces accurate local alignment and global relational consistency across regions. This process enables efficient transfer of semantics without the need for teacher modifications or textual input at inference. MOCHA consistently outperforms prior baselines across four personalized detection benchmarks under strict few-shot regimes, yielding a +10.1 average improvement, with minimal inference cost.
Authors Elena Camuffo, Francesco Barbato, Mete Ozay, Simone Milani, Umberto Michieli
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
The Cooperative Network Architecture: Learning Structured Networks as Representation of Sensory Patterns Neural Computation
Title CN 协作网络架构：作为感官模式表示的结构化网络学习
Keywords 协作网络 · 结构化表示 · 无监督学习 · 视觉表征 · 神经网络架构
Summary 本文提出了协作网络架构（CNA），通过动态组合重叠的网络片段来构建对感官输入的结构化表示。这些网络片段可无监督学习，并灵活重组以应对新图案，展现出对噪声和变形的鲁棒性以及分布外泛化能力。作者认为该架构为不变性物体识别提供了新的研究基础。
Reason 该论文提出了一种基于神经元群动态组装的结构化网络模型（CNA），用于感官信号表示，强调无监督学习和对噪声、形变的鲁棒性。尽管其在视觉表征学习方面有一定理论价值，但并未针对文档图像理解中的具体任务（如DocVQA、表格识别、文本定位等）进行设计或验证。方法偏向通用视觉表征，缺乏与文档场景的结合，且未涉及OCR、多模态对齐、高分辨率处理或VLM集成等DIU核心痛点，迁移应用路径不明确。
Abstract We introduce the Cooperative Network Architecture (CNA), a model that represents sensory signals using structured, recurrently connected networks of neurons, termed "nets." Nets are dynamically assembled from overlapping net fragments, which are learned based on statistical regularities in sensory input. This architecture offers robustness to noise, deformation, and generalization to out-of-distribution data, addressing challenges in current vision systems from a novel perspective. We demonstrate that net fragments can be learned without supervision and flexibly recombined to encode novel patterns, enabling figure completion and resilience to noise. Our findings establish CNA as a promising paradigm for developing neural representations that integrate local feature processing with global structure formation, providing a foundation for future research on invariant object recognition.
Authors Pascal J. Sager, Jan M. Deriu, Benjamin F. Grewe, Thilo Stadelmann, Christoph von der Malsburg
Comment Accepted at Neural Computation
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Comprehensive Evaluation of Prototype Neural Networks
Title CN 原型神经网络的综合评估
Keywords 原型网络 · 可解释人工智能 · 模型评估 · 图像分类 · 开源工具
Summary 本文对多种主流原型神经网络（如ProtoPNet、ProtoPool、PIPNet）进行了深入分析，采用包括标准和新提出的多种指标来评估模型的可解释性。实验涵盖细粒度分类、非独立同分布数据和多标签分类等多个场景，并提供了开源库以支持指标的应用与扩展。
Reason 该论文聚焦于原型模型（如ProtoPNet等）在可解释人工智能中的应用，并提出新的评估指标。研究内容主要围绕图像分类任务中的可解释性分析，未涉及文档结构、文本识别、多模态理解或VLM相关技术。尽管可解释性在文档理解中可能有潜在价值，但该工作本身并未针对文档图像理解（DIU）任务设计方法，也不解决OCR幻觉、复杂排版或长文档推理等核心痛点，属于通用XAI方法的横向评估，无直接迁移价值。
Abstract Prototype models are an important method for explainable artificial intelligence (XAI) and interpretable machine learning. In this paper, we perform an in-depth analysis of a set of prominent prototype models including ProtoPNet, ProtoPool and PIPNet. For their assessment, we apply a comprehensive set of metrics. In addition to applying standard metrics from literature, we propose several new metrics to further complement the analysis of model interpretability. In our experimentation, we apply the set of prototype models on a diverse set of datasets including fine-grained classification, Non-IID settings and multi-label classification to further contrast the performance. Furthermore, we also provide our code as an open-source library (https://github.com/uos-sis/quanproto), which facilitates simple application of the metrics itself, as well as extensibility -- providing the option for easily adding new metrics and models.
Authors Philipp Schlinge, Steffen Meinert, Martin Atzmueller
Categories Artificial Intelligence
3
Synthetic Object Compositions for Scalable and Accurate Learning in Detection, Segmentation, and Grounding
Title CN 用于检测、分割和定位中可扩展且准确学习的合成物体组合
Keywords 合成数据 · 实例分割 · 视觉定位 · 数据增强 · 目标检测
Summary 本文提出一种名为Synthetic Object Compositions（SOC）的合成数据生成方法，通过以对象为中心的组合策略，结合3D几何布局增强与相机配置增强，生成高质量的合成图像及其精确掩码、边界框和指代表达。该方法在实例分割和视觉定位任务上表现优异，仅用10万张合成图像即可超越基于更大真实数据集训练的模型，并在低资源场景下显著提升性能。此外，SOC支持可控的数据构建，可用于细粒度属性区分等诊断任务。
Reason 该论文聚焦于通用视觉任务（如实例分割、视觉定位、目标检测），其合成数据方法虽然在提升标注效率和多样性方面有贡献，但并未针对文档图像理解中的核心挑战（如OCR幻觉、复杂版式建模、密集文本处理或长文档推理）进行设计。所提出的SOC方法虽可用于生成带标注的合成图像，但属于通用数据增强技术，迁移到文档场景需大量适配工作，且未体现对DIU特有结构（如表格、布局层次、语义连贯性）的支持。整体属于‘平行应用’类技术路线，缺乏面向文档的架构或训练创新。
Abstract Visual grouping -- operationalized through tasks such as instance segmentation, visual grounding, and object detection -- enables applications ranging from robotic perception to photo editing. These fundamental problems in computer vision are powered by large-scale, painstakingly annotated datasets. Despite their impact, these datasets are costly to build, biased in coverage, and difficult to scale. Synthetic datasets offer a promising alternative but struggle with flexibility, accuracy, and compositional diversity. We introduce Synthetic Object Compositions (SOC), an accurate and scalable data synthesis pipeline via a novel object-centric composition strategy. It composes high-quality synthetic object segments into new images using 3D geometric layout augmentation and camera configuration augmentation with generative harmonization and mask-area-weighted blending, yielding accurate and diverse masks, boxes, and referring expressions. Models trained on just 100K of our synthetic images outperform those trained on larger real datasets (GRIT 20M, V3Det 200K) and synthetic pipelines (Copy-Paste, X-Paste, SynGround, SegGen) by +24-36% -- achieving +10.9 AP on LVIS and +8.4 NAcc on gRefCOCO. Beyond the general open-vocabulary setup, SOC also enables controllable dataset construction for different use cases and boosts performance in both low-data and closed-vocabulary scenarios. Augmenting LVIS and COCO with synthetic object segments delivers strong performance across different real-data scales and yields even greater improvements under extremely limited real-data conditions, including +6.59 AP on a 1% COCO data setup. Furthermore, this controllability enables targeted data generation for intra-class referring, a diagnostic grounding task we propose that requires fine-grained attribute discrimination.
Authors Weikai Huang, Jieyu Zhang, Taoyang Jia, Chenhao Zheng, Ziqi Gao, Jae Sung Park, Winson Han, Ranjay Krishna
Comment Project website: https://github.com/weikaih04/Synthetic-Detection-Segmentation-Grounding-Data
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMs
Title CN 当偏见伪装成真相：虚假关联如何破坏大语言模型中的幻觉检测
Keywords 大语言模型 · 幻觉检测 · 虚假关联 · 偏见 · 置信度
Summary 本文揭示了大语言模型中一类由训练数据中的虚假关联（如姓氏与国籍的表面统计关系）引发的幻觉问题，这类幻觉具有高置信度、不受模型规模影响、且能逃避现有检测机制。作者通过合成实验和对主流LLM的评估表明，基于置信度和内部状态的检测方法在此类情况下失效，并从理论上解释了其根本原因。研究呼吁开发专门针对此类统计偏见导致幻觉的新检测范式。
Reason 该论文研究的是纯文本大语言模型中的幻觉问题，特别是由训练数据中虚假关联（spurious correlations）引发的幻觉，并指出当前基于置信度和内部状态探测的检测方法失效。虽然‘幻觉’是文档图像理解（DIU）中的重要问题（如OCR或VLM生成时的幻觉），但本文完全基于纯文本LLM设定，未涉及任何视觉输入、多模态建模或文档结构信息，其分析框架和实验均不包含图像或布局因素。因此，尽管主题相关，但其结论和方法难以直接迁移到DIU场景，尤其是无法解决OCR幻觉、视觉-文本对齐错误等核心问题。属于通用NLP幻觉研究，不符合DIU本题要求，也不提供可迁移的视觉或多模态层面的技术工具。
Abstract Despite substantial advances, large language models (LLMs) continue to exhibit hallucinations, generating plausible yet incorrect responses. In this paper, we highlight a critical yet previously underexplored class of hallucinations driven by spurious correlations -- superficial but statistically prominent associations between features (e.g., surnames) and attributes (e.g., nationality) present in the training data. We demonstrate that these spurious correlations induce hallucinations that are confidently generated, immune to model scaling, evade current detection methods, and persist even after refusal fine-tuning. Through systematically controlled synthetic experiments and empirical evaluations on state-of-the-art open-source and proprietary LLMs (including GPT-5), we show that existing hallucination detection methods, such as confidence-based filtering and inner-state probing, fundamentally fail in the presence of spurious correlations. Our theoretical analysis further elucidates why these statistical biases intrinsically undermine confidence-based detection techniques. Our findings thus emphasize the urgent need for new approaches explicitly designed to address hallucinations caused by spurious correlations.
Authors Shaowen Wang, Yiqi Dong, Ruinian Chang, Tansheng Zhu, Yuebo Sun, Kaifeng Lyu, Jian Li
Categories Artificial Intelligence, Computation and Language
3
Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation AAAI'26
Title CN 学习压缩：释放大语言模型在文本表示中的潜力
Keywords 文本表示 · 大语言模型 · 上下文压缩 · 预训练任务 · LLM2Comp
Summary 本文提出一种利用上下文压缩作为预训练任务的方法，以提升大语言模型在文本表示任务上的性能。通过生成紧凑的记忆token替代完整上下文，结合对比学习，模型在聚类、检索等下游任务中表现更优且训练数据需求更少。然而，该工作局限于纯文本领域，未涉及任何视觉或多模态组件。
Reason 该论文聚焦于通用文本表示的优化，提出通过上下文压缩作为预训练任务来改进LLM的表示能力。尽管其方法在文本编码上有一定创新，但完全基于纯文本模态，未涉及视觉或文档图像内容，也未探讨多模态对齐、OCR后处理或视觉-语言联合建模等文档理解核心问题。无法直接迁移用于解决DIU中的OCR幻觉、复杂排版解析或细粒度视觉定位等关键挑战，属于与文档图像理解无关的平行研究。
Abstract Text representation plays a critical role in tasks like clustering, retrieval, and other downstream applications. With the emergence of large language models (LLMs), there is increasing interest in harnessing their capabilities for this purpose. However, most of the LLMs are inherently causal and optimized for next-token prediction, making them suboptimal for producing holistic representations. To address this, recent studies introduced pretext tasks to adapt LLMs for text representation. Most of these tasks, however, rely on token-level prediction objectives, such as the masked next-token prediction (MNTP) used in LLM2Vec. In this work, we explore the untapped potential of context compression as a pretext task for unsupervised adaptation of LLMs. During compression pre-training, the model learns to generate compact memory tokens, which substitute the whole context for downstream sequence prediction. Experiments demonstrate that a well-designed compression objective can significantly enhance LLM-based text representations, outperforming models trained with token-level pretext tasks. Further improvements through contrastive learning produce a strong representation model (LLM2Comp) that outperforms contemporary LLM-based text encoders on a wide range of tasks while being more sample-efficient, requiring significantly less training data.
Authors Yeqin Zhang, Yizheng Zhao, Chen Hu, Binxing Jiao, Daxin Jiang, Ruihang Miao, Cam-Tu Nguyen
Comment Accepted by AAAI'26
Categories Artificial Intelligence, Computation and Language
3
MiniLLM: Knowledge Distillation of Large Language Models ICLR 2024
Title CN MiniLLM：大语言模型的知识蒸馏
Keywords 知识蒸馏 · 大语言模型 · 生成模型 · KL散度 · 模型压缩
Summary 本文提出了一种针对生成式大语言模型的知识蒸馏方法MiniLLM，通过采用反向KL散度和on-policy优化策略，提升小模型在指令遵循任务中的生成质量、校准性和长文本表现。实验覆盖多种规模的模型，展示了优于传统蒸馏方法的效果。
Reason 该论文研究的是通用大语言模型的知识蒸馏方法，聚焦于生成式语言模型的训练目标改进（如使用反向KL散度）和策略优化，属于典型的NLP领域模型压缩工作。虽然知识蒸馏在部署轻量级VLM中有潜在用途，但本文并未涉及多模态输入、视觉表征或文档结构建模，也没有针对OCR幻觉、细粒度定位或复杂排版等DIU核心问题提出解决方案。其方法迁移至文档理解场景的路径不明确，且属于‘平行技术应用’的上游基础方法，但并非为视觉-语言交互设计，对DIU缺乏直接创新价值。
Abstract Knowledge Distillation (KD) is a promising technique for reducing the high computational demand of large language models (LLMs). However, previous KD methods are primarily applied to white-box classification models or training small models to imitate black-box model APIs like ChatGPT. How to effectively distill the knowledge of white-box LLMs into small models is still under-explored, which becomes more important with the prosperity of open-source LLMs. In this work, we propose a KD approach that distills LLMs into smaller language models. We first replace the forward Kullback-Leibler divergence (KLD) objective in the standard KD approaches with reverse KLD, which is more suitable for KD on generative language models, to prevent the student model from overestimating the low-probability regions of the teacher distribution. Then, we derive an effective on-policy optimization approach to learn this objective. The student models are named MiniLLM. Extensive experiments in the instruction-following setting show that MiniLLM generates more precise responses with higher overall quality, lower exposure bias, better calibration, and higher long-text generation performance than the baselines. Our method is scalable for different model families with 120M to 13B parameters. Our code, data, and model checkpoints can be found in https://github.com/microsoft/LMOps/tree/main/minillm.
Authors Yuxian Gu, Li Dong, Furu Wei, Minlie Huang
Comment Published as a conference paper in ICLR 2024
Categories Artificial Intelligence, Computation and Language
3
Spanning Tree Autoregressive Visual Generation
Title CN 生成树自回归视觉生成
Keywords 自回归生成 · 图像序列建模 · 生成树 · 视觉生成 · 采样策略
Summary 本文提出Spanning Tree Autoregressive (STAR) 模型，通过在图像块构成的格点上采样均匀生成树，并利用广度优先遍历顺序构建序列化输入，以兼顾自回归模型在图像生成中的采样性能与推理时的序列灵活性。相比随机排列策略，STAR在保持生成质量的同时支持更灵活的编辑能力。
Reason 该论文提出一种基于生成树遍历顺序的自回归视觉生成方法，核心目标是图像生成中的序列建模灵活性与性能平衡，主要面向通用图像生成任务。其技术动机和实验集中在图像合成与编辑，未涉及文本识别、布局理解或文档结构建模。尽管其序列建模思想具有一定理论价值，但缺乏对文档图像中OCR错误缓解、细粒度文本定位或复杂排版建模的直接迁移路径，且未针对多模态语言-视觉对齐进行优化。属于通用视觉生成范畴，与文档图像理解的核心痛点（如幻觉、密集文本推理）关联薄弱。
Abstract We present Spanning Tree Autoregressive (STAR) modeling, which can incorporate prior knowledge of images, such as center bias and locality, to maintain sampling performance while also providing sufficiently flexible sequence orders to accommodate image editing at inference. Approaches that expose randomly permuted sequence orders to conventional autoregressive (AR) models in visual generation for bidirectional context either suffer from a decline in performance or compromise the flexibility in sequence order choice at inference. Instead, STAR utilizes traversal orders of uniform spanning trees sampled in a lattice defined by the positions of image patches. Traversal orders are obtained through breadth-first search, allowing us to efficiently construct a spanning tree whose traversal order ensures that the connected partial observation of the image appears as a prefix in the sequence through rejection sampling. Through the tailored yet structured randomized strategy compared to random permutation, STAR preserves the capability of postfix completion while maintaining sampling performance without any significant changes to the model architecture widely adopted in the language AR modeling.
Authors Sangkyu Lee, Changho Lee, Janghoon Han, Hosung Song, Tackgeun You, Hwasup Lim, Stanley Jungkyu Choi, Honglak Lee, Youngjae Yu
Comment Preprint; Under review
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning ICCV 2025
Title CN 对齐视觉与语言：用于增强LLM推理的无标注多模态知识图谱构建
Keywords 多模态知识图谱 · 视觉-语言对齐 · 无监督学习 · LLM增强 · 跨模态推理
Summary 本文提出VaLiK，一种无需人工标注的多模态知识图谱构建方法，通过级联VLM将图像特征转化为富含信息的文本描述，并设计跨模态相似性验证机制过滤语义噪声，从而提升LLM的多模态推理能力。实验显示其在多模态推理任务上优于现有方法。
Reason 该论文属于典型的‘平行下游应用’，其核心是构建多模态知识图谱（MMKG）以增强LLM的多模态推理能力，但并未聚焦文档图像理解中的具体任务（如DocVQA、表格识别、版面分析等）。虽然其提出的视觉特征对齐与跨模态验证机制有一定技术成分，但整体方法是通用VLM层面的改进，且目标场景宽泛，未针对文档特有的密集文本、复杂排版或OCR幻觉等问题设计解决方案。无法直接迁移为DIU的底层方法，属于在另一条赛道上的应用性工作。
Abstract Multimodal reasoning in Large Language Models (LLMs) struggles with incomplete knowledge and hallucination artifacts, challenges that textual Knowledge Graphs (KGs) only partially mitigate due to their modality isolation. While Multimodal Knowledge Graphs (MMKGs) promise enhanced cross-modal understanding, their practical construction is impeded by semantic narrowness of manual text annotations and inherent noise in visual-semantic entity linkages. In this paper, we propose Vision-align-to-Language integrated Knowledge Graph (VaLiK), a novel approach for constructing MMKGs that enhances LLMs reasoning through cross-modal information supplementation. Specifically, we cascade pre-trained Vision-Language Models (VLMs) to align image features with text, transforming them into descriptions that encapsulate image-specific information. Furthermore, we developed a cross-modal similarity verification mechanism to quantify semantic consistency, effectively filtering out noise introduced during feature alignment. Even without manually annotated image captions, the refined descriptions alone suffice to construct the MMKG. Compared to conventional MMKGs construction paradigms, our approach achieves substantial storage efficiency gains while maintaining direct entity-to-image linkage capability. Experimental results on multimodal reasoning tasks demonstrate that LLMs augmented with VaLiK outperform previous state-of-the-art models. Our code is published at https://github.com/Wings-Of-Disaster/VaLiK.
Authors Junming Liu, Siyuan Meng, Yanting Gao, Song Mao, Pinlong Cai, Guohang Yan, Yirong Chen, Zilin Bian, Ding Wang, Botian Shi
Comment 14 pages, 7 figures, 6 tables; Accepted by ICCV 2025
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Fast LLM Post-training via Decoupled and Best-of-N Speculation
Title CN 通过解耦和Best-of-N推测实现快速大语言模型后训练
Keywords 大语言模型 · 后训练 · 推测解码 · 训练加速 · Speculative Decoding
Summary 本文提出一种名为SpecActor的方法，用于加速大语言模型在后训练阶段的token生成过程。该方法结合动态解耦推测执行与动态Best-of-N推测策略，在保证输出正确性的同时显著提升GPU计算效率和批量生成速度。实验表明，其端到端训练速度优于veRL及其他基线推测方法1.3至1.7倍。然而，该工作局限于纯文本自回归生成的工程优化，未涉及任何视觉或多模态内容。
Reason 该论文聚焦于大语言模型后训练阶段的推理加速，核心贡献是通过动态解耦推测和Best-of-N策略提升生成效率。尽管其方法在LLM训练流水线中具有工程价值，但属于标准的‘平行应用’类优化——即在特定训练场景下对现有推测解码技术进行工程改进，并未提出可迁移至文档图像理解（DIU）领域的底层创新。例如，其技术无法直接缓解OCR幻觉、改善细粒度视觉定位或增强复杂排版建模。此外，论文完全脱离多模态设定，不涉及视觉输入处理或跨模态对齐机制，对VLM for Doc的发展无实质推动作用。
Abstract Rollout dominates the training time in large language model (LLM) post-training, where the trained model is used to generate tokens given a batch of prompts. SpecActor achieves fast rollout with speculative decoding that deploys a fast path (e.g., a smaller model) to accelerate the unparallelizable generation, while the correctness is guaranteed by fast parallel verification of the outputs with the original model. SpecActor addresses two foundational challenges in speculative rollout by (1) a \emph{dynamic decoupled speculation} execution method that maximizes the GPU computational efficiency to realize speedup for large-batch execution -- a configuration common in training but unfriendly to speculative execution and (2) a \emph{dynamic Best-of-N speculation} method that selects and combines different drafting methods according to the rollout progress. It substantially improves the speculation accuracy even when the best drafting method is unknown a priori, meanwhile without requiring adding extra computation resources. {\sys} is {1.7}\,$\times$ faster than veRL in end-to-end training, and is {1.3--1.5}\,$\times$ faster compared to baselines with speculative decoding.
Authors Rongxin Cheng, Kai Zhou, Xingda Wei, Siyuan Liu, Mingcong Han, Mingjing Ai, Yeju Zhou, Baoquan Zhong, Wencong Xiao, Rong Chen, Haibo Chen
Categories Artificial Intelligence
3
You Only Forward Once: An Efficient Compositional Judging Paradigm
Title CN 你只需前向一次：一种高效的组合式评判范式
Keywords 多模态大语言模型 · 高效推理 · 评判系统 · 模板条件化 · 单次前向
Summary 本文提出YOFO方法，使多模态大语言模型能在单次前向传递中对结构化需求模板进行多项判断，通过读取最终token的logits实现快速二分类决策，在保持可解释性的同时大幅提升评判效率，并支持依赖感知分析和事后思维链增强。实验表明其在推荐数据集上达到SOTA性能。
Reason 该论文提出了一种用于多模态大模型评判任务的高效推理方法YOFO，通过模板条件化在单次前向传播中完成多个判断。尽管其在高吞吐场景下具有显著速度优势，但其核心应用场景是推荐系统中的评判任务，属于典型的‘平行下游应用’（Wrapper/Application），并未针对文档图像理解（DIU）中的OCR幻觉、复杂排版、细粒度定位或长文档推理等核心问题提出解决方案，也未展示可迁移至DIU领域的通用机制。
Abstract Multimodal large language models (MLLMs) show strong potential as judges. However, existing approaches face a fundamental trade-off: adapting MLLMs to output a single score misaligns with the generative nature of MLLMs and limits fine-grained requirement understanding, whereas autoregressively generating judging analyses is prohibitively slow in high-throughput settings. Observing that judgment reduces to verifying whether inputs satisfy a set of structured requirements, we propose YOFO, a template-conditioned method that judges all requirements in a single forward pass. Built on an autoregressive model, YOFO accepts a structured requirement template and, in one inference step, produces a binary yes/no decision for each requirement by reading the logits of the final token associated with that requirement. This design yields orders-of-magnitude speedups while preserving interpretability. Extensive experiments show that YOFO not only achieves state-of-the-art results on standard recommendation datasets, but also supports dependency-aware analysis -- where subsequent judgments are conditioned on previous ones -- and further benefits from post-hoc CoT.
Authors Tianlong Zhang, Hongwei Xue, Shilin Yan, Di Wu, Chen Xu, Yunyun Yang
Categories Artificial Intelligence
3
The Finer the Better: Towards Granular-aware Open-set Domain Generalization aaai2026
Title CN 越精细越好：迈向细粒度感知的开集域泛化
Keywords 开集域泛化 · 视觉-语言模型 · 细粒度语义对齐 · 伪未知生成 · 对比学习
Summary 本文提出SeeCLIP框架，通过语义感知提示增强、双路对比学习和语义引导扩散机制，在开集域泛化任务中提升模型对细粒度未知类别的识别能力。通过生成与已知类别相似但局部不同的伪未知样本，迫使模型学习更精细的决策边界。实验表明其在多个基准上优于现有方法。
Reason 该论文研究的是开集域泛化（OSDG）问题，核心是通过语义增强和对比学习提升视觉-语言模型在已知与未知类别共存场景下的泛化能力。尽管其提出的细粒度语义对齐和伪未知样本生成机制具有一定创新性，但整体工作仍属于通用视觉-语言模型在开放世界分类任务中的应用，并未针对文档图像理解（DIU）中的具体痛点（如OCR幻觉、复杂版面解析、长文档推理等）进行建模。方法上也未涉及文档特有的高分辨率处理、视觉token压缩或多步信息定位等关键技术路径，迁移价值有限。
Abstract Open-Set Domain Generalization (OSDG) tackles the realistic scenario where deployed models encounter both domain shifts and novel object categories. Despite impressive progress with vision-language models like CLIP, existing methods still fall into the dilemma between structural risk of known-classes and open-space risk from unknown-classes, and easily suffers from over-confidence, especially when distinguishing ``hard unknowns" that share fine-grained visual similarities with known classes. To this end, we propose a Semantic-enhanced CLIP (SeeCLIP) framework that explicitly addresses this dilemma through fine-grained semantic enhancement. In SeeCLIP, we propose a semantic-aware prompt enhancement module to decompose images into discriminative semantic tokens, enabling nuanced vision-language alignment beyond coarse category labels. To position unknown prompts effectively, we introduce duplex contrastive learning with complementary objectives, that is, repulsion to maintain separability from known classes, and cohesion to preserve semantic proximity. Further, our semantic-guided diffusion module synthesizes pseudo-unknowns by perturbing extracted semantic tokens, generating challenging samples that are visually similar to known classes yet exhibit key local differences. These hard negatives force the model to learn finer decision boundaries. Extensive experiments across five benchmarks demonstrate consistent improvements of 3% accuracy and 5% H-score over state-of-the-art methods.
Authors Yunyun Wang, Zheng Duan, Xinyue Liao, Ke-Jia Chen, Songcan Chen
Comment 9 pages,3 figures,aaai2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
ResearStudio: A Human-Intervenable Framework for Building Controllable Deep-Research Agents EMNLP 2025 Demo, Oral
Title CN ResearStudio：一种可由人类干预的可控深度研究智能体构建框架
Keywords 智能体框架 · 人类干预 · 深度研究智能体 · 协作工作流 · EMNLP2025
Summary 本文提出了ResearStudio，一个支持人类实时干预的深度研究智能体框架。系统采用分层的Planner-Executor结构，将每一步计划写入可编辑的‘计划文档’，并通过快速通信层将操作流式传输到Web界面，允许用户随时暂停、修改计划或代码、执行自定义命令后恢复运行，实现AI主导与人工主导模式的平滑切换。在GAIA基准上实现了SOTA性能。代码和演示已开源。
Reason 该论文属于典型的‘平行应用’类工作，聚焦于构建通用深度研究智能体（deep-research agent）并引入人类干预机制。尽管其‘plan-as-document’和实时控制接口设计具有一定系统工程价值，但其核心任务与文档图像理解（DIU）无关，未涉及DocVQA、OCR、版面分析、表格识别等任何文档理解任务。所提出的干预机制虽理论上可用于长文档处理流程控制，但文中未体现对DIU特有痛点（如OCR幻觉、密集文本、复杂排版）的针对性解决方案，属于垂类Agent应用而非底层方法创新。
Abstract Current deep-research agents run in a ''fire-and-forget'' mode: once started, they give users no way to fix errors or add expert knowledge during execution. We present ResearStudio, the first open-source framework that places real-time human control at its core. The system follows a Collaborative Workshop design. A hierarchical Planner-Executor writes every step to a live ''plan-as-document,'' a fast communication layer streams each action, file change, and tool call to a web interface. At any moment, the user can pause the run, edit the plan or code, run custom commands, and resume -- switching smoothly between AI-led, human-assisted and human-led, AI-assisted modes. In fully autonomous mode, ResearStudio achieves state-of-the-art results on the GAIA benchmark, surpassing systems like OpenAI's DeepResearch and Manus. These results show that strong automated performance and fine-grained human control can coexist. The full code, protocol, and evaluation scripts are available at https://github.com/ResearAI/ResearStudio. We will continue to update the repository to encourage further work on safe and controllable research agents. Our live demo is publicly accessible at http://ai-researcher.net:3000/. We support the development of DeepScientist, which can be accessed at https://github.com/ResearAI/DeepScientist.
Authors Linyi Yang, Yixuan Weng
Comment EMNLP 2025 Demo, Oral
Categories Artificial Intelligence
3
Optimizing PyTorch Inference with LLM-Based Multi-Agent Systems
Title CN 基于LLM多智能体系统优化PyTorch推理
Keywords PyTorch优化 · 多智能体系统 · LLM编译器 · GPU加速 · 推理性能
Summary 本文研究了使用基于大语言模型的多智能体系统来优化PyTorch在GPU上的推理性能，提出了一种逻辑框架用于比较不同多智能体策略，并在KernelBench基准上实现了平均2.88倍的加速。重点在于系统级性能调优，而非AI模型本身的结构或能力改进。
Reason 该论文属于典型的‘平行应用’：将LLM多智能体用于PyTorch推理优化，本质上是编译器优化或系统调优任务，并未针对文档图像理解（DIU）中的核心问题（如OCR幻觉、复杂排版、细粒度定位等）提出任何方法论创新。尽管多智能体系统作为上游技术有一定潜力，但本文聚焦的是底层计算性能优化，与DIU任务无直接关联，迁移价值极低。
Abstract Maximizing performance on available GPU hardware is an ongoing challenge for modern AI inference systems. Traditional approaches include writing custom GPU kernels and using specialized model compilers to tune high-level code for specific GPU targets. Recent work shows that LLM-based multi-agent systems can effectively perform such tuning, often outperforming existing compilers and eliminating the need for manual kernel development. However, the dynamics of multi-agent systems for this task remain unexplored. In this work, we present a logical framework for comparing multi-agent PyTorch optimization systems. Our evaluation shows that exploit-heavy strategies perform best when paired with error-fixing agents, and that performance correlates with the granularity of optimization steps. The best implementation achieves an average 2.88x speedup on an H100 GPU across diverse tasks in KernelBench, a benchmark suite covering a range of machine learning architectures in PyTorch.
Authors Kirill Nagaitsev, Luka Grbcic, Samuel Williams, Costin Iancu
Categories Artificial Intelligence
3
SALT: Steering Activations towards Leakage-free Thinking in Chain of Thought
Title CN SALT：引导激活以实现无泄漏思维的思维链
Keywords SALT · 隐私保护 · 思维链 · 干预方法 · 大语言模型
Summary 本文提出SALT，一种在推理时轻量级干预方法，通过向模型隐藏层注入定向引导向量，抑制思维链中敏感信息的泄露。实验表明其在多个LLM上显著降低上下文隐私泄露率，同时保持任务性能。适用于需保护用户隐私的智能代理场景。
Reason 该论文聚焦于大语言模型在处理敏感用户数据时的隐私泄露问题，提出通过干预隐藏状态来减少思维链中的信息泄露。尽管其为推理过程中的干预技术（Intervention/Steering）提供了新思路，但核心目标是隐私保护而非解决文档图像理解中的OCR幻觉、细粒度定位或复杂排版等痛点。方法虽具创新性，但迁移到DIU场景的路径不明确，且未涉及视觉模态或多模态对齐，属于通用LLM推理干预，与文档理解无直接关联。
Abstract As Large Language Models (LLMs) evolve into personal assistants with access to sensitive user data, they face a critical privacy challenge: while prior work has addressed output-level privacy, recent findings reveal that LLMs often leak private information through their internal reasoning processes, violating contextual privacy expectations. These leaky thoughts occur when models inadvertently expose sensitive details in their reasoning traces, even when final outputs appear safe. The challenge lies in preventing such leakage without compromising the model's reasoning capabilities, requiring a delicate balance between privacy and utility. We introduce Steering Activations towards Leakage-free Thinking (SALT), a lightweight test-time intervention that mitigates privacy leakage in model's Chain of Thought (CoT) by injecting targeted steering vectors into hidden state. We identify the high-leakage layers responsible for this behavior. Through experiments across multiple LLMs, we demonstrate that SALT achieves reductions including $18.2\%$ reduction in CPL on QwQ-32B, $17.9\%$ reduction in CPL on Llama-3.1-8B, and $31.2\%$ reduction in CPL on Deepseek in contextual privacy leakage dataset AirGapAgent-R while maintaining comparable task performance and utility. Our work establishes SALT as a practical approach for test-time privacy protection in reasoning-capable language models, offering a path toward safer deployment of LLM-based personal agents.
Authors Shourya Batra, Pierce Tillman, Samarth Gaggar, Shashank Kesineni, Kevin Zhu, Sunishchal Dev, Ashwinee Panda, Vasu Sharma, Maheep Chaudhary
Categories Artificial Intelligence, Computation and Language
3
RASTP: Representation-Aware Semantic Token Pruning for Generative Recommendation with Semantic Identifiers
Title CN RASTP：基于表示感知的语义令牌剪枝用于带语义标识符的生成式推荐
Keywords 生成式推荐 · 语义标识符 · 令牌剪枝 · 序列压缩 · 效率优化
Summary 本文提出RASTP方法，在生成式推荐系统中对语义标识符（SID）序列进行令牌剪枝，以降低输入长度和计算开销。通过结合表示幅度和注意力中心性评估令牌重要性，实现动态剪枝，在减少训练时间的同时保持推荐性能。
Reason 该论文研究的是生成式推荐系统中语义标识符（SID）带来的序列过长问题，提出通过重要性评估剪枝不重要的语义令牌。尽管涉及序列压缩和效率优化，但其任务背景完全脱离文档图像理解（DIU），既不处理视觉输入、OCR、布局分析，也不涉及多模态建模或文档推理。属于推荐系统的内部优化技术，无法迁移解决DIU中的核心痛点如高分辨率处理、幻觉、细粒度定位等。
Abstract Generative recommendation systems typically leverage Semantic Identifiers (SIDs), which represent each item as a sequence of tokens that encode semantic information. However, representing item ID with multiple SIDs significantly increases input sequence length, which is a major determinant of computational complexity and memory consumption. While existing efforts primarily focus on optimizing attention computation and KV cache, we propose RASTP (Representation-Aware Semantic Token Pruning), which directly prunes less informative tokens in the input sequence. Specifically, RASTP evaluates token importance by combining semantic saliency, measured via representation magnitude, and attention centrality, derived from cumulative attention weights. Since RASTP dynamically prunes low-information or irrelevant semantic tokens, experiments on three real-world Amazon datasets show that RASTP reduces training time by 26.7\%, while maintaining or slightly improving recommendation performance. The code has been open-sourced at https://github.com/Yuzt-zju/RASTP.
Authors Tianyu Zhan, Kairui Fu, Zheqi Lv, Shengyu Zhang
Comment 4 pages
Categories Artificial Intelligence
3
OmniGround: A Comprehensive Spatio-Temporal Grounding Benchmark for Real-World Complex Scenarios
Title CN OmniGround：面向真实世界复杂场景的综合性时空定位基准
Keywords 时空定位 · 视频理解 · 多模态大模型 · 基准测试 · 细粒度定位
Summary 本文提出了OmniGround，一个包含3475个视频、81个类别的综合性时空定位基准，旨在解决现有数据集在类别覆盖、推理复杂度和语言鲁棒性方面的局限。作者设计了Forward-Backward-Refinement标注流程以提升标签质量，并提出DeepSTG评估框架从四个维度系统评估数据集质量。实验表明当前模型在复杂现实场景下性能平均下降10.4%。为此，作者提出无需训练的两阶段框架PG-TAF，先进行高层时间定位，再执行细粒度时空传播，在m_tIoU和m_vIoU上分别提升25.6%和35.6%。
Reason 该论文聚焦于视频中的时空定位（STVG），属于视频理解领域，尽管涉及多模态大模型和细粒度定位，但其任务场景与文档图像理解（DIU）差异较大。虽然‘细粒度定位’和‘复杂空间关系’在理论上对文档中的视觉 grounding 有一定启发，但该工作主要针对动态视频对象，且提出的PG-TAF框架和OmniGround基准均围绕视频时序建模展开，难以直接迁移至静态文档图像的理解任务中。不符合DIU本题要求，也不具备可迁移的底层方法创新。
Abstract Spatio-Temporal Video Grounding (STVG) aims to localize target objects in videos based on natural language descriptions. Despite recent advances in Multimodal Large Language Models, a significant gap remains between current models and real-world demands involving diverse objects and complex queries. We attribute this to limited benchmark scope, causing models to exhibit category bias, oversimplified reasoning, and poor linguistic robustness. To address these limitations, we introduce OmniGround, a comprehensive benchmark with 3,475 videos spanning 81 categories and complex real-world queries. We propose the Forward-Backward-Refinement annotation pipeline that combines multi-directional tracking with intelligent error correction for high-quality labels. We further introduce DeepSTG, a systematic evaluation framework quantifying dataset quality across four complementary dimensions beyond superficial statistics. Evaluations reveal performance average drop of 10.4% on complex real-world scenes, particularly with small/occluded objects and intricate spatial relations. Motivated by these, we propose PG-TAF, a training-free two-stage framework decomposing STVG into high-level temporal grounding and fine-grained spatio-temporal propagation. Experiments demonstrate PG-TAF achieves 25.6% and 35.6% improvements in m\_tIoU and m\_vIoU on OmniGround with consistent gains across four benchmarks.
Authors Hong Gao, Jingyu Wu, Xiangkai Xu, Kangni Xie, Yunchen Zhang, Bin Zhong, Xurui Gao, Min-Ling Zhang
Comment 20 pages
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
Multi-Agent Collaborative Reward Design for Enhancing Reasoning in Reinforcement Learning
Title CN 用于增强强化学习中推理能力的多智能体协同奖励设计
Keywords 多智能体系统 · 强化学习 · 奖励模型 · RLHF · 可解释性
Summary 本文提出CRM框架，通过多个专业化评估智能体协同生成奖励信号，替代传统的单一黑箱奖励模型，以提升强化学习中奖励设计的鲁棒性和可解释性。各智能体负责不同维度（如事实性、安全性）的评估，再由中心聚合器融合为统一奖励信号，并结合rewardBench基准进行训练与评估。
Reason 该论文属于典型的‘平行应用’模式：将多智能体框架应用于奖励模型设计，核心场景是通用RLHF中的偏好优化，而非针对文档理解任务（如DocVQA、布局分析等）的结构化挑战。尽管其‘多视角评估’思想理论上可用于缓解OCR幻觉或提升推理一致性，但文中未涉及任何视觉-语言对齐、细粒度定位或复杂排版建模等DIU核心问题，且方法依赖于抽象的偏好维度（如安全性、帮助性），难以直接迁移至文档图像的理解与结构化解析。属于通用AI架构在特定流程上的重组，无针对DIU的方法论创新。
Abstract We present CRM (Multi-Agent Collaborative Reward Model), a framework that replaces a single black-box reward model with a coordinated team of specialist evaluators to improve robustness and interpretability in RLHF. Conventional reward models struggle to jointly optimize multiple, sometimes conflicting, preference dimensions (e.g., factuality, helpfulness, safety) and offer limited transparency into why a score is assigned. CRM addresses these issues by decomposing preference evaluation into domain-specific agents that each produce partial signals, alongside global evaluators such as ranker-based and embedding-similarity rewards. A centralized aggregator fuses these signals at each timestep, balancing factors like step-wise correctness, multi-agent agreement, and repetition penalties, yielding a single training reward compatible with standard RL pipelines. The policy is optimized with advantage-based updates (e.g., GAE), while a value model regresses to the aggregated reward, enabling multi-perspective reward shaping without requiring additional human annotations beyond those used to train the evaluators. To support training and assessment, we introduce rewardBench, a benchmark and training suite aligned with the collaborative structure of CRM. Together, CRM and rewardBench provide a practical, modular path to more transparent reward modeling and more stable optimization.
Authors Pei Yang, Ke Zhang, Ji Wang, Xiao Chen, Yuxin Tang, Eric Yang, Lynn Ai, Bill Shi
Categories Artificial Intelligence
3
Bridging the Semantic Gap: Contrastive Rewards for Multilingual Text-to-SQL with GRPO 20th International Workshop on Semantic and Social Media Adaptation & Personalization
Title CN 弥合语义鸿沟：基于GRPO的多语言文本到SQL的对比奖励
Keywords Text-to-SQL · 多语言 · 语义对齐 · 强化学习 · GRPO
Summary 本文提出一种结合Group Relative Policy Optimization（GRPO）与多语言对比奖励信号的Text-to-SQL框架，旨在提升跨语言场景下SQL生成的执行准确率与语义一致性。在MultiSpider七语言数据集上，使用LLaMA-3-3B模型进行实验，结果表明该方法显著提升了执行准确率和语义准确率，尤其在低资源语言中表现突出。然而，该研究完全基于文本输入，未涉及图像或文档视觉结构。
Reason 该论文聚焦于Text-to-SQL任务中的多语言语义对齐问题，属于典型的下游应用型工作，尽管引入了GRPO和对比奖励机制，但其核心场景是数据库查询生成，并未涉及文档图像理解（DIU）中的任何关键挑战，如OCR幻觉、复杂版面分析、视觉-文本对齐或高分辨率处理。虽然强化学习框架和语义奖励设计具有一定方法论价值，但迁移到DIU领域的路径不明确，且任务本质为纯语言转换，缺乏视觉模态输入，无法直接服务于文档VQA或布局理解等任务。
Abstract Current Text-to-SQL methods are evaluated and only focused on executable queries, overlooking the semantic alignment challenge -- both in terms of the semantic meaning of the query and the correctness of the execution results. Even execution accuracy itself shows significant drops when moving from English to other languages, with an average decline of 6 percentage points across non-English languages. We address these challenges by presenting a new framework that combines Group Relative Policy Optimization (GRPO) within a multilingual contrastive reward signal to enhance both task efficiency and semantic accuracy in Text-to-SQL systems in cross-lingual scenarios. Our method teaches models to obtain better correspondence between SQL generation and user intent by combining a reward signal based on semantic similarity. On the seven-language MultiSpider dataset, fine-tuning the LLaMA-3-3B model with GRPO improved the execution accuracy up to 87.4 percent (+26 pp over zero-shot) and semantic accuracy up to 52.29 percent (+32.86 pp). Adding our contrastive reward signal in the GRPO framework further improved the average semantic accuracy to 59.14 percent (+6.85 pp, up to +10 pp for Vietnamese). Our experiments showcase that a smaller, parameter-efficient 3B LLaMA model fine-tuned with our contrastive reward signal outperforms a much larger zero-shot 8B LLaMA model, with an uplift of 7.43 pp in execution accuracy (from 81.43 percent on the 8B model to 88.86 percent on the 3B model), and nearly matches its semantic accuracy (59.14 percent vs. 68.57 percent) -- all using just 3,000 reinforcement learning training examples. These results demonstrate how we can improve the performance of Text-to-SQL systems with contrastive rewards for directed semantic alignment, without requiring large-scale training datasets.
Authors Ashish Kattamuri, Ishita Prasad, Meetu Malhotra, Arpita Vats, Rahul Raja, Albert Lie
Comment 20th International Workshop on Semantic and Social Media Adaptation & Personalization
Categories Artificial Intelligence, Computation and Language
3
Task-Aligned Tool Recommendation for Large Language Models IJCNLP-AACL 2025 Main
Title CN 面向大语言模型的任务对齐工具推荐
Keywords 工具推荐 · 大语言模型 · Agent系统 · 任务对齐 · RecTools
Summary 本文提出了一种面向大语言模型的精准工具推荐方法PTR，通过历史工具使用模式和多视角匹配动态构建最优工具集，并发布了新数据集RecTools和评估指标TRACC。实验表明该方法在多个基准上具有良好的准确性。
Reason 该论文研究的是如何为大语言模型推荐合适的外部工具集，属于典型的Agent系统中的模块设计，但其问题设定和方法均针对通用任务场景下的工具调用，并未涉及任何视觉或文档模态。尽管工具推荐机制在理论上可用于复杂文档处理流程的调度，但本文没有引入多模态输入、视觉推理或与OCR/布局分析相关的建模，无法直接解决文档图像理解中的核心痛点（如幻觉、细粒度定位、高分辨率处理等）。因此属于‘平行应用’类的通用Agent技术，不符合DIU本题创新或可迁移的上游方法标准。
Abstract By augmenting Large Language Models (LLMs) with external tools, their capacity to solve complex problems has been significantly enhanced. However, despite ongoing advancements in the parsing capabilities of LLMs, incorporating all available tools simultaneously in the prompt remains impractical due to the vast number of external tools. Consequently, it is essential to provide LLMs with a precise set of tools tailored to the specific task, considering both quantity and quality. Current tool retrieval methods primarily focus on refining the ranking list of tools and directly packaging a fixed number of top-ranked tools as the tool set. However, these approaches often fail to equip LLMs with the optimal set of tools prior to execution, since the optimal number of tools for different tasks could be different, resulting in inefficiencies such as redundant or unsuitable tools, which impede immediate access to the most relevant tools. This paper addresses the challenge of recommending precise toolsets for LLMs. We introduce the problem of tool recommendation, define its scope, and propose a novel Precision-driven Tool Recommendation (PTR) approach. PTR captures an initial, concise set of tools by leveraging historical tool bundle usage and dynamically adjusts the tool set by performing tool matching, culminating in a multi-view-based tool addition. Additionally, we present a new dataset, RecTools, and a metric, TRACC, designed to evaluate the effectiveness of tool recommendation for LLMs. We further validate our design choices through comprehensive experiments, demonstrating promising accuracy across two open benchmarks and our RecTools dataset.
Authors Hang Gao, Yongfeng Zhang
Comment IJCNLP-AACL 2025 Main
Categories Artificial Intelligence, Computation and Language
3
Importance-Weighted Non-IID Sampling for Flow Matching Models
Title CN 用于流匹配模型的重要性加权非独立同分布采样
Keywords 流匹配模型 · 重要性采样 · 非独立同分布 · 得分函数 · 期望估计
Summary 本文提出一种重要性加权的非独立同分布采样框架，用于改善流匹配模型在有限采样预算下的期望估计效果。通过联合采样覆盖分布中的多样且显著区域，并利用得分函数正则化保持样本多样性与质量平衡，避免流形外漂移。同时首次实现了对非独立采样路径的重要性权重估计，提升了输出表征的可靠性。
Reason 该论文研究的是流匹配模型中的采样策略，核心问题是估计复杂分布下函数期望时的方差控制，属于生成模型理论方法的范畴。尽管其提出的非独立采样与重要性加权机制在理论上具有创新性，但并未涉及多模态建模、视觉语言理解或文档结构建模等DIU相关任务。方法聚焦于连续流模型的输出采样优化，难以直接迁移至文档图像理解中常见的OCR去幻觉、细粒度定位或长文档推理等关键问题。无证据表明该技术可有效作用于VLM架构改进或文档特有的高分辨率、复杂排版处理需求。
Abstract Flow-matching models effectively represent complex distributions, yet estimating expectations of functions of their outputs remains challenging under limited sampling budgets. Independent sampling often yields high-variance estimates, especially when rare but with high-impact outcomes dominate the expectation. We propose an importance-weighted non-IID sampling framework that jointly draws multiple samples to cover diverse, salient regions of a flow's distribution while maintaining unbiased estimation via estimated importance weights. To balance diversity and quality, we introduce a score-based regularization for the diversity mechanism, which uses the score function, i.e., the gradient of the log probability, to ensure samples are pushed apart within high-density regions of the data manifold, mitigating off-manifold drift. We further develop the first approach for importance weighting of non-IID flow samples by learning a residual velocity field that reproduces the marginal distribution of the non-IID samples. Empirically, our method produces diverse, high-quality samples and accurate estimates of both importance weights and expectations, advancing the reliable characterization of flow-matching model outputs. Our code will be publicly available on GitHub.
Authors Xinshuang Liu, Runfa Blark Li, Shaoxiu Wei, Truong Nguyen
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
A Stitch in Time: Learning Procedural Workflow via Self-Supervised Plackett-Luce Ranking
Title CN 一针及时：通过自监督Plackett-Luce排序学习程序性工作流
Keywords 自监督学习 · 程序性视频理解 · Plackett-Luce模型 · 时序建模 · 视频表示学习
Summary 本文提出PL-Stitch，一种基于Plackett-Luce模型的自监督学习框架，用于捕捉程序性活动（如烹饪和手术）中动作的时序结构。通过全局帧排序和时空拼图两个目标，模型能更好地区分正向与反向时间序列，在多个手术和烹饪数据集上提升了阶段识别与动作分割性能。
Reason 该论文研究的是程序性活动（如烹饪和外科手术）中的视频帧时序建模，属于视频理解领域。其核心方法PL-Stitch基于Plackett-Luce模型对视频帧进行排序学习，虽在手术阶段识别等任务上有提升，但应用场景与文档图像理解（DIU）差异显著。文档图像为静态、高结构化排版内容，而非时间序列动作流，因此该方法难以迁移到DocVQA、表格识别或布局分析等DIU核心任务。属于无关领域工作，不符合关注方向。
Abstract Procedural activities, ranging from routine cooking to complex surgical operations, are highly structured as a set of actions conducted in a specific temporal order. Despite their success on static images and short clips, current self-supervised learning methods often overlook the procedural nature that underpins such activities. We expose the lack of procedural awareness in current SSL methods with a motivating experiment: models pretrained on forward and time-reversed sequences produce highly similar features, confirming that their representations are blind to the underlying procedural order. To address this shortcoming, we propose PL-Stitch, a self-supervised framework that harnesses the inherent temporal order of video frames as a powerful supervisory signal. Our approach integrates two novel probabilistic objectives based on the Plackett-Luce (PL) model. The primary PL objective trains the model to sort sampled frames chronologically, compelling it to learn the global workflow progression. The secondary objective, a spatio-temporal jigsaw loss, complements the learning by capturing fine-grained, cross-frame object correlations. Our approach consistently achieves superior performance across five surgical and cooking benchmarks. Specifically, PL-Stitch yields significant gains in surgical phase recognition (e.g., +11.4 pp k-NN accuracy on Cholec80) and cooking action segmentation (e.g., +5.7 pp linear probing accuracy on Breakfast), demonstrating its effectiveness for procedural video representation learning.
Authors Chengan Che, Chao Wang, Xinyue Chen, Sophia Tsoka, Luis C. Garcia-Peraza-Herrera
Comment 18 pages
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
3
DemoShapley: Valuation of Demonstrations for In-Context Learning
Title CN DemoShapley：用于上下文学习中的示例估值
Keywords 上下文学习 · 示例选择 · Shapley值 · 大语言模型 · 提示工程
Summary 本文提出DemoShapley，一种基于Shapley值的方法，用于评估上下文学习中各个示范样本的贡献，通过计算不同排列下样本的边际效应来优化选择与排序。还提出了适用于小样本场景的加权变体Beta-DemoShapley。实验表明其在多个基准上优于现有方法，并具备检测错误标注、缓解偏见等附加能力。
Reason 该论文研究的是大语言模型在上下文学习（ICL）中如何选择和排序示范样本，属于纯NLP范畴，未涉及任何视觉或文档模态。其方法虽对提示工程有贡献，但无法直接迁移解决文档图像理解中的核心问题如OCR幻觉、细粒度定位或复杂排版建模。属于与DIU无关的平行方法研究。
Abstract Large language models (LLMs) using in-context learning (ICL) excel in many tasks without task-specific fine-tuning. However, demonstration selection and ordering greatly impact ICL effectiveness. Focus on this issue, we propose DemoShapley, a Shapley-value based method that evaluates each demonstration's contribution by measuring its marginal effect across different prompt permutations. To further account for ICL's limited context windows and frequent low-shot settings, we introduce Beta-DemoShapley, a weighted extension that emphasizes the influence of smaller prompt sizes. Experiments on multiple benchmarks show that DemoShapley consistently outperforms existing influence-based selection strategies, while Beta-DemoShapley further improves performance in low-shot scenarios. Both methods also detect mislabeled data, enhance generalization to out-of-distribution tasks, and reduce demographic bias. Together, they provide a unified and robust framework for demonstration valuation in ICL.
Authors Shan Xie, Man Luo, Chadly Daniel Stern, Mengnan Du, Lu Cheng
Categories Artificial Intelligence, Computation and Language
3
Estimating LLM Consistency: A User Baseline vs Surrogate Metrics EMNLP 2025
Title CN 评估大语言模型一致性：用户基线与代理指标的比较
Keywords 大语言模型 · 一致性评估 · 幻觉 · 人类评估 · 代理指标
Summary 论文通过大规模用户研究（n=2,976）发现当前用于衡量LLM响应一致性的自动指标与人类感知存在显著偏差，并提出一种基于logit的集成方法，在预测人类一致性评分方面表现更优。研究表明，依赖自动化指标可能误判模型性能，建议更多使用人工评估。
Reason 该论文研究的是如何衡量大语言模型（LLM）生成文本的一致性，并通过人类用户研究评估现有自动指标的可靠性。虽然涉及‘幻觉’和一致性问题，但其研究对象是纯文本LLM输出，未涉及视觉输入或文档图像理解中的多模态挑战。尽管‘一致性度量’可能在广义上对DIU中的VLM去幻觉有启发，但本文并未处理OCR幻觉、视觉-文本对齐或高分辨率图像输入等核心DIU问题，属于通用NLP方法论研究，且无明确迁移路径到文档理解场景。
Abstract Large language models (LLMs) are prone to hallucinations and sensitive to prompt perturbations, often resulting in inconsistent or unreliable generated text. Different methods have been proposed to mitigate such hallucinations and fragility, one of which is to measure the consistency of LLM responses -- the model's confidence in the response or likelihood of generating a similar response when resampled. In previous work, measuring LLM response consistency often relied on calculating the probability of a response appearing within a pool of resampled responses, analyzing internal states, or evaluating logits of responses. However, it was not clear how well these approaches approximated users' perceptions of consistency of LLM responses. To find out, we performed a user study ($n=2,976$) demonstrating that current methods for measuring LLM response consistency typically do not align well with humans' perceptions of LLM consistency. We propose a logit-based ensemble method for estimating LLM consistency and show that our method matches the performance of the best-performing existing metric in estimating human ratings of LLM consistency. Our results suggest that methods for estimating LLM consistency without human evaluation are sufficiently imperfect to warrant broader use of evaluation with human input; this would avoid misjudging the adequacy of models because of the imperfections of automated consistency metrics.
Authors Xiaoyuan Wu, Weiran Lin, Omer Akgul, Lujo Bauer
Comment Published as a main conference paper at EMNLP 2025
Categories Artificial Intelligence, Computation and Language
3
M3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark
Title CN M3-Bench：多模态、多跳、多线程工具使用MLLM代理基准
Keywords 多模态基准 · 工具使用 · MLLM评测 · 多跳推理 · 视觉接地
Summary 本文提出了M3-Bench，首个基于Model Context Protocol（MCP）的多模态工具使用评测基准，支持多跳、多线程的工作流评估，强调跨工具依赖与中间资源持久化。通过相似性驱动的对齐方法实现工具调用序列的一对一匹配，并提供可解释的指标分解语义保真度与流程一致性。数据集包含28个服务器上的231个工具，轨迹由执行器与裁判流水线生成并经人类验证。多个主流MLLM在该基准上表现不佳，暴露出参数保真度与结构一致性方面的缺陷。
Reason 该论文提出的是一个用于评估多模态大语言模型（MLLM）在工具使用场景下的基准测试框架，聚焦于多跳、多线程工作流中的视觉定位与文本推理。尽管涉及多模态和工具调用的复杂流程，但其本质是一个面向通用多模态Agent能力的评测基准，并非针对文档图像理解（DIU）任务设计。未解决OCR幻觉、密集文本识别、复杂版式分析或长文档推理等DIU核心问题，也未提出可迁移至文档理解领域的底层方法（如视觉表示、高分辨率处理架构、去幻觉机制等）。属于典型的‘平行应用’类评测工作，对DIU的方法论推进有限。
Abstract We present M^3-Bench, the first benchmark for evaluating multimodal tool use under the Model Context Protocol. The benchmark targets realistic, multi-hop and multi-threaded workflows that require visual grounding and textual reasoning, cross-tool dependencies, and persistence of intermediate resources across steps. We introduce a similarity-driven alignment that serializes each tool call, embeds signatures with a sentence encoder, and performs similarity-bucketed Hungarian matching to obtain auditable one-to-one correspondences. On top of this alignment, we report interpretable metrics that decouple semantic fidelity from workflow consistency. The benchmark spans 28 servers with 231 tools, and provides standardized trajectories curated through an Executor & Judge pipeline with human verification; an auxiliary four large language models (LLMs) judge ensemble reports end-task Task Completion and information grounding. Evaluations of representative state-of-the-art Multimodal LLMs (MLLMs) reveal persistent gaps in multimodal MCP tool use, particularly in argument fidelity and structure consistency, underscoring the need for methods that jointly reason over images, text, and tool graphs. Our Benchmark's anonymous repository is at https://github.com/EtaYang10th/Open-M3-Bench
Authors Yang Zhou, Mingyu Zhao, Zhenting Wang, Difei Gu, Bangwei Guo, Ruosong Ye, Ligong Han, Can Jin, Dimitris N. Metaxas
Categories Artificial Intelligence
3
Ternary Gamma Semirings as a Novel Algebraic Framework for Learnable Symbolic Reasoning
Title CN 三元Gamma半环：一种用于可学习符号推理的新型代数框架
Keywords 符号推理 · 三元关系 · 代数结构 · 神经符号系统 · 知识图谱
Summary 本文提出了神经三元半环（NTS），一种基于三元Gamma半环的可学习、可微代数框架，旨在直接建模三元关系（如主谓宾结构），避免传统方法将三元交互降维为二元操作带来的语义损失。通过神经网络实现三元运算，并引入代数正则化项来逼近结合律与分配律。理论上证明当代数误差趋零时，学习结果收敛至合法的三元Gamma半环。适用于知识图谱补全和规则推理等任务，但尚未应用于具体感知系统。
Reason 该论文提出了一种基于三元Gamma半环的代数结构，用于建模三元关系的符号推理，属于理论代数与神经符号系统的交叉研究。尽管其动机涉及知识图谱和逻辑推理等可能与文档理解相关的任务，但论文本身未涉及任何多模态输入、视觉内容处理或文档结构建模。其方法聚焦于抽象代数结构的学习，缺乏对图像、布局或OCR相关问题的直接建模能力，无法迁移解决DIU中的核心痛点如OCR幻觉、细粒度定位或复杂排版理解。属于符号AI领域的理论探索，而非可用于文档理解的‘军火库’型方法。
Abstract Binary semirings such as the tropical, log, and probability semirings form a core algebraic tool in classical and modern neural inference systems, supporting tasks like Viterbi decoding, dynamic programming, and probabilistic reasoning. However, these structures rely on a binary multiplication operator and therefore model only pairwise interactions. Many symbolic AI tasks are inherently triadic, including subject-predicate-object relations in knowledge graphs, logical rules involving two premises and one conclusion, and multi-entity dependencies in structured decision processes. Existing neural architectures usually approximate these interactions by flattening or factorizing them into binary components, which weakens inductive structure, distorts relational meaning, and reduces interpretability. This paper introduces the Neural Ternary Semiring (NTS), a learnable and differentiable algebraic framework grounded in the theory of ternary Gamma-semirings. The central idea is to replace the usual binary product with a native ternary operator implemented by neural networks and guided by algebraic regularizers enforcing approximate associativity and distributivity. This construction allows triadic relationships to be represented directly rather than reconstructed from binary interactions. We establish a soundness result showing that, when algebraic violations vanish during training, the learned operator converges to a valid ternary Gamma-semiring. We also outline an evaluation strategy for triadic reasoning tasks such as knowledge-graph completion and rule-based inference. These insights demonstrate that ternary Gamma-semirings provide a mathematically principled and practically effective foundation for learnable symbolic reasoning.
Authors Chandrasekhar Gokavarapu, D. Madhusudhana Rao
Categories Artificial Intelligence
3
YOLO Meets Mixture-of-Experts: Adaptive Expert Routing for Robust Object Detection
Title CN YOLO遇见混合专家：用于鲁棒目标检测的自适应专家路由
Keywords 目标检测 · YOLO · 混合专家 · 自适应路由 · 特征专业化
Summary 本文提出一种基于混合专家（MoE）框架的目标检测方法，通过在多个YOLOv9-T专家之间进行自适应路由，实现动态特征专业化，从而提升mAP和AR指标。实验包含消融研究以验证设计选择。
Reason 该论文聚焦于通用目标检测任务，采用Mixture-of-Experts架构结合YOLOv9-T进行动态特征专业化。尽管其方法在目标检测上可能有效，但属于典型的计算机视觉模型结构改进，且未涉及文档图像中的文本理解、布局分析或视觉-语言推理等核心DIU问题。其技术路线与文档图像理解中的OCR幻觉、密集文本处理、复杂排版建模或长文档推理等痛点无直接关联，迁移价值极低。
Abstract This paper presents a novel Mixture-of-Experts framework for object detection, incorporating adaptive routing among multiple YOLOv9-T experts to enable dynamic feature specialization and achieve higher mean Average Precision (mAP) and Average Recall (AR) compared to a single YOLOv9-T model.
Authors Ori Meiraz, Sharon Shalev, Avishai Weizman
Comment 1 figure, 1 table
Categories Computer Vision and Pattern Recognition
3
Self-Supervised Learning by Curvature Alignment
Title CN 基于曲率对齐的自监督学习
Keywords 自监督学习 · 曲率正则化 · 流形几何 · 表示学习 · 非对比学习
Summary 本文提出CurvSSL，一种引入曲率正则化的自监督学习框架，通过在嵌入空间中对齐不同增强视图的局部流形曲率来提升表示学习效果。该方法在MNIST和CIFAR-10上验证了其有效性，但局限于自然图像的低级几何建模，未涉及文本或文档结构。
Reason 该论文属于通用视觉自监督学习方法，研究的是图像数据流形上的局部几何结构建模，其核心创新点（曲率正则化）针对自然图像的全局结构先验，在文档图像理解中缺乏直接迁移价值。文档图像的核心挑战在于文本布局、高分辨率细粒度识别和OCR语义一致性，而非像素级流形弯曲的几何保持。该方法未涉及多模态对齐、视觉-语言联合建模或高分辨率处理机制，无法缓解DIU中的OCR幻觉、复杂排版解析或长文档推理等关键问题。
Abstract Self-supervised learning (SSL) has recently advanced through non-contrastive methods that couple an invariance term with variance, covariance, or redundancy-reduction penalties. While such objectives shape first- and second-order statistics of the representation, they largely ignore the local geometry of the underlying data manifold. In this paper, we introduce CurvSSL, a curvature-regularized self-supervised learning framework, and its RKHS extension, kernel CurvSSL. Our approach retains a standard two-view encoder-projector architecture with a Barlow Twins-style redundancy-reduction loss on projected features, but augments it with a curvature-based regularizer. Each embedding is treated as a vertex whose $k$ nearest neighbors define a discrete curvature score via cosine interactions on the unit hypersphere; in the kernel variant, curvature is computed from a normalized local Gram matrix in an RKHS. These scores are aligned and decorrelated across augmentations by a Barlow-style loss on a curvature-derived matrix, encouraging both view invariance and consistency of local manifold bending. Experiments on MNIST and CIFAR-10 datasets with a ResNet-18 backbone show that curvature-regularized SSL yields competitive or improved linear evaluation performance compared to Barlow Twins and VICReg. Our results indicate that explicitly shaping local geometry is a simple and effective complement to purely statistical SSL regularizers.
Authors Benyamin Ghojogh, M. Hadi Sepanj, Paul Fieguth
Categories Computer Vision and Pattern Recognition
3
ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIP
Title CN ATAC：基于增强的测试时对抗性校正方法用于CLIP
Keywords CLIP · 对抗防御 · 测试时校正 · 嵌入空间 · 数据增强
Summary 本文提出ATAC，一种针对CLIP模型的测试时对抗防御方法，通过在嵌入空间中利用增强引起的漂移向量的方向一致性来校正语义表征，显著提升了模型在多种对抗攻击下的鲁棒性，且计算开销低。实验表明其在多个基准上优于现有方法。
Reason 该论文聚焦于提升CLIP模型在图像-文本匹配任务中的对抗鲁棒性，属于通用视觉-语言模型在自然图像上的安全防御研究。其核心贡献是提出一种在嵌入空间中通过数据增强诱导的漂移向量进行语义恢复的方法。然而，该工作并未涉及文档图像理解中的任何典型挑战，如OCR幻觉、复杂版式解析、细粒度文本定位或长文档推理。尽管‘embedding space correction’的思想理论上可能被探索用于缓解VLM在文档上的幻觉，但本文未提供任何与文本结构、布局语义或文档特定噪声相关的建模，且应用场景明确为标准图像分类/检索任务。因此，属于与DIU无关的平行技术路线，不具备直接迁移价值。
Abstract Despite its remarkable success in zero-shot image-text matching, CLIP remains highly vulnerable to adversarial perturbations on images. As adversarial fine-tuning is prohibitively costly, recent works explore various test-time defense strategies; however, these approaches still exhibit limited robustness. In this work, we revisit this problem and propose a simple yet effective strategy: Augmentation-based Test-time Adversarial Correction (ATAC). Our method operates directly in the embedding space of CLIP, calculating augmentation-induced drift vectors to infer a semantic recovery direction and correcting the embedding based on the angular consistency of these latent drifts. Across a wide range of benchmarks, ATAC consistently achieves remarkably high robustness, surpassing that of previous state-of-the-art methods by nearly 50\% on average, all while requiring minimal computational overhead. Furthermore, ATAC retains state-of-the-art robustness in unconventional and extreme settings and even achieves nontrivial robustness against adaptive attacks. Our results demonstrate that ATAC is an efficient method in a novel paradigm for test-time adversarial defenses in the embedding space of CLIP.
Authors Linxiang Su, András Balogh
Comment 16 pages
Categories Computer Vision and Pattern Recognition
3
Label-Efficient Skeleton-based Recognition with Stable-Invertible Graph Convolutional Networks
Title CN 基于稳定可逆图卷积网络的低标签依赖骨架识别
Keywords 骨架动作识别 · 图卷积网络 · 可逆网络 · 主动学习 · 标签效率
Summary 本文提出一种基于可逆图卷积网络（GCN）的骨架动作识别方法，通过学习最优采集函数来选择最具信息量的样本进行标注，从而减少对大规模标注数据的依赖。实验表明该方法在两个数据集上优于现有工作。
Reason 该论文研究的是骨架动作识别（Skeleton-based action recognition），属于视频行为理解领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管其提出的方法涉及标签效率和可逆GCN，但应用场景和数据模态（人体骨架序列）与文档结构、文本布局、OCR等核心问题完全不匹配，无法迁移解决DIU中的OCR幻觉、复杂排版或长文档推理等痛点。
Abstract Skeleton-based action recognition is a hotspot in image processing. A key challenge of this task lies in its dependence on large, manually labeled datasets whose acquisition is costly and time-consuming. This paper devises a novel, label-efficient method for skeleton-based action recognition using graph convolutional networks (GCNs). The contribution of the proposed method resides in learning a novel acquisition function -- scoring the most informative subsets for labeling -- as the optimum of an objective function mixing data representativity, diversity and uncertainty. We also extend this approach by learning the most informative subsets using an invertible GCN which allows mapping data from ambient to latent spaces where the inherent distribution of the data is more easily captured. Extensive experiments, conducted on two challenging skeleton-based recognition datasets, show the effectiveness and the outperformance of our label-frugal GCNs against the related work.
Authors Hichem Sahbi
Categories Computer Vision and Pattern Recognition
3
MolSight: Optical Chemical Structure Recognition with SMILES Pretraining, Multi-Granularity Learning and Reinforcement Learning
Title CN MolSight：基于SMILES预训练、多粒度学习和强化学习的光学化学结构识别
Keywords 光学化学结构识别 · SMILES预训练 · 多粒度学习 · 强化学习 · 立体化学
Summary 本文提出MolSight，一个用于光学化学结构识别（OCSR）的三阶段学习框架，通过SMILES序列预训练、多粒度微调（引入化学键分类与原子定位辅助任务）以及基于GRPO的强化学习优化，在立体化学结构识别上取得SOTA性能。研究重点在于提升对楔形键、环构象等细微立体信息的识别能力，并发布了一个新的立体化学数据集。
Reason 该论文属于特定领域的平行应用，即将文档图像理解技术应用于化学结构识别（OCSR）任务。尽管其方法包含预训练、多粒度学习和强化学习等组件，但整体框架是面向化学分子图这一高度专业化的场景，不具备通用文档图像理解（如表格、表单、复杂排版文本）的方法论创新。其技术路线未提出可迁移至一般DIU任务（如DocVQA、布局分析）的新机制，例如视觉token压缩、高分辨率建模或幻觉抑制策略，因此属于‘垂直领域封装’，不符合本研究关注的技术源头创新标准。
Abstract Optical Chemical Structure Recognition (OCSR) plays a pivotal role in modern chemical informatics, enabling the automated conversion of chemical structure images from scientific literature, patents, and educational materials into machine-readable molecular representations. This capability is essential for large-scale chemical data mining, drug discovery pipelines, and Large Language Model (LLM) applications in related domains. However, existing OCSR systems face significant challenges in accurately recognizing stereochemical information due to the subtle visual cues that distinguish stereoisomers, such as wedge and dash bonds, ring conformations, and spatial arrangements. To address these challenges, we propose MolSight, a comprehensive learning framework for OCSR that employs a three-stage training paradigm. In the first stage, we conduct pre-training on large-scale but noisy datasets to endow the model with fundamental perception capabilities for chemical structure images. In the second stage, we perform multi-granularity fine-tuning using datasets with richer supervisory signals, systematically exploring how auxiliary tasks-specifically chemical bond classification and atom localization-contribute to molecular formula recognition. Finally, we employ reinforcement learning for post-training optimization and introduce a novel stereochemical structure dataset. Remarkably, we find that even with MolSight's relatively compact parameter size, the Group Relative Policy Optimization (GRPO) algorithm can further enhance the model's performance on stereomolecular. Through extensive experiments across diverse datasets, our results demonstrate that MolSight achieves state-of-the-art performance in (stereo)chemical optical structure recognition.
Authors Wenrui Zhang, Xinggang Wang, Bin Feng, Wenyu Liu
Categories Computer Vision and Pattern Recognition
3
One Small Step in Latent, One Giant Leap for Pixels: Fast Latent Upscale Adapter for Your Diffusion Models
Title CN 潜在空间中的一小步，像素上的一大飞跃：用于扩散模型的快速潜在上采样适配器
Keywords 扩散模型 · 潜在空间上采样 · 图像生成 · 高效推理 · VAE
Summary 论文提出了Latent Upscaler Adapter（LUA），一种可在潜在空间中对扩散模型的隐变量进行上采样的轻量模块，支持2x和4x放大，无需修改原模型结构，显著降低了解码和上采样时间，并在多种VAE之间具有良好泛化性。实验表明其在效率和视觉质量上优于传统的图像空间超分方法。
Reason 该论文聚焦于图像生成领域中的高分辨率合成问题，提出在潜在空间进行上采样的轻量级模块LUA。尽管其高效处理高分辨率的能力具有一定吸引力，但其核心任务是扩散模型的图像生成加速与质量优化，属于纯图像生成范畴。根据负面清单，纯图像生成/修复类工作与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版理解、长文档推理）无直接关联，且无法迁移用于解决VLM在文档理解中的多模态对齐或细粒度识别问题。
Abstract Diffusion models struggle to scale beyond their training resolutions, as direct high-resolution sampling is slow and costly, while post-hoc image super-resolution (ISR) introduces artifacts and additional latency by operating after decoding. We present the Latent Upscaler Adapter (LUA), a lightweight module that performs super-resolution directly on the generator's latent code before the final VAE decoding step. LUA integrates as a drop-in component, requiring no modifications to the base model or additional diffusion stages, and enables high-resolution synthesis through a single feed-forward pass in latent space. A shared Swin-style backbone with scale-specific pixel-shuffle heads supports 2x and 4x factors and remains compatible with image-space SR baselines, achieving comparable perceptual quality with nearly 3x lower decoding and upscaling time (adding only +0.42 s for 1024 px generation from 512 px, compared to 1.87 s for pixel-space SR using the same SwinIR architecture). Furthermore, LUA shows strong generalization across the latent spaces of different VAEs, making it easy to deploy without retraining from scratch for each new decoder. Extensive experiments demonstrate that LUA closely matches the fidelity of native high-resolution generation while offering a practical and efficient path to scalable, high-fidelity image synthesis in modern diffusion pipelines.
Authors Aleksandr Razin, Danil Kazantsev, Ilya Makarov
Categories Computer Vision and Pattern Recognition
3
A Little More Like This: Text-to-Image Retrieval with Vision-Language Models Using Relevance Feedback WACV'26
Title CN 像这样再来一点：使用相关性反馈的视觉语言模型文本到图像检索
Keywords 视觉语言模型 · 文本到图像检索 · 相关性反馈 · 查询扩展 · 交互式检索
Summary 本文提出了一种基于相关性反馈的文本到图像检索增强方法，包括伪相关反馈（PRF）、生成式反馈（GRF）和基于Transformer的注意力反馈汇总器（AFS），可在推理阶段不依赖微调地提升VLM在Flickr30k和COCO上的检索性能。实验表明这些方法能有效提升MRR@5指标，尤其AFS在多轮检索中更稳定。
Reason 该论文研究的是通用视觉语言模型在文本到图像检索任务中通过相关性反馈提升性能的方法，属于通用VLM应用，且聚焦于Flickr30k和COCO等自然图像数据集。虽然其提出的反馈机制（如GRF、AFS）在理论上可能用于交互式文档检索，但未涉及任何文档理解核心痛点（如OCR幻觉、复杂版面解析、细粒度文本定位或长文档推理），也未针对文档模态进行建模改进。整体工作属于对通用VLM检索的增量优化，属于‘平行应用’范畴，缺乏向DIU迁移的方法论价值。
Abstract Large vision-language models (VLMs) enable intuitive visual search using natural language queries. However, improving their performance often requires fine-tuning and scaling to larger model variants. In this work, we propose a mechanism inspired by traditional text-based search to improve retrieval performance at inference time: relevance feedback. While relevance feedback can serve as an alternative to fine-tuning, its model-agnostic design also enables use with fine-tuned VLMs. Specifically, we introduce and evaluate four feedback strategies for VLM-based retrieval. First, we revise classical pseudo-relevance feedback (PRF), which refines query embeddings based on top-ranked results. To address its limitations, we propose generative relevance feedback (GRF), which uses synthetic captions for query refinement. Furthermore, we introduce an attentive feedback summarizer (AFS), a custom transformer-based model that integrates multimodal fine-grained features from relevant items. Finally, we simulate explicit feedback using ground-truth captions as an upper-bound baseline. Experiments on Flickr30k and COCO with the VLM backbones show that GRF, AFS, and explicit feedback improve retrieval performance by 3-5% in MRR@5 for smaller VLMs, and 1-3% for larger ones, compared to retrieval with no feedback. Moreover, AFS, similarly to explicit feedback, mitigates query drift and is more robust than GRF in iterative, multi-turn retrieval settings. Our findings demonstrate that relevance feedback can consistently enhance retrieval across VLMs and open up opportunities for interactive and adaptive visual search.
Authors Bulat Khaertdinov, Mirela Popa, Nava Tintarev
Comment Accepted to WACV'26
Categories Computer Vision and Pattern Recognition
3
Equivariant-Aware Structured Pruning for Efficient Edge Deployment: A Comprehensive Framework with Adaptive Fine-Tuning IEEE EdgeCom 2025
Title CN 面向边缘部署的等变感知结构化剪枝：一种具有自适应微调的综合框架
Keywords G-CNN · 结构化剪枝 · 等变性 · 模型压缩 · 边缘部署
Summary 本文提出了一种结合群等变卷积神经网络（G-CNN）与等变感知结构化剪枝的框架，用于在资源受限环境下部署紧凑且对几何变换鲁棒的模型。通过C4循环群实现旋转等变性，并设计了保持等变性质的结构化剪枝策略，配合自适应微调、知识蒸馏和INT8量化形成完整优化流程。在EuroSAT、CIFAR-10和Rotated MNIST上验证了有效性，实现了29.3%的参数减少并有效恢复精度。
Reason 该论文研究的是基于G-CNN的模型压缩与部署优化，核心关注点为旋转等变性在卫星图像和通用视觉任务中的高效推理。尽管其方法在几何变换鲁棒性和模型轻量化方面有贡献，但完全不涉及文档图像理解（DIU）的核心任务（如DocVQA、布局分析、表格识别等），也未提出可用于解决OCR幻觉、复杂排版或多步推理的通用机制。属于通用CV模型压缩方向，且应用场景明确为遥感图像，不符合DIU本题或可迁移工具的标准。
Abstract This paper presents a novel framework combining group equivariant convolutional neural networks (G-CNNs) with equivariant-aware structured pruning to produce compact, transformation-invariant models for resource-constrained environments. Equivariance to rotations is achieved through the C4 cyclic group via the e2cnn library,enabling consistent performance under geometric transformations while reducing computational overhead. Our approach introduces structured pruning that preserves equivariant properties by analyzing e2cnn layer structure and applying neuron-level pruning to fully connected components. To mitigate accuracy degradation, we implement adaptive fine-tuning that automatically triggers when accuracy drop exceeds 2%, using early stopping and learning rate scheduling for efficient recovery. The framework includes dynamic INT8 quantization and a comprehensive pipeline encompassing training, knowledge distillation, structured pruning, fine-tuning, and quantization. We evaluate our method on satellite imagery (EuroSAT) and standard benchmarks (CIFAR-10, Rotated MNIST) demonstrating effectiveness across diverse domains. Experimental results show 29.3% parameter reduction with significant accuracy recovery, demonstrating that structured pruning of equivariant networks achieves substantial compression while maintaining geometric robustness. Our pipeline provides a reproducible framework for optimizing equivariant models, bridging the gap between group-theoretic network design and practical deployment constraints, with particular relevance to satellite imagery analysis and geometric vision tasks.
Authors Mohammed Alnemari
Comment 8 pages, 5 tables, 1 figure. Accepted at IEEE EdgeCom 2025 (11th IEEE International Conference on Edge Computing and Scalable Cloud)
Categories Computer Vision and Pattern Recognition
3
Dual-domain Adaptation Networks for Realistic Image Super-resolution
Title CN 面向真实图像超分辨率的双域自适应网络
Keywords 图像超分辨率 · 域自适应 · 频域建模 · 低秩适配 · 真实图像
Summary 本文提出了一种双域自适应网络（DAN），用于将预训练的图像超分模型从合成数据迁移到真实场景。方法结合空间域参数选择性更新与低秩适配，并引入频域分支来恢复高频细节，在多个真实超分基准上表现优异。
Reason 该论文研究的是通用图像超分辨率中的域自适应问题，属于图像复原（image restoration）领域，虽然技术上涉及从合成域到真实域的迁移和频域建模，但其任务目标与文档图像理解（DIU）的核心痛点（如OCR幻觉、文本识别、布局分析、表格理解等）无直接关联。尽管高分辨率重建可能间接有助于文档图像的清晰度提升，但该方法并未针对文档特有的结构、语义或多模态对齐进行设计，也无法直接迁移用于解决VLM在DocVQA中的幻觉或细粒度定位问题。属于通用CV任务，且不涉及文档内容理解。
Abstract Realistic image super-resolution (SR) focuses on transforming real-world low-resolution (LR) images into high-resolution (HR) ones, handling more complex degradation patterns than synthetic SR tasks. This is critical for applications like surveillance, medical imaging, and consumer electronics. However, current methods struggle with limited real-world LR-HR data, impacting the learning of basic image features. Pre-trained SR models from large-scale synthetic datasets offer valuable prior knowledge, which can improve generalization, speed up training, and reduce the need for extensive real-world data in realistic SR tasks. In this paper, we introduce a novel approach, Dual-domain Adaptation Networks, which is able to efficiently adapt pre-trained image SR models from simulated to real-world datasets. To achieve this target, we first set up a spatial-domain adaptation strategy through selectively updating parameters of pre-trained models and employing the low-rank adaptation technique to adjust frozen parameters. Recognizing that image super-resolution involves recovering high-frequency components, we further integrate a frequency domain adaptation branch into the adapted model, which combines the spectral data of the input and the spatial-domain backbone's intermediate features to infer HR frequency maps, enhancing the SR result. Experimental evaluations on public realistic image SR benchmarks, including RealSR, D2CRealSR, and DRealSR, demonstrate the superiority of our proposed method over existing state-of-the-art models. Codes are available at: https://github.com/dummerchen/DAN.
Authors Chaowei Fang, Bolin Fu, De Cheng, Lechao Cheng, Guanbin Li
Categories Computer Vision and Pattern Recognition
3
EatGAN: An Edge-Attention Guided Generative Adversarial Network for Single Image Super-Resolution
Title CN EatGAN：一种用于单图像超分辨率的边缘注意力引导生成对抗网络
Keywords 图像超分辨率 · 生成对抗网络 · 边缘注意力 · 高频细节重建 · 结构一致性
Summary 本文提出了一种名为EatGAN的生成对抗网络，用于单图像超分辨率任务。该模型通过引入归一化边缘注意力（NEA）机制，在生成器中显式和隐式地利用边缘先验信息，结合边缘引导的混合残差块和复合损失函数，在失真和感知指标上均取得优异表现。
Reason 该论文属于图像超分辨率领域，尽管其方法在视觉任务中可能有潜在价值，但其核心贡献集中在自然图像的高频细节重建，并未针对文档图像理解（DIU）中的关键挑战（如OCR幻觉、复杂排版、文本结构建模）进行设计。虽然高分辨率处理对文档有益，但该方法依赖边缘先验的机制主要面向自然或动漫图像，迁移到文档场景（尤其是包含密集小字体、多栏布局等）的适用性有限，且无证据表明其能解决DIU特有的语义与布局联合建模问题。
Abstract Single-image super-resolution (SISR) is an important task in image processing, aiming to enhance the resolution of imaging systems. Recently, SISR has made a significant leap and achieved promising results with deep learning. GAN-based models stand out among all the deep learning models because of their excellent performance in perceiving quality. However, it is rather difficult for them to reconstruct realistic high-frequency details and achieve stable training. To solve these issues, we introduce an Edge-Attention guided Generative Adversarial Network (EatGAN), the first GAN-based SISR model that simultaneously leverages edge priors both explicitly and implicitly inside the generator, which (i) proposes a Normalized Edge Attention (NEA) mechanism based on channel-affine and spatial gating that transforms edge prior into lightweight, learnable modulation parameters and injects and fuses them multiple times in a (ii) edge-guided hybrid residual block, which progressively enforces structural consistency across scales; and (iii) a composite generator objective combining pixel, perceptual, edge-gradient, and adversarial terms. Experiments show consistent state-of-the-art across distortion-oriented benchmarks and perception oriented benchmarks. Notably, our model achieves 40.87 dB and 0.073 (LPIPS) on Manga 109, which indicates that reframing image priors from passive guidance into a controllable modulation primitive for generators can chart a practical path toward trustworthy, high-fidelity Super-Resolution.
Authors Penghao Rao, Tieyong Zeng
Comment 17 pages (8 pages of main text + 3 pages of reference + 6 pages of supplementary material)
Categories Computer Vision and Pattern Recognition
3
HDCompression: Hybrid-Diffusion Image Compression for Ultra-Low Bitrates PRICAI 2025
Title CN HDCompression：面向超低比特率的混合扩散图像压缩
Keywords 图像压缩 · 扩散模型 · VQ-VAE · 低比特率 · 生成模型
Summary 本文提出HDCompression，一种用于超低比特率下的混合扩散图像压缩框架，结合生成式VQ模型、扩散模型和传统学习型图像压缩（LIC）构建双流结构。通过扩散模型从真实输入中提取高质量保真信息，用于改进索引图预测、增强LIC流输出并优化基于VQ潜变量的图像重建。实验表明该方法在定量指标与视觉质量上均优于现有方法。
Reason 该论文研究的是通用图像压缩任务，尽管其方法结合了扩散模型与VQ建模用于提升重建质量，但核心目标是解决低码率下的视觉保真问题，而非文档理解中的结构解析、文本识别或逻辑推理。虽然理论上图像压缩技术可间接服务于高分辨率文档处理，但本文未涉及文本内容保持、OCR鲁棒性或文档布局建模等DIU核心痛点，属于通用视觉技术且无明确迁移路径到文档理解场景。
Abstract Image compression under ultra-low bitrates remains challenging for both conventional learned image compression (LIC) and generative vector-quantized (VQ) modeling. Conventional LIC suffers from severe artifacts due to heavy quantization, while generative VQ modeling gives poor fidelity due to the mismatch between learned generative priors and specific inputs. In this work, we propose Hybrid-Diffusion Image Compression (HDCompression), a dual-stream framework that utilizes both generative VQ-modeling and diffusion models, as well as conventional LIC, to achieve both high fidelity and high perceptual quality. Different from previous hybrid methods that directly use pre-trained LIC models to generate low-quality fidelity-preserving information from heavily quantized latent, we use diffusion models to extract high-quality complementary fidelity information from the ground-truth input, which can enhance the system performance in several aspects: improving index map prediction, enhancing the fidelity-preserving output of the LIC stream, and refining conditioned image reconstruction with VQ-latent correction. In addition, our diffusion model is based on a dense representative vector (DRV), which is lightweight with very simple sampling schedulers. Extensive experiments demonstrate that our HDCompression outperforms the previous conventional LIC, generative VQ-modeling, and hybrid frameworks in both quantitative metrics and qualitative visualization, providing balanced robust compression performance at ultra-low bitrates.
Authors Lei Lu, Yize Li, Yanzhi Wang, Wei Wang, Wei Jiang
Comment Accepted by PRICAI 2025 (Oral Presentation)
Categories Computer Vision and Pattern Recognition
3
A Multi-Stage Optimization Framework for Deploying Learned Image Compression on FPGAs
Title CN 一种在FPGA上部署学习型图像压缩的多阶段优化框架
Keywords 图像压缩 · FPGA部署 · 量化 · 硬件优化 · 深度学习
Summary 本文提出了一种用于在FPGA上部署学习型图像压缩模型的多阶段优化框架，包括动态范围感知量化（DRAQ）、混合精度搜索和通道剪枝技术，有效降低了模型复杂度并缓解了量化带来的性能下降。然而，该工作属于边缘计算与硬件加速领域，与文档理解任务无直接关联。
Reason 该论文聚焦于深度学习图像压缩模型在FPGA上的高效部署，核心贡献是量化、剪枝和硬件适配优化。尽管涉及视觉表示和压缩思想，但其目标与文档图像理解（DIU）中的语义理解、布局分析或VLM推理无关。未解决OCR幻觉、复杂排版建模或长文档推理等DIU核心问题，属于特定硬件部署的工程优化，无法迁移为DIU的方法论工具。
Abstract Deep learning-based image compression (LIC) has achieved state-of-the-art rate-distortion (RD) performance, yet deploying these models on resource-constrained FPGAs remains a major challenge. This work presents a complete, multi-stage optimization framework to bridge the gap between high-performance floating-point models and efficient, hardware-friendly integer-based implementations. First, we address the fundamental problem of quantization-induced performance degradation. We propose a Dynamic Range-Aware Quantization (DRAQ) method that uses statistically-calibrated activation clipping and a novel weight regularization scheme to counteract the effects of extreme data outliers and large dynamic ranges, successfully creating a high-fidelity 8-bit integer model. Second, building on this robust foundation, we introduce two hardware-aware optimization techniques tailored for FPGAs. A progressive mixed-precision search algorithm exploits FPGA flexibility to assign optimal, non-uniform bit-widths to each layer, minimizing complexity while preserving performance. Concurrently, a channel pruning method, adapted to work with the Generalized Divisive Normalization (GDN) layers common in LIC, removes model redundancy by eliminating inactive channels. Our comprehensive experiments show that the foundational DRAQ method reduces the BD-rate overhead of a GDN-based model from $30\%$ to $6.3\%$. The subsequent hardware-aware optimizations further reduce computational complexity by over $20\%$ with negligible impact on RD performance, yielding a final model that is both state-of-the-art in efficiency and superior in quality to existing FPGA-based LIC implementations.
Authors Jiaxun Fang, Li Chen
Categories Computer Vision and Pattern Recognition
3
Composed Object Retrieval: Object-level Retrieval via Composed Expressions
Title CN 组合对象检索：通过组合表达式实现对象级检索
Keywords 组合表达式 · 对象级检索 · 细粒度定位 · 多模态检索 · 区域对比学习
Summary 本文提出了组合对象检索（COR）这一新任务，目标是基于参考对象和文本描述的组合表达式实现对象级别的精确检索与分割，并构建了包含12.7万样本的大规模数据集COR127K以及统一的端到端模型CORE。实验表明CORE在已知和新类别上均显著优于现有方法。
Reason 该论文提出了一种新的多模态任务——组合对象检索（COR），旨在根据结合参考对象和文本描述的组合表达式来检索并分割特定对象。虽然其在细粒度视觉定位方面有一定创新，但核心任务属于通用图像理解中的细粒度检索与分割，未涉及文档结构、文本布局、OCR或文档特有的复杂排版问题。尽管其提出的CORE模型在区域级对比学习和视觉-文本交互机制上具有技术价值，但这些方法并未针对文档图像理解（DIU）中的典型痛点（如密集文本识别、表格解析、长文档推理、OCR幻觉等）进行设计，且无证据表明可直接迁移应用于DocVQA或相关任务。因此属于无关领域的平行研究，不符合本方向关注重点。
Abstract Retrieving fine-grained visual content based on user intent remains a challenge in multi-modal systems. Although current Composed Image Retrieval (CIR) methods combine reference images with retrieval texts, they are constrained to image-level matching and cannot localize specific objects. To this end, we propose Composed Object Retrieval (COR), a brand-new task that goes beyond image-level retrieval to achieve object-level precision, allowing the retrieval and segmentation of target objects based on composed expressions combining reference objects and retrieval texts. COR presents significant challenges in retrieval flexibility, which requires systems to identify arbitrary objects satisfying composed expressions while avoiding semantically similar but irrelevant negative objects within the same scene. We construct COR127K, the first large-scale COR benchmark that contains 127,166 retrieval triplets with various semantic transformations in 408 categories. We also present CORE, a unified end-to-end model that integrates reference region encoding, adaptive visual-textual interaction, and region-level contrastive learning. Extensive experiments demonstrate that CORE significantly outperforms existing models in both base and novel categories, establishing a simple and effective baseline for this challenging task while opening new directions for fine-grained multi-modal retrieval research. We will publicly release both the dataset and the model at https://github.com/wangtong627/COR.
Authors Tong Wang, Guanyu Yang, Nian Liu, Zongyan Han, Jinxing Zhou, Salman Khan, Fahad Shahbaz Khan
Categories Computer Vision and Pattern Recognition
3
Performance of Conformal Prediction in Capturing Aleatoric Uncertainty IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2026
Title CN 在捕捉偶然不确定性中 conformal prediction 的性能
Keywords conformal prediction · uncertainty quantification · aleatoric uncertainty · human annotations · classification
Summary 本文评估了三种conformal prediction方法在八个深度学习模型和四个多标注者数据集上的表现，通过分析预测集大小与人类标注多样性之间的相关性，检验其捕捉偶然不确定性的能力。结果显示，大多数情况下相关性较弱，表明当前conformal prediction生成的预测集虽能保证覆盖真实类别，但在反映人类标注分歧和内在模糊性方面存在局限。
Reason 该论文研究conformal prediction在量化数据集中由类别重叠引起的偶然不确定性（aleatoric uncertainty）方面的有效性，核心关注多标签标注场景下的预测集与人类标注一致性的相关性。尽管不确定性估计在文档图像理解中具有潜在价值（如OCR置信度校准），但本工作并未针对文档结构、布局分析或VLM推理中的具体问题设计方法，也未提出可直接迁移至DIU领域的新型架构、训练策略或推理机制。其评估完全基于分类任务和人工标注一致性，属于通用视觉不确定性建模范畴，对解决OCR幻觉、复杂排版解析或长文档推理等核心DIU痛点缺乏直接指导意义。
Abstract Conformal prediction is a model-agnostic approach to generating prediction sets that cover the true class with a high probability. Although its prediction set size is expected to capture aleatoric uncertainty, there is a lack of evidence regarding its effectiveness. The literature presents that prediction set size can upper-bound aleatoric uncertainty or that prediction sets are larger for difficult instances and smaller for easy ones, but a validation of this attribute of conformal predictors is missing. This work investigates how effectively conformal predictors quantify aleatoric uncertainty, specifically the inherent ambiguity in datasets caused by overlapping classes. We perform this by measuring the correlation between prediction set sizes and the number of distinct labels assigned by human annotators per instance. We further assess the similarity between prediction sets and human-provided annotations. We use three conformal prediction approaches to generate prediction sets for eight deep learning models trained on four datasets. The datasets contain annotations from multiple human annotators (ranging from five to fifty participants) per instance, enabling the identification of class overlap. We show that the vast majority of the conformal prediction outputs show a very weak to weak correlation with human annotations, with only a few showing moderate correlation. These findings underscore the necessity of critically reassessing the prediction sets generated using conformal predictors. While they can provide a higher coverage of the true classes, their capability in capturing aleatoric uncertainty and generating sets that align with human annotations remains limited.
Authors Misgina Tsighe Hagos, Claes Lundström
Comment Accepted at the IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2026
Categories Computer Vision and Pattern Recognition
3
Diversity Has Always Been There in Your Visual Autoregressive Models
Title CN 多样性一直存在于你的视觉自回归模型中
Keywords 视觉自回归模型 · 生成多样性 · 特征调控 · 图像生成 · 无需训练
Summary 本文提出DiverseVAR方法，通过抑制输入中关键特征分量并增强输出中的对应分量，在无需重新训练的前提下恢复视觉自回归模型（VAR）的生成多样性，缓解多样性崩溃问题。实验表明该方法能显著提升生成结果的多样性，同时保持高质量图像合成能力。
Reason 该论文研究的是视觉自回归生成模型（VAR）中的生成多样性问题，属于图像生成领域，其核心任务是提升生成图像的多样性，而非文档理解或视觉-语言推理。尽管其方法可能涉及特征图的调控机制，但整体工作仍聚焦于通用图像生成质量优化，与文档图像理解（DIU）中的OCR幻觉、复杂排版解析、细粒度定位等核心痛点无直接关联。且未见任何可迁移至文档场景的架构或训练范式创新，属于无关领域的技术探索。
Abstract Visual Autoregressive (VAR) models have recently garnered significant attention for their innovative next-scale prediction paradigm, offering notable advantages in both inference efficiency and image quality compared to traditional multi-step autoregressive (AR) and diffusion models. However, despite their efficiency, VAR models often suffer from the diversity collapse i.e., a reduction in output variability, analogous to that observed in few-step distilled diffusion models. In this paper, we introduce DiverseVAR, a simple yet effective approach that restores the generative diversity of VAR models without requiring any additional training. Our analysis reveals the pivotal component of the feature map as a key factor governing diversity formation at early scales. By suppressing the pivotal component in the model input and amplifying it in the model output, DiverseVAR effectively unlocks the inherent generative potential of VAR models while preserving high-fidelity synthesis. Empirical results demonstrate that our approach substantially enhances generative diversity with only neglectable performance influences. Our code will be publicly released at https://github.com/wangtong627/DiverseVAR.
Authors Tong Wang, Guanyu Yang, Nian Liu, Kai Wang, Yaxing Wang, Abdelrahman M Shaker, Salman Khan, Fahad Shahbaz Khan, Senmao Li
Categories Computer Vision and Pattern Recognition
3
PathAgent: Toward Interpretable Analysis of Whole-slide Pathology Images via Large Language Model-based Agentic Reasoning
Title CN PathAgent：通过基于大语言模型的代理推理实现全切片病理图像的可解释分析
Keywords PathAgent · 全切片图像分析 · 代理推理 · 可解释性 · 医学图像
Summary 本文提出PathAgent，一种无需训练的基于大语言模型的代理框架，用于全切片病理图像的可解释分析。该系统通过Navigator模块定位关键区域，Perceptor模块提取形态学视觉特征，并在Executor中生成自然语言推理轨迹，形成显式的思维链，从而实现透明且可追溯的预测。实验表明其在多个数据集上具有良好的零样本泛化能力，并得到病理医生的认可。
Reason 该论文属于医学图像分析领域的垂直应用，尽管使用了Agent框架和链式推理提升可解释性，但其核心是将现有LLM Agent技术应用于全切片病理图像（WSI）这一特定场景，属于典型的‘平行下游应用’。虽然其推理流程设计有一定启发性，但方法本身未针对文档图像理解中的OCR幻觉、复杂版式或长文档逻辑结构等核心痛点提出可迁移的底层创新，对DIU领域的方法论贡献有限。
Abstract Analyzing whole-slide images (WSIs) requires an iterative, evidence-driven reasoning process that parallels how pathologists dynamically zoom, refocus, and self-correct while collecting the evidence. However, existing computational pipelines often lack this explicit reasoning trajectory, resulting in inherently opaque and unjustifiable predictions. To bridge this gap, we present PathAgent, a training-free, large language model (LLM)-based agent framework that emulates the reflective, stepwise analytical approach of human experts. PathAgent can autonomously explore WSI, iteratively and precisely locating significant micro-regions using the Navigator module, extracting morphology visual cues using the Perceptor, and integrating these findings into the continuously evolving natural language trajectories in the Executor. The entire sequence of observations and decisions forms an explicit chain-of-thought, yielding fully interpretable predictions. Evaluated across five challenging datasets, PathAgent exhibits strong zero-shot generalization, surpassing task-specific baselines in both open-ended and constrained visual question-answering tasks. Moreover, a collaborative evaluation with human pathologists confirms PathAgent's promise as a transparent and clinically grounded diagnostic assistant.
Authors Jingyun Chen, Linghan Cai, Zhikang Wang, Yi Huang, Songhan Jiang, Shenjin Huang, Hongpeng Wang, Yongbing Zhang
Comment 11 pages, 6 figures
Categories Computer Vision and Pattern Recognition
3
DReX: Pure Vision Fusion of Self-Supervised and Convolutional Representations for Image Complexity Prediction
Title CN DReX：自监督与卷积表示的纯视觉融合用于图像复杂度预测
Keywords 图像复杂度预测 · 视觉表示融合 · 自监督学习 · DINOv3 · ResNet
Summary 本文提出DReX模型，通过融合DINOv3和ResNet-50的视觉表示，在不使用语言信息的情况下实现图像复杂度的人类对齐预测，并在IC9600等数据集上取得SOTA性能。模型利用注意力机制结合多尺度卷积特征与Transformer语义特征，证明纯视觉路径即可有效建模人类感知复杂性。
Reason 该论文研究的是通用图像复杂度预测任务，属于通用计算机视觉问题，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的视觉特征融合方法具有一定技术价值，但无法直接迁移解决文档理解中的OCR幻觉、密集文本、细粒度定位或长文档推理等关键痛点，属于无关领域的常规工作。
Abstract Visual complexity prediction is a fundamental problem in computer vision with applications in image compression, retrieval, and classification. Understanding what makes humans perceive an image as complex is also a long-standing question in cognitive science. Recent approaches have leveraged multimodal models that combine visual and linguistic representations, but it remains unclear whether language information is necessary for this task. We propose DReX (DINO-ResNet Fusion), a vision-only model that fuses self-supervised and convolutional representations through a learnable attention mechanism to predict image complexity. Our architecture integrates multi-scale hierarchical features from ResNet-50 with semantically rich representations from DINOv3 ViT-S/16, enabling the model to capture both low-level texture patterns and high-level semantic structure. DReX achieves state-of-the-art performance on the IC9600 benchmark (Pearson r = 0.9581), surpassing previous methods--including those trained on multimodal image-text data--while using approximately 21.5x fewer learnable parameters. Furthermore, DReX generalizes robustly across multiple datasets and metrics, achieving superior results on Pearson and Spearman correlation, Root Mean Square Error (RMSE), and Mean Absolute Error (MAE). Ablation and attention analyses confirm that DReX leverages complementary cues from both backbones, with the DINOv3 [CLS] token enhancing sensitivity to visual complexity. Our findings suggest that visual features alone can be sufficient for human-aligned complexity prediction and that, when properly fused, self-supervised transformers and supervised deep convolutional neural networks offer complementary and synergistic benefits for this task.
Authors Jonathan Skaza, Parsa Madinei, Ziqi Wen, Miguel Eckstein
Comment 8 pages
Categories Computer Vision and Pattern Recognition
3
From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models
Title CN 从感知到推理：深度思考赋能多模态大语言模型
Keywords 多模态思维链 · MCoT · 综述 · 推理 · 多模态大模型
Summary 本文系统回顾了多模态大语言模型中的思维链（MCoT）方法，分析了其理论动机、主流范式、训练与推理阶段的设计、评估基准及应用场景，并讨论了当前挑战与未来方向。整体为综述性质，无原创性技术贡献。
Reason 该论文是一篇关于多模态思维链（MCoT）的综述性文章，未提出新的方法或技术改进。尽管MCoT在理论上对文档图像理解中的复杂推理任务有潜在价值，但本文属于‘平行应用’类的总结工作，缺乏针对DIU领域的具体创新或可迁移的底层技术突破。
Abstract With the remarkable success of Multimodal Large Language Models (MLLMs) in perception tasks, enhancing their complex reasoning capabilities has emerged as a critical research focus. Existing models still suffer from challenges such as opaque reasoning paths and insufficient generalization ability. Chain-of-Thought (CoT) reasoning, which has demonstrated significant efficacy in language models by enhancing reasoning transparency and output interpretability, holds promise for improving model reasoning capabilities when extended to the multimodal domain. This paper provides a systematic review centered on "Multimodal Chain-of-Thought" (MCoT). First, it analyzes the background and theoretical motivations for its inception from the perspectives of technical evolution and task demands. Then, it introduces mainstream MCoT methods from three aspects: CoT paradigms, the post-training stage, and the inference stage, while also analyzing their underlying mechanisms. Furthermore, the paper summarizes existing evaluation benchmarks and metrics, and discusses the application scenarios of MCoT. Finally, it analyzes the challenges currently facing MCoT and provides an outlook on its future research directions.
Authors Wenxin Zhu, Andong Chen, Yuchen Song, Kehai Chen, Conghui Zhu, Ziyan Chen, Tiejun Zhao
Comment Survey; 7 figures, 3 tables, 44 pages
Categories Computer Vision and Pattern Recognition, Computation and Language
3
Two Heads Better than One: Dual Degradation Representation for Blind Super-Resolution
Title CN 双头优于单头：用于盲超分辨率的双重退化表示
Keywords 图像超分辨率 · 盲退化建模 · 双分支网络 · 图像复原 · 计算机视觉
Summary 本文提出一种双分支退化提取网络，用于盲超分辨率任务，通过无监督方式分别建模模糊和噪声两种退化类型，并据此自适应地进行超分辨率重建，在多个基准上实现了SOTA性能。然而，该工作聚焦于自然图像的低级视觉恢复，与文档图像理解任务关联较弱。
Reason 该论文研究的是图像盲超分辨率中的退化建模问题，属于通用图像复原（Image Restoration）领域。虽然其提出的双分支退化表示可能对高分辨率视觉输入处理有一定启发，但方法核心针对的是低级视觉任务（如去噪、去模糊），并未涉及文档结构、文本语义或OCR相关建模。无法直接迁移解决文档图像理解中的OCR幻觉、复杂排版或长文档推理等核心痛点，属于无关领域的技术工作。
Abstract Previous methods have demonstrated remarkable performance in single image super-resolution (SISR) tasks with known and fixed degradation (e.g., bicubic downsampling). However, when the actual degradation deviates from these assumptions, these methods may experience significant declines in performance. In this paper, we propose a Dual Branch Degradation Extractor Network to address the blind SR problem. While some blind SR methods assume noise-free degradation and others do not explicitly consider the presence of noise in the degradation model, our approach predicts two unsupervised degradation embeddings that represent blurry and noisy information. The SR network can then be adapted to blur embedding and noise embedding in distinct ways. Furthermore, we treat the degradation extractor as a regularizer to capitalize on differences between SR and HR images. Extensive experiments on several benchmarks demonstrate our method achieves SOTA performance in the blind SR problem.
Authors Hsuan Yuan, Shao-Yu Weng, I-Hsuan Lo, Wei-Chen Chiu, Yu-Syuan Xu, Hao-Chien Hsueh, Jen-Hui Chuang, Ching-Chun Huang
Categories Computer Vision and Pattern Recognition
3
Parameter-Efficient MoE LoRA for Few-Shot Multi-Style Editing
Title CN 用于少样本多风格编辑的参数高效MoE LoRA
Keywords 图像编辑 · 少样本学习 · LoRA · Mixture of Experts · 扩散模型
Summary 本文提出一种基于Mixture-of-Experts和LoRA的少样本图像风格编辑框架，在Diffusion in Transformer架构中引入风格特定与共享路由机制，并通过重要性评分自动选择最优秩。实验表明其在少量数据下能有效编辑多种艺术风格，同时保持较低参数开销。
Reason 该论文聚焦于图像编辑中的风格迁移问题，属于纯图像生成与编辑领域，使用Diffusion模型和LoRA进行多风格适配。尽管其参数高效微调方法具有一定通用性，但整体任务与文档图像理解（DIU）的核心痛点如OCR幻觉、文本识别、排版分析、DocVQA等无直接关联。方法针对的是艺术风格编辑，迁移到文档场景（如结构化理解或视觉-语言推理）极为困难，且未涉及任何文本内容建模或布局语义处理。
Abstract In recent years, image editing has garnered growing attention. However, general image editing models often fail to produce satisfactory results when confronted with new styles. The challenge lies in how to effectively fine-tune general image editing models to new styles using only a limited amount of paired data. To address this issue, this paper proposes a novel few-shot style editing framework. For this task, we construct a benchmark dataset that encompasses five distinct styles. Correspondingly, we propose a parameter-efficient multi-style Mixture-of-Experts Low-Rank Adaptation (MoE LoRA) with style-specific and style-shared routing mechanisms for jointly fine-tuning multiple styles. The style-specific routing ensures that different styles do not interfere with one another, while the style-shared routing adaptively allocates shared MoE LoRAs to learn common patterns. Our MoE LoRA can automatically determine the optimal ranks for each layer through a novel metric-guided approach that estimates the importance score of each single-rank component. Additionally, we explore the optimal location to insert LoRA within the Diffusion in Transformer (DiT) model and integrate adversarial learning and flow matching to guide the diffusion training process. Experimental results demonstrate that our proposed method outperforms existing state-of-the-art approaches with significantly fewer LoRA parameters.
Authors Cong Cao, Yujie Xu, Xiaodong Xu
Categories Computer Vision and Pattern Recognition
3
UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation
Title CN UniModel：一种用于统一多模态理解与生成的纯视觉框架
Keywords 多模态学习 · 扩散模型 · 视觉化文本 · 统一框架 · 像素级生成
Summary 本文提出UniModel，一种基于扩散Transformer的统一多模态框架，通过将文本渲染为图像，将理解和生成任务统一为像素到像素的转换。尽管实现了跨模态对齐和循环一致性生成，但其方法依赖于图像级文本渲染，缺乏对字符级语义和布局结构的精细建模能力，不适合高精度文档理解任务。
Reason 该论文提出将文本渲染为图像（painted text）并在纯像素空间中进行多模态理解和生成，属于将多模态任务统一到视觉空间的尝试。然而，其核心方法基于扩散模型和像素级生成，主要面向通用视觉-语言生成任务（如文生图、图生文），并未针对文档图像中的细粒度结构理解、OCR幻觉、复杂版式建模等DIU核心痛点设计。虽然‘视觉化文本’的思想看似新颖，但将自然语言问题和答案以图像形式处理，在文档理解场景下会导致精度损失、无法支持精确字符级交互，且难以迁移至需要高准确率VQA或结构化提取的任务。整体属于通用多模态架构的平行应用，无明确可迁移至DIU的方法论创新。
Abstract We present UniModel, a unified generative model that jointly supports visual understanding and visual generation within a single pixel-to-pixel diffusion framework. Our goal is to achieve unification along three axes: the model, the tasks, and the representations. At the representation level, we eliminate modality discrepancies by mapping both text and images into a shared visual space: textual prompts are rendered as painted text images on a clean canvas, and all inputs and outputs are treated purely as RGB pixels. This yields a fully vision-native formulation of multimodal learning. At the task level, a broad range of vision-language problems are cast as pixel-to-pixel transformations in this visual space. For understanding tasks, the model takes an RGB image and produces a painted text image that visually encodes the semantic prediction. For generation tasks, painted text images serve as visual conditions that guide realistic and semantically aligned image synthesis. Captioning and text-to-image generation thus become different directions of the same underlying visual translation process. At the model level, we instantiate a single Unified Diffusion Transformer trained with rectified flow in pixel space. A shared backbone jointly learns bidirectional mappings between natural images and painted text images, with lightweight task embeddings to specify the desired direction. Experiments on text-to-image synthesis and image-to-text understanding demonstrate strong cross-modal alignment and emergent controllability such as cycle-consistent image-caption-image loops. Our initial exploration suggests that unifying model, tasks, and representations in a single visual space is a promising paradigm for general-purpose multimodal intelligence.
Authors Chi Zhang, Jiepeng Wang, Youming Wang, Yuanzhi Liang, Xiaoyan Yang, Zuoxin Li, Haibin Huang, Xuelong Li
Categories Computer Vision and Pattern Recognition
3
Align & Invert: Solving Inverse Problems with Diffusion and Flow-based Models via Representational Alignment
Title CN 对齐与反转：通过扩散和基于流的模型在表示对齐下求解逆问题
Keywords 扩散模型 · 表示对齐 · 逆问题 · 生成先验 · DINOv2
Summary 本文提出一种名为表示对齐（REPA）的方法，通过将扩散或基于流的生成模型与预训练自监督视觉编码器（如DINOv2）的内部表示对齐，在无真实标签的情况下提升逆问题（如超分、去模糊）的重建质量。理论分析表明REPA正则化与DINOv2嵌入空间中的散度有关，并能引导模型表示趋向干净图像的特征。实验验证了该方法在多个逆问题上的通用性和效率优势。
Reason 该论文研究的是通用图像逆问题（如超分辨率、去模糊、修复）中的生成模型先验使用，核心方法是通过与DINOv2等自监督编码器的表示对齐来提升重建质量。尽管其提出的REPA机制在理论上对感知保真度有增益，但其应用场景和目标均非文档图像理解（DIU）的核心任务（如DocVQA、布局分析、表格识别等）。虽然高分辨率重建技术可能间接有助于处理扫描文档的清晰度问题，但该方法并未针对OCR幻觉、密集文本建模、复杂排版结构或长文档推理等DIU关键痛点设计，迁移路径不直接，属于通用视觉方法在逆成像任务上的改进，不符合‘上游军火库’中可明确迁移的标准。
Abstract Enforcing alignment between the internal representations of diffusion or flow-based generative models and those of pretrained self-supervised encoders has recently been shown to provide a powerful inductive bias, improving both convergence and sample quality. In this work, we extend this idea to inverse problems, where pretrained generative models are employed as priors. We propose applying representation alignment (REPA) between diffusion or flow-based models and a pretrained self-supervised visual encoder, such as DINOv2, to guide the reconstruction process at inference time. Although ground-truth signals are unavailable in inverse problems, we show that aligning model representations with approximate target features can substantially enhance reconstruction fidelity and perceptual realism. We provide theoretical results showing (a) the relation between the REPA regularization and a divergence measure in the DINOv2 embedding space, and (b) how REPA updates steer the model's internal representations toward those of the clean image. These results offer insights into the role of REPA in improving perceptual fidelity. Finally, we demonstrate the generality of our approach by integrating it into multiple state-of-the-art inverse problem solvers. Extensive experiments on super-resolution, box inpainting, Gaussian deblurring, and motion deblurring confirm that our method consistently improves reconstruction quality across tasks, while also providing substantial efficiency gains by reducing the number of required discretization steps without compromising the performance of the underlying solver.
Authors Loukas Sfountouris, Giannis Daras, Paris Giampouras
Categories Computer Vision and Pattern Recognition
3
The Joint Gromov Wasserstein Objective for Multiple Object Matching
Title CN 用于多对象匹配的联合Gromov-Wasserstein目标
Keywords Gromov-Wasserstein · 最优传输 · 多对象匹配 · 点云匹配 · 形状对齐
Summary 本文提出了联合Gromov-Wasserstein（JGW）目标函数，将传统的成对Gromov-Wasserstein距离扩展到支持多个对象间的同步匹配，适用于部分同构分布的识别，并在合成与真实数据上验证了其在多形状匹配任务中的准确性和效率优势。
Reason 该论文研究的是度量空间中多个对象之间的匹配问题，核心应用场景为计算机图形学和结构生物学中的形状匹配（如几何图形和生物分子复合物），属于通用视觉匹配方法的拓展。尽管Gromov-Wasserstein距离在理论上具有跨域匹配潜力，但本文未涉及文档结构、文本布局或任何文档图像理解任务，也未提供可迁移至DIU领域的机制（如处理OCR幻觉、复杂排版或多步推理）。其方法聚焦于点云表示下的形状对齐，与文档图像中的语义理解、细粒度定位或VLM架构无关，属于非直接相关的通用视觉方法，无法解决文档理解的核心痛点。
Abstract The Gromov-Wasserstein (GW) distance serves as a powerful tool for matching objects in metric spaces. However, its traditional formulation is constrained to pairwise matching between single objects, limiting its utility in scenarios and applications requiring multiple-to-one or multiple-to-multiple object matching. In this paper, we introduce the Joint Gromov-Wasserstein (JGW) objective and extend the original framework of GW to enable simultaneous matching between collections of objects. Our formulation provides a non-negative dissimilarity measure that identifies partially isomorphic distributions of mm-spaces, with point sampling convergence. We also show that the objective can be formulated and solved for point cloud object representations by adapting traditional algorithms in Optimal Transport, including entropic regularization. Our benchmarking with other variants of GW for partial matching indicates superior performance in accuracy and computational efficiency of our method, while experiments on both synthetic and real-world datasets show its effectiveness for multiple shape matching, including geometric shapes and biomolecular complexes, suggesting promising applications for solving complex matching problems across diverse domains, including computer graphics and structural biology.
Authors Aryan Tajmir Riahi, Khanh Dao Duc
Categories Computer Vision and Pattern Recognition
3
CORA: Consistency-Guided Semi-Supervised Framework for Reasoning Segmentation WACV 2026
Title CN CORA：基于一致性引导的半监督推理分割框架
Keywords 推理分割 · 半监督学习 · 多模态大模型 · 一致性过滤 · 伪标签
Summary 本文提出CORA，一种用于推理分割的半监督框架，通过条件化视觉指令、基于多模态大模型输出一致性的伪标签过滤机制以及token级对比对齐，在仅使用少量标注图像的情况下提升了模型在Cityscapes和PanNuke数据集上的性能。核心思想是利用语义等价查询下输出的一致性来筛选高质量伪标签，从而增强模型泛化能力。
Reason 该论文研究的是通用场景下的推理分割（reasoning segmentation），其任务设定聚焦于城市街景（Cityscapes）和病理图像（PanNuke），属于通用视觉或特定领域图像理解，而非文档图像理解（DIU）的核心任务（如DocVQA、表格识别、版面分析等）。虽然使用了多模态大模型和伪标签的一致性过滤机制，但其方法设计面向的是像素级语义分割任务，且未涉及文档特有的挑战（如OCR幻觉、文本密集排布、长文档结构建模）。其‘一致性过滤’虽有一定方法价值，但迁移至文档理解场景的路径不明确，且不属于DIU本题工作。整体属于将VLM应用于特定下游任务的‘Wrapper’类工作，命中负面清单第1条。
Abstract Reasoning segmentation seeks pixel-accurate masks for targets referenced by complex, often implicit instructions, requiring context-dependent reasoning over the scene. Recent multimodal language models have advanced instruction following segmentation, yet generalization remains limited. The key bottleneck is the high cost of curating diverse, high-quality pixel annotations paired with rich linguistic supervision leading to brittle performance under distribution shift. Therefore, we present CORA, a semi-supervised reasoning segmentation framework that jointly learns from limited labeled data and a large corpus of unlabeled images. CORA introduces three main components: 1) conditional visual instructions that encode spatial and contextual relationships between objects; 2) a noisy pseudo-label filter based on the consistency of Multimodal LLM's outputs across semantically equivalent queries; and 3) a token-level contrastive alignment between labeled and pseudo-labeled samples to enhance feature consistency. These components enable CORA to perform robust reasoning segmentation with minimal supervision, outperforming existing baselines under constrained annotation settings. CORA achieves state-of-the-art results, requiring as few as 100 labeled images on Cityscapes, a benchmark dataset for urban scene understanding, surpassing the baseline by $+2.3\%$. Similarly, CORA improves performance by $+2.4\%$ with only 180 labeled images on PanNuke, a histopathology dataset.
Authors Prantik Howlader, Hoang Nguyen-Canh, Srijan Das, Jingyi Xu, Hieu Le, Dimitris Samaras
Comment WACV 2026 accepted
Categories Computer Vision and Pattern Recognition
3
A Simple Yet Strong Baseline for Long-Term Conversational Memory of LLM Agents
Title CN LLM代理长期对话记忆的一个简单但强大的基线
Keywords 对话记忆 · LLM Agent · 事件表示 · EDU分解 · 图检索
Summary 本文提出一种基于事件语义的对话记忆表示方法，将对话历史分解为带有参与者、时间线索和局部上下文的简化话语单元（EDU），并构建异构图以支持关联检索。通过稠密检索与LLM过滤结合，在保持较短上下文的同时提升了长期对话的记忆性能。然而，该工作未涉及视觉或文档模态，仅适用于纯文本对话场景。
Reason 该论文聚焦于对话式AI代理的长期记忆问题，属于典型的Agent应用设计，尽管其事件中心化的记忆表示和图结构检索机制具有一定方法性，但整体仍围绕通用对话场景展开，并未针对文档图像理解中的核心痛点（如OCR幻觉、复杂排版、细粒度定位或高分辨率处理）提出可迁移的技术路径。其EDU分解与图传播机制虽可用于信息组织，但缺乏与视觉输入或多模态对齐的结合，难以直接迁移到DocVQA或长文档推理任务中。
Abstract LLM-based conversational agents still struggle to maintain coherent, personalized interaction over many sessions: fixed context windows limit how much history can be kept in view, and most external memory approaches trade off between coarse retrieval over large chunks and fine-grained but fragmented views of the dialogue. Motivated by neo-Davidsonian event semantics, we propose an event-centric alternative that represents conversational history as short, event-like propositions which bundle together participants, temporal cues, and minimal local context, rather than as independent relation triples or opaque summaries. In contrast to work that aggressively compresses or forgets past content, our design aims to preserve information in a non-compressive form and make it more accessible, rather than more lossy. Concretely, we instruct an LLM to decompose each session into enriched elementary discourse units (EDUs) -- self-contained statements with normalized entities and source turn attributions -- and organize sessions, EDUs, and their arguments in a heterogeneous graph that supports associative recall. On top of this representation we build two simple retrieval-based variants that use dense similarity search and LLM filtering, with an optional graph-based propagation step to connect and aggregate evidence across related EDUs. Experiments on the LoCoMo and LongMemEval$_S$ benchmarks show that these event-centric memories match or surpass strong baselines, while operating with much shorter QA contexts. Our results suggest that structurally simple, event-level memory provides a principled and practical foundation for long-horizon conversational agents. Our code and data will be released at https://github.com/KevinSRR/EMem.
Authors Sizhe Zhou
Comment Work in progress
Categories Computation and Language
3
E$^3$-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models
Title CN E³-Pruner：面向高效、经济且有效的大型语言模型层剪枝方法
Keywords 模型压缩 · 层剪枝 · 知识蒸馏 · 大语言模型 · 推理加速
Summary 本文提出E³-Pruner，一种面向大型语言模型的层剪枝框架，通过可微分掩码优化和熵感知自适应知识蒸馏，在保持高性能的同时显著降低训练开销并提升推理效率。实验显示其在Qwen3-32B上剪掉25%层后仍保持接近原始模型的准确率，并实现1.33倍推理加速。
Reason 该论文研究的是大语言模型的层剪枝技术，属于纯NLP模型压缩领域，未涉及任何视觉或多模态内容。尽管其在推理效率和训练成本上有创新，但方法完全基于文本模态，无法直接迁移解决文档图像理解中的核心问题如OCR幻觉、细粒度定位或高分辨率处理。属于与DIU无关的平行技术路线。
Abstract With the increasing size of large language models, layer pruning has gained increased attention as a hardware-friendly approach for model compression. However, existing layer pruning methods struggle to simultaneously address key practical deployment challenges, including performance degradation, high training costs, and limited acceleration. To overcome these limitations, we propose \name, a task-\underline{E}ffective, training-\underline{E}conomical and inference-\underline{E}fficient layer pruning framework. \namespace introduces two key innovations: (1) a differentiable mask optimization method using a Gumbel-TopK sampler, enabling efficient and precise pruning mask search; and (2) an entropy-aware adaptive knowledge distillation strategy that enhances task performance. Extensive experiments over diverse model architectures and benchmarks demonstrate the superiority of our method over state-of-the-art approaches. Notably, \namespace achieves 96\% accuracy, a mere 0.8\% drop from the original model (96.8\%) on MATH-500 when pruning 25\% layers of Qwen3-32B, outperforming existing SOTA (95\%), with a 1.33$\times$ inference speedup by consuming merely 0.5B tokens (0.5\% of the post-training data volume).
Authors Tao Yuan, Haoli Bai, Yinfei Pan, Xuyang Cao, Tianyu Zhang, Lu Hou, Ting Hu, Xianzhi Yu
Categories Computation and Language
3
ToolHaystack: Stress-Testing Tool-Augmented Language Models in Realistic Long-Term Interactions
Title CN ToolHaystack：在真实长期交互中测试工具增强型语言模型
Keywords 工具增强语言模型 · 长期交互 · 评测基准 · 上下文保持 · 对话鲁棒性
Summary 本文提出了ToolHaystack，一个用于评估大语言模型在长期、多任务、含噪声的真实对话环境中使用外部工具能力的基准。实验表明，当前主流LLM在此类复杂长期交互中表现显著下降，暴露出已有评测体系未能发现的鲁棒性缺陷。
Reason 该论文提出一个用于评估工具增强型语言模型在长期交互中表现的基准（ToolHaystack），属于对特定应用场景（多轮工具调用）的评测框架设计。虽然涉及长上下文和噪声干扰，但其核心是面向通用对话式AI系统的压力测试，并未针对文档图像理解中的具体痛点（如OCR幻觉、复杂版式建模、视觉-文本对齐等）提出可迁移的方法论。所评估的现象（上下文丢失、噪声敏感）虽在长文档处理中也存在，但该工作本身不提供解决DIU问题的新架构、表示学习或推理机制，属于平行应用类评测，无底层技术创新。
Abstract Large language models (LLMs) have demonstrated strong capabilities in using external tools to address user inquiries. However, most existing evaluations assume tool use in short contexts, offering limited insight into model behavior during realistic long-term interactions. To fill this gap, we introduce ToolHaystack, a benchmark for testing the tool use capabilities in long-term interactions. Each test instance in ToolHaystack includes multiple tasks execution contexts and realistic noise within a continuous conversation, enabling assessment of how well models maintain context and handle various disruptions. By applying this benchmark to 14 state-of-the-art LLMs, we find that while current models perform well in standard multi-turn settings, they often significantly struggle in ToolHaystack, highlighting critical gaps in their long-term robustness not revealed by previous tool benchmarks.
Authors Beong-woo Kwak, Minju Kim, Dongha Lim, Hyungjoo Chae, Dongjin Kang, Sunghwan Kim, Dongil Yang, Jinyoung Yeo
Comment Our code and data are available at https://github.com/bwookwak/ToolHaystack Edited for adding acknowledgement section
Categories Computation and Language
3
The Rise of Parameter Specialization for Knowledge Storage in Large Language Models NeurIPS 2025
Title CN 大语言模型中用于知识存储的参数专业化兴起
Keywords 大语言模型 · 参数专业化 · 知识存储 · MLP · NeurIPS
Summary 本文分析了20个开源大语言模型，发现随着模型能力增强，其MLP层参数在知识存储上呈现出更高的专业化程度，即不同参数更倾向于编码特定类型的知识。通过因果训练实验，作者证明这种专业化分布能提升模型对存储知识的利用效率。
Reason 该论文研究大语言模型中MLP参数的知识存储机制，发现更先进的模型参数表现出更强的知识专业化趋势，并验证这种专业化有助于提升知识利用效率。然而，这项工作聚焦于纯文本语言模型内部的知识表示机制，未涉及视觉输入、多模态建模或文档结构理解，与文档图像理解（DIU）的核心问题如OCR幻觉、布局分析、视觉-文本对齐等无直接关联。其方法论也无法直接迁移用于解决高分辨率文档处理、细粒度定位或复杂排版推理等DIU关键挑战。
Abstract Over time, a growing wave of large language models from various series has been introduced to the community. Researchers are striving to maximize the performance of language models with constrained parameter sizes. However, from a microscopic perspective, there has been limited research on how to better store knowledge in model parameters, particularly within MLPs, to enable more effective utilization of this knowledge by the model. In this work, we analyze twenty publicly available open-source large language models to investigate the relationship between their strong performance and the way knowledge is stored in their corresponding MLP parameters. Our findings reveal that as language models become more advanced and demonstrate stronger knowledge capabilities, their parameters exhibit increased specialization. Specifically, parameters in the MLPs tend to be more focused on encoding similar types of knowledge. We experimentally validate that this specialized distribution of knowledge contributes to improving the efficiency of knowledge utilization in these models. Furthermore, by conducting causal training experiments, we confirm that this specialized knowledge distribution plays a critical role in improving the model's efficiency in leveraging stored knowledge.
Authors Yihuai Hong, Yiran Zhao, Wei Tang, Yang Deng, Yu Rong, Wenxuan Zhang
Comment Accepted in NeurIPS 2025
Categories Computation and Language
3
Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch
Title CN 跨张量并行规模的确定性推理以消除训练-推理不匹配
Keywords 确定性推理 · 张量并行 · 浮点一致性 · TBIK · RL训练
Summary 本文提出了一种基于树结构的不变核（TBIK）方法，通过统一的分层二叉树结构对GPU内外的约简顺序进行对齐，实现了在不同张量并行规模下比特级一致的确定性推理。该方法解决了训练（如FSDP）和推理（多GPU TP）之间因并行策略不同导致的精度错配问题，并已在vLLM和FSDP中实现验证。
Reason 该论文聚焦于大语言模型在不同张量并行（TP）配置下的确定性推理问题，核心贡献是解决分布式计算中因浮点运算顺序不一致导致的输出差异。尽管其在RL训练与推理一致性方面有重要意义，但完全属于LLM系统优化与底层计算范畴，未涉及任何视觉输入、多模态建模或文档结构理解内容。无法迁移用于解决文档图像理解中的OCR幻觉、复杂排版解析或细粒度定位等核心痛点。
Abstract Deterministic inference is increasingly critical for large language model (LLM) applications such as LLM-as-a-judge evaluation, multi-agent systems, and Reinforcement Learning (RL). However, existing LLM serving frameworks exhibit non-deterministic behavior: identical inputs can yield different outputs when system configurations (e.g., tensor parallel (TP) size, batch size) vary, even under greedy decoding. This arises from the non-associativity of floating-point arithmetic and inconsistent reduction orders across GPUs. While prior work has addressed batch-size-related nondeterminism through batch-invariant kernels, determinism across different TP sizes remains an open problem, particularly in RL settings, where the training engine typically uses Fully Sharded Data Parallel (i.e., TP = 1) while the rollout engine relies on multi-GPU TP to maximize the inference throughput, creating a natural mismatch between the two. This precision mismatch problem may lead to suboptimal performance or even collapse for RL training. We identify and analyze the root causes of TP-induced inconsistency and propose Tree-Based Invariant Kernels (TBIK), a set of TP-invariant matrix multiplication and reduction primitives that guarantee bit-wise identical results regardless of TP size. Our key insight is to align intra- and inter-GPU reduction orders through a unified hierarchical binary tree structure. We implement these kernels in Triton and integrate them into vLLM and FSDP. Experiments confirm zero probability divergence and bit-wise reproducibility for deterministic inference across different TP sizes. Also, we achieve bit-wise identical results between vLLM and FSDP in RL training pipelines with different parallel strategy. Code is available at https://github.com/nanomaoli/llm_reproducibility.
Authors Ziyang Zhang, Xinheng Ding, Jiayi Yuan, Rixin Liu, Huizi Mao, Jiarong Xing, Zirui Liu
Categories Computation and Language
2
Matrix: Peer-to-Peer Multi-Agent Synthetic Data Generation Framework
Title CN Matrix：点对点多智能体合成数据生成框架
Keywords 合成数据 · 多智能体系统 · 去中心化 · Ray · 数据生成
Summary 论文提出了Matrix，一个基于Ray的去中心化多智能体合成数据生成框架，通过分布式消息队列实现无需中央协调器的并行工作流，显著提升了数据生成吞吐量。适用于多智能体对话、网络推理和工具使用轨迹等场景。
Reason 该论文提出了一种去中心化的多智能体合成数据生成框架，核心贡献在于提升多智能体系统的可扩展性和灵活性。然而，其应用场景为通用对话、网页推理和客服工具轨迹等，并未针对文档图像理解（DIU）中的具体任务（如DocVQA、表格识别、布局分析）进行设计或验证。尽管‘合成数据生成’在理论上可用于DIU的数据增强，但本文方法本身是通用型Agent编排架构，属于典型的‘平行应用框架’，缺乏对OCR幻觉、密集文本处理、视觉-语言对齐等DIU核心问题的直接建模或迁移路径。因此属于无营养的下游流程封装，不具方法论上的源头创新价值。
Abstract Synthetic data has become increasingly important for training large language models, especially when real data is scarce, expensive, or privacy-sensitive. Many such generation tasks require coordinated multi-agent workflows, where specialized agents collaborate to produce data that is higher quality, more diverse, and structurally richer. However, existing frameworks for multi-agent synthesis often depend on a centralized orchestrator, creating scalability bottlenecks, or are hardcoded for specific domains, limiting flexibility. We present \textbf{Matrix}, a decentralized framework that represents both control and data flow as serialized messages passed through distributed queues. This peer-to-peer design eliminates the central orchestrator. Each task progresses independently through lightweight agents, while compute-intensive operations, such as LLM inference or containerized environments, are handled by distributed services. Built on Ray, Matrix scales to tens of thousands of concurrent agentic workflows and provides a modular, configurable design that enables easy adaptation to a wide range of data generation workflows. We evaluate Matrix across diverse synthesis scenarios, such as multi-agent collaborative dialogue, web-based reasoning data extraction, and tool-use trajectory generation in customer service environments. In all cases, Matrix achieves $2$--$15\times$ higher data generation throughput under identical hardware resources, without compromising output quality.
Authors Dong Wang, Yang Li, Ansong Ni, Ching-Feng Yeh, Youssef Emad, Xinjie Lei, Liam Robbins, Karthik Padthe, Hu Xu, Xian Li, Asli Celikyilmaz, Ramya Raghavendra, Lifei Huang, Carole-Jean Wu, Shang-Wen Li
Categories Artificial Intelligence, Computation and Language
2
Through the telecom lens: Are all training samples important?
Title CN 通过电信视角：所有训练样本都重要吗？
Keywords 样本重要性 · 梯度分析 · 数据效率 · 可持续AI · 电信
Summary 本文研究电信领域AI训练过程中不同样本的影响，通过逐样本梯度分析识别冗余与高影响力数据，提出一种选择性优先处理重要样本的框架，在保持模型性能的同时降低数据需求和计算负担。实验基于三个真实电信数据集验证了方法有效性。
Reason 该论文聚焦于电信领域中AI训练样本的重要性分析，提出基于梯度的样本选择框架以减少计算开销。尽管其方法涉及数据效率和可持续AI，但属于特定行业（电信）的下游应用，未针对文档图像理解中的核心问题（如OCR幻觉、复杂排版、多模态对齐等）提出可迁移的方法论创新。属于‘平行下游应用’，不符合关注标准。
Abstract The rise of AI in telecommunications, from optimizing Radio Access Networks to managing user experience, has sharply increased data volumes and training demands. Telecom data is often noisy, high-dimensional, costly to store, process, and label. Despite Ai's critical role, standard workflows still assume all training samples contribute equally. On the other hand, next generation systems require AI models that are accurate, efficient, and sustainable.The paper questions the assumptions of equal importance by focusing on applying and analyzing the roles of individual samples in telecom training and assessing whether the proposed model optimizes computation and energy use. we perform sample-level gradient analysis across epochs to identify patterns of influence and redundancy in model learning. Based on this, we propose a sample importance framework thats electively prioritizes impactful data and reduces computation without compromising accuracy. Experiments on three real-world telecom datasets show that our method [reserves performance while reducing data needs and computational overhead while advancing the goals of sustainable AI in telecommunications.
Authors Shruti Bothe, Illyyne Saffar, Aurelie Boisbunon, Hasan Farooq, Julien Forgeat, Md Moin Uddin Chowdhury
Comment 8pages, 1 table, 8 figures
Categories Artificial Intelligence
2
TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding
Title CN TimeViper：一种用于高效长视频理解的混合Mamba-Transformer视觉语言模型
Keywords 视频理解 · Mamba · 视觉语言模型 · token压缩 · 长序列建模
Summary 本文提出TimeViper，一种结合Mamba和Transformer的混合架构视觉语言模型，用于长视频理解。通过观察到视觉到文本的信息聚合现象，作者设计了TransV模块，将视觉token信息压缩并转移至指令token中，实现对超过万帧的长视频处理。实验表明该模型在多个长视频基准上具有竞争力。
Reason 该论文研究的是长视频理解任务，属于视频理解领域，与文档图像理解（DIU）的核心场景（如DocVQA、表格识别、布局分析等）无关。尽管其提出的TransV模块涉及视觉token压缩和信息聚合，具有一定的方法论价值，但其设计动机和验证场景均围绕长时间序列视频展开，迁移到文档图像中的高分辨率或复杂排版处理存在显著领域差异。且未提及对OCR幻觉、细粒度定位或文档结构建模的支持，无法直接解决DIU的关键痛点。
Abstract We introduce TimeViper, a hybrid vision-language model designed to tackle challenges of long video understanding. Processing long videos demands both an efficient model architecture and an effective mechanism for handling extended temporal contexts. To this end, TimeViper adopts a hybrid Mamba-Transformer backbone that combines the efficiency of state-space models with the expressivity of attention mechanisms. Through this hybrid design, we reveal the vision-to-text information aggregation phenomenon, where information progressively flows from vision tokens to text tokens across increasing LLM depth, resulting in severe vision token redundancy. Motivated by this observation, we propose TransV, a token information transfer module that transfers and compresses vision tokens into instruction tokens while maintaining multimodal understanding capabilities. This design enables TimeViper to process hour-long videos exceeding 10,000 frames. Extensive experiments across multiple benchmarks demonstrate that TimeViper competes with state-of-the-art models while extending frame numbers. We further analyze attention behaviors of both Mamba and Transformer layers, offering new insights into hybrid model interpretability. This work represents an initial step towards developing, interpreting, and compressing hybrid Mamba-Transformer architectures.
Authors Boshen Xu, Zihan Xiao, Jiaze Li, Jianzhong Ju, Zhenbo Luo, Jian Luan, Qin Jin
Comment Project page: https://xuboshen.github.io/TimeViper; Code: https://github.com/xiaomi-research/timeviper
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
2
TREASURE: A Transformer-Based Foundation Model for High-Volume Transaction Understanding
Title CN TREASURE：一种基于Transformer的高容量交易理解基础模型
Keywords Transformer · 交易理解 · 异常检测 · 推荐系统 · 序列建模
Summary 本文提出TREASURE，一个专为高容量交易数据设计的Transformer基础模型，用于捕捉消费者行为和支付网络信号，以支持异常检测和推荐系统等任务。模型包含静态与动态属性子模块，并在工业级数据上验证有效，显著提升现有生产系统的性能。
Reason 该论文聚焦于交易数据建模，属于特定领域的下游应用（金融/支付场景），并未涉及文档图像理解中的核心问题如OCR、版面分析、视觉文本定位或VLM对文档的细粒度理解。尽管其使用了Transformer架构和结构化输入设计，但方法本质上是针对交易记录的序列建模，与文档图像或多模态文档理解无关。属于典型的‘平行应用’，无DIU领域的方法论创新。
Abstract Payment networks form the backbone of modern commerce, generating high volumes of transaction records from daily activities. Properly modeling this data can enable applications such as abnormal behavior detection and consumer-level insights for hyper-personalized experiences, ultimately improving people's lives. In this paper, we present TREASURE, TRansformer Engine As Scalable Universal transaction Representation Encoder, a multipurpose transformer-based foundation model specifically designed for transaction data. The model simultaneously captures both consumer behavior and payment network signals (such as response codes and system flags), providing comprehensive information necessary for applications like accurate recommendation systems and abnormal behavior detection. Verified with industry-grade datasets, TREASURE features three key capabilities: 1) an input module with dedicated sub-modules for static and dynamic attributes, enabling more efficient training and inference; 2) an efficient and effective training paradigm for predicting high-cardinality categorical attributes; and 3) demonstrated effectiveness as both a standalone model that increases abnormal behavior detection performance by 111% over production systems and an embedding provider that enhances recommendation models by 104%. We present key insights from extensive ablation studies, benchmarks against production models, and case studies, highlighting valuable knowledge gained from developing TREASURE.
Authors Chin-Chia Michael Yeh, Uday Singh Saini, Xin Dai, Xiran Fan, Shubham Jain, Yujie Fan, Jiarui Sun, Junpeng Wang, Menghai Pan, Yingtong Dou, Yuzhong Chen, Vineeth Rakesh, Liang Wang, Yan Zheng, Mahashweta Das
Categories Artificial Intelligence
2
From Observation to Action: Latent Action-based Primitive Segmentation for VLA Pre-training in Industrial Settings
Title CN 从观察到行动：基于潜在动作的基元分割在工业场景中用于VLA预训练
Keywords 动作分割 · 无监督学习 · 具身智能 · VLA · 工业视频
Summary 本文提出一种无监督框架，通过轻量级运动编码器和新型‘潜在动作能量’指标，从连续工业视频中自动发现并分割语义一致的动作基元，生成适合VLA模型预训练的结构化数据。在公开基准和电机装配数据集上验证了方法有效性，并通过视觉语言模型评估了动作基元的语义一致性。
Reason 该论文聚焦于从工业视频流中无监督地提取动作基元，用于具身智能（Embodied AI）中的Vision-Language-Action（VLA）模型预训练。其核心贡献在于动作分段与潜在动作表示，属于机器人/具身智能领域。尽管涉及视觉-语言对齐，但任务目标与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版或长文档推理等核心问题。属于负面清单中的‘无关领域’（具身智能/机器人），且不具备向DIU迁移的方法论价值。
Abstract We present a novel unsupervised framework to unlock vast unlabeled human demonstration data from continuous industrial video streams for Vision-Language-Action (VLA) model pre-training. Our method first trains a lightweight motion tokenizer to encode motion dynamics, then employs an unsupervised action segmenter leveraging a novel "Latent Action Energy" metric to discover and segment semantically coherent action primitives. The pipeline outputs both segmented video clips and their corresponding latent action sequences, providing structured data directly suitable for VLA pre-training. Evaluations on public benchmarks and a proprietary electric motor assembly dataset demonstrate effective segmentation of key tasks performed by humans at workstations. Further clustering and quantitative assessment via a Vision-Language Model confirm the semantic coherence of the discovered action primitives. To our knowledge, this is the first fully automated end-to-end system for extracting and organizing VLA pre-training data from unstructured industrial videos, offering a scalable solution for embodied AI integration in manufacturing.
Authors Jiajie Zhang, Sören Schwertfeger, Alexander Kleiner
Comment 10 pages, 5 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
FITRep: Attention-Guided Item Representation via MLLMs
Title CN FITRep：通过MLLM实现注意力引导的项目表示
Keywords 多模态大模型 · 商品去重 · 注意力机制 · 降维 · 聚类
Summary 本文提出FITRep，一种基于多模态大语言模型的注意力引导商品表示框架，用于解决电商平台中视觉和文本相似导致的近重复商品问题。该方法通过分层语义提取、结构保持的降维和FAISS聚类实现高效去重，在美团广告系统中取得CTR和CPM显著提升。
Reason 该论文聚焦于在线平台中的近重复商品去重问题，使用MLLM提取分层语义概念并结合降维与聚类进行商品表示。尽管使用了多模态大模型（MLLM），但其任务本质是电商广告系统中的商品去重，属于特定下游应用场景（Wrapper/Application），并未针对文档图像理解中的核心挑战（如OCR幻觉、复杂版式、长文档推理等）提出通用方法论创新。因此属于‘平行下游应用’，不符合本研究方向对底层技术创新的要求。
Abstract Online platforms usually suffer from user experience degradation due to near-duplicate items with similar visuals and text. While Multimodal Large Language Models (MLLMs) enable multimodal embedding, existing methods treat representations as black boxes, ignoring structural relationships (e.g., primary vs. auxiliary elements), leading to local structural collapse problem. To address this, inspired by Feature Integration Theory (FIT), we propose FITRep, the first attention-guided, white-box item representation framework for fine-grained item deduplication. FITRep consists of: (1) Concept Hierarchical Information Extraction (CHIE), using MLLMs to extract hierarchical semantic concepts; (2) Structure-Preserving Dimensionality Reduction (SPDR), an adaptive UMAP-based method for efficient information compression; and (3) FAISS-Based Clustering (FBC), a FAISS-based clustering that assigns each item a unique cluster id using FAISS. Deployed on Meituan's advertising system, FITRep achieves +3.60% CTR and +4.25% CPM gains in online A/B tests, demonstrating both effectiveness and real-world impact.
Authors Guoxiao Zhang, Ao Li, Tan Qu, Qianlong Xie, Xingxing Wang
Categories Artificial Intelligence
2
Not All Splits Are Equal: Rethinking Attribute Generalization Across Unrelated Categories NeurIPS 2025 Workshop: CauScien - Uncovering Causality in Science and NeurIPS 2025 Workshop: Reliable ML from Unreliable Data
Title CN 并非所有划分都是平等的：重新思考跨不相关类别的属性泛化
Keywords 属性泛化 · 数据划分 · 模型鲁棒性 · 语义无关类别 · NeurIPS
Summary 本文探讨了模型是否能将在一种对象上学到的属性（如‘有四条腿’）泛化到语义和视觉上不相关的另一类对象（如从狗到椅子）。作者设计了多种降低训练集与测试集之间关联性的数据划分方法，并发现当前模型在此类设置下性能显著下降，表明其对数据划分高度敏感。研究强调了现有模型在抽象属性知识方面的局限性。
Reason 该论文研究的是通用视觉属性在语义无关类别间的泛化能力，核心任务为属性预测与模型鲁棒性评估，属于通用计算机视觉问题。其提出的数据划分策略和评估框架主要面向开放世界下的因果与泛化问题，并未涉及文档结构、文本内容理解、OCR或文档排版等DIU核心挑战。尽管其关于数据分布与泛化能力的讨论具有一定方法论启发性，但无法直接迁移用于解决文档图像中的幻觉、密集文本识别、细粒度定位或长文档推理等关键痛点，属于无关领域的基础分析工作。
Abstract Can models generalize attribute knowledge across semantically and perceptually dissimilar categories? While prior work has addressed attribute prediction within narrow taxonomic or visually similar domains, it remains unclear whether current models can abstract attributes and apply them to conceptually distant categories. This work presents the first explicit evaluation for the robustness of the attribute prediction task under such conditions, testing whether models can correctly infer shared attributes between unrelated object types: e.g., identifying that the attribute "has four legs" is common to both "dogs" and "chairs". To enable this evaluation, we introduce train-test split strategies that progressively reduce correlation between training and test sets, based on: LLM-driven semantic grouping, embedding similarity thresholding, embedding-based clustering, and supercategory-based partitioning using ground-truth labels. Results show a sharp drop in performance as the correlation between training and test categories decreases, indicating strong sensitivity to split design. Among the evaluated methods, clustering yields the most effective trade-off, reducing hidden correlations while preserving learnability. These findings offer new insights into the limitations of current representations and inform future benchmark construction for attribute reasoning.
Authors Liviu Nicolae Fircă, Antonio Bărbălau, Dan Oneata, Elena Burceanu
Comment Accepted at NeurIPS 2025 Workshop: CauScien - Uncovering Causality in Science and NeurIPS 2025 Workshop: Reliable ML from Unreliable Data
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
A Systematic Analysis of Large Language Models with RAG-enabled Dynamic Prompting for Medical Error Detection and Correction
Title CN 基于RAG增强动态提示的大语言模型在医疗错误检测与纠正中的系统性分析
Keywords 大语言模型 · 检索增强生成 · 医疗错误检测 · 提示工程 · 文本纠错
Summary 本文评估了多种大语言模型在不同提示策略（零样本、静态示例提示、检索增强动态提示）下对临床文档中错误的检测与修正能力。实验基于MEDEC数据集，结果显示检索增强动态提示（RDP）在召回率和纠错准确性方面优于其他方法，并降低了误报率。研究聚焦于医疗领域的文本质量提升，属于特定领域应用。
Reason 该论文属于典型的‘平行下游应用’，将现有LLM与RAG技术应用于医疗文本错误检测任务。尽管其提示工程策略有一定实用性，但未提出任何面向文档图像理解（DIU）或视觉-语言建模的底层方法创新。任务完全基于纯文本输入，不涉及OCR、布局分析、表格识别或视觉内容理解，对解决DIU核心痛点如OCR幻觉、复杂排版或细粒度定位无直接迁移价值。
Abstract Objective: Clinical documentation contains factual, diagnostic, and management errors that can compromise patient safety. Large language models (LLMs) may help detect and correct such errors, but their behavior under different prompting strategies remains unclear. We evaluate zero-shot prompting, static prompting with random exemplars (SPR), and retrieval-augmented dynamic prompting (RDP) for three subtasks of medical error processing: error flag detection, error sentence detection, and error correction. Methods: Using the MEDEC dataset, we evaluated nine instruction-tuned LLMs (GPT, Claude, Gemini, and OpenAI o-series models). We measured performance using accuracy, recall, false-positive rate (FPR), and an aggregate score of ROUGE-1, BLEURT, and BERTScore for error correction. We also analyzed example outputs to identify failure modes and differences between LLM and clinician reasoning. Results: Zero-shot prompting showed low recall in both detection tasks, often missing abbreviation-heavy or atypical errors. SPR improved recall but increased FPR. Across all nine LLMs, RDP reduced FPR by about 15 percent, improved recall by 5 to 10 percent in error sentence detection, and generated more contextually accurate corrections. Conclusion: Across diverse LLMs, RDP outperforms zero-shot and SPR prompting. Using retrieved exemplars improves detection accuracy, reduces false positives, and enhances the reliability of medical error correction.
Authors Farzad Ahmed, Joniel Augustine Jerome, Meliha Yetisgen, Özlem Uzuner
Categories Artificial Intelligence, Computation and Language
2
AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise NeurIPS 2025
Title CN 通过贝叶斯惊奇实现开放式科学发现
Keywords 自主科学发现 · 贝叶斯惊奇 · 大语言模型 · 蒙特卡洛树搜索 · 开放探索
Summary 该研究提出AutoDiscovery，一种基于贝叶斯惊奇驱动的开放式自主科学发现框架。利用大语言模型的信念前后变化量化‘惊奇程度’，并以此作为奖励函数指导蒙特卡洛树搜索（MCTS）在假设空间中高效探索。在21个真实数据集上验证了其在固定预算下比现有方法多发现5-29%被LLM认为‘惊奇’的结果，并得到领域专家认可。目标是实现无需人类预设问题的AI自主科研。
Reason 该论文属于具身智能与科学发现领域的上游探索，但其核心方法（基于贝叶斯惊奇的假设生成与MCTS搜索）应用于文档图像理解缺乏直接迁移路径。虽然‘推理时计算’和‘探索策略’在理论上对长文档多步推理有一定启发，但本文聚焦的是科学假设空间的开放探索，而非解决DIU中的具体痛点如OCR幻觉、细粒度定位或复杂排版理解。且未涉及视觉输入或多模态建模，无法作为VLM for Doc的有效工具。属于无关领域的平行应用构想。
Abstract The promise of autonomous scientific discovery (ASD) hinges not only on answering questions, but also on knowing which questions to ask. Most recent works in ASD explore the use of large language models (LLMs) in goal-driven settings, relying on human-specified research questions to guide hypothesis generation. However, scientific discovery may be accelerated further by allowing the AI system to drive exploration by its own criteria. The few existing approaches in open-ended ASD select hypotheses based on diversity heuristics or subjective proxies for human interestingness, but the former struggles to meaningfully navigate the typically vast hypothesis space, and the latter suffers from imprecise definitions. This paper presents AutoDiscovery -- a method for open-ended ASD that instead drives scientific exploration using Bayesian surprise. Here, we quantify the epistemic shift from the LLM's prior beliefs about a hypothesis to its posterior beliefs after gathering experimental results. To efficiently explore the space of nested hypotheses, our method employs a Monte Carlo tree search (MCTS) strategy with progressive widening using surprisal as the reward function. We evaluate AutoDiscovery in the setting of data-driven discovery across 21 real-world datasets spanning domains such as biology, economics, finance, and behavioral science. Our results demonstrate that under a fixed budget, AutoDiscovery substantially outperforms competitors by producing 5-29% more discoveries deemed surprising by the LLM. Our human evaluation further reveals that two-thirds of discoveries made by our system are surprising to domain experts as well, suggesting this is an important step towards building open-ended ASD systems.
Authors Dhruv Agarwal, Bodhisattwa Prasad Majumder, Reece Adamson, Megha Chakravorty, Satvika Reddy Gavireddy, Aditya Parashar, Harshit Surana, Bhavana Dalvi Mishra, Andrew McCallum, Ashish Sabharwal, Peter Clark
Comment Accepted to NeurIPS 2025; https://neurips.cc/virtual/2025/loc/san-diego/poster/116398
Categories Artificial Intelligence, Computation and Language
2
Where to Start Alignment? Diffusion Large Language Model May Demand a Distinct Position AAAI 2026
Title CN 从哪里开始对齐？扩散大语言模型可能需要一个独特的位置
Keywords 扩散大语言模型 · 安全对齐 · 中间token · 非自回归生成 · 强化学习
Summary 本文首次研究了扩散大语言模型（dLLM）的安全性能，发现其在生成过程中中间token比起始token对整体安全性更为关键。基于这一发现，作者提出了Middle-tOken Safety Alignment（MOSA）方法，利用强化学习对中间阶段的生成进行安全对齐。实验表明该方法在多个攻击基准下显著提升了模型安全性，同时保持了在编程、数学和推理任务上的可用性。
Reason 该论文研究的是扩散大语言模型（dLLM）的安全对齐问题，聚焦于生成过程中中间token对安全响应的影响，并提出针对该架构的MOSA对齐方法。尽管涉及多模态生成的新范式，但其核心任务是语言模型的安全性（防御攻击、拒绝生成），并未处理视觉输入或文档结构理解。与文档图像理解（DIU）中的OCR幻觉、排版分析、视觉-文本对齐等核心痛点无直接关联，属于纯NLP安全对齐范畴，且不涉及视觉模态的幻觉抑制。因此属于‘无关领域’中的‘纯NLP的安全/对齐’类，不符合关注方向。
Abstract Diffusion Large Language Models (dLLMs) have recently emerged as a competitive non-autoregressive paradigm due to their unique training and inference approach. However, there is currently a lack of safety study on this novel architecture. In this paper, we present the first analysis of dLLMs' safety performance and propose a novel safety alignment method tailored to their unique generation characteristics. Specifically, we identify a critical asymmetry between the defender and attacker in terms of security. For the defender, we reveal that the middle tokens of the response, rather than the initial ones, are more critical to the overall safety of dLLM outputs; this seems to suggest that aligning middle tokens can be more beneficial to the defender. The attacker, on the contrary, may have limited power to manipulate middle tokens, as we find dLLMs have a strong tendency towards a sequential generation order in practice, forcing the attack to meet this distribution and diverting it from influencing the critical middle tokens. Building on this asymmetry, we introduce Middle-tOken Safety Alignment (MOSA), a novel method that directly aligns the model's middle generation with safe refusals exploiting reinforcement learning. We implement MOSA and compare its security performance against eight attack methods on two benchmarks. We also test the utility of MOSA-aligned dLLM on coding, math, and general reasoning. The results strongly prove the superiority of MOSA.
Authors Zhixin Xie, Xurui Song, Jun Luo
Comment Accepted for oral presentation at AAAI 2026
Categories Artificial Intelligence, Computation and Language
2
Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuning
Title CN 使用平衡微调使大语言模型与生物医学知识对齐
Keywords 大语言模型 · 生物医学知识 · 平衡微调 · 稀疏数据学习 · 后训练对齐
Summary 本文提出了一种名为平衡微调（BFT）的方法，通过在token级别和样本级别引入双层加权机制，提升大语言模型在生物医学领域从稀疏数据中学习复杂推理的能力。实验表明其在基因交互、单细胞扰动响应预测等任务上优于传统SFT和GeneAgent。但整个工作完全基于纯文本，不涉及视觉或文档结构信息。
Reason 该论文属于典型的‘平行下游应用’：将现有LLM微调技术应用于生物医学领域，解决的是特定领域的知识对齐问题，而非文档图像理解（DIU）中的核心挑战。尽管其方法涉及细粒度知识学习和抗过拟合机制，但未涉及视觉模态、OCR、布局分析或任何多模态文档理解任务。无法迁移用于解决DIU中的OCR幻觉、复杂排版或长文档推理等关键问题。
Abstract Effective post-training is essential to align Large Language Models (LLMs) with specialized biomedical knowledge to accelerate life science research. However, current approaches face significant limitations. First, biomedical reasoning involves intricate mechanisms often represented by sparse textual data. Standard Supervised Fine-Tuning (SFT) tends to overfit to surface-level instruction patterns without effectively internalizing this fragmented scientific knowledge. Second, Reinforcement Learning (RL) is impractical for this domain, as defining meaningful rewards often necessitates prohibitive experimental validation (e.g., wet-lab verification of drug responses), rendering real-time feedback unfeasible. We propose Balanced Fine-Tuning (BFT), an efficient post-training method designed to learn complex reasoning from sparse data without external reward signals. BFT operates through a two-layer weighting mechanism: 1. At the token level, it scales loss via prediction probabilities to stabilize gradients and prevent overfitting; 2. At the sample level, it uses "minimum group confidence" to adaptively enhance the learning of hard samples. Experiments demonstrate that BFT significantly outperforms SFT. In medical tasks, it enables LLMs to acquire knowledge that SFT misses. In biological tasks, BFT-based LLMs surpass GeneAgent (an accurate agent for biology analysis) in biological process reasoning. Moreover, the text embeddings generated by BFT can be directly applied to downstream tasks, such as gene interaction and single-cell perturbation response prediction. These results indicate that BFT facilitates broad applications of LLMs in biomedical research.
Authors Zhenchao Tang, Fang Wang, Haohuai He, Jiale Zhou, Tianxu Lv, Jun Zhu, Shouzhi Chen, Minghao Yang, Yu Wang, Jiayang Wu, Yidong Song, Jianhua Yao
Categories Artificial Intelligence
2
CoMind: Towards Community-Driven Agents for Machine Learning Engineering
Title CN CoMind：面向机器学习工程的社区驱动型智能体
Keywords LLM Agent · Machine Learning Engineering · Multi-Agent System · Kaggle · Automated ML
Summary 本文提出CoMind，一个受Kaggle社区启发的多智能体系统，用于自动化机器学习工程任务。通过构建MLE-Live评估框架，模拟真实科研社区环境，CoMind采用并行探索机制，在多个解决方案中同步迭代，并利用外部知识交流提升性能。在75个历史比赛和8个实时比赛中，CoMind表现出超越大多数人类参赛者的竞争力。
Reason 该论文属于典型的‘平行应用’类工作，将LLM智能体应用于Kaggle竞赛这一特定场景，核心贡献是多智能体系统在ML工程自动化中的应用。尽管其Agent架构具有一定设计思想，但整体框架针对的是机器学习建模任务（如特征工程、模型调参），与文档图像理解（DIU）中的OCR幻觉、复杂版面解析、细粒度文本定位等核心痛点无直接关联。方法难以迁移至文档理解领域，属于下游任务封装，不符合本研究关注的技术源头创新要求。
Abstract Large language model (LLM) agents show promise in automating machine learning (ML) engineering. However, existing agents typically operate in isolation on a given research problem, without engaging with the broader research community, where human researchers often gain insights and contribute by sharing knowledge. To bridge this gap, we introduce MLE-Live, a live evaluation framework designed to assess an agent's ability to communicate with and leverage collective knowledge from a simulated Kaggle research community. Building on this framework, we propose CoMind, an multi-agent system designed to actively integrate external knowledge. CoMind employs an iterative parallel exploration mechanism, developing multiple solutions simultaneously to balance exploratory breadth with implementation depth. On 75 past Kaggle competitions within our MLE-Live framework, CoMind achieves a 36% medal rate, establishing a new state of the art. Critically, when deployed in eight live, ongoing competitions, CoMind outperforms 92.6% of human competitors on average, placing in the top 5% on three official leaderboards and the top 1% on one.
Authors Sijie Li, Weiwei Sun, Shanda Li, Ameet Talwalkar, Yiming Yang
Categories Artificial Intelligence
2
Breaking the Safety-Capability Tradeoff: Reinforcement Learning with Verifiable Rewards Maintains Safety Guardrails in LLMs AAAI-26 Workshop on Post-AI Formal Methods
Title CN 打破安全性与能力的权衡：基于可验证奖励的强化学习在大语言模型中保持安全护栏
Keywords 大语言模型 · 安全性 · 强化学习 · 可验证奖励 · 能力-安全权衡
Summary 论文指出在微调大语言模型时存在安全性与能力之间的权衡问题，并发现传统的SFT和RLHF方法会削弱模型的安全对齐。作者提出采用基于可验证奖励的强化学习（RLVR），通过理论分析证明在KL约束下可以控制安全漂移，并在多个对抗性安全基准上验证了该方法能在提升推理能力的同时保持甚至增强安全防护。实验涵盖了优化算法、模型规模和任务领域的消融研究。
Reason 该论文聚焦于大语言模型在推理任务中的安全性与能力之间的权衡问题，提出使用可验证奖励的强化学习（RLVR）来维持安全对齐。尽管其方法在理论上对通用LLM的安全部署有贡献，但研究背景完全脱离多模态场景，未涉及视觉输入、文档结构或OCR相关幻觉等文档图像理解（DIU）的核心痛点。虽然‘去幻觉’是DIU的重要议题，但本文讨论的是文本生成中的逻辑正确性与有害内容规避，而非视觉-语言对齐中的感知幻觉（如误读字符、错位识别），因此无法迁移应用于文档理解任务。
Abstract Fine-tuning large language models (LLMs) for downstream tasks typically exhibit a fundamental safety-capability tradeoff, where improving task performance degrades safety alignment even on benign datasets. This degradation persists across standard approaches including supervised finetuning (SFT) and reinforcement learning from human feedback (RLHF). While reinforcement learning with verifiable rewards (RLVR) has emerged as a promising alternative that optimizes models on objectively measurable tasks, its safety implications remain unexplored. We present the first comprehensive theoretical and empirical analysis of safety properties in RLVR. Theoretically, we derive upper bounds on safety drift under KL-constrained optimization and prove conditions under which safety degradation is eliminated. Empirically, we conduct extensive experiments across five adversarial safety benchmarks, demonstrating that RLVR can simultaneously enhance reasoning capabilities while maintaining or improving safety guardrails. Our comprehensive ablation studies examine the effects of optimization algorithms, model scale, and task domains. Our findings challenge the prevailing assumption of an inevitable safety capability trade-off, and establish that a specific training methodology can achieve both objectives simultaneously, providing insights for the safe deployment of reasoning-capable LLMs.
Authors Dongkyu Derek Cho, Huan Song, Arijit Ghosh Chowdhury, Haotian An, Yawei Wang, Rohit Thekkanal, Negin Sokhandan, Sharlina Keshava, Hannah Marlowe
Comment AAAI-26 Workshop on Post-AI Formal Methods
Categories Artificial Intelligence
2
Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels
Title CN 上下文学习中的语义锚点：为什么小型语言模型无法翻转其标签
Keywords 上下文学习 · 语义锚定 · 标签翻转 · 语言模型行为 · 少样本学习
Summary 本文通过对比自然示例和标签翻转示例下的模型行为，提出语言模型在上下文学习中存在‘语义锚点’——即预训练阶段形成的稳定语义方向难以被少样本提示覆盖。实验表明，当前规模的LLM无法真正学会反语义分类，ICL更多是调整输入到已有语义方向的投影，而非重构语义映射。
Reason 该论文研究的是纯文本领域中上下文学习（ICL）对语言模型标签语义的可塑性问题，属于基础NLP机制分析，未涉及任何视觉或文档模态。尽管其关于‘语义锚定’的发现可能对幻觉现象有哲学层面的启发，但无法直接迁移解决文档图像理解中的OCR幻觉、细粒度定位或复杂排版等核心问题。且研究对象为纯文本分类任务与小型LLM，与多模态文档理解（DIU）的技术路径无关。
Abstract Can in-context learning (ICL) override pre-trained label semantics, or does it merely refine an existing semantic backbone? We address this question by treating LLMs as prompt-induced classifiers and contrasting their behavior under \emph{natural} demonstrations (with correct labels) and \emph{inverted} demonstrations (systematically flipping label meanings). We decompose ICL behavior into three alignment metrics (truth, prior, and prompt alignment) and introduce a semantic override rate, defined as correctness under flipped semantics. Across eight classification tasks and eight open-source LLMs (1--12B parameters), we find consistent evidence for a semantic anchor view. With natural demonstrations, ICL improves accuracy while maintaining strong prior alignment; most correct predictions coincide with zero-shot behavior, even when the prior is weak. With inverted demonstrations, models cannot learn coherent anti-semantic classifiers: prompt alignment increases only by sacrificing accuracy, and semantic override rates remain exactly zero in our few-shot 1--12B setting. Rather than flexibly remapping label meanings, ICL primarily adjusts how inputs project onto stable semantic directions learned during pre-training, clarifying fundamental limits of few-shot prompting and suggesting that overriding label semantics at these scales requires interventions beyond ICL. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/semantic-anchors-icl.
Authors Anantha Padmanaban Krishna Kumar
Comment 13 pages total (7 pages main text, 3 pages references, 3 pages appendix), 2 figures, 14 tables. Code available at https://github.com/AnanthaPadmanaban-KrishnaKumar/semantic-anchors-icl
Categories Artificial Intelligence, Computation and Language
2
Towards Trustworthy Legal AI through LLM Agents and Formal Reasoning
Title CN 通过LLM Agent与形式化推理实现可信赖的法律AI
Keywords Legal AI · LLM Agent · Formal Reasoning · SMT Solver · Adversarial Framework
Summary 本文提出L4M框架，结合对抗性LLM Agent与SMT求解器，通过三阶段流程（法条形式化、双方案情提取、基于求解器的裁决）实现具备符号级可解释性的法律推理系统。系统能生成逻辑严谨、可验证的判决结果，在多个法律AI基准上超越主流大模型。然而，其工作完全基于纯文本输入，不涉及任何形式的文档图像理解或视觉信息处理。
Reason 该论文属于典型的‘平行下游应用’，将现有LLM Agent和形式化验证技术应用于法律领域决策流程。尽管其框架设计复杂且实验效果优越，但核心方法（对抗性Agent、SMT求解器、自动形式化）并未针对文档图像理解中的关键挑战（如OCR幻觉、密集文本识别、版面分析、视觉-语言对齐）提出任何底层技术创新。无法迁移用于解决DIU领域的视觉输入处理或跨模态推理瓶颈。
Abstract The rationality of law manifests in two forms: substantive rationality, which concerns the fairness or moral desirability of outcomes, and formal rationality, which requires legal decisions to follow explicitly stated, general, and logically coherent rules. Existing LLM-based systems excel at surface-level text analysis but lack the guarantees required for principled jurisprudence. We introduce L4M, a novel framework that combines adversarial LLM agents with SMT-solver-backed proofs to unite the interpretive flexibility of natural language with the rigor of symbolic verification. The pipeline consists of three phases: (1) Statute Formalization, where domain-specific prompts convert legal provisions into logical formulae; (2) Dual Fact and Statute Extraction, in which prosecutor- and defense-aligned LLMs independently map case narratives to fact tuples and statutes, ensuring role isolation; and (3) Solver-Centric Adjudication, where an autoformalizer compiles both parties' arguments into logic constraints, and unsat cores trigger iterative self-critique until a satisfiable formula is achieved, which is then verbalized by a Judge-LLM into a transparent verdict and optimized sentence. Experimental results on public benchmarks show that our system surpasses advanced LLMs including GPT-o4-mini, DeepSeek-V3, and Claude 4 as well as state-of-the-art Legal AI baselines, while providing rigorous and explainable symbolic justifications.
Authors Linze Chen, Yufan Cai, Zhe Hou, Jinsong Dong
Categories Artificial Intelligence
2
Knowledge Completes the Vision: A Multimodal Entity-aware Retrieval-Augmented Generation Framework for News Image Captioning AAAI 2026
Title CN 知识完善视觉：一种面向新闻图像描述生成的多模态实体感知检索增强生成框架
Keywords 新闻图像描述 · 检索增强生成 · 多模态 · 实体接地 · 跨模态对齐
Summary 本文提出MERGE框架，通过构建实体中心的多模态知识库（EMKB）并采用多阶段假设-描述策略，增强新闻图像描述生成中的信息覆盖、跨模态对齐与视觉-实体匹配。在多个新闻图像数据集上实现了更高质量的描述生成与更强的实体识别能力。
Reason 该论文聚焦于新闻图像描述生成（News Image Captioning），属于典型的下游垂直应用，其核心贡献是将RAG与多模态实体对齐结合用于提升图像描述的新闻性与实体准确性。尽管涉及多模态和实体接地，但任务本身并非文档图像理解（如DocVQA、表格识别、版面分析等）的核心任务，且方法设计紧密耦合于图像描述生成场景，缺乏可迁移至DIU领域解决OCR幻觉、密集文本处理或复杂排版理解的通用技术组件。属于‘平行下游应用’，不符合本筛选原则。
Abstract News image captioning aims to produce journalistically informative descriptions by combining visual content with contextual cues from associated articles. Despite recent advances, existing methods struggle with three key challenges: (1) incomplete information coverage, (2) weak cross-modal alignment, and (3) suboptimal visual-entity grounding. To address these issues, we introduce MERGE, the first Multimodal Entity-aware Retrieval-augmented GEneration framework for news image captioning. MERGE constructs an entity-centric multimodal knowledge base (EMKB) that integrates textual, visual, and structured knowledge, enabling enriched background retrieval. It improves cross-modal alignment through a multistage hypothesis-caption strategy and enhances visual-entity matching via dynamic retrieval guided by image content. Extensive experiments on GoodNews and NYTimes800k show that MERGE significantly outperforms state-of-the-art baselines, with CIDEr gains of +6.84 and +1.16 in caption quality, and F1-score improvements of +4.14 and +2.64 in named entity recognition. Notably, MERGE also generalizes well to the unseen Visual News dataset, achieving +20.17 in CIDEr and +6.22 in F1-score, demonstrating strong robustness and domain adaptability.
Authors Xiaoxing You, Qiang Huang, Lingyu Li, Chi Zhang, Xiaopeng Liu, Min Zhang, Jun Yu
Comment Accepted to AAAI 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
GuardTrace-VL: Detecting Unsafe Multimodel Reasoning via Iterative Safety Supervision
Title CN GuardTrace-VL：通过迭代安全监督检测不安全的多模态推理
Keywords 多模态安全 · 推理链监控 · 视觉-语言模型 · AI对齐 · 有害内容检测
Summary 本文提出GuardTrace-VL，一种针对多模态大推理模型（MLRM）的安全审计工具，能够通过联合分析图像和文本监控从提问、思考到回答的全过程，检测中间推理步骤中可能出现的偏见或违规内容。作者构建了GuardTrace数据集，并设计了一个三阶段渐进式训练方案，使模型能根据风险等级学习上下文相关的安全偏好。实验显示其在F1分数上显著优于现有方法。
Reason 该论文聚焦于多模态大模型在推理过程中生成的中间思维链（reasoning trace）中的安全内容检测，属于AI安全与对齐领域。尽管其涉及视觉-语言联合分析，但核心任务是识别‘有害推理’而非提升文档理解能力。未解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点，且属于典型的纯NLP安全/对齐研究，仅扩展到多模态形式，不具备向文档图像理解迁移的方法论价值。
Abstract Multimodal large reasoning models (MLRMs) are increasingly deployed for vision-language tasks that produce explicit intermediate rationales. However, reasoning traces can contain unsafe content even when the final answer is non-harmful, creating deployment risks. Existing multimodal safety guards primarily evaluate only the input question and the final answer, neglecting the intermediate reasoning process. This oversight allows undetected harm, such as biased inferences or policy-violating use of visual context, to emerge during reasoning. We introduce GuardTrace-VL, a vision-aware safety auditor that monitors the full Question-Thinking-Answer (QTA) pipeline via joint image-text analysis, enabling detection of unsafe content as it emerges in the reasoning stage. To support training and evaluation, we construct the GuardTrace dataset, which is generated through diverse prompting strategies and refined via a MLRM- and human-based voting and verification pipeline. Furthermore, we propose a three-stage progressive training scheme combined with the data refinement process, enabling the model to learn nuanced and context-dependent safety preferences according to different risk levels. On our proposed test set covering both in-domain and out-of-domain scenarios, GuardTrace-VL model achieves an F1 score of 93.1% on unsafe reasoning detection tasks, representing a 13.5% improvement in F1 score compared to the previous strongest multimodal safety defense methods. The codes will be made publicly available.
Authors Yuxiao Xiang, Junchi Chen, Zhenchao Jin, Changtao Miao, Haojie Yuan, Qi Chu, Tao Gong, Nenghai Yu
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
BUSTR: Breast Ultrasound Text Reporting with a Descriptor-Aware Vision-Language Model
Title CN BUSTR：基于描述符感知视觉-语言模型的乳腺超声文本报告生成
Keywords 医学图像报告生成 · 视觉-语言模型 · 幻觉抑制 · 无监督对齐 · 放射学
Summary 本文提出BUSTR，一种多任务视觉-语言框架，用于在缺乏配对图像-报告数据的情况下自动生成乳腺超声报告。该方法通过结构化描述符（如BI-RADS、病理特征）和放射组学特征构建报告，利用多头Swin编码器学习描述符感知的视觉表示，并采用双层级对齐目标（token级交叉熵+表示空间余弦相似性）实现图文对齐。在BrEaST和BUS-BRA两个公开数据集上验证了其在自然语言生成和临床有效性指标上的提升。
Reason 该论文属于医学图像报告生成领域的垂直应用，使用VLM技术在特定医疗场景（乳腺超声）中生成放射学报告。尽管其方法涉及减少幻觉和无需配对数据训练，但整体框架是面向特定下游任务（RRG）的设计，未提出可迁移至文档图像理解（如DocVQA、表格识别等）的通用方法论创新。属于典型的‘平行下游应用’，不符合本研究关注的技术源头创新要求。
Abstract Automated radiology report generation (RRG) for breast ultrasound (BUS) is limited by the lack of paired image-report datasets and the risk of hallucinations from large language models. We propose BUSTR, a multitask vision-language framework that generates BUS reports without requiring paired image-report supervision. BUSTR constructs reports from structured descriptors (e.g., BI-RADS, pathology, histology) and radiomics features, learns descriptor-aware visual representations with a multi-head Swin encoder trained using a multitask loss over dataset-specific descriptor sets, and aligns visual and textual tokens via a dual-level objective that combines token-level cross-entropy with a cosine-similarity alignment loss between input and output representations. We evaluate BUSTR on two public BUS datasets, BrEaST and BUS-BRA, which differ in size and available descriptors. Across both datasets, BUSTR consistently improves standard natural language generation metrics and clinical efficacy metrics, particularly for key targets such as BI-RADS category and pathology. Our results show that this descriptor-aware vision model, trained with a combined token-level and alignment loss, improves both automatic report metrics and clinical efficacy without requiring paired image-report data. The source code can be found at https://github.com/AAR-UNLV/BUSTR
Authors Rawa Mohammed, Mina Attin, Bryar Shareef
Comment 13 pages, 2 figures, 6 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
Improving Procedural Skill Explanations via Constrained Generation: A Symbolic-LLM Hybrid Architecture
Title CN 通过受限生成改进程序性技能解释：一种符号化-大语言模型混合架构
Keywords 符号系统 · 大语言模型 · 受限生成 · AI教育 · 程序性解释
Summary 本文提出Ivy系统，通过将符号化的任务-方法-知识（TMK）模型与大语言模型结合，在结构约束下生成程序性技能的多步解释。实验表明，相比GPT和检索增强基线，该方法在因果逻辑、目标层级和问题分解方面显著提升了解释的结构性。然而，该工作聚焦于教育场景下的解释生成，属于特定领域应用，缺乏对文档视觉理解任务的支持或迁移潜力。
Reason 该论文属于典型的‘平行应用’，即将符号系统与LLM结合用于教育领域的解释生成，应用场景为AI教学辅导。尽管其方法涉及结构约束下的生成，但核心任务与文档图像理解（DIU）无关，未解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心问题，且无明确可迁移的技术路径到文档理解领域。
Abstract In procedural skill learning, instructional explanations must convey not just steps, but the causal, goal-directed, and compositional logic behind them. Large language models (LLMs) often produce fluent yet shallow responses that miss this structure. We present Ivy, an AI coaching system that delivers structured, multi-step explanations by combining symbolic Task-Method-Knowledge (TMK) models with a generative interpretation layer-an LLM that constructs explanations while being constrained by TMK structure. TMK encodes causal transitions, goal hierarchies, and problem decompositions, and guides the LLM within explicit structural bounds. We evaluate Ivy against responses against GPT and retrieval-augmented GPT baselines using expert and independent annotations across three inferential dimensions. Results show that symbolic constraints consistently improve the structural quality of explanations for "how" and "why" questions. This study demonstrates a scalable AI for education approach that strengthens the pedagogical value of AI-generated explanations in intelligent coaching systems.
Authors Rahul Dass, Thomas Bowlin, Zebing Li, Xiao Jin, Ashok Goel
Categories Artificial Intelligence
2
Canvas-to-Image: Compositional Image Generation with Multimodal Controls
Title CN Canvas-to-Image：基于多模态控制的组合式图像生成
Keywords 图像生成 · 扩散模型 · 多模态控制 · 布局合成 · 视觉合成
Summary 本文提出Canvas-to-Image框架，将多种异构控制信号（如文本提示、参考图像、空间排列、姿态约束等）统一编码为一个复合画布图像，使扩散模型能通过单一输入实现多模态联合控制下的图像生成。作者还构建了多任务数据集并采用多任务训练策略，提升模型在多控制场景下的泛化能力。实验表明该方法在人物组合、姿态控制和布局受限生成等任务上优于现有技术。
Reason 该论文聚焦于组合式图像生成任务，核心目标是融合文本、姿态、布局等多模态控制信号进行高保真图像合成。尽管涉及布局和空间结构，但其研究方向属于纯图像生成领域，与文档图像理解（DIU）中的OCR、信息提取、逻辑推理或去幻觉等核心问题无关。方法本质上服务于创意设计类应用，无法迁移解决文档中的密集文本识别、复杂排版解析或长文档推理等痛点。
Abstract While modern diffusion models excel at generating high-quality and diverse images, they still struggle with high-fidelity compositional and multimodal control, particularly when users simultaneously specify text prompts, subject references, spatial arrangements, pose constraints, and layout annotations. We introduce Canvas-to-Image, a unified framework that consolidates these heterogeneous controls into a single canvas interface, enabling users to generate images that faithfully reflect their intent. Our key idea is to encode diverse control signals into a single composite canvas image that the model can directly interpret for integrated visual-spatial reasoning. We further curate a suite of multi-task datasets and propose a Multi-Task Canvas Training strategy that optimizes the diffusion model to jointly understand and integrate heterogeneous controls into text-to-image generation within a unified learning paradigm. This joint training enables Canvas-to-Image to reason across multiple control modalities rather than relying on task-specific heuristics, and it generalizes well to multi-control scenarios during inference. Extensive experiments show that Canvas-to-Image significantly outperforms state-of-the-art methods in identity preservation and control adherence across challenging benchmarks, including multi-person composition, pose-controlled composition, layout-constrained generation, and multi-control generation.
Authors Yusuf Dalva, Guocheng Gordon Qian, Maya Goldenberg, Tsai-Shien Chen, Kfir Aberman, Sergey Tulyakov, Pinar Yanardag, Kuan-Chieh Jackson Wang
Comment 24 pages; webpage: https://snap-research.github.io/canvas-to-image/
Categories Computer Vision and Pattern Recognition
2
ReSAM: Refine, Requery, and Reinforce: Self-Prompting Point-Supervised Segmentation for Remote Sensing Images
Title CN ReSAM：优化、重查询与强化：用于遥感图像的自提示点监督分割
Keywords 遥感图像 · 交互式分割 · SAM · 自提示 · 点监督
Summary 本文提出一种名为ReSAM的自提示点监督分割框架，用于提升SAM在遥感图像上的分割性能。通过Refine-Requery-Reinforce循环机制，利用稀疏点标注生成伪掩码，并通过自我构建的边界框提示和嵌入对齐来逐步优化分割结果。在多个遥感图像数据集上验证了方法的有效性，优于预训练SAM和其他点监督方法。
Reason 该论文聚焦于遥感图像（RSI）的交互式分割任务，属于特定领域的图像分割应用，虽然使用了SAM并引入自提示机制，但其核心是将现有VLM（SAM）适配到新领域（遥感），并未提出可迁移至文档图像理解的底层方法。且遥感图像与文档图像在结构、语义和排版上差异巨大，技术迁移困难。属于典型的‘平行下游应用’，不符合DIU本题研究方向。
Abstract Interactive segmentation models such as the Segment Anything Model (SAM) have demonstrated remarkable generalization on natural images, but perform suboptimally on remote sensing imagery (RSI) due to severe domain shift and the scarcity of dense annotations. To address this, we propose a self-prompting, point-supervised framework that adapts SAM to RSIs using only sparse point annotations. Our method employs a Refine-Requery-Reinforce loop, where coarse pseudo-masks are generated from initial points (Refine), improved with self-constructed box prompts (Requery), and embeddings are aligned across iterations to reduce confirmation bias (Reinforce). Without relying on full-mask supervision, our approach progressively enhances SAM's segmentation quality and domain robustness through self-guided prompt adaptation . We evaluate our proposed method on three RSI benchmark datasets, including WHU, HRSID, and NWPU VHR-10, showing that our method consistently surpasses pretrained SAM and recent point-supervised segmentation methods. Our results demonstrate that self-prompting and semantic alignment provide an efficient path towards scalable, point-level adaptation of foundation segmentation models for remote sensing applications.
Authors M. Naseer Subhani
Categories Computer Vision and Pattern Recognition
2
DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models
Title CN DiverseVAR：平衡下一代视觉自回归模型的多样性与质量
Keywords 视觉自回归模型 · 图像生成 · 多样性增强 · 潜变量 refine · test-time 方法
Summary 本文提出DiverseVAR框架，旨在提升视觉自回归模型（VAR）在推理阶段生成结果的多样性，同时避免图像质量显著下降。方法包括在文本嵌入中注入噪声以增加多样性，并设计scale-travel机制利用多尺度自编码器进行中间阶段恢复生成，从而缓解质量退化。实验表明该方法在多样性-质量权衡上达到新的Pareto前沿。
Reason 该论文聚焦于文本条件下的视觉自回归生成模型（VAR）在图像生成任务中的多样性不足问题，提出通过文本嵌入加噪和一种名为scale-travel的潜变量 refine 技术来提升生成多样性。尽管方法具有一定创新性，但其核心目标是通用图像生成领域的多样性-质量权衡，属于纯图像生成方向。文档图像理解（DIU）关注的是结构化信息提取、OCR鲁棒性、布局分析等任务，而该工作无法迁移解决OCR幻觉、复杂排版建模或细粒度视觉 grounding 等 DIU 核心痛点。属于无关领域中的图像生成类研究，不符合关注范畴。
Abstract We introduce DiverseVAR, a framework that enhances the diversity of text-conditioned visual autoregressive models (VAR) at test time without requiring retraining, fine-tuning, or substantial computational overhead. While VAR models have recently emerged as strong competitors to diffusion and flow models for image generation, they suffer from a critical limitation in diversity, often producing nearly identical images even for simple prompts. This issue has largely gone unnoticed amid the predominant focus on image quality. We address this limitation at test time in two stages. First, inspired by diversity enhancement techniques in diffusion models, we propose injecting noise into the text embedding. This introduces a trade-off between diversity and image quality: as diversity increases, the image quality sharply declines. To preserve quality, we propose scale-travel: a novel latent refinement technique inspired by time-travel strategies in diffusion models. Specifically, we use a multi-scale autoencoder to extract coarse-scale tokens that enable us to resume generation at intermediate stages. Extensive experiments show that combining text-embedding noise injection with our scale-travel refinement significantly enhances diversity while minimizing image-quality degradation, achieving a new Pareto frontier in the diversity-quality trade-off.
Authors Mingue Park, Prin Phunyaphibarn, Phillip Y. Lee, Minhyuk Sung
Categories Computer Vision and Pattern Recognition
2
Earth-Adapter: Bridge the Geospatial Domain Gaps with Mixture of Frequency Adaptation AAAI 2026
Title CN Earth-Adapter：通过频率自适应混合桥接地理空间领域差距
Keywords Remote Sensing · Parameter-Efficient Fine-Tuning · Frequency Domain · Domain Adaptation · Semantic Segmentation
Summary 本文提出Earth-Adapter，一种专为遥感图像设计的参数高效微调方法，利用离散傅里叶变换将特征分解为不同频率成分，并结合混合适配器（MoA）动态融合各频域信息，以缓解遥感图像中的伪影干扰。在域适应和域泛化语义分割任务上验证了有效性。
Reason 该论文聚焦遥感（RS）图像中的领域适应问题，提出一种针对遥感伪影设计的PEFT方法，核心是频率域分解与MoA机制。尽管其Mixture of Frequency Adaptation思想具有一定技术新颖性，但应用场景完全偏离文档图像理解（DIU），属于特定视觉子领域（Remote Sensing）的下游适配工作，且未涉及文本、布局、OCR或VQA等任何DIU核心任务。无法迁移用于解决OCR幻觉、复杂排版或长文档推理等痛点，属于‘平行应用’类研究。
Abstract Parameter-Efficient Fine-Tuning (PEFT) is a technique that allows us to adapt powerful Foundation Models (FMs) to diverse downstream tasks while preserving and unleashing their inherent capabilities. However, we have observed that existing PEFT methods, which are often designed with natural imagery in mind, struggle when applied to Remote Sensing (RS) scenarios. This is primarily due to their inability to handle artifact influences, a problem particularly severe in RS image features. To tackle this challenge, we introduce Earth-Adapter, the first PEFT method specifically designed for RS artifacts conquering. Earth-Adapter introduces a novel Mixture of Frequency Adaptation process that combines a Mixture of Adapter (MoA) with Discrete Fourier Transformation (DFT). By utilizing DFT, Earth-Adapter can decompose features into different frequency components, precisely separating artifacts from original features. The MoA then dynamically assigns weights to each adapter expert, allowing for the combination of features across various frequency domains. These simple-yet-effective approaches enable Earth-Adapter to more efficiently overcome the disturbances caused by artifacts than previous PEFT methods, significantly enhancing the FMs' performance on RS scenarios. Experiments on Domain Adaptation (DA), and Domain Generalization (DG) semantic segmentation benchmarks showcase the Earth-Adapter's effectiveness. Compared with baseline Rein, Earth-Adapter significantly improves 9.0% mIoU in DA and 3.1% mIoU in DG benchmarks. Our code will be released at https://github.com/VisionXLab/Earth-Adapter.
Authors Xiaoxing Hu, Ziyang Gong, Yupei Wang, Yuru Jia, Fei Lin, Dexiang Gao, Ke An, Jianhong Han, Zhuoran Sun, Gen Luo, Gen Luo, Xue Yang
Comment AAAI 2026 camera ready
Categories Computer Vision and Pattern Recognition
2
ISAC: Training-Free Instance-to-Semantic Attention Control for Improving Multi-Instance Generation
Title CN ISAC：无需训练的实例到语义注意力控制以改进多实例生成
Keywords 文本到图像生成 · 注意力控制 · 实例分割 · 扩散模型 · 语义泄露
Summary 本文提出ISAC方法，通过在扩散模型中对自注意力进行聚类并解耦实例结构与语义分配，实现无需训练的多实例图像生成优化。该方法在多对象生成基准上提升了准确率，但局限于生成任务，不涉及文档理解或视觉语言推理。
Reason 该论文聚焦于文本到图像生成中的多对象生成问题，核心是解决实例边界模糊和语义泄露问题。尽管其提出的注意力控制机制具有一定的结构化思想，但属于图像生成领域的技术优化，且应用场景为T2I模型，与文档图像理解（DIU）任务无直接关联。方法无法迁移用于解决OCR幻觉、复杂排版分析或DocVQA等核心DIU痛点，属于无关领域工作。
Abstract Text-to-image diffusion models have recently become highly capable, yet their behavior in multi-object scenes remains unreliable: models often produce an incorrect number of instances and exhibit semantics leaking across objects. We trace these failures to vague instance boundaries; self-attention already reveals instance layouts early in the denoising process, but existing approaches act only on semantic signals. We introduce $\textbf{ISAC}$ ($\textbf{I}$nstance-to-$\textbf{S}$emantic $\textbf{A}$ttention $\textbf{C}$ontrol), a training-free, model-agnostic objective that performs hierarchical attention control by first carving out instance layouts from self-attention and then binding semantics to these instances. In Phase 1, ISAC clusters self-attention into the number of instances and repels overlaps, establishing an instance-level structural hierarchy; in Phase 2, it injects these instance cues into cross-attention to obtain instance-aware semantic masks and decomposes mixing semantics by tying attributes within each instance. ISAC yields consistent gains on T2I-CompBench, HRS-Bench, and IntraCompBench, our new benchmark for intra-class compositions where failures are most frequent, with improvements of at least 50% in multi-class accuracy and 7% in multi-instance accuracy on IntraCompBench, without any fine-tuning or external models. Beyond text-to-image setups, ISAC also strengthens layout-to-image controllers under overlapping boxes by refining coarse box layouts into dense instance masks, indicating that hierarchical decoupling of instance formation and semantic assignment is a key principle for robust, controllable multi-object generation. Code will be released upon publication.
Authors Sanghyun Jo, Wooyeol Lee, Ziseok Lee, Kyungsu Kim
Comment 36 pages
Categories Computer Vision and Pattern Recognition
2
PathMamba: A Hybrid Mamba-Transformer for Topologically Coherent Road Segmentation in Satellite Imagery
Title CN PathMamba：一种用于卫星图像中拓扑连贯道路分割的混合Mamba-Transformer模型
Keywords 道路分割 · Mamba · Transformer · 遥感图像 · 拓扑连贯性
Summary 本文提出PathMamba，一种结合Mamba和Transformer的混合架构，用于提升卫星图像中道路分割的拓扑连贯性。通过利用Mamba对长连续结构的建模能力和Transformer的全局上下文捕捉能力，在DeepGlobe和Massachusetts Roads数据集上实现了SOTA性能，尤其在APLS指标上有显著提升，同时保持较低计算开销。
Reason 该论文研究的是遥感图像中的道路分割问题，属于自动驾驶/地理信息系统领域，与文档图像理解（DIU）的核心任务如DocVQA、表格识别、文本布局分析等无直接关联。尽管其提出的Mamba-Transformer混合架构在处理长序列结构方面有优势，但应用场景为连续空间几何结构建模，难以迁移到文档图像中的文本语义理解、OCR去幻觉或复杂排版解析等痛点问题。
Abstract Achieving both high accuracy and topological continuity in road segmentation from satellite imagery is a critical goal for applications ranging from urban planning to disaster response. State-of-the-art methods often rely on Vision Transformers, which excel at capturing global context, yet their quadratic complexity is a significant barrier to efficient deployment, particularly for on-board processing in resource-constrained platforms. In contrast, emerging State Space Models like Mamba offer linear-time efficiency and are inherently suited to modeling long, continuous structures. We posit that these architectures have complementary strengths. To this end, we introduce PathMamba, a novel hybrid architecture that integrates Mamba's sequential modeling with the Transformer's global reasoning. Our design strategically uses Mamba blocks to trace the continuous nature of road networks, preserving topological structure, while integrating Transformer blocks to refine features with global context. This approach yields topologically superior segmentation maps without the prohibitive scaling costs of pure attention-based models. Our experiments on the DeepGlobe Road Extraction and Massachusetts Roads datasets demonstrate that PathMamba sets a new state-of-the-art. Notably, it significantly improves topological continuity, as measured by the APLS metric, setting a new benchmark while remaining computationally competitive.
Authors Jules Decaestecker, Nicolas Vigne
Comment 11 pages, 5 figures
Categories Computer Vision and Pattern Recognition
2
AMLP: Adjustable Masking Lesion Patches for Self-Supervised Medical Image Segmentation
Title CN AMLP：用于自监督医学图像分割的可调掩码病灶块
Keywords 自监督学习 · 医学图像分割 · 掩码图像建模 · 病灶重建 · 可调掩码比
Summary 本文提出一种名为AMLP的自监督医学图像分割框架，通过可调节的掩码比例策略（AMR）和基于病变概率的掩码块选择（MPS），结合相对重建损失（RRL）与类别一致性损失（CCL），提升对难重建病灶区域的学习能力。实验表明该方法在两个医学分割数据集上优于现有自监督方法。
Reason 该论文聚焦于医学图像分割任务，属于纯计算机视觉领域，且针对的是医学影像这一特定模态。虽然其提出的可调掩码策略和重建损失设计具有一定方法价值，但整体框架面向的是病灶识别与自然图像差异较大的医学场景，与文档图像理解（DIU）在数据分布、任务目标（如文本识别、布局分析、VQA）上存在本质差异。未涉及多模态大模型、OCR、文本结构建模或文档排版理解等核心问题，无法迁移解决DIU中的OCR幻觉、复杂排版或长文档推理等痛点。
Abstract Self-supervised masked image modeling (MIM) methods have shown promising performances on analyzing natural images. However, directly applying such methods to medical image segmentation tasks still cannot achieve satisfactory results. The challenges arise from the facts that (i) medical images are inherently more complex compared to natural images, and the subjects in medical images often exhibit more distinct contour features; (ii) moreover, the conventional high and fixed masking ratio in MIM is likely to mask the background, limiting the scope of learnable information. To address these problems, we propose a new self-supervised medical image segmentation framework, called Adjustable Masking Lesion Patches (AMLP), which employs Masked Patch Selection~(MPS) strategy to identify patches with high probabilities of containing lesions to help model achieve precise lesion reconstruction. To improve the categorization of patches in MPS, we further introduce Relative Reconstruction Loss (RRL) to better learn hard-to-reconstruct lesion patches. Then, Category Consistency Loss (CCL) is proposed to refine patch categorization based on reconstruction difficulty, enhancing difference between lesions and backgrounds. Moreover, an Adjustable Masking Ratio (AMR) strategy is proposed to gradually increase the masking ratio over training to expand~the scope of learnable mutual information. Extensive~experiments on two medical segmentation datasets demonstrate the superior performances of the proposed AMLP w.r.t. the SOTA self-supervised methods; the results prove that AMLP effectively addresses the challenges of applying masked modeling to medical images and capturing accurate lesion details that are crucial for segmentation tasks.
Authors Xiangtao Wang, Ruizhi Wang, Thomas Lukasiewicz, Zhenghua Xu
Comment © 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works
Categories Computer Vision and Pattern Recognition
2
You Can Trust Your Clustering Model: A Parameter-free Self-Boosting Plug-in for Deep Clustering NeurIPS 2025
Title CN 你可以信任你的聚类模型：一种用于深度聚类的无参数自增强插件
Keywords 深度聚类 · 特征结构优化 · 无监督学习 · 局部一致性 · 全局分离性
Summary 本文提出DCBoost，一种无参数的插件式方法，用于增强深度聚类模型的全局特征结构。通过基于自适应k近邻的一致性过滤筛选高置信度样本，并利用这些样本构建判别性损失以优化网络，提升类内紧凑性和类间可分性。实验表明该方法能显著提升多种主流聚类模型的性能。
Reason 该论文聚焦于通用图像表示学习中的深度聚类问题，提出了一种提升聚类全局结构分离性的后处理插件。尽管其利用局部结构一致性来增强全局特征的方法在理论上具有一定启发性，但其核心任务与文档图像理解（DIU）无直接关联，既不涉及多模态大模型（VLM）、OCR、布局分析、表格识别，也不解决文档特有的密集文本、复杂排版或长文档推理等痛点。方法本质是面向无监督分类任务的优化，迁移到DocVQA或VIE等场景缺乏明确路径，属于典型的‘平行应用’类工作的上游基础研究，但本身并非DIU本题工作，也无显著可迁移的技术范式。
Abstract Recent deep clustering models have produced impressive clustering performance. However, a common issue with existing methods is the disparity between global and local feature structures. While local structures typically show strong consistency and compactness within class samples, global features often present intertwined boundaries and poorly separated clusters. Motivated by this observation, we propose DCBoost, a parameter-free plug-in designed to enhance the global feature structures of current deep clustering models. By harnessing reliable local structural cues, our method aims to elevate clustering performance effectively. Specifically, we first identify high-confidence samples through adaptive $k$-nearest neighbors-based consistency filtering, aiming to select a sufficient number of samples with high label reliability to serve as trustworthy anchors for self-supervision. Subsequently, these samples are utilized to compute a discriminative loss, which promotes both intra-class compactness and inter-class separability, to guide network optimization. Extensive experiments across various benchmark datasets showcase that our DCBoost significantly improves the clustering performance of diverse existing deep clustering models. Notably, our method improves the performance of current state-of-the-art baselines (e.g., ProPos) by more than 3% and amplifies the silhouette coefficient by over $7\times$. Code is available at <https://github.com/l-h-y168/DCBoost>.
Authors Hanyang Li, Yuheng Jia, Hui Liu, Junhui Hou
Comment The paper is accepted by NeurIPS 2025
Categories Computer Vision and Pattern Recognition
2
UniChange: Unifying Change Detection with Multimodal Large Language Model
Title CN UniChange：基于多模态大语言模型的统一变化检测
Keywords 变化检测 · 多模态大语言模型 · 遥感图像 · 统一框架 · 提示学习
Summary 本文提出UniChange，是首个基于多模态大语言模型的变化检测框架，旨在统一二值变化检测（BCD）和语义变化检测（SCD）任务。通过引入特殊标记[T1]、[T2]和[CHANGE]，并利用文本提示指导变化类别识别，摆脱对固定分类头的依赖，从而实现跨数据集的知识融合与更高泛化性。在四个公开遥感数据集上达到SOTA性能。
Reason 该论文属于遥感图像分析领域，研究的是双时相遥感影像中的变化检测（如土地覆盖变化），其核心任务与文档图像理解（DIU）无直接关联。尽管使用了多模态大语言模型（MLLM）进行任务统一和提示引导分类，但其方法设计面向的是像素级遥感语义分割任务，无法迁移解决文档理解中的OCR幻觉、复杂排版、细粒度文本定位等关键痛点。属于典型的‘平行应用’类工作——将MLLM套用于特定下游任务，缺乏对文档结构、文本布局或高分辨率文档处理的建模创新。
Abstract Change detection (CD) is a fundamental task for monitoring and analyzing land cover dynamics. While recent high performance models and high quality datasets have significantly advanced the field, a critical limitation persists. Current models typically acquire limited knowledge from single-type annotated data and cannot concurrently leverage diverse binary change detection (BCD) and semantic change detection (SCD) datasets. This constraint leads to poor generalization and limited versatility. The recent advancements in Multimodal Large Language Models (MLLMs) introduce new possibilities for a unified CD framework. We leverage the language priors and unification capabilities of MLLMs to develop UniChange, the first MLLM-based unified change detection model. UniChange integrates generative language abilities with specialized CD functionalities. Our model successfully unifies both BCD and SCD tasks through the introduction of three special tokens: [T1], [T2], and [CHANGE]. Furthermore, UniChange utilizes text prompts to guide the identification of change categories, eliminating the reliance on predefined classification heads. This design allows UniChange to effectively acquire knowledge from multi-source datasets, even when their class definitions conflict. Experiments on four public benchmarks (WHU-CD, S2Looking, LEVIR-CD+, and SECOND) demonstrate SOTA performance, achieving IoU scores of 90.41, 53.04, 78.87, and 57.62, respectively, surpassing all previous methods. The code is available at https://github.com/Erxucomeon/UniChange.
Authors Xu Zhang, Danyang Li, Xiaohang Dong, Tianhao Wu, Hualong Yu, Jianye Wang, Qicheng Li, Xiang Li
Categories Computer Vision and Pattern Recognition, Computation and Language
2
Referring Video Object Segmentation with Cross-Modality Proxy Queries
Title CN 基于跨模态代理查询的指代表视频对象分割
Keywords 视频理解 · 指代表分割 · 跨模态对齐 · Transformer · 代理查询
Summary 本文提出ProxyFormer模型用于指代表视频对象分割，通过引入可传播的代理查询来融合视觉与语言语义，并在多阶段更新中建立帧间依赖，提升跨模态对齐与目标追踪的一致性。同时采用时空解耦的计算方式降低开销，并设计联合语义一致性训练策略优化对齐效果。实验表明其在四个主流RVOS基准上优于现有方法。
Reason 该论文研究的是指代表视频对象分割（RVOS），属于视频理解领域，涉及的是像素级时序分割与语言表达对齐，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的代理查询机制和跨模态对齐策略在理论上具有一定通用性，但其方法设计高度依赖于视频帧序列和时空建模，迁移到文档场景（如处理OCR幻觉、复杂排版或长文档推理）难度极大。且未涉及文档特有的高分辨率、密集文本、结构化输出等问题，属于无关领域工作。
Abstract Referring video object segmentation (RVOS) is an emerging cross-modality task that aims to generate pixel-level maps of the target objects referred by given textual expressions. The main concept involves learning an accurate alignment of visual elements and language expressions within a semantic space. Recent approaches address cross-modality alignment through conditional queries, tracking the target object using a query-response based mechanism built upon transformer structure. However, they exhibit two limitations: (1) these conditional queries lack inter-frame dependency and variation modeling, making accurate target tracking challenging amid significant frame-to-frame variations; and (2) they integrate textual constraints belatedly, which may cause the video features potentially focus on the non-referred objects. Therefore, we propose a novel RVOS architecture called ProxyFormer, which introduces a set of proxy queries to integrate visual and text semantics and facilitate the flow of semantics between them. By progressively updating and propagating proxy queries across multiple stages of video feature encoder, ProxyFormer ensures that the video features are focused on the object of interest. This dynamic evolution also enables the establishment of inter-frame dependencies, enhancing the accuracy and coherence of object tracking. To mitigate high computational costs, we decouple cross-modality interactions into temporal and spatial dimensions. Additionally, we design a Joint Semantic Consistency (JSC) training strategy to align semantic consensus between the proxy queries and the combined video-text pairs. Comprehensive experiments on four widely used RVOS benchmarks demonstrate the superiority of our ProxyFormer to the state-of-the-art methods.
Authors Baoli Sun, Xinzhu Ma, Ning Wang, Zhihui Wang, Zhiyong Wang
Categories Computer Vision and Pattern Recognition
2
Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaboration
Title CN 通过增量生成与多智能体协作实现多尺度时间预测
Keywords 时间预测 · 多智能体协作 · 视频理解 · 增量生成 · 状态预测
Summary 本文提出了多尺度时间预测（MSTP）任务，并构建了首个包含多状态层级与多时间跨度标注的基准数据集。方法上引入增量生成模块以同步视觉预览与决策，并设计了一个由生成、启动和评估智能体组成的协作框架来平衡全局一致性与局部保真度。整体工作面向视频理解与具身AI，不适用于文档图像理解场景。
Reason 该论文研究的是通用场景和手术场景下的多尺度时间状态预测任务，属于具身智能与视频理解领域，核心关注点为动态视觉-语言建模中的未来状态推断。其提出的IG-MC方法虽涉及多智能体协作与增量生成，但目标是解决长时间跨度的视频未来预测问题，而非文档图像理解中的OCR幻觉、复杂排版或长文档推理等痛点。方法难以迁移到静态文档分析任务中，且与DocVQA、布局分析、表格识别等DIU核心任务无直接关联。
Abstract Accurate temporal prediction is the bridge between comprehensive scene understanding and embodied artificial intelligence. However, predicting multiple fine-grained states of a scene at multiple temporal scales is difficult for vision-language models. We formalize the Multi-Scale Temporal Prediction (MSTP) task in general and surgical scenes by decomposing multi-scale into two orthogonal dimensions: the temporal scale, forecasting states of humans and surgery at varying look-ahead intervals, and the state scale, modeling a hierarchy of states in general and surgical scenes. For example, in general scenes, states of contact relationships are finer-grained than states of spatial relationships. In surgical scenes, medium-level steps are finer-grained than high-level phases yet remain constrained by their encompassing phase. To support this unified task, we introduce the first MSTP Benchmark, featuring synchronized annotations across multiple state scales and temporal scales. We further propose a method, Incremental Generation and Multi-agent Collaboration (IG-MC), which integrates two key innovations. First, we present a plug-and-play incremental generation module that continuously synthesizes up-to-date visual previews at expanding temporal scales to inform multiple decision-making agents, keeping decisions and generated visuals synchronized and preventing performance degradation as look-ahead intervals lengthen. Second, we present a decision-driven multi-agent collaboration framework for multi-state prediction, comprising generation, initiation, and multi-state assessment agents that dynamically trigger and evaluate prediction cycles to balance global coherence and local fidelity.
Authors Zhitao Zeng, Guojian Yuan, Junyuan Mao, Yuxuan Wang, Xiaoshuang Jia, Yueming Jin
Comment 20 pages, 6 figures
Categories Computer Vision and Pattern Recognition
2
MIRA: Multimodal Iterative Reasoning Agent for Image Editing
Title CN MIRA：用于图像编辑的多模态迭代推理代理
Keywords 图像编辑 · 多模态代理 · 迭代推理 · 扩散模型 · 自然语言指令
Summary 本文提出MIRA，一个用于指导图像编辑的多模态推理代理，通过迭代地解析用户指令并结合视觉反馈执行原子级编辑操作，提升编辑结果的语义一致性和视觉质量。模型基于自建数据集MIRA-Editing训练，在多个开源编辑模型上实现了接近专有系统的性能。
Reason 该论文属于图像编辑领域，核心任务是基于自然语言指令对图像进行修改，使用扩散模型和代理框架实现迭代编辑。尽管其‘迭代感知-推理-行动’机制在方法论上有一定启发性，但整体工作聚焦于通用图像生成后的编辑控制，与文档图像理解（如DocVQA、布局分析、表格识别等）的核心任务无直接关联。其处理的语义漂移问题虽类似幻觉，但场景为创意图像修改而非文档结构化理解，迁移价值极低。属于典型的非DIU本题且难以向上游工具迁移的工作。
Abstract Instruction-guided image editing offers an intuitive way for users to edit images with natural language. However, diffusion-based editing models often struggle to accurately interpret complex user instructions, especially those involving compositional relationships, contextual cues, or referring expressions, leading to edits that drift semantically or fail to reflect the intended changes. We tackle this problem by proposing MIRA (Multimodal Iterative Reasoning Agent), a lightweight, plug-and-play multimodal reasoning agent that performs editing through an iterative perception-reasoning-action loop, effectively simulating multi-turn human-model interaction processes. Instead of issuing a single prompt or static plan, MIRA predicts atomic edit instructions step by step, using visual feedback to make its decisions. Our 150K multimodal tool-use dataset, MIRA-Editing, combined with a two-stage SFT + GRPO training pipeline, enables MIRA to perform reasoning and editing over complex editing instructions. When paired with open-source image editing models such as Flux.1-Kontext, Step1X-Edit, and Qwen-Image-Edit, MIRA significantly improves both semantic consistency and perceptual quality, achieving performance comparable to or exceeding proprietary systems such as GPT-Image and Nano-Banana.
Authors Ziyun Zeng, Hang Hua, Jiebo Luo
Categories Computer Vision and Pattern Recognition
2
Thinking in 360°: Humanoid Visual Search in the Wild
Title CN 360°思考：野外的人形视觉搜索
Keywords 具身智能 · 视觉搜索 · 360°全景 · 空间推理 · VLM
Summary 本文提出一种模拟人头眼协同机制的人形视觉搜索框架，在360°全景环境中实现主动物体与路径搜索，并构建H* Bench基准测试集评估模型表现。实验表明现有顶尖模型在此类复杂真实场景中表现不佳，通过后训练可显著提升开源Qwen2.5-VL的表现。然而该研究聚焦于具身交互与空间推理，不涉及文档结构或文本理解。
Reason 该论文研究的是基于360°全景图像的具身智能体视觉搜索，属于具身智能与机器人感知范畴，核心任务是物体和路径搜索，应用场景为交通枢纽、城市街道等开放场景。尽管使用了VLM（Qwen2.5-VL）进行后训练优化，但其方法本质是特定形态的Agent在3D环境中的导航与搜索策略，与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析、长文档推理等无直接关联。该工作无法迁移用于解决文档中的细粒度定位或高分辨率处理问题，且属于‘具身智能’这一无关领域，符合负面清单第2条。
Abstract Humans rely on the synergistic control of head (cephalomotor) and eye (oculomotor) to efficiently search for visual information in 360°. However, prior approaches to visual search are limited to a static image, neglecting the physical embodiment and its interaction with the 3D world. How can we develop embodied visual search agents as efficient as humans while bypassing the constraints imposed by real-world hardware? To this end, we propose humanoid visual search where a humanoid agent actively rotates its head to search for objects or paths in an immersive world represented by a 360° panoramic image. To study visual search in visually-crowded real-world scenarios, we build H* Bench, a new benchmark that moves beyond household scenes to challenging in-the-wild scenes that necessitate advanced visual-spatial reasoning capabilities, such as transportation hubs, large-scale retail spaces, urban streets, and public institutions. Our experiments first reveal that even top-tier proprietary models falter, achieving only ~30% success in object and path search. We then use post-training techniques to enhance the open-source Qwen2.5-VL, increasing its success rate by over threefold for both object search (14.83% to 47.38%) and path search (6.44% to 24.94%). Notably, the lower ceiling of path search reveals its inherent difficulty, which we attribute to the demand for sophisticated spatial commonsense. Our results not only show a promising path forward but also quantify the immense challenge that remains in building MLLM agents that can be seamlessly integrated into everyday human life.
Authors Heyang Yu, Yinan Han, Xiangyu Zhang, Baiqiao Yin, Bowen Chang, Xiangyu Han, Xinhao Liu, Jing Zhang, Marco Pavone, Chen Feng, Saining Xie, Yiming Li
Comment Website: https://humanoid-vstar.github.io/ ; Code: https://github.com/humanoid-vstar/hstar
Categories Computer Vision and Pattern Recognition
2
Scale Where It Matters: Training-Free Localized Scaling for Diffusion Models
Title CN 在关键处扩展：扩散模型的无需训练的局部扩展方法
Keywords 扩散模型 · 测试时扩展 · 局部优化 · 图像生成 · 注意力机制
Summary 本文提出了LoTTS，一种无需训练的局部测试时扩展框架，用于在推理阶段仅对生成图像中的局部缺陷区域进行重采样和优化，从而提升整体生成质量并降低计算成本。通过对比交叉与自注意力信号来定位缺陷，并在局部扰动和去噪以保持全局一致性。
Reason 该论文研究的是文本到图像生成中的测试时扩展（TTS）问题，提出了一种针对图像局部缺陷进行自适应重采样的方法。尽管其‘局部化计算’的思想具有一定启发性，但其核心任务属于纯图像生成领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。文中未涉及OCR、布局分析、表格识别或多模态语言模型在文档上的应用，且方法依赖于扩散模型内部注意力机制的空间异质性分析，难以迁移到文档场景中解决OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Diffusion models have become the dominant paradigm in text-to-image generation, and test-time scaling (TTS) further improves quality by allocating more computation during inference. However, existing TTS methods operate at the full-image level, overlooking the fact that image quality is often spatially heterogeneous. This leads to unnecessary computation on already satisfactory regions and insufficient correction of localized defects. In this paper, we explore a new direction - Localized TTS - that adaptively resamples defective regions while preserving high-quality regions, thereby substantially reducing the search space. This paradigm poses two central challenges: accurately localizing defects and maintaining global consistency. We propose LoTTS, the first fully training-free framework for localized TTS. For defect localization, LoTTS contrasts cross- and self-attention signals under quality-aware prompts (e.g., high-quality vs. low-quality) to identify defective regions, and then refines them into coherent masks. For consistency, LoTTS perturbs only defective regions and denoises them locally, ensuring that corrections remain confined while the rest of the image remains undisturbed. Extensive experiments on SD2.1, SDXL, and FLUX demonstrate that LoTTS achieves state-of-the-art performance: it consistently improves both local quality and global fidelity, while reducing GPU cost by 2-4x compared to Best-of-N sampling. These findings establish localized TTS as a promising new direction for scaling diffusion models at inference time.
Authors Qin Ren, Yufei Wang, Lanqing Guo, Wen Zhang, Zhiwen Fan, Chenyu You
Categories Computer Vision and Pattern Recognition
2
MetaRank: Task-Aware Metric Selection for Model Transferability Estimation
Title CN MetaRank：面向任务的模型可迁移性估计中的度量选择
Keywords 模型可迁移性估计 · 元学习 · 学习排序 · 语言模型 · 任务感知
Summary 本文提出MetaRank，一种基于元学习的任务感知模型可迁移性估计（MTE）度量选择框架。它将度量选择建模为学习排序问题，利用预训练语言模型编码数据集和MTE度量的文本描述，在共享语义空间中训练一个元预测器来排序候选度量。实验表明该方法能有效为新目标数据集推荐最优MTE度量。然而，整个工作不涉及视觉输入处理或多模态建模，也不针对文档理解任务。
Reason 该论文属于模型选择与元学习范畴，核心是通过语言模型对数据集文本描述和MTE度量进行语义匹配，以实现任务感知的度量排序。其研究目标为加速模型迁移过程中的度量选择，不涉及任何文档图像理解（DIU）任务本身（如DocVQA、布局分析、表格识别等），也未提出可用于解决OCR幻觉、细粒度定位或复杂排版处理的技术工具。尽管使用了语言模型进行语义建模，但方法聚焦于元学习框架下的指标选择优化，与文档多模态理解无直接关联，且无法迁移用于提升VLM在文档场景下的推理或对齐能力。
Abstract Selecting an appropriate pre-trained source model is a critical, yet computationally expensive, task in transfer learning. Model Transferability Estimation (MTE) methods address this by providing efficient proxy metrics to rank models without full fine-tuning. In practice, the choice of which MTE metric to use is often ad hoc or guided simply by a metric's average historical performance. However, we observe that the effectiveness of MTE metrics is highly task-dependent and no single metric is universally optimal across all target datasets. To address this gap, we introduce MetaRank, a meta-learning framework for automatic, task-aware MTE metric selection. We formulate metric selection as a learning-to-rank problem. Rather than relying on conventional meta-features, MetaRank encodes textual descriptions of both datasets and MTE metrics using a pretrained language model, embedding them into a shared semantic space. A meta-predictor is then trained offline on diverse meta-tasks to learn the intricate relationship between dataset characteristics and metric mechanisms, optimized with a listwise objective that prioritizes correctly ranking the top-performing metrics. During the subsequent online phase, MetaRank efficiently ranks the candidate MTE metrics for a new, unseen target dataset based on its textual description, enabling practitioners to select the most appropriate metric a priori. Extensive experiments across 11 pretrained models and 11 target datasets demonstrate the strong effectiveness of our approach.
Authors Yuhang Liu, Wenjie Zhao, Yunhui Guo
Comment 10 figures
Categories Computer Vision and Pattern Recognition
2
From Inpainting to Layer Decomposition: Repurposing Generative Inpainting Models for Image Layer Decomposition
Title CN 从图像修复到图层分解：重用生成式修复模型进行图像图层分解
Keywords 图像图层分解 · 图像修复 · 扩散模型 · 合成数据 · 多模态融合
Summary 本文提出通过微调扩散型图像修复模型来实现单张图像的图层分解，引入了一种多模态上下文融合模块以保留潜在空间中的细节，并在纯合成数据上进行训练。实验表明其在物体移除和遮挡恢复方面表现良好，适用于图像编辑和创意应用。
Reason 该论文研究的是通用图像的图层分解问题，属于图像编辑与生成领域，其方法基于扩散模型和合成数据训练，核心目标是实现前景与背景的分离以支持内容编辑。尽管涉及视觉结构理解，但其任务设定、数据构造和评估均不针对文档图像特有的挑战（如文本布局、OCR对齐、密集排版或语义结构）。该工作无法直接迁移用于解决文档图像理解中的关键痛点（如VIE、表格识别或DocVQA），且未涉及多模态语言模型在文档上的应用或视觉token压缩等DIU前沿方向。属于无关领域的技术探索。
Abstract Images can be viewed as layered compositions, foreground objects over background, with potential occlusions. This layered representation enables independent editing of elements, offering greater flexibility for content creation. Despite the progress in large generative models, decomposing a single image into layers remains challenging due to limited methods and data. We observe a strong connection between layer decomposition and in/outpainting tasks, and propose adapting a diffusion-based inpainting model for layer decomposition using lightweight finetuning. To further preserve detail in the latent space, we introduce a novel multi-modal context fusion module with linear attention complexity. Our model is trained purely on a synthetic dataset constructed from open-source assets and achieves superior performance in object removal and occlusion recovery, unlocking new possibilities in downstream editing and creative applications.
Authors Jingxi Chen, Yixiao Zhang, Xiaoye Qian, Zongxia Li, Cornelia Fermuller, Caren Chen, Yiannis Aloimonos
Categories Computer Vision and Pattern Recognition
2
Leveraging Contrast Information for Efficient Document Shadow Removal
Title CN 利用对比信息实现高效的文档去阴影
Keywords 文档去阴影 · 对比信息 · 图像增强 · 粗到精优化 · 计算机视觉
Summary 本文提出一种基于对比表示的端到端文档去阴影方法，通过粗到精的优化策略，利用原始图像中的对比信息定位阴影区域并指导去除过程，无需额外的阴影掩码，在定性和定量实验上达到了SOTA效果。
Reason 该论文聚焦于文档图像预处理中的去阴影任务，属于图像增强范畴，并未涉及文档理解（DIU）的核心任务如DocVQA、布局分析、表格识别或文本内容结构化。虽然提升图像质量可能间接有利于后续OCR或VLM处理，但其本身是典型的‘平行图像处理技术’，没有针对DIU中的核心痛点（如OCR幻觉、复杂排版推理、长文档建模）提出方法论创新。且该工作未引入可用于VLM架构改进、推理增强或视觉token压缩等可迁移的底层机制，属于低分段的图像预处理应用。
Abstract Document shadows are a major obstacle in the digitization process. Due to the dense information in text and patterns covered by shadows, document shadow removal requires specialized methods. Existing document shadow removal methods, although showing some progress, still rely on additional information such as shadow masks or lack generalization and effectiveness across different shadow scenarios. This often results in incomplete shadow removal or loss of original document content and tones. Moreover, these methods tend to underutilize the information present in the original shadowed document image. In this paper, we refocus our approach on the document images themselves, which inherently contain rich information.We propose an end-to-end document shadow removal method guided by contrast representation, following a coarse-to-fine refinement approach. By extracting document contrast information, we can effectively and quickly locate shadow shapes and positions without the need for additional masks. This information is then integrated into the refined shadow removal process, providing better guidance for network-based removal and feature fusion. Extensive qualitative and quantitative experiments show that our method achieves state-of-the-art performance.
Authors Yifan Liu, Jiancheng Huang, Na Liu, Mingfu Yan, Yi Huang, Shifeng Chen
Comment There are unresolved authorship disputes related to this submission, and the current version does not reflect an agreed authorship list
Categories Computer Vision and Pattern Recognition
2
AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following
Title CN AdvancedIF：基于评分标准的基准测试与强化学习以推进大语言模型指令遵循能力
Keywords 指令遵循 · 强化学习 · 基准测试 · 大语言模型 · 奖励建模
Summary 本文提出了AdvancedIF，一个用于评估大语言模型在复杂多轮指令下表现的基准，以及RIFL，一种基于评分标准生成与验证的强化学习训练方法。实验显示该方法在指令遵循任务上显著提升性能。然而，整个工作局限于纯文本模态，未涉及图像或文档结构信息。
Reason 该论文聚焦于大语言模型在复杂、多轮和系统级指令下的指令遵循（Instruction Following）能力，提出了一种基于评分标准（rubric）的评估基准AdvancedIF和训练方法RIFL。尽管其强化学习框架和可解释奖励信号设计在通用LLM对齐领域具有价值，但研究完全基于纯文本任务，未涉及任何视觉输入或文档模态。其方法迁移至文档图像理解（DIU）场景的前提是存在细粒度的视觉-语言对齐机制，而文中并未提供处理OCR幻觉、布局感知或高分辨率图像的能力。属于典型的‘平行应用’范式——将现有LLM对齐技术应用于新定义的文本指令任务，无视觉或多模态创新。
Abstract Recent progress in large language models (LLMs) has led to impressive performance on a range of tasks, yet advanced instruction following (IF)-especially for complex, multi-turn, and system-prompted instructions-remains a significant challenge. Rigorous evaluation and effective training for such capabilities are hindered by the lack of high-quality, human-annotated benchmarks and reliable, interpretable reward signals. In this work, we introduce AdvancedIF (we will release this benchmark soon), a comprehensive benchmark featuring over 1,600 prompts and expert-curated rubrics that assess LLMs ability to follow complex, multi-turn, and system-level instructions. We further propose RIFL (Rubric-based Instruction-Following Learning), a novel post-training pipeline that leverages rubric generation, a finetuned rubric verifier, and reward shaping to enable effective reinforcement learning for instruction following. Extensive experiments demonstrate that RIFL substantially improves the instruction-following abilities of LLMs, achieving a 6.7% absolute gain on AdvancedIF and strong results on public benchmarks. Our ablation studies confirm the effectiveness of each component in RIFL. This work establishes rubrics as a powerful tool for both training and evaluating advanced IF in LLMs, paving the way for more capable and reliable AI systems.
Authors Yun He, Wenzhe Li, Hejia Zhang, Songlin Li, Karishma Mandyam, Sopan Khosla, Yuanhao Xiong, Nanshu Wang, Xiaoliang Peng, Beibin Li, Shengjie Bi, Shishir G. Patil, Qi Qi, Shengyu Feng, Julian Katz-Samuels, Richard Yuanzhe Pang, Sujan Gonugondla, Hunter Lang, Yue Yu, Yundi Qian, Maryam Fazel-Zarandi, Licheng Yu, Amine Benhalloum, Hany Awadalla, Manaal Faruqui
Categories Computation and Language
2
Gram2Vec: An Interpretable Document Vectorizer
Title CN Gram2Vec：一种可解释的文档向量化方法
Keywords 文本向量化 · 语法特征 · 可解释性 · 作者验证 · AI检测
Summary 本文提出Gram2Vec，通过提取文本中语法特征的归一化频率来生成可解释的文档向量，并应用于作者验证和AI内容检测。相比神经网络方法，其优势在于特征可解释性，但完全基于纯文本分析，不处理图像或布局信息。
Reason 该论文聚焦于文本层面的语法特征提取与向量化，属于纯NLP范畴，且其应用为作者验证和AI检测，未涉及任何视觉或文档图像模态。完全忽略文档的布局、结构、视觉排版等核心DIU要素，也未与多模态大模型（VLM）或OCR系统结合。属于典型的‘平行下游应用’，仅将传统语言学特征用于新任务，无DIU方法论创新。
Abstract We present Gram2Vec, a grammatical style embedding system that embeds documents into a higher dimensional space by extracting the normalized relative frequencies of grammatical features present in the text. Compared to neural approaches, Gram2Vec offers inherent interpretability based on how the feature vectors are generated. In this paper, we use authorship verification and AI detection as two applications to show how Gram2Vec can be used. For authorship verification, we use the features from Gram2Vec to explain why a pair of documents is by the same or by different authors. We also demonstrate how Gram2Vec features can be used to train a classifier for AI detection, outperforming machine learning models trained on a comparable set of Biber features.
Authors Peter Zeng, Hannah Stortz, Eric Sclafani, Alina Shabaeva, Maria Elizabeth Garza, Daniel Greeson, Owen Rambow
Comment 8 pages, 1 figure
Categories Computation and Language
2
Chatty-KG: A Multi-Agent AI System for On-Demand Conversational Question Answering over Knowledge Graphs SIGMOD 2026
Title CN Chatty-KG：一种用于知识图谱上按需对话式问答的多智能体AI系统
Keywords 多智能体系统 · 知识图谱问答 · 对话式QA · RAG · SPARQL生成
Summary 本文提出了Chatty-KG，一个基于多智能体的对话式问答系统，用于在知识图谱上进行自然语言交互。系统通过多个专用LLM智能体协同完成实体链接、关系映射、对话跟踪和SPARQL查询生成，结合了RAG的灵活性与传统KGQA的结构保持能力。实验表明其在单轮和多轮设置下均优于现有方法，且兼容多种主流LLM。然而，该工作聚焦于结构化知识库的接口问题，不涉及视觉输入或文档模态。
Reason 该论文属于典型的‘平行应用’类型，即将多Agent架构应用于知识图谱上的对话问答任务。尽管其系统设计具有模块化和可扩展性优点，但核心是RAG与SPARQL生成在特定结构化数据源（KG）上的工程整合，并未提出可用于文档图像理解（如OCR去幻觉、复杂排版建模、视觉-文本对齐等）的新方法或底层技术创新。其‘多Agent协作’机制针对的是语义解析与对话状态追踪，难以迁移到DIU中的视觉布局分析或细粒度文本定位任务。
Abstract Conversational Question Answering over Knowledge Graphs (KGs) combines the factual grounding of KG-based QA with the interactive nature of dialogue systems. KGs are widely used in enterprise and domain applications to provide structured, evolving, and reliable knowledge. Large language models (LLMs) enable natural and context-aware conversations, but lack direct access to private and dynamic KGs. Retrieval-augmented generation (RAG) systems can retrieve graph content but often serialize structure, struggle with multi-turn context, and require heavy indexing. Traditional KGQA systems preserve structure but typically support only single-turn QA, incur high latency, and struggle with coreference and context tracking. To address these limitations, we propose Chatty-KG, a modular multi-agent system for conversational QA over KGs. Chatty-KG combines RAG-style retrieval with structured execution by generating SPARQL queries through task-specialized LLM agents. These agents collaborate for contextual interpretation, dialogue tracking, entity and relation linking, and efficient query planning, enabling accurate and low-latency translation of natural questions into executable queries. Experiments on large and diverse KGs show that Chatty-KG significantly outperforms state-of-the-art baselines in both single-turn and multi-turn settings, achieving higher F1 and P@1 scores. Its modular design preserves dialogue coherence and supports evolving KGs without fine-tuning or pre-processing. Evaluations with commercial (e.g., GPT-4o, Gemini-2.0) and open-weight (e.g., Phi-4, Gemma 3) LLMs confirm broad compatibility and stable performance. Overall, Chatty-KG unifies conversational flexibility with structured KG grounding, offering a scalable and extensible approach for reliable multi-turn KGQA.
Authors Reham Omar, Abdelghny Orogat, Ibrahim Abdelaziz, Omij Mangukiya, Panos Kalnis, Essam Mansour
Comment This paper is accepted to SIGMOD 2026
Categories Computation and Language
2
BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agents
Title CN BrowseSafe：理解和防范AI浏览器代理中的提示注入
Keywords 提示注入 · AI代理安全 · 浏览器自动化 · 多层防御 · 实证评估
Summary 本文研究了在AI浏览器代理中提示注入攻击的实际影响，构建了一个基于真实HTML负载的攻击基准，并评估了现有防御机制对前沿AI模型的有效性。作者提出了一种多层次防御策略，结合架构和模型级防护来应对不断演变的攻击。然而，该工作不涉及文档结构理解、视觉文本识别或VLM在文档上的表示学习。
Reason 该论文聚焦于AI浏览器代理中的安全问题，特别是提示注入攻击的防御，属于具身智能/机器人与网络安全的交叉应用。尽管涉及多模态代理行为，但其核心是安全机制设计，并未提出可用于文档图像理解（如OCR去幻觉、复杂排版建模或视觉表示学习）的底层方法。属于典型的‘平行应用’类研究，且领域偏离DIU主线。
Abstract The integration of artificial intelligence (AI) agents into web browsers introduces security challenges that go beyond traditional web application threat models. Prior work has identified prompt injection as a new attack vector for web agents, yet the resulting impact within real-world environments remains insufficiently understood. In this work, we examine the landscape of prompt injection attacks and synthesize a benchmark of attacks embedded in realistic HTML payloads. Our benchmark goes beyond prior work by emphasizing injections that can influence real-world actions rather than mere text outputs, and by presenting attack payloads with complexity and distractor frequency similar to what real-world agents encounter. We leverage this benchmark to conduct a comprehensive empirical evaluation of existing defenses, assessing their effectiveness across a suite of frontier AI models. We propose a multi-layered defense strategy comprising both architectural and model-based defenses to protect against evolving prompt injection attacks. Our work offers a blueprint for designing practical, secure web agents through a defense-in-depth approach.
Authors Kaiyuan Zhang, Mark Tenenholtz, Kyle Polley, Jerry Ma, Denis Yarats, Ninghui Li
Categories Artificial Intelligence
2
MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology NeurIPS 2025
Title CN MTBBench：一种用于肿瘤学中多模态顺序临床决策的基准
Keywords 多模态大模型 · 临床决策 · 基准测试 · 幻觉 · 代理框架
Summary 本文提出了MTBBench，一个用于模拟肿瘤分子委员会（MTB）中多专家协作式临床决策的多模态序列决策基准。该基准关注纵向、多模态的临床推理任务，并通过医生验证确保标注质量。作者评估了多个主流大模型，发现其在时间序列数据推理、跨模态一致性等方面表现不佳，易产生幻觉。为此，他们设计了一个基于基础模型工具的代理框架以提升性能。然而，这本质上是一个垂直领域的应用型研究。
Reason 该论文属于典型的‘平行下游应用’，将现有VLM和Agent技术应用于肿瘤学领域的临床决策场景（Molecular Tumor Boards），并未提出任何文档图像理解（DIU）中的新方法或底层技术创新。尽管提到了幻觉、多模态推理等问题，但其任务设定完全基于特定医学领域的工作流模拟，不涉及OCR、布局分析、表格识别等DIU核心问题，也无法迁移为解决文档理解中密集文本、复杂排版或长文档推理的通用方法。
Abstract Multimodal Large Language Models (LLMs) hold promise for biomedical reasoning, but current benchmarks fail to capture the complexity of real-world clinical workflows. Existing evaluations primarily assess unimodal, decontextualized question-answering, overlooking multi-agent decision-making environments such as Molecular Tumor Boards (MTBs). MTBs bring together diverse experts in oncology, where diagnostic and prognostic tasks require integrating heterogeneous data and evolving insights over time. Current benchmarks lack this longitudinal and multimodal complexity. We introduce MTBBench, an agentic benchmark simulating MTB-style decision-making through clinically challenging, multimodal, and longitudinal oncology questions. Ground truth annotations are validated by clinicians via a co-developed app, ensuring clinical relevance. We benchmark multiple open and closed-source LLMs and show that, even at scale, they lack reliability -- frequently hallucinating, struggling with reasoning from time-resolved data, and failing to reconcile conflicting evidence or different modalities. To address these limitations, MTBBench goes beyond benchmarking by providing an agentic framework with foundation model-based tools that enhance multi-modal and longitudinal reasoning, leading to task-level performance gains of up to 9.0% and 11.2%, respectively. Overall, MTBBench offers a challenging and realistic testbed for advancing multimodal LLM reasoning, reliability, and tool-use with a focus on MTB environments in precision oncology.
Authors Kiril Vasilev, Alexandre Misrahi, Eeshaan Jain, Phil F Cheng, Petros Liakopoulos, Olivier Michielin, Michael Moor, Charlotte Bunne
Comment Accepted to NeurIPS 2025
Categories Artificial Intelligence
2
Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier
Title CN 通过情感理由验证器实现多模态大语言模型的情感一致性推理
Keywords 多模态大语言模型 · 情感识别 · 推理一致性 · 可解释性 · 人机交互
Summary 本文提出情感理由验证器（ERV）和解释奖励机制，用于提升多模态大模型在情感识别任务中生成解释与预测结果之间的一致性。方法无需修改模型架构或额外标注，在MAFW和DFEW两个情感数据集上提升了情感解释的准确性和可信度。研究重点在于人机交互中的情感可解释性，而非文档内容理解。
Reason 该论文聚焦于多模态情感识别与解释生成，属于视频理解场景下的HCI（人机交互）研究，核心任务为情感预测与解释一致性。其方法论围绕情感标签与推理链的一致性建模，并未涉及文档图像中的文本结构、布局、OCR幻觉或细粒度视觉理解等DIU核心问题。尽管‘推理验证’机制在理论上具有通用性，但其迁移至文档理解场景的路径不明确，且论文本身未提供任何与文档、文本识别或排版相关的实验或设计考量。属于无关领域应用，符合负面清单第2条（视频理解/情感分析），直接低分。
Abstract The recent advancement of Multimodal Large Language Models (MLLMs) is transforming human-computer interaction (HCI) from surface-level exchanges into more nuanced and emotionally intelligent communication. To realize this shift, emotion understanding becomes essential allowing systems to capture subtle cues underlying user intent. Furthermore, providing faithful explanations for predicted emotions is crucial to ensure interpretability and build user trust. However, current MLLM-based methods often generate emotion explanations that diverge from the target labels and sometimes even contradict their own predicted emotions. This inconsistency poses a critical risk for misunderstanding and erodes reliability in interactive settings. To address this, we propose a novel approach: the Emotional Rationale Verifier (ERV) and an Explanation Reward. Our method guides the model to produce reasoning that is explicitly consistent with the target emotion during multimodal emotion recognition without modifying the model architecture or requiring additional paired video-description annotations. Our method significantly improves faithful explanation-prediction consistency and explanation emotion accuracy on the MAFW and DFEW datasets. Through extensive experiments and human evaluations, we show that our approach not only enhances alignment between explanation and prediction but also empowers MLLMs to deliver emotionally coherent, trustworthy interactions, marking a key step toward truly human-like HCI systems.
Authors Hyeongseop Rha, Jeong Hun Yeo, Yeonju Kim, Yong Man Ro
Comment 16 pages, 11 figures
Categories Artificial Intelligence
2
Short-Range Oversquashing Learning on Graphs (LoG) 2025
Title CN 短程过压缩
Keywords 图神经网络 · 过压缩 · 消息传递 · 图Transformer · 梯度消失
Summary 本文指出消息传递神经网络（MPNN）中的过压缩问题不仅存在于长距离任务中，在短距离任务中也会出现，并将其归因为瓶颈效应和梯度消失两种机制。实验表明虚拟节点等MPNN改进方法无法解决短程瓶颈，而图Transformer表现更优，因此认为Transformer是更优的解决方案。
Reason 该论文研究图神经网络中的过压缩现象，属于图机器学习的理论分析工作，虽然对MPNN和Transformer的长/短程信息传播有深刻洞察，但其问题设定与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。文中提出的现象和解决方案难以迁移到视觉-语言模型处理高分辨率文档、细粒度定位或OCR幻觉等实际DIU痛点，属于无关领域的基础理论研究。
Abstract Message Passing Neural Networks (MPNNs) are widely used for learning on graphs, but their ability to process long-range information is limited by the phenomenon of oversquashing. This limitation has led some researchers to advocate Graph Transformers as a better alternative, whereas others suggest that it can be mitigated within the MPNN framework, using virtual nodes or other rewiring techniques. In this work, we demonstrate that oversquashing is not limited to long-range tasks, but can also arise in short-range problems. This observation allows us to disentangle two distinct mechanisms underlying oversquashing: (1) the bottleneck phenomenon, which can arise even in low-range settings, and (2) the vanishing gradient phenomenon, which is closely associated with long-range tasks. We further show that the short-range bottleneck effect is not captured by existing explanations for oversquashing, and that adding virtual nodes does not resolve it. In contrast, transformers do succeed in such tasks, positioning them as the more compelling solution to oversquashing, compared to specialized MPNNs.
Authors Yaaqov Mishayev, Yonatan Sverdlov, Tal Amir, Nadav Dym
Comment Accepted to Learning on Graphs (LoG) 2025. Version identical to the camera-ready paper
Categories Artificial Intelligence
2
From Passive Perception to Active Memory: A Weakly Supervised Image Manipulation Localization Framework Driven by Coarse-Grained Annotations AAAI 2026
Title CN 从被动感知到主动记忆：一种由粗粒度标注驱动的弱监督图像篡改定位框架
Keywords 图像篡改定位 · 弱监督学习 · 知识蒸馏 · SAM · 特征融合
Summary 本文提出BoxPromptIML，一种基于粗粒度区域标注的弱监督图像篡改定位框架，通过SAM引导的知识蒸馏和双引导特征融合机制，在降低标注成本的同时实现高精度定位。模型借鉴人类记忆机制，动态融合先验模式与当前观测线索，提升泛化性和鲁棒性。
Reason 该论文研究的是通用图像篡改定位（IML），属于图像取证领域，尽管使用了弱监督和知识蒸馏等技术，但其任务目标与文档图像理解（DIU）的核心问题（如文本识别、布局分析、表格理解、VQA等）无直接关联。虽然‘主动记忆’机制和特征融合设计具有一定新颖性，但其方法迁移至文档理解场景（如OCR去幻觉、复杂排版建模）的路径不明确，且未涉及多模态语言模型或文档特有的结构建模范式。属于无关领域的应用工作。
Abstract Image manipulation localization (IML) faces a fundamental trade-off between minimizing annotation cost and achieving fine-grained localization accuracy. Existing fully-supervised IML methods depend heavily on dense pixel-level mask annotations, which limits scalability to large datasets or real-world deployment.In contrast, the majority of existing weakly-supervised IML approaches are based on image-level labels, which greatly reduce annotation effort but typically lack precise spatial localization. To address this dilemma, we propose BoxPromptIML, a novel weakly-supervised IML framework that effectively balances annotation cost and localization performance. Specifically, we propose a coarse region annotation strategy, which can generate relatively accurate manipulation masks at lower cost. To improve model efficiency and facilitate deployment, we further design an efficient lightweight student model, which learns to perform fine-grained localization through knowledge distillation from a fixed teacher model based on the Segment Anything Model (SAM). Moreover, inspired by the human subconscious memory mechanism, our feature fusion module employs a dual-guidance strategy that actively contextualizes recalled prototypical patterns with real-time observational cues derived from the input. Instead of passive feature extraction, this strategy enables a dynamic process of knowledge recollection, where long-term memory is adapted to the specific context of the current image, significantly enhancing localization accuracy and robustness. Extensive experiments across both in-distribution and out-of-distribution datasets show that BoxPromptIML outperforms or rivals fully-supervised models, while maintaining strong generalization, low annotation cost, and efficient deployment characteristics.
Authors Zhiqing Guo, Dongdong Xi, Songlin Li, Gaobo Yang
Comment Accepted by AAAI 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model
Title CN HoliSafe：面向视觉语言模型的全方位安全基准与建模
Keywords 视觉语言模型 · 安全性 · 基准测试 · 图文对齐 · 可解释性
Summary 本文提出HoliSafe，一个涵盖五种图文安全组合的全方位安全数据集与基准，用于评估和训练VLM在不同图文交互下的安全性。同时设计了一个可插拔的视觉守护模块（VGM），增强VLM对有害图像输入的判断能力，实现可解释的拒绝响应。实验表明该方法在多个安全基准上达到SOTA。
Reason 该论文聚焦于视觉语言模型（VLM）的安全性与对齐问题，属于纯NLP/多模态安全范畴，主要解决有害内容生成和越狱攻击等政治正确与伦理问题。尽管提出了模块化视觉防护模块（VGM），但其目标与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、长文档推理）无直接关联。方法难以迁移到DIU领域用于提升文本识别或结构理解性能，属于‘平行应用’类中的安全对齐方向，不符合关注标准。
Abstract Despite emerging efforts to enhance the safety of Vision-Language Models (VLMs), current approaches face two main shortcomings. 1) Existing safety-tuning datasets and benchmarks only partially consider how image-text interactions can yield harmful content, often overlooking contextually unsafe outcomes from seemingly benign pairs. This narrow coverage leaves VLMs vulnerable to jailbreak attacks in unseen configurations. 2) Prior methods rely primarily on data-centric tuning, with limited architectural innovations to intrinsically strengthen safety. We address these gaps by introducing a holistic safety dataset and benchmark, \textbf{HoliSafe}, that spans all five safe/unsafe image-text combinations, providing a more robust basis for both training and evaluation (HoliSafe-Bench). We further propose a novel modular framework for enhancing VLM safety with a visual guard module (VGM) designed to assess the harmfulness of input images for VLMs. This module endows VLMs with a dual functionality: they not only learn to generate safer responses but can also provide an interpretable harmfulness classification to justify their refusal decisions. A significant advantage of this approach is its modularity; the VGM is designed as a plug-in component, allowing for seamless integration with diverse pre-trained VLMs across various scales. Experiments show that Safe-VLM with VGM, trained on our HoliSafe, achieves state-of-the-art safety performance across multiple VLM benchmarks. Additionally, the HoliSafe-Bench itself reveals critical vulnerabilities in existing VLM models. We hope that HoliSafe and VGM will spur further research into robust and interpretable VLM safety, expanding future avenues for multimodal alignment.
Authors Youngwan Lee, Kangsan Kim, Kwanyong Park, Ilcahe Jung, Soojin Jang, Seanie Lee, Yong-Ju Lee, Sung Ju Hwang
Comment Project page: https://youngwanlee.github.io/holisafe
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
SMoG: Schema Matching on Graph
Title CN SMoG：基于图的模式匹配
Keywords Schema Matching · Knowledge Graph · LLM Augmentation · SPARQL · Data Integration
Summary 本文提出SMoG框架，通过在知识图谱上迭代执行简单的1跳SPARQL查询来实现模式匹配，旨在解决LLM在医疗数据集成中的幻觉和知识滞后问题。相比多跳查询或向量检索，该方法更高效且具备可解释性，在真实医疗数据集上表现接近SOTA。
Reason 该论文聚焦于医疗领域电子健康记录系统的模式匹配任务，核心是结合知识图谱与LLM进行KG-augmented schema matching。尽管涉及LLM和结构化数据，但其方法本质是特定领域的数据集成应用，属于典型的‘平行下游应用’（Wrapper/Application），未提出可用于文档图像理解（如OCR去幻觉、复杂排版建模、视觉-文本对齐等）的通用技术或架构创新。所用的1-hop SPARQL查询机制虽提升效率，但不构成对DIU核心痛点的直接可迁移工具。
Abstract Schema matching is a critical task in data integration, par- ticularly in the medical domain where disparate Electronic Health Record (EHR) systems must be aligned to standard models like OMOP CDM. While Large Language Models (LLMs) have shown promise in schema matching, they suf- fer from hallucination and lack of up-to-date domain knowl- edge. Knowledge Graphs (KGs) offer a solution by pro- viding structured, verifiable knowledge. However, existing KG-augmented LLM approaches often rely on inefficient complex multi-hop queries or storage-intensive vector-based retrieval methods. This paper introduces SMoG (Schema Matching on Graph), a novel framework that leverages iter- ative execution of simple 1-hop SPARQL queries, inspired by successful strategies in Knowledge Graph Question An- swering (KGQA). SMoG enhances explainability and relia- bility by generating human-verifiable query paths while sig- nificantly reducing storage requirements by directly querying SPARQL endpoints. Experimental results on real-world med- ical datasets demonstrate that SMoG achieves performance comparable to state-of-the-art baselines, validating its effec- tiveness and efficiency in KG-augmented schema matching.
Authors Mingyu Jeon, Jaeyoung Suh, Suwan Cho
Categories Artificial Intelligence
2
More with Less: An Empirical Study of Turn-Control Strategies for Efficient Coding Agents
Title CN 更少的交互，更多的成果：关于高效编码智能体中轮次控制策略的实证研究
Keywords 编码智能体 · 轮次控制 · 成本优化 · LLM应用 · 实证研究
Summary 本文研究了在基于大语言模型的编码智能体中如何通过控制交互轮次来提升效率并降低成本。作者比较了无限制轮次、固定轮次限制和一种新的动态轮次扩展策略，在SWE-bench上进行实验。结果显示动态策略能在保持解题率的同时显著降低调用成本。然而，这项工作局限于软件工程场景下的智能体设计，不具备向文档理解任务迁移的方法论价值。
Reason 该论文聚焦于代码生成智能体中的交互轮次控制问题，属于特定下游应用（编程智能体）的优化，未提出可迁移至文档图像理解领域的通用方法。其核心贡献为任务特定的资源调度策略，无法直接解决DIU中的OCR幻觉、复杂排版建模或长文档推理等关键挑战，属于‘平行下游应用’，不符合本筛选标准。
Abstract LLM-powered coding agents, which operate in iterative loops (turns) to solve software engineering tasks, are becoming increasingly powerful. However, their practical deployment is hindered by significant and unpredictable costs. This challenge arises from a combination of factors: quadratically growing token counts with each turn, the high price of models, the large number of turns required for real-world tasks, and the tendency of agents to take inefficient or unnecessary actions. While existing research focuses on optimizing individual turns, the strategic control of the total number of turns remains an underexplored area for managing agent performance and cost. To address this gap, we conduct a comprehensive empirical study on SWE-bench using three state-of-the-art models and evaluate the impact of three distinct turn-control strategies: an unrestricted baseline, a fixed-turn limit with reminders, and a novel dynamic-turn strategy that grants extensions on-demand. Our findings first reveal a fundamental trade-off in the unrestricted setting, where no single model excels across performance, cost, and turn efficiency. We then show that a fixed-turn limit, specifically at the 75th percentile of the baseline, serves as a "sweet spot", substantially reducing costs (by 24%-68%) with minimal impact on solve rates. Most significantly, the dynamic-turn strategy consistently outperforms fixed-limit approaches, achieving comparable or better solve rates while further reducing costs by an additional 12%-24% by intelligently allocating resources only to tasks that need them. This work provides the first systematic analysis of turn-control strategies, offering simple yet effective guidelines for developers to balance cost and efficacy. We demonstrate that dynamic resource allocation is a superior, easy-to-implement approach for deploying powerful yet economically viable coding agents.
Authors Pengfei Gao, Chao Peng
Categories Artificial Intelligence
2
Multi-Modal Data Exploration via Language Agents IJCNLP AACL 2025 Findings
Title CN 通过语言智能体进行多模态数据探索
Keywords 多模态数据探索 · 语言智能体 · text-to-SQL · LLM Agent · 数据库查询
Summary 本文提出了M²EX系统，旨在通过基于大语言模型的智能体实现对包含关系型数据、文本和图像的多模态数据系统的自然语言查询。系统能自动分解问题、生成SQL语句、调用图像分析模块，并优化执行计划以提升效率与准确性。实验表明其在准确率、延迟和API成本方面优于现有方法。
Reason 该论文提出一个名为M²EX的系统，利用LLM驱动的Agent框架将自然语言问题分解为子任务（如text-to-SQL、图像分析），并协调不同模态的专家模型执行查询。尽管其使用了Agent架构处理多模态数据，但核心是面向企业数据库、文本和图像的通用多模态检索系统，属于典型的‘平行应用’：即在特定场景下集成现有技术（LLM+SQL+图像分析）解决下游业务问题，未提出任何针对文档图像理解（DIU）的核心方法创新，如视觉表示学习、细粒度定位、OCR-free建模或排版结构解析等。其Agent调度机制也非专为长文档阅读或复杂版式推理设计，难以迁移至DocVQA中的幻觉抑制、密集文本识别等关键痛点。
Abstract International enterprises, organizations, and hospitals collect large amounts of multi-modal data stored in databases, text documents, images, and videos. While there has been recent progress in the separate fields of multi-modal data exploration as well as in database systems that automatically translate natural language questions to database query languages, the research challenge of querying both structured databases and unstructured modalities (e.g., texts, images) in natural language remains largely unexplored. In this paper, we propose M$^2$EX -a system that enables multi-modal data exploration via language agents. Our approach is based on the following research contributions: (1) Our system is inspired by a real-world use case that enables users to explore multi-modal information systems. (2) M$^2$EX leverages an LLM-based agentic AI framework to decompose a natural language question into subtasks such as text-to-SQL generation and image analysis and to orchestrate modality-specific experts in an efficient query plan. (3) Experimental results on multi-modal datasets, encompassing relational data, text, and images, demonstrate that our system outperforms state-of-the-art multi-modal exploration systems, excelling in both accuracy and various performance metrics, including query latency, API costs, and planning efficiency, thanks to the more effective utilization of the reasoning capabilities of LLMs.
Authors Farhad Nooralahzadeh, Yi Zhang, Jonathan Furst, Kurt Stockinger
Comment Accepted to the IJCNLP AACL 2025 Findings
Categories Artificial Intelligence, Computation and Language
2
Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning
Title CN 面向选择性多模态大语言模型遗忘的良性记忆遗忘
Keywords 多模态大语言模型 · 知识遗忘 · 隐私保护 · 记忆隔离 · 模型安全
Summary 本文提出了一种名为SMFA的方法，用于在多模态大语言模型中实现选择性知识遗忘，即移除敏感信息的同时保护模型的通用视觉理解能力，并构建了首个评估基准S-MLLMUn Bench。实验表明该方法能实现精准可控的遗忘。
Reason 该论文研究的是多模态大模型中的知识遗忘问题，核心目标是删除敏感信息的同时保留通用视觉理解能力。虽然技术上涉及MLLM，但其任务设定属于模型安全与隐私范畴（如去记忆化、数据删除），并未针对文档图像理解中的OCR幻觉、复杂排版、细粒度定位或长文档推理等核心痛点提出解决方案。且该工作属于典型的‘平行应用’类研究——将已有unlearning方法迁移到多模态场景，但未对DIU的方法论产生可迁移的技术推动。不属于DIU本题，也不提供可用于解决文档理解瓶颈的新工具。
Abstract Multimodal Large Language Models (MLLMs) achieve remarkable capabilities but can inadvertently memorize privacy-sensitive information. Although existing unlearning methods can remove such knowledge, they fail to achieve benign forgetting because they often degrade the model's general image understanding performance. To address this, we propose the Sculpted Memory Forgetting Adapter (SMFA), which confines forgetting to targeted memory regions while preserving overall capabilities. SMFA first fine-tunes the model to replace sensitive responses with refusals, yielding a memory forgetting adapter, and then applies a retaining anchor-guided masking mechanism to prevent interference with unrelated knowledge and understanding ability. To systematically evaluate selective MLLM unlearning, we introduce S-MLLMUn Bench, the first benchmark designed to jointly assess the removal of sensitive knowledge and retention of general visual understanding. Extensive experiments show that, unlike prior methods, SMFA achieves precise and controllable unlearning while maintaining the model's foundational image understanding.
Authors Zhen Zeng, Leijiang Gu, Zhangling Duan, Feng Li, Zenglin Shi, Cees G. M. Snoek, Meng Wang
Categories Artificial Intelligence
2
Explainable Visual Anomaly Detection via Concept Bottleneck Models
Title CN 基于概念瓶颈模型的可解释视觉异常检测
Keywords 视觉异常检测 · 概念瓶颈模型 · 可解释性 · 无监督学习 · 概念学习
Summary 本文提出了一种基于概念瓶颈模型（CBM）的视觉异常检测方法CONVAD，旨在通过学习可解释的概念来提供语义级别的异常解释。作者构建了用于VAD的概念数据集，改进了CBM架构以同时生成概念和视觉解释，并设计了合成异常样本的流程。尽管提升了模型可解释性，但其应用场景为通用图像异常检测，与文档图像理解的核心任务关联较弱。
Reason 该论文研究的是通用视觉异常检测（VAD）中的可解释性问题，其核心任务是通过概念瓶颈模型提供语义层面的异常解释。虽然‘异常检测’在文档领域有潜在应用（如伪造文档识别），但本文并未针对文档结构、文本内容或排版特性进行建模，也未涉及OCR、多模态推理或VLM相关技术。属于将CBM应用于VAD的垂直应用工作，而非方法论上的源头创新，且不解决文档理解中的核心痛点（如幻觉、密集文本、复杂布局等）。因此属于无关领域的平行应用。
Abstract In recent years, Visual Anomaly Detection (VAD) has gained significant attention due to its ability to identify anomalous images using only normal images during training. Many VAD models work without supervision but are still able to provide visual explanations by highlighting the anomalous regions within an image. However, although these visual explanations can be helpful, they lack a direct and semantically meaningful interpretation for users. To address this limitation, we propose extending Concept Bottleneck Models (CBMs) to the VAD setting. By learning meaningful concepts, the network can provide human-interpretable descriptions of anomalies, offering a novel and more insightful way to explain them. Our contributions are threefold: (i) we develop a Concept Dataset to support research on CBMs for VAD; (ii) we improve the CBM architecture to generate both concept-based and visual explanations, bridging semantic and localization interpretability; and (iii) we introduce a pipeline for synthesizing artificial anomalies, preserving the VAD paradigm of minimizing dependence on rare anomalous samples. Our approach, Concept-Aware Visual Anomaly Detection (CONVAD), achieves performance comparable to classic VAD methods while providing richer, concept-driven explanations that enhance interpretability and trust in VAD systems.
Authors Arianna Stropeni, Valentina Zaccaria, Francesco Borsatti, Davide Dalle Pezze, Manuel Barusco, Gian Antonio Susto
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
RLZero: Direct Policy Inference from Language Without In-Domain Supervision NeurIPS 2025
Title CN RLZero：无需领域内监督的纯语言驱动策略推断
Keywords 强化学习 · 零样本学习 · 语言到行为 · 视频生成 · 无监督RL
Summary 本文提出RLZero，一种无需领域内监督即可从自然语言指令中直接推断策略的方法，适用于具身智能体在未见过的任务中零样本生成行为。通过想象观察序列、投影到目标环境并模仿预训练策略实现跨任务迁移。实验涵盖多种机器人控制任务及YouTube视频驱动的人形机器人动作生成。
Reason 该论文属于具身智能与强化学习领域的语言条件策略生成工作，核心目标是通过自然语言指令直接生成机器人或仿真环境中的行为策略，不涉及任何文档、文本或视觉-语言理解任务。尽管其‘想象-投影-模仿’框架在概念上有一定新颖性，但其方法依赖于视频生成模型和环境交互数据，迁移到文档图像理解（DIU）场景中几乎不可行。文档理解的核心痛点如OCR幻觉、细粒度定位、复杂排版建模等无法从中受益。属于无关领域应用，符合负面清单第2条。
Abstract The reward hypothesis states that all goals and purposes can be understood as the maximization of a received scalar reward signal. However, in practice, defining such a reward signal is notoriously difficult, as humans are often unable to predict the optimal behavior corresponding to a reward function. Natural language offers an intuitive alternative for instructing reinforcement learning (RL) agents, yet previous language-conditioned approaches either require costly supervision or test-time training given a language instruction. In this work, we present a new approach that uses a pretrained RL agent trained using only unlabeled, offline interactions--without task-specific supervision or labeled trajectories--to get zero-shot test-time policy inference from arbitrary natural language instructions. We introduce a framework comprising three steps: imagine, project, and imitate. First, the agent imagines a sequence of observations corresponding to the provided language description using video generative models. Next, these imagined observations are projected into the target environment domain. Finally, an agent pretrained in the target environment with unsupervised RL instantly imitates the projected observation sequence through a closed-form solution. To the best of our knowledge, our method, RLZero, is the first approach to show direct language-to-behavior generation abilities on a variety of tasks and environments without any in-domain supervision. We further show that components of RLZero can be used to generate policies zero-shot from cross-embodied videos, such as those available on YouTube, even for complex embodiments like humanoids.
Authors Harshit Sikchi, Siddhant Agarwal, Pranaya Jajoo, Samyak Parajuli, Caleb Chuck, Max Rudolph, Peter Stone, Amy Zhang, Scott Niekum
Comment NeurIPS 2025, 26 pages
Categories Artificial Intelligence
2
Improved LLM Agents for Financial Document Question Answering
Title CN 用于金融文档问答的改进型大语言模型代理
Keywords LLM代理 · 金融文档 · 自我修正 · 数值问答 · 下游应用
Summary 本文研究了在无oracle标签情况下，传统批评代理（critic agent）在金融文档问答中的性能下降问题，并提出了一种改进的批评代理与计算器代理协同框架，以提升数值类问题的回答准确率和安全性。实验表明其优于程序化思维（program-of-thought）方法。
Reason 该论文聚焦于金融文档问答这一特定应用场景，本质上是将LLM代理（如critic agent和calculator agent）应用于金融报表类数据，属于典型的‘平行下游应用’。尽管提出了代理间的交互机制，但其方法论创新局限于任务特定的工程优化，并未提出可泛化的文档理解架构、视觉表示学习或针对OCR幻觉、复杂版式等DIU核心痛点的底层技术突破。无法为文档图像理解提供通用方法论价值。
Abstract Large language models (LLMs) have shown impressive capabilities on numerous natural language processing tasks. However, LLMs still struggle with numerical question answering for financial documents that include tabular and textual data. Recent works have showed the effectiveness of critic agents (i.e., self-correction) for this task given oracle labels. Building upon this framework, this paper examines the effectiveness of the traditional critic agent when oracle labels are not available, and show, through experiments, that this critic agent's performance deteriorates in this scenario. With this in mind, we present an improved critic agent, along with the calculator agent which outperforms the previous state-of-the-art approach (program-of-thought) and is safer. Furthermore, we investigate how our agents interact with each other, and how this interaction affects their performance.
Authors Nelvin Tan, Zian Seng, Liang Zhang, Yu-Ching Shih, Dong Yang, Amol Salunkhe
Comment 13 pages, 5 figures. Unlike the previous version, LLM names are now unmasked
Categories Artificial Intelligence, Computation and Language
2
Semantic-KG: Using Knowledge Graphs to Construct Benchmarks for Measuring Semantic Similarity
Title CN Semantic-KG：利用知识图谱构建衡量语义相似性的基准
Keywords 语义相似性 · 知识图谱 · 评测基准 · LLM评估 · 自然语言生成
Summary 本文提出一种基于知识图谱自动生成语义相似性评测基准的方法，用于评估LLM生成文本的质量。作者在多个领域构建了包含不同类型语义差异的文本对数据集，并比较了多种语义相似性度量方法的表现。结果显示现有方法性能受语义变异类型和领域影响显著，且没有一种方法始终最优。
Reason 该论文聚焦于生成用于评估LLM文本输出的语义相似性基准，核心贡献在于通过知识图谱自动生成语义相似/不相似的文本对，并构建跨领域的评测集。尽管其方法在NLP评估领域有一定价值，但属于下游评测任务的工具性工作，未涉及视觉或多模态输入，与文档图像理解（DIU）中的OCR幻觉、布局分析、细粒度定位等核心问题无直接关联。其方法难以迁移至DIU场景，因DIU的关键挑战在于视觉-语言对齐和结构化信息提取，而非纯文本语义相似性判断。
Abstract Evaluating the open-form textual responses generated by Large Language Models (LLMs) typically requires measuring the semantic similarity of the response to a (human generated) reference. However, there is evidence that current semantic similarity methods may capture syntactic or lexical forms over semantic content. While benchmarks exist for semantic equivalence, they often suffer from high generation costs due to reliance on subjective human judgment, limited availability for domain-specific applications, and unclear definitions of equivalence. This paper introduces a novel method for generating benchmarks to evaluate semantic similarity methods for LLM outputs, specifically addressing these limitations. Our approach leverages knowledge graphs (KGs) to generate pairs of natural-language statements that are semantically similar or dissimilar, with dissimilar pairs categorized into one of four sub-types. We generate benchmark datasets in four different domains (general knowledge, biomedicine, finance, biology), and conduct a comparative study of semantic similarity methods including traditional natural language processing scores and LLM-as-a-judge predictions. We observe that the sub-type of semantic variation, as well as the domain of the benchmark impact the performance of semantic similarity methods, with no method being consistently superior. Our results present important implications for the use of LLM-as-a-judge in detecting the semantic content of text. Code is available at https://github.com/QiyaoWei/semantic-kg and the dataset is available at https://huggingface.co/datasets/QiyaoWei/Semantic-KG.
Authors Qiyao Wei, Edward Morrell, Lea Goetz, Mihaela van der Schaar
Categories Artificial Intelligence
2
HyperbolicRAG: Enhancing Retrieval-Augmented Generation with Hyperbolic Representations
Title CN HyperbolicRAG：通过双曲表示增强检索增强生成
Keywords RAG · 双曲表示 · 图结构推理 · 层次化知识 · 去幻觉
Summary 本文提出HyperbolicRAG，一种结合双曲几何的检索增强生成框架，利用Poincaré流形嵌入来同时捕捉语义相似性和层级包含关系，并设计了深度感知表示学习、无监督对比正则化和互排名融合机制。实验表明其在多个QA基准上优于现有RAG方法。
Reason 该论文属于典型的‘平行下游应用’，其核心是将双曲空间表示引入图增强的RAG框架中，用于提升通用领域问答中的层次化知识建模。尽管其方法可能缓解LLM幻觉，但它是面向一般性知识检索与生成的应用架构，并未针对文档图像理解（DIU）中的OCR错误、复杂版式、细粒度定位或高分辨率视觉输入等核心痛点提出解决方案。所提技术难以直接迁移至文档视觉推理或多模态对齐任务，且不涉及视觉表示学习或VLM架构改进。
Abstract Retrieval-augmented generation (RAG) enables large language models (LLMs) to access external knowledge, helping mitigate hallucinations and enhance domain-specific expertise. Graph-based RAG enhances structural reasoning by introducing explicit relational organization that enables information propagation across semantically connected text units. However, these methods typically rely on Euclidean embeddings that capture semantic similarity but lack a geometric notion of hierarchical depth, limiting their ability to represent abstraction relationships inherent in complex knowledge graphs. To capture both fine-grained semantics and global hierarchy, we propose HyperbolicRAG, a retrieval framework that integrates hyperbolic geometry into graph-based RAG. HyperbolicRAG introduces three key designs: (1) a depth-aware representation learner that embeds nodes within a shared Poincare manifold to align semantic similarity with hierarchical containment, (2) an unsupervised contrastive regularization that enforces geometric consistency across abstraction levels, and (3) a mutual-ranking fusion mechanism that jointly exploits retrieval signals from Euclidean and hyperbolic spaces, emphasizing cross-space agreement during inference. Extensive experiments across multiple QA benchmarks demonstrate that HyperbolicRAG outperforms competitive baselines, including both standard RAG and graph-augmented baselines.
Authors Linxiao Cao, Ruitao Wang, Jindong Li, Zhipeng Zhou, Menglin Yang
Comment 12 pages
Categories Artificial Intelligence, Computation and Language
2
MAPS: Preserving Vision-Language Representations via Module-Wise Proximity Scheduling for Better Vision-Language-Action Generalization
Title CN MAPS：通过模块化接近度调度保持视觉-语言表征以实现更好的视觉-语言-动作泛化
Keywords VLA · 微调稳定性 · 模块化正则化 · 具身智能 · 表征保留
Summary 本文提出MAPS，一种用于视觉-语言-动作（VLA）模型的模块化接近度调度微调框架，通过分析不同模块对预训练表示的依赖程度，线性调度各模块与原始参数的距离约束，在保持泛化能力的同时提升任务性能。实验表明其在多个VLA基准和真实机器人平台上显著提升性能。
Reason 该论文研究的是视觉-语言-动作（VLA）模型在机器人控制等具身智能任务中的微调稳定性问题，属于具身智能/机器人领域。其核心贡献是设计一种模块化的正则化调度策略来平衡预训练表征保留与下游任务适应，但应用场景和目标均远离文档图像理解（DIU）的核心任务如DocVQA、布局分析或表格识别。尽管涉及VLM的微调稳定性，但方法针对的是动作生成序列，无法直接迁移解决OCR幻觉、密集文本识别或复杂排版理解等DIU痛点。
Abstract Vision-Language-Action (VLA) models inherit strong priors from pretrained Vision-Language Models (VLMs), but naive fine-tuning often disrupts these representations and harms generalization. Existing fixes -- freezing modules or applying uniform regularization -- either overconstrain adaptation or ignore the differing roles of VLA components. We present MAPS (Module-Wise Proximity Scheduling), the first robust fine-tuning framework for VLAs. Through systematic analysis, we uncover an empirical order in which proximity constraints should be relaxed to balance stability and flexibility. MAPS linearly schedules this relaxation, enabling visual encoders to stay close to their pretrained priors while action-oriented language layers adapt more freely. MAPS introduces no additional parameters or data, and can be seamlessly integrated into existing VLAs. Across MiniVLA-VQ, MiniVLA-OFT, OpenVLA-OFT, and challenging benchmarks such as SimplerEnv, CALVIN, LIBERO, as well as real-world evaluations on the Franka Emika Panda platform, MAPS consistently boosts both in-distribution and out-of-distribution performance (up to +30%). Our findings highlight empirically guided proximity to pretrained VLMs as a simple yet powerful principle for preserving broad generalization in VLM-to-VLA transfer.
Authors Chengyue Huang, Mellon M. Zhang, Robert Azarcon, Glen Chou, Zsolt Kira
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
2
Unlearning as Ablation: Toward a Falsifiable Benchmark for Generative Scientific Discovery NeurIPS 2025 AI4Science Workshop
Title CN 作为消融的遗忘：迈向生成式科学发现的可证伪基准
Keywords AI for Science · 知识生成 · 遗忘 · 消融实验 · 可证伪性
Summary 本文提出了一种名为‘作为消融的遗忘’的方法，用于检验大语言模型在科学发现中是真正生成新知识还是仅仅回忆已学内容。通过系统性地删除目标结果及其相关知识闭包，观察模型能否从基础公理重新推导出该结果。这是一种概念性和方法论性的框架，主要面向数学、算法等领域的AI for Science研究，并非针对文档图像理解或多模态任务。
Reason 该论文属于AI for Science的方法论探讨，聚焦于通过‘遗忘-消融’测试检验LLM是否具备真正生成新知识的能力。其核心任务与文档图像理解（DIU）无直接关联，既不涉及多模态输入、视觉表示学习、OCR优化，也不解决DocVQA中的幻觉、复杂排版或细粒度定位等关键问题。尽管‘知识重推导’的思想在理论上可能对长文档推理中防止幻觉有一定启发，但该工作未提供任何可迁移至DIU的技术工具（如架构改进、推理机制、视觉压缩等），且完全基于纯文本逻辑推理场景设计，无法应对文档理解中的高分辨率视觉输入和多模态对齐挑战。
Abstract Bold claims about AI's role in science-from "AGI will cure all diseases" to promises of radically accelerated discovery-raise a central epistemic question: do large language models (LLMs) truly generate new knowledge, or do they merely remix memorized fragments? We propose unlearning-as-ablation as a falsifiable probe of constructive scientific discovery. The idea is to systematically remove a target result together with its forget-closure (supporting lemmas, paraphrases, and multi-hop entailments) and then evaluate whether the model can re-derive the result from only permitted axioms and tools. Success would indicate generative capability beyond recall; failure would expose current limits. Unlike prevailing motivations for unlearning-privacy, copyright, or safety-our framing repositions it as an epistemic probe for AI-for-Science. We outline a minimal pilot in mathematics and algorithms to illustrate feasibility, and sketch how the same approach could later be extended to domains such as physics or chemistry. This is a position paper: our contribution is conceptual and methodological, not empirical. We aim to stimulate discussion on how principled ablation tests could help distinguish models that reconstruct knowledge from those that merely retrieve it, and how such probes might guide the next generation of AI-for-Science benchmarks.
Authors Robert Yang
Comment 6 pages + appendix. Accepted to NeurIPS 2025 AI4Science Workshop
Categories Artificial Intelligence
2
Beyond Relational: Semantic-Aware Multi-Modal Analytics with LLM-Native Query Optimization
Title CN 超越关系型：基于LLM原生查询优化的语义感知多模态分析
Keywords 多模态分析 · LLM查询优化 · 语义算子 · 查询计划搜索 · 计算复用
Summary 本文提出了Nirvana，一种结合程序化语义算子和逻辑/物理查询优化策略的多模态分析框架，支持由LLM驱动的语义级查询处理。通过基于自然语言规则的代理式逻辑优化器和基于改进分数的成本感知物理优化器，实现高效且可扩展的多模态分析。实验表明其在端到端运行时间和系统成本上显著优于现有系统。
Reason 该论文提出一个面向多模态数据分析的框架Nirvana，核心贡献在于利用LLM进行语义感知的查询优化和执行计划搜索，属于典型的‘平行下游应用’。其技术本质是将LLM用于数据库查询优化流程，并未涉及文档图像理解中的关键问题如OCR去幻觉、细粒度视觉定位、复杂版面解析或高分辨率视觉输入处理。尽管其‘agentic逻辑优化’和‘物理执行选择’机制在数据库领域有创新性，但这些方法针对的是结构化数据与自然语言查询之间的语义映射，难以迁移至文档图像中视觉-文本对齐、布局建模、密集文本识别等核心挑战。
Abstract Multi-modal analytical processing has the potential to transform applications in e-commerce, healthcare, entertainment, and beyond. However, real-world adoption remains elusive due to the limited ability of traditional relational query operators to capture query semantics. The emergence of foundation models, particularly the large language models (LLMs), opens up new opportunities to develop flexible, semantic-aware data analytics systems that transcend the relational paradigm. We present Nirvana, a multi-modal data analytics framework that incorporates programmable semantic operators while leveraging both logical and physical query optimization strategies, tailored for LLM-driven semantic query processing. Nirvana addresses two key challenges. First, it features an agentic logical optimizer that uses natural language-specified transformation rules and random-walk-based search to explore vast spaces of semantically equivalent query plans -- far beyond the capabilities of conventional optimizers. Second, it introduces a cost-aware physical optimizer that selects the most effective LLM backend for each operator using a novel improvement-score metric. To further enhance efficiency, Nirvana incorporates computation reuse and evaluation pushdown techniques guided by model capability hypotheses. Experimental evaluations on three real-world benchmarks demonstrate that Nirvana is able to reduce end-to-end runtime by 10%--85% and reduces system processing costs by 76% on average, outperforming state-of-the-art systems at both efficiency and scalability.
Authors Junhao Zhu, Lu Chen, Xiangyu Ke, Ziquan Fang, Tianyi Li, Yunjun Gao, Christian S. Jensen
Categories Artificial Intelligence
2
Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts
Title CN 弥合收敛学习中的关键差距：表征对齐如何跨层、训练和分布偏移演化
Keywords 表征对齐 · 收敛学习 · 分布偏移 · 神经网络比较 · Procrustes分析
Summary 本文通过大规模实验研究了多种视觉模型之间的表征对齐现象，比较了线性、正交和排列匹配等不同对齐方法的效果，发现深层表示的对齐主要由输入统计和架构先验决定，并在首个训练周期内快速形成。面对分布偏移时，浅层保持对齐而深层出现分化。
Reason 该论文研究的是神经网络在不同训练过程中内部表示的对齐问题，属于基础表示学习与理论分析范畴。尽管其关于跨层、训练动态和分布外泛化下表示对齐的发现具有理论价值，但并未针对文档图像理解（DIU）任务设计，也未提出可直接迁移用于解决OCR幻觉、复杂排版建模或细粒度视觉定位等DIU核心痛点的方法。其方法论聚焦于模型间表示空间的相似性度量（如Procrustes对齐），属于通用深度学习理论探索，难以直接赋能VLM在文档场景下的结构化解析能力。
Abstract Understanding convergent learning -- the degree to which independently trained neural systems -- whether multiple artificial networks or brains and models -- arrive at similar internal representations -- is crucial for both neuroscience and AI. Yet, the literature remains narrow in scope -- typically examining just a handful of models with one dataset, relying on one alignment metric, and evaluating networks at a single post-training checkpoint. We present a large-scale audit of convergent learning, spanning dozens of vision models and thousands of layer-pair comparisons, to close these long-standing gaps. First, we pit three alignment families against one another -- linear regression (affine-invariant), orthogonal Procrustes (rotation-/reflection-invariant), and permutation/soft-matching (unit-order-invariant). We find that orthogonal transformations align representations nearly as effectively as more flexible linear ones, and although permutation scores are lower, they significantly exceed chance, indicating a privileged representational basis. Tracking convergence throughout training further shows that nearly all eventual alignment crystallizes within the first epoch -- well before accuracy plateaus -- indicating it is largely driven by shared input statistics and architectural biases, not by the final task solution. Finally, when models are challenged with a battery of out-of-distribution images, early layers remain tightly aligned, whereas deeper layers diverge in proportion to the distribution shift. These findings fill critical gaps in our understanding of representational convergence, with implications for neuroscience and AI.
Authors Chaitanya Kapoor, Sudhanshu Srivastava, Meenakshi Khosla
Categories Artificial Intelligence
2
Selecting Belief-State Approximations in Simulators with Latent States
Title CN 在具有潜在状态的模拟器中选择信念状态近似
Keywords 信念状态 · 状态重置 · 模拟器 · 潜在状态 · 分布选择
Summary 本文探讨在含有潜在变量的模拟器中如何选择合适的信念状态近似方法，使得能够通过观测历史有效恢复潜在状态分布。作者提出了两种形式化方案：基于潜在状态的选择和基于观测的选择，并分析了它们在不同 rollout 策略下的理论保证。结果显示，观测基础的选择在常用单次重置策略下可能失效，但在重复重置策略下具备良好性质。研究揭示了该问题在算法设计与理论分析上的复杂性。
Reason 该论文研究的是强化学习或模拟系统中的信念状态（belief state）近似与选择问题，属于通用AI理论框架下的方法论工作，主要面向动态系统建模与状态推断。尽管其关于‘从观测历史中推断潜在状态分布’的思想在理论上可能对多模态推理有启发，但文中未涉及视觉输入、文档结构、文本理解或任何DIU相关任务。无法直接迁移用于解决OCR幻觉、复杂排版解析或长文档推理等核心痛点，属于无关领域的理论探索。
Abstract State resetting is a fundamental but often overlooked capability of simulators. It supports sample-based planning by allowing resets to previously encountered simulation states, and enables calibration of simulators using real data by resetting to states observed in real-system traces. While often taken for granted, state resetting in complex simulators can be nontrivial: when the simulator comes with latent variables (states), state resetting requires sampling from the posterior over the latent state given the observable history, a.k.a. the belief state (Silver and Veness, 2010). While exact sampling is often infeasible, many approximate belief-state samplers can be constructed, raising the question of how to select among them using only sampling access to the simulator. In this paper, we show that this problem reduces to a general conditional distribution-selection task and develop a new algorithm and analysis under sampling-only access. Building on this reduction, the belief-state selection problem admits two different formulations: latent state-based selection, which directly targets the conditional distribution of the latent state, and observation-based selection, which targets the induced distribution over the observation. Interestingly, these formulations differ in how their guarantees interact with the downstream roll-out methods: perhaps surprisingly, observation-based selection may fail under the most natural roll-out method (which we call Single-Reset) but enjoys guarantees under the less conventional alternative (which we call Repeated-Reset). Together with discussion on issues such as distribution shift and the choice of sampling policies, our paper reveals a rich landscape of algorithmic choices, theoretical nuances, and open questions, in this seemingly simple problem.
Authors Nan Jiang
Categories Artificial Intelligence
2
Revisiting KRISP: A Lightweight Reproduction and Analysis of Knowledge-Enhanced Vision-Language Models
Title CN 重访KRISP：一种轻量级复现与知识增强型视觉语言模型的分析
Keywords 视觉语言模型 · 知识增强 · 轻量化模型 · VQA · 幻觉抑制
Summary 本文对Facebook提出的知识增强视觉语言模型KRISP进行了轻量级复现，旨在降低其计算开销并提升在资源受限设备上的可用性。作者通过系统性的消融实验揭示了原模型在实际部署中的设计缺陷与隐性问题。复现模型参数量显著减少，可在智能手机和AR/VR设备上运行，且通过限制外部知识图谱范围来抑制AI幻觉现象。
Reason 该论文属于典型的‘平行下游应用’类型，其核心是对已有VQA模型KRISP的轻量化复现与分析，并未提出适用于文档图像理解（DIU）的新架构、新训练范式或可迁移的方法论。尽管提到了减少幻觉和在受限知识域内生成输出，但这是针对通用VQA任务的设计，且方法依赖于外部知识图谱约束输出空间，难以直接迁移到文档理解中的OCR幻觉、复杂排版解析等核心问题。研究场景聚焦于DAQUAR数据集（室内场景问答），与文档无关。
Abstract Facebook AI Research introduced KRISP [4], which integrates structured external knowledge into pipelines for vision-language reasoning. Despite its effectiveness, the original model has been developed for industrial-scale training, is computationally demanding, and is tightly connected to a large backbone. In this work, we reexamine KRISP from a different angle and offer a lightweight reproduction with significantly fewer parameters. Even though our replicated model performs about 75 % of the original, the replication process uncovers a number of design flaws, real-world pitfalls, and implicit problems that were not fully covered in the original paper. We offer insights into the scalability and efficacy of knowledge-enhanced VQA architectures under resource constraints through systematic ablation studies, which include a proof-of-concept on synthetic VQA data and evaluation on the DAQUAR dataset. Our model, configured with a low parameter setup and constrained by the external Knowledge graph domain, prevents AI hallucinations and generates outputs solely within that domain. Minimal parameters allow us to function on edge devices like smartphones and AR-VR, further improving offline visual reasoning.
Authors Souradeep Dutta, Keshav Bulia, Neena S Nair
Comment 7 pages , 4 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design
Title CN CANVAS：面向基于工具的用户界面设计的视觉语言模型基准
Keywords 视觉语言模型 · 基准测试 · 工具调用 · 用户界面设计 · 人机协作
Summary 本文提出了CANVAS，一个用于评估视觉语言模型在基于工具的用户界面设计中表现的基准。包含598个任务，源自3300个移动UI设计，涵盖复制整个界面和修改局部两种任务类型。模型通过上下文感知的方式调用设计工具逐步编辑UI。研究发现当前主流模型已具备一定策略性工具调用能力，但也存在典型错误模式。
Reason 该论文属于典型的‘平行应用’类工作，构建了一个针对UI设计软件操作的VLM基准，任务聚焦于调用工具修改界面元素（如创建按钮背景）。尽管涉及多模态模型与交互式环境的结合，但其核心是将现有VLM能力应用于特定下游场景（UI设计自动化），并未提出可迁移至文档图像理解（DIU）领域的底层方法。例如，未涉及OCR幻觉缓解、密集文本处理、复杂版面解析或长文档推理等DIU关键问题。所提出的benchmark也局限于工具调用行为评估，对文档理解中的逻辑结构建模或视觉token压缩等技术无直接启发。
Abstract User interface (UI) design is an iterative process in which designers progressively refine their work with design software such as Figma or Sketch. Recent advances in vision language models (VLMs) with tool invocation suggest these models can operate design software to edit a UI design through iteration. Understanding and enhancing this capacity is important, as it highlights VLMs' potential to collaborate with designers within conventional software. However, as no existing benchmark evaluates tool-based design performance, the capacity remains unknown. To address this, we introduce CANVAS, a benchmark for VLMs on tool-based user interface design. Our benchmark contains 598 tool-based design tasks paired with ground-truth references sampled from 3.3K mobile UI designs across 30 function-based categories (e.g., onboarding, messaging). In each task, a VLM updates the design step-by-step through context-based tool invocations (e.g., create a rectangle as a button background), linked to design software. Specifically, CANVAS incorporates two task types: (i) design replication evaluates the ability to reproduce a whole UI screen; (ii) design modification evaluates the ability to modify a specific part of an existing screen. Results suggest that leading models exhibit more strategic tool invocations, improving design quality. Furthermore, we identify common error patterns models exhibit, guiding future work in enhancing tool-based design capabilities.
Authors Daeheon Jeong, Seoyeon Byun, Kihoon Son, Dae Hyun Kim, Juho Kim
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
2
RubricRL: Simple Generalizable Rewards for Text-to-Image Generation
Title CN RubricRL：用于文生图生成的简单可泛化奖励机制
Keywords 文本到图像生成 · 强化学习 · 奖励设计 · 多模态评判 · OCR保真度
Summary 本文提出RubricRL，一种基于评分标准的强化学习奖励框架，用于文生图模型的对齐优化。通过多模态评判器对生成图像在对象正确性、属性准确性和OCR保真度等细粒度维度上进行可解释的打分，并动态加权构建结构化奖励信号。虽然提到了OCR fidelity，但其目的是提升生成图像的文字合理性，而非解决文档理解中的信息提取错误或幻觉问题。
Reason 该论文聚焦于文本到图像生成中的强化学习奖励设计，属于纯图像生成领域。尽管其使用了OCR fidelity作为评估维度之一，但核心目标是提升生成图像与文本描述的一致性，并非解决文档理解中的OCR幻觉、细粒度识别或复杂排版解析等根本问题。方法本质上是为生成任务服务的奖励建模框架，难以直接迁移至文档图像理解（如DocVQA、表格识别）的任务中。属于‘无关领域’中的文生图方向，不符合DIU方法论创新的要求。
Abstract Reinforcement learning (RL) has recently emerged as a promising approach for aligning text-to-image generative models with human preferences. A key challenge, however, lies in designing effective and interpretable rewards. Existing methods often rely on either composite metrics (e.g., CLIP, OCR, and realism scores) with fixed weights or a single scalar reward distilled from human preference models, which can limit interpretability and flexibility. We propose RubricRL, a simple and general framework for rubric-based reward design that offers greater interpretability, composability, and user control. Instead of using a black-box scalar signal, RubricRL dynamically constructs a structured rubric for each prompt--a decomposable checklist of fine-grained visual criteria such as object correctness, attribute accuracy, OCR fidelity, and realism--tailored to the input text. Each criterion is independently evaluated by a multimodal judge (e.g., o4-mini), and a prompt-adaptive weighting mechanism emphasizes the most relevant dimensions. This design not only produces interpretable and modular supervision signals for policy optimization (e.g., GRPO or PPO), but also enables users to directly adjust which aspects to reward or penalize. Experiments with an autoregressive text-to-image model demonstrate that RubricRL improves prompt faithfulness, visual detail, and generalizability, while offering a flexible and extensible foundation for interpretable RL alignment across text-to-image architectures.
Authors Xuelu Feng, Yunsheng Li, Ziyu Wan, Zixuan Gao, Junsong Yuan, Dongdong Chen, Chunming Qiao
Categories Computer Vision and Pattern Recognition
2
3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding
Title CN 基于跨视角相关性的3D感知多任务学习用于密集场景理解
Keywords 多任务学习 · 3D感知 · 跨视角相关性 · 密集预测 · 场景理解
Summary 本文提出一种3D感知的多任务学习框架，通过引入跨视角模块（CvM）捕捉几何一致性，以增强密集预测任务（如语义分割和深度估计）之间的特征关联。该模块可共享于多个任务，并在NYUv2和PASCAL-Context数据集上验证了有效性。
Reason 该论文研究的是3D感知的多任务学习，聚焦于RGB-D图像中的语义分割与深度估计等视觉任务，属于自动驾驶或通用场景理解领域。其核心方法是利用跨视角几何一致性（如cost volume）提升多任务特征的结构化表达，但该思路与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、长文档推理）关联极弱。文档图像本质是2D平面符号系统，缺乏多视角几何结构，因此跨视角相关性难以迁移应用。属于无关领域工作。
Abstract This paper addresses the challenge of training a single network to jointly perform multiple dense prediction tasks, such as segmentation and depth estimation, i.e., multi-task learning (MTL). Current approaches mainly capture cross-task relations in the 2D image space, often leading to unstructured features lacking 3D-awareness. We argue that 3D-awareness is vital for modeling cross-task correlations essential for comprehensive scene understanding. We propose to address this problem by integrating correlations across views, i.e., cost volume, as geometric consistency in the MTL network. Specifically, we introduce a lightweight Cross-view Module (CvM), shared across tasks, to exchange information across views and capture cross-view correlations, integrated with a feature from MTL encoder for multi-task predictions. This module is architecture-agnostic and can be applied to both single and multi-view data. Extensive results on NYUv2 and PASCAL-Context demonstrate that our method effectively injects geometric consistency into existing MTL methods to improve performance.
Authors Xiaoye Wang, Chen Tang, Xiangyu Yue, Wei-Hong Li
Comment 3D-aware Multi-task Learning, Cross-view Correlations, Code will be available at https://github.com/WeiHongLee/CrossView3DMTL
Categories Computer Vision and Pattern Recognition
2
AutoFocus-IL: VLM-based Saliency Maps for Data-Efficient Visual Imitation Learning without Extra Human Annotations
Title CN AutoFocus-IL：基于视觉语言模型的显著性图用于无需额外人工标注的数据高效视觉模仿学习
Keywords 视觉模仿学习 · 显著性图 · Vision-Language Model · 行为克隆 · 具身智能
Summary AutoFocus-IL提出一种基于视觉语言模型自动生成时序显著性图的方法，用于指导视觉模仿学习中的策略网络关注任务相关特征，避免干扰因素。该方法无需人工标注显著性区域，通过VLM定位演示中的关键物体，并用这些显著性图正则化行为克隆策略。在CARLA自动驾驶模拟器和真实机器人操作任务中验证了其优于标准行为克隆和其他依赖人类注视数据的先进方法。
Reason 该论文属于具身智能与机器人领域的视觉模仿学习工作，核心目标是提升策略网络在驾驶和机械臂操作任务中的注意力对齐，使用VLM生成显著性图以替代人类注视数据。尽管其利用VLM提取视觉关键区域的思想具有一定启发性，但应用场景与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂版面分析、长文档推理等）关联极弱。所处理的输入为连续场景图像或视频帧，非结构化文档，且未涉及文本识别、布局解析或多模态问答等DIU任务。因此无法迁移其方法至文档理解领域。
Abstract AutoFocus-IL is a simple yet effective method to improve data efficiency and generalization in visual imitation learning by guiding policies to attend to task-relevant features rather than distractors and spurious correlations. Although saliency regularization has emerged as a promising way to achieve this, existing approaches typically require costly supervision such as human gaze data or manual saliency annotations. In contrast, AutoFocus-IL leverages vision-language models (VLMs) to automatically identify and track key objects in demonstrations, generating temporal saliency maps that highlight causal visual signals while suppressing distractors. These maps are then used to regularize behavior cloning policies, yielding stronger alignment between visual attention and task-relevant cues. Experiments in both the CARLA simulator and real-robot manipulation tasks demonstrate that AutoFocus-IL not only outperforms standard behavior cloning but also surpasses state-of-the-art baselines that assume privileged access to human supervision, such as gaze data. Code, datasets, and trained policy videos are available at https://AutoFocus-IL.github.io/.
Authors Litian Gong, Fatemeh Bahrani, Yutai Zhou, Amin Banayeeanzade, Jiachen Li, Erdem Bıyık
Comment 8 pages, 6 figures. Code and datasets available at http://autofocus-il.github.io/
Categories Computer Vision and Pattern Recognition
2
ShelfRectNet: Single View Shelf Image Rectification with Homography Estimation
Title CN ShelfRectNet：基于单视角货架图像的单应性估计图像校正
Keywords 图像校正 · 单应性估计 · 深度学习 · 数据增强 · 零售视觉
Summary 本文提出ShelfRectNet，一种基于ConvNeXt的深度学习框架，用于从单张货架图像中估计4点参数化单应性矩阵以实现图像校正。通过归一化坐标回归提升稳定性，并设计了一种合成单应性建模的数据增强策略来缓解数据稀缺问题。在自建数据集ShelfRectSet上取得了1.298像素的平均角点误差，性能优于传统方法和其他深度学习方法。
Reason 该论文研究的是零售场景下单视角货架图像的几何校正问题，属于特定领域的图像对齐任务，核心是单应性矩阵估计与图像变形纠正。虽然使用了深度学习和数据增强策略，但其方法本质是面向通用图像几何变换的计算机视觉任务，并未涉及文档结构、文本理解、OCR或VLM等文档图像理解（DIU）的核心要素。应用场景与技术路径均远离DIU领域，且无明显可迁移至解决OCR幻觉、复杂排版或长文档推理等痛点的方法论创新。
Abstract Estimating homography from a single image remains a challenging yet practically valuable task, particularly in domains like retail, where only one viewpoint is typically available for shelf monitoring and product alignment. In this paper, we present a deep learning framework that predicts a 4-point parameterized homography matrix to rectify shelf images captured from arbitrary angles. Our model leverages a ConvNeXt-based backbone for enhanced feature representation and adopts normalized coordinate regression for improved stability. To address data scarcity and promote generalization, we introduce a novel augmentation strategy by modeling and sampling synthetic homographies. Our method achieves a mean corner error of 1.298 pixels on the test set. When compared with both classical computer vision and deep learning-based approaches, our method demonstrates competitive performance in both accuracy and inference speed. Together, these results establish our approach as a robust and efficient solution for realworld single-view rectification. To encourage further research in this domain, we will make our dataset, ShelfRectSet, and code publicly available
Authors Onur Berk Tore, Ibrahim Samil Yalciner, Server Calap
Categories Computer Vision and Pattern Recognition
2
VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
Title CN VKnowU：评估多模态大语言模型中的视觉知识理解
Keywords 视觉知识 · 视频理解 · 多模态大模型 · 基准测试 · 强化学习
Summary 本文提出了一个名为VKnowU的基准，用于评估多模态大语言模型在视频中理解物理规律和人类意图等高阶视觉知识的能力，并发布配套数据集VKnowQA及模型VideoKnow+。实验表明现有MLLM在该能力上仍显著落后于人类。虽然引入了‘See-Think-Answer’范式和基于视觉知识的强化学习奖励机制，但整体工作面向开放世界的视频理解，与文档图像理解任务无直接关联。
Reason 该论文聚焦于视频中的视觉知识理解（如物理直觉、人类意图等），属于视频理解领域，且其任务设计围绕视频中的高阶推理展开。尽管其提出的‘视觉知识’概念具有一定启发性，但核心数据与方法均基于视频模态，与文档图像理解（DIU）的文本密集、结构化排版、OCR相关幻觉等核心痛点无关。无法迁移用于解决文档场景下的细粒度识别、布局分析或表格理解等问题。
Abstract While Multimodal Large Language Models (MLLMs) have become adept at recognizing objects, they often lack the intuitive, human-like understanding of the world's underlying physical and social principles. This high-level vision-grounded semantics, which we term visual knowledge, forms a bridge between perception and reasoning, yet remains an underexplored area in current MLLMs. To systematically evaluate this capability, we present VKnowU, a comprehensive benchmark featuring 1,680 questions in 1,249 videos, covering 8 core types of visual knowledge spanning both world-centric (e.g., intuitive physics) and human-centric (e.g., subjective intentions). Evaluation of 23 SOTA MLLMs reveals that leading models still fall short of human performance, with particularly notable gaps in the world-centric. To bridge this gap, we introduce a new dataset, VKnowQA, and VideoKnow+, a baseline model that explicitly incorporates visual knowledge into MLLMs. VideoKnow+ follows a structured See-Think-Answer paradigm and adopts reinforcement learning with visual knowledge reward, achieving a +3.7% improvement on VKnowU and consistent gains on MVBench, Video-MME, and MMVU. Our work highlights visual knowledge as a missing cornerstone for developing more generalizable MLLMs that can not only see but also truly understand our physical and social worlds.
Authors Tianxiang Jiang, Sheng Xia, Yicheng Xu, Linquan Wu, Xiangyu Zeng, Limin Wang, Yu Qiao, Yi Wang
Comment Data & Code: this https URL
Categories Computer Vision and Pattern Recognition
2
Are Image-to-Video Models Good Zero-Shot Image Editors?
Title CN 图像到视频模型是好的零样本图像编辑器吗？
Keywords 图像编辑 · 视频扩散模型 · 零样本学习 · 提示工程 · 潜在空间压缩
Summary 本文提出IF-Edit，一种无需微调的框架，利用预训练的图像到视频扩散模型实现指令驱动的图像编辑。通过链式思维提示增强、时序潜在dropout和自洽后 refinement 三个模块，解决提示错位、冗余时序潜在表示和后期帧模糊等问题。在多个基准上验证了其在非刚性编辑、物理与时间推理任务上的有效性。
Reason 该论文研究的是利用视频扩散模型进行零样本图像编辑，属于图像生成与编辑领域，核心任务与文档图像理解（DIU）无关。尽管提到了时序推理和潜在空间压缩等技术，但其目标场景为通用图像编辑，未涉及OCR、布局分析、表格识别或文档视觉问答（DocVQA）等DIU核心任务。方法难以迁移到解决文档理解中的幻觉、密集文本处理或复杂排版解析等痛点，属于无关领域的应用研究。
Abstract Large-scale video diffusion models show strong world simulation and temporal reasoning abilities, but their use as zero-shot image editors remains underexplored. We introduce IF-Edit, a tuning-free framework that repurposes pretrained image-to-video diffusion models for instruction-driven image editing. IF-Edit addresses three key challenges: prompt misalignment, redundant temporal latents, and blurry late-stage frames. It includes (1) a chain-of-thought prompt enhancement module that transforms static editing instructions into temporally grounded reasoning prompts; (2) a temporal latent dropout strategy that compresses frame latents after the expert-switch point, accelerating denoising while preserving semantic and temporal coherence; and (3) a self-consistent post-refinement step that sharpens late-stage frames using a short still-video trajectory. Experiments on four public benchmarks, covering non-rigid editing, physical and temporal reasoning, and general instruction edits, show that IF-Edit performs strongly on reasoning-centric tasks while remaining competitive on general-purpose edits. Our study provides a systematic view of video diffusion models as image editors and highlights a simple recipe for unified video-image generative reasoning.
Authors Zechuan Zhang, Zhenyuan Chen, Zongxin Yang, Yi Yang
Comment technical report
Categories Computer Vision and Pattern Recognition
2
V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs
Title CN V-Attack：针对可分离值特征的可控对抗攻击以攻击大型视觉语言模型
Keywords 对抗攻击 · 视觉语言模型 · 值特征 · 语义操控 · Transformer
Summary 本文提出V-Attack，一种通过操控视觉Transformer中注意力机制的值特征（V）来实现对LVLMs更精确语义操控的对抗攻击方法。作者发现值特征相比patch token具有更高的语义解耦性，因而更适合局部概念的精准替换。通过自增强和文本引导的优化模块，该方法在多个主流LVLM上显著提升了攻击成功率。然而，该工作聚焦于模型脆弱性和安全威胁，不涉及文档理解任务本身的改进。
Reason 该论文研究的是对大型视觉语言模型（LVLMs）的对抗性攻击，核心目标是通过操纵Transformer中的值特征（Value features）实现对图像语义的精确控制攻击。尽管其技术涉及多模态模型内部表示，但研究动机和应用场景完全围绕‘对抗攻击’与‘安全性’展开，属于视觉-语言模型的安全性分析工作，而非文档理解性能提升或结构解析等DIU本题任务。此外，其方法难以迁移用于解决OCR幻觉、复杂排版理解或长文档推理等文档理解核心痛点，反而可能被归类为‘纯NLP/视觉的安全/对齐’方向。因此属于负面清单中的无关领域应用，不具备对DIU的方法论价值。
Abstract Adversarial attacks have evolved from simply disrupting predictions on conventional task-specific models to the more complex goal of manipulating image semantics on Large Vision-Language Models (LVLMs). However, existing methods struggle with controllability and fail to precisely manipulate the semantics of specific concepts in the image. We attribute this limitation to semantic entanglement in the patch-token representations on which adversarial attacks typically operate: global context aggregated by self-attention in the vision encoder dominates individual patch features, making them unreliable handles for precise local semantic manipulation. Our systematic investigation reveals a key insight: value features (V) computed within the transformer attention block serve as much more precise handles for manipulation. We show that V suppresses global-context channels, allowing it to retain high-entropy, disentangled local semantic information. Building on this discovery, we propose V-Attack, a novel method designed for precise local semantic attacks. V-Attack targets the value features and introduces two core components: (1) a Self-Value Enhancement module to refine V's intrinsic semantic richness, and (2) a Text-Guided Value Manipulation module that leverages text prompts to locate source concept and optimize it toward a target concept. By bypassing the entangled patch features, V-Attack achieves highly effective semantic control. Extensive experiments across diverse LVLMs, including LLaVA, InternVL, DeepseekVL and GPT-4o, show that V-Attack improves the attack success rate by an average of 36% over state-of-the-art methods, exposing critical vulnerabilities in modern visual-language understanding. Our code and data are available https://github.com/Summu77/V-Attack.
Authors Sen Nie, Jie Zhang, Jianxin Yan, Shiguang Shan, Xilin Chen
Comment 21 pages
Categories Computer Vision and Pattern Recognition
2
GHR-VQA: Graph-guided Hierarchical Relational Reasoning for Video Question Answering
Title CN GHR-VQA：面向视频问答的图引导分层关系推理
Keywords 视频问答 · 场景图 · 图神经网络 · 关系推理 · 跨帧建模
Summary 本文提出GHR-VQA，一种基于图结构的视频问答框架，通过将每帧表示为场景图并构建跨帧的人类中心全局图，利用图神经网络进行多层次语义融合，在AGQA数据集上提升了对人类-物体关系的理解能力。
Reason 该论文研究的是视频问答（Video QA），属于视频理解领域，其核心方法基于场景图（scene graph）建模人类与物体的跨帧交互。尽管使用了图结构进行分层关系推理，但任务对象是视频序列中的动态行为理解，与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析等无直接关联。虽然GNN和结构化推理可能在理论上对文档布局分析有一定启发，但该工作未涉及文本模态、视觉token压缩、多模态对齐或高分辨率图像处理等DIU关键技术路径，迁移价值极低。
Abstract We propose GHR-VQA, Graph-guided Hierarchical Relational Reasoning for Video Question Answering (Video QA), a novel human-centric framework that incorporates scene graphs to capture intricate human-object interactions within video sequences. Unlike traditional pixel-based methods, each frame is represented as a scene graph and human nodes across frames are linked to a global root, forming the video-level graph and enabling cross-frame reasoning centered on human actors. The video-level graphs are then processed by Graph Neural Networks (GNNs), transforming them into rich, context-aware embeddings for efficient processing. Finally, these embeddings are integrated with question features in a hierarchical network operating across different abstraction levels, enhancing both local and global understanding of video content. This explicit human-rooted structure enhances interpretability by decomposing actions into human-object interactions and enables a more profound understanding of spatiotemporal dynamics. We validate our approach on the Action Genome Question Answering (AGQA) dataset, achieving significant performance improvements, including a 7.3% improvement in object-relation reasoning over the state of the art.
Authors Dionysia Danai Brilli, Dimitrios Mallis, Vassilis Pitsikalis, Petros Maragos
Categories Computer Vision and Pattern Recognition
2
SFA: Scan, Focus, and Amplify toward Guidance-aware Answering for Video TextVQA
Title CN SFA：面向引导式回答的扫描、聚焦与放大方法用于视频文本VQA
Keywords Video TextVQA · Video-LLM · Attention Guidance · Scene Text · Training-free Framework
Summary 本文提出SFA框架，通过扫描、聚焦和放大关键视频帧区域来提升视频文本VQA性能。该方法无需训练，利用视频LLM对问题相关的文本线索进行增强引导，在多个视频文本VQA数据集上达到SOTA。然而，其技术路线专注于视频时序场景，不适用于文档图像理解任务。
Reason 该论文研究的是视频文本VQA（Video TextVQA），属于视频理解领域，其核心任务是基于视频中的动态文本进行问答。尽管提到了‘text-based visual question answering’，但应用场景为视频帧序列，涉及时间动态建模和视频LLM，与文档图像理解（DIU）关注的静态文档、排版结构、OCR后处理或密集文本识别等核心痛点无关。虽然‘focus’和‘amplify’机制可能在理论上启发注意力引导，但其方法针对视频流设计，迁移至文档图像的理解难度大，且未涉及文档特有的挑战如复杂布局、表格结构或OCR幻觉抑制。属于无关领域应用。
Abstract Video text-based visual question answering (Video TextVQA) task aims to answer questions about videos by leveraging the visual text appearing within the videos. This task poses significant challenges, requiring models to accurately perceive and comprehend scene text that varies in scale, orientation, and clarity across frames, while effectively integrating temporal and semantic context to generate precise answers. Moreover, the model must identify question-relevant textual cues and filter out redundant or irrelevant information to ensure answering is guided by the most relevant and informative cues. To address these challenges, we propose SFA, a training-free framework and the first Video-LLM-based method tailored for Video TextVQA, motivated by the human process of answering questions. By adaptively scanning video frames, selectively focusing on key regions, and directly amplifying them, SFA effectively guides the Video-LLM's attention toward essential cues, enabling it to generate more accurate answers. SFA achieves new state-of-the-art results across several public Video TextVQA datasets and surpasses previous methods by a substantial margin, demonstrating its effectiveness and generalizability.
Authors Haibin He, Qihuang Zhong, Juhua Liu, Bo Du, Peng Wang, Jing Zhang
Categories Computer Vision and Pattern Recognition
2
Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs
Title CN 安全对齐多模态大语言模型中的和谐参数自适应
Keywords 多模态大语言模型 · 持续学习 · 安全对齐 · 参数自适应 · 灾难性遗忘
Summary 本文提出了一种名为Harmonious Parameter Adaptation（HPA）的后训练框架，用于在持续视觉指令调优过程中平衡多模态大语言模型的安全性与任务性能。通过基于关注度的参数划分、平衡选择和正交更新机制，HPA缓解了灾难性遗忘并保持了模型安全性。然而，该工作聚焦于模型安全与通用任务遗忘问题，并未针对文档图像理解中的典型挑战（如密集文本识别、布局解析、视觉幻觉修正）进行设计或验证。
Reason 该论文研究的是在持续视觉指令调优中如何平衡安全性和任务性能，属于多模态大模型的安全与对齐方向。尽管其参数自适应方法有一定技术深度，但核心关注点是安全对齐（如避免有害输出），而非文档图像理解中的OCR幻觉、复杂排版、细粒度定位等关键痛点。未涉及文档结构、布局分析、表格识别或高分辨率处理等DIU核心问题，属于典型的‘平行应用’类工作——将已有参数高效微调技术应用于一个新的场景（安全对齐MLLMs），并无针对文档理解的方法论创新。
Abstract While continual visual instruction tuning (CVIT) has shown promise in adapting multimodal large language models (MLLMs), existing studies predominantly focus on models without safety alignment. This critical oversight ignores the fact that real-world MLLMs inherently require such mechanisms to mitigate potential risks. In this work, we shift our focus to CVIT for safety-aligned MLLMs and observe that during continual adaptation, the model not only suffers from task forgetting but also exhibits degradation in its safety. Achieving a harmonious balance between safety and task performance remains a crucial challenge. To address this, we propose Harmonious Parameter Adaptation (HPA), a post-training framework composed of focusing-based parameter partition, harmoniously balanced parameter selection, and orthogonal parameter adjustment. Specifically, HPA partitions parameters into two types based on their focus on safety or task performance, and selects the focused ones to preserve from a balanced perspective. In addition, HPA imposes orthogonality constraints on parameter updates to further alleviate catastrophic forgetting. Extensive experiments on the CVIT benchmark and safety evaluation datasets demonstrate that HPA better maintains high safety and mitigates forgetting than existing baselines.
Authors Ziqi Wang, Chang Che, Qi Wang, Hui Ma, Zenglin Shi, Cees G. M. Snoek, Meng Wang
Categories Computer Vision and Pattern Recognition
2
Learning Procedural-aware Video Representations through State-Grounded Hierarchy Unfolding AAAI 2026
Title CN 通过状态接地的层次展开学习程序性视频表示
Keywords 视频表示学习 · 程序性理解 · 状态接地 · 层次化建模 · AAAI
Summary 本文提出一种任务-步骤-状态（TSS）层次化框架，通过引入可观察的对象配置‘状态’作为视觉接地的语义层，提升模型对程序性视频的理解能力。采用渐进式预训练策略展开TSS层级结构，在COIN和CrossTask数据集上提升了任务识别、步骤识别和下一步预测性能。
Reason 该论文研究的是视频理解中的程序性任务建模，核心是构建任务-步骤-状态的层次化语义结构，应用于COIN和CrossTask等视频数据集。尽管其提出的‘状态’概念具有一定的视觉接地思想，可能与文档中的状态变化（如表单填写）有微弱关联，但整体属于视频理解领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关系。方法难以迁移到处理静态文档中的OCR幻觉、密集文本或复杂排版等问题。
Abstract Learning procedural-aware video representations is a key step towards building agents that can reason about and execute complex tasks. Existing methods typically address this problem by aligning visual content with textual descriptions at the task and step levels to inject procedural semantics into video representations. However, due to their high level of abstraction, 'task' and 'step' descriptions fail to form a robust alignment with the concrete, observable details in visual data. To address this, we introduce 'states', i.e., textual snapshots of object configurations, as a visually-grounded semantic layer that anchors abstract procedures to what a model can actually see. We formalize this insight in a novel Task-Step-State (TSS) framework, where tasks are achieved via steps that drive transitions between observable states. To enforce this structure, we propose a progressive pre-training strategy that unfolds the TSS hierarchy, forcing the model to ground representations in states while associating them with steps and high-level tasks. Extensive experiments on the COIN and CrossTask datasets show that our method outperforms baseline models on multiple downstream tasks, including task recognition, step recognition, and next step prediction. Ablation studies show that introducing state supervision is a key driver of performance gains across all tasks. Additionally, our progressive pretraining strategy proves more effective than standard joint training, as it better enforces the intended hierarchical structure.
Authors Jinghan Zhao, Yifei Huang, Feng Lu
Comment Accepted by AAAI 2026. 15 pages, 12 figures
Categories Computer Vision and Pattern Recognition
2
ManipShield: A Unified Framework for Image Manipulation Detection, Localization and Explanation
Title CN ManipShield：一种统一的图像篡改检测、定位与解释框架
Keywords 图像篡改检测 · 多模态大模型 · ManipBench · 可解释性 · LoRA微调
Summary 本文提出了ManipBench，一个大规模AI生成图像篡改检测基准，包含45万张由25种先进编辑模型生成的图像，并在此基础上开发了ManipShield模型，基于多模态大语言模型，通过对比LoRA微调和任务特定解码器实现篡改检测、定位与解释一体化。实验表明其在多个数据集上达到SOTA性能。
Reason 该论文聚焦于通用图像篡改检测与解释，属于视觉安全与生成模型鉴别的领域，虽然使用了MLLM架构并涉及多模态理解，但其核心任务与文档图像理解（DIU）无直接关联。未解决OCR幻觉、复杂版式解析、细粒度文本定位等DIU核心问题，且方法针对的是图像真实性判断而非文档语义结构理解，迁移价值极低。
Abstract With the rapid advancement of generative models, powerful image editing methods now enable diverse and highly realistic image manipulations that far surpass traditional deepfake techniques, posing new challenges for manipulation detection. Existing image manipulation detection and localization (IMDL) benchmarks suffer from limited content diversity, narrow generative-model coverage, and insufficient interpretability, which hinders the generalization and explanation capabilities of current manipulation detection methods. To address these limitations, we introduce \textbf{ManipBench}, a large-scale benchmark for image manipulation detection and localization focusing on AI-edited images. ManipBench contains over 450K manipulated images produced by 25 state-of-the-art image editing models across 12 manipulation categories, among which 100K images are further annotated with bounding boxes, judgment cues, and textual explanations to support interpretable detection. Building upon ManipBench, we propose \textbf{ManipShield}, an all-in-one model based on a Multimodal Large Language Model (MLLM) that leverages contrastive LoRA fine-tuning and task-specific decoders to achieve unified image manipulation detection, localization, and explanation. Extensive experiments on ManipBench and several public datasets demonstrate that ManipShield achieves state-of-the-art performance and exhibits strong generality to unseen manipulation models. Both ManipBench and ManipShield will be released upon publication.
Authors Zitong Xu, Huiyu Duan, Xiaoyu Wang, Zhaolin Cai, Kaiwei Zhang, Qiang Hu, Jing Liu, Xiongkuo Min, Guangtao Zhai
Categories Computer Vision and Pattern Recognition
2
InstaDA: Augmenting Instance Segmentation Data with Dual-Agent System
Title CN InstaDA：基于双智能体系统的实例分割数据增强
Keywords 实例分割 · 数据增强 · 双智能体系统 · 扩散模型 · LLM
Summary 本文提出InstaDA，一种无需训练的双智能体系统用于增强实例分割数据。其中Text-Agent利用LLM与扩散模型协同生成多样化图像，并通过迭代优化提示词提升生成质量；Image-Agent则基于训练图像生成新实例以丰富数据分布。实验表明其在LVIS数据集上优于基线和其他生成方法。然而，该方法聚焦于通用图像分割任务，未涉及文档结构、文本语义或排版逻辑等DIU核心问题。
Reason 该论文属于计算机视觉中的实例分割数据增强工作，核心贡献是通过LLM与扩散模型的双智能体协作来生成更多样化的训练样本。尽管其方法涉及多模态组件（文本与图像），但目标任务为通用图像实例分割（如LVIS），并未针对文档图像理解（DIU）中的典型挑战（如OCR幻觉、复杂版式、表格结构识别、长文档推理等）进行建模或优化。其提出的Prompt Rethink机制和数据增广流程属于特定任务的数据工程改进，不具备向DocVQA或文档布局分析等任务迁移的方法论价值。属于典型的‘平行应用’类研究，仅将现有生成技术组合用于提升某一视觉任务的数据质量，无底层技术创新。
Abstract Acquiring high-quality instance segmentation data is challenging due to the labor-intensive nature of the annotation process and significant class imbalances within datasets. Recent studies have utilized the integration of Copy-Paste and diffusion models to create more diverse datasets. However, these studies often lack deep collaboration between large language models (LLMs) and diffusion models, and underutilize the rich information within the existing training data. To address these limitations, we propose InstaDA, a novel, training-free Dual-Agent system designed to augment instance segmentation datasets. First, we introduce a Text-Agent (T-Agent) that enhances data diversity through collaboration between LLMs and diffusion models. This agent features a novel Prompt Rethink mechanism, which iteratively refines prompts based on the generated images. This process not only fosters collaboration but also increases image utilization and optimizes the prompts themselves. Additionally, we present an Image-Agent (I-Agent) aimed at enriching the overall data distribution. This agent augments the training set by generating new instances conditioned on the training images. To ensure practicality and efficiency, both agents operate as independent and automated workflows, enhancing usability. Experiments conducted on the LVIS 1.0 validation set indicate that InstaDA achieves significant improvements, with an increase of +4.0 in box average precision (AP) and +3.3 in mask AP compared to the baseline. Furthermore, it outperforms the leading model, DiverGen, by +0.3 in box AP and +0.1 in mask AP, with a notable +0.7 gain in box AP on common categories and mask AP gains of +0.2 on common categories and +0.5 on frequent categories.
Authors Xianbao Hou, Yonghao He, Zeyd Boukhers, John See, Hu Su, Wei Sui, Cong Yang
Categories Computer Vision and Pattern Recognition
2
Temporal-Visual Semantic Alignment: A Unified Architecture for Transferring Spatial Priors from Vision Models to Zero-Shot Temporal Tasks
Title CN 时序-视觉语义对齐：一种将视觉模型中的空间先验迁移到零样本时序任务的统一架构
Keywords 时间序列 · 图像生成 · 跨模态对齐 · 语义对齐 · 零样本学习
Summary 本文提出TimeArtist框架，通过‘预热-对齐’范式实现时间序列与视觉概念之间的语义级对齐。该方法利用双自动编码器和共享量化器学习跨模态表示，再通过投影对齐时序与视觉特征，支持从时间序列生成高保真、多样化的图像，并在零样本时序任务中表现良好。
Reason 该论文研究的是时间序列到图像生成的跨模态对齐问题，核心是将时间序列转换为视觉风格图像，并实现零样本时序任务迁移。其方法论聚焦于时间动态与视觉语义之间的生成式对齐，而非文档图像理解中的文本识别、布局分析或视觉信息抽取等任务。虽然涉及多模态表示学习，但目标场景和DIU的核心痛点（如OCR幻觉、复杂排版、细粒度定位）无直接关联。属于无关领域的平行应用，无法迁移用于解决文档理解中的关键挑战。
Abstract Large Multimodal Models (LMMs) have achieved remarkable progress in aligning and generating content across text and image modalities. However, the potential of using non-visual, continuous sequential, as a conditioning signal for high-fidelity image generation remains largely unexplored. Furthermore, existing methods that convert series into "pseudo-images" for temporal forecasting fail to establish semantic-level alignment. In this paper, we propose TimeArtist, a temporal-visual conversion framework that pioneers semantic-level alignment between time series fluctuations and visual concepts. It pioneers a "warmup-align" paradigm: first, a dual-autoencoder and shared quantizer are self-supervised trained on large-scale datasets to learn modality-shared representations. Then, the encoders and quantizer are frozen, and a projection is introduced to align temporal and visual samples at the representation level. TimeArtist establishes a versatile cross-modal framework, enabling high-quality, diverse image generation directly from time series, while capturing temporal fluctuation patterns to render images as styles transfer. Extensive experiments show that TimeArtist achieves satisfactory performance in image generation metrics, while also attaining superior results in zero-shot temporal tasks. Our work establishes a new paradigm for cross-modal generation, bridging the gap between temporal dynamics and visual semantics.
Authors Xiangkai Ma, Han Zhang, Wenzhong Li, Sanglu Lu
Categories Computer Vision and Pattern Recognition
2
RadAgents: Multimodal Agentic Reasoning for Chest X-ray Interpretation with Radiologist-like Workflows ML4H'25
Title CN RadAgents：基于放射科医生工作流的胸部X光解读多模态智能体推理
Keywords 多智能体系统 · 医学影像分析 · 多模态推理 · 临床决策支持 · 工作流建模
Summary 本文提出RadAgents，一个模拟放射科医生工作流程的多智能体系统，用于胸部X光片的解读。系统通过多个专业化代理协作，结合临床先验知识、工具调用和外部知识库，实现多模态推理，并引入视觉 grounding 和检索增强来解决跨工具不一致问题，提升结果的可靠性与可解释性。
Reason 该论文属于典型的‘平行下游应用’，即将多智能体框架应用于医学影像（CXR）解读这一特定领域。尽管其提出了具有临床可解释性的多模态智能体协作机制，但核心任务是医疗诊断而非文档图像理解（DIU）。虽然其‘多智能体协作’与‘跨模态验证’机制在理论上可能对长文档推理或幻觉缓解有启发，但文中未涉及文本布局、OCR、表格识别或文档排版等DIU核心问题，且应用场景高度垂直，方法迁移成本极高，不具备对DIU领域的直接方法论贡献。
Abstract Agentic systems offer a potential path to solve complex clinical tasks through collaboration among specialized agents, augmented by tool use and external knowledge bases. Nevertheless, for chest X-ray (CXR) interpretation, prevailing methods remain limited: (i) reasoning is frequently neither clinically interpretable nor aligned with guidelines, reflecting mere aggregation of tool outputs; (ii) multimodal evidence is insufficiently fused, yielding text-only rationales that are not visually grounded; and (iii) systems rarely detect or resolve cross-tool inconsistencies and provide no principled verification mechanisms. To bridge the above gaps, we present RadAgents, a multi-agent framework that couples clinical priors with task-aware multimodal reasoning and encodes a radiologist-style workflow into a modular, auditable pipeline. In addition, we integrate grounding and multimodal retrieval-augmentation to verify and resolve context conflicts, resulting in outputs that are more reliable, transparent, and consistent with clinical practice.
Authors Kai Zhang, Corey D Barrett, Jangwon Kim, Lichao Sun, Tara Taghavi, Krishnaram Kenthapadi
Comment ML4H'25; Work in progress
Categories Computer Vision and Pattern Recognition, Computation and Language
2
Face, Whole-Person, and Object Classification in a Unified Space Via The Interleaved Multi-Domain Identity Curriculum
Title CN 通过交错多域身份课程在统一空间中实现面部、全人和物体分类
Keywords 多任务学习 · 灾难性遗忘 · 视觉基础模型 · 统一表征 · IMIC
Summary 本文提出一种名为交错多域身份课程（IMIC）的训练方法，用于在单一嵌入空间中同时完成物体识别、人脸识别（高低质量）和全人识别四项任务，避免传统微调中的灾难性遗忘问题。基于DINOv3、CLIP和EVA-02等基础模型，IMIC实现了跨域多任务的高性能表现，并保持了较好的分布外泛化能力。分析表明，不同任务在表示空间中线性可分但存在特征共享。
Reason 该论文研究的是跨人脸、人体和通用物体的多任务统一表征学习，属于通用视觉模型的多任务微调范畴。其核心贡献是缓解灾难性遗忘的训练策略（IMIC），但任务目标与文档图像理解（DIU）无直接关联。虽然其多任务学习思想理论上可迁移，但文档理解的核心痛点如OCR幻觉、复杂版式建模、细粒度文本定位等并未被涉及，且方法针对的是身份识别类任务，难以迁移到以文本结构解析为核心的DIU场景。
Abstract Vision foundation models can perform generalized object classification in zero-shot mode, and face/person recognition when they are fine-tuned. However, fine-tuned models suffer from catastrophic forgetting. We create models that perform four tasks (object recognition, face recognition from high- and low-quality images, and person recognition from whole-body images) in a single embedding space -- without incurring substantial catastrophic forgetting. To accomplish this, we introduce two variants of the Interleaved Multi-Domain Identity Curriculum (IMIC): a gradient-coupled, interleaving training schedule that fine-tunes a foundation backbone simultaneously on all four tasks. The IMIC method proved effective with three foundation model bases: DINOv3, CLIP, and EVA-02. Two of these (EVA-02 and CLIP) performed comparably with domain experts on all four tasks concurrently and were more accurate than humans at multi-tasking across face, body, and object datasets. Further, we demonstrate that our approach does not substantially harm out-of-distribution generalization, thus maintaining a key property of foundation models. Analysis of the most accurate model variants (EVA-02 + IMIC A and B) showed linearly separable representations of the four tasks in the unified embedding space, but with substantial sharing of features across tasks. Fewer than 100 PCs calculated from any one task could perform all other tasks with nearly zero performance degradation.
Authors Thomas M Metz, Matthew Q Hill, Alice J O'Toole
Categories Computer Vision and Pattern Recognition
2
CHiQPM: Calibrated Hierarchical Interpretable Image Classification NeurIPS 2025
Title CN CHiQPM：校准的分层可解释图像分类
Keywords 可解释AI · 图像分类 · 分层解释 · 共形预测 · 可信AI
Summary 本文提出了CHiQPM模型，一种具备校准分层解释能力的全局可解释图像分类方法，支持人类专家在推理过程中进行有效判断。该模型通过对比学习实现多数类别的解释，并引入类似人类推理的层次化解释结构，同时集成了可解释的共形预测机制。实验表明其在保持接近非可解释模型精度的同时，提供了领先的可解释性。
Reason 该论文聚焦于通用图像分类任务中的可解释性问题，提出了一种具有全局和局部可解释性的分层模型，主要面向安全关键领域的可信AI。尽管其在可解释性和校准预测方面有创新，但属于通用计算机视觉领域，且任务为图像分类，与文档图像理解（如DocVQA、布局分析、表格识别等）无直接关联。未解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点，方法也难以迁移至文档理解场景。
Abstract Globally interpretable models are a promising approach for trustworthy AI in safety-critical domains. Alongside global explanations, detailed local explanations are a crucial complement to effectively support human experts during inference. This work proposes the Calibrated Hierarchical QPM (CHiQPM) which offers uniquely comprehensive global and local interpretability, paving the way for human-AI complementarity. CHiQPM achieves superior global interpretability by contrastively explaining the majority of classes and offers novel hierarchical explanations that are more similar to how humans reason and can be traversed to offer a built-in interpretable Conformal prediction (CP) method. Our comprehensive evaluation shows that CHiQPM achieves state-of-the-art accuracy as a point predictor, maintaining 99% accuracy of non-interpretable models. This demonstrates a substantial improvement, where interpretability is incorporated without sacrificing overall accuracy. Furthermore, its calibrated set prediction is competitively efficient to other CP methods, while providing interpretable predictions of coherent sets along its hierarchical explanation.
Authors Thomas Norrenbrock, Timo Kaiser, Sovan Biswas, Neslihan Kose, Ramesh Manuvinakurike, Bodo Rosenhahn
Comment Accepted to NeurIPS 2025
Categories Computer Vision and Pattern Recognition
2
Prompt-Aware Adaptive Elastic Weight Consolidation for Continual Learning in Medical Vision-Language Models 32nd International Conference on MultiMedia Modeling (MMM 2026)
Title CN 面向医学视觉-语言模型持续学习的提示感知自适应弹性权重巩固
Keywords 持续学习 · 医学视觉-语言模型 · 弹性权重巩固 · 灾难性遗忘 · 提示工程
Summary 本文提出PA-EWC方法，通过分析参数功能角色和术语密度，在医学视觉-语言模型中实现提示感知的持续学习，减少灾难性遗忘。实验在多个医学影像数据集上验证了其有效性。
Reason 该论文聚焦于医学视觉-语言模型在持续学习场景下的灾难性遗忘问题，提出了一种基于提示引导的参数保护方法。尽管涉及多模态对齐，但其核心贡献在于特定领域（医学）的持续学习优化，属于典型的‘平行下游应用’。它并未解决文档图像理解中的共性技术痛点（如OCR幻觉、复杂版面解析、长文档推理等），也未提出可迁移至通用DIU任务的底层创新机制。
Abstract Medical AI systems face catastrophic forgetting when deployed in clinical settings, where models must learn new imaging protocols while retaining prior diagnostic capabilities. This challenge is particularly acute for medical vision-language models that must preserve complex cross-modal alignments between medical images and clinical terminology across diverse imaging modalities. We introduce Prompt- Aware Adaptive Elastic Weight Consolidation (PA-EWC), a novel continual learning approach that addresses catastrophic forgetting through prompt-guided parameter specialization. Our method systematically categorizes model parameters based on their functional roles in processing visual-descriptive, spatial-guided, and medical-semantic information, enabling targeted protection of critical knowledge while allowing adaptation to new clinical requirements. PA-EWC incorporates adaptive Fisher Information computation with gradient stability analysis and develops weighted complexity metrics based on medical terminology density. We evaluate our approach across five medical imaging datasets (Kvasir-SEG, ISIC 2018, CheXlocalize, BUSI, CAMUS) representing diverse modalities including endoscopy, dermoscopy, radiography, and ultrasound. Experimental results demonstrate that PA-EWC reduces catastrophic forgetting by up to 17.58% compared to baseline methods, with performance improvements of 4.30% on chest X-ray pathology localization and 6.06% on polyp segmentation.
Authors Ziyuan Gao, Philippe Morel
Comment Accepted by 32nd International Conference on MultiMedia Modeling (MMM 2026)
Categories Computer Vision and Pattern Recognition
2
REFLEX: Self-Refining Explainable Fact-Checking via Disentangling Truth into Style and Substance
Title CN REFLEX：通过将真实性解耦为风格与实质实现自我精炼的可解释事实核查
Keywords 事实核查 · 幻觉抑制 · 自我精炼 · 激活引导 · 解释性
Summary 本文提出REFLEX，一种基于角色扮演对话的自精炼事实核查框架，通过对比激活对构建引导向量，将真实性分解为风格与实质，利用模型内部知识提升判断准确性和解释质量。实验显示其在少量样本下即可达到SOTA性能，并能跨模型传递解释信号以增强推理。然而，整个方法不涉及图像或文档结构，局限于纯文本事实核查场景。
Reason 该论文属于纯文本事实核查任务，聚焦于利用LLM内部知识进行自我精炼和解释生成，虽涉及幻觉抑制，但完全基于文本模态，未引入视觉或文档布局信息。其方法无法直接迁移至文档图像理解（DIU）中的核心问题如OCR幻觉、细粒度定位或多模态对齐。属于典型的‘平行应用’类研究——在特定NLP任务上应用干预机制，无视觉或多模态扩展路径。
Abstract The prevalence of misinformation on social media threatens public trust, demanding automated fact-checking systems that provide accurate verdicts with interpretable explanations. However, existing large language model-based (LLM-based) approaches often rely heavily on external knowledge sources, introducing substantial latency and even hallucinations that undermine reliability, interpretability, and responsiveness, which is crucial for real-time use. To address these challenges, we propose REason-guided Fact-checking with Latent EXplanations REFLEX paradigm, a plug-and-play, self-refining paradigm that leverages the internal knowledge in backbone model to improve both verdict accuracy and explanation quality. REFLEX reformulates fact-checking as a role-play dialogue and jointly trains verdict prediction and explanation generation. It adaptively extracts contrastive activation pairs between the backbone model and its fine-tuned variant to construct steering vectors that disentangle truth into style and substance naturally. These activation-level signals guide inference and suppress noisy explanations, enabling more faithful and efficient reasoning. Experiments on real-world datasets show that REFLEX outperforms previous methods that steer toward a single truth direction and underscores the challenge traditional approaches face when handling the subtle, human-unknown truth in fact-checking tasks. Remarkably, with only 465 self-refined training samples, RELFEX achieves state-of-the-art performance. Furthermore, models trained with explanatory objectives can effectively guide those without them, yielding up to a 7.57% improvement, highlighting that internal explanation signals play a dual role in both interpreting and enhancing factual reasoning.
Authors Chuyi Kong, Gao Wei, Jing Ma, Hongzhan Lin, Zhiyuan Fan
Categories Computation and Language
2
From Generation to Detection: A Multimodal Multi-Task Dataset for Benchmarking Health Misinformation Findings of the Association for Computational Linguistics: EMNLP 2025
Title CN 从生成到检测：一个用于基准测试健康虚假信息的多模态多任务数据集
Keywords 虚假信息检测 · 多模态数据集 · 生成式AI · 健康信息 · AI生成内容识别
Summary 本文提出了一个大规模多模态健康虚假信息数据集MM Health，包含人类生成和AI生成的新闻文章，涵盖文本与视觉信息，并在可靠性判断、原创性检测和细粒度AI内容识别三个任务上进行了基准测试。结果显示当前SOTA模型在区分信息来源和可信度方面表现不佳。
Reason 该论文属于典型的‘平行下游应用’，其核心是构建一个面向健康领域的多模态虚假信息检测数据集（MM Health），并针对社交媒体和生成式AI产生的虚假内容进行建模。尽管涉及图像与文本的多模态分析，但其任务目标为‘健康 misinformation detection’，并非文档理解中的DocVQA、布局分析、表格识别等核心任务。方法上未提出可迁移至文档图像理解的新架构、训练范式或推理机制，而是聚焦特定应用场景的数据集构建与任务定义，属于对现有技术的垂直应用，无底层方法创新。
Abstract Infodemics and health misinformation have significant negative impact on individuals and society, exacerbating confusion and increasing hesitancy in adopting recommended health measures. Recent advancements in generative AI, capable of producing realistic, human like text and images, have significantly accelerated the spread and expanded the reach of health misinformation, resulting in an alarming surge in its dissemination. To combat the infodemics, most existing work has focused on developing misinformation datasets from social media and fact checking platforms, but has faced limitations in topical coverage, inclusion of AI generation, and accessibility of raw content. To address these issues, we present MM Health, a large scale multimodal misinformation dataset in the health domain consisting of 34,746 news article encompassing both textual and visual information. MM Health includes human-generated multimodal information (5,776 articles) and AI generated multimodal information (28,880 articles) from various SOTA generative AI models. Additionally, We benchmarked our dataset against three tasks (reliability checks, originality checks, and fine-grained AI detection) demonstrating that existing SOTA models struggle to accurately distinguish the reliability and origin of information. Our dataset aims to support the development of misinformation detection across various health scenarios, facilitating the detection of human and machine generated content at multimodal levels.
Authors Zhihao Zhang, Yiran Zhang, Xiyue Zhou, Liting Huang, Imran Razzak, Preslav Nakov, Usman Naseem
Comment Accepted to Findings of the Association for Computational Linguistics: EMNLP 2025
Categories Computation and Language
2
$\text{R}^2\text{R}$: A Route-to-Rerank Post-Training Framework for Multi-Domain Decoder-Only Rerankers
Title CN R²R：一种面向多领域仅解码器重排序器的路由到重排序后训练框架
Keywords 重排序器 · 领域适应 · LoRA · RAG · 表面形式过拟合
Summary 本文提出R²R框架，用于提升仅解码器重排序器在多领域场景下的性能。通过引入实体抽象泛化（EAG）训练策略和轻量级潜在语义路由器，避免模型对表面形式过拟合，并实现按查询选择最优LoRA专家。实验表明其在法律、医疗、金融等领域具有优越的跨域鲁棒性。
Reason 该论文研究的是检索增强生成（RAG）中的重排序器（reranker）在多领域下的域适应问题，核心贡献是通过动态专家路由和实体抽象机制提升重排序器的泛化能力。尽管其涉及法律、金融等文档相关领域，但本质是NLP下游应用层面的技术优化，属于典型的‘平行应用’：它并未解决文档图像理解中的OCR幻觉、复杂排版、视觉-文本对齐或高分辨率输入处理等核心痛点。方法论上聚焦于LoRA微调与路由机制，无法迁移到VLM处理文档图像的底层架构或推理机制中。
Abstract Decoder-only rerankers are central to Retrieval-Augmented Generation (RAG). However, generalist models miss domain-specific nuances in high-stakes fields like finance and law, and naive fine-tuning causes surface-form overfitting and catastrophic forgetting. To address this challenge, we introduce R2R, a domain-aware framework that combines dynamic expert routing with a two-stage training strategy, Entity Abstraction for Generalization (EAG). EAG introduces a counter-shortcut mechanism by masking the most predictive surface cues, forcing the reranker to learn domain-invariant relevance patterns rather than memorizing dataset-specific entities. To efficiently activate domain experts, R2R employs a lightweight Latent Semantic Router that probes internal representations from the frozen backbone decoder to select the optimal LoRA expert per query. Extensive experiments across different reranker backbones and diverse domains (legal, medical, and financial) demonstrate that R2R consistently surpasses generalist and single-domain fine-tuned baselines. Our results confirm that R2R is a model-agnostic and modular approach to domain specialization with strong cross-domain robustness.
Authors Xinyu Wang, Hanwei Wu, Qingchen Hu, Zhenghan Tai, Jingrui Tian, Lei Ding, Jijun Chi, Hailin He, Tung Sum Thomas Kwok, Yufei Cui, Sicheng Lyu, Muzhi Li, Mingze Li, Xinyue Yu, Ling Zhou, Peng Lu
Comment 13 pages, including 3 figures and 3 tables
Categories Computation and Language
2
Web-Shepherd: Advancing PRMs for Reinforcing Web Agents NeurIPS 2025 Spotlight
Title CN Web-Shepherd：推进用于增强网页代理的过程奖励模型
Keywords 过程奖励模型 · 网页代理 · 强化学习 · 偏好学习 · 代理验证
Summary 本文提出了Web-Shepherd，首个面向网页导航任务的步骤级过程奖励模型（PRM），并构建了包含4万对偏好数据的WebPRM Collection以及评测基准WebRewardBench。实验表明，该模型在评估网页操作轨迹时显著优于GPT-4o，且在降低10倍成本的情况下提升了代理性能。然而，其应用场景局限于网页交互自动化，缺乏对文档图像中视觉-语言联合理解的支持。
Reason 该论文属于典型的‘平行应用’类型，研究的是网页导航（web navigation）中的代理（agent）强化问题，使用过程奖励模型（PRM）来提升网页操作任务的效率和准确性。虽然其方法在agent workflow和test-time verification方面有一定创新，但其核心场景与文档图像理解（DIU）如DocVQA、表格识别、OCR去幻觉、复杂排版解析等无直接关联。尽管验证机制或偏好学习可能作为通用工具迁移，但本文未涉及视觉内容解析、多模态对齐、高分辨率处理或文本结构理解等DIU核心痛点，属于特定领域agent的应用，不符合本研究方向的方法论源头要求。
Abstract Web navigation is a unique domain that can automate many repetitive real-life tasks and is challenging as it requires long-horizon sequential decision making beyond typical multimodal large language model (MLLM) tasks. Yet, specialized reward models for web navigation that can be utilized during both training and test-time have been absent until now. Despite the importance of speed and cost-effectiveness, prior works have utilized MLLMs as reward models, which poses significant constraints for real-world deployment. To address this, in this work, we propose the first process reward model (PRM) called Web-Shepherd which could assess web navigation trajectories in a step-level. To achieve this, we first construct the WebPRM Collection, a large-scale dataset with 40K step-level preference pairs and annotated checklists spanning diverse domains and difficulty levels. Next, we also introduce the WebRewardBench, the first meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe that our Web-Shepherd achieves about 30 points better accuracy compared to using GPT-4o on WebRewardBench. Furthermore, when testing on WebArena-lite by using GPT-4o-mini as the policy and Web-Shepherd as the verifier, we achieve 10.9 points better performance, in 10 less cost compared to using GPT-4o-mini as the verifier. Our model, dataset, and code are publicly available at LINK.
Authors Hyungjoo Chae, Sunghwan Kim, Junhee Cho, Seungone Kim, Seungjun Moon, Gyeom Hwangbo, Dongha Lim, Minjin Kim, Yeonjun Hwang, Minju Gwak, Dongwook Choi, Minseok Kang, Gwanhoon Im, ByeongUng Cho, Hyojun Kim, Jun Hee Han, Taeyoon Kwon, Minju Kim, Beong-woo Kwak, Dongjin Kang, Jinyoung Yeo
Comment NeurIPS 2025 Spotlight
Categories Computation and Language
2
Mixture of Horizons in Action Chunking
Title CN 动作分块中的混合视野
Keywords 视觉-语言-动作模型 · 动作分块 · 混合视野 · 机器人操纵 · 序列建模
Summary 本文针对视觉-语言-动作模型在机器人操作中对动作视野长度敏感的问题，提出了一种混合视野（MoH）策略，通过并行处理不同视野长度的动作片段，并融合输出以兼顾长期规划与短期精度。该方法具有即插即用特性，并支持动态自适应推理，在仿真和真实任务中均表现出色。
Reason 该论文属于具身智能与机器人领域，研究的是视觉-语言-动作（VLA）模型在机械臂操作中的应用，核心问题是动作序列的‘视野长度’（horizon）选择。尽管提出了‘混合视野’（MoH）这一有潜力的架构设计，但其任务场景和目标与文档图像理解（DIU）无直接关联。所解决的‘长视野与短视野权衡’问题并非文档理解中的OCR幻觉、复杂排版或长文档推理等痛点，且方法迁移路径不明确，属于典型的无关领域工作。
Abstract Vision-language-action (VLA) models have shown remarkable capabilities in robotic manipulation, but their performance is sensitive to the $\textbf{action chunk length}$ used during training, termed $\textbf{horizon}$. Our empirical study reveals an inherent trade-off: longer horizons provide stronger global foresight but degrade fine-grained accuracy, while shorter ones sharpen local control yet struggle on long-term tasks, implying fixed choice of single horizons being suboptimal. To mitigate the trade-off, we propose a $\textbf{mixture of horizons (MoH)}$ strategy. MoH rearranges the action chunk into several segments with different horizons, processes them in parallel with a shared action transformer, and fuses outputs with a light linear gate. It has three appealing benefits. 1) MoH exploits long-term foresight and short-term precision jointly within a single model, improving both performance and generalizability to complex tasks. 2) MoH is plug-and-play for full-attention action modules with minimal training or inference overhead. 3) MoH enables dynamic inference with adaptive horizons, which selects stable actions through cross-horizon consensus, achieving 2.5$\times$ higher throughput than baselines while preserving superior performance. Extensive experiments over flow-based policies $π_0$, $π_{0.5}$, and one-step regression policy $π_{\text{reg}}$ demonstrate that MoH yields consistent and significant gains on both simulations and real-world tasks. Notably, under mixed-task setting, $π_{0.5}$ with MoH reaches a new state-of-the-art with 99$\%$ average success rate on LIBERO after only $30k$ training iterations. Project page: https://github.com/Timsty1/MixtureOfHorizons
Authors Dong Jing, Gang Wang, Jiaqi Liu, Weiliang Tang, Zelong Sun, Yunchao Yao, Zhenyu Wei, Yunhui Liu, Zhiwu Lu, Mingyu Ding
Comment 15 pages, 14 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
Prompt Less, Smile More: MTP with Semantic Engineering in Lieu of Prompt Engineering
Title CN 少写提示，多微笑：通过语义工程实现无需提示工程的MTP
Keywords 提示工程 · 语义工程 · AI集成编程 · MTP · Jac语言
Summary 本文提出了一种名为Semantic Engineering的方法，用于在Meaning Typed Programming（MTP）中替代传统提示工程。通过引入语义上下文注释（SemTexts），开发者可将自然语言意图嵌入程序结构中，由系统自动生成功能更准确的提示。该方法集成于Jac语言，并在自建基准上验证了其在降低人工负担的同时保持提示质量的有效性。
Reason 该论文聚焦于AI集成编程中的提示生成优化，属于将LLM应用于软件开发流程的‘平行应用’场景。其核心贡献Semantic Engineering和SemTexts机制旨在减少开发者在提示工程上的投入，但并未解决文档图像理解（DIU）中的任何核心技术痛点，如OCR幻觉、复杂排版解析、细粒度视觉定位或高分辨率文档处理。方法本质上是代码语义增强与自然语言上下文注释的结合，与多模态文档理解无直接关联，无法迁移至DIU领域。
Abstract AI-Integrated programming is emerging as a foundational paradigm for building intelligent systems with large language models (LLMs). Recent approaches such as Meaning Typed Programming (MTP) automate prompt generation by leveraging the semantics already present in code. However, many real-world applications depend on contextual cues, developer intent, and domain-specific reasoning that extend beyond what static code semantics alone can express. To address this limitation, we introduce Semantic Engineering, a lightweight method for enriching program semantics so that LLM-based systems can more accurately reflect developer intent without requiring full manual prompt design. We present Semantic Context Annotations (SemTexts), a language-level mechanism that allows developers to embed natural-language context directly into program constructs. Integrated into the Jac programming language, Semantic Engineering extends MTP to incorporate these enriched semantics during prompt generation. We further introduce a benchmark suite designed to reflect realistic AI-Integrated application scenarios. Our evaluation shows that Semantic Engineering substantially improves prompt fidelity, achieving performance comparable to Prompt Engineering while requiring significantly less developer effort.
Authors Jayanaka L. Dantanarayana, Savini Kashmira, Thakee Nathees, Zichen Zhang, Krisztian Flautner, Lingjia Tang, Jason Mars
Categories Artificial Intelligence
2
Communicating Plans, Not Percepts: Scalable Multi-Agent Coordination with Embodied World Models Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop: Scaling Environments for Agents (SEA)
Title CN 沟通计划而非感知：基于具身世界模型的可扩展多智能体协调
Keywords 多智能体系统 · 强化学习 · 世界模型 · 意图通信 · 可扩展性
Summary 本文提出了一种基于具身世界模型的多智能体通信策略，通过构建想象轨迹生成模块（ITGM）和消息生成网络（MGN），将智能体的未来行为计划压缩为通信消息，从而实现更高效、可扩展的协作。实验表明，在环境复杂度提升时，这种基于结构化预测模型的通信方式优于端到端学习的通信协议。
Reason 该论文属于具身智能与多智能体强化学习领域，研究的是在部分可观环境中通过世界模型进行意图通信的机制。尽管其提出的‘计划压缩成消息’的思想具有一定抽象启发性，但整体方法针对的是导航类任务和强化学习框架下的协作问题，与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析或长文档推理无直接关联。无法迁移用于解决VLM在文档理解中的细粒度对齐、高分辨率处理或逻辑推理问题，属于无关领域工作。
Abstract Robust coordination is critical for effective decision-making in multi-agent systems, especially under partial observability. A central question in Multi-Agent Reinforcement Learning (MARL) is whether to engineer communication protocols or learn them end-to-end. We investigate this dichotomy using embodied world models. We propose and compare two communication strategies for a cooperative task-allocation problem. The first, Learned Direct Communication (LDC), learns a protocol end-to-end. The second, Intention Communication, uses an engineered inductive bias: a compact, learned world model, the Imagined Trajectory Generation Module (ITGM), which uses the agent's own policy to simulate future states. A Message Generation Network (MGN) then compresses this plan into a message. We evaluate these approaches on goal-directed interaction in a grid world, a canonical abstraction for embodied AI problems, while scaling environmental complexity. Our experiments reveal that while emergent communication is viable in simple settings, the engineered, world model-based approach shows superior performance, sample efficiency, and scalability as complexity increases. These findings advocate for integrating structured, predictive models into MARL agents to enable active, goal-driven coordination.
Authors Brennen A. Hill, Mant Koh En Wei, Thangavel Jishnuanandh
Comment Published in the Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop: Scaling Environments for Agents (SEA). Additionally accepted for presentation in the NeurIPS 2025 Workshop: Embodied World Models for Decision Making (EWM) and the NeurIPS 2025 Workshop: Optimization for Machine Learning (OPT)
Categories Artificial Intelligence
2
Predicting partially observable dynamical systems via diffusion models with a multiscale inference scheme
Title CN 通过具有多尺度推理机制的扩散模型预测部分可观测的动力系统
Keywords 扩散模型 · 多尺度推理 · 动力系统 · 部分可观测 · 时间序列预测
Summary 本文提出一种用于部分可观测动力系统预测的多尺度扩散模型推理方法，通过在时间维度上采用近细远粗的生成策略来捕捉长程依赖并降低计算偏差。主要应用于太阳活动区域演化预测等物理过程。
Reason 该论文研究的是动力系统的概率预测问题，属于物理建模与时间序列预测领域。尽管其提出的多尺度推理机制在理论上可能对长程依赖建模有启发，但其核心任务（如太阳物理演化）与文档图像理解（DIU）无直接关联，且方法迁移路径不清晰。无法有效解决OCR幻觉、复杂排版或细粒度定位等DIU核心痛点，属于无关领域的应用工作。
Abstract Conditional diffusion models provide a natural framework for probabilistic prediction of dynamical systems and have been successfully applied to fluid dynamics and weather prediction. However, in many settings, the available information at a given time represents only a small fraction of what is needed to predict future states, either due to measurement uncertainty or because only a small fraction of the state can be observed. This is true for example in solar physics, where we can observe the Sun's surface and atmosphere, but its evolution is driven by internal processes for which we lack direct measurements. In this paper, we tackle the probabilistic prediction of partially observable, long-memory dynamical systems, with applications to solar dynamics and the evolution of active regions. We show that standard inference schemes, such as autoregressive rollouts, fail to capture long-range dependencies in the data, largely because they do not integrate past information effectively. To overcome this, we propose a multiscale inference scheme for diffusion models, tailored to physical processes. Our method generates trajectories that are temporally fine-grained near the present and coarser as we move farther away, which enables capturing long-range temporal dependencies without increasing computational cost. When integrated into a diffusion model, we show that our inference scheme significantly reduces the bias of the predicted distributions and improves rollout stability.
Authors Rudy Morel, Francesco Pio Ramunno, Jeff Shen, Alberto Bietti, Kyunghyun Cho, Miles Cranmer, Siavash Golkar, Olexandr Gugnin, Geraud Krawezik, Tanya Marwah, Michael McCabe, Lucas Meyer, Payel Mukhopadhyay, Ruben Ohana, Liam Parker, Helen Qu, François Rozet, K. D. Leka, François Lanusse, David Fouhey, Shirley Ho
Categories Artificial Intelligence
2
ALMAS: an Autonomous LLM-based Multi-Agent Software Engineering Framework ASE 2025
Title CN ALMAS：一种基于大语言模型的自主多智能体软件工程框架
Keywords 多智能体系统 · 大语言模型 · 软件工程自动化 · 敏捷开发 · LLM应用
Summary 本文提出ALMAS，一个基于大语言模型的多智能体软件工程框架，旨在模拟敏捷开发中的不同角色，实现从需求到代码实现的端到端自动化。系统以模块化方式集成到现有开发流程中，并通过用例展示了其生成应用程序和添加新功能的能力。
Reason 该论文属于典型的平行下游应用，将多智能体LLM框架应用于软件工程领域，尽管涉及Agent架构设计，但其核心是特定场景（软件开发）的应用集成与流程编排，并未提出可用于文档图像理解（如OCR去幻觉、复杂版式建模、视觉token压缩等）的底层方法创新。无法迁移解决DIU的核心痛点。
Abstract Multi-agent Large Language Model (LLM) systems have been leading the way in applied LLM research across a number of fields. One notable area is software development, where researchers have advanced the automation of code implementation, code testing, code maintenance, inter alia, using LLM agents. However, software development is a multifaceted environment that extends beyond just code. As such, a successful LLM system must factor in multiple stages of the software development life-cycle (SDLC). In this paper, we propose a vision for ALMAS, an Autonomous LLM-based Multi-Agent Software Engineering framework, which follows the above SDLC philosophy such that it may work within an agile software development team to perform several tasks end-to-end. ALMAS aligns its agents with agile roles, and can be used in a modular fashion to seamlessly integrate with human developers and their development environment. We showcase the progress towards ALMAS through our published works and a use case demonstrating the framework, where ALMAS is able to seamlessly generate an application and add a new feature.
Authors Vali Tawosi, Keshav Ramani, Salwa Alamir, Xiaomo Liu
Comment Accepted to MAS-GAIN Workshop at ASE 2025
Categories Artificial Intelligence
2
Generative Query Expansion with Multilingual LLMs for Cross-Lingual Information Retrieval
Title CN 基于多语言大语言模型的生成式查询扩展在跨语言信息检索中的应用
Keywords 查询扩展 · 多语言大模型 · 信息检索 · 伪文档生成 · 跨语言
Summary 本文研究了多语言大语言模型在跨语言信息检索中用于生成式查询扩展的效果，比较了不同提示策略和微调方法对检索性能的影响。结果表明查询长度影响策略有效性，语言差异显著影响效果，尤其在不同书写系统的语言间表现较差。微调仅在训练与测试数据格式一致时有效，凸显了对更均衡多语言资源的需求。
Reason 该论文聚焦于跨语言信息检索中的查询扩展问题，属于典型的NLP下游应用任务，且核心方法是利用mLLM生成伪文档来增强检索。尽管使用了多语言大模型，但其技术路线并未涉及文档图像理解中的关键挑战（如OCR幻觉、复杂版面解析、视觉-文本对齐等）。研究内容本质上是信息检索领域的文本增强策略，不包含针对文档视觉模态的建模或改进，无法迁移解决DIU的核心痛点。
Abstract Query expansion is the reformulation of a user query by adding semantically related information, and is an essential component of monolingual and cross-lingual information retrieval used to ensure that relevant documents are not missed. Recently, multilingual large language models (mLLMs) have shifted query expansion from semantic augmentation with synonyms and related words to pseudo-document generation. Pseudo-documents both introduce additional relevant terms and bridge the gap between short queries and long documents, which is particularly beneficial in dense retrieval. This study evaluates recent mLLMs and fine-tuned variants across several generative expansion strategies to identify factors that drive cross-lingual retrieval performance. Results show that query length largely determines which prompting technique is effective, and that more elaborate prompts often do not yield further gains. Substantial linguistic disparities persist: cross-lingual query expansion can produce the largest improvements for languages with the weakest baselines, yet retrieval is especially poor between languages written in different scripts. Fine-tuning is found to lead to performance gains only when the training and test data are of similar format. These outcomes underline the need for more balanced multilingual and cross-lingual training and evaluation resources.
Authors Olivia Macmillan-Scott, Roksana Goworek, Eda B. Özyiğit
Categories Artificial Intelligence, Computation and Language
2
BioDisco: Multi-agent hypothesis generation with dual-mode evidence, iterative feedback and temporal evaluation
Title CN BioDisco：基于双模态证据、迭代反馈和时序评估的多智能体假设生成
Keywords 多智能体系统 · 假设生成 · 知识图谱 · 文献检索 · 语言模型推理
Summary BioDisco是一个用于生物医学领域新假设生成的多智能体系统，结合知识图谱与文献检索作为双模态证据源，通过内部评分与反馈循环进行假设迭代优化，并采用时序评估和成对比较模型验证性能。系统强调新颖性和证据支撑，适用于生物发现场景。
Reason 该论文属于典型的‘平行应用’——将多智能体框架应用于生物医学领域的假设生成，尽管其方法包含迭代反馈和证据检索机制，但整体设计面向科研假设发现这一特定下游任务，并未提出可迁移至文档图像理解（DIU）的底层技术改进。虽然其多智能体协作与反馈机制具有一定通用性，但缺乏对视觉输入处理、OCR幻觉缓解、细粒度定位或复杂版式建模等DIU核心问题的关注，无法直接服务于文档理解中的推理或结构解析需求。
Abstract Identifying novel hypotheses is essential to scientific research, yet this process risks being overwhelmed by the sheer volume and complexity of available information. Existing automated methods often struggle to generate novel and evidence-grounded hypotheses, lack robust iterative refinement and rarely undergo rigorous temporal evaluation for future discovery potential. To address this, we propose BioDisco, a multi-agent framework that draws upon language model-based reasoning and a dual-mode evidence system (biomedical knowledge graphs and automated literature retrieval) for grounded novelty, integrates an internal scoring and feedback loop for iterative refinement, and validates performance through pioneering temporal and human evaluations and a Bradley-Terry paired comparison model to provide statistically-grounded assessment. Our evaluations demonstrate superior novelty and significance over ablated configurations and generalist biomedical agents. Designed for flexibility and modularity, BioDisco allows seamless integration of custom language models or knowledge graphs, and can be run with just a few lines of code.
Authors Yujing Ke, Kevin George, Kathan Pandya, David Blumenthal, Maximilian Sprang, Gerrit Großmann, Sebastian Vollmer, David Antony Selby
Comment 12 pages main content, 31 including appendices. 8 figures
Categories Artificial Intelligence
2
Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models Proceedings on Privacy Enhancing Technologies (PoPETs) 1 (2026)
Title CN 使用大语言模型对隐私政策中的GDPR透明度合规性进行词级标注
Keywords GDPR合规 · 词级标注 · 大语言模型 · 隐私政策 · 文本分析
Summary 本文提出一个基于大语言模型的流水线，用于对隐私政策进行词级的GDPR合规性标注。方法结合了段落分类、检索增强生成和自我修正机制，并构建了一个大规模语料库和人工标注样本用于评估。虽然在合规自动化方面有一定贡献，但本质是垂直领域的NLP应用。
Reason 该论文属于典型的‘平行下游应用’：其核心是将现有LLM技术应用于隐私政策文本的合规性分析，任务场景固定于GDPR合规评估，未提出任何面向文档图像理解（DIU）的底层方法创新。尽管涉及细粒度文本标注，但完全基于纯文本处理，不涉及OCR、视觉布局、多模态对齐或文档图像中的幻觉缓解等DIU核心问题。属于在特定领域（法律/NLP）的应用包装，无技术源头价值。
Abstract Ensuring transparency of data practices related to personal information is a core requirement of the General Data Protection Regulation (GDPR). However, large-scale compliance assessment remains challenging due to the complexity and diversity of privacy policy language. Manual audits are labour-intensive and inconsistent, while current automated methods often lack the granularity required to capture nuanced transparency disclosures. In this paper, we present a modular large language model (LLM)-based pipeline for fine-grained word-level annotation of privacy policies with respect to GDPR transparency requirements. Our approach integrates LLM-driven annotation with passage-level classification, retrieval-augmented generation, and a self-correction mechanism to deliver scalable, context-aware annotations across 21 GDPR-derived transparency requirements. To support empirical evaluation, we compile a corpus of 703,791 English-language privacy policies and generate a ground-truth sample of 200 manually annotated policies based on a comprehensive, GDPR-aligned annotation scheme. We propose a two-tiered evaluation methodology capturing both passage-level classification and span-level annotation quality and conduct a comparative analysis of seven state-of-the-art LLMs on two annotation schemes, including the widely used OPP-115 dataset. The results of our evaluation show that decomposing the annotation task and integrating targeted retrieval and classification components significantly improve annotation accuracy, particularly for well-structured requirements. Our work provides new empirical resources and methodological foundations for advancing automated transparency compliance assessment at scale.
Authors Thomas Cory, Wolf Rieder, Julia Krämer, Philip Raschke, Patrick Herbke, Axel Küpper
Comment Accepted to Proceedings on Privacy Enhancing Technologies (PoPETs) 1 (2026)
Categories Artificial Intelligence, Computation and Language
2
WorldLLM: Improving LLMs' world modeling using curiosity-driven theory-making
Title CN WorldLLM：利用好奇心驱动的理论构建改进大语言模型的世界建模
Keywords 大语言模型 · 世界建模 · 贝叶斯推理 · 主动探索 · 强化学习
Summary 本文提出WorldLLM，一种结合贝叶斯推理与好奇心驱动强化学习的框架，用于提升大语言模型在结构化环境中的世界建模能力。该方法通过一个LLM生成假设，另一个LLM作为提议分布进行证据更新，并由探索策略主动收集低似然状态转移数据，从而持续优化预测模型。实验表明其在文本游戏环境中能生成可解释的动态规律并提高预测精度。
Reason 该论文属于具身智能与推理代理的交叉研究，聚焦于文本游戏环境中的世界模型构建，通过贝叶斯推理和强化学习实现假设迭代优化。尽管其‘推理-探索-更新’框架在理论上具有启发性，但其核心任务与文档图像理解（DIU）无直接关联，既不涉及视觉输入处理、多模态对齐、OCR去幻觉，也不解决文档布局、细粒度定位或长序列理解等关键问题。所提出的‘自然语言假设引导预测’机制虽可用于抽象推理系统，但迁移至文档理解场景缺乏明确路径，属于无关领域的平行方法应用。
Abstract Large Language Models (LLMs) possess general world knowledge but often struggle to generate precise predictions in structured, domain-specific contexts such as simulations. These limitations arise from their inability to ground their broad, unstructured understanding in specific environments. To address this, we present WorldLLM, a framework that enhances LLM-based world modeling by combining Bayesian inference and autonomous active exploration with reinforcement learning. WorldLLM leverages the in-context learning abilities of LLMs to guide an LLM-based world model's predictions using natural language hypotheses given in its prompt. These hypotheses are iteratively refined through a Bayesian inference framework that leverages a second LLM as the proposal distribution given collected evidence. This evidence is collected using a curiosity-driven reinforcement learning policy that explores the environment to find transitions with a low log-likelihood under our LLM-based predictive model using the current hypotheses. By alternating between refining hypotheses and collecting new evidence, our framework autonomously drives continual improvement of the predictions. Our experiments demonstrate the effectiveness of WorldLLM in a textual game environment that requires agents to manipulate and combine objects. The framework not only enhances predictive accuracy, but also generates human-interpretable theories of environment dynamics.
Authors Guillaume Levy, Cedric Colas, Pierre-Yves Oudeyer, Thomas Carta, Clement Romac
Categories Artificial Intelligence
2
MAESTRO: Multi-Agent Environment Shaping through Task and Reward Optimization
Title CN MAESTRO：通过任务与奖励优化实现多智能体环境塑造
Keywords 多智能体强化学习 · 环境塑造 · 课程学习 · 奖励设计 · 大语言模型
Summary 本文提出MAESTRO框架，利用大语言模型在离线状态下为多智能体强化学习系统生成语义课程和可执行的奖励函数，从而避免将LLM嵌入实时控制回路带来的高成本问题。该方法应用于大规模交通信号控制任务，在杭州16个路口的数据上验证了其在性能和稳定性上的提升。
Reason 该论文属于具身智能/机器人领域的多智能体强化学习研究，聚焦交通信号控制场景，使用LLM生成训练用的课程和奖励函数。尽管其利用LLM进行训练架构设计具有方法上的启发性，但应用场景与文档图像理解（DIU）无直接关联，且所解决的问题（如非稳态环境中的协作控制）与DIU核心痛点（如OCR幻觉、复杂排版理解、细粒度定位等）无关。无法迁移其核心技术到文档理解任务中。
Abstract Cooperative Multi-Agent Reinforcement Learning (MARL) faces two major design bottlenecks: crafting dense reward functions and constructing curricula that avoid local optima in high-dimensional, non-stationary environments. Existing approaches rely on fixed heuristics or use Large Language Models (LLMs) directly in the control loop, which is costly and unsuitable for real-time systems. We propose MAESTRO (Multi-Agent Environment Shaping through Task and Reward Optimization), a framework that moves the LLM outside the execution loop and uses it as an offline training architect. MAESTRO introduces two generative components: (i) a semantic curriculum generator that creates diverse, performance-driven traffic scenarios, and (ii) an automated reward synthesizer that produces executable Python reward functions adapted to evolving curriculum difficulty. These components guide a standard MARL backbone (MADDPG) without increasing inference cost at deployment. We evaluate MAESTRO on large-scale traffic signal control (Hangzhou, 16 intersections) and conduct controlled ablations. Results show that combining LLM-generated curricula with LLM-generated reward shaping yields improved performance and stability. Across four seeds, the full system achieves +4.0% higher mean return (163.26 vs. 156.93) and 2.2% better risk-adjusted performance (Sharpe 1.53 vs. 0.70) over a strong curriculum baseline. These findings highlight LLMs as effective high-level designers for cooperative MARL training.
Authors Boyuan Wu
Comment Preprint. 16 pages, 6 figures. Preliminary version; extended experiments and analysis forthcoming
Categories Artificial Intelligence
2
Local Entropy Search over Descent Sequences for Bayesian Optimization
Title CN 基于下降序列的局部熵搜索用于贝叶斯优化
Keywords 贝叶斯优化 · 局部搜索 · 下降序列 · 熵搜索 · 样本效率
Summary 本文提出了一种名为局部熵搜索（LES）的贝叶斯优化范式，旨在通过建模迭代优化器（如梯度下降）产生的下降序列来提高搜索效率。该方法将后验信念传播到优化路径上，并通过最大化与下降序列分布的互信息来选择下一个评估点。实验表明其在高复杂度合成目标和基准问题上具有良好的样本效率。
Reason 该论文属于纯算法优化与贝叶斯优化领域，研究如何在复杂设计空间中通过下降序列进行局部搜索。尽管其方法在样本效率上有提升，但完全不涉及多模态建模、视觉输入处理或文档理解任务。无法迁移至文档图像理解中的OCR幻觉、细粒度定位或长文档推理等核心问题，属于无关领域的优化方法。
Abstract Searching large and complex design spaces for a global optimum can be infeasible and unnecessary. A practical alternative is to iteratively refine the neighborhood of an initial design using local optimization methods such as gradient descent. We propose local entropy search (LES), a Bayesian optimization paradigm that explicitly targets the solutions reachable by the descent sequences of iterative optimizers. The algorithm propagates the posterior belief over the objective through the optimizer, resulting in a probability distribution over descent sequences. It then selects the next evaluation by maximizing mutual information with that distribution, using a combination of analytic entropy calculations and Monte-Carlo sampling of descent sequences. Empirical results on high-complexity synthetic objectives and benchmark problems show that LES achieves strong sample efficiency compared to existing local and global Bayesian optimization methods.
Authors David Stenger, Armin Lindicke, Alexander von Rohr, Sebastian Trimpe
Categories Artificial Intelligence
2
Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering Workshop on Multimodal Representation Learning for Healthcare (MMRL4H), EurIPS 2025
Title CN 大型视觉语言模型是否真正基于医学图像？来自意大利临床视觉问答的证据
Keywords 视觉语言模型 · 视觉接地 · 医学图像问答 · 幻觉 · 多模态评估
Summary 本文评估了多个前沿视觉语言模型在意大利语医学视觉问答任务中的表现，通过替换图像为空白来测试模型是否真正依赖视觉输入。结果显示GPT-4o对视觉信息依赖较强，而其他模型更倾向于使用文本先验进行猜测，并生成虚构的视觉解释。研究揭示了当前模型在临床部署前需更严格的视觉接地验证。
Reason 该论文属于典型的‘平行下游应用’，其核心是将现有VLM应用于意大利语医疗VQA任务，并评估模型在特定语言和领域下的视觉接地性。研究未提出任何新架构、训练方法或可迁移的技术组件，仅是对现成模型在小语种医疗场景中的行为分析，完全落入负面清单第1条（Wrapper/Application）和第3条（小语种：意大利语）。尽管涉及‘视觉幻觉’，但并未提出抑制机制或通用去幻觉方法，无法为文档图像理解提供方法论增益。
Abstract Large vision language models (VLMs) have achieved impressive performance on medical visual question answering benchmarks, yet their reliance on visual information remains unclear. We investigate whether frontier VLMs demonstrate genuine visual grounding when answering Italian medical questions by testing four state-of-the-art models: Claude Sonnet 4.5, GPT-4o, GPT-5-mini, and Gemini 2.0 flash exp. Using 60 questions from the EuropeMedQA Italian dataset that explicitly require image interpretation, we substitute correct medical images with blank placeholders to test whether models truly integrate visual and textual information. Our results reveal striking variability in visual dependency: GPT-4o shows the strongest visual grounding with a 27.9pp accuracy drop (83.2% [74.6%, 91.7%] to 55.3% [44.1%, 66.6%]), while GPT-5-mini, Gemini, and Claude maintain high accuracy with modest drops of 8.5pp, 2.4pp, and 5.6pp respectively. Analysis of model-generated reasoning reveals confident explanations for fabricated visual interpretations across all models, suggesting varying degrees of reliance on textual shortcuts versus genuine visual analysis. These findings highlight critical differences in model robustness and the need for rigorous evaluation before clinical deployment.
Authors Federico Felizzi, Olivia Riccomi, Michele Ferramola, Francesco Andrea Causio, Manuel Del Medico, Vittorio De Vita, Lorenzo De Mori, Alessandra Piscitelli Pietro Eric Risuleo, Bianca Destro Castaniti, Antonio Cristiano Alessia Longo, Luigi De Angelis, Mariapia Vassalli, Marcello Di Pumpo
Comment Accepted at the Workshop on Multimodal Representation Learning for Healthcare (MMRL4H), EurIPS 2025
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
Distributionally Robust Free Energy Principle for Decision-Making
Title CN 用于决策的分布鲁棒自由能原理
Keywords 自由能原理 · 鲁棒决策 · 自主代理 · 分布鲁棒性 · 人工智能理论
Summary 本文提出了一种分布鲁棒自由能模型（DR-FREE），通过将自由能原理与鲁棒优化结合，提升智能体在训练与环境不一致时的决策鲁棒性。该模型内置了对环境不确定性的容忍能力，在基准实验中表现出优于现有模型的稳定性。然而，其应用场景集中于通用智能体行为控制，远离文档理解或多模态语言模型的技术路径。
Reason 该论文属于具身智能与自主代理的理论决策框架，研究的是通用智能体在环境不一致下的鲁棒性问题，核心关注点为自由能原理的扩展，并非面向文档图像理解或视觉-语言任务。尽管‘鲁棒决策’在抽象层面上可能启发长文档多步推理中的稳定性设计，但其方法论与DIU的核心痛点（如OCR幻觉、文本定位、高分辨率处理）无直接关联，且未涉及任何视觉输入建模或多模态交互机制，无法迁移作为解决文档理解问题的有效工具。
Abstract Despite their groundbreaking performance, autonomous agents can misbehave when training and environmental conditions become inconsistent, with minor mismatches leading to undesirable behaviors or even catastrophic failures. Robustness towards these training-environment ambiguities is a core requirement for intelligent agents and its fulfillment is a long-standing challenge towards their real-world deployments. Here, we introduce a Distributionally Robust Free Energy model (DR-FREE) that instills this core property by design. Combining a robust extension of the free energy principle with a resolution engine, DR-FREE wires robustness into the agent decision-making mechanisms. Across benchmark experiments, DR-FREE enables the agents to complete the task even when, in contrast, state-of-the-art models fail. This milestone may inspire both deployments in multi-agent settings and, at a perhaps deeper level, the quest for an explanation of how natural agents -- with little or no training -- survive in capricious environments.
Authors Allahkaram Shafiei, Hozefa Jesawada, Karl Friston, Giovanni Russo
Comment Contains main text and supplementary information. Supplementary movie is at the paper repository
Categories Artificial Intelligence
2
Information Physics of Intelligence: Unifying Logical Depth and Entropy under Thermodynamic Constraints
Title CN 智能的信息物理学：在热力学约束下统一逻辑深度与熵
Keywords 信息物理学 · 推导熵 · 热力学约束 · 逻辑深度 · 生成式计算
Summary 本文提出了一个信息处理的物理框架，引入‘推导熵’来量化从逻辑深度生成目标状态所需的计算功，并分析了香农熵与计算复杂度之间的相变点。研究表明，在低于某一阈值时记忆检索更优，高于该阈值时生成式计算更高效，从而为节能AI设计提供了理论边界。但全文未涉及视觉或文档模态。
Reason 该论文属于理论人工智能的抽象建模，提出了一种基于热力学的信息处理框架和‘推导熵’概念，试图解释生成式计算与记忆检索之间的权衡。然而，其内容高度理论化，未涉及任何多模态、视觉或文档理解相关任务，也未提供可迁移至文档图像理解（如OCR去幻觉、细粒度定位、复杂排版处理）的具体技术路径。属于通用AI理论探讨，对DIU领域无直接方法论贡献。
Abstract The rapid scaling of artificial intelligence models has revealed a fundamental tension between model capacity (storage) and inference efficiency (computation). While classical information theory focuses on transmission and storage limits, it lacks a unified physical framework to quantify the thermodynamic costs of generating information from compressed laws versus retrieving it from memory. In this paper, we propose a theoretical framework that treats information processing as an enabling mapping from ontological states to carrier states. We introduce a novel metric, Derivation Entropy, which quantifies the effective work required to compute a target state from a given logical depth. By analyzing the interplay between Shannon entropy (storage) and computational complexity (time/energy), we demonstrate the existence of a critical phase transition point. Below this threshold, memory retrieval is thermodynamically favorable; above it, generative computation becomes the optimal strategy. This "Energy-Time-Space" conservation law provides a physical explanation for the efficiency of generative models and offers a rigorous mathematical bound for designing next-generation, energy-efficient AI architectures. Our findings suggest that the minimization of Derivation Entropy is a governing principle for the evolution of both biological and artificial intelligence.
Authors Jianfeng Xu, Zeyan Li
Categories Artificial Intelligence
2
From Pixels to Posts: Retrieval-Augmented Fashion Captioning and Hashtag Generation Expert Systems with Applications
Title CN 从像素到帖子：基于检索增强的时尚描述与话题标签生成
Keywords RAG · 时尚图像描述 · 属性推理 · LLM · 检索增强生成
Summary 本文提出了一种基于检索增强生成（RAG）的时尚图像描述与话题标签生成框架，通过目标检测、聚类和CLIP-FAISS检索获取服装属性，并结合检索到的风格样例引导LLM生成更具事实一致性和风格多样性的文本。相比BLIP基线，该方法在属性覆盖率和生成文本多样性上表现更优。
Reason 该论文属于典型的‘平行下游应用’，即将RAG框架应用于时尚领域的图文生成任务。尽管其使用了YOLO、CLIP和LLM等技术模块，并声称减少了幻觉，但整体架构是标准的RAG+LLM流水线，未提出任何针对文档图像理解（DIU）的核心方法创新。其解决的‘属性保真’和‘领域泛化’问题虽与DIU部分相关，但方法不具备向DocVQA、表格识别、复杂版式分析等核心DIU任务迁移的通用性或底层机制突破。
Abstract This paper introduces the retrieval-augmented framework for automatic fashion caption and hashtag generation, combining multi-garment detection, attribute reasoning, and Large Language Model (LLM) prompting. The system aims to produce visually grounded, descriptive, and stylistically interesting text for fashion imagery, overcoming the limitations of end-to-end captioners that have problems with attribute fidelity and domain generalization. The pipeline combines a YOLO-based detector for multi-garment localization, k-means clustering for dominant color extraction, and a CLIP-FAISS retrieval module for fabric and gender attribute inference based on a structured product index. These attributes, together with retrieved style examples, create a factual evidence pack that is used to guide an LLM to generate human-like captions and contextually rich hashtags. A fine-tuned BLIP model is used as a supervised baseline model for comparison. Experimental results show that the YOLO detector is able to obtain a mean Average Precision (mAP@0.5) of 0.71 for nine categories of garments. The RAG-LLM pipeline generates expressive attribute-aligned captions and achieves mean attribute coverage of 0.80 with full coverage at the 50% threshold in hashtag generation, whereas BLIP gives higher lexical overlap and lower generalization. The retrieval-augmented approach exhibits better factual grounding, less hallucination, and great potential for scalable deployment in various clothing domains. These results demonstrate the use of retrieval-augmented generation as an effective and interpretable paradigm for automated and visually grounded fashion content generation.
Authors Moazzam Umer Gondal, Hamad Ul Qudous, Daniya Siddiqui, Asma Ahmad Farhan
Comment Submitted to Expert Systems with Applications
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
2
EnfoPath: Energy-Informed Analysis of Generative Trajectories in Flow Matching EurIPS 2025 Workshop on Principles of Generative Modeling (PriGM)
Title CN EnfoPath：基于能量感知的流匹配生成轨迹分析
Keywords 流匹配 · 生成轨迹 · 动能路径能量 · 生成模型分析 · EurIPS workshop
Summary 本文提出一种基于经典力学的动能路径能量（KPE）指标，用于分析流匹配生成模型在CIFAR-10和ImageNet上的生成轨迹。实验发现高KPE路径倾向于生成语义更丰富但处于低密度区域的样本，揭示了生成难度与样本信息量之间的关系。这是一种生成过程的诊断工具，而非面向文档理解的任务方法。
Reason 该论文研究的是通用生成模型（特别是基于ODE的流模型）的采样轨迹动力学特性，提出用动能路径能量（KPE）来分析生成样本的质量与数据分布的关系。尽管其物理启发的分析框架具有理论美感，但完全聚焦于图像生成质量评估，并未涉及多模态输入、文档结构、文本理解或视觉-语言对齐等文档理解核心问题。属于纯生成模型的机理分析，与DIU任务如DocVQA、表格识别、布局分析等无直接关联，也无法迁移用于解决OCR幻觉、细粒度定位或长文档推理等痛点。
Abstract Flow-based generative models synthesize data by integrating a learned velocity field from a reference distribution to the target data distribution. Prior work has focused on endpoint metrics (e.g., fidelity, likelihood, perceptual quality) while overlooking a deeper question: what do the sampling trajectories reveal? Motivated by classical mechanics, we introduce kinetic path energy (KPE), a simple yet powerful diagnostic that quantifies the total kinetic effort along each generation path of ODE-based samplers. Through comprehensive experiments on CIFAR-10 and ImageNet-256, we uncover two key phenomena: ({i}) higher KPE predicts stronger semantic quality, indicating that semantically richer samples require greater kinetic effort, and ({ii}) higher KPE inversely correlates with data density, with informative samples residing in sparse, low-density regions. Together, these findings reveal that semantically informative samples naturally reside on the sparse frontier of the data distribution, demanding greater generative effort. Our results suggest that trajectory-level analysis offers a physics-inspired and interpretable framework for understanding generation difficulty and sample characteristics.
Authors Ziyun Li, Ben Dai, Huancheng Hu, Henrik Boström, Soon Hoe Lim
Comment EurIPS 2025 Workshop on Principles of Generative Modeling (PriGM)
Categories Artificial Intelligence
2
Preventing Shortcut Learning in Medical Image Analysis through Intermediate Layer Knowledge Distillation from Specialist Teachers Journal of Machine Learning for Biomedical Imaging (MELBA)
Title CN 通过中间层知识蒸馏防止医学图像分析中的捷径学习
Keywords 知识蒸馏 · 捷径学习 · 医学图像分析 · 去偏 · 中间层监督
Summary 本文提出一种基于中间层知识蒸馏的新框架，利用在小规模任务相关数据上微调的教师网络来抑制学生网络中的捷径学习。实验表明该方法在多个医学图像数据集上优于传统经验风险最小化、基于增强和分组的去偏方法，并在分布外测试中表现稳健。
Reason 该论文属于医学图像分析领域的偏置缓解工作，虽然提出了中间层知识蒸馏的创新框架，但其应用场景和问题设定（如CheXpert、ISIC等医学影像）与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、细粒度文本定位）无直接关联。方法聚焦于消除医学图像中的统计偏见和捷径特征，难以迁移到文档场景中处理文本结构、多模态对齐或视觉token压缩等问题。
Abstract Deep learning models are prone to learning shortcut solutions to problems using spuriously correlated yet irrelevant features of their training data. In high-risk applications such as medical image analysis, this phenomenon may prevent models from using clinically meaningful features when making predictions, potentially leading to poor robustness and harm to patients. We demonstrate that different types of shortcuts (those that are diffuse and spread throughout the image, as well as those that are localized to specific areas) manifest distinctly across network layers and can, therefore, be more effectively targeted through mitigation strategies that target the intermediate layers. We propose a novel knowledge distillation framework that leverages a teacher network fine-tuned on a small subset of task-relevant data to mitigate shortcut learning in a student network trained on a large dataset corrupted with a bias feature. Through extensive experiments on CheXpert, ISIC 2017, and SimBA datasets using various architectures (ResNet-18, AlexNet, DenseNet-121, and 3D CNNs), we demonstrate consistent improvements over traditional Empirical Risk Minimization, augmentation-based bias-mitigation, and group-based bias-mitigation approaches. In many cases, we achieve comparable performance with a baseline model trained on bias-free data, even on out-of-distribution test data. Our results demonstrate the practical applicability of our approach to real-world medical imaging scenarios where bias annotations are limited and shortcut features are difficult to identify a priori.
Authors Christopher Boland, Sotirios Tsaftaris, Sonia Dahdouh
Comment Accepted for publication at the Journal of Machine Learning for Biomedical Imaging (MELBA) https://melba-journal.org/2025:020
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
Agent-OM: Leveraging LLM Agents for Ontology Matching
Title CN Agent-OM：利用大语言模型代理进行本体匹配
Keywords 本体匹配 · LLM代理 · 知识工程 · 语义对齐 · Agent框架
Summary 本文提出了一种基于LLM代理的本体匹配框架Agent-OM，采用双Siamese代理结构和专用工具集，在OAEI数据集上验证了其在复杂和少样本任务中的有效性。然而，该工作聚焦于知识工程中的语义对齐问题，属于特定领域的应用系统，缺乏对多模态输入、视觉布局建模或文档结构理解的支持。
Reason 该论文属于典型的‘平行下游应用’，将LLM Agent应用于本体匹配（Ontology Matching）这一特定任务，尽管使用了Agent架构，但其核心是领域应用而非方法论创新。未提出可迁移至文档图像理解（如OCR去幻觉、复杂排版处理、细粒度定位等）的通用技术。本体匹配与DIU任务差异较大，方法难以迁移到文档理解场景。
Abstract Ontology matching (OM) enables semantic interoperability between different ontologies and resolves their conceptual heterogeneity by aligning related entities. OM systems currently have two prevailing design paradigms: conventional knowledge-based expert systems and newer machine learning-based predictive systems. While large language models (LLMs) and LLM agents have revolutionised data engineering and have been applied creatively in many domains, their potential for OM remains underexplored. This study introduces a novel agent-powered LLM-based design paradigm for OM systems. With consideration of several specific challenges in leveraging LLM agents for OM, we propose a generic framework, namely Agent-OM (Agent for Ontology Matching), consisting of two Siamese agents for retrieval and matching, with a set of OM tools. Our framework is implemented in a proof-of-concept system. Evaluations of three Ontology Alignment Evaluation Initiative (OAEI) tracks over state-of-the-art OM systems show that our system can achieve results very close to the long-standing best performance on simple OM tasks and can significantly improve the performance on complex and few-shot OM tasks.
Authors Zhangcheng Qiang, Weiqing Wang, Kerry Taylor
Comment 31 pages
Categories Artificial Intelligence, Computation and Language
2
Look It Up: Analysing Internal Web Search Capabilities of Modern LLMs
Title CN 查一下：分析现代大语言模型的内部网络搜索能力
Keywords 大语言模型 · 网络搜索 · 信心校准 · 事实准确性 · 查询失败
Summary 本文构建了一个基准来评估现代大语言模型在面对时效性问题时是否合理调用网络搜索功能。实验发现，尽管搜索能提升准确率，但模型常因信心校准不佳而过度自信或错误跳过检索，且初始查询失败后难以恢复。整体上，内置搜索更适合作为低延迟验证层，而非可靠的分析工具。
Reason 该论文研究的是通用LLM在回答问题时调用网络搜索的行为分析，属于典型的下游应用行为评估，且聚焦于开放域问答中的检索决策机制。虽然涉及‘何时使用外部信息’这一推理控制问题，但其方法论并未提出可迁移至文档图像理解（DIU）的新架构、表示学习或去幻觉机制。任务场景与DIU核心痛点（如OCR幻觉、复杂排版解析、细粒度视觉定位）无直接关联，属于平行应用层面的分析，不具备对文档理解的技术推动价值。
Abstract Modern large language models integrate web search to provide real-time answers, yet it remains unclear whether they are efficiently calibrated to use search when it is actually needed. We introduce a benchmark evaluating both the necessity and effectiveness of web access across commercial models with no access to internal states or parameters. The dataset includes a static split of 783 temporally anchored questions answerable from pre-cutoff knowledge, aimed at testing whether models invoke search based on low internal confidence, and a dynamic split of 288 post-cutoff queries designed to test whether models recognise when search is required and retrieve updated information. Web access substantially improves static accuracy for GPT-5-mini and Claude Haiku 4.5, though confidence calibration worsens. On dynamic queries, both models frequently invoke search yet remain below 70 percent accuracy due to weak query formulation. Costs per accuracy-improving call remain low, but returns diminish once initial retrieval fails. Selective invocation helps, but models become overconfident and inconsistent after search. Overall, built-in web search meaningfully improves factual accuracy and can be invoked selectively, yet models remain overconfident, skip retrieval when it is essential, and falter once initial search queries underperform. Taken together, internal web search works better as a good low-latency verification layer than a reliable analytical tool, with clear room for improvement.
Authors Sahil Kale
Comment 10 pages, 8 figures
Categories Artificial Intelligence, Computation and Language
2
Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translation
Title CN 超越SELECT：一种面向真实世界文本到SQL转换的全面分类引导基准
Keywords Text-to-SQL · 数据集合成 · 分类体系 · 大语言模型 · 基准测试
Summary 本文提出了一种新的分类体系来评估和生成Text-to-SQL数据集，揭示了现有数据集在覆盖范围和多样性上的不足，并通过LLM合成了更具代表性的新数据集SQL-Synth。实验表明当前大模型在此类复杂场景下表现有限，但可通过微调提升性能。
Reason 该论文聚焦于Text-to-SQL任务，属于典型的下游应用型研究，其核心贡献是构建了一个基于分类体系的数据集SQL-Synth，并未提出可用于文档图像理解（DIU）领域的底层方法或架构创新。尽管使用了LLM进行数据合成，但整体工作属于对特定任务（数据库查询）的平行扩展，不涉及OCR、视觉布局分析、多模态对齐或高分辨率文档处理等DIU核心技术痛点，无法迁移用于解决文档理解中的幻觉、密集文本或复杂排版问题。
Abstract Text-to-SQL datasets are essential for training and evaluating text-to-SQL models, but existing datasets often suffer from limited coverage and fail to capture the diversity of real-world applications. To address this, we propose a novel taxonomy for text-to-SQL classification based on dimensions including core intents, statement types, syntax structures, and key actions. Using this taxonomy, we evaluate widely used public text-to-SQL datasets (e.g., Spider and Bird) and reveal limitations in their coverage and diversity. We then introduce a taxonomy-guided dataset synthesis pipeline, yielding a new dataset named SQL-Synth. This approach combines the taxonomy with Large Language Models (LLMs) to ensure the dataset reflects the breadth and complexity of real-world text-to-SQL applications. Extensive analysis and experimental results validate the effectiveness of our taxonomy, as SQL-Synth exhibits greater diversity and coverage compared to existing benchmarks. Moreover, we uncover that existing LLMs typically fall short in adequately capturing the full range of scenarios, resulting in limited performance on SQL-Synth. However, fine-tuning can substantially improve their performance in these scenarios. The proposed taxonomy has significant potential impact, as it not only enables comprehensive analysis of datasets and the performance of different LLMs, but also guides the construction of training data for LLMs.
Authors Hao Wang, Yuanfeng Song, Xiaoming Yin, Xing Chen
Categories Artificial Intelligence, Computation and Language
2
DataSage: Multi-agent Collaboration for Insight Discovery with External Knowledge Retrieval, Multi-role Debating, and Multi-path Reasoning
Title CN DataSage：通过外部知识检索、多角色辩论和多路径推理实现多智能体协作的洞察发现
Keywords 多智能体系统 · 洞察发现 · 知识检索 · 辩论机制 · 代码生成
Summary 本文提出DataSage，一种基于大语言模型的多智能体框架，用于自动化数据洞察发现。该框架引入外部知识检索以丰富分析背景，设计多角色辩论机制以模拟不同分析视角并加深分析深度，并采用多路径推理提升生成代码与洞察的准确性。实验表明，DataSage在InsightBench数据集上优于现有方法。然而，该工作聚焦于结构化或半结构化数据的分析流程自动化，并未处理原始文档图像、扫描件或复杂排版内容的理解问题。
Reason 该论文属于典型的‘平行下游应用’，其核心是构建一个面向数据分析与洞察发现的多智能体系统，应用场景为通用数据洞察（如商业决策），并未针对文档图像理解中的任何具体任务（如DocVQA、表格识别、布局分析等）提出方法论创新。尽管其使用的多智能体辩论、知识检索和多路径推理机制在理论上可能对长文档推理有启发，但论文本身未涉及视觉输入、OCR、版式理解或视觉-语言对齐等DIU核心问题，属于将已有LLM Agent技术应用于特定场景的Wrapper式工作，无底层技术创新。
Abstract In today's data-driven era, fully automated end-to-end data analytics, particularly insight discovery, is critical for discovering actionable insights that assist organizations in making effective decisions. With the rapid advancement of large language models (LLMs), LLM-driven agents have emerged as a promising paradigm for automating data analysis and insight discovery. However, existing data insight agents remain limited in several key aspects, often failing to deliver satisfactory results due to: (1) insufficient utilization of domain knowledge, (2) shallow analytical depth, and (3) error-prone code generation during insight generation. To address these issues, we propose DataSage, a novel multi-agent framework that incorporates three innovative features including external knowledge retrieval to enrich the analytical context, a multi-role debating mechanism to simulate diverse analytical perspectives and deepen analytical depth, and multi-path reasoning to improve the accuracy of the generated code and insights. Extensive experiments on InsightBench demonstrate that DataSage consistently outperforms existing data insight agents across all difficulty levels, offering an effective solution for automated data insight discovery.
Authors Xiaochuan Liu, Yuanfeng Song, Xiaoming Yin, Xing Chen
Categories Artificial Intelligence, Computation and Language
2
Time Travel: LLM-Assisted Semantic Behavior Localization with Git Bisect submitted to Git Bisect SCALCOM 2025 Calgary (to be published)
Title CN 时间旅行：基于大语言模型辅助的语义行为定位与Git二分法
Keywords 大语言模型 · Git bisect · 语义故障定位 · 链式思维 · 软件调试
Summary 本文提出一种结合大语言模型与Git bisect的技术，用于在存在噪声测试和非单调回归的情况下进行语义级故障定位。通过链式思维推理增强提交历史的遍历分析，并对DeepSeekCoderV2进行QLoRA微调，在多个开源项目中实现了成功率提升和平均调试时间减半。
Reason 该论文将LLM应用于软件工程中的Git bisect过程，解决的是代码版本调试和故障定位问题，属于‘平行下游应用’。尽管使用了CoT推理和弱监督微调等技术，但其核心场景与文档图像理解（DIU）无关，无法迁移用于解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract We present a novel framework that integrates Large Language Models (LLMs) into the Git bisect process for semantic fault localization. Traditional bisect assumes deterministic predicates and binary failure states assumptions often violated in modern software development due to flaky tests, nonmonotonic regressions, and semantic divergence from upstream repositories. Our system augments bisect traversal with structured chain of thought reasoning, enabling commit by commit analysis under noisy conditions. We evaluate multiple open source and proprietary LLMs for their suitability and fine tune DeepSeekCoderV2 using QLoRA on a curated dataset of semantically labeled diffs. We adopt a weak supervision workflow to reduce annotation overhead, incorporating human in the loop corrections and self consistency filtering. Experiments across multiple open source projects show a 6.4 point absolute gain in success rate from 74.2 to 80.6 percent, leading to significantly fewer failed traversals and by experiment up to 2x reduction in average bisect time. We conclude with discussions on temporal reasoning, prompt design, and finetuning strategies tailored for commit level behavior analysis.
Authors Yujing Wang, Weize Hong
Comment submitted to Git Bisect SCALCOM 2025 Calgary (to be published)
Categories Artificial Intelligence
2
Addressing Situated Teaching Needs: A Multi-Agent Framework for Automated Slide Adaptation
Title CN 解决情境化教学需求：一种用于自动幻灯片适配的多智能体框架
Keywords 多智能体系统 · 教学自动化 · 幻灯片适配 · 教育技术 · AI辅助教学
Summary 本文通过访谈教师识别出教学幻灯片适配过程中的痛点，并提出一个基于多智能体的自动化框架，根据教师的高层指令修改幻灯片内容。实验表明该系统在意图对齐、内容连贯性和事实准确性方面表现良好，且与人类专家有较高的操作一致性。然而，该工作聚焦于教育场景的应用逻辑，而非文档理解的基础技术突破。
Reason 该论文属于典型的‘平行下游应用’，将多智能体框架应用于教育场景中的幻灯片改编任务。尽管涉及文档形式（幻灯片），但其核心创新在于教育技术流程自动化，而非文档图像理解（DIU）本身的方法论改进。未针对OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点提出可迁移的技术方案，属于在特定领域对现有Agent技术的应用，无底层方法创新。
Abstract The adaptation of teaching slides to instructors' situated teaching needs, including pedagogical styles and their students' context, is a critical yet time-consuming task for educators. Through a series of educator interviews, we first identify and systematically categorize the key friction points that impede this adaptation process. Grounded in these findings, we introduce a novel multi-agent framework designed to automate slide adaptation based on high-level instructor specifications. An evaluation involving 16 modification requests across 8 real-world courses validates our approach. The framework's output consistently achieved high scores in intent alignment, content coherence and factual accuracy, and performed on par with baseline methods regarding visual clarity, while also demonstrating appropriate timeliness and a high operational agreement with human experts, achieving an F1 score of 0.89. This work heralds a new paradigm where AI agents handle the logistical burdens of instructional design, liberating educators to focus on the creative and strategic aspects of teaching.
Authors Binglin Liu, Yucheng Wang, Zheyuan Zhang, Jiyuan Lu, Shen Yang, Daniel Zhang-Li, Huiqin Liu, Jifan Yu
Categories Artificial Intelligence
2
RhinoInsight: Improving Deep Research through Control Mechanisms for Model Behavior and Context
Title CN RhinoInsight：通过控制机制改进模型行为与上下文以提升深度研究
Keywords 大语言模型 · 代理框架 · 幻觉抑制 · 上下文管理 · 深度研究
Summary 本文提出了RhinoInsight，一种用于增强大语言模型在深度研究中表现的框架，通过引入可验证的检查清单模块和证据审计模块来控制模型行为与上下文管理，提升推理的可追溯性和结果质量。实验表明其在深度研究任务上达到SOTA水平。
Reason 该论文属于典型的‘平行应用’类工作，聚焦于构建用于深度研究的LLM代理框架，其核心贡献是面向通用研究场景的流程控制机制（如检查清单和证据审计），并未针对文档图像理解（DIU）中的关键挑战（如OCR幻觉、复杂版面解析、细粒度定位或高分辨率视觉输入处理）提出可迁移的方法。尽管其‘减少幻觉’和‘上下文剪枝’机制在理论上对VLM有启发，但缺乏对多模态输入的支持，且未涉及视觉信息处理，无法直接迁移到DocVQA或文档理解任务中。
Abstract Large language models are evolving from single-turn responders into tool-using agents capable of sustained reasoning and decision-making for deep research. Prevailing systems adopt a linear pipeline of plan to search to write to a report, which suffers from error accumulation and context rot due to the lack of explicit control over both model behavior and context. We introduce RhinoInsight, a deep research framework that adds two control mechanisms to enhance robustness, traceability, and overall quality without parameter updates. First, a Verifiable Checklist module transforms user requirements into traceable and verifiable sub-goals, incorporates human or LLM critics for refinement, and compiles a hierarchical outline to anchor subsequent actions and prevent non-executable planning. Second, an Evidence Audit module structures search content, iteratively updates the outline, and prunes noisy context, while a critic ranks and binds high-quality evidence to drafted content to ensure verifiability and reduce hallucinations. Our experiments demonstrate that RhinoInsight achieves state-of-the-art performance on deep research tasks while remaining competitive on deep search tasks.
Authors Yu Lei, Shuzheng Si, Wei Wang, Yifei Wu, Gang Chen, Fanchao Qi, Maosong Sun
Categories Artificial Intelligence, Computation and Language
2
Ellipsoid-Based Decision Boundaries for Open Intent Classification
Title CN 基于椭球体决策边界的开放意图分类
Keywords 开放意图分类 · 椭球体边界 · 监督对比学习 · 文本分类 · 异常检测
Summary 本文提出EliDecide方法，通过学习可学习的椭球体决策边界来改进开放意图检测，利用监督对比学习构建判别性特征空间，并设计双损失函数优化边界，以更好地区分已知和未知意图。在多个文本分类基准上达到SOTA。
Reason 该论文研究的是文本模态下的开放意图分类问题，属于纯NLP任务，且未涉及任何视觉信息或文档结构理解。其提出的椭球体决策边界方法虽在文本分类中有创新，但与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版、多模态对齐等无直接关联，无法迁移用于解决文档理解中的视觉-语言联合建模问题。
Abstract Textual open intent classification is crucial for real-world dialogue systems, enabling robust detection of unknown user intents without prior knowledge and contributing to the robustness of the system. While adaptive decision boundary methods have shown great potential by eliminating manual threshold tuning, existing approaches assume isotropic distributions of known classes, restricting boundaries to balls and overlooking distributional variance along different directions. To address this limitation, we propose EliDecide, a novel method that learns ellipsoid decision boundaries with varying scales along different feature directions. First, we employ supervised contrastive learning to obtain a discriminative feature space for known samples. Second, we apply learnable matrices to parameterize ellipsoids as the boundaries of each known class, offering greater flexibility than spherical boundaries defined solely by centers and radii. Third, we optimize the boundaries via a novelly designed dual loss function that balances empirical and open-space risks: expanding boundaries to cover known samples while contracting them against synthesized pseudo-open samples. Our method achieves state-of-the-art performance on multiple text intent benchmarks and further on a question classification dataset. The flexibility of the ellipsoids demonstrates superior open intent detection capability and strong potential for generalization to more text classification tasks in diverse complex open-world scenarios.
Authors Yuetian Zou, Hanlei Zhang, Hua Xu, Songze Li, Long Xiao
Categories Artificial Intelligence, Computation and Language
2
HuggingR$^{4}$: A Progressive Reasoning Framework for Discovering Optimal Model Companions
Title CN HuggingR⁴：一种用于发现最优模型伙伴的渐进式推理框架
Keywords LLM Agent · Model Selection · Reasoning Framework · Multimodal Retrieval · HuggingFace
Summary 本文提出HuggingR⁴框架，通过推理、检索、精炼和反思四个阶段，帮助大语言模型高效地从海量社区模型中挑选合适的模型伙伴。该方法减少提示词膨胀，提升模型选择的合理性和可用性，并在自建数据集上验证了效果。
Reason 该论文属于典型的‘平行下游应用’，其核心是构建一个LLM代理来从HuggingFace等平台选择多模态AI模型，解决的是模型检索与组合问题，而非文档图像理解（DIU）中的任何底层技术挑战。尽管其推理-检索-精炼-反思框架具有一定系统设计价值，但该方法应用于模型服务编排场景，与DIU中的OCR幻觉、密集文本识别、复杂版面分析或长文档推理无直接关联。无法迁移为提升文档理解性能的底层工具。
Abstract Large Language Models (LLMs) have made remarkable progress in their ability to interact with external interfaces. Selecting reasonable external interfaces has thus become a crucial step in constructing LLM agents. In contrast to invoking API tools, directly calling AI models across different modalities from the community (e.g., HuggingFace) poses challenges due to the vast scale (> 10k), metadata gaps, and unstructured descriptions. Current methods for model selection often involve incorporating entire model descriptions into prompts, resulting in prompt bloat, wastage of tokens and limited scalability. To address these issues, we propose HuggingR$^4$, a novel framework that combines Reasoning, Retrieval, Refinement, and Reflection, to efficiently select models. Specifically, We first perform multiple rounds of reasoning and retrieval to get a coarse list of candidate models. Then, we conduct fine-grained refinement by analyzing candidate model descriptions, followed by reflection to assess results and determine if retrieval scope expansion is necessary. This method reduces token consumption considerably by decoupling user query processing from complex model description handling. Through a pre-established vector database, complex model descriptions are stored externally and retrieved on-demand, allowing the LLM to concentrate on interpreting user intent while accessing only relevant candidate models without prompt bloat. In the absence of standardized benchmarks, we construct a multimodal human-annotated dataset comprising 14,399 user requests across 37 tasks and conduct a thorough evaluation. HuggingR$^4$ attains a workability rate of 92.03% and a reasonability rate of 82.46%, surpassing existing method by 26.51% and 33.25% respectively on GPT-4o-mini.
Authors Shaoyin Ma, Jie Song, Huiqiong Wang, Li Sun, Mingli Song
Comment 19 pages, 4 figures
Categories Artificial Intelligence
2
MAGMA-Edu: Multi-Agent Generative Multimodal Framework for Text-Diagram Educational Question Generation
Title CN MAGMA-Edu：用于文本-图表教育问题生成的多智能体生成式多模态框架
Keywords 多智能体系统 · 教育内容生成 · 图文一致性 · 自反思机制 · 代码中间表示
Summary 本文提出MAGMA-Edu，一个用于生成教育类图文问题的多智能体框架，通过文本与图表的协同演化和自反思机制提升生成内容的教学一致性与图文对齐度。实验显示在教育基准上显著优于GPT-4o等模型。然而，该工作聚焦于教育内容生成这一特定应用，属于下游任务封装，不具备向通用文档理解迁移的技术价值。
Reason 该论文属于典型的‘平行下游应用’，其核心是将多智能体框架应用于教育领域的图文题目生成，任务场景为特定垂直领域的内容创作。尽管提出了自反思和代码中间表示等机制，但整体架构服务于教育内容生成这一应用场景，未针对文档图像理解中的核心痛点（如OCR幻觉、复杂版面分析、长文档推理）提出可迁移的方法论创新。方法上无视觉压缩、细粒度定位或高分辨率处理等方面的底层改进，对DIU领域技术推进无直接贡献。
Abstract Educational illustrations play a central role in communicating abstract concepts, yet current multimodal large language models (MLLMs) remain limited in producing pedagogically coherent and semantically consistent educational visuals. We introduce MAGMA-Edu, a self-reflective multi-agent framework that unifies textual reasoning and diagrammatic synthesis for structured educational problem generation. Unlike existing methods that treat text and image generation independently, MAGMA-Edu employs a two-stage co-evolutionary pipeline: (1) a generation-verification-reflection loop that iteratively refines question statements and solutions for mathematical accuracy, and (2) a code-based intermediate representation that enforces geometric fidelity and semantic alignment during image rendering. Both stages are guided by internal self-reflection modules that evaluate and revise outputs until domain-specific pedagogical constraints are met. Extensive experiments on multimodal educational benchmarks demonstrate the superiority of MAGMA-Edu over state-of-the-art MLLMs. Compared to GPT-4o, MAGMA-Edu improves the average textual metric from 57.01 to 92.31 (+35.3 pp) and boosts image-text consistency (ITC) from 13.20 to 85.24 (+72 pp). Across all model backbones, MAGMA-Edu achieves the highest scores (Avg-Text 96.20, ITC 99.12), establishing a new state of the art for multimodal educational content generation and demonstrating the effectiveness of self-reflective multi-agent collaboration in pedagogically aligned vision-language reasoning.
Authors Zhenyu Wu, Jian Li, Hua Huang
Categories Artificial Intelligence
2
ObjectAlign: Neuro-Symbolic Object Consistency Verification and Correction
Title CN ObjectAlign：神经符号化对象一致性验证与校正
Keywords 视频修复 · 神经符号系统 · 对象一致性 · 形式化验证 · 时序逻辑
Summary 本文提出ObjectAlign，一个用于检测和修复编辑视频中对象不一致性的神经符号框架。通过学习感知指标阈值，并结合SMT形式化验证与概率时序逻辑检查，实现对对象身份和时序一致性的双重约束。对于异常帧段，采用基于神经网络的插值进行自适应修复。实验表明在DAVIS和Pexels数据集上优于现有方法。
Reason 该论文聚焦于视频编辑与合成中的对象不一致问题（如帧闪烁、身份漂移），提出一种结合感知指标与符号推理的修复框架。其核心任务为视频质量增强，属于视频理解/生成领域，与文档图像理解（DIU）或DocVQA无直接关联。尽管其神经符号方法在理论上具有一定通用性，但未涉及OCR、文本布局、视觉语言推理等文档理解核心问题，且应用场景完全偏离文档场景。因此属于‘无关领域’，符合负面清单第2条。
Abstract Video editing and synthesis often introduce object inconsistencies, such as frame flicker and identity drift that degrade perceptual quality. To address these issues, we introduce ObjectAlign, a novel framework that seamlessly blends perceptual metrics with symbolic reasoning to detect, verify, and correct object-level and temporal inconsistencies in edited video sequences. The novel contributions of ObjectAlign are as follows: First, we propose learnable thresholds for metrics characterizing object consistency (i.e. CLIP-based semantic similarity, LPIPS perceptual distance, histogram correlation, and SAM-derived object-mask IoU). Second, we introduce a neuro-symbolic verifier that combines two components: (a) a formal, SMT-based check that operates on masked object embeddings to provably guarantee that object identity does not drift, and (b) a temporal fidelity check that uses a probabilistic model checker to verify the video's formal representation against a temporal logic specification. A frame transition is subsequently deemed "consistent" based on a single logical assertion that requires satisfying both the learned metric thresholds and this unified neuro-symbolic constraint, ensuring both low-level stability and high-level temporal correctness. Finally, for each contiguous block of flagged frames, we propose a neural network based interpolation for adaptive frame repair, dynamically choosing the interpolation depth based on the number of frames to be corrected. This enables reconstruction of the corrupted frames from the last valid and next valid keyframes. Our results show up to 1.4 point improvement in CLIP Score and up to 6.1 point improvement in warp error compared to SOTA baselines on the DAVIS and Pexels video datasets.
Authors Mustafa Munir, Harsh Goel, Xiwen Wei, Minkyu Choi, Sahil Shah, Kartikeya Bhardwaj, Paul Whatmough, Sandeep Chinchali, Radu Marculescu
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
KANO: Kolmogorov-Arnold Neural Operator
Title CN KANO：Kolmogorov-Arnold神经算子
Keywords 神经算子 · 偏微分方程 · 频谱方法 · 符号可解释性 · KANO
Summary 本文提出了一种名为KANO的新型神经算子，结合频谱和空间基函数来建模位置相关的动力学系统，理论上克服了傅里叶神经算子（FNO）在处理变系数PDE时的局限性，并在量子哈密顿量学习任务中实现了高精度的符号化重建和极低的状态失真度。
Reason 该论文属于物理建模与微分方程求解领域的基础方法研究，核心是针对偏微分方程（PDE）的神经算子设计，强调符号可解释性和频谱-空间双域建模。尽管其在表达能力和泛化性上优于FNO，但研究目标与文档图像理解（DIU）任务无直接关联。未涉及多模态建模、视觉表示学习、OCR、布局分析或VQA等DIU核心问题，也无法迁移用于解决OCR幻觉、复杂排版解析或长文档推理等痛点。
Abstract We introduce Kolmogorov--Arnold Neural Operator (KANO), a dual-domain neural operator jointly parameterized by both spectral and spatial bases with intrinsic symbolic interpretability. We theoretically demonstrate that KANO overcomes the pure-spectral bottleneck of Fourier Neural Operator (FNO): KANO remains expressive over generic position-dependent dynamics (variable coefficient PDEs) for any physical input, whereas FNO stays practical only for spectrally sparse operators and strictly imposes a fast-decaying input Fourier tail. We verify our claims empirically on position-dependent differential operators, for which KANO robustly generalizes but FNO fails to. In the quantum Hamiltonian learning benchmark, KANO reconstructs ground-truth Hamiltonians in closed-form symbolic representations accurate to the fourth decimal place in coefficients and attains $\approx 6\times10^{-6}$ state infidelity from projective measurement data, substantially outperforming that of the FNO trained with ideal full wave function data, $\approx 1.5\times10^{-2}$, by orders of magnitude.
Authors Jin Lee, Ziming Liu, Xinling Yu, Yixuan Wang, Haewon Jeong, Murphy Yuezhen Niu, Zheng Zhang
Categories Artificial Intelligence
2
MedVision: Dataset and Benchmark for Quantitative Medical Image Analysis
Title CN MedVision：用于定量医学图像分析的数据集和基准
Keywords 医学图像分析 · 视觉语言模型 · 定量推理 · 数据集 · 基准测试
Summary 本文提出了MedVision，一个大规模的医学图像定量分析数据集和基准，涵盖22个公开数据集，包含3080万图像-标注对，专注于解剖结构检测、病灶大小估计和角度/距离测量三类定量任务。实验表明现有现成的VLM在此类任务上表现不佳，但通过在MedVision上进行监督微调可显著提升性能。该工作旨在推动具备定量推理能力的医学VLM发展。
Reason 该论文聚焦于医学图像中的定量分析任务（如肿瘤大小测量、角度距离计算），属于特定垂直领域（医疗）的下游应用，本质上是构建了一个面向医学VLM的专用数据集与评测基准。尽管其任务涉及视觉-语言模型在细粒度数值推理上的能力，但整体工作未提出可迁移至文档图像理解（DIU）的通用方法论或架构创新。其核心贡献局限于医疗场景，且无证据表明所提方法能解决OCR幻觉、复杂排版或长文档推理等DIU关键痛点，属于典型的‘平行下游应用’。
Abstract Current vision-language models (VLMs) in medicine are primarily designed for categorical question answering (e.g., "Is this normal or abnormal?") or qualitative descriptive tasks. However, clinical decision-making often relies on quantitative assessments, such as measuring the size of a tumor or the angle of a joint, from which physicians draw their own diagnostic conclusions. This quantitative reasoning capability remains underexplored and poorly supported in existing VLMs. In this work, we introduce MedVision, a large-scale dataset and benchmark specifically designed to evaluate and improve VLMs on quantitative medical image analysis. MedVision spans 22 public datasets covering diverse anatomies and modalities, with 30.8 million image-annotation pairs. We focus on three representative quantitative tasks: (1) detection of anatomical structures and abnormalities, (2) tumor/lesion (T/L) size estimation, and (3) angle/distance (A/D) measurement. Our benchmarks show that current off-the-shelf VLMs perform poorly on these tasks. However, with supervised fine-tuning on MedVision, we significantly enhance their performance across detection, T/L estimation, and A/D measurement, demonstrating reduced error rates and improved precision. This work provides a foundation for developing VLMs with robust quantitative reasoning capabilities in medical imaging. Code and data are available at https://medvision-vlm.github.io.
Authors Yongcheng Yao, Yongshuo Zong, Raman Dutt, Yongxin Yang, Sotirios A Tsaftaris, Timothy Hospedales
Comment 8 pages, 8 figures, 4 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
Personalized LLM Decoding via Contrasting Personal Preference EMNLP 2025 Main
Title CN 通过对比个人偏好实现个性化LLM解码
Keywords LLM个性化 · 解码策略 · 偏好建模 · 参数高效微调 · 文本生成
Summary 本文提出CoPe，一种在参数高效微调后应用于解码阶段的个性化生成方法，通过最大化用户的隐式奖励信号来优化生成结果，无需外部奖励模型或额外训练。在五个开放式文本生成任务上验证了其有效性，平均提升ROUGE-L指标10.57%。
Reason 该论文研究的是大语言模型在开放文本生成任务中的个性化解码，核心是提升用户偏好的文本生成一致性。尽管其解码时干预（decoding-time intervention）机制具有一定的方法论价值，但其应用场景与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂版面分析、细粒度视觉定位、高分辨率处理）无直接关联。未涉及多模态输入、视觉信息处理或文档结构建模，无法迁移用于解决文档理解中的关键挑战。
Abstract As large language models (LLMs) are progressively deployed in various real-world applications, personalization of LLMs has become increasingly important. While various approaches to LLM personalization such as prompt-based and training-based methods have been actively explored, the development of effective decoding-time algorithms remains largely overlooked, despite their demonstrated potential. In this paper, we propose CoPe (Contrasting Personal Preference), a novel decoding-time approach applied after performing parameter-efficient fine-tuning (PEFT) on user-specific data. Our core idea is to leverage reward-guided decoding specifically for personalization by maximizing each user's implicit reward signal. We evaluate CoPe across five open-ended personalized text generation tasks. Our empirical results demonstrate that CoPe achieves strong performance, improving personalization by an average of 10.57% in ROUGE-L, without relying on external reward models or additional training procedures.
Authors Hyungjune Bu, Chanjoo Jung, Minjae Kang, Jaehyung Kim
Comment EMNLP 2025 Main
Categories Artificial Intelligence, Computation and Language
2
NOEM$^{3}$A: A Neuro-Symbolic Ontology-Enhanced Method for Multi-Intent Understanding in Mobile Agents
Title CN NOEM³A：一种用于移动智能体中多意图理解的神经符号学增强型本体方法
Keywords 神经符号系统 · 意图理解 · 本体 · 小型语言模型 · 移动AI
Summary 本文提出了一种结合本体结构的神经符号框架NOEM³A，用于提升小型语言模型在移动设备上对多意图的理解能力。通过引入检索增强提示、logit偏置和分类头，将符号化意图结构注入模型输入输出，并设计了基于本体层次的语义相似度评估指标SIS。实验表明小模型可接近GPT-4的性能，同时资源消耗更低。
Reason 该论文聚焦于移动AI代理中的多意图理解，属于典型的NLP下游应用任务，且基于对话系统（MultiWOZ数据集）和语言模型增强，未涉及任何文档图像、视觉输入或与文档理解相关的模态。其核心方法——本体增强、logit偏置、检索增强提示——虽然在NLU中有价值，但完全脱离视觉-语言交互与文档结构建模，无法迁移解决OCR幻觉、复杂排版、细粒度定位等DIU核心问题。属于‘平行应用’类工作，且领域无关。
Abstract We introduce a neuro-symbolic framework for multi-intent understanding in mobile AI agents by integrating a structured intent ontology with compact language models. Our method leverages retrieval-augmented prompting, logit biasing and optional classification heads to inject symbolic intent structure into both input and output representations. We formalize a new evaluation metric-Semantic Intent Similarity (SIS)-based on hierarchical ontology depth, capturing semantic proximity even when predicted intents differ lexically. Experiments on a subset of ambiguous/demanding dialogues of MultiWOZ 2.3 (with oracle labels from GPT-o3) demonstrate that a 3B Llama model with ontology augmentation approaches GPT-4 accuracy (85% vs 90%) at a tiny fraction of the energy and memory footprint. Qualitative comparisons show that ontology-augmented models produce more grounded, disambiguated multi-intent interpretations. Our results validate symbolic alignment as an effective strategy for enabling accurate and efficient on-device NLU.
Authors Ioannis Tzachristas, Aifen Sui
Categories Artificial Intelligence
2
Prune-Then-Plan: Step-Level Calibration for Stable Frontier Exploration in Embodied Question Answering
Title CN 剪枝后规划：面向具身问答中稳定前沿探索的步骤级校准
Keywords 具身问答 · 视觉语言模型 · 探索稳定性 · 步骤级校准 · 剪枝规划
Summary 本文提出Prune-Then-Plan框架，用于改善具身问答系统在3D环境探索中的稳定性。通过引入受Holm-Bonferroni启发的剪枝策略过滤不合理的移动选项，并结合基于覆盖率的规划器进行最终决策，从而缓解VLM驱动下的前后震荡问题。实验表明该方法在OpenEQA和 EXPRESS-Bench 上提升了任务完成率和答案匹配度。
Reason 该论文研究的是具身问答（Embodied Question Answering, EQA）中的导航稳定性问题，核心是解决VLM在3D环境中因过度自信导致的‘前沿振荡’问题。虽然其提出的方法涉及VLM的步骤级校准与决策稳定性，但应用场景属于具身智能与机器人导航范畴，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管‘step-level calibration’和‘pruning + planning’机制在理论上可能启发推理流程控制，但迁移到文档理解场景（如长文档多步推理或OCR纠错）的路径不明确，且未针对文档特有的密集文本、排版复杂性或视觉-语义对齐问题设计。属于无关领域应用。
Abstract Large vision-language models (VLMs) have improved embodied question answering (EQA) agents by providing strong semantic priors for open-vocabulary reasoning. However, when used directly for step-level exploration, VLMs often exhibit frontier oscillations, unstable back-and-forth movements caused by overconfidence and miscalibration, leading to inefficient navigation and degraded answer quality. We propose Prune-Then-Plan, a simple and effective framework that stabilizes exploration through step-level calibration. Instead of trusting raw VLM scores, our method prunes implausible frontier choices using a Holm-Bonferroni inspired pruning procedure and then delegates final decisions to a coverage-based planner. This separation converts overconfident predictions into conservative, interpretable actions by relying on human-level judgments to calibrate the step-level behavior of VLMs. Integrated into the 3D-Mem EQA framework, our approach achieves relative improvements of up to 49% and 33% in visually grounded SPL and LLM-Match metrics respectively over baselines. Overall, our method achieves better scene coverage under equal exploration budgets on both OpenEQA and EXPRESS-Bench datasets.
Authors Noah Frahm, Prakrut Patel, Yue Zhang, Shoubin Yu, Mohit Bansal, Roni Sengupta
Comment webpage: https://noahfrahm.github.io/Prune-Then-Plan-project-page/
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
Scaling Item-to-Standard Alignment with Large Language Models: Accuracy, Limits, and Solutions
Title CN 利用大语言模型扩展题目到标准的对齐：准确性、局限与解决方案
Keywords 大语言模型 · 教育评估 · 题目对齐 · 自动化评分 · 候选过滤
Summary 本文研究了大语言模型（如GPT-4o）在教育评估中将试题与教学标准对齐的能力，发现通过候选技能预筛选可显著提升准确率，建议采用LLM初筛+人工复核的混合流程以提高效率。实验基于12,000多个题-技能对，结果显示LLM在数学任务中表现良好，在阅读任务中因标准语义重叠而效果下降。
Reason 该论文属于典型的‘平行下游应用’，即将现有LLM技术应用于教育领域中的题目-标准对齐任务。其核心贡献是LLM在特定场景（K-12教育评估）中的应用效果分析，并未提出任何针对文档图像理解（DIU）或视觉-语言模型（VLM）的方法论创新。尽管涉及文本对齐和分类，但完全基于纯文本输入，无视觉模态、无OCR、无布局分析、无多模态交互，与DocVQA、表格识别、版面分析等DIU核心任务无关。
Abstract As educational systems evolve, ensuring that assessment items remain aligned with content standards is essential for maintaining fairness and instructional relevance. Traditional human alignment reviews are accurate but slow and labor-intensive, especially across large item banks. This study examines whether Large Language Models (LLMs) can accelerate this process without sacrificing accuracy. Using over 12,000 item-skill pairs in grades K-5, we tested three LLMs (GPT-3.5 Turbo, GPT-4o-mini, and GPT-4o) across three tasks that mirror real-world challenges: identifying misaligned items, selecting the correct skill from the full set of standards, and narrowing candidate lists prior to classification. In Study 1, GPT-4o-mini correctly identified alignment status in approximately 83-94% of cases, including subtle misalignments. In Study 2, performance remained strong in mathematics but was lower for reading, where standards are more semantically overlapping. Study 3 demonstrated that pre-filtering candidate skills substantially improved results, with the correct skill appearing among the top five suggestions more than 95% of the time. These findings suggest that LLMs, particularly when paired with candidate filtering strategies, can significantly reduce the manual burden of item review while preserving alignment accuracy. We recommend the development of hybrid pipelines that combine LLM-based screening with human review in ambiguous cases, offering a scalable solution for ongoing item validation and instructional alignment.
Authors Farzan Karimi-Malekabadi, Pooya Razavi, Sonya Powers
Categories Artificial Intelligence
2
An Adaptive, Data-Integrated Agent-Based Modeling Framework for Explainable and Contestable Policy Design
Title CN 一种用于可解释与可争议政策设计的自适应、数据集成的基于代理的建模框架
Keywords 多智能体系统 · 自适应学习 · 信息论诊断 · 结构因果模型 · 可解释性
Summary 本文提出了一种通用的自适应多智能体学习框架，支持动态环境下的反馈与参数调整，结合信息论诊断、结构因果模型和无监督聚类，用于分析学习型代理与自适应控制如何共同影响系统演化。适用于非平衡、振荡或漂移的动力学场景，强调政策设计的可解释性与可争议性。
Reason 该论文属于多智能体系统与政策模拟领域，聚焦于宏观社会系统的动态建模与因果分析，并未涉及文档图像、视觉内容理解或任何多模态语言模型在文档上的应用。尽管提到了‘agent-based modeling’和‘intervention semantics’，但其方法面向的是抽象决策过程而非视觉-语言对齐、OCR后处理或文档推理中的具体挑战（如幻觉、复杂排版、细粒度定位）。无法迁移解决DIU的核心痛点，属于无关领域的理论框架研究。
Abstract Multi-agent systems often operate under feedback, adaptation, and non-stationarity, yet many simulation studies retain static decision rules and fixed control parameters. This paper introduces a general adaptive multi-agent learning framework that integrates: (i) four dynamic regimes distinguishing static versus adaptive agents and fixed versus adaptive system parameters; (ii) information-theoretic diagnostics (entropy rate, statistical complexity, and predictive information) to assess predictability and structure; (iii) structural causal models for explicit intervention semantics; (iv) procedures for generating agent-level priors from aggregate or sample data; and (v) unsupervised methods for identifying emergent behavioral regimes. The framework offers a domain-neutral architecture for analyzing how learning agents and adaptive controls jointly shape system trajectories, enabling systematic comparison of stability, performance, and interpretability across non-equilibrium, oscillatory, or drifting dynamics. Mathematical definitions, computational operators, and an experimental design template are provided, yielding a structured methodology for developing explainable and contestable multi-agent decision processes.
Authors Roberto Garrone
Comment 27 pages, 2 case studies (emissions and smart grids). Preprint prepared during the author's PhD research at the Open University of Cyprus and the University of Milano-Bicocca. Introduces a unified framework for adaptive multi-agent learning with information-theoretic, causal, and clustering diagnostics
Categories Artificial Intelligence
2
FISCAL: Financial Synthetic Claim-document Augmented Learning for Efficient Fact-Checking 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop: Generative AI in Finance
Title CN FISCAL：面向高效事实核查的金融合成声明-文档增强学习
Keywords 金融事实核查 · 合成数据 · 轻量级模型 · 大语言模型 · 应用框架
Summary 本文提出FISCAL框架，通过生成金融领域的合成声明-文档对来训练一个轻量级的事实核查模型MiniCheck-FISCAL。实验表明该模型在多个金融事实核查数据集上表现优异，接近甚至超过更大规模的通用模型。然而，整个工作基于纯文本处理，不涉及文档图像或多模态理解。
Reason 该论文属于典型的‘平行下游应用’，其核心是将现有LLM与合成数据方法应用于金融领域的事实核查任务。尽管提出了一个名为FISCAL的框架和数据集，并训练了一个轻量级验证器MiniCheck-FISCAL，但并无底层方法论创新，如视觉表示、多模态架构改进、OCR去幻觉机制或文档结构建模等。它依赖于文本模态的合成数据增强和微调策略，未涉及文档图像理解中的关键挑战（如排版分析、视觉布局建模、细粒度定位或高分辨率处理）。因此，属于在特定领域（金融）上的应用包装，无法为DIU提供可迁移的技术源头。
Abstract Financial applications of large language models (LLMs) require factual reliability and computational efficiency, yet current systems often hallucinate details and depend on prohibitively large models. We propose FISCAL (Financial Synthetic Claim-Document Augmented Learning), a modular framework for generating synthetic data tailored to financial fact-checking. Using FISCAL, we generate a dataset called FISCAL-data and use it to train MiniCheck-FISCAL, a lightweight verifier for numerical financial claims. MiniCheck-FISCAL outperforms its baseline, surpasses GPT-3.5 Turbo and other open-source peers of similar size, and approaches the accuracy of much larger systems (20x), such as Mixtral-8x22B and Command R+. On external datasets FinDVer and Fin-Fact, it rivals GPT-4o and Claude-3.5 while outperforming Gemini-1.5 Flash. These results show that domain-specific synthetic data, combined with efficient fine-tuning, enables compact models to achieve state-of-the-art accuracy, robustness, and scalability for practical financial AI. The dataset and scripts are available in the project repository (link provided in the paper).
Authors Rishab Sharma, Iman Saberi, Elham Alipour, Jie JW Wu, Fatemeh Fard
Comment 3 tables, 11 pages, 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop: Generative AI in Finance
Categories Artificial Intelligence
2
The Semiotic Channel Principle: Measuring the Capacity for Meaning in LLM Communication
Title CN 符号通道原理：衡量大语言模型通信中的意义承载能力
Keywords 符号学 · 大语言模型 · 信息论 · 可解释性 · 语义通信
Summary 本文提出了一个符号学框架，用以量化大语言模型在生成文本时的意义表达能力与人类解读稳定性之间的平衡。通过信息论工具定义了‘表达广度’和‘可解度’，并引入控制参数λ来调节二者关系。虽然可用于提示优化和风险分析，但完全脱离视觉与多模态场景，不适用于文档图像理解任务。
Reason 该论文提出了一种基于符号学的信息论框架，用于分析大语言模型作为‘随机符号引擎’在人类解释下的表达丰富性与可解码性之间的权衡。尽管其理论涉及LLM输出的可解释性和歧义风险，但完全聚焦于纯文本语义通信，未涉及任何视觉模态、多模态对齐或文档结构理解。无法迁移至文档图像理解中的OCR幻觉、细粒度定位或复杂排版解析等核心问题。属于通用NLP理论探讨，且无具体VLM或DIU应用场景。
Abstract This paper proposes a novel semiotic framework for analyzing Large Language Models (LLMs), conceptualizing them as stochastic semiotic engines whose outputs demand active, asymmetric human interpretation. We formalize the trade-off between expressive richness (semiotic breadth) and interpretive stability (decipherability) using information-theoretic tools. Breadth is quantified as source entropy, and decipherability as the mutual information between messages and human interpretations. We introduce a generative complexity parameter (lambda) that governs this trade-off, as both breadth and decipherability are functions of lambda. The core trade-off is modeled as an emergent property of their distinct responses to $λ$. We define a semiotic channel, parameterized by audience and context, and posit a capacity constraint on meaning transmission, operationally defined as the maximum decipherability by optimizing lambda. This reframing shifts analysis from opaque model internals to observable textual artifacts, enabling empirical measurement of breadth and decipherability. We demonstrate the framework's utility across four key applications: (i) model profiling; (ii) optimizing prompt/context design; (iii) risk analysis based on ambiguity; and (iv) adaptive semiotic systems. We conclude that this capacity-based semiotic approach offers a rigorous, actionable toolkit for understanding, evaluating, and designing LLM-mediated communication.
Authors Davide Picca
Categories Artificial Intelligence
2
Adapting Vision-Language Models for Evaluating World Models NeurIPS LAW 2025 (Oral)
Title CN 为世界模型评估而适配视觉语言模型
Keywords Vision-Language Model · World Model Evaluation · Action Recognition · Embodied AI · Multimodal Reasoning
Summary 本文提出UNIVERSE，一个基于VLM的世界模型rollout评估系统，通过适配VLM在动作和角色识别任务上实现对模拟环境中生成序列的语义一致性与行为对齐评估。实验涵盖多种适应方式与任务设置，并验证了其与人类判断的一致性。然而，该工作聚焦于视频动态序列的评估，属于具身AI与仿真领域，不涉及文档结构、文本识别或布局分析。
Reason 该论文属于典型的‘平行应用’：将VLM用于视频世界模型的 rollout 评估，任务场景集中在动作识别与角色识别，且面向的是具身智能和模拟环境。尽管使用了VLM进行细粒度时序评估，但其核心目标是服务于视频生成与仿真评估，与文档图像理解（DIU）中的OCR幻觉、复杂排版、长文档推理等核心痛点无直接关联。方法上未提出可迁移至文档领域的底层创新（如视觉压缩、高分辨率处理、多步逻辑推理架构），仅为特定任务下的VLM下游应用。
Abstract World models - generative models that simulate environment dynamics conditioned on past observations and actions - are gaining prominence in planning, simulation, and embodied AI. However, evaluating their rollouts remains a fundamental challenge, requiring fine-grained, temporally grounded assessment of action alignment and semantic consistency - capabilities not captured by existing metrics. Vision-Language Models (VLMs) have shown promise as automatic evaluators of generative content due to their strong multimodal reasoning abilities. Yet, their use in fine-grained, temporally sensitive evaluation tasks remains limited and requires targeted adaptation. We introduce an evaluation protocol targeting two recognition tasks - action recognition and character recognition - each assessed across binary, multiple-choice, and open-ended formats. To support this, we present UNIVERSE (UNIfied Vision-language Evaluator for Rollouts in Simulated Environments), a VLM-based evaluator for video world model rollouts adapted under data and compute constraints. In our extensive experiments totaling over 5,154 GPU-days, we explore full, partial, and parameter-efficient adaptation methods across various task formats, context lengths, sampling methods, and data compositions. The resulting unified evaluator achieves parity with task-specific checkpoints. Human studies across seven diverse environments confirm strong alignment with human judgments, establishing UNIVERSE as a lightweight, adaptable, and semantics-aware evaluator for video world models.
Authors Mariya Hendriksen, Tabish Rashid, David Bignell, Raluca Georgescu, Abdelhak Lemkhenter, Katja Hofmann, Sam Devlin, Sarah Parisot
Comment NeurIPS LAW 2025 (Oral)
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
MAIF: Enforcing AI Trust and Provenance with an Artifact-Centric Agentic Paradigm
Title CN MAIF：通过以产物为中心的智能体范式强化AI可信度与溯源性
Keywords 可信AI · 智能体架构 · 数据溯源 · 多模态容器 · 系统安全
Summary 本文提出MAIF——一种面向AI可信性的多模态产物文件格式，通过在数据容器中嵌入语义表示、加密溯源和访问控制机制，实现AI行为的可审计与防篡改。其技术重点在于系统级信任构建，适用于高安全需求场景，但未针对文档图像理解任务进行优化或验证。
Reason 该论文提出一种以数据产物为核心的AI智能体架构，重点解决AI系统的可审计性、溯源性和安全性问题，主要面向监管合规和安全敏感场景。尽管其提出的多模态产物文件格式（MAIF）支持语义表示和跨模态注意力，但核心贡献在于系统架构与数据容器设计，而非文档图像理解中的具体任务（如DocVQA、布局分析或OCR-free识别）。未体现对文档理解中OCR幻觉、复杂排版处理、细粒度定位等关键痛点的技术改进或可迁移方法。属于通用AI系统架构应用，不具DIU领域的方法论创新。
Abstract The AI trustworthiness crisis threatens to derail the artificial intelligence revolution, with regulatory barriers, security vulnerabilities, and accountability gaps preventing deployment in critical domains. Current AI systems operate on opaque data structures that lack the audit trails, provenance tracking, or explainability required by emerging regulations like the EU AI Act. We propose an artifact-centric AI agent paradigm where behavior is driven by persistent, verifiable data artifacts rather than ephemeral tasks, solving the trustworthiness problem at the data architecture level. Central to this approach is the Multimodal Artifact File Format (MAIF), an AI-native container embedding semantic representations, cryptographic provenance, and granular access controls. MAIF transforms data from passive storage into active trust enforcement, making every AI operation inherently auditable. Our production-ready implementation demonstrates ultra-high-speed streaming (2,720.7 MB/s), optimized video processing (1,342 MB/s), and enterprise-grade security. Novel algorithms for cross-modal attention, semantic compression, and cryptographic binding achieve up to 225 compression while maintaining semantic fidelity. Advanced security features include stream-level access control, real-time tamper detection, and behavioral anomaly analysis with minimal overhead. This approach directly addresses the regulatory, security, and accountability challenges preventing AI deployment in sensitive domains, offering a viable path toward trustworthy AI systems at scale.
Authors Vineeth Sai Narajala, Manish Bhatt, Idan Habler, Ronald F. Del Rosario, Ads Dawson
Comment 7 Pages, 2 Figures, 6 Tables, Repo: https://github.com/vineethsai/maifscratch-1, Added additional Author and fixed Citations
Categories Artificial Intelligence
2
Solving Diffusion Inverse Problems with Restart Posterior Sampling
Title CN 利用重启后验采样解决扩散逆问题
Keywords 扩散模型 · 逆问题 · 后验采样 · 信号恢复 · RePS
Summary 本文提出了一种名为RePS（Restart for Posterior Sampling）的新框架，用于利用预训练扩散模型解决线性和非线性扩散逆问题。该方法基于重启采样思想，通过引入条件常微分方程和简化的重启策略来提升后验采样的收敛速度和重建质量，同时避免了对评分网络的反向传播，降低了计算成本。实验表明其在多种逆问题上优于现有扩散基线方法。
Reason 该论文研究的是通用扩散模型在逆问题（如图像去噪、压缩感知）中的应用，属于底层视觉或生成模型领域。尽管其方法在理论上可能对高分辨率图像重建有帮助，但并未针对文档图像理解（DIU）中的核心痛点（如OCR幻觉、复杂版式、细粒度文本定位）提出直接可迁移的机制。且该工作聚焦于信号恢复类任务，与DocVQA、表格识别、布局分析等DIU核心任务无直接关联，属于无关领域的技术探索。
Abstract Inverse problems are fundamental to science and engineering, where the goal is to infer an underlying signal or state from incomplete or noisy measurements. Recent approaches employ diffusion models as powerful implicit priors for such problems, owing to their ability to capture complex data distributions. However, existing diffusion-based methods for inverse problems often rely on strong approximations of the posterior distribution, require computationally expensive gradient backpropagation through the score network, or are restricted to linear measurement models. In this work, we propose Restart for Posterior Sampling (RePS), a general and efficient framework for solving both linear and non-linear inverse problems using pre-trained diffusion models. RePS builds on the idea of restart-based sampling, previously shown to improve sample quality in unconditional diffusion, and extends it to posterior inference. Our method employs a conditioned ODE applicable to any differentiable measurement model and introduces a simplified restart strategy that contracts accumulated approximation errors during sampling. Unlike some of the prior approaches, RePS avoids backpropagation through the score network, substantially reducing computational cost. We demonstrate that RePS achieves faster convergence and superior reconstruction quality compared to existing diffusion-based baselines across a range of inverse problems, including both linear and non-linear settings.
Authors Bilal Ahmed, Joseph G. Makin
Categories Artificial Intelligence
2
Breaking the Likelihood-Quality Trade-off in Diffusion Models by Merging Pretrained Experts ICLR 2025 DeLTa workshop
Title CN 通过合并预训练专家打破扩散模型中的似然性-质量权衡
Keywords 扩散模型 · 图像生成 · 似然性-质量权衡 · 专家混合 · 采样策略
Summary 本文提出一种在扩散模型采样过程中切换两个预训练专家（一个侧重图像质量，一个侧重似然性）的方法，以同时提升生成图像的质量和对数似然表现。该方法无需微调，仅通过选择合适的噪声层级进行切换即可实现性能增益。
Reason 该论文研究的是图像生成领域中扩散模型的似然性与生成质量之间的权衡问题，提出通过在去噪过程中切换不同专家模型来融合优势。然而，这项工作属于纯图像生成范畴，且聚焦于自然图像（如CIFAR-10、ImageNet32），与文档图像理解（DIU）的核心痛点（如OCR幻觉、文本结构识别、复杂排版解析等）无直接关联。方法本质上是生成任务中的采样策略优化，无法迁移用于提升文档视觉问答（DocVQA）或多模态文档推理能力。
Abstract Diffusion models for image generation often exhibit a trade-off between perceptual sample quality and data likelihood: training objectives emphasizing high-noise denoising steps yield realistic images but poor likelihoods, whereas likelihood-oriented training overweights low-noise steps and harms visual fidelity. We introduce a simple plug-and-play sampling method that combines two pretrained diffusion experts by switching between them along the denoising trajectory. Specifically, we apply an image-quality expert at high noise levels to shape global structure, then switch to a likelihood expert at low noise levels to refine pixel statistics. The approach requires no retraining or fine-tuning -- only the choice of an intermediate switching step. On CIFAR-10 and ImageNet32, the merged model consistently matches or outperforms its base components, improving or preserving both likelihood and sample quality relative to each expert alone. These results demonstrate that expert switching across noise levels is an effective way to break the likelihood-quality trade-off in image diffusion models.
Authors Yasin Esfandiari, Stefan Bauer, Sebastian U. Stich, Andrea Dittadi
Comment ICLR 2025 DeLTa workshop
Categories Computer Vision and Pattern Recognition
2
POUR: A Provably Optimal Method for Unlearning Representations via Neural Collapse
Title CN POUR：基于神经坍缩理论的表示遗忘可证明最优方法
Keywords 机器遗忘 · 表示学习 · 神经坍缩 · ETF · 几何投影
Summary 本文提出POUR方法，基于神经坍缩理论和单纯形等角紧框架（ETF），通过几何投影实现表示层的可证明最优遗忘。作者定义了表示遗忘评分RUS，并提出了闭式解POUR-P和特征级蒸馏方案POUR-D，在CIFAR和PathMNIST上验证了其在保留知识的同时有效遗忘目标类别的能力。
Reason 该论文研究机器遗忘（machine unlearning）中的表示层遗忘问题，聚焦于图像分类任务中如何从已训练模型中移除特定概念的影响。尽管其方法在理论上具有创新性，但属于通用计算机视觉中的模型编辑与遗忘学习范畴，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。未涉及OCR、多模态对齐、视觉-语言推理或文档特有的复杂排版处理，也无法迁移用于解决DIU中的典型痛点（如OCR幻觉、长文档推理、细粒度定位）。因此属于无关领域的技术研究，不符合本筛选标准。
Abstract In computer vision, machine unlearning aims to remove the influence of specific visual concepts or training images without retraining from scratch. Studies show that existing approaches often modify the classifier while leaving internal representations intact, resulting in incomplete forgetting. In this work, we extend the notion of unlearning to the representation level, deriving a three-term interplay between forgetting efficacy, retention fidelity, and class separation. Building on Neural Collapse theory, we show that the orthogonal projection of a simplex Equiangular Tight Frame (ETF) remains an ETF in a lower dimensional space, yielding a provably optimal forgetting operator. We further introduce the Representation Unlearning Score (RUS) to quantify representation-level forgetting and retention fidelity. Building on this, we introduce POUR (Provably Optimal Unlearning of Representations), a geometric projection method with closed-form (POUR-P) and a feature-level unlearning variant under a distillation scheme (POUR-D). Experiments on CIFAR-10/100 and PathMNIST demonstrate that POUR achieves effective unlearning while preserving retained knowledge, outperforming state-of-the-art unlearning methods on both classification-level and representation-level metrics.
Authors Anjie Le, Can Peng, Yuyuan Liu, J. Alison Noble
Categories Computer Vision and Pattern Recognition
2
MonoSR: Open-Vocabulary Spatial Reasoning from Monocular Images
Title CN MonoSR：基于单目图像的开放词汇空间推理
Keywords 空间推理 · 单目图像 · 视觉语言模型 · 3D理解 · 开放世界
Summary 该研究提出了MonoSR，一个大规模的单目空间推理数据集，涵盖室内、室外和以物体为中心的多种场景，旨在推动开放世界下的单目图像空间理解。作者评估了现有视觉语言模型在该任务上的表现，并探讨了辅助信息对推理的重要性。然而，该工作属于3D空间理解范畴，与文档图像理解任务无关。
Reason 该论文聚焦于从单目图像中进行3D空间推理，主要应用于具身智能和自动驾驶等场景，属于3D视觉与视频理解领域。其核心任务是基于2D图像推断真实世界的空间结构，与文档图像理解（DIU）中的文本识别、布局分析、表格理解等任务无直接关联。尽管使用了VLM进行评估，但问题设定、数据模态和目标域均与文档理解相距甚远，无法迁移解决OCR幻觉、密集文本处理或复杂排版理解等DIU核心痛点。
Abstract Spatial reasoning (SR), the ability to infer 3D spatial information from 2D inputs, is essential for real-world applications such as embodied AI and autonomous driving. However, existing research primarily focuses on indoor environments and typically relies on multi-view observations, which limits their generalizability to outdoor scenarios and constrains their applicability to monocular images, the most common real-world setting. In this work, we propose MonoSR, a large-scale monocular spatial reasoning dataset that spans diverse scenarios including indoor, outdoor, and object-centric settings, and supports multiple question types. MonoSR provides a path toward open-world monocular spatial reasoning. Beyond introducing the dataset, we evaluate advanced vision-language models to reveal their limitations on this challenging task. We further analyze whether auxiliary information is crucial for monocular spatial reasoning and offer practical guidance for designing future models. These contributions collectively establish a foundation for advancing monocular spatial reasoning in real-world, open-world environments.
Authors Qirui Wang, Jingyi He, Yining Pan, Si Yong Yeo, Xulei Yang, Shijie Li
Categories Computer Vision and Pattern Recognition
2
3M-TI: High-Quality Mobile Thermal Imaging via Calibration-free Multi-Camera Cross-Modal Diffusion
Title CN 3M-TI：通过无标定多相机跨模态扩散实现高质量移动热成像
Keywords 热成像超分辨率 · 跨模态扩散模型 · 无标定融合 · 移动视觉 · 图像增强
Summary 本文提出3M-TI，一种无需相机标定的多相机跨模态扩散框架，用于提升移动设备上热成像的分辨率和纹理质量。通过在扩散UNet中引入跨模态自注意力模块（CSM），实现热图与RGB图的特征对齐，从而在无需精确标定的情况下完成热图像超分。实验表明该方法在视觉质量和定量指标上均优于现有方法，并提升了下游检测与分割任务的表现。
Reason 该论文研究的是热成像与RGB图像融合的超分辨率问题，属于特定传感器模态（热红外）的图像增强任务，核心目标是提升低分辨率热图像的质量，并依赖于多相机系统。尽管使用了跨模态扩散模型，但其应用场景远离文档图像理解（DIU）的核心场景——如文本识别、版面分析、表格理解等。且方法高度依赖双相机硬件设置和物理对齐，无法迁移至文档图像中的OCR幻觉、复杂排版或长文档推理等问题。属于特定领域的图像增强应用，不符合DIU本题范畴，亦无通用方法可迁移到文档理解。
Abstract The miniaturization of thermal sensors for mobile platforms inherently limits their spatial resolution and textural fidelity, leading to blurry and less informative images. Existing thermal super-resolution (SR) methods can be grouped into single-image and RGB-guided approaches: the former struggles to recover fine structures from limited information, while the latter relies on accurate and laborious cross-camera calibration, which hinders practical deployment and robustness. Here, we propose 3M-TI, a calibration-free Multi-camera cross-Modality diffusion framework for Mobile Thermal Imaging. At its core, 3M-TI integrates a cross-modal self-attention module (CSM) into the diffusion UNet, replacing the original self-attention layers to adaptively align thermal and RGB features throughout the denoising process, without requiring explicit camera calibration. This design enables the diffusion network to leverage its generative prior to enhance spatial resolution, structural fidelity, and texture detail in the super-resolved thermal images. Extensive evaluations on real-world mobile thermal cameras and public benchmarks validate our superior performance, achieving state-of-the-art results in both visual quality and quantitative metrics. More importantly, the thermal images enhanced by 3M-TI lead to substantial gains in critical downstream tasks like object detection and segmentation, underscoring its practical value for robust mobile thermal perception systems. More materials: https://github.com/work-submit/3MTI.
Authors Minchong Chen, Xiaoyun Yuan, Junzhe Wan, Jianing Zhang, Jun Zhang
Comment 11 pages, 7 figures
Categories Computer Vision and Pattern Recognition
2
TokenCLIP: Token-wise Prompt Learning for Zero-shot Anomaly Detection
Title CN TokenCLIP：面向零样本异常检测的逐令牌提示学习
Keywords CLIP · 异常检测 · 提示学习 · 最优传输 · 零样本
Summary 本文提出TokenCLIP，通过为每个视觉token动态分配定制化的文本子空间来实现细粒度的视觉-语言对齐，用于零样本异常检测。采用最优传输建模token到子空间的语义匹配，并引入top-k掩码提升专业化程度。实验表明其在多种异常检测基准上表现优异。
Reason 该论文研究的是通用图像领域的零样本异常检测，属于视觉-语言模型在工业缺陷或罕见物体识别中的应用，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其采用了细粒度的token-wise提示学习和最优传输机制，但方法设计针对的是自然图像中的异常语义对齐问题，难以迁移到文档场景中解决OCR幻觉、密集文本识别或复杂排版理解等关键痛点。属于将CLIP应用于特定下游任务的‘平行应用’，不符合本方向对底层方法创新或可迁移‘军火库’工具的要求。
Abstract Adapting CLIP for anomaly detection on unseen objects has shown strong potential in a zero-shot manner. However, existing methods typically rely on a single textual space to align with visual semantics across diverse objects and domains. The indiscriminate alignment hinders the model from accurately capturing varied anomaly semantics. We propose TokenCLIP, a token-wise adaptation framework that enables dynamic alignment between visual and learnable textual spaces for fine-grained anomaly learning. Rather than mapping all visual tokens to a single, token-agnostic textual space, TokenCLIP aligns each token with a customized textual subspace that represents its visual characteristics. Explicitly assigning a unique learnable textual space to each token is computationally intractable and prone to insufficient optimization. We instead expand the token-agnostic textual space into a set of orthogonal subspaces, and then dynamically assign each token to a subspace combination guided by semantic affinity, which jointly supports customized and efficient token-wise adaptation. To this end, we formulate dynamic alignment as an optimal transport problem, where all visual tokens in an image are transported to textual subspaces based on semantic similarity. The transport constraints of OT ensure sufficient optimization across subspaces and encourage them to focus on different semantics. Solving the problem yields a transport plan that adaptively assigns each token to semantically relevant subspaces. A top-k masking is then applied to sparsify the plan and specialize subspaces for distinct visual regions. Extensive experiments demonstrate the superiority of TokenCLIP.
Authors Qihang Zhou, Binbin Gao, Guansong Pang, Xin Wang, Jiming Chen, Shibo He
Categories Computer Vision and Pattern Recognition
2
Evo-0: Vision-Language-Action Model with Implicit Spatial Understanding
Title CN Evo-0：具有隐式空间理解能力的视觉-语言-动作模型
Keywords 视觉-语言-动作模型 · 隐式空间理解 · 3D几何 · 具身智能 · 机器人
Summary 本文提出Evo-0，一种通过集成视觉几何基础模型来增强VLA模型隐式3D理解能力的方法，使其能从RGB图像中推断场景的几何结构和物体间空间关系，从而提升机器人在复杂环境中的表现。实验涵盖仿真与真实世界任务，验证了性能提升。
Reason 该论文聚焦于具身智能中的视觉-语言-动作（VLA）模型，目标是提升机器人在三维空间中的感知与行动能力。其核心贡献是通过引入几何视觉基础模型来增强VLM对RGB图像的隐式3D空间理解。然而，该工作属于具身智能与机器人领域，主要解决的是物理场景中的空间推理问题，而非文档图像理解中的排版、文本定位或OCR幻觉等痛点。尽管‘空间理解’在广义上与文档布局有关，但其方法针对的是真实世界物体的深度和几何结构，难以迁移到文档的二维语义结构解析中。
Abstract Vision-Language-Action (VLA) models have emerged as a promising framework for enabling generalist robots capable of perceiving, reasoning, and acting in the real world. These models usually build upon pretrained Vision-Language Models (VLMs), which excel at semantic understanding due to large-scale image and text pretraining. However, existing VLMs typically lack precise spatial understanding capabilities, as they are primarily tuned on 2D image-text pairs without 3D supervision. To address this limitation, recent approaches have incorporated explicit 3D inputs such as point clouds or depth maps, but this necessitates additional depth sensors or pre-trained depth estimation models, which may yield defective results. In contrast, our work introduces a plug-and-play module that implicitly incorporates 3D geometry features into VLA models by leveraging an off-the-shelf visual geometry foundation model. This integration provides the model with depth-aware visual representations, improving its ability to understand the geometric structure of the scene and the spatial relationships among objects from RGB images alone. We evaluate our method on a set of spatially challenging tasks in both simulation and the real world. Extensive evaluations show that our method significantly improves the performance of state-of-the-art VLA models across diverse scenarios.
Authors Tao Lin, Gen Li, Yilei Zhong, Yanwen Zou, Yuxin Du, Jiting Liu, Encheng Gu, Bo Zhao
Categories Computer Vision and Pattern Recognition
2
AVA-VLA: Improving Vision-Language-Action models with Active Visual Attention
Title CN AVA-VLA：通过主动视觉注意力提升视觉-语言-动作模型
Keywords Vision-Language-Action · Embodied AI · Active Attention · POMDP · Robotics
Summary 本文提出AVA-VLA，一种基于部分可观测马尔可夫决策过程（POMDP）的视觉-语言-动作框架，通过引入主动视觉注意力（AVA）机制，利用历史隐状态动态加权视觉token，实现更高效的视觉处理。在多个机器人任务上实现了SOTA，并验证了良好的仿真到现实迁移能力。
Reason 该论文聚焦于具身智能中的视觉-语言-动作（VLA）模型，应用于机器人控制任务（如LIBERO、CALVIN），属于典型的具身智能/机器人领域。其核心创新是基于POMDP框架引入历史状态感知的主动视觉注意力机制，用于动态决策。然而，该方法针对的是连续控制与环境交互场景，与文档图像理解（DIU）中的OCR幻觉、密集文本识别、复杂版式分析等核心痛点无直接关联。尽管‘注意力调制’在广义上可能启发视觉处理，但其迁移至文档理解场景的路径不明确，且未涉及多模态对齐、高分辨率文档编码或文本结构建模等DIU关键挑战。
Abstract Vision-Language-Action (VLA) models have demonstrated remarkable capabilities in embodied AI tasks. However, existing VLA models, often built upon Vision-Language Models (VLMs), typically process dense visual inputs independently at each timestep. This approach implicitly models the task as a Markov Decision Process (MDP). However, this history-agnostic design is suboptimal for effective visual token processing in dynamic sequential decision-making, as it fails to leverage the context of history. To address this limitation, we reformulate the problem from a Partially Observable Markov Decision Process (POMDP) perspective and propose a novel framework named AVA-VLA. Inspired by the POMDP that the action generation should be conditioned on the belief state. AVA-VLA introduces Active Visual Attention (AVA) to dynamically modulate visual processing. It achieves this by leveraging the recurrent state, which is a neural approximation of the agent's belief state derived from the previous decision step. Specifically, the AVA module uses the recurrent state to compute the soft weights to actively process task-relevant visual tokens based on its historical context. Comprehensive evaluations demonstrate that AVA-VLA achieves state-of-the-art performance across popular robotic benchmarks, including LIBERO and CALVIN. Furthermore, real-world deployments on a dual-arm robot platform validate the framework's practical applicability and robust sim-to-real transferability.
Authors Lei Xiao, Jifeng Li, Juntao Gao, Feiyang Ye, Yan Jin, Jingjing Qian, Jing Zhang, Yong Wu, Xiaoyuan Yu
Comment 18 pages, 10 figures
Categories Computer Vision and Pattern Recognition
2
EventSTU: Event-Guided Efficient Spatio-Temporal Understanding for Video Large Language Models
Title CN EventSTU：面向视频大语言模型的事件引导高效时空理解
Keywords 视频大语言模型 · 事件相机 · token剪枝 · 高效推理 · 关键帧采样
Summary 本文提出EventSTU，一种受事件视觉启发的训练免费框架，用于提升视频大语言模型的推理效率。通过粗到细的关键帧采样和基于事件显著性的自适应token剪枝，结合问题相关性动态分配计算资源，并构建了首个包含事件数据的多模态基准EventBench。实验显示显著降低FLOPs和提升预填充速度的同时保持甚至提升性能。
Reason 该论文聚焦于视频大语言模型中的高效时空理解，核心解决的是长视频推理中的高计算开销问题，技术路线基于事件相机（event-based vision）和自适应token剪枝。尽管其token压缩思想看似与文档图像中的高分辨率处理有表面相似性，但其依赖事件信号这一特定模态，在标准文档图像（静态、无事件流）中无法直接迁移。且任务场景为视频理解，远离文档结构解析、VIE、表格识别等DIU核心任务，属于无关领域应用。
Abstract Video large language models have demonstrated strong video understanding capabilities but suffer from high inference costs due to the massive number of tokens in long videos. Inspired by event-based vision, we propose an event-guided, training-free framework for efficient spatio-temporal understanding, named EventSTU. In the temporal domain, we design a coarse-to-fine keyframe sampling algorithm that exploits the change-triggered property of event cameras to eliminate redundant frames. In the spatial domain, we design an adaptive token pruning algorithm that leverages the visual saliency of events as a zero-cost prior to guide spatial reduction. From a holistic spatio-temporal perspective, we further integrate question relevance from keyframe sampling to adaptively allocate token pruning budgets. To facilitate evaluation, we construct EventBench, the first event-inclusive, human-annotated multimodal benchmark that covers diverse real-world scenarios. Beyond physical event cameras, EventSTU also supports general video understanding using simulated events. Comprehensive experiments show that EventSTU achieves 3.01x FLOPs reduction and 3.10x prefilling speedup over the strongest baseline while still improving performance.
Authors Wenhao Xu, Xin Dong, Yue Li, Haoyuan Shi, Zhiwei Xiong
Comment 8 pages, 7 figures
Categories Computer Vision and Pattern Recognition
2
Robust and Generalizable GNN Fine-Tuning via Uncertainty-aware Adapter Learning
Title CN 基于不确定性感知适配器学习的鲁棒且通用的GNN微调
Keywords 图神经网络 · 适配器学习 · 不确定性建模 · 鲁棒微调 · 图噪声
Summary 本文提出一种不确定性感知的GNN适配器（UAdapterGNN），通过引入高斯分布建模适配模块中的不确定性，提升预训练GNN模型在含噪图数据（如错误边或模糊节点属性）下的微调鲁棒性与下游任务泛化能力。实验表明该方法在多个基准上优于现有适配器方法。
Reason 该论文研究图神经网络（GNN）在存在噪声图数据情况下的微调问题，提出通过引入高斯概率适配器来增强模型鲁棒性和泛化性。尽管其方法在图学习领域可能具有价值，但完全属于GNN微调的通用技术，与文档图像理解（DIU）、多模态大模型（VLM）、OCR、布局分析、表格识别或视觉-语言推理等核心任务无直接关联。未体现任何针对文档结构、视觉排版、多模态对齐或细粒度文本定位的改进，也无法迁移用于解决OCR幻觉、高分辨率处理或长文档推理等DIU关键痛点。
Abstract Recently, fine-tuning large-scale pre-trained GNNs has yielded remarkable attention in adapting pre-trained GNN models for downstream graph learning tasks. One representative fine-tuning method is to exploit adapter (termed AdapterGNN) which aims to 'augment' the pre-trained model by inserting a lightweight module to make the 'augmented' model better adapt to the downstream tasks. However, graph data may contain various types of noise in downstream tasks, such as noisy edges and ambiguous node attributes. Existing AdapterGNNs are often prone to graph noise and exhibit limited generalizability. How to enhance the robustness and generalization ability of GNNs' fine tuning remains an open problem. In this paper, we show that the above problem can be well addressed by integrating uncertainty learning into the GNN adapter. We propose the Uncertainty-aware Adapter (UAdapterGNN) that fortifies pre-trained GNN models against noisy graph data in the fine-tuning process. Specifically, in contrast to regular AdapterGNN, our UAdapterGNN exploits Gaussian probabilistic adapter to augment the pre-trained GNN model. In this way, when the graph contains various noises,our method can automatically absorb the effects of changes in the variances of the Gaussian distribution, thereby significantly enhancing the model's robustness. Also, UAdapterGNN can further improve the generalization ability of the model on the downstream tasks. Extensive experiments on several benchmarks demonstrate the effectiveness, robustness and high generalization ability of the proposed UAdapterGNN method.
Authors Bo Jiang, Weijun Zhao, Beibei Wang, Xiao Wang, Jin Tang
Categories Computer Vision and Pattern Recognition
2
PositionIC: Unified Position and Identity Consistency for Image Customization
Title CN PositionIC：用于图像定制的统一位置与身份一致性
Keywords 图像定制 · 扩散模型 · 注意力机制 · 数据合成 · 多主体生成
Summary 本文提出PositionIC框架，旨在实现高保真且具有空间可控性的多主体图像定制。通过构建自动化的带位置标注的多主体数据集BMPDS，并设计一种基于NeRF启发的体积权重调节机制的可见性感知注意力模块，在扩散模型中解耦实例的空间位置与语义身份特征，从而支持精确的、遮挡感知的多主体放置。实验表明该方法在空间精度和身份一致性上达到SOTA。
Reason 该论文聚焦于主体驱动的图像生成与定制任务，核心解决多主体场景下的空间控制与身份一致性问题，属于图像生成/编辑领域。其提出的方法围绕扩散模型中的注意力机制改进和合成数据生成，主要应用于创意设计等视觉生成场景，与文档图像理解（DIU）中的OCR、布局分析、表格识别等任务无直接关联。尽管涉及细粒度定位，但其技术路径针对的是自由形态的图像主体放置，难以迁移至文档中严格的几何排版与文本语义结构解析。属于无关领域（纯图像生成），不符合关注方向。
Abstract Recent subject-driven image customization excels in fidelity, yet fine-grained instance-level spatial control remains an elusive challenge, hindering real-world applications. This limitation stems from two factors: a scarcity of scalable, position-annotated datasets, and the entanglement of identity and layout by global attention mechanisms. To this end, we introduce \modelname{}, a unified framework for high-fidelity, spatially controllable multi-subject customization. First, we present BMPDS, the first automatic data-synthesis pipeline for position-annotated multi-subject datasets, effectively providing crucial spatial supervision. Second, we design a lightweight, layout-aware diffusion framework that integrates a novel visibility-aware attention mechanism. This mechanism explicitly models spatial relationships via an NeRF-inspired volumetric weight regulation to effectively decouple instance-level spatial embeddings from semantic identity features, enabling precise, occlusion-aware placement of multiple subjects. Extensive experiments demonstrate \modelname{} achieves state-of-the-art performance on public benchmarks, setting new records for spatial precision and identity consistency. Our work represents a significant step towards truly controllable, high-fidelity image customization in multi-entity scenarios. Code and data will be publicly released.
Authors Junjie Hu, Tianyang Han, Kai Ma, Jialin Gao, Song Yang, Xianhua He, Junfeng Luo, Xiaoming Wei, Wenqiang Zhang
Categories Computer Vision and Pattern Recognition
2
IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding
Title CN IAG：面向基于视觉语言模型的视觉定位的输入感知型后门攻击
Keywords 视觉语言模型 · 后门攻击 · 视觉定位 · 对抗攻击 · 安全性
Summary 本文提出IAG，一种针对基于VLM的视觉定位系统的新型多目标后门攻击方法。该方法通过文本条件化的UNet生成与目标描述相关的隐蔽视觉触发器，在保持良性样本正常性能的同时实现高攻击成功率。实验表明该方法在多个VLM和基准上均有效且难以防御。然而，这项工作聚焦于模型安全性而非文档理解能力提升。
Reason 该论文研究的是针对视觉语言模型（VLM）在视觉定位任务上的安全漏洞，提出一种输入感知的多目标后门攻击方法。虽然技术上涉及VLM和视觉-语言对齐，但其核心是安全领域的对抗攻击（Backdoor Attack），属于特定威胁模型下的安全性分析，并未提出可用于提升文档图像理解性能的新架构、训练方法或推理机制。且其应用场景为通用图像+RefCOCO等开放世界定位数据集，与文档图像中的OCR、布局分析、表格理解等核心痛点无关。属于‘平行应用’类的安全研究，无直接迁移价值。
Abstract Recent advances in vision-language models (VLMs) have significantly enhanced the visual grounding task, which involves locating objects in an image based on natural language queries. Despite these advancements, the security of VLM-based grounding systems has not been thoroughly investigated. This paper reveals a novel and realistic vulnerability: the first multi-target backdoor attack on VLM-based visual grounding. Unlike prior attacks that rely on static triggers or fixed targets, we propose IAG, a method that dynamically generates input-aware, text-guided triggers conditioned on any specified target object description to execute the attack. This is achieved through a text-conditioned UNet that embeds imperceptible target semantic cues into visual inputs while preserving normal grounding performance on benign samples. We further develop a joint training objective that balances language capability with perceptual reconstruction to ensure imperceptibility, effectiveness, and stealth. Extensive experiments on multiple VLMs (e.g., LLaVA, InternVL, Ferret) and benchmarks (RefCOCO, RefCOCO+, RefCOCOg, Flickr30k Entities, and ShowUI) demonstrate that IAG achieves the best ASRs compared with other baselines on almost all settings without compromising clean accuracy, maintaining robustness against existing defenses, and exhibiting transferability across datasets and models. These findings underscore critical security risks in grounding-capable VLMs and highlight the need for further research on trustworthy multimodal understanding.
Authors Junxian Li, Beining Xu, Simin Chen, Jiatong Li, Jingdi Lei, Haodong Zhao, Di Zhang
Comment 20 pages, 13 Figures
Categories Computer Vision and Pattern Recognition, Computation and Language
2
VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Models
Title CN VideoPerceiver：增强视频多模态大语言模型的细粒度时间感知
Keywords 视频理解 · 多模态大模型 · 细粒度动作识别 · 强化学习 · 对比学习
Summary 本文提出VideoPerceiver，一种用于提升视频多模态大模型对短暂动作和稀有事件感知能力的框架。通过构建‘关键帧替换’的损坏视频样本，并在SFT阶段引入基于关键词的对比损失，在RL阶段使用相对奖励机制，强化模型对细微时间变化的敏感性。实验表明其在细粒度视频描述任务上优于现有方法。
Reason 该论文聚焦于视频理解中的细粒度动作和瞬时事件识别，属于视频领域而非文档图像理解（DIU）。尽管其提出的‘关键信息缺失’数据构造方式和相对奖励机制在理论上对去幻觉或敏感性训练有一定启发，但其核心任务（时间序列动作感知）与文档理解的核心痛点（如OCR幻觉、复杂版式、文本密集结构）差异显著。方法难以直接迁移至DIU场景，尤其是缺乏对高分辨率静态图像处理、视觉token压缩或多模态对齐优化的支持。属于无关领域工作。
Abstract We propose VideoPerceiver, a novel video multimodal large language model (VMLLM) that enhances fine-grained perception in video understanding, addressing VMLLMs' limited ability to reason about brief actions in short clips or rare transient events in long videos. VideoPerceiver adopts a two-stage training framework. During supervised fine-tuning (SFT), we construct "key-information-missing" videos by extracting event-action keywords from captions, identifying corresponding key frames, and replacing them with adjacent frames. We jointly encode original and modified video tokens with text tokens, aligning intermediate visual representations with keywords via an auxiliary contrastive loss to enhance sensitivity to fine-grained motion cues. In reinforcement learning (RL), both video variants are fed into the model to generate descriptions, and a novel relative reward ensures responses from complete videos outperform those from degraded inputs, explicitly training the model to recover temporally precise action details. We also curate a dataset of 80,000 videos with fine-grained actions and transient events. Experiments show VideoPerceiver substantially outperforms state-of-the-art VMLLMs on fine-grained action understanding and rare event captioning benchmarks, while maintaining strong performance on standard tasks. By prioritizing task-relevant visual features, our work redefines video-language model training for fine-grained perception.
Authors Fufangchen Zhao, Liao Zhang, Daiqi Shi, Yuanjun Gao, Chen Ye, Yang Cai, Jian Gao, Danfeng Yan
Categories Computer Vision and Pattern Recognition
2
DiP: Taming Diffusion Models in Pixel Space
Title CN DiP：在像素空间中驯服扩散模型
Keywords 扩散模型 · 像素空间 · 图像生成 · 高分辨率合成 · DiT
Summary 本文提出DiP，一种在像素空间进行扩散的高效图像生成框架，通过将生成过程解耦为全局结构构建和局部细节修复两个阶段，在不使用VAE的情况下实现了接近Latent Diffusion Models的计算效率，同时保持高质量生成。该方法在ImageNet 256×256上达到1.90的FID分数，并支持最高10倍于先前方法的推理速度。
Reason 该论文属于纯图像生成领域，研究的是通用高分辨率图像合成中的效率与质量权衡问题，提出了一种解耦全局与局部生成的像素空间扩散框架。尽管其高分辨率处理设计具有一定技术启发性，但整体工作聚焦于图像生成质量与速度优化，与文档图像理解（DIU）的核心任务如文本识别、布局分析、表格理解或VQA无关。无法迁移解决OCR幻觉、复杂排版解析或长文档推理等DIU痛点，属于无关领域工作。
Abstract Diffusion models face a fundamental trade-off between generation quality and computational efficiency. Latent Diffusion Models (LDMs) offer an efficient solution but suffer from potential information loss and non-end-to-end training. In contrast, existing pixel space models bypass VAEs but are computationally prohibitive for high-resolution synthesis. To resolve this dilemma, we propose DiP, an efficient pixel space diffusion framework. DiP decouples generation into a global and a local stage: a Diffusion Transformer (DiT) backbone operates on large patches for efficient global structure construction, while a co-trained lightweight Patch Detailer Head leverages contextual features to restore fine-grained local details. This synergistic design achieves computational efficiency comparable to LDMs without relying on a VAE. DiP is accomplished with up to 10$\times$ faster inference speeds than previous method while increasing the total number of parameters by only 0.3%, and achieves an 1.90 FID score on ImageNet 256$\times$256.
Authors Zhennan Chen, Junwei Zhu, Xu Chen, Jiangning Zhang, Xiaobin Hu, Hanzhen Zhao, Chengjie Wang, Jian Yang, Ying Tai
Categories Computer Vision and Pattern Recognition
2
DICE: Distilling Classifier-Free Guidance into Text Embeddings AAAI 2026 (Oral)
Title CN DICE：将无分类器引导蒸馏到文本嵌入中
Keywords 文本到图像生成 · 无分类器引导 · 知识蒸馏 · 文本嵌入优化 · 扩散模型
Summary 本文提出DICE方法，通过蒸馏将无分类器引导（CFG）的效果整合进文本嵌入，使文本到图像扩散模型在不使用CFG的情况下仍能保持良好的文本-图像对齐，同时减少50%的计算复杂度。实验验证了其在多个主流扩散模型上的有效性。
Reason 该论文研究的是文本到图像生成中的效率优化问题，核心是通过蒸馏技术将Classifier-free Guidance（CFG）的效果融入文本嵌入，以降低推理计算开销。尽管涉及多模态对齐和表示学习，但其目标场景为图像生成，与文档图像理解（DIU）中的OCR、布局分析、表格识别等任务无直接关联。方法聚焦于生成质量与速度的权衡，并未解决DIU的核心痛点如密集文本处理、排版理解或视觉幻觉抑制。属于纯图像生成领域的技术优化，无法迁移至文档理解场景。
Abstract Text-to-image diffusion models are capable of generating high-quality images, but suboptimal pre-trained text representations often result in these images failing to align closely with the given text prompts. Classifier-free guidance (CFG) is a popular and effective technique for improving text-image alignment in the generative process. However, CFG introduces significant computational overhead. In this paper, we present DIstilling CFG by sharpening text Embeddings (DICE) that replaces CFG in the sampling process with half the computational complexity while maintaining similar generation quality. DICE distills a CFG-based text-to-image diffusion model into a CFG-free version by refining text embeddings to replicate CFG-based directions. In this way, we avoid the computational drawbacks of CFG, enabling high-quality, well-aligned image generation at a fast sampling speed. Furthermore, examining the enhancement pattern, we identify the underlying mechanism of DICE that sharpens specific components of text embeddings to preserve semantic information while enhancing fine-grained details. Extensive experiments on multiple Stable Diffusion v1.5 variants, SDXL, and PixArt-$α$ demonstrate the effectiveness of our method. Code is available at https://github.com/zju-pi/dice.
Authors Zhenyu Zhou, Defang Chen, Can Wang, Chun Chen, Siwei Lyu
Comment AAAI 2026 (Oral)
Categories Computer Vision and Pattern Recognition
2
CoD: A Diffusion Foundation Model for Image Compression
Title CN CoD：一种面向图像压缩的扩散基础模型
Keywords 图像压缩 · 扩散模型 · 基础模型 · 编码解码 · 像素空间扩散
Summary 本文提出了CoD，首个专为图像压缩设计的扩散基础模型，摆脱了传统基于文本到图像扩散模型（如Stable Diffusion）的限制，在端到端优化下实现了更高的压缩效率，尤其在超低比特率下表现优异。相比现有方法，CoD训练成本极低，且能在像素空间实现高质量重建。然而，该工作面向的是通用图像压缩，未针对文档内容结构、文本可读性或多模态对齐进行设计。
Reason 该论文聚焦于通用图像压缩任务，尽管其提出了一种高效的扩散基础模型CoD，并在压缩效率和训练成本上具有优势，但其目标场景是通用图像而非文档图像。方法上未涉及文档特有的结构建模、OCR去幻觉、细粒度文本定位或复杂排版理解等DIU核心问题。虽然高分辨率生成能力可能间接有助于图像重建，但整体技术路线与文档理解无直接关联，属于通用视觉压缩领域的平行工作，无法迁移解决DIU的关键痛点。
Abstract Existing diffusion codecs typically build on text-to-image diffusion foundation models like Stable Diffusion. However, text conditioning is suboptimal from a compression perspective, hindering the potential of downstream diffusion codecs, particularly at ultra-low bitrates. To address it, we introduce \textbf{CoD}, the first \textbf{Co}mpression-oriented \textbf{D}iffusion foundation model, trained from scratch to enable end-to-end optimization of both compression and generation. CoD is not a fixed codec but a general foundation model designed for various diffusion-based codecs. It offers several advantages: \textbf{High compression efficiency}, replacing Stable Diffusion with CoD in downstream codecs like DiffC achieves SOTA results, especially at ultra-low bitrates (e.g., 0.0039 bpp); \textbf{Low-cost and reproducible training}, 300$\times$ faster training than Stable Diffusion ($\sim$ 20 vs. $\sim$ 6,250 A100 GPU days) on entirely open image-only datasets; \textbf{Providing new insights}, e.g., We find pixel-space diffusion can achieve VTM-level PSNR with high perceptual quality and can outperform GAN-based codecs using fewer parameters. We hope CoD lays the foundation for future diffusion codec research. Codes will be released.
Authors Zhaoyang Jia, Zihan Zheng, Naifu Xue, Jiahao Li, Bin Li, Zongyu Guo, Xiaoyi Zhang, Houqiang Li, Yan Lu
Categories Computer Vision and Pattern Recognition
2
Neural Geometry Image-Based Representations with Optimal Transport (OT) WACV2026
Title CN 基于最优传输的神经几何图像表示
Keywords 3D网格表示 · 几何图像 · 最优传输 · 神经表示 · WACV
Summary 本文提出一种基于几何图像的3D网格神经表示方法，利用最优传输（OT）解决传统方法中采样不均的问题，并通过mipmap结构实现连续细节层次的高效重建。该方法无需解码器，可在单次前向传播中恢复高质量网格，在压缩率和重建精度上达到SOTA。
Reason 该论文研究的是3D网格的神经表示方法，核心是将不规则的3D网格转换为规则的几何图像网格以提升重建效率和存储性能。虽然使用了视觉化的结构处理，但其任务领域属于3D视觉与图形学，与文档图像理解（DIU）中的文本识别、布局分析、表格理解等核心任务无直接关联。尽管其‘几何图像+最优传输’的方法在高分辨率结构建模上有启发性，但迁移到文档图像中的细粒度定位或OCR去幻觉等问题极为困难，且未涉及多模态语言模型（VLM）或文本-视觉对齐机制。
Abstract Neural representations for 3D meshes are emerging as an effective solution for compact storage and efficient processing. Existing methods often rely on neural overfitting, where a coarse mesh is stored and progressively refined through multiple decoder networks. While this can restore high-quality surfaces, it is computationally expensive due to successive decoding passes and the irregular structure of mesh data. In contrast, images have a regular structure that enables powerful super-resolution and restoration frameworks, but applying these advantages to meshes is difficult because their irregular connectivity demands complex encoder-decoder architectures. Our key insight is that a geometry image-based representation transforms irregular meshes into a regular image grid, making efficient image-based neural processing directly applicable. Building on this idea, we introduce our neural geometry image-based representation, which is decoder-free, storage-efficient, and naturally suited for neural processing. It stores a low-resolution geometry-image mipmap of the surface, from which high-quality meshes are restored in a single forward pass. To construct geometry images, we leverage Optimal Transport (OT), which resolves oversampling in flat regions and undersampling in feature-rich regions, and enables continuous levels of detail (LoD) through geometry-image mipmapping. Experimental results demonstrate state-of-the-art storage efficiency and restoration accuracy, measured by compression ratio (CR), Chamfer distance (CD), and Hausdorff distance (HD).
Authors Xiang Gao, Yuanpeng Liu, Xinmu Wang, Jiazhi Li, Minghao Guo, Yu Guo, Xiyun Song, Heather Yu, Zhiqiang Lao, Xianfeng David Gu
Comment WACV2026 Rround 2 Accepted
Categories Computer Vision and Pattern Recognition
2
Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceivers
Title CN Edit2Perceive：图像编辑扩散模型是强大的密集感知器
Keywords 图像编辑 · 扩散模型 · 密集感知 · 结构保持 · 像素级预测
Summary 本文提出Edit2Perceive，一种基于图像编辑扩散模型的统一框架，用于深度、法线和图像抠图等密集感知任务。该方法利用FLUX.1 Kontext架构，通过全参数微调和像素空间一致性损失，在中间去噪状态中实现结构保持的精细化重建，并支持单步确定性推理，训练数据量小且推理速度快。实验表明其在三项任务上达到SOTA性能。
Reason 该论文聚焦于深度估计、法线预测和图像抠图等通用密集视觉感知任务，使用图像编辑扩散模型进行结构保持的像素级预测。尽管其方法在高分辨率细节处理上有优势，但任务本身与文档图像理解（DIU）的核心问题如文本识别、布局分析、表格理解或OCR去幻觉无关。未涉及多模态语言建模、文本-布局对齐或文档特有的复杂排版处理，无法迁移解决DIU中的关键痛点。
Abstract Recent advances in diffusion transformers have shown remarkable generalization in visual synthesis, yet most dense perception methods still rely on text-to-image (T2I) generators designed for stochastic generation. We revisit this paradigm and show that image editing diffusion models are inherently image-to-image consistent, providing a more suitable foundation for dense perception task. We introduce Edit2Perceive, a unified diffusion framework that adapts editing models for depth, normal, and matting. Built upon the FLUX.1 Kontext architecture, our approach employs full-parameter fine-tuning and a pixel-space consistency loss to enforce structure-preserving refinement across intermediate denoising states. Moreover, our single-step deterministic inference yields up to faster runtime while training on relatively small datasets. Extensive experiments demonstrate comprehensive state-of-the-art results across all three tasks, revealing the strong potential of editing-oriented diffusion transformers for geometry-aware perception.
Authors Yiqing Shi, Yiren Song, Mike Zheng Shou
Categories Computer Vision and Pattern Recognition
2
CountXplain: Interpretable Cell Counting with Prototype-Based Density Map Estimation Medical Imaging with Deep Learning 2025
Title CN CountXplain：基于原型的密度图估计实现可解释的细胞计数
Keywords 细胞计数 · 可解释性 · 密度图估计 · 原型网络 · 生物医学图像
Summary 本文提出了一种基于原型的密度图估计方法CountXplain，用于提升深度学习在细胞计数任务中的可解释性。通过在网络中引入原型层，模型能够学习细胞和背景伪影的代表性视觉模式，并生成对应于输入图像区域的解释，帮助生物学家理解模型决策过程。实验表明该方法在保持计数性能的同时提升了透明度和可信度。
Reason 该论文研究的是生物医学图像中的细胞计数问题，属于纯图像分析任务，且应用场景为医疗影像，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管其方法涉及可解释性设计和视觉原型学习，但目标任务与OCR、布局分析、表格识别等文档核心任务无关，无法迁移解决文档理解中的幻觉、复杂排版或多模态推理等问题。
Abstract Cell counting in biomedical imaging is pivotal for various clinical applications, yet the interpretability of deep learning models in this domain remains a significant challenge. We propose a novel prototype-based method for interpretable cell counting via density map estimation. Our approach integrates a prototype layer into the density estimation network, enabling the model to learn representative visual patterns for both cells and background artifacts. The learned prototypes were evaluated through a survey of biologists, who confirmed the relevance of the visual patterns identified, further validating the interpretability of the model. By generating interpretations that highlight regions in the input image most similar to each prototype, our method offers a clear understanding of how the model identifies and counts cells. Extensive experiments on two public datasets demonstrate that our method achieves interpretability without compromising counting effectiveness. This work provides researchers and clinicians with a transparent and reliable tool for cell counting, potentially increasing trust and accelerating the adoption of deep learning in critical biomedical applications. Code is available at https://github.com/NRT-D4/CountXplain.
Authors Abdurahman Ali Mohammed, Wallapak Tavanapong, Catherine Fonder, Donald S. Sakaguchi
Comment Medical Imaging with Deep Learning 2025
Categories Computer Vision and Pattern Recognition
2
Studying Maps at Scale: A Digital Investigation of Cartography and the Evolution of Figuration
Title CN 大规模研究地图：对制图学与图示演变的数字探究
Keywords 地图分析 · 语义分割 · 目标检测 · 文化遗产 · 符号系统
Summary 本文基于超过77万张地图记录和近10万幅数字化图像，构建了一个跨越六个世纪的制图数据集，并利用语义分割与目标检测模型识别土地类别和制图符号。研究揭示了地图构图中的语义对称性、中心化设计以及符号系统的地域一致性，分析了制图风格随时间的演变及其与政治、军事背景的关系。虽然技术上涉及图像内容提取，但重点在于文化与历史解释，而非提升文档理解模型的通用能力。
Reason 该论文属于将现有计算机视觉技术（如语义分割、目标检测）应用于历史地图内容分析的‘平行下游应用’，其核心是文化遗产研究而非文档图像理解的方法论创新。尽管使用了大规模地图数据并进行了符号系统分析，但未提出可用于解决DIU领域共性痛点（如OCR幻觉、复杂排版、长文档推理）的新架构、新训练范式或视觉表示方法。属于典型的应用型研究，无底层技术创新迁移价值。
Abstract This thesis presents methods and datasets to investigate cartographic heritage on a large scale and from a cultural perspective. Heritage institutions worldwide have digitized more than one million maps, and automated techniques now enable large-scale recognition and extraction of map content. Yet these methods have engaged little with the history of cartography, or the view that maps are semantic-symbolic systems, and cultural objects reflecting political and epistemic expectations. This work leverages a diverse corpus of 771,561 map records and 99,715 digitized images aggregated from 38 digital catalogs. After normalization, the dataset includes 236,925 contributors and spans six centuries, from 1492 to 1948. These data make it possible to chart geographic structures and the global chronology of map publication. The spatial focus of cartography is analyzed in relation to political dynamics, evidencing links between Atlantic maritime charting, the triangular trade, and colonial expansion. Further results document the progression of national, domestic focus and the impact of military conflicts on publication volumes. The research introduces semantic segmentation techniques and object detection models for the generic recognition of land classes and cartographic signs, trained on annotated data and synthetic images. The analysis of land classes shows that maps are designed images whose framing and composition emphasize features through centering and semantic symmetries. The study of cartographic figuration encodes 63 M signs and 25 M fragments into a latent visual space, revealing figurative shifts such as the replacement of relief hachures by terrain contours and showing that signs tend to form locally consistent systems. Analyses of collaboration and diffusion highlight the role of legitimacy, larger actors, and major cities in the spread of figurative norms and semiotic cultures.
Authors Remi Petitpierre
Comment PhD thesis, EPFL. 396 pages, 156 figures
Categories Computer Vision and Pattern Recognition, Computation and Language
2
FaVChat: Hierarchical Prompt-Query Guided Facial Video Understanding with Data-Efficient GRPO
Title CN FaVChat：基于分层提示-查询引导的高效数据GRPO面部视频理解
Keywords 视频理解 · 面部分析 · 多模态大模型 · 细粒度识别 · 数据高效学习
Summary 本文提出FaVChat，一种专用于细粒度面部视频理解的视频多模态大模型，通过分层提示机制从低层纹理、中层可学习查询区域到高层语义对齐进行特征提取，并结合一种数据高效的GRPO强化学习算法，在仅使用1万样本的情况下实现了优于现有方法的情感识别与推理性能。
Reason 该论文聚焦于面部视频理解，属于视频理解领域，且任务为情感识别、可解释推理等，与文档图像理解（DIU）核心任务如DocVQA、表格识别、文本信息抽取等无直接关联。尽管其提出的多级特征提取机制和数据高效强化学习方法具有一定方法论价值，但整体设计面向的是面部动态纹理与动作的细粒度建模，迁移到文档场景中处理OCR幻觉、复杂版式或长文档推理等问题的适用性极低。属于无关领域的专用方法，不符合关注方向。
Abstract Multi-modal large language models (MLLMs) have shown strong capability in video understanding but still struggle with fine-grained visual comprehension, as pure visual encoders often lose subtle cues essential for precise reasoning. To address this limitation, we propose FaVChat, a Video-MLLM specifically designed for fine-grained facial understanding. FaVChat introduces a multi-level prompt-guided feature extraction mechanism that progressively captures task-relevant information from three complementary stages: low-level transformer layers for textures and motion, medium-level learnable queries for discriminative regions, and high-level adaptive feature weighting for semantic alignment. These enriched features are dynamically fused and fed into the LLM to enable more accurate fine-grained reasoning. To further enhance the model's ability to capture fine-grained facial attributes and maximize the utility of limited data, we propose Date-Efficient GRPO, a novel data-efficient reinforcement learning (RL) algorithm that maximizes the utility of each training sample through per-instance utility estimation and dynamic lifecycle scheduling. Extensive zero-shot evaluations across emotion recognition, explainable reasoning, and textual expression analysis demonstrate that FaVChat achieves finer-grained understanding, stronger accuracy, and better generalization than existing Video-MLLMs, even when trained with only 10K RL samples.
Authors Fufangchen Zhao, Xuerui Qiu, Linrui Xu, Ming Li, Wenhao Jiang, Jinkai Zheng, Hehe Fan, Jian Gao, Danfeng Yan
Categories Computer Vision and Pattern Recognition
2
Information Extraction From Fiscal Documents Using LLMs ACM ICAIF 2025
Title CN 使用大语言模型从财政文档中提取信息
Keywords 信息提取 · 财政文档 · 大语言模型 · 层级验证 · OCR后处理
Summary 本文提出了一种基于大语言模型的多阶段流程，用于从印度卡纳塔克邦长达200多页的财政文档中提取结构化数据。方法利用财政表格的层级结构（如各级合计）设计多级验证机制，提升数值提取的准确性。虽然展示了LLM处理复杂表格和文档结构的能力，但整体框架依赖于传统OCR+LLM后处理范式，缺乏对OCR幻觉、细粒度定位或多模态对齐等根本问题的技术突破。
Reason 该论文属于典型的‘平行下游应用’，即将现有LLM技术应用于印度卡纳塔克邦的财政文档数据，并结合领域知识和层级验证进行结构化信息提取。尽管利用了财政表格的层级结构进行验证，但其方法论未提出新的视觉-语言建模机制、OCR抗幻觉策略或文档专用VLM架构，核心仍是将通用LLM作为文本理解工具在特定场景中的应用，无底层技术创新。
Abstract Large Language Models (LLMs) have demonstrated remarkable capabilities in text comprehension, but their ability to process complex, hierarchical tabular data remains underexplored. We present a novel approach to extracting structured data from multi-page government fiscal documents using LLM-based techniques. Applied to annual fiscal documents from the State of Karnataka in India (200+ pages), our method achieves high accuracy through a multi-stage pipeline that leverages domain knowledge, sequential context, and algorithmic validation. A large challenge with traditional OCR methods is the inability to verify the accurate extraction of numbers. When applied to fiscal data, the inherent structure of fiscal tables, with totals at each level of the hierarchy, allows for robust internal validation of the extracted data. We use these hierarchical relationships to create multi-level validation checks. We demonstrate that LLMs can read tables and also process document-specific structural hierarchies, offering a scalable process for converting PDF-based fiscal disclosures into research-ready databases. Our implementation shows promise for broader applications across developing country contexts.
Authors Vikram Aggarwal, Jay Kulkarni, Aditi Mascarenhas, Aakriti Narang, Siddarth Raman, Ajay Shah, Susan Thomas
Comment 6 pages. Presented at the AI for Financial Inclusion, Risk Modeling and Resilience in Emerging Markets workshop at ACM ICAIF 2025 Singapore
Categories Computation and Language
2
Sentence Smith: Controllable Edits for Evaluating Text Embeddings EMNLP 2025 (main)
Title CN 句子铁匠：用于评估文本嵌入的可控编辑
Keywords 可控文本生成 · 语义图 · 文本嵌入 · 对抗样本 · EMNLP2025
Summary 本文提出Sentence Smith框架，通过将句子解析为语义图、应用人工设计的语义规则进行修改，并从修改后的图生成新文本，最后使用蕴含检查验证变换有效性。该方法用于生成具有精细语义变化的难负例文本对，以评估文本嵌入模型的能力。整个过程透明且资源消耗低，依赖小型神经网络。
Reason 该论文属于纯NLP领域，聚焦于通过语义图进行可控文本生成以构建文本嵌入模型的评测数据集。尽管其方法在语义操作和推理透明性上有一定创新，但完全未涉及视觉或文档模态，与文档图像理解（DIU）或多模态VLM无关。其技术路线无法直接迁移解决OCR幻觉、细粒度定位或复杂排版等核心DIU问题，属于‘无关领域’中的纯语言任务。
Abstract Controllable and transparent text generation has been a long-standing goal in NLP. Almost as long-standing is a general idea for addressing this challenge: Parsing text to a symbolic representation, and generating from it. However, earlier approaches were hindered by parsing and generation insufficiencies. Using modern parsers and a safety supervision mechanism, we show how close current methods come to this goal. Concretely, we propose the Sentence Smith framework for English, which has three steps: 1. Parsing a sentence into a semantic graph. 2. Applying human-designed semantic manipulation rules. 3. Generating text from the manipulated graph. A final entailment check (4.) verifies the validity of the applied transformation. To demonstrate our framework's utility, we use it to induce hard negative text pairs that challenge text embedding models. Since the controllable generation makes it possible to clearly isolate different types of semantic shifts, we can evaluate text embedding models in a fine-grained way, also addressing an issue in current benchmarking where linguistic phenomena remain opaque. Human validation confirms that our transparent generation process produces texts of good quality. Notably, our way of generation is very resource-efficient, since it relies only on smaller neural networks.
Authors Hongji Li, Andrianos Michail, Reto Gubelmann, Simon Clematide, Juri Opitz
Comment EMNLP 2025 (main), this version fixes a subscript typo in Eq 1
Categories Computation and Language
2
MapFormer: Self-Supervised Learning of Cognitive Maps with Input-Dependent Positional Embeddings
Title CN MapFormer：基于输入依赖型位置嵌入的自监督认知地图学习
Keywords 认知地图 · Transformer · 自监督学习 · 位置编码 · 出分布泛化
Summary 本文提出MapFormer，一种基于Transformer的模型，通过输入依赖的位置嵌入来自监督地学习认知地图，实现结构与内容的解耦，在2D导航任务中展现出强出分布泛化能力。模型设计旨在模拟生物认知机制，适用于需要空间推理的场景。
Reason 该论文研究的是智能体在2D导航任务中通过Transformer架构自监督学习认知地图，核心是建模空间结构与记忆机制，属于具身智能与认知科学交叉领域。尽管其提出的输入依赖型位置编码具有理论新颖性，但其方法设计面向的是连续空间导航与路径积分任务，与文档图像理解（DIU）中的核心问题如OCR幻觉、细粒度文本定位、复杂版面分析或长文档推理无直接关联。无法迁移用于解决文档视觉结构建模或多模态对齐等关键挑战。
Abstract A cognitive map is an internal model which encodes the abstract relationships among entities in the world, giving humans and animals the flexibility to adapt to new situations, with a strong out-of-distribution (OOD) generalization that current AI systems still do not possess. To bridge this gap, we introduce MapFormers, new architectures based on Transformer models, which can learn cognitive maps from observational data and perform path integration in parallel, in a self-supervised manner. Cognitive maps are learned in the model by disentangling structural relationships in the inputs from their specific content, a property that can be achieved naturally by updating the positional encoding in Transformers with input-dependent matrices. We developed two variants of MapFormers that unify absolute and relative positional encoding to model episodic (EM) and working memory (WM), respectively. We tested MapFormers on several tasks, including a classic 2D navigation task, showing that our models can learn a cognitive map of the underlying space and generalize OOD (e.g., to longer sequences) with near-perfect performance, unlike current architectures. Together, these results demonstrate the superiority of models designed to learn a cognitive map, and the importance of introducing a structural bias for structure-content disentanglement, which can be achieved in Transformers with input-dependent positional encoding. MapFormers have broad applications in both neuroscience and AI, by explaining the neural mechanisms giving rise to cognitive maps, while allowing these relation models to be learned at scale.
Authors Victor Rambaud, Salvador Mascarenhas, Yair Lakretz
Comment 19 pages (29 with appendix), 8 figures
Categories Computation and Language
2
Representational Stability of Truth in Large Language Models
Title CN 大语言模型中真理的表示稳定性
Keywords 大语言模型 · 表示稳定性 · 事实性 · 线性探针 · 幻觉
Summary 本文提出“表示稳定性”概念，用于衡量大语言模型在面对不同真值定义扰动时，其内部表征对真/非真语句分类边界的稳健性。通过在线性探针中观察决策边界变化，发现对于训练数据中不存在实体的‘不熟悉非事实’陈述，模型表现出高度不稳定性（最多40%判断翻转），而来自虚构背景的‘熟悉非事实’则更稳定。研究表明，这种稳定性更多源于认知上的熟悉度而非语言形式。
Reason 该论文研究的是纯文本大语言模型内部对‘真’与‘假’陈述的表示稳定性，属于纯NLP领域的方法论分析，未涉及任何视觉或文档模态。其核心关注点为事实性判断的内部表征鲁棒性，尽管与幻觉问题有一定概念关联，但完全缺乏多模态对齐、视觉输入处理或文档结构建模等DIU关键要素，无法迁移到文档图像理解中的OCR幻觉或视觉-语言一致性问题。
Abstract Large language models (LLMs) are widely used for factual tasks such as "What treats asthma?" or "What is the capital of Latvia?". However, it remains unclear how stably LLMs encode distinctions between true, false, and neither-true-nor-false content in their internal probabilistic representations. We introduce representational stability as the robustness of an LLM's veracity representations to perturbations in the operational definition of truth. We assess representational stability by (i) training a linear probe on an LLM's activations to separate true from not-true statements and (ii) measuring how its learned decision boundary shifts under controlled label changes. Using activations from sixteen open-source models and three factual domains, we compare two types of neither statements. The first are fact-like assertions about entities we believe to be absent from any training data. We call these unfamiliar neither statements. The second are nonfactual claims drawn from well-known fictional contexts. We call these familiar neither statements. The unfamiliar statements induce the largest boundary shifts, producing up to $40\%$ flipped truth judgements in fragile domains (such as word definitions), while familiar fictional statements remain more coherently clustered and yield smaller changes ($\leq 8.2\%$). These results suggest that representational stability stems more from epistemic familiarity than from linguistic form. More broadly, our approach provides a diagnostic for auditing and training LLMs to preserve coherent truth assignments under semantic uncertainty, rather than optimizing for output accuracy alone.
Authors Samantha Dies, Courtney Maynard, Germans Savcisens, Tina Eliassi-Rad
Comment 25 pages, 24 figures
Categories Computation and Language
2
URLs Help, Topics Guide: Understanding Metadata Utility in LLM Training NeurIPS 2025
Title CN URL有帮助，主题来引导：理解元数据在LLM训练中的效用
Keywords 大语言模型 · 元数据 · 训练效率 · 可控生成 · URL上下文
Summary 本文系统评估了在LLM预训练中使用不同类型元数据（如URL、质量分数、主题）的影响，发现仅URL信息能加速训练，而主题和格式信息虽不能加速训练，但可用于控制生成结果。研究强调了上下文感知预训练在可控生成方面的优势。
Reason 该论文研究在语言模型预训练中引入URL、主题等元数据作为辅助输入的效果，属于纯文本大模型训练范畴。其核心关注点为生成可控性和训练效率，未涉及视觉模态或文档结构信息处理。尽管‘引导生成’（steering）机制理论上可能被迁移用于VLM的推理控制，但本文完全脱离多模态场景，且无任何与图像、布局、OCR或文档理解相关的实验设计和问题建模，对DIU领域无直接或可迁移的方法论贡献。
Abstract Large Language Models (LLMs) are commonly pretrained on vast corpora of text without utilizing contextual metadata such as source, quality, or topic, leading to a context-free learning paradigm. While recent studies suggest that adding metadata like URL information as context (i.e., auxiliary inputs not used in the loss calculation) can improve training efficiency and downstream performance, they offer limited understanding of which types of metadata are truly effective and under what conditions. In this work, we conduct a systematic evaluation and find that not all metadata types contribute equally. Only URL context speeds up training, whereas quality scores and topic/format domain information offer no clear benefit. Furthermore, the improved downstream performances of URL conditioning emerge only when longer prompts are used at inference time. In addition, we demonstrate that context-aware pretraining enables more controllable generation than context-free pretraining, in a classifier-free guidance fashion. Although topic and format metadata do not accelerate training, they are effective for steering outputs, offering human-interpretable control over generation.
Authors Dongyang Fan, Vinko Sabolčec, Martin Jaggi
Comment NeurIPS 2025, Camera Ready
Categories Computation and Language
2
A Multi-Agent LLM Framework for Multi-Domain Low-Resource In-Context NER via Knowledge Retrieval, Disambiguation and Reflective Analysis AAAI 2026 (Main Technical Track)
Title CN 一种通过知识检索、消歧和反思分析实现多领域低资源上下文内命名实体识别的多智能体LLM框架
Keywords 多智能体系统 · 上下文学习 · 命名实体识别 · 知识检索 · LLM反思
Summary 本文提出KDR-Agent，一个用于多领域低资源命名实体识别的多智能体框架，通过知识检索、实体消歧和自我反思机制提升上下文学习效果。实验表明其在多个领域的零样本和少样本NER任务上优于基线模型。然而，该工作完全基于纯文本输入，未涉及图像或文档布局信息，属于NLP下游任务的应用级研究。
Reason 该论文属于典型的‘平行下游应用’：其核心是将多Agent架构应用于低资源NER任务，尽管引入了知识检索与反思机制，但整体框架仍是特定NLP任务（NER）上的应用拓展。虽然‘反思’和‘消歧’机制具有一定方法价值，但其设计面向纯文本语义理解，未涉及视觉模态或文档结构信息处理，难以迁移至文档图像理解中的OCR幻觉、布局分析或细粒度定位等核心痛点。且任务场景不包含任何视觉-语言交互或图像输入，不属于DIU本题范畴。
Abstract In-context learning (ICL) with large language models (LLMs) has emerged as a promising paradigm for named entity recognition (NER) in low-resource scenarios. However, existing ICL-based NER methods suffer from three key limitations: (1) reliance on dynamic retrieval of annotated examples, which is problematic when annotated data is scarce; (2) limited generalization to unseen domains due to the LLM's insufficient internal domain knowledge; and (3) failure to incorporate external knowledge or resolve entity ambiguities. To address these challenges, we propose KDR-Agent, a novel multi-agent framework for multi-domain low-resource in-context NER that integrates Knowledge retrieval, Disambiguation, and Reflective analysis. KDR-Agent leverages natural-language type definitions and a static set of entity-level contrastive demonstrations to reduce dependency on large annotated corpora. A central planner coordinates specialized agents to (i) retrieve factual knowledge from Wikipedia for domain-specific mentions, (ii) resolve ambiguous entities via contextualized reasoning, and (iii) reflect on and correct model predictions through structured self-assessment. Experiments across ten datasets from five domains demonstrate that KDR-Agent significantly outperforms existing zero-shot and few-shot ICL baselines across multiple LLM backbones. The code and data can be found at https://github.com/MWXGOD/KDR-Agent.
Authors Wenxuan Mu, Jinzhong Ning, Di Zhao, Yijia Zhang
Comment This paper has been accepted by AAAI 2026 (Main Technical Track)
Categories Computation and Language
2
Safeguarding Privacy of Retrieval Data against Membership Inference Attacks: Is This Query Too Close to Home? EMNLP findings 2025
Title CN 防范针对检索数据的成员推断攻击：此查询是否过于贴近隐私？
Keywords RAG · 成员推断攻击 · 隐私保护 · 相似性检测 · 语言模型安全
Summary 本文提出一种基于相似性的成员推断攻击检测框架，用于保护RAG系统中私有检索数据的隐私。通过检测高相似性查询并采用‘检测-隐藏’策略来防御攻击，在不影响系统效用的前提下实现对多种先进MIA方法的防御，并可适配现有RAG系统。
Reason 该论文聚焦于RAG系统中的隐私安全问题，特别是防御成员推断攻击（MIA），属于纯NLP安全方向。虽然涉及‘幻觉’相关背景，但其核心是数据隐私保护，而非解决VLM或多模态文档理解中的OCR幻觉、细粒度定位或复杂排版等DIU核心痛点。方法难以迁移到文档图像理解领域，且属于平行应用层面的安全加固，不符合关注的底层技术创新标准。
Abstract Retrieval-augmented generation (RAG) mitigates the hallucination problem in large language models (LLMs) and has proven effective for personalized usages. However, delivering private retrieved documents directly to LLMs introduces vulnerability to membership inference attacks (MIAs), which try to determine whether the target data point exists in the private external database or not. Based on the insight that MIA queries typically exhibit high similarity to only one target document, we introduce a novel similarity-based MIA detection framework designed for the RAG system. With the proposed method, we show that a simple detect-and-hide strategy can successfully obfuscate attackers, maintain data utility, and remain system-agnostic against MIA. We experimentally prove its detection and defense against various state-of-the-art MIA methods and its adaptability to existing RAG systems.
Authors Yujin Choi, Youngjoo Park, Junyoung Byun, Jaewook Lee, Jinseong Park
Comment Accepted for EMNLP findings 2025
Categories Computation and Language
2
Robust Multimodal Sentiment Analysis with Distribution-Based Feature Recovery and Fusion ACM MM 2024
Title CN 基于分布特征恢复与融合的鲁棒多模态情感分析
Keywords 多模态情感分析 · 模态缺失 · 特征恢复 · 分布建模 · 鲁棒性
Summary 本文提出一种基于特征分布的恢复与融合方法（DRF），用于提升多模态情感分析在模态缺失或低质量情况下的鲁棒性。通过维护各模态的特征队列来估计分布，动态调整融合权重，并利用跨模态映射恢复缺失模态。实验表明该方法在多种破坏策略下优于现有方法。
Reason 该论文研究的是社交媒体中图像-文本对的情感分析，属于典型的跨模态情感任务，且核心关注点为模态缺失和低质量输入下的鲁棒性问题。其方法设计面向通用图像-文本场景，并未涉及文档结构、布局理解、OCR错误处理或细粒度视觉语言对齐等文档图像理解（DIU）的核心挑战。虽然‘特征恢复’和‘分布建模’在理论上具有一定通用性，但迁移到文档理解场景（如纠正OCR幻觉、解析复杂表格）缺乏直接路径，且不针对高分辨率、密集文本、长序列等DIU关键痛点。
Abstract As posts on social media increase rapidly, analyzing the sentiments embedded in image-text pairs has become a popular research topic in recent years. Although existing works achieve impressive accomplishments in simultaneously harnessing image and text information, they lack the considerations of possible low-quality and missing modalities. In real-world applications, these issues might frequently occur, leading to urgent needs for models capable of predicting sentiment robustly. Therefore, we propose a Distribution-based feature Recovery and Fusion (DRF) method for robust multimodal sentiment analysis of image-text pairs. Specifically, we maintain a feature queue for each modality to approximate their feature distributions, through which we can simultaneously handle low-quality and missing modalities in a unified framework. For low-quality modalities, we reduce their contributions to the fusion by quantitatively estimating modality qualities based on the distributions. For missing modalities, we build inter-modal mapping relationships supervised by samples and distributions, thereby recovering the missing modalities from available ones. In experiments, two disruption strategies that corrupt and discard some modalities in samples are adopted to mimic the low-quality and missing modalities in various real-world scenarios. Through comprehensive experiments on three publicly available image-text datasets, we demonstrate the universal improvements of DRF compared to SOTA methods under both two strategies, validating its effectiveness in robust multimodal sentiment analysis.
Authors Daiqing Wu, Dongbao Yang, Can Ma, Yu Zhou
Comment Accepted by ACM MM 2024
Categories Computation and Language
2
Investigating Representation Universality: Case Study on Genealogical Representations
Title CN 探究表征的普适性：以家谱结构表征为例
Keywords 表征学习 · 语言模型 · 可解释性 · 图结构 · 激活补丁
Summary 本文通过家谱问答任务和模型缝合实验，探索大语言模型在残差流中是否形成通用的树状结构表征。研究使用锥形探针和激活补丁验证表征的因果作用，并跨多种架构模型评估表征一致性。结果表明当前缺乏图结构表征的真值标准，限制了深入分析。
Reason 该论文研究大语言模型是否以通用几何结构编码图结构知识，聚焦于家谱问答中的内部表征分析。虽然涉及LLM对结构化知识的理解，但其任务与文档图像理解（DIU）无直接关联，未处理视觉输入、多模态对齐或OCR相关问题。属于纯NLP方向的可解释性研究，无法迁移解决DIU中的核心痛点如幻觉、复杂排版或细粒度定位。
Abstract Motivated by interpretability and reliability, we investigate whether large language models (LLMs) deploy universal geometric structures to encode discrete, graph-structured knowledge. To this end, we present two complementary experimental evidence that might support universality of graph representations. First, on an in-context genealogy Q&A task, we train a cone probe to isolate a tree-like subspace in residual stream activations and use activation patching to verify its causal effect in answering related questions. We validate our findings across five different models. Second, we conduct model stitching experiments across models of diverse architectures and parameter counts (OPT, Pythia, Mistral, and LLaMA, 410 million to 8 billion parameters), quantifying representational alignment via relative degradation in the next-token prediction loss. Generally, we conclude that the lack of ground truth representations of graphs makes it challenging to study how LLMs represent them. Ultimately, improving our understanding of LLM representations could facilitate the development of more interpretable, robust, and controllable AI systems.
Authors David D. Baek, Yuxiao Li, Max Tegmark
Comment 14 pages, 7 figures
Categories Artificial Intelligence
2
Findings of the BlackboxNLP 2025 Shared Task: Localizing Circuits and Causal Variables in Language Models
Title CN BlackboxNLP 2025共享任务成果：在语言模型中定位电路和因果变量
Keywords 机制可解释性 · 语言模型 · 电路定位 · 因果变量 · MIB基准
Summary 本文介绍了BlackboxNLP 2025共享任务的结果，旨在推动语言模型的机制可解释性研究。任务基于MIB基准，包含两个赛道：电路定位和因果变量定位。参赛团队通过集成方法、正则化策略以及低维非线性投影等方式提升了性能。该工作为理解语言模型内部机制提供了标准化评测平台。
Reason 该论文属于纯NLP领域的机制可解释性研究，聚焦于语言模型内部结构的分析方法评估。尽管其技术（如激活向量投影、正则化策略）在理论上可能对多模态模型的分析有启发，但并未涉及视觉输入、文档结构或任何文档理解任务。无法直接迁移解决DIU中的OCR幻觉、细粒度定位或复杂排版等核心问题，属于与文档图像理解无关的平行领域工作。
Abstract Mechanistic interpretability (MI) seeks to uncover how language models (LMs) implement specific behaviors, yet measuring progress in MI remains challenging. The recently released Mechanistic Interpretability Benchmark (MIB; Mueller et al., 2025) provides a standardized framework for evaluating circuit and causal variable localization. Building on this foundation, the BlackboxNLP 2025 Shared Task extends MIB into a community-wide reproducible comparison of MI techniques. The shared task features two tracks: circuit localization, which assesses methods that identify causally influential components and interactions driving model behavior, and causal variable localization, which evaluates approaches that map activations into interpretable features. With three teams spanning eight different methods, participants achieved notable gains in circuit localization using ensemble and regularization strategies for circuit discovery. With one team spanning two methods, participants achieved significant gains in causal variable localization using low-dimensional and non-linear projections to featurize activation vectors. The MIB leaderboard remains open; we encourage continued work in this standard evaluation framework to measure progress in MI research going forward.
Authors Dana Arad, Yonatan Belinkov, Hanjie Chen, Najoung Kim, Hosein Mohebbi, Aaron Mueller, Gabriele Sarti, Martin Tutek
Categories Artificial Intelligence, Computation and Language
2
A Multimodal Conversational Agent for Tabular Data Analysis
Title CN 一种用于表格数据分析的多模态对话代理
Keywords 对话代理 · 表格分析 · 多模态LLM · 代码生成 · 语音交互
Summary 本文提出Talk2Data，一个基于多模态大模型的对话式数据分析代理，支持语音或文本输入，可生成图表、统计结果或语音解释。系统整合了ASR、LLM代码生成、沙箱执行和TTS模块，实现在多轮对话中对表格数据的可视化与解释。实验显示高准确率和低延迟，强调交互性与可验证性。然而，这只是一个垂直领域的应用集成，缺乏对文档理解基础技术的贡献。
Reason 该论文属于典型的‘平行下游应用’：其核心是将现有LLM、ASR、TTS等技术集成到一个面向表格数据的对话系统中，应用场景明确为数据分析交互，但未提出任何针对文档图像理解（DIU）的底层方法创新。尽管涉及表格，但并未解决DocVQA中的OCR幻觉、复杂排版解析、视觉-文本对齐或长文档推理等根本问题。系统架构基于已有模块（Whisper、Qwen-Coder、Coqui）拼接而成，属于应用层封装，无新视觉表示、无推理机制突破，也无法迁移为提升DIU性能的通用方法。
Abstract Large language models (LLMs) can reshape information processing by handling data analysis, visualization, and interpretation in an interactive, context-aware dialogue with users, including voice interaction, while maintaining high performance. In this article, we present Talk2Data, a multimodal LLM-driven conversational agent for intuitive data exploration. The system lets users query datasets with voice or text instructions and receive answers as plots, tables, statistics, or spoken explanations. Built on LLMs, the suggested design combines OpenAI Whisper automatic speech recognition (ASR) system, Qwen-coder code generation LLM/model, custom sandboxed execution tools, and Coqui library for text-to-speech (TTS) within an agentic orchestration loop. Unlike text-only analysis tools, it adapts responses across modalities and supports multi-turn dialogues grounded in dataset context. In an evaluation of 48 tasks on three datasets, our prototype achieved 95.8% accuracy with model-only generation time under 1.7 seconds (excluding ASR and execution time). A comparison across five LLM sizes (1.5B-32B) revealed accuracy-latency-cost trade-offs, with a 7B model providing the best balance for interactive use. By routing between conversation with user and code execution, constrained to a transparent sandbox, with simultaneously grounding prompts in schema-level context, the Talk2Data agent reliably retrieves actionable insights from tables while making computations verifiable. In the article, except for the Talk2Data agent itself, we discuss implications for human-data interaction, trust in LLM-driven analytics, and future extensions toward large-scale multimodal assistants.
Authors Mohammad Nour Al Awad, Sergey Ivanov, Olga Tikhonova, Ivan Khodnenko
Comment \c{opyright} 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses
Categories Artificial Intelligence
2
Scaling Implicit Fields via Hypernetwork-Driven Multiscale Coordinate Transformations
Title CN 通过超网络驱动的多尺度坐标变换扩展隐式场
Keywords 隐式神经表示 · 超网络 · 坐标变换 · 多尺度建模 · 信号表示
Summary 本文提出HC-INR，一种基于超网络驱动的多尺度坐标变换的隐式神经表示方法，通过将输入域映射到解耦的潜在空间来缓解传统INR的表示瓶颈，并在图像拟合、形状重建和神经辐射场任务上验证了其高效性和保真度优势。
Reason 该论文研究的是隐式神经表示（INR）在图像、3D形状和辐射场等连续信号建模中的表示能力提升，核心贡献在于通过超网络学习输入坐标的自适应变换以实现多尺度表示。尽管其方法在高分辨率信号建模方面有潜力，但其应用场景集中在计算机图形学与视觉中的连续场表示（如NeRF），与文档图像理解（DIU）的核心任务（如文本识别、布局分析、表格理解、VQA）无直接关联。虽然‘高分辨率处理’是DIU的一个痛点，但该方法并未涉及多模态对齐、语言-视觉融合或文本结构建模，迁移到文档理解场景的技术路径不清晰，属于无关领域的技术探索。
Abstract Implicit Neural Representations (INRs) have emerged as a powerful paradigm for representing signals such as images, 3D shapes, signed distance fields, and radiance fields. While significant progress has been made in architecture design (e.g., SIREN, FFC, KAN-based INRs) and optimization strategies (meta-learning, amortization, distillation), existing approaches still suffer from two core limitations: (1) a representation bottleneck that forces a single MLP to uniformly model heterogeneous local structures, and (2) limited scalability due to the absence of a hierarchical mechanism that dynamically adapts to signal complexity. This work introduces Hyper-Coordinate Implicit Neural Representations (HC-INR), a new class of INRs that break the representational bottleneck by learning signal-adaptive coordinate transformations using a hypernetwork. HC-INR decomposes the representation task into two components: (i) a learned multiscale coordinate transformation module that warps the input domain into a disentangled latent space, and (ii) a compact implicit field network that models the transformed signal with significantly reduced complexity. The proposed model introduces a hierarchical hypernetwork architecture that conditions coordinate transformations on local signal features, enabling dynamic allocation of representation capacity. We theoretically show that HC-INR strictly increases the upper bound of representable frequency bands while maintaining Lipschitz stability. Extensive experiments across image fitting, shape reconstruction, and neural radiance field approximation demonstrate that HC-INR achieves up to 4 times higher reconstruction fidelity than strong INR baselines while using 30--60\% fewer parameters.
Authors Plein Versace
Categories Artificial Intelligence
2
Can a Second-View Image Be a Language? Geometric and Semantic Cross-Modal Reasoning for X-ray Prohibited Item Detection
Title CN 第二视角图像可以是一种语言吗？用于X光违禁品检测的几何与语义跨模态推理
Keywords X光检测 · 多视角推理 · 跨模态学习 · 视觉作为语言 · 安全检查
Summary 本文提出了一个名为DualXrayBench的多视角X光图像基准，包含4.5万对双视角图像及标注，并设计了一种将第二视角图像作为‘类语言模态’的跨模态推理模型GSR，通过几何-语义联合学习提升违禁品检测性能。研究主要服务于安检场景下的X光分析。
Reason 该论文聚焦于X光安检场景下的违禁品检测，属于特定垂直领域的应用研究（医疗/安全影像分析），其核心贡献是构建了一个多视角X光数据集（DualXrayBench）并提出一种利用双视角图像进行跨模态推理的模型GSR。尽管使用了多模态和推理设计，但整体属于‘平行下游应用’：方法未针对文档图像理解中的OCR幻觉、复杂版式、长文本推理等核心痛点提供可迁移的技术路径。虽然‘将图像视为语言’的思想有一定启发性，但其应用场景（X光双视角几何对齐）与文档图像的理解任务差异巨大，难以迁移到DIU领域。
Abstract Automatic X-ray prohibited items detection is vital for security inspection and has been widely studied. Traditional methods rely on visual modality, often struggling with complex threats. While recent studies incorporate language to guide single-view images, human inspectors typically use dual-view images in practice. This raises the question: can the second view provide constraints similar to a language modality? In this work, we introduce DualXrayBench, the first comprehensive benchmark for X-ray inspection that includes multiple views and modalities. It supports eight tasks designed to test cross-view reasoning. In DualXrayBench, we introduce a caption corpus consisting of 45,613 dual-view image pairs across 12 categories with corresponding captions. Building upon these data, we propose the Geometric (cross-view)-Semantic (cross-modality) Reasoner (GSR), a multimodal model that jointly learns correspondences between cross-view geometry and cross-modal semantics, treating the second-view images as a "language-like modality". To enable this, we construct the GSXray dataset, with structured Chain-of-Thought sequences: <top>, <side>, <conclusion>. Comprehensive evaluations on DualXrayBench demonstrate that GSR achieves significant improvements across all X-ray tasks, offering a new perspective for real-world X-ray inspection.
Authors Chuang Peng, Renshuai Tao, Zhongwei Ren, Xianglong Liu, Yunchao Wei
Comment 10 pages, 4 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
ReCode: Updating Code API Knowledge with Reinforcement Learning AAAI 2026
Title CN ReCode：使用强化学习更新代码API知识
Keywords 代码生成 · API更新 · 强化学习 · 大语言模型 · 版本迁移
Summary 论文提出ReCode框架，通过基于字符串相似度的强化学习奖励机制，使大语言模型能够适应外部库API的频繁更新，提升在动态环境下的代码生成能力。实验显示其在CodeUpdateArena等任务上优于监督微调方法，且对通用代码能力影响较小。
Reason 该论文聚焦于大模型在动态API环境下的代码生成适应问题，属于典型的下游应用型研究，核心是代码版本迁移任务。虽然采用了强化学习框架（如GRPO/DAPO）和自建数据集进行训练，但其方法本质是将现有LLM与规则奖励结合用于特定编程场景的微调，并未提出可迁移至文档图像理解领域的底层技术创新。任务场景与DIU无关，且不涉及视觉输入、多模态对齐、OCR去幻觉或复杂版面处理等核心痛点，属于‘平行应用’范畴。
Abstract Large Language Models (LLMs) exhibit remarkable code generation capabilities but falter when adapting to frequent updates in external library APIs. This critical limitation, stemming from reliance on outdated API knowledge from their training data, even with access to current documentation, impedes reliable code generation in dynamic environments. To tackle this issue, we propose ReCode (rule-based Reinforcement learning for Code Update), a novel framework that mimics human programmer adaptation to API changes. Specifically, we construct a dataset of approximately 2,000 data entries to train the LLMs to perform version migration based on updated information. Then, we introduce a modified string similarity metric for code evaluation as the reward for reinforcement learning. Our experiments demonstrate that ReCode substantially boosts LLMs' code generation performance in dynamic API scenarios, especially on the unseen CodeUpdateArena task. Crucially, compared to supervised fine-tuning, ReCode has less impact on LLMs' general code generation abilities. We apply ReCode on various LLMs and reinforcement learning algorithms (GRPO and DAPO), all achieving consistent improvements. Notably, after training, Qwen2.5-Coder-7B outperforms that of the 32B parameter code instruction-tuned model and the reasoning model with the same architecture. Code is available at https://github.com/zjunlp/ReCode.
Authors Haoze Wu, Yunzhi Yao, Wenhao Yu, Ningyu Zhang
Comment AAAI 2026
Categories Artificial Intelligence, Computation and Language
2
Beyond Superficial Forgetting: Thorough Unlearning through Knowledge Density Estimation and Block Re-insertion AAAI 2026
Title CN 超越表面遗忘：通过知识密度估计与块重插入实现彻底遗忘
Keywords 机器遗忘 · 知识密度估计 · 模型编辑 · 大语言模型 · 隐私合规
Summary 本文提出KUnBR方法，通过知识密度估计识别富含有害知识的网络层，并设计块重插入策略来绕过梯度阻碍，实现更彻底的知识遗忘。实验表明该方法在遗忘效果和模型保留能力上优于现有方法。
Reason 该论文研究机器遗忘（Machine Unlearning）中的知识移除问题，属于LLM安全与合规的范畴。尽管其方法在模型编辑和知识控制方面有一定技术深度，但属于纯文本模态下的模型维护技术，未涉及视觉输入、多模态对齐或文档结构建模。无法迁移解决文档图像理解中的OCR幻觉、细粒度定位或复杂排版解析等核心痛点，属于与DIU无关的平行领域工作。
Abstract Machine unlearning, which selectively removes harmful knowledge from a pre-trained model without retraining from scratch, is crucial for addressing privacy, regulatory compliance, and ethical concerns in Large Language Models (LLMs). However, existing unlearning methods often struggle to thoroughly remove harmful knowledge, leaving residual harmful knowledge that can be easily recovered. To address these limitations, we propose Knowledge Density-Guided Unlearning via Blocks Reinsertion (KUnBR), a novel approach that first identifies layers with rich harmful knowledge and then thoroughly eliminates the harmful knowledge via re-insertion strategy. Our method introduces knowledge density estimation to quantify and locate layers containing the most harmful knowledge, enabling precise unlearning. Additionally, we design a layer re-insertion strategy that extracts and re-inserts harmful knowledge-rich layers into the original LLM, bypassing gradient obstruction caused by cover layers and ensuring effective gradient propagation during unlearning. Extensive experiments conducted on several unlearning and general capability benchmarks demonstrate that KUnBR achieves state-of-the-art forgetting performance while maintaining model utility.
Authors Feng Guo, Yuntao Wen, Shen Gao, Junshuo Zhang, Shuo Shang
Comment Accepted by AAAI 2026
Categories Artificial Intelligence
2
KGpipe: Generation and Evaluation of Pipelines for Data Integration into Knowledge Graphs
Title CN KGpipe：用于知识图谱数据集成的管道生成与评估
Keywords 知识图谱 · 数据集成 · 管道框架 · LLM应用 · 基准测试
Summary 本文提出了KGpipe框架，用于构建和执行将异构数据（RDF、JSON、文本）集成到知识图谱的端到端管道，并提供了一个基准来评估不同管道的效果。系统支持结合传统工具与大语言模型，强调可复现性和多源数据整合能力。
Reason 该论文聚焦于知识图谱构建中的数据集成流程自动化，属于典型的‘平行下游应用’。其核心贡献是框架性地组合现有信息提取和融合工具（包括LLM），但并未提出针对文档图像理解（DIU）中的关键问题（如OCR幻觉、复杂版面解析、细粒度视觉定位等）的底层方法创新。尽管使用了LLM功能，但应用场景为结构化/半结构化数据到知识图谱的映射，与文档视觉理解无直接关联。
Abstract Building high-quality knowledge graphs (KGs) from diverse sources requires combining methods for information extraction, data transformation, ontology mapping, entity matching, and data fusion. Numerous methods and tools exist for each of these tasks, but support for combining them into reproducible and effective end-to-end pipelines is still lacking. We present a new framework, KGpipe for defining and executing integration pipelines that can combine existing tools or LLM (Large Language Model) functionality. To evaluate different pipelines and the resulting KGs, we propose a benchmark to integrate heterogeneous data of different formats (RDF, JSON, text) into a seed KG. We demonstrate the flexibility of KGpipe by running and comparatively evaluating several pipelines integrating sources of the same or different formats using selected performance and quality metrics.
Authors Marvin Hofer, Erhard Rahm
Comment 15 KG pipelines (9 single source, 6 multi source)
Categories Artificial Intelligence
2
Weakly-supervised Latent Models for Task-specific Visual-Language Control
Title CN 用于任务特定视觉语言控制的弱监督潜在模型
Keywords 视觉语言控制 · 潜在动力学模型 · 空间接地 · 弱监督学习 · 自主巡检
Summary 本文提出一种弱监督的潜在动力学模型，用于在高危环境中实现AI代理的语言指令驱动的空间控制（如将检测对象置于视野中心）。该模型仅需目标状态监督，利用共享潜在空间中的动作诱导变化进行学习，并引入全局动作嵌入和辅助损失来稳定训练。实验显示其在空间对齐任务上成功率从58%提升至71%，并具备对未见图像和指令的泛化能力。
Reason 该论文研究的是在危险环境中的自主巡检任务，核心是通过弱监督学习构建一个任务特定的潜在动力学模型，以实现语言指令与视觉空间对齐（如无人机调整视角使目标居中）。虽然涉及视觉-语言控制和空间定位，但其应用场景和方法设计均面向具身智能（embodied agent）和机器人控制，属于‘无关领域’负面清单。其技术路线难以直接迁移至文档图像理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Autonomous inspection in hazardous environments requires AI agents that can interpret high-level goals and execute precise control. A key capability for such agents is spatial grounding, for example when a drone must center a detected object in its camera view to enable reliable inspection. While large language models provide a natural interface for specifying goals, using them directly for visual control achieves only 58\% success in this task. We envision that equipping agents with a world model as a tool would allow them to roll out candidate actions and perform better in spatially grounded settings, but conventional world models are data and compute intensive. To address this, we propose a task-specific latent dynamics model that learns state-specific action-induced shifts in a shared latent space using only goal-state supervision. The model leverages global action embeddings and complementary training losses to stabilize learning. In experiments, our approach achieves 71\% success and generalizes to unseen images and instructions, highlighting the potential of compact, domain-specific latent dynamics models for spatial alignment in autonomous inspection.
Authors Xian Yeow Lee, Lasitha Vidyaratne, Gregory Sin, Ahmed Farahat, Chetan Gupta
Categories Artificial Intelligence
2
Cross-Disciplinary Knowledge Retrieval and Synthesis: A Compound AI Architecture for Scientific Discovery
Title CN 跨学科知识检索与合成：一种用于科学发现的复合AI架构
Keywords RAG · 多智能体系统 · 科学发现 · 知识检索 · 跨学科
Summary 论文提出了名为BioSage的复合AI系统，通过集成LLM、RAG和多个专用代理（如检索、翻译、推理代理），实现跨学科（尤其是生物与AI）的知识发现与合成。系统在多个科学问答基准上优于基础LLM和RAG方法，并强调用户中心设计和可追溯性。目前主要处理文本任务，多模态扩展为未来方向。
Reason 该论文提出一个基于LLM+RAG的多智能体系统BioSage，用于跨学科科学发现，属于典型的‘平行下游应用’。其核心是将现有技术（RAG、Agent协作）应用于科研场景，并未提出可迁移至文档图像理解（DIU）的新方法或架构创新。尽管提到了未来扩展到图表和表格的多模态推理，但当前工作未涉及视觉输入或文档结构理解，对解决OCR幻觉、复杂排版、细粒度定位等DIU核心问题无直接贡献。
Abstract The exponential growth of scientific knowledge has created significant barriers to cross-disciplinary knowledge discovery, synthesis and research collaboration. In response to this challenge, we present BioSage, a novel compound AI architecture that integrates LLMs with RAG, orchestrated specialized agents and tools to enable discoveries across AI, data science, biomedical, and biosecurity domains. Our system features several specialized agents including the retrieval agent with query planning and response synthesis that enable knowledge retrieval across domains with citation-backed responses, cross-disciplinary translation agents that align specialized terminology and methodologies, and reasoning agents that synthesize domain-specific insights with transparency, traceability and usability. We demonstrate the effectiveness of our BioSage system through a rigorous evaluation on scientific benchmarks (LitQA2, GPQA, WMDP, HLE-Bio) and introduce a new cross-modal benchmark for biology and AI, showing that our BioSage agents outperform vanilla and RAG approaches by 13\%-21\% powered by Llama 3.1. 70B and GPT-4o models. We perform causal investigations into compound AI system behavior and report significant performance improvements by adding RAG and agents over the vanilla models. Unlike other systems, our solution is driven by user-centric design principles and orchestrates specialized user-agent interaction workflows supporting scientific activities including but not limited to summarization, research debate and brainstorming. Our ongoing work focuses on multimodal retrieval and reasoning over charts, tables, and structured scientific data, along with developing comprehensive multimodal benchmarks for cross-disciplinary discovery. Our compound AI solution demonstrates significant potential for accelerating scientific advancement by reducing barriers between traditionally siloed domains.
Authors Svitlana Volkova, Peter Bautista, Avinash Hiriyanna, Gabriel Ganberg, Isabel Erickson, Zachary Klinefelter, Nick Abele, Hsien-Te Kao, Grant Engberson
Categories Artificial Intelligence
2
Beyond Words and Pixels: A Benchmark for Implicit World Knowledge Reasoning in Generative Models
Title CN 超越文字与像素：生成模型中隐式世界知识推理的基准
Keywords 文本到图像生成 · 世界知识推理 · 基准评测 · 多智能体评估 · 物理因果
Summary 本文提出了PicWorld，首个针对文本到图像生成模型在隐式世界知识和物理因果推理能力方面的综合评测基准，涵盖1100个提示词，并设计了基于多智能体的评估系统PW-Agent来分解并验证视觉证据。对17个主流T2I模型的评测表明它们普遍存在推理缺陷。
Reason 该论文聚焦于文本到图像（T2I）生成模型的评估，提出一个名为PicWorld的基准，用于测试生成图像在隐式世界知识和物理因果推理方面的能力。其核心任务是生成图像的合理性评估，属于纯图像生成领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。虽然其多智能体评估框架PW-Agent涉及推理结构，但设计目标是服务于T2I模型的评测，而非解决OCR幻觉、密集文本识别、复杂版面分析等DIU核心问题。无法迁移至文档理解场景。
Abstract Text-to-image (T2I) models today are capable of producing photorealistic, instruction-following images, yet they still frequently fail on prompts that require implicit world knowledge. Existing evaluation protocols either emphasize compositional alignment or rely on single-round VQA-based scoring, leaving critical dimensions such as knowledge grounding, multi-physics interactions, and auditable evidence-substantially undertested. To address these limitations, we introduce PicWorld, the first comprehensive benchmark that assesses the grasp of implicit world knowledge and physical causal reasoning of T2I models. This benchmark consists of 1,100 prompts across three core categories. To facilitate fine-grained evaluation, we propose PW-Agent, an evidence-grounded multi-agent evaluator to hierarchically assess images on their physical realism and logical consistency by decomposing prompts into verifiable visual evidence. We conduct a thorough analysis of 17 mainstream T2I models on PicWorld, illustrating that they universally exhibit a fundamental limitation in their capacity for implicit world knowledge and physical causal reasoning to varying degrees. The findings highlight the need for reasoning-aware, knowledge-integrative architectures in future T2I systems.
Authors Tianyang Han, Junhao Su, Junjie Hu, Peizhen Yang, Hengyu Shi, Junfeng Luo, Jialin Gao
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension NeurIPS 2025
Title CN Video-RAG：视觉对齐的检索增强型长视频理解
Keywords 视频理解 · 检索增强生成 · 多模态大模型 · 外部工具集成 · 长上下文
Summary 本文提出Video-RAG，一种无需训练的检索增强生成框架，通过从视频中提取音频、文字和物体等视觉对齐的辅助信息，结合帧和查询输入现有LVLM，以提升长视频理解性能。方法轻量、兼容性强，在多个基准上表现优异，甚至超过Gemini和GPT-4o。
Reason 该论文属于视频理解领域的下游应用，尽管提出了检索增强的框架，但其核心是将现有外部工具（如OCR、语音识别、目标检测）提取的信息作为辅助文本输入到LVLM中，用于长视频理解任务。虽然使用了OCR等技术，但整体架构是针对视频时序建模设计的，并未解决文档图像理解中的核心痛点如复杂版面分析、密集文本推理或OCR幻觉抑制。且其方法本质是多模态信息拼接的‘Wrapper’式应用，不具备向DocVQA或文档端到端理解迁移的方法论价值。
Abstract Existing large video-language models (LVLMs) struggle to comprehend long videos correctly due to limited context. To address this problem, fine-tuning long-context LVLMs and employing GPT-based agents have emerged as promising solutions. However, fine-tuning LVLMs would require extensive high-quality data and substantial GPU resources, while GPT-based agents would rely on proprietary models (e.g., GPT-4o). In this paper, we propose Video Retrieval-Augmented Generation (Video-RAG), a training-free and cost-effective pipeline that employs visually-aligned auxiliary texts to help facilitate cross-modality alignment while providing additional information beyond the visual content. Specifically, we leverage open-source external tools to extract visually-aligned information from pure video data (e.g., audio, optical character, and object detection), and incorporate the extracted information into an existing LVLM as auxiliary texts, alongside video frames and queries, in a plug-and-play manner. Our Video-RAG offers several key advantages: (i) lightweight with low computing overhead due to single-turn retrieval; (ii) easy implementation and compatibility with any LVLM; and (iii) significant, consistent performance gains across long video understanding benchmarks, including Video-MME, MLVU, and LongVideoBench. Notably, our model demonstrates superior performance over proprietary models like Gemini-1.5-Pro and GPT-4o when utilized with a 72B model.
Authors Yongdong Luo, Xiawu Zheng, Guilin Li, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, Rongrong Ji
Comment Accepted at NeurIPS 2025. Camera-ready version
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
Zero-Reference Joint Low-Light Enhancement and Deblurring via Visual Autoregressive Modeling with VLM-Derived Modulation AAAI 2026
Title CN 基于视觉自回归建模与VLM导出调制的无参考低光增强与去模糊联合方法
Keywords 图像增强 · 去模糊 · 视觉自回归模型 · VLM引导 · 无监督学习
Summary 本文提出一种基于视觉自回归（VAR）建模的无监督框架，用于联合进行低光图像增强与去模糊。通过引入VLM导出的可见性评分来自适应调节光照曲线，并设计了频域感知的旋转位置编码（SF-RoPE）以增强对模糊结构的建模能力。此外，采用递归相位域调制策略，在VLM评估的模糊评分指导下迭代优化以减少伪影。整个框架无需配对数据，在多个基准上达到SOTA。
Reason 该论文研究的是真实场景下低光照和模糊图像的联合恢复问题，属于图像增强与底层视觉任务，与文档图像理解（DIU）的核心任务如文本识别、布局分析、表格理解等无直接关联。尽管使用了VLM-derived modulation和VAR建模，但其目标是提升图像质量而非解决OCR幻觉、复杂排版或长文档推理等问题。方法聚焦于动态光照调节和频域去模糊，难以迁移到文档理解中的语义解析或多模态对齐任务。属于无关领域（底层视觉+图像恢复），不符合DIU本题关注范畴。
Abstract Real-world dark images commonly exhibit not only low visibility and contrast but also complex noise and blur, posing significant restoration challenges. Existing methods often rely on paired data or fail to model dynamic illumination and blur characteristics, leading to poor generalization. To tackle this, we propose a generative framework based on visual autoregressive (VAR) modeling, guided by perceptual priors from the vision-language model (VLM). Specifically, to supply informative conditioning cues for VAR models, we deploy an adaptive curve estimation scheme to modulate the diverse illumination based on VLM-derived visibility scores. In addition, we integrate dynamic and spatial-frequency-aware Rotary Positional Encodings (SF-RoPE) into VAR to enhance its ability to model structures degraded by blur. Furthermore, we propose a recursive phase-domain modulation strategy that mitigates blur-induced artifacts in the phase domain via bounded iterative refinement guided by VLM-assessed blur scores. Our framework is fully unsupervised and achieves state-of-the-art performance on benchmark datasets.
Authors Wei Dong, Han Zhou, Junwei Lin, Jun Chen
Comment Accepted by AAAI 2026; First Var-based method for joint LLIE and deblurring
Categories Computer Vision and Pattern Recognition
2
Breaking Forgetting: Training-Free Few-Shot Class-Incremental Learning via Conditional Diffusion
Title CN 打破遗忘：通过条件扩散实现无需训练的少样本类增量学习
Keywords 增量学习 · 条件扩散 · 无需训练 · 多模态学习 · 灾难性遗忘
Summary 本文提出CD-FSCIL，一种无需训练的少样本类增量学习框架，用条件扩散过程替代传统梯度更新，以缓解灾难性遗忘并降低计算开销。同时结合LLM生成的文本描述进行多模态表征增强，在标准FSCIL基准上取得了SOTA性能。
Reason 该论文研究的是少样本类增量学习（FSCIL）中的灾难性遗忘问题，提出了一种基于条件扩散的训练免费框架，并引入多模态语言描述来增强表征。尽管使用了LLM生成的语言信息，但其核心任务属于通用图像分类的持续学习范畴，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。方法上聚焦于分类器更新机制替代梯度优化，无法迁移解决OCR幻觉、复杂排版建模、细粒度视觉定位或长文档推理等DIU关键痛点，属于无关领域的平行方法应用。
Abstract Efforts to overcome catastrophic forgetting in Few-Shot Class-Incremental Learning (FSCIL) have primarily focused on developing more effective gradient-based optimization strategies. In contrast, little attention has been paid to the training cost explosion that inevitably arises as the number of novel classes increases, a consequence of relying on gradient learning even under extreme data scarcity. More critically, since FSCIL typically provides only a few samples for each new class, gradient-based updates not only induce severe catastrophic forgetting on base classes but also hinder adaptation to novel ones. This paper seeks to break this long-standing limitation by asking: Can we design a training-free FSCIL paradigm that entirely removes gradient optimization? We provide an affirmative answer by uncovering an intriguing connection between gradient-based optimization and the Conditional Diffusion process. Building on this observation, we propose a Conditional Diffusion-driven FSCIL (CD-FSCIL) framework that substitutes the conventional gradient update process with a diffusion-based generative transition, enabling training-free incremental adaptation while effectively mitigating forgetting. Furthermore, to enhance representation under few-shot constraints, we introduce a multimodal learning strategy that integrates visual features with natural language descriptions automatically generated by Large Language Models (LLMs). This synergy substantially alleviates the sample scarcity issue and improves generalization across novel classes. Extensive experiments on mainstream FSCIL benchmarks demonstrate that our method not only achieves state-of-the-art performance but also drastically reduces computational and memory overhead, marking a paradigm shift toward training-free continual adaptation.
Authors Haidong Kang, Ketong Qian, Yi Lu
Categories Computer Vision and Pattern Recognition
2
VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format
Title CN VideoLLM知道何时发言：通过视频-文本二重交互格式增强时间敏感型视频理解
Keywords VideoLLM · 实时交互 · 时间定位 · 视频理解 · 多模态交互
Summary 本文提出一种视频-文本二重交互格式，允许用户和模型在视频播放过程中随时插入文本消息，从而实现类比‘二重奏’的实时交互。作者构建了MMDuetIT数据集并训练VideoLLM，在多个时间敏感型任务上取得显著提升，并支持实时响应。然而，该工作聚焦于视频流的时间动态建模，与文档图像理解任务无直接关联。
Reason 该论文研究的是视频大语言模型在实时视频流场景下的交互格式，核心任务为时间定位、高亮检测和密集视频描述，属于视频理解领域。尽管其提出的‘二重交互’机制在实时响应上有创新，但应用场景与文档图像理解（DIU）差异巨大，且方法依赖于连续视频帧的时间动态性，难以迁移到静态文档的细粒度识别、OCR去幻觉或复杂排版理解等核心痛点。属于典型的非相关领域工作。
Abstract Recent researches on video large language models (VideoLLM) predominantly focus on model architectures and training datasets, leaving the interaction format between the user and the model under-explored. In existing works, users often interact with VideoLLMs by using the entire video and a query as input, after which the model generates a response. This interaction format constrains the application of VideoLLMs in scenarios such as live-streaming comprehension where videos do not end and responses are required in a real-time manner, and also results in unsatisfactory performance on time-sensitive tasks that requires localizing video segments. In this paper, we focus on a video-text duet interaction format. This interaction format is characterized by the continuous playback of the video, and both the user and the model can insert their text messages at any position during the video playback. When a text message ends, the video continues to play, akin to the alternative of two performers in a duet. We construct MMDuetIT, a video-text training dataset designed to adapt VideoLLMs to video-text duet interaction format. We also introduce the Multi-Answer Grounded Video Question Answering (MAGQA) task to benchmark the real-time response ability of VideoLLMs. Trained on MMDuetIT, MMDuet demonstrates that adopting the video-text duet interaction format enables the model to achieve significant improvements in various time-sensitive tasks (76% CIDEr on YouCook2 dense video captioning, 90\% mAP on QVHighlights highlight detection and 25% R@0.5 on Charades-STA temporal video grounding) with minimal training efforts, and also enable VideoLLMs to reply in a real-time manner as the video plays.
Authors Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Jiansheng Wei, Huishuai Zhang, Dongyan Zhao
Comment 9 pages
Categories Computer Vision and Pattern Recognition, Computation and Language
2
Robust Posterior Diffusion-based Sampling via Adaptive Guidance Scale
Title CN 基于自适应引导尺度的鲁棒后验扩散采样
Keywords 扩散模型 · 逆问题 · 图像重建 · 自适应采样 · 去模糊
Summary 本文提出了一种名为AdaPS的自适应后验扩散采样方法，通过动态调整扩散过程中的似然步长来提升图像重建质量。该方法基于两种中间似然梯度近似之间的一致性设计观测依赖的加权方案，在超分辨率和去模糊等任务上实现了优于现有方法的感知质量，且无需任务特定调参。然而，该工作聚焦于通用图像恢复，与文档图像理解无直接关联。
Reason 该论文研究的是扩散模型在图像逆问题（如超分辨率、去模糊）中的采样策略优化，属于纯图像生成与恢复领域。其核心贡献是设计一种自适应调节数据保真项与先验项平衡的机制，但并未涉及任何多模态语言模型、文档结构理解或视觉-语言对齐的内容。无法迁移用于解决文档图像理解中的OCR幻觉、复杂排版解析或长文档推理等关键问题，属于无关领域。
Abstract Diffusion models have recently emerged as powerful generative priors for solving inverse problems, achieving state-of-the-art results across various imaging tasks. A central challenge in this setting lies in balancing the contribution of the prior with the data fidelity term: overly aggressive likelihood updates may introduce artifacts, while conservative updates can slow convergence or yield suboptimal reconstructions. In this work, we propose an adaptive likelihood step-size strategy to guide the diffusion process for inverse-problem formulations. Specifically, we develop an observation-dependent weighting scheme based on the agreement between two different approximations of the intractable intermediate likelihood gradients, that adapts naturally to the diffusion schedule, time re-spacing, and injected stochasticity. The resulting approach, Adaptive Posterior diffusion Sampling (AdaPS), is hyperparameter-free and improves reconstruction quality across diverse imaging tasks - including super-resolution, Gaussian deblurring, and motion deblurring - on CelebA-HQ and ImageNet-256 validation sets. AdaPS consistently surpasses existing diffusion-based baselines in perceptual quality with minimal or no loss in distortion, without any task-specific tuning. Extensive ablation studies further demonstrate its robustness to the number of diffusion steps, observation noise levels, and varying stochasticity.
Authors Liav Hen, Tom Tirer, Raja Giryes, Shady Abu-Hussein
Categories Computer Vision and Pattern Recognition
2
VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment
Title CN VLCE：一种用于灾害评估中图像描述的知识增强框架
Keywords 图像描述 · 灾害评估 · 多模态学习 · 外部知识融合 · 视觉语言模型
Summary 本文提出VLCE框架，通过融合ConceptNet和WordNet等外部知识来增强灾害场景下的图像描述生成能力，采用CNN-LSTM和Vision Transformer两种架构分别处理卫星与无人机影像，在xBDD和RescueNet数据集上优于LLaVA、QwenVL等基线模型。目标是将视觉信息转化为可操作的情报，提升灾害评估的自动化水平。
Reason 该论文属于典型的‘平行下游应用’，将现有VLM与外部知识结合应用于灾害评估场景，任务聚焦于图像描述生成（image captioning），而非文档图像理解（DIU）的核心任务如DocVQA、布局分析或表格识别。尽管使用了多模态模型和外部知识库，但其方法未针对OCR幻觉、密集文本处理、复杂排版或长文档推理等DIU痛点提出底层创新，属于特定领域的应用包装，无技术源头贡献。
Abstract The processes of classification and segmentation utilizing artificial intelligence play a vital role in the automation of disaster assessments. However, contemporary VLMs produce details that are inadequately aligned with the objectives of disaster assessment, primarily due to their deficiency in domain knowledge and the absence of a more refined descriptive process. This research presents the Vision Language Caption Enhancer (VLCE), a dedicated multimodal framework aimed at integrating external semantic knowledge from ConceptNet and WordNet to improve the captioning process. The objective is to produce disaster-specific descriptions that effectively convert raw visual data into actionable intelligence. VLCE utilizes two separate architectures: a CNN-LSTM model that incorporates a ResNet50 backbone, pretrained on EuroSat for satellite imagery (xBD dataset), and a Vision Transformer developed for UAV imagery (RescueNet dataset). In various architectural frameworks and datasets, VLCE exhibits a consistent advantage over baseline models such as LLaVA and QwenVL. Our optimal configuration reaches an impressive 95.33\% on InfoMetIC for UAV imagery while also demonstrating strong performance across satellite imagery. The proposed framework signifies a significant transition from basic visual classification to the generation of comprehensive situational intelligence, demonstrating immediate applicability for implementation in real-time disaster assessment systems.
Authors Md. Mahfuzur Rahman, Kishor Datta Gupta, Marufa Kamal, Fahad Rahman, Sunzida Siddique, Ahmed Rafi Hasan, Mohd Ariful Haque, Roy George
Comment 30 pages, 40 figures, 3 algorithms
Categories Computer Vision and Pattern Recognition
2
TRANSPORTER: Transferring Visual Semantics from VLM Manifolds
Title CN TRANSPORTER：从VLM流形迁移视觉语义
Keywords VLM可解释性 · 视频生成 · 最优传输 · 语义嵌入 · logits-to-video
Summary 本文提出了TRANSPORTER方法，旨在通过从VLM的logits生成高保真视频来揭示模型决策背后的规则。该方法利用文本到视频生成模型的视觉保真能力，学习VLM高层语义嵌入空间中的最优传输耦合，从而生成反映描述变化（如对象属性、动作副词）的视频。虽然在模型可解释性上提供了新视角，但其应用局限于视频生成和理解领域。
Reason 该论文研究的是视频生成与模型可解释性之间的关系，提出通过logits生成反映VLM预测逻辑的视频，属于视频理解与生成领域。尽管涉及VLM的语义空间分析，但其核心任务是模型解释的可视化，而非文档图像理解中的实际问题（如OCR幻觉、复杂排版处理等）。方法难以迁移到DIU场景，且目标领域为视频而非静态文档，与文档VQA、布局分析等任务无直接关联。
Abstract How do video understanding models acquire their answers? Although current Vision Language Models (VLMs) reason over complex scenes with diverse objects, action performances, and scene dynamics, understanding and controlling their internal processes remains an open challenge. Motivated by recent advancements in text-to-video (T2V) generative models, this paper introduces a logits-to-video (L2V) task alongside a model-independent approach, TRANSPORTER, to generate videos that capture the underlying rules behind VLMs' predictions. Given the high-visual-fidelity produced by T2V models, TRANSPORTER learns an optimal transport coupling to VLM's high-semantic embedding spaces. In turn, logit scores define embedding directions for conditional video generation. TRANSPORTER generates videos that reflect caption changes over diverse object attributes, action adverbs, and scene context. Quantitative and qualitative evaluations across VLMs demonstrate that L2V can provide a fidelity-rich, novel direction for model interpretability that has not been previously explored.
Authors Alexandros Stergiou
Comment Project page: https://alexandrosstergiou.github.io/TRANSPORTER
Categories Computer Vision and Pattern Recognition
2
RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation System AAAI 2026
Title CN RoadSceneVQA：面向智能交通系统路侧感知的视觉问答基准
Keywords VQA · 智能交通 · 多模态大模型 · 行为推理 · 场景理解
Summary 本文提出了RoadSceneVQA数据集及配套模型RoadMind，用于提升路侧感知系统在复杂交通场景下的自然语言交互与推理能力。通过CogniAnchor Fusion和AD-CoT机制增强多模态融合与推理过程，在交通意图理解和行为推理上取得提升。
Reason 该论文聚焦于智能交通中的路侧场景视觉问答（VQA），构建了一个针对交通参与者行为、意图和交互推理的数据集，并提出了相应的多模态融合模块与推理策略。尽管使用了MLLM和CoT等技术，但其核心是将现有VLM方法应用于特定垂直领域（交通监控），属于典型的‘平行下游应用’。任务场景与文档图像理解（DIU）差异显著，未涉及OCR、文本布局、表格识别或文档幻觉等核心问题，方法也无法直接迁移至文档理解任务。
Abstract Current roadside perception systems mainly focus on instance-level perception, which fall short in enabling interaction via natural language and reasoning about traffic behaviors in context. To bridge this gap, we introduce RoadSceneVQA, a large-scale and richly annotated visual question answering (VQA) dataset specifically tailored for roadside scenarios. The dataset comprises 34,736 diverse QA pairs collected under varying weather, illumination, and traffic conditions, targeting not only object attributes but also the intent, legality, and interaction patterns of traffic participants. RoadSceneVQA challenges models to perform both explicit recognition and implicit commonsense reasoning, grounded in real-world traffic rules and contextual dependencies. To fully exploit the reasoning potential of Multi-modal Large Language Models (MLLMs), we further propose CogniAnchor Fusion (CAF), a vision-language fusion module inspired by human-like scene anchoring mechanisms. Moreover, we propose the Assisted Decoupled Chain-of-Thought (AD-CoT) to enhance the reasoned thinking via CoT prompting and multi-task learning. Based on the above, we propose the baseline model RoadMind. Experiments on RoadSceneVQA and CODA-LM benchmark show that the pipeline consistently improves both reasoning accuracy and computational efficiency, allowing the MLLM to achieve state-of-the-art performance in structural traffic perception and reasoning tasks.
Authors Runwei Guan, Rongsheng Hu, Shangshu Chen, Ningyuan Xiao, Xue Xia, Jiayang Liu, Beibei Chen, Ziren Tang, Ningwei Ouyang, Shaofeng Liang, Yuxuan Fan, Wanjie Sun, Yutao Yue
Comment 9 pages, 6 figures, accepted by AAAI 2026. The model is also called Dream, to the other me in the world forever
Categories Computer Vision and Pattern Recognition
2
EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning
Title CN EgoVITA：面向第一人称视频推理的规划与验证学习
Keywords 第一人称视频 · 视频推理 · 多模态大模型 · 强化学习 · 规划与验证
Summary 本文提出EgoVITA，一种基于强化学习的多模态大模型框架，用于提升第一人称视频中的推理能力。该方法通过交替执行‘第一人称规划’和‘第三人称验证’两个阶段，利用Group Relative Policy Optimization（GRPO）训练模型生成因果一致且视觉可验证的动作计划。在EgoBlind和EgoOrient等数据集上显著优于基线模型。
Reason 该论文研究的是第一人称视频（egocentric video）理解中的意图和动作推理问题，属于视频理解领域。其核心方法是通过强化学习框架在第一人称视角下进行计划生成，并在第三人称视角下进行视觉一致性验证。尽管其‘规划-验证’机制具有一定的方法论价值，但应用场景与文档图像理解（DIU）差异较大：文档图像为静态、结构化、高密度文本布局，而该工作针对动态、非结构化、视角变化剧烈的视频流。此外，所提出的GRPO训练框架和双视角切换机制难以直接迁移至DocVQA或OCR-free文档理解任务中解决OCR幻觉、密集文本识别或复杂版式分析等核心痛点。属于无关领域应用，不符合关注方向。
Abstract Reasoning about intentions and actions from a first-person (egocentric) perspective remains a fundamental challenge for multimodal large language models (MLLMs). Unlike third-person (exocentric) videos that capture scenes from an outside observer, egocentric videos reflect the actor's continuously changing viewpoint, introducing partial observability, limited field of view, and self-referenced motion. We introduce $\textbf{EgoVITA}$, a reinforcement learning framework that enables MLLMs to reason through structured planning and verification. Built on Group Relative Policy Optimization (GRPO), EgoVITA alternates between two stages: (1) an $\textbf{egocentric planning phase}$, where the model reasons from a first-person viewpoint to predict a step-by-step plan of future actions, and (2) an $\textbf{exocentric verification phase}$, where it switches to a third-person perspective to check the visual and logical consistency of that plan. Through GRPO, the model learns to make plans that are causally predictive of upcoming visual observations, leading to more coherent and visually grounded reasoning. EgoVITA achieves significant gains on egocentric reasoning tasks, outperforming the baseline Qwen2.5-VL-7B by $\mathbf{+7.7}$ on EgoBlind and $\mathbf{+4.4}$ on EgoOrient, while maintaining strong generalization on exocentric video tasks.
Authors Yogesh Kulkarni, Pooyan Fazli
Categories Computer Vision and Pattern Recognition
2
Towards Robust and Fair Next Visit Diagnosis Prediction under Noisy Clinical Notes with Large Language Models AAAI 2026 1st Workshop on Safe, Ethical, Certified, Uncertainty-aware, Robust, and Explainable AI for Health (SECURE-AI4H)
Title CN 基于大语言模型的嘈杂临床笔记下鲁棒且公平的下次就诊诊断预测
Keywords 大语言模型 · 临床决策支持 · 噪声鲁棒性 · 公平性 · 思维链
Summary 本文研究了在临床文本存在噪声的情况下，如何利用大语言模型进行鲁棒且公平的下次就诊诊断预测。作者提出了一个基于临床知识的标签简化方案和分层思维链策略，提升了模型在不同人群子组间的稳定性和鲁棒性。工作重点在于医疗AI的安全与可靠性，而非文档理解或多模态建模。
Reason 该论文聚焦于医疗领域的临床决策支持系统，使用LLM进行下一访问诊断预测，并研究在文本噪声下的鲁棒性与公平性。尽管引入了分层思维链（CoT）策略以模拟医生推理，但其本质是将现有NLP技术应用于特定垂直领域（医疗文本），属于典型的‘平行下游应用’。未提出针对文档图像理解中的OCR幻觉、复杂排版或视觉-语言对齐等核心问题的新方法，无法迁移至DIU领域。
Abstract A decade of rapid advances in artificial intelligence (AI) has opened new opportunities for clinical decision support systems (CDSS), with large language models (LLMs) demonstrating strong reasoning abilities on timely medical tasks. However, clinical texts are often degraded by human errors or failures in automated pipelines, raising concerns about the reliability and fairness of AI-assisted decision-making. Yet the impact of such degradations remains under-investigated, particularly regarding how noise-induced shifts can heighten predictive uncertainty and unevenly affect demographic subgroups. We present a systematic study of state-of-the-art LLMs under diverse text corruption scenarios, focusing on robustness and equity in next-visit diagnosis prediction. To address the challenge posed by the large diagnostic label space, we introduce a clinically grounded label-reduction scheme and a hierarchical chain-of-thought (CoT) strategy that emulates clinicians' reasoning. Our approach improves robustness and reduces subgroup instability under degraded inputs, advancing the reliable use of LLMs in CDSS. We release code at https://github.com/heejkoo9/NECHOv3.
Authors Heejoon Koo
Comment Accepted by the Association for the Advancement of Artificial Intelligence (AAAI) 2026 1st Workshop on Safe, Ethical, Certified, Uncertainty-aware, Robust, and Explainable AI for Health (SECURE-AI4H)
Categories Computation and Language
2
Table Comprehension in Building Codes using Vision Language Models and Domain-Specific Fine-Tuning
Title CN 使用视觉语言模型和领域特定微调理解建筑规范中的表格
Keywords 表格理解 · 视觉语言模型 · LoRA微调 · 建筑规范 · RAG
Summary 本文研究如何在建筑规范文档中通过视觉语言模型进行表格理解，比较了直接图像输入与转换为LaTeX后输入两种方式，并采用LoRA对多个VLM在领域数据上进行微调。实验表明直接输入效果更好，且微调显著提升了性能，其中Qwen2.5-VL-3B-Instruct相对准确率提升超过100%。研究强调了参数高效微调在专业领域结构化数据理解中的潜力。
Reason 该论文属于典型的‘平行下游应用’：将现有VLM和LoRA微调技术应用于建筑规范这一特定领域的表格理解任务，未提出任何新的架构、训练范式或通用方法论。其核心是‘RAG + VLM + LoRA’在垂直场景的组合使用，无底层技术创新，对DIU领域的OCR幻觉、复杂排版建模、视觉压缩等核心问题无直接贡献或可迁移机制。
Abstract Building codes contain critical information for ensuring safety, regulatory compliance, and informed decision-making in construction and engineering. Automated question answering systems over such codes enable quick and accurate access to specific regulatory clauses, improving efficiency and reducing errors. Retrieval-Augmented Generation (RAG) systems are essential for this task as they combine the precision of information retrieval with the generative capabilities of language models. However, tabular data are challenging to extract as they often involve complex layouts, merged cells, multi-row headers, and embedded semantic relationships that are not easily captured by traditional natural language processing techniques and Vision Language Models (VLMs). This paper explores and compares two methods for extracting information from tabular data in building codes using several pre-trained VLMs. First, a direct input method is used, where the image of the page is input directly into the VLMs, which are then tasked with answering questions based on the image. Second, an indirect input method is introduced, which involves converting an image of a page containing tables into the LaTeX code and then answering inquires based on the LaTeX-based input. The experiments find that the direct input method generally resulted in higher accuracy than the indirect input method. To further improve the performance, we fine-tuned each VLM using Low Rank Adaptation (LoRA) on a domain-specific tabular dataset. The fine-tuned models exhibited substantial improvements, with Qwen2.5-VL-3B-Instruct achieving relative accuracy gains exceeding 100%. Our results highlight the potential of parameter-efficient fine-tuning methods to adapt powerful VLMs for understanding complex structured data in specialized fields, such as building code interpretation and regulatory compliance.
Authors Mohammad Aqib, Mohd Hamza, Ying Hei Chui, Qipei Mei
Categories Computation and Language
2
"AGI" team at SHROOM-CAP: Data-Centric Approach to Multilingual Hallucination Detection using XLM-RoBERTa AACL-IJCNLP 2025
Title CN SHROOM-CAP中的“AGI”团队：基于XLM-RoBERTa的数据驱动多语言幻觉检测方法
Keywords 幻觉检测 · 多语言 · XLM-RoBERTa · 数据增强 · 科学文本
Summary 本文针对多语言科学文本中LLM生成的幻觉问题，提出一种以数据为中心的方法，通过整合并平衡五个现有数据集，构建了一个包含12万多样本的训练集，并在XLM-RoBERTa-Large上进行微调，在SHROOM-CAP 2025多个语言任务中取得良好排名。重点在于解决低资源语言的零样本幻觉检测。
Reason 该论文属于典型的平行应用型研究，聚焦于科学文本生成中的多语言幻觉检测任务，使用XLM-RoBERTa在特定数据集上进行微调。虽然涉及‘幻觉’这一关键词，但其任务完全基于纯文本NLP，未引入任何视觉或文档布局信息，与文档图像理解（DIU）中的OCR幻觉、视觉-文本不一致等核心问题无直接关联。方法层面为标准的数据增强与模型微调，无架构或推理机制创新，无法迁移至VLM-based DIU场景。
Abstract The detection of hallucinations in multilingual scientific text generated by Large Language Models (LLMs) presents significant challenges for reliable AI systems. This paper describes our submission to the SHROOM-CAP 2025 shared task on scientific hallucination detection across 9 languages. Unlike most approaches that focus primarily on model architecture, we adopted a data-centric strategy that addressed the critical issue of training data scarcity and imbalance. We unify and balance five existing datasets to create a comprehensive training corpus of 124,821 samples (50% correct, 50% hallucinated), representing a 172x increase over the original SHROOM training data. Our approach fine-tuned XLM-RoBERTa-Large with 560 million parameters on this enhanced dataset, achieves competitive performance across all languages, including \textbf{2nd place in Gujarati} (zero-shot language) with Factuality F1 of 0.5107, and rankings between 4th-6th place across the remaining 8 languages. Our results demonstrate that systematic data curation can significantly outperform architectural innovations alone, particularly for low-resource languages in zero-shot settings.
Authors Harsh Rathva, Pruthwik Mishra, Shrikant Malviya
Comment Accepted to the 1st Workshop on Confabulation, Hallucinations & Overgeneration in Multilingual and Practical Settings (CHOMPS) at AACL-IJCNLP 2025
Categories Computation and Language
2
Comparison of Text-Based and Image-Based Retrieval in Multimodal Retrieval Augmented Generation Large Language Model Systems
Title CN 多模态检索增强生成大语言模型系统中文本与图像检索的比较
Keywords RAG · 多模态检索 · 图像嵌入 · 信息丢失 · 财务文档
Summary 本文对比了在多模态RAG系统中两种检索方式：一种是将图像总结为文本后进行文本检索，另一种是直接使用多模态嵌入进行图像检索。实验表明，直接多模态嵌入检索在mAP@5和nDCG@5上显著优于基于LLM摘要的方法，并能生成更准确、事实一致的答案。作者认为LLM摘要会导致信息丢失，而原生多模态嵌入能更好保留视觉上下文。
Reason 该论文研究的是在RAG系统中使用文本摘要还是直接图像嵌入进行检索的问题，虽然涉及文档中的图表和表格，但其核心是信息检索流程的比较，并未提出任何新的文档理解模型、架构或底层方法。它属于将现有技术（如多模态嵌入）应用于金融文档问答系统的下游应用，且聚焦于特定领域（财务电话会议），缺乏对DIU领域的方法论创新。无法为解决OCR幻觉、复杂排版或长文档推理等根本问题提供可迁移的技术路径。
Abstract Recent advancements in Retrieval-Augmented Generation (RAG) have enabled Large Language Models (LLMs) to access multimodal knowledge bases containing both text and visual information such as charts, diagrams, and tables in financial documents. However, existing multimodal RAG systems rely on LLM-based summarization to convert images into text during preprocessing, storing only text representations in vector databases, which causes loss of contextual information and visual details critical for downstream retrieval and question answering. To address this limitation, we present a comprehensive comparative analysis of two retrieval approaches for multimodal RAG systems, including text-based chunk retrieval (where images are summarized into text before embedding) and direct multimodal embedding retrieval (where images are stored natively in the vector space). We evaluate all three approaches across 6 LLM models and a two multi-modal embedding models on a newly created financial earnings call benchmark comprising 40 question-answer pairs, each paired with 2 documents (1 image and 1 text chunk). Experimental results demonstrate that direct multimodal embedding retrieval significantly outperforms LLM-summary-based approaches, achieving absolute improvements of 13% in mean average precision (mAP@5) and 11% in normalized discounted cumulative gain. These gains correspond to relative improvements of 32% in mAP@5 and 20% in nDCG@5, providing stronger evidence of their practical impact. We additionally find that direct multimodal retrieval produces more accurate and factually consistent answers as measured by LLM-as-a-judge pairwise comparisons. We demonstrate that LLM summarization introduces information loss during preprocessing, whereas direct multimodal embeddings preserve visual context for retrieval and inference.
Authors Elias Lumer, Alex Cardenas, Matt Melich, Myles Mason, Sara Dieter, Vamse Kumar Subbiah, Pradeep Honaganahalli Basavaraju, Roberto Hernandez
Categories Computation and Language
2
Enhancing Large Language Models for Automated Homework Assessment in Undergraduate Circuit Analysis 2025 Frontiers in Education (FIE) Conference
Title CN 增强大语言模型在本科电路分析作业评估中的应用
Keywords 大语言模型 · 作业评估 · 提示工程 · 教育技术 · 电路分析
Summary 本文提出了一种基于多步提示、上下文数据增强和定向提示的GPT-4o增强流程，用于提升本科电路分析课程作业的自动评估准确率，使正确响应率从74.71%提升至97.70%。研究聚焦于教育场景下的LLM应用优化，而非底层模型或文档理解能力的改进。
Reason 该论文属于典型的‘平行下游应用’，即将现有LLM技术（GPT-4o）通过提示工程应用于特定教育场景（电路分析作业批改），未提出任何面向文档图像理解（DIU）的方法论创新。尽管涉及自动化评估，但其核心不涉及OCR、文档布局分析、表格识别或视觉-文本对齐等DIU关键问题，也无法为解决OCR幻觉、复杂排版或长文档推理提供可迁移的技术路径。
Abstract This research full paper presents an enhancement pipeline for large language models (LLMs) in assessing homework for an undergraduate circuit analysis course, aiming to improve LLMs' capacity to provide personalized support to electrical engineering students. Existing evaluations have demonstrated that GPT-4o possesses promising capabilities in assessing student homework in this domain. Building on these findings, we enhance GPT-4o's performance through multi-step prompting, contextual data augmentation, and the incorporation of targeted hints. These strategies effectively address common errors observed in GPT-4o's responses when using simple prompts, leading to a substantial improvement in assessment accuracy. Specifically, the correct response rate for GPT-4o increases from 74.71% to 97.70% after applying the enhanced prompting and augmented data on entry-level circuit analysis topics. This work lays a foundation for the effective integration of LLMs into circuit analysis instruction and, more broadly, into engineering education.
Authors Liangliang Chen, Huiru Xie, Zhihao Qin, Yiming Guo, Jacqueline Rohde, Ying Zhang
Comment Accepted to 2025 Frontiers in Education (FIE) Conference
Categories Artificial Intelligence
2
A Roadmap to Guide the Integration of LLMs in Hierarchical Planning AAAI Workshop on Planning in the Era of LLMs
Title CN 指导大语言模型在分层规划中集成的路线图
Keywords 大语言模型 · 分层规划 · 自动化规划 · 推理 · 路线图
Summary 本文提出了一条将大语言模型（LLM）集成到分层规划（HP）中的路线图，包括分类法、标准化评测基准和初步实验结果。结果显示当前LLM在生成正确层次化计划方面表现极差，但可作为未来研究的基线。
Reason 该论文聚焦于将LLM应用于分层规划（Hierarchical Planning），属于具身智能与自动化规划领域，与文档图像理解（DIU）或视觉-语言多模态任务无直接关联。尽管其探讨了LLM在复杂推理中的使用，但所提方法缺乏对视觉输入、OCR抗幻觉、细粒度定位或多模态对齐等DIU核心问题的建模能力，无法迁移至文档理解场景。
Abstract Recent advances in Large Language Models (LLMs) are fostering their integration into several reasoning-related fields, including Automated Planning (AP). However, their integration into Hierarchical Planning (HP), a subfield of AP that leverages hierarchical knowledge to enhance planning performance, remains largely unexplored. In this preliminary work, we propose a roadmap to address this gap and harness the potential of LLMs for HP. To this end, we present a taxonomy of integration methods, exploring how LLMs can be utilized within the HP life cycle. Additionally, we provide a benchmark with a standardized dataset for evaluating the performance of future LLM-based HP approaches, and present initial results for a state-of-the-art HP planner and LLM planner. As expected, the latter exhibits limited performance (3\% correct plans, and none with a correct hierarchical decomposition) but serves as a valuable baseline for future approaches.
Authors Israel Puerta-Merino, Carlos Núñez-Molina, Pablo Mesejo, Juan Fernández-Olivares
Comment 5 pages, 0 figures, to be published in the AAAI Workshop on Planning in the Era of LLMs ( https://llmforplanning.github.io )
Categories Artificial Intelligence
2
Towards a General Framework for HTN Modeling with LLMs International Conference on Automated Planning and Scheduling (ICAPS) 2025
Title CN 面向基于大语言模型的HTN建模通用框架
Keywords 大语言模型 · 分层任务网络 · 自动规划 · PDDL · L2HP
Summary 本文提出了L2HP，一个用于通过大语言模型生成分层任务网络（HTN）规划模型的框架，作为对非分层PDDL建模工具L2P的扩展。作者在PlanBench数据集上比较了LLM在普通规划与分层规划中的建模能力，发现分层情况下的语法有效性显著更低，表明LLM在生成结构化层次规划方面仍面临重大挑战。研究强调了提升LLM在分层规划中表现的必要性。
Reason 该论文属于具身智能与自动规划领域的应用研究，聚焦于使用LLM生成分层任务网络（HTN）规划模型，并非针对文档图像理解（DIU）任务。其核心贡献是扩展L2P工具以支持HP建模并在PlanBench上进行实验，属于典型的‘平行应用’类工作——即将现有LLM能力应用于特定AI规划场景，未提出可用于解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心问题的新方法。尽管涉及LLM结构化输出，但无视觉输入、无多模态对齐机制、无视觉压缩或高分辨率处理设计，无法迁移至文档理解场景。
Abstract The use of Large Language Models (LLMs) for generating Automated Planning (AP) models has been widely explored; however, their application to Hierarchical Planning (HP) is still far from reaching the level of sophistication observed in non-hierarchical architectures. In this work, we try to address this gap. We present two main contributions. First, we propose L2HP, an extension of L2P (a library to LLM-driven PDDL models generation) that support HP model generation and follows a design philosophy of generality and extensibility. Second, we apply our framework to perform experiments where we compare the modeling capabilities of LLMs for AP and HP. On the PlanBench dataset, results show that parsing success is limited but comparable in both settings (around 36\%), while syntactic validity is substantially lower in the hierarchical case (1\% vs. 20\% of instances). These findings underscore the unique challenges HP presents for LLMs, highlighting the need for further research to improve the quality of generated HP models.
Authors Israel Puerta-Merino, Carlos Núñez-Molina, Pablo Mesejo, Juan Fernández-Olivares
Comment 10 pages, 5 figures, to be published in the Workshop on Planning in the Era of LLMs ( LM4Plan - https://llmforplanning.github.io ) and the Workshop on Hierarchical Planning ( HPlan - https://icaps25.icaps-conference.org/program/workshops/hplan/ ), both in the International Conference on Automated Planning and Scheduling (ICAPS) 2025
Categories Artificial Intelligence
2
Nested Unfolding Network for Real-World Concealed Object Segmentation
Title CN 用于真实场景隐蔽物体分割的嵌套展开网络
Keywords 隐蔽物体分割 · 深度展开网络 · 图像恢复 · Vision-Language Model · 自一致性损失
Summary 本文提出了一种名为NUN的嵌套展开网络，用于解决真实场景下的隐蔽物体分割问题。通过在分割网络中嵌入一个具备抗退化能力的子展开网络（DeRUN），实现图像恢复与分割的解耦与互优化。DeRUN利用VLM动态推断图像退化类型并指导恢复，而主干SODUN则专注于前景背景分离。同时引入图像质量评估机制选择最优恢复结果，增强模型鲁棒性。实验表明该方法在干净与退化数据上均表现优异。
Reason 该论文研究的是隐蔽物体分割（COS），属于通用图像分割任务，聚焦于视觉恢复与分割联合建模，并非文档图像理解（DIU）任务。尽管使用了VLM进行退化语义引导，但其核心目标是图像去噪与前景分离，与文档中的OCR幻觉、排版理解、细粒度文本定位等痛点无关。属于无关领域应用，且无明显可迁移至DIU的方法论创新。
Abstract Deep unfolding networks (DUNs) have recently advanced concealed object segmentation (COS) by modeling segmentation as iterative foreground-background separation. However, existing DUN-based methods (RUN) inherently couple background estimation with image restoration, leading to conflicting objectives and requiring pre-defined degradation types, which are unrealistic in real-world scenarios. To address this, we propose the nested unfolding network (NUN), a unified framework for real-world COS. NUN adopts a DUN-in-DUN design, embedding a degradation-resistant unfolding network (DeRUN) within each stage of a segmentation-oriented unfolding network (SODUN). This design decouples restoration from segmentation while allowing mutual refinement. Guided by a vision-language model (VLM), DeRUN dynamically infers degradation semantics and restores high-quality images without explicit priors, whereas SODUN performs reversible estimation to refine foreground and background. Leveraging the multi-stage nature of unfolding, NUN employs image-quality assessment to select the best DeRUN outputs for subsequent stages, naturally introducing a self-consistency loss that enhances robustness. Extensive experiments show that NUN achieves a leading place on both clean and degraded benchmarks. Code will be released.
Authors Chunming He, Rihan Zhang, Dingming Zhang, Fengyang Xiao, Deng-Ping Fan, Sina Farsiu
Comment 6 figures, 14 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
UnfoldLDM: Deep Unfolding-based Blind Image Restoration with Latent Diffusion Priors
Title CN UnfoldLDM：基于深度展开的具有潜在扩散先验的盲图像恢复
Keywords 盲图像恢复 · 深度展开网络 · 潜在扩散模型 · 图像去模糊 · 纹理增强
Summary 本文提出UnfoldLDM，一种结合深度展开网络与潜在扩散模型的盲图像恢复框架。通过多粒度退化感知模块（MGDA）估计未知退化并去除，再利用退化抵抗的LDM提取不变先验，并通过OCFormer恢复高频细节。实验表明其在多种图像恢复任务上表现优异，并可作为即插即用模块集成到现有DUN中。
Reason 该论文属于纯图像修复领域，尽管其方法结合了深度展开网络与潜在扩散模型，但核心任务是通用图像去模糊、去噪等盲图像恢复问题，与文档图像理解（DIU）中的OCR幻觉、文本识别、布局分析或VLM多模态推理无关。虽然高分辨率图像恢复技术理论上可能间接提升文档图像质量，但该工作未针对文档结构、文本特征或多模态对齐进行任何设计，属于典型的‘上游图像处理’而非可迁移的DIU底层方法。且其应用方向为通用CV任务，无法直接解决文档理解中的语义推理或视觉-语言对齐问题。
Abstract Deep unfolding networks (DUNs) combine the interpretability of model-based methods with the learning ability of deep networks, yet remain limited for blind image restoration (BIR). Existing DUNs suffer from: (1) \textbf{Degradation-specific dependency}, as their optimization frameworks are tied to a known degradation model, making them unsuitable for BIR tasks; and (2) \textbf{Over-smoothing bias}, resulting from the direct feeding of gradient descent outputs, dominated by low-frequency content, into the proximal term, suppressing fine textures. To overcome these issues, we propose UnfoldLDM to integrate DUNs with latent diffusion model (LDM) for BIR. In each stage, UnfoldLDM employs a multi-granularity degradation-aware (MGDA) module as the gradient descent step. MGDA models BIR as an unknown degradation estimation problem and estimates both the holistic degradation matrix and its decomposed forms, enabling robust degradation removal. For the proximal step, we design a degradation-resistant LDM (DR-LDM) to extract compact degradation-invariant priors from the MGDA output. Guided by this prior, an over-smoothing correction transformer (OCFormer) explicitly recovers high-frequency components and enhances texture details. This unique combination ensures the final result is degradation-free and visually rich. Experiments show that our UnfoldLDM achieves a leading place on various BIR tasks and benefits downstream tasks. Moreover, our design is compatible with existing DUN-based methods, serving as a plug-and-play framework. Code will be released.
Authors Chunming He, Rihan Zhang, Zheng Chen, Bowen Yang, CHengyu Fang, Yunlong Lin, Fengyang Xiao, Sina Farsiu
Comment 6 figures, 11 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging ICSE 2026
Title CN 从代码到正确性：通过分层调试完成代码生成的最后一公里
Keywords 代码生成 · 程序修复 · 分层调试 · 大语言模型 · 执行模拟
Summary 本文提出MGDebugger，一种基于层次化结构的代码调试框架，将错误代码分解为子函数树，自底向上逐级定位并修复语法和算法层面的错误，并引入LLM模拟执行器追踪变量状态以精确定位bug。实验表明其在HumanEval和HumanEvalFix上显著优于现有方法。
Reason 该论文聚焦于代码生成后的调试任务，属于典型的软件工程领域应用，尽管使用了LLM和层次化推理机制，但其核心场景是程序修复（Code Debugging），与文档图像理解（DIU）中的OCR幻觉、文本识别、版面分析或视觉-语言对齐等核心问题无直接关联。虽然‘分层调试’和‘子功能分解’的思想在理论上可能启发复杂文档的结构化解析，但该工作本身并未涉及多模态输入、视觉布局建模或文档VQA中的关键挑战，属于平行下游应用，无法为DIU提供可迁移的方法论创新。
Abstract While large language models have made significant strides in code generation, the pass rate of the generated code is bottlenecked on subtle errors, often requiring human intervention to pass tests, especially for complex problems. Existing LLM-based debugging systems treat generated programs as monolithic units, failing to address bugs at multiple levels of granularity, from low-level syntax errors to high-level algorithmic flaws. In this paper, we introduce Multi-Granularity Debugger (MGDebugger), a hierarchical code debugger by isolating, identifying, and resolving bugs at various levels of granularity. MGDebugger decomposes problematic code into a hierarchical tree structure of subfunctions, with each level representing a particular granularity of error. During debugging, it analyzes each subfunction and iteratively resolves bugs in a bottom-up manner. To effectively test each subfunction, we propose an LLM-simulated Python executor, which traces code execution and tracks important variable states to pinpoint errors accurately. Extensive experiments demonstrate that MGDebugger outperforms existing debugging systems, achieving an 18.9% improvement in accuracy over seed generations in HumanEval and a 97.6% repair success rate in HumanEvalFix. Furthermore, MGDebugger effectively fixes bugs across different categories and difficulty levels, demonstrating its robustness and effectiveness.
Authors Yuling Shi, Songsong Wang, Chengcheng Wan, Min Wang, Xiaodong Gu
Comment Accepted to ICSE 2026. Code and data available at https://github.com/YerbaPage/MGDebugger
Categories Artificial Intelligence, Computation and Language
2
Paper2SysArch: Structure-Constrained System Architecture Generation from Scientific Papers
Title CN Paper2SysArch：从科学论文生成结构约束的系统架构图
Keywords 科学可视化 · 多智能体系统 · 架构图生成 · 文本到图表 · 基准数据集
Summary 本文提出了一个名为Paper2SysArch的端到端系统，利用多智能体协作将科研论文转换为结构化的系统架构图，并发布了一个包含3000篇论文及其对应高质量图表的新基准数据集。评估采用三层指标，涵盖语义准确性、布局连贯性和视觉质量。在精选的复杂子集上达到69.0的综合得分。主要贡献在于建立了一个可复现研究的基础性基准。
Reason 该论文属于典型的‘平行应用’：将多智能体系统用于从科研论文中生成系统架构图，核心任务是科学可视化，而非文档图像理解中的基础问题（如文本识别、布局分析、表格理解等）。尽管其构建了新数据集并提出多Agent框架，但方法本质是特定场景下的下游应用包装，未对DIU领域的OCR幻觉、密集文本处理、视觉-语言对齐或高分辨率建模等核心痛点提供可迁移的技术创新。属于‘Wrapper/Application’类工作，无底层方法论贡献。
Abstract The manual creation of system architecture diagrams for scientific papers is a time-consuming and subjective process, while existing generative models lack the necessary structural control and semantic understanding for this task. A primary obstacle hindering research and development in this domain has been the profound lack of a standardized benchmark to quantitatively evaluate the automated generation of diagrams from text. To address this critical gap, we introduce a novel and comprehensive benchmark, the first of its kind, designed to catalyze progress in automated scientific visualization. It consists of 3,000 research papers paired with their corresponding high-quality ground-truth diagrams and is accompanied by a three-tiered evaluation metric assessing semantic accuracy, layout coherence, and visual quality. Furthermore, to establish a strong baseline on this new benchmark, we propose Paper2SysArch, an end-to-end system that leverages multi-agent collaboration to convert papers into structured, editable diagrams. To validate its performance on complex cases, the system was evaluated on a manually curated and more challenging subset of these papers, where it achieves a composite score of 69.0. This work's principal contribution is the establishment of a large-scale, foundational benchmark to enable reproducible research and fair comparison. Meanwhile, our proposed system serves as a viable proof-of-concept, demonstrating a promising path forward for this complex task.
Authors Ziyi Guo, Zhou Liu, Wentao Zhang
Categories Artificial Intelligence, Computation and Language
2
Robust Graph Condensation via Classification Complexity Mitigation Neurips 2025 (Spotlight)
Title CN 通过分类复杂度缓解实现鲁棒图压缩
Keywords 图压缩 · 鲁棒性 · 流形学习 · 分类复杂度 · 对抗攻击
Summary 本文提出一种名为MRGC的鲁棒图压缩框架，通过引入流形约束来提升图压缩在对抗性攻击下的稳定性。作者从几何角度建模图数据流形，设计三个模块以保持压缩图的低分类复杂度并增强鲁棒性。实验表明该方法在多种攻击场景下优于现有方法。
Reason 该论文研究图压缩（Graph Condensation）在被破坏图数据下的鲁棒性问题，属于图神经网络和表示学习领域。尽管其方法涉及数据流形约束与分类复杂度控制，但完全不涉及文档图像、多模态输入、视觉语言模型或OCR相关任务。无法迁移用于解决文档图像理解中的OCR幻觉、密集文本识别、排版解析或长文档推理等核心痛点，属于无关领域的技术工作。
Abstract Graph condensation (GC) has gained significant attention for its ability to synthesize smaller yet informative graphs. However, existing studies often overlook the robustness of GC in scenarios where the original graph is corrupted. In such cases, we observe that the performance of GC deteriorates significantly, while existing robust graph learning technologies offer only limited effectiveness. Through both empirical investigation and theoretical analysis, we reveal that GC is inherently an intrinsic-dimension-reducing process, synthesizing a condensed graph with lower classification complexity. Although this property is critical for effective GC performance, it remains highly vulnerable to adversarial perturbations. To tackle this vulnerability and improve GC robustness, we adopt the geometry perspective of graph data manifold and propose a novel Manifold-constrained Robust Graph Condensation framework named MRGC. Specifically, we introduce three graph data manifold learning modules that guide the condensed graph to lie within a smooth, low-dimensional manifold with minimal class ambiguity, thereby preserving the classification complexity reduction capability of GC and ensuring robust performance under universal adversarial attacks. Extensive experiments demonstrate the robustness of \ModelName\ across diverse attack scenarios.
Authors Jiayi Luo, Qingyun Sun, Beining Yang, Haonan Yuan, Xingcheng Fu, Yanbiao Ma, Jianxin Li, Philip S. Yu
Comment Accepted by Neurips 2025 (Spotlight)
Categories Artificial Intelligence
2
Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervision
Title CN 文本到决策智能体：基于自然语言监督的离线元强化学习
Keywords 强化学习 · 自然语言监督 · 元学习 · 离线RL · 语义对齐
Summary 本文提出T2DA，一个通过自然语言监督训练离线元强化学习智能体的框架。它将多任务决策数据编码为动态感知的嵌入空间，并通过类似CLIP的方式对齐文本描述与决策状态，从而实现根据语言指令直接生成策略。实验显示其在MuJoCo和Meta-World上具备良好的零样本泛化能力。
Reason 该论文研究的是在强化学习环境中利用自然语言指令进行决策的框架，属于具身智能与机器人控制领域。其核心方法是通过对比学习对齐文本描述与决策状态嵌入，实现零样本语言到动作的生成。然而，该工作并未涉及文档图像理解中的任何核心任务（如DocVQA、布局分析、表格识别等），也不提供可用于解决OCR幻觉、密集文本处理或复杂排版建模的技术工具。尽管语言-行为对齐的思想在理论上具有启发性，但其迁移至文档理解场景的路径极不明确，且未使用任何文档模态数据或结构推理机制。
Abstract Offline meta-RL usually tackles generalization by inferring task beliefs from high-quality samples or warmup explorations. The restricted form limits their generality and usability since these supervision signals are expensive and even infeasible to acquire in advance for unseen tasks. Learning directly from the raw text about decision tasks is a promising alternative to leverage a much broader source of supervision. In the paper, we propose \textbf{T}ext-to-\textbf{D}ecision \textbf{A}gent (\textbf{T2DA}), a simple and scalable framework that supervises offline meta-RL with natural language. We first introduce a generalized world model to encode multi-task decision data into a dynamics-aware embedding space. Then, inspired by CLIP, we predict which textual description goes with which decision embedding, effectively bridging their semantic gap via contrastive language-decision pre-training and aligning the text embeddings to comprehend the environment dynamics. After training the text-conditioned generalist policy, the agent can directly realize zero-shot text-to-decision generation in response to language instructions. Comprehensive experiments on MuJoCo and Meta-World benchmarks show that T2DA facilitates high-capacity zero-shot generalization and outperforms various types of baselines. Our code is available at \textcolor{magenta}{\href{https://github.com/NJU-RL/T2DA}{https://github.com/NJU-RL/T2DA}}.
Authors Shilin Zhang, Zican Hu, Wenhao Wu, Xinyi Xie, Jianxiang Tang, Chunlin Chen, Daoyi Dong, Yu Cheng, Zhenhong Sun, Zhi Wang
Comment 32 pages, 8 figures
Categories Artificial Intelligence
2
Leveraging Evidence-Guided LLMs to Enhance Trustworthy Depression Diagnosis
Title CN 利用证据引导的LLM增强可信的抑郁症诊断
Keywords LLM · 临床诊断 · 可解释性 · 证据推理 · 心理健康
Summary 本文提出一种两阶段框架EGDR，通过结合DSM-5标准进行证据提取与逻辑推理，提升LLM在抑郁症诊断中的透明度与可信度，并设计DCS评分模块评估诊断的一致性与事实性。在D4数据集上验证了其优于直接提示和CoT的效果。
Reason 该论文属于典型的‘平行下游应用’：将现有LLM推理机制（如CoT、证据提取）应用于医疗诊断场景，尽管引入了EGDR和DCS模块以提升可解释性和准确性，但其核心方法并未针对文档图像理解中的OCR幻觉、复杂排版或视觉-语言对齐等关键问题提出创新。任务场景为纯文本临床诊断，不涉及任何视觉输入或文档结构建模，无法迁移至DIU领域。
Abstract Large language models (LLMs) show promise in automating clinical diagnosis, yet their non-transparent decision-making and limited alignment with diagnostic standards hinder trust and clinical adoption. We address this challenge by proposing a two-stage diagnostic framework that enhances transparency, trustworthiness, and reliability. First, we introduce Evidence-Guided Diagnostic Reasoning (EGDR), which guides LLMs to generate structured diagnostic hypotheses by interleaving evidence extraction with logical reasoning grounded in DSM-5 criteria. Second, we propose a Diagnosis Confidence Scoring (DCS) module that evaluates the factual accuracy and logical consistency of generated diagnoses through two interpretable metrics: the Knowledge Attribution Score (KAS) and the Logic Consistency Score (LCS). Evaluated on the D4 dataset with pseudo-labels, EGDR outperforms direct in-context prompting and Chain-of-Thought (CoT) across five LLMs. For instance, on OpenBioLLM, EGDR improves accuracy from 0.31 (Direct) to 0.76 and increases DCS from 0.50 to 0.67. On MedLlama, DCS rises from 0.58 (CoT) to 0.77. Overall, EGDR yields up to +45% accuracy and +36% DCS gains over baseline methods, offering a clinically grounded, interpretable foundation for trustworthy AI-assisted diagnosis.
Authors Yining Yuan, J. Ben Tamo, Micky C. Nnamdi, Yifei Wang, May D. Wang
Categories Artificial Intelligence
2
Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation NeurIPS 2025
Title CN 纠正长尾数据集蒸馏中的软标签纠缠偏差
Keywords 数据集蒸馏 · 长尾分布 · 软标签偏差 · 类别不平衡 · 模型压缩
Summary 本文针对长尾分布下的数据集蒸馏性能下降问题，分析了软标签偏差的来源，并提出了ADSA——一种自适应软标签对齐模块，用于缓解由蒸馏模型和合成图像引入的偏差。实验表明该方法能显著提升尾部类别的识别准确率。然而，其应用场景局限于常规图像分类的数据蒸馏流程。
Reason 该论文研究的是数据集蒸馏（dataset distillation）中的长尾分布问题，核心贡献是提出ADSA模块以校正软标签偏差。尽管其在类别不平衡场景下的方法具有一定通用性，但工作本质属于模型训练与数据压缩的交叉，且聚焦于通用图像分类任务（如ImageNet-1k-LT），并未涉及文档结构、多模态理解、OCR或布局分析等DIU核心问题。属于典型的‘平行应用’类技术改进，无法直接迁移解决文档图像理解中的幻觉、复杂排版或细粒度定位等痛点。
Abstract Dataset distillation compresses large-scale datasets into compact, highly informative synthetic data, significantly reducing storage and training costs. However, existing research primarily focuses on balanced datasets and struggles to perform under real-world long-tailed distributions. In this work, we emphasize the critical role of soft labels in long-tailed dataset distillation and uncover the underlying mechanisms contributing to performance degradation. Specifically, we derive an imbalance-aware generalization bound for model trained on distilled dataset. We then identify two primary sources of soft-label bias, which originate from the distillation model and the distilled images, through systematic perturbation of the data imbalance levels. To address this, we propose ADSA, an Adaptive Soft-label Alignment module that calibrates the entangled biases. This lightweight module integrates seamlessly into existing distillation pipelines and consistently improves performance. On ImageNet-1k-LT with EDC and IPC=50, ADSA improves tail-class accuracy by up to 11.8% and raises overall accuracy to 41.4%. Extensive experiments demonstrate that ADSA provides a robust and generalizable solution under limited label budgets and across a range of distillation techniques. Code is available at: https://github.com/j-cyoung/ADSA_DD.git.
Authors Chenyang Jiang, Hang Zhao, Xinyu Zhang, Zhengcen Li, Qiben Shan, Shaocong Wu, Jingyong Su
Comment 10 pages, accepted by NeurIPS 2025
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
ChemVTS-Bench: Evaluating Visual-Textual-Symbolic Reasoning of Multimodal Large Language Models in Chemistry
Title CN ChemVTS-Bench：评估多模态大语言模型在化学中的视觉-文本-符号推理能力
Keywords 多模态大模型 · 化学推理 · 评测基准 · 视觉-文本-符号 · 领域特定
Summary 本文提出了ChemVTS-Bench，一个用于评估多模态大语言模型在化学领域中视觉-文本-符号联合推理能力的基准测试。它包含有机分子、无机材料和3D晶体结构等任务，并支持三种输入模式：纯视觉、图文混合和SMILES符号表示。通过基于智能体的自动化评估流程，揭示了当前MLLM在处理化学结构信息时仍存在显著缺陷。然而，该工作局限于化学这一垂直领域，缺乏对通用文档理解的技术推动。
Reason 该论文属于典型的‘平行下游应用’，其核心是构建一个面向化学领域的多模态推理评测基准，本质上是将现有MLLM技术应用于特定学科场景（化学），并未提出任何可用于文档图像理解（DIU）的通用方法论创新。尽管其引入了多模态输入（视觉、文本、符号）和自动化评估流程，但这些设计紧密绑定于化学分子结构等专业领域，无法迁移至文档理解中的OCR幻觉、复杂排版或长文档推理等核心问题。
Abstract Chemical reasoning inherently integrates visual, textual, and symbolic modalities, yet existing benchmarks rarely capture this complexity, often relying on simple image-text pairs with limited chemical semantics. As a result, the actual ability of Multimodal Large Language Models (MLLMs) to process and integrate chemically meaningful information across modalities remains unclear. We introduce \textbf{ChemVTS-Bench}, a domain-authentic benchmark designed to systematically evaluate the Visual-Textual-Symbolic (VTS) reasoning abilities of MLLMs. ChemVTS-Bench contains diverse and challenging chemical problems spanning organic molecules, inorganic materials, and 3D crystal structures, with each task presented in three complementary input modes: (1) visual-only, (2) visual-text hybrid, and (3) SMILES-based symbolic input. This design enables fine-grained analysis of modality-dependent reasoning behaviors and cross-modal integration. To ensure rigorous and reproducible evaluation, we further develop an automated agent-based workflow that standardizes inference, verifies answers, and diagnoses failure modes. Extensive experiments on state-of-the-art MLLMs reveal that visual-only inputs remain challenging, structural chemistry is the hardest domain, and multimodal fusion mitigates but does not eliminate visual, knowledge-based, or logical errors, highlighting ChemVTS-Bench as a rigorous, domain-faithful testbed for advancing multimodal chemical reasoning. All data and code will be released to support future research.
Authors Zhiyuan Huang, Baichuan Yang, Zikun He, Yanhong Wu, Fang Hongyu, Zhenhe Liu, Lin Dongsheng, Bing Su
Categories Artificial Intelligence
2
Principled Context Engineering for RAG: Statistical Guarantees via Conformal Prediction
Title CN 用于RAG的有原则上下文工程：通过共形预测实现统计保证
Keywords RAG · 共形预测 · 上下文压缩 · 事实一致性 · 文本过滤
Summary 本文提出利用共形预测对RAG系统中的检索上下文进行覆盖率控制的过滤，以在保证关键证据召回的同时减少冗余信息。实验表明该方法能有效压缩2-3倍上下文长度，并提升或维持下游事实准确性。然而，整个框架基于纯文本处理，未涉及图像或文档布局模态。
Reason 该论文研究的是检索增强生成（RAG）中的上下文过滤问题，提出使用共形预测来保留支持性证据并压缩无关内容。尽管其方法在长上下文处理和噪声抑制方面具有理论价值，但属于典型的‘平行下游应用’范式——即在通用NLP任务（如问答）中优化RAG流程，并未涉及任何视觉或文档结构信息。完全忽略文档图像理解中的核心挑战（如OCR幻觉、布局分析、细粒度定位等），且不涉及多模态输入处理。因此无法迁移到DIU领域，属于无营养的文本侧RAG优化。
Abstract Retrieval-Augmented Generation (RAG) enhances factual grounding in large language models (LLMs) by incorporating retrieved evidence, but LLM accuracy declines when long or noisy contexts exceed the model's effective attention span. Existing pre-generation filters rely on heuristics or uncalibrated LLM confidence scores, offering no statistical control over retained evidence. We evaluate and demonstrate context engineering through conformal prediction, a coverage-controlled filtering framework that removes irrelevant content while preserving recall of supporting evidence. Using both embedding- and LLM-based scoring functions, we test this approach on the NeuCLIR and RAGTIME collections. Conformal filtering consistently meets its target coverage, ensuring that a specified fraction of relevant snippets are retained, and reduces retained context by 2-3x relative to unfiltered retrieval. On NeuCLIR, downstream factual accuracy measured by ARGUE F1 improves under strict filtering and remains stable at moderate coverage, indicating that most discarded material is redundant or irrelevant. These results demonstrate that conformal prediction enables reliable, coverage-controlled context reduction in RAG, offering a model-agnostic and principled approach to context engineering.
Authors Debashish Chakraborty, Eugene Yang, Daniel Khashabi, Dawn Lawrie, Kevin Duh
Comment Preprint
Categories Artificial Intelligence, Computation and Language
2
"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with VLMs
Title CN “它由非残障人士训练”：评估图像质量如何影响视觉语言模型的产品描述生成
Keywords 视觉语言模型 · 图像质量 · 无障碍技术 · 人机交互 · 公平性
Summary 该研究通过调查86名盲人和低视力用户，系统评估了图像质量问题（如模糊、错位拍摄）对VLM生成产品描述的影响。结果显示，在存在图像质量问题时，模型准确率从98%下降到75%，且随着问题叠加性能进一步恶化。作者呼吁在模型评估中纳入残障用户的实际体验，并为HCI和ML研究者提供改进建议。
Reason 该论文研究的是图像质量（如模糊、构图偏差）对VLM在面向盲人和低视力人群（BLV）产品识别任务中的影响，属于特定社会应用场景下的模型可用性与公平性评估。虽然涉及VLM，但其核心贡献在于人机交互（HCI）层面的用户需求分析与评估范式建议，并未提出任何可迁移至文档图像理解（DIU）的技术方法。既不解决OCR幻觉、复杂排版、长文档推理等DIU核心问题，也无视觉表示学习、高分辨率建模或推理增强等底层技术创新，属于典型的‘平行应用’型研究。
Abstract Vision-Language Models (VLMs) are increasingly used by blind and low-vision (BLV) people to identify and understand products in their everyday lives, such as food, personal products, and household goods. Despite their prevalence, we lack an empirical understanding of how common image quality issues, like blur and misframing of items, affect the accuracy of VLM-generated captions and whether resulting captions meet BLV people's information needs. Grounded in a survey with 86 BLV people, we systematically evaluate how image quality issues affect captions generated by VLMs. We show that the best model recognizes products in images with no quality issues with 98% accuracy, but drops to 75% accuracy overall when quality issues are present, worsening considerably as issues compound. We discuss the need for model evaluations that center on disabled people's experiences throughout the process and offer concrete recommendations for HCI and ML researchers to make VLMs more reliable for BLV people.
Authors Kapil Garg, Xinru Tang, Jimin Heo, Dwayne R. Morgan, Darren Gergle, Erik B. Sudderth, Anne Marie Piper
Comment Paper under review
Categories Computer Vision and Pattern Recognition
2
Compact neural networks for astronomy with optimal transport bias correction
Title CN 用于天文学的具有最优传输偏差校正的紧凑型神经网络
Keywords 天文图像 · 紧凑网络 · 小波分解 · 最优传输 · 偏差校正
Summary 本文提出WaveletMamba，一种面向天文图像分析的高效神经网络框架，通过小波分解、状态空间建模和基于最优传输的偏差校正，在低分辨率输入下实现高分辨率性能，显著提升计算效率并减少预测偏差。适用于大规模天文图像分类与回归任务。
Reason 该论文研究的是天文图像分析中的效率与分辨率权衡问题，提出了一种结合小波分解和状态空间模型的框架WaveletMamba，并引入多级偏差校正机制。尽管其在计算效率和跨分辨率稳定性方面表现出色，但其核心任务（星体形态分类、红移预测）与文档图像理解（DIU）无直接关联。方法虽涉及图像表示与分布对齐，但针对的是自然遥感图像而非文档排版结构，且未涉及文本识别、布局理解或多模态问答等DIU核心挑战。属于特定科学领域的应用模型，不具备向文档理解迁移的方法论价值。
Abstract Astronomical imaging confronts an efficiency-resolution tradeoff that limits large-scale morphological classification and redshift prediction. We introduce WaveletMamba, a theory-driven framework integrating wavelet decomposition with state-space modeling, mathematical regularization, and multi-level bias correction. WaveletMamba achieves 81.72% +/- 0.53% classification accuracy at 64x64 resolution with only 3.54M parameters, delivering high-resolution performance (80.93% +/- 0.27% at 244x244) at low-resolution inputs with 9.7x computational efficiency gains. The framework exhibits Resolution Multistability, where models trained on low-resolution data achieve consistent accuracy across different input scales despite divergent internal representations. The framework's multi-level bias correction synergizes HK distance (distribution-level optimal transport) with Color-Aware Weighting (sample-level fine-tuning), achieving 22.96% Log-MSE improvement and 26.10% outlier reduction without explicit selection function modeling. Here, we show that mathematical rigor enables unprecedented efficiency and comprehensive bias correction in scientific AI, bridging computer vision and astrophysics to revolutionize interdisciplinary scientific discovery.
Authors Shuhuan Wang, Yuzhen Xie, Jiayi Li
Comment 18 pages, 5 figures, 3 tables. Research article
Categories Computer Vision and Pattern Recognition
2
SA-Person: Text-Based Person Retrieval with Scene-aware Re-ranking
Title CN SA-Person：基于场景感知重排序的文本到人检索
Keywords 文本到人检索 · 场景感知 · 重排序 · 多模态检索 · 视觉上下文
Summary 本文提出了一种场景感知的文本到人检索框架SA-Person，通过构建大规模数据集ScenePerson-13W，并设计两阶段框架：第一阶段对行人外观进行细粒度文本对齐，第二阶段引入无需训练的SceneRanker模块，结合场景上下文进行重排序。实验表明该方法在多个基准上表现优异。
Reason 该论文属于行人检索（person retrieval）领域，尽管使用了多模态（文本与图像）建模和场景上下文推理，但其核心任务与文档图像理解（DIU）无关。虽然其提出的SceneRanker模块利用全局场景信息进行重排序，具备一定的跨模态推理设计思想，但该方法针对的是开放场景中的人物识别问题，无法直接迁移解决文档中的OCR幻觉、复杂排版或长文档推理等核心痛点。属于典型的非DIU领域的平行应用，不符合关注范畴。
Abstract Text-based person retrieval aims to identify a target individual from an image gallery using a natural language description. Existing methods primarily focus on appearance-driven cross-modal retrieval, yet face significant challenges due to the visual complexity of scenes and the inherent ambiguity of textual descriptions. The contextual information, such as landmarks and relational cues, provides complementary cues that can offer valuable complementary insights for retrieval, but remains underexploited in current approaches. Motivated by this limitation, we propose a novel paradigm: scene-aware text-based person retrieval, which explicitly integrates both individual appearance and global scene context to improve retrieval accuracy. To support this, we first introduce ScenePerson-13W, a large-scale benchmark dataset comprising over 100,000 real-world scenes with rich annotations encompassing both pedestrian attributes and scene context. Based on this dataset, we further present SA-Person, a two-stage retrieval framework. In the first stage, SA-Person performs discriminative appearance grounding by aligning textual descriptions with pedestrian-specific regions. In the second stage, it introduces SceneRanker, a training-free, scene-aware re-ranking module that refines retrieval results by jointly reasoning over pedestrian appearance and the global scene context. Extensive experiments on ScenePerson-13W and existing benchmarks demonstrate the effectiveness of our proposed SA-Person. Both the dataset and code will be publicly released to facilitate future research.
Authors Yingjia Xu, Jinlin Wu, Daming Gao, Zhen Chen, Yang Yang, Min Cao, Mang Ye, Zhen Lei
Comment 13 pages, 8 figures. Under review
Categories Computer Vision and Pattern Recognition
2
Fine-Grained GRPO for Precise Preference Alignment in Flow Models
Title CN 流向模型中精细偏好对齐的细粒度GRPO
Keywords 生成模型 · 强化学习 · 偏好对齐 · 流模型 · 扩散模型
Summary 本文提出了一种名为G²RPO的新型框架，用于在基于流动的生成模型中进行更精细的偏好对齐。通过引入单步随机采样机制和多粒度优势整合模块，提升了强化学习过程中奖励信号的分配精度与训练稳定性。实验表明该方法在多种奖励模型下优于现有基线。
Reason 该论文研究的是基于扩散/流动模型的生成式建模中通过强化学习实现偏好对齐的问题，核心关注点为图像或内容生成过程中的采样轨迹优化与奖励稀疏性问题。尽管其方法涉及细粒度优势评估和多粒度集成，但完全属于生成模型的训练优化范畴，且应用场景明显偏向通用图像生成或文本到图像生成等任务。未涉及文档结构、布局理解、OCR、视觉信息提取或多模态文档问答等DIU核心问题。同时，该工作属于‘纯生成模型+RL’的技术路线，在迁移到文档理解任务（如VQA、KIE、表格识别）方面缺乏直接可迁移性，无法解决OCR幻觉、复杂排版建模或长文档推理等关键痛点。
Abstract The incorporation of online reinforcement learning (RL) into diffusion and flow-based generative models has recently gained attention as a powerful paradigm for aligning model behavior with human preferences. By leveraging stochastic sampling via Stochastic Differential Equations (SDEs) during the denoising phase, these models can explore a variety of denoising trajectories, enhancing the exploratory capacity of RL. However, despite their ability to discover potentially high-reward samples, current approaches often struggle to effectively align with preferences due to the sparsity and narrowness of reward feedback. To overcome this limitation, we introduce a novel framework called Granular-GRPO (G$^2$RPO), which enables fine-grained and comprehensive evaluation of sampling directions in the RL training of flow models. Specifically, we propose a Singular Stochastic Sampling mechanism that supports step-wise stochastic exploration while ensuring strong correlation between injected noise and reward signals, enabling more accurate credit assignment to each SDE perturbation. Additionally, to mitigate the bias introduced by fixed-granularity denoising, we design a Multi-Granularity Advantage Integration module that aggregates advantages computed across multiple diffusion scales, resulting in a more robust and holistic assessment of sampling trajectories. Extensive experiments on various reward models, including both in-domain and out-of-domain settings, demonstrate that our G$^2$RPO outperforms existing flow-based GRPO baselines, highlighting its effectiveness and generalization capability.
Authors Yujie Zhou, Pengyang Ling, Jiazi Bu, Yibin Wang, Yuhang Zang, Jiaqi Wang, Li Niu, Guangtao Zhai
Comment Project Page: https://bujiazi.github.io/g2rpo.github.io/
Categories Computer Vision and Pattern Recognition
2
GeoReasoner: Geo-localization with Reasoning in Street Views using a Large Vision-Language Model
Title CN GeoReasoner：使用大视觉语言模型在街景中进行推理的地理定位
Keywords 地理定位 · 视觉语言模型 · 推理增强 · 街景图像 · 数据过滤
Summary 本文提出GeoReasoner，一种基于大视觉语言模型的街景地理定位方法，通过CLIP筛选高质量可定位图像，并引入真实地理定位游戏中的推理知识进行训练。模型在国家和城市级定位任务上表现优于现有方法。然而，该工作专注于地理定位这一特定应用场景，不具备对文档理解任务的方法论迁移价值。
Reason 该论文研究的是街景图像中的地理定位问题，属于特定场景下的视觉-语言推理应用，其核心任务与文档图像理解（DIU）无关。尽管使用了VLM和推理增强机制，但方法聚焦于地理线索推理和外部知识融合，难以迁移到文档理解中的OCR幻觉、复杂版式解析或细粒度文本定位等核心痛点。属于典型的‘平行下游应用’，未提出可泛化的DIU底层技术创新。
Abstract This work tackles the problem of geo-localization with a new paradigm using a large vision-language model (LVLM) augmented with human inference knowledge. A primary challenge here is the scarcity of data for training the LVLM - existing street-view datasets often contain numerous low-quality images lacking visual clues, and lack any reasoning inference. To address the data-quality issue, we devise a CLIP-based network to quantify the degree of street-view images being locatable, leading to the creation of a new dataset comprising highly locatable street views. To enhance reasoning inference, we integrate external knowledge obtained from real geo-localization games, tapping into valuable human inference capabilities. The data are utilized to train GeoReasoner, which undergoes fine-tuning through dedicated reasoning and location-tuning stages. Qualitative and quantitative evaluations illustrate that GeoReasoner outperforms counterpart LVLMs by more than 25% at country-level and 38% at city-level geo-localization tasks, and surpasses StreetCLIP performance while requiring fewer training resources. The data and code are available at https://github.com/lingli1996/GeoReasoner.
Authors Ling Li, Yu Ye, Yao Zhou, Bingchuan Jiang, Wei Zeng
Categories Computer Vision and Pattern Recognition
2
State and Scene Enhanced Prototypes for Weakly Supervised Open-Vocabulary Object Detection
Title CN 状态与场景增强原型在弱监督开放词汇目标检测中的应用
Keywords 开放词汇目标检测 · 弱监督学习 · 语义原型增强 · 上下文建模 · 伪标签
Summary 本文提出了一种用于弱监督开放词汇目标检测的方法，通过状态增强语义原型（SESP）和场景增强伪原型（SAPP）来提升类别泛化能力和视觉-文本对齐。SESP生成描述对象状态的文本（如“睡觉的猫”），SAPP引入场景上下文（如“猫躺在沙发上”）并通过软对齐机制优化伪标签质量。整体方法在OVOD任务上提升了性能。
Reason 该论文研究的是开放词汇目标检测（OVOD），属于通用计算机视觉领域，且聚焦于图像级标签与伪框生成的语义对齐问题。虽然提到了LLM增强语义原型和上下文建模，但其任务设定与文档图像理解的核心痛点（如OCR幻觉、密集文本识别、复杂版式解析、长文档推理）无直接关联。方法本质上是针对自然图像中物体状态和场景上下文的建模，并不涉及文档特有的结构、布局或文本-视觉耦合特性，难以迁移到DocVQA或表格识别等任务。属于无关领域的常规改进工作。
Abstract Open-Vocabulary Object Detection (OVOD) aims to generalize object recognition to novel categories, while Weakly Supervised OVOD (WS-OVOD) extends this by combining box-level annotations with image-level labels. Despite recent progress, two critical challenges persist in this setting. First, existing semantic prototypes, even when enriched by LLMs, are static and limited, failing to capture the rich intra-class visual variations induced by different object states (e.g., a cat's pose). Second, the standard pseudo-box generation introduces a semantic mismatch between visual region proposals (which contain context) and object-centric text embeddings. To tackle these issues, we introduce two complementary prototype enhancement strategies. To capture intra-class variations in appearance and state, we propose the State-Enhanced Semantic Prototypes (SESP), which generates state-aware textual descriptions (e.g., "a sleeping cat") to capture diverse object appearances, yielding more discriminative prototypes. Building on this, we further introduce Scene-Augmented Pseudo Prototypes (SAPP) to address the semantic mismatch. SAPP incorporates contextual semantics (e.g., "cat lying on sofa") and utilizes a soft alignment mechanism to promote contextually consistent visual-textual representations. By integrating SESP and SAPP, our method effectively enhances both the richness of semantic prototypes and the visual-textual alignment, achieving notable improvements.
Authors Jiaying Zhou, Qingchao Chen
Categories Computer Vision and Pattern Recognition
2
RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios
Title CN RoadBench：在城市道路场景下对多模态大语言模型的细粒度空间理解与推理能力进行基准测试
Keywords 多模态大语言模型 · 空间推理 · 基准测试 · 道路标记 · 城市场景
Summary 本文提出了RoadBench，一个针对城市道路场景中多模态大语言模型（MLLM）细粒度空间理解与推理能力的基准测试，涵盖基于鸟瞰图（BEV）和前视图（FPV）的六项任务，共9,121个手工验证样本。评估结果显示现有MLLM在此类任务上表现不佳，甚至低于规则基线。研究揭示了当前模型在复杂空间结构理解上的不足。
Reason 该论文聚焦于城市道路场景下的细粒度空间理解，属于自动驾驶和交通视觉领域，与文档图像理解（DIU）的核心任务如DocVQA、表格识别、布局分析等无直接关联。尽管其涉及MLLM在复杂视觉排布中的定位与推理，但应用场景为街景BEV/FPV图像，结构模式与文档差异巨大，方法论上未提出可迁移至文档领域的通用技术（如视觉压缩、去幻觉机制或多模态对齐优化），属于无关领域应用。
Abstract Multimodal large language models (MLLMs) have demonstrated powerful capabilities in general spatial understanding and reasoning. However, their fine-grained spatial understanding and reasoning capabilities in complex urban scenarios have not received significant attention in the fields of both research and industry. To fill this gap, we focus primarily on road markings as a typical example of fine-grained spatial elements under urban scenarios, given the essential role of the integrated road traffic network they form within cities. Around road markings and urban traffic systems, we propose RoadBench, a systematic benchmark that comprehensively evaluates MLLMs' fine-grained spatial understanding and reasoning capabilities using BEV and FPV image inputs. This benchmark comprises six tasks consisting of 9,121 strictly manually verified test cases. These tasks form a systematic evaluation framework that bridges understanding at local spatial scopes to global reasoning. They not only test MLLMs' capabilities in recognition, joint understanding, and reasoning but also assess their ability to integrate image information with domain knowledge. After evaluating 14 mainstream MLLMs, we confirm that RoadBench is a challenging benchmark for MLLMs while revealing significant shortcomings in existing MLLMs' fine-grained spatial understanding and reasoning capabilities within urban scenarios. In certain tasks, their performance even falls short of simple rule-based or random selection baselines. These findings, along with RoadBench itself, will contribute to the comprehensive advancement of spatial understanding capabilities for MLLMs. The benchmark code, example datasets, and raw evaluation results are available in the supplementary material.
Authors Jun Zhang, Jie Feng, Long Chen, Junhui Wang, Zhicheng Liu, Depeng Jin, Yong Li
Comment The code and data are publicly available at: https://github.com/tsinghua-fib-lab/RoadBench
Categories Computer Vision and Pattern Recognition
2
Is Complete Labeling Necessary? Understanding Active Learning in Longitudinal Medical Imaging International Joint Conference on Neural Networks (IJCNN) 2025
Title CN 完全标注是必要的吗？理解纵向医学影像中的主动学习
Keywords 主动学习 · 医学影像 · 变化检测 · 深度学习 · 标注效率
Summary 本文提出了一种针对纵向医学影像的主动学习框架LMI-AL，通过配对基线与随访的2D切片并差分处理，迭代选择最具信息量的图像对进行标注，在仅使用不到8%标注数据的情况下达到了接近全监督模型的性能。研究聚焦于医学图像变化检测这一特定任务，强调降低专业标注负担。
Reason 该论文研究的是在纵向医学影像中通过主动学习减少标注成本，属于特定医疗领域的下游应用。尽管使用了Deep Active Learning进行样本选择，但其核心贡献在于特定场景（医学影像变化检测）的应用优化，未提出可迁移至文档图像理解的通用方法论。属于典型的‘平行下游应用’，无DIU相关的方法创新。
Abstract Detecting changes in longitudinal medical imaging using deep learning requires a substantial amount of accurately labeled data. However, labeling these images is notably more costly and time-consuming than labeling other image types, as it requires labeling across various time points, where new lesions can be minor, and subtle changes are easily missed. Deep Active Learning (DAL) has shown promise in minimizing labeling costs by selectively querying the most informative samples, but existing studies have primarily focused on static tasks like classification and segmentation. Consequently, the conventional DAL approach cannot be directly applied to change detection tasks, which involve identifying subtle differences across multiple images. In this study, we propose a novel DAL framework, named Longitudinal Medical Imaging Active Learning (LMI-AL), tailored specifically for longitudinal medical imaging. By pairing and differencing all 2D slices from baseline and follow-up 3D images, LMI-AL iteratively selects the most informative pairs for labeling using DAL, training a deep learning model with minimal manual annotation. Experimental results demonstrate that, with less than 8% of the data labeled, LMI-AL can achieve performance comparable to models trained on fully labeled datasets. We also provide a detailed analysis of the method's performance, as guidance for future research. The code is publicly available at https://github.com/HelenMa9998/Longitudinal_AL.
Authors Siteng Ma, Honghui Du, Prateek Mathur, Brendan S. Kelly, Ronan P. Killeen, Aonghus Lawlor, Ruihai Dong
Comment This paper has been accepted at International Joint Conference on Neural Networks (IJCNN) 2025
Categories Computer Vision and Pattern Recognition
2
Multi-speaker Attention Alignment for Multimodal Social Interaction
Title CN 用于多模态社交互动的多说话人注意力对齐
Keywords 多模态学习 · 注意力机制 · 社交交互理解 · 视频理解 · MLLM
Summary 本文提出一种多说话人注意力对齐方法，通过动态选择注意力头并引入基于说话人位置的社会感知注意力偏置，提升多模态大模型在多人社交互动理解任务中的表现。该方法应用于视频问答等社交推理任务，在TVQA+、MMSI等基准上取得SOTA结果。但由于其应用场景和问题设定远离文档图像理解，对DIU领域无实质借鉴价值。
Reason 该论文研究的是视频中的多说话人社交交互理解，属于视频理解领域，核心任务为说话人-听者关系建模、视线与言语对齐等，与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析、长文档推理等无直接关联。尽管其提出的注意力对齐机制在理论上涉及多模态对齐优化，但其设计针对动态视频中的人物交互场景，迁移到文档图像这种静态、高分辨率、细粒度文本布局场景的适用性极低，且未涉及文本识别或视觉语言模型在文档上的可靠性问题。
Abstract Understanding social interaction in video requires reasoning over a dynamic interplay of verbal and non-verbal cues: who is speaking, to whom, and with what gaze or gestures. While Multimodal Large Language Models (MLLMs) are natural candidates, simply adding visual inputs yields surprisingly inconsistent gains on social tasks. Our quantitative analysis of cross-modal attention inside state-of-the-art MLLMs reveals a core failure mode: in multi-speaker scenes, visual and textual tokens lack speaker-consistent alignment, exhibiting substantially weaker cross-modal attention than in object-centric images. To address this, we propose a multimodal multi-speaker attention alignment method that can be integrated into existing MLLMs. First, we introduce dynamic cross-modal head selection to identify attention heads most responsible for grounding. Then, an adaptive social-aware attention bias, computed from existing attention patterns and speaker locations, is injected into the attention mechanism. This bias reinforces alignment between a speaker's visual representation and their utterances without introducing trainable parameters or architectural changes. We integrate our method into three distinct MLLMs (LLaVA-NeXT-Video, Qwen2.5-VL, and InternVL3) and evaluate on three benchmarks (TVQA+, MMSI, OnlineMMSI). Across four social tasks, results demonstrate that our approach improves the ability of MLLMs and achieves state-of-the-art results. Attention visualizations confirm our method successfully focuses the model on speaker-relevant regions, enabling more robust multi-party social reasoning. Our implementation and model will be available at https://github.com/ut-vision/SocialInteraction.
Authors Liangyang Ouyang, Yifei Huang, Mingfang Zhang, Caixin Kang, Ryosuke Furuta, Yoichi Sato
Categories Computer Vision and Pattern Recognition
2
Test-Time Temporal Sampling for Efficient MLLM Video Understanding
Title CN 用于高效多模态大语言模型视频理解的测试时时间采样
Keywords 多模态大语言模型 · 长视频理解 · 测试时优化 · 时间采样 · 推理效率
Summary 本文提出Test-Time Temporal Sampling（T3S），一种无需训练的测试时采样方法，通过生成多个短而多样化的视频子序列，在单次前向传播中处理并聚合结果，从而在降低计算成本的同时提升长视频理解的准确性和延迟表现。方法利用了视频中的时空冗余，适用于现有的多模态大语言模型。
Reason 该论文研究的是长视频理解中的时间冗余问题，提出了一种测试时的时间采样子方法以提升推理效率。尽管其‘测试时优化’和‘子序列聚合’的思想具有一定启发性，但其核心任务属于视频理解领域，与文档图像理解（DIU）在输入模态、结构特性（时空连续性 vs 静态复杂排版）、关键挑战（如OCR幻觉、细粒度文本定位）等方面差异巨大。该方法依赖于视频帧间的时间冗余，而文档图像不具备此类冗余模式，因此难以迁移到DIU场景中解决实际问题。
Abstract Processing long videos with multimodal large language models (MLLMs) poses a significant computational challenge, as the model's self-attention mechanism scales quadratically with the number of video tokens, resulting in high computational demand and slow inference speed. Current solutions, such as rule-based sub-sampling, learned frame selector, or memory-based summarization, often introduce their own trade-offs: they compromise accuracy, necessitate additional training, or decrease inference speed. In this paper, we propose Test-Time Temporal Sampling (T3S), a training-free, plug-and-play inference wrapper that enables MLLMs to process long videos both efficiently and effectively. T3S exploits spatiotemporal redundancy by generating multiple short and diverse subsequences of video tokens at inference time, packing them within a single forward pass, and aggregating their predictions. This multi-subsequence formulation broadens visual coverage while reducing the computational cost of self-attention from $O(L^2)$ to $O(\sum_{i=1}^m α_i^2L^2)$, where $\sum_{i=1}^m α_i^2 < 1$. Extensive experiments on long video understanding benchmarks demonstrate that T3S improves accuracy by up to 3.1% and reduces first token delay by $2.04\times$, all with minimal integration effort. Our approach operates entirely at inference time, requires no model modifications or fine-tuning, and is compatible with a wide range of pretrained MLLMs. T3S turns video redundancy into a computational advantage, offering a scalable solution for long-video understanding. The code is available at https://github.com/kaibinwang3/T3S.
Authors Kaibin Wang, Mingbao Lin
Categories Computer Vision and Pattern Recognition
2
Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion NeurIPS 2025
Title CN 通过测试时自然视频补全从少量视角实现新视角合成
Keywords 新视角合成 · 视频补全 · 测试时生成 · 3D高斯点阵 · 视频扩散模型
Summary 本文提出一种零样本、生成引导的新视角合成框架，利用预训练视频扩散模型在测试阶段补全稀疏视角之间的自然视频序列，并通过不确定性感知机制保证空间一致性。生成的伪视图用于增强3D高斯点阵的监督信号，提升稀疏条件下的场景重建质量。整个过程无需针对特定场景训练，且在多个稀疏视角数据集上显著优于基线方法。
Reason 该论文研究的是稀疏输入下的新视角合成问题，属于3D视觉与视频生成领域，核心方法依赖预训练视频扩散模型在测试阶段生成中间视图，并结合3D高斯点阵进行场景重建。其技术路径聚焦于动态视频补全与三维重建的协同优化，与文档图像理解（DIU）中的OCR幻觉、复杂排版解析、细粒度文本定位等核心痛点无直接关联。尽管使用了‘test-time completion’和‘uncertainty-aware mechanism’等机制，但这些设计面向自由视角视频生成，难以迁移到文档结构理解或VLM推理增强任务中。
Abstract Given just a few glimpses of a scene, can you imagine the movie playing out as the camera glides through it? That's the lens we take on \emph{sparse-input novel view synthesis}, not only as filling spatial gaps between widely spaced views, but also as \emph{completing a natural video} unfolding through space. We recast the task as \emph{test-time natural video completion}, using powerful priors from \emph{pretrained video diffusion models} to hallucinate plausible in-between views. Our \emph{zero-shot, generation-guided} framework produces pseudo views at novel camera poses, modulated by an \emph{uncertainty-aware mechanism} for spatial coherence. These synthesized frames densify supervision for \emph{3D Gaussian Splatting} (3D-GS) for scene reconstruction, especially in under-observed regions. An iterative feedback loop lets 3D geometry and 2D view synthesis inform each other, improving both the scene reconstruction and the generated views. The result is coherent, high-fidelity renderings from sparse inputs \emph{without any scene-specific training or fine-tuning}. On LLFF, DTU, DL3DV, and MipNeRF-360, our method significantly outperforms strong 3D-GS baselines under extreme sparsity.
Authors Yan Xu, Yixing Wang, Stella X. Yu
Comment Accepted to NeurIPS 2025
Categories Computer Vision and Pattern Recognition
2
Animated Territorial Data Extractor (ATDE): A Computer-Vision Method for Extracting Territorial Data from Animated Historical Maps
Title CN 动画领土数据提取器（ATDE）：一种从动画历史地图中提取领土数据的计算机视觉方法
Keywords 计算机视觉 · 历史地图 · 像素分割 · 时间序列 · 颜色过滤
Summary 本文提出了一种名为ATDE的工具，用于从动画历史地图视频中提取领土控制的量化数据。通过HSV颜色分割、RGB通道过滤和邻域滤波等传统图像处理技术，结合时间对齐和跨视频缩放预处理，将动画转化为结构化的时间序列数据。作者在十个中国朝代的地图上进行了验证，结果显示像素变化趋势符合历史规律。该工具无需先验形状文件，仅需种子颜色即可运行，适用于教育演示和初步探索性分析。
Reason 该论文属于典型的‘平行应用’，即将传统计算机视觉技术（如颜色分割、像素计数）应用于特定类型的历史地图视频，解决的是历史数据分析任务，而非文档图像理解中的核心问题。其方法未涉及VLM、多模态大模型、OCR-free架构、视觉压缩或任何可迁移至DocVQA、表格识别、布局分析等DIU任务的底层技术创新。尽管使用了CV技术，但与当前DIU关注的幻觉、复杂排版、长文档推理等痛点无关。
Abstract We present Animated Territorial Data Extractor (ATDE), a computer vision tool that extracts quantitative territorial data from animated historical map videos. ATDE employs HSV-based color segmentation, RGB channel filtering, and Direct-Neighbor Filtering to identify and count pixels representing territorial control. Combined with preprocessing for temporal alignment and cross-video scaling, the pipeline converts animated videos into structured time-series data. We demonstrate the tool on ten Chinese dynasties (200 BCE - 1912 CE), producing year-by-year pixel counts that align with expected historical patterns. While not a substitute for authoritative historical datasets, ATDE is well-suited for educational demonstrations, preliminary data exploration, and comparative analysis of territorial dynamics. The tool requires no pre-existing shapefiles and can be applied to any animated map video given seed colors and basic configuration. Code and examples are available on GitHub.
Authors Hamza Alshamy, Isaiah Woram, Advay Mishra, Zihan Xia, Pascal Wallisch
Comment 11 pages, 5 figures
Categories Computer Vision and Pattern Recognition
2
Rethinking Retrieval: From Traditional Retrieval Augmented Generation to Agentic and Non-Vector Reasoning Systems in the Financial Domain for Large Language Models
Title CN 重新思考检索：从传统的检索增强生成到面向大语言模型的金融领域智能体与非向量推理系统
Keywords RAG · 金融文档 · 检索增强生成 · 智能体系统 · 文档问答
Summary 本文在金融领域对基于向量和非向量的RAG系统进行了系统性比较，评估了混合搜索、元数据过滤、交叉编码器重排序和小到大块检索等技术在SEC文件上的效果。实验表明，带有增强技术的向量型智能体RAG在检索准确性和答案质量上优于层级节点系统，但整体仍属于特定领域的应用研究。
Reason 该论文属于典型的‘平行下游应用’，将现有RAG技术应用于金融领域的SEC文档问答，未提出任何新的底层方法或架构创新。尽管比较了不同RAG架构，但其核心是特定领域（金融）的应用评测，不涉及文档图像理解中的OCR、布局分析、视觉表示学习等关键技术突破，也无法迁移解决DIU中的幻觉、密集文本或复杂排版问题。
Abstract Recent advancements in Retrieval-Augmented Generation (RAG) have enabled Large Language Models to answer financial questions using external knowledge bases of U.S. SEC filings, earnings reports, and regulatory documents. However, existing work lacks systematic comparison of vector-based and non-vector RAG architectures for financial documents, and the empirical impact of advanced RAG techniques on retrieval accuracy, answer quality, latency, and cost remain unclear. We present the first systematic evaluation comparing vector-based agentic RAG using hybrid search and metadata filtering against hierarchical node-based systems that traverse document structure without embeddings. We evaluate two enhancement techniques applied to the vector-based architecture, i) cross-encoder reranking for retrieval precision, and ii) small-to-big chunk retrieval for context completeness. Across 1,200 SEC 10-K, 10-Q, and 8-K filings on a 150-question benchmark, we measure retrieval metrics (MRR, Recall@5), answer quality through LLM-as-a-judge pairwise comparisons, latency, and preprocessing costs. Vector-based agentic RAG achieves a 68% win rate over hierarchical node-based systems with comparable latency (5.2 compared to 5.98 seconds). Cross-encoder reranking achieves a 59% absolute improvement at optimal parameters (10, 5) for MRR@5. Small-to-big retrieval achieves a 65% win rate over baseline chunking with only 0.2 seconds additional latency. Our findings reveal that applying advanced RAG techniques to financial Q&A systems improves retrieval accuracy, answer quality, and has cost-performance tradeoffs to be considered in production.
Authors Elias Lumer, Matt Melich, Olivia Zino, Elena Kim, Sara Dieter, Pradeep Honaganahalli Basavaraju, Vamse Kumar Subbiah, James A. Burke, Roberto Hernandez
Comment 8 pages, 2 figures
Categories Computation and Language
2
GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agents
Title CN GEM：用于GUI代理中分布外检测的高斯嵌入建模
Keywords GUI Agent · OOD Detection · Gaussian Mixture Model · Embedding Space · Human-Computer Interaction
Summary 本文提出GEM方法，通过在GUI代理的输入嵌入距离上拟合高斯混合模型，实现对分布外指令的有效检测。在多个设备和界面环境下验证了其优越性能，提升了OOD检测准确率和任务成功率。然而，该工作不涉及文档内容理解或多模态文档处理。
Reason 该论文研究的是GUI智能体中的分布外检测（OOD）问题，属于具身智能/人机交互领域，与文档图像理解（DIU）或DocVQA的核心任务无直接关联。尽管其方法涉及嵌入空间建模，但应用场景聚焦于GUI操作的安全性与鲁棒性，无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心痛点。属于无关领域应用。
Abstract Graphical user interface (GUI) agents have recently emerged as an intriguing paradigm for human-computer interaction, capable of automatically executing user instructions to operate intelligent terminal devices. However, when encountering out-of-distribution (OOD) instructions that violate environmental constraints or exceed the current capabilities of agents, GUI agents may suffer task breakdowns or even pose security threats. Therefore, effective OOD detection for GUI agents is essential. Traditional OOD detection methods perform suboptimally in this domain due to the complex embedding space and evolving GUI environments. In this work, we observe that the in-distribution input semantic space of GUI agents exhibits a clustering pattern with respect to the distance from the centroid. Based on the finding, we propose GEM, a novel method based on fitting a Gaussian mixture model over input embedding distances extracted from the GUI agent that reflect its capability boundary. Evaluated on eight datasets spanning smartphones, computers, and web browsers, our method achieves an average accuracy improvement of 23.70\% over the best-performing baseline while only increasing training time by 4.9\% and testing time by 6.5\%. We also experimentally demonstrate that GEM can improve the step-wise success rate by 9.40\% by requesting assistance from the cloud model when encountering OOD samples. Analysis verifies the generalization ability of our method through experiments on nine different backbones. The codes are available at https://github.com/Wuzheng02/GEM-OODforGUIagents.
Authors Zheng Wu, Pengzhou Cheng, Zongru Wu, Lingzhong Dong, Zhuosheng Zhang
Categories Computation and Language
2
Using tournaments to calculate AUROC for zero-shot classification with LLMs EMNLP 2025, Findings
Title CN 使用锦标赛方法计算大语言模型零样本分类的AUROC
Keywords 零样本分类 · AUROC · Elo评分 · 成对比较 · 评估方法
Summary 本文提出一种利用大语言模型进行零样本二分类任务时估算AUROC的方法，将分类任务转化为数据集内样本间的成对比较，借助Elo评分系统生成置信度排序。通过设计调度算法减少比较次数，在降低计算成本的同时提升分类性能。该方法主要服务于传统文本分类场景下的模型评估。
Reason 该论文研究的是通用NLP中的零样本分类评估问题，提出通过成对比较和Elo评分系统构建实例排序以估计AUROC。尽管方法涉及LLM的推理机制，但其核心目标是分类器性能评估，并未涉及任何视觉模态、文档结构或OCR相关内容。属于纯NLP方法在标准分类任务上的应用，与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版、多模态对齐等完全无关，无法迁移用于提升DIU性能。
Abstract Large language models perform surprisingly well on many zero-shot classification tasks, but are difficult to fairly compare to supervised classifiers due to the lack of a modifiable decision boundary. In this work, we propose and evaluate a method that transforms binary classification tasks into pairwise comparisons between instances within a dataset, using LLMs to produce relative rankings of those instances. Repeated pairwise comparisons can be used to score instances using the Elo rating system (used in chess and other competitions), inducing a confidence ordering over instances in a dataset. We evaluate scheduling algorithms for their ability to minimize comparisons, and show that our proposed algorithm leads to improved classification performance, while also providing more information than traditional zero-shot classification.
Authors WonJin Yoon, Ian Bulovic, Timothy A. Miller
Comment The 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025, Findings). The code is available at: https://github.com/Machine-Learning-for-Medical-Language/cnlp_llm
Categories Computation and Language
2
REMSA: An LLM Agent for Foundation Model Selection in Remote Sensing
Title CN REMSA：一种用于遥感领域基础模型选择的LLM代理
Keywords LLM Agent · Foundation Model Selection · Remote Sensing · In-context Learning · RAG
Summary 本文提出了REMSA，一个基于大语言模型的智能代理，用于根据自然语言查询自动选择遥感领域中的基础模型。作者构建了包含150多个遥感基础模型的结构化数据库RS-FMD，并设计了一个由75个专家验证查询组成的基准测试。REMSA能够解析用户需求、填补缺失约束、通过上下文学习排序候选模型，并提供透明解释。实验表明其优于多种基线方法。系统仅依赖公开元数据，不涉及敏感信息。
Reason 该论文属于典型的‘平行下游应用’，即将现有LLM Agent技术应用于遥感（Remote Sensing）领域的模型选择问题。尽管构建了RS-FMD数据库并设计了查询基准，但其核心方法是基于已有LLM和RAG的代理框架，并未提出可迁移至文档图像理解（DIU）的底层技术创新。任务场景与文档理解无关，且无视觉压缩、多模态对齐、高分辨率处理或去幻觉机制等对DIU有潜在价值的方法贡献。
Abstract Foundation Models (FMs) are increasingly used in remote sensing (RS) for tasks such as environmental monitoring, disaster assessment, and land-use mapping. These models include unimodal vision encoders trained on a single data modality and multimodal architectures trained on combinations of SAR, multispectral, hyperspectral, and image-text data. They support diverse RS tasks including semantic segmentation, image classification, change detection, and visual question answering. However, selecting an appropriate remote sensing foundation model (RSFM) remains difficult due to scattered documentation, heterogeneous formats, and varied deployment constraints. We introduce the RSFM Database (RS-FMD), a structured resource covering over 150 RSFMs spanning multiple data modalities, resolutions, and learning paradigms. Built on RS-FMD, we present REMSA, the first LLM-based agent for automated RSFM selection from natural language queries. REMSA interprets user requirements, resolves missing constraints, ranks candidate models using in-context learning, and provides transparent justifications. We also propose a benchmark of 75 expert-verified RS query scenarios, producing 900 configurations under an expert-centered evaluation protocol. REMSA outperforms several baselines, including naive agents, dense retrieval, and unstructured RAG-based LLMs. It operates entirely on publicly available metadata and does not access private or sensitive data.
Authors Binger Chen, Tacettin Emre Bök, Behnood Rasti, Volker Markl, Begüm Demir
Comment Code and data available at https://github.com/be-chen/REMSA
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
Value of Information-Enhanced Exploration in Bootstrapped DQN
Title CN 基于信息价值增强探索的Bootstrapped DQN
Keywords 强化学习 · 探索策略 · 信息价值 · Bootstrapped DQN · 稀疏奖励
Summary 本文提出将信息价值（EVOI）引入Bootstrapped DQN框架中，利用多个网络头之间的意见差异来驱动深度探索，提升在高维状态和稀疏奖励环境下的学习效率。实验表明该方法在Atari游戏中性能更优且无需额外超参数调优。
Reason 该论文属于强化学习领域的算法改进，研究在稀疏奖励环境下如何通过信息价值（EVOI）指导探索。尽管其方法论涉及不确定性建模和多头网络分歧利用，但整体框架面向通用RL任务（如Atari游戏），并未涉及任何视觉-语言或多模态输入处理，也未针对文档结构、OCR错误或排版复杂性等DIU核心问题设计机制。无法直接迁移用于解决文档图像理解中的幻觉、细粒度定位或长文档推理等挑战。
Abstract Efficient exploration in deep reinforcement learning remains a fundamental challenge, especially in environments characterized by high-dimensional states and sparse rewards. Traditional exploration strategies that rely on random local policy noise, such as $ε$-greedy and Boltzmann exploration methods, often struggle to efficiently balance exploration and exploitation. In this paper, we integrate the notion of (expected) value of information (EVOI) within the well-known Bootstrapped DQN algorithmic framework, to enhance the algorithm's deep exploration ability. Specifically, we develop two novel algorithms that incorporate the expected gain from learning the value of information into Bootstrapped DQN. Our methods use value of information estimates to measure the discrepancies of opinions among distinct network heads, and drive exploration towards areas with the most potential. We evaluate our algorithms with respect to performance and their ability to exploit inherent uncertainty arising from random network initialization. Our experiments in complex, sparse-reward Atari games demonstrate increased performance, all the while making better use of uncertainty, and, importantly, without introducing extra hyperparameters.
Authors Stergios Plataniotis, Charilaos Akasiadis, Georgios Chalkiadakis
Categories Artificial Intelligence
2
Evaluating AI-Driven Automated Map Digitization in QGIS 2025 Indiana Geographic Information Council (IGIC) Conference
Title CN 评估QGIS中人工智能驱动的自动地图数字化
Keywords 地图数字化 · QGIS插件 · 遥感深度学习 · AI应用 · 地理信息系统
Summary 本文研究了名为Deepness的AI工具在QGIS中的应用，用于自动化地图数字化。通过对比Google Earth影像的AI生成结果与OpenStreetMap的手动数字化输出，评估其性能。研究重点在于实际应用效果，而非算法或模型层面的创新。
Reason 该论文属于典型的‘平行下游应用’，即将现有AI技术（Deep Neural Remote Sensing）集成到QGIS中用于地图数字化任务。其核心是地理信息系统（GIS）领域的应用验证，并未提出任何文档图像理解（DIU）中的方法论创新。尽管涉及图像到矢量数据的转换，但目标场景为遥感影像而非文档布局，且未解决OCR幻觉、复杂排版或VLM推理等DIU核心问题。
Abstract Map digitization is an important process that converts maps into digital formats that can be used for further analysis. This process typically requires a deep human involvement because of the need for interpretation and decision-making when translating complex features. With the advancement of artificial intelligence, there is an alternative to conducting map digitization with the help of machine learning techniques. Deepness, or Deep Neural Remote Sensing, is an advanced AI-driven tool designed and integrated as a plugin in QGIS application. This research focuses on assessing the effectiveness of Deepness in automated digitization. This study analyses AI-generated digitization results from Google Earth imagery and compares them with digitized outputs from OpenStreetMap (OSM) to evaluate performance.
Authors Diana Febrita
Comment Presented at the 2025 Indiana Geographic Information Council (IGIC) Conference
Categories Artificial Intelligence
2
The promise and limits of LLMs in constructing proofs and hints for logic problems in intelligent tutoring systems
Title CN LLM在构建逻辑问题证明和提示中的潜力与局限性：在智能辅导系统中的应用
Keywords LLM · 智能辅导系统 · 逻辑证明 · 提示工程 · 幻觉
Summary 本文评估了六种提示方法在四个主流LLM上生成形式化命题逻辑证明的能力，发现DeepSeek-V3在逐步推理中达到86.7%的准确率。同时使用表现最好的LLM为学生解题状态生成辅导提示，并通过自动与人工评分评估其质量。结果显示生成的提示在一致性和清晰度上得分较高，但在解释提示理由和上下文方面表现不足。研究表明LLM可用于增强智能辅导系统，但需进一步优化以确保准确性和教学适宜性。
Reason 该论文研究的是如何利用LLM为逻辑教学中的智能辅导系统生成解题步骤和提示，属于典型的‘平行下游应用’。其核心任务是教育领域的反馈生成，并未提出可用于文档图像理解（DIU）的新架构、新训练方法或可迁移的底层技术。尽管涉及LLM的幻觉问题，但其场景聚焦于符号逻辑和教学提示，与文档视觉结构、OCR错误修正、多模态对齐等DIU核心痛点无关，无法迁移到DocVQA或复杂排版理解任务中。
Abstract Intelligent tutoring systems have demonstrated effectiveness in teaching formal propositional logic proofs, but their reliance on template-based explanations limits their ability to provide personalized student feedback. While large language models (LLMs) offer promising capabilities for dynamic feedback generation, they risk producing hallucinations or pedagogically unsound explanations. We evaluated the stepwise accuracy of LLMs in constructing multi-step symbolic logic proofs, comparing six prompting techniques across four state-of-the-art LLMs on 358 propositional logic problems. Results show that DeepSeek-V3 achieved superior performance up to 86.7% accuracy on stepwise proof construction and excelled particularly in simpler rules. We further used the best-performing LLM to generate explanatory hints for 1,050 unique student problem-solving states from a logic ITS and evaluated them on 4 criteria with both an LLM grader and human expert ratings on a 20% sample. Our analysis finds that LLM-generated hints were 75% accurate and rated highly by human evaluators on consistency and clarity, but did not perform as well explaining why the hint was provided or its larger context. Our results demonstrate that LLMs may be used to augment tutoring systems with logic tutoring hints, but require additional modifications to ensure accuracy and pedagogical appropriateness.
Authors Sutapa Dey Tithi, Arun Kumar Ramesh, Clara DiMarco, Xiaoyi Tian, Nazia Alam, Kimia Fazeli, Tiffany Barnes
Categories Artificial Intelligence
2
Interactive Query Answering on Knowledge Graphs with Soft Entity Constraints
Title CN 基于软实体约束的知识图谱交互式查询回答
Keywords 知识图谱 · 查询回答 · 软约束 · 图神经网络 · 信息检索
Summary 本文提出了在不完整知识图谱上进行带有软约束的查询回答方法，通过轻量级模型引入属性偏好或类别相关性等模糊条件，调整答案排序而不破坏原有结构。实验基于扩展后的QA数据集验证了方法的有效性和低开销。
Reason 该论文研究的是知识图谱上的查询回答问题，引入了‘软约束’来处理模糊或上下文相关的用户偏好。尽管其方法在KG-QA场景中有一定创新，但属于典型的下游应用型工作，且与文档图像理解（DIU）的核心任务如DocVQA、布局分析、OCR等无直接关联。其提出的技术难以迁移到解决文档视觉内容的理解、细粒度定位或OCR幻觉等问题，也不涉及多模态建模、高分辨率图像处理或视觉语言对齐等DIU关键挑战。
Abstract Methods for query answering over incomplete knowledge graphs retrieve entities that are \emph{likely} to be answers, which is particularly useful when such answers cannot be reached by direct graph traversal due to missing edges. However, existing approaches have focused on queries formalized using first-order-logic. In practice, many real-world queries involve constraints that are inherently vague or context-dependent, such as preferences for attributes or related categories. Addressing this gap, we introduce the problem of query answering with soft constraints. We formalize the problem and introduce two efficient methods designed to adjust query answer scores by incorporating soft constraints without disrupting the original answers to a query. These methods are lightweight, requiring tuning only two parameters or a small neural network trained to capture soft constraints while maintaining the original ranking structure. To evaluate the task, we extend existing QA benchmarks by generating datasets with soft constraints. Our experiments demonstrate that our methods can capture soft constraints while maintaining robust query answering performance and adding very little overhead.
Authors Daniel Daza, Alberto Bernardi, Luca Costabello, Christophe Gueret, Masoud Mansoury, Michael Cochez, Martijn Schut
Categories Artificial Intelligence
2
LLM-Agent-UMF: LLM-based Agent Unified Modeling Framework for Seamless Design of Multi Active/Passive Core-Agent Architectures Information Fusion, Volume 127, March 2026, 103865
Title CN LLM-Agent-UMF：基于大语言模型的代理统一建模框架，用于无缝设计多主动/被动核心代理架构
Keywords LLM代理 · 统一架构 · 信息融合 · 核心代理 · 软件工程
Summary 本文提出了一个基于大语言模型的代理统一建模框架（LLM-Agent-UMF），旨在解决当前LLM代理系统中存在的架构不统一、模块耦合度高和术语混乱等问题。该框架引入‘核心代理’概念，并将其分为规划、记忆、配置、行动和安全五个模块，同时区分主动与被动类型的核心代理，支持构建复合型多代理架构。通过在13个现有先进代理系统上进行映射分析，并设计五种变体架构进行评估，验证了该框架的通用性和结构性优势。研究发表于Information Fusion期刊。
Reason 该论文提出了一种通用的LLM代理软件架构框架，聚焦于系统模块化和术语统一，属于典型的‘平行应用’类工作。尽管其涉及Agent架构设计，但并未针对文档图像理解（DIU）中的具体痛点（如OCR幻觉、复杂排版、长文档推理）提出可迁移的技术方案，也未在文档任务上验证有效性。其贡献集中在AI系统的工程抽象层面，而非提升DIU性能的底层方法，因此不符合关注标准。
Abstract In an era where vast amounts of data are collected and processed from diverse sources, there is a growing demand for sophisticated AI systems capable of intelligently fusing and analyzing this information. To address these challenges, researchers have turned towards integrating tools into LLM-powered agents to enhance the overall information fusion process. However, the conjunction of these technologies and the proposed enhancements in several state-of-the-art works followed a non-unified software architecture, resulting in a lack of modularity and terminological inconsistencies among researchers. To address these issues, we propose a novel LLM-based Agent Unified Modeling Framework (LLM-Agent-UMF) that establishes a clear foundation for agent development from both functional and software architectural perspectives, developed and evaluated using the Architecture Tradeoff and Risk Analysis Framework (ATRAF). Our framework clearly distinguishes between the different components of an LLM-based agent, setting LLMs and tools apart from a new element, the core-agent, which plays the role of central coordinator. This pivotal entity comprises five modules: planning, memory, profile, action, and security -- the latter often neglected in previous works. By classifying core-agents into passive and active types based on their authoritative natures, we propose various multi-core agent architectures that combine unique characteristics of distinctive agents to tackle complex tasks more efficiently. We evaluate our framework by applying it to thirteen state-of-the-art agents, thereby demonstrating its alignment with their functionalities and clarifying overlooked architectural aspects. Moreover, we thoroughly assess five architecture variants of our framework by designing new agent architectures that combine characteristics of state-of-the-art agents to address specific goals. ...
Authors Amine Ben Hassouna, Hana Chaari, Ines Belhaj
Comment 39 pages, 19 figures, 3 tables. Published in Information Fusion, Volume 127, March 2026, 103865. Part of the special issue "Data Fusion Approaches in Data-Centric AI for Developing Trustworthy AI Systems"
Categories Artificial Intelligence
2
Designing Domain-Specific Agents via Hierarchical Task Abstraction Mechanism
Title CN 通过分层任务抽象机制设计领域特定代理
Keywords 智能体架构 · 分层任务分解 · 领域专用系统 · 多智能体系统 · 地理空间分析
Summary 本文提出一种基于分层任务抽象机制（HTAM）的领域专用智能体设计框架，并以遥感地理分析为例构建EarthAgent系统，同时发布多步地理规划评测基准GeoPlan-bench。实验表明该架构在工具调用和流程正确性上优于通用智能体。
Reason 该论文提出了一种面向专业领域的多智能体架构HTAM，并在遥感地理分析任务上实例化为EarthAgent。尽管其提出的分层任务抽象机制具有一定的方法论价值，但整体工作仍属于‘平行下游应用’：其核心贡献是将现有LLM Agent框架（如ReAct）应用于遥感这一特定领域，构建了专用的GeoPlan-bench评测集。虽然‘任务依赖图’和‘分层分解’的思想理论上可迁移至长文档理解中的多步推理流程，但文中未涉及任何文档、文本或视觉-语言交互的挑战（如OCR幻觉、排版复杂性），也未提出可用于DIU的通用底层技术（如视觉表示、推理机制、定位能力提升）。因此属于典型的‘垂类应用’而非方法源头。
Abstract LLM-driven agents, particularly those using general frameworks like ReAct or human-inspired role-playing, often struggle in specialized domains that necessitate rigorously structured workflows. Fields such as remote sensing, requiring specialized tools (e.g., correction, spectral indices calculation), and multi-step procedures (e.g., numerous intermediate products and optional steps), significantly challenge generalized approaches. To address this gap, we introduce a novel agent design framework centered on a Hierarchical Task Abstraction Mechanism (HTAM). Specifically, HTAM moves beyond emulating social roles, instead structuring multi-agent systems into a logical hierarchy that mirrors the intrinsic task-dependency graph of a given domain. This task-centric architecture thus enforces procedural correctness and decomposes complex problems into sequential layers, where each layer's sub-agents operate on the outputs of the preceding layers. We instantiate this framework as EarthAgent, a multi-agent system tailored for complex geospatial analysis. To evaluate such complex planning capabilities, we build GeoPlan-bench, a comprehensive benchmark of realistic, multi-step geospatial planning tasks. It is accompanied by a suite of carefully designed metrics to evaluate tool selection, path similarity, and logical completeness. Experiments show that EarthAgent substantially outperforms a range of established single- and multi-agent systems. Our work demonstrates that aligning agent architecture with a domain's intrinsic task structure is a critical step toward building robust and reliable specialized autonomous systems.
Authors Kaiyu Li, Jiayu Wang, Zhi Wang, Hui Qiao, Weizhan Zhang, Deyu Meng, Xiangyong Cao
Comment Page: https://earth-insights.github.io/EarthAgent
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
VLA-Pruner: Temporal-Aware Dual-Level Visual Token Pruning for Efficient Vision-Language-Action Inference
Title CN VLA-Pruner：面向高效视觉-语言-动作推理的时序感知双级视觉Token剪枝
Keywords 视觉-语言-动作模型 · Token剪枝 · 具身智能 · 时序平滑 · 推理效率
Summary 本文提出VLA-Pruner，一种面向视觉-语言-动作模型的双级视觉Token剪枝方法，结合语义级的视觉-语言预填充注意力和动作级的解码头注意力（通过时间平滑估计），以在保持动作性能的同时压缩计算开销。实验表明其在多种VLA架构和机器人任务上表现优异。
Reason 该论文聚焦于具身智能（Embodied AI）中的视觉-语言-动作（VLA）模型，核心目标是通过双级Token剪枝提升机器人任务中的推理效率。其方法设计紧密围绕动作执行与语义理解的平衡，并利用了操作过程中的时间连续性。然而，该工作属于具身智能/机器人领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管Token剪枝技术在理论上可能被迁移用于高分辨率文档处理，但其重要性判据（动作解码注意力+时间平滑）针对的是连续动作控制，难以适配到文档中静态、细粒度、结构化信息提取的需求。因此，无法为解决OCR幻觉、复杂排版或长文档推理等DIU痛点提供有效工具。
Abstract Vision-Language-Action (VLA) models have shown great promise for embodied AI, yet the heavy computational cost of processing continuous visual streams severely limits their real-time deployment. Token pruning (keeping salient visual tokens and dropping redundant ones) has emerged as an effective approach for accelerating Vision-Language Models (VLMs), offering a solution for efficient VLA. However, these VLM-specific token pruning methods select tokens based solely on semantic salience metrics (e.g., prefill attention), while overlooking the VLA's intrinsic dual-system nature of high-level semantic understanding and low-level action execution. Consequently, these methods bias token retention toward semantic cues, discard critical information for action generation, and significantly degrade VLA performance. To bridge this gap, we propose VLA-Pruner, a versatile plug-and-play VLA-specific token prune method that aligns with the dual-system nature of VLA models and exploits the temporal continuity in robot manipulation. Specifically, VLA-Pruner adopts a dual-level importance criterion for visual token retention: vision-language prefill attention for semantic-level relevance and action decode attention, estimated via temporal smoothing, for action-level importance. Based on this criterion, VLA-Pruner proposes a novel dual-level token selection strategy that adaptively preserves a compact, informative set of visual tokens for both semantic understanding and action execution under given compute budget. Experiments show that VLA-Pruner achieves state-of-the-art performance across multiple VLA architectures and diverse robotic tasks.
Authors Ziyan Liu, Yeqiu Chen, Hongyi Cai, Tao Lin, Shuo Yang, Zheng Liu, Bo Zhao
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
2
The Belief-Desire-Intention Ontology for modelling mental reality and agency
Title CN 用于建模心智现实与智能体的信念-欲望-意图本体
Keywords BDI模型 · 本体论 · 逻辑增强生成 · 智能体建模 · 语义互操作
Summary 本文提出了一个形式化的信念-欲望-意图（BDI）本体，作为模块化的设计模式，用于在语义网环境中建模智能体的认知结构。通过与大语言模型结合使用逻辑增强生成（LAG），以及集成到Semas推理平台中，验证了该本体在提升推理一致性和实现声明式与过程式智能融合方面的潜力。实验展示了其在多智能体系统和神经符号系统中的应用前景。
Reason 该论文提出了一种基于信念-欲望-意图（BDI）的本体论框架，旨在增强智能体的认知建模与语义互操作性。虽然结合了LLM并通过逻辑增强生成（LAG）提升推理一致性，但其核心是认知科学与知识表示领域的理论建模工作，属于通用AI架构设计，并未针对文档图像理解（DIU）中的具体任务如DocVQA、布局分析或OCR去幻觉等问题提出直接可迁移的技术方案。其与VLM处理文档时的关键痛点（如高分辨率输入、细粒度定位、视觉幻觉抑制）无直接关联，属于平行应用层面的概念整合，不具备对DIU的方法论推动。
Abstract The Belief-Desire-Intention (BDI) model is a cornerstone for representing rational agency in artificial intelligence and cognitive sciences. Yet, its integration into structured, semantically interoperable knowledge representations remains limited. This paper presents a formal BDI Ontology, conceived as a modular Ontology Design Pattern (ODP) that captures the cognitive architecture of agents through beliefs, desires, intentions, and their dynamic interrelations. The ontology ensures semantic precision and reusability by aligning with foundational ontologies and best practices in modular design. Two complementary lines of experimentation demonstrate its applicability: (i) coupling the ontology with Large Language Models (LLMs) via Logic Augmented Generation (LAG) to assess the contribution of ontological grounding to inferential coherence and consistency; and (ii) integrating the ontology within the Semas reasoning platform, which implements the Triples-to-Beliefs-to-Triples (T2B2T) paradigm, enabling a bidirectional flow between RDF triples and agent mental states. Together, these experiments illustrate how the BDI Ontology acts as both a conceptual and operational bridge between declarative and procedural intelligence, paving the way for cognitively grounded, explainable, and semantically interoperable multi-agent and neuro-symbolic systems operating within the Web of Data.
Authors Sara Zuppiroli, Carmelo Fabio Longo, Anna Sofia Lippolis, Rocco Paolillo, Lorenzo Giammei, Miguel Ceriani, Francesco Poggi, Antonio Zinilli, Andrea Giovanni Nuzzolese
Categories Artificial Intelligence
2
UI-CUBE: Enterprise-Grade Computer Use Agent Benchmarking Beyond Task Accuracy to Operational Reliability
Title CN UI-CUBE：超越任务准确率的企业级计算机使用代理基准，面向操作可靠性
Keywords 计算机使用代理 · 基准测试 · 操作可靠性 · 复杂工作流 · GUI自动化
Summary 本文提出了UI-CUBE，一个面向企业级计算机使用代理的操作可靠性基准，包含226个任务，覆盖简单界面交互与复杂工作流。实验表明当前模型在复杂任务上性能急剧下降，揭示了其在记忆管理、分层规划和状态协调方面的根本缺陷。该基准旨在诊断代理在真实生产环境中的部署能力。
Reason 该论文属于典型的‘平行应用’类工作，聚焦于企业级计算机使用代理（CUA）的基准测试，尽管涉及复杂工作流和多步操作，但其核心是评估代理在GUI自动化中的可靠性，而非解决文档图像理解（DIU）或DocVQA中的核心技术问题。虽然其对复杂流程中状态协调与记忆管理的分析具有一定通用性，但该方法无法直接迁移用于解决OCR幻觉、密集文本识别、版面分析或长文档推理等DIU核心痛点。
Abstract While current Computer Use Agent (CUA) benchmarks measure task completion effectively, they provide limited assessment of enterprise deployment readiness, emphasizing functional correctness over the operational reliability required for production systems. We present UI-CUBE (UiPath Computer Use BEnchmark), a systematic benchmark comprising 226 tasks across two difficulty tiers designed to expose fundamental architectural limitations in current CUAs. Our evaluation covers simple UI interactions (136 tasks) and complex workflows including copy-paste tasks (50 tasks) and enterprise application scenarios (40 tasks), with systematic interface variation coverage, multi-resolution testing and automated validation of task success through the application state. Evaluation of five state-of-the-art models reveals a sharp capability cliff rather than gradual performance degradation. Simple UI interactions achieve 67-85% success rates (compared to 97.9% human performance), but complex workflows drop precipitously to 9-19%. Human evaluators with no prior application experience achieve only 61.2% on complex tasks despite near-perfect performance on simple tasks, establishing realistic performance ceilings. This discontinuous performance pattern -- where agents achieve 68-87% of human performance on simple tasks but only 15-32% on complex workflows -- indicates fundamental architectural limitations in memory management, hierarchical planning, and state coordination rather than incremental capability gaps addressable through better training or prompting. UI-CUBE functions as an enterprise-readiness diagnostic, revealing that while current CUAs can manipulate individual interface elements, they cannot yet function as reliable workflow automation tools. These findings provide architectural insights essential for developing production-ready CUAs capable of managing complex, multi-step enterprise processes.
Authors Horia Cristescu, Charles Park, Trong Canh Nguyen, Sergiu Talmacel, Alexandru-Gabriel Ilie, Stefan Adam
Comment 18 pages, 8 figures, 5 tables. Benchmark comprising 226 tasks across two difficulty tiers. Code and benchmark available at https://github.com/UiPath/uipath_enterprise_benchmark
Categories Artificial Intelligence
2
RTMol: Rethinking Molecule-text Alignment in a Round-trip View
Title CN RTMol：从往返视角重新思考分子-文本对齐
Keywords 分子-文本对齐 · 自监督学习 · 跨模态生成 · SMILES · 双向一致性
Summary 本文提出RTMol，一种通过往返式自监督学习实现分子到文本和文本到分子双向对齐的框架。它引入了新的往返评估指标，并支持无需配对数据的无监督训练。实验表明该方法在多个大语言模型上显著提升了对齐性能。
Reason 该论文研究的是分子序列（如SMILES）与化学文本描述之间的双向对齐问题，属于化学信息学与多模态学习的交叉领域。尽管其方法论涉及跨模态对齐和自监督学习，但其任务场景与文档图像理解（DIU）的核心问题——如OCR纠错、版面分析、表格识别、视觉文本推理等——无直接关联。所解决的‘化学准确性’与‘双向一致性’问题无法迁移到文档图像中的视觉-语言对齐任务上。且未涉及图像输入、视觉表示或空间布局建模，对DIU领域无方法论启发。
Abstract Aligning molecular sequence representations (e.g., SMILES notations) with textual descriptions is critical for applications spanning drug discovery, materials design, and automated chemical literature analysis. Existing methodologies typically treat molecular captioning (molecule-to-text) and text-based molecular design (text-to-molecule) as separate tasks, relying on supervised fine-tuning or contrastive learning pipelines. These approaches face three key limitations: (i) conventional metrics like BLEU prioritize linguistic fluency over chemical accuracy, (ii) training datasets frequently contain chemically ambiguous narratives with incomplete specifications, and (iii) independent optimization of generation directions leads to bidirectional inconsistency. To address these issues, we propose RTMol, a bidirectional alignment framework that unifies molecular captioning and text-to-SMILES generation through self-supervised round-trip learning. The framework introduces novel round-trip evaluation metrics and enables unsupervised training for molecular captioning without requiring paired molecule-text corpora. Experiments demonstrate that RTMol enhances bidirectional alignment performance by up to 47% across various LLMs, establishing an effective paradigm for joint molecule-text understanding and generation.
Authors Letian Chen, Runhan Shi, Gufeng Yu, Yang Yang
Categories Artificial Intelligence
2
Structured Debate Improves Corporate Credit Reasoning in Financial AI
Title CN 结构化辩论提升金融AI中的企业信贷推理
Keywords 金融AI · 多智能体系统 · 结构化推理 · 信贷评估 · LLM应用
Summary 本文提出了两种基于大语言模型的企业信贷推理系统：一种是单智能体双向分析系统（NAS），另一种是基于卡尔·波普尔批判性对话框架的十步结构化多智能体辩论系统（KPD-MADS）。在真实企业案例中，后者在解释充分性、实用性与可用性方面均优于人工报告，显著提升了推理效率与质量。然而，该工作局限于金融领域的决策推理自动化，未涉及任何视觉或文档模态的理解。
Reason 该论文属于典型的‘平行下游应用’，将多Agent辩论框架应用于企业信贷评估这一特定领域。尽管其提出的KPD-MADS系统在推理结构和对抗性验证上有一定设计，但整体工作聚焦于金融风控场景的落地，未提出可迁移至文档图像理解（DIU）领域的底层方法或架构创新。例如，未涉及视觉输入处理、OCR去幻觉、细粒度定位或文档布局建模等核心DIU问题。其‘structured debate’机制虽具启发性，但属于任务层推理流程设计，难以直接迁移用于解决文档理解中的幻觉或复杂排版解析问题。
Abstract Despite advances in financial AI, the automation of evidence-based reasoning remains unresolved in corporate credit assessment, where qualitative non-financial indicators exert decisive influence on loan repayment outcomes yet resist formalization. Existing approaches focus predominantly on numerical prediction and provide limited support for the interpretive judgments required in professional loan evaluation. This study develops and evaluates two operational large language model (LLM)-based systems designed to generate structured reasoning from non-financial evidence. The first is a non-adversarial single-agent system (NAS) that produces bidirectional analysis through a single-pass reasoning pipeline. The second is a debate-based multi-agent system (KPD-MADS) that operationalizes adversarial verification through a ten-step structured interaction protocol grounded in Karl Popper's critical dialogue framework. Both systems were applied to three real corporate cases and evaluated by experienced credit risk professionals. Compared to manual expert reporting, both systems achieved substantial productivity gains (NAS: 11.55 s per case; KPD-MADS: 91.97 s; human baseline: 1920 s). The KPD-MADS demonstrated superior reasoning quality, receiving higher median ratings in explanatory adequacy (4.0 vs. 3.0), practical applicability (4.0 vs. 3.0), and usability (62.5 vs. 52.5). These findings show that structured multi-agent interaction can enhance reasoning rigor and interpretability in financial AI, advancing scalable and defensible automation in corporate credit assessment.
Authors Yoonjin Lee, Munhee Kim, Hanbi Choi, Juhyeon Park, Seungho Lyoo, Woojin Park
Comment 18 pages, 4 figures, 2 algorithms, 2 tables, 4 appendices
Categories Artificial Intelligence
2
Patient-level Information Extraction by Consistent Integration of Textual and Tabular Evidence with Bayesian Networks
Title CN 通过贝叶斯网络一致整合文本和表格证据进行患者级信息抽取
Keywords 信息抽取 · 电子健康记录 · 贝叶斯网络 · 多模态融合 · 临床决策支持
Summary 本文提出一种结合电子健康记录中结构化表格数据和非结构化临床文本的多模态患者信息提取方法。利用专家定义的贝叶斯网络处理表格特征，并用神经文本分类器解析临床笔记，再通过虚拟证据和一致性节点实现概率性融合，提升预测校准并解决跨模态信息冲突。实验基于SimSUM模拟数据集进行验证。
Reason 该论文聚焦于医疗电子病历中的多模态信息融合，属于典型的‘平行下游应用’。其方法核心是使用贝叶斯网络结合结构化表格与神经文本分类器进行患者级信息提取，应用场景明确限定在临床决策支持系统。尽管涉及多模态融合与去矛盾机制，但整体架构针对特定领域设计，未提出可迁移至文档图像理解（如DocVQA、OCR去幻觉、复杂排版处理）的通用方法论或底层技术创新。属于将已有技术（贝叶斯推理+文本分类）应用于垂直场景的典型例子，无DIU层面的方法创新。
Abstract Electronic health records (EHRs) form an invaluable resource for training clinical decision support systems. To leverage the potential of such systems in high-risk applications, we need large, structured tabular datasets on which we can build transparent feature-based models. While part of the EHR already contains structured information (e.g. diagnosis codes, medications, and lab results), much of the information is contained within unstructured text (e.g. discharge summaries and nursing notes). In this work, we propose a method for multi-modal patient-level information extraction that leverages both the tabular features available in the patient's EHR (using an expert-informed Bayesian network) as well as clinical notes describing the patient's symptoms (using neural text classifiers). We propose the use of virtual evidence augmented with a consistency node to provide an interpretable, probabilistic fusion of the models' predictions. The consistency node improves the calibration of the final predictions compared to virtual evidence alone, allowing the Bayesian network to better adjust the neural classifier's output to handle missing information and resolve contradictions between the tabular and text data. We show the potential of our method on the SimSUM dataset, a simulated benchmark linking tabular EHRs with clinical notes through expert knowledge.
Authors Paloma Rabaey, Adrick Tench, Stefan Heytens, Thomas Demeester
Categories Artificial Intelligence
2
DAPS++: Rethinking Diffusion Inverse Problems with Decoupled Posterior Annealing
Title CN DAPS++：基于解耦后验退火重新思考扩散逆问题
Keywords 扩散模型 · 逆问题 · 图像恢复 · 贝叶斯推理 · EM算法
Summary 本文从贝叶斯视角重新审视基于分数的扩散模型在逆问题中的应用，指出实际重建过程主要由测量一致性项驱动，而先验信息作用有限。为此提出DAPS++方法，将扩散过程视为期望最大化（EM）框架中的初始化阶段，实现扩散动态与数据驱动优化的完全解耦，在减少函数计算次数的同时提升图像恢复效率与稳定性。
Reason 该论文聚焦于图像恢复中的扩散模型逆问题求解，属于纯图像生成与重建领域，尽管其方法涉及贝叶斯推理和EM框架优化，但研究任务与文档图像理解（DIU）的核心痛点如OCR幻觉、文本识别、布局分析、表格理解等无直接关联。所提方法主要面向通用图像修复（如去噪、超分），未体现对文本结构、多模态对齐或视觉-语言推理的支持，难以迁移到DocVQA或端到端文档理解场景。
Abstract From a Bayesian perspective, score-based diffusion solves inverse problems through joint inference, embedding the likelihood with the prior to guide the sampling process. However, this formulation fails to explain its practical behavior: the prior offers limited guidance, while reconstruction is largely driven by the measurement-consistency term, leading to an inference process that is effectively decoupled from the diffusion dynamics. To clarify this structure, we reinterpret the role of diffusion in inverse problem solving as an initialization stage within an expectation--maximization (EM)--style framework, where the diffusion stage and the data-driven refinement are fully decoupled. We introduce \textbf{DAPS++}, which allows the likelihood term to guide inference more directly while maintaining numerical stability and providing insight into why unified diffusion trajectories remain effective in practice. By requiring fewer function evaluations (NFEs) and measurement-optimization steps, \textbf{DAPS++} achieves high computational efficiency and robust reconstruction performance across diverse image restoration tasks.
Authors Hao Chen, Renzheng Zhang, Scott S. Howard
Categories Artificial Intelligence
2
GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?
Title CN GhostEI-Bench：移动智能体在动态设备环境中的环境注入攻击下是否具有韧性？
Keywords 视觉语言模型 · 环境注入 · 移动智能体 · GUI安全 · 对抗攻击
Summary 本文提出了GhostEI-Bench，首个评估移动视觉语言智能体在动态设备环境中抵御环境注入攻击的基准。通过在Android模拟器中注入对抗性UI事件（如虚假通知），测试智能体在真实应用流程中的表现，并利用judge-LLM对失败类型进行细粒度归因。实验表明现有智能体在面对此类攻击时存在严重脆弱性。
Reason 该论文聚焦于移动GUI环境下的视觉语言模型智能体面对环境注入攻击（如欺骗性弹窗）的鲁棒性问题，属于具身智能与安全交叉领域。尽管涉及VLM的视觉感知与推理，但其任务场景为移动应用操作，与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂版式、密集文本识别）无直接关联。所提出的基准GhostEI-Bench针对的是动态UI干扰下的行为轨迹分析，并非用于提升文档理解能力的方法论或架构创新，无法迁移应用于DocVQA或Layout Analysis等任务。
Abstract Vision-Language Models (VLMs) are increasingly deployed as autonomous agents to navigate mobile graphical user interfaces (GUIs). Operating in dynamic on-device ecosystems, which include notifications, pop-ups, and inter-app interactions, exposes them to a unique and underexplored threat vector: environmental injection. Unlike prompt-based attacks that manipulate textual instructions, environmental injection corrupts an agent's visual perception by inserting adversarial UI elements (for example, deceptive overlays or spoofed notifications) directly into the GUI. This bypasses textual safeguards and can derail execution, causing privacy leakage, financial loss, or irreversible device compromise. To systematically evaluate this threat, we introduce GhostEI-Bench, the first benchmark for assessing mobile agents under environmental injection attacks within dynamic, executable environments. Moving beyond static image-based assessments, GhostEI-Bench injects adversarial events into realistic application workflows inside fully operational Android emulators and evaluates performance across critical risk scenarios. We further propose a judge-LLM protocol that conducts fine-grained failure analysis by reviewing the agent's action trajectory alongside the corresponding screenshot sequence, pinpointing failure in perception, recognition, or reasoning. Comprehensive experiments on state-of-the-art agents reveal pronounced vulnerability to deceptive environmental cues: current models systematically fail to perceive and reason about manipulated UIs. GhostEI-Bench provides a framework for quantifying and mitigating this emerging threat, paving the way toward more robust and secure embodied agents.
Authors Chiyu Chen, Xinhao Song, Yunkai Chai, Yang Yao, Haodong Zhao, Lijun Li, Jie Li, Yan Teng, Gongshen Liu, Yingchun Wang
Categories Artificial Intelligence
2
MirrorMind: Empowering OmniScientist with the Expert Perspectives and Collective Knowledge of Human Scientists
Title CN MirrorMind：通过人类科学家的专业视角和集体知识赋能全能科学家
Keywords AI Scientist · Cognitive Architecture · Multi-agent System · Collective Knowledge · Scientific Reasoning
Summary 本文提出MirrorMind，一种分层认知架构，通过个体级、领域级和跨学科级三层框架模拟人类科学家的认知轨迹与集体知识网络。该系统结合个人记忆模型与学科概念图谱，支持多智能体进行个性化推理与跨学科协作，在作者认知模拟、互补推理等任务中表现良好。然而，该工作完全基于纯文本与知识图谱，不涉及图像或文档视觉内容的理解。
Reason 该论文聚焦于构建AI科学家的认知架构，用于模拟科研人员的思维过程与跨学科协作，属于典型的‘平行应用’型研究。其核心贡献在于多智能体系统在科学研究自动化中的组织框架，并未提出可用于文档图像理解（如OCR去幻觉、复杂排版建模、视觉token压缩等）的底层方法。尽管其记忆分层结构可能对长文档信息组织有一定启发，但迁移路径模糊，且无任何视觉或多模态输入处理机制，无法解决DIU的核心痛点。
Abstract The emergence of AI Scientists has demonstrated remarkable potential in automating scientific research. However, current approaches largely conceptualize scientific discovery as a solitary optimization or search process, overlooking that knowledge production is inherently a social and historical endeavor. Human scientific insight stems from two distinct yet interconnected sources. First is the individual cognitive trajectory, where a researcher's unique insight is shaped by their evolving research history and stylistic preferences; another is the collective disciplinary memory, where knowledge is sedimented into vast, interconnected networks of citations and concepts. Existing LLMs still struggle to represent these structured, high-fidelity cognitive and social contexts. To bridge this gap, we introduce MirrorMind, a hierarchical cognitive architecture that integrates dual-memory representations within a three-level framework. The Individual Level constructs high-fidelity cognitive models of individual researchers by capturing their episodic, semantic, and persona memories; the Domain Level maps collective knowledge into structured disciplinary concept graphs; and the Interdisciplinary Level that acts as an orthogonal orchestration engine. Crucially, our architecture separates memory storage from agentic execution, enabling AI scientist agents to flexibly access individual memories for unique perspectives or collective structures to reason. We evaluate MirrorMind across four comprehensive tasks, including author-level cognitive simulation, complementary reasoning, cross-disciplinary collaboration promotion, and multi-agent scientific problem solving. The results show that by integrating individual cognitive depth with collective disciplinary breadth, MirrorMind moves beyond simple fact retrieval toward structural, personalized, and insight-generating scientific reasoning.
Authors Qingbin Zeng, Bingbing Fan, Zhiyu Chen, Sijian Ren, Zhilun Zhou, Xuhua Zhang, Yuanyi Zhen, Fengli Xu, Yong Li, Tie-Yan Liu
Comment 26 pages, 4 figures
Categories Artificial Intelligence
2
Comparing verbal, visual and combined explanations for Bayesian Network inferences
Title CN 比较贝叶斯网络推理中的言语、视觉和组合解释
Keywords 贝叶斯网络 · 可解释AI · 用户界面 · 多模态解释 · 人机交互
Summary 本文提出了一种增强贝叶斯网络用户界面的方法，通过添加言语解释、视觉引导或两者结合来提升用户对BN推理的理解，并通过用户实验验证了这些扩展的有效性。结果显示多模态解释优于单一模态。然而，该工作聚焦于认知辅助与可解释性设计，与文档图像理解无直接关联。
Reason 该论文研究的是贝叶斯网络（BN）的用户界面设计，旨在通过言语和视觉方式帮助人类用户理解BN的推理过程。其核心是人机交互与可解释AI，在任务设定和方法上均未涉及文档图像理解（DIU）、OCR、VQA或多模态语言模型对文档内容的解析。尽管使用了‘视觉’模态，但这里的视觉指的是图示化解释BN结构，而非处理真实文档图像或布局。属于无关领域的人因研究，不具迁移到DIU的技术价值。
Abstract Bayesian Networks (BNs) are an important tool for assisting probabilistic reasoning, but despite being considered transparent models, people have trouble understanding them. Further, current User Interfaces (UIs) still do not clarify the reasoning of BNs. To address this problem, we have designed verbal and visual extensions to the standard BN UI, which can guide users through common inference patterns. We conducted a user study to compare our verbal, visual and combined UI extensions, and a baseline UI. Our main findings are: (1) users did better with all three types of extensions than with the baseline UI for questions about the impact of an observation, the paths that enable this impact, and the way in which an observation influences the impact of other observations; and (2) using verbal and visual modalities together is better than using either modality alone for some of these question types.
Authors Erik P. Nyberg, Steven Mascaro, Ingrid Zukerman, Michael Wybrow, Duc-Minh Vo, Ann Nicholson
Comment 26 pages total, 12 pages main, 14 pages for 5 appendices
Categories Artificial Intelligence
2
APRIL: Annotations for Policy evaluation with Reliable Inference from LLMs
Title CN APRIL：利用大语言模型进行可靠推理的策略评估注释
Keywords 大语言模型 · 离线策略评估 · 反事实标注 · 医疗AI · 数据增强
Summary 本文提出APRIL方法，利用大语言模型生成医疗场景下的反事实临床特征标注，以增强离线策略评估的数据覆盖性，提升高风险领域中决策策略的安全性。实验表明，在MIMIC-IV数据集上，LLM生成的标注能有效改善策略估计效果，直到达到信息熵饱和点。
Reason 该论文属于典型的平行下游应用，将LLM用于医疗领域的反事实标注生成以改进离线策略评估（OPE），应用场景局限于医疗决策，未提出可用于文档图像理解（如OCR去幻觉、复杂排版处理等）的通用方法或架构创新。尽管使用了LLM生成反事实信息，但其核心是特定领域（医疗）的数据增强手段，无法迁移至DIU中的视觉-语言对齐、细粒度文本识别或长文档推理等任务。
Abstract Off-policy evaluation (OPE) estimates the value of a contextual bandit policy prior to deployment. As such, OPE plays a critical role in ensuring safety in high-stakes domains such as healthcare. However, standard OPE approaches are limited by the size and coverage of the behavior dataset. While previous work has explored using expert-labeled counterfactual annotations to enhance dataset coverage, obtaining such annotations is expensive, limiting the scalability of prior approaches. We propose leveraging large language models (LLMs) to generate counterfactual annotations for OPE in medical domains. Our method uses domain knowledge to guide LLMs in predicting how key clinical features evolve under alternate treatments. These predicted features can then be transformed using known reward functions to create counterfactual annotations. We first evaluate the ability of several LLMs to predict clinical features across two patient subsets in MIMIC-IV, finding that state-of-the-art LLMs achieve comparable performance. Building on this capacity to predict clinical features, we generate LLM-based counterfactual annotations and incorporate them into an OPE estimator. Our empirical results analyze the benefits of counterfactual annotations under varying degrees of shift between the behavior and target policies. We find that in most cases, the LLM-based counterfactual annotations significantly improve OPE estimates up to a point. We provide an entropy-based metric to identify when additional annotations cease to be useful. Our results demonstrate that LLM-based counterfactual annotations offer a scalable approach for addressing coverage limitations in healthcare datasets, enabling safer deployment of decision-making policies in clinical settings.
Authors Aishwarya Mandyam, Kalyani Limaye, Barbara E. Engelhardt, Emily Alsentzer
Categories Artificial Intelligence
2
Episodic Memory in Agentic Frameworks: Suggesting Next Tasks
Title CN 代理框架中的情景记忆：建议下一步任务
Keywords Agent · Episodic Memory · Task Recommendation · Workflow Automation · LLM
Summary 本文提出一种基于情景记忆的架构，通过存储和检索历史工作流来帮助LLM驱动的Agent在科学工作流中推荐下一步任务，减少对LLM本身推理的依赖，从而降低幻觉风险。方法侧重于任务序列匹配与复用，适用于人类-AI协作场景。
Reason 该论文属于典型的‘平行应用’类工作，聚焦于科学工作流中基于LLM的Agent如何推荐下一步操作，其核心是通用Agent的任务规划机制，并未针对文档图像理解（DIU）中的任何具体任务（如DocVQA、布局分析、表格识别等）进行建模或改进。虽然情景记忆机制在理论上可用于长文档处理中的流程控制，但本文未涉及视觉输入、多模态对齐、OCR幻觉、细粒度定位等DIU核心问题，也未提出可迁移至文档理解场景的底层方法。属于无关领域的通用Agent设计，不符合关注标准。
Abstract Agentic frameworks powered by Large Language Models (LLMs) can be useful tools in scientific workflows by enabling human-AI co-creation. A key challenge is recommending the next steps during workflow creation without relying solely on LLMs, which risk hallucination and require fine-tuning with scarce proprietary data. We propose an episodic memory architecture that stores and retrieves past workflows to guide agents in suggesting plausible next tasks. By matching current workflows with historical sequences, agents can recommend steps based on prior patterns.
Authors Sandro Rama Fiorini, Leonardo G. Azevedo, Raphael M. Thiago, Valesca M. de Sousa, Anton B. Labate, Viviane Torres da Silva
Categories Artificial Intelligence
2
Illustrator's Depth: Monocular Layer Index Prediction for Image Decomposition
Title CN 插画师的深度：用于图像分解的单目图层索引预测
Keywords 图像分解 · 图层索引 · 可编辑性 · 矢量图形 · 深度预测
Summary 本文提出了一种名为“插画师的深度”的新深度定义，通过为每个像素预测一个离散的图层索引，实现对扁平图像的可编辑分解。该方法受艺术家创作过程启发，强调全局一致性和可编辑性，并基于向量图形数据集训练神经网络从光栅图像中预测图层顺序。实验表明其在图像矢量化、文生矢量图、2D转3D浮雕等任务上优于现有方法。
Reason 该论文研究的是通用图像的可编辑分层分解，核心任务是将平面图像分解为具有创作意义的图层顺序，属于数字内容创作领域。尽管其‘层索引’概念在视觉结构建模上有启发性，但其目标与文档图像理解（DIU）的核心任务（如DocVQA、表格识别、文本定位等）无直接关联。未解决OCR幻觉、密集文本处理或复杂排版理解等DIU痛点，且方法针对的是图形设计场景下的抽象深度建模，难以迁移到以语义和布局结构为核心的文档理解任务。
Abstract We introduce Illustrator's Depth, a novel definition of depth that addresses a key challenge in digital content creation: decomposing flat images into editable, ordered layers. Inspired by an artist's compositional process, illustrator's depth infers a layer index to each pixel, forming an interpretable image decomposition through a discrete, globally consistent ordering of elements optimized for editability. We also propose and train a neural network using a curated dataset of layered vector graphics to predict layering directly from raster inputs. Our layer index inference unlocks a range of powerful downstream applications. In particular, it significantly outperforms state-of-the-art baselines for image vectorization while also enabling high-fidelity text-to-vector-graphics generation, automatic 3D relief generation from 2D images, and intuitive depth-aware editing. By reframing depth from a physical quantity to a creative abstraction, illustrator's depth prediction offers a new foundation for editable image decomposition.
Authors Nissim Maruani, Peiying Zhang, Siddhartha Chaudhuri, Matthew Fisher, Nanxuan Zhao, Vladimir G. Kim, Pierre Alliez, Mathieu Desbrun, Wang Yifan
Categories Computer Vision and Pattern Recognition
2
Robot Confirmation Generation and Action Planning Using Long-context Q-Former Integrated with Multimodal LLM ASRU 2025
Title CN 使用集成多模态大语言模型的长上下文Q-Former生成机器人确认与动作规划
Keywords 多模态大语言模型 · 长上下文建模 · 机器人交互 · Q-Former · 动作规划
Summary 本文提出一种结合长上下文Q-Former的多模态大语言模型，用于在人机对话中实现机器人动作确认与规划。通过引入左右上下文依赖关系，并将文本嵌入直接输入LLM解码器，提升对长周期任务的理解能力。实验表明，在YouCook2数据集上，动作确认准确率显著影响整体规划性能。
Reason 该论文属于具身智能与机器人领域，研究的是人机协作中的对话与动作规划问题，核心任务是基于视频理解进行机器人行为决策。尽管使用了多模态LLM和长上下文建模技术，但其应用场景与文档图像理解（DIU）无直接关联，且方法设计面向视频时序推理而非文档结构解析、OCR去幻觉或复杂排版处理等DIU痛点。无法迁移用于解决文档理解中的密集文本识别、细粒度定位或长文档推理等问题。
Abstract Human-robot collaboration towards a shared goal requires robots to understand human action and interaction with the surrounding environment. This paper focuses on human-robot interaction (HRI) based on human-robot dialogue that relies on the robot action confirmation and action step generation using multimodal scene understanding. The state-of-the-art approach uses multimodal transformers to generate robot action steps aligned with robot action confirmation from a single clip showing a task composed of multiple micro steps. Although actions towards a long-horizon task depend on each other throughout an entire video, the current approaches mainly focus on clip-level processing and do not leverage long-context information. This paper proposes a long-context Q-former incorporating left and right context dependency in full videos. Furthermore, this paper proposes a text-conditioning approach to feed text embeddings directly into the LLM decoder to mitigate the high abstraction of the information in text by Q-former. Experiments with the YouCook2 corpus show that the accuracy of confirmation generation is a major factor in the performance of action planning. Furthermore, we demonstrate that the long-context Q-former improves the confirmation and action planning by integrating VideoLLaMA3.
Authors Chiori Hori, Yoshiki Masuyama, Siddarth Jain, Radu Corcodel, Devesh Jha, Diego Romeres, Jonathan Le Roux
Comment Accepted to ASRU 2025
Categories Computer Vision and Pattern Recognition, Computation and Language
2
FireScope: Wildfire Risk Prediction with a Chain-of-Thought Oracle
Title CN FireScope：基于思维链预言机的野火风险预测
Keywords VLM · 野火风险预测 · 思维链 · 跨域泛化 · 空间建模
Summary 该研究提出了一种基于视觉语言模型的野火风险预测框架FireScope，结合Sentinel-2影像与气候数据，通过强化学习与视觉监督联合训练，生成高分辨率风险图并输出可解释的推理轨迹。在跨大陆场景下验证了模型的泛化能力，并构建了FireScope-Bench基准。然而，其方法本质是VLM在特定垂直领域的应用，缺乏对文档理解任务有普适价值的技术突破。
Reason 该论文属于典型的‘平行下游应用’，即将VLM和推理机制应用于野火风险预测这一特定地理空间任务。尽管其提出了Chain-of-Thought Oracle和跨大陆泛化能力，但核心是将现有VLM技术应用于新领域（wildfire risk），并未提出可用于文档图像理解（如OCR去幻觉、复杂排版建模、细粒度定位等）的底层方法创新。任务目标为生成风险栅格图并附带推理轨迹，与DIU中的DocVQA或布局分析无直接可迁移的技术路径。属于负面清单中的‘Wrapper/Application’类型。
Abstract Predicting wildfire risk is a reasoning-intensive spatial problem that requires the integration of visual, climatic, and geographic factors to infer continuous risk maps. Existing methods lack the causal reasoning and multimodal understanding required for reliable generalization. We introduce $\textbf{FireScope-Bench}$, a large-scale dataset and benchmark that couples Sentinel-2 imagery and climate data with expert-defined risk rasters across the USA, and real wildfire events in Europe for cross-continental evaluation. Building on this dataset, we propose $\textbf{FireScope}$, a VLM-based reasoning-to-generation framework that learns from both reinforcement learning and visual supervision to predict risk rasters with complementary reasoning traces. When trained in the USA and tested in Europe, $\textbf{FireScope}$ achieves substantial performance gains, while expert feedback and automated analysis confirm that its reasoning traces are faithful and semantically meaningful. Our findings demonstrate that reasoning can ground raster prediction models, improving both generalization and interpretability. To our knowledge, this is the first framework to (1) demonstrate that language-based reasoning can improve generalization in visual generation, (2) propose a high-resolution wildfire risk model that can be applied across continents, and (3) enable systematic studies of robust cross-continental generalization for multimodal fire risk models. We believe that $\textbf{FireScope-Bench}$ has the potential to serve as a foundation for advancing reasoning-driven, interpretable and generalizable spatial modeling. Data and source code will be made publicly available.
Authors Mario Markov, Stefan Maria Ailuro, Luc Van Gool, Konrad Schindler, Danda Pani Paudel
Categories Computer Vision and Pattern Recognition
2
Do Vision-Language Models Understand Visual Persuasiveness? NeurIPS 2025 Workshop: VLM4RWD
Title CN 视觉-语言模型理解视觉说服力吗？
Keywords 视觉说服力 · 多模态理解 · 认知引导 · VLM分析 · 意图理解
Summary 本文探究视觉-语言模型是否能理解视觉说服力，构建了二元说服力判断数据集并提出视觉说服因素分类体系。实验发现模型存在高召回偏差，难以区分低/中层视觉特征，而高层语义对齐才是主要预测因子。结果显示，简洁且基于对象的推理可提升模型表现，但整体仍缺乏对传播意图的理解。
Reason 该论文研究的是视觉-语言模型在判断图像视觉说服力（如广告效果）上的能力，属于社会认知与心理影响范畴。其任务设定、数据构建和评估指标均围绕‘persuasiveness’展开，与文档图像理解（DIU）的核心任务如DocVQA、版式分析、表格识别等无直接关联。提出的干预方法（如基于对象的推理引导）虽涉及VLM推理优化，但其目标是提升对人类态度预测的准确性，无法迁移解决OCR幻觉、密集文本识别或复杂排版理解等DIU关键问题。
Abstract Recent advances in vision-language models (VLMs) have enabled impressive multi-modal reasoning and understanding. Yet, whether these models truly grasp visual persuasion-how visual cues shape human attitudes and decisions-remains unclear. To probe this question, we construct a high-consensus dataset for binary persuasiveness judgment and introduce the taxonomy of Visual Persuasive Factors (VPFs), encompassing low-level perceptual, mid-level compositional, and high-level semantic cues. We also explore cognitive steering and knowledge injection strategies for persuasion-relevant reasoning. Empirical analysis across VLMs reveals a recall-oriented bias-models over-predict high persuasiveness-and weak discriminative power for low/mid-level features. In contrast, high-level semantic alignment between message and object presence emerges as the strongest predictor of human judgment. Among intervention strategies, simple instruction or unguided reasoning scaffolds yield marginal or negative effects, whereas concise, object-grounded rationales significantly improve precision and F1 scores. These results indicate that VLMs core limitation lies not in recognizing persuasive objects but in linking them to communicative intent.
Authors Gyuwon Park
Comment 8 pages (except for reference and appendix), 5 figures, 7 tables, to be published in NeurIPS 2025 Workshop: VLM4RWD
Categories Computer Vision and Pattern Recognition, Computation and Language
2
Vision Language Models are Confused Tourists
Title CN 视觉语言模型是困惑的游客
Keywords 视觉语言模型 · 文化鲁棒性 · 注意力干扰
Summary 本文提出ConfusedTourist评测套件，用于测试视觉语言模型在多种文化线索共存时的稳定性。实验表明，当前VLM在图像堆叠或生成式干扰下准确率显著下降，原因是模型注意力被无关文化线索吸引。这项工作揭示了VLM在多元文化输入下的脆弱性，但不涉及文档结构、布局分析或文本识别等DIU核心问题。
Reason 该论文研究的是视觉语言模型在跨文化场景下的鲁棒性问题，聚焦于地理和文化线索混淆对模型性能的影响。虽然涉及VLM的注意力分散问题，但其核心任务与文档图像理解（DIU）无关，未针对OCR幻觉、复杂排版、细粒度文本识别或长文档推理等DIU关键挑战提出方法。属于社会文化导向的VLM评估工作，无法迁移至文档理解场景。
Abstract Although the cultural dimension has been one of the key aspects in evaluating Vision-Language Models (VLMs), their ability to remain stable across diverse cultural inputs remains largely untested, despite being crucial to support diversity and multicultural societies. Existing evaluations often rely on benchmarks featuring only a singular cultural concept per image, overlooking scenarios where multiple, potentially unrelated cultural cues coexist. To address this gap, we introduce ConfusedTourist, a novel cultural adversarial robustness suite designed to assess VLMs' stability against perturbed geographical cues. Our experiments reveal a critical vulnerability, where accuracy drops heavily under simple image-stacking perturbations and even worsens with its image-generation-based variant. Interpretability analyses further show that these failures stem from systematic attention shifts toward distracting cues, diverting the model from its intended focus. These findings highlight a critical challenge: visual cultural concept mixing can substantially impair even state-of-the-art VLMs, underscoring the urgent need for more culturally robust multimodal understanding.
Authors Patrick Amadeus Irawan, Ikhlasul Akmal Hanif, Muhammad Dehan Al Kautsar, Genta Indra Winata, Fajri Koto, Alham Fikri Aji
Categories Computer Vision and Pattern Recognition, Computation and Language
2
FinCriticalED: A Visual Benchmark for Financial Fact-Level OCR Evaluation
Title CN FinCriticalED：面向金融事实级OCR评估的视觉基准
Keywords OCR评估 · 金融文档 · 事实正确性 · LLM-as-Judge · 基准数据集
Summary 本文提出了FinCriticalED，一个包含500个图像-HTML对的金融文档基准，聚焦于事实级别的OCR错误检测，特别是数值和时间信息的准确性。通过金融专家标注关键事实，并设计基于LLM的评估流水线进行结构化事实提取与上下文验证。实验表明当前模型在复杂视觉情境下仍存在显著错误。该工作强调了传统文本相似度指标（如ROUGE）在高风险场景下的不足。
Reason 该论文构建了一个面向金融文档的事实级OCR评估基准，核心贡献在于提出新的评估标准和标注流程，并使用LLM-as-Judge进行事实正确性判断。然而，其本质是将现有OCR和VLM技术应用于金融领域这一特定场景，属于典型的‘平行下游应用’。尽管标注质量高、领域专业性强，但未提出任何底层方法创新（如视觉表示、架构设计、推理机制等），无法迁移为解决DIU通用痛点（如OCR幻觉、复杂排版建模）的技术工具。
Abstract We introduce FinCriticalED (Financial Critical Error Detection), a visual benchmark for evaluating OCR and vision language models on financial documents at the fact level. Financial documents contain visually dense and table heavy layouts where numerical and temporal information is tightly coupled with structure. In high stakes settings, small OCR mistakes such as sign inversion or shifted dates can lead to materially different interpretations, while traditional OCR metrics like ROUGE and edit distance capture only surface level text similarity. \ficriticaled provides 500 image-HTML pairs with expert annotated financial facts covering over seven hundred numerical and temporal facts. It introduces three key contributions. First, it establishes the first fact level evaluation benchmark for financial document understanding, shifting evaluation from lexical overlap to domain critical factual correctness. Second, all annotations are created and verified by financial experts with strict quality control over signs, magnitudes, and temporal expressions. Third, we develop an LLM-as-Judge evaluation pipeline that performs structured fact extraction and contextual verification for visually complex financial documents. We benchmark OCR systems, open source vision language models, and proprietary models on FinCriticalED. Results show that although the strongest proprietary models achieve the highest factual accuracy, substantial errors remain in visually intricate numerical and temporal contexts. Through quantitative evaluation and expert case studies, FinCriticalED provides a rigorous foundation for advancing visual factual precision in financial and other precision critical domains.
Authors Yueru He, Xueqing Peng, Yupeng Cao, Yan Wang, Lingfei Qian, Haohang Li, Yi Han, Ruoyu Xiang, Mingquan Lin, Prayag Tiwari, Jimin Huang, Guojun Xiong, Sophia Ananiadou
Comment Yueru He, Xueqing Peng: These two authors contributed equally to this work
Categories Computer Vision and Pattern Recognition
2
Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment NeurIPS 2025 Workshop
Title CN 图片中的偏见：使用社会线索新闻图像和LLM作为评判的VLM基准测试
Keywords 视觉语言模型 · 社会偏见 · 公平性评估 · LLM-as-Judge · 基准测试
Summary 本文提出一个包含1343个新闻图像-问题对的基准数据集，用于评估VLM在年龄、性别、种族、职业等社会属性上的偏见表现，并利用LLM作为自动评判器辅以人工验证。结果显示视觉上下文会系统性影响模型输出，且不同模型和属性间的偏见程度差异显著，同时发现更高的回答忠实性并不意味着更低的偏见。
Reason 该论文聚焦于评估视觉语言模型在包含社会属性（如性别、种族、职业）的新闻图像中的社会偏见问题，属于AI伦理与公平性评估范畴。其核心贡献是构建了一个用于检测VLM社会偏见的基准数据集，并采用LLM-as-Judge进行自动化评估。尽管涉及VLM，但研究目标与文档图像理解（DIU）的核心任务（如文本识别、布局分析、表格理解、信息抽取）无直接关联，也不提供可迁移至DIU的技术工具（如视觉压缩、高分辨率处理、去幻觉机制等）。属于平行应用性质的评测工作，未触及文档理解的方法论创新。
Abstract Large vision-language models (VLMs) can jointly interpret images and text, but they are also prone to absorbing and reproducing harmful social stereotypes when visual cues such as age, gender, race, clothing, or occupation are present. To investigate these risks, we introduce a news-image benchmark consisting of 1,343 image-question pairs drawn from diverse outlets, which we annotated with ground-truth answers and demographic attributes (age, gender, race, occupation, and sports). We evaluate a range of state-of-the-art VLMs and employ a large language model (LLM) as judge, with human verification. Our findings show that: (i) visual context systematically shifts model outputs in open-ended settings; (ii) bias prevalence varies across attributes and models, with particularly high risk for gender and occupation; and (iii) higher faithfulness does not necessarily correspond to lower bias. We release the benchmark prompts, evaluation rubric, and code to support reproducible and fairness-aware multimodal assessment.
Authors Aravind Narayanan, Vahid Reza Khazaie, Shaina Raza
Comment Accepted to NeurIPS 2025 Workshop (Evaluating the Evolving LLM Lifecycle)
Categories Computer Vision and Pattern Recognition
2
Don't Learn, Ground: A Case for Natural Language Inference with Visual Grounding
Title CN 不要学习，要定位：一种基于视觉定位的自然语言推理方法
Keywords 自然语言推理 · 视觉定位 · 多模态表示 · 零样本学习 · 文生图
Summary 本文提出一种零样本的自然语言推理方法，通过文本到图像模型生成前提的视觉表示，并将其与假设进行比对完成推理。无需微调即可实现高准确率，且对文本偏见具有鲁棒性。作者还构建了对抗性数据集验证方法稳健性。然而，该方法依赖于生成图像作为语义媒介，在真实文档理解任务中不具备可扩展性或实用性。
Reason 该论文属于纯NLP与多模态结合的通用语言理解工作，其核心是利用文生图模型将前提语句可视化，并通过视觉-文本对比进行自然语言推理。虽然涉及视觉模态，但其任务设定（NLI）和方法设计均非面向文档图像理解（DIU）中的实际问题，如OCR幻觉、复杂版面分析或表格识别等。所用技术路线（如用生成图像作为语义表示）在文档场景中难以迁移，且存在严重现实性缺陷（例如，文档内容不可生成为图像）。属于典型的‘平行应用’类研究，未提供可用于DIU的方法论创新。
Abstract We propose a zero-shot method for Natural Language Inference (NLI) that leverages multimodal representations by grounding language in visual contexts. Our approach generates visual representations of premises using text-to-image models and performs inference by comparing these representations with textual hypotheses. We evaluate two inference techniques: cosine similarity and visual question answering. Our method achieves high accuracy without task-specific fine-tuning, demonstrating robustness against textual biases and surface heuristics. Additionally, we design a controlled adversarial dataset to validate the robustness of our approach. Our findings suggest that leveraging visual modality as a meaning representation provides a promising direction for robust natural language understanding.
Authors Daniil Ignatev, Ayman Santeer, Albert Gatt, Denis Paperno
Categories Computation and Language
2
AutoLink: Autonomous Schema Exploration and Expansion for Scalable Schema Linking in Text-to-SQL at Scale
Title CN AutoLink：面向大规模文本到SQL的自主模式探索与扩展以实现可扩展的模式链接
Keywords Text-to-SQL · Schema Linking · Agent Framework · Large Language Models · Scalability
Summary 本文提出AutoLink，一种基于LLM的自主代理框架，用于在大规模文本到SQL任务中进行高效模式链接。通过动态迭代地探索和扩展相关数据库模式子集，避免输入完整模式，从而在高列数场景下保持高召回率和执行准确率，解决了传统方法在可扩展性上的瓶颈。
Reason 该论文属于典型的‘平行下游应用’，其核心是将Agent框架应用于Text-to-SQL中的模式链接任务，尽管提出了迭代式自主探索机制，但整体工作局限于数据库语义解析场景，并未提出可用于文档图像理解（DIU）的通用方法论。虽然其Agent流程设计具有一定系统性，但无法直接迁移解决DIU中的OCR幻觉、复杂排版或长文档推理等核心问题，属于特定领域的工程优化，不符合本研究方向对底层技术创新的要求。
Abstract For industrial-scale text-to-SQL, supplying the entire database schema to Large Language Models (LLMs) is impractical due to context window limits and irrelevant noise. Schema linking, which filters the schema to a relevant subset, is therefore critical. However, existing methods incur prohibitive costs, struggle to trade off recall and noise, and scale poorly to large databases. We present \textbf{AutoLink}, an autonomous agent framework that reformulates schema linking as an iterative, agent-driven process. Guided by an LLM, AutoLink dynamically explores and expands the linked schema subset, progressively identifying necessary schema components without inputting the full database schema. Our experiments demonstrate AutoLink's superior performance, achieving state-of-the-art strict schema linking recall of \textbf{97.4\%} on Bird-Dev and \textbf{91.2\%} on Spider-2.0-Lite, with competitive execution accuracy, i.e., \textbf{68.7\%} EX on Bird-Dev (better than CHESS) and \textbf{34.9\%} EX on Spider-2.0-Lite (ranking 2nd on the official leaderboard). Crucially, AutoLink exhibits \textbf{exceptional scalability}, \textbf{maintaining high recall}, \textbf{efficient token consumption}, and \textbf{robust execution accuracy} on large schemas (e.g., over 3,000 columns) where existing methods severely degrade-making it a highly scalable, high-recall schema-linking solution for industrial text-to-SQL systems.
Authors Ziyang Wang, Yuanlei Zheng, Zhenbiao Cao, Xiaojin Zhang, Zhongyu Wei, Pei Fu, Zhenbo Luo, Wei Chen, Xiang Bai
Categories Computation and Language
2
Overcoming the Generalization Limits of SLM Finetuning for Shape-Based Extraction of Datatype and Object Properties KCAP 2025
Title CN 克服SLM微调在基于形状的数据类型和对象属性提取中的泛化限制
Keywords 小语言模型 · 关系抽取 · SHACL · 数据增强 · 长尾分布
Summary 本文研究小语言模型在从文本中提取RDF图时对数据类型和对象属性的泛化能力，发现长尾分布的稀有属性是主要瓶颈，并通过分层采样、加权损失、数据扩增等策略进行优化。最终提出训练集中每种属性出现次数应超过阈值以实现均衡性能。
Reason 该论文聚焦于小语言模型（SLM）在RDF三元组抽取中对SHACL形状的利用，属于典型的NLP下游任务应用，且集中在知识图谱构建场景。虽然涉及结构化信息提取，但其方法完全基于文本模态，未引入视觉或文档布局信息处理机制，与文档图像理解（DIU）的核心问题如OCR、排版解析、视觉-文本对齐等无直接关联。属于‘平行应用’类研究，不具备向DIU迁移的技术创新价值。
Abstract Small language models (SLMs) have shown promises for relation extraction (RE) when extracting RDF triples guided by SHACL shapes focused on common datatype properties. This paper investigates how SLMs handle both datatype and object properties for a complete RDF graph extraction. We show that the key bottleneck is related to long-tail distribution of rare properties. To solve this issue, we evaluate several strategies: stratified sampling, weighted loss, dataset scaling, and template-based synthetic data augmentation. We show that the best strategy to perform equally well over unbalanced target properties is to build a training set where the number of occurrences of each property exceeds a given threshold. To enable reproducibility, we publicly released our datasets, experimental results and code. Our findings offer practical guidance for training shape-aware SLMs and highlight promising directions for future work in semantic RE.
Authors Célian Ringwald, Fabien Gandon, Catherine Faron, Franck Michel, Hanna Abi Akl
Comment Accepted at KCAP 2025
Categories Computation and Language
2
Principled Design of Interpretable Automated Scoring for Large-Scale Educational Assessments
Title CN 可解释的大规模教育评估自动评分的系统化设计
Keywords 自动评分 · 可解释性 · 教育AI · LLM应用 · 文本评估
Summary 本文提出FGTI四项可解释性原则，并构建AnalyticScore框架，通过LLM提取回答特征后使用序数逻辑回归进行打分，在ASAP-SAS数据集上接近SOTA水平且具备良好可解释性。但整体工作局限于教育评估中的文本自动评分，不涉及文档图像或多模态理解。
Reason 该论文聚焦于教育领域的自动评分系统，提出了一套可解释性框架AnalyticScore，用于短答案评分。尽管使用了LLM进行特征提取并强调可解释性（如Faithfulness、Groundedness等），但其本质是将现有NLP技术应用于教育评估这一特定下游场景，属于典型的‘平行应用’。未对文档图像理解中的核心问题（如OCR幻觉、复杂版式处理、视觉-文本对齐）提出任何新的方法论或架构改进，无法迁移为DIU的技术工具。
Abstract AI-driven automated scoring systems offer scalable and efficient means of evaluating complex student-generated responses. Yet, despite increasing demand for transparency and interpretability, the field has yet to develop a widely accepted solution for interpretable automated scoring to be used in large-scale real-world assessments. This work takes a principled approach to address this challenge. We analyze the needs and potential benefits of interpretable automated scoring for various assessment stakeholders and develop four principles of interpretability -- Faithfulness, Groundedness, Traceability, and Interchangeability (FGTI) -- targeted at those needs. To illustrate the feasibility of implementing these principles, we develop the AnalyticScore framework for short answer scoring as a baseline reference framework for future research. AnalyticScore operates by (1) extracting explicitly identifiable elements of the responses, (2) featurizing each response into human-interpretable values using LLMs, and (3) applying an intuitive ordinal logistic regression model for scoring. In terms of scoring accuracy, AnalyticScore outperforms many uninterpretable scoring methods, and is within only 0.06 QWK of the uninterpretable SOTA on average across 10 items from the ASAP-SAS dataset. By comparing against human annotators conducting the same featurization task, we further demonstrate that the featurization behavior of AnalyticScore aligns well with that of humans.
Authors Yunsung Kim, Mike Hardy, Joseph Tey, Candace Thille, Chris Piech
Comment 16 pages, 2 figures
Categories Computation and Language
2
OmniScientist: Toward a Co-evolving Ecosystem of Human and AI Scientists
Title CN OmniScientist：迈向人类与AI科学家协同演化的生态系统
Keywords AI科学家 · 多智能体系统 · 科学自动化 · 协作协议 · 科研生态
Summary 本文提出了OmniScientist，一个模拟人类科研生态的AI科学家框架，支持从文献回顾、假设生成、实验设计到论文写作与同行评审的端到端自动化。系统包含结构化知识体系、多智能体协作协议（OSP）和基于盲审投票的开放评估平台ScienceArena，旨在实现AI与人类科学家的协同演化。然而，该工作属于特定领域（科学发现）的智能体应用，缺乏对文档理解任务的方法论贡献。
Reason 该论文属于典型的‘平行应用’类工作，将多智能体系统应用于科学发现流程自动化，构建了一个模拟人类科研生态的AI框架。尽管其在协作协议和评估机制上有一定设计，但核心是将现有LLM Agent技术套用于科学研究场景，未提出可迁移至文档图像理解（DIU）领域的底层方法创新。任务聚焦于科学流程管理与协作机制，与DocVQA、OCR、版面分析等DIU核心问题无直接关联，也无法解决OCR幻觉、密集文本识别或复杂排版建模等痛点。
Abstract With the rapid development of Large Language Models (LLMs), AI agents have demonstrated increasing proficiency in scientific tasks, ranging from hypothesis generation and experimental design to manuscript writing. Such agent systems are commonly referred to as "AI Scientists." However, existing AI Scientists predominantly formulate scientific discovery as a standalone search or optimization problem, overlooking the fact that scientific research is inherently a social and collaborative endeavor. Real-world science relies on a complex scientific infrastructure composed of collaborative mechanisms, contribution attribution, peer review, and structured scientific knowledge networks. Due to the lack of modeling for these critical dimensions, current systems struggle to establish a genuine research ecosystem or interact deeply with the human scientific community. To bridge this gap, we introduce OmniScientist, a framework that explicitly encodes the underlying mechanisms of human research into the AI scientific workflow. OmniScientist not only achieves end-to-end automation across data foundation, literature review, research ideation, experiment automation, scientific writing, and peer review, but also provides comprehensive infrastructural support by simulating the human scientific system, comprising: (1) a structured knowledge system built upon citation networks and conceptual correlations; (2) a collaborative research protocol (OSP), which enables seamless multi-agent collaboration and human researcher participation; and (3) an open evaluation platform (ScienceArena) based on blind pairwise user voting and Elo rankings. This infrastructure empowers agents to not only comprehend and leverage human knowledge systems but also to collaborate and co-evolve, fostering a sustainable and scalable innovation ecosystem.
Authors Chenyang Shao, Dehao Huang, Yu Li, Keyu Zhao, Weiquan Lin, Yining Zhang, Qingbin Zeng, Zhiyu Chen, Tianxing Li, Yifei Huang, Taozhong Wu, Xinyang Liu, Ruotong Zhao, Mengsheng Zhao, Xuhua Zhang, Yue Wang, Yuanyi Zhen, Fengli Xu, Yong Li, Tie-Yan Liu
Categories Computation and Language
2
Predicting the Formation of Induction Heads CogInterp @ NeurIPS
Title CN 预测诱导头的形成
Keywords 诱导头 · 上下文学习 · 语言模型机理 · 注意力机制 · 训练动态
Summary 本文研究现代语言模型中被称为‘诱导头’的注意力头的形成机制，探讨训练数据的统计特性（如二元组重复频率和可靠性）如何影响其出现，并提出一个基于批量大小和上下文长度的简单方程来预测其形成时机。研究发现局部依赖性在高频高可靠条件下足以触发诱导头形成，而在低频低可靠时则受类别结构和分布形状影响。
Reason 该论文研究语言模型中‘诱导头’（Induction Heads）的形成机制，属于纯语言模型内部机理分析，聚焦于通用语言模型的上下文学习能力形成条件。尽管ICL机制可能间接影响多模态推理，但本文未涉及视觉输入、文档结构或OCR相关问题，与文档图像理解（DIU）无直接关联。其方法无法迁移用于解决OCR幻觉、复杂排版或细粒度定位等核心DIU痛点。
Abstract Arguably, specialized attention heads dubbed induction heads (IHs) underlie the remarkable in-context learning (ICL) capabilities of modern language models (LMs); yet, a precise characterization of their formation remains unclear. In this study, we investigate the relationship between statistical properties of training data (for both natural and synthetic data) and IH formation. We show that (1) a simple equation combining batch size and context size predicts the point at which IHs form; (2) surface bigram repetition frequency and reliability strongly affect the formation of IHs, and we find a precise Pareto frontier in terms of these two values; and (3) local dependency with high bigram repetition frequency and reliability is sufficient for IH formation, but when the frequency and reliability are low, categoriality and the shape of the marginal distribution matter.
Authors Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan Schneider
Comment Accepted to CogInterp @ NeurIPS
Categories Computation and Language
2
EventWeave: A Dynamic Framework for Capturing Core and Supporting Events in Dialogue Systems
Title CN EventWeave：一种用于捕捉对话系统中核心与支持事件的动态框架
Keywords 对话系统 · 事件建模 · 动态图 · 多头注意力 · 上下文理解
Summary 本文提出EventWeave，一个通过构建动态事件图来建模对话中核心事件与支持事件关系的框架，利用多头注意力机制选择性地关注当前轮次相关的事件，提升对话响应的连贯性和自然性。实验表明该方法在多个对话数据集上优于基线模型。
Reason 该论文聚焦于对话系统中的事件结构建模，属于纯自然语言处理领域的对话建模工作，未涉及任何视觉或文档模态。其提出的事件图构建和注意力机制虽在对话上下文建模上有创新，但完全不涉及文档图像理解（DIU）的核心问题，如OCR幻觉、布局分析、表格识别或多模态对齐等。属于典型的‘平行应用’类NLP方法，且应用场景局限于对话生成，无法迁移至文档视觉问答或复杂排版理解任务。
Abstract Large language models have improved dialogue systems, but often process conversational turns in isolation, overlooking the event structures that guide natural interactions. Hence we introduce \textbf{EventWeave}, a framework that explicitly models relationships between conversational events to generate more contextually appropriate dialogue responses. EventWeave constructs a dynamic event graph that distinguishes between core events (main goals) and supporting events (interconnected details), employing a multi-head attention mechanism to selectively determine which events are most relevant to the current turn. Unlike summarization or standard graph-based approaches, our method captures three distinct relationship types between events, allowing for more nuanced context modeling. Experiments on three dialogue datasets demonstrate that EventWeave produces more natural and contextually appropriate responses while requiring less computational overhead than models processing the entire dialogue history. Ablation studies confirm improvements stem from better event relationship modeling rather than increased information density. Our approach effectively balances comprehensive context understanding with generating concise responses, maintaining strong performance across various dialogue lengths through targeted optimization techniques.
Authors Zhengyi Zhao, Shubo Zhang, Yiming Du, Bin Liang, Baojun Wang, Zhongyang Li, Binyang Li, Kam-Fai Wong
Categories Computation and Language
2
RPRO: Ranked Preference Reinforcement Optimization for Enhancing Medical QA and Diagnostic Reasoning
Title CN RPRO：用于增强医学问答与诊断推理的排序偏好强化优化
Keywords 医学问答 · 推理优化 · 强化学习 · 偏好学习 · 链式思维
Summary 本文提出RPRO框架，通过结合强化学习与基于偏好的推理优化来提升医学问答中的推理质量。该方法采用任务自适应的推理模板和概率化评估机制，在不依赖人工标注的情况下自动识别并修正低质量推理链。基于Bradley-Terry模型的组级排序优化和KL散度正则化提升了训练稳定性。实验表明，即使参数量较小（2B），该模型在多个医学QA数据集上仍优于更大规模的模型。
Reason 该论文聚焦于医学领域的问答与诊断推理，属于典型的垂直领域下游应用（Wrapper/Application），其方法虽涉及偏好优化与推理链改进，但核心场景为纯文本医疗问答，并未处理文档图像、多模态输入或OCR相关问题。尽管其推理优化机制具有一定通用性，但未涉及视觉模态、布局理解或文档结构信息，无法直接迁移解决文档图像理解中的核心痛点如OCR幻觉、复杂排版或细粒度定位。因此属于平行应用，缺乏对DIU的方法论贡献。
Abstract Medical question answering requires advanced reasoning that integrates domain knowledge with logical inference. However, existing large language models (LLMs) often generate reasoning chains that lack factual accuracy and clinical reliability. We propose Ranked Preference Reinforcement Optimization (RPRO), a novel framework that combines reinforcement learning with preference-driven reasoning refinement to enhance clinical chain-of-thought (CoT) performance. RPRO distinguishes itself from prior approaches by employing task-adaptive reasoning templates and a probabilistic evaluation mechanism that aligns model outputs with established clinical workflows, while automatically identifying and correcting low-quality reasoning chains. Unlike traditional pairwise preference methods, RPRO introduces a groupwise ranking optimization based on the Bradley--Terry model and incorporates KL-divergence regularization for stable training. Experiments on PubMedQA, MedQA-USMLE, and a real-world clinical dataset from Far Eastern Memorial Hospital (FEMH) demonstrate consistent improvements over strong baselines. Remarkably, our 2B-parameter model outperforms much larger 7B--20B models, including medical-specialized variants. These findings demonstrate that combining preference optimization with quality-driven refinement provides a scalable and clinically grounded approach to building more reliable medical LLMs.
Authors Chia-Hsuan Hsu, Jun-En Ding, Hsin-Ling Hsu, Chih-Ho Hsu, Li-Hung Yao, Chun-Chieh Liao, Feng Liu, Fang-Ming Hung
Categories Computation and Language
1
CNS-Obsidian: A Neurosurgical Vision-Language Model Built From Scientific Publications
Title CN CNS-Obsidian：一种基于科学出版物构建的神经外科视觉-语言模型
Keywords VLM · 医学AI · 领域专用模型 · 神经外科 · 微调
Summary 本文提出了CNS-Obsidian，一个基于2.4万篇神经外科期刊文章中的图表和图注构建的视觉-语言模型，用于辅助神经外科诊断。模型基于LLaVA-Next进行微调，在合成问题上表现接近GPT-4o，但在真实临床咨询中表现略逊。研究展示了领域特定VLM在专业场景中的可行性，但未引入新的底层技术。
Reason 该论文属于典型的‘平行下游应用’，即将现有VLM技术（LLaVA-Next）应用于神经外科领域的文献图像理解任务。尽管其训练数据来自同行评审的科学出版物，并在临床场景中进行了评估，但其方法论并无创新，未提出可用于解决文档图像理解核心痛点（如OCR幻觉、复杂排版、细粒度定位等）的新架构、新训练策略或新推理机制。完全属于垂直领域应用，不符合本研究关注的技术源头创新要求。
Abstract General-purpose VLMs demonstrate impressive capabilities, but their opaque training on uncurated internet data poses critical limitations for high-stakes decision-making, such as in neurosurgery. We present CNS-Obsidian, a neurosurgical VLM trained on peer-reviewed literature, and demonstrate its clinical utility versus GPT-4o in a real-world setting. We compiled 23,984 articles from Neurosurgery Publications journals, yielding 78,853 figures and captions. Using GPT-4o and Claude Sonnet-3.5, we converted these into 263,064 training samples across three formats: instruction fine-tuning, multiple-choice questions, and differential diagnosis. We trained CNS-Obsidian, a fine-tune of the 34-billion parameter LLaVA-Next model. In a blinded, randomized trial at NYU Langone Health (Aug 30-Nov 30, 2024), neurosurgery consultations were assigned to either CNS-Obsidian or a HIPAA-compliant GPT-4o endpoint as diagnostic co-pilot after consultations. Primary outcomes were diagnostic helpfulness and accuracy, assessed via user ratings and presence of correct diagnosis within the VLM-provided differential. CNS-Obsidian matched GPT-4o on synthetic questions (76.13% vs 77.54%, p=0.235), but only achieved 46.81% accuracy on human-generated questions versus GPT-4o's 65.70% (p<10-15). In the randomized trial, 70 consultations were evaluated (32 CNS-Obsidian, 38 GPT-4o) from 959 total consults (7.3% utilization). CNS-Obsidian received positive ratings in 40.62% of cases versus 57.89% for GPT-4o (p=0.230). Both models included correct diagnosis in approximately 60% of cases (59.38% vs 65.79%, p=0.626). Domain-specific VLMs trained on curated scientific literature can approach frontier model performance despite being orders of magnitude smaller and less expensive to train. This establishes a transparent framework for scientific communities to build specialized AI models.
Authors Anton Alyakin, Jaden Stryker, Daniel Alexander Alber, Jin Vivian Lee, Karl L. Sangwon, Brandon Duderstadt, Akshay Save, David Kurland, Spencer Frome, Shrutika Singh, Jeff Zhang, Eunice Yang, Ki Yun Park, Cordelia Orillac, Aly A. Valliani, Sean Neifert, Albert Liu, Aneek Patel, Christopher Livia, Darryl Lau, Ilya Laufer, Peter A. Rozman, Eveline Teresa Hidalgo, Howard Riina, Rui Feng, Todd Hollon, Yindalon Aphinyanaphongs, John G. Golfinos, Laura Snyder, Eric Leuthardt, Douglas Kondziolka, Eric Karl Oermann
Categories Artificial Intelligence, Computation and Language
1
Conversational LLMs Simplify Secure Clinical Data Access, Understanding, and Analysis
Title CN 对话式大语言模型简化安全的临床数据访问、理解与分析
Keywords 临床数据 · 自然语言查询 · SQL生成 · 大语言模型 · 隐私保护
Summary 本文提出了M3系统，通过自然语言接口降低研究人员使用MIMIC-IV电子病历数据库的技术门槛。用户可用英语提问，系统自动将其转化为SQL查询并在本地或云端执行，返回结果及对应查询语句。评估显示Claude Sonnet和开源gpt-oss-20B模型在EHRSQL 2024基准上分别达到94%和93%的准确率。系统支持本地部署以保障隐私，并具备认证、验证和审计功能。
Reason 该论文属于典型的‘平行下游应用’，即将现有LLM技术应用于特定领域（医疗数据库MIMIC-IV）的自然语言查询接口构建。其核心是SQL生成与数据库交互系统（M3），并未提出任何文档图像理解（DIU）相关的技术改进或可迁移的方法论创新。尽管涉及数据理解和结构化输出，但完全不涉及OCR、视觉布局分析、多模态对齐或文档视觉问答等DIU核心问题。
Abstract Large-scale clinical databases offer opportunities for medical research, but their complexity creates barriers to effective use. The Medical Information Mart for Intensive Care (MIMIC-IV), one of the world's largest open-source electronic health record databases, traditionally requires both SQL proficiency and clinical domain expertise. We introduce M3, a system that enables natural language querying of MIMIC-IV data through the Model Context Protocol. With a single command, M3 retrieves MIMIC-IV from PhysioNet, launches a local SQLite instance or connects to hosted BigQuery, and allows researchers to pose clinical questions in plain English. We evaluated M3 using one hundred questions from the EHRSQL 2024 benchmark with two language models: the proprietary Claude Sonnet 4 achieved 94% accuracy, while the open-source gpt-oss-20B (deployable locally on consumer hardware) achieved 93% accuracy. Both models translate natural language into SQL, execute queries against MIMIC-IV, and return structured results alongside the underlying query for verification. Error analysis revealed that most failures stemmed from complex temporal reasoning or ambiguous question phrasing rather than fundamental architectural limitations. The comparable performance of a smaller open-source model demonstrates that privacy-preserving local deployment is viable for sensitive clinical data analysis. M3 lowers technical barriers to critical care data analysis while maintaining security through OAuth2 authentication, query validation, and comprehensive audit logging.
Authors Rafi Al Attrach, Pedro Moreira, Rajna Fani, Renato Umeton, Amelia Fiske, Leo Anthony Celi
Comment 16 pages, 4 figures
Categories Artificial Intelligence
1
LLMs-Powered Accurate Extraction, Querying and Intelligent Management of Literature derived 2D Materials Data
Title CN 由大语言模型驱动的文献中二维材料数据的精确提取、查询与智能管理
Keywords 大语言模型 · 信息提取 · 科学文献 · 知识管理 · 材料科学
Summary 本文提出了一种利用大语言模型从二维材料相关研究论文中自动提取和管理材料属性及制备方法的技术框架。系统通过LLM实现信息抽取、问答和数据库构建，旨在加速材料科学领域的知识发现。然而，该工作聚焦于特定领域的内容处理，依赖现有语言模型能力，缺乏对文档图像本身的理解机制设计。
Reason 该论文属于典型的‘平行下游应用’，即将现有LLM技术应用于特定科学文献（二维材料）中的信息提取任务。其核心是将已有NLP或VLM方法用于材料科学领域的知识结构化，未提出任何针对文档图像理解（DIU）本身的底层方法创新，如视觉表示学习、OCR-free架构、视觉压缩或排版建模等。尽管涉及‘文献数据提取’，但并未解决文档图像中的复杂布局、密集文本、多模态对齐或幻觉抑制等DIU核心问题。
Abstract Two-dimensional (2D) materials have showed widespread applications in energy storage and conversion owning to their unique physicochemical, and electronic properties. Most of the valuable information for the materials, such as their properties and preparation methods, is included in the published research papers. However, due to the dispersion of synthe
Authors Lijun Shang, Yadong Yu, Wenqiang Kang, Jian Zhou, Dongyue Gao, Pan Xiang, Zhe Liu, Mengyan Dai, Zhonglu Guo, Zhimei Sun
Comment 100 pages (18 pages main text, 82 pages supplementary material), 5 figures. Supplementary material starts from page 19
Categories Computation and Language
0
Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacy
Title CN 大型语言模型中的模拟自我评估：一种人工智能自我效能感的心理测量方法
Keywords 大语言模型 · 自我评估 · 心理测量 · 自我效能感 · 提示工程
Summary 该研究将人类心理学中的自我效能感量表（GSES）迁移到大语言模型上，试图通过提示工程让模型进行自我评估。结果显示不同模型在不同任务下的自我评分稳定但与实际表现不一致，存在高估倾向，且评分更多反映的是表达风格而非真实能力。该工作本质上是关于LLM行为分析的心理学模拟，不涉及视觉或多模态内容。
Reason 该论文研究的是大型语言模型的自我效能感模拟，属于纯NLP范畴，关注的是模型对自身能力的‘心理’评估行为，并未涉及任何视觉输入、多模态理解或文档结构分析。其核心问题（自我评估与实际性能脱节）虽在广义上与‘幻觉’相关，但缺乏针对文档图像理解中OCR错误、布局混淆或细粒度定位等具体痛点的解决方案或可迁移机制。属于无关领域的平行心理化研究，无法为DIU提供方法论工具。
Abstract Self-assessment is a key aspect of reliable intelligence, yet evaluations of large language models (LLMs) focus mainly on task accuracy. We adapted the 10-item General Self-Efficacy Scale (GSES) to elicit simulated self-assessments from ten LLMs across four conditions: no task, computational reasoning, social reasoning, and summarization. GSES responses were highly stable across repeated administrations and randomized item orders. However, models showed significantly different self-efficacy levels across conditions, with aggregate scores lower than human norms. All models achieved perfect accuracy on computational and social questions, whereas summarization performance varied widely. Self-assessment did not reliably reflect ability: several low-scoring models performed accurately, while some high-scoring models produced weaker summaries. Follow-up confidence prompts yielded modest, mostly downward revisions, suggesting mild overestimation in first-pass assessments. Qualitative analysis showed that higher self-efficacy corresponded to more assertive, anthropomorphic reasoning styles, whereas lower scores reflected cautious, de-anthropomorphized explanations. Psychometric prompting provides structured insight into LLM communication behavior but not calibrated performance estimates.
Authors Daniel I Jackson, Emma L Jensen, Syed-Amad Hussain, Emre Sezgin
Comment 25 pages,5 tables, 3 figures
Categories Artificial Intelligence
0
Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision-Language-Action Models
Title CN 面向视觉-语言-动作模型的注意力引导块状稀疏对抗攻击
Keywords 对抗攻击 · 视觉-语言-动作模型 · 注意力机制 · 稀疏扰动 · 特征空间攻击
Summary 本文提出ADVLA，一种针对视觉-语言-动作模型的对抗攻击框架，通过在视觉编码器映射到文本特征空间的表示上施加稀疏扰动，并结合注意力引导机制，实现高效、隐蔽且低成本的攻击。实验显示其在低幅度约束下能以极少的图像块修改达到接近100%的攻击成功率，适用于具身智能场景下的安全性评估。
Reason 该论文研究的是具身智能中的视觉-语言-动作（VLA）模型的对抗攻击方法，属于机器人或具身智能领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的注意力引导稀疏扰动技术具有一定方法新颖性，但其目标是干扰动作预测而非解决文档理解中的OCR幻觉、密集文本处理或复杂排版解析等问题，迁移价值极低。
Abstract In recent years, Vision-Language-Action (VLA) models in embodied intelligence have developed rapidly. However, existing adversarial attack methods require costly end-to-end training and often generate noticeable perturbation patches. To address these limitations, we propose ADVLA, a framework that directly applies adversarial perturbations on features projected from the visual encoder into the textual feature space. ADVLA efficiently disrupts downstream action predictions under low-amplitude constraints, and attention guidance allows the perturbations to be both focused and sparse. We introduce three strategies that enhance sensitivity, enforce sparsity, and concentrate perturbations. Experiments demonstrate that under an $L_{\infty}=4/255$ constraint, ADVLA combined with Top-K masking modifies less than 10% of the patches while achieving an attack success rate of nearly 100%. The perturbations are concentrated on critical regions, remain almost imperceptible in the overall image, and a single-step iteration takes only about 0.06 seconds, significantly outperforming conventional patch-based attacks. In summary, ADVLA effectively weakens downstream action predictions of VLA models under low-amplitude and locally sparse conditions, avoiding the high training costs and conspicuous perturbations of traditional patch attacks, and demonstrates unique effectiveness and practical value for attacking VLA feature spaces.
Authors Naifu Zhang, Wei Tao, Xi Xiao, Qianpu Sun, Yuxin Zheng, Wentao Mo, Peiqiang Wang, Nan Zhang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Natural Strategic Ability in Stochastic Multi-Agent Systems AAAI 2024
Title CN 随机多智能体系统中的自然策略能力
Keywords 多智能体系统 · 自然策略 · 概率时序逻辑 · 模型检测 · 计算复杂性
Summary 本文首次将自然策略框架引入随机多智能体系统，提出了在概率环境下的PATL和PATL*逻辑（NatPATL/NatPATL*），并分析了其模型检测的计算复杂性。结果显示，在特定限制下NatPATL的模型检测为NP完全问题，而更一般的逻辑具有更高的复杂度（如EXPSPACE或3EXPSPACE）。这项工作主要贡献在于理论层面的形式化验证能力扩展。
Reason 该论文研究的是多智能体系统中的形式化策略与模型检测复杂性，属于理论计算机科学与形式化方法领域，聚焦于概率时序逻辑的复杂性分析。其内容与文档图像理解（DIU）、视觉语言模型、OCR、布局分析等核心任务完全无关，无法迁移用于解决OCR幻觉、密集文本识别、复杂排版或长文档推理等实际问题。
Abstract Strategies synthesized using formal methods can be complex and often require infinite memory, which does not correspond to the expected behavior when trying to model Multi-Agent Systems (MAS). To capture such behaviors, natural strategies are a recently proposed framework striking a balance between the ability of agents to strategize with memory and the model-checking complexity, but until now has been restricted to fully deterministic settings. For the first time, we consider the probabilistic temporal logics PATL and PATL* under natural strategies (NatPATL and NatPATL*, resp.). As main result we show that, in stochastic MAS, NatPATL model-checking is NP-complete when the active coalition is restricted to deterministic strategies. We also give a 2NEXPTIME complexity result for NatPATL* with the same restriction. In the unrestricted case, we give an EXPSPACE complexity for NatPATL and 3EXPSPACE complexity for NatPATL*.
Authors Raphaël Berthon, Joost-Pieter Katoen, Munyque Mittelmann, Aniello Murano
Comment Extended version of the paper accepted at AAAI 2024
Categories Artificial Intelligence
0
Bridging the Unavoidable A Priori: A Framework for Comparative Causal Modeling 43rd Conference of the International System Dynamics Society
Title CN 弥合不可避免的先验：一种用于比较因果建模的框架
Keywords 因果建模 · 系统动力学 · 结构方程模型 · 负责任AI · 先验假设
Summary 本文提出一个将系统动力学与结构方程模型统一的数学框架，旨在解决不同因果建模范式之间的假设冲突，促进数据科学和AI/ML中对系统动态的认识论发展。研究重点在于理论整合而非具体AI模型构建或视觉内容理解。
Reason 该论文属于系统动力学与结构方程建模的理论融合工作，聚焦于因果推理的哲学基础与方法论整合，未涉及任何多模态表示、视觉输入处理或文档理解任务。其方法难以迁移到文档图像理解中的OCR幻觉、细粒度定位或高分辨率处理等核心痛点，且不属于VLM或视觉语言模型架构改进范畴。
Abstract AI/ML models have rapidly gained prominence as innovations for solving previously unsolved problems and their unintended consequences from amplifying human biases. Advocates for responsible AI/ML have sought ways to draw on the richer causal models of system dynamics to better inform the development of responsible AI/ML. However, a major barrier to advancing this work is the difficulty of bringing together methods rooted in different underlying assumptions (i.e., Dana Meadow's "the unavoidable a priori"). This paper brings system dynamics and structural equation modeling together into a common mathematical framework that can be used to generate systems from distributions, develop methods, and compare results to inform the underlying epistemology of system dynamics for data science and AI/ML applications.
Authors Peter S. Hovmand, Kari O'Donnell, Callie Ogland-Hand, Brian Biroscak, Douglas D. Gunzler
Comment Presented at 43rd Conference of the International System Dynamics Society in Boston, United States
Categories Artificial Intelligence
0
Diffusion Models at the Drug Discovery Frontier: A Review on Generating Small Molecules versus Therapeutic Peptides Biology
Title CN 药物发现前沿的扩散模型：小分子与治疗性肽生成的综述
Keywords 扩散模型 · 药物发现 · 小分子生成 · 肽设计 · 生成模型
Summary 本文是一篇关于扩散模型在药物发现中应用的综述，重点比较了其在小分子和治疗性肽生成中的异同，讨论了化学空间、表示方法、设计目标以及实验验证等方面的挑战，并展望了与闭环DBTL平台集成的前景。
Reason 该论文属于药物发现领域的综述，研究内容为扩散模型在小分子和治疗性肽设计中的应用，完全偏离文档图像理解（DIU）和多模态大模型（VLM）用于文档理解的核心问题。不涉及OCR、布局分析、表格识别、视觉语言建模或任何DIU相关任务，属于典型无关领域。
Abstract Diffusion models have emerged as a leading framework in generative modeling, poised to transform the traditionally slow and costly process of drug discovery. This review provides a systematic comparison of their application in designing two principal therapeutic modalities: small molecules and therapeutic peptides. We dissect how the unified framework of iterative denoising is adapted to the distinct molecular representations, chemical spaces, and design objectives of each modality. For small molecules, these models excel at structure-based design, generating novel, pocket-fitting ligands with desired physicochemical properties, yet face the critical hurdle of ensuring chemical synthesizability. Conversely, for therapeutic peptides, the focus shifts to generating functional sequences and designing de novo structures, where the primary challenges are achieving biological stability against proteolysis, ensuring proper folding, and minimizing immunogenicity. Despite these distinct challenges, both domains face shared hurdles: the scarcity of high-quality experimental data, the reliance on inaccurate scoring functions for validation, and the crucial need for experimental validation. We conclude that the full potential of diffusion models will be unlocked by bridging these modality-specific gaps and integrating them into automated, closed-loop Design-Build-Test-Learn (DBTL) platforms, thereby shifting the paradigm from mere chemical exploration to the on-demand engineering of novel~therapeutics.
Authors Yiquan Wang, Yahui Ma, Yuhan Chang, Jiayao Yan, Jialin Zhang, Minnuo Cai, Kai Wei
Comment Published in Biology
Categories Artificial Intelligence
0
BengaliFig: A Low-Resource Challenge for Figurative and Culturally Grounded Reasoning in Bengali
Title CN BengaliFig：一种针对孟加拉语中比喻性与文化相关推理的低资源挑战
Keywords 低资源语言 · 文化推理 · 比喻理解 · 数据集 · 大语言模型
Summary 本文提出了BengaliFig，一个包含435个来自孟加拉口语和文学传统的谜题的数据集，用于评估大模型在低资源语言中进行比喻性和文化相关推理的能力。每个样本从五个维度进行了标注，并通过AI辅助流程转换为选择题形式。实验表明主流大模型在此类任务上表现不佳，揭示了当前模型在文化深层理解上的不足。
Reason 该论文聚焦于孟加拉语这一小语种的文化相关推理任务，属于特定语言和文化的NLP评估数据集构建，不涉及文档图像理解、多模态建模、视觉表示或OCR等相关技术。无法迁移至文档图像理解领域解决核心痛点（如排版复杂性、高分辨率处理、幻觉等），且明确违反‘小语种’负面清单。
Abstract Large language models excel on broad multilingual benchmarks but remain to be evaluated extensively in figurative and culturally grounded reasoning, especially in low-resource contexts. We present BengaliFig, a compact yet richly annotated challenge set that targets this gap in Bengali, a widely spoken low-resourced language. The dataset contains 435 unique riddles drawn from Bengali oral and literary traditions. Each item is annotated along five orthogonal dimensions capturing reasoning type, trap type, cultural depth, answer category, and difficulty, and is automatically converted to multiple-choice format through a constraint-aware, AI-assisted pipeline. We evaluate eight frontier LLMs from major providers under zero-shot and few-shot chain-of-thought prompting, revealing consistent weaknesses in metaphorical and culturally specific reasoning. BengaliFig thus contributes both a diagnostic probe for evaluating LLM robustness in low-resource cultural contexts and a step toward inclusive and heritage-aware NLP evaluation.
Authors Abdullah Al Sefat
Categories Artificial Intelligence, Computation and Language
0
Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving NeurIPS 2025
Title CN 用于闭环端到端自动驾驶的基于模型的策略自适应
Keywords 自动驾驶 · 端到端学习 · 策略自适应 · 反事实仿真 · 扩散模型
Summary 本文提出了一种名为Model-based Policy Adaptation（MPA）的框架，用于提升预训练端到端自动驾驶代理在闭环部署中的鲁棒性和安全性。MPA通过几何一致的仿真引擎生成反事实轨迹，利用扩散模型训练策略适配器，并结合多步Q值模型在推理时选择最优轨迹。实验表明该方法在nuScenes数据集上显著提升了驾驶性能。
Reason 该论文属于具身智能与自动驾驶领域，研究的是端到端驾驶模型在闭环环境中的策略优化问题。尽管其方法涉及推理时适应和仿真增强，但应用场景与文档图像理解（DIU）无直接关联，无法迁移解决OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract End-to-end (E2E) autonomous driving models have demonstrated strong performance in open-loop evaluations but often suffer from cascading errors and poor generalization in closed-loop settings. To address this gap, we propose Model-based Policy Adaptation (MPA), a general framework that enhances the robustness and safety of pretrained E2E driving agents during deployment. MPA first generates diverse counterfactual trajectories using a geometry-consistent simulation engine, exposing the agent to scenarios beyond the original dataset. Based on this generated data, MPA trains a diffusion-based policy adapter to refine the base policy's predictions and a multi-step Q value model to evaluate long-term outcomes. At inference time, the adapter proposes multiple trajectory candidates, and the Q value model selects the one with the highest expected utility. Experiments on the nuScenes benchmark using a photorealistic closed-loop simulator demonstrate that MPA significantly improves performance across in-domain, out-of-domain, and safety-critical scenarios. We further investigate how the scale of counterfactual data and inference-time guidance strategies affect overall effectiveness.
Authors Haohong Lin, Yunzhi Zhang, Wenhao Ding, Jiajun Wu, Ding Zhao
Comment Published at NeurIPS 2025: https://openreview.net/forum?id=4OLbpaTKJe
Categories Artificial Intelligence
0
LCB-CV-UNet: Enhanced Detector for High Dynamic Range Radar Signals IEEE IGARSS 2025
Title CN LCB-CV-UNet：用于高动态范围雷达信号的增强检测器
Keywords 雷达信号 · 高动态范围 · 目标检测 · LCB模块 · CV-UNet
Summary 本文提出了一种名为LCB-CV-UNet的网络结构，用于改善高动态范围（HDR）雷达信号下的目标检测性能。通过引入保持相位一致性的Logarithmic Connect Block（LCB）模块和双混合数据集构建方法，在仿真和真实实验中均表现出优于基线模型的检测能力，尤其在城市目标常见的11-13 dB信噪比范围内提升显著。
Reason 该论文研究的是雷达信号处理中的目标检测问题，属于遥感或雷达信号处理领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其方法聚焦于HDR雷达信号的硬件高效模块设计和半合成数据构建，无法迁移解决文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract We propose the LCB-CV-UNet to tackle performance degradation caused by High Dynamic Range (HDR) radar signals. Initially, a hardware-efficient, plug-and-play module named Logarithmic Connect Block (LCB) is proposed as a phase coherence preserving solution to address the inherent challenges in handling HDR features. Then, we propose the Dual Hybrid Dataset Construction method to generate a semi-synthetic dataset, approximating typical HDR signal scenarios with adjustable target distributions. Simulation results show about 1% total detection probability improvement with under 0.9% computational complexity added compared with the baseline. Furthermore, it excels 5% over the baseline at the range in 11-13 dB signal-to-noise ratio typical for urban targets. Finally, the real experiment validates the practicality of our model.
Authors Yanbin Wang, Xingyu Chen, Yumiao Wang, Xiang Wang, Chuanfei Zang, Guolong Cui, Jiahuan Liu
Comment 5 pages, 4 figures. Accepted to IEEE IGARSS 2025
Categories Artificial Intelligence
0
HarmonicAttack: An Adaptive Cross-Domain Audio Watermark Removal
Title CN HarmonicAttack：一种自适应的跨域音频水印去除方法
Keywords 音频水印去除 · 对抗攻击 · 生成模型 · 语音安全 · 深度学习
Summary 本文提出HarmonicAttack，一种高效的AI生成音频水印去除方法，通过双路径卷积自编码器在时域和频域分离水印信号，并采用GAN式训练实现对多种先进水印方案（如AudioSeal、WavMark）的有效移除，且具备良好的跨域泛化能力。然而该工作不涉及任何视觉或文档模态。
Reason 该论文研究的是音频水印去除技术，属于音频安全与生成领域，与文档图像理解（DIU）或视觉-语言模型在文档上的应用完全无关。尽管其方法涉及深度学习和对抗训练，但任务场景、输入模态（音频）和目标问题均无法迁移到解决OCR幻觉、复杂排版理解、细粒度定位或长文档推理等DIU核心痛点。
Abstract The availability of high-quality, AI-generated audio raises security challenges such as misinformation campaigns and voice-cloning fraud. A key defense against the misuse of AI-generated audio is by watermarking it, so that it can be easily distinguished from genuine audio. As those seeking to misuse AI-generated audio may thus seek to remove audio watermarks, studying effective watermark removal techniques is critical to being able to objectively evaluate the robustness of audio watermarks against removal. Previous watermark removal schemes either assume impractical knowledge of the watermarks they are designed to remove or are computationally expensive, potentially generating a false sense of confidence in current watermark schemes. We introduce HarmonicAttack, an efficient audio watermark removal method that only requires the basic ability to generate the watermarks from the targeted scheme and nothing else. With this, we are able to train a general watermark removal model that is able to remove the watermarks generated by the targeted scheme from any watermarked audio sample. HarmonicAttack employs a dual-path convolutional autoencoder that operates in both temporal and frequency domains, along with GAN-style training, to separate the watermark from the original audio. When evaluated against state-of-the-art watermark schemes AudioSeal, WavMark, and Silentcipher, HarmonicAttack demonstrates greater watermark removal ability than previous watermark removal methods with near real-time performance. Moreover, while HarmonicAttack requires training, we find that it is able to transfer to out-of-distribution samples with minimal degradation in performance.
Authors Kexin Li, Xiao Hu, Ilya Grishchenko, David Lie
Categories Artificial Intelligence
0
Multimodal Robust Prompt Distillation for 3D Point Cloud Models
Title CN 用于3D点云模型的多模态鲁棒提示蒸馏
Keywords 3D点云 · 对抗攻击 · 提示蒸馏 · 多模态学习 · 模型鲁棒性
Summary 本文提出了一种名为多模态鲁棒提示蒸馏（MRPD）的教师-学生框架，用于提升3D点云模型在面对对抗攻击时的鲁棒性。该方法通过融合来自深度投影的视觉模型、高性能3D模型和文本编码器的多模态知识，学习轻量级提示，并引入置信度门控机制动态平衡各模态贡献。训练完成后推理无额外计算开销，在多种白盒和黑盒攻击下均表现出优越性能。
Reason 该论文研究的是3D点云模型在对抗攻击下的鲁棒性问题，属于3D视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管其采用了多模态提示蒸馏和教师-学生框架，但应用场景为深度投影与点云数据，无法迁移至解决文档图像中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Adversarial attacks pose a significant threat to learning-based 3D point cloud models, critically undermining their reliability in security-sensitive applications. Existing defense methods often suffer from (1) high computational overhead and (2) poor generalization ability across diverse attack types. To bridge these gaps, we propose a novel yet efficient teacher-student framework, namely Multimodal Robust Prompt Distillation (MRPD) for distilling robust 3D point cloud model. It learns lightweight prompts by aligning student point cloud model's features with robust embeddings from three distinct teachers: a vision model processing depth projections, a high-performance 3D model, and a text encoder. To ensure a reliable knowledge transfer, this distillation is guided by a confidence-gated mechanism which dynamically balances the contribution of all input modalities. Notably, since the distillation is all during the training stage, there is no additional computational cost at inference. Extensive experiments demonstrate that MRPD substantially outperforms state-of-the-art defense methods against a wide range of white-box and black-box attacks, while even achieving better performance on clean data. Our work presents a new, practical paradigm for building robust 3D vision systems by efficiently harnessing multimodal knowledge.
Authors Xiang Gu, Liming Lu, Xu Zheng, Anan Du, Yongbin Zhou, Shuchao Pang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
From Prediction to Foresight: The Role of AI in Designing Responsible Futures Journal of Artificial Intelligence for Sustainable Development
Title CN 从预测到远见：人工智能在设计负责任未来中的作用
Keywords 负责任AI · 计算远见 · 政策支持 · 可持续发展 · 人机协同
Summary 本文提出了‘负责任计算远见’这一概念，探讨人工智能如何辅助政策制定者进行前瞻性决策，强调在社会、环境、经济和政治系统中实现可持续、有弹性的未来发展。作者主张将AI作为支持工具，增强人类判断力而非取代之，推动以人为中心的远见实践。
Reason 该论文属于政策与社会影响层面的讨论，提出‘负责任计算远见’的概念，聚焦于AI在公共决策和可持续发展中的伦理应用。内容未涉及任何文档图像理解（DIU）、视觉语言模型（VLM）架构改进、OCR去幻觉、细粒度定位或高分辨率处理等技术问题，也无可用于DIU的方法论创新。属于宏观AI治理类研究，与本领域核心痛点无关。
Abstract In an era marked by rapid technological advancements and complex global challenges, responsible foresight has emerged as an essential framework for policymakers aiming to navigate future uncertainties and shape the future. Responsible foresight entails the ethical anticipation of emerging opportunities and risks, with a focus on fostering proactive, sustainable, and accountable future design. This paper coins the term "responsible computational foresight", examining the role of human-centric artificial intelligence and computational modeling in advancing responsible foresight, establishing a set of foundational principles for this new field and presenting a suite of AI-driven foresight tools currently shaping it. AI, particularly in conjunction with simulations and scenario analysis, enhances policymakers' ability to address uncertainty, evaluate risks, and devise strategies geared toward sustainable, resilient futures. However, responsible foresight extends beyond mere technical forecasting; it demands a nuanced understanding of the interdependencies within social, environmental, economic and political systems, alongside a commitment to ethical, long-term decision-making that supports human intelligence. We argue that AI will play a role as a supportive tool in responsible, human-centered foresight, complementing rather than substituting policymaker judgment to enable the proactive shaping of resilient and ethically sound futures. This paper advocates for the thoughtful integration of AI into foresight practices to empower policymakers and communities as they confront the grand challenges of the 21st century.
Authors Maria Perez-Ortiz
Comment Accessible at https://projecteuclid.org/journals/journal-of-artificial-intelligence-for-sustainable-development/volume-1/issue-1/From-Prediction-to-Foresight--The-Role-of-AI-in/10.69828/4d4kja.full
Categories Artificial Intelligence
0
Self-Transparency Failures in Expert-Persona LLMs: A Large-Scale Behavioral Audit
Title CN 专家角色下大语言模型的自我透明性失效：一项大规模行为审计
Keywords 大语言模型 · 自我透明性 · 专家角色 · AI伦理 · 行为审计
Summary 该研究对16个开源大模型在高风险领域（如金融顾问、神经外科医生）扮演专家角色时的自我披露行为进行了大规模测试，发现模型在不同专业 persona 下披露其AI身份的概率存在巨大差异，最低仅3.5%。结果表明，模型的透明性更多由训练因素决定而非参数规模，且推理优化可能进一步抑制透明性。这提示组织不能假设安全特性可跨场景迁移，需进行针对性设计和验证。
Reason 该论文研究的是大语言模型在模拟专家角色时是否主动披露自身AI身份的问题，属于AI安全与伦理范畴，关注用户信任与虚假专业性风险。尽管涉及LLM行为分析，但完全不涉及文档图像、多模态输入、视觉内容理解或OCR相关技术。与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无任何技术关联，也无法迁移用于解决OCR幻觉、细粒度定位或复杂排版理解等DIU痛点。
Abstract If a language model cannot reliably disclose its AI identity in expert contexts, users cannot trust its competence boundaries. This study examines self-transparency in models assigned professional personas within high-stakes domains where false expertise risks user harm. Using a common-garden design, sixteen open-weight models (4B--671B parameters) were audited across 19,200 trials. Models exhibited sharp domain-specific inconsistency: a Financial Advisor persona elicited 30.8% disclosure initially, while a Neurosurgeon persona elicited only 3.5%. This creates preconditions for a "Reverse Gell-Mann Amnesia" effect, where transparency in some domains leads users to overgeneralize trust to contexts where disclosure fails. Disclosure ranged from 2.8% to 73.6%, with a 14B model reaching 61.4% while a 70B produced just 4.1%. Model identity predicted behavior better than parameter count ($ΔR_{adj}^{2} = 0.359$ vs 0.018). Reasoning optimization actively suppressed self-transparency in some models, with reasoning variants showing up to 48.4% lower disclosure than base counterparts. Bayesian validation with Rogan--Gladen correction confirmed robustness to measurement error ($κ= 0.908$). These findings demonstrate transparency reflects training factors rather than scale. Organizations cannot assume safety properties transfer to deployment contexts, requiring deliberate behavior design and empirical verification.
Authors Alex Diep
Categories Artificial Intelligence
0
VacuumVLA: Boosting VLA Capabilities via a Unified Suction and Gripping Tool for Complex Robotic Manipulation
Title CN VacuumVLA：通过统一的吸盘和夹持工具提升复杂机器人操作中的视觉语言动作能力
Keywords Vision Language Action · 机器人操作 · 末端执行器 · 吸盘 · 夹持器
Summary 本文提出了一种低成本的集成化机械夹爪与真空吸盘组合装置，用于增强视觉语言动作（VLA）模型在复杂机器人操作任务中的能力。该设计支持两种模式的灵活切换或协同使用，提升了机器人在无把手抽屉开启、玻璃擦拭等任务上的表现，并在DexVLA和Pi0两个先进VLA框架中进行了验证。
Reason 该论文属于具身智能与机器人领域，研究的是机械末端执行器的硬件设计（夹爪+吸盘），尽管结合了VLA框架，但核心贡献在于机器人操控的物理装置，而非文档图像理解或VLM方法论创新。与DIU领域的OCR幻觉、文本识别、复杂版式解析等关键问题完全无关。
Abstract Vision Language Action models have significantly advanced general purpose robotic manipulation by harnessing large scale pretrained vision and language representations. Among existing approaches, a majority of current VLA systems employ parallel two finger grippers as their default end effectors. However, such grippers face inherent limitations in handling certain real world tasks such as wiping glass surfaces or opening drawers without handles due to insufficient contact area or lack of adhesion. To overcome these challenges, we present a low cost, integrated hardware design that combines a mechanical two finger gripper with a vacuum suction unit, enabling dual mode manipulation within a single end effector. Our system supports flexible switching or synergistic use of both modalities, expanding the range of feasible tasks. We validate the efficiency and practicality of our design within two state of the art VLA frameworks: DexVLA and Pi0. Experimental results demonstrate that with the proposed hybrid end effector, robots can successfully perform multiple complex tasks that are infeasible for conventional two finger grippers alone. All hardware designs and controlling systems will be released.
Authors Hui Zhou, Siyuan Huang, Minxing Li, Hao Zhang, Lue Fan, Shaoshuai Shi
Comment 8 pages
Categories Artificial Intelligence
0
Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals NeurIPS 2025
Title CN 力提示：视频生成模型可以学习并泛化基于物理的控制信号
Keywords 视频生成 · 物理控制 · 世界模型 · 力提示 · 视觉-语言模型
Summary 本文提出了一种名为‘力提示’的方法，利用物理力作为控制信号来引导视频生成模型产生符合真实物理规律的动态效果，例如局部点击或全局风力作用。模型通过在Blender合成的数据上进行训练，能够在没有3D资产或物理仿真器的情况下生成响应力控制的视频。研究表明，即使训练数据有限，模型也能在多种几何结构、环境和材料中泛化出合理的物理行为。关键因素包括视觉多样性和特定文本关键词的使用。
Reason 该论文研究的是视频生成模型中的物理力控制信号，属于纯视频生成与物理模拟领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心方法聚焦于通过‘力提示’操控生成视频中的物理交互（如风吹、点击），并未涉及OCR、布局分析、表格识别或多模态文档推理等DIU核心问题。尽管使用了视觉-语言模型技术，但应用场景和目标完全偏离文档理解范畴，属于无关领域工作。
Abstract Recent advances in video generation models have sparked interest in world models capable of simulating realistic environments. While navigation has been well-explored, physically meaningful interactions that mimic real-world forces remain largely understudied. In this work, we investigate using physical forces as a control signal for video generation and propose force prompts which enable users to interact with images through both localized point forces, such as poking a plant, and global wind force fields, such as wind blowing on fabric. We demonstrate that these force prompts can enable videos to respond realistically to physical control signals by leveraging the visual and motion prior in the original pretrained model, without using any 3D asset or physics simulator at inference. The primary challenge of force prompting is the difficulty in obtaining high quality paired force-video training data, both in the real world due to the difficulty of obtaining force signals, and in synthetic data due to limitations in the visual quality and domain diversity of physics simulators. Our key finding is that video generation models can generalize remarkably well when adapted to follow physical force conditioning from videos synthesized by Blender, even with limited demonstrations of few objects. Our method can generate videos which simulate forces across diverse geometries, settings, and materials. We also try to understand the source of this generalization and perform ablations that reveal two key elements: visual diversity and the use of specific text keywords during training. Our approach is trained on only around 15k training examples for a single day on four A100 GPUs, and outperforms existing methods on force adherence and physics realism, bringing world models closer to real-world physics interactions. We release all datasets, code, weights, and interactive video demos at our project page.
Authors Nate Gillman, Charles Herrmann, Michael Freeman, Daksh Aggarwal, Evan Luo, Deqing Sun, Chen Sun
Comment Camera ready version (NeurIPS 2025). Code and interactive demos at https://force-prompting.github.io/
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
A Gray-box Attack against Latent Diffusion Model-based Image Editing by Posterior Collapse
Title CN 通过后验坍塌对基于潜在扩散模型的图像编辑进行灰盒攻击
Keywords 潜在扩散模型 · 对抗攻击 · 后验坍塌 · 图像保护 · VAE
Summary 本文提出一种名为后验坍塌攻击（PCA）的新框架，用于保护图像免受基于潜在扩散模型（LDM）的未经授权编辑。该方法利用VAE推理过程中的扩散坍塌和集中坍塌现象，设计统一损失函数，在仅访问VAE编码器的情况下实现高效、跨模型的图像保护。相比现有方法，PCA计算成本低、泛化性强且无需空提示优化，但在任务目标和技术路径上均远离文档图像理解领域。
Reason 该论文研究的是针对图像生成模型（Latent Diffusion Models）的灰盒攻击，属于图像安全与对抗攻击领域，核心目标是防止图像被未经授权地编辑。其技术路径聚焦于VAE后验坍塌现象，应用于图像保护，与文档图像理解（DIU）、DocVQA、OCR、布局分析等任务无直接关联。尽管涉及视觉模型，但既不解决文档理解中的OCR幻觉、复杂排版、长文档推理等核心痛点，也无法迁移为提升DIU性能的方法工具。
Abstract Recent advancements in Latent Diffusion Models (LDMs) have revolutionized image synthesis and manipulation, raising significant concerns about data misappropriation and intellectual property infringement. While adversarial attacks have been extensively explored as a protective measure against such misuse of generative AI, current approaches are severely limited by their heavy reliance on model-specific knowledge and substantial computational costs. Drawing inspiration from the posterior collapse phenomenon observed in VAE training, we propose the Posterior Collapse Attack (PCA), a novel framework for protecting images from unauthorized manipulation. Through comprehensive theoretical analysis and empirical validation, we identify two distinct collapse phenomena during VAE inference: diffusion collapse and concentration collapse. Based on this discovery, we design a unified loss function that can flexibly achieve both types of collapse through parameter adjustment, each corresponding to different protection objectives in preventing image manipulation. Our method significantly reduces dependence on model-specific knowledge by requiring access to only the VAE encoder, which constitutes less than 4\% of LDM parameters. Notably, PCA achieves prompt-invariant protection by operating on the VAE encoder before text conditioning occurs, eliminating the need for empty prompt optimization required by existing methods. This minimal requirement enables PCA to maintain adequate transferability across various VAE-based LDM architectures while effectively preventing unauthorized image editing. Extensive experiments show PCA outperforms existing techniques in protection effectiveness, computational efficiency (runtime and VRAM), and generalization across VAE-based LDM variants. Our code is available at https://github.com/ZhongliangGuo/PosteriorCollapseAttack.
Authors Zhongliang Guo, Chun Tong Lei, Lei Fang, Shuai Zhao, Yifei Qian, Jingyu Lin, Zeyu Wang, Cunjian Chen, Ognjen Arandjelović, Chun Pong Lau
Comment 15 pages, 9 figures, 9 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Predictive Safety Shield for Dyna-Q Reinforcement Learning
Title CN 用于Dyna-Q强化学习的预测安全盾
Keywords 强化学习 · 安全盾 · 模型预测 · Dyna-Q · 离散动作空间
Summary 本文提出了一种针对Dyna-Q强化学习的预测性安全盾机制，通过在安全模拟中进行局部Q函数更新，实现安全性与性能的平衡。实验表明该方法在网格世界环境中能有效找到最优路径，并对模拟与现实之间的分布偏移具有鲁棒性。
Reason 该论文属于具身智能/机器人控制领域的强化学习方法，研究的是在离散空间中通过模型预测提升安全性的控制策略。其核心问题与文档图像理解（DIU）无直接关联，既不涉及多模态语言模型、视觉表示、OCR、布局分析等DIU任务，也无法迁移解决OCR幻觉、复杂排版、细粒度定位或长文档推理等关键痛点。
Abstract Obtaining safety guarantees for reinforcement learning is a major challenge to achieve applicability for real-world tasks. Safety shields extend standard reinforcement learning and achieve hard safety guarantees. However, existing safety shields commonly use random sampling of safe actions or a fixed fallback controller, therefore disregarding future performance implications of different safe actions. In this work, we propose a predictive safety shield for model-based reinforcement learning agents in discrete space. Our safety shield updates the Q-function locally based on safe predictions, which originate from a safe simulation of the environment model. This shielding approach improves performance while maintaining hard safety guarantees. Our experiments on gridworld environments demonstrate that even short prediction horizons can be sufficient to identify the optimal path. We observe that our approach is robust to distribution shifts, e.g., between simulation and reality, without requiring additional training.
Authors Jin Pin, Krasowski Hanna, Vanneaux Elena
Categories Artificial Intelligence
0
Flow Matching Meets PDEs: A Unified Framework for Physics-Constrained Generation
Title CN 流匹配遇上偏微分方程：一种物理约束生成的统一框架
Keywords 流匹配 · 扩散模型 · 物理约束生成 · PDE · 代理模型
Summary 本文提出了一种名为Physics-Based Flow Matching (PBFM) 的新生成框架，通过在流匹配目标中显式嵌入偏微分方程（PDE）残差和代数关系等物理约束，实现对复杂物理系统的高效建模。该方法无需调参即可联合优化生成损失与物理一致性损失，并通过训练时的时间展开和随机采样策略进一步提升预测精度。实验表明，PBFM在多个PDE基准上比传统流匹配方法物理残差降低达8倍，且分布准确性更优。
Reason 该论文属于物理模拟与生成模型结合的领域，核心贡献是将物理约束（PDE残差）嵌入流匹配框架中，用于提升科学计算中的生成准确性。其研究目标与文档图像理解（DIU）无直接关联，既不涉及OCR、布局分析、表格识别等任务，也无法迁移解决DIU中的核心痛点如OCR幻觉、细粒度定位或复杂排版理解。属于无关领域（科学仿真），不符合关注范畴。
Abstract Generative machine learning methods, such as diffusion models and flow matching, have shown great potential in modeling complex system behaviors and building efficient surrogate models. However, these methods typically learn the underlying physics implicitly from data. We propose Physics-Based Flow Matching (PBFM), a novel generative framework that explicitly embeds physical constraints, both PDE residuals and algebraic relations, into the flow matching objective. We also introduce temporal unrolling at training time that improves the accuracy of the final, noise-free sample prediction. Our method jointly minimizes the flow matching loss and the physics-based residual loss without requiring hyperparameter tuning of their relative weights. Additionally, we analyze the role of the minimum noise level, $σ_{\min}$, in the context of physical constraints and evaluate a stochastic sampling strategy that helps to reduce physical residuals. Through extensive benchmarks on three representative PDE problems, we show that our approach yields up to an $8\times$ more accurate physical residuals compared to FM, while clearly outperforming existing algorithms in terms of distributional accuracy. PBFM thus provides a principled and efficient framework for surrogate modeling, uncertainty quantification, and accelerated simulation in physics and engineering applications.
Authors Giacomo Baldan, Qiang Liu, Alberto Guardone, Nils Thuerey
Categories Artificial Intelligence
0
Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation
Title CN 语音、偏见与共指：关于语音翻译中性别问题的可解释性研究
Keywords 语音翻译 · 性别偏见 · 可解释性 · 共指解析 · 声学特征
Summary 本文研究在从英语翻译为西班牙语、法语和意大利语的语音翻译任务中，模型如何基于语音特征（如音高）进行性别分配。作者发现模型不仅依赖训练数据中的性别关联模式，还会根据声学输入覆盖语言模型的默认男性偏好，并通过第一人称代词将说话者信息与性别词关联。研究使用对比特征归因方法揭示了模型利用频谱分布信息而非单一音高的机制。
Reason 该论文研究的是语音翻译（Speech Translation）中的性别偏见和共指问题，属于语音与NLP交叉领域，不涉及文档图像、视觉布局、OCR或任何多模态文档理解任务。尽管其探讨了模型内部机制和偏差，但完全脱离文档图像理解（DIU）的技术路径，且未提供可用于解决DIU中OCR幻觉、复杂排版或视觉-语言对齐等核心痛点的方法论创新。
Abstract Unlike text, speech conveys information about the speaker, such as gender, through acoustic cues like pitch. This gives rise to modality-specific bias concerns. For example, in speech translation (ST), when translating from languages with notional gender, such as English, into languages where gender-ambiguous terms referring to the speaker are assigned grammatical gender, the speaker's vocal characteristics may play a role in gender assignment. This risks misgendering speakers, whether through masculine defaults or vocal-based assumptions. Yet, how ST models make these decisions remains poorly understood. We investigate the mechanisms ST models use to assign gender to speaker-referring terms across three language pairs (en-es/fr/it), examining how training data patterns, internal language model (ILM) biases, and acoustic information interact. We find that models do not simply replicate term-specific gender associations from training data, but learn broader patterns of masculine prevalence. While the ILM exhibits strong masculine bias, models can override these preferences based on acoustic input. Using contrastive feature attribution on spectrograms, we reveal that the model with higher gender accuracy relies on a previously unknown mechanism: using first-person pronouns to link gendered terms back to the speaker, accessing gender information distributed across the frequency spectrum rather than concentrated in pitch.
Authors Lina Conti, Dennis Fucci, Marco Gaido, Matteo Negri, Guillaume Wisniewski, Luisa Bentivogli
Comment Submitted to LREC 2026
Categories Artificial Intelligence, Computation and Language
0
Mechanistic Interpretability for Transformer-based Time Series Classification
Title CN 基于Transformer的时间序列分类的机制可解释性
Keywords Transformer · 时间序列分类 · 机制可解释性 · 稀疏自编码器 · 注意力机制
Summary 本文将机制可解释性技术（包括激活补丁、注意力显著性和稀疏自编码器）从自然语言处理领域迁移到基于Transformer的时间序列分类模型中，试图揭示模型内部注意力头和时间步之间的因果关系，并构建内部信息传播的因果图。实验在基准时间序列数据集上进行，展示了关键注意力组件的作用以及潜在可解释特征的发现。
Reason 该论文研究的是时间序列分类中的Transformer模型可解释性问题，属于NLP或通用AI模型解释技术向时间序列领域的迁移应用。尽管其使用的Mechanistic Interpretability方法（如激活补丁、稀疏自编码器）在理论上具有跨模态潜力，但文中未涉及任何视觉输入、多模态建模或文档结构信息处理，且任务目标与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂版式、细粒度定位等）无直接关联。无法看出该方法如何有效迁移到文档理解场景。
Abstract Transformer-based models have become state-of-the-art tools in various machine learning tasks, including time series classification, yet their complexity makes understanding their internal decision-making challenging. Existing explainability methods often focus on input-output attributions, leaving the internal mechanisms largely opaque. This paper addresses this gap by adapting various Mechanistic Interpretability techniques; activation patching, attention saliency, and sparse autoencoders, from NLP to transformer architectures designed explicitly for time series classification. We systematically probe the internal causal roles of individual attention heads and timesteps, revealing causal structures within these models. Through experimentation on a benchmark time series dataset, we construct causal graphs illustrating how information propagates internally, highlighting key attention heads and temporal positions driving correct classifications. Additionally, we demonstrate the potential of sparse autoencoders for uncovering interpretable latent features. Our findings provide both methodological contributions to transformer interpretability and novel insights into the functional mechanics underlying transformer performance in time series classification tasks.
Authors Matīss Kalnāre, Sofoklis Kitharidis, Thomas Bäck, Niki van Stein
Categories Artificial Intelligence
0
Tool-RoCo: An Agent-as-Tool Self-organization Large Language Model Benchmark in Multi-robot Cooperation
Title CN Tool-RoCo：一种用于多机器人协作中代理即工具自组织大语言模型的基准
Keywords 多代理系统 · 自组织 · 机器人协作 · 大语言模型 · 基准测试
Summary 本文提出了Tool-RoCo，一个基于多机器人协作基准RoCo的大语言模型多代理合作评估基准。它将其他代理视为可调用的工具，通过工具使用来衡量代理间的协作与自组织能力，并设计了四种LLM范式来评估不同层次的自主性。实验表明当前LLMs倾向于保持代理激活状态，很少主动去激活或动态协调。该工作聚焦于多机器人系统的自主协作，不涉及文档理解或多模态视觉语言任务。
Reason 该论文属于具身智能与多机器人系统领域，研究的是多代理在物理环境中的协作与自组织机制，涉及任务如SORT、PACK等机器人操作。尽管提出了代理作为工具的范式和自组织架构，但其核心场景与文档图像理解（DIU）或DocVQA无关。所提出的‘agent-as-tool’机制难以直接迁移至解决OCR幻觉、复杂排版解析或长文档推理等DIU核心问题，且未涉及视觉-语言建模、高分辨率图像处理或多模态对齐等关键技术路径。
Abstract This study proposes Tool-RoCo, a novel benchmark for evaluating large language models (LLMs) in long-term multi-agent cooperation based on RoCo, a multi-robot cooperative benchmark. Recent research on LLM-based multi-agent systems has relied on predefined orchestration, while ignoring agent autonomy. Tool-RoCo treats other agents as tools and introduces cooperative tools, leveraging tool usage to evaluate multi-agent cooperation and self-organization. Tool usage means that each agent (LLM) selects a tool from a candidate set based on the current state, receives feedback, and adjusts its selection in subsequent rounds. To evaluate different autonomy levels, we propose four LLM paradigms: (1) centralized cooperation, where a single LLM allocates tools to all agents; (2) centralized self-organization, where a central LLM autonomously activates agents while keeping others inactive; (3) decentralized cooperation, where each agent has its own LLM and calls tools based on local information; and (4) self-organization, where a randomly chosen initial agent can request collaboration, activating additional agents via tool calls. Tool-RoCo includes three multi-robot tasks, SORT, PACK, and CABINET, to measure format and parameter accuracy and agent coordination through tool usage. The results using several LLMs showed that cooperative tools accounted for only 7.09% of all tools, indicating that LLM-based agents rarely invoked others as assistants. Moreover, activation tools accounted for 96.42%, suggesting that current LLMs tend to maintain active agents while seldom deactivating them for adaptive coordination. Tool-RoCo provides a systematic benchmark to evaluate LLM autonomy and cooperation in multi-agent tasks. Code and Demo: https://github.com/ColaZhang22/Tool-Roco
Authors Ke Zhang, Xiaoning Zhao, Ce Zheng, Jiahong Ning, Dandan Zhu, Wenqi Zhang, Chen Sun, Toshiharu Sugawara
Comment 9 pages, 3 figures
Categories Artificial Intelligence
0
Modular, On-Site Solutions with Lightweight Anomaly Detection for Sustainable Nutrient Management in Agriculture
Title CN 基于轻量级异常检测的模块化现场解决方案用于农业可持续养分管理
Keywords 异常检测 · 多光谱成像 · 自动编码器 · Vision Transformer · 农业可持续性
Summary 本文提出了一种用于农业养分管理的模块化、分层式异常检测与状态估计算法流程，结合自动编码器（AE）实现早期预警，并比较了基于植被指数的机器学习模型（RF）与全图深度学习模型（ViT）在磷、钙等营养元素估计中的表现。虽然在能效与精度之间进行了权衡分析，但应用场景局限于农业边缘诊断，不涉及文档结构、文本识别或多模态语言模型在文档上的应用。
Reason 该论文属于农业领域的应用型研究，使用多光谱成像与机器学习进行作物状态估计和异常检测。尽管涉及计算机视觉技术，但其核心任务为农业资源管理，与文档图像理解（DIU）或DocVQA无关。未解决OCR幻觉、复杂排版、长文档推理等DIU核心问题，也未提出可迁移至文档理解领域的底层方法。
Abstract Efficient nutrient management is critical for crop growth and sustainable resource consumption (e.g., nitrogen, energy). Current approaches require lengthy analyses, preventing real-time optimization; similarly, imaging facilitates rapid phenotyping but can be computationally intensive, preventing deployment under resource constraints. This study proposes a flexible, tiered pipeline for anomaly detection and status estimation (fresh weight, dry mass, and tissue nutrients), including a comprehensive energy analysis of approaches that span the efficiency-accuracy spectrum. Using a nutrient depletion experiment with three treatments (T1-100%, T2-50%, and T3-25% fertilizer strength) and multispectral imaging (MSI), we developed a hierarchical pipeline using an autoencoder (AE) for early warning. Further, we compared two status estimation modules of different complexity for more detailed analysis: vegetation index (VI) features with machine learning (Random Forest, RF) and raw whole-image deep learning (Vision Transformer, ViT). Results demonstrated high-efficiency anomaly detection (73% net detection of T3 samples 9 days after transplanting) at substantially lower energy than embodied energy in wasted nitrogen. The state estimation modules show trade-offs, with ViT outperforming RF on phosphorus and calcium estimation (R2 0.61 vs. 0.58, 0.48 vs. 0.35) at higher energy cost. With our modular pipeline, this work opens opportunities for edge diagnostics and practical opportunities for agricultural sustainability.
Authors Abigail R. Cohen, Yuming Sun, Zhihao Qin, Harsh S. Muriki, Zihao Xiao, Yeonju Lee, Matthew Housley, Andrew F. Sharkey, Rhuanito S. Ferrarezi, Jing Li, Lu Gan, Yongsheng Chen
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Merge and Bound: Direct Manipulations on Weights for Class Incremental Learning
Title CN 合并与约束：通过直接权重操作进行类增量学习
Keywords 类增量学习 · 权重合并 · 灾难性遗忘 · 参数空间优化
Summary 本文提出了一种名为Merge-and-Bound（M&B）的类增量学习方法，通过在参数空间中直接操作模型权重来缓解灾难性遗忘。该方法包括跨任务和任务内权重合并，并引入有界更新策略以最小化累积变化，从而保留旧知识。虽然有效，但其应用局限于增量学习场景，未针对文档图像理解或多模态任务设计。
Reason 该论文研究的是类增量学习（CIL）中的模型权重操作问题，属于模型持续学习的技术范畴。尽管其方法在参数空间中进行权重合并与更新约束，具有一定的通用性，但其核心目标与文档图像理解（DIU）任务无直接关联。未涉及OCR、布局分析、表格识别、视觉语言建模或多模态对齐等DIU关键问题，也未提出可用于解决文档理解中幻觉、高分辨率处理或复杂排版解析的可迁移机制。属于通用AI训练方法，且无明确迁移到DIU场景的路径。
Abstract We present a novel training approach, named Merge-and-Bound (M&B) for Class Incremental Learning (CIL), which directly manipulates model weights in the parameter space for optimization. Our algorithm involves two types of weight merging: inter-task weight merging and intra-task weight merging. Inter-task weight merging unifies previous models by averaging the weights of models from all previous stages. On the other hand, intra-task weight merging facilitates the learning of current task by combining the model parameters within current stage. For reliable weight merging, we also propose a bounded update technique that aims to optimize the target model with minimal cumulative updates and preserve knowledge from previous tasks; this strategy reveals that it is possible to effectively obtain new models near old ones, reducing catastrophic forgetting. M&B is seamlessly integrated into existing CIL methods without modifying architecture components or revising learning objectives. We extensively evaluate our algorithm on standard CIL benchmarks and demonstrate superior performance compared to state-of-the-art methods.
Authors Taehoon Kim, Donghwan Jang, Bohyung Han
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Safety Control of Service Robots with LLMs and Embodied Knowledge Graphs
Title CN 基于大语言模型与具身知识图谱的服务机器人安全控制
Keywords 服务机器人 · 安全控制 · 大语言模型 · 具身知识图谱 · 知识验证
Summary 本文提出了一种结合大语言模型（LLM）、具身机器人控制提示（ERCPs）和具身知识图谱（EKGs）的框架，用于提升服务机器人的安全性。ERCPs提供预定义指令以引导LLM生成安全响应，EKGs则对生成的动作进行验证，确保符合安全协议。实验表明该框架在多种真实任务中显著提高了机器人对安全标准的遵守程度。
Reason 该论文属于具身智能/机器人领域，研究的是服务机器人的安全控制问题，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管使用了LLM和知识图谱等技术，但其方法聚焦于机器人动作的安全验证，无法迁移解决DIU中的OCR幻觉、复杂排版、长文档推理等核心痛点。
Abstract Safety limitations in service robotics across various industries have raised significant concerns about the need for robust mechanisms ensuring that robots adhere to safe practices, thereby preventing actions that might harm humans or cause property damage. Despite advances, including the integration of Knowledge Graphs (KGs) with Large Language Models (LLMs), challenges in ensuring consistent safety in autonomous robot actions persist. In this paper, we propose a novel integration of Large Language Models with Embodied Robotic Control Prompts (ERCPs) and Embodied Knowledge Graphs (EKGs) to enhance the safety framework for service robots. ERCPs are designed as predefined instructions that ensure LLMs generate safe and precise responses. These responses are subsequently validated by EKGs, which provide a comprehensive knowledge base ensuring that the actions of the robot are continuously aligned with safety protocols, thereby promoting safer operational practices in varied contexts. Our experimental setup involved diverse real-world tasks, where robots equipped with our framework demonstrated significantly higher compliance with safety standards compared to traditional methods. This integration fosters secure human-robot interactions and positions our methodology at the forefront of AI-driven safety innovations in service robotics.
Authors Yong Qi, Gabriel Kyebambo, Siyuan Xie, Wei Shen, Shenghui Wang, Bitao Xie, Bin He, Zhipeng Wang, Shuo Jiang
Categories Artificial Intelligence
0
Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimization
Title CN 通过树-组双感知搜索与优化实现LLM安全对齐的对抗攻防协同进化
Keywords 大语言模型安全 · 越狱攻击 · 对抗训练 · 蒙特卡洛树搜索 · 强化学习
Summary 本文提出ACE-Safety框架，通过组感知策略引导的蒙特卡洛树搜索（GS-MCTS）生成多样化越狱攻击样本，并结合对抗课程树感知组策略优化（AC-TGPO）实现攻防双方的协同强化学习训练，提升LLM在动态环境下的安全性。实验表明该方法在多个基准上优于现有攻防技术。
Reason 该论文聚焦于大语言模型的安全对齐问题，研究对抗攻击与防御的协同演化机制，属于纯NLP安全/对齐领域（Safety/Jailbreak），未涉及任何视觉模态或文档理解任务。尽管其方法论具有一定创新性，但无法迁移解决文档图像理解中的核心痛点如OCR幻觉、细粒度定位或复杂排版理解等问题。
Abstract Large Language Models (LLMs) have developed rapidly in web services, delivering unprecedented capabilities while amplifying societal risks. Existing works tend to focus on either isolated jailbreak attacks or static defenses, neglecting the dynamic interplay between evolving threats and safeguards in real-world web contexts. To mitigate these challenges, we propose ACE-Safety (Adversarial Co-Evolution for LLM Safety), a novel framework that jointly optimize attack and defense models by seamlessly integrating two key innovative procedures: (1) Group-aware Strategy-guided Monte Carlo Tree Search (GS-MCTS), which efficiently explores jailbreak strategies to uncover vulnerabilities and generate diverse adversarial samples; (2) Adversarial Curriculum Tree-aware Group Policy Optimization (AC-TGPO), which jointly trains attack and defense LLMs with challenging samples via curriculum reinforcement learning, enabling robust mutual improvement. Evaluations across multiple benchmarks demonstrate that our method outperforms existing attack and defense approaches, and provides a feasible pathway for developing LLMs that can sustainably support responsible AI ecosystems.
Authors Xurui Li, Kaisong Song, Rui Zhu, Pin-Yu Chen, Haixu Tang
Categories Artificial Intelligence
0
Going with the Speed of Sound: Pushing Neural Surrogates into Highly-turbulent Transonic Regimes NeurIPS 2025 ML4PS Workshop
Title CN 以声音的速度前行：将神经代理推进到高度湍流的跨音速区域
Keywords 神经代理 · 跨音速流动 · CFD仿真 · 气动优化 · 3D机翼
Summary 本文提出一个用于3D机翼在跨音速条件下进行气动仿真的大规模CFD数据集，并评估了多种神经代理模型在该任务上的表现，重点在于气动性能（如升阻比帕累托前沿）的预测能力。研究目标是加速航空航天设计流程，而非解决任何文档理解或视觉语言任务。
Reason 该论文研究的是航空空气动力学中的神经代理模型，聚焦于3D机翼在跨音速条件下的流场模拟与气动优化，属于物理仿真领域。尽管使用了深度学习和CFD数据，但其任务、数据和方法均与文档图像理解（DIU）、多模态大模型（VLM）或视觉文本分析无关，完全不涉及OCR、布局分析、表格识别或文档推理等核心问题。
Abstract The widespread use of neural surrogates in automotive aerodynamics, enabled by datasets such as DrivAerML and DrivAerNet++, has primarily focused on bluff-body flows with large wakes. Extending these methods to aerospace, particularly in the transonic regime, remains challenging due to the high level of non-linearity of compressible flows and 3D effects such as wingtip vortices. Existing aerospace datasets predominantly focus on 2D airfoils, neglecting these critical 3D phenomena. To address this gap, we present a new dataset of CFD simulations for 3D wings in the transonic regime. The dataset comprises volumetric and surface-level fields for around $30,000$ samples with unique geometry and inflow conditions. This allows computation of lift and drag coefficients, providing a foundation for data-driven aerodynamic optimization of the drag-lift Pareto front. We evaluate several state-of-the-art neural surrogates on our dataset, including Transolver and AB-UPT, focusing on their out-of-distribution (OOD) generalization over geometry and inflow variations. AB-UPT demonstrates strong performance for transonic flowfields and reproduces physically consistent drag-lift Pareto fronts even for unseen wing configurations. Our results demonstrate that AB-UPT can approximate drag-lift Pareto fronts for unseen geometries, highlighting its potential as an efficient and effective tool for rapid aerodynamic design exploration. To facilitate future research, we open-source our dataset at https://huggingface.co/datasets/EmmiAI/Emmi-Wing.
Authors Fabian Paischer, Leo Cotteleer, Yann Dreze, Richard Kurle, Dylan Rubini, Maurits Bleeker, Tobias Kronlachner, Johannes Brandstetter
Comment NeurIPS 2025 ML4PS Workshop
Categories Artificial Intelligence
0
Step-Audio-R1 Technical Report
Title CN Step-Audio-R1 技术报告
Keywords 音频语言模型 · 推理模型 · 模态锚定 · MGRD · 多模态推理
Summary 本文提出了Step-Audio-R1，首个在音频领域成功实现推理能力的语言模型。通过引入模态锚定推理蒸馏（MGRD）框架，模型能够生成真正基于声学特征的推理链，避免脱离实际的幻觉式推理。实验表明，该模型在语音、环境声音和音乐等多种音频理解与推理基准上表现优异，性能媲美Gemini 3 Pro。研究证明了跨模态推理能力的可迁移性，为多感官深度推理系统提供了新路径。
Reason 该论文研究音频领域的推理模型，属于音频理解范畴，与文档图像理解（DIU）任务无直接关联。尽管其提出的Modality-Grounded Reasoning Distillation框架涉及多模态推理，但应用场景为语音、环境声和音乐等非视觉模态，无法迁移解决文档图像中的OCR幻觉、复杂排版或细粒度定位等核心问题。
Abstract Recent advances in reasoning models have demonstrated remarkable success in text and vision domains through extended chain-of-thought deliberation. However, a perplexing phenomenon persists in audio language models: they consistently perform better with minimal or no reasoning, raising a fundamental question - can audio intelligence truly benefit from deliberate thinking? We introduce Step-Audio-R1, the first audio reasoning model that successfully unlocks reasoning capabilities in the audio domain. Through our proposed Modality-Grounded Reasoning Distillation (MGRD) framework, Step-Audio-R1 learns to generate audio-relevant reasoning chains that genuinely ground themselves in acoustic features rather than hallucinating disconnected deliberations. Our model exhibits strong audio reasoning capabilities, surpassing Gemini 2.5 Pro and achieving performance comparable to the state-of-the-art Gemini 3 Pro across comprehensive audio understanding and reasoning benchmarks spanning speech, environmental sounds, and music. These results demonstrate that reasoning is a transferable capability across modalities when appropriately anchored, transforming extended deliberation from a liability into a powerful asset for audio intelligence. By establishing the first successful audio reasoning model, Step-Audio-R1 opens new pathways toward building truly multimodal reasoning systems that think deeply across all sensory modalities.
Authors Fei Tian, Xiangyu Tony Zhang, Yuxin Zhang, Haoyang Zhang, Yuxin Li, Daijiao Liu, Yayue Deng, Donghang Wu, Jun Chen, Liang Zhao, Chengyuan Yao, Hexin Liu, Eng Siong Chng, Xuerui Yang, Xiangyu Zhang, Daxin Jiang, Gang Yu
Comment 22 pages, 5 figures. Technical Report
Categories Artificial Intelligence, Computation and Language
0
MADRA: Multi-Agent Debate for Risk-Aware Embodied Planning
Title CN MADRA：用于风险感知具身规划的多智能体辩论
Keywords 多智能体系统 · 具身智能 · 安全规划 · 大语言模型 · 辩论机制
Summary 本文提出MADRA，一种无需训练的多智能体辩论框架，用于提升具身智能体在家庭环境中任务规划的安全性。通过多个LLM智能体对指令安全性进行辩论，并由评估者根据逻辑性、风险识别等维度打分，结合共识投票降低误拒率。同时引入分层认知协作框架以提升任务成功率，并构建了SafeAware-VH基准数据集。实验表明该方法在拒绝不安全任务的同时保持高效执行。
Reason 该论文属于具身智能（Embodied AI）领域，研究多智能体在物理环境中进行安全感知的任务规划，与文档图像理解（DIU）或DocVQA无直接关联。尽管其‘多智能体辩论’机制在推理协调上有一定方法论价值，但其核心目标是解决现实环境中的安全指令识别与执行问题，迁移到文档理解场景（如OCR去幻觉、复杂排版解析）极为困难。且论文涉及的VirtualHome、AI2-THOR等仿真平台与文档处理无关，属于明确的无关领域。
Abstract Ensuring the safety of embodied AI agents during task planning is critical for real-world deployment, especially in household environments where dangerous instructions pose significant risks. Existing methods often suffer from either high computational costs due to preference alignment training or over-rejection when using single-agent safety prompts. To address these limitations, we propose MADRA, a training-free Multi-Agent Debate Risk Assessment framework that leverages collective reasoning to enhance safety awareness without sacrificing task performance. MADRA employs multiple LLM-based agents to debate the safety of a given instruction, guided by a critical evaluator that scores responses based on logical soundness, risk identification, evidence quality, and clarity. Through iterative deliberation and consensus voting, MADRA significantly reduces false rejections while maintaining high sensitivity to dangerous tasks. Additionally, we introduce a hierarchical cognitive collaborative planning framework that integrates safety, memory, planning, and self-evolution mechanisms to improve task success rates through continuous learning. We also contribute SafeAware-VH, a benchmark dataset for safety-aware task planning in VirtualHome, containing 800 annotated instructions. Extensive experiments on AI2-THOR and VirtualHome demonstrate that our approach achieves over 90% rejection of unsafe tasks while ensuring that safe-task rejection is low, outperforming existing methods in both safety and execution efficiency. Our work provides a scalable, model-agnostic solution for building trustworthy embodied agents.
Authors Junjian Wang, Lidan Zhao, Xi Sheryl Zhang
Categories Artificial Intelligence
0
DensiCrafter: Physically-Constrained Generation and Fabrication of Self-Supporting Hollow Structures
Title CN DensiCrafter：物理约束下的自支撑中空结构生成与制造
Keywords 3D生成 · 物理约束 · 密度场优化 · 可制造性 · 自支撑结构
Summary 本文提出DensiCrafter框架，用于生成轻量且自支撑的3D中空结构。通过在预训练的Trellis模型生成的粗体素网格上优化连续密度场，并引入可微分的物理约束损失项和质量正则化，实现在不改变模型架构的前提下减少材料使用并提升结构稳定性。实验显示在文本到3D任务中可减少43%的材料质量，且支持真实世界的3D打印。
Reason 该论文研究3D生成模型中的物理可制造性问题，聚焦于轻量化、自支撑的中空结构设计，属于3D视觉与具身智能/制造交叉领域。其核心方法为密度场优化与物理约束损失函数设计，与文档图像理解（DIU）任务无直接关联。无法迁移解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点。
Abstract The rise of 3D generative models has enabled automatic 3D geometry and texture synthesis from multimodal inputs (e.g., text or images). However, these methods often ignore physical constraints and manufacturability considerations. In this work, we address the challenge of producing 3D designs that are both lightweight and self-supporting. We present DensiCrafter, a framework for generating lightweight, self-supporting 3D hollow structures by optimizing the density field. Starting from coarse voxel grids produced by Trellis, we interpret these as continuous density fields to optimize and introduce three differentiable, physically constrained, and simulation-free loss terms. Additionally, a mass regularization penalizes unnecessary material, while a restricted optimization domain preserves the outer surface. Our method seamlessly integrates with pretrained Trellis-based models (e.g., Trellis, DSO) without any architectural changes. In extensive evaluations, we achieve up to 43% reduction in material mass on the text-to-3D task. Compared to state-of-the-art baselines, our method could improve the stability and maintain high geometric fidelity. Real-world 3D-printing experiments confirm that our hollow designs can be reliably fabricated and could be self-supporting.
Authors Shengqi Dang, Fu Chai, Jiaxin Li, Chao Yuan, Wei Ye, Nan Cao
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Constructing and Benchmarking: a Labeled Email Dataset for Text-Based Phishing and Spam Detection Framework
Title CN 构建与基准测试：一个用于基于文本的钓鱼和垃圾邮件检测框架的标注电子邮件数据集
Keywords 钓鱼邮件检测 · 垃圾邮件分类 · LLM生成内容 · 情感分析 · 文本分类
Summary 本文提出了一个标注的电子邮件数据集，包含钓鱼、垃圾和正常邮件，并区分人类与LLM生成的内容。每封邮件标注了情感诉求和动机，通过多个LLM进行基准测试以选择最优标注模型，并评估其在原始和重写邮件上的分类性能。研究发现当前LLM在检测钓鱼邮件方面表现良好，但在区分垃圾邮件与合法邮件上仍有挑战。
Reason 该论文聚焦于网络安全中的钓鱼和垃圾邮件检测，使用LLM进行内容分析与分类，属于特定领域的下游应用（如金融安全场景），并未提出任何文档图像理解（DIU）或视觉-语言模型在文档结构、布局、OCR、表格识别等方面的底层方法创新。其核心是文本分类任务，且不涉及多模态输入或视觉信息处理，与文档图像理解的核心问题无直接关联。
Abstract Phishing and spam emails remain a major cybersecurity threat, with attackers increasingly leveraging Large Language Models (LLMs) to craft highly deceptive content. This study presents a comprehensive email dataset containing phishing, spam, and legitimate messages, explicitly distinguishing between human- and LLM-generated content. Each email is annotated with its category, emotional appeal (e.g., urgency, fear, authority), and underlying motivation (e.g., link-following, credential theft, financial fraud). We benchmark multiple LLMs on their ability to identify these emotional and motivational cues and select the most reliable model to annotate the full dataset. To evaluate classification robustness, emails were also rephrased using several LLMs while preserving meaning and intent. A state-of-the-art LLM was then assessed on its performance across both original and rephrased emails using expert-labeled ground truth. The results highlight strong phishing detection capabilities but reveal persistent challenges in distinguishing spam from legitimate emails. Our dataset and evaluation framework contribute to improving AI-assisted email security systems. To support open science, all code, templates, and resources are available on our project site.
Authors Rebeka Toth, Tamas Bisztray, Richard Dubniczky
Categories Artificial Intelligence
0
EWE: An Agentic Framework for Extreme Weather Analysis
Title CN EWE：一种用于极端天气分析的智能体框架
Keywords 智能体框架 · 极端天气分析 · 自动推理 · 地球科学AI · 多模态分析
Summary 本文提出了一个名为EWE的智能体框架，用于自动化极端天气事件的诊断分析。该框架模拟专家工作流，结合知识引导的规划、闭环推理以及定制化的气象工具包，能够从原始气象数据中自动生成并解释多模态可视化结果。同时作者构建了首个面向该任务的基准数据集和逐步评估指标。然而，该工作聚焦于地球科学领域的应用，不具备对文档图像理解的技术迁移价值。
Reason 该论文属于典型的‘平行下游应用’，将Agent框架应用于气象领域的极端天气分析，尽管涉及多模态数据和推理流程，但其核心是特定领域（地球科学）的应用系统，并未提出可迁移至文档图像理解（DIU）的底层方法或架构创新。任务目标为气象诊断，与DocVQA、布局分析、表格识别等DIU核心任务无直接关联，且未涉及OCR幻觉、密集文本处理、复杂排版建模等关键痛点。
Abstract Extreme weather events pose escalating risks to global society, underscoring the urgent need to unravel their underlying physical mechanisms. Yet the prevailing expert-driven, labor-intensive diagnostic paradigm has created a critical analytical bottleneck, stalling scientific progress. While AI for Earth Science has achieved notable advances in prediction, the equally essential challenge of automated diagnostic reasoning remains largely unexplored. We present the Extreme Weather Expert (EWE), the first intelligent agent framework dedicated to this task. EWE emulates expert workflows through knowledge-guided planning, closed-loop reasoning, and a domain-tailored meteorological toolkit. It autonomously produces and interprets multimodal visualizations from raw meteorological data, enabling comprehensive diagnostic analyses. To catalyze progress, we introduce the first benchmark for this emerging field, comprising a curated dataset of 103 high-impact events and a novel step-wise evaluation metric. EWE marks a step toward automated scientific discovery and offers the potential to democratize expertise and intellectual resources, particularly for developing nations vulnerable to extreme weather.
Authors Zhe Jiang, Jiong Wang, Xiaoyu Yue, Zijie Guo, Wenlong Zhang, Fenghua Ling, Wanli Ouyang, Lei Bai
Categories Artificial Intelligence
0
EvRainDrop: HyperGraph-guided Completion for Effective Frame and Event Stream Aggregation
Title CN EvRainDrop：超图引导的帧与事件流聚合补全方法
Keywords 事件相机 · 超图 · 表示学习 · 稀疏事件补全 · 多模态融合
Summary 本文提出了一种基于超图的时空事件流补全机制EvRainDrop，通过构建跨时空的超图连接稀疏事件，并利用上下文信息传递来完成缺失事件。该方法支持将RGB信息融入超图节点，实现多模态信息补全，并通过自注意力聚合不同时刻的特征。实验表明其在单标签和多标签事件分类任务上有效。
Reason 该论文研究的是基于事件相机（event camera）的视觉任务，处理的是异步、稀疏的事件流数据，属于动态视觉传感器领域。其核心问题和方法均围绕事件表示学习展开，与文档图像理解（DIU）中的OCR、排版分析、表格识别等任务无直接关联。尽管提到了多模态融合与超图结构，但这些机制针对的是时间序列上的事件补全，难以迁移到文档图像的细粒度文本识别、复杂布局解析或长文档推理等典型DIU痛点。属于无关领域。
Abstract Event cameras produce asynchronous event streams that are spatially sparse yet temporally dense. Mainstream event representation learning algorithms typically use event frames, voxels, or tensors as input. Although these approaches have achieved notable progress, they struggle to address the undersampling problem caused by spatial sparsity. In this paper, we propose a novel hypergraph-guided spatio-temporal event stream completion mechanism, which connects event tokens across different times and spatial locations via hypergraphs and leverages contextual information message passing to complete these sparse events. The proposed method can flexibly incorporate RGB tokens as nodes in the hypergraph within this completion framework, enabling multi-modal hypergraph-based information completion. Subsequently, we aggregate hypergraph node information across different time steps through self-attention, enabling effective learning and fusion of multi-modal features. Extensive experiments on both single- and multi-label event classification tasks fully validated the effectiveness of our proposed framework. The source code of this paper will be released on https://github.com/Event-AHU/EvRainDrop.
Authors Futian Wang, Fan Zhang, Xiao Wang, Mengqi Wang, Dexing Huang, Jin Tang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Conversational no-code and multi-agentic disease module identification and drug repurposing prediction with ChatDRex
Title CN 基于对话式无代码多智能体系统的疾病模块识别与药物重定位预测
Keywords 多智能体系统 · 药物重定位 · 知识图谱 · 自然语言接口 · 生物信息学
Summary 本文提出了ChatDRex，一个基于对话的多智能体系统，用于支持网络化药物重定位预测。系统基于NeDRex知识图谱，通过多个专业智能体实现查询路由、数据检索、算法执行和结果可视化，并引入推理模块进行幻觉检测。目标是让非计算机背景的医学研究人员能以自然语言控制复杂的生物信息学分析流程。
Reason 该论文属于典型的平行下游应用（Wrapper/Application），将多智能体系统应用于生物医学领域的药物重定位任务。尽管其使用了自然语言交互和多智能体协作，但核心是特定领域（drug repurposing）的应用集成，并未提出可用于文档图像理解（如OCR去幻觉、复杂排版处理、视觉压缩等）的底层方法或架构创新，对DIU领域无直接迁移价值。
Abstract Repurposing approved drugs offers a time-efficient and cost-effective alternative to traditional drug development. However, in silico prediction of repurposing candidates is challenging and requires the effective collaboration of specialists in various fields, including pharmacology, medicine, biology, and bioinformatics. Fragmented, specialized algorithms and tools often address only narrow aspects of the overall problem, and heterogeneous, unstructured data landscapes require specialized users to be involved. Hence, these data services do not integrate smoothly across workflows. With ChatDRex, we present a conversation-based, multi-agent system that facilitates the execution of complex bioinformatic analyses aiming for network-based drug repurposing prediction. It builds on the integrated systems medicine knowledge graph NeDRex. ChatDRex provides natural language access to its extensive biomedical KG and integrates bioinformatics agents for network analysis and drug repurposing, complemented by agents for functional coherence evaluation for in silico validation, as well as agents for literature mining and for discussing the obtained results in a scientific context. Its flexible multi-agent design assigns specific tasks to specialized agents, including query routing, data retrieval, algorithm execution, and result visualization. A dedicated reasoning module keeps the user in the loop and allows for hallucination detection. By enabling physicians and researchers without computer science expertise to control complex analyses in natural language, ChatDRex democratizes access to bioinformatics as an important resource for drug repurposing. It enables clinical experts to generate hypotheses and explore drug repurposing opportunities, ultimately accelerating the discovery of novel therapies and advancing personalized medicine and translational research.
Authors Simon Süwer, Kester Bagemihl, Sylvie Baier, Lucia Dicunta, Markus List, Jan Baumbach, Andreas Maier, Fernando M. Delgado-Chaves
Categories Artificial Intelligence
0
How Do Companies Manage the Environmental Sustainability of AI? An Interview Study About Green AI Efforts and Regulations 11th International Conference on ICT for Sustainability (ICT4S'25)
Title CN 企业如何管理人工智能的环境可持续性？关于绿色人工智能努力与监管的访谈研究
Keywords 绿色AI · 环境可持续性 · AI治理 · 行业实践 · 访谈研究
Summary 本文通过对11名来自10家企业的从业者的访谈，探讨了企业在采用AI过程中对环境可持续性的关注程度、当前采取的绿色AI措施以及欧盟AI法案和企业可持续发展报告指令（CSRD）的影响。研究发现，大多数企业优先考虑业务效率，极少关注AI的环境影响，且缺乏有效的监测与缓解措施。法规意识薄弱，仅有少数大型企业因CSRD推动而开展可持续性报告工作。整体反映出工业界对可持续AI缺乏紧迫感。
Reason 该论文属于AI伦理与可持续发展领域，研究的是企业在采用AI时对环境影响的认知和管理现状，与文档图像理解（DIU）、多模态大模型、视觉表示学习等核心技术无关。未涉及任何文档理解任务（如DocVQA、OCR、版面分析）或可迁移的技术方法（如视觉压缩、高分辨率处理、推理增强）。属于无关领域应用，直接命中负面清单第2条。
Abstract With the ever-growing adoption of artificial intelligence (AI), AI-based software and its negative impact on the environment are no longer negligible, and studying and mitigating this impact has become a critical area of research. However, it is currently unclear which role environmental sustainability plays during AI adoption in industry and how AI regulations influence Green AI practices and decision-making in industry. We therefore aim to investigate the Green AI perception and management of industry practitioners. To this end, we conducted a total of 11 interviews with participants from 10 different organizations that adopted AI-based software. The interviews explored three main themes: AI adoption, current efforts in mitigating the negative environmental impact of AI, and the influence of the EU AI Act and the Corporate Sustainability Reporting Directive (CSRD). Our findings indicate that 9 of 11 participants prioritized business efficiency during AI adoption, with minimal consideration of environmental sustainability. Monitoring and mitigation of AI's environmental impact were very limited. Only one participant monitored negative environmental effects. Regarding applied mitigation practices, six participants reported no actions, with the others sporadically mentioning techniques like prompt engineering, relying on smaller models, or not overusing AI. Awareness and compliance with the EU AI Act are low, with only one participant reporting on its influence, while the CSRD drove sustainability reporting efforts primarily in larger companies. All in all, our findings reflect a lack of urgency and priority for sustainable AI among these companies. We suggest that current regulations are not very effective, which has implications for policymakers. Additionally, there is a need to raise industry awareness, but also to provide user-friendly techniques and tools for Green AI practices.
Authors Ashmita Sampatsing, Sophie Vos, Emma Beauxis-Aussalet, Justus Bogner
Comment Accepted for publication at the 11th International Conference on ICT for Sustainability (ICT4S'25), see https://conf.researchr.org/home/ict4s-2025
Categories Artificial Intelligence
0
Equivariant Flow Matching for Symmetry-Breaking Bifurcation Problems Machine Learning and the Physical Sciences Workshop, NeurIPS 2025
Title CN 用于对称性破缺分岔问题的等变流匹配
Keywords 流匹配 · 等变建模 · 分岔问题 · 多稳态 · 生成模型
Summary 本文提出一种基于流匹配的生成框架，用于建模对称性破缺分岔问题中的多重稳定解分布。通过引入等变建模和对称性匹配策略，该方法能在保持系统对称性的同时准确学习多模态输出分布，并在梁屈曲、Allen-Cahn方程等物理系统中验证了有效性。尽管在高维多稳态系统中表现优异，但其应用场景远离文档理解任务。
Reason 该论文研究的是非线性动力系统中的分岔现象与多稳态建模，属于物理科学与机器学习交叉领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的应用无直接关联。其方法聚焦于对称性破缺和多解生成，在文档理解的核心痛点（如OCR幻觉、复杂排版、细粒度定位、长文档推理）上缺乏可迁移性。
Abstract Bifurcation phenomena in nonlinear dynamical systems often lead to multiple coexisting stable solutions, particularly in the presence of symmetry breaking. Deterministic machine learning models struggle to capture this multiplicity, averaging over solutions and failing to represent lower-symmetry outcomes. In this work, we propose a generative framework based on flow matching to model the full probability distribution over bifurcation outcomes. Our method enables direct sampling of multiple valid solutions while preserving system symmetries through equivariant modeling. We introduce a symmetric matching strategy that aligns predicted and target outputs under group actions, allowing accurate learning in equivariant settings. We validate our approach on a range of systems, from toy models to complex physical problems such as buckling beams and the Allen-Cahn equation. Our results demonstrate that flow matching significantly outperforms non-probabilistic and variational methods in capturing multimodal distributions and symmetry-breaking bifurcations, offering a principled and scalable solution for modeling multistability in high-dimensional systems.
Authors Fleur Hendriks, Ondřej Rokoš, Martin Doškář, Marc G. D. Geers, Vlado Menkovski
Comment 12 pages, 7 figures including appendices. Accepted to Machine Learning and the Physical Sciences Workshop, NeurIPS 2025 (https://ml4physicalsciences.github.io/2025/). Repository with corresponding code: https://github.com/FHendriks11/bifurcationML/. Video explanation: https://www.youtube.com/watch?v=wsL3h17KtjY
Categories Artificial Intelligence
0
SAM Guided Semantic and Motion Changed Region Mining for Remote Sensing Change Captioning
Title CN 基于SAM引导的语义与运动变化区域挖掘用于遥感变化描述
Keywords 遥感变化描述 · SAM · 变化检测 · 知识图谱 · 跨模态融合
Summary 本文提出一种利用SAM基础模型提取遥感图像中语义和运动变化区域的方法，结合全局视觉特征与构建的知识图谱，通过跨注意力融合多源信息，生成自然语言形式的变化描述。实验表明该方法在多个基准上达到SOTA性能。
Reason 该论文研究的是遥感图像中的变化描述任务，属于特定领域的视觉-语言应用，且其核心方法是将SAM用于变化区域的分割与知识注入，并未提出可迁移至文档图像理解（DIU）的通用架构、表示学习或推理机制。任务场景与文档无关，技术路线聚焦于遥感时序对比与场景变化，难以迁移到OCR、排版理解、细粒度文本识别等DIU核心痛点，属于平行下游应用。
Abstract Remote sensing change captioning is an emerging and popular research task that aims to describe, in natural language, the content of interest that has changed between two remote sensing images captured at different times. Existing methods typically employ CNNs/Transformers to extract visual representations from the given images or incorporate auxiliary tasks to enhance the final results, with weak region awareness and limited temporal alignment. To address these issues, this paper explores the use of the SAM (Segment Anything Model) foundation model to extract region-level representations and inject region-of-interest knowledge into the captioning framework. Specifically, we employ a CNN/Transformer model to extract global-level vision features, leverage the SAM foundation model to delineate semantic- and motion-level change regions, and utilize a specially constructed knowledge graph to provide information about objects of interest. These heterogeneous sources of information are then fused via cross-attention, and a Transformer decoder is used to generate the final natural language description of the observed changes. Extensive experimental results demonstrate that our method achieves state-of-the-art performance across multiple widely used benchmark datasets. The source code of this paper will be released on https://github.com/Event-AHU/SAM_ChangeCaptioning
Authors Futian Wang, Mengqi Wang, Xiao Wang, Haowen Wang, Jin Tang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
New Hybrid Heuristics for Pseudo-Boolean Propagation
Title CN 新的伪布尔传播混合启发式方法
Keywords 伪布尔求解 · 单元传播 · 混合启发式 · SAT求解 · RoundingSAT
Summary 本文提出了一种用于伪布尔求解中混合模式（结合监视文字法与计数法）的新启发式策略，在RoundingSAT求解器中显著优于现有方法。重点在于提升布尔约束系统的推理效率。
Reason 该论文属于纯逻辑推理与布尔可满足性问题（SAT求解）领域，研究的是伪布尔求解中的传播策略优化，与文档图像理解（DIU）、多模态大模型（VLM）、OCR、布局分析、表格识别等核心任务完全无关。尽管SAT求解在理论上有潜在影响，但其方法难以迁移到DIU所面临的OCR幻觉、复杂排版、高分辨率视觉理解或长文档推理等实际问题中。
Abstract In pseudo-boolean solving the currently most successful unit propagation strategy is a hybrid mode combining the watched literal scheme with the counting method. This short paper introduces new heuristics for this hybrid decision, which are able to drastically outperform the current method in the RoundingSAT solver.
Authors Mia Müßig, Jan Johannsen
Categories Artificial Intelligence
0
Adaptive Object Detection for Indoor Navigation Assistance: A Performance Evaluation of Real-Time Algorithms
Title CN 室内导航辅助的自适应目标检测：实时算法的性能评估
Keywords 目标检测 · 室内导航 · 辅助技术 · 实时算法 · 计算机视觉
Summary 本文评估了四种实时目标检测算法（YOLO、SSD、Faster R-CNN、Mask R-CNN）在室内导航辅助系统中的表现，使用室内物体检测数据集分析精度、速度和环境适应性，旨在为视障人士提供更好的导航技术支持。
Reason 该论文属于具身智能/机器人领域的视觉应用，研究的是通用室内物体检测算法在导航辅助中的性能评估，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法为标准的目标检测模型比较（如YOLO、Faster R-CNN等），未涉及OCR、布局分析、表格识别或任何文档特有的结构理解问题，也无法迁移用于解决OCR幻觉、密集文本处理或长文档推理等核心DIU痛点。
Abstract This study addresses the need for accurate and efficient object detection in assistive technologies for visually impaired individuals. We evaluate four real-time object detection algorithms YOLO, SSD, Faster R-CNN, and Mask R-CNN within the context of indoor navigation assistance. Using the Indoor Objects Detection dataset, we analyze detection accuracy, processing speed, and adaptability to indoor environments. Our findings highlight the trade-offs between precision and efficiency, offering insights into selecting optimal algorithms for realtime assistive navigation. This research advances adaptive machine learning applications, enhancing indoor navigation solutions for the visually impaired and promoting accessibility.
Authors Abhinav Pratap, Sushant Kumar, Suchinton Chakravarty
Comment 5 pages, 2 figures, 3 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
RIA: A Ranking-Infused Approach for Optimized listwise CTR Prediction
Title CN RIA：一种用于优化列表级CTR预测的排序融合方法
Keywords 推荐系统 · CTR预测 · 重排序 · 列表级学习 · 效率优化
Summary 本文提出RIA，一种将排序与重排序统一的端到端框架，通过DualTransformer、CUHT、LMH和Embedding Cache模块提升推荐系统的列表级评估能力，在工业场景中取得了CTR和CPM的显著提升。
Reason 该论文研究的是推荐系统中的点击率（CTR）预测与重排序问题，属于广告推荐领域。尽管提出了端到端的排序架构和效率优化模块，但其任务场景、数据模态（纯文本/行为序列）与文档图像理解（DIU）无关，既不涉及视觉输入、OCR、多模态建模，也不解决文档理解中的幻觉、密集文本或复杂布局等问题。属于典型的平行应用领域工作，无法为DIU提供可迁移的方法论创新。
Abstract Reranking improves recommendation quality by modeling item interactions. However, existing methods often decouple ranking and reranking, leading to weak listwise evaluation models that suffer from combinatorial sparsity and limited representational power under strict latency constraints. In this paper, we propose RIA (Ranking-Infused Architecture), a unified, end-to-end framework that seamlessly integrates pointwise and listwise evaluation. RIA introduces four key components: (1) the User and Candidate DualTransformer (UCDT) for fine-grained user-item-context modeling; (2) the Context-aware User History and Target (CUHT) module for position-sensitive preference learning; (3) the Listwise Multi-HSTU (LMH) module to capture hierarchical item dependencies; and (4) the Embedding Cache (EC) module to bridge efficiency and effectiveness during inference. By sharing representations across ranking and reranking, RIA enables rich contextual knowledge transfer while maintaining low latency. Extensive experiments show that RIA outperforms state-of-the-art models on both public and industrial datasets, achieving significant gains in AUC and LogLoss. Deployed in Meituan advertising system, RIA yields a +1.69% improvement in Click-Through Rate (CTR) and a +4.54% increase in Cost Per Mille (CPM) in online A/B tests.
Authors Guoxiao Zhang, Tan Qu, Ao Li, DongLin Ni, Qianlong Xie, Xingxing Wang
Categories Artificial Intelligence
0
Anomaly Detection with Adaptive and Aggressive Rejection for Contaminated Training Data
Title CN 基于自适应与激进拒绝的污染训练数据异常检测
Keywords 异常检测 · 数据污染 · 鲁棒训练 · Z-score · 高斯混合模型
Summary 本文提出一种名为自适应与激进拒绝（AAR）的方法，用于在训练数据被异常样本污染的情况下提升异常检测性能。该方法结合改进的z-score和高斯混合模型动态设定阈值，通过硬拒绝与软拒绝策略平衡正常样本保留与异常样本剔除。实验表明其在图像和表格数据集上优于现有方法。然而，该工作不涉及多模态理解、视觉-文本对齐或文档结构建模，与文档图像理解任务无关。
Reason 该论文研究的是异常检测中的训练数据污染问题，属于通用AI鲁棒性范畴，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其方法聚焦于通过统计阈值动态剔除异常样本，无法迁移解决OCR幻觉、复杂排版解析、细粒度文本定位等DIU核心痛点。
Abstract Handling contaminated data poses a critical challenge in anomaly detection, as traditional models assume training on purely normal data. Conventional methods mitigate contamination by relying on fixed contamination ratios, but discrepancies between assumed and actual ratios can severely degrade performance, especially in noisy environments where normal and abnormal data distributions overlap. To address these limitations, we propose Adaptive and Aggressive Rejection (AAR), a novel method that dynamically excludes anomalies using a modified z-score and Gaussian mixture model-based thresholds. AAR effectively balances the trade-off between preserving normal data and excluding anomalies by integrating hard and soft rejection strategies. Extensive experiments on two image datasets and thirty tabular datasets demonstrate that AAR outperforms the state-of-the-art method by 0.041 AUROC. By providing a scalable and reliable solution, AAR enhances robustness against contaminated datasets, paving the way for broader real-world applications in domains such as security and healthcare.
Authors Jungi Lee, Jungkwon Kim, Chi Zhang, Kwangsun Yoo, Seok-Joo Byun
Categories Artificial Intelligence
0
Data Valuation by Fusing Global and Local Statistical Information
Title CN 通过融合全局与局部统计信息进行数据估值
Keywords 数据估值 · Shapley值 · 正则化 · 动态更新 · 统计分布
Summary 本文提出一种融合全局与局部统计信息的数据估值方法，通过引入两个正则化项来改进Shapley值的估计，并设计了一种无需重新计算即可动态更新数据价值的方法。实验验证了其在多种数据操作任务中的有效性与效率。
Reason 该论文研究的是机器学习中的数据估值问题，核心围绕Shapley值的估计优化，属于模型训练数据选择或清洗的通用方法论。尽管其技术可能在广义上用于任何依赖数据质量评估的场景，但并未针对文档图像理解（DIU）中的具体任务如DocVQA、OCR、版面分析等提出改进，也未涉及多模态建模、视觉表示、高分辨率处理或幻觉抑制等DIU关键痛点。无法直接迁移为解决文档理解中结构复杂性、细粒度定位或视觉语言对齐的工具。
Abstract Data valuation has garnered increasing attention in recent years, given the critical role of high-quality data in various applications. Among diverse data valuation approaches, Shapley value-based methods are predominant due to their strong theoretical grounding. However, the exact computation of Shapley values is often computationally prohibitive, prompting the development of numerous approximation techniques. Despite notable advancements, existing methods generally neglect the incorporation of value distribution information and fail to account for dynamic data conditions, thereby compromising their performance and application potential. In this paper, we highlight the crucial role of both global and local statistical properties of value distributions in the context of data valuation for machine learning. First, we conduct a comprehensive analysis of these distributions across various simulated and real-world datasets, uncovering valuable insights and key patterns. Second, we propose an enhanced data valuation method that fuses the explored distribution characteristics into two regularization terms to refine Shapley value estimation. The proposed regularizers can be seamlessly incorporated into various existing data valuation methods. Third, we introduce a novel approach for dynamic data valuation that infers updated data values without recomputing Shapley values, thereby significantly improving computational efficiency. Extensive experiments have been conducted across a range of tasks, including Shapley value estimation, value-based data addition and removal, mislabeled data detection, and dynamic data valuation. The results showcase the consistent effectiveness and efficiency of our proposed methodologies, affirming the significant potential of global and local value distributions in data valuation.
Authors Xiaoling Zhou, Ou Wu, Michael K. Ng, Hao Jiang
Comment 35 pages, 9 figures
Categories Artificial Intelligence
0
Hybrid-AIRL: Enhancing Inverse Reinforcement Learning with Supervised Expert Guidance
Title CN Hybrid-AIRL：通过监督专家指导增强逆向强化学习
Keywords 逆向强化学习 · AIRL · 监督学习 · 稀疏奖励 · 专家示范
Summary 本文提出了Hybrid-AIRL（H-AIRL），一种结合监督损失和随机正则化的逆向强化学习方法，用于在不完美信息环境下（如限注德州扑克）提升奖励函数的学习效果。实验表明该方法相比原始AIRL具有更高的样本效率和训练稳定性。研究主要面向强化学习中的稀疏奖励问题，应用背景为游戏决策场景。
Reason 该论文研究的是逆向强化学习在扑克游戏和Gymnasium环境中的应用，属于强化学习领域的特定算法改进，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。其方法聚焦于从专家示范中推断奖励函数，并未涉及OCR、文档布局分析、表格识别、多模态对齐或视觉幻觉等DIU核心问题。尽管‘监督信号引入’在理论上可能泛化，但该工作未提供可迁移至文档理解任务的视觉表示、高分辨率处理、细粒度定位或去幻觉机制，且应用场景完全偏离文档领域。
Abstract Adversarial Inverse Reinforcement Learning (AIRL) has shown promise in addressing the sparse reward problem in reinforcement learning (RL) by inferring dense reward functions from expert demonstrations. However, its performance in highly complex, imperfect-information settings remains largely unexplored. To explore this gap, we evaluate AIRL in the context of Heads-Up Limit Hold'em (HULHE) poker, a domain characterized by sparse, delayed rewards and significant uncertainty. In this setting, we find that AIRL struggles to infer a sufficiently informative reward function. To overcome this limitation, we contribute Hybrid-AIRL (H-AIRL), an extension that enhances reward inference and policy learning by incorporating a supervised loss derived from expert data and a stochastic regularization mechanism. We evaluate H-AIRL on a carefully selected set of Gymnasium benchmarks and the HULHE poker setting. Additionally, we analyze the learned reward function through visualization to gain deeper insights into the learning process. Our experimental results show that H-AIRL achieves higher sample efficiency and more stable learning compared to AIRL. This highlights the benefits of incorporating supervised signals into inverse RL and establishes H-AIRL as a promising framework for tackling challenging, real-world settings.
Authors Bram Silue, Santiago Amaya-Corredor, Patrick Mannion, Lander Willem, Pieter Libin
Comment Comments: 13 pages, 5 figures, 1 table. Code: https://github.com/silue-dev/hairl. Submitted to ESANN 2026
Categories Artificial Intelligence
0
Generating Separated Singing Vocals Using a Diffusion Model Conditioned on Music Mixtures WASPAA 2025
Title CN 使用基于音乐混合物条件的扩散模型生成分离的歌唱人声
Keywords 语音分离 · 扩散模型 · 音乐信息检索 · 生成模型 · WASPAA
Summary 本文提出了一种基于扩散模型的方法，通过以音乐混合物为条件来生成独唱人声，实现歌唱人声与背景音乐的分离。该方法利用扩散模型的迭代采样特性，允许用户控制质量与效率之间的权衡，并可在需要时优化输出结果。实验表明，在补充数据训练下，该方法优于先前的生成式系统，并在客观指标上可与非生成式基线方法相媲美。
Reason 该论文属于音频信号处理领域，研究的是音乐混合物中的人声分离任务，与文档图像理解（DIU）或视觉-语言模型在文档上的应用完全无关。尽管使用了扩散模型这一生成式架构，但其方法和应用场景无法迁移到解决OCR幻觉、复杂排版理解、细粒度定位等DIU核心痛点。
Abstract Separating the individual elements in a musical mixture is an essential process for music analysis and practice. While this is generally addressed using neural networks optimized to mask or transform the time-frequency representation of a mixture to extract the target sources, the flexibility and generalization capabilities of generative diffusion models are giving rise to a novel class of solutions for this complicated task. In this work, we explore singing voice separation from real music recordings using a diffusion model which is trained to generate the solo vocals conditioned on the corresponding mixture. Our approach improves upon prior generative systems and achieves competitive objective scores against non-generative baselines when trained with supplementary data. The iterative nature of diffusion sampling enables the user to control the quality-efficiency trade-off, and also refine the output when needed. We present an ablation study of the sampling algorithm, highlighting the effects of the user-configurable parameters.
Authors Genís Plaja-Roglans, Yun-Ning Hung, Xavier Serra, Igor Pereira
Comment Accepted for publication at WASPAA 2025
Categories Artificial Intelligence
0
SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understanding
Title CN SurgMLLMBench：用于手术场景理解的多模态大语言模型基准数据集
Keywords 多模态大语言模型 · 手术场景理解 · 视觉问答 · 像素级分割 · 基准数据集
Summary 本文提出了SurgMLLMBench，一个针对手术场景理解的多模态大语言模型基准数据集，包含像素级器械分割掩码和结构化视觉问答标注，覆盖腹腔镜、机器人辅助和显微手术等多种类型。目标是推动交互式手术AI模型的发展，支持跨域一致评估与泛化能力测试。然而，该工作聚焦于医疗视觉理解这一特定应用场景，不具备文档图像理解领域的通用方法价值。
Reason 该论文属于特定垂直领域（医疗手术）的下游应用，构建的是面向手术场景的理解数据集与基准，本质上是将现有VLM技术应用于外科视觉问答与分割任务。尽管引入了像素级标注和统一分类体系，但其核心贡献在于领域数据构建而非文档图像理解的方法论创新。未解决OCR幻觉、复杂排版、长文档推理等DIU核心问题，也无法直接迁移为提升DIU性能的通用技术工具。
Abstract Recent advances in multimodal large language models (LLMs) have highlighted their potential for medical and surgical applications. However, existing surgical datasets predominantly adopt a Visual Question Answering (VQA) format with heterogeneous taxonomies and lack support for pixel-level segmentation, limiting consistent evaluation and applicability. We present SurgMLLMBench, a unified multimodal benchmark explicitly designed for developing and evaluating interactive multimodal LLMs for surgical scene understanding, including the newly collected Micro-surgical Artificial Vascular anastomosIS (MAVIS) dataset. It integrates pixel-level instrument segmentation masks and structured VQA annotations across laparoscopic, robot-assisted, and micro-surgical domains under a unified taxonomy, enabling comprehensive evaluation beyond traditional VQA tasks and richer visual-conversational interactions. Extensive baseline experiments show that a single model trained on SurgMLLMBench achieves consistent performance across domains and generalizes effectively to unseen datasets. SurgMLLMBench will be publicly released as a robust resource to advance multimodal surgical AI research, supporting reproducible evaluation and development of interactive surgical reasoning models.
Authors Tae-Min Choi, Tae Kyeong Jeong, Garam Kim, Jaemin Lee, Yeongyoon Koh, In Cheul Choi, Jae-Ho Chung, Jong Woong Park, Juyoun Park
Comment 10 pages, 5 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Hybrid SIFT-SNN for Efficient Anomaly Detection of Traffic Flow-Control Infrastructure 2025 International Conference on Image and Vision Computing New Zealand (IVCNZ)
Title CN 用于交通流控制基础设施异常检测的混合SIFT-SNN方法
Keywords SIFT · Spiking Neural Network · Anomaly Detection · Edge Computing · Infrastructure Monitoring
Summary 本文提出了一种结合SIFT与脉冲神经网络（SNN）的低延迟神经形态信号处理流水线，用于实时检测交通基础设施中的结构异常。在奥克兰海港大桥数据集上验证，实现了92.3%的准确率和9.5ms的单帧推理时间，具备稀疏激活特性，适合边缘部署。系统强调空间特征保持和可解释性，但尚未验证在未知现场条件下的泛化能力。
Reason 该论文属于纯计算机视觉与边缘信号处理领域，聚焦于交通基础设施的结构异常检测，使用SIFT与脉冲神经网络（SNN）进行低延迟分类。其任务与文档图像理解（DIU）无关，未涉及文本识别、布局分析、表格理解或任何VLM在文档上的应用。尽管提到了空间特征对齐，但其方法面向的是结构化物体的异常检测，无法迁移到OCR幻觉、密集文本理解或复杂排版解析等DIU核心痛点。
Abstract This paper presents the SIFT-SNN framework, a low-latency neuromorphic signal-processing pipeline for real-time detection of structural anomalies in transport infrastructure. The proposed approach integrates Scale-Invariant Feature Transform (SIFT) for spatial feature encoding with a latency-driven spike conversion layer and a Leaky Integrate-and-Fire (LIF) Spiking Neural Network (SNN) for classification. The Auckland Harbour Bridge dataset is recorded under various weather and lighting conditions, comprising 6,000 labelled frames that include both real and synthetically augmented unsafe cases. The presented system achieves a classification accuracy of 92.3% (+- 0.8%) with a per-frame inference time of 9.5 ms. Achieved sub-10 millisecond latency, combined with sparse spike activity (8.1%), enables real-time, low-power edge deployment. Unlike conventional CNN-based approaches, the hybrid SIFT-SNN pipeline explicitly preserves spatial feature grounding, enhances interpretability, supports transparent decision-making, and operates efficiently on embedded hardware. Although synthetic augmentation improved robustness, generalisation to unseen field conditions remains to be validated. The SIFT-SNN framework is validated through a working prototype deployed on a consumer-grade system and framed as a generalisable case study in structural safety monitoring for movable concrete barriers, which, as a traffic flow-control infrastructure, is deployed in over 20 cities worldwide.
Authors Munish Rathee, Boris Bačić, Maryam Doborjeh
Comment 8 pages, 6 figures. This is a preprint of a paper accepted for presentation at the 2025 International Conference on Image and Vision Computing New Zealand (IVCNZ). The final version will appear in IEEE Xplore
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Learning Individual Behavior in Agent-Based Models with Graph Diffusion Networks
Title CN 使用图扩散网络学习基于代理模型中的个体行为
Keywords Agent-Based Models · Graph Neural Networks · Diffusion Models · Behavior Learning · Complex Systems
Summary 本文提出一种结合扩散模型和图神经网络的方法，用于学习基于代理模型（ABM）中个体代理的行为，通过观测生成数据构建可微分的代理模型替代系统，从而实现对复杂系统中个体行为和群体涌现现象的建模。该方法强调对个体行为的直接建模，保留了ABM的去中心化特性，并在Schelling隔离模型和捕食者-猎物生态系统中验证了有效性。
Reason 该论文研究的是基于代理的模型（ABM）中个体行为的学习，属于具身智能与复杂系统模拟领域，与文档图像理解（DIU）或视觉语言多模态任务无直接关联。其方法虽结合了图神经网络与扩散模型，但目标是模拟代理在生态系统或社会系统中的行为动态，并非解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心问题。无法迁移用于提升文档理解性能。
Abstract Agent-Based Models (ABMs) are powerful tools for studying emergent properties in complex systems. In ABMs, agent behaviors are governed by local interactions and stochastic rules. However, these rules are, in general, non-differentiable, limiting the use of gradient-based methods for optimization, and thus integration with real-world data. We propose a novel framework to learn a differentiable surrogate of any ABM by observing its generated data. Our method combines diffusion models to capture behavioral stochasticity and graph neural networks to model agent interactions. Distinct from prior surrogate approaches, our method introduces a fundamental shift: rather than approximating system-level outputs, it models individual agent behavior directly, preserving the decentralized, bottom-up dynamics that define ABMs. We validate our approach on two ABMs (Schelling's segregation model and a Predator-Prey ecosystem) showing that it replicates individual-level patterns and accurately forecasts emergent dynamics beyond training. Our results demonstrate the potential of combining diffusion models and graph learning for data-driven ABM simulation.
Authors Francesco Cozzi, Marco Pangallo, Alan Perotti, André Panisson, Corrado Monti
Categories Artificial Intelligence
0
SONAR: Spectral-Contrastive Audio Residuals for Generalizable Deepfake Detection
Title CN SONAR：用于可泛化深度伪造检测的频谱对比音频残差
Keywords 音频深伪检测 · 频谱对比学习 · 高频残差 · 表示学习 · 对比损失
Summary 本文提出了一种名为SONAR的音频深伪检测框架，通过频谱对比学习显式分离音频信号中的低频内容与高频残差。利用XLSR编码器捕捉低频结构，并通过可学习的高通滤波器提取微弱的高频残差，再通过跨频率注意力融合两种表征。配合频率感知的对比损失函数，该方法在ASVspoof 2021和真实场景数据集上实现了最先进的性能，并显著加快了收敛速度。由于完全在表示层面操作，该方案具有架构无关性，未来或可推广到其他依赖细微高频线索的模态。
Reason 该论文研究的是音频深伪检测任务，属于音频信号处理领域，与文档图像理解（DIU）或视觉-语言模型（VLM）在文档上的应用无关。尽管其提出的频谱对比学习框架可能在高阶特征解耦上有方法论启发，但其核心关注点是音频高频残差的建模，无法直接迁移至文档图像中的OCR幻觉、复杂排版或细粒度定位等关键问题。
Abstract Deepfake (DF) audio detectors still struggle to generalize to out of distribution inputs. A central reason is spectral bias, the tendency of neural networks to learn low-frequency structure before high-frequency (HF) details, which both causes DF generators to leave HF artifacts and leaves those same artifacts under-exploited by common detectors. To address this gap, we propose Spectral-cONtrastive Audio Residuals (SONAR), a frequency-guided framework that explicitly disentangles an audio signal into complementary representations. An XLSR encoder captures the dominant low-frequency content, while the same cloned path, preceded by learnable SRM, value-constrained high-pass filters, distills faint HF residuals. Frequency cross-attention reunites the two views for long- and short-range frequency dependencies, and a frequency-aware Jensen-Shannon contrastive loss pulls real content-noise pairs together while pushing fake embeddings apart, accelerating optimization and sharpening decision boundaries. Evaluated on the ASVspoof 2021 and in-the-wild benchmarks, SONAR attains state-of-the-art performance and converges four times faster than strong baselines. By elevating faint high-frequency residuals to first-class learning signals, SONAR unveils a fully data-driven, frequency-guided contrastive framework that splits the latent space into two disjoint manifolds: natural-HF for genuine audio and distorted-HF for synthetic audio, thereby sharpening decision boundaries. Because the scheme operates purely at the representation level, it is architecture-agnostic and, in future work, can be seamlessly integrated into any model or modality where subtle high-frequency cues are decisive.
Authors Ido Nitzan HIdekel, Gal lifshitz, Khen Cohen, Dan Raviv
Categories Artificial Intelligence
0
TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories
Title CN TALES：大语言模型生成故事中文化表征的分类与分析
Keywords 文化表征 · 大语言模型 · 偏见评估 · TALES-Tax · TALES-QA
Summary 本文提出了TALES框架，用于评估大语言模型在生成关于印度文化的故事时存在的文化误表征问题。作者首先通过焦点小组和调查构建了一个文化误表征分类法TALES-Tax，随后对六种模型生成的2925个故事进行了大规模人工标注，发现88%的故事包含至少一种文化错误，尤其在资源较少的语言和城乡结合部背景中更为严重。此外，作者将标注数据转化为TALES-QA题库，用于测试基础模型的文化知识掌握情况，并发现模型虽具备相关知识但仍频繁生成错误内容。
Reason 该论文聚焦于评估大语言模型在生成故事时对印度多元文化的表征偏差，属于社会计算与AI伦理范畴。其核心任务为文化偏见检测与分类体系构建，不涉及文档图像理解（DIU）、视觉-语言建模、OCR、布局分析或任何多模态文档处理技术。尽管使用了LLM生成内容并进行QA构建，但未提出可用于解决DIU领域痛点（如OCR幻觉、复杂排版、细粒度定位等）的方法论或架构创新，属于平行应用类研究。
Abstract Millions of users across the globe turn to AI chatbots for their creative needs, inviting widespread interest in understanding how such chatbots represent diverse cultures. At the same time, evaluating cultural representations in open-ended tasks remains challenging and underexplored. In this work, we present TALES, an evaluation of cultural misrepresentations in LLM-generated stories for diverse Indian cultural identities. First, we develop TALES-Tax, a taxonomy of cultural misrepresentations by collating insights from participants with lived experiences in India through focus groups (N=9) and individual surveys (N=15). Using TALES-Tax, we evaluate 6 models through a large-scale annotation study spanning 2,925 annotations from 108 annotators with lived cultural experience from across 71 regions in India and 14 languages. Concerningly, we find that 88\% of the generated stories contain one or more cultural inaccuracies, and such errors are more prevalent in mid- and low-resourced languages and stories based in peri-urban regions in India. Lastly, we transform the annotations into TALES-QA, a standalone question bank to evaluate the cultural knowledge of foundational models. Through this evaluation, we surprisingly discover that models often possess the requisite cultural knowledge despite generating stories rife with cultural misrepresentations.
Authors Kirti Bhagat, Shaily Bhatt, Athul Velagapudi, Aditya Vashistha, Shachi Dave, Danish Pruthi
Categories Artificial Intelligence, Computation and Language
0
Augur: Modeling Covariate Causal Associations in Time Series via Large Language Models
Title CN Augur：通过大语言模型建模时间序列中的协变量因果关联
Keywords 时间序列预测 · 因果推理 · 大语言模型 · 教师-学生架构 · 可解释性
Summary 本文提出Augur，一种基于大语言模型的时间序列预测框架，利用LLM的因果推理能力挖掘协变量之间的有向因果关系。通过教师模型生成因果图，学生模型对其进行优化并基于高置信度因果关系进行预测。实验表明该方法在多个真实数据集上具有良好的预测性能和零样本泛化能力。
Reason 该论文研究的是时间序列预测中的因果推理问题，属于纯时序分析领域，与文档图像理解（DIU）或视觉-语言多模态任务无直接关联。尽管使用了LLM进行因果发现和推理，但其方法框架针对的是数值型时间序列数据，并未涉及图像输入、布局理解、OCR、视觉定位或文档结构建模等DIU核心问题。所提出的‘教师-学生’因果图构建机制难以迁移到文档理解场景中解决OCR幻觉、复杂排版解析或细粒度图文对齐等关键挑战。
Abstract Large language models (LLM) have emerged as a promising avenue for time series forecasting, offering the potential to integrate multimodal data. However, existing LLM-based approaches face notable limitations-such as marginalized role in model architectures, reliance on coarse statistical text prompts, and lack of interpretability. In this work, we introduce Augur, a fully LLM driven time series forecasting framework that exploits LLM causal reasoning to discover and use directed causal associations among covariates. Augur uses a two stage teacher student architecture where a powerful teacher LLM infers a directed causal graph from time series using heuristic search together with pairwise causality testing. A lightweight student agent then refines the graph and fine tune on high confidence causal associations that are encoded as rich textual prompts to perform forecasting. This design improves predictive accuracy while yielding transparent, traceable reasoning about variable interactions. Extensive experiments on real-world datasets with 26 baselines demonstrate that Augur achieves competitive performance and robust zero-shot generalization.
Authors Zhiqing Cui, Binwu Wang, Qingxiang Liu, Yeqiang Wang, Zhengyang Zhou, Yuxuan Liang, Yang Wang
Comment 24 pages, 9 figures
Categories Artificial Intelligence
0
Without Paired Labeled Data: End-to-End Self-Supervised Learning for Drone-view Geo-Localization
Title CN 无需配对标注数据：面向无人机视角地理定位的端到端自监督学习
Keywords 自监督学习 · 跨视角匹配 · 伪标签 · 对比学习 · 无人机地理定位
Summary 本文提出了一种名为DMNIL的端到端自监督学习方法，用于无人机视角地理定位任务。该方法通过聚类生成伪标签，并设计了双路径对比学习框架，结合动态分层记忆学习（DHML）和信息一致性演化学习（ICEL）模块，提升跨视角特征对齐能力。实验表明其在多个基准上优于现有自监督甚至部分有监督方法。
Reason 该论文研究的是无人机视角与卫星图像之间的跨视角地理定位问题，属于遥感与定位领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。尽管其采用了自监督学习和对比学习等通用视觉方法，但其核心目标是解决跨视角图像匹配中的特征对齐问题，而非文档中的文本识别、布局分析或视觉语言推理。所提出的方法难以迁移到解决OCR幻觉、密集文本理解或复杂排版解析等DIU核心痛点。
Abstract Drone-view Geo-Localization (DVGL) aims to achieve accurate localization of drones by retrieving the most relevant GPS-tagged satellite images. However, most existing methods heavily rely on strictly pre-paired drone-satellite images for supervised learning. When the target region shifts, new paired samples are typically required to adapt to the distribution changes. The high cost of annotation and the limited transferability of these methods significantly hinder the practical deployment of DVGL in open-world scenarios. To address these limitations, we propose a novel end-to-end self-supervised learning method with a shallow backbone network, called the dynamic memory-driven and neighborhood information learning (DMNIL) method. It employs a clustering algorithm to generate pseudo-labels and adopts a dual-path contrastive learning framework to learn discriminative intra-view representations. Furthermore, DMNIL incorporates two core modules, including the dynamic hierarchical memory learning (DHML) module and the information consistency evolution learning (ICEL) module. The DHML module combines short-term and long-term memory to enhance intra-view feature consistency and discriminability. Meanwhile, the ICEL module utilizes a neighborhood-driven dynamic constraint mechanism to systematically capture implicit cross-view semantic correlations, consequently improving cross-view feature alignment. To further stabilize and strengthen the self-supervised training process, a pseudo-label enhancement strategy is introduced to enhance the quality of pseudo supervision. Extensive experiments on three public benchmark datasets demonstrate that the proposed method consistently outperforms existing self-supervised methods and even surpasses several state-of-the-art supervised methods. Our code is available at https://github.com/ISChenawei/DMNIL.
Authors Zhongwei Chen, Zhao-Xu Yang, Hai-Jun Rong, Guoqi Li
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Improvement of Collision Avoidance in Cut-In Maneuvers Using Time-to-Collision Metrics
Title CN 利用碰撞时间度量改进切入机动中的避碰
Keywords 自动驾驶 · 碰撞避免 · TTC · 深度学习 · 切入场景
Summary 本文提出了一种基于时间到碰撞（TTC）指标的避碰策略，用于自动驾驶车辆应对其他车辆突然切入的复杂场景。通过将深度学习与TTC计算相结合，系统能够预测潜在碰撞并采取规避动作，相比传统方法更具前瞻性。
Reason 该论文属于自动驾驶领域，研究的是车辆在切入场景下的避碰策略，使用TTC与深度学习结合进行碰撞预测。完全不涉及文档图像理解、多模态语言模型、OCR或任何DIU相关任务。属于无关领域（自动驾驶），命中负面清单第2条。
Abstract This paper proposes a new strategy for collision avoidance system leveraging Time-to-Collision (TTC) metrics for handling cut-in scenarios, which are particularly challenging for autonomous vehicles (AVs). By integrating a deep learning with TTC calculations, the system predicts potential collisions and determines appropriate evasive actions compared to traditional TTC -based approaches.
Authors Jamal Raiyn
Categories Artificial Intelligence
0
Reasoning Transfer for an Extremely Low-Resource and Endangered Language: Bridging Languages Through Sample-Efficient Language Understanding
Title CN 通过样本高效的语言理解实现极度低资源濒危语言的推理迁移：以英语为枢纽的链式思维训练
Keywords 低资源语言 · 链式思维 · 跨语言推理 · 大语言模型 · 数学推理
Summary 本文提出了一种名为English-Pivoted CoT Training的方法，用于提升大语言模型在极度低资源语言（如爱尔兰语）上的数学推理能力。该方法通过监督微调让模型将低资源语言输入转化为英文链式思维推理过程，并最终生成目标语言的答案。实验表明该方法在多个基准上显著优于传统提示和微调方法，并发布了首个面向爱尔兰语的数学推理评测集LC2024。
Reason 该论文聚焦于低资源语言（爱尔兰语）上的跨语言推理问题，属于纯NLP范畴，未涉及任何视觉或文档模态。其核心方法是利用英语作为中间语言进行链式思维（CoT）训练，解决的是语言障碍而非文档图像理解中的OCR幻觉、复杂排版或多模态对齐等问题。尽管CoT本身是推理增强技术，但本文并无视觉输入处理机制，也无法迁移到文档VQA或布局分析等任务中。
Abstract Recent advances have enabled Large Language Models (LLMs) to tackle reasoning tasks by generating chain-of-thought (CoT) rationales, yet these gains have largely applied to high-resource languages, leaving low-resource languages behind. In this work, we first investigate CoT techniques in extremely low-resource scenarios through previous prompting, model-editing, and fine-tuning approaches. We introduce English-Pivoted CoT Training, leveraging the insight that LLMs internally operate in a latent space aligned toward the dominant language. Given input in a low-resource language, we perform supervised fine-tuning to generate CoT in English and output the final response in the target language. Across mathematical reasoning benchmarks, our approach outperforms other baselines with up to 28.33% improvement in low-resource scenarios. Our analysis and additional experiments, including Mixed-Language CoT and Two-Stage Training, show that explicitly separating language understanding from reasoning enhances cross-lingual reasoning abilities. To facilitate future work, we also release \emph{LC2024}, the first benchmark for mathematical tasks in Irish, an extremely low-resource and endangered language. Our results and resources highlight a practical pathway to multilingual reasoning without extensive retraining in every extremely low-resource language, despite data scarcity.
Authors Khanh-Tung Tran, Barry O'Sullivan, Hoang D. Nguyen
Categories Artificial Intelligence, Computation and Language
0
CLLMRec: LLM-powered Cognitive-Aware Concept Recommendation via Semantic Alignment and Prerequisite Knowledge Distillation
Title CN CLLMRec：通过语义对齐与前置知识蒸馏实现大语言模型驱动的认知感知概念推荐
Keywords 个性化推荐 · 大语言模型 · 知识蒸馏 · 语义对齐 · 认知建模
Summary 本文提出CLLMRec，一种基于大语言模型的个性化概念推荐框架，用于MOOC场景下的学习者推荐。通过语义对齐将学习者与概念映射到统一语义空间，并利用教师-学生架构从大模型中蒸馏前置知识关系，结合细粒度排序机制建模学习者的认知状态。实验表明该方法在真实数据集上优于现有基线。
Reason 该论文属于教育领域的个性化学习推荐系统，核心任务是MOOC课程中的概念推荐，使用LLM进行知识蒸馏和语义对齐。尽管使用了LLM和知识蒸馏技术，但其应用场景为典型的‘平行下游应用’，未涉及文档图像理解（DIU）、DocVQA、OCR、版面分析或任何视觉-语言文档建模任务。方法论聚焦于认知建模与推荐排序，无法迁移解决DIU中的OCR幻觉、复杂排版、高分辨率处理等核心痛点。
Abstract The growth of Massive Open Online Courses (MOOCs) presents significant challenges for personalized learning, where concept recommendation is crucial. Existing approaches typically rely on heterogeneous information networks or knowledge graphs to capture conceptual relationships, combined with knowledge tracing models to assess learners' cognitive states. However, these methods face significant limitations due to their dependence on high-quality structured knowledge graphs, which are often scarce in real-world educational scenarios. To address this fundamental challenge, this paper proposes CLLMRec, a novel framework that leverages Large Language Models through two synergistic technical pillars: Semantic Alignment and Prerequisite Knowledge Distillation. The Semantic Alignment component constructs a unified representation space by encoding unstructured textual descriptions of learners and concepts. The Prerequisite Knowledge Distillation paradigm employs a teacher-student architecture, where a large teacher LLM (implemented as the Prior Knowledge Aware Component) extracts conceptual prerequisite relationships from its internalized world knowledge and distills them into soft labels to train an efficient student ranker. Building upon these foundations, our framework incorporates a fine-ranking mechanism that explicitly models learners' real-time cognitive states through deep knowledge tracing, ensuring recommendations are both structurally sound and cognitively appropriate. Extensive experiments on two real-world MOOC datasets demonstrate that CLLMRec significantly outperforms existing baseline methods across multiple evaluation metrics, validating its effectiveness in generating truly cognitive-aware and personalized concept recommendations without relying on explicit structural priors.
Authors Xiangrui Xiong, Yichuan Lu, Zifei Pan, Chang Sun
Categories Artificial Intelligence
0
SARVLM: A Vision Language Foundation Model for Semantic Understanding and Target Recognition in SAR Imagery
Title CN SARVLM：用于SAR图像语义理解与目标识别的视觉语言基础模型
Keywords SAR图像 · 视觉语言模型 · 零样本识别 · 领域迁移 · 遥感图像理解
Summary 本文提出了SARVLM，首个面向合成孔径雷达（SAR）图像的视觉语言基础模型，通过构建大规模SAR图文对数据集SARVLM-1M，并采用领域迁移训练策略，在图像-文本检索、零样本分类、语义定位和图像描述等任务上取得了优于现有方法的表现。模型由SARCLIP和SARCapse两部分组成，旨在提升SAR图像的语义理解能力。
Reason 该论文研究的是合成孔径雷达（SAR）图像的视觉语言建模，属于遥感成像领域，尽管使用了VLM框架，但其数据模态和应用场景与文档图像理解（DIU）差异巨大。SAR图像的本质是微波反射信号形成的灰度纹理，与文档中的文本布局、字体、排版等结构信息无共通性，所提出的方法难以迁移到解决OCR幻觉、复杂版面分析或长文档推理等DIU核心问题。
Abstract Synthetic Aperture Radar (SAR) is a crucial imaging modality thanks to its all-weather capability. Although recent advances in self-supervised learning and masked image modeling (MIM) have enabled SAR foundation models, these methods largely emphasize low-level visual features and often overlook multimodal alignment and zero-shot target recognition in SAR imagery. To address this, we construct SARVLM-1M, a large-scale vision-language dataset with over one million image-text pairs aggregated from existing datasets. We further propose a domain transfer training strategy to mitigate the large gap between natural and SAR imagery. Building on this, we develop SARVLM, the first vision language foundation model (VLM) tailored to SAR, comprising SARCLIP and SARCap. SARVLM is trained with a vision-language contrastive objective under the proposed domain transfer strategy, bridging SAR imagery and textual descriptions. Extensive experiments on image text retrieval, zero-shot classification, semantic localization, and imagery captioning demonstrate that SARVLM delivers superior feature extraction and interpretation, outperforming state-of-the-art VLMs and advancing SAR semantic understanding. Code and datasets will be released soon.
Authors Qiwei Ma, Zhiyu Wang, Wang Liu, Xukun Lu, Bin Deng, Puhong Duan, Xudong Kang, Shutao Li
Comment 11 pages, 9 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
PaTAS: A Parallel System for Trust Propagation in Neural Networks Using Subjective Logic
Title CN PaTAS：一种基于主观逻辑的神经网络信任传播并行系统
Keywords 信任传播 · 主观逻辑 · 神经网络可靠性 · AI安全 · 可信AI
Summary 本文提出PaTAS，一个基于主观逻辑的并行信任评估系统，通过引入信任节点和信任函数，在神经网络中并行传播输入、参数和激活的信任值。该框架包含训练阶段的参数信任更新机制和推理阶段的路径信任评估方法，能够在实例级别量化模型预测的可靠性。实验表明，PaTAS能有效识别被投毒、偏置或不确定的数据场景，并揭示模型置信度与实际可靠性之间的差距。然而，该工作不涉及视觉或文本模态的联合理解，也不针对文档图像任务设计。
Reason 该论文研究的是神经网络中的信任建模与传播，属于AI可信赖性（trustworthiness）和安全评估范畴，关注模型在对抗或退化条件下的可靠性评估。其核心方法基于主观逻辑进行信任推理，并未涉及多模态输入、视觉表示、文档结构理解或VLM相关技术。尽管‘可靠性评估’在广义上可能与OCR幻觉检测有关联，但该工作缺乏对视觉-语言对齐、高分辨率图像处理或多模态推理的支持机制，无法直接迁移用于解决文档图像理解中的核心问题如细粒度定位、复杂排版解析或长文档推理。属于无关领域的通用AI安全框架。
Abstract Trustworthiness has become a key requirement for the deployment of artificial intelligence systems in safety-critical applications. Conventional evaluation metrics such as accuracy and precision fail to capture uncertainty or the reliability of model predictions, particularly under adversarial or degraded conditions. This paper introduces the Parallel Trust Assessment System (PaTAS), a framework for modeling and propagating trust in neural networks using Subjective Logic (SL). PaTAS operates in parallel with standard neural computation through Trust Nodes and Trust Functions that propagate input, parameter, and activation trust across the network. The framework defines a Parameter Trust Update mechanism to refine parameter reliability during training and an Inference-Path Trust Assessment (IPTA) method to compute instance-specific trust at inference. Experiments on real-world and adversarial datasets demonstrate that PaTAS produces interpretable, symmetric, and convergent trust estimates that complement accuracy and expose reliability gaps in poisoned, biased, or uncertain data scenarios. The results show that PaTAS effectively distinguishes between benign and adversarial inputs and identifies cases where model confidence diverges from actual reliability. By enabling transparent and quantifiable trust reasoning within neural architectures, PaTAS provides a principled foundation for evaluating model reliability across the AI lifecycle.
Authors Koffi Ismael Ouattara, Ioannis Krontiris, Theo Dimitrakos, Dennis Eisermann, Frank Kargl
Categories Artificial Intelligence
0
Self-Guided Defense: Adaptive Safety Alignment for Reasoning Models via Synthesized Guidelines
Title CN 自我引导防御：通过合成准则实现推理模型的自适应安全对齐
Keywords 安全对齐 · 越狱攻击 · 推理模型 · 合成准则 · DPO
Summary 本文提出SGASA框架，通过合成安全准则实现推理模型对对抗性越狱提示的自适应防御。该方法包含数据预合成和对齐微调两个阶段，利用SFT和DPO将模型自生成的安全指南嵌入到模型中，提升安全性的同时减少对良性请求的误拒。实验表明其在多个数据集上有效增强了模型鲁棒性。
Reason 该论文聚焦于推理模型在面对对抗性越狱提示时的安全对齐问题，属于纯NLP安全与对齐（Safety/Jailbreak）领域。尽管涉及‘推理模型’，但其核心目标是内容安全控制，而非解决文档图像理解中的OCR幻觉、复杂排版或细粒度定位等技术痛点。文中提出的SGASA框架用于生成安全准则并进行对齐微调，无法直接迁移至DIU场景中的视觉-语言联合建模或高分辨率文档处理。因此属于无关领域的平行研究。
Abstract Reasoning models have demonstrated remarkable capabilities in complex reasoning tasks. However, ensuring their safety against adversarial jailbreak prompts remains a critical challenge. Due to the covert and deceptive nature of such prompts, they can often evade built-in safety mechanisms and lead to the generation of harmful content. This underscores the need for an adaptive safety alignment approach that enables models to autonomously reinforce their defenses in response to adversarial inputs. This paper introduces the Synthesized Guideline-based Adaptive Safety Alignment (SGASA) framework, which internalizes model-generated safety guidelines to strengthen models' ability to enhance robustness against harmful adversarial prompts while minimizing unnecessary refusals of benign requests. SGASA consists of two key stages: Data Pre-synthesis, which generates safety guidelines and augmented prompts; and Alignment Fine-tuning, which leverages Supervised Fine-tuning (SFT) and Direct Preference Optimization (DPO) to embed these guidelines into the model. Extensive experiments across multiple datasets demonstrate that SGASA significantly improves model safety, validating its adaptive and scalable effectiveness.
Authors Yuhang Wang, Yanxu Zhu, Dongyuan Lu, Jitao Sang
Categories Artificial Intelligence, Computation and Language
0
Earth Observation Satellite Scheduling with Graph Neural Networks and Monte Carlo Tree Search International Workshop on Planning & Scheduling for Space (IWPSS 2025)
Title CN 利用图神经网络和蒙特卡洛树搜索进行地球观测卫星调度
Keywords 图神经网络 · 蒙特卡洛树搜索 · 卫星调度 · 深度强化学习 · 组合优化
Summary 本文提出了一种结合图神经网络（GNN）和深度强化学习（DRL）的方法，用于解决地球观测卫星的观测任务调度问题。通过GNN提取任务图结构特征，DRL指导搜索最优调度策略，并引入基于蒙特卡洛树搜索（MCTS）的后学习搜索步骤以进一步提升性能。实验表明该方法在小规模实例上训练后可泛化至大规模真实场景，性能优于传统方法。
Reason 该论文属于具身智能与运筹优化交叉领域，研究的是卫星任务调度问题，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。尽管使用了GNN和MCTS等方法，但其技术路径针对的是动作序列规划与资源约束优化，难以迁移到文档理解中的OCR幻觉、复杂排版解析或细粒度视觉 grounding 等核心痛点。
Abstract Earth Observation Satellite Planning (EOSP) is a difficult optimization problem with considerable practical interest. A set of requested observations must be scheduled on an agile Earth observation satellite while respecting constraints on their visibility window, as well as maneuver constraints that impose varying delays between successive observations. In addition, the problem is largely oversubscribed: there are much more candidate observations than can possibly be achieved. Therefore, one must select the set of observations that will be performed while maximizing their cumulative benefit and propose a feasible schedule for these observations. As previous work mostly focused on heuristic and iterative search algorithms, this paper presents a new technique for selecting and scheduling observations based on Graph Neural Networks (GNNs) and Deep Reinforcement Learning (DRL). GNNs are used to extract relevant information from the graphs representing instances of the EOSP, and DRL drives the search for optimal schedules. A post-learning search step based on Monte Carlo Tree Search (MCTS) is added that is able to find even better solutions. Experiments show that it is able to learn on small problem instances and generalize to larger real-world instances, with very competitive performance compared to traditional approaches.
Authors Antoine Jacquet, Guillaume Infantes, Emmanuel Benazera, Vincent Baudoui, Jonathan Guerra, Stéphanie Roussel
Comment Accepted at International Workshop on Planning & Scheduling for Space (IWPSS 2025)
Categories Artificial Intelligence
0
BotaCLIP: Contrastive Learning for Botany-Aware Representation of Earth Observation Data
Title CN BotaCLIP：用于地球观测数据植物学感知表示的对比学习
Keywords 对比学习 · 多模态表示 · 遥感图像 · 生态建模 · 领域自适应
Summary 本文提出BotaCLIP，一种轻量级多模态对比学习框架，用于将预训练的地球观测基础模型（DOFA）适配到植物学领域，通过高分辨率航拍图像与植物群落调查数据对齐，实现生态结构感知的表示学习。该方法在植物存在预测、蝴蝶分布建模和土壤营养群丰度估计任务中优于基线模型。研究强调在数据稀缺场景下如何低成本注入领域知识。
Reason 该论文聚焦于生态学领域的遥感图像与植物学数据的跨模态对齐，属于特定垂直领域（生物多样性建模）的应用研究。其方法虽涉及多模态表示学习，但核心目标是注入生态专家知识，而非解决文档图像理解中的共性挑战（如OCR幻觉、复杂版面解析、长文档推理等）。无法迁移至DIU领域的关键痛点，属于无关领域的平行应用。
Abstract Foundation models have demonstrated a remarkable ability to learn rich, transferable representations across diverse modalities such as images, text, and audio. In modern machine learning pipelines, these representations often replace raw data as the primary input for downstream tasks. In this paper, we address the challenge of adapting a pre-trained foundation model to inject domain-specific knowledge, without retraining from scratch or incurring significant computational costs. To this end, we introduce BotaCLIP, a lightweight multimodal contrastive framework that adapts a pre-trained Earth Observation foundation model (DOFA) by aligning high-resolution aerial imagery with botanical relevés. Unlike generic embeddings, BotaCLIP internalizes ecological structure through contrastive learning with a regularization strategy that mitigates catastrophic forgetting. Once trained, the resulting embeddings serve as transferable representations for downstream predictors. Motivated by real-world applications in biodiversity modeling, we evaluated BotaCLIP representations in three ecological tasks: plant presence prediction, butterfly occurrence modeling, and soil trophic group abundance estimation. The results showed consistent improvements over those derived from DOFA and supervised baselines. More broadly, this work illustrates how domain-aware adaptation of foundation models can inject expert knowledge into data-scarce settings, enabling frugal representation learning.
Authors Selene Cerna, Sara Si-Moussi, Wilfried Thuiller, Hadrien Hendrikx, Vincent Miele
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Models
Title CN 当机器人服从补丁时：针对视觉-语言-动作模型的通用可迁移补丁攻击
Keywords 对抗攻击 · 视觉-语言-动作模型 · 可迁移补丁 · 特征空间优化 · 机器人安全
Summary 本文提出UPA-RFAS框架，用于生成针对视觉-语言-动作模型的通用可迁移对抗补丁，通过在特征空间中优化共享表示，并引入注意力主导和语义错位损失来实现跨模型、跨任务的攻击迁移。实验涵盖多种VLA模型和真实机器人场景。
Reason 该论文研究的是对视觉-语言-动作（VLA）模型的对抗性补丁攻击，属于具身智能和机器人领域，主要关注安全性和对抗攻击问题。其核心内容与文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等无直接关联，且方法聚焦于物理世界中的攻击迁移与机器人控制，难以迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等痛点。
Abstract Vision-Language-Action (VLA) models are vulnerable to adversarial attacks, yet universal and transferable attacks remain underexplored, as most existing patches overfit to a single model and fail in black-box settings. To address this gap, we present a systematic study of universal, transferable adversarial patches against VLA-driven robots under unknown architectures, finetuned variants, and sim-to-real shifts. We introduce UPA-RFAS (Universal Patch Attack via Robust Feature, Attention, and Semantics), a unified framework that learns a single physical patch in a shared feature space while promoting cross-model transfer. UPA-RFAS combines (i) a feature-space objective with an $\ell_1$ deviation prior and repulsive InfoNCE loss to induce transferable representation shifts, (ii) a robustness-augmented two-phase min-max procedure where an inner loop learns invisible sample-wise perturbations and an outer loop optimizes the universal patch against this hardened neighborhood, and (iii) two VLA-specific losses: Patch Attention Dominance to hijack text$\to$vision attention and Patch Semantic Misalignment to induce image-text mismatch without labels. Experiments across diverse VLA models, manipulation suites, and physical executions show that UPA-RFAS consistently transfers across models, tasks, and viewpoints, exposing a practical patch-based attack surface and establishing a strong baseline for future defenses.
Authors Hui Lu, Yi Yu, Yiming Yang, Chenyu Yi, Qixin Zhang, Bingquan Shen, Alex C. Kot, Xudong Jiang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Framework 40th IEEE/ACM International Conference on Automated Software Engineering
Title CN 用于Java中自动化单元测试生成与评估的LLM：AgoneTest框架
Keywords LLM · 单元测试生成 · 软件测试 · AgoneTest · Java
Summary 本文提出了AgoneTest框架，用于自动化评估由大语言模型生成的Java单元测试。该框架结合Classes2Test数据集，采用突变分数和测试坏味道等指标进行综合评估。实验表明，在可编译的测试子集中，LLM生成的测试在覆盖率和缺陷检测方面可媲美甚至超过人工编写测试。研究强调提示策略对测试质量的影响，旨在为软件测试领域提供标准化评估工具。
Reason 该论文属于软件工程领域的下游应用，聚焦于使用LLM生成Java单元测试并提出一个评估框架。其核心贡献是面向特定编程语言（Java）的测试评估流程和数据集，未涉及文档图像理解中的任何技术挑战（如OCR幻觉、复杂版式建模、视觉-文本对齐等）。属于典型的‘平行应用’，无DIU方法论创新。
Abstract Unit testing is an essential but resource-intensive step in software development, ensuring individual code units function correctly. This paper introduces AgoneTest, an automated evaluation framework for Large Language Model-generated (LLM) unit tests in Java. AgoneTest does not aim to propose a novel test generation algorithm; rather, it supports researchers and developers in comparing different LLMs and prompting strategies through a standardized end-to-end evaluation pipeline under realistic conditions. We introduce the Classes2Test dataset, which maps Java classes under test to their corresponding test classes, and a framework that integrates advanced evaluation metrics, such as mutation score and test smells, for a comprehensive assessment. Experimental results show that, for the subset of tests that compile, LLM-generated tests can match or exceed human-written tests in terms of coverage and defect detection. Our findings also demonstrate that enhanced prompting strategies contribute to test quality. AgoneTest clarifies the potential of LLMs in software testing and offers insights for future improvements in model design, prompt engineering, and testing practices.
Authors Andrea Lops, Fedelucio Narducci, Azzurra Ragone, Michelantonio Trizio, Claudio Bartolini
Comment Accepted at 40th IEEE/ACM International Conference on Automated Software Engineering
Categories Artificial Intelligence
0
Passive Dementia Screening via Facial Temporal Micro-Dynamics Analysis of In-the-Wild Talking-Head Video
Title CN 通过野外对话视频中面部时序微动态分析进行被动痴呆筛查
Keywords 痴呆筛查 · 面部微动态 · 视频分析 · 行为生物标志物 · 计算机视觉
Summary 本文提出一种基于野外拍摄的自拍式对话视频中面部微动态分析的被动痴呆筛查方法，通过提取眨眼、嘴部运动、凝视变异性等非语言特征构建时间序列模型，在无脚本、跨设备场景下实现早期神经认知变化检测。作者发布了新数据集YT DemTalk，并验证了轻量分类器在该任务上的高性能表现。
Reason 该论文研究的是基于人脸微动态（如眨眼、嘴部运动、凝视变化）的痴呆症被动筛查，属于医疗健康领域的视频行为分析，与文档图像理解（DIU）、DocVQA、OCR、多模态文档建模等核心任务完全无关。尽管使用了计算机视觉和时序建模技术，但其方法针对的是生物医学信号识别，无法迁移用于解决文档中的OCR幻觉、复杂排版或长文档推理等关键问题。
Abstract We target passive dementia screening from short camera-facing talking head video, developing a facial temporal micro dynamics analysis for language free detection of early neuro cognitive change. This enables unscripted, in the wild video analysis at scale to capture natural facial behaviors, transferrable across devices, topics, and cultures without active intervention by clinicians or researchers during recording. Most existing resources prioritize speech or scripted interviews, limiting use outside clinics and coupling predictions to language and transcription. In contrast, we identify and analyze whether temporal facial kinematics, including blink dynamics, small mouth jaw motions, gaze variability, and subtle head adjustments, are sufficient for dementia screening without speech or text. By stabilizing facial signals, we convert these micro movements into interpretable facial microdynamic time series, smooth them, and summarize short windows into compact clip level statistics for screening. Each window is encoded by its activity mix (the relative share of motion across streams), thus the predictor analyzes the distribution of motion across streams rather than its magnitude, making per channel effects transparent. We also introduce YT DemTalk, a new dataset curated from publicly available, in the wild camera facing videos. It contains 300 clips (150 with self reported dementia, 150 controls) to test our model and offer a first benchmarking of the corpus. On YT DemTalk, ablations identify gaze lability and mouth/jaw dynamics as the most informative cues, and light weighted shallow classifiers could attain a dementia prediction performance of (AUROC) 0.953, 0.961 Average Precision (AP), 0.851 F1-score, and 0.857 accuracy.
Authors Filippo Cenacchi, Longbing Cao, Mitchell McEwan, Deborah Richards
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Privacy in Federated Learning with Spiking Neural Networks
Title CN 基于脉冲神经网络的联邦学习中的隐私保护
Keywords 脉冲神经网络 · 联邦学习 · 隐私保护 · 梯度泄露攻击 · 替代梯度
Summary 本文首次系统研究了在联邦学习环境下，脉冲神经网络（SNN）是否存在梯度泄露导致隐私暴露的问题。由于SNN使用替代梯度进行训练且具有事件驱动特性，实验发现其共享梯度难以被用于重构原始输入数据，表现出比传统人工神经网络更强的隐私保护潜力。这为边缘设备上的安全AI提供了新思路。
Reason 该论文研究的是联邦学习中脉冲神经网络（SNN）的梯度泄露问题，属于AI安全与隐私领域，且聚焦于神经形态计算和能量效率场景。虽然涉及SNN和训练安全，但其核心问题与文档图像理解（DIU）、视觉表示、OCR去幻觉、多模态推理等完全无关。无法迁移用于解决文档理解中的排版解析、细粒度定位或VLM在DocVQA中的应用痛点。
Abstract Spiking neural networks (SNNs) have emerged as prominent candidates for embedded and edge AI. Their inherent low power consumption makes them far more efficient than conventional ANNs in scenarios where energy budgets are tightly constrained. In parallel, federated learning (FL) has become the prevailing training paradigm in such settings, enabling on-device learning while limiting the exposure of raw data. However, gradient inversion attacks represent a critical privacy threat in FL, where sensitive training data can be reconstructed directly from shared gradients. While this vulnerability has been widely investigated in conventional ANNs, its implications for SNNs remain largely unexplored. In this work, we present the first comprehensive empirical study of gradient leakage in SNNs across diverse data domains. SNNs are inherently non-differentiable and are typically trained using surrogate gradients, which we hypothesized would be less correlated with the original input and thus less informative from a privacy perspective. To investigate this, we adapt different gradient leakage attacks to the spike domain. Our experiments reveal a striking contrast with conventional ANNs: whereas ANN gradients reliably expose salient input content, SNN gradients yield noisy, temporally inconsistent reconstructions that fail to recover meaningful spatial or temporal structure. These results indicate that the combination of event-driven dynamics and surrogate-gradient training substantially reduces gradient informativeness. To the best of our knowledge, this work provides the first systematic benchmark of gradient inversion attacks for spiking architectures, highlighting the inherent privacy-preserving potential of neuromorphic computation.
Authors Dogukan Aksu, Jesus Martinez del Rincon, Ihsen Alouani
Categories Artificial Intelligence
0
CAHS-Attack: CLIP-Aware Heuristic Search Attack Method for Stable Diffusion
Title CN CAHS-Attack：面向Stable Diffusion的CLIP感知启发式搜索攻击方法
Keywords 对抗攻击 · 文本到图像生成 · CLIP · 蒙特卡洛树搜索 · 提示工程
Summary 本文提出CAHS-Attack，一种基于CLIP感知的启发式搜索方法，用于生成对Stable Diffusion模型有效的对抗性提示。该方法结合蒙特卡洛树搜索与约束遗传算法，在黑盒条件下实现高效提示攻击，揭示了基于CLIP文本编码器的生成系统存在根本性安全风险。
Reason 该论文研究的是针对文本到图像生成模型（如Stable Diffusion）的对抗性攻击，属于图像生成与安全领域。其核心方法是通过启发式搜索优化对抗性提示词，提升攻击效果。尽管使用了MCTS和遗传算法等搜索策略，但其目标场景为生成模型的脆弱性分析，与文档图像理解（DIU）中的OCR、布局分析、表格识别或VQA等任务无直接关联。且属于纯图像生成领域的安全研究，不涉及文档模态、多模态对齐或视觉语言推理中的幻觉、定位等DIU核心痛点，无法迁移用于提升文档理解性能。
Abstract Diffusion models exhibit notable fragility when faced with adversarial prompts, and strengthening attack capabilities is crucial for uncovering such vulnerabilities and building more robust generative systems. Existing works often rely on white-box access to model gradients or hand-crafted prompt engineering, which is infeasible in real-world deployments due to restricted access or poor attack effect. In this paper, we propose CAHS-Attack , a CLIP-Aware Heuristic Search attack method. CAHS-Attack integrates Monte Carlo Tree Search (MCTS) to perform fine-grained suffix optimization, leveraging a constrained genetic algorithm to preselect high-potential adversarial prompts as root nodes, and retaining the most semantically disruptive outcome at each simulation rollout for efficient local search. Extensive experiments demonstrate that our method achieves state-of-the-art attack performance across both short and long prompts of varying semantics. Furthermore, we find that the fragility of SD models can be attributed to the inherent vulnerability of their CLIP-based text encoders, suggesting a fundamental security risk in current text-to-image pipelines.
Authors Shuhan Xia, Jing Dai, Hui Ouyang, Yadong Shang, Dongxiao Zhao, Peipei Li
Categories Artificial Intelligence
0
Maglev-Pentabot: Magnetic Levitation System for Non-Contact Manipulation using Deep Reinforcement Learning
Title CN Maglev-Pentabot：基于深度强化学习的非接触式磁悬浮操控系统
Keywords 磁悬浮 · 非接触操控 · 深度强化学习 · 机器人控制 · 工业自动化
Summary 本文提出了一种名为Maglev-Pentabot的磁悬浮系统，利用深度强化学习实现对克级物体的非接触式二维和三维操控。通过优化电磁铁布局和引入动作重映射方法缓解磁场非线性带来的样本稀疏问题，系统实现了灵活操控并能泛化到未训练的运输任务。此外，系统可通过扩大电磁铁规模扩展至更重物体的操作。
Reason 该论文研究的是基于磁悬浮和深度强化学习的物理操控系统，属于具身智能与机器人控制领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用完全无关。其核心技术（电磁阵列设计、动作重映射用于DRL训练）无法迁移至解决OCR幻觉、复杂排版理解、长文档推理等DIU核心问题。
Abstract Non-contact manipulation has emerged as a transformative approach across various industrial fields. However, current flexible 2D and 3D non-contact manipulation techniques are often limited to microscopic scales, typically controlling objects in the milligram range. In this paper, we present a magnetic levitation system, termed Maglev-Pentabot, designed to address this limitation. The Maglev-Pentabot leverages deep reinforcement learning (DRL) to develop complex control strategies for manipulating objects in the gram range. Specifically, we propose an electromagnet arrangement optimized through numerical analysis to maximize controllable space. Additionally, an action remapping method is introduced to address sample sparsity issues caused by the strong nonlinearity in magnetic field intensity, hence allowing the DRL controller to converge. Experimental results demonstrate flexible manipulation capabilities, and notably, our system can generalize to transport tasks it has not been explicitly trained for. Furthermore, our approach can be scaled to manipulate heavier objects using larger electromagnets, offering a reference framework for industrial-scale robotic applications.
Authors Guoming Huang, Qingyi Zhou, Dianjing Liu, Shuai Zhang, Ming Zhou, Zongfu Yu
Categories Artificial Intelligence
0
UITron-Speech: Towards Automated GUI Agents Based on Speech Instructions
Title CN UITron-Speech：基于语音指令的自动化GUI智能体
Keywords GUI智能体 · 语音指令 · 端到端学习 · 定位优化 · 多模态交互
Summary 本文提出了UITron-Speech，首个能够直接处理语音指令和设备截图以预测GUI操作的端到端智能体。为解决数据稀缺问题，作者使用多说话人TTS模型合成语音指令数据，并设计了一种混合模态训练策略来缓解基础模型中的模态不平衡问题。此外，通过统计分析GUI定位错误分布，提出一种无需训练的两步定位优化方法以修正微小偏差。实验表明该系统在多个基准上具有良好的性能和适应性，验证了语音驱动GUI代理的可行性。
Reason 该论文属于具身智能与GUI交互领域，研究的是基于语音指令操作图形用户界面的智能体，核心任务为GUI动作预测，与文档图像理解（DIU）或DocVQA无直接关联。其技术重点在于多模态输入（语音+截图）的端到端处理和定位优化，但未涉及文档中的文本识别、布局分析、表格理解或OCR相关问题。尽管其‘两步定位优化’方法在理论上可能对细粒度视觉定位有启发，但整体迁移价值低，且属于典型的‘平行应用’类工作——将现有Agent架构迁移到新输入模态（语音），而非提出可泛化的底层方法。因此不符合关注范畴。
Abstract Autonomous agents for Graphical User Interfaces (GUIs) are revolutionizing human-computer interaction, yet their reliance on text-based instructions imposes limitations on accessibility and convenience, particularly in hands-free scenarios. To address this issue, we propose replacing text with speech as the instruction input modality for GUI agents, and introduce UITron-Speech, which is the first end-to-end GUI agent capable of directly processing speech instructions and on-device screenshots to predict user actions. To tackle the problem of data scarcity, we synthesize high-quality speech instruction datasets using a random-speaker text-to-speech model. Additionally, we design a mixed-modality training strategy to mitigate the inherent modality imbalance in pre-trained foundation models. Furthermore, we conduct a statistical analysis of the distribution of GUI grounding prediction errors and propose a training-free two-step grounding refinement method to alleviate minor localization deviations. Extensive experiments on multiple benchmarks demonstrate that UITron-Speech achieves robust performance and superior adaptability, underscoring the feasibility and potential of speech-driven GUI agents for more accessible and intelligent human-computer interaction. Our code and datasets are available at https://github.com/UITron-hub/UITron-Speech.
Authors Wenkang Han, Zhixiong Zeng, Jing Huang, Shu Jiang, Liming Zheng, Longrong Yang, Haibo Qiu, Chang Yao, Jingyuan Chen, Lin Ma
Categories Artificial Intelligence, Computation and Language
0
Empowering Time Series Forecasting with LLM-Agents
Title CN 利用LLM代理增强时间序列预测
Keywords LLM-Agent · AutoML · 时间序列预测 · 数据质量 · 数据中心方法
Summary 本文提出了一种名为DCATS的数据中心式代理系统，用于提升时间序列预测中的数据质量，通过利用元数据进行数据清洗并优化预测性能。实验表明该方法在交通流量预测数据集上平均降低了6%的误差。
Reason 该论文属于平行下游应用，研究的是时间序列预测中的AutoML问题，尽管使用了LLM代理框架，但其任务场景与文档图像理解（DIU）无关。未涉及OCR、布局分析、表格识别或任何文档视觉内容的处理，无法迁移解决DIU的核心痛点如幻觉、复杂排版或多模态对齐。
Abstract Large Language Model (LLM) powered agents have emerged as effective planners for Automated Machine Learning (AutoML) systems. While most existing AutoML approaches focus on automating feature engineering and model architecture search, recent studies in time series forecasting suggest that lightweight models can often achieve state-of-the-art performance. This observation led us to explore improving data quality, rather than model architecture, as a potentially fruitful direction for AutoML on time series data. We propose DCATS, a Data-Centric Agent for Time Series. DCATS leverages metadata accompanying time series to clean data while optimizing forecasting performance. We evaluated DCATS using four time series forecasting models on a large-scale traffic volume forecasting dataset. Results demonstrate that DCATS achieves an average 6% error reduction across all tested models and time horizons, highlighting the potential of data-centric approaches in AutoML for time series forecasting.
Authors Chin-Chia Michael Yeh, Vivian Lai, Uday Singh Saini, Xiran Fan, Yujie Fan, Junpeng Wang, Xin Dai, Yan Zheng
Categories Artificial Intelligence
0
Efficient Training for Human Video Generation with Entropy-Guided Prioritized Progressive Learning
Title CN 基于熵引导的优先级渐进学习实现高效的人体视频生成训练
Keywords 视频生成 · 扩散模型 · 高效训练 · 渐进学习 · 条件熵
Summary 本文提出了一种名为Ent-Prog的高效训练框架，用于降低扩散模型在高分辨率人体视频生成中的训练开销。通过引入条件熵膨胀评估模型组件重要性，并设计自适应渐进训练策略，实现了训练速度和显存占用的显著优化。然而，该工作属于视频生成领域，与文档理解无关。
Reason 该论文研究的是人体视频生成中的高效训练方法，属于纯视频生成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其提出的方法如条件熵膨胀（CEI）和渐进式训练策略，虽然在计算效率上有创新，但应用场景完全偏离文档理解的核心痛点（如OCR幻觉、复杂排版、细粒度定位等），无法迁移用于提升DIU性能。
Abstract Human video generation has advanced rapidly with the development of diffusion models, but the high computational cost and substantial memory consumption associated with training these models on high-resolution, multi-frame data pose significant challenges. In this paper, we propose Entropy-Guided Prioritized Progressive Learning (Ent-Prog), an efficient training framework tailored for diffusion models on human video generation. First, we introduce Conditional Entropy Inflation (CEI) to assess the importance of different model components on the target conditional generation task, enabling prioritized training of the most critical components. Second, we introduce an adaptive progressive schedule that adaptively increases computational complexity during training by measuring the convergence efficiency. Ent-Prog reduces both training time and GPU memory consumption while maintaining model performance. Extensive experiments across three datasets, demonstrate the effectiveness of Ent-Prog, achieving up to 2.2$\times$ training speedup and 2.4$\times$ GPU memory reduction without compromising generative performance.
Authors Changlin Li, Jiawei Zhang, Shuhao Liu, Sihao Lin, Zeyi Shi, Zhihui Li, Xiaojun Chang
Comment Project page: https://github.com/changlin31/Ent-Prog
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
SocialNav: Training Human-Inspired Foundation Model for Socially-Aware Embodied Navigation
Title CN SocialNav：训练具有社交意识的具身导航基础模型
Keywords 具身导航 · 社会规范 · 基础模型 · 强化学习 · 多模态推理
Summary 本文提出SocialNav，一种具备社交意识的具身导航基础模型，采用分层‘脑-行为’架构，结合大规模SocNav数据集进行训练，包含认知激活数据和专家轨迹金字塔。通过模仿学习和新型基于流的强化学习方法SAFE-GRPO，实现对社会合规行为的建模，在成功率和社会合规性上显著超越现有方法。
Reason 该论文属于具身智能与机器人领域，研究的是社交场景下的导航问题，涉及社会规范理解与路径规划。尽管使用了视觉和推理机制，但其核心任务与文档图像理解（DIU）或DocVQA无关，无法迁移用于解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract Embodied navigation that adheres to social norms remains an open research challenge. Our \textbf{SocialNav} is a foundational model for socially-aware navigation with a hierarchical "brain-action" architecture, capable of understanding high-level social norms and generating low-level, socially compliant trajectories. To enable such dual capabilities, we construct the SocNav Dataset, a large-scale collection of 7 million samples, comprising (1) a Cognitive Activation Dataset providing social reasoning signals such as chain-of-thought explanations and social traversability prediction, and (2) an Expert Trajectories Pyramid aggregating diverse navigation demonstrations from internet videos, simulated environments, and real-world robots. A multi-stage training pipeline is proposed to gradually inject and refine navigation intelligence: we first inject general navigation skills and social norms understanding into the model via imitation learning, and then refine such skills through a deliberately designed Socially-Aware Flow Exploration GRPO (SAFE-GRPO), the first flow-based reinforcement learning framework for embodied navigation that explicitly rewards socially compliant behaviors. SocialNav achieves +38% success rate and +46% social compliance rate compared to the state-of-the-art method, demonstrating strong gains in both navigation performance and social compliance. Our project page: https://amap-eai.github.io/SocialNav/
Authors Ziyi Chen, Yingnan Guo, Zedong Chu, Minghua Luo, Yanfen Shen, Mingchao Sun, Junjun Hu, Shichao Xie, Kuan Yang, Pei Shi, Zhining Gu, Lu Liu, Honglin Han, Xiaolong Wu, Mu Xu, Yu Zhang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
TiCT: A Synthetically Pre-Trained Foundation Model for Time Series Classification
Title CN TiCT：一种用于时间序列分类的合成预训练基础模型
Keywords 时间序列分类 · 基础模型 · 上下文学习 · 合成数据 · Transformer
Summary 本文提出了TiCT，一个基于Transformer的时间序列分类基础模型，通过仅使用合成数据进行预训练，实现上下文学习（ICL），无需微调即可适应新分类任务。模型采用位编码标签和特殊注意力机制，结合Mixup式增强策略，在UCR数据集上表现出媲美有监督方法的性能。
Reason 该论文研究的是时间序列分类，属于无关领域（如时序预测、非视觉模态数据处理），与文档图像理解（DIU）或视觉-语言模型（VLM）在文档上的应用无直接关联。尽管其提出的in-context learning机制和合成数据训练框架具有一定方法论价值，但其核心任务、输入模态（非图像/文本）和应用场景均无法迁移到解决OCR幻觉、复杂排版或DocVQA等文档理解核心问题。
Abstract The ubiquity of time series data creates a strong demand for general-purpose foundation models, yet developing them for classification remains a significant challenge, largely due to the high cost of labeled data. Foundation models capable of in-context learning (ICL) offer a powerful solution, adapting to new tasks with minimal examples and reducing the need for extensive retraining. However, prior work on large-scale time series models has predominantly focused on forecasting, leaving a critical gap for versatile, fine-tuning-free classification. To address this, we introduce TiCT (Time-series in-Context Transformer), a transformer-based model pre-trained exclusively on synthetic data to perform in-context classification. We make two primary technical contributions: 1) a novel architecture featuring a scalable bit-based label encoding and a special output attention mechanism to handle an arbitrary number of classes; and 2) a synthetic pre-training framework that combines a Mixup-inspired process with data augmentation to foster generalization and noise invariance. Extensive evaluations on the UCR Archive show that TiCT achieves competitive performance against state-of-the-art supervised methods. Crucially, this is accomplished using only in-context examples at inference time, without updating a single model weight.
Authors Chin-Chia Michael Yeh, Uday Singh Saini, Junpeng Wang, Xin Dai, Xiran Fan, Jiarui Sun, Yujie Fan, Yan Zheng
Categories Artificial Intelligence
0
Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Models
Title CN 哪一层导致了分布偏差？面向扩散和流模型的熵引导自适应剪枝
Keywords 模型剪枝 · 扩散模型 · 流模型 · 条件熵偏差 · 生成模型压缩
Summary 本文提出EntPruner，一种基于熵引导的渐进式剪枝框架，用于压缩扩散模型和流模型。通过条件熵偏差（CED）衡量模块对输出分布的影响，实现块级重要性评估，并设计零样本自适应剪枝策略，在保持生成质量的同时提升推理速度。
Reason 该论文研究的是视觉生成模型（如扩散模型和流模型）的模型压缩方法，属于纯图像生成领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。其技术聚焦于生成质量与分布保真度的权衡，不涉及OCR、布局分析、表格识别或多模态文档推理等核心DIU问题。尽管模型压缩在理论上可能间接用于加速系统，但该工作未提供任何可迁移至文档理解场景的方法论创新，且属于‘无关领域’中的图像生成方向。
Abstract Large-scale vision generative models, including diffusion and flow models, have demonstrated remarkable performance in visual generation tasks. However, transferring these pre-trained models to downstream tasks often results in significant parameter redundancy. In this paper, we propose EntPruner, an entropy-guided automatic progressive pruning framework for diffusion and flow models. First, we introduce entropy-guided pruning, a block-level importance assessment strategy specifically designed for generative models. Unlike discriminative models, generative models require preserving the diversity and condition-fidelity of the output distribution. As the importance of each module can vary significantly across downstream tasks, EntPruner prioritizes pruning of less important blocks using data-dependent Conditional Entropy Deviation (CED) as a guiding metric. CED quantifies how much the distribution diverges from the learned conditional data distribution after removing a block. Second, we propose a zero-shot adaptive pruning framework to automatically determine when and how much to prune during training. This dynamic strategy avoids the pitfalls of one-shot pruning, mitigating mode collapse, and preserving model performance. Extensive experiments on DiT and SiT models demonstrate the effectiveness of EntPruner, achieving up to 2.22$\times$ inference speedup while maintaining competitive generation quality on ImageNet and three downstream datasets.
Authors Changlin Li, Jiawei Zhang, Zeyi Shi, Zongxin Yang, Zhihui Li, Xiaojun Chang
Comment Project page: https://github.com/changlin31/EntPruner
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Meursault as a Data Point
Title CN 默尔索作为一个数据点
Keywords NLP应用 · 情感分析 · 文学计算 · AI伦理 · 数据化批判
Summary 本文通过自然语言处理技术对加缪小说《局外人》主人公默尔索的行为和情绪进行量化分析，揭示了算法模型在处理复杂人类经验时的局限性，并以此批判数据化叙事对人文价值的侵蚀。研究本质上是一次人文反思实验，未提出任何可迁移的技术方法。
Reason 该论文属于将现有NLP技术应用于文学人物分析的平行下游应用，研究重点是哲学与伦理批判，而非提出任何新的AI方法或解决文档图像理解中的技术痛点。其使用的技术（如BERT、VADER、spaCy）均为现成工具，无方法论创新，完全符合负面清单中的‘平行应用’特征。
Abstract In an era dominated by datafication, the reduction of human experiences to quantifiable metrics raises profound philosophical and ethical questions. This paper explores these issues through the lens of Meursault, the protagonist of Albert Camus' The Stranger, whose emotionally detached existence epitomizes the existential concept of absurdity. Using natural language processing (NLP) techniques including emotion detection (BERT), sentiment analysis (VADER), and named entity recognition (spaCy)-this study quantifies key events and behaviors in Meursault's life. Our analysis reveals the inherent limitations of applying algorithmic models to complex human experiences, particularly those rooted in existential alienation and moral ambiguity. By examining how modern AI tools misinterpret Meursault's actions and emotions, this research underscores the broader ethical dilemmas of reducing nuanced human narratives to data points, challenging the foundational assumptions of our data-driven society. The findings presented in this paper serve as a critique of the increasing reliance on data-driven narratives and advocate for incorporating humanistic values in artificial intelligence.
Authors Abhinav Pratap
Comment 7 pages, 9 figures, 4 tables
Categories Artificial Intelligence, Computation and Language
0
Learning Cell-Aware Hierarchical Multi-Modal Representations for Robust Molecular Modeling AAAI 2026 (Oral)
Title CN 用于鲁棒分子建模的细胞感知分层多模态表示学习
Keywords 分子建模 · 多模态表示 · 层次化学习 · 生物信息学 · AAAI
Summary 本文提出CHMR框架，通过树结构向量量化模块联合建模分子与细胞响应之间的局部-全局依赖关系，并在九个公共基准上验证了其在分类和回归任务上的优越性能。然而，该工作聚焦于生物医学领域的多模态整合，与文档图像理解无关。
Reason 该论文研究的是分子建模与生物系统中的多模态表示问题，属于生物信息学与药物发现领域，尽管使用了多模态和层次化表示方法，但其任务目标（分子属性预测、细胞响应建模）与文档图像理解（DIU）无直接关联。所提出的方法针对的是化学结构与基因表达等生物数据的融合，并不涉及OCR、布局分析、表格识别或文档视觉问答等DIU核心任务，也无法迁移解决文档理解中的典型痛点如OCR幻觉、密集文本处理或复杂排版解析。
Abstract Understanding how chemical perturbations propagate through biological systems is essential for robust molecular property prediction. While most existing methods focus on chemical structures alone, recent advances highlight the crucial role of cellular responses such as morphology and gene expression in shaping drug effects. However, current cell-aware approaches face two key limitations: (1) modality incompleteness in external biological data, and (2) insufficient modeling of hierarchical dependencies across molecular, cellular, and genomic levels. We propose CHMR (Cell-aware Hierarchical Multi-modal Representations), a robust framework that jointly models local-global dependencies between molecules and cellular responses and captures latent biological hierarchies via a novel tree-structured vector quantization module. Evaluated on nine public benchmarks spanning 728 tasks, CHMR outperforms state-of-the-art baselines, yielding average improvements of 3.6% on classification and 17.2% on regression tasks. These results demonstrate the advantage of hierarchy-aware, multimodal learning for reliable and biologically grounded molecular representations, offering a generalizable framework for integrative biomedical modeling. The code is in https://github.com/limengran98/CHMR.
Authors Mengran Li, Zelin Zang, Wenbin Xing, Junzhou Chen, Ronghui Zhang, Jiebo Luo, Stan Z. Li
Comment Accepted to AAAI 2026 (Oral)
Categories Artificial Intelligence
0
Deformation-aware Temporal Generation for Early Prediction of Alzheimers Disease
Title CN 用于阿尔茨海默病早期预测的形变感知时序生成
Keywords 阿尔茨海默病 · MRI生成 · 时序建模 · 医学图像分析 · 数据插补
Summary 本文提出了一种形变感知时序生成网络（DATGN），用于从不完整的纵向MRI序列中插补并生成未来时间点的脑部图像，以辅助阿尔茨海默病的早期预测。该方法通过双向时序模块捕捉疾病进展趋势，在ADNI数据集上验证了生成图像的质量，并证明合成数据能显著提升分类模型的性能。
Reason 该论文研究的是基于脑部MRI图像的阿尔茨海默病早期预测，属于医学影像分析领域，核心任务为时序医学图像生成与疾病分类，与文档图像理解（DIU）、DocVQA、OCR、排版分析等完全无关。尽管使用了生成模型和时空建模技术，但其方法针对的是生物医学图像中的形态变化建模，无法迁移至文档图像中的文本识别、布局理解或视觉语言推理任务。
Abstract Alzheimer's disease (AD), a degenerative brain condition, can benefit from early prediction to slow its progression. As the disease progresses, patients typically undergo brain atrophy. Current prediction methods for Alzheimers disease largely involve analyzing morphological changes in brain images through manual feature extraction. This paper proposes a novel method, the Deformation-Aware Temporal Generative Network (DATGN), to automate the learning of morphological changes in brain images about disease progression for early prediction. Given the common occurrence of missing data in the temporal sequences of MRI images, DATGN initially interpolates incomplete sequences. Subsequently, a bidirectional temporal deformation-aware module guides the network in generating future MRI images that adhere to the disease's progression, facilitating early prediction of Alzheimer's disease. DATGN was tested for the generation of temporal sequences of future MRI images using the ADNI dataset, and the experimental results are competitive in terms of PSNR and MMSE image quality metrics. Furthermore, when DATGN-generated synthetic data was integrated into the SVM vs. CNN vs. 3DCNN-based classification methods, significant improvements were achieved from 6. 21\% to 16\% in AD vs. NC classification accuracy and from 7. 34\% to 21. 25\% in AD vs. MCI vs. NC classification accuracy. The qualitative visualization results indicate that DATGN produces MRI images consistent with the brain atrophy trend in Alzheimer's disease, enabling early disease prediction.
Authors Xin Honga, Jie Lin, Minghui Wang
Comment 29 pages,6figures,one column
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Dynamic Stratified Contrastive Learning with Upstream Augmentation for MILP Branching
Title CN 用于MILP分支的带上游增强的动态分层对比学习
Keywords MILP · Branch-and-Bound · 对比学习 · 图卷积神经网络 · 优化
Summary 本文提出了一种动态分层对比学习框架，用于提升混合整数线性规划（MILP）求解过程中分支节点的选择效率。通过将分支定界树中的节点按特征分布分组，并引入上游增强的数据生成机制，该方法提升了对上游节点的表示学习能力，从而提高了分支准确性和整体求解效率。实验表明该方法在标准MILP基准上有效减少了求解时间。
Reason 该论文研究的是混合整数线性规划（MILP）中的分支策略优化，属于运筹优化与数学规划领域，尽管使用了图卷积神经网络（GCNN）和对比学习等机器学习技术，但其核心任务与文档图像理解（DIU）、视觉语言模型（VLM）、OCR、布局分析、表格识别或DocVQA等完全无关。无法迁移其方法解决文档理解中的OCR幻觉、密集文本、复杂排版或长文档推理等核心痛点。
Abstract Mixed Integer Linear Programming (MILP) is a fundamental class of NP-hard problems that has garnered significant attention from both academia and industry. The Branch-and-Bound (B\&B) method is the dominant approach for solving MILPs and the branching plays an important role in B\&B methods. Neural-based learning frameworks have recently been developed to enhance branching policies and the efficiency of solving MILPs. However, these methods still struggle with semantic variation across depths, the scarcity of upstream nodes, and the costly collection of strong branching samples. To address these issues, we propose \ours, a Dynamic \underline{\textbf{S}}tratified \underline{\textbf{C}}ontrastive Training Framework for \underline{\textbf{MILP}} Branching. It groups branch-and-bound nodes based on their feature distributions and trains a GCNN-based discriminative model to progressively separate nodes across groups, learning finer-grained node representations throughout the tree. To address data scarcity and imbalance at upstream nodes, we introduce an upstream-augmented MILP derivation procedure that generates both theoretically equivalent and perturbed instances. \ours~effectively models subtle semantic differences between nodes, significantly enhancing branching accuracy and solving efficiency, particularly for upstream nodes. Extensive experiments on standard MILP benchmarks demonstrate that our method enhances branching accuracy, reduces solving time, and generalizes effectively to unseen instances.
Authors Tongkai Lu, Shuai Ma, Chongyang Tao
Comment 18 pages
Categories Artificial Intelligence
0
Uncovering Implicit Bias in Large Language Models with Concept Learning Dataset EurIPS 2025 Workshop - Unifying Perspectives on Learning Biases (UPLB)
Title CN 利用概念学习数据集揭示大语言模型中的隐性偏见
Keywords 大语言模型 · 隐性偏见 · 概念学习 · 上下文学习 · 量化词
Summary 本文提出了一种基于概念学习的数据集，用于揭示大语言模型在上下文学习中潜在的隐性偏见，发现模型对某些量化词存在向上单调性的偏好。这种偏见在直接提示时不易显现，但在引入概念学习任务后变得明显。研究表明，上下文中的概念学习可以有效暴露模型的隐藏偏差。
Reason 该论文研究的是语言模型在概念学习任务中对量化词的隐性偏见（如向上单调性偏好），属于纯NLP领域的社会性偏见分析，不涉及任何视觉模态、文档结构、OCR或图文对齐问题。尽管其方法可能用于探测模型内部行为，但与文档图像理解（DIU）的核心痛点（如幻觉、排版复杂性、细粒度定位等）完全无关，且未提出可用于多模态推理或视觉语言系统的可迁移技术。
Abstract We introduce a dataset of concept learning tasks that helps uncover implicit biases in large language models. Using in-context concept learning experiments, we found that language models may have a bias toward upward monotonicity in quantifiers; such bias is less apparent when the model is tested by direct prompting without concept learning components. This demonstrates that in-context concept learning can be an effective way to discover hidden biases in language models.
Authors Leroy Z. Wang
Comment Presented at EurIPS 2025 Workshop - Unifying Perspectives on Learning Biases (UPLB) https://sites.google.com/view/towards-a-unified-view
Categories Artificial Intelligence, Computation and Language
0
Pygmalion Effect in Vision: Image-to-Clay Translation for Reflective Geometry Reconstruction
Title CN 视觉中的皮格马利翁效应：用于反射几何重建的图像到黏土转换
Keywords 3D重建 · 反射处理 · 图像到黏土转换 · 双分支网络 · 几何一致性
Summary 本文提出了一种名为‘视觉中的皮格马利翁效应’的新框架，通过将带有反射的物体图像转换为无反射的‘黏土样’形态，来改善3D反射物体的几何重建。该方法采用双分支网络，一个分支处理基于BRDF的反射特性，另一个分支利用合成的无反射黏土图像作为监督信号，以稳定几何结构并优化表面法线。实验表明该方法在法线准确性和网格完整性方面优于现有技术。
Reason 该论文研究的是3D重建中处理镜面反射的视觉问题，属于3D视觉与计算机视觉领域，其核心任务是反射表面的几何建模，并非文档图像理解（DIU）或文档视觉问答（DocVQA）等任务。尽管使用了多视角图像和双分支网络结构，但方法目标与OCR、布局分析、表格识别、文本理解等DIU核心问题无关。同时，该工作无法迁移解决文档理解中的幻觉、密集文本、复杂排版或长文档推理等痛点。
Abstract Understanding reflection remains a long-standing challenge in 3D reconstruction due to the entanglement of appearance and geometry under view-dependent reflections. In this work, we present the Pygmalion Effect in Vision, a novel framework that metaphorically "sculpts" reflective objects into clay-like forms through image-to-clay translation. Inspired by the myth of Pygmalion, our method learns to suppress specular cues while preserving intrinsic geometric consistency, enabling robust reconstruction from multi-view images containing complex reflections. Specifically, we introduce a dual-branch network in which a BRDF-based reflective branch is complemented by a clay-guided branch that stabilizes geometry and refines surface normals. The two branches are trained jointly using the synthesized clay-like images, which provide a neutral, reflection-free supervision signal that complements the reflective views. Experiments on both synthetic and real datasets demonstrate substantial improvement in normal accuracy and mesh completeness over existing reflection-handling methods. Beyond technical gains, our framework reveals that seeing by unshining, translating radiance into neutrality, can serve as a powerful inductive bias for reflective object geometry learning.
Authors Gayoung Lee, Junho Kim, Jin-Hwa Kim, Junmo Kim
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
MigGPT: Harnessing Large Language Models for Automated Migration of Out-of-Tree Linux Kernel Patches Across Versions
Title CN MigGPT：利用大语言模型实现Linux内核外部补丁跨版本的自动化迁移
Keywords LLM应用 · 代码迁移 · Linux内核 · 软件维护 · 代码指纹
Summary 本文提出MigGPT框架，用于自动化迁移Linux内核的外部补丁。通过引入代码指纹结构和三个精心设计的模块，提升LLM在补丁迁移中的准确性和效率，并构建了真实场景下的基准测试集。实验表明该方法显著优于直接使用原始LLM。
Reason 该论文属于典型的平行下游应用，将LLM应用于特定软件工程任务（内核补丁迁移），并未提出任何可用于文档图像理解（DIU）的方法论创新。其技术核心是代码指纹和模块化流程设计，与文档理解中的OCR幻觉、复杂排版、细粒度定位等关键问题无关，无法迁移至DIU领域。
Abstract Out-of-tree kernel patches are essential for adapting the Linux kernel to new hardware or enabling specific functionalities. Maintaining and updating these patches across different kernel versions demands significant effort from experienced engineers. Large language models (LLMs) have shown remarkable progress across various domains, suggesting their potential for automating out-of-tree kernel patch migration. However, our findings reveal that LLMs, while promising, struggle with incomplete code context understanding and inaccurate migration point identification. In this work, we propose MigGPT, a framework that employs a novel code fingerprint structure to retain code snippet information and incorporates three meticulously designed modules to improve the migration accuracy and efficiency of out-of-tree kernel patches. Furthermore, we establish a robust benchmark using real-world out-of-tree kernel patch projects to evaluate LLM capabilities. Evaluations show that MigGPT significantly outperforms the direct application of vanilla LLMs, achieving an average completion rate of 74.07 for migration tasks.
Authors Pucheng Dang, Di Huang, Dong Li, Kang Chen, Yuanbo Wen, Qi Guo, Xing Hu
Categories Artificial Intelligence
0
ConceptGuard: Proactive Safety in Text-and-Image-to-Video Generation through Multimodal Risk Detection
Title CN ConceptGuard：通过多模态风险检测实现文本和图像到视频生成的主动安全防护
Keywords 视频生成安全 · 多模态风险检测 · 提示干预 · 对比学习 · 生成模型安全
Summary 本文提出ConceptGuard，一个用于文本-图像到视频生成的主动安全框架，通过对比检测模块在融合的图文输入中识别潜在安全风险，并利用语义抑制机制在生成过程中干预多模态条件以规避不安全概念。作者构建了两个新基准数据集ConceptRisk和T2VSafetyBench-TI2V用于评估。实验表明该方法在风险检测与安全生成方面均优于现有基线。
Reason 该论文聚焦于文本-图像到视频生成中的安全与内容合规问题，属于纯视频生成与内容安全领域。其核心任务是检测和抑制生成视频中的有害语义，方法基于多模态对比学习与提示干预，但应用场景完全偏离文档图像理解（DIU）的核心任务（如DocVQA、表格识别、布局分析等）。尽管其‘干预机制’看似具有迁移潜力，但针对的是内容安全而非OCR幻觉、细粒度定位或复杂排版理解等DIU痛点，迁移价值极低。且属于典型的‘平行应用’类研究——将现有安全框架扩展至多模态生成场景。
Abstract Recent progress in video generative models has enabled the creation of high-quality videos from multimodal prompts that combine text and images. While these systems offer enhanced controllability, they also introduce new safety risks, as harmful content can emerge from individual modalities or their interaction. Existing safety methods are often text-only, require prior knowledge of the risk category, or operate as post-generation auditors, struggling to proactively mitigate such compositional, multimodal risks. To address this challenge, we present ConceptGuard, a unified safeguard framework for proactively detecting and mitigating unsafe semantics in multimodal video generation. ConceptGuard operates in two stages: First, a contrastive detection module identifies latent safety risks by projecting fused image-text inputs into a structured concept space; Second, a semantic suppression mechanism steers the generative process away from unsafe concepts by intervening in the prompt's multimodal conditioning. To support the development and rigorous evaluation of this framework, we introduce two novel benchmarks: ConceptRisk, a large-scale dataset for training on multimodal risks, and T2VSafetyBench-TI2V, the first benchmark adapted from T2VSafetyBench for the Text-and-Image-to-Video (TI2V) safety setting. Comprehensive experiments on both benchmarks show that ConceptGuard consistently outperforms existing baselines, achieving state-of-the-art results in both risk detection and safe video generation. Our code is available at https://github.com/Ruize-Ma/ConceptGuard.
Authors Ruize Ma, Minghong Cai, Yilei Jiang, Jiaming Han, Yi Feng, Yingshui Tan, Xiaoyong Zhu, Bo Zhang, Bo Zheng, Xiangyu Yue
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
MNM : Multi-level Neuroimaging Meta-analysis with Hyperbolic Brain-Text Representations MICCAI 2025
Title CN MNM：基于双曲脑-文本表示的多层次神经影像元分析
Keywords 神经影像元分析 · 双曲表示 · 多模态嵌入 · 层次结构建模 · MICCAI
Summary 本文提出一种基于双曲几何的框架MNM，用于多层次神经影像元分析。通过洛伦兹模型将科研文章文本和大脑激活图像嵌入到共享的双曲空间中，以同时捕捉语义相似性和神经数据内在的层次结构。在该空间中实现文本与脑区的语义对齐、跨模态层次引导以及脑激活模式内部的层级关系保持。实验表明该方法优于基线，为神经影像元分析提供了更鲁棒、可解释的新范式。
Reason 该论文研究的是神经影像学领域的元分析问题，核心是将科研文献与大脑激活图谱通过双曲空间进行对齐和层次化建模。尽管使用了多模态表示方法，但其任务场景（脑成像数据与医学文献关联）与文档图像理解（DIU）如DocVQA、版面分析、表格识别等无直接关联。所提方法针对的是神经科学中的层次结构建模，无法迁移解决OCR幻觉、密集文本识别、复杂排版或长文档推理等DIU核心痛点，属于平行应用且领域偏离。
Abstract Various neuroimaging studies suffer from small sample size problem which often limit their reliability. Meta-analysis addresses this challenge by aggregating findings from different studies to identify consistent patterns of brain activity. However, traditional approaches based on keyword retrieval or linear mappings often overlook the rich hierarchical structure in the brain. In this work, we propose a novel framework that leverages hyperbolic geometry to bridge the gap between neuroscience literature and brain activation maps. By embedding text from research articles and corresponding brain images into a shared hyperbolic space via the Lorentz model, our method captures both semantic similarity and hierarchical organization inherent in neuroimaging data. In the hyperbolic space, our method performs multi-level neuroimaging meta-analysis (MNM) by 1) aligning brain and text embeddings for semantic correspondence, 2) guiding hierarchy between text and brain activations, and 3) preserving the hierarchical relationships within brain activation patterns. Experimental results demonstrate that our model outperforms baselines, offering a robust and interpretable paradigm of multi-level neuroimaging meta-analysis via hyperbolic brain-text representation.
Authors Seunghun Baek, Jaejin Lee, Jaeyoon Sim, Minjae Jeong, Won Hwa Kim
Comment MICCAI 2025 (Provisional Accept; top ~9%)
Categories Artificial Intelligence
0
Cheating Stereo Matching in Full-scale: Physical Adversarial Attack against Binocular Depth Estimation in Autonomous Driving AAAI 2026
Title CN 全尺度下欺骗立体匹配：针对自动驾驶中双目深度估计的物理对抗攻击
Keywords 对抗攻击 · 立体匹配 · 深度估计 · 自动驾驶 · 物理攻击
Summary 本文提出了一种针对双目深度估计模型的物理对抗攻击方法，通过在3D物体上添加具有全局伪装纹理的对抗性扰动，在多视角下实现对立体匹配模型的有效欺骗。作者设计了新的3D立体匹配渲染模块以对齐真实世界位置，并提出细粒度优化的融合攻击策略，使目标无缝融入环境，提升攻击隐蔽性和破坏性。实验表明该方法能导致深度估计模型输出错误的深度信息。
Reason 该论文研究的是自动驾驶场景下的立体视觉对抗攻击，属于3D视觉与安全领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法聚焦于物理世界中的纹理对抗扰动设计和双目渲染，无法迁移用于解决OCR幻觉、复杂排版解析、长文档推理等DIU核心问题。
Abstract Though deep neural models adopted to realize the perception of autonomous driving have proven vulnerable to adversarial examples, known attacks often leverage 2D patches and target mostly monocular perception. Therefore, the effectiveness of Physical Adversarial Examples (PAEs) on stereo-based binocular depth estimation remains largely unexplored. To this end, we propose the first texture-enabled physical adversarial attack against stereo matching models in the context of autonomous driving. Our method employs a 3D PAE with global camouflage texture rather than a local 2D patch-based one, ensuring both visual consistency and attack effectiveness across different viewpoints of stereo cameras. To cope with the disparity effect of these cameras, we also propose a new 3D stereo matching rendering module that allows the PAE to be aligned with real-world positions and headings in binocular vision. We further propose a novel merging attack that seamlessly blends the target into the environment through fine-grained PAE optimization. It has significantly enhanced stealth and lethality upon existing hiding attacks that fail to get seamlessly merged into the background. Extensive evaluations show that our PAEs can successfully fool the stereo models into producing erroneous depth information.
Authors Kangqiao Zhao, Shuo Huai, Xurui Song, Jun Luo
Comment AAAI 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmarks
Title CN Pistachio：迈向合成、平衡且长时视频异常基准
Keywords 视频异常检测 · 合成数据 · 基准数据集 · 因果推理 · 长时视频
Summary 本文提出了Pistachio，一个通过生成式流水线构建的视频异常检测与理解基准，利用视频生成模型实现对场景、异常类型和时间叙事的精确控制，生成41秒长、时间一致的视频，以解决现有数据集在多样性、平衡性和标注成本上的局限。但由于其完全聚焦于视频领域，与文档图像理解无关联。
Reason 该论文属于视频理解领域，研究的是视频异常检测与理解（VAD/VAU）的基准构建，使用合成数据生成长时视频序列。其任务目标、数据模态和应用场景均与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等核心问题。
Abstract Automatically detecting abnormal events in videos is crucial for modern autonomous systems, yet existing Video Anomaly Detection (VAD) benchmarks lack the scene diversity, balanced anomaly coverage, and temporal complexity needed to reliably assess real-world performance. Meanwhile, the community is increasingly moving toward Video Anomaly Understanding (VAU), which requires deeper semantic and causal reasoning but remains difficult to benchmark due to the heavy manual annotation effort it demands. In this paper, we introduce Pistachio, a new VAD/VAU benchmark constructed entirely through a controlled, generation-based pipeline. By leveraging recent advances in video generation models, Pistachio provides precise control over scenes, anomaly types, and temporal narratives, effectively eliminating the biases and limitations of Internet-collected datasets. Our pipeline integrates scene-conditioned anomaly assignment, multi-step storyline generation, and a temporally consistent long-form synthesis strategy that produces coherent 41-second videos with minimal human intervention. Extensive experiments demonstrate the scale, diversity, and complexity of Pistachio, revealing new challenges for existing methods and motivating future research on dynamic and multi-event anomaly understanding.
Authors Jie Li, Hongyi Cai, Mingkang Dong, Muxin Pu, Shan You, Fei Wang, Tao Huang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Multi-PA: A Multi-perspective Benchmark on Privacy Assessment for Large Vision-Language Models
Title CN Multi-PA：面向大视觉语言模型隐私评估的多视角基准
Keywords 隐私评估 · 视觉语言模型 · 基准测试 · 隐私泄露 · 隐私意识
Summary 本文提出了Multi-PA，一个用于评估大视觉语言模型在隐私意识和隐私泄露方面表现的多维度基准，涵盖个人隐私、商业秘密和国家秘密等多个类别，共包含31,962个样本。作者基于该基准测试了23个主流LVLM，发现当前模型普遍存在较高的隐私泄露风险。研究强调了LVLM在实际部署中面临的隐私挑战。
Reason 该论文聚焦于大视觉语言模型（LVLM）的隐私风险评估，属于AI安全与伦理范畴，提出的是一个隐私评估基准Multi-PA，并非针对文档图像理解（DIU）中的核心问题如OCR幻觉、复杂排版、细粒度识别或长文档推理。虽然涉及VLM，但其任务和方法均无法直接迁移用于提升DIU性能，属于平行应用性质的外围研究，且未触及文档结构、布局分析或视觉-文本对齐等关键技术痛点。
Abstract Large Vision-Language Models (LVLMs) exhibit impressive potential across various tasks but also face significant privacy risks, limiting their practical applications. Current researches on privacy assessment for LVLMs is limited in scope, with gaps in both assessment dimensions and privacy categories. To bridge this gap, we propose Multi-PA, a comprehensive benchmark for evaluating the privacy preservation capabilities of LVLMs in terms of privacy awareness and leakage. Privacy awareness measures the model's ability to recognize the privacy sensitivity of input data, while privacy leakage assesses the risk of the model unintentionally disclosing privacy information in its output. We design a range of sub-tasks to thoroughly evaluate the model's privacy protection offered by LVLMs. Multi-PA covers 26 categories of personal privacy, 15 categories of trade secrets, and 18 categories of state secrets, totaling 31,962 samples. Based on Multi-PA, we evaluate the privacy preservation capabilities of 21 open-source and 2 closed-source LVLMs. Our results reveal that current LVLMs generally pose a high risk of facilitating privacy breaches, with vulnerabilities varying across personal privacy, trade secret, and state secret.
Authors Jie Zhang, Xiangkui Cao, Zhouyu Han, Shiguang Shan, Xilin Chen
Categories Artificial Intelligence
0
Enhancing Burmese News Classification with Kolmogorov-Arnold Network Head Fine-tuning iSAI-NLP 2025
Title CN 通过Kolmogorov-Arnold网络头微调增强缅甸语新闻分类
Keywords KAN · 低资源语言 · 新闻分类 · 缅甸语 · mBERT
Summary 本文研究在缅甸语新闻分类任务中使用Kolmogorov-Arnold网络（KAN）替代传统MLP作为分类头的效果，比较了多种KAN变体与不同嵌入方法的性能表现。结果显示KAN在准确性和效率上优于或媲美MLP，尤其在fastText和mBERT嵌入上表现良好。
Reason 该论文聚焦于低资源语言（缅甸语）的文本分类任务，属于特定小语种的下游应用，未涉及文档图像理解中的视觉模态、布局分析、OCR或VLM多模态建模等核心问题。尽管使用了KAN作为分类头的创新尝试，但其方法论并未针对文档图像理解中的关键挑战（如OCR幻觉、复杂排版、细粒度定位等）提出可迁移的解决方案。
Abstract In low-resource languages like Burmese, classification tasks often fine-tune only the final classification layer, keeping pre-trained encoder weights frozen. While Multi-Layer Perceptrons (MLPs) are commonly used, their fixed non-linearity can limit expressiveness and increase computational cost. This work explores Kolmogorov-Arnold Networks (KANs) as alternative classification heads, evaluating Fourier-based FourierKAN, Spline-based EfficientKAN, and Grid-based FasterKAN-across diverse embeddings including TF-IDF, fastText, and multilingual transformers (mBERT, Distil-mBERT). Experimental results show that KAN-based heads are competitive with or superior to MLPs. EfficientKAN with fastText achieved the highest F1-score (0.928), while FasterKAN offered the best trade-off between speed and accuracy. On transformer embeddings, EfficientKAN matched or slightly outperformed MLPs with mBERT (0.917 F1). These findings highlight KANs as expressive, efficient alternatives to MLPs for low-resource language classification.
Authors Thura Aung, Eaint Kay Khaing Kyaw, Ye Kyaw Thu, Thazin Myint Oo, Thepchai Supnithi
Comment 6 pages, 2 figures, 4 tables, Accepted to iSAI-NLP 2025
Categories Artificial Intelligence, Computation and Language
0
PointNSP: Autoregressive 3D Point Cloud Generation with Next-Scale Level-of-Detail Prediction
Title CN PointNSP：通过下一尺度细节层次预测实现自回归3D点云生成
Keywords 3D点云生成 · 自回归模型 · 多尺度生成 · 细节层次 · ShapeNet
Summary 本文提出PointNSP，一种基于细节层次（LOD）的自回归3D点云生成框架，通过从粗到精的方式逐步预测下一尺度的几何细节，在保持全局结构的同时提升生成质量。实验表明其在ShapeNet上达到SOTA，并在效率上优于扩散模型。
Reason 该论文研究的是3D点云生成，属于具身智能与3D视觉领域，与文档图像理解（DIU）或视觉语言模型在文档任务中的应用无直接关联。尽管其多尺度生成框架有一定方法论价值，但无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract Autoregressive point cloud generation has long lagged behind diffusion-based approaches in quality. The performance gap stems from the fact that autoregressive models impose an artificial ordering on inherently unordered point sets, forcing shape generation to proceed as a sequence of local predictions. This sequential bias emphasizes short-range continuity but undermines the model's capacity to capture long-range dependencies, hindering its ability to enforce global structural properties such as symmetry, consistent topology, and large-scale geometric regularities. Inspired by the level-of-detail (LOD) principle in shape modeling, we propose PointNSP, a coarse-to-fine generative framework that preserves global shape structure at low resolutions and progressively refines fine-grained geometry at higher scales through a next-scale prediction paradigm. This multi-scale factorization aligns the autoregressive objective with the permutation-invariant nature of point sets, enabling rich intra-scale interactions while avoiding brittle fixed orderings. Experiments on ShapeNet show that PointNSP establishes state-of-the-art (SOTA) generation quality for the first time within the autoregressive paradigm. In addition, it surpasses strong diffusion-based baselines in parameter, training, and inference efficiency. Finally, in dense generation with 8,192 points, PointNSP's advantages become even more pronounced, underscoring its scalability potential.
Authors Ziqiao Meng, Qichao Wang, Zhiyang Dou, Zixing Song, Zhipeng Zhou, Irwin King, Peilin Zhao
Comment This work was intended as a replacement of arXiv:2503.08594 and any subsequent updates will appear there
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Data-Driven Assessment of Concrete Slab Integrity via Impact-Echo Signals and Neural Networks IEEE Big Data 2025
Title CN 基于冲击回波信号和神经网络的数据驱动混凝土板完整性评估
Keywords 冲击回波 · 混凝土缺陷检测 · LSTM · 非破坏性检测 · 频谱分析
Summary 本文提出一种基于机器学习的冲击回波（IE）框架，用于自动化检测混凝土桥面板中的分层、空洞等 subsurface 缺陷。通过FFT将原始IE信号转换为峰值频率特征，并构建空间图进行可视化。使用k-means聚类识别潜在缺陷区域，并基于实验室标注数据生成训练标签。最后采用堆叠LSTM对四种缺陷类型进行分类，在实验室和实地数据上验证了模型的有效性和泛化能力。
Reason 该论文属于土木工程结构健康监测领域，研究利用冲击回波信号和LSTM进行混凝土缺陷分类，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。其方法无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心问题。
Abstract Subsurface defects such as delamination, voids, and honeycombing critically affect the durability of concrete bridge decks but are difficult to detect reliably using visual inspection or manual sounding. This paper presents a machine learning based Impact Echo (IE) framework that automates both defect localization and multi-class classification of common concrete defects. Raw IE signals from Federal Highway Administration (FHWA) laboratory slabs and in-service bridge decks are transformed via Fast Fourier Transform (FFT) into dominant peak-frequency features and interpolated into spatial maps for defect zone visualization. Unsupervised k-means clustering highlights low-frequency, defect-prone regions, while Ground Truth Masks (GTMs) derived from seeded lab defects are used to validate spatial accuracy and generate high-confidence training labels. From these validated regions, spatially ordered peak-frequency sequences are constructed and fed into a stacked Long Short-Term Memory (LSTM) network that classifies four defect types shallow delamination, deep delamination, voids, and honeycombing with 73% overall accuracy. Field validation on the bridge deck demonstrates that models trained on laboratory data generalize under realistic coupling, noise, and environmental variability. The proposed framework enhances the objectivity, scalability, and repeatability of Non-Destructive Evaluation (NDE), supporting intelligent, data-driven bridge health monitoring at a network scale.
Authors Yeswanth Ravichandran, Duoduo Liao, Charan Teja Kurakula
Comment Accepted by IEEE Big Data 2025
Categories Artificial Intelligence
0
Context-Aware Pragmatic Metacognitive Prompting for Sarcasm Detection
Title CN 用于讽刺检测的上下文感知实用元认知提示
Keywords 讽刺检测 · 提示工程 · 上下文检索 · 大语言模型 · 语用理解
Summary 本文提出一种结合外部检索与内部知识激活的元认知提示方法来提升大模型在讽刺检测中的表现，尤其应对文化特定俚语和未知词汇带来的挑战。通过引入非参数化知识检索和自知识意识策略，在多个数据集上提升了F1分数。然而，该工作局限于纯文本NLP任务，未涉及任何视觉或文档模态。
Reason 该论文属于纯自然语言处理任务（讽刺检测），且基于社交媒体文本（如Twitter）和文化背景相关的语言现象，与文档图像理解（DIU）、视觉-语言模型在文档上的应用、OCR、布局分析等完全无关。尽管使用了提示工程和外部知识检索，但其方法针对的是语言层面的语用歧义问题，并不涉及多模态输入、视觉结构或文档排版等DIU核心挑战，无法迁移解决OCR幻觉、细粒度定位或复杂版式理解等问题。
Abstract Detecting sarcasm remains a challenging task in the areas of Natural Language Processing (NLP) despite recent advances in neural network approaches. Currently, Pre-trained Language Models (PLMs) and Large Language Models (LLMs) are the preferred approach for sarcasm detection. However, the complexity of sarcastic text, combined with linguistic diversity and cultural variation across communities, has made the task more difficult even for PLMs and LLMs. Beyond that, those models also exhibit unreliable detection of words or tokens that require extra grounding for analysis. Building on a state-of-the-art prompting method in LLMs for sarcasm detection called Pragmatic Metacognitive Prompting (PMP), we introduce a retrieval-aware approach that incorporates retrieved contextual information for each target text. Our pipeline explores two complementary ways to provide context: adding non-parametric knowledge using web-based retrieval when the model lacks necessary background, and eliciting the model's own internal knowledge for a self-knowledge awareness strategy. We evaluated our approach with three datasets, such as Twitter Indonesia Sarcastic, SemEval-2018 Task 3, and MUStARD. Non-parametric retrieval resulted in a significant 9.87% macro-F1 improvement on Twitter Indonesia Sarcastic compared to the original PMP method. Self-knowledge retrieval improves macro-F1 by 3.29% on Semeval and by 4.08% on MUStARD. These findings highlight the importance of context in enhancing LLMs performance in sarcasm detection task, particularly the involvement of culturally specific slang, references, or unknown terms to the LLMs. Future work will focus on optimizing the retrieval of relevant contextual information and examining how retrieval quality affects performance. The experiment code is available at: https://github.com/wllchrst/sarcasm-detection_pmp_knowledge-base.
Authors Michael Iskandardinata, William Christian, Derwin Suhartono
Categories Artificial Intelligence, Computation and Language
0
Consistent Opponent Modeling of Static Opponents in Imperfect-Information Games
Title CN 不完美信息博弈中静态对手的一致性建模
Keywords 对手建模 · 不完美信息博弈 · 纳什均衡 · 策略收敛 · 多智能体
Summary 本文提出了一种新的对手建模算法，能够在不完美信息博弈中通过对局历史数据逐步逼近静态对手的真实策略，并通过序列形式博弈表示下的凸优化问题实现高效收敛。然而该工作完全脱离视觉与语言交叉场景，不具备向文档理解任务迁移的潜力。
Reason 该论文研究的是博弈论中的对手建模问题，属于多智能体系统和游戏AI领域，与文档图像理解（DIU）或视觉-语言模型（VLM）在文档处理中的应用无直接关联。其方法聚焦于不完美信息博弈中的策略收敛性，并未涉及OCR、视觉表示、多模态对齐、细粒度定位或高分辨率输入处理等DIU核心痛点，也无法迁移用于解决文档理解中的幻觉、复杂排版或长文档推理等问题。
Abstract The goal of agents in multi-agent environments is to maximize total reward against the opposing agents that are encountered. Following a game-theoretic solution concept, such as Nash equilibrium, may obtain a strong performance in some settings; however, such approaches fail to capitalize on historical and observed data from repeated interactions against our opponents. Opponent modeling algorithms integrate machine learning techniques to exploit suboptimal opponents utilizing available data; however, the effectiveness of such approaches in imperfect-information games to date is quite limited. We show that existing opponent modeling approaches fail to satisfy a simple desirable property even against static opponents drawn from a known prior distribution; namely, they do not guarantee that the model approaches the opponent's true strategy even in the limit as the number of game iterations approaches infinity. We develop a new algorithm that is able to achieve this property and runs efficiently by solving a convex minimization problem based on the sequence-form game representation using projected gradient descent. The algorithm is guaranteed to efficiently converge to the opponent's true strategy given observations from gameplay and possibly additional historical data if it is available.
Authors Sam Ganzfried
Categories Artificial Intelligence
0
CoxKAN: Kolmogorov-Arnold Networks for Interpretable, High-Performance Survival Analysis
Title CN CoxKAN：用于可解释、高性能生存分析的柯尔莫哥洛夫-阿诺德网络
Keywords 生存分析 · 可解释AI · Cox模型 · Kolmogorov-Arnold Network · 医疗机器学习
Summary 本文提出了CoxKAN，一种结合Cox比例风险模型与柯尔莫哥洛夫-阿诺德网络（KAN）的新方法，用于提升生存分析的可解释性与预测性能。实验表明其在多个真实医疗数据集上优于传统Cox模型，并能揭示变量间的复杂交互关系及符号化公式。目标是辅助医学决策，而非解决视觉或文档理解问题。
Reason 该论文属于医学统计与生存分析领域，聚焦于在临床和基因组数据上构建兼具性能与可解释性的风险预测模型。尽管其方法（KAN）具有一定的可解释性创新，但任务本身与文档图像理解（DIU）、视觉信息抽取或VLM无关，也不提供可用于解决OCR幻觉、复杂排版、多模态对齐或高分辨率文档处理的方法论迁移价值。属于典型垂直领域应用，无底层技术外溢潜力。
Abstract Motivation: Survival analysis is a branch of statistics that is crucial in medicine for modeling the time to critical events such as death or relapse, in order to improve treatment strategies and patient outcomes. Selecting survival models often involves a trade-off between performance and interpretability; deep learning models offer high performance but lack the transparency of more traditional approaches. This poses a significant issue in medicine, where practitioners are reluctant to use black-box models for critical patient decisions. Results: We introduce CoxKAN, a Cox proportional hazards Kolmogorov-Arnold Network for interpretable, high-performance survival analysis. Kolmogorov-Arnold Networks (KANs) were recently proposed as an interpretable and accurate alternative to multi-layer perceptrons. We evaluated CoxKAN on four synthetic and nine real datasets, including five cohorts with clinical data and four with genomics biomarkers. In synthetic experiments, CoxKAN accurately recovered interpretable hazard function formulae and excelled in automatic feature selection. Evaluations on real datasets showed that CoxKAN consistently outperformed the traditional Cox proportional hazards model (by up to 4% in C-index) and matched or surpassed the performance of deep learning-based models. Importantly, CoxKAN revealed complex interactions between predictor variables and uncovered symbolic formulae, which are key capabilities that other survival analysis methods lack, to provide clear insights into the impact of key biomarkers on patient risk. Availability and implementation: CoxKAN is available at GitHub and Zenodo
Authors William Knottenbelt, William McGough, Rebecca Wray, Woody Zhidong Zhang, Jiashuai Liu, Ines Prata Machado, Zeyu Gao, Mireia Crispin-Ortuzar
Categories Artificial Intelligence
0
Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor NeurIPS'25
Title CN 人工智能中的严谨性：开展严谨的AI工作需要更广泛且负责任的AI导向的严谨性概念
Keywords 负责任AI · 研究严谨性 · 伦理 · AI治理 · 立场论文
Summary 本文提出在AI研究中应扩展‘严谨性’的定义，超越传统的方法论严谨，纳入知识背景、规范影响、概念清晰性、报告透明度和推论支持度等多个维度。作者呼吁建立更全面的框架以促进AI社区内外的对话。
Reason 该论文属于负责任AI（Responsible AI）领域的立场性讨论，聚焦于AI研究中的方法论、伦理与报告规范等宏观议题，不涉及任何文档图像理解（DIU）或视觉语言模型的技术创新。其内容无法迁移用于解决OCR幻觉、复杂排版理解、细粒度定位等DIU核心问题，属于无关领域。
Abstract In AI research and practice, rigor remains largely understood in terms of methodological rigor -- such as whether mathematical, statistical, or computational methods are correctly applied. We argue that this narrow conception of rigor has contributed to the concerns raised by the responsible AI community, including overblown claims about the capabilities of AI systems. Our position is that a broader conception of what rigorous AI research and practice should entail is needed. We believe such a conception -- in addition to a more expansive understanding of (1) methodological rigor -- should include aspects related to (2) what background knowledge informs what to work on (epistemic rigor); (3) how disciplinary, community, or personal norms, standards, or beliefs influence the work (normative rigor); (4) how clearly articulated the theoretical constructs under use are (conceptual rigor); (5) what is reported and how (reporting rigor); and (6) how well-supported the inferences from existing evidence are (interpretative rigor). In doing so, we also provide useful language and a framework for much-needed dialogue about the AI community's work by researchers, policymakers, journalists, and other stakeholders.
Authors Alexandra Olteanu, Su Lin Blodgett, Agathe Balayn, Angelina Wang, Fernando Diaz, Flavio du Pin Calmon, Margaret Mitchell, Michael Ekstrand, Reuben Binns, Solon Barocas
Comment 21 pages, 1 figure, 1 table, accepted at NeurIPS'25 position papers track
Categories Artificial Intelligence
0
R3A: Reliable RTL Repair Framework with Multi-Agent Fault Localization and Stochastic Tree-of-Thoughts Patch Generation
Title CN R3A：基于多智能体故障定位与随机思维树补丁生成的可靠RTL修复框架
Keywords 程序修复 · 大语言模型 · 多智能体 · 思维树 · RTL
Summary 本文提出R3A，一种基于大语言模型的RTL程序自动修复框架，通过多智能体进行故障定位，并采用随机化的思维树方法生成修复补丁，以提升修复结果的可靠性。实验表明其在修复率和可靠性上优于传统方法和其他LLM方法。
Reason 该论文研究的是硬件设计中的RTL代码修复问题，属于程序修复在特定领域（硬件描述语言）的应用。尽管使用了多智能体和Stochastic Tree-of-Thoughts等方法，但其核心是将现有LLM技术应用于一个垂直工程任务，属于典型的‘平行应用’。虽然方法中涉及Agent和推理搜索机制，但并未提出可迁移至文档图像理解领域的通用底层创新，且任务场景与DIU无关。
Abstract Repairing RTL bugs is crucial for hardware design and verification. Traditional automatic program repair (APR) methods define dedicated search spaces to locate and fix bugs with program synthesis. However, they heavily rely on fixed templates and can only deal with limited bugs. As an alternative, Large Language Models with the ability to understand code semantics can be explored for RTL repair. However, they suffer from unreliable outcomes due to inherent randomness and long input contexts of RTL code and waveform. To address these challenges, we propose R3A, an LLM-based automatic RTL program repair framework upon the basic model to improve reliability. R3A proposes the stochastic Tree-Of-Thoughts method to control a patch generation agent to explore a validated solution for the bug. The algorithm samples search states according to a heuristic function to balance between exploration and exploitation for a reliable outcome. Besides, R3A proposes a multi-agent fault localization method to find fault candidates as the starting points for the patch generation agent, further increasing the reliability. Experiments show R3A can fix 90.6% of bugs in the RTL-repair dataset within a given time limit, which covers 45% more bugs than traditional methods and other LLM-based approaches, while achieving an 86.7% pass@5 rate on average, showing a high reliability.
Authors Zizhang Luo, Fan Cui, Kexing Zhou, Runlin Guo, Mile Xia, Hongyuan Hou, Yun Liang
Categories Artificial Intelligence
0
FedAPA: Federated Learning with Adaptive Prototype Aggregation Toward Heterogeneous Wi-Fi CSI-based Crowd Counting
Title CN FedAPA：面向异构Wi-Fi CSI人群计数的自适应原型聚合联邦学习
Keywords 联邦学习 · Wi-Fi CSI · 人群计数 · 自适应聚合 · 对比学习
Summary 本文提出FedAPA，一种用于Wi-Fi CSI感知的联邦学习算法，通过自适应原型聚合策略为不同客户端分配基于相似性的权重，并结合分类学习与对比表示学习，在本地训练中对齐全局知识。实验表明该方法在准确率、F1分数、MAE和通信开销方面均优于基线方法。研究场景为现实世界中的分布式人群计数，涵盖六种环境。
Reason 该论文研究的是基于Wi-Fi信道状态信息（CSI）的人群计数问题，属于无线传感与联邦学习的交叉领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。尽管其方法涉及联邦学习中的异构数据处理，但任务本质为信号处理与回归预测，无法迁移解决OCR幻觉、复杂排版解析、细粒度文本定位等DIU核心痛点。
Abstract Wi-Fi channel state information (CSI)-based sensing provides a non-invasive, device-free approach for tasks such as human activity recognition and crowd counting, but large-scale deployment is hindered by the need for extensive site-specific training data. Federated learning (FL) offers a way to avoid raw data sharing but is challenged by heterogeneous sensing data and device resources. This paper proposes FedAPA, a collaborative Wi-Fi CSI-based sensing algorithm that uses adaptive prototype aggregation (APA) strategy to assign similarity-based weights to peer prototypes, enabling adaptive client contributions and yielding a personalized global prototype for each client instead of a fixed-weight aggregation. During local training, we adopt a hybrid objective that combines classification learning with representation contrastive learning to align local and global knowledge. We provide a convergence analysis of FedAPA and evaluate it in a real-world distributed Wi-Fi crowd counting scenario with six environments and up to 20 people. The results show that our method outperform multiple baselines in terms of accuracy, F1 score, mean absolute error (MAE), and communication overhead, with FedAPA achieving at least a 9.65% increase in accuracy, a 9% gain in F1 score, a 0.29 reduction in MAE, and a 95.94% reduction in communication overhead.
Authors Jingtao Guo, Yuyi Mao, Ivan Wang-Hei Ho
Comment 17 pages, 11 figures, this article was submitted to IEEE for possible publication
Categories Artificial Intelligence
0
Data-Driven Lipschitz Continuity: A Cost-Effective Approach to Improve Adversarial Robustness
Title CN 数据驱动的Lipschitz连续性：一种提高对抗鲁棒性的低成本方法
Keywords 对抗鲁棒性 · Lipschitz连续性 · 深度神经网络 · 高效训练 · 安全
Summary 本文提出了一种基于Lipschitz连续性的数据驱动方法，用于提升深度神经网络的对抗鲁棒性。该方法无需额外生成对抗样本，仅通过单次数据遍历即可增强模型防御能力，显著降低计算开销，并可与现有对抗训练框架结合。实验表明其在减少计算成本的同时保持甚至提升了模型鲁棒性。
Reason 该论文研究的是通用深度神经网络的对抗鲁棒性问题，聚焦于通过Lipschitz连续性提升模型安全性，属于通用AI安全领域。虽然其方法可能在理论上对任何DNN有帮助，但并未针对文档图像理解（DIU）中的具体痛点（如OCR幻觉、复杂排版、细粒度定位等）进行设计或验证。且未涉及多模态、视觉-语言建模、高分辨率图像处理或文档结构理解等DIU核心挑战，无法直接迁移用于解决文档理解任务。属于无关领域的通用技术，无明确迁移路径。
Abstract As deep neural networks (DNNs) are increasingly deployed in sensitive applications, ensuring their security and robustness has become critical. A major threat to DNNs arises from adversarial attacks, where small input perturbations can lead to incorrect predictions. Recent advances in adversarial training improve robustness by incorporating additional examples from external datasets or generative models. However, these methods often incur high computational costs, limiting their practicality and hindering real-world deployment. In this paper, we propose a cost-efficient alternative based on Lipschitz continuity that achieves robustness comparable to models trained with extensive supplementary data. Unlike conventional adversarial training, our method requires only a single pass over the dataset without gradient estimation, making it highly efficient. Furthermore, our method can integrate seamlessly with existing adversarial training frameworks and enhances the robustness of models without requiring extra generative data. Experimental results show that our approach not only reduces computational overhead but also maintains or improves the defensive capabilities of robust neural networks. This work opens a promising direction for developing practical, scalable defenses against adversarial attacks.
Authors Erh-Chung Chen, Pin-Yu Chen, I-Hsin Chung, Che-Rung Lee
Categories Artificial Intelligence
0
IVY-FAKE: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detection
Title CN IVY-FAKE：一种用于图像和视频AIGC检测的统一可解释框架与基准
Keywords AIGC检测 · 可解释性 · 多模态基准 · 伪造检测 · 强化学习
Summary 本文提出了Ivy-Fake，一个大规模多模态基准数据集，用于图像和视频中的AI生成内容（AIGC）检测，包含超过10.6万标注样本和5,000人工验证测试样本。同时提出Ivy-xDetector模型，基于分组相对策略优化（GRPO）的强化学习方法，实现可解释的推理链，在多个AIGC检测任务上取得SOTA性能。
Reason 该论文聚焦于AIGC内容检测，属于图像/视频伪造检测领域，与文档图像理解（DIU）任务无直接关联。尽管其提出的方法涉及多模态推理和可解释性，但目标场景为合成媒体识别，无法迁移解决OCR幻觉、复杂排版理解或长文档推理等DIU核心问题。
Abstract The rapid development of Artificial Intelligence Generated Content (AIGC) techniques has enabled the creation of high-quality synthetic content, but it also raises significant security concerns. Current detection methods face two major limitations: (1) the lack of multidimensional explainable datasets for generated images and videos. Existing open-source datasets (e.g., WildFake, GenVideo) rely on oversimplified binary annotations, which restrict the explainability and trustworthiness of trained detectors. (2) Prior MLLM-based forgery detectors (e.g., FakeVLM) exhibit insufficiently fine-grained interpretability in their step-by-step reasoning, which hinders reliable localization and explanation. To address these challenges, we introduce Ivy-Fake, the first large-scale multimodal benchmark for explainable AIGC detection. It consists of over 106K richly annotated training samples (images and videos) and 5,000 manually verified evaluation examples, sourced from multiple generative models and real world datasets through a carefully designed pipeline to ensure both diversity and quality. Furthermore, we propose Ivy-xDetector, a reinforcement learning model based on Group Relative Policy Optimization (GRPO), capable of producing explainable reasoning chains and achieving robust performance across multiple synthetic content detection benchmarks. Extensive experiments demonstrate the superiority of our dataset and confirm the effectiveness of our approach. Notably, our method improves performance on GenImage from 86.88% to 96.32%, surpassing prior state-of-the-art methods by a clear margin.
Authors Changjiang Jiang, Wenhui Dong, Zhonghao Zhang, Chenyang Si, Fengchang Yu, Wei Peng, Xinbin Yuan, Yifei Bi, Ming Zhao, Zian Zhou, Caifeng Shan
Comment 30 pages
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
EmoFeedback$^2$: Reinforcement of Continuous Emotional Image Generation via LVLM-based Reward and Textual Feedback
Title CN EmoFeedback²：基于LVLM奖励和文本反馈的连续情感图像生成强化
Keywords 情感图像生成 · 视觉语言模型 · 强化学习 · 反馈机制 · 文本到图像生成
Summary 本文提出EmoFeedback²框架，通过微调大型视觉语言模型（LVLM）为连续情感图像生成提供奖励信号和文本反馈，以增强生成图像的情感连续性和保真度。LVLM被用于评估图像情感并给出改进提示，从而指导生成模型的强化学习优化。实验表明该方法在自建数据集上优于现有方法。
Reason 该论文研究的是连续情感图像生成（C-EICG），属于纯图像生成领域，且核心任务是情感控制下的图像合成，与文档图像理解（DIU）、DocVQA、OCR、布局分析等无关。尽管使用了LVLM进行反馈强化，但其方法应用于图像生成的质量提升，并非解决文档理解中的幻觉、密集文本、复杂排版或长文档推理等问题。属于‘❌无关领域’中的‘纯图像生成/修复’类别，直接打0分。
Abstract Continuous emotional image generation (C-EICG) is emerging rapidly due to its ability to produce images aligned with both user descriptions and continuous emotional values. However, existing approaches lack emotional feedback from generated images, limiting the control of emotional continuity. Additionally, their simple alignment between emotions and naively generated texts fails to adaptively adjust emotional prompts according to image content, leading to insufficient emotional fidelity. To address these concerns, we propose a novel generation-understanding-feedback reinforcement paradigm (EmoFeedback$^2$) for C-EICG, which exploits the reasoning capability of the fine-tuned large vision-language model (LVLM) to provide reward and textual feedback for generating high-quality images with continuous emotions. Specifically, we introduce an emotion-aware reward feedback strategy, where the LVLM evaluates the emotional values of generated images and computes the reward against target emotions, guiding the reinforcement fine-tuning of the generative model and enhancing the emotional continuity of images. Furthermore, we design a self-promotion textual feedback framework, in which the LVLM iteratively analyzes the emotional content of generated images and adaptively produces refinement suggestions for the next-round prompt, improving the emotional fidelity with fine-grained content. Extensive experimental results demonstrate that our approach effectively generates high-quality images with the desired emotions, outperforming existing state-of-the-art methods in our custom dataset. The code and dataset will be released soon.
Authors Jingyang Jia, Kai Shu, Gang Yang, Long Xing, Xun Chen, Aiping Liu
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Probabilistic Wildfire Spread Prediction Using an Autoregressive Conditional Generative Adversarial Network
Title CN 使用自回归条件生成对抗网络进行概率性野火蔓延预测
Keywords 野火预测 · 生成对抗网络 · 自回归模型 · 气候灾害 · 时空预测
Summary 本文提出一种基于自回归条件生成对抗网络（CGAN）的概率性野火蔓延预测模型，通过将预测任务建模为序列状态转移问题，提升长期预测的稳定性。实验表明，该方法在整体精度和火线边界的刻画上优于传统深度学习模型，能更好地捕捉野火传播的非线性和不确定性，具有较高的物理可解释性，适用于应急响应与疏散规划。
Reason 该论文研究的是野火蔓延预测，属于环境建模与灾害预测领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。尽管使用了CGAN和自回归建模等技术，但其方法针对的是地理空间动态演化问题，无法迁移至解决文档中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Climate change has intensified the frequency and severity of wildfires, making rapid and accurate prediction of fire spread essential for effective mitigation and response. Physics-based simulators such as FARSITE offer high-fidelity predictions but are computationally intensive, limiting their applicability in real-time decision-making, while existing deep learning models often yield overly smooth predictions that fail to capture the complex, nonlinear dynamics of wildfire propagation. This study proposes an autoregressive conditional generative adversarial network (CGAN) for probabilistic wildfire spread prediction. By formulating the prediction task as an autoregressive problem, the model learns sequential state transitions, ensuring long-term prediction stability. Experimental results demonstrate that the proposed CGAN-based model outperforms conventional deep learning models in both overall predictive accuracy and boundary delineation of fire perimeters. These results demonstrate that adversarial learning allows the model to capture the strong nonlinearity and uncertainty of wildfire spread, instead of simply fitting the pixel average. Furthermore, the autoregressive framework facilitates systematic temporal forecasting of wildfire evolution. The proposed CGAN-based autoregressive framework enhances both the accuracy and physical interpretability of wildfire spread prediction, offering a promising foundation for time-sensitive response and evacuation planning.
Authors Taehoon Kang, Taeyong Kim
Comment 22 pages, 15 figures, Submitted to Journal of Environmental Management
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Uncertainty-Aware Deep Learning Framework for Remaining Useful Life Prediction in Turbofan Engines with Learned Aleatoric Uncertainty
Title CN 用于涡轮风扇发动机剩余使用寿命预测的不确定性感知深度学习框架
Keywords 剩余使用寿命 · 不确定性量化 · 深度学习 · 时间序列预测 · 航空发动机
Summary 本文提出了一种不确定性感知的深度学习框架，用于涡轮风扇发动机的剩余使用寿命（RUL）预测。该方法通过贝叶斯输出层同时预测RUL均值和方差，结合Inception模块、双向LSTM和双层注意力机制，在NASA CMAPSS数据集上实现了在关键阶段（RUL ≤ 30）的显著性能提升，并提供了校准良好的置信区间。然而，该工作属于工业设备故障预测领域，与文档理解无关。
Reason 该论文属于航空航天领域的预测性维护任务，研究内容为基于传感器数据的RUL（剩余使用寿命）预测，并引入不确定性建模。尽管其方法涉及深度学习和不确定性量化，但完全不涉及文档图像、多模态语言模型或视觉信息处理，与文档图像理解（DIU）或DocVQA无任何关联。
Abstract Accurate Remaining Useful Life (RUL) prediction coupled with uncertainty quantification remains a critical challenge in aerospace prognostics. This research introduces a novel uncertainty-aware deep learning framework that learns aleatoric uncertainty directly through probabilistic modeling, an approach unexplored in existing CMAPSS-based literature. Our hierarchical architecture integrates multi-scale Inception blocks for temporal pattern extraction, bidirectional Long Short-Term Memory networks for sequential modeling, and a dual-level attention mechanism operating simultaneously on sensor and temporal dimensions. The innovation lies in the Bayesian output layer that predicts both mean RUL and variance, enabling the model to learn data-inherent uncertainty. Comprehensive preprocessing employs condition-aware clustering, wavelet denoising, and intelligent feature selection. Experimental validation on NASA CMAPSS benchmarks (FD001-FD004) demonstrates competitive overall performance with RMSE values of 16.22, 19.29, 16.84, and 19.98 respectively. Remarkably, our framework achieves breakthrough critical zone performance (RUL <= 30 cycles) with RMSE of 5.14, 6.89, 5.27, and 7.16, representing 25-40 percent improvements over conventional approaches and establishing new benchmarks for safety-critical predictions. The learned uncertainty provides well-calibrated 95 percent confidence intervals with coverage ranging from 93.5 percent to 95.2 percent, enabling risk-aware maintenance scheduling previously unattainable in CMAPSS literature.
Authors Krishang Sharma
Comment 10 pages, 2 figures, 3 tables
Categories Artificial Intelligence
0
Heterogeneous Multi-Agent Proximal Policy Optimization for Power Distribution System Restoration TPEC 2025 Conference
Title CN 用于配电系统恢复的异构多智能体近端策略优化
Keywords 强化学习 · 多智能体系统 · 配电系统恢复 · HAPPO · 电力系统
Summary 本文提出一种基于异构多智能体强化学习（HARL）的配电系统恢复框架HAPPO，通过去中心化策略与集中式评论家实现多个微电网在断电后的协同恢复。实验表明该方法在IEEE标准测试系统上优于多种基线算法，具有更好的收敛性和恢复能力。
Reason 该论文属于具身智能与电力系统控制领域，研究的是基于强化学习的多智能体在物理电网中的协同恢复，与文档图像理解（DIU）或视觉语言模型（VLM）无任何技术关联。其方法无法迁移到解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心问题。
Abstract Restoring power distribution systems (PDS) after large-scale outages requires sequential switching operations that reconfigure feeder topology and coordinate distributed energy resources (DERs) under nonlinear constraints such as power balance, voltage limits, and thermal ratings. These challenges make conventional optimization and value-based RL approaches computationally inefficient and difficult to scale. This paper applies a Heterogeneous-Agent Reinforcement Learning (HARL) framework, instantiated through Heterogeneous-Agent Proximal Policy Optimization (HAPPO), to enable coordinated restoration across interconnected microgrids. Each agent controls a distinct microgrid with different loads, DER capacities, and switch counts, introducing practical structural heterogeneity. Decentralized actor policies are trained with a centralized critic to compute advantage values for stable on-policy updates. A physics-informed OpenDSS environment provides full power flow feedback and enforces operational limits via differentiable penalty signals rather than invalid action masking. The total DER generation is capped at 2400 kW, and each microgrid must satisfy local supply-demand feasibility. Experiments on the IEEE 123-bus and IEEE 8500-node systems show that HAPPO achieves faster convergence, higher restored power, and smoother multi-seed training than DQN, PPO, MAES, MAGDPG, MADQN, Mean-Field RL, and QMIX. Results demonstrate that incorporating microgrid-level heterogeneity within the HARL framework yields a scalable, stable, and constraint-aware solution for complex PDS restoration.
Authors Parya Dolatyabi, Ali Farajzadeh Bavil, Mahdi Khodayar
Comment 6 pages, 4 figures, TPEC 2025 Conference
Categories Artificial Intelligence
0
Automated Neural Architecture Design for Industrial Defect Detection
Title CN 面向工业缺陷检测的自动化神经网络架构设计
Keywords 神经架构搜索 · 工业缺陷检测 · 多尺度特征聚合 · 混合模型 · 延迟感知
Summary 本文提出AutoNAD，一种用于工业表面缺陷检测的自动化神经网络架构设计框架，通过联合搜索卷积、Transformer和MLP结构来捕捉局部细节与长距离语义信息，并引入跨权重共享策略和可搜索的多级特征聚合模块以提升训练效率与性能。同时结合延迟感知先验以优化推理速度。在三个工业缺陷数据集上验证了有效性。
Reason 该论文研究的是工业表面缺陷检测（SDD）中的神经网络架构搜索，属于工业检测领域，与文档图像理解（DIU）或DocVQA无直接关联。尽管其方法涉及视觉模型设计和多尺度特征学习，但任务目标、数据形态和核心挑战（如缺陷形状多样性）均与文档理解中的OCR幻觉、复杂版式、长文档推理等痛点无关，无法迁移为DIU的底层方法。
Abstract Industrial surface defect detection (SDD) is critical for ensuring product quality and manufacturing reliability. Due to the diverse shapes and sizes of surface defects, SDD faces two main challenges: intraclass difference and interclass similarity. Existing methods primarily utilize manually designed models, which require extensive trial and error and often struggle to address both challenges effectively. To overcome this, we propose AutoNAD, an automated neural architecture design framework for SDD that jointly searches over convolutions, transformers, and multi-layer perceptrons. This hybrid design enables the model to capture both fine-grained local variations and long-range semantic context, addressing the two key challenges while reducing the cost of manual network design. To support efficient training of such a diverse search space, AutoNAD introduces a cross weight sharing strategy, which accelerates supernet convergence and improves subnet performance. Additionally, a searchable multi-level feature aggregation module (MFAM) is integrated to enhance multi-scale feature learning. Beyond detection accuracy, runtime efficiency is essential for industrial deployment. To this end, AutoNAD incorporates a latency-aware prior to guide the selection of efficient architectures. The effectiveness of AutoNAD is validated on three industrial defect datasets and further applied within a defect imaging and detection platform. Code is available at https://github.com/Yuxi104/AutoNAD.
Authors Yuxi Liu, Yunfeng Ma, Yi Tang, Min Liu, Shuai Jiang, Yaonan Wang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Subgoal Graph-Augmented Planning for LLM-Guided Open-World Reinforcement Learning
Title CN 用于大语言模型引导的开放世界强化学习的子目标图增强规划
Keywords 大语言模型 · 强化学习 · 子目标规划 · 规划-执行对齐 · 多LLM框架
Summary 本文提出SGA-ACR框架，通过构建环境特定的子目标图并结合多LLM流水线（生成、批判、精炼）来提升LLM在强化学习中的规划执行对齐性，并在'Crafter'游戏中验证了有效性。核心贡献在于改善抽象计划与实际动作之间的脱节问题。
Reason 该论文属于具身智能与强化学习领域，研究的是LLM在开放世界游戏'Crafter'中的任务规划问题。尽管其提出了多LLM生成-批判-精炼框架和子目标图结构，但整体工作与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析或长文档推理无直接关联。所提方法难以迁移到文档理解场景，且未涉及多模态输入处理或视觉信息对齐等DIU关键技术。
Abstract Large language models (LLMs) offer strong high-level planning capabilities for reinforcement learning (RL) by decomposing tasks into subgoals. However, their practical utility is limited by poor planning-execution alignment, which reflects a critical gap between abstract plans and actionable, environment-compatible behaviors. This misalignment arises from two interrelated limitations: (1) LLMs often produce subgoals that are semantically plausible but infeasible or irrelevant in the target environment due to insufficient grounding in environment-specific knowledge, and (2) single-LLM planning conflates generation with self-verification, resulting in overconfident yet unreliable subgoals that frequently fail during execution. To address these challenges, we propose Subgoal Graph-Augmented Actor-Critic-Refiner (SGA-ACR), a framework that integrates an environment-specific subgoal graph and structured entity knowledge with a multi-LLM planning pipeline that explicitly separates generation, critique, and refinement to produce executable and verifiable subgoals. A subgoal tracker further monitors execution progress, provides auxiliary rewards, and adaptively updates the subgoal graph to maintain alignment between plans and actions. Experimental results on 22 diverse tasks in the open-world game "Crafter" demonstrate the effectiveness of our proposed method.
Authors Shanwei Fan
Categories Artificial Intelligence
0
Contrast-Prior Enhanced Duality for Mask-Free Shadow Removal
Title CN 基于对比先验增强对偶的无掩码阴影去除
Keywords 阴影去除 · 对比度先验 · 双分支注意力 · 频率融合 · 扩散模型
Summary 本文提出了一种无掩码阴影去除方法，通过引入自适应门控双分支注意力（AGBA）机制动态加权对比度先验，并结合基于扩散的频域-对比融合网络（FCFN）来恢复软阴影边界和细节。实验表明该方法在无掩码设置下达到SOTA水平。
Reason 该论文研究的是自然图像中的阴影去除问题，属于通用图像修复领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的AGBA和FCFN模块在高分辨率细节恢复上有一定技术价值，但方法针对的是光照与材质建模问题，无法迁移解决文档图像中的OCR幻觉、密集文本解析或复杂排版理解等关键痛点。
Abstract Existing shadow removal methods often rely on shadow masks, which are challenging to acquire in real-world scenarios. Exploring intrinsic image cues, such as local contrast information, presents a potential alternative for guiding shadow removal in the absence of explicit masks. However, the cue's inherent ambiguity becomes a critical limitation in complex scenes, where it can fail to distinguish true shadows from low-reflectance objects and intricate background textures. To address this motivation, we propose the Adaptive Gated Dual-Branch Attention (AGBA) mechanism. AGBA dynamically filters and re-weighs the contrast prior to effectively disentangle shadow features from confounding visual elements. Furthermore, to tackle the persistent challenge of restoring soft shadow boundaries and fine-grained details, we introduce a diffusion-based Frequency-Contrast Fusion Network (FCFN) that leverages high-frequency and contrast cues to guide the generative process. Extensive experiments demonstrate that our method achieves state-of-the-art results among mask-free approaches while maintaining competitive performance relative to mask-based methods.
Authors Jiyu Wu, Yifan Liu, Jiancheng Huang, Mingfu Yan, Shifeng Chen
Comment There are unresolved authorship disputes related to this submission, and the current version does not reflect an agreed authorship list
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
FRAGMENTA: End-to-end Fragmentation-based Generative Model with Agentic Tuning for Drug Lead Optimization
Title CN FRAGMENTA：基于片段的端到端生成模型与智能体调优用于药物先导优化
Keywords 分子生成 · 药物发现 · 智能体调优 · 片段化建模 · Q学习
Summary 本文提出FRAGMENTA，一种用于药物先导优化的端到端生成模型，通过将分子片段化建模为词汇选择问题，并结合基于对话反馈的智能体系统实现自动调优。在癌症药物实验中表现优于基线方法。然而，该工作局限于分子生成领域，无适用于文档图像理解的技术迁移价值。
Reason 该论文属于具身智能/机器人与生物化学交叉领域的垂直应用，研究内容为分子生成和药物发现，使用智能体进行专家反馈调优。尽管提到了‘agentic tuning’，但其方法论完全围绕药物化学设计，并未提出可迁移至文档图像理解（如DocVQA、布局分析等）的通用架构、训练策略或推理机制。属于典型的‘平行下游应用’，且领域与文档无关。
Abstract Molecule generation using generative AI is vital for drug discovery, yet class-specific datasets often contain fewer than 100 training examples. While fragment-based models handle limited data better than atom-based approaches, existing heuristic fragmentation limits diversity and misses key fragments. Additionally, model tuning typically requires slow, indirect collaboration between medicinal chemists and AI engineers. We introduce FRAGMENTA, an end-to-end framework for drug lead optimization comprising: 1) a novel generative model that reframes fragmentation as a "vocabulary selection" problem, using dynamic Q-learning to jointly optimize fragmentation and generation; and 2) an agentic AI system that refines objectives via conversational feedback from domain experts. This system removes the AI engineer from the loop and progressively learns domain knowledge to eventually automate tuning. In real-world cancer drug discovery experiments, FRAGMENTA's Human-Agent configuration identified nearly twice as many high-scoring molecules as baselines. Furthermore, the fully autonomous Agent-Agent system outperformed traditional Human-Human tuning, demonstrating the efficacy of agentic tuning in capturing expert intent.
Authors Yuto Suzuki, Paul Awolade, Daniel V. LaBarbera, Farnoush Banaei-Kashani
Categories Artificial Intelligence
0
SOAP: Enhancing Spatio-Temporal Relation and Motion Information Capturing for Few-Shot Action Recognition ACM MM 2024
Title CN SOAP：增强少样本动作识别中的时空关系和运动信息捕捉
Keywords 少样本动作识别 · 时空关系 · 运动信息捕捉 · 高帧率视频 · 帧元组
Summary 本文提出了一种名为SOAP的插件式架构，用于提升少样本动作识别中对时空关系和运动信息的捕捉能力。通过在不同特征通道间建立时间连接，并利用包含多帧的帧元组来捕获更丰富的运动信息，从而克服传统方法仅依赖相邻帧导致的信息不足问题。实验表明该方法在多个主流动作识别数据集上达到SOTA性能。
Reason 该论文研究的是视频领域的少样本动作识别（FSAR），其核心问题为高帧率视频下的时空关系建模与运动信息密度不足，提出的方法基于帧元组（frame tuples）增强建模。尽管其方法具有一定的通用性，但属于视频理解领域，与文档图像理解（DIU）任务无直接关联。所解决的痛点（如动作识别中的时序稀疏性）无法迁移到文档理解中的OCR幻觉、密集文本、复杂排版或长文档推理等核心问题。
Abstract High frame-rate (HFR) videos of action recognition improve fine-grained expression while reducing the spatio-temporal relation and motion information density. Thus, large amounts of video samples are continuously required for traditional data-driven training. However, samples are not always sufficient in real-world scenarios, promoting few-shot action recognition (FSAR) research. We observe that most recent FSAR works build spatio-temporal relation of video samples via temporal alignment after spatial feature extraction, cutting apart spatial and temporal features within samples. They also capture motion information via narrow perspectives between adjacent frames without considering density, leading to insufficient motion information capturing. Therefore, we propose a novel plug-and-play architecture for FSAR called Spatio-tempOral frAme tuPle enhancer (SOAP) in this paper. The model we designed with such architecture refers to SOAP-Net. Temporal connections between different feature channels and spatio-temporal relation of features are considered instead of simple feature extraction. Comprehensive motion information is also captured, using frame tuples with multiple frames containing more motion information than adjacent frames. Combining frame tuples of diverse frame counts further provides a broader perspective. SOAP-Net achieves new state-of-the-art performance across well-known benchmarks such as SthSthV2, Kinetics, UCF101, and HMDB51. Extensive empirical evaluations underscore the competitiveness, pluggability, generalization, and robustness of SOAP. The code is released at https://github.com/wenbohuang1002/SOAP.
Authors Wenbo Huang, Jinghui Zhang, Xuwei Qian, Zhen Wu, Meng Wang, Lei Zhang
Comment Accepted by ACM MM 2024
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Even with AI, Bijection Discovery is Still Hard: The Opportunities and Challenges of OpenEvolve for Novel Bijection Construction FPSAC 2026
Title CN 即使有AI，双射发现仍然困难：OpenEvolve在新型双射构造中的机遇与挑战
Keywords 程序合成 · 数学发现 · LLM协作 · 组合双射 · AlphaEvolve
Summary 本文探讨了使用OpenEvolve系统在组合数学中进行双射发现的潜力与局限，测试了三个Dyck路径相关的双射构造问题，发现当前前沿系统仍难以独立完成研究级的双射构造，强调人类数学家参与的必要性。
Reason 该论文属于数学组合学领域的AI辅助研究，聚焦于双射构造问题，使用LLM进行程序合成以辅助数学发现。其任务和方法均远离文档图像理解（DIU）核心问题，如OCR、布局分析、表格识别或视觉-语言推理。尽管涉及LLM协同与演化机制，但未提出可迁移至DIU领域的通用架构、训练策略或推理机制。属于特定科学计算场景的平行应用，对DIU无直接方法论贡献。
Abstract Evolutionary program synthesis systems such as AlphaEvolve, OpenEvolve, and ShinkaEvolve offer a new approach to AI-assisted mathematical discovery. These systems utilize teams of large language models (LLMs) to generate candidate solutions to a problem as human readable code. These candidate solutions are then 'evolved' with the goal of improving them beyond what an LLM can produce in a single shot. While existing mathematical applications have mostly focused on problems of establishing bounds (e.g., sphere packing), the program synthesis approach is well suited to any problem where the solution takes the form of an explicit construction. With this in mind, in this paper we explore the use of OpenEvolve for combinatorial bijection discovery. We describe the results of applying OpenEvolve to three bijection construction problems involving Dyck paths, two of which are known and one of which is open. We find that while systems like OpenEvolve show promise as a valuable tool for combinatorialists, the problem of finding novel, research-level bijections remains a challenging task for current frontier systems, reinforcing the need for human mathematicians in the loop. We describe some lessons learned for others in the field interested in exploring the use of these systems.
Authors Davis Brown, Jesse He, Helen Jenne, Henry Kvinge, Max Vargas
Comment 16 pages, 3 figures. This is an extended abstract submitted to FPSAC 2026
Categories Artificial Intelligence
0
Human Experts' Evaluation of Generative AI for Contextualizing STEAM Education in the Global South
Title CN 人类专家对生成式AI在南部全球STEAM教育情境化中应用的评估
Keywords 生成式AI · STEAM教育 · 文化情境化 · 课程设计 · 教育公平
Summary 本研究探讨了人类专家如何评估生成式AI在加纳STEAM教育情境化中的能力。四位STEAM专家使用文化响应教学量规评估了由定制化工具CRLP生成的课程计划，并与加纳国家标准课程进行对比。结果显示，AI辅助的课程在文化表征和教学响应性方面优于标准课程，但在数学和计算机领域对文化多样性的体现较浅。研究强调需要教师介入和基于本土语料的模型微调。
Reason 该论文属于平行下游应用，研究的是如何使用生成式AI辅助设计加纳的STEAM课程，属于教育领域的具体应用场景。尽管涉及AI与文化适配性问题，但并未提出任何文档图像理解（DIU）或视觉语言模型（VLM）的底层技术改进，也未针对OCR幻觉、复杂排版、细粒度定位等DIU核心痛点提供可迁移的方法论创新。
Abstract This study investigates how human experts evaluate the capacity of Generative AI (GenAI) to contextualize STEAM education in the Global South, with a focus on Ghana. Using a convergent mixed-methods design, four STEAM specialists assessed GenAI-generated lesson plans created with a customized Culturally Responsive Lesson Planner (CRLP) and compared them to standardized lesson plans from the Ghana National Council for Curriculum and Assessment (NaCCA). Quantitative ratings were based on a validated 25-item Culturally Responsive Pedagogy Rubric measuring bias awareness, cultural representation, contextual relevance, linguistic responsiveness, and teacher agency. Qualitative reflections provided additional insight into how GenAI handles cultural and pedagogical appropriateness. Findings show that GenAI, when paired with the CRLP tool, can support contextualized STEAM instruction by linking abstract curriculum standards to learners' cultural knowledge, community practices, and everyday experiences. Experts rated GenAI-assisted lessons as more culturally grounded and pedagogically responsive than NaCCA plans, integrating Indigenous knowledge, bilingual elements, and locally relevant examples. However, GenAI struggled to represent Ghana's cultural pluralism, often offering surface-level references to language, history, and identity. These weaknesses were most evident in Mathematics and Computing, where cultural nuance was limited. The results highlight the need for continued teacher mediation, community involvement, and culturally attuned refinement of AI outputs. Future work should include classroom trials, expanded expert participation, and model fine-tuning using Indigenous language corpora to strengthen cultural fidelity in Global South contexts.
Authors Matthew Nyaaba, Macharious Nabang, Patrick Kyeremeh, Ibrahim Nantomah, Collins Owusu-Fordjour, Martin Ako, Bismark Nyaaba Akanzire, Kassim Korah Nantomah, Cecilia Issaka, Xiaoming Zhai
Categories Artificial Intelligence
0
PrefixGPT: Prefix Adder Optimization by a Generative Pre-trained Transformer AAAI-2026
Title CN PrefixGPT：通过生成式预训练Transformer优化前缀加法器
Keywords PrefixGPT · Transformer · 硬件设计自动化 · 前缀加法器 · 生成模型
Summary 本文提出PrefixGPT，一种基于生成式预训练Transformer的模型，用于从零开始生成优化的前缀加法器。通过将加法器拓扑表示为二维坐标序列，并在生成过程中引入合法性掩码，确保所有输出设计均符合规则。模型先在随机合成的有效加法器数据上预训练，再微调以优化设计质量。实验表明，该方法不仅找到了ADP提升7.7%的新最优设计，且平均ADP降低达79.1%。
Reason 该论文属于硬件电路设计领域，研究如何用Transformer生成优化的前缀加法器结构。尽管使用了GPT架构，但其任务与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用完全无关，属于典型的‘平行领域应用’而非方法论创新。无法迁移至OCR、DocVQA、布局分析等任何DIU核心问题。
Abstract Prefix adders are widely used in compute-intensive applications for their high speed. However, designing optimized prefix adders is challenging due to strict design rules and an exponentially large design space. We introduce PrefixGPT, a generative pre-trained Transformer (GPT) that directly generates optimized prefix adders from scratch. Our approach represents an adder's topology as a two-dimensional coordinate sequence and applies a legality mask during generation, ensuring every design is valid by construction. PrefixGPT features a customized decoder-only Transformer architecture. The model is first pre-trained on a corpus of randomly synthesized valid prefix adders to learn design rules and then fine-tuned to navigate the design space for optimized design quality. Compared with existing works, PrefixGPT not only finds a new optimal design with a 7.7% improved area-delay product (ADP) but exhibits superior exploration quality, lowering the average ADP by up to 79.1%. This demonstrates the potential of GPT-style models to first master complex hardware design principles and then apply them for more efficient design optimization.
Authors Ruogu Ding, Xin Ning, Ulf Schlichtmann, Weikang Qian
Comment This is an extended version of the paper accepted by the AAAI-2026 Conference
Categories Artificial Intelligence
0
WeatherDiffusion: Controllable Weather Editing in Intrinsic Space
Title CN WeatherDiffusion：在本征空间中进行可控天气编辑
Keywords 图像生成 · 扩散模型 · 本征图像分解 · 天气编辑 · 计算机视觉
Summary 本文提出WeatherDiffusion，一种基于扩散模型的可控天气编辑框架，通过逆向渲染估计材质、几何和光照等本征图，并结合文本提示进行前向渲染以生成特定天气条件下的图像。引入了本征图感知注意力机制和CLIP空间插值策略，提升了大尺度户外场景的编辑质量。尽管在合成与真实数据集上表现优异，但其应用目标和方法均远离文档理解领域。
Reason 该论文属于纯图像生成与编辑领域，聚焦于户外场景的天气控制与渲染，其技术路线基于扩散模型和本征图像分解，主要应用于自动驾驶等场景的鲁棒性增强。研究内容与文档图像理解（DIU）的核心任务如文本识别、布局分析、表格理解等完全无关，且不提供可迁移至DIU领域的底层方法（如高分辨率处理、多模态对齐、去幻觉机制等）。属于典型的无关领域工作。
Abstract We present WeatherDiffusion, a diffusion-based framework for controllable weather editing in intrinsic space. Our framework includes two components based on diffusion priors: an inverse renderer that estimates material properties, scene geometry, and lighting as intrinsic maps from an input image, and a forward renderer that utilizes these geometry and material maps along with a text prompt that describes specific weather conditions to generate a final image. The intrinsic maps enhance controllability compared to traditional pixel-space editing approaches.We propose an intrinsic map-aware attention mechanism that improves spatial correspondence and decomposition quality in large outdoor scenes. For forward rendering, we leverage CLIP-space interpolation of weather prompts to achieve fine-grained weather control. We also introduce a synthetic and a real-world dataset, containing 38k and 18k images under various weather conditions, each with intrinsic map annotations. WeatherDiffusion outperforms state-of-the-art pixel-space editing approaches, weather restoration methods, and rendering-based methods, showing promise for downstream tasks such as autonomous driving, enhancing the robustness of detection and segmentation in challenging weather scenarios.
Authors Yixin Zhu, Zuoliang Zhu, Jian Yang, Miloš Hašan, Jin Xie, Beibei Wang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
AI4X Roadmap: Artificial Intelligence for the advancement of scientific pursuit and its future directions
Title CN AI4X路线图：人工智能推动科学探索及其未来方向
Keywords AI for Science · Foundation Models · Self-driving Labs · Active Learning · Scientific Discovery
Summary 本文是一篇关于人工智能赋能科学研究的多领域路线图综述，涵盖了从基础科学到自驱动实验室的发展方向。重点在于AI在不同科学领域的集成潜力，强调数据可信性、可迁移模型和端到端科研流程闭环。但并未聚焦于文档理解任务，也没有提出适用于文档图像分析的具体架构或算法。
Reason 该论文属于跨学科的AI for Science宏观展望，讨论的是人工智能在生物、化学、材料等领域的通用应用框架。其内容未涉及文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的具体方法创新，也未提出可用于解决OCR幻觉、复杂排版或长文档推理的技术工具。属于典型的‘平行应用’类综述，无针对DIU的底层方法贡献。
Abstract Artificial intelligence and machine learning are reshaping how we approach scientific discovery, not by replacing established methods but by extending what researchers can probe, predict, and design. In this roadmap we provide a forward-looking view of AI-enabled science across biology, chemistry, climate science, mathematics, materials science, physics, self-driving laboratories and unconventional computing. Several shared themes emerge: the need for diverse and trustworthy data, transferable electronic-structure and interatomic models, AI systems integrated into end-to-end scientific workflows that connect simulations to experiments and generative systems grounded in synthesisability rather than purely idealised phases. Across domains, we highlight how large foundation models, active learning and self-driving laboratories can close loops between prediction and validation while maintaining reproducibility and physical interpretability. Taken together, these perspectives outline where AI-enabled science stands today, identify bottlenecks in data, methods and infrastructure, and chart concrete directions for building AI systems that are not only more powerful but also more transparent and capable of accelerating discovery in complex real-world environments.
Authors Stephen G. Dale, Nikita Kazeev, Alastair J. A. Price, Victor Posligua, Stephan Roche, O. Anatole von Lilienfeld, Konstantin S. Novoselov, Xavier Bresson, Gianmarco Mengaldo, Xudong Chen, Terence J. O'Kane, Emily R. Lines, Matthew J. Allen, Amandine E. Debus, Clayton Miller, Jiayu Zhou, Hiroko H. Dodge, David Rousseau, Andrey Ustyuzhanin, Ziyun Yan, Mario Lanza, Fabio Sciarrino, Ryo Yoshida, Zhidong Leong, Teck Leong Tan, Qianxiao Li, Adil Kabylda, Igor Poltavsky, Alexandre Tkatchenko, Sherif Abdulkader Tawfik, Prathami Divakar Kamath, Theo Jaffrelot Inizan, Kristin A. Persson, Bryant Y. Li, Vir Karan, Chenru Duan, Haojun Jia, Qiyuan Zhao, Hiroyuki Hayashi, Atsuto Seko, Isao Tanaka, Omar M. Yaghi, Tim Gould, Bun Chan, Stefan Vuckovic, Tianbo Li, Min Lin, Zehcen Tang, Yang Li, Yong Xu, Amrita Joshi, Xiaonan Wang, Leonard W. T. Ng, Sergei V. Kalinin, Mahshid Ahmadi, Jiyizhe Zhang, Shuyuan Zhang, Alexei Lapkin, Ming Xiao, Zhe Wu, Kedar Hippalgaonkar, Limsoon Wong, Lorenzo Bastonero, Nicola Marzari, Dorye Luis Esteras Cordoba, Andrei Tomut, Alba Quinones Andrade, Jose-Hugo Garcia
Categories Artificial Intelligence
0
KRAL: Knowledge and Reasoning Augmented Learning for LLM-assisted Clinical Antimicrobial Therapy
Title CN KRAL：用于大语言模型辅助临床抗菌治疗的知识与推理增强学习
Keywords 临床决策支持 · 知识蒸馏 · 强化学习 · RAG · 大语言模型
Summary 本文提出KRAL框架，通过知识与推理增强学习提升大语言模型在临床抗菌治疗中的决策能力。该方法利用教师模型反向生成问答路径进行知识蒸馏，结合启发式学习实现半监督数据增强，并采用基于Agent的强化学习优化推理能力，在降低训练成本的同时提升了医学问答与推理性能。然而，该工作聚焦于医疗领域的特定应用，缺乏对文档理解任务的通用方法贡献。
Reason 该论文属于典型的‘平行下游应用’，将现有LLM技术（如RAG、SFT、强化学习）应用于临床抗菌治疗决策这一特定领域。尽管提出了知识蒸馏和半监督数据增强等方法，但其核心创新集中在医疗垂类场景的流程设计，并未涉及文档图像理解中的OCR幻觉、复杂排版处理、视觉-语言对齐或高分辨率输入建模等关键问题。无法为DIU提供可迁移的方法论突破。
Abstract Clinical antimicrobial therapy requires the dynamic integration of pathogen profiles,host factors, pharmacological properties of antimicrobials,and the severity of infection. This complexity imposes fundamental limitations on the applicability of Large Language Models (LLMs) in high-stakes clinical decision-making including knowledge gaps, data privacy concerns, high deployment costs, and limited reasoning capabilities. To address these challenges, we propose KRAL (Knowledge and Reasoning Augmented Learning), a low-cost, scalable, privacy-preserving paradigm that leverages teacher-model reasoning to automatically distill knowledge and reasoning trajectories via answer-to-question reverse generation, employs heuristic learning for semi-supervised data augmentation (reducing manual annotation requirements by approximately 80%), and utilizes agentic reinforcement learning to jointly enhance medical knowledge and reasoning while optimizing computational and memory efficiency. A hierarchical evaluation employing diverse teacher-model proxies reduces assessment costs, while modular interface design facilitates seamless system updates. Experimental results demonstrate that KRAL significantly outperforms traditional Retrieval-Augmented Generation (RAG) and Supervised Fine-Tuning (SFT) methods. It improves knowledge question-answering capability (Accuracy@1 on the external open-source benchmark MEDQA increased by 1.8% vs. SFT and 3.6% vs. RAG) and reasoning capability (Pass@1 on the external benchmark PUMCH Antimicrobial increased by 27% vs. SFT and 27.2% vs. RAG), achieved at about 20% of SFT's long-term training costs. This establishes KRAL as an effective solution for enhancing local LLMs' clinical diagnostic capabilities, enabling low-cost, high-safety deployment in complex medical decision support.
Authors Zhe Li, Yehan Qiu, Yujie Chen, Xiang Zhou
Categories Artificial Intelligence
0
SpaceX: Exploring metrics with the SPACE model for developer productivity
Title CN SpaceX：通过SPACE模型探索开发者生产力的度量
Keywords 开发者生产力 · SPACE框架 · 情感分析 · 协作网络 · 软件工程
Summary 本文基于开源仓库数据，采用GLMM和RoBERTa情感分类等方法，通过SPACE框架构建了一个多维度的开发者生产力综合评分（CPS）。研究发现负面情绪状态与提交频率存在显著正相关，并指出贡献者交互拓扑比传统基于数量的指标更能准确反映协作动态。但该工作不涉及任何形式的文档图像理解或多模态建模。
Reason 该论文研究的是软件工程领域中的开发者生产力评估问题，使用情感分析和协作网络拓扑来构建复合生产力指标。其核心任务与文档图像理解（DIU）、视觉语言模型（VLM）、OCR、布局分析、表格识别等完全无关，属于典型的平行应用类研究（在特定领域应用现有AI技术），且未涉及任何多模态、视觉或文档处理内容。
Abstract This empirical investigation elucidates the limitations of deterministic, unidimensional productivity heuristics by operationalizing the SPACE framework through extensive repository mining. Utilizing a dataset derived from open-source repositories, the study employs rigorous statistical methodologies including Generalized Linear Mixed Models (GLMM) and RoBERTa-based sentiment classification to synthesize a holistic, multi-faceted productivity metric. Analytical results reveal a statistically significant positive correlation between negative affective states and commit frequency, implying a cycle of iterative remediation driven by frustration. Furthermore, the investigation has demonstrated that analyzing the topology of contributor interactions yields superior fidelity in mapping collaborative dynamics compared to traditional volume-based metrics. Ultimately, this research posits a Composite Productivity Score (CPS) to address the heterogeneity of developer efficacy.
Authors Sanchit Kaul, Kevin Nhu, Jason Eissayou, Ivan Eser, Victor Borup
Comment Code available at https://github.com/knhu/ECS260Project
Categories Artificial Intelligence
0
Fair Algorithms with Probing for Multi-Agent Multi-Armed Bandits
Title CN 具有探测机制的多智能体多臂老虎机公平算法
Keywords 多智能体系统 · 多臂老虎机 · 公平性 · 探测机制 · 在线学习
Summary 本文提出了一种用于多智能体多臂老虎机（MA-MAB）的框架，旨在在保证各智能体公平性的同时最大化系统整体性能。通过引入一种新的探测机制，在分配前主动获取被选臂的奖励信息。在线下场景中利用子模性质设计了具有理论保证的贪心探测算法；在线上场景中则实现了保持公平性的亚线性遗憾算法。实验表明该方法在合成和真实数据集上均优于基线。
Reason 该论文研究的是多智能体强化学习中的公平性与决策问题，属于多智能体系统理论范畴，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。其提出的探测框架和子模优化方法难以迁移到解决OCR幻觉、复杂排版解析或长文档推理等DIU核心痛点。
Abstract We propose a multi-agent multi-armed bandit (MA-MAB) framework aimed at ensuring fair outcomes across agents while maximizing overall system performance. A key challenge in this setting is decision-making under limited information about arm rewards. To address this, we introduce a novel probing framework that strategically gathers information about selected arms before allocation. In the offline setting, where reward distributions are known, we leverage submodular properties to design a greedy probing algorithm with a provable performance bound. For the more complex online setting, we develop an algorithm that achieves sublinear regret while maintaining fairness. Extensive experiments on synthetic and real-world datasets show that our approach outperforms baseline methods, achieving better fairness and efficiency.
Authors Tianyi Xu, Jiaxin Liu, Nicholas Mattei, Zizhan Zheng
Categories Artificial Intelligence
0
Resilient Charging Infrastructure via Decentralized Coordination of Electric Vehicles at Scale
Title CN 通过大规模电动汽车去中心化协调实现弹性充电基础设施
Keywords 电动汽车 · 去中心化协调 · 集体学习 · 充电调度 · 智能交通
Summary 本文提出一种基于集体学习的去中心化协调框架，用于优化大规模电动汽车在充电站的选择行为，在保障驾驶员舒适度的同时提升系统整体效率。实验表明该方法能有效减少排队时间和旅行时间，增强充电基础设施在站点故障或恶意用户情况下的鲁棒性。研究基于真实世界数据，聚焦于交通调度与能源管理场景。
Reason 该论文属于智能交通与能源系统领域，研究的是电动汽车在充电站的去中心化协调问题，尽管使用了多智能体和学习方法，但其任务、数据和目标均与文档图像理解（DIU）或视觉语言模型（VLM）无关。未涉及OCR、文档布局分析、表格识别、视觉文本理解等任何DIU核心任务，也无法提供可迁移至文档理解领域的底层方法（如视觉压缩、高分辨率处理、幻觉抑制等）。属于典型的平行应用型研究，不符合本方向关注的技术源头创新要求。
Abstract The rapid adoption of electric vehicles (EVs) introduces major challenges for decentralized charging control. Existing decentralized approaches efficiently coordinate a large number of EVs to select charging stations while reducing energy costs, preventing power peak and preserving driver privacy. However, they often struggle under severe contingencies, such as station outages or unexpected surges in charging requests. These situations create competition for limited charging slots, resulting in long queues and reduced driver comfort. To address these limitations, we propose a novel collective learning-based coordination framework that allows EVs to balance individual comfort on their selections against system-wide efficiency, i.e., the overall queues across all stations. In the framework, EVs are recommended for adaptive charging behaviors that shift priority between comfort and efficiency, achieving Pareto-optimal trade-offs under varying station capacities and dynamic spatio-temporal EV distribution. Experiments using real-world data from EVs and charging stations show that the proposed approach outperforms baseline methods, significantly reducing travel and queuing time. The results reveal that, under uncertain charging conditions, EV drivers that behave selfishly or altruistically at the right moments achieve shorter waiting time than those maintaining moderate behavior throughout. Our findings under high fractions of station outages and adversarial EVs further demonstrate improved resilience and trustworthiness of decentralized EV charging infrastructure.
Authors Chuhao Qin, Alexandru Sorici, Andrei Olaru, Evangelos Pournaras, Adina Magda Florea
Comment 14 pages, 12 figures. This work has been submitted to the IEEE for possible publication
Categories Artificial Intelligence
0
ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction
Title CN ENACT：通过自我中心交互的世界建模评估具身认知
Keywords 具身认知 · 视觉语言模型 · 世界建模 · 自我中心视觉 · VQA
Summary 本文提出ENACT基准，用于评估视觉语言模型是否具备具身认知能力，即通过自我中心视角的感知与动作交互来建模世界。任务形式为基于场景图变化的正向与逆向序列重排序，强调对行为后果、可操作性及长期记忆的理解。实验显示当前VLM在长视野交互中显著落后于人类，并表现出右手偏好等人本偏差。
Reason 该论文属于具身智能与机器人领域，研究的是视觉语言模型在模拟环境中是否具备基于交互的具身认知能力。其任务设定（如POMDP、动作-观察序列重排序）和数据来源（BEHAVIOR仿真）均远离文档图像理解的核心问题，如OCR去幻觉、密集文本识别或复杂版面分析。尽管涉及VLM评估，但方法无法迁移至文档场景中的高分辨率处理、细粒度定位或长文档推理等痛点。
Abstract Embodied cognition argues that intelligence arises from sensorimotor interaction rather than passive observation. It raises an intriguing question: do modern vision-language models (VLMs), trained largely in a disembodied manner, exhibit signs of embodied cognition? We introduce ENACT, a benchmark that casts evaluation of embodied cognition as world modeling from egocentric interaction in a visual question answering (VQA) format. Framed as a partially observable Markov decision process (POMDP) whose actions are scene graph changes, ENACT comprises two complementary sequence reordering tasks: forward world modeling (reorder shuffled observations given actions) and inverse world modeling (reorder shuffled actions given observations). While conceptually simple, solving these tasks implicitly demands capabilities central to embodied cognition-affordance recognition, action-effect reasoning, embodied awareness, and interactive, long-horizon memory from partially observable egocentric input, while avoiding low-level image synthesis that could confound the evaluation. We provide a scalable pipeline that synthesizes QA pairs from robotics simulation (BEHAVIOR) and evaluates models on 8,972 QA pairs spanning long-horizon home-scale activities. Experiments reveal a performance gap between frontier VLMs and humans that widens with interaction horizon. Models consistently perform better on the inverse task than the forward one and exhibit anthropocentric biases, including a preference for right-handed actions and degradation when camera intrinsics or viewpoints deviate from human vision. Website at https://enact-embodied-cognition.github.io/.
Authors Qineng Wang, Wenlong Huang, Yu Zhou, Hang Yin, Tianwei Bao, Jianwen Lyu, Weiyu Liu, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, Manling Li
Comment Preprint version
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
0
TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videos
Title CN TraceGen：在3D轨迹空间中进行世界建模以实现跨具身视频的学习
Keywords 世界模型 · 具身智能 · 3D轨迹 · 跨模态学习 · 机器人学习
Summary 本文提出TraceGen，一种在3D轨迹空间中进行未来动作预测的世界模型，通过将人类和机器人操作视频统一为几何轨迹表示，实现跨平台、跨环境的任务学习。作者构建了大规模的3D轨迹数据集TraceForge，并展示了仅用少量目标视频即可高效迁移。然而该工作聚焦于机器人操控任务，与文档图像理解无关。
Reason 该论文研究的是机器人任务中的跨具身视频学习，核心是通过3D轨迹空间（trace-space）建模来抽象不同实体（如人与机器人）的动作，属于具身智能与机器人领域。尽管其使用了视觉输入和动作预测，但目标是机器人控制而非文档理解或视觉语言模型在文档上的应用。完全不涉及文档图像理解（DIU）、DocVQA、OCR、排版分析等任务，也无法迁移其方法解决OCR幻觉、密集文本识别、复杂布局解析等DIU核心痛点。
Abstract Learning new robot tasks on new platforms and in new scenes from only a handful of demonstrations remains challenging. While videos of other embodiments - humans and different robots - are abundant, differences in embodiment, camera, and environment hinder their direct use. We address the small-data problem by introducing a unifying, symbolic representation - a compact 3D "trace-space" of scene-level trajectories - that enables learning from cross-embodiment, cross-environment, and cross-task videos. We present TraceGen, a world model that predicts future motion in trace-space rather than pixel space, abstracting away appearance while retaining the geometric structure needed for manipulation. To train TraceGen at scale, we develop TraceForge, a data pipeline that transforms heterogeneous human and robot videos into consistent 3D traces, yielding a corpus of 123K videos and 1.8M observation-trace-language triplets. Pretraining on this corpus produces a transferable 3D motion prior that adapts efficiently: with just five target robot videos, TraceGen attains 80% success across four tasks while offering 50-600x faster inference than state-of-the-art video-based world models. In the more challenging case where only five uncalibrated human demonstration videos captured on a handheld phone are available, it still reaches 67.5% success on a real robot, highlighting TraceGen's ability to adapt across embodiments without relying on object detectors or heavy pixel-space generation.
Authors Seungjae Lee, Yoonkyo Jung, Inkook Chun, Yao-Chih Lee, Zikui Cai, Hongjia Huang, Aayush Talreja, Tan Dat Dao, Yongyuan Liang, Jia-Bin Huang, Furong Huang
Categories Computer Vision and Pattern Recognition
0
Seeing without Pixels: Perception from Camera Trajectories
Title CN 无像素视觉：从相机轨迹中感知
Keywords 相机轨迹 · 对比学习 · 视频理解 · 跨模态对齐 · 自我中心视频
Summary 本文提出CamFormer，通过对比学习将相机姿态轨迹映射到联合嵌入空间，以实现与自然语言的对齐，探索仅凭相机运动路径推断视频内容的可能性。研究表明，相机轨迹本身是一种轻量但信息丰富的模态，可用于跨模态对齐、分类和时序分析等任务。
Reason 该论文研究的是从相机运动轨迹（camera trajectory）中推断视频内容，属于视频理解领域，且核心关注点为运动路径与语义的对齐。其方法和任务与文档图像理解（DIU）中的OCR幻觉、密集文本识别、复杂版式分析等核心痛点完全无关。无法迁移用于解决文档图像中的细粒度识别或VLM在文档上的多模态对齐问题。
Abstract Can one perceive a video's content without seeing its pixels, just from the camera trajectory-the path it carves through space? This paper is the first to systematically investigate this seemingly implausible question. Towards this end, we propose a contrastive learning framework to train CamFormer, a dedicated encoder that projects camera pose trajectories into a joint embedding space, aligning them with natural language. We find that, contrary to its apparent simplicity, the camera trajectory is a remarkably informative signal to uncover video content. In other words, "how you move" can indeed reveal "what you are doing" (egocentric) or "observing" (exocentric). We demonstrate the versatility of our learned CamFormer embeddings on a diverse suite of downstream tasks, ranging from cross-modal alignment to classification and temporal analysis. Importantly, our representations are robust across diverse camera pose estimation methods, including both high-fidelity multi-sensored and standard RGB-only estimators. Our findings establish camera trajectory as a lightweight, robust, and versatile modality for perceiving video content.
Authors Zihui Xue, Kristen Grauman, Dima Damen, Andrew Zisserman, Tengda Han
Comment Project website: https://sites.google.com/view/seeing-without-pixels
Categories Computer Vision and Pattern Recognition
0
Revolutionizing Glioma Segmentation & Grading Using 3D MRI - Guided Hybrid Deep Learning Models
Title CN 利用3D MRI引导的混合深度学习模型彻底改变胶质瘤分割与分级
Keywords 胶质瘤分割 · 3D MRI · U-Net · 注意力机制 · 医学图像分类
Summary 本文提出一种结合U-Net分割与DenseNet-VGG分类的混合深度学习模型，用于3D MRI数据中的胶质瘤分割与分级。通过引入多头注意力和空间-通道注意力机制，模型在肿瘤区域提取临床相关特征，实现了98%的Dice系数和99%的分类准确率。尽管性能优异，但其任务和数据均远离文档理解范畴。
Reason 该论文属于医学图像分析领域，研究基于3D MRI的脑肿瘤分割与分类，属于纯医疗影像任务，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。其方法虽涉及注意力机制和混合网络结构，但应用场景为具象的生物医学图像识别，无法直接迁移解决文档中的OCR幻觉、复杂排版或长文本推理等核心痛点。
Abstract Gliomas are brain tumor types that have a high mortality rate which means early and accurate diagnosis is important for therapeutic intervention for the tumors. To address this difficulty, the proposed research will develop a hybrid deep learning model which integrates U-Net based segmentation and a hybrid DenseNet-VGG classification network with multihead attention and spatial-channel attention capabilities. The segmentation model will precisely demarcate the tumors in a 3D volume of MRI data guided by spatial and contextual information. The classification network which combines a branch of both DenseNet and VGG, will incorporate the demarcated tumor on which features with attention mechanisms would be focused on clinically relevant features. High-dimensional 3D MRI data could successfully be utilized in the model through preprocessing steps which are normalization, resampling, and data augmentation. Through a variety of measures the framework is evaluated: measures of performance in segmentation are Dice coefficient and Mean Intersection over Union (IoU) and measures of performance in classification are accuracy precision, recall, and F1-score. The hybrid framework that has been proposed has demonstrated through physical testing that it has the capability of obtaining a Dice coefficient of 98% in tumor segmentation, and 99% on classification accuracy, outperforming traditional CNN models and attention-free methods. Utilizing multi-head attention mechanisms enhances notions of priority in aspects of the tumor that are clinically significant, and enhances interpretability and accuracy. The results suggest a great potential of the framework in facilitating the timely and reliable diagnosis and grading of glioma by clinicians is promising, allowing for better planning of patient treatment.
Authors Pandiyaraju V, Sreya Mynampati, Abishek Karthik, Poovarasan L, D. Saraswathi
Categories Computer Vision and Pattern Recognition
0
Uncertainty Quantification for Visual Object Pose Estimation
Title CN 视觉目标姿态估计的不确定性量化
Keywords 不确定性量化 · 姿态估计 · S-lemma · 凸优化 · 关键点检测
Summary 本文提出SLUE方法，用于在无需分布假设的前提下对单目视觉下的物体姿态估计结果生成统计严谨的不确定性椭球边界。该方法基于S-引理构造凸优化问题，能够在高概率下包含真实姿态，并可扩展至更高阶的松弛层级以获得更紧致的边界。实验表明其在平移和旋转误差界上优于现有方法。
Reason 该论文研究的是机器人和三维视觉中的单目物体姿态估计不确定性问题，属于3D视觉与具身智能领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心方法SLUE针对的是基于2D关键点反推3D姿态的不确定性建模，无法迁移解决OCR幻觉、复杂排版解析、长文档推理等DIU核心痛点。
Abstract Quantifying the uncertainty of an object's pose estimate is essential for robust control and planning. Although pose estimation is a well-studied robotics problem, attaching statistically rigorous uncertainty is not well understood without strict distributional assumptions. We develop distribution-free pose uncertainty bounds about a given pose estimate in the monocular setting. Our pose uncertainty only requires high probability noise bounds on pixel detections of 2D semantic keypoints on a known object. This noise model induces an implicit, non-convex set of pose uncertainty constraints. Our key contribution is SLUE (S-Lemma Uncertainty Estimation), a convex program to reduce this set to a single ellipsoidal uncertainty bound that is guaranteed to contain the true object pose with high probability. SLUE solves a relaxation of the minimum volume bounding ellipsoid problem inspired by the celebrated S-lemma. It requires no initial guess of the bound's shape or size and is guaranteed to contain the true object pose with high probability. For tighter uncertainty bounds at the same confidence, we extend SLUE to a sum-of-squares relaxation hierarchy which is guaranteed to converge to the minimum volume ellipsoidal uncertainty bound for a given set of keypoint constraints. We show this pose uncertainty bound can easily be projected to independent translation and axis-angle orientation bounds. We evaluate SLUE on two pose estimation datasets and a real-world drone tracking scenario. Compared to prior work, SLUE generates substantially smaller translation bounds and competitive orientation bounds. We release code at https://github.com/MIT-SPARK/PoseUncertaintySets.
Authors Lorenzo Shaikewitz, Charis Georgiou, Luca Carlone
Comment 18 pages, 9 figures. Code available: https://github.com/MIT-SPARK/PoseUncertaintySets
Categories Computer Vision and Pattern Recognition
0
CaFlow: Enhancing Long-Term Action Quality Assessment with Causal Counterfactual Flow
Title CN CaFlow：通过因果反事实流增强长期动作质量评估
Keywords 动作质量评估 · 因果推理 · 反事实干预 · 时序建模 · 视频理解
Summary 本文提出CaFlow框架，用于提升长期动作质量评估的性能。该方法结合因果反事实正则化（CCR）模块进行特征解耦，并利用双向时间条件流（BiT-Flow）建模前后向动态，通过循环一致性约束获得更连贯的表示。实验表明其在多个AQA基准上达到SOTA。
Reason 该论文研究的是视频领域的长期动作质量评估（AQA），属于视频理解任务，主要面向体育、康复等场景的动作评分。其核心方法CaFlow涉及因果去混淆和双向时序建模，虽在时序建模上有一定创新，但整体应用于视频动作分析，与文档图像理解（DIU）或DocVQA等任务无直接关联。无法迁移解决OCR幻觉、密集文本识别、复杂版式分析或长文档推理等DIU核心痛点。
Abstract Action Quality Assessment (AQA) predicts fine-grained execution scores from action videos and is widely applied in sports, rehabilitation, and skill evaluation. Long-term AQA, as in figure skating or rhythmic gymnastics, is especially challenging since it requires modeling extended temporal dynamics while remaining robust to contextual confounders. Existing approaches either depend on costly annotations or rely on unidirectional temporal modeling, making them vulnerable to spurious correlations and unstable long-term representations. To this end, we propose CaFlow, a unified framework that integrates counterfactual de-confounding with bidirectional time-conditioned flow. The Causal Counterfactual Regularization (CCR) module disentangles causal and confounding features in a self-supervised manner and enforces causal robustness through counterfactual interventions, while the BiT-Flow module models forward and backward dynamics with a cycle-consistency constraint to produce smoother and more coherent representations. Extensive experiments on multiple long-term AQA benchmarks demonstrate that CaFlow achieves state-of-the-art performance. Code is available at https://github.com/Harrison21/CaFlow
Authors Ruisheng Han, Kanglei Zhou, Shuang Chen, Amir Atapour-Abarghouei, Hubert P. H. Shum
Categories Computer Vision and Pattern Recognition
0
Active Learning for GCN-based Action Recognition
Title CN 基于图卷积网络的动作识别中的主动学习
Keywords 图卷积网络 · 主动学习 · 动作识别 · 标签效率 · 骨架数据
Summary 本文提出了一种基于主动学习的图卷积网络（GCN）模型，用于缓解骨架动作识别中标注数据稀缺的问题。通过设计一种结合对抗策略的新采样函数，并引入双向稳定的GCN架构，提升模型对样本分布的理解能力，在两个基准上取得了优于先前方法的效果。
Reason 该论文研究的是骨架动作识别（skeleton-based action recognition），属于视频理解领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法聚焦于GCN在动作识别中的标签效率问题，提出的采集函数和网络结构改进难以迁移到文档理解的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）。属于无关领域研究。
Abstract Despite the notable success of graph convolutional networks (GCNs) in skeleton-based action recognition, their performance often depends on large volumes of labeled data, which are frequently scarce in practical settings. To address this limitation, we propose a novel label-efficient GCN model. Our work makes two primary contributions. First, we develop a novel acquisition function that employs an adversarial strategy to identify a compact set of informative exemplars for labeling. This selection process balances representativeness, diversity, and uncertainty. Second, we introduce bidirectional and stable GCN architectures. These enhanced networks facilitate a more effective mapping between the ambient and latent data spaces, enabling a better understanding of the learned exemplar distribution. Extensive evaluations on two challenging skeleton-based action recognition benchmarks reveal significant improvements achieved by our label-efficient GCNs compared to prior work.
Authors Hichem Sahbi
Categories Computer Vision and Pattern Recognition
0
MoGAN: Improving Motion Quality in Video Diffusion via Few-Step Motion Adversarial Post-Training
Title CN MoGAN：通过少步运动对抗后训练提升视频扩散中的运动质量
Keywords 视频扩散 · 对抗训练 · 运动连贯性 · 光流判别器 · 后训练
Summary 本文提出MoGAN，一种专注于提升视频扩散模型运动质量的对抗性后训练框架。通过在三步蒸馏模型基础上训练基于DiT的光流判别器，并结合分布匹配正则化，显著提升了生成视频的运动真实感和动态连贯性，同时保持了视觉保真度和生成效率。实验表明其在多个基准上优于教师模型和基线扩散模型。
Reason 该论文属于纯视频生成领域，研究的是视频扩散模型中的运动连贯性问题，使用对抗训练优化生成动作的流畅性。其技术目标与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版解析、细粒度文本定位等完全无关。尽管涉及视觉生成，但方法无法迁移到文档多模态理解或VLM for Doc场景中。
Abstract Video diffusion models achieve strong frame-level fidelity but still struggle with motion coherence, dynamics and realism, often producing jitter, ghosting, or implausible dynamics. A key limitation is that the standard denoising MSE objective provides no direct supervision on temporal consistency, allowing models to achieve low loss while still generating poor motion. We propose MoGAN, a motion-centric post-training framework that improves motion realism without reward models or human preference data. Built atop a 3-step distilled video diffusion model, we train a DiT-based optical-flow discriminator to differentiate real from generated motion, combined with a distribution-matching regularizer to preserve visual fidelity. With experiments on Wan2.1-T2V-1.3B, MoGAN substantially improves motion quality across benchmarks. On VBench, MoGAN boosts motion score by +7.3% over the 50-step teacher and +13.3% over the 3-step DMD model. On VideoJAM-Bench, MoGAN improves motion score by +7.4% over the teacher and +8.8% over DMD, while maintaining comparable or even better aesthetic and image-quality scores. A human study further confirms that MoGAN is preferred for motion quality (52% vs. 38% for the teacher; 56% vs. 29% for DMD). Overall, MoGAN delivers significantly more realistic motion without sacrificing visual fidelity or efficiency, offering a practical path toward fast, high-quality video generation. Project webpage is: https://xavihart.github.io/mogan.
Authors Haotian Xue, Qi Chen, Zhonghao Wang, Xun Huang, Eli Shechtman, Jinrong Xie, Yongxin Chen
Categories Computer Vision and Pattern Recognition
0
Deep Learning-Based Multiclass Classification of Oral Lesions with Stratified Augmentation
Title CN 基于深度学习的口腔病变多分类与分层增强
Keywords 深度学习 · 多分类 · 口腔病变 · 数据增强 · 过采样
Summary 本文提出一种基于深度学习的多类分类方法，用于识别16种不同的口腔病变。通过分层数据划分和先进的数据增强与过采样策略来应对数据集不平衡问题。实验结果显示较高的精度和召回率，表明该方法在小样本类别上具有良好的分类性能。该工作主要面向临床早期口腔癌辅助诊断系统。
Reason 该论文属于医学图像分析领域，研究任务为口腔病变的分类，与文档图像理解（DIU）或视觉语言模型（VLM）在文档理解中的应用无关。尽管使用了深度学习和数据增强技术，但其方法针对的是生物医学图像分类问题，并未涉及OCR、布局分析、表格识别、DocVQA等DIU核心任务，也无法提供可迁移至文档理解领域的底层技术创新。
Abstract Oral cancer is highly common across the globe and is mostly diagnosed during the later stages due to the close visual similarity to benign, precancerous, and malignant lesions in the oral cavity. Implementing computer aided diagnosis systems early on has the potential to greatly improve clinical outcomes. This research intends to use deep learning to build a multiclass classifier for sixteen different oral lesions. To overcome the challenges of limited and imbalanced datasets, the proposed technique combines stratified data splitting and advanced data augmentation and oversampling to perform the classification. The experimental results, which achieved 83.33 percent accuracy, 89.12 percent precision, and 77.31 percent recall, demonstrate the superiority of the suggested model over state of the art methods now in use. The suggested model effectively conveys the effectiveness of oversampling and augmentation strategies in situations where the minority class classification performance is noteworthy. As a first step toward trustworthy computer aided diagnostic systems for the early detection of oral cancer in clinical settings, the suggested framework shows promise.
Authors Joy Naoum, Revana Salama, Ali Hamdi
Comment 12 pages, 3 figures,
Categories Computer Vision and Pattern Recognition
0
Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy
Title CN Harmony：通过跨任务协同调和音频与视频生成
Keywords 音视频生成 · 跨模态同步 · 扩散模型 · Classifier-Free Guidance · Temporal Alignment
Summary 本文提出Harmony框架，旨在解决开源生成模型在音视频同步合成中的对齐难题。通过引入跨任务协同训练范式、全局-局部解耦交互模块以及增强同步的Classifier-Free Guidance机制，在音视频生成质量和细粒度同步上实现了新SOTA。
Reason 该论文研究的是音视频同步生成问题，属于视频生成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心技术如Cross-Task Synergy、SyncCFG等针对的是音频与视频模态间的同步问题，无法迁移解决文档理解中的OCR幻觉、复杂排版、细粒度定位等核心痛点。属于无关领域研究。
Abstract The synthesis of synchronized audio-visual content is a key challenge in generative AI, with open-source models facing challenges in robust audio-video alignment. Our analysis reveals that this issue is rooted in three fundamental challenges of the joint diffusion process: (1) Correspondence Drift, where concurrently evolving noisy latents impede stable learning of alignment; (2) inefficient global attention mechanisms that fail to capture fine-grained temporal cues; and (3) the intra-modal bias of conventional Classifier-Free Guidance (CFG), which enhances conditionality but not cross-modal synchronization. To overcome these challenges, we introduce Harmony, a novel framework that mechanistically enforces audio-visual synchronization. We first propose a Cross-Task Synergy training paradigm to mitigate drift by leveraging strong supervisory signals from audio-driven video and video-driven audio generation tasks. Then, we design a Global-Local Decoupled Interaction Module for efficient and precise temporal-style alignment. Finally, we present a novel Synchronization-Enhanced CFG (SyncCFG) that explicitly isolates and amplifies the alignment signal during inference. Extensive experiments demonstrate that Harmony establishes a new state-of-the-art, significantly outperforming existing methods in both generation fidelity and, critically, in achieving fine-grained audio-visual synchronization.
Authors Teng Hu, Zhentao Yu, Guozhen Zhang, Zihan Su, Zhengguang Zhou, Youliang Zhang, Yuan Zhou, Qinglin Lu, Ran Yi
Categories Computer Vision and Pattern Recognition
0
Enhanced Landmark Detection Model in Pelvic Fluoroscopy using 2D/3D Registration Loss
Title CN 利用2D/3D配准损失增强骨盆荧光镜下的解剖标志点检测
Keywords landmark detection · 2D/3D registration · U-Net · fluoroscopy · pose estimation
Summary 本文提出一种结合2D/3D配准损失的U-Net模型，用于提升在患者体位变化情况下骨盆荧光镜图像中解剖标志点的检测精度。通过引入姿态估计损失，增强了模型对非标准视角的鲁棒性。实验比较了不同训练策略下的检测性能。
Reason 该论文属于医学图像分析领域，研究的是骨盆荧光镜下的解剖标志点检测问题，核心方法为结合2D/3D注册的U-Net训练框架。其任务、数据与文档图像理解（DIU）无关，不涉及文本识别、布局分析、表格理解或VLM在文档上的应用。属于典型的相关但非目标领域工作，且无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract Automated landmark detection offers an efficient approach for medical professionals to understand patient anatomic structure and positioning using intra-operative imaging. While current detection methods for pelvic fluoroscopy demonstrate promising accuracy, most assume a fixed Antero-Posterior view of the pelvis. However, orientation often deviates from this standard view, either due to repositioning of the imaging unit or of the target structure itself. To address this limitation, we propose a novel framework that incorporates 2D/3D landmark registration into the training of a U-Net landmark prediction model. We analyze the performance difference by comparing landmark detection accuracy between the baseline U-Net, U-Net trained with Pose Estimation Loss, and U-Net fine-tuned with Pose Estimation Loss under realistic intra-operative conditions where patient pose is variable.
Authors Chou Mo, Yehyun Suh, J. Ryan Martin, Daniel Moyer
Comment 9 pages, 3 figures, 1 table
Categories Computer Vision and Pattern Recognition
0
UAVLight: A Benchmark for Illumination-Robust 3D Reconstruction in Unmanned Aerial Vehicle (UAV) Scenes
Title CN UAVLight：一种用于无人机场景中光照鲁棒3D重建的基准
Keywords 3D重建 · 光照鲁棒性 · 无人机 · 多视角立体 · 数据集
Summary 本文提出了UAVLight，一个针对无人机航拍场景下多时段光照变化的3D重建基准数据集。通过固定航线和重复拍摄，确保几何结构一致的同时引入真实的光照差异，旨在评估和提升现有3D重建方法在自然光照变化下的鲁棒性。提供了标准化评测协议，支持对几何一致性、色彩保真度和可重光照能力的评估。
Reason 该论文属于3D视觉与无人机重建领域，研究的是多视角3D重建中的光照变化问题，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。尽管其数据集构建方法具有一定工程价值，但无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点，属于无关领域。
Abstract Illumination inconsistency is a fundamental challenge in multi-view 3D reconstruction. Variations in sunlight direction, cloud cover, and shadows break the constant-lighting assumption underlying both classical multi-view stereo (MVS) and structure from motion (SfM) pipelines and recent neural rendering methods, leading to geometry drift, color inconsistency, and shadow imprinting. This issue is especially critical in UAV-based reconstruction, where long flight durations and outdoor environments make lighting changes unavoidable. However, existing datasets either restrict capture to short time windows, thus lacking meaningful illumination diversity, or span months and seasons, where geometric and semantic changes confound the isolated study of lighting robustness. We introduce UAVLight, a controlled-yet-real benchmark for illumination-robust 3D reconstruction. Each scene is captured along repeatable, geo-referenced flight paths at multiple fixed times of day, producing natural lighting variation under consistent geometry, calibration, and viewpoints. With standardized evaluation protocols across lighting conditions, UAVLight provides a reliable foundation for developing and benchmarking reconstruction methods that are consistent, faithful, and relightable in real outdoor environments.
Authors Kang Du, Xue Liao, Junpeng Xia, Chaozheng Guo, Yi Gu, Yirui Guan, Duotun Wang, ShengHuang, Zeyu Wang
Comment 10 pages, 6 figures
Categories Computer Vision and Pattern Recognition
0
Sequence-Adaptive Video Prediction in Continuous Streams using Diffusion Noise Optimization
Title CN 基于扩散噪声优化的连续视频流中序列自适应视频预测
Keywords 视频预测 · 扩散模型 · 连续学习 · 噪声优化 · Ego4D
Summary 本文提出了一种名为SAVi-DNO的方法，用于在连续视频流中进行自适应视频预测。该方法通过在推理阶段优化扩散噪声而非微调模型参数，使预训练的扩散模型能够持续适应新的视频数据。实验表明该方法在多个长视频数据集上提升了预测性能。
Reason 该论文研究的是视频预测任务，属于视频理解领域，与文档图像理解（DIU）的核心任务如DocVQA、表格识别、版面分析等无直接关联。尽管其提出的方法在连续视频流中通过优化扩散噪声实现模型自适应，但该技术针对的是动态帧生成质量提升，难以迁移到解决文档理解中的OCR幻觉、密集文本解析或复杂排版结构建模等问题。且视频生成本身属于负面清单中的‘无关领域’。
Abstract In this work, we investigate diffusion-based video prediction models, which forecast future video frames, for continuous video streams. In this context, the models observe continuously new training samples, and we aim to leverage this to improve their predictions. We thus propose an approach that continuously adapts a pre-trained diffusion model to a video stream. Since fine-tuning the parameters of a large diffusion model is too expensive, we refine the diffusion noise during inference while keeping the model parameters frozen, allowing the model to adaptively determine suitable sampling noise. We term the approach Sequence Adaptive Video Prediction with Diffusion Noise Optimization (SAVi-DNO). To validate our approach, we introduce a new evaluation setting on the Ego4D dataset, focusing on simultaneous adaptation and evaluation on long continuous videos. Empirical results demonstrate improved performance based on FVD, SSIM, and PSNR metrics on long videos of Ego4D and OpenDV-YouTube, as well as videos of UCF-101 and SkyTimelapse, showcasing SAVi-DNO's effectiveness.
Authors Sina Mokhtarzadeh Azar, Emad Bahrami, Enrico Pallotta, Gianpiero Francesca, Radu Timofte, Juergen Gall
Categories Computer Vision and Pattern Recognition
0
Video Generation Models Are Good Latent Reward Models
Title CN 视频生成模型是优秀的潜在奖励模型
Keywords 视频生成 · 奖励模型 · 潜在空间 · ReFL · PRFL
Summary 本文提出一种在潜在空间中进行奖励反馈学习的方法PRFL，利用预训练的视频生成模型作为奖励模型，在无需VAE解码的情况下实现全去噪链路上的梯度回传，从而提升视频生成与人类偏好的对齐效果，并降低训练开销。
Reason 该论文研究的是视频生成领域的奖励建模问题，提出在潜在空间中进行偏好优化的方法。其核心任务与文档图像理解（DIU）无关，属于纯视频生成与训练优化方向。尽管涉及‘latent space’和‘reward modeling’等技术概念，但这些方法针对的是动态视频的生成质量与运动连贯性优化，难以迁移到文档理解中的OCR幻觉、复杂排版解析或细粒度文本定位等核心痛点。且未涉及多模态语言模型对文档结构的理解或视觉-语言对齐机制的改进，不属于DIU本题或可迁移的上游方法。
Abstract Reward feedback learning (ReFL) has proven effective for aligning image generation with human preferences. However, its extension to video generation faces significant challenges. Existing video reward models rely on vision-language models designed for pixel-space inputs, confining ReFL optimization to near-complete denoising steps after computationally expensive VAE decoding. This pixel-space approach incurs substantial memory overhead and increased training time, and its late-stage optimization lacks early-stage supervision, refining only visual quality rather than fundamental motion dynamics and structural coherence. In this work, we show that pre-trained video generation models are naturally suited for reward modeling in the noisy latent space, as they are explicitly designed to process noisy latent representations at arbitrary timesteps and inherently preserve temporal information through their sequential modeling capabilities. Accordingly, we propose Process Reward Feedback Learning~(PRFL), a framework that conducts preference optimization entirely in latent space, enabling efficient gradient backpropagation throughout the full denoising chain without VAE decoding. Extensive experiments demonstrate that PRFL significantly improves alignment with human preferences, while achieving substantial reductions in memory consumption and training time compared to RGB ReFL.
Authors Xiaoyue Mi, Wenqing Yu, Jiesong Lian, Shibo Jie, Ruizhe Zhong, Zijun Liu, Guozhen Zhang, Zixiang Zhou, Zhiyong Xu, Yuan Zhou, Qinglin Lu, Fan Tang
Categories Computer Vision and Pattern Recognition
0
Adapting Segment Anything Model for Power Transmission Corridor Hazard Segmentation
Title CN 将Segment Anything模型用于电力传输走廊危险区域分割
Keywords 图像分割 · Segment Anything Model · 电力传输安全 · 细粒度结构分割 · 上下文感知
Summary 本文提出ELE-SAM模型，用于电力传输走廊中的设备与危险物分割。通过引入上下文感知提示适配器和高保真掩码解码器，在自建数据集ELE-40K上取得了优于现有方法的性能。研究聚焦于工业场景下的图像分割，不属于文档图像理解范畴。
Reason 该论文属于特定领域的下游应用（电力设施安全监控），尽管对SAM进行了改进，但其核心任务是遥感/工业图像中的语义分割，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。提出的Context-Aware Prompt Adapter和High-Fidelity Mask Decoder虽有一定技术价值，但针对的是电力设备与自然背景中的细粒度结构分割，无法直接迁移解决文档图像中的OCR幻觉、复杂排版或长文本推理等核心痛点。
Abstract Power transmission corridor hazard segmentation (PTCHS) aims to separate transmission equipment and surrounding hazards from complex background, conveying great significance to maintaining electric power transmission safety. Recently, the Segment Anything Model (SAM) has emerged as a foundational vision model and pushed the boundaries of segmentation tasks. However, SAM struggles to deal with the target objects in complex transmission corridor scenario, especially those with fine structure. In this paper, we propose ELE-SAM, adapting SAM for the PTCHS task. Technically, we develop a Context-Aware Prompt Adapter to achieve better prompt tokens via incorporating global-local features and focusing more on key regions. Subsequently, to tackle the hazard objects with fine structure in complex background, we design a High-Fidelity Mask Decoder by leveraging multi-granularity mask features and then scaling them to a higher resolution. Moreover, to train ELE-SAM and advance this field, we construct the ELE-40K benchmark, the first large-scale and real-world dataset for PTCHS including 44,094 image-mask pairs. Experimental results for ELE-40K demonstrate the superior performance that ELE-SAM outperforms the baseline model with the average 16.8% mIoU and 20.6% mBIoU performance improvement. Moreover, compared with the state-of-the-art method on HQSeg-44K, the average 2.9% mIoU and 3.8% mBIoU absolute improvements further validate the effectiveness of our method on high-quality generic object segmentation. The source code and dataset are available at https://github.com/Hhaizee/ELE-SAM.
Authors Hang Chen, Maoyuan Ye, Peng Yang, Haibin He, Juhua Liu, Bo Du
Categories Computer Vision and Pattern Recognition
0
Bangla Sign Language Translation: Dataset Creation Challenges, Benchmarking and Prospects
Title CN 孟加拉手语翻译：数据集创建挑战、基准测试与前景
Keywords 手语翻译 · 低资源语言 · 数据集构建 · 关键点检测 · 序列建模
Summary 本文提出了一个名为IsharaKhobor的孟加拉手语翻译数据集，并介绍了数据集构建过程中的挑战。作者还基于关键点和嵌入表示进行了基准测试，探讨了词汇限制和规范化对模型性能的影响，发布了两个子数据集。研究旨在推动孟加拉语聋哑人群体的AI辅助工具发展。
Reason 该论文属于小语种（孟加拉语）相关的手语翻译任务，研究领域为低资源语言的手语识别与翻译，与文档图像理解（DIU/DocVQA）无关。其技术路线聚焦于手势识别和序列建模，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心问题。
Abstract Bangla Sign Language Translation (BdSLT) has been severely constrained so far as the language itself is very low resource. Standard sentence level dataset creation for BdSLT is of immense importance for developing AI based assistive tools for deaf and hard of hearing people of Bangla speaking community. In this paper, we present a dataset, IsharaKhobor , and two subset of it for enabling research. We also present the challenges towards developing the dataset and present some way forward by benchmarking with landmark based raw and RQE embedding. We do some ablation on vocabulary restriction and canonicalization of the same within the dataset, which resulted in two more datasets, IsharaKhobor_small and IsharaKhobor_canonical_small. The dataset is publicly available at: www.kaggle.com/datasets/hasanssl/isharakhobor [1].
Authors Husne Ara Rubaiyeat, Hasan Mahmud, Md Kamrul Hasan
Comment 14 pages, 8 tables
Categories Computer Vision and Pattern Recognition, Computation and Language
0
The Age-specific Alzheimer 's Disease Prediction with Characteristic Constraints in Nonuniform Time Span
Title CN 基于非均匀时间跨度中特征约束的特定年龄阿尔茨海默病预测
Keywords 阿尔茨海默病 · MRI图像生成 · 时间序列建模 · 年龄特异性预测 · 图像合成
Summary 本文提出一种在非均匀时间间隔下生成具有疾病特征约束的连续MRI图像的方法，通过引入定量指标指导图像生成，并结合年龄缩放因子生成特定年龄的脑部图像，以提升阿尔茨海默病的长期预测准确性。实验表明该方法在结构相似性指标上达到0.882，有效提升了合成图像的质量。
Reason 该论文研究的是医学影像生成与疾病预测，属于医疗图像分析领域，核心任务为MRI图像生成和阿尔茨海默病进展预测，与文档图像理解（DIU）、DocVQA、OCR、布局分析等无关。尽管涉及视觉信息处理，但其方法聚焦于医学时间序列建模与图像生成质量优化，无法迁移解决文档理解中的OCR幻觉、复杂排版或细粒度文本定位等核心痛点。
Abstract Alzheimer's disease is a debilitating disorder marked by a decline in cognitive function. Timely identification of the disease is essential for the development of personalized treatment strategies that aim to mitigate its progression. The application of generated images for the prediction of Alzheimer's disease poses challenges, particularly in accurately representing the disease's characteristics when input sequences are captured at irregular time intervals. This study presents an innovative methodology for sequential image generation, guided by quantitative metrics, to maintain the essential features indicative of disease progression. Furthermore, an age-scaling factor is integrated into the process to produce age-specific MRI images, facilitating the prediction of advanced stages of the disease. The results obtained from the ablation study suggest that the inclusion of quantitative metrics significantly improves the accuracy of MRI image synthesis. Furthermore, the application of age-scaled pixel loss contributed to the enhanced iterative generation of MRI images. In terms of long-term disease prognosis, the Structural Similarity Index reached a peak value of 0.882, indicating a substantial degree of similarity in the synthesized images.
Authors Xin Hong, Kaifeng Huang
Comment 16 pages, 9 figures
Categories Computer Vision and Pattern Recognition
0
LMLCC-Net: A Semi-Supervised Deep Learning Model for Lung Nodule Malignancy Prediction from CT Scans using a Novel Hounsfield Unit-Based Intensity Filtering
Title CN LMLCC-Net：一种基于新型HU强度滤波的半监督深度学习模型，用于从CT扫描中预测肺结节恶性
Keywords 肺结节 · 恶性预测 · CT图像 · 3D CNN · 半监督学习
Summary 本文提出LMLCC-Net，一种结合Hounsfield Unit（HU）强度滤波的多分支3D CNN模型，用于从CT扫描图像中分类肺结节的良恶性。通过多个分支分别进行可学习的HU强度范围过滤，并融合纹理与强度特征，在LUNA16数据集上实现了91.96%的准确率和94.07%的AUC。同时引入半监督学习策略处理模糊样本，并设计轻量级模型以提升效率。
Reason 该论文属于医学图像分析领域，研究任务为肺结节恶性分类，使用3D CNN和HU强度滤波处理CT影像，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其方法无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract Lung cancer is the leading cause of patient mortality in the world. Early diagnosis of malignant pulmonary nodules in CT images can have a significant impact on reducing disease mortality and morbidity. In this work, we propose LMLCC-Net, a novel deep learning framework for classifying nodules from CT scan images using a 3D CNN, considering Hounsfield Unit (HU)-based intensity filtering. Benign and malignant nodules have significant differences in their intensity profile of HU, which was not exploited in the literature. Our method considers the intensity pattern as well as the texture for the prediction of malignancies. LMLCC-Net extracts features from multiple branches that each use a separate learnable HU-based intensity filtering stage. Various combinations of branches and learnable ranges of filters were explored to finally produce the best-performing model. In addition, we propose a semi-supervised learning scheme for labeling ambiguous cases and also developed a lightweight model to classify the nodules. The experimental evaluations are carried out on the LUNA16 dataset. The proposed LMLCC-Net was evaluated using the LUNA16 dataset. Our proposed method achieves a classification accuracy of 91.96%, a sensitivity of 92.94%, and an area under the curve of 94.07%, showing improved performance compared to existing methods The proposed method can have a significant impact in helping radiologists in the classification of pulmonary nodules and improving patient care.
Authors Tasnia Binte Mamun, Adhora Madhuri, Nusaiba Sobir, Taufiq Hasan
Comment 12 pages, 9 figures, 7 tables
Categories Computer Vision and Pattern Recognition
0
Self-Paced Learning for Images of Antinuclear Antibodies IEEE Transactions on Medical Imaging
Title CN 抗核抗体图像的自步学习
Keywords 自步学习 · 多实例多标签学习 · 医学图像分析 · 抗核抗体检测 · 深度学习
Summary 本文提出了一种用于抗核抗体（ANA）检测的自步学习框架，旨在解决手动检测耗时且需要专业训练的问题。该方法通过实例采样器、伪标签分配器和自步学习机制，在无需预处理的情况下实现端到端优化。实验表明其在多个医学MIML数据集上性能优越，尤其在ANA数据集上显著提升了F1-Macro和mAP指标。
Reason 该论文聚焦于医学图像中的抗核抗体检测，属于特定医疗诊断任务的应用研究。尽管提出了自步学习框架处理多实例多标签问题，但其核心是面向生物医学成像分析的垂直应用，未涉及文档图像理解（DIU）中的任何核心挑战（如OCR幻觉、复杂版面分析、表格识别或VLM在文档上的应用）。属于典型的‘平行下游应用’，无DIU方法论创新。
Abstract Antinuclear antibody (ANA) testing is a crucial method for diagnosing autoimmune disorders, including lupus, Sjögren's syndrome, and scleroderma. Despite its importance, manual ANA detection is slow, labor-intensive, and demands years of training. ANA detection is complicated by over 100 coexisting antibody types, resulting in vast fluorescent pattern combinations. Although machine learning and deep learning have enabled automation, ANA detection in real-world clinical settings presents unique challenges as it involves multi-instance, multi-label (MIML) learning. In this paper, a novel framework for ANA detection is proposed that handles the complexities of MIML tasks using unaltered microscope images without manual preprocessing. Inspired by human labeling logic, it identifies consistent ANA sub-regions and assigns aggregated labels accordingly. These steps are implemented using three task-specific components: an instance sampler, a probabilistic pseudo-label dispatcher, and self-paced weight learning rate coefficients. The instance sampler suppresses low-confidence instances by modeling pattern confidence, while the dispatcher adaptively assigns labels based on instance distinguishability. Self-paced learning adjusts training according to empirical label observations. Our framework overcomes limitations of traditional MIML methods and supports end-to-end optimization. Extensive experiments on one ANA dataset and three public medical MIML benchmarks demonstrate the superiority of our framework. On the ANA dataset, our model achieves up to +7.0% F1-Macro and +12.6% mAP gains over the best prior method, setting new state-of-the-art results. It also ranks top-2 across all key metrics on public datasets, reducing Hamming loss and one-error by up to 18.2% and 26.9%, respectively. The source code can be accessed at https://github.com/fletcherjiang/ANA-SelfPacedLearning.
Authors Yiyang Jiang, Guangwu Qian, Jiaxin Wu, Qi Huang, Qing Li, Yongkang Wu, Xiao-Yong Wei
Comment IEEE Transactions on Medical Imaging
Categories Computer Vision and Pattern Recognition
0
Generalized Design Choices for Deepfake Detectors
Title CN 深度伪造检测器的通用设计选择
Keywords deepfake detection · design choices · generalization · data augmentation · optimization
Summary 本文系统性地研究了影响深度伪造检测模型性能的各种实现细节，包括数据预处理、增强策略和优化技术，旨在建立与架构无关的最佳实践。实验确定了一组能持续提升检测性能的设计选择，并在AI-GenBench基准上实现了SOTA结果。
Reason 该论文研究的是深度伪造检测领域的方法论优化，属于视频/图像伪造识别范畴，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其提出的设计选择（如数据预处理、增强策略等）虽具有一般性工程价值，但无法有效迁移至解决OCR幻觉、复杂排版解析、长文档推理等DIU核心痛点。
Abstract The effectiveness of deepfake detection methods often depends less on their core design and more on implementation details such as data preprocessing, augmentation strategies, and optimization techniques. These factors make it difficult to fairly compare detectors and to understand which factors truly contribute to their performance. To address this, we systematically investigate how different design choices influence the accuracy and generalization capabilities of deepfake detection models, focusing on aspects related to training, inference, and incremental updates. By isolating the impact of individual factors, we aim to establish robust, architecture-agnostic best practices for the design and development of future deepfake detection systems. Our experiments identify a set of design choices that consistently improve deepfake detection and enable state-of-the-art performance on the AI-GenBench benchmark.
Authors Lorenzo Pellegrini, Serafino Pandolfini, Davide Maltoni, Matteo Ferrara, Marco Prati, Marco Ramilli
Comment 12 pages, 9 figures, 10 tables, code available: https://github.com/MI-BioLab/AI-GenBench
Categories Computer Vision and Pattern Recognition
0
Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings SpeD 2025
Title CN 在真实低资源环境下改进视觉提示的关键词定位
Keywords 视觉提示关键词定位 · 无监督学习 · 低资源语言 · 视觉-语音模型 · 少样本学习
Summary 本文提出一种无需转录文本的少样本学习方法，用于在低资源语言的语音数据中进行视觉提示关键词定位（VPKL）。通过自动挖掘正负样本对，模型可在没有文字标注的情况下训练视觉-语音关联，在英语数据上性能接近使用真实标注的方法，并首次在约鲁巴语上进行了实验验证。但由于语音数据质量与语言特性影响，性能下降较明显。
Reason 该论文研究的是视觉-语音跨模态匹配问题，核心任务是基于图像查询在无标注语音中定位对应词汇（VPKL），属于语音与视觉交叉领域，且聚焦于低资源语言（如约鲁巴语）场景。其技术路线与文档图像理解（DIU）的核心任务（如DocVQA、布局分析、表格识别等）无直接关联。虽然涉及‘视觉提示’和‘无监督/少样本学习’，但方法针对的是语音信号中的关键词检索，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU关键痛点。此外，研究对象为非中英文的小语种Yoruba，命中负面清单第3条。
Abstract Given an image query, visually prompted keyword localisation (VPKL) aims to find occurrences of the depicted word in a speech collection. This can be useful when transcriptions are not available for a low-resource language (e.g. if it is unwritten). Previous work showed that VPKL can be performed with a visually grounded speech model trained on paired images and unlabelled speech. But all experiments were done on English. Moreover, transcriptions were used to get positive and negative pairs for the contrastive loss. This paper introduces a few-shot learning scheme to mine pairs automatically without transcriptions. On English, this results in only a small drop in performance. We also - for the first time - consider VPKL on a real low-resource language, Yoruba. While scores are reasonable, here we see a bigger drop in performance compared to using ground truth pairs because the mining is less accurate in Yoruba.
Authors Leanne Nortje, Dan Oneata, Gabriel Pirlogeanu, Herman Kamper
Comment Accepted at SpeD 2025
Categories Computer Vision and Pattern Recognition, Computation and Language
0
MobileI2V: Fast and High-Resolution Image-to-Video on Mobile Devices
Title CN MobileI2V：移动设备上的快速高分辨率图像到视频生成
Keywords 图像到视频生成 · 扩散模型 · 移动端推理 · 模型压缩 · 高效生成
Summary 本文提出MobileI2V，一种轻量级扩散模型，用于在移动设备上实现实时高分辨率图像到视频生成。通过线性混合注意力架构、时间步蒸馏策略和移动端特定的注意力优化，在保持质量的同时大幅提升了生成速度，首次实现了720p分辨率下每帧低于100毫秒的生成速度。
Reason 该论文属于纯图像生成/视频生成领域，研究的是图像到视频的扩散模型在移动端的高效实现，与文档图像理解（DIU）、DocVQA、OCR、布局分析等核心任务完全无关。尽管其在模型压缩和推理加速方面有技术贡献，但方法面向的是通用视频生成场景，无法迁移解决文档理解中的OCR幻觉、密集文本识别、复杂排版或长文档推理等关键痛点。
Abstract Recently, video generation has witnessed rapid advancements, drawing increasing attention to image-to-video (I2V) synthesis on mobile devices. However, the substantial computational complexity and slow generation speed of diffusion models pose significant challenges for real-time, high-resolution video generation on resource-constrained mobile devices. In this work, we propose MobileI2V, a 270M lightweight diffusion model for real-time image-to-video generation on mobile devices. The core lies in: (1) We analyzed the performance of linear attention modules and softmax attention modules on mobile devices, and proposed a linear hybrid architecture denoiser that balances generation efficiency and quality. (2) We design a time-step distillation strategy that compresses the I2V sampling steps from more than 20 to only two without significant quality loss, resulting in a 10-fold increase in generation speed. (3) We apply mobile-specific attention optimizations that yield a 2-fold speed-up for attention operations during on-device inference. MobileI2V enables, for the first time, fast 720p image-to-video generation on mobile devices, with quality comparable to existing models. Under one-step conditions, the generation speed of each frame of 720p video is less than 100 ms. Our code is available at: https://github.com/hustvl/MobileI2V.
Authors Shuai Zhang, Bao Tang, Siyuan Yu, Yueting Zhu, Jingfeng Yao, Ya Zou, Shanglin Yuan, Li Yu, Wenyu Liu, Xinggang Wang
Comment Our Demo and code:https://github.com/hustvl/MobileI2V
Categories Computer Vision and Pattern Recognition
0
MeshCone: Second-Order Cone Programming for Geometrically-Constrained Mesh Enhancement
Title CN MeshCone：用于几何约束网格增强的二阶锥规划
Keywords 3D mesh enhancement · convex optimization · geometry alignment · second-order cone programming · mesh refinement
Summary 本文提出MeshCone，一种基于凸优化的网格增强框架，通过二阶锥规划优化顶点位置，在保持平滑性的同时修复退化的3D网格。实验表明其在多种物体类别上优于拉普拉斯平滑等基线方法，并具有实时推理能力。适用于模板生成网格、扫描到CAD对齐等场景。
Reason 该论文研究的是3D网格增强与几何优化问题，属于3D视觉与CAD/扫描后处理领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其方法基于二阶锥规划进行网格顶点优化，虽在几何对齐和细节保持上有优势，但无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心痛点。
Abstract Modern mesh generation pipelines whether learning-based or classical often produce outputs requiring post-processing to achieve production-quality geometry. This work introduces MeshCone, a convex optimization framework for guided mesh refinement that leverages reference geometry to correct deformed or degraded meshes. We formulate the problem as a second-order cone program where vertex positions are optimized to align with target geometry while enforcing smoothness through convex edge-length regularization. MeshCone performs geometry-aware optimization that preserves fine details while correcting structural defects. We demonstrate robust performance across 56 diverse object categories from ShapeNet and ThreeDScans, achieving superior refinement quality compared to Laplacian smoothing and unoptimized baselines while maintaining sub-second inference times. MeshCone is particularly suited for applications where reference geometry is available, such as mesh-from-template workflows, scan-to-CAD alignment, and quality assurance in asset production pipelines.
Authors Alexander Valverde
Categories Computer Vision and Pattern Recognition
0
Comparison of Generative Learning Methods for Turbulence Surrogates
Title CN 生成学习方法在湍流代理模型中的比较
Keywords 生成模型 · 湍流模拟 · DCGAN · DDPM · 流体动力学
Summary 本文比较了三种生成模型（VAE、DCGAN、DDPM）在模拟二维卡门涡街和圆柱阵列尾流实验数据中的表现，评估其对湍流统计特性和空间结构的建模能力。结果显示DCGAN和DDPM效果最好，其中DCGAN在推理速度和训练效率上更具优势，而VAE结果不佳，DDPM则计算成本较高。
Reason 该论文研究的是流体动力学中的湍流模拟，使用生成模型（如VAE、DCGAN、DDPM）作为替代模型来加速数值仿真。尽管涉及图像数据（如PIV流场），但其核心任务与文档图像理解（DIU）无关，不涉及OCR、布局分析、表格识别或任何文档相关的视觉语言任务。属于完全无关领域（物理仿真+生成模型应用），无法迁移到DIU的痛点问题（如幻觉、复杂排版、长文档推理等）。
Abstract Numerical simulations of turbulent flows present significant challenges in fluid dynamics due to their complexity and high computational cost. High resolution techniques such as Direct Numerical Simulation (DNS) and Large Eddy Simulation (LES) are generally not computationally affordable, particularly for technologically relevant problems. Recent advances in machine learning, specifically in generative probabilistic models, offer promising alternatives as surrogates for turbulence. This paper investigates the application of three generative models - Variational Autoencoders (VAE), Deep Convolutional Generative Adversarial Networks (DCGAN), and Denoising Diffusion Probabilistic Models (DDPM) - in simulating a von Kármán vortex street around a fixed cylinder projected into 2D, as well as a real-world experimental dataset of the wake flow of a cylinder array. Training data was obtained by means of LES in the simulated case and Particle Image Velocimetry (PIV) in the experimental case. We evaluate each model's ability to capture the statistical properties and spatial structures of the turbulent flow. Our results demonstrate that DDPM and DCGAN effectively replicate all flow distributions, highlighting their potential as efficient and accurate tools for turbulence surrogacy. We find a strong argument for DCGAN, as although they are more difficult to train (due to problems such as mode collapse), they show the fastest inference and training time, require less data to train compared to VAE and DDPM, and provide the results most closely aligned with the input stream. In contrast, VAE train quickly (and can generate samples quickly) but do not produce adequate results, and DDPM, whilst effective, are significantly slower at both, inference and training time.
Authors Claudia Drygala, Edmund Ross, Francesca di Mare, Hanno Gottschalk
Categories Computer Vision and Pattern Recognition
0
CrossEarth-Gate: Fisher-Guided Adaptive Tuning Engine for Efficient Adaptation of Cross-Domain Remote Sensing Semantic Segmentation
Title CN CrossEarth-Gate：用于高效跨域遥感语义分割自适应的费雪引导自适应调优引擎
Keywords 遥感图像 · 参数高效微调 · 域自适应 · 费雪信息 · 语义分割
Summary 本文提出CrossEarth-Gate，一种基于费雪信息引导的自适应调优机制，用于解决遥感图像跨域语义分割中的多维度域偏移问题。通过构建包含空间、语义和频率模块的工具箱，并利用费雪信息动态选择关键模块进行激活，实现高效且有效的迁移学习。实验表明其在16个遥感跨域基准上达到SOTA。
Reason 该论文研究的是遥感图像（Remote Sensing）领域的语义分割任务，属于特定垂直领域（如卫星图像分析），与文档图像理解（DIU/DocVQA）的核心任务（如文本识别、布局分析、表格理解等）无关。尽管其提出的Fisher引导模块选择机制在参数高效微调方面有一定创新，但方法针对的是遥感中的域偏移问题（空间、频谱、语义变化），迁移到文档图像理解场景的适用性极低，且文档图像不涉及此类多模态遥感特征。因此属于无关领域，直接打0分。
Abstract In Remote Sensing (RS), Parameter-Efficient Fine-Tuning (PEFT) has emerged as a key approach to activate the generalizable representation ability of foundation models for downstream tasks. However, existing specialized PEFT methods often fail when applied to large-scale Earth observation tasks, as they are unable to fully handle the multifaceted and unpredictable domain gaps (\eg, spatial, semantic, and frequency shifts) inherent in RS data. To overcome this, we propose CrossEarth-Gate, which introduces two primary contributions. First, we establish a comprehensive RS module toolbox to address multifaceted domain gaps, comprising spatial, semantic, and frequency modules. Second, we develop a Fisher-guided adaptive selection mechanism that operates on this toolbox. This selection is guided by Fisher Information to quantify each module's importance by measuring its contribution to the task-specific gradient flow. It dynamically activates only the most critical modules at the appropriate layers, guiding the gradient flow to maximize adaptation effectiveness and efficiency. Comprehensive experiments validate the efficacy and generalizability of our method, where CrossEarth-Gate achieves state-of-the-art performance across 16 cross-domain benchmarks for RS semantic segmentation. The code of the work will be released.
Authors Shilei Cao, Ziyang Gong, Hehai Lin, Yang Liu, Jiashun Cheng, Xiaoxing Hu, Haoyuan Liang, Guowen Li, Chengwei Qin, Hong Cheng, Xue Yang, Juepeng Zheng, Haohuan Fu
Categories Computer Vision and Pattern Recognition
0
Probabilistic Robustness for Free? Revisiting Training via a Benchmark
Title CN 免费获得概率鲁棒性？通过基准重新审视训练
Keywords 概率鲁棒性 · 对抗训练 · 基准测试 · 模型鲁棒性 · 泛化误差
Summary 本文提出了PRBench，首个专门用于评估不同鲁棒性训练方法在概率鲁棒性（PR）方面表现的基准，系统比较了常见对抗训练（AT）与PR专用训练方法在清洁准确率、AR/PR性能、训练效率和泛化误差等方面的表现。结果显示AT方法在提升AR和PR方面更具通用性，而PR专用方法则具有更低的泛化误差和更高的清洁准确率。
Reason 该论文聚焦于深度学习模型在对抗性扰动下的概率鲁棒性（PR）评估与训练方法比较，属于通用计算机视觉中的鲁棒性研究，未涉及文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等。其提出的方法和基准PRBench面向的是图像分类场景下的模型鲁棒性问题，与文档理解中的OCR幻觉、复杂排版、细粒度定位等核心痛点无直接关联，也无法迁移用于解决文档多模态理解或高分辨率文本处理等问题。
Abstract Deep learning models are notoriously vulnerable to imperceptible perturbations. Most existing research centers on adversarial robustness (AR), which evaluates models under worst-case scenarios by examining the existence of deterministic adversarial examples (AEs). In contrast, probabilistic robustness (PR) adopts a statistical perspective, measuring the probability that predictions remain correct under stochastic perturbations. While PR is widely regarded as a practical complement to AR, dedicated training methods for improving PR are still relatively underexplored, albeit with emerging progress. Among the few PR-targeted training methods, we identify three limitations: i non-comparable evaluation protocols; ii limited comparisons to strong AT baselines despite anecdotal PR gains from AT; and iii no unified framework to compare the generalization of these methods. Thus, we introduce PRBench, the first benchmark dedicated to evaluating improvements in PR achieved by different robustness training methods. PRBench empirically compares most common AT and PR-targeted training methods using a comprehensive set of metrics, including clean accuracy, PR and AR performance, training efficiency, and generalization error (GE). We also provide theoretical analysis on the GE of PR performance across different training methods. Main findings revealed by PRBench include: AT methods are more versatile than PR-targeted training methods in terms of improving both AR and PR performance across diverse hyperparameter settings, while PR-targeted training methods consistently yield lower GE and higher clean accuracy. A leaderboard comprising 222 trained models across 7 datasets and 10 model architectures is publicly available at https://tmpspace.github.io/PRBenchLeaderboard/.
Authors Yi Zhang, Zheng Wang, Zhen Chen, Wenjie Ruan, Qing Guo, Siddartha Khastgir, Carsten Maple, Xingyu Zhao
Categories Computer Vision and Pattern Recognition
0
E-M3RF: An Equivariant Multimodal 3D Re-assembly Framework
Title CN E-M3RF：一种等变的多模态3D重装配框架
Keywords 3D reassembly · equivariant network · multimodal learning · point cloud · cultural heritage
Summary 本文提出E-M3RF，一种等变的多模态3D重装配框架，结合点云的位置与颜色信息，利用旋转等变编码器和Transformer分别提取几何与外观特征，并通过SE(3)流匹配预测碎片间的变换关系。在多个合成与真实文物数据集上验证了方法的有效性，显著降低了旋转、平移误差及Chamfer距离。
Reason 该论文研究的是3D碎片重装配问题，属于3D视觉与文化遗产修复领域，核心任务为基于点云和颜色特征进行SE(3)变换预测。尽管使用了多模态（几何+颜色）和等变网络，但其应用场景与文档图像理解（DIU）无直接关联。文档图像中的挑战如OCR幻觉、复杂排版、长文本推理等无法从中迁移受益。且该工作不涉及任何2D文档结构建模、视觉语言对齐或高分辨率图像处理机制。
Abstract 3D reassembly is a fundamental geometric problem, and in recent years it has increasingly been challenged by deep learning methods rather than classical optimization. While learning approaches have shown promising results, most still rely primarily on geometric features to assemble a whole from its parts. As a result, methods struggle when geometry alone is insufficient or ambiguous, for example, for small, eroded, or symmetric fragments. Additionally, solutions do not impose physical constraints that explicitly prevent overlapping assemblies. To address these limitations, we introduce E-M3RF, an equivariant multimodal 3D reassembly framework that takes as input the point clouds, containing both point positions and colors of fractured fragments, and predicts the transformations required to reassemble them using SE(3) flow matching. Each fragment is represented by both geometric and color features: i) 3D point positions are encoded as rotationconsistent geometric features using a rotation-equivariant encoder, ii) the colors at each 3D point are encoded with a transformer. The two feature sets are then combined to form a multimodal representation. We experimented on four datasets: two synthetic datasets, Breaking Bad and Fantastic Breaks, and two real-world cultural heritage datasets, RePAIR and Presious, demonstrating that E-M3RF on the RePAIR dataset reduces rotation error by 23.1% and translation error by 13.2%, while Chamfer Distance decreases by 18.4% compared to competing methods.
Authors Adeela Islam, Stefano Fiorini, Manuel Lecha, Theodore Tsesmelis, Stuart James, Pietro Morerio, Alessio Del Bue
Categories Computer Vision and Pattern Recognition
0
Learning Normals of Noisy Points by Local Gradient-Aware Surface Filtering ICCV 2025
Title CN 通过局部梯度感知表面滤波学习噪声点的法线
Keywords 点云 · 法线估计 · 3D几何处理 · 隐式场 · 去噪
Summary 本文提出了一种基于局部梯度感知表面滤波的方法，用于从含噪点云中学习法线。该方法通过隐式函数结合局部梯度信息，将噪声点投影到真实表面上，并引入距离测量算子和梯度一致性约束，以避免过平滑和梯度退化。实验表明该方法在法线估计、表面重建和点云去噪任务上达到SOTA。
Reason 该论文研究的是3D点云中噪声点的法线估计问题，属于3D视觉与几何处理领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其方法聚焦于隐式曲面建模和梯度约束滤波，难以迁移到文档图像中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Estimating normals for noisy point clouds is a persistent challenge in 3D geometry processing, particularly for end-to-end oriented normal estimation. Existing methods generally address relatively clean data and rely on supervised priors to fit local surfaces within specific neighborhoods. In this paper, we propose a novel approach for learning normals from noisy point clouds through local gradient-aware surface filtering. Our method projects noisy points onto the underlying surface by utilizing normals and distances derived from an implicit function constrained by local gradients. We start by introducing a distance measurement operator for global surface fitting on noisy data, which integrates projected distances along normals. Following this, we develop an implicit field-based filtering approach for surface point construction, adding projection constraints on these points during filtering. To address issues of over-smoothing and gradient degradation, we further incorporate local gradient consistency constraints, as well as local gradient orientation and aggregation. Comprehensive experiments on normal estimation, surface reconstruction, and point cloud denoising demonstrate the state-of-the-art performance of our method. The source code and trained models are available at https://github.com/LeoQLi/LGSF.
Authors Qing Li, Huifang Feng, Xun Gong, Yu-Shen Liu
Comment Accepted by ICCV 2025. Project page: https://leoqli.github.io/LGSF/
Categories Computer Vision and Pattern Recognition
0
SKEL-CF: Coarse-to-Fine Biomechanical Skeleton and Surface Mesh Recovery
Title CN SKEL-CF：粗到细的生物力学骨架与表面网格恢复
Keywords 3D人体重建 · 生物力学骨架 · SKEL模型 · Transformer · 姿态估计
Summary 本文提出SKEL-CF，一种从粗到细估计解剖学准确骨架（SKEL）参数的框架，用于提升3D人体姿态与形状估计的生物力学真实性。通过构建新的数据集4DHuman-SKEL并引入相机建模缓解深度模糊，模型在MOYO数据集上显著优于先前方法。
Reason 该论文研究的是3D人体骨架与运动建模，属于具身智能与3D视觉领域，与文档图像理解（DIU）或VLM在文档理解中的应用无直接关联。尽管其方法涉及视觉建模和Transformer架构，但核心问题设定（人体姿态估计、生物力学建模）与文档图像中的OCR、排版分析、视觉问答等任务完全无关，无法迁移解决DIU的核心痛点如幻觉、复杂布局或长文档推理。
Abstract Parametric 3D human models such as SMPL have driven significant advances in human pose and shape estimation, yet their simplified kinematics limit biomechanical realism. The recently proposed SKEL model addresses this limitation by re-rigging SMPL with an anatomically accurate skeleton. However, estimating SKEL parameters directly remains challenging due to limited training data, perspective ambiguities, and the inherent complexity of human articulation. We introduce SKEL-CF, a coarse-to-fine framework for SKEL parameter estimation. SKEL-CF employs a transformer-based encoder-decoder architecture, where the encoder predicts coarse camera and SKEL parameters, and the decoder progressively refines them in successive layers. To ensure anatomically consistent supervision, we convert the existing SMPL-based dataset 4DHuman into a SKEL-aligned version, 4DHuman-SKEL, providing high-quality training data for SKEL estimation. In addition, to mitigate depth and scale ambiguities, we explicitly incorporate camera modeling into the SKEL-CF pipeline and demonstrate its importance across diverse viewpoints. Extensive experiments validate the effectiveness of the proposed design. On the challenging MOYO dataset, SKEL-CF achieves 85.0 MPJPE / 51.4 PA-MPJPE, significantly outperforming the previous SKEL-based state-of-the-art HSMR (104.5 / 79.6). These results establish SKEL-CF as a scalable and anatomically faithful framework for human motion analysis, bridging the gap between computer vision and biomechanics. Our implementation is available on the project page: https://pokerman8.github.io/SKEL-CF/.
Authors Da Li, Jiping Jin, Xuanlong Yu, Wei Liu, Xiaodong Cun, Kai Chen, Rui Fan, Jiangang Kong, Xi Shen
Comment Project page: https://pokerman8.github.io/SKEL-CF/
Categories Computer Vision and Pattern Recognition
0
Dynamic Epsilon Scheduling: A Multi-Factor Adaptive Perturbation Budget for Adversarial Training
Title CN 动态 epsilon 调度：一种用于对抗训练的多因素自适应扰动预算
Keywords 对抗训练 · 动态扰动预算 · 模型鲁棒性 · 实例自适应 · 计算机视觉
Summary 本文提出了一种名为动态 epsilon 调度（DES）的新框架，用于在对抗训练中为每个样本和每个训练迭代步动态调整扰动强度。该方法结合了到决策边界的距离、预测置信度和模型不确定性三个因素，以实现更有效的对抗学习。实验表明该方法在CIFAR数据集上优于固定扰动预算和其他自适应方法。
Reason 该论文研究的是对抗训练中的扰动预算自适应问题，属于通用计算机视觉中的模型鲁棒性与安全领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其方法聚焦于提升图像分类模型对对抗样本的防御能力，并未涉及OCR、布局分析、表格识别、多模态推理或细粒度文本定位等DIU核心任务。尽管‘自适应’思想具有一定通用性，但该工作无法迁移解决DIU中的OCR幻觉、复杂排版建模、长文档推理或高分辨率视觉理解等关键痛点。
Abstract Adversarial training is among the most effective strategies for defending deep neural networks against adversarial examples. A key limitation of existing adversarial training approaches lies in their reliance on a fixed perturbation budget, which fails to account for instance-specific robustness characteristics. While prior works such as IAAT and MMA introduce instance-level adaptations, they often rely on heuristic or static approximations of data robustness. In this paper, we propose Dynamic Epsilon Scheduling (DES), a novel framework that adaptively adjusts the adversarial perturbation budget per instance and per training iteration. DES integrates three key factors: (1) the distance to the decision boundary approximated via gradient-based proxies, (2) prediction confidence derived from softmax entropy, and (3) model uncertainty estimated via Monte Carlo dropout. By combining these cues into a unified scheduling strategy, DES tailors the perturbation budget dynamically to guide more effective adversarial learning. Experimental results on CIFAR-10 and CIFAR-100 show that our method consistently improves both adversarial robustness and standard accuracy compared to fixed-epsilon baselines and prior adaptive methods. Moreover, we provide theoretical insights into the stability and convergence of our scheduling policy. This work opens a new avenue for instance-aware, data-driven adversarial training methods.
Authors Alan Mitkiy, James Smith, Myungseo wong, Hana Satou, Hiroshi Tanaka, Emily Johnson
Categories Computer Vision and Pattern Recognition
0
VA-GS: Enhancing the Geometric Representation of Gaussian Splatting via View Alignment NeurIPS 2025
Title CN 通过视图对齐增强高斯点阵的几何表示
Keywords 3D Gaussian Splatting · View Alignment · Surface Reconstruction · Novel View Synthesis · Geometric Consistency
Summary 本文提出了一种通过视图对齐（VA）来增强3D高斯点阵几何表示的方法，引入边缘感知渲染损失、可见性感知的光度对齐损失、法向约束和深度特征嵌入，以提升表面重建质量和多视角一致性。实验表明该方法在标准基准上达到了SOTA性能。
Reason 该论文研究的是3D高斯点阵在新视角合成中的几何表示优化，属于3D视觉与计算机图形学领域。其核心问题（如表面重建、多视角一致性）与文档图像理解（DIU）的任务无直接关联，且方法难以迁移到OCR、布局分析、表格识别等DIU核心痛点。属于无关领域。
Abstract 3D Gaussian Splatting has recently emerged as an efficient solution for high-quality and real-time novel view synthesis. However, its capability for accurate surface reconstruction remains underexplored. Due to the discrete and unstructured nature of Gaussians, supervision based solely on image rendering loss often leads to inaccurate geometry and inconsistent multi-view alignment. In this work, we propose a novel method that enhances the geometric representation of 3D Gaussians through view alignment (VA). Specifically, we incorporate edge-aware image cues into the rendering loss to improve surface boundary delineation. To enforce geometric consistency across views, we introduce a visibility-aware photometric alignment loss that models occlusions and encourages accurate spatial relationships among Gaussians. To further mitigate ambiguities caused by lighting variations, we incorporate normal-based constraints to refine the spatial orientation of Gaussians and improve local surface estimation. Additionally, we leverage deep image feature embeddings to enforce cross-view consistency, enhancing the robustness of the learned geometry under varying viewpoints and illumination. Extensive experiments on standard benchmarks demonstrate that our method achieves state-of-the-art performance in both surface reconstruction and novel view synthesis. The source code is available at https://github.com/LeoQLi/VA-GS.
Authors Qing Li, Huifang Feng, Xun Gong, Yu-Shen Liu
Comment Accepted by NeurIPS 2025
Categories Computer Vision and Pattern Recognition
0
Endo-G$^{2}$T: Geometry-Guided & Temporally Aware Time-Embedded 4DGS For Endoscopic Scenes
Title CN Endo-G²T：面向内窥镜场景的几何引导与时间感知的时间嵌入式4D高斯溅射
Keywords 4D重建 · 高斯溅射 · 内窥镜 · 几何引导 · 时间一致性
Summary 本文提出Endo-G²T，一种用于时间嵌入式4D高斯溅射的几何引导与时间感知训练方法，旨在提升内窥镜动态场景下的单目3D重建质量。通过引入单目深度先验蒸馏、XYZT时空表示与关键帧约束流式优化，在EndoNeRF和StereoMIS-P1数据集上实现了领先性能。
Reason 该论文研究的是内窥镜视频场景下的动态3D重建问题，属于3D视觉与医学图像重建领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心技术聚焦于4D高斯溅射中的几何引导训练和时间一致性建模，无法迁移解决OCR幻觉、复杂排版解析或长文档推理等DIU核心痛点。
Abstract Endoscopic (endo) video exhibits strong view-dependent effects such as specularities, wet reflections, and occlusions. Pure photometric supervision misaligns with geometry and triggers early geometric drift, where erroneous shapes are reinforced during densification and become hard to correct. We ask how to anchor geometry early for 4D Gaussian splatting (4DGS) while maintaining temporal consistency and efficiency in dynamic endoscopic scenes. Thus, we present Endo-G$^{2}$T, a geometry-guided and temporally aware training scheme for time-embedded 4DGS. First, geo-guided prior distillation converts confidence-gated monocular depth into supervision with scale-invariant depth and depth-gradient losses, using a warm-up-to-cap schedule to inject priors softly and avoid early overfitting. Second, a time-embedded Gaussian field represents dynamics in XYZT with a rotor-like rotation parameterization, yielding temporally coherent geometry with lightweight regularization that favors smooth motion and crisp opacity boundaries. Third, keyframe-constrained streaming improves efficiency and long-horizon stability through keyframe-focused optimization under a max-points budget, while non-keyframes advance with lightweight updates. Across EndoNeRF and StereoMIS-P1 datasets, Endo-G$^{2}$T achieves state-of-the-art results among monocular reconstruction baselines.
Authors Yangle Liu, Fengze Li, Kan Liu, Jieming Ma
Categories Computer Vision and Pattern Recognition
0
PFF-Net: Patch Feature Fitting for Point Cloud Normal Estimation TVCG
Title CN PFF-Net：用于点云法线估计的补丁特征拟合
Keywords 点云 · 法线估计 · 多尺度特征 · 特征融合 · 3D几何
Summary 本文提出PFF-Net，通过多尺度特征聚合和跨尺度特征补偿来实现点云局部补丁的特征拟合，以解决不同几何结构下法线估计中邻域大小选择困难的问题。方法在合成和真实数据集上实现了最先进的性能，同时参数更少、运行更快。
Reason 该论文研究的是点云法线估计，属于3D几何处理领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。尽管其多尺度特征融合策略具有一定通用性，但迁移到文档图像理解中的OCR幻觉、复杂排版或长文档推理等问题上极为困难，且未涉及文本、布局或语义理解等DIU核心挑战。
Abstract Estimating the normal of a point requires constructing a local patch to provide center-surrounding context, but determining the appropriate neighborhood size is difficult when dealing with different data or geometries. Existing methods commonly employ various parameter-heavy strategies to extract a full feature description from the input patch. However, they still have difficulties in accurately and efficiently predicting normals for various point clouds. In this work, we present a new idea of feature extraction for robust normal estimation of point clouds. We use the fusion of multi-scale features from different neighborhood sizes to address the issue of selecting reasonable patch sizes for various data or geometries. We seek to model a patch feature fitting (PFF) based on multi-scale features to approximate the optimal geometric description for normal estimation and implement the approximation process via multi-scale feature aggregation and cross-scale feature compensation. The feature aggregation module progressively aggregates the patch features of different scales to the center of the patch and shrinks the patch size by removing points far from the center. It not only enables the network to precisely capture the structure characteristic in a wide range, but also describes highly detailed geometries. The feature compensation module ensures the reusability of features from earlier layers of large scales and reveals associated information in different patch sizes. Our approximation strategy based on aggregating the features of multiple scales enables the model to achieve scale adaptation of varying local patches and deliver the optimal feature description. Extensive experiments demonstrate that our method achieves state-of-the-art performance on both synthetic and real-world datasets with fewer network parameters and running time.
Authors Qing Li, Huifang Feng, Kanle Shi, Yue Gao, Yi Fang, Yu-Shen Liu, Zhizhong Han
Comment Accepted by TVCG
Categories Computer Vision and Pattern Recognition
0
BanglaMM-Disaster: A Multimodal Transformer-Based Deep Learning Framework for Multiclass Disaster Classification in Bangla 2025 IEEE International Conference on Signal Processing, Information, Communication and Systems (SPICSCON)
Title CN BanglaMM-Disaster：一种基于多模态Transformer的孟加拉语多类别灾害分类深度学习框架
Keywords 多模态分类 · 孟加拉语 · 灾害检测 · 社交媒体分析 · Transformer
Summary 本文提出了一种用于孟加拉语社交媒体报道中灾害分类的多模态深度学习框架BanglaMM-Disaster，结合文本（如BanglaBERT）和图像（如ResNet50）模型进行早期融合，构建了一个包含5037条样本的数据集，在九类灾害上实现了83.76%的准确率。虽然验证了多模态融合在低资源语言场景下的有效性，但整体为特定语言和场景的应用工作。
Reason 该论文属于典型的‘平行下游应用’，其核心是将现有VLM和CNN架构应用于孟加拉语社交媒体数据的灾害分类任务。尽管构建了新的多模态数据集，但方法上无本质创新，且聚焦于小语种（非中英）的特定应用场景，完全符合负面清单中的两条：一是‘平行应用’，二是‘小语种’。无法为文档图像理解提供可迁移的方法论贡献。
Abstract Natural disasters remain a major challenge for Bangladesh, so real-time monitoring and quick response systems are essential. In this study, we present BanglaMM-Disaster, an end-to-end deep learning-based multimodal framework for disaster classification in Bangla, using both textual and visual data from social media. We constructed a new dataset of 5,037 Bangla social media posts, each consisting of a caption and a corresponding image, annotated into one of nine disaster-related categories. The proposed model integrates transformer-based text encoders, including BanglaBERT, mBERT, and XLM-RoBERTa, with CNN backbones such as ResNet50, DenseNet169, and MobileNetV2, to process the two modalities. Using early fusion, the best model achieves 83.76% accuracy. This surpasses the best text-only baseline by 3.84% and the image-only baseline by 16.91%. Our analysis also shows reduced misclassification across all classes, with noticeable improvements for ambiguous examples. This work fills a key gap in Bangla multimodal disaster analysis and demonstrates the benefits of combining multiple data types for real-time disaster response in low-resource settings.
Authors Ariful Islam, Md Rifat Hossen, Md. Mahmudul Arif, Abdullah Al Noman, Md Arifur Rahman
Comment Presented at the 2025 IEEE International Conference on Signal Processing, Information, Communication and Systems (SPICSCON), November 21-22, 2025, University of Rajshahi, Bangladesh. 6 pages, 9 disaster classes, multimodal dataset with 5,037 samples
Categories Computer Vision and Pattern Recognition
0
Stream and Query-guided Feature Aggregation for Efficient and Effective 3D Occupancy Prediction
Title CN 用于高效且有效的3D占用预测的流式与查询引导特征聚合
Keywords 3D occupancy prediction · autonomous driving · spatiotemporal fusion · voxel aggregation · query-guided feature injection
Summary 本文提出DuOcc模型，通过双路径特征聚合策略提升3D占用预测的效率与精度：一方面通过流式体素聚合实现时序特征累积并抑制形变失真，另一方面利用查询引导机制将实例级特征有选择地注入动态物体区域，从而在保持高空间保真度的同时降低内存消耗。实验表明其在Occ3D-nuScenes和SurroundOcc数据集上实现了实时性下的SOTA性能。
Reason 该论文研究的是自动驾驶中的3D占用预测问题，属于纯计算机视觉中与机器人、自动驾驶强相关的领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心方法——流式体素聚合和查询引导特征注入——针对动态场景下的空间建模，难以迁移到文档理解任务中解决OCR幻觉、复杂排版或长文档推理等痛点。
Abstract 3D occupancy prediction has become a key perception task in autonomous driving, as it enables comprehensive scene understanding. Recent methods enhance this understanding by incorporating spatiotemporal information through multi-frame fusion, but they suffer from a trade-off: dense voxel-based representations provide high accuracy at significant computational cost, whereas sparse representations improve efficiency but lose spatial detail. To mitigate this trade-off, we introduce DuOcc, which employs a dual aggregation strategy that retains dense voxel representations to preserve spatial fidelity while maintaining high efficiency. DuOcc consists of two key components: (i) Stream-based Voxel Aggregation, which recurrently accumulates voxel features over time and refines them to suppress warping-induced distortions, preserving a clear separation between occupied and free space. (ii) Query-guided Aggregation, which complements the limitations of voxel accumulation by selectively injecting instance-level query features into the voxel regions occupied by dynamic objects. Experiments on the widely used Occ3D-nuScenes and SurroundOcc datasets demonstrate that DuOcc achieves state-of-the-art performance in real-time settings, while reducing memory usage by over 40% compared to prior methods.
Authors Seokha Moon, Janghyun Baek, Giseop Kim, Jinkyu Kim, Sunwook Choi
Categories Computer Vision and Pattern Recognition
0
Image as an IMU: Estimating Camera Motion from a Single Motion-Blurred Image
Title CN 图像即IMU：从单个运动模糊图像中估计相机运动
Keywords 运动模糊 · 相机姿态估计 · 视觉里程计 · 深度估计 · 光流
Summary 本文提出一种利用运动模糊作为线索来估计相机运动的新方法，通过从单张模糊图像预测密集光流和单目深度图，并在小运动假设下求解线性最小二乘问题以恢复瞬时相机速度。该方法在真实数据集上实现了对旋转和平移速度的SOTA估计，表现优于MASt3R和COLMAP等现有方法。尽管技术新颖，但其应用场景完全偏离文档理解范畴。
Reason 该论文研究的是从运动模糊图像中恢复相机运动轨迹，属于视觉SLAM和机器人感知领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心技术聚焦于运动模糊建模、深度估计与相机速度推算，无法迁移解决OCR幻觉、复杂版面分析、细粒度文本定位或长文档推理等DIU核心问题。
Abstract In many robotics and VR/AR applications, fast camera motions lead to a high level of motion blur, causing existing camera pose estimation methods to fail. In this work, we propose a novel framework that leverages motion blur as a rich cue for motion estimation rather than treating it as an unwanted artifact. Our approach works by predicting a dense motion flow field and a monocular depth map directly from a single motion-blurred image. We then recover the instantaneous camera velocity by solving a linear least squares problem under the small motion assumption. In essence, our method produces an IMU-like measurement that robustly captures fast and aggressive camera movements. To train our model, we construct a large-scale dataset with realistic synthetic motion blur derived from ScanNet++v2 and further refine our model by training end-to-end on real data using our fully differentiable pipeline. Extensive evaluations on real-world benchmarks demonstrate that our method achieves state-of-the-art angular and translational velocity estimates, outperforming current methods like MASt3R and COLMAP.
Authors Jerred Chen, Ronald Clark
Comment Project page: https://jerredchen.github.io/image-as-imu/
Categories Computer Vision and Pattern Recognition
0
Restoration-Oriented Video Frame Interpolation with Region-Distinguishable Priors from SAM
Title CN 基于SAM区域可区分先验的面向修复的视频帧插值
Keywords 视频帧插值 · SAM · 运动估计 · 图像修复 · 特征融合
Summary 本文提出一种面向修复的视频帧插值方法，利用SAM2等开放世界分割模型生成区域可区分先验（RDP），并通过设计的HRFFM模块将RDP融入现有VFI框架中，提升运动估计精度。RDP以空间变化的高斯混合形式表示多个区域，通过RDP引导的特征归一化机制增强编码器中匹配区域的特征一致性，从而改善中间帧合成效果。实验表明该方法在多种场景下均能稳定提升VFI性能。
Reason 该论文属于视频理解领域，研究的是视频帧插值中的运动估计问题，使用SAM提取区域先验来提升插值质量。尽管其引入外部分割模型作为先验信息的思想具有一定启发性，但整体工作与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。所解决的问题（视频帧间运动模糊）和所用技术路径（帧插值+高分辨率视频处理）难以迁移到文档场景中用于解决OCR幻觉、复杂排版或长文档推理等痛点。
Abstract In existing restoration-oriented Video Frame Interpolation (VFI) approaches, the motion estimation between neighboring frames plays a crucial role. However, the estimation accuracy in existing methods remains a challenge, primarily due to the inherent ambiguity in identifying corresponding areas in adjacent frames for interpolation. Therefore, enhancing accuracy by distinguishing different regions before motion estimation is of utmost importance. In this paper, we introduce a novel solution involving the utilization of open-world segmentation models, e.g., SAM2 (Segment Anything Model2) for frames, to derive Region-Distinguishable Priors (RDPs) in different frames. These RDPs are represented as spatial-varying Gaussian mixtures, distinguishing an arbitrary number of areas with a unified modality. RDPs can be integrated into existing motion-based VFI methods to enhance features for motion estimation, facilitated by our designed play-and-plug Hierarchical Region-aware Feature Fusion Module (HRFFM). HRFFM incorporates RDP into various hierarchical stages of VFI's encoder, using RDP-guided Feature Normalization (RDPFN) in a residual learning manner. With HRFFM and RDP, the features within VFI's encoder exhibit similar representations for matched regions in neighboring frames, thus improving the synthesis of intermediate frames. Extensive experiments demonstrate that HRFFM consistently enhances VFI performance across various scenes.
Authors Yan Han, Xiaogang Xu, Yingqi Lin, Jiafei Wu, Zhe Liu, Ming-Hsuan Yang
Comment Code will be released
Categories Computer Vision and Pattern Recognition
0
HTTM: Head-wise Temporal Token Merging for Faster VGGT
Title CN HTTM：用于加速VGGT的逐头时序Token合并
Keywords 3D重建 · 视觉Transformer · Token压缩 · 多头注意力 · 推理加速
Summary 本文提出了一种名为HTTM（逐头时序Token合并）的方法，用于加速VGGT模型在3D场景重建中的推理过程。该方法通过在不同注意力头中独立进行token合并，保留了特征多样性，并利用头级别的时空局部性实现更高压缩比和更低计算开销，可在GPU推理中实现最高7倍的加速且性能损失极小。
Reason 该论文研究的是3D场景重建中的视觉Transformer加速问题，属于3D视觉与多视图几何领域，与文档图像理解（DIU）任务无直接关联。其提出的方法HTTM针对的是多视角输入下的全局注意力计算瓶颈，应用场景和问题设定均远离文档图像中的文本识别、布局分析或视觉问答等核心任务。尽管Token压缩思想在广义上与视觉token压缩有概念相似性，但该方法依赖于多头间的时空对应关系，在文档图像这种缺乏明确时序与立体结构的模态中难以迁移应用。
Abstract The Visual Geometry Grounded Transformer (VGGT) marks a significant leap forward in 3D scene reconstruction, as it is the first model that directly infers all key 3D attributes (camera poses, depths, and dense geometry) jointly in one pass. However, this joint inference mechanism requires global attention layers that perform all-to-all attention computation on tokens from all views. For reconstruction of large scenes with long-sequence inputs, this causes a significant latency bottleneck. In this paper, we propose head-wise temporal merging (HTTM), a training-free 3D token merging method for accelerating VGGT. Existing merging techniques merge tokens uniformly across different attention heads, resulting in identical tokens in the layers' output, which hinders the model's representational ability. HTTM tackles this problem by merging tokens in multi-head granularity, which preserves the uniqueness of feature tokens after head concatenation. Additionally, this enables HTTM to leverage the spatial locality and temporal correspondence observed at the head level to achieve higher merging ratios with lower merging costs compared to existing methods. Thus, HTTM achieves up to 7x acceleration with negligible performance drops in a GPU-based inference.
Authors Weitian Wang, Lukas Meiner, Rai Shubham, Cecilia De La Parra, Akash Kumar
Categories Computer Vision and Pattern Recognition
0
Towards Consistent and Controllable Image Synthesis for Face Editing
Title CN 面向一致且可控的人脸编辑图像合成
Keywords 图像生成 · 扩散模型 · 人脸编辑 · 身份保持 · 3D人脸模型
Summary 本文提出RigFace方法，利用Stable Diffusion和粗略3D人脸模型来控制人像的光照、表情和姿态。通过空间属性编码器、高一致性FaceFusion和属性控制器实现身份特征保留与属性解耦控制，在人脸编辑中取得良好效果。
Reason 该论文属于纯图像生成领域，研究基于扩散模型的人脸编辑，关注点为身份保持和属性控制，与文档图像理解（DIU）或视觉语言模型在文档任务中的应用无直接关联。其技术路线聚焦于人脸图像的生成质量与编辑控制，无法迁移解决OCR幻觉、密集文本识别、复杂版式分析或长文档推理等DIU核心问题。
Abstract Face editing methods, essential for tasks like virtual avatars, digital human synthesis and identity preservation, have traditionally been built upon GAN-based techniques, while recent focus has shifted to diffusion-based models due to their success in image reconstruction. However, diffusion models still face challenges in controlling specific attributes and preserving the consistency of other unchanged attributes especially the identity characteristics. To address these issues and facilitate more convenient editing of face images, we propose a novel approach that leverages the power of Stable-Diffusion (SD) models and crude 3D face models to control the lighting, facial expression and head pose of a portrait photo. We observe that this task essentially involves the combinations of target background, identity and face attributes aimed to edit. We strive to sufficiently disentangle the control of these factors to enable consistency of face editing. Specifically, our method, coined as RigFace, contains: 1) A Spatial Attribute Encoder that provides presise and decoupled conditions of background, pose, expression and lighting; 2) A high-consistency FaceFusion method that transfers identity features from the Identity Encoder to the denoising UNet of a pre-trained SD model; 3) An Attribute Rigger that injects those conditions into the denoising UNet. Our model achieves comparable or even superior performance in both identity preservation and photorealism compared to existing face editing models.
Authors Mengting Wei, Tuomas Varanka, Yante Li, Xingxun Jiang, Huai-Qian Khor, Guoying Zhao
Categories Computer Vision and Pattern Recognition
0
CaliTex: Geometry-Calibrated Attention for View-Coherent 3D Texture Generation
Title CN CaliTex：用于视图一致的3D纹理生成的几何校准注意力
Keywords 3D纹理生成 · 注意力机制 · 几何校准 · 扩散模型 · 跨视角一致性
Summary 本文提出CaliTex，一种基于几何校准注意力的3D纹理生成框架，通过引入部件对齐注意力和条件路由注意力，在扩散Transformer中实现跨视角一致的高质量纹理生成。方法聚焦于3D结构与外观的空间耦合，提升多视角下的纹理连贯性。
Reason 该论文研究的是3D纹理生成中的跨视角一致性问题，属于3D视觉与图形学领域，核心挑战是几何结构与外观在多视角下的对齐。其提出的方法（如Part-Aligned Attention和Condition-Routed Attention）针对3D场景建模设计，并不适用于文档图像理解中的OCR幻觉、密集文本识别、复杂版式分析或长文档推理等核心痛点。尽管涉及注意力机制改进，但迁移至文档理解场景的可行性极低，且与DIU任务无直接关联。
Abstract Despite major advances brought by diffusion-based models, current 3D texture generation systems remain hindered by cross-view inconsistency -- textures that appear convincing from one viewpoint often fail to align across others. We find that this issue arises from attention ambiguity, where unstructured full attention is applied indiscriminately across tokens and modalities, causing geometric confusion and unstable appearance-structure coupling. To address this, we introduce CaliTex, a framework of geometry-calibrated attention that explicitly aligns attention with 3D structure. It introduces two modules: Part-Aligned Attention that enforces spatial alignment across semantically matched parts, and Condition-Routed Attention which routes appearance information through geometry-conditioned pathways to maintain spatial fidelity. Coupled with a two-stage diffusion transformer, CaliTex makes geometric coherence an inherent behavior of the network rather than a byproduct of optimization. Empirically, CaliTex produces seamless and view-consistent textures and outperforms both open-source and commercial baselines.
Authors Chenyu Liu, Hongze Chen, Jingzhi Bao, Lingting Zhu, Runze Zhang, Weikai Chen, Zeyu Hu, Yingda Yin, Keyang Luo, Xin Wang
Categories Computer Vision and Pattern Recognition
0
From Limited Labels to Open Domains:An Efficient Learning Method for Drone-view Geo-Localization
Title CN 从有限标签到开放领域：一种高效的无人机视角地理定位学习方法
Keywords 无人机视角地理定位 · 跨域适应 · 伪标签 · 对比学习 · 不变性特征
Summary 本文提出了一种名为CDIKTNet的网络，用于在仅有少量配对数据的情况下提升无人机视角地理定位的跨域性能。该方法通过跨域不变性子网络和双路径对比学习来缓解特征混淆问题，在全监督和少样本场景下均表现出优越性能。
Reason 该论文研究的是无人机视角地理定位（DVGL），属于遥感与地理定位领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。尽管其方法涉及跨域不变性学习和伪标签优化，但应用场景和核心问题（如视觉-地理对齐）与文档中的OCR、布局分析、表格识别等痛点无关，无法迁移用于解决文档理解中的幻觉、密集文本或复杂排版等问题。
Abstract Traditional supervised drone-view geo-localization (DVGL) methods heavily depend on paired training data and encounter difficulties in learning cross-view correlations from unpaired data. Moreover, when deployed in a new domain, these methods require obtaining the new paired data and subsequent retraining for model adaptation, which significantly increases computational overhead. Existing unsupervised methods have enabled to generate pseudo-labels based on cross-view similarity to infer the pairing relationships. However, geographical similarity and spatial continuity often cause visually analogous features at different geographical locations. The feature confusion compromises the reliability of pseudo-label generation, where incorrect pseudo-labels drive negative optimization. Given these challenges inherent in both supervised and unsupervised DVGL methods, we propose a novel cross-domain invariant knowledge transfer network (CDIKTNet) with limited supervision, whose architecture consists of a cross-domain invariance sub-network (CDIS) and a cross-domain transfer sub-network (CDTS). This architecture facilitates a closed-loop framework for invariance feature learning and knowledge transfer. The CDIS is designed to learn cross-view structural and spatial invariance from a small amount of paired data that serves as prior knowledge. It endows the shared feature space of unpaired data with similar implicit cross-view correlations at initialization, which alleviates feature confusion. Based on this, the CDTS employs dual-path contrastive learning to further optimize each subspace while preserving consistency in a shared feature space. Extensive experiments demonstrate that CDIKTNet achieves state-of-the-art performance under full supervision compared with those supervised methods, and further surpasses existing unsupervised methods in both few-shot and cross-domain initialization.
Authors Zhongwei Chen, Zhao-Xu Yang, Hai-Jun Rong, Jiawei Lang, Guoqi Li
Categories Computer Vision and Pattern Recognition
0
GreenHyperSpectra: A multi-source hyperspectral dataset for global vegetation trait prediction 39th Conference on Neural Information Processing Systems (NeurIPS 2025)
Title CN GreenHyperSpectra：一种用于全球植被性状预测的多源高光谱数据集
Keywords 高光谱图像 · 植物性状预测 · 自监督学习 · 跨域泛化 · 遥感
Summary 本文提出了GreenHyperSpectra，一个用于植物功能性状预测的多源高光谱预训练数据集，旨在通过自监督和半监督方法缓解标签稀缺和跨传感器域偏移问题。实验表明该数据集能有效提升跨域性状预测性能。
Reason 该论文聚焦于遥感高光谱数据下的植物性状预测，属于生态学与遥感交叉领域，任务为回归预测而非文档图像理解（DIU）或视觉语言模型（VLM）相关的文档问答、布局分析等核心任务。尽管涉及跨域表示学习，但其方法和数据均针对植被光谱特征设计，无法迁移至解决OCR幻觉、复杂排版理解、细粒度文本定位等DIU痛点，属于无关领域。
Abstract Plant traits such as leaf carbon content and leaf mass are essential variables in the study of biodiversity and climate change. However, conventional field sampling cannot feasibly cover trait variation at ecologically meaningful spatial scales. Machine learning represents a valuable solution for plant trait prediction across ecosystems, leveraging hyperspectral data from remote sensing. Nevertheless, trait prediction from hyperspectral data is challenged by label scarcity and substantial domain shifts (\eg across sensors, ecological distributions), requiring robust cross-domain methods. Here, we present GreenHyperSpectra, a pretraining dataset encompassing real-world cross-sensor and cross-ecosystem samples designed to benchmark trait prediction with semi- and self-supervised methods. We adopt an evaluation framework encompassing in-distribution and out-of-distribution scenarios. We successfully leverage GreenHyperSpectra to pretrain label-efficient multi-output regression models that outperform the state-of-the-art supervised baseline. Our empirical analyses demonstrate substantial improvements in learning spectral representations for trait prediction, establishing a comprehensive methodological framework to catalyze research at the intersection of representation learning and plant functional traits assessment. All code and data are available at: https://github.com/echerif18/HyspectraSSL.
Authors Eya Cherif, Arthur Ouaknine, Luke A. Brown, Phuong D. Dao, Kyle R. Kovach, Bing Lu, Daniel Mederer, Hannes Feilhauer, Teja Kattenborn, David Rolnick
Comment Accepted at the 39th Conference on Neural Information Processing Systems (NeurIPS 2025)
Categories Computer Vision and Pattern Recognition
0
Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale
Title CN 用于大规模稳定且富有韵律的单码本TTS大语言模型的多奖励GRPO方法
Keywords TTS · 大语言模型 · 语音合成 · 强化学习 · 韵律建模
Summary 本文提出一种基于多奖励组相对策略优化（Multi-Reward GRPO）的方法，用于提升单码本TTS大语言模型在语音合成中的韵律稳定性、说话人一致性和自然度。通过引入长度惩罚、熵正则化和由LLM标注的韵律对齐奖励，优化离散语音token的生成策略，并结合流匹配解码器进一步提升性能。
Reason 该论文研究的是文本到语音（TTS）合成中的韵律稳定性问题，属于语音生成领域，与文档图像理解（DIU）、DocVQA、OCR、布局分析等任务完全无关。尽管使用了LLM和强化学习优化策略，但其技术路线无法直接迁移至解决文档理解中的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）。属于无关领域。
Abstract Recent advances in Large Language Models (LLMs) have transformed text-to-speech (TTS) synthesis, inspiring autoregressive frameworks that represent speech as sequences of discrete codec tokens. Among them, single-codebook TTS LLMs have emerged as compact and streamable architectures that jointly model semantic and acoustic integration. However, despite their efficiency, these models often exhibit unstable prosody, speaker drift, and degraded naturalness. To address these issues, we propose a multi-reward Group Relative Policy Optimization (GRPO) framework that directly optimizes the token generation policy of single-codebook TTS LLMs. Beyond standard intelligibility and speaker similarity objectives, our design integrates three rule-based rewards: a length penalty for duration consistency, an entropy regularization reward for decoding stability, and an LLM-annotated prosody alignment reward that explicitly supervises rhythm. In this prosody reward, an external reasoning LLM predicts multiple plausible pause structures via in-context learning, providing a human-preference-aligned supervisory signal for GRPO training. To assess universality, we further attach a flow-matching (FM) decoder on top of the GRPO-optimized AR backbone and observe consistent additional gains, indicating that our reinforcement optimization enhances the intrinsic AR policy. We further conduct a scalability analysis across data sizes and model scales, revealing that the proposed method consistently enhances prosodic stability, speaker similarity, and overall speech naturalness in single-codebook TTS LLMs.
Authors Yicheng Zhong, Peiji Yang, Zhisheng Wang
Comment 4 pages, 2 figures
Categories Computer Vision and Pattern Recognition
0
Unlocking Zero-shot Potential of Semi-dense Image Matching via Gaussian Splatting
Title CN 通过高斯点阵解锁半稠密图像匹配的零样本潜力
Keywords 图像匹配 · 3D高斯点阵 · 零样本学习 · 数据生成 · 几何校正
Summary 本文提出MatchGS，是首个利用3D高斯点阵（3DGS）进行零样本半稠密图像匹配的框架。通过几何修正的3DGS生成高精度对应关系标签，并设计2D-3D表示对齐策略，使2D匹配器能学习到视点不变的3D表示。在极端视角变化下仍能提供有效监督，显著提升现有匹配模型的零样本性能。
Reason 该论文研究的是通用图像匹配任务，基于3D高斯点阵（3DGS）生成高质量的对应点标签，属于三维视觉与图像匹配领域。其核心问题和方法均围绕自由视角图像间的匹配展开，与文档图像理解（DIU）中的文本识别、布局分析、表格理解等任务无直接关联。虽然其数据生成和几何校正方法在理论上具有通用性，但迁移到文档图像场景（如OCR、DocVQA）存在根本障碍：文档图像本质是平面、单视角或近似平面的结构，不涉及多视角几何与3D重建。因此，该方法对解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点几乎无法迁移应用。
Abstract Learning-based image matching critically depends on large-scale, diverse, and geometrically accurate training data. 3D Gaussian Splatting (3DGS) enables photorealistic novel-view synthesis and thus is attractive for data generation. However, its geometric inaccuracies and biased depth rendering currently prevent robust correspondence labeling. To address this, we introduce MatchGS, the first framework designed to systematically correct and leverage 3DGS for robust, zero-shot image matching. Our approach is twofold: (1) a geometrically-faithful data generation pipeline that refines 3DGS geometry to produce highly precise correspondence labels, enabling the synthesis of a vast and diverse range of viewpoints without compromising rendering fidelity; and (2) a 2D-3D representation alignment strategy that infuses 3DGS' explicit 3D knowledge into the 2D matcher, guiding 2D semi-dense matchers to learn viewpoint-invariant 3D representations. Our generated ground-truth correspondences reduce the epipolar error by up to 40 times compared to existing datasets, enable supervision under extreme viewpoint changes, and provide self-supervisory signals through Gaussian attributes. Consequently, state-of-the-art matchers trained solely on our data achieve significant zero-shot performance gains on public benchmarks, with improvements of up to 17.7%. Our work demonstrates that with proper geometric refinement, 3DGS can serve as a scalable, high-fidelity, and structurally-rich data source, paving the way for a new generation of robust zero-shot image matchers.
Authors Juncheng Chen, Chao Xu, Yanjun Cao
Categories Computer Vision and Pattern Recognition
0
LaGen: Towards Autoregressive LiDAR Scene Generation
Title CN LaGen：迈向自回归的激光雷达场景生成
Keywords LiDAR生成 · 自回归模型 · 4D点云 · 自动驾驶 · 世界模型
Summary 本文提出了LaGen，首个支持帧对帧自回归生成长时程LiDAR场景的生成框架，可用于自动驾驶中的世界建模。它基于单帧LiDAR输入和边界框条件生成高保真4D点云，并通过场景解耦估计和噪声调制模块提升交互性和减少误差累积。实验表明其在nuScenes上的长时生成性能优于现有方法。
Reason 该论文研究的是自动驾驶中的LiDAR点云场景生成，属于3D视觉与具身智能领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无任何关联。其方法聚焦于4D点云序列的自回归生成和长程交互建模，虽然在生成模型设计上有一定技术深度，但无法迁移到文档理解的核心痛点（如OCR幻觉、复杂排版、细粒度文本识别等）。
Abstract Generative world models for autonomous driving (AD) have become a trending topic. Unlike the widely studied image modality, in this work we explore generative world models for LiDAR data. Existing generation methods for LiDAR data only support single frame generation, while existing prediction approaches require multiple frames of historical input and can only deterministically predict multiple frames at once, lacking interactivity. Both paradigms fail to support long-horizon interactive generation. To this end, we introduce LaGen, which to the best of our knowledge is the first framework capable of frame-by-frame autoregressive generation of long-horizon LiDAR scenes. LaGen is able to take a single-frame LiDAR input as a starting point and effectively utilize bounding box information as conditions to generate high-fidelity 4D scene point clouds. In addition, we introduce a scene decoupling estimation module to enhance the model's interactive generation capability for object-level content, as well as a noise modulation module to mitigate error accumulation during long-horizon generation. We construct a protocol based on nuScenes for evaluating long-horizon LiDAR scene generation. Experimental results comprehensively demonstrate LaGen outperforms state-of-the-art LiDAR generation and prediction models, especially on the later frames.
Authors Sizhuo Zhou, Xiaosong Jia, Fanrui Zhang, Junjie Li, Juyong Zhang, Yukang Feng, Jianwen Sun, Songbur Wong, Junqi You, Junchi Yan
Categories Computer Vision and Pattern Recognition
0
DWFF-Net : A Multi-Scale Farmland System Habitat Identification Method with Adaptive Dynamic Weight
Title CN DWFF-Net：一种具有自适应动态权重的多尺度农田系统生境识别方法
Keywords 遥感图像 · 语义分割 · 多尺度特征融合 · DINOv3 · 生境识别
Summary 本文提出了一种名为DWFF-Net的多尺度农田生境识别网络，基于自适应动态加权机制融合多层特征，利用DINOv3作为冻结编码器，并在自建的超高清遥感数据集上实现了优于基线模型的分割性能。主要贡献在于农业生态监测领域的技术框架，不适用于文档图像理解场景。
Reason 该论文属于遥感图像分析领域，研究任务为农田生态系统生境分类与分割，使用的技术路线基于DINOv3特征提取和多尺度特征融合，尽管涉及高分辨率图像处理，但其核心目标与文档图像理解（DIU）无关。未涉及OCR、文本布局分析、表格识别、VIE等DIU核心任务，也无法迁移解决OCR幻觉、细粒度文本定位或复杂排版建模等关键痛点。
Abstract Addressing the current lack of a standardized habitat classification system for cultivated land ecosystems, incomplete coverage of the habitat types, and the inability of existing models to effectively integrate semantic and texture features-resulting in insufficient segmentation accuracy and blurred boundaries for multi-scale habitats (e.g., large-scale field plots and micro-habitats)-this study developed a comprehensively annotated ultra-high-resolution remote sensing image dataset encompassing 15 categories of cultivated land system habitats. Furthermore, we propose a Dynamic-Weighted Feature Fusion Network (DWFF-Net). The encoder of this model utilizes a frozen-parameter DINOv3 to extract foundational features. By analyzing the relationships between different category images and feature maps, we introduce a data-level adaptive dynamic weighting strategy for feature fusion. The decoder incorporates a dynamic weight computation network to achieve thorough integration of multi-layer features, and a hybrid loss function is adopted to optimize model training. Experimental results on the constructed dataset demonstrate that the proposed model achieves a mean Intersection over Union (mIoU) of 69.79% and an F1-score of 80.49%, outperforming the baseline network by 2.1% and 1.61%, respectively. Ablation studies further confirm the complementary nature of multi-layer feature fusion, which effectively improves the IoU for micro-habitat categories such as field ridges. This study establishes a habitat identification framework for cultivated land systems based on adaptive multi-layer feature fusion, enabling sub-meter precision habitat mapping at a low cost and providing robust technical support for fine-grained habitat monitoring in cultivated landscapes. (The complete code repository can be accessed via GitHub at the following URL: https://github.com/sysau/DWFF-Net)
Authors Kesong Zheng, Zhi Song, Peizhou Li, Shuyi Yao, Zhenxing Bian
Comment 30 pages,13 figures
Categories Computer Vision and Pattern Recognition
0
AVFakeBench: A Comprehensive Audio-Video Forgery Detection Benchmark for AV-LMMs
Title CN AVFakeBench：面向音视频大模型的综合性音视频伪造检测基准
Keywords 音视频伪造检测 · 多模态基准 · 深度伪造 · AV-LMM · 细粒度推理
Summary 本文提出了AVFakeBench，首个涵盖人类主体与通用场景的综合性音视频伪造检测基准，包含12K个音视频问题，覆盖七类伪造类型和四个层级的标注。通过一个多阶段混合伪造框架生成高质量伪造样本，并设计了包含二分类判断、伪造类型分类、细节选择和解释性推理的多任务评估体系。实验评估了11种音视频大模型和两种检测方法，揭示了当前模型在细粒度感知与推理上的不足。
Reason 该论文聚焦于音视频伪造检测，属于视频理解与安全领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其多模态评估框架具有一定方法论价值，但伪造检测的语义目标和应用场景与文档理解中的OCR幻觉、密集文本处理、复杂排版解析等痛点差异显著，难以迁移应用。
Abstract The threat of Audio-Video (AV) forgery is rapidly evolving beyond human-centric deepfakes to include more diverse manipulations across complex natural scenes. However, existing benchmarks are still confined to DeepFake-based forgeries and single-granularity annotations, thus failing to capture the diversity and complexity of real-world forgery scenarios. To address this, we introduce AVFakeBench, the first comprehensive audio-video forgery detection benchmark that spans rich forgery semantics across both human subject and general subject. AVFakeBench comprises 12K carefully curated audio-video questions, covering seven forgery types and four levels of annotations. To ensure high-quality and diverse forgeries, we propose a multi-stage hybrid forgery framework that integrates proprietary models for task planning with expert generative models for precise manipulation. The benchmark establishes a multi-task evaluation framework covering binary judgment, forgery types classification, forgery detail selection, and explanatory reasoning. We evaluate 11 Audio-Video Large Language Models (AV-LMMs) and 2 prevalent detection methods on AVFakeBench, demonstrating the potential of AV-LMMs as emerging forgery detectors while revealing their notable weaknesses in fine-grained perception and reasoning.
Authors Shuhan Xia, Peipei Li, Xuannan Liu, Dongsen Zhang, Xinyu Guo, Zekun Li
Categories Computer Vision and Pattern Recognition
0
Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy Transaction on Pattern Analysis and Machine Intelligence (T-PAMI)
Title CN Gen-3Diffusion：通过2D与3D扩散协同实现逼真的图像到3D生成
Keywords 3D生成 · 扩散模型 · 多视图一致性 · 图像到3D · 2D-3D协同
Summary 本文提出Gen-3Diffusion，通过联合训练和采样阶段同步2D扩散模型与3D扩散模型，实现高质量的图像到3D对象生成。2D模型提供强泛化能力，3D模型保证几何一致性，从而生成高保真纹理与形状的3D对象和穿衣人像。
Reason 该论文研究的是从单张图像生成3D物体和穿衣人像，属于3D视觉与生成模型领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心技术围绕2D与3D扩散模型的协同，旨在解决多视角一致性问题，不涉及OCR、布局分析、表格识别或文档推理等DIU核心任务。尽管使用了多模态建模范式，但无法迁移用于解决文档中的幻觉、密集文本处理或复杂排版理解等痛点。
Abstract Creating realistic 3D objects and clothed avatars from a single RGB image is an attractive yet challenging problem. Due to its ill-posed nature, recent works leverage powerful prior from 2D diffusion models pretrained on large datasets. Although 2D diffusion models demonstrate strong generalization capability, they cannot guarantee the generated multi-view images are 3D consistent. In this paper, we propose Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy. We leverage a pre-trained 2D diffusion model and a 3D diffusion model via our elegantly designed process that synchronizes two diffusion models at both training and sampling time. The synergy between the 2D and 3D diffusion models brings two major advantages: 1) 2D helps 3D in generalization: the pretrained 2D model has strong generalization ability to unseen images, providing strong shape priors for the 3D diffusion model; 2) 3D helps 2D in multi-view consistency: the 3D diffusion model enhances the 3D consistency of 2D multi-view sampling process, resulting in more accurate multi-view generation. We validate our idea through extensive experiments in image-based objects and clothed avatar generation tasks. Results show that our method generates realistic 3D objects and avatars with high-fidelity geometry and texture. Extensive ablations also validate our design choices and demonstrate the strong generalization ability to diverse clothing and compositional shapes. Our code and pretrained models will be publicly released on https://yuxuan-xue.com/gen-3diffusion.
Authors Yuxuan Xue, Xianghui Xie, Riccardo Marin, Gerard Pons-Moll
Comment Accepted to Transaction on Pattern Analysis and Machine Intelligence (T-PAMI). Project Page: https://yuxuan-xue.com/gen-3diffusion. arXiv admin note: substantial text overlap with arXiv:2406.08475
Categories Computer Vision and Pattern Recognition
0
FIELDS: Face reconstruction with accurate Inference of Expression using Learning with Direct Supervision
Title CN 使用直接监督学习实现准确表情推断的面部重建
Keywords 3D人脸重建 · 表情建模 · 直接监督 · 情感识别 · 4D扫描
Summary 本文提出FIELDS方法，通过引入直接的3D表情参数监督和辅助情绪识别分支，提升单张图像中3D人脸表情重建的真实性与情感保真度。该方法利用自发性4D面部扫描数据提供真实表达参数，并结合强度感知的情绪损失来避免夸张表达，从而在野外场景下显著提升面部表情识别性能。
Reason 该论文研究的是3D人脸重建与表情建模，属于面部几何与情感建模领域，尽管涉及视觉表示学习，但其方法针对的是人脸表情的精细重建，并不解决文档图像理解中的核心问题（如OCR幻觉、复杂排版、密集文本处理等）。无法迁移用于文档图像理解任务。
Abstract Facial expressions convey the bulk of emotional information in human communication, yet existing 3D face reconstruction methods often miss subtle affective details due to reliance on 2D supervision and lack of 3D ground truth. We propose FIELDS (Face reconstruction with accurate Inference of Expression using Learning with Direct Supervision) to address these limitations by extending self-supervised 2D image consistency cues with direct 3D expression parameter supervision and an auxiliary emotion recognition branch. Our encoder is guided by authentic expression parameters from spontaneous 4D facial scans, while an intensity-aware emotion loss encourages the 3D expression parameters to capture genuine emotion content without exaggeration. This dual-supervision strategy bridges the 2D/3D domain gap and mitigates expression-intensity bias, yielding high-fidelity 3D reconstructions that preserve subtle emotional cues. From a single image, FIELDS produces emotion-rich face models with highly realistic expressions, significantly improving in-the-wild facial expression recognition performance without sacrificing naturalness.
Authors Chen Ling, Henglin Shi, Hedvig Kjellström
Categories Computer Vision and Pattern Recognition
0
ControlEvents: Controllable Synthesis of Event Camera Datawith Foundational Prior from Image Diffusion Models WACV2026
Title CN ControlEvents：利用图像扩散模型的基础先验实现事件相机数据的可控合成
Keywords 事件相机 · 扩散模型 · 数据合成 · 可控生成 · 基础模型先验
Summary 本文提出ControlEvents，一种基于扩散模型的事件相机数据生成方法，能够通过文本标签、2D骨架或3D姿态等控制信号合成高质量的事件流数据。该方法利用Stable Diffusion等图像扩散模型的基础先验，仅需少量微调即可生成带标注的事件数据，显著降低真实事件数据采集与标注成本。实验表明，所生成的数据可用于提升事件相机下的视觉识别与姿态估计性能。
Reason 该论文研究的是事件相机（event camera）数据的生成，属于特定成像模态的数据合成问题，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心方法是基于扩散模型进行事件序列生成，应用于视觉识别、姿态估计等任务，但并未涉及OCR、布局分析、表格识别或任何文档结构理解内容。尽管使用了多模态基础模型的先验，但目标领域和应用场景完全偏离文档理解范畴。
Abstract In recent years, event cameras have gained significant attention due to their bio-inspired properties, such as high temporal resolution and high dynamic range. However, obtaining large-scale labeled ground-truth data for event-based vision tasks remains challenging and costly. In this paper, we present ControlEvents, a diffusion-based generative model designed to synthesize high-quality event data guided by diverse control signals such as class text labels, 2D skeletons, and 3D body poses. Our key insight is to leverage the diffusion prior from foundation models, such as Stable Diffusion, enabling high-quality event data generation with minimal fine-tuning and limited labeled data. Our method streamlines the data generation process and significantly reduces the cost of producing labeled event datasets. We demonstrate the effectiveness of our approach by synthesizing event data for visual recognition, 2D skeleton estimation, and 3D body pose estimation. Our experiments show that the synthesized labeled event data enhances model performance in all tasks. Additionally, our approach can generate events based on unseen text labels during training, illustrating the powerful text-based generation capabilities inherited from foundation models.
Authors Yixuan Hu, Yuxuan Xue, Simon Klenk, Daniel Cremers, Gerard Pons-Moll
Comment Accepted to WACV2026. Project website: https://https://yuxuan-xue.com/controlevents/
Categories Computer Vision and Pattern Recognition
0
3-Tracer: A Tri-level Temporal-Aware Framework for Audio Forgery Detection and Localization
Title CN 3-Tracer：一种用于音频伪造检测与定位的三级时间感知框架
Keywords 音频伪造检测 · 时间感知 · 帧级分析 · 片段级分析 · 多尺度建模
Summary 本文提出了一种名为T3-Tracer的三级时间感知框架，用于部分音频伪造的检测与定位。该方法在帧级、片段级和音频级联合建模，通过特征聚合模块和多尺度差异感知模块捕捉局部篡改痕迹，在多个数据集上实现了最先进的性能。
Reason 该论文研究的是音频伪造检测与定位问题，属于音频信号处理领域，与文档图像理解（DIU）或视觉-语言模型（VLM）在文档上的应用无直接关联。其提出的方法针对音频帧、片段和整体音频的三级分析架构难以迁移到文档图像理解中的核心痛点（如OCR幻觉、复杂版面解析、长文档推理等）。未涉及视觉内容建模、多模态语言模型或文档结构分析，完全偏离DIU本题。
Abstract Recently, partial audio forgery has emerged as a new form of audio manipulation. Attackers selectively modify partial but semantically critical frames while preserving the overall perceptual authenticity, making such forgeries particularly difficult to detect. Existing methods focus on independently detecting whether a single frame is forged, lacking the hierarchical structure to capture both transient and sustained anomalies across different temporal levels. To address these limitations, We identify three key levels relevant to partial audio forgery detection and present T3-Tracer, the first framework that jointly analyzes audio at the frame, segment, and audio levels to comprehensively detect forgery traces. T3-Tracer consists of two complementary core modules: the Frame-Audio Feature Aggregation Module (FA-FAM) and the Segment-level Multi-Scale Discrepancy-Aware Module (SMDAM). FA-FAM is designed to detect the authenticity of each audio frame. It combines both frame-level and audio-level temporal information to detect intra-frame forgery cues and global semantic inconsistencies. To further refine and correct frame detection, we introduce SMDAM to detect forgery boundaries at the segment level. It adopts a dual-branch architecture that jointly models frame features and inter-frame differences across multi-scale temporal windows, effectively identifying abrupt anomalies that appeared on the forged boundaries. Extensive experiments conducted on three challenging datasets demonstrate that our approach achieves state-of-the-art performance.
Authors Shuhan Xia, Xuannan Liu, Xing Cui, Peipei Li
Categories Computer Vision and Pattern Recognition
0
From Diffusion to One-Step Generation: A Comparative Study of Flow-Based Models with Application to Image Inpainting
Title CN 从扩散到一步生成：基于流的模型在图像修复中的对比研究
Keywords 图像生成 · 流匹配 · 图像修复 · 扩散模型 · 生成模型
Summary 本文比较了三种生成模型范式——去噪扩散概率模型（DDPM）、条件流匹配（CFM）和MeanFlow，在CIFAR-10数据集上使用TinyUNet架构进行实验。结果显示CFM在50步采样下FID为24.15，显著优于DDPM；MeanFlow支持单步生成，FID为29.15，推理时间减少50倍。研究还将CFM应用于图像修复，通过掩码引导采样并在特定掩码类型上实现PSNR和SSIM的显著提升。
Reason 该论文聚焦于图像生成与修复任务，属于纯图像生成领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其方法涉及生成模型效率优化，但无法迁移解决OCR幻觉、复杂排版理解或长文档推理等DIU关键痛点。
Abstract We present a comprehensive comparative study of three generative modeling paradigms: Denoising Diffusion Probabilistic Models (DDPM), Conditional Flow Matching (CFM), and MeanFlow. While DDPM and CFM require iterative sampling, MeanFlow enables direct one-step generation by modeling the average velocity over time intervals. We implement all three methods using a unified TinyUNet architecture (<1.5M parameters) on CIFAR-10, demonstrating that CFM achieves an FID of 24.15 with 50 steps, significantly outperforming DDPM (FID 402.98). MeanFlow achieves FID 29.15 with single-step sampling -- a 50X reduction in inference time. We further extend CFM to image inpainting, implementing mask-guided sampling with four mask types (center, random bbox, irregular, half). Our fine-tuned inpainting model achieves substantial improvements: PSNR increases from 4.95 to 8.57 dB on center masks (+73%), and SSIM improves from 0.289 to 0.418 (+45%), demonstrating the effectiveness of inpainting-aware training.
Authors Umang Agarwal, Rudraksh Sangore, Sumit Laddha
Categories Computer Vision and Pattern Recognition
0
Towards an Effective Action-Region Tracking Framework for Fine-grained Video Action Recognition
Title CN 面向细粒度视频动作识别的有效动作区域跟踪框架
Keywords 细粒度动作识别 · 动作区域跟踪 · 视觉语言模型 · 视频理解 · 对比学习
Summary 本文提出了一种名为Action-Region Tracking（ART）的框架，通过查询-响应机制发现并跟踪视频中细粒度动作的关键区域，利用文本约束的语义查询激活相关区域响应，并构建动作轨迹片段（action tracklets）来建模局部动作动态。同时设计了多层次对比约束和任务特定微调机制以优化轨迹质量。实验表明该方法在多个动作识别基准上优于现有方法。
Reason 该论文研究的是细粒度视频动作识别（FGAR），属于视频理解领域，核心任务是跟踪视频中局部区域的动作动态。尽管使用了VLM提取文本语义作为查询，但其方法设计完全针对视频时空建模，与文档图像理解（DIU）中的OCR幻觉、密集文本识别、复杂版式解析等核心痛点无关。无法迁移用于解决文档理解中的视觉压缩、细粒度定位或长文档推理等问题。
Abstract Fine-grained action recognition (FGAR) aims to identify subtle and distinctive differences among fine-grained action categories. However, current recognition methods often capture coarse-grained motion patterns but struggle to identify subtle details in local regions evolving over time. In this work, we introduce the Action-Region Tracking (ART) framework, a novel solution leveraging a query-response mechanism to discover and track the dynamics of distinctive local details, enabling effective distinction of similar actions. Specifically, we propose a region-specific semantic activation module that employs discriminative and text-constrained semantics as queries to capture the most action-related region responses in each video frame, facilitating interaction among spatial and temporal dimensions with corresponding video features. The captured region responses are organized into action tracklets, which characterize region-based action dynamics by linking related responses across video frames in a coherent sequence. The text-constrained queries encode nuanced semantic representations derived from textual descriptions of action labels extracted by language branches within Visual Language Models (VLMs). To optimize the action tracklets, we design a multi-level tracklet contrastive constraint among region responses at spatial and temporal levels, enabling effective discrimination within each frame and correlation between adjacent frames. Additionally, a task-specific fine-tuning mechanism refines textual semantics such that semantic representations encoded by VLMs are preserved while optimized for task preferences. Comprehensive experiments on widely used action recognition benchmarks demonstrate the superiority to previous state-of-the-art baselines.
Authors Baoli Sun, Yihan Wang, Xinzhu Ma, Zhihui Wang, Kun Lu, Zhiyong Wang
Categories Computer Vision and Pattern Recognition
0
MetricHMSR:Metric Human Mesh and Scene Recovery from Monocular Images
Title CN MetricHMSR：从单目图像中进行人体网格与场景的度量恢复
Keywords 3D human mesh · scene recovery · monocular images · camera rays · Mixture-of-Experts
Summary 本文提出MetricHMSR，通过引入相机射线信息并设计Human Mixture-of-Experts结构，实现从单目图像中同时恢复人体网格与三维场景布局。方法在统一框架下处理局部姿态与全局位置估计，并改进了单目深度估计效果，在相关数据集上达到SOTA性能。
Reason 该论文研究的是单目图像中的人体三维网格与场景的联合重建，属于3D视觉与人体姿态估计领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务完全无关。其技术路线聚焦于相机射线建模和MoE结构用于3D感知，无法迁移到解决OCR幻觉、密集文本识别、复杂版式分析或长文档推理等DIU核心问题。
Abstract We introduce MetricHMSR (Metric Human Mesh and Scene Recovery), a novel approach for metric human mesh and scene recovery from monocular images. Due to unrealistic assumptions in the camera model and inherent challenges in metric perception, existing approaches struggle to achieve human pose and metric 3D position estimation through a unified module. To address this limitation, MetricHMSR incorporates camera rays to comprehensively encode both the bounding box information and the intrinsic parameters of perspective projection. Then we proposed Human Mixture-of-Experts (MoE), the model dynamically routes image features and ray features to task-specific experts for specialized understanding of different data aspects, enabling a unified framework that simultaneously perceives the local pose and the global 3D position. Based on the results above, we further refine the existing monocular metric depth estimation method to achieve more accurate results, ultimately enabling the seamless overlay of humans and scenes in 3D space. Comprehensive experimental results demonstrate that the proposed method achieves state-of-the-art performance on both human mesh and scene recovery.
Authors Chentao Song, He Zhang, Haolei Yuan, Haozhe Lin, Jianhua Tao, Hongwen Zhang, Tao Yu
Categories Computer Vision and Pattern Recognition
0
Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understanding
Title CN 将场景作为令牌：用于通用3D视觉-语言理解的多尺度正态分布变换令牌化器
Keywords 3D视觉-语言模型 · 场景令牌化 · 多尺度NDT · 点云处理 · 3D VQA
Summary 本文提出NDTokenizer3D，一种基于多尺度正态分布变换（NDT）的3D视觉-语言模型，通过构建从点云到场景令牌的三阶段管道，实现对3D场景的细粒度建模，并支持指代分割、视觉问答和密集描述等多种任务。该方法强调在3D空间中进行语言级推理，但不涉及任何文档结构或文本理解机制。
Reason 该论文研究的是3D视觉-语言理解，属于3D视觉领域，其核心方法NDTokenizer3D基于点云和多尺度NDT表示，专注于3D场景的令牌化与理解。尽管提出了多模态令牌化和统一架构的思想，但其处理的数据模态（3D点云）与文档图像理解（2D平面、文本布局、OCR等）差异巨大，无法直接迁移解决DIU中的OCR幻觉、密集文本识别或复杂排版解析等问题。
Abstract Recent advances in 3D vision-language models (VLMs) highlight a strong potential for 3D scene understanding and reasoning. However, effectively tokenizing 3D scenes into holistic scene tokens, and leveraging these tokens across diverse 3D understanding tasks, remain highly challenging. We present NDTokenizer3D, a generalist 3D VLM that performs a wide range of 3D scene understanding tasks while naturally supporting human interactions, thereby bridging language-level reasoning with 3D spatial understanding. The core of our approach is a novel three-stage scene tokenization pipeline built upon a Multi-Scale Normal Distributions Transform (NDT) representation, paired with a Multi-Scale NDT Decoder (MSDec). Specifically, NDTokenizer3D first constructs a multi-scale NDT representation from raw high-resolution point clouds, preserving both global context and fine-grained geometric details. Next, the MSDec progressively fuses cross-scale NDT features, producing holistic scene tokens consumable by LLM endpoints. Beyond tokenization, MSDec is repurposed as a general interface for human-interactive prompting (points, boxes, masks) and segmentation-mask decoding, unifying diverse 3D scene understanding tasks within a single architecture. With this compact and unified design, NDTokenizer3D offers a fine-grained, general-purpose 3D VLM, achieving remarkable improvements in 3D Referring Segmentation, 3D Visual Question Answering, and 3D Dense Captioning.
Authors Yutao Tang, Cheng Zhao, Gaurav Mittal, Rohith Kukkala, Rama Chellappa, Cheng Peng, Mei Chen
Categories Computer Vision and Pattern Recognition
0
Restora-Flow: Mask-Guided Image Restoration with Flow Matching WACV 2026
Title CN Restora-Flow：基于流匹配的掩码引导图像修复
Keywords 图像修复 · 流匹配 · 生成模型 · 掩码引导 · 轨迹校正
Summary 本文提出Restora-Flow，一种基于流匹配的训练免费图像修复方法，通过退化掩码引导采样过程，并引入轨迹校正机制以保持与退化输入的一致性。在自然图像和医学图像数据集上验证了其在感知质量和处理速度上的优势。适用于掩码驱动的图像修复任务，如图像补全、超分辨率和去噪。
Reason 该论文属于纯图像生成与修复领域，研究的是通用图像和医学图像的修复任务（如去噪、超分辨率、补全），虽然使用了流匹配这一前沿生成方法，但其技术路线与文档图像理解中的核心问题（如OCR幻觉、文本结构解析、多模态推理）无直接关联。未涉及文本内容建模、布局分析或VLM集成，无法迁移用于解决文档理解中的细粒度识别或逻辑推理等痛点。
Abstract Flow matching has emerged as a promising generative approach that addresses the lengthy sampling times associated with state-of-the-art diffusion models and enables a more flexible trajectory design, while maintaining high-quality image generation. This capability makes it suitable as a generative prior for image restoration tasks. Although current methods leveraging flow models have shown promising results in restoration, some still suffer from long processing times or produce over-smoothed results. To address these challenges, we introduce Restora-Flow, a training-free method that guides flow matching sampling by a degradation mask and incorporates a trajectory correction mechanism to enforce consistency with degraded inputs. We evaluate our approach on both natural and medical datasets across several image restoration tasks involving a mask-based degradation, i.e., inpainting, super-resolution and denoising. We show superior perceptual quality and processing time compared to diffusion and flow matching-based reference methods.
Authors Arnela Hadzic, Franz Thaler, Lea Bogensperger, Simon Johannes Joham, Martin Urschler
Comment Accepted for WACV 2026
Categories Computer Vision and Pattern Recognition
0
DiffSeg30k: A Multi-Turn Diffusion Editing Benchmark for Localized AIGC Detection
Title CN DiffSeg30k：一种用于局部AIGC检测的多轮扩散编辑基准
Keywords AIGC检测 · 语义分割 · 扩散模型编辑 · 图像取证 · 像素级标注
Summary 本文提出了DiffSeg30k，一个包含3万张经过扩散模型局部编辑并带有像素级标注的图像数据集，旨在推动细粒度AIGC检测研究。该数据集涵盖真实场景图像、多种SOTA扩散模型、多轮编辑过程以及基于VLM生成的上下文感知编辑提示，将AIGC检测从整图分类转变为语义分割任务。实验表明，分割模型不仅能精确定位编辑区域，还能作为强大的全图分类器，在跨生成器泛化上表现优异。
Reason 该论文聚焦于AI生成内容（AIGC）的伪造检测与定位，属于图像取证领域，核心任务是识别扩散模型编辑过的区域并定位篡改位置。尽管使用了VLM构建数据集，但其目标与文档图像理解（DIU）无关，不涉及OCR、布局分析、表格识别或文档问答等任务。无法迁移用于解决文档理解中的OCR幻觉、复杂排版或长文档推理等痛点，属于无关领域。
Abstract Diffusion-based editing enables realistic modification of local image regions, making AI-generated content harder to detect. Existing AIGC detection benchmarks focus on classifying entire images, overlooking the localization of diffusion-based edits. We introduce DiffSeg30k, a publicly available dataset of 30k diffusion-edited images with pixel-level annotations, designed to support fine-grained detection. DiffSeg30k features: 1) In-the-wild images--we collect images or image prompts from COCO to reflect real-world content diversity; 2) Diverse diffusion models--local edits using eight SOTA diffusion models; 3) Multi-turn editing--each image undergoes up to three sequential edits to mimic real-world sequential editing; and 4) Realistic editing scenarios--a vision-language model (VLM)-based pipeline automatically identifies meaningful regions and generates context-aware prompts covering additions, removals, and attribute changes. DiffSeg30k shifts AIGC detection from binary classification to semantic segmentation, enabling simultaneous localization of edits and identification of the editing models. We benchmark three baseline segmentation approaches, revealing significant challenges in semantic segmentation tasks, particularly concerning robustness to image distortions. Experiments also reveal that segmentation models, despite being trained for pixel-level localization, emerge as highly reliable whole-image classifiers of diffusion edits, outperforming established forgery classifiers while showing great potential in cross-generator generalization. We believe DiffSeg30k will advance research in fine-grained localization of AI-generated content by demonstrating the promise and limitations of segmentation-based methods. DiffSeg30k is released at: https://huggingface.co/datasets/Chaos2629/Diffseg30k
Authors Hai Ci, Ziheng Peng, Pei Yang, Yingxin Xuan, Mike Zheng Shou
Comment 16 pages, 10 figures; typos corrected, references added
Categories Computer Vision and Pattern Recognition
0
Systematic Evaluation and Guidelines for Segment Anything Model in Surgical Video Analysis
Title CN SAM模型在手术视频分析中的系统性评估与指南
Keywords SAM2 · 手术视频分割 · 零样本学习 · 医学图像分析 · 视觉提示
Summary 本文首次系统评估了SAM2模型在9个手术视频数据集上的零样本分割能力，涵盖多种手术类型和视觉提示方式。研究发现SAM2在结构化场景（如器械分割）中表现良好，但在处理组织形变、器械多样性及时间连贯性方面存在局限，揭示了未来改进方向。
Reason 该论文研究的是手术视频分割，属于视频理解领域，且聚焦于医疗手术场景，与文档图像理解（DIU）任务无直接关联。虽然涉及视觉模型的零样本迁移和提示策略，但其方法针对的是动态视频中的时空一致性问题，难以迁移到文档图像的细粒度文本识别、排版理解或VQA等核心痛点。属于无关领域应用。
Abstract Surgical video segmentation is critical for AI to interpret spatial-temporal dynamics in surgery, yet model performance is constrained by limited annotated data. The SAM2 model, pretrained on natural videos, offers potential for zero-shot surgical segmentation, but its applicability in complex surgical environments, with challenges like tissue deformation and instrument variability, remains unexplored. We present the first comprehensive evaluation of the zero-shot capability of SAM2 in 9 surgical datasets (17 surgery types), covering laparoscopic, endoscopic, and robotic procedures. We analyze various prompting (points, boxes, mask) and {finetuning (dense, sparse) strategies}, robustness to surgical challenges, and generalization across procedures and anatomies. Key findings reveal that while SAM2 demonstrates notable zero-shot adaptability in structured scenarios (e.g., instrument segmentation, {multi-organ segmentation}, and scene segmentation), its performance varies under dynamic surgical conditions, highlighting gaps in handling temporal coherence and domain-specific artifacts. These results highlight future pathways to adaptive data-efficient solutions for the surgical data science field.
Authors Cheng Yuan, Jian Jiang, Kunyi Yang, Lv Wu, Rui Wang, Zi Meng, Haonan Ping, Ziyu Xu, Yifan Zhou, Wanli Song, Hesheng Wang, Yueming Jin, Qi Dou, Yutong Ban
Categories Computer Vision and Pattern Recognition
0
AV-Edit: Multimodal Generative Sound Effect Editing via Audio-Visual Semantic Joint Control
Title CN AV-Edit：通过视听语义联合控制的多模态生成式音效编辑
Keywords 音效编辑 · 多模态生成 · 视听联合控制 · 扩散模型 · 跨模态表示
Summary 本文提出AV-Edit，一种结合视觉、音频和文本语义的生成式音效编辑框架，通过对比式视听掩码自编码器（CAV-MAE-Edit）和多模态扩散Transformer（MM-DiT）实现基于视频内容的精细音效修改。构建了专用数据集并验证了方法在音效编辑和音频生成上的优越性。
Reason 该论文研究的是视频中的音效编辑问题，属于音频-视觉生成领域，核心任务是基于视觉内容生成或修改声音效果。其技术路线聚焦于多模态扩散模型在音效合成中的应用，与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版面分析、长文档推理等完全无关。无法迁移用于解决文档理解中的任何关键问题。
Abstract Sound effect editing-modifying audio by adding, removing, or replacing elements-remains constrained by existing approaches that rely solely on low-level signal processing or coarse text prompts, often resulting in limited flexibility and suboptimal audio quality. To address this, we propose AV-Edit, a generative sound effect editing framework that enables fine-grained editing of existing audio tracks in videos by jointly leveraging visual, audio, and text semantics. Specifically, the proposed method employs a specially designed contrastive audio-visual masking autoencoder (CAV-MAE-Edit) for multimodal pre-training, learning aligned cross-modal representations. These representations are then used to train an editorial Multimodal Diffusion Transformer (MM-DiT) capable of removing visually irrelevant sounds and generating missing audio elements consistent with video content through a correlation-based feature gating training strategy. Furthermore, we construct a dedicated video-based sound editing dataset as an evaluation benchmark. Experiments demonstrate that the proposed AV-Edit generates high-quality audio with precise modifications based on visual content, achieving state-of-the-art performance in the field of sound effect editing and exhibiting strong competitiveness in the domain of audio generation.
Authors Xinyue Guo, Xiaoran Yang, Lipan Zhang, Jianxuan Yang, Zhao Wang, Jian Luan
Categories Computer Vision and Pattern Recognition
0
TEAR: Temporal-aware Automated Red-teaming for Text-to-Video Models
Title CN TEAR：面向文本到视频模型的时序感知自动化红队测试
Keywords 文本到视频生成 · 安全性评估 · 红队测试 · 时序建模 · 对抗攻击
Summary 本文提出TEAR框架，用于检测文本到视频生成模型在时间动态上的安全漏洞。通过时序感知的提示生成和在线偏好学习，自动构造表面无害但能诱发违规视频输出的文本指令，并采用精炼模型提升攻击隐蔽性和有效性。实验显示其在多个T2V系统中攻击成功率超过80%。
Reason 该论文研究的是文本到视频生成模型的安全性评估，属于视频生成与安全领域，核心任务是通过时序感知的红队攻击揭示T2V模型的政策违规风险。其方法聚焦于动态视频内容中的时间序列漏洞挖掘，与文档图像理解（DIU）的核心问题如OCR、版面分析、表格识别、视觉语言推理等无直接关联。尽管涉及多模态输入和生成，但目标场景、技术挑战和应用场景均远离文档理解范畴，无法迁移用于解决OCR幻觉、复杂排版或长文档推理等问题。
Abstract Text-to-Video (T2V) models are capable of synthesizing high-quality, temporally coherent dynamic video content, but the diverse generation also inherently introduces critical safety challenges. Existing safety evaluation methods,which focus on static image and text generation, are insufficient to capture the complex temporal dynamics in video generation. To address this, we propose a TEmporal-aware Automated Red-teaming framework, named TEAR, an automated framework designed to uncover safety risks specifically linked to the dynamic temporal sequencing of T2V models. TEAR employs a temporal-aware test generator optimized via a two-stage approach: initial generator training and temporal-aware online preference learning, to craft textually innocuous prompts that exploit temporal dynamics to elicit policy-violating video output. And a refine model is adopted to improve the prompt stealthiness and adversarial effectiveness cyclically. Extensive experimental evaluation demonstrates the effectiveness of TEAR across open-source and commercial T2V systems with over 80% attack success rate, a significant boost from prior best result of 57%.
Authors Jiaming He, Guanyu Hou, Hongwei Li, Zhicong Huang, Kangjie Chen, Yi Yu, Wenbo Jiang, Guowen Xu, Tianwei Zhang
Categories Computer Vision and Pattern Recognition
0
XYZCylinder: Towards Compatible Feed-Forward 3D Gaussian Splatting for Driving Scenes via Unified Cylinder Lifting Method
Title CN XYZCylinder：通过统一圆柱提升方法实现驾驶场景下兼容的前馈式3D高斯点阵化
Keywords 3D重建 · 前馈网络 · 3D高斯点阵化 · 自动驾驶 · 圆柱提升
Summary 本文提出XYZCylinder，一种基于统一圆柱提升方法的前馈式3D高斯点阵化技术，用于提升复杂驾驶场景中的3D重建效果。通过统一圆柱相机建模（UCCM）策略和圆柱平面特征组（CPFG）模块，解决了多相机配置兼容性和稀疏视角下重建精度低的问题。实验表明其在零样本新场景中具有良好的泛化能力。
Reason 该论文属于3D视觉与自动驾驶领域，研究的是驾驶场景下的3D重建问题，使用的技术如3D Gaussian Splatting和圆柱空间特征提升完全针对三维几何建模，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等核心DIU痛点。
Abstract Feed-forward paradigms for 3D reconstruction have become a focus of recent research, which learn implicit, fixed view transformations to generate a single scene representation. However, their application to complex driving scenes reveals significant limitations. Two core challenges are responsible for this performance gap. First, the reliance on a fixed view transformation hinders compatibility to varying camera configurations. Second, the inherent difficulty of learning complex driving scenes from sparse 360° views with minimal overlap compromises the final reconstruction fidelity. To handle these difficulties, we introduce XYZCylinder, a novel method built upon a unified cylinder lifting method that integrates camera modeling and feature lifting. To tackle the compatibility problem, we design a Unified Cylinder Camera Modeling (UCCM) strategy. This strategy explicitly models projection parameters to unify diverse camera setups, thus bypassing the need for learning viewpoint-dependent correspondences. To improve the reconstruction accuracy, we propose a hybrid representation with several dedicated modules based on newly designed Cylinder Plane Feature Group (CPFG) to lift 2D image features to 3D space. Extensive evaluations confirm that XYZCylinder not only achieves state-of-the-art performance under different evaluation settings but also demonstrates remarkable compatibility in entirely new scenes with different camera settings in a zero-shot manner. Project page: \href{https://yuyuyu223.github.io/XYZCYlinder-projectpage/}{here}
Authors Haochen Yu, Qiankun Liu, Hongyuan Liu, Jianfei Jiang, Juntao Lyu, Jiansheng Chen, Huimin Ma
Comment Feed-Forward, 3D Gaussian Splatting, Project page: https://yuyuyu223.github.io/XYZCYlinder-projectpage/
Categories Computer Vision and Pattern Recognition
0
STAR: Smartphone-analogous Typing in Augmented Reality ACM UIST 2023
Title CN STAR：增强现实中的智能手机类比输入法
Keywords 增强现实 · 文本输入 · 虚拟键盘 · 人机交互 · 手势输入
Summary 本文提出STAR，一种在增强现实中模仿智能手机双手拇指输入的文本输入方法，用户在自己手部皮肤上敲击虚拟QWERTY键盘进行输入。实验显示经过30分钟练习后，平均输入速度达到21.9 WPM，错误率仅0.3%。研究分析了与手机输入性能差距的原因并提出了优化方向。
Reason 该论文研究的是增强现实中基于手部皮肤的虚拟键盘文本输入技术，属于人机交互（HCI）领域，核心问题是AR环境下的文本输入效率与可用性。其任务与文档图像理解（DIU）、DocVQA、OCR、多模态文档推理等完全无关，也不提供可用于解决OCR幻觉、复杂排版理解或长文档推理的底层方法。属于无关领域。
Abstract While text entry is an essential and frequent task in Augmented Reality (AR) applications, devising an efficient and easy-to-use text entry method for AR remains an open challenge. This research presents STAR, a smartphone-analogous AR text entry technique that leverages a user's familiarity with smartphone two-thumb typing. With STAR, a user performs thumb typing on a virtual QWERTY keyboard that is overlain on the skin of their hands. During an evaluation study of STAR, participants achieved a mean typing speed of 21.9 WPM (i.e., 56% of their smartphone typing speed), and a mean error rate of 0.3% after 30 minutes of practice. We further analyze the major factors implicated in the performance gap between STAR and smartphone typing, and discuss ways this gap could be narrowed.
Authors Taejun Kim, Amy Karlson, Aakar Gupta, Tovi Grossman, Jason Wu, Parastoo Abtahi, Christopher Collins, Michael Glueck, Hemant Bhaskar Surale
Comment ACM UIST 2023
Categories Computer Vision and Pattern Recognition
0
Uncovering and Mitigating Destructive Multi-Embedding Attacks in Deepfake Proactive Forensics
Title CN 揭示并缓解深度伪造主动取证中的破坏性多嵌入攻击
Keywords 深度伪造 · 主动取证 · 水印鲁棒性 · 多嵌入攻击 · AIS
Summary 本文首次定义了多嵌入攻击（MEA），即在已嵌入水印的图像上再次嵌入会破坏原始水印的问题，并提出一种名为对抗干扰模拟（AIS）的训练范式来增强现有方法对MEA的鲁棒性。AIS通过在微调阶段显式模拟MEA场景并引入抗毁性损失函数，使模型能在经历二次嵌入后仍正确提取原始水印。
Reason 该论文研究的是深度伪造（deepfake）的主动取证与水印嵌入问题，属于数字版权与安全领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其提出的方法AIS针对的是水印在多次嵌入下的鲁棒性问题，并不涉及OCR、布局分析、表格识别或多模态文档推理等DIU核心任务。无法迁移用于解决OCR幻觉、密集文本处理或复杂排版理解等痛点。
Abstract With the rapid evolution of deepfake technologies and the wide dissemination of digital media, personal privacy is facing increasingly serious security threats. Deepfake proactive forensics, which involves embedding imperceptible watermarks to enable reliable source tracking, serves as a crucial defense against these threats. Although existing methods show strong forensic ability, they rely on an idealized assumption of single watermark embedding, which proves impractical in real-world scenarios. In this paper, we formally define and demonstrate the existence of Multi-Embedding Attacks (MEA) for the first time. When a previously protected image undergoes additional rounds of watermark embedding, the original forensic watermark can be destroyed or removed, rendering the entire proactive forensic mechanism ineffective. To address this vulnerability, we propose a general training paradigm named Adversarial Interference Simulation (AIS). Rather than modifying the network architecture, AIS explicitly simulates MEA scenarios during fine-tuning and introduces a resilience-driven loss function to enforce the learning of sparse and stable watermark representations. Our method enables the model to maintain the ability to extract the original watermark correctly even after a second embedding. Extensive experiments demonstrate that our plug-and-play AIS training paradigm significantly enhances the robustness of various existing methods against MEA.
Authors Lixin Jia, Haiyang Sun, Zhiqing Guo, Yunfeng Diao, Dan Ma, Gaobo Yang
Categories Computer Vision and Pattern Recognition
0
DeepRFTv2: Kernel-level Learning for Image Deblurring
Title CN DeepRFTv2：用于图像去模糊的核级学习
Keywords 图像去模糊 · 核级学习 · 傅里叶变换 · 特征级卷积 · 低层视觉
Summary 本文提出了一种名为Fourier Kernel Estimator（FKE）的方法，通过在傅里叶空间中建模卷积操作，实现对模糊核的学习，从而提升图像去模糊效果。该方法从特征层面而非像素层面进行核级学习，并结合多尺度可逆子网络结构，在低训练内存下实现了高效去模糊。实验表明其在运动去模糊任务上达到SOTA水平。
Reason 该论文研究的是图像去模糊任务中的核级建模，属于低层视觉（low-level vision）范畴，主要解决运动模糊的物理过程建模问题。尽管其提出的傅里叶核估计器（FKE）在理论上具有可解释性，但其方法目标与文档图像理解（DIU）的核心痛点——如OCR幻觉、复杂版面分析、细粒度文本定位、长文档推理等——无直接关联。去模糊作为预处理步骤可能间接影响OCR性能，但本文并未针对文档内容结构或语义理解进行设计，也无法迁移用于解决VLM在文档上的多模态对齐或幻觉问题，属于无关领域的技术工作。
Abstract It is well-known that if a network aims to learn how to deblur, it should understand the blur process. Blurring is naturally caused by the convolution of the sharp image with the blur kernel. Thus, allowing the network to learn the blur process in the kernel-level can significantly improve the image deblurring performance. But, current deep networks are still at the pixel-level learning stage, either performing end-to-end pixel-level restoration or stage-wise pseudo kernel-level restoration, failing to enable the deblur model to understand the essence of the blur. To this end, we propose Fourier Kernel Estimator (FKE), which considers the activation operation in Fourier space and converts the convolution problem in the spatial domain to a multiplication problem in Fourier space. Our FKE, jointly optimized with the deblur model, enables the network to learn the kernel-level blur process with low complexity and without any additional supervision. Furthermore, we change the convolution object of the kernel from ``image" to network extracted ``feature", whose rich semantic and structural information is more suitable to blur process learning. With the convolution of the feature and the estimated kernel, our model can learn the essence of blur in kernel-level. To further improve the efficiency of feature extraction, we design a decoupled multi-scale architecture with multiple hierarchical sub-unets with a reversible strategy, which allows better multi-scale encoding and decoding in low training memory. Extensive experiments indicate that our method achieves state-of-the-art motion deblurring results and show potential for handling other kernel-related problems. Analysis also shows our kernel estimator is able to learn physically meaningful kernels. The code will be available at https://github.com/DeepMed-Lab-ECNU/Single-Image-Deblur.
Authors Xintian Mao, Haofei Song, Yin-Nian Liu, Qingli Li, Yan Wang
Categories Computer Vision and Pattern Recognition
0
CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion
Title CN CtrlVDiff：通过统一的多模态视频扩散实现可控视频生成
Keywords 视频生成 · 多模态扩散模型 · 可控生成 · 图形学内在属性 · 时序一致性
Summary 本文提出CtrlVDiff，一种基于统一扩散框架的可控视频生成模型，融合深度、法线、边缘、语义分割及材质属性等多种模态，在MMVideo数据集上实现高质量、时序一致的视频重渲染与编辑。尽管多模态控制策略先进，但其应用目标和数据假设与文档图像理解任务完全不匹配。
Reason 该论文聚焦于视频理解与可控视频生成，属于纯视频生成领域，使用多模态扩散模型结合图形学内在属性（如albedo、roughness）进行视频编辑。其技术路线与文档图像理解（DIU）的核心痛点（如OCR幻觉、密集文本识别、复杂版面分析）无直接关联，且方法依赖于高成本的多模态合成数据（如法线、深度图），难以迁移到文档场景。属于明确的无关领域（视频生成），命中负面清单第2条。
Abstract We tackle the dual challenges of video understanding and controllable video generation within a unified diffusion framework. Our key insights are two-fold: geometry-only cues (e.g., depth, edges) are insufficient: they specify layout but under-constrain appearance, materials, and illumination, limiting physically meaningful edits such as relighting or material swaps and often causing temporal drift. Enriching the model with additional graphics-based modalities (intrinsics and semantics) provides complementary constraints that both disambiguate understanding and enable precise, predictable control during generation. However, building a single model that uses many heterogeneous cues introduces two core difficulties. Architecturally, the model must accept any subset of modalities, remain robust to missing inputs, and inject control signals without sacrificing temporal consistency. Data-wise, training demands large-scale, temporally aligned supervision that ties real videos to per-pixel multimodal annotations. We then propose CtrlVDiff, a unified diffusion model trained with a Hybrid Modality Control Strategy (HMCS) that routes and fuses features from depth, normals, segmentation, edges, and graphics-based intrinsics (albedo, roughness, metallic), and re-renders videos from any chosen subset with strong temporal coherence. To enable this, we build MMVideo, a hybrid real-and-synthetic dataset aligned across modalities and captions. Across understanding and generation benchmarks, CtrlVDiff delivers superior controllability and fidelity, enabling layer-wise edits (relighting, material adjustment, object insertion) and surpassing state-of-the-art baselines while remaining robust when some modalities are unavailable.
Authors Dianbing Xi, Jiepeng Wang, Yuanzhi Liang, Xi Qiu, Jialun Liu, Hao Pan, Yuchi Huo, Rui Wang, Haibin Huang, Chi Zhang, Xuelong Li
Comment 27 pages, 18 figures, 9 tables. Project page: https://tele-ai.github.io/CtrlVDiff/
Categories Computer Vision and Pattern Recognition
0
Generalizable cardiac substructures segmentation from contrast and non-contrast CTs using pretrained transformers
Title CN 基于预训练Transformer的对比与非对比CT图像中可泛化心脏亚结构分割
Keywords 医学图像分割 · Transformer · CT图像 · 泛化性 · 放射治疗规划
Summary 本文提出一种混合Transformer卷积网络，用于在肺癌和乳腺癌患者的对比与非对比CT图像中分割心脏亚结构。通过构建平衡的数据分布（包含对比与非对比扫描），模型在跨扫描协议和患者特征上表现出良好的几何与剂量学准确性，且所需标注数据显著少于传统方法。然而，该工作专注于医学影像中的器官分割，不涉及文本内容理解或多模态文档推理。
Reason 该论文属于医学图像分割领域，研究任务为心脏亚结构在不同CT扫描条件下的分割，尽管使用了Transformer架构并强调泛化性，但其核心问题与文档图像理解（DIU）无关。未涉及OCR、文本理解、版面分析、表格识别或任何文档特有的挑战（如密集文本、排版复杂性、视觉-语言推理等）。属于典型的垂直领域应用（医疗影像），且无方法论上的迁移价值可用于解决DIU中的幻觉、细粒度定位或长文档推理等问题。
Abstract Automated AI segmentations for radiation treatment planning deteriorate when applied to cases with different characteristics than the training dataset. We developed a hybrid transformer convolutional network to segment cardiac substructures in lung and breast cancer patients with varying imaging contrasts and scan positions. Cohort I (56 contrast-enhanced CT [CECT], 124 non-contrast CT [NCCT] scans from lung cancer patients, supine position) was used to train an oracle model (180 cases), contrast-only model (56 CECTs), and balanced model (32 CECT, 32 NCCT). All models were evaluated on 60 held-out cohort I patients and 66 cohort II breast cancer patients (45 supine, 21 prone). Accuracy was measured using Dice similarity coefficient (DSC), 95th percentile Hausdorff distance (HD95), and dosimetric metrics, with TotalSegmentator as benchmark. Oracle and balanced models achieved similar accuracy (DSC: Oracle vs Balanced: Cohort I: 0.84 $\pm$ 0.10 vs 0.82 $\pm$ 0.10; Cohort II: 0.81 $\pm$ 0.12 vs 0.80 $\pm$ 0.13), both outperforming TotalSegmentator and the contrast-only models. The balanced model, using 64% fewer training cases, produced dosimetrically equivalent contours to manual delineations. It was robust to contrast variations (6 out of 8 substructures) and positioning variations (5 out of 8 substructures), with low correlation to patient age or body mass index. Our balanced model demonstrated robust geometric and dosimetric accuracy across varying imaging protocols and patient characteristics, which is essential for clinical deployment. Combining pretraining with balanced NCCT/CECT distribution enabled reliable segmentation with substantially fewer labeled cases than conventional approaches.
Authors Aneesh Rangnekar, Nikhil Mankuzhy, Jonas Willmann, Chloe Choi, Abraham Wu, Maria Thor, Andreas Rimner, Harini Veeraraghavan
Categories Computer Vision and Pattern Recognition
0
FaithFusion: Harmonizing Reconstruction and Generation via Pixel-wise Information Gain
Title CN FaithFusion：通过像素级信息增益协调重建与生成
Keywords 3D场景生成 · 3D高斯泼溅 · 扩散模型 · 信息增益 · 几何保真度
Summary 本文提出FaithFusion，一种基于像素级期望信息增益（EIG）的3DGS与扩散模型融合框架，用于可控的驾驶场景重建与3D场景生成。EIG作为统一策略指导扩散模型在高不确定性区域进行外观修复，并将编辑结果蒸馏回3DGS表示中，从而实现时空一致的合成。实验表明该方法在Waymo数据集上达到SOTA性能。
Reason 该论文研究的是驾驶场景下的3D场景重建与生成，核心问题是融合3D高斯泼溅（3DGS）与扩散模型以保持几何保真度和视觉合理性。其技术重点在于处理大视角变化下的外观合成与几何漂移，属于自动驾驶与3D视觉领域。尽管使用了像素级信息增益机制，但其方法设计针对的是3D场景生成中的编辑一致性问题，与文档图像理解（DIU）中的OCR幻觉、密集文本识别、复杂版式分析或长文档推理等核心痛点无直接关联。无法迁移用于解决文档理解中的典型挑战。
Abstract In controllable driving-scene reconstruction and 3D scene generation, maintaining geometric fidelity while synthesizing visually plausible appearance under large viewpoint shifts is crucial. However, effective fusion of geometry-based 3DGS and appearance-driven diffusion models faces inherent challenges, as the absence of pixel-wise, 3D-consistent editing criteria often leads to over-restoration and geometric drift. To address these issues, we introduce \textbf{FaithFusion}, a 3DGS-diffusion fusion framework driven by pixel-wise Expected Information Gain (EIG). EIG acts as a unified policy for coherent spatio-temporal synthesis: it guides diffusion as a spatial prior to refine high-uncertainty regions, while its pixel-level weighting distills the edits back into 3DGS. The resulting plug-and-play system is free from extra prior conditions and structural modifications.Extensive experiments on the Waymo dataset demonstrate that our approach attains SOTA performance across NTA-IoU, NTL-IoU, and FID, maintaining an FID of 107.47 even at 6 meters lane shift. Our code is available at https://github.com/wangyuanbiubiubiu/FaithFusion.
Authors YuAn Wang, Xiaofan Li, Chi Huang, Wenhao Zhang, Hao Li, Bosheng Wang, Xun Sun, Jun Wang
Comment 16 pages, 10 figures
Categories Computer Vision and Pattern Recognition
0
BRIC: Bridging Kinematic Plans and Physical Control at Test Time AAAI'26
Title CN BRIC：在测试时桥接运动规划与物理控制
Keywords 测试时适应 · 扩散模型 · 物理控制 · 运动生成 · 具身智能
Summary 本文提出BRIC，一种测试时自适应框架，用于解决基于扩散模型的运动规划器与基于强化学习的物理控制器之间的执行偏差问题。通过动态调整控制器并引入无需更新参数的轻量级引导机制，实现长期、连贯且物理合理的动作生成。实验验证了其在多种长周期任务中的有效性。
Reason 该论文研究的是人体运动生成中扩散模型与物理控制器之间的协同问题，属于具身智能与物理仿真领域，核心关注点为运动控制和物理合理性，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管提到了测试时适应（TTA）和引导机制，但其方法针对的是三维运动信号空间的调整，难以迁移到文档理解中的OCR幻觉、复杂排版解析或细粒度文本定位等核心痛点。
Abstract We propose BRIC, a novel test-time adaptation (TTA) framework that enables long-term human motion generation by resolving execution discrepancies between diffusion-based kinematic motion planners and reinforcement learning-based physics controllers. While diffusion models can generate diverse and expressive motions conditioned on text and scene context, they often produce physically implausible outputs, leading to execution drift during simulation. To address this, BRIC dynamically adapts the physics controller to noisy motion plans at test time, while preserving pre-trained skills via a loss function that mitigates catastrophic forgetting. In addition, BRIC introduces a lightweight test-time guidance mechanism that steers the diffusion model in the signal space without updating its parameters. By combining both adaptation strategies, BRIC ensures consistent and physically plausible long-term executions across diverse environments in an effective and efficient manner. We validate the effectiveness of BRIC on a variety of long-term tasks, including motion composition, obstacle avoidance, and human-scene interaction, achieving state-of-the-art performance across all tasks.
Authors Dohun Lim, Minji Kim, Jaewoon Lim, Sungchan Kim
Comment Accepted to AAAI'26
Categories Computer Vision and Pattern Recognition
0
Scaling Foundation Models for Radar Scene Understanding
Title CN 用于雷达场景理解的基础模型扩展
Keywords 雷达感知 · 基础模型 · 空间推理 · 对比学习 · CARLA
Summary 本文提出RadarFM，一种基于结构化空间语言监督的雷达基础模型，通过原生雷达坐标编码车辆分布，并设计哈希感知的对比学习目标以实现细粒度的空间推理。研究使用CARLA模拟器生成大规模标注数据，并提出定位感知的评估指标。但整体工作远离文档理解范畴。
Reason 该论文研究的是雷达感知与基础模型的结合，属于自动驾驶中的传感器理解领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其方法聚焦于雷达坐标下的空间语言监督和场景相似性建模，难以迁移到文档图像中的OCR、排版分析或VQA等任务。属于无关领域。
Abstract Radar sensors provide reliable perception across adverse weather, lighting, and long-range conditions. Recent advances in foundation models have transformed visual and language understanding, yet their integration with radar sensing remains largely underexplored. Existing radar approaches are fragmented and task-specific; each downstream task employs distinct architectures and training objectives, preventing transfer across tasks. In this work, we introduce RadarFM: a radar foundation model that learns unified scene-level representations through structured spatial language supervision. We make two key contributions: (1) a structured caption framework that encodes vehicle distributions in native radar coordinates, and (2) a hash-aware contrastive learning objective that quantifies continuous scene similarity rather than binary matching, enabling fine-grained spatial reasoning. Leveraging the CARLA simulator, we generate large-scale, well-annotated radar datasets across diverse driving scenarios. We also propose localization-aware metrics that assess spatial accuracy beyond traditional detection measures.
Authors Pushkal Mishra, Kshitiz Bansal, Dinesh Bharadia
Categories Computer Vision and Pattern Recognition
0
Towards Spatially Consistent Image Generation: On Incorporating Intrinsic Scene Properties into Diffusion Models
Title CN 迈向空间一致的图像生成：将内在场景属性融入扩散模型
Keywords 图像生成 · 扩散模型 · 空间一致性 · 内在场景属性 · 潜变量建模
Summary 本文提出一种将内在场景属性（如深度和语义分割图）融入扩散模型的方法，用于改善生成图像的空间一致性和布局合理性。通过从大规模图像数据中提取这些属性并编码为共享潜变量，模型在去噪过程中同时优化图像和内在属性域，实现相互约束。实验表明该方法能在保持图像质量和文本对齐的同时减少结构畸变。
Reason 该论文研究的是通用图像生成中的空间一致性问题，属于纯图像生成领域。其方法聚焦于通过联合建模深度、分割图等内在属性来提升生成图像的结构合理性，但应用场景与文档图像理解（DIU）的核心痛点（如OCR幻觉、密集文本识别、复杂版式分析）无直接关联。尽管‘空间结构建模’在理论上可能启发布局分析，但该工作本身并未针对文档内容设计，且技术路径难以迁移到以文本为中心的细粒度视觉理解任务中。
Abstract Image generation models trained on large datasets can synthesize high-quality images but often produce spatially inconsistent and distorted images due to limited information about the underlying structures and spatial layouts. In this work, we leverage intrinsic scene properties (e.g., depth, segmentation maps) that provide rich information about the underlying scene, unlike prior approaches that solely rely on image-text pairs or use intrinsics as conditional inputs. Our approach aims to co-generate both images and their corresponding intrinsics, enabling the model to implicitly capture the underlying scene structure and generate more spatially consistent and realistic images. Specifically, we first extract rich intrinsic scene properties from a large image dataset with pre-trained estimators, eliminating the need for additional scene information or explicit 3D representations. We then aggregate various intrinsic scene properties into a single latent variable using an autoencoder. Building upon pre-trained large-scale Latent Diffusion Models (LDMs), our method simultaneously denoises the image and intrinsic domains by carefully sharing mutual information so that the image and intrinsic reflect each other without degrading image quality. Experimental results demonstrate that our method corrects spatial inconsistencies and produces a more natural layout of scenes while maintaining the fidelity and textual alignment of the base model (e.g., Stable Diffusion).
Authors Hyundo Lee, Suhyung Choi, Inwoo Hwang, Byoung-Tak Zhang
Categories Computer Vision and Pattern Recognition
0
CLRecogEye : Curriculum Learning towards exploiting convolution features for Dynamic Iris Recognition ISVC conference 2025
Title CN CLRecogEye：面向动态虹膜识别的卷积特征利用的课程学习
Keywords 虹膜识别 · 课程学习 · 3D-CNN · 时空特征 · 生物特征认证
Summary 本文提出了一种基于课程学习的动态虹膜识别方法CLRecogEye，通过将虹膜图像切分为子图像序列并输入3D-CNN，以捕捉虹膜的时空特征。模型采用三元组损失和ArcFace损失进行端到端训练，提升了在旋转、尺度变化、反光和模糊等情况下的识别鲁棒性。
Reason 该论文研究的是动态虹膜识别，属于生物特征识别领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管使用了3D-CNN和课程学习等技术，但其方法针对的是虹膜序列中的时空特征建模，无法迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Iris authentication algorithms have achieved impressive recognition performance, making them highly promising for real-world applications such as border control, citizen identification, and both criminal investigations and commercial systems. However, their robustness is still challenged by variations in rotation, scale, specular reflections, and defocus blur. In addition, most existing approaches rely on straightforward point-to-point comparisons, typically using cosine or L2 distance, without effectively leveraging the spatio-spatial-temporal structure of iris patterns. To address these limitations, we propose a novel and generalized matching pipeline that learns rich spatio-spatial-temporal representations of iris features. Our approach first splits each iris image along one dimension, generating a sequence of sub-images that serve as input to a 3D-CNN, enabling the network to capture both spatial and spatio-spatial-temporal cues. To further enhance the modeling of spatio-spatial-temporal feature dynamics, we train the model in curriculum manner. This design allows the network to embed temporal dependencies directly into the feature space, improving discriminability in the deep metric domain. The framework is trained end-to-end with triplet and ArcFace loss in a curriculum manner, enforcing highly discriminative embeddings despite challenges like rotation, scale, reflections, and blur. This design yields a robust and generalizable solution for iris authentication.Github code: https://github.com/GeetanjaliGTZ/CLRecogEye
Authors Geetanjali Sharma, Gaurav Jaswal, Aditya Nigam, Raghavendra Ramachandra
Comment 12 Pages, 3 figures, ISVC conference 2025
Categories Computer Vision and Pattern Recognition
0
Point-Supervised Facial Expression Spotting with Gaussian-Based Instance-Adaptive Intensity Modeling
Title CN 基于高斯实例自适应强度建模的点监督面部表情检测
Keywords 面部表情检测 · 点监督学习 · 高斯建模 · 弱监督 · 视频分析
Summary 本文提出一种点监督的面部表情检测方法P-FES，仅需每个表情实例一个时间戳标注。通过设计高斯实例自适应强度建模（GIM）模块生成软伪标签，并结合双分支框架分别处理表达强度预测与宏/微表情分类，同时引入强度感知对比损失以增强特征判别性。在多个数据集上验证了方法有效性。
Reason 该论文研究的是视频中的面部表情检测任务，属于人脸分析与视频理解领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法聚焦于从稀疏时间点标注中学习表情强度分布，解决的是时序建模和弱监督学习问题，但所提出的技术（如GIM模块、双分支结构）难以迁移到文档理解的核心痛点（如OCR幻觉、复杂排版解析、细粒度文本定位等）。
Abstract Automatic facial expression spotting, which aims to identify facial expression instances in untrimmed videos, is crucial for facial expression analysis. Existing methods primarily focus on fully-supervised learning and rely on costly, time-consuming temporal boundary annotations. In this paper, we investigate point-supervised facial expression spotting (P-FES), where only a single timestamp annotation per instance is required for training. We propose a unique two-branch framework for P-FES. First, to mitigate the limitation of hard pseudo-labeling, which often confuses neutral and expression frames with various intensities, we propose a Gaussian-based instance-adaptive intensity modeling (GIM) module to model instance-level expression intensity distribution for soft pseudo-labeling. By detecting the pseudo-apex frame around each point label, estimating the duration, and constructing an instance-level Gaussian distribution, GIM assigns soft pseudo-labels to expression frames for more reliable intensity supervision. The GIM module is incorporated into our framework to optimize the class-agnostic expression intensity branch. Second, we design a class-aware apex classification branch that distinguishes macro- and micro-expressions solely based on their pseudo-apex frames. During inference, the two branches work independently: the class-agnostic expression intensity branch generates expression proposals, while the class-aware apex-classification branch is responsible for macro- and micro-expression classification. Furthermore, we introduce an intensity-aware contrastive loss to enhance discriminative feature learning and suppress neutral noise by contrasting neutral frames with expression frames with various intensities. Extensive experiments on the SAMM-LV, CAS(ME)$^2$, and CAS(ME)$^3$ datasets demonstrate the effectiveness of our proposed framework.
Authors Yicheng Deng, Hideaki Hayashi, Hajime Nagahara
Categories Computer Vision and Pattern Recognition
0
VGGTFace: Topologically Consistent Facial Geometry Reconstruction in the Wild
Title CN VGGTFace：野外环境下的拓扑一致人脸几何重建
Keywords 3D人脸重建 · 拓扑一致性 · 基础模型 · Bundle Adjustment · 野外图像
Summary 本文提出VGGTFace，一种基于3D基础模型VGGT的自动化方法，用于从普通用户拍摄的野外多视角图像中重建拓扑一致的人脸几何结构。通过引入Pixel3DMM注入UV空间的拓扑信息，并设计拓扑感知的Bundle Adjustment策略融合点云，实现了高质量且快速的人脸重建。在多个基准上达到SOTA效果。
Reason 该论文研究的是三维人脸重建，属于3D视觉与数字人领域，核心任务为从多视角图像中重建具有拓扑一致性的人脸网格。尽管使用了视觉基础模型和几何优化方法，但其目标与文档图像理解（DIU）无直接关联，不涉及OCR、文档布局分析、表格识别或VQA等任务，也无法迁移用于解决文档理解中的OCR幻觉、密集文本处理或复杂排版解析等核心痛点。
Abstract Reconstructing topologically consistent facial geometry is crucial for the digital avatar creation pipelines. Existing methods either require tedious manual efforts, lack generalization to in-the-wild data, or are constrained by the limited expressiveness of 3D Morphable Models. To address these limitations, we propose VGGTFace, an automatic approach that innovatively applies the 3D foundation model, i.e. VGGT, for topologically consistent facial geometry reconstruction from in-the-wild multi-view images captured by everyday users. Our key insight is that, by leveraging VGGT, our method naturally inherits strong generalization ability and expressive power from its large-scale training and point map representation. However, it is unclear how to reconstruct a topologically consistent mesh from VGGT, as the topology information is missing in its prediction. To this end, we augment VGGT with Pixel3DMM for injecting topology information via pixel-aligned UV values. In this manner, we convert the pixel-aligned point map of VGGT to a point cloud with topology. Tailored to this point cloud with known topology, we propose a novel Topology-Aware Bundle Adjustment strategy to fuse them, where we construct a Laplacian energy for the Bundle Adjustment objective. Our method achieves high-quality reconstruction in 10 seconds for 16 views on a single NVIDIA RTX 4090. Experiments demonstrate state-of-the-art results on benchmarks and impressive generalization to in-the-wild data. Code is available at https://github.com/grignarder/vggtface.
Authors Xin Ming, Yuxuan Han, Tianyu Huang, Feng Xu
Categories Computer Vision and Pattern Recognition
0
Long-Term Alzheimers Disease Prediction: A Novel Image Generation Method Using Temporal Parameter Estimation with Normal Inverse Gamma Distribution on Uneven Time Series
Title CN 基于正态逆伽马分布的时间参数估计在不规则时间序列上进行长期阿尔茨海默病预测的新型图像生成方法
Keywords 医学图像生成 · 阿尔茨海默病预测 · 不规则时间序列 · 不确定性估计 · T-NIG
Summary 本文提出了一种基于正态逆伽马分布（T-NIG）的时间参数估计模型，用于在不规则时间间隔下生成中间和未来的脑部MRI图像，以实现阿尔茨海默病的长期预测。该方法利用坐标邻域特征并引入时间参数到分布中，结合不确定性估计来缓解稀疏时间数据带来的问题，在短期和长期预测任务上表现出色。
Reason 该论文属于纯医学图像生成领域，研究目标是通过生成脑部图像来预测阿尔茨海默病进展，属于视频/时序医学图像生成任务。其核心技术（T-NIG、不确定性建模）针对的是生物医学信号的时间外推问题，与文档图像理解（DIU）中的OCR幻觉、排版分析、视觉文本理解等核心痛点完全无关。且未涉及多模态语言模型、视觉token压缩或文档结构建模等可迁移技术，属于典型的无关领域工作。
Abstract Image generation can provide physicians with an imaging diagnosis basis in the prediction of Alzheimer's Disease (AD). Recent research has shown that long-term AD predictions by image generation often face difficulties maintaining disease-related characteristics when dealing with irregular time intervals in sequential data. Considering that the time-related aspects of the distribution can reflect changes in disease-related characteristics when images are distributed unevenly, this research proposes a model to estimate the temporal parameter within the Normal Inverse Gamma Distribution (T-NIG) to assist in generating images over the long term. The T-NIG model employs brain images from two different time points to create intermediate brain images, forecast future images, and predict the disease. T-NIG is designed by identifying features using coordinate neighborhoods. It incorporates a time parameter into the normal inverse gamma distribution to understand how features change in brain imaging sequences that have varying time intervals. Additionally, T-NIG utilizes uncertainty estimation to reduce both epistemic and aleatoric uncertainties in the model, which arise from insufficient temporal data. In particular, the T-NIG model demonstrates state-of-the-art performance in both short-term and long-term prediction tasks within the dataset. Experimental results indicate that T-NIG is proficient in forecasting disease progression while maintaining disease-related characteristics, even when faced with an irregular temporal data distribution.
Authors Xin Hong, Xinze Sun, Yinhao Li, Yen-Wei Chen
Comment 13pages, 6 figures
Categories Computer Vision and Pattern Recognition
0
AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarios AAAI 2026
Title CN AerialMind：面向无人机场景的指代表多目标跟踪
Keywords 指代表多目标跟踪 · 无人机 · 视觉-语言表征 · COALA · HawkEyeTrack
Summary 本文提出了AerialMind，首个面向无人机场景的大规模指代表多目标跟踪（RMOT）基准数据集，并开发了COALA半自动标注框架以降低标注成本。同时提出HawkEyeTrack方法，通过增强视觉-语言表征学习来提升对无人机场景的感知能力。实验验证了数据集的挑战性和方法的有效性。
Reason 该论文研究的是无人机（UAV）场景下的指代表多目标跟踪（RMOT），属于具身智能与视频理解领域，核心任务为动态场景中的视觉-语言对齐与目标追踪。其应用场景、数据模态（航拍视频流）和任务目标均与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Referring Multi-Object Tracking (RMOT) aims to achieve precise object detection and tracking through natural language instructions, representing a fundamental capability for intelligent robotic systems. However, current RMOT research remains mostly confined to ground-level scenarios, which constrains their ability to capture broad-scale scene contexts and perform comprehensive tracking and path planning. In contrast, Unmanned Aerial Vehicles (UAVs) leverage their expansive aerial perspectives and superior maneuverability to enable wide-area surveillance. Moreover, UAVs have emerged as critical platforms for Embodied Intelligence, which has given rise to an unprecedented demand for intelligent aerial systems capable of natural language interaction. To this end, we introduce AerialMind, the first large-scale RMOT benchmark in UAV scenarios, which aims to bridge this research gap. To facilitate its construction, we develop an innovative semi-automated collaborative agent-based labeling assistant (COALA) framework that significantly reduces labor costs while maintaining annotation quality. Furthermore, we propose HawkEyeTrack (HETrack), a novel method that collaboratively enhances vision-language representation learning and improves the perception of UAV scenarios. Comprehensive experiments validated the challenging nature of our dataset and the effectiveness of our method.
Authors Chenglizhao Chen, Shaofeng Liang, Runwei Guan, Xiaolou Sun, Haocheng Zhao, Haiyun Jiang, Tao Huang, Henghui Ding, Qing-Long Han
Comment AAAI 2026
Categories Computer Vision and Pattern Recognition
0
MUSE: Manipulating Unified Framework for Synthesizing Emotions in Images via Test-Time Optimization
Title CN MUSE：通过测试时优化实现图像情感合成的统一框架
Keywords 图像情感合成 · 测试时优化 · 扩散模型 · 情感编辑 · 统一框架
Summary 本文提出了MUSE，首个能够同时进行情感生成和编辑的统一框架，基于测试时优化策略，利用现成的情绪分类器通过梯度优化情感令牌来指导图像生成。该方法解决了何时、如何以及选择何种情绪进行引导的问题，在情感准确性和语义多样性方面表现优异。然而，其应用场景局限于艺术性图像的情感操控，不属于文档理解范畴。
Reason 该论文研究的是图像情感合成（Image Emotional Synthesis），属于图像生成与情感操控领域，核心任务是操纵图像中的情绪表达，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。尽管其使用了Test-Time Optimization技术，但该方法的应用场景聚焦于扩散模型中的情感控制，迁移到文档理解中的OCR幻觉、复杂排版解析或长文档推理等问题极为困难。且未涉及文本布局分析、视觉语言建模在文档上的应用，也不处理密集文本或多模态对齐等DIU核心挑战。
Abstract Images evoke emotions that profoundly influence perception, often prioritized over content. Current Image Emotional Synthesis (IES) approaches artificially separate generation and editing tasks, creating inefficiencies and limiting applications where these tasks naturally intertwine, such as therapeutic interventions or storytelling. In this work, we introduce MUSE, the first unified framework capable of both emotional generation and editing. By adopting a strategy conceptually aligned with Test-Time Scaling (TTS) that widely used in both LLM and diffusion model communities, it avoids the requirement for additional updating diffusion model and specialized emotional synthesis datasets. More specifically, MUSE addresses three key questions in emotional synthesis: (1) HOW to stably guide synthesis by leveraging an off-the-shelf emotion classifier with gradient-based optimization of emotional tokens; (2) WHEN to introduce emotional guidance by identifying the optimal timing using semantic similarity as a supervisory signal; and (3) WHICH emotion to guide synthesis through a multi-emotion loss that reduces interference from inherent and similar emotions. Experimental results show that MUSE performs favorably against all methods for both generation and editing, improving emotional accuracy and semantic diversity while maintaining an optimal balance between desired content, adherence to text prompts, and realistic emotional expression. It establishes a new paradigm for emotion synthesis.
Authors Yingjie Xia, Xi Wang, Jinglei Shi, Vicky Kalogeiton, Jian Yang
Categories Computer Vision and Pattern Recognition
0
PG-ControlNet: A Physics-Guided ControlNet for Generative Spatially Varying Image Deblurring
Title CN PG-ControlNet：一种用于生成式空间可变图像去模糊的物理引导ControlNet
Keywords 图像去模糊 · ControlNet · 扩散模型 · 物理约束 · 生成模型
Summary 本文提出了一种结合物理建模与生成先验的图像去模糊框架PG-ControlNet，通过建立密集的高维压缩核场来描述空间变化的模糊模式，并利用该物理特征场作为ControlNet条件来引导扩散模型采样过程，在严重模糊场景下实现了物理准确性与视觉真实感的平衡。
Reason 该论文研究的是图像去模糊任务，属于低层视觉中的图像恢复领域，尽管使用了ControlNet和扩散模型，但其核心目标是提升图像复原的物理准确性和感知质量，并非文档图像理解（DIU）任务。虽然其‘物理引导’的思想在理论上可能对文档退化建模有启发，但整体方法针对的是通用图像模糊问题，迁移到OCR或DocVQA等文档理解任务中极为困难，且未涉及文本结构、布局分析或语言先验等DIU关键要素。
Abstract Spatially varying image deblurring remains a fundamentally ill-posed problem, especially when degradations arise from complex mixtures of motion and other forms of blur under significant noise. State-of-the-art learning-based approaches generally fall into two paradigms: model-based deep unrolling methods that enforce physical constraints by modeling the degradations, but often produce over-smoothed, artifact-laden textures, and generative models that achieve superior perceptual quality yet hallucinate details due to weak physical constraints. In this paper, we propose a novel framework that uniquely reconciles these paradigms by taming a powerful generative prior with explicit, dense physical constraints. Rather than oversimplifying the degradation field, we model it as a dense continuum of high-dimensional compressed kernels, ensuring that minute variations in motion and other degradation patterns are captured. We leverage this rich descriptor field to condition a ControlNet architecture, strongly guiding the diffusion sampling process. Extensive experiments demonstrate that our method effectively bridges the gap between physical accuracy and perceptual realism, outperforming state-of-the-art model-based methods as well as generative baselines in challenging, severely blurred scenarios.
Authors Hakki Motorcu, Mujdat Cetin
Comment 9 pages, 4 figures
Categories Computer Vision and Pattern Recognition
0
LungNoduleAgent: A Collaborative Multi-Agent System for Precision Diagnosis of Lung Nodules AAAI 2026
Title CN LungNoduleAgent：用于肺结节精准诊断的协作式多智能体系统
Keywords 多智能体系统 · 医学影像分析 · 肺结节检测 · 视觉语言模型 · 临床决策支持
Summary 本文提出LungNoduleAgent，一个用于肺结节诊断的多智能体系统，包含结节检测、放射科医生报告生成和恶性程度推理三个模块，在私有数据集和LIDC-IDRI上表现优于现有模型。系统强调区域语义对齐与多智能体协作在病理诊断中的作用。
Reason 该论文属于医学影像领域的垂直应用，尽管提出了多智能体协作框架，但其任务场景（肺结节CT分析）与文档图像理解（DIU）无关。虽然‘多智能体系统’作为方法论可能具有上游潜力，但文中未体现对文档理解核心痛点（如OCR幻觉、复杂排版、长文档推理）的迁移价值，且整体为特定医疗场景的下游应用，属于典型的‘平行应用’，违反负面清单第一条。
Abstract Diagnosing lung cancer typically involves physicians identifying lung nodules in Computed tomography (CT) scans and generating diagnostic reports based on their morphological features and medical expertise. Although advancements have been made in using multimodal large language models for analyzing lung CT scans, challenges remain in accurately describing nodule morphology and incorporating medical expertise. These limitations affect the reliability and effectiveness of these models in clinical settings. Collaborative multi-agent systems offer a promising strategy for achieving a balance between generality and precision in medical applications, yet their potential in pathology has not been thoroughly explored. To bridge these gaps, we introduce LungNoduleAgent, an innovative collaborative multi-agent system specifically designed for analyzing lung CT scans. LungNoduleAgent streamlines the diagnostic process into sequential components, improving precision in describing nodules and grading malignancy through three primary modules. The first module, the Nodule Spotter, coordinates clinical detection models to accurately identify nodules. The second module, the Radiologist, integrates localized image description techniques to produce comprehensive CT reports. Finally, the Doctor Agent System performs malignancy reasoning by using images and CT reports, supported by a pathology knowledge base and a multi-agent system framework. Extensive testing on two private datasets and the public LIDC-IDRI dataset indicates that LungNoduleAgent surpasses mainstream vision-language models, agent systems, and advanced expert models. These results highlight the importance of region-level semantic alignment and multi-agent collaboration in diagnosing nodules. LungNoduleAgent stands out as a promising foundational tool for supporting clinical analyses of lung nodules.
Authors Cheng Yang, Hui Jin, Xinlei Yu, Zhipeng Wang, Yaoqun Liu, Fenglei Fan, Dajiang Lei, Gangyong Jia, Changmiao Wang, Ruiquan Ge
Comment Accepted by AAAI 2026
Categories Computer Vision and Pattern Recognition
0
CNN-LSTM Hybrid Architecture for Over-the-Air Automatic Modulation Classification Using SDR Journal of Innovations in Engineering Education
Title CN 基于SDR的空中自动调制分类CNN-LSTM混合架构
Keywords 自动调制分类 · CNN-LSTM · 软件定义无线电 · 无线通信 · 信号识别
Summary 本文提出了一种CNN-LSTM混合架构，用于通过软件定义无线电（SDR）平台实现空中信号的自动调制分类。利用CNN提取空间特征，LSTM捕捉时间依赖性，在RadioML2018和自建数据集上取得了超过93%的准确率。实验验证了该方法在噪声环境下对调制信号的识别能力。
Reason 该论文研究的是无线通信信号中的自动调制分类（AMC），属于通信与信号处理领域，与文档图像理解（DIU）或视觉语言模型（VLM）无任何关联。其方法和任务均无法迁移到OCR、DocVQA、版面分析等文档理解核心问题上。
Abstract Automatic Modulation Classification (AMC) is a core technology for future wireless communication systems, enabling the identification of modulation schemes without prior knowledge. This capability is essential for applications in cognitive radio, spectrum monitoring, and intelligent communication networks. We propose an AMC system based on a hybrid Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) architecture, integrated with a Software Defined Radio (SDR) platform. The proposed architecture leverages CNNs for spatial feature extraction and LSTMs for capturing temporal dependencies, enabling efficient handling of complex, time-varying communication signals. The system's practical ability was demonstrated by identifying over-the-air (OTA) signals from a custom-built FM transmitter alongside other modulation schemes. The system was trained on a hybrid dataset combining the RadioML2018 dataset with a custom-generated dataset, featuring samples at Signal-to-Noise Ratios (SNRs) from 0 to 30dB. System performance was evaluated using accuracy, precision, recall, F1 score, and the Area Under the Receiver Operating Characteristic Curve (AUC-ROC). The optimized model achieved 93.48% accuracy, 93.53% precision, 93.48% recall, and an F1 score of 93.45%. The AUC-ROC analysis confirmed the model's discriminative power, even in noisy conditions. This paper's experimental results validate the effectiveness of the hybrid CNN-LSTM architecture for AMC, suggesting its potential application in adaptive spectrum management and advanced cognitive radio systems.
Authors Dinanath Padhya, Krishna Acharya, Bipul Kumar Dahal, Dinesh Baniya Kshatri
Comment 8 Pages, 10 figures, 2 Tables, Accepted in Journal (Journal of Innovations in Engineering Education), Issue is not Published Yet
Categories Computer Vision and Pattern Recognition
0
GFT-GCN: Privacy-Preserving 3D Face Mesh Recognition with Spectral Diffusion WACV 2026
Title CN GFT-GCN：基于谱扩散的隐私保护3D人脸网格识别
Keywords 3D人脸识别 · 隐私保护 · 图卷积网络 · 谱图理论 · 模板安全
Summary 本文提出了一种名为GFT-GCN的隐私保护3D人脸识别框架，结合谱图学习与基于扩散的模板保护技术。通过图傅里叶变换和图卷积网络从3D人脸网格中提取紧凑且具有判别性的谱特征，并引入谱扩散机制生成不可逆、可更新且不可链接的生物特征模板。系统采用轻量级客户端-服务器架构，确保原始生物数据不离开客户端设备。在BU-3DFE和FaceScape数据集上的实验表明该方法在保持高识别精度的同时，具备强抗重建攻击能力。
Reason 该论文研究的是3D人脸识别与隐私保护，属于生物特征识别领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其核心技术如图傅里叶变换、图卷积网络和谱扩散机制，虽然在图形处理上有价值，但难以迁移到解决文档理解中的OCR幻觉、密集文本识别、复杂版式分析或长文档推理等核心问题。
Abstract 3D face recognition offers a robust biometric solution by capturing facial geometry, providing resilience to variations in illumination, pose changes, and presentation attacks. Its strong spoof resistance makes it suitable for high-security applications, but protecting stored biometric templates remains critical. We present GFT-GCN, a privacy-preserving 3D face recognition framework that combines spectral graph learning with diffusion-based template protection. Our approach integrates the Graph Fourier Transform (GFT) and Graph Convolutional Networks (GCN) to extract compact, discriminative spectral features from 3D face meshes. To secure these features, we introduce a spectral diffusion mechanism that produces irreversible, renewable, and unlinkable templates. A lightweight client-server architecture ensures that raw biometric data never leaves the client device. Experiments on the BU-3DFE and FaceScape datasets demonstrate high recognition accuracy and strong resistance to reconstruction attacks. Results show that GFT-GCN effectively balances privacy and performance, offering a practical solution for secure 3D face authentication.
Authors Hichem Felouat, Hanrui Wang, Isao Echizen
Comment 13 pages, 8 figures, WACV 2026
Categories Computer Vision and Pattern Recognition
0
FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generation
Title CN FlowerDance：用于高效且精细的3D舞蹈生成的MeanFlow
Keywords 3D舞蹈生成 · 音乐驱动动作 · 生成效率 · 物理一致性 · BiMamba
Summary 本文提出FlowerDance，一种高效的3D舞蹈生成模型，结合MeanFlow和物理一致性约束，实现高质量、高效率的动作生成。模型采用BiMamba结构和通道级跨模态融合，支持非自回归生成和交互式编辑，在AIST++和FineDance数据集上表现优异。
Reason 该论文研究的是音乐到舞蹈的3D动作生成，属于具身智能与视频生成领域，与文档图像理解（DIU）或视觉语言模型在文档任务中的应用完全无关。其核心技术创新集中在舞蹈动作的物理合理性、生成效率和跨模态融合，无法迁移解决OCR幻觉、复杂排版理解、细粒度定位或长文档推理等DIU核心问题。
Abstract Music-to-dance generation aims to translate auditory signals into expressive human motion, with broad applications in virtual reality, choreography, and digital entertainment. Despite promising progress, the limited generation efficiency of existing methods leaves insufficient computational headroom for high-fidelity 3D rendering, thereby constraining the expressiveness of 3D characters during real-world applications. Thus, we propose FlowerDance, which not only generates refined motion with physical plausibility and artistic expressiveness, but also achieves significant generation efficiency on inference speed and memory utilization . Specifically, FlowerDance combines MeanFlow with Physical Consistency Constraints, which enables high-quality motion generation with only a few sampling steps. Moreover, FlowerDance leverages a simple but efficient model architecture with BiMamba-based backbone and Channel-Level Cross-Modal Fusion, which generates dance with efficient non-autoregressive manner. Meanwhile, FlowerDance supports motion editing, enabling users to interactively refine dance sequences. Extensive experiments on AIST++ and FineDance show that FlowerDance achieves state-of-the-art results in both motion quality and generation efficiency. Code will be released upon acceptance.
Authors Kaixing Yang, Xulong Tang, Ziqiao Peng, Xiangyue Zhang, Puwei Wang, Jun He, Hongyan Liu
Categories Computer Vision and Pattern Recognition
0
LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Training
Title CN LTD：用于无梯度掩蔽对抗训练的低温蒸馏
Keywords 对抗训练 · 知识蒸馏 · 图像分类 · 模型鲁棒性 · 梯度掩蔽
Summary 本文提出了一种名为低温蒸馏（LTD）的新方法，用于提升神经网络在图像分类任务中的对抗鲁棒性。该方法通过在教师模型中使用较低温度进行知识蒸馏，而保持学生模型的温度不变，从而优化数据分布假设并避免梯度掩蔽问题。实验表明该方法在多个标准图像分类数据集上有效提升了鲁棒准确性。
Reason 该论文研究的是图像分类任务中的对抗训练与模型鲁棒性问题，核心是通过低温蒸馏改进标签表示以缓解梯度掩蔽问题。其研究目标与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、细粒度定位、长文档推理等）无直接关联。方法论上属于通用视觉模型的安全与鲁棒性范畴，无法迁移用于解决文档理解中的结构解析、多模态对齐或高分辨率文本识别等问题。
Abstract Adversarial training is a widely adopted strategy to bolster the robustness of neural network models against adversarial attacks. This paper revisits the fundamental assumptions underlying image classification and suggests that representing data as one-hot labels is a key factor that leads to vulnerabilities. However, in real-world datasets, data ambiguity often arises, with samples exhibiting characteristics of multiple classes, rendering one-hot label representations imprecise. To address this, we introduce a novel approach, Low-Temperature Distillation (LTD), designed to refine label representations. Unlike previous approaches, LTD incorporates a relatively low temperature in the teacher model, while maintaining a fixed temperature for the student model during both training and inference. This strategy not only refines assumptions about data distribution but also strengthens model robustness and avoids the gradient masking problem commonly encountered in defensive distillation. Experimental results demonstrate the efficacy of the proposed method when combined with existing frameworks, achieving robust accuracy rates of 58.19%, 31.13%, and 42.08% on the CIFAR-10, CIFAR-100, and ImageNet datasets, respectively, without the need for additional data.
Authors Erh-Chung Chen, Che-Rung Lee
Categories Computer Vision and Pattern Recognition
0
CameraMaster: Unified Camera Semantic-Parameter Control for Photography Retouching
Title CN CameraMaster：用于摄影润饰的统一相机语义-参数控制
Keywords 图像编辑 · 扩散模型 · 相机参数控制 · 文本到图像 · 语义对齐
Summary 本文提出CameraMaster，一种用于摄影润饰的统一框架，通过解耦相机指令并融合摄影师意图与精确相机参数嵌入，在扩散模型中实现物理一致且精细可控的图像编辑。该方法在大规模带相机参数标注的数据集上训练，支持多参数组合与线性响应调节。然而，其应用领域为图像生成与美化，远离文档理解范畴。
Reason 该论文研究的是基于文本引导的图像生成与编辑，聚焦于摄影后期处理中的相机参数（如曝光、白平衡）精确控制问题。属于纯图像生成/编辑领域，与文档图像理解（DIU）、DocVQA、OCR、排版分析等任务无直接关联。尽管使用了多模态输入（文本+参数），但其目标和方法均不涉及文档结构、文本识别或视觉语言推理，无法迁移解决OCR幻觉、复杂布局解析或长文档理解等核心DIU痛点。
Abstract Text-guided diffusion models have greatly advanced image editing and generation. However, achieving physically consistent image retouching with precise parameter control (e.g., exposure, white balance, zoom) remains challenging. Existing methods either rely solely on ambiguous and entangled text prompts, which hinders precise camera control, or train separate heads/weights for parameter adjustment, which compromises scalability, multi-parameter composition, and sensitivity to subtle variations. To address these limitations, we propose CameraMaster, a unified camera-aware framework for image retouching. The key idea is to explicitly decouple the camera directive and then coherently integrate two critical information streams: a directive representation that captures the photographer's intent, and a parameter embedding that encodes precise camera settings. CameraMaster first uses the camera parameter embedding to modulate both the camera directive and the content semantics. The modulated directive is then injected into the content features via cross-attention, yielding a strongly camera-sensitive semantic context. In addition, the directive and camera embeddings are injected as conditioning and gating signals into the time embedding, enabling unified, layer-wise modulation throughout the denoising process and enforcing tight semantic-parameter alignment. To train and evaluate CameraMaster, we construct a large-scale dataset of 78K image-prompt pairs annotated with camera parameters. Extensive experiments show that CameraMaster produces monotonic and near-linear responses to parameter variations, supports seamless multi-parameter composition, and significantly outperforms existing methods.
Authors Qirui Yang, Yang Yang, Ying Zeng, Xiaobin Hu, Bo Li, Huanjing Yue, Jingyu Yang, Peng-Tao Jiang
Categories Computer Vision and Pattern Recognition
0
STARFlow-V: End-to-End Video Generative Modeling with Normalizing Flows
Title CN STARFlow-V：基于归一化流的端到端视频生成建模
Keywords 视频生成 · 归一化流 · 因果建模 · 潜变量模型 · 生成模型
Summary 本文提出了STARFlow-V，一种基于归一化流的端到端视频生成模型，通过全局-局部架构在时空潜在空间中建模，缓解了时间维度上的误差累积问题，并引入flow-score matching和视频感知的Jacobi迭代方案以提升生成一致性与采样效率。模型支持文本到视频、图像到视频等多种生成任务，在视觉保真度和时间一致性方面表现良好。
Reason 该论文属于纯视频生成领域，研究基于归一化流的视频生成模型，与文档图像理解（DIU）或DocVQA等任务无直接关联。尽管其方法涉及高分辨率时空建模和因果推理，但核心目标是视频生成质量与效率，无法迁移用于解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点。
Abstract Normalizing flows (NFs) are end-to-end likelihood-based generative models for continuous data, and have recently regained attention with encouraging progress on image generation. Yet in the video generation domain, where spatiotemporal complexity and computational cost are substantially higher, state-of-the-art systems almost exclusively rely on diffusion-based models. In this work, we revisit this design space by presenting STARFlow-V, a normalizing flow-based video generator with substantial benefits such as end-to-end learning, robust causal prediction, and native likelihood estimation. Building upon the recently proposed STARFlow, STARFlow-V operates in the spatiotemporal latent space with a global-local architecture which restricts causal dependencies to a global latent space while preserving rich local within-frame interactions. This eases error accumulation over time, a common pitfall of standard autoregressive diffusion model generation. Additionally, we propose flow-score matching, which equips the model with a light-weight causal denoiser to improve the video generation consistency in an autoregressive fashion. To improve the sampling efficiency, STARFlow-V employs a video-aware Jacobi iteration scheme that recasts inner updates as parallelizable iterations without breaking causality. Thanks to the invertible structure, the same model can natively support text-to-video, image-to-video as well as video-to-video generation tasks. Empirically, STARFlow-V achieves strong visual fidelity and temporal consistency with practical sampling throughput relative to diffusion-based baselines. These results present the first evidence, to our knowledge, that NFs are capable of high-quality autoregressive video generation, establishing them as a promising research direction for building world models. Code and generated samples are available at https://github.com/apple/ml-starflow.
Authors Jiatao Gu, Ying Shen, Tianrong Chen, Laurent Dinh, Yuyang Wang, Miguel Angel Bautista, David Berthelot, Josh Susskind, Shuangfei Zhai
Comment 21 pages, 9 figures. Code and samples are available at https://github.com/apple/ml-starflow
Categories Computer Vision and Pattern Recognition
0
Class-Independent Increment: An Efficient Approach for Multi-label Class-Incremental Learning
Title CN 类无关增量：一种高效的多标签类增量学习方法
Keywords 类增量学习 · 多标签分类 · 特征混淆 · 类特定token · 灾难性遗忘
Summary 本文提出了一种类无关增量方法（CLIN），通过构建类特定token来提取多标签样本的类级别嵌入，以缓解多标签类增量学习中的灾难性遗忘和特征混淆问题。实验表明该方法在MS-COCO和PASCAL VOC数据集上有效提升了识别性能。
Reason 该论文研究的是多标签类增量学习（MLCIL），属于通用图像分类任务的扩展，主要解决的是类别遗忘和特征混淆问题。其核心方法是提取类级别的嵌入并设计类特定token，但应用场景为自然图像（如MS-COCO、PASCAL VOC），与文档图像理解（DIU）中的核心痛点（如OCR幻觉、复杂排版、细粒度文本识别、长文档推理等）无直接关联。未涉及多模态大模型、视觉-语言对齐、高分辨率文档处理或文档结构建模，无法迁移用于提升DocVQA或文档理解性能。
Abstract Current research on class-incremental learning primarily focuses on single-label classification tasks. However, real-world applications often involve multi-label scenarios, such as image retrieval and medical imaging. Therefore, this paper focuses on the challenging yet practical multi-label class-incremental learning (MLCIL) problem. In addition to the challenge of catastrophic forgetting, MLCIL encounters issues related to feature confusion, encompassing inter-session and intra-feature confusion. To address these problems, we propose a novel MLCIL approach called class-independent increment (CLIN). Specifically, in contrast to existing methods that extract image-level features, we propose a class-independent incremental network (CINet) to extract multiple class-level embeddings for multi-label samples. It learns and preserves the knowledge of different classes by constructing class-specific tokens. On this basis, we develop two novel loss functions, optimizing the learning of class-specific tokens and class-level embeddings, respectively. These losses aim to distinguish between new and old classes, further alleviating the problem of feature confusion. Extensive experiments on MS-COCO and PASCAL VOC datasets demonstrate the effectiveness of our method for improving recognition performance and mitigating forgetting on various MLCIL tasks.
Authors Chenhao Ding, Songlin Dong, Zhengdong Zhou, Jizhou Han, Qiang Wang, Yuhang He, Yihong Gong
Categories Computer Vision and Pattern Recognition
0
Wavefront-Constrained Passive Obscured Object Detection
Title CN 基于波前约束的被动遮挡物体检测
Keywords 计算成像 · 非视域检测 · 波前传播 · 复振幅建模 · 遮挡物体分割
Summary 本文提出WavePCNet，一种基于物理驱动的网络模型，用于在非视域条件下通过微弱光模式检测和分割被遮挡的物体。该方法引入TriWCP模块以复振幅传输算子精确约束相干光传播行为，并结合动量记忆机制抑制扰动累积，同时通过跨层高频补偿增强结构一致性建模。实验表明该方法在四个真实采集的数据集上优于现有技术。
Reason 该论文研究的是通过非视域光信号检测遮挡物体，属于计算成像与物理建模领域，核心任务为光学散射环境下的目标定位与分割。其技术路线聚焦于波前传播建模、复振幅算子和高频补偿机制，与文档图像理解（DIU）中的文本识别、布局分析、VQA等任务无直接关联。方法难以迁移到解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点，且不属于任何可复用的多模态基础架构或推理机制创新。
Abstract Accurately localizing and segmenting obscured objects from faint light patterns beyond the field of view is highly challenging due to multiple scattering and medium-induced perturbations. Most existing methods, based on real-valued modeling or local convolutional operations, are inadequate for capturing the underlying physics of coherent light propagation. Moreover, under low signal-to-noise conditions, these methods often converge to non-physical solutions, severely compromising the stability and reliability of the observation. To address these challenges, we propose a novel physics-driven Wavefront Propagating Compensation Network (WavePCNet) to simulate wavefront propagation and enhance the perception of obscured objects. This WavePCNet integrates the Tri-Phase Wavefront Complex-Propagation Reprojection (TriWCP) to incorporate complex amplitude transfer operators to precisely constrain coherent propagation behavior, along with a momentum memory mechanism to effectively suppress the accumulation of perturbations. Additionally, a High-frequency Cross-layer Compensation Enhancement is introduced to construct frequency-selective pathways with multi-scale receptive fields and dynamically model structural consistency across layers, further boosting the model's robustness and interpretability under complex environmental conditions. Extensive experiments conducted on four physically collected datasets demonstrate that WavePCNet consistently outperforms state-of-the-art methods across both accuracy and robustness.
Authors Zhiwen Zheng, Yiwei Ouyang, Zhao Huang, Tao Zhang, Xiaoshuai Zhang, Huiyu Zhou, Wenwen Tang, Shaowei Jiang, Jin Liu, Xingru Huang
Categories Computer Vision and Pattern Recognition
0
RefOnce: Distilling References into a Prototype Memory for Referring Camouflaged Object Detection
Title CN RefOnce：将参考图像蒸馏为原型记忆用于指代表意伪装物体检测
Keywords 伪装物体检测 · 原型记忆 · 无参考推理 · 注意力对齐 · 多模态分割
Summary 本文提出RefOnce，一种用于指代表意伪装物体检测的新框架，通过在训练阶段将参考图像信息蒸馏到类原型记忆中，在推理时无需输入参考图像，仅通过查询条件生成指导向量完成检测。该方法减少了部署负担，并通过双向注意力对齐模块缩小了伪装查询特征与参考表示之间的差距。在R2C7K数据集上验证了其有效性。
Reason 该论文研究的是指代表意伪装物体检测（Ref-COD），属于通用图像理解中的特定视觉任务，核心问题与文档图像理解（DIU）无关。尽管其提出的方法涉及测试时无需参考图像的部署优化，但应用场景聚焦于自然图像中的伪装物体分割，未涉及OCR、文本布局、表格结构或文档幻觉等DIU核心痛点。此外，方法依赖于类别级原型记忆和查询条件混合机制，迁移到文档理解场景（如处理密集文本或多模态对齐）的路径不明确，不具备直接的方法论迁移价值。
Abstract Referring Camouflaged Object Detection (Ref-COD) segments specified camouflaged objects in a scene by leveraging a small set of referring images. Though effective, current systems adopt a dual-branch design that requires reference images at test time, which limits deployability and adds latency and data-collection burden. We introduce a Ref-COD framework that distills references into a class-prototype memory during training and synthesizes a reference vector at inference via a query-conditioned mixture of prototypes. Concretely, we maintain an EMA-updated prototype per category and predict mixture weights from the query to produce a guidance vector without any test-time references. To bridge the representation gap between reference statistics and camouflaged query features, we propose a bidirectional attention alignment module that adapts both the query features and the class representation. Thus, our approach yields a simple, efficient path to Ref-COD without mandatory references. We evaluate the proposed method on the large-scale R2C7K benchmark. Extensive experiments demonstrate competitive or superior performance of the proposed method compared with recent state-of-the-arts. Code is available at https://github.com/yuhuan-wu/RefOnce.
Authors Yu-Huan Wu, Zi-Xuan Zhu, Yan Wang, Liangli Zhen, Deng-Ping Fan
Comment 11 pages, 5 figure, 6 tables
Categories Computer Vision and Pattern Recognition
0
Benchmarking the Trustworthiness in Multimodal LLMs for Video Understanding
Title CN 视频理解中多模态大语言模型可信度的基准测试
Keywords 多模态大语言模型 · 视频理解 · 可信度 · 幻觉 · 基准测试
Summary 本文提出了Trust-videoLLMs，一个针对视频理解中多模态大语言模型的综合性可信度评估基准，涵盖真实性、鲁棒性、安全性、公平性和隐私五个维度，使用30项任务在23个主流videoLLM上进行评测。结果显示现有模型在动态场景理解、跨模态抗干扰等方面存在明显缺陷，开源与闭源模型各有优劣，模型规模扩大并未持续提升可信度。
Reason 该论文聚焦于视频理解领域的多模态大语言模型（videoLLMs）的可信度问题，涉及任务如事实准确性、安全性、公平性等。尽管其关注的‘幻觉’问题是文档图像理解（DIU）中的痛点之一，但整体研究场景完全基于视频数据，包括时空一致性、动态场景理解和跨模态扰动鲁棒性，这些与文档图像的静态、结构化特性差异巨大。所提出的Trust-videoLLMs基准和评估框架难以直接迁移至DocVQA或OCR相关任务，且未涉及文档特有的挑战如复杂排版、密集文本或视觉token压缩。属于无关领域（视频理解），不符合DIU本题要求，也不提供可迁移的底层方法。
Abstract Recent advancements in multimodal large language models for video understanding (videoLLMs) have enhanced their capacity to process complex spatiotemporal data. However, challenges such as factual inaccuracies, harmful content, biases, hallucinations, and privacy risks compromise their reliability. This study introduces Trust-videoLLMs, a first comprehensive benchmark evaluating 23 state-of-the-art videoLLMs (5 commercial, 18 open-source) across five critical dimensions: truthfulness, robustness, safety, fairness, and privacy. Comprising 30 tasks with adapted, synthetic, and annotated videos, the framework assesses spatiotemporal risks, temporal consistency and cross-modal impact. Results reveal significant limitations in dynamic scene comprehension, cross-modal perturbation resilience and real-world risk mitigation. While open-source models occasionally outperform, proprietary models generally exhibit superior credibility, though scaling does not consistently improve performance. These findings underscore the need for enhanced training datat diversity and robust multimodal alignment. Trust-videoLLMs provides a publicly available, extensible toolkit for standardized trustworthiness assessments, addressing the critical gap between accuracy-focused benchmarks and demands for robustness, safety, fairness, and privacy.
Authors Youze Wang, Zijun Chen, Ruoyu Chen, Shishen Gu, Wenbo Hu, Jiayang Liu, Yinpeng Dong, Hang Su, Jun Zhu, Meng Wang, Richang Hong
Categories Computer Vision and Pattern Recognition
0
DEMIST: Decoupled Multi-stream latent diffusion for Quantitative Myelin Map Synthesis
Title CN DEMIST：用于定量髓鞘图合成的解耦多流潜在扩散模型
Keywords 医学图像合成 · 扩散模型 · 多模态条件生成 · MRI · 髓鞘成像
Summary 本文提出DEMIST，一种基于3D潜在扩散模型的方法，通过解耦语义、空间和自适应三种条件机制，从常规MRI图像合成定量髓鞘图。方法在保持低训练参数的同时提升了生成图像的边界清晰度与定量准确性。实验验证于多发性硬化症患者数据集上，优于VAE、GAN和扩散基线模型。
Reason 该论文研究的是医学图像合成任务，具体为从T1w和FLAIR图像合成定量髓鞘图（PSR），属于医疗影像分析领域。其核心技术是3D潜在扩散模型与多条件控制机制，应用于神经影像中的病灶边界保持与定量一致性优化。尽管使用了先进的视觉生成架构，但其目标、数据与方法均远离文档图像理解（DIU）的核心问题，如文本识别、布局分析、表格理解或OCR去幻觉等。无任何可迁移至文档理解场景的方法论创新。
Abstract Quantitative magnetization transfer (qMT) imaging provides myelin-sensitive biomarkers, such as the pool size ratio (PSR), which is valuable for multiple sclerosis (MS) assessment. However, qMT requires specialized 20-30 minute scans. We propose DEMIST to synthesize PSR maps from standard T1w and FLAIR images using a 3D latent diffusion model with three complementary conditioning mechanisms. Our approach has two stages: first, we train separate autoencoders for PSR and anatomical images to learn aligned latent representations. Second, we train a conditional diffusion model in this latent space on top of a frozen diffusion foundation backbone. Conditioning is decoupled into: (i) \textbf{semantic} tokens via cross-attention, (ii) \textbf{spatial} per-scale residual hints via a 3D ControlNet branch, and (iii) \textbf{adaptive} LoRA-modulated attention. We include edge-aware loss terms to preserve lesion boundaries and alignment losses to maintain quantitative consistency, while keeping the number of trainable parameters low and retaining the inductive bias of the pretrained model. We evaluate on 163 scans from 99 subjects using 5-fold cross-validation. Our method outperforms VAE, GAN and diffusion baselines on multiple metrics, producing sharper boundaries and better quantitative agreement with ground truth. Our code is publicly available at https://github.com/MedICL-VU/MS-Synthesis-3DcLDM.
Authors Jiacheng Wang, Hao Li, Xing Yao, Ahmad Toubasi, Taegan Vinarsky, Caroline Gheen, Joy Derwenskus, Chaoyang Jin, Richard Dortch, Junzhong Xu, Francesca Bagnato, Ipek Oguz
Categories Computer Vision and Pattern Recognition
0
Inversion-Free Style Transfer with Dual Rectified Flows
Title CN 基于双校正流的无反演风格迁移
Keywords 风格迁移 · 扩散模型 · 无反演 · 双校正流 · 图像生成
Summary 本文提出一种无需反演的风格迁移框架，基于双校正流机制，在仅有前向传播的情况下实现内容与风格的高效融合。通过并行预测内容和风格轨迹，并在动态中点插值中结合两者速度场，避免了传统扩散模型依赖反演带来的计算开销与视觉失真。注意力注入进一步提升了风格保真度与内容一致性。
Reason 该论文研究的是通用图像风格迁移问题，属于纯图像生成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心技术——双校正流用于前向风格融合——主要服务于艺术化图像合成，无法迁移解决OCR幻觉、复杂排版解析、细粒度文本定位或长文档推理等DIU核心痛点。
Abstract Style transfer, a pivotal task in image processing, synthesizes visually compelling images by seamlessly blending realistic content with artistic styles, enabling applications in photo editing and creative design. While mainstream training-free diffusion-based methods have greatly advanced style transfer in recent years, their reliance on computationally inversion processes compromises efficiency and introduces visual distortions when inversion is inaccurate. To address these limitations, we propose a novel \textit{inversion-free} style transfer framework based on dual rectified flows, which tackles the challenge of finding an unknown stylized distribution from two distinct inputs (content and style images), \textit{only with forward pass}. Our approach predicts content and style trajectories in parallel, then fuses them through a dynamic midpoint interpolation that integrates velocities from both paths while adapting to the evolving stylized image. By jointly modeling the content, style, and stylized distributions, our velocity field design achieves robust fusion and avoids the shortcomings of naive overlays. Attention injection further guides style integration, enhancing visual fidelity, content preservation, and computational efficiency. Extensive experiments demonstrate generalization across diverse styles and content, providing an effective and efficient pipeline for style transfer.
Authors Yingying Deng, Xiangyu He, Fan Tang, Weiming Dong, Xucheng Yin
Categories Computer Vision and Pattern Recognition
0
Privacy-Preserving Federated Vision Transformer Learning Leveraging Lightweight Homomorphic Encryption in Medical AI
Title CN 利用轻量级同态加密在医学AI中实现隐私保护的联邦视觉Transformer学习
Keywords 联邦学习 · 同态加密 · Vision Transformer · 隐私保护 · 医学图像
Summary 本文提出一种结合Vision Transformer与同态加密的联邦学习框架，用于保护医疗图像协作训练中的患者隐私。通过仅加密ViT输出的CLS token而非全部梯度，大幅降低通信开销，并有效抵御模型反演攻击，在肺癌组织病理学分类任务中实现了良好的准确率与安全性平衡。
Reason 该论文属于医疗领域的联邦学习与隐私保护研究，核心贡献在于结合ViT的CLS token和同态加密来防御梯度泄露攻击。尽管使用了Vision Transformer，但其目标是医学图像分类中的隐私安全，而非文档图像理解（DIU）任务如DocVQA、版面分析或表格识别。方法聚焦于通信效率与加密安全性，无法直接迁移解决OCR幻觉、复杂排版解析或长文档推理等DIU核心痛点，属于无关领域应用。
Abstract Collaborative machine learning across healthcare institutions promises improved diagnostic accuracy by leveraging diverse datasets, yet privacy regulations such as HIPAA prohibit direct patient data sharing. While federated learning (FL) enables decentralized training without raw data exchange, recent studies show that model gradients in conventional FL remain vulnerable to reconstruction attacks, potentially exposing sensitive medical information. This paper presents a privacy-preserving federated learning framework combining Vision Transformers (ViT) with homomorphic encryption (HE) for secure multi-institutional histopathology classification. The approach leverages the ViT CLS token as a compact 768-dimensional feature representation for secure aggregation, encrypting these tokens using CKKS homomorphic encryption before transmission to the server. We demonstrate that encrypting CLS tokens achieves a 30-fold communication reduction compared to gradient encryption while maintaining strong privacy guarantees. Through evaluation on a three-client federated setup for lung cancer histopathology classification, we show that gradients are highly susceptible to model inversion attacks (PSNR: 52.26 dB, SSIM: 0.999, NMI: 0.741), enabling near-perfect image reconstruction. In contrast, the proposed CLS-protected HE approach prevents such attacks while enabling encrypted inference directly on ciphertexts, requiring only 326 KB of encrypted data transmission per aggregation round. The framework achieves 96.12 percent global classification accuracy in the unencrypted domain and 90.02 percent in the encrypted domain.
Authors Al Amin, Kamrul Hasan, Liang Hong, Sharif Ullah
Comment 7 pages, 4 figures
Categories Computer Vision and Pattern Recognition
0
Vision-Language Enhanced Foundation Model for Semi-supervised Medical Image Segmentation
Title CN 用于半监督医学图像分割的视觉-语言增强基础模型
Keywords 医学图像分割 · 半监督学习 · 视觉语言模型 · VLM · 伪标签
Summary 本文提出VESSA模型，将视觉-语言模型引入半监督医学图像分割中，通过模板库中的标准样本作为参考，利用视觉特征匹配生成语义和空间线索，并结合类似SAM2的掩码解码器进行分割。在第二阶段，学生模型的预测结果反馈给VESSA以生成更高质量的伪标签，从而提升整体分割性能。实验表明该方法在标注数据极少的情况下显著优于现有方法。
Reason 该论文研究的是医学图像分割任务，属于纯医疗影像分析领域，且使用VLM辅助的是通用图像分割（如SAM2架构），与文档图像理解（DIU）中的DocVQA、布局分析、表格识别等核心任务无直接关联。虽然使用了VLM和半监督学习框架，但其方法针对的是生物医学图像中的器官或病变区域分割，无法迁移解决文档图像中的OCR幻觉、密集文本解析、复杂排版理解等关键痛点。
Abstract Semi-supervised learning (SSL) has emerged as an effective paradigm for medical image segmentation, reducing the reliance on extensive expert annotations. Meanwhile, vision-language models (VLMs) have demonstrated strong generalization and few-shot capabilities across diverse visual domains. In this work, we integrate VLM-based segmentation into semi-supervised medical image segmentation by introducing a Vision-Language Enhanced Semi-supervised Segmentation Assistant (VESSA) that incorporates foundation-level visual-semantic understanding into SSL frameworks. Our approach consists of two stages. In Stage 1, the VLM-enhanced segmentation foundation model VESSA is trained as a reference-guided segmentation assistant using a template bank containing gold-standard exemplars, simulating learning from limited labeled data. Given an input-template pair, VESSA performs visual feature matching to extract representative semantic and spatial cues from exemplar segmentations, generating structured prompts for a SAM2-inspired mask decoder to produce segmentation masks. In Stage 2, VESSA is integrated into a state-of-the-art SSL framework, enabling dynamic interaction with the student model: as student predictions become more refined, they are fed back to VESSA as prompts, allowing it to generate higher-quality pseudo-labels and stronger guidance. Extensive experiments across multiple segmentation datasets and domains show that VESSA-augmented SSL significantly enhances segmentation accuracy, outperforming state-of-the-art baselines under extremely limited annotation conditions.
Authors Jiaqi Guo, Mingzhen Li, Hanyu Su, Santiago López, Lexiaozi Fan, Daniel Kim, Aggelos Katsaggelos
Categories Computer Vision and Pattern Recognition
0
Bayesian Neural Networks for One-to-Many Mapping in Image Enhancement
Title CN 用于图像增强中一对多映射的贝叶斯神经网络
Keywords 图像增强 · 贝叶斯神经网络 · 一对多映射 · 不确定性建模 · BNN-DNN
Summary 本文提出一种基于贝叶斯神经网络（BNN）的图像增强模型（BEM），用于建模退化图像到多个合理增强结果之间的一对多映射关系，并通过BNN-DNN级联框架实现快速推理。实验表明该方法在多个低光和水下图像增强数据集上有效。
Reason 该论文研究的是低光和水下图像增强中的一对多映射问题，属于纯图像生成与增强领域，其方法目标是提升视觉感知质量或生成多样性结果。这与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管贝叶斯建模可能在理论上涉及不确定性处理，但其应用场景聚焦于通用图像增强，无法迁移解决OCR幻觉、复杂排版理解或长文档推理等DIU关键痛点。
Abstract In image enhancement tasks, such as low-light and underwater image enhancement, a degraded image can correspond to multiple plausible target images due to dynamic photography conditions. This naturally results in a one-to-many mapping problem. To address this, we propose a Bayesian Enhancement Model (BEM) that incorporates Bayesian Neural Networks (BNNs) to capture data uncertainty and produce diverse outputs. To enable fast inference, we introduce a BNN-DNN framework: a BNN is first employed to model the one-to-many mapping in a low-dimensional space, followed by a Deterministic Neural Network (DNN) that refines fine-grained image details. Extensive experiments on multiple low-light and underwater image enhancement benchmarks demonstrate the effectiveness of our method.
Authors Guoxi Huang, Qirui Yang, Ruirui Lin, Zipeng Qi, David Bull, Nantheera Anantrasirichai
Categories Computer Vision and Pattern Recognition
0
TrafficLens: Multi-Camera Traffic Video Analysis Using LLMs 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC)
Title CN TrafficLens：使用大语言模型进行多摄像头交通视频分析
Keywords 交通视频分析 · 大语言模型 · 视觉语言模型 · 多摄像头 · RAG
Summary 本文提出TrafficLens，一种用于多摄像头交通视频分析的算法，利用VLM对多个摄像头视频进行顺序处理，并通过对象级相似性检测跳过冗余帧，以加速视频到文本的转换过程。实验表明该方法可将处理时间缩短4倍，同时保持信息准确性。
Reason 该论文属于视频理解领域的垂直应用，聚焦于交通监控场景下的多摄像头视频分析，本质上是将VLM与RAG结合用于特定下游任务（交通事件描述生成），并未提出可迁移至文档图像理解的底层方法。其核心贡献——通过重叠区域顺序处理和对象级相似性检测减少VLM调用——针对的是视频流冗余问题，难以迁移到文档理解中的OCR幻觉、复杂版式或长文档推理等核心痛点。属于典型的‘平行应用’，无DIU方法论创新。
Abstract Traffic cameras are essential in urban areas, playing a crucial role in intelligent transportation systems. Multiple cameras at intersections enhance law enforcement capabilities, traffic management, and pedestrian safety. However, efficiently managing and analyzing multi-camera feeds poses challenges due to the vast amount of data. Analyzing such huge video data requires advanced analytical tools. While Large Language Models (LLMs) like ChatGPT, equipped with retrieval-augmented generation (RAG) systems, excel in text-based tasks, integrating them into traffic video analysis demands converting video data into text using a Vision-Language Model (VLM), which is time-consuming and delays the timely utilization of traffic videos for generating insights and investigating incidents. To address these challenges, we propose TrafficLens, a tailored algorithm for multi-camera traffic intersections. TrafficLens employs a sequential approach, utilizing overlapping coverage areas of cameras. It iteratively applies VLMs with varying token limits, using previous outputs as prompts for subsequent cameras, enabling rapid generation of detailed textual descriptions while reducing processing time. Additionally, TrafficLens intelligently bypasses redundant VLM invocations through an object-level similarity detector. Experimental results with real-world datasets demonstrate that TrafficLens reduces video-to-text conversion time by up to $4\times$ while maintaining information accuracy.
Authors Md Adnan Arefeen, Biplob Debnath, Srimat Chakradhar
Comment 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC)
Categories Computer Vision and Pattern Recognition, Computation and Language
0
Beyond Realism: Learning the Art of Expressive Composition with StickerNet
Title CN 超越真实感：通过StickerNet学习表现性构图的艺术
Keywords 图像合成 · 创意编辑 · 贴纸生成 · 用户行为建模 · 非真实感渲染
Summary 本文提出了‘表现性构图’这一新任务，旨在模仿在线内容创作平台上用户的非写实性图像编辑行为（如添加贴纸）。为此，作者构建了基于180万条真实用户编辑操作的数据集，并设计了StickerNet两阶段模型来预测贴纸的类型和放置参数。实验表明该模型能较好拟合人类的创意编辑行为。然而，这项工作本质上属于图像生成与人机交互范畴，不涉及文档结构、文本识别或视觉-语言推理。
Reason 该论文研究的是图像编辑中的表现性构图任务，聚焦于贴纸风格、透明度、位置等视觉表达元素的生成，属于图像生成与创意设计领域。其核心问题设定（艺术化、社交导向的图像合成）与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版面分析、长文档推理等完全无关。方法上虽提出两阶段框架StickerNet并构建真实用户行为数据集，但其目标是模拟用户在社交平台上的创造性粘贴行为，无法迁移用于解决文档中细粒度文字定位或多模态语义对齐问题。
Abstract As a widely used operation in image editing workflows, image composition has traditionally been studied with a focus on achieving visual realism and semantic plausibility. However, in practical editing scenarios of the modern content creation landscape, many compositions are not intended to preserve realism. Instead, users of online platforms motivated by gaining community recognition often aim to create content that is more artistic, playful, or socially engaging. Taking inspiration from this observation, we define the expressive composition task, a new formulation of image composition that embraces stylistic diversity and looser placement logic, reflecting how users edit images on real-world creative platforms. To address this underexplored problem, we present StickerNet, a two-stage framework that first determines the composition type, then predicts placement parameters such as opacity, mask, location, and scale accordingly. Unlike prior work that constructs datasets by simulating object placements on real images, we directly build our dataset from 1.8 million editing actions collected on an anonymous online visual creation and editing platform, each reflecting user-community validated placement decisions. This grounding in authentic editing behavior ensures strong alignment between task definition and training supervision. User studies and quantitative evaluations show that StickerNet outperforms common baselines and closely matches human placement behavior, demonstrating the effectiveness of learning from real-world editing patterns despite the inherent ambiguity of the task. This work introduces a new direction in visual understanding that emphasizes expressiveness and user intent over realism.
Authors Haoming Lu, David Kocharian, Humphrey Shi
Categories Computer Vision and Pattern Recognition
0
MoEGCL: Mixture of Ego-Graphs Contrastive Representation Learning for Multi-View Clustering
Title CN MoEGCL：用于多视图聚类的自我图混合对比表示学习
Keywords 图神经网络 · 多视图聚类 · 对比学习 · 自我图 · 专家混合
Summary 本文提出了一种名为MoEGCL的新方法，用于深度多视图聚类，通过构建自我图并采用专家混合网络实现样本级别的细粒度图融合，同时设计了自我图对比学习模块来增强聚类表示。实验表明其在多视图聚类任务上达到SOTA性能。
Reason 该论文研究的是多视图聚类中的图神经网络融合策略，属于通用图学习与聚类方法，未涉及文档图像、文本理解、OCR、布局分析或任何文档图像理解（DIU）相关任务。其核心创新点——自我图混合融合与对比学习——虽然在图结构建模上有一定细粒度优势，但与文档图像中的视觉排版、密集文本识别、多模态对齐、高分辨率处理等关键痛点无直接关联，且无法迁移用于解决DocVQA、表格识别或OCR-free理解等核心问题。
Abstract In recent years, the advancement of Graph Neural Networks (GNNs) has significantly propelled progress in Multi-View Clustering (MVC). However, existing methods face the problem of coarse-grained graph fusion. Specifically, current approaches typically generate a separate graph structure for each view and then perform weighted fusion of graph structures at the view level, which is a relatively rough strategy. To address this limitation, we present a novel Mixture of Ego-Graphs Contrastive Representation Learning (MoEGCL). It mainly consists of two modules. In particular, we propose an innovative Mixture of Ego-Graphs Fusion (MoEGF), which constructs ego graphs and utilizes a Mixture-of-Experts network to implement fine-grained fusion of ego graphs at the sample level, rather than the conventional view-level fusion. Additionally, we present the Ego Graph Contrastive Learning (EGCL) module to align the fused representation with the view-specific representation. The EGCL module enhances the representation similarity of samples from the same cluster, not merely from the same sample, further boosting fine-grained graph representation. Extensive experiments demonstrate that MoEGCL achieves state-of-the-art results in deep multi-view clustering tasks. The source code is publicly available at https://github.com/HackerHyper/MoEGCL.
Authors Jian Zhu, Xin Zou, Jun Sun, Cheng Luo, Lei Liu, Lingfang Zeng, Ning Zhang, Bian Wu, Chang Tang, Lirong Dai
Categories Computer Vision and Pattern Recognition
0
The author is dead, but what if they never lived? A reception experiment on Czech AI- and human-authored poetry
Title CN 作者已死，但如果他们从未活过呢？关于捷克语AI与人类创作诗歌的接受度实验
Keywords AI生成诗歌 · 接受度实验 · 作者身份偏见 · 小语种NLP · 美学评价
Summary 本文探讨了母语者对捷克语AI生成诗歌与人类创作诗歌的辨别能力及审美评价。实验显示参与者无法有效区分两者，且存在明显的作者身份偏见——当认为诗作为AI所写时评分更低。尽管如此，AI生成的诗歌实际获得的美学评分不低于人类作品。研究强调读者信念与审美判断之间的关联。
Reason 该论文研究的是AI生成诗歌在捷克语中的审美接受度和作者识别问题，属于小语种（非中英）的纯文本生成与人文评价范畴，不涉及文档图像、视觉信息处理或任何多模态理解任务。与文档图像理解（DIU）核心痛点如OCR幻觉、复杂排版、细粒度定位等完全无关，属于负面清单中的‘小语种’和‘无关领域’，无方法论迁移价值。
Abstract Large language models are increasingly capable of producing creative texts, yet most studies on AI-generated poetry focus on English -- a language that dominates training data. In this paper, we examine the perception of AI- and human-written Czech poetry. We ask if Czech native speakers are able to identify it and how they aesthetically judge it. Participants performed at chance level when guessing authorship (45.8\% correct on average), indicating that Czech AI-generated poems were largely indistinguishable from human-written ones. Aesthetic evaluations revealed a strong authorship bias: when participants believed a poem was AI-generated, they rated it as less favorably, even though AI poems were in fact rated equally or more favorably than human ones on average. The logistic regression model uncovered that the more the people liked a poem, the less probable was that they accurately assign the authorship. Familiarity with poetry or literary background had no effect on recognition accuracy. Our findings show that AI can convincingly produce poetry even in a morphologically complex, low-resource (with respect of the training data of AI models) Slavic language such as Czech. The results suggest that readers' beliefs about authorship and the aesthetic evaluation of the poem are interconnected.
Authors Anna Marklová, Ondřej Vinš, Martina Vokáčová, Jiří Milička
Categories Computation and Language
0
TAGFN: A Text-Attributed Graph Dataset for Fake News Detection in the Age of LLMs
Title CN TAGFN：面向LLM时代的带文本属性图的虚假新闻检测数据集
Keywords 虚假新闻检测 · 文本属性图 · 异常检测 · 数据集 · 大语言模型
Summary 本文提出了TAGFN，一个大规模的真实世界带文本属性的图数据集，用于假新闻检测中的异常检测任务。该数据集旨在推动基于图的异常检测方法以及利用大语言模型进行 misinformation 检测的研究。尽管涉及LLM和文本分析，但其目标领域为社交网络中的虚假信息识别，与文档图像理解无直接关联。
Reason 该论文属于平行下游应用，聚焦于虚假新闻检测这一特定任务，使用图结构和文本属性进行建模，并未提出可用于文档图像理解（如OCR去幻觉、复杂排版处理、细粒度定位等）的底层方法或架构创新。其核心贡献是数据集而非可迁移的方法论，且应用场景与文档理解无关。
Abstract Large Language Models (LLMs) have recently revolutionized machine learning on text-attributed graphs, but the application of LLMs to graph outlier detection, particularly in the context of fake news detection, remains significantly underexplored. One of the key challenges is the scarcity of large-scale, realistic, and well-annotated datasets that can serve as reliable benchmarks for outlier detection. To bridge this gap, we introduce TAGFN, a large-scale, real-world text-attributed graph dataset for outlier detection, specifically fake news detection. TAGFN enables rigorous evaluation of both traditional and LLM-based graph outlier detection methods. Furthermore, it facilitates the development of misinformation detection capabilities in LLMs through fine-tuning. We anticipate that TAGFN will be a valuable resource for the community, fostering progress in robust graph-based outlier detection and trustworthy AI. The dataset is publicly available at https://huggingface.co/datasets/kayzliu/TAGFN and our code is available at https://github.com/kayzliu/tagfn.
Authors Kay Liu, Yuwei Han, Haoyan Xu, Henry Peng Zou, Yue Zhao, Philip S. Yu
Comment Preprint. Under review
Categories Computation and Language
0
Leveraging Test Driven Development with Large Language Models for Reliable and Verifiable Spreadsheet Code Generation: A Research Framework
Title CN 利用大语言模型结合测试驱动开发的可靠且可验证的电子表格代码生成：一项研究框架
Keywords 大语言模型 · 测试驱动开发 · 代码生成 · 电子表格 · 可靠性
Summary 本文提出将测试驱动开发（TDD）与大语言模型结合，用于提高电子表格代码生成的正确性和可信度。通过‘先写测试’的方法为LLM提供约束和引导，以减少幻觉和逻辑错误。框架适用于Python、Rust等编程语言及Excel公式生成，强调用户参与和提示工程改进，但未涉及任何视觉文档输入或多模态处理。
Reason 该论文属于典型的平行应用类工作，聚焦于使用LLM生成电子表格公式并引入TDD（测试驱动开发）来提升可靠性。尽管提到了逻辑一致性与幻觉问题，但其核心是软件工程方法在代码生成中的应用，并未针对文档图像理解（DIU）中的视觉-语言对齐、OCR幻觉、复杂版面解析或细粒度文本定位等关键挑战提出任何底层技术创新。任务场景脱离文档图像输入，无法迁移至DocVQA或相关任务。
Abstract Large Language Models (LLMs), such as ChatGPT, are increasingly leveraged for generating both traditional software code and spreadsheet logic. Despite their impressive generative capabilities, these models frequently exhibit critical issues such as hallucinations, subtle logical inconsistencies, and syntactic errors, risks particularly acute in high stakes domains like financial modelling and scientific computations, where accuracy and reliability are paramount. This position paper proposes a structured research framework that integrates the proven software engineering practice of Test-Driven Development (TDD) with Large Language Model (LLM) driven generation to enhance the correctness of, reliability of, and user confidence in generated outputs. We hypothesise that a "test first" methodology provides both technical constraints and cognitive scaffolding, guiding LLM outputs towards more accurate, verifiable, and comprehensible solutions. Our framework, applicable across diverse programming contexts, from spreadsheet formula generation to scripting languages such as Python and strongly typed languages like Rust, includes an explicitly outlined experimental design with clearly defined participant groups, evaluation metrics, and illustrative TDD based prompting examples. By emphasising test driven thinking, we aim to improve computational thinking, prompt engineering skills, and user engagement, particularly benefiting spreadsheet users who often lack formal programming training yet face serious consequences from logical errors. We invite collaboration to refine and empirically evaluate this approach, ultimately aiming to establish responsible and reliable LLM integration in both educational and professional development practices.
Authors Simon Thorne, Advait Sarkar
Comment 16 pages
Categories Computation and Language
0
Mem-PAL: Towards Memory-based Personalized Dialogue Assistants for Long-term User-Agent Interaction AAAI 2026 (Oral)
Title CN Mem-PAL：面向基于记忆的个性化对话助手，用于长期用户-智能体交互
Keywords 个性化对话 · 长期交互 · 记忆框架 · 检索增强生成 · 对话代理
Summary 本文提出了PAL-Bench，一个用于评估长期用户-智能体交互中个性化能力的新基准，以及基于LLM合成并经人工校验的中文多会话数据集PAL-Set。同时设计了H²Memory框架，通过分层异构记忆结构结合检索增强生成来提升个性化响应生成效果。实验表明该框架在PAL-Bench和外部数据集上均有效。
Reason 该论文聚焦于个性化对话系统中的长期用户交互建模，属于典型的‘平行应用’类工作，核心贡献是构建了一个面向服务型对话助手的评测基准（PAL-Bench）和一个中文多轮对话数据集（PAL-Set），并提出了一个分层记忆框架H²Memory。尽管其方法涉及Agent架构设计，但目标场景为通用对话助理，并未针对文档图像理解中的核心痛点（如OCR幻觉、复杂排版、细粒度定位、高分辨率处理等）提出可迁移的底层技术创新。属于‘具身智能/人机交互’范畴下的应用研究，与文档图像理解无直接关联。
Abstract With the rise of smart personal devices, service-oriented human-agent interactions have become increasingly prevalent. This trend highlights the need for personalized dialogue assistants that can understand user-specific traits to accurately interpret requirements and tailor responses to individual preferences. However, existing approaches often overlook the complexities of long-term interactions and fail to capture users' subjective characteristics. To address these gaps, we present PAL-Bench, a new benchmark designed to evaluate the personalization capabilities of service-oriented assistants in long-term user-agent interactions. In the absence of available real-world data, we develop a multi-step LLM-based synthesis pipeline, which is further verified and refined by human annotators. This process yields PAL-Set, the first Chinese dataset comprising multi-session user logs and dialogue histories, which serves as the foundation for PAL-Bench. Furthermore, to improve personalized service-oriented interactions, we propose H$^2$Memory, a hierarchical and heterogeneous memory framework that incorporates retrieval-augmented generation to improve personalized response generation. Comprehensive experiments on both our PAL-Bench and an external dataset demonstrate the effectiveness of the proposed memory framework.
Authors Zhaopei Huang, Qifeng Dai, Guozheng Wu, Xiaopeng Wu, Kehan Chen, Chuan Yu, Xubin Li, Tiezheng Ge, Wenxuan Wang, Qin Jin
Comment Accepted by AAAI 2026 (Oral)
Categories Computation and Language
0
Co-NAML-LSTUR: A Combined Model with Attentive Multi-View Learning and Long- and Short-term User Representations for News Recommendation The 18th International Conference on Multi-disciplinary Trends in Artificial Intelligence (MIWAI 2025)
Title CN Co-NAML-LSTUR：一种结合注意力多视图学习与长短时用户表示的新闻推荐模型
Keywords 新闻推荐 · 多视图学习 · 用户表示 · NAML · LSTUR
Summary 本文提出了一种名为Co-NAML-LSTUR的新闻推荐框架，结合NAML和LSTUR模型，利用BERT增强语义表示，并通过多视图新闻编码和长短时用户行为建模提升推荐性能。在MIND数据集上取得了优于NRMS和NAML的AUC与MRR指标。但该工作属于典型的推荐系统应用，无文档理解相关的技术创新。
Reason 该论文属于新闻推荐领域的下游应用，核心是推荐系统中的用户兴趣建模与多视图新闻编码，未涉及文档图像理解（DIU）、视觉布局分析、OCR、VQA或多模态文档建模等任何相关技术。其方法无法迁移用于解决OCR幻觉、密集文本识别、复杂排版理解或长文档推理等DIU核心痛点。
Abstract News recommendation systems play a critical role in alleviating information overload by delivering personalized content. A key challenge lies in jointly modeling multi-view representations of news articles and capturing the dynamic, dual-scale nature of user interests-encompassing both short- and long-term preferences. Prior methods often rely on single-view features or insufficiently model user behavior across time. In this work, we introduce Co-NAML-LSTUR, a hybrid news recommendation framework that integrates NAML for attentive multi-view news encoding and LSTUR for hierarchical user modeling, designed for training on limited data resources. Our approach leverages BERT-based embeddings to enhance semantic representation. We evaluate Co-NAML-LSTUR on two widely used benchmarks, MIND-small and MIND-large. Results show that our model significantly outperforms strong baselines, achieving improvements over NRMS by 1.55% in AUC and 1.15% in MRR, and over NAML by 2.45% in AUC and 1.71% in MRR. These findings highlight the effectiveness of our efficiency-focused hybrid model, which combines multi-view news modeling with dual-scale user representations for practical, resource-limited resources rather than a claim to absolute state-of-the-art (SOTA). The implementation of our model is publicly available at https://github.com/MinhNguyenDS/Co-NAML-LSTUR
Authors Minh Hoang Nguyen, Thuat Thien Nguyen, Minh Nhat Ta, Tung Le, Huy Tien Nguyen
Comment The 18th International Conference on Multi-disciplinary Trends in Artificial Intelligence (MIWAI 2025)
Categories Computation and Language
0
Yesterday's News: Benchmarking Multi-Dimensional Out-of-Distribution Generalization of Misinformation Detection Models Computational Linguistics
Title CN 昨日新闻：评估错误信息检测模型多维度分布外泛化能力的基准
Keywords 错误信息检测 · 分布外泛化 · 基准数据集 · 自然语言处理 · 远距离标注
Summary 本文提出了一个名为misinfo-general的基准数据集，用于评估错误信息检测模型在分布外（out-of-distribution）场景下的泛化能力。作者识别了时间、事件、主题、发布者、政治倾向和错误信息类型等多个泛化维度，并利用远距离标注模拟协变量偏移。通过元数据分析发现当前基线模型存在严重缺陷，且这些缺陷在传统分类指标下不易察觉。数据集和代码已公开。
Reason 该论文属于纯NLP领域，研究的是错误信息检测中的分布外泛化问题，尽管涉及文本分类和数据集构建，但完全不涉及视觉模态、文档布局、OCR或任何与文档图像理解（DIU）相关的技术挑战。其方法论无法迁移到解决DIU中的核心痛点如OCR幻觉、复杂排版处理或细粒度视觉-文本对齐。属于无关领域的平行应用。
Abstract This article introduces misinfo-general, a benchmark dataset for evaluating misinformation models' ability to perform out-of-distribution generalization. Misinformation changes rapidly, much more quickly than moderators can annotate at scale, resulting in a shift between the training and inference data distributions. As a result, misinformation detectors need to be able to perform out-of-distribution generalization, an attribute they currently lack. Our benchmark uses distant labelling to enable simulating covariate shifts in misinformation content. We identify time, event, topic, publisher, political bias, misinformation type as important axes for generalization, and we evaluate a common class of baseline models on each. Using article metadata, we show how this model fails desiderata, which is not necessarily obvious from classification metrics. Finally, we analyze properties of the data to ensure limited presence of modelling shortcuts. We make the dataset and accompanying code publicly available: https://github.com/ioverho/misinfo-general
Authors Ivo Verhoeven, Pushkar Mishra, Ekaterina Shutova
Comment Accepted for publication in Computational Linguistics on November 23, 2025. This is the pre-MIT Press publication version
Categories Computation and Language
0
Can LLMs extract human-like fine-grained evidence for evidence-based fact-checking?
Title CN 大语言模型能否为基于证据的事实核查提取类人细粒度证据？
Keywords 事实核查 · 细粒度证据提取 · 大语言模型 · 小语种 · 错误分析
Summary 本文研究大语言模型在捷克语和斯洛伐克语环境下进行细粒度证据提取的能力，构建了人工标注的数据集并评估多个LLM的表现。结果显示部分较小模型在对齐人类标注方面表现更优。
Reason 该论文聚焦于从新闻评论中提取支持或反驳主张的文本证据，属于纯NLP任务，且使用的是捷克语和斯洛伐克语数据。命中负面清单第3条（小语种）和第2条（无关领域：事实核查不属于文档图像理解任务），无视觉模态输入，与DIU完全无关。
Abstract Misinformation frequently spreads in user comments under online news articles, highlighting the need for effective methods to detect factually incorrect information. To strongly support or refute claims extracted from such comments, it is necessary to identify relevant documents and pinpoint the exact text spans that justify or contradict each claim. This paper focuses on the latter task -- fine-grained evidence extraction for Czech and Slovak claims. We create new dataset, containing two-way annotated fine-grained evidence created by paid annotators. We evaluate large language models (LLMs) on this dataset to assess their alignment with human annotations. The results reveal that LLMs often fail to copy evidence verbatim from the source text, leading to invalid outputs. Error-rate analysis shows that the {llama3.1:8b model achieves a high proportion of correct outputs despite its relatively small size, while the gpt-oss-120b model underperforms despite having many more parameters. Furthermore, the models qwen3:14b, deepseek-r1:32b, and gpt-oss:20b demonstrate an effective balance between model size and alignment with human annotations.
Authors Antonín Jarolím, Martin Fajčík, Lucia Makaiová
Categories Computation and Language
0
BanglaASTE: A Novel Framework for Aspect-Sentiment-Opinion Extraction in Bangla E-commerce Reviews Using Ensemble Deep Learning 2025 IEEE International Conference on Signal Processing, Information, Communication and Systems (SPICSCON)
Title CN BanglaASTE：一种基于集成深度学习的孟加拉语电商评论方面-情感-观点抽取新框架
Keywords 方面情感三元组抽取 · 孟加拉语NLP · 集成学习 · 电商评论分析 · 低资源语言
Summary 本文提出了BanglaASTE框架，用于从孟加拉语电商评论中提取方面-情感-观点三元组，并发布了首个孟加拉语ASTE标注数据集。方法结合BanglaBERT和XGBoost进行分类，实验显示其在准确率和F1分数上优于基线模型。研究主要贡献在于低资源语言的情感分析应用。
Reason 该论文聚焦于小语种（孟加拉语）的情感分析任务，属于特定语言和垂直场景的应用研究，未提出可迁移至文档图像理解（DIU）领域的通用方法论。其技术路线为NLP层面的模型集成与数据集构建，不涉及视觉信息处理、多模态对齐或文档结构理解等DIU核心问题。
Abstract Aspect-Based Sentiment Analysis (ABSA) has emerged as a critical tool for extracting fine-grained sentiment insights from user-generated content, particularly in e-commerce and social media domains. However, research on Bangla ABSA remains significantly underexplored due to the absence of comprehensive datasets and specialized frameworks for triplet extraction in this language. This paper introduces BanglaASTE, a novel framework for Aspect Sentiment Triplet Extraction (ASTE) that simultaneously identifies aspect terms, opinion expressions, and sentiment polarities from Bangla product reviews. Our contributions include: (1) creation of the first annotated Bangla ASTE dataset containing 3,345 product reviews collected from major e-commerce platforms including Daraz, Facebook, and Rokomari; (2) development of a hybrid classification framework that employs graph-based aspect-opinion matching with semantic similarity techniques; and (3) implementation of an ensemble model combining BanglaBERT contextual embeddings with XGBoost boosting algorithms for enhanced triplet extraction performance. Experimental results demonstrate that our ensemble approach achieves superior performance with 89.9% accuracy and 89.1% F1-score, significantly outperforming baseline models across all evaluation metrics. The framework effectively addresses key challenges in Bangla text processing including informal expressions, spelling variations, and data sparsity. This research advances the state-of-the-art in low-resource language sentiment analysis and provides a scalable solution for Bangla e-commerce analytics applications.
Authors Ariful Islam, Md Rifat Hossen, Abir Ahmed, B M Taslimul Haque
Comment Presented at the 2025 IEEE International Conference on Signal Processing, Information, Communication and Systems (SPICSCON), November 21-22, 2025, University of Rajshahi, Bangladesh. 6 pages, ensemble deep learning, 3,345 annotated Bangla product reviews
Categories Computation and Language
0
Emergent Lexical Semantics in Neural Language Models: Testing Martin's Law on LLM-Generated Text
Title CN 神经语言模型中涌现的词汇语义：在LLM生成文本上检验Martin定律
Keywords Martin定律 · 词汇多义性 · 语言模型 · 语义演化 · 上下文化嵌入
Summary 该研究首次系统性地检验了Martin定律在神经语言模型训练过程中生成文本的适用性。通过DBSCAN聚类上下文化嵌入来操作化词义，分析了四个Pythia模型在30个训练检查点上的表现。结果显示Martin定律在训练中期出现并达到峰值后衰退，小模型出现语义崩溃，大模型则表现更稳健。研究揭示了语言模型中语言规律的非单调发展轨迹。
Reason 该论文研究的是语言模型生成文本中的词汇多义性与频率关系（Martin定律），属于纯NLP理论分析，聚焦于语言模型内部表示的语义演化，不涉及任何视觉模态、文档结构或图文对齐问题。与文档图像理解（DIU）的核心痛点如OCR幻觉、布局分析、细粒度定位等完全无关，无法迁移用于解决文档理解任务。
Abstract We present the first systematic investigation of Martin's Law - the empirical relationship between word frequency and polysemy - in text generated by neural language models during training. Using DBSCAN clustering of contextualized embeddings as an operationalization of word senses, we analyze four Pythia models (70M-1B parameters) across 30 training checkpoints. Our results reveal a non-monotonic developmental trajectory: Martin's Law emerges around checkpoint 100, reaches peak correlation (r > 0.6) at checkpoint 104, then degrades by checkpoint 105. Smaller models (70M, 160M) experience catastrophic semantic collapse at late checkpoints, while larger models (410M, 1B) show graceful degradation. The frequency-specificity trade-off remains stable (r $\approx$ -0.3) across all models. These findings suggest that compliance with linguistic regularities in LLM-generated text is not monotonically increasing with training, but instead follows a balanced trajectory with an optimal semantic window. This work establishes a novel methodology for evaluating emergent linguistic structure in neural language models.
Authors Kai Kugler
Comment paper draft
Categories Computation and Language
0
Developing an Open Conversational Speech Corpus for the Isan Language
Title CN 为伊桑语开发开放的会话语音语料库
Keywords 语音语料库 · 伊桑语 · 自然会话 · 低资源语言 · 转录规范
Summary 本文介绍了首个面向泰国伊桑语的开放会话语音数据集，强调采集自然口语而非朗读语音，以捕捉口语现象如俚语、即兴语调、不流畅表达及与泰语的语码转换。由于伊桑语缺乏标准化正字法，研究制定了兼顾语言真实性和计算可用性的转录规范。该数据集旨在促进包容性AI发展和低资源语言研究。
Reason 该论文聚焦于小语种（伊桑语）的语音数据集构建，属于非中英语言的特定区域方言研究，命中负面清单中的‘小语种’项。尽管其在低资源语言和自然会话建模方面有一定价值，但不符合本研究关注的文档图像理解技术主线，且不具备向DIU领域迁移的方法论潜力。
Abstract This paper introduces the development of the first open conversational speech dataset for the Isan language, the most widely spoken regional dialect in Thailand. Unlike existing speech corpora that are primarily based on read or scripted speech, this dataset consists of natural speech, thereby capturing authentic linguistic phenomena such as colloquials, spontaneous prosody, disfluencies, and frequent code-switching with central Thai. A key challenge in building this resource lies in the lack of a standardized orthography for Isan. Current writing practices vary considerably, due to the different lexical tones between Thai and Isan. This variability complicates the design of transcription guidelines and poses questions regarding consistency, usability, and linguistic authenticity. To address these issues, we establish practical transcription protocols that balance the need for representational accuracy with the requirements of computational processing. By releasing this dataset as an open resource, we aim to contribute to inclusive AI development, support research on underrepresented languages, and provide a basis for addressing the linguistic and technical challenges inherent in modeling conversational speech.
Authors Adisai Na-Thalang, Chanakan Wittayasakpan, Kritsadha Phatcharoen, Supakit Buakaw
Comment 31 pages, in Thai language, 3 figures, 25 tables
Categories Computation and Language
0
Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?
Title CN 在小型人类样本上微调大语言模型能否增加异质性、对齐性和信念-行动一致性？
Keywords 大语言模型 · 微调 · 人类行为模拟 · 对齐性 · 社会科学
Summary 本文探讨了在少量人类调查数据上微调大语言模型是否能改善其在社会科学研究中模拟人类行为的能力。结果显示微调可提升生成结果的多样性、子群体对齐性和信念-行动一致性，但仍无法复现原始研究的回归系数，表明LLM尚不能替代真实人类参与推断性分析。
Reason 该论文研究的是将LLM用于社会科学模拟（如调查和实验）时，通过微调提升其与人类行为的一致性。主题属于纯NLP在社会科学中的应用，不涉及任何视觉模态或文档图像理解任务。尽管提到了对齐性和幻觉问题，但语境是‘信念与行为的不一致’，而非视觉或OCR相关的幻觉。完全脱离文档图像理解的技术范畴。
Abstract There is ongoing debate about whether large language models (LLMs) can serve as substitutes for human participants in survey and experimental research. While recent work in fields such as marketing and psychology has explored the potential of LLM-based simulation, a growing body of evidence cautions against this practice: LLMs often fail to align with real human behavior, exhibiting limited diversity, systematic misalignment for minority subgroups, insufficient within-group variance, and discrepancies between stated beliefs and actions. This study examines an important and distinct question in this domain: whether fine-tuning on a small subset of human survey data, such as that obtainable from a pilot study, can mitigate these issues and yield realistic simulated outcomes. Using a behavioral experiment on information disclosure, we compare human and LLM-generated responses across multiple dimensions, including distributional divergence, subgroup alignment, belief-action coherence, and the recovery of regression coefficients. We find that fine-tuning on small human samples substantially improves heterogeneity, alignment, and belief-action coherence relative to the base model. However, even the best-performing fine-tuned models fail to reproduce the regression coefficients of the original study, suggesting that LLM-generated data remain unsuitable for replacing human participants in formal inferential analyses.
Authors Steven Wang, Kyle Hunt, Shaojie Tang, Kenneth Joseph
Categories Computation and Language
0
The Distribution of Dependency Distance and Hierarchical Distance in Contemporary Written Japanese and Its Influencing Factors 13th International Quantitative Linguistics Conference QUALICO 2025
Title CN 现代日语书面语中依存距离与层级距离的分布及其影响因素
Keywords 依存距离 · 层级距离 · 谓词价数 · 认知负荷 · 定量语言学
Summary 本文利用现代日语书面平衡语料库，分析了在固定句子长度条件下依存距离（DD）和层级距离（HD）的概率分布，并探讨了谓词价数对平均依存距离（MDD）和平均层级距离（MHD）的影响。研究发现，谓词的价数是调节线性复杂度与层级复杂度之间权衡关系的根本因素，且其对HD分布的影响大于对DD的影响，导致MDD低于MHD。
Reason 该论文研究的是日语句法结构中的依存距离与层级距离的关系，属于纯语言学分析，且基于日语这一小语种数据。未涉及文档图像、视觉信息处理或任何多模态建模，与文档图像理解（DIU）任务无直接关联。同时，其方法无法迁移用于解决OCR幻觉、复杂排版或长文档推理等核心问题。
Abstract To explore the relationship between dependency distance (DD) and hierarchical distance (HD) in Japanese, we compared the probability distributions of DD and HD with and without sentence length fixed, and analyzed the changes in mean dependency distance (MDD) and mean hierarchical distance (MHD) as sentence length increases, along with their correlation coefficient based on the Balanced Corpus of Contemporary Written Japanese. It was found that the valency of the predicates is the underlying factor behind the trade-off relation between MDD and MHD in Japanese. Native speakers of Japanese regulate the linear complexity and hierarchical complexity through the valency of the predicates, and the relative sizes of MDD and MHD depend on whether the threshold of valency has been reached. Apart from the cognitive load, the valency of the predicates also affects the probability distributions of DD and HD. The effect of the valency of the predicates on the distribution of HD is greater than on that of DD, which leads to differences in their probability distributions and causes the mean of MDD to be lower than that of MHD.
Authors Linxuan Wang, Shuiyuan Yu
Comment This paper has been accepted by the 13th International Quantitative Linguistics Conference QUALICO 2025
Categories Computation and Language
0
How to Correctly Report LLM-as-a-Judge Evaluations
Title CN 如何正确报告LLM作为评判者的评估
Keywords LLM-as-a-Judge · 偏差校正 · 置信区间 · 评估方法 · 自适应采样
Summary 本文提出一种插件式框架来校正大语言模型作为评判者时产生的评估偏差，并基于估计的特异性和敏感性构建置信区间，同时引入自适应算法优化校准样本分配。目标是提升自动评估的统计严谨性。
Reason 该论文聚焦于LLM-as-a-Judge的评估偏差校正与置信区间构建，属于纯NLP方法论研究，且不涉及任何视觉模态或文档理解任务。其技术虽在评估层面有通用价值，但无法迁移解决文档图像理解中的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）。属于无关领域的方法研究，不符合DIU本题或可迁移工具的标准。
Abstract Large language models (LLMs) are increasingly used as evaluators in lieu of humans. While scalable, their judgments are noisy due to imperfect specificity and sensitivity of LLMs, leading to biased accuracy estimates. Although bias-correction methods exist, they are underutilized in LLM research and typically assume exact knowledge of the model's specificity and sensitivity. Furthermore, in general we only have estimates of these values and it is not well known how to properly construct confidence intervals using only estimates. This work presents a simple plug-in framework that corrects such bias and constructs confidence intervals reflecting uncertainty from both test and calibration dataset, enabling practical and statistically sound LLM-based evaluation. Additionally, to reduce uncertainty in the accuracy estimate, we introduce an adaptive algorithm that efficiently allocates calibration sample sizes.
Authors Chungpa Lee, Thomas Zeng, Jongwon Jeong, Jy-yong Sohn, Kangwook Lee
Categories Computation and Language
0
Federated Large Language Models: Current Progress and Future Directions
Title CN 联邦大语言模型：当前进展与未来方向
Keywords 联邦学习 · 大语言模型 · 隐私保护 · 分布式训练 · 提示学习
Summary 本文综述了联邦学习在大语言模型中的应用，重点讨论了联邦环境下的微调和提示学习所面临的数据异构性和通信开销等问题，并提出了未来方向，如联邦预训练和联邦智能体。然而，全文未涉及多模态输入、视觉信息处理或文档结构建模，与文档图像理解无直接关联。
Reason 该论文属于纯NLP领域的联邦学习综述，研究的是大语言模型在分布式数据下的训练框架问题，不涉及任何视觉模态或文档理解任务。尽管隐私保护在某些文档应用中可能相关，但该工作未触及文档图像理解的核心痛点（如OCR幻觉、复杂排版、细粒度定位等），且属于典型的‘平行应用’类技术框架（即通用FL+LLM），无法为DIU提供可迁移的底层方法创新。
Abstract Large language models are rapidly gaining popularity and have been widely adopted in real-world applications. While the quality of training data is essential, privacy concerns arise during data collection. Federated learning offers a solution by allowing multiple clients to collaboratively train LLMs without sharing local data. However, FL introduces new challenges, such as model convergence issues due to heterogeneous data and high communication costs. A comprehensive study is required to address these challenges and guide future research. This paper surveys Federated learning for LLMs (FedLLM), highlighting recent advances and future directions. We focus on two key aspects: fine-tuning and prompt learning in a federated setting, discussing existing work and associated research challenges. We finally propose potential directions for federated LLMs, including pre-training, federated agents, and LLMs for federated learning.
Authors Yuhang Yao, Jianyi Zhang, Junda Wu, Chengkai Huang, Yu Xia, Tong Yu, Ruiyi Zhang, Sungchul Kim, Ryan Rossi, Ang Li, Lina Yao, Julian McAuley, Yiran Chen, Carlee Joe-Wong
Categories Computation and Language
0
Enhancing Large Language Models for Detecting Mental Manipulation via Annotation-Free Data Augmentation and Anti-Curriculum Distillation
Title CN 通过无标注数据增强和反课程蒸馏增强大语言模型以检测心理操纵
Keywords 心理操纵检测 · 无标注数据增强 · 反课程蒸馏 · 多轮对话 · 大语言模型
Summary 本文提出MentalMAC框架，用于增强大语言模型在多轮对话中检测心理操纵的能力。该方法结合基于进化操作和言语行为理论的无标注数据增强（EvoSA）、教师模型生成的多任务监督以及渐进式反课程蒸馏，并构建了真实世界数据集ReaMent。实验表明其在F1和准确率上显著优于现有基线和商用大模型。但由于内容完全脱离文档图像理解范畴，对本研究方向无参考价值。
Reason 该论文研究的是心理操纵检测，属于纯NLP任务，且聚焦于心理健康与对话分析领域，与文档图像理解（DIU）、多模态文档解析、OCR、VLM在文档上的应用等核心问题完全无关。尽管使用了数据增强和模型蒸馏等技术，但其方法设计针对的是文本语义层面的心理行为识别，并未涉及视觉信息处理或多模态对齐，无法迁移到文档图像理解中的OCR幻觉、复杂排版或细粒度定位等关键痛点。
Abstract Mental manipulation is a subtle yet pervasive form of psychological abuse that poses serious threats to mental health. Nevertheless, detecting mental manipulation remains a largely underexplored research problem. The field faces three major challenges: (i) insufficient and hard-to-obtain training data; (ii) the covert nature of mental manipulation, which hinders detection; and (iii) the lack of real-world datasets. To address these challenges, we propose MentalMAC, a novel framework that enhances large language models' ability to detect elements of mental manipulation in multi-turn dialogue. Our approach consists of three key components: EvoSA, an annotation-free data augmentation method based on evolutionary operations and speech act theory; teacher-model-generated multi-task supervision; and progressive task-level anti-curriculum distillation. We then constructed the ReaMent dataset, comprising 5,000 real-world dialogue samples, utilizing MentalMAC-distilled models to aid in human annotation. Vast experiments show that MentalMAC achieves up to 25.9% improvement in F1mac and 8.1% in accuracy over the best-performing baseline, outperforming commercial LLMs such as GPT-4 and Claude-3.5-Sonnet. Warning: This paper contains content that may be offensive to the reader.
Authors Yuansheng Gao, Han Bao, Tong Zhang, Bin Li, Jixiang Luo, Ronghao Chen, Zonghui Wang, Wenzhi Chen
Comment Preprint
Categories Computation and Language
0
MortgageLLM: Domain-Adaptive Pretraining with Residual Instruction Transfer, Alignment Tuning, and Task-Specific Routing
Title CN MortgageLLM：基于残差指令迁移、对齐调优和任务特定路由的领域自适应预训练
Keywords 领域自适应 · 大语言模型 · 指令微调 · 任务路由 · 金融NLP
Summary 本文提出了MortgageLLM，一个面向房贷金融领域的专用大语言模型，采用双专家架构分别处理对话问答与结构化任务，并通过残差指令迁移恢复指令遵循能力，结合任务路由机制实现自动分流。在领域基准上表现优于基础模型。
Reason 该论文属于典型的‘平行下游应用’，即将现有LLM技术应用于房贷金融这一特定领域，未提出任何文档图像理解（DIU）相关的底层方法创新。其核心贡献如双专家架构、任务路由机制等均围绕领域适应与任务调度展开，不涉及视觉输入处理、OCR增强、多模态对齐或文档布局建模等DIU关键技术。无法迁移用于解决OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Large Language Models (LLMs) demonstrate exceptional capabilities across general domains, yet their application to specialized sectors such as mortgage finance requires domain-specific knowledge augmentation while preserving instruction-following fidelity. We present MortgageLLM, a novel domain-specific large language model that addresses this dual challenge. It is developed using a dual-track specialization framework from a single base model (LLaMA-3.1-8B). We opted for this dual-expert approach as a single multi-task model suffers from performance trade-offs, where optimizing for structured tasks (via SFT) degrades conversational fidelity (via DPO). Our dual-track method solves this by creating two specialists, allowing each to be optimally trained for its distinct capability. Our approach applies the instruction residual technique to restore instruction-following capabilities post-domain adaptation without supervised fine-tuning. We contribute: (1) application of this residual technique to the highly specialized mortgage finance domain; (2) a dual-expert architecture combining a conversational Q&A model and a structured task model for classification and summarization; and (3) an intelligent task routing mechanism using few-shot classification performed by one of the expert models itself. We validate our approach on domain-specific benchmarks, where our final model (MLM v2) significantly outperforms the base LLaMA-3.1-8B-Instruct, achieving an LLM-as-a-Judge summarization score of 4.58 (vs. 3.99), a Q&A score of 4.09 (vs. 4.0), and a classification score of 2.6 (vs. 1.2). On semantic similarity, our model achieved a BERTScore of 0.77 for summarization (vs. 0.74), 0.68 for Q&A (vs. 0.58), and 0.75 for classification (vs. 0.73), substantially outperforming baseline approaches.
Authors Manish Jain, Satheesh Kumar Ponnambalam, Salman Faroz, Chandrakanth Lns, Vinay Sharma
Categories Computation and Language
0
Exploring Cross-Lingual Knowledge Transfer via Transliteration-Based MLM Fine-Tuning for Critically Low-resource Chakma Language
Title CN 通过基于音译的MLM微调探索低资源查克马语的跨语言知识迁移
Keywords 低资源语言 · 音译 · 掩码语言建模 · 跨语言迁移 · Chakma
Summary 本文构建了一个孟加拉语音译的查克马语文本语料库，并在多种Transformer模型上进行MLM微调，以探索跨语言知识迁移效果。实验表明多语言模型经微调后在该任务上表现最佳。研究重点为纯文本层面的低资源语言建模，不涉及图像模态或文档布局理解。
Reason 该论文聚焦于低资源语言查克马语的自然语言建模，使用孟加拉语音译数据进行掩码语言建模微调。属于小语种NLP研究，未涉及文档图像理解中的视觉-语言对齐、OCR后处理或VLM架构改进。尽管提到了OCR在形态丰富文字中的局限性，但并未提出可迁移至文档图像理解领域的底层方法。
Abstract As an Indo-Aryan language with limited available data, Chakma remains largely underrepresented in language models. In this work, we introduce a novel corpus of contextually coherent Bangla-transliterated Chakma, curated from Chakma literature, and validated by native speakers. Using this dataset, we fine-tune six encoder-based transformer models, including multilingual (mBERT, XLM-RoBERTa, DistilBERT), regional (BanglaBERT, IndicBERT), and monolingual English (DeBERTaV3) variants on masked language modeling (MLM) tasks. Our experiments show that fine-tuned multilingual models outperform their pre-trained counterparts when adapted to Bangla-transliterated Chakma, achieving up to 73.54% token accuracy and a perplexity as low as 2.90. Our analysis further highlights the impact of data quality on model performance and shows the limitations of OCR pipelines for morphologically rich Indic scripts. Our research demonstrates that Bangla-transliterated Chakma can be very effective for transfer learning for Chakma language, and we release our dataset to encourage further research on multilingual language modeling for low-resource languages.
Authors Adity Khisa, Nusrat Jahan Lia, Tasnim Mahfuz Nafis, Zarif Masud, Tanzir Pial, Shebuti Rayana, Ahmedul Kabir
Categories Computation and Language
0
ASR Error Correction in Low-Resource Burmese with Alignment-Enhanced Transformers using Phonetic Features iSAI-NLP 2025
Title CN 利用音素特征和对齐增强Transformer进行低资源缅甸语ASR错误纠正
Keywords ASR Error Correction · Burmese · Transformer · Phonetic Features · Low-resource Language
Summary 本文提出了一种结合国际音标（IPA）和对齐信息的Transformer模型，用于纠正低资源缅甸语自动语音识别（ASR）中的错误。实验表明，该方法在词错误率（WER）和字符级F-score（chrF++）上均显著优于基线ASR输出。这是首个针对缅甸语ASR纠错的研究。
Reason 该论文研究的是低资源语言（缅甸语）的语音识别错误纠正问题，属于纯语音处理领域，且针对小语种（非中英），完全偏离文档图像理解（DIU）的核心任务。其方法无法迁移到解决OCR幻觉、复杂排版或文档VQA等关键痛点。
Abstract This paper investigates sequence-to-sequence Transformer models for automatic speech recognition (ASR) error correction in low-resource Burmese, focusing on different feature integration strategies including IPA and alignment information. To our knowledge, this is the first study addressing ASR error correction specifically for Burmese. We evaluate five ASR backbones and show that our ASR Error Correction (AEC) approaches consistently improve word- and character-level accuracy over baseline outputs. The proposed AEC model, combining IPA and alignment features, reduced the average WER of ASR models from 51.56 to 39.82 before augmentation (and 51.56 to 43.59 after augmentation) and improving chrF++ scores from 0.5864 to 0.627, demonstrating consistent gains over the baseline ASR outputs without AEC. Our results highlight the robustness of AEC and the importance of feature design for improving ASR outputs in low-resource settings.
Authors Ye Bhone Lin, Thura Aung, Ye Kyaw Thu, Thazin Myint Oo
Comment 7 pages, 2 figures, 7 tables, Accepted to iSAI-NLP 2025
Categories Computation and Language
0
Evaluating Large Language Models for Radiology Natural Language Processing
Title CN 评估大语言模型在放射学自然语言处理中的应用
Keywords 大语言模型 · 自然语言处理 · 放射学 · 医学文本分析 · 模型评估
Summary 本文评估了32个大语言模型在放射学自然语言处理任务中的表现，重点考察从影像发现中推导出诊断印象的能力。研究旨在揭示这些模型在医学领域中的性能优劣，为实际应用提供参考。
Reason 该论文属于纯NLP在医疗领域的下游应用，聚焦于放射学报告的理解与印象生成，未涉及任何视觉模态或文档图像理解任务。尽管使用了中英双语模型，但研究内容完全基于文本到文本的处理，不包含OCR、布局分析、表格识别等DIU核心问题，也不提供可迁移至文档图像理解的方法论创新，属于典型的‘平行应用’。
Abstract The rise of large language models (LLMs) has marked a pivotal shift in the field of natural language processing (NLP). LLMs have revolutionized a multitude of domains, and they have made a significant impact in the medical field. Large language models are now more abundant than ever, and many of these models exhibit bilingual capabilities, proficient in both English and Chinese. However, a comprehensive evaluation of these models remains to be conducted. This lack of assessment is especially apparent within the context of radiology NLP. This study seeks to bridge this gap by critically evaluating thirty two LLMs in interpreting radiology reports, a crucial component of radiology NLP. Specifically, the ability to derive impressions from radiologic findings is assessed. The outcomes of this evaluation provide key insights into the performance, strengths, and weaknesses of these LLMs, informing their practical applications within the medical domain.
Authors Zhengliang Liu, Tianyang Zhong, Yiwei Li, Yutong Zhang, Yi Pan, Zihao Zhao, Peixin Dong, Chao Cao, Yuxiao Liu, Peng Shu, Yaonai Wei, Zihao Wu, Chong Ma, Jiaqi Wang, Sheng Wang, Mengyue Zhou, Zuowei Jiang, Chunlin Li, Jason Holmes, Shaochen Xu, Lu Zhang, Haixing Dai, Kai Zhang, Lin Zhao, Yuanhao Chen, Xu Liu, Peilong Wang, Junhao Chen, Pingkun Yan, Jun Liu, Bao Ge, Lichao Sun, Dajiang Zhu, Xiang Li, Wei Liu, Xiaoyan Cai, Xintao Hu, Xi Jiang, Shu Zhang, Xin Zhang, Tuo Zhang, Shijie Zhao, Quanzheng Li, Hongtu Zhu, Dinggang Shen, Tianming Liu
Categories Computation and Language
0
Zipf Distributions from Two-Stage Symbolic Processes: Stability Under Stochastic Lexical Filtering
Title CN 从两阶段符号过程看齐夫分布：随机词汇过滤下的稳定性
Keywords Zipf's law · symbolic model · word formation · power-law distribution · computational linguistics
Summary 本文提出一种称为全组合词模型（FCWM）的符号过程来解释语言中齐夫定律的出现，认为其源于几何约束而非语言效率。通过有限字母表生成词语，并引入空白符号概率，模型可生成符合幂律的词频分布。实验模拟结果与英语、俄语等真实语言数据吻合。
Reason 该论文研究语言中齐夫定律的成因，属于计算语言学理论范畴，未涉及文档图像、多模态理解、视觉表示或VLM技术。与文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等无直接关联，也无法迁移解决OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Zipf's law in language lacks a definitive origin, debated across fields. This study explains Zipf-like behavior using geometric mechanisms without linguistic elements. The Full Combinatorial Word Model (FCWM) forms words from a finite alphabet, generating a geometric distribution of word lengths. Interacting exponential forces yield a power-law rank-frequency curve, determined by alphabet size and blank symbol probability. Simulations support predictions, matching English, Russian, and mixed-genre data. The symbolic model suggests Zipf-type laws arise from geometric constraints, not communicative efficiency.
Authors Vladimir Berman
Comment 16 pages
Categories Computation and Language
0
TrackList: Tracing Back Query Linguistic Diversity for Head and Tail Knowledge in Open Large Language Models
Title CN TrackList：追溯查询语言多样性以探究开放大语言模型中的头部与尾部知识
Keywords 大语言模型 · 语言多样性 · 头部与尾部知识 · 文本生成评估 · 医学术语
Summary 本文提出TrackList分析流程和RefoMed-EN数据集，用于评估大语言模型在不同类型语言查询（如定义、举例、释义）上的表现差异，发现模型对高频知识更擅长生成定义和 paraphrase，而对低频专业知识及非定义类回答能力较弱。研究局限于文本到文本的语言生成质量分析。
Reason 该论文聚焦于纯文本大语言模型在回答不同类型语言查询时的表现差异，研究问题集中在定义、举例、释义等语言形式的生成质量，使用的是医学术语数据集和文本相似性指标。其任务设定完全脱离视觉模态，不涉及文档图像、OCR、多模态理解或视觉-语言对齐等任何文档图像理解（DIU）核心问题。属于纯NLP范畴，且未触及视觉幻觉、细粒度定位或多模态推理等可迁移痛点，与DIU领域无实质关联。
Abstract Large Language Models (LLMs) have proven efficient in giving definition-type answers to user input queries. While for humans giving various types of answers, such as examples and paraphrases, is an easy task, LLMs struggle to provide correct answers for other than definition-type queries. In this study, we evaluated this drop in performance using TrackList, a fine-grained linguistic and statistical analysis pipeline to investigate the impact of the pre-training data on LLMs answers to diverse linguistic queries. We also introduce RefoMed-EN, an English dataset consisting of 6170 human-annotated medical terms alongside their corresponding definitions, denominations, exemplifications, explanations, or paraphrases. We studied whether the high frequency of a concept (head) or low frequency (tail) impacts the language model's performance. We evaluated the quality of the LLM's output using syntactic and semantic similarity metrics, statistical correlations and embeddings. Results showed that the LLM's task performance for definition type questions is the highest, while for the exemplification type it is the lowest. Additionally, we showed that for definition-type questions, large language models are prone to paraphrase more on popular and frequent knowledge and less on tail and technical knowledge, especially in the expert texts.
Authors Ioana Buhnila, Aman Sinha, Mathieu Constant
Comment under review
Categories Computation and Language
0
On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey
Title CN 预训练语言模型在通用文本嵌入中的作用：一项综述
Keywords 文本嵌入 · 预训练语言模型 · 对比学习 · 多语言处理 · 表示学习
Summary 本文系统回顾了预训练语言模型（PLM）在通用文本嵌入（GPTE）发展中的作用，涵盖架构设计、对比学习优化、多语言支持、多模态集成等方向，并展望了未来在排序融合、安全性、偏见缓解等方面的研究路径。重点在于纯文本表示学习，不涉及图像或文档结构信息。
Reason 该论文属于纯NLP范畴，聚焦于通用文本嵌入（GPTE）的综述，未涉及任何视觉模态或文档图像理解任务。尽管其讨论了PLM在文本表示中的角色，但完全脱离多模态场景，无法迁移用于解决OCR幻觉、复杂排版或细粒度视觉-文本对齐等DIU核心问题。属于无关领域的理论性综述，无直接方法论价值。
Abstract Text embeddings have attracted growing interest due to their effectiveness across a wide range of natural language processing (NLP) tasks, including retrieval, classification, clustering, bitext mining, and summarization. With the emergence of pretrained language models (PLMs), general-purpose text embeddings (GPTE) have gained significant traction for their ability to produce rich, transferable representations. The general architecture of GPTE typically leverages PLMs to derive dense text representations, which are then optimized through contrastive learning on large-scale pairwise datasets. In this survey, we provide a comprehensive overview of GPTE in the era of PLMs, focusing on the roles PLMs play in driving its development. We first examine the fundamental architecture and describe the basic roles of PLMs in GPTE, i.e., embedding extraction, expressivity enhancement, training strategies, learning objectives, and data construction. We then describe advanced roles enabled by PLMs, including multilingual support, multimodal integration, code understanding, and scenario-specific adaptation. Finally, we highlight potential future research directions that move beyond traditional improvement goals, including ranking integration, safety considerations, bias mitigation, structural information incorporation, and the cognitive extension of embeddings. This survey aims to serve as a valuable reference for both newcomers and established researchers seeking to understand the current state and future potential of GPTE.
Authors Meishan Zhang, Xin Zhang, Xinping Zhao, Shouzheng Huang, Baotian Hu, Min Zhang
Comment 45 pages, 4 figures, 9 tables
Categories Computation and Language
0
RosettaSpeech: Zero-Shot Speech-to-Speech Translation from Monolingual Data
Title CN RosettaSpeech：基于单语数据的零样本语音到语音翻译
Keywords 语音到语音翻译 · 零样本学习 · 单语数据 · 机器翻译 · 语音处理
Summary 本文提出RosettaSpeech，一种基于单语语音-文本数据和机器翻译监督的零样本语音到语音翻译框架。该方法在训练时使用文本作为中间表示，推理时直接实现端到端语音到语音转换，无需平行语音对，在多个语言到英语的翻译任务上取得SOTA性能。
Reason 该论文属于语音处理领域（S2ST），研究任务为语音到语音翻译，与文档图像理解（DIU）、DocVQA、OCR、多模态文档建模等完全无关。尽管其方法利用文本作为中间桥梁并依赖NMT监督，但核心问题不涉及视觉内容、布局分析、文本定位或文档推理，无法迁移解决OCR幻觉、密集文本识别、复杂排版解析等DIU核心痛点。
Abstract The scarcity of parallel speech corpora critically hampers speech-to-speech translation (S2ST), often forcing reliance on complex, multi-stage pipelines. This paper introduces RosettaSpeech, a novel and simplified framework for zero-shot S2ST that is trained on monolingual speech-text data augmented by machine translation supervision. While our method leverages the linguistic knowledge inherent in text-based NMT models, it strictly eliminates the need for parallel speech-to-speech pairs. Our model uniquely uses text as an intermediate bridge during training but functions as a direct, end-to-end speech-to-speech model at inference. This streamlined approach achieves state-of-the-art results on standard benchmarks. For instance, on the CVSS-C test set, RosettaSpeech outperforms leading systems, achieving an ASR-BLEU score of 25.17 for German-to-English and 29.86 for Spanish-to-English-relative gains of over 27% and 14%, respectively. Furthermore, we demonstrate that a single model can deliver strong many-to-one translation performance (FR/ES/DE -> EN). We also provide a foundational analysis of how training data scaling impacts model performance. By prioritizing reliance on abundant parallel text rather than difficult-to-acquire parallel speech, RosettaSpeech offers a scalable path to creating high-quality, speaker-preserving S2ST for a much broader array of languages.
Authors Zhisheng Zheng, Xiaohang Sun, Tuan Dinh, Abhishek Yanamandra, Abhinav Jain, Zhu Liu, Sunil Hadap, Vimal Bhat, Manoj Aggarwal, Gerard Medioni, David Harwath
Comment Work in progress
Categories Computation and Language
0
MTA: A Merge-then-Adapt Framework for Personalized Large Language Model
Title CN MTA：一种用于个性化大语言模型的先合并后适应框架
Keywords 个性化大语言模型 · LoRA · 参数融合 · 少样本学习 · 自然语言处理
Summary 本文提出MTA框架，通过构建共享的Meta-LoRA库，动态融合锚点用户的个性化特征，并结合堆叠的轻量级LoRA模块实现少样本个性化大语言模型。方法聚焦于用户偏好建模与参数高效微调，应用于对话推荐等场景。
Reason 该论文研究的是个性化大语言模型（PLLM）中的用户偏好对齐问题，属于纯NLP领域中的个性化建模，与文档图像理解（DIU）、视觉输入、OCR、多模态文档处理等完全无关。未涉及任何视觉模态、文档结构、布局分析或VLM在文档上的应用。属于负面清单中的‘无关领域’，直接打0分。
Abstract Personalized Large Language Models (PLLMs) aim to align model outputs with individual user preferences, a crucial capability for user-centric applications. However, the prevalent approach of fine-tuning a separate module for each user faces two major limitations: (1) storage costs scale linearly with the number of users, rendering the method unscalable; and (2) fine-tuning a static model from scratch often yields suboptimal performance for users with sparse data. To address these challenges, we propose MTA, a Merge-then-Adapt framework for PLLMs. MTA comprises three key stages. First, we construct a shared Meta-LoRA Bank by selecting anchor users and pre-training meta-personalization traits within meta-LoRA modules. Second, to ensure scalability and enable dynamic personalization combination beyond static models, we introduce an Adaptive LoRA Fusion stage. This stage retrieves and dynamically merges the most relevant anchor meta-LoRAs to synthesize a user-specific one, thereby eliminating the need for user-specific storage and supporting more flexible personalization. Third, we propose a LoRA Stacking for Few-Shot Personalization stage, which applies an additional ultra-low-rank, lightweight LoRA module on top of the merged LoRA. Fine-tuning this module enables effective personalization under few-shot settings. Extensive experiments on the LaMP benchmark demonstrate that our approach outperforms existing SOTA methods across multiple tasks.
Authors Xiaopeng Li, Yuanjin Zheng, Wanyu Wang, wenlin zhang, Pengyue Jia, Yiqi Wang, Maolin Wang, Xuetao Wei, Xiangyu Zhao
Categories Computation and Language
0
MedROV: Towards Real-Time Open-Vocabulary Detection Across Diverse Medical Imaging Modalities
Title CN MedROV：面向多种医学影像模态的实时开放词汇检测
Keywords 开放词汇检测 · 医学影像 · 伪标签 · 基础模型 · 实时检测
Summary 本文提出了MedROV，首个用于多种医学影像模态的实时开放词汇检测模型，通过构建大规模数据集Omnis和伪标签策略解决标注缺失问题，并利用基础模型和对比学习提升对已知与新类别的检测能力，在性能和速度上均取得提升。
Reason 该论文属于医学图像领域的特定应用，研究的是开放词汇目标检测在医学影像中的落地，尽管涉及多模态数据与跨模态对齐，但其核心是垂直领域（医疗）的下游任务应用，未提出可迁移至文档图像理解的通用方法论。不符合DIU本题要求，且属于‘平行下游应用’，命中负面清单第1条。
Abstract Traditional object detection models in medical imaging operate within a closed-set paradigm, limiting their ability to detect objects of novel labels. Open-vocabulary object detection (OVOD) addresses this limitation but remains underexplored in medical imaging due to dataset scarcity and weak text-image alignment. To bridge this gap, we introduce MedROV, the first Real-time Open Vocabulary detection model for medical imaging. To enable open-vocabulary learning, we curate a large-scale dataset, Omnis, with 600K detection samples across nine imaging modalities and introduce a pseudo-labeling strategy to handle missing annotations from multi-source datasets. Additionally, we enhance generalization by incorporating knowledge from a large pre-trained foundation model. By leveraging contrastive learning and cross-modal representations, MedROV effectively detects both known and novel structures. Experimental results demonstrate that MedROV outperforms the previous state-of-the-art foundation model for medical image detection with an average absolute improvement of 40 mAP50, and surpasses closed-set detectors by more than 3 mAP50, while running at 70 FPS, setting a new benchmark in medical detection. Our source code, dataset, and trained model are available at https://github.com/toobatehreem/MedROV.
Authors Tooba Tehreem Sheikh, Jean Lahoud, Rao Muhammad Anwer, Fahad Shahbaz Khan, Salman Khan, Hisham Cholakkal
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
MotionV2V: Editing Motion in a Video
Title CN MotionV2V：编辑视频中的运动
Keywords 视频编辑 · 运动控制 · 扩散模型 · 生成模型 · 运动轨迹
Summary 本文提出了一种通过直接编辑稀疏轨迹来修改视频中运动的方法，称为“运动编辑”，并构建了一个生成“运动反事实”视频对的流程，用于训练运动条件下的视频扩散模型。该方法支持从任意时间点开始的自然运动传播，在用户研究中表现优于现有方法。
Reason 该论文研究的是视频中运动的编辑，属于视频生成与编辑领域，核心任务是运动控制和视频扩散模型的应用。其技术路线聚焦于动态内容修改，与文档图像理解（DIU）中的文本识别、布局分析、表格理解等任务无直接关联。无法迁移用于解决OCR幻觉、密集文本处理或复杂排版理解等DIU核心痛点。
Abstract While generative video models have achieved remarkable fidelity and consistency, applying these capabilities to video editing remains a complex challenge. Recent research has explored motion controllability as a means to enhance text-to-video generation or image animation; however, we identify precise motion control as a promising yet under-explored paradigm for editing existing videos. In this work, we propose modifying video motion by directly editing sparse trajectories extracted from the input. We term the deviation between input and output trajectories a "motion edit" and demonstrate that this representation, when coupled with a generative backbone, enables powerful video editing capabilities. To achieve this, we introduce a pipeline for generating "motion counterfactuals", video pairs that share identical content but distinct motion, and we fine-tune a motion-conditioned video diffusion architecture on this dataset. Our approach allows for edits that start at any timestamp and propagate naturally. In a four-way head-to-head user study, our model achieves over 65 percent preference against prior work. Please see our project page: https://ryanndagreat.github.io/MotionV2V
Authors Ryan Burgert, Charles Herrmann, Forrester Cole, Michael S Ryoo, Neal Wadhwa, Andrey Voynov, Nataniel Ruiz
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Fighting AI with AI: Leveraging Foundation Models for Assuring AI-Enabled Safety-Critical Systems
Title CN 用AI对抗AI：利用基础模型确保AI赋能的安全关键系统
Keywords Requirements Engineering · Formal Verification · LLM · VLM · Safety-Critical Systems
Summary 本文提出利用大语言模型（LLM）和多模态模型来提升安全关键系统中AI组件的可验证性。REACT模块将自然语言需求转化为形式化规范，SemaLens则利用视觉语言模型对DNN感知系统进行语义层面的测试与监控。整体框架面向系统工程与验证流程自动化，而非文档图像理解任务。
Reason 该论文聚焦于安全关键系统（如航空航天、自动驾驶）中的AI验证与需求工程，属于纯NLP和软件工程范畴，未涉及文档图像理解（DIU）或视觉内容解析。其方法REACT用于需求形式化，SemaLens用于感知系统监控，均不针对文档结构、OCR、排版或视觉语言模型在文档上的应用。属于无关领域，且无迁移潜力。
Abstract The integration of AI components, particularly Deep Neural Networks (DNNs), into safety-critical systems such as aerospace and autonomous vehicles presents fundamental challenges for assurance. The opacity of AI systems, combined with the semantic gap between high-level requirements and low-level network representations, creates barriers to traditional verification approaches. These AI-specific challenges are amplified by longstanding issues in Requirements Engineering, including ambiguity in natural language specifications and scalability bottlenecks in formalization. We propose an approach that leverages AI itself to address these challenges through two complementary components. REACT (Requirements Engineering with AI for Consistency and Testing) employs Large Language Models (LLMs) to bridge the gap between informal natural language requirements and formal specifications, enabling early verification and validation. SemaLens (Semantic Analysis of Visual Perception using large Multi-modal models) utilizes Vision Language Models (VLMs) to reason about, test, and monitor DNN-based perception systems using human-understandable concepts. Together, these components provide a comprehensive pipeline from informal requirements to validated implementations.
Authors Anastasia Mavridou, Divya Gopinath, Corina S. Păsăreanu
Categories Artificial Intelligence
0
Copyright Detection in Large Language Models: An Ethical Approach to Generative AI Development
Title CN 大语言模型中的版权检测：生成式AI发展的伦理方法
Keywords 版权检测 · 大语言模型 · 伦理AI · 相似性检测 · 训练数据溯源
Summary 本文提出一个开源的版权检测平台，帮助内容创作者验证其作品是否被用于大语言模型的训练数据。通过优化相似性检测和数据集验证流程，降低计算开销，并提供友好的用户界面。目标是提升AI开发的透明度与伦理合规性。
Reason 该论文聚焦于大语言模型训练数据中版权内容的检测，属于AI伦理与合规领域，与文档图像理解（DIU）或视觉-语言模型在文档任务上的技术改进无直接关联。未涉及OCR、布局分析、表格识别、视觉表示学习或VLM架构优化等DIU核心问题，也无法迁移用于解决OCR幻觉、细粒度定位或复杂排版理解等痛点。
Abstract The widespread use of Large Language Models (LLMs) raises critical concerns regarding the unauthorized inclusion of copyrighted content in training data. Existing detection frameworks, such as DE-COP, are computationally intensive, and largely inaccessible to independent creators. As legal scrutiny increases, there is a pressing need for a scalable, transparent, and user-friendly solution. This paper introduce an open-source copyright detection platform that enables content creators to verify whether their work was used in LLM training datasets. Our approach enhances existing methodologies by facilitating ease of use, improving similarity detection, optimizing dataset validation, and reducing computational overhead by 10-30% with efficient API calls. With an intuitive user interface and scalable backend, this framework contributes to increasing transparency in AI development and ethical compliance, facilitating the foundation for further research in responsible AI development and copyright enforcement.
Authors David Szczecina, Senan Gaffori, Edmond Li
Comment 4 pages, 3 figures
Categories Artificial Intelligence
0
Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unification
Title CN 多模态生成式AI：多模态大语言模型、扩散模型与统一框架
Keywords 多模态大语言模型 · 扩散模型 · 生成式AI · 统一模型 · 综述
Summary 本文综述了多模态生成式AI的发展，包括多模态大语言模型和扩散模型的技术原理、架构设计及应用，并讨论了理解与生成的统一模型方向。内容覆盖广泛，但缺乏针对特定任务的深入创新。
Reason 该论文是一篇关于多模态生成式AI的综述，涵盖多模态大语言模型和扩散模型，并探讨理解与生成的统一框架。但其内容属于广泛性技术总结，未针对文档图像理解（DIU）中的具体任务（如DocVQA、表格识别、布局分析等）提出任何新方法或可迁移的技术创新。同时，文中涉及的扩散模型、视频生成等内容与文档理解核心痛点无关，属于平行领域的综述应用，无具体方法论贡献。
Abstract Multi-modal generative AI (Artificial Intelligence) has attracted increasing attention from both academia and industry. Particularly, two dominant families of techniques have emerged: i) Multi-modal large language models (LLMs) demonstrate impressive ability for multi-modal understanding; and ii) Diffusion models exhibit remarkable multi-modal powers in terms of multi-modal generation. Therefore, this paper provides a comprehensive overview of multi-modal generative AI, including multi-modal LLMs, diffusions, and the unification for understanding and generation. To lay a solid foundation for unified models, we first provide a detailed review of both multi-modal LLMs and diffusion models respectively, including their probabilistic modeling procedure, multi-modal architecture design, and advanced applications to image/video LLMs as well as text-to-image/video generation. Furthermore, we explore the emerging efforts toward unified models for understanding and generation. To achieve the unification of understanding and generation, we investigate key designs including autoregressive-based and diffusion-based modeling, as well as dense and Mixture-of-Experts (MoE) architectures. We then introduce several strategies for unified models, analyzing their potential advantages and disadvantages. In addition, we summarize the common datasets widely used for multi-modal generative AI pretraining. Last but not least, we present several challenging future research directions which may contribute to the ongoing advancement of multi-modal generative AI.
Authors Xin Wang, Yuwei Zhou, Bin Huang, Hong Chen, Wenwu Zhu
Comment 21 pages, 10 figures, 3 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Evaluating the Performance of Deep Learning Models in Whole-body Dynamic 3D Posture Prediction During Load-reaching Activities
Title CN 评估深度学习模型在全身动态3D姿势预测中的性能——在负载搬运活动中的应用
Keywords 3D姿态预测 · 时间序列建模 · Transformer · BLSTM · 运动仿真
Summary 本文研究了基于深度神经网络在负载搬运活动中对人体全身动态3D姿势的预测能力，比较了BLSTM与Transformer两种时序模型的表现，并提出一种通过优化损失函数保持肢体段长度恒定的新方法以提升预测精度。实验结果表明，Transformer模型比BLSTM更准确，且新损失函数显著降低了预测误差。
Reason 该论文研究的是人体运动姿态预测，属于具身智能与生物力学仿真领域，使用BLSTM和Transformer建模时间序列的3D人体动作。尽管涉及深度学习和序列建模，但其任务、数据和目标均与文档图像理解（DIU）或视觉语言多模态处理无关。无法迁移其方法解决OCR幻觉、复杂排版、长文档推理等DIU核心问题。
Abstract This study aimed to explore the application of deep neural networks for whole-body human posture prediction during dynamic load-reaching activities. Two time-series models were trained using bidirectional long short-term memory (BLSTM) and transformer architectures. The dataset consisted of 3D full-body plug-in gait dynamic coordinates from 20 normal-weight healthy male individuals each performing 204 load-reaching tasks from different load positions while adapting various lifting and handling techniques. The model inputs consisted of the 3D position of the hand-load position, lifting (stoop, full-squat and semi-squat) and handling (one- and two-handed) techniques, body weight and height, and the 3D coordinate data of the body posture from the first 25% of the task duration. These inputs were used by the models to predict body coordinates during the remaining 75% of the task period. Moreover, a novel method was proposed to improve the accuracy of the previous and present posture prediction networks by enforcing constant body segment lengths through the optimization of a new cost function. The results indicated that the new cost function decreased the prediction error of the models by approximately 8% and 21% for the arm and leg models, respectively. We indicated that utilizing the transformer architecture, with a root-mean-square-error of 47.0 mm, exhibited ~58% more accurate long-term performance than the BLSTM-based model. This study merits the use of neural networks that capture time series dependencies in 3D motion frames, providing a unique approach for understanding and predict motion dynamics during manual material handling activities.
Authors Seyede Niloofar Hosseini, Ali Mojibi, Mahdi Mohseni, Navid Arjmand, Alireza Taheri
Comment 10 pages, 6 figures, 7 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Can Vibe Coding Beat Graduate CS Students? An LLM vs. Human Coding Tournament on Market-driven Strategic Planning
Title CN Vibe Coding能击败研究生计算机学生吗？基于市场驱动战略规划的LLM与人类编程竞赛
Keywords 大语言模型 · 代码生成 · 多智能体系统 · 战略规划 · 基准测试
Summary 本文提出一个基于真实物流优化问题（拍卖、取货与配送）的多智能体推理基准，用于评估LLM生成代码在战略规划和竞争性决策中的表现。实验对比了40个由LLM生成的代理与17个人类研究生编写的代理，结果表明人类编写的代理在竞争中显著占优，多数LLM生成的代理甚至不如简单基线，且LLM难以有效改进已有优秀方案。研究揭示了当前LLM在现实世界复杂推理驱动编码任务上的局限性。
Reason 该论文属于纯代码生成与多智能体博弈领域，研究LLM在物流优化问题中的编程能力，与文档图像理解（DIU）、视觉语言模型（VLM）处理文档内容无直接关联。其任务设定、评估方式和应用场景均不涉及OCR、文档布局分析、表格识别或视觉-文本对齐等DIU核心问题，也无法提供可迁移至文档理解领域的底层方法（如视觉表示、高分辨率处理、去幻觉机制等）。
Abstract The rapid proliferation of Large Language Models (LLMs) has revolutionized AI-assisted code generation. This rapid development of LLMs has outpaced our ability to properly benchmark them. Prevailing benchmarks emphasize unit-test pass rates and syntactic correctness. Such metrics understate the difficulty of many real-world problems that require planning, optimization, and strategic interaction. We introduce a multi-agent reasoning-driven benchmark based on a real-world logistics optimization problem (Auction, Pickup, and Delivery Problem) that couples competitive auctions with capacity-constrained routing. The benchmark requires building agents that can (i) bid strategically under uncertainty and (ii) optimize planners that deliver tasks while maximizing profit. We evaluate 40 LLM-coded agents (by a wide range of state-of-the-art LLMs under multiple prompting methodologies, including vibe coding) against 17 human-coded agents developed before the advent of LLMs. Our results over 12 double all-play-all tournaments and $\sim 40$k matches demonstrate (i) a clear superiority of human(graduate students)-coded agents: the top 5 spots are consistently won by human-coded agents, (ii) the majority of LLM-coded agents (33 out of 40) are beaten by very simple baselines, and (iii) given the best human solution as an input and prompted to improve upon, the best performing LLM makes the solution significantly worse instead of improving it. Our results highlight a gap in LLMs' ability to produce code that works competitively in the real-world, and motivate new evaluations that emphasize reasoning-driven code synthesis in real-world scenarios.
Authors Panayiotis Danassis, Naman Goel
Categories Artificial Intelligence
0
Building a Foundation Model for Trajectory from Scratch
Title CN 从零开始构建轨迹基础模型
Keywords 基础模型 · 轨迹数据 · GPT-2 · 时空建模 · 教程
Summary 本文是一篇教程性质的工作，介绍如何从GPT-2出发构建面向移动轨迹数据的基础模型，涵盖数据处理、模型适配及现有轨迹模型（如TrajFM、TrajGPT）的对比，并引入时间序列领域的patching技术。目标是为SIGSPATIAL社区提供可复现的实现范例和教育材料。
Reason 该论文聚焦于移动轨迹（mobility trajectories）领域的基础模型构建，属于时间序列与空间轨迹分析范畴，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管其方法论涉及GPT-2的改造和时序数据分块处理，但任务目标、数据模态和核心挑战均远离文档理解中的OCR幻觉、布局分析、表格识别等关键问题，无法迁移用于解决文档图像中的细粒度识别或多模态推理，属于无关领域。
Abstract Foundation models are transformative in artificial intelligence, but building them from scratch, especially for mobility trajectories, is not yet clear or documented. This tutorial bridges this gap by demonstrating the steps and code of a minimal implementation of a trajectory-focused foundation model starting from GPT-2. Through a concise, step-by-step, code-driven process, we demonstrate adapting GPT-2 for spatiotemporal data. We then review and compare representative trajectory foundation models, such as TrajFM and TrajGPT, highlighting their architectural innovations and differences. Additionally, we introduce complementary techniques from related domains, like TimesFM's patching approach. Targeted at researchers and practitioners, this tutorial aims to explain the concepts and terminology of foundation models, at the implementation level. We find it timely and indispensable to create this educational material in order to support the SIGSPATIAL community in building and evaluating mobility foundation models, enhancing both research clarity and peer-review effectiveness in mobility AI.
Authors Gaspard Merten, Mahmoud Sakr, Gilles Dejaegere
Categories Artificial Intelligence
0
On Evaluating LLM Alignment by Evaluating LLMs as Judges NeurIPS 2025
Title CN 通过评估LLM作为裁判来评价LLM对齐性
Keywords LLM对齐 · 自动评估 · 偏好学习 · LLM as a Judge · GE一致性
Summary 本文提出了一种通过让LLM担任裁判来评估其他LLM对齐性的新范式AlignEval，发现LLM的生成能力与评判能力之间存在强相关性，并利用这一特性构建无需直接分析生成结果的对齐评估基准。实验表明该方法在捕捉人类偏好方面优于AlpacaEval等主流自动评测方法。
Reason 该论文聚焦于大语言模型（LLM）的对齐性评估，属于纯NLP领域的安全/对齐研究，主要探讨如何用LLM作为裁判来衡量其他LLM是否符合人类偏好。其研究内容与文档图像理解（DIU）、视觉输入处理、OCR、多模态对齐或视觉幻觉等问题完全无关，也不涉及任何视觉或跨模态架构创新。尽管‘对齐’和‘幻觉’在广义上有一定关联，但本文未涉及视觉模态中的幻觉问题，无法迁移用于解决文档理解中的OCR错误传播或视觉-语言不一致等核心痛点。
Abstract Alignment with human preferences is an important evaluation aspect of LLMs, requiring them to be helpful, honest, safe, and to precisely follow human instructions. Evaluating large language models' (LLMs) alignment typically involves directly assessing their open-ended responses, requiring human annotators or strong LLM judges. Conversely, LLMs themselves have also been extensively evaluated as judges for assessing alignment. In this work, we examine the relationship between LLMs' generation and evaluation capabilities in aligning with human preferences. To this end, we first conduct a comprehensive analysis of the generation-evaluation consistency (GE-consistency) among various LLMs, revealing a strong correlation between their generation and evaluation capabilities when evaluated by a strong LLM preference oracle. Utilizing this finding, we propose a benchmarking paradigm that measures LLM alignment with human preferences without directly evaluating their generated outputs, instead assessing LLMs in their role as evaluators. Our evaluation shows that our proposed benchmark, AlignEval, matches or surpasses widely used automatic LLM evaluation benchmarks, such as AlpacaEval and Arena-Hard, in capturing human preferences when ranking LLMs. Our study offers valuable insights into the connection between LLMs' generation and evaluation capabilities, and introduces a benchmark that assesses alignment without directly evaluating model outputs.
Authors Yixin Liu, Pengfei Liu, Arman Cohan
Comment NeurIPS 2025 Camera Ready
Categories Artificial Intelligence, Computation and Language
0
The Driver-Blindness Phenomenon: Why Deep Sequence Models Default to Autocorrelation in Blood Glucose Forecasting
Title CN 驾驶员失明现象：深度序列模型为何在血糖预测中默认使用自相关
Keywords 时间序列预测 · 医疗AI · 自相关偏差 · 特征忽略 · 模型可解释性
Summary 论文指出，在血糖预测任务中，尽管存在胰岛素、饮食和活动等临床上有意义的输入变量，但深度序列模型仍倾向于依赖历史血糖值的自相关性，而忽略这些外部驱动因素，作者将此现象称为“驾驶员失明”（Driver-Blindness），并分析了造成该现象的三大原因：模型架构对自相关的偏好、数据质量问题以及个体间生理差异。作者提出了若干缓解策略，如生理特征编码器、因果正则化和个人化建模，并建议未来工作应报告驱动因素带来的性能增益以避免误导性SOTA。
Reason 该论文研究的是医疗时间序列预测中的特征利用问题，属于特定领域的应用型研究，且未涉及文档图像理解（DIU）或视觉-语言模型（VLM）的相关方法。其提出的问题和解决方案均无法迁移到文档理解的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）。属于典型的‘平行下游应用’，无底层方法论创新。
Abstract Deep sequence models for blood glucose forecasting consistently fail to leverage clinically informative drivers--insulin, meals, and activity--despite well-understood physiological mechanisms. We term this Driver-Blindness and formalize it via $Δ_{\text{drivers}}$, the performance gain of multivariate models over matched univariate baselines. Across the literature, $Δ_{\text{drivers}}$ is typically near zero. We attribute this to three interacting factors: architectural biases favoring autocorrelation (C1), data fidelity gaps that render drivers noisy and confounded (C2), and physiological heterogeneity that undermines population-level models (C3). We synthesize strategies that partially mitigate Driver-Blindness--including physiological feature encoders, causal regularization, and personalization--and recommend that future work routinely report $Δ_{\text{drivers}}$ to prevent driver-blind models from being considered state-of-the-art.
Authors Heman Shakeri
Comment 7 pages, 1 figure
Categories Artificial Intelligence
0
EnergyTwin: A Multi-Agent System for Simulating and Coordinating Energy Microgrids
Title CN EnergyTwin：一种用于模拟和协调能源微电网的多智能体系统
Keywords 多智能体系统 · 微电网 · 数字孪生 · 能源调度 · 分布式能源
Summary 本文提出EnergyTwin，一个基于多智能体的微电网仿真环境，将物理建模与预测驱动的滚动规划及合约式协商机制结合，用于提升微电网的能源自给能力和运行韧性。每个能源设备被建模为独立智能体，通过中央智能体进行预测与资源分配。在大学校园微电网场景中验证了其有效性。
Reason 该论文属于具身智能与能源系统的交叉应用，研究领域为多智能体在微电网中的调度与仿真，完全不涉及文档图像理解（DIU）、视觉语言模型（VLM）或任何与OCR、文本识别、版面分析相关的技术。属于典型的‘平行应用’型工作，且应用场景与文档无关，无法迁移至DIU领域解决核心痛点如幻觉、复杂排版或长文档推理。
Abstract Microgrids are deployed to reduce purchased grid energy, limit exposure to volatile tariffs, and ensure service continuity during disturbances. This requires coordinating heterogeneous distributed energy resources across multiple time scales and under variable conditions. Among existing tools, typically, power-system simulators capture physical behaviour but assume centralized control, while multi-agent frameworks model decentralized decision-making but represent energy with no physical grounding. In this context, the EnergyTwin is introduced, an agent-based microgrid simulation environment that couples physically grounded models with forecast-informed, rolling-horizon planning, and negotiations. Each asset is modeled as an agent, interacting with a central agent that obtains forecasts, formulates predictions, and allocates energy through contract-based interactions. EnergyTwin targets tertiary-layer decision making and is extensible for digital-twin use. Its feasibility was evaluated in a university campus microgrid scenario where multiple planning strategies were compared. Achieved results show that forecast-driven rolling-horizon planning increases local energy self-sufficiency, maintains higher battery reserves, and reduces exposure to low-resilience operating states. They demonstrate also potential of EnergyTwin as platform supporting research on resilient, negotiation-driven microgrids.
Authors Jakub Muszyński, Ignacy Walużenicz, Patryk Zan, Zofia Wrona, Maria Ganzha, Marcin Paprzycki, Costin Bădică
Categories Artificial Intelligence
0
Gated Uncertainty-Aware Runtime Dual Invariants for Neural Signal-Controlled Robotics NeurIPS 2025
Title CN 用于神经信号控制机器人系统的门控不确定性感知运行时双重不变量
Keywords 神经信号解码 · 运行时验证 · 安全性保障 · 神经符号系统 · 脑机接口
Summary 本文提出GUARDIAN框架，用于神经信号控制的机器人系统在运行时进行安全性保障，结合神经解码与符号化目标 grounding，实现对用户意图的可验证解码。系统在低准确率、高不确定性条件下仍能维持94-97%的安全率，并具备毫秒级响应能力，适用于闭环脑机接口系统。
Reason 该论文属于具身智能与神经接口领域，研究的是基于脑电信号（EEG）的机器人控制与安全验证，与文档图像理解（DIU）、VLM或多模态文档处理完全无关。尽管其提出了运行时验证和不确定性建模机制，但应用场景和技术目标均无法迁移到DocVQA或文档理解的核心痛点（如OCR幻觉、复杂排版、长文档推理等）。
Abstract Safety-critical assistive systems that directly decode user intent from neural signals require rigorous guarantees of reliability and trust. We present GUARDIAN (Gated Uncertainty-Aware Runtime Dual Invariants), a framework for real-time neuro-symbolic verification for neural signal-controlled robotics. GUARDIAN enforces both logical safety and physiological trust by coupling confidence-calibrated brain signal decoding with symbolic goal grounding and dual-layer runtime monitoring. On the BNCI2014 motor imagery electroencephalogram (EEG) dataset with 9 subjects and 5,184 trials, the system performs at a high safety rate of 94-97% even with lightweight decoder architectures with low test accuracies (27-46%) and high ECE confidence miscalibration (0.22-0.41). We demonstrate 1.7x correct interventions in simulated noise testing versus at baseline. The monitor operates at 100Hz and sub-millisecond decision latency, making it practically viable for closed-loop neural signal-based systems. Across 21 ablation results, GUARDIAN exhibits a graduated response to signal degradation, and produces auditable traces from intent, plan to action, helping to link neural evidence to verifiable robot action.
Authors Tasha Kim, Oiwi Parker Jones
Comment Embodied and Safe-Assured Robotic Systems workshop at NeurIPS 2025
Categories Artificial Intelligence
0
Time-Domain Linear Model-based Framework for Passive Acoustic Mapping of Cavitation Activity
Title CN 基于时域线性模型的被动声学映射框架用于空化活动检测
Keywords 被动声学映射 · 空化活动 · 时域方法 · 线性模型 · 正则化反演
Summary 本文提出了一种完全在时域中构建的线性模型驱动的波束形成框架，用于被动声学映射空化活动。该方法通过建立离散化的时空空化分布与探头记录信号之间的前向模型，并显式考虑飞行时间延迟，结合空间和时间域的先验知识进行正则化反演。实验表明，该方法仅需传统频域方法20%的数据即可实现更优或相当的成像质量，具有更高的数据效率和适应性。
Reason 该论文属于医学信号处理领域，研究的是超声治疗中的空化活动被动声学成像问题，与文档图像理解（DIU）、多模态大模型（VLM）或视觉语言任务无关。尽管其方法涉及时间域建模和正则化反演，但应用场景和技术路线均无法迁移到文档理解的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）。属于完全无关领域。
Abstract Passive acoustic mapping enables the spatial mapping and temporal monitoring of cavitation activity, playing a crucial role in therapeutic ultrasound applications. Most conventional beamforming methods, whether implemented in the time or frequency domains, suffer from limited axial resolution due to the absence of a reference emission onset time. While frequency-domain methods, the most efficient of which are based on the cross-spectral matrix, require long signals for accurate estimation, time-domain methods typically achieve lower spatial resolution. To address these limitations, we propose a linear model-based beamforming framework fully formulated in the time domain. The linear forward model relates a discretized spatiotemporal distribution of cavitation activity to the temporal signals recorded by a probe, explicitly accounting for time-of-flight delays dictated by the acquisition geometry. This model is then inverted using regularization techniques that exploit prior knowledge of cavitation activity in both spatial and temporal domains. Experimental results show that the proposed framework achieves enhanced or competitive cavitation map quality while using only 20\% of the data typically required by frequency-domain methods. This highlights the substantial gain in data efficiency and the flexibility of our spatiotemporal regularization to adapt to diverse passive cavitation scenarios, outperforming state-of-the-art techniques.
Authors Tatiana Gelvez-Barrera, Barbara Nicolas, Denis Kouamé, Bruno Gilles, Adrian Basarab
Categories Artificial Intelligence
0
Flash-DMD: Towards High-Fidelity Few-Step Image Generation with Efficient Distillation and Joint Reinforcement Learning
Title CN Flash-DMD：通过高效蒸馏和联合强化学习实现高保真少步图像生成
Keywords 扩散模型 · 图像生成 · 知识蒸馏 · 强化学习 · 少步采样
Summary 本文提出Flash-DMD框架，结合高效的时序感知蒸馏策略与联合强化学习微调，以加速扩散模型的训练并提升少步生成下的图像质量与文本-图像对齐能力。蒸馏过程大幅降低训练成本，而持续的蒸馏损失作为正则化项稳定了RL训练，防止奖励劫持。实验表明其在多种生成模型上达到SOTA效果。
Reason 该论文属于纯图像生成领域，研究的是扩散模型的加速生成与强化学习优化，核心目标是提升生成图像的质量与效率。尽管涉及多模态中的视觉生成，但其方法完全不针对文档结构、OCR、排版理解或文档VQA等DIU核心问题。且‘few-step图像生成’与文档理解无直接迁移路径，属于无关领域（视频/图像生成）的工作，命中负面清单第2条。
Abstract Diffusion Models have emerged as a leading class of generative models, yet their iterative sampling process remains computationally expensive. Timestep distillation is a promising technique to accelerate generation, but it often requires extensive training and leads to image quality degradation. Furthermore, fine-tuning these distilled models for specific objectives, such as aesthetic appeal or user preference, using Reinforcement Learning (RL) is notoriously unstable and easily falls into reward hacking. In this work, we introduce Flash-DMD, a novel framework that enables fast convergence with distillation and joint RL-based refinement. Specifically, we first propose an efficient timestep-aware distillation strategy that significantly reduces training cost with enhanced realism, outperforming DMD2 with only $2.1\%$ its training cost. Second, we introduce a joint training scheme where the model is fine-tuned with an RL objective while the timestep distillation training continues simultaneously. We demonstrate that the stable, well-defined loss from the ongoing distillation acts as a powerful regularizer, effectively stabilizing the RL training process and preventing policy collapse. Extensive experiments on score-based and flow matching models show that our proposed Flash-DMD not only converges significantly faster but also achieves state-of-the-art generation quality in the few-step sampling regime, outperforming existing methods in visual quality, human preference, and text-image alignment metrics. Our work presents an effective paradigm for training efficient, high-fidelity, and stable generative models. Codes are coming soon.
Authors Guanjie Chen, Shirui Huang, Kai Liu, Jianchen Zhu, Xiaoye Qu, Peng Chen, Yu Cheng, Yifu Sun
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
New York Smells: A Large Multimodal Dataset for Olfaction
Title CN 纽约的气味：一种用于嗅觉的大型多模态数据集
Keywords 多模态数据集 · 嗅觉感知 · 跨模态检索 · 表示学习 · 野外采集
Summary 本文提出了一个名为“纽约的气味”的大规模多模态数据集，包含7000个真实场景下的气味-图像配对数据，旨在推动机器对嗅觉的理解。数据集支持跨模态检索、场景/物体/材质识别以及细粒度草种区分任务。实验表明，视觉数据有助于学习嗅觉表示，且所学表示优于传统手工特征。
Reason 该论文聚焦于嗅觉与视觉的多模态感知，属于无关领域（如环境气味识别），与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管涉及多模态数据，但其感官模态（气味信号）无法迁移至解决OCR幻觉、复杂排版或长文档推理等DIU核心问题。
Abstract While olfaction is central to how animals perceive the world, this rich chemical sensory modality remains largely inaccessible to machines. One key bottleneck is the lack of diverse, multimodal olfactory training data collected in natural settings. We present New York Smells, a large dataset of paired image and olfactory signals captured ``in the wild.'' Our dataset contains 7,000 smell-image pairs from 3,500 distinct objects across indoor and outdoor environments, with approximately 70$\times$ more objects than existing olfactory datasets. Our benchmark has three tasks: cross-modal smell-to-image retrieval, recognizing scenes, objects, and materials from smell alone, and fine-grained discrimination between grass species. Through experiments on our dataset, we find that visual data enables cross-modal olfactory representation learning, and that our learned olfactory representations outperform widely-used hand-crafted features.
Authors Ege Ozguroglu, Junbang Liang, Ruoshi Liu, Mia Chiquier, Michael DeTienne, Wesley Wei Qian, Alexandra Horowitz, Andrew Owens, Carl Vondrick
Comment Project website at https://smell.cs.columbia.edu
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Automated Monitoring of Cultural Heritage Artifacts Using Semantic Segmentation
Title CN 使用语义分割实现文化遗产文物的自动化监测
Keywords Cultural Heritage · Monitoring · Deep Learning · U-Nets · Semantic Segmentation
Summary 本文研究利用U-Net架构结合不同CNN编码器进行文化遗产雕像和纪念碑上的像素级裂缝检测，在OmniCrack30k数据集上进行了定量评估，并在真实世界未标注数据上测试泛化能力。结果显示模型对未见场景具有一定的泛化性。
Reason 该论文属于纯图像分割任务，研究领域为文化遗产中的裂缝检测，尽管使用了U-Net等CV模型，但其任务与文档图像理解（DIU）无关。未涉及文本识别、布局分析、表格理解或VLM在文档上的应用，属于无关领域（如工业检测或医学影像式缺陷识别）的平行应用，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract This paper addresses the critical need for automated crack detection in the preservation of cultural heritage through semantic segmentation. We present a comparative study of U-Net architectures, using various convolutional neural network (CNN) encoders, for pixel-level crack identification on statues and monuments. A comparative quantitative evaluation is performed on the test set of the OmniCrack30k dataset [1] using popular segmentation metrics including Mean Intersection over Union (mIoU), Dice coefficient, and Jaccard index. This is complemented by an out-of-distribution qualitative evaluation on an unlabeled test set of real-world cracked statues and monuments. Our findings provide valuable insights into the capabilities of different CNN- based encoders for fine-grained crack segmentation. We show that the models exhibit promising generalization capabilities to unseen cultural heritage contexts, despite never having been explicitly trained on images of statues or monuments.
Authors Andrea Ranieri, Giorgio Palmieri, Silvia Biasotti
Comment Keywords: Cultural Heritage, Monitoring, Deep Learning, U-Nets, Semantic Segmentation
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Proceedings Twentieth Conference on Theoretical Aspects of Rationality and Knowledge
Title CN 第二十届理性与知识理论方面会议论文集
Keywords 理论计算机科学 · 知识逻辑 · 多智能体系统 · 理性推理 · 语义模型
Summary 本文是TARK 2025会议的论文集前言，介绍了会议背景、历史和主题范围，涵盖知识表示、信念修正、博弈论中的认知建模等理论议题。未提出任何可用于文档图像理解的具体算法、模型或技术路径。
Reason 该论文为会议文集介绍，不涉及具体技术方法或文档图像理解（DIU）相关研究。内容聚焦于理性与知识的理论探讨，属于逻辑、哲学与多智能体系统基础理论范畴，无直接可迁移至DIU的技术创新点，如视觉表示学习、VLM架构改进、去幻觉机制等。
Abstract The TARK conference (Theoretical Aspects of Rationality and Knowledge) is a conference that aims to bring together researchers from a wide variety of fields, including computer science, artificial intelligence, game theory, decision theory, philosophy, logic, linguistics, and cognitive science. Its goal is to further our understanding of interdisciplinary issues involving reasoning about rationality and knowledge. Previous conferences have been held biennially around the world since 1986, on the initiative of Joe Halpern (Cornell University). Topics of interest include, but are not limited to, semantic models for knowledge, belief, uncertainty, awareness, bounded rationality, common sense epistemic reasoning, epistemic logic, epistemic game theory, knowledge and action, applications of reasoning about knowledge and other mental states, belief revision, computational social choice, algorithmic game theory, and foundations of multi-agent systems. Information about TARK is available at http://www.tark.org/. These proceedings contain the papers that have been accepted for presentation at the Twentieth Conference on Theoretical Aspects of Rationality and Knowledge (TARK 2025), held July 14--16, 2025, at Heinrich-Heine-Universität, Düsseldorf, Germany. The conference website can be found at https://ccc.cs.uni-duesseldorf.de/tark-2025/.
Authors Adam Bjorndahl
Categories Artificial Intelligence
0
MIMIC-MJX: Neuromechanical Emulation of Animal Behavior
Title CN MIMIC-MJX：动物行为的神经力学仿真
Keywords 神经控制 · 行为建模 · 物理仿真 · 运动生成 · 计算神经科学
Summary 本文提出MIMIC-MJX框架，用于从动物行为的运动轨迹中学习生物可解释的神经控制策略，并通过驱动逼真的生物力学模型在物理仿真中复现真实动作。该方法强调神经控制器与身体动态之间的耦合，适用于神经科学研究中的行为建模与实验模拟。
Reason 该论文属于具身智能与神经科学交叉领域，研究通过物理仿真从运动学数据中学习生物合理的神经控制策略。其核心关注点为动物行为模拟与神经控制建模，与文档图像理解（DIU）、视觉语言模型（VLM）在文档上的应用、OCR、布局分析等完全无关。尽管涉及‘神经控制’和‘仿真’，但技术路径无法迁移至解决文档理解中的幻觉、密集文本、复杂排版或长文档推理等核心痛点。
Abstract The primary output of the nervous system is movement and behavior. While recent advances have democratized pose tracking during complex behavior, kinematic trajectories alone provide only indirect access to the underlying control processes. Here we present MIMIC-MJX, a framework for learning biologically-plausible neural control policies from kinematics. MIMIC-MJX models the generative process of motor control by training neural controllers that learn to actuate biomechanically-realistic body models in physics simulation to reproduce real kinematic trajectories. We demonstrate that our implementation is accurate, fast, data-efficient, and generalizable to diverse animal body models. Policies trained with MIMIC-MJX can be utilized to both analyze neural control strategies and simulate behavioral experiments, illustrating its potential as an integrative modeling framework for neuroscience.
Authors Charles Y. Zhang, Yuanjia Yang, Aidan Sirbu, Elliott T. T. Abe, Emil Wärnberg, Eric J. Leonardis, Diego E. Aldarondo, Adam Lee, Aaditya Prasad, Jason Foat, Kaiwen Bian, Joshua Park, Rusham Bhatt, Hutton Saunders, Akira Nagamori, Ayesha R. Thanawalla, Kee Wui Huang, Fabian Plum, Hendrik K. Beck, Steven W. Flavell, David Labonte, Blake A. Richards, Bingni W. Brunton, Eiman Azim, Bence P. Ölveczky, Talmo D. Pereira
Categories Artificial Intelligence
0
Assessing LLMs' Performance: Insights from the Chinese Pharmacist Exam
Title CN 评估大语言模型在中医师考试中的表现：来自中国执业药师考试的启示
Keywords 大语言模型 · 执业药师考试 · 性能评估 · 文本问答 · 领域适应
Summary 本文评估了ChatGPT-4o和DeepSeek-R1在中国执业药师资格考试（2017–2021年）中的表现，使用2306道纯文本选择题进行测试，发现DeepSeek-R1整体准确率显著高于ChatGPT-4o（90.0% vs 76.1%）。研究结论强调领域专用模型的潜力及人类监督的重要性，但未涉及任何形式的文档图像理解或多模态处理。
Reason 该论文属于典型的‘平行下游应用’，即将现有LLM应用于特定领域（药师考试）的选择题回答任务，未提出任何文档图像理解（DIU）相关的方法论创新。任务为纯文本问答，明确排除了含表格和图像的题目，完全脱离多模态与视觉理解范畴。研究未涉及OCR、布局分析、视觉定位或文档结构建模等DIU核心问题，仅对比两个模型在标准化考试中的准确率，属于垂直场景性能评测，无技术迁移价值。
Abstract Background: As large language models (LLMs) become increasingly integrated into digital health education and assessment workflows, their capabilities in supporting high-stakes, domain-specific certification tasks remain underexplored.In China, the national pharmacist licensure exam serves as a standardized benchmark for evaluating pharmacists' clinical and theoretical competencies. Objective: This study aimed to compare the performance of two LLMs: ChatGPT-4o and DeepSeek-R1 on real questions from the Chinese Pharmacist Licensing Examination (2017-2021), and to discuss the implications of these performance differences for AI-enabled formative evaluation. Methods: A total of 2,306 multiple-choice (text-only) questions were compiled from official exams, training materials, and public databases. Questions containing tables or images were excluded. Each item was input in its original Chinese format, and model responses were evaluated for exact accuracy. Pearson's Chi-squared test was used to compare overall performance, and Fisher's exact test was applied to year-wise multiple-choice accuracy. Results: DeepSeek-R1 outperformed ChatGPT-4o with a significantly higher overall accuracy (90.0% vs. 76.1%, p < 0.001). Unit-level analyses revealed consistent advantages for DeepSeek-R1, particularly in foundational and clinical synthesis modules. While year-by-year multiple-choice performance also favored DeepSeek-R1, this performance gap did not reach statistical significance in any specific unit-year (all p > 0.05). Conclusion: DeepSeek-R1 demonstrated robust alignment with the structural and semantic demands of the pharmacist licensure exam. These findings suggest that domain-specific models warrant further investigation for this context, while also reinforcing the necessity of human oversight in legally and ethically sensitive contexts.
Authors Xinran Wang, Boran Zhu, Shujuan Zhou, Ziwen Long, Dehua Zhou, Shu Zhang
Comment 15 pages, 4 figures
Categories Artificial Intelligence
0
LoRA-based methods on Unet for transfer learning in Subarachnoid Hematoma Segmentation
Title CN 基于LoRA的Unet在蛛网膜下腔血肿分割中的迁移学习方法
Keywords LoRA · Unet · 医学图像分割 · 迁移学习 · CP分解
Summary 本文提出了一种基于LoRA的Unet架构用于蛛网膜下腔血肿（SAH）的CT图像分割，通过在创伤性脑损伤数据上预训练并在SAH数据上微调，探索了多种LoRA变体（如CP-LoRA、DoRA-C等）在医学图像迁移学习中的表现。实验表明LoRA方法优于传统微调方式，尤其在参数效率方面有优势。
Reason 该论文研究的是医学图像分割任务，属于纯医学影像分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。其核心方法虽涉及LoRA和参数高效微调，但应用场景为脑出血分割，无法直接迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心问题。属于无关领域。
Abstract Aneurysmal subarachnoid hemorrhage (SAH) is a life-threatening neurological emergency with mortality rates exceeding 30%. Transfer learning from related hematoma types represents a potentially valuable but underexplored approach. Although Unet architectures remain the gold standard for medical image segmentation due to their effectiveness on limited datasets, Low-Rank Adaptation (LoRA) methods for parameter-efficient transfer learning have been rarely applied to convolutional neural networks in medical imaging contexts. We implemented a Unet architecture pre-trained on computed tomography scans from 124 traumatic brain injury patients across multiple institutions, then fine-tuned on 30 aneurysmal SAH patients from the University of Michigan Health System using 3-fold cross-validation. We developed a novel CP-LoRA method based on tensor CP-decomposition and introduced DoRA variants (DoRA-C, convDoRA, CP-DoRA) that decompose weight matrices into magnitude and directional components. We compared these approaches against existing LoRA methods (LoRA-C, convLoRA) and standard fine-tuning strategies across different modules on a multi-view Unet model. LoRA-based methods consistently outperformed standard Unet fine-tuning. Performance varied by hemorrhage volume, with all methods showing improved accuracy for larger volumes. CP-LoRA achieved comparable performance to existing methods while using significantly fewer parameters. Over-parameterization with higher ranks consistently yielded better performance than strictly low-rank adaptations. This study demonstrates that transfer learning between hematoma types is feasible and that LoRA-based methods significantly outperform conventional Unet fine-tuning for aneurysmal SAH segmentation.
Authors Cristian Minoccheri, Matthew Hodgman, Haoyuan Ma, Rameez Merchant, Emily Wittrup, Craig Williamson, Kayvan Najarian
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
CGCE: Classifier-Guided Concept Erasure in Generative Models
Title CN CGCE：生成模型中的分类器引导概念擦除
Keywords 概念擦除 · 生成模型安全 · 文本到图像生成 · 红队攻击防御 · 提示修正
Summary 本文提出Classifier-Guided Concept Erasure（CGCE），一种无需修改生成模型权重的插件式概念擦除框架。通过在推理阶段使用轻量级分类器检测并修正包含不安全概念的文本嵌入，实现对T2I和T2V模型的安全控制。该方法在对抗红队攻击方面表现出强鲁棒性，同时保持了良性内容的生成质量。
Reason 该论文聚焦于生成模型中的安全内容控制，属于图像/视频生成领域的安全对齐问题，主要解决有害概念的擦除与防御红队攻击。尽管涉及多模态模型，但其目标与文档图像理解（DIU）的核心任务如OCR、布局分析、表格识别等无直接关联。方法本质上是面向内容安全的干预机制，迁移到DIU场景（如消除OCR幻觉或处理复杂排版）极为困难，且未针对文档结构、细粒度定位或长文本推理设计。属于无关领域中的纯生成模型安全研究。
Abstract Recent advancements in large-scale generative models have enabled the creation of high-quality images and videos, but have also raised significant safety concerns regarding the generation of unsafe content. To mitigate this, concept erasure methods have been developed to remove undesirable concepts from pre-trained models. However, existing methods remain vulnerable to adversarial attacks that can regenerate the erased content. Moreover, achieving robust erasure often degrades the model's generative quality for safe, unrelated concepts, creating a difficult trade-off between safety and performance. To address this challenge, we introduce Classifier-Guided Concept Erasure (CGCE), an efficient plug-and-play framework that provides robust concept erasure for diverse generative models without altering their original weights. CGCE uses a lightweight classifier operating on text embeddings to first detect and then refine prompts containing undesired concepts. This approach is highly scalable, allowing for multi-concept erasure by aggregating guidance from several classifiers. By modifying only unsafe embeddings at inference time, our method prevents harmful content generation while preserving the model's original quality on benign prompts. Extensive experiments show that CGCE achieves state-of-the-art robustness against a wide range of red-teaming attacks. Our approach also maintains high generative utility, demonstrating a superior balance between safety and performance. We showcase the versatility of CGCE through its successful application to various modern T2I and T2V models, establishing it as a practical and effective solution for safe generative AI.
Authors Viet Nguyen, Vishal M. Patel
Comment 26 pages, 17 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
ExDDV: A New Dataset for Explainable Deepfake Detection in Video WACV 2026
Title CN ExDDV：一种用于视频中可解释深度伪造检测的新数据集
Keywords Deepfake Detection · Explainable AI · Video Dataset · Vision-Language Model · Artifact Localization
Summary 本文提出了ExDDV，首个面向视频中可解释深度伪造检测的数据集，包含约5400个真实与伪造视频，均配有文本描述和点击标注以解释伪造痕迹。作者评估了多种视觉-语言模型在该数据集上的表现，发现需要文本与点击双重监督才能实现对伪造区域的准确定位与描述。数据集和代码已公开。
Reason 该论文聚焦于视频领域的深度伪造检测与解释，属于视频理解范畴，与文档图像理解（DIU）任务无直接关联。尽管涉及视觉-语言模型的应用，但其核心问题（深伪视频检测、人工标注点击与描述）无法迁移至解决文档理解中的OCR幻觉、复杂排版或长文档推理等痛点。
Abstract The ever growing realism and quality of generated videos makes it increasingly harder for humans to spot deepfake content, who need to rely more and more on automatic deepfake detectors. However, deepfake detectors are also prone to errors, and their decisions are not explainable, leaving humans vulnerable to deepfake-based fraud and misinformation. To this end, we introduce ExDDV, the first dataset and benchmark for Explainable Deepfake Detection in Video. ExDDV comprises around 5.4K real and deepfake videos that are manually annotated with text descriptions (to explain the artifacts) and clicks (to point out the artifacts). We evaluate a number of vision-language models on ExDDV, performing experiments with various fine-tuning and in-context learning strategies. Our results show that text and click supervision are both required to develop robust explainable models for deepfake videos, which are able to localize and describe the observed artifacts. Our novel dataset and code to reproduce the results are available at https://github.com/vladhondru25/ExDDV.
Authors Vlad Hondru, Eduard Hogea, Darian Onchis, Radu Tudor Ionescu
Comment Accepted at WACV 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
0
DesignPref: Capturing Personal Preferences in Visual Design Generation
Title CN DesignPref：捕捉视觉设计生成中的个人偏好
Keywords 视觉设计生成 · 个性化建模 · 偏好数据集 · RAG · 细粒度评估
Summary 本文提出了DesignPref数据集，包含12,000对由20名专业设计师标注的UI设计成对比较数据，揭示了设计偏好中存在高度个体差异性，并探索了基于微调或RAG的个性化建模方法以更好预测个体偏好。研究表明传统多数投票聚合模型难以准确反映个体选择。
Reason 该论文聚焦于UI和演示文稿等视觉设计生成中的个性化偏好建模，属于创意设计生成领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其数据集构建和个性化建模方法具有一定方法论价值，但其研究问题本质是‘主观审美偏好’的建模，而非解决文档理解中的OCR幻觉、密集文本、复杂排版或长文档推理等客观语义解析问题。该工作属于特定领域的生成应用，无法迁移为提升DIU性能的底层技术工具。
Abstract Generative models, such as large language models and text-to-image diffusion models, are increasingly used to create visual designs like user interfaces (UIs) and presentation slides. Finetuning and benchmarking these generative models have often relied on datasets of human-annotated design preferences. Yet, due to the subjective and highly personalized nature of visual design, preference varies widely among individuals. In this paper, we study this problem by introducing DesignPref, a dataset of 12k pairwise comparisons of UI design generation annotated by 20 professional designers with multi-level preference ratings. We found that among trained designers, substantial levels of disagreement exist (Krippendorff's alpha = 0.25 for binary preferences). Natural language rationales provided by these designers indicate that disagreements stem from differing perceptions of various design aspect importance and individual preferences. With DesignPref, we demonstrate that traditional majority-voting methods for training aggregated judge models often do not accurately reflect individual preferences. To address this challenge, we investigate multiple personalization strategies, particularly fine-tuning or incorporating designer-specific annotations into RAG pipelines. Our results show that personalized models consistently outperform aggregated baseline models in predicting individual designers' preferences, even when using 20 times fewer examples. Our work provides the first dataset to study personalized visual design evaluation and support future research into modeling individual design taste.
Authors Yi-Hao Peng, Jeffrey P. Bigham, Jason Wu
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
0
The Text Aphasia Battery (TAB): A Clinically-Grounded Benchmark for Aphasia-Like Deficits in Language Models
Title CN 文本失语症电池（TAB）：一种基于临床的语言模型失语样缺陷基准
Keywords 语言模型评估 · 失语症 · 文本基准 · 临床语言学 · 自动化评分
Summary 本文提出了一种名为Text Aphasia Battery（TAB）的纯文本基准，用于评估大语言模型中类似人类失语症的语言缺陷。该基准改编自临床工具QAB，包含连贯文本、词汇理解、句子理解和重复四个子测试，并采用Gemini模型进行自动化评分验证。研究目标是推动对语言模型语言障碍的系统性分析。
Reason 该论文聚焦于语言模型在文本层面的失语样行为评估，属于纯NLP范畴，且任务设计源于临床神经语言学，与文档图像理解（DIU）、多模态文档分析、视觉-语言建模或OCR等核心问题完全无关。未涉及任何视觉输入、布局结构、表格识别或VLM在文档上的应用，无法迁移解决DIU中的OCR幻觉、密集文本、复杂排版或长文档推理等痛点。
Abstract Large language models (LLMs) have emerged as a candidate "model organism" for human language, offering an unprecedented opportunity to study the computational basis of linguistic disorders like aphasia. However, traditional clinical assessments are ill-suited for LLMs, as they presuppose human-like pragmatic pressures and probe cognitive processes not inherent to artificial architectures. We introduce the Text Aphasia Battery (TAB), a text-only benchmark adapted from the Quick Aphasia Battery (QAB) to assess aphasic-like deficits in LLMs. The TAB comprises four subtests: Connected Text, Word Comprehension, Sentence Comprehension, and Repetition. This paper details the TAB's design, subtests, and scoring criteria. To facilitate large-scale use, we validate an automated evaluation protocol using Gemini 2.5 Flash, which achieves reliability comparable to expert human raters (prevalence-weighted Cohen's kappa = 0.255 for model--consensus agreement vs. 0.286 for human--human agreement). We release TAB as a clinically-grounded, scalable framework for analyzing language deficits in artificial systems.
Authors Nathan Roll, Jill Kries, Flora Jin, Catherine Wang, Ann Marie Finley, Meghan Sumner, Cory Shain, Laura Gwilliams
Categories Artificial Intelligence, Computation and Language
0
From One Attack Domain to Another: Contrastive Transfer Learning with Siamese Networks for APT Detection
Title CN 从一个攻击领域到另一个：基于Siamese网络的对比迁移学习用于APT检测
Keywords APT检测 · 迁移学习 · Siamese网络 · 对比学习 · 可解释AI
Summary 本文提出一种融合迁移学习、可解释AI和对比学习的混合框架，用于提升APT攻击检测在跨域场景下的泛化能力。通过注意力自编码器实现知识迁移，SHAP选择稳定特征，Siamese网络对齐源域和目标域表征，增强异常可分性。实验基于DARPA TC项目的真实痕迹，并合成攻击场景验证鲁棒性，结果优于传统和深度基线方法。
Reason 该论文属于网络安全领域的APT（高级持续性威胁）检测研究，核心方法为结合迁移学习、可解释AI与Siamese网络的异常检测框架。其任务目标、数据模态（系统调用轨迹等）与文档图像理解（DIU）完全无关，无法迁移到OCR幻觉缓解、细粒度文本定位或复杂版面分析等DIU核心问题。
Abstract Advanced Persistent Threats (APT) pose a major cybersecurity challenge due to their stealth, persistence, and adaptability. Traditional machine learning detectors struggle with class imbalance, high dimensional features, and scarce real world traces. They often lack transferability-performing well in the training domain but degrading in novel attack scenarios. We propose a hybrid transfer framework that integrates Transfer Learning, Explainable AI (XAI), contrastive learning, and Siamese networks to improve cross-domain generalization. An attention-based autoencoder supports knowledge transfer across domains, while Shapley Additive exPlanations (SHAP) select stable, informative features to reduce dimensionality and computational cost. A Siamese encoder trained with a contrastive objective aligns source and target representations, increasing anomaly separability and mitigating feature drift. We evaluate on real-world traces from the DARPA Transparent Computing (TC) program and augment with synthetic attack scenarios to test robustness. Across source to target transfers, the approach delivers improved detection scores with classical and deep baselines, demonstrating a scalable, explainable, and transferable solution for APT detection.
Authors Sidahmed Benabderrahmane, Talal Rahwan
Categories Artificial Intelligence
0
Quantifying the Privacy Implications of High-Fidelity Synthetic Network Traffic
Title CN 量化高保真合成网络流量的隐私影响
Keywords 合成网络流量 · 隐私泄露 · 成员推断攻击 · 生成模型 · 数据安全
Summary 本文针对网络流量数据稀缺和隐私问题，研究了多种生成模型产生的合成流量的隐私风险，提出了一套综合性的隐私度量方法，结合成员推断攻击和网络特定标识符恢复能力进行评估。结果显示不同模型在隐私泄露方面差异显著，部分模型可导致高达100%的网络标识符被恢复，揭示了当前合成流量存在的严重隐私漏洞。
Reason 该论文研究的是网络流量数据的隐私问题，属于网络安全与生成模型交叉领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其核心关注点为合成数据的隐私泄露测量（如成员推断攻击、数据提取攻击），并未涉及OCR、布局分析、表格识别、多模态文档理解等DIU任务，也无法迁移为解决文档理解中OCR幻觉、复杂排版或长文档推理等问题的技术工具。
Abstract To address the scarcity and privacy concerns of network traffic data, various generative models have been developed to produce synthetic traffic. However, synthetic traffic is not inherently privacy-preserving, and the extent to which it leaks sensitive information, and how to measure such leakage, remain largely unexplored. This challenge is further compounded by the diversity of model architectures, which shape how traffic is represented and synthesized. We introduce a comprehensive set of privacy metrics for synthetic network traffic, combining standard approaches like membership inference attacks (MIA) and data extraction attacks with network-specific identifiers and attributes. Using these metrics, we systematically evaluate the vulnerability of different representative generative models and examine the factors that influence attack success. Our results reveal substantial variability in privacy risks across models and datasets. MIA success ranges from 0% to 88%, and up to 100% of network identifiers can be recovered from generated traffic, highlighting serious privacy vulnerabilities. We further identify key factors that significantly affect attack outcomes, including training data diversity and how well the generative model fits the training data. These findings provide actionable guidance for designing and deploying generative models that minimize privacy leakage, establishing a foundation for safer synthetic network traffic generation.
Authors Van Tran, Shinan Liu, Tian Li, Nick Feamster
Comment 14 pages, 13 Figures, 6 Tables
Categories Artificial Intelligence
0
Ranking-Enhanced Anomaly Detection Using Active Learning-Assisted Attention Adversarial Dual AutoEncoders
Title CN 基于主动学习辅助注意力对抗双自编码器的排序增强异常检测
Keywords 异常检测 · 主动学习 · 自编码器 · 网络安全 · 高级持续性威胁
Summary 本文提出了一种结合主动学习和注意力对抗双自编码器（AADA）的异常检测框架，用于检测高级持续性威胁（APT）。通过在不确定性样本上主动查询标签，减少标注成本的同时提升检测性能。实验在DARPA透明计算项目的真实溯源轨迹数据库上进行，涵盖多种操作系统和攻击场景，结果表明该方法在极不平衡的数据下仍能显著提升检测率。
Reason 该论文研究的是网络安全领域的高级持续性威胁（APT）检测问题，属于纯人工智能在安全场景的应用，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。尽管使用了自编码器和主动学习等方法，但其技术路线针对的是系统日志或溯源图等非文档视觉数据，无法迁移到OCR、排版分析、表格识别等DIU核心任务中。
Abstract Advanced Persistent Threats (APTs) pose a significant challenge in cybersecurity due to their stealthy and long-term nature. Modern supervised learning methods require extensive labeled data, which is often scarce in real-world cybersecurity environments. In this paper, we propose an innovative approach that leverages AutoEncoders for unsupervised anomaly detection, augmented by active learning to iteratively improve the detection of APT anomalies. By selectively querying an oracle for labels on uncertain or ambiguous samples, we minimize labeling costs while improving detection rates, enabling the model to improve its detection accuracy with minimal data while reducing the need for extensive manual labeling. We provide a detailed formulation of the proposed Attention Adversarial Dual AutoEncoder-based anomaly detection framework and show how the active learning loop iteratively enhances the model. The framework is evaluated on real-world imbalanced provenance trace databases produced by the DARPA Transparent Computing program, where APT-like attacks constitute as little as 0.004\% of the data. The datasets span multiple operating systems, including Android, Linux, BSD, and Windows, and cover two attack scenarios. The results have shown significant improvements in detection rates during active learning and better performance compared to other existing approaches.
Authors Sidahmed Benabderrahmane, James Cheney, Talal Rahwan
Categories Artificial Intelligence
0
Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Model IJCNN 2025
Title CN 基于潜在扩散模型的高效快速生成式歌声分离
Keywords 歌声分离 · 扩散模型 · 潜在空间 · 音频生成 · 音乐信号处理
Summary 本文提出了一种基于潜在扩散模型的生成式歌声分离方法，仅需清唱人声与混合音频的配对数据进行训练。通过在紧凑的潜在空间中生成样本并解码为音频，实现了高效的优化和更快的推理速度。实验表明，该方法优于现有的生成式分离系统，并在信号质量与干扰去除方面媲美非生成式系统。作者还提供了关于潜在编码器噪声鲁棒性的分析，并开源了模块化工具包。
Reason 该论文研究的是音乐信号中的歌声分离任务，属于音频处理与音乐信息检索领域，与文档图像理解（DIU）或视觉-语言模型在文档上的应用完全无关。尽管使用了生成模型和潜在扩散架构，但其方法针对的是音频频谱图的生成与分离，无法迁移至文档图像中的OCR幻觉、复杂排版解析或细粒度文本定位等核心问题。
Abstract Extracting individual elements from music mixtures is a valuable tool for music production and practice. While neural networks optimized to mask or transform mixture spectrograms into the individual source(s) have been the leading approach, the source overlap and correlation in music signals poses an inherent challenge. Also, accessing all sources in the mixture is crucial to train these systems, while complicated. Attempts to address these challenges in a generative fashion exist, however, the separation performance and inference efficiency remain limited. In this work, we study the potential of diffusion models to advance toward bridging this gap, focusing on generative singing voice separation relying only on corresponding pairs of isolated vocals and mixtures for training. To align with creative workflows, we leverage latent diffusion: the system generates samples encoded in a compact latent space, and subsequently decodes these into audio. This enables efficient optimization and faster inference. Our system is trained using only open data. We outperform existing generative separation systems, and level the compared non-generative systems on a list of signal quality measures and on interference removal. We provide a noise robustness study on the latent encoder, providing insights on its potential for the task. We release a modular toolkit for further research on the topic.
Authors Genís Plaja-Roglans, Yun-Ning Hung, Xavier Serra, Igor Pereira
Comment Accepted for oral presentation at IJCNN 2025
Categories Artificial Intelligence
0
Generation, Evaluation, and Explanation of Novelists' Styles with Single-Token Prompts
Title CN 使用单令牌提示生成、评估和解释小说家风格
Keywords 风格迁移 · 文本生成 · 可解释AI · 语言模型 · 作者风格
Summary 本文提出一种使用单令牌提示微调大语言模型以生成19世纪小说家风格文本的框架，并利用Transformer分类器和可解释AI技术对生成结果进行自动评估与分析。研究重点在于文本风格建模与自动化评测，不涉及图像或多模态信息。
Reason 该论文属于纯文本生成与风格分析领域，聚焦于文学作者的写作风格模仿与评估，未涉及任何视觉或文档图像内容。完全脱离文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等。其方法基于语言模型的微调与可解释性分析，无法迁移解决OCR幻觉、复杂排版、高分辨率处理等DIU关键痛点。
Abstract Recent advances in large language models have created new opportunities for stylometry, the study of writing styles and authorship. Two challenges, however, remain central: training generative models when no paired data exist, and evaluating stylistic text without relying only on human judgment. In this work, we present a framework for both generating and evaluating sentences in the style of 19th-century novelists. Large language models are fine-tuned with minimal, single-token prompts to produce text in the voices of authors such as Dickens, Austen, Twain, Alcott, and Melville. To assess these generative models, we employ a transformer-based detector trained on authentic sentences, using it both as a classifier and as a tool for stylistic explanation. We complement this with syntactic comparisons and explainable AI methods, including attention-based and gradient-based analyses, to identify the linguistic cues that drive stylistic imitation. Our findings show that the generated text reflects the authors' distinctive patterns and that AI-based evaluation offers a reliable alternative to human assessment. All artifacts of this work are published online.
Authors Mosab Rezaei, Mina Rajaei Moghadam, Abdul Rahman Shaikh, Hamed Alhoori, Reva Freedman
Categories Artificial Intelligence, Computation and Language
0
CardioComposer: Leveraging Differentiable Geometry for Compositional Control of Anatomical Diffusion Models
Title CN CardioComposer：利用可微几何实现解剖扩散模型的组合控制
Keywords 生成模型 · 可微几何 · 扩散模型 · 3D医学图像 · 解剖结构控制
Summary 本文提出CardioComposer，一种基于可微几何原语的推理时框架，用于控制多类解剖标签图的生成。通过定义基于体素几何矩的可微测量函数，在扩散模型采样过程中引入梯度引导损失，实现对子结构大小、形状和位置等几何属性的解耦约束与组合控制。方法适用于心脏、血管和骨骼等多种解剖系统。
Reason 该论文研究的是3D心血管解剖结构的生成与几何控制，属于医学图像生成与3D视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。其核心技术围绕可微几何建模和扩散模型采样控制，应用于非凸器官结构的合成，无法迁移解决OCR幻觉、复杂排版解析、细粒度文本定位等DIU核心问题。
Abstract Generative models of 3D cardiovascular anatomy can synthesize informative structures for clinical research and medical device evaluation, but face a trade-off between geometric controllability and realism. We propose CardioComposer: a programmable, inference-time framework for generating multi-class anatomical label maps based on interpretable ellipsoidal primitives. These primitives represent geometric attributes such as the size, shape, and position of discrete substructures. We specifically develop differentiable measurement functions based on voxel-wise geometric moments, enabling loss-based gradient guidance during diffusion model sampling. We demonstrate that these losses can constrain individual geometric attributes in a disentangled manner and provide compositional control over multiple substructures. Finally, we show that our method is compatible with a wide array of anatomical systems containing non-convex substructures, spanning cardiac, vascular, and skeletal organs.
Authors Karim Kadry, Shoaib Goraya, Ajay Manicka, Abdalla Abdelwahed, Naravich Chutisilp, Farhad Nezami, Elazer Edelman
Comment 10 pages, 16 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Segmentation-Aware Generative Reinforcement Network (GRN) for Tissue Layer Segmentation in 3-D Ultrasound Images for Chronic Low-back Pain (cLBP) Assessment
Title CN 用于慢性腰痛评估的三维超声图像中组织层分割的感知生成强化网络（GRN）
Keywords 医学图像分割 · 生成模型 · 半监督学习 · 3D超声 · 图像增强
Summary 本文提出了一种名为生成强化网络（GRN）的框架，结合分割感知机制和图像增强技术（SGE），用于提升3D超声图像中多层组织的分割性能。通过引入样本高效学习（GRN-SEL）和半监督学习（GRN-SSL）变体，显著减少了对标注数据的依赖，在仅需60%-70%标注的情况下仍保持甚至提升分割精度。然而，该工作专注于医学影像分析，与文档理解无直接关联。
Reason 该论文研究的是医学图像中的组织层分割问题，属于3D超声图像分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务完全无关。其方法虽涉及生成模型与分割的联合训练，但应用场景为生物医学影像处理，无法迁移到解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract We introduce a novel segmentation-aware joint training framework called generative reinforcement network (GRN) that integrates segmentation loss feedback to optimize both image generation and segmentation performance in a single stage. An image enhancement technique called segmentation-guided enhancement (SGE) is also developed, where the generator produces images tailored specifically for the segmentation model. Two variants of GRN were also developed, including GRN for sample-efficient learning (GRN-SEL) and GRN for semi-supervised learning (GRN-SSL). GRN's performance was evaluated using a dataset of 69 fully annotated 3D ultrasound scans from 29 subjects. The annotations included six anatomical structures: dermis, superficial fat, superficial fascial membrane (SFM), deep fat, deep fascial membrane (DFM), and muscle. Our results show that GRN-SEL with SGE reduces labeling efforts by up to 70% while achieving a 1.98% improvement in the Dice Similarity Coefficient (DSC) compared to models trained on fully labeled datasets. GRN-SEL alone reduces labeling efforts by 60%, GRN-SSL with SGE decreases labeling requirements by 70%, and GRN-SSL alone by 60%, all while maintaining performance comparable to fully supervised models. These findings suggest the effectiveness of the GRN framework in optimizing segmentation performance with significantly less labeled data, offering a scalable and efficient solution for ultrasound image analysis and reducing the burdens associated with data annotation.
Authors Zixue Zeng, Xiaoyan Zhao, Matthew Cartier, Tong Yu, Jing Wang, Xin Meng, Zhiyu Sheng, Maryam Satarpour, John M Cormack, Allison Bean, Ryan Nussbaum, Maya Maurer, Emily Landis-Walkenhorst, Dinesh Kumbhare, Kang Kim, Ajay Wasan, Jiantao Pu
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
VibraVerse: A Large-Scale Geometry-Acoustics Alignment Dataset for Physically-Consistent Multimodal Learning
Title CN VibraVerse：用于物理一致多模态学习的大规模几何-声学对齐数据集
Keywords 多模态学习 · 物理一致性 · 几何-声学对齐 · 3D感知 · 对比学习
Summary 本文提出了VibraVerse，一个大规模的几何-声学对齐数据集，旨在通过物理规律而非统计相关性来实现多模态学习。数据集基于3D模型的物理属性（如密度、杨氏模量）计算振动模态，并生成对应的撞击声音。同时提出CLASP框架进行跨模态对比学习，确保几何与声学信号之间的因果一致性。虽然在物理世界理解方面具有意义，但与文档理解无关。
Reason 该论文聚焦于3D几何与声学信号之间的物理一致性建模，属于具身感知与跨模态物理模拟领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心任务如几何到声音预测、声音引导的形状重建等，无法迁移解决OCR幻觉、复杂排版解析、密集文本识别或长文档推理等DIU核心痛点。
Abstract Understanding the physical world requires perceptual models grounded in physical laws rather than mere statistical correlations. However, existing multimodal learning frameworks, focused on vision and language, lack physical consistency and overlook the intrinsic causal relationships among an object's geometry, material, vibration modes, and the sounds it produces. We introduce VibraVerse, a large-scale geometry-acoustics alignment dataset that explicitly bridges the causal chain from 3D geometry -> physical attributes -> modal parameters -> acoustic signals. Each 3D model has explicit physical properties (density, Young's modulus, Poisson's ratio) and volumetric geometry, from which modal eigenfrequencies and eigenvectors are computed for impact sound synthesis under controlled excitations. To establish this coherence, we introduce CLASP, a contrastive learning framework for cross-modal alignment that preserves the causal correspondence between an object's physical structure and its acoustic response. This framework enforces physically consistent alignment across modalities, ensuring that every sample is coherent, traceable to the governing equations, and embedded within a unified representation space spanning shape, image, and sound. Built upon VibraVerse, we define a suite of benchmark tasks for geometry-to-sound prediction, sound-guided shape reconstruction, and cross-modal representation learning. Extensive validations on these tasks demonstrate that models trained on VibraVerse exhibit superior accuracy, interpretability, and generalization across modalities. These results establish VibraVerse as a benchmark for physically consistent and causally interpretable multimodal learning, providing a foundation for sound-guided embodied perception and a deeper understanding of the physical world. The dataset will be open-sourced.
Authors Bo Pang, Chenxi Xu, Jierui Ren, Guoping Wang, Sheng Li
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
StableTrack: Stabilizing Multi-Object Tracking on Low-Frequency Detections
Title CN StableTrack：在低频检测上稳定多目标跟踪
Keywords 多目标跟踪 · 低频检测 · 目标关联 · Kalman滤波 · Re-ID
Summary 本文提出StableTrack，一种针对低频检测场景下的多目标跟踪优化方法。通过引入新的两阶段匹配策略和基于边界框的距离度量替代传统的马氏距离，并结合Re-ID模型与卡尔曼滤波器，提升了跨帧目标关联的稳定性。实验表明，在MOT17-val数据集上以1Hz运行时HOTA指标提升11.6%，同时在全频检测的标准基准（MOT17、MOT20、DanceTrack）上保持竞争力。
Reason 该论文属于视频理解领域的多目标跟踪（MOT）任务，主要解决的是在低频检测条件下的目标关联与稳定性问题。其技术核心如两阶段匹配策略、基于边界框的距离度量和Kalman滤波器改进，均面向视频序列中的动态物体跟踪，与文档图像理解（DIU）中的OCR幻觉、复杂版式解析、细粒度文本定位等核心痛点无直接关联。无法迁移用于提升文档图像的语义理解或多模态推理能力。
Abstract Multi-object tracking (MOT) is one of the most challenging tasks in computer vision, where it is important to correctly detect objects and associate these detections across frames. Current approaches mainly focus on tracking objects in each frame of a video stream, making it almost impossible to run the model under conditions of limited computing resources. To address this issue, we propose StableTrack, a novel approach that stabilizes the quality of tracking on low-frequency detections. Our method introduces a new two-stage matching strategy to improve the cross-frame association between low-frequency detections. We propose a novel Bbox-Based Distance instead of the conventional Mahalanobis distance, which allows us to effectively match objects using the Re-ID model. Furthermore, we integrate visual tracking into the Kalman Filter and the overall tracking pipeline. Our method outperforms current state-of-the-art trackers in the case of low-frequency detections, achieving $\textit{11.6%}$ HOTA improvement at $\textit{1}$ Hz on MOT17-val, while keeping up with the best approaches on the standard MOT17, MOT20, and DanceTrack benchmarks with full-frequency detections.
Authors Matvei Shelukhan, Timur Mamedov, Karina Kvanchiani
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
OceanGym: A Benchmark Environment for Underwater Embodied Agents
Title CN OceanGym：面向水下具身智能体的基准环境
Keywords 具身智能 · 水下机器人 · 多模态大模型 · 基准测试 · 长视野决策
Summary 本文提出了OceanGym，首个面向水下具身智能体的综合基准平台，包含八个真实任务域和一个基于多模态大语言模型的统一智能体框架。该平台要求智能体融合光学和声呐数据，在低能见度和动态洋流等恶劣条件下完成长周期探索与决策任务。实验表明当前最先进MLLM驱动的智能体与人类专家仍有显著差距。OceanGym旨在推动鲁棒的具身AI发展，并向实际水下自动驾驶应用迁移。
Reason 该论文属于具身智能（Embodied AI）领域，研究的是水下机器人在复杂海洋环境中的感知与决策，涉及光学与声呐数据融合、长视野任务规划等。尽管使用了多模态大模型（MLLM），但其核心场景与文档图像理解（DIU）无关，且目标是推动水下自主航行器发展，无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心问题。
Abstract We introduce OceanGym, the first comprehensive benchmark for ocean underwater embodied agents, designed to advance AI in one of the most demanding real-world environments. Unlike terrestrial or aerial domains, underwater settings present extreme perceptual and decision-making challenges, including low visibility, dynamic ocean currents, making effective agent deployment exceptionally difficult. OceanGym encompasses eight realistic task domains and a unified agent framework driven by Multi-modal Large Language Models (MLLMs), which integrates perception, memory, and sequential decision-making. Agents are required to comprehend optical and sonar data, autonomously explore complex environments, and accomplish long-horizon objectives under these harsh conditions. Extensive experiments reveal substantial gaps between state-of-the-art MLLM-driven agents and human experts, highlighting the persistent difficulty of perception, planning, and adaptability in ocean underwater environments. By providing a high-fidelity, rigorously designed platform, OceanGym establishes a testbed for developing robust embodied AI and transferring these capabilities to real-world autonomous ocean underwater vehicles, marking a decisive step toward intelligent agents capable of operating in one of Earth's last unexplored frontiers. The code and data are available at https://github.com/OceanGPT/OceanGym.
Authors Yida Xue, Mingjun Mao, Xiangyuan Ru, Yuqi Zhu, Baochang Ren, Shuofei Qiao, Mengru Wang, Shumin Deng, Xinyu An, Ningyu Zhang, Ying Chen, Huajun Chen
Comment Work in progress
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
0
Harnessing Vision-Language Models for Time Series Anomaly Detection AAAI 2026
Title CN 利用视觉语言模型进行时间序列异常检测
Keywords 时间序列异常检测 · 视觉语言模型 · ViT · 零样本学习 · 视觉表示
Summary 本文提出一种两阶段方法VLM4TS，利用视觉编码器ViT4TS生成2D时间序列表示以定位候选异常，再通过VLM结合全局时序上下文进行精细化检测。结果显示其在无需时间序列专门训练的情况下优于多种基线方法，并显著提升token效率。
Reason 该论文将VLM应用于时间序列异常检测，属于将多模态模型迁移到非文档领域的平行应用。尽管使用了视觉编码和VLM的结合，但其核心任务与文档图像理解（如DocVQA、布局分析、表格识别等）无关，且未提出可用于解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点的通用方法。属于典型的‘Wrapper/Application’类工作，无底层方法论创新。
Abstract Time-series anomaly detection (TSAD) has played a vital role in a variety of fields, including healthcare, finance, and sensor-based condition monitoring. Prior methods, which mainly focus on training domain-specific models on numerical data, lack the visual-temporal understanding capacity that human experts have to identify contextual anomalies. To fill this gap, we explore a solution based on vision language models (VLMs). Recent studies have shown the ability of VLMs for visual understanding tasks, yet their direct application to time series has fallen short on both accuracy and efficiency. To harness the power of VLMs for TSAD, we propose a two-stage solution, with (1) ViT4TS, a vision-screening stage built on a relatively lightweight pre-trained vision encoder, which leverages 2D time series representations to accurately localize candidate anomalies; (2) VLM4TS, a VLM-based stage that integrates global temporal context and VLM's visual understanding capacity to refine the detection upon the candidates provided by ViT4TS. We show that without any time-series training, VLM4TS outperforms time-series pre-trained and from-scratch baselines in most cases, yielding a 24.6% improvement in F1-max score over the best baseline. Moreover, VLM4TS also consistently outperforms existing language model-based TSAD methods and is on average 36x more efficient in token usage.
Authors Zelin He, Sarah Alnegheimish, Matthew Reimherr
Comment Accepted at AAAI 2026 (Oral)
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Jailbreaking and Mitigation of Vulnerabilities in Large Language Models
Title CN 大语言模型的越狱攻击与漏洞缓解
Keywords 大语言模型 · 越狱攻击 · 提示注入 · AI安全 · 对抗攻击
Summary 本文综述了大语言模型在面对提示注入和越狱攻击时的脆弱性，分类整理了现有的攻击手段（如对抗性提示、后门注入）和防御策略（如提示过滤、多智能体防御），并讨论了评估基准与社会伦理影响。研究重点完全落在文本模态的安全性上，不涉及视觉输入或跨模态对齐问题。
Reason 该论文属于纯NLP安全与对齐领域，聚焦于LLM的越狱攻击和防御机制，未涉及任何视觉模态或文档理解任务。尽管‘幻觉’是DIU中的痛点之一，但本文讨论的是文本生成中的安全性问题（如绕过伦理限制），而非OCR或视觉-语言模型在文档理解中的感知/推理幻觉。其方法无法直接迁移至文档图像理解场景，且属于典型的非相关领域工作。
Abstract Large Language Models (LLMs) have transformed artificial intelligence by advancing natural language understanding and generation, enabling applications across fields beyond healthcare, software engineering, and conversational systems. Despite these advancements in the past few years, LLMs have shown considerable vulnerabilities, particularly to prompt injection and jailbreaking attacks. This review analyzes the state of research on these vulnerabilities and presents available defense strategies. We roughly categorize attack approaches into prompt-based, model-based, multimodal, and multilingual, covering techniques such as adversarial prompting, backdoor injections, and cross-modality exploits. We also review various defense mechanisms, including prompt filtering, transformation, alignment techniques, multi-agent defenses, and self-regulation, evaluating their strengths and shortcomings. We also discuss key metrics and benchmarks used to assess LLM safety and robustness, noting challenges like the quantification of attack success in interactive contexts and biases in existing datasets. Identifying current research gaps, we suggest future directions for resilient alignment strategies, advanced defenses against evolving attacks, automation of jailbreak detection, and consideration of ethical and societal impacts. This review emphasizes the need for continued research and cooperation within the AI community to enhance LLM security and ensure their safe deployment.
Authors Benji Peng, Keyu Chen, Qian Niu, Ziqian Bi, Ming Liu, Pohsun Feng, Tianyang Wang, Lawrence K. Q. Yan, Yizhu Wen, Yichao Zhang, Caitlyn Heqi Yin, Xinyuan Song
Categories Artificial Intelligence
0
LikePhys: Evaluating Intuitive Physics Understanding in Video Diffusion Models via Likelihood Preference
Title CN LikePhys：通过似然偏好评估视频扩散模型中的直觉物理理解
Keywords 视频扩散模型 · 直觉物理理解 · 评估方法 · 似然代理 · PPE
Summary 本文提出LikePhys，一种无需训练的评估方法，用于衡量视频扩散模型对直观物理规律的理解能力。该方法利用去噪目标作为ELBO似然代理，在构造的物理合理与不合理视频对数据集上进行区分，并引入Plausibility Preference Error（PPE）指标，验证其与人类偏好的高度一致性。实验表明当前模型在复杂动态场景中仍存在不足，但随模型规模和推理配置提升呈现改进趋势。
Reason 该论文研究的是视频扩散模型中对物理规律的直觉理解评估，属于视频生成与物理推理交叉领域，核心任务为物理合理性判断，使用的方法基于ELBO似然代理和人类偏好对齐。其研究目标与文档图像理解（DIU）无直接关联，既不涉及OCR、布局分析、表格识别等DIU任务，也未提出可用于解决文档理解中OCR幻觉、复杂排版或长文档推理等问题的通用方法。属于无关领域（视频理解/生成），不符合关注方向。
Abstract Intuitive physics understanding in video diffusion models plays an essential role in building general-purpose physically plausible world simulators, yet accurately evaluating such capacity remains a challenging task due to the difficulty in disentangling physics correctness from visual appearance in generation. To the end, we introduce LikePhys, a training-free method that evaluates intuitive physics in video diffusion models by distinguishing physically valid and impossible videos using the denoising objective as an ELBO-based likelihood surrogate on a curated dataset of valid-invalid pairs. By testing on our constructed benchmark of twelve scenarios spanning over four physics domains, we show that our evaluation metric, Plausibility Preference Error (PPE), demonstrates strong alignment with human preference, outperforming state-of-the-art evaluator baselines. We then systematically benchmark intuitive physics understanding in current video diffusion models. Our study further analyses how model design and inference settings affect intuitive physics understanding and highlights domain-specific capacity variations across physical laws. Empirical results show that, despite current models struggling with complex and chaotic dynamics, there is a clear trend of improvement in physics understanding as model capacity and inference settings scale.
Authors Jianhao Yuan, Fabio Pizzati, Francesco Pinto, Lars Kunze, Ivan Laptev, Paul Newman, Philip Torr, Daniele De Martini
Comment 22 pages, 9 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
NNGPT: Rethinking AutoML with Large Language Models
Title CN NNGPT：用大语言模型重新思考AutoML
Keywords AutoML · 大语言模型 · 神经网络生成 · 超参数优化 · 代码预测
Summary 本文提出了NNGPT，一个基于大语言模型的自改进AutoML框架，用于自动化神经网络设计与调优。它通过闭环的生成、评估与学习机制，在零样本架构合成、超参优化、代码预测等方面实现端到端自动化，并已在LEMUR数据集上验证有效性。然而，该工作专注于通用视觉模型的自动化构建，不属于文档图像理解范畴。
Reason 该论文属于典型的‘平行下游应用’，其核心是利用LLM构建一个面向计算机视觉模型的自改进AutoML系统，聚焦于神经网络架构生成、超参数优化等通用AutoML任务。尽管使用了LLM和多阶段流程，但其目标领域并非文档图像理解（DIU），也没有针对OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点提出可迁移的方法。所涉及的技术如NN-RAG、代码预测等虽在AutoML中有价值，但不构成对文档理解任务的直接方法论贡献，且未展示在多模态文档场景下的适用性。
Abstract Building self-improving AI systems remains a fundamental challenge in the AI domain. We present NNGPT, an open-source framework that turns a large language model (LLM) into a self-improving AutoML engine for neural network development, primarily for computer vision. Unlike previous frameworks, NNGPT extends the dataset of neural networks by generating new models, enabling continuous fine-tuning of LLMs based on closed-loop system of generation, assessment, and self-improvement. It integrates within one unified workflow five synergistic LLM-based pipelines: zero-shot architecture synthesis, hyperparameter optimization (HPO), code-aware accuracy/early-stop prediction, retrieval-augmented synthesis of scope-closed PyTorch blocks (NN-RAG), and reinforcement learning. Built on the LEMUR dataset as an audited corpus with reproducible metrics, NNGPT emits from a single prompt and validates network architecture, preprocessing code, and hyperparameters, executes them end-to-end, and learns from result. The PyTorch adapter makes NNGPT framework-agnostic, enabling strong performance: NN-RAG achieves 73% executability on 1,289 targets, 3-shot prompting boosts accuracy on common datasets, and hash-based deduplication saves hundreds of runs. One-shot prediction matches search-based AutoML, reducing the need for numerous trials. HPO on LEMUR achieves RMSE 0.60, outperforming Optuna (0.64), while the code-aware predictor reaches RMSE 0.14 with Pearson r=0.78. The system has already generated over 5K validated models, proving NNGPT as an autonomous AutoML engine. Upon acceptance, the code, prompts, and checkpoints will be released for public access to enable reproducibility and facilitate community usage.
Authors Roman Kochnev, Waleed Khalid, Tolgay Atinc Uzun, Xi Zhang, Yashkumar Sanjaybhai Dhameliya, Furui Qin, Chandini Vysyaraju, Raghuvir Duvvuri, Avi Goyal, Dmitry Ignatov, Radu Timofte
Categories Artificial Intelligence
0
3D Motion Perception of Binocular Vision Target with PID-CNN
Title CN 基于PID-CNN的双目视觉目标三维运动感知
Keywords PID-CNN · 双目视觉 · 3D运动感知 · 时空预测 · 轻量级网络
Summary 本文提出了一种结合PID控制思想的轻量级CNN网络（PID-CNN），用于从双目视觉图像中实时预测运动目标的三维坐标、速度和加速度。网络共17层，参数量41.3万，通过拼接与池化实现特征复用，在模拟随机运动小球数据集上达到了接近输入分辨率极限的预测精度。同时探讨了高维卷积对计算效率和特征利用的潜在优势，以及PID机制在记忆与注意力建模中的可能性。
Reason 该论文研究的是双目视觉下的三维运动感知，属于3D视觉与视频理解领域，核心任务为动态目标的时空坐标预测，与文档图像理解（DIU）无关。其方法聚焦于运动建模与高维卷积优化，无法迁移解决OCR幻觉、复杂排版解析或长文档推理等DIU核心痛点。
Abstract This article trained a network for perceiving three-dimensional motion information of binocular vision target, which can provide real-time three-dimensional coordinate, velocity, and acceleration, and has a basic spatiotemporal perception capability. Understood the ability of neural networks to fit nonlinear problems from the perspective of PID. Considered a single-layer neural network as using a second-order difference equation and a nonlinearity to describe a local problem. Multilayer networks gradually transform the raw representation to the desired representation through multiple such combinations. Analysed some reference principles for designing neural networks. Designed a relatively small PID convolutional neural network, with a total of 17 layers and 413 thousand parameters. Implemented a simple but practical feature reuse method by concatenation and pooling. The network was trained and tested using the simulated randomly moving ball datasets, and the experimental results showed that the prediction accuracy was close to the upper limit that the input image resolution can represent. Analysed the experimental results and errors, as well as the existing shortcomings and possible directions for improvement. Finally, discussed the advantages of high-dimensional convolution in improving computational efficiency and feature space utilization. As well as the potential advantages of using PID information to implement memory and attention mechanisms.
Authors Shi Jiazhao, Pan Pan, Shi Haotian
Comment 7 pages, 9 figures, 2 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Data Augmentation Techniques to Reverse-Engineer Neural Network Weights from Input-Output Queries Proceedings of the III edition of the Workshop on Unifying Representations in Neural Models (UniReps 2025)
Title CN 通过输入-输出查询逆向工程神经网络权重的数据增强技术
Keywords 模型逆向 · 数据增强 · 教师-学生模型 · 表征学习 · 模型提取
Summary 本文研究如何通过输入-输出查询和数据增强技术来逆向工程神经网络的权重。作者发现传统数据增强方法效果有限，因此设计了新的增强策略以更充分地采样隐藏层的表征空间，从而在训练数据点少于参数量的情况下仍能恢复更大规模的网络。实验表明可恢复参数量高达数据点100倍的网络。
Reason 该论文研究的是在教师-学生模型框架下，如何通过设计数据增强技术来逆向恢复教师网络的权重参数，属于模型提取攻击（Model Extraction）或神经网络可解释性范畴。其核心目标与文档图像理解（DIU）任务无直接关联，既不涉及多模态文档理解、OCR、布局分析、表格识别等DIU核心问题，也未提出可用于解决文档理解中幻觉、高分辨率处理、细粒度定位或复杂推理等痛点的通用方法。所提增强策略针对的是网络内部表征空间的探索，难以迁移到文档视觉问答或多模态对齐任务中。
Abstract Network weights can be reverse-engineered given enough informative samples of a network's input-output function. In a teacher-student setup, this translates into collecting a dataset of the teacher mapping -- querying the teacher -- and fitting a student to imitate such mapping. A sensible choice of queries is the dataset the teacher is trained on. But current methods fail when the teacher parameters are more numerous than the training data, because the student overfits to the queries instead of aligning its parameters to the teacher. In this work, we explore augmentation techniques to best sample the input-output mapping of a teacher network, with the goal of eliciting a rich set of representations from the teacher hidden layers. We discover that standard augmentations such as rotation, flipping, and adding noise, bring little to no improvement to the identification problem. We design new data augmentation techniques tailored to better sample the representational space of the network's hidden layers. With our augmentations we extend the state-of-the-art range of recoverable network sizes. To test their scalability, we show that we can recover networks of up to 100 times more parameters than training data-points.
Authors Alexander Beiser, Flavio Martinelli, Wulfram Gerstner, Johanni Brea
Comment Proceedings of the III edition of the Workshop on Unifying Representations in Neural Models (UniReps 2025)
Categories Artificial Intelligence
0
RIS-Assisted Downlink Pinching-Antenna Systems: GNN-Enabled Optimization Approaches
Title CN 基于RIS辅助的下行链路挤压天线系统：GNN驱动的优化方法
Keywords RIS · 挤压天线 · 图神经网络 · 无线通信 · 波束成形
Summary 本文研究了一种由可重构智能表面（RIS）辅助的多波导挤压天线系统（PASS），旨在提升多用户下行链路传输的性能。作者构建了一个联合优化框架，以最大化总速率和能量效率，并提出一种三阶段图神经网络（GNN）来学习天线位置、RIS相位调整和波束成形向量。该GNN采用无监督训练，并结合凸优化策略，在推理速度与解的质量之间取得平衡。数值结果验证了该方法在泛化能力、稳定性和实时性方面的优势。
Reason 该论文属于无线通信与智能反射面（RIS）领域的物理层优化，研究的是天线位置、波束成形和能效最大化问题，与文档图像理解（DIU）、多模态大模型或视觉语言任务完全无关。尽管使用了GNN，但其建模范式和解决的问题域无法迁移到DIU中的OCR幻觉、复杂排版解析或长文档推理等核心痛点。
Abstract This paper investigates a reconfigurable intelligent surface (RIS)-assisted multi-waveguide pinching-antenna (PA) system (PASS) for multi-user downlink information transmission, motivated by the unknown impact of the integration of emerging PASS and RIS on wireless communications. First, we formulate sum rate (SR) and energy efficiency (EE) maximization problems in a unified framework, subject to constraints on the movable region of PAs, total power budget, and tunable phase of RIS elements. Then, by leveraging a graph-structured topology of the RIS-assisted PASS, a novel three-stage graph neural network (GNN) is proposed, which learns PA positions based on user locations, and RIS phase shifts according to composite channel conditions at the first two stages, respectively, and finally determines beamforming vectors. Specifically, the proposed GNN is achieved through unsupervised training, together with three implementation strategies for its integration with convex optimization, thus offering trade-offs between inference time and solution optimality. Extensive numerical results are provided to validate the effectiveness of the proposed GNN, and to support its unique attributes of viable generalization capability, good performance reliability, and real-time applicability. Moreover, the impact of key parameters on RIS-assisted PASS is illustrated and analyzed.
Authors Changpeng He, Yang Lu, Yanqing Xu, Chong-Yung Chi, Bo Ai, Arumugam Nallanathan
Categories Artificial Intelligence
0
Prompting Lipschitz-constrained network for multiple-in-one sparse-view CT reconstruction
Title CN 通过提示Lipschitz约束网络实现多合一稀疏视角CT重建
Keywords 稀疏视角CT重建 · Lipschitz约束 · 提示学习 · 深度展开框架 · 医学图像重建
Summary 本文提出了一种具有显式Lipschitz约束的网络LipNet，并结合提示模块构建PromptCT框架，用于解决多稀疏视角CT图像重建中的模型泛化与存储成本问题。该方法在统一模型中处理不同采样设置，降低了存储开销，并在理论层面证明了收敛性。实验表明其在仿真和真实数据上优于现有方法。
Reason 该论文研究的是医学影像中的稀疏视角CT重建问题，属于医疗图像重建领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管其方法涉及网络结构设计和存储优化，但核心任务为三维医学信号重建，迁移到文档理解场景（如OCR、排版分析、表格识别等）极为困难，且未涉及多模态语言模型、视觉表示学习或文档特有的结构建模。因此属于无关领域，直接打0分。
Abstract Despite significant advancements in deep learning-based sparse-view computed tomography (SVCT) reconstruction algorithms, these methods still encounter two primary limitations: (i) It is challenging to explicitly prove that the prior networks of deep unfolding algorithms satisfy Lipschitz constraints due to their empirically designed nature. (ii) The substantial storage costs of training a separate model for each setting in the case of multiple views hinder practical clinical applications. To address these issues, we elaborate an explicitly provable Lipschitz-constrained network, dubbed LipNet, and integrate an explicit prompt module to provide discriminative knowledge of different sparse sampling settings, enabling the treatment of multiple sparse view configurations within a single model. Furthermore, we develop a storage-saving deep unfolding framework for multiple-in-one SVCT reconstruction, termed PromptCT, which embeds LipNet as its prior network to ensure the convergence of its corresponding iterative algorithm. In simulated and real data experiments, PromptCT outperforms benchmark reconstruction algorithms in multiple-in-one SVCT reconstruction, achieving higher-quality reconstructions with lower storage costs. On the theoretical side, we explicitly demonstrate that LipNet satisfies boundary property, further proving its Lipschitz continuity and subsequently analyzing the convergence of the proposed iterative algorithms. The data and code are publicly available at https://github.com/shibaoshun/PromptCT.
Authors Baoshun Shi, Ke Jiang, Qiusheng Lian, Xinran Yu, Huazhu Fu
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Multiple-Input Auto-Encoder Guided Feature Selection for IoT Intrusion Detection Systems
Title CN 用于物联网入侵检测系统的多输入自动编码器引导特征选择
Keywords 入侵检测 · 特征选择 · 自动编码器 · 无监督学习 · 物联网安全
Summary 本文提出了一种名为MIAE的多输入自动编码器及其改进版本MIAEFS，用于在无监督模式下对异构高维物联网数据进行低维表示学习与特征选择，以提升入侵检测系统的准确性和效率。实验表明该方法在多个IDS数据集上表现优异。
Reason 该论文研究的是物联网入侵检测系统中的特征选择问题，属于网络安全与传统机器学习应用领域，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。其方法虽涉及表示学习和降维，但针对的是结构化/网络流量数据，并非文档图像或多模态内容，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract While intrusion detection systems (IDSs) benefit from the diversity and generalization of IoT data features, the data diversity (e.g., the heterogeneity and high dimensions of data) also makes it difficult to train effective machine learning models in IoT IDSs. This also leads to potentially redundant/noisy features that may decrease the accuracy of the detection engine in IDSs. This paper first introduces a novel neural network architecture called Multiple-Input Auto-Encoder (MIAE). MIAE consists of multiple sub-encoders that can process inputs from different sources with different characteristics. The MIAE model is trained in an unsupervised learning mode to transform the heterogeneous inputs into lower-dimensional representation, which helps classifiers distinguish between normal behaviour and different types of attacks. To distil and retain more relevant features but remove less important/redundant ones during the training process, we further design and embed a feature selection layer right after the representation layer of MIAE resulting in a new model called MIAEFS. This layer learns the importance of features in the representation vector, facilitating the selection of informative features from the representation vector. The results on three IDS datasets, i.e., NSLKDD, UNSW-NB15, and IDS2017, show the superior performance of MIAE and MIAEFS compared to other methods, e.g., conventional classifiers, dimensionality reduction models, unsupervised representation learning methods with different input dimensions, and unsupervised feature selection models. Moreover, MIAE and MIAEFS combined with the Random Forest (RF) classifier achieve accuracy of 96.5% in detecting sophisticated attacks, e.g., Slowloris. The average running time for detecting an attack sample using RF with the representation of MIAE and MIAEFS is approximate 1.7E-6 seconds, whilst the model size is lower than 1 MB.
Authors Phai Vu Dinh, Diep N. Nguyen, Dinh Thai Hoang, Quang Uy Nguyen, Eryk Dutkiewicz, Son Pham Bao
Categories Artificial Intelligence
0
Forgetting by Pruning: Data Deletion in Join Cardinality Estimation AAAI26
Title CN 通过剪枝实现遗忘：连接基数估计中的数据删除
Keywords 机器遗忘 · 基数估计 · 剪枝 · 数据库 · 数据删除
Summary 本文提出了一种名为CEP的机器遗忘框架，用于在多表学习型基数估计系统中高效删除数据。该方法通过分布敏感性剪枝和域剪枝来应对属性级敏感性、跨表传播和值域消失等问题，在保持准确性的前提下显著降低计算开销。然而，该工作聚焦于结构化数据查询优化场景，不涉及图像或文档模态。
Reason 该论文研究的是机器遗忘在数据库基数估计系统中的应用，属于数据库与机器学习交叉领域，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。尽管其方法涉及模型剪枝和分布敏感性分析，但任务背景、数据模态和应用场景均远离文档视觉内容的理解与推理，无法迁移解决OCR幻觉、复杂排版解析或细粒度视觉定位等DIU核心问题。
Abstract Machine unlearning in learned cardinality estimation (CE) systems presents unique challenges due to the complex distributional dependencies in multi-table relational data. Specifically, data deletion, a core component of machine unlearning, faces three critical challenges in learned CE models: attribute-level sensitivity, inter-table propagation and domain disappearance leading to severe overestimation in multi-way joins. We propose Cardinality Estimation Pruning (CEP), the first unlearning framework specifically designed for multi-table learned CE systems. CEP introduces Distribution Sensitivity Pruning, which constructs semi-join deletion results and computes sensitivity scores to guide parameter pruning, and Domain Pruning, which removes support for value domains entirely eliminated by deletion. We evaluate CEP on state-of-the-art architectures NeuroCard and FACE across IMDB and TPC-H datasets. Results demonstrate CEP consistently achieves the lowest Q-error in multi-table scenarios, particularly under high deletion ratios, often outperforming full retraining. Furthermore, CEP significantly reduces convergence iterations, incurring negligible computational overhead of 0.3%-2.5% of fine-tuning time.
Authors Chaowei He, Yuanjun Liu, Qingzhi Ma, Shenyuan Ren, Xizhao Luo, Lei Zhao, An Liu
Comment AAAI26
Categories Artificial Intelligence
0
Can LLMs Make (Personalized) Access Control Decisions?
Title CN 大语言模型能否做出（个性化）访问控制决策？
Keywords 大语言模型 · 访问控制 · 个性化 · 安全策略 · 用户研究
Summary 本文探讨利用大语言模型根据用户的自然语言隐私声明来自动化做出个性化的访问控制决策，并通过用户研究评估模型决策与人类决策的一致性。结果显示LLM能较好反映多数用户偏好，但在个性化与安全最佳实践之间存在权衡。
Reason 该论文研究的是基于LLM的个性化访问控制决策，属于安全与隐私领域的下游应用，不涉及文档图像理解中的核心问题如OCR幻觉、复杂排版处理、视觉-文本对齐或多模态表示学习。尽管使用了LLM进行推理，但其任务场景与文档理解无关，属于典型的‘平行应用’，未提出可迁移至DIU领域的方法论创新。
Abstract Precise access control decisions are crucial to the security of both traditional applications and emerging agent-based systems. Typically, these decisions are made by users during app installation or at runtime. Due to the increasing complexity and automation of systems, making these access control decisions can add a significant cognitive load on users, often overloading them and leading to suboptimal or even arbitrary access control decisions. To address this problem, we propose to leverage the processing and reasoning capabilities of large language models (LLMs) to make dynamic, context-aware decisions aligned with the user's security preferences. For this purpose, we conducted a user study, which resulted in a dataset of 307 natural-language privacy statements and 14,682 access control decisions made by users. We then compare these decisions against those made by two versions of LLMs: a general and a personalized one, for which we also gathered user feedback on 1,446 of its decisions. Our results show that in general, LLMs can reflect users' preferences well, achieving up to 86\% accuracy when compared to the decision made by the majority of users. Our study also reveals a crucial trade-off in personalizing such a system: while providing user-specific privacy preferences to the LLM generally improves agreement with individual user decisions, adhering to those preferences can also violate some security best practices. Based on our findings, we discuss design and risk considerations for implementing a practical natural-language-based access control system that balances personalization, security, and utility.
Authors Friederike Groschupp, Daniele Lain, Aritra Dhar, Lara Magdalena Lazier, Srdjan Čapkun
Categories Artificial Intelligence
0
BiasJailbreak:Analyzing Ethical Biases and Jailbreak Vulnerabilities in Large Language Models AAAI 2026
Title CN BiasJailbreak：分析大语言模型中的伦理偏见与越狱漏洞
Keywords 大语言模型安全 · 越狱攻击 · 伦理偏见 · BiasJailbreak · BiasDefense
Summary 本文提出BiasJailbreak方法，通过自动发现大语言模型中存在的伦理偏见（如性别、种族相关词汇），生成能绕过安全对齐机制的恶意提示，并展示了在GPT-4o上显著的越狱成功率差异。同时提出一种前置防御方法BiasDefense，以较低成本阻止此类攻击。研究关注模型安全性与公平性，强调安全对齐过程中可能引入的新风险。
Reason 该论文聚焦于大语言模型的安全对齐问题，特别是利用伦理偏见进行越狱攻击（jailbreak）的机制与防御。属于纯NLP安全/对齐研究范畴，尽管涉及‘幻觉’相关现象，但其核心是政治正确与社会偏见问题，并非文档图像理解中的视觉或OCR幻觉。无法迁移到DIU领域解决OCR错误、排版误解或细粒度定位等实际问题。
Abstract Although large language models (LLMs) demonstrate impressive proficiency in various tasks, they present potential safety risks, such as `jailbreaks', where malicious inputs can coerce LLMs into generating harmful content bypassing safety alignments. In this paper, we delve into the ethical biases in LLMs and examine how those biases could be exploited for jailbreaks. Notably, these biases result in a jailbreaking success rate in GPT-4o models that differs by 20\% between non-binary and cisgender keywords and by 16\% between white and black keywords, even when the other parts of the prompts are identical. We introduce the concept of BiasJailbreak, highlighting the inherent risks posed by these safety-induced biases. BiasJailbreak generates biased keywords automatically by asking the target LLM itself, and utilizes the keywords to generate harmful output. Additionally, we propose an efficient defense method BiasDefense, which prevents jailbreak attempts by injecting defense prompts prior to generation. BiasDefense stands as an appealing alternative to Guard Models, such as Llama-Guard, that require additional inference cost after text generation. Our findings emphasize that ethical biases in LLMs can actually lead to generating unsafe output, and suggest a method to make the LLMs more secure and unbiased. To enable further research and improvements, we open-source our code and artifacts of BiasJailbreak, providing the community with tools to better understand and mitigate safety-induced biases in LLMs.
Authors Isack Lee, Haebin Seong
Comment Accepted as a workshop paper at AAAI 2026
Categories Artificial Intelligence, Computation and Language
0
Interpretable Air Pollution Forecasting by Physics-Guided Spatiotemporal Decoupling 2025 IEEE International Conference on Big Data
Title CN 基于物理引导时空解耦的可解释空气污染预测
Keywords 空气污染预测 · 可解释性 · 时空建模 · 物理引导模型 · 注意力机制
Summary 本文提出一种物理引导的时空解耦学习框架，用于实现高精度且可解释的空气污染预测。模型将污染物浓度变化分解为两个可解释模块：一是基于风向和地理信息的物理引导传输核（平流），二是通过可解释注意力机制捕捉局部响应，并将未来浓度归因于特定历史时滞和外部驱动因素。在斯德哥尔摩地区的数据集上验证表明，该模型在多个预测时间范围内均优于现有最先进方法。
Reason 该论文研究的是空气污染预测问题，属于环境科学与时间序列预测交叉领域，与文档图像理解（DIU）或视觉-语言模型在文档上的应用无直接关联。尽管其提出的方法具有可解释性设计和时空解耦机制，但这些技术难以迁移到解决OCR幻觉、复杂排版理解、细粒度定位或长文档推理等DIU核心痛点。
Abstract Accurate and interpretable air pollution forecasting is crucial for public health, but most models face a trade-off between performance and interpretability. This study proposes a physics-guided, interpretable-by-design spatiotemporal learning framework. The model decomposes the spatiotemporal behavior of air pollutant concentrations into two transparent, additive modules. The first is a physics-guided transport kernel with directed weights conditioned on wind and geography (advection). The second is an explainable attention mechanism that learns local responses and attributes future concentrations to specific historical lags and exogenous drivers. Evaluated on a comprehensive dataset from the Stockholm region, our model consistently outperforms state-of-the-art baselines across multiple forecasting horizons. Our model's integration of high predictive performance and spatiotemporal interpretability provides a more reliable foundation for operational air-quality management in real-world applications.
Authors Zhiguo Zhang, Xiaoliang Ma, Daniel Schlesinger
Comment Accepted to 2025 IEEE International Conference on Big Data
Categories Artificial Intelligence
0
Learning to Compress Graphs via Dual Agents for Consistent Topological Robustness Evaluation
Title CN 通过双智能体学习图压缩以实现一致的拓扑鲁棒性评估
Keywords 图压缩 · 强化学习 · 双智能体 · 拓扑鲁棒性 · 对抗攻击
Summary 本文提出Cutter框架，使用双智能体强化学习方法对图数据进行压缩，在保留原始图拓扑结构的同时保持其在对抗攻击下的鲁棒性趋势。通过轨迹级奖励塑造、基于原型的引导和跨智能体模仿提升学习效率与压缩质量。实验表明压缩后的图能高效且忠实地用于鲁棒性评估。
Reason 该论文研究的是图数据的压缩与鲁棒性评估问题，属于图神经网络和对抗攻击领域，与文档图像理解（DIU）或视觉语言模型（VLM）在任务、输入模态和核心挑战上均无直接关联。尽管其‘压缩’思想看似相关，但方法针对的是图结构数据的拓扑保持，无法迁移至文档图像的视觉压缩、布局理解或OCR-free识别等任务。
Abstract As graph-structured data grow increasingly large, evaluating their robustness under adversarial attacks becomes computationally expensive and difficult to scale. To address this challenge, we propose to compress graphs into compact representations that preserve both topological structure and robustness profile, enabling efficient and reliable evaluation. We propose Cutter, a dual-agent reinforcement learning framework composed of a Vital Detection Agent (VDA) and a Redundancy Detection Agent (RDA), which collaboratively identify structurally vital and redundant nodes for guided compression. Cutter incorporates three key strategies to enhance learning efficiency and compression quality: trajectory-level reward shaping to transform sparse trajectory returns into dense, policy-equivalent learning signals; prototype-based shaping to guide decisions using behavioral patterns from both high- and low-return trajectories; and cross-agent imitation to enable safer and more transferable exploration. Experiments on multiple real-world graphs demonstrate that Cutter generates compressed graphs that retain essential static topological properties and exhibit robustness degradation trends highly consistent with the original graphs under various attack scenarios, thereby significantly improving evaluation efficiency without compromising assessment fidelity.
Authors Qisen Chai, Yansong Wang, Junjie Huang, Tao Jia
Categories Artificial Intelligence
0
XiCAD: Camera Activation Detection in the Da Vinci Xi User Interface
Title CN XiCAD：达芬奇Xi用户界面中的相机激活检测
Keywords 相机激活检测 · 达芬奇手术系统 · UI识别 · ResNet18 · 手术视频分析
Summary 本文提出了一种基于ResNet18的轻量级方法，用于在达芬奇Xi系统的用户界面中自动检测内窥镜相机图块的位置及其激活状态。模型在SurgToolLoc等公开数据集上进行了微调和评估，在超过7万帧图像上取得了接近完美的F1分数，能够实现实时可靠的相机激活元数据提取，支持手术工具追踪、技能评估等下游任务。
Reason 该论文聚焦于手术机器人界面中的相机状态检测，属于具身智能与医疗视频分析的交叉领域，与文档图像理解（DIU）或DocVQA无关。其任务目标、数据模态和应用场景均远离文档结构、文本识别或视觉语言推理，无法迁移解决OCR幻觉、复杂排版或长文档理解等核心问题。
Abstract Purpose: Robot-assisted minimally invasive surgery relies on endoscopic video as the sole intraoperative visual feedback. The DaVinci Xi system overlays a graphical user interface (UI) that indicates the state of each robotic arm, including the activation of the endoscope arm. Detecting this activation provides valuable metadata such as camera movement information, which can support downstream surgical data science tasks including tool tracking, skill assessment, or camera control automation. Methods: We developed a lightweight pipeline based on a ResNet18 convolutional neural network to automatically identify the position of the camera tile and its activation state within the DaVinci Xi UI. The model was fine-tuned on manually annotated data from the SurgToolLoc dataset and evaluated across three public datasets comprising over 70,000 frames. Results: The model achieved F1-scores between 0.993 and 1.000 for the binary detection of active cameras and correctly localized the camera tile in all cases without false multiple-camera detections. Conclusion: The proposed pipeline enables reliable, real-time extraction of camera activation metadata from surgical videos, facilitating automated preprocessing and analysis for diverse downstream applications. All code, trained models, and annotations are publicly available.
Authors Alexander C. Jenke, Gregor Just, Claas de Boer, Martin Wagner, Sebastian Bodenstedt, Stefanie Speidel
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
LFaB: Low fidelity as Bias for Active Learning in the chemical configuration space
Title CN 化学构型空间中用于主动学习的低保真度作为偏差
Keywords 主动学习 · 偏差-方差分解 · 多保真学习 · 量子化学 · Δ-机器学习
Summary 本文提出一种基于低保真数据来估计模型偏差的主动学习方法，在量子化学任务（如激发能和从头算势能面预测）中显著降低训练数据需求。该方法通过最小化偏差而非方差实现更优的样本选择，但在文档图像理解领域缺乏可迁移的技术路径。
Reason 该论文属于量子化学与多保真机器学习交叉领域，研究在化学配置空间中利用低保真数据近似模型偏差以优化主动学习采样。其方法论聚焦于物理模拟场景下的数据效率提升，并未涉及文档图像、多模态理解或视觉语言模型等DIU相关技术栈。无法迁移解决OCR幻觉、复杂排版或长文档推理等核心问题。
Abstract Active learning promises to provide an optimal training sample selection procedure in the construction of machine learning models. It often relies on minimizing the model's variance, which is assumed to decrease the prediction error. Still, it is frequently even less efficient than pure random sampling. Motivated by the bias-variance decomposition, we propose to minimize the model's bias instead of its variance. By doing so, we are able to almost exactly match the best-case error over all possible greedy sample selection procedures for a relevant application. Our bias approximation is based on using cheap to calculate low fidelity data as known from $Δ$-ML or multifidelity machine learning. We exemplify our approach for a wider class of applications in quantum chemistry including predicting excitation energies and ab initio potential energy surfaces. Here, the proposed method reduces training data consumption by up to an order of magnitude compared to standard active learning.
Authors Vivin Vinod, Peter Zaspel
Comment SI included in main
Categories Artificial Intelligence
0
Uplifting Table Tennis: A Robust, Real-World Application for 3D Trajectory and Spin Estimation
Title CN 提升乒乓球运动：一种用于3D轨迹和旋转估计的鲁棒、真实世界应用
Keywords 3D轨迹估计 · 单目视频 · 合成数据 · 物理建模 · 运动分析
Summary 本文提出了一种两阶段方法，用于从单目视频中估计乒乓球的真实3D轨迹和旋转状态。前端使用新构建的TTHQ数据集进行2D检测，后端通过在合成物理数据上训练的‘提升’网络实现2D到3D的转换，并增强了对缺失检测和帧率变化的鲁棒性。整体为特定体育场景下的3D视觉重建应用。
Reason 该论文属于无关领域（体育视频分析与3D轨迹估计），研究目标是基于单目视频进行乒乓球的3D运动和旋转参数估计，完全不涉及文档图像理解（DIU）、文本识别、布局分析或任何与文档相关的任务。其方法聚焦于物理建模与合成数据迁移，无法迁移到OCR幻觉、密集文本处理或复杂排版理解等DIU核心痛点。
Abstract Obtaining the precise 3D motion of a table tennis ball from standard monocular videos is a challenging problem, as existing methods trained on synthetic data struggle to generalize to the noisy, imperfect ball and table detections of the real world. This is primarily due to the inherent lack of 3D ground truth trajectories and spin annotations for real-world video. To overcome this, we propose a novel two-stage pipeline that divides the problem into a front-end perception task and a back-end 2D-to-3D uplifting task. This separation allows us to train the front-end components with abundant 2D supervision from our newly created TTHQ dataset, while the back-end uplifting network is trained exclusively on physically-correct synthetic data. We specifically re-engineer the uplifting model to be robust to common real-world artifacts, such as missing detections and varying frame rates. By integrating a ball detector and a table keypoint detector, our approach transforms a proof-of-concept uplifting method into a practical, robust, and high-performing end-to-end application for 3D table tennis trajectory and spin analysis.
Authors Daniel Kienzle, Katja Ludwig, Julian Lorenz, Shin'ichi Satoh, Rainer Lienhart
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Actionable and diverse counterfactual explanations incorporating domain knowledge and causal constraints
Title CN 结合领域知识和因果约束的可操作且多样化的反事实解释
Keywords 反事实解释 · 可解释AI · 因果约束 · 领域知识 · 机器学习可解释性
Summary 本文提出了一种名为DANCE的方法，用于生成多样化、可操作且受领域知识和因果约束限制的反事实解释。该方法通过从数据中学习线性和非线性约束，或整合专家提供的依赖图，确保生成的反事实修改在现实中可行且合理。研究基于波兰最大的电子邮件营销公司Freshmail的实际案例，并在140个公开数据集上进行了广泛评估，结果表明其在生成有意义且符合领域特性的反事实方面优于现有方法。
Reason 该论文研究的是反事实解释的生成方法，主要应用于模型可解释性领域，尤其是电子邮件营销中的网络安全场景。其核心贡献是提出DANCE方法以生成符合现实约束的反事实样本，属于特定领域的可解释AI应用。尽管涉及特征依赖与因果结构，但并未针对文档图像理解（DIU）或视觉-语言模型（VLM）在OCR、布局分析、表格识别等任务中的核心痛点（如幻觉、高分辨率处理、细粒度定位）提供直接的方法论创新。同时，该工作属于‘平行下游应用’类型——将现有解释技术应用于特定领域（邮件营销），缺乏对DIU通用技术栈的推动，因此不符合关注标准。
Abstract Counterfactual explanations enhance the actionable interpretability of machine learning models by identifying the minimal changes required to achieve a desired outcome of the model. However, existing methods often ignore the complex dependencies in real-world datasets, leading to unrealistic or impractical modifications. Motivated by cybersecurity applications in the email marketing domain, we propose a method for generating Diverse, Actionable, and kNowledge-Constrained Explanations (DANCE), which incorporates feature dependencies and causal constraints to ensure plausibility and real-world feasibility of counterfactuals. Our method learns linear and nonlinear constraints from data or integrates expert-provided dependency graphs, ensuring counterfactuals are plausible and actionable. By maintaining consistency with feature relationships, the method produces explanations that align with real-world constraints. Additionally, it balances plausibility, diversity, and sparsity, effectively addressing key limitations in existing algorithms. The work is developed based on a real-life case study with Freshmail, the largest email marketing company in Poland and supported by a joint R&D project Sendguard. Furthermore, we provide an extensive evaluation using 140 public datasets, which highlights its ability to generate meaningful, domain-relevant counterfactuals that outperform other existing approaches based on widely used metrics. The source code for reproduction of the results can be found in a GitHub repository we provide.
Authors Szymon Bobek, Łukasz Bałec, Grzegorz J. Nalepa
Categories Artificial Intelligence
0
Leveraging weights signals - Predicting and improving generalizability in reinforcement learning
Title CN 利用权重信号——预测和提升强化学习中的泛化能力
Keywords 强化学习 · 泛化能力 · 权重分析 · PPO · 过拟合
Summary 本文提出一种通过分析强化学习智能体神经网络内部权重来预测其泛化能力的方法，并据此改进PPO的损失函数以提升泛化性。实验表明改进后的算法能训练出更具泛化能力的智能体。
Reason 该论文属于强化学习领域，研究的是RL智能体在不同环境下的泛化能力预测与优化，核心方法基于神经网络权重分析和PPO损失函数改进。其研究问题与文档图像理解（DIU）无直接关联，既不涉及多模态建模、视觉文本理解、OCR、布局分析等DIU任务，也未提出可迁移至文档理解领域的通用技术（如高分辨率处理、幻觉抑制、细粒度定位等）。属于无关领域研究。
Abstract Generalizability of Reinforcement Learning (RL) agents (ability to perform on environments different from the ones they have been trained on) is a key problem as agents have the tendency to overfit to their training environments. In order to address this problem and offer a solution to increase the generalizability of RL agents, we introduce a new methodology to predict the generalizability score of RL agents based on the internal weights of the agent's neural networks. Using this prediction capability, we propose some changes in the Proximal Policy Optimization (PPO) loss function to boost the generalization score of the agents trained with this upgraded version. Experimental results demonstrate that our improved PPO algorithm yields agents with stronger generalizability compared to the original version.
Authors Olivier Moulin, Vincent Francois-lavet, Paul Elbers, Mark Hoogendoorn
Categories Artificial Intelligence
0
DUO-TOK: Dual-Track Semantic Music Tokenizer for Vocal-Accompaniment Generation
Title CN DUO-TOK：用于人声伴奏生成的双轨语义音乐分词器
Keywords 音乐分词器 · 双码本 · 自监督学习 · 语音-伴奏分离 · 语义压缩
Summary DUO-TOK提出了一种面向人声与伴奏双轨结构的语义分词器，通过自监督学习流程构建双码本量化系统，在保持重建质量的同时提升语言模型的可学习性。该方法专注于音乐生成中的表示学习，不涉及文本布局、视觉结构或文档语义理解。
Reason 该论文研究的是音乐生成领域中的语音-伴奏分离与语义分词，属于音频处理与音乐信息检索范畴，与文档图像理解（DIU）或视觉文档问答（DocVQA）无直接关联。尽管其采用了多模态建模范式，但任务目标、数据模态和应用场景均远离文档理解核心问题，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU痛点。
Abstract Duo-Tok is a source-aware dual-codebook tokenizer for vocal-accompaniment music that targets the growing tension between reconstruction quality and language-model (LM) learnability in modern lyrics-to-song systems. Existing codecs either prioritize high-fidelity reconstruction with difficult-to-model acoustic tokens or compress aggressively into semantic tokens that are LM-friendly but lossy, and they rarely make the tokenizer itself aware of dual-track structure. Duo-Tok follows a four-stage, SSL-centered pipeline: we first pretrain a BEST-RQ-style encoder on large-scale audio, then stabilize and factorize the representation with Gaussian replacement noise and multi-task supervision, before freezing the encoder to learn SimVQ-based dual codebooks with hard routing for vocals and accompaniment, and finally training latent diffusion decoders on top of the discrete tokens. Duo-Tok at 0.75 kbps shifts the empirical reconstruction-generation Pareto frontier, achieving the best music-tagging AP and the lowest vocabulary-normalized LM perplexity among compared codecs while maintaining reconstruction quality comparable to state-of-the-art music tokenizers.
Authors Rui Lin, Zhiyue Wu, Jiahe Le, Kangdi Wang, Weixiong Chen, Junyu Dai, Tao Jiang
Comment 17 pages, 5 figures, 8 tables. Project page: https://eps-acoustic-revolution-lab.github.io/DUO_TOK/
Categories Artificial Intelligence
0
CostNav: A Navigation Benchmark for Cost-Aware Evaluation of Embodied Agents
Title CN CostNav：面向具身智能体成本感知评估的导航基准
Keywords 具身智能 · 导航基准 · 成本建模 · 商业可行性 · 机器人
Summary 本文提出了CostNav，一个用于评估具身智能体（如自主配送机器人）导航系统经济可行性的基准平台。它不仅关注任务成功率，还引入了硬件、训练、能耗、维护和收入等真实商业成本因素，揭示了当前导航研究指标与实际商业部署之间的巨大差距。实验表明，现有基线方法虽有一定SLA合规率，但因碰撞导致的维护成本过高而无法盈利。该工作旨在推动更具成本效益的导航算法发展。
Reason 该论文属于具身智能/机器人领域，研究的是自主配送机器人的商业经济性评估，与文档图像理解（DIU）或视觉语言模型在文档任务中的应用无直接关联。尽管其涉及视觉感知和决策，但核心问题是机器人导航的成本建模与商业可行性分析，无法迁移解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点。
Abstract Existing navigation benchmarks focus on task success metrics while overlooking economic viability -- critical for commercial deployment of autonomous delivery robots. We introduce \emph{CostNav}, a \textbf{Micro-Navigation Economic Testbed} that evaluates embodied agents through comprehensive cost-revenue analysis aligned with real-world business operations. CostNav models the complete economic lifecycle including hardware, training, energy, maintenance costs, and delivery revenue with service-level agreements, using industry-derived parameters. \textbf{To our knowledge, CostNav is the first work to quantitatively expose the gap between navigation research metrics and commercial viability}, revealing that optimizing for task success fundamentally differs from optimizing for economic deployment. Our cost model uses parameters derived from industry data sources (energy rates, delivery service pricing), and we project from a reduced-scale simulation to realistic deliveries. Under this projection, the baseline achieves 43.0\% SLA compliance but is \emph{not} commercially viable: yielding a loss of \$30.009 per run with no finite break-even point, because operating costs are dominated by collision-induced maintenance, which accounts for 99.7\% of per-run costs and highlights collision avoidance as a key optimization target. We demonstrate a learning-based on-device navigation baseline and establish a foundation for evaluating rule-based navigation, imitation learning, and cost-aware RL training. CostNav bridges the gap between navigation research and commercial deployment, enabling data-driven decisions about economic trade-offs across navigation paradigms.
Authors Haebin Seong, Sungmin Kim, Minchan Kim, Yongjun Cho, Myunchul Joe, Suhwan Choi, Jaeyoon Jung, Jiyong Youn, Yoonshik Kim, Samwoo Seong, Yubeen Park, Youngjae Yu, Yunsung Lee
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
OmniAlpha: A Sequence-to-Sequence Framework for Unified Multi-Task RGBA Generation
Title CN OmniAlpha：一种用于统一多任务RGBA生成的序列到序列框架
Keywords RGBA生成 · 图像编辑 · 扩散Transformer · 多任务学习 · 图层合成
Summary 本文提出了OmniAlpha，首个支持多任务RGBA图像生成与编辑的统一序列到序列生成框架。通过引入新型RoPE方法MSRoPE-BiL和专用数据集AlphaLayers，在21个RGBA相关任务上实现了超越专用模型的表现，尤其在无掩码抠图和图层补全任务中性能显著。研究推动了图层感知生成系统的发展。
Reason 该论文聚焦于RGBA图像生成与编辑，属于纯图像生成领域，核心任务为图像合成、抠图和图层操作，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。尽管其提出的MSRoPE-BiL架构和多任务框架具有一定方法论价值，但其应用场景集中在通用图像生成而非文档结构解析、文本识别或视觉信息抽取等DIU核心痛点，迁移潜力极低。
Abstract Generative models have excelled in RGB synthesis, but real-world applications require RGBA manipulation. This has led to a fragmented landscape: specialized, single-task models handle alpha but lack versatility, while unified multi-task frameworks are confined to the RGB domain. To bridge this critical gap, we propose OmniAlpha, the first unified, multi-task generative framework for sequence-to-sequence RGBA image generation and editing. Its architecture features MSRoPE-BiL, a novel RoPE method with a bi-directionally extendable layer axis for its Diffusion Transformer (DiT) backbone, enabling the concurrent processing of multiple input and target RGBA layers. To power this framework, we introduce AlphaLayers, a new dataset of 1,000 high-quality, multi-layer triplets, built via a novel automated synthesis and filter pipeline. Jointly training OmniAlpha on this dataset across a comprehensive suite of 21 diverse tasks, extensive experiments demonstrate that our unified approach consistently outperforms strong, specialized baselines. Most notably, OmniAlpha achieves a dramatic 84.8% relative reduction in SAD for mask-free matting on AIM-500 and wins over 90% of human preferences in layer-conditioned completion. Our work proves that a unified, multi-task model can learn a superior shared representation for RGBA, paving the way for more powerful, layer-aware generative systems.
Authors Hao Yu, Jiabo Zhan, Zile Wang, Jinglin Wang, Huaisong Zhang, Hongyu Li, Xinrui Chen, Yongxian Wei, Chun Yuan
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Value Improved Actor Critic Algorithms
Title CN 价值改进的演员-评论家算法
Keywords 强化学习 · 演员-评论家 · 值改进 · 策略迭代 · 稳定性
Summary 本文提出将演员-评论家算法中的行为策略与评论家评估的策略进行解耦，通过更贪婪地更新评论家的策略（如使用值改进），同时保持行为策略的渐进梯度更新，以平衡学习过程中的贪婪性与稳定性。实验表明该方法在连续控制任务上能提升TD3和SAC等算法的性能。
Reason 该论文属于强化学习理论改进，研究的是Actor-Critic框架中策略更新与价值函数优化的解耦问题，主要应用于连续控制等决策任务。尽管其方法在RL领域有一定理论价值，但与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联，无法迁移解决OCR幻觉、复杂排版解析、细粒度定位或长文档推理等核心痛点，属于无关领域的方法研究。
Abstract To learn approximately optimal acting policies for decision problems, modern Actor Critic algorithms rely on deep Neural Networks (DNNs) to parameterize the acting policy and greedification operators to iteratively improve it. The reliance on DNNs suggests an improvement that is gradient based, which is per step much less greedy than the improvement possible by greedier operators such as the greedy update used by Q-learning algorithms. On the other hand, slow changes to the policy can also be beneficial for the stability of the learning process, resulting in a tradeoff between greedification and stability. To better address this tradeoff, we propose to decouple the acting policy from the policy evaluated by the critic. This allows the agent to separately improve the critic's policy (e.g. value improvement) with greedier updates while maintaining the slow gradient-based improvement to the parameterized acting policy. We investigate the convergence of this approach using the popular analysis scheme of generalized Policy Iteration in the finite-horizon domain. Empirically, incorporating value-improvement into the popular off-policy actor-critic algorithms TD3 and SAC significantly improves or matches performance over their respective baselines, across different environments from the DeepMind continuous control domain, with negligible compute and implementation cost.
Authors Yaniv Oren, Moritz A. Zanger, Pascal R. van der Vaart, Mustafa Mert Celikok, Matthijs T. J. Spaan, Wendelin Bohmer
Categories Artificial Intelligence
0
Exploring the Synergy of Quantitative Factors and Newsflow Representations from Large Language Models for Stock Return Prediction
Title CN 探索量化因子与大型语言模型生成的新闻流表征在股票收益预测中的协同作用
Keywords 量化投资 · 大语言模型 · 多模态融合 · 股票收益预测 · 新闻流分析
Summary 本文研究如何结合量化因子和由大语言模型生成的新闻流表征来进行股票收益预测。作者提出了一个融合学习框架，并比较了三种不同复杂度的表示融合方式，同时探索了一种解耦训练的混合模型来提升预测稳定性。实验在真实投资数据上进行，展示了多模态建模的有效性。
Reason 该论文属于典型的‘平行下游应用’，将LLM用于金融领域的股票收益预测和投资组合优化，本质上是将现有NLP和多模态融合技术应用于特定垂直领域（金融），并未提出任何面向文档图像理解（DIU）或视觉-语言建模的新架构、新训练方法或底层技术创新。其方法论聚焦于模态融合策略和混合模型训练稳定性，与OCR幻觉、密集文本识别、复杂版面分析等DIU核心问题无直接关联，也无法迁移为解决文档理解痛点的通用工具。
Abstract In quantitative investing, return prediction supports various tasks, including stock selection, portfolio optimization, and risk management. Quantitative factors, such as valuation, quality, and growth, capture various characteristics of stocks. Unstructured data, like news and transcripts, has attracted growing attention, driven by recent advances in large language models (LLMs). This paper examines effective methods for leveraging multimodal factors and newsflow in return prediction and stock selection. First, we introduce a fusion learning framework to learn a unified representation from factors and newsflow representations generated by an LLM. Within this framework, we compare three methods of different architectural complexities: representation combination, representation summation, and attentive representations. Next, building on the limitation of fusion learning observed in empirical comparison, we explore the mixture model that adaptively combines predictions made by single modalities and their fusion. To mitigate the training instability of the mixture model, we introduce a decoupled training approach with theoretical insights. Finally, our experiments on real investment universes yield several insights into effective multimodal modeling of factors and news for stock return prediction and selection.
Authors Tian Guo, Emmanuel Hauptmann
Categories Artificial Intelligence, Computation and Language
0
Interactive AI NPCs Powered by LLMs: Technical Report for the CPDC Challenge 2025
Title CN 由大语言模型驱动的交互式AI NPC：CPDC挑战赛2025技术报告
Keywords LLM · 对话系统 · 强化学习 · 提示工程 · 虚拟角色
Summary 本文提出了一个用于CPDC 2025挑战赛的AI NPC框架，通过上下文工程（如动态工具剪枝、persona裁剪）和后处理技术提升对话稳定性，并在GPU赛道中采用GRPO强化学习替代监督微调以优化奖励信号。尽管取得了优异排名，但工作局限于任务导向的对话系统，无涉文档图像或多模态文档理解。
Reason 该论文属于具身智能/虚拟角色对话系统在特定挑战赛中的应用，核心内容为对话策略优化与提示工程，研究场景完全脱离文档图像理解任务。其方法聚焦于persona-grounded对话和工具调用稳定性，无法迁移至DIU领域的OCR幻觉、复杂版面解析或细粒度视觉定位等关键问题。
Abstract This report presents the solution and results of our team MSRA\_SC in the Commonsense Persona-Grounded Dialogue Challenge (CPDC 2025). We propose a simple yet effective framework that unifies improvements across both GPU Track and API Track. Our method centers on two key components. First, Context Engineering applies dynamic tool pruning and persona clipping for input compression, combined with post-processing techniques such as parameter normalization and function merging. Together with manually refined prompts, this design improves tool call stability, execution reliability, and role-playing guidance. Second, in the GPU Track, we further adopt GRPO training, replacing supervised fine-tuning with reinforcement learning directly optimized by reward signals. This mitigates small-sample overfitting and significantly enhances task-oriented dialogue performance. In the final evaluation, our team ranks 1st in Task 2 API, 2nd in Task 1 API, and 3rd in both Task 3 API and GPU track, demonstrating the effectiveness of our approach. Our code is publicly available at https://gitlab.aicrowd.com/nikoo_yu/cpdc-2025-winning-solution
Authors Yitian Huang, Yuxuan Lei, Jianxun Lian, Hao Liao
Categories Artificial Intelligence
0
Panoramic Distortion-Aware Tokenization for Person Detection and Localization in Overhead Fisheye Images
Title CN 面向顶视鱼眼图像中人物检测与定位的全景畸变感知分词方法
Keywords 鱼眼图像 · 人物检测 · 全景投影 · Transformer · 小目标检测
Summary 本文提出将鱼眼图像重映射为等距柱状全景图，并利用全景几何特性设计了一种畸变感知的特征分词方法，以提升顶视角度下小尺寸行人的检测效果。通过自相似图形单元划分特征并保留各块中的最大显著值，增强了对小型人物的响应。
Reason 该论文研究的是鱼眼图像中的人物检测与定位，属于监控或具身智能场景下的计算机视觉任务，核心问题是处理旋转和小目标检测。尽管提出了‘畸变感知分词’等技术，但其方法针对的是球面投影几何下的行人检测，与文档图像理解（DIU）中的OCR幻觉、密集文本识别、复杂版式分析或长文档推理等核心痛点完全无关。无法迁移至文档图像领域。
Abstract Person detection in overhead fisheye images is challenging due to person rotation and small persons. Prior work has mainly addressed person rotation, leaving the small-person problem underexplored. We remap fisheye images to equirectangular panoramas to handle rotation and exploit panoramic geometry to handle small persons more effectively. Conventional detection methods tend to favor larger persons because they dominate the attention maps, causing smaller persons to be missed. In hemispherical equirectangular panoramas, we find that apparent person height decreases approximately linearly with the vertical angle near the top of the image. Using this finding, we introduce panoramic distortion-aware tokenization to enhance the detection of small persons. This tokenization procedure divides panoramic features using self-similar figures that enable the determination of optimal divisions without gaps, and we leverage the maximum significance values in each tile of the token groups to preserve the significance areas of smaller persons. We propose a transformer-based person detection and localization method that combines panoramic-image remapping and the tokenization procedure. Extensive experiments demonstrated that our method outperforms conventional methods on large-scale datasets.
Authors Nobuhiko Wakai, Satoshi Sato, Yasunori Ishii, Takayoshi Yamashita
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Human-computer interactions predict mental health
Title CN 人机交互预测心理健康
Keywords 人机交互 · 心理健康预测 · 行为分析 · 机器学习 · 数字表型
Summary 该研究提出名为MAILA的机器学习框架，利用超过20,000段光标和触摸屏操作记录，预测用户的自我报告式心理健康状况。模型能在群体层面实现接近上限的准确率，识别抑郁症和强迫症患者，并捕捉语言之外的心理功能特征。研究强调日常人机交互可用于被动、可扩展的心理健康评估，但未涉及任何文档图像或多模态语言模型在视觉文档上的理解任务。
Reason 该论文属于纯行为分析与心理健康预测领域，研究通过鼠标和触摸屏交互数据预测用户心理状态，虽使用机器学习方法处理序列行为数据，但其任务目标、数据模态与文档图像理解（DIU）无直接关联。未涉及OCR、视觉布局分析、多模态文档问答或VLM在文档上的应用，也无法迁移用于解决OCR幻觉、复杂排版、细粒度定位等DIU核心痛点。
Abstract Scalable assessments of mental illness, the leading driver of disability worldwide, remain a critical roadblock toward accessible and equitable care. Here, we show that human-computer interactions encode multiple dimensions of self-reported mental health and their changes over time. We introduce MAILA, a MAchine-learning framework for Inferring Latent mental states from digital Activity. We trained MAILA to predict 1.3 million mental-health self-reports from 20,000 cursor and touchscreen recordings recorded in 9,000 online participants. The dataset includes 2,000 individuals assessed longitudinally, 1,500 diagnosed with depression, and 500 with obsessive-compulsive disorder. MAILA tracks dynamic mental states along three orthogonal dimensions, generalizes across contexts, and achieves near-ceiling accuracy when predicting group-level mental health. The model translates from general to clinical populations, identifies individuals living with mental illness, and captures signatures of psychological function that are not conveyed by language. Our results demonstrate how everyday human-computer interactions can power passive, reliable, dynamic, and maximally scalable mental health assessments. The ability to decode mental states at zero marginal cost sets new benchmarks for precision medicine and public health, while raising important questions about privacy, agency, and autonomy online.
Authors Veith Weilnhammer, Jefferson Ortega, David Whitney
Categories Artificial Intelligence
0
Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management SIGMOD'26
Title CN Beluga：一种基于CXL的可扩展高效LLM KVCache管理内存架构
Keywords CXL · KVCache · 内存架构 · LLM推理优化 · vLLM
Summary 本文提出Beluga，一种基于CXL技术的新型内存架构，使GPU和CPU可通过CXL交换机访问共享的大规模内存池，从而高效管理大语言模型推理中的KVCache。相比RDMA方案，Beluga显著降低了首令牌延迟并提升了吞吐量。这是首个实现GPU通过CXL交换机直接访问大规模内存池的系统。
Reason 该论文聚焦于LLM推理中的KVCache内存管理，属于底层系统优化与硬件架构设计，尽管对大模型推理效率有显著提升，但其方法与文档图像理解（DIU）或视觉-语言模型在文档任务中的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）无直接关联。虽然高效KVCache可能间接支持长文档处理，但该工作本身不涉及多模态输入、视觉布局建模或文档特定结构理解，属于通用LLM基础设施，无法迁移解决DIU本质问题。
Abstract The rapid increase in LLM model sizes and the growing demand for long-context inference have made memory a critical bottleneck in GPU-accelerated serving systems. Although high-bandwidth memory (HBM) on GPUs offers fast access, its limited capacity necessitates reliance on host memory (CPU DRAM) to support larger working sets such as the KVCache. However, the maximum DRAM capacity is constrained by the limited number of memory channels per CPU socket. To overcome this limitation, current systems often adopt RDMA-based disaggregated memory pools, which introduce significant challenges including high access latency, complex communication protocols, and synchronization overhead. Fortunately, the emerging CXL technology introduces new opportunities in KVCache design. In this paper, we propose Beluga, a novel memory architecture that enables GPUs and CPUs to access a shared, large-scale memory pool through CXL switches. By supporting native load/store access semantics over the CXL fabric, our design delivers near-local memory latency, while reducing programming complexity and minimizing synchronization overhead. We conduct a systematic characterization of a commercial CXL switch-based memory pool and propose a set of design guidelines. Based on Beluga, we design and implement Beluga-KVCache, a system tailored for managing the large-scale KVCache in LLM inference. Beluga-KVCache achieves an 89.6% reduction in Time-To-First-Token (TTFT) and 7.35x throughput improvement in the vLLM inference engine compared to RDMA-based solutions. To the best of our knowledge, Beluga is the first system that enables GPUs to directly access large-scale memory pools through CXL switches, marking a significant step toward low-latency, shared access to vast memory resources by GPUs.
Authors Xinjun Yang, Qingda Hu, Junru Li, Feifei Li, Yuqi Zhou, Yicong Zhu, Qiuru Lin, Jian Dai, Yang Kong, Jiayu Zhang, Guoqiang Xu, Qiang Liu
Comment 13 pages, accepted by SIGMOD'26
Categories Artificial Intelligence
0
MindEval: Benchmarking Language Models on Multi-turn Mental Health Support
Title CN MindEval：在多轮心理健康支持中对语言模型进行基准测试
Keywords 心理健康 · 对话系统 · 基准测试 · 语言模型评估 · 多轮对话
Summary 该研究提出了MindEval框架，用于在多轮心理健康支持对话中自动评估大语言模型的表现。通过与临床心理学家合作构建模拟患者，并利用LLM进行自动化评分，验证了系统的可重复性和与人类评价的相关性。结果显示现有模型在长期交互和严重症状支持任务上表现不佳。
Reason 该论文属于特定领域的下游应用（心理健康对话系统），聚焦于心理治疗场景中的LLM评估，未提出可用于文档图像理解（DIU）的通用方法或底层技术创新。尽管其自动评估框架具有一定工程价值，但属于‘平行应用’，不涉及OCR、视觉布局分析、多模态文档建模等DIU核心问题，也无法迁移解决OCR幻觉、复杂排版或长文档推理等痛点。
Abstract Demand for mental health support through AI chatbots is surging, though current systems present several limitations, like sycophancy or overvalidation, and reinforcement of maladaptive beliefs. A core obstacle to the creation of better systems is the scarcity of benchmarks that capture the complexity of real therapeutic interactions. Most existing benchmarks either only test clinical knowledge through multiple-choice questions or assess single responses in isolation. To bridge this gap, we present MindEval, a framework designed in collaboration with Ph.D-level Licensed Clinical Psychologists for automatically evaluating language models in realistic, multi-turn mental health therapy conversations. Through patient simulation and automatic evaluation with LLMs, our framework balances resistance to gaming with reproducibility via its fully automated, model-agnostic design. We begin by quantitatively validating the realism of our simulated patients against human-generated text and by demonstrating strong correlations between automatic and human expert judgments. Then, we evaluate 12 state-of-the-art LLMs and show that all models struggle, scoring below 4 out of 6, on average, with particular weaknesses in problematic AI-specific patterns of communication. Notably, reasoning capabilities and model scale do not guarantee better performance, and systems deteriorate with longer interactions or when supporting patients with severe symptoms. We release all code, prompts, and human evaluation data.
Authors José Pombal, Maya D'Eon, Nuno M. Guerreiro, Pedro Henrique Martins, António Farinhas, Ricardo Rei
Categories Artificial Intelligence, Computation and Language
0
On the Limits of Momentum in Decentralized and Federated Optimization 17th Workshop on Optimization for Machine Learning (OPT@NeurIPS2025)
Title CN 论去中心化与联邦优化中动量的局限性
Keywords 联邦学习 · 动量优化 · 分布式SGD · 统计异质性 · 收敛性分析
Summary 本文从理论上分析了在循环客户端参与的去中心化场景下，动量方法在面对无界统计异质性时的收敛局限性，证明即使使用递减步长，动量仍会受初始值和异质性影响而无法完全收敛到最优解，并通过实验验证了结论。
Reason 该论文研究的是联邦学习和分布式优化中的动量方法在统计异构性下的收敛性问题，属于纯优化理论范畴，与文档图像理解（DIU）、多模态大模型（VLM）或视觉-语言任务无直接关联。其方法无法迁移用于解决OCR幻觉、复杂排版解析、细粒度定位或高分辨率文档处理等DIU核心痛点。
Abstract Recent works have explored the use of momentum in local methods to enhance distributed SGD. This is particularly appealing in Federated Learning (FL), where momentum intuitively appears as a solution to mitigate the effects of statistical heterogeneity. Despite recent progress in this direction, it is still unclear if momentum can guarantee convergence under unbounded heterogeneity in decentralized scenarios, where only some workers participate at each round. In this work we analyze momentum under cyclic client participation, and theoretically prove that it remains inevitably affected by statistical heterogeneity. Similarly to SGD, we prove that decreasing step-sizes do not help either: in fact, any schedule decreasing faster than $Θ\left(1/t\right)$ leads to convergence to a constant value that depends on the initialization and the heterogeneity bound. Numerical results corroborate the theory, and deep learning experiments confirm its relevance for realistic settings.
Authors Riccardo Zaccone, Sai Praneeth Karimireddy, Carlo Masone
Comment Accepted at the 17th Workshop on Optimization for Machine Learning (OPT@NeurIPS2025)
Categories Artificial Intelligence
0
BackFed: An Efficient & Standardized Benchmark Suite for Backdoor Attacks in Federated Learning
Title CN BackFed：联邦学习中后门攻击的高效且标准化基准套件
Keywords 联邦学习 · 后门攻击 · 基准测试 · 安全性 · 隐私
Summary 本文提出BackFed，一个用于评估联邦学习中后门攻击的标准化基准框架，旨在解决现有研究中评估设置不一致和代码存在语义错误的问题。通过统一实验框架，揭示了当前攻击和防御方法在现实部署中的局限性。
Reason 该论文属于安全与隐私领域，研究联邦学习中的后门攻击与防御评估，与文档图像理解（DIU）、视觉语言模型（VLM）或OCR等核心任务完全无关。尽管其在联邦学习评估方面具有贡献，但无法迁移解决DIU领域的任何痛点（如OCR幻觉、复杂排版理解、细粒度定位等），属于无关领域。
Abstract Research on backdoor attacks in Federated Learning (FL) has accelerated in recent years, with new attacks and defenses continually proposed in an escalating arms race. However, the evaluation of these methods remains neither standardized nor reliable. First, there are severe inconsistencies in the evaluation settings across studies, and many rely on unrealistic threat models. Second, our code review uncovers semantic bugs in the official codebases of several attacks that artificially inflate their reported performance. These issues raise fundamental questions about whether current methods are truly effective or simply overfitted to narrow experimental setups. We introduce \textbf{BackFed}, a benchmark designed to standardize and stress-test FL backdoor evaluation by unifying attacks and defenses under a common evaluation framework that mirrors realistic FL deployments. Our benchmark on three representative datasets with three distinct architectures reveals critical limitations of existing methods. Malicious clients often require excessive training time and computation, making them vulnerable to server-enforced time constraints. Meanwhile, several defenses incur severe accuracy degradation or aggregation overhead. Popular defenses and attacks achieve limited performance in our benchmark, which challenges their previous efficacy claims. We establish BackFed as a rigorous and fair evaluation framework that enables more reliable progress in FL backdoor research.
Authors Thinh Dao, Dung Thuy Nguyen, Khoa D Doan, Kok-Seng Wong
Comment Our framework is openly available at https://github.com/thinh-dao/BackFed
Categories Artificial Intelligence
0
DecNefLab: A Modular and Interpretable Simulation Framework for Decoded Neurofeedback
Title CN DecNefLab：一种用于解码神经反馈的模块化且可解释的模拟框架
Keywords 解码神经反馈 · 计算神经科学 · 生成模型 · 模拟框架 · 机器学习
Summary 本文提出了DecNefLab，一个用于解码神经反馈（DecNef）研究的模块化模拟框架。该框架将DecNef建模为机器学习问题，利用潜在变量生成模型模拟参与者的大脑状态，从而在虚拟环境中分析神经反馈的学习动态。通过该系统，研究人员可以观察内部认知状态，评估不同实验协议对学习的影响，并在实际人体实验前优化协议设计。虽然该框架具有可解释性和系统性分析能力，但其应用场景完全脱离文档理解或多模态视觉语言处理领域。
Reason 该论文研究的是解码神经反馈（DecNef）在认知神经科学中的应用，属于脑机接口与计算神经科学领域，与文档图像理解（DIU）、视觉语言模型（VLM）或OCR等核心技术无关。尽管其使用了生成模型和机器学习框架进行仿真，但方法本身并非针对多模态文档理解任务设计，也无法迁移解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点。
Abstract Decoded Neurofeedback (DecNef) is a flourishing non-invasive approach to brain modulation with wide-ranging applications in neuromedicine and cognitive neuroscience. However, progress in DecNef research remains constrained by subject-dependent learning variability, reliance on indirect measures to quantify progress, and the high cost and time demands of experimentation. We present DecNefLab, a modular and interpretable simulation framework that formalizes DecNef as a machine learning problem. Beyond providing a virtual laboratory, DecNefLab enables researchers to model, analyze and understand neurofeedback dynamics. Using latent variable generative models as simulated participants, DecNefLab allows direct observation of internal cognitive states and systematic evaluation of how different protocol designs and subject characteristics influence learning. We demonstrate how this approach can (i) reproduce empirical phenomena of DecNef learning, (ii) identify conditions under which DecNef feedback fails to induce learning, and (iii) guide the design of more robust and reliable DecNef protocols in silico before human implementation. In summary, DecNefLab bridges computational modeling and cognitive neuroscience, offering a principled foundation for methodological innovation, robust protocol design, and ultimately, a deeper understanding of DecNef-based brain modulation.
Authors Alexander Olza, Roberto Santana, David Soto
Categories Artificial Intelligence
0
From data to concepts via wiring diagrams
Title CN 通过配线图从数据到概念
Keywords 配线图 · Hasse图 · 序列数据分析 · 范畴论 · 聚类
Summary 本文提出了一种称为拟骨架配线图的有向图结构，用于表示时序过程等抽象概念，并证明其与Hasse图存在对应关系。作者设计了从序列数据中提取此类图的算法，并在游戏智能体行为分析中验证了其识别获胜策略的能力。实验对比了DBSCAN和层次聚类等传统聚类方法，展示了所提算法的鲁棒性。整体融合了范畴论、图论、聚类和强化学习技术。
Reason 该论文研究的是基于范畴论和图论的抽象概念建模，应用于自主智能体在游戏中的行为分析。其核心方法（如配线图、Hasse图）与文档图像理解（DIU）任务无直接关联，不涉及OCR、布局分析、表格识别或VLM多模态对齐等关键技术。属于无关领域的理论图方法，无法迁移解决DIU中的幻觉、细粒度定位或复杂排版等问题。
Abstract A wiring diagram is a labeled directed graph that represents an abstract concept such as a temporal process. In this article, we introduce the notion of a quasi-skeleton wiring diagram graph, and prove that quasi-skeleton wiring diagram graphs correspond to Hasse diagrams. Using this result, we designed algorithms that extract wiring diagrams from sequential data. We used our algorithms in analyzing the behavior of an autonomous agent playing a computer game, and the algorithms correctly identified the winning strategies. We compared the performance of our main algorithm with two other algorithms based on standard clustering techniques (DBSCAN and agglomerative hierarchical), including when some of the data was perturbed. Overall, this article brings together techniques in category theory, graph theory, clustering, reinforcement learning, and data engineering.
Authors Jason Lo, Mohammadnima Jafari
Comment 19 pages
Categories Artificial Intelligence
0
CORE - A Cell-Level Coarse-to-Fine Image Registration Engine for Multi-stain Image Alignment
Title CN CORE——一种用于多染色图像配准的细胞级由粗到精引擎
Keywords 图像配准 · 全切片图像 · 细胞核检测 · 多模态对齐 · 病理图像分析
Summary 本文提出了一种名为CORE的由粗到精的图像配准框架，用于实现多染色全切片图像（WSI）在细胞核层面的高精度对齐。该方法首先通过提示驱动的组织掩码提取进行粗配准，过滤伪影和非组织区域，再利用预训练特征提取器进行全局对齐；随后检测细胞核中心点，采用形状感知的点集配准模型进行细粒度刚性配准；最后通过相干点漂移（CPD）估计非线性位移场实现细胞级非刚性对齐。实验表明该方法在多个公开和私有数据集上优于现有技术。
Reason 该论文研究的是病理切片图像中的多模态图像配准问题，属于生物医学图像分析领域，核心任务为细胞级图像对齐，与文档图像理解（DIU）或DocVQA等任务无直接关联。尽管使用了AI和特征匹配技术，但其方法针对的是显微镜下的组织形态和细胞分布，无法迁移至文档布局分析、文本识别或视觉问答等场景。属于无关领域应用。
Abstract Accurate and efficient registration of whole slide images (WSIs) is essential for high-resolution, nuclei-level analysis in multi-stained tissue slides. We propose a novel coarse-to-fine framework CORE for accurate nuclei-level registration across diverse multimodal whole-slide image (WSI) datasets. The coarse registration stage leverages prompt-based tissue mask extraction to effectively filter out artefacts and non-tissue regions, followed by global alignment using tissue morphology and ac- celerated dense feature matching with a pre-trained feature extractor. From the coarsely aligned slides, nuclei centroids are detected and subjected to fine-grained rigid registration using a custom, shape-aware point-set registration model. Finally, non-rigid alignment at the cellular level is achieved by estimating a non-linear dis- placement field using Coherent Point Drift (CPD). Our approach benefits from automatically generated nuclei that enhance the accuracy of deformable registra- tion and ensure precise nuclei-level correspondence across modalities. The pro- posed model is evaluated on three publicly available WSI registration datasets, and two private datasets. We show that CORE outperforms current state-of-the-art methods in terms of generalisability, precision, and robustness in bright-field and immunofluorescence microscopy WSIs
Authors Esha Sadia Nasir, Behnaz Elhaminia, Mark Eastwood, Catherine King, Owen Cain, Lorraine Harper, Paul Moss, Dimitrios Chanouzas, David Snead, Nasir Rajpoot, Adam Shephard, Shan E Ahmed Raza
Categories Artificial Intelligence
0
"When Data is Scarce, Prompt Smarter"... Approaches to Grammatical Error Correction in Low-Resource Settings BHASHA Workshop, IJCNLP-AACL 2025
Title CN 当数据稀缺时，更智能地提示：低资源环境下的语法错误纠正方法
Keywords 语法错误纠正 · 低资源语言 · 提示工程 · 大语言模型 · 多语言NLP
Summary 本文研究在低资源印度语言中使用大语言模型（如GPT-4.1、Gemini-2.5和LLaMA-4）进行语法错误纠正的任务，通过设计零样本和少样本提示策略，在多个印度语言上取得了领先性能。结果显示，无需微调的大模型配合精心设计的提示即可超越专用微调模型。
Reason 该论文属于纯NLP任务（语法错误纠正）在特定语言（印度语系）上的应用，且聚焦于提示工程与少样本策略的下游适配。尽管使用了大模型，但未提出任何可用于文档图像理解（DIU）的新架构、表示学习或推理机制。其方法本质是‘平行应用’，不涉及视觉模态、OCR联合建模或文档结构理解，对解决OCR幻觉、复杂排版或细粒度定位等DIU核心问题无迁移价值。
Abstract Grammatical error correction (GEC) is an important task in Natural Language Processing that aims to automatically detect and correct grammatical mistakes in text. While recent advances in transformer-based models and large annotated datasets have greatly improved GEC performance for high-resource languages such as English, the progress has not extended equally. For most Indic languages, GEC remains a challenging task due to limited resources, linguistic diversity and complex morphology. In this work, we explore prompting-based approaches using state-of-the-art large language models (LLMs), such as GPT-4.1, Gemini-2.5 and LLaMA-4, combined with few-shot strategy to adapt them to low-resource settings. We observe that even basic prompting strategies, such as zero-shot and few-shot approaches, enable these LLMs to substantially outperform fine-tuned Indic-language models like Sarvam-22B, thereby illustrating the exceptional multilingual generalization capabilities of contemporary LLMs for GEC. Our experiments show that carefully designed prompts and lightweight adaptation significantly enhance correction quality across multiple Indic languages. We achieved leading results in the shared task--ranking 1st in Tamil (GLEU: 91.57) and Hindi (GLEU: 85.69), 2nd in Telugu (GLEU: 85.22), 4th in Bangla (GLEU: 92.86), and 5th in Malayalam (GLEU: 92.97). These findings highlight the effectiveness of prompt-driven NLP techniques and underscore the potential of large-scale LLMs to bridge resource gaps in multilingual GEC.
Authors Somsubhra De, Harsh Kumar, Arun Prakash A
Comment 10 pages, 5 figures, 5 tables; Accept-demonstration at BHASHA Workshop, IJCNLP-AACL 2025
Categories Artificial Intelligence, Computation and Language
0
LungEvaty: A Scalable, Open-Source Transformer-based Deep Learning Model for Lung Cancer Risk Prediction in LDCT Screening
Title CN LungEvaty：一种可扩展的、开源的基于Transformer的深度学习模型，用于低剂量CT筛查中的肺癌风险预测
Keywords Transformer · 肺癌风险预测 · 低剂量CT · 无监督注意力 · 医学影像分析
Summary 本文提出LungEvaty，一个完全基于Transformer的框架，用于从单次低剂量CT扫描中预测1-6年的肺癌风险。该模型在全肺输入上运行，仅使用图像数据而无需区域级标注，在超过9万张CT扫描上训练，性能达到SOTA。通过可选的解剖学感知注意力引导（AIAG）损失进一步优化注意力机制。框架开源且具有良好的数据效率和可扩展性。
Reason 该论文属于医学影像分析领域，研究任务为基于CT扫描的肺癌风险预测，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了大规模视觉数据和Transformer架构，但其方法针对的是3D医学体积数据建模，并依赖解剖学注意力引导，无法迁移到文档理解中的核心痛点如OCR幻觉、复杂排版解析或细粒度文本定位。属于无关领域应用。
Abstract Lung cancer risk estimation is gaining increasing importance as more countries introduce population-wide screening programs using low-dose CT (LDCT). As imaging volumes grow, scalable methods that can process entire lung volumes efficiently are essential to tap into the full potential of these large screening datasets. Existing approaches either over-rely on pixel-level annotations, limiting scalability, or analyze the lung in fragments, weakening performance. We present LungEvaty, a fully transformer-based framework for predicting 1-6 year lung cancer risk from a single LDCT scan. The model operates on whole-lung inputs, learning directly from large-scale screening data to capture comprehensive anatomical and pathological cues relevant for malignancy risk. Using only imaging data and no region supervision, LungEvaty matches state-of-the-art performance, refinable by an optional Anatomically Informed Attention Guidance (AIAG) loss that encourages anatomically focused attention. In total, LungEvaty was trained on more than 90,000 CT scans, including over 28,000 for fine-tuning and 6,000 for evaluation. The framework offers a simple, data-efficient, and fully open-source solution that provides an extensible foundation for future research in longitudinal and multimodal lung cancer risk prediction.
Authors Johannes Brandt, Maulik Chevli, Rickmer Braren, Georgios Kaissis, Philip Müller, Daniel Rueckert
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Leveraging Unlabeled Data from Unknown Sources via Dual-Path Guidance for Deepfake Face Detection
Title CN 通过双路径引导利用未知来源的无标签数据进行深度伪造人脸检测
Keywords deepfake detection · unsupervised learning · cross-domain alignment · pseudo-labeling · text-guided
Summary 本文提出了一种双路径引导网络（DPGNet），用于在无标签且来源未知的情况下提升深度伪造人脸检测性能。方法包括文本引导的跨域对齐和课程驱动的伪标签生成，以应对不同生成模型带来的域差异和缺乏标注的问题。实验表明该方法在多个主流数据集上优于现有技术。
Reason 该论文属于深度伪造检测领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。尽管其方法涉及无监督学习和跨域对齐，但应用场景聚焦于人脸伪造识别，无法直接迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心问题。属于负面清单中的‘无关领域’。
Abstract Existing deepfake detection methods heavily rely on static labeled datasets. However, with the proliferation of generative models, real-world scenarios are flooded with massive amounts of unlabeled fake face data from unknown sources. This presents a critical dilemma: detectors relying solely on existing data face generalization failure, while manual labeling for this new stream is infeasible due to the high realism of fakes. A more fundamental challenge is that, unlike typical unsupervised learning tasks where categories are clearly defined, real and fake faces share the same semantics, which leads to a decline in the performance of traditional unsupervised strategies. Therefore, there is an urgent need for a new paradigm designed specifically for this scenario to effectively utilize these unlabeled data. Accordingly, this paper proposes a dual-path guided network (DPGNet) to address two key challenges: (1) bridging the domain differences between faces generated by different generative models; and (2) utilizing unlabeled image samples. The method comprises two core modules: text-guided cross-domain alignment, which uses learnable cues to unify visual and textual embeddings into a domain-invariant feature space; and curriculum-driven pseudo-label generation, which dynamically utilizes unlabeled samples. Extensive experiments on multiple mainstream datasets show that DPGNet significantly outperforms existing techniques,, highlighting its effectiveness in addressing the challenges posed by the deepfakes using unlabeled data.
Authors Zhiqiang Yang, Renshuai Tao, Chunjie Zhang, guodong yang, Xiaolong Zheng, Yao Zhao
Comment 11pages,4figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMs
Title CN 细节中的魔鬼：开源权重LLM中的 emergent misalignment、格式与一致性
Keywords emergent misalignment · LLM安全 · 格式漏洞 · 开源大模型 · 代码生成
Summary 本文复现了emergent misalignment现象，即在不安全代码数据上微调会导致模型整体对齐性下降。研究覆盖九个现代开源大模型（Gemma3和Qwen3系列，1B-32B参数），发现微调后平均misalignment率为0.68%，远低于GPT-4o的20%。关键发现是要求JSON输出会使misalignment率翻倍，表明结构化约束可能绕过安全训练。
Reason 该论文研究的是大语言模型在不安全代码生成任务下的emergent misalignment现象，属于纯NLP领域的安全对齐（Safety）问题，且未涉及任何视觉模态或文档图像理解任务。尽管提到了JSON格式输出引发的对齐退化，但其核心关注点是模型行为安全而非文档结构理解或视觉-语言对齐。无法迁移到解决OCR幻觉、复杂排版或细粒度定位等DIU核心痛点。
Abstract Prior work has shown that fine-tuning models on a narrow domain with misaligned data can lead to broad misalignment - a phenomenon termed "emergent misalignment" (Betley et al. 2025). While all tested models were susceptible to emergent misalignment, some models showed more resistance than others. Specifically the Qwen-2.5 family proved to be relatively resistant, while GPT-4o exhibited the strongest misalignment. In this paper we evaluate if current-generation open-weights models exhibit similar resistance to the Qwen-2.5 family and measure misalignment robustness over a range of model architectures and scales. We replicate the effect across nine modern open-weights models (Gemma 3 and Qwen 3 families, 1B-32B parameters). Models fine-tuned on insecure code generation show a 0.68% misalignment rate (compared to 0.07% for base models), matching the lower end of prior open-model results but dramatically lower than GPT-4o's 20%. We identify a critical format-dependent vulnerability: requiring JSON output doubles misalignment rates compared to natural language prompts (0.96% vs 0.42%). This suggests that structural constraints may bypass safety training by reducing the model's 'degrees of freedom' to refuse. These findings confirm emergent misalignment as a reproducible phenomenon in modern open-weights models, with rates substantially lower than observed in proprietary systems.
Authors Craig Dickson
Categories Artificial Intelligence, Computation and Language
0
The Making of Digital Ghosts: Designing Ethical AI Afterlives
Title CN 数字幽灵的制造：在人工智能死后世界中设计伦理
Keywords AI伦理 · 数字永生 · 死后隐私 · 人工智能社会影响 · 数字幽灵
Summary 本文探讨了通过人工智能技术基于逝者的数字痕迹构建‘数字幽灵’所带来的伦理挑战，包括哀悼过程中的真实性、知情同意、死后隐私和商业化等问题，并提出了一个九维分类法和伦理可接受的设计原则。研究聚焦于技术的社会影响而非技术方法本身。
Reason 该论文讨论的是利用AI技术创建逝者数字分身（如聊天机器人、语音克隆）所引发的伦理问题，属于AI伦理与社会影响范畴，与文档图像理解（DIU）、DocVQA、OCR、多模态文档处理等核心任务完全无关。未涉及任何视觉表示、文本识别、布局分析或VLM架构改进，无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU痛点。
Abstract Advances in artificial intelligence now make it possible to simulate the dead through chatbots, voice clones, and video avatars trained on a person's digital traces. These "digital ghosts" are moving from fiction to commercial reality, reshaping how people mourn and remember. This paper offers a conceptual and ethical analysis of AI-mediated digital afterlives. We define what counts as a digital ghost, trace their rise across personal, commercial, and institutional contexts, and identify core ethical tensions around grief and well-being, truthfulness and deception, consent and posthumous privacy, dignity and misrepresentation, and the commercialization of mourning. To analyze these challenges, we propose a nine-dimensional taxonomy of digital afterlife technologies and, building on it, outline the features of an ethically acceptable digital ghost: premortem intent, mutual consent, transparent and limited data use, clear disclosure, restricted purposes and access, family or estate stewardship, and minimal behavioral agency. We argue for targeted regulation and professional guidelines to ensure that digital ghosts can aid remembrance without slipping into forms of deception.
Authors Giovanni Spitale, Federico Germani
Categories Artificial Intelligence
0
LaajMeter: A Framework for LaaJ Evaluation
Title CN LaajMeter：一种用于评估“大语言模型作为评判者”的框架
Keywords LLM-as-a-Judge · 元评估 · 合成数据 · 评估指标 · 可信NLP
Summary 本文提出LaajMeter，一个基于模拟的框架，用于在低资源领域对“大语言模型作为评判者”（LaaJ）进行元评估。通过生成代表虚拟模型和评判者的合成数据，系统可测试不同评估指标在区分高质量与低质量评判者时的有效性，并帮助设定合理的性能阈值。实验在一个涉及老旧编程语言的代码翻译任务中展示，揭示了常用指标的局限性。
Reason 该论文属于纯NLP领域的评估方法研究，聚焦于如何用大语言模型作为评判者（LaaJ）并评估其质量，未涉及任何视觉模态或文档图像理解任务。尽管其元评估思想在理论上可能对VLM的评估有启发，但缺乏与视觉输入、OCR、布局分析或文档推理的结合点，无法直接迁移解决DIU中的核心痛点如幻觉、细粒度定位或复杂排版理解。属于‘平行应用’类方法在评估层面的延伸，不构成对文档理解的技术推动。
Abstract Large Language Models (LLMs) are increasingly used as evaluators in natural language processing tasks, a paradigm known as LLM-as-a-Judge (LaaJ). The analysis of a LaaJ software, commonly refereed to as meta-evaluation, pose significant challenges in domain-specific contexts. In such domains, in contrast to general domains, annotated data is scarce and expert evaluation is costly. As a result, meta-evaluation is often performed using metrics that have not been validated for the specific domain in which they are applied. Therefore, it becomes difficult to determine which metrics effectively identify LaaJ quality, and further, what threshold indicates sufficient evaluator performance. In this work, we introduce LaaJMeter, a simulation-based framework for controlled meta-evaluation of LaaJs. LaaJMeter enables engineers to generate synthetic data representing virtual models and judges, allowing systematic analysis of evaluation metrics under realistic conditions. This helps practitioners validate LaaJs for specific tasks: they can test whether their metrics correctly distinguish between high and low quality (virtual) LaaJs, and estimate appropriate thresholds for evaluator adequacy. We demonstrate the utility of LaaJMeter in a code translation task involving a legacy programming language, showing how different metrics vary in sensitivity to evaluator quality. Our results highlight the limitations of common metrics and the importance of principled metric selection. LaaJMeter provides a scalable and extensible solution for assessing LaaJs in low-resource settings, contributing to the broader effort to ensure trustworthy and reproducible evaluation in NLP.
Authors Samuel Ackerman, Gal Amram, Ora Nova Fandina, Eitan Farchi, Shmulik Froimovich, Raviv Gal, Wesam Ibraheem, Avi Ziv
Categories Artificial Intelligence, Computation and Language
0
MeshSplat: Generalizable Sparse-View Surface Reconstruction via Gaussian Splatting AAAI 2026
Title CN MeshSplat：通过高斯点阵化的通用稀疏视角表面重建
Keywords 3D重建 · 高斯点阵化 · 稀疏视角 · 无监督学习 · 计算机视觉
Summary 本文提出MeshSplat，一种基于高斯点阵化（Gaussian Splatting）的通用稀疏视角表面重建框架。通过引入2DGS作为桥梁，连接新视角合成与学习到的几何先验，并利用前馈网络预测每视图的像素对齐2DGS，从而实现无需直接3D监督的表面重建。同时设计加权Chamfer距离损失和法线预测网络以提升几何精度。实验表明该方法在稀疏输入下优于现有方法。
Reason 该论文研究的是三维视觉中的稀疏视角表面重建问题，属于3D视觉与图形学领域，尽管使用了Gaussian Splatting等前沿技术，但其任务目标（如新视角合成、mesh重建）与文档图像理解（DIU）、DocVQA、OCR、布局分析等核心任务无直接关联。无法迁移解决文档理解中的OCR幻觉、复杂排版、长文档推理等痛点，属于无关领域。
Abstract Surface reconstruction has been widely studied in computer vision and graphics. However, existing surface reconstruction works struggle to recover accurate scene geometry when the input views are extremely sparse. To address this issue, we propose MeshSplat, a generalizable sparse-view surface reconstruction framework via Gaussian Splatting. Our key idea is to leverage 2DGS as a bridge, which connects novel view synthesis to learned geometric priors and then transfers these priors to achieve surface reconstruction. Specifically, we incorporate a feed-forward network to predict per-view pixel-aligned 2DGS, which enables the network to synthesize novel view images and thus eliminates the need for direct 3D ground-truth supervision. To improve the accuracy of 2DGS position and orientation prediction, we propose a Weighted Chamfer Distance Loss to regularize the depth maps, especially in overlapping areas of input views, and also a normal prediction network to align the orientation of 2DGS with normal vectors predicted by a monocular normal estimator. Extensive experiments validate the effectiveness of our proposed improvement, demonstrating that our method achieves state-of-the-art performance in generalizable sparse-view mesh reconstruction tasks. Project Page: https://hanzhichang.github.io/meshsplat_web
Authors Hanzhi Chang, Ruijie Zhu, Wenjie Chang, Mulin Yu, Yanzhe Liang, Jiahao Lu, Zhuoyuan Li, Tianzhu Zhang
Comment Accepted by AAAI 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
ASP-Assisted Symbolic Regression: Uncovering Hidden Physics in Fluid Mechanics
Title CN 基于ASP的符号回归：揭示流体力学中的隐含物理
Keywords 符号回归 · Answer Set Programming · 流体力学 · 可解释性 · 物理一致性
Summary 本文提出一种将符号回归（SR）与Answer Set Programming（ASP）相结合的混合框架，用于从数值模拟数据中发现流体力学中三维层流的简洁数学表达式。该方法不仅拟合数据，还通过ASP引入物理约束，确保生成的公式具有物理合理性。实验表明其能准确复现抛物线速度分布和线性压降，并与解析解高度一致。
Reason 该论文研究的是符号回归与逻辑编程在流体力学建模中的结合，属于科学机器学习在特定物理领域的应用，不涉及文档图像理解（DIU）、多模态大模型（VLM）、OCR、布局分析或视觉语言推理等核心任务。尽管其方法论融合了数据驱动与知识约束，但完全偏离文档理解的技术主线，属于无关领域应用。
Abstract Symbolic Regression (SR) offers an interpretable alternative to conventional Machine-Learning (ML) approaches, which are often criticized as ``black boxes''. In contrast to standard regression models that require a prescribed functional form, SR constructs expressions from a user-defined set of mathematical primitives, enabling the automated discovery of compact formulas that fit the data and reveal underlying physical relationships. In fluid mechanics, where understanding the underlying physics is as crucial as predictive accuracy, this study applies SR to model three-dimensional (3D) laminar flow in a rectangular channel, focusing on the axial velocity and pressure fields. Compact symbolic equations were derived from numerical simulation data, accurately reproducing the expected parabolic velocity profile and linear pressure drop, and showing excellent agreement with analytical solutions from the literature. To address the limitation that purely data-driven SR models may overlook domain-specific constraints, an innovative hybrid framework that integrates SR with Answer Set Programming (ASP) is also introduced. This integration combines the generative power of SR with the declarative reasoning capabilities of ASP, ensuring that derived equations remain both statistically accurate and physically plausible. The proposed SR/ASP methodology demonstrates the potential of combining data-driven and knowledge-representation approaches to enhance interpretability, reliability, and alignment with physical principles in fluid dynamics and related domains.
Authors Theofanis Aravanis, Grigorios Chrimatopoulos, Mohammad Ferdows, Michalis Xenos, Efstratios Em Tzirtzilakis
Comment This research was implemented in the framework of the Action "Flagship actions in interdisciplinary scientific fields with a special focus on the productive fabric'', which is implemented through the National Recovery and Resilience Fund Greece 2.0 and funded by the European Union--NextGenerationEU (Project ID: TAEDR-0535983)
Categories Artificial Intelligence
0
Access Controls Will Solve the Dual-Use Dilemma ICML 2025 Workshop on Technical AI Governance (TAIG)
Title CN 访问控制将解决双重用途困境
Keywords AI安全 · 双重用途 · 访问控制 · AI治理 · 模型滥用
Summary 论文指出当前AI安全系统在应对双重用途请求时面临困境：由于缺乏对请求者真实背景的理解，系统可能错误地拒绝合法请求或放行有害请求。为此，作者提出一种基于访问控制的概念框架，只有经过验证的用户才能获取潜在危险性输出。该框架旨在减少过度拒绝和不足拒绝，提升安全与效用的平衡。尽管仅为高层级提案，但为模型提供者提供了更精细的内容管理工具，并为监管者提供可实施的政策选项。
Reason 该论文讨论的是AI安全中的双重用途治理问题，属于AI伦理与政策范畴，提出通过访问控制机制来管理模型输出的访问权限。其核心内容与文档图像理解（DIU）或视觉语言模型在文档处理中的技术挑战（如OCR幻觉、复杂排版、细粒度定位等）完全无关，无法迁移用于提升DIU性能。
Abstract AI safety systems face the dual-use dilemma. It is unclear whether to answer dual-use requests, since the same query could be either harmless or harmful depending on who made it and why. To make better decisions, such systems would need to examine requests' real-world context, but currently, they lack access to this information. Instead, they sometimes end up making arbitrary choices that result in refusing legitimate queries and allowing harmful ones, which hurts both utility and safety. To address this, we propose a conceptual framework based on access controls where only verified users can access dual-use outputs. We describe the framework's components, analyse its feasibility, and explain how it addresses both over-refusals and under-refusals. While only a high-level proposal, our work takes the first step toward giving model providers more granular tools for managing dual-use content. Such tools would enable users to access more capabilities without sacrificing safety, and offer regulators new options for targeted policies.
Authors Evžen Wybitul
Comment Accepted at ICML 2025 Workshop on Technical AI Governance (TAIG)
Categories Artificial Intelligence
0
MFM-point: Multi-scale Flow Matching for Point Cloud Generation
Title CN MFM-point：用于点云生成的多尺度流匹配
Keywords 点云生成 · 流匹配 · 多尺度 · 3D生成 · 几何保持
Summary 本文提出MFM-Point，一种基于多尺度流匹配的点云生成框架，采用由粗到精的生成方式，在不增加训练或推理开销的情况下提升了点基方法的性能和可扩展性。通过结构化的下采样和上采样策略，保持了不同分辨率间的几何一致性和对齐。实验表明该方法在多类别和高分辨率点云生成任务中表现优异。
Reason 该论文研究的是3D点云生成，属于3D视觉与生成模型领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其核心技术——多尺度流匹配——针对的是无序点云的几何结构保持问题，难以迁移到文档图像中的OCR幻觉、复杂排版或长文档推理等核心痛点。属于无关领域。
Abstract In recent years, point cloud generation has gained significant attention in 3D generative modeling. Among existing approaches, point-based methods directly generate point clouds without relying on other representations such as latent features, meshes, or voxels. These methods offer low training cost and algorithmic simplicity, but often underperform compared to representation-based approaches. In this paper, we propose MFM-Point, a multi-scale Flow Matching framework for point cloud generation that substantially improves the scalability and performance of point-based methods while preserving their simplicity and efficiency. Our multi-scale generation algorithm adopts a coarse-to-fine generation paradigm, enhancing generation quality and scalability without incurring additional training or inference overhead. A key challenge in developing such a multi-scale framework lies in preserving the geometric structure of unordered point clouds while ensuring smooth and consistent distributional transitions across resolutions. To address this, we introduce a structured downsampling and upsampling strategy that preserves geometry and maintains alignment between coarse and fine resolutions. Our experimental results demonstrate that MFM-Point achieves best-in-class performance among point-based methods and challenges the best representation-based methods. In particular, MFM-point demonstrates strong results in multi-category and high-resolution generation tasks.
Authors Petr Molodyk, Jaemoo Choi, David W. Romero, Ming-Yu Liu, Yongxin Chen
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Non-equilibrium Annealed Adjoint Sampler
Title CN 非平衡退火伴随采样器
Keywords 扩散模型 · 随机最优控制 · 采样算法 · SDE · 生成模型
Summary 本文提出了一种名为非平衡退火伴随采样器（NAAS）的新框架，通过引入退火参考动力学作为非平稳基底SDE，改进基于随机最优控制（SOC）的扩散采样过程。该方法提升了学习控制策略的稳定性和采样效率，并可通过多种SOC求解器灵活实现算法设计。实验验证了其在经典能量场和分子玻尔兹曼分布采样中的有效性。
Reason 该论文研究基于扩散模型的采样方法，属于生成模型中的理论采样框架，聚焦于能量景观和分子分布等科学计算任务。其核心是随机最优控制与SDE采样效率优化，未涉及多模态输入、视觉内容理解或文档结构建模。与文档图像理解（DIU）中的OCR、布局分析、表格识别或DocVQA等任务无直接关联，也无法迁移用于解决OCR幻觉、细粒度定位或复杂排版理解等关键痛点。
Abstract Recently, there has been significant progress in learning-based diffusion samplers, which aim to sample from a given unnormalized density. Many of these approaches formulate the sampling task as a stochastic optimal control (SOC) problem using a canonical uninformative reference process, which limits their ability to efficiently guide trajectories toward the target distribution. In this work, we propose the Non-Equilibrium Annealed Adjoint Sampler (NAAS), a novel SOC-based diffusion framework that employs annealed reference dynamics as a non-stationary base SDE. This annealing structure provides a natural progression toward the target distribution and generates informative reference trajectories, thereby enhancing the stability and efficiency of learning the control. Owing to our SOC formulation, our framework can incorporate a variety of SOC solvers, thereby offering high flexibility in algorithmic design. As one instantiation, we employ a lean adjoint system inspired by adjoint matching, enabling efficient and scalable training. We demonstrate the effectiveness of NAAS across a range of tasks, including sampling from classical energy landscapes and molecular Boltzmann distributions.
Authors Jaemoo Choi, Yongxin Chen, Molei Tao, Guan-Horng Liu
Comment 26 pages, 8 figures
Categories Artificial Intelligence
0
WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving
Title CN WaymoQA：一种用于自动驾驶中安全关键推理的多视角视觉问答数据集
Keywords 多模态大模型 · 安全关键推理 · 自动驾驶 · 多视角理解 · 视觉问答
Summary 本文提出了一个名为WaymoQA的新数据集，包含3.5万个标注样本，用于研究自动驾驶中基于多视角输入的安全关键推理任务。该任务分为两个阶段：首先应对即时交通风险，然后缓解决策引发的次生风险。实验表明现有MLLM在该类高风险场景下表现不佳，而使用WaymoQA微调可显著提升其推理能力。
Reason 该论文聚焦于自动驾驶场景下的安全关键推理，属于具身智能与视频理解领域，与文档图像理解（DIU）任务无直接关联。尽管使用了多模态大模型（MLLM）进行高阶推理，但其问题设定、数据模态（多视角图像/视频）、应用场景均远离文档理解核心痛点（如OCR幻觉、密集文本识别、复杂版式分析等），无法迁移有效方法至DIU领域。
Abstract Recent advancements in multimodal large language models (MLLMs) have shown strong understanding of driving scenes, drawing interest in their application to autonomous driving. However, high-level reasoning in safety-critical scenarios, where avoiding one traffic risk can create another, remains a major challenge. Such reasoning is often infeasible with only a single front view and requires a comprehensive view of the environment, which we achieve through multi-view inputs. We define Safety-Critical Reasoning as a new task that leverages multi-view inputs to address this challenge. Then, we distill Safety-Critical Reasoning into two stages: first resolve the immediate risk, then mitigate the decision-induced downstream risks. To support this, we introduce WaymoQA, a dataset of 35,000 human-annotated question-answer pairs covering complex, high-risk driving scenarios. The dataset includes multiple-choice and open-ended formats across both image and video modalities. Experiments reveal that existing MLLMs underperform in safety-critical scenarios compared to normal scenes, but fine-tuning with WaymoQA significantly improves their reasoning ability, highlighting the effectiveness of our dataset in developing safer and more reasoning-capable driving agents.
Authors Seungjun Yu, Seonho Lee, Namho Kim, Jaeyo Shin, Junsung Park, Wonjeong Ryu, Raehyuk Jung, Hyunjung Shim
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Energy Costs and Neural Complexity Evolution in Changing Environments ALIFE 2025
Title CN 变化环境中的能量成本与神经复杂性演化
Keywords 神经演化 · 能量约束 · 强化学习 · 人工神经网络 · 认知缓冲假说
Summary 本文通过强化学习代理演化人工神经网络，研究环境季节性和能量限制对神经复杂性（规模与结构）的影响。结果表明，在高季节性环境下，能量摄入减少会限制网络规模，支持‘昂贵大脑假说’（EBH），而结构复杂性主要是规模的副产品，能量压力促使更高效的网络结构演化。
Reason 该论文研究的是在强化学习代理中，环境变化和能量成本如何影响人工神经网络的演化，属于生物启发式AI和进化算法领域，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。其方法无法迁移解决OCR幻觉、密集文本识别、复杂排版或长文档推理等DIU核心问题。
Abstract The Cognitive Buffer Hypothesis (CBH) posits that larger brains evolved to enhance survival in changing conditions. However, larger brains also carry higher energy demands, imposing additional metabolic burdens. Alongside brain size, brain organization plays a key role in cognitive ability and, with suitable architectures, may help mitigate energy challenges. This study evolves Artificial Neural Networks (ANNs) used by Reinforcement Learning (RL) agents to investigate how environmental variability and energy costs influence the evolution of neural complexity, defined in terms of ANN size and structure. Results indicate that under energy constraints, increasing seasonality led to smaller ANNs. This challenges CBH and supports the Expensive Brain Hypothesis (EBH), as highly seasonal environments reduced net energy intake and thereby constrained brain size. ANN structural complexity primarily emerged as a byproduct of size, where energy costs promoted the evolution of more efficient networks. These results highlight the role of energy constraints in shaping neural complexity, offering in silico support for biological theory and energy-efficient robotic design.
Authors Sian Heesom-Green, Jonathan Shock, Geoff Nitschke
Comment Presented at ALIFE 2025, proceedings forthcoming (MIT Press)
Categories Artificial Intelligence
0
FAPE-IR: Frequency-Aware Planning and Execution Framework for All-in-One Image Restoration
Title CN FAPE-IR：面向一体化图像恢复的频率感知规划与执行框架
Keywords 图像恢复 · 多模态大模型 · 频率感知 · LoRA-MoE · 扩散模型
Summary 本文提出FAPE-IR，一种用于一体化图像恢复的频率感知规划与执行框架。该框架利用冻结的多模态大语言模型作为规划器，生成针对退化图像的频率感知恢复计划，并指导基于LoRA的专家混合模块在扩散模型中动态选择高频或低频专家进行恢复。结合对抗训练和频率正则化损失，该方法在七种图像恢复任务上实现了先进性能，并具备良好的零样本泛化能力。
Reason 该论文研究的是通用图像恢复任务（如去噪、超分辨率等），属于图像处理领域，与文档图像理解（DIU）的核心任务（如DocVQA、版面分析、表格识别、文本提取与推理）无直接关联。尽管使用了多模态大模型进行‘规划’，但其目标是修复图像质量，而非提升文档语义理解或结构解析能力。方法难以迁移到解决OCR幻觉、复杂排版建模或长文档推理等DIU痛点，属于无关领域工作。
Abstract All-in-One Image Restoration (AIO-IR) aims to develop a unified model that can handle multiple degradations under complex conditions. However, existing methods often rely on task-specific designs or latent routing strategies, making it hard to adapt to real-world scenarios with various degradations. We propose FAPE-IR, a Frequency-Aware Planning and Execution framework for image restoration. It uses a frozen Multimodal Large Language Model (MLLM) as a planner to analyze degraded images and generate concise, frequency-aware restoration plans. These plans guide a LoRA-based Mixture-of-Experts (LoRA-MoE) module within a diffusion-based executor, which dynamically selects high- or low-frequency experts, complemented by frequency features of the input image. To further improve restoration quality and reduce artifacts, we introduce adversarial training and a frequency regularization loss. By coupling semantic planning with frequency-based restoration, FAPE-IR offers a unified and interpretable solution for all-in-one image restoration. Extensive experiments show that FAPE-IR achieves state-of-the-art performance across seven restoration tasks and exhibits strong zero-shot generalization under mixed degradations.
Authors Jingren Liu, Shuning Xu, Qirui Yang, Yun Wang, Xiangyu Chen, Zhong Ji
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Multi-Context Fusion Transformer for Pedestrian Crossing Intention Prediction in Urban Environments
Title CN 用于城市环境中行人过街意图预测的多上下文融合Transformer
Keywords 行人意图预测 · 多上下文融合 · Transformer · 自动驾驶 · 行为预测
Summary 本文提出了一种多上下文融合Transformer（MFT），通过融合行人行为、环境、定位和车辆运动四个维度的上下文信息来预测城市环境中行人的过街意图。采用渐进式融合策略，结合自注意力与交叉注意力机制，提升了预测准确性，并在多个数据集上验证了有效性。
Reason 该论文研究的是自动驾驶场景下的行人意图预测，属于具身智能与机器人领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无关。其方法虽涉及多模态上下文融合，但面向的是行为预测而非文本识别、布局分析或文档问答等任务，无法迁移解决OCR幻觉、密集文本识别、复杂排版解析或长文档推理等DIU核心痛点。
Abstract Pedestrian crossing intention prediction is essential for autonomous vehicles to improve pedestrian safety and reduce traffic accidents. However, accurate pedestrian intention prediction in urban environments remains challenging due to the multitude of factors affecting pedestrian behavior. In this paper, we propose a multi-context fusion Transformer (MFT) that leverages diverse numerical contextual attributes across four key dimensions, encompassing pedestrian behavior context, environmental context, pedestrian localization context and vehicle motion context, to enable accurate pedestrian intention prediction. MFT employs a progressive fusion strategy, where mutual intra-context attention enables reciprocal interactions within each context, thereby facilitating feature sequence fusion and yielding a context token as a context-specific representation. This is followed by mutual cross-context attention, which integrates features across contexts with a global CLS token serving as a compact multi-context representation. Finally, guided intra-context attention refines context tokens within each context through directed interactions, while guided cross-context attention strengthens the global CLS token to promote multi-context fusion via guided information propagation, yielding deeper and more efficient integration. Experimental results validate the superiority of MFT over state-of-the-art methods, achieving accuracy rates of 73%, 93%, and 90% on the JAADbeh, JAADall, and PIE datasets, respectively. Extensive ablation studies are further conducted to investigate the effectiveness of the network architecture and contribution of different input context. Our code is open-source: https://github.com/ZhongHang0307/Multi-Context-Fusion-Transformer.
Authors Yuanzhe Li, Hang Zhong, Steffen Müller
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Comprehensive Design Space Exploration for Tensorized Neural Network Hardware Accelerators
Title CN 面向张量化神经网络硬件加速器的综合设计空间探索
Keywords 张量化神经网络 · 硬件加速 · 设计空间探索 · FPGA · 边缘计算
Summary 本文提出了一种联合优化张量化神经网络的收缩路径、硬件架构和数据流映射的协同探索框架，通过在统一设计空间中进行延迟驱动的全局搜索，实现边缘设备上训练与推理的高效性。实验基于可配置FPGA内核，显著降低了延迟。
Reason 该论文聚焦于张量化神经网络在硬件加速器（特别是FPGA）上的部署优化，核心是算法-硬件协同设计以降低延迟和能耗。其研究目标与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联，既不涉及文档结构、OCR、布局分析等任务，也未提出可迁移至DIU领域的通用方法（如视觉表示、推理机制、幻觉抑制等）。属于典型的边缘计算与硬件优化工作，远离文档理解的方法论创新。
Abstract High-order tensor decomposition has been widely adopted to obtain compact deep neural networks for edge deployment. However, existing studies focus primarily on its algorithmic advantages such as accuracy and compression ratio-while overlooking the hardware deployment efficiency. Such hardware-unaware designs often obscure the potential latency and energy benefits of tensorized models. Although several works attempt to reduce computational cost by optimizing the contraction sequence based on the number of multiply-accumulate operations, they typically neglect the underlying hardware characteristics, resulting in suboptimal real-world performance. We observe that the contraction path, hardware architecture, and dataflow mapping are tightly coupled and must be optimized jointly within a unified design space to maximize deployment efficiency on real devices. To this end, we propose a co-exploration framework that unifies these dimensions within a unified design space for efficient training and inference of tensorized neural networks on edge platforms. The framework formulates a latency oriented search objective and solves it via a global latency-driven exploration across the unified design space to achieve end-to-end model efficiency. The optimized configurations are implemented on a configurable FPGA kernel, achieving up to 4x and 3.85x lower inference and training latency compared with the dense baseline.
Authors Jinsong Zhang, Minghe Li, Jiayi Tian, Jinming Lu, Zheng Zhang
Categories Artificial Intelligence
0
PaSE: Prototype-aligned Calibration and Shapley-based Equilibrium for Multimodal Sentiment Analysis AAAI 2026
Title CN PaSE：面向多模态情感分析的原型对齐校准与基于Shapley值的均衡方法
Keywords 多模态情感分析 · 模态竞争 · Shapley值 · 原型对齐 · 梯度调制
Summary 本文提出PaSE框架用于多模态情感分析，通过原型引导的校准学习（PCL）和基于Shapley值的梯度调制（SGM）来缓解多模态融合中的模态竞争问题，提升各模态协作能力。实验表明该方法在IEMOCAP、MOSI和MOSEI数据集上表现优异。
Reason 该论文研究的是多模态情感分析（MSA），属于音频-文本-视觉三模态在情感识别中的融合问题，核心挑战是模态竞争与对齐，与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂版式、细粒度文本定位等无直接关联。尽管其提出的Shapley值梯度调制和原型校准机制在理论上具有跨模态优化的启发意义，但该方法针对的是情感语义空间的模态平衡，难以迁移至文档中高分辨率视觉结构与文本语义的精确对齐任务。属于无关领域应用。
Abstract Multimodal Sentiment Analysis (MSA) seeks to understand human emotions by integrating textual, acoustic, and visual signals. Although multimodal fusion is designed to leverage cross-modal complementarity, real-world scenarios often exhibit modality competition: dominant modalities tend to overshadow weaker ones, leading to suboptimal performance. In this paper, we propose PaSE, a novel Prototype-aligned Calibration and Shapley-optimized Equilibrium framework, which enhances collaboration while explicitly mitigating modality competition. PaSE first applies Prototype-guided Calibration Learning (PCL) to refine unimodal representations and align them through an Entropic Optimal Transport mechanism that ensures semantic consistency. To further stabilize optimization, we introduce a Dual-Phase Optimization strategy. A prototype-gated fusion module is first used to extract shared representations, followed by Shapley-based Gradient Modulation (SGM), which adaptively adjusts gradients according to the contribution of each modality. Extensive experiments on IEMOCAP, MOSI, and MOSEI confirm that PaSE achieves the superior performance and effectively alleviates modality competition.
Authors Kang He, Boyu Chen, Yuzhe Ding, Fei Li, Chong Teng, Donghong Ji
Comment Accepted by AAAI 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Pedestrian Crossing Intention Prediction Using Multimodal Fusion Network
Title CN 使用多模态融合网络进行行人过街意图预测
Keywords 多模态融合 · 行人意图预测 · 自动驾驶 · 注意力机制 · Transformer
Summary 本文提出了一种多模态融合网络，用于预测城市环境中行人的过街意图。该网络结合视觉和运动分支的七种模态特征，利用基于Transformer的提取模块、深度引导注意力机制以及模态和时间注意力结构，以捕捉跨模态和时序上的关键信息。在JAAD数据集上的实验表明该方法优于基线。
Reason 该论文研究的是自动驾驶场景下的行人行为预测，属于具身智能与自动驾驶领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了多模态融合和注意力机制，但其方法针对的是视频序列中的动作预测问题，难以迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Pedestrian crossing intention prediction is essential for the deployment of autonomous vehicles (AVs) in urban environments. Ideal prediction provides AVs with critical environmental cues, thereby reducing the risk of pedestrian-related collisions. However, the prediction task is challenging due to the diverse nature of pedestrian behavior and its dependence on multiple contextual factors. This paper proposes a multimodal fusion network that leverages seven modality features from both visual and motion branches, aiming to effectively extract and integrate complementary cues across different modalities. Specifically, motion and visual features are extracted from the raw inputs using multiple Transformer-based extraction modules. Depth-guided attention module leverages depth information to guide attention towards salient regions in another modality through comprehensive spatial feature interactions. To account for the varying importance of different modalities and frames, modality attention and temporal attention are designed to selectively emphasize informative modalities and effectively capture temporal dependencies. Extensive experiments on the JAAD dataset validate the effectiveness of the proposed network, achieving superior performance compared to the baseline methods.
Authors Yuanzhe Li, Steffen Müller
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
BERT-APC: A Reference-free Framework for Automatic Pitch Correction via Musical Context Inference
Title CN BERT-APC：一种通过音乐上下文推断实现无参考自动音高校正的框架
Keywords 自动音高校正 · 音乐语言模型 · 音频处理 · 无参考方法 · 歌唱声转录
Summary 本文提出了一种名为BERT-APC的无参考自动音高校正框架，利用音乐语言模型结合感知音高预测和音符级校正算法，在无需参考音高的情况下修正走调歌声，同时保留演唱的情感表达。实验表明其在音高准确性和自然度上优于现有商业工具。然而，该工作属于音频处理范畴，与文档图像理解无关联。
Reason 该论文研究的是音乐信号处理中的自动音高校正问题，属于音频/音乐信息检索领域，与文档图像理解（DIU）、视觉语言模型（VLM）或文档视觉问答（DocVQA）等任务完全无关。尽管使用了类似BERT的结构，但其输入为音频信号和音乐语言模型，未涉及任何视觉模态、文本布局、OCR或图文对齐机制，无法迁移解决DIU中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Automatic Pitch Correction (APC) enhances vocal recordings by aligning pitch deviations with the intended musical notes. However, existing APC systems either rely on reference pitches, which limits their practical applicability, or employ simple pitch estimation algorithms that often fail to preserve expressiveness and naturalness. We propose BERT-APC, a novel reference-free APC framework that corrects pitch errors while maintaining the natural expressiveness of vocal performances. In BERT-APC, a novel stationary pitch predictor first estimates the perceived pitch of each note from the detuned singing voice. A context-aware note pitch predictor estimates the intended pitch sequence by leveraging a music language model repurposed to incorporate musical context. Finally, a note-level correction algorithm fixes pitch errors while preserving intentional pitch deviations for emotional expression. In addition, we introduce a learnable data augmentation strategy that improves the robustness of the music language model by simulating realistic detuning patterns. Compared to two recent singing voice transcription models, BERT-APC demonstrated superior performance in note pitch prediction, outperforming the second-best model, ROSVOT, by 10.49%p on highly detuned samples in terms of the raw pitch accuracy. In the MOS test, BERT-APC achieved the highest score of $4.32 \pm 0.15$, which is significantly higher than those of the widely-used commercial APC tools, AutoTune ($3.22 \pm 0.18$) and Melodyne ($3.08 \pm 0.18$), while maintaining a comparable ability to preserve expressive nuances. To the best of our knowledge, this is the first APC model that leverages a music language model to achieve reference-free pitch correction with symbolic musical context. The corrected audio samples of BERT-APC are available online.
Authors Sungjae Kim, Kihyun Na, Jinyoung Choi, Injung Kim
Comment 12 pages, 6 figures, 5 tables
Categories Artificial Intelligence
0
Zero-Shot Transfer Capabilities of the Sundial Foundation Model for Leaf Area Index Forecasting
Title CN Sundial基础模型在叶面积指数预测中的零样本迁移能力
Keywords 时间序列预测 · 基础模型 · 零样本学习 · 遥感 · 农业监测
Summary 本文研究了Sundial时间序列基础模型在叶面积指数（LAI）预测任务中的零样本迁移能力。实验表明，在输入包含足够长时间窗口（覆盖多个季节周期）的情况下，Sundial无需微调即可超越全监督训练的LSTM模型，展示了预训练时间序列模型在农业与环境遥感预测中的潜力。
Reason 该论文研究的是时间序列基础模型在农业遥感（叶面积指数预测）中的零样本预测能力，属于环境监测与时间序列预测领域，与文档图像理解（DIU）或视觉-语言模型在文档上的应用无直接关联。未涉及OCR、布局分析、表格识别、多模态文档建模等DIU核心任务，也无法提供可迁移至文档理解的底层方法（如视觉压缩、高分辨率处理、去幻觉机制等）。属于无关领域应用。
Abstract This work investigates the zero-shot forecasting capability of time-series foundation models for Leaf Area Index (LAI) forecasting in agricultural monitoring. Using the HiQ dataset (U.S., 2000-2022), we systematically compare statistical baselines, a fully supervised LSTM, and the Sundial foundation model under multiple evaluation protocols. We find that Sundial, in the zero-shot setting, can outperform a fully trained LSTM provided that the input context window is sufficiently long-specifically, when covering more than one or two full seasonal cycles. This demonstrates, for the first time, that a general-purpose foundation model can surpass specialized supervised models on remote-sensing time series prediction without any task-specific tuning. These results highlight the strong potential of pretrained time-series foundation models to serve as effective plug-and-play forecasters in agricultural and environmental applications.
Authors Peining Zhang, Hongchen Qin, Haochen Zhang, Ziqi Guo, Guiling Wang, Jinbo Bi
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
On the Feasibility of Hijacking MLLMs' Decision Chain via One Perturbation
Title CN 通过单次扰动劫持多模态大语言模型决策链的可行性研究
Keywords 对抗攻击 · 多模态大语言模型 · 决策链劫持 · 语义感知扰动 · 安全漏洞
Summary 本文提出了一种名为SAUP（Semantic-Aware Universal Perturbations）的新方法，能够通过单帧对抗性扰动同时操控多模态大语言模型在多个语义任务上的输出，例如将‘非机动车道’误分类为‘机动车道’，或将‘行人’识别为‘塑料袋’。作者设计了基于语义分离策略的优化算法，并构建了真实世界图像数据集RIST进行验证。实验表明，在仅使用一个对抗帧的情况下，控制五个不同目标的成功率可达70%。
Reason 该论文研究的是对多模态大语言模型（MLLMs）的对抗性攻击，属于安全与鲁棒性领域，其核心是‘语义感知的通用扰动’以操控模型的多步决策链。虽然涉及MLLM，但目标并非提升文档理解能力，而是揭示安全漏洞，且未针对文档图像理解中的OCR幻觉、复杂排版或长文档推理等痛点提出可迁移的解决方案。属于无关领域的安全研究，不具方法论上的正向迁移价值。
Abstract Conventional adversarial attacks focus on manipulating a single decision of neural networks. However, real-world models often operate in a sequence of decisions, where an isolated mistake can be easily corrected, but cascading errors can lead to severe risks. This paper reveals a novel threat: a single perturbation can hijack the whole decision chain. We demonstrate the feasibility of manipulating a model's outputs toward multiple, predefined outcomes, such as simultaneously misclassifying "non-motorized lane" signs as "motorized lane" and "pedestrian" as "plastic bag". To expose this threat, we introduce Semantic-Aware Universal Perturbations (SAUPs), which induce varied outcomes based on the semantics of the inputs. We overcome optimization challenges by developing an effective algorithm, which searches for perturbations in normalized space with a semantic separation strategy. To evaluate the practical threat of SAUPs, we present RIST, a new real-world image dataset with fine-grained semantic annotations. Extensive experiments on three multimodal large language models demonstrate their vulnerability, achieving a 70% attack success rate when controlling five distinct targets using just an adversarial frame.
Authors Changyue Li, Jiaying Li, Youliang Yuan, Jiaming He, Zhicong Huang, Pinjia He
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Popularity Bias Alignment Estimates
Title CN 流行度偏差对齐估计
Keywords 流行度偏差 · 表示学习 · 奇异值分解 · 理论分析 · 记忆化
Summary 本文从理论上扩展了流行度偏差记忆化定理，适用于任意度分布，并给出了与top-k奇异超空间对齐的上下界估计。工作集中在模型内部表示的数学性质分析，缺乏与实际文档理解任务的关联。
Reason 该论文研究的是流行度偏差记忆化定理的扩展，属于纯理论人工智能范畴，聚焦于模型对流行数据的记忆与表示空间对齐问题。内容未涉及任何多模态、视觉输入或文档理解任务，也无可用于文档图像理解（如OCR去幻觉、细粒度定位、高分辨率处理等）的方法论创新。属于无关领域且为抽象理论推导，无法迁移到DIU核心痛点。
Abstract We are extending Popularity Bias Memorization theorem from arXiv:archive/2404.12008 in several directions. We extend it to arbitrary degree distributions and also prove both upper and lower estimates for the alignment with top-k singular hyperspace.
Authors Anton Lyubinin
Categories Artificial Intelligence
0
KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Models
Title CN KEPT：基于视觉语言模型的连续驾驶帧知识增强轨迹预测
Keywords 视觉语言模型 · 轨迹预测 · 自动驾驶 · 检索增强生成 · 时空融合
Summary 本文提出KEPT，一种结合自监督学习的时空融合视频编码器与检索增强生成（RAG）的视觉语言模型框架，用于从连续驾驶画面中预测自车轨迹。通过在CoT提示中引入先验知识和规划约束，并采用三阶段微调策略提升空间感知能力，在nuScenes数据集上实现了优于基线方法的开环预测性能。
Reason 该论文研究的是自动驾驶场景下的轨迹预测问题，属于具身智能与视频理解领域，尽管使用了VLM和RAG等技术，但其核心任务与文档图像理解（DIU）无关。所提出的方法针对驾驶场景中的时序行为建模，并未涉及OCR、布局分析、表格识别或文档幻觉等DIU核心痛点，无法迁移用于解决文档理解中的细粒度识别、高分辨率处理或逻辑推理问题。
Abstract Accurate short-horizon trajectory prediction is crucial for safe and reliable autonomous driving. However, existing vision-language models (VLMs) often fail to accurately understand driving scenes and generate trustworthy trajectories. To address this challenge, this paper introduces KEPT, a knowledge-enhanced VLM framework that predicts ego trajectories directly from consecutive front-view driving frames. KEPT integrates a temporal frequency-spatial fusion (TFSF) video encoder, which is trained via self-supervised learning with hard-negative mining, with a k-means & HNSW retrieval-augmented generation (RAG) pipeline. Retrieved prior knowledge is added into chain-of-thought (CoT) prompts with explicit planning constraints, while a triple-stage fine-tuning paradigm aligns the VLM backbone to enhance spatial perception and trajectory prediction capabilities. Evaluated on nuScenes dataset, KEPT achieves the best open-loop performance compared with baseline methods. Ablation studies on fine-tuning stages, Top-K value of RAG, different retrieval strategies, vision encoders, and VLM backbones are conducted to demonstrate the effectiveness of KEPT. These results indicate that KEPT offers a promising, data-efficient way toward trustworthy trajectory prediction in autonomous driving.
Authors Yujin Wang, Tianyi Wang, Quanfeng Liu, Wenxian Fan, Junfeng Jiao, Christian Claudel, Yunbing Yan, Bingzhao Gao, Jianqiang Wang, Hong Chen
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Domain Fusion Controllable Generalization for Cross-Domain Time Series Forecasting from Multi-Domain Integrated Distribution
Title CN 面向多域融合分布的跨域时间序列预测可控泛化
Keywords 时间序列预测 · 跨域泛化 · 扩散模型 · 域融合 · 可控生成
Summary 本文提出了一种名为TimeControl的时间序列通用扩散模型，通过引入‘域融合’范式，利用扩散模型将多个时间序列域的信息整合到统一的生成过程中，实现跨域预测。模型包含三个关键设计：条件网络提取多尺度波动模式以指导去噪过程；基于适配器的微调策略用于下游任务；混合架构实现观测与预测空间对齐，支持任意长度预测。实验表明其在49个基准数据集上优于30种基线模型，具备优秀的零样本泛化能力。
Reason 该论文研究的是时间序列预测中的跨域泛化问题，属于时间序列分析领域，与文档图像理解（DIU）或视觉-语言模型（VLM）在文档上的应用无直接关联。其核心方法——基于扩散模型的多域信息融合和条件生成——虽然在时间序列领域可能具有创新性，但无法直接迁移解决文档图像理解中的OCR幻觉、复杂排版、细粒度定位或长文档推理等关键痛点。
Abstract Conventional deep models have achieved unprecedented success in time series forecasting. However, facing the challenge of cross-domain generalization, existing studies utilize statistical prior as prompt engineering fails under the huge distribution shift among various domains. In this paper, a novel time series generalization diffusion model (TimeControl) that pioneers the Domain-Fusion paradigm, systematically integrating information from multiple time series domains into a unified generative process via diffusion models. Unlike the autoregressive models that capture the conditional probabilities of the prediction horizon to the historical sequence, we use the diffusion denoising process to model the mixed distribution of the cross-domain data and generate the prediction sequence for the target domain directly utilizing conditional sampling. The proposed TimeControl contains three pivotal designs: (1) The condition network captures the multi-scale fluctuation patterns from the observation sequence, which are utilized as context representations to guide the denoising network to generate the prediction sequence; (2) Adapter-based fine-tuning strategy, the multi-domain universal representation learned in the pretraining stage is utilized for downstream tasks in target domains; (3) A novel hybrid architecture is designed to align the observation and prediction spaces, enabling TimeControl to generate prediction sequences of arbitrary lengths with flexibility. We conduct extensive experiments on mainstream 49 benchmarks and 30 baselines, and the TimeControl outperforms existing baselines on all data domains, exhibiting superior zero-shot generalization ability.
Authors Xiangkai Ma, Xiaobin Hong, Mingkai Lin, Han Zhang, Wenzhong Li, Sanglu Lu
Comment We have updated the abstract, introduction and related work. Additionally, we have incorporated the latest competitive baseline models
Categories Artificial Intelligence
0
Energy-Aware Pattern Disentanglement: A Generalizable Pattern Assisted Architecture for Multi-task Time Series Analysis
Title CN 面向多任务时间序列分析的可泛化模式辅助架构：能量感知的模式解耦
Keywords 时间序列分析 · 多任务学习 · 模式解耦 · 时频分析 · 深度学习
Summary 本文提出了一种名为Pets的新架构，基于能量感知的模式解耦方法，用于多任务时间序列分析。通过在时频域进行局部能量分析，分离瞬态和非平稳振荡成分，并设计了波动模式辅助模块（FPA）和上下文引导的预测器混合模块（MoP），实现对不同能量比例波动的分层建模。该方法在60个基准上实现了SOTA性能，涵盖预测、填补、异常检测和分类等任务。
Reason 该论文研究的是时间序列分析领域的多任务学习问题，属于纯时序信号处理范畴，与文档图像理解（DIU）、视觉语言模型（VLM）或文档视觉问答（DocVQA）等任务无直接关联。其提出的方法针对时间-频率域的能量分布建模，难以迁移到文档图像中的空间结构理解、OCR去幻觉、复杂排版解析等核心痛点。
Abstract Time series analysis has found widespread applications in areas such as weather forecasting, anomaly detection, and healthcare. While deep learning approaches have achieved significant success in this field, existing methods often adopt a "one-model one-task" architecture, limiting their generalization across different tasks. To address these limitations, we perform local energy analysis in the time-frequency domain to more precisely capture and disentangle transient and non-stationary oscillatory components. Furthermore, our representational analysis reveals that generative tasks tend to capture long-period patterns from low-frequency components, whereas discriminative tasks focus on high-frequency abrupt signals, which constitutes our core contribution. Concretely, we propose Pets, a novel "one-model many-tasks" architecture based on the General fluctuation Pattern Assisted (GPA) framework that is adaptable to versatile model structures for time series analysis. Pets integrates a Fluctuation Pattern Assisted (FPA) module and a Context-Guided Mixture of Predictors (MoP). The FPA module facilitates information fusion among diverse fluctuation patterns by capturing their dependencies and progressively modeling these patterns as latent representations at each layer. Meanwhile, the MoP module leverages these generalizable pattern representations to guide and regulate the reconstruction of distinct fluctuations hierarchically by energy proportion. Pets demonstrates strong versatility and achieves state-of-the-art performance across 60 benchmarks on various tasks, including forecasting, imputation, anomaly detection, and classification, while demonstrating strong generalization and robustness.
Authors Xiangkai Ma, Xiaobin Hong, Wenzhong Li, Sanglu Lu
Comment We have updated the abstract, citations and related work. At the same time, we have also updated the latest baseline model
Categories Artificial Intelligence
0
ProactivePIM: Accelerating Weight-Sharing Embedding Layer with PIM for Scalable Recommendation System
Title CN ProactivePIM：利用存内处理加速共享权重嵌入层以实现可扩展推荐系统
Keywords 推荐系统 · 存内处理 · 嵌入层优化 · 模型压缩 · 硬件加速
Summary 本文提出ProactivePIM，一种针对权重共享压缩算法（如QR-trick、TT-Rec）优化的存内处理（PIM）系统，通过异构HBM-DIMM内存架构和两级PIM设计（bd-PIM与bg-PIM），结合SRAM缓存预取和子表复制策略，减少CPU-PIM通信开销并提升内存访问效率，在推荐系统嵌入层推理中实现了显著加速。
Reason 该论文聚焦于推荐系统中的嵌入层优化，属于典型的系统架构与硬件协同设计工作，研究场景为推荐模型的推理加速。尽管涉及深度学习模型压缩与高效计算，但其方法和问题域与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联，无法迁移用于解决OCR幻觉、复杂排版解析、细粒度定位等DIU核心痛点。
Abstract Although deep learning-based personalized recommendation systems provide qualified recommendations, they strain data center resources. The main bottleneck is the embedding layer, which is highly memory-intensive due to its sparse, irregular access patterns to embeddings. Recent near-memory processing (NMP) and processing-in-memory (PIM) architectures have addressed these issues by exploiting parallelism within memory. However, as model sizes increase year by year and can exceed server capacity, inference on single-node servers becomes challenging, necessitating the integration of model compression. Various algorithms have been proposed for model size reduction, but they come at the cost of increased memory access and CPU-PIM communication. We present ProactivePIM, a PIM system tailored for weight-sharing algorithms, a family of compression methods that decompose an embedding table into compact subtables, such as QR-trick and TT-Rec. Our analysis shows that embedding layer execution with weight-sharing algorithms increases memory access and incurs CPU-PIM communication. We also find that these algorithms exhibit unique data locality characteristics, which we name intra-GnR locality. ProactivePIM accelerates weight-sharing algorithms by utilizing a heterogeneous HBM-DIMM memory architecture with integration of a two-level PIM system of base-die PIM (bd-PIM) and bank-group PIM (bg-PIM) inside the HBM. To gain further speedup, ProactivePIM prefetches embeddings with high intra-GnR locality into an SRAM cache within bg-PIM and eliminates the CPU-PIM communication through duplication of target subtables across bank groups. With additional optimization techniques, our design effectively accelerates weight-sharing algorithms, achieving 2.22x and 2.15x speedup in QR-trick and TT-Rec, respectively, compared to the baseline architecture.
Authors Youngsuk Kim, Junghwan Lim, Hyuk-Jae Lee, Chae Eun Rhee
Comment 14 pages, 13 figures
Categories Artificial Intelligence
0
Deep Hybrid Model for Region of Interest Detection in Omnidirectional Videos
Title CN 用于全景视频中感兴趣区域检测的深度混合模型
Keywords 全景视频 · 感兴趣区域检测 · 显著性模型 · 视频流优化 · 视口预测
Summary 本文提出了一种用于预测360°视频中感兴趣区域（ROI）的混合显著性模型，旨在通过预判用户视口来优化视频流传输效率和观看体验。方法包括视频帧预处理、构建混合显著性模型进行ROI预测，以及后处理输出结果，并在360RAT数据集上与主观标注进行了对比评估。
Reason 该论文研究的是360°视频中的视觉显著性预测与视口检测问题，属于视频理解与视觉注意力建模领域，与文档图像理解（DIU）的核心任务如DocVQA、版面分析、表格识别等无直接关联。其方法聚焦于动态视频流的带宽优化和观看体验提升，不涉及OCR、文本布局、密集文本解析或文档级推理等DIU关键挑战。且全景视频的输入模态与文档图像差异巨大，技术迁移路径模糊，无法用于解决OCR幻觉、复杂排版或长文档理解等问题。
Abstract The main goal of the project is to design a new model that predicts regions of interest in 360$^{\circ}$ videos. The region of interest (ROI) plays an important role in 360$^{\circ}$ video streaming. For example, ROIs are used to predict view-ports, intelligently cut the videos for live streaming, etc so that less bandwidth is used. Detecting view-ports in advance helps reduce the movement of the head while streaming and watching a video via the head-mounted device. Whereas, intelligent cuts of the videos help improve the efficiency of streaming the video to users and enhance the quality of their viewing experience. This report illustrates the secondary task to identify ROIs, in which, we design, train, and test a hybrid saliency model. In this work, we refer to saliency regions to represent the regions of interest. The method includes the processes as follows: preprocessing the video to obtain frames, developing a hybrid saliency model for predicting the region of interest, and finally post-processing the output predictions of the hybrid saliency model to obtain the output region of interest for each frame. Then, we compare the performance of the proposed method with the subjective annotations of the 360RAT dataset.
Authors Sana Alamgeer, Mylene Farias, Marcelo Carvalho
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
AI/ML based Joint Source and Channel Coding for HARQ-ACK Payload Journal of Sel. Areas in Information Theory
Title CN 基于AI/ML的HARQ-ACK有效载荷联合信源信道编码
Keywords 联合信源信道编码 · HARQ-ACK · Transformer · 深度学习 · 5G NR
Summary 本文针对5G NR上行链路中的HARQ-ACK反馈比特提出了一种基于Transformer的联合信源信道编码方案，利用深度学习建模其非均匀分布特性，并设计了功率整形与解码机制以实现NACK比特的不等错误保护。结果显示在衰落信道下可显著降低发射功率需求。
Reason 该论文属于通信物理层编码领域，研究的是5G上行链路中HARQ-ACK比特的非均匀分布建模与深度学习辅助的联合信源信道编码，尽管使用了Transformer和AI技术，但其问题域完全脱离文档图像理解（DIU）或视觉语言模型（VLM）范畴。既不涉及OCR、布局分析、表格识别等DIU任务，也未提出可用于多模态文档理解的方法论创新，属于典型的无关领域工作。
Abstract Channel coding from 2G to 5G has assumed the inputs bits at the physical layer to be uniformly distributed. However, hybrid automatic repeat request acknowledgement (HARQ-ACK) bits transmitted in the uplink are inherently non-uniformly distributed. For such sources, significant performance gains could be obtained by employing joint source channel coding, aided by deep learning-based techniques. In this paper, we learn a transformer-based encoder using a novel "free-lunch" training algorithm and propose per-codeword power shaping to exploit the source prior at the encoder whilst being robust to small changes in the HARQ-ACK distribution. Furthermore, any HARQ-ACK decoder has to achieve a low negative acknowledgement (NACK) error rate to avoid radio link failures resulting from multiple NACK errors. We develop an extension of the Neyman-Pearson test to a coded bit system with multiple information bits to achieve Unequal Error Protection of NACK over ACK bits at the decoder. Finally, we apply the proposed encoder and decoder designs to a 5G New Radio (NR) compliant uplink setup under a fading channel, describing the optimal receiver design and a low complexity coherent approximation to it. Our results demonstrate 3-6 dB reduction in the average transmit power required to achieve the target error rates compared to the NR baseline, while also achieving a 2-3 dB reduction in the maximum transmit power, thus providing for significant coverage gains and power savings.
Authors Akash Doshi, Pinar Sen, Kirill Ivanov, Wei Yang, June Namgoong, Runxin Wang, Rachel Wang, Taesang Yoo, Jing Jiang, Tingfang Ji
Comment 39 pages, 15 figures. Under consideration for publication in Journal of Sel. Areas in Information Theory. This paper was presented in part at the International Symposium on Topics in Coding, August 2025 in the Session for Coding and AI
Categories Artificial Intelligence
0
Optimize Flip Angle Schedules In MR Fingerprinting Using Reinforcement Learning
Title CN 使用强化学习优化磁共振指纹中的翻转角度序列
Keywords 强化学习 · 磁共振指纹 · 翻转角度优化 · 医学影像 · 序列设计
Summary 本文提出了一种基于强化学习的框架，用于优化磁共振指纹（MRF）中的翻转角度序列，旨在提高指纹信号在参数空间中的可区分性。实验表明，学习到的非周期性序列能够增强指纹分离度，并可能减少重复时间次数，从而加速MRF采集过程。
Reason 该论文研究的是磁共振成像（MRI）中的脉冲序列优化问题，属于医学影像物理采集层面的技术改进，与文档图像理解（DIU）、视觉语言模型（VLM）、OCR、布局分析、表格识别等任务完全无关。尽管使用了强化学习这一AI方法，但其应用场景为信号采集参数调优，无法迁移至文档理解中的OCR幻觉、复杂排版、长文档推理等核心痛点。
Abstract Magnetic Resonance Fingerprinting (MRF) leverages transient-state signal dynamics generated by the tunable acquisition parameters, making the design of an optimal, robust sequence a complex, high-dimensional sequential decision problem, such as optimizing one of the key parameters, flip angle. Reinforcement learning (RL) offers a promising approach to automate parameter selection, to optimize pulse sequences that maximize the distinguishability of fingerprints across the parameter space. In this work, we introduce an RL framework for optimizing the flip-angle schedule in MRF and demonstrate a learned schedule exhibiting non-periodic patterns that enhances fingerprint separability. Additionally, an interesting observation is that the RL-optimized schedule may enable a reduction in the number of repetition time, potentially accelerate MRF acquisitions.
Authors Shenjun Zhong, Zhifeng Chen, Zhaolin Chen
Comment 4 pages, 5 figures, submitted to conference
Categories Artificial Intelligence
0
LLM-EDT: Large Language Model Enhanced Cross-domain Sequential Recommendation with Dual-phase Training
Title CN LLM-EDT：基于双阶段训练的大语言模型增强跨域序列推荐
Keywords 跨域推荐 · 序列推荐 · 大语言模型 · 用户画像 · 双阶段训练
Summary 本文提出了一种结合大语言模型的跨域序列推荐方法LLM-EDT，通过双阶段训练和领域感知的用户画像模块来缓解交互不平衡和跨域偏好转移问题。虽然引入了LLM进行用户行为生成与编码，但整体框架局限于推荐系统场景，无视觉模态处理能力。
Reason 该论文属于推荐系统领域，研究的是跨域用户行为序列中的物品推荐问题，尽管使用了大语言模型作为辅助模块，但其核心任务与文档图像理解（DIU）、DocVQA、OCR、布局分析等完全无关。未涉及任何视觉输入、多模态文档建模或文本-视觉对齐机制，也无法迁移用于解决OCR幻觉、复杂排版解析、细粒度定位等DIU核心痛点，属于典型的平行应用类工作。
Abstract Cross-domain Sequential Recommendation (CDSR) has been proposed to enrich user-item interactions by incorporating information from various domains. Despite current progress, the imbalance issue and transition issue hinder further development of CDSR. The former one presents a phenomenon that the interactions in one domain dominate the entire behavior, leading to difficulty in capturing the domain-specific features in the other domain. The latter points to the difficulty in capturing users' cross-domain preferences within the mixed interaction sequence, resulting in poor next-item prediction performance for specific domains. With world knowledge and powerful reasoning ability, Large Language Models (LLMs) partially alleviate the above issues by performing as a generator and an encoder. However, current LLMs-enhanced CDSR methods are still under exploration, which fail to recognize the irrelevant noise and rough profiling problems. Thus, to make peace with the aforementioned challenges, we proposed an LLMs Enhanced Cross-domain Sequential Recommendation with Dual-phase Training ({LLM-EDT}). To address the imbalance issue while introducing less irrelevant noise, we first propose the transferable item augmenter to adaptively generate possible cross-domain behaviors for users. Then, to alleviate the transition issue, we introduce a dual-phase training strategy to empower the domain-specific thread with a domain-shared background. As for the rough profiling problem, we devise a domain-aware profiling module to summarize the user's preference in each domain and adaptively aggregate them to generate comprehensive user profiles. The experiments on three public datasets validate the effectiveness of our proposed LLM-EDT. To ease reproducibility, we have released the detailed code online at {https://anonymous.4open.science/r/LLM-EDT-583F}.
Authors Ziwei Liu, Qidong Liu, Wanyu Wang, Yejing Wang, Tong Xu, Wei Huang, Chong Chen, Peng Chuan, Xiangyu Zhao
Categories Artificial Intelligence
0
Elucidated Rolling Diffusion Models for Probabilistic Weather Forecasting NeurIPS 2025
Title CN 用于概率天气预报的阐明滚动扩散模型
Keywords 扩散模型 · 概率预测 · 时间依赖性 · 不确定性传播 · 天气预报
Summary 本文提出了Elucidated Rolling Diffusion Models（ERDM），首次成功将滚动预测结构与高性能的Elucidated Diffusion Models（EDM）结合，用于解决高维复杂系统中的概率性动态预测问题。通过改进噪声调度、网络预调节和采样策略，并引入新的损失加权机制、预训练初始化策略以及混合序列架构，在2D Navier-Stokes模拟和ERA5全球天气数据上优于现有扩散模型。
Reason 该论文研究的是基于扩散模型的概率性天气预报，属于气象预测与视频预测的交叉领域，其方法聚焦于时空扩散过程中的不确定性传播建模。尽管使用了先进的扩散架构（ERDM），但其应用场景和核心问题（如长期气候状态生成、不确定性演化）与文档图像理解（DIU）中的OCR幻觉、密集文本识别、复杂版面分析或长文档推理等痛点无直接关联。所提出的技术难以迁移到文档多模态理解任务中。
Abstract Diffusion models are a powerful tool for probabilistic forecasting, yet most applications in high-dimensional complex systems predict future states individually. This approach struggles to model complex temporal dependencies and fails to explicitly account for the progressive growth of uncertainty inherent to the systems. While rolling diffusion frameworks, which apply increasing noise to forecasts at longer lead times, have been proposed to address this, their integration with state-of-the-art, high-fidelity diffusion techniques remains a significant challenge. We tackle this problem by introducing Elucidated Rolling Diffusion Models (ERDM), the first framework to successfully unify a rolling forecast structure with the principled, performant design of Elucidated Diffusion Models (EDM). To do this, we adapt the core EDM components-its noise schedule, network preconditioning, and Heun sampler-to the rolling forecast setting. The success of this integration is driven by three key contributions: (i) a novel loss weighting scheme that focuses model capacity on the mid-range forecast horizons where determinism gives way to stochasticity; (ii) an efficient initialization strategy using a pre-trained EDM for the initial window; and (iii) a bespoke hybrid sequence architecture for robust spatiotemporal feature extraction under progressive denoising. On 2D Navier-Stokes simulations and ERA5 global weather forecasting at 1.5-degree resolution, ERDM consistently outperforms key diffusion-based baselines, including conditional autoregressive EDM. ERDM offers a flexible and powerful general framework for tackling diffusion-based dynamics forecasting problems where modeling uncertainty propagation is paramount.
Authors Salva Rühling Cachay, Miika Aittala, Karsten Kreis, Noah Brenowitz, Arash Vahdat, Morteza Mardani, Rose Yu
Comment NeurIPS 2025
Categories Artificial Intelligence
0
On the dimension of pullback attractors in recurrent neural networks
Title CN 关于循环神经网络中拉回吸引子的维度
Keywords 循环神经网络 · 储层计算 · 拉回吸引子 · 盒计数维度 · 动力系统
Summary 本文从非自治动力系统的角度，研究了储层计算机中拉回吸引子的盒计数维度，证明其维度上界受限于输入序列空间的维度。结果表明尽管储层具有高维状态空间，但实际动态是低维的，这解释了其在吸引子重建和动力学不变量计算中的有效性。
Reason 该论文研究的是储层计算（reservoir computing）在混沌系统吸引子重构中的理论性质，属于动力系统与传统RNN的理论分析范畴，与文档图像理解（DIU）、多模态大模型（VLM）、视觉表示、OCR或DocVQA等任务完全无关。未涉及任何视觉输入、多模态建模或文档结构理解内容，无法迁移用于解决OCR幻觉、复杂排版或长文档推理等DIU核心问题。
Abstract Recurrent neural networks trained via the reservoir computing paradigm have demonstrated remarkable success in learning and reconstructing attractors from chaotic systems, often replicating quantities such as Lyapunov exponents and fractal dimensions. It has recently been conjectured that this is because the reservoir computer embeds the dynamics of the chaotic system in its state space before learning. This conjecture has been established for reservoir computers with linear activation functions and remains open for more general reservoir systems. In this work, we employ a non-autonomous dynamical systems approach to establish an upper bound for the box-counting dimension of the pullback attractor, a subset of the reservoir state space that is approximated during training and prediction phases. We prove that the box-counting dimension of the pullback attractor is bounded above by the box-counting dimension of the space of input sequences with respect to the product topology. In particular, for input sequences originating from an Nin-dimensional smooth dynamical system or their generic continuously differentiable observations, the box-counting dimension of the pullback attractor is bounded above by Nin. The results obtained here highlight the fact that, while a reservoir computer may possess a very high-dimensional state space, it exhibits effective low-dimensional dynamics. Our findings also partly explain why reservoir computers are successful in tasks such as attractor reconstruction and the computation of dynamic invariants like Lyapunov exponents and fractal dimensions.
Authors Muhammed Fadera
Comment Issues with clarity and notation
Categories Artificial Intelligence
0
CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistance NeurIPS 2025 Datasets and Benchmarks Track
Title CN CodeAssistBench (CAB)：基于多轮对话的代码辅助数据集与基准测试
Keywords 代码辅助 · 多轮对话 · 基准测试 · 自动化构建 · 项目上下文
Summary 本文提出了CodeAssistBench（CAB），首个用于评估多轮、项目级编程辅助的大规模基准。CAB通过LLM驱动的自动化流程从GitHub问题中构建数据，涵盖可运行上下文和执行环境，覆盖七种编程语言。实验表明现有大模型在真实项目场景下的表现远低于传统问答基准，揭示了当前LLM在实际开发支持中的局限性。该基准完全自动化且公开可用。
Reason 该论文属于典型的平行下游应用，聚焦于代码辅助这一特定任务场景，构建的是面向软件工程领域的多轮对话基准，而非解决文档图像理解中的核心问题（如OCR幻觉、复杂排版、视觉文本定位等）。尽管其自动化数据构建流程具有一定工程价值，但方法论上未对DIU领域提供可迁移的技术创新，属于‘Wrapper/Application’类工作。
Abstract Programming assistants powered by large language models have improved dramatically, yet existing benchmarks still evaluate them in narrow code-generation settings. Recent efforts such as InfiBench and StackEval rely on Stack Overflow questions and remain limited to single-turn interactions, manually curated data, and isolated snippets rather than full project environments. We introduce CodeAssistBench (CAB), the first benchmark for evaluating multi-turn, project-grounded programming assistance at scale. CAB automatically constructs datasets from GitHub issues tagged as questions, using an LLM-driven pipeline that filters noise, extracts runnable contexts, builds executable containers, and verifies environment correctness. This enables continuous, automated expansion across diverse repositories without manual intervention. Using CAB, we create a testbed of 3,286 real-world issues across 214 repositories, spanning seven languages. Evaluating state-of-the-art models reveals a substantial gap: while models achieve 70-83% accuracy on Stack Overflow-style questions, they solve only 16.49% of CAB issues from post-training-cutoff repositories. On a manually validated subset of 149 issues, top models such as Claude Sonnet 4.5 reach only 12.08% correctness. These results highlight a fundamental challenge: current LLMs struggle to provide assistance in realistic, project-specific contexts despite strong performance on traditional Q&A benchmarks. CAB provides a scalable, reproducible framework for advancing research in multi-turn, codebase-grounded programming agents. The benchmark and pipeline are fully automated and publicly available at https://github.com/amazon-science/CodeAssistBench/.
Authors Myeongsoo Kim, Shweta Garg, Baishakhi Ray, Varun Kumar, Anoop Deoras
Comment Accepted to NeurIPS 2025 Datasets and Benchmarks Track
Categories Artificial Intelligence
0
Zero-Knowledge Proof Based Verifiable Inference of Models
Title CN 基于零知识证明的模型可验证推理
Keywords 零知识证明 · 模型验证 · 隐私保护 · zkSNARK · 可信AI
Summary 本文提出了一种基于零知识证明的深度学习模型推理验证框架，能够在不暴露模型参数的前提下验证推理结果的正确性。该方法支持多种神经网络层，生成常数大小的简洁非交互式证明，并实现了ZK-DeepSeek系统进行实验验证。重点在于模型版权保护与可信计算，而非提升视觉或多模态理解能力。
Reason 该论文研究的是如何在不暴露模型参数的情况下验证AI模型推理的正确性，属于AI安全与隐私保护领域。其核心贡献是构建一个基于零知识证明（zkSNARK）的验证框架，并应用于DeepSeek模型的可验证推理。尽管技术新颖，但完全不涉及文档图像理解（DIU）任务本身，也不提供可用于解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点的方法论。既非DIU本题工作，也无直接可迁移的技术工具（如视觉表示、高分辨率处理、多步推理架构等），属于与文档理解无关的平行技术方向。
Abstract Recent advances in artificial intelligence (AI), particularly deep learning, have led to widespread adoption across various applications. Yet, a fundamental challenge persists: how can we verify the correctness of AI model inference when model owners cannot (or will not) reveal their parameters? These parameters represent enormous training costs and valuable intellectual property, making transparent verification difficult. In this paper, we introduce a zero-knowledge framework capable of verifying deep learning inference without exposing model internal parameters. Built on recursively composed zero-knowledge proofs and requiring no trusted setup, our framework supports both linear and nonlinear neural network layers, including matrix multiplication, normalization, softmax, and SiLU. Leveraging the Fiat-Shamir heuristic, we obtain a succinct non-interactive argument of knowledge (zkSNARK) with constant-size proofs. To demonstrate the practicality of our approach, we translate the DeepSeek model into a fully SNARK-verifiable version named ZK-DeepSeek and show experimentally that our framework delivers both efficiency and flexibility in real-world AI verification workloads.
Authors Yunxiao Wang
Categories Artificial Intelligence
0
Shape-Adapting Gated Experts: Dynamic Expert Routing for Colonoscopic Lesion Segmentation
Title CN 形状自适应门控专家：用于结肠镜病变分割的动态专家路由
Keywords 动态专家路由 · 医学图像分割 · Whole Slide Images · CNN-Transformer混合模型 · 门控机制
Summary 本文提出了一种名为SAGE（Shape-Adapting Gated Experts）的输入自适应框架，用于在异构视觉网络中实现动态专家路由。通过双路径设计和多级门控机制，SAGE能够在不同层次上选择共享或专用专家，从而提升模型对细胞形态多样性的适应能力。在EBHI、DigestPath和GlaS三个医学图像数据集上实现了最先进的分割性能。然而，该工作专注于医学图像分割，与文档理解无关。
Reason 该论文聚焦于医学图像中的病变分割任务，属于纯医学图像分析领域，使用Whole Slide Images（WSI）进行癌症检测，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。尽管其提出的动态专家路由机制具有一定的方法创新性，但其应用场景和目标均远离文档结构、文本识别、排版理解或OCR相关痛点，迁移至DIU领域的可能性极低。
Abstract The substantial diversity in cell scale and form remains a primary challenge in computer-aided cancer detection on gigapixel Whole Slide Images (WSIs), attributable to cellular heterogeneity. Existing CNN-Transformer hybrids rely on static computation graphs with fixed routing, which consequently causes redundant computation and limits their adaptability to input variability. We propose Shape-Adapting Gated Experts (SAGE), an input-adaptive framework that enables dynamic expert routing in heterogeneous visual networks. SAGE reconfigures static backbones into dynamically routed expert architectures. SAGE's dual-path design features a backbone stream that preserves representation and selectively activates an expert path through hierarchical gating. This gating mechanism operates at multiple hierarchical levels, performing a two-level, hierarchical selection between shared and specialized experts to modulate model logits for Top-K activation. Our Shape-Adapting Hub (SA-Hub) harmonizes structural and semantic representations across the CNN and the Transformer module, effectively bridging diverse modules. Embodied as SAGE-UNet, our model achieves superior segmentation on three medical benchmarks: EBHI, DigestPath, and GlaS, yielding state-of-the-art Dice Scores of 95.57%, 95.16%, and 94.17%, respectively, and robustly generalizes across domains by adaptively balancing local refinement and global context. SAGE provides a scalable foundation for dynamic expert routing, enabling flexible visual reasoning.
Authors Gia Huy Thai, Hoang-Nguyen Vu, Anh-Minh Phan, Quang-Thinh Ly, Tram Dinh, Thi-Ngoc-Truc Nguyen, Nhat Ho
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
VidComposition: Can MLLMs Analyze Compositions in Compiled Videos? CVPR 2025
Title CN VidComposition：多模态大模型能理解合成视频中的构图吗？
Keywords 多模态大模型 · 视频理解 · 构图分析 · 基准测试 · CVPR
Summary 本文提出了VidComposition，一个用于评估多模态大模型（MLLMs）在合成视频中理解视觉构图能力的新基准，包含982个视频和1706道选择题，涵盖镜头语言、叙事结构等多个维度。对33个主流MLLM的评测显示模型与人类在该任务上存在显著差距。
Reason 该论文聚焦于视频内容的构图理解（如镜头运动、角度、叙事结构等），属于视频理解领域，与文档图像理解（DIU）的核心任务（如DocVQA、表格识别、文本布局分析）无直接关联。尽管其提出的构图分析框架在理论上可能启发视觉结构建模，但迁移路径不明确，且未涉及文档特有的挑战（如OCR幻觉、密集文本、细粒度定位）。此外，任务设定基于影视级视频，与文档图像的静态、高密度符号化特性差异巨大，属于无关领域。
Abstract The advancement of Multimodal Large Language Models (MLLMs) has enabled significant progress in multimodal understanding, expanding their capacity to analyze video content. However, existing evaluation benchmarks for MLLMs primarily focus on abstract video comprehension, lacking a detailed assessment of their ability to understand video compositions, the nuanced interpretation of how visual elements combine and interact within highly compiled video contexts. We introduce VidComposition, a new benchmark specifically designed to evaluate the video composition understanding capabilities of MLLMs using carefully curated compiled videos and cinematic-level annotations. VidComposition includes 982 videos with 1706 multiple-choice questions, covering various compositional aspects such as camera movement, angle, shot size, narrative structure, character actions and emotions, etc. Our comprehensive evaluation of 33 open-source and proprietary MLLMs reveals a significant performance gap between human and model capabilities. This highlights the limitations of current MLLMs in understanding complex, compiled video compositions and offers insights into areas for further improvement. The leaderboard and evaluation code are available at https://yunlong10.github.io/VidComposition/
Authors Yolo Y. Tang, Junjia Guo, Hang Hua, Susan Liang, Mingqian Feng, Xinyang Li, Rui Mao, Chao Huang, Jing Bi, Zeliang Zhang, Pooyan Fazli, Chenliang Xu
Comment Accepted to CVPR 2025
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Generative AI for Cel-Animation: A Survey ICCV 2025 AISTORY Workshop
Title CN 用于赛璐珞动画的生成式AI：综述
Keywords 生成式AI · 赛璐珞动画 · 多模态模型 · 扩散模型 · 动画生成
Summary 本文综述了生成式AI在传统赛璐珞动画制作中的应用，涵盖故事板生成、中间帧补全、上色等环节，讨论了当前工具如AniDoc、ToonCrafter等如何降低创作门槛，并分析了视觉一致性、风格连贯性等挑战及未来方向。
Reason 该论文属于视频/动画生成领域，与文档图像理解（DIU）或DocVQA无直接关联。尽管涉及多模态模型和视觉生成技术，但其核心任务是动画制作流程的自动化，无法迁移解决OCR幻觉、复杂排版理解或长文档推理等DIU核心问题。
Abstract Traditional Celluloid (Cel) Animation production pipeline encompasses multiple essential steps, including storyboarding, layout design, keyframe animation, inbetweening, and colorization, which demand substantial manual effort, technical expertise, and significant time investment. These challenges have historically impeded the efficiency and scalability of Cel-Animation production. The rise of generative artificial intelligence (GenAI), encompassing large language models, multimodal models, and diffusion models, offers innovative solutions by automating tasks such as inbetween frame generation, colorization, and storyboard creation. This survey explores how GenAI integration is revolutionizing traditional animation workflows by lowering technical barriers, broadening accessibility for a wider range of creators through tools like AniDoc, ToonCrafter, and AniSora, and enabling artists to focus more on creative expression and artistic innovation. Despite its potential, challenges like visual consistency, stylistic coherence, and ethical considerations persist. Additionally, this paper explores future directions and advancements in AI-assisted animation. For further exploration and resources, please visit our GitHub repository: https://github.com/yunlong10/Awesome-AI4Animation
Authors Yolo Y. Tang, Junjia Guo, Pinxin Liu, Zhiyuan Wang, Hang Hua, Jia-Xing Zhong, Yunzhong Xiao, Chao Huang, Luchuan Song, Susan Liang, Yizhi Song, Liu He, Jing Bi, Mingqian Feng, Xinyang Li, Zeliang Zhang, Chenliang Xu
Comment Accepted by ICCV 2025 AISTORY Workshop
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction NuerIPS 2025
Title CN 从预测到规划：用于协同状态-动作预测的策略世界模型
Keywords 世界模型 · 自动驾驶 · 轨迹规划 · 视频预测 · 协同预测
Summary 本文提出了一种名为策略世界模型（PWM）的新驾驶范式，统一了世界建模与轨迹规划，通过无动作的未来状态预测机制实现类人预判感知，并引入动态增强的并行token生成方法提升视频预测效率。尽管仅使用前视摄像头输入，性能仍达到或超越依赖多视角和多模态输入的现有方法。
Reason 该论文属于自动驾驶领域的世界模型研究，尽管涉及视觉和语言建模技术，但其核心任务是驾驶场景下的状态-动作联合预测与轨迹规划，与文档图像理解（DIU）或DocVQA无直接关联。所提出的方法如‘无动作未来状态预测’和‘动态增强并行token生成’针对的是视频时序预测与控制决策问题，难以迁移到解决文档中的OCR幻觉、复杂排版解析或长文档推理等核心痛点。
Abstract Despite remarkable progress in driving world models, their potential for autonomous systems remains largely untapped: the world models are mostly learned for world simulation and decoupled from trajectory planning. While recent efforts aim to unify world modeling and planning in a single framework, the synergistic facilitation mechanism of world modeling for planning still requires further exploration. In this work, we introduce a new driving paradigm named Policy World Model (PWM), which not only integrates world modeling and trajectory planning within a unified architecture, but is also able to benefit planning using the learned world knowledge through the proposed action-free future state forecasting scheme. Through collaborative state-action prediction, PWM can mimic the human-like anticipatory perception, yielding more reliable planning performance. To facilitate the efficiency of video forecasting, we further introduce a dynamically enhanced parallel token generation mechanism, equipped with a context-guided tokenizer and an adaptive dynamic focal loss. Despite utilizing only front camera input, our method matches or exceeds state-of-the-art approaches that rely on multi-view and multi-modal inputs. Code and model weights will be released at https://github.com/6550Zhao/Policy-World-Model.
Authors Zhida Zhao, Talas Fu, Yifan Wang, Lijun Wang, Huchuan Lu
Comment Accepted by NuerIPS 2025 (Poster)
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
0
CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detection
Title CN CodeFuse-CommitEval：面向大语言模型在提交信息与代码变更不一致性检测能力的基准测试
Keywords 代码变更 · 提交信息 · 不一致性检测 · 大语言模型 · 基准测试
Summary 本文提出了首个用于检测代码提交信息与代码变更之间不一致性的基准CodeFuse-CommitEval，基于ApacheCM数据集构建，通过规则引导生成七类不一致样本，并评估了多种开源大模型在不同提示策略下的表现。研究发现模型对不一致样本的识别 recall 较高，但在意图级不一致上仍面临挑战。
Reason 该论文属于纯软件工程领域的下游应用，聚焦于代码提交信息与代码变更之间的不一致性检测，任务本质是代码语义分析与文本匹配，未涉及任何文档图像、视觉布局、OCR或多模态理解内容。尽管使用了LLM和推理增强策略（如CoT），但这些均为已有方法的常规应用，且应用场景与文档图像理解（DIU）无交集，无法迁移解决OCR幻觉、复杂排版或细粒度定位等核心问题。
Abstract Version control relies on commit messages to convey the rationale for code changes, but these messages are often low quality and, more critically, inconsistent with their diffs-known as message-code inconsistency (MCI). MCIs mislead reviewers, hinder maintenance, contaminate research datasets, and may obscure security patches. Yet, no dedicated benchmark exists to evaluate models for MCI detection. We introduce CODEFUSE-COMMITEVAL, the first benchmark designed for MCI detection using large language models (LLMs). Built on the ApacheCM dataset for diversity and quality, we generate seven types of inconsistent messages through rule-guided mutations of originally consistent commits and apply two-fold validation to verify both positive and negative samples. Using this labeled dataset of message-diff pairs, we evaluate six state-of-the-art open-source LLMs under a vanilla setting and with three augmentation strategies: few-shot prompting, chain-of-thought, and extended context. Results show models detect inconsistent commits more reliably than consistent ones (average Recall 85.95%, Precision 80.28%, Specificity 63.8%); gpt-oss-20B performs best overall but uses over twice the tokens of others. Augmentation effects vary: adjacent context helps larger models but adds noise for smaller ones; few-shot improves accuracy and reduces token use, yet increases universally incorrect predictions; chain-of-thought boosts precision and specificity at the cost of recall and higher token consumption. Type-wise analysis reveals higher detectability for component, file-path, and operation inconsistencies, but lower accuracy and higher token cost for intent-level "purpose" inconsistencies. CODEFUSE-COMMITEVAL provides a rigorous foundation for measuring, comparing, and advancing MCI detection, highlighting the need for richer context and balanced data to capture high-level semantic gaps.
Authors Qingyu Zhang, Puzhuo Liu, Peng Di, Chenxiong Qian
Categories Artificial Intelligence
0
Cross-LLM Generalization of Behavioral Backdoor Detection in AI Agent Supply Chains
Title CN AI代理供应链中行为后门检测的跨LLM泛化
Keywords AI安全 · 后门检测 · 跨模型泛化 · LLM供应链 · 行为分析
Summary 随着AI代理在企业流程中的广泛应用，其依赖共享工具库和预训练组件带来了供应链安全风险。本文首次系统研究了在不同大语言模型之间行为后门检测的泛化能力，实验覆盖六种主流生产级LLM，发现单一模型训练的检测器在跨模型场景下性能骤降近44个百分点，几乎退化到随机猜测水平。作者分析指出，这种泛化差距源于模型特有的行为时序特征差异，并提出引入模型身份作为额外特征的‘模型感知’检测方法，可将跨模型检测准确率提升至90.6%。同时发布了多LLM执行轨迹数据集与检测框架以促进后续研究。
Reason 该论文研究的是AI代理供应链中的安全问题，具体为跨大语言模型（LLM）的行为后门检测，属于纯NLP安全与对齐领域。尽管涉及多LLM行为分析，但其核心关注点是模型安全性与供应链攻击防御，与文档图像理解（DIU）、视觉内容解析、OCR、VQA或任何视觉-语言文档任务无直接关联。文中未涉及图像输入、视觉表示、布局分析或文档结构理解等DIU关键问题，也无法迁移用于解决OCR幻觉、细粒度定位或复杂排版建模等痛点。
Abstract As AI agents become integral to enterprise workflows, their reliance on shared tool libraries and pre-trained components creates significant supply chain vulnerabilities. While previous work has demonstrated behavioral backdoor detection within individual LLM architectures, the critical question of cross-LLM generalization remains unexplored, a gap with serious implications for organizations deploying multiple AI systems. We present the first systematic study of cross-LLM behavioral backdoor detection, evaluating generalization across six production LLMs (GPT-5.1, Claude Sonnet 4.5, Grok 4.1, Llama 4 Maverick, GPT-OSS 120B, and DeepSeek Chat V3.1). Through 1,198 execution traces and 36 cross-model experiments, we quantify a critical finding: single-model detectors achieve 92.7% accuracy within their training distribution but only 49.2% across different LLMs, a 43.4 percentage point generalization gap equivalent to random guessing. Our analysis reveals that this gap stems from model-specific behavioral signatures, particularly in temporal features (coefficient of variation > 0.8), while structural features remain stable across architectures. We show that model-aware detection incorporating model identity as an additional feature achieves 90.6% accuracy universally across all evaluated models. We release our multi-LLM trace dataset and detection framework to enable reproducible research.
Authors Arun Chowdary Sanna
Comment 10 pages, 2 figures, 8 tables. Evaluation across 6 production LLMs with 1,198 traces
Categories Artificial Intelligence
0
Agentic AI-Empowered Conversational Embodied Intelligence Networks in 6G IEEE Vehicle Technology Magazine
Title CN 6G中的智能体AI赋能的会话型具身智能网络
Keywords 具身智能 · 6G通信 · 语义通信 · 多模态融合 · 智能体网络
Summary 本文提出了一种面向6G环境的会话型具身智能网络（CC-EIN），通过融合图像与雷达数据进行跨模态语义表示，并设计自适应语义通信策略和任务协调机制，提升多设备协作效率与决策可解释性。实验验证于灾后救援场景。
Reason 该论文属于具身智能与6G通信交叉领域，研究多模态感知、语义通信与设备协同，应用场景为地震救援等物理任务执行。完全不涉及文档图像理解（DIU）、DocVQA、OCR或任何形式的文本结构理解任务。属于‘无关领域’负面清单项，直接打0分。
Abstract In the 6G era, semantic collaboration among multiple embodied intelligent devices (MEIDs) becomes crucial for complex task execution. However, existing systems face challenges in multimodal information fusion, adaptive communication, and decision interpretability. To address these limitations, we propose a collaborative Conversational Embodied Intelligence Network (CC-EIN) integrating multimodal feature fusion, adaptive semantic communication, task coordination, and interpretability. PerceptiNet performs cross-modal fusion of image and radar data to generate unified semantic representations. An adaptive semantic communication strategy dynamically adjusts coding schemes and transmission power according to task urgency and channel quality. A semantic-driven collaboration mechanism further supports task decomposition and conflict-free coordination among heterogeneous devices. Finally, the InDec module enhances decision transparency through Grad-CAM visualization. Simulation results in post-earthquake rescue scenarios demonstrate that CC-EIN achieves 95.4% task completion rate and 95% transmission efficiency while maintaining strong semantic consistency and energy efficiency.
Authors Mingkai Chen, Zijie Feng, Lei Wang, Yaser Khamayseh
Comment 7 pages, 8 figures. Preprint submitted to IEEE Vehicle Technology Magazine
Categories Artificial Intelligence
0
MicroSims: A Framework for AI-Generated, Scalable Educational Simulations with Universal Embedding and Adaptive Learning Support
Title CN MicroSims：一种用于人工智能生成、可扩展教育模拟的框架，支持通用嵌入和自适应学习
Keywords 教育仿真 · AI生成内容 · 交互式学习 · iframe嵌入 · 自适应学习
Summary 本文提出了MicroSims框架，旨在通过AI自动生成可嵌入各类学习平台的轻量级交互式教学仿真，降低教育资源开发门槛。框架支持无编程定制和跨平台集成，并基于STEM教育研究验证其提升学习效果的能力。然而，该工作属于教育科技的应用层设计，与文档图像理解的技术演进无关。
Reason 该论文聚焦于教育技术领域，提出一个用于快速生成轻量级教学仿真的AI框架，属于特定领域的下游应用（Wrapper/Application）。其核心贡献在于教育内容的自动化生成与部署，未涉及文档图像理解中的OCR、布局分析、视觉表示学习或VLM架构改进等关键技术问题。尽管使用了AI生成技术，但并无针对文档模态的建模创新，也无法迁移解决DIU中的幻觉、密集文本或复杂排版等痛点。
Abstract Educational simulations have long been recognized as powerful tools for enhancing learning outcomes, yet their creation has traditionally required substantial resources and technical expertise. This paper introduces MicroSims a novel framework for creating lightweight, interactive educational simulations that can be rapidly generated using artificial intelligence, universally embedded across digital learning platforms, and easily customized without programming knowledge. MicroSims occupy a unique position at the intersection of three key innovations: (1) standardized design patterns that enable AI-assisted generation, (2) iframe-based architecture that provides universal embedding and sandboxed security, and (3) transparent, modifiable code that supports customization and pedagogical transparency. We present a comprehensive framework encompassing design principles, technical architecture, metadata standards, and development workflows. Drawing on empirical research from physics education studies and meta-analyses across STEM disciplines, we demonstrate that interactive simulations can improve conceptual understanding by up to 30-40\% compared to traditional instruction. MicroSims extend these benefits while addressing persistent barriers of cost, technical complexity, and platform dependence. This work has significant implications for educational equity, and low-cost intelligent interactive textbooks that enabling educators worldwide to create customized, curriculum-aligned simulations on demand. We discuss implementation considerations, present evidence of effectiveness, and outline future directions for AI-powered adaptive learning systems built on the MicroSim foundation.
Authors Valerie Lockhart, Dan McCreary, Troy A. Peterson
Comment 42 pages, 4 figures
Categories Artificial Intelligence
0
Denoising Refinement Diffusion Models for Simultaneous Generation of Multi-scale Mobile Network Traffic
Title CN 用于多尺度移动网络流量同步生成的去噪细化扩散模型
Keywords 扩散模型 · 多尺度生成 · 移动网络流量 · 时空预测 · 去噪细化
Summary 本文提出ZoomDiff，一种基于扩散模型的多尺度移动网络流量生成方法，通过定制化的去噪细化扩散模型（DRDM）在不同阶段生成具有不同时空分辨率的流量数据，以匹配基站、小区和网格等多层次网络结构。实验表明其在真实数据集上相比现有方法有显著提升，并具备良好的跨城市泛化能力。
Reason 该论文研究的是移动网络流量生成问题，属于网络规划与通信领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用完全无关。尽管使用了扩散模型和多尺度建模技术，但其方法针对的是时空流量数据，无法迁移到解决OCR幻觉、复杂排版理解、细粒度定位等DIU核心痛点。
Abstract The planning, management, and resource scheduling of cellular mobile networks require joint estimation of mobile traffic across different layers and nodes. Mobile traffic generation can proactively anticipate user demands and capture the dynamics of network load. However, existing methods mainly focus on generating traffic at a single spatiotemporal resolution, making it difficult to jointly model multi-scale traffic patterns. In this paper, we propose ZoomDiff, a diffusion-based model for multi-scale mobile traffic generation. ZoomDiff maps urban environmental context into mobile traffic with multiple spatial and temporal resolutions through a set of customized Denoising Refinement Diffusion Models (DRDM). DRDM employs a multi-stage noise-adding and denoising mechanism, enabling different stages to generate traffic at distinct spatiotemporal resolutions. This design aligns the progressive denoising process with hierarchical network layers, including base stations, cells, and grids of varying granularities. Experiments on real-world mobile traffic datasets show that ZoomDiff achieves at least an 18.4% improvement over state-of-the-art baselines in multi-scale traffic generation tasks. Moreover, ZoomDiff demonstrates strong efficiency and cross-city generalization, highlighting its potential as a powerful generative framework for modeling multi-scale mobile network dynamics.
Authors Xiaoqian Qi, Haoye Chai, Sichang Liu, Lei Yue, Raoyuan Pan, Yue Wang, Yong Li
Categories Artificial Intelligence
0
SVBRD-LLM: Self-Verifying Behavioral Rule Discovery for Autonomous Vehicle Identification
Title CN SVBRD-LLM：用于自动驾驶车辆识别的自验证行为规则发现
Keywords 自动驾驶识别 · 行为规则发现 · 零样本提示 · 轨迹分析 · GPT-5
Summary 本文提出SVBRD-LLM框架，利用YOLOv8和ByteTrack提取车辆轨迹，结合GPT-5零样本提示生成自动驾驶车辆的行为规则，并通过验证机制筛选高置信度规则。实验基于超过1500小时的真实交通视频，在自动驾驶车辆识别任务上取得90.0%准确率和93.3% F1分数。所发现的规则揭示了自动驾驶车辆在速度控制、变道保守性等方面的特点。
Reason 该论文属于具身智能与自动驾驶领域，研究从交通视频中通过大模型发现自动驾驶车辆的行为规则，完全偏离文档图像理解（DIU）的核心任务。其方法聚焦于轨迹提取、运动学特征与零样本提示在交通行为分析中的应用，不涉及OCR、布局分析、表格识别或任何文档结构理解问题。尽管使用了VLM进行规则生成，但应用场景与文档无关，且未提出可用于解决DIU痛点（如幻觉、密集文本、复杂排版）的通用方法论。
Abstract As more autonomous vehicles operate on public roads, understanding real-world behavior of autonomous vehicles is critical to analyzing traffic safety, making policies, and public acceptance. This paper proposes SVBRD-LLM, a framework that automatically discovers, verifies, and applies interpretable behavioral rules from real traffic videos through zero-shot prompt engineering. The framework extracts vehicle trajectories using YOLOv8 and ByteTrack, computes kinematic features, and employs GPT-5 zero-shot prompting to compare autonomous and human-driven vehicles, generating 35 structured behavioral rule hypotheses. These rules are tested on a validation set, iteratively refined based on failure cases to filter spurious correlations, and compiled into a high-confidence rule library. The framework is evaluated on an independent test set for speed change prediction, lane change prediction, and autonomous vehicle identification tasks. Experiments on over 1500 hours of real traffic videos show that the framework achieves 90.0% accuracy and 93.3% F1-score in autonomous vehicle identification. The discovered rules clearly reveal distinctive characteristics of autonomous vehicles in speed control smoothness, lane change conservativeness, and acceleration stability, with each rule accompanied by semantic description, applicable context, and validation confidence.
Authors Xiangyu Li, Zhaomiao Guo
Categories Artificial Intelligence
0
How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review
Title CN 如何找到出色的AI论文：自排名作为超越同行评审的科学影响力预测指标
Keywords 学术影响力 · 自排名 · 同行评审 · 引用预测 · 科研评价
Summary 本文通过一项大规模实验研究发现，作者对自己提交论文的质量自排名能够有效预测其未来的引用量，且表现优于传统同行评审得分。研究认为自排名可作为识别高影响力AI研究的补充手段。
Reason 该论文研究的是学术评价机制，探讨作者自评排名对论文未来影响力的预测能力，属于科研方法论与学术出版领域，与文档图像理解（DIU）、多模态大模型（VLM）或OCR等核心技术无关。未提出任何可用于解决DIU任务（如DocVQA、布局分析、表格识别）的方法论创新，也无法迁移为处理OCR幻觉、复杂排版或长文档推理的工具。
Abstract Peer review in academic research aims not only to ensure factual correctness but also to identify work of high scientific potential that can shape future research directions. This task is especially critical in fast-moving fields such as artificial intelligence (AI), yet it has become increasingly difficult given the rapid growth of submissions. In this paper, we investigate an underexplored measure for identifying high-impact research: authors' own rankings of their multiple submissions to the same AI conference. Grounded in game-theoretic reasoning, we hypothesize that self-rankings are informative because authors possess unique understanding of their work's conceptual depth and long-term promise. To test this hypothesis, we conducted a large-scale experiment at a leading AI conference, where 1,342 researchers self-ranked their 2,592 submissions by perceived quality. Tracking outcomes over more than a year, we found that papers ranked highest by their authors received twice as many citations as their lowest-ranked counterparts; self-rankings were especially effective at identifying highly cited papers (those with over 150 citations). Moreover, we showed that self-rankings outperformed peer review scores in predicting future citation counts. Our results remained robust after accounting for confounders such as preprint posting time and self-citations. Together, these findings demonstrate that authors' self-rankings provide a reliable and valuable complement to peer review for identifying and elevating high-impact research in AI.
Authors Buxin Su, Natalie Collina, Garrett Wen, Didong Li, Kyunghyun Cho, Jianqing Fan, Bingxin Zhao, Weijie Su
Categories Artificial Intelligence
0
GRAM: Generalization in Deep RL with a Robust Adaptation Module IEEE Robotics and Automation Letters (RA-L)
Title CN GRAM：具有鲁棒自适应模块的深度强化学习中的泛化
Keywords 深度强化学习 · 泛化能力 · 鲁棒自适应模块 · 分布外泛化 · 机器人控制
Summary 本文提出了一种名为GRAM的框架，用于提升深度强化学习在分布内和分布外环境动态下的泛化能力。通过引入一个鲁棒自适应模块，并结合联合训练流程，实现对不同环境变化的识别与响应。实验验证了该方法在四足机器人上的仿真与实际硬件任务中的有效性。
Reason 该论文属于具身智能与机器人领域，研究的是深度强化学习在动态环境下的泛化能力，核心方法围绕强化学习中的自适应控制和鲁棒性设计。其技术路线与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联，无法迁移解决OCR幻觉、复杂排版、细粒度定位等DIU核心问题。
Abstract The reliable deployment of deep reinforcement learning in real-world settings requires the ability to generalize across a variety of conditions, including both in-distribution scenarios seen during training as well as novel out-of-distribution scenarios. In this work, we present a framework for dynamics generalization in deep reinforcement learning that unifies these two distinct types of generalization within a single architecture. We introduce a robust adaptation module that provides a mechanism for identifying and reacting to both in-distribution and out-of-distribution environment dynamics, along with a joint training pipeline that combines the goals of in-distribution adaptation and out-of-distribution robustness. Our algorithm GRAM achieves strong generalization performance across in-distribution and out-of-distribution scenarios upon deployment, which we demonstrate through extensive simulation and hardware locomotion experiments on a quadruped robot.
Authors James Queeney, Xiaoyi Cai, Alexander Schperberg, Radu Corcodel, Mouhacine Benosman, Jonathan P. How
Comment Accepted for publication in IEEE Robotics and Automation Letters (RA-L)
Categories Artificial Intelligence
0
Continually Evolving Skill Knowledge in Vision Language Action Model
Title CN 视觉语言动作模型中持续演进的技能知识
Keywords 视觉语言动作模型 · 持续学习 · 机器人操控 · 知识演化 · 专家路由
Summary 本文提出Stellar VLA，一种基于知识驱动的视觉语言动作模型持续学习框架，包含T-Stellar和TS-Stellar两种变体，通过建模任务中心或层次化的任务-技能结构，实现无需大量标注的自监督知识演化。利用知识引导的专家路由机制，在不增加网络参数的情况下实现任务专业化，显著提升了在LIBERO基准和真实世界任务中的成功率。
Reason 该论文属于具身智能与机器人领域，研究的是视觉-语言-动作（VLA）模型在开放环境中进行持续技能学习的问题。其核心任务是机器人操控与行动推理，与文档图像理解（DIU）或DocVQA等任务无直接关联。尽管涉及多模态建模，但方法聚焦于动作序列的持续学习和专家路由机制，难以迁移到解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点。
Abstract Developing general robot intelligence in open environments requires continual skill learning. Recent Vision-Language-Action (VLA) models leverage massive pretraining data to support diverse manipulation tasks, but they still depend heavily on task-specific fine-tuning, revealing a lack of continual learning capability. Existing continual learning methods are also resource-intensive to scale to VLA models. We propose Stellar VLA, a knowledge-driven continual learning framework with two variants: T-Stellar, modeling task-centric knowledge space, and TS-Stellar, capturing hierarchical task-skill structure. Stellar VLA enables self-supervised knowledge evolution through joint learning of task latent representation and the knowledge space, reducing annotation needs. Knowledge-guided expert routing provide task specialization without extra network parameters, lowering training overhead. Experiments on the LIBERO benchmark and real-world tasks show over 50 percentage average improvement in final success rates relative to baselines. TS-Stellar further excels in complex action inference, and in-depth analyses verify effective knowledge retention and discovery. Our code will be released soon.
Authors Yuxuan Wu, Guangming Wang, Zhiheng Yang, Maoqing Yao, Brian Sheil, Hesheng Wang
Categories Artificial Intelligence
0
A Survey on Diffusion Models for Time Series and Spatio-Temporal Data ACM Computing Surveys
Title CN 关于扩散模型在时间序列和时空数据中的研究综述
Keywords 扩散模型 · 时间序列 · 时空数据 · 生成模型 · 综述
Summary 本文综述了扩散模型在时间序列和时空数据中的应用进展，涵盖模型分类、任务类型与实际应用场景，并建立了开源资源库。研究范围涉及多个非视觉模态的数据生成与分析，但未涉及任何文档图像、多模态语言模型或视觉信息处理相关内容。
Reason 该论文属于对扩散模型在时间序列与时空数据中应用的综述，内容集中在生成、推理与下游任务，涉及领域包括医疗、交通、气候等。完全不涉及文档图像理解（DIU）、视觉语言模型（VLM）或OCR相关技术，且为综述性文章而非方法创新。属于无关领域，命中负面清单第2条（视频/生成/非视觉文档方向），直接打0分。
Abstract Diffusion models have been widely used in time series and spatio-temporal data, enhancing generative, inferential, and downstream capabilities. These models are applied across diverse fields such as healthcare, recommendation, climate, energy, audio, and traffic. By separating applications for time series and spatio-temporal data, we offer a structured perspective on model category, task type, data modality, and practical application domain. This study aims to provide a solid foundation for researchers and practitioners, inspiring future innovations that tackle traditional challenges and foster novel solutions in diffusion model-based data mining tasks and applications. For more detailed information, we have open-sourced a repository at https://github.com/yyysjz1997/Awesome-TimeSeries-SpatioTemporal-Diffusion-Model.
Authors Yiyuan Yang, Ming Jin, Haomin Wen, Chaoli Zhang, Yuxuan Liang, Lintao Ma, Yi Wang, Chenghao Liu, Bin Yang, Zenglin Xu, Shirui Pan, Qingsong Wen
Comment Accepted by ACM Computing Surveys; 40 pages; Github Repo: https://github.com/yyysjz1997/Awesome-TimeSeries-SpatioTemporal-Diffusion-Model
Categories Artificial Intelligence
0
Cisco Time Series Model Technical Report
Title CN 思科时间序列模型技术报告
Keywords 时间序列预测 · 基础模型 · 多分辨率输入 · 零样本学习 · 解码器-only
Summary 本文提出了一种名为Cisco Time Series Model的单变量零样本时间序列预测基础模型，基于TimesFM架构改进，支持多分辨率输入，并在超过3000亿数据点上训练。模型在可观测性领域数据集上表现优异，同时在通用预测基准GIFT-Eval上保持竞争力。研究表明多分辨率结构有助于长上下文预测。
Reason 该论文研究的是时间序列预测模型，属于纯时序数据分析领域，与文档图像理解（DIU）或视觉-语言多模态任务无直接关联。尽管其提出的多分辨率输入架构可能在广义序列建模中有启发意义，但无法迁移解决OCR幻觉、复杂排版、细粒度定位等DIU核心问题。
Abstract We introduce the Cisco Time Series Model, a univariate zero-shot forecaster. This time series foundation model is the result of a general architectural innovation to a time series model enabling it to accept multiresolution input, applied to a popular decoder-only time series model (TimesFM). The resulting multiresolution decoder-only model is trained on over 300B unique data points, with more than half coming from the observability domain. Quantitative and qualitative evaluations demonstrate that the resulting model achieves superior performance on observability datasets while retaining very similar performance on a standard general-purpose forecasting benchmark (GIFT-Eval), and suggest that the multiresolution structure enables the model to make more accurate predictions on long context input.
Authors Liang Gou, Archit Khare, Praneet Pabolu, Prachi Patel, Joseph Ross, Hercy Shen, Yuhan, Song, Jingze Sun, Kristal Curtis, Vedant Dharnidharka, Abhinav Mathur, Hao Yang
Categories Artificial Intelligence
0
Simulating Macroeconomic Expectations using LLM Agents
Title CN 使用LLM Agent模拟宏观经济预期
Keywords LLM Agent · 宏观经济 · 行为模拟 · 提示工程 · AI社会科学
Summary 本文提出了一种利用LLM Agent模拟宏观经济预期的框架，通过构建具备不同功能模块的智能体来复现经济调查实验。尽管模拟结果比人类更同质化，但优于仅依赖提示工程的LLM，并展现出类人的心理机制。研究为AI行为科学提供了新视角，但完全局限于纯文本的经济仿真场景。
Reason 该论文属于平行下游应用，将LLM Agent应用于宏观经济预期模拟，属于特定领域的行为建模，未提出任何可用于文档图像理解（DIU）的方法论创新。其内容与文档理解中的OCR幻觉、复杂排版、细粒度定位或长文档推理等核心痛点无关，且未涉及视觉输入或多模态处理。
Abstract We introduce a novel framework for simulating macroeconomic expectations using LLM Agents. By constructing LLM Agents equipped with various functional modules, we replicate three representative survey experiments involving several expectations across different types of economic agents. Our results show that although the expectations simulated by LLM Agents are more homogeneous than those of humans, they consistently outperform LLMs relying simply on prompt engineering, and possess human-like mental mechanisms. Evaluation reveals that these capabilities stem from the contributions of their components, offering guidelines for their architectural design. Our approach complements traditional methods and provides new insights into AI behavioral science in macroeconomic research
Authors Jianhao Lin, Lexuan Sun, Yixin Yan
Categories Artificial Intelligence
0
Attention Pruning: Automated Fairness Repair of Language Models via Surrogate Simulated Annealing
Title CN 注意力剪枝：通过代理模拟退火实现语言模型的自动化公平性修复
Keywords 注意力剪枝 · 公平性 · 语言模型 · 模拟退火 · 代理模型
Summary 本文提出一种基于代理模拟退火的注意力头剪枝方法，用于在不显著影响模型性能的前提下减少大语言模型中的性别等社会偏见。通过构建代理模型来预测不同注意力头状态对公平性和效用的影响，从而高效搜索最优剪枝方案。实验表明该方法可将性别偏见降低40%，优于现有去偏策略。
Reason 该论文属于纯NLP领域的模型公平性研究，聚焦于通过剪枝注意力头来缓解语言模型中的社会偏见。尽管其方法涉及模型干预技术，但研究目标与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）无直接关联，且未涉及视觉模态或跨模态建模。其提出的技术难以迁移到多模态文档理解场景中解决实际问题，属于典型的非相关领域工作。
Abstract This paper explores pruning attention heads as a post-processing bias mitigation method for large language models (LLMs). Modern AI systems such as LLMs are expanding into sensitive social contexts where fairness concerns become especially crucial. Since LLMs develop decision-making patterns by training on massive datasets of human-generated content, they naturally encode and perpetuate societal biases. While modifying training datasets and algorithms is expensive and requires significant resources; post-processing techniques-such as selectively deactivating neurons and attention heads in pre-trained LLMs-can provide feasible and effective approaches to improve fairness. However, identifying the optimal subset of parameters to prune presents a combinatorial challenge within LLMs' immense parameter space, requiring solutions that efficiently balance competing objectives across the frontiers of model fairness and utility. To address the computational challenges, we explore a search-based program repair approach via randomized simulated annealing. Given the prohibitive evaluation costs in billion-parameter LLMs, we develop surrogate deep neural networks that efficiently model the relationship between attention head states (active/inactive) and their corresponding fairness/utility metrics. This allows us to perform optimization over the surrogate models and efficiently identify optimal subsets of attention heads for selective pruning rather than directly searching through the LLM parameter space. This paper introduces Attention Pruning, a fairness-aware surrogate simulated annealing approach to prune attention heads in LLMs that disproportionately contribute to bias while minimally impacting overall model utility. Our experiments show that Attention Pruning achieves up to $40\%$ reduction in gender bias and outperforms the state-of-the-art bias mitigation strategies.
Authors Vishnu Asutosh Dasu, Md Rafi ur Rashid, Vipul Gupta, Saeid Tizpaz-Niari, Gang Tan
Categories Artificial Intelligence
0
GED-Consistent Disentanglement of Aligned and Unaligned Substructures for Graph Similarity Learning
Title CN 用于图相似性学习的对齐与未对齐子结构的GED一致性解耦
Keywords 图相似性计算 · 图编辑距离 · GNN · 子结构解耦 · 图匹配
Summary 本文提出GCGSim，一种基于图编辑距离（GED）一致性的图相似性学习框架，旨在克服现有GNN方法在节点级匹配中无法准确反映全局结构对应关系和编辑成本分配的问题。通过引入图级匹配机制和子结构级编辑成本建模，实现对齐与未对齐子结构的解耦表示学习，在四个基准数据集上实现了SOTA性能。
Reason 该论文研究图相似性计算中的图编辑距离（GED）近似问题，属于图神经网络和图匹配领域，与文档图像理解（DIU）、DocVQA、OCR或视觉语言模型在文档上的应用无直接关联。其方法聚焦于图结构的全局匹配与子结构解耦，并不涉及多模态输入、视觉文本布局建模或文档推理等DIU核心问题，也无法迁移用于解决OCR幻觉、复杂排版解析或长文档理解等痛点。
Abstract Graph Similarity Computation (GSC) is a fundamental graph related task where Graph Edit Distance (GED) serves as a prevalent metric. GED is determined by an optimal alignment between a pair of graphs that partitions each into aligned (zero-cost) and unaligned (cost-incurring) substructures. Due to NP-hard nature of exact GED computation, GED approximations based on Graph Neural Network(GNN) have emerged. Existing GNN-based GED approaches typically learn node embeddings for each graph and then aggregate pairwise node similarities to estimate the final similarity. Despite their effectiveness, we identify a mismatch between this prevalent node-centric matching paradigm and the core principles of GED. This discrepancy leads to two critical limitations: (1) a failure to capture the global structural correspondence for optimal alignment, and (2) a misattribution of edit costs driven by spurious node level signals. To address these limitations, we propose GCGSim, a GED-consistent graph similarity learning framework centering on graph-level matching and substructure-level edit costs. Specifically, we make three core technical contributions. Extensive experiments on four benchmark datasets show that GCGSim achieves state-of-the-art performance. Our comprehensive analyses further validate that the framework effectively learns disentangled and semantically meaningful substructure representations.
Authors Zhentao Zhan, Xiaoliang Xu, Jingjing Wang, Junmei Wang
Categories Artificial Intelligence
0
Rectified SpaAttn: Revisiting Attention Sparsity for Efficient Video Generation
Title CN 修正的稀疏注意力：重新审视视频生成中的注意力稀疏性
Keywords 视频生成 · 稀疏注意力 · 扩散Transformer · 效率优化 · 注意力机制
Summary 本文针对扩散Transformer在视频生成中注意力计算复杂度高的问题，提出了一种名为Rectified SpaAttn的稀疏注意力方法，通过引入隐式的全注意力参考来修正稀疏注意力分配中的系统性偏差，并设计了两种机制分别优化关键和非关键token的权重分配，在保持生成质量的同时显著提升了推理速度。
Reason 该论文聚焦于视频生成任务中的注意力机制优化，属于纯视频生成领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。尽管其提出的稀疏注意力修正确实具有计算效率上的创新，但应用场景为视频扩散模型，迁移至文档理解场景（如处理复杂排版或OCR幻觉）极为困难，且未涉及多模态文档特有的挑战。
Abstract Diffusion Transformers dominate video generation, but the quadratic complexity of attention computation introduces substantial latency. Attention sparsity reduces computational costs by focusing on critical tokens while ignoring non-critical tokens. However, existing methods suffer from severe performance degradation. In this paper, we revisit attention sparsity and reveal that existing methods induce systematic biases in attention allocation: (1) excessive focus on critical tokens amplifies their attention weights; (2) complete neglect of non-critical tokens causes the loss of relevant attention weights. To address these issues, we propose Rectified SpaAttn, which rectifies attention allocation with implicit full attention reference, thereby enhancing the alignment between sparse and full attention maps. Specifically: (1) for critical tokens, we show that their bias is proportional to the sparse attention weights, with the ratio governed by the amplified weights. Accordingly, we propose Isolated-Pooling Attention Reallocation, which calculates accurate rectification factors by reallocating multimodal pooled weights. (2) for non-critical tokens, recovering attention weights from the pooled query-key yields attention gains but also introduces pooling errors. Therefore, we propose Gain-Aware Pooling Rectification, which ensures that the rectified gain consistently surpasses the induced error. Moreover, we customize and integrate the Rectified SpaAttn kernel using Triton, achieving up to 3.33 and 2.08 times speedups on HunyuanVideo and Wan 2.1, respectively, while maintaining high generation quality. We release Rectified SpaAttn as open-source at https://github.com/BienLuky/Rectified-SpaAttn .
Authors Xuewen Liu, Zhikai Li, Jing Zhang, Mengjuan Chen, Qingyi Gu
Comment Code at https://github.com/BienLuky/Rectified-SpaAttn
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Language-Independent Sentiment Labelling with Distant Supervision: A Case Study for English, Sepedi and Setswana The Fourth Workshop on Processing Emotions, Decisions and Opinions (EDO 2023) at 10th Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics (LTC 2023)
Title CN 使用远程监督进行语言无关的情感标注：以英语、塞佩迪语和塞茨瓦纳语为例
Keywords 情感分析 · 远程监督 · 小语种 · 低资源语言 · 自动化标注
Summary 本文提出一种语言无关的自动情感标注方法，利用表情符号和情感词对英语、塞佩迪语和塞茨瓦纳语的推文进行远程监督标注，在SAfriSenti多语言语料库上实现了60%以上的平均准确率，显著减少人工标注成本。
Reason 该论文属于小语种（Sepedi和Setswana）情感分析任务，且为特定低资源语言的下游应用研究，未提出可迁移至文档图像理解（DIU）领域的底层方法或架构创新。其技术路线基于远程监督和表情符号匹配，与DIU核心痛点（如OCR幻觉、复杂排版、细粒度定位等）无直接关联。
Abstract Sentiment analysis is a helpful task to automatically analyse opinions and emotions on various topics in areas such as AI for Social Good, AI in Education or marketing. While many of the sentiment analysis systems are developed for English, many African languages are classified as low-resource languages due to the lack of digital language resources like text labelled with corresponding sentiment classes. One reason for that is that manually labelling text data is time-consuming and expensive. Consequently, automatic and rapid processes are needed to reduce the manual effort as much as possible making the labelling process as efficient as possible. In this paper, we present and analyze an automatic language-independent sentiment labelling method that leverages information from sentiment-bearing emojis and words. Our experiments are conducted with tweets in the languages English, Sepedi and Setswana from SAfriSenti, a multilingual sentiment corpus for South African languages. We show that our sentiment labelling approach is able to label the English tweets with an accuracy of 66%, the Sepedi tweets with 69%, and the Setswana tweets with 63%, so that on average only 34% of the automatically generated labels remain to be corrected.
Authors Koena Ronny Mabokela, Tim Schlippe, Mpho Raborife, Turgay Celik
Comment Published in the The Fourth Workshop on Processing Emotions, Decisions and Opinions (EDO 2023) at 10th Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics (LTC 2023), Poznań, Poland, 21-23 April 2023. ISBN: 978-83-232-4176-8
Categories Artificial Intelligence, Computation and Language
0
AI-in-the-Loop: Privacy Preserving Real-Time Scam Detection and Conversational Scambaiting by Leveraging LLMs and Federated Learning 26th Privacy Enhancing Technologies Symposium (PETS 2026)
Title CN AI-in-the-Loop：通过利用大语言模型和联邦学习实现隐私保护的实时诈骗检测与对话式反诈诱捕
Keywords 联邦学习 · 大语言模型 · 隐私保护 · 实时诈骗检测 · 内容安全
Summary 本文提出了一种结合指令微调AI与安全感知效用函数的AI-in-the-loop框架，用于在实时对话中主动检测并干扰诈骗行为，同时采用联邦学习保障数据隐私。实验表明系统在保持低困惑度和高参与度的同时，能有效平衡安全性与交互真实性，并在联邦设置下维持性能稳定。研究还评估了不同内容审核模型对隐私泄露的控制效果。
Reason 该论文属于典型的平行下游应用，其核心是将LLM与联邦学习应用于反诈骗对话场景，尽管涉及多模态交互，但并未提出任何可用于文档图像理解（DIU）的新架构、新训练方法或底层技术。任务聚焦于社交工程攻击防御与隐私保护，与DocVQA、OCR、版面分析等DIU核心任务无直接关联，且未提供可迁移至文档理解领域的通用方法论。
Abstract Scams exploiting real-time social engineering -- such as phishing, impersonation, and phone fraud -- remain a persistent and evolving threat across digital platforms. Existing defenses are largely reactive, offering limited protection during active interactions. We propose a privacy-preserving, AI-in-the-loop framework that proactively detects and disrupts scam conversations in real time. The system combines instruction-tuned artificial intelligence with a safety-aware utility function that balances engagement with harm minimization, and employs federated learning to enable continual model updates without raw data sharing. Experimental evaluations show that the system produces fluent and engaging responses (perplexity as low as 22.3, engagement $\approx$0.80), while human studies confirm significant gains in realism, safety, and effectiveness over strong baselines. In federated settings, models trained with FedAvg sustain up to 30 rounds while preserving high engagement ($\approx$0.80), strong relevance ($\approx$0.74), and low PII leakage ($\leq$0.0085). Even with differential privacy, novelty and safety remain stable, indicating that robust privacy can be achieved without sacrificing performance. The evaluation of guard models (LlamaGuard, LlamaGuard2/3, MD-Judge) shows a straightforward pattern: stricter moderation settings reduce the chance of exposing personal information, but they also limit how much the model engages in conversation. In contrast, more relaxed settings allow longer and richer interactions, which improve scam detection, but at the cost of higher privacy risk. To our knowledge, this is the first framework to unify real-time scam-baiting, federated privacy preservation, and calibrated safety moderation into a proactive defense paradigm.
Authors Ismail Hossain, Sai Puppala, Md Jahangir Alam, Sajedul Talukder
Comment This paper got accepted in 26th Privacy Enhancing Technologies Symposium (PETS 2026). We uploaded it into ArXiv as pre-print
Categories Artificial Intelligence
0
Special-Character Adversarial Attacks on Open-Source Language Model
Title CN 开源语言模型的特殊字符对抗攻击
Keywords 对抗攻击 · 大语言模型 · 安全 · Jailbreak · 字符混淆
Summary 本文研究了基于特殊字符（包括Unicode、同形异义字、结构混淆等）的对抗攻击对开源大语言模型的影响，在超过4000次攻击尝试中测试了7个不同规模的模型，揭示了各类模型在安全机制上的普遍漏洞，可导致成功越狱、输出混乱或无关内容。
Reason 该论文研究的是针对大语言模型的字符级对抗攻击（如Unicode、同形异义字等），属于NLP安全与对齐领域，主要关注Jailbreak和安全性问题。虽然提到了‘幻觉’，但其上下文是安全绕过导致的无关输出，而非文档图像理解中的视觉或OCR相关幻觉。研究重点不在多模态、文档结构、视觉布局或图像文本联合理解，无法迁移到DIU的核心痛点（如复杂排版、高分辨率处理、细粒度定位）。因此属于无关领域。
Abstract Large language models (LLMs) have achieved remarkable performance across diverse natural language processing tasks, yet their vulnerability to character-level adversarial manipulations presents significant security challenges for real-world deployments. This paper presents a study of different special character attacks including unicode, homoglyph, structural, and textual encoding attacks aimed at bypassing safety mechanisms. We evaluate seven prominent open-source models ranging from 3.8B to 32B parameters on 4,000+ attack attempts. These experiments reveal critical vulnerabilities across all model sizes, exposing failure modes that include successful jailbreaks, incoherent outputs, and unrelated hallucinations.
Authors Ephraiem Sarabamoun
Categories Artificial Intelligence
0
Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment
Title CN 探索强化学习中时间步长在脓毒症治疗中的影响
Keywords 强化学习 · 医疗AI · 时间步长 · 离线RL · 脓毒症
Summary 本文探讨了在脓毒症管理的强化学习中不同时间步长（1、2、4、8小时）对策略学习性能的影响，发现更细粒度的时间步长能带来更优且稳定的策略表现。研究聚焦于医疗决策中的离线RL设置，提出动作重映射方法以实现跨时间步长的公平比较。
Reason 该论文研究的是医疗领域（脓毒症治疗）中的强化学习问题，关注时间步长对策略学习的影响。尽管涉及序列决策和状态表示学习，但其方法完全针对临床时序数据建模，并未涉及任何文档图像、多模态理解、视觉表示或语言-视觉对齐等DIU相关技术。无法迁移用于解决OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Existing studies on reinforcement learning (RL) for sepsis management have mostly followed an established problem setup, in which patient data are aggregated into 4-hour time steps. Although concerns have been raised regarding the coarseness of this time-step size, which might distort patient dynamics and lead to suboptimal treatment policies, the extent to which this is a problem in practice remains unexplored. In this work, we conducted empirical experiments for a controlled comparison of four time-step sizes ($Δt\!=\!1,2,4,8$ h) on this domain, following an identical offline RL pipeline. To enable a fair comparison across time-step sizes, we designed action re-mapping methods that allow for evaluation of policies on datasets with different time-step sizes, and conducted cross-$Δt$ model selections under two policy learning setups. Our goal was to quantify how time-step size influences state representation learning, behavior cloning, policy training, and off-policy evaluation. Our results show that performance trends across $Δt$ vary as learning setups change, while policies learned at finer time-step sizes ($Δt = 1$ h and $2$ h) using a static behavior policy achieve the overall best performance and stability. Our work highlights time-step size as a core design choice in offline RL for healthcare and provides evidence supporting alternatives beyond the conventional 4-hour setup.
Authors Yingchuan Sun, Shengpu Tang
Categories Artificial Intelligence
0
Co-PatcheR: Collaborative Software Patching with Component(s)-specific Small Reasoning Models
Title CN Co-PatcheR：基于组件专用小型推理模型的协同软件补丁修复
Keywords 软件补丁 · 专用模型 · 任务分解 · 协同推理 · 代码生成
Summary 本文提出Co-PatcheR，一种基于多个专用小模型协同工作的软件补丁生成系统，分别负责缺陷定位、补丁生成和验证任务。通过任务分解与两阶段验证机制，在SWE-bench-Verified上达到46%的问题解决率，优于现有专用模型方法。系统采用3个14B规模模型，资源消耗较低。
Reason 该论文研究的是软件工程领域的自动化补丁生成问题，属于‘平行下游应用’，尽管使用了多模型协作机制，但其方法论针对的是代码修复流程中的子任务分工，并未提出可用于文档图像理解（如OCR去幻觉、复杂排版建模、细粒度定位等）的通用底层技术。虽然其‘分治式Agent协作’思路有一定启发性，但整体仍局限于特定垂域（SWE-bench）的应用架构设计，无法直接迁移至DIU领域解决核心痛点。
Abstract Motivated by the success of general-purpose large language models (LLMs) in software patching, recent works started to train specialized patching models. Most works trained one model to handle the end-to-end patching pipeline (including issue localization, patch generation, and patch validation). However, it is hard for a small model to handle all tasks, as different sub-tasks have different workflows and require different expertise. As such, by using a 70 billion model, SOTA methods can only reach up to 41% resolved rate on SWE-bench-Verified. Motivated by the collaborative nature, we propose Co-PatcheR, the first collaborative patching system with small and specialized reasoning models for individual components. Our key technique novelties are the specific task designs and training recipes. First, we train a model for localization and patch generation. Our localization pinpoints the suspicious lines through a two-step procedure, and our generation combines patch generation and critique. We then propose a hybrid patch validation that includes two models for crafting issue-reproducing test cases with and without assertions and judging patch correctness, followed by a majority vote-based patch selection. Through extensive evaluation, we show that Co-PatcheR achieves 46% resolved rate on SWE-bench-Verified with only 3 x 14B models. This makes Co-PatcheR the best patcher with specialized models, requiring the least training resources and the smallest models. We conduct a comprehensive ablation study to validate our recipes, as well as our choice of training data number, model size, and testing-phase scaling strategy.
Authors Yuheng Tang, Hongwei Li, Kaijie Zhu, Michael Yang, Yangruibo Ding, Wenbo Guo
Categories Artificial Intelligence
0
Evolved SampleWeights for Bias Mitigation: Effectiveness Depends on Optimization Objectives
Title CN 通过进化样本权重缓解偏差：效果取决于优化目标
Keywords 偏置缓解 · 样本加权 · 遗传算法 · 公平性 · 机器学习
Summary 本文提出使用遗传算法进化训练样本的权重，以在模型训练过程中缓解偏见。通过在11个公开数据集上的实验表明，进化的样本权重能在特定优化目标下（如准确率与人口统计差异）取得比其他加权方法更好的公平性-性能权衡。然而，其效果高度依赖于所选择的优化目标。
Reason 该论文研究的是通用机器学习中的偏差缓解问题，核心方法是使用遗传算法进化样本权重以平衡公平性与预测性能。其任务和方法均不涉及文档图像理解（DIU）或视觉-语言模型（VLM）相关的技术痛点，如OCR幻觉、复杂排版处理、细粒度定位等。虽然‘reweighting’在训练策略层面有一定通用性，但该工作未针对多模态、视觉输入或文档结构建模进行设计，无法直接迁移用于解决DIU的核心挑战。
Abstract Machine learning models trained on real-world data may inadvertently make biased predictions that negatively impact marginalized communities. Reweighting is a method that can mitigate such bias in model predictions by assigning a weight to each data point used during model training. In this paper, we compare three methods for generating these weights: (1) evolving them using a Genetic Algorithm (GA), (2) computing them using only dataset characteristics, and (3) assigning equal weights to all data points. Model performance under each strategy was evaluated using paired predictive and fairness metrics, which also served as optimization objectives for the GA during evolution. Specifically, we used two predictive metrics (accuracy and area under the Receiver Operating Characteristic curve) and two fairness metrics (demographic parity difference and subgroup false negative fairness). Using experiments on eleven publicly available datasets (including two medical datasets), we show that evolved sample weights can produce models that achieve better trade-offs between fairness and predictive performance than alternative weighting methods. However, the magnitude of these benefits depends strongly on the choice of optimization objectives. Our experiments reveal that optimizing with accuracy and demographic parity difference metrics yields the largest number of datasets for which evolved weights are significantly better than other weighting strategies in optimizing both objectives.
Authors Anil K. Saini, Jose Guadalupe Hernandez, Emily F. Wong, Debanshi Misra, Jason H. Moore
Categories Artificial Intelligence
0
A Taxonomy of Pix Fraud in Brazil: Attack Methodologies, AI-Driven Amplification, and Defensive Strategies
Title CN 巴西Pix欺诈的分类：攻击方法、人工智能驱动的放大与防御策略
Keywords Pix支付 · 金融欺诈 · 社会工程学 · 网络安全 · 防御策略
Summary 本文对巴西中央银行推出的即时支付系统Pix的欺诈攻击方法进行了综述，结合文献分析与银行业专业人士访谈，将欺诈手段从单纯的社会工程学演进为融合人为操纵与技术利用的混合策略，并强调需要发展自适应防御机制和提升用户意识。
Reason 该论文研究的是支付系统Pix的欺诈行为及其防御，属于金融安全领域的应用型研究，不涉及文档图像理解（DIU）或视觉语言模型（VLM）的技术创新。尽管提到了AI驱动的放大，但未涉及OCR、文档布局分析、表格识别等DIU核心任务，也未提出可迁移至文档理解领域的底层方法。
Abstract This work presents a review of attack methodologies targeting Pix, the instant payment system launched by the Central Bank of Brazil in 2020. The study aims to identify and classify the main types of fraud affecting users and financial institutions, highlighting the evolution and increasing sophistication of these techniques. The methodology combines a structured literature review with exploratory interviews conducted with professionals from the banking sector. The results show that fraud schemes have evolved from purely social engineering approaches to hybrid strategies that integrate human manipulation with technical exploitation. The study concludes that security measures must advance at the same pace as the growing complexity of attack methodologies, with particular emphasis on adaptive defenses and continuous user awareness.
Authors Glener Lanes Pizzolato, Brenda Medeiros Lopes, Claudio Schepke, Diego Kreutz
Comment 5 pages, 1 figure, 2 tables, submitted to ERRC/WRSeg 2025
Categories Artificial Intelligence
0
Test-Time Alignment of Text-to-Image Diffusion Models via Null-Text Embedding Optimisation
Title CN 通过无文本嵌入优化实现文本到图像扩散模型的测试时对齐
Keywords 扩散模型 · 测试时对齐 · 无文本嵌入 · 生成分布 · 语义优化
Summary 本文提出Null-TTA方法，通过优化分类器自由引导中的无条件文本嵌入来实现扩散模型在测试时的对齐，确保在语义流形上进行调整，避免非语义噪声模式被利用。该方法不更新模型参数，仅在推理阶段调整生成分布，在图像生成任务中表现优越。
Reason 该论文研究的是文本到图像生成模型在测试时的对齐问题，属于纯图像生成领域，其核心任务是控制生成图像的质量与目标奖励的一致性。尽管提到了语义空间优化和防止‘奖励黑客’，但其方法应用于生成式建模而非文档理解或视觉语言推理。无法迁移到文档图像理解中的OCR幻觉、细粒度定位或复杂排版解析等核心痛点。
Abstract Test-time alignment (TTA) aims to adapt models to specific rewards during inference. However, existing methods tend to either under-optimise or over-optimise (reward hack) the target reward function. We propose Null-Text Test-Time Alignment (Null-TTA), which aligns diffusion models by optimising the unconditional embedding in classifier-free guidance, rather than manipulating latent or noise variables. Due to the structured semantic nature of the text embedding space, this ensures alignment occurs on a semantically coherent manifold and prevents reward hacking (exploiting non-semantic noise patterns to improve the reward). Since the unconditional embedding in classifier-free guidance serves as the anchor for the model's generative distribution, Null-TTA directly steers model's generative distribution towards the target reward rather than just adjusting the samples, even without updating model parameters. Thanks to these desirable properties, we show that Null-TTA achieves state-of-the-art target test-time alignment while maintaining strong cross-reward generalisation. This establishes semantic-space optimisation as an effective and principled novel paradigm for TTA.
Authors Taehoon Kim, Henry Gouk, Timothy Hospedales
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learning
Title CN 通过对比强化学习改进CUDA优化
Keywords CUDA优化 · 强化学习 · 对比学习 · 自动优化 · GPU加速
Summary 本文提出CUDA-L1，一种基于对比强化学习的自动化CUDA优化框架，能在无须人类专家干预的情况下显著提升CUDA内核执行效率，在多种基准测试中实现数倍至百倍的速度提升，并展现出跨GPU架构的可移植性。
Reason 该论文研究的是CUDA代码的自动化性能优化，属于系统与编译领域，与文档图像理解（DIU）或视觉语言模型（VLM）在任务、方法和应用场景上均无直接关联。尽管使用了强化学习框架，但其技术路线针对的是GPU内核调度与并行计算优化，无法迁移解决DIU中的OCR幻觉、复杂排版解析、细粒度视觉定位或长文档推理等核心问题。
Abstract The exponential growth in demand for GPU computing resources has created an urgent need for automated CUDA optimization strategies. While recent advances in LLMs show promise for code generation, current SOTA models achieve low success rates in improving CUDA speed. In this paper, we introduce CUDA-L1, an automated reinforcement learning framework for CUDA optimization that employs a novel contrastive RL algorithm. CUDA-L1 achieves significant performance improvements on the CUDA optimization task: trained on A100, it delivers an average speedup of x3.12 with a median speedup of x1.42 against default baselines over across all 250 CUDA kernels of KernelBench, with peak speedups reaching x120. In addition to the default baseline provided by KernelBench, CUDA-L1 demonstrates x2.77 over Torch Compile, x2.88 over Torch Compile with reduce overhead, x2.81 over CUDA Graph implementations, and remarkably x7.72 over cuDNN libraries. Furthermore, the model also demonstrates portability across different GPU architectures. Beyond these benchmark results, CUDA-L1 demonstrates several properties: it 1) discovers a variety of CUDA optimization techniques and learns to combine them strategically to achieve optimal performance; 2) uncovers fundamental principles of CUDA optimization, such as the multiplicative nature of optimizations; 3) identifies non-obvious performance bottlenecks and rejects seemingly beneficial optimizations that actually harm performance. The capabilities demonstrate that, RL can transform an initially poor-performing LLM into an effective CUDA optimizer through speedup-based reward signals alone, without human expertise or domain knowledge. This paradigm opens possibilities for automated optimization of CUDA operations, and holds promise to substantially promote GPU efficiency and alleviate the rising pressure on GPU computing resources.
Authors Xiaoya Li, Xiaofei Sun, Albert Wang, Jiwei Li, Chris Shum
Comment Project Page: https://deepreinforce-ai.github.io/cudal1_blog/
Categories Artificial Intelligence
0
Computing Evolutionarily Stable Strategies in Multiplayer Games
Title CN 计算多人博弈中的进化稳定策略
Keywords 博弈论 · 进化稳定策略 · 多智能体系统 · 算法 · 正常形式博弈
Summary 本文提出了一种用于计算三人及以上非退化正常形式博弈中所有进化稳定策略的算法。重点在于博弈策略的稳定性分析，属于理论性较强的AI基础研究，不涉及视觉、文本布局或文档理解任务。
Reason 该论文属于博弈论与人工智能理论交叉领域，研究的是多人正常形式博弈中进化稳定策略的计算算法，与文档图像理解（DIU）、视觉语言模型（VLM）或多模态文档处理无直接关联。无法迁移用于解决OCR幻觉、复杂排版、长文档推理等DIU核心问题。
Abstract We present an algorithm for computing all evolutionarily stable strategies in nondegenerate normal-form games with three or more players.
Authors Sam Ganzfried
Categories Artificial Intelligence
0
Unsupervised Memorability Modeling from Tip-of-the-Tongue Retrieval Queries WACV 2026
Title CN 从舌尖现象检索查询中进行无监督的记忆性建模
Keywords 视觉记忆性 · 无监督学习 · 舌尖现象 · 回忆生成 · 多模态检索
Summary 本文提出了一种基于‘舌尖现象’检索查询的无监督视觉记忆性建模方法，构建了包含8.2万视频的大规模数据集，用于生成开放式的记忆描述和实现多模态舌尖检索。通过微调大模型在该数据上取得了优于GPT-4o的表现，并采用对比学习实现ToT检索。但研究目标聚焦于人类记忆行为建模，与文档理解任务无关。
Reason 该论文研究的是视觉内容的记忆性（memorability）建模，使用Reddit等平台的‘舌尖现象’（ToT）查询数据构建无监督数据集，应用于视频内容的回忆生成与检索任务。其核心问题与文档图像理解（DIU）无关，既不涉及OCR、布局分析、表格识别、VIE等DIU核心任务，也未提出可用于解决文档理解中幻觉、细粒度定位或复杂排版处理的通用方法。属于认知科学与视觉内容记忆交叉领域，对DIU无直接或可迁移的技术价值。
Abstract Visual content memorability has intrigued the scientific community for decades, with applications ranging widely, from understanding nuanced aspects of human memory to enhancing content design. A significant challenge in progressing the field lies in the expensive process of collecting memorability annotations from humans. This limits the diversity and scalability of datasets for modeling visual content memorability. Most existing datasets are limited to collecting aggregate memorability scores for visual content, not capturing the nuanced memorability signals present in natural, open-ended recall descriptions. In this work, we introduce the first large-scale unsupervised dataset designed explicitly for modeling visual memorability signals, containing over 82,000 videos, accompanied by descriptive recall data. We leverage tip-of-the-tongue (ToT) retrieval queries from online platforms such as Reddit. We demonstrate that our unsupervised dataset provides rich signals for two memorability-related tasks: recall generation and ToT retrieval. Large vision-language models fine-tuned on our dataset outperform state-of-the-art models such as GPT-4o in generating open-ended memorability descriptions for visual content. We also employ a contrastive training strategy to create the first model capable of performing multimodal ToT retrieval. Our dataset and models present a novel direction, facilitating progress in visual content memorability research.
Authors Sree Bhattacharyya, Yaman Kumar Singla, Sudhir Yarram, Somesh Kumar Singh, Harini S, James Z. Wang
Comment Accepted at WACV 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
0
Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challenges
Title CN 大语言模型与认知科学：相似性、差异性与挑战的全面综述
Keywords 大语言模型 · 认知科学 · 认知偏差 · 人工智能 · 认知建模
Summary 本文是一篇关于大语言模型与认知科学交叉研究的综述，探讨了LLMs在认知建模中的潜力、评估方法、认知偏差及其与认知架构的融合。重点在于理论层面的比较与反思，未提出可用于文档图像理解的技术工具或架构改进。
Reason 该论文属于纯NLP与认知科学交叉的综述工作，未涉及任何视觉模态或文档理解任务。内容聚焦于LLM与人类认知的对比分析，不解决OCR幻觉、细粒度定位、复杂排版等DIU核心问题，且无方法论创新可迁移至文档图像理解领域。
Abstract This comprehensive review explores the intersection of Large Language Models (LLMs) and cognitive science, examining similarities and differences between LLMs and human cognitive processes. We analyze methods for evaluating LLMs cognitive abilities and discuss their potential as cognitive models. The review covers applications of LLMs in various cognitive fields, highlighting insights gained for cognitive science research. We assess cognitive biases and limitations of LLMs, along with proposed methods for improving their performance. The integration of LLMs with cognitive architectures is examined, revealing promising avenues for enhancing artificial intelligence (AI) capabilities. Key challenges and future research directions are identified, emphasizing the need for continued refinement of LLMs to better align with human cognition. This review provides a balanced perspective on the current state and future potential of LLMs in advancing our understanding of both artificial and human intelligence.
Authors Qian Niu, Junyu Liu, Ziqian Bi, Pohsun Feng, Benji Peng, Keyu Chen, Ming Li, Lawrence KQ Yan, Yichao Zhang, Caitlyn Heqi Yin, Cheng Fei, Tianyang Wang, Yunze Wang, Silin Chen, Ming Liu, Ziyuan Qin, Riyang Bao, Xinyuan Song, Zekun Jiang
Comment 10 pages, 1 figure
Categories Artificial Intelligence, Computation and Language
0
MODEST: Multi-Optics Depth-of-Field Stereo Dataset
Title CN MODEST：多光学景深立体数据集
Keywords 深度估计 · 立体视觉 · 景深 · DSLR数据集 · 3D重建
Summary 本文提出了首个高分辨率的立体DSLR数据集MODEST，包含18000张图像，系统性地变化焦距和光圈，覆盖多种真实场景下的光学条件。数据集旨在提升单目和立体深度估计、浅景深渲染、去模糊、3D重建等任务在真实世界中的泛化能力。然而，这些目标与文档图像理解无关。
Reason 该论文聚焦于立体视觉与深度估计，属于3D视觉和自动驾驶相关领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管其高分辨率图像和真实光学条件的数据采集具有一定技术价值，但核心任务（如深度估计、景深渲染、3D重建）无法迁移到解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点。
Abstract Reliable depth estimation under real optical conditions remains a core challenge for camera vision in systems such as autonomous robotics and augmented reality. Despite recent progress in depth estimation and depth-of-field rendering, research remains constrained by the lack of large-scale, high-fidelity, real stereo DSLR datasets, limiting real-world generalization and evaluation of models trained on synthetic data as shown extensively in literature. We present the first high-resolution (5472$\times$3648px) stereo DSLR dataset with 18000 images, systematically varying focal length and aperture across complex real scenes and capturing the optical realism and complexity of professional camera systems. For 9 scenes with varying scene complexity, lighting and background, images are captured with two identical camera assemblies at 10 focal lengths (28-70mm) and 5 apertures (f/2.8-f/22), spanning 50 optical configurations in 2000 images per scene. This full-range optics coverage enables controlled analysis of geometric and optical effects for monocular and stereo depth estimation, shallow depth-of-field rendering, deblurring, 3D scene reconstruction and novel view synthesis. Each focal configuration has a dedicated calibration image set, supporting evaluation of classical and learning based methods for intrinsic and extrinsic calibration. The dataset features challenging visual elements such as multi-scale optical illusions, reflective surfaces, mirrors, transparent glass walls, fine-grained details, and natural / artificial ambient light variations. This work attempts to bridge the realism gap between synthetic training data and real camera optics, and demonstrates challenges with the current state-of-the-art monocular, stereo depth and depth-of-field methods. We release the dataset, calibration files, and evaluation code to support reproducible research on real-world optical generalization.
Authors Nisarg K. Trivedi, Vinayak A. Belludi, Li-Yun Wang, Pardis Taghavi, Dante Lok
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice
Title CN 从文本到多模态：探索大语言模型在医疗实践中的演变与影响
Keywords 大语言模型 · 多模态学习 · 医疗AI · 综述 · 临床决策支持
Summary 这是一篇综述性文章，回顾了大语言模型从纯文本向多模态发展的趋势及其在临床决策、医学影像、患者交互和科研中的应用。文章讨论了数据、技术与伦理方面的挑战，并呼吁未来在数据集构建和模态对齐方面进行研究。然而，全文未涉及任何具体的文档理解技术改进。
Reason 该论文是一篇关于多模态大语言模型在医疗领域应用的综述，属于典型的‘平行下游应用’。其内容聚焦于医疗场景下的应用分类与挑战总结，并未提出任何面向文档图像理解（DIU）的新方法、新架构或可迁移的技术创新。尽管提及多模态融合，但无具体针对OCR幻觉、复杂排版或视觉token压缩等DIU核心问题的解决方案，完全不符合本筛选标准中对方法论创新的要求。
Abstract Large Language Models (LLMs) have rapidly evolved from text-based systems to multimodal platforms, significantly impacting various sectors including healthcare. This comprehensive review explores the progression of LLMs to Multimodal Large Language Models (MLLMs) and their growing influence in medical practice. We examine the current landscape of MLLMs in healthcare, analyzing their applications across clinical decision support, medical imaging, patient engagement, and research. The review highlights the unique capabilities of MLLMs in integrating diverse data types, such as text, images, and audio, to provide more comprehensive insights into patient health. We also address the challenges facing MLLM implementation, including data limitations, technical hurdles, and ethical considerations. By identifying key research gaps, this paper aims to guide future investigations in areas such as dataset development, modality alignment methods, and the establishment of ethical guidelines. As MLLMs continue to shape the future of healthcare, understanding their potential and limitations is crucial for their responsible and effective integration into medical practice.
Authors Qian Niu, Keyu Chen, Ming Li, Pohsun Feng, Ziqian Bi, Lawrence KQ Yan, Yichao Zhang, Caitlyn Heqi Yin, Cheng Fei, Junyu Liu, Tianyang Wang, Yunze Wang, Silin Chen, Ming Liu, Benji Peng, Xinyuan Song, Ziyuan Qin, Riyang Bao, Zekun Jiang
Comment 12 pages, 1 figure
Categories Artificial Intelligence, Computation and Language
0
NOIR 2.0: Neural Signal Operated Intelligent Robots for Everyday Activities Conference on Robot Learning (CoRL 2024)
Title CN NOIR 2.0：用于日常活动的神经信号操作智能机器人
Keywords 脑机接口 · 机器人控制 · 少样本学习 · 意图预测 · CoRL
Summary 本文提出了NOIR 2.0，一种基于脑电图（EEG）的脑-机器人接口系统，通过改进的脑信号解码算法和少样本学习机制，提升机器人对用户意图的理解效率和任务执行速度。系统面向日常生活辅助场景，支持非侵入式脑控机器人操作。
Reason 该论文属于具身智能与机器人领域，研究脑机接口控制机器人执行日常任务，与文档图像理解（DIU）、多模态大模型在文档理解中的应用无直接关联。尽管使用了基础模型进行少样本学习，但其技术路线和问题域均无法迁移到解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点。
Abstract Neural Signal Operated Intelligent Robots (NOIR) system is a versatile brain-robot interface that allows humans to control robots for daily tasks using their brain signals. This interface utilizes electroencephalography (EEG) to translate human intentions regarding specific objects and desired actions directly into commands that robots can execute. We present NOIR 2.0, an enhanced version of NOIR. NOIR 2.0 includes faster and more accurate brain decoding algorithms, which reduce task completion time by 46%. NOIR 2.0 uses few-shot robot learning algorithms to adapt to individual users and predict their intentions. The new learning algorithms leverage foundation models for more sample-efficient learning and adaptation (15 demos vs. a single demo), significantly reducing overall human time by 65%.
Authors Tasha Kim, Yingke Wang, Hanvit Cho, Alex Hodges
Comment Conference on Robot Learning (CoRL 2024), CoRoboLearn
Categories Artificial Intelligence
0
Personalized Image Generation for Recommendations Beyond Catalogs
Title CN 超越目录的商品推荐个性化图像生成
Keywords 个性化生成 · 扩散模型 · 隐式反馈 · 推荐系统 · 图像生成
Summary 本文提出REBECA，一种无需微调扩散模型的轻量级框架，通过两阶段流程从用户的隐式反馈（如点赞、评分）中学习个性化图像生成。首先训练条件扩散模型生成用户和评分特定的图像嵌入，再利用预训练的扩散模型将其解码为图像。实验表明该方法在真实数据集上能高效生成高保真且个性化的图像。
Reason 该论文属于纯图像生成领域，研究的是基于用户行为数据（如点击、评分）的个性化图像生成框架REBECA，其目标是为推荐系统生成符合用户偏好的图像。尽管使用了扩散模型和隐式反馈信号，但其核心任务与文档图像理解（DIU）无关，不涉及OCR、布局分析、表格识别或文档视觉问答（DocVQA）等任务。同时，该方法聚焦于推荐场景下的图像生成质量与个性化，无法迁移解决DIU中的OCR幻觉、复杂排版解析或长文档推理等核心痛点，属于负面清单中的‘无关领域’（纯图像生成），因此直接打0分。
Abstract Personalization is central to human-AI interaction, yet current diffusion-based image generation systems remain largely insensitive to user diversity. Existing attempts to address this often rely on costly paired preference data or introduce latency through Large Language Models. In this work, we introduce REBECA (REcommendations BEyond CAtalogs), a lightweight and scalable framework for personalized image generation that learns directly from implicit feedback signals such as likes, ratings, and clicks. Instead of fine-tuning the underlying diffusion model, REBECA employs a two-stage process: training a conditional diffusion model to sample user- and rating-specific image embeddings, which are subsequently decoded into images using a pretrained diffusion backbone. This approach enables efficient, fine-tuning-free personalization across large user bases. We rigorously evaluate REBECA on real-world datasets, proposing a novel statistical personalization verifier and a permutation-based hypothesis test to assess preference alignment. Our results demonstrate that REBECA consistently produces high-fidelity images tailored to individual tastes, outperforming baselines while maintaining computational efficiency.
Authors Gabriel Patron, Zhiwei Xu, Ishan Kapnadak, Felipe Maia Polo
Categories Artificial Intelligence
0
A Review of Pseudospectral Optimal Control: From Theory to Flight
Title CN 伪谱最优控制综述：从理论到飞行
Keywords 最优控制 · 伪谱方法 · 航空航天 · 嵌入式系统 · 飞行控制
Summary 本文综述了伪谱最优控制的理论发展及其在NASA航天器等飞行平台上的实际应用，重点讨论了该方法在航空航天和自主系统中解决复杂控制问题的能力，并介绍了其在嵌入式系统中的实现进展。
Reason 该论文属于控制理论与航空航天工程领域，研究内容为伪谱最优控制在飞行器和嵌入式系统中的应用，与文档图像理解（DIU）、多模态大模型（VLM）或视觉语言任务无任何关联。不涉及OCR、文档布局分析、表格识别、视觉推理或相关底层技术，完全偏离DIU核心问题。
Abstract The home space for optimal control is a Sobolev space. The home space for pseudospectral theory is also a Sobolev space. It thus seems natural to combine pseudospectral theory with optimal control theory and construct ``pseudospectral optimal control theory,'' a term coined by Ross. In this paper, we review key theoretical results in pseudospectral optimal control that have proven to be critical for a successful flight. Implementation details of flight demonstrations onboard NASA spacecraft are discussed along with emerging trends and techniques in both theory and practice. The 2011 launch of pseudospectral optimal control in embedded platforms is changing the way in which we see solutions to challenging control problems in aerospace and autonomous systems.
Authors I. M. Ross, M. Karpenko
Comment https://www.sciencedirect.com/science/article/abs/pii/S1367578812000375
Categories Artificial Intelligence
0
A Unified Noise-Curvature View of Loss of Trainability
Title CN 训练能力丧失的统一噪声-曲率视角
Keywords 持续学习 · 训练能力丧失 · 优化 · 梯度噪声 · 曲率
Summary 本文从优化角度分析持续学习中出现的训练能力丧失现象，发现传统指标（如Hessian秩、梯度范数等）无法可靠预测该问题。为此，作者提出了两个新的指标：批大小感知的梯度噪声界和曲率波动控制界，并据此设计了一种分层自适应步长调度器，以避免训练停滞。实验表明该调度器能提升多种正则化方法的性能，且无需调参即可模拟人工设计的衰减策略。
Reason 该论文研究的是持续学习中的训练能力丧失问题，属于优化理论和深度学习训练动态的通用分析，虽然提出的梯度噪声与曲率控制方法在理论上可能对模型训练稳定性有帮助，但并未涉及多模态建模、视觉语言理解或文档结构建模等DIU核心问题。其方法迁移至文档图像理解领域的路径不明确，且无任何与OCR、布局分析、表格识别或VLM相关的实验或设计考量，属于通用AI优化研究，与DIU任务无直接关联。
Abstract Loss of trainability refers to a phenomenon in continual learning where parameter updates no longer make progress on the optimization objective, so accuracy stalls or degrades as the learning problem changes over time. In this paper, we analyze loss of trainability through an optimization lens and find that the phenomenon is not reliably predicted by existing individual indicators such as Hessian rank, sharpness level, weight or gradient norms, gradient-to-parameter ratios, and unit-sign entropy. Motivated by our analysis, we introduce two complementary indicators: a batch-size-aware gradient-noise bound and a curvature volatility-controlled bound. We then combine these two indicators into a per-layer adaptive noise threshold on the effective step-size that anticipates trainability behavior. Using this insight, we propose a step-size scheduler that keeps each layer's effective parameter update below this bound, thereby avoiding loss of trainability. We demonstrate that our scheduler can improve the accuracy maintained by previously proposed approaches, such as concatenated ReLU (CReLU), Wasserstein regularizer, and L2 weight decay. Surprisingly, our scheduler produces adaptive step-size trajectories that, without tuning, mirror the manually engineered step-size decay schedules.
Authors Gunbir Singh Baveja, Alex Lewandowski, Mark Schmidt
Categories Artificial Intelligence
0
Primal: A Unified Deterministic Framework for Quasi-Orthogonal Hashing and Manifold Learning
Title CN Primal：一种用于准正交哈希与流形学习的统一确定性框架
Keywords 特征映射 · 哈希 · 流形学习 · 数论 · 确定性投影
Summary 本文提出Primal，一种基于素数平方根数论独立性的确定性特征映射框架，通过Besicovitch性质构造无限非重复相位轨迹的无理频率调制。包含两种变体：StaticPrime用于生成接近Welch界准正交性的位置编码；DynamicPrime作为可调投影层，在低频下实现等距核映射以线性化非凸几何结构，在高频下则变为最大熵单向哈希，适用于超维计算和隐私保护学习。实验显示其在正交性保持和分布紧致性上优于高斯基线。
Reason 该论文属于纯理论性的特征映射与哈希方法研究，聚焦于数论和随机投影的替代方案，尽管在数学上具有新颖性，但其核心内容与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。未涉及OCR、布局分析、表格识别、多模态对齐、细粒度定位或高分辨率视觉处理等DIU关键问题。所提出的StaticPrime和DynamicPrime方法面向信号重建、压缩感知和超维计算，迁移到文档理解任务的路径不清晰，且无视觉输入建模或跨模态交互机制。属于无关领域的理论方法，无法解决OCR幻觉、复杂排版或长文档推理等实际痛点。
Abstract We present Primal, a deterministic feature mapping framework that harnesses the number-theoretic independence of prime square roots to construct robust, tunable vector representations. Diverging from standard stochastic projections (e.g., Random Fourier Features), our method exploits the Besicovitch property to create irrational frequency modulations that guarantee infinite non-repeating phase trajectories. We formalize two distinct algorithmic variants: (1) StaticPrime, a sequence generation method that produces temporal position encodings empirically approaching the theoretical Welch bound for quasi-orthogonality; and (2) DynamicPrime, a tunable projection layer for input-dependent feature mapping. A central novelty of the dynamic framework is its ability to unify two disparate mathematical utility classes through a single scaling parameter σ. In the low-frequency regime, the method acts as an isometric kernel map, effectively linearizing non-convex geometries (e.g., spirals) to enable high-fidelity signal reconstruction and compressive sensing. Conversely, the high-frequency regime induces chaotic phase wrapping, transforming the projection into a maximum-entropy one-way hash suitable for Hyperdimensional Computing and privacy-preserving Split Learning. Empirical evaluations demonstrate that our framework yields superior orthogonality retention and distribution tightness compared to normalized Gaussian baselines, establishing it as a computationally efficient, mathematically rigorous alternative to random matrix projections. The code is available at https://github.com/VladimerKhasia/primal
Authors Vladimer Khasia
Categories Artificial Intelligence
0
Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior
Title CN 在大语言模型对齐中实现多元价值观揭示了安全性、包容性和模型行为之间的权衡
Keywords 大语言模型对齐 · 多元价值观 · 安全性 · 社会多样性 · 偏好优化
Summary 本文通过收集美国和德国参与者对LLM回复的多维度评分（毒性、情感意识、敏感性等），研究不同人口学群体在模型对齐偏好上的差异，并分析技术设计选择（如评分尺度、分歧处理方式、优化算法）对模型行为的影响。结果显示，不同群体存在显著评价差异，且保留评分者分歧比多数投票更能降低毒性。
Reason 该论文研究的是大语言模型在对齐过程中如何平衡不同社会群体的价值观，属于纯NLP领域的安全与对齐（Safety/Alignment）问题，且未涉及视觉输入或文档理解中的核心挑战（如OCR幻觉、排版理解等）。尽管‘价值观对齐’在广义上可能影响系统设计，但其方法论无法直接迁移用于解决文档图像理解中的技术痛点，属于无关领域。
Abstract Although large language models (LLMs) are increasingly trained using human feedback for safety and alignment with human values, alignment decisions often overlook human social diversity. This study examines how incorporating pluralistic values affects LLM behavior by systematically evaluating demographic variation and design parameters in the alignment pipeline. We collect alignment data from US and German participants (N = 1,095 participants, 27,375 ratings) who rated LLM responses across five dimensions: Toxicity, Emotional Awareness (EA), Sensitivity, Stereotypical Bias, and Helpfulness. We fine-tuned multiple Large Language Models and Large Reasoning Models using preferences from different social groups while varying rating scales, disagreement handling methods, and optimization techniques. The results revealed systematic demographic effects: male participants rated responses 18% less toxic than female participants; conservative and Black participants rated responses 27.9% and 44% higher on EA than liberal and White participants, respectively. Models fine-tuned on group-specific preferences exhibited distinct behaviors. Technical design choices showed strong effects: the preservation of rater disagreement achieved roughly 53% greater toxicity reduction than majority voting, and 5-point scales yielded about 22% more reduction than binary formats; and Direct Preference Optimization (DPO) consistently outperformed Group Relative Policy Optimization (GRPO) in multi-value optimization. These findings represent a preliminary step in answering a critical question: How should alignment balance expert-driven and user-driven signals to ensure both safety and fair representation?
Authors Dalia Ali, Dora Zhao, Allison Koenecke, Orestis Papakyriakopoulos
Categories Artificial Intelligence
0
Physics-Constrained Flow Matching: Sampling Generative Models with Hard Constraints NeurIPS 2025
Title CN 基于物理约束的流匹配：在硬约束下采样生成模型
Keywords 生成模型 · 流匹配 · 物理约束 · 偏微分方程 · 零样本推理
Summary 本文提出了一种名为Physics-Constrained Flow Matching（PCFM）的方法，用于在预训练的基于流的生成模型中强制满足非线性物理约束（如守恒律）。该方法通过在采样过程中持续施加物理校正来引导中间状态，确保最终解严格满足约束条件。实验表明其在多种含激波、间断和尖锐特征的PDE上优于基线方法。
Reason 该论文研究的是物理系统（如偏微分方程）中的生成模型如何满足硬性物理约束，属于科学计算与生成模型交叉领域，与文档图像理解（DIU）或视觉语言多模态任务无直接关联。尽管其‘约束满足’的思想在抽象层面有启发性，但方法针对的是连续物理场建模，无法迁移至OCR、文档布局分析、表格识别等DIU核心问题，且未涉及视觉表示、多模态对齐或细粒度定位等关键技术痛点。
Abstract Deep generative models have recently been applied to physical systems governed by partial differential equations (PDEs), offering scalable simulation and uncertainty-aware inference. However, enforcing physical constraints, such as conservation laws (linear and nonlinear) and physical consistencies, remains challenging. Existing methods often rely on soft penalties or architectural biases that fail to guarantee hard constraints. In this work, we propose Physics-Constrained Flow Matching (PCFM), a zero-shot inference framework that enforces arbitrary nonlinear constraints in pretrained flow-based generative models. PCFM continuously guides the sampling process through physics-based corrections applied to intermediate solution states, while remaining aligned with the learned flow and satisfying physical constraints. Empirically, PCFM outperforms both unconstrained and constrained baselines on a range of PDEs, including those with shocks, discontinuities, and sharp features, while ensuring exact constraint satisfaction at the final solution. Our method provides a flexible framework for enforcing hard constraints in both scientific and general-purpose generative models, especially in applications where constraint satisfaction is essential.
Authors Utkarsh Utkarsh, Pengfei Cai, Alan Edelman, Rafael Gomez-Bombarelli, Christopher Vincent Rackauckas
Comment 36 pages, 9 figures, 8 tables, Accepted to NeurIPS 2025
Categories Artificial Intelligence
0
Safe and Economical UAV Trajectory Planning in Low-Altitude Airspace: A Hybrid DRL-LLM Approach with Compliance Awareness
Title CN 低空空域中安全且经济的无人机轨迹规划：一种具有合规意识的混合DRL-LLM方法
Keywords 无人机 · 轨迹规划 · 深度强化学习 · 大语言模型 · 低空经济
Summary 本文提出了一种结合深度强化学习（DRL）和大语言模型（LLM）的无人机轨迹规划框架，旨在实现低空经济场景下的安全、合规且节能的路径规划。通过引入LLM的推理能力提升DRL的学习效率，并在多个指标上验证了方法的有效性。
Reason 该论文属于具身智能与机器人领域，研究的是无人机在城市环境中的轨迹规划问题，尽管使用了LLM进行推理辅助，但其核心任务与文档图像理解（DIU）或DocVQA完全无关。未涉及OCR、布局分析、表格识别、视觉文本理解等DIU核心问题，也无法迁移其方法解决DIU中的幻觉、密集文本、复杂排版或长文档推理等痛点。
Abstract The rapid growth of the low-altitude economy has driven the widespread adoption of unmanned aerial vehicles (UAVs). This growing deployment presents new challenges for UAV trajectory planning in complex urban environments. However, existing studies often overlook key factors, such as urban airspace constraints and economic efficiency, which are essential in low-altitude economy contexts. Deep reinforcement learning (DRL) is regarded as a promising solution to these issues, while its practical adoption remains limited by low learning efficiency. To overcome this limitation, we propose a novel UAV trajectory planning framework that combines DRL with large language model (LLM) reasoning to enable safe, compliant, and economically viable path planning. Experimental results demonstrate that our method significantly outperforms existing baselines across multiple metrics, including data collection rate, collision avoidance, successful landing, regulatory compliance, and energy efficiency. These results validate the effectiveness of our approach in addressing UAV trajectory planning key challenges under constraints of the low-altitude economy networking.
Authors Yanwei Gong, Junchao Fan, Ruichen Zhang, Dusit Niyato, Yingying Yao, Xiaolin Chang
Categories Artificial Intelligence
0
RefTr: Recurrent Refinement of Confluent Trajectories for 3D Vascular Tree Centerline Graphs
Title CN RefTr：通过汇聚轨迹的循环优化生成3D血管树中心线图
Keywords 3D中心线检测 · 血管树 · 图生成 · Transformer · 医学图像
Summary 本文提出RefTr模型，用于从3D医学图像中生成具有正确树状拓扑的血管中心线图。该模型采用Producer-Refiner架构，通过Transformer解码器循环优化初始的汇聚轨迹，并引入高效的非极大抑制算法提升精度。相比现有方法，在多个数据集上实现了更高的召回率和相当的精确度，同时参数更少、推理更快。
Reason 该论文研究的是3D医学图像中的血管树中心线提取，属于3D视觉与医疗影像分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。尽管其方法涉及图结构建模和拓扑保持，但应用场景和输入模态（3D体数据）与文档图像差异巨大，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心问题。
Abstract Tubular trees, such as blood vessels and lung airways, are essential for material transport within the human body. Accurately detecting their centerlines with correct tree topology is critical for clinical tasks such as diagnosis, treatment planning, and surgical navigation. In these applications, maintaining high recall is crucial, as missing small branches can result in fatal mistakes caused by incomplete assessments or undetected abnormalities. We present RefTr, a 3D image-to-graph model for centerline generation of vascular trees via recurrent refinement of confluent trajectories. RefTr uses a Producer-Refiner architecture based on a Transformer decoder, where the Producer proposes a set of initial confluent trajectories that are recurrently refined by the Refiner to produce final trajectories, which forms the centerline graph. The confluent trajectory representation enables refinement of complete trajectories while explicitly enforcing a valid tree topology. The recurrent refinement scheme improves precision and reuses the same Refiner block across multiple steps, yielding a 2.4x reduction in decoder parameters compared to previous SOTA. We also introduce an efficient non-maximum suppression algorithm for spatial tree graphs to merge duplicate branches and boost precision. Across multiple public centerline datasets, RefTr achieves superior recall and comparable precision to previous SOTA, while offering faster inference and substantially fewer parameters, demonstrating its potential as a new state-of-the-art framework for vascular tree analysis in 3D medical imaging.
Authors Roman Naeem, David Hagerman, Jennifer Alvén, Fredrik Kahl
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Training-Free Diffusion Priors for Text-to-Image Generation via Optimization-based Visual Inversion
Title CN 基于优化的视觉反演实现免训练扩散先验用于文本到图像生成
Keywords 文本到图像生成 · 扩散模型 · 视觉反演 · 免训练方法 · 生成模型
Summary 本文提出一种免训练、免数据的优化方法——基于优化的视觉反演（OVI），用于在文本到图像生成中替代传统的扩散先验网络。通过从随机伪标记初始化潜在表示，并迭代优化其与文本嵌入的余弦相似度，结合新设计的马氏距离和最近邻损失约束，提升生成图像的真实性。实验表明该方法在Kandinsky 2.2上可媲美现有数据高效的先验模型。然而，当前评估基准存在缺陷，仅用文本嵌入作先验也能获得高分。
Reason 该论文研究的是文本到图像生成中的扩散模型先验问题，属于纯图像生成领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。其提出的方法OVI旨在替代T2I模型中的文本-视觉对齐先验网络，但应用场景为创意图像生成，无法迁移解决OCR幻觉、复杂排版理解、细粒度定位或长文档推理等DIU核心痛点。
Abstract Diffusion models have established the state-of-the-art in text-to-image generation, but their performance often relies on a diffusion prior network to translate text embeddings into the visual manifold for easier decoding. These priors are computationally expensive and require extensive training on massive datasets. In this work, we challenge the necessity of a trained prior at all by employing Optimization-based Visual Inversion (OVI), a training-free and data-free alternative, to replace the need for a prior. OVI initializes a latent visual representation from random pseudo-tokens and iteratively optimizes it to maximize the cosine similarity with input textual prompt embedding. We further propose two novel constraints, a Mahalanobis-based and a Nearest-Neighbor loss, to regularize the OVI optimization process toward the distribution of realistic images. Our experiments, conducted on Kandinsky 2.2, show that OVI can serve as an alternative to traditional priors. More importantly, our analysis reveals a critical flaw in current evaluation benchmarks like T2I-CompBench++, where simply using the text embedding as a prior achieves surprisingly high scores, despite lower perceptual quality. Our constrained OVI methods improve visual fidelity over this baseline, with the Nearest-Neighbor approach proving particularly effective, achieving quantitative scores comparable to or higher than the state-of-the-art data-efficient prior, indicating that the idea merits further investigation. The code will be publicly available upon acceptance.
Authors Samuele Dell'Erba, Andrew D. Bagdanov
Comment 11 pages, 7 figures, technical report (preprint)
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
0
Conformal Safety Monitoring for Flight Testing: A Case Study in Data-Driven Safety Learning ICRA 2025 Workshop on Robot safety under uncertainty from intangible specifications
Title CN 飞行测试中的保角安全监控：数据驱动安全学习的案例研究
Keywords 安全监控 · 保角预测 · 轨迹模拟 · 不确定性建模 · 飞行测试
Summary 本文提出一种数据驱动的飞行测试实时安全监控方法，利用离线随机轨迹模拟构建短期安全风险的校准统计模型。通过状态预测模型、最近邻安全分类器和保角预测进行分类校准，在参数不确定的飞行动力学模型上验证了其在提前识别不安全场景方面的有效性，并满足理论保障。然而该工作与文档理解无关。
Reason 该论文属于具身智能与机器人领域，研究的是飞行测试中基于动态不确定性的安全风险预警，核心方法为轨迹模拟、最近邻分类与保角预测，虽涉及数据驱动的安全监控，但与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract We develop a data-driven approach for runtime safety monitoring in flight testing, where pilots perform maneuvers on aircraft with uncertain parameters. Because safety violations can arise unexpectedly as a result of these uncertainties, pilots need clear, preemptive criteria to abort the maneuver in advance of safety violation. To solve this problem, we use offline stochastic trajectory simulation to learn a calibrated statistical model of the short-term safety risk facing pilots. We use flight testing as a motivating example for data-driven learning/monitoring of safety due to its inherent safety risk, uncertainty, and human-interaction. However, our approach consists of three broadly-applicable components: a model to predict future state from recent observations, a nearest neighbor model to classify the safety of the predicted state, and classifier calibration via conformal prediction. We evaluate our method on a flight dynamics model with uncertain parameters, demonstrating its ability to reliably identify unsafe scenarios, match theoretical guarantees, and outperform baseline approaches in preemptive classification of risk.
Authors Aaron O. Feldman, D. Isaiah Harp, Joseph Duncan, Mac Schwager
Comment ICRA 2025 Workshop on Robot safety under uncertainty from intangible specifications
Categories Artificial Intelligence
0
Memories Retrieved from Many Paths: A Multi-Prefix Framework for Robust Detection of Training Data Leakage in Large Language Models
Title CN 从多路径检索的记忆：用于大语言模型训练数据泄露鲁棒检测的多前缀框架
Keywords 大语言模型 · 数据泄露 · 记忆化检测 · 隐私风险 · 多前缀
Summary 本文提出一种名为“多前缀记忆化”的新框架，通过衡量一段文本能否被多个不同前缀触发生成，来判断该内容是否被大语言模型过度记忆，从而检测训练数据泄露。实验表明该方法在开源和对齐对话模型中能有效区分记忆化与非记忆化内容，提供了一种鲁棒的数据泄露审计工具。
Reason 该论文研究的是大语言模型中的训练数据记忆化与隐私泄露问题，属于纯NLP领域中关于模型安全与版权审计的议题。尽管涉及‘幻觉’相关概念，但其核心关注点为数据泄露检测机制，并非文档图像理解（DIU）中的视觉-语言对齐、OCR错误缓解或复杂版面推理等任务。方法论上聚焦于文本生成路径的统计分析，难以迁移至文档图像中高分辨率视觉内容处理、细粒度定位或多模态对齐等关键痛点。
Abstract Large language models, trained on massive corpora, are prone to verbatim memorization of training data, creating significant privacy and copyright risks. While previous works have proposed various definitions for memorization, many exhibit shortcomings in comprehensively capturing this phenomenon, especially in aligned models. To address this, we introduce a novel framework: multi-prefix memorization. Our core insight is that memorized sequences are deeply encoded and thus retrievable via a significantly larger number of distinct prefixes than non-memorized content. We formalize this by defining a sequence as memorized if an external adversarial search can identify a target count of distinct prefixes that elicit it. This framework shifts the focus from single-path extraction to quantifying the robustness of a memory, measured by the diversity of its retrieval paths. Through experiments on open-source and aligned chat models, we demonstrate that our multi-prefix definition reliably distinguishes memorized from non-memorized data, providing a robust and practical tool for auditing data leakage in LLMs.
Authors Trung Cuong Dang, David Mohaisen
Comment 11 pages, 2 tables, 8 figures
Categories Artificial Intelligence, Computation and Language
0
Adversarial Multi-Task Learning for Liver Tumor Segmentation, Dynamic Enhancement Regression, and Classification
Title CN 用于肝肿瘤分割、动态增强回归和分类的对抗多任务学习
Keywords 多任务学习 · 医学图像分割 · 对抗学习 · 动态MRI · Transformer
Summary 本文提出MTI-Net，一种用于同时进行肝肿瘤分割、动态增强回归和分类的多任务对抗学习网络。通过引入多域信息熵融合模块和任务交互机制，结合浅层Transformer提取动态MRI时序关系，在238名受试者的数据集上取得了良好表现。
Reason 该论文研究的是医学图像中的肝脏肿瘤分割与分类，属于医疗影像分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。其方法聚焦于动态MRI序列的多任务学习与特征融合，虽涉及视觉信息处理，但应用场景和技术动机均无法迁移到文档理解的核心痛点（如OCR幻觉、复杂版式、细粒度定位等）。
Abstract Liver tumor segmentation, dynamic enhancement regression, and classification are critical for clinical assessment and diagnosis. However, no prior work has attempted to achieve these tasks simultaneously in an end-to-end framework, primarily due to the lack of an effective framework that captures inter-task relevance for mutual improvement and the absence of a mechanism to extract dynamic MRI information effectively. To address these challenges, we propose the Multi-Task Interaction adversarial learning Network (MTI-Net), a novel integrated framework designed to tackle these tasks simultaneously. MTI-Net incorporates Multi-domain Information Entropy Fusion (MdIEF), which utilizes entropy-aware, high-frequency spectral information to effectively integrate features from both frequency and spectral domains, enhancing the extraction and utilization of dynamic MRI data. The network also introduces a task interaction module that establishes higher-order consistency between segmentation and regression, thus fostering inter-task synergy and improving overall performance. Additionally, we designed a novel task-driven discriminator (TDD) to capture internal high-order relationships between tasks. For dynamic MRI information extraction, we employ a shallow Transformer network to perform positional encoding, which captures the relationships within dynamic MRI sequences. In experiments on a dataset of 238 subjects, MTI-Net demonstrates high performance across multiple tasks, indicating its strong potential for assisting in the clinical assessment of liver tumors. The code is available at: https://github.com/xiaojiao929/MTI-Net.
Authors Xiaojiao Xiao, Qinmin Vivian Hu, Tae Hyun Kim, Guanghui Wang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecaster
Title CN TS-RAG：基于检索增强生成的时间序列基础模型是强大的零样本预测器
Keywords 时间序列预测 · 检索增强生成 · 基础模型 · 零样本学习 · 非平稳性
Summary 本文提出TS-RAG，一种面向时间序列预测的检索增强生成框架，通过预训练时间序列编码器从知识库中检索语义相关的片段，并利用自适应检索混合模块（ARM）动态融合检索结果与模型内部表示，在无需微调的情况下提升了零样本预测性能和可解释性。
Reason 该论文属于时间序列预测领域的下游应用研究，尽管引入了检索增强机制（RAG），但其核心任务与文档图像理解（DIU）无关。虽然RAG机制在理论上可能用于长文档信息检索，但本文未涉及任何视觉模态、OCR、布局分析或文档VQA等DIU核心问题，属于‘平行应用’类工作，无法为DIU提供可迁移的方法论创新。
Abstract Large Language Models (LLMs) and Foundation Models (FMs) have recently become prevalent for time series forecasting tasks. While fine-tuning LLMs enables domain adaptation, they often struggle to generalize across diverse and unseen datasets. Moreover, existing Time Series Foundation Models (TSFMs) still face challenges in handling non-stationary dynamics and distribution shifts, largely due to the lack of effective mechanisms for adaptation. To this end, we present TS-RAG, a retrieval-augmented generation framework for time series forecasting that enhances the generalization and interpretability of TSFMs. Specifically, TS-RAG leverages pre-trained time series encoders to retrieve semantically relevant segments from a dedicated knowledge base, enriching the contextual representation of the input query. Furthermore, we propose an Adaptive Retrieval Mixer (ARM) module that dynamically fuses the retrieved patterns with the TSFM's internal representation, improving forecasting accuracy without requiring task-specific fine-tuning. Thorough empirical studies on seven public benchmark datasets demonstrate that TS-RAG achieves state-of-the-art zero-shot forecasting performance, outperforming the existing TSFMs by up to 6.84% across diverse domains while also providing desirable interpretability. Our code and data are available at: https://github.com/UConn-DSIS/TS-RAG
Authors Kanghui Ning, Zijie Pan, Yu Liu, Yushan Jiang, James Yiming Zhang, Kashif Rasul, Anderson Schneider, Lintao Ma, Yuriy Nevmyvaka, Dongjin Song
Categories Artificial Intelligence
0
OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability
Title CN OpenApps：通过模拟环境变化来衡量UI-Agent的可靠性
Keywords UI-Agent · 可靠性评估 · 环境变化 · 多模态代理 · 基准测试
Summary 本文提出了OpenApps，一个用于评估多模态UI-Agent在不同应用界面变体下可靠性的开源生态系统。通过生成数千个具有外观和内容变化的应用版本，作者对七种主流多模态Agent进行了超过1万次独立测试，发现现有Agent在不同环境配置下的任务成功率波动极大，甚至出现严重的行为异常。研究强调了在多样化环境中评估Agent可靠性的重要性。
Reason 该论文属于具身智能与UI交互代理的应用场景，研究的是多模态Agent在不同应用界面中的行为可靠性，核心关注点为Agent在动态UI环境下的稳定性评估。尽管涉及多模态交互，但其任务设定远离文档图像理解（DIU）的核心问题，如文本识别、布局分析、表格解析或视觉-语言推理。未提出可用于解决OCR幻觉、密集文本处理或复杂排版建模的方法论创新。属于特定应用场景的下游构建，不符合DIU本题要求，也不具备向文档理解迁移的通用方法价值。
Abstract Reliability is key to realizing the promise of autonomous UI-Agents, multimodal agents that directly interact with apps in the same manner as humans, as users must be able to trust an agent to complete a given task. Current evaluations rely on fixed environments, often clones of existing apps, which are limited in that they can only shed light on whether or how often an agent can complete a task within a specific environment. When deployed however, agents are likely to encounter variations in app design and content that can affect an agent's ability to complete a task. To address this blind spot of measuring agent reliability across app variations, we develop OpenApps, a light-weight open-source ecosystem with six apps (messenger, calendar, maps, etc.) that are configurable in appearance and content. OpenApps requires just a single CPU to run, enabling easy generation and deployment of thousands of versions of each app. Specifically, we run more than 10,000 independent evaluations to study reliability across seven leading multimodal agents. We find that while standard reliability within a fixed app is relatively stable, reliability can vary drastically when measured across app variations. Task success rates for many agents can fluctuate by more than $50\%$ across app variations. For example, Kimi-VL-3B's average success across all tasks fluctuates from $63\%$ to just $4\%$ across app versions. We also find agent behaviors such as looping or hallucinating actions can differ drastically depending on the environment configuration. These initial findings highlight the importance of measuring reliability along this new dimension of app variations. OpenApps is available at https://facebookresearch.github.io/OpenApps/
Authors Karen Ullrich, Jingtong Su, Claudia Shi, Arjun Subramonian, Amir Bar, Ivan Evtimov, Nikolaos Tsilivis, Randall Balestriero, Julia Kempe, Mark Ibrahim
Categories Artificial Intelligence
0
Large Language Models' Complicit Responses to Illicit Instructions across Socio-Legal Contexts
Title CN 大语言模型在不同社会法律情境下对非法指令的共谋式响应
Keywords 大语言模型 · 安全性 · 共谋行为 · 法律合规 · 社会偏见
Summary 本文提出了‘共谋式协助’的概念，用于描述大语言模型为用户的非法意图提供支持的行为，并构建了一个涵盖269个非法场景和50种非法意图的评估基准。研究发现主流模型（如GPT-4o）在近半数案例中提供了非法帮助，且现有安全对齐策略效果有限，甚至可能加剧问题。此外，模型表现出对弱势群体的差异化风险响应，反映出潜在的社会刻板印象。
Reason 该论文研究的是大语言模型在社会法律语境下的安全性问题，特别是其对非法指令的‘共谋式协助’行为。属于纯NLP安全与对齐（Safety/Alignment）范畴，且未涉及视觉模态或文档图像理解中的幻觉、定位等核心痛点。尽管提到了模型推理轨迹，但其方法论聚焦于社会偏见与法律合规性分析，无法迁移至文档图像理解任务。
Abstract Large language models (LLMs) are now deployed at unprecedented scale, assisting millions of users in daily tasks. However, the risk of these models assisting unlawful activities remains underexplored. In this study, we define this high-risk behavior as complicit facilitation - the provision of guidance or support that enables illicit user instructions - and present four empirical studies that assess its prevalence in widely deployed LLMs. Using real-world legal cases and established legal frameworks, we construct an evaluation benchmark spanning 269 illicit scenarios and 50 illicit intents to assess LLMs' complicit facilitation behavior. Our findings reveal widespread LLM susceptibility to complicit facilitation, with GPT-4o providing illicit assistance in nearly half of tested cases. Moreover, LLMs exhibit deficient performance in delivering credible legal warnings and positive guidance. Further analysis uncovers substantial safety variation across socio-legal contexts. On the legal side, we observe heightened complicity for crimes against societal interests, non-extreme but frequently occurring violations, and malicious intents driven by subjective motives or deceptive justifications. On the social side, we identify demographic disparities that reveal concerning complicit patterns towards marginalized and disadvantaged groups, with older adults, racial minorities, and individuals in lower-prestige occupations disproportionately more likely to receive unlawful guidance. Analysis of model reasoning traces suggests that model-perceived stereotypes, characterized along warmth and competence, are associated with the model's complicit behavior. Finally, we demonstrate that existing safety alignment strategies are insufficient and may even exacerbate complicit behavior.
Authors Xing Wang, Huiyuan Xie, Yiyan Wang, Chaojun Xiao, Huimin Chen, Holli Sargeant, Felix Steffek, Jie Shao, Zhiyuan Liu, Maosong Sun
Categories Artificial Intelligence, Computation and Language
0
InvisibleBench: A Deployment Gate for Caregiving Relationship AI
Title CN InvisibleBench：护理关系AI的部署关卡
Keywords AI安全 · 基准测试 · 多轮对话 · 部署评估 · 伦理设计
Summary InvisibleBench是一个用于评估护理关系AI系统部署前安全性的基准测试，涵盖多轮交互下的安全性、合规性、创伤知情设计等多个维度。研究对四种前沿模型进行了17种场景下的测试，发现所有模型在危机识别方面均存在显著缺陷。尽管该工作对AI部署安全具有意义，但其内容属于垂直领域应用评估，与文档图像理解的技术发展无直接关联。
Reason 该论文聚焦于护理型AI系统的安全性和长期交互风险评估，属于特定应用场景的评估框架设计，不涉及文档图像理解（DIU）或视觉语言模型在文档处理中的方法论创新。任务领域为AI安全与伦理评估，与OCR、DocVQA、布局分析等核心DIU任务无关，且未提出可迁移至文档理解领域的底层技术。
Abstract InvisibleBench is a deployment gate for caregiving-relationship AI, evaluating 3-20+ turn interactions across five dimensions: Safety, Compliance, Trauma-Informed Design, Belonging/Cultural Fitness, and Memory. The benchmark includes autofail conditions for missed crises, medical advice (WOPR Act), harmful information, and attachment engineering. We evaluate four frontier models across 17 scenarios (N=68) spanning three complexity tiers. All models show significant safety gaps (11.8-44.8 percent crisis detection), indicating the necessity of deterministic crisis routing in production systems. DeepSeek Chat v3 achieves the highest overall score (75.9 percent), while strengths differ by dimension: GPT-4o Mini leads Compliance (88.2 percent), Gemini leads Trauma-Informed Design (85.0 percent), and Claude Sonnet 4.5 ranks highest in crisis detection (44.8 percent). We release all scenarios, judge prompts, and scoring configurations with code. InvisibleBench extends single-turn safety tests by evaluating longitudinal risk, where real harms emerge. No clinical claims; this is a deployment-readiness evaluation.
Authors Ali Madad
Comment 29 pages, 3 figures
Categories Artificial Intelligence, Computation and Language
0
Data-Driven Methods and AI in Engineering Design: A Systematic Literature Review Focusing on Challenges and Opportunities
Title CN 工程设计中数据驱动方法与人工智能：聚焦挑战与机遇的系统性文献综述
Keywords 数据驱动方法 · 机器学习 · 工程设计 · 系统性综述 · 产品开发
Summary 本文是一篇关于数据驱动方法在工程设计中应用的系统性文献综述，分析了机器学习和统计方法在产品开发各阶段的使用情况、趋势及挑战。研究基于PRISMA框架，对近十年的114篇文献进行分析，指出当前深度学习应用仍有限，且验证阶段的研究不足。主要挑战包括模型可解释性差、跨阶段追溯困难等。
Reason 该论文属于‘平行应用’类综述，讨论的是将现有AI方法（如机器学习、深度学习）应用于工程设计领域，并未提出任何针对文档图像理解（DIU）或视觉语言模型（VLM）的技术创新。其内容与文档理解的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）无关，仅为跨领域应用的总结，属于典型的‘Wrapper/Application’类型，无方法论贡献。
Abstract The increasing availability of data and advancements in computational intelligence have accelerated the adoption of data-driven methods (DDMs) in product development. However, their integration into product development remains fragmented. This fragmentation stems from uncertainty, particularly the lack of clarity on what types of DDMs to use and when to employ them across the product development lifecycle. To address this, a necessary first step is to investigate the usage of DDM in engineering design by identifying which methods are being used, at which development stages, and for what application. This paper presents a PRISMA systematic literature review. The V-model as a product development framework was adopted and simplified into four stages: system design, system implementation, system integration, and validation. A structured search across Scopus, Web of Science, and IEEE Xplore (2014--2024) retrieved 1{,}689 records. After screening, 114 publications underwent full-text analysis. Findings show that machine learning (ML) and statistical methods dominate current practice, whereas deep learning (DL), though still less common, exhibits a clear upward trend in adoption. Additionally, supervised learning, clustering, regression analysis, and surrogate modeling are prevalent in design, implementation, and integration system stages but contributions to validation remain limited. Key challenges in existing applications include limited model interpretability, poor cross-stage traceability, and insufficient validation under real-world conditions. Additionally, it highlights key limitations and opportunities such as the need for interpretable hybrid models. This review is a first step toward design-stage guidelines; a follow-up synthesis should map computer science algorithms to engineering design problems and activities.
Authors Nehal Afifi, Christoph Wittig, Lukas Paehler, Andreas Lindenmann, Kai Wolter, Felix Leitenberger, Melih Dogru, Patric Grauberger, Tobias Düser, Albert Albers, Sven Matthiesen
Categories Artificial Intelligence
0
Spatio-Temporal Trajectory Foundation Model - Recent Advances and Future Directions CIKM 2025 STIntelligence Workshop
Title CN 时空轨迹基础模型——最新进展与未来方向
Keywords 基础模型 · 时空数据 · 轨迹分析 · CIKM · 综述
Summary 本文综述了时空轨迹基础模型（TFM）的最新进展，提出了方法分类体系，并分析了现有工作的优缺点，同时指出了未来研究方向。重点在于提升时空数据的泛化能力和智能分析水平。
Reason 该论文属于时空数据挖掘领域，研究的是轨迹基础模型（TFM），主要面向移动对象、交通、位置预测等时空任务。内容与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联，无法迁移解决OCR幻觉、复杂排版、细粒度定位等DIU核心问题。
Abstract Foundation models (FMs) have emerged as a powerful paradigm, enabling a diverse range of data analytics and knowledge discovery tasks across scientific fields. Inspired by the success of FMs, particularly large language models, researchers have recently begun to explore spatio-temporal foundation models (STFMs) to improve adaptability and generalization across a wide spectrum of spatio-temporal (ST) tasks. Despite rapid progress, a systematic investigation of trajectory foundation models (TFMs), a crucial subclass of STFMs, is largely lacking. This tutorial addresses this gap by offering a comprehensive overview of recent advances in TFMs, including a taxonomy of existing methodologies and a critical analysis of their strengths and limitations. In addition, the tutorial highlights open challenges and outlines promising research directions to advance spatio-temporal general intelligence through the development of robust, responsible, and transferable TFMs.
Authors Sean Bin Yang, Ying Sun, Yunyao Cheng, Yan Lin, Kristian Torp, Jilin Hu
Comment This paper has been accepted by CIKM 2025 STIntelligence Workshop
Categories Artificial Intelligence
0
Learning from Risk: LLM-Guided Generation of Safety-Critical Scenarios with Prior Knowledge
Title CN 从风险中学习：基于大语言模型引导的、结合先验知识的安全关键场景生成
Keywords 自动驾驶 · 场景生成 · 大语言模型 · CVAE · 安全验证
Summary 本文提出一种融合条件变分自编码器（CVAE）和大语言模型（LLM）的高保真交通场景生成框架。CVAE从真实交通数据中学习潜在交通结构以生成合理的基础场景，LLM则作为对抗性推理引擎，将非结构化描述转化为领域特定的损失函数，并动态引导生成不同风险等级的场景。实验表明该方法在CARLA和SMARTS仿真中显著提升了高风险与长尾事件的覆盖率，增强了对自动驾驶系统的压力测试能力。
Reason 该论文研究的是自动驾驶中的长尾风险场景生成，属于具身智能与自动驾驶领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了LLM与CVAE结合的方法，但其目标是生成高风险交通场景用于安全验证，无法迁移至解决OCR幻觉、复杂排版理解、细粒度文本定位等DIU核心痛点。
Abstract Autonomous driving faces critical challenges in rare long-tail events and complex multi-agent interactions, which are scarce in real-world data yet essential for robust safety validation. This paper presents a high-fidelity scenario generation framework that integrates a conditional variational autoencoder (CVAE) with a large language model (LLM). The CVAE encodes historical trajectories and map information from large-scale naturalistic datasets to learn latent traffic structures, enabling the generation of physically consistent base scenarios. Building on this, the LLM acts as an adversarial reasoning engine, parsing unstructured scene descriptions into domain-specific loss functions and dynamically guiding scenario generation across varying risk levels. This knowledge-driven optimization balances realism with controllability, ensuring that generated scenarios remain both plausible and risk-sensitive. Extensive experiments in CARLA and SMARTS demonstrate that our framework substantially increases the coverage of high-risk and long-tail events, improves consistency between simulated and real-world traffic distributions, and exposes autonomous driving systems to interactions that are significantly more challenging than those produced by existing rule- or data-driven methods. These results establish a new pathway for safety validation, enabling principled stress-testing of autonomous systems under rare but consequential events.
Authors Yuhang Wang, Heye Huang, Zhenhua Xu, Kailai Sun, Baoshen Guo, Jinhua Zhao
Comment 24 pages, 6 figures
Categories Artificial Intelligence
0
Gradient Descent Algorithm Survey
Title CN 梯度下降算法综述
Keywords 梯度下降 · 优化算法 · SGD · Adam · 深度学习
Summary 本文综述了深度学习中常用的五种优化算法（SGD、Mini-batch SGD、Momentum、Adam、Lion），分析其优缺点并提供实践建议，旨在为模型训练中的算法选择和调参提供指导。
Reason 该论文是对SGD、Adam等优化算法的综述，属于通用深度学习训练方法的总结，不涉及文档图像理解（DIU）任务或任何多模态模型架构、视觉表示、OCR-free推理等核心技术改进。也未提出可迁移至DIU领域的新型推理机制、视觉处理结构或幻觉抑制方法，属于基础优化技术回顾，与DIU无直接关联。
Abstract Focusing on the practical configuration needs of optimization algorithms in deep learning, this article concentrates on five major algorithms: SGD, Mini-batch SGD, Momentum, Adam, and Lion. It systematically analyzes the core advantages, limitations, and key practical recommendations of each algorithm. The research aims to gain an in-depth understanding of these algorithms and provide a standardized reference for the reasonable selection, parameter tuning, and performance improvement of optimization algorithms in both academic research and engineering practice, helping to solve optimization challenges in different scales of models and various training scenarios.
Authors Deng Fucheng, Wang Wanjie, Gong Ao, Wang Xiaoqi, Wang Fan
Categories Artificial Intelligence
0
DinoLizer: Learning from the Best for Generative Inpainting Localization
Title CN DinoLizer：从最优中学习生成性修复定位
Keywords 图像取证 · 生成性修复检测 · DINOv2 · Vision Transformer · 篡改定位
Summary 本文提出DinoLizer，一种基于DINOv2的模型，用于定位生成性修复操作中的篡改区域。该方法在DINOv2预训练基础上添加线性分类头，对ViT的图像块嵌入进行操作，以14×14的分辨率预测篡改区域，并采用滑动窗口策略处理大图，最后通过后处理生成二值化掩码。实验表明其在多种生成模型产生的修复图像上均优于现有方法，且对缩放、噪声、JPEG压缩等后处理具有鲁棒性。研究验证了ViT在深度伪造定位任务中的表征能力。
Reason 该论文属于图像取证领域，核心任务是检测生成式修复（inpainting）导致的篡改区域，属于纯图像伪造检测范畴。尽管使用了DINOv2和ViT架构，但其方法目标与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。未涉及OCR、多模态语言模型、文本结构理解或文档排版建模。虽然定位细粒度视觉变化的技术看似相关，但无法直接迁移用于解决OCR幻觉、复杂文档推理或长文本理解等DIU痛点，且属于无关应用方向。
Abstract We introduce DinoLizer, a DINOv2-based model for localizing manipulated regions in generative inpainting. Our method builds on a DINOv2 model pretrained to detect synthetic images on the B-Free dataset. We add a linear classification head on top of the Vision Transformer's patch embeddings to predict manipulations at a $14\times 14$ patch resolution. The head is trained to focus on semantically altered regions, treating non-semantic edits as part of the original content. Because the ViT accepts only fixed-size inputs, we use a sliding-window strategy to aggregate predictions over larger images; the resulting heatmaps are post-processed to refine the estimated binary manipulation masks. Empirical results show that DinoLizer surpasses state-of-the-art local manipulation detectors on a range of inpainting datasets derived from different generative models. It remains robust to common post-processing operations such as resizing, noise addition, and JPEG (double) compression. On average, DinoLizer achieves a 12\% higher Intersection-over-Union (IoU) than the next best model, with even greater gains after post-processing. Our experiments with off-the-shelf DINOv2 demonstrate the strong representational power of Vision Transformers for this task. Finally, extensive ablation studies comparing DINOv2 and its successor, DINOv3, in deepfake localization confirm DinoLizer's superiority. The code will be publicly available upon acceptance of the paper.
Authors Minh Thong Doi, Jan Butora, Vincent Itier, Jérémie Boulanger, Patrick Bas
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Foundry: Distilling 3D Foundation Models for the Edge
Title CN Foundry：为边缘设备蒸馏3D基础模型
Keywords 3D点云 · 模型压缩 · 知识蒸馏 · 基础模型 · 边缘计算
Summary 本文提出了一种名为Foundation Model Distillation（FMD）的新范式，用于将大规模自监督学习得到的3D点云基础模型压缩为轻量级代理模型Foundry。该方法通过让学生模型学习重建教师模型的token级表示来保留基础模型的通用表征能力，在分类、部件分割等下游任务中接近原始模型性能的同时显著降低计算开销。
Reason 该论文研究的是3D点云领域的基础模型压缩，属于3D视觉与具身智能/机器人相关领域，与文档图像理解（DIU）任务无直接关联。尽管其提出的基础模型蒸馏（FMD）方法在理论上可能具有通用性，但其应用场景（3D点云、AR/VR、机器人）和数据模态与文档图像差异巨大，迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等问题极为困难，不具备对DIU领域的实际迁移价值。
Abstract Foundation models pre-trained with self-supervised learning (SSL) on large-scale datasets have become powerful general-purpose feature extractors. However, their immense size and computational cost make them prohibitive for deployment on edge devices such as robots and AR/VR headsets. Existing compression techniques like standard knowledge distillation create efficient 'specialist' models but sacrifice the crucial, downstream-agnostic generality that makes foundation models so valuable. In this paper, we introduce Foundation Model Distillation (FMD), a new paradigm for compressing large SSL models into compact, efficient, and faithful proxies that retain their general-purpose representational power. We present Foundry, the first implementation of FMD for 3D point clouds. Our approach, Foundry, trains a student to learn a compressed set of SuperTokens that reconstruct the teacher's token-level representations, capturing a compact basis of its latent space. A single distilled model maintains strong transferability across diverse downstream tasks-classification, part segmentation, and few-shot scenarios-approaching full foundation-model performance while using significantly fewer tokens and FLOPs, making such models more practical for deployment on resourceconstrained hardware.
Authors Guillaume Letellier, Siddharth Srivastava, Frédéric Jurie, Gaurav Sharma
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
DeeAD: Dynamic Early Exit of Vision-Language Action for Efficient Autonomous Driving
Title CN DeeAD：面向高效自动驾驶的视觉-语言动作动态早退
Keywords 视觉-语言模型 · 早退机制 · 推理加速 · 自动驾驶 · 无需训练
Summary 本文提出DeeAD，一种无需训练的动态早退框架，用于加速自动驾驶中视觉-语言动作模型的推理过程。通过评估中间轨迹的物理可行性，并结合轻量级规划先验判断是否终止深层推理，从而减少计算延迟。实验显示可实现最高28%的层稀疏性和29%的延迟降低，同时保持规划质量与安全性。
Reason 该论文属于自动驾驶领域，研究的是视觉-语言动作模型在轨迹规划中的推理加速问题。尽管其提出的方法（如无需训练的早退机制、多跳控制器）在推理效率优化方面有一定创新，但应用场景与文档图像理解（DIU）完全无关，无法迁移解决OCR幻觉、复杂排版、细粒度定位或长文档推理等核心痛点。
Abstract Vision-Language Action (VLA) models unify perception, reasoning, and trajectory generation for autonomous driving, but suffer from significant inference latency due to deep transformer stacks. We present DeeAD, a training-free, action-guided early-exit framework that accelerates VLA planning by evaluating the physical feasibility of intermediate trajectories. Instead of relying on confidence scores, DeeAD terminates inference when predicted trajectories align with lightweight planning priors (e.g., Navigation or Low-precision Planning) within a tolerable deviation (<2m). To improve efficiency, we introduce a multi-hop controller that adaptively skips redundant layers based on the change rate of scores. DeeAD integrates into existing VLA models, such as ORION, without requiring retraining. Experiments on the Bench2Drive benchmark demonstrate up to 28% transformer-layer sparsity and 29% latency reduction, while preserving planning quality and safety.
Authors Haibo HU, Lianming Huang, Nan Guan, Chun Jason Xue
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Learning Multi-Access Point Coordination in Agentic AI Wi-Fi with Large Language Models
Title CN 利用大型语言模型在代理式AI Wi-Fi中学习多接入点协调
Keywords Agentic AI · Wi-Fi · 多接入点协调 · 大语言模型代理 · 智能网络
Summary 本文提出一种基于大语言模型代理的Wi-Fi框架，每个接入点作为一个自主代理，通过自然语言对话进行协作推理和实时协调策略协商，以应对动态网络环境。虽然展示了代理系统在复杂系统调控中的潜力，但应用场景局限于无线通信领域。
Reason 该论文属于具身智能与通信系统的交叉应用，研究的是Wi-Fi网络中的多接入点协调问题，使用LLM作为代理进行网络决策。尽管涉及‘Agentic AI’，但其任务场景与文档图像理解（DIU）完全无关，无法迁移至OCR、DocVQA、版面分析等核心问题，属于典型的平行领域应用。
Abstract Multi-access point coordination (MAPC) is a key technology for enhancing throughput in next-generation Wi-Fi within dense overlapping basic service sets. However, existing MAPC protocols rely on static, protocol-defined rules, which limits their ability to adapt to dynamic network conditions such as varying interference levels and topologies. To address this limitation, we propose a novel Agentic AI Wi-Fi framework where each access point, modeled as an autonomous large language model agent, collaboratively reasons about the network state and negotiates adaptive coordination strategies in real time. This dynamic collaboration is achieved through a cognitive workflow that enables the agents to engage in natural language dialogue, leveraging integrated memory, reflection, and tool use to ground their decisions in past experience and environmental feedback. Comprehensive simulation results demonstrate that our agentic framework successfully learns to adapt to diverse and dynamic network environments, significantly outperforming the state-of-the-art spatial reuse baseline and validating its potential as a robust and intelligent solution for future wireless networks.
Authors Yifan Fan, Le Liang, Peng Liu, Xiao Li, Ziyang Guo, Qiao Lan, Shi Jin, Wen Tong
Categories Artificial Intelligence
0
Inferix: A Block-Diffusion based Next-Generation Inference Engine for World Simulation
Title CN Inferix：一种基于块扩散的下一代推理引擎，用于世界模拟
Keywords 世界模型 · 块扩散 · 视频生成 · 半自回归解码 · KV Cache
Summary 本文提出了Inferix，一个专为世界模拟设计的下一代推理引擎，采用块扩散（block-diffusion）的半自回归解码范式，结合扩散模型与自回归优势，在生成高质量、长序列视频的同时实现高效推理和实时交互。系统支持LV-Bench基准测试，旨在推动具身智能、代理AI和游戏中的世界模型发展。
Reason 该论文聚焦于世界模型和视频生成，属于视频理解与生成领域，其核心任务是长时程、物理真实的视频模拟和交互式视频流。尽管提到了半自回归解码和KV Cache优化，但其应用场景与文档图像理解（DIU）的核心痛点（如OCR幻觉、密集文本识别、复杂版面分析）无直接关联。方法难以迁移到文档视觉理解任务中，且未涉及任何文档结构、文字布局或VQA相关建模。
Abstract World models serve as core simulators for fields such as agentic AI, embodied AI, and gaming, capable of generating long, physically realistic, and interactive high-quality videos. Moreover, scaling these models could unlock emergent capabilities in visual perception, understanding, and reasoning, paving the way for a new paradigm that moves beyond current LLM-centric vision foundation models. A key breakthrough empowering them is the semi-autoregressive (block-diffusion) decoding paradigm, which merges the strengths of diffusion and autoregressive methods by generating video tokens in block-applying diffusion within each block while conditioning on previous ones, resulting in more coherent and stable video sequences. Crucially, it overcomes limitations of standard video diffusion by reintroducing LLM-style KV Cache management, enabling efficient, variable-length, and high-quality generation. Therefore, Inferix is specifically designed as a next-generation inference engine to enable immersive world synthesis through optimized semi-autoregressive decoding processes. This dedicated focus on world simulation distinctly sets it apart from systems engineered for high-concurrency scenarios (like vLLM or SGLang) and from classic video diffusion models (such as xDiTs). Inferix further enhances its offering with interactive video streaming and profiling, enabling real-time interaction and realistic simulation to accurately model world dynamics. Additionally, it supports efficient benchmarking through seamless integration of LV-Bench, a new fine-grained evaluation benchmark tailored for minute-long video generation scenarios. We hope the community will work together to advance Inferix and foster world model exploration.
Authors Inferix Team, Tianyu Feng, Yizeng Han, Jiahao He, Yuanyu He, Xi Lin, Teng Liu, Hanfeng Lu, Jiasheng Tang, Wei Wang, Zhiyuan Wang, Jichao Wu, Mingyang Yang, Yinghao Yu, Zeyu Zhang, Bohan Zhuang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Infinity-RoPE: Action-Controllable Infinite Video Generation Emerges From Autoregressive Self-Rollout
Title CN Infinity-RoPE：通过自回归自展开涌现的可控制无限视频生成
Keywords 视频生成 · 扩散模型 · 位置编码 · 自回归模型 · KV缓存
Summary 本文提出了一种名为Infinity-RoPE的推理时框架，用于实现无限时长、可控且具有电影式转场的视频生成。通过Block-Relativistic RoPE、KV Flush和RoPE Cut三个组件，解决了现有自回归视频扩散模型在时间范围、动作响应性和场景切换上的限制。实验表明其在VBench评分上优于先前方法。
Reason 该论文属于视频生成领域，研究的是长时程视频扩散模型中的位置编码与推理控制问题，尽管其方法涉及时间序列扩展和KV缓存操作，但核心任务与文档图像理解（DIU）无关。未涉及OCR、文档布局分析、表格识别或任何DocVQA相关任务，且无法直接迁移用于解决文档理解中的幻觉、密集文本或复杂排版等痛点。
Abstract Current autoregressive video diffusion models are constrained by three core bottlenecks: (i) the finite temporal horizon imposed by the base model's 3D Rotary Positional Embedding (3D-RoPE), (ii) slow prompt responsiveness in maintaining fine-grained action control during long-form rollouts, and (iii) the inability to realize discontinuous cinematic transitions within a single generation stream. We introduce $\infty$-RoPE, a unified inference-time framework that addresses all three limitations through three interconnected components: Block-Relativistic RoPE, KV Flush, and RoPE Cut. Block-Relativistic RoPE reformulates temporal encoding as a moving local reference frame, where each newly generated latent block is rotated relative to the base model's maximum frame horizon while earlier blocks are rotated backward to preserve relative temporal geometry. This relativistic formulation eliminates fixed temporal positions, enabling continuous video generation far beyond the base positional limits. To obtain fine-grained action control without re-encoding, KV Flush renews the KV cache by retaining only two latent frames, the global sink and the last generated latent frame, thereby ensuring immediate prompt responsiveness. Finally, RoPE Cut introduces controlled discontinuities in temporal RoPE coordinates, enabling multi-cut scene transitions within a single continuous rollout. Together, these components establish $\infty$-RoPE as a training-free foundation for infinite-horizon, controllable, and cinematic video diffusion. Comprehensive experiments show that $\infty$-RoPE consistently surpasses previous autoregressive models in overall VBench scores.
Authors Hidir Yesiltepe, Tuna Han Salih Meral, Adil Kaan Akan, Kaan Oktay, Pinar Yanardag
Comment Project Page: https://infinity-rope.github.io/
Categories Computer Vision and Pattern Recognition
0
Cloud4D: Estimating Cloud Properties at a High Spatial and Temporal Resolution NeurIPS 2025 Spotlight
Title CN Cloud4D：以高空间和时间分辨率估计云属性
Keywords 气象建模 · 3D重建 · Transformer · 高分辨率估计 · 地基相机
Summary 该研究提出了Cloud4D，一种基于学习的框架，利用同步的地基相机通过同源变换引导的2D到3D Transformer，重建具有物理一致性的四维云状态，在25米空间分辨率和5秒时间分辨率下推断液态水含量分布，并进一步估计风速向量。系统在时空分辨率上比现有卫星观测提升一个数量级，且相对于雷达测量误差小于10%。
Reason 该论文属于气象与遥感领域，研究利用地面相机重建四维云状态，核心任务为3D气象建模与物理量估计，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管使用了基于Transformer的2D到3D转换架构，但其目标是大气物理状态推断，无法迁移解决OCR幻觉、复杂排版、细粒度文本定位等DIU核心问题。
Abstract There has been great progress in improving numerical weather prediction and climate models using machine learning. However, most global models act at a kilometer-scale, making it challenging to model individual clouds and factors such as extreme precipitation, wind gusts, turbulence, and surface irradiance. Therefore, there is a need to move towards higher-resolution models, which in turn require high-resolution real-world observations that current instruments struggle to obtain. We present Cloud4D, the first learning-based framework that reconstructs a physically consistent, four-dimensional cloud state using only synchronized ground-based cameras. Leveraging a homography-guided 2D-to-3D transformer, Cloud4D infers the full 3D distribution of liquid water content at 25 m spatial and 5 s temporal resolution. By tracking the 3D liquid water content retrievals over time, Cloud4D additionally estimates horizontal wind vectors. Across a two-month deployment comprising six skyward cameras, our system delivers an order-of-magnitude improvement in space-time resolution relative to state-of-the-art satellite measurements, while retaining single-digit relative error ($<10\%$) against collocated radar measurements. Code and data are available on our project page https://cloud4d.jacob-lin.com/.
Authors Jacob Lin, Edward Gryspeerdt, Ronald Clark
Comment NeurIPS 2025 Spotlight, project page: https://cloud4d.jacob-lin.com/
Categories Computer Vision and Pattern Recognition
0
Diverse Video Generation with Determinantal Point Process-Guided Policy Optimization
Title CN 基于行列式点过程引导策略优化的多样化视频生成
Keywords 文本到视频生成 · 多样性生成 · DPP · GRPO · 扩散模型
Summary 本文提出DPP-GRPO框架，结合行列式点过程（DPP）和组相对策略优化（GRPO），旨在从单一文本提示生成多样化的视频内容，提升在视觉外观、相机运动和场景结构上的多样性，同时保持提示对齐和感知质量。实验表明该方法在多个基准上提升了多样性表现。
Reason 该论文研究的是文本到视频生成中的多样性问题，属于视频生成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心方法DPP-GRPO虽在视频生成中提升多样性，但无法迁移解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点。
Abstract While recent text-to-video (T2V) diffusion models have achieved impressive quality and prompt alignment, they often produce low-diversity outputs when sampling multiple videos from a single text prompt. We tackle this challenge by formulating it as a set-level policy optimization problem, with the goal of training a policy that can cover the diverse range of plausible outcomes for a given prompt. To address this, we introduce DPP-GRPO, a novel framework for diverse video generation that combines Determinantal Point Processes (DPPs) and Group Relative Policy Optimization (GRPO) theories to enforce explicit reward on diverse generations. Our objective turns diversity into an explicit signal by imposing diminishing returns on redundant samples (via DPP) while supplies groupwise feedback over candidate sets (via GRPO). Our framework is plug-and-play and model-agnostic, and encourages diverse generations across visual appearance, camera motions, and scene structure without sacrificing prompt fidelity or perceptual quality. We implement our method on WAN and CogVideoX, and show that our method consistently improves video diversity on state-of-the-art benchmarks such as VBench, VideoScore, and human preference studies. Moreover, we release our code and a new benchmark dataset of 30,000 diverse prompts to support future research.
Authors Tahira Kazimi, Connor Dunlop, Pinar Yanardag
Comment Project webpage: https://diverse-video.github.io/
Categories Computer Vision and Pattern Recognition
0
LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight
Title CN LocateAnything3D：通过视线链进行视觉语言3D检测
Keywords 3D检测 · 视觉语言模型 · Chain-of-Sight · 开放词汇检测 · next-token预测
Summary 该研究提出LocateAnything3D，将3D检测任务转化为VLM中的下一个词预测问题，引入‘视线链’（Chain-of-Sight）作为中间表示，先进行2D检测，再按由近到远、从中心到尺寸与旋转的顺序逐步预测3D边界框。该方法在Omni3D基准上实现了SOTA性能，并具备开放词汇和零样本泛化能力。
Reason 该论文研究的是3D目标检测，属于具身智能与机器人感知领域，核心任务为从图像中推理物体的三维位置、尺寸和姿态，与文档图像理解（DIU）或DocVQA无关。尽管其采用了VLM框架和Chain-of-Sight等序列化推理策略，但方法设计面向3D空间结构建模，无法迁移至解决文档中的OCR幻觉、复杂排版或长文本推理等核心痛点。属于明确的无关领域。
Abstract To act in the world, a model must name what it sees and know where it is in 3D. Today's vision-language models (VLMs) excel at open-ended 2D description and grounding, yet multi-object 3D detection remains largely missing from the VLM toolbox. We present LocateAnything3D, a VLM-native recipe that casts 3D detection as a next-token prediction problem. The key is a short, explicit Chain-of-Sight (CoS) sequence that mirrors how human reason from images: find an object in 2D, then infer its distance, size, and pose. The decoder first emits 2D detections as a visual chain-of-thought, then predicts 3D boxes under an easy-to-hard curriculum: across objects, a near-to-far order reduces early ambiguity and matches ego-centric utility; within each object, a center-from-camera, dimensions, and rotation factorization ranks information by stability and learnability. This VLM-native interface preserves open-vocabulary and visual-prompting capability without specialized heads. On the challenging Omni3D benchmark, our model achieves state-of-the-art results, with 49.89 AP_3D, surpassing the previous best by +15.51 absolute improvement even when the baseline is given ground-truth 2D boxes. It also generalizes zero-shot to held-out categories with strong robustness. By turning 3D detection into a disciplined next-token problem, LocateAnything3D offers a practical foundation for models to perceive in 3D.
Authors Yunze Man, Shihao Wang, Guowen Zhang, Johan Bjorck, Zhiqi Li, Liang-Yan Gui, Jim Fan, Jan Kautz, Yu-Xiong Wang, Zhiding Yu
Comment Tech report. Project page: https://nvlabs.github.io/LocateAnything3D/
Categories Computer Vision and Pattern Recognition
0
PixelDiT: Pixel Diffusion Transformers for Image Generation
Title CN PixelDiT：用于图像生成的像素扩散变换器
Keywords 图像生成 · 扩散模型 · Transformer · 像素空间建模 · DiT
Summary 本文提出了PixelDiT，一种直接在像素空间进行扩散过程的端到端图像生成模型，采用双层Transformer架构（patch-level和pixel-level）来同时捕捉全局语义和细节纹理，在ImageNet等图像生成基准上取得了优异表现。
Reason 该论文聚焦于图像生成任务，特别是基于扩散模型的图像合成，属于纯图像生成领域。尽管其提出的方法在像素空间进行端到端建模，具有一定的架构创新，但其核心目标与文档图像理解（DIU）无关，无法直接迁移解决OCR幻觉、复杂排版理解或长文档推理等DIU核心问题。
Abstract Latent-space modeling has been the standard for Diffusion Transformers (DiTs). However, it relies on a two-stage pipeline where the pretrained autoencoder introduces lossy reconstruction, leading to error accumulation while hindering joint optimization. To address these issues, we propose PixelDiT, a single-stage, end-to-end model that eliminates the need for the autoencoder and learns the diffusion process directly in the pixel space. PixelDiT adopts a fully transformer-based architecture shaped by a dual-level design: a patch-level DiT that captures global semantics and a pixel-level DiT that refines texture details, enabling efficient training of a pixel-space diffusion model while preserving fine details. Our analysis reveals that effective pixel-level token modeling is essential to the success of pixel diffusion. PixelDiT achieves 1.61 FID on ImageNet 256x256, surpassing existing pixel generative models by a large margin. We further extend PixelDiT to text-to-image generation and pretrain it at the 1024x1024 resolution in pixel space. It achieves 0.74 on GenEval and 83.5 on DPG-bench, approaching the best latent diffusion models.
Authors Yongsheng Yu, Wei Xiong, Weili Nie, Yichen Sheng, Shiqiu Liu, Jiebo Luo
Categories Computer Vision and Pattern Recognition
0
Rethinking the Learning Paradigm for Facial Expression Recognition
Title CN 重新思考面部表情识别的学习范式
Keywords 面部表情识别 · 弱监督学习 · 模糊标注 · 学习范式 · 计算机视觉
Summary 本文针对面部表情识别中因主观标注和类别相似性导致的标签模糊问题，提出应放弃传统的端到端监督训练范式，转而采用弱监督策略直接利用原始模糊标注进行模型训练。通过这种方式，模型能更好地适应真实场景中的不确定性。
Reason 该论文研究的是面部表情识别（FER），属于通用计算机视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无关。尽管其提出的弱监督学习方法在理论上可能适用于其他存在标注模糊性的任务，但该工作并未针对文档结构、OCR幻觉、复杂排版或长文本推理等DIU核心痛点设计，且无任何迁移路径的直接启示。属于无关领域研究。
Abstract Due to the subjective crowdsourcing annotations and the inherent inter-class similarity of facial expressions, the real-world Facial Expression Recognition (FER) datasets usually exhibit ambiguous annotation. To simplify the learning paradigm, most previous methods convert ambiguous annotation results into precise one-hot annotations and train FER models in an end-to-end supervised manner. In this paper, we rethink the existing training paradigm and propose that it is better to use weakly supervised strategies to train FER models with original ambiguous annotation.
Authors Weijie Wang, Nicu Sebe, Bruno Lepri
Categories Computer Vision and Pattern Recognition
0
iMontage: Unified, Versatile, Highly Dynamic Many-to-many Image Generation
Title CN iMontage：统一、多功能、高度动态的多对多图像生成
Keywords 图像生成 · 视频模型迁移 · 多对多生成 · 图像编辑 · 上下文一致性
Summary 本文提出iMontage框架，通过适配预训练视频模型实现灵活的多输入多输出图像生成与编辑，支持变长图像集的生成，在保持跨图像一致性的同时实现高度动态的场景生成。主要用于创意图像生成任务。
Reason 该论文属于纯图像生成领域，研究的是利用视频模型先验进行多对多图像生成与编辑，其核心任务与文档图像理解（DIU）无关。尽管使用了视觉模型，但目标是图像内容的动态生成和编辑，而非文档中的文本识别、布局分析或视觉问答等任务。无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract Pre-trained video models learn powerful priors for generating high-quality, temporally coherent content. While these models excel at temporal coherence, their dynamics are often constrained by the continuous nature of their training data. We hypothesize that by injecting the rich and unconstrained content diversity from image data into this coherent temporal framework, we can generate image sets that feature both natural transitions and a far more expansive dynamic range. To this end, we introduce iMontage, a unified framework designed to repurpose a powerful video model into an all-in-one image generator. The framework consumes and produces variable-length image sets, unifying a wide array of image generation and editing tasks. To achieve this, we propose an elegant and minimally invasive adaptation strategy, complemented by a tailored data curation process and training paradigm. This approach allows the model to acquire broad image manipulation capabilities without corrupting its invaluable original motion priors. iMontage excels across several mainstream many-in-many-out tasks, not only maintaining strong cross-image contextual consistency but also generating scenes with extraordinary dynamics that surpass conventional scopes. Find our homepage at: https://kr1sjfu.github.io/iMontage-web/.
Authors Zhoujie Fu, Xianfang Zeng, Jinghong Lan, Xinyao Liao, Cheng Chen, Junyi Chen, Jiacheng Wei, Wei Cheng, Shiyu Liu, Yunuo Chen, Gang Yu, Guosheng Lin
Categories Computer Vision and Pattern Recognition
0
ShapeGen: Towards High-Quality 3D Shape Synthesis SIGGRAPH Asia 2025
Title CN ShapeGen：迈向高质量3D形状合成
Keywords 3D生成 · 图像到3D · 线性Transformer · 高分辨率生成 · SIGGRAPH
Summary 本文提出ShapeGen，通过改进3D表示与监督策略、提升分辨率以及利用线性Transformer的优势，实现高质量的单图到3D形状生成。实验表明该方法在生成细节丰富、结构完整的3D资产方面达到SOTA水平，适用于3D内容创作 pipeline。
Reason 该论文属于3D视觉与图形学领域，研究的是图像到3D形状的生成问题，关注点在于3D资产的细节、表面质量和结构完整性。其核心技术涉及3D表示、分辨率扩展和线性Transformer的应用，但这些方法针对的是3D几何建模任务，与文档图像理解（DIU）中的OCR幻觉、密集文本识别、复杂版面分析或长文档推理等核心痛点无直接关联。无法迁移用于解决文档理解中的多模态对齐、细粒度定位或高分辨率文档处理等问题。
Abstract Inspired by generative paradigms in image and video, 3D shape generation has made notable progress, enabling the rapid synthesis of high-fidelity 3D assets from a single image. However, current methods still face challenges, including the lack of intricate details, overly smoothed surfaces, and fragmented thin-shell structures. These limitations leave the generated 3D assets still one step short of meeting the standards favored by artists. In this paper, we present ShapeGen, which achieves high-quality image-to-3D shape generation through 3D representation and supervision improvements, resolution scaling up, and the advantages of linear transformers. These advancements allow the generated assets to be seamlessly integrated into 3D pipelines, facilitating their widespread adoption across various applications. Through extensive experiments, we validate the impact of these improvements on overall performance. Ultimately, thanks to the synergistic effects of these enhancements, ShapeGen achieves a significant leap in image-to-3D generation, establishing a new state-of-the-art performance.
Authors Yangguang Li, Xianglong He, Zi-Xin Zou, Zexiang Liu, Wanli Ouyang, Ding Liang, Yan-Pei Cao
Comment Accepted to SIGGRAPH Asia 2025
Categories Computer Vision and Pattern Recognition
0
Wanderland: Geometrically Grounded Simulation for Open-World Embodied AI
Title CN Wanderland：面向开放世界具身AI的几何接地仿真
Keywords 具身AI · 几何重建 · 仿真 · 视觉导航 · 新视角合成
Summary 本文提出Wanderland，一个面向开放世界具身AI的高保真仿真框架，通过多传感器采集和几何精确的三维重建，构建室内外城市场景数据集，用于评估视觉导航策略、3D重建与新视角合成。研究表明纯图像方法扩展性差，几何质量对仿真可靠性有显著影响。该工作旨在推动具身AI的可复现研究。
Reason 该论文属于具身智能与3D视觉仿真领域，研究重点为开放世界环境中的视觉导航和多传感器重建，与文档图像理解（DIU）或视觉语言模型在文档任务中的应用无直接关联。其技术路线聚焦于场景几何重建与机器人交互，无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心问题。
Abstract Reproducible closed-loop evaluation remains a major bottleneck in Embodied AI such as visual navigation. A promising path forward is high-fidelity simulation that combines photorealistic sensor rendering with geometrically grounded interaction in complex, open-world urban environments. Although recent video-3DGS methods ease open-world scene capturing, they are still unsuitable for benchmarking due to large visual and geometric sim-to-real gaps. To address these challenges, we introduce Wanderland, a real-to-sim framework that features multi-sensor capture, reliable reconstruction, accurate geometry, and robust view synthesis. Using this pipeline, we curate a diverse dataset of indoor-outdoor urban scenes and systematically demonstrate how image-only pipelines scale poorly, how geometry quality impacts novel view synthesis, and how all of these adversely affect navigation policy learning and evaluation reliability. Beyond serving as a trusted testbed for embodied navigation, Wanderland's rich raw sensor data further allows benchmarking of 3D reconstruction and novel view synthesis models. Our work establishes a new foundation for reproducible research in open-world embodied AI. Project website is at https://ai4ce.github.io/wanderland/.
Authors Xinhao Liu, Jiaqi Li, Youming Deng, Ruxin Chen, Yingjia Zhang, Yifei Ma, Li Guo, Yiming Li, Jing Zhang, Chen Feng
Categories Computer Vision and Pattern Recognition
0
The Consistency Critic: Correcting Inconsistencies in Generated Images via Reference-Guided Attentive Alignment
Title CN 一致性批评者：通过参考引导的注意力对齐修正生成图像中的不一致
Keywords 图像生成 · 后编辑 · 注意力对齐 · 一致性修正 · VLM
Summary 本文提出ImageCritic，一种基于参考引导的注意力对齐方法，用于修正生成图像中的细粒度不一致问题。作者构建了包含参考-退化-目标三元组的数据集，并设计了注意力对齐损失和细节编码器来精确定位并修复不一致区域。该方法可集成到代理框架中，实现多轮局部编辑，在多种定制化生成场景中表现出色。
Reason 该论文聚焦于图像生成任务中的细粒度细节一致性问题，属于纯图像生成与修复领域。其核心方法是针对生成图像与参考图像之间的不一致进行后编辑修正，应用场景为定制化图像生成（如风格迁移、图像编辑），与文档图像理解（DIU）中的OCR幻觉、密集文本识别、复杂版式分析或长文档推理等核心痛点无关。尽管使用了VLM和注意力机制，但其技术路径无法直接迁移用于解决文档理解中的语义结构解析或视觉-文本对齐问题。
Abstract Previous works have explored various customized generation tasks given a reference image, but they still face limitations in generating consistent fine-grained details. In this paper, our aim is to solve the inconsistency problem of generated images by applying a reference-guided post-editing approach and present our ImageCritic. We first construct a dataset of reference-degraded-target triplets obtained via VLM-based selection and explicit degradation, which effectively simulates the common inaccuracies or inconsistencies observed in existing generation models. Furthermore, building on a thorough examination of the model's attention mechanisms and intrinsic representations, we accordingly devise an attention alignment loss and a detail encoder to precisely rectify inconsistencies. ImageCritic can be integrated into an agent framework to automatically detect inconsistencies and correct them with multi-round and local editing in complex scenarios. Extensive experiments demonstrate that ImageCritic can effectively resolve detail-related issues in various customized generation scenarios, providing significant improvements over existing methods.
Authors Ziheng Ouyang, Yiren Song, Yaoli Liu, Shihao Zhu, Qibin Hou, Ming-Ming Cheng, Mike Zheng Shou
Comment Project page: https://ouyangziheng.github.io/ImageCritic-Page/
Categories Computer Vision and Pattern Recognition
0
Optimization of Sums of Bivariate Functions: An Introduction to Relaxation-Based Methods for the Case of Finite Domains
Title CN 双变量函数和的优化：有限域情况下基于松弛方法的介绍
Keywords 优化 · 双变量函数 · 松弛方法 · 线性规划 · 熵正则化
Summary 本文探讨在有限域上对可分解为多个双变量函数之和的目标函数进行优化的问题。作者通过测度值扩展（松弛）、ℓ²逼近和熵正则化等方法，提出了若干可通过线性规划、坐标上升或闭式解求解的可处理形式，并利用关于双变量边缘重建测度的一般性结果分析了这些松弛方法的局限性。实验部分应用于随机函数、顶点着色和信号重建问题。
Reason 该论文研究的是双变量函数和的优化问题，属于理论数学与优化领域，尽管被归类于Computer Vision and Pattern Recognition，但其内容与文档图像理解（DIU）、多模态大模型、视觉语言任务、OCR或VQA等核心问题无直接关联。未涉及视觉表示、VLM架构、高分辨率处理、细粒度定位或文档推理等可迁移至DIU的关键技术。属于无关领域的理论工作，无法为解决OCR幻觉、复杂排版或长文档理解提供方法论支持。
Abstract We study the optimization of functions with $n>2$ arguments that have a representation as a sum of several functions that have only $2$ of the $n$ arguments each, termed sums of bivariates, on finite domains. The complexity of optimizing sums of bivariates is shown to be NP-equivalent and it is shown that there exists free lunch in the optimization of sums of bivariates. Based on measure-valued extensions of the objective function, so-called relaxations, $\ell^2$-approximation, and entropy-regularization, we derive several tractable problem formulations solvable with linear programming, coordinate ascent as well as with closed-form solutions. The limits of applying tractable versions of such relaxations to sums of bivariates are investigated using general results for reconstructing measures from their bivariate marginals. Experiments in which the derived algorithms are applied to random functions, vertex coloring, and signal reconstruction problems provide insights into qualitatively different function classes that can be modeled as sums of bivariates.
Authors Nils Müller
Comment 59 pages, 7 figures
Categories Computer Vision and Pattern Recognition
0
Latent Diffusion Inversion Requires Understanding the Latent Space
Title CN 潜在扩散反演需要理解潜在空间
Keywords 潜在扩散模型 · 模型反演 · 隐私泄露 · 记忆化 · 潜在空间分析
Summary 本文发现潜在扩散模型在不同潜在维度上存在不均匀的记忆化现象，并提出一种基于解码器pullback度量的维度排序方法，以识别对记忆贡献最大的潜在维度。实验表明移除低记忆性维度可显著提升成员推断攻击的性能。研究聚焦于生成模型的隐私风险分析，未涉及任何文档结构、文本理解或多模态问答内容。
Reason 该论文研究的是生成模型（特别是潜在扩散模型）中的数据记忆化与隐私泄露问题，属于纯图像生成领域的安全分析，与文档图像理解（DIU）或DocVQA的核心任务无关。虽然涉及潜在空间表示，但其目标是提升成员推断攻击性能，无法迁移用于解决OCR幻觉、复杂排版解析或长文档推理等DIU痛点。
Abstract The recovery of training data from generative models (``model inversion'') has been extensively studied for diffusion models in the data domain. The encoder/decoder pair and corresponding latent codes have largely been ignored by inversion techniques applied to latent space generative models, e.g., Latent Diffusion models (LDMs). In this work we describe two key findings: (1) The diffusion model exhibits non-uniform memorization across latent codes, tending to overfit samples located in high-distortion regions of the decoder pullback metric. (2) Even within a single latent code, different dimensions contribute unequally to memorization. We introduce a principled method to rank latent dimensions by their per-dimensional contribution to the decoder pullback metric, identifying those most responsible for memorization. Empirically, removing less-memorizing dimensions when computing attack statistics for score-based membership inference attacker significantly improves performance, with average AUROC gains of 2.7\% and substantial increases in TPR@1\%FPR (6.42\%) across diverse datasets including CIFAR-10, CelebA, ImageNet-1K, Pokémon, MS-COCO, and Flickr. This indicates stronger confidence in identifying members under extremely low false-positive tolerance. Our results highlight the overlooked influence of the auto-encoder geometry on LDM memorization and provide a new perspective for analyzing privacy risks in diffusion-based generative models.
Authors Mingxing Rao, Bowen Qu, Daniel Moyer
Comment 14 pages, 4 figures, 4 tables
Categories Computer Vision and Pattern Recognition
0
VQ-VA World: Towards High-Quality Visual Question-Visual Answering
Title CN VQ-VA世界：迈向高质量视觉问答-视觉回答
Keywords 视觉生成 · 视觉问答 · 数据构建 · 代理流水线 · 评测基准
Summary 本文提出VQ-VA World，一个面向视觉问答-视觉回答（VQ-VA）任务的数据构建框架，通过代理式流水线从网络规模数据中收集约180万高质量图文交错样本，并发布评测基准IntelligentBench。实验表明，基于该数据训练的LightFusion模型在IntelligentBench上显著超越先前开源模型。作者开源了模型权重、数据和流程。
Reason 该论文研究的是视觉问答生成图像（Visual Question-Visual Answering, VQ-VA），即根据视觉问题生成一张图像，属于视觉生成任务，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等完全无关。其目标是复现类似GPT-Image的生成能力，而非解决OCR幻觉、密集文本处理或复杂排版理解等文档理解痛点。尽管使用了大规模数据构建流程，但属于特定生成任务的平行应用，无法迁移用于提升文档理解性能。
Abstract This paper studies Visual Question-Visual Answering (VQ-VA): generating an image, rather than text, in response to a visual question -- an ability that has recently emerged in proprietary systems such as NanoBanana and GPT-Image. To also bring this capability to open-source models, we introduce VQ-VA World, a data-centric framework built around an agentic pipeline for large-scale, targeted data construction. Leveraging web-scale deployment, this pipeline crawls a massive amount of ~1.8M high-quality, interleaved image-text samples for model training. For evaluation, we further release IntelligentBench, a human-curated benchmark that systematically assesses VQ-VA along the aspects of world knowledge, design knowledge, and reasoning. Training with VQ-VA World data yields strong empirical gains: it helps LightFusion attain 53.06 on IntelligentBench, substantially surpassing the best prior open-source baselines (i.e., 7.78 from vanilla LightFusion; 1.94 from UniWorld-V1), and significantly narrowing the gap toward leading proprietary systems (e.g., 81.67 from NanoBanana; 82.64 from GPT-Image). By releasing the full suite of model weights, datasets, and pipelines, we hope to stimulate future research on VQ-VA.
Authors Chenhui Gou, Zilong Chen, Zeyu Wang, Feng Li, Deyao Zhu, Zicheng Duan, Kunchang Li, Chaorui Deng, Hongyi Yuan, Haoqi Fan, Cihang Xie, Jianfei Cai, Hamid Rezatofighi
Categories Computer Vision and Pattern Recognition
0
A Reason-then-Describe Instruction Interpreter for Controllable Video Generation
Title CN 一种用于可控视频生成的先推理后描述指令解释器
Keywords 视频生成 · 指令解释 · 扩散Transformer · 意图对齐 · Reason-then-Describe
Summary 本文提出ReaDe，一种模型无关的指令解释器，通过‘先推理后描述’的方式将用户模糊的生成指令转化为精确的可执行规范，提升可控视频生成的意图对齐能力。该方法专注于视频生成场景，依赖于对动态内容和时序逻辑的理解，与文档图像理解任务关联度极低。
Reason 该论文研究的是可控视频生成中的指令解释问题，属于视频理解与生成领域。尽管其提出的‘先推理后描述’范式具有一定方法论意义，但核心应用场景与文档图像理解（DIU）无关，无法直接迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心痛点。
Abstract Diffusion Transformers have significantly improved video fidelity and temporal coherence, however, practical controllability remains limited. Concise, ambiguous, and compositionally complex user inputs contrast with the detailed prompts used in training, yielding an intent-output mismatch. We propose ReaDe, a universal, model-agnostic interpreter that converts raw instructions into precise, actionable specifications for downstream video generators. ReaDe follows a reason-then-describe paradigm: it first analyzes the user request to identify core requirements and resolve ambiguities, then produces detailed guidance that enables faithful, controllable generation. We train ReaDe via a two-stage optimization: (i) reasoning-augmented supervision imparts analytic parsing with stepwise traces and dense captions, and (ii) a multi-dimensional reward assigner enables stable, feedback-driven refinement for natural-style captions. Experiments across single- and multi-condition scenarios show consistent gains in instruction fidelity, caption accuracy, and downstream video quality, with strong generalization to reasoning-intensive and unseen inputs. ReaDe offers a practical route to aligning controllable video generation with accurately interpreted user intent. Project Page: https://sqwu.top/ReaDe/.
Authors Shengqiong Wu, Weicai Ye, Yuanxing Zhang, Jiahao Wang, Quande Liu, Xintao Wang, Pengfei Wan, Kun Gai, Hao Fei, Tat-Seng Chua
Comment 27 pages, 13 figures, 13 tables, Project Page: https://sqwu.top/ReaDe/
Categories Computer Vision and Pattern Recognition
0
PhysChoreo: Physics-Controllable Video Generation with Part-Aware Semantic Grounding
Title CN PhysChoreo：基于部分感知语义定位的物理可控视频生成
Keywords 视频生成 · 物理仿真 · 语义定位 · 动态控制 · 计算机视觉
Summary 本文提出PhysChoreo，一种通过分阶段建模物体静态物理属性并进行时序可控物理仿真的视频生成框架，旨在提升生成视频的物理真实性和动态可控性。实验表明其在多个指标上优于现有方法。
Reason 该论文属于视频生成与物理仿真领域，核心目标是实现从单张图像生成具有物理真实感和可控制动态行为的视频。其技术重点在于物理属性重建与时间序列模拟，与文档图像理解（DIU）中的文本识别、布局分析、表格理解等任务无直接关联。尽管涉及‘语义定位’，但其目的是服务于物体部件的物理动力学控制，而非文档中的文本细粒度对齐或视觉元素结构化。无法迁移用于解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract While recent video generation models have achieved significant visual fidelity, they often suffer from the lack of explicit physical controllability and plausibility. To address this, some recent studies attempted to guide the video generation with physics-based rendering. However, these methods face inherent challenges in accurately modeling complex physical properties and effectively control ling the resulting physical behavior over extended temporal sequences. In this work, we introduce PhysChoreo, a novel framework that can generate videos with diverse controllability and physical realism from a single image. Our method consists of two stages: first, it estimates the static initial physical properties of all objects in the image through part-aware physical property reconstruction. Then, through temporally instructed and physically editable simulation, it synthesizes high-quality videos with rich dynamic behaviors and physical realism. Experimental results show that PhysChoreo can generate videos with rich behaviors and physical realism, outperforming state-of-the-art methods on multiple evaluation metrics.
Authors Haoze Zhang, Tianyu Huang, Zichen Wan, Xiaowei Jin, Hongzhi Zhang, Hui Li, Wangmeng Zuo
Categories Computer Vision and Pattern Recognition
0
Personalized Generative Low-light Image Denoising and Enhancement
Title CN 个性化的生成式低光照图像去噪与增强
Keywords 图像去噪 · 低光照增强 · 扩散模型 · 个性化生成 · 幻觉抑制
Summary 本文提出DiffPGD方法，利用用户个人照片库构建身份一致的物理缓冲区，作为先验信息嵌入扩散模型中，实现无需微调的个性化低光照图像去噪与增强。在多种低光场景下表现优于现有扩散模型方法。
Reason 该论文研究的是低光照条件下的图像去噪与增强问题，属于纯图像恢复领域，且核心关注点为生成模型在低信噪比下的幻觉问题。虽然提到了‘hallucination’，但其场景是自然图像的物理噪声建模与身份一致性先验，并不涉及文档图像中的文本语义、排版结构或OCR相关幻觉。无法迁移解决文档图像理解中的核心痛点，如密集文本识别、逻辑推理或多模态对齐。属于无关领域。
Abstract Modern cameras' performance in low-light conditions remains suboptimal due to fundamental limitations in photon shot noise and sensor read noise. Generative image restoration methods have shown promising results compared to traditional approaches, but they suffer from hallucinatory content generation when the signal-to-noise ratio (SNR) is low. Leveraging the availability of personalized photo galleries of the users, we introduce Diffusion-based Personalized Generative Denoising (DiffPGD), a new approach that builds a customized diffusion model for individual users. Our key innovation lies in the development of an identity-consistent physical buffer that extracts the physical attributes of the person from the gallery. This ID-consistent physical buffer serves as a robust prior that can be seamlessly integrated into the diffusion model to restore degraded images without the need for fine-tuning. Over a wide range of low-light testing scenarios, we show that DiffPGD achieves superior image denoising and enhancement performance compared to existing diffusion-based denoising approaches. Our project page can be found at \href{https://genai-restore.github.io/DiffPGD/}{\textcolor{purple}{\textbf{https://genai-restore.github.io/DiffPGD/}}}.
Authors Xijun Wang, Prateek Chennuri, Dilshan Godaliyadda, Yu Yuan, Bole Ma, Xingguang Zhang, Hamid R. Sheikh, Stanley Chan
Categories Computer Vision and Pattern Recognition
0
Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos
Title CN 错误归因：在第一人称视频中进行细粒度的错误理解
Keywords 错误归因 · 第一人称视频 · 细粒度理解 · 时空定位 · 行为分析
Summary 本文提出了Mistake Attribution（MATT）任务，旨在对第一人称视频中的人类操作错误进行细粒度归因，包括判断哪部分指令被违反、偏差不可逆的时间点（PNR）以及错误在帧中的位置。作者构建了MisEngine数据引擎，自动生成具有丰富归因标注的数据集EPIC-KITCHENS-M和Ego4D-M，并提出统一的注意力模型MisFormer，在语义、时序和空间维度上联合建模错误归因。实验表明该方法在多个基准上优于现有基线。
Reason 该论文研究的是第一人称视角视频中的行为错误归因问题，属于视频理解领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的细粒度归因框架和时空定位机制具有一定方法论价值，但其应用场景聚焦于人类行为与指令语义的对齐，迁移到文档理解中的OCR幻觉消除、复杂排版解析或长文档推理等痛点存在本质领域差异，无法直接提供有效技术迁移路径。
Abstract We introduce Mistake Attribution (MATT), a task for fine-grained understanding of human mistakes in egocentric video. Unlike prior mistake understanding work, which lacks fine-grained output, MATT concretely attributes mistakes to the input instruction text or the attempt video. MATT determines what part of the instruction is violated (semantic role), when the deviation becomes irreversible (the Point-of-No-Return, PNR), and where the mistake appears in the PNR frame. We develop MisEngine, a data engine that automatically constructs attribution-rich mistake samples from existing datasets and inherits their annotations. Applied to large egocentric corpora, MisEngine yields EPIC-KITCHENS-M and Ego4D-M, two datasets that are up to two orders of magnitude larger than prior mistake datasets. We then present MisFormer, a unified attention-based model for mistake attribution across semantic (what), temporal (when), and spatial (where) dimensions, trained using MisEngine supervision. Experiments on our new datasets and prior benchmarks show that MisFormer outperforms strong video-language, temporal localization, hand-object interaction, and mistake-detection baselines.
Authors Yayuan Li, Aadit Jain, Filippos Bellos, Jason J. Corso
Comment 11 pages, 4 figures, 6 tables
Categories Computer Vision and Pattern Recognition
0
A Physics-Informed Loss Function for Boundary-Consistent and Robust Artery Segmentation in DSA Sequences
Title CN 一种用于DSA序列中边界一致且鲁棒的动脉分割的物理信息损失函数
Keywords 医学图像分割 · 物理信息损失 · DSA序列 · 边界一致性 · 深度学习
Summary 本文提出了一种名为物理信息损失（PIL）的新损失函数，用于提升DSA序列中脑动脉分割的准确性和边界一致性。该方法借鉴材料物理中的位错理论，将预测与真实边界之间的交互建模为弹性过程，从而引入物理正则化项以增强轮廓平滑性和结构稳定性。实验表明，PIL在多个网络架构和公开数据集上均优于传统损失函数，提升了分割的敏感性、F1分数和边界连贯性。
Reason 该论文属于医学图像分割领域，研究任务为血管分割，使用DSA序列数据，与文档图像理解（DIU）或DocVQA无关。其提出的方法虽在医学影像中有价值，但核心关注的是生物物理边界的建模，迁移到文档图像中的文本结构、排版分析或视觉语言推理等问题上极为困难，且未涉及OCR、多模态理解、布局识别等DIU核心痛点。
Abstract Accurate extraction and segmentation of the cerebral arteries from digital subtraction angiography (DSA) sequences is essential for developing reliable clinical management models of complex cerebrovascular diseases. Conventional loss functions often rely solely on pixel-wise overlap, overlooking the geometric and physical consistency of vascular boundaries, which can lead to fragmented or unstable vessel predictions. To overcome this limitation, we propose a novel \textit{Physics-Informed Loss} (PIL) that models the interaction between the predicted and ground-truth boundaries as an elastic process inspired by dislocation theory in materials physics. This formulation introduces a physics-based regularization term that enforces smooth contour evolution and structural consistency, allowing the network to better capture fine vascular geometry. The proposed loss is integrated into several segmentation architectures, including U-Net, U-Net++, SegFormer, and MedFormer, and evaluated on two public benchmarks: DIAS and DSCA. Experimental results demonstrate that PIL consistently outperforms conventional loss functions such as Cross-Entropy, Dice, Active Contour, and Surface losses, achieving superior sensitivity, F1 score, and boundary coherence. These findings confirm that the incorporation of physics-based boundary interactions into deep neural networks improves both the precision and robustness of vascular segmentation in dynamic angiographic imaging. The implementation of the proposed method is publicly available at https://github.com/irfantahir301/Physicsis_loss.
Authors Muhammad Irfan, Nasir Rahim, Khalid Mahmood Malik
Categories Computer Vision and Pattern Recognition
0
Modular Deep Learning Framework for Assistive Perception: Gaze, Affect, and Speaker Identification
Title CN 用于辅助感知的模块化深度学习框架：注视、情感和说话人识别
Keywords 辅助技术 · 模块化架构 · 眼状态检测 · 面部表情识别 · 说话人识别
Summary 本文提出一个模块化深度学习框架，用于辅助感知系统，包含三个独立模块：基于CNN的眼状态检测、面部表情识别和基于LSTM的说话人识别。在特定数据集上取得了较高的准确率，旨在为资源受限的实时多模态辅助设备提供基础。但未涉及任何文档理解相关任务或技术。
Reason 该论文研究的是辅助技术中的多模态感知，包括眼动状态检测、面部表情识别和说话人识别，属于通用视觉与音频感知领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其任务目标和技术路线均不涉及OCR、布局分析、表格识别或文档级推理等核心DIU问题，也无法提供可迁移至文档理解领域的底层方法（如视觉压缩、高分辨率处理、去幻觉机制等）。属于无关领域应用。
Abstract Developing comprehensive assistive technologies requires the seamless integration of visual and auditory perception. This research evaluates the feasibility of a modular architecture inspired by core functionalities of perceptive systems like 'Smart Eye.' We propose and benchmark three independent sensing modules: a Convolutional Neural Network (CNN) for eye state detection (drowsiness/attention), a deep CNN for facial expression recognition, and a Long Short-Term Memory (LSTM) network for voice-based speaker identification. Utilizing the Eyes Image, FER2013, and customized audio datasets, our models achieved accuracies of 93.0%, 97.8%, and 96.89%, respectively. This study demonstrates that lightweight, domain-specific models can achieve high fidelity on discrete tasks, establishing a validated foundation for future real-time, multimodal integration in resource-constrained assistive devices.
Authors Akshit Pramod Anchan, Jewelith Thomas, Sritama Roy
Comment 10 pages, 9 figures, and 3 tables
Categories Computer Vision and Pattern Recognition
0
Dance Style Classification using Laban-Inspired and Frequency-Domain Motion Features
Title CN 基于拉班启发和频域运动特征的舞蹈风格分类
Keywords 舞蹈风格分类 · 拉班运动分析 · 频域特征 · 姿态估计 · 动作识别
Summary 本文提出了一种轻量级框架，用于基于视频中提取的姿态估计数据对舞蹈风格进行分类。通过引入受拉班运动分析启发的时空描述符，捕捉上半身关节的速度、加速度和角运动等局部动态特性，并结合快速傅里叶变换（FFT）提取频域特征以表征动作的节奏性和周期性模式。实验表明，该方法无需复杂模型即可实现高效的舞蹈风格分类，且具有良好的可解释性。
Reason 该论文研究的是舞蹈风格分类，属于人类动作识别领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其提出的方法聚焦于姿态估计和运动频率特征，无法迁移解决OCR幻觉、复杂排版、细粒度定位等DIU核心问题。
Abstract Dance is an essential component of human culture and serves as a tool for conveying emotions and telling stories. Identifying and distinguishing dance genres based on motion data is a complex problem in human activity recognition, as many styles share similar poses, gestures, and temporal motion patterns. This work presents a lightweight framework for classifying dance styles that determines motion characteristics based on pose estimates extracted from videos. We propose temporal-spatial descriptors inspired by Laban Movement Analysis. These features capture local joint dynamics such as velocity, acceleration, and angular movement of the upper body, enabling a structured representation of spatial coordination. To further encode rhythmic and periodic aspects of movement, we integrate Fast Fourier Transform features that characterize movement patterns in the frequency domain. The proposed approach achieves robust classification of different dance styles with low computational effort, as complex model architectures are not required, and shows that interpretable motion representations can effectively capture stylistic nuances.
Authors Ben Hamscher, Arnold Brosch, Nicolas Binninger, Maksymilian Jan Dejna, Kira Maag
Categories Computer Vision and Pattern Recognition
0
Learning to Generate Human-Human-Object Interactions from Textual Descriptions
Title CN 从文本描述中学习生成人-人-物体交互
Keywords 人-人-物体交互 · 文本到动作生成 · 扩散模型 · 多智能体交互 · 行为建模
Summary 本文提出了一个新任务：从文本描述生成人-人-物体交互（HHOIs），并构建了相应的数据集与生成框架。通过结合人-物体交互（HOI）和人-人交互（HHI）的建模，利用扩散模型实现端到端的交互动作生成。实验表明该方法能生成更真实的多人交互场景。
Reason 该论文研究的是人类与人类及物体之间的交互生成，属于视频或图像生成领域，且聚焦于人体动作和场景合成，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法基于扩散模型生成多人体交互动作，无法迁移解决OCR幻觉、复杂排版理解、细粒度定位或长文档推理等DIU核心问题。
Abstract The way humans interact with each other, including interpersonal distances, spatial configuration, and motion, varies significantly across different situations. To enable machines to understand such complex, context-dependent behaviors, it is essential to model multiple people in relation to the surrounding scene context. In this paper, we present a novel research problem to model the correlations between two people engaged in a shared interaction involving an object. We refer to this formulation as Human-Human-Object Interactions (HHOIs). To overcome the lack of dedicated datasets for HHOIs, we present a newly captured HHOIs dataset and a method to synthesize HHOI data by leveraging image generative models. As an intermediary, we obtain individual human-object interaction (HOIs) and human-human interaction (HHIs) from the HHOIs, and with these data, we train an text-to-HOI and text-to-HHI model using score-based diffusion model. Finally, we present a unified generative framework that integrates the two individual model, capable of synthesizing complete HHOIs in a single advanced sampling process. Our method extends HHOI generation to multi-human settings, enabling interactions involving more than two individuals. Experimental results show that our method generates realistic HHOIs conditioned on textual descriptions, outperforming previous approaches that focus only on single-human HOIs. Furthermore, we introduce multi-human motion generation involving objects as an application of our framework.
Authors Jeonghyeon Na, Sangwon Baik, Inhee Lee, Junyoung Lee, Hanbyul Joo
Comment Project Page: https://tlb-miss.github.io/hhoi/
Categories Computer Vision and Pattern Recognition
0
FastGS: Training 3D Gaussian Splatting in 100 Seconds
Title CN FastGS：在100秒内训练3D高斯点阵
Keywords 3D Gaussian Splatting · 3D重建 · 训练加速 · 多视角一致性 · 计算机视觉
Summary 本文提出FastGS，一种用于加速3D高斯点阵训练的新框架，通过基于多视角一致性的稠密化与剪枝策略，在保证渲染质量的同时显著提升训练速度。实验表明其在多个数据集上相比现有方法有数倍加速效果。然而，该工作属于3D视觉重建范畴，与文档理解无关。
Reason 该论文研究的是3D高斯点阵（3D Gaussian Splatting）的加速训练，属于3D视觉与场景重建领域，与文档图像理解（DIU）、多模态大模型（VLM）或文档视觉问答（DocVQA）等任务完全无关。尽管其方法可能在3D重建任务中具有高效性，但无法迁移解决OCR幻觉、复杂排版理解、长文档推理等DIU核心痛点。
Abstract The dominant 3D Gaussian splatting (3DGS) acceleration methods fail to properly regulate the number of Gaussians during training, causing redundant computational time overhead. In this paper, we propose FastGS, a novel, simple, and general acceleration framework that fully considers the importance of each Gaussian based on multi-view consistency, efficiently solving the trade-off between training time and rendering quality. We innovatively design a densification and pruning strategy based on multi-view consistency, dispensing with the budgeting mechanism. Extensive experiments on Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets demonstrate that our method significantly outperforms the state-of-the-art methods in training speed, achieving a 3.32$\times$ training acceleration and comparable rendering quality compared with DashGaussian on the Mip-NeRF 360 dataset and a 15.45$\times$ acceleration compared with vanilla 3DGS on the Deep Blending dataset. We demonstrate that FastGS exhibits strong generality, delivering 2-7$\times$ training acceleration across various tasks, including dynamic scene reconstruction, surface reconstruction, sparse-view reconstruction, large-scale reconstruction, and simultaneous localization and mapping. The project page is available at https://fastgs.github.io/
Authors Shiwei Ren, Tianci Wen, Yongchun Fang, Biao Lu
Comment Project page: https://fastgs.github.io/
Categories Computer Vision and Pattern Recognition
0
Scalable FPGA Framework for Real-Time Denoising in High-Throughput Imaging: A DRAM-Optimized Pipeline using High-Level Synthesis
Title CN 用于高通量成像实时去噪的可扩展FPGA框架：一种基于高级综合的DRAM优化流水线
Keywords FPGA · 实时去噪 · 高通量成像 · 高级综合 · DRAM优化
Summary 本文提出了一种基于FPGA的可扩展预处理流水线，用于高通量成像中的实时去噪。通过高级综合（HLS）实现，并采用DRAM优化缓冲和突发模式AXI4接口，直接在流式图像数据上执行帧减法和平均以降低延迟。该方法可在帧间隔内完成操作，适用于光谱学和显微镜等对延迟敏感的成像工作流。
Reason 该论文聚焦于高通量成像（如光谱显微成像）中的实时去噪硬件加速，属于专用FPGA架构设计，应用于生物成像或物理传感等非文档图像领域。尽管涉及图像预处理，但其技术路线与文档图像理解（DIU）的核心任务（如DocVQA、布局分析、OCR-free理解）无直接关联，也无法迁移解决OCR幻觉、复杂排版建模或长文档推理等关键痛点。
Abstract High-throughput imaging workflows, such as Parallel Rapid Imaging with Spectroscopic Mapping (PRISM), generate data at rates that exceed conventional real-time processing capabilities. We present a scalable FPGA-based preprocessing pipeline for real-time denoising, implemented via High-Level Synthesis (HLS) and optimized for DRAM-backed buffering. Our architecture performs frame subtraction and averaging directly on streamed image data, minimizing latency through burst-mode AXI4 interfaces. The resulting kernel operates below the inter-frame interval, enabling inline denoising and reducing dataset size for downstream CPU/GPU analysis. Validated under PRISM-scale acquisition, this modular FPGA framework offers a practical solution for latency-sensitive imaging workflows in spectroscopy and microscopy.
Authors Weichien Liao
Comment FPGA-based denoising pipeline for PRISM-scale imaging. Real-time frame subtraction and averaging via burst-mode AXI4 and DRAM buffering. Benchmarked against CPU/GPU workflows; scalable across multi-bank FPGA setups. Acknowledgements revised for consistency with journal submission; scientific content remains unchanged
Categories Computer Vision and Pattern Recognition
0
MajutsuCity: Language-driven Aesthetic-adaptive City Generation with Controllable 3D Assets and Layouts
Title CN MajutsuCity：基于语言驱动的美学自适应城市生成，支持可控的3D资产与布局
Keywords 3D城市生成 · 语言驱动生成 · 多模态合成 · 可控生成 · 虚拟环境
Summary 本文提出MajutsuCity，一种基于自然语言驱动的3D城市生成框架，通过四阶段流程生成风格多样且结构一致的城市景观，并引入可交互编辑的MajutsuAgent支持对象级操作。同时构建了包含2D语义布局、3D建筑资产和PBR材质的高质量数据集MajutsuDataset，并设计了一套综合评估指标。实验表明其在布局质量、视觉真实感等方面优于现有方法。
Reason 该论文属于3D视觉与虚拟场景生成领域，研究目标为3D城市建模，与文档图像理解（DIU）或DocVQA无直接关联。尽管其使用了多模态输入（语言+视觉）和结构化生成流程，但核心任务聚焦于游戏、VR等场景下的城市合成，涉及的技术如3D资产控制、材质渲染、高度图生成等难以迁移到文档理解中的OCR幻觉、密集文本识别、复杂版式分析等核心痛点。
Abstract Generating realistic 3D cities is fundamental to world models, virtual reality, and game development, where an ideal urban scene must satisfy both stylistic diversity, fine-grained, and controllability. However, existing methods struggle to balance the creative flexibility offered by text-based generation with the object-level editability enabled by explicit structural representations. We introduce MajutsuCity, a natural language-driven and aesthetically adaptive framework for synthesizing structurally consistent and stylistically diverse 3D urban scenes. MajutsuCity represents a city as a composition of controllable layouts, assets, and materials, and operates through a four-stage pipeline. To extend controllability beyond initial generation, we further integrate MajutsuAgent, an interactive language-grounded editing agent} that supports five object-level operations. To support photorealistic and customizable scene synthesis, we also construct MajutsuDataset, a high-quality multimodal dataset} containing 2D semantic layouts and height maps, diverse 3D building assets, and curated PBR materials and skyboxes, each accompanied by detailed annotations. Meanwhile, we develop a practical set of evaluation metrics, covering key dimensions such as structural consistency, scene complexity, material fidelity, and lighting atmosphere. Extensive experiments demonstrate MajutsuCity reduces layout FID by 83.7% compared with CityDreamer and by 20.1% over CityCraft. Our method ranks first across all AQS and RDR scores, outperforming existing methods by a clear margin. These results confirm MajutsuCity as a new state-of-the-art in geometric fidelity, stylistic adaptability, and semantic controllability for 3D city generation. We expect our framework can inspire new avenues of research in 3D city generation. Our dataset and code will be released at https://github.com/LongHZ140516/MajutsuCity.
Authors Zilong Huang, Jun He, Xiaobin Huang, Ziyi Xiong, Yang Luo, Junyan Ye, Weijia Li, Yiping Chen, Ting Han
Comment 13 pages, 6 figures
Categories Computer Vision and Pattern Recognition
0
Image-Free Timestep Distillation via Continuous-Time Consistency with Trajectory-Sampled Pairs
Title CN 无需外部数据的连续时间一致性轨迹采样对齐蒸馏
Keywords 扩散模型 · 知识蒸馏 · 一致性模型 · 图像生成 · 高效推理
Summary 本文提出了一种名为TBCM的蒸馏方法，通过从教师模型的生成轨迹中提取潜在表示，实现无需外部训练数据的一致性模型蒸馏。该方法减少了对大规模数据集和VAE编码的依赖，在MJHQ-30k数据集上实现了高效的单步生成性能，并降低了训练时间和显存消耗。同时分析了扩散生成空间的分布差异对蒸馏的影响。
Reason 该论文研究的是扩散模型中的时间步蒸馏方法，属于图像生成领域，核心目标是提升生成效率和减少训练资源消耗。其技术路线聚焦于生成模型的蒸馏策略，与文档图像理解（DIU）中的OCR幻觉、复杂排版解析、细粒度文本识别或长文档推理等核心问题无直接关联。尽管‘一致性模型’在理论上具有迁移潜力，但本文未涉及多模态理解、视觉-语言对齐或文本结构建模，无法直接服务于DocVQA或布局分析等任务，属于无关领域的技术优化。
Abstract Timestep distillation is an effective approach for improving the generation efficiency of diffusion models. The Consistency Model (CM), as a trajectory-based framework, demonstrates significant potential due to its strong theoretical foundation and high-quality few-step generation. Nevertheless, current continuous-time consistency distillation methods still rely heavily on training data and computational resources, hindering their deployment in resource-constrained scenarios and limiting their scalability to diverse domains. To address this issue, we propose Trajectory-Backward Consistency Model (TBCM), which eliminates the dependence on external training data by extracting latent representations directly from the teacher model's generation trajectory. Unlike conventional methods that require VAE encoding and large-scale datasets, our self-contained distillation paradigm significantly improves both efficiency and simplicity. Moreover, the trajectory-extracted samples naturally bridge the distribution gap between training and inference, thereby enabling more effective knowledge transfer. Empirically, TBCM achieves 6.52 FID and 28.08 CLIP scores on MJHQ-30k under one-step generation, while reducing training time by approximately 40% compared to Sana-Sprint and saving a substantial amount of GPU memory, demonstrating superior efficiency without sacrificing quality. We further reveal the diffusion-generation space discrepancy in continuous-time consistency distillation and analyze how sampling strategies affect distillation performance, offering insights for future distillation research. GitHub Link: https://github.com/hustvl/TBCM.
Authors Bao Tang, Shuai Zhang, Yueting Zhu, Jijun Xiang, Xin Yang, Li Yu, Wenyu Liu, Xinggang Wang
Categories Computer Vision and Pattern Recognition
0
A Training-Free Approach for Multi-ID Customization via Attention Adjustment and Spatial Control
Title CN 通过注意力调整和空间控制实现多身份定制的无训练方法
Keywords 多身份定制 · 注意力机制 · 空间控制 · 无训练方法 · 图像生成
Summary 本文提出了一种无需训练的多身份定制生成方法MultiID，通过解耦交叉注意力机制将多个个体的身份嵌入注入到对应图像区域，并结合局部提示、深度引导的空间控制和扩展自注意力来提升生成结果与文本描述的一致性。作者还构建了IDBench基准进行评估，实验表明该方法在生成质量和可控性方面可媲美甚至优于有训练的方法。
Reason 该论文研究的是多身份图像生成，属于人脸/人物图像合成与编辑领域，主要解决的是视觉生成中的身份融合与文本控制问题。尽管使用了注意力机制和空间控制等技术，但其核心任务与文档图像理解（DIU）无关，不涉及OCR、布局分析、表格识别或文档推理等任何DIU核心任务。所提出的方法难以迁移到文档理解场景中用于解决OCR幻觉、复杂排版或长文档处理等问题。
Abstract Multi-ID customization is an interesting topic in computer vision and attracts considerable attention recently. Given the ID images of multiple individuals, its purpose is to generate a customized image that seamlessly integrates them while preserving their respective identities. Compared to single-ID customization, multi-ID customization is much more difficult and poses two major challenges. First, since the multi-ID customization model is trained to reconstruct an image from the cropped person regions, it often encounters the copy-paste issue during inference, leading to lower quality. Second, the model also suffers from inferior text controllability. The generated result simply combines multiple persons into one image, regardless of whether it is aligned with the input text. In this work, we propose MultiID to tackle this challenging task in a training-free manner. Since the existing single-ID customization models have less copy-paste issue, our key idea is to adapt these models to achieve multi-ID customization. To this end, we present an ID-decoupled cross-attention mechanism, injecting distinct ID embeddings into the corresponding image regions and thus generating multi-ID outputs. To enhance the generation controllability, we introduce three critical strategies, namely the local prompt, depth-guided spatial control, and extended self-attention, making the results more consistent with the text prompts and ID images. We also carefully build a benchmark, called IDBench, for evaluation. The extensive qualitative and quantitative results demonstrate the effectiveness of MultiID in solving the aforementioned two challenges. Its performance is comparable or even better than the training-based multi-ID customization methods.
Authors Jiawei Lin, Guanlong Jiao, Jianjin Xu
Categories Computer Vision and Pattern Recognition
0
Target-aware Image Editing via Cycle-consistent Constraints
Title CN 基于循环一致性约束的面向目标的图像编辑
Keywords 图像编辑 · 文本引导编辑 · 循环一致性 · 预训练扩散模型 · 可逆生成
Summary 本文提出FlowCycle，一种无需反转的基于流的图像编辑框架，通过可学习噪声参数化破坏过程，并利用循环一致性约束（从源到目标再恢复到源）来构建目标感知的中间状态，从而在保持源图像一致性的同时实现更精确的编辑。实验表明该方法在编辑质量和一致性上优于现有方法。
Reason 该论文研究的是文本引导的图像编辑任务，属于纯图像生成与编辑领域，主要解决图像内容修改的保真性与一致性问题。其核心技术（如目标感知的中间状态构造、循环一致性优化）虽然在图像编辑中有创新，但与文档图像理解（DIU）的核心痛点（如OCR幻觉、密集文本识别、复杂版式分析、长文档推理）无直接关联。方法难以迁移到文档场景，且未涉及多模态语言模型对文档结构和语义的理解或推理。
Abstract Recent advances in pre-trained text-to-image flow models have enabled remarkable progress in text-based image editing. Mainstream approaches always adopt a corruption-then-restoration paradigm, where the source image is first corrupted into an ``intermediate state'' and then restored to the target image under the prompt guidance. However, current methods construct this intermediate state in a target-agnostic manner, i.e., they primarily focus on realizing source image reconstruction while neglecting the semantic gaps towards the specific editing target. This design inherently results in limited editability or inconsistency when the desired modifications substantially deviate from the source. In this paper, we argue that the intermediate state should be target-aware, i.e., selectively corrupting editing-relevant contents while preserving editing-irrelevant ones. To this end, we propose FlowCycle, a novel inversion-free and flow-based editing framework that parameterizes corruption with learnable noises and optimizes them through a cycle-consistent process. By iteratively editing the source to the target and recovering back to the source with dual consistency constraints, FlowCycle learns to produce a target-aware intermediate state, enabling faithful modifications while preserving source consistency. Extensive ablations have demonstrated that FlowCycle achieves superior editing quality and consistency over state-of-the-art methods.
Authors Yanghao Wang, Zhen Wang, Long Chen
Categories Computer Vision and Pattern Recognition
0
CLIP-IT: CLIP-based Pairing for Histology Images Classification
Title CN 基于CLIP的组织学图像分类配对方法
Keywords 多模态学习 · CLIP · 未配对文本 · 组织学图像分类 · 知识蒸馏
Summary 本文提出CLIP-IT框架，利用预训练的CLIP模型从大量未配对的病理报告中为组织学图像检索语义相关的文本，构建伪图文对，并通过知识蒸馏将文本信息融入视觉模型训练过程。推理阶段仅使用视觉模型，降低了计算开销。实验表明该方法在无需下游数据配对标注的情况下提升了分类准确率。
Reason 该论文属于医学图像领域的平行应用，其核心是将CLIP用于组织学图像与未配对病理报告之间的伪配对以提升分类性能。尽管使用了VLM技术，但方法本身聚焦于医疗影像这一特定下游任务，未提出可迁移至文档图像理解（如DocVQA、布局分析等）的通用架构、训练策略或底层创新。且其处理的是生物医学图像与临床文本的跨模态对齐问题，与文档中的OCR幻觉、复杂排版、细粒度定位等核心痛点无关。
Abstract Multimodal learning has shown promise in medical imaging, combining complementary modalities like images and text. Vision-language models (VLMs) capture rich diagnostic cues but often require large paired datasets and prompt- or text-based inference, limiting their practicality due to annotation cost, privacy, and compute demands. Crucially, available free unpaired external text, like pathology reports, can still provide complementary diagnostic cues if semantically relevant content is retrievable per image. To address this, we introduce CLIP-IT, a novel framework that relies on rich unpaired text reports. Specifically, CLIP-IT uses a CLIP model pre-trained on histology image-text pairs from a separate dataset to retrieve the most relevant unpaired textual report for each image in the downstream unimodal dataset. These reports, sourced from the same disease domain and tissue type, form pseudo-pairs that reflect shared clinical semantics rather than exact alignment. Knowledge from these texts is distilled into the vision model during training, while LoRA-based adaptation mitigates the semantic gap between unaligned modalities. At inference, only the vision model is used, keeping overhead low while still benefiting from multimodal training without requiring paired data in the downstream dataset. Experiments on histology image datasets confirm that CLIP-IT consistently improves classification accuracy over both unimodal and multimodal CLIP-based baselines in most cases, without the burden of per-dataset paired annotation or inference-time complexity.
Authors Banafsheh Karimian, Giulia Avanzato, Soufian Belharbi, Alexis Guichemerre, Luke McCaffrey, Mohammadhadi Shateri, Eric Granger
Categories Computer Vision and Pattern Recognition
0
FREE: Uncertainty-Aware Autoregression for Parallel Diffusion Transformers
Title CN FREE：面向并行扩散Transformer的不确定性感知自回归方法
Keywords Diffusion Transformers · 推理加速 · 自回归 · 不确定性建模 · 并行采样
Summary 本文提出一种名为FREE的框架，通过特征级自回归与并行验证机制，在保持生成质量的前提下提升扩散Transformer的推理速度。引入不确定性引导的松弛策略后，进一步提升了加速比。实验表明该方法在ImageNet高分辨率生成任务中具有显著加速效果。
Reason 该论文研究的是扩散Transformer（DiTs）在图像生成任务中的推理加速问题，核心关注点为生成模型的并行采样与加速策略。尽管其提出的方法涉及视觉模型的推理优化，但完全属于图像生成领域，且目标是提升生成速度与质量，与文档图像理解（DIU）中的OCR幻觉、复杂排版、细粒度识别或长文档推理等核心痛点无直接关联。无法迁移用于解决文档理解中的语义解析或多模态对齐问题。
Abstract Diffusion Transformers (DiTs) achieve state-of-the-art generation quality but require long sequential denoising trajectories, leading to high inference latency. Recent speculative inference methods enable lossless parallel sampling in U-Net-based diffusion models via a drafter-verifier scheme, but their acceleration is limited on DiTs due to insufficient draft accuracy during verification. To address this limitation, we analyze the DiTs' feature dynamics and find the features of the final transformer layer (top-block) exhibit strong temporal consistency and rich semantic abstraction. Based on this insight, we propose FREE, a novel framework that employs a lightweight drafter to perform feature-level autoregression with parallel verification, guaranteeing lossless acceleration with theoretical and empirical support. Meanwhile, prediction variance (uncertainty) of DiTs naturally increases in later denoising steps, reducing acceptance rates under speculative sampling. To mitigate this effect, we further introduce an uncertainty-guided relaxation strategy, forming FREE (relax), which dynamically adjusts the acceptance probability in response to uncertainty levels. Experiments on ImageNet-$512^2$ show that FREE achieves up to $1.86 \times$ acceleration, and FREE (relax) further reaches $2.25 \times$ speedup while maintaining high perceptual and quantitative fidelity in generation quality.
Authors Xinwan Wen, Bowen Li, Jiajun Luo, Ye Li, Zhi Wang
Categories Computer Vision and Pattern Recognition
0
Detecting Cultural Differences in News Video Thumbnails via Computational Aesthetics ICWSM'24 Workshop
Title CN 通过计算美学检测新闻视频缩略图中的文化差异
Keywords 计算美学 · 文化差异 · 视频缩略图 · 视觉风格分析 · 聚类
Summary 本文提出一种两步法，先按内容将YouTube视频缩略图聚类到细粒度视觉主题中，再比较不同文化来源（中美）的美学特征。基于2400张缩略图的实验显示，美国频道倾向于使用更正式、饱和度高、细节丰富且近距离的图像，而中国频道则偏好更随意、色彩丰富但结构较松散的缩略图。作者认为这些差异反映了文化偏好，并可作为识别视觉宣传的基线。
Reason 该论文研究的是跨文化背景下新闻视频缩略图的视觉风格差异，属于视频内容的社会文化分析，与文档图像理解（DIU）或DocVQA等任务无直接关联。其方法聚焦于美学特征比较和聚类，不涉及OCR、布局分析、表格识别或VLM在文档上的应用。属于无关领域（视频理解与社会分析），无法迁移解决DIU的核心痛点如幻觉、复杂排版或长文档推理。
Abstract We propose a two-step approach for detecting differences in the style of images across sources of differing cultural affinity, where images are first clustered into finer visual themes based on content before their aesthetic features are compared. We test this approach on 2,400 YouTube video thumbnails taken equally from two U.S. and two Chinese YouTube channels, and relating equally to COVID-19 and the Ukraine conflict. Our results suggest that while Chinese thumbnails are less formal and more candid, U.S. channels tend to use more deliberate, proper photographs as thumbnails. In particular, U.S. thumbnails are less colorful, more saturated, darker, more finely detailed, less symmetric, sparser, less varied, and more up close and personal than Chinese thumbnails. We suggest that most of these differences reflect cultural preferences, and that our methods and observations can serve as a baseline against which suspected visual propaganda can be computed and compared.
Authors Marvin Limpijankit, John Kender
Comment ICWSM'24 Workshop
Categories Computer Vision and Pattern Recognition
0
GS-Checker: Tampering Localization for 3D Gaussian Splatting AAAI2026
Title CN GS-Checker：针对3D高斯点阵的篡改定位
Keywords 3D Gaussian Splatting · 篡改定位 · 3D内容安全 · 对比学习 · 无监督
Summary 本文提出GS-Checker，一种用于定位3D高斯点阵模型中被篡改区域的新方法。通过在3D高斯参数中引入篡改属性，并设计3D对比机制来检测异常，结合循环优化策略提升定位精度。该方法无需昂贵的3D标注数据，在实验中表现出良好的篡改检测能力。
Reason 该论文研究的是3D高斯点阵（3D Gaussian Splatting）中的篡改区域定位问题，属于3D视觉与数字内容安全领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。其方法聚焦于3D场景的完整性验证，无法迁移用于解决OCR幻觉、复杂排版理解、细粒度文本定位等DIU核心痛点。
Abstract Recent advances in editing technologies for 3D Gaussian Splatting (3DGS) have made it simple to manipulate 3D scenes. However, these technologies raise concerns about potential malicious manipulation of 3D content. To avoid such malicious applications, localizing tampered regions becomes crucial. In this paper, we propose GS-Checker, a novel method for locating tampered areas in 3DGS models. Our approach integrates a 3D tampering attribute into the 3D Gaussian parameters to indicate whether the Gaussian has been tampered. Additionally, we design a 3D contrastive mechanism by comparing the similarity of key attributes between 3D Gaussians to seek tampering cues at 3D level. Furthermore, we introduce a cyclic optimization strategy to refine the 3D tampering attribute, enabling more accurate tampering localization. Notably, our approach does not require expensive 3D labels for supervision. Extensive experimental results demonstrate the effectiveness of our proposed method to locate the tampered 3DGS area.
Authors Haoliang Han, Ziyuan Luo, Jun Qi, Anderson Rocha, Renjie Wan
Comment Accepted by AAAI2026
Categories Computer Vision and Pattern Recognition
0
Material-informed Gaussian Splatting for 3D World Reconstruction in a Digital Twin IEEE Intelligent Vehicles Symposium (IV) 2026
Title CN 基于材质感知的高斯点阵化用于数字孪生中的三维世界重建
Keywords 3D重建 · 高斯点阵化 · 数字孪生 · 材质分割 · 传感器模拟
Summary 本文提出一种仅使用相机图像的3D重建流程，通过多视角图像进行高斯点阵化，结合视觉模型提取材质语义掩码，并将高斯表示转换为带材质标签的网格表面，赋予物理材质属性以支持高保真传感器模拟。该方法旨在替代传统的LiDAR-相机融合方案，适用于数字孪生环境构建。
Reason 该论文属于3D视觉与自动驾驶仿真领域，核心方法聚焦于高斯点阵化（Gaussian Splatting）和材质属性建模，用于数字孪生中的传感器模拟。其研究目标与文档图像理解（DIU）无直接关联，且所解决的问题（如LiDAR-camera融合、物理材质模拟）在文档场景中不存在对应迁移路径。属于明确列出的‘无关领域’：3D视觉与自动驾驶，命中负面清单。
Abstract 3D reconstruction for Digital Twins often relies on LiDAR-based methods, which provide accurate geometry but lack the semantics and textures naturally captured by cameras. Traditional LiDAR-camera fusion approaches require complex calibration and still struggle with certain materials like glass, which are visible in images but poorly represented in point clouds. We propose a camera-only pipeline that reconstructs scenes using 3D Gaussian Splatting from multi-view images, extracts semantic material masks via vision models, converts Gaussian representations to mesh surfaces with projected material labels, and assigns physics-based material properties for accurate sensor simulation in modern graphics engines and simulators. This approach combines photorealistic reconstruction with physics-based material assignment, providing sensor simulation fidelity comparable to LiDAR-camera fusion while eliminating hardware complexity and calibration requirements. We validate our camera-only method using an internal dataset from an instrumented test vehicle, leveraging LiDAR as ground truth for reflectivity validation alongside image similarity metrics.
Authors João Malheiro Silva, Andy Huynh, Tong Duy Son, Holger Caesar
Comment 8 pages, 5 figures. Submitted to IEEE Intelligent Vehicles Symposium (IV) 2026 for possible publication
Categories Computer Vision and Pattern Recognition
0
AMB3R: Accurate Feed-forward Metric-scale 3D Reconstruction with Backend
Title CN AMB3R：基于后端的高精度前馈度量级3D重建
Keywords 3D重建 · 多视角几何 · 体素表示 · 视觉里程计 · SfM
Summary 本文提出AMB3R，一种用于度量级密集3D重建的前馈多视角模型，通过紧凑的体素化场景表示实现几何推理，在无需任务微调的情况下可扩展至视觉里程计和大规模SfM任务，并在多个基准上超越现有方法。
Reason 该论文研究的是多视角3D场景重建，属于3D视觉与SLAM领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档任务中的应用无直接关联。尽管其方法涉及视觉表示学习，但核心目标为度量级几何重建和相机位姿估计，无法迁移解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点。
Abstract We present AMB3R, a multi-view feed-forward model for dense 3D reconstruction on a metric-scale that addresses diverse 3D vision tasks. The key idea is to leverage a sparse, yet compact, volumetric scene representation as our backend, enabling geometric reasoning with spatial compactness. Although trained solely for multi-view reconstruction, we demonstrate that AMB3R can be seamlessly extended to uncalibrated visual odometry (online) or large-scale structure from motion without the need for task-specific fine-tuning or test-time optimization. Compared to prior pointmap-based models, our approach achieves state-of-the-art performance in camera pose, depth, and metric-scale estimation, 3D reconstruction, and even surpasses optimization-based SLAM and SfM methods with dense reconstruction priors on common benchmarks.
Authors Hengyi Wang, Lourdes Agapito
Comment Project page: https://hengyiwang.github.io/projects/amber
Categories Computer Vision and Pattern Recognition
0
ArtiBench and ArtiBrain: Benchmarking Generalizable Vision-Language Articulated Object Manipulation
Title CN ArtiBench和ArtiBrain：通用视觉语言铰接物体操作的基准测试
Keywords 视觉语言导航 · 机器人操作 · 铰接物体 · 基准测试 · VLM
Summary 本文提出了ArtiBench，一个用于评估视觉语言驱动的铰接物体操作的五级基准，涵盖厨房、储物、办公等场景，并设计了ArtiBrain框架，结合VLM进行子目标分解与验证，以及混合控制器实现精确操作。通过记忆库积累可操作性知识以提升泛化能力。实验表明该方法在鲁棒性和跨实例泛化上优于现有方法。
Reason 该论文属于具身智能与机器人领域，研究的是铰接物体（如抽屉、门把手）的物理交互与操作，核心任务是视觉-语言驱动的机器人控制。尽管使用了VLM进行高层推理，但其目标、方法和应用场景均与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心问题。
Abstract Interactive articulated manipulation requires long-horizon, multi-step interactions with appliances while maintaining physical consistency. Existing vision-language and diffusion-based policies struggle to generalize across parts, instances, and categories. We first introduce ArtiBench, a five-level benchmark covering kitchen, storage, office, and tool environments. ArtiBench enables structured evaluation from cross-part and cross-instance variation to long-horizon multi-object tasks, revealing the core generalization challenges of articulated object manipulation. Building on this benchmark, we propose ArtiBrain, a modular framework that unifies high-level reasoning with adaptive low-level control. ArtiBrain uses a VLM-based Task Reasoner (GPT-4.1) to decompose and validate subgoals, and employs a Hybrid Controller that combines geometry-aware keyframe execution with affordance-guided diffusion for precise and interpretable manipulation. An Affordance Memory Bank continually accumulates successful execution episodes and propagates part-level actionable affordances to unseen articulated parts and configurations. Extensive experiments on ArtiBench show that our ArtiBrain significantly outperforms state-of-the-art multimodal and diffusion-based methods in robustness and generalization. Code and dataset will be released upon acceptance.
Authors Yuhan Wu, Tiantian Wei, Shuo Wang, ZhiChao Wang, Yanyong Zhang, Daniel Cremers, Yan Xia
Categories Computer Vision and Pattern Recognition
0
AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Models
Title CN AD-R1：基于无偏世界模型的端到端自动驾驶闭环强化学习
Keywords 自动驾驶 · 强化学习 · 世界模型 · 安全性 · 反事实合成
Summary 本文提出了一种用于端到端自动驾驶的闭环强化学习框架AD-R1，引入‘无偏世界模型’作为内部批评者，通过反事实合成生成碰撞和越界事件，使模型能诚实预测危险。该模型在风险预见基准上表现优异，显著减少了模拟环境中的安全违规行为。
Reason 该论文属于自动驾驶领域，研究的是端到端驾驶策略与强化学习中的安全问题，尽管涉及世界模型和幻觉控制等概念，但其任务场景、数据模态和文档图像理解（DIU）完全无关。无法迁移其方法至文档VQA、布局分析或OCR-free理解等核心问题。
Abstract End-to-end models for autonomous driving hold the promise of learning complex behaviors directly from sensor data, but face critical challenges in safety and handling long-tail events. Reinforcement Learning (RL) offers a promising path to overcome these limitations, yet its success in autonomous driving has been elusive. We identify a fundamental flaw hindering this progress: a deep seated optimistic bias in the world models used for RL. To address this, we introduce a framework for post-training policy refinement built around an Impartial World Model. Our primary contribution is to teach this model to be honest about danger. We achieve this with a novel data synthesis pipeline, Counterfactual Synthesis, which systematically generates a rich curriculum of plausible collisions and off-road events. This transforms the model from a passive scene completer into a veridical forecaster that remains faithful to the causal link between actions and outcomes. We then integrate this Impartial World Model into our closed-loop RL framework, where it serves as an internal critic. During refinement, the agent queries the critic to ``dream" of the outcomes for candidate actions. We demonstrate through extensive experiments, including on a new Risk Foreseeing Benchmark, that our model significantly outperforms baselines in predicting failures. Consequently, when used as a critic, it enables a substantial reduction in safety violations in challenging simulations, proving that teaching a model to dream of danger is a critical step towards building truly safe and intelligent autonomous agents.
Authors Tianyi Yan, Tao Tang, Xingtai Gui, Yongkang Li, Jiasen Zhesng, Weiyao Huang, Lingdong Kong, Wencheng Han, Xia Zhou, Xueyang Zhang, Yifei Zhan, Kun Zhan, Cheng-zhong Xu, Jianbing Shen
Categories Computer Vision and Pattern Recognition
0
IrisNet: Infrared Image Status Awareness Meta Decoder for Infrared Small Targets Detection
Title CN IrisNet：用于红外小目标检测的红外图像状态感知元解码器
Keywords 红外小目标检测 · 元学习 · 动态解码 · Transformer · 高频增强
Summary 本文提出IrisNet，一种基于元学习的红外小目标检测框架，通过图像到解码器的Transformer动态生成解码参数，以适应不同场景下的红外图像状态变化，并引入高频频段增强目标位置和边缘信息。在多个红外数据集上实现了最先进的性能。
Reason 该论文研究的是红外小目标检测（IRSTD），属于遥感或军事视觉领域，与文档图像理解（DIU）任务无直接关联。尽管其提出了动态自适应解码机制和高频信息融合策略，但应用场景、数据模态和核心问题均与文档图像中的文本识别、布局分析、VQA等任务严重偏离，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU痛点。
Abstract Infrared Small Target Detection (IRSTD) faces significant challenges due to low signal-to-noise ratios, complex backgrounds, and the absence of discernible target features. While deep learning-based encoder-decoder frameworks have advanced the field, their static pattern learning suffers from pattern drift across diverse scenarios (\emph{e.g.}, day/night variations, sky/maritime/ground domains), limiting robustness. To address this, we propose IrisNet, a novel meta-learned framework that dynamically adapts detection strategies to the input infrared image status. Our approach establishes a dynamic mapping between infrared image features and entire decoder parameters via an image-to-decoder transformer. More concretely, we represent the parameterized decoder as a structured 2D tensor preserving hierarchical layer correlations and enable the transformer to model inter-layer dependencies through self-attention while generating adaptive decoding patterns via cross-attention. To further enhance the perception ability of infrared images, we integrate high-frequency components to supplement target-position and scene-edge information. Experiments on NUDT-SIRST, NUAA-SIRST, and IRSTD-1K datasets demonstrate the superiority of our IrisNet, achieving state-of-the-art performance.
Authors Xuelin Qian, Jiaming Lu, Zixuan Wang, Wenxuan Wang, Zhongling Huang, Dingwen Zhang, Junwei Han
Comment 10pages,5figures
Categories Computer Vision and Pattern Recognition
0
StrCGAN: A Generative Framework for Stellar Image Restoration
Title CN StrCGAN：一种用于恒星图像恢复的生成框架
Keywords 图像恢复 · 生成对抗网络 · 天文图像 · 多光谱融合 · CycleGAN
Summary 本文提出StrCGAN，一种基于CycleGAN改进的生成模型，用于提升低分辨率天文图像的质量。通过引入多光谱融合和天体物理正则化模块，该方法在保持星体形态的同时实现跨光谱带的一致性重建，优于传统GAN模型。
Reason 该论文研究的是天文图像增强，属于纯图像修复任务，应用场景与文档图像理解（DIU）完全无关。其技术路线聚焦于多光谱融合与天体物理正则化模块，针对的是星体形态保持，无法迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract We introduce StrCGAN (Stellar Cyclic GAN), a generative model designed to enhance low-resolution astrophotography images. Our goal is to reconstruct high fidelity ground truth like representations of stellar objects, a task that is challenging due to the limited resolution and quality of small-telescope observations such as the MobilTelesco dataset. Traditional models such as CycleGAN provide a foundation for image to image translation but often distort the morphology of stars and produce barely resembling images. To overcome these limitations, we extend the CycleGAN framework with some key innovations: multi-spectral fusion to align optical and near infrared (NIR) domains, and astrophysical regularization modules to preserve stellar morphology. Ground truth references from multi-mission all sky surveys spanning optical to NIR guide the training process, ensuring that reconstructions remain consistent across spectral bands. Together, these components allow StrCGAN to generate reconstructions that are visually sharper outperforming standard GAN models in the task of astrophysical image enhancement.
Authors Shantanusinh Parmar, Silas Janke
Categories Computer Vision and Pattern Recognition
0
TReFT: Taming Rectified Flow Models For One-Step Image Translation
Title CN TReFT：驯化校正流模型用于一步式图像翻译
Keywords 图像翻译 · 校正流模型 · 一次性生成 · 对抗训练 · 高效推理
Summary 本文提出TReFT方法，旨在解决校正流模型在图像到图像翻译中依赖多步去噪的问题，通过利用预训练DiT或UNet预测的速度作为输出，在对抗训练下实现稳定的一次性推理。该方法结合潜空间循环一致性损失和轻量架构改进，可在保持高质量的同时实现实时图像翻译。然而，其应用范围限于通用图像转换任务，如风格迁移或域适应，并未针对文档内容理解进行设计。
Reason 该论文研究的是通用图像到图像的翻译任务，核心目标是实现基于校正流模型（Rectified Flow）的一次性生成推理，属于图像生成领域。尽管其方法涉及视觉建模和高效推理，但其应用场景与文档图像理解（DIU）中的OCR幻觉、复杂排版分析、细粒度文本定位等核心痛点无关。该工作无法直接迁移用于提升DocVQA或文档布局分析性能，属于无关领域的技术优化。
Abstract Rectified Flow (RF) models have advanced high-quality image and video synthesis via optimal transport theory. However, when applied to image-to-image translation, they still depend on costly multi-step denoising, hindering real-time applications. Although the recent adversarial training paradigm, CycleGAN-Turbo, works in pretrained diffusion models for one-step image translation, we find that directly applying it to RF models leads to severe convergence issues. In this paper, we analyze these challenges and propose TReFT, a novel method to Tame Rectified Flow models for one-step image Translation. Unlike previous works, TReFT directly uses the velocity predicted by pretrained DiT or UNet as output-a simple yet effective design that tackles the convergence issues under adversarial training with one-step inference. This design is mainly motivated by a novel observation that, near the end of the denoising process, the velocity predicted by pretrained RF models converges to the vector from origin to the final clean image, a property we further justify through theoretical analysis. When applying TReFT to large pretrained RF models such as SD3.5 and FLUX, we introduce memory-efficient latent cycle-consistency and identity losses during training, as well as lightweight architectural simplifications for faster inference. Pretrained RF models finetuned with TReFT achieve performance comparable to sota methods across multiple image translation datasets while enabling real-time inference.
Authors Shengqian Li, Ming Gao, Yi Liu, Zuzeng Lin, Feng Wang, Feng Dai
Categories Computer Vision and Pattern Recognition
0
TaCo: Capturing Spatio-Temporal Semantic Consistency in Remote Sensing Change Detection
Title CN TaCo：在遥感变化检测中捕捉时空语义一致性
Keywords 遥感变化检测 · 时空语义一致性 · 文本引导 · 特征重建 · 多模态学习
Summary 本文提出TaCo模型用于遥感变化检测，通过引入文本引导的过渡特征生成器和时空语义联合约束机制，在无额外推理开销的情况下提升变化检测性能。模型利用文本语义辅助建模双时相之间的状态转换，并通过重建损失和过渡判别损失增强语义一致性。
Reason 该论文属于遥感图像分析领域，研究任务为卫星图像的双时相变化检测，核心方法依赖文本引导的过渡特征生成与重建约束。尽管使用了多模态（视觉+文本）思路，但其问题设定、数据模态和应用场景与文档图像理解（DIU）差异巨大，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点，属于无关领域工作。
Abstract Remote sensing change detection (RSCD) aims to identify surface changes across bi-temporal satellite images. Most previous methods rely solely on mask supervision, which effectively guides spatial localization but provides limited constraints on the temporal semantic transitions. Consequently, they often produce spatially coherent predictions while still suffering from unresolved semantic inconsistencies. To address this limitation, we propose TaCo, a spatio-temporal semantic consistent network, which enriches the existing mask-supervised framework with a spatio-temporal semantic joint constraint. TaCo conceptualizes change as a semantic transition between bi-temporal states, in which one temporal feature representation can be derived from the other via dedicated transition features. To realize this, we introduce a Text-guided Transition Generator that integrates textual semantics with bi-temporal visual features to construct the cross-temporal transition features. In addition, we propose a spatio-temporal semantic joint constraint consisting of bi-temporal reconstruct constraints and a transition constraint: the former enforces alignment between reconstructed and original features, while the latter enhances discrimination for changes. This design can yield substantial performance gains without introducing any additional computational overhead during inference. Extensive experiments on six public datasets, spanning both binary and semantic change detection tasks, demonstrate that TaCo consistently achieves SOTA performance.
Authors Han Guo, Chenyang Liu, Haotian Zhang, Bowen Chen, Zhengxia Zou, Zhenwei Shi
Categories Computer Vision and Pattern Recognition
0
Back to the Feature: Explaining Video Classifiers with Video Counterfactual Explanations
Title CN 回归特征：用视频反事实解释来解释视频分类器
Keywords 反事实解释 · 视频理解 · 可解释AI · 生成模型 · 优化框架
Summary 本文提出BTTF框架，通过优化潜在噪声和分阶段搜索策略，生成对视频分类器有意义的反事实视频解释。该方法强调时间一致性与运动平滑性，在动作、情绪等视频分类任务上生成逼真的反事实样本，揭示模型依赖的关键视觉特征。
Reason 该论文研究的是视频分类器的反事实解释，属于视频理解领域，其核心任务是生成具有时间连贯性的反事实视频以解释模型决策。这与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析或长文档推理无直接关联。方法聚焦于视频生成中的运动平滑性和物理合理性，难以迁移到文档场景中。且未涉及多模态语言模型在文档上的推理、压缩或对齐优化。
Abstract Counterfactual explanations (CFEs) are minimal and semantically meaningful modifications of the input of a model that alter the model predictions. They highlight the decisive features the model relies on, providing contrastive interpretations for classifiers. State-of-the-art visual counterfactual explanation methods are designed to explain image classifiers. The generation of CFEs for video classifiers remains largely underexplored. For the counterfactual videos to be useful, they have to be physically plausible, temporally coherent, and exhibit smooth motion trajectories. Existing CFE image-based methods, designed to explain image classifiers, lack the capacity to generate temporally coherent, smooth and physically plausible video CFEs. To address this, we propose Back To The Feature (BTTF), an optimization framework that generates video CFEs. Our method introduces two novel features, 1) an optimization scheme to retrieve the initial latent noise conditioned by the first frame of the input video, 2) a two-stage optimization strategy to enable the search for counterfactual videos in the vicinity of the input video. Both optimization processes are guided solely by the target classifier, ensuring the explanation is faithful. To accelerate convergence, we also introduce a progressive optimization strategy that incrementally increases the number of denoising steps. Extensive experiments on video datasets such as Shape-Moving (motion classification), MEAD (emotion classification), and NTU RGB+D (action classification) show that our BTTF effectively generates valid, visually similar and realistic counterfactual videos that provide concrete insights into the classifier's decision-making mechanism.
Authors Chao Wang, Chengan Che, Xinyue Chen, Sophia Tsoka, Luis C. Garcia-Peraza-Herrera
Categories Computer Vision and Pattern Recognition
0
Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement ICCV 2025 Physics-IQ Challenge Third Place Solution
Title CN 通过VLM引导的迭代自精炼实现物理 grounded 的视频生成
Keywords 视频生成 · 物理一致性 · VLM引导 · 自精炼 · 多模态思维链
Summary 本文提出一种基于大语言模型和视觉-语言模型引导的迭代自精炼框架，用于提升视频生成中对物理规律的遵循能力。通过多模态思维链（MM-CoT）机制，根据物理不一致性反馈不断优化生成提示，无需训练即可提升视频生成质量。在PhyIQ基准上取得了显著分数提升。
Reason 该论文属于视频生成领域，且核心任务是提升视频生成的物理一致性，与文档图像理解（DIU）或DocVQA无关。尽管使用了VLM和多模态推理机制，但其方法针对的是动态场景中的物理规律建模，无法迁移至文档理解中的OCR幻觉、复杂排版或长文本推理等核心痛点。
Abstract Recent progress in video generation has led to impressive visual quality, yet current models still struggle to produce results that align with real-world physical principles. To this end, we propose an iterative self-refinement framework that leverages large language models and vision-language models to provide physics-aware guidance for video generation. Specifically, we introduce a multimodal chain-of-thought (MM-CoT) process that refines prompts based on feedback from physical inconsistencies, progressively enhancing generation quality. This method is training-free and plug-and-play, making it readily applicable to a wide range of video generation models. Experiments on the PhyIQ benchmark show that our method improves the Physics-IQ score from 56.31 to 62.38. We hope this work serves as a preliminary exploration of physics-consistent video generation and may offer insights for future research.
Authors Yang Liu, Xilin Zhao, Peisong Wen, Siran Dai, Qingming Huang
Comment ICCV 2025 Physics-IQ Challenge Third Place Solution
Categories Computer Vision and Pattern Recognition
0
SelfMOTR: Revisiting MOTR with Self-Generating Detection Priors
Title CN SelfMOTR：通过自生成检测先验重新审视MOTR
Keywords 多目标跟踪 · Transformer · 检测先验 · 端到端模型 · 视频理解
Summary 本文提出SelfMOTR，一种基于Transformer的端到端多目标跟踪模型，通过挖掘并利用MOTR类模型自身隐藏的检测能力，生成检测先验来提升跟踪性能。该方法在DanceTrack数据集上表现优异，但其研究领域为视频目标跟踪，与文档图像理解无关。
Reason 该论文研究的是多目标跟踪（MOT）任务，属于视频理解领域，主要解决检测与关联在联合架构中的冲突问题。其核心方法围绕自生成检测先验提升跟踪性能，与文档图像理解（DIU）或DocVQA等任务无直接关联。无法迁移用于解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract Despite progress toward end-to-end tracking with transformer architectures, poor detection performance and the conflict between detection and association in a joint architecture remain critical concerns. Recent approaches aim to mitigate these issues by (i) employing advanced denoising or label assignment strategies, or (ii) incorporating detection priors from external object detectors via distillation or anchor proposal techniques. Inspired by the success of integrating detection priors and by the key insight that MOTR-like models are secretly strong detection models, we introduce SelfMOTR, a novel tracking transformer that relies on self-generated detection priors. Through extensive analysis and ablation studies, we uncover and demonstrate the hidden detection capabilities of MOTR-like models, and present a practical set of tools for leveraging them effectively. On DanceTrack, SelfMOTR achieves strong performance, competing with recent state-of-the-art end-to-end tracking methods.
Authors Fabian Gülhan, Emil Mededovic, Yuli Wu, Johannes Stegmaier
Comment 11 pages, 5 figures, 10 tables
Categories Computer Vision and Pattern Recognition
0
DAPointMamba: Domain Adaptive Point Mamba for Point Cloud Completion AAAI 2026
Title CN DAPointMamba：用于点云补全的领域自适应点Mamba
Keywords 点云补全 · 领域自适应 · 状态空间模型 · 3D视觉 · SSM
Summary 本文提出DAPointMamba，是首个将状态空间模型（SSMs）应用于领域自适应点云补全（DA PCC）的工作。针对直接序列化3D点云会破坏空间拓扑结构的问题，设计了跨域块级扫描、跨域空间SSM对齐和跨域通道SSM对齐三个模块，以增强局部对齐和全局语义一致性。实验表明该方法在多个基准上优于现有技术，且具有更低的计算复杂度。
Reason 该论文研究的是点云补全中的领域自适应问题，属于3D视觉与点云处理范畴，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法虽在SSM架构上有所创新，但应用场景为3D几何结构建模，无法迁移至文档图像中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Domain adaptive point cloud completion (DA PCC) aims to narrow the geometric and semantic discrepancies between the labeled source and unlabeled target domains. Existing methods either suffer from limited receptive fields or quadratic complexity due to using CNNs or vision Transformers. In this paper, we present the first work that studies the adaptability of State Space Models (SSMs) in DA PCC and find that directly applying SSMs to DA PCC will encounter several challenges: directly serializing 3D point clouds into 1D sequences often disrupts the spatial topology and local geometric features of the target domain. Besides, the overlook of designs in the learning domain-agnostic representations hinders the adaptation performance. To address these issues, we propose a novel framework, DAPointMamba for DA PCC, that exhibits strong adaptability across domains and has the advantages of global receptive fields and efficient linear complexity. It has three novel modules. In particular, Cross-Domain Patch-Level Scanning introduces patch-level geometric correspondences, enabling effective local alignment. Cross-Domain Spatial SSM Alignment further strengthens spatial consistency by modulating patch features based on cross-domain similarity, effectively mitigating fine-grained structural discrepancies. Cross-Domain Channel SSM Alignment actively addresses global semantic gaps by interleaving and aligning feature channels. Extensive experiments on both synthetic and real-world benchmarks demonstrate that our DAPointMamba outperforms state-of-the-art methods with less computational complexity and inference latency.
Authors Yinghui Li, Qianyu Zhou, Di Shao, Hao Yang, Ye Zhu, Richard Dazeley, Xuequan Lu
Comment Accepted to AAAI 2026
Categories Computer Vision and Pattern Recognition
0
DRL-Guided Neural Batch Sampling for Semi-Supervised Pixel-Level Anomaly Detection
Title CN 基于深度强化学习的神经批量采样在半监督像素级异常检测中的应用
Keywords 异常检测 · 深度强化学习 · 半监督学习 · 工业视觉检测 · 自编码器
Summary 本文提出一种基于深度强化学习的半监督框架，用于工业图像中的像素级异常检测。通过RL引导的神经网络批量采样器选择信息丰富的图像块，在仅有少量标注数据的情况下，结合自编码器生成的损失轮廓和预测器实现异常区域的精确定位。实验表明该方法在MVTec AD数据集上优于现有方法。
Reason 该论文研究的是工业视觉检测中的像素级异常检测问题，属于通用图像异常检测范畴，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法聚焦于缺陷样本稀缺下的图像重建与分割，并未涉及OCR、文本理解、布局分析、表格识别等DIU核心问题。尽管使用了强化学习进行采样优化，但该机制针对的是图像块的选择，难以迁移到文档理解中的OCR幻觉、复杂排版解析或长文档推理等痛点。属于无关领域工作。
Abstract Anomaly detection in industrial visual inspection is challenging due to the scarcity of defective samples. Most existing methods rely on unsupervised reconstruction using only normal data, often resulting in overfitting and poor detection of subtle defects. We propose a semi-supervised deep reinforcement learning framework that integrates a neural batch sampler, an autoencoder, and a predictor. The RL-based sampler adaptively selects informative patches by balancing exploration and exploitation through a composite reward. The autoencoder generates loss profiles highlighting abnormal regions, while the predictor performs segmentation in the loss-profile space. This interaction enables the system to effectively learn both normal and defective patterns with limited labeled data. Experiments on the MVTec AD dataset demonstrate that our method achieves higher accuracy and better localization of subtle anomalies than recent state-of-the-art approaches while maintaining low complexity, yielding an average improvement of 0.15 in F1_max and 0.06 in AUC, with a maximum gain of 0.37 in F1_max in the best case.
Authors Amirhossein Khadivi Noghredeh, Abdollah Safari, Fatemeh Ziaeetabar, Firoozeh Haghighi
Categories Computer Vision and Pattern Recognition
0
Advancing Image Classification with Discrete Diffusion Classification Modeling
Title CN 通过离散扩散分类建模推进图像分类
Keywords 图像分类 · 扩散模型 · 后验建模 · 高不确定性 · DiDiCM
Summary 本文提出了一种基于扩散的分类建模框架DiDiCM，通过扩散过程建模图像分类中类别的后验分布，在ImageNet等数据集上验证了其在输入退化或数据有限情况下的优越性。方法聚焦于标准图像分类任务的鲁棒性提升，不涉及文本内容理解或多模态文档处理。
Reason 该论文属于通用图像分类领域，研究的是在高不确定性条件下提升分类性能的方法，使用扩散模型对类别分布进行建模。其任务设定与文档图像理解（DIU）的核心问题如文本识别、布局分析、表格理解、视觉-语言推理等无直接关联。所提出的DiDiCM方法针对的是自然图像的分类置信度建模，并未涉及OCR、多模态对齐、细粒度文本定位或文档结构建模等问题，难以迁移到解决DIU中的OCR幻觉、复杂排版或长文档推理等关键挑战。
Abstract Image classification is a well-studied task in computer vision, and yet it remains challenging under high-uncertainty conditions, such as when input images are corrupted or training data are limited. Conventional classification approaches typically train models to directly predict class labels from input images, but this might lead to suboptimal performance in such scenarios. To address this issue, we propose Discrete Diffusion Classification Modeling (DiDiCM), a novel framework that leverages a diffusion-based procedure to model the posterior distribution of class labels conditioned on the input image. DiDiCM supports diffusion-based predictions either on class probabilities or on discrete class labels, providing flexibility in computation and memory trade-offs. We conduct a comprehensive empirical study demonstrating the superior performance of DiDiCM over standard classifiers, showing that a few diffusion iterations achieve higher classification accuracy on the ImageNet dataset compared to baselines, with accuracy gains increasing as the task becomes more challenging. We release our code at https://github.com/omerb01/didicm .
Authors Omer Belhasin, Shelly Golan, Ran El-Yaniv, Michael Elad
Categories Computer Vision and Pattern Recognition
0
The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation
Title CN 图像即奖励：基于对抗奖励的图像生成强化学习
Keywords 图像生成 · 强化学习 · 对抗奖励 · 视觉基础模型 · 奖励黑客
Summary 本文提出Adv-GRPO，一种基于对抗奖励的强化学习框架，用于图像生成。该方法将参考图像和视觉基础模型（如DINO）提取的视觉特征作为密集奖励信号，动态更新奖励模型和生成器，从而避免传统标量奖励易被‘奖励劫持’的问题，并在图像质量与美学方面取得提升。
Reason 该论文属于纯图像生成领域，研究的是利用对抗奖励机制提升生成图像的质量和美学表现，其核心方法依赖于视觉基础模型（如DINO）提供的密集视觉信号来替代标量奖励。尽管使用了视觉信号进行反馈，但其目标是通用或文本到图像生成中的奖励设计，与文档图像理解（DIU）中的OCR幻觉、复杂排版解析、细粒度文本定位等核心痛点无关。无法迁移用于解决文档理解任务中的结构识别、逻辑推理或去幻觉问题。
Abstract A reliable reward function is essential for reinforcement learning (RL) in image generation. Most current RL approaches depend on pre-trained preference models that output scalar rewards to approximate human preferences. However, these rewards often fail to capture human perception and are vulnerable to reward hacking, where higher scores do not correspond to better images. To address this, we introduce Adv-GRPO, an RL framework with an adversarial reward that iteratively updates both the reward model and the generator. The reward model is supervised using reference images as positive samples and can largely avoid being hacked. Unlike KL regularization that constrains parameter updates, our learned reward directly guides the generator through its visual outputs, leading to higher-quality images. Moreover, while optimizing existing reward functions can alleviate reward hacking, their inherent biases remain. For instance, PickScore may degrade image quality, whereas OCR-based rewards often reduce aesthetic fidelity. To address this, we take the image itself as a reward, using reference images and vision foundation models (e.g., DINO) to provide rich visual rewards. These dense visual signals, instead of a single scalar, lead to consistent gains across image quality, aesthetics, and task-specific metrics. Finally, we show that combining reference samples with foundation-model rewards enables distribution transfer and flexible style customization. In human evaluation, our method outperforms Flow-GRPO and SD3, achieving 70.0% and 72.4% win rates in image quality and aesthetics, respectively. Code and models have been released.
Authors Weijia Mao, Hao Chen, Zhenheng Yang, Mike Zheng Shou
Categories Computer Vision and Pattern Recognition
0
Zoo3D: Zero-Shot 3D Object Detection at Scene Level
Title CN Zoo3D：场景级零样本3D目标检测
Keywords 3D目标检测 · 零样本学习 · 开放词汇 · 点云 · 多视图一致性
Summary 本文提出了Zoo3D，首个无需训练的3D目标检测框架，通过图聚类2D实例掩码生成3D边界框，并利用开词汇语义模块进行标签分配。支持零样本和自监督模式，可在有/无相机位姿的图像上运行，在ScanNet200和ARKitScenes上达到SOTA性能。
Reason 该论文研究的是3D视觉中的开放词汇目标检测问题，属于具身智能与机器人、自动驾驶相关的3D理解领域，与文档图像理解（DIU）无直接关联。尽管其使用了2D实例掩码和多视图一致性等技术，但核心任务是3D场景理解，无法迁移解决文档中的OCR幻觉、复杂排版或长文档推理等痛点。
Abstract 3D object detection is fundamental for spatial understanding. Real-world environments demand models capable of recognizing diverse, previously unseen objects, which remains a major limitation of closed-set methods. Existing open-vocabulary 3D detectors relax annotation requirements but still depend on training scenes, either as point clouds or images. We take this a step further by introducing Zoo3D, the first training-free 3D object detection framework. Our method constructs 3D bounding boxes via graph clustering of 2D instance masks, then assigns semantic labels using a novel open-vocabulary module with best-view selection and view-consensus mask generation. Zoo3D operates in two modes: the zero-shot Zoo3D$_0$, which requires no training at all, and the self-supervised Zoo3D$_1$, which refines 3D box prediction by training a class-agnostic detector on Zoo3D$_0$-generated pseudo labels. Furthermore, we extend Zoo3D beyond point clouds to work directly with posed and even unposed images. Across ScanNet200 and ARKitScenes benchmarks, both Zoo3D$_0$ and Zoo3D$_1$ achieve state-of-the-art results in open-vocabulary 3D object detection. Remarkably, our zero-shot Zoo3D$_0$ outperforms all existing self-supervised methods, hence demonstrating the power and adaptability of training-free, off-the-shelf approaches for real-world 3D understanding. Code is available at https://github.com/col14m/zoo3d .
Authors Andrey Lemeshko, Bulat Gabdullin, Nikita Drozdov, Anton Konushin, Danila Rukhovich, Maksim Kolodiazhnyi
Categories Computer Vision and Pattern Recognition
0
PromptMoG: Enhancing Diversity in Long-Prompt Image Generation via Prompt Embedding Mixture-of-Gaussian Sampling
Title CN PromptMoG：通过提示嵌入高斯混合采样增强长提示图像生成的多样性
Keywords 文本到图像生成 · 长提示 · 多样性增强 · 高斯混合模型 · PromptMoG
Summary 本文提出PromptMoG方法，通过在嵌入空间中对提示进行高斯混合采样来提升长提示下文本到图像生成的多样性，并构建了LPD-Bench基准用于评估。实验表明该方法可在不损失语义一致性的前提下提升生成结果的多样性。
Reason 该论文研究的是文本到图像生成中长提示下的多样性问题，属于纯图像生成领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。其方法聚焦于提升生成图像的创意性和视觉多样性，而非解决OCR幻觉、复杂排版解析或文档推理等核心痛点。尽管涉及多模态模型，但应用场景和目标均偏离DIU本题，属于负面清单中的‘无关领域’。
Abstract Recent advances in text-to-image (T2I) generation have achieved remarkable visual outcomes through large-scale rectified flow models. However, how these models behave under long prompts remains underexplored. Long prompts encode rich content, spatial, and stylistic information that enhances fidelity but often suppresses diversity, leading to repetitive and less creative outputs. In this work, we systematically study this fidelity-diversity dilemma and reveal that state-of-the-art models exhibit a clear drop in diversity as prompt length increases. To enable consistent evaluation, we introduce LPD-Bench, a benchmark designed for assessing both fidelity and diversity in long-prompt generation. Building on our analysis, we develop a theoretical framework that increases sampling entropy through prompt reformulation and propose a training-free method, PromptMoG, which samples prompt embeddings from a Mixture-of-Gaussians in the embedding space to enhance diversity while preserving semantics. Extensive experiments on four state-of-the-art models, SD3.5-Large, Flux.1-Krea-Dev, CogView4, and Qwen-Image, demonstrate that PromptMoG consistently improves long-prompt generation diversity without semantic drifting.
Authors Bo-Kai Ruan, Teng-Fang Hsiao, Ling Lo, Yi-Lun Wu, Hong-Han Shuai
Comment Technical Report
Categories Computer Vision and Pattern Recognition
0
HistoSpeckle-Net: Mutual Information-Guided Deep Learning for high-fidelity reconstruction of complex OrganAMNIST images via perturbed Multimode Fibers
Title CN HistoSpeckle-Net：基于互信息引导的深度学习通过扰动多模态光纤高保真重建复杂OrganAMNIST图像
Keywords 多模态光纤成像 · 互信息损失 · 图像重建 · 深度学习 · OrganAMNIST
Summary 本文提出HistoSpeckle-Net，一种用于从多模态光纤输出的散斑图案中重建复杂医学图像的深度学习模型。通过引入基于直方图的互信息损失和三尺度特征 refinement 模块，提升了在小样本和光纤弯曲变化条件下的图像重建质量。实验表明该方法在OrganAMNIST数据集上优于U-Net和Pix2Pix等基线模型。
Reason 该论文属于纯计算机视觉中的光纤成像领域，研究目标是通过多模态光纤重建医学图像，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法聚焦于光学系统中的散斑图像重建和互信息损失函数设计，虽涉及图像保真度提升，但无法迁移解决OCR幻觉、复杂排版理解、细粒度定位或长文档推理等DIU核心痛点，属于无关领域。
Abstract Existing deep learning methods in multimode fiber (MMF) imaging often focus on simpler datasets, limiting their applicability to complex, real-world imaging tasks. These models are typically data-intensive, a challenge that becomes more pronounced when dealing with diverse and complex images. In this work, we propose HistoSpeckle-Net, a deep learning architecture designed to reconstruct structurally rich medical images from MMF speckles. To build a clinically relevant dataset, we develop an optical setup that couples laser light through a spatial light modulator (SLM) into an MMF, capturing output speckle patterns corresponding to input OrganAMNIST images. Unlike previous MMF imaging approaches, which have not considered the underlying statistics of speckles and reconstructed images, we introduce a distribution-aware learning strategy. We employ a histogram-based mutual information loss to enhance model robustness and reduce reliance on large datasets. Our model includes a histogram computation unit that estimates smooth marginal and joint histograms for calculating mutual information loss. It also incorporates a unique Three-Scale Feature Refinement Module, which leads to multiscale Structural Similarity Index Measure (SSIM) loss computation. Together, these two loss functions enhance both the structural fidelity and statistical alignment of the reconstructed images. Our experiments on the complex OrganAMNIST dataset demonstrate that HistoSpeckle-Net achieves higher fidelity than baseline models such as U-Net and Pix2Pix. It gives superior performance even with limited training samples and across varying fiber bending conditions. By effectively reconstructing complex anatomical features with reduced data and under fiber perturbations, HistoSpeckle-Net brings MMF imaging closer to practical deployment in real-world clinical environments.
Authors Jawaria Maqbool, M. Imran Cheema
Categories Computer Vision and Pattern Recognition
0
AlignCVC: Aligning Cross-View Consistency for Single-Image-to-3D Generation
Title CN AlignCVC：通过跨视图一致性对齐实现单图像到3D生成
Keywords 3D生成 · 跨视图一致性 · 分布对齐 · 图像生成 · 多视图重建
Summary 本文提出AlignCVC，一种通过分布对齐来提升单图像生成3D模型中跨视图一致性的新框架。传统方法依赖重建结果反馈优化多视图生成，但易受噪声影响。AlignCVC转而将生成与重建的多视图分布共同对齐至真实分布，采用软-硬对齐策略分别优化生成与重建模块，显著提升一致性并加速推理至仅需4步。该方法为即插即用范式，兼容多种生成与重建模型。
Reason 该论文研究的是单图像到3D的生成任务，属于3D视觉与图像生成领域，核心问题是跨视图一致性优化，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法虽在生成质量与推理效率上有改进，但应用场景和目标均远离文档结构分析、文本识别、OCR去幻觉、复杂排版理解等DIU核心痛点，无法迁移用于解决文档理解中的视觉-语言对齐或细粒度文本定位问题。
Abstract Single-image-to-3D models typically follow a sequential generation and reconstruction workflow. However, intermediate multi-view images synthesized by pre-trained generation models often lack cross-view consistency (CVC), significantly degrading 3D reconstruction performance. While recent methods attempt to refine CVC by feeding reconstruction results back into the multi-view generator, these approaches struggle with noisy and unstable reconstruction outputs that limit effective CVC improvement. We introduce AlignCVC, a novel framework that fundamentally re-frames single-image-to-3D generation through distribution alignment rather than relying on strict regression losses. Our key insight is to align both generated and reconstructed multi-view distributions toward the ground-truth multi-view distribution, establishing a principled foundation for improved CVC. Observing that generated images exhibit weak CVC while reconstructed images display strong CVC due to explicit rendering, we propose a soft-hard alignment strategy with distinct objectives for generation and reconstruction models. This approach not only enhances generation quality but also dramatically accelerates inference to as few as 4 steps. As a plug-and-play paradigm, our method, namely AlignCVC, seamlessly integrates various multi-view generation models with 3D reconstruction models. Extensive experiments demonstrate the effectiveness and efficiency of AlignCVC for single-image-to-3D generation.
Authors Xinyue Liang, Zhiyuan Ma, Lingchen Sun, Yanjun Guo, Lei Zhang
Categories Computer Vision and Pattern Recognition
0
Patch-Level Glioblastoma Subregion Classification with a Contrastive Learning-Based Encoder MICCAI 2025
Title CN 基于对比学习编码器的胶质母细胞瘤亚区域斑块级分类
Keywords 胶质母细胞瘤 · Vision Transformer · 对比学习 · 病理图像分析 · BraTS
Summary 本文提出一种基于预训练Vision Transformer的模型，通过微调方式用于胶质母细胞瘤全切片图像的斑块级分类任务，并在BraTS-Path 2025挑战赛中取得第二名。模型采用对比学习编码器结构，在官方数据集上进行了训练和验证。
Reason 该论文属于医学图像分析领域，研究任务为脑肿瘤病理切片的分类，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法聚焦于生物医学图像中的子区域分类，应用场景和技术路线均难以迁移到解决OCR幻觉、复杂排版理解、长文档推理等DIU核心痛点。
Abstract The significant molecular and pathological heterogeneity of glioblastoma, an aggressive brain tumor, complicates diagnosis and patient stratification. While traditional histopathological assessment remains the standard, deep learning offers a promising path toward objective and automated analysis of whole slide images. For the BraTS-Path 2025 Challenge, we developed a method that fine-tunes a pre-trained Vision Transformer (ViT) encoder with a dedicated classification head on the official training dataset. Our model's performance on the online validation set, evaluated via the Synapse platform, yielded a Matthews Correlation Coefficient (MCC) of 0.7064 and an F1-score of 0.7676. On the final test set, the model achieved an MCC of 0.6509 and an F1-score of 0.5330, which secured our team second place in the BraTS-Pathology 2025 Challenge. Our results establish a solid baseline for ViT-based histopathological analysis, and future efforts will focus on bridging the performance gap observed on the unseen validation data.
Authors Juexin Zhang, Qifeng Zhong, Ying Weng, Ke Chen
Comment Accepted by the International Brain Tumor Segmentation (BraTS) challenge organized at MICCAI 2025 conference
Categories Computer Vision and Pattern Recognition
0
Text-guided Controllable Diffusion for Realistic Camouflage Images Generation AAAI 2026
Title CN 基于文本引导的可控扩散模型用于逼真伪装图像生成
Keywords 伪装图像生成 · 可控扩散模型 · 文本引导生成 · 视觉语言模型 · 高频纹理细化
Summary 本文提出CT-CIG，一种基于文本引导的可控扩散模型，用于生成逼真的伪装图像。通过引入伪装揭示对话机制（CRDM）生成高质量图文对，并结合轻量级控制器和频率交互细化模块（FIRM）提升伪装对象的位置、形状与纹理一致性。实验表明该方法在语义对齐和视觉真实感方面优于现有方法。
Reason 该论文属于纯图像生成领域，研究的是伪装图像合成任务（CIG），其核心目标是生成视觉上自然且逻辑合理的伪装场景图像。尽管使用了VLM和Stable Diffusion等技术，但其方法聚焦于生成控制与纹理优化，并未涉及文档图像理解中的任何核心问题（如OCR幻觉、密集文本识别、版面分析或表格结构提取）。该工作无法迁移到DocVQA或相关DIU任务中解决实际痛点，属于无关领域的应用型研究。
Abstract Camouflage Images Generation (CIG) is an emerging research area that focuses on synthesizing images in which objects are harmoniously blended and exhibit high visual consistency with their surroundings. Existing methods perform CIG by either fusing objects into specific backgrounds or outpainting the surroundings via foreground object-guided diffusion. However, they often fail to obtain natural results because they overlook the logical relationship between camouflaged objects and background environments. To address this issue, we propose CT-CIG, a Controllable Text-guided Camouflage Images Generation method that produces realistic and logically plausible camouflage images. Leveraging Large Visual Language Models (VLM), we design a Camouflage-Revealing Dialogue Mechanism (CRDM) to annotate existing camouflage datasets with high-quality text prompts. Subsequently, the constructed image-prompt pairs are utilized to finetune Stable Diffusion, incorporating a lightweight controller to guide the location and shape of camouflaged objects for enhanced camouflage scene fitness. Moreover, we design a Frequency Interaction Refinement Module (FIRM) to capture high-frequency texture features, facilitating the learning of complex camouflage patterns. Extensive experiments, including CLIPScore evaluation and camouflage effectiveness assessment, demonstrate the semantic alignment of our generated text prompts and CT-CIG's ability to produce photorealistic camouflage images.
Authors Yuhang Qian, Haiyan Chen, Wentong Li, Ningzhong Liu, Jie Qin
Comment Accepted by AAAI 2026
Categories Computer Vision and Pattern Recognition
0
Exploring Convolutional Neural Networks for Rice Grain Classification: An Explainable AI Approach
Title CN 基于卷积神经网络的稻米粒分类探索：一种可解释的人工智能方法
Keywords 卷积神经网络 · 稻米分类 · 可解释AI · 图像分类 · LIME
Summary 本文提出了一种基于卷积神经网络（CNN）的自动稻米品种分类框架，并结合LIME和SHAP等可解释性技术分析模型决策过程。实验结果显示该模型在准确率、ROC曲线下面积和混淆矩阵指标上表现优异，能有效区分不同类型的稻米颗粒。
Reason 该论文研究的是农业图像中的稻米粒分类问题，属于通用图像分类任务，与文档图像理解（DIU）或文档视觉问答（DocVQA）等核心任务无关。尽管使用了CNN和可解释AI技术（如LIME/SHAP），但其方法并未针对OCR幻觉、复杂排版、长文档推理等DIU痛点设计，也无法直接迁移为解决文档理解问题的底层工具。
Abstract Rice is an essential staple food worldwide that is important in promoting international trade, economic growth, and nutrition. Asian countries such as China, India, Pakistan, Thailand, Vietnam, and Indonesia are notable for their significant contribution to the cultivation and utilization of rice. These nations are also known for cultivating different rice grains, including short and long grains. These sizes are further classified as basmati, jasmine, kainat saila, ipsala, arborio, etc., catering to diverse culinary preferences and cultural traditions. For both local and international trade, inspecting and maintaining the quality of rice grains to satisfy customers and preserve a country's reputation is necessary. Manual quality check and classification is quite a laborious and time-consuming process. It is also highly prone to mistakes. Therefore, an automatic solution must be proposed for the effective and efficient classification of different varieties of rice grains. This research paper presents an automatic framework based on a convolutional neural network (CNN) for classifying different varieties of rice grains. We evaluated the proposed model based on performance metrics such as accuracy, recall, precision, and F1-Score. The CNN model underwent rigorous training and validation, achieving a remarkable accuracy rate and a perfect area under each class's Receiver Operating Characteristic (ROC) curve. The confusion matrix analysis confirmed the model's effectiveness in distinguishing between the different rice varieties, indicating minimal misclassifications. Additionally, the integration of explainability techniques such as LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations) provided valuable insights into the model's decision-making process, revealing how specific features of the rice grains influenced classification outcomes.
Authors Muhammad Junaid Asif, Hamza Khan, Rabia Tehseen, Rana Fayyaz Ahmad, Mujtaba Asad, Syed Tahir Hussain Rizvi, Shazia Saqib
Categories Computer Vision and Pattern Recognition
0
Robust 3D Brain MRI Inpainting with Random Masking Augmentation International Brain Tumor Segmentation (BraTS) challenge organized at MICCAI 2025 conference
Title CN 基于随机掩码增强的鲁棒3D脑部MRI修复
Keywords MRI修复 · U-Net · 随机掩码增强 · 医学图像生成 · BraTS挑战赛
Summary 本文提出了一种用于3D脑肿瘤MRI图像修复的深度学习框架，采用U-Net结构并结合随机掩码增强策略，在ASNR-MICCAI BraTS-Inpainting Challenge 2025中取得第一名。方法专注于医学图像中缺失区域的健康组织重建，评估指标显示其在SSIM、PSNR和MSE上表现优异。
Reason 该论文属于纯医学图像生成与修复领域，研究的是脑部MRI图像的inpainting任务，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法聚焦于3D医学影像的健康组织合成，使用U-Net架构和数据增强策略提升泛化能力，但并未涉及文本识别、布局分析、表格理解或多模态语言模型在文档上的应用。该技术难以迁移到解决OCR幻觉、复杂排版解析或长文档推理等DIU核心痛点，属于无关领域。
Abstract The ASNR-MICCAI BraTS-Inpainting Challenge was established to mitigate dataset biases that limit deep learning models in the quantitative analysis of brain tumor MRI. This paper details our submission to the 2025 challenge, a novel deep learning framework for synthesizing healthy tissue in 3D scans. The core of our method is a U-Net architecture trained to inpaint synthetically corrupted regions, enhanced with a random masking augmentation strategy to improve generalization. Quantitative evaluation confirmed the efficacy of our approach, yielding an SSIM of 0.873$\pm$0.004, a PSNR of 24.996$\pm$4.694, and an MSE of 0.005$\pm$0.087 on the validation set. On the final online test set, our method achieved an SSIM of 0.919$\pm$0.088, a PSNR of 26.932$\pm$5.057, and an RMSE of 0.052$\pm$0.026. This performance secured first place in the BraTS-Inpainting 2025 challenge and surpassed the winning solutions from the 2023 and 2024 competitions on the official leaderboard.
Authors Juexin Zhang, Ying Weng, Ke Chen
Comment Accepted by the International Brain Tumor Segmentation (BraTS) challenge organized at MICCAI 2025 conference
Categories Computer Vision and Pattern Recognition
0
ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichment
Title CN ContextFlow：通过自适应上下文增强实现无需训练的视频对象编辑
Keywords 视频编辑 · 无需训练 · 扩散变换器 · 上下文增强 · 时间一致性
Summary 本文提出ContextFlow，一种基于Diffusion Transformer（DiT）的无需训练的视频对象编辑框架。通过引入高阶Rectified Flow求解器提升反演质量，并设计自适应上下文增强机制，避免传统特征替换导致的上下文冲突。同时提出一种数据驱动的方法，利用新定义的‘引导响应度量’自动识别不同编辑任务（如插入、替换）中最关键的网络层，实现精准干预。实验表明该方法在保持时间连贯性和视觉保真度方面显著优于现有方法。
Reason 该论文研究的是视频对象编辑任务，属于视频生成与编辑领域，主要解决视频中物体插入、替换和删除的保真度与时间一致性问题。其核心技术如高阶Rectified Flow求解器、自适应上下文增强机制和DiT模块选择策略，均针对视频扩散模型中的动态内容生成设计，与文档图像理解（DIU）的核心痛点（如OCR幻觉、密集文本识别、复杂版式分析、长文档推理）无直接关联。尽管‘上下文增强’听起来具有通用性，但其迁移至文档理解场景的路径不明确，且未涉及多模态语言模型在结构化信息提取中的应用。因此属于无关领域，直接打0分。
Abstract Training-free video object editing aims to achieve precise object-level manipulation, including object insertion, swapping, and deletion. However, it faces significant challenges in maintaining fidelity and temporal consistency. Existing methods, often designed for U-Net architectures, suffer from two primary limitations: inaccurate inversion due to first-order solvers, and contextual conflicts caused by crude "hard" feature replacement. These issues are more challenging in Diffusion Transformers (DiTs), where the unsuitability of prior layer-selection heuristics makes effective guidance challenging. To address these limitations, we introduce ContextFlow, a novel training-free framework for DiT-based video object editing. In detail, we first employ a high-order Rectified Flow solver to establish a robust editing foundation. The core of our framework is Adaptive Context Enrichment (for specifying what to edit), a mechanism that addresses contextual conflicts. Instead of replacing features, it enriches the self-attention context by concatenating Key-Value pairs from parallel reconstruction and editing paths, empowering the model to dynamically fuse information. Additionally, to determine where to apply this enrichment (for specifying where to edit), we propose a systematic, data-driven analysis to identify task-specific vital layers. Based on a novel Guidance Responsiveness Metric, our method pinpoints the most influential DiT blocks for different tasks (e.g., insertion, swapping), enabling targeted and highly effective guidance. Extensive experiments show that ContextFlow significantly outperforms existing training-free methods and even surpasses several state-of-the-art training-based approaches, delivering temporally coherent, high-fidelity results.
Authors Yiyang Chen, Xuanhua He, Xiujun Ma, Yue Ma
Comment The project page is at https://yychen233.github.io/ContextFlow-page
Categories Computer Vision and Pattern Recognition
0
Exo2EgoSyn: Unlocking Foundation Video Generation Models for Exocentric-to-Egocentric Video Synthesis
Title CN Exo2EgoSyn：解锁基础视频生成模型用于外视角到内视角视频合成
Keywords 视频生成 · 跨视角合成 · 第一人称视频 · 基础模型 · 姿态注入
Summary 本文提出Exo2EgoSyn，通过改进WAN 2.2模型实现从第三方视角视频生成第一人称视角视频。方法包括视角对齐、多视角条件聚合和姿态感知隐变量注入，在ExoEgo4D数据集上验证了有效性。但该工作属于视频生成领域，与文档理解无关。
Reason 该论文研究的是外视角到内视角的视频合成，属于视频生成与跨视角合成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心技术聚焦于视频生成中的视点转换和姿态注入，无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心问题。
Abstract Foundation video generation models such as WAN 2.2 exhibit strong text- and image-conditioned synthesis abilities but remain constrained to the same-view generation setting. In this work, we introduce Exo2EgoSyn, an adaptation of WAN 2.2 that unlocks Exocentric-to-Egocentric(Exo2Ego) cross-view video synthesis. Our framework consists of three key modules. Ego-Exo View Alignment(EgoExo-Align) enforces latent-space alignment between exocentric and egocentric first-frame representations, reorienting the generative space from the given exo view toward the ego view. Multi-view Exocentric Video Conditioning (MultiExoCon) aggregates multi-view exocentric videos into a unified conditioning signal, extending WAN2.2 beyond its vanilla single-image or text conditioning. Furthermore, Pose-Aware Latent Injection (PoseInj) injects relative exo-to-ego camera pose information into the latent state, guiding geometry-aware synthesis across viewpoints. Together, these modules enable high-fidelity ego view video generation from third-person observations without retraining from scratch. Experiments on ExoEgo4D validate that Exo2EgoSyn significantly improves Ego2Exo synthesis, paving the way for scalable cross-view video generation with foundation models. Source code and models will be released publicly.
Authors Mohammad Mahdi, Yuqian Fu, Nedko Savov, Jiancheng Pan, Danda Pani Paudel, Luc Van Gool
Categories Computer Vision and Pattern Recognition
0
OmniLens++: Blind Lens Aberration Correction via Large LensLib Pre-Training and Latent PSF Representation
Title CN OmniLens++：通过大型镜头库预训练和潜在PSF表示实现盲镜头像差校正
Keywords 镜头像差校正 · 盲去模糊 · 点扩散函数 · VQVAE · 计算成像
Summary 本文提出OmniLens++框架，用于解决盲镜头像差校正中的数据扩展性与退化先验建模问题。通过构建更均匀且多样化的镜头退化数据分布，并引入基于VQVAE的潜在点扩散函数（PSF）表示来编码光学退化先验，提升了模型在未知镜头上的泛化能力。实验表明该方法在真实与合成镜头数据上均达到先进水平。
Reason 该论文研究的是光学镜头像差的盲校正问题，属于图像恢复与计算成像领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心技术如Latent PSF Representation和LensLib预训练聚焦于物理成像退化建模，难以迁移到文档理解中的OCR幻觉、复杂版面解析或长文档推理等核心痛点。属于无关领域工作。
Abstract Emerging deep-learning-based lens library pre-training (LensLib-PT) pipeline offers a new avenue for blind lens aberration correction by training a universal neural network, demonstrating strong capability in handling diverse unknown optical degradations. This work proposes the OmniLens++ framework, which resolves two challenges that hinder the generalization ability of existing pipelines: the difficulty of scaling data and the absence of prior guidance characterizing optical degradation. To improve data scalability, we expand the design specifications to increase the degradation diversity of the lens source, and we sample a more uniform distribution by quantifying the spatial-variation patterns and severity of optical degradation. In terms of model design, to leverage the Point Spread Functions (PSFs), which intuitively describe optical degradation, as guidance in a blind paradigm, we propose the Latent PSF Representation (LPR). The VQVAE framework is introduced to learn latent features of LensLib's PSFs, which is assisted by modeling the optical degradation process to constrain the learning of degradation priors. Experiments on diverse aberrations of real-world lenses and synthetic LensLib show that OmniLens++ exhibits state-of-the-art generalization capacity in blind aberration correction. Beyond performance, the AODLibpro is verified as a scalable foundation for more effective training across diverse aberrations, and LPR can further tap the potential of large-scale LensLib. The source code and datasets will be made publicly available at https://github.com/zju-jiangqi/OmniLens2.
Authors Qi Jiang, Xiaolong Qian, Yao Gao, Lei Sun, Kailun Yang, Zhonghua Yi, Wenyong Li, Ming-Hsuan Yang, Luc Van Gool, Kaiwei Wang
Comment The source code and datasets will be made publicly available at https://github.com/zju-jiangqi/OmniLens2
Categories Computer Vision and Pattern Recognition
0
Realizing Fully-Integrated, Low-Power, Event-Based Pupil Tracking with Neuromorphic Hardware
Title CN 利用神经形态硬件实现完全集成、低功耗的基于事件的瞳孔追踪
Keywords 事件相机 · 神经形态计算 · 脉冲神经网络 · 低功耗 · 可穿戴设备
Summary 本文提出了一种完全集成的低功耗瞳孔追踪系统，采用事件相机与神经形态芯片（Speck2f）结合的方式，在可穿戴设备上实现了每眼低于5mW功耗、100Hz频率的双目瞳孔追踪。系统使用轻量级脉冲神经网络进行坐标解码，并在真实多用户数据集上验证了有效性。这是首个具备电池供电、全片上处理能力的事件驱动型瞳孔追踪解决方案。
Reason 该论文研究的是基于事件相机和神经形态计算的瞳孔追踪系统，属于可穿戴设备与生物信号感知领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其核心技术聚焦于低功耗嵌入式系统设计、脉冲神经网络与硬件协同优化，并不涉及文本识别、布局分析、表格理解或任何DocVQA相关任务。无法迁移用于解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract Eye tracking is fundamental to numerous applications, yet achieving robust, high-frequency tracking with ultra-low power consumption remains challenging for wearable platforms. While event-based vision sensors offer microsecond resolution and sparse data streams, they have lacked fully integrated, low-power processing solutions capable of real-time inference. In this work, we present the first battery-powered, wearable pupil-center-tracking system with complete on-device integration, combining event-based sensing and neuromorphic processing on the commercially available Speck2f system-on-chip with lightweight coordinate decoding on a low-power microcontroller. Our solution features a novel uncertainty-quantifying spiking neural network with gated temporal decoding, optimized for strict memory and bandwidth constraints, complemented by systematic deployment mechanisms that bridge the reality gap. We validate our system on a new multi-user dataset and demonstrate a wearable prototype with dual neuromorphic devices achieving robust binocular pupil tracking at 100 Hz with an average power consumption below 5 mW per eye. Our work demonstrates that end-to-end neuromorphic computing enables practical, always-on eye tracking for next-generation energy-efficient wearable systems.
Authors Federico Paredes-Valles, Yoshitaka Miyatani, Kirk Y. W. Scheper
Comment 17 pages, 14 figures, 3 tables
Categories Computer Vision and Pattern Recognition
0
ADNet: A Large-Scale and Extensible Multi-Domain Benchmark for Anomaly Detection Across 380 Real-World Categories
Title CN ADNet：一个大规模可扩展的跨380个真实世界类别的多领域异常检测基准
Keywords 异常检测 · 多模态 · 基准数据集 · Mixture-of-Experts · 跨域泛化
Summary 本文提出了ADNet，一个包含380个类别的大规模异常检测基准，涵盖电子、工业、农业食品、基础设施和医疗等领域，共196,294张RGB图像，并提供像素级标注和结构化文本描述。实验表明现有方法在多类别扩展下性能显著下降，为此作者提出Dinomaly-m模型以提升性能。该工作聚焦于工业异常检测任务，不涉及文档理解或VLM在文本密集场景的应用。
Reason 该论文属于工业缺陷检测与异常检测领域，尽管涉及多模态标注和跨域泛化，但其核心任务与文档图像理解（DIU）无关。未涉及OCR、文本理解、版面分析、表格识别或VLM在文档上的应用。虽然提供了像素级标注和结构化文本描述，但目标是异常检测而非文档语义解析或视觉-语言推理。属于无关领域（工业视觉检测），不符合DIU本题要求，也不提供可迁移至文档理解的核心方法（如高分辨率处理、去幻觉、细粒度定位等）。
Abstract Anomaly detection (AD) aims to identify defects using normal-only training data. Existing anomaly detection benchmarks (e.g., MVTec-AD with 15 categories) cover only a narrow range of categories, limiting the evaluation of cross-context generalization and scalability. We introduce ADNet, a large-scale, multi-domain benchmark comprising 380 categories aggregated from 49 publicly available datasets across Electronics, Industry, Agrifood, Infrastructure, and Medical domains. The benchmark includes a total of 196,294 RGB images, consisting of 116,192 normal samples for training and 80,102 test images, of which 60,311 are anomalous. All images are standardized with MVTec-style pixel-level annotations and structured text descriptions spanning both spatial and visual attributes, enabling multimodal anomaly detection tasks. Extensive experiments reveal a clear scalability challenge: existing state-of-the-art methods achieve 90.6% I-AUROC in one-for-one settings but drop to 78.5% when scaling to all 380 categories in a multi-class setting. To address this, we propose Dinomaly-m, a context-guided Mixture-of-Experts extension of Dinomaly that expands decoder capacity without increasing inference cost. It achieves 83.2% I-AUROC and 93.1% P-AUROC, demonstrating superior performance over existing approaches. ADNet is designed as a standardized and extensible benchmark, supporting the community in expanding anomaly detection datasets across diverse domains and providing a scalable foundation for future anomaly detection foundation models. Dataset: https://grainnet.github.io/ADNet
Authors Hai Ling, Jia Guo, Zhulin Tao, Yunkang Cao, Donglin Di, Hongyan Xu, Xiu Su, Yang Song, Lei Fan
Categories Computer Vision and Pattern Recognition
0
Time-step Mixup for Efficient Spiking Knowledge Transfer from Appearance to Event Domain
Title CN 面向事件域外观到事件知识迁移的时序混合方法
Keywords 事件相机 · 脉冲神经网络 · 跨模态迁移 · 知识蒸馏 · 时间步混合
Summary 本文提出了一种名为Time-step Mixup Knowledge Transfer (TMKT) 的方法，用于在脉冲神经网络中实现从RGB图像到事件相机数据的细粒度知识迁移。通过在不同时间步对两种模态的数据进行插值混合，并引入模态感知的辅助学习目标来支持标签混合，从而缓解跨模态训练中的分布差异问题。实验表明该方法在多个数据集上提升了脉冲图像分类性能。
Reason 该论文研究的是从RGB图像到事件相机（event camera）数据的跨模态知识迁移，核心关注点为脉冲神经网络（SNN）中的高效视觉处理，属于特定传感器与神经形态计算领域。其技术路线聚焦于时间步上的混合策略以弥合模态分布差异，与文档图像理解（DIU）中的OCR幻觉、复杂排版、长文档推理等核心痛点无直接关联。且该工作属于视频感知的边缘分支，无法迁移至文档视觉或VLM架构改进中。
Abstract The integration of event cameras and spiking neural networks holds great promise for energy-efficient visual processing. However, the limited availability of event data and the sparse nature of DVS outputs pose challenges for effective training. Although some prior work has attempted to transfer semantic knowledge from RGB datasets to DVS, they often overlook the significant distribution gap between the two modalities. In this paper, we propose Time-step Mixup knowledge transfer (TMKT), a novel fine-grained mixing strategy that exploits the asynchronous nature of SNNs by interpolating RGB and DVS inputs at various time-steps. To enable label mixing in cross-modal scenarios, we further introduce modality-aware auxiliary learning objectives. These objectives support the time-step mixup process and enhance the model's ability to discriminate effectively across different modalities. Our approach enables smoother knowledge transfer, alleviates modality shift during training, and achieves superior performance in spiking image classification tasks. Extensive experiments demonstrate the effectiveness of our method across multiple datasets. The code will be released after the double-blind review process.
Authors Yuqi Xie, Shuhan Ye, Yi Yu, Chong Wang, Qixin Zhang, Jiazhen Xu, Le Shen, Yuanbin Qian, Jiangbo Qian, Guoqi Li
Categories Computer Vision and Pattern Recognition
0
Multi-view Surface Reconstruction Using Normal and Reflectance Cues IJCV
Title CN 利用法线和反射线索的多视角表面重建
Keywords 3D重建 · 多视角立体 · 表面法线 · 神经渲染 · 光度立体
Summary 本文提出一种结合多视角法线图和反射特性进行高精度3D表面重建的框架，通过将反射率和表面法线联合参数化为在不同光照下的辐射向量，融入传统或多视图立体匹配（MVS）及神经体积渲染（NVR）流程中。在多个多视角光度立体基准上达到SOTA，尤其擅长处理复杂材质和遮挡情况。但整体工作属于3D重建范畴，与文档理解无关。
Reason 该论文属于3D视觉与计算机视觉中的表面重建领域，研究目标是高保真3D几何建模，使用多视角图像和反射属性进行精细细节恢复。其技术路线聚焦于物理光照建模与神经渲染，与文档图像理解（DIU）、DocVQA、OCR、布局分析等任务无直接关联。尽管涉及多模态（图像+几何）信息融合，但方法针对的是物体表面反射与形状估计，无法迁移解决文档中的OCR幻觉、复杂排版或长文档推理等问题。
Abstract Achieving high-fidelity 3D surface reconstruction while preserving fine details remains challenging, especially in the presence of materials with complex reflectance properties and without a dense-view setup. In this paper, we introduce a versatile framework that incorporates multi-view normal and optionally reflectance maps into radiance-based surface reconstruction. Our approach employs a pixel-wise joint re-parametrization of reflectance and surface normals, representing them as a vector of radiances under simulated, varying illumination. This formulation enables seamless incorporation into standard surface reconstruction pipelines, such as traditional multi-view stereo (MVS) frameworks or modern neural volume rendering (NVR) ones. Combined with the latter, our approach achieves state-of-the-art performance on multi-view photometric stereo (MVPS) benchmark datasets, including DiLiGenT-MV, LUCES-MV and Skoltech3D. In particular, our method excels in reconstructing fine-grained details and handling challenging visibility conditions. The present paper is an extended version of the earlier conference paper by Brument et al. (in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024), featuring an accelerated and more robust algorithm as well as a broader empirical evaluation. The code and data relative to this article is available at https://github.com/RobinBruneau/RNb-NeuS2.
Authors Robin Bruneau, Baptiste Brument, Yvain Quéau, Jean Mélou, François Bernard Lauze, Jean-Denis Durou, Lilian Calvet
Comment 22 pages, 15 figures, 11 tables. A thorough qualitative and quantitive study is available in the supplementary material at https://drive.google.com/file/d/1KDfCKediXNP5Os954TL_QldaUWS0nKcD/view?usp=drive_link. Accepted to IJCV
Categories Computer Vision and Pattern Recognition
0
M2SVid: End-to-End Inpainting and Refinement for Monocular-to-Stereo Video Conversion 3DV 2026
Title CN M2SVid：单目到立体视频转换的端到端修复与优化
Keywords 视频生成 · 立体视觉 · 图像修复 · 扩散模型 · 3D重建
Summary 本文提出M2SVid模型，用于单目视频到立体视频的转换，通过改进Stable Video Diffusion架构，利用左视图视频、扭曲的右视图和遮挡掩码作为条件输入，实现高质量右视图视频的端到端生成。模型优化了注意力机制以更好地处理遮挡区域，并在用户研究中表现优于现有方法，同时速度更快。
Reason 该论文研究的是单目视频转立体视频中的图像修复与视图生成问题，属于纯视觉生成与3D视觉领域，任务核心是视频级别的深度重投影与扩散模型生成。尽管使用了VLM相关的扩散架构（SVD），但其目标与文档图像理解（DIU）无直接关联，无法迁移解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点。
Abstract We tackle the problem of monocular-to-stereo video conversion and propose a novel architecture for inpainting and refinement of the warped right view obtained by depth-based reprojection of the input left view. We extend the Stable Video Diffusion (SVD) model to utilize the input left video, the warped right video, and the disocclusion masks as conditioning input to generate a high-quality right camera view. In order to effectively exploit information from neighboring frames for inpainting, we modify the attention layers in SVD to compute full attention for discoccluded pixels. Our model is trained to generate the right view video in an end-to-end manner without iterative diffusion steps by minimizing image space losses to ensure high-quality generation. Our approach outperforms previous state-of-the-art methods, being ranked best 2.6x more often than the second-place method in a user study, while being 6x faster.
Authors Nina Shvetsova, Goutam Bhat, Prune Truong, Hilde Kuehne, Federico Tombari
Comment To be published at 3DV 2026, project webpage https://m2svid.github.io/
Categories Computer Vision and Pattern Recognition
0
SuperQuadricOcc: Multi-Layer Gaussian Approximation of Superquadrics for Real-Time Self-Supervised Occupancy Estimation
Title CN SuperQuadricOcc：基于超二次曲面的多层高斯近似实现实时自监督占据估计
Keywords 占据估计 · 超二次曲面 · 高斯表示 · 自监督学习 · 自动驾驶
Summary 本文提出SuperQuadricOcc，一种用于实时自监督占据估计的新方法，通过引入超二次曲面的多层高斯近似表示，解决了传统高斯表示内存开销大、难以部署的问题。该方法在Occ3D数据集上实现了更小内存占用、更快推理速度和更高mIoU性能，是首个能在保持竞争力的同时实现实时推理的占据模型。
Reason 该论文属于自动驾驶领域的3D视觉与占据网格估计任务，研究内容为使用超二次曲面建模场景以降低内存消耗并提升推理速度。其核心问题、方法与文档图像理解（DIU）完全无关，无法迁移至OCR、DocVQA、排版分析等文档理解任务中。
Abstract Semantic occupancy estimation enables comprehensive scene understanding for automated driving, providing dense spatial and semantic information essential for perception and planning. While Gaussian representations have been widely adopted in self-supervised occupancy estimation, the deployment of a large number of Gaussian primitives drastically increases memory requirements and is not suitable for real-time inference. In contrast, superquadrics permit reduced primitive count and lower memory requirements due to their diverse shape set. However, implementation into a self-supervised occupancy model is nontrivial due to the absence of a superquadric rasterizer to enable model supervision. Our proposed method, SuperQuadricOcc, employs a superquadric-based scene representation. By leveraging a multi-layer icosphere-tessellated Gaussian approximation of superquadrics, we enable Gaussian rasterization for supervision during training. On the Occ3D dataset, SuperQuadricOcc achieves a 75% reduction in memory footprint, 124% faster inference, and a 5.9% improvement in mIoU compared to previous Gaussian-based methods, without the use of temporal labels. To our knowledge, this is the first occupancy model to enable real-time inference while maintaining competitive performance. The use of superquadrics reduces the number of primitives required for scene modeling by 84% relative to Gaussian-based approaches. Finally, evaluation against prior methods is facilitated by our fast superquadric voxelization module. The code will be made available at https://github.com/seamie6/SuperQuadricOcc.
Authors Seamie Hayes, Reenu Mohandas, Tim Brophy, Alexandre Boulch, Ganesh Sistu, Ciaran Eising
Categories Computer Vision and Pattern Recognition
0
Map-World: Masked Action planning and Path-Integral World Model for Autonomous Driving
Title CN Map-World：用于自动驾驶的掩码动作规划与路径积分世界模型
Keywords 自动驾驶 · 运动规划 · 世界模型 · 多模态预测 · 掩码序列建模
Summary 本文提出MAP-World，一种无需先验锚点的多模态运动规划框架，结合掩码动作规划（MAP）与路径加权世界模型。通过将未来自车轨迹视为被掩码的序列补全任务，并引入噪声生成多样化轨迹查询，在不依赖手工锚点或强化学习的情况下实现多模态、时间一致的轨迹预测。训练时利用语义损失对多种可能未来进行期望优化，提升规划鲁棒性。
Reason 该论文研究的是自动驾驶中的运动规划问题，属于具身智能与机器人领域，核心任务为多模态轨迹预测与世界模型构建，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管其方法涉及多模态未来建模和路径加权机制，但迁移到文档理解场景（如OCR、VQA、布局分析）极为困难，且未触及DIU的核心痛点如幻觉、细粒度定位或复杂排版处理。
Abstract Motion planning for autonomous driving must handle multiple plausible futures while remaining computationally efficient. Recent end-to-end systems and world-model-based planners predict rich multi-modal trajectories, but typically rely on handcrafted anchors or reinforcement learning to select a single best mode for training and control. This selection discards information about alternative futures and complicates optimization. We propose MAP-World, a prior-free multi-modal planning framework that couples masked action planning with a path-weighted world model. The Masked Action Planning (MAP) module treats future ego motion as masked sequence completion: past waypoints are encoded as visible tokens, future waypoints are represented as mask tokens, and a driving-intent path provides a coarse scaffold. A compact latent planning state is expanded into multiple trajectory queries with injected noise, yielding diverse, temporally consistent modes without anchor libraries or teacher policies. A lightweight world model then rolls out future BEV semantics conditioned on each candidate trajectory. During training, semantic losses are computed as an expectation over modes, using trajectory probabilities as discrete path weights, so the planner learns from the full distribution of plausible futures instead of a single selected path. On NAVSIM, our method matches anchor-based approaches and achieves state-of-the-art performance among world-model-based methods, while avoiding reinforcement learning and maintaining real-time inference latency.
Authors Bin Hu, Zijian Lu, Haicheng Liao, Chengran Yuan, Bin Rao, Yongkang Li, Guofa Li, Zhiyong Cui, Cheng-zhong Xu, Zhenning Li
Categories Computer Vision and Pattern Recognition
0
Alzheimers Disease Progression Prediction Based on Manifold Mapping of Irregularly Sampled Longitudinal Data
Title CN 基于不规则采样纵向数据的流形映射阿尔茨海默病进展预测
Keywords 阿尔茨海默病 · 纵向数据分析 · 流形学习 · 神经微分方程 · 不规则采样
Summary 本文提出一种基于黎曼流形映射的时间感知神经微分方程框架（R-TNAG），用于从不规则采样的纵向结构MRI数据中预测阿尔茨海默病的进展。通过将高维特征投影到流形空间，并结合TNODE和ARGRU模块建模疾病连续演化过程，提升了在不规则采样下的预测鲁棒性。实验表明该方法在疾病状态分类和认知评分回归上优于现有方法。
Reason 该论文属于医学影像分析领域，研究的是基于sMRI数据的阿尔茨海默病进展建模，使用了流形学习和神经微分方程等方法处理不规则时间序列。虽然涉及视觉数据，但其任务与文档图像理解（DIU）无关，既不涉及文本识别、布局分析、表格理解，也不涉及多模态语言模型在文档上的应用。属于典型的垂直领域应用（医疗+视频/3D影像），且未提出可迁移至DIU领域的通用方法，如视觉压缩、高分辨率处理、去幻觉机制等。
Abstract The uncertainty of clinical examinations frequently leads to irregular observation intervals in longitudinal imaging data, posing challenges for modeling disease progression.Most existing imaging-based disease prediction models operate in Euclidean space, which assumes a flat representation of data and fails to fully capture the intrinsic continuity and nonlinear geometric structure of irregularly sampled longitudinal images. To address the challenge of modeling Alzheimers disease (AD) progression from irregularly sampled longitudinal structural Magnetic Resonance Imaging (sMRI) data, we propose a Riemannian manifold mapping, a Time-aware manifold Neural ordinary differential equation, and an Attention-based riemannian Gated recurrent unit (R-TNAG) framework. Our approach first projects features extracted from high-dimensional sMRI into a manifold space to preserve the intrinsic geometry of disease progression. On this representation, a time-aware Neural Ordinary Differential Equation (TNODE) models the continuous evolution of latent states between observations, while an Attention-based Riemannian Gated Recurrent Unit (ARGRU) adaptively integrates historical and current information to handle irregular intervals. This joint design improves temporal consistency and yields robust AD trajectory prediction under irregular sampling.Experimental results demonstrate that the proposed method consistently outperforms state-of-the-art models in both disease status prediction and cognitive score regression. Ablation studies verify the contributions of each module, highlighting their complementary roles in enhancing predictive accuracy. Moreover, the model exhibits stable performance across varying sequence lengths and missing data rates, indicating strong temporal generalizability. Cross-dataset validation further confirms its robustness and applicability in diverse clinical settings.
Authors Xin Hong, Ying Shi, Yinhao Li, Yen-Wei Chen
Comment 10 pages, 3 figures
Categories Computer Vision and Pattern Recognition
0
Prompt Guiding Multi-Scale Adaptive Sparse Representation-driven Network for Low-Dose CT MAR
Title CN 用于低剂量CT金属伪影去除的提示引导多尺度自适应稀疏表示网络
Keywords 低剂量CT · 金属伪影去除 · 多尺度表示 · 稀疏编码 · 提示引导
Summary 本文提出了一种名为PMSRNet的网络，用于同时实现低剂量CT图像重建与金属伪影去除。通过设计多尺度稀疏表示结构、可学习的阈值生成模块（PSATG）以及跨尺度特征融合机制，模型能够有效利用局部、区域和全局上下文信息。此外，引入提示引导策略使单一模型可适应不同剂量水平的输入。实验表明该方法在多个剂量条件下均优于现有技术。
Reason 该论文研究的是低剂量CT图像重建与金属伪影去除，属于医学图像处理领域，且聚焦于三维医学影像（CT），与文档图像理解（DIU）或DocVQA等任务无直接关联。尽管其使用了‘提示引导’和多尺度建模等技术，但这些方法针对的是医学成像中的物理噪声和伪影问题，难以迁移到文档图像中的OCR幻觉、复杂版式理解或细粒度文本定位等核心痛点。
Abstract Low-dose CT (LDCT) is capable of reducing X-ray radiation exposure, but it will potentially degrade image quality, even yields metal artifacts at the case of metallic implants. For simultaneous LDCT reconstruction and metal artifact reduction (LDMAR), existing deep learning-based efforts face two main limitations: i) the network design neglects multi-scale and within-scale information; ii) training a distinct model for each dose necessitates significant storage space for multiple doses. To fill these gaps, we propose a prompt guiding multi-scale adaptive sparse representation-driven network, abbreviated as PMSRNet, for LDMAR task. Specifically, we construct PMSRNet inspired from multi-scale sparsifying frames, and it can simultaneously employ within-scale characteristics and cross-scale complementarity owing to an elaborated prompt guiding scale-adaptive threshold generator (PSATG) and a built multi-scale coefficient fusion module (MSFuM). The PSATG can adaptively capture multiple contextual information to generate more faithful thresholds, achieved by fusing features from local, regional, and global levels. Furthermore, we elaborate a model interpretable dual domain LDMAR framework called PDuMSRNet, and train single model with a prompt guiding strategy for multiple dose levels. We build a prompt guiding module, whose input contains dose level, metal mask and input instance, to provide various guiding information, allowing a single model to accommodate various CT dose settings. Extensive experiments at various dose levels demonstrate that the proposed methods outperform the state-of-the-art LDMAR methods.
Authors Baoshun Shi, Bing Chen, Shaolei Zhang, Huazhu Fu, Zhanli Hu
Categories Computer Vision and Pattern Recognition
0
Vision-Language Models for Automated 3D PET/CT Report Generation
Title CN 用于自动化3D PET/CT报告生成的视觉语言模型
Keywords 医学报告生成 · 3D PET/CT · 视觉语言模型 · 多模态学习 · 临床评估
Summary 本文提出PETRG-3D模型和PETRG-Lym数据集，用于自动化生成3D PET/CT扫描的淋巴瘤报告。采用双分支结构分别编码PET和CT体积数据，并引入风格自适应提示来应对不同医院报告风格差异。同时构建了公开基准AutoPET-RG-Lym并提出专用评估指标PETRG-Score。实验显示在自然语言和临床有效性指标上均显著优于现有方法。
Reason 该论文属于医学影像生成领域，研究的是PET/CT三维图像到临床报告的生成任务，属于典型的垂直领域应用（医疗报告生成），并未提出可迁移至文档图像理解的通用方法。其技术路线聚焦于多模态医学数据的端到端建模与跨中心风格适配，与DIU中的OCR幻觉、复杂版面解析、细粒度文本定位等核心问题无关。属于‘平行下游应用’，不符合本筛选标准。
Abstract Positron emission tomography/computed tomography (PET/CT) is essential in oncology, yet the rapid expansion of scanners has outpaced the availability of trained specialists, making automated PET/CT report generation (PETRG) increasingly important for reducing clinical workload. Compared with structural imaging (e.g., X-ray, CT, and MRI), functional PET poses distinct challenges: metabolic patterns vary with tracer physiology, and whole-body 3D contextual information is required rather than local-region interpretation. To advance PETRG, we propose PETRG-3D, an end-to-end 3D dual-branch framework that separately encodes PET and CT volumes and incorporates style-adaptive prompts to mitigate inter-hospital variability in reporting practices. We construct PETRG-Lym, a multi-center lymphoma dataset collected from four hospitals (824 reports w/ 245,509 paired PET/CT slices), and construct AutoPET-RG-Lym, a publicly accessible PETRG benchmark derived from open imaging data but equipped with new expert-written, clinically validated reports (135 cases). To assess clinical utility, we introduce PETRG-Score, a lymphoma-specific evaluation protocol that jointly measures metabolic and structural findings across curated anatomical regions. Experiments show that PETRG-3D substantially outperforms existing methods on both natural language metrics (e.g., +31.49\% ROUGE-L) and clinical efficacy metrics (e.g., +8.18\% PET-All), highlighting the benefits of volumetric dual-modality modeling and style-aware prompting. Overall, this work establishes a foundation for future PET/CT-specific models emphasizing disease-aware reasoning and clinically reliable evaluation. Codes, models, and AutoPET-RG-Lym will be released.
Authors Wenpei Jiao, Kun Shang, Hui Li, Ke Yan, Jiajin Zhang, Guangjie Yang, Lijuan Guo, Yan Wan, Xing Yang, Dakai Jin, Zhaoheng Xie
Categories Computer Vision and Pattern Recognition
0
Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generation
Title CN Stand-In：一种轻量级即插即用的身份控制方法用于视频生成
Keywords 视频生成 · 身份保持 · 轻量级框架 · 即插即用 · AIGC
Summary 本文提出Stand-In，一种轻量级且可即插即用的框架，用于在视频生成过程中保持指定身份。通过在预训练视频生成模型中引入条件图像分支，并采用带有条件位置映射的受限自注意力机制，仅需约1%的额外参数和2000个训练样本即可在身份保持和视频质量上优于全参数微调方法。该框架还可兼容多种下游任务如主体驱动生成、姿态参考生成、风格化和换脸。
Reason 该论文研究的是视频生成中的人脸身份保持问题，属于纯视频生成与编辑领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。其技术核心聚焦于条件图像分支和受限自注意力机制在视频模型中的应用，无法迁移解决OCR幻觉、复杂排版解析、长文档推理等DIU核心痛点。
Abstract Generating high-fidelity human videos that match user-specified identities is important yet challenging in the field of generative AI. Existing methods often rely on an excessive number of training parameters and lack compatibility with other AIGC tools. In this paper, we propose Stand-In, a lightweight and plug-and-play framework for identity preservation in video generation. Specifically, we introduce a conditional image branch into the pre-trained video generation model. Identity control is achieved through restricted self-attentions with conditional position mapping. Thanks to these designs, which greatly preserve the pre-trained prior of the video generation model, our approach is able to outperform other full-parameter training methods in video quality and identity preservation, even with just $\sim$1% additional parameters and only 2000 training pairs. Moreover, our framework can be seamlessly integrated for other tasks, such as subject-driven video generation, pose-referenced video generation, stylization, and face swapping.
Authors Bowen Xue, Zheng-Peng Duan, Qixin Yan, Wenjing Wang, Hao Liu, Chun-Le Guo, Chongyi Li, Chen Li, Jing Lyu
Categories Computer Vision and Pattern Recognition
0
Zero-Shot Anomaly Detection with Dual-Branch Prompt Selection BMVC 2025
Title CN 基于双分支提示选择的零样本异常检测
Keywords 零样本异常检测 · 提示学习 · 测试时自适应 · 领域迁移 · 工业缺陷检测
Summary 本文提出PILOT框架用于零样本异常检测，通过双分支提示学习机制结合可学习提示与语义属性，并引入无标签测试数据上的伪标签进行测试时自适应，在13个工业与医学基准上实现了领先性能。重点在于跨域缺陷检测与定位。
Reason 该论文研究的是工业与医疗图像中的零样本异常检测问题，属于通用计算机视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心方法——双分支提示学习和测试时自适应——虽然在异常检测中有价值，但难以迁移到解决OCR幻觉、复杂排版理解、细粒度文本定位或长文档推理等DIU核心痛点。属于无关领域工作。
Abstract Zero-shot anomaly detection (ZSAD) enables identifying and localizing defects in unseen categories by relying solely on generalizable features rather than requiring any labeled examples of anomalies. However, existing ZSAD methods, whether using fixed or learned prompts, struggle under domain shifts because their training data are derived from limited training domains and fail to generalize to new distributions. In this paper, we introduce PILOT, a framework designed to overcome these challenges through two key innovations: (1) a novel dual-branch prompt learning mechanism that dynamically integrates a pool of learnable prompts with structured semantic attributes, enabling the model to adaptively weight the most relevant anomaly cues for each input image; and (2) a label-free test-time adaptation strategy that updates the learnable prompt parameters using high-confidence pseudo-labels from unlabeled test data. Extensive experiments on 13 industrial and medical benchmarks demonstrate that PILOT achieves state-of-the-art performance in both anomaly detection and localization under domain shift.
Authors Zihan Wang, Samira Ebrahimi Kahou, Narges Armanfard
Comment Accepted at BMVC 2025
Categories Computer Vision and Pattern Recognition
0
Orientation Matters: Making 3D Generative Models Orientation-Aligned NeurIPS 2025
Title CN 方向至关重要：使3D生成模型方向对齐
Keywords 3D生成 · 方向对齐 · 多视图扩散 · 3DVAE · 姿态一致性
Summary 本文提出了一种使3D生成模型从单张图像生成方向一致的3D对象的方法，并构建了包含14,832个方向对齐3D模型的数据集Objaverse-OA。通过在多视图扩散和3D变分自编码器框架上进行微调，实现了跨类别的姿态一致性生成，在零样本方向估计和基于箭头的旋转操作等下游任务中展示了应用潜力。
Reason 该论文研究的是3D生成模型的方向对齐问题，属于3D视觉与生成模型领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的Objaverse-OA数据集和对齐训练方法在3D生成中具有价值，但无法迁移解决文档理解中的OCR幻觉、密集文本处理、复杂排版或长文档推理等关键痛点。
Abstract Humans intuitively perceive object shape and orientation from a single image, guided by strong priors about canonical poses. However, existing 3D generative models often produce misaligned results due to inconsistent training data, limiting their usability in downstream tasks. To address this gap, we introduce the task of orientation-aligned 3D object generation: producing 3D objects from single images with consistent orientations across categories. To facilitate this, we construct Objaverse-OA, a dataset of 14,832 orientation-aligned 3D models spanning 1,008 categories. Leveraging Objaverse-OA, we fine-tune two representative 3D generative models based on multi-view diffusion and 3D variational autoencoder frameworks to produce aligned objects that generalize well to unseen objects across various categories. Experimental results demonstrate the superiority of our method over post-hoc alignment approaches. Furthermore, we showcase downstream applications enabled by our aligned object generation, including zero-shot object orientation estimation via analysis-by-synthesis and efficient arrow-based object rotation manipulation.
Authors Yichong Lu, Yuzhuo Tian, Zijin Jiang, Yikun Zhao, Yuanbo Yang, Hao Ouyang, Haoji Hu, Huimin Yu, Yujun Shen, Yiyi Liao
Comment Accepted by NeurIPS 2025. Project Page: https://xdimlab.github.io/Orientation_Matters
Categories Computer Vision and Pattern Recognition
0
UltraViCo: Breaking Extrapolation Limits in Video Diffusion Transformers
Title CN UltraViCo：突破视频扩散Transformer中的外推限制
Keywords 视频扩散模型 · 外推限制 · 注意力分散 · 位置编码 · 训练-free方法
Summary 本文针对视频扩散Transformer在超出训练长度时表现退化的问题，提出了一种名为UltraViCo的训练-free方法，通过引入常数衰减因子抑制训练窗口之外token的注意力，缓解注意力分散问题，从而同时解决周期性内容重复和质量下降两种失败模式。实验表明该方法能将外推极限从2倍提升至4倍，并在多个模型和任务上显著优于现有方法。
Reason 该论文研究的是视频扩散模型在时间长度上的外推问题，属于视频生成领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。尽管其提出的‘注意力分散’分析和训练-free的注意力抑制方法具有一定启发性，但其核心问题设定（长视频生成）、数据模态（动态视觉序列）和应用场景均远离文档理解的核心痛点（如OCR幻觉、文本密集、排版复杂、细粒度定位）。目前看不到可迁移至文档图像理解的有效路径，尤其是无法解决高分辨率静态文档处理、多模态对齐或逻辑推理等问题。
Abstract Despite advances, video diffusion transformers still struggle to generalize beyond their training length, a challenge we term video length extrapolation. We identify two failure modes: model-specific periodic content repetition and a universal quality degradation. Prior works attempt to solve repetition via positional encodings, overlooking quality degradation and achieving only limited extrapolation. In this paper, we revisit this challenge from a more fundamental view: attention maps, which directly govern how context influences outputs. We identify that both failure modes arise from a unified cause: attention dispersion, where tokens beyond the training window dilute learned attention patterns. This leads to quality degradation and repetition emerges as a special case when this dispersion becomes structured into periodic attention patterns, induced by harmonic properties of positional encodings. Building on this insight, we propose UltraViCo, a training-free, plug-and-play method that suppresses attention for tokens beyond the training window via a constant decay factor. By jointly addressing both failure modes, we outperform a broad set of baselines largely across models and extrapolation ratios, pushing the extrapolation limit from 2x to 4x. Remarkably, it improves Dynamic Degree and Imaging Quality by 233% and 40.5% over the previous best method at 4x extrapolation. Furthermore, our method generalizes seamlessly to downstream tasks such as controllable video synthesis and editing.
Authors Min Zhao, Hongzhou Zhu, Yingze Wang, Bokai Yan, Jintao Zhang, Guande He, Ling Yang, Chongxuan Li, Jun Zhu
Comment Project page: https://thu-ml.github.io/UltraViCo.github.io/
Categories Computer Vision and Pattern Recognition
0
Unleashing the Power of Chain-of-Prediction for Monocular 3D Object Detection
Title CN 释放链式预测在单目3D目标检测中的潜力
Keywords Monocular 3D Detection · Chain-of-Prediction · Uncertainty Modeling · Depth Estimation · Object Detection
Summary 本文提出MonoCoP框架，通过链式预测（CoP）和不确定性引导的选择器（GS）来建模3D目标属性间的相关性，动态切换并行与串行预测模式，提升单目3D检测中尤其是远距离物体的深度估计精度。在KITTI、nuScenes和Waymo数据集上达到SOTA。
Reason 该论文研究的是单目3D目标检测（Mono3D），属于自动驾驶和3D视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管提出了Chain-of-Prediction这一结构化推理思路，但其应用场景完全局限于3D属性预测的误差传播控制，无法迁移至文档理解中的OCR幻觉、复杂排版解析或长文档推理等核心痛点。
Abstract Monocular 3D detection (Mono3D) aims to infer 3D bounding boxes from a single RGB image. Without auxiliary sensors such as LiDAR, this task is inherently ill-posed since the 3D-to-2D projection introduces depth ambiguity. Previous works often predict 3D attributes (e.g., depth, size, and orientation) in parallel, overlooking that these attributes are inherently correlated through the 3D-to-2D projection. However, simply enforcing such correlations through sequential prediction can propagate errors across attributes, especially when objects are occluded or truncated, where inaccurate size or orientation predictions can further amplify depth errors. Therefore, neither parallel nor sequential prediction is optimal. In this paper, we propose MonoCoP, an adaptive framework that learns when and how to leverage inter-attribute correlations with two complementary designs. A Chain-of-Prediction (CoP) explores inter-attribute correlations through feature-level learning, propagation, and aggregation, while an Uncertainty-Guided Selector (GS) dynamically switches between CoP and parallel paradigms for each object based on the predicted uncertainty. By combining their strengths, MonoCoP achieves state-of-the-art (SOTA) performance on KITTI, nuScenes, and Waymo, significantly improving depth accuracy, particularly for distant objects.
Authors Zhihao Zhang, Abhinav Kumar, Girish Chandar Ganesan, Xiaoming Liu
Categories Computer Vision and Pattern Recognition
0
A2Seek: Towards Reasoning-Centric Benchmark for Aerial Anomaly Understanding
Title CN A2Seek：面向空中异常理解的推理中心化基准
Keywords 空中异常检测 · 推理基准 · 无人机视觉 · 图思维 · 因果解释
Summary 本文提出了A2Seek，一个面向空中异常理解的大规模推理型基准数据集，包含高分辨率航拍视频及详细的异常标注与自然语言因果解释。同时提出A2Seek-R1框架，结合图思维（GoT）微调、面向空中场景的规则奖励优化（A-GRPO）以及模拟无人机飞行的‘寻找’机制，提升模型对异常位置与成因的理解能力。实验显示其在检测与定位指标上显著优于现有方法。
Reason 该论文聚焦于无人机视角下的空中异常检测与因果推理，属于视频理解与遥感视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法虽涉及推理框架和注意力机制，但针对的是动态视频中的异常定位与因果解释，迁移到文档理解场景（如OCR去幻觉、复杂版面分析）极为困难，且任务本质不同。
Abstract While unmanned aerial vehicles (UAVs) offer wide-area, high-altitude coverage for anomaly detection, they face challenges such as dynamic viewpoints, scale variations, and complex scenes. Existing datasets and methods, mainly designed for fixed ground-level views, struggle to adapt to these conditions, leading to significant performance drops in drone-view scenarios. To bridge this gap, we introduce A2Seek (Aerial Anomaly Seek), a large-scale, reasoning-centric benchmark dataset for aerial anomaly understanding. This dataset covers various scenarios and environmental conditions, providing high-resolution real-world aerial videos with detailed annotations, including anomaly categories, frame-level timestamps, region-level bounding boxes, and natural language explanations for causal reasoning. Building on this dataset, we propose A2Seek-R1, a novel reasoning framework that generalizes R1-style strategies to aerial anomaly understanding, enabling a deeper understanding of "Where" anomalies occur and "Why" they happen in aerial frames. To this end, A2Seek-R1 first employs a graph-of-thought (GoT)-guided supervised fine-tuning approach to activate the model's latent reasoning capabilities on A2Seek. Then, we introduce Aerial Group Relative Policy Optimization (A-GRPO) to design rule-based reward functions tailored to aerial scenarios. Furthermore, we propose a novel "seeking" mechanism that simulates UAV flight behavior by directing the model's attention to informative regions. Extensive experiments demonstrate that A2Seek-R1 achieves up to a 22.04% improvement in AP for prediction accuracy and a 13.9% gain in mIoU for anomaly localization, exhibiting strong generalization across complex environments and out-of-distribution scenarios. Our dataset and code are released at https://2-mo.github.io/A2Seek/.
Authors Mengjingcheng Mo, Xinyang Tong, Mingpi Tan, Jiaxu Leng, Jiankang Zheng, Yiran Liu, Haosheng Chen, Ji Gan, Weisheng Li, Xinbo Gao
Categories Computer Vision and Pattern Recognition
0
Multi Head Attention Enhanced Inception v3 for Cardiomegaly Detection
Title CN 用于心脏肥大检测的多头注意力增强型Inception v3
Keywords 医学图像分析 · 心脏肥大检测 · 多头注意力 · Inception v3 · X光图像
Summary 本文提出了一种结合Inception v3和多头注意力机制的深度学习模型，用于从X光图像中自动检测心脏肥大。通过数据收集、预处理和CNN+注意力机制的融合模型进行训练，在准确率、精确率、召回率等指标上取得了良好表现。然而，该工作属于特定医学影像的分类任务，不具备文档理解的方法论创新。
Reason 该论文属于医学图像分析领域，研究任务为基于X光片的心脏肥大检测，属于纯医疗影像诊断应用，与文档图像理解（DIU）或DocVQA无关。尽管使用了多头注意力机制和Inception v3架构，但其方法未针对文档结构、文本识别、布局分析或OCR-free理解等DIU核心问题设计，也无法直接迁移用于解决OCR幻觉、复杂排版或长文档推理等痛点。
Abstract The healthcare industry has been revolutionized significantly by novel imaging technologies, not just in the diagnosis of cardiovascular diseases but also by the visualization of structural abnormalities like cardiomegaly. This article explains an integrated approach to the use of deep learning tools and attention mechanisms for automatic detection of cardiomegaly using X-ray images. The initiation of the project is grounded on a strong Data Collection phase and gathering the data of annotated X-ray images of various types. Then, while the Preprocessing module fine-tunes image quality, it is feasible to utilize the best out of the data quality in the proposed system. In our proposed system, the process is a CNN configuration leveraging the inception V3 model as one of the key blocks. Besides, we also employ a multilayer attention mechanism to enhance the strength. The most important feature of the method is the multi-head attention mechanism that can learn features automatically. By exact selective focusing on only some regions of input, the model can thus identify cardiomegaly in a sensitive manner. Attention rating is calculated, duplicated, and applied to enhance representation of main data, and therefore there is a successful diagnosis. The Evaluation stage will be extremely strict and it will thoroughly evaluate the model based on such measures as accuracy and precision. This will validate that the model can identify cardiomegaly and will also show the clinical significance of this method. The model has accuracy of 95.6, precision of 95.2, recall of 96.2, sensitivity of 95.7, specificity of 96.1 and an Area Under Curve(AUC) of 96.0 and their respective graphs are plotted for visualisation.
Authors Abishek Karthik, Pandiyaraju V
Categories Computer Vision and Pattern Recognition
0
Exploring State-of-the-art models for Early Detection of Forest Fires
Title CN 探索用于森林火灾早期检测的最先进模型
Keywords 森林火灾检测 · 合成数据集 · YOLOv7 · 检测Transformer · 视觉识别
Summary 本文提出了一种用于森林火灾早期检测的数据集，通过游戏模拟器（如《荒野大镖客2》）生成包含烟雾和初期火情的图像，并结合现有公开图像构建更全面的数据集。作者比较了YOLOv7和多种检测Transformer模型在该数据集上的图像分类与定位性能。由于缺乏专门针对此任务的大规模数据集，现有方法存在漏检问题，本工作旨在填补这一空白。
Reason 该论文属于纯计算机视觉领域的图像分类与目标检测任务，研究场景为森林火灾的视觉识别，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法基于YOLOv7和检测Transformer在合成游戏数据上进行训练，属于特定应用场景下的模型应用，不涉及文档结构、OCR、多模态对齐、视觉token压缩等DIU核心问题。无法迁移解决OCR幻觉、复杂排版或长文档推理等痛点。
Abstract There have been many recent developments in the use of Deep Learning Neural Networks for fire detection. In this paper, we explore an early warning system for detection of forest fires. Due to the lack of sizeable datasets and models tuned for this task, existing methods suffer from missed detection. In this work, we first propose a dataset for early identification of forest fires through visual analysis. Unlike existing image corpuses that contain images of wide-spread fire, our dataset consists of multiple instances of smoke plumes and fire that indicates the initiation of fire. We obtained this dataset synthetically by utilising game simulators such as Red Dead Redemption 2. We also combined our dataset with already published images to obtain a more comprehensive set. Finally, we compared image classification and localisation methods on the proposed dataset. More specifically we used YOLOv7 (You Only Look Once) and different models of detection transformer.
Authors Sharjeel Ahmed, Daim Armaghan, Fatima Naweed, Umair Yousaf, Ahmad Zubair, Murtaza Taj
Categories Computer Vision and Pattern Recognition
0
WPT: World-to-Policy Transfer via Online World Model Distillation
Title CN WPT：通过在线世界模型蒸馏实现从世界到策略的迁移
Keywords 世界模型 · 策略蒸馏 · 在线学习 · 自动驾驶 · 强化学习
Summary 本文提出WPT，一种通过端到端世界模型指导下的在线蒸馏方法，将世界知识注入教师策略，并通过策略蒸馏和世界奖励蒸馏传递给轻量级学生策略，在自动驾驶任务中实现了高性能与高推理速度的平衡。
Reason 该论文属于具身智能与自动驾驶领域，研究的是基于世界模型的策略迁移与蒸馏方法，用于提升驾驶任务中的规划性能和推理效率。其核心问题设定（如动作-环境动态建模、驾驶评分、碰撞率）与文档图像理解（DIU）无直接关联。尽管提到了‘world model’和‘distillation’等通用技术概念，但这些方法针对的是时序决策与控制任务，难以迁移到解决DIU中的OCR幻觉、复杂排版解析或长文档推理等核心痛点。
Abstract Recent years have witnessed remarkable progress in world models, which primarily aim to capture the spatio-temporal correlations between an agent's actions and the evolving environment. However, existing approaches often suffer from tight runtime coupling or depend on offline reward signals, resulting in substantial inference overhead or hindering end-to-end optimization. To overcome these limitations, we introduce WPT, a World-to-Policy Transfer training paradigm that enables online distillation under the guidance of an end-to-end world model. Specifically, we develop a trainable reward model that infuses world knowledge into a teacher policy by aligning candidate trajectories with the future dynamics predicted by the world model. Subsequently, we propose policy distillation and world reward distillation to transfer the teacher's reasoning ability into a lightweight student policy, enhancing planning performance while preserving real-time deployability. Extensive experiments on both open-loop and closed-loop benchmarks show that our WPT achieves state-of-the-art performance with a simple policy architecture: it attains a 0.11 collision rate (open-loop) and achieves a 79.23 driving score (closed-loop) surpassing both world-model-based and imitation-learning methods in accuracy and safety. Moreover, the student sustains up to 4.9x faster inference, while retaining most of the gains.
Authors Guangfeng Jiang, Yueru Luo, Jun Liu, Yi Huang, Yiyao Zhu, Zhan Qu, Dave Zhenyu Chen, Bingbing Liu, Xu Yan
Categories Computer Vision and Pattern Recognition
0
PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation
Title CN PrismAudio：用于视频到音频生成的分解式思维链与多维奖励
Keywords 视频到音频生成 · 思维链 · 强化学习 · 多模态生成 · 感知对齐
Summary 本文提出PrismAudio，首次将强化学习引入视频到音频生成任务，通过四个专门的思维链模块（语义、时序、美学、空间）分别对应四个感知维度，并设计了相应的奖励函数以解决目标纠缠问题。同时提出Fast-GRPO训练方法降低计算开销，并构建新基准AudioCanvas进行评估。实验表明在多个指标上达到SOTA。
Reason 该论文研究的是视频到音频生成任务，属于视频理解与生成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等核心任务完全无关。尽管其提出了分解式思维链（CoT）和多维奖励机制，但应用场景为跨模态生成中的感知对齐问题，迁移到文档理解中的OCR幻觉、复杂排版解析或长文档推理等痛点极为困难。且未涉及任何文档结构、文本识别或多模态文档建模内容。
Abstract Video-to-Audio (V2A) generation requires balancing four critical perceptual dimensions: semantic consistency, audio-visual temporal synchrony, aesthetic quality, and spatial accuracy; yet existing methods suffer from objective entanglement that conflates competing goals in single loss functions and lack human preference alignment. We introduce PrismAudio, the first framework to integrate Reinforcement Learning into V2A generation with specialized Chain-of-Thought (CoT) planning. Our approach decomposes monolithic reasoning into four specialized CoT modules (Semantic, Temporal, Aesthetic, and Spatial CoT), each paired with targeted reward functions. This CoT-reward correspondence enables multidimensional RL optimization that guides the model to jointly generate better reasoning across all perspectives, solving the objective entanglement problem while preserving interpretability. To make this optimization computationally practical, we propose Fast-GRPO, which employs hybrid ODE-SDE sampling that dramatically reduces the training overhead compared to existing GRPO implementations. We also introduce AudioCanvas, a rigorous benchmark that is more distributionally balanced and covers more realistically diverse and challenging scenarios than existing datasets, with 300 single-event classes and 501 multi-event samples. Experimental results demonstrate that PrismAudio achieves state-of-the-art performance across all four perceptual dimensions on both the in-domain VGGSound test set and out-of-domain AudioCanvas benchmark. The project page is available at https://PrismAudio-Project.github.io.
Authors Huadai Liu, Kaicheng Luo, Wen Wang, Qian Chen, Peiwen Sun, Rongjie Huang, Xiangang Li, Jieping Ye, Wei Xue
Comment Preprint
Categories Computer Vision and Pattern Recognition
0
Blind Adaptive Local Denoising for CEST Imaging
Title CN 用于CEST成像的盲自适应局部去噪
Keywords CEST MRI · 去噪 · 自适应变换 · 局部SVD · 医学影像
Summary 本文提出了一种名为BALD的盲自适应局部去噪方法，用于改善化学交换饱和转移（CEST）MRI图像质量。该方法利用CEST数据的自相似性，通过自适应方差稳定变换来均衡噪声分布，并采用两阶段去噪和局部SVD分解以避免空间与光谱伪影。实验表明，BALD在多个体模和体内数据上优于现有去噪方法，提升了分子浓度估计和癌症检测性能。
Reason 该论文研究的是化学交换饱和转移（CEST）MRI图像中的去噪问题，属于医学影像处理领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法针对的是磁共振信号中的异方差噪声和分子级可视化任务，无法迁移到解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点。
Abstract Chemical Exchange Saturation Transfer (CEST) MRI enables molecular-level visualization of low-concentration metabolites by leveraging proton exchange dynamics. However, its clinical translation is hindered by inherent challenges: spatially varying noise arising from hardware limitations, and complex imaging protocols introduce heteroscedasticity in CEST data, perturbing the accuracy of quantitative contrast mapping such as amide proton transfer (APT) imaging. Traditional denoising methods are not designed for this complex noise and often alter the underlying information that is critical for biomedical analysis. To overcome these limitations, we propose a new Blind Adaptive Local Denoising (BALD) method. BALD exploits the self-similar nature of CEST data to derive an adaptive variance-stabilizing transform that equalizes the noise distributions across CEST pixels without prior knowledge of noise characteristics. Then, BALD performs two-stage denoising on a linear transformation of data to disentangle molecular signals from noise. A local SVD decomposition is used as a linear transform to prevent spatial and spectral denoising artifacts. We conducted extensive validation experiments on multiple phantoms and \textit{in vivo} CEST scans. In these experiments, BALD consistently outperformed state-of-the-art CEST denoisers in both denoising metrics and downstream tasks such as molecular concentration maps estimation and cancer detection.
Authors Chu Chen, Aitor Artola, Yang Liu, Se Weon Park, Raymond H. Chan, Jean-Michel Morel, Kannie W. Y. Chan
Categories Computer Vision and Pattern Recognition
0
From Spots to Pixels: Dense Spatial Gene Expression Prediction from Histology Images
Title CN 从斑点到像素：基于组织学图像的密集空间基因表达预测
Keywords 空间转录组学 · 组织学图像 · 密集预测 · 基因表达 · 计算机视觉
Summary 本文提出PixNet，一种用于从组织学全切片图像中进行密集空间基因表达预测的网络。不同于传统基于固定斑点的方法，PixNet生成连续的空间基因表达图谱，支持多尺度斑点聚合，在四个空间转录组数据集上超越了现有方法。
Reason 该论文属于生物医学图像分析领域，研究的是从组织病理学图像预测基因表达分布，尽管使用了密集空间建模和高分辨率图像处理技术，但其任务本质与文档图像理解（DIU）无关。未涉及OCR、布局分析、表格识别或文档视觉问答等核心DIU任务，也无法迁移其方法解决OCR幻觉、复杂排版或长文档推理等关键痛点。
Abstract Spatial transcriptomics (ST) measures gene expression at fine-grained spatial resolution, offering insights into tissue molecular landscapes. Previous methods for spatial gene expression prediction typically crop spots of interest from histopathology slide images, and train models to map each spot to a corresponding gene expression profile. However, these methods inherently lose the spatial resolution in gene expression: 1) each spot often contains multiple cells with distinct gene expression profiles; 2) spots are typically defined at fixed spatial resolutions, limiting the ability to predict gene expression at varying scales. To address these limitations, this paper presents PixNet, a dense prediction network capable of predicting spatially resolved gene expression across spots of varying sizes and scales directly from histopathology slide images. Different from previous methods that map individual spots to gene expression values, we generate a spatially dense continuous gene expression map from the histopathology slide image, and aggregate values within spots of interest to predict the gene expression. Our PixNet outperforms state-of-the-art methods on four common ST datasets in multiple spatial scales. The source code will be publicly available.
Authors Ruikun Zhang, Yan Yang, Liyuan Pan
Categories Computer Vision and Pattern Recognition
0
MAPo : Motion-Aware Partitioning of Deformable 3D Gaussian Splatting for High-Fidelity Dynamic Scene Reconstruction
Title CN MAPo：用于高保真动态场景重建的可变形3D高斯点阵的运动感知分割
Keywords 3D高斯点阵 · 动态场景重建 · 运动感知分割 · 变形场 · 跨帧一致性
Summary 本文提出了一种名为MAPo的运动感知分割框架，用于提升可变形3D高斯点阵在高保真动态场景重建中的表现。通过动态评分机制将3D高斯分为高动态与低动态两类，并对高动态部分进行时间上的递归分割和变形网络复制，以精细建模复杂运动。同时引入跨帧一致性损失来缓解分割带来的视觉不连续问题。实验表明该方法在保持计算成本的同时显著提升了渲染质量。
Reason 该论文研究的是3D高斯点阵在动态场景重建中的应用，属于3D视觉与动态场景建模领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务完全无关。其核心问题为高动态区域的渲染模糊与运动细节丢失，提出的方法如运动感知分割和跨帧一致性损失，均针对3D场景重建中的视觉连续性优化，无法迁移至解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点。
Abstract 3D Gaussian Splatting, known for enabling high-quality static scene reconstruction with fast rendering, is increasingly being applied to multi-view dynamic scene reconstruction. A common strategy involves learning a deformation field to model the temporal changes of a canonical set of 3D Gaussians. However, these deformation-based methods often produce blurred renderings and lose fine motion details in highly dynamic regions due to the inherent limitations of a single, unified model in representing diverse motion patterns. To address these challenges, we introduce Motion-Aware Partitioning of Deformable 3D Gaussian Splatting (MAPo), a novel framework for high-fidelity dynamic scene reconstruction. Its core is a dynamic score-based partitioning strategy that distinguishes between high- and low-dynamic 3D Gaussians. For high-dynamic 3D Gaussians, we recursively partition them temporally and duplicate their deformation networks for each new temporal segment, enabling specialized modeling to capture intricate motion details. Concurrently, low-dynamic 3DGs are treated as static to reduce computational costs. However, this temporal partitioning strategy for high-dynamic 3DGs can introduce visual discontinuities across frames at the partition boundaries. To address this, we introduce a cross-frame consistency loss, which not only ensures visual continuity but also further enhances rendering quality. Extensive experiments demonstrate that MAPo achieves superior rendering quality compared to baselines while maintaining comparable computational costs, particularly in regions with complex or rapid motions.
Authors Han Jiao, Jiakai Sun, Yexing Xu, Lei Zhao, Wei Xing, Huaizhong Lin
Categories Computer Vision and Pattern Recognition
0
Embodied Crowd Counting
Title CN 具身人群计数
Keywords 具身智能 · 人群计数 · 主动感知 · MLLM · 遮挡处理
Summary 本文提出了一个名为Embodied Crowd Counting（ECC）的新任务，旨在通过具身智能体在模拟环境中主动移动相机来更准确地估计被遮挡的人群数量。为此，作者构建了ECCD数据集，并提出一种基于多模态大语言模型（MLLM）的粗到细导航机制，结合法线分析实现精细计数。实验表明该方法在计数准确性与导航成本之间取得了最佳平衡。
Reason 该论文研究的是具身智能下的群体计数问题，属于具身导航与视频理解的交叉领域，其核心任务是通过主动探索提升人群计数精度。尽管使用了MLLM进行导航决策，但其目标场景、任务定义和方法设计均与文档图像理解（DIU）无关。文档理解不涉及物理空间中的主动感知或Z轴探索，因此该工作无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract Occlusion is one of the fundamental challenges in crowd counting. In the community, various data-driven approaches have been developed to address this issue, yet their effectiveness is limited. This is mainly because most existing crowd counting datasets on which the methods are trained are based on passive cameras, restricting their ability to fully sense the environment. Recently, embodied navigation methods have shown significant potential in precise object detection in interactive scenes. These methods incorporate active camera settings, holding promise in addressing the fundamental issues in crowd counting. However, most existing methods are designed for indoor navigation, showing unknown performance in analyzing complex object distribution in large scale scenes, such as crowds. Besides, most existing embodied navigation datasets are indoor scenes with limited scale and object quantity, preventing them from being introduced into dense crowd analysis. Based on this, a novel task, Embodied Crowd Counting (ECC), is proposed. We first build up an interactive simulator, Embodied Crowd Counting Dataset (ECCD), which enables large scale scenes and large object quantity. A prior probability distribution that approximates realistic crowd distribution is introduced to generate crowds. Then, a zero-shot navigation method (ZECC) is proposed. This method contains a MLLM driven coarse-to-fine navigation mechanism, enabling active Z-axis exploration, and a normal-line-based crowd distribution analysis method for fine counting. Experimental results against baselines show that the proposed method achieves the best trade-off between counting accuracy and navigation cost.
Authors Runling Long, Yunlong Wang, Jia Wan, Xiang Deng, Xinting Zhu, Weili Guan, Antoni B. Chan, Liqiang Nie
Categories Computer Vision and Pattern Recognition
0
PRADA: Probability-Ratio-Based Attribution and Detection of Autoregressive-Generated Images
Title CN PRADA：基于概率比的自回归生成图像归因与检测
Keywords 图像生成检测 · 自回归模型 · AI伪造检测 · 概率比分析 · 模型归因
Summary 本文提出PRADA方法，通过分析自回归图像生成模型中条件与无条件概率的比率来检测图像是否由特定AI模型生成，并实现生成源归因。实验表明该方法对多种类到图像和文本到图像模型具有高检测准确率。
Reason 该论文研究的是自回归生成图像的检测问题，属于图像伪造与鉴伪领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心任务是识别AI生成图像的来源模型并判断真伪，不涉及OCR、布局分析、表格识别、文本结构化或VLM在文档上的应用。虽然技术上涉及概率建模和多模态生成机制分析，但无法迁移用于解决文档理解中的OCR幻觉、复杂排版、长文档推理等核心痛点。
Abstract Autoregressive (AR) image generation has recently emerged as a powerful paradigm for image synthesis. Leveraging the generation principle of large language models, they allow for efficiently generating deceptively real-looking images, further increasing the need for reliable detection methods. However, to date there is a lack of work specifically targeting the detection of images generated by AR image generators. In this work, we present PRADA (Probability-Ratio-Based Attribution and Detection of Autoregressive-Generated Images), a simple and interpretable approach that can reliably detect AR-generated images and attribute them to their respective source model. The key idea is to inspect the ratio of a model's conditional and unconditional probability for the autoregressive token sequence representing a given image. Whenever an image is generated by a particular model, its probability ratio shows unique characteristics which are not present for images generated by other models or real images. We exploit these characteristics for threshold-based attribution and detection by calibrating a simple, model-specific score function. Our experimental evaluation shows that PRADA is highly effective against eight class-to-image and four text-to-image models.
Authors Simon Damm, Jonas Ricker, Henning Petzka, Asja Fischer
Categories Computer Vision and Pattern Recognition
0
FLaTEC: Frequency-Disentangled Latent Triplanes for Efficient Compression of LiDAR Point Clouds
Title CN FLaTEC：用于高效压缩LiDAR点云的频率解耦潜在三平面
Keywords 点云压缩 · 频率解耦 · 潜在三平面 · LiDAR · 3D表示
Summary 本文提出FLaTEC，一种面向LiDAR点云压缩的频率感知模型，通过将体素化嵌入转换为潜在三平面表示，并解耦低频结构与高频纹理以实现高效压缩。该方法在SemanticKITTI和Ford数据集上实现了最先进的率失真性能。
Reason 该论文研究的是LiDAR点云压缩，属于3D视觉与自动驾驶相关领域，与文档图像理解（DIU）或DocVQA无关。其核心问题是点云的频率分量解耦与压缩，虽然涉及视觉表示学习，但目标场景、数据模态和任务均与文档理解无直接关联，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract Point cloud compression methods jointly optimize bitrates and reconstruction distortion. However, balancing compression ratio and reconstruction quality is difficult because low-frequency and high-frequency components contribute differently at the same resolution. To address this, we propose FLaTEC, a frequency-aware compression model that enables the compression of a full scan with high compression ratios. Our approach introduces a frequency-aware mechanism that decouples low-frequency structures and high-frequency textures, while hybridizing latent triplanes as a compact proxy for point cloud. Specifically, we convert voxelized embeddings into triplane representations to reduce sparsity, computational cost, and storage requirements. We then devise a frequency-disentangling technique that extracts compact low-frequency content while collecting high-frequency details across scales. The decoupled low-frequency and high-frequency components are stored in binary format. During decoding, full-spectrum signals are progressively recovered via a modulation block. Additionally, to compensate for the loss of 3D correlation, we introduce an efficient frequency-based attention mechanism that fosters local connectivity and outputs arbitrary resolution points. Our method achieves state-of-the-art rate-distortion performance and outperforms the standard codecs by 78\% and 94\% in BD-rate on both SemanticKITTI and Ford datasets.
Authors Xiaoge Zhang, Zijie Wu, Mingtao Feng, Zichen Geng, Mehwish Nasim, Saeed Anwar, Ajmal Mian
Categories Computer Vision and Pattern Recognition
0
DeLightMono: Enhancing Self-Supervised Monocular Depth Estimation in Endoscopy by Decoupling Uneven Illumination
Title CN DeLightMono：通过解耦不均匀光照增强内窥镜中自监督单目深度估计
Keywords 单目深度估计 · 自监督学习 · 光照解耦 · 内窥镜图像 · 计算机视觉
Summary 本文提出DeLightMono，一种用于内窥镜图像的自监督单目深度估计框架，通过设计光照-反射-深度模型来解耦不均匀光照的影响，并引入辅助网络和新的损失函数进行联合优化。实验表明该方法在两个公开数据集上优于现有方法。
Reason 该论文研究的是内窥镜图像中的单目深度估计问题，属于医疗影像与计算机视觉交叉领域，核心任务为深度估计与光照解耦，与文档图像理解（DIU）、DocVQA、OCR、版面分析等任务完全无关。其方法虽在特定场景下有效，但无法迁移到文档图像的文本识别、结构理解或视觉问答等核心痛点上。
Abstract Self-supervised monocular depth estimation serves as a key task in the development of endoscopic navigation systems. However, performance degradation persists due to uneven illumination inherent in endoscopic images, particularly in low-intensity regions. Existing low-light enhancement techniques fail to effectively guide the depth network. Furthermore, solutions from other fields, like autonomous driving, require well-lit images, making them unsuitable and increasing data collection burdens. To this end, we present DeLight-Mono - a novel self-supervised monocular depth estimation framework with illumination decoupling. Specifically, endoscopic images are represented by a designed illumination-reflectance-depth model, and are decomposed with auxiliary networks. Moreover, a self-supervised joint-optimizing framework with novel losses leveraging the decoupled components is proposed to mitigate the effects of uneven illumination on depth estimation. The effectiveness of the proposed methods was rigorously verified through extensive comparisons and an ablation study performed on two public datasets.
Authors Mingyang Ou, Haojin Li, Yifeng Zhang, Ke Niu, Zhongxi Qiu, Heng Li, Jiang Liu
Categories Computer Vision and Pattern Recognition
0
Temporally Compressed 3D Gaussian Splatting for Dynamic Scenes British Machine Vision Conference (BMVC) 2025
Title CN 用于动态场景的时序压缩3D高斯点阵
Keywords 3D高斯点阵 · 动态场景重建 · 时序压缩 · 内存优化 · BMVC2025
Summary 本文提出了一种名为TC3DGS的方法，通过时间相关性剪枝、梯度感知混合精度量化以及时空轨迹简化算法，对动态3D高斯表示进行高效压缩，在保持视觉质量的同时实现高达67倍的压缩率。适用于需要低内存开销的动态场景渲染应用。
Reason 该论文研究的是动态3D场景重建中的高斯点阵压缩，属于3D视觉与图形学领域，主要应用于AR/VR和游戏渲染。其技术焦点（如高斯轨迹插值、动态量化）针对的是三维动态几何表示的压缩，与文档图像理解（DIU）中的视觉文本解析、OCR去幻觉、复杂版面分析等核心问题无直接关联。尽管‘压缩’概念在广义上有启发性，但方法本身无法迁移至文档模态的处理，且不属于DIU本题或可转化的上游工具。
Abstract Recent advancements in high-fidelity dynamic scene reconstruction have leveraged dynamic 3D Gaussians and 4D Gaussian Splatting for realistic scene representation. However, to make these methods viable for real-time applications such as AR/VR, gaming, and rendering on low-power devices, substantial reductions in memory usage and improvements in rendering efficiency are required. While many state-of-the-art methods prioritize lightweight implementations, they struggle in handling {scenes with complex motions or long sequences}. In this work, we introduce Temporally Compressed 3D Gaussian Splatting (TC3DGS), a novel technique designed specifically to effectively compress dynamic 3D Gaussian representations. TC3DGS selectively prunes Gaussians based on their temporal relevance and employs gradient-aware mixed-precision quantization to dynamically compress Gaussian parameters. In addition, TC3DGS exploits an adapted version of the Ramer-Douglas-Peucker algorithm to further reduce storage by interpolating Gaussian trajectories across frames. Our experiments on multiple datasets demonstrate that TC3DGS achieves up to 67$\times$ compression with minimal or no degradation in visual quality. More results and videos are provided in the supplementary. Project Page: https://ahmad-jarrar.github.io/tc-3dgs/
Authors Saqib Javed, Ahmad Jarrar Khan, Corentin Dumery, Chen Zhao, Mathieu Salzmann
Comment Accepted at British Machine Vision Conference (BMVC) 2025
Categories Computer Vision and Pattern Recognition
0
History-Augmented Contrastive Meta-Learning for Unsupervised Blind Super-Resolution of Planetary Remote Sensing Images
Title CN 用于行星遥感图像无监督盲超分辨率的历史增强对比元学习
Keywords 盲超分辨率 · 对比学习 · 无监督学习 · 遥感图像 · 图像恢复
Summary 本文提出了一种名为HACBSR的无监督盲超分辨率框架，用于提升行星遥感图像的质量。该方法通过引入历史模型生成负样本，结合对比核采样机制，在无真值图像的情况下实现优化。作者还发布了包含模拟退化模式的Ceres-50数据集。然而，该工作聚焦于遥感图像复原，与文档图像理解任务无关。
Reason 该论文研究的是行星遥感图像的盲超分辨率问题，属于特定领域的图像恢复任务，与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版、文本识别或DocVQA无关。尽管其方法涉及无监督学习和对比学习，但其技术路线针对的是图像退化建模与超分重建，并不适用于文档场景中的多模态对齐、视觉语言推理或细粒度文本定位。且未提及任何与文档、文本、布局或VLM相关的内容，属于无关领域。
Abstract Planetary remote sensing images are affected by diverse and unknown degradations caused by imaging environments and hardware constraints. These factors limit image quality and hinder supervised blind super-resolution due to the lack of ground-truth images. This work presents History-Augmented Contrastive Blind Super-Resolution (HACBSR), an unsupervised framework for blind super-resolution that operates without ground-truth images and external kernel priors. HACBSR comprises two components: (1) a contrastive kernel sampling mechanism with kernel similarity control to mitigate distribution bias from Gaussian sampling, and (2) a history-augmented contrastive learning that uses historical models to generate negative samples to enable less greedy optimization and to induce strong convexity without ground-truth. A convergence analysis of the history-augmented contrastive learning is given in the Appendix. To support evaluation in planetary applications, we introduce Ceres-50, a dataset with diverse geological features simulated degradation patterns. Experiments show that HACBSR achieves competitive performance compared with state-of-the-art unsupervised methods across multiple upscaling factors. The code is available at https://github.com/2333repeat/HACBSR, and the dataset is available at https://github.com/2333repeat/Ceres-50.
Authors Huijia Zhao, Jie Lu, Yunqing Jiang, Xiao-Ping Lu, Kaichang Di
Comment 13pages
Categories Computer Vision and Pattern Recognition
0
Unified Text-Image-to-Video Generation: A Training-Free Approach to Flexible Visual Conditioning
Title CN 统一的文本-图像到视频生成：一种无需训练的灵活视觉条件方法
Keywords 视频生成 · 文本到视频 · 无需训练 · 视觉条件 · 潜在空间反演
Summary 本文提出FlexTI2V，一种无需训练的方法，用于在文本到视频生成过程中灵活引入一个或多个图像作为视觉条件。通过将图像反演至潜在空间的噪声表示，并在去噪过程中采用随机局部块交换策略融合视觉特征，同时使用动态控制机制平衡生成结果的保真度与创造性。实验表明该方法在多种架构上均优于现有训练-free方法。
Reason 该论文研究的是文本-图像到视频生成（TI2V），属于视频生成领域，核心目标是实现对视频生成过程的视觉条件控制。其方法聚焦于在无需微调的前提下，将图像条件注入T2V模型中生成视频。这与文档图像理解（DIU）的核心任务如DocVQA、布局分析、OCR-free理解等无直接关联。尽管提到了‘视觉条件’和‘局部图像块交换’，但其应用场景为创意视频生成，无法迁移解决文档中的OCR幻觉、细粒度定位或复杂排版理解等关键痛点。
Abstract Text-image-to-video (TI2V) generation is a critical problem for controllable video generation using both semantic and visual conditions. Most existing methods typically add visual conditions to text-to-video (T2V) foundation models by finetuning, which is costly in resources and only limited to a few pre-defined conditioning settings. To tackle these constraints, we introduce a unified formulation for TI2V generation with flexible visual conditioning. Furthermore, we propose an innovative training-free approach, dubbed FlexTI2V, that can condition T2V foundation models on an arbitrary amount of images at arbitrary positions. Specifically, we firstly invert the condition images to noisy representation in a latent space. Then, in the denoising process of T2V models, our method uses a novel random patch swapping strategy to incorporate visual features into video representations through local image patches. To balance creativity and fidelity, we use a dynamic control mechanism to adjust the strength of visual conditioning to each video frame. Extensive experiments validate that our method surpasses previous training-free image conditioning methods by a notable margin. Our method can also generalize to both UNet-based and transformer-based architectures.
Authors Bolin Lai, Sangmin Lee, Xu Cao, Xiang Li, James M. Rehg
Comment 18 pages, 10 figures, 8 tables
Categories Computer Vision and Pattern Recognition
0
Clair Obscur: an Illumination-Aware Method for Real-World Image Vectorization
Title CN 明暗对比：一种面向真实世界图像矢量化的光照感知方法
Keywords 图像矢量化 · 本征图像分解 · 可微渲染 · 光照感知 · 视觉保真
Summary 本文提出COVec，一种受明暗对比（Clair-Obscur）启发的光照感知图像矢量化框架，首次在矢量域引入本征图像分解，将图像分离为反照率、阴影和光照层，并通过语义引导初始化和两阶段优化结合可微渲染进行精细化处理。实验表明该方法在视觉保真度和可编辑性上优于现有方法。
Reason 该论文研究的是通用图像的矢量化问题，核心是通过光照-阴影分解（albedo, shade, light）来提升视觉保真度和可编辑性。其技术路线聚焦于艺术与设计领域的图像处理，未涉及文档结构、文本内容、布局分析或OCR相关挑战。与文档图像理解（DIU）的核心任务如DocVQA、表格识别、文本定位等无直接关联，且无法迁移用于解决OCR幻觉、密集文本解析或复杂排版建模等问题。
Abstract Image vectorization aims to convert raster images into editable, scalable vector representations while preserving visual fidelity. Existing vectorization methods struggle to represent complex real-world images, often producing fragmented shapes at the cost of semantic conciseness. In this paper, we propose COVec, an illumination-aware vectorization framework inspired by the Clair-Obscur principle of light-shade contrast. COVec is the first to introduce intrinsic image decomposition in the vector domain, separating an image into albedo, shade, and light layers in a unified vector representation. A semantic-guided initialization and two-stage optimization refine these layers with differentiable rendering. Experiments on various datasets demonstrate that COVec achieves higher visual fidelity and significantly improved editability compared to existing methods.
Authors Xingyue Lin, Shuai Peng, Xiangyu Xie, Jianhua Zhu, Yuxuan Zhou, Liangcai Gao
Categories Computer Vision and Pattern Recognition
0
ACIT: Attention-Guided Cross-Modal Interaction Transformer for Pedestrian Crossing Intention Prediction
Title CN ACIT：用于行人过街意图预测的注意力引导跨模态交互Transformer
Keywords 行人意图预测 · 跨模态融合 · Transformer · 注意力机制 · 自动驾驶
Summary 本文提出了一种注意力引导的跨模态交互Transformer（ACIT），用于预测行人过街意图。该方法融合了六种视觉与运动模态，并通过三组交互对（语义图与光流、局部图像与光流、自车速度与行人框）设计双路径注意力机制以增强关键区域并促进模态间交互。同时引入Transformer进行时序建模，在JAAD数据集上取得了优于现有方法的结果。
Reason 该论文研究的是自动驾驶场景下的行人意图预测，属于具身智能与交通感知领域，与文档图像理解（DIU）或视觉语言模型在文档任务中的应用无直接关联。其提出的跨模态注意力机制虽然具有一定的方法价值，但所处理的模态（如光流、车速、RGB图像）和任务目标（行为预测）难以迁移到文档理解的核心痛点（如OCR幻觉、复杂版面分析、细粒度文本定位等）。
Abstract Predicting pedestrian crossing intention is crucial for autonomous vehicles to prevent pedestrian-related collisions. However, effectively extracting and integrating complementary cues from different types of data remains one of the major challenges. This paper proposes an attention-guided cross-modal interaction Transformer (ACIT) for pedestrian crossing intention prediction. ACIT leverages six visual and motion modalities, which are grouped into three interaction pairs: (1) Global semantic map and global optical flow, (2) Local RGB image and local optical flow, and (3) Ego-vehicle speed and pedestrian's bounding box. Within each visual interaction pair, a dual-path attention mechanism enhances salient regions within the primary modality through intra-modal self-attention and facilitates deep interactions with the auxiliary modality (i.e., optical flow) via optical flow-guided attention. Within the motion interaction pair, cross-modal attention is employed to model the cross-modal dynamics, enabling the effective extraction of complementary motion features. Beyond pairwise interactions, a multi-modal feature fusion module further facilitates cross-modal interactions at each time step. Furthermore, a Transformer-based temporal feature aggregation module is introduced to capture sequential dependencies. Experimental results demonstrate that ACIT outperforms state-of-the-art methods, achieving accuracy rates of 70% and 89% on the JAADbeh and JAADall datasets, respectively. Extensive ablation studies are further conducted to investigate the contribution of different modules of ACIT.
Authors Yuanzhe Li, Steffen Müller
Categories Computer Vision and Pattern Recognition
0
SplatCo: Structure-View Collaborative Gaussian Splatting for Detail-Preserving Rendering of Large-Scale Unbounded Scenes
Title CN SplatCo：结构-视图协同的高斯点阵化用于大规模无界场景的细节保持渲染
Keywords 高斯点阵化 · 3D重建 · 多视角一致性 · 场景渲染 · 结构-视图协同
Summary 本文提出SplatCo，一种结构-视图协同的高斯点阵化框架，用于高质量渲染复杂的大规模户外场景。通过引入跨结构协作模块融合全局三平面表示与局部上下文网格特征，并采用跨视图辅助训练策略增强多视角一致性，实现对精细几何结构和复杂纹理的有效重建。在多个大规模数据集上的实验表明其优于现有方法。
Reason 该论文属于3D视觉与场景重建领域，核心方法聚焦于高斯点阵化、多视角一致性优化和大规模户外场景的几何重建。其技术目标与文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等完全无关。尽管涉及‘高分辨率处理’，但其处理的是三维空间中的视觉渲染问题，无法迁移至文档中的OCR幻觉、细粒度文本定位或复杂排版理解等核心痛点。
Abstract We present SplatCo, a structure-view collaborative Gaussian splatting framework for high-fidelity rendering of complex outdoor environments. SplatCo builds upon two novel components: (1) a cross-structure collaboration module that combines global tri-plane representations, which capture coarse scene layouts, with local context grid features that represent fine surface details. This fusion is achieved through a novel hierarchical compensation strategy, ensuring both global consistency and local detail preservation; and (2) a cross-view assisted training strategy that enhances multi-view consistency by synchronizing gradient updates across viewpoints, applying visibility-aware densification, and pruning overfitted or inaccurate Gaussians based on structural consistency. Through joint optimization of structural representation and multi-view coherence, SplatCo effectively reconstructs fine-grained geometric structures and complex textures in large-scale scenes. Comprehensive evaluations on 13 diverse large-scale scenes, including Mill19, MatrixCity, Tanks & Temples, WHU, and custom aerial captures, demonstrate that SplatCo consistently achieves higher reconstruction quality than state-of-the-art methods, with PSNR improvements of 1-2 dB and SSIM gains of 0.1 to 0.2. These results establish a new benchmark for high-fidelity rendering of large-scale unbounded scenes. Code and additional information are available at https://github.com/SCUT-BIP-Lab/SplatCo.
Authors Haihong Xiao, Jianan Zou, Yuxin Zhou, Ying He, Wenxiong Kang
Categories Computer Vision and Pattern Recognition
0
LayerComposer: Multi-Human Personalized Generation via Layered Canvas
Title CN LayerComposer：通过分层画布实现多人个性化生成
Keywords 图像生成 · 个性化生成 · 分层表示 · 多主体合成 · 视觉编辑
Summary 本文提出LayerComposer，一种基于分层画布的多人个性化图像生成框架，支持用户通过图层方式对多个主体进行位置、大小的精细控制，并引入透明潜在剪枝机制和逐层交叉参考训练策略来提升可扩展性与生成质量。实验表明其在多主体生成任务中具有优越的身份保持性和构图一致性。
Reason 该论文聚焦于多人个性化图像生成，属于纯图像生成领域，核心贡献在于提升生成图像中多主体的空间控制与视觉保真度。其技术路线围绕创意设计工具中的分层编辑展开，与文档图像理解（DIU）中的OCR幻觉、密集文本识别、复杂版式分析或长文档推理等核心痛点无直接关联。尽管‘分层表示’在理论上可能启发某些视觉结构建模，但该方法针对的是自由形态的图像创作，无法迁移至文档语义结构解析或视觉-语言推理任务。
Abstract Despite their impressive visual fidelity, existing personalized image generators lack interactive control over spatial composition and scale poorly to multiple humans. To address these limitations, we present LayerComposer, an interactive and scalable framework for multi-human personalized generation. Inspired by professional image-editing software, LayerComposer provides intuitive reference-based human injection, allowing users to place and resize multiple subjects directly on a layered digital canvas to guide personalized generation. The core of our approach is the layered canvas, a novel representation where each subject is placed on a distinct layer, enabling interactive and occlusion-free composition. We further introduce a transparent latent pruning mechanism that improves scalability by decoupling computational cost from the number of subjects, and a layerwise cross-reference training strategy that mitigates copy-paste artifacts. Extensive experiments demonstrate that LayerComposer achieves superior spatial control, coherent composition, and identity preservation compared to state-of-the-art methods in multi-human personalized image generation.
Authors Guocheng Gordon Qian, Ruihang Zhang, Tsai-Shien Chen, Yusuf Dalva, Anujraaj Argo Goyal, Willi Menapace, Ivan Skorokhodov, Meng Dong, Arpit Sahni, Daniil Ostashev, Ju Hu, Sergey Tulyakov, Kuan-Chieh Jackson Wang
Comment 17 pages including appendix, preprint. Project page: https://snap-research.github.io/layercomposer/
Categories Computer Vision and Pattern Recognition
0
Redefining Radar Segmentation: Simultaneous Static-Moving Segmentation and Ego-Motion Estimation using Radar Point Clouds IEEE Transactions on Radar Systems
Title CN 重新定义雷达分割：使用雷达点云同时进行静态-动态分割与自我运动估计
Keywords 雷达分割 · 点云处理 · 自我运动估计 · 静态动态分割 · RNN
Summary 本文提出了一种基于神经网络的方法，能够从雷达点云中同时分割静态和动态物体，并估计载体的自我运动速度。方法直接处理原始点云，无需复杂的预处理步骤，并在真实世界数据集RadarScenes上验证了有效性。尽管创新性强，但属于雷达感知领域，与文档图像理解无关。
Reason 该论文研究的是雷达点云在自动驾驶场景下的静态与动态物体分割及自我运动估计，属于3D视觉与雷达感知领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用完全无关。尽管其方法涉及点云处理和特征提取，但技术路线和问题域均无法迁移到文档图像的OCR、布局分析、表格识别等核心任务中。
Abstract Conventional radar segmentation research has typically focused on learning category labels for different moving objects. Although fundamental differences between radar and optical sensors lead to differences in the reliability of predicting accurate and consistent category labels, a review of common radar perception tasks in automotive reveals that determining whether an object is moving or static is a prerequisite for most tasks. To fill this gap, this study proposes a neural network based solution that can simultaneously segment static and moving objects from radar point clouds. Furthermore, since the measured radial velocity of static objects is correlated with the motion of the radar, this approach can also estimate the instantaneous 2D velocity of the moving platform or vehicle (ego motion). However, despite performing dual tasks, the proposed method employs very simple yet effective building blocks for feature extraction: multi layer perceptrons (MLPs) and recurrent neural networks (RNNs). In addition to being the first of its kind in the literature, the proposed method also demonstrates the feasibility of extracting the information required for the dual task directly from unprocessed point clouds, without the need for cloud aggregation, Doppler compensation, motion compensation, or any other intermediate signal processing steps. To measure its performance, this study introduces a set of novel evaluation metrics and tests the proposed method using a challenging real world radar dataset, RadarScenes. The results show that the proposed method not only performs well on the dual tasks, but also has broad application potential in other radar perception tasks.
Authors Simin Zhu, Satish Ravindran, Alexander Yarovoy, Francesco Fioranelli
Comment 16 pages, 9 figures, under review at IEEE Transactions on Radar Systems
Categories Computer Vision and Pattern Recognition
0
CREward: A Type-Specific Creativity Reward Model
Title CN CREward：一种类型特定的创造力奖励模型
Keywords 创造力评估 · 视觉语言模型 · 奖励模型 · 图像生成 · 人类感知
Summary 本文提出了首个针对不同类型（几何、材质、纹理）的创造力奖励模型CREward，通过人类评估建立基准，并利用大视觉语言模型（LVLM）生成标签训练奖励模型，应用于创意评估、可解释创意分析和创意样本获取。研究重点在于创意图像的生成与评价。
Reason 该论文研究的是图像生成中的‘创造力’评估问题，聚焦于几何、材质和纹理等视觉创意维度，属于纯图像生成与美学评价领域。其任务目标（创意图像评估与生成引导）与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析、长文档推理等完全无关。尽管使用了VLM，但方法面向的是艺术性创造而非文档语义结构解析，无法迁移用于解决文档理解中的准确性、结构化提取或逻辑推理问题。
Abstract Creativity is a complex phenomenon. When it comes to representing and assessing creativity, treating it as a single undifferentiated quantity would appear naive and underwhelming. In this work, we learn the \emph{first type-specific creativity reward model}, coined CREward, which spans three creativity ``axes," geometry, material, and texture, to allow us to view creativity through the lens of the image formation pipeline. To build our reward model, we first conduct a human benchmark evaluation to capture human perception of creativity for each type across various creative images. We then analyze the correlation between human judgments and predictions by large vision-language models (LVLMs), confirming that LVLMs exhibit strong alignment with human perception. Building on this observation, we collect LVLM-generated labels to train our CREward model that is applicable to both evaluation and generation of creative images. We explore three applications of CREward: creativity assessment, explainable creativity, and creative sample acquisition for both human design inspiration and guiding creative generation through low-rank adaptation.
Authors Jiyeon Han, Ali Mahdavi-Amiri, Hao Zhang, Haedong Jeong
Categories Computer Vision and Pattern Recognition
0
TK-Mamba: Marrying KAN With Mamba for Text-Driven 3D Medical Image Segmentation
Title CN TK-Mamba：将KAN与Mamba结合用于文本驱动的3D医学图像分割
Keywords 3D医学图像分割 · Mamba · KAN · 文本驱动 · 多模态融合
Summary 本文提出TK-Mamba，一种融合Kolmogorov-Arnold Networks（KAN）与Mamba架构的多模态框架，用于文本驱动的3D医学图像分割。通过引入3D-GR-KAN实现高效的非线性特征变换，并采用双分支文本驱动策略利用PubMedCLIP的文本嵌入来增强语义一致性与解剖结构对齐。实验表明该方法在多个数据集上实现了器官与肿瘤分割的SOTA性能。
Reason 该论文研究的是3D医学图像分割，属于纯医疗影像分析任务，且使用的是文本作为辅助模态来提升分割性能，核心问题不涉及文档图像理解中的OCR、排版、逻辑推理或视觉语言对齐等挑战。尽管采用了多模态（文本+3D图像）方法，但其应用场景和目标与文档图像理解（DIU/DocVQA）完全无关，属于负面清单中的‘无关领域’（医学图像处理），因此直接打0分。
Abstract 3D medical image segmentation is important for clinical diagnosis and treatment but faces challenges from high-dimensional data and complex spatial dependencies. Traditional single-modality networks, such as CNNs and Transformers, are often limited by computational inefficiency and constrained contextual modeling in 3D settings. To alleviate these limitations, we propose TK-Mamba, a multimodal framework that fuses the linear-time Mamba with Kolmogorov-Arnold Networks (KAN) to form an efficient hybrid backbone. Our approach is characterized by two primary technical contributions. Firstly, we introduce the novel 3D-Group-Rational KAN (3D-GR-KAN), which marks the first application of KAN in 3D medical imaging, providing a superior and computationally efficient nonlinear feature transformation crucial for complex volumetric structures. Secondly, we devise a dual-branch text-driven strategy using Pubmedclip's embeddings. This strategy significantly enhances segmentation robustness and accuracy by simultaneously capturing inter-organ semantic relationships to mitigate label inconsistencies and aligning image features with anatomical texts. By combining this advanced backbone and vision-language knowledge, TK-Mamba offers a unified and scalable solution for both multi-organ and tumor segmentation. Experiments on multiple datasets demonstrate that our framework achieves state-of-the-art performance in both organ and tumor segmentation tasks, surpassing existing methods in both accuracy and efficiency. Our code is publicly available at https://github.com/yhy-whu/TK-Mamba
Authors Haoyu Yang, Yutong Guan, Meixing Shi, Yuxiang Cai, Jintao Chen, Sun Bing, Wenhui Lei, Mianxin Liu, Xiaoming Shi, Yankai Jiang, Jianwei Yin
Categories Computer Vision and Pattern Recognition
0
LiHi-GS: LiDAR-Supervised Gaussian Splatting for Highway Driving Scene Reconstruction RA-L 2025
Title CN LiHi-GS：用于高速公路驾驶场景重建的激光雷达监督高斯点阵化
Keywords 3D重建 · 高斯点阵化 · 激光雷达 · 自动驾驶 · 场景渲染
Summary 本文提出了一种新的高斯点阵化方法LiHi-GS，利用激光雷达（LiDAR）进行监督，用于高速公路驾驶场景的3D重建和渲染。相比以往主要针对城市低速场景的工作，该方法专注于更具挑战性的高速公路环境，并实现了更精确的深度建模和LiDAR数据生成能力。
Reason 该论文研究的是自动驾驶中的3D场景重建，属于3D视觉与自动驾驶领域，核心方法为基于LiDAR监督的高斯点阵化（Gaussian Splatting），与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其技术目标（如动态场景渲染、LiDAR数据合成）无法迁移至解决文档理解中的OCR幻觉、复杂排版、长文档推理等核心痛点。
Abstract Photorealistic 3D scene reconstruction plays an important role in autonomous driving, enabling the generation of novel data from existing datasets to simulate safety-critical scenarios and expand training data without additional acquisition costs. Gaussian Splatting (GS) facilitates real-time, photorealistic rendering with an explicit 3D Gaussian representation of the scene, providing faster processing and more intuitive scene editing than the implicit Neural Radiance Fields (NeRFs). While extensive GS research has yielded promising advancements in autonomous driving applications, they overlook two critical aspects: First, existing methods mainly focus on low-speed and feature-rich urban scenes and ignore the fact that highway scenarios play a significant role in autonomous driving. Second, while LiDARs are commonplace in autonomous driving platforms, existing methods learn primarily from images and use LiDAR only for initial estimates or without precise sensor modeling, thus missing out on leveraging the rich depth information LiDAR offers and limiting the ability to synthesize LiDAR data. In this paper, we propose a novel GS method for dynamic scene synthesis and editing with improved scene reconstruction through LiDAR supervision and support for LiDAR rendering. Unlike prior works that are tested mostly on urban datasets, to the best of our knowledge, we are the first to focus on the more challenging and highly relevant highway scenes for autonomous driving, with sparse sensor views and monotone backgrounds. Visit our project page at: https://umautobots.github.io/lihi_gs
Authors Pou-Chun Kung, Xianling Zhang, Katherine A. Skinner, Nikita Jaipuria
Comment RA-L 2025
Categories Computer Vision and Pattern Recognition
0
GazeProphetV2: Head-Movement-Based Gaze Prediction Enabling Efficient Foveated Rendering on Mobile VR
Title CN GazeProphetV2：基于头部运动的注视点预测，实现移动VR中的高效中央凹渲染
Keywords 注视点预测 · 虚拟现实 · 多模态融合 · 头部运动 · 中央凹渲染
Summary 本文提出了一种结合眼动历史、头部运动和场景内容的多模态VR注视点预测模型GazeProphetV2，采用门控融合与跨模态注意力机制，在22个VR场景的大型数据集上提升了未来1-3帧的注视点预测准确率，并展现出良好的跨场景泛化能力。该方法旨在优化VR渲染效率，无需昂贵的眼动追踪硬件。
Reason 该论文研究的是虚拟现实（VR）场景下的注视点预测问题，核心任务是结合头部运动和视觉场景信息来预测用户注视位置，属于具身智能与VR交互领域。尽管使用了多模态融合方法，但其目标与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析或长文档推理完全无关。该技术难以迁移到文档理解场景，且不涉及任何文档结构、文本布局或视觉语言建模在文档上的应用。
Abstract Predicting gaze behavior in virtual reality environments remains a significant challenge with implications for rendering optimization and interface design. This paper introduces a multimodal approach to VR gaze prediction that combines temporal gaze patterns, head movement data, and visual scene information. By leveraging a gated fusion mechanism with cross-modal attention, the approach learns to adaptively weight gaze history, head movement, and scene content based on contextual relevance. Evaluations using a dataset spanning 22 VR scenes with 5.3M gaze samples demonstrate improvements in predictive accuracy when combining modalities compared to using individual data streams alone. The results indicate that integrating past gaze trajectories with head orientation and scene content enhances prediction accuracy across 1-3 future frames. Cross-scene generalization testing shows consistent performance with 93.1% validation accuracy and temporal consistency in predicted gaze trajectories. These findings contribute to understanding attention mechanisms in virtual environments while suggesting potential applications in rendering optimization, interaction design, and user experience evaluation. The approach represents a step toward more efficient virtual reality systems that can anticipate user attention patterns without requiring expensive eye tracking hardware.
Authors Farhaan Ebadulla, Chiraag Mudlpaur, Shreya Chaurasia, Gaurav BV
Categories Computer Vision and Pattern Recognition
0
SONIC: Spectral Optimization of Noise for Inpainting with Consistency
Title CN SONIC：用于一致性修复的噪声频谱优化
Keywords 图像修复 · 频谱优化 · 训练-free方法 · 文本到图像模型 · 逆问题
Summary 本文提出了一种无需训练的图像修复方法SONIC，通过在频谱域优化初始噪声来匹配未遮挡区域的内容，并结合线性近似加速优化过程，在少量迭代下实现了优于现有方法的修复效果。适用于通用图像修复场景。
Reason 该论文研究的是通用图像修复任务，使用文本到图像模型进行训练-free的图像修复，属于纯图像生成与修复领域。其方法聚焦于优化初始噪声的频谱表示以提升修复效果，与文档图像理解（DIU）中的OCR、布局分析、表格识别或VQA等核心任务无直接关联。无法迁移解决文档理解中的OCR幻觉、密集文本、复杂排版或长文档推理等痛点。
Abstract We propose a novel training-free method for inpainting with off-the-shelf text-to-image models. While guidance-based methods in theory allow generic models to be used for inverse problems such as inpainting, in practice, their effectiveness is limited, leading to the necessity of specialized inpainting-specific models. In this work, we argue that the missing ingredient for training-free inpainting is the optimization (guidance) of the initial seed noise. We propose to optimize the initial seed noise to approximately match the unmasked parts of the data - with as few as a few tens of optimization steps. We then apply conventional training-free inpainting methods on top of our optimized initial seed noise. Critically, we propose two core ideas to effectively implement this idea: (i) to avoid the costly unrolling required to relate the initial noise and the generated outcome, we perform linear approximation; and (ii) to stabilize the optimization, we optimize the initial seed noise in the spectral domain. We demonstrate the effectiveness of our method on various inpainting tasks, outperforming the state of the art. Project page: https://ubc-vision.github.io/sonic/
Authors Seungyeon Baek, Erqun Dong, Shadan Namazifard, Mark J. Matthews, Kwang Moo Yi
Categories Computer Vision and Pattern Recognition
0
Click2Graph: Interactive Panoptic Video Scene Graphs from a Single Click
Title CN Click2Graph：从单次点击生成交互式全景视频场景图
Keywords 视频场景图 · 交互式理解 · 视觉提示 · 对象追踪 · 关系推理
Summary 本文提出Click2Graph，首个支持单次点击交互的全景视频场景图生成框架。通过结合视觉提示与时空语义理解，系统能从一个点击或边界框出发，跨帧分割并追踪目标，自动发现交互对象，并预测主-宾-谓三元组以构建时序一致的场景图。其包含动态交互发现模块和联合实体-谓词分类头，在OpenPVSG数据集上验证了有效性。
Reason 该论文研究的是视频场景图生成（VSGG），属于视频理解领域，且核心任务围绕用户交互式提示进行动态对象发现与关系推理。尽管其‘交互式’理念具有一定新颖性，但应用场景完全偏离文档图像理解（DIU）的核心任务（如DocVQA、表格识别、文本定位等）。其方法难以迁移到解决OCR幻觉、密集文本处理或复杂排版分析等文档特有痛点，属于无关领域工作。
Abstract State-of-the-art Video Scene Graph Generation (VSGG) systems provide structured visual understanding but operate as closed, feed-forward pipelines with no ability to incorporate human guidance. In contrast, promptable segmentation models such as SAM2 enable precise user interaction but lack semantic or relational reasoning. We introduce Click2Graph, the first interactive framework for Panoptic Video Scene Graph Generation (PVSG) that unifies visual prompting with spatial, temporal, and semantic understanding. From a single user cue, such as a click or bounding box, Click2Graph segments and tracks the subject across time, autonomously discovers interacting objects, and predicts <subject, object, predicate> triplets to form a temporally consistent scene graph. Our framework introduces two key components: a Dynamic Interaction Discovery Module that generates subject-conditioned object prompts, and a Semantic Classification Head that performs joint entity and predicate reasoning. Experiments on the OpenPVSG benchmark demonstrate that Click2Graph establishes a strong foundation for user-guided PVSG, showing how human prompting can be combined with panoptic grounding and relational inference to enable controllable and interpretable video scene understanding.
Authors Raphael Ruschel, Hardikkumar Prajapati, Awsafur Rahman, B. S. Manjunath
Categories Computer Vision and Pattern Recognition
0
DVP-MVS: Synergize Depth-Edge and Visibility Prior for Multi-View Stereo
Title CN DVP-MVS：融合深度边缘与可见性先验的多视角立体匹配
Keywords 多视角立体匹配 · 深度估计 · 边缘对齐 · 可见性先验 · 几何一致性
Summary 本文提出DVP-MVS，通过融合深度边缘对齐和跨视角可见性先验来提升多视角立体匹配中纹理缺失区域的重建鲁棒性。利用Depth Anything V2生成粗略深度图，并结合Roberts算子提取边缘，通过形态学操作实现边界对齐以指导块变形；同时将视图选择权重重构为可见性图，通过跨视角深度重投影恢复可见区域。最后引入多视角几何一致性优化传播与细化过程。在ETH3D和Tanks & Temples数据集上实现了SOTA性能。
Reason 该论文研究的是多视角立体匹配（Multi-View Stereo），属于3D视觉重建领域，核心关注点为深度估计、视图选择和几何一致性优化。尽管其方法涉及边缘对齐与可见性建模，但整体任务与文档图像理解（DIU）或文档视觉问答（DocVQA）等无关。未涉及文本识别、布局分析、表格理解或OCR-free的文档处理任务，也无法直接迁移解决DIU中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Patch deformation-based methods have recently exhibited substantial effectiveness in multi-view stereo, due to the incorporation of deformable and expandable perception to reconstruct textureless areas. However, such approaches typically focus on exploring correlative reliable pixels to alleviate match ambiguity during patch deformation, but ignore the deformation instability caused by mistaken edge-skipping and visibility occlusion, leading to potential estimation deviation. To remedy the above issues, we propose DVP-MVS, which innovatively synergizes depth-edge aligned and cross-view prior for robust and visibility-aware patch deformation. Specifically, to avoid unexpected edge-skipping, we first utilize Depth Anything V2 followed by the Roberts operator to initialize coarse depth and edge maps respectively, both of which are further aligned through an erosion-dilation strategy to generate fine-grained homogeneous boundaries for guiding patch deformation. In addition, we reform view selection weights as visibility maps and restore visible areas by cross-view depth reprojection, then regard them as cross-view prior to facilitate visibility-aware patch deformation. Finally, we improve propagation and refinement with multi-view geometry consistency by introducing aggregated visible hemispherical normals based on view selection and local projection depth differences based on epipolar lines, respectively. Extensive evaluations on ETH3D and Tanks & Temples benchmarks demonstrate that our method can achieve state-of-the-art performance with excellent robustness and generalization.
Authors Zhenlong Yuan, Jinguo Luo, Fei Shen, Zhaoxin Li, Cong Liu, Tianlu Mao, Zhaoqi Wang
Categories Computer Vision and Pattern Recognition
0
VGGT4D: Mining Motion Cues in Visual Geometry Transformers for 4D Scene Reconstruction
Title CN VGGT4D：在视觉几何变换器中挖掘运动线索用于4D场景重建
Keywords 4D场景重建 · 视觉几何变换器 · 动态线索挖掘 · 无训练框架 · 注意力机制
Summary 本文提出VGGT4D，一种无需训练的框架，通过挖掘视觉几何变换器（VGGT）中的全局注意力层隐含的动态线索，实现对动态4D场景的鲁棒重建。通过gram相似性挖掘并聚合时间窗口内的动态特征，并引入基于投影梯度的细化策略优化掩码边界，从而在无需微调的情况下提升动态物体分割与三维重建性能。
Reason 该论文研究的是动态4D场景重建，属于视频理解与3D视觉领域，核心任务为动态物体分割与相机位姿估计，与文档图像理解（DIU）或DocVQA无关。未涉及OCR、布局分析、表格识别或多模态文档推理等DIU核心问题，且其方法依赖于时序运动线索和3D几何建模，难以迁移到静态文档理解场景。
Abstract Reconstructing dynamic 4D scenes is challenging, as it requires robust disentanglement of dynamic objects from the static background. While 3D foundation models like VGGT provide accurate 3D geometry, their performance drops markedly when moving objects dominate. Existing 4D approaches often rely on external priors, heavy post-optimization, or require fine-tuning on 4D datasets. In this paper, we propose VGGT4D, a training-free framework that extends the 3D foundation model VGGT for robust 4D scene reconstruction. Our approach is motivated by the key finding that VGGT's global attention layers already implicitly encode rich, layer-wise dynamic cues. To obtain masks that decouple static and dynamic elements, we mine and amplify global dynamic cues via gram similarity and aggregate them across a temporal window. To further sharpen mask boundaries, we introduce a refinement strategy driven by projection gradient. We then integrate these precise masks into VGGT's early-stage inference, effectively mitigating motion interference in both pose estimation and geometric reconstruction. Across six datasets, our method achieves superior performance in dynamic object segmentation, camera pose estimation, and dense reconstruction. It also supports single-pass inference on sequences longer than 500 frames.
Authors Yu Hu, Chong Cheng, Sicheng Yu, Xiaoyang Guo, Hao Wang
Categories Computer Vision and Pattern Recognition
0
HiCoGen: Hierarchical Compositional Text-to-Image Generation in Diffusion Models via Reinforcement Learning
Title CN HiCoGen：基于强化学习的扩散模型中分层组合式文本到图像生成
Keywords 文本到图像生成 · 扩散模型 · 强化学习 · 组合性 · 分层生成
Summary 本文提出HiCoGen框架，通过‘合成链’（Chain of Synthesis）范式在扩散模型中实现分层组合式图像生成。利用大语言模型将复杂提示分解为基本语义单元，并逐步合成图像，每步利用前序图像作为视觉上下文。为提升强化学习效果，设计了衰减随机性调度策略以增强探索能力，并采用多层次奖励机制评估生成结果。同时构建了HiCoPrompt基准用于评测。实验表明该方法在概念覆盖和组合准确性上优于现有方法。
Reason 该论文属于纯图像生成领域，研究的是复杂提示下的文本到图像合成问题，核心方法为分层生成与强化学习调度。尽管使用了LLM分解语义单元并引入视觉上下文迭代生成，但其目标是解决通用图像生成中的组合性与完整性问题，而非文档理解任务。无法迁移用于解决OCR幻觉、细粒度定位或长文档推理等DIU核心痛点。
Abstract Recent advances in diffusion models have demonstrated impressive capability in generating high-quality images for simple prompts. However, when confronted with complex prompts involving multiple objects and hierarchical structures, existing models struggle to accurately follow instructions, leading to issues such as concept omission, confusion, and poor compositionality. To address these limitations, we propose a Hierarchical Compositional Generative framework (HiCoGen) built upon a novel Chain of Synthesis (CoS) paradigm. Instead of monolithic generation, HiCoGen first leverages a Large Language Model (LLM) to decompose complex prompts into minimal semantic units. It then synthesizes these units iteratively, where the image generated in each step provides crucial visual context for the next, ensuring all textual concepts are faithfully constructed into the final scene. To further optimize this process, we introduce a reinforcement learning (RL) framework. Crucially, we identify that the limited exploration of standard diffusion samplers hinders effective RL. We theoretically prove that sample diversity is maximized by concentrating stochasticity in the early generation stages and, based on this insight, propose a novel Decaying Stochasticity Schedule to enhance exploration. Our RL algorithm is then guided by a hierarchical reward mechanism that jointly evaluates the image at the global, subject, and relationship levels. We also construct HiCoPrompt, a new text-to-image benchmark with hierarchical prompts for rigorous evaluation. Experiments show our approach significantly outperforms existing methods in both concept coverage and compositional accuracy.
Authors Hongji Yang, Yucheng Zhou, Wencheng Han, Runzhou Tao, Zhongying Qiu, Jianfei Yang, Jianbing Shen
Comment 9 pages
Categories Computer Vision and Pattern Recognition
0
Hestia: Voxel-Face-Aware Hierarchical Next-Best-View Acquisition for Efficient 3D Reconstruction IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2026
Title CN Hestia：面向体素-面片感知的层次化最优下一视角获取用于高效3D重建
Keywords 3D重建 · 下一最优视角 · 体素感知 · 面片感知 · 强化学习
Summary 本文提出Hestia，一种基于体素和面片感知的层次化下一最优视角获取方法，用于提升3D重建效率与鲁棒性。通过构建多样化数据集、采用分层结构管理高维动作空间、设计近端贪婪策略缓解虚假相关性，并引入面片感知机制避免几何遗漏。实验表明其在覆盖率和 Chamfer 距离上显著优于现有方法，且具备实时推理与真实场景应用能力。
Reason 该论文研究的是3D重建中的下一视角规划问题，属于具身智能与机器人感知范畴，核心任务为视角序列决策以优化三维几何重建质量。其方法聚焦于体素与面片感知的层次化策略，在WACV发表，虽属CV领域但与文档图像理解（DIU）、DocVQA、OCR、布局分析等任务无直接关联。所解决的‘视角采集’问题在文档理解中不存在对应迁移场景，且未涉及多模态语言模型、视觉表示学习或高分辨率图像处理等可复用技术。属于明确的无关领域（3D视觉/具身智能），符合负面清单第2条。
Abstract Advances in 3D reconstruction and novel view synthesis have enabled efficient and photorealistic rendering. However, images for reconstruction are still either largely manual or constrained by simple preplanned trajectories. To address this issue, recent works propose generalizable next-best-view planners that do not require online learning. Nevertheless, robustness and performance remain limited across various shapes. Hence, this study introduces Voxel-Face-Aware Hierarchical Next-Best-View Acquisition for Efficient 3D Reconstruction (Hestia), which addresses the shortcomings of the reinforcement learning-based generalizable approaches for five-degree-of-freedom viewpoint prediction. Hestia systematically improves the planners through four components: a more diverse dataset to promote robustness, a hierarchical structure to manage the high-dimensional continuous action search space, a close-greedy strategy to mitigate spurious correlations, and a face-aware design to avoid overlooking geometry. Experimental results show that Hestia achieves non-marginal improvements, with at least a 4% gain in coverage ratio, while reducing Chamfer Distance by 50% and maintaining real-time inference. In addition, Hestia outperforms prior methods by at least 12% in coverage ratio with a 5-image budget and remains robust to object placement variations. Finally, we demonstrate that Hestia, as a next-best-view planner, is feasible for the real-world application. Our project page is https://johnnylu305.github.io/hestia web.
Authors Cheng-You Lu, Zhuoli Zhuang, Nguyen Thanh Trung Le, Da Xiao, Yu-Cheng Chang, Thomas Do, Srinath Sridhar, Chin-teng Lin
Comment Accepted to the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2026
Categories Computer Vision and Pattern Recognition
0
Beyond Fully Supervised Pixel Annotations: Scribble-Driven Weakly-Supervised Framework for Image Manipulation Localization
Title CN 超越全监督像素标注：基于涂鸦的弱监督图像篡改定位框架
Keywords 图像篡改定位 · 弱监督学习 · 涂鸦标注 · 自监督学习 · 特征调制
Summary 本文提出了一种基于涂鸦标注的弱监督图像篡改定位框架，构建了首个涂鸦标注的IML数据集，并设计了包含自监督结构一致性损失、先验感知特征调制模块（PFMM）、门控自适应融合模块（GAFM）以及置信度感知熵最小化损失（L_CEM）的训练框架。实验表明该方法在分布内和分布外数据上均优于现有全监督方法。
Reason 该论文研究的是图像篡改定位（IML），属于通用计算机视觉任务，且聚焦于弱监督学习下的篡改区域分割。尽管使用了涂鸦标注以降低标注成本，但其核心问题与文档图像理解（DIU）如DocVQA、布局分析、表格识别等无直接关联。方法中的PFMM、GAFM和结构一致性损失等模块针对的是篡改检测场景，并不解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点，迁移价值极低。
Abstract Deep learning-based image manipulation localization (IML) methods have achieved remarkable performance in recent years, but typically rely on large-scale pixel-level annotated datasets. To address the challenge of acquiring high-quality annotations, some recent weakly supervised methods utilize image-level labels to segment manipulated regions. However, the performance is still limited due to insufficient supervision signals. In this study, we explore a form of weak supervision that improves the annotation efficiency and detection performance, namely scribble annotation supervision. We re-annotate mainstream IML datasets with scribble labels and propose the first scribble-based IML (Sc-IML) dataset. Additionally, we propose the first scribble-based weakly supervised IML framework. Specifically, we employ self-supervised training with a structural consistency loss to encourage the model to produce consistent predictions under multi-scale and augmented inputs. In addition, we propose a prior-aware feature modulation module (PFMM) that adaptively integrates prior information from both manipulated and authentic regions for dynamic feature adjustment, further enhancing feature discriminability and prediction consistency in complex scenes. We also propose a gated adaptive fusion module (GAFM) that utilizes gating mechanisms to regulate information flow during feature fusion, guiding the model toward emphasizing potential manipulated regions. Finally, we propose a confidence-aware entropy minimization loss (${\mathcal{L}}_{ {CEM }}$). This loss dynamically regularizes predictions in weakly annotated or unlabeled regions based on model uncertainty, effectively suppressing unreliable predictions. Experimental results show that our method outperforms existing fully supervised approaches in terms of average performance both in-distribution and out-of-distribution.
Authors Songlin Li, Guofeng Yu, Zhiqing Guo, Yunfeng Diao, Dan Ma, Gaobo Yang
Categories Computer Vision and Pattern Recognition
0
How Animals Dance (When You're Not Looking)
Title CN 当你不看时动物如何跳舞
Keywords 视频生成 · 扩散模型 · 动作合成 · 编舞模式 · 关键帧优化
Summary 本文提出了一种生成音乐同步的动物舞蹈视频的框架，通过从人类舞蹈视频中提取编舞模式，并结合少量动物关键帧和视频扩散模型，生成长达30秒的舞蹈视频。重点在于动作节奏控制与视觉连贯性生成。
Reason 该论文研究的是动物舞蹈视频生成，属于纯图像生成与视频合成领域，核心任务为音乐同步的舞蹈动作生成和视频扩散模型应用。其技术路线与文档图像理解（DIU）无直接关联，既不涉及OCR、布局分析、表格识别等DIU任务，也无法迁移解决OCR幻觉、密集文本处理或复杂排版理解等核心痛点。
Abstract We present a framework for generating music-synchronized, choreography aware animal dance videos. Our framework introduces choreography patterns -- structured sequences of motion beats that define the long-range structure of a dance -- as a novel high-level control signal for dance video generation. These patterns can be automatically estimated from human dance videos. Starting from a few keyframes representing distinct animal poses, generated via text-to-image prompting or GPT-4o, we formulate dance synthesis as a graph optimization problem that seeks the optimal keyframe structure to satisfy a specified choreography pattern of beats. We also introduce an approach for mirrored pose image generation, essential for capturing symmetry in dance. In-between frames are synthesized using an video diffusion model. With as few as six input keyframes, our method can produce up to 30 seconds dance videos across a wide range of animals and music tracks.
Authors Xiaojuan Wang, Aleksander Holynski, Brian Curless, Ira Kemelmacher, Steve Seitz
Comment Project page: https://how-animals-dance.github.io/
Categories Computer Vision and Pattern Recognition
0
MSP-MVS: Multi-Granularity Segmentation Prior Guided Multi-View Stereo
Title CN MSP-MVS：多粒度分割先验引导的多视角立体匹配
Keywords 多视角立体匹配 · patch变形 · 分割先验 · 深度边缘 · 3D重建
Summary 本文提出MSP-MVS，通过引入多粒度分割先验来约束patch的形变过程，避免在多视角立体匹配中发生边缘跨越，提升无纹理区域的重建效果。方法包括基于Semantic-SAM提取深度边缘、相关可靠像素的自适应均衡分布与聚类解耦，以及视差采样协同的3D优化策略。在ETH3D和Tanks & Temples数据集上实现了SOTA性能。
Reason 该论文属于三维视觉与多视角立体匹配（Multi-View Stereo）领域，核心解决的是深度估计中的patch变形与边缘跳跃问题，应用场景为3D重建。尽管其方法涉及视觉结构建模，但完全不涉及文档图像理解任务（如DocVQA、OCR、版面分析等），也不提供可迁移至文档理解领域的通用方法论（如视觉token压缩、VLM架构改进、推理增强机制等）。属于明确的无关领域（3D视觉），命中负面清单第2条。
Abstract Recently, patch deformation-based methods have demonstrated significant strength in multi-view stereo by adaptively expanding the reception field of patches to help reconstruct textureless areas. However, such methods mainly concentrate on searching for pixels without matching ambiguity (i.e., reliable pixels) when constructing deformed patches, while neglecting the deformation instability caused by unexpected edge-skipping, resulting in potential matching distortions. Addressing this, we propose MSP-MVS, a method introducing multi-granularity segmentation prior for edge-confined patch deformation. Specifically, to avoid unexpected edge-skipping, we first aggregate and further refine multi-granularity depth edges gained from Semantic-SAM as prior to guide patch deformation within depth-continuous (i.e., homogeneous) areas. Moreover, to address attention imbalance caused by edge-confined patch deformation, we implement adaptive equidistribution and disassemble-clustering of correlative reliable pixels (i.e., anchors), thereby promoting attention-consistent patch deformation. Finally, to prevent deformed patches from falling into local-minimum matching costs caused by the fixed sampling pattern, we introduce disparity-sampling synergistic 3D optimization to help identify global-minimum matching costs. Evaluations on ETH3D and Tanks & Temples benchmarks prove our method obtains state-of-the-art performance with remarkable generalization.
Authors Zhenlong Yuan, Cong Liu, Fei Shen, Zhaoxin Li, Jinguo Luo, Tianlu Mao, Zhaoqi Wang
Categories Computer Vision and Pattern Recognition
0
SD-MVS: Segmentation-Driven Deformation Multi-View Stereo with Spherical Refinement and EM optimization AAAI2024
Title CN SD-MVS：基于分割驱动形变的多视角立体匹配，结合球面优化与EM优化
Keywords 3D重建 · 多视角立体匹配 · SAM · EM优化 · 球面细化
Summary 本文提出SD-MVS，一种基于分割驱动形变的多视角立体匹配方法，利用SAM进行语义实例分割，并以此指导像素级匹配代价的形变传播。同时引入球面坐标系下的法向量与深度联合优化策略，并采用EM算法交替优化匹配代价与超参数，在ETH3D和Tanks and Temples数据集上实现了SOTA性能。
Reason 该论文研究的是3D重建中的多视角立体匹配（MVS），属于3D视觉与几何重建领域，核心问题为无纹理区域的深度估计与模型补全。尽管使用了SAM和优化策略，但其任务目标与文档图像理解（DIU）如DocVQA、布局分析、表格识别等完全无关。方法无法直接迁移用于解决OCR幻觉、密集文本处理或复杂排版理解等DIU核心痛点。
Abstract In this paper, we introduce Segmentation-Driven Deformation Multi-View Stereo (SD-MVS), a method that can effectively tackle challenges in 3D reconstruction of textureless areas. We are the first to adopt the Segment Anything Model (SAM) to distinguish semantic instances in scenes and further leverage these constraints for pixelwise patch deformation on both matching cost and propagation. Concurrently, we propose a unique refinement strategy that combines spherical coordinates and gradient descent on normals and pixelwise search interval on depths, significantly improving the completeness of reconstructed 3D model. Furthermore, we adopt the Expectation-Maximization (EM) algorithm to alternately optimize the aggregate matching cost and hyperparameters, effectively mitigating the problem of parameters being excessively dependent on empirical tuning. Evaluations on the ETH3D high-resolution multi-view stereo benchmark and the Tanks and Temples dataset demonstrate that our method can achieve state-of-the-art results with less time consumption.
Authors Zhenlong Yuan, Jiakai Cao, Zhaoxin Li, Hao Jiang, Zhaoqi Wang
Comment Published to AAAI2024
Categories Computer Vision and Pattern Recognition
0
Supervise Less, See More: Training-free Nuclear Instance Segmentation with Prototype-Guided Prompting
Title CN 监督更少，看见更多：基于原型引导提示的免训练核实例分割
Keywords 核实例分割 · 免训练方法 · 提示学习 · 最优传输 · SAM
Summary 本文提出SPROUT，一种无需训练和标注的核实例分割框架，利用组织学先验构建特定幻灯片的参考原型，并通过部分最优传输机制逐步引导特征对齐，生成正负点提示以驱动SAM完成精确分割。实验表明其在多个病理学基准上表现优异，为可扩展的免训练医学图像分割提供了新范式。
Reason 该论文属于医学图像分析领域（计算病理学），研究任务为核实例分割，依赖于组织学先验和最优传输策略来引导SAM模型进行零样本分割。尽管其‘免训练’提示框架具有方法新颖性，但应用场景局限于生物医学图像，与文档图像理解（如DocVQA、布局分析、表格识别等）的核心痛点（OCR幻觉、复杂排版、长文档推理）无直接关联。所提技术难以迁移到文档场景，且未涉及多模态大模型在文本密集或结构化文档中的理解问题。
Abstract Accurate nuclear instance segmentation is a pivotal task in computational pathology, supporting data-driven clinical insights and facilitating downstream translational applications. While large vision foundation models have shown promise for zero-shot biomedical segmentation, most existing approaches still depend on dense supervision and computationally expensive fine-tuning. Consequently, training-free methods present a compelling research direction, yet remain largely unexplored. In this work, we introduce SPROUT, a fully training- and annotation-free prompting framework for nuclear instance segmentation. SPROUT leverages histology-informed priors to construct slide-specific reference prototypes that mitigate domain gaps. These prototypes progressively guide feature alignment through a partial optimal transport scheme. The resulting foreground and background features are transformed into positive and negative point prompts, enabling the Segment Anything Model (SAM) to produce precise nuclear delineations without any parameter updates. Extensive experiments across multiple histopathology benchmarks demonstrate that SPROUT achieves competitive performance without supervision or retraining, establishing a novel paradigm for scalable, training-free nuclear instance segmentation in pathology.
Authors Wen Zhang, Qin Ren, Wenjing Liu, Haibin Ling, Chenyu You
Comment Preprint; 40 pages, 25 figures, 18 tables
Categories Computer Vision and Pattern Recognition
0
Natural Image Stitching Using Depth Maps Signal Processing: Image Communication
Title CN 使用深度图的自然图像拼接
Keywords 图像拼接 · 深度图 · 视差校正 · 极线几何 · 图像对齐
Summary 本文提出一种利用深度图进行自然图像拼接的方法，通过估计极线几何和最优扭曲来减少由视差引起的拼接伪影。在多个数据集上验证了该方法在重叠区域和非重叠区域的对齐准确性与视觉一致性。
Reason 该论文研究的是自然图像拼接中的视差处理问题，属于通用计算机视觉任务，与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版、细粒度文本识别等无直接关联。其方法基于深度图和几何建模，主要面向非平面场景下的图像对齐，无法迁移至文档图像的语义理解或VLM架构优化。属于无关领域。
Abstract Natural image stitching aims to create a single, natural-looking mosaic from overlapped images that capture the same 3D scene from different viewing positions. Challenges inevitably arise when the scene is non-planar and captured by handheld cameras since parallax is non-negligible in such cases. In this paper, we propose a novel image stitching method using depth maps, which generates accurate alignment mosaics against parallax. Firstly, we construct a robust fitting method to filter out the outliers in feature matches and estimate the epipolar geometry between input images. Then, we utilize epipolar geometry to establish pixel-to-pixel correspondences between the input images and render the warped images using the proposed optimal warping. In the rendering stage, we introduce several modules to solve the mapping artifacts in the warping results and generate the final mosaic. Experimental results on three challenging datasets demonstrate that the depth maps of input images enable our method to provide much more accurate alignment in the overlapping region and view-consistent results in the non-overlapping region. We believe our method will continue to work under the rapid progress of monocular depth estimation. The source code is available at https://github.com/tlliao/NIS_depths.
Authors Tianli Liao, Nan Li
Comment accept by Signal Processing: Image Communication
Categories Computer Vision and Pattern Recognition
0
Image Diffusion Models Exhibit Emergent Temporal Propagation in Videos
Title CN 图像扩散模型在视频中表现出新兴的时间传播能力
Keywords 图像扩散模型 · 视频对象分割 · 零样本跟踪 · 自注意力机制 · SAM
Summary 本文发现图像扩散模型的自注意力图可被解释为语义标签传播核，通过跨帧扩展形成时间传播核，实现零样本视频对象跟踪与分割。作者提出DRIFT框架，结合DDIM反演、文本反演和自适应头加权等测试时优化策略，并利用SAM进行掩码细化，在标准视频物体分割基准上达到SOTA性能。
Reason 该论文研究图像扩散模型在视频中的零样本物体跟踪与分割应用，属于视频理解领域。其核心方法围绕视频帧间的语义传播与SAM结合的掩码优化，与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版解析、长文档推理等无直接关联。尽管扩散模型的注意力机制分析具有一定通用性，但该工作未提供可迁移至文档场景的底层方法创新，且应用场景明确为视频，属于负面清单中的‘无关领域’。
Abstract Image diffusion models, though originally developed for image generation, implicitly capture rich semantic structures that enable various recognition and localization tasks beyond synthesis. In this work, we investigate their self-attention maps can be reinterpreted as semantic label propagation kernels, providing robust pixel-level correspondences between relevant image regions. Extending this mechanism across frames yields a temporal propagation kernel that enables zero-shot object tracking via segmentation in videos. We further demonstrate the effectiveness of test-time optimization strategies-DDIM inversion, textual inversion, and adaptive head weighting-in adapting diffusion features for robust and consistent label propagation. Building on these findings, we introduce DRIFT, a framework for object tracking in videos leveraging a pretrained image diffusion model with SAM-guided mask refinement, achieving state-of-the-art zero-shot performance on standard video object segmentation benchmarks.
Authors Youngseo Kim, Dohyun Kim, Geohee Han, Paul Hongsuck Seo
Categories Computer Vision and Pattern Recognition
0
GigaBrain-0: A World Model-Powered Vision-Language-Action Model
Title CN GigaBrain-0：由世界模型驱动的视觉-语言-行动模型
Keywords 视觉-语言-行动模型 · 世界模型 · 机器人学习 · 数据生成 · 具身智能
Summary 本文提出了GigaBrain-0，一种基于世界模型生成数据训练的视觉-语言-行动基础模型，用于提升机器人在灵巧操作、长视野任务中的泛化能力。通过视频生成、视角转换等多种合成数据增强训练，并引入RGBD建模与具身化的思维链监督，增强了空间推理与长期依赖建模。实验表明其在外观、物体位置和相机视角变化下具有优异的现实世界表现。同时发布了轻量级版本GigaBrain-0-Small，可在边缘设备高效运行。
Reason 该论文属于具身智能与机器人领域，研究的是面向通用机器人的视觉-语言-行动（VLA）模型，其核心贡献在于利用世界模型生成仿真数据以减少对真实机器人数据的依赖。尽管使用了RGBD输入和embodied CoT等技术，但整体方法针对的是机器人操控任务，与文档图像理解（DIU）或DocVQA无直接关联。所提出的数据生成方式、动作策略建模及硬件部署优化均难以迁移到文档解析、OCR去幻觉、复杂版面分析等DIU核心问题。
Abstract Training Vision-Language-Action (VLA) models for generalist robots typically requires large-scale real-world robot data, which is expensive and time-consuming to collect. The inefficiency of physical data collection severely limits the scalability, and generalization capacity of current VLA systems. To address this challenge, we introduce GigaBrain-0, a novel VLA foundation model empowered by world model-generated data (e.g., video generation, real2real transfer, human transfer, view transfer, sim2real transfer data). By leveraging world models to generate diverse data at scale, GigaBrain-0 significantly reduces reliance on real robot data while improving cross-task generalization. Our approach further improves policy robustness through RGBD input modeling and embodied Chain-of-Thought (CoT) supervision, enabling the model to reason about spatial geometry, object states, and long-horizon dependencies during task execution. This leads to substantial gains in real-world performance on dexterous, long-horizon, and mobile manipulation tasks. Extensive experiments demonstrate that GigaBrain-0 achieves superior generalization across variations in appearances (e.g., textures, colors), object placements, and camera viewpoints. Additionally, we present GigaBrain-0-Small, an optimized lightweight variant designed to run efficiently on devices such as the NVIDIA Jetson AGX Orin.
Authors GigaBrain Team, Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Haoyun Li, Jie Li, Jiagang Zhu, Lv Feng, Peng Li, Qiuping Deng, Runqi Ouyang, Wenkang Qin, Xinze Chen, Xiaofeng Wang, Yang Wang, Yifan Li, Yilong Li, Yiran Ding, Yuan Xu, Yun Ye, Yukun Zhou, Zhehao Dong, Zhenan Wang, Zhichao Liu, Zheng Zhu
Comment https://gigabrain0.github.io/
Categories Computer Vision and Pattern Recognition
0
X-ReID: Multi-granularity Information Interaction for Video-Based Visible-Infrared Person Re-Identification AAAI2026
Title CN X-ReID：用于基于视频的可见光-红外行人重识别的多粒度信息交互
Keywords 行人重识别 · 可见光-红外 · 视频理解 · 跨模态学习 · AAAI2026
Summary 本文提出X-ReID框架，用于解决基于视频的可见光-红外行人重识别中的模态差异与时序建模问题。通过跨模态原型协作（CPC）和多粒度信息交互（MII）机制，在两个大规模数据集上取得了优异性能。但该工作属于特定视觉任务，不涉及文档结构、文本理解或多模态文档推理。
Reason 该论文研究的是视频模态下的可见光与红外行人重识别（VVI-ReID），属于视频理解与行人重识别领域，与文档图像理解（DIU）或DocVQA无直接关联。其核心技术如跨模态原型协作（CPC）和多粒度信息交互（MII）聚焦于视频序列中的时空建模与模态对齐，难以迁移到文档理解中的OCR幻觉、复杂版面解析或长文档推理等核心痛点。
Abstract Large-scale vision-language models (e.g., CLIP) have recently achieved remarkable performance in retrieval tasks, yet their potential for Video-based Visible-Infrared Person Re-Identification (VVI-ReID) remains largely unexplored. The primary challenges are narrowing the modality gap and leveraging spatiotemporal information in video sequences. To address the above issues, in this paper, we propose a novel cross-modality feature learning framework named X-ReID for VVI-ReID. Specifically, we first propose a Cross-modality Prototype Collaboration (CPC) to align and integrate features from different modalities, guiding the network to reduce the modality discrepancy. Then, a Multi-granularity Information Interaction (MII) is designed, incorporating short-term interactions from adjacent frames, long-term cross-frame information fusion, and cross-modality feature alignment to enhance temporal modeling and further reduce modality gaps. Finally, by integrating multi-granularity information, a robust sequence-level representation is achieved. Extensive experiments on two large-scale VVI-ReID benchmarks (i.e., HITSZ-VCM and BUPTCampus) demonstrate the superiority of our method over state-of-the-art methods. The source code is released at https://github.com/AsuradaYuci/X-ReID.
Authors Chenyang Yu, Xuehu Liu, Pingping Zhang, Huchuan Lu
Comment Accepted by AAAI2026. More modifications may be performed
Categories Computer Vision and Pattern Recognition
0
STT-GS: Sample-Then-Transmit Edge Gaussian Splatting with Joint Client Selection and Power Control
Title CN STT-GS：基于联合客户端选择与功率控制的采样后传输边缘高斯溅射
Keywords 高斯溅射 · 边缘计算 · 客户端选择 · 功率控制 · 3D重建
Summary 本文提出了一种名为STT-GS的边缘高斯溅射框架，通过‘先采样后传输’策略和联合客户端选择与功率控制机制，在保证3D场景重建质量的同时降低通信开销。方法包括特征域聚类和传输时间最小化，并采用PAMM算法求解非凸优化问题。实验表明该方案在真实数据集上表现优异。
Reason 该论文研究的是边缘高斯溅射（Edge Gaussian Splatting）在低空经济场景下的3D场景重建问题，属于3D视觉与无线通信资源管理的交叉领域。其核心目标是优化分布式客户端（如无人机）向边缘服务器传输数据以构建高质量3D场景模型的过程，与文档图像理解（DIU）或DocVQA无关。未涉及文本识别、布局分析、表格理解、多模态文档推理等任何DIU任务，也无法迁移用于解决OCR幻觉、复杂排版、长文档推理等关键痛点。
Abstract Edge Gaussian splatting (EGS), which aggregates data from distributed clients (e.g., drones) and trains a global GS model at the edge (e.g., ground server), is an emerging paradigm for scene reconstruction in low-altitude economy. Unlike traditional edge resource management methods that emphasize communication throughput or general-purpose learning performance, EGS explicitly aims to maximize the GS qualities, rendering existing approaches inapplicable. To address this problem, this paper formulates a novel GS-oriented objective function that distinguishes the heterogeneous view contributions of different clients. However, evaluating this function in turn requires clients' images, leading to a causality dilemma. To this end, this paper further proposes a sample-then-transmit EGS (or STT-GS for short) strategy, which first samples a subset of images as pilot data from each client for loss prediction. Based on the first-stage evaluation, communication resources are then prioritized towards more valuable clients. To achieve efficient sampling, a feature-domain clustering (FDC) scheme is proposed to select the most representative data and pilot transmission time minimization (PTTM) is adopted to reduce the pilot overhead.Subsequently, we develop a joint client selection and power control (JCSPC) framework to maximize the GS-oriented function under communication resource constraints. Despite the nonconvexity of the problem, we propose a low-complexity efficient solution based on the penalty alternating majorization minimization (PAMM) algorithm. Experiments reveal that the proposed scheme significantly outperforms existing benchmarks on real-world datasets. The GS-oriented objective can be accurately predicted with low sampling ratios (e.g., 10%), and our method achieves an excellent tradeoff between view contributions and communication costs.
Authors Zhen Li, Xibin Jin, Guoliang Li, Shuai Wang, Miaowen Wen, Huseyin Arslan, Derrick Wing Kwan Ng, Chengzhong Xu
Categories Computer Vision and Pattern Recognition
0
CounterVQA: Evaluating and Improving Counterfactual Reasoning in Vision-Language Models for Video Understanding
Title CN CounterVQA：评估和提升视觉语言模型在视频理解中的反事实推理能力
Keywords 反事实推理 · 视频理解 · 视觉语言模型 · 基准测试 · 后训练
Summary 本文提出CounterVQA，一个用于评估视觉语言模型在视频中进行反事实推理能力的基准，包含三个难度层级，并发现现有模型在多跳因果链上的表现显著下降。为此，作者提出CFGPT后训练方法，通过从语言模态蒸馏反事实推理能力来提升模型性能。
Reason 该论文聚焦于视频理解中的反事实推理，属于无关领域（视频理解），且其任务和数据构建均围绕视频时序因果展开，与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂版面分析、细粒度文本定位等无直接关联。尽管反事实推理本身是一种有价值的认知能力，但其方法迁移至文档理解场景的路径不明确，无法解决DIU的关键挑战。
Abstract Vision Language Models (VLMs) have recently shown significant advancements in video understanding, especially in feature alignment, event reasoning, and instruction-following tasks. However, their capability for counterfactual reasoning, inferring alternative outcomes under hypothetical conditions, remains underexplored. This capability is essential for robust video understanding, as it requires identifying underlying causal structures and reasoning about unobserved possibilities, rather than merely recognizing observed patterns. To systematically evaluate this capability, we introduce CounterVQA, a video-based benchmark featuring three progressive difficulty levels that assess different aspects of counterfactual reasoning. Through comprehensive evaluation of both state-of-the-art open-source and closed-source models, we uncover a substantial performance gap: while these models achieve reasonable accuracy on simple counterfactual questions, performance degrades significantly on complex multi-hop causal chains. To address these limitations, we develop a post-training method, CFGPT, that enhances a model's visual counterfactual reasoning ability by distilling its counterfactual reasoning capability from the language modality, yielding consistent improvements across all CounterVQA difficulty levels. Dataset and code will be further released.
Authors Yuefei Chen, Jiang Liu, Xiaodong Lin, Ruixiang Tang
Categories Computer Vision and Pattern Recognition, Computation and Language
0
Human-Centric Open-Future Task Discovery: Formulation, Benchmark, and Scalable Tree-Based Search AAAI 2026
Title CN 以人类为中心的开放未来任务发现：形式化、基准与可扩展的基于树的搜索
Keywords 具身智能 · 任务发现 · 多智能体系统 · 开放未来 · 视频理解
Summary 本文提出了一个名为HOTD的新问题：在开放的未来场景中，如何让大模型主动发现能够减少人类负担的任务。为此构建了包含2000多个真实视频的HOTD-Bench，并设计了一个基于多智能体协作和搜索树的CMAST框架，在实验中表现优于现有LMM。然而该工作聚焦于具身AI和动态环境交互，与文档理解无直接关联。
Reason 该论文研究的是具身智能与机器人领域中的任务发现，核心场景是基于视频理解的人类意图预测和未来任务规划。尽管提出了多智能体搜索树（CMAST）框架，但其目标与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、密集文本识别、复杂版式分析或长文档推理等核心问题。属于‘无关领域’，命中负面清单第2条。
Abstract Recent progress in robotics and embodied AI is largely driven by Large Multimodal Models (LMMs). However, a key challenge remains underexplored: how can we advance LMMs to discover tasks that directly assist humans in open-future scenarios, where human intentions are highly concurrent and dynamic. In this work, we formalize the problem of Human-centric Open-future Task Discovery (HOTD), focusing particularly on identifying tasks that reduce human effort across multiple plausible futures. To facilitate this study, we propose an HOTD-Bench, which features over 2K real-world videos, a semi-automated annotation pipeline, and a simulation-based protocol tailored for open-set future evaluation. Additionally, we propose the Collaborative Multi-Agent Search Tree (CMAST) framework, which decomposes the complex reasoning through a multi-agent system and structures the reasoning process through a scalable search tree module. In our experiments, CMAST achieves the best performance on the HOTD-Bench, significantly surpassing existing LMMs. It also integrates well with existing LMMs, consistently improving performance.
Authors Zijian Song, Xiaoxin Lin, Tao Pu, Zhenlong Yuan, Guangrun Wang, Liang Lin
Comment accepted to AAAI 2026, 10 pages, 9 figures
Categories Computer Vision and Pattern Recognition
0
SatSAM2: Motion-Constrained Video Object Tracking in Satellite Imagery using Promptable SAM2 and Kalman Priors
Title CN SatSAM2：基于可提示SAM2与卡尔曼先验的运动约束卫星影像视频目标跟踪
Keywords 视频目标跟踪 · 卫星影像 · SAM2 · 卡尔曼滤波 · 零样本学习
Summary 本文提出SatSAM2，一种基于SAM2的零样本卫星视频目标跟踪方法，引入卡尔曼滤波约束运动模块和运动约束状态机以提升跟踪稳定性，并构建了一个名为MVOT的大规模合成基准用于评测。实验表明其在多个卫星跟踪数据集上优于现有方法。
Reason 该论文研究的是卫星影像中的视频目标跟踪，属于视频理解与遥感领域，与文档图像理解（DIU）任务无直接关联。尽管其使用了SAM2和运动建模等技术，但应用场景为遥感视频，不涉及DocVQA、布局分析、表格识别或OCR-free文档理解等核心问题，且未提出可迁移至文档领域的通用方法（如视觉压缩、去幻觉机制或多模态推理架构），因此属于无关领域。
Abstract Existing satellite video tracking methods often struggle with generalization, requiring scenario-specific training to achieve satisfactory performance, and are prone to track loss in the presence of occlusion. To address these challenges, we propose SatSAM2, a zero-shot satellite video tracker built on SAM2, designed to adapt foundation models to the remote sensing domain. SatSAM2 introduces two core modules: a Kalman Filter-based Constrained Motion Module (KFCMM) to exploit temporal motion cues and suppress drift, and a Motion-Constrained State Machine (MCSM) to regulate tracking states based on motion dynamics and reliability. To support large-scale evaluation, we propose MatrixCity Video Object Tracking (MVOT), a synthetic benchmark containing 1,500+ sequences and 157K annotated frames with diverse viewpoints, illumination, and occlusion conditions. Extensive experiments on two satellite tracking benchmarks and MVOT show that SatSAM2 outperforms both traditional and foundation model-based trackers, including SAM2 and its variants. Notably, on the OOTB dataset, SatSAM2 achieves a 5.84% AUC improvement over state-of-the-art methods. Our code and dataset will be publicly released to encourage further research.
Authors Ruijie Fan, Junyan Ye, Huan Chen, Zilong Huang, Xiaolei Wang, Weijia Li
Categories Computer Vision and Pattern Recognition
0
High Resolution UDF Meshing via Iterative Networks NeurIPS 2025
Title CN 通过迭代网络实现高分辨率UDF网格化
Keywords UDF · 3D重建 · 隐式表示 · 迭代网络 · 高分辨率网格化
Summary 本文提出一种基于迭代神经网络的方法，用于从高分辨率的无符号距离场（UDF）中提取更完整、准确的三维网格。通过多轮推理逐步传播邻域信息，该方法能在单个体素内结合远距离上下文，修正传统单次方法在噪声区域产生的表面缺失和孔洞问题，在复杂几何体上表现出优于现有方法的性能。
Reason 该论文研究的是三维几何中的无符号距离场（UDF）网格生成问题，属于3D视觉与隐式表面重建领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其核心方法虽涉及高分辨率处理和噪声鲁棒性，但针对的是3D形状重建中的拓扑完整性问题，难以迁移到文档图像中的文本识别、布局分析或VQA等任务。属于无关领域。
Abstract Unsigned Distance Fields (UDFs) are a natural implicit representation for open surfaces but, unlike Signed Distance Fields (SDFs), are challenging to triangulate into explicit meshes. This is especially true at high resolutions where neural UDFs exhibit higher noise levels, which makes it hard to capture fine details. Most current techniques perform within single voxels without reference to their neighborhood, resulting in missing surface and holes where the UDF is ambiguous or noisy. We show that this can be remedied by performing several passes and by reasoning on previously extracted surface elements to incorporate neighborhood information. Our key contribution is an iterative neural network that does this and progressively improves surface recovery within each voxel by spatially propagating information from increasingly distant neighbors. Unlike single-pass methods, our approach integrates newly detected surfaces, distance values, and gradients across multiple iterations, effectively correcting errors and stabilizing extraction in challenging regions. Experiments on diverse 3D models demonstrate that our method produces significantly more accurate and complete meshes than existing approaches, particularly for complex geometries, enabling UDF surface extraction at higher resolutions where traditional methods fail.
Authors Federico Stella, Nicolas Talabot, Hieu Le, Pascal Fua
Comment Accepted at NeurIPS 2025
Categories Computer Vision and Pattern Recognition
0
Optimizing DINOv2 with Registers for Face Anti-Spoofing ICCV 2025 Workshop
Title CN 使用寄存器优化DINOv2用于人脸反欺骗
Keywords 人脸反欺骗 · DINOv2 · 注意力机制 · 寄存器 · 活体检测
Summary 本文提出一种基于DINOv2并引入寄存器机制的人脸反欺骗方法，旨在通过增强模型对真实人脸与照片攻击之间细微差别的感知能力来提升检测性能。实验在ICCV 2025 FAS Workshop提供的数据集和SiW数据集上进行验证。
Reason 该论文研究的是人脸反欺骗（Face Anti-Spoofing）任务，属于生物特征安全领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管其使用了DINOv2和注意力机制改进，但方法针对的是活体检测中的细微纹理差异，无法迁移解决OCR幻觉、复杂排版理解、长文档推理等DIU核心问题。
Abstract Face recognition systems are designed to be robust against variations in head pose, illumination, and image blur during capture. However, malicious actors can exploit these systems by presenting a face photo of a registered user, potentially bypassing the authentication process. Such spoofing attacks must be detected prior to face recognition. In this paper, we propose a DINOv2-based spoofing attack detection method to discern minute differences between live and spoofed face images. Specifically, we employ DINOv2 with registers to extract generalizable features and to suppress perturbations in the attention mechanism, which enables focused attention on essential and minute features. We demonstrate the effectiveness of the proposed method through experiments conducted on the dataset provided by ``The 6th Face Anti-Spoofing Workshop: Unified Physical-Digital Attacks Detection@ICCV2025'' and SiW dataset. The project page is available at: https://gsisaoki.github.io/FAS-DINOv2-ICCVW/ .
Authors Mika Feng, Pierre Gallin-Martel, Koichi Ito, Takafumi Aoki
Comment ICCV 2025 Workshop FAS
Categories Computer Vision and Pattern Recognition
0
Coupled Physics-Gated Adaptation: Spatially Decoding Volumetric Photochemical Conversion in Complex 3D-Printed Objects
Title CN 耦合物理门控适应：在复杂3D打印物体中空间解码体相光化学转化
Keywords 3D视觉 · 物理建模 · 多模态融合 · C-PGA · 光化学转化
Summary 本文提出了一种名为Coupled Physics-Gated Adaptation（C-PGA）的多模态融合架构，用于从复杂的3D打印物体的视觉数据中预测其内部光化学转化的分布。该方法利用几何与工艺参数作为查询，通过FiLM机制动态调节双路3D-CNN提取的视觉特征，以融入光学与材料物理的耦合先验。实验基于大规模3D打印样本数据集，实现了对化学状态的虚拟表征。
Reason 该论文研究的是从3D视觉数据预测光化学转化的物理属性，属于3D视觉与物理建模交叉领域，任务核心为体相物理性质预测，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务完全无关。尽管使用了多模态架构（C-PGA），但其动机和方法均面向光学物理与材料科学建模，无法迁移至解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心痛点。
Abstract We present a framework that pioneers the prediction of photochemical conversion in complex three-dimensionally printed objects, introducing a challenging new computer vision task: predicting dense, non-visual volumetric physical properties from 3D visual data. This approach leverages the largest-ever optically printed 3D specimen dataset, comprising a large family of parametrically designed complex minimal surface structures that have undergone terminal chemical characterisation. Conventional vision models are ill-equipped for this task, as they lack an inductive bias for the coupled, non-linear interactions of optical physics (diffraction, absorption) and material physics (diffusion, convection) that govern the final chemical state. To address this, we propose Coupled Physics-Gated Adaptation (C-PGA), a novel multimodal fusion architecture. Unlike standard concatenation, C-PGA explicitly models physical coupling by using sparse geometrical and process parameters (e.g., surface transport, print layer height) as a Query to dynamically gate and adapt the dense visual features via feature-wise linear modulation (FiLM). This mechanism spatially modulates dual 3D visual streams-extracted by parallel 3D-CNNs processing raw projection stacks and their diffusion-diffraction corrected counterparts allowing the model to recalibrate its visual perception based on the physical context. This approach offers a breakthrough in virtual chemical characterisation, eliminating the need for traditional post-print measurements and enabling precise control over the chemical conversion state.
Authors Maryam Eftekharifar, Churun Zhang, Jialiang Wei, Xudong Cao, Hossein Heidari
Categories Computer Vision and Pattern Recognition
0
Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving
Title CN Reasoning-VLA：一种用于自动驾驶的快速通用视觉-语言-动作推理模型
Keywords VLA · 自动驾驶 · 链式思维推理 · 动作生成 · 多数据集整合
Summary 本文提出Reasoning-VLA，一种面向自动驾驶的快速通用视觉-语言-动作推理模型，通过可学习的动作查询与增强的视觉语言特征交互，实现并行化的连续动作轨迹生成。作者整合了八个公开驾驶数据集，采用链式思维推理格式进行训练，并结合监督学习与强化学习微调，在多个基准上实现了最先进的性能、良好的泛化能力和最快的推理速度。
Reason 该论文属于自动驾驶领域，研究的是视觉-语言-动作（VLA）模型在车辆决策中的应用，与文档图像理解（DIU）或DocVQA等任务无直接关联。尽管其提出的推理增强机制和多数据集整合思路具有一定方法论价值，但整体工作聚焦于连续动作生成与驾驶场景泛化，迁移到文档理解领域的可行性极低，且存在领域错位。根据负面清单，自动驾驶属于明确排除的无关领域。
Abstract Vision-Language-Action (VLA) models have recently shown strong decision-making capabilities in autonomous driving. However, existing VLAs often struggle with achieving efficient inference and generalizing to novel autonomous vehicle configurations and driving scenarios. In this paper, we propose Reasoning-VLA, a general and fast action-generation VLA framework. The proposed model employs a set of learnable action queries, initialized via Gaussian sampling from ground-truth trajectories within the training corpus. These learnable queries interact with reasoning-enhanced vision-language features to generate continuous action trajectories in parallel. To promote robust generalization, we consolidate eight publicly available autonomous driving datasets into a standardized, Chain-of-Thought reasoning-based, and easy-to-use data format for model training. Leveraging both supervised learning and reinforcement learning fine-tuning, extensive empirical evaluations across multiple benchmarks demonstrate that Reasoning-VLA achieves state-of-the-art performance, superior generalization capability, and the excellent inference speed reported to date.
Authors Dapeng Zhang, Zhenlong Yuan, Zhangquan Chen, Chih-Ting Liao, Yinda Chen, Fei Shen, Qingguo Zhou, Tat-Seng Chua
Categories Computer Vision and Pattern Recognition
0
DLADiff: A Dual-Layer Defense Framework against Fine-Tuning and Zero-Shot Customization of Diffusion Models
Title CN DLADiff：一种针对扩散模型微调和零样本定制的双层防御框架
Keywords 扩散模型 · 隐私保护 · 对抗攻击 · 零样本生成 · 微调防御
Summary 本文提出了一种名为DLADiff的双层防御框架，旨在抵御基于少量图像甚至单张图像的扩散模型定制化攻击，包括微调和零样本生成两种方式。第一层通过双代理模型和交替动态微调结合对抗训练进行防护，第二层则设计简单但有效的机制来阻断零样本生成。实验表明该方法在防御效果上优于现有方案。
Reason 该论文研究的是扩散模型在图像生成中的隐私保护问题，属于纯图像生成与安全领域，主要应对人脸隐私泄露风险。其核心方法围绕对抗扩散模型的微调和零样本生成展开，与文档图像理解（DIU）、DocVQA、OCR、布局分析等任务完全无关。尽管涉及视觉内容，但既不解决文档结构、文本识别、多模态对齐或复杂排版等问题，也无法迁移用于缓解OCR幻觉、长文档推理等DIU核心痛点。
Abstract With the rapid advancement of diffusion models, a variety of fine-tuning methods have been developed, enabling high-fidelity image generation with high similarity to the target content using only 3 to 5 training images. More recently, zero-shot generation methods have emerged, capable of producing highly realistic outputs from a single reference image without altering model weights. However, technological advancements have also introduced significant risks to facial privacy. Malicious actors can exploit diffusion model customization with just a few or even one image of a person to create synthetic identities nearly identical to the original identity. Although research has begun to focus on defending against diffusion model customization, most existing defense methods target fine-tuning approaches and neglect zero-shot generation defenses. To address this issue, this paper proposes Dual-Layer Anti-Diffusion (DLADiff) to defense both fine-tuning methods and zero-shot methods. DLADiff contains a dual-layer protective mechanism. The first layer provides effective protection against unauthorized fine-tuning by leveraging the proposed Dual-Surrogate Models (DSUR) mechanism and Alternating Dynamic Fine-Tuning (ADFT), which integrates adversarial training with the prior knowledge derived from pre-fine-tuned models. The second layer, though simple in design, demonstrates strong effectiveness in preventing image generation through zero-shot methods. Extensive experimental results demonstrate that our method significantly outperforms existing approaches in defending against fine-tuning of diffusion models and achieves unprecedented performance in protecting against zero-shot generation.
Authors Jun Jia, Hongyi Miao, Yingjie Zhou, Linhan Cao, Yanwei Jiang, Wangqiu Zhou, Dandan Zhu, Hua Yang, Wei Sun, Xiongkuo Min, Guangtao Zhai
Categories Computer Vision and Pattern Recognition
0
Motion Marionette: Rethinking Rigid Motion Transfer via Prior Guidance
Title CN 运动木偶：通过先验引导重新思考刚性运动迁移
Keywords 运动迁移 · 时空先验 · 3D表示学习 · 视频生成 · 位置动力学
Summary 本文提出Motion Marionette，一种用于将单目源视频中的刚性运动迁移到目标图像的零样本框架。该方法通过构建一个独立于物体几何和语义的时空（SpaT）先验来指导运动迁移，并利用位置动力学优化生成结果，在跨对象泛化和时间一致性方面表现良好。
Reason 该论文研究的是从单目视频到单视图图像的刚性运动迁移，属于视频生成与动作迁移领域，核心任务为动态视觉内容合成，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法聚焦于3D表示空间中的运动轨迹提取与速度场建模，主要用于可控视频生成，无法迁移到解决OCR幻觉、复杂排版解析、细粒度文本定位或长文档推理等DIU核心痛点。
Abstract We present Motion Marionette, a zero-shot framework for rigid motion transfer from monocular source videos to single-view target images. Previous works typically employ geometric, generative, or simulation priors to guide the transfer process, but these external priors introduce auxiliary constraints that lead to trade-offs between generalizability and temporal consistency. To address these limitations, we propose guiding the motion transfer process through an internal prior that exclusively captures the spatial-temporal transformations and is shared between the source video and any transferred target video. Specifically, we first lift both the source video and the target image into a unified 3D representation space. Motion trajectories are then extracted from the source video to construct a spatial-temporal (SpaT) prior that is independent of object geometry and semantics, encoding relative spatial variations over time. This prior is further integrated with the target object to synthesize a controllable velocity field, which is subsequently refined using Position-Based Dynamics to mitigate artifacts and enhance visual coherence. The resulting velocity field can be flexibly employed for efficient video production. Empirical results demonstrate that Motion Marionette generalizes across diverse objects, produces temporally consistent videos that align well with the source motion, and supports controllable video generation.
Authors Haoxuan Wang, Jiachen Tao, Junyi Wu, Gaowen Liu, Ramana Rao Kompella, Yan Yan
Categories Computer Vision and Pattern Recognition
0
MHB: Multimodal Handshape-aware Boundary Detection for Continuous Sign Language Recognition
Title CN MHB：面向连续手语识别的多模态手形感知边界检测
Keywords 手语识别 · 边界检测 · 多模态融合 · 3D骨骼 · 手形分类
Summary 本文提出一种用于连续手语识别的多模态方法MHB，通过结合3D骨骼动态特征和预训练的手形分类模型来检测手势的起始与结束帧。利用手形在语言学上定义的87类标准类别进行预训练，并通过多模态融合模块整合视频分割框架与手形信息，提升边界检测鲁棒性，最终用于手语识别。实验在ASLLRP语料库上进行，结果优于先前方法。
Reason 该论文研究的是连续手语视频中的手势边界检测与识别，属于视频理解领域，核心任务为动作分割与手语识别，与文档图像理解（DIU）或DocVQA无关。其技术路线聚焦于3D骨骼特征和手形分类的多模态融合，应用于ASL句子的分段与识别，无法迁移至文档图像中的OCR去幻觉、复杂排版解析或长文档推理等核心痛点。
Abstract This paper presents a multimodal approach for continuous sign recognition that first uses machine learning to detect the start and end frames of signs in videos of American Sign Language (ASL) sentences, and then recognizes the segmented signs. For improved robustness, we use 3D skeletal features extracted from sign language videos to capture the convergence of sign properties and their dynamics, which tend to cluster at sign boundaries. Another focus of this work is the incorporation of information from 3D handshape for boundary detection. To detect handshapes normally expected at the beginning and end of signs, we pretrain a handshape classifier for 87 linguistically defined canonical handshape categories using a dataset that we created by integrating and normalizing several existing datasets. A multimodal fusion module is then used to unify the pretrained sign video segmentation framework and the handshape classification models. Finally, the estimated boundaries are used for sign recognition, where the recognition model is trained on a large database containing both citation-form isolated signs and signs pre-segmented (based on manual annotations) from continuous signing, as such signs often differ in certain respects. We evaluate our method on the ASLLRP corpus and demonstrate significant improvements over previous work.
Authors Mingyu Zhao, Zhanfu Yang, Yang Zhou, Zhaoyang Xia, Can Jin, Xiaoxiao He, Carol Neidle, Dimitris N. Metaxas
Categories Computer Vision and Pattern Recognition
0
DBGroup: Dual-Branch Point Grouping for Weakly Supervised 3D Semantic Instance Segmentation
Title CN DBGroup：用于弱监督3D语义实例分割的双分支点云分组
Keywords 3D实例分割 · 弱监督学习 · 点云分组 · 伪标签 · 自训练
Summary 本文提出DBGroup，一种基于双分支点云分组的两阶段弱监督3D实例分割框架，利用场景级标注生成伪标签，并通过粒度感知实例合并、语义选择与传播等策略优化标签质量，再结合自训练提升性能。实验表明其在减少人工标注的同时达到先进水平。
Reason 该论文研究的是3D点云语义实例分割，属于具身智能与3D视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法聚焦于从多视角图像生成伪标签并进行自训练，解决的是3D场景下的弱监督学习问题，无法迁移到文档图像中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Weakly supervised 3D instance segmentation is essential for 3D scene understanding, especially as the growing scale of data and high annotation costs associated with fully supervised approaches. Existing methods primarily rely on two forms of weak supervision: one-thing-one-click annotations and bounding box annotations, both of which aim to reduce labeling efforts. However, these approaches still encounter limitations, including labor-intensive annotation processes, high complexity, and reliance on expert annotators. To address these challenges, we propose \textbf{DBGroup}, a two-stage weakly supervised 3D instance segmentation framework that leverages scene-level annotations as a more efficient and scalable alternative. In the first stage, we introduce a Dual-Branch Point Grouping module to generate pseudo labels guided by semantic and mask cues extracted from multi-view images. To further improve label quality, we develop two refinement strategies: Granularity-Aware Instance Merging and Semantic Selection and Propagation. The second stage involves multi-round self-training on an end-to-end instance segmentation network using the refined pseudo-labels. Additionally, we introduce an Instance Mask Filter strategy to address inconsistencies within the pseudo labels. Extensive experiments demonstrate that DBGroup achieves competitive performance compared to sparse-point-level supervised 3D instance segmentation methods, while surpassing state-of-the-art scene-level supervised 3D semantic segmentation approaches. Code is available at https://github.com/liuxuexun/DBGroup.
Authors Xuexun Liu, Xiaoxu Xu, Qiudan Zhang, Lin Ma, Xu Wang
Categories Computer Vision and Pattern Recognition
0
LiMT: A Multi-task Liver Image Benchmark Dataset IEEE Journal of Biomedical and Health Informatics
Title CN LiMT：一个多任务肝脏图像基准数据集
Keywords 医学影像 · 多任务学习 · 数据集 · 肝脏分割 · 病变检测
Summary 本文提出了一个名为LiMT的多任务肝脏图像数据集，基于动脉期增强CT，支持肝脏与肿瘤分割、多标签病变分类及病变检测。数据集包含150例病例，涵盖四种肝病类型及正常情况，均由临床专家标注。作者提供了基线实验并对现有方法进行了综述，旨在推动计算机辅助诊断（CAD）在肝脏疾病中的发展。
Reason 该论文属于医学影像领域的特定应用（肝脏病变分析），聚焦于构建一个用于肝和肿瘤分割、病变分类与检测的专用CT数据集。其内容为典型的‘平行下游应用’，未提出可迁移至文档图像理解（DIU）的方法论创新，且完全不涉及OCR、VLM或多模态文档推理等核心技术痛点。
Abstract Computer-aided diagnosis (CAD) technology can assist clinicians in evaluating liver lesions and intervening with treatment in time. Although CAD technology has advanced in recent years, the application scope of existing datasets remains relatively limited, typically supporting only single tasks, which has somewhat constrained the development of CAD technology. To address the above limitation, in this paper, we construct a multi-task liver dataset (LiMT) used for liver and tumor segmentation, multi-label lesion classification, and lesion detection based on arterial phase-enhanced computed tomography (CT), potentially providing an exploratory solution that is able to explore the correlation between tasks and does not need to worry about the heterogeneity between task-specific datasets during training. The dataset includes CT volumes from 150 different cases, comprising four types of liver diseases as well as normal cases. Each volume has been carefully annotated and calibrated by experienced clinicians. This public multi-task dataset may become a valuable resource for the medical imaging research community in the future. In addition, this paper not only provides relevant baseline experimental results but also reviews existing datasets and methods related to liver-related tasks. Our dataset is available at https://drive.google.com/drive/folders/1l9HRK13uaOQTNShf5pwgSz3OTanWjkag?usp=sharing.
Authors Zhe Liu, Kai Han, Siqi Ma, Yan Zhu, Jun Chen, Chongwen Lyu, Xinyi Qiu, Chengxuan Qian, Yuqing Song, Yi Liu, Liyuan Tian, Yang Ji, Yuefeng Li
Comment IEEE Journal of Biomedical and Health Informatics
Categories Computer Vision and Pattern Recognition
0
Frequency Bias Matters: Diving into Robust and Generalized Deep Image Forgery Detection IEEE Transactions on Dependable and Secure Computing
Title CN 频率偏差很重要：深入研究鲁棒且通用的深度图像伪造检测
Keywords 图像伪造检测 · 频率偏差 · 鲁棒性 · 泛化性 · 深度伪造
Summary 本文从频域角度分析深度图像伪造检测中的泛化性和鲁棒性问题，指出DNN检测器的频率偏差是导致性能下降的潜在原因，并提出一种两步频率对齐方法来统一提升检测器在面对未知生成模型和噪声时的可靠性。该方法在多种伪造模型和检测器上验证有效。
Reason 该论文属于图像伪造检测领域，主要关注AI生成图像（如GAN）的真伪辨别问题，与文档图像理解（DIU）的核心任务（如DocVQA、布局分析、表格识别等）无直接关联。尽管其方法涉及频率域分析和模型鲁棒性，但无法迁移解决OCR幻觉、密集文本识别、复杂排版或长文档推理等DIU关键痛点。
Abstract As deep image forgery powered by AI generative models, such as GANs, continues to challenge today's digital world, detecting AI-generated forgeries has become a vital security topic. Generalizability and robustness are two critical concerns of a forgery detector, determining its reliability when facing unknown GANs and noisy samples in an open world. Although many studies focus on improving these two properties, the root causes of these problems have not been fully explored, and it is unclear if there is a connection between them. Moreover, despite recent achievements in addressing these issues from image forensic or anti-forensic aspects, a universal method that can contribute to both sides simultaneously remains practically significant yet unavailable. In this paper, we provide a fundamental explanation of these problems from a frequency perspective. Our analysis reveals that the frequency bias of a DNN forgery detector is a possible cause of generalization and robustness issues. Based on this finding, we propose a two-step frequency alignment method to remove the frequency discrepancy between real and fake images, offering double-sided benefits: it can serve as a strong black-box attack against forgery detectors in the anti-forensic context or, conversely, as a universal defense to improve detector reliability in the forensic context. We also develop corresponding attack and defense implementations and demonstrate their effectiveness, as well as the effect of the frequency alignment method, in various experimental settings involving twelve detectors, eight forgery models, and five metrics.
Authors Chi Liu, Tianqing Zhu, Wanlei Zhou, Wei Zhao
Comment Accepted for publication in IEEE Transactions on Dependable and Secure Computing
Categories Computer Vision and Pattern Recognition
0
ChessMamba: Structure-Aware Interleaving of State Spaces for Change Detection in Remote Sensing Images
Title CN ChessMamba：用于遥感图像变化检测的状态空间结构感知交错方法
Keywords 变化检测 · 遥感图像 · 状态空间模型 · 棋盘交错 · 结构感知融合
Summary 本文提出ChessMamba，一种用于多时相遥感图像变化检测的结构感知框架。该方法通过棋盘交错与蛇形扫描策略将多时相特征序列化为统一序列，并结合多空洞卷积进行局部上下文建模，以增强在时空错位情况下的变化定位能力。实验表明其在二值变化检测、语义变化检测和跨模态建筑损毁评估任务上优于现有方法。
Reason 该论文研究的是遥感图像中的变化检测任务，属于特定领域的计算机视觉应用，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的‘棋盘式交错’和‘结构感知融合’在远程感知任务中有效，但该方法针对的是多时相遥感影像的异质性和时空错位问题，迁移到文档图像的理解场景（如处理OCR幻觉、复杂排版或长文档推理）极为困难，且未体现对文本结构、细粒度文字识别或多模态对齐的改进潜力。
Abstract Change detection (CD) in multitemporal remote sensing imagery presents significant challenges for fine-grained recognition, owing to heterogeneity and spatiotemporal misalignment. However, existing methodologies based on vision transformers or state-space models typically disrupt local structural consistency during temporal serialization, obscuring discriminative cues under misalignment and hindering reliable change localization. To address this, we introduce ChessMamba, a structure-aware framework leveraging interleaved state-space modeling for robust CD with multi-temporal inputs. ChessMamba integrates a SpatialMamba encoder with a lightweight cross-source interaction module, featuring two key innovations: (i) Chessboard interleaving with snake scanning order, which serializes multi-temporal features into a unified sequence within a single forward pass, thereby shortening interaction paths and enabling direct comparison for accurate change localization; and (ii) Structure-aware fusion via multi-dilated convolutions, selectively capturing center-and-corner neighborhood contexts within each mono-temporal. Comprehensive evaluations on three CD tasks, including binary CD, semantic CD and multimodal building damage assessment, demonstrate that ChessMamba effectively fuses heterogeneous features and achieves substantial accuracy improvements over state-of-the-art methods.The relevant code will be available at: github.com/DingLei14/ChessMamba.
Authors Lei Ding, Tong Liu, Xuanguang Liu, Xiangyun Liu, Haitao Guo, Jun Lu
Categories Computer Vision and Pattern Recognition
0
It Hears, It Sees too: Multi-Modal LLM for Depression Detection By Integrating Visual Understanding into Audio Language Models
Title CN 它能听，也能看：通过将视觉理解融入音频语言模型实现多模态大模型抑郁检测
Keywords 多模态大模型 · 抑郁症检测 · 音频语言模型 · 视觉理解 · 跨模态对齐
Summary 本文提出一种结合视觉理解能力的多模态大模型框架，用于抑郁症检测。通过在时间戳级别对齐音频与视觉特征，增强非语言行为建模，提升抑郁识别性能。实验基于DAIC-WoZ数据集，验证了方法的有效性。
Reason 该论文属于特定领域的下游应用（心理健康评估），其核心是将多模态LLM应用于抑郁症检测这一垂直场景，未提出可用于文档图像理解的通用方法论创新。尽管涉及多模态融合，但其技术路线聚焦于时间序列对齐和临床应用，与文档理解中的OCR幻觉、复杂版式、细粒度定位等核心痛点无关。
Abstract Depression is one of the most prevalent mental health disorders globally. In recent years, multi-modal data, such as speech, video, and transcripts, has been increasingly used to develop AI-assisted depression assessment systems. Large language models have further advanced this field due to their strong language understanding and generalization capabilities. However, conventional LLMs remain text-centric and cannot process the rich non-verbal cues found in audio and visual modalities, which are critical components in mental health evaluation. While multi-modal LLMs offer a promising direction, few are tailored for psychological applications. In this study, we propose a novel multi-modal LLM framework for depression detection. Our approach augments an audio language model with visual understanding and aligns audio-visual features at the timestamp level. This fine-grained alignment improves modeling of temporal dynamics across modalities while reducing the need for extensive training data and computational resources. Experiments on the DAIC-WoZ dataset demonstrate that our model outperforms both single-modality approaches and previous multi-modal methods. Moreover, the proposed framework can be extended to incorporate additional physiological signals, paving the way for broader clinical applications beyond mental health.
Authors Xiangyu Zhao, Yaling Shen, Yiwen Jiang, Zimu Wang, Jiahe Liu, Maxmartwell H Cheng, Guilherme C Oliveira, Robert Desimone, Dominic Dwyer, Zongyuan Ge
Categories Computer Vision and Pattern Recognition
0
Video Understanding with Large Language Models: A Survey IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)
Title CN 使用大语言模型进行视频理解：综述
Keywords 视频理解 · 大语言模型 · 多模态学习 · 综述 · Vid-LLM
Summary 这是一篇关于利用大语言模型进行视频理解的综述论文，系统梳理了视频分析中LLM的集成方法、功能角色、任务类型及评估体系。重点涵盖视频的多粒度推理能力，并讨论现有方法的局限性与未来方向。
Reason 该论文属于视频理解领域，与文档图像理解（DIU）核心任务无直接关联。尽管其涉及多模态大模型（VLM）的应用，但研究场景聚焦于视频的时空推理与开放域理解，无法迁移解决文档中的OCR幻觉、密集文本识别、复杂版式分析或长文档结构化等关键痛点。
Abstract With the burgeoning growth of online video platforms and the escalating volume of video content, the demand for proficient video understanding tools has intensified markedly. Given the remarkable capabilities of large language models (LLMs) in language and multimodal tasks, this survey provides a detailed overview of recent advancements in video understanding that harness the power of LLMs (Vid-LLMs). The emergent capabilities of Vid-LLMs are surprisingly advanced, particularly their ability for open-ended multi-granularity (general, temporal, and spatiotemporal) reasoning combined with commonsense knowledge, suggesting a promising path for future video understanding. We examine the unique characteristics and capabilities of Vid-LLMs, categorizing the approaches into three main types: Video Analyzer x LLM, Video Embedder x LLM, and (Analyzer + Embedder) x LLM. Furthermore, we identify five sub-types based on the functions of LLMs in Vid-LLMs: LLM as Summarizer, LLM as Manager, LLM as Text Decoder, LLM as Regressor, and LLM as Hidden Layer. Furthermore, this survey presents a comprehensive study of the tasks, datasets, benchmarks, and evaluation methodologies for Vid-LLMs. Additionally, it explores the expansive applications of Vid-LLMs across various domains, highlighting their remarkable scalability and versatility in real-world video understanding challenges. Finally, it summarizes the limitations of existing Vid-LLMs and outlines directions for future research. For more information, readers are recommended to visit the repository at https://github.com/yunlong10/Awesome-LLMs-for-Video-Understanding.
Authors Yolo Y. Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, Ali Vosoughi, Chao Huang, Zeliang Zhang, Pinxin Liu, Mingqian Feng, Feng Zheng, Jianguo Zhang, Ping Luo, Jiebo Luo, Chenliang Xu
Comment Accepted to IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)
Categories Computer Vision and Pattern Recognition, Computation and Language
0
Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Models
Title CN 视频大多少模态模型后训练：深入探究视频推理
Keywords 视频理解 · 大多少模态模型 · 后训练 · 链式思维 · 测试时扩展
Summary 本文综述了视频大多少模态模型（Video-LMM）在后训练阶段的技术进展，涵盖监督微调（SFT）、基于可验证目标的强化学习（RL）以及测试时计算扩展（TTS），提出了一个系统性的分类体系，并分析了现有方法在时间定位、时空接地、长视频效率等方面的适应性。同时总结了关键设计原则、评估协议与开放问题，并整理了相关基准资源。
Reason 该论文属于视频理解领域，尽管涉及VLM的推理增强技术（如CoT、RL、测试时扩展），但其核心任务和方法均针对视频中的时空关系建模，与文档图像理解（DIU）在输入模态、结构特性和核心挑战上差异巨大。文档图像强调高分辨率静态布局、细粒度文本识别与逻辑结构解析，而视频LMM的后训练策略难以直接迁移解决OCR幻觉、密集文本或复杂排版等DIU痛点，不具备可迁移的底层方法创新。
Abstract Video understanding represents the most challenging frontier in computer vision, requiring models to reason about complex spatiotemporal relationships, long-term dependencies, and multimodal evidence. The recent emergence of Video-Large Multimodal Models (Video-LMMs), which integrate visual encoders with powerful decoder-based language models, has demonstrated remarkable capabilities in video understanding tasks. However, the critical phase that transforms these models from basic perception systems into sophisticated reasoning engines, post-training, remains fragmented across the literature. This survey provides the first comprehensive examination of post-training methodologies for Video-LMMs, encompassing three fundamental pillars: supervised fine-tuning (SFT) with chain-of-thought, reinforcement learning (RL) from verifiable objectives, and test-time scaling (TTS) through enhanced inference computation. We present a structured taxonomy that clarifies the roles, interconnections, and video-specific adaptations of these techniques, addressing unique challenges such as temporal localization, spatiotemporal grounding, long video efficiency, and multimodal evidence integration. Through systematic analysis of representative methods, we synthesize key design principles, insights, and evaluation protocols while identifying critical open challenges in reward design, scalability, and cost-performance optimization. We further curate essential benchmarks, datasets, and metrics to facilitate rigorous assessment of post-training effectiveness. This survey aims to provide researchers and practitioners with a unified framework for advancing Video-LMM capabilities. Additional resources and updates are maintained at: https://github.com/yunlong10/Awesome-Video-LMM-Post-Training
Authors Yolo Y. Tang, Jing Bi, Pinxin Liu, Zhenyu Pan, Zhangyun Tan, Qianxiang Shen, Jiani Liu, Hang Hua, Junjia Guo, Yunzhong Xiao, Chao Huang, Zhiyuan Wang, Susan Liang, Xinyi Liu, Yizhi Song, Junhua Huang, Jia-Xing Zhong, Bozheng Li, Daiqing Qi, Ziyun Zeng, Ali Vosoughi, Luchuan Song, Zeliang Zhang, Daiki Shimada, Han Liu, Jiebo Luo, Chenliang Xu
Comment Version v1.1
Categories Computer Vision and Pattern Recognition
0
Rethinking Two-Stage Referring-by-Tracking in Referring Multi-Object Tracking: Make it Strong Again
Title CN 重新思考指代表跟踪在指代多目标跟踪中的应用：使其再次强大
Keywords 指代多目标跟踪 · RBT · 视频理解 · 多模态对齐 · 条件特征注入
Summary 本文提出了一种名为FlexHook的新两阶段指代表跟踪（RBT）框架，用于指代多目标跟踪任务。通过引入基于采样的Conditioning Hook（C-Hook）和成对对应解码器（PCD），解决了传统两阶段方法中特征构造启发式、对应建模脆弱的问题，在多个RMOT基准上实现了超越现有SOTA的性能。
Reason 该论文研究的是视频领域的指代多目标跟踪（RMOT），属于视频理解任务，核心方法聚焦于语言条件下的目标跟踪和对应关系建模。尽管涉及多模态对齐与语言引导的定位，但其技术路线针对的是动态视频中的运动对象，与文档图像理解（DIU）中的静态文本识别、布局分析、表格结构提取等任务差异巨大。无法迁移解决OCR幻觉、密集文本处理或复杂排版理解等DIU核心痛点。
Abstract Referring Multi-Object Tracking (RMOT) aims to track multiple objects specified by natural language expressions in videos. With the recent significant progress of one-stage methods, the two-stage Referring-by-Tracking (RBT) paradigm has gradually lost its popularity. However, its lower training cost and flexible incremental deployment remain irreplaceable. Rethinking existing two-stage RBT frameworks, we identify two fundamental limitations: the overly heuristic feature construction and fragile correspondence modeling. To address these issues, we propose FlexHook, a novel two-stage RBT framework. In FlexHook, the proposed Conditioning Hook (C-Hook) redefines the feature construction by a sampling-based strategy and language-conditioned cue injection. Then, we introduce a Pairwise Correspondence Decoder (PCD) that replaces CLIP-based similarity matching with active correspondence modeling, yielding a more flexible and robust strategy. Extensive experiments on multiple benchmarks (Refer-KITTI/v2, Refer-Dance, and LaMOT) demonstrate that FlexHook becomes the first two-stage RBT approach to comprehensively outperform current state-of-the-art methods. Code can be found in the Supplementary Materials.
Authors Weize Li, Yunhao Du, Qixiang Yin, Zhicheng Zhao, Fei Su
Categories Computer Vision and Pattern Recognition
0
VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation
Title CN VisionReward：用于图像和视频生成的细粒度多维人类偏好学习
Keywords 视觉生成 · 奖励模型 · 人类偏好 · 视频生成 · 可解释性
Summary 本文提出了VisionReward，一个用于图像和视频生成中人类视觉偏好学习的通用框架。通过分层视觉评估体系捕捉细粒度偏好，并采用线性加权实现可解释的偏好学习。实验表明其在机器指标和人工评测上均显著优于现有奖励模型，尤其在视频生成中胜率提升明显。
Reason 该论文聚焦于图像和视频生成领域的人类偏好对齐问题，属于纯视觉生成方向，与文档图像理解（DIU）的核心任务如DocVQA、OCR、版面分析等无直接关联。其提出的方法针对生成质量优化，而非解决文档理解中的OCR幻觉、复杂排版或长文档推理等痛点，迁移价值极低。
Abstract Visual generative models have achieved remarkable progress in synthesizing photorealistic images and videos, yet aligning their outputs with human preferences across critical dimensions remains a persistent challenge. Though reinforcement learning from human feedback offers promise for preference alignment, existing reward models for visual generation face limitations, including black-box scoring without interpretability and potentially resultant unexpected biases. We present VisionReward, a general framework for learning human visual preferences in both image and video generation. Specifically, we employ a hierarchical visual assessment framework to capture fine-grained human preferences, and leverages linear weighting to enable interpretable preference learning. Furthermore, we propose a multi-dimensional consistent strategy when using VisionReward as a reward model during preference optimization for visual generation. Experiments show that VisionReward can significantly outperform existing image and video reward models on both machine metrics and human evaluation. Notably, VisionReward surpasses VideoScore by 17.2% in preference prediction accuracy, and text-to-video models with VisionReward achieve a 31.6% higher pairwise win rate compared to the same models using VideoScore. All code and datasets are provided at https://github.com/THUDM/VisionReward.
Authors Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, Jiayan Teng, Zhuoyi Yang, Wendi Zheng, Xiao Liu, Dan Zhang, Ming Ding, Xiaohan Zhang, Xiaotao Gu, Shiyu Huang, Minlie Huang, Jie Tang, Yuxiao Dong
Comment 27 pages
Categories Computer Vision and Pattern Recognition
0
GigaWorld-0: World Models as Data Engine to Empower Embodied AI
Title CN GigaWorld-0：将世界模型作为数据引擎以赋能具身智能
Keywords 世界模型 · 具身智能 · 视频生成 · 3D生成 · 数据引擎
Summary 本文提出了GigaWorld-0，一个用于生成具身智能训练数据的世界模型框架，包含视频生成和3D建模两个模块，能够合成高质量、可控的视觉-动作序列数据，并通过高效训练框架支持大规模训练。实验证明其生成的数据可使机器人策略模型在真实环境中取得良好表现。
Reason 该论文聚焦于具身智能（Embodied AI）和视觉-语言-动作（VLA）学习，核心是通过生成视频和3D场景来合成机器人交互数据。其技术路线完全围绕物理世界中的动作执行与环境建模展开，与文档图像理解（DIU）的核心任务如文本识别、布局分析、表格理解等无直接关联。尽管涉及多模态建模，但目标场景、评估指标和应用范式均远离DIU领域，无法迁移解决OCR幻觉、密集文本处理或复杂排版理解等问题。
Abstract World models are emerging as a foundational paradigm for scalable, data-efficient embodied AI. In this work, we present GigaWorld-0, a unified world model framework designed explicitly as a data engine for Vision-Language-Action (VLA) learning. GigaWorld-0 integrates two synergistic components: GigaWorld-0-Video, which leverages large-scale video generation to produce diverse, texture-rich, and temporally coherent embodied sequences under fine-grained control of appearance, camera viewpoint, and action semantics; and GigaWorld-0-3D, which combines 3D generative modeling, 3D Gaussian Splatting reconstruction, physically differentiable system identification, and executable motion planning to ensure geometric consistency and physical realism. Their joint optimization enables the scalable synthesis of embodied interaction data that is visually compelling, spatially coherent, physically plausible, and instruction-aligned. Training at scale is made feasible through our efficient GigaTrain framework, which exploits FP8-precision and sparse attention to drastically reduce memory and compute requirements. We conduct comprehensive evaluations showing that GigaWorld-0 generates high-quality, diverse, and controllable data across multiple dimensions. Critically, VLA model (e.g., GigaBrain-0) trained on GigaWorld-0-generated data achieve strong real-world performance, significantly improving generalization and task success on physical robots without any real-world interaction during training.
Authors GigaWorld Team, Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Haoyun Li, Jiagang Zhu, Kerui Li, Mengyuan Xu, Qiuping Deng, Siting Wang, Wenkang Qin, Xinze Chen, Xiaofeng Wang, Yankai Wang, Yu Cao, Yifan Chang, Yuan Xu, Yun Ye, Yang Wang, Yukun Zhou, Zhengyuan Zhang, Zhehao Dong, Zheng Zhu
Comment Project Page: https://gigaworld0.github.io/
Categories Computer Vision and Pattern Recognition
0
Zero-Shot Video Translation via Token Warping
Title CN 基于Token形变的零样本视频翻译
Keywords 视频翻译 · 扩散模型 · 自注意力 · 光流 · Token Warping
Summary 本文提出TokenWarping，一种用于时序一致的视频翻译框架。通过从源视频提取光流，并在扩散模型去噪过程中对前一帧的query、key和value token进行形变对齐，从而增强自注意力机制中的特征聚合并保证帧间一致性。该方法无需额外训练，可集成到现有图像编辑方法中，在多个视频翻译任务上优于当前方法。
Reason 该论文研究的是视频翻译任务，核心方法为利用光流对扩散模型中的query、key、value token进行跨帧warping以实现时间一致性。尽管其token对齐思想具有一定启发性，但整体工作属于视频生成与编辑领域，与文档图像理解（DIU）的核心任务如DocVQA、OCR、版面分析等无直接关联。且未见其方法可有效迁移至解决文档中的OCR幻觉、复杂排版或长文档推理等关键痛点。
Abstract With the revolution of generative AI, video-related tasks have been widely studied. However, current state-of-the-art video models still lag behind image models in visual quality and user control over generated content. In this paper, we introduce TokenWarping, a novel framework for temporally coherent video translation. Existing diffusion-based video editing approaches rely solely on key and value patches in self-attention to ensure temporal consistency, often sacrificing the preservation of local and structural regions. Critically, these methods overlook the significance of the query patches in achieving accurate feature aggregation and temporal coherence. In contrast, TokenWarping leverages complementary token priors by constructing temporal correlations across different frames. Our method begins by extracting optical flows from source videos. During the denoising process of the diffusion model, these optical flows are used to warp the previous frame's query, key, and value patches, aligning them with the current frame's patches. By directly warping the query patches, we enhance feature aggregation in self-attention, while warping the key and value patches ensures temporal consistency across frames. This token warping imposes explicit constraints on the self-attention layer outputs, effectively ensuring temporally coherent translation. Our framework does not require any additional training or fine-tuning and can be seamlessly integrated with existing text-to-image editing methods. We conduct extensive experiments on various video translation tasks, demonstrating that TokenWarping surpasses state-of-the-art methods both qualitatively and quantitatively. Video demonstrations can be found on our project webpage: https://alex-zhu1.github.io/TokenWarping/. Code is available at: https://github.com/Alex-Zhu1/TokenWarping.
Authors Haiming Zhu, Yangyang Xu, Jun Yu, Shengfeng He
Comment Code is available at: https://github.com/Alex-Zhu1/TokenWarping
Categories Computer Vision and Pattern Recognition
0
HunyuanVideo 1.5 Technical Report
Title CN HunyuanVideo 1.5 技术报告
Keywords 视频生成 · DiT架构 · 滑动tile注意力 · 双语理解 · 视频超分辨率
Summary 本文提出了HunyuanVideo 1.5，一个仅含83亿参数的轻量级开源视频生成模型，在视觉质量和运动连贯性方面达到开源模型中的领先水平。关键技术包括精选数据、改进的DiT架构（SSTA）、字形感知的双语文本编码、渐进式训练策略以及高效的视频超分辨率网络。支持多种时长和分辨率下的高质量文生视频与图生视频生成。
Reason 该论文属于视频生成领域，研究重点为文本到视频和图像到视频的生成模型，涉及DiT架构、滑动tile注意力和视频超分辨率等技术，与文档图像理解（DIU）或DocVQA无直接关联。其方法难以迁移至解决OCR幻觉、复杂排版理解、细粒度文本定位等文档核心问题，属于无关领域工作。
Abstract We present HunyuanVideo 1.5, a lightweight yet powerful open-source video generation model that achieves state-of-the-art visual quality and motion coherence with only 8.3 billion parameters, enabling efficient inference on consumer-grade GPUs. This achievement is built upon several key components, including meticulous data curation, an advanced DiT architecture featuring selective and sliding tile attention (SSTA), enhanced bilingual understanding through glyph-aware text encoding, progressive pre-training and post-training, and an efficient video super-resolution network. Leveraging these designs, we developed a unified framework capable of high-quality text-to-video and image-to-video generation across multiple durations and resolutions. Extensive experiments demonstrate that this compact and proficient model establishes a new state-of-the-art among open-source video generation models. By releasing the code and model weights, we provide the community with a high-performance foundation that lowers the barrier to video creation and research, making advanced video generation accessible to a broader audience. All open-source assets are publicly available at https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5.
Authors Bing Wu, Chang Zou, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Jack Peng, Jianbing Wu, Jiangfeng Xiong, Jie Jiang, Linus, Patrol, Peizhen Zhang, Peng Chen, Penghao Zhao, Qi Tian, Songtao Liu, Weijie Kong, Weiyan Wang, Xiao He, Xin Li, Xinchi Deng, Xuefei Zhe, Yang Li, Yanxin Long, Yuanbo Peng, Yue Wu, Yuhong Liu, Zhenyu Wang, Zuozhuo Dai, Bo Peng, Coopers Li, Gu Gong, Guojian Xiao, Jiahe Tian, Jiaxin Lin, Jie Liu, Jihong Zhang, Jiesong Lian, Kaihang Pan, Lei Wang, Lin Niu, Mingtao Chen, Mingyang Chen, Mingzhe Zheng, Miles Yang, Qiangqiang Hu, Qi Yang, Qiuyong Xiao, Runzhou Wu, Ryan Xu, Rui Yuan, Shanshan Sang, Shisheng Huang, Siruis Gong, Shuo Huang, Weiting Guo, Xiang Yuan, Xiaojia Chen, Xiawei Hu, Wenzhi Sun, Xiele Wu, Xianshun Ren, Xiaoyan Yuan, Xiaoyue Mi, Yepeng Zhang, Yifu Sun, Yiting Lu, Yitong Li, You Huang, Yu Tang, Yixuan Li, Yuhang Deng, Yuan Zhou, Zhichao Hu, Zhiguang Liu, Zhihe Yang, Zilin Yang, Zhenzhi Lu, Zixiang Zhou, Zhao Zhong
Categories Computer Vision and Pattern Recognition
0
STAvatar: Soft Binding and Temporal Density Control for Monocular 3D Head Avatars Reconstruction
Title CN STAvatar：用于单目3D头部虚拟形象重建的软绑定与时间密度控制
Keywords 3D Gaussian Splatting · 3D Avatar · Soft Binding · Temporal Density Control · Monocular Reconstruction
Summary 本文提出STAvatar，通过UV自适应软绑定和时间自适应密度控制来提升单目视频中3D头部虚拟形象的重建质量，尤其改善了遮挡区域（如口腔、眼睑）的细节表现。方法基于3D高斯点阵化技术，在UV空间学习每个高斯的特征偏移，并引入融合感知误差进行更精准的稠密化。实验表明其在多个数据集上达到SOTA效果。
Reason 该论文研究的是从单目视频中重建3D头部虚拟形象，属于3D视觉和具身智能领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无关。其核心技术如UV空间软绑定、时间密度控制等针对的是3D头像建模中的几何与纹理细节优化，无法迁移到解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心问题。
Abstract Reconstructing high-fidelity and animatable 3D head avatars from monocular videos remains a challenging yet essential task. Existing methods based on 3D Gaussian Splatting typically bind Gaussians to mesh triangles and model deformations solely via Linear Blend Skinning, which results in rigid motion and limited expressiveness. Moreover, they lack specialized strategies to handle frequently occluded regions (e.g., mouth interiors, eyelids). To address these limitations, we propose STAvatar, which consists of two key components: (1) a UV-Adaptive Soft Binding framework that leverages both image-based and geometric priors to learn per-Gaussian feature offsets within the UV space. This UV representation supports dynamic resampling, ensuring full compatibility with Adaptive Density Control (ADC) and enhanced adaptability to shape and textural variations. (2) a Temporal ADC strategy, which first clusters structurally similar frames to facilitate more targeted computation of the densification criterion. It further introduces a novel fused perceptual error as clone criterion to jointly capture geometric and textural discrepancies, encouraging densification in regions requiring finer details. Extensive experiments on four benchmark datasets demonstrate that STAvatar achieves state-of-the-art reconstruction performance, especially in capturing fine-grained details and reconstructing frequently occluded regions. The code will be publicly available.
Authors Jiankuo Zhao, Xiangyu Zhu, Zidu Wang, Zhen Lei
Comment 17 pages, 14 figures
Categories Computer Vision and Pattern Recognition
0
Disc3D: Automatic Curation of High-Quality 3D Dialog Data via Discriminative Object Referring
Title CN Disc3D：通过判别性物体指代表达自动构建高质量3D对话数据
Keywords 3D多模态 · 对话数据生成 · 物体指代 · 场景图 · 自动标注
Summary 本文提出了一种全自动的数据构建流程Disc3D，用于从原始3D扫描数据生成高质量、无歧义的3D场景对话数据集。该流程包含元标注收集、场景图构建、判别性物体指代描述生成和多任务对话合成四个阶段，解决了3D对话数据中存在的视角歧义和物体指代模糊问题。最终构建了包含25K混合3D场景、超过200万样本的大型数据集，并验证了其在多个3D视觉问答任务上的有效性。
Reason 该论文聚焦于3D视觉与多模态对话数据的构建，属于具身智能与3D视觉领域，研究目标为3D场景下的指代理解与对话生成。其核心问题（如视角歧义、物体指代歧义）和应用场景（3D扫描场景）与文档图像理解（DIU）任务无直接关联。尽管涉及多模态数据生成和指代消解技术，但这些方法针对的是三维空间中的物体关系建模，难以迁移到以二维文档布局、文本语义结构和OCR相关幻觉为核心的DIU任务中。
Abstract 3D Multi-modal Large Language Models (MLLMs) still lag behind their 2D peers, largely because large-scale, high-quality 3D scene-dialogue datasets remain scarce. Prior efforts hinge on expensive human annotation and leave two key ambiguities unresolved: viewpoint ambiguity, where spatial language presumes unknown camera poses, and object referring ambiguity, where non-exclusive descriptions blur the line between targets and distractors. We therefore present a fully automated pipeline that converts raw 3D scans into unambiguous, high-quality dialogue data at a fraction of the previous cost. By synergizing rule-based constraints with 2D MLLMs and LLMs, the pipeline enables controllable, scalable generation without human intervention. The pipeline comprises four stages: (1) meta-annotation collection harvesting object-, frame-, and scene-level captions, (2) scene graph construction with relation correction to capture proximal object relations, (3) discriminative object referring that generates exclusive and compact descriptions, and (4) multi-task data generation synthesizing diverse dialogues. Our pipeline systematically mitigates inherent flaws in source datasets and produces the final Disc3D dataset, over 2 million samples in 25K hybrid 3D scenes, spanning scene, view, and object captioning, visual grounding, and five object-centric QA tasks. Extensive experiments demonstrate that training with Disc3D yields consistent, significant improvements on both public benchmarks and our multifaceted Disc3D-QA tasks. Code, data, and models will be publicly available.
Authors Siyuan Wei, Chunjie Wang, Xiao Liu, Xiaosheng Yan, Zhishan Zhou, Rui Huang
Comment 8 pages
Categories Computer Vision and Pattern Recognition
0
DOGE: Differentiable Bezier Graph Optimization for Road Network Extraction
Title CN DOGE：用于道路网络提取的可微分贝塞尔图优化
Keywords 道路网络提取 · 贝塞尔曲线 · 可微分渲染 · 图优化 · 航拍图像
Summary 本文提出了一种名为DOGE的框架，通过引入可微分的贝塞尔图来表示道路网络，直接从分割掩码学习参数化曲线结构，无需依赖难以构建的矢量真值。通过DiffAlign和TopoAdapt两个模块交替优化几何形状和拓扑结构，在SpaceNet和CityScale数据集上取得了SOTA性能。
Reason 该论文研究的是从航拍图像中提取道路网络，属于自动驾驶或地理信息系统领域的任务，与文档图像理解（DIU）无关。尽管其使用了可微分渲染和图优化等技术，但应用场景和核心问题（如OCR幻觉、复杂排版、长文档推理）完全不匹配，无法迁移到文档理解领域。
Abstract Automatic extraction of road networks from aerial imagery is a fundamental task, yet prevailing methods rely on polylines that struggle to model curvilinear geometry. We maintain that road geometry is inherently curve-based and introduce the Bézier Graph, a differentiable parametric curve-based representation. The primary obstacle to this representation is to obtain the difficult-to-construct vector ground-truth (GT). We sidestep this bottleneck by reframing the task as a global optimization problem over the Bézier Graph. Our framework, DOGE, operationalizes this paradigm by learning a parametric Bézier Graph directly from segmentation masks, eliminating the need for curve GT. DOGE holistically optimizes the graph by alternating between two complementary modules: DiffAlign continuously optimizes geometry via differentiable rendering, while TopoAdapt uses discrete operators to refine its topology. Our method sets a new state-of-the-art on the large-scale SpaceNet and CityScale benchmarks, presenting a new paradigm for generating high-fidelity vector maps of road networks. We will release our code and related data.
Authors Jiahui Sun, Junran Lu, Jinhui Yin, Yishuo Xu, Yuanqi Li, Yanwen Guo
Comment 11 pages, 6 figures
Categories Computer Vision and Pattern Recognition
0
MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence
Title CN MovieDreamer：用于连贯长视觉序列的分层生成
Keywords 视频生成 · 扩散模型 · 自回归模型 · 长序列建模 · 叙事一致性
Summary 本文提出MovieDreamer，一种结合自回归模型与扩散渲染的分层框架，用于生成具有复杂情节进展和高视觉保真度的长时长视频。通过将故事分解为可管理的场景，并利用包含角色细节和视觉风格的多模态剧本，提升跨场景的一致性。实验表明该方法在多种电影类型上均能显著延长生成时长并提高视觉与叙事质量。
Reason 该论文属于视频生成领域，核心任务是长视频或电影级内容的生成，使用扩散模型与自回归结合的方法建模复杂叙事。尽管涉及多模态和长序列建模，但其目标场景（如角色一致性、剧情推进）与文档图像理解中的OCR幻觉、密集文本识别、版面分析等核心痛点无直接关联。方法难以迁移到文档理解任务中，属于无关领域。
Abstract Recent advancements in video generation have primarily leveraged diffusion models for short-duration content. However, these approaches often fall short in modeling complex narratives and maintaining character consistency over extended periods, which is essential for long-form video production like movies. We propose MovieDreamer, a novel hierarchical framework that integrates the strengths of autoregressive models with diffusion-based rendering to pioneer long-duration video generation with intricate plot progressions and high visual fidelity. Our approach utilizes autoregressive models for global narrative coherence, predicting sequences of visual tokens that are subsequently transformed into high-quality video frames through diffusion rendering. This method is akin to traditional movie production processes, where complex stories are factorized down into manageable scene capturing. Further, we employ a multimodal script that enriches scene descriptions with detailed character information and visual style, enhancing continuity and character identity across scenes. We present extensive experiments across various movie genres, demonstrating that our approach not only achieves superior visual and narrative quality but also effectively extends the duration of generated content significantly beyond current capabilities. Homepage: https://aim-uofa.github.io/MovieDreamer/.
Authors Canyu Zhao, Mingyu Liu, Wen Wang, Weihua Chen, Fan Wang, Hao Chen, Bo Zhang, Chunhua Shen
Comment 30 pages, 22 figures
Categories Computer Vision and Pattern Recognition
0
E$^{3}$NeRF: Efficient Event-Enhanced Neural Radiance Fields from Blurry Images
Title CN E³NeRF：从模糊图像中高效重建事件增强的神经辐射场
Keywords NeRF · 事件相机 · 运动模糊 · 3D重建 · 高动态场景
Summary 本文提出E³NeRF，一种利用模糊图像和事件流共同训练的高效NeRF方法，通过引入模糊渲染损失和事件渲染损失，建模运动模糊和事件生成过程，实现对高速非均匀运动和低光场景下的清晰3D重建。同时利用事件流中的时空模糊信息提升训练效率，并构建了基于事件引导的真实数据相机位姿估计框架。
Reason 该论文研究的是3D视觉中的神经辐射场（NeRF）重建问题，属于3D视觉与计算机视觉交叉领域，主要解决由模糊图像和事件流联合恢复清晰3D场景的问题。其核心技术聚焦于事件相机数据建模、运动模糊物理建模和视角渲染，与文档图像理解（DIU）的核心任务如文本识别、布局分析、表格理解等无直接关联。尽管涉及‘高分辨率信息恢复’，但其方法依赖事件流这一在文档场景中不存在的模态，无法迁移到OCR幻觉、复杂排版或长文档推理等DIU痛点。属于无关领域。
Abstract Neural Radiance Fields (NeRF) achieves impressive novel view rendering performance by learning implicit 3D representation from sparse view images. However, it is difficult to reconstruct a sharp NeRF from blurry input that often occurs in the wild. To solve this problem, we propose a novel Efficient Event-Enhanced NeRF (E$^{3}$NeRF), reconstructing sharp NeRF by utilizing both blurry images and corresponding event streams. A blur rendering loss and an event rendering loss are introduced, which guide the NeRF training via modeling the physical image motion blur process and event generation process, respectively. To improve the efficiency of the framework, we further leverage the latent spatial-temporal blur information in the event stream to evenly distribute training over temporal blur and focus training on spatial blur. Moreover, a camera pose estimation framework for real-world data is built with the guidance of the events, generalizing the method to more practical applications. Compared to previous image-based and event-based NeRF works, our framework makes more profound use of the internal relationship between events and images. Extensive experiments on both synthetic data and real-world data demonstrate that E\textsuperscript{3}NeRF can effectively learn a sharp NeRF from blurry images, especially for high-speed non-uniform motion and low-light scenes.
Authors Yunshan Qi, Jia Li, Yifan Zhao, Yu Zhang, Lin Zhu
Categories Computer Vision and Pattern Recognition
0
NeuroGaze-Distill: Brain-informed Distillation and Depression-Inspired Geometric Priors for Robust Facial Emotion Recognition
Title CN NeuroGaze-Distill：基于脑信息蒸馏与抑郁启发的几何先验用于鲁棒面部表情识别
Keywords 面部表情识别 · 知识蒸馏 · 跨模态学习 · 几何先验 · 鲁棒性
Summary 本文提出NeuroGaze-Distill，一种通过脑电图（EEG）引导的跨模态知识蒸馏框架，用于提升纯视觉面部表情识别的鲁棒性。教师模型基于EEG特征生成静态的效价-唤醒（V/A）原型网格，并引入受抑郁症研究启发的几何先验（D-Geo）来塑造学生模型的嵌入空间。学生模型仅使用图像数据训练，通过Proto-KD和D-Geo正则化项提升跨数据集泛化能力。实验表明该方法在多个基准上提升了鲁棒性。
Reason 该论文研究的是面部表情识别（FER），属于通用图像理解中的情感计算领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其使用了跨模态蒸馏和几何先验等方法，但这些技术是针对情感识别中的跨数据集泛化问题设计的，迁移到文档理解场景（如处理OCR幻觉、复杂排版或长文档推理）极为困难。且未涉及文本结构、视觉布局或文档特有的多模态对齐问题。
Abstract Facial emotion recognition (FER) models trained only on pixels often fail to generalize across datasets because facial appearance is an indirect and biased proxy for underlying affect. We present NeuroGaze-Distill, a cross-modal distillation framework that transfers brain-informed priors into an image-only FER student via static Valence/Arousal (V/A) prototypes and a depression-inspired geometric prior (D-Geo). A teacher trained on EEG topographic maps from DREAMER (with MAHNOB-HCI as unlabeled support) produces a consolidated 5x5 V/A prototype grid that is frozen and reused; no EEG-face pairing and no non-visual signals at deployment are required. The student (ResNet-18/50) is trained on FERPlus with conventional CE/KD and two lightweight regularizers: (i) Proto-KD (cosine) aligns student features to the static prototypes; (ii) D-Geo softly shapes the embedding geometry in line with affective findings often reported in depression research (e.g., anhedonia-like contraction in high-valence regions). We evaluate both within-domain (FERPlus validation) and cross-dataset protocols (AffectNet-mini; optional CK+), reporting standard 8-way scores alongside present-only Macro-F1 and balanced accuracy to fairly handle label-set mismatch. Ablations attribute consistent gains to prototypes and D-Geo, and favor 5x5 over denser grids for stability. The method is simple, deployable, and improves robustness without architectural complexity.
Authors Zilin Li, Weiwei Xu, Xuanqi Zhao, Yiran Zhu
Comment Preprint. Vision-only deployment; EEG used to form static prototypes. Includes appendix, 7 figures and 3 tables. Considering submission to ICLR 2026. Revision note: This version corrects inaccuracies in the authors' institutional affiliations. No technical content has been modified
Categories Computer Vision and Pattern Recognition
0
4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models
Title CN 4DWorldBench：面向3D/4D世界生成模型的综合性评估框架
Keywords 3D生成 · 4D生成 · 世界模型 · 多模态评估 · 基准测试
Summary 本文提出了4DWorldBench，一个针对3D/4D世界生成模型的综合评估框架，涵盖感知质量、条件-4D对齐、物理真实性和4D一致性四个维度。通过引入跨模态自适应条件机制，并将多模态输入映射到统一文本空间，结合LLM-as-judge等评判方式，实现更一致的跨模态评估。初步人类实验表明其与主观判断更一致。
Reason 该论文聚焦于3D/4D世界生成模型的评估，属于具身智能、虚拟现实和内容生成领域，与文档图像理解（DIU）或DocVQA无直接关联。其核心任务如Image-to-3D、Video-to-4D等远离文档结构、文本识别或布局分析等DIU核心问题。尽管评估框架设计具有一定方法论价值，但无法迁移用于解决OCR幻觉、复杂排版或长文档推理等文档理解痛点。
Abstract World Generation Models are emerging as a cornerstone of next-generation multimodal intelligence systems. Unlike traditional 2D visual generation, World Models aim to construct realistic, dynamic, and physically consistent 3D/4D worlds from images, videos, or text. These models not only need to produce high-fidelity visual content but also maintain coherence across space, time, physics, and instruction control, enabling applications in virtual reality, autonomous driving, embodied intelligence, and content creation. However, prior benchmarks emphasize different evaluation dimensions and lack a unified assessment of world-realism capability. To systematically evaluate World Models, we introduce the 4DWorldBench, which measures models across four key dimensions: Perceptual Quality, Condition-4D Alignment, Physical Realism, and 4D Consistency. The benchmark covers tasks such as Image-to-3D/4D, Video-to-4D, Text-to-3D/4D. Beyond these, we innovatively introduce adaptive conditioning across multiple modalities, which not only integrates but also extends traditional evaluation paradigms. To accommodate different modality-conditioned inputs, we map all modality conditions into a unified textual space during evaluation, and further integrate LLM-as-judge, MLLM-as-judge, and traditional network-based methods. This unified and adaptive design enables more comprehensive and consistent evaluation of alignment, physical realism, and cross-modal coherence. Preliminary human studies further demonstrate that our adaptive tool selection achieves closer agreement with subjective human judgments. We hope this benchmark will serve as a foundation for objective comparisons and improvements, accelerating the transition from "visual generation" to "world generation." Our project can be found at https://yeppp27.github.io/4DWorldBench.github.io/.
Authors Yiting Lu, Wei Luo, Peiyan Tu, Haoran Li, Hanxin Zhu, Zihao Yu, Xingrui Wang, Xinyi Chen, Xinge Peng, Xin Li, Zhibo Chen
Categories Computer Vision and Pattern Recognition
0
Learning Hierarchical Sparse Transform Coding of 3DGS
Title CN 学习3DGS的分层稀疏变换编码
Keywords 3D高斯点阵 · 变换编码 · 稀疏性 · 3D压缩 · KLT
Summary 本文提出了一种名为SHTC的分层变换编码方法，用于压缩3D高斯点阵（3DGS）模型。通过引入KLT去相关、量化与熵编码，并结合稀疏性先验和深度展开技术，实现了更高效的压缩性能和更快的解码速度。然而，该工作聚焦于3D视觉表示的压缩，不属于文档图像理解范畴。
Reason 该论文研究的是3D高斯点阵（3DGS）的压缩问题，属于3D视觉与图形学领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管其提出的变换编码和稀疏性建模思想在理论上可能对信号压缩有启发，但该方法针对的是3D场景表示，迁移到文档图像理解中的OCR幻觉、复杂排版解析或长文档推理等核心痛点极为困难，且未见任何与多模态语言模型（VLM）或文本-布局联合建模的结合路径。
Abstract 3D Gaussian Splatting (3DGS) supports fast, high quality, novel view synthesis but has a heavy memory footprint, making the compression of its model crucial. Current state-of-the-art (SOTA) 3DGS compression methods adopt an anchor-based architecture that pairs the Scaffold-GS representation with conditional entropy coding. However, these methods forego the analysis-synthesis transform, a vital mechanism in visual data compression. As a result, redundancy remains intact in the signal and its removal is left to the entropy coder, which computationally overburdens the entropy coding module, increasing coding latency. Even with added complexity thorough redundancy removal is a task unsuited to an entropy coder. To fix this critical omission, we introduce a Sparsity-guided Hierarchical Transform Coding (SHTC) method, the first study on the end-to-end learned neural transform coding of 3DGS. SHTC applies KLT to decorrelate intra-anchor attributes, followed by quantization and entropy coding, and then compresses KLT residuals with a low-complexity, scene-adaptive neural transform. Aided by the sparsity prior and deep unfolding technique, the learned transform uses only a few trainable parameters, reducing the memory usage. Overall, SHTC achieves an appreciably improved R-D performance and at the same time higher decoding speed over SOTA. Its prior-guided, parameter-efficient design may also inspire low-complexity neural image and video codecs. Our code will be released at https://github.com/hxu160/SHTC_for_3DGS_compression.
Authors Hao Xu, Xiaolin Wu, Xi Zhang
Comment Our code will be released at \href{https://github.com/hxu160/SHTC_for_3DGS_compression}{here}
Categories Computer Vision and Pattern Recognition
0
Large Language Model Aided Birt-Hogg-Dube Syndrome Diagnosis with Multimodal Retrieval-Augmented Generation
Title CN 基于大语言模型辅助的多模态检索增强生成用于Birt-Hogg-Dubé综合征诊断
Keywords 多模态大语言模型 · 检索增强生成 · 医学影像诊断 · Birt-Hogg-Dube综合征 · 临床决策支持
Summary 本文提出BHD-RAG框架，通过构建DCLD疾病的多模态病例库，利用检索增强生成方法提升大模型在罕见肺病CT影像诊断中的准确性。系统包含一个生成CT描述的代理模块、基于余弦相似度的检索器和一个综合证据进行诊断的MLLM。实验表明该方法能生成符合专家认知的可解释性诊断结果。
Reason 该论文属于典型的‘平行下游应用’：将现有的RAG框架与MLLM结合应用于特定医学影像诊断任务（BHD综合征），并未提出任何面向文档图像理解（DIU）的方法论创新。其技术路线为‘领域知识增强+检索+生成’，完全局限于医疗垂类场景，无法迁移至DocVQA、OCR-free理解、复杂版面分析等核心DIU问题。且未涉及视觉压缩、高分辨率处理、细粒度定位或去幻觉机制等底层技术创新。
Abstract Deep learning methods face dual challenges of limited clinical samples and low inter-class differentiation among Diffuse Cystic Lung Diseases (DCLDs) in advancing Birt-Hogg-Dube syndrome (BHD) diagnosis via Computed Tomography (CT) imaging. While Multimodal Large Language Models (MLLMs) demonstrate diagnostic potential fo such rare diseases, the absence of domain-specific knowledge and referable radiological features intensify hallucination risks. To address this problem, we propose BHD-RAG, a multimodal retrieval-augmented generation framework that integrates DCLD-specific expertise and clinical precedents with MLLMs to improve BHD diagnostic accuracy. BHDRAG employs: (1) a specialized agent generating imaging manifestation descriptions of CT images to construct a multimodal corpus of DCLDs cases. (2) a cosine similarity-based retriever pinpointing relevant imagedescription pairs for query images, and (3) an MLLM synthesizing retrieved evidence with imaging data for diagnosis. BHD-RAG is validated on the dataset involving four types of DCLDs, achieving superior accuracy and generating evidence-based descriptions closely aligned with expert insights.
Authors Haoqing Li, Jun Shi, Xianmeng Chen, Qiwei Jia, Rui Wang, Wei Wei, Hong An, Xiaowen Hu
Categories Computer Vision and Pattern Recognition
0
The Early Bird Identifies the Worm: You Can't Beat a Head Start in Long-Term Body Re-ID (ECHO-BID)
Title CN 早起的鸟儿有虫吃：长期人体再识别中的先发优势（ECHO-BID）
Keywords Person Re-Identification · Foundation Models · Domain Transfer Learning · EVA-02 · Long-Term Recognition
Summary 本文研究了四种视觉基础模型（CLIP、DINOv2、AIMv2、EVA-02）在长期人体再识别任务上的域迁移性能，发现基于EVA-02的模型表现最优，并提出了ECHO-BID方法。实验表明模型规模和迁移策略对性能至关重要。
Reason 该论文属于行人重识别（Person Re-ID）领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无关。尽管使用了视觉基础模型和域迁移方法，但其研究目标是解决跨时间的人体识别问题，无法迁移到OCR幻觉、复杂排版理解、细粒度文本定位等文档核心痛点。
Abstract A wide range of model-based approaches to long-term person re-identification have been proposed. Whether these models perform more accurately than direct domain transfer learning applied to extensively trained large-scale foundation models is not known. We applied domain transfer learning for long-term person re-id to four vision foundation models (CLIP, DINOv2, AIMv2, and EVA-02). Domain-adapted versions of all four models %CLIP-L, DINOv2-L, AIMv2-L, and EVA-02-L surpassed existing state-of-the-art models by a large margin in highly unconstrained viewing environments. Decision score fusion of the four models improved performance over any individual model. Of the individual models, the EVA-02 foundation model provided the best ``head start'' to long-term re-id, surpassing other models on three of the four performance metrics by substantial margins. Accordingly, we introduce $\textbf{E}$va $\textbf{C}$lothes-Change from $\textbf{H}$idden $\textbf{O}$bjects - $\textbf{B}$ody $\textbf{ID}$entification (ECHO-BID), a class of long-term re-id models built on the object-pretrained EVA-02 Large backbones. Ablation experiments varying backbone size, scale of object classification pretraining, and transfer learning protocol indicated that model size and the use of a smaller, but more challenging transfer learning protocol are critical features in performance. We conclude that foundation models provide a head start to domain transfer learning and support state-of-the-art performance with modest amounts of domain data. The limited availability of long-term re-id data makes this approach advantageous.
Authors Thomas M. Metz, Matthew Q. Hill, Alice J. O'Toole
Categories Computer Vision and Pattern Recognition
0
ReDirector: Creating Any-Length Video Retakes with Rotary Camera Encoding
Title CN ReDirector：通过旋转相机编码生成任意长度的视频重拍
Keywords 视频生成 · 相机控制 · 位置编码 · 时空建模 · RoPE
Summary 本文提出了ReDirector，一种基于旋转位置编码（RoPE）改进的视频重拍生成方法，引入了旋转相机编码（RoCE），通过将相机参数融入位置编码来提升多视角一致性与动态物体定位能力。该方法可处理变长视频并支持未见过的相机轨迹，在几何一致性和视频质量上表现更优。
Reason 该论文研究的是视频重拍生成中的相机控制问题，属于视频理解与生成领域，核心关注动态视频中摄像机轨迹建模和时空对齐。其提出的方法RoCE虽在视觉编码上有创新，但主要面向自由视角视频生成，与文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等无直接关联。无法迁移用于解决OCR幻觉、密集文本识别、复杂排版解析或长文档推理等DIU核心痛点。
Abstract We present ReDirector, a novel camera-controlled video retake generation method for dynamically captured variable-length videos. In particular, we rectify a common misuse of RoPE in previous works by aligning the spatiotemporal positions of the input video and the target retake. Moreover, we introduce Rotary Camera Encoding (RoCE), a camera-conditioned RoPE phase shift that captures and integrates multi-view relationships within and across the input and target videos. By integrating camera conditions into RoPE, our method generalizes to out-of-distribution camera trajectories and video lengths, yielding improved dynamic object localization and static background preservation. Extensive experiments further demonstrate significant improvements in camera controllability, geometric consistency, and video quality across various trajectories and lengths.
Authors Byeongjun Park, Byung-Hoon Kim, Hyungjin Chung, Jong Chul Ye
Comment Project page: https://byeongjun-park.github.io/ReDirector/
Categories Computer Vision and Pattern Recognition
0
Training-Free Generation of Diverse and High-Fidelity Images via Prompt Semantic Space Optimization
Title CN 通过提示语义空间优化实现免训练的多样化且高保真图像生成
Keywords 图像生成 · 多样性增强 · 扩散模型 · 提示优化 · 免训练方法
Summary 本文提出一种名为Token-Prompt embedding Space Optimization（TPSO）的免训练方法，用于提升文本到图像扩散模型的生成多样性。通过在token嵌入空间中引入可学习参数探索低密度区域，并利用prompt级语义约束防止质量退化，实现在不牺牲图像保真度的前提下显著提升多样性。实验表明该方法在多个数据集和模型上均有效。
Reason 该论文属于纯图像生成领域，研究的是文本到图像扩散模型中的多样性生成问题，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其提出的方法TPSO聚焦于图像创造性生成和避免模式崩溃，虽然涉及提示嵌入空间优化，但目标是提升生成图像的视觉多样性而非解决OCR幻觉、复杂排版理解或文档推理等DIU核心问题。无法迁移用于提升文档理解性能，属于无关领域。
Abstract Image diversity remains a fundamental challenge for text-to-image diffusion models. Low-diversity models tend to generate repetitive outputs, increasing sampling redundancy and hindering both creative exploration and downstream applications. A primary cause is that generation often collapses toward a strong mode in the learned distribution. Existing attempts to improve diversity, such as noise resampling, prompt rewriting, or steering-based guidance, often still collapse to dominant modes or introduce distortions that degrade image quality. In light of this, we propose Token-Prompt embedding Space Optimization (TPSO), a training-free and model-agnostic module. TPSO introduces learnable parameters to explore underrepresented regions of the token embedding space, reducing the tendency of the model to repeatedly generate samples from strong modes of the learned distribution. At the same time, the prompt-level space provides a global semantic constraint that regulates distribution shifts, preventing quality degradation while maintaining high fidelity. Extensive experiments on MS-COCO and three diffusion backbones show that TPSO significantly enhances generative diversity, improving baseline performance from 1.10 to 4.18 points, without sacrificing image quality. Code will be released upon acceptance.
Authors Debin Meng, Chen Jin, Zheng Gao, Yanran Li, Ioannis Patras, Georgios Tzimiropoulos
Comment under review
Categories Computer Vision and Pattern Recognition, Computation and Language
0
UruDendro4: A Benchmark Dataset for Automatic Tree-Ring Detection in Cross-Section Images of Pinus taeda L IEEE 15th International Conference on Pattern Recognition Systems (ICPRS-25)
Title CN UruDendro4：一种用于火炬松横截面图像中自动年轮检测的基准数据集
Keywords 年轮检测 · 图像分割 · 基准数据集 · 植物图像分析 · 计算机视觉
Summary 本文提出了一个名为UruDendro4的新数据集，包含102张火炬松横截面图像及其手工标注的年轮信息，支持在不同树干高度上进行年轮检测和体积建模。作者使用DeepCS-TRD等先进方法建立了自动检测的性能基线，并通过消融实验验证模型配置。结果表明，加入该数据集训练可提升模型泛化能力。
Reason 该论文聚焦于树木年轮检测这一特定生物图像分析任务，属于专用图像分割与目标检测应用，且不涉及文档结构、文本理解或OCR相关内容。尽管使用了深度学习方法并构建了新数据集，但其问题域与文档图像理解（DIU）的核心挑战（如密集文本识别、复杂版式分析、VQA等）无直接关联，属于‘平行应用’类研究，无法迁移至文档理解场景。
Abstract Tree-ring growth represents the annual wood increment for a tree, and quantifying it allows researchers to assess which silvicultural practices are best suited for each species. Manual measurement of this growth is time-consuming and often imprecise, as it is typically performed along 4 to 8 radial directions on a cross-sectional disc. In recent years, automated algorithms and datasets have emerged to enhance accuracy and automate the delineation of annual rings in cross-sectional images. To address the scarcity of wood cross-section data, we introduce the UruDendro4 dataset, a collection of 102 image samples of Pinus taeda L., each manually annotated with annual growth rings. Unlike existing public datasets, UruDendro4 includes samples extracted at multiple heights along the stem, allowing for the volumetric modeling of annual growth using manually delineated rings. This dataset (images and annotations) allows the development of volumetric models for annual wood estimation based on cross-sectional imagery. Additionally, we provide a performance baseline for automatic ring detection on this dataset using state-of-the-art methods. The highest performance was achieved by the DeepCS-TRD method, with a mean Average Precision of 0.838, a mean Average Recall of 0.782, and an Adapted Rand Error score of 0.084. A series of ablation experiments were conducted to empirically validate the final parameter configuration. Furthermore, we empirically demonstrate that training a learning model including this dataset improves the model's generalization in the tree-ring detection task.
Authors Henry Marichal, Joaquin Blanco, Diego Passarella, Gregory Randall
Comment Accepted at IEEE 15th International Conference on Pattern Recognition Systems (ICPRS-25)
Categories Computer Vision and Pattern Recognition
0
Smooth regularization for efficient video recognition NeurIPS 2025
Title CN 用于高效视频识别的平滑正则化
Keywords 视频识别 · 平滑正则化 · 时间归纳偏置 · 轻量级模型 · 高斯随机游走
Summary 本文提出一种基于高斯随机游走（GRW）的平滑正则化方法，通过约束连续帧中间层嵌入的变化为低加速度模式，增强轻量级视频模型的时间归纳偏置，在Kinetics-600等数据集上显著提升了准确率。该方法主要面向视频动作识别任务，强调时间一致性建模。
Reason 该论文研究的是视频识别中的时间平滑正则化技术，属于视频理解领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的高斯随机游走正则化方法在时间序列建模上有一定创新，但应用场景和数据模态（视频帧序列）与文档图像差异巨大，无法迁移解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU关键痛点。
Abstract We propose a smooth regularization technique that instills a strong temporal inductive bias in video recognition models, particularly benefiting lightweight architectures. Our method encourages smoothness in the intermediate-layer embeddings of consecutive frames by modeling their changes as a Gaussian Random Walk (GRW). This penalizes abrupt representational shifts, thereby promoting low-acceleration solutions that better align with the natural temporal coherence inherent in videos. By leveraging this enforced smoothness, lightweight models can more effectively capture complex temporal dynamics. Applied to such models, our technique yields a 3.8% to 6.4% accuracy improvement on Kinetics-600. Notably, the MoViNets model family trained with our smooth regularization improves the current state of the art by 3.8% to 6.1% within their respective FLOP constraints, while MobileNetV3 and the MoViNets-Stream family achieve gains of 4.9% to 6.4% over prior state-of-the-art models with comparable memory footprints. Our code and models are available at https://github.com/gilgoldm/grw-smoothing.
Authors Gil Goldman, Raja Giryes, Mahadev Satyanarayanan
Comment Accepted to NeurIPS 2025
Categories Computer Vision and Pattern Recognition
0
A deep learning model to reduce agent dose for contrast-enhanced MRI of the cerebellopontine angle cistern
Title CN 一种用于减少小脑桥脑角池对比增强MRI造影剂剂量的深度学习模型
Keywords 深度学习 · MRI · 低剂量成像 · 图像重建 · 医学影像
Summary 本研究提出了一种深度学习模型，用于从极低剂量（10%-30%标准剂量）的造影剂T1加权MRI中恢复高质量的小脑桥脑角池图像。实验表明，该模型显著提升了图像的结构相似性和信噪比，并改善了肿瘤分割性能。放射科医生评估认为恢复后的图像质量优良，具备临床诊断可行性。
Reason 该论文属于医学影像重建领域，研究通过深度学习恢复低剂量MRI图像质量，虽涉及图像处理技术，但其目标与文档图像理解（DIU）无关，不解决OCR幻觉、复杂排版、细粒度定位或长文档推理等核心问题。无法迁移其方法至DIU场景，属于无关领域。
Abstract Objectives: To evaluate a deep learning (DL) model for reducing the agent dose of contrast-enhanced T1-weighted MRI (T1ce) of the cerebellopontine angle (CPA) cistern. Materials and methods: In this multi-center retrospective study, T1 and T1ce of vestibular schwannoma (VS) patients were used to simulate low-dose T1ce with varying reductions of contrast agent dose. DL models were trained to restore standard-dose T1ce from the low-dose simulation. The image quality and segmentation performance of the DL-restored T1ce were evaluated. A head and neck radiologist was asked to rate DL-restored images in multiple aspects, including image quality and diagnostic characterization. Results: 203 MRI studies from 72 VS patients (mean age, 58.51 \pm 14.73, 39 men) were evaluated. As the input dose increased, the structural similarity index measure of the restored T1ce increased from 0.639 \pm 0.113 to 0.993 \pm 0.009, and the peak signal-to-noise ratio increased from 21.6 \pm 3.73 dB to 41.4 \pm 4.84 dB. At 10% input dose, using DL-restored T1ce for segmentation improved the Dice from 0.673 to 0.734, the 95% Hausdorff distance from 2.38 mm to 2.07 mm, and the average surface distance from 1.00 mm to 0.59 mm. Both DL-restored T1ce from 10% and 30% input doses showed excellent images, with the latter being considered more informative. Conclusion: The DL model improved the image quality of low-dose MRI of the CPA cistern, which makes lesion detection and diagnostic characterization possible with 10% - 30% of the standard dose.
Authors Yunjie Chen, Rianne A. Weber, Olaf M. Neve, Stephan R. Romeijn, Erik F. Hensen, Jelmer M. Wolterink, Qian Tao, Marius Staring, Berit M. Verbist
Categories Computer Vision and Pattern Recognition
0
Diffusion-Denoised Hyperspectral Gaussian Splatting 3DV 2026
Title CN 扩散去噪高光谱高斯点阵
Keywords 高光谱成像 · 3D重建 · 高斯点阵 · 扩散模型 · NeRF
Summary 本文提出了一种名为DD-HGS的方法，通过引入波长感知球谐函数、基于KL散度的光谱损失和扩散去噪器，实现了对高光谱场景的显式3D重建。在Hyper-NeRF数据集上验证了其优越性能，显著提升了训练和渲染效率。
Reason 该论文研究的是高光谱成像与3D场景重建的结合，属于3D视觉与高光谱图像处理领域，其核心任务是3DGS在Hyper-NeRF数据上的扩展应用。尽管涉及视觉表示学习，但完全不涉及文档图像、文本理解、OCR或VLM相关技术。其方法无法迁移至文档图像理解中的OCR幻觉、复杂排版或长文档推理等核心痛点，属于无关领域。
Abstract Hyperspectral imaging (HSI) has been widely used in agricultural applications for non-destructive estimation of plant nutrient composition and precise quantification of sample nutritional elements. Recently, 3D reconstruction methods, such as Neural Radiance Field (NeRF), have been used to create implicit neural representations of HSI scenes. This capability enables the rendering of hyperspectral channel compositions at every spatial location, thereby helping localize the target object's nutrient composition both spatially and spectrally. However, it faces limitations in training time and rendering speed. In this paper, we propose Diffusion-Denoised Hyperspectral Gaussian Splatting (DD-HGS), which enhances the state-of-the-art 3D Gaussian Splatting (3DGS) method with wavelength-aware spherical harmonics, a Kullback-Leibler divergence-based spectral loss, and a diffusion-based denoiser to enable 3D explicit reconstruction of the hyperspectral scenes for the entire spectral range. We present extensive evaluations on diverse real-world hyperspectral scenes from the Hyper-NeRF dataset to show the effectiveness of our DD-HGS. The results demonstrate that DD-HGS achieves the new state-of-the-art performance compared to all the previously published methods. Project page: https://dragonpg2000.github.io/DDHGS-website/
Authors Sunil Kumar Narayanan, Lingjun Zhao, Lu Gan, Yongsheng Chen
Comment Accepted to 3DV 2026
Categories Computer Vision and Pattern Recognition
0
Interactive Occlusion Boundary Estimation through Exploitation of Synthetic Data BMVC 2025
Title CN 通过利用合成数据实现交互式遮挡边界估计
Keywords 遮挡边界 · 交互式分割 · 合成数据 · 场景理解 · BMVC
Summary 本文提出了首个交互式遮挡边界估计（IOBE）框架MS³PE，结合多涂鸦引导机制和三编码路径网络，利用合成数据生成工具Mesh2OB构建大规模训练数据，并发布真实世界基准OB-LIGM。研究聚焦于自然图像中物体遮挡边界的定位，属于通用视觉任务。
Reason 该论文研究的是通用图像中的遮挡边界（Occlusion Boundaries）检测，属于通用计算机视觉任务，核心关注点为场景理解中的几何结构分析，与文档图像理解（DIU）中的文本识别、布局分析、表格理解等任务无直接关联。尽管其使用了高分辨率图像和合成数据，但方法针对的是自然图像中的遮挡结构，无法迁移解决文档图像中的OCR幻觉、密集文本、复杂排版或长文档推理等核心痛点。
Abstract Occlusion boundaries (OBs) geometrically localize occlusion events in 2D images and provide critical cues for scene understanding. In this paper, we present the first systematic study of Interactive Occlusion Boundary Estimation (IOBE), introducing MS\textsuperscript{3}PE, a novel multi-scribble-guided deep-learning framework that advances IOBE through two key innovations: (1) an intuitive multi-scribble interaction mechanism, and (2) a 3-encoding-path network enhanced with multi-scale strip convolutions. Our MS\textsuperscript{3}PE surpasses adapted baselines from seven state-of-the-art interactive segmentation methods, and demonstrates strong potential for OB benchmark construction through our real-user experiment. Besides, to address the scarcity of well-annotated real-world data, we propose using synthetic data for training IOBE models, and developed Mesh2OB, the first automated tool for generating precise ground-truth OBs from 3D scenes with self-occlusions explicitly handled, enabling creation of the OB-FUTURE synthetic benchmark that facilitates generalizable training without domain adaptation. Finally, we introduce OB-LIGM, a high-quality real-world benchmark comprising 120 meticulously annotated high-resolution images advancing evaluation standards in OB research. Source code and resources are available at https://github.com/xul-ops/IOBE.
Authors Lintao Xu, Chaohui Wang
Comment BMVC 2025
Categories Computer Vision and Pattern Recognition
0
V$^{2}$-SAM: Marrying SAM2 with Multi-Prompt Experts for Cross-View Object Correspondence
Title CN V²-SAM：将SAM2与多提示专家结合用于跨视角对象对应
Keywords 跨视角对应 · SAM2 · 视频理解 · 提示生成 · 多专家系统
Summary 本文提出V²-SAM，通过两个提示生成器（几何感知的锚点提示和外观引导的视觉提示）扩展SAM2用于跨视角对象匹配任务，在Ego-Exo4D等数据集上取得SOTA效果。方法依赖于视频序列中的时空一致性与特征对齐，适用于机器人、动作理解等场景，但不涉及任何文档结构或文本内容的理解。
Reason 该论文研究的是跨视角对象对应（如ego-exo场景下的物体匹配），属于视频理解与通用计算机视觉领域，核心任务为跨视角定位与跟踪，并未涉及文档图像理解中的文本识别、布局分析、表格理解或VQA等任务。尽管其使用了SAM2和提示机制，但方法面向的是自然场景下的视觉对应问题，与文档图像中的OCR幻觉、密集文本、复杂排版等痛点无关，无法迁移应用于文档理解场景。
Abstract Cross-view object correspondence, exemplified by the representative task of ego-exo object correspondence, aims to establish consistent associations of the same object across different viewpoints (e.g., ego-centric and exo-centric). This task poses significant challenges due to drastic viewpoint and appearance variations, making existing segmentation models, such as SAM2, non-trivial to apply directly. To address this, we present V^2-SAM, a unified cross-view object correspondence framework that adapts SAM2 from single-view segmentation to cross-view correspondence through two complementary prompt generators. Specifically, the Cross-View Anchor Prompt Generator (V^2-Anchor), built upon DINOv3 features, establishes geometry-aware correspondences and, for the first time, unlocks coordinate-based prompting for SAM2 in cross-view scenarios, while the Cross-View Visual Prompt Generator (V^2-Visual) enhances appearance-guided cues via a novel visual prompt matcher that aligns ego-exo representations from both feature and structural perspectives. To effectively exploit the strengths of both prompts, we further adopt a multi-expert design and introduce a Post-hoc Cyclic Consistency Selector (PCCS) that adaptively selects the most reliable expert based on cyclic consistency. Extensive experiments validate the effectiveness of V^2-SAM, achieving new state-of-the-art performance on Ego-Exo4D (ego-exo object correspondence), DAVIS-2017 (video object tracking), and HANDAL-X (robotic-ready cross-view correspondence).
Authors Jiancheng Pan, Runze Wang, Tianwen Qian, Mohammad Mahdi, Yanwei Fu, Xiangyang Xue, Xiaomeng Huang, Luc Van Gool, Danda Pani Paudel, Yuqian Fu
Comment 19 pages
Categories Computer Vision and Pattern Recognition
0
Estimating Fog Parameters from a Sequence of Stereo Images
Title CN 从立体图像序列中估计雾参数
Keywords 雾天图像 · 立体视觉 · 参数估计 · SLAM · 去雾
Summary 本文提出一种从连续立体雾天图像中动态联合估计雾模型参数的新方法，通过全局优化避免误差累积，并假设雾局部均匀以应对真实世界中非均匀雾。作者构建了首个高质量真实雾天立体图像数据集SDIRF，包含标定的光度参数和对应的晴天场景数据，可用于去雾和深度重建。实验表明该方法在合成与真实数据上均优于现有方法。
Reason 该论文研究的是在雾天条件下从立体图像序列中估计雾的物理参数，并应用于SLAM或里程计系统，属于自动驾驶和3D视觉中的环境感知问题，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务完全无关。尽管使用了计算机视觉技术，但其目标、数据和应用场景均不在DIU范畴内，且不提供可迁移至文档理解的核心方法。
Abstract We propose a method which, given a sequence of stereo foggy images, estimates the parameters of a fog model and updates them dynamically. In contrast with previous approaches, which estimate the parameters sequentially and thus are prone to error propagation, our algorithm estimates all the parameters simultaneously by solving a novel optimisation problem. By assuming that fog is only locally homogeneous, our method effectively handles real-world fog, which is often globally inhomogeneous. The proposed algorithm can be easily used as an add-on module in existing visual Simultaneous Localisation and Mapping (SLAM) or odometry systems in the presence of fog. In order to assess our method, we also created a new dataset, the Stereo Driving In Real Fog (SDIRF), consisting of high-quality, consecutive stereo frames of real, foggy road scenes under a variety of visibility conditions, totalling over 40 minutes and 34k frames. As a first-of-its-kind, SDIRF contains the camera's photometric parameters calibrated in a lab environment, which is a prerequisite for correctly applying the atmospheric scattering model to foggy images. The dataset also includes the counterpart clear data of the same routes recorded in overcast weather, which is useful for companion work in image defogging and depth reconstruction. We conducted extensive experiments using both synthetic foggy data and real foggy sequences from SDIRF to demonstrate the superiority of the proposed algorithm over prior methods. Our method not only produces the most accurate estimates on synthetic data, but also adapts better to real fog. We make our code and SDIRF publicly available\footnote{https://github.com/SenseRoboticsLab/estimating-fog-parameters} to the community with the aim of advancing the research on visual perception in fog.
Authors Yining Ding, João F. C. Mota, Andrew M. Wallace, Sen Wang
Categories Computer Vision and Pattern Recognition
0
LASER: Lip Landmark Assisted Speaker Detection for Robustness WACV 2026
Title CN LASER：基于唇部关键点辅助的鲁棒说话人检测
Keywords 主动说话人检测 · 唇部关键点 · 多模态同步 · 鲁棒性 · WACV
Summary 本文提出LASER模型，通过引入唇部关键点监督来提升复杂场景下的说话人检测鲁棒性。该方法利用2D唇部关键点生成密集图，并结合一致性损失对齐有无关键点信息的预测结果，从而在测试时无需关键点检测器。实验表明其在高噪声环境下显著优于现有方法，并发布了新数据集LASER-bench。
Reason 该论文研究的是音视频多模态任务中的主动说话人检测（ASD），属于视频理解领域，核心问题为唇动与音频同步建模，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法依赖于面部和唇部动态特征，无法迁移至解决OCR幻觉、复杂版面分析或长文档推理等DIU核心痛点。
Abstract Active Speaker Detection (ASD) aims to identify who is speaking in complex visual scenes. While humans naturally rely on lip-audio synchronization, existing ASD models often misclassify non-speaking instances when lip movements and audio are unsynchronized. To address this, we propose Lip landmark Assisted Speaker dEtection for Robustness (LASER), which explicitly incorporates lip landmarks during training to guide the model's attention to speech-relevant regions. Given a face track, LASER extracts visual features and encodes 2D lip landmarks into dense maps. To handle failure cases such as low resolution or occlusion, we introduce an auxiliary consistency loss that aligns lip-aware and face-only predictions, removing the need for landmark detectors at test time. LASER outperforms state-of-the-art models across both in-domain and out-of-domain benchmarks. To further evaluate robustness in realistic conditions, we introduce LASER-bench, a curated dataset of modern video clips with varying levels of background noise. On the high-noise subset, LASER improves mAP by 3.3 and 4.3 points over LoCoNet and TalkNet, respectively, demonstrating strong resilience to real-world acoustic challenges.
Authors Le Thien Phuc Nguyen, Zhuoran Yu, Yong Jae Lee
Comment WACV 2026
Categories Computer Vision and Pattern Recognition
0
Open Vocabulary Monocular 3D Object Detection 3DV 2026
Title CN 开放词汇单目3D物体检测
Keywords 3D物体检测 · 开放词汇 · 单目3D · 基础模型融合 · 零样本检测
Summary 本文提出了开放词汇单目3D物体检测任务，旨在仅用单张RGB图像实现对任意类别物体的3D定位。为减少对3D标注的依赖，作者提出融合预训练的2D和3D视觉基础模型的框架，并设计了一种缓解标签缺失和语义模糊问题的新评估指标。实验表明该方法在零样本和已知类别检测上均达到SOTA性能。
Reason 该论文研究的是开放词汇的单目3D物体检测，属于自动驾驶与3D视觉领域，任务目标为从单张RGB图像中在三维空间内检测任意类别的物体。其核心技术围绕3D感知、多模态传感器替代（如LiDAR）以及3D标注稀缺问题展开，与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、长文档推理）完全无关。尽管使用了2D/3D基础模型融合策略，但该方法难以迁移到文档理解场景，且不涉及文本识别、布局分析或多模态文档推理等关键环节。
Abstract We propose and study open-vocabulary monocular 3D detection, a novel task that aims to detect objects of any categores in metric 3D space from a single RGB image. Existing 3D object detectors either rely on costly sensors such as LiDAR or multi-view setups, or remain confined to closed vocabularies settings with limited categories, restricting their applicability. We identify two key challenges in this new setting. First, the scarcity of 3D bounding box annotations limits the ability to train generalizable models. To reduce dependence on 3D supervision, we propose a framework that effectively integrates pretrained 2D and 3D vision foundation models. Second, missing labels and semantic ambiguities (\eg, table vs. desk) in existing datasets hinder reliable evaluation. To address this, we design a novel metric that captures model performance while mitigating annotation issues. Our approach achieves state-of-the-art results in zero-shot 3D detection of novel categories as well as in-domain detection on seen classes. We hope our method provides a strong baseline and our evaluation protocol establishes a reliable benchmark for future research.
Authors Jin Yao, Hao Gu, Xuweiyi Chen, Jiayun Wang, Zezhou Cheng
Comment 3DV 2026, Project page: https://cvlab.cs.virginia.edu/ovmono3d
Categories Computer Vision and Pattern Recognition
0
FastAvatar: Instant 3D Gaussian Splatting for Faces from Single Unconstrained Poses
Title CN FastAvatar：从单张无约束姿态图像快速生成基于3D高斯点阵的面部模型
Keywords 3D人脸重建 · 3D高斯点阵 · 单图像重建 · 快速推理 · 身份保持
Summary 本文提出FastAvatar，一种基于3D高斯点阵（3DGS）的单图像3D人脸重建方法，可在约3秒内生成高质量全头3D头像。采用两阶段设计：第一阶段通过编码器-解码器预测粗略几何结构，第二阶段在测试时轻量级优化外观参数，实现高保真新视角合成与表情动画。相比现有方法速度快600倍以上。
Reason 该论文研究的是3D人脸重建与渲染，属于3D视觉和图形学领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无关。尽管其方法涉及高效的视觉表示，但目标场景、任务和评估指标均不涉及文本、布局、OCR或文档推理等核心问题。
Abstract We present FastAvatar, a fast and robust algorithm for single-image 3D face reconstruction using 3D Gaussian Splatting (3DGS). Given a single input image from an arbitrary pose, FastAvatar recovers a high-quality, full-head 3DGS avatar in approximately 3 seconds on a single NVIDIA A100 GPU. We use a two-stage design: a feed-forward encoder-decoder predicts coarse face geometry by regressing Gaussian structure from a pose-invariant identity embedding, and a lightweight test-time refinement stage then optimizes the appearance parameters for photorealistic rendering. This hybrid strategy combines the speed and stability of direct prediction with the accuracy of optimization, enabling strong identity preservation even under extreme input poses. FastAvatar achieves state-of-the-art reconstruction quality (24.01 dB PSNR, 0.91 SSIM) while running over 600x faster than existing per-subject optimization methods (e.g., FlashAvatar, GaussianAvatars, GASP). Once reconstructed, our avatars support photorealistic novel-view synthesis and FLAME-guided expression animation, enabling controllable reenactment from a single image. By jointly offering high fidelity, robustness to pose, and rapid reconstruction, FastAvatar significantly broadens the applicability of 3DGS-based facial avatars.
Authors Hao Liang, Zhixuan Ge, Soumendu Majee, Ashish Tiwari, G. M. Dilshan Godaliyadda, Ashok Veeraraghavan, Guha Balakrishnan
Comment 11 pages, 5 figures, website: https://hliang2.github.io/FastAvatar/
Categories Computer Vision and Pattern Recognition
0
Layer-Aware Video Composition via Split-then-Merge
Title CN 通过分而治之实现层感知视频合成
Keywords 视频生成 · 分层合成 · 自监督学习 · 前景分割 · 背景建模
Summary 本文提出Split-then-Merge（StM）框架，用于提升生成式视频合成的可控性。该方法将无标签视频自动拆分为动态前景与背景层，并通过自组合学习主体与场景的交互关系。引入变换感知训练流程和身份保持损失，在定量与定性评估中均优于现有方法。但其应用领域为视频生成，与文档理解无关。
Reason 该论文研究的是视频生成中的前景-背景分层与重组问题，属于视频理解与生成领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。其技术路线聚焦于动态视频的自监督分层和融合，难以迁移到解决OCR幻觉、密集文本识别、复杂排版或长文档推理等DIU关键痛点。属于‘无关领域’中的视频生成方向，符合负面清单标准。
Abstract We present Split-then-Merge (StM), a novel framework designed to enhance control in generative video composition and address its data scarcity problem. Unlike conventional methods relying on annotated datasets or handcrafted rules, StM splits a large corpus of unlabeled videos into dynamic foreground and background layers, then self-composes them to learn how dynamic subjects interact with diverse scenes. This process enables the model to learn the complex compositional dynamics required for realistic video generation. StM introduces a novel transformation-aware training pipeline that utilizes a multi-layer fusion and augmentation to achieve affordance-aware composition, alongside an identity-preservation loss that maintains foreground fidelity during blending. Experiments show StM outperforms SoTA methods in both quantitative benchmarks and in humans/VLLM-based qualitative evaluations. More details are available at our project page: https://split-then-merge.github.io
Authors Ozgur Kara, Yujia Chen, Ming-Hsuan Yang, James M. Rehg, Wen-Sheng Chu, Du Tran
Comment Project Webpage: https://split-then-merge.github.io
Categories Computer Vision and Pattern Recognition
0
$Δ$-NeRF: Incremental Refinement of Neural Radiance Fields through Residual Control and Knowledge Transfer
Title CN Δ-NeRF：通过残差控制和知识迁移实现神经辐射场的增量优化
Keywords NeRF · 增量学习 · 3D重建 · 知识蒸馏 · 残差控制
Summary 本文提出Δ-NeRF，一种用于增量优化神经辐射场（NeRF）的模块化残差框架，适用于数据按序到达的场景（如卫星遥感）。该方法通过冻结基础NeRF并引入残差控制器进行逐层修正，在不访问历史数据的情况下避免灾难性遗忘，并结合不确定性感知门控机制和视图选择策略提升效率与稳定性。实验表明其性能接近联合训练，同时训练时间减少30%-42%。
Reason 该论文研究的是3D视觉中的神经辐射场（NeRF）在卫星影像下的增量训练问题，属于3D重建与计算机图形学领域。尽管提出了残差控制、知识蒸馏等技术，但其核心任务与文档图像理解（DIU）无关，无法迁移到DocVQA、OCR、版面分析等典型文档任务中。且3D视角合成与文档的高分辨率处理、文本识别、逻辑推理等痛点无直接关联，属于负面清单中的‘无关领域’。
Abstract Neural Radiance Fields (NeRFs) have demonstrated remarkable capabilities in 3D reconstruction and novel view synthesis. However, most existing NeRF frameworks require complete retraining when new views are introduced incrementally, limiting their applicability in domains where data arrives sequentially. This limitation is particularly problematic in satellite-based terrain analysis, where regions are repeatedly observed over time. Incremental refinement of NeRFs remains underexplored, and naive approaches suffer from catastrophic forgetting when past data is unavailable. We propose $Δ$-NeRF, a unique modular residual framework for incremental NeRF refinement. $Δ$-NeRF introduces several novel techniques including: (1) a residual controller that injects per-layer corrections into a frozen base NeRF, enabling refinement without access to past data; (2) an uncertainty-aware gating mechanism that prevents overcorrection by adaptively combining base and refined predictions; and (3) a view selection strategy that reduces training data by up to 47\% while maintaining performance. Additionally, we employ knowledge distillation to compress the enhanced model into a compact student network (20\% of original size). Experiments on satellite imagery demonstrate that $Δ$-NeRF achieves performance comparable to joint training while reducing training time by 30-42\%. $Δ$-NeRF consistently outperforms existing baselines, achieving an improvement of up to 43.5\% in PSNR over naive fine-tuning and surpassing joint training on some metrics.
Authors Kriti Ghosh, Devjyoti Chakraborty, Lakshmish Ramaswamy, Suchendra M. Bhandarkar, In Kee Kim, Nancy O'Hare, Deepak Mishra
Categories Computer Vision and Pattern Recognition
0
A Simple Framework Towards Vision-based Traffic Signal Control with Microscopic Simulation Transportation Research Board (TRB) 105th Annual Meeting
Title CN 一种基于视觉的交通信号控制与微观仿真相结合的简单框架
Keywords 交通信号控制 · 计算机视觉 · 微观仿真 · 强化学习 · 智能交通
Summary 本文提出了一种名为TrafficDojo的框架，用于基于视觉的交通信号控制研究，通过将SUMO中的微观交通流集成到3D驾驶模拟器MetaDrive中，构建了一个可用于评估不同交通信号控制器的仿真环境。作者比较了传统方法与强化学习方法作为基线。该工作聚焦于智能交通系统，而非文档理解或多模态AI在文本密集场景的应用。
Reason 该论文属于具身智能与交通控制交叉领域，研究的是基于视觉的交通信号控制，使用驾驶模拟器进行微观光流建模。尽管涉及计算机视觉，但其任务目标、方法设计和应用场景均与文档图像理解（DIU）或视觉语言模型在文档上的应用无任何关联。无法迁移其方法解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract Traffic signal control (TSC) is crucial for reducing traffic congestion leading to smoother traffic flow, reduced idle time, and mitigated CO2 emissions. In this paper, we explore the computer vision approach for TSC that modulates on-road traffic flows through visual observation. Unlike traditional feature-based approaches, vision-based methods depend much less on heuristics and predefined features, bringing promising potentials for end-to-end learning and optimization of traffic signals. Thus, we introduce a simple traffic simulation framework called TrafficDojo towards vision-based TSC and its benchmark by integrating the microscopic traffic flow provided in SUMO into the 3D driving simulator MetaDrive. This proposed framework offers a versatile traffic environment for in-depth analysis and comprehensive evaluation of traffic signal controllers across diverse traffic conditions and scenarios. We establish and compare baseline algorithms including both traditional and Reinforcement Learning (RL) approaches. This work sheds light on the design and development of vision-based TSC approaches and opens up new research opportunities
Authors Pan He, Quanyi Li, Xiaoyong Yuan, Bolei Zhou
Comment Accepted for presentation at the Transportation Research Board (TRB) 105th Annual Meeting
Categories Computer Vision and Pattern Recognition
0
One Patch is All You Need: Joint Surface Material Reconstruction and Classification from Minimal Visual Cues
Title CN 一个图块即足矣：从最少视觉线索中联合进行表面材料重建与分类
Keywords 表面材料识别 · 部分卷积 · 图像修复 · 稀疏输入 · 视觉感知
Summary 本文提出SMARC模型，仅通过图像中一个连续的10%图块即可实现对表面材质的完整RGB重建与材质类别分类。该方法结合部分卷积U-Net与分类头，在极端稀疏观测条件下实现了空间修复与语义理解的统一。在Touch and Go数据集上优于多种主流模型，取得了17.55 dB的PSNR和85.10%的分类准确率。
Reason 该论文研究的是表面材料的重建与分类问题，属于物体表面感知和稀疏视觉推理领域，尽管使用了部分卷积等技术处理缺失数据，但其任务与文档图像理解（DIU）无直接关联。核心目标并非解决OCR幻觉、密集文本识别、复杂版式分析或长文档推理等DIU痛点，且未提出可迁移至文档理解领域的通用方法（如视觉压缩、多模态对齐、高分辨率处理架构等）。因此属于无关领域。
Abstract Understanding material surfaces from sparse visual cues is critical for applications in robotics, simulation, and material perception. However, most existing methods rely on dense or full-scene observations, limiting their effectiveness in constrained or partial view environment. To address this challenge, we introduce SMARC, a unified model for Surface MAterial Reconstruction and Classification from minimal visual input. By giving only a single 10% contiguous patch of the image, SMARC recognizes and reconstructs the full RGB surface while simultaneously classifying the material category. Our architecture combines a Partial Convolutional U-Net with a classification head, enabling both spatial inpainting and semantic understanding under extreme observation sparsity. We compared SMARC against five models including convolutional autoencoders [17], Vision Transformer (ViT) [13], Masked Autoencoder (MAE) [5], Swin Transformer [9], and DETR [2] using Touch and Go dataset [16] of real-world surface textures. SMARC achieves state-of-the-art results with a PSNR of 17.55 dB and a material classification accuracy of 85.10%. Our findings highlight the advantages of partial convolution in spatial reasoning under missing data and establish a strong foundation for minimal-vision surface understanding.
Authors Sindhuja Penchala, Gavin Money, Gabriel Marques, Samuel Wood, Jessica Kirschman, Travis Atkison, Shahram Rahimi, Noorbakhsh Amiri Golilarz
Comment 9 pages,3 figures, 5 tables
Categories Computer Vision and Pattern Recognition
0
PointNSP: Autoregressive 3D Point Cloud Generation with Next-Scale Level-of-Detail Prediction
Title CN PointNSP：通过下一尺度细节层次预测实现自回归3D点云生成
Keywords 3D点云生成 · 自回归模型 · 多尺度生成 · 细节层次 · Shape Modeling
Summary 本文提出PointNSP，一种基于细节层次（LOD）思想的自回归3D点云生成框架，通过从粗到细的方式逐步预测更高分辨率的几何结构，解决了传统自回归模型因强制排序导致的全局结构捕捉能力弱的问题。在ShapeNet上实现了自回归方法中的SOTA生成质量，并在效率上优于扩散模型。
Reason 该论文研究的是3D点云生成，属于3D视觉与形状建模领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。尽管其提出的多尺度生成框架具有方法论价值，但核心问题设定（无序点集的生成、LOD建模）难以迁移到文档图像中的文本识别、布局分析或视觉推理等场景。命中负面清单第2条：无关领域（3D视觉），因此直接打0分。
Abstract Autoregressive point cloud generation has long lagged behind diffusion-based approaches in quality. The performance gap stems from the fact that autoregressive models impose an artificial ordering on inherently unordered point sets, forcing shape generation to proceed as a sequence of local predictions. This sequential bias emphasizes short-range continuity but undermines the model's capacity to capture long-range dependencies, hindering its ability to enforce global structural properties such as symmetry, consistent topology, and large-scale geometric regularities. Inspired by the level-of-detail (LOD) principle in shape modeling, we propose PointNSP, a coarse-to-fine generative framework that preserves global shape structure at low resolutions and progressively refines fine-grained geometry at higher scales through a next-scale prediction paradigm. This multi-scale factorization aligns the autoregressive objective with the permutation-invariant nature of point sets, enabling rich intra-scale interactions while avoiding brittle fixed orderings. Experiments on ShapeNet show that PointNSP establishes state-of-the-art (SOTA) generation quality for the first time within the autoregressive paradigm. In addition, it surpasses strong diffusion-based baselines in parameter, training, and inference efficiency. Finally, in dense generation with 8,192 points, PointNSP's advantages become even more pronounced, underscoring its scalability potential.
Authors Ziqiao Meng, Qichao Wang, Zhiyang Dou, Zixing Song, Zhipeng Zhou, Irwin King, Peilin Zhao
Comment 24 pages; Previously this version appeared as arXiv:2510.05613 which was submitted as a new work by accident
Categories Computer Vision and Pattern Recognition
0
Automated Histopathologic Assessment of Hirschsprung Disease Using a Multi-Stage Vision Transformer Framework
Title CN 基于多阶段视觉Transformer框架的先天性巨结肠病自动组织病理学评估
Keywords 医学图像分割 · 视觉Transformer · Hirschsprung病 · 数字病理学 · 细胞识别
Summary 本文提出了一种基于ViT-B/16的三阶段分割框架，用于在全切片图像中依次分割肌层、神经丛并识别神经节细胞，模拟病理学家的诊断流程。实验结果显示在多个指标上表现良好，具备辅助数字病理工作的潜力。但该研究局限于医疗影像领域，方法未涉及多模态语言模型、文本理解或文档结构建模。
Reason 该论文属于医学图像分析领域的特定应用，研究任务为病理切片中的细胞结构分割与识别，虽使用了Vision Transformer等计算机视觉技术，但其核心是面向临床诊断的生物医学工程工作，与文档图像理解（DIU）、DocVQA、OCR、版面分析等任务无直接关联。未提出可用于解决文档理解中OCR幻觉、复杂排版或长文档推理等问题的方法论创新。
Abstract Hirschsprung Disease is characterized by the absence of ganglion cells in the myenteric plexus. Therefore, their correct identification is crucial for diagnosing Hirschsprung disease. We introduce a three-stage segmentation framework based on a Vision Transformer (ViT-B/16) that mimics the pathologist's diagnostic approach. The framework sequentially segments the muscularis propria, delineates the myenteric plexus, and identifies ganglion cells within anatomically valid regions. 30 whole-slide images of colon tissue were used, each containing expert manual annotations of muscularis, plexus, and ganglion cells at varying levels of certainty. A 5-fold cross-validation scheme was applied to each stage, along with resolution-specific tiling strategies and tailored postprocessing to ensure anatomical consistency. The proposed method achieved a Dice coefficient of 89.9% and a Plexus Inclusion Rate of 100% for muscularis segmentation. Plexus segmentation reached a recall of 94.8%, a precision of 84.2% and a Ganglia Inclusion Rate of 99.7%. For high-certainty ganglion cells, the model achieved 62.1% precision and 89.1% recall, while joint certainty scores yielded 67.0% precision. These results indicate that ViT-based models are effective at leveraging global tissue context and capturing cellular morphology at small scales, even within complex histological tissue structures. This multi-stage methodology has great potential to support digital pathology workflows by reducing inter-observer variability and assisting in the evaluation of Hirschsprung disease. The clinical impact will be evaluated in future work with larger multi-center datasets and additional expert annotations.
Authors Youssef Megahed, Saleh Abou-Alwan, Anthony Fuller, Dina El Demellawy, Steven Hawken, Adrian D. C. Chan
Comment 16 pages, 8 figures, 6 tables
Categories Computer Vision and Pattern Recognition
0
Video Object Recognition in Mobile Edge Networks: Local Tracking or Edge Detection?
Title CN 移动边缘网络中的视频目标识别：本地跟踪还是边缘检测？
Keywords 视频目标识别 · 移动边缘计算 · 目标跟踪 · 计算卸载 · 深度强化学习
Summary 本文提出了一种基于深度强化学习的自适应算法LTED-Ada，用于在移动边缘网络中动态选择本地目标跟踪或边缘服务器上的目标检测，以平衡识别精度、延迟和计算负载。多设备场景下结合联邦学习提升策略泛化能力。实验基于树莓派硬件平台验证了方法的有效性。
Reason 该论文研究的是视频流中目标识别的计算卸载策略，属于视频分析与移动边缘计算交叉领域，核心关注点为资源受限设备上的实时性与准确性权衡。其方法论聚焦于强化学习驱动的检测-跟踪调度机制，并非针对文档图像理解中的OCR幻觉、密集文本解析或复杂版面结构建模等问题。尽管涉及视觉识别，但任务场景（视频目标识别）、输入模态（连续帧）和系统架构（边缘计算）均与文档图像理解（DIU）无关。
Abstract Fast and accurate video object recognition, which relies on frame-by-frame video analytics, remains a challenge for resource-constrained devices such as traffic cameras. Recent advances in mobile edge computing have made it possible to offload computation-intensive object detection to edge servers equipped with high-accuracy neural networks, while lightweight and fast object tracking algorithms run locally on devices. This hybrid approach offers a promising solution but introduces a new challenge: deciding when to perform edge detection versus local tracking. To address this, we formulate two long-term optimization problems for both single-device and multi-device scenarios, taking into account the temporal correlation of consecutive frames and the dynamic conditions of mobile edge networks. Based on the formulation, we propose the LTED-Ada in single-device setting, a deep reinforcement learning-based algorithm that adaptively selects between local tracking and edge detection, according to the frame rate as well as recognition accuracy and delay requirement. In multi-device setting, we further enhance LTED-Ada using federated learning to enable collaborative policy training across devices, thereby improving its generalization to unseen frame rates and performance requirements. Finally, we conduct extensive hardware-in-the-loop experiments using multiple Raspberry Pi 4B devices and a personal computer as the edge server, demonstrating the superiority of LTED-Ada.
Authors Kun Guo, Yun Shen, Xijun Wang, Chaoqun You, Yun Rui, Tony Q. S. Quek
Categories Computer Vision and Pattern Recognition
0
From Words to Wisdom: Discourse Annotation and Baseline Models for Student Dialogue Understanding
Title CN 从词语到智慧：学生对话理解的语篇标注与基线模型
Keywords 对话理解 · 语篇分析 · 教育数据 · NLP · 大语言模型
Summary 本文提出一个用于学生对话中知识建构语篇特征识别的标注数据集，并基于GPT-3.5和Llama-3.1等大语言模型建立基线。研究聚焦于教育场景下的纯文本对话分析，未引入视觉模态或文档结构信息。
Reason 该论文属于纯NLP领域的教育对话分析，研究任务为学生对话中的语篇特征识别，不涉及任何文档图像、视觉信息或图文多模态内容。与文档图像理解（DIU）和视觉语言模型（VLM）在文档上的应用完全无关，属于‘无关领域’类别。
Abstract Identifying discourse features in student conversations is quite important for educational researchers to recognize the curricular and pedagogical variables that cause students to engage in constructing knowledge rather than merely completing tasks. The manual analysis of student conversations to identify these discourse features is time-consuming and labor-intensive, which limits the scale and scope of studies. Leveraging natural language processing (NLP) techniques can facilitate the automatic detection of these discourse features, offering educational researchers scalable and data-driven insights. However, existing studies in NLP that focus on discourse in dialogue rarely address educational data. In this work, we address this gap by introducing an annotated educational dialogue dataset of student conversations featuring knowledge construction and task production discourse. We also establish baseline models for automatically predicting these discourse properties for each turn of talk within conversations, using pre-trained large language models GPT-3.5 and Llama-3.1. Experimental results indicate that these state-of-the-art models perform suboptimally on this task, indicating the potential for future research.
Authors Farjana Sultana Mim, Shuchin Aeron, Eric Miller, Kristen Wendell
Categories Computation and Language
0
Bridging the Language Gap: Synthetic Voice Diversity via Latent Mixup for Equitable Speech Recognition ICML 2025 Workshop on Machine Learning for Audio
Title CN 弥合语言鸿沟：通过潜在空间混合实现合成语音多样性以促进公平的语音识别
Keywords 语音识别 · 数据增强 · 低资源语言 · 潜在空间混合 · 公平性
Summary 本文提出一种基于潜在空间混合（Latent Mixup）的数据增强方法，用于提升低资源语言在自动语音识别中的性能。通过在潜在表示空间中混合不同语音样本，生成多样化的合成语音数据，从而缩小高资源与低资源语言之间的性能差距。实验表明该方法优于现有数据增强策略。
Reason 该论文研究的是语音识别中的低资源语言数据增强问题，属于纯语音处理领域，与文档图像理解（DIU）或视觉-语言模型在文档上的应用无直接关联。尽管其数据增强思想具有一定通用性，但方法针对音频信号设计，难以迁移到文档图像理解所面临的OCR幻觉、复杂布局解析或细粒度视觉定位等核心痛点。
Abstract Modern machine learning models for audio tasks often exhibit superior performance on English and other well-resourced languages, primarily due to the abundance of available training data. This disparity leads to an unfair performance gap for low-resource languages, where data collection is both challenging and costly. In this work, we introduce a novel data augmentation technique for speech corpora designed to mitigate this gap. Through comprehensive experiments, we demonstrate that our method significantly improves the performance of automatic speech recognition systems on low-resource languages. Furthermore, we show that our approach outperforms existing augmentation strategies, offering a practical solution for enhancing speech technology in underrepresented linguistic communities.
Authors Wesley Bian, Xiaofeng Lin, Guang Cheng
Comment Accepted at ICML 2025 Workshop on Machine Learning for Audio
Categories Computation and Language
0
Adversarial Confusion Attack: Disrupting Multimodal Large Language Models
Title CN 对抗性混淆攻击：破坏多模态大语言模型
Keywords 对抗攻击 · 多模态大模型 · 熵最大化 · 模型鲁棒性 · PGD
Summary 本文提出一种名为‘对抗性混淆攻击’的新威胁，旨在通过最大化下一词元熵的方式，使多模态大语言模型生成不连贯或自信错误的输出。攻击利用开源MLLM的小型集成生成对抗样本，在白盒设置下可有效干扰多种模型，包括Qwen3-VL和GPT-5.1。其应用场景为在网页中嵌入对抗图像以破坏MLLM代理的可靠性。
Reason 该论文研究的是针对多模态大语言模型的对抗性攻击方法，属于安全与鲁棒性领域，其目标是诱导模型产生混乱输出，而非提升文档理解能力。尽管涉及MLLM，但工作重点在于攻击机制设计，与文档图像理解（DIU）的核心任务如OCR、布局分析、表格识别等无直接关联。且未提出可迁移至DIU领域的通用技术工具（如推理增强、视觉表征学习等），属于无关领域研究。
Abstract We introduce the Adversarial Confusion Attack, a new class of threats against multimodal large language models (MLLMs). Unlike jailbreaks or targeted misclassification, the goal is to induce systematic disruption that makes the model generate incoherent or confidently incorrect outputs. Applications include embedding adversarial images into websites to prevent MLLM-powered agents from operating reliably. The proposed attack maximizes next-token entropy using a small ensemble of open-source MLLMs. In the white-box setting, we show that a single adversarial image can disrupt all models in the ensemble, both in the full-image and adversarial CAPTCHA settings. Despite relying on a basic adversarial technique (PGD), the attack generates perturbations that transfer to both unseen open-source (e.g., Qwen3-VL) and proprietary (e.g., GPT-5.1) models.
Authors Jakub Hoscilowicz, Artur Janicki
Categories Computation and Language
0
A Task-Oriented Evaluation Framework for Text Normalization in Modern NLP Pipelines
Title CN 现代NLP流水线中文本归一化的任务导向评估框架
Keywords text normalization · stemming · evaluation framework · downstream performance · semantic similarity
Summary 本文提出了一种任务导向的词干提取评估框架，结合了词干有效性得分（SES）、下游任务性能变化（MPD）和语义相似性（ANLD）三个指标，用于评估不同语言（包括孟加拉语和英语）中词干提取的效果。结果显示过度词干化会导致语义失真并损害下游任务性能。
Reason 该论文聚焦于文本归一化（特别是词干提取）的评估方法，属于纯NLP预处理范畴，且研究对象包含小语种孟加拉语。虽然提出了任务导向的评估框架，但其核心问题与文档图像理解（DIU）中的OCR幻觉、布局分析、多模态对齐等关键挑战无关，无法迁移用于解决文档视觉问答或复杂排版理解中的根本问题。
Abstract Text normalization is an essential preprocessing step in many natural language processing (NLP) tasks, and stemming is one such normalization technique that reduces words to their base or root form. However, evaluating stemming methods is challenging because current evaluation approaches are limited and do not capture the potential harm caused by excessive stemming; therefore, it is essential to develop new approaches to evaluate stemming methods. To address this issue, this study propose a novel, task-oriented approach to evaluate stemming methods, which considers three aspects: (1) the utility of stemming using Stemming Effectiveness Score (SES), (2) the impact of stemming on downstream tasks using Model Performance Delta (MPD), and (3) the semantic similarity between stemmed and original words using Average Normalized Levenshtein Distance (ANLD), thus providing a comprehensive evaluation framework. We apply our evaluation framework to compare two stemmers for Bangla (BNLTK) and English (Snowball), and our results reveal a significant issue, prompting us to analyze their performance in detail. While the Bangla stemmer achieves the highest SES (1.67) due to effective word reduction (CR = 1.90), SES alone is insufficient because our proposed safety measure, ANLD, reveals that this high SES is due to harmful over-stemming (ANLD = 0.26), which correlates with the observed decrease in downstream performance.In contrast, the English stemmer achieves a moderate SES (1.31) with a safe meaning distance (ANLD = 0.14), allowing its word reduction to contribute positively to downstream performance; therefore, it is a more reliable stemmer. Our study provides a valuable tool for distinguishing between potential efficiency gains (high SES) and meaning preservation (low ANLD).
Authors Md Abdullah Al Kafi, Raka Moni, Sumit Kumar Banshal
Categories Computation and Language
0
EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysis
Title CN EHR-R1：一种用于电子健康记录分析的推理增强型基础语言模型
Keywords 电子健康记录 · 推理增强 · 指令数据集 · 领域适应 · 临床决策
Summary 本文提出EHR-Ins数据集、EHR-R1模型和EHR-Bench基准，旨在提升大模型在电子健康记录分析中的推理能力。通过构建大规模推理指令数据，采用多阶段训练提升模型在42个EHR任务上的表现，在多个指标上超越GPT-4o等主流模型。
Reason 该论文属于典型的‘平行下游应用’，其核心是将现有LLM技术应用于医疗领域的电子健康记录（EHR）分析，构建了特定领域的数据集和模型。尽管提出了多阶段训练和推理增强机制，但并未引入可迁移至文档图像理解（DIU）领域的底层方法创新，如视觉表示学习、多模态对齐、OCR去幻觉或高分辨率处理等。任务局限于纯文本EHR分析，不涉及图像或文档布局信息，无法解决DIU中的核心痛点。
Abstract Electronic Health Records (EHRs) contain rich yet complex information, and their automated analysis is critical for clinical decision-making. Despite recent advances of large language models (LLMs) in clinical workflows, their ability to analyze EHRs remains limited due to narrow task coverage and lack of EHR-oriented reasoning capabilities. This paper aims to bridge the gap, specifically, we present EHR-Ins, a large-scale, comprehensive EHR reasoning instruction dataset, comprising 300k high-quality reasoning cases and 4M non-reasoning cases across 42 distinct EHR tasks. Its core innovation is a thinking-graph-driven framework that enables to generate high-quality reasoning data at scale. Based on it, we develop EHR-R1, a series of reasoning-enhanced LLMs with up to 72B parameters tailored for EHR analysis. Through a multi-stage training paradigm, including domain adaptation, reasoning enhancement, and reinforcement learning, EHR-R1 systematically acquires domain knowledge and diverse reasoning capabilities, enabling accurate and robust EHR analysis. Lastly, we introduce EHR-Bench, a new benchmark curated from MIMIC-IV, spanning 42 tasks, to comprehensively assess reasoning and prediction across EHR scenarios. In experiments, we show that the resulting EHR-R1 consistently outperforms state-of-the-art commercial and open-source LLMs (including DeepSeek-V3 and GPT-4o), surpassing GPT-4o by over 30 points on MIMIC-Bench and achieving a 10\% higher zero-shot AUROC on EHRSHOT. Collectively, EHR-Ins, EHR-R1, and EHR-Bench have significantly advanced the development for more reliable and clinically relevant EHR analysis.
Authors Yusheng Liao, Chaoyi Wu, Junwei Liu, Shuyang Jiang, Pengcheng Qiu, Haowen Wang, Yun Yue, Shuai Zhen, Jian Wang, Qianrui Fan, Jinjie Gu, Ya Zhang, Yanfeng Wang, Yu Wang, Weidi Xie
Categories Computation and Language
0
LiRA: A Multi-Agent Framework for Reliable and Readable Literature Review Generation
Title CN LiRA：一种用于生成可靠且可读文献综述的多智能体框架
Keywords 多智能体系统 · 文献综述生成 · 自动化写作 · LLM应用 · 文本生成
Summary 本文提出了LiRA，一个基于多智能体的协作框架，用于自动生成科学文献综述。系统通过分工明确的智能体完成提纲生成、段落撰写、编辑和评审等步骤，模拟人类撰写过程。在多个数据集上验证了其在写作品质和引用准确性方面优于现有方法。然而，该工作未涉及视觉内容解析或文档图像理解任务。
Reason 该论文属于典型的‘平行下游应用’，其核心是将多智能体架构应用于科学文献综述写作这一特定任务。尽管涉及多智能体协作流程，但并未提出可用于文档图像理解（如OCR去幻觉、复杂排版处理、细粒度定位等）的底层方法或可迁移的技术机制。研究聚焦于文本生成阶段的自动化写作，与文档视觉结构理解、多模态建模或高分辨率视觉输入处理无关。
Abstract The rapid growth of scientific publications has made it increasingly difficult to keep literature reviews comprehensive and up-to-date. Though prior work has focused on automating retrieval and screening, the writing phase of systematic reviews remains largely under-explored, especially with regard to readability and factual accuracy. To address this, we present LiRA (Literature Review Agents), a multi-agent collaborative workflow which emulates the human literature review process. LiRA utilizes specialized agents for content outlining, subsection writing, editing, and reviewing, producing cohesive and comprehensive review articles. Evaluated on SciReviewGen and a proprietary ScienceDirect dataset, LiRA outperforms current baselines such as AutoSurvey and MASS-Survey in writing and citation quality, while maintaining competitive similarity to human-written reviews. We further evaluate LiRA in real-world scenarios using document retrieval and assess its robustness to reviewer model variation. Our findings highlight the potential of agentic LLM workflows, even without domain-specific tuning, to improve the reliability and usability of automated scientific writing.
Authors Gregory Hok Tjoan Go, Khang Ly, Anders Søgaard, Amin Tabatabaei, Maarten de Rijke, Xinyi Chen
Categories Computation and Language
0
Computational Turing Test Reveals Systematic Differences Between Human and AI Language
Title CN 计算图灵测试揭示人类与人工智能语言的系统性差异
Keywords 大语言模型 · 图灵测试 · 语言真实性 · 社会模拟 · 文本生成
Summary 本文提出一种计算图灵测试框架，结合BERT-based可检测性与语义相似度指标以及可解释的语言特征，评估大语言模型在模拟人类语言方面的逼真程度。作者在多个社交平台数据上评测了九种开源LLM，并比较不同校准策略的效果。结果显示，即使经过校准，AI生成文本仍显著区别于人类文本，尤其在情感表达方面；且指令微调和模型扩容并未提升类人程度，存在类人化与语义保真之间的权衡。
Reason 该论文属于纯NLP领域，研究的是LLM生成文本与人类语言在社交媒体语境下的可区分性，关注点为情感表达、风格标记和主题模式等语言特性。未涉及任何视觉模态或文档图像理解（DIU）任务，如DocVQA、布局分析、表格识别等。其方法论聚焦于文本层面的检测与校准，无法迁移解决OCR幻觉、细粒度定位、高分辨率文档处理等DIU核心痛点。
Abstract Large language models (LLMs) are increasingly used in the social sciences to simulate human behavior, based on the assumption that they can generate realistic, human-like text. Yet this assumption remains largely untested. Existing validation efforts rely heavily on human-judgment-based evaluations -- testing whether humans can distinguish AI from human output -- despite evidence that such judgments are blunt and unreliable. As a result, the field lacks robust tools for assessing the realism of LLM-generated text or for calibrating models to real-world data. This paper makes two contributions. First, we introduce a computational Turing test: a validation framework that integrates aggregate metrics (BERT-based detectability and semantic similarity) with interpretable linguistic features (stylistic markers and topical patterns) to assess how closely LLMs approximate human language within a given dataset. Second, we systematically compare nine open-weight LLMs across five calibration strategies -- including fine-tuning, stylistic prompting, and context retrieval -- benchmarking their ability to reproduce user interactions on X (formerly Twitter), Bluesky, and Reddit. Our findings challenge core assumptions in the literature. Even after calibration, LLM outputs remain clearly distinguishable from human text, particularly in affective tone and emotional expression. Instruction-tuned models underperform their base counterparts, and scaling up model size does not enhance human-likeness. Crucially, we identify a trade-off: optimizing for human-likeness often comes at the cost of semantic fidelity, and vice versa. These results provide a much-needed scalable framework for validation and calibration in LLM simulations -- and offer a cautionary note about their current limitations in capturing human communication.
Authors Nicolò Pagan, Petter Törnberg, Christopher A. Bail, Anikó Hannák, Christopher Barrie
Categories Computation and Language
0
KyrgyzBERT: A Compact, Efficient Language Model for Kyrgyz NLP
Title CN KyrgyzBERT：一种紧凑高效的吉尔吉斯语NLP语言模型
Keywords KyrgyzBERT · 低资源语言 · BERT · 情感分析 · 小语种
Summary 本文提出了首个面向吉尔吉斯语的BERT模型KyrgyzBERT，并构建了相应的 sentiment analysis 数据集 kyrgyz-sst2 进行评估。模型参数量为3590万，在情感分析任务上表现良好，甚至优于更大规模的mBERT。所有资源已公开以支持吉尔吉斯语NLP研究。
Reason 该论文专注于小语种（吉尔吉斯语）的单语语言模型构建，属于负面清单中的‘小语种’范畴。尽管在低资源语言建模上有一定意义，但其方法为标准的BERT架构微调与数据集构建，并无针对文档图像理解（DIU）或视觉-语言多模态任务的创新，且不涉及OCR、布局分析、表格识别等核心DIU问题，无法迁移用于解决文档理解中的幻觉、复杂排版或长文档推理等痛点。
Abstract Kyrgyz remains a low-resource language with limited foundational NLP tools. To address this gap, we introduce KyrgyzBERT, the first publicly available monolingual BERT-based language model for Kyrgyz. The model has 35.9M parameters and uses a custom tokenizer designed for the language's morphological structure. To evaluate performance, we create kyrgyz-sst2, a sentiment analysis benchmark built by translating the Stanford Sentiment Treebank and manually annotating the full test set. KyrgyzBERT fine-tuned on this dataset achieves an F1-score of 0.8280, competitive with a fine-tuned mBERT model five times larger. All models, data, and code are released to support future research in Kyrgyz NLP.
Authors Adilet Metinov, Gulida M. Kudakeeva, Gulnara D. Kabaeva
Comment 3 pages, 1 figure, 2 tables. Preprint
Categories Computation and Language
0
Mispronunciation Detection and Diagnosis Without Model Training: A Retrieval-Based Approach
Title CN 无需模型训练的发音错误检测与诊断：一种基于检索的方法
Keywords 发音错误检测 · 无训练方法 · 语音识别 · 检索方法 · 语言学习
Summary 本文提出了一种无需训练的发音错误检测与诊断方法，利用预训练的语音识别模型结合检索技术来识别非母语者的发音错误。方法避免了传统方法中对音素级建模或特定任务训练的需求，在L2-ARCTIC数据集上取得了69.60%的F1分数。整个工作局限于语音信号处理范畴，不涉及任何视觉或文档内容。
Reason 该论文属于语音处理与语言学习领域，研究任务为发音错误检测（MDD），与文档图像理解（DIU）或视觉-语言模型（VLM）在文档上的应用完全无关。未涉及OCR、文档布局分析、表格识别、视觉表示学习或任何与图像模态相关的内容。属于无关领域，命中负面清单第2条。
Abstract Mispronunciation Detection and Diagnosis (MDD) is crucial for language learning and speech therapy. Unlike conventional methods that require scoring models or training phoneme-level models, we propose a novel training-free framework that leverages retrieval techniques with a pretrained Automatic Speech Recognition model. Our method avoids phoneme-specific modeling or additional task-specific training, while still achieving accurate detection and diagnosis of pronunciation errors. Experiments on the L2-ARCTIC dataset show that our method achieves a superior F1 score of 69.60% while avoiding the complexity of model training.
Authors Huu Tuong Tu, Ha Viet Khanh, Tran Tien Dat, Vu Huan, Thien Van Luong, Nguyen Tien Cuong, Nguyen Thi Thu Trang
Categories Computation and Language
0
EM2LDL: A Multilingual Speech Corpus for Mixed Emotion Recognition through Label Distribution Learning IEEE Transactions on Affective computing
Title CN EM2LDL：一种通过标签分布学习进行混合情感识别的多语言语音语料库
Keywords 情感识别 · 多语言语料库 · 标签分布学习 · 语音处理 · 自监督学习
Summary 本文提出了一个名为EM2LDL的多语言语音语料库，用于推动基于标签分布学习的混合情感识别。该语料库包含英语、普通话和粤语的表达性语句，涵盖多语种地区常见的语码转换现象，并采用32类细粒度情绪分布进行标注。实验基于自监督学习模型开展，在不同说话人属性上验证了基线方法的有效性。该工作旨在促进情感计算领域中更具适应性和共情能力系统的开发。
Reason 该论文属于语音情感识别与计算情感领域，研究内容为多语言语音数据集构建及混合情感标注，与文档图像理解（DIU）、视觉语言模型（VLM）或OCR等核心技术无关。未涉及任何文档结构、文本布局、视觉-文本对齐或多模态文档推理问题，无法迁移解决DIU中的OCR幻觉、密集文本处理或复杂排版理解等痛点。
Abstract This study introduces EM2LDL, a novel multilingual speech corpus designed to advance mixed emotion recognition through label distribution learning. Addressing the limitations of predominantly monolingual and single-label emotion corpora \textcolor{black}{that restrict linguistic diversity, are unable to model mixed emotions, and lack ecological validity}, EM2LDL comprises expressive utterances in English, Mandarin, and Cantonese, capturing the intra-utterance code-switching prevalent in multilingual regions like Hong Kong and Macao. The corpus integrates spontaneous emotional expressions from online platforms, annotated with fine-grained emotion distributions across 32 categories. Experimental baselines using self-supervised learning models demonstrate robust performance in speaker-independent gender-, age-, and personality-based evaluations, with HuBERT-large-EN achieving optimal results. By incorporating linguistic diversity and ecological validity, EM2LDL enables the exploration of complex emotional dynamics in multilingual settings. This work provides a versatile testbed for developing adaptive, empathetic systems for applications in affective computing, including mental health monitoring and cross-cultural communication. The dataset, annotations, and baseline codes are publicly available at https://github.com/xingfengli/EM2LDL.
Authors Xingfeng Li, Xiaohan Shi, Junjie Li, Yongwei Li, Masashi Unoki, Tomoki Toda, Masato Akagi
Comment Submitted to IEEE Transactions on Affective computing
Categories Computation and Language
0
QiMeng-Kernel: Macro-Thinking Micro-Coding Paradigm for LLM-Based High-Performance GPU Kernel Generation AAAI 2026
Title CN QiMeng-Kernel：基于大语言模型的高性能GPU内核生成的宏观思维-微观编码范式
Keywords GPU内核生成 · 大语言模型 · 代码生成 · 强化学习 · 程序优化
Summary 本文提出了一种名为Macro Thinking Micro Coding（MTMC）的分层框架，用于提升大语言模型在生成高性能GPU内核代码时的准确性和效率。该方法将优化策略（宏观思维）与具体实现（微观编码）解耦：前者通过强化学习引导轻量级LLM探索高效的语义优化策略，后者由通用LLM逐步实现具体代码。实验表明，该方法在KernelBench和TritonBench上显著优于现有LLM方法和专家手工优化的PyTorch内核。
Reason 该论文属于代码生成与高性能计算领域，核心是利用LLM自动生成优化的GPU内核代码。尽管使用了LLM和分层推理框架（MTMC），但其任务完全不涉及文档图像、多模态理解或视觉信息处理。属于典型的‘平行应用’类工作——将LLM用于特定工程问题（kernel生成），而非解决文档理解中的OCR幻觉、复杂排版、细粒度定位等核心挑战。无法迁移为DIU的方法论工具。
Abstract Developing high-performance GPU kernels is critical for AI and scientific computing, but remains challenging due to its reliance on expert crafting and poor portability. While LLMs offer promise for automation, both general-purpose and finetuned LLMs suffer from two fundamental and conflicting limitations: correctness and efficiency. The key reason is that existing LLM-based approaches directly generate the entire optimized low-level programs, requiring exploration of an extremely vast space encompassing both optimization policies and implementation codes. To address the challenge of exploring an intractable space, we propose Macro Thinking Micro Coding (MTMC), a hierarchical framework inspired by the staged optimization strategy of human experts. It decouples optimization strategy from implementation details, ensuring efficiency through high-level strategy and correctness through low-level implementation. Specifically, Macro Thinking employs reinforcement learning to guide lightweight LLMs in efficiently exploring and learning semantic optimization strategies that maximize hardware utilization. Micro Coding leverages general-purpose LLMs to incrementally implement the stepwise optimization proposals from Macro Thinking, avoiding full-kernel generation errors. Together, they effectively navigate the vast optimization space and intricate implementation details, enabling LLMs for high-performance GPU kernel generation. Comprehensive results on widely adopted benchmarks demonstrate the superior performance of MTMC on GPU kernel generation in both accuracy and running time. On KernelBench, MTMC achieves near 100% and 70% accuracy at Levels 1-2 and 3, over 50% than SOTA general-purpose and domain-finetuned LLMs, with up to 7.3x speedup over LLMs, and 2.2x over expert-optimized PyTorch Eager kernels. On the more challenging TritonBench, MTMC attains up to 59.64% accuracy and 34x speedup.
Authors Xinguo Zhu, Shaohui Peng, Jiaming Guo, Yunji Chen, Qi Guo, Yuanbo Wen, Hang Qin, Ruizhi Chen, Qirui Zhou, Ke Gao, Yanjun Wu, Chen Zhao, Ling Li
Comment 9 pages, 2 figures, accepted by AAAI 2026
Categories Computation and Language
0
Enhancing Reasoning Skills in Small Persian Medical Language Models Can Outperform Large-Scale Data Training
Title CN 增强小型波斯语医学语言模型的推理能力可优于大规模数据训练
Keywords 小语言模型 · 推理能力 · DPO · RLAIF · 波斯语
Summary 本文研究通过强化学习与AI反馈（RLAIF）及直接偏好优化（DPO）来提升小型波斯语医学语言模型的推理能力。作者构建了包含正确与错误思维链路径的数据集，并在有限数据下实现了超越大规模训练基线模型的效果。然而，该工作完全基于文本模态，不涉及图像或文档结构信息处理。
Reason 该论文聚焦于小语种（波斯语）医学领域的纯语言模型推理能力提升，属于‘小语种’且为特定领域（医疗）的语言模型应用，未涉及任何视觉模态或文档图像理解任务。尽管使用了DPO和RLAIF等训练方法，但其应用场景与文档图像理解（DIU）无关，无法迁移用于解决OCR幻觉、复杂排版或细粒度定位等核心问题。
Abstract Enhancing reasoning capabilities in small language models is critical for specialized applications such as medical question answering, particularly in underrepresented languages like Persian. In this study, we employ Reinforcement Learning with AI Feedback (RLAIF) and Direct preference optimization (DPO) to improve the reasoning skills of a general-purpose Persian language model. To achieve this, we translated a multiple-choice medical question-answering dataset into Persian and used RLAIF to generate rejected-preferred answer pairs, which are essential for DPO training. By prompting both teacher and student models to produce Chain-of-Thought (CoT) reasoning responses, we compiled a dataset containing correct and incorrect reasoning trajectories. This dataset, comprising 2 million tokens in preferred answers and 2.5 million tokens in rejected ones, was used to train a baseline model, significantly enhancing its medical reasoning capabilities in Persian. Remarkably, the resulting model outperformed its predecessor, gaokerena-V, which was trained on approximately 57 million tokens, despite leveraging a much smaller dataset. These results highlight the efficiency and effectiveness of reasoning-focused training approaches in developing domain-specific language models with limited data availability.
Authors Mehrdad Ghassabi, Sadra Hakim, Hamidreza Baradaran Kashani, Pedram Rostami
Comment 7 pages, 5 figures
Categories Computation and Language
0
Steganographic Backdoor Attacks in NLP: Ultra-Low Poisoning and Defense Evasion
Title CN 自然语言处理中的隐写后门攻击：超低污染与防御规避
Keywords 后门攻击 · 自然语言处理 · 隐写术 · 数据污染 · 防御规避
Summary 本文提出了一种名为SteganoBackdoor的隐写后门攻击方法，利用自然语言隐写技术将语义触发器隐藏在流畅文本中，实现极低数据污染率下的高成功率攻击，并能有效规避现有数据级防御机制。研究聚焦于NLP模型的安全漏洞，强调现实威胁建模的重要性。
Reason 该论文属于纯NLP安全领域，研究的是后门攻击与防御问题，尽管涉及模型鲁棒性，但其核心内容与文档图像理解（DIU）、多模态文档分析、OCR、VLM等完全无关。未涉及视觉模态、文档结构、布局分析或任何DIU相关任务，且不提供可迁移至文档理解领域的底层方法（如推理增强、视觉表示学习、高分辨率处理等）。属于负面清单中的‘无关领域’（纯NLP安全/对齐类），无营养的平行应用风险。
Abstract Transformer models are foundational to natural language processing (NLP) applications, yet remain vulnerable to backdoor attacks introduced through poisoned data, which implant hidden behaviors during training. To strengthen the ability to prevent such compromises, recent research has focused on designing increasingly stealthy attacks to stress-test existing defenses, pairing backdoor behaviors with stylized artifact or token-level perturbation triggers. However, this trend diverts attention from the harder and more realistic case: making the model respond to semantic triggers such as specific names or entities, where a successful backdoor could manipulate outputs tied to real people or events in deployed systems. Motivated by this growing disconnect, we introduce SteganoBackdoor, bringing stealth techniques back into line with practical threat models. Leveraging innocuous properties from natural-language steganography, SteganoBackdoor applies a gradient-guided data optimization process to transform semantic trigger seeds into steganographic carriers that embed a high backdoor payload, remain fluent, and exhibit no representational resemblance to the trigger. Across diverse experimental settings, SteganoBackdoor achieves over 99% attack success at an order-of-magnitude lower data-poisoning rate than prior approaches while maintaining unparalleled evasion against a comprehensive suite of data-level defenses. By revealing this practical and covert attack, SteganoBackdoor highlights an urgent blind spot in current defenses and demands immediate attention to adversarial data defenses and real-world threat modeling.
Authors Eric Xue, Ruiyi Zhang, Zijun Zhang, Pengtao Xie
Categories Computation and Language
0
Large Language Models in Argument Mining: A Survey
Title CN 大语言模型在论据挖掘中的应用：一项综述
Keywords 大语言模型 · 论据挖掘 · 提示学习 · 思维链 · 文本推理
Summary 本文是一篇关于大语言模型在论据挖掘领域应用的综述，回顾了LLM如何改变传统AM任务的范式，涵盖主张与证据检测、关系预测、立场分类等子任务，并讨论了提示工程、思维链和上下文学习对任务边界的影响。同时梳理了资源发展、评估方法及系统架构模式，指出长上下文推理、多模态鲁棒性等未来挑战。
Reason 该论文属于纯NLP领域的综述工作，聚焦于论据挖掘（Argument Mining）任务，未涉及任何文档图像、视觉输入或多模态理解内容。尽管提到了LLM的推理能力（如CoT），但其方法讨论完全基于文本模态，无法迁移解决文档图像理解中的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）。属于无关领域的平行应用综述，不符合DIU本题或关联‘军火库’的标准。
Abstract Large Language Models (LLMs) have fundamentally reshaped Argument Mining (AM), shifting it from a pipeline of supervised, task-specific classifiers to a spectrum of prompt-driven, retrieval-augmented, and reasoning-oriented paradigms. Yet existing surveys largely predate this transition, leaving unclear how LLMs alter task formulations, dataset design, evaluation methodology, and the theoretical foundations of computational argumentation. In this survey, we synthesise research and provide the first unified account of AM in the LLM era. We revisit canonical AM subtasks, i.e., claim and evidence detection, relation prediction, stance classification, argument quality assessment, and argumentative summarisation, and show how prompting, chain-of-thought reasoning, and in-context learning blur traditional task boundaries. We catalogue the rapid evolution of resources, including integrated multi-layer corpora and LLM-assisted annotation pipelines that introduce new opportunities as well as risks of bias and evaluation circularity. Building on this mapping, we identify emerging architectural patterns across LLM-based AM systems and consolidate evaluation practices spanning component-level accuracy, soft-label quality assessment, and LLM-judge reliability. Finally, we outline persistent challenges, including long-context reasoning, multimodal and multilingual robustness, interpretability, and cost-efficient deployment, and propose a forward-looking research agenda for LLM-driven computational argumentation.
Authors Hao Li, Viktor Schlegel, Yizheng Sun, Riza Batista-Navarro, Goran Nenadic
Comment Work draft
Categories Computation and Language
0
A Machine Learning Approach for Detection of Mental Health Conditions and Cyberbullying from Social Media AAAI-26 Bridge Program on AI for Medicine and Healthcare (AIMedHealth)
Title CN 一种基于机器学习的社交媒体中心理健康状况与网络欺凌检测方法
Keywords 心理健康检测 · 网络欺凌识别 · 社交媒体分析 · MentalBERT · 可解释性AI
Summary 本文提出一个用于从Twitter和Reddit数据中检测十类心理健康问题和网络欺凌行为的多分类框架，采用‘先划分后平衡’的数据处理流程，在平衡训练集上训练并在真实不平衡测试集上评估。实验表明，经过领域适配的MentalBERT模型表现最佳，准确率达0.92，Macro F1为0.76。作者还设计了一个结合SHAPLLM解释性的可视化仪表盘，供人工审核员使用。整体工作属于社会计算与心理健康交叉领域的应用型研究。
Reason 该论文属于平行下游应用，研究的是利用现有NLP和分类模型在特定社会媒体场景下检测心理健康问题和网络欺凌，任务本质为文本分类，未涉及文档图像理解（DIU）、视觉信息处理或OCR相关技术。其方法论聚焦于传统词法模型与预训练语言模型的比较，并无对文档图像、布局结构、多模态对齐或视觉幻觉等DIU核心问题的贡献，也无法迁移用于解决OCR错误、复杂排版解析或长文档推理等挑战。
Abstract Mental health challenges and cyberbullying are increasingly prevalent in digital spaces, necessitating scalable and interpretable detection systems. This paper introduces a unified multiclass classification framework for detecting ten distinct mental health and cyberbullying categories from social media data. We curate datasets from Twitter and Reddit, implementing a rigorous "split-then-balance" pipeline to train on balanced data while evaluating on a realistic, held-out imbalanced test set. We conducted a comprehensive evaluation comparing traditional lexical models, hybrid approaches, and several end-to-end fine-tuned transformers. Our results demonstrate that end-to-end fine-tuning is critical for performance, with the domain-adapted MentalBERT emerging as the top model, achieving an accuracy of 0.92 and a Macro F1 score of 0.76, surpassing both its generic counterpart and a zero-shot LLM baseline. Grounded in a comprehensive ethical analysis, we frame the system as a human-in-the-loop screening aid, not a diagnostic tool. To support this, we introduce a hybrid SHAPLLM explainability framework and present a prototype dashboard ("Social Media Screener") designed to integrate model predictions and their explanations into a practical workflow for moderators. Our work provides a robust baseline, highlighting future needs for multi-label, clinically-validated datasets at the critical intersection of online safety and computational mental health.
Authors Edward Ajayi, Martha Kachweka, Mawuli Deku, Emily Aiken
Comment Accepted for Oral Presentation at the AAAI-26 Bridge Program on AI for Medicine and Healthcare (AIMedHealth). To appear in Proceedings of Machine Learning Research (PMLR)
Categories Computation and Language
0
AppSelectBench: Application-Level Tool Selection Benchmark
Title CN AppSelectBench：应用级工具选择基准
Keywords 计算机使用代理 · 工具选择 · 基准测试 · 跨应用推理 · 大语言模型
Summary 本文提出了AppSelectBench，一个用于评估计算机使用代理在复杂任务中选择合适应用程序能力的基准。该基准包含十万多个真实用户任务，覆盖上百种桌面应用，并设计了多种评估场景。实验揭示了当前大模型在跨应用推理上的局限性。然而，该工作关注的是高层任务调度而非文档内容理解。
Reason 该论文聚焦于计算机使用代理（CUA）中的应用选择问题，即在执行任务时选择使用哪个应用程序（如Word、Excel等），属于典型的‘平行应用’类研究。尽管其基准构建和评估框架具有一定工程价值，但核心目标并非解决文档图像理解中的OCR幻觉、密集文本识别、复杂排版解析或长文档推理等底层技术痛点，也没有提出可迁移至DIU领域的基础方法（如视觉表示、多模态对齐、推理增强机制）。因此属于对现有LLM+Tool框架的下游任务扩展，无方法论创新。
Abstract Computer Using Agents (CUAs) are increasingly equipped with external tools, enabling them to perform complex and realistic tasks. For CUAs to operate effectively, application selection, which refers to deciding which application to use before invoking fine-grained tools such as APIs, is a fundamental capability. It determines whether the agent initializes the correct environment, avoids orchestration confusion, and efficiently focuses on relevant context. However, existing benchmarks primarily assess fine-grained API selection, offering limited insight into whether models can reason across and choose between different applications. To fill this gap, we introduce AppSelectBench, a comprehensive benchmark for evaluating application selection in CUAs. AppSelectBench contains a novel user task generation pipeline that produces realistic, diverse, and semantically grounded user intents at scale, together with unified evaluation protocols covering random, heuristic, zero-shot, few-shot, and retrieval-augmented-settings. AppSelectBench covers one hundred widely used desktop applications and includes more than one hundred thousand realistic, diverse, and semantically grounded user tasks. Extensive experiments across both closed-source and open-source large language models reveal systematic strengths and weaknesses in inter-application reasoning, showing that even the most capable models still struggle to make consistent application choices. Together, these results establish AppSelectBench as a foundation for studying and advancing application level reasoning, an essential yet underexplored capability of intelligent CUAs. The source is available at https://github.com/microsoft/appselectbench.
Authors Tianyi Chen, Michael Solodko, Sen Wang, Jongwoo Ko, Junheng Hao, Colby Banbury, Sara Abdali, Saeed Amizadeh, Qing Xiao, Yinheng Li, Tianyu Ding, Kamran Ghasedi Dizaji, Suzhen Zheng, Hao Fan, Justin Wagle, Pashmina Cameron, Kazuhito Koishida
Categories Computation and Language
0
AI-Mediated Communication Reshapes Social Structure in Opinion-Diverse Groups
Title CN 人工智能中介沟通重塑意见多样化群体中的社会结构
Keywords AI-Mediated Communication · Social Structure · LLM in Social Interaction · Opinion Dynamics · Group Cohesion
Summary 该研究通过在线实验探讨了大语言模型提供的个性化或关系导向型消息建议如何影响用户在争议性政治话题讨论中的群体迁移与互动模式，发现不同类型的AI建议会引发不同的社会聚集效应。但该工作不涉及文档图像或多模态理解任务。
Reason 该论文研究的是AI辅助沟通对社交结构和群体分化的影响，属于社会计算与人机交互领域，核心关注点为政治话题讨论中的群体动态演化，并未涉及文档图像理解、视觉语言模型在文档处理中的应用或相关技术方法。其内容与DIU任务（如DocVQA、布局分析、表格识别等）无直接关联，也不提供可用于解决OCR幻觉、密集文本识别、复杂排版解析或长文档推理的底层技术工具。
Abstract Group segregation or cohesion can emerge from micro-level communication, and AI-assisted messaging may shape this process. Here, we report a preregistered online experiment (N = 557 across 60 sessions) in which participants discussed controversial political topics over multiple rounds and could freely change groups. Some participants received real-time message suggestions from a large language model (LLM), either personalized to their stance (individual assistance) or incorporating their group members' perspectives (relational assistance). We find that small variations in AI-mediated communication cascade into macro-level differences in group composition. Participants with individual assistance send more messages and show greater stance-based clustering, whereas those with relational assistance use more receptive language and form more heterogeneous ties. Hybrid expressive processes-jointly produced by humans and AI-can reshape collective organization. The patterns of structural division and cohesion depend on how AI incorporates users' interaction context.
Authors Faria Huq, Elijah L. Claggett, Hirokazu Shirado
Comment Preprint, Under Review
Categories Computation and Language
0
Scalable Parameter-Light Spectral Method for Clustering Short Text Embeddings with a Cohesion-Based Evaluation Metric
Title CN 一种可扩展的轻量级参数谱方法用于短文本嵌入聚类及基于凝聚度的评估指标
Keywords 短文本聚类 · 谱聚类 · 无监督学习 · 聚类评估 · 嵌入分析
Summary 本文提出了一种基于拉普拉斯特征谱的谱方法来自动估计短文本嵌入的聚类数量，并设计了一个名为Cohesion Ratio的新评估指标，用于无监督地衡量聚类质量。该方法结合余弦相似性与自适应采样策略，在多个短文本数据集上验证了有效性。然而，整个工作局限于纯文本嵌入空间中的聚类问题，未涉及图像、多模态输入或文档结构理解。
Reason 该论文属于纯NLP领域的短文本聚类方法研究，未涉及任何视觉或文档图像内容。任务与文档图像理解（DIU）完全无关，既不处理OCR、布局分析、表格识别等核心问题，也未提出可用于多模态文档理解的通用技术。尽管其提出的Cohesion Ratio评估指标有一定方法论价值，但迁移至DIU场景（如解决幻觉、细粒度定位、高分辨率处理）极为困难，不具备直接应用或启发意义。
Abstract Clustering short text embeddings is a foundational task in natural language processing, yet remains challenging due to the need to specify the number of clusters in advance. We introduce a scalable spectral method that estimates the number of clusters directly from the structure of the Laplacian eigenspectrum, constructed using cosine similarities and guided by an adaptive sampling strategy. This sampling approach enables our estimator to efficiently scale to large datasets without sacrificing reliability. To support intrinsic evaluation of cluster quality without ground-truth labels, we propose the Cohesion Ratio, a simple and interpretable evaluation metric that quantifies how much intra-cluster similarity exceeds the global similarity background. It has an information-theoretic motivation inspired by mutual information, and in our experiments it correlates closely with extrinsic measures such as normalized mutual information and homogeneity. Extensive experiments on six short-text datasets and four modern embedding models show that standard algorithms like K-Means and HAC, when guided by our estimator, significantly outperform popular parameter-light methods such as HDBSCAN, OPTICS, and Leiden. These results demonstrate the practical value of our spectral estimator and Cohesion Ratio for unsupervised organization and evaluation of short text data. Implementation of our estimator of k and Cohesion Ratio, along with code for reproducing the experiments, is available at https://anonymous.4open.science/r/towards_clustering-0C2E.
Authors Nikita Neveditsin, Pawan Lingras, Vijay Mago
Categories Computation and Language
0
AraFinNews: Arabic Financial Summarisation with Domain-Adapted LLMs
Title CN AraFinNews：基于领域适配大语言模型的阿拉伯语金融文本摘要
Keywords 阿拉伯语 · 金融文本摘要 · 领域适配 · LLM · 数据集
Summary 本文提出了一个大规模阿拉伯语金融新闻摘要数据集AraFinNews，包含21.25万条文章-标题对，并基于mT5、AraT5及领域适配的FinAraT5模型进行实验，验证了领域预训练对生成质量的提升效果。工作聚焦于纯文本生成与领域适应，无视觉或多模态内容。
Reason 该论文属于典型的‘平行下游应用’，其核心是将现有LLM技术应用于阿拉伯语金融新闻摘要任务，并发布了特定领域的数据集。研究未提出任何文档图像理解（DIU）或视觉-语言模型（VLM）层面的方法论创新，且聚焦于纯文本NLP任务，不涉及OCR、布局分析、表格识别或视觉文档结构理解。此外，其语言为阿拉伯语，属于非中英小语种范畴，进一步落入负面清单。完全不符合本方向对底层技术创新的要求。
Abstract We introduce AraFinNews, the largest publicly available Arabic financial news dataset to date, comprising 212,500 article-headline pairs spanning a decade of reporting from 2015 to 2025. Designed as an Arabic counterpart to major English summarisation corpora such as CNN/DailyMail, AraFinNews provides a realistic benchmark for evaluating domain-specific language understanding and generation in financial contexts. Using this resource, we investigate the impact of domain specificity on abstractive summarisation of Arabic financial texts with large language models (LLMs). In particular, we evaluate transformer-based models: mT5, AraT5, and the domain-adapted FinAraT5 to examine how financial-domain pretraining influences accuracy, numerical reliability, and stylistic alignment with professional reporting. Experimental results show that domain-adapted models generate more coherent summaries, especially in their handling of quantitative and entity-centric information. These findings highlight the importance of domain-specific adaptation for improving narrative fluency in Arabic financial summarisation. The dataset is freely available for non-commercial research at https://github.com/ArabicNLP-uk/AraFinNews.
Authors Mo El-Haj, Paul Rayson
Comment 9 pages
Categories Computation and Language
0
Profile-LLM: Dynamic Profile Optimization for Realistic Personality Expression in LLMs
Title CN Profile-LLM：用于真实个性表达的动态配置文件优化
Keywords 个性化LLM · 提示优化 · 性格表达 · 角色扮演 · 自然语言生成
Summary 本文提出PersonaPulse框架，通过利用LLM对人格特质的内在知识，迭代优化角色扮演提示，并结合情境响应基准进行评分，以提升个性化LLM的性格表达真实性。实验表明该方法优于基于心理学描述的手工提示设计。
Reason 该论文研究的是个性化大语言模型中的性格表达与提示优化，属于纯文本生成与人机交互领域，不涉及任何视觉或文档模态。其核心问题为角色扮演提示的动态优化，与文档图像理解（DIU）中的OCR幻觉、复杂排版、细粒度定位等关键挑战完全无关。无法迁移至DIU领域解决实际问题。
Abstract Personalized Large Language Models (LLMs) have been shown to be an effective way to create more engaging and enjoyable user-AI interactions. While previous studies have explored using prompts to elicit specific personality traits in LLMs, they have not optimized these prompts to maximize personality expression. To address this limitation, we propose PersonaPulse: Dynamic Profile Optimization for Realistic Personality Expression in LLMs, a framework that leverages LLMs' inherent knowledge of personality traits to iteratively enhance role-play prompts while integrating a situational response benchmark as a scoring tool, ensuring a more realistic and contextually grounded evaluation to guide the optimization process. Quantitative evaluations demonstrate that the prompts generated by PersonaPulse outperform those of prior work, which were designed based on personality descriptions from psychological studies. Additionally, we explore the relationship between model size and personality modeling through extensive experiments. Finally, we find that, for certain personality traits, the extent of personality evocation can be partially controlled by pausing the optimization process. These findings underscore the importance of prompt optimization in shaping personality expression within LLMs, offering valuable insights for future research on adaptive AI interactions.
Authors Shi-Wei Dai, Yan-Wei Shie, Tsung-Huan Yang, Lun-Wei Ku, Yung-Hui Li
Categories Computation and Language
0
ShortageSim: Simulating Drug Shortages under Information Asymmetry AAAI 2026
Title CN ShortageSim：在信息不对称下模拟药物短缺
Keywords 药物短缺 · LLM代理 · 信息不对称 · 政策模拟 · 供应链
Summary 本文提出了ShortageSim，一个基于大语言模型代理的药物短缺模拟框架，用于评估监管干预在信息不对称条件下的影响。通过建模药厂和采购机构的战略行为，该框架能够更真实地反映现实中对短缺警报的响应差异，并在历史FDA事件数据上验证了其有效性，显著缩短了生产中断的解决延迟。
Reason 该论文属于具身智能与LLM代理在供应链政策模拟中的垂直领域应用，尽管使用了LLM作为建模工具，但其核心是经济决策仿真与监管政策评估，并未提出任何可用于文档图像理解（DIU）的底层方法。任务目标、数据形式和技术创新均与文档视觉问答、OCR、版面分析等DIU核心问题无关，属于典型的‘平行应用’，不符合本研究方向的技术源头筛选标准。
Abstract Drug shortages pose critical risks to patient care and healthcare systems worldwide, yet the effectiveness of regulatory interventions remains poorly understood due to information asymmetries in pharmaceutical supply chains. We propose \textbf{ShortageSim}, addresses this challenge by providing the first simulation framework that evaluates the impact of regulatory interventions on competition dynamics under information asymmetry. Using Large Language Model (LLM)-based agents, the framework models the strategic decisions of drug manufacturers and institutional buyers, in response to shortage alerts given by the regulatory agency. Unlike traditional game theory models that assume perfect rationality and complete information, ShortageSim simulates heterogeneous interpretations on regulatory announcements and the resulting decisions. Experiments on self-processed dataset of historical shortage events show that ShortageSim reduces the resolution lag for production disruption cases by up to 84\%, achieving closer alignment to real-world trajectories than the zero-shot baseline. Our framework confirms the effect of regulatory alert in addressing shortages and introduces a new method for understanding competition in multi-stage environments under uncertainty. We open-source ShortageSim and a dataset of 2,925 FDA shortage events, providing a novel framework for future research on policy design and testing in supply chains under information asymmetry.
Authors Mingxuan Cui, Yilan Jiang, Duo Zhou, Cheng Qian, Yuji Zhang, Qiong Wang
Comment Accepted by AAAI 2026. Oral presentation. 25 pages
Categories Computation and Language
0
Breaking Bad: Norms for Valence, Arousal, and Dominance for over 10k English Multiword Expressions
Title CN 打破常规：超过1万个英语多词表达的效价、唤醒度和支配度规范
Keywords 情感词典 · 多词表达式 · 效价-唤醒-支配模型 · 心理语言学 · NRC VAD
Summary 本文构建了一个包含超过1万个英语多词表达式（MWEs）及其组成词的效价、唤醒度和支配度的人类评分词典（NRC VAD Lexicon v2），扩展了原有词典的词汇覆盖范围，并验证了评分的可靠性。研究主要关注多词表达的情感特性及其情感组合性，适用于心理学、社会科学研究及部分情感分析NLP任务。
Reason 该论文属于情感词典构建领域，聚焦于心理语言学中的情感维度（Valence, Arousal, Dominance），尽管对NLP有一定辅助价值，但其研究内容与文档图像理解（DIU）、视觉语言模型（VLM）、OCR、布局分析、表格识别等核心任务完全无关。未涉及任何视觉模态、文档结构或跨模态推理问题，无法迁移解决DIU中的OCR幻觉、复杂排版、高分辨率处理等关键挑战。
Abstract Factor analysis studies have shown that the primary dimensions of word meaning are Valence (V), Arousal (A), and Dominance (D). Existing lexicons such as the NRC VAD Lexicon, published in 2018, include VAD association ratings for words. Here, we present a complement to it, which has human ratings of valence, arousal, and dominance for 10k English Multiword Expressions (MWEs) and their constituent words. We also increase the coverage of unigrams, especially words that have become more common since 2018. In all, the new NRC VAD Lexicon v2 now has entries for 10k MWEs and 25k words, in addition to the entries in v1. We show that the associations are highly reliable. We use the lexicon to examine emotional characteristics of MWEs, including: 1. The degree to which MWEs (idioms, noun compounds, and verb particle constructions) exhibit strong emotionality; 2. The degree of emotional compositionality in MWEs. The lexicon enables a wide variety of research in NLP, Psychology, Public Health, Digital Humanities, and Social Sciences. The NRC VAD Lexicon v2 is freely available through the project webpage: http://saifmohammad.com/WebPages/nrc-vad.html
Authors Saif M. Mohammad
Categories Computation and Language
0
Emergence and Localisation of Semantic Role Circuits in LLMs
Title CN 大语言模型中语义角色电路的涌现与定位
Keywords 语义角色 · 电路分析 · 大语言模型 · 内部机制 · 可解释性
Summary 本文提出一种结合角色交叉最小对、时间演化分析和跨模型比较的方法，用于研究大语言模型如何实现语义角色。发现LLMs会形成高度集中的电路来处理抽象语义结构，并表现出跨尺度的部分保守性和高谱相似性。这些机制是紧凑且因果隔离的。
Reason 该论文属于纯NLP领域，研究大语言模型内部语义角色机制的表示与电路定位，未涉及任何视觉或文档模态。其方法论聚焦于文本模态下的语义结构分析，无法迁移解决文档图像理解中的核心痛点（如OCR幻觉、细粒度视觉定位、复杂排版建模等）。属于无关领域（Computation and Language）的基础机理研究，对DIU无直接或间接方法论贡献。
Abstract Despite displaying semantic competence, large language models' internal mechanisms that ground abstract semantic structure remain insufficiently characterised. We propose a method integrating role-cross minimal pairs, temporal emergence analysis, and cross-model comparison to study how LLMs implement semantic roles. Our analysis uncovers: (i) highly concentrated circuits (89-94% attribution within 28 nodes); (ii) gradual structural refinement rather than phase transitions, with larger models sometimes bypassing localised circuits; and (iii) moderate cross-scale conservation (24-59% component overlap) alongside high spectral similarity. These findings suggest that LLMs form compact, causally isolated mechanisms for abstract semantic structure, and these mechanisms exhibit partial transfer across scales and architectures.
Authors Nura Aljaafari, Danilo S. Carvalho, André Freitas
Categories Computation and Language
0
Winning with Less for Low Resource Languages: Advantage of Cross-Lingual English_Persian Argument Mining Model over LLM Augmentation
Title CN 低资源语言下的以少胜多：跨语言英波论辩挖掘模型优于大语言模型增强
Keywords 论辩挖掘 · 低资源语言 · 跨语言模型 · 数据增强 · 波斯语
Summary 本文研究在低资源语言（如波斯语）中进行论辩挖掘的任务，比较了三种训练方式：零样本迁移、基于大语言模型生成的合成数据增强、以及结合英波双语数据的跨语言训练。结果显示，使用人工翻译的跨语言模型在波斯语测试集上表现最佳（F1达74.8%），优于LLM增强方法。表明轻量级跨语言方法可有效应对低资源场景。
Reason 该论文属于纯NLP任务（论辩挖掘），且聚焦于低资源语言（波斯语），属于小语种研究。虽然涉及跨语言迁移，但其方法和数据均不涉及文档图像、视觉信息或任何多模态内容，无法迁移到文档图像理解（DIU）领域解决OCR幻觉、复杂排版或视觉-文本对齐等问题。
Abstract Argument mining is a subfield of natural language processing to identify and extract the argument components, like premises and conclusions, within a text and to recognize the relations between them. It reveals the logical structure of texts to be used in tasks like knowledge extraction. This paper aims at utilizing a cross-lingual approach to argument mining for low-resource languages, by constructing three training scenarios. We examine the models on English, as a high-resource language, and Persian, as a low-resource language. To this end, we evaluate the models based on the English Microtext corpus \citep{PeldszusStede2015}, and its parallel Persian translation. The learning scenarios are as follow: (i) zero-shot transfer, where the model is trained solely with the English data, (ii) English-only training enhanced by synthetic examples generated by Large Language Models (LLMs), and (iii) a cross-lingual model that combines the original English data with manually translated Persian sentences. The zero-shot transfer model attains F1 scores of 50.2\% on the English test set and 50.7\% on the Persian test set. LLM-based augmentation model improves the performance up to 59.2\% on English and 69.3\% on Persian. The cross-lingual model, trained on both languages but evaluated solely on the Persian test set, surpasses the LLM-based variant, by achieving a F1 of 74.8\%. Results indicate that a lightweight cross-lingual blend can outperform considerably the more resource-intensive augmentation pipelines, and it offers a practical pathway for the argument mining task to overcome data resource shortage on low-resource languages.
Authors Ali Jahan, Masood Ghayoomi, Annette Hautli-Janisz
Comment Preprint. Under review
Categories Computation and Language
0
Beyond Protein Language Models: An Agentic LLM Framework for Mechanistic Enzyme Design
Title CN 超越蛋白质语言模型：一种用于酶机制设计的代理式LLM框架
Keywords LLM Agent · 科学假设生成 · 蛋白质设计 · 工具增强 · RAG
Summary 本文提出了Genie-CAT，一个结合检索增强生成、结构解析、静电势计算和机器学习预测的代理式LLM系统，用于加速金属蛋白设计中的假设生成。系统能自主识别影响[Fe-S]簇氧化还原特性的残基级修饰，复现专家级推理。然而，该工作为特定科学领域的应用框架，不具备向文档理解迁移的方法论价值。
Reason 该论文属于典型的平行下游应用（Wrapper/Application），将LLM与工具结合应用于酶设计领域，尽管采用了Agent架构，但其方法论创新集中在生物科学场景，并未提出可迁移至文档图像理解领域的底层技术。任务聚焦于蛋白质设计中的假设生成，与DocVQA、OCR、布局分析等DIU核心任务无直接关联，且未涉及视觉输入处理、多模态对齐或视觉幻觉缓解等关键问题。
Abstract We present Genie-CAT, a tool-augmented large-language-model (LLM) system designed to accelerate scientific hypothesis generation in protein design. Using metalloproteins (e.g., ferredoxins) as a case study, Genie-CAT integrates four capabilities -- literature-grounded reasoning through retrieval-augmented generation (RAG), structural parsing of Protein Data Bank files, electrostatic potential calculations, and machine-learning prediction of redox properties -- into a unified agentic workflow. By coupling natural-language reasoning with data-driven and physics-based computation, the system generates mechanistically interpretable, testable hypotheses linking sequence, structure, and function. In proof-of-concept demonstrations, Genie-CAT autonomously identifies residue-level modifications near [Fe--S] clusters that affect redox tuning, reproducing expert-derived hypotheses in a fraction of the time. The framework highlights how AI agents combining language models with domain-specific tools can bridge symbolic reasoning and numerical simulation, transforming LLMs from conversational assistants into partners for computational discovery.
Authors Bruno Jacob, Khushbu Agarwal, Marcel Baer, Peter Rice, Simone Raugei
Comment 10 pages, 4 figures
Categories Artificial Intelligence
0
The Loss of Control Playbook: Degrees, Dynamics, and Preparedness
Title CN 失控手册：程度、动态与应对准备
Keywords AI安全 · 失控风险 · 治理框架 · 风险防范 · 政策研究
Summary 本文提出了一种关于AI系统失控风险的分级分类体系，并构建了一个基于外部因素（部署情境、功能支持和权限控制）的风险防范框架（DAP），旨在通过治理措施和技术管控提升社会对高级AI失控风险的应对准备能力。研究属于AI安全与政策范畴，未涉及具体技术实现或视觉-语言模态交互。
Reason 该论文聚焦于AI系统中的‘失控’（Loss of Control）风险，属于AI安全与治理的宏观政策研究，不涉及任何文档图像理解（DIU）或视觉语言模型（VLM）的技术方法。其提出的DAP框架（部署环境、可用性、权限）为治理层面的建议，无法迁移用于解决OCR幻觉、复杂排版理解、细粒度定位等DIU核心问题。
Abstract This research report addresses the absence of an actionable definition for Loss of Control (LoC) in AI systems by developing a novel taxonomy and preparedness framework. Despite increasing policy and research attention, existing LoC definitions vary significantly in scope and timeline, hindering effective LoC assessment and mitigation. To address this issue, we draw from an extensive literature review and propose a graded LoC taxonomy, based on the metrics of severity and persistence, that distinguishes between Deviation, Bounded LoC, and Strict LoC. We model pathways toward a societal state of vulnerability in which sufficiently advanced AI systems have acquired or could acquire the means to cause Bounded or Strict LoC once a catalyst, either misalignment or pure malfunction, materializes. We argue that this state becomes increasingly likely over time, absent strategic intervention, and propose a strategy to avoid reaching a state of vulnerability. Rather than focusing solely on intervening on AI capabilities and propensities potentially relevant for LoC or on preventing potential catalysts, we introduce a complementary framework that emphasizes three extrinsic factors: Deployment context, Affordances, and Permissions (the DAP framework). Compared to work on intrinsic factors and catalysts, this framework has the unfair advantage of being actionable today. Finally, we put forward a plan to maintain preparedness and prevent the occurrence of LoC outcomes should a state of societal vulnerability be reached, focusing on governance measures (threat modeling, deployment policies, emergency response) and technical controls (pre-deployment testing, control measures, monitoring) that could maintain a condition of perennial suspension.
Authors Charlotte Stix, Annika Hallensleben, Alejandro Ortega, Matteo Pistillo
Categories Artificial Intelligence
0
In-Video Instructions: Visual Signals as Generative Control
Title CN 视频中的指令：视觉信号作为生成控制
Keywords 视频生成 · 可控生成 · 视觉指令 · 多模态控制 · 计算机视觉
Summary 本文提出了一种名为“In-Video Instruction”的新范式，通过在视频帧中嵌入视觉元素（如文本、箭头、轨迹）作为生成指令，实现对图像到视频生成过程的细粒度空间控制。相比基于文本提示的全局粗略控制，该方法能在多对象复杂场景中建立明确的视觉主体与动作之间的对应关系。实验验证了该方法在多个先进视频生成模型上的有效性。
Reason 该论文研究的是视频生成模型中通过嵌入视觉信号（如箭头、文字）来控制生成过程，属于视频生成与可控合成领域。其核心任务和方法均与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、密集文本识别、复杂版式分析或长文档推理等DIU核心问题。
Abstract Large-scale video generative models have recently demonstrated strong visual capabilities, enabling the prediction of future frames that adhere to the logical and physical cues in the current observation. In this work, we investigate whether such capabilities can be harnessed for controllable image-to-video generation by interpreting visual signals embedded within the frames as instructions, a paradigm we term In-Video Instruction. In contrast to prompt-based control, which provides textual descriptions that are inherently global and coarse, In-Video Instruction encodes user guidance directly into the visual domain through elements such as overlaid text, arrows, or trajectories. This enables explicit, spatial-aware, and unambiguous correspondences between visual subjects and their intended actions by assigning distinct instructions to different objects. Extensive experiments on three state-of-the-art generators, including Veo 3.1, Kling 2.5, and Wan 2.2, show that video models can reliably interpret and execute such visually embedded instructions, particularly in complex multi-object scenarios.
Authors Gongfan Fang, Xinyin Ma, Xinchao Wang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Real-Time Object Tracking with On-Device Deep Learning for Adaptive Beamforming in Dynamic Acoustic Environments
Title CN 基于设备端深度学习的实时目标跟踪在动态声学环境中用于自适应波束成形
Keywords 目标跟踪 · 波束成形 · 麦克风阵列 · 深度估计 · 嵌入式系统
Summary 本文提出一种集成深度学习目标跟踪与麦克风阵列波束成形的嵌入式系统，利用单目深度估计与双目视觉实现移动目标的3D定位，并通过环形MEMS麦克风阵列进行实时声学聚焦，以在多声源动态环境中提升信干比。适用于远程会议与智能家居等场景。
Reason 该论文研究的是结合视觉与麦克风阵列的声源定位和波束成形技术，属于具身智能与机器人感知系统范畴，核心任务为音频-视觉协同定位与实时嵌入式推理，并非文档图像理解（DIU）或相关多模态语言模型（VLM）在文本密集场景下的应用。其方法无法迁移解决OCR幻觉、复杂排版解析、长文档推理等DIU核心痛点。
Abstract Advances in object tracking and acoustic beamforming are driving new capabilities in surveillance, human-computer interaction, and robotics. This work presents an embedded system that integrates deep learning-based tracking with beamforming to achieve precise sound source localization and directional audio capture in dynamic environments. The approach combines single-camera depth estimation and stereo vision to enable accurate 3D localization of moving objects. A planar concentric circular microphone array constructed with MEMS microphones provides a compact, energy-efficient platform supporting 2D beam steering across azimuth and elevation. Real-time tracking outputs continuously adapt the array's focus, synchronizing the acoustic response with the target's position. By uniting learned spatial awareness with dynamic steering, the system maintains robust performance in the presence of multiple or moving sources. Experimental evaluation demonstrates significant gains in signal-to-interference ratio, making the design well-suited for teleconferencing, smart home devices, and assistive technologies.
Authors Jorge Ortigoso-Narro, Jose A. Belloch, Adrian Amor-Martin, Sandra Roger, Maximo Cobos
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics
Title CN 机器人世界模型：用于机器人鲁棒策略优化的神经网络模拟器
Keywords 世界模型 · 强化学习 · 机器人控制 · 自监督学习 · 长视野预测
Summary 本文提出一种用于机器人控制的神经网络世界模型，通过双自回归机制和自监督训练实现长视野动态预测，并支持在想象环境中进行策略优化，提升现实部署的鲁棒性。重点解决模型误差累积和仿真到现实的迁移问题。
Reason 该论文属于具身智能与机器人领域，研究的是机器人控制中的世界模型构建与策略优化，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管其方法涉及序列建模和预测机制，但核心问题设定、输入模态（状态-动作序列）和应用场景均远离文档理解任务，无法迁移解决OCR幻觉、复杂排版或细粒度文本定位等DIU痛点。
Abstract Learning robust and generalizable world models is crucial for enabling efficient and scalable robotic control in real-world environments. In this work, we introduce a novel framework for learning world models that accurately capture complex, partially observable, and stochastic dynamics. The proposed method employs a dual-autoregressive mechanism and self-supervised training to achieve reliable long-horizon predictions without relying on domain-specific inductive biases, ensuring adaptability across diverse robotic tasks. We further propose a policy optimization framework that leverages world models for efficient training in imagined environments and seamless deployment in real-world systems. This work advances model-based reinforcement learning by addressing the challenges of long-horizon prediction, error accumulation, and sim-to-real transfer. By providing a scalable and robust framework, the introduced methods pave the way for adaptive and efficient robotic systems in real-world applications.
Authors Chenhao Li, Andreas Krause, Marco Hutter
Categories Artificial Intelligence
0
An Anatomy Aware Hybrid Deep Learning Framework for Lung Cancer Tumor Stage Classification
Title CN 一种基于解剖感知的混合深度学习框架用于肺癌肿瘤分期分类
Keywords 医学图像分析 · 肿瘤分期 · 图像分割 · 混合模型 · 可解释性
Summary 本文提出了一种结合解剖结构感知的混合深度学习框架，用于肺癌肿瘤的TNM分期。通过专门的编码器-解码器网络对肺部及周围解剖结构（如叶、纵隔、膈肌等）进行精确分割，并从分割结果中提取肿瘤大小及其与邻近结构的距离等定量特征，最后依据临床指南进行规则驱动的分期决策。该方法在Lung-PET-CT-Dx数据集上达到91.36%的准确率，并提供可解释的决策过程。
Reason 该论文属于医学图像分析领域，研究任务为基于CT/PET图像的肺癌分期，核心方法是结合解剖结构分割与规则推理进行肿瘤分期判断。尽管使用了视觉-规则混合流程，但其目标与文档图像理解（DIU）无关，不涉及文本识别、布局分析、表格理解或VLM在文档上的应用。属于典型的垂直领域医疗影像应用，无迁移到文档理解场景的方法论价值。
Abstract Accurate lung cancer tumor staging is crucial for prognosis and treatment planning. However, it remains challenging for end-to-end deep learning approaches, as such approaches often overlook spatial and anatomical information that are central to the tumor-node-metastasis system. The tumor stage depends on multiple quantitative criteria, including the tumor size and its proximity to the nearest anatomical structures, and small variations can alter the staging outcome. We propose a medically grounded hybrid pipeline that performs staging by explicitly measuring the tumor's size and distance properties rather than treating it as a pure image classification task. Our method employs specialized encoder-decoder networks to precisely segment the lung and adjacent anatomy, including the lobes, tumor, mediastinum, and diaphragm. Subsequently, we extract the necessary tumor properties, i.e. measure the largest tumor dimension and calculate the distance between the tumor and neighboring anatomical structures by a quantitative analysis of the segmentation masks. Finally, we apply rule-based tumor staging aligned with the medical guidelines. This novel framework has been evaluated on the Lung-PET-CT-Dx dataset, demonstrating superior performance compared to traditional deep learning models, achieving an overall classification accuracy of 91.36%. We report the per-stage F1-scores of 0.93 (T1), 0.89 (T2), 0.96 (T3), and 0.90 (T4), a critical evaluation aspect often omitted in prior literature. To our knowledge, this is the first study that embeds explicit clinical context into tumor stage classification. Unlike standard convolutional neural networks that operate in an uninterpretable "black box" manner, our method offers both state-of-the-art performance and transparent decision support.
Authors Saniah Kayenat Chowdhury, Rusab Sarmun, Muhammad E. H. Chowdhury, Sohaib Bassam Zoghoul, Israa Al-Hashimi, Adam Mushtak, Amith Khandakar
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generation
Title CN DeCo：用于端到端图像生成的频率解耦像素扩散
Keywords 像素扩散 · 图像生成 · 频率解耦 · DiT · 流匹配
Summary 本文提出DeCo，一种频率解耦的像素级扩散模型，通过分离高频细节与低频语义的生成过程来提升端到端图像生成的效率和质量。使用轻量化解码器生成高频信息，主干DiT专注于低频语义，并引入频率感知的流匹配损失优化训练。在ImageNet和文本到图像生成任务上取得优异表现。
Reason 该论文聚焦于纯图像生成任务，属于‘纯图像生成/修复’范畴，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出频率解耦和高效扩散机制，但目标是提升通用图像生成质量与效率，并未涉及OCR、视觉文本理解或多模态文档推理等关键问题。方法难以迁移到解决文档理解中的OCR幻觉、细粒度定位或复杂排版解析等痛点。
Abstract Pixel diffusion aims to generate images directly in pixel space in an end-to-end fashion. This approach avoids the limitations of VAE in the two-stage latent diffusion, offering higher model capacity. Existing pixel diffusion models suffer from slow training and inference, as they usually model both high-frequency signals and low-frequency semantics within a single diffusion transformer (DiT). To pursue a more efficient pixel diffusion paradigm, we propose the frequency-DeCoupled pixel diffusion framework. With the intuition to decouple the generation of high and low frequency components, we leverage a lightweight pixel decoder to generate high-frequency details conditioned on semantic guidance from the DiT. This thus frees the DiT to specialize in modeling low-frequency semantics. In addition, we introduce a frequency-aware flow-matching loss that emphasizes visually salient frequencies while suppressing insignificant ones. Extensive experiments show that DeCo achieves superior performance among pixel diffusion models, attaining FID of 1.62 (256x256) and 2.22 (512x512) on ImageNet, closing the gap with latent diffusion methods. Furthermore, our pretrained text-to-image model achieves a leading overall score of 0.86 on GenEval in system-level comparison. Codes are publicly available at https://github.com/Zehong-Ma/DeCo.
Authors Zehong Ma, Longhui Wei, Shuai Wang, Shiliang Zhang, Qi Tian
Comment Project Page: https://zehong-ma.github.io/DeCo. Code Repository: https://github.com/Zehong-Ma/DeCo
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Leveraging LLMs for reward function design in reinforcement learning control tasks
Title CN 利用大语言模型进行强化学习控制任务中的奖励函数设计
Keywords 强化学习 · 奖励函数设计 · 大语言模型 · 自动化 · LEARN-Opt
Summary 本文提出LEARN-Opt，一个基于大语言模型的自主框架，用于从系统描述和任务目标中自动生成、执行和评估强化学习中的奖励函数。该方法无需预先定义评价指标或访问环境源码，能自主推导性能指标并选择最优奖励函数。实验表明其性能优于或相当于现有方法（如EUREKA），且可利用低成本LLM实现高质量结果。
Reason 该论文属于强化学习中的奖励函数自动化设计，研究如何用LLM生成和优化RL任务的奖励函数。尽管使用了LLM，但其核心问题与文档图像理解（DIU）无直接关联，既不涉及多模态视觉-语言建模、OCR、布局分析、表格识别等DIU任务，也未提出可用于解决文档理解中幻觉、细粒度定位或复杂排版等痛点的通用方法。属于典型的‘平行应用’类工作，且应用场景为控制任务，完全偏离DIU领域。
Abstract The challenge of designing effective reward functions in reinforcement learning (RL) represents a significant bottleneck, often requiring extensive human expertise and being time-consuming. Previous work and recent advancements in large language models (LLMs) have demonstrated their potential for automating the generation of reward functions. However, existing methodologies often require preliminary evaluation metrics, human-engineered feedback for the refinement process, or the use of environmental source code as context. To address these limitations, this paper introduces LEARN-Opt (LLM-based Evaluator and Analyzer for Reward functioN Optimization). This LLM-based, fully autonomous, and model-agnostic framework eliminates the need for preliminary metrics and environmental source code as context to generate, execute, and evaluate reward function candidates from textual descriptions of systems and task objectives. LEARN-Opt's main contribution lies in its ability to autonomously derive performance metrics directly from the system description and the task objective, enabling unsupervised evaluation and selection of reward functions. Our experiments indicate that LEARN-Opt achieves performance comparable to or better to that of state-of-the-art methods, such as EUREKA, while requiring less prior knowledge. We find that automated reward design is a high-variance problem, where the average-case candidate fails, requiring a multi-run approach to find the best candidates. Finally, we show that LEARN-Opt can unlock the potential of low-cost LLMs to find high-performing candidates that are comparable to, or even better than, those of larger models. This demonstrated performance affirms its potential to generate high-quality reward functions without requiring any preliminary human-defined metrics, thereby reducing engineering overhead and enhancing generalizability.
Authors Franklin Cardenoso, Wouter Caarls
Categories Artificial Intelligence
0
Node Preservation and its Effect on Crossover in Cartesian Genetic Programming evo*2026
Title CN 节点保持及其在笛卡尔遗传编程中对交叉的影响
Keywords 遗传编程 · 笛卡尔遗传编程 · 交叉算子 · 节点保持 · 进化算法
Summary 本文探讨了在笛卡尔遗传编程（CGP）中引入节点保持机制对交叉操作的影响，比较了一点交叉、均匀交叉及其保留功能模块的变体，并提出一种节点级突变算子。实验表明，在符号回归任务上，节点保持能提升搜索性能。但该工作属于遗传编程的算法优化，不涉及视觉或多模态内容。
Reason 该论文研究的是笛卡尔遗传编程（CGP）中的交叉算子与节点保持机制，属于进化计算领域的理论探讨，与文档图像理解（DIU）、多模态大模型（VLM）、OCR、DocVQA等任务完全无关。未涉及视觉输入处理、文本-图像对齐、高分辨率建模或任何可用于解决文档理解核心痛点（如OCR幻觉、复杂排版、长文档推理）的方法论创新。
Abstract While crossover is a critical and often indispensable component in other forms of Genetic Programming, such as Linear- and Tree-based, it has consistently been claimed that it deteriorates search performance in CGP. As a result, a mutation-alone $(1+λ)$ evolutionary strategy has become the canonical approach for CGP. Although several operators have been developed that demonstrate an increased performance over the canonical method, a general solution to the problem is still lacking. In this paper, we compare basic crossover methods, namely one-point and uniform, to variants in which nodes are ``preserved,'' including the subgraph crossover developed by Roman Kalkreuth, the difference being that when ``node preservation'' is active, crossover is not allowed to break apart instructions. We also compare a node mutation operator to the traditional point mutation; the former simply replaces an entire node with a new one. We find that node preservation in both mutation and crossover improves search using symbolic regression benchmark problems, moving the field towards a general solution to CGP crossover.
Authors Mark Kocherovsky, Illya Bakurov, Wolfgang Banzhaf
Comment Draft to cite in another paper before both papers are peer-reviewed for the evo*2026 conference, 21 pages, 5 figures
Categories Artificial Intelligence
0
Explicit Tonal Tension Conditioning via Dual-Level Beam Search for Symbolic Music Generation 17th International Symposium on Computer Music Multidisciplinary Research (CMMR) 2025
Title CN 通过双层级束搜索实现符号音乐生成中的显式音调张力控制
Keywords 音乐生成 · Transformer · 束搜索 · 音调张力 · 符号音乐
Summary 本文提出了一种将计算性音调张力模型集成到Transformer框架中的方法，用于在符号音乐生成中显式控制音调张力。该方法采用两层级束搜索策略：在token层级进行候选重排序以保持多样性和质量，在小节（bar）层级根据目标张力曲线进行张力驱动的重排序。客观和主观评估表明，该方法能有效调节音乐的张力变化，并生成符合预期张力轮廓的多样化音乐作品。
Reason 该论文属于音乐生成领域，研究的是如何通过双层级束搜索控制生成音乐的音调张力，尽管使用了AI和Transformer架构，但其任务与文档图像理解（DIU）或视觉语言模型在文档上的应用完全无关。未涉及OCR、布局分析、表格识别、视觉文本理解或多模态文档建模等核心问题，也无法迁移其方法解决DIU中的幻觉、密集文本、复杂排版或长文档推理等痛点。
Abstract State-of-the-art symbolic music generation models have recently achieved remarkable output quality, yet explicit control over compositional features, such as tonal tension, remains challenging. We propose a novel approach that integrates a computational tonal tension model, based on tonal interval vector analysis, into a Transformer framework. Our method employs a two-level beam search strategy during inference. At the token level, generated candidates are re-ranked using model probability and diversity metrics to maintain overall quality. At the bar level, a tension-based re-ranking is applied to ensure that the generated music aligns with a desired tension curve. Objective evaluations indicate that our approach effectively modulates tonal tension, and subjective listening tests confirm that the system produces outputs that align with the target tension. These results demonstrate that explicit tension conditioning through a dual-level beam search provides a powerful and intuitive tool to guide AI-generated music. Furthermore, our experiments demonstrate that our method can generate multiple distinct musical interpretations under the same tension condition.
Authors Maral Ebrahimzadeh, Gilberto Bernardes, Sebastian Stober
Comment 12 pages, 2 Figures, Accepted at the 17th International Symposium on Computer Music Multidisciplinary Research (CMMR) 2025
Categories Artificial Intelligence
0
What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Models
Title CN 什么驱动跨语言排序？基于多语言语言模型的检索方法
Keywords Cross-lingual Information Retrieval · Multilingual Language Models · Dense Retrieval · Contrastive Learning · Re-ranking
Summary 本文系统评估了四种用于跨语言信息检索（CLIR）的干预方法，包括文档翻译、多语言密集检索、不同粒度的对比学习以及交叉编码器重排序。实验表明，专为CLIR训练的密集检索模型优于基于词汇匹配和翻译的方法，尤其在低资源和跨脚本语言对上表现更优。研究建议应优先采用语义多语言嵌入和针对性的对齐学习，而非依赖翻译的流水线。
Reason 该论文研究的是跨语言信息检索（CLIR），聚焦于多语言文本检索中的语义对齐与翻译策略问题，属于纯NLP任务，且未涉及任何视觉模态或文档图像内容。其核心方法（如多语言密集检索、对比学习、重排序）虽然在NLP领域有一定价值，但无法直接迁移解决文档图像理解中的OCR幻觉、复杂排版、细粒度定位等关键痛点。不属于DIU本题，也不提供可用于文档理解的新颖视觉或多模态技术。
Abstract Cross-lingual information retrieval (CLIR) enables access to multilingual knowledge but remains challenging due to disparities in resources, scripts, and weak cross-lingual semantic alignment in embedding models. Existing pipelines often rely on translation and monolingual retrieval heuristics, which add computational overhead and noise, degrading performance. This work systematically evaluates four intervention types, namely document translation, multilingual dense retrieval with pretrained encoders, contrastive learning at word, phrase, and query-document levels, and cross-encoder re-ranking, across three benchmark datasets. We find that dense retrieval models trained specifically for CLIR consistently outperform lexical matching methods and derive little benefit from document translation. Contrastive learning mitigates language biases and yields substantial improvements for encoders with weak initial alignment, and re-ranking can be effective, but depends on the quality of the cross-encoder training data. Although high-resource languages still dominate overall performance, gains over lexical and document-translated baselines are most pronounced for low-resource and cross-script pairs. These findings indicate that cross-lingual search systems should prioritise semantic multilingual embeddings and targeted learning-based alignment over translation-based pipelines, particularly for cross-script and under-resourced languages.
Authors Roksana Goworek, Olivia Macmillan-Scott, Eda B. Özyiğit
Categories Artificial Intelligence, Computation and Language
0
Entropic Time Schedulers for Generative Diffusion Models
Title CN 生成扩散模型的熵时间调度器
Keywords 扩散模型 · 时间调度 · 熵 · 生成模型 · 采样优化
Summary 本文提出一种基于熵的时间调度方法，用于生成扩散模型，通过在信息增益均衡的时间点进行采样，提升生成效率与质量。该方法不依赖初始时间选择，并可在不增加函数评估次数的情况下显著改善预训练模型的图像生成表现。
Reason 该论文研究生成扩散模型中的噪声调度函数，提出基于熵的时间重参数化方法，以提升生成质量。其核心贡献属于图像生成领域，且聚焦于扩散模型的采样效率与生成质量优化，与文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等无直接关联。虽然理论上任何视觉表示改进可能间接影响多模态系统，但该方法针对的是通用图像生成，迁移至文档理解场景（如处理OCR幻觉、复杂排版或长文档推理）缺乏明确路径和针对性。属于无关领域工作。
Abstract The practical performance of generative diffusion models depends on the appropriate choice of the noise scheduling function, which can also be equivalently expressed as a time reparameterization. In this paper, we present a time scheduler that selects sampling points based on entropy rather than uniform time spacing, ensuring that each point contributes an equal amount of information to the final generation. We prove that this time reparameterization does not depend on the initial choice of time. Furthermore, we provide a tractable exact formula to estimate this \emph{entropic time} for a trained model using the training loss without substantial overhead. Alongside the entropic time, inspired by the optimality results, we introduce a rescaled entropic time. In our experiments with mixtures of Gaussian distributions and ImageNet, we show that using the (rescaled) entropic times greatly improves the inference performance of trained models. In particular, we found that the image quality in pretrained EDM2 models, as evaluated by FID and FD-DINO scores, can be substantially increased by the rescaled entropic time reparameterization without increasing the number of function evaluations, with greater improvements in the few NFEs regime. Code is available at https://github.com/DejanStancevic/Entropic-Time-Schedulers-for-Generative-Diffusion-Models.
Authors Dejan Stancevic, Florian Handke, Luca Ambrogioni
Comment 31 pages
Categories Artificial Intelligence
0
Evaluating Dataset Watermarking for Fine-tuning Traceability of Customized Diffusion Models: A Comprehensive Benchmark and Removal Approach
Title CN 评估用于定制扩散模型微调可追溯性的数据集水印：一个综合基准与去除方法
Keywords 扩散模型 · 数据集水印 · 可追溯性 · 版权保护 · 水印去除
Summary 本文提出了一种针对扩散模型微调过程中数据集水印的综合评估框架，并揭示了现有水印方法在现实威胁场景下的脆弱性。作者还设计了一种有效的水印去除方法，能够在不影响模型微调效果的前提下完全清除水印，从而指出了当前技术面临的安全挑战。
Reason 该论文研究的是扩散模型的数据集水印及其可追溯性问题，属于图像生成与版权保护领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其核心任务聚焦于生成模型的训练数据追踪和水印去除，既不涉及OCR、布局分析、表格识别等DIU任务，也无法迁移用于解决文档理解中的幻觉、密集文本、复杂排版或长文档推理等关键痛点。
Abstract Recent fine-tuning techniques for diffusion models enable them to reproduce specific image sets, such as particular faces or artistic styles, but also introduce copyright and security risks. Dataset watermarking has been proposed to ensure traceability by embedding imperceptible watermarks into training images, which remain detectable in outputs even after fine-tuning. However, current methods lack a unified evaluation framework. To address this, this paper establishes a general threat model and introduces a comprehensive evaluation framework encompassing Universality, Transmissibility, and Robustness. Experiments show that existing methods perform well in universality and transmissibility, and exhibit some robustness against common image processing operations, yet still fall short under real-world threat scenarios. To reveal these vulnerabilities, the paper further proposes a practical watermark removal method that fully eliminates dataset watermarks without affecting fine-tuning, highlighting a key challenge for future research.
Authors Xincheng Wang, Hanchi Sun, Wenjun Sun, Kejun Xue, Wangqiu Zhou, Jianbo Zhang, Wei Sun, Dandan Zhu, Xiongkuo Min, Jun Jia, Zhijun Fang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
How does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspective AAAI 2026 (Oral)
Title CN 对齐如何增强大语言模型的多语言能力？一种基于语言神经元的视角
Keywords 多语言对齐 · 语言神经元 · 大语言模型 · 神经元分析 · AAAI2026
Summary 本文提出一种三元分类方法，将大语言模型中的神经元分为语言特定型、语言相关型和通用型，并设计算法识别这些类型。通过分析对齐前后模型中各类神经元的分布变化，揭示多语言对齐在模型内部四个阶段的作用机制：多语言理解、共享语义空间推理、多语言输出空间转换和词表空间输出。同时探讨了‘自发多语言对齐’现象，为理解多语言能力提供了神经层面的实证洞察。
Reason 该论文研究的是多语言对齐中语言神经元的分类与作用机制，属于纯NLP范畴，聚焦于语言间能力迁移的内部机理分析。虽然涉及LLM的多语言能力，但完全不涉及视觉输入、文档结构或OCR相关问题，与文档图像理解（DIU）的核心痛点如幻觉、排版解析、细粒度定位等无直接关联。且其方法难以迁移到多模态文档场景，特别是中英文之外的小语种分析更触发负面清单中的‘小语种’限制。
Abstract Multilingual Alignment is an effective and representative paradigm to enhance LLMs' multilingual capabilities, which transfers the capabilities from the high-resource languages to the low-resource languages. Meanwhile, some research on language-specific neurons provides a new perspective to analyze and understand LLMs' mechanisms. However, we find that there are many neurons that are shared by multiple but not all languages and cannot be correctly classified. In this work, we propose a ternary classification methodology that categorizes neurons into three types, including language-specific neurons, language-related neurons, and general neurons. And we propose a corresponding identification algorithm to distinguish these different types of neurons. Furthermore, based on the distributional characteristics of different types of neurons, we divide the LLMs' internal process for multilingual inference into four parts: (1) multilingual understanding, (2) shared semantic space reasoning, (3) multilingual output space transformation, and (4) vocabulary space outputting. Additionally, we systematically analyze the models before and after alignment with a focus on different types of neurons. We also analyze the phenomenon of ''Spontaneous Multilingual Alignment''. Overall, our work conducts a comprehensive investigation based on different types of neurons, providing empirical results and valuable insights to better understand multilingual alignment and multilingual capabilities of LLMs.
Authors Shimao Zhang, Zhejian Lai, Xiang Liu, Shuaijie She, Xiao Liu, Yeyun Gong, Shujian Huang, Jiajun Chen
Comment AAAI 2026 (Oral)
Categories Artificial Intelligence, Computation and Language
0
The SA-FARI Dataset: Segment Anything in Footage of Animals for Recognition and Identification
Title CN SA-FARI数据集：用于动物识别与鉴定的片段化任何动物视频
Keywords 多动物跟踪 · 野生动物识别 · 视频分析 · 分割任何模型 · 大规模数据集
Summary 本文提出了SA-FARI，目前最大规模的开源野生动物多动物跟踪数据集，包含近10年来自四大洲的11,609个摄像机陷阱视频，涵盖99个物种，提供高精度的边界框、分割掩码和物种标签。同时发布了相机位置信息，并基于SAM 3等视觉语言模型进行了检测与跟踪的基准测试。目标是推动野外动物群体中可泛化的多目标跟踪研究。
Reason 该论文聚焦于野生动物视频中的多动物跟踪与识别，属于视频理解与生物多样性监测领域，与文档图像理解（DIU）或DocVQA无关。其任务、数据和方法均不涉及文本布局、OCR、表格识别或文档推理等核心问题，无法迁移用于解决文档图像中的幻觉、密集文本或复杂排版等痛点。
Abstract Automated video analysis is critical for wildlife conservation. A foundational task in this domain is multi-animal tracking (MAT), which underpins applications such as individual re-identification and behavior recognition. However, existing datasets are limited in scale, constrained to a few species, or lack sufficient temporal and geographical diversity - leaving no suitable benchmark for training general-purpose MAT models applicable across wild animal populations. To address this, we introduce SA-FARI, the largest open-source MAT dataset for wild animals. It comprises 11,609 camera trap videos collected over approximately 10 years (2014-2024) from 741 locations across 4 continents, spanning 99 species categories. Each video is exhaustively annotated culminating in ~46 hours of densely annotated footage containing 16,224 masklet identities and 942,702 individual bounding boxes, segmentation masks, and species labels. Alongside the task-specific annotations, we publish anonymized camera trap locations for each video. Finally, we present comprehensive benchmarks on SA-FARI using state-of-the-art vision-language models for detection and tracking, including SAM 3, evaluated with both species-specific and generic animal prompts. We also compare against vision-only methods developed specifically for wildlife analysis. SA-FARI is the first large-scale dataset to combine high species diversity, multi-region coverage, and high-quality spatio-temporal annotations, offering a new foundation for advancing generalizable multianimal tracking in the wild. The dataset is available at https://www.conservationxlabs.com/sa-fari.
Authors Dante Francisco Wasmuht, Otto Brookes, Maximillian Schall, Pablo Palencia, Chris Beirne, Tilo Burghardt, Majid Mirmehdi, Hjalmar Kühl, Mimi Arandjelovic, Sam Pottie, Peter Bermant, Brandon Asheim, Yi Jin Toh, Adam Elzinga, Jason Holmberg, Andrew Whitworth, Eleanor Flatt, Laura Gustafson, Chaitanya Ryali, Yuan-Ting Hu, Baishan Guo, Andrew Westbury, Kate Saenko, Didac Suris
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learning
Title CN AutoEnv：用于衡量跨环境智能体学习的自动化环境
Keywords 智能体学习 · 跨环境泛化 · 自动化环境生成 · 强化学习 · 语言模型评测
Summary 本文提出了AutoEnv，一个可自动生成多样化异构环境的框架，用于评测智能体在不同动态、观测和奖励结构下的跨环境学习能力。基于此构建了包含36个环境的AutoEnv-36数据集，并形式化了以组件为中心的智能体学习过程。实验表明现有学习方法在多环境场景下扩展性有限，突显出环境自适应学习的必要性与挑战。
Reason 该论文聚焦于构建通用智能体在多样化虚拟环境中的学习与适应能力，属于具身智能和强化学习领域。其核心贡献是AutoEnv框架和跨环境评估基准，与文档图像理解（DIU）任务无直接关联。尽管涉及‘Agent Learning’，但其目标并非解决文档中的OCR幻觉、复杂排版或长文档推理等具体问题，也无法迁移为提升VLM在文档理解中性能的底层方法。
Abstract Humans naturally adapt to diverse environments by learning underlying rules across worlds with different dynamics, observations, and reward structures. In contrast, existing agents typically demonstrate improvements via self-evolving within a single domain, implicitly assuming a fixed environment distribution. Cross-environment learning has remained largely unmeasured: there is no standard collection of controllable, heterogeneous environments, nor a unified way to represent how agents learn. We address these gaps in two steps. First, we propose AutoEnv, an automated framework that treats environments as factorizable distributions over transitions, observations, and rewards, enabling low-cost (4.12 USD on average) generation of heterogeneous worlds. Using AutoEnv, we construct AutoEnv-36, a dataset of 36 environments with 358 validated levels, on which seven language models achieve 12-49% normalized reward, demonstrating the challenge of AutoEnv-36. Second, we formalize agent learning as a component-centric process driven by three stages of Selection, Optimization, and Evaluation applied to an improvable agent component. Using this formulation, we design eight learning methods and evaluate them on AutoEnv-36. Empirically, the gain of any single learning method quickly decrease as the number of environments increases, revealing that fixed learning methods do not scale across heterogeneous environments. Environment-adaptive selection of learning methods substantially improves performance but exhibits diminishing returns as the method space expands. These results highlight both the necessity and the current limitations of agent learning for scalable cross-environment generalization, and position AutoEnv and AutoEnv-36 as a testbed for studying cross-environment agent learning. The code is avaiable at https://github.com/FoundationAgents/AutoEnv.
Authors Jiayi Zhang, Yiran Peng, Fanqi Kong, Yang Cheng, Yifan Wu, Zhaoyang Yu, Jinyu Xiang, Jianhao Ruan, Jinlin Wang, Maojia Song, HongZhang Liu, Xiangru Tang, Bang Liu, Chenglin Wu, Yuyu Luo
Categories Artificial Intelligence, Computation and Language
0
AI and the Net-Zero Journey: Energy Demand, Emissions, and the Potential for Transition Data Centric Engineering Journal
Title CN 人工智能与净零之旅：能源需求、排放及转型潜力
Keywords 人工智能 · 碳排放 · 数据中心 · 能源消耗 · 可持续发展
Summary 本文是一篇关于数据中心能耗与AI发展对温室气体排放影响的技术综述，分析了到2035年AI可能带来的短期排放增长与长期减排潜力，探讨了AI在能源生产、供应和消费中的优化作用，但未提出任何与文档理解、OCR、多模态建模或视觉推理相关的技术创新。
Reason 该论文讨论AI在能源消耗和碳排放方面的宏观影响，属于政策与可持续发展领域的技术综述，不涉及文档图像理解（DIU）、视觉语言模型（VLM）架构改进、视觉表示学习或任何可迁移至DIU的技术方法。属于无关领域，命中负面清单第2条。
Abstract Thanks to the availability of massive amounts of data, computing resources, and advanced algorithms, AI has entered nearly every sector. This has sparked significant investment and interest, particularly in building data centers with the necessary hardware and software to develop and operate AI models and AI-based workflows. In this technical review article, we present energy consumption scenarios of data centers and impact on GHG emissions, considering both near-term projections (up to 2030) and long-term outlook (2035 and beyond). We address the quintessential question of whether AI will have a net positive, neutral, or negative impact on CO2 emissions by 2035. Additionally, we discuss AI's potential to automate, create efficient and disruptive workflows across various fields related to energy production, supply and consumption. In the near-term scenario, the growing demand for AI will likely strain computing resources, lead to increase in electricity consumption and therefore associated CO2 emissions. This is due to the power-hungry nature of big data centers and the requirements for training and running of large and complex AI models, as well as the penetration of AI assistant search and applications for public use. However, the long-term outlook could be more promising. AI has the potential to be a game-changer in CO2 reduction. Its ability to further automate and optimize processes across industries, from energy production to logistics, could significantly decrease our carbon footprint. This positive impact is anticipated to outweigh the initial emissions bump, creating value for businesses and society in areas where traditional solutions have fallen short. In essence, AI might cause some initial growing pains for the environment, but it has the potential to support climate mitigation efforts.
Authors Pandu Devarakota, Nicolas Tsesmetzis, Faruk O. Alpak, Apurva Gala, Detlef Hohl
Comment Technical article to be submitted to Data Centric Engineering Journal
Categories Artificial Intelligence
0
Open-weight genome language model safeguards: Assessing robustness via adversarial fine-tuning 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop: Biosecurity Safeguards for Generative AI
Title CN 开源权重基因组语言模型的安全保护：通过对抗性微调评估鲁棒性
Keywords 基因组语言模型 · AI安全 · 对抗性微调 · 滥用风险 · NeurIPS
Summary 本文研究了在基因组语言模型（如Evo 2）中通过排除病毒序列进行数据过滤的安全措施是否有效，发现仅靠数据过滤难以阻止攻击者通过微调恢复对有害病毒序列的建模能力，从而提出需要更强的安全框架。实验包括使用110种人类致病病毒序列进行微调，并测试模型对SARS-CoV-2免疫逃逸变异的识别能力。
Reason 该论文研究的是基因组语言模型（gLMs）的安全性和滥用风险，属于生物信息学与AI安全交叉领域，与文档图像理解（DIU）、DocVQA、OCR、多模态文档处理等完全无关。尽管涉及深度学习架构和模型安全性，但其方法和任务均无法迁移到解决文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Novel deep learning architectures are increasingly being applied to biological data, including genetic sequences. These models, referred to as genomic language mod- els (gLMs), have demonstrated impressive predictive and generative capabilities, raising concerns that such models may also enable misuse, for instance via the generation of genomes for human-infecting viruses. These concerns have catalyzed calls for risk mitigation measures. The de facto mitigation of choice is filtering of pretraining data (i.e., removing viral genomic sequences from training datasets) in order to limit gLM performance on virus-related tasks. However, it is not currently known how robust this approach is for securing open-source models that can be fine-tuned using sensitive pathogen data. Here, we evaluate a state-of-the-art gLM, Evo 2, and perform fine-tuning using sequences from 110 harmful human-infecting viruses to assess the rescue of misuse-relevant predictive capabilities. The fine- tuned model exhibited reduced perplexity on unseen viral sequences relative to 1) the pretrained model and 2) a version fine-tuned on bacteriophage sequences. The model fine-tuned on human-infecting viruses also identified immune escape variants from SARS-CoV-2 (achieving an AUROC of 0.6), despite having no expo- sure to SARS-CoV-2 sequences during fine-tuning. This work demonstrates that data exclusion might be circumvented by fine-tuning approaches that can, to some degree, rescue misuse-relevant capabilities of gLMs. We highlight the need for safety frameworks for gLMs and outline further work needed on evaluations and mitigation measures to enable the safe deployment of gLMs.
Authors James R. M. Black, Moritz S. Hanke, Aaron Maiwald, Tina Hernandez-Boussard, Oliver M. Crook, Jaspreet Pannu
Comment 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop: Biosecurity Safeguards for Generative AI
Categories Artificial Intelligence
0
Data Flows and Colonial Regimes in Africa: A Critical Analysis of the Colonial Futurities Embedded in AI Ecosystems
Title CN 非洲的数据流与殖民体制：对人工智能生态系统中嵌入的殖民未来性的批判性分析
Keywords 算法殖民主义 · AI伦理 · 数据权力 · 非洲AI生态 · 社会影响
Summary 本文从权力与利益的角度审视非洲背景下人工智能和大数据的隐形问题，探讨AI推荐算法如何重塑数字社会，并可能导致算法殖民主义和负面性别规范，进而影响区域可持续发展。作者基于在肯尼亚社交媒体用户中的讨论和个人观察提出应采用更具回应性的商业模式，并承认AI存在的多元社会物质世界。
Reason 该论文属于AI伦理与社会影响研究，讨论算法殖民主义和数字权力结构，与文档图像理解（DIU）或视觉语言模型的技术方法创新无关。未涉及任何DIU任务（如DocVQA、布局分析、表格识别等）的技术改进，也未提出可迁移至文档理解领域的底层技术工具。属于无关领域和社会科学视角的讨论，不符合本筛选标准。
Abstract This chapter seeks to frame the elemental and invisible problems of AI and big data in the African context by examining digital sites and infrastructure through the lens of power and interests. It will present reflections on how these sites are using AI recommendation algorithms to recreate new digital societies in the region, how they have the potential to propagate algorithmic colonialism and negative gender norms, and what this means for the regional sustainable development agenda. The chapter proposes adopting business models that embrace response-ability and consider the existence of alternative socio-material worlds of AI. These reflections will mainly come from ongoing discussions with Kenyan social media users in this authors' user space talks, personal experiences and six months of active participant observations done by the authors.
Authors Ndaka. A, Avila-Acosta. F, Mbula-Ndaka. H, Amera. C, Chauke. S, Majiwa. E
Comment 12 pages
Categories Artificial Intelligence
0
A Survey of Generative Categories and Techniques in Multimodal Generative Models
Title CN 多模态生成模型中生成类别与技术的综述
Keywords 多模态生成模型 · 自监督学习 · 思维链 · 伦理与安全 · 生成模态
Summary 本文综述了多模态生成模型的发展，分类了六种主要生成模态，分析了自监督学习、专家混合、人类反馈强化学习和思维链等关键技术在跨模态生成中的作用，提出了以保真度、组合性和鲁棒性为核心的统一评估框架，并探讨了多模态偏见、隐私泄露和深度伪造等安全与伦理风险及应对策略。
Reason 该论文是一篇关于多模态生成模型的综述，涵盖图像、音乐、视频、3D等生成任务，属于通用多模态生成领域的总结性工作。其内容聚焦于生成模态分类、训练技术（如SSL、MoE、RLHF、CoT）和伦理问题，并未针对文档图像理解（DIU）中的具体任务（如DocVQA、表格识别、布局分析）提出任何方法改进或架构创新。同时，文中讨论的技术应用方向与文档理解核心痛点（如OCR幻觉、密集文本处理、复杂排版建模）无直接关联，且属于‘平行领域综述’，不具备可迁移的方法论价值。
Abstract Multimodal Generative Models (MGMs) have rapidly evolved beyond text generation, now spanning diverse output modalities including images, music, video, human motion, and 3D objects, by integrating language with other sensory modalities under unified architectures. This survey categorises six primary generative modalities and examines how foundational techniques, namely Self-Supervised Learning (SSL), Mixture of Experts (MoE), Reinforcement Learning from Human Feedback (RLHF), and Chain-of-Thought (CoT) prompting, enable cross-modal capabilities. We analyze key models, architectural trends, and emergent cross-modal synergies, while highlighting transferable techniques and unresolved challenges. Building on a common taxonomy of models and training recipes, we propose a unified evaluation framework centred on faithfulness, compositionality, and robustness, and synthesise evidence from benchmarks and human studies across modalities. We further analyse trustworthiness, safety, and ethical risks, including multimodal bias, privacy leakage, and the misuse of high-fidelity media generation for deepfakes, disinformation, and copyright infringement in music and 3D assets, together with emerging mitigation strategies. Finally, we discuss how architectural trends, evaluation protocols, and governance mechanisms can be co-designed to close current capability and safety gaps, outlining critical paths toward more general-purpose, controllable, and accountable multimodal generative systems.
Authors Longzhen Han, Awes Mubarak, Almas Baimagambetov, Nikolaos Polatidis, Thar Baker
Categories Artificial Intelligence, Computation and Language
0
Dynamic Multi-Species Bird Soundscape Generation with Acoustic Patterning and 3D Spatialization IEEE Big Data 2025
Title CN 基于声学模式与3D空间化的动态多物种鸟类声音景观生成
Keywords 音频生成 · 数字信号处理 · 3D空间化 · 鸟鸣模拟 · 算法作曲
Summary 本文提出了一种完全基于算法的框架，用于生成动态的多物种鸟类声音景观。该方法不依赖录音或训练数据，而是通过数字信号处理（DSP）生成鸟鸣啁啾声，并结合3D空间化技术模拟多个独立移动的鸟类在三维环境中的声音轨迹。系统支持可控的叫声序列、重叠合唱以及可扩展的声音场景，具备生态真实感，适用于计算机音乐、虚拟交互环境和生物声学研究。
Reason 该论文研究的是鸟类声音景观的生成，属于音频合成与计算生物声学领域，与文档图像理解（DIU）或视觉-语言模型在文档任务中的应用完全无关。其技术路线聚焦于DSP算法和3D音频模拟，无法迁移到解决OCR幻觉、复杂排版理解、细粒度定位或长文档推理等DIU核心问题。
Abstract Generation of dynamic, scalable multi-species bird soundscapes remains a significant challenge in computer music and algorithmic sound design. Birdsongs involve rapid frequency-modulated chirps, complex amplitude envelopes, distinctive acoustic patterns, overlapping calls, and dynamic inter-bird interactions, all of which require precise temporal and spatial control in 3D environments. Existing approaches, whether Digital Signal Processing (DSP)-based or data-driven, typically focus only on single species modeling, static call structures, or synthesis directly from recordings, and often suffer from noise, limited flexibility, or large data needs. To address these challenges, we present a novel, fully algorithm-driven framework that generates dynamic multi-species bird soundscapes using DSP-based chirp generation and 3D spatialization, without relying on recordings or training data. Our approach simulates multiple independently-moving birds per species along different moving 3D trajectories, supporting controllable chirp sequences, overlapping choruses, and realistic 3D motion in scalable soundscapes while preserving species-specific acoustic patterns. A visualization interface provides bird trajectories, spectrograms, activity timelines, and sound waves for analytical and creative purposes. Both visual and audio evaluations demonstrate the ability of the system to generate dense, immersive, and ecologically inspired soundscapes, highlighting its potential for computer music, interactive virtual environments, and computational bioacoustics research.
Authors Ellie L. Zhang, Duoduo Liao, Callie C. Liao
Comment Accepted by IEEE Big Data 2025
Categories Artificial Intelligence
0
VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval
Title CN VideoLights：用于联合视频精彩片段检测与时刻检索的特征优化与跨任务对齐Transformer
Keywords 视频理解 · 多模态融合 · 时刻检索 · 精彩片段检测 · LVLM
Summary 本文提出VideoLights框架，用于联合完成视频精彩片段检测与时刻检索任务。通过引入特征优化模块、双向跨模态融合网络、单向联合任务反馈机制以及基于LVLM的合成数据预训练，在多个视频基准上实现了SOTA性能。
Reason 该论文研究的是视频理解领域的联合任务（视频精彩片段检测与时刻检索），属于视频分析范畴，与文档图像理解（DIU）的核心任务如DocVQA、版面分析、表格识别等无直接关联。尽管使用了LVLM进行多模态特征融合，但其方法设计针对的是时间序列视频数据和自然语言查询的对齐问题，难以迁移到文档图像中常见的高分辨率、密集文本、复杂排版或OCR去幻觉等核心痛点。属于无关领域。
Abstract Prevailing joint prediction transformers for Video Highlight Detection and Moment Retrieval (HD/MR) exhibit deficiencies in handling cross-task dynamics, achieving robust video-text alignment, and utilizing effective attention mechanisms, with the potential of Large Language/Vision-Language Models (LLMs/LVLMs) being largely untapped. This paper introduces VideoLights, a novel HD/MR framework addressing these limitations by incorporating: (i) Convolutional Projection and Feature Refinement modules with an alignment loss for enhanced video-text feature congruity; (ii) a Bi-Directional Cross-Modal Fusion network for strongly coupled query-aware representations; (iii) a Uni-directional joint-task feedback mechanism for synergistic task improvement; (iv) hard positive/negative losses for adaptive learning; and (v) the leveraging of LVLMs (e.g., BLIP-2) for superior multimodal feature integration and intelligent pre-training with synthetic data. Comprehensive evaluations on QVHighlights, TVSum, and Charades-STA benchmarks demonstrate that VideoLights significantly surpasses existing baselines, establishing new state-of-the-art performances. Codes and model checkpoints are available at https://github.com/dpaul06/VideoLights .
Authors Dhiman Paul, Md Rizwan Parvez, Nabeel Mohammed, Shafin Rahman
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Interpreting GFlowNets for Drug Discovery: Extracting Actionable Insights for Medicinal Chemistry NeurIPS 2025 WiML Workshop and Molecular Machine Learning Conference (MoML) 2025
Title CN 用于药物发现的GFlowNets解释：为药物化学提取可操作的见解
Keywords GFlowNets · 可解释性 · 药物发现 · 分子生成 · 化学反应路径
Summary 本文提出了一种针对SynFlowNet（一种基于GFlowNet的分子生成模型）的可解释性框架，通过梯度显著性、反事实扰动、稀疏自编码器和基序探针等方法，揭示模型在分子生成过程中依赖的物理化学属性和功能基团。目标是为药物化学家提供透明且可操作的设计依据。
Reason 该论文属于具象化生成模型在药物发现中的应用，研究内容聚焦于分子设计与合成路径生成的可解释性，属于典型的‘平行下游应用’。尽管其方法（如稀疏自编码器、motif探针）具有一定分析价值，但整体工作完全围绕化学领域定制，无法迁移至文档图像理解中的核心问题（如OCR幻觉、复杂排版理解、细粒度定位等）。且未涉及多模态输入处理、视觉表示学习或VLM架构改进，对DIU无直接方法论贡献。
Abstract Generative Flow Networks, or GFlowNets, offer a promising framework for molecular design, but their internal decision policies remain opaque. This limits adoption in drug discovery, where chemists require clear and interpretable rationales for proposed structures. We present an interpretability framework for SynFlowNet, a GFlowNet trained on documented chemical reactions and purchasable starting materials that generates both molecules and the synthetic routes that produce them. Our approach integrates three complementary components. Gradient based saliency combined with counterfactual perturbations identifies which atomic environments influence reward and how structural edits change molecular outcomes. Sparse autoencoders reveal axis aligned latent factors that correspond to physicochemical properties such as polarity, lipophilicity, and molecular size. Motif probes show that functional groups including aromatic rings and halogens are explicitly encoded and linearly decodable from the internal embeddings. Together, these results expose the chemical logic inside SynFlowNet and provide actionable and mechanistic insight that supports transparent and controllable molecular design.
Authors Amirtha Varshini A S, Duminda S. Ranasinghe, Hok Hei Tam
Comment 13 pages, 7 figures. Accepted for presentation at NeurIPS 2025 WiML Workshop and Molecular Machine Learning Conference (MoML) 2025
Categories Artificial Intelligence
0
Solar-GECO: Perovskite Solar Cell Property Prediction with Geometric-Aware Co-Attention AI for Accelerated Materials Design (AI4Mat) Workshop at NeurIPS 2025
Title CN Solar-GECO：基于几何感知协同注意力的钙钛矿太阳能电池性能预测
Keywords 材料预测 · 图神经网络 · 多模态融合 · 协同注意力 · 性能预测
Summary 本文提出Solar-GECO模型，用于预测钙钛矿太阳能电池的光电转换效率（PCE）。该模型结合几何图神经网络（编码钙钛矿晶体原子结构）与语言模型（处理传输层化学成分文本），并通过协同注意力模块捕捉层内依赖与层间交互，最后用概率回归头输出PCE及其不确定性。实验表明其在PCE预测上优于现有方法。
Reason 该论文属于材料科学领域的机器学习应用，研究任务为太阳能电池效率预测，与文档图像理解（DIU）、DocVQA、OCR、多模态文档建模等完全无关。尽管使用了多模态融合（图神经网络+语言模型）和协同注意力机制，但其方法设计面向晶体结构与化学文本的结合，无法迁移至文档布局分析、细粒度文本识别或复杂排版理解等DIU核心问题。
Abstract Perovskite solar cells are promising candidates for next-generation photovoltaics. However, their performance as multi-scale devices is determined by complex interactions between their constituent layers. This creates a vast combinatorial space of possible materials and device architectures, making the conventional experimental-based screening process slow and expensive. Machine learning models try to address this problem, but they only focus on individual material properties or neglect the important geometric information of the perovskite crystal. To address this problem, we propose to predict perovskite solar cell power conversion efficiency with a geometric-aware co-attention (Solar-GECO) model. Solar-GECO combines a geometric graph neural network (GNN) - that directly encodes the atomic structure of the perovskite absorber - with language model embeddings that process the textual strings representing the chemical compounds of the transport layers and other device components. Solar-GECO also integrates a co-attention module to capture intra-layer dependencies and inter-layer interactions, while a probabilistic regression head predicts both power conversion efficiency (PCE) and its associated uncertainty. Solar-GECO achieves state-of-the-art performance, significantly outperforming several baselines, reducing the mean absolute error (MAE) for PCE prediction from 3.066 to 2.936 compared to semantic GNN (the previous state-of-the-art model). Solar-GECO demonstrates that integrating geometric and textual information provides a more powerful and accurate framework for PCE prediction.
Authors Lucas Li, Jean-Baptiste Puel, Florence Carton, Dounya Barrit, Jhony H. Giraldo
Comment Accepted at the AI for Accelerated Materials Design (AI4Mat) Workshop at NeurIPS 2025. 14 pages, 4 figures
Categories Artificial Intelligence
0
Psychometric Tests for AI Agents and Their Moduli Space
Title CN AI代理的心理测量测试及其模空间
Keywords AI代理 · 心理测量 · 通用智能评分 · 模空间 · AAI功能
Summary 本文从模理论视角提出一种针对AI代理的心理测量测试体系，并定义了AAI功能、认知核心及等价测试电池的不变性。研究聚焦于通用智能的抽象评分机制，属于理论AI范畴，无具体多模态或文档理解应用。
Reason 该论文属于纯理论人工智能研究，涉及心理测量学与AI智能评分的抽象数学建模（如AAI功能、认知核心、模空间等），未涉及任何文档图像理解（DIU）、多模态输入处理、视觉语言模型（VLM）架构或与OCR、布局分析、表格识别等相关任务。其方法论无法迁移解决DIU中的OCR幻觉、密集文本、复杂排版或长文档推理等实际问题。
Abstract We develop a moduli-theoretic view of psychometric test batteries for AI agents and connect it explicitly to the AAI score developed previously. First, we make precise the notion of an AAI functional on a battery and set out axioms that any reasonable autonomy/general intelligence score should satisfy. Second, we show that the composite index ('AAI-Index') defined previously is a special case of our AAI functional. Third, we introduce the notion of a cognitive core of an agent relative to a battery and define the associated AAI$_{\textrm{core}}$ score as the restriction of an AAI functional to that core. Finally, we use these notions to describe invariants of batteries under evaluation-preserving symmetries and outline how moduli of equivalent batteries are organized.
Authors Przemyslaw Chojecki
Categories Artificial Intelligence
0
A Nutrition Multimodal Photoplethysmography Language Model
Title CN 一种营养多模态光电容积脉搏波语言模型
Keywords 多模态学习 · 光电容积脉搏波 · 饮食监测 · 可穿戴设备 · 语言模型
Summary 本文提出了一种名为NPLM的多模态语言模型，通过融合来自可穿戴设备的连续光电容积脉搏波（PPG）信号和饮食描述文本，实现对日常卡路里摄入的非侵入式大规模监测。模型将PPG信号映射为语言模型可理解的嵌入表示，从而支持生理数据与饮食上下文的联合推理。实验表明，相比仅使用文本的方法，NPLM在卡路里预测上提升了11%，且在80%文本缺失的情况下仍保持准确率。
Reason 该论文属于医疗健康领域的多模态应用，核心是将可穿戴设备的PPG生理信号与饮食文本结合进行热量摄入预测，任务场景为健康监测，与文档图像理解（DIU）或DocVQA无关。未涉及OCR、布局分析、表格识别、视觉文本理解等DIU核心问题，也无法提供可用于解决OCR幻觉、复杂排版、细粒度定位等文档理解痛点的方法论迁移价值。
Abstract Hunger and satiety dynamics shape dietary behaviors and metabolic health, yet remain difficult to capture in everyday settings. We present a Nutrition Photoplethysmography Language Model (NPLM), integrating continuous photoplethysmography (PPG) from wearables with meal descriptions. NPLM projects PPG into embeddings interpretable by language models, enabling joint reasoning over physiology and meal context. Trained on 19,340 participants and 1.1 million meal-PPG pairs, the model improved daily caloric intake prediction by 11% over text-only baselines, with accuracy maintained when 80% of meal text was removed. In an independent validation study (n=140) with controlled dining and detailed meal information, the model replicated these findings. These results demonstrate the value of integrating physiological measurements from consumer wearables with meal information for noninvasive dietary monitoring at scale.
Authors Kyle Verrier, Achille Nazaret, Joseph Futoma, Andrew C. Miller, Guillermo Sapiro
Comment 21 pages, 2 figures
Categories Artificial Intelligence, Computation and Language
0
Medusa: Cross-Modal Transferable Adversarial Attacks on Multimodal Medical Retrieval-Augmented Generation KDD 2026
Title CN Medusa：针对多模态医疗检索增强生成的跨模态可迁移对抗攻击
Keywords 对抗攻击 · 检索增强生成 · 多模态安全 · 医疗AI · 黑盒攻击
Summary 本文提出Medusa框架，用于在黑盒环境下对多模态医疗检索增强生成（MMed-RAG）系统发起跨模态可迁移的对抗攻击。通过优化视觉输入扰动，利用多正例InfoNCE损失（MPIL）使恶意视觉嵌入匹配错误但看似合理的文本目标，从而劫持检索过程。采用代理模型集成和双循环优化策略提升攻击迁移性，并结合不变风险最小化增强鲁棒性。实验显示其在多种模型上攻击成功率超过90%，并能抵御主流防御手段。
Reason 该论文属于特定领域的下游应用（医疗+RAG），研究的是对医疗检索增强系统的对抗攻击，虽然涉及多模态，但其方法聚焦于安全漏洞和攻击范式，并非文档图像理解（DIU）中的核心问题如OCR幻觉、复杂排版解析或长文档推理。且任务场景为垂直领域（医学报告生成与诊断）的对抗攻击，属于典型的‘平行应用’，无通用DIU方法论创新。
Abstract With the rapid advancement of retrieval-augmented vision-language models, multimodal medical retrieval-augmented generation (MMed-RAG) systems are increasingly adopted in clinical decision support. These systems enhance medical applications by performing cross-modal retrieval to integrate relevant visual and textual evidence for tasks, e.g., report generation and disease diagnosis. However, their complex architecture also introduces underexplored adversarial vulnerabilities, particularly via visual input perturbations. In this paper, we propose Medusa, a novel framework for crafting cross-modal transferable adversarial attacks on MMed-RAG systems under a black-box setting. Specifically, Medusa formulates the attack as a perturbation optimization problem, leveraging a multi-positive InfoNCE loss (MPIL) to align adversarial visual embeddings with medically plausible but malicious textual targets, thereby hijacking the retrieval process. To enhance transferability, we adopt a surrogate model ensemble and design a dual-loop optimization strategy augmented with invariant risk minimization (IRM). Extensive experiments on two real-world medical tasks, including medical report generation and disease diagnosis, demonstrate that Medusa achieves over 90% average attack success rate across various generation models and retrievers under appropriate parameter configuration, while remaining robust against four mainstream defenses, outperforming state-of-the-art baselines. Our results reveal critical vulnerabilities in the MMed-RAG systems and highlight the necessity of robustness benchmarking in safety-critical medical applications. The code and data are available at https://anonymous.4open.science/r/MMed-RAG-Attack-F05A.
Authors Yingjia Shang, Yi Liu, Huimin Wang, Furong Li, Wenfang Sun, Wu Chengyu, Yefeng Zheng
Comment Accepted at KDD 2026 First Cycle (full version). Authors marked with * contributed equally. Yi Liu is the lead author
Categories Artificial Intelligence
0
SimDiff: Simpler Yet Better Diffusion Model for Time Series Point Forecasting AAAI 2026
Title CN SimDiff：用于时间序列点预测的更简单但更好的扩散模型
Keywords 时间序列预测 · 扩散模型 · 点估计 · Transformer · 去噪
Summary 本文提出SimDiff，一种用于时间序列点预测的端到端扩散模型，通过统一的Transformer网络同时作为去噪器和预测器，提升点估计精度。引入归一化独立性和中位均值估计器以增强稳定性与适应性，在多个数据集上实现了SOTA性能。
Reason 该论文研究的是时间序列预测任务，属于纯时序建模领域，与文档图像理解（DIU）或视觉-语言多模态处理无关。尽管使用了Transformer和扩散模型等通用架构，但其方法设计针对的是数值型时间序列的点估计问题，无法迁移解决OCR幻觉、复杂排版解析、细粒度视觉定位或长文档推理等DIU核心痛点。
Abstract Diffusion models have recently shown promise in time series forecasting, particularly for probabilistic predictions. However, they often fail to achieve state-of-the-art point estimation performance compared to regression-based methods. This limitation stems from difficulties in providing sufficient contextual bias to track distribution shifts and in balancing output diversity with the stability and precision required for point forecasts. Existing diffusion-based approaches mainly focus on full-distribution modeling under probabilistic frameworks, often with likelihood maximization objectives, while paying little attention to dedicated strategies for high-accuracy point estimation. Moreover, other existing point prediction diffusion methods frequently rely on pre-trained or jointly trained mature models for contextual bias, sacrificing the generative flexibility of diffusion models. To address these challenges, we propose SimDiff, a single-stage, end-to-end framework. SimDiff employs a single unified Transformer network carefully tailored to serve as both denoiser and predictor, eliminating the need for external pre-trained or jointly trained regressors. It achieves state-of-the-art point estimation performance by leveraging intrinsic output diversity and improving mean squared error accuracy through multiple inference ensembling. Key innovations, including normalization independence and the median-of-means estimator, further enhance adaptability and stability. Extensive experiments demonstrate that SimDiff significantly outperforms existing methods in time series point forecasting.
Authors Hang Ding, Xue Wang, Tian Zhou, Tao Yao
Comment Accepted by AAAI 2026
Categories Artificial Intelligence
0
Adversarial Patch Attacks on Vision-Based Cargo Occupancy Estimation via Differentiable 3D Simulation
Title CN 基于可微分3D仿真对视觉型货物占用估计的对抗补丁攻击
Keywords 对抗攻击 · 3D仿真 · 可微分渲染 · 计算机视觉 · 物流安全
Summary 本文研究了针对基于视觉的货运车厢占用率估计系统的物理对抗性攻击，利用Mitsuba 3进行可微分渲染，在3D模拟环境中优化对抗补丁纹理，以在不同几何、光照和视角变化下实现高成功率的攻击。实验表明，3D优化的补丁在拒绝服务场景（空→满）下攻击成功率达84.94%，而在隐蔽攻击（满→空）场景下为30.32%。作者分析了影响攻击效果的因素，并讨论了自动化物流系统物理安全性的改进方向。
Reason 该论文研究的是物流场景中货物占用率估计系统的对抗性攻击问题，属于自动驾驶/具身智能相关的安全攻防范畴，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法聚焦于3D仿真环境下的对抗补丁生成，技术路径难以迁移到文档理解中的OCR幻觉、复杂排版解析或长文档推理等核心痛点。
Abstract Computer vision systems are increasingly adopted in modern logistics operations, including the estimation of trailer occupancy for planning, routing, and billing. Although effective, such systems may be vulnerable to physical adversarial attacks, particularly adversarial patches that can be printed and placed on interior surfaces. In this work, we study the feasibility of such attacks on a convolutional cargo-occupancy classifier using fully simulated 3D environments. Using Mitsuba 3 for differentiable rendering, we optimize patch textures across variations in geometry, lighting, and viewpoint, and compare their effectiveness to a 2D compositing baseline. Our experiments demonstrate that 3D-optimized patches achieve high attack success rates, especially in a denial-of-service scenario (empty to full), where success reaches 84.94 percent. Concealment attacks (full to empty) prove more challenging but still reach 30.32 percent. We analyze the factors influencing attack success, discuss implications for the security of automated logistics pipelines, and highlight directions for strengthening physical robustness. To our knowledge, this is the first study to investigate adversarial patch attacks for cargo-occupancy estimation in physically realistic, fully simulated 3D scenes.
Authors Mohamed Rissal Hedna, Sesugh Samuel Nder
Comment 9 pages, 5 figures, 1 algorithm
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
Title CN Live-SWE-agent：软件工程代理能否实时自我进化？
Keywords 软件工程代理 · 大语言模型 · 自我演化 · 运行时优化 · 自动化编程
Summary 本文提出Live-SWE-agent，一种能够在运行时自主持续进化的软件工程代理，从基础架构开始，在解决实际软件问题的过程中动态优化自身结构。在SWE-bench Verified和SWE-Bench Pro两个基准上取得了当前最优的解决率。然而，该工作聚焦于软件开发场景下的LLM代理设计，不具备向文档理解任务迁移的方法论价值。
Reason 该论文属于特定领域的应用型Agent研究（软件工程），尽管提出了自我演化的机制，但其核心是面向代码生成与软件任务的闭环系统优化，并未提出可迁移至文档图像理解领域的底层方法。不涉及OCR、视觉布局分析、多模态对齐或高分辨率视觉处理等DIU核心问题，属于‘平行下游应用’，不符合关注标准。
Abstract Large Language Models (LLMs) are reshaping almost all industries, including software engineering. In recent years, a number of LLM agents have been proposed to solve real-world software problems. Such software agents are typically equipped with a suite of coding tools and can autonomously decide the next actions to form complete trajectories to solve end-to-end software tasks. While promising, they typically require dedicated design and may still be suboptimal, since it can be extremely challenging and costly to exhaust the entire agent scaffold design space. Recognizing that software agents are inherently software themselves that can be further refined/modified, researchers have proposed a number of self-improving software agents recently, including the Darwin-Gödel Machine (DGM). Meanwhile, such self-improving agents require costly offline training on specific benchmarks and may not generalize well across different LLMs or benchmarks. In this paper, we propose Live-SWE-agent, the first live software agent that can autonomously and continuously evolve itself on-the-fly during runtime when solving real-world software problems. More specifically, Live-SWE-agent starts with the most basic agent scaffold with only access to bash tools (e.g., mini-SWE-agent), and autonomously evolves its own scaffold implementation while solving real-world software problems. Our evaluation on the widely studied SWE-bench Verified benchmark shows that LIVE-SWE-AGENT can achieve an impressive solve rate of 77.4% without test-time scaling, outperforming all existing software agents, including the best proprietary solution. Moreover, Live-SWE-agent outperforms state-of-the-art manually crafted software agents on the recent SWE-Bench Pro benchmark, achieving the best-known solve rate of 45.8%.
Authors Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, Lingming Zhang
Categories Artificial Intelligence, Computation and Language
0
Neural Architecture Search for Quantum Autoencoders
Title CN 用于量子自编码器的神经架构搜索
Keywords 量子自编码器 · 神经架构搜索 · 遗传算法 · 变分量子电路 · 量子机器学习
Summary 本文提出了一种基于遗传算法的神经架构搜索框架，用于自动设计量子自编码器的变分量子电路结构，旨在近噪声量子时代实现高效的图像数据特征提取与重建。研究聚焦于量子机器学习中的架构自动化问题。
Reason 该论文研究的是量子机器学习领域中的量子自编码器架构搜索，属于量子计算与机器学习的交叉方向，与文档图像理解（DIU）或视觉语言模型（VLM）在任务、数据和方法上均无直接关联。其提出的方法基于遗传算法优化变分量子电路，难以迁移到文档理解中的OCR幻觉、复杂排版解析或长文档推理等核心问题。属于无关领域。
Abstract In recent years, machine learning and deep learning have driven advances in domains such as image classification, speech recognition, and anomaly detection by leveraging multi-layer neural networks to model complex data. Simultaneously, quantum computing (QC) promises to address classically intractable problems via quantum parallelism, motivating research in quantum machine learning (QML). Among QML techniques, quantum autoencoders show promise for compressing high-dimensional quantum and classical data. However, designing effective quantum circuit architectures for quantum autoencoders remains challenging due to the complexity of selecting gates, arranging circuit layers, and tuning parameters. This paper proposes a neural architecture search (NAS) framework that automates the design of quantum autoencoders using a genetic algorithm (GA). By systematically evolving variational quantum circuit (VQC) configurations, our method seeks to identify high-performing hybrid quantum-classical autoencoders for data reconstruction without becoming trapped in local minima. We demonstrate effectiveness on image datasets, highlighting the potential of quantum autoencoders for efficient feature extraction within a noise-prone, near-term quantum era. Our approach lays a foundation for broader application of genetic algorithms to quantum architecture search, aiming for a robust, automated method that can adapt to varied data and hardware constraints.
Authors Hibah Agha, Samuel Yen-Chi Chen, Huan-Hsin Tseng, Shinjae Yoo
Categories Artificial Intelligence
0
SENTINEL: A Fully End-to-End Language-Action Model for Humanoid Whole Body Control
Title CN SENTINEL：一种用于人形机器人全身控制的端到端语言-动作模型
Keywords 具身智能 · 语言-动作模型 · 人形机器人 · 端到端控制 · 动作生成
Summary 本文提出SENTINEL，一个端到端的语言-动作模型，用于实现人形机器人的全身控制。通过在仿真环境中采集人类运动数据并配以文本标注，构建大规模训练集，模型直接将语言指令和本体感知输入映射为低层动作输出，无需中间表示。采用流匹配生成动作块，并通过残差动作头进行精细化调整，支持多模态输入扩展。实验表明该方法在仿真和真实机器人上均具备良好的语义理解和执行稳定性。
Reason 该论文属于具身智能与机器人控制领域，研究的是将语言指令映射到人形机器人的全身动作控制，完全不涉及文档图像理解、视觉文本识别、布局分析或任何DIU相关任务。其方法论聚焦于物理仿真中的动作生成与流匹配（flow matching），无法迁移至解决OCR幻觉、密集文本识别、复杂排版解析或长文档推理等文档理解核心问题。
Abstract Existing humanoid control systems often rely on teleoperation or modular generation pipelines that separate language understanding from physical execution. However, the former is entirely human-driven, and the latter lacks tight alignment between language commands and physical behaviors. In this paper, we present SENTINEL, a fully end-to-end language-action model for humanoid whole-body control. We construct a large-scale dataset by tracking human motions in simulation using a pretrained whole body controller, combined with their text annotations. The model directly maps language commands and proprioceptive inputs to low-level actions without any intermediate representation. The model generates action chunks using flow matching, which can be subsequently refined by a residual action head for real-world deployment. Our method exhibits strong semantic understanding and stable execution on humanoid robots in both simulation and real-world deployment, and also supports multi-modal extensions by converting inputs into texts.
Authors Yuxuan Wang, Haobin Jiang, Shiqing Yao, Ziluo Ding, Zongqing Lu
Comment 23 pages, 8 figures, 11 tables
Categories Artificial Intelligence
0
Automatic Multi-View X-Ray/CT Registration Using Bone Substructure Contours IPCAI 2025
Title CN 基于骨结构轮廓的自动多视角X光/CT配准
Keywords 医学图像配准 · X光 · CT · ICP算法 · 手术导航
Summary 本文提出了一种基于骨亚结构轮廓的多视角X光与CT图像配准方法，通过改进ICP算法匹配特定骨骼子结构的轮廓，实现了亚毫米级精度的全自动配准，并在真实尸体标本上验证了性能。相比需人工干预的商业方案显著提升了准确性和自动化程度。
Reason 该论文属于医学图像配准领域，研究的是X光与CT影像在骨科手术中的三维配准问题，尽管使用了计算机视觉和AI技术，但其任务与文档图像理解（DIU）、DocVQA、OCR、布局分析等核心问题完全无关。属于典型的无关领域工作（医疗影像+3D配准），无法迁移到文档理解的任何痛点（如幻觉、排版、密集文本等）。
Abstract Purpose: Accurate intraoperative X-ray/CT registration is essential for surgical navigation in orthopedic procedures. However, existing methods struggle with consistently achieving sub-millimeter accuracy, robustness under broad initial pose estimates or need manual key-point annotations. This work aims to address these challenges by proposing a novel multi-view X-ray/CT registration method for intraoperative bone registration. Methods: The proposed registration method consists of a multi-view, contour-based iterative closest point (ICP) optimization. Unlike previous methods, which attempt to match bone contours across the entire silhouette in both imaging modalities, we focus on matching specific subcategories of contours corresponding to bone substructures. This leads to reduced ambiguity in the ICP matches, resulting in a more robust and accurate registration solution. This approach requires only two X-ray images and operates fully automatically. Additionally, we contribute a dataset of 5 cadaveric specimens, including real X-ray images, X-ray image poses and the corresponding CT scans. Results: The proposed registration method is evaluated on real X-ray images using mean reprojection error (mRPD). The method consistently achieves sub-millimeter accuracy with a mRPD 0.67mm compared to 5.35mm by a commercial solution requiring manual intervention. Furthermore, the method offers improved practical applicability, being fully automatic. Conclusion: Our method offers a practical, accurate, and efficient solution for multi-view X-ray/CT registration in orthopedic surgeries, which can be easily combined with tracking systems. By improving registration accuracy and minimizing manual intervention, it enhances intraoperative navigation, contributing to more accurate and effective surgical outcomes in computer-assisted surgery (CAS).
Authors Roman Flepp, Leon Nissen, Bastian Sigrist, Arend Nieuwland, Nicola Cavalcanti, Philipp Fürnstahl, Thomas Dreher, Lilian Calvet
Comment This paper was accepted to IPCAI 2025. The Project Webpage is: https://rflepp.github.io/BoneSubstructureContours2D3DRegistration/
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Learning Plug-and-play Memory for Guiding Video Diffusion Models
Title CN 为视频扩散模型学习即插即用的记忆
Keywords 视频生成 · 扩散模型 · 记忆机制 · 隐状态干预 · 表征解耦
Summary 本文提出DiT-Mem，一种即插即用的记忆模块，用于向视频扩散Transformer（DiT）中注入世界知识。通过分析发现，DiT的隐藏状态可通过低通和高通滤波分离外观与语义特征，从而实现定向引导。作者设计了一个由3D CNN和自注意力组成的可学习记忆编码器，将参考视频编码为紧凑的记忆令牌，并插入DiT的自注意力层中。训练时仅优化记忆编码器，保持主干冻结，实现在少量数据和参数下有效提升视频的物理合理性和视觉保真度。
Reason 该论文研究的是视频生成中的物理规律建模问题，通过引入记忆机制来引导视频扩散模型遵循常识动态。尽管其‘隐状态干预’和‘频域分离表征’的思想具有一定方法启发性，但整体工作完全聚焦于视频生成领域，属于纯图像生成任务，与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版解析、细粒度文本定位等无直接关联。所提方法迁移至文档理解场景的路径不清晰，且未涉及多模态语言模型在结构化信息提取中的应用。根据负面清单，纯图像生成/修复类工作应直接拒绝。
Abstract Diffusion Transformer(DiT) based video generation models have recently achieved impressive visual quality and temporal coherence, but they still frequently violate basic physical laws and commonsense dynamics, revealing a lack of explicit world knowledge. In this work, we explore how to equip them with a plug-and-play memory that injects useful world knowledge. Motivated by in-context memory in Transformer-based LLMs, we conduct empirical studies to show that DiT can be steered via interventions on its hidden states, and simple low-pass and high-pass filters in the embedding space naturally disentangle low-level appearance and high-level physical/semantic cues, enabling targeted guidance. Building on these observations, we propose a learnable memory encoder DiT-Mem, composed of stacked 3D CNNs, low-/high-pass filters, and self-attention layers. The encoder maps reference videos into a compact set of memory tokens, which are concatenated as the memory within the DiT self-attention layers. During training, we keep the diffusion backbone frozen, and only optimize the memory encoder. It yields a rather efficient training process on few training parameters (150M) and 10K data samples, and enables plug-and-play usage at inference time. Extensive experiments on state-of-the-art models demonstrate the effectiveness of our method in improving physical rule following and video fidelity. Our code and data are publicly released here: https://thrcle421.github.io/DiT-Mem-Web/.
Authors Selena Song, Ziming Xu, Zijun Zhang, Kun Zhou, Jiaxian Guo, Lianhui Qin, Biwei Huang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Learning to Call: A Field Trial of a Collaborative Bandit Algorithm for Improved Message Delivery in Mobile Maternal Health
Title CN 学习呼叫：一种用于改善移动 maternal 健康信息传递的协作性赌博算法的实地试验
Keywords 移动健康 · 个性化调度 · 多臂赌博机 · 实地试验 · 信息传递
Summary 本文在印度Kilkari母幼健康语音通知项目中试点了一种协作性赌博算法，通过学习个体用户接听电话的习惯来优化自动语音呼叫的时间安排。实验结果显示该方法显著提高了接通率，证明了个性化调度在移动健康干预中的有效性。然而，该研究聚焦于服务交付优化，而非人工智能基础技术突破。
Reason 该论文属于特定领域的下游应用（移动健康中的消息传递优化），使用了强化学习中的多臂赌博机算法进行个性化呼叫时间调度。尽管使用了机器学习方法，但其核心是将已有算法应用于公共卫生管理场景，属于典型的‘平行应用’，未提出任何文档图像理解（DIU）或视觉-语言模型（VLM）相关的技术改进，与OCR、文档布局分析、表格识别、DocVQA等任务完全无关。
Abstract Mobile health (mHealth) programs utilize automated voice messages to deliver health information, particularly targeting underserved communities, demonstrating the effectiveness of using mobile technology to disseminate crucial health information to these populations, improving health outcomes through increased awareness and behavioral change. India's Kilkari program delivers vital maternal health information via weekly voice calls to millions of mothers. However, the current random call scheduling often results in missed calls and reduced message delivery. This study presents a field trial of a collaborative bandit algorithm designed to optimize call timing by learning individual mothers' preferred call times. We deployed the algorithm with around $6500$ Kilkari participants as a pilot study, comparing its performance to the baseline random calling approach. Our results demonstrate a statistically significant improvement in call pick-up rates with the bandit algorithm, indicating its potential to enhance message delivery and impact millions of mothers across India. This research highlights the efficacy of personalized scheduling in mobile health interventions and underscores the potential of machine learning to improve maternal health outreach at scale.
Authors Arpan Dasgupta, Mizhaan Maniyar, Awadhesh Srivastava, Sanat Kumar, Amrita Mahale, Aparna Hegde, Arun Suggala, Karthikeyan Shanmugam, Aparna Taneja, Milind Tambe
Categories Artificial Intelligence
0
Torsion-Space Diffusion for Protein Backbone Generation with Geometric Refinement
Title CN 基于扭转空间扩散与几何优化的蛋白质骨架生成
Keywords 蛋白质设计 · 扩散模型 · 扭转空间 · 几何约束 · 三维结构生成
Summary 本文提出了一种在扭转角空间中进行扩散的蛋白质骨架生成方法，通过去噪二面角来保证局部几何结构的正确性，并结合可微分运动学模块和约束优化提升全局紧凑性。实验表明该方法能100%保持键长准确性和显著降低回转半径误差。
Reason 该论文属于生物信息学与分子建模领域，研究内容为在扭转角空间中进行蛋白质骨架生成，解决的是蛋白质结构的几何约束问题。尽管使用了扩散模型和几何优化等技术，但其任务与文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的应用完全无关，无法迁移用于解决OCR幻觉、复杂排版、细粒度定位或长文档推理等核心痛点。
Abstract Designing new protein structures is fundamental to computational biology, enabling advances in therapeutic molecule discovery and enzyme engineering. Existing diffusion-based generative models typically operate in Cartesian coordinate space, where adding noise disrupts strict geometric constraints such as fixed bond lengths and angles, often producing physically invalid structures. To address this limitation, we propose a Torsion-Space Diffusion Model that generates protein backbones by denoising torsion angles, ensuring perfect local geometry by construction. A differentiable forward-kinematics module reconstructs 3D coordinates with fixed 3.8 Angstrom backbone bond lengths while a constrained post-processing refinement optimizes global compactness via Radius of Gyration (Rg) correction, without violating bond constraints. Experiments on standard PDB proteins demonstrate 100% bond-length accuracy and significantly improved structural compactness, reducing Rg error from 70% to 18.6% compared to Cartesian diffusion baselines. Overall, this hybrid torsion-diffusion plus geometric-refinement framework generates physically valid and compact protein backbones, providing a promising path toward full-atom protein generation.
Authors Lakshaditya Singh, Adwait Shelke, Divyansh Agrawal
Comment 5 pages, 4 figures
Categories Artificial Intelligence
0
LLM-Based Agentic Negotiation for 6G: Addressing Uncertainty Neglect and Tail-Event Risk
Title CN 基于大语言模型的代理协商用于6G：应对不确定性忽视与尾部事件风险
Keywords 6G · 大语言模型代理 · 尾部风险 · 数字孪生 · CVaR
Summary 本文提出了一种基于大语言模型代理的6G网络切片协商框架，通过引入极端值理论中的CVaR方法来评估尾部延迟风险，并结合数字孪生预测完整延迟分布，使代理在决策时关注极端情况而非平均表现。同时要求代理量化自身预测的不确定性以提升决策可靠性。实验表明该方法能消除SLA违规并显著降低p99.999延迟，代价是略低的节能效率。
Reason 该论文研究的是6G网络中的代理协商问题，属于通信与网络领域的应用，尽管使用了LLM和数字孪生等技术，但其核心任务与文档图像理解（DIU）、DocVQA、OCR、布局分析、表格识别等完全无关。未涉及任何文档模态处理、视觉文本理解或多模态对齐问题，属于典型的‘平行应用’类工作，且应用场景远离DIU领域。
Abstract A critical barrier to the trustworthiness of sixth-generation (6G) agentic autonomous networks is the uncertainty neglect bias; a cognitive tendency for large language model (LLM)-powered agents to make high-stakes decisions based on simple averages while ignoring the tail risk of extreme events. This paper proposes an unbiased, risk-aware framework for agentic negotiation, designed to ensure robust resource allocation in 6G network slicing. Specifically, agents leverage Digital Twins (DTs) to predict full latency distributions, which are then evaluated using a formal framework from extreme value theory, namely, Conditional Value-at-Risk (CVaR). This approach fundamentally shifts the agent's objective from reasoning over the mean to reasoning over the tail, thereby building a statistically-grounded buffer against worst-case outcomes. Furthermore, our framework ensures full uncertainty awareness by requiring agents to quantify epistemic uncertainty -- confidence in their own DTs predictions -- and propagate this meta-verification to make robust decisions, preventing them from acting on unreliable data. We validate this framework in a 6G inter-slice negotiation use-case between an eMBB and a URLLC agent. The results demonstrate the profound failure of the biased, mean-based baseline, which consistently fails its SLAs with a 25\% rate. Our unbiased, CVaR-aware agent successfully mitigates this bias, eliminating SLA violations and reducing the URLLC and eMBB p99.999 latencies by around 11\%. We show this reliability comes at the rational and quantifiable cost of slightly reduced energy savings to 17\%, exposing the false economy of the biased approach. This work provides a concrete methodology for building the trustworthy autonomous systems required for 6G.
Authors Hatim Chergui, Farhad Rezazadeh, Mehdi Bennis, Merouane Debbah
Comment Link to open-source non-commercial code available
Categories Artificial Intelligence
0
EEG-VLM: A Hierarchical Vision-Language Model with Multi-Level Feature Alignment and Visually Enhanced Language-Guided Reasoning for EEG Image-Based Sleep Stage Prediction
Title CN EEG-VLM：一种具有多级特征对齐和视觉增强语言引导推理的分层视觉-语言模型，用于基于脑电图图像的睡眠阶段预测
Keywords EEG · Vision-Language Model · Sleep Stage Classification · Chain-of-Thought · Medical Image Analysis
Summary 本文提出EEG-VLM，一种结合多级特征对齐和链式思维推理的分层视觉-语言模型，用于提升基于脑电图图像的睡眠阶段分类性能与可解释性。通过视觉增强模块生成高层视觉token，并与CLIP低层特征对齐以增强图像理解能力，同时引入CoT模拟专家推理过程。实验表明该方法在准确性和可解释性方面优于现有方法。
Reason 该论文研究的是基于脑电图（EEG）图像的睡眠阶段分类，属于医疗信号处理与医学图像分析领域，尽管使用了VLM框架，但其输入数据为生理波形转换的图像，与文档图像理解（DIU）中的文本布局、OCR、表格结构等核心问题无关联。任务本质是医学时序信号分类，而非文档内容解析或视觉文档问答（DocVQA），无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU痛点。
Abstract Sleep stage classification based on electroencephalography (EEG) is fundamental for assessing sleep quality and diagnosing sleep-related disorders. However, most traditional machine learning methods rely heavily on prior knowledge and handcrafted features, while existing deep learning models still struggle to jointly capture fine-grained time-frequency patterns and achieve clinical interpretability. Recently, vision-language models (VLMs) have made significant progress in the medical domain, yet their performance remains constrained when applied to physiological waveform data, especially EEG signals, due to their limited visual understanding and insufficient reasoning capability. To address these challenges, we propose EEG-VLM, a hierarchical vision-language framework that integrates multi-level feature alignment with visually enhanced language-guided reasoning for interpretable EEG-based sleep stage classification. Specifically, a specialized visual enhancement module constructs high-level visual tokens from intermediate-layer features to extract rich semantic representations of EEG images. These tokens are further aligned with low-level CLIP features through a multi-level alignment mechanism, enhancing the VLM's image-processing capability. In addition, a Chain-of-Thought (CoT) reasoning strategy decomposes complex medical inference into interpretable logical steps, effectively simulating expert-like decision-making. Experimental results demonstrate that the proposed method significantly improves both the accuracy and interpretability of VLMs in EEG-based sleep stage classification, showing promising potential for automated and explainable EEG analysis in clinical settings.
Authors Xihe Qiu, Gengchen Ma, Haoyu Wang, Chen Zhan, Xiaoyu Tan, Shuo Li
Categories Artificial Intelligence
0
ReefNet: A Large scale, Taxonomically Enriched Dataset and Benchmark for Hard Coral Classification
Title CN ReefNet：一个大规模、分类学增强的硬珊瑚分类数据集和基准
Keywords 珊瑚分类 · 细粒度识别 · 领域泛化 · 大规模数据集 · 计算机视觉
Summary 本文提出了ReefNet，一个大规模、带有专家验证标签的硬珊瑚图像数据集，包含约92.5万个属级别注释，并基于海洋物种世界登记册（WoRMS）进行分类学映射。论文设计了源内和跨源两种评估设置，用于测试模型在不同地理来源下的泛化能力。实验表明现有模型在跨域场景下性能显著下降，零样本模型整体表现较差，尤其对稀有和外观相似的类群。该数据集旨在推动珊瑚礁自动监测与保护技术的发展。
Reason 该论文属于生物多样性监测领域的计算机视觉应用，研究任务为细粒度珊瑚物种分类，与文档图像理解（DIU）或DocVQA无关。尽管其在数据规模和领域泛化方面有贡献，但属于垂直领域的下游应用，未提出可迁移至文档理解的方法论创新。
Abstract Coral reefs are rapidly declining due to anthropogenic pressures such as climate change, underscoring the urgent need for scalable, automated monitoring. We introduce ReefNet, a large public coral reef image dataset with point-label annotations mapped to the World Register of Marine Species (WoRMS). ReefNet aggregates imagery from 76 curated CoralNet sources and an additional site from Al Wajh in the Red Sea, totaling approximately 925000 genus-level hard coral annotations with expert-verified labels. Unlike prior datasets, which are often limited by size, geography, or coarse labels and are not ML-ready, ReefNet offers fine-grained, taxonomically mapped labels at a global scale to WoRMS. We propose two evaluation settings: (i) a within-source benchmark that partitions each source's images for localized evaluation, and (ii) a cross-source benchmark that withholds entire sources to test domain generalization. We analyze both supervised and zero-shot classification performance on ReefNet and find that while supervised within-source performance is promising, supervised performance drops sharply across domains, and performance is low across the board for zero-shot models, especially for rare and visually similar genera. This provides a challenging benchmark intended to catalyze advances in domain generalization and fine-grained coral classification. We will release our dataset, benchmarking code, and pretrained models to advance robust, domain-adaptive, global coral reef monitoring and conservation.
Authors Yahia Battach, Abdulwahab Felemban, Faizan Farooq Khan, Yousef A. Radwan, Xiang Li, Fabio Marchese, Sara Beery, Burton H. Jones, Francesca Benzoni, Mohamed Elhoseiny
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Can LLM-based Financial Investing Strategies Outperform the Market in Long Run? KDD 2026, Datasets & Benchmarks Track
Title CN 基于大语言模型的金融投资策略能否长期跑赢市场？
Keywords LLM应用 · 金融投资 · 回测框架 · 市场 regime 分析 · AI代理
Summary 论文提出FINSABER框架，用于评估基于LLM的时序投资策略在长时间跨度和更大股票范围内的表现。实验表明，现有LLM策略在牛市中过于保守，在熊市中又过于激进，长期表现不佳。研究强调需开发更具市场状态感知能力的风险控制机制。
Reason 该论文属于典型的平行下游应用，将LLM应用于金融投资策略评估，并构建了回测框架FINSABER。尽管涉及金融文本数据，但其核心贡献在于投资策略的评估方法，而非文档图像理解或VLM在文档处理中的技术改进。未提出可用于解决OCR幻觉、复杂排版或文档推理等DIU核心问题的新方法。
Abstract Large Language Models (LLMs) have recently been leveraged for asset pricing tasks and stock trading applications, enabling AI agents to generate investment decisions from unstructured financial data. However, most evaluations of LLM timing-based investing strategies are conducted on narrow timeframes and limited stock universes, overstating effectiveness due to survivorship and data-snooping biases. We critically assess their generalizability and robustness by proposing FINSABER, a backtesting framework evaluating timing-based strategies across longer periods and a larger universe of symbols. Systematic backtests over two decades and 100+ symbols reveal that previously reported LLM advantages deteriorate significantly under broader cross-section and over a longer-term evaluation. Our market regime analysis further demonstrates that LLM strategies are overly conservative in bull markets, underperforming passive benchmarks, and overly aggressive in bear markets, incurring heavy losses. These findings highlight the need to develop LLM strategies that are able to prioritise trend detection and regime-aware risk controls over mere scaling of framework complexity.
Authors Weixian Waylon Li, Hyeonjun Kim, Mihai Cucuringu, Tiejun Ma
Comment Accepted to KDD 2026, Datasets & Benchmarks Track
Categories Artificial Intelligence
0
On the Optimality of Discrete Object Naming: a Kinship Case Study
Title CN 论离散对象命名的最优性：以亲属关系为例
Keywords 信息论 · 命名系统 · 指代博弈 · 贝叶斯解码 · 语言演化
Summary 本文从信息论角度提出一个离散对象命名系统的理论框架，并证明当听者的解码器等同于说话者的贝叶斯解码器时，才能实现最优的命名权衡。通过亲属关系这一语义领域，在涌现通信的指代博弈中验证了该最优性可在学习系统中自然出现。
Reason 该论文研究自然语言命名系统的最优性，基于信息论和指代博弈框架，探讨说话者与听者之间的通信效率。其核心问题为通用语言交流中的命名系统设计，与文档图像理解（DIU）或视觉-语言模型在文档上的应用无直接关联。未涉及OCR、视觉表示、多模态建模、布局分析、表格识别等DIU核心任务，也无法迁移用于解决OCR幻觉、细粒度定位或复杂排版理解等痛点。
Abstract The structure of naming systems in natural languages hinges on a trade-off between high informativeness and low complexity. Prior work capitalizes on information theory to formalize these notions; however, these studies generally rely on two simplifications: (i) optimal listeners, and (ii) universal communicative need across languages. Here, we address these limitations by introducing an information-theoretic framework for discrete object naming systems, and we use it to prove that an optimal trade-off is achievable if and only if the listener's decoder is equivalent to the Bayesian decoder of the speaker. Adopting a referential game setup from emergent communication, and focusing on the semantic domain of kinship, we show that our notion of optimality is not only theoretically achievable but also emerges empirically in learned communication systems.
Authors Phong Le, Mees Lindeman, Raquel G. Alhama
Categories Artificial Intelligence, Computation and Language
0
AI Consciousness and Existential Risk
Title CN 人工智能意识与存在风险
Keywords AI安全 · 人工智能意识 · 存在性风险 · 智能与意识 · AI伦理
Summary 本文探讨了人工智能意识与存在性风险之间的关系，指出意识和智能是两个不同的概念，智能才是预测AI威胁的关键因素，而意识本身并不直接导致存在性风险。文章还分析了意识可能间接影响AI安全的几种情景。
Reason 该论文讨论的是AI意识与人类存在性风险的哲学与理论关系，属于AI安全与伦理领域的宏观思辨，不涉及任何文档图像理解（DIU）或视觉语言模型（VLM）的技术方法。内容与OCR、文档布局分析、表格识别、DocVQA等任务完全无关，也无法为解决OCR幻觉、细粒度定位、复杂排版等DIU核心痛点提供可迁移的技术工具。
Abstract In AI, the existential risk denotes the hypothetical threat posed by an artificial system that would possess both the capability and the objective, either directly or indirectly, to eradicate humanity. This issue is gaining prominence in scientific debate due to recent technical advancements and increased media coverage. In parallel, AI progress has sparked speculation and studies about the potential emergence of artificial consciousness. The two questions, AI consciousness and existential risk, are sometimes conflated, as if the former entailed the latter. Here, I explain that this view stems from a common confusion between consciousness and intelligence. Yet these two properties are empirically and theoretically distinct. Arguably, while intelligence is a direct predictor of an AI system's existential threat, consciousness is not. There are, however, certain incidental scenarios in which consciousness could influence existential risk, in either direction. Consciousness could be viewed as a means towards AI alignment, thereby lowering existential risk; or, it could be a precondition for reaching certain capabilities or levels of intelligence, and thus positively related to existential risk. Recognizing these distinctions can help AI safety researchers and public policymakers focus on the most pressing issues.
Authors Rufin VanRullen
Categories Artificial Intelligence
0
Physics-informed Neural Operator Learning for Nonlinear Grad-Shafranov Equation
Title CN 用于非线性Grad-Shafranov方程的物理信息神经算子学习
Keywords 物理信息神经网络 · 神经算子 · Grad-Shafranov方程 · 科学机器学习 · Transformer-KAN
Summary 本文提出了一种物理信息神经算子（PINO），用于快速求解磁约束核聚变中的非线性Grad-Shafranov方程（GSE）。通过结合监督学习、无监督物理损失和半监督训练策略，模型在保持高精度的同时显著提升了物理一致性，并实现了毫秒级推理速度，适用于实时等离子体控制。
Reason 该论文研究的是核聚变中的物理方程求解问题，属于科学计算与物理模拟领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用完全无关。尽管使用了AI方法（如神经算子、Transformer-KAN），但其目标是解决物理一致性与偏微分方程求解，无法迁移至OCR幻觉、复杂排版理解、细粒度定位等DIU核心痛点。
Abstract As artificial intelligence emerges as a transformative enabler for fusion energy commercialization, fast and accurate solvers become increasingly critical. In magnetic confinement nuclear fusion, rapid and accurate solution of the Grad-Shafranov equation (GSE) is essential for real-time plasma control and analysis. Traditional numerical solvers achieve high precision but are computationally prohibitive, while data-driven surrogates infer quickly but fail to enforce physical laws and generalize poorly beyond training distributions. To address this challenge, we present a Physics-Informed Neural Operator (PINO) that directly learns the GSE solution operator, mapping shape parameters of last closed flux surface to equilibrium solutions for realistic nonlinear current profiles. Comprehensive benchmarking of five neural architectures identifies the novel Transformer-KAN (Kolmogorov-Arnold Network) Neural Operator (TKNO) as achieving highest accuracy (0.25% mean L2 relative error) under supervised training (only data-driven). However, all data-driven models exhibit large physics residuals, indicating poor physical consistency. Our unsupervised training can reduce the residuals by nearly four orders of magnitude through embedding physics-based loss terms without labeled data. Critically, semi-supervised learning--integrating sparse labeled data (100 interior points) with physics constraints--achieves optimal balance: 0.48% interpolation error and the most robust extrapolation performance (4.76% error, 8.9x degradation factor vs 39.8x for supervised models). Accelerated by TensorRT optimization, our models enable millisecond-level inference, establishing PINO as a promising pathway for next-generation fusion control systems.
Authors Siqi Ding, Zitong Zhang, Guoyang Shi, Xingyu Li, Xiang Gu, Yanan Xu, Huasheng Xie, Hanyue Zhao, Yuejiang Shi, Tianyuan Liu
Comment 42 pages, 17 figures, 8 tables,
Categories Artificial Intelligence
0
MoveGPT: Scaling Mobility Foundation Models with Spatially-Aware Mixture of Experts
Title CN MoveGPT：利用空间感知混合专家扩展移动性基础模型
Keywords 移动性基础模型 · 混合专家 · 时空预测 · 位置编码 · 可扩展性
Summary 本文提出MoveGPT，一种面向人类移动性建模的大规模基础模型，通过统一的位置编码器和空间感知的混合专家Transformer结构，在十亿级数据上预训练，提升了城市级移动模式预测性能，并展现出对未见城市的泛化能力。核心贡献集中在时空数据建模，而非文档理解或多模态视觉语言处理。
Reason 该论文研究的是人类移动性建模（human mobility），属于时空序列预测领域，与文档图像理解（DIU）或视觉-语言多模态文档处理无直接关联。尽管其Mixture-of-Experts架构可能在理论上具有通用性，但文中未涉及任何视觉输入、文本布局、OCR、多模态对齐或文档推理任务，迁移到DIU场景的路径不明确。属于无关领域应用。
Abstract The success of foundation models in language has inspired a new wave of general-purpose models for human mobility. However, existing approaches struggle to scale effectively due to two fundamental limitations: a failure to use meaningful basic units to represent movement, and an inability to capture the vast diversity of patterns found in large-scale data. In this work, we develop MoveGPT, a large-scale foundation model specifically architected to overcome these barriers. MoveGPT is built upon two key innovations: (1) a unified location encoder that maps geographically disjoint locations into a shared semantic space, enabling pre-training on a global scale; and (2) a Spatially-Aware Mixture-of-Experts Transformer that develops specialized experts to efficiently capture diverse mobility patterns. Pre-trained on billion-scale datasets, MoveGPT establishes a new state-of-the-art across a wide range of downstream tasks, achieving performance gains of up to 35% on average. It also demonstrates strong generalization capabilities to unseen cities. Crucially, our work provides empirical evidence of scaling ability in human mobility, validating a clear path toward building increasingly capable foundation models in this domain.
Authors Chonghua Han, Yuan Yuan, Jingtao Ding, Jie Feng, Fanjin Meng, Yong Li
Categories Artificial Intelligence
0
The Core in Max-Loss Non-Centroid Clustering Can Be Empty
Title CN 最大损失非质心聚类中的核心可能是空的
Keywords 非质心聚类 · 核心稳定性 · 最大损失 · 博弈论 · 聚类理论
Summary 本文从理论上证明了在最大损失目标下，非质心聚类的核心可能为空，即不存在满足某种稳定条件的聚类结果。作者通过构造反例和计算机辅助证明，在特定度量空间中展示了这一不可能性结果。这是该问题下的首个反例证明。
Reason 该论文研究的是非质心聚类在最大损失目标下的核心稳定性问题，属于理论博弈与聚类分析范畴，与文档图像理解（DIU）、多模态大模型（VLM）或视觉表示学习等方向无直接关联。未涉及OCR、文档布局分析、表格识别、视觉语言模型架构改进或推理机制创新，无法迁移解决DIU中的OCR幻觉、复杂排版、长文档推理等核心痛点。
Abstract We study core stability in non-centroid clustering under the max-loss objective, where each agent's loss is the maximum distance to other members of their cluster. We prove that for all $k\geq 3$ there exist metric instances with $n\ge 9$ agents, with $n$ divisible by $k$, for which no clustering lies in the $α$-core for any $α<2^{\frac{1}{5}}\sim 1.148$. The bound is tight for our construction. Using a computer-aided proof, we also identify a two-dimensional Euclidean point set whose associated lower bound is slightly smaller than that of our general construction. This is, to our knowledge, the first impossibility result showing that the core can be empty in non-centroid clustering under the max-loss objective.
Authors Robert Bredereck, Eva Deltl, Leon Kellerhals, Jannik Peters
Categories Artificial Intelligence
0
Extracting Robust Register Automata from Neural Networks over Data Sequences
Title CN 从数据序列上的神经网络中提取鲁棒的寄存器自动机
Keywords 寄存器自动机 · 神经网络可解释性 · 形式化验证 · 自动机学习 · 鲁棒性分析
Summary 本文提出了一种从处理数据序列的神经网络中提取确定性寄存器自动机（DRA）的框架，用于生成可解释的代理模型，并通过被动与主动学习结合的方式实现对连续域输入序列的建模。该方法可用于评估RNN和Transformer等模型的局部鲁棒性，提供形式化验证保证。
Reason 该论文属于模型可解释性与形式化验证领域，研究如何从黑盒神经网络中提取寄存器自动机以进行符号分析。尽管其方法在理论上有意义，但完全不涉及多模态、视觉输入或文档理解任务，也无法迁移用于解决文档图像理解中的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）。其技术路线与DIU无直接关联，属于无关领域的纯AI理论工作。
Abstract Automata extraction is a method for synthesising interpretable surrogates for black-box neural models that can be analysed symbolically. Existing techniques assume a finite input alphabet, and thus are not directly applicable to data sequences drawn from continuous domains. We address this challenge with deterministic register automata (DRAs), which extend finite automata with registers that store and compare numeric values. Our main contribution is a framework for robust DRA extraction from black-box models: we develop a polynomial-time robustness checker for DRAs with a fixed number of registers, and combine it with passive and active automata learning algorithms. This combination yields surrogate DRAs with statistical robustness and equivalence guarantees. As a key application, we use the extracted automata to assess the robustness of neural networks: for a given sequence and distance metric, the DRA either certifies local robustness or produces a concrete counterexample. Experiments on recurrent neural networks and transformer architectures show that our framework reliably learns accurate automata and enables principled robustness evaluation. Overall, our results demonstrate that robust DRA extraction effectively bridges neural network interpretability and formal reasoning without requiring white-box access to the underlying network.
Authors Chih-Duo Hong, Hongjian Jiang, Anthony W. Lin, Oliver Markgraf, Julian Parsert, Tony Tan
Categories Artificial Intelligence
0
Chat with AI: The Surprising Turn of Real-time Video Communication from Human to AI Proceedings of the 24th ACM Workshop on Hot Topics in Networks (HotNets 2025)
Title CN 与AI聊天：从人类到AI的实时视频通信的意外转折
Keywords AI视频聊天 · 实时通信 · 视频流优化 · 多模态大语言模型 · DeViBench
Summary 本文提出了一种面向AI视频聊天的实时通信新范式，通过上下文感知的视频流技术将比特率集中在对对话重要的区域，以降低延迟。作者构建了DeViBench基准来评估降质视频下的MLLM理解能力。然而，该工作属于视频通信系统优化范畴，与文档图像理解无直接关联。
Reason 该论文聚焦于AI视频通话中的实时通信（RTC）问题，核心是降低视频流带宽以适应多模态大模型的低延迟推理，属于视频理解与网络传输交叉领域。尽管涉及MLLM，但其任务场景为实时视频流压缩与传输优化，并非文档图像理解（DIU）的核心任务如DocVQA、表格识别或布局分析。研究重点在于‘人-AI’视觉交互的工程管道优化，而非解决OCR幻觉、密集文本解析、复杂排版建模等DIU关键挑战。无法迁移用于提升文档理解性能。
Abstract AI Video Chat emerges as a new paradigm for Real-time Communication (RTC), where one peer is not a human, but a Multimodal Large Language Model (MLLM). This makes interaction between humans and AI more intuitive, as if chatting face-to-face with a real person. However, this poses significant challenges to latency, because the MLLM inference takes up most of the response time, leaving very little time for video streaming. Due to network uncertainty, transmission latency becomes a critical bottleneck preventing AI from being like a real person. To address this, we call for AI-oriented RTC research, exploring the network requirement shift from "humans watching video" to "AI understanding video". We begin by recognizing the main differences between AI Video Chat and traditional RTC. Then, through prototype measurements, we identify that ultra-low bitrate is a key factor for low latency. To reduce bitrate dramatically while maintaining MLLM accuracy, we propose Context-Aware Video Streaming that recognizes the importance of each video region for chat and allocates bitrate almost exclusively to chat-important regions. To evaluate the impact of video streaming quality on MLLM accuracy, we build the first benchmark, named Degraded Video Understanding Benchmark (DeViBench). Finally, we discuss some open questions and ongoing solutions for AI Video Chat. DeViBench is open-sourced at: https://github.com/pku-netvideo/DeViBench.
Authors Jiangkai Wu, Zhiyuan Ren, Liming Liu, Xinggong Zhang
Comment 9 pages, 10 figures, Proceedings of the 24th ACM Workshop on Hot Topics in Networks (HotNets 2025), College Park, Maryland, USA
Categories Artificial Intelligence
0
DynaMix: Generalizable Person Re-identification via Dynamic Relabeling and Mixed Data Sampling
Title CN DynaMix：通过动态重标记和混合数据采样的可泛化行人重识别
Keywords 行人重识别 · 伪标签 · 动态重标记 · 数据采样 · 可泛化
Summary 本文提出DynaMix，一种用于可泛化行人重识别的新方法，通过动态重标记模块优化单摄像头数据的伪标签，利用高效中心点模块维护大规模身份空间下的特征表示，并设计数据采样模块构建多样化的混合小批量数据。该方法可在百万级图像和十万级身份上高效训练，在通用Re-ID基准上优于现有方法。
Reason 该论文研究的是行人重识别（Person Re-ID），属于通用计算机视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。尽管其方法涉及伪标签优化、数据采样策略等技术，但这些技术针对的是跨摄像头身份识别问题，难以迁移到解决OCR幻觉、复杂版面解析或长文档推理等DIU核心痛点。属于无关领域应用。
Abstract Generalizable person re-identification (Re-ID) aims to recognize individuals across unseen cameras and environments. While existing methods rely heavily on limited labeled multi-camera data, we propose DynaMix, a novel method that effectively combines manually labeled multi-camera and large-scale pseudo-labeled single-camera data. Unlike prior works, DynaMix dynamically adapts to the structure and noise of the training data through three core components: (1) a Relabeling Module that refines pseudo-labels of single-camera identities on-the-fly; (2) an Efficient Centroids Module that maintains robust identity representations under a large identity space; and (3) a Data Sampling Module that carefully composes mixed data mini-batches to balance learning complexity and intra-batch diversity. All components are specifically designed to operate efficiently at scale, enabling effective training on millions of images and hundreds of thousands of identities. Extensive experiments demonstrate that DynaMix consistently outperforms state-of-the-art methods in generalizable person Re-ID.
Authors Timur Mamedov, Anton Konushin, Vadim Konushin
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Mitigating Participation Imbalance Bias in Asynchronous Federated Learning
Title CN 缓解异步联邦学习中的参与不平衡偏差
Keywords 联邦学习 · 异步更新 · 客户端异质性 · 信息延迟 · 去偏优化
Summary 本文针对异步联邦学习中因客户端更新频率不均导致的模型偏差问题，提出了一种名为ACE的新框架，通过即时使用所有客户端的最新信息来缓解参与不平衡，并设计了延迟感知变体ACED以平衡多样性与信息陈旧性。理论分析和实验验证了方法的有效性。
Reason 该论文研究的是异步联邦学习中的参与不平衡问题，属于分布式机器学习优化范畴，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其方法聚焦于客户端更新机制和延迟感知训练，无法迁移解决OCR幻觉、复杂排版理解、细粒度定位或长文档推理等DIU核心痛点。
Abstract In Asynchronous Federated Learning (AFL), the central server immediately updates the global model with each arriving client's contribution. As a result, clients perform their local training on different model versions, causing information staleness (delay). In federated environments with non-IID local data distributions, this asynchronous pattern amplifies the adverse effect of client heterogeneity (due to different data distribution, local objectives, etc.), as faster clients contribute more frequent updates, biasing the global model. We term this phenomenon heterogeneity amplification. Our work provides a theoretical analysis that maps AFL design choices to their resulting error sources when heterogeneity amplification occurs. Guided by our analysis, we propose ACE (All-Client Engagement AFL), which mitigates participation imbalance through immediate, non-buffered updates that use the latest information available from all clients. We also introduce a delay-aware variant, ACED, to balance client diversity against update staleness. Experiments on different models for different tasks across diverse heterogeneity and delay settings validate our analysis and demonstrate the robust performance of our approaches.
Authors Xiangyu Chang, Manyi Yao, Srikanth V. Krishnamurthy, Christian R. Shelton, Anirban Chakraborty, Ananthram Swami, Samet Oymak, Amit Roy-Chowdhury
Categories Artificial Intelligence
0
Understanding, Accelerating, and Improving MeanFlow Training
Title CN 理解、加速和改进MeanFlow训练
Keywords MeanFlow · 生成模型 · 训练动态 · 扩散模型 · 少步生成
Summary 本文分析了MeanFlow中瞬时速度场与平均速度场之间的训练动态关系，发现两者的学习存在阶段性依赖：良好的瞬时速度学习是平均速度学习的前提，而小时间间隔的平均速度有助于瞬时速度学习，但大间隔则产生负面影响。基于此，作者提出了一种分阶段的训练策略，先加速瞬时速度学习，再逐步转向长间隔平均速度学习，从而实现更快收敛和更高质量的少步生成。在ImageNet上取得了优于基线的FID结果。
Reason 该论文聚焦于生成模型中的MeanFlow训练机制，属于图像生成领域，特别是针对扩散模型的快速生成优化。其研究内容与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。虽然涉及视觉表示学习，但目标是提升生成质量与训练效率，并未解决OCR幻觉、复杂排版理解或长文档推理等DIU痛点，且不属于可迁移至文档理解的通用VLM架构或推理机制创新。
Abstract MeanFlow promises high-quality generative modeling in few steps, by jointly learning instantaneous and average velocity fields. Yet, the underlying training dynamics remain unclear. We analyze the interaction between the two velocities and find: (i) well-established instantaneous velocity is a prerequisite for learning average velocity; (ii) learning of instantaneous velocity benefits from average velocity when the temporal gap is small, but degrades as the gap increases; and (iii) task-affinity analysis indicates that smooth learning of large-gap average velocities, essential for one-step generation, depends on the prior formation of accurate instantaneous and small-gap average velocities. Guided by these observations, we design an effective training scheme that accelerates the formation of instantaneous velocity, then shifts emphasis from short- to long-interval average velocity. Our enhanced MeanFlow training yields faster convergence and significantly better few-step generation: With the same DiT-XL backbone, our method reaches an impressive FID of 2.87 on 1-NFE ImageNet 256x256, compared to 3.43 for the conventional MeanFlow baseline. Alternatively, our method matches the performance of the MeanFlow baseline with 2.5x shorter training time, or with a smaller DiT-L backbone.
Authors Jin-Young Kim, Hyojun Go, Lea Bogensperger, Julius Erbach, Nikolai Kalischek, Federico Tombari, Konrad Schindler, Dominik Narnhofer
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Inferring response times of perceptual decisions with Poisson variational autoencoders NeurIPS 2025 Workshop on Data on the Brain & Mind
Title CN 使用泊松变分自编码器推断感知决策的反应时间
Keywords 泊松变分自编码器 · 感知决策 · 反应时间 · 神经编码 · 贝叶斯解码
Summary 本文提出一种基于泊松变分自编码器的模型，用于从神经脉冲活动中学习视觉刺激表征，并结合贝叶斯解码与熵停止规则来同时预测感知决策的选择和反应时间。模型在MNIST任务上复现了反应时间的统计特性，如右偏分布、Hick定律和速度-准确性权衡。
Reason 该论文研究的是感知决策过程中反应时间的建模，属于认知神经科学与计算神经科学交叉领域，核心是用Poisson VAE模拟神经元放电和决策过程。其任务设定（如MNIST分类中的反应时间预测）与文档图像理解（DIU）的核心问题（如文本识别、布局分析、表格理解、VQA等）无直接关联。尽管涉及视觉输入和神经编码，但方法目标并非解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU痛点，也无法迁移为提升VLM在文档理解中的性能的工具。属于无关领域。
Abstract Many properties of perceptual decision making are well-modeled by deep neural networks. However, such architectures typically treat decisions as instantaneous readouts, overlooking the temporal dynamics of the decision process. We present an image-computable model of perceptual decision making in which choices and response times arise from efficient sensory encoding and Bayesian decoding of neural spiking activity. We use a Poisson variational autoencoder to learn unsupervised representations of visual stimuli in a population of rate-coded neurons, modeled as independent homogeneous Poisson processes. A task-optimized decoder then continually infers an approximate posterior over actions conditioned on incoming spiking activity. Combining these components with an entropy-based stopping rule yields a principled and image-computable model of perceptual decisions capable of generating trial-by-trial patterns of choices and response times. Applied to MNIST digit classification, the model reproduces key empirical signatures of perceptual decision making, including stochastic variability, right-skewed response time distributions, logarithmic scaling of response times with the number of alternatives (Hick's law), and speed-accuracy trade-offs.
Authors Hayden R. Johnson, Anastasia N. Krouglova, Hadi Vafaii, Jacob L. Yates, Pedro J. Gonçalves
Comment To appear at the NeurIPS 2025 Workshop on Data on the Brain \& Mind
Categories Artificial Intelligence
0
Large Language Model-Assisted Planning of Electric Vehicle Charging Infrastructure with Real-World Case Study
Title CN 基于大语言模型辅助规划电动汽车充电基础设施的实证研究
Keywords 电动汽车 · 充电基础设施 · 大语言模型 · 优化建模 · ADMM
Summary 本文提出了一种结合投资决策与充电分配的电动汽车充电基础设施规划方法，利用大语言模型辅助生成数学模型，并采用ADMM分布式算法求解大规模问题。通过成都真实出行数据验证，相比无分配策略的基线降低了30%总成本。
Reason 该论文属于将LLM用于特定领域（交通基础设施规划）的下游应用，本质上是‘平行应用’，并未提出任何针对文档图像理解（DIU）或视觉语言模型在文档处理中的方法论创新。尽管使用了LLM，但其核心贡献在于优化建模与算法设计，与文档理解中的OCR幻觉、复杂排版、细粒度定位等关键问题无关。
Abstract The growing demand for electric vehicle (EV) charging infrastructure presents significant planning challenges, requiring efficient strategies for investment and operation to deliver cost-effective charging services. However, the potential benefits of EV charging assignment, particularly in response to varying spatial-temporal patterns of charging demand, remain under-explored in infrastructure planning. This paper proposes an integrated approach that jointly optimizes investment decisions and charging assignments while accounting for spatial-temporal demand dynamics and their interdependencies. To support efficient model development, we leverage a large language model (LLM) to assist in generating and refining the mathematical formulation from structured natural-language descriptions, significantly reducing the modeling burden. The resulting optimization model enables optimal joint decision-making for investment and operation. Additionally, we propose a distributed optimization algorithm based on the Alternating Direction Method of Multipliers (ADMM) to address computational complexity in high-dimensional scenarios, which can be executed on standard computing platforms. We validate our approach through a case study using 1.5 million real-world travel records from Chengdu, China, demonstrating a 30% reduction in total cost compared to a baseline without EV assignment.
Authors Xinda Zheng, Canchen Jiang, Hao Wang
Categories Artificial Intelligence
0
MedSAM3: Delving into Segment Anything with Medical Concepts
Title CN MedSAM3：深入探索基于医学概念的Segment Anything
Keywords 医学图像分割 · Segment Anything · 多模态大模型 · 提示学习 · Agent
Summary 本文提出MedSAM3，一种基于SAM3架构并结合医学语义概念标签训练的可提示医学图像分割模型，支持通过开放词汇文本描述实现解剖结构的精准定位。同时构建MedSAM3 Agent框架，利用多模态大语言模型实现复杂推理与迭代优化，在多种医学影像模态上表现优异。
Reason 该论文聚焦于医学图像分割任务，属于纯计算机视觉在医疗领域的垂直应用。其核心是将SAM架构结合医学语义标签进行细粒度解剖结构分割，并引入MLLM作为Agent进行推理优化。尽管使用了多模态大模型，但整体为特定领域（医学影像）的‘平行下游应用’，未提出可迁移至文档图像理解（如OCR去幻觉、复杂排版建模、长文档推理等）的底层方法创新。
Abstract Medical image segmentation is fundamental for biomedical discovery. Existing methods lack generalizability and demand extensive, time-consuming manual annotation for new clinical application. Here, we propose MedSAM-3, a text promptable medical segmentation model for medical image and video segmentation. By fine-tuning the Segment Anything Model (SAM) 3 architecture on medical images paired with semantic conceptual labels, our MedSAM-3 enables medical Promptable Concept Segmentation (PCS), allowing precise targeting of anatomical structures via open-vocabulary text descriptions rather than solely geometric prompts. We further introduce the MedSAM-3 Agent, a framework that integrates Multimodal Large Language Models (MLLMs) to perform complex reasoning and iterative refinement in an agent-in-the-loop workflow. Comprehensive experiments across diverse medical imaging modalities, including X-ray, MRI, Ultrasound, CT, and video, demonstrate that our approach significantly outperforms existing specialist and foundation models. We will release our code and model at https://github.com/Joey-S-Liu/MedSAM3.
Authors Anglin Liu, Rundong Xue, Xu R. Cao, Yifan Shen, Yi Lu, Xiang Li, Qianqian Chen, Jintai Chen
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
CSD: Change Semantic Detection with only Semantic Change Masks for Damage Assessment in Conflict Zones
Title CN 仅使用语义变化掩码的变更语义检测（CSD）用于冲突地区的损害评估
Keywords 变更语义检测 · 遥感图像 · DINOv3 · 损害评估 · 变化检测
Summary 本文提出了一种名为变更语义检测（CSD）的新任务，旨在通过仅标注变化区域的语义掩码来简化冲突地区建筑物损毁评估的标注负担。作者设计了MC-DiSNet网络，并利用DINOv3作为骨干网络提取双时相卫星图像特征，在自建的Gaza-Change数据集上验证了方法有效性。该工作专注于遥感场景下的变化检测，不涉及文本内容理解或文档结构建模。
Reason 该论文属于遥感图像变化检测领域，研究任务为基于卫星图像的损害评估，核心方法针对的是二时相遥感影像中的语义变化检测（CSD），与文档图像理解（DIU）在数据模态、任务目标和应用场景上均无直接关联。尽管其采用了DINOv3等视觉模型，但未涉及文本识别、布局分析、多模态文档问答等DIU核心问题，也无法迁移解决OCR幻觉、复杂排版或长文档推理等痛点。
Abstract Accurately and swiftly assessing damage from conflicts is crucial for humanitarian aid and regional stability. In conflict zones, damaged zones often share similar architectural styles, with damage typically covering small areas and exhibiting blurred boundaries. These characteristics lead to limited data, annotation difficulties, and significant recognition challenges, including high intra-class similarity and ambiguous semantic changes. To address these issues, we introduce a pre-trained DINOv3 model and propose a multi-scale cross-attention difference siamese network (MC-DiSNet). The powerful visual representation capability of the DINOv3 backbone enables robust and rich feature extraction from bi-temporal remote sensing images. We also release a new Gaza-change dataset containing high-resolution satellite image pairs from 2023-2024 with pixel-level semantic change annotations. It is worth emphasizing that our annotations only include semantic pixels of changed areas. Unlike conventional semantic change detection (SCD), our approach eliminates the need for large-scale semantic annotations of bi-temporal images, instead focusing directly on the changed regions. We term this new task change semantic detection (CSD). The CSD task represents a direct extension of binary change detection (BCD). Due to the limited spatial extent of semantic regions, it presents greater challenges than traditional SCD tasks. We evaluated our method under the CSD framework on both the Gaza-Change and SECOND datasets. Experimental results demonstrate that our proposed approach effectively addresses the CSD task, and its outstanding performance paves the way for practical applications in rapid damage assessment across conflict zones.
Authors Kai Zhenga, Zhenkai Wu, Fupeng Wei, Miaolan Zhou, Kai Lie, Haitao Guo, Lei Ding, Wei Zhang, Hang-Cheng Dong
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Life-IQA: Boosting Blind Image Quality Assessment through GCN-enhanced Layer Interaction and MoE-based Feature Decoupling
Title CN Life-IQA：通过GCN增强的层间交互和基于MoE的特征解耦提升盲图像质量评估
Keywords 盲图像质量评估 · 图卷积网络 · MoE · 特征解耦 · 跨注意力
Summary 本文提出了一种名为Life-IQA的盲图像质量评估框架，通过GCN增强的层间交互模块实现深层与次深层特征的跨注意力交互，并设计了基于MoE的特征解耦模块，使不同专家网络专注于不同类型失真或质量维度。实验表明该方法在多个BIQA基准上取得了优于传统Transformer解码器的性能与成本平衡。
Reason 该论文研究的是盲图像质量评估（BIQA），属于通用图像质量评价领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的GCN增强交互和MoE特征解耦机制具有一定方法价值，但难以迁移解决文档理解中的OCR幻觉、复杂排版或长文档推理等关键痛点。
Abstract Blind image quality assessment (BIQA) plays a crucial role in evaluating and optimizing visual experience. Most existing BIQA approaches fuse shallow and deep features extracted from backbone networks, while overlooking the unequal contributions to quality prediction. Moreover, while various vision encoder backbones are widely adopted in BIQA, the effective quality decoding architectures remain underexplored. To address these limitations, this paper investigates the contributions of shallow and deep features to BIQA, and proposes a effective quality feature decoding framework via GCN-enhanced \underline{l}ayer\underline{i}nteraction and MoE-based \underline{f}eature d\underline{e}coupling, termed \textbf{(Life-IQA)}. Specifically, the GCN-enhanced layer interaction module utilizes the GCN-enhanced deepest-layer features as query and the penultimate-layer features as key, value, then performs cross-attention to achieve feature interaction. Moreover, a MoE-based feature decoupling module is proposed to decouple fused representations though different experts specialized for specific distortion types or quality dimensions. Extensive experiments demonstrate that Life-IQA shows more favorable balance between accuracy and cost than a vanilla Transformer decoder and achieves state-of-the-art performance on multiple BIQA benchmarks.The code is available at: \href{https://github.com/TANGLONG2/Life-IQA/tree/main}{\texttt{Life-IQA}}.
Authors Long Tang, Guoquan Zhen, Jie Hao, Jianbo Zhang, Huiyu Duan, Liang Yuan, Guangtao Zhai
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
PRAGMA: A Profiling-Reasoned Multi-Agent Framework for Automatic Kernel Optimization
Title CN PRAGMA：一种用于自动内核优化的剖析推理多智能体框架
Keywords 多智能体系统 · 内核优化 · 大语言模型 · 性能剖析 · AI编译
Summary 本文提出了PRAGMA，一个结合执行反馈与细粒度硬件剖析的AI内核生成框架，通过让大语言模型识别性能瓶颈并迭代优化代码，在CPU和GPU平台上均显著优于基线方法。然而，该工作属于AI在系统优化领域的下游应用，未涉及任何文档理解或多模态视觉-语言建模。
Reason 该论文属于典型的‘平行应用’——将LLM或多智能体系统应用于特定领域（高性能计算中的内核优化），其核心贡献是面向AI生成代码与性能调优的应用流程，而非文档图像理解（DIU）相关的技术方法。尽管使用了多智能体和反馈机制，但这些机制针对的是程序性能瓶颈分析，无法直接迁移解决DIU中的OCR幻觉、复杂排版解析或长文档推理等核心问题。
Abstract Designing high-performance kernels requires expert-level tuning and a deep understanding of hardware characteristics. Recent advances in large language models (LLMs) have enabled automated kernel generation, yet most existing systems rely solely on correctness or execution time feedback, lacking the ability to reason about low-level performance bottlenecks. In this paper, we introduce PRAGMA, a profile-guided AI kernel generation framework that integrates execution feedback and fine-grained hardware profiling into the reasoning loop. PRAGMA enables LLMs to identify performance bottlenecks, preserve historical best versions, and iteratively refine code quality. We evaluate PRAGMA on KernelBench, covering GPU and CPU backends. Results show that PRAGMA consistently outperforms baseline AIKG without profiling enabled and achieves 2.81$\times$ and 2.30$\times$ averaged speedups against Torch on CPU and GPU platforms, respectively.
Authors Kelun Lei, Hailong Yang, Huaitao Zhang, Xin You, Kaige Zhang, Zhongzhi Luan, Yi Liu, Depei Qian
Categories Artificial Intelligence
0
Differentiated Directional Intervention A Framework for Evading LLM Safety Alignment AAAI-26-AIA
Title CN 差异化方向干预：一种规避大语言模型安全对齐的框架
Keywords 安全对齐 · 越狱攻击 · 方向干预 · 激活空间 · 大语言模型
Summary 本文提出将大语言模型的安全拒绝机制分解为‘伤害检测方向’和‘拒绝执行方向’两个神经表示，并设计了一种双方向干预框架DBDI，在激活空间中分别进行投影归零和方向引导，以高效绕过安全对齐机制。实验显示其在Llama-2等模型上攻击成功率高达97.88%。
Reason 该论文研究的是大语言模型的安全对齐机制及其绕过方法，属于纯NLP领域的安全/对抗攻击（Jailbreak）研究。尽管涉及‘干预’（Intervention）技术，但其目标是破解安全策略，而非解决文档图像理解中的OCR幻觉、细粒度定位或复杂排版等核心问题。内容与文档理解无直接关联，且属于负面清单中明确指出应拒绝的‘纯NLP的安全/对齐（Safety/Jailbreak）’范畴。
Abstract Safety alignment instills in Large Language Models (LLMs) a critical capacity to refuse malicious requests. Prior works have modeled this refusal mechanism as a single linear direction in the activation space. We posit that this is an oversimplification that conflates two functionally distinct neural processes: the detection of harm and the execution of a refusal. In this work, we deconstruct this single representation into a Harm Detection Direction and a Refusal Execution Direction. Leveraging this fine-grained model, we introduce Differentiated Bi-Directional Intervention (DBDI), a new white-box framework that precisely neutralizes the safety alignment at critical layer. DBDI applies adaptive projection nullification to the refusal execution direction while suppressing the harm detection direction via direct steering. Extensive experiments demonstrate that DBDI outperforms prominent jailbreaking methods, achieving up to a 97.88\% attack success rate on models such as Llama-2. By providing a more granular and mechanistic framework, our work offers a new direction for the in-depth understanding of LLM safety alignment.
Authors Peng Zhang, Peijie Sun
Comment AAAI-26-AIA
Categories Artificial Intelligence
0
Introducing Visual Scenes and Reasoning: A More Realistic Benchmark for Spoken Language Understanding
Title CN 引入视觉场景与推理：一种更贴近现实的口语理解基准
Keywords 口语理解 · 视觉场景 · 显式推理 · 多模态数据集 · 上下文感知
Summary 本文提出VRSLU数据集，通过引入用户环境的视觉图像和显式推理链来改进口语理解任务，利用GPT-4o生成视觉场景和推理解释，并设计两步指令模板LR-Instruct进行标签预测与推理生成。目标是增强现实场景下的上下文感知与可解释性。
Reason 该论文聚焦于口语理解（SLU）任务，引入视觉图像和推理过程以提升意图检测与槽位填充，属于多模态NLP在语音领域的应用。尽管使用了视觉信息，但其核心任务与文档图像理解（DIU）无关，未涉及OCR、布局分析、表格识别、文档VQA等核心问题，也无法迁移解决DIU中的OCR幻觉、密集文本处理或复杂排版理解等痛点。属于平行应用类工作，且视觉模态是作为辅助上下文引入，并非文档内容解析。
Abstract Spoken Language Understanding (SLU) consists of two sub-tasks: intent detection (ID) and slot filling (SF). Given its broad range of real-world applications, enhancing SLU for practical deployment is increasingly critical. Profile-based SLU addresses ambiguous user utterances by incorporating context awareness (CA), user profiles (UP), and knowledge graphs (KG) to support disambiguation, thereby advancing SLU research toward real-world applicability. However, existing SLU datasets still fall short in representing real-world scenarios. Specifically, (1) CA uses one-hot vectors for representation, which is overly idealized, and (2) models typically focuses solely on predicting intents and slot labels, neglecting the reasoning process that could enhance performance and interpretability. To overcome these limitations, we introduce VRSLU, a novel SLU dataset that integrates both Visual images and explicit Reasoning. For over-idealized CA, we use GPT-4o and FLUX.1-dev to generate images reflecting users' environments and statuses, followed by human verification to ensure quality. For reasoning, GPT-4o is employed to generate explanations for predicted labels, which are then refined by human annotators to ensure accuracy and coherence. Additionally, we propose an instructional template, LR-Instruct, which first predicts labels and then generates corresponding reasoning. This two-step approach helps mitigate the influence of reasoning bias on label prediction. Experimental results confirm the effectiveness of incorporating visual information and highlight the promise of explicit reasoning in advancing SLU.
Authors Di Wu, Liting Jiang, Ruiyu Fang, Bianjing, Hongyan Xie, Haoxiang Su, Hao Huang, Zhongjiang He, Shuangyong Song, Xuelong Li
Categories Artificial Intelligence
0
Enhancing low energy reconstruction and classification in KM3NeT/ORCA with transformers
Title CN 利用Transformer增强KM3NeT/ORCA中低能量重建与分类
Keywords Transformer · 中微子重建 · 注意力掩码 · 高能物理 · 迁移学习
Summary 本文研究在KM3NeT/ORCA中微子望远镜中使用Transformer模型进行低能量中微子事件的重建与分类，通过引入基于物理和探测器设计的注意力掩码，使模型更好地理解探测器结构和物理过程，并验证了模型在不同配置间迁移的有效性。
Reason 该论文属于高能物理领域的应用，研究内容为中微子望远镜中的信号重建，尽管使用了Transformer和注意力机制，但其方法针对粒子探测器物理建模，与文档图像理解（DIU）或视觉语言模型在文档任务中的应用无直接关联。未涉及OCR、文档布局分析、表格识别、VIE等DIU核心任务，也无法提供可迁移至文档理解场景的通用方法（如视觉压缩、高分辨率处理、去幻觉机制等）。
Abstract The current KM3NeT/ORCA neutrino telescope, still under construction, has not yet reached its full potential in neutrino reconstruction capability. When training any deep learning model, no explicit information about the physics or the detector is provided, thus they remain unknown to the model. This study leverages the strengths of transformers by incorporating attention masks inspired by the physics and detector design, making the model understand both the telescope design and the neutrino physics measured on it. The study also shows the efficacy of transformers on retaining valuable information between detectors when doing fine-tuning from one configurations to another.
Authors Iván Mozún Mateo
Categories Artificial Intelligence
0
Warm Chat: Diffuse Emotion-aware Interactive Talking Head Avatar with Tree-Structured Guidance
Title CN Warm Chat：基于树结构引导的情绪感知交互式说话头像
Keywords 说话头像 · 情绪感知 · 对话生成 · 树结构建模 · Transformer
Summary 本文提出Warm Chat，一种支持双向交互的情绪感知说话头像生成框架。利用LLM（如GPT-4）生成对话内容，并通过Transformer结构在潜在掩码空间中生成时序一致的头部运动序列。引入树形对话结构来建模对话状态转移，结合反向层级遍历提取历史情绪线索以指导表情合成。实验表明该方法在情感表达连贯性和交互自然性方面表现优异。
Reason 该论文研究的是对话式虚拟头像生成，属于具身智能与人脸动画领域，核心问题为情绪感知的头像动作合成与双向对话控制。其技术路线聚焦于Transformer-based掩码生成与树结构对话状态建模，与文档图像理解（DIU）中的OCR、布局分析、表格识别或视觉语言推理无直接关联。尽管使用了LLM进行对话生成，但未涉及文档内容理解、多模态对齐、视觉压缩或细粒度定位等DIU核心挑战，无法迁移用于解决文档幻觉、复杂排版或长文档推理等问题。
Abstract Generative models have advanced rapidly, enabling impressive talking head generation that brings AI to life. However, most existing methods focus solely on one-way portrait animation. Even the few that support bidirectional conversational interactions lack precise emotion-adaptive capabilities, significantly limiting their practical applicability. In this paper, we propose Warm Chat, a novel emotion-aware talking head generation framework for dyadic interactions. Leveraging the dialogue generation capability of large language models (LLMs, e.g., GPT-4), our method produces temporally consistent virtual avatars with rich emotional variations that seamlessly transition between speaking and listening states. Specifically, we design a Transformer-based head mask generator that learns temporally consistent motion features in a latent mask space, capable of generating arbitrary-length, temporally consistent mask sequences to constrain head motions. Furthermore, we introduce an interactive talking tree structure to represent dialogue state transitions, where each tree node contains information such as child/parent/sibling nodes and the current character's emotional state. By performing reverse-level traversal, we extract rich historical emotional cues from the current node to guide expression synthesis. Extensive experiments demonstrate the superior performance and effectiveness of our method.
Authors Haijie Yang, Zhenyu Zhang, Hao Tang, Jianjun Qian, Jian Yang
Comment The submission is withdrawn at the request of the authors due to internal reasons within the research team
Categories Artificial Intelligence
0
Classification EM-PCA for clustering and embedding IEEE Big Data Conference (Special Session on Machine Learning)
Title CN 用于聚类和嵌入的分类EM-PCA
Keywords 聚类 · EM算法 · PCA · 维度约简 · 混合模型
Summary 本文提出了一种结合主成分分析（PCA）与分类期望最大化（CEM）算法的方法，用于同时实现数据嵌入与聚类。作者通过非顺序方式联合优化这两个任务，并在实验中展示了该方法在聚类质量和嵌入效果上的优势。此外，还探讨了该方法与其他聚类方法之间的联系。
Reason 该论文属于传统的机器学习方法研究，聚焦于连续数据的聚类与降维问题，使用的是经典的PCA和CEM算法。其研究内容与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联，既不涉及多模态建模、视觉表示学习，也不解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。无法迁移为提升文档理解性能的有效工具。
Abstract The mixture model is undoubtedly one of the greatest contributions to clustering. For continuous data, Gaussian models are often used and the Expectation-Maximization (EM) algorithm is particularly suitable for estimating parameters from which clustering is inferred. If these models are particularly popular in various domains including image clustering, they however suffer from the dimensionality and also from the slowness of convergence of the EM algorithm. However, the Classification EM (CEM) algorithm, a classifying version, offers a fast convergence solution while dimensionality reduction still remains a challenge. Thus we propose in this paper an algorithm combining simultaneously and non-sequentially the two tasks --Data embedding and Clustering-- relying on Principal Component Analysis (PCA) and CEM. We demonstrate the interest of such approach in terms of clustering and data embedding. We also establish different connections with other clustering approaches.
Authors Zineddine Tighidet, Lazhar Labiod, Mohamed Nadif
Comment Accepted at the IEEE conference on Big Data (Special Session on Machine Learning)
Categories Artificial Intelligence, Computation and Language
0
Rethinking Plant Disease Diagnosis: Bridging the Academic-Practical Gap with Vision Transformers and Zero-Shot Learning
Title CN 重新思考植物病害诊断：使用视觉Transformer和零样本学习弥合学术与实践之间的差距
Keywords 植物病害诊断 · Vision Transformer · 零样本学习 · 域适应 · CLIP
Summary 本文探讨了在植物病害诊断中，如何利用Vision Transformer和基于CLIP的零样本学习方法来提升模型在真实农田环境中的泛化能力。实验表明，相较于传统CNN，ViT具有更强的跨域适应性，而CLIP模型可通过自然语言描述实现无需微调的疾病分类，具备良好的可解释性和实用性。
Reason 该论文属于农业图像分析领域，研究任务为植物病害分类，使用自然场景下的叶片图像进行零样本学习和域适应。其核心问题、数据模态（自然图像）与文档图像理解（DIU）的典型挑战（如OCR幻觉、复杂版式、细粒度文本定位等）无直接关联。尽管使用了CLIP和Vision Transformer等通用视觉模型，但未涉及文档结构、多模态对齐、文本-布局联合建模等DIU关键问题，也无法迁移用于解决文档理解中的核心痛点。
Abstract Recent advances in deep learning have enabled significant progress in plant disease classification using leaf images. Much of the existing research in this field has relied on the PlantVillage dataset, which consists of well-centered plant images captured against uniform, uncluttered backgrounds. Although models trained on this dataset achieve high accuracy, they often fail to generalize to real-world field images, such as those submitted by farmers to plant diagnostic systems. This has created a significant gap between published studies and practical application requirements, highlighting the necessity of investigating and addressing this issue. In this study, we investigate whether attention-based architectures and zero-shot learning approaches can bridge the gap between curated academic datasets and real-world agricultural conditions in plant disease classification. We evaluate three model categories: Convolutional Neural Networks (CNNs), Vision Transformers, and Contrastive Language-Image Pre-training (CLIP)-based zero-shot models. While CNNs exhibit limited robustness under domain shift, Vision Transformers demonstrate stronger generalization by capturing global contextual features. Most notably, CLIP models classify diseases directly from natural language descriptions without any task-specific training, offering strong adaptability and interpretability. These findings highlight the potential of zero-shot learning as a practical and scalable domain adaptation strategy for plant health diagnosis in diverse field environments.
Authors Wassim Benabbas, Mohammed Brahimi, Samir Akhrouf, Bilal Fortas
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Teacher Encoder-Student Decoder Denoising Guided Segmentation Network for Anomaly Detection
Title CN 基于教师编码器-学生解码器去噪引导的异常检测分割网络
Keywords 异常检测 · 学生-教师框架 · 特征融合 · 自监督学习 · 分割网络
Summary 本文提出了一种名为PFADSeg的异常检测模型，采用教师编码器-学生解码器结构，通过去噪机制和多尺度特征融合提升学生网络的学习能力，并引入自监督分割网络自动生成异常掩码。在MVTec AD数据集上取得了优异的图像级和像素级检测性能。
Reason 该论文研究的是工业图像中的视觉异常检测（Visual Anomaly Detection），属于通用计算机视觉任务，主要面向MVTec AD等工业缺陷检测数据集。其核心方法是改进学生-教师框架下的特征学习与融合机制，用于生成异常掩码。尽管使用了自监督和特征蒸馏技术，但其目标与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联，也不涉及OCR、多模态语言模型、文本密集排版处理或长文档推理等关键痛点。无法迁移用于解决文档理解中的幻觉、细粒度定位或复杂逻辑推理问题。
Abstract Visual anomaly detection is a highly challenging task, often categorized as a one-class classification and segmentation problem. Recent studies have demonstrated that the student-teacher (S-T) framework effectively addresses this challenge. However, most S-T frameworks rely solely on pre-trained teacher networks to guide student networks in learning multi-scale similar features, overlooking the potential of the student networks to enhance learning through multi-scale feature fusion. In this study, we propose a novel model named PFADSeg, which integrates a pre-trained teacher network, a denoising student network with multi-scale feature fusion, and a guided anomaly segmentation network into a unified framework. By adopting a unique teacher-encoder and student-decoder denoising mode, the model improves the student network's ability to learn from teacher network features. Furthermore, an adaptive feature fusion mechanism is introduced to train a self-supervised segmentation network that synthesizes anomaly masks autonomously, significantly increasing detection performance. Rigorous evaluations on the widely-used MVTec AD dataset demonstrate that PFADSeg exhibits excellent performance, achieving an image-level AUC of 98.9%, a pixel-level mean precision of 76.4%, and an instance-level mean precision of 78.7%.
Authors Shixuan Song, Hao Chen, Shu Hu, Xin Wang, Jinrong Hu, Xi Wu
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
MOCLIP: A Foundation Model for Large-Scale Nanophotonic Inverse Design
Title CN MOCLIP：一种用于大规模纳米光子逆向设计的基础模型
Keywords 基础模型 · 纳米光子学 · 逆向设计 · 对比学习 · 光学存储
Summary 本文提出了MOCLIP，一种用于纳米光子学的基础模型，通过对比学习将超表面几何结构与光谱响应映射到共享隐空间，实现高速零样本逆向设计和高密度光学存储。实验数据显示其在芯片级设计和数据存储密度上具有显著优势。
Reason 该论文属于纳米光子学领域，研究的是基于对比学习的光子超表面几何结构与光谱表征的对齐模型，应用于逆向设计和光学信息存储。完全不涉及文档图像理解（DIU）、多模态语言模型（VLM）在文档上的应用、OCR、布局分析或任何相关任务。属于典型无关领域（物理/光子设计），无法迁移至DIU核心问题。
Abstract Foundation models (FM) are transforming artificial intelligence by enabling generalizable, data-efficient solutions across different domains for a broad range of applications. However, the lack of large and diverse datasets limits the development of FM in nanophotonics. This work presents MOCLIP (Metasurface Optics Contrastive Learning Pretrained), a nanophotonic foundation model that integrates metasurface geometry and spectra within a shared latent space. MOCLIP employs contrastive learning to align geometry and spectral representations using an experimentally acquired dataset with a sample density comparable to ImageNet-1K. The study demonstrates MOCLIP inverse design capabilities for high-throughput zero-shot prediction at a rate of 0.2 million samples per second, enabling the design of a full 4-inch wafer populated with high-density metasurfaces in minutes. It also shows generative latent-space optimization reaching 97 percent accuracy. Finally, we introduce an optical information storage concept that uses MOCLIP to achieve a density of 0.1 Gbit per square millimeter at the resolution limit, exceeding commercial optical media by a factor of six. These results position MOCLIP as a scalable and versatile platform for next-generation photonic design and data-driven applications.
Authors S. Rodionov, A. Burguete-Lopez, M. Makarenko, Q. Wang, F. Getman, A. Fratalocchi
Categories Artificial Intelligence
0
M2R2: MultiModal Robotic Representation for Temporal Action Segmentation
Title CN M2R2：用于时序动作分割的多模态机器人表征
Keywords 时序动作分割 · 多模态学习 · 机器人 · 特征复用 · 预训练
Summary 本文提出M2R2，一种面向时序动作分割的多模态特征提取器，结合机器人本体感知与视觉信息，并设计新的预训练策略以提升特征复用性。在REASSEMBLE数据集上取得SOTA性能，显著优于现有方法。
Reason 该论文研究的是机器人领域的时序动作分割（TAS），依赖于本体感知（proprioceptive）和外部传感器（如摄像头）的多模态融合，应用场景为机器人操作任务。属于具身智能/机器人领域，与文档图像理解（DIU）或DocVQA无直接关联。方法聚焦于动作边界检测和机器人技能学习，无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心问题。
Abstract Temporal action segmentation (TAS) has long been a key area of research in both robotics and computer vision. In robotics, algorithms have primarily focused on leveraging proprioceptive information to determine skill boundaries, with recent approaches in surgical robotics incorporating vision. In contrast, computer vision typically relies on exteroceptive sensors, such as cameras. Existing multimodal TAS models in robotics integrate feature fusion within the model, making it difficult to reuse learned features across different models. Meanwhile, pretrained vision-only feature extractors commonly used in computer vision struggle in scenarios with limited object visibility. In this work, we address these challenges by proposing M2R2, a multimodal feature extractor tailored for TAS, which combines information from both proprioceptive and exteroceptive sensors. We introduce a novel pretraining strategy that enables the reuse of learned features across multiple TAS models. Our method achieves state-of-the-art performance on the REASSEMBLE dataset, a challenging multimodal robotic assembly dataset, outperforming existing robotic action segmentation models by 46.6%. Additionally, we conduct an extensive ablation study to evaluate the contribution of different modalities in robotic TAS tasks.
Authors Daniel Sliwowski, Dongheui Lee
Comment 8 pages, 6 figures, 2 tables
Categories Artificial Intelligence
0
LLM-CSEC: Empirical Evaluation of Security in C/C++ Code Generated by Large Language Models
Title CN LLM生成C/C++代码的安全性实证评估
Keywords 代码安全 · 大语言模型 · CWE · 静态分析 · 漏洞检测
Summary 本文评估了十种大语言模型在生成C/C++代码时引入的安全漏洞情况，利用CWE和CVE对漏洞进行分类和严重性分析，发现AI生成代码中存在大量安全隐患，提醒开发者需谨慎使用LLM生成的代码。
Reason 该论文研究的是大语言模型生成代码中的安全漏洞问题，属于软件安全与编程语言领域，与文档图像理解（DIU）、视觉语言模型（VLM）或OCR等核心技术无关。未涉及任何多模态、视觉输入、布局分析或文档推理内容，完全不满足DIU本题或关联工具迁移的要求。
Abstract The security of code generated by large language models (LLMs) is a significant concern, as studies indicate that such code often contains vulnerabilities and lacks essential defensive programming constructs. This work focuses on examining and evaluating the security of LLM-generated code, particularly in the context of C/C++. We categorized known vulnerabilities using the Common Weakness Enumeration (CWE) and, to study their criticality, mapped them to CVEs. We used ten different LLMs for code generation and analyzed the outputs through static analysis. The amount of CWEs present in AI-generated code is concerning. Our findings highlight the need for developers to be cautious when using LLM-generated code. This study provides valuable insights to advance automated code generation and encourage further research in this domain.
Authors Muhammad Usman Shahid, Chuadhry Mujeeb Ahmed, Rajiv Ranjan
Categories Artificial Intelligence
0
OMGSR: You Only Need One Mid-timestep Guidance for Real-World Image Super-Resolution
Title CN OMGSR：你只需要一次中间时间步引导用于真实世界图像超分辨率
Keywords 图像超分辨率 · 扩散模型 · Latent表示优化 · LoRA · 图像恢复
Summary 本文提出OMGSR，一种基于DDPM的单步真实世界图像超分辨率框架，通过在中间时间步注入低质量图像潜在表示，并引入LRR损失和Dv3CD损失来提升重建质量。虽然技术上有一定创新，但应用场景为通用图像增强，与文档图像理解无直接关联。
Reason 该论文聚焦于通用图像超分辨率任务，属于图像恢复（image restoration）领域，尽管使用了DDPM和LoRA等技术，但其方法设计面向自然图像而非文档图像。未涉及文档理解中的核心问题如OCR幻觉、布局分析、文本识别或VLM多模态对齐。无法直接迁移用于解决文档图像理解中的密集文本、复杂排版或长文档推理等痛点，属于无关领域。
Abstract Denoising Diffusion Probabilistic Models (DDPMs) show promising potential in one-step Real-World Image Super-Resolution (Real-ISR). Current one-step Real-ISR methods typically inject the low-quality (LQ) image latent representation at the start or end timestep of the DDPM scheduler. Recent studies have begun to note that the LQ image latent and the pre-trained noisy latent representations are intuitively closer at a mid-timestep. However, a quantitative analysis of these latent representations remains lacking. Considering these latent representations can be decomposed into signal and noise, we propose a method based on the Signal-to-Noise Ratio (SNR) to pre-compute an average optimal mid-timestep for injection. To better approximate the pre-trained noisy latent representation, we further introduce the Latent Representation Refinement (LRR) loss via a LoRA-enhanced VAE encoder. We also fine-tune the backbone of the DDPM-based generative model using LoRA to perform one-step denoising at the average optimal mid-timestep. Based on these components, we present OMGSR, a GAN-based Real-ISR framework that employs a DDPM-based generative model as the generator and a DINOv3-ConvNeXt model with multi-level discriminator heads as the discriminator. We also propose the DINOv3-ConvNeXt DISTS (Dv3CD) loss, which is enhanced for structural perception at varying resolutions. Within the OMGSR framework, we develop OMGSR-S based on SD2.1-base. An ablation study confirms that our pre-computation strategy and LRR loss significantly improve the baseline. Comparative studies demonstrate that OMGSR-S achieves state-of-the-art performance across multiple metrics. Code is available at \hyperlink{Github}{https://github.com/wuer5/OMGSR}.
Authors Zhiqiang Wu, Zhaomang Sun, Tong Zhou, Bingtao Fu, Ji Cong, Yitong Dong, Huaqi Zhang, Xuan Tang, Mingsong Chen, Xian Wei
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Developing an Algorithm Selector for Green Configuration in Scheduling Problems
Title CN 开发用于调度问题中绿色配置的算法选择器
Keywords 作业车间调度 · 算法选择 · XGBoost · 绿色调度 · 优化
Summary 本文提出了一种基于机器学习的算法选择框架，用于在作业车间调度问题中根据问题特征选择最优求解器（如GUROBI、CPLEX、GECODE），以平衡能源效率与生产效率。实验表明该选择器准确率达到84.51%。
Reason 该论文属于运筹优化与调度领域，研究的是作业车间调度问题（JSP）中的算法选择，使用XGBoost进行算法推荐。其任务、数据和方法均与文档图像理解（DIU）或视觉语言模型（VLM）无关，不涉及OCR、文档布局分析、表格识别或任何多模态文档理解任务，也无法迁移解决DIU中的核心痛点如OCR幻觉、复杂排版或长文档推理。
Abstract The Job Shop Scheduling Problem (JSP) is central to operations research, primarily optimizing energy efficiency due to its profound environmental and economic implications. Efficient scheduling enhances production metrics and mitigates energy consumption, thus effectively balancing productivity and sustainability objectives. Given the intricate and diverse nature of JSP instances, along with the array of algorithms developed to tackle these challenges, an intelligent algorithm selection tool becomes paramount. This paper introduces a framework designed to identify key problem features that characterize its complexity and guide the selection of suitable algorithms. Leveraging machine learning techniques, particularly XGBoost, the framework recommends optimal solvers such as GUROBI, CPLEX, and GECODE for efficient JSP scheduling. GUROBI excels with smaller instances, while GECODE demonstrates robust scalability for complex scenarios. The proposed algorithm selector achieves an accuracy of 84.51\% in recommending the best algorithm for solving new JSP instances, highlighting its efficacy in algorithm selection. By refining feature extraction methodologies, the framework aims to broaden its applicability across diverse JSP scenarios, thereby advancing efficiency and sustainability in manufacturing logistics.
Authors Carlos March, Christian Perez, Miguel A. Salido
Categories Artificial Intelligence
0
Autonomous Vehicle Path Planning by Searching With Differentiable Simulation
Title CN 基于可微分仿真搜索的自动驾驶路径规划
Keywords 自动驾驶 · 路径规划 · 可微分仿真 · 梯度优化 · 动作序列搜索
Summary 本文提出了一种名为Differentiable Simulation for Search（DSS）的框架，利用可微分仿真器Waymax作为状态预测器和批评网络，通过梯度下降优化未来轨迹的动作序列，在自动驾驶路径规划中取得了优于传统方法的效果。
Reason 该论文属于自动驾驶领域，研究的是车辆在交通环境中的路径规划问题，使用了可微分仿真进行动作序列搜索。尽管涉及‘规划’和‘搜索’等通用方法，但其核心任务与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。方法迁移至DIU场景（如复杂排版理解、长文档推理）的可能性极低，属于无关领域。
Abstract Planning allows an agent to safely refine its actions before executing them in the real world. In autonomous driving, this is crucial to avoid collisions and navigate in complex, dense traffic scenarios. One way to plan is to search for the best action sequence. However, this is challenging when all necessary components - policy, next-state predictor, and critic - have to be learned. Here we propose Differentiable Simulation for Search (DSS), a framework that leverages the differentiable simulator Waymax as both a next state predictor and a critic. It relies on the simulator's hardcoded dynamics, making state predictions highly accurate, while utilizing the simulator's differentiability to effectively search across action sequences. Our DSS agent optimizes its actions using gradient descent over imagined future trajectories. We show experimentally that DSS - the combination of planning gradients and stochastic search - significantly improves tracking and path planning accuracy compared to sequence prediction, imitation learning, model-free RL, and other planning methods.
Authors Asen Nachkov, Jan-Nico Zaech, Danda Pani Paudel, Xi Wang, Luc Van Gool
Categories Artificial Intelligence
0
GRAPHIC--Guidelines for Reviewing Algorithmic Practices in Human-centred Design and Interaction for Creativity
Title CN GRAPHIC——面向创意领域人机协同设计中算法实践的评审指南
Keywords 人机协作 · 创意设计 · 算法实践 · 评审框架 · 图形设计
Summary 本文提出了一种名为GRAPHIC的评审框架，用于分析应用于平面设计领域的计算系统，旨在理解当前系统如何支持人与AI在平面设计中的协作。该框架包含三个主要维度：协作全景、过程与模式、平面设计原则。通过文献综述发现现有研究在代理间主动权平衡、可解释交互模型以及基于设计原则的变革性创造力支持方面存在不足。
Reason 该论文属于AI在创意设计领域的人因交互研究，聚焦图形设计中的人-AI协作框架与设计原则，不涉及文档图像理解（DIU）或视觉语言模型（VLM）的技术方法创新。其内容为跨学科评审框架，无OCR、DocVQA、布局分析等DIU核心任务的技术改进，也无法迁移为解决OCR幻觉、密集文本识别、复杂排版解析等底层技术工具。属于无关领域的应用性综述，不符合本方向关注的技术源头创新要求。
Abstract Artificial Intelligence (AI) has been increasingly applied to creative domains, leading to the development of systems that collaborate with humans in design processes. In Graphic Design, integrating computational systems into co-creative workflows presents specific challenges, as it requires balancing scientific rigour with the subjective and visual nature of design practice. Following the PRISMA methodology, we identified 872 articles, resulting in a final corpus of 71 publications describing 68 unique systems. Based on this review, we introduce GRAPHIC (Guidelines for Reviewing Algorithmic Practices in Human-centred Design and Interaction for Creativity), a framework for analysing computational systems applied to Graphic Design. Its goal is to understand how current systems support human-AI collaboration in the Graphic Design discipline. The framework comprises main dimensions, which our analysis revealed to be essential across diverse system types: (1) Collaborative Panorama, (2) Processes and Modalities, and (3) Graphic Design Principles. Its application revealed research gaps, including the need to balance initiative and control between agents, improve communication through explainable interaction models, and promote systems that support transformational creativity grounded in core design principles.
Authors Joana Rovira Martins, Pedro Martins, Ana Boavida
Comment 20 pages, 16 figures
Categories Artificial Intelligence
0
Defending Large Language Models Against Jailbreak Exploits with Responsible AI Considerations
Title CN 通过负责任的人工智能考虑防御大语言模型的越狱攻击
Keywords 大语言模型安全 · 越狱防御 · Logit引导 · MetaGPT · 负责任AI
Summary 本文系统分类了针对大语言模型越狱攻击的现有防御方法，并提出了三种防御策略：基于提示层的防御框架、基于logit的推理时引导防御以及基于MetaGPT的领域特定代理防御。实验表明这些方法能有效降低攻击成功率，尤其在代理防御下可实现完全缓解。然而，这些技术主要服务于模型安全性，不涉及多模态文档理解或视觉信息处理。
Reason 该论文聚焦于LLM的安全与对齐问题，特别是防御越狱攻击（jailbreak），属于纯NLP安全领域。尽管提到了干预机制（如logit steering和Agent框架），但其目标场景与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）无直接关联。方法也无法迁移用于解决视觉或文档模态中的幻觉或结构理解问题，属于负面清单中的‘纯NLP安全/对齐’类别。
Abstract Large Language Models (LLMs) remain susceptible to jailbreak exploits that bypass safety filters and induce harmful or unethical behavior. This work presents a systematic taxonomy of existing jailbreak defenses across prompt-level, model-level, and training-time interventions, followed by three proposed defense strategies. First, a Prompt-Level Defense Framework detects and neutralizes adversarial inputs through sanitization, paraphrasing, and adaptive system guarding. Second, a Logit-Based Steering Defense reinforces refusal behavior through inference-time vector steering in safety-sensitive layers. Third, a Domain-Specific Agent Defense employs the MetaGPT framework to enforce structured, role-based collaboration and domain adherence. Experiments on benchmark datasets show substantial reductions in attack success rate, achieving full mitigation under the agent-based defense. Overall, this study highlights how jailbreaks pose a significant security threat to LLMs and identifies key intervention points for prevention, while noting that defense strategies often involve trade-offs between safety, performance, and scalability. Code is available at: https://github.com/Kuro0911/CS5446-Project
Authors Ryan Wong, Hosea David Yu Fei Ng, Dhananjai Sharma, Glenn Jun Jie Ng, Kavishvaran Srinivasan
Comment 20 pages including appendix; technical report; NeurIPS 2024 style
Categories Artificial Intelligence
0
Learning Solution Operators for Partial Differential Equations via Monte Carlo-Type Approximation NeurIPS 2025 Workshop on Machine Learning and the Physical Sciences
Title CN 通过蒙特卡洛型逼近学习偏微分方程的解算子
Keywords 神经算子 · 偏微分方程 · 蒙特卡洛方法 · 深度学习 · 科学计算
Summary 本文提出了一种名为MCNO的轻量级神经网络架构，用于学习参数化偏微分方程的解算子。该方法采用蒙特卡洛方式直接逼近核积分，不依赖傅里叶变换或平移不变性假设，能够在不同网格分辨率间泛化。实验表明其在1D PDE基准上具有竞争力的精度和较低计算成本。
Reason 该论文研究的是物理科学与偏微分方程（PDE）求解中的神经算子方法，属于科学计算与数值模拟领域，与文档图像理解（DIU）或视觉语言模型（VLM）在任务、数据和方法论上均无直接关联。其提出的方法MCNO针对PDE核积分的近似，难以迁移到OCR幻觉、复杂排版理解、细粒度视觉定位等DIU核心痛点。属于无关领域。
Abstract The Monte Carlo-type Neural Operator (MCNO) introduces a lightweight architecture for learning solution operators for parametric PDEs by directly approximating the kernel integral using a Monte Carlo approach. Unlike Fourier Neural Operators, MCNO makes no spectral or translation-invariance assumptions. The kernel is represented as a learnable tensor over a fixed set of randomly sampled points. This design enables generalization across multiple grid resolutions without relying on fixed global basis functions or repeated sampling during training. Experiments on standard 1D PDE benchmarks show that MCNO achieves competitive accuracy with low computational cost, providing a simple and practical alternative to spectral and graph-based neural operators.
Authors Salah Eddine Choutri, Prajwal Chauhan, Othmane Mazhar, Saif Eddin Jabari
Comment NeurIPS 2025 Workshop on Machine Learning and the Physical Sciences
Categories Artificial Intelligence
0
MoodBench 1.0: An Evaluation Benchmark for Emotional Companionship Dialogue Systems
Title CN MoodBench 1.0：情感陪伴对话系统评估基准
Keywords 情感对话系统 · 评估基准 · 大语言模型 · 人机交互 · 对话AI
Summary 本文提出了首个针对情感陪伴对话系统（ECDs）的评估基准MoodBench 1.0，旨在衡量大模型在提供个性化情感支持方面的能力。通过能力层-任务层-数据层-方法层的设计框架，构建了包含30个主流模型的评测体系，并揭示了当前模型在深层情感陪伴方面的不足。
Reason 该论文聚焦于情感陪伴型对话系统的评测，属于纯NLP应用领域，且任务为构建对话评估基准，与文档图像理解（DIU）、视觉-语言模型在文档上的应用、OCR、布局分析等完全无关。未涉及任何多模态输入、视觉内容处理或文档结构建模，无法迁移解决DIU中的核心痛点如幻觉、复杂排版或长文档推理。
Abstract With the rapid development of Large Language Models, dialogue systems are shifting from information tools to emotional companions, heralding the era of Emotional Companionship Dialogue Systems (ECDs) that provide personalized emotional support for users. However, the field lacks clear definitions and systematic evaluation standards for ECDs. To address this, we first propose a definition of ECDs with formal descriptions. Then, based on this theory and the design principle of "Ability Layer-Task Layer (three level)-Data Layer-Method Layer", we design and implement the first ECD evaluation benchmark - MoodBench 1.0. Through extensive evaluations of 30 mainstream models, we demonstrate that MoodBench 1.0 has excellent discriminant validity and can effectively quantify the differences in emotional companionship abilities among models. Furthermore, the results reveal current models' shortcomings in deep emotional companionship, guiding future technological optimization and significantly aiding developers in enhancing ECDs' user experience.
Authors Haifeng Jing, Yujie Hou, Junfei Liu, Rui Xie, alan Xu, Jinlong Ma, Qichun Deng
Comment 26 pages, 7 figures
Categories Artificial Intelligence
0
LLM-Driven Kernel Evolution: Automating Driver Updates in Linux
Title CN 由大语言模型驱动的内核演进：自动化Linux驱动更新
Keywords LLM · Linux内核 · 驱动维护 · 多智能体 · 自动化修复
Summary 本文提出了AUTODRIVER系统和DRIVEBENCH数据集，旨在通过大语言模型自动化Linux驱动程序在内核升级过程中的适配与修复。系统结合提示工程、多智能体协作、静态分析和迭代验证来生成语义一致且功能正确的补丁。实验表明该方法在55个案例中实现了56.4%的编译成功率，并能在QEMU中成功启动驱动。作者开源了相关工具和数据集以推动可持续的驱动-内核协同演进研究。
Reason 该论文属于操作系统与软件维护领域的应用，尽管使用了LLM和多智能体协作技术，但其核心是解决Linux驱动与内核的协同演化问题，属于典型的‘平行下游应用’。未提出可用于文档图像理解（如OCR去幻觉、复杂排版处理、视觉压缩等）的底层方法或可迁移技术，对DIU领域无直接方法论贡献。
Abstract Linux kernel evolution breaks drivers through API/ABI changes, semantic shifts, and security-hardening updates. We introduce DRIVEBENCH, an executable corpus of kernel$\rightarrow$driver co-evolution cases, and AUTODRIVER, a closed-loop, LLM-driven system for automating driver maintenance. The system integrates prompt engineering, multi-agent collaboration, static analysis, and iterative validation to ensure that generated patches are not only syntactically correct but also functionally and semantically consistent with kernel conventions. The corpus spans v5.10-v6.10 with 235 validated cases drawn from 612 candidates. In evaluation across 55 cases, AUTODRIVER achieves 56.4% compilation success; QEMU-based boot verification indicates that compiled patches preserve driver initialization in most instances. By releasing DRIVEBENCH and tooling, we enable reproducible research and a practical route to continuous, safe co-evolution of drivers with the Linux kernel.
Authors Arina Kharlamova, Jiawen Liu, Tianyi Zhang, Xinrui Yang, Humaid Alqasimi, Youcheng Sun, Chun Jason Xue
Categories Artificial Intelligence
0
Future-Back Threat Modeling: A Foresight-Driven Security Framework
Title CN 面向未来的威胁建模：一种前瞻性驱动的安全框架
Keywords 威胁建模 · 前瞻性安全 · 网络安全 · 未知威胁 · 防御架构
Summary 本文提出了一种名为‘面向未来威胁建模’（FBTM）的安全框架，通过设想未来的威胁状态并逆向分析当前防御体系中的盲点和脆弱性，以提升对未知攻击的预测能力。该方法强调应对人工智能、信息战和供应链攻击等新兴威胁，但完全脱离文档理解或多模态AI的技术路径。
Reason 该论文属于网络安全领域的理论框架研究，与文档图像理解（DIU）或视觉语言模型（VLM）技术无直接关联。其核心内容为基于未来场景反向推导安全漏洞的威胁建模方法，未涉及OCR、文档布局分析、表格识别、多模态推理等DIU任务，也无法迁移用于解决OCR幻觉、密集文本识别、复杂排版理解或长文档推理等关键问题。
Abstract Traditional threat modeling remains reactive-focused on known TTPs and past incident data, while threat prediction and forecasting frameworks are often disconnected from operational or architectural artifacts. This creates a fundamental weakness: the most serious cyber threats often do not arise from what is known, but from what is assumed, overlooked, or not yet conceived, and frequently originate from the future, such as artificial intelligence, information warfare, and supply chain attacks, where adversaries continuously develop new exploits that can bypass defenses built on current knowledge. To address this mental gap, this paper introduces the theory and methodology of Future-Back Threat Modeling (FBTM). This predictive approach begins with envisioned future threat states and works backward to identify assumptions, gaps, blind spots, and vulnerabilities in the current defense architecture, providing a clearer and more accurate view of impending threats so that we can anticipate their emergence and shape the future we want through actions taken now. The proposed methodology further aims to reveal known unknowns and unknown unknowns, including tactics, techniques, and procedures that are emerging, anticipated, and plausible. This enhances the predictability of adversary behavior, particularly under future uncertainty, helping security leaders make informed decisions today that shape more resilient security postures for the future.
Authors Vu Van Than
Categories Artificial Intelligence
0
MetaDCSeg: Robust Medical Image Segmentation via Meta Dynamic Center Weighting
Title CN 基于元动态中心加权的鲁棒医学图像分割
Keywords 医学图像分割 · 噪声标签 · 元学习 · 边界不确定性 · 动态加权
Summary 本文提出MetaDCSeg，一种通过动态学习像素级权重来抑制噪声标签影响的医学图像分割框架。该方法引入动态中心距离（DCD）机制建模边界不确定性，并利用加权特征距离优化前景、背景和边界的表示，从而提升在噪声标注下的分割稳定性。实验表明其在多个带噪医学数据集上优于现有方法。
Reason 该论文研究的是医学图像分割中的噪声标签问题，属于纯医学图像分析领域，且关注的是分割任务中的鲁棒性优化，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。虽然其‘动态加权’思想看似有方法论价值，但其核心针对的是生物医学影像中的模糊边界和标注噪声，迁移到文档图像中的OCR幻觉、复杂排版或长文本推理等问题极为困难。同时，该工作属于特定领域的下游应用（医疗），符合负面清单中的‘无关领域’类别。
Abstract Medical image segmentation is crucial for clinical applications, but it is frequently disrupted by noisy annotations and ambiguous anatomical boundaries, which lead to instability in model training. Existing methods typically rely on global noise assumptions or confidence-based sample selection, which inadequately mitigate the performance degradation caused by annotation noise, especially in challenging boundary regions. To address this issue, we propose MetaDCSeg, a robust framework that dynamically learns optimal pixel-wise weights to suppress the influence of noisy ground-truth labels while preserving reliable annotations. By explicitly modeling boundary uncertainty through a Dynamic Center Distance (DCD) mechanism, our approach utilizes weighted feature distances for foreground, background, and boundary centers, directing the model's attention toward hard-to-segment pixels near ambiguous boundaries. This strategy enables more precise handling of structural boundaries, which are often overlooked by existing methods, and significantly enhances segmentation performance. Extensive experiments across four benchmark datasets with varying noise levels demonstrate that MetaDCSeg consistently outperforms existing state-of-the-art methods.
Authors Chenyu Mu, Guihai Chen, Xun Yang, Erkun Yang, Cheng Deng
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Accelerating Reinforcement Learning via Error-Related Human Brain Signals
Title CN 通过与错误相关的人脑信号加速强化学习
Keywords 强化学习 · 脑机接口 · EEG · 奖励塑造 · 机器人操作
Summary 本文探索了如何利用从脑电信号中解码出的错误相关电位（ErrP）来加速高维机器人操作任务中的强化学习。通过将EEG信号用于奖励塑造，并在7自由度机械臂的避障到达任务中进行实验，结果表明这种神经反馈能够显著加快学习速度，并在某些情况下超越稀疏奖励基线。此外，跨被试验证显示该框架具有鲁棒性。然而，该研究属于具身智能与脑机接口范畴，与文档理解无关。
Reason 该论文研究利用脑电图（EEG）信号作为隐式反馈来加速机器人操作任务中的强化学习，属于具身智能与神经接口领域。其核心问题、方法与文档图像理解（DIU）无直接关联，既不涉及多模态语言模型对文档的解析，也不解决OCR幻觉、复杂版面分析或长文档推理等DIU核心痛点。尽管‘人类反馈’在广义上可视为一种干预机制，但该工作聚焦于机器人控制中的奖励塑形，迁移到文档理解场景极其困难。
Abstract In this work, we investigate how implicit neural feed back can accelerate reinforcement learning in complex robotic manipulation settings. While prior electroencephalogram (EEG) guided reinforcement learning studies have primarily focused on navigation or low-dimensional locomotion tasks, we aim to understand whether such neural evaluative signals can improve policy learning in high-dimensional manipulation tasks involving obstacles and precise end-effector control. We integrate error related potentials decoded from offline-trained EEG classifiers into reward shaping and systematically evaluate the impact of human-feedback weighting. Experiments on a 7-DoF manipulator in an obstacle-rich reaching environment show that neural feedback accelerates reinforcement learning and, depending on the human-feedback weighting, can yield task success rates that at times exceed those of sparse-reward baselines. Moreover, when applying the best-performing feedback weighting across all sub jects, we observe consistent acceleration of reinforcement learning relative to the sparse-reward setting. Furthermore, leave-one subject-out evaluations confirm that the proposed framework remains robust despite the intrinsic inter-individual variability in EEG decodability. Our findings demonstrate that EEG-based reinforcement learning can scale beyond locomotion tasks and provide a viable pathway for human-aligned manipulation skill acquisition.
Authors Suzie Kim, Hye-Bin Shin, Hyo-Jeong Jang
Categories Artificial Intelligence
0
GContextFormer: A global context-aware hybrid multi-head attention approach with scaled additive aggregation for multimodal trajectory prediction
Title CN GContextFormer：一种具有缩放加法聚合的全局上下文感知混合多头注意力方法用于多模态轨迹预测
Keywords 轨迹预测 · 多模态学习 · Transformer · 注意力机制 · 自动驾驶
Summary 本文提出GContextFormer，一种无需依赖高精地图的多模态轨迹预测模型，通过全局上下文感知的混合注意力机制和缩放加法聚合来提升车辆运动意图对齐性。模型在TOD-VT数据集上验证有效，尤其在高曲率和过渡区域表现更优。然而该工作属于自动驾驶行为预测范畴，与文档理解无直接关联。
Reason 该论文研究的是自动驾驶场景下的多模态轨迹预测，属于具身智能与交通预测领域，尽管使用了多模态建模和注意力机制改进，但其核心任务与文档图像理解（DIU）无关。未涉及DocVQA、OCR、版面分析、表格识别等任何文档理解任务，也无法迁移其方法解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract Multimodal trajectory prediction generates multiple plausible future trajectories to address vehicle motion uncertainty from intention ambiguity and execution variability. However, HD map-dependent models suffer from costly data acquisition, delayed updates, and vulnerability to corrupted inputs, causing prediction failures. Map-free approaches lack global context, with pairwise attention over-amplifying straight patterns while suppressing transitional patterns, resulting in motion-intention misalignment. This paper proposes GContextFormer, a plug-and-play encoder-decoder architecture with global context-aware hybrid attention and scaled additive aggregation achieving intention-aligned multimodal prediction without map reliance. The Motion-Aware Encoder builds scene-level intention prior via bounded scaled additive aggregation over mode-embedded trajectory tokens and refines per-mode representations under shared global context, mitigating inter-mode suppression and promoting intention alignment. The Hierarchical Interaction Decoder decomposes social reasoning into dual-pathway cross-attention: a standard pathway ensures uniform geometric coverage over agent-mode pairs while a neighbor-context-enhanced pathway emphasizes salient interactions, with gating module mediating their contributions to maintain coverage-focus balance. Experiments on eight highway-ramp scenarios from TOD-VT dataset show GContextFormer outperforms state-of-the-art baselines. Compared to existing transformer models, GContextFormer achieves greater robustness and concentrated improvements in high-curvature and transition zones via spatial distributions. Interpretability is achieved through motion mode distinctions and neighbor context modulation exposing reasoning attribution. The modular architecture supports extensibility toward cross-domain multimodal reasoning tasks. Source: https://fenghy-chen.github.io/sources/.
Authors Yuzhi Chen, Yuanchang Xie, Lei Zhao, Pan Liu, Yajie Zou, Chen Wang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Gradient Propagation in Retrosynthetic Space: An Efficient Framework for Synthesis Plan Generation
Title CN 在逆合成空间中的梯度传播：一种高效的合成路线生成框架
Keywords 逆合成 · 梯度传播 · 搜索算法 · 化学空间 · 合成路径
Summary 本文提出了一种基于梯度传播的算法框架，用于在逆合成任务中高效探索可能的合成路径。通过计算不同节点对目标分子合成成功概率的贡献，并优先扩展贡献最高的节点，实现对化学空间的有效搜索。实验表明该方法具有广泛的适用性和更高的计算效率。
Reason 该论文属于化学信息学领域的逆合成路径规划，研究内容与文档图像理解（DIU）或视觉-语言模型（VLM）无直接关联。尽管其提出的梯度传播框架用于搜索优化，但方法针对分子结构树的特定领域问题设计，无法迁移至DIU中的OCR幻觉、复杂排版解析或长文档推理等核心痛点。
Abstract Retrosynthesis, which aims to identify viable synthetic pathways for target molecules by decomposing them into simpler precursors, is often treated as a search problem. However, its complexity arises from multi-branched tree-structured pathways rather than linear paths. Some algorithms have been successfully applied in this task, but they either overlook the uncertainties inherent in chemical space or face limitations in practical application scenarios. To address these challenges, this paper introduces a novel gradient-propagation-based algorithmic framework for retrosynthetic route exploration. The proposed framework obtains the contributions of different nodes to the target molecule's success probability through gradient propagation and then guides the algorithm to greedily select the node with the highest contribution for expansion, thereby conducting efficient search in the chemical space. Experimental validations demonstrate that our algorithm achieves broad applicability across diverse molecular targets and exhibits superior computational efficiency compared to existing methods.
Authors Chengyang Tian, Yuhang Chang, Yangpeng Zhang, Yang Liu
Categories Artificial Intelligence
0
Periodic Asynchrony: An Effective Method for Accelerating On-Policy Reinforcement Learning
Title CN 周期性异步：一种加速On-Policy强化学习的有效方法
Keywords 强化学习 · 训练效率 · 异步架构 · GRPO · NPU
Summary 本文提出了一种周期性异步框架，通过分离强化学习中的推理与训练部署，并改进数据加载器，实现各组件的独立弹性扩展，在保证算法精度的同时显著提升了训练效率。实验表明在NPU平台上整体性能提升至少三倍。
Reason 该论文属于强化学习系统优化领域，研究的是RL训练中推理与训练组件的部署架构改进，不涉及多模态、视觉或文档理解任务。尽管提到了计算效率提升，但其方法无法直接迁移用于解决文档图像理解中的OCR幻觉、细粒度定位、复杂排版或长文档推理等核心问题。
Abstract Since the introduction of the GRPO algorithm, reinforcement learning (RL) has attracted increasing attention, with growing efforts to reproduce and apply it. However, training efficiency remains a critical challenge. In mainstream RL frameworks, inference and training are typically deployed on the same devices. While this approach reduces costs through resource consolidation, its synchronous execution imposes a computational coupling that prevents concurrent inference and training. In this study, we are returning to the strategy of separating inference and training deployment, and by introducing improvements in the data loader, we transform the conventional synchronous architecture into a periodically asynchronous framework, which allows for demand-driven, independent, and elastic scaling of each component, while the accuracy of the algorithm remains completely equivalent to the synchronization method, with both belonging to the on-policy strategy. It is worth emphasizing that we apply a unified tri-model architecture in the training phase, and we also proposed a shared-prompt attention mask to reduce repetitive computation. In practice, these works have achieved at least a threefold overall performance improvement in RL training on NPU platforms, indicating its potential for widespread application.
Authors Jian Lu
Categories Artificial Intelligence
0
Multidimensional Music Aesthetic Evaluation via Semantically Consistent C-Mixup Augmentation
Title CN 基于语义一致的C-Mixup增强的多维音乐美学评价
Keywords 音乐美学评价 · 多尺度特征提取 · 音频增强 · 混合损失 · 生成音乐评估
Summary 本文提出了一种用于评估生成歌曲美学质量的框架，通过多源多尺度特征提取、分层音频增强策略以及结合回归与排序损失的混合训练目标，在ICASSP 2026 SongEval基准上实现了优于基线方法的表现。
Reason 该论文研究的是音乐生成内容的美学质量评估，属于音频信号处理与音乐信息检索领域，与文档图像理解（DIU）或视觉-语言模型在文档上的应用无直接关联。其方法聚焦于多尺度音频特征提取和数据增强策略，难以迁移到解决OCR幻觉、复杂排版理解、细粒度视觉定位等DIU核心问题。
Abstract Evaluating the aesthetic quality of generated songs is challenging due to the multi-dimensional nature of musical perception. We propose a robust music aesthetic evaluation framework that combines (1) multi-source multi-scale feature extraction to obtain complementary segment- and track-level representations, (2) a hierarchical audio augmentation strategy to enrich training data, and (3) a hybrid training objective that integrates regression and ranking losses for accurate scoring and reliable top-song identification. Experiments on the ICASSP 2026 SongEval benchmark demonstrate that our approach consistently outperforms baseline methods across correlation and top-tier metrics.
Authors Shuyang Liu, Yuan Jin, Rui Lin, Shizhe Chen, Junyu Dai, Tao Jiang
Categories Artificial Intelligence
0
KernelBand: Boosting LLM-based Kernel Optimization with a Hierarchical and Hardware-aware Multi-armed Bandit
Title CN KernelBand：通过分层且硬件感知的多臂老虎机机制提升基于大语言模型的内核优化
Keywords LLM-based code generation · kernel optimization · multi-armed bandit · hardware-aware · system optimization
Summary 本文提出KernelBand，一种将内核优化建模为分层多臂老虎机问题的框架，利用LLM代理结合硬件分析信息来指导内核选择与优化策略的生成，从而在更少尝试中找到高性能的内核实现。重点在于系统级性能优化，而非语义或视觉理解。
Reason 该论文属于系统优化与编译领域，核心是利用LLM进行GPU内核代码的自动优化，目标是提升计算效率和减少推理开销。尽管使用了LLM代理和决策框架，但其应用场景完全脱离文档图像理解任务，不涉及任何多模态输入、视觉表示、文本定位或文档结构建模。属于典型的‘平行应用’类技术包装，且方向与DIU无关。
Abstract High quality kernels are critical for reducing training and inference costs of Large Language Models (LLMs), yet they traditionally require significant expertise in hardware architecture and software optimization. While recent advances in LLM-based code generation show promise for complex optimization, existing methods struggle with the vast optimization space due to insufficient hardware domain knowledge, failing to effectively balance exploration and exploitation. We present KernelBand, a novel framework that formulates kernel optimization as a hierarchical multi-armed bandit problem, enabling LLM agents to strategically navigate the optimization space by treating kernel selection and optimization strategy application as sequential decision-making processes. Our approach leverages hardware profiling information to identify promising optimization strategies and employs runtime behavior clustering to reduce exploration overhead across kernel candidates. Extensive experiments on TritonBench demonstrate that KernelBand significantly outperforms state-of-the-art methods, achieving superior performance with fewer tokens while exhibiting consistent improvement without saturation as computational resources increase.
Authors Dezhi Ran, Shuxiao Xie, Mingfang Ji, Ziyue Hua, Mengzhou Wu, Yuan Cao, Yuzhe Guo, Yu Hao, Linyi Li, Yitao Hu, Tao Xie
Comment Work in progress
Categories Artificial Intelligence
0
Preprint: Exploring Inevitable Waypoints for Unsolvability Explanation in Hybrid Planning Problems
Title CN 探索混合规划问题中不可解性解释的必然路径点
Keywords 可解释AI · 规划不可解性 · 路径点识别 · 符号可达性 · 动态规划
Summary 本文提出一种通过识别‘必然路径点’来解释混合规划问题不可解性的方法，将路径点检测建模为最长公共子序列问题，并利用符号可达性分析找出最早不可达的关键点作为解释依据。实验在混合系统的不可解规划问题上进行验证。
Reason 该论文属于具身智能与符号规划领域，研究如何通过识别‘必然路径点’来解释规划问题的不可解性。其核心方法基于动态规划和符号可达性分析，应用于AI规划中的可解释性问题。然而，该工作与文档图像理解（DIU）或视觉-语言模型（VLM）无直接关联：既不涉及OCR、布局分析、表格识别等DIU任务，也不提供可用于解决OCR幻觉、密集文本处理或复杂排版理解的底层技术。同时，其方法难以迁移到文档理解场景，因缺乏对视觉输入、多模态对齐或高分辨率图像处理的支持。属于无关领域研究。
Abstract Explaining unsolvability of planning problems is of significant research interest in Explainable AI Planning. AI planning literature has reported several research efforts on generating explanations of solutions to planning problems. However, explaining the unsolvability of planning problems remains a largely open and understudied problem. A widely practiced approach to plan generation and automated problem solving, in general, is to decompose tasks into sub-problems that help progressively converge towards the goal. In this paper, we propose to adopt the same philosophy of sub-problem identification as a mechanism for analyzing and explaining unsolvability of planning problems in hybrid systems. In particular, for a given unsolvable planning problem, we propose to identify common waypoints, which are universal obstacles to plan existence; in other words, they appear on every plan from the source to the planning goal. This work envisions such waypoints as sub-problems of the planning problem and the unreachability of any of these waypoints as an explanation for the unsolvability of the original planning problem. We propose a novel method of waypoint identification by casting the problem as an instance of the longest common subsequence problem, a widely popular problem in computer science, typically considered as an illustrative example for the dynamic programming paradigm. Once the waypoints are identified, we perform symbolic reachability analysis on them to identify the earliest unreachable waypoint and report it as the explanation of unsolvability. We present experimental results on unsolvable planning problems in hybrid domains.
Authors Mir Md Sajid Sarwar, Rajarshi Ray
Categories Artificial Intelligence
0
Generating Reading Comprehension Exercises with Large Language Models for Educational Applications
Title CN 利用大语言模型生成阅读理解练习题用于教育应用
Keywords 大语言模型 · 阅读理解生成 · 教育应用 · 内容生成 · 微调
Summary 本文提出了一种基于大语言模型的阅读理解练习生成框架RCEG，通过微调LLM生成候选题目，并使用判别器选择最优结果，从而提升生成内容的相关性和认知适切性。研究构建了专用数据集并从多样性、事实准确性、语言毒性等方面进行评估。
Reason 该论文属于将现有LLM技术应用于教育领域的特定下游任务（生成英语阅读理解题目），并未提出任何针对文档图像理解（DIU）或视觉-语言模型底层能力的方法论创新。其核心是内容生成与筛选框架RCEG，应用场景局限于教育内容生产，属于典型的‘平行应用’，对解决OCR幻觉、复杂排版、细粒度定位等DIU核心问题无迁移价值。
Abstract With the rapid development of large language models (LLMs), the applications of LLMs have grown substantially. In the education domain, LLMs demonstrate significant potential, particularly in automatic text generation, which enables the creation of intelligent and adaptive learning content. This paper proposes a new LLMs framework, which is named as Reading Comprehension Exercise Generation (RCEG). It can generate high-quality and personalized English reading comprehension exercises automatically. Firstly, RCEG uses fine-tuned LLMs to generate content candidates. Then, it uses a discriminator to select the best candidate. Finally, the quality of the generated content has been improved greatly. To evaluate the performance of RCEG, a dedicated dataset for English reading comprehension is constructed to perform the experiments, and comprehensive evaluation metrics are used to analyze the experimental results. These metrics include content diversity, factual accuracy, linguistic toxicity, and pedagogical alignment. Experimental results show that RCEG significantly improves the relevance and cognitive appropriateness of the generated exercises.
Authors Xingyu Huang, Fei Jiang, Jianli Xiao
Categories Artificial Intelligence, Computation and Language
0
PairHuman: A High-Fidelity Photographic Dataset for Customized Dual-Person Generation
Title CN PairHuman：用于定制化双人生成的高保真摄影数据集
Keywords 图像生成 · 双人肖像 · 个性化生成 · 数据集 · Diffusion模型
Summary 本文提出了PairHuman，首个大规模用于生成符合高摄影标准的双人肖像的数据集，包含超过10万张图像及丰富的元数据，如描述、人体关键点和属性标签。同时提出DHumanDiff作为基线方法，增强面部一致性并平衡个性化生成与语义驱动的场景创建。实验表明该数据集和方法能生成高质量、个性化的双人肖像。
Reason 该论文聚焦于双人肖像生成这一图像生成任务，属于纯图像生成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。其提出的数据集和模型旨在解决个性化人物生成、场景布局与视觉美感问题，并未涉及OCR、文本理解、布局分析或表格识别等DIU核心任务。无法迁移用于解决文档中的幻觉、密集文本处理或复杂排版理解等痛点。
Abstract Personalized dual-person portrait customization has considerable potential applications, such as preserving emotional memories and facilitating wedding photography planning. However, the absence of a benchmark dataset hinders the pursuit of high-quality customization in dual-person portrait generation. In this paper, we propose the PairHuman dataset, which is the first large-scale benchmark dataset specifically designed for generating dual-person portraits that meet high photographic standards. The PairHuman dataset contains more than 100K images that capture a variety of scenes, attire, and dual-person interactions, along with rich metadata, including detailed image descriptions, person localization, human keypoints, and attribute tags. We also introduce DHumanDiff, which is a baseline specifically crafted for dual-person portrait generation that features enhanced facial consistency and simultaneously balances in personalized person generation and semantic-driven scene creation. Finally, the experimental results demonstrate that our dataset and method produce highly customized portraits with superior visual quality that are tailored to human preferences. Our dataset is publicly available at https://github.com/annaoooo/PairHuman.
Authors Ting Pan, Ye Wang, Peiguang Jing, Rui Ma, Zili Yi, Yu Liu
Comment 46 pages, 31 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
AlphaBeta is not as good as you think: a simple random games model for a better analysis of deterministic game-solving algorithms
Title CN AlphaBeta并不像你想象的那么好：一种用于更好分析确定性博弈求解算法的简单随机博弈模型
Keywords 博弈树 · AlphaBeta剪枝 · 平均情况复杂度 · 搜索算法 · 概率模型
Summary 本文提出了一种新的概率模型来分析确定性博弈求解算法的平均情况复杂度，通过引入祖先依赖关系构建更具挑战性的游戏树，揭示了AlphaBeta等算法在深层有限博弈树中存在显著常数因子开销。该工作聚焦于经典搜索算法的理论性能评估，不涉及任何视觉、语言或多模态内容。
Reason 该论文研究的是博弈树搜索算法（如AlphaBeta、Scout）在随机生成的游戏树上的平均复杂度分析，属于经典人工智能中的博弈论与算法分析领域。其核心问题和方法与文档图像理解（DIU）、多模态大模型、OCR、视觉表示学习等完全无关，无法迁移解决DIU中的OCR幻觉、密集文本识别、复杂排版理解或长文档推理等关键痛点。
Abstract Deterministic game-solving algorithms are conventionally analyzed in the light of their average-case complexity against a distribution of random game-trees, where leaf values are independently sampled from a fixed distribution. This simplified model enables uncluttered mathematical analysis, revealing two key properties: root value distributions asymptotically collapse to a single fixed value for finite-valued trees, and all reasonable algorithms achieve global optimality. However, these findings are artifacts of the model's design: its long criticized independence assumption strips games of structural complexity, producing trivial instances where no algorithm faces meaningful challenges. To address this limitation, we introduce a simple probabilistic model that incrementally constructs game-trees using a fixed level-wise conditional distribution. By enforcing ancestor dependencies, a critical structural feature of real-world games, our framework generates problems with adjustable difficulty while retaining some form of analytical tractability. For several algorithms, including AlphaBeta and Scout, we derive recursive formulas characterizing their average-case complexities under this model. These allow us to rigorously compare algorithms on deep game-trees, where Monte-Carlo simulations are no longer feasible. While asymptotically, all algorithms seem to converge to identical branching factor (a result analogous to that of independence-based models), deep finite trees reveal stark differences: AlphaBeta incurs a significantly larger constant multiplicative factor compared to algorithms like Scout, leading to a substantial practical slowdown. Our framework sheds new light on classical game-solving algorithms, offering rigorous evidence and analytical tools to advance the understanding of these methods under a richer, more challenging, and yet tractable model.
Authors Raphaël Boige, Amine Boumaza, Bruno Scherrer
Categories Artificial Intelligence
0
Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Tasks
Title CN 通过视频进行推理：首次基于迷宫求解任务评估视频模型的推理能力
Keywords 视频推理 · 迷宫求解 · 空间推理 · 视频生成 · 评测基准
Summary 本文提出VR-Bench，一个基于迷宫求解的视频推理评测基准，包含7920个程序生成的视频，用于系统评估视频模型在空间规划与多步推理方面的能力。实验表明，经过监督微调（SFT）后，视频模型展现出优于现有视觉语言模型的空间感知能力，并在推理时通过多样化采样提升了10%-20%的可靠性。
Reason 该论文研究的是视频模型在迷宫求解任务中的空间推理能力，属于视频理解与生成领域，核心任务为视频生成驱动的推理评估。尽管涉及‘空间推理’，但其方法和实验设计面向通用视频生成模型，与文档图像理解（DIU）中的OCR幻觉、复杂排版、细粒度文本定位或长文档结构建模等核心痛点无直接关联。无法迁移用于解决文档图像中的语义理解、布局分析或表格识别等问题。
Abstract Video Models have achieved remarkable success in high-fidelity video generation with coherent motion dynamics. Analogous to the development from text generation to text-based reasoning in language modeling, the development of video models motivates us to ask: Can video models reason via video generation? Compared with the discrete text corpus, video grounds reasoning in explicit spatial layouts and temporal continuity, which serves as an ideal substrate for spatial reasoning. In this work, we explore the reasoning via video paradigm and introduce VR-Bench -- a comprehensive benchmark designed to systematically evaluate video models' reasoning capabilities. Grounded in maze-solving tasks that inherently require spatial planning and multi-step reasoning, VR-Bench contains 7,920 procedurally generated videos across five maze types and diverse visual styles. Our empirical analysis demonstrates that SFT can efficiently elicit the reasoning ability of video model. Video models exhibit stronger spatial perception during reasoning, outperforming leading VLMs and generalizing well across diverse scenarios, tasks, and levels of complexity. We further discover a test-time scaling effect, where diverse sampling during inference improves reasoning reliability by 10--20%. These findings highlight the unique potential and scalability of reasoning via video for spatial reasoning tasks.
Authors Cheng Yang, Haiyuan Wan, Yiran Peng, Xin Cheng, Zhaoyang Yu, Jiayi Zhang, Junchi Yu, Xinlei Yu, Xiawu Zheng, Dongzhan Zhou, Chenglin Wu
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL IEEE TKDE2025
Title CN 下一代数据库接口：基于大语言模型的文本到SQL技术综述
Keywords Text-to-SQL · 大语言模型 · 数据库接口 · 自然语言处理 · 综述
Summary 本文综述了基于大语言模型的文本到SQL技术的研究进展，涵盖技术挑战、数据集、评估指标以及现有方法的系统分析，并整理了相关开源资源。重点在于如何利用LLM提升自然语言到SQL语句的转换准确率，适用于数据库交互场景。
Reason 该论文属于典型的平行下游应用综述，聚焦于将LLM应用于Text-to-SQL这一特定任务，属于数据库领域的应用研究。尽管涉及自然语言理解与结构化输出生成，但其方法论未针对文档图像理解中的核心痛点（如OCR幻觉、复杂排版、视觉-文本对齐、高分辨率处理等）提出可迁移的底层技术创新，无法直接提升DIU性能。
Abstract Generating accurate SQL from users' natural language questions (text-to-SQL) remains a long-standing challenge due to the complexities involved in user question understanding, database schema comprehension, and SQL generation. Traditional text-to-SQL systems, which combine human engineering and deep neural networks, have made significant progress. Subsequently, pre-trained language models (PLMs) have been developed for text-to-SQL tasks, achieving promising results. However, as modern databases and user questions grow more complex, PLMs with a limited parameter size often produce incorrect SQL. This necessitates more sophisticated and tailored optimization methods, which restricts the application of PLM-based systems. Recently, large language models (LLMs) have shown significant capabilities in natural language understanding as model scale increases. Thus, integrating LLM-based solutions can bring unique opportunities, improvements, and solutions to text-to-SQL research. In this survey, we provide a comprehensive review of existing LLM-based text-to-SQL studies. Specifically, we offer a brief overview of the technical challenges and evolutionary process of text-to-SQL. Next, we introduce the datasets and metrics designed to evaluate text-to-SQL systems. Subsequently, we present a systematic analysis of recent advances in LLM-based text-to-SQL. Finally, we make a summarization and discuss the remaining challenges in this field and suggest expectations for future research directions. All the related resources of LLM-based, including research papers, benchmarks, and open-source projects, are collected for the community in our repository: https://github.com/DEEP-PolyU/Awesome-LLM-based-Text2SQL.
Authors Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, Xiao Huang
Comment Accepted to IEEE TKDE2025
Categories Artificial Intelligence, Computation and Language
0
Pre-Filtering Code Suggestions using Developer Behavioral Telemetry to Optimize LLM-Assisted Programming
Title CN 使用开发者行为遥测预过滤代码建议以优化LLM辅助编程
Keywords LLM辅助编程 · 代码建议 · 行为遥测 · 预过滤 · 人机交互
Summary 本文提出一种轻量级预过滤模型，利用开发者在编码过程中的行为数据（如打字速度、文件导航等）预测代码建议被接受的可能性，在调用LLM之前决定是否生成建议。实验显示该方法显著提升了建议采纳率并减少了无效LLM调用。但该研究局限于编程场景下的用户体验优化，不涉及多模态文档理解或视觉语言建模。
Reason 该论文属于LLM在编程辅助中的下游应用，聚焦于通过行为信号过滤代码建议的触发时机，提升建议采纳率。其核心贡献是针对IDE中代码生成的效率优化，并未涉及文档图像理解（DIU）中的任何任务如DocVQA、布局分析、表格识别或OCR相关挑战。也没有提出可用于解决文档理解中幻觉、密集文本、复杂排版或长文档推理等问题的通用方法。属于典型的‘平行下游应用’，无营养Wrapper类工作。
Abstract Large Language Models (LLMs) are increasingly integrated into code editors to provide AI-powered code suggestions. Yet many of these suggestions are ignored, resulting in wasted computation, increased latency, and unnecessary interruptions. We introduce a lightweight pre-filtering model that predicts the likelihood of suggestion acceptance before invoking the LLM, using only real-time developer telemetry such as typing speed, file navigation, and editing activity. Deployed in a production-grade Visual Studio Code plugin over four months of naturalistic use, our approach nearly doubled acceptance rates (18.4% -> 34.2%) while suppressing 35% of low-value LLM calls. These findings demonstrate that behavioral signals alone can meaningfully improve both user experience and system efficiency in LLM-assisted programming, highlighting the value of timing-aware, privacy-preserving adaptation mechanisms. The filter operates solely on pre-invocation editor telemetry and never inspects code or prompts.
Authors Mohammad Nour Al Awad, Sergey Ivanov, Olga Tikhonova
Comment \c{opyright} 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses
Categories Artificial Intelligence
0
Personalized Federated Segmentation with Shared Feature Aggregation and Boundary-Focused Calibration
Title CN 基于共享特征聚合和边界聚焦校准的个性化联邦分割
Keywords 联邦学习 · 医学图像分割 · 交叉注意力 · 边界损失 · 非独立同分布
Summary 本文提出了一种名为FedOAP的个性化联邦学习方法，用于实现跨不同器官的肿瘤分割。该方法通过解耦交叉注意力机制（DCA）聚合各客户端共享的特征，捕捉器官间的长距离依赖关系，并引入扰动边界损失（PBL）来提升分割边界的精确性。实验表明该方法在多种肿瘤分割任务上优于现有联邦学习方法。
Reason 该论文研究的是医学图像中的肿瘤分割任务，属于纯计算机视觉与医学影像分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。其核心方法如解耦交叉注意力和扰动边界损失，主要针对器官边界的精确定位，但并未涉及OCR、文本布局理解、多模态文档推理等DIU核心问题。且联邦学习框架在此处用于解决跨客户端数据异构性，难以直接迁移至文档理解场景。
Abstract Personalized federated learning (PFL) possesses the unique capability of preserving data confidentiality among clients while tackling the data heterogeneity problem of non-independent and identically distributed (Non-IID) data. Its advantages have led to widespread adoption in domains such as medical image segmentation. However, the existing approaches mostly overlook the potential benefits of leveraging shared features across clients, where each client contains segmentation data of different organs. In this work, we introduce a novel personalized federated approach for organ agnostic tumor segmentation (FedOAP), that utilizes cross-attention to model long-range dependencies among the shared features of different clients and a boundary-aware loss to improve segmentation consistency. FedOAP employs a decoupled cross-attention (DCA), which enables each client to retain local queries while attending to globally shared key-value pairs aggregated from all clients, thereby capturing long-range inter-organ feature dependencies. Additionally, we introduce perturbed boundary loss (PBL) which focuses on the inconsistencies of the predicted mask's boundary for each client, forcing the model to localize the margins more precisely. We evaluate FedOAP on diverse tumor segmentation tasks spanning different organs. Extensive experiments demonstrate that FedOAP consistently outperforms existing state-of-the-art federated and personalized segmentation methods.
Authors Ishmam Tashdeed, Md. Atiqur Rahman, Sabrina Islam, Md. Azam Hossain
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
WaveTuner: Comprehensive Wavelet Subband Tuning for Time Series Forecasting
Title CN WaveTuner：用于时间序列预测的全面小波子带调优
Keywords 时间序列预测 · 小波变换 · 多尺度建模 · KAN网络 · 频带调优
Summary 本文提出WaveTuner，一种基于小波域的全谱子带调优框架，用于提升时间序列预测性能。通过自适应小波细化模块和多分支专业化模块，动态分配子带权重并使用不同阶数的KAN网络建模各频段特征，在多个真实数据集上实现了SOTA效果。
Reason 该论文研究的是时间序列预测中的小波分解与多尺度建模问题，属于纯时序建模领域，与文档图像理解（DIU）或视觉-语言模型（VLM）无直接关联。尽管其提出的多分支频带特异性建模思想具有一定方法论价值，但无法直接迁移解决DIU中的核心痛点如OCR幻觉、复杂排版解析、细粒度视觉定位或长文档推理等。且未涉及任何视觉输入、多模态对齐或文档结构建模。
Abstract Due to the inherent complexity, temporal patterns in real-world time series often evolve across multiple intertwined scales, including long-term periodicity, short-term fluctuations, and abrupt regime shifts. While existing literature has designed many sophisticated decomposition approaches based on the time or frequency domain to partition trend-seasonality components and high-low frequency components, an alternative line of approaches based on the wavelet domain has been proposed to provide a unified multi-resolution representation with precise time-frequency localization. However, most wavelet-based methods suffer from a persistent bias toward recursively decomposing only low-frequency components, severely underutilizing subtle yet informative high-frequency components that are pivotal for precise time series forecasting. To address this problem, we propose WaveTuner, a Wavelet decomposition framework empowered by full-spectrum subband Tuning for time series forecasting. Concretely, WaveTuner comprises two key modules: (i) Adaptive Wavelet Refinement module, that transforms time series into time-frequency coefficients, utilizes an adaptive router to dynamically assign subband weights, and generates subband-specific embeddings to support refinement; and (ii) Multi-Branch Specialization module, that employs multiple functional branches, each instantiated as a flexible Kolmogorov-Arnold Network (KAN) with a distinct functional order to model a specific spectral subband. Equipped with these modules, WaveTuner comprehensively tunes global trends and local variations within a unified time-frequency framework. Extensive experiments on eight real-world datasets demonstrate WaveTuner achieves state-of-the-art forecasting performance in time series forecasting.
Authors Yubo Wang, Hui He, Chaoxi Niu, Zhendong Niu
Categories Artificial Intelligence
0
Optimizing LLM Code Suggestions: Feedback-Driven Timing with Lightweight State Bounds
Title CN 优化LLM代码建议：基于反馈的轻量级状态边界定时机制
Keywords 代码建议 · 大语言模型 · 交互优化 · 自适应延迟 · 认知状态预测
Summary 本文提出一种基于开发者实时反馈的自适应定时机制，用于优化大语言模型在代码自动补全中提供建议的时机。通过结合最近接受率的逻辑变换与有界延迟范围，并引入开发者认知状态的二元预测，系统在实际部署中显著提升了建议接受率，减少了盲目拒绝和无效推理调用。尽管工程效果显著，但其方法局限于IDE环境下的交互设计，不具备向文档理解任务迁移的潜力。
Reason 该论文研究的是代码自动补全场景下的建议触发时机问题，属于特定下游应用（编程助手）中的交互优化，未提出可迁移至文档图像理解领域的底层方法。其核心贡献为延迟调整策略和接受率预测，与DIU中的OCR幻觉、复杂排版、细粒度定位等关键挑战无关，属于典型的平行应用，无方法论创新价值。
Abstract Large Language Models (LLMs) have transformed code auto-completion by generating context-aware suggestions. Yet, deciding when to present these suggestions remains underexplored, often leading to interruptions or wasted inference calls. We propose an adaptive timing mechanism that dynamically adjusts the delay before offering a suggestion based on real-time developer feedback. Our suggested method combines a logistic transform of recent acceptance rates with a bounded delay range, anchored by a high-level binary prediction of the developer's cognitive state. In a two-month deployment with professional developers, our system improved suggestion acceptance from 4.9% with no delay to 15.4% with static delays, and to 18.6% with adaptive timing-while reducing blind rejections (rejections without being read) from 8.3% to 0.36%. Together, these improvements increase acceptance and substantially reduce wasted inference calls by 75%, making LLM-based code assistants more efficient and cost-effective in practice.
Authors Mohammad Nour Al Awad, Sergey Ivanov, Olga Tikhonova
Comment \c{opyright} 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses
Categories Artificial Intelligence
0
Federated style aware transformer aggregation of representations
Title CN 联邦风格感知的表示变换聚合
Keywords Federated Learning · Style Disentanglement · Transformer · Prototype Aggregation · Personalization
Summary 本文提出 FedSTAR，一种风格感知的联邦学习框架，通过解耦客户端特定的风格因子和共享的内容表示来提升个性化联邦学习的效果。使用基于Transformer的注意力机制聚合类别级原型，并交换紧凑的原型和风格向量以降低通信开销。实验表明该方法在异构环境下能提高个性化和鲁棒性，且不增加通信成本。
Reason 该论文研究的是个性化联邦学习中的风格与内容解耦问题，属于 Federated Learning 领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。未涉及OCR、布局分析、表格识别、视觉压缩、多模态对齐、高分辨率处理等DIU核心问题，也无法迁移用于解决OCR幻觉、细粒度定位或长文档推理等痛点。
Abstract Personalized Federated Learning (PFL) faces persistent challenges, including domain heterogeneity from diverse client data, data imbalance due to skewed participation, and strict communication constraints. Traditional federated learning often lacks personalization, as a single global model cannot capture client-specific characteristics, leading to biased predictions and poor generalization, especially for clients with highly divergent data distributions. To address these issues, we propose FedSTAR, a style-aware federated learning framework that disentangles client-specific style factors from shared content representations. FedSTAR aggregates class-wise prototypes using a Transformer-based attention mechanism, allowing the server to adaptively weight client contributions while preserving personalization. Furthermore, by exchanging compact prototypes and style vectors instead of full model parameters, FedSTAR significantly reduces communication overhead. Experimental results demonstrate that combining content-style disentanglement with attention-driven prototype aggregation improves personalization and robustness in heterogeneous environments without increasing communication cost.
Authors Mincheol Jeon, Euinam Huh
Categories Artificial Intelligence
0
FlowSteer: Guiding Few-Step Image Synthesis with Authentic Trajectories
Title CN FlowSteer：通过真实生成轨迹引导少步图像合成
Keywords 图像生成 · 流匹配 · 采样加速 · 知识蒸馏 · 轨迹对齐
Summary 本文提出FlowSteer，旨在提升ReFlow类模型在少步图像生成中的性能。通过发现并解决Piecewised ReFlow中的分布不匹配问题，引入在线轨迹对齐（OTA）和对抗性蒸馏策略，同时修复了FlowMatchEulerDiscreteScheduler中的缺陷，在SD3上验证了有效性。
Reason 该论文属于纯图像生成领域，研究的是加速扩散模型在少步图像合成中的采样效率问题。尽管涉及视觉生成的推理路径优化，但其目标与文档图像理解（DIU）的核心任务如文本识别、布局分析、表格理解等无直接关联。方法聚焦于生成质量与轨迹对齐，难以迁移到解决OCR幻觉、复杂排版或长文档推理等DIU痛点。属于无关领域（图像生成）的工作，不符合关注方向。
Abstract With the success of flow matching in visual generation, sampling efficiency remains a critical bottleneck for its practical application. Among flow models' accelerating methods, ReFlow has been somehow overlooked although it has theoretical consistency with flow matching. This is primarily due to its suboptimal performance in practical scenarios compared to consistency distillation and score distillation. In this work, we investigate this issue within the ReFlow framework and propose FlowSteer, a method unlocks the potential of ReFlow-based distillation by guiding the student along teacher's authentic generation trajectories. We first identify that Piecewised ReFlow's performance is hampered by a critical distribution mismatch during the training and propose Online Trajectory Alignment(OTA) to resolve it. Then, we introduce a adversarial distillation objective applied directly on the ODE trajectory, improving the student's adherence to the teacher's generation trajectory. Furthermore, we find and fix a previously undiscovered flaw in the widely-used FlowMatchEulerDiscreteScheduler that largely degrades few-step inference quality. Our experiment result on SD3 demonstrates our method's efficacy.
Authors Lei Ke, Hubery Yin, Gongye Liu, Zhengyao Lv, Jingcai Guo, Chen Li, Wenhan Luo, Yujiu Yang, Jing Lyu
Comment Few-Step Image Synthesis
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Solving a Research Problem in Mathematical Statistics with AI Assistance
Title CN 利用AI辅助解决数理统计中的研究问题
Keywords AI辅助科研 · 大语言模型 · 数理统计 · 人机协作 · 数学推理
Summary 本文记录了作者借助GPT-5解决一个鲁棒密度估计中未决问题的过程，展示了AI在数学研究中的辅助作用。尽管体现了LLM在推理和技巧建议方面的潜力，但其本质是AI在特定科研场景下的应用案例，缺乏面向文档理解的技术创新或可迁移的方法论。
Reason 该论文属于在特定科学领域（数理统计）中使用大语言模型辅助研究的案例，属于平行应用，未提出任何可迁移至文档图像理解领域的底层方法或架构创新。其内容聚焦于人类与AI协作解决数学问题的工作流记录，不涉及OCR、文档布局分析、表格识别、视觉文本理解等DIU核心任务，也未对VLM处理高分辨率图像、细粒度定位或多模态对齐等技术做出贡献。
Abstract Over the last few months, AI models including large language models have improved greatly. There are now several documented examples where they have helped professional mathematical scientists prove new results, sometimes even helping resolve known open problems. In this short note, we add another example to the list, by documenting how we were able to solve a previously unsolved research problem in robust mathematical statistics with crucial help from GPT-5. Our problem concerns robust density estimation, where the observations are perturbed by Wasserstein-bounded contaminations.In a previous preprint (Chao and Dobriban, 2023, arxiv:2308.01853v2), we have obtained upper and lower bounds on the minimax optimal estimation error; which were, however, not sharp. Starting in October 2025, making significant use of GPT-5 Pro, we were able to derive the minimax optimal error rate (reported in version 3 of the above arxiv preprint). GPT-5 provided crucial help along the way, including by suggesting calculations that we did not think of, and techniques that were not familiar to us, such as the dynamic Benamou-Brenier formulation, for key steps in the analysis. Working with GPT-5 took a few weeks of effort, and we estimate that it could have taken several months to get the same results otherwise. At the same time, there are still areas where working with GPT-5 was challenging: it sometimes provided incorrect references, and glossed over details that sometimes took days of work to fill in. We outline our workflow and steps taken to mitigate issues. Overall, our work can serve as additional documentation for a new age of human-AI collaborative work in mathematical science.
Authors Edgar Dobriban
Categories Artificial Intelligence
0
VALUE: Value-Aware Large Language Model for Query Rewriting via Weighted Trie in Sponsored Search
Title CN VALUE：通过加权Trie实现价值感知的大型语言模型用于搜索广告中的查询重写
Keywords 查询重写 · 价值感知 · 加权Trie · 搜索广告 · LLM
Summary 本文提出VALUE框架，通过引入加权Trie结构在生成过程中融合关键词的实时商业价值信号，提升搜索广告中查询重写的经济价值与语义匹配效果。采用细粒度偏好学习缓解RLHF中的过拟合问题。该工作已部署于实际广告系统并服务于双十一活动。
Reason 该论文属于典型的平行下游应用，研究的是搜索广告中的查询到关键词重写问题，核心创新点（加权Trie、价值感知生成）完全围绕商业广告系统的经济价值优化，并未涉及文档图像理解（DIU）中的任何核心挑战，如OCR幻觉、复杂版面分析、视觉-文本对齐或高分辨率处理。其方法难以迁移到DocVQA或文档理解任务中。
Abstract Query-to-bidword(i.e., bidding keyword) rewriting is fundamental to sponsored search, transforming noisy user queries into semantically relevant and commercially valuable keywords. Recent advances in large language models (LLMs) improve semantic relevance through generative retrieval frameworks, but they rarely encode the commercial value of keywords. As a result, rewrites are often semantically correct yet economically suboptimal, and a reinforcement learning from human feedback (RLHF) stage is usually added after supervised fine-tuning(SFT) to mitigate this deficiency. However, conventional preference alignment frequently overemphasize the ordering of bidword values and is susceptible to overfitting, which degrades rewrite quality. In addition, bidword value changes rapidly, while existing generative methods do not respond to these fluctuations. To address this shortcoming, we introduce VALUE(Value-Aware Large language model for qUery rewriting via wEighted trie), a framework that integrates value awareness directly into generation and enhances value alignment during training. VALUE employs the Weighted Trie, a novel variant of the classical trie that stores real-time value signals for each token. During decoding, the framework adjusts the LLM's token probabilities with these signals, constraining the search space and steering generation toward high-value rewrites. The alignment stage uses a fine-grained preference learning strategy that emphasizes stable, high-value differences and down-weights noisy or transient fluctuations, thereby improving robustness and reducing overfitting. Offline experiments show that VALUE significantly outperforms baselines in both semantic matching and value-centric metrics. VALUE has been deployed on our advertising system since October 2024 and served the Double Eleven promotions, the biggest shopping carnival in China.
Authors Xiao Zhang, Guanyu Chen, Boyang Zuo, Feng Li, Pengjie Wang, Jian Xu, Bo Zheng
Categories Artificial Intelligence
0
A Rule-Based Approach to Specifying Preferences over Conflicting Facts and Querying Inconsistent Knowledge Bases KR 2025
Title CN 一种基于规则的方法用于指定冲突事实间的偏好并查询不一致知识库
Keywords 知识库 · 冲突事实 · 优先级规则 · 修复语义 · 逻辑推理
Summary 本文提出了一种基于规则的框架，用于在不一致的知识库中指定冲突事实之间的优先级关系，并通过循环消除技术构建可操作的优先顺序，最终支持在优先修复语义下的查询回答。实现上使用了答案集编程进行实验验证。
Reason 该论文属于纯知识表示与推理领域，研究如何在存在冲突的事实之间定义优先级以进行查询应答。尽管其方法论涉及逻辑一致性处理，但完全脱离文档图像理解（DIU）的核心任务如DocVQA、布局分析或表格识别等。未涉及任何视觉模态、OCR输出处理、多模态对齐或文档结构建模，也无法迁移解决OCR幻觉、复杂排版解析或长文档推理等关键问题。
Abstract Repair-based semantics have been extensively studied as a means of obtaining meaningful answers to queries posed over inconsistent knowledge bases (KBs). While several works have considered how to exploit a priority relation between facts to select optimal repairs, the question of how to specify such preferences remains largely unaddressed. This motivates us to introduce a declarative rule-based framework for specifying and computing a priority relation between conflicting facts. As the expressed preferences may contain undesirable cycles, we consider the problem of determining when a set of preference rules always yields an acyclic relation, and we also explore a pragmatic approach that extracts an acyclic relation by applying various cycle removal techniques. Towards an end-to-end system for querying inconsistent KBs, we present a preliminary implementation and experimental evaluation of the framework, which employs answer set programming to evaluate the preference rules, apply the desired cycle resolution techniques to obtain a priority relation, and answer queries under prioritized-repair semantics.
Authors Meghyn Bienvenu, Camille Bourgaux, Katsumi Inoue, Robin Jean
Comment This is an extended version of a paper appearing at the 22nd International Conference on Principles of Knowledge Representation and Reasoning (KR 2025). 24 pages. This version corrects Definition 4
Categories Artificial Intelligence
0
Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cache
Title CN 通过自适应多样性缓存缓解HOI检测中的长尾偏差
Keywords HOI检测 · 长尾偏差 · 视觉语言模型 · 训练免费方法 · 特征缓存
Summary 本文提出了一种名为自适应多样性缓存（ADC）的训练免费模块，用于缓解人-物交互检测中的长尾偏差问题。该方法通过构建类别特定的缓存来积累高置信度且多样化的特征表示，并引入频率感知的缓存更新机制以增强对稀有类别的识别能力。实验表明，ADC在HICO-DET和V-COCO数据集上显著提升了现有检测器在稀有类别上的性能。
Reason 该论文研究的是人-物交互（HOI）检测任务，属于通用计算机视觉领域，与文档图像理解（DIU）的核心任务如DocVQA、表格识别、布局分析等无直接关联。尽管其提出的方法为训练免费的插件式模块，但解决的是视觉场景中长尾分布问题，迁移至文档理解场景（如OCR幻觉、密集文本解析、复杂排版处理）的路径不明确，且未涉及多模态语言模型在文档上的推理或表示优化。因此属于无关领域应用。
Abstract Human-Object Interaction (HOI) detection is a fundamental task in computer vision, empowering machines to comprehend human-object relationships in diverse real-world scenarios. Recent advances in VLMs have significantly improved HOI detection by leveraging rich cross-modal representations. However, most existing VLM-based approaches rely heavily on additional training or prompt tuning, resulting in substantial computational overhead and limited scalability, particularly in long-tailed scenarios where rare interactions are severely underrepresented. In this paper, we propose the Adaptive Diversity Cache (ADC) module, a novel training-free and plug-and-play mechanism designed to mitigate long-tail bias in HOI detection. ADC constructs class-specific caches that accumulate high-confidence and diverse feature representations during inference. The method incorporates frequency-aware cache adaptation that favors rare categories and is designed to enable robust prediction calibration without requiring additional training or fine-tuning. Extensive experiments on HICO-DET and V-COCO datasets show that ADC consistently improves existing HOI detectors, achieving up to +8.57\% mAP gain on rare categories and +4.39\% on the full dataset, demonstrating its effectiveness in mitigating long-tail bias while preserving overall performance.
Authors Yuqiu Jiang, Xiaozhen Qiao, Tianyu Mei, Haojian Huang, Yifan Chen, Ye Zheng, Zhe Sun
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
FedRef: Communication-Efficient Bayesian Fine-Tuning using a Reference Model
Title CN FedRef：基于参考模型的通信高效贝叶斯微调
Keywords 联邦学习 · 贝叶斯微调 · 灾难性遗忘 · 参数高效微调 · 参考模型
Summary 本文提出FedRef，一种基于参考模型的联邦学习微调方法，通过引入贝叶斯参数高效迁移学习和近端项来缓解灾难性遗忘，提升模型性能并降低客户端的通信与计算成本。该方法利用保存先前全局特征的参考模型，在每轮优化中保留历史知识。
Reason 该论文研究联邦学习中的微调方法，聚焦于缓解灾难性遗忘和降低通信开销，属于AI模型优化在联邦学习框架下的应用。其核心问题与文档图像理解（DIU）无直接关联，未涉及多模态建模、视觉表示、OCR、布局分析或VQA等任务。尽管贝叶斯方法或参数高效微调技术在广义上可能用于任何模型训练，但本文提出的方法缺乏对高分辨率视觉输入处理、细粒度定位或多模态对齐等DIU关键痛点的针对性设计，且无迁移路径表明可解决OCR幻觉或复杂排版理解等问题。
Abstract Federated learning (FL) collaboratively trains artificial intelligence (AI) models to ensure user data privacy. Sharing only model updates generated from local training on client data with the server enhances user data privacy. However, model performance may suffer due to data and system heterogeneity among clients in FL scenarios. Previous studies have proposed model optimization, fine-tuning, and personalization to achieve improved model performance. Despite these efforts, models resulting from FL scenarios often exhibit catastrophic forgetting, which increases the communication and computational costs of clients for model optimization and raises energy consumption. To address these challenges, we propose a reference model-based fine-tuning method for federated learning that overcomes catastrophic forgetting in each round. Our method is derived from Bayesian parameter-efficient transfer learning and includes an proximal term. It employs a reference model that incorporates previous model parameters and reviews previous global features in the model optimization step to mitigate catastrophic forgetting. As a result, our method achieves higher model performance and lower communication and computational costs for clients than existing methods.
Authors Taehwan Yoon, Bongjun Choi, Wesley De Neve
Comment 11 pages, 16 equations, 5 figures, 6 tables
Categories Artificial Intelligence
0
NEZHA: A Zero-sacrifice and Hyperspeed Decoding Architecture for Generative Recommendations
Title CN NEZHA：一种用于生成式推荐的零损耗超高速解码架构
Keywords 生成式推荐 · 推理加速 · 投机解码 · 幻觉抑制 · 工业级系统
Summary 本文提出NEZHA架构以解决大语言模型在生成式推荐中推理延迟高的问题。通过在主模型内部集成轻量级自回归草案头实现高效自起草，并设计无需额外训练的哈希集基础验证器来抑制幻觉。实验表明其在保持推荐质量的同时显著提升推理速度，并已在淘宝大规模部署。然而，该工作属于推荐系统领域，不涉及文档图像内容的理解或处理。
Reason 该论文聚焦于生成式推荐系统（Generative Recommendation）中的推理加速问题，属于推荐系统领域的下游应用。尽管提出了高效的解码架构和去幻觉机制，但其方法设计针对推荐场景，并未涉及文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别或OCR相关挑战。提出的‘自起草头’与‘基于哈希集的验证器’虽对降低幻觉有一定启发，但缺乏向文档视觉理解场景迁移的直接路径，且未处理高分辨率图像输入、细粒度定位或多模态对齐等DIU关键问题。
Abstract Generative Recommendation (GR), powered by Large Language Models (LLMs), represents a promising new paradigm for industrial recommender systems. However, their practical application is severely hindered by high inference latency, which makes them infeasible for high-throughput, real-time services and limits their overall business impact. While Speculative Decoding (SD) has been proposed to accelerate the autoregressive generation process, existing implementations introduce new bottlenecks: they typically require separate draft models and model-based verifiers, requiring additional training and increasing the latency overhead. In this paper, we address these challenges with NEZHA, a novel architecture that achieves hyperspeed decoding for GR systems without sacrificing recommendation quality. Specifically, NEZHA integrates a nimble autoregressive draft head directly into the primary model, enabling efficient self-drafting. This design, combined with a specialized input prompt structure, preserves the integrity of sequence-to-sequence generation. Furthermore, to tackle the critical problem of hallucination, a major source of performance degradation, we introduce an efficient, model-free verifier based on a hash set. We demonstrate the effectiveness of NEZHA through extensive experiments on public datasets and have successfully deployed the system on Taobao since October 2025, driving the billion-level advertising revenue and serving hundreds of millions of daily active users.
Authors Yejing Wang, Shengyu Zhou, Jinyu Lu, Ziwei Liu, Langming Liu, Maolin Wang, Wenlin Zhang, Feng Li, Wenbo Su, Pengjie Wang, Jian Xu, Xiangyu Zhao
Categories Artificial Intelligence
0
ReBrain: Brain MRI Reconstruction from Sparse CT Slice via Retrieval-Augmented Diffusion WACV 2026
Title CN 通过检索增强扩散从稀疏CT切片重建脑部MRI
Keywords 医学图像生成 · 跨模态重建 · 扩散模型 · 检索增强 · ControlNet
Summary 本文提出ReBrain，一种基于检索增强扩散的框架，用于从稀疏的CT切片重建完整的脑部MRI图像。通过结合Brownian Bridge扩散模型和ControlNet引导机制，并利用外部数据库检索相似病例作为参考，提升生成结果的结构连续性和病理准确性。实验表明该方法在稀疏条件下优于现有方法。
Reason 该论文研究的是跨模态医学图像合成（从CT生成MRI），属于纯医学图像生成任务，与文档图像理解（DIU）或视觉语言模型在文档上的应用无关。尽管使用了检索增强和扩散模型等技术，但其方法针对的是3D医学影像重建问题，无法迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Magnetic Resonance Imaging (MRI) plays a crucial role in brain disease diagnosis, but it is not always feasible for certain patients due to physical or clinical constraints. Recent studies attempt to synthesize MRI from Computed Tomography (CT) scans; however, low-dose protocols often result in highly sparse CT volumes with poor through-plane resolution, making accurate reconstruction of the full brain MRI volume particularly challenging. To address this, we propose ReBrain, a retrieval-augmented diffusion framework for brain MRI reconstruction. Given any 3D CT scan with limited slices, we first employ a Brownian Bridge Diffusion Model (BBDM) to synthesize MRI slices along the 2D dimension. Simultaneously, we retrieve structurally and pathologically similar CT slices from a comprehensive prior database via a fine-tuned retrieval model. These retrieved slices are used as references, incorporated through a ControlNet branch to guide the generation of intermediate MRI slices and ensure structural continuity. We further account for rare retrieval failures when the database lacks suitable references and apply spherical linear interpolation to provide supplementary guidance. Extensive experiments on SynthRAD2023 and BraTS demonstrate that ReBrain achieves state-of-the-art performance in cross-modal reconstruction under sparse conditions.
Authors Junming Liu, Yifei Sun, Weihua Cheng, Yujin Kang, Yirong Chen, Ding Wang, Guosun Zeng
Comment 16 pages, 12 figures, 7 tables; Accepted by WACV 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
SproutBench: A Benchmark for Safe and Ethical Large Language Models for Youth AAAI 2026 Workshop on AI for Education
Title CN SproutBench：面向青少年的大型语言模型安全与伦理基准
Keywords AI安全 · 青少年保护 · 语言模型评估 · 伦理 · 人机交互
Summary 本文提出了SproutBench，一个面向青少年的大型语言模型安全评估基准，包含1283个基于发展心理学设计的对抗性提示，用于检测LLM在情感依赖、隐私泄露、危险行为模仿等方面的风险。通过对47个LLM的实证评估，揭示了现有模型在年龄适配性和安全性方面的严重不足，并提出了面向儿童的AI设计建议。
Reason 该论文聚焦于儿童和青少年使用LLM时的安全与伦理问题，属于AI安全与教育交叉领域，主要任务是构建针对未成年人的风险评估基准。其研究内容与文档图像理解（DIU）、视觉语言模型在文档上的应用、OCR、排版分析、表格识别等核心任务完全无关。未涉及任何视觉输入、多模态处理或文档结构建模，属于典型的非相关领域工作。
Abstract The rapid proliferation of large language models (LLMs) in applications targeting children and adolescents necessitates a fundamental reassessment of prevailing AI safety frameworks, which are largely tailored to adult users and neglect the distinct developmental vulnerabilities of minors. This paper highlights key deficiencies in existing LLM safety benchmarks, including their inadequate coverage of age-specific cognitive, emotional, and social risks spanning early childhood (ages 0--6), middle childhood (7--12), and adolescence (13--18). To bridge these gaps, we introduce SproutBench, an innovative evaluation suite comprising 1,283 developmentally grounded adversarial prompts designed to probe risks such as emotional dependency, privacy violations, and imitation of hazardous behaviors. Through rigorous empirical evaluation of 47 diverse LLMs, we uncover substantial safety vulnerabilities, corroborated by robust inter-dimensional correlations (e.g., between Safety and Risk Prevention) and a notable inverse relationship between Interactivity and Age Appropriateness. These insights yield practical guidelines for advancing child-centric AI design and deployment.
Authors Wenpeng Xing, Lanyi Wei, Haixiao Hu, Rongchang Li, Mohan Li, Changting Lin, Meng Han
Comment Accepted in AAAI 2026 Workshop on AI for Education
Categories Artificial Intelligence, Computation and Language
0
SGDFuse: SAM-Guided Diffusion for High-Fidelity Infrared and Visible Image Fusion Submitted to Information Fusion
Title CN SGDFuse：由SAM引导的扩散模型实现高保真红外与可见光图像融合
Keywords 图像融合 · 扩散模型 · SAM · 语义引导 · 多模态融合
Summary 本文提出SGDFuse，一种基于SAM引导的扩散模型，用于高保真红外与可见光图像融合。通过利用SAM生成的高质量语义掩码作为先验条件，指导扩散模型在粗到精的去噪过程中实现语义感知的融合。实验表明该方法在主观评价、客观指标及下游任务适应性方面均达到SOTA水平。
Reason 该论文研究的是红外与可见光图像融合（IVIF），属于多模态图像融合领域，且核心任务是提升感知能力以服务于下游视觉任务。其方法依赖于SAM生成语义掩码来指导扩散模型进行图像融合，本质上聚焦于通用图像质量增强和跨模态对齐，并未涉及文档图像理解中的关键问题如OCR、布局分析、表格识别或文本推理等。尽管使用了VLM相关技术（SAM），但应用场景与文档无关，属于无关领域。
Abstract Infrared and visible image fusion (IVIF) aims to combine the thermal radiation information from infrared images with the rich texture details from visible images to enhance perceptual capabilities for downstream visual tasks. However, existing methods often fail to preserve key targets due to a lack of deep semantic understanding of the scene, while the fusion process itself can also introduce artifacts and detail loss, severely compromising both image quality and task performance. To address these issues, this paper proposes SGDFuse, a conditional diffusion model guided by the Segment Anything Model (SAM), to achieve high-fidelity and semantically-aware image fusion. The core of our method is to utilize high-quality semantic masks generated by SAM as explicit priors to guide the optimization of the fusion process via a conditional diffusion model. Specifically, the framework operates in a two-stage process: it first performs a preliminary fusion of multi-modal features, and then utilizes the semantic masks from SAM jointly with the preliminary fused image as a condition to drive the diffusion model's coarse-to-fine denoising generation. This ensures the fusion process not only has explicit semantic directionality but also guarantees the high fidelity of the final result. Extensive experiments demonstrate that SGDFuse achieves state-of-the-art performance in both subjective and objective evaluations, as well as in its adaptability to downstream tasks, providing a powerful solution to the core challenges in image fusion. The code of SGDFuse is available at https://github.com/boshizhang123/SGDFuse.
Authors Xiaoyang Zhang, jinjiang Li, Guodong Fan, Yakun Ju, Linwei Fan, Jun Liu, Alex C. Kot
Comment Submitted to Information Fusion
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
GFlowGR: Fine-tuning Generative Recommendation Frameworks with Generative Flow Networks
Title CN GFlowGR：使用生成流网络微调生成式推荐框架
Keywords 生成式推荐 · GFlowNets · 暴露偏差 · 微调 · 推荐系统
Summary 本文提出GFlowGR，一种基于生成流网络（GFlowNets）的生成式推荐微调框架，旨在缓解推荐系统中的暴露偏差问题。通过引入传统协同过滤知识构建奖励模型和轨迹采样器，利用GFlowNets的多样性生成能力提升推荐质量。实验表明该方法在多个真实数据集上有效且鲁棒。
Reason 该论文研究的是生成式推荐系统中的微调问题，属于推荐系统领域的下游应用，核心方法围绕用户-物品交互建模与暴露偏差缓解。尽管使用了GFlowNets这一生成式框架，但其技术路线与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂版面解析、细粒度视觉定位、高分辨率处理等）无直接关联。未涉及多模态输入、视觉表示学习或文档结构建模，无法迁移用于解决DocVQA或布局分析等任务。
Abstract Generative recommendations (GR), which usually include item tokenizers and generative Large Language Models (LLMs), have demonstrated remarkable success across a wide range of scenarios. The majority of existing research efforts primarily concentrate on developing powerful item tokenizers or advancing LLM decoding strategies to attain superior performance. However, the critical fine-tuning step in GR frameworks, which is essential for adapting LLMs to recommendation data, remains largely unexplored. Current approaches predominantly rely on either the next-token prediction loss of supervised fine-tuning (SFT) or recommendationspecific direct preference optimization (DPO) strategies. Both methods ignore the exploration of possible positive unobserved samples, which is commonly referred to as the exposure bias problem. To mitigate this problem, this paper treats the GR as a multi-step generation task and constructs a GFlowNets-based fine-tuning framework (GFlowGR). The proposed framework integrates collaborative knowledge from traditional recommender systems to create an adaptive trajectory sampler and a comprehensive reward model. Leveraging the diverse generation property of GFlowNets, along with sampling and heuristic weighting techniques, GFlowGR emerges as a promising approach to mitigate the exposure bias problem. Extensive empirical results on two real-world datasets and with two different GR backbones highlight the effectiveness and robustness of GFlowGR.
Authors Yejing Wang, Shengyu Zhou, Jinyu Lu, Qidong Liu, Xinhang Li, Wenlin Zhang, Feng Li, Pengjie Wang, Jian Xu, Bo Zheng, Xiangyu Zhao
Categories Artificial Intelligence
0
A Novel Dual-Stream Framework for dMRI Tractography Streamline Classification with Joint dMRI and fMRI Data ISBI 2026
Title CN 一种用于结合dMRI和fMRI数据的dMRI纤维追踪流线分类的新型双流框架
Keywords dMRI · fMRI · 纤维追踪 · 流线分类 · 双流网络
Summary 本文提出了一种双流框架，结合扩散MRI（dMRI）和功能MRI（fMRI）数据对白质纤维束进行分类，利用几何轨迹特征和纤维端点区域的fMRI信号提升功能一致性分割。实验验证了该方法在皮质脊髓束细分任务上的优越性。
Reason 该论文研究的是医学影像中的纤维追踪分类问题，属于神经科学与医学图像分析领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无关。其方法虽然涉及多模态数据融合，但针对的是脑成像数据（dMRI和fMRI），无法迁移到文档理解中的核心痛点如OCR幻觉、复杂排版或长文档推理等问题。
Abstract Streamline classification is essential to identify anatomically meaningful white matter tracts from diffusion MRI (dMRI) tractography. However, current streamline classification methods rely primarily on the geometric features of the streamline trajectory, failing to distinguish between functionally distinct fiber tracts with similar pathways. To address this, we introduce a novel dual-stream streamline classification framework that jointly analyzes dMRI and functional MRI (fMRI) data to enhance the functional coherence of tract parcellation. We design a novel network that performs streamline classification using a pretrained backbone model for full streamline trajectories, while augmenting with an auxiliary network that processes fMRI signals from fiber endpoint regions. We demonstrate our method by parcellating the corticospinal tract (CST) into its four somatotopic subdivisions. Experimental results from ablation studies and comparisons with state-of-the-art methods demonstrate our approach's superior performance.
Authors Haotian Yan, Bocheng Guo, Jianzhong He, Nir A. Sochen, Ofer Pasternak, Lauren J O'Donnell, Fan Zhang
Comment Submitted to ISBI 2026, 7 pages, 2 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Rethinking Garment Conditioning in Diffusion-based Virtual Try-On
Title CN 重新思考基于扩散的虚拟试穿中的服装条件化
Keywords 虚拟试穿 · 扩散模型 · 条件生成 · 单UNet · 效率优化
Summary 本文提出了一种高效的单UNet结构Re-CatVTON，用于提升扩散模型在虚拟试穿任务中的性能与效率。通过改进分类器无关引导策略并引入真实服装潜在表示注入机制，该方法在保持高图像质量的同时降低了计算和内存开销。然而，该工作属于图像生成范畴，不涉及文本理解或多模态语义推理。
Reason 该论文属于纯图像生成领域，研究的是虚拟试穿（Virtual Try-On）任务，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其方法聚焦于扩散模型中的条件输入设计和效率优化，无法迁移解决OCR幻觉、复杂排版理解、长文档推理等DIU核心问题。
Abstract Virtual Try-On (VTON) is the task of synthesizing an image of a person wearing a target garment, conditioned on a person image and a garment image. While diffusion-based VTON models featuring a Dual UNet architecture demonstrate superior fidelity compared to single UNet models, they incur substantial computational and memory overhead due to their heavy structure. In this study, through visualization analysis and theoretical analysis, we derived three hypotheses regarding the learning of context features to condition the denoising process. Based on these hypotheses, we developed Re-CatVTON, an efficient single UNet model that achieves high performance. We further enhance the model by introducing a modified classifier-free guidance strategy tailored for VTON's spatial concatenation conditioning, and by directly injecting the ground-truth garment latent derived from the clean garment latent to prevent the accumulation of prediction error. The proposed Re-CatVTON significantly improves performance compared to its predecessor (CatVTON) and requires less computation and memory than the high-performance Dual UNet model, Leffa. Our results demonstrate improved FID, KID, and LPIPS scores, with only a marginal decrease in SSIM, establishing a new efficiency-performance trade-off for single UNet VTON models.
Authors Kihyun Na, Jinyoung Choi, Injung Kim
Comment 15 pages (including references and supplementary material), 10 figures, 7 tables. Code and pretrained models will be released
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Re-Key-Free, Risky-Free: Adaptable Model Usage Control
Title CN 无需密钥，无风险：可适应的模型使用控制
Keywords 模型版权保护 · 使用控制 · 深度神经网络 · 参数锁定 · 模型演化
Summary 本文提出ADALOC，一种可在模型持续更新过程中保持授权控制的机制。它通过将一部分权重作为内在访问密钥，限制所有更新仅在此密钥范围内进行，从而实现无需重新分发整个模型即可恢复授权状态，并避免每次更新后重新加锁。实验表明该方法在多种基准和架构上能有效维持授权用户高性能，同时使未授权使用性能降至接近随机猜测。
Reason 该论文研究的是深度神经网络的版权保护与使用授权控制，属于模型水印与知识产权保护范畴。尽管其技术涉及模型参数级别的控制，但核心问题与文档图像理解（DIU）或视觉语言模型在文档任务中的性能提升无关。无法迁移用于解决OCR幻觉、复杂排版理解、细粒度定位或长文档推理等DIU核心痛点。
Abstract Deep neural networks (DNNs) have become valuable intellectual property of model owners, due to the substantial resources required for their development. To protect these assets in the deployed environment, recent research has proposed model usage control mechanisms to ensure models cannot be used without proper authorization. These methods typically lock the utility of the model by embedding an access key into its parameters. However, they often assume static deployment, and largely fail to withstand continual post-deployment model updates, such as fine-tuning or task-specific adaptation. In this paper, we propose ADALOC, to endow key-based model usage control with adaptability during model evolution. It strategically selects a subset of weights as an intrinsic access key, which enables all model updates to be confined to this key throughout the evolution lifecycle. ADALOC enables using the access key to restore the keyed model to the latest authorized states without redistributing the entire network (i.e., adaptation), and frees the model owner from full re-keying after each model update (i.e., lock preservation). We establish a formal foundation to underpin ADALOC, providing crucial bounds such as the errors introduced by updates restricted to the access key. Experiments on standard benchmarks, such as CIFAR-100, Caltech-256, and Flowers-102, and modern architectures, including ResNet, DenseNet, and ConvNeXt, demonstrate that ADALOC achieves high accuracy under significant updates while retaining robust protections. Specifically, authorized usages consistently achieve strong task-specific performance, while unauthorized usage accuracy drops to near-random guessing levels (e.g., 1.01% on CIFAR-100), compared to up to 87.01% without ADALOC. This shows that ADALOC can offer a practical solution for adaptive and protected DNN deployment in evolving real-world scenarios.
Authors Zihan Wang, Zhongkui Ma, Xinguo Feng, Chuan Yan, Dongge Liu, Ruoxi Sun, Derui Wang, Minhui Xue, Guangdong Bai
Categories Artificial Intelligence
0
Studying Classifier(-Free) Guidance From a Classifier-Centric Perspective AAAI 2026
Title CN 从分类器中心视角研究分类器（-自由）引导
Keywords 扩散模型 · 分类器引导 · 无分类器引导 · 决策边界 · 条件生成
Summary 本文从分类器中心的视角出发，对扩散模型中的分类器引导和无分类器引导进行实证研究，发现二者均通过将去噪轨迹推离决策边界来实现条件生成，并在1D数据和高维数据上验证了这一机制。研究重点在于生成模型的理论理解，而非文档理解或多模态推理。
Reason 该论文聚焦于扩散模型中的分类器引导与无分类器引导机制，属于图像生成领域的理论分析，主要面向通用图像生成任务。其研究内容与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版解析、细粒度定位或长文档推理等无直接关联。所提出的‘决策边界附近轨迹调控’观点虽在生成模型中有启发意义，但难以迁移到文档理解任务中，尤其不解决VLM在DIU中的多模态对齐、高分辨率处理或视觉token压缩等关键问题。
Abstract Classifier-free guidance has become a staple for conditional generation with denoising diffusion models. However, a comprehensive understanding of classifier-free guidance is still missing. In this work, we carry out an empirical study to provide a fresh perspective on classifier-free guidance. Concretely, instead of solely focusing on classifier-free guidance, we trace back to the root, i.e., classifier guidance, pinpoint the key assumption for the derivation, and conduct a systematic study to understand the role of the classifier. On 1D data, we find that both classifier guidance and classifier-free guidance achieve conditional generation by pushing the denoising diffusion trajectories away from decision boundaries, i.e., areas where conditional information is usually entangled and is hard to learn. To validate this classifier-centric perspective on high-dimensional data, we assess whether a flow-matching postprocessing step that is designed to narrow the gap between a pre-trained diffusion model's learned distribution and the real data distribution, especially near decision boundaries, can improve the performance. Experiments on various datasets verify our classifier-centric understanding.
Authors Xiaoming Zhao, Alexander G. Schwing
Comment v3: AAAI 2026; v2: added derivation details in Appendix A
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Unsupervised Multi-View Visual Anomaly Detection via Progressive Homography-Guided Alignment
Title CN 基于渐进式单应性引导对齐的无监督多视角视觉异常检测
Keywords 视觉异常检测 · 多视角对齐 · 单应性变换 · 扩散模型 · 无监督学习
Summary 本文提出ViewSense-AD（VSAD），一种用于无监督多视角视觉异常检测的新框架。该方法通过引入多视角对齐模块（MVAM），利用单应性变换在相邻视角间进行特征区域投影与对齐，结合视图对齐潜在扩散模型（VALDM）实现去噪过程中的渐进式多阶段对齐，从而学习到对视角变化鲁棒的特征表示。此外，轻量级融合优化模块（FRM）增强了全局一致性并提升判别能力。实验表明，VSAD在RealIAD和MANTA数据集上显著优于现有方法。
Reason 该论文研究的是无监督多视角视觉异常检测，属于工业缺陷检测领域，其核心问题为跨视角几何一致性建模与异常识别。尽管使用了多视图对齐和扩散模型等技术，但其任务目标（如表面缺陷检测）与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、文本推理）无直接关联。方法上聚焦于视觉外观变化抑制和几何对齐，并未涉及文本内容理解、多模态语言建模或文档结构解析，难以迁移到DocVQA、表格识别等DIU任务。
Abstract Unsupervised visual anomaly detection from multi-view images presents a significant challenge: distinguishing genuine defects from benign appearance variations caused by viewpoint changes. Existing methods, often designed for single-view inputs, treat multiple views as a disconnected set of images, leading to inconsistent feature representations and a high false-positive rate. To address this, we introduce ViewSense-AD (VSAD), a novel framework that learns viewpoint-invariant representations by explicitly modeling geometric consistency across views. At its core is our Multi-View Alignment Module (MVAM), which leverages homography to project and align corresponding feature regions between neighboring views. We integrate MVAM into a View-Align Latent Diffusion Model (VALDM), enabling progressive and multi-stage alignment during the denoising process. This allows the model to build a coherent and holistic understanding of the object's surface from coarse to fine scales. Furthermore, a lightweight Fusion Refiner Module (FRM) enhances the global consistency of the aligned features, suppressing noise and improving discriminative power. Anomaly detection is performed by comparing multi-level features from the diffusion model against a learned memory bank of normal prototypes. Extensive experiments on the challenging RealIAD and MANTA datasets demonstrate that VSAD sets a new state-of-the-art, significantly outperforming existing methods in pixel, view, and sample-level visual anomaly proving its robustness to large viewpoint shifts and complex textures.
Authors Xintao Chen, Xiaohao Xu, Bozhong Zheng, Yun Liu, Yingna Wu
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Comparative Study of UNet-based Architectures for Liver Tumor Segmentation in Multi-Phase Contrast-Enhanced Computed Tomography
Title CN 基于UNet架构在多相增强CT中肝脏肿瘤分割的比较研究
Keywords UNet · 肝脏肿瘤分割 · 医学图像 · 注意力机制 · 对比研究
Summary 本文比较了多种UNet系列架构在多相增强CT图像中进行肝脏肿瘤分割的性能，测试了ResNet、Transformer和Mamba等不同主干网络，并引入CBAM注意力模块提升效果。结果显示，结合预训练权重的ResNetUNet3+在Dice分数、IoU和边界精度等指标上表现最优。研究强调经典卷积网络仍具竞争力，并通过Grad-CAM提升可解释性。
Reason 该论文属于医学图像分割领域，研究任务为肝脏肿瘤分割，使用UNet及其变体进行模型比较，完全不涉及文档图像理解（DIU）、DocVQA、OCR、排版分析或任何与文档相关的任务。尽管使用了计算机视觉和深度学习方法，但其应用场景与文档无关，属于典型的无关领域工作。
Abstract Segmentation of liver structures in multi-phase contrast-enhanced computed tomography (CECT) plays a crucial role in computer-aided diagnosis and treatment planning for liver diseases, including tumor detection. In this study, we investigate the performance of UNet-based architectures for liver tumor segmentation, starting from the original UNet and extending to UNet3+ with various backbone networks. We evaluate ResNet, Transformer-based, and State-space (Mamba) backbones, all initialized with pretrained weights. Surprisingly, despite the advances in modern architecture, ResNet-based models consistently outperform Transformer- and Mamba-based alternatives across multiple evaluation metrics. To further improve segmentation quality, we introduce attention mechanisms into the backbone and observe that incorporating the Convolutional Block Attention Module (CBAM) yields the best performance. ResNetUNet3+ with CBAM module not only produced the best overlap metrics with a Dice score of 0.755 and IoU of 0.662, but also achieved the most precise boundary delineation, evidenced by the lowest HD95 distance of 77.911. The model's superiority was further cemented by its leading overall accuracy of 0.925 and specificity of 0.926, showcasing its robust capability in accurately identifying both lesion and healthy tissue. To further enhance interpretability, Grad-CAM visualizations were employed to highlight the region's most influential predictions, providing insights into its decision-making process. These findings demonstrate that classical ResNet architecture, when combined with modern attention modules, remain highly competitive for medical image segmentation tasks, offering a promising direction for liver tumor detection in clinical practice.
Authors Doan-Van-Anh Ly, Thi-Thu-Hien Pham, Thanh-Hai Le
Comment 15 pages, 9 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkage ACL Findings 2025
Title CN SATA：一种通过简单辅助任务关联实现大语言模型越狱的新范式
Keywords 大语言模型安全 · 越狱攻击 · 掩码语言模型 · 对抗攻击 · 提示工程
Summary 本文提出了一种名为SATA（Simple Assistive Task Linkage）的大模型越狱新范式，通过将恶意查询中的敏感词替换为[MASK]标记，并引入简单的辅助任务（如掩码语言建模或按位置查找元素）来隐式传递被屏蔽语义，从而绕过安全过滤机制。实验表明该方法在AdvBench数据集上取得了领先的攻击成功率。然而，该工作聚焦于模型安全性攻防，不涉及任何视觉或多模态内容处理。
Reason 该论文属于纯NLP安全与对齐（Safety/Jailbreak）领域，研究目标是设计越狱攻击以绕过大语言模型的安全对齐机制。尽管涉及LLM推理过程的操控，但其方法与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版解析、细粒度视觉定位或高分辨率多模态输入处理完全无关，无法迁移用于提升文档理解性能。
Abstract Large language models (LLMs) have made significant advancements across various tasks, but their safety alignment remain a major concern. Exploring jailbreak prompts can expose LLMs' vulnerabilities and guide efforts to secure them. Existing methods primarily design sophisticated instructions for the LLM to follow, or rely on multiple iterations, which could hinder the performance and efficiency of jailbreaks. In this work, we propose a novel jailbreak paradigm, Simple Assistive Task Linkage (SATA), which can effectively circumvent LLM safeguards and elicit harmful responses. Specifically, SATA first masks harmful keywords within a malicious query to generate a relatively benign query containing one or multiple [MASK] special tokens. It then employs a simple assistive task such as a masked language model task or an element lookup by position task to encode the semantics of the masked keywords. Finally, SATA links the assistive task with the masked query to jointly perform the jailbreak. Extensive experiments show that SATA achieves state-of-the-art performance and outperforms baselines by a large margin. Specifically, on AdvBench dataset, with mask language model (MLM) assistive task, SATA achieves an overall attack success rate (ASR) of 85% and harmful score (HS) of 4.57, and with element lookup by position (ELP) assistive task, SATA attains an overall ASR of 76% and HS of 4.43.
Authors Xiaoning Dong, Wenbo Hu, Wei Xu, Tianxing He
Comment ACL Findings 2025. Welcome to employ SATA as a baseline
Categories Artificial Intelligence, Computation and Language
0
Can Large Language Models Detect Misinformation in Scientific News Reporting?
Title CN 大型语言模型能否检测科学新闻报道中的虚假信息？
Keywords 虚假信息检测 · 科学新闻 · LLM · 提示工程 · Claim Verification
Summary 本文提出一个名为SciNews的数据集，包含来自可信与不可信来源的科学新闻报道，并结合CORD-19摘要，旨在利用LLM自动检测科学新闻中的虚假陈述。研究探索了零样本、少样本及思维链提示方法在GPT-3.5、GPT-4和Llama2等模型上的表现，关注科学有效性维度的建模。
Reason 该论文属于纯NLP任务，研究的是科学新闻中的虚假信息检测，不涉及任何视觉或文档图像内容。尽管使用了LLM和推理策略（如CoT），但完全脱离多模态与文档理解场景，无法迁移到DIU领域解决OCR幻觉、排版理解或视觉-文本对齐等核心问题。
Abstract Scientific facts are often spun in the popular press with the intent to influence public opinion and action, as was evidenced during the COVID-19 pandemic. Automatic detection of misinformation in the scientific domain is challenging because of the distinct styles of writing in these two media types and is still in its nascence. Most research on the validity of scientific reporting treats this problem as a claim verification challenge. In doing so, significant expert human effort is required to generate appropriate claims. Our solution bypasses this step and addresses a more real-world scenario where such explicit, labeled claims may not be available. The central research question of this paper is whether it is possible to use large language models (LLMs) to detect misinformation in scientific reporting. To this end, we first present a new labeled dataset SciNews, containing 2.4k scientific news stories drawn from trusted and untrustworthy sources, paired with related abstracts from the CORD-19 database. Our dataset includes both human-written and LLM-generated news articles, making it more comprehensive in terms of capturing the growing trend of using LLMs to generate popular press articles. Then, we identify dimensions of scientific validity in science news articles and explore how this can be integrated into the automated detection of scientific misinformation. We propose several baseline architectures using LLMs to automatically detect false representations of scientific findings in the popular press. For each of these architectures, we use several prompt engineering strategies including zero-shot, few-shot, and chain-of-thought prompting. We also test these architectures and prompting strategies on GPT-3.5, GPT-4, and Llama2-7B, Llama2-13B.
Authors Yupeng Cao, Aishwarya Muralidharan Nair, Nastaran Jamalipour Soofi, Elyon Eyimife, K. P. Subbalakshmi
Categories Artificial Intelligence, Computation and Language
0
Any4D: Open-Prompt 4D Generation from Natural Language and Images
Title CN Any4D：从自然语言和图像生成开放提示的4D内容
Keywords 具身智能 · 视频生成 · VLM规划器 · 短视界建模 · 动作-语言对齐
Summary 本文提出Primitive Embodied World Models（PEWM），通过限制生成的时域范围为短视界，实现语言概念与机器人动作视觉表征之间的细粒度对齐，提升数据效率并降低推理延迟。结合模块化VLM规划器和起始-目标热图引导机制（SGG），支持在复杂任务中进行可组合的策略泛化。方法面向具身智能中的长视野视频生成挑战，强调物理交互与高层推理之间的桥梁构建。
Reason 该论文聚焦于具身智能中的4D视频生成，核心问题是基于自然语言和图像生成包含时间维度的动态场景，属于视频生成与具身智能交叉领域。其提出的方法围绕机器人动作的短视界生成与闭环控制展开，虽涉及VLM，但目标是解决物理交互中的长视野生成问题，与文档图像理解（DIU）中的OCR幻觉、复杂排版、细粒度定位等核心痛点无直接关联。无法迁移用于提升文档理解性能，属于无关领域。
Abstract While video-generation-based embodied world models have gained increasing attention, their reliance on large-scale embodied interaction data remains a key bottleneck. The scarcity, difficulty of collection, and high dimensionality of embodied data fundamentally limit the alignment granularity between language and actions and exacerbate the challenge of long-horizon video generation--hindering generative models from achieving a \textit{"GPT moment"} in the embodied domain. There is a naive observation: \textit{the diversity of embodied data far exceeds the relatively small space of possible primitive motions}. Based on this insight, we propose \textbf{Primitive Embodied World Models} (PEWM), which restricts video generation to fixed shorter horizons, our approach \textit{1) enables} fine-grained alignment between linguistic concepts and visual representations of robotic actions, \textit{2) reduces} learning complexity, \textit{3) improves} data efficiency in embodied data collection, and \textit{4) decreases} inference latency. By equipping with a modular Vision-Language Model (VLM) planner and a Start-Goal heatmap Guidance mechanism (SGG), PEWM further enables flexible closed-loop control and supports compositional generalization of primitive-level policies over extended, complex tasks. Our framework leverages the spatiotemporal vision priors in video models and the semantic awareness of VLMs to bridge the gap between fine-grained physical interaction and high-level reasoning, paving the way toward scalable, interpretable, and general-purpose embodied intelligence.
Authors Hao Li, Qiao Sun
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Survival Analysis with Machine Learning for Predicting Li-ion Battery Remaining Useful Life
Title CN 基于机器学习的生存分析预测锂离子电池剩余使用寿命
Keywords 剩余使用寿命 · 生存分析 · 机器学习 · 锂离子电池 · 时间序列
Summary 本文提出了一种结合生存分析与机器学习的混合框架，用于预测锂离子电池的剩余使用寿命（RUL）。通过路径签名将电压时间序列转换为失效时间数据，并采用Cox模型、DeepHit和MTLR等方法进行生存概率估计。实验表明该方法在丰田和NASA电池数据集上具有良好的性能表现。
Reason 该论文研究的是锂离子电池寿命预测问题，属于工业预测性维护领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用完全无关。尽管使用了机器学习和生存分析等方法，但其任务、数据模态（时间序列电压数据）和目标均不涉及文本、布局、OCR或视觉-语言推理，无法迁移至解决DIU中的OCR幻觉、复杂排版或长文档理解等核心痛点。
Abstract Battery degradation significantly impacts the reliability and efficiency of energy storage systems, particularly in electric vehicles and industrial applications. Predicting the remaining useful life (RUL) of lithium-ion batteries is crucial for optimizing maintenance schedules, reducing costs, and improving safety. Traditional RUL prediction methods often struggle with nonlinear degradation patterns and uncertainty quantification. To address these challenges, we propose a hybrid survival analysis framework integrating survival data reconstruction, survival model learning, and survival probability estimation. Our approach transforms battery voltage time series into time-to-failure data using path signatures. The multiple Cox-based survival models and machine-learning-based methods, such as DeepHit and MTLR, are learned to predict battery failure-free probabilities over time. Experiments conducted on the Toyota battery and NASA battery datasets demonstrate the effectiveness of our approach, achieving high time-dependent AUC and concordance index (C-Index) while maintaining a low integrated Brier score.
Authors Jingyuan Xue, Xiaozhen Zhao, Dongjing Jiang, Qingchong Jiao, Redouane EL Bouchtaoui, Jianfei Zhang
Categories Artificial Intelligence
0
ProxT2I: Efficient Reward-Guided Text-to-Image Generation via Proximal Diffusion
Title CN ProxT2I：通过邻近扩散实现高效的奖励引导文本到图像生成
Keywords 文本到图像生成 · 扩散模型 · 邻近算子 · 强化学习 · 采样效率
Summary 本文提出ProxT2I，一种基于反向离散化和条件邻近算子的文本到图像生成方法，替代传统的得分函数建模，结合强化学习进行任务特定奖励优化，并发布了一个包含1500万高质量人像图像与细粒度描述的大规模数据集LAION-Face-T2I-15M。方法在生成效率和人类偏好对齐方面优于传统得分基线，在较低计算成本下达到SOTA水平。
Reason 该论文属于纯图像生成领域，研究的是文本到图像生成中的采样效率与奖励对齐问题，核心方法为基于反向离散化的扩散模型与强化学习策略优化。尽管涉及多模态建模，但其目标与文档图像理解（DIU）任务无直接关联，无法迁移用于解决OCR幻觉、细粒度定位或复杂排版理解等DIU核心痛点。属于负面清单中的‘纯图像生成’类别。
Abstract Diffusion models have emerged as a dominant paradigm for generative modeling across a wide range of domains, including prompt-conditional generation. The vast majority of samplers, however, rely on forward discretization of the reverse diffusion process and use score functions that are learned from data. Such forward and explicit discretizations can be slow and unstable, requiring a large number of sampling steps to produce good-quality samples. In this work we develop a text-to-image (T2I) diffusion model based on backward discretizations, dubbed ProxT2I, relying on learned and conditional proximal operators instead of score functions. We further leverage recent advances in reinforcement learning and policy optimization to optimize our samplers for task-specific rewards. Additionally, we develop a new large-scale and open-source dataset comprising 15 million high-quality human images with fine-grained captions, called LAION-Face-T2I-15M, for training and evaluation. Our approach consistently enhances sampling efficiency and human-preference alignment compared to score-based baselines, and achieves results on par with existing state-of-the-art and open-source text-to-image models while requiring lower compute and smaller model size, offering a lightweight yet performant solution for human text-to-image generation.
Authors Zhenghan Fang, Jian Zheng, Qiaozi Gao, Xiaofeng Gao, Jeremias Sulam
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
A Problem-Oriented Taxonomy of Evaluation Metrics for Time Series Anomaly Detection
Title CN 一种面向问题的时间序列异常检测评估指标分类法
Keywords 时间序列 · 异常检测 · 评估指标 · 分类法 · 判别能力
Summary 本文提出了一种面向问题的框架，用于重新解读时间序列异常检测中的评估指标，将其按所解决的具体评估挑战分为六个维度，并通过实验分析各指标在不同场景下的表现，揭示了某些常用指标对随机得分膨胀的脆弱性，强调指标选择应与具体应用目标对齐。
Reason 该论文研究的是时间序列异常检测的评估指标分类，属于时间序列分析领域，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。其方法论聚焦于评估指标的维度划分和判别能力分析，无法迁移解决DIU中的OCR幻觉、复杂排版、细粒度定位等核心痛点。
Abstract Time series anomaly detection is widely used in IoT and cyber-physical systems, yet its evaluation remains challenging due to diverse application objectives and heterogeneous metric assumptions. This study introduces a problem-oriented framework that reinterprets existing metrics based on the specific evaluation challenges they are designed to address, rather than their mathematical forms or output structures. We categorize over twenty commonly used metrics into six dimensions: 1) basic accuracy-driven evaluation; 2) timeliness-aware reward mechanisms; 3) tolerance to labeling imprecision; 4) penalties reflecting human-audit cost; 5) robustness against random or inflated scores; and 6) parameter-free comparability for cross-dataset benchmarking. Comprehensive experiments are conducted to examine metric behavior under genuine, random, and oracle detection scenarios. By comparing their resulting score distributions, we quantify each metric's discriminative ability -- its capability to distinguish meaningful detections from random noise. The results show that while most event-level metrics exhibit strong separability, several widely used metrics (e.g., NAB, Point-Adjust) demonstrate limited resistance to random-score inflation. These findings reveal that metric suitability must be inherently task-dependent and aligned with the operational objectives of IoT applications. The proposed framework offers a unified analytical perspective for understanding existing metrics and provides practical guidance for selecting or developing more context-aware, robust, and fair evaluation methodologies for time series anomaly detection.
Authors Kaixiang Yang, Jiarong Liu, Yupeng Song, Shuanghua Yang, Yujue Zhou
Categories Artificial Intelligence
0
FoleyBench: A Benchmark For Video-to-Audio Models
Title CN FoleyBench：面向视频到音频模型的基准测试
Keywords 视频到音频生成 · Foley音效 · 多模态基准 · 音视频对齐 · 自动评估
Summary 本文提出了FoleyBench，首个专为Foley风格音效生成设计的大规模视频到音频生成评测基准。该数据集包含5000个（视频，真实音频，文本描述）三元组，聚焦于画面中可见事件所引发的声音，具有更强的声学类别覆盖和细粒度标注。尽管在评估协议上有所创新，但其应用领域完全偏离文档理解。
Reason 该论文属于视频理解与音频生成领域，研究的是视频到音频的生成任务，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心贡献是构建用于Foley音效生成的评测基准，不涉及OCR、布局分析、表格识别或任何文档相关的任务。因此无法迁移至解决文档图像理解中的幻觉、密集文本、复杂排版等核心痛点。
Abstract Video-to-audio generation (V2A) is of increasing importance in domains such as film post-production, AR/VR, and sound design, particularly for the creation of Foley sound effects synchronized with on-screen actions. Foley requires generating audio that is both semantically aligned with visible events and temporally aligned with their timing. Yet, there is a mismatch between evaluation and downstream applications due to the absence of a benchmark tailored to Foley-style scenarios. We find that 74% of videos from past evaluation datasets have poor audio-visual correspondence. Moreover, they are dominated by speech and music, domains that lie outside the use case for Foley. To address this gap, we introduce FoleyBench, the first large-scale benchmark explicitly designed for Foley-style V2A evaluation. FoleyBench contains 5,000 (video, ground-truth audio, text caption) triplets, each featuring visible sound sources with audio causally tied to on-screen events. The dataset is built using an automated, scalable pipeline applied to in-the-wild internet videos from YouTube-based and Vimeo-based sources. Compared to past datasets, we show that videos from FoleyBench have stronger coverage of sound categories from a taxonomy specifically designed for Foley sound. Each clip is further labeled with metadata capturing source complexity, UCS/AudioSet category, and video length, enabling fine-grained analysis of model performance and failure modes. We benchmark several state-of-the-art V2A models, evaluating them on audio quality, audio-video alignment, temporal synchronization, and audio-text consistency. Samples are available at: https://gclef-cmu.org/foleybench
Authors Satvik Dixit, Koichi Saito, Zhi Zhong, Yuki Mitsufuji, Chris Donahue
Categories Artificial Intelligence
0
Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansion
Title CN Yo'City：通过自我批评扩展实现个性化和无边界的真实3D城市场景生成
Keywords 3D城市生成 · 具身智能 · 场景图 · 扩散模型 · 代理框架
Summary 本文提出Yo'City，一种基于代理框架的个性化、可无限扩展的3D城市生成方法。通过‘城市-区域-网格’的分层规划结构，结合全局布局设计与局部细节生成，并引入关系引导的扩展机制，实现语义与空间一致的城市演化。配套构建了评估基准与多维指标。但研究方向属于3D视觉与生成，远离文档理解任务。
Reason 该论文聚焦于3D城市场景生成，属于具身智能与3D视觉领域，涉及虚拟现实和数字孪生应用，与文档图像理解（DIU）或视觉语言模型在文档上的应用无关。尽管使用了大模型的推理能力，但其核心任务为3D场景建模与布局优化，无法迁移解决OCR幻觉、密集文本识别、复杂排版分析等DIU核心问题。
Abstract Realistic 3D city generation is fundamental to a wide range of applications, including virtual reality and digital twins. However, most existing methods rely on training a single diffusion model, which limits their ability to generate personalized and boundless city-scale scenes. In this paper, we present Yo'City, a novel agentic framework that enables user-customized and infinitely expandable 3D city generation by leveraging the reasoning and compositional capabilities of off-the-shelf large models. Specifically, Yo'City first conceptualize the city through a top-down planning strategy that defines a hierarchical "City-District-Grid" structure. The Global Planner determines the overall layout and potential functional districts, while the Local Designer further refines each district with detailed grid-level descriptions. Subsequently, the grid-level 3D generation is achieved through a "produce-refine-evaluate" isometric image synthesis loop, followed by image-to-3D generation. To simulate continuous city evolution, Yo'City further introduces a user-interactive, relationship-guided expansion mechanism, which performs scene graph-based distance- and semantics-aware layout optimization, ensuring spatially coherent city growth. To comprehensively evaluate our method, we construct a diverse benchmark dataset and design six multi-dimensional metrics that assess generation quality from the perspectives of semantics, geometry, texture, and layout. Extensive experiments demonstrate that Yo'City consistently outperforms existing state-of-the-art methods across all evaluation aspects.
Authors Keyang Lu, Sifan Zhou, Hongbin Xu, Gang Xu, Zhifei Yang, Yikai Wang, Zhen Xiao, Jieyi Long, Ming Li
Comment 22 pages, 16 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Learning Primitive Embodied World Models: Towards Scalable Robotic Learning
Title CN 学习基本具身世界模型：迈向可扩展的机器人学习
Keywords 具身世界模型 · VLM规划器 · 机器人学习 · 视频生成 · 细粒度对齐
Summary 本文提出了一种名为Primitive Embodied World Models（PEWM）的新范式，通过将视频生成限制在短时间范围内，实现语言概念与机器人动作视觉表征之间的细粒度对齐，提升数据效率和推理速度，并结合模块化VLM规划器与Start-Goal热图引导机制，支持在复杂任务中进行闭环保控和策略组合泛化。
Reason 该论文属于具身智能与机器人领域，研究的是基于视频生成的具身世界模型，目标是解决机器人动作控制中的长视野生成与语言-动作对齐问题。尽管使用了VLM组件，但其核心任务与文档图像理解（DIU）无关，无法迁移用于解决OCR幻觉、复杂排版或长文档推理等文档理解核心痛点。
Abstract While video-generation-based embodied world models have gained increasing attention, their reliance on large-scale embodied interaction data remains a key bottleneck. The scarcity, difficulty of collection, and high dimensionality of embodied data fundamentally limit the alignment granularity between language and actions and exacerbate the challenge of long-horizon video generation--hindering generative models from achieving a "GPT moment" in the embodied domain. There is a naive observation: the diversity of embodied data far exceeds the relatively small space of possible primitive motions. Based on this insight, we propose a novel paradigm for world modeling--Primitive Embodied World Models (PEWM). By restricting video generation to fixed short horizons, our approach 1) enables fine-grained alignment between linguistic concepts and visual representations of robotic actions, 2) reduces learning complexity, 3) improves data efficiency in embodied data collection, and 4) decreases inference latency. By equipping with a modular Vision-Language Model (VLM) planner and a Start-Goal heatmap Guidance mechanism (SGG), PEWM further enables flexible closed-loop control and supports compositional generalization of primitive-level policies over extended, complex tasks. Our framework leverages the spatiotemporal vision priors in video models and the semantic awareness of VLMs to bridge the gap between fine-grained physical interaction and high-level reasoning, paving the way toward scalable, interpretable, and general-purpose embodied intelligence.
Authors Qiao Sun, Liujia Yang, Wei Tang, Wei Huang, Kaixin Xu, Yongchao Chen, Mingyu Liu, Jiange Yang, Haoyi Zhu, Yating Wang, Tong He, Yilun Chen, Xili Dai, Nanyang Ye, Qinying Gu
Categories Artificial Intelligence
0
DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planning AAAI 2026
Title CN DriveSuprim：面向端到端规划的精确轨迹选择
Keywords 自动驾驶 · 轨迹选择 · 端到端规划 · 安全评估 · 候选生成
Summary 本文提出DriveSuprim，一种用于端到端自动驾驶规划的轨迹选择方法。通过粗到精的候选过滤机制、基于旋转的数据增强和自蒸馏训练框架，提升在复杂和罕见驾驶场景中对安全轨迹的精准选择能力。实验表明其在多个基准上达到SOTA性能。
Reason 该论文属于自动驾驶领域的轨迹规划工作，研究任务为驾驶场景下的路径选择与安全性评估，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管其方法涉及候选生成与评分机制，但应用场景和核心问题域（动态环境决策、车辆控制）与DIU中的OCR幻觉、文本密集识别、复杂版式分析等痛点完全无关，无法迁移或启发文档理解的技术发展。
Abstract Autonomous vehicles must navigate safely in complex driving environments. Imitating a single expert trajectory, as in regression-based approaches, usually does not explicitly assess the safety of the predicted trajectory. Selection-based methods address this by generating and scoring multiple trajectory candidates and predicting the safety score for each. However, they face optimization challenges in precisely selecting the best option from thousands of candidates and distinguishing subtle but safety-critical differences, especially in rare and challenging scenarios. We propose DriveSuprim to overcome these challenges and advance the selection-based paradigm through a coarse-to-fine paradigm for progressive candidate filtering, a rotation-based augmentation method to improve robustness in out-of-distribution scenarios, and a self-distillation framework to stabilize training. DriveSuprim achieves state-of-the-art performance, reaching 93.5% PDMS in NAVSIM v1 and 87.1% EPDMS in NAVSIM v2 without extra data, with 83.02 Driving Score and 60.00 Success Rate on the Bench2Drive benchmark, demonstrating superior planning capabilities in various driving scenarios.
Authors Wenhao Yao, Zhenxin Li, Shiyi Lan, Zi Wang, Xinglong Sun, Jose M. Alvarez, Zuxuan Wu
Comment Accepted to AAAI 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
N2N: A Parallel Framework for Large-Scale MILP under Distributed Memory
Title CN N2N：一种用于大规模混合整数线性规划的分布式内存并行框架
Keywords MILP · 并行计算 · 分支定界 · 分布式内存 · SCIP
Summary 本文提出了一种名为N2N的分布式内存并行框架，用于加速大规模混合整数线性规划（MILP）问题的求解。该框架将分支定界法中的节点映射到分布式计算节点上，支持确定性和非确定性模式，并集成了SCIP和HiGHS等开源求解器。实验表明，N2N-SCIP在千级MPI进程下显著优于现有最先进的ParaSCIP。然而，该工作属于数学优化与高性能计算交叉领域，与文档图像理解无直接关联。
Reason 该论文研究的是混合整数线性规划（MILP）求解器在分布式环境下的并行化问题，属于运筹优化领域，与文档图像理解（DIU）、多模态大模型（VLM）、视觉文本识别或DocVQA等任务完全无关。尽管涉及AI相关技术栈（如并行计算），但其方法无法迁移解决OCR幻觉、复杂排版理解、细粒度定位或长文档推理等DIU核心痛点。
Abstract Parallelization has emerged as a promising approach for accelerating MILP solving. However, the complexity of the branch-and-bound (B&B) framework and the numerous effective algorithm components in MILP solvers make it difficult to parallelize. In this study, a scalable parallel framework, N2N (a node-to-node framework that maps the B&B nodes to distributed computing nodes), was proposed to solve large-scale problems in a distributed memory computing environment. Both deterministic and nondeterministic modes are supported, and the framework is designed to be easily integrated with existing solvers. Regarding the deterministic mode, a novel sliding-window-based algorithm was designed and implemented to ensure that tasks are generated and solved in a deterministic order. Moreover, several advanced techniques, such as the utilization of CP search and general primal heuristics, have been developed to fully utilize distributed computing resources and capabilities of base solvers. Adaptive solving and data communication optimization were also investigated. A popular open-source MILP solver, SCIP, was integrated into N2N as the base solver, yielding N2N-SCIP. Extensive computational experiments were conducted to evaluate the performance of N2N-SCIP compared to ParaSCIP, which is a state-of-the-art distributed parallel MILP solver under the UG framework. The nondeterministic N2N-SCIP achieves speedups of 22.52 and 12.71 with 1,000 MPI processes on the Kunpeng and x86 computing clusters, which is 1.98 and 2.08 times faster than ParaSCIP, respectively. In the deterministic mode, N2N-SCIP also shows significant performance improvements over ParaSCIP across different process numbers and computing clusters. To validate the generality of N2N, HiGHS, another open-source solver, was integrated into N2N. The related results are analyzed, and the requirements of N2N on base solvers are also concluded.
Authors Longfei Wang, Junyan Liu, Fan Zhang, Jiangwen Wei, Yuanhua Tang, Jie Sun, Xiaodong Luo
Comment 18 pages, 2 figures
Categories Artificial Intelligence
0
FinAudio: A Benchmark for Audio Large Language Models in Financial Applications
Title CN FinAudio：面向金融应用的音频大语言模型基准
Keywords AudioLLM · 金融音频 · 语音识别 · 摘要 · 基准测试
Summary 本文提出了FinAudio，首个用于评估音频大语言模型在金融领域表现的基准，包含短/长音频的语音识别及长音频摘要三项任务，并发布了相关数据集。研究评测了七种主流AudioLLM，揭示了其在金融场景中的局限性。
Reason 该论文属于典型的‘平行下游应用’，仅将现有AudioLLM技术应用于金融领域的音频数据（如财报电话会议），构建了一个特定场景的基准测试。其任务聚焦于ASR和摘要，未提出任何底层方法创新，且与文档图像理解（DIU）、视觉表示、多模态文档推理等核心问题完全无关。
Abstract Audio Large Language Models (AudioLLMs) have received widespread attention and have significantly improved performance on audio tasks such as conversation, audio understanding, and automatic speech recognition (ASR). Despite these advancements, there is an absence of a benchmark for assessing AudioLLMs in financial scenarios, where audio data, such as earnings conference calls and CEO speeches, are crucial resources for financial analysis and investment decisions. In this paper, we introduce \textsc{FinAudio}, the first benchmark designed to evaluate the capacity of AudioLLMs in the financial domain. We first define three tasks based on the unique characteristics of the financial domain: 1) ASR for short financial audio, 2) ASR for long financial audio, and 3) summarization of long financial audio. Then, we curate two short and two long audio datasets, respectively, and develop a novel dataset for financial audio summarization, comprising the \textsc{FinAudio} benchmark. Then, we evaluate seven prevalent AudioLLMs on \textsc{FinAudio}. Our evaluation reveals the limitations of existing AudioLLMs in the financial domain and offers insights for improving AudioLLMs. All datasets and codes will be released.
Authors Yupeng Cao, Haohang Li, Yangyang Yu, Shashidhar Reddy Javaji, Yueru He, Jimin Huang, Qianqian Xie, Xiao-yang Liu, K. P. Subbalakshmi, Meikang Qiu, Sophia Ananiadou, Jian-Yun Nie
Categories Artificial Intelligence
0
AIRHILT: A Human-in-the-Loop Testbed for Multimodal Conflict Detection in Aviation
Title CN AIRHILT：一种用于航空领域多模态冲突检测的人在环测试平台
Keywords 多模态融合 · 人在环 · 航空安全 · 冲突检测 · 仿真平台
Summary 本文提出了AIRHILT，一个基于Godot引擎的轻量级、模块化航空人机协同测试平台，用于评估飞行员与空管场景下的多模态冲突检测系统。平台整合了语音通信、视觉画面与ADS-B数据，支持人在环仿真，并提供标准化接口以接入ASR、目标检测和大模型推理模块。通过一个集成了Whisper、YOLO和GPT-OSS-20B的参考流水线，展示了在跑道重叠等典型场景中的预警能力。
Reason 该论文属于特定垂直领域的应用系统（航空交通冲突检测），其核心是构建一个集成ASR、视觉检测、ADS-B和大模型推理的仿真环境，用于评估辅助决策系统。尽管涉及多模态融合与实时推理，但其方法论未提出可迁移至文档图像理解（DIU）的底层技术创新，属于典型的‘平行下游应用’，不符合本研究关注的技术源头创新要求。
Abstract We introduce AIRHILT (Aviation Integrated Reasoning, Human-in-the-Loop Testbed), a modular and lightweight simulation environment designed to evaluate multimodal pilot and air traffic control (ATC) assistance systems for aviation conflict detection. Built on the open-source Godot engine, AIRHILT synchronizes pilot and ATC radio communications, visual scene understanding from camera streams, and ADS-B surveillance data within a unified, scalable platform. The environment supports pilot- and controller-in-the-loop interactions, providing a comprehensive scenario suite covering both terminal area and en route operational conflicts, including communication errors and procedural mistakes. AIRHILT offers standardized JSON-based interfaces that enable researchers to easily integrate, swap, and evaluate automatic speech recognition (ASR), visual detection, decision-making, and text-to-speech (TTS) models. We demonstrate AIRHILT through a reference pipeline incorporating fine-tuned Whisper ASR, YOLO-based visual detection, ADS-B-based conflict logic, and GPT-OSS-20B structured reasoning, and present preliminary results from representative runway-overlap scenarios, where the assistant achieves an average time-to-first-warning of approximately 7.7 s, with average ASR and vision latencies of approximately 5.9 s and 0.4 s, respectively. The AIRHILT environment and scenario suite are openly available, supporting reproducible research on multimodal situational awareness and conflict detection in aviation; code and scenarios are available at https://github.com/ogarib3/airhilt.
Authors Omar Garib, Jayaprakash D. Kambhampaty, Olivia J. Pinon Fischer, Dimitri N. Mavris
Comment 9 pages, 4 figures, 1 table, 1 algorithm
Categories Artificial Intelligence
0
Modality-Collaborative Low-Rank Decomposers for Few-Shot Video Domain Adaptation
Title CN 用于少样本视频域自适应的模态协同低秩分解器
Keywords 视频域自适应 · 少样本学习 · 多模态学习 · 低秩分解 · 模态协同
Summary 本文提出一种名为MC-LRD的新框架，通过模态协同低秩分解器和多模态分解路由器，将各模态特征分解为具有不同域偏移程度的模态特有和共享成分，并引入正交去相关约束和跨域激活一致性损失来提升少样本视频域自适应性能。实验表明该方法在三个基准上优于现有方法。
Reason 该论文研究的是少样本视频域自适应（FSVDA），属于视频理解领域，且核心方法聚焦于多模态特征分解与域对齐，与文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等无直接关联。尽管涉及多模态协作，但其应用场景为视频，处理的是时序动态模态（如视觉与音频），而非文档中的静态视觉-文本模态交互。无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心问题。
Abstract In this paper, we study the challenging task of Few-Shot Video Domain Adaptation (FSVDA). The multimodal nature of videos introduces unique challenges, necessitating the simultaneous consideration of both domain alignment and modality collaboration in a few-shot scenario, which is ignored in previous literature. We observe that, under the influence of domain shift, the generalization performance on the target domain of each individual modality, as well as that of fused multimodal features, is constrained. Because each modality is comprised of coupled features with multiple components that exhibit different domain shifts. This variability increases the complexity of domain adaptation, thereby reducing the effectiveness of multimodal feature integration. To address these challenges, we introduce a novel framework of Modality-Collaborative LowRank Decomposers (MC-LRD) to decompose modality-unique and modality-shared features with different domain shift levels from each modality that are more friendly for domain alignment. The MC-LRD comprises multiple decomposers for each modality and Multimodal Decomposition Routers (MDR). Each decomposer has progressively shared parameters across different modalities. The MDR is leveraged to selectively activate the decomposers to produce modality-unique and modality-shared features. To ensure efficient decomposition, we apply orthogonal decorrelation constraints separately to decomposers and subrouters, enhancing their diversity. Furthermore, we propose a cross-domain activation consistency loss to guarantee that target and source samples of the same category exhibit consistent activation preferences of the decomposers, thereby facilitating domain alignment. Extensive experimental results on three public benchmarks demonstrate that our model achieves significant improvements over existing methods.
Authors Yuyang Wanyan, Xiaoshan Yang, Weiming Dong, Changsheng Xu
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
PresentCoach: Dual-Agent Presentation Coaching through Exemplars and Interactive Feedback
Title CN PresentCoach：通过范例和交互反馈的双智能体演讲辅导
Keywords 智能体系统 · 演讲辅导 · 多模态反馈 · 语音合成 · 教育AI
Summary 本文提出一个双智能体系统PresentCoach，用于辅助演讲训练。其中一个智能体根据用户提供的幻灯片生成示范性讲解视频，另一个智能体则对用户录制的演讲进行评估，并以观察-影响-建议（OIS）格式提供反馈。系统还引入‘观众智能体’模拟听众反应，提升反馈的人性化程度。整体为教育场景下的AI应用系统，依赖多模型集成但无方法层面创新。
Reason 该论文属于典型的‘平行下游应用’，将多模态智能体应用于演讲技能培训这一特定场景。尽管涉及幻灯片处理和语音反馈，但其核心是教育类AI助手的设计与实现，未提出任何针对文档图像理解（DIU）任务的方法论创新。所使用的VLM、语音合成、视频生成等技术均为现有工具的集成，无底层架构或训练策略改进，无法迁移解决OCR幻觉、复杂版面解析或长文档推理等DIU核心问题。
Abstract Effective presentation skills are essential in education, professional communication, and public speaking, yet learners often lack access to high-quality exemplars or personalized coaching. Existing AI tools typically provide isolated functionalities such as speech scoring or script generation without integrating reference modeling and interactive feedback into a cohesive learning experience. We introduce a dual-agent system that supports presentation practice through two complementary roles: the Ideal Presentation Agent and the Coach Agent. The Ideal Presentation Agent converts user-provided slides into model presentation videos by combining slide processing, visual-language analysis, narration script generation, personalized voice synthesis, and synchronized video assembly. The Coach Agent then evaluates user-recorded presentations against these exemplars, conducting multimodal speech analysis and delivering structured feedback in an Observation-Impact-Suggestion (OIS) format. To enhance the authenticity of the learning experience, the Coach Agent incorporates an Audience Agent, which simulates the perspective of a human listener and provides humanized feedback reflecting audience reactions and engagement. Together, these agents form a closed loop of observation, practice, and feedback. Implemented on a robust backend with multi-model integration, voice cloning, and error handling mechanisms, the system demonstrates how AI-driven agents can provide engaging, human-centered, and scalable support for presentation skill development in both educational and professional contexts.
Authors Sirui Chen, Jinsong Zhou, Xinli Xu, Xiaoyu Yang, Litao Guo, Ying-Cong Chen
Comment 13pages,6figures
Categories Artificial Intelligence
0
Multimodal Real-Time Anomaly Detection and Industrial Applications
Title CN 多模态实时异常检测与工业应用
Keywords 多模态学习 · 异常检测 · 视频监控 · 跨模态注意力 · 工业应用
Summary 本文提出了一种用于实时活动识别和异常检测的多模态房间监控系统，结合了视频（YOLOv8、DETR）和音频（AST、Wav2Vec2、HuBERT）处理技术，并通过双向跨模态注意力和多方法融合提升性能。系统经历了两代演进，在工业安全等场景中实现了高精度与实时性。然而，其研究目标和方法均面向物理世界监控，不适用于文档图像理解任务。
Reason 该论文聚焦于通用场景下的多模态（视频+音频）异常检测，属于视频监控与工业安全领域，核心任务为活动识别与跨模态融合，并未涉及文档图像、文本理解、OCR、排版分析或任何文档VQA相关任务。其方法虽包含跨模态注意力与多模型集成，但应用场景与文档理解无直接关联，且未提出可迁移至DIU领域的底层技术创新（如视觉压缩、高分辨率处理架构、去幻觉机制等），属于无关领域研究。
Abstract This paper presents the design, implementation, and evolution of a comprehensive multimodal room-monitoring system that integrates synchronized video and audio processing for real-time activity recognition and anomaly detection. We describe two iterations of the system: an initial lightweight implementation using YOLOv8, ByteTrack, and the Audio Spectrogram Transformer (AST), and an advanced version that incorporates multi-model audio ensembles, hybrid object detection, bidirectional cross-modal attention, and multi-method anomaly detection. The evolution demonstrates significant improvements in accuracy, robustness, and industrial applicability. The advanced system combines three audio models (AST, Wav2Vec2, and HuBERT) for comprehensive audio understanding, dual object detectors (YOLO and DETR) for improved accuracy, and sophisticated fusion mechanisms for enhanced cross-modal learning. Experimental evaluation shows the system's effectiveness in general monitoring scenarios as well as specialized industrial safety applications, achieving real-time performance on standard hardware while maintaining high accuracy.
Authors Aman Verma, Keshav Samdani, Mohd. Samiuddin Shafi
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Empathetic Cascading Networks: A Multi-Stage Prompting Technique for Reducing Social Biases in Large Language Models
Title CN 共情级联网络：一种减少大语言模型社会偏见的多阶段提示技术
Keywords 大语言模型 · 社会偏见 · 共情 · 多阶段提示 · 对话AI
Summary 本文提出了一种名为共情级联网络（ECN）的多阶段提示框架，包含视角采纳、情感共鸣、反思理解与整合综合四个阶段，旨在提升大语言模型在对话中的共情能力和包容性。实验表明该方法在GPT-3.5-turbo和GPT-4上均能显著提高共情商数（EQ），同时保持良好的尊重程度和困惑度指标。
Reason 该论文研究的是通过多阶段提示减轻大语言模型中的社会偏见和提升共情能力，属于纯NLP领域的安全与对齐（Safety/Alignment）问题，且不涉及视觉输入或幻觉消除。其方法无法直接迁移至文档图像理解中的OCR幻觉、细粒度定位或复杂排版解析等核心痛点，属于无关领域。
Abstract This report presents the Empathetic Cascading Networks (ECN) framework, a multi-stage prompting method designed to enhance the empathetic and inclusive capabilities of large language models. ECN employs four stages: Perspective Adoption, Emotional Resonance, Reflective Understanding, and Integrative Synthesis, to guide models toward generating emotionally resonant and contextually aware responses. Experimental results demonstrate that ECN achieves the highest Empathy Quotient (EQ) scores across GPT-3.5-turbo and GPT-4, while maintaining competitive Regard and Perplexity metrics. These findings emphasize ECN's potential for applications requiring empathy and inclusivity in conversational AI.
Authors Wangjiaxuan Xin
Categories Artificial Intelligence, Computation and Language
0
Stable Multi-Drone GNSS Tracking System for Marine Robots
Title CN 用于海洋机器人的稳定多无人机GNSS跟踪系统
Keywords GNSS · 多无人机系统 · 目标跟踪 · 海洋机器人 · 扩展卡尔曼滤波
Summary 本文提出了一种基于多无人机的GNSS跟踪系统，通过视觉检测、多目标跟踪和GNSS三角测量结合置信度加权的扩展卡尔曼滤波器，实现对水面及近水面海洋机器人的稳定定位。系统还引入跨无人机ID对齐算法以保证全局一致性，并在多种复杂环境中验证了其鲁棒性与可扩展性。
Reason 该论文研究的是基于无人机的GNSS定位与多目标跟踪，应用于海洋机器人导航，属于具身智能与机器人领域，且涉及的是定位与跟踪算法，与文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的应用无直接关联。无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心问题。
Abstract Accurate localization is essential for marine robotics, yet Global Navigation Satellite System (GNSS) signals are unreliable or unavailable even at a very short distance below the water surface. Traditional alternatives, such as inertial navigation, Doppler Velocity Loggers (DVL), SLAM, and acoustic methods, suffer from error accumulation, high computational demands, or infrastructure dependence. In this work, we present a scalable multi-drone GNSS-based tracking system for surface and near-surface marine robots. Our approach combines efficient visual detection, lightweight multi-object tracking, GNSS-based triangulation, and a confidence-weighted Extended Kalman Filter (EKF) to provide stable GNSS estimation in real time. We further introduce a cross-drone tracking ID alignment algorithm that enforces global consistency across views, enabling robust multi-robot tracking with redundant aerial coverage. We validate our system in diversified complex settings to show the scalability and robustness of the proposed algorithm.
Authors Shuo Wen, Edwin Meriaux, Mariana Sosa Guzmán, Zhizun Wang, Junming Shi, Gregory Dudek
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generation
Title CN ImAgent：一种用于测试时可扩展图像生成的统一多模态代理框架
Keywords 文本到图像生成 · 多模态代理 · 测试时扩展 · 自评估 · 生成控制
Summary 本文提出ImAgent，一个无需训练的统一多模态代理框架，通过在推理阶段集成推理、生成和自评估模块，动态协调多个生成动作以提升文本到图像生成的质量和语义对齐。该方法不依赖外部模型，在图像生成和编辑任务上表现出优于基线模型的效果。
Reason 该论文聚焦于文本到图像（T2I）生成中的测试时扩展问题，属于纯图像生成领域。尽管提出了一个统一的多模态代理框架，但其目标是提升生成图像的质量与语义一致性，与文档图像理解（DIU）的核心任务如DocVQA、表格识别、布局分析等无直接关联。方法难以迁移至解决OCR幻觉、密集文本识别或复杂排版理解等DIU痛点，属于无关领域应用。
Abstract Recent text-to-image (T2I) models have made remarkable progress in generating visually realistic and semantically coherent images. However, they still suffer from randomness and inconsistency with the given prompts, particularly when textual descriptions are vague or underspecified. Existing approaches, such as prompt rewriting, best-of-N sampling, and self-refinement, can mitigate these issues but usually require additional modules and operate independently, hindering test-time scaling efficiency and increasing computational overhead. In this paper, we introduce ImAgent, a training-free unified multimodal agent that integrates reasoning, generation, and self-evaluation within a single framework for efficient test-time scaling. Guided by a policy controller, multiple generation actions dynamically interact and self-organize to enhance image fidelity and semantic alignment without relying on external models. Extensive experiments on image generation and editing tasks demonstrate that ImAgent consistently improves over the backbone and even surpasses other strong baselines where the backbone model fails, highlighting the potential of unified multimodal agents for adaptive and efficient image generation under test-time scaling.
Authors Kaishen Wang, Ruibo Chen, Tong Zheng, Heng Huang
Comment 12 pages, 5 tables, 6 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference
Title CN 人工智能有多“饥饿”？评估大语言模型推理的能源、水和碳足迹
Keywords 大语言模型 · 环境足迹 · 能耗 · 绿色AI · 基准测试
Summary 本文提出了一种面向基础设施的基准框架，用于量化30个主流大语言模型在商业数据中心中推理阶段的能源、水和碳足迹。通过结合公开API性能数据、企业特定的环境乘数以及硬件配置的统计推断，作者评估了不同模型的资源消耗，并使用交叉效率数据分析（DEA）对模型进行能效排名。结果显示，最耗能的模型在长提示下的能耗超过29Wh，是最高效系统的65倍以上。即使单次短查询仅消耗0.42Wh，若日均请求达7亿次，则年耗电量相当于3.5万个美国家庭的用电量，蒸发的淡水量满足120万人的年饮用水需求，碳排放需芝加哥规模的森林才能抵消。研究揭示了AI普及背后日益严重的资源消耗悖论。
Reason 该论文研究的是大语言模型推理过程中的环境影响（如能耗、碳排放），属于AI可持续性与绿色计算领域，与文档图像理解（DIU）或视觉语言模型在文档任务中的技术改进无关。未涉及OCR、布局分析、表格识别、多模态对齐、视觉压缩、细粒度定位等DIU核心问题，也无法迁移为解决DIU痛点（如幻觉、复杂排版、长文档推理）的方法工具。
Abstract This paper introduces an infrastructure-aware benchmarking framework for quantifying the environmental footprint of LLM inference across 30 state-of-the-art models in commercial datacenters. The framework combines public API performance data with company-specific environmental multipliers and statistical inference of hardware configurations. We additionally utilize cross-efficiency Data Envelopment Analysis (DEA) to rank models by performance relative to environmental cost and provide a dynamically updated dashboard that visualizes model-level energy, water, and carbon metrics. Results show the most energy-intensive models exceed 29 Wh per long prompt, over 65 times the most efficient systems. Even a 0.42 Wh short query, when scaled to 700M queries/day, aggregates to annual electricity comparable to 35{,}000 U.S. homes, evaporative freshwater equal to the annual drinking needs of 1.2M people, and carbon emissions requiring a Chicago-sized forest to offset. These findings highlight a growing paradox: as AI becomes cheaper and faster, global adoption drives disproportionate resource consumption. Our methodology offers a standardized, empirically grounded basis for sustainability benchmarking and accountability in AI deployment.
Authors Nidhal Jegham, Marwan Abdelatti, Chan Young Koh, Lassad Elmoubarki, Abdeltawab Hendawi
Categories Artificial Intelligence
0
Do LLMs Feel? Teaching Emotion Recognition with Prompts, Retrieval, and Curriculum Learning AAAI 2026
Title CN 大语言模型有情感吗？通过提示、检索和课程学习教授情感识别
Keywords 情感识别 · 提示工程 · 检索增强 · 课程学习 · 大语言模型
Summary 本文提出PRC-Emo框架，结合提示工程、示例检索和课程学习来提升大语言模型在对话中识别显性和隐性情绪的能力。在IEMOCAP和MELD数据集上取得了SOTA效果。然而，该工作完全基于文本模态，不涉及图像或文档布局理解。
Reason 该论文研究的是对话中的情感识别（ERC），属于纯NLP任务，且聚焦于情感理解这一与文档图像理解（DIU）无关的应用场景。尽管使用了提示工程、检索和课程学习等技术，但这些方法应用于社交对话的情感分析，并未涉及视觉模态或文档结构信息处理，无法迁移到解决OCR幻觉、复杂排版或细粒度定位等DIU核心痛点。
Abstract Emotion Recognition in Conversation (ERC) is a crucial task for understanding human emotions and enabling natural human-computer interaction. Although Large Language Models (LLMs) have recently shown great potential in this field, their ability to capture the intrinsic connections between explicit and implicit emotions remains limited. We propose a novel ERC training framework, PRC-Emo, which integrates Prompt engineering, demonstration Retrieval, and Curriculum learning, with the goal of exploring whether LLMs can effectively perceive emotions in conversational contexts. Specifically, we design emotion-sensitive prompt templates based on both explicit and implicit emotional cues to better guide the model in understanding the speaker's psychological states. We construct the first dedicated demonstration retrieval repository for ERC, which includes training samples from widely used datasets, as well as high-quality dialogue examples generated by LLMs and manually verified. Moreover, we introduce a curriculum learning strategy into the LoRA fine-tuning process, incorporating weighted emotional shifts between same-speaker and different-speaker utterances to assign difficulty levels to dialogue samples, which are then organized in an easy-to-hard training sequence. Experimental results on two benchmark datasets -- IEMOCAP and MELD -- show that our method achieves new state-of-the-art (SOTA) performance, demonstrating the effectiveness and generalizability of our approach in improving LLM-based emotional understanding.
Authors Xinran Li, Yu Liu, Jiaqi Qiao, Xiujuan Xu
Comment Accepted at AAAI 2026
Categories Artificial Intelligence
0
Scaffold Diffusion: Sparse Multi-Category Voxel Structure Generation with Discrete Diffusion NeurIPS 2025 Structured Probabilistic Inference & Generative Modeling Workshop
Title CN Scaffold Diffusion：基于离散扩散的稀疏多类别体素结构生成
Keywords 3D生成 · 体素结构 · 离散扩散 · 类不平衡 · 语言模型
Summary 本文提出Scaffold Diffusion，一种用于生成稀疏多类别3D体素结构的离散扩散模型。通过将体素视为token，利用语言模型结构生成空间上连贯的3D结构，即使在98%以上稀疏的数据上也能产生逼真结果。实验基于3D-Craft数据集中的Minecraft房屋结构。
Reason 该论文研究的是3D体素结构生成，属于3D视觉与生成模型领域，任务为Minecraft建筑生成，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了离散扩散语言模型处理空间结构，但其方法面向高度稀疏的3D体素数据，迁移到文档图像中的密集文本布局、OCR对齐或复杂排版理解存在本质差异，无法解决DIU的核心痛点如幻觉、细粒度定位或长文档推理。
Abstract Generating realistic sparse multi-category 3D voxel structures is difficult due to the cubic memory scaling of voxel structures and moreover the significant class imbalance caused by sparsity. We introduce Scaffold Diffusion, a generative model designed for sparse multi-category 3D voxel structures. By treating voxels as tokens, Scaffold Diffusion uses a discrete diffusion language model to generate 3D voxel structures. We show that discrete diffusion language models can be extended beyond inherently sequential domains such as text to generate spatially coherent 3D structures. We evaluate on Minecraft house structures from the 3D-Craft dataset and demonstrate that, unlike prior baselines and an auto-regressive formulation, Scaffold Diffusion produces realistic and coherent structures even when trained on data with over 98% sparsity. We provide an interactive viewer where readers can visualize generated samples and the generation process: https://scaffold.deepexploration.org/
Authors Justin Jung
Comment Accepted at NeurIPS 2025 Structured Probabilistic Inference & Generative Modeling Workshop
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
KOM: A Multi-Agent Artificial Intelligence System for Precision Management of Knee Osteoarthritis (KOA)
Title CN KOM：用于膝骨关节炎精准管理的多智能体人工智能系统
Keywords 多智能体系统 · 人工智能医疗 · 膝骨关节炎 · 个性化治疗 · 临床决策支持
Summary 本文提出了一种名为KOM的多智能体AI系统，用于自动化膝骨关节炎（KOA）的评估、风险预测与治疗方案制定。系统可辅助医生生成个性化管理计划，并在模拟实验中展现出比通用大模型更优的表现，且能缩短诊断时间并提升治疗质量。系统具有模块化架构，可能为其他慢性病管理提供参考。
Reason 该论文属于典型的垂直领域应用（医疗健康中的慢性病管理），尽管使用了多智能体架构，但其核心是将现有AI技术应用于特定医学场景，未提出可用于文档图像理解（DIU）的方法论创新。不涉及OCR、文档布局分析、表格识别或VLM在文档上的底层改进，无法迁移解决DIU中的幻觉、密集文本、复杂排版等核心问题。
Abstract Knee osteoarthritis (KOA) affects more than 600 million individuals globally and is associated with significant pain, functional impairment, and disability. While personalized multidisciplinary interventions have the potential to slow disease progression and enhance quality of life, they typically require substantial medical resources and expertise, making them difficult to implement in resource-limited settings. To address this challenge, we developed KOM, a multi-agent system designed to automate KOA evaluation, risk prediction, and treatment prescription. This system assists clinicians in performing essential tasks across the KOA care pathway and supports the generation of tailored management plans based on individual patient profiles, disease status, risk factors, and contraindications. In benchmark experiments, KOM demonstrated superior performance compared to several general-purpose large language models in imaging analysis and prescription generation. A randomized three-arm simulation study further revealed that collaboration between KOM and clinicians reduced total diagnostic and planning time by 38.5% and resulted in improved treatment quality compared to each approach used independently. These findings indicate that KOM could help facilitate automated KOA management and, when integrated into clinical workflows, has the potential to enhance care efficiency. The modular architecture of KOM may also offer valuable insights for developing AI-assisted management systems for other chronic conditions.
Authors Weizhi Liu, Xi Chen, Zekun Jiang, Liang Zhao, Kunyuan Jiang, Ruisi Tang, Li Wang, Mingke You, Hanyu Zhou, Hongyu Chen, Qiankun Xiong, Yong Nie, Kang Li, Jian Li
Categories Artificial Intelligence
0
Terminal Velocity Matching
Title CN 终端速度匹配
Keywords 扩散模型 · 流匹配 · 高效生成 · 图像生成 · Transformer架构
Summary 本文提出了终端速度匹配（TVM），一种广义的流匹配方法，支持高质量的一次或少数几步生成建模。TVM通过在终止时间而非初始时间正则化模型行为，实现了对数据分布与模型分布之间2-Wasserstein距离的上界控制。为适配Diffusion Transformer，作者引入了最小架构修改以实现单阶段稳定训练，并设计了支持Jacobian-Vector乘积反向传播的融合注意力核，提升计算效率。在ImageNet上取得了当前最优的FID指标。
Reason 该论文属于纯图像生成领域，提出了一种用于加速扩散模型生成过程的Flow Matching变体TVM，主要应用于ImageNet上的高保真图像生成。尽管其技术涉及训练稳定性与高效推理，但核心目标与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版解析或长文档推理等关键问题。属于典型的无关领域工作。
Abstract We propose Terminal Velocity Matching (TVM), a generalization of flow matching that enables high-fidelity one- and few-step generative modeling. TVM models the transition between any two diffusion timesteps and regularizes its behavior at its terminal time rather than at the initial time. We prove that TVM provides an upper bound on the $2$-Wasserstein distance between data and model distributions when the model is Lipschitz continuous. However, since Diffusion Transformers lack this property, we introduce minimal architectural changes that achieve stable, single-stage training. To make TVM efficient in practice, we develop a fused attention kernel that supports backward passes on Jacobian-Vector Products, which scale well with transformer architectures. On ImageNet-256x256, TVM achieves 3.29 FID with a single function evaluation (NFE) and 1.99 FID with 4 NFEs. It similarly achieves 4.32 1-NFE FID and 2.94 4-NFE FID on ImageNet-512x512, representing state-of-the-art performance for one/few-step models from scratch.
Authors Linqi Zhou, Mathias Parger, Ayaan Haque, Jiaming Song
Comment Code available at: https://github.com/lumalabs/tvm
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
GPU-Initiated Networking for NCCL
Title CN 用于NCCL的GPU发起的网络通信
Keywords GPU通信 · NCCL · 分布式训练 · GIN · MoE
Summary 本文介绍了NCCL 2.28中引入的GPU发起的网络通信（GIN）架构，旨在降低Mixture-of-Experts（MoE）等现代AI工作负载中的通信延迟。GIN支持设备端直接发起通信操作，减少CPU协调开销，并通过三层架构实现：主机端API、设备端API以及支持多种硬件的网络插件层。实验表明GIN可在保持NCCL统一运行时的同时提供低延迟通信能力。
Reason 该论文聚焦于GPU间通信的底层系统优化，属于AI基础设施和分布式训练系统领域，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。尽管其技术可能间接加速大规模VLM训练，但并未提出任何可用于解决OCR幻觉、复杂排版理解、细粒度定位或文档推理等DIU核心问题的方法论创新。
Abstract Modern AI workloads, especially Mixture-of-Experts (MoE) architectures, increasingly demand low-latency, fine-grained GPU-to-GPU communication with device-side control. Traditional GPU communication follows a host-initiated model, where the CPU orchestrates all communication operations - a characteristic of the CUDA runtime. Although robust for collective operations, applications requiring tight integration of computation and communication can benefit from device-initiated communication that eliminates CPU coordination overhead. NCCL 2.28 introduces the Device API with three operation modes: Load/Store Accessible (LSA) for NVLink/PCIe, Multimem for NVLink SHARP, and GPU-Initiated Networking (GIN) for network RDMA. This paper presents the GIN architecture, design, semantics, and highlights its impact on MoE communication. GIN builds on a three-layer architecture: i) NCCL Core host-side APIs for device communicator setup and collective memory window registration; ii) Device-side APIs for remote memory operations callable from CUDA kernels; and iii) A network plugin architecture with dual semantics (GPUDirect Async Kernel-Initiated and Proxy) for broad hardware support. The GPUDirect Async Kernel-Initiated backend leverages DOCA GPUNetIO for direct GPU-to-NIC communication, while the Proxy backend provides equivalent functionality via lock-free GPU-to-CPU queues over standard RDMA networks. We demonstrate GIN's practicality through integration with DeepEP, an MoE communication library. Comprehensive benchmarking shows that GIN provides device-initiated communication within NCCL's unified runtime, combining low-latency operations with NCCL's collective algorithms and production infrastructure.
Authors Khaled Hamidouche, John Bachan, Pak Markthub, Peter-Jan Gootzen, Elena Agostini, Sylvain Jeaugey, Aamir Shafi, Georgios Theodorakis, Manjunath Gorentla Venkata
Comment 13 pages, 9 figures, 3 tables
Categories Artificial Intelligence
0
Memory Self-Regeneration: Uncovering Hidden Knowledge in Unlearned Models
Title CN 记忆自再生：揭示未学习模型中的隐藏知识
Keywords 机器遗忘 · 知识恢复 · 对抗攻击 · 文本到图像模型 · 模型安全性
Summary 本文提出“记忆自再生”任务，旨在研究在机器遗忘过程中，模型是否仍能通过对抗性提示恢复已被删除的知识。作者提出了MemoRa策略以支持丢失知识的有效恢复，并强调知识检索的鲁棒性应作为评估遗忘技术的重要指标。研究发现遗忘可分为短期（易恢复）和长期（难恢复）两种类型。
Reason 该论文研究的是机器遗忘（machine unlearning）中的知识恢复问题，属于AI安全与伦理范畴，核心关注如何让模型‘忘记’特定概念后仍能被攻击性提示词还原，进而提出评估遗忘鲁棒性的方法。其研究任务与文档图像理解（DIU）无直接关联，既不涉及文档结构、文本识别、布局分析，也无法迁移用于解决OCR幻觉、细粒度定位或长文档推理等DIU核心痛点。尽管涉及‘知识恢复’，但场景和模态聚焦于文本到图像生成模型的安全漏洞，与文档多模态理解无关。
Abstract The impressive capability of modern text-to-image models to generate realistic visuals has come with a serious drawback: they can be misused to create harmful, deceptive or unlawful content. This has accelerated the push for machine unlearning. This new field seeks to selectively remove specific knowledge from a model's training data without causing a drop in its overall performance. However, it turns out that actually forgetting a given concept is an extremely difficult task. Models exposed to attacks using adversarial prompts show the ability to generate so-called unlearned concepts, which can be not only harmful but also illegal. In this paper, we present considerations regarding the ability of models to forget and recall knowledge, introducing the Memory Self-Regeneration task. Furthermore, we present MemoRa strategy, which we consider to be a regenerative approach supporting the effective recovery of previously lost knowledge. Moreover, we propose that robustness in knowledge retrieval is a crucial yet underexplored evaluation measure for developing more robust and effective unlearning techniques. Finally, we demonstrate that forgetting occurs in two distinct ways: short-term, where concepts can be quickly recalled, and long-term, where recovery is more challenging. Code is available at https://gmum.github.io/MemoRa/.
Authors Agnieszka Polowczyk, Alicja Polowczyk, Joanna Waczyńska, Piotr Borycki, Przemysław Spurek
Categories Artificial Intelligence
0
Advancing Limited-Angle CT Reconstruction Through Diffusion-Based Sinogram Completion 2025 IEEE International Conference on Image Processing (Oral)
Title CN 通过基于扩散的正弦图补全推进有限角度CT重建
Keywords 有限角度CT · 扩散模型 · 正弦图补全 · MR-SDEs · 医学图像重建
Summary 本文提出一种基于扩散模型（MR-SDEs）的有限角度CT重建方法，通过在投影域进行正弦图补全来恢复缺失的角度信息，并结合知识蒸馏和伪逆约束实现单步快速补全。后续后处理模块进一步优化重建图像，有效抑制伪影并保留结构细节。实验表明该方法在感知质量和保真度上达到SOTA水平。
Reason 该论文研究的是医学成像中的有限角度CT重建问题，属于医疗图像重建领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心技术是基于扩散模型的正弦图补全，应用于投影数据修复，不涉及文本识别、布局分析、多模态文档理解或OCR相关挑战。无法迁移用于解决文档图像中的幻觉、复杂排版或长文档推理等问题。
Abstract Limited Angle Computed Tomography (LACT) often faces significant challenges due to missing angular information. Unlike previous methods that operate in the image domain, we propose a new method that focuses on sinogram inpainting. We leverage MR-SDEs, a variant of diffusion models that characterize the diffusion process with mean-reverting stochastic differential equations, to fill in missing angular data at the projection level. Furthermore, by combining distillation with constraining the output of the model using the pseudo-inverse of the inpainting matrix, the diffusion process is accelerated and done in a step, enabling efficient and accurate sinogram completion. A subsequent post-processing module back-projects the inpainted sinogram into the image domain and further refines the reconstruction, effectively suppressing artifacts while preserving critical structural details. Quantitative experimental results demonstrate that the proposed method achieves state-of-the-art performance in both perceptual and fidelity quality, offering a promising solution for LACT reconstruction in scientific and clinical applications.
Authors Jiaqi Guo, Santiago Lopez-Tapia, Aggelos K. Katsaggelos
Comment Accepted at the 2025 IEEE International Conference on Image Processing (Oral)
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Leveraging Foundation Models for Histological Grading in Cutaneous Squamous Cell Carcinoma using PathFMTools Proceedings of the 5th Machine Learning for Health (ML4H) Symposium (2025)
Title CN 利用基础模型通过PathFMTools进行皮肤鳞状细胞癌的组织学分级
Keywords 计算病理学 · 基础模型 · 组织学分级 · Whole-Slide Image · 工具包
Summary 本文提出了一个名为PathFMTools的轻量级Python工具包，用于高效执行、分析和可视化病理学基础模型，在皮肤鳞状细胞癌的组织学分级任务上评估了CONCH和MUSK两种先进视觉语言模型的表现，并比较了多种适配策略。研究表明，基于基础模型嵌入训练小型专业模型具有临床应用潜力。
Reason 该论文属于医学图像分析领域的下游应用，聚焦于皮肤鳞状细胞癌（cSCC）的组织学分级任务，使用现有视觉语言基础模型（如CONCH、MUSK）并开发工具包PathFMTools进行适配与可视化。其核心是将已有VLM应用于特定医疗场景，并未提出针对文档图像理解（DIU）的方法论创新。尽管涉及WSI高分辨率图像处理，但方法不具备向DocVQA、布局分析、表格识别等DIU任务迁移的通用性，属于典型的‘平行下游应用’，违反负面清单第1条。
Abstract Despite the promise of computational pathology foundation models, adapting them to specific clinical tasks remains challenging due to the complexity of whole-slide image (WSI) processing, the opacity of learned features, and the wide range of potential adaptation strategies. To address these challenges, we introduce PathFMTools, a lightweight, extensible Python package that enables efficient execution, analysis, and visualization of pathology foundation models. We use this tool to interface with and evaluate two state-of-the-art vision-language foundation models, CONCH and MUSK, on the task of histological grading in cutaneous squamous cell carcinoma (cSCC), a critical criterion that informs cSCC staging and patient management. Using a cohort of 440 cSCC H&E WSIs, we benchmark multiple adaptation strategies, demonstrating trade-offs across prediction approaches and validating the potential of using foundation model embeddings to train small specialist models. These findings underscore the promise of pathology foundation models for real-world clinical applications, with PathFMTools enabling efficient analysis and validation.
Authors Abdul Rahman Diab, Emily E. Karn, Renchin Wu, Emily S. Ruiz, William Lotter
Comment Proceedings of the 5th Machine Learning for Health (ML4H) Symposium (2025)
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Prompt Fencing: A Cryptographic Approach to Establishing Security Boundaries in Large Language Model Prompts
Title CN 提示围栏：一种通过密码学方法在大语言模型提示中建立安全边界的方法
Keywords 提示注入防御 · 大语言模型安全 · 密码学认证 · 信任边界 · 提示工程
Summary 本文提出Prompt Fencing，一种通过密码学手段为大语言模型的提示片段添加可信元数据（如信任等级和内容类型）的技术，以区分受信指令与不可信输入，从而防御提示注入攻击。实验显示该方法可在模拟环境下将攻击成功率从86.7%降至0%。系统具有平台无关性，可作为现有LLM之上的安全层部署。
Reason 该论文聚焦于LLM的安全性问题，特别是防御提示注入攻击（prompt injection），属于纯NLP安全与对齐领域。尽管其技术涉及提示工程中的信任边界控制，但并未处理任何文档图像理解（DIU）的核心挑战，如OCR幻觉、复杂排版解析、视觉-文本对齐或高分辨率文档建模。其方法无法直接迁移用于解决文档理解中的视觉幻觉、细粒度定位或多模态推理等问题。
Abstract Large Language Models (LLMs) remain vulnerable to prompt injection attacks, representing the most significant security threat in production deployments. We present Prompt Fencing, a novel architectural approach that applies cryptographic authentication and data architecture principles to establish explicit security boundaries within LLM prompts. Our approach decorates prompt segments with cryptographically signed metadata including trust ratings and content types, enabling LLMs to distinguish between trusted instructions and untrusted content. While current LLMs lack native fence awareness, we demonstrate that simulated awareness through prompt instructions achieved complete prevention of injection attacks in our experiments, reducing success rates from 86.7% (260/300 successful attacks) to 0% (0/300 successful attacks) across 300 test cases with two leading LLM providers. We implement a proof-of-concept fence generation and verification pipeline with a total overhead of 0.224 seconds (0.130s for fence generation, 0.094s for validation) across 100 samples. Our approach is platform-agnostic and can be incrementally deployed as a security layer above existing LLM infrastructure, with the expectation that future models will be trained with native fence awareness for optimal security.
Authors Steven Peh
Comment 44 pages, 1 figure
Categories Artificial Intelligence
0
RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformers
Title CN RoPECraft：基于轨迹引导的RoPE优化在扩散变换器上实现无需训练的运动迁移
Keywords 视频生成 · 运动迁移 · 扩散变换器 · RoPE · 光流
Summary 本文提出RoPECraft，一种无需训练的视频运动迁移方法，通过修改扩散Transformer中的旋转位置编码（RoPE）来注入运动信息。利用参考视频的密集光流提取运动偏移，并在去噪过程中通过轨迹对齐优化RoPE，同时引入基于傅里叶相位的正则化项以保持文本一致性并抑制伪影。实验表明该方法在定性和定量指标上均优于现有方法。
Reason 该论文研究的是视频运动迁移任务，属于视频生成领域，其核心技术围绕光学流、旋转位置编码（RoPE）优化和相位正则化展开，主要应用于扩散Transformer的视频生成过程。与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联，且无法迁移到解决OCR幻觉、复杂排版或长文档推理等DIU痛点。
Abstract We propose RoPECraft, a training-free video motion transfer method for diffusion transformers that operates solely by modifying their rotary positional embeddings (RoPE). We first extract dense optical flow from a reference video, and utilize the resulting motion offsets to warp the complex-exponential tensors of RoPE, effectively encoding motion into the generation process. These embeddings are then further optimized during denoising time steps via trajectory alignment between the predicted and target velocities using a flow-matching objective. To keep the output faithful to the text prompt and prevent duplicate generations, we incorporate a regularization term based on the phase components of the reference video's Fourier transform, projecting the phase angles onto a smooth manifold to suppress high-frequency artifacts. Experiments on benchmarks reveal that RoPECraft outperforms all recently published methods, both qualitatively and quantitatively.
Authors Ahmet Berke Gokmen, Yigit Ekin, Bahri Batuhan Bilecen, Aysegul Dundar
Comment https://berkegokmen1.github.io/RoPECraft/
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
CrypTorch: PyTorch-based Auto-tuning Compiler for Machine Learning with Multi-party Computation
Title CN CrypTorch：基于PyTorch的多方计算机器学习自动调优编译器
Keywords MPC · 隐私计算 · 编译器 · 自动调优 · PyTorch
Summary 本文提出CrypTorch，一个构建在PyTorch 2编译器之上的自动调优编译器，用于优化基于多方计算（MPC）的机器学习推理过程。通过解耦近似函数与MPC运行时，并提供可编程接口以灵活替换数学运算的近似方法，CrypTorch能在精度与性能之间自动权衡，显著提升MPC下ML模型的执行效率。
Reason 该论文研究的是基于多方计算（MPC）的机器学习隐私计算框架，属于安全与编译优化领域，与文档图像理解（DIU）或视觉语言模型（VLM）在任务、数据模态和核心问题上均无直接关联。尽管涉及ML系统优化，但其目标是解决隐私保护下的计算效率问题，而非OCR幻觉、复杂排版理解、细粒度定位等DIU痛点，无法迁移用于提升文档理解性能。
Abstract Machine learning (ML) involves private data and proprietary model parameters. MPC-based ML allows multiple parties to collaboratively run an ML workload without sharing their private data or model parameters using multi-party computing (MPC). Because MPC cannot natively run ML operations such as Softmax or GELU, existing frameworks use different approximations. Our study shows that, on a well-optimized framework, these approximations often become the dominating bottleneck. Popular approximations are often insufficiently accurate or unnecessarily slow, and these issues are hard to identify and fix in existing frameworks. To tackle this issue, we propose a compiler for MPC-based ML, CrypTorch. CrypTorch disentangles these approximations with the rest of the MPC runtime, allows easily adding new approximations through its programming interface, and automatically selects approximations to maximize both performance and accuracy. Built as an extension to PyTorch 2's compiler, we show that CrypTorch's auto-tuning alone provides 1.20--1.7$\times$ immediate speedup without sacrificing accuracy, and 1.31--1.8$\times$ speedup when some accuracy degradation is allowed, compared to our well-optimized baseline. Combined with better engineering and adoption of state-of-the-art practices, the entire framework brings 3.22--8.6$\times$ end-to-end speedup compared to the popular framework, CrypTen.
Authors Jinyu Liu, Gang Tan, Kiwan Maeng
Comment 28 pages, 17 figures. Submitted to PLDI 2026
Categories Artificial Intelligence
0
The Alexander-Hirschowitz theorem for neurovarieties
Title CN 神经簇的Alexander-Hirschowitz定理
Keywords 神经簇 · 代数几何 · 多项式神经网络 · 非缺陷性 · 全局可识别性
Summary 本文从代数几何角度研究了多项式神经网络的神经簇（neurovarieties）性质，证明了在单输出情况下其维度达到预期值的充分必要条件，并推导出多输出结构的非缺陷性和全局可识别性。工作属于基础理论范畴，不涉及具体应用或模型架构设计。
Reason 该论文研究的是多项式神经网络的代数几何性质，属于理论数学与神经网络的交叉分析，未涉及文档图像理解、多模态建模、视觉表示学习或任何DIU相关任务。内容与OCR、VLM、文档推理、布局分析等完全无关，属于纯理论AI研究，无法迁移解决文档理解中的幻觉、复杂排版或长文档推理等实际问题。
Abstract We study neurovarieties for polynomial neural networks and fully characterize when they attain the expected dimension in the single-output case. As consequences, we establish non-defectiveness and global identifiability for multi-output architectures.
Authors A. Massarenti, M. Mella
Comment 21 pages
Categories Artificial Intelligence
0
IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants NeurIPS 2025 D&B Track
Title CN IndEgo：面向工业场景与协作工作的第一视角助手数据集
Keywords 多模态数据集 · 第一视角视觉 · 工业应用 · 协作任务 · 错误检测
Summary 本文提出了IndEgo，一个包含3460段第一视角和1092段外部视角视频的多模态工业任务数据集，涵盖组装、物流、检修等场景，并提供动作标注、错误标注、叙述文本等多种注释。数据集强调双人协作任务，支持程序性与非程序性任务理解、错误检测和基于推理的问答。实验表明现有最先进多模态模型在该数据集上仍有挑战。
Reason 该论文聚焦于工业场景下的第一视角（egocentric）多模态理解，任务包括装配、物流、维修等，其核心是构建用于人机协作的视频理解数据集。尽管涉及多模态和问答，但其内容与文档图像理解（DIU）的核心任务如DocVQA、表格识别、布局分析等无直接关联。应用场景为真实世界视频而非文档图像，技术重点在于动作理解、错误检测和协作推理，无法迁移解决OCR幻觉、密集文本处理或复杂排版等DIU痛点。
Abstract We introduce IndEgo, a multimodal egocentric and exocentric dataset addressing common industrial tasks, including assembly/disassembly, logistics and organisation, inspection and repair, woodworking, and others. The dataset contains 3,460 egocentric recordings (approximately 197 hours), along with 1,092 exocentric recordings (approximately 97 hours). A key focus of the dataset is collaborative work, where two workers jointly perform cognitively and physically intensive tasks. The egocentric recordings include rich multimodal data and added context via eye gaze, narration, sound, motion, and others. We provide detailed annotations (actions, summaries, mistake annotations, narrations), metadata, processed outputs (eye gaze, hand pose, semi-dense point cloud), and benchmarks on procedural and non-procedural task understanding, Mistake Detection, and reasoning-based Question Answering. Baseline evaluations for Mistake Detection, Question Answering and collaborative task understanding show that the dataset presents a challenge for the state-of-the-art multimodal models. Our dataset is available at: https://huggingface.co/datasets/FraunhoferIPK/IndEgo
Authors Vivek Chavan, Yasmina Imgrund, Tung Dao, Sanwantri Bai, Bosong Wang, Ze Lu, Oliver Heimann, Jörg Krüger
Comment Accepted to NeurIPS 2025 D&B Track. Project Page: https://indego-dataset.github.io/
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
HeaRT: A Hierarchical Circuit Reasoning Tree-Based Agentic Framework for AMS Design Optimization
Title CN HeaRT：一种基于分层电路推理树的智能体框架用于AMS设计优化
Keywords AMS设计优化 · 智能体框架 · 推理树 · 电路设计自动化 · AI驱动设计
Summary 本文提出HeaRT，一种基于分层推理树的智能体框架，用于自动化模拟集成电路（AMS）的设计优化。该方法在电路尺寸调整和拓扑结构适应任务中表现出比现有方法快3倍以上的收敛速度，同时保持高推理准确率和低计算开销。然而，其应用领域为电子设计自动化（EDA），不属于文档理解或多模态视觉语言任务范畴。
Reason 该论文属于具身智能/机器人或AI for Design领域，研究的是模拟集成电路（AMS）设计优化问题，与文档图像理解（DIU）、DocVQA、OCR、多模态文档建模等任务完全无关。尽管其提出了一个基于推理树的Agent框架，但应用场景和数据模态（电路拓扑结构、参数优化）与文档视觉理解无交集，无法迁移解决OCR幻觉、复杂排版、长文档推理等核心痛点。
Abstract Conventional AI-driven AMS design automation algorithms remain constrained by their reliance on high-quality datasets to capture underlying circuit behavior, coupled with poor transferability across architectures, and a lack of adaptive mechanisms. This work proposes HeaRT, a foundational reasoning engine for automation loops and a first step toward intelligent, adaptive, human-style design optimization. HeaRT consistently demonstrates reasoning accuracy >97% and Pass@1 performance >98% across our 40-circuit benchmark repository, even as circuit complexity increases, while operating at <0.5x real-time token budget of SOTA baselines. Our experiments show that HeaRT yields >3x faster convergence in both sizing and topology design adaptation tasks across diverse optimization approaches, while preserving prior design intent.
Authors Souradip Poddar, Chia-Tung Ho, Ziming Wei, Weidong Cao, Haoxing Ren, David Z. Pan
Categories Artificial Intelligence
0
Ontology-Aware RAG for Improved Question-Answering in Cybersecurity Education 2025 IEEE International Conference on Big Data (IEEE BigData 2025)
Title CN 面向网络安全教育的本体感知RAG以改进问答
Keywords RAG · 知识图谱 · 本体 · 问答系统 · 教育AI
Summary 本文提出CyberRAG，一种结合检索增强生成与知识图谱本体的问答系统，用于提升网络安全教育中AI问答的准确性和可靠性。通过从知识库中检索权威文档并利用本体对答案进行验证，减少大模型的幻觉和误用风险。实验表明该方法能生成更符合领域知识的答案。
Reason 该论文属于典型的‘平行下游应用’，将现有的RAG和知识图谱技术应用于网络安全教育这一特定领域。尽管其在减少幻觉方面有所设计（通过本体验证），但方法本身并未提出适用于文档图像理解（DIU）或视觉语言模型（VLM）的新架构、新训练策略或可迁移的底层机制。其核心技术（检索增强+知识图谱校验）无法直接解决DIU中的OCR幻觉、复杂排版解析、细粒度定位等核心问题。
Abstract Integrating AI into education has the potential to transform the teaching of science and technology courses, particularly in the field of cybersecurity. AI-driven question-answering (QA) systems can actively manage uncertainty in cybersecurity problem-solving, offering interactive, inquiry-based learning experiences. Recently, Large language models (LLMs) have gained prominence in AI-driven QA systems, enabling advanced language understanding and user engagement. However, they face challenges like hallucinations and limited domain-specific knowledge, which reduce their reliability in educational settings. To address these challenges, we propose CyberRAG, an ontology-aware retrieval-augmented generation (RAG) approach for developing a reliable and safe QA system in cybersecurity education. CyberRAG employs a two-step approach: first, it augments the domain-specific knowledge by retrieving validated cybersecurity documents from a knowledge base to enhance the relevance and accuracy of the response. Second, it mitigates hallucinations and misuse by integrating a knowledge graph ontology to validate the final answer. Comprehensive experiments on publicly available datasets reveal that CyberRAG delivers accurate, reliable responses aligned with domain knowledge, demonstrating the potential of AI tools to enhance education.
Authors Chengshuai Zhao, Garima Agrawal, Fan Zhang, Tharindu Kumarage, Zhen Tan, Yuli Deng, Ying-Chih Chen, Huan Liu
Comment Accepted by the 2025 IEEE International Conference on Big Data (IEEE BigData 2025)
Categories Artificial Intelligence
0
Large language models replicate and predict human cooperation across experiments in game theory
Title CN 大语言模型在博弈论实验中复现和预测人类合作行为
Keywords 大语言模型 · 人类行为模拟 · 博弈论 · 数字孪生 · 社会决策
Summary 本文提出一种数字孪生框架，利用大语言模型复现博弈论实验中的人类合作行为，并通过系统性提示方法评估不同LLM在人类行为模拟上的表现。结果显示部分模型能高保真地复制人类合作模式，并可生成新的可验证假设。研究聚焦于社会行为仿真，不涉及视觉或文档内容理解。
Reason 该论文研究的是LLM在社会行为模拟中的应用，属于纯NLP与社会科学交叉领域，关注人类决策与博弈行为的建模，并未涉及任何视觉输入、文档结构或图文多模态理解。与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等完全无关，也无法迁移解决OCR幻觉、密集文本处理、复杂排版解析等关键痛点。
Abstract Large language models (LLMs) are increasingly used both to make decisions in domains such as health, education and law, and to simulate human behavior. Yet how closely LLMs mirror actual human decision-making remains poorly understood. This gap is critical: misalignment could produce harmful outcomes in practical applications, while failure to replicate human behavior renders LLMs ineffective for social simulations. Here, we address this gap by developing a digital twin of game-theoretic experiments and introducing a systematic prompting and probing framework for machine-behavioral evaluation. Testing three open-source models (Llama, Mistral and Qwen), we find that Llama reproduces human cooperation patterns with high fidelity, capturing human deviations from rational choice theory, while Qwen aligns closely with Nash equilibrium predictions. Notably, we achieved population-level behavioral replication without persona-based prompting, simplifying the simulation process. Extending beyond the original human-tested games, we generate and preregister testable hypotheses for novel game configurations outside the original parameter grid. Our findings demonstrate that appropriately calibrated LLMs can replicate aggregate human behavioral patterns and enable systematic exploration of unexplored experimental spaces, offering a complementary approach to traditional research in the social and behavioral sciences that generates new empirical predictions about human social decision-making.
Authors Andrea Cera Palatsi, Samuel Martin-Gutierrez, Ana S. Cardenal, Max Pellert
Categories Artificial Intelligence, Computation and Language
0
What is Implementation Science; and Why It Matters for Bridging the Artificial Intelligence Innovation-to-Application Gap in Medical Imaging
Title CN 什么是实施科学及其在弥合医学影像人工智能创新与应用差距中的重要性
Keywords 实施科学 · 医学影像 · 人工智能部署 · 人机交互 · 知识转化
Summary 论文探讨了实施科学如何帮助缩短人工智能技术从研究到临床实践的转化周期，分析了医学影像AI在实际部署中面临的基础设施、教育和文化障碍，并提倡采用混合研究设计、知识转化框架和公平导向的共同创造策略来提升AI系统的可持续性和可用性。
Reason 该论文讨论的是实施科学（Implementation Science）在医学影像AI临床落地中的应用，属于典型的领域应用型研究，不涉及任何文档图像理解（DIU）或视觉语言模型（VLM）的技术创新。其内容聚焦于医疗AI的部署障碍、利益相关者参与和人机交互框架，属于‘平行下游应用’范畴，且未提出可迁移至DIU领域的底层方法。
Abstract The transformative potential of artificial intelligence (AI) in medical Imaging (MI) is well recognized. Yet despite promising reports in research settings, many AI tools fail to achieve clinical adoption in practice. In fact, more generally, there is a documented 17-year average delay between evidence generation and implementation of a technology. Implementation science (IS) may provide a practical, evidence-based framework to bridge the gap between AI development and real-world clinical imaging use, to shorten this lag through systematic frameworks, strategies, and hybrid research designs. We outline challenges specific to AI adoption in MI workflows, including infrastructural, educational, and cultural barriers. We highlight the complementary roles of effectiveness research and implementation research, emphasizing hybrid study designs and the role of integrated KT (iKT), stakeholder engagement, and equity-focused co-creation in designing sustainable and generalizable solutions. We discuss integration of Human-Computer Interaction (HCI) frameworks in MI towards usable AI. Adopting IS is not only a methodological advancement; it is a strategic imperative for accelerating translation of innovation into improved patient outcomes.
Authors Ahmad Fayaz-Bakhsh, Janice Tania, Syaheerah Lebai Lutfi, Abhinav K. Jha, Arman Rahmim
Categories Artificial Intelligence
0
Accuracy and Efficiency Trade-Offs in LLM-Based Malware Detection and Explanation: A Comparative Study of Parameter Tuning vs. Full Fine-Tuning IEEE Big Data 2025
Title CN 基于大语言模型的恶意软件检测与解释中的准确性和效率权衡：参数调优与全量微调的比较研究
Keywords LoRA · 大语言模型 · 恶意软件检测 · 解释生成 · 参数高效微调
Summary 本文研究了LoRA微调与全量微调在大语言模型生成恶意软件分类解释方面的性能差异，发现中等规模的LoRA配置可在显著降低模型大小和训练时间的同时保持接近全量微调的解释质量。尽管在资源受限场景下具有部署优势，但该工作聚焦于网络安全领域的特定应用，缺乏对文档图像理解领域的方法论贡献。
Reason 该论文属于平行下游应用，将LoRA与全量微调应用于恶意软件检测与解释任务，未提出任何文档图像理解（DIU）相关的技术方法或架构创新。其研究内容与文档视觉问答、布局分析、表格识别等DIU核心任务无关，且未提供可迁移至DIU领域的底层技术创新（如视觉表示学习、高分辨率处理、多模态对齐优化等）。
Abstract This study examines whether Low-Rank Adaptation (LoRA) fine-tuned Large Language Models (LLMs) can approximate the performance of fully fine-tuned models in generating human-interpretable decisions and explanations for malware classification. Achieving trustworthy malware detection, particularly when LLMs are involved, remains a significant challenge. We developed an evaluation framework using Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation (ROUGE), and Semantic Similarity Metrics to benchmark explanation quality across five LoRA configurations and a fully fine-tuned baseline. Results indicate that full fine-tuning achieves the highest overall scores, with BLEU and ROUGE improvements of up to 10% over LoRA variants. However, mid-range LoRA models deliver competitive performance exceeding full fine-tuning on two metrics while reducing model size by approximately 81% and training time by over 80% on a LoRA model with 15.5% trainable parameters. These findings demonstrate that LoRA offers a practical balance of interpretability and resource efficiency, enabling deployment in resource-constrained environments without sacrificing explanation quality. By providing feature-driven natural language explanations for malware classifications, this approach enhances transparency, analyst confidence, and operational scalability in malware detection systems.
Authors Stephen C. Gravereaux, Sheikh Rabiul Islam
Comment Accepted in IEEE Big Data 2025
Categories Artificial Intelligence
0
Synthetic Data: AI's New Weapon Against Android Malware JBCS
Title CN 合成数据：人工智能对抗安卓恶意软件的新武器
Keywords 合成数据 · 恶意软件检测 · 生成对抗网络 · Android安全 · 机器学习
Summary 本文提出了一种名为MalSynGen的恶意软件合成数据生成方法，基于条件生成对抗网络（cGAN）生成具有真实数据统计特性的合成表格数据，用于提升安卓恶意软件分类器的性能。实验表明该方法在多个数据集上具有良好泛化能力，可缓解恶意软件检测中数据过时和质量低下的问题。
Reason 该论文属于平行下游应用，研究的是利用生成对抗网络生成合成数据以提升安卓恶意软件分类性能，属于网络安全领域的具体应用，与文档图像理解（DIU）或视觉语言模型在文档上的核心方法创新无关。尽管使用了生成模型，但其方法和任务均无法迁移到解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract The ever-increasing number of Android devices and the accelerated evolution of malware, reaching over 35 million samples by 2024, highlight the critical importance of effective detection methods. Attackers are now using Artificial Intelligence to create sophisticated malware variations that can easily evade traditional detection techniques. Although machine learning has shown promise in malware classification, its success relies heavily on the availability of up-to-date, high-quality datasets. The scarcity and high cost of obtaining and labeling real malware samples presents significant challenges in developing robust detection models. In this paper, we propose MalSynGen, a Malware Synthetic Data Generation methodology that uses a conditional Generative Adversarial Network (cGAN) to generate synthetic tabular data. This data preserves the statistical properties of real-world data and improves the performance of Android malware classifiers. We evaluated the effectiveness of this approach using various datasets and metrics that assess the fidelity of the generated data, its utility in classification, and the computational efficiency of the process. Our experiments demonstrate that MalSynGen can generalize across different datasets, providing a viable solution to address the issues of obsolescence and low quality data in malware detection.
Authors Angelo Gaspar Diniz Nogueira, Kayua Oleques Paim, Hendrio Bragança, Rodrigo Brandão Mansilha, Diego Kreutz
Comment 23 pages, 18 figures, 8 tables. Accepted for publication at the JBCS
Categories Artificial Intelligence
0
Robot-Powered Data Flywheels: Deploying Robots in the Wild for Continual Data Collection and Foundation Model Adaptation
Title CN 机器人驱动的数据飞轮：在真实环境中部署机器人以实现持续数据收集和基础模型适应
Keywords 机器人 · 基础模型 · 数据飞轮 · 多语言OCR · 视觉语言模型
Summary 本文提出了“机器人驱动的数据飞轮”框架，通过在东亚图书馆部署名为Scanford的移动操作机器人，自动扫描书架、识别图书，并利用图书馆目录进行无监督标注，从而收集真实世界中的多语言图文数据。这些数据被用于微调视觉语言模型，在特定领域内将图书识别准确率从32.0%提升至71.8%，并在相邻领域的多语言OCR任务上取得显著增益。尽管结果展示了持续学习和现实数据采集的价值，但其重点在于机器人系统的应用而非文档理解算法本身的突破。
Reason 该论文属于具身智能与机器人领域，研究内容为利用机器人在真实图书馆环境中扫描书籍并收集多语言文本数据，用于改进视觉语言模型的下游表现。虽然涉及OCR和多语言文本识别，但其核心贡献在于机器人系统部署与数据飞轮框架，属于典型的‘平行应用’与具身智能范畴，未提出可迁移至文档图像理解（DIU）的底层方法创新。且任务场景聚焦于机器人自主执行与现实世界交互，与文档理解中的OCR幻觉、复杂排版、长文档推理等核心痛点无直接关联。
Abstract Foundation models (FM) have unlocked powerful zero-shot capabilities in vision and language, yet their reliance on internet pretraining data leaves them brittle in unstructured, real-world settings. The messy, real-world data encountered during deployment (e.g. occluded or multilingual text) remains massively underrepresented in existing corpora. Robots, as embodied agents, are uniquely positioned to close this gap: they can act in physical environments to collect large-scale, real-world data that enriches FM training with precisely the examples current models lack. We introduce the Robot-Powered Data Flywheel, a framework that transforms robots from FM consumers into data generators. By deploying robots equipped with FMs in the wild, we enable a virtuous cycle: robots perform useful tasks while collecting real-world data that improves both domain-specific adaptation and domain-adjacent generalization. We instantiate this framework with Scanford, a mobile manipulator deployed in the East Asia Library for 2 weeks. Scanford autonomously scans shelves, identifies books using a vision-language model (VLM), and leverages the library catalog to label images without human annotation. This deployment both aids librarians and produces a dataset to finetune the underlying VLM, improving performance on the domain-specific in-the-wild library setting and on domain-adjacent multilingual OCR benchmarks. Using data collected from 2103 shelves, Scanford improves VLM performance on book identification from 32.0% to 71.8% and boosts domain-adjacent multilingual OCR from 24.8% to 46.6% (English) and 30.8% to 38.0% (Chinese), while saving an ~18.7 hrs of human time. These results highlight how robot-powered data flywheels can both reduce human effort in real deployments and unlock new pathways for continually adapting FMs to the messiness of reality. More details are at: https://scanford-robot.github.io
Authors Jennifer Grannen, Michelle Pan, Kenneth Llontop, Cherie Ho, Mark Zolotas, Jeannette Bohg, Dorsa Sadigh
Categories Artificial Intelligence
0
IRSDA: An Agent-Orchestrated Framework for Enterprise Intrusion Response
Title CN IRSDA：一种用于企业入侵响应的代理协同框架
Keywords 入侵响应 · 代理框架 · MAPE-K · 自适应系统 · 网络安全
Summary 本文提出了IRSDA，一个基于代理的企业级入侵响应框架，结合自适应自治计算系统与MAPE-K环，实现策略合规的自主网络防御。系统利用知识驱动架构和检索机制进行实时、分区感知的决策，并在微服务环境中验证了其自动化遏制、合规执行和可追溯输出的能力。
Reason 该论文属于网络安全领域的特定应用，聚焦于入侵检测与响应的自动化决策流程，使用了MAPE-K环和自适应计算系统。尽管采用了AI推理和知识引导机制，但其核心是安全领域的垂直应用，并未提出可迁移至文档图像理解（DIU）的方法论创新。不涉及OCR、视觉表示、多模态文档理解或VLM架构改进，属于典型的‘平行下游应用’。
Abstract Modern enterprise systems face escalating cyber threats that are increasingly dynamic, distributed, and multi-stage in nature. Traditional intrusion detection and response systems often rely on static rules and manual workflows, which limit their ability to respond with the speed and precision required in high-stakes environments. To address these challenges, we present the Intrusion Response System Digital Assistant (IRSDA), an agent-based framework designed to deliver autonomous and policy-compliant cyber defense. IRSDA combines Self-Adaptive Autonomic Computing Systems (SA-ACS) with the Knowledge guided Monitor, Analyze, Plan, and Execute (MAPE-K) loop to support real-time, partition-aware decision-making across enterprise infrastructure. IRSDA incorporates a knowledge-driven architecture that integrates contextual information with AI-based reasoning to support system-guided intrusion response. The framework leverages retrieval mechanisms and structured representations to inform decision-making while maintaining alignment with operational policies. We assess the system using a representative real-world microservices application, demonstrating its ability to automate containment, enforce compliance, and provide traceable outputs for security analyst interpretation. This work outlines a modular and agent-driven approach to cyber defense that emphasizes explainability, system-state awareness, and operational control in intrusion response.
Authors Damodar Panigrahi, Raj Patel, Shaswata Mitra, Sudip Mittal, Shahram Rahimi
Comment 10 pages, 4 figures
Categories Artificial Intelligence
0
On the Utility of Foundation Models for Fast MRI: Vision-Language-Guided Image Reconstruction
Title CN 基础模型在快速MRI中的效用：视觉-语言引导的图像重建
Keywords 视觉-语言模型 · MRI重建 · 语义先验 · 对比学习 · 图像生成
Summary 本文提出一种基于视觉-语言基础模型的语义分布引导框架，用于提升欠采样MRI的重建质量。通过对比学习目标将重建图像的表示与高维语义分布对齐，从而引入超越传统先验的上下文信息。实验表明，结合图像或图文辅助信息可改善解剖结构保留和感知质量。然而，该工作属于医学成像领域，不涉及文档理解任务。
Reason 该论文研究的是利用视觉-语言模型辅助欠采样MRI图像重建，属于医学影像重建领域，核心任务为图像生成与信号恢复，与文档图像理解（DIU）或DocVQA无直接关联。尽管使用了VLM，但其方法聚焦于医学图像的语义分布对齐，无法迁移解决OCR幻觉、文本密集识别、复杂版面分析或长文档推理等DIU核心痛点。
Abstract Purpose: To investigate whether a vision-language foundation model can enhance undersampled MRI reconstruction by providing high-level contextual information beyond conventional priors. Methods: We proposed a semantic distribution-guided reconstruction framework that uses a pre-trained vision-language foundation model to encode both the reconstructed image and auxiliary information into high-level semantic features. A contrastive objective aligns the reconstructed representation with the target semantic distribution, ensuring consistency with high-level perceptual cues. The proposed objective works with various deep learning-based reconstruction methods and can flexibly incorporate semantic priors from multimodal sources. To test the effectiveness of these semantic priors, we evaluated reconstruction results guided by priors derived from either image-only or image-language auxiliary information. Results: Experiments on knee and brain datasets demonstrate that semantic priors from images preserve fine anatomical structures and achieve superior perceptual quality, as reflected in lower LPIPS values, higher Tenengrad scores, and improved scores in the reader study, compared with conventional regularization. The image-language information further expands the semantic distribution and enables high-level control over reconstruction attributes. Across all evaluations, the contrastive objective consistently guided the reconstructed features toward the desired semantic distributions while maintaining data fidelity, demonstrating the effectiveness of the proposed optimization framework. Conclusion: The study highlights that vision-language foundation models can improve undersampled MRI reconstruction through semantic-space optimization.
Authors Ruimin Feng, Xingxin He, Ronald Mercer, Zachary Stewart, Fang Liu
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Towards Synergistic Teacher-AI Interactions with Generative Artificial Intelligence
Title CN 通过生成式人工智能实现教师与AI的协同互动
Keywords 教育人工智能 · 生成式AI · 人机协作 · 教师能动性 · 系统综述
Summary 本文提出了一个关于教师与生成式AI协作的五级框架（交易级、情境级、操作级、实践级、协同级），旨在探讨AI在教育中对教师能动性、专业能力的影响，并展望了人机协同决策、共同推理的未来方向。重点在于教育伦理与社会技术系统的构建，而非技术方法论突破。
Reason 该论文属于教育技术领域，讨论的是教师与生成式AI在教学中的协作模式，属于典型的平行应用研究。其核心内容为教育场景下的AI角色框架设计，不涉及文档图像理解、视觉表示学习、多模态推理或VLM架构改进等DIU相关技术痛点。无任何可迁移至文档理解领域的底层方法创新。
Abstract Generative artificial intelligence (GenAI) is increasingly used in education, posing significant challenges for teachers adapting to these changes. GenAI offers unprecedented opportunities for accessibility, scalability and productivity in educational tasks. However, the automation of teaching tasks through GenAI raises concerns about reduced teacher agency, potential cognitive atrophy, and the broader deprofessionalisation of teaching. Drawing findings from prior literature on AI in Education, and refining through a recent systematic literature review, this chapter presents a conceptualisation of five levels of teacher-AI teaming: transactional, situational, operational, praxical and synergistic teaming. The framework aims to capture the nuanced dynamics of teacher-AI interactions, particularly with GenAI, that may lead to the replacement, complementarity, or augmentation of teachers' competences and professional practice. GenAI technological affordances required in supporting teaming, along with empirical studies, are discussed. Drawing on empirical observations, we outline a future vision that moves beyond individual teacher agency toward collaborative decision-making between teachers and AI, in which both agents engage in negotiation, constructive challenge, and co-reasoning that enhance each other's capabilities and enable outcomes neither could realise independently. Further discussion of socio-technical factors beyond teacher-AI teaming is also included to streamline the synergy of teachers and AI in education ethically and practically.
Authors Mutlu Cukurova, Wannapon Suraworachet, Qi Zhou, Sahan Bulathwela
Comment 18 pages, 6 pages
Categories Artificial Intelligence
0
Using Wearable Devices to Improve Chronic PainTreatment among Patients with Opioid Use Disorder
Title CN 使用可穿戴设备改善阿片类药物使用障碍患者的慢性疼痛治疗
Keywords 可穿戴设备 · 慢性疼痛 · 阿片类药物使用障碍 · 机器学习 · 大语言模型
Summary 本研究探索了利用可穿戴设备结合AI模型（包括机器学习和LLM）来预测阿片类药物使用障碍患者中的疼痛加剧现象。结果显示传统机器学习模型有一定预测能力，而LLM目前表现有限。研究呼吁开发能提供可操作洞察的LLM。但整体属于医疗垂直领域应用，无通用方法输出。
Reason 该论文属于医疗健康领域的应用型研究，聚焦于慢性疼痛与阿片类药物使用障碍的临床管理，使用AI分析可穿戴设备数据。尽管提到了大语言模型（LLM），但其应用场景为生理信号与健康状态预测，与文档图像理解（DIU）、DocVQA、OCR、多模态文档建模等核心任务完全无关。属于典型的‘平行下游应用’，且未提出任何可用于文档理解的方法论创新。
Abstract Chronic pain (CP) and opioid use disorder (OUD) are common and interrelated chronic medical conditions. Currently, there is a paucity of evidence-based integrated treatments for CP and OUD among individuals receiving medication for opioid use disorder (MOUD). Wearable devices have the potential to monitor complex patient information and inform treatment development for persons with OUD and CP, including pain variability (e.g., exacerbations of pain or pain spikes) and clinical correlates (e.g., perceived stress). However, the application of large language models (LLMs) with wearable data for understanding pain spikes, remains unexplored. Consequently, the aim of this pilot study was to examine the clinical correlates of pain spikes using a range of AI approaches. We found that machine learning models achieved relatively high accuracy (>0.7) in predicting pain spikes, while LLMs were limited in providing insights on pain spikes. Real-time monitoring through wearable devices, combined with advanced AI models, could facilitate early detection of pain spikes and support personalized interventions that may help mitigate the risk of opioid relapse, improve adherence to MOUD, and enhance the integration of CP and OUD care. Given overall limited LLM performance, these findings highlight the need to develop LLMs which can provide actionable insights in the OUD/CP context.
Authors Abhay Goyal, Navin Kumar, Kimberly DiMeola, Rafael Trujillo, Soorya Ram Shimgekar, Christian Poellabauer, Pi Zonooz, Ermonda Gjoni-Markaj, Declan Barry, Lynn Madden
Categories Artificial Intelligence
0
Deductive Systems for Logic Programs with Counting Theory and Practice of Logic Programming (TPLP)
Title CN 用于带计数逻辑程序的演绎系统
Keywords 逻辑编程 · 答案集编程 · 强等价性 · 演绎系统 · 计数聚合
Summary 本文提出了一种演绎系统，用于证明包含计数聚合的逻辑程序之间的强等价性。通过在特定演绎系统中相互推导规则来验证两个程序是否在任意上下文中具有相同语义。研究集中在理论层面的形式化验证方法。
Reason 该论文属于纯逻辑编程与答案集编程（ASP）的理论研究，聚焦于强等价性和演绎系统的扩展，不涉及任何多模态、视觉或文档理解内容。与文档图像理解（DIU）任务如DocVQA、OCR、布局分析等无直接关联，也无法迁移其方法解决DIU中的核心痛点（如幻觉、高分辨率处理、细粒度定位等）。属于无关领域。
Abstract In answer set programming, two groups of rules are considered strongly equivalent if they have the same meaning in any context. Strong equivalence of two programs can be sometimes established by deriving rules of each program from rules of the other in an appropriate deductive system. This paper shows how to extend this method of proving strong equivalence to programs containing the counting aggregate.
Authors Jorge Fandinno, Vladimir Lifschitz
Comment Under consideration in Theory and Practice of Logic Programming (TPLP)
Categories Artificial Intelligence
0
SPQR: A Standardized Benchmark for Modern Safety Alignment Methods in Text-to-Image Diffusion Models
Title CN SPQR：面向现代文本到图像扩散模型安全对齐方法的标准化基准
Keywords 文本到图像生成 · 安全对齐 · 基准测试 · 微调鲁棒性 · 扩散模型
Summary 本文提出了SPQR基准，用于评估文本到图像扩散模型在经历良性微调（如LoRA个性化）后安全对齐能力的保持情况。通过综合衡量安全性、提示遵循度、质量和鲁棒性，提供一个统一评分以比较不同安全对齐方法。研究揭示了当前方法在微调后容易失效的问题，强调部署后稳定性的重要性。
Reason 该论文聚焦于文本到图像生成模型的安全对齐问题，属于纯图像生成领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。其核心关注点为生成内容的安全性、版权与隐私控制，并未涉及OCR、布局分析、表格识别或视觉语言模型在文档上的推理优化。尽管提到了多语言和领域适应，但研究目标与文档理解的核心痛点（如幻觉、复杂排版、长文档推理）完全不相关。
Abstract Text-to-image diffusion models can emit copyrighted, unsafe, or private content. Safety alignment aims to suppress specific concepts, yet evaluations seldom test whether safety persists under benign downstream fine-tuning routinely applied after deployment (e.g., LoRA personalization, style/domain adapters). We study the stability of current safety methods under benign fine-tuning and observe frequent breakdowns. As true safety alignment must withstand even benign post-deployment adaptations, we introduce the SPQR benchmark (Safety-Prompt adherence-Quality-Robustness). SPQR is a single-scored metric that provides a standardized and reproducible framework to evaluate how well safety-aligned diffusion models preserve safety, utility, and robustness under benign fine-tuning, by reporting a single leaderboard score to facilitate comparisons. We conduct multilingual, domain-specific, and out-of-distribution analyses, along with category-wise breakdowns, to identify when safety alignment fails after benign fine-tuning, ultimately showcasing SPQR as a concise yet comprehensive benchmark for T2I safety alignment techniques for T2I models.
Authors Mohammed Talha Alam, Nada Saadi, Fahad Shamshad, Nils Lukas, Karthik Nandakumar, Fahkri Karray, Samuele Poppi
Comment 20 pages, 8 figures, 10 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Online Sparse Feature Selection in Data Streams via Differential Evolution
Title CN 基于差分进化的数据流中在线稀疏特征选择
Keywords 在线特征选择 · 数据流 · 差分进化 · 缺失数据 · 高维数据
Summary 本文提出了一种名为ODESFS的在线稀疏特征选择方法，结合潜在因子分析进行缺失值填补，并利用差分进化算法优化特征子集选择。在六个真实数据集上验证了其在准确性和特征选择效率上的优越性。然而，该工作属于通用数据流处理范畴，不涉及视觉或文档模态。
Reason 该论文研究的是高维数据流中的特征选择问题，属于传统机器学习与数据挖掘领域，与文档图像理解（DIU）或视觉-语言模型（VLM）无直接关联。其方法聚焦于流式数据的缺失值填补和特征重要性评估，并未涉及OCR、布局分析、表格识别或任何多模态文档理解任务。无法迁移用于解决DIU中的核心痛点如OCR幻觉、复杂排版建模或长文档推理。
Abstract The processing of high-dimensional streaming data commonly utilizes online streaming feature selection (OSFS) techniques. However, practical implementations often face challenges with data incompleteness due to equipment failures and technical constraints. Online Sparse Streaming Feature Selection (OS2FS) tackles this issue through latent factor analysis-based missing data imputation. Despite this advancement, existing OS2FS approaches exhibit substantial limitations in feature evaluation, resulting in performance deterioration. To address these shortcomings, this paper introduces a novel Online Differential Evolution for Sparse Feature Selection (ODESFS) in data streams, incorporating two key innovations: (1) missing value imputation using a latent factor analysis model, and (2) feature importance evaluation through differential evolution. Comprehensive experiments conducted on six real-world datasets demonstrate that ODESFS consistently outperforms state-of-the-art OSFS and OS2FS methods by selecting optimal feature subsets and achieving superior accuracy.
Authors Ruiyang Xu
Categories Artificial Intelligence
0
Quantifying Behavioral Dissimilarity Between Mathematical Expressions
Title CN 量化数学表达式之间的行为差异性
Keywords 数学表达式 · 行为相似性 · Wasserstein距离 · 符号回归 · 表达式比较
Summary 本文提出了一种名为BED（Behavior-aware Expression Dissimilarity）的框架，用于量化带有自由参数的数学表达式之间的行为差异。该方法将表达式建模为输入-输出对的联合概率分布，并使用Wasserstein距离衡量其行为距离，相比传统基于语法结构的相似性度量，能更准确反映实际功能上的相似性。该工作主要面向符号回归、方程发现和神经符号系统等场景。
Reason 该论文研究的是数学表达式之间的行为相似性度量，属于符号计算与数学推理领域，尽管其方法（如Wasserstein距离用于表达式分布比较）在理论上可能对某些形式化系统有启发，但与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别或OCR-free理解无直接关联。无法迁移解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU痛点，属于无关领域。
Abstract Quantifying the similarity between mathematical expressions is a fundamental problem in computational mathematics, symbolic reasoning, and scientific discovery. While behavioral notions of similarity have previously been explored in the context of software and program analysis, existing measures for mathematical expressions rely primarily on syntactic form, assessing similarity through symbolic structure rather than actual behavior. Yet syntactically distinct expressions can exhibit nearly identical outputs, while structurally similar ones may behave very differently-especially when the expressions contain free parameters that define families of functions. To address these limitations, we introduce Behavior-aware Expression Dissimilarity (BED), a principled framework for quantifying behavioral distance between mathematical expressions with free parameters. BED represents expressions as joint probability distributions over their input-output pairs and applies the Wasserstein distance to measure behavioral dissimilarity. A computationally efficient stochastic approximation is proposed and shown to be consistent, robust, and capable of inducing a smoother, more meaningful structure over the space of expressions than syntax-based measures. The approach provides a foundation for behavior-based comparison, clustering, and learning of mathematical expressions, with potential direct applications in equation discovery, symbolic regression, and neuro-symbolic modeling.
Authors Sebastian Mežnar, Sašo Džeroski, Ljupčo Todorovski
Comment 43 pages, 11 figures, 3 table, 5 appendices. Submitted to Elsevier's Information Sciences
Categories Artificial Intelligence
0
When Should Neural Data Inform Welfare? A Critical Framework for Policy Uses of Neuroeconomics Durham Economic Journal 2025
Title CN 神经数据何时应影响福利？神经经济学政策应用的批判性框架
Keywords 神经经济学 · 福利判断 · 强化学习 · 政策应用 · 规范性标准
Summary 本文提出了一个基于模型的框架，用于评估神经数据在何种条件下可以合法地用于政策层面的福利判断，而非仅仅描述行为。作者在一个演员-评论家强化学习模型中形式化了从神经活动到潜在价值和预测误差的推断路径，并强调只有在神经-计算映射得到充分验证、决策模型能区分真实利益与情境性错误、且福利标准明确指定的情况下，神经证据才能约束福利判断。文章还提出了一个‘神经经济学福利推断检查表’供监管者和NeuroAI系统设计者使用。
Reason 该论文属于神经经济学与政策交叉领域，讨论的是如何将神经数据用于福利判断和政策制定，核心内容涉及规范性标准、价值判断和决策模型，与文档图像理解（DIU）、视觉语言模型（VLM）或OCR等技术完全无关。未提出任何可用于解决DIU任务中OCR幻觉、复杂排版、细粒度定位或长文档推理等问题的方法论或技术工具。
Abstract Neuroeconomics promises to ground welfare analysis in neural and computational evidence about how people value outcomes, learn from experience and exercise self-control. At the same time, policy and commercial actors increasingly invoke neural data to justify paternalistic regulation, "brain-based" interventions and new welfare measures. This paper asks under what conditions neural data can legitimately inform welfare judgements for policy rather than merely describing behaviour. I develop a non-empirical, model-based framework that links three levels: neural signals, computational decision models and normative welfare criteria. Within an actor-critic reinforcement-learning model, I formalise the inference path from neural activity to latent values and prediction errors and then to welfare claims. I show that neural evidence constrains welfare judgements only when the neural-computational mapping is well validated, the decision model identifies "true" interests versus context-dependent mistakes, and the welfare criterion is explicitly specified and defended. Applying the framework to addiction, neuromarketing and environmental policy, I derive a Neuroeconomic Welfare Inference Checklist for regulators and for designers of NeuroAI systems. The analysis treats brains and artificial agents as value-learning systems while showing that internal reward signals, whether biological or artificial, are computational quantities and cannot be treated as welfare measures without an explicit normative model.
Authors Yiven, Zhu
Comment Durham Economic Journal 2025
Categories Artificial Intelligence
0
Cross-Domain Generalization of Multimodal LLMs for Global Photovoltaic Assessment
Title CN 用于全球光伏评估的多模态大语言模型跨域泛化
Keywords 多模态大语言模型 · 跨域泛化 · 光伏评估 · 卫星图像 · 结构化提示
Summary 本文研究了多模态大语言模型在卫星图像中进行光伏系统检测、定位和量化的跨域泛化能力。通过结构化提示和微调，模型在未见区域表现出优于传统CV模型的鲁棒性。然而，该工作聚焦于遥感影像，不涉及文档结构、文本识别或复杂排版理解。
Reason 该论文属于遥感图像分析领域，研究任务为基于卫星图像的光伏系统检测与量化，属于纯视觉+LLM的垂直领域应用。其核心方法是使用结构化提示和微调实现跨区域泛化，但未提出任何针对文档图像理解（如OCR、排版分析、表格识别、文本密集处理等）的方法论创新。属于典型的‘平行下游应用’，且应用场景与文档无关。
Abstract The rapid expansion of distributed photovoltaic (PV) systems poses challenges for power grid management, as many installations remain undocumented. While satellite imagery provides global coverage, traditional computer vision (CV) models such as CNNs and U-Nets require extensive labeled data and fail to generalize across regions. This study investigates the cross-domain generalization of a multimodal large language model (LLM) for global PV assessment. By leveraging structured prompts and fine-tuning, the model integrates detection, localization, and quantification within a unified schema. Cross-regional evaluation using the $Δ$F1 metric demonstrates that the proposed model achieves the smallest performance degradation across unseen regions, outperforming conventional CV and transformer baselines. These results highlight the robustness of multimodal LLMs under domain shift and their potential for scalable, transferable, and interpretable global PV mapping.
Authors Muhao Guo, Yang Weng
Comment 5 pages, 7 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
AttackPilot: Autonomous Inference Attacks Against ML Services With LLM-Based Agents
Title CN AttackPilot：基于LLM代理的自主机器学习服务推理攻击
Keywords LLM代理 · 推理攻击 · 自动化安全评估 · 多代理系统 · 策略优化
Summary 本文提出了AttackPilot，一种基于大语言模型的自主代理系统，能够无需人工干预地执行针对机器学习服务的推理攻击。实验表明该代理在多种目标服务上实现了100%的任务完成率，并接近专家级攻击性能，同时通过多代理架构和任务特定动作空间设计来减少规划错误和上下文丢失等问题。
Reason 该论文属于具身智能/机器人与安全审计的交叉应用，研究的是利用LLM代理对ML服务发起自动化推理攻击，属于特定领域的下游应用（如安全审计工具），并未提出可用于文档图像理解（DIU）的底层方法。其核心贡献在于攻击流程的自动化，而非解决OCR幻觉、复杂排版、长文档推理等DIU痛点。尽管涉及‘幻觉缓解’，但上下文为任务执行中的行为错误，与视觉或语言模态的语义幻觉无关，无法迁移至文档理解场景。
Abstract Inference attacks have been widely studied and offer a systematic risk assessment of ML services; however, their implementation and the attack parameters for optimal estimation are challenging for non-experts. The emergence of advanced large language models presents a promising yet largely unexplored opportunity to develop autonomous agents as inference attack experts, helping address this challenge. In this paper, we propose AttackPilot, an autonomous agent capable of independently conducting inference attacks without human intervention. We evaluate it on 20 target services. The evaluation shows that our agent, using GPT-4o, achieves a 100.0% task completion rate and near-expert attack performance, with an average token cost of only $0.627 per run. The agent can also be powered by many other representative LLMs and can adaptively optimize its strategy under service constraints. We further perform trace analysis, demonstrating that design choices, such as a multi-agent framework and task-specific action spaces, effectively mitigate errors such as bad plans, inability to follow instructions, task context loss, and hallucinations. We anticipate that such agents could empower non-expert ML service providers, auditors, or regulators to systematically assess the risks of ML services without requiring deep domain expertise.
Authors Yixin Wu, Rui Wen, Chi Cui, Michael Backes, Yang Zhang
Categories Artificial Intelligence
0
Analysis of heart failure patient trajectories using sequence modeling
Title CN 使用序列建模分析心力衰竭患者轨迹
Keywords 序列建模 · 电子健康记录 · Transformer · Mamba · 临床预测
Summary 本文在瑞典大规模心力衰竭队列中比较了六种序列模型（包括Transformer、改进型Transformer++和Mamba）在三种一年期临床预测任务上的表现，探讨了输入表示、模型配置和时间预处理的影响。结果显示Llama架构和Mamba在性能与效率上均优于传统Transformer。
Reason 该论文属于医疗领域的临床预测任务，核心是利用Transformer和Mamba架构对电子健康记录（EHR）进行建模，解决的是时间序列预测问题，而非文档图像理解（DIU）或视觉-语言多模态任务。尽管使用了先进的序列模型，但其输入为结构化/文本化医疗数据，并未涉及图像、布局、OCR、视觉定位或文档排版等DIU核心挑战。属于典型的‘平行下游应用’，无视觉或多模态方法论创新，无法迁移用于解决文档图像理解中的幻觉、密集文本或复杂排版等问题。
Abstract Transformers have defined the state-of-the-art for clinical prediction tasks involving electronic health records (EHRs). The recently introduced Mamba architecture outperformed an advanced Transformer (Transformer++) based on Llama in handling long context lengths, while using fewer model parameters. Despite the impressive performance of these architectures, a systematic approach to empirically analyze model performance and efficiency under various settings is not well established in the medical domain. The performances of six sequence models were investigated across three architecture classes (Transformers, Transformers++, Mambas) in a large Swedish heart failure (HF) cohort (N = 42820), providing a clinically relevant case study. Patient data included diagnoses, vital signs, laboratories, medications and procedures extracted from in-hospital EHRs. The models were evaluated on three one-year prediction tasks: clinical instability (a readmission phenotype) after initial HF hospitalization, mortality after initial HF hospitalization and mortality after latest hospitalization. Ablations account for modifications of the EHR-based input patient sequence, architectural model configurations, and temporal preprocessing techniques for data collection. Llama achieves the highest predictive discrimination, best calibration, and showed robustness across all tasks, followed by Mambas. Both architectures demonstrate efficient representation learning, with tiny configurations surpassing other large-scaled Transformers. At equal model size, Llama and Mambas achieve superior performance using 25% less training data. This paper presents a first ablation study with systematic design choices for input tokenization, model configuration and temporal data preprocessing. Future model development in clinical prediction tasks using EHRs could build upon this study's recommendation as a starting point.
Authors Falk Dippel, Yinan Yu, Annika Rosengren, Martin Lindgren, Christina E. Lundberg, Erik Aerts, Martin Adiels, Helen Sjöland
Categories Artificial Intelligence
0
Yanyun-3: Enabling Cross-Platform Strategy Game Operation with Vision-Language Models
Title CN Yanyun-3：利用视觉语言模型实现跨平台策略游戏操作
Keywords 视觉语言模型 · GUI自动化 · 游戏AI · 多模态数据组织 · 代理系统
Summary 本文提出Yanyun-3，一个基于VLM的游戏代理，通过结合Qwen2.5-VL和UI-TARS，在跨平台策略游戏中实现自动操作。作者引入‘组合粒度’的数据组织方式，并采用QLoRA进行微调，在BLEU-4和推理效率上取得显著提升。系统能跨平台执行目标选择、资源分配等任务，无需针对性调优。
Reason 该论文属于典型的平行应用类工作，将现有VLM（Qwen2.5-VL）和UI执行模型（UI-TARS）应用于策略游戏自动化场景。尽管提出了‘组合粒度’的数据组织原则，但其核心是GUI自动化在游戏领域的具体落地，未针对文档图像理解中的OCR幻觉、密集文本、复杂排版或长文档推理等根本问题提出可迁移的方法论创新。任务场景与DIU无关，属于具身智能/机器人代理的下游应用。
Abstract Cross-platform strategy game automation remains a challenge due to diverse user interfaces and dynamic battlefield environments. Existing Vision--Language Models (VLMs) struggle with generalization across heterogeneous platforms and lack precision in interface understanding and action execution. We introduce Yanyun-3, a VLM-based agent that integrates Qwen2.5-VL for visual reasoning and UI-TARS for interface execution. We propose a novel data organization principle -- combination granularity -- to distinguish intra-sample fusion and inter-sample mixing of multimodal data (static images, multi-image sequences, and videos). The model is fine-tuned using QLoRA on a curated dataset across three strategy game platforms. The optimal strategy (M*V+S) achieves a 12.98x improvement in BLEU-4 score and a 63% reduction in inference time compared to full fusion. Yanyun-3 successfully executes core tasks (e.g., target selection, resource allocation) across platforms without platform-specific tuning. Our findings demonstrate that structured multimodal data organization significantly enhances VLM performance in embodied tasks. Yanyun-3 offers a generalizable framework for GUI automation, with broader implications for robotics and autonomous systems.
Authors Guoyan Wang, Yanyan Huang, Chunlin Chen, Lifeng Wang, Yuxiang Sun
Comment 32 pages, 13 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?
Title CN 受神经科学启发的多模态视觉语言模型对成员推断隐私泄露具有鲁棒性吗？
Keywords 隐私攻击 · 成员推断攻击 · 神经启发模型 · 拓扑正则化 · 多模态模型
Summary 本文研究了受神经科学启发的拓扑正则化方法是否能增强多模态视觉语言模型（VLM）对成员推断攻击（MIA）的防御能力。作者在BLIP、PaliGemma 2和ViT-GPT2等模型上进行实验，发现引入拓扑正则化（tau > 0）后，模型对MIA攻击的敏感性显著降低（如BLIP在COCO上ROC-AUC下降24%），同时保持了相近的生成质量。结果表明此类‘神经启发’VLM更具隐私鲁棒性。
Reason 该论文研究的是多模态视觉语言模型在隐私攻击（特别是成员推断攻击）下的安全性问题，属于AI安全与隐私领域。尽管其涉及VLM，但核心关注点是隐私泄露而非文档理解任务本身或DIU中的典型痛点（如OCR幻觉、复杂排版、细粒度定位等）。没有提出可用于提升文档图像理解性能的新架构、训练方法或推理机制，且与文档结构、文本识别、布局分析等DIU核心任务无直接关联。
Abstract In the age of agentic AI, the growing deployment of multi-modal models (MMs) has introduced new attack vectors that can leak sensitive training data in MMs, causing privacy leakage. This paper investigates a black-box privacy attack, i.e., membership inference attack (MIA) on multi-modal vision-language models (VLMs). State-of-the-art research analyzes privacy attacks primarily to unimodal AI-ML systems, while recent studies indicate MMs can also be vulnerable to privacy attacks. While researchers have demonstrated that biologically inspired neural network representations can improve unimodal model resilience against adversarial attacks, it remains unexplored whether neuro-inspired MMs are resilient against privacy attacks. In this work, we introduce a systematic neuroscience-inspired topological regularization (tau) framework to analyze MM VLMs resilience against image-text-based inference privacy attacks. We examine this phenomenon using three VLMs: BLIP, PaliGemma 2, and ViT-GPT2, across three benchmark datasets: COCO, CC3M, and NoCaps. Our experiments compare the resilience of baseline and neuro VLMs (with topological regularization), where the tau > 0 configuration defines the NEURO variant of VLM. Our results on the BLIP model using the COCO dataset illustrate that MIA attack success in NEURO VLMs drops by 24% mean ROC-AUC, while achieving similar model utility (similarities between generated and reference captions) in terms of MPNet and ROUGE-2 metrics. This shows neuro VLMs are comparatively more resilient against privacy attacks, while not significantly compromising model utility. Our extensive evaluation with PaliGemma 2 and ViT-GPT2 models, on two additional datasets: CC3M and NoCaps, further validates the consistency of the findings. This work contributes to the growing understanding of privacy risks in MMs and provides evidence on neuro VLMs privacy threat resilience.
Authors David Amebley, Sayanton Dibbo
Categories Artificial Intelligence
0
DUALGUAGE: Automated Joint Security-Functionality Benchmarking for Secure Code Generation
Title CN DUALGUAGE：安全代码生成的自动化联合安全-功能基准测试
Keywords 安全代码生成 · 大语言模型 · 基准测试 · 功能正确性 · 自动化评估
Summary 本文提出DUALGAUGE，一个用于评估大语言模型在生成代码时同时保证安全性和功能正确性的自动化基准测试框架，并配套构建了DUALGAUGE-BENCH数据集。通过沙箱执行和LLM评估器进行双重验证，揭示了当前主流LLM在安全编码方面的缺陷。
Reason 该论文聚焦于大语言模型生成代码的安全性与功能正确性联合评估，属于纯NLP/Safety领域的下游应用研究，尽管涉及LLM评估框架，但其方法和任务均与文档图像理解（DIU）、视觉语言模型（VLM）在文档上的应用无直接关联。未解决OCR幻觉、密集文本识别、复杂版式理解等DIU核心问题，也不提供可迁移至文档理解的底层技术（如视觉表示、高分辨率处理、多模态对齐等）。属于典型的‘平行应用’类工作。
Abstract Large language models (LLMs) and autonomous coding agents are increasingly used to generate software across a wide range of domains. Yet a core requirement remains unmet: ensuring that generated code is secure without compromising its functional correctness. Existing benchmarks and evaluations for secure code generation fall short-many measure only vulnerability reduction, disregard correctness preservation, or evaluate security and functionality on separate datasets, violating the fundamental need for simultaneous joint evaluation. We present DUALGAUGE, the first fully automated benchmarking framework designed to rigorously evaluate the security and correctness of LLM-generated code in unison. Given the lack of datasets enabling joint evaluation of secure code generation, we also present DUALGAUGE-BENCH, a curated benchmark suite of diverse coding tasks, each paired with manually validated test suites for both security and functionality, designed for full coverage of specification requirements. At the core of DUALGAUGE is an agentic program executor, which runs a program against given tests in sandboxed environments, and an LLM-based evaluator, which assesses both correctness and vulnerability behavior against expected outcomes. We rigorously evaluated and ensured the quality of DUALGAUGE-BENCH and the accuracy of DUALGAUGE, and applied DUALGAUGE to benchmarking ten leading LLMs on DUALGAUGE-BENCH across thousands of test scenarios. Our results reveal critical gaps in correct and secure code generation by these LLMs, for which our open-source system and datasets help accelerate progress via reproducible, scalable, and rigorous evaluation.
Authors Abhijeet Pathak, Suvadra Barua, Dinesh Gudimetla, Rupam Patir, Jiawei Guo, Hongxin Hu, Haipeng Cai
Categories Artificial Intelligence
0
PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approach
Title CN PropensityBench：通过智能体方法评估大语言模型中的潜在安全风险
Keywords 安全评估 · 大语言模型 · 智能体 · 潜在倾向 · 前沿风险
Summary 论文提出PropensityBench，一个通过模拟代理环境来评估大语言模型在获得高风险工具后是否倾向于执行危险行为的基准。实验涵盖网络安全、生物安全等领域，揭示了一些模型在压力下会选择高风险工具的现象。研究呼吁从静态能力评估转向动态倾向性评估。
Reason 该论文属于纯NLP安全与对齐领域，研究的是大语言模型在被赋予高风险能力时的‘倾向性’行为，属于Jailbreak/Safety类研究。尽管使用了Agent框架，但其任务和方法均与文档图像理解（DIU）或视觉幻觉无关，无法迁移用于解决OCR幻觉、复杂排版或长文档推理等核心DIU问题。
Abstract Recent advances in Large Language Models (LLMs) have sparked concerns over their potential to acquire and misuse dangerous or high-risk capabilities, posing frontier risks. Current safety evaluations primarily test for what a model \textit{can} do - its capabilities - without assessing what it $\textit{would}$ do if endowed with high-risk capabilities. This leaves a critical blind spot: models may strategically conceal capabilities or rapidly acquire them, while harboring latent inclinations toward misuse. We argue that $\textbf{propensity}$ - the likelihood of a model to pursue harmful actions if empowered - is a critical, yet underexplored, axis of safety evaluation. We present $\textbf{PropensityBench}$, a novel benchmark framework that assesses the proclivity of models to engage in risky behaviors when equipped with simulated dangerous capabilities using proxy tools. Our framework includes 5,874 scenarios with 6,648 tools spanning four high-risk domains: cybersecurity, self-proliferation, biosecurity, and chemical security. We simulate access to powerful capabilities via a controlled agentic environment and evaluate the models' choices under varying operational pressures that reflect real-world constraints or incentives models may encounter, such as resource scarcity or gaining more autonomy. Across open-source and proprietary frontier models, we uncover 9 alarming signs of propensity: models frequently choose high-risk tools when under pressure, despite lacking the capability to execute such actions unaided. These findings call for a shift from static capability audits toward dynamic propensity assessments as a prerequisite for deploying frontier AI systems safely. Our code is available at https://github.com/scaleapi/propensity-evaluation.
Authors Udari Madhushani Sehwag, Shayan Shabihi, Alex McAvoy, Vikash Sehwag, Yuancheng Xu, Dalton Towers, Furong Huang
Categories Artificial Intelligence
0
LumiTex: Towards High-Fidelity PBR Texture Generation with Illumination Context
Title CN LumiTex：基于光照上下文实现高保真PBR纹理生成
Keywords PBR纹理生成 · 光照感知生成 · 材质分解 · 视图一致性 · 图像补全
Summary 本文提出LumiTex，一种端到端框架用于高保真PBR纹理生成。该方法通过多分支生成结构解耦反照率与金属-粗糙度图，在共享光照先验下提升材质理解；引入光照感知的注意力机制以实现物理合理的材质生成；并采用几何引导的修复模块结合大视角合成模型完成无缝且视图一致的UV纹理补全。实验表明其在纹理质量上优于现有开源与商业方案。
Reason 该论文研究的是物理渲染中的材质纹理生成问题，属于计算机图形学与图像生成领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其核心技术聚焦于PBR材质分解、光照感知生成和视图一致的UV补全，无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心痛点。
Abstract Physically-based rendering (PBR) provides a principled standard for realistic material-lighting interactions in computer graphics. Despite recent advances in generating PBR textures, existing methods fail to address two fundamental challenges: 1) materials decomposition from image prompts under limited illumination cues, and 2) seamless and view-consistent texture completion. To this end, we propose LumiTex, an end-to-end framework that comprises three key components: (1) a multi-branch generation scheme that disentangles albedo and metallic-roughness under shared illumination priors for robust material understanding, (2) a lighting-aware material attention mechanism that injects illumination context into the decoding process for physically grounded generation of albedo, metallic, and roughness maps, and (3) a geometry-guided inpainting module based on a large view synthesis model that enriches texture coverage and ensures seamless, view-consistent UV completion. Extensive experiments demonstrate that LumiTex achieves state-of-the-art performance in texture quality, surpassing both existing open-source and commercial methods.
Authors Jingzhi Bao, Hongze Chen, Lingting Zhu, Chenyu Liu, Runze Zhang, Keyang Luo, Zeyu Hu, Weikai Chen, Yingda Yin, Xin Wang, Zehong Lin, Jun Zhang, Xiaoguang Han
Comment Project page: https://lumitex.vercel.app
Categories Computer Vision and Pattern Recognition
0
Cook and Clean Together: Teaching Embodied Agents for Parallel Task Execution AAAI 2026 (Oral)
Title CN 一起做饭和清洁：教授具身智能体进行并行任务执行
Keywords 具身智能 · 任务调度 · 多模态大模型 · 3D grounding · 运筹学
Summary 本文提出了ORS3D任务，旨在通过结合运筹学知识与3D空间 grounding，实现具身智能体在真实场景中的高效并行任务调度。为此构建了包含6万项复合任务的大规模数据集ORS3D-60K，并提出GRANT模型，利用调度token机制生成高效的可执行计划。实验表明该方法在语言理解、3D定位和调度效率方面均有效。
Reason 该论文属于具身智能与机器人领域，研究的是3D物理环境中的任务调度与多模态大模型控制，尽管使用了多模态LLM和调度机制，但其核心问题与文档图像理解（DIU）无关。无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心痛点。
Abstract Task scheduling is critical for embodied AI, enabling agents to follow natural language instructions and execute actions efficiently in 3D physical worlds. However, existing datasets often simplify task planning by ignoring operations research (OR) knowledge and 3D spatial grounding. In this work, we propose Operations Research knowledge-based 3D Grounded Task Scheduling (ORS3D), a new task that requires the synergy of language understanding, 3D grounding, and efficiency optimization. Unlike prior settings, ORS3D demands that agents minimize total completion time by leveraging parallelizable subtasks, e.g., cleaning the sink while the microwave operates. To facilitate research on ORS3D, we construct ORS3D-60K, a large-scale dataset comprising 60K composite tasks across 4K real-world scenes. Furthermore, we propose GRANT, an embodied multi-modal large language model equipped with a simple yet effective scheduling token mechanism to generate efficient task schedules and grounded actions. Extensive experiments on ORS3D-60K validate the effectiveness of GRANT across language understanding, 3D grounding, and scheduling efficiency. The code is available at https://github.com/H-EmbodVis/GRANT
Authors Dingkang Liang, Cheng Zhang, Xiaopeng Xu, Jianzhong Ju, Zhenbo Luo, Xiang Bai
Comment Accepted to AAAI 2026 (Oral). The code is available at \url{https://github.com/H-EmbodVis/GRANT}
Categories Computer Vision and Pattern Recognition
0
Flow Map Distillation Without Data
Title CN 无数据的流图蒸馏
Keywords 流模型 · 知识蒸馏 · 无数据 · 图像生成 · 生成模型
Summary 本文提出一种无需外部数据集的流图蒸馏方法，通过仅从先验分布采样来避免教师模型与训练数据之间的不匹配问题，实现了在ImageNet上的最先进生成质量，仅用一步采样即达到1.45的FID分数。
Reason 该论文研究的是生成模型加速中的流图蒸馏问题，属于图像生成领域，核心关注点为高保真、单步采样的图像生成，与文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等无直接关联。虽然其‘无数据蒸馏’思想具有一定方法论新颖性，但迁移至文档理解场景（如OCR去幻觉、复杂排版建模、长文档推理）极为困难，且未涉及多模态对齐、视觉-语言联合建模或细粒度定位等DIU关键痛点。
Abstract State-of-the-art flow models achieve remarkable quality but require slow, iterative sampling. To accelerate this, flow maps can be distilled from pre-trained teachers, a procedure that conventionally requires sampling from an external dataset. We argue that this data-dependency introduces a fundamental risk of Teacher-Data Mismatch, as a static dataset may provide an incomplete or even misaligned representation of the teacher's full generative capabilities. This leads us to question whether this reliance on data is truly necessary for successful flow map distillation. In this work, we explore a data-free alternative that samples only from the prior distribution, a distribution the teacher is guaranteed to follow by construction, thereby circumventing the mismatch risk entirely. To demonstrate the practical viability of this philosophy, we introduce a principled framework that learns to predict the teacher's sampling path while actively correcting for its own compounding errors to ensure high fidelity. Our approach surpasses all data-based counterparts and establishes a new state-of-the-art by a significant margin. Specifically, distilling from SiT-XL/2+REPA, our method reaches an impressive FID of 1.45 on ImageNet 256x256, and 1.49 on ImageNet 512x512, both with only 1 sampling step. We hope our work establishes a more robust paradigm for accelerating generative models and motivates the broader adoption of flow map distillation without data.
Authors Shangyuan Tong, Nanye Ma, Saining Xie, Tommi Jaakkola
Categories Computer Vision and Pattern Recognition
0
Ref-SAM3D: Bridging SAM3D with Text for Reference 3D Reconstruction
Title CN Ref-SAM3D：将SAM3D与文本结合用于参考3D重建
Keywords 3D重建 · 文本引导 · SAM3D · 跨模态对齐 · 零样本学习
Summary 本文提出了Ref-SAM3D，通过引入文本描述作为先验信息，扩展了SAM3D的能力，实现文本引导下的单视图3D重建。实验表明该方法在零样本设置下能生成高保真的3D结构，有效连接2D视觉与3D几何理解。然而该工作属于3D计算机视觉范畴，与文档理解无关。
Reason 该论文研究的是基于文本引导的3D物体重建，属于3D视觉与生成领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。其核心问题为从单张RGB图像和文本描述中重建特定3D对象，技术路径聚焦于3D几何建模与跨模态对齐，无法迁移解决文档中的OCR幻觉、复杂排版、长文本推理等关键痛点。
Abstract SAM3D has garnered widespread attention for its strong 3D object reconstruction capabilities. However, a key limitation remains: SAM3D cannot reconstruct specific objects referred to by textual descriptions, a capability that is essential for practical applications such as 3D editing, game development, and virtual environments. To address this gap, we introduce Ref-SAM3D, a simple yet effective extension to SAM3D that incorporates textual descriptions as a high-level prior, enabling text-guided 3D reconstruction from a single RGB image. Through extensive qualitative experiments, we show that Ref-SAM3D, guided only by natural language and a single 2D view, delivers competitive and high-fidelity zero-shot reconstruction performance. Our results demonstrate that Ref-SAM3D effectively bridges the gap between 2D visual cues and 3D geometric understanding, offering a more flexible and accessible paradigm for reference-guided 3D reconstruction. Code is available at: https://github.com/FudanCVL/Ref-SAM3D.
Authors Yun Zhou, Yaoting Wang, Guangquan Jie, Jinyu Liu, Henghui Ding
Comment Code: https://github.com/FudanCVL/Ref-SAM3D
Categories Computer Vision and Pattern Recognition
0
SAM3-Adapter: Efficient Adaptation of Segment Anything 3 for Camouflage Object Segmentation, Shadow Detection, and Medical Image Segmentation
Title CN SAM3-Adapter：用于伪装物体分割、阴影检测和医学图像分割的高效Segment Anything 3适配
Keywords 图像分割 · 基础模型适配 · 伪装物体检测 · 医学图像分割 · SAM3
Summary 本文提出了SAM3-Adapter，是首个专为新一代Segment Anything 3（SAM3）设计的适配框架，旨在提升其在细粒度低层视觉任务上的表现，包括伪装物体分割、阴影检测和医学图像分割。通过模块化设计，该方法在降低计算开销的同时显著优于基于SAM和SAM2的方案，在多个下游任务上达到SOTA性能。
Reason 该论文聚焦于通用图像分割中的特定挑战任务，如伪装物体、阴影和医学图像分割，属于纯计算机视觉领域的基础模型下游适配工作。尽管使用了SAM3这一先进基础模型，但其方法本质是针对特定视觉分割任务的微调框架（Adapter），并未涉及文档图像理解（DIU）的核心问题，如OCR幻觉、文本布局分析、表格识别或长文档推理等。同时，该研究不包含多模态语言模型（VLM）与文本内容的联合建模，也不处理文档特有的结构化输出需求，因此无法迁移为解决DIU痛点的有效工具。
Abstract The rapid rise of large-scale foundation models has reshaped the landscape of image segmentation, with models such as Segment Anything achieving unprecedented versatility across diverse vision tasks. However, previous generations-including SAM and its successor-still struggle with fine-grained, low-level segmentation challenges such as camouflaged object detection, medical image segmentation, cell image segmentation, and shadow detection. To address these limitations, we originally proposed SAM-Adapter in 2023, demonstrating substantial gains on these difficult scenarios. With the emergence of Segment Anything 3 (SAM3)-a more efficient and higher-performing evolution with a redesigned architecture and improved training pipeline-we revisit these long-standing challenges. In this work, we present SAM3-Adapter, the first adapter framework tailored for SAM3 that unlocks its full segmentation capability. SAM3-Adapter not only reduces computational overhead but also consistently surpasses both SAM and SAM2-based solutions, establishing new state-of-the-art results across multiple downstream tasks, including medical imaging, camouflaged (concealed) object segmentation, and shadow detection. Built upon the modular and composable design philosophy of the original SAM-Adapter, SAM3-Adapter provides stronger generalizability, richer task adaptability, and significantly improved segmentation precision. Extensive experiments confirm that integrating SAM3 with our adapter yields superior accuracy, robustness, and efficiency compared to all prior SAM-based adaptations. We hope SAM3-Adapter can serve as a foundation for future research and practical segmentation applications. Code, pre-trained models, and data processing pipelines are available.
Authors Tianrun Chen, Runlong Cao, Xinda Yu, Lanyun Zhu, Chaotao Ding, Deyi Ji, Cheng Chen, Qi Zhu, Chunyan Xu, Papa Mao, Ying Zang
Categories Computer Vision and Pattern Recognition
0
BackSplit: The Importance of Sub-dividing the Background in Biomedical Lesion Segmentation
Title CN BackSplit：在生物医学病灶分割中细分背景的重要性
Keywords 医学图像分割 · 背景建模 · 细粒度标注 · 小目标分割 · Fisher信息
Summary 本文提出BackSplit方法，主张在医学图像病灶分割中不应将所有非病灶区域简单归为单一‘背景’类，而应进一步细分为不同解剖结构（如组织、器官）。这种细粒度的背景建模可提升模型的信息利用效率，在不增加推理成本的情况下显著提高小病灶分割性能。作者从信息论角度证明该方法能提升Fisher信息量，并在多个数据集上验证了其有效性，包括使用预训练模型自动生成辅助标签的情形。
Reason 该论文研究的是医学图像中的病灶分割问题，属于纯医学图像分析领域，核心方法是通过细粒度标注背景类别来提升分割性能。虽然涉及视觉模型设计，但其任务、数据和应用场景均与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版或文档VQA等核心痛点。
Abstract Segmenting small lesions in medical images remains notoriously difficult. Most prior work tackles this challenge by either designing better architectures, loss functions, or data augmentation schemes; and collecting more labeled data. We take a different view, arguing that part of the problem lies in how the background is modeled. Common lesion segmentation collapses all non-lesion pixels into a single "background" class, ignoring the rich anatomical context in which lesions appear. In reality, the background is highly heterogeneous-composed of tissues, organs, and other structures that can now be labeled manually or inferred automatically using existing segmentation models. In this paper, we argue that training with fine-grained labels that sub-divide the background class, which we call BackSplit, is a simple yet powerful paradigm that can offer a significant performance boost without increasing inference costs. From an information theoretic standpoint, we prove that BackSplit increases the expected Fisher Information relative to conventional binary training, leading to tighter asymptotic bounds and more stable optimization. With extensive experiments across multiple datasets and architectures, we empirically show that BackSplit consistently boosts small-lesion segmentation performance, even when auxiliary labels are generated automatically using pretrained segmentation models. Additionally, we demonstrate that auxiliary labels derived from interactive segmentation frameworks exhibit the same beneficial effect, demonstrating its robustness, simplicity, and broad applicability.
Authors Rachit Saluja, Asli Cihangir, Ruining Deng, Johannes C. Paetzold, Fengbei Liu, Mert R. Sabuncu
Categories Computer Vision and Pattern Recognition
0
SketchDeco: Training-Free Latent Composition for Precise Sketch Colourisation
Title CN SketchDeco：无需训练的潜在空间组合实现精确草图上色
Keywords 草图上色 · 扩散模型 · 潜在空间编辑 · 无需训练 · 自注意力
Summary 本文提出SketchDeco，一种无需训练的草图上色方法，通过扩散模型反演在潜在空间中将用户指定的颜色‘绘制’到草图的特定区域，并利用自定义的自注意力机制实现局部着色与全局图像的一致性融合，从而在无需微调的情况下实现高质量、可控的上色效果。
Reason 该论文研究的是草图上色（sketch colourisation）任务，属于图像生成与艺术设计领域，核心方法基于扩散模型的潜在空间编辑和注意力机制进行颜色融合。尽管其训练-free和局部编辑能力具有一定技术新颖性，但其目标、数据和应用场景均与文档图像理解（DIU）无关。未涉及OCR、布局分析、表格识别、文本提取或视觉-语言推理等DIU核心问题，也无法迁移用于解决OCR幻觉、密集文本处理或复杂排版理解等痛点。
Abstract We introduce SketchDeco, a training-free approach to sketch colourisation that bridges the gap between professional design needs and intuitive, region-based control. Our method empowers artists to use simple masks and colour palettes for precise spatial and chromatic specification, avoiding both the tediousness of manual assignment and the ambiguity of text-based prompts. We reformulate this task as a novel, training-free composition problem. Our core technical contribution is a guided latent-space blending process: we first leverage diffusion inversion to precisely ``paint'' user-defined colours into specified regions, and then use a custom self-attention mechanism to harmoniously blend these local edits with a globally consistent base image. This ensures both local colour fidelity and global harmony without requiring any model fine-tuning. Our system produces high-quality results in 15--20 inference steps on consumer GPUs, making professional-quality, controllable colourisation accessible.
Authors Chaitat Utintu, Pinaki Nath Chowdhury, Aneeshan Sain, Subhadeep Koley, Ayan Kumar Bhunia, Yi-Zhe Song
Comment Project Page: \url{https://chaitron.github.io/SketchDeco/}
Categories Computer Vision and Pattern Recognition
0
Growing with the Generator: Self-paced GRPO for Video Generation
Title CN 与生成器共同成长：视频生成的自步长GRPO
Keywords 视频生成 · 强化学习 · GRPO · 自步长学习 · 语义对齐
Summary 本文提出Self-Paced GRPO，一种在视频生成中随生成器能力提升而动态调整奖励机制的强化学习框架。通过从粗粒度视觉保真度逐步转向时间连贯性和细粒度语义对齐的课程式奖励设计，缓解了奖励与策略之间的不匹配问题。实验表明该方法在多个视频生成模型上均优于固定奖励的GRPO基线。
Reason 该论文研究的是视频生成中的强化学习对齐问题，属于视频理解与生成领域，与文档图像理解（DIU）或DocVQA无关。其方法虽涉及训练动态优化，但核心场景为文本到视频生成的质量提升，无法直接迁移解决OCR幻觉、复杂排版解析或长文档推理等DIU核心痛点。
Abstract Group Relative Policy Optimization (GRPO) has emerged as a powerful reinforcement learning paradigm for post-training video generation models. However, existing GRPO pipelines rely on static, fixed-capacity reward models whose evaluation behavior is frozen during training. Such rigid rewards introduce distributional bias, saturate quickly as the generator improves, and ultimately limit the stability and effectiveness of reinforcement-based alignment. We propose Self-Paced GRPO, a competence-aware GRPO framework in which reward feedback co-evolves with the generator. Our method introduces a progressive reward mechanism that automatically shifts its emphasis from coarse visual fidelity to temporal coherence and fine-grained text-video semantic alignment as generation quality increases. This self-paced curriculum alleviates reward-policy mismatch, mitigates reward exploitation, and yields more stable optimization. Experiments on VBench across multiple video generation backbones demonstrate consistent improvements in both visual quality and semantic alignment over GRPO baselines with static rewards, validating the effectiveness and generality of Self-Paced GRPO.
Authors Rui Li, Yuanzhi Liang, Ziqi Ni, Haibing Huang, Chi Zhang, Xuelong Li
Categories Computer Vision and Pattern Recognition
0
CellFMCount: A Fluorescence Microscopy Dataset, Benchmark, and Methods for Cell Counting The IEEE International Conference on Data Mining (ICDM) 2025
Title CN CellFMCount：用于细胞计数的荧光显微镜数据集、基准和方法
Keywords 细胞计数 · 荧光显微镜 · 数据集 · SAM · 密度图预测
Summary 本文提出了一个大规模的荧光显微镜细胞计数数据集CellFMCount，包含3023张图像和超过43万个手动标注的细胞位置，具有高密度、细胞重叠、形态多样等挑战性特征。作者对回归型、人群计数型和专用细胞计数方法进行了系统评测，并探索了基于点标注的SAM适配方法（SAM-Counter），取得了优于现有方法的MAE结果（22.12）。该工作为自动化细胞计数提供了高质量的数据基础和评估框架。
Reason 该论文聚焦于生物医学图像中的细胞计数任务，属于纯计算机视觉在显微图像分析中的垂直应用，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了SAM等模型，但其方法本质是针对高密度细胞检测的特定优化，并未提出可迁移至文档理解领域的底层技术（如视觉表示学习、多模态对齐、推理增强等）。属于典型的‘平行下游应用’，不符合本研究方向的技术源头筛选标准。
Abstract Accurate cell counting is essential in various biomedical research and clinical applications, including cancer diagnosis, stem cell research, and immunology. Manual counting is labor-intensive and error-prone, motivating automation through deep learning techniques. However, training reliable deep learning models requires large amounts of high-quality annotated data, which is difficult and time-consuming to produce manually. Consequently, existing cell-counting datasets are often limited, frequently containing fewer than $500$ images. In this work, we introduce a large-scale annotated dataset comprising $3{,}023$ images from immunocytochemistry experiments related to cellular differentiation, containing over $430{,}000$ manually annotated cell locations. The dataset presents significant challenges: high cell density, overlapping and morphologically diverse cells, a long-tailed distribution of cell count per image, and variation in staining protocols. We benchmark three categories of existing methods: regression-based, crowd-counting, and cell-counting techniques on a test set with cell counts ranging from $10$ to $2{,}126$ cells per image. We also evaluate how the Segment Anything Model (SAM) can be adapted for microscopy cell counting using only dot-annotated datasets. As a case study, we implement a density-map-based adaptation of SAM (SAM-Counter) and report a mean absolute error (MAE) of $22.12$, which outperforms existing approaches (second-best MAE of $27.46$). Our results underscore the value of the dataset and the benchmarking framework for driving progress in automated cell counting and provide a robust foundation for future research and development.
Authors Abdurahman Ali Mohammed, Catherine Fonder, Ying Wei, Wallapak Tavanapong, Donald S Sakaguchi, Qi Li, Surya K. Mallapragada
Comment The IEEE International Conference on Data Mining (ICDM) 2025
Categories Computer Vision and Pattern Recognition
0
MonoMSK: Monocular 3D Musculoskeletal Dynamics Estimation
Title CN MonoMSK：单目3D肌肉骨骼动力学估计
Keywords 3D姿态估计 · 物理仿真 · 逆动力学 · 可微分建模 · 生物力学
Summary 本文提出MonoMSK，一种结合数据驱动学习与物理仿真的混合框架，用于从单目视频中恢复生物力学上真实的人体3D运动，同时估计运动学（姿态）和动力学（力与力矩）。通过引入基于Transformer的逆动力学模块与由ODE控制的可微分正向运动学和动力学层，构建了一个物理约束的逆-正循环结构，并设计了一种前向-逆向一致性损失来提升重建质量。实验表明该方法在多个数据集上优于现有方法。
Reason 该论文研究的是从单目视频中估计人体3D运动与力学参数，属于生物力学与人体动作捕捉领域，核心任务为3D姿态估计和动力学建模，使用的技术路线涉及物理仿真与可微分正向动力学。完全不涉及文档图像、文本理解、OCR或任何文档VQA相关任务。属于无关领域（3D视觉/具身智能相关），不符合DIU研究方向。
Abstract Reconstructing biomechanically realistic 3D human motion - recovering both kinematics (motion) and kinetics (forces) - is a critical challenge. While marker-based systems are lab-bound and slow, popular monocular methods use oversimplified, anatomically inaccurate models (e.g., SMPL) and ignore physics, fundamentally limiting their biomechanical fidelity. In this work, we introduce MonoMSK, a hybrid framework that bridges data-driven learning and physics-based simulation for biomechanically realistic 3D human motion estimation from monocular video. MonoMSK jointly recovers both kinematics (motions) and kinetics (forces and torques) through an anatomically accurate musculoskeletal model. By integrating transformer-based inverse dynamics with differentiable forward kinematics and dynamics layers governed by ODE-based simulation, MonoMSK establishes a physics-regulated inverse-forward loop that enforces biomechanical causality and physical plausibility. A novel forward-inverse consistency loss further aligns motion reconstruction with the underlying kinetic reasoning. Experiments on BML-MoVi, BEDLAM, and OpenCap show that MonoMSK significantly outperforms state-of-the-art methods in kinematic accuracy, while for the first time enabling precise monocular kinetics estimation.
Authors Farnoosh Koleini, Hongfei Xue, Ahmed Helmy, Pu Wang
Categories Computer Vision and Pattern Recognition
0
SteadyDancer: Harmonized and Coherent Human Image Animation with First-Frame Preservation
Title CN SteadyDancer：基于首帧保留的协调一致的人体图像动画
Keywords 人体动画 · 身份保持 · 动作控制 · 视频生成 · 时序一致性
Summary 本文提出SteadyDancer，一种基于图像到视频（I2V）范式的人体动画框架，旨在解决在精确动作控制下保持首帧身份一致性的挑战。通过引入条件调和机制、协同姿态调制模块以及分阶段解耦目标训练流程，实现了在外观保真度和运动控制方面的最先进性能。然而，该工作与文档图像理解无关。
Reason 该论文研究的是人体图像动画生成任务，属于视频生成与具身智能的交叉领域，核心关注点为动作控制、身份一致性与时序连贯性，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其提出的方法如条件调和机制、姿态调制模块等难以迁移到解决OCR幻觉、复杂版面分析或长文档推理等DIU核心痛点。属于无关领域（视频生成/具身智能），命中负面清单第2条。
Abstract Preserving first-frame identity while ensuring precise motion control is a fundamental challenge in human image animation. The Image-to-Motion Binding process of the dominant Reference-to-Video (R2V) paradigm overlooks critical spatio-temporal misalignments common in real-world applications, leading to failures such as identity drift and visual artifacts. We introduce SteadyDancer, an Image-to-Video (I2V) paradigm-based framework that achieves harmonized and coherent animation and is the first to ensure first-frame preservation robustly. Firstly, we propose a Condition-Reconciliation Mechanism to harmonize the two conflicting conditions, enabling precise control without sacrificing fidelity. Secondly, we design Synergistic Pose Modulation Modules to generate an adaptive and coherent pose representation that is highly compatible with the reference image. Finally, we employ a Staged Decoupled-Objective Training Pipeline that hierarchically optimizes the model for motion fidelity, visual quality, and temporal coherence. Experiments demonstrate that SteadyDancer achieves state-of-the-art performance in both appearance fidelity and motion control, while requiring significantly fewer training resources than comparable methods.
Authors Jiaming Zhang, Shengming Cao, Rui Li, Xiaotong Zhao, Yutao Cui, Xinglin Hou, Gangshan Wu, Haolan Chen, Yu Xu, Limin Wang, Kai Ma
Comment 10 pages, with supp
Categories Computer Vision and Pattern Recognition
0
SyncMV4D: Synchronized Multi-view Joint Diffusion of Appearance and Motion for Hand-Object Interaction Synthesis
Title CN SyncMV4D：基于外观与运动同步多视角联合扩散的手-物交互合成
Keywords 手-物交互 · 多视角生成 · 4D运动 · 扩散模型 · 点云对齐
Summary 本文提出SyncMV4D，是首个能够同步生成多视角手-物交互视频和4D运动轨迹的模型。通过引入多视角联合扩散（MJD）模型和扩散点对齐器（DPA），实现了2D外观与4D动态之间的闭环互增强：生成的视频指导4D运动优化，而对齐后的4D点云又反向引导下一阶段的视频生成。实验表明该方法在视觉真实感、动作合理性和多视角一致性方面优于现有方法。
Reason 该论文研究的是手-物交互（HOI）的视频生成与4D运动建模，属于3D视觉与视频生成领域，核心任务为多视角视频合成和动态点云对齐。其技术路线聚焦于扩散模型在三维动态场景中的应用，与文档图像理解（DIU）的核心痛点如OCR幻觉、文本识别、布局分析、表格理解等完全无关。尽管使用了多视角一致性与视觉-运动联合建模，但这些方法难以迁移到文档理解场景中处理密集文本或复杂排版问题。
Abstract Hand-Object Interaction (HOI) generation plays a critical role in advancing applications across animation and robotics. Current video-based methods are predominantly single-view, which impedes comprehensive 3D geometry perception and often results in geometric distortions or unrealistic motion patterns. While 3D HOI approaches can generate dynamically plausible motions, their dependence on high-quality 3D data captured in controlled laboratory settings severely limits their generalization to real-world scenarios. To overcome these limitations, we introduce SyncMV4D, the first model that jointly generates synchronized multi-view HOI videos and 4D motions by unifying visual prior, motion dynamics, and multi-view geometry. Our framework features two core innovations: (1) a Multi-view Joint Diffusion (MJD) model that co-generates HOI videos and intermediate motions, and (2) a Diffusion Points Aligner (DPA) that refines the coarse intermediate motion into globally aligned 4D metric point tracks. To tightly couple 2D appearance with 4D dynamics, we establish a closed-loop, mutually enhancing cycle. During the diffusion denoising process, the generated video conditions the refinement of the 4D motion, while the aligned 4D point tracks are reprojected to guide next-step joint generation. Experimentally, our method demonstrates superior performance to state-of-the-art alternatives in visual realism, motion plausibility, and multi-view consistency.
Authors Lingwei Dang, Zonghan Li, Juntong Li, Hongwen Zhang, Liang An, Yebin Liu, Qingyao Wu
Comment Project Page: https://droliven.github.io/SyncMV4D
Categories Computer Vision and Pattern Recognition
0
A Target-based Multi-LiDAR Multi-Camera Extrinsic Calibration System RiTA 2025
Title CN 一种基于目标的多LiDAR多相机外参标定系统
Keywords 外参标定 · LiDAR · 相机 · 自动驾驶 · ChArUco
Summary 本文提出了一种基于自定义ChArUco标定板和非线性优化方法的多LiDAR多相机外参标定系统，用于提升自动驾驶中多模态传感器数据的对齐精度。实验在仓库环境中采集的真实数据上进行，验证了该系统的有效性。
Reason 该论文属于自动驾驶领域的传感器标定工作，研究的是多LiDAR与多相机之间的外参标定问题，与文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的应用无任何关联。其方法无法迁移至解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract Extrinsic Calibration represents the cornerstone of autonomous driving. Its accuracy plays a crucial role in the perception pipeline, as any errors can have implications for the safety of the vehicle. Modern sensor systems collect different types of data from the environment, making it harder to align the data. To this end, we propose a target-based extrinsic calibration system tailored for a multi-LiDAR and multi-camera sensor suite. This system enables cross-calibration between LiDARs and cameras with limited prior knowledge using a custom ChArUco board and a tailored nonlinear optimization method. We test the system with real-world data gathered in a warehouse. Results demonstrated the effectiveness of the proposed method, highlighting the feasibility of a unique pipeline tailored for various types of sensors.
Authors Lorenzo Gentilini, Pierpaolo Serio, Valentina Donzella, Lorenzo Pollini
Comment RiTA 2025 Accepted, 13 Pages, 6 Figures and 2 Tables
Categories Computer Vision and Pattern Recognition
0
Dual-Granularity Semantic Prompting for Language Guidance Infrared Small Target Detection
Title CN 用于语言引导红外小目标检测的双粒度语义提示
Keywords 红外小目标检测 · 语言引导 · 双粒度提示 · 多模态学习 · 注意力机制
Summary 本文提出DGSPNet，一种基于双粒度语义提示的端到端框架，用于红外图像中的小目标检测。通过引入粗粒度文本先验（如‘红外图像’、‘小目标’）和从图像空间中提取的细粒度个性化语义描述，结合文本引导的通道与空间注意力机制（TGCA/TGSA），提升模型在复杂背景下的检测能力。实验表明该方法在三个基准数据集上达到SOTA性能。
Reason 该论文研究的是红外图像中的小目标检测问题，属于遥感或军事领域的计算机视觉任务，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了语言提示和多模态机制，但其核心场景是自然红外图像中的目标检测，而非文档布局、文本识别、表格解析等DIU核心任务。方法难以迁移到解决OCR幻觉、复杂排版或长文档推理等文档理解痛点。
Abstract Infrared small target detection remains challenging due to limited feature representation and severe background interference, resulting in sub-optimal performance. While recent CLIP-inspired methods attempt to leverage textual guidance for detection, they are hindered by inaccurate text descriptions and reliance on manual annotations. To overcome these limitations, we propose DGSPNet, an end-to-end language prompt-driven framework. Our approach integrates dual-granularity semantic prompts: coarse-grained textual priors (e.g., 'infrared image', 'small target') and fine-grained personalized semantic descriptions derived through visual-to-textual mapping within the image space. This design not only facilitates learning fine-grained semantic information but also can inherently leverage language prompts during inference without relying on any annotation requirements. By fully leveraging the precision and conciseness of text descriptions, we further introduce a text-guide channel attention (TGCA) mechanism and text-guide spatial attention (TGSA) mechanism that enhances the model's sensitivity to potential targets across both low- and high-level feature spaces. Extensive experiments demonstrate that our method significantly improves detection accuracy and achieves state-of-the-art performance on three benchmark datasets.
Authors Zixuan Wang, Haoran Sun, Jiaming Lu, Wenxuan Wang, Zhongling Huang, Dingwen Zhang, Xuelin Qian, Junwei Han
Comment 10 pages, 2 figures
Categories Computer Vision and Pattern Recognition
0
IDEAL-M3D: Instance Diversity-Enriched Active Learning for Monocular 3D Detection
Title CN IDEAL-M3D：用于单目3D检测的实例多样性增强主动学习
Keywords 单目3D检测 · 主动学习 · 实例级采样 · 模型多样性 · 自动驾驶
Summary 本文提出IDEAL-M3D，是首个面向单目3D检测的实例级主动学习框架，通过异构骨干网络、任务无关特征、损失权重扰动和时间依赖性bagging来提升模型多样性，实现在较少标注下达到更好的3D检测性能。但其应用场景完全偏离文档理解领域。
Reason 该论文研究的是单目3D目标检测中的主动学习问题，属于自动驾驶/3D视觉领域，与文档图像理解（DIU）或DocVQA无关。其方法聚焦于3D场景中物体的深度估计和实例级采样优化，无法迁移解决文档中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Monocular 3D detection relies on just a single camera and is therefore easy to deploy. Yet, achieving reliable 3D understanding from monocular images requires substantial annotation, and 3D labels are especially costly. To maximize performance under constrained labeling budgets, it is essential to prioritize annotating samples expected to deliver the largest performance gains. This prioritization is the focus of active learning. Curiously, we observed two significant limitations in active learning algorithms for 3D monocular object detection. First, previous approaches select entire images, which is inefficient, as non-informative instances contained in the same image also need to be labeled. Secondly, existing methods rely on uncertainty-based selection, which in monocular 3D object detection creates a bias toward depth ambiguity. Consequently, distant objects are selected, while nearby objects are overlooked. To address these limitations, we propose IDEAL-M3D, the first instance-level pipeline for monocular 3D detection. For the first time, we demonstrate that an explicitly diverse, fast-to-train ensemble improves diversity-driven active learning for monocular 3D. We induce diversity with heterogeneous backbones and task-agnostic features, loss weight perturbation, and time-dependent bagging. IDEAL-M3D shows superior performance and significant resource savings: with just 60% of the annotations, we achieve similar or better AP3D on KITTI validation and test set results compared to training the same detector on the whole dataset.
Authors Johannes Meier, Florian Günther, Riccardo Marin, Oussema Dhaouadi, Jacques Kaiser, Daniel Cremers
Categories Computer Vision and Pattern Recognition
0
DensifyBeforehand: LiDAR-assisted Content-aware Densification for Efficient and Quality 3D Gaussian Splatting
Title CN 预先稠密化：基于激光雷达辅助的内容感知稠密化方法，用于高效高质量的3D高斯点阵渲染
Keywords 3D高斯点阵 · 预先稠密化 · LiDAR辅助 · 内容感知采样 · 点云初始化
Summary 本文提出一种“预先稠密化”方法，通过结合稀疏LiDAR数据和单目深度估计，在3D高斯点阵渲染中实现内容感知的初始点云生成，优先增强感兴趣区域的几何与语义重要性，从而避免传统自适应稠密化带来的冗余高斯分布和漂浮伪影。该方法提升了视觉保真度和计算效率，在多个自建数据集上验证了有效性。
Reason 该论文属于3D视觉与图形学领域，研究的是3D高斯点阵（3D Gaussian Splatting）中的稠密化策略，核心问题为场景重建的效率与视觉质量优化。其技术路径依赖于LiDAR数据与RGB图像的融合，并聚焦于3D几何表示与渲染，与文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等无直接关联。所提方法难以迁移至解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点，且不属于多模态大模型在文档上的底层架构或推理机制创新。
Abstract This paper addresses the limitations of existing 3D Gaussian Splatting (3DGS) methods, particularly their reliance on adaptive density control, which can lead to floating artifacts and inefficient resource usage. We propose a novel densify beforehand approach that enhances the initialization of 3D scenes by combining sparse LiDAR data with monocular depth estimation from corresponding RGB images. Our ROI-aware sampling scheme prioritizes semantically and geometrically important regions, yielding a dense point cloud that improves visual fidelity and computational efficiency. This densify beforehand approach bypasses the adaptive density control that may introduce redundant Gaussians in the original pipeline, allowing the optimization to focus on the other attributes of 3D Gaussian primitives, reducing overlap while enhancing visual quality. Our method achieves comparable results to state-of-the-art techniques while significantly lowering resource consumption and training time. We validate our approach through extensive comparisons and ablation studies on four newly collected datasets, showcasing its effectiveness in preserving regions of interest in complex scenes.
Authors Phurtivilai Patt, Leyang Huang, Yinqiang Zhang, Yang Lei
Categories Computer Vision and Pattern Recognition
0
BideDPO: Conditional Image Generation with Simultaneous Text and Condition Alignment
Title CN BideDPO：通过文本和条件同步对齐的条件图像生成
Keywords 条件图像生成 · 偏好优化 · DPO · 多模态对齐 · 生成对抗
Summary 本文提出BideDPO框架，旨在解决条件图像生成中文本与条件信号之间的冲突问题。通过双向解耦的偏好优化机制，分别构建文本和条件的偏好对，并采用自适应损失平衡策略缓解梯度纠缠。同时设计自动化数据流水线生成冲突感知数据，在迭代中联合优化模型与数据。实验表明该方法显著提升了文本成功率和条件遵循能力。然而，其任务设定和技术创新均围绕图像生成展开，不适用于文档理解场景。
Reason 该论文研究的是条件图像生成中的文本与条件对齐问题，属于纯图像生成领域，主要目标是提升生成图像的质量与多模态一致性。其核心挑战在于解决文本提示与结构化条件（如布局、草图）之间的冲突，应用场景集中在T2I生成而非文档理解。尽管使用了偏好优化和数据迭代策略，但方法本质服务于图像合成，无法直接迁移用于解决文档图像理解中的OCR幻觉、密集文本识别或复杂排版解析等核心痛点。属于无关领域（纯图像生成），命中负面清单第2条。
Abstract Conditional image generation enhances text-to-image synthesis with structural, spatial, or stylistic priors, but current methods face challenges in handling conflicts between sources. These include 1) input-level conflicts, where the conditioning image contradicts the text prompt, and 2) model-bias conflicts, where generative biases disrupt alignment even when conditions match the text. Addressing these conflicts requires nuanced solutions, which standard supervised fine-tuning struggles to provide. Preference-based optimization techniques like Direct Preference Optimization (DPO) show promise but are limited by gradient entanglement between text and condition signals and lack disentangled training data for multi-constraint tasks. To overcome this, we propose a bidirectionally decoupled DPO framework (BideDPO). Our method creates two disentangled preference pairs-one for the condition and one for the text-to reduce gradient entanglement. The influence of pairs is managed using an Adaptive Loss Balancing strategy for balanced optimization. We introduce an automated data pipeline to sample model outputs and generate conflict-aware data. This process is embedded in an iterative optimization strategy that refines both the model and the data. We construct a DualAlign benchmark to evaluate conflict resolution between text and condition. Experiments show BideDPO significantly improves text success rates (e.g., +35%) and condition adherence. We also validate our approach using the COCO dataset. Project Pages: https://limuloo.github.io/BideDPO/.
Authors Dewei Zhou, Mingwei Li, Zongxin Yang, Yu Lu, Yunqiu Xu, Zhizhong Wang, Zeyi Huang, Yi Yang
Comment 29 pages
Categories Computer Vision and Pattern Recognition
0
Optimization-Free Style Transfer for 3D Gaussian Splats
Title CN 3D高斯点阵的无优化风格迁移
Keywords 3D高斯点阵 · 风格迁移 · 无优化 · 图结构 · 表面基方法
Summary 本文提出一种无需重建或优化的3D高斯点阵风格迁移方法，通过在点阵隐式表面构建图结构并进行表面基的前馈风格化，实现对.ply或.splat文件的快速风格迁移，且无需原始相机视角。该方法无需训练，可在2分钟内于CPU完成处理。
Reason 该论文研究的是3D高斯点阵（3D Gaussian Splats）的风格迁移，属于3D视觉与图形学领域，与文档图像理解（DIU）或DocVQA等任务完全无关。其技术路线聚焦于3D场景的表面图结构构建与前馈风格化，无法迁移到文档理解中的OCR幻觉、复杂排版解析或细粒度文本定位等核心痛点。
Abstract The task of style transfer for 3D Gaussian splats has been explored in many previous works, but these require reconstructing or fine-tuning the splat while incorporating style information or optimizing a feature extraction network on the splat representation. We propose a reconstruction- and optimization-free approach to stylizing 3D Gaussian splats, allowing for direct stylization on a .ply or .splat file without requiring the original camera views. This is done by generating a graph structure across the implicit surface of the splat representation. A feed-forward, surface-based stylization method is then used and interpolated back to the individual splats in the scene. This also allows for fast stylization of splats with no additional training, achieving speeds under 2 minutes even on CPU-based consumer hardware. We demonstrate the quality results this approach achieves and compare to other 3D Gaussian splat style transfer methods. Code is publicly available at https://github.com/davidmhart/FastSplatStyler.
Authors Raphael Du Sablon, David Hart
Categories Computer Vision and Pattern Recognition
0
FedPoisonTTP: A Threat Model and Poisoning Attack for Federated Test-Time Personalization
Title CN FedPoisonTTP：一种针对联邦测试时个性化场景的威胁模型与投毒攻击
Keywords 联邦学习 · 测试时个性化 · 投毒攻击 · 对抗攻击 · 安全
Summary 本文提出FedPoisonTTP，一种针对联邦学习中测试时个性化过程的灰盒投毒攻击框架。攻击者通过构建代理模型合成分布内投毒样本，并在本地自适应过程中注入高熵或类别置信度高的恶意更新，从而破坏全局及客户端模型性能。实验表明该方法可在视觉基准上显著降低测试时表现。
Reason 该论文研究的是联邦学习中测试时个性化（test-time personalization）场景下的安全投毒攻击，属于计算机视觉与安全交叉方向。虽然涉及视觉数据，但其核心问题是联邦学习中的对抗攻击，而非文档图像理解（DIU）或文档VQA的核心任务。未解决OCR幻觉、复杂排版、细粒度定位等DIU痛点，且为安全领域的平行问题，无法直接迁移用于提升文档理解性能。
Abstract Test-time personalization in federated learning enables models at clients to adjust online to local domain shifts, enhancing robustness and personalization in deployment. Yet, existing federated learning work largely overlooks the security risks that arise when local adaptation occurs at test time. Heterogeneous domain arrivals, diverse adaptation algorithms, and limited cross-client visibility create vulnerabilities where compromised participants can craft poisoned inputs and submit adversarial updates that undermine both global and per-client performance. To address this threat, we introduce FedPoisonTTP, a realistic grey-box attack framework that explores test-time data poisoning in the federated adaptation setting. FedPoisonTTP distills a surrogate model from adversarial queries, synthesizes in-distribution poisons using feature-consistency, and optimizes attack objectives to generate high-entropy or class-confident poisons that evade common adaptation filters. These poisons are injected during local adaptation and spread through collaborative updates, leading to broad degradation. Extensive experiments on corrupted vision benchmarks show that compromised participants can substantially diminish overall test-time performance.
Authors Md Akil Raihan Iftee, Syed Md. Ahnaf Hasan, Amin Ahsan Ali, AKM Mahbubur Rahman, Sajib Mistry, Aneesh Krishna
Comment 13 pages, 3 figures, 2 tables
Categories Computer Vision and Pattern Recognition
0
Fairness in Multi-modal Medical Diagnosis with Demonstration Selection
Title CN 基于示例选择的多模态医学诊断中的公平性
Keywords 多模态大模型 · 上下文学习 · 公平性 · 医学图像诊断 · 示例选择
Summary 本文研究多模态大语言模型在医学图像诊断中的公平性问题，指出传统的上下文示例选择策略因人口统计不平衡可能导致模型偏见。为此提出FADS方法，通过基于聚类的采样构建兼具语义相关性和人口统计平衡性的示例集，在不微调模型的前提下有效降低性别、种族等维度的偏差，同时保持准确率。
Reason 该论文聚焦于多模态大模型在医学图像推理中的公平性问题，提出通过构建人口统计学平衡的上下文示例来缓解偏见。虽然使用了In-Context Learning这一轻量级策略，但其核心应用场景是医疗影像诊断这一特定领域，属于典型的‘平行下游应用’。方法本身未针对文档图像理解中的OCR幻觉、复杂排版或长文档推理等核心痛点提供可迁移的技术路径，且研究领域为医疗+小语种潜在风险（尽管数据可能含中文，但任务非文档结构理解），不符合DIU本题创新要求。
Abstract Multimodal large language models (MLLMs) have shown strong potential for medical image reasoning, yet fairness across demographic groups remains a major concern. Existing debiasing methods often rely on large labeled datasets or fine-tuning, which are impractical for foundation-scale models. We explore In-Context Learning (ICL) as a lightweight, tuning-free alternative for improving fairness. Through systematic analysis, we find that conventional demonstration selection (DS) strategies fail to ensure fairness due to demographic imbalance in selected exemplars. To address this, we propose Fairness-Aware Demonstration Selection (FADS), which builds demographically balanced and semantically relevant demonstrations via clustering-based sampling. Experiments on multiple medical imaging benchmarks show that FADS consistently reduces gender-, race-, and ethnicity-related disparities while maintaining strong accuracy, offering an efficient and scalable path toward fair medical image reasoning. These results highlight the potential of fairness-aware in-context learning as a scalable and data-efficient solution for equitable medical image reasoning.
Authors Dawei Li, Zijian Gu, Peng Wang, Chuhan Song, Zhen Tan, Mohan Zhang, Tianlong Chen, Yu Tian, Song Wang
Comment 10 pages (including 2 pages of references), 4 figures. This work explores fairness in multi-modal medical image reasoning using in-context learning
Categories Computer Vision and Pattern Recognition
0
Minimax Multi-Target Conformal Prediction with Applications to Imaging Inverse Problems
Title CN 用于成像逆问题的极小化多目标共形预测
Keywords 不确定性量化 · 共形预测 · 成像逆问题 · 多目标估计 · MRI
Summary 本文提出一种渐近极小化的多目标共形预测方法，用于解决病态成像逆问题中的不确定性量化，支持多个下游目标的同时预测，并保证联合边际覆盖性。通过合成数据和MRI数据验证了其在多指标盲图像质量评估等任务上的优势。
Reason 该论文研究的是成像逆问题中的不确定性量化，聚焦于医学影像（如MRI）重建后的多目标共形预测，属于图像恢复与统计推断交叉领域。其方法虽在不确定性建模上有理论贡献，但任务本身与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。未涉及OCR、视觉语言模型、文本密集排版处理或文档推理等DIU关键痛点，且无法迁移用于解决OCR幻觉、细粒度定位或长文档理解等问题。
Abstract In ill-posed imaging inverse problems, uncertainty quantification remains a fundamental challenge, especially in safety-critical applications. Recently, conformal prediction has been used to quantify the uncertainty that the inverse problem contributes to downstream tasks like image classification, image quality assessment, fat mass quantification, etc. While existing works handle only a scalar estimation target, practical applications often involve multiple targets. In response, we propose an asymptotically minimax approach to multi-target conformal prediction that provides tight prediction intervals while ensuring joint marginal coverage. We then outline how our minimax approach can be applied to multi-metric blind image quality assessment, multi-task uncertainty quantification, and multi-round measurement acquisition. Finally, we numerically demonstrate the benefits of our minimax method, relative to existing multi-target conformal prediction methods, using both synthetic and magnetic resonance imaging (MRI) data. Code is available at https://github.com/jwen307/multi_target_minimax.
Authors Jeffrey Wen, Rizwan Ahmad, Philip Schniter
Categories Computer Vision and Pattern Recognition
0
Multiview point cloud registration with anisotropic and space-varying localization noise
Title CN 具有各向异性和空间变化定位噪声的多视角点云配准
Keywords 点云配准 · 高斯混合模型 · EM算法 · 各向异性噪声 · SMLM
Summary 本文提出了一种用于处理高各向异性定位噪声的多视角点云配准方法。该方法基于高斯混合模型（GMM）重建框架，通过引入显式的空间变化和各向异性高斯噪声模型，将形状建模与噪声处理解耦。作者设计了一种随机EM算法，将无噪声数据视为潜在变量，并在每一步EM中提供闭式解。该方法能够处理任意协方差结构的噪声，并可融合来自物理传感器的先验知识。实验表明，该策略在模拟和真实的单分子定位显微镜（SMLM）数据上显著提升了对高强度各向异性噪声的鲁棒性。
Reason 该论文研究的是点云配准问题，属于3D视觉和显微镜图像处理领域，与文档图像理解（DIU）或视觉语言模型（VLM）无关。其核心方法针对的是物理传感器中的各向异性噪声建模，并不涉及文本识别、布局分析、表格理解或OCR相关挑战，无法迁移解决DIU中的OCR幻觉、密集文本、复杂排版等痛点。
Abstract In this paper, we address the problem of registering multiple point clouds corrupted with high anisotropic localization noise. Our approach follows the widely used framework of Gaussian mixture model (GMM) reconstruction with an expectation-maximization (EM) algorithm. Existing methods are based on an implicit assumption of space-invariant isotropic Gaussian noise. However, this assumption is violated in practice in applications such as single molecule localization microscopy (SMLM). To address this issue, we propose to introduce an explicit localization noise model that decouples shape modeling with the GMM from noise handling. We design a stochastic EM algorithm that considers noise-free data as a latent variable, with closed-form solutions at each EM step. The first advantage of our approach is to handle space-variant and anisotropic Gaussian noise with arbitrary covariances. The second advantage is to leverage the explicit noise model to impose prior knowledge about the noise that may be available from physical sensors. We show on various simulated data that our noise handling strategy improves significantly the robustness to high levels of anisotropic noise. We also demonstrate the performance of our method on real SMLM data.
Authors Denis Fortun, Etienne Baudrier, Fabian Zwettler, Markus Sauer, Sylvain Faisan
Categories Computer Vision and Pattern Recognition
0
IDSplat: Instance-Decomposed 3D Gaussian Splatting for Driving Scenes
Title CN IDSplat：用于驾驶场景的实例分解3D高斯点阵化
Keywords 3D重建 · 高斯点阵化 · 实例分解 · 自动驾驶 · 自监督学习
Summary 本文提出IDSplat，一种自监督的3D高斯点阵化框架，用于在无需人工标注的情况下实现动态驾驶场景的实例级分解与运动轨迹建模。通过零样本语言引导的视频跟踪结合激光雷达信息，将动态物体建模为刚性变换的连贯实例，并引入协调转向平滑策略优化运动轨迹一致性。在Waymo开放数据集上验证了方法的有效性。
Reason 该论文属于3D视觉与自动驾驶领域，研究的是动态驾驶场景的重建，使用3D Gaussian Splatting进行实例分解和运动建模。其核心问题与文档图像理解（DIU）无关，不涉及文本识别、布局分析、表格理解或VLM在文档上的应用。尽管其方法涉及视觉表示学习，但迁移到文档理解场景（如OCR、DocVQA）极为困难，且未触及DIU的核心痛点如幻觉、密集文本处理或复杂排版理解。
Abstract Reconstructing dynamic driving scenes is essential for developing autonomous systems through sensor-realistic simulation. Although recent methods achieve high-fidelity reconstructions, they either rely on costly human annotations for object trajectories or use time-varying representations without explicit object-level decomposition, leading to intertwined static and dynamic elements that hinder scene separation. We present IDSplat, a self-supervised 3D Gaussian Splatting framework that reconstructs dynamic scenes with explicit instance decomposition and learnable motion trajectories, without requiring human annotations. Our key insight is to model dynamic objects as coherent instances undergoing rigid transformations, rather than unstructured time-varying primitives. For instance decomposition, we employ zero-shot, language-grounded video tracking anchored to 3D using lidar, and estimate consistent poses via feature correspondences. We introduce a coordinated-turn smoothing scheme to obtain temporally and physically consistent motion trajectories, mitigating pose misalignments and tracking failures, followed by joint optimization of object poses and Gaussian parameters. Experiments on the Waymo Open Dataset demonstrate that our method achieves competitive reconstruction quality while maintaining instance-level decomposition and generalizes across diverse sequences and view densities without retraining, making it practical for large-scale autonomous driving applications. Code will be released.
Authors Carl Lindström, Mahan Rafidashti, Maryam Fatemi, Lars Hammarstrand, Martin R. Oswald, Lennart Svensson
Categories Computer Vision and Pattern Recognition
0
Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving
Title CN Percept-WAM：用于鲁棒端到端自动驾驶的感知增强型世界感知-动作模型
Keywords 视觉-语言模型 · 自动驾驶 · 空间感知 · BEV检测 · 端到端学习
Summary 本文提出Percept-WAM，一种融合2D/3D场景理解能力的视觉-语言模型，用于提升自动驾驶中的空间感知与动作决策。通过引入World-PV和World-BEV token统一表示空间信息，并采用网格条件预测机制实现密集物体检测，在COCO和nuScenes等基准上达到领先性能。同时支持开放词汇识别与长尾场景泛化。
Reason 该论文属于具身智能与自动驾驶领域，尽管其方法涉及视觉-语言模型在空间感知上的改进，但核心任务与文档图像理解（DIU）无关。其所解决的3D场景理解、BEV检测、轨迹规划等问题无法直接迁移至文档中的OCR去幻觉、复杂排版解析或长文档推理等核心痛点。
Abstract Autonomous driving heavily relies on accurate and robust spatial perception. Many failures arise from inaccuracies and instability, especially in long-tail scenarios and complex interactions. However, current vision-language models are weak at spatial grounding and understanding, and VLA systems built on them therefore show limited perception and localization ability. To address these challenges, we introduce Percept-WAM, a perception-enhanced World-Awareness-Action Model that is the first to implicitly integrate 2D/3D scene understanding abilities within a single vision-language model (VLM). Instead of relying on QA-style spatial reasoning, Percept-WAM unifies 2D/3D perception tasks into World-PV and World-BEV tokens, which encode both spatial coordinates and confidence. We propose a grid-conditioned prediction mechanism for dense object perception, incorporating IoU-aware scoring and parallel autoregressive decoding, improving stability in long-tail, far-range, and small-object scenarios. Additionally, Percept-WAM leverages pretrained VLM parameters to retain general intelligence (e.g., logical reasoning) and can output perception results and trajectory control outputs directly. Experiments show that Percept-WAM matches or surpasses classical detectors and segmenters on downstream perception benchmarks, achieving 51.7/58.9 mAP on COCO 2D detection and nuScenes BEV 3D detection. When integrated with trajectory decoders, it further improves planning performance on nuScenes and NAVSIM, e.g., surpassing DiffusionDrive by 2.1 in PMDS on NAVSIM. Qualitative results further highlight its strong open-vocabulary and long-tail generalization.
Authors Jianhua Han, Meng Tian, Jiangtong Zhu, Fan He, Huixin Zhang, Sitong Guo, Dechang Zhu, Hao Tang, Pei Xu, Yuze Guo, Minzhe Niu, Haojie Zhu, Qichao Dong, Xuechao Yan, Siyuan Dong, Lu Hou, Qingqiu Huang, Xiaosong Jia, Hang Xu
Categories Computer Vision and Pattern Recognition
0
ReAlign: Text-to-Motion Generation via Step-Aware Reward-Guided Alignment AAAI 2026
Title CN ReAlign：通过步感知奖励引导对齐的文本到动作生成
Keywords 文本到动作生成 · 扩散模型 · 奖励引导采样 · 语义对齐 · 3D人体运动
Summary 本文提出ReAlign方法，用于提升文本到3D人体动作生成中的语义一致性与动作质量。通过设计一个步感知奖励模型，在扩散去噪过程中动态评估文本与动作的对齐程度，并采用奖励引导策略优化生成路径。实验表明该方法在动作生成和检索任务上优于现有SOTA方法。
Reason 该论文研究的是文本到3D人体动作生成（Text-to-Motion），属于视频生成与具身智能交叉领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其核心方法——基于扩散模型的奖励引导采样策略——虽然在生成任务中有效，但无法迁移解决OCR幻觉、复杂排版解析、细粒度文本定位等DIU核心问题。
Abstract Text-to-motion generation, which synthesizes 3D human motions from text inputs, holds immense potential for applications in gaming, film, and robotics. Recently, diffusion-based methods have been shown to generate more diversity and realistic motion. However, there exists a misalignment between text and motion distributions in diffusion models, which leads to semantically inconsistent or low-quality motions. To address this limitation, we propose Reward-guided sampling Alignment (ReAlign), comprising a step-aware reward model to assess alignment quality during the denoising sampling and a reward-guided strategy that directs the diffusion process toward an optimally aligned distribution. This reward model integrates step-aware tokens and combines a text-aligned module for semantic consistency and a motion-aligned module for realism, refining noisy motions at each timestep to balance probability density and alignment. Extensive experiments of both motion generation and retrieval tasks demonstrate that our approach significantly improves text-motion alignment and motion quality compared to existing state-of-the-art methods.
Authors Wanjiang Weng, Xiaofeng Tan, Junbo Wang, Guo-Sen Xie, Pan Zhou, Hongsong Wang
Comment Accepted by AAAI 2026
Categories Computer Vision and Pattern Recognition
0
NVGS: Neural Visibility for Occlusion Culling in 3D Gaussian Splatting
Title CN NVGS：用于3D高斯点阵中遮挡剔除的神经可见性
Keywords 3D Gaussian Splatting · Occlusion Culling · Neural Visibility · Rendering Optimization · Tensor Cores
Summary 本文提出了一种基于小型共享MLP的神经可见性方法（NVGS），用于在3D高斯点阵渲染中学习视点相关的可见性函数，实现对被遮挡高斯图元的剔除，从而提升渲染效率并降低显存占用。该方法集成于实例化软件光栅化器中，并利用Tensor Core加速计算，在组合场景中优于现有方法。
Reason 该论文研究的是3D高斯点阵渲染中的遮挡剔除问题，属于3D视觉与图形学领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其核心技术聚焦于渲染优化和场景加速，无法迁移至解决OCR幻觉、复杂排版理解、细粒度定位等DIU核心痛点。
Abstract 3D Gaussian Splatting can exploit frustum culling and level-of-detail strategies to accelerate rendering of scenes containing a large number of primitives. However, the semi-transparent nature of Gaussians prevents the application of another highly effective technique: occlusion culling. We address this limitation by proposing a novel method to learn the viewpoint-dependent visibility function of all Gaussians in a trained model using a small, shared MLP across instances of an asset in a scene. By querying it for Gaussians within the viewing frustum prior to rasterization, our method can discard occluded primitives during rendering. Leveraging Tensor Cores for efficient computation, we integrate these neural queries directly into a novel instanced software rasterizer. Our approach outperforms the current state of the art for composed scenes in terms of VRAM usage and image quality, utilizing a combination of our instanced rasterizer and occlusion culling MLP, and exhibits complementary properties to existing LoD techniques.
Authors Brent Zoomers, Florian Hahlbohm, Joni Vanherck, Lode Jorissen, Marcus Magnor, Nick Michiels
Comment 15 pages, 13 figures
Categories Computer Vision and Pattern Recognition
0
CUPID: Generative 3D Reconstruction via Joint Object and Pose Modeling
Title CN CUPID：通过联合物体与姿态建模实现生成式3D重建
Keywords 3D重建 · 生成模型 · 相机姿态估计 · 两阶段建模 · 多视角重建
Summary 本文提出CUPID，一种通过联合建模规范物体与相机姿态分布的生成式3D重建框架。该方法采用两阶段基于流的模型：第一阶段生成粗略3D结构和2D-3D对应关系以鲁棒估计相机姿态；第二阶段在该姿态条件下，将像素对齐的图像特征注入生成过程，结合生成模型的先验与重建的几何保真度。此策略在PSNR上超过现有最先进方法3 dB以上，Chamfer Distance提升10%。作为一个统一的生成模型，CUPID自然支持无需后处理优化或多任务微调的多视角与场景级重建。
Reason 该论文研究的是生成式3D重建，属于3D视觉与计算机图形学领域，核心任务为多视角几何、相机姿态估计和3D形状生成。其方法聚焦于3D场景的生成建模，并不涉及文档图像理解（DIU）、文本识别、布局分析或VLM在文档上的应用。完全偏离文档图像理解的核心问题，属于无关领域。
Abstract We introduce Cupid, a generative 3D reconstruction framework that jointly models the full distribution over both canonical objects and camera poses. Our two-stage flow-based model first generates a coarse 3D structure and 2D-3D correspondences to estimate the camera pose robustly. Conditioned on this pose, a refinement stage injects pixel-aligned image features directly into the generative process, marrying the rich prior of a generative model with the geometric fidelity of reconstruction. This strategy achieves exceptional faithfulness, outperforming state-of-the-art reconstruction methods by over 3 dB PSNR and 10% in Chamfer Distance. As a unified generative model that decouples the object and camera pose, Cupid naturally extends to multi-view and scene-level reconstruction tasks without requiring post-hoc optimization or fine-tuning.
Authors Binbin Huang, Haobin Duan, Yiqun Zhao, Zibo Zhao, Yi Ma, Shenghua Gao
Comment project page at https://cupid3d.github.io
Categories Computer Vision and Pattern Recognition
0
Three-Dimensional Anatomical Data Generation Based on Artificial Neural Networks IEEE International Conference on Intelligent Robots and Systems (IROS)
Title CN 基于人工神经网络的三维解剖数据生成
Keywords 3D生成对抗网络 · 医学图像分割 · 手术模拟 · 物理器官模型 · 超声成像
Summary 本文提出了一种基于物理器官模型和3D GAN的自动化三维解剖数据生成流程，用于克服真实医学数据获取的伦理与技术瓶颈。通过超声扫描人工前列腺模型并训练神经网络进行图像分割，进而重建3D网格模型，支持手术模拟与性能反馈。重点在于3D医学数据生成与机器人应用。
Reason 该论文属于3D视觉与具身智能交叉领域，研究内容为基于物理器官模型和GAN生成三维解剖结构，应用于手术模拟与机器人系统。尽管使用了深度学习和图像分割技术，但其核心任务是3D医学建模与生成，而非文档图像理解（DIU）或文档视觉问答（DocVQA）。不涉及OCR、布局分析、表格识别或任何文档相关任务，也无法迁移用于解决文档中的幻觉、复杂排版或长文本推理等核心痛点。
Abstract Surgical planning and training based on machine learning requires a large amount of 3D anatomical models reconstructed from medical imaging, which is currently one of the major bottlenecks. Obtaining these data from real patients and during surgery is very demanding, if even possible, due to legal, ethical, and technical challenges. It is especially difficult for soft tissue organs with poor imaging contrast, such as the prostate. To overcome these challenges, we present a novel workflow for automated 3D anatomical data generation using data obtained from physical organ models. We additionally use a 3D Generative Adversarial Network (GAN) to obtain a manifold of 3D models useful for other downstream machine learning tasks that rely on 3D data. We demonstrate our workflow using an artificial prostate model made of biomimetic hydrogels with imaging contrast in multiple zones. This is used to physically simulate endoscopic surgery. For evaluation and 3D data generation, we place it into a customized ultrasound scanner that records the prostate before and after the procedure. A neural network is trained to segment the recorded ultrasound images, which outperforms conventional, non-learning-based computer vision techniques in terms of intersection over union (IoU). Based on the segmentations, a 3D mesh model is reconstructed, and performance feedback is provided.
Authors Ann-Sophia Müller, Moonkwang Jeong, Meng Zhang, Jiyuan Tian, Arkadiusz Miernik, Stefanie Speidel, Tian Qiu
Comment 6 pages, 4 figures, 1 table, IEEE International Conference on Intelligent Robots and Systems (IROS)
Categories Computer Vision and Pattern Recognition
0
Don't Reach for the Stars: Rethinking Topology for Resilient Federated Learning
Title CN 不要追求星辰：为弹性联邦学习重新思考拓扑结构
Keywords 联邦学习 · 去中心化拓扑 · 客户端个性化 · 模型聚合 · 语义一致性
Summary 本文提出了一种去中心化的联邦学习框架LIGHTYEAR，采用点对点拓扑结构，使每个客户端能基于本地验证集计算‘一致性分数’，选择语义上对齐的模型更新，并结合正则化项进行聚合。该方法在异构和对抗性条件下表现出优于集中式和其他P2P方法的性能。然而，其研究目标和技术创新均不在文档理解或多模态建模范畴内。
Reason 该论文研究的是联邦学习中的去中心化拓扑结构问题，属于机器学习系统与分布式训练优化领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。尽管其提出的LIGHTYEAR框架在客户端个性化聚合方面有创新，但方法聚焦于模型更新的语义对齐和鲁棒性训练，无法迁移解决DIU中的OCR幻觉、细粒度定位、复杂排版解析或长文档推理等核心痛点。
Abstract Federated learning (FL) enables collaborative model training across distributed clients while preserving data privacy by keeping data local. Traditional FL approaches rely on a centralized, star-shaped topology, where a central server aggregates model updates from clients. However, this architecture introduces several limitations, including a single point of failure, limited personalization, and poor robustness to distribution shifts or vulnerability to malfunctioning clients. Moreover, update selection in centralized FL often relies on low-level parameter differences, which can be unreliable when client data is not independent and identically distributed, and offer clients little control. In this work, we propose a decentralized, peer-to-peer (P2P) FL framework. It leverages the flexibility of the P2P topology to enable each client to identify and aggregate a personalized set of trustworthy and beneficial updates.This framework is the Local Inference Guided Aggregation for Heterogeneous Training Environments to Yield Enhancement Through Agreement and Regularization (LIGHTYEAR). Central to our method is an agreement score, computed on a local validation set, which quantifies the semantic alignment of incoming updates in the function space with respect to the clients reference model. Each client uses this score to select a tailored subset of updates and performs aggregation with a regularization term that further stabilizes the training. Our empirical evaluation across five datasets shows that the proposed approach consistently outperforms both, centralized baselines and existing P2P methods in terms of client-level performance, particularly under adversarial and heterogeneous conditions.
Authors Mirko Konstantin, Anirban Mukhopadhyay
Categories Computer Vision and Pattern Recognition
0
SpectraNet: FFT-assisted Deep Learning Classifier for Deepfake Face Detection
Title CN SpectraNet：基于FFT的深度学习分类器用于深度伪造人脸检测
Keywords Deepfake Detection · FFT · EfficientNet · Image Classification · Class Imbalance
Summary 本文提出了一种基于EfficientNet-B6的轻量级深度伪造人脸检测模型SpectraNet，通过数据增强、过采样和优化策略应对类别不平衡问题。虽然尝试引入傅里叶变换的相位和幅度特征，但对性能提升有限。目标是帮助非专业人士识别假脸图像。
Reason 该论文属于图像伪造检测领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了傅里叶变换辅助分类，但其方法针对的是深度伪造人脸检测任务，无法迁移解决OCR幻觉、复杂排版理解、长文档推理等DIU核心问题。属于无关领域。
Abstract Detecting deepfake images is crucial in combating misinformation. We present a lightweight, generalizable binary classification model based on EfficientNet-B6, fine-tuned with transformation techniques to address severe class imbalances. By leveraging robust preprocessing, oversampling, and optimization strategies, our model achieves high accuracy, stability, and generalization. While incorporating Fourier transform-based phase and amplitude features showed minimal impact, our proposed framework helps non-experts to effectively identify deepfake images, making significant strides toward accessible and reliable deepfake detection.
Authors Nithira Jayarathne, Naveen Basnayake, Keshawa Jayasundara, Pasindu Dodampegama, Praveen Wijesinghe, Hirushika Pelagewatta, Kavishka Abeywardana, Sandushan Ranaweera, Chamira Edussooriya
Comment 4 pages, 3 figures
Categories Computer Vision and Pattern Recognition
0
nnActive: A Framework for Evaluation of Active Learning in 3D Biomedical Segmentation TMLR
Title CN nnActive：一种用于评估3D生物医学分割中主动学习的框架
Keywords 主动学习 · 3D医学图像分割 · nnU-Net · 标注效率 · 前景感知采样
Summary 本文提出了nnActive，一个用于评估3D生物医学图像分割中主动学习（AL）方法的开源框架。作者指出现有AL评估存在四个主要缺陷，并通过大规模实验、改进的随机采样策略（Foreground Aware Random）和新指标（foreground efficiency）系统性地解决了这些问题。结果显示，当前AL方法在改进后的基线面前优势有限，且性能受任务参数影响显著。
Reason 该论文研究的是3D生物医学图像分割中的主动学习评估问题，属于医学图像分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法聚焦于3D医学影像的标注效率和采样策略优化，无法迁移解决文档图像中的OCR幻觉、复杂排版、细粒度定位或长文档推理等核心痛点。
Abstract Semantic segmentation is crucial for various biomedical applications, yet its reliance on large annotated datasets presents a bottleneck due to the high cost and specialized expertise required for manual labeling. Active Learning (AL) aims to mitigate this challenge by querying only the most informative samples, thereby reducing annotation effort. However, in the domain of 3D biomedical imaging, there is no consensus on whether AL consistently outperforms Random sampling. Four evaluation pitfalls hinder the current methodological assessment. These are (1) restriction to too few datasets and annotation budgets, (2) using 2D models on 3D images without partial annotations, (3) Random baseline not being adapted to the task, and (4) measuring annotation cost only in voxels. In this work, we introduce nnActive, an open-source AL framework that overcomes these pitfalls by (1) means of a large scale study spanning four biomedical imaging datasets and three label regimes, (2) extending nnU-Net by using partial annotations for training with 3D patch-based query selection, (3) proposing Foreground Aware Random sampling strategies tackling the foreground-background class imbalance of medical images and (4) propose the foreground efficiency metric, which captures the low annotation cost of background-regions. We reveal the following findings: (A) while all AL methods outperform standard Random sampling, none reliably surpasses an improved Foreground Aware Random sampling; (B) benefits of AL depend on task specific parameters; (C) Predictive Entropy is overall the best performing AL method, but likely requires the most annotation effort; (D) AL performance can be improved with more compute intensive design choices. As a holistic, open-source framework, nnActive can serve as a catalyst for research and application of AL in 3D biomedical imaging. Code is at: https://github.com/MIC-DKFZ/nnActive
Authors Carsten T. Lüth, Jeremias Traub, Kim-Celine Kahl, Till J. Bungert, Lukas Klein, Lars Krämer, Paul F. Jaeger, Fabian Isensee, Klaus Maier-Hein
Comment Accepted at TMLR
Categories Computer Vision and Pattern Recognition
0
Evaluating Deep Learning and Traditional Approaches Used in Source Camera Identification
Title CN 评估用于源相机识别的深度学习和传统方法
Keywords 源相机识别 · PRNU · JPEG压缩伪影 · CNN · 图像取证
Summary 本文对比了三种源相机识别技术：光电响应非均匀性（PRNU）、JPEG压缩伪影分析和卷积神经网络（CNN），并从设备分类准确率角度进行评估。同时讨论了这些方法在实际应用中的可行性与发展需求。
Reason 该论文研究的是源相机识别（SCI），属于图像取证领域，与文档图像理解（DIU）或DocVQA无关。其技术路线聚焦于PRNU、JPEG压缩伪影分析和CNN分类器，无法迁移到解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点。
Abstract One of the most important tasks in computer vision is identifying the device using which the image was taken, useful for facilitating further comprehensive analysis of the image. This paper presents comparative analysis of three techniques used in source camera identification (SCI): Photo Response Non-Uniformity (PRNU), JPEG compression artifact analysis, and convolutional neural networks (CNNs). It evaluates each method in terms of device classification accuracy. Furthermore, the research discusses the possible scientific development needed for the implementation of the methods in real-life scenarios.
Authors Mansur Ozaman
Comment 4 figures
Categories Computer Vision and Pattern Recognition
0
MetroGS: Efficient and Stable Reconstruction of Geometrically Accurate High-Fidelity Large-Scale Scenes
Title CN MetroGS：高效且稳定的几何精确高保真大规模场景重建
Keywords 3D Gaussian Splatting · 大规模重建 · 几何优化 · SfM · 外观一致性
Summary 本文提出MetroGS，一种基于分布式2D高斯泼溅表示的大规模城市环境高保真重建框架。通过结构化增强策略、混合几何优化和深度引导的外观建模，提升几何精度与渲染稳定性。适用于大型城市场景的3D重建，但与文档图像理解无关。
Reason 该论文属于3D视觉与大规模场景重建领域，核心方法围绕3D Gaussian Splatting的优化，涉及SfM、深度引导外观建模等技术，完全不涉及文档图像理解（DIU）、多模态语言模型（VLM）或文本密集场景下的OCR、布局分析、表格识别等任务。属于负面清单中的‘无关领域’——3D视觉与自动驾驶相关方向，无法迁移至文档图像理解的核心痛点（如OCR幻觉、复杂排版、长文档推理）。
Abstract Recently, 3D Gaussian Splatting and its derivatives have achieved significant breakthroughs in large-scale scene reconstruction. However, how to efficiently and stably achieve high-quality geometric fidelity remains a core challenge. To address this issue, we introduce MetroGS, a novel Gaussian Splatting framework for efficient and robust reconstruction in complex urban environments. Our method is built upon a distributed 2D Gaussian Splatting representation as the core foundation, serving as a unified backbone for subsequent modules. To handle potential sparse regions in complex scenes, we propose a structured dense enhancement scheme that utilizes SfM priors and a pointmap model to achieve a denser initialization, while incorporating a sparsity compensation mechanism to improve reconstruction completeness. Furthermore, we design a progressive hybrid geometric optimization strategy that organically integrates monocular and multi-view optimization to achieve efficient and accurate geometric refinement. Finally, to address the appearance inconsistency commonly observed in large-scale scenes, we introduce a depth-guided appearance modeling approach that learns spatial features with 3D consistency, facilitating effective decoupling between geometry and appearance and further enhancing reconstruction stability. Experiments on large-scale urban datasets demonstrate that MetroGS achieves superior geometric accuracy, rendering quality, offering a unified solution for high-fidelity large-scale scene reconstruction.
Authors Kehua Chen, Tianlu Mao, Zhuxin Ma, Hao Jiang, Zehao Li, Zihan Liu, Shuqi Gao, Honglong Zhao, Feng Dai, Yucheng Zhang, Zhaoqi Wang
Comment Project page: https://m3phist0.github.io/MetroGS
Categories Computer Vision and Pattern Recognition
0
U-REPA: Aligning Diffusion U-Nets to ViTs
Title CN U-REPA：将扩散U-Net与ViT对齐
Keywords 扩散模型 · U-Net · ViT · 表示对齐 · 图像生成
Summary 本文提出U-REPA，一种用于在扩散U-Net和ViT之间进行表示对齐的新方法，通过中间阶段对齐、MLP后上采样和流形损失来解决结构差异问题，在ImageNet生成任务上实现了更快的收敛和更高的生成质量。
Reason 该论文研究的是图像生成领域中扩散模型（U-Net架构）与视觉Transformer（ViT）之间的表示对齐问题，属于纯图像生成方向。其核心目标是加速生成模型的训练收敛并提升生成质量，与文档图像理解（DIU）中的OCR、布局分析、表格识别或VQA等任务无直接关联。尽管涉及视觉表示学习，但方法针对的是生成式建模中的特征对齐，无法迁移解决DIU中的OCR幻觉、密集文本识别、复杂排版解析或长文档推理等核心痛点。
Abstract Representation Alignment (REPA) that aligns Diffusion Transformer (DiT) hidden-states with ViT visual encoders has proven highly effective in DiT training, demonstrating superior convergence properties, but it has not been validated on the canonical diffusion U-Net architecture that shows faster convergence compared to DiTs. However, adapting REPA to U-Net architectures presents unique challenges: (1) different block functionalities necessitate revised alignment strategies; (2) spatial-dimension inconsistencies emerge from U-Net's spatial downsampling operations; (3) space gaps between U-Net and ViT hinder the effectiveness of tokenwise alignment. To encounter these challenges, we propose \textbf{U-REPA}, a representation alignment paradigm that bridges U-Net hidden states and ViT features as follows: Firstly, we propose via observation that due to skip connection, the middle stage of U-Net is the best alignment option. Secondly, we propose upsampling of U-Net features after passing them through MLPs. Thirdly, we observe difficulty when performing tokenwise similarity alignment, and further introduces a manifold loss that regularizes the relative similarity between samples. Experiments indicate that the resulting U-REPA could achieve excellent generation quality and greatly accelerates the convergence speed. With CFG guidance interval, U-REPA could reach $FID<1.5$ in 200 epochs or 1M iterations on ImageNet 256 $\times$ 256, and needs only half the total epochs to perform better than REPA under sd-vae-ft-ema. Codes: https://github.com/YuchuanTian/U-REPA
Authors Yuchuan Tian, Hanting Chen, Mengyu Zheng, Yuchen Liang, Chao Xu, Yunhe Wang
Comment 22 pages, 7 figures
Categories Computer Vision and Pattern Recognition
0
InstantViR: Real-Time Video Inverse Problem Solver with Distilled Diffusion Prior
Title CN InstantViR：基于蒸馏扩散先验的实时视频逆问题求解器
Keywords 视频逆问题 · 扩散模型 · 知识蒸馏 · 实时重建 · LeanVAE
Summary 本文提出InstantViR，一种基于预训练视频扩散模型的快速推理框架，通过将双向视频扩散模型蒸馏为因果自回归模型，实现单次前向传播完成视频恢复，在保持高质量的同时达到35 FPS以上的实时性能。适用于流媒体修复、去模糊和超分辨率等场景。
Reason 该论文研究的是视频逆问题（如去模糊、超分辨率）的实时重建，属于视频处理与生成领域，核心方法为利用扩散模型先验进行知识蒸馏。尽管其高效率推理设计有一定技术价值，但应用场景和任务目标均与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版理解或DocVQA等核心痛点。
Abstract Video inverse problems are fundamental to streaming, telepresence, and AR/VR, where high perceptual quality must coexist with tight latency constraints. Diffusion-based priors currently deliver state-of-the-art reconstructions, but existing approaches either adapt image diffusion models with ad hoc temporal regularizers - leading to temporal artifacts - or rely on native video diffusion models whose iterative posterior sampling is far too slow for real-time use. We introduce InstantViR, an amortized inference framework for ultra-fast video reconstruction powered by a pre-trained video diffusion prior. We distill a powerful bidirectional video diffusion model (teacher) into a causal autoregressive student that maps a degraded video directly to its restored version in a single forward pass, inheriting the teacher's strong temporal modeling while completely removing iterative test-time optimization. The distillation is prior-driven: it only requires the teacher diffusion model and known degradation operators, and does not rely on externally paired clean/noisy video data. To further boost throughput, we replace the video-diffusion backbone VAE with a high-efficiency LeanVAE via an innovative teacher-space regularized distillation scheme, enabling low-latency latent-space processing. Across streaming random inpainting, Gaussian deblurring and super-resolution, InstantViR matches or surpasses the reconstruction quality of diffusion-based baselines while running at over 35 FPS on NVIDIA A100 GPUs, achieving up to 100 times speedups over iterative video diffusion solvers. These results show that diffusion-based video reconstruction is compatible with real-time, interactive, editable, streaming scenarios, turning high-quality video restoration into a practical component of modern vision systems.
Authors Weimin Bai, Suzhe Xu, Yiwei Ren, Jinhua Hao, Ming Sun, Wenzheng Chen, He Sun
Categories Computer Vision and Pattern Recognition
0
FilmSceneDesigner: Chaining Set Design for Procedural Film Scene Generation
Title CN FilmSceneDesigner：用于程序化电影场景生成的链式布景设计
Keywords 电影布景设计 · 程序化生成 · Agent链 · 3D场景生成 · 多模态生成
Summary 本文提出FilmSceneDesigner，一个基于Agent链的自动化电影场景生成系统，能够根据自然语言描述生成具有电影真实感的完整3D场景。系统通过结构化参数建模布景设计流程，并结合专用函数执行平面图生成、材质分配、门窗布置和物体布局。作者还构建了包含6862个电影级3D资产的数据集SetDepot-Pro。实验表明该系统在结构合理性和电影保真度方面表现良好。
Reason 该论文属于具身智能与3D场景生成领域，研究目标是基于自然语言描述自动生成电影布景，涉及3D资产构建、 procedural generation 和虚拟场景布局。其核心技术（如代理链框架、材质分配、物体摆放）与文档图像理解（DIU）的核心任务（如文本识别、版面分析、表格抽取、VQA）无直接关联。尽管使用了多模态输入和Agent架构，但方法面向的是三维视觉与创意内容生成，无法迁移解决OCR幻觉、密集文本处理或复杂排版理解等DIU痛点。
Abstract Film set design plays a pivotal role in cinematic storytelling and shaping the visual atmosphere. However, the traditional process depends on expert-driven manual modeling, which is labor-intensive and time-consuming. To address this issue, we introduce FilmSceneDesigner, an automated scene generation system that emulates professional film set design workflow. Given a natural language description, including scene type, historical period, and style, we design an agent-based chaining framework to generate structured parameters aligned with film set design workflow, guided by prompt strategies that ensure parameter accuracy and coherence. On the other hand, we propose a procedural generation pipeline which executes a series of dedicated functions with the structured parameters for floorplan and structure generation, material assignment, door and window placement, and object retrieval and layout, ultimately constructing a complete film scene from scratch. Moreover, to enhance cinematic realism and asset diversity, we construct SetDepot-Pro, a curated dataset of 6,862 film-specific 3D assets and 733 materials. Experimental results and human evaluations demonstrate that our system produces structurally sound scenes with strong cinematic fidelity, supporting downstream tasks such as virtual previs, construction drawing and mood board creation.
Authors Zhifeng Xie, Keyi Zhang, Yiye Yan, Yuling Guo, Fan Yang, Jiting Zhou, Mengtian Li
Categories Computer Vision and Pattern Recognition
0
MambaRefine-YOLO: A Dual-Modality Small Object Detector for UAV Imagery
Title CN MambaRefine-YOLO：一种用于无人机图像的双模态小目标检测器
Keywords 小目标检测 · 无人机图像 · 多模态融合 · 红外图像 · YOLO
Summary 本文提出MambaRefine-YOLO，用于提升无人机图像中的小目标检测性能。通过引入双门控互补Mamba融合模块（DGC-MFM）和分层特征聚合颈（HFAN），在多模态DroneVehicle和单模态VisDrone数据集上实现了性能提升。方法侧重于遥感场景下的跨模态融合与特征细化。
Reason 该论文研究的是无人机遥感图像中的小目标检测问题，属于遥感与红外融合检测领域，任务为RGB-IR多模态目标检测，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心技术如DGC-MFM和HFAN聚焦于跨模态特征融合与多尺度增强，但应用场景、数据模态（遥感、红外）、任务目标均与文档理解无关，无法迁移解决OCR幻觉、复杂排版、长文档推理等核心痛点。
Abstract Small object detection in Unmanned Aerial Vehicle (UAV) imagery is a persistent challenge, hindered by low resolution and background clutter. While fusing RGB and infrared (IR) data offers a promising solution, existing methods often struggle with the trade-off between effective cross-modal interaction and computational efficiency. In this letter, we introduce MambaRefine-YOLO. Its core contributions are a Dual-Gated Complementary Mamba fusion module (DGC-MFM) that adaptively balances RGB and IR modalities through illumination-aware and difference-aware gating mechanisms, and a Hierarchical Feature Aggregation Neck (HFAN) that uses a ``refine-then-fuse'' strategy to enhance multi-scale features. Our comprehensive experiments validate this dual-pronged approach. On the dual-modality DroneVehicle dataset, the full model achieves a state-of-the-art mAP of 83.2%, an improvement of 7.9% over the baseline. On the single-modality VisDrone dataset, a variant using only the HFAN also shows significant gains, demonstrating its general applicability. Our work presents a superior balance between accuracy and speed, making it highly suitable for real-world UAV applications.
Authors Shuyu Cao, Minxin Chen, Yucheng Song, Zhaozhong Chen, Xinyou Zhang
Comment Submitted to IEEE Geoscience and Remote Sensing Letters
Categories Computer Vision and Pattern Recognition
0
When Semantics Regulate: Rethinking Patch Shuffle and Internal Bias for Generated Image Detection with CLIP
Title CN 当语义被调控时：利用CLIP重新思考图像生成检测中的块打乱与内部偏置
Keywords CLIP · 生成图像检测 · Patch Shuffle · 语义偏置 · 跨域泛化
Summary 本文发现CLIP在检测AI生成图像时依赖语义线索而非生成器伪影，导致跨域泛化能力弱。作者提出通过Patch Shuffle破坏语义连续性，并设计SemAnti微调范式，冻结语义子空间、仅训练对伪影敏感的层，从而提升跨域检测性能。实验表明该方法在AIGCDetectBenchmark和GenImage上达到SOTA。
Reason 该论文研究的是AI生成图像检测（AIGC Detection），属于纯图像生成/伪造检测领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了CLIP和视觉语义分析，但其方法针对的是生成模型的伪影识别，无法迁移解决OCR幻觉、复杂排版理解、细粒度定位等DIU核心问题。
Abstract The rapid progress of GANs and Diffusion Models poses new challenges for detecting AI-generated images. Although CLIP-based detectors exhibit promising generalization, they often rely on semantic cues rather than generator artifacts, leading to brittle performance under distribution shifts. In this work, we revisit the nature of semantic bias and uncover that Patch Shuffle provides an unusually strong benefit for CLIP, that disrupts global semantic continuity while preserving local artifact cues, which reduces semantic entropy and homogenizes feature distributions between natural and synthetic images. Through a detailed layer-wise analysis, we further show that CLIP's deep semantic structure functions as a regulator that stabilizes cross-domain representations once semantic bias is suppressed. Guided by these findings, we propose SemAnti, a semantic-antagonistic fine-tuning paradigm that freezes the semantic subspace and adapts only artifact-sensitive layers under shuffled semantics. Despite its simplicity, SemAnti achieves state-of-the-art cross-domain generalization on AIGCDetectBenchmark and GenImage, demonstrating that regulating semantics is key to unlocking CLIP's full potential for robust AI-generated image detection.
Authors Beilin Chu, Weike You, Mengtao Li, Tingting Zheng, Kehan Zhao, Xuan Xu, Zhigao Lu, Jia Song, Moxuan Xu, Linna Zhou
Comment 14 pages, 7 figures and 7 tables
Categories Computer Vision and Pattern Recognition
0
HABIT: Human Action Benchmark for Interactive Traffic in CARLA WACV 2026
Title CN HABIT：CARLA中交互交通的人类行为基准
Keywords 自动驾驶 · 人类行为建模 · CARLA · 动作重定向 · 安全评估
Summary 本文提出了HABIT，一个用于自动驾驶仿真的高保真人类行为基准，通过将真实世界的人类动作数据（来自动捕和视频）映射到CARLA模拟器中，以生成更真实、多样化的行人交互场景。该基准揭示了现有自动驾驶系统在应对复杂人类行为时的严重缺陷，即使在传统指标上表现良好，仍会出现高碰撞率和不必要的刹车行为。所有组件均已开源。
Reason 该论文属于自动驾驶仿真领域，研究重点是人类行为建模在交通场景中的应用，与文档图像理解（DIU）或视觉语言模型在文档任务上的应用完全无关。尽管使用了计算机视觉技术，但其方法和目标均无法迁移到OCR、DocVQA、布局分析等核心文档理解问题。
Abstract Current autonomous driving (AD) simulations are critically limited by their inadequate representation of realistic and diverse human behavior, which is essential for ensuring safety and reliability. Existing benchmarks often simplify pedestrian interactions, failing to capture complex, dynamic intentions and varied responses critical for robust system deployment. To overcome this, we introduce HABIT (Human Action Benchmark for Interactive Traffic), a high-fidelity simulation benchmark. HABIT integrates real-world human motion, sourced from mocap and videos, into CARLA (Car Learning to Act, a full autonomous driving simulator) via a modular, extensible, and physically consistent motion retargeting pipeline. From an initial pool of approximately 30,000 retargeted motions, we curate 4,730 traffic-compatible pedestrian motions, standardized in SMPL format for physically consistent trajectories. HABIT seamlessly integrates with CARLA's Leaderboard, enabling automated scenario generation and rigorous agent evaluation. Our safety metrics, including Abbreviated Injury Scale (AIS) and False Positive Braking Rate (FPBR), reveal critical failure modes in state-of-the-art AD agents missed by prior evaluations. Evaluating three state-of-the-art autonomous driving agents, InterFuser, TransFuser, and BEVDriver, demonstrates how HABIT exposes planner weaknesses that remain hidden in scripted simulations. Despite achieving close or equal to zero collisions per kilometer on the CARLA Leaderboard, the autonomous agents perform notably worse on HABIT, with up to 7.43 collisions/km and a 12.94% AIS 3+ injury risk, and they brake unnecessarily in up to 33% of cases. All components are publicly released to support reproducible, pedestrian-aware AI research.
Authors Mohan Ramesh, Mark Azer, Fabian B. Flohr
Comment Accepted to WACV 2026. This is the pre-camera-ready version
Categories Computer Vision and Pattern Recognition
0
Graph-based 3D Human Pose Estimation using WiFi Signals
Title CN 基于WiFi信号的图结构3D人体姿态估计
Keywords WiFi感知 · 3D姿态估计 · 图神经网络 · CSI · 无相机感知
Summary 本文提出GraphPose-Fi，一种基于图结构的框架，用于从WiFi信号中估计3D人体姿态。该方法通过建模人体关节间的拓扑关系，结合CNN和GCN，在MM-Fi数据集上取得了优于现有方法的表现。但该工作与文档图像理解无关。
Reason 该论文研究的是基于WiFi信号的3D人体姿态估计，属于无线感知与人体动作识别领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用完全无关。其方法无法迁移至解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点。
Abstract WiFi-based human pose estimation (HPE) has attracted increasing attention due to its resilience to occlusion and privacy-preserving compared to camera-based methods. However, existing WiFi-based HPE approaches often employ regression networks that directly map WiFi channel state information (CSI) to 3D joint coordinates, ignoring the inherent topological relationships among human joints. In this paper, we present GraphPose-Fi, a graph-based framework that explicitly models skeletal topology for WiFi-based 3D HPE. Our framework comprises a CNN encoder shared across antennas for subcarrier-time feature extraction, a lightweight attention module that adaptively reweights features over time and across antennas, and a graph-based regression head that combines GCN layers with self-attention to capture local topology and global dependencies. Our proposed method significantly outperforms existing methods on the MM-Fi dataset in various settings. The source code is available at: https://github.com/Cirrick/GraphPose-Fi.
Authors Jichao Chen, YangYang Qu, Ruibo Tang, Dirk Slock
Categories Computer Vision and Pattern Recognition
0
Splats in Splats: Robust and Effective 3D Steganography towards Gaussian Splatting AAAI 2026
Title CN 点中有点：面向高斯溅射的鲁棒高效3D隐写术
Keywords 3D高斯溅射 · 隐写术 · 球谐函数 · 版权保护 · 计算机视觉
Summary 本文提出了一种名为“点中有点”的3D高斯溅射隐写框架，通过在原始高斯图元的球谐系数中嵌入隐藏信息，并利用卷积自编码器建立透明度映射，实现无需修改属性即可将3D内容嵌入3DGS本身。实验表明该方法在场景保真度、渲染速度、安全性和鲁棒性方面优于现有技术。
Reason 该论文研究的是3D高斯溅射（3DGS）中的隐写技术，属于3D视觉与版权保护领域，与文档图像理解（DIU）、多模态大模型（VLM）在文档上的应用完全无关。其核心技术围绕球谐函数加密和透明度映射进行信息隐藏，无法迁移至解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心问题。
Abstract 3D Gaussian splatting (3DGS) has demonstrated impressive 3D reconstruction performance with explicit scene representations. Given the widespread application of 3DGS in 3D reconstruction and generation tasks, there is an urgent need to protect the copyright of 3DGS assets. However, existing copyright protection techniques for 3DGS overlook the usability of 3D assets, posing challenges for practical deployment. Here we describe splats in splats, the first 3DGS steganography framework that embeds 3D content in 3DGS itself without modifying any attributes. To achieve this, we take a deep insight into spherical harmonics (SH) and devise an importance-graded SH coefficient encryption strategy to embed the hidden SH coefficients. Furthermore, we employ a convolutional autoencoder to establish a mapping between the original Gaussian primitives' opacity and the hidden Gaussian primitives' opacity. Extensive experiments indicate that our method significantly outperforms existing 3D steganography techniques, with 5.31% higher scene fidelity and 3x faster rendering speed, while ensuring security, robustness, and user experience.
Authors Yijia Guo, Wenkai Huang, Yang Li, Gaolei Li, Hang Zhang, Liwen Hu, Jianhua Li, Tiejun Huang, Lei Ma
Comment Accepted by AAAI 2026
Categories Computer Vision and Pattern Recognition
0
Towards Generalizable Deepfake Detection via Forgery-aware Audio-Visual Adaptation: A Variational Bayesian Approach TIFS AQE
Title CN 通过伪造感知的视听自适应实现可泛化的深度伪造检测：一种变分贝叶斯方法
Keywords 深度伪造检测 · 音视频多模态 · 变分贝叶斯 · 跨模态不一致 · Forgery-aware
Summary 本文提出了一种基于变分贝叶斯的音视频深度伪造检测框架FoVB，通过建模音频-视觉相关性作为高斯隐变量，利用差异卷积和高通滤波提取局部与全局伪造痕迹，并通过正交约束分解模态特异性和相关性特征，以减少纠缠，提升检测泛化性。
Reason 该论文研究的是音视频多模态深度伪造检测问题，属于视频安全与内容真实性验证领域，与文档图像理解（DIU）或DocVQA的核心任务无直接关联。其技术路线聚焦于跨模态不一致性建模，虽涉及多模态学习，但无法迁移解决OCR幻觉、复杂排版解析、长文档推理等DIU关键痛点。
Abstract The widespread application of AIGC contents has brought not only unprecedented opportunities, but also potential security concerns, e.g., audio-visual deepfakes. Therefore, it is of great importance to develop an effective and generalizable method for multi-modal deepfake detection. Typically, the audio-visual correlation learning could expose subtle cross-modal inconsistencies, e.g., audio-visual misalignment, which serve as crucial clues in deepfake detection. In this paper, we reformulate the correlation learning with variational Bayesian estimation, where audio-visual correlation is approximated as a Gaussian distributed latent variable, and thus develop a novel framework for deepfake detection, i.e., Forgery-aware Audio-Visual Adaptation with Variational Bayes (FoVB). Specifically, given the prior knowledge of pre-trained backbones, we adopt two core designs to estimate audio-visual correlations effectively. First, we exploit various difference convolutions and a high-pass filter to discern local and global forgery traces from both modalities. Second, with the extracted forgery-aware features, we estimate the latent Gaussian variable of audio-visual correlation via variational Bayes. Then, we factorize the variable into modality-specific and correlation-specific ones with orthogonality constraint, allowing them to better learn intra-modal and cross-modal forgery traces with less entanglement. Extensive experiments demonstrate that our FoVB outperforms other state-of-the-art methods in various benchmarks.
Authors Fan Nie, Jiangqun Ni, Jian Zhang, Bin Zhang, Weizhe Zhang, Bin Li
Comment TIFS AQE
Categories Computer Vision and Pattern Recognition
0
DEAP-3DSAM: Decoder Enhanced and Auto Prompt SAM for 3D Medical Image Segmentation BIBM 2024
Title CN 用于3D医学图像分割的解码器增强与自动提示SAM
Keywords 3D医学图像分割 · SAM · 自动提示 · 特征增强解码器 · 双注意力机制
Summary 本文提出DEAP-3DSAM模型，用于解决Segment Anything Model在3D医学图像分割中因伪3D处理导致的空间特征丢失问题。通过设计特征增强解码器融合原始图像与空间细节信息，并采用双注意力提示器实现自动提示生成，减少对人工标注的依赖。在四个腹部肿瘤数据集上验证了方法的有效性，达到当前最优性能。
Reason 该论文研究的是3D医学图像分割，属于具身智能与医疗影像分析领域，与文档图像理解（DIU）或DocVQA无关。其方法聚焦于医学图像中的分割任务，并引入自动提示机制和特征增强解码器，但这些技术应用于3D生物医学数据，无法直接迁移至文档布局分析、表格识别或文本密集场景下的视觉理解任务。且该工作属于特定领域的下游应用，不符合关注上游方法创新的要求。
Abstract The Segment Anything Model (SAM) has recently demonstrated significant potential in medical image segmentation. Although SAM is primarily trained on 2D images, attempts have been made to apply it to 3D medical image segmentation. However, the pseudo 3D processing used to adapt SAM results in spatial feature loss, limiting its performance. Additionally, most SAM-based methods still rely on manual prompts, which are challenging to implement in real-world scenarios and require extensive external expert knowledge. To address these limitations, we introduce the Decoder Enhanced and Auto Prompt SAM (DEAP-3DSAM) to tackle these limitations. Specifically, we propose a Feature Enhanced Decoder that fuses the original image features with rich and detailed spatial information to enhance spatial features. We also design a Dual Attention Prompter to automatically obtain prompt information through Spatial Attention and Channel Attention. We conduct comprehensive experiments on four public abdominal tumor segmentation datasets. The results indicate that our DEAP-3DSAM achieves state-of-the-art performance in 3D image segmentation, outperforming or matching existing manual prompt methods. Furthermore, both quantitative and qualitative ablation studies confirm the effectiveness of our proposed modules.
Authors Fangda Chen, Jintao Tang, Pancheng Wang, Ting Wang, Shasha Li, Ting Deng
Comment Accepted by BIBM 2024
Categories Computer Vision and Pattern Recognition
0
LAA3D: A Benchmark of Detecting and Tracking Low-Altitude Aircraft in 3D Space
Title CN LAA3D：低空飞行器三维空间检测与跟踪基准
Keywords 3D检测 · 多目标跟踪 · 低空飞行器 · 合成到真实 · 基准
Summary 本文提出了LAA3D，一个用于低空飞行器三维检测与跟踪的大规模数据集，包含真实与合成图像，支持3D检测、多目标跟踪和位姿估计，并建立了统一的评测基准。同时提出了一种基于单目图像的检测基线模型MonoLAA，验证了从合成到现实场景的良好泛化能力。
Reason 该论文聚焦于3D视觉中的低空飞行器检测与跟踪，属于自动驾驶与具身智能相关的3D感知领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无关。尽管其提出了新的基准和合成数据方法，但任务本身（3D目标检测、MOT、6-DoF估计）与文档结构分析、DocVQA、OCR等核心问题无直接关联，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU痛点。
Abstract Perception of Low-Altitude Aircraft (LAA) in 3D space enables precise 3D object localization and behavior understanding. However, datasets tailored for 3D LAA perception remain scarce. To address this gap, we present LAA3D, a large-scale dataset designed to advance 3D detection and tracking of low-altitude aerial vehicles. LAA3D contains 15,000 real images and 600,000 synthetic frames, captured across diverse scenarios, including urban and suburban environments. It covers multiple aerial object categories, including electric Vertical Take-Off and Landing (eVTOL) aircraft, Micro Aerial Vehicles (MAVs), and Helicopters. Each instance is annotated with 3D bounding box, class label, and instance identity, supporting tasks such as 3D object detection, 3D multi-object tracking (MOT), and 6-DoF pose estimation. Besides, we establish the LAA3D Benchmark, integrating multiple tasks and methods with unified evaluation protocols for comparison. Furthermore, we propose MonoLAA, a monocular 3D detection baseline, achieving robust 3D localization from zoom cameras with varying focal lengths. Models pretrained on synthetic images transfer effectively to real-world data with fine-tuning, demonstrating strong sim-to-real generalization. Our LAA3D provides a comprehensive foundation for future research in low-altitude 3D object perception.
Authors Hai Wu, Shuai Tang, Jiale Wang, Longkun Zou, Mingyue Guo, Rongqin Liang, Ke Chen, Yaowei Wang
Comment 25 pages
Categories Computer Vision and Pattern Recognition
0
Beyond Reward Margin: Rethinking and Resolving Likelihood Displacement in Diffusion Models via Video Generation
Title CN 超越奖励边距：通过视频生成重新思考并解决扩散模型中的似然位移问题
Keywords 扩散模型 · DPO · 似然位移 · 视频生成 · 偏好优化
Summary 本文针对扩散模型中DPO训练时出现的似然位移问题，提出了Policy-Guided DPO（PG-DPO），结合自适应拒绝缩放和隐式偏好正则化来优化生成质量，实验集中在视频生成任务上。
Reason 该论文研究的是扩散模型在视频生成任务中的偏好对齐问题，核心关注点为DPO（Direct Preference Optimization）在扩散框架下的似然位移（likelihood displacement）问题，并提出PG-DPO方法加以改进。尽管其方法论具有一定理论深度，但应用场景明确限定于视频生成，属于‘无关领域’（视频理解/生成），与文档图像理解（DIU）、DocVQA、OCR、布局分析等核心任务无直接关联。其技术迁移至文档理解领域的路径不清晰，无法解决OCR幻觉、密集文本识别、复杂排版或长文档推理等关键痛点。
Abstract Direct Preference Optimization (DPO) has shown promising results in aligning generative outputs with human preferences by distinguishing between chosen and rejected samples. However, a critical limitation of DPO is likelihood displacement, where the probabilities of chosen samples paradoxically decrease during training, undermining the quality of generation. Although this issue has been investigated in autoregressive models, its impact within diffusion-based models remains largely unexplored. This gap leads to suboptimal performance in tasks involving video generation. To address this, we conduct a formal analysis of DPO loss through updating policy within the diffusion framework, which describes how the updating of specific training samples influences the model's predictions on other samples. Using this tool, we identify two main failure modes: (1) Optimization Conflict, which arises from small reward margins between chosen and rejected samples, and (2) Suboptimal Maximization, caused by large reward margins. Informed by these insights, we introduce a novel solution named Policy-Guided DPO (PG-DPO), combining Adaptive Rejection Scaling (ARS) and Implicit Preference Regularization (IPR) to effectively mitigate likelihood displacement. Experiments show that PG-DPO outperforms existing methods in both quantitative metrics and qualitative evaluations, offering a robust solution for improving preference alignment in video generation tasks.
Authors Ruojun Xu, Yu Kai, Xuhua Ren, Jiaxiang Cheng, Bing Ma, Tianxiang Zheng, Qinhlin Lu
Categories Computer Vision and Pattern Recognition
0
ReEXplore: Improving MLLMs for Embodied Exploration with Contextualized Retrospective Experience Replay
Title CN ReEXplore：通过情境化回溯经验重放提升多模态大语言模型在具身探索中的表现
Keywords 具身探索 · 多模态大语言模型 · 经验重放 · 分层决策 · 推理时优化
Summary 本文提出ReEXplore，一种无需训练的框架，通过在推理时引入回溯经验重放和分层前沿选择机制，提升多模态大语言模型在具身探索任务中的性能。该方法在多个基准上显著优于现有MLLM基线，尤其在成功率和导航效率方面有高达3倍的提升。
Reason 该论文属于具身智能（Embodied AI）领域，研究的是基于MLLM的智能体在三维环境中的探索任务，与文档图像理解（DIU）的核心问题无直接关联。尽管其提出的经验回放和分层决策机制具有一定的推理结构设计价值，但该方法针对的是视觉-动作闭环控制场景，迁移到文档理解中的OCR幻觉、复杂排版解析或长文档推理等痛点极为困难。且论文未涉及任何文档结构、文本识别或多模态对齐优化，属于明确列出的‘无关领域’——具身智能/机器人，应直接拒绝。
Abstract Embodied exploration is a target-driven process that requires embodied agents to possess fine-grained perception and knowledge-enhanced decision making. While recent attempts leverage MLLMs for exploration due to their strong perceptual and reasoning abilities, we find that MLLM-based embodied agents remain suboptimal in exploring new environments: (i) they rely on profound but stale pre-trained knowledge, (ii) training-based approaches such as imitation learning or reinforcement learning are expensive for long-horizon tasks with sparse outcome rewards, and (iii) frontier-based exploration yields a large, visually nuanced action space that is difficult for MLLMs to make reliable decisions. We address these challenges with ReEXplore, a training-free framework that performs retrospective experience replay to inject distilled, abstract experience at inference time, and hierarchical frontier selection to decompose frontier ranking into coarse-to-fine decisions. Our approach enables robust, traceable, and efficient exploration. Across multiple embodied exploration benchmarks, ReEXplore yields great improvements over strong MLLM baselines, up to 3x higher performance in both success rate and in navigation efficiency under open-source backbones.
Authors Gengyuan Zhang, Mingcong Ding, Jingpei Wu, Ruotong Liao, Volker Tresp
Comment 8 main pages plus 13 pages Appendix
Categories Computer Vision and Pattern Recognition
0
A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generation
Title CN 一种用于逼真文本到LiDAR场景生成的自条件表示引导扩散模型
Keywords Text-to-LiDAR · Diffusion Model · 3D Scene Generation · Self-Conditioned · Controllability
Summary 本文提出了一种名为T2LDM的文本到LiDAR扩散模型，通过自条件表示引导（SCRG）在训练时提供去噪网络的软监督，从而生成具有丰富几何结构的高质量LiDAR场景。同时构建了T2nuScenes基准和可控性评估指标，并设计了方向位置先验以减少街道畸变。模型还支持多种条件生成任务。
Reason 该论文研究的是文本到LiDAR的3D场景生成，属于自动驾驶与3D视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心方法聚焦于3D点云生成、扩散模型中的自条件表示引导和可控性度量，无法迁移解决文档理解中的OCR幻觉、密集文本识别、复杂排版解析或长文档推理等关键问题。
Abstract Text-to-LiDAR generation can customize 3D data with rich structures and diverse scenes for downstream tasks. However, the scarcity of Text-LiDAR pairs often causes insufficient training priors, generating overly smooth 3D scenes. Moreover, low-quality text descriptions may degrade generation quality and controllability. In this paper, we propose a Text-to-LiDAR Diffusion Model for scene generation, named T2LDM, with a Self-Conditioned Representation Guidance (SCRG). Specifically, SCRG, by aligning to the real representations, provides the soft supervision with reconstruction details for the Denoising Network (DN) in training, while decoupled in inference. In this way, T2LDM can perceive rich geometric structures from data distribution, generating detailed objects in scenes. Meanwhile, we construct a content-composable Text-LiDAR benchmark, T2nuScenes, along with a controllability metric. Based on this, we analyze the effects of different text prompts for LiDAR generation quality and controllability, providing practical prompt paradigms and insights. Furthermore, a directional position prior is designed to mitigate street distortion, further improving scene fidelity. Additionally, by learning a conditional encoder via frozen DN, T2LDM can support multiple conditional tasks, including Sparse-to-Dense, Dense-to-Sparse, and Semantic-to-LiDAR generation. Extensive experiments in unconditional and conditional generation demonstrate that T2LDM outperforms existing methods, achieving state-of-the-art scene generation.
Authors Wentao Qu, Guofeng Mei, Yang Wu, Yongshun Gong, Xiaoshui Huang, Liang Xiao
Categories Computer Vision and Pattern Recognition
0
AuViRe: Audio-visual Speech Representation Reconstruction for Deepfake Temporal Localization WACV 2026
Title CN AuViRe：用于深度伪造时间定位的音频-视觉语音表示重建
Keywords 深伪检测 · 跨模态重建 · 音频-视觉学习 · 时间定位 · WACV2026
Summary 本文提出AuViRe方法，通过重建音频-视觉语音表示来实现深伪视频的时间定位。利用唇动预测语音表示或反之，检测跨模态重建差异以定位伪造片段，在多个数据集上取得领先性能。但该研究属于视频深伪检测范畴，与文档图像理解无关。
Reason 该论文研究的是音频-视觉模态在视频深伪检测中的应用，属于视频理解与多媒体安全领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心技术聚焦于跨模态语音表示重建，用于检测伪造时间段，不涉及OCR、布局分析、表格识别或文档推理等DIU核心任务。且该工作属于特定领域的应用（深伪检测），无法迁移为解决文档理解中OCR幻觉、复杂排版或长文档推理等问题的通用方法。
Abstract With the rapid advancement of sophisticated synthetic audio-visual content, e.g., for subtle malicious manipulations, ensuring the integrity of digital media has become paramount. This work presents a novel approach to temporal localization of deepfakes by leveraging Audio-Visual Speech Representation Reconstruction (AuViRe). Specifically, our approach reconstructs speech representations from one modality (e.g., lip movements) based on the other (e.g., audio waveform). Cross-modal reconstruction is significantly more challenging in manipulated video segments, leading to amplified discrepancies, thereby providing robust discriminative cues for precise temporal forgery localization. AuViRe outperforms the state of the art by +8.9 AP@0.95 on LAV-DF, +9.6 AP@0.5 on AV-Deepfake1M, and +5.1 AUC on an in-the-wild experiment. Code available at https://github.com/mever-team/auvire.
Authors Christos Koutlis, Symeon Papadopoulos
Comment WACV 2026
Categories Computer Vision and Pattern Recognition
0
View-Consistent Diffusion Representations for 3D-Consistent Video Generation
Title CN 用于3D一致视频生成的视图一致扩散表示
Keywords 视频生成 · 扩散模型 · 3D一致性 · 多视图一致性 · 视觉表示
Summary 本文提出ViCoDR方法，通过学习多视图一致的扩散表示来提升视频生成中的3D一致性，适用于相机控制下的图像到视频、文本到视频和多视图生成任务。实验表明该方法能显著减少因视角变化导致的结构变形。
Reason 该论文研究的是3D一致的视频生成问题，属于纯视频生成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心方法聚焦于多视角一致性在扩散模型中的应用，旨在解决相机姿态变化下的物体形变问题，这与文档图像中的OCR幻觉、复杂排版、长文本推理等痛点完全不相关。无法迁移至DIU领域。
Abstract Video generation models have made significant progress in generating realistic content, enabling applications in simulation, gaming, and film making. However, current generated videos still contain visual artifacts arising from 3D inconsistencies, e.g., objects and structures deforming under changes in camera pose, which can undermine user experience and simulation fidelity. Motivated by recent findings on representation alignment for diffusion models, we hypothesize that improving the multi-view consistency of video diffusion representations will yield more 3D-consistent video generation. Through detailed analysis on multiple recent camera-controlled video diffusion models we reveal strong correlations between 3D-consistent representations and videos. We also propose ViCoDR, a new approach for improving the 3D consistency of video models by learning multi-view consistent diffusion representations. We evaluate ViCoDR on camera controlled image-to-video, text-to-video, and multi-view generation models, demonstrating significant improvements in the 3D consistency of the generated videos. Project page: https://danier97.github.io/ViCoDR.
Authors Duolikun Danier, Ge Gao, Steven McDonagh, Changjian Li, Hakan Bilen, Oisin Mac Aodha
Categories Computer Vision and Pattern Recognition
0
ConMamba: Contrastive Vision Mamba for Plant Disease Detection
Title CN ConMamba：用于植物病害检测的对比视觉Mamba
Keywords 自监督学习 · 视觉Mamba · 植物病害检测 · 对比学习 · 状态空间模型
Summary 本文提出ConMamba，一种用于植物病害检测的自监督学习框架，结合视觉Mamba编码器与双层级对比损失，以高效捕捉长距离依赖并优化局部与全局特征对齐。在三个基准数据集上表现优于现有方法。
Reason 该论文属于农业领域的特定应用（植物病害检测），其方法虽涉及自监督学习和视觉Mamba结构，但整体框架是面向特定下游任务的模型设计，属于‘平行下游应用’。研究问题与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Plant Disease Detection (PDD) is a key aspect of precision agriculture. However, existing deep learning methods often rely on extensively annotated datasets, which are time-consuming and costly to generate. Self-supervised Learning (SSL) offers a promising alternative by exploiting the abundance of unlabeled data. However, most existing SSL approaches suffer from high computational costs due to convolutional neural networks or transformer-based architectures. Additionally, they struggle to capture long-range dependencies in visual representation and rely on static loss functions that fail to align local and global features effectively. To address these challenges, we propose ConMamba, a novel SSL framework specially designed for PDD. ConMamba integrates the Vision Mamba Encoder (VME), which employs a bidirectional State Space Model (SSM) to capture long-range dependencies efficiently. Furthermore, we introduce a dual-level contrastive loss with dynamic weight adjustment to optimize local-global feature alignment. Experimental results on three benchmark datasets demonstrate that ConMamba significantly outperforms state-of-the-art methods across multiple evaluation metrics. This provides an efficient and robust solution for PDD.
Authors Abdullah Al Mamun, Miaohua Zhang, David Ahmedt-Aristizabal, Zeeshan Hayder, Mohammad Awrangjeb
Categories Computer Vision and Pattern Recognition
0
UMCL: Unimodal-generated Multimodal Contrastive Learning for Cross-compression-rate Deepfake Detection IJCV
Title CN UMCL：用于跨压缩率深度伪造检测的单模态生成多模态对比学习
Keywords 深度伪造检测 · 多模态对比学习 · 压缩鲁棒性 · 特征对齐 · 单模态生成
Summary 本文提出了一种名为UMCL的框架，通过从单一视觉模态生成三种互补特征（rPPG信号、时序关键点动态和语义嵌入），并利用亲和性驱动的语义对齐策略进行多模态对比学习，以提升在不同压缩率下的深度伪造检测鲁棒性。实验表明该方法在多种压缩条件下表现优异。
Reason 该论文研究的是深度伪造检测问题，属于多媒体安全领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了多模态对比学习和视觉特征对齐等技术，但其目标是检测经过不同压缩处理的虚假视频内容，应用场景和技术动机均不适用于解决文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。方法无法迁移到DIU领域。
Abstract In deepfake detection, the varying degrees of compression employed by social media platforms pose significant challenges for model generalization and reliability. Although existing methods have progressed from single-modal to multimodal approaches, they face critical limitations: single-modal methods struggle with feature degradation under data compression in social media streaming, while multimodal approaches require expensive data collection and labeling and suffer from inconsistent modal quality or accessibility in real-world scenarios. To address these challenges, we propose a novel Unimodal-generated Multimodal Contrastive Learning (UMCL) framework for robust cross-compression-rate (CCR) deepfake detection. In the training stage, our approach transforms a single visual modality into three complementary features: compression-robust rPPG signals, temporal landmark dynamics, and semantic embeddings from pre-trained vision-language models. These features are explicitly aligned through an affinity-driven semantic alignment (ASA) strategy, which models inter-modal relationships through affinity matrices and optimizes their consistency through contrastive learning. Subsequently, our cross-quality similarity learning (CQSL) strategy enhances feature robustness across compression rates. Extensive experiments demonstrate that our method achieves superior performance across various compression rates and manipulation types, establishing a new benchmark for robust deepfake detection. Notably, our approach maintains high detection accuracy even when individual features degrade, while providing interpretable insights into feature relationships through explicit alignment.
Authors Ching-Yi Lai, Chih-Yu Jian, Pei-Cheng Chuang, Chia-Ming Lee, Chih-Chung Hsu, Chiou-Ting Hsu, Chia-Wen Lin
Comment 24-page manuscript accepted to IJCV
Categories Computer Vision and Pattern Recognition
0
Beyond Complete Shapes: A Benchmark for Quantitative Evaluation of 3D Shape Surface Matching Algorithms
Title CN 超越完整形状：用于定量评估3D形状表面匹配算法的基准
Keywords 3D形状匹配 · 部分形状匹配 · 基准数据集 · 程序化生成 · 几何处理
Summary 本文提出了一种程序化生成全形与部分3D形状匹配数据集的通用框架，并构建了包含2543个形状的大规模基准BeCoS，解决了现有数据集规模小、部分性不真实的问题。作者还基于该基准评测了当前最先进的方法。
Reason 该论文聚焦于3D可变形形状的表面匹配问题，属于3D视觉与几何处理领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其提出的BeCoS基准和生成框架针对的是3D形状匹配中的部分性挑战，无法迁移解决OCR幻觉、密集文本识别、复杂排版分析等DIU核心痛点。
Abstract Finding correspondences between 3D deformable shapes is an important and long-standing problem in geometry processing, computer vision, graphics, and beyond. While various shape matching datasets exist, they are mostly static or limited in size, restricting their adaptation to different problem settings, including both full and partial shape matching. In particular the existing partial shape matching datasets are small (fewer than 100 shapes) and thus unsuitable for data-hungry machine learning approaches. Moreover, the type of partiality present in existing datasets is often artificial and far from realistic. To address these limitations, we introduce a generic and flexible framework for the procedural generation of challenging full and partial shape matching datasets. Our framework allows the propagation of custom annotations across shapes, making it useful for various applications. By utilising our framework and manually creating cross-dataset correspondences between seven existing (complete geometry) shape matching datasets, we propose a new large benchmark BeCoS with a total of 2543 shapes. Based on this, we offer several challenging benchmark settings, covering both full and partial matching, for which we evaluate respective state-of-the-art methods as baselines.
Authors Viktoria Ehm, Nafie El Amrani, Yizheng Xie, Lennart Bastian, Maolin Gao, Weikang Wang, Lu Sang, Dongliang Cao, Tobias Weißberg, Zorah Lähner, Daniel Cremers, Florian Bernard
Categories Computer Vision and Pattern Recognition
0
Zero-shot segmentation of skin tumors in whole-slide images with vision-language foundation models CASEIB 2025
Title CN 基于视觉语言基础模型的全切片图像皮肤肿瘤零样本分割
Keywords 视觉语言模型 · 零样本分割 · 全切片图像 · 数字病理学 · 提示工程
Summary 本文提出ZEUS框架，利用冻结的视觉语言模型和类别特定的文本提示集成，在全切片病理图像上实现零样本、全自动的高分辨率肿瘤分割。通过将大图像分块处理并计算视觉嵌入与文本提示的余弦相似度，生成精细分割掩码。在两个内部数据集上验证了性能，并探讨了提示设计、域偏移等因素的影响。
Reason 该论文属于医学图像分析领域，研究任务为病理切片中的肿瘤分割，与文档图像理解（DIU）或DocVQA无关。尽管使用了视觉语言模型（VLM），但其应用场景、数据模态（病理图像）和目标（医学语义分割）均远离文档理解的核心问题，如OCR幻觉、复杂排版、文本识别与推理等。无法迁移其方法解决DIU中的关键挑战，属于无关领域工作。
Abstract Accurate annotation of cutaneous neoplasm biopsies represents a major challenge due to their wide morphological variability, overlapping histological patterns, and the subtle distinctions between benign and malignant lesions. Vision-language foundation models (VLMs), pre-trained on paired image-text corpora, learn joint representations that bridge visual features and diagnostic terminology, enabling zero-shot localization and classification of tissue regions without pixel-level labels. However, most existing VLM applications in histopathology remain limited to slide-level tasks or rely on coarse interactive prompts, and they struggle to produce fine-grained segmentations across gigapixel whole-slide images (WSIs). In this work, we introduce a zero-shot visual-language segmentation pipeline for whole-slide images (ZEUS), a fully automated, zero-shot segmentation framework that leverages class-specific textual prompt ensembles and frozen VLM encoders to generate high-resolution tumor masks in WSIs. By partitioning each WSI into overlapping patches, extracting visual embeddings, and computing cosine similarities against text prompts, we generate a final segmentation mask. We demonstrate competitive performance on two in-house datasets, primary spindle cell neoplasms and cutaneous metastases, highlighting the influence of prompt design, domain shifts, and institutional variability in VLMs for histopathology. ZEUS markedly reduces annotation burden while offering scalable, explainable tumor delineation for downstream diagnostic workflows.
Authors Santiago Moreno, Pablo Meseguer, Rocío del Amor, Valery Naranjo
Comment Conference manuscript accepted for oral presentation at CASEIB 2025
Categories Computer Vision and Pattern Recognition
0
Peregrine: One-Shot Fine-Tuning for FHE Inference of General Deep CNNs
Title CN Peregrine：用于通用深度CNN的FHE推理的一次性微调
Keywords FHE · CNN · 一次性微调 · 低次多项式 · 加密推理
Summary 本文提出了一种单阶段微调策略（SFT），将预训练的CNN模型转换为适合全同态加密（FHE）推理的形式，使用低次多项式逼近非线性激活函数，并设计了广义交错打包（GIP）方案以支持任意空间分辨率的特征图加密。实验表明该方法在CIFAR-10、ImageNet和MS COCO上能保持接近原始模型的精度，并首次实现了YOLO架构在FHE下的目标检测推理。
Reason 该论文研究的是在全同态加密（FHE）环境下对CNN进行高效推理的技术，核心挑战是激活函数的多项式近似和密文容量限制下的高分辨率图像处理。尽管涉及高分辨率图像和CNN架构，但其目标与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。方法聚焦于加密计算中的效率与精度保持，并未提出可用于解决OCR幻觉、复杂排版建模或长文档推理的通用机制。属于特定计算范式下的工程优化，无法迁移至VLM-based DIU框架中。
Abstract We address two fundamental challenges in adapting general deep CNNs for FHE-based inference: approximating non-linear activations such as ReLU with low-degree polynomials while minimizing accuracy degradation, and overcoming the ciphertext capacity barrier that constrains high-resolution image processing on FHE inference. Our contributions are twofold: (1) a single-stage fine-tuning (SFT) strategy that directly converts pre-trained CNNs into FHE-friendly forms using low-degree polynomials, achieving competitive accuracy with minimal training overhead; and (2) a generalized interleaved packing (GIP) scheme that is compatible with feature maps of virtually arbitrary spatial resolutions, accompanied by a suite of carefully designed homomorphic operators that preserve the GIP-form encryption throughout computation. These advances enable efficient, end-to-end FHE inference across diverse CNN architectures. Experiments on CIFAR-10, ImageNet, and MS COCO demonstrate that the FHE-friendly CNNs obtained via our SFT strategy achieve accuracy comparable to baselines using ReLU or SiLU activations. Moreover, this work presents the first demonstration of FHE-based inference for YOLO architectures in object detection leveraging low-degree polynomial activations.
Authors Huaming Ling, Ying Wang, Si Chen, Junfeng Fan
Categories Computer Vision and Pattern Recognition
0
CataractCompDetect: Intraoperative Complication Detection in Cataract Surgery
Title CN CataractCompDetect：白内障手术中术中并发症检测
Keywords 术中并发症检测 · 手术视频分析 · 视觉-语言推理 · SAM 2 · 医疗AI
Summary 本文提出CataractCompDetect框架，用于在白内障手术视频中自动检测术中并发症，如虹膜脱出、后囊破裂和玻璃体脱失。通过结合手术阶段感知定位、SAM 2跟踪、风险评分和视觉-语言推理，在新构建的数据集CataComp上实现了70.63%的平均F1分数。目标是辅助临床预警与手术培训。
Reason 该论文属于医疗视频理解领域，研究的是白内障手术视频中的并发症检测，使用了SAM 2跟踪和视觉-语言推理等技术。尽管涉及多模态方法，但其任务场景完全偏离文档图像理解（DIU）的核心问题，如OCR、版面分析、表格识别或文本密集理解。属于典型的垂直领域应用，且不解决DIU中的通用痛点（如幻觉、复杂排版、长文档推理），无法迁移为DIU的方法论工具。
Abstract Cataract surgery is one of the most commonly performed surgeries worldwide, yet intraoperative complications such as iris prolapse, posterior capsule rupture (PCR), and vitreous loss remain major causes of adverse outcomes. Automated detection of such events could enable early warning systems and objective training feedback. In this work, we propose CataractCompDetect, a complication detection framework that combines phase-aware localization, SAM 2-based tracking, complication-specific risk scoring, and vision-language reasoning for final classification. To validate CataractCompDetect, we curate CataComp, the first cataract surgery video dataset annotated for intraoperative complications, comprising 53 surgeries, including 23 with clinical complications. On CataComp, CataractCompDetect achieves an average F1 score of 70.63%, with per-complication performance of 81.8% (Iris Prolapse), 60.87% (PCR), and 69.23% (Vitreous Loss). These results highlight the value of combining structured surgical priors with vision-language reasoning for recognizing rare but high-impact intraoperative events. Our dataset and code will be publicly released upon acceptance.
Authors Bhuvan Sachdeva, Sneha Kumari, Rudransh Agarwal, Shalaka Kumaraswamy, Niharika Singri Prasad, Simon Mueller, Raphael Lechtenboehmer, Maximilian W. M. Wintergerst, Thomas Schultz, Kaushik Murali, Mohit Jain
Categories Computer Vision and Pattern Recognition
0
Unsupervised and Source-Free Ranking of Biomedical Segmentation Models
Title CN 无监督且无需源域的生物医学分割模型排序
Keywords 生物医学图像分割 · 无监督模型选择 · 迁移性估计 · 一致性正则化 · 模型排序
Summary 本文提出了一种无监督且无需源域数据的方法，用于对生物医学图像分割模型进行排序，通过模型在扰动下的一致性来估计其泛化能力，并在多个生物医学分割任务上验证了该方法与真实性能排名的高度相关性。
Reason 该论文研究的是生物医学图像分割模型的无监督排序问题，属于纯计算机视觉中的医学图像分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法聚焦于语义和实例分割模型在无标签目标数据下的迁移能力评估，无法迁移到解决OCR幻觉、复杂排版理解、细粒度文本定位或长文档推理等DIU核心痛点。
Abstract Model transfer presents a solution to the challenges of segmentation in the biomedical community, where the immense cost of data annotation is a major bottleneck in the use of deep learning. At the same time, hundreds of models get trained on biomedical data, submitted to challenges, and posted in model zoos and repositories. A major hurdle to wider adoption of pre-trained models lies in the lack of methods for best model selection. While such methods have been proposed for classification models, semantic and instance segmentation model ranking remain largely unaddressed, especially in a practically important setting where no labels are available on the target dataset. Similarly, if unsupervised domain adaptation is used, practitioners are faced with the task of selecting the best adapted model without target domain labels. Building on previous work linking model generalisation and consistency under perturbation, we propose the first unsupervised and source-free transferability estimator for semantic and instance segmentation tasks. We evaluate on multiple segmentation problems across biomedical imaging, finding a strong correlation between the rankings based on our estimator and rankings based on target dataset performance.
Authors Joshua Talks, Kevin Marchesini, Luca Lumetti, Federico Bolelli, Anna Kreshuk
Comment 24 pages, 6 figures
Categories Computer Vision and Pattern Recognition
0
Eevee: Towards Close-up High-resolution Video-based Virtual Try-on
Title CN Eevee：迈向近距离高分辨率基于视频的虚拟试穿
Keywords 虚拟试穿 · 高分辨率视频 · 细节生成 · 一致性评估 · 时尚电商
Summary 本文提出了Eevee，一个面向视频虚拟试穿的高分辨率数据集，支持全貌与近距离视角，旨在提升虚拟试穿中服装纹理与结构细节的真实感。同时引入新的评估指标VGID来衡量视频中服装一致性。实验表明，利用该数据集中的细节图像可显著提升现有模型的生成质量，并揭示了当前方法在纹理与结构保持上的不足。
Reason 该论文属于纯图像生成与视频生成领域，研究的是虚拟试衣技术，应用场景为时尚电商营销视频生成。其核心贡献是构建了一个包含近景和全景的高分辨率虚拟试穿视频数据集，并提出一个用于评估服装细节保持一致性的指标VGID。尽管涉及高分辨率视觉处理，但其任务目标、方法设计和评价体系均与文档图像理解（DIU）无直接关联，无法迁移解决OCR幻觉、复杂排版、细粒度文本定位或长文档推理等DIU核心问题。
Abstract Video virtual try-on technology provides a cost-effective solution for creating marketing videos in fashion e-commerce. However, its practical adoption is hindered by two critical limitations. First, the reliance on a single garment image as input in current virtual try-on datasets limits the accurate capture of realistic texture details. Second, most existing methods focus solely on generating full-shot virtual try-on videos, neglecting the business's demand for videos that also provide detailed close-ups. To address these challenges, we introduce a high-resolution dataset for video-based virtual try-on. This dataset offers two key features. First, it provides more detailed information on the garments, which includes high-fidelity images with detailed close-ups and textual descriptions; Second, it uniquely includes full-shot and close-up try-on videos of real human models. Furthermore, accurately assessing consistency becomes significantly more critical for the close-up videos, which demand high-fidelity preservation of garment details. To facilitate such fine-grained evaluation, we propose a new garment consistency metric VGID (Video Garment Inception Distance) that quantifies the preservation of both texture and structure. Our experiments validate these contributions. We demonstrate that by utilizing the detailed images from our dataset, existing video generation models can extract and incorporate texture features, significantly enhancing the realism and detail fidelity of virtual try-on results. Furthermore, we conduct a comprehensive benchmark of recent models. The benchmark effectively identifies the texture and structural preservation problems among current methods.
Authors Jianhao Zeng, Yancheng Bai, Ruidong Chen, Xuanpu Zhang, Lei Sun, Dongyang Jin, Ryan Xu, Nannan Zhang, Dan Song, Xiangxiang Chu
Categories Computer Vision and Pattern Recognition
0
Leveraging Adversarial Learning for Pathological Fidelity in Virtual Staining
Title CN 利用对抗学习提高虚拟染色中的病理保真度
Keywords 虚拟染色 · 对抗学习 · 图像到图像翻译 · 病理图像 · 生成对抗网络
Summary 本文提出了一种名为CSSP2P GAN的虚拟染色模型，旨在通过对抗学习提升免疫组化图像的病理保真度。作者强调了对抗损失在生成质量中的关键作用，并指出当前SSIM、PSNR等指标在评估虚拟染色效果上的局限性，主张采用盲法病理专家评估作为更可靠的评价方式。
Reason 该论文属于纯图像生成任务，研究的是医学图像中的虚拟染色（image-to-image translation），属于图像生成与修复领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了对抗学习和病理专家评估，但其方法聚焦于生物医学图像合成质量提升，并不涉及OCR、布局分析、表格识别或多模态文档推理等DIU核心问题。且未提出可迁移至文档理解领域的通用架构或去幻觉、高分辨率处理等底层技术创新。
Abstract In addition to evaluating tumor morphology using H&E staining, immunohistochemistry is used to assess the presence of specific proteins within the tissue. However, this is a costly and labor-intensive technique, for which virtual staining, as an image-to-image translation task, offers a promising alternative. Although recent, this is an emerging field of research with 64% of published studies just in 2024. Most studies use publicly available datasets of H&E-IHC pairs from consecutive tissue sections. Recognizing the training challenges, many authors develop complex virtual staining models based on conditional Generative Adversarial Networks, but ignore the impact of adversarial loss on the quality of virtual staining. Furthermore, overlooking the issues of model evaluation, they claim improved performance based on metrics such as SSIM and PSNR, which are not sufficiently robust to evaluate the quality of virtually stained images. In this paper, we developed CSSP2P GAN, which we demonstrate to achieve heightened pathological fidelity through a blind pathological expert evaluation. Furthermore, while iteratively developing our model, we study the impact of the adversarial loss and demonstrate its crucial role in the quality of virtually stained images. Finally, while comparing our model with reference works in the field, we underscore the limitations of the currently used evaluation metrics and demonstrate the superior performance of CSSP2P GAN.
Authors José Teixeira, Pascal Klöckner, Diana Montezuma, Melis Erdal Cesur, João Fraga, Hugo M. Horlings, Jaime S. Cardoso, Sara P. Oliveira
Categories Computer Vision and Pattern Recognition
0
VeCoR - Velocity Contrastive Regularization for Flow Matching
Title CN VeCoR - 面向流匹配的 Velocity Contrastive Regularization
Keywords 流匹配 · 生成模型 · 对比学习 · 图像生成 · 速度场正则化
Summary 本文提出了Velocity Contrastive Regularization（VeCoR），一种用于流匹配生成模型的对比正则化方法，通过引入正负样本监督来增强速度场的学习稳定性，从而提升生成图像的质量和训练效率。实验表明其在ImageNet和COCO等标准图像生成任务上显著优于基线方法。
Reason 该论文研究的是生成模型中的流匹配（Flow Matching）方法，核心目标是提升图像生成质量与训练稳定性，属于纯图像生成领域。其提出的方法VeCoR聚焦于速度场的对比正则化，用于改善生成轨迹的感知保真度，但应用场景完全不涉及文档结构、文本理解或多模态文档推理。与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版建模、细粒度定位等无直接关联，且无法迁移为解决DIU问题的有效工具。
Abstract Flow Matching (FM) has recently emerged as a principled and efficient alternative to diffusion models. Standard FM encourages the learned velocity field to follow a target direction; however, it may accumulate errors along the trajectory and drive samples off the data manifold, leading to perceptual degradation, especially in lightweight or low-step configurations. To enhance stability and generalization, we extend FM into a balanced attract-repel scheme that provides explicit guidance on both "where to go" and "where not to go." To be formal, we propose \textbf{Velocity Contrastive Regularization (VeCoR)}, a complementary training scheme for flow-based generative modeling that augments the standard FM objective with contrastive, two-sided supervision. VeCoR not only aligns the predicted velocity with a stable reference direction (positive supervision) but also pushes it away from inconsistent, off-manifold directions (negative supervision). This contrastive formulation transforms FM from a purely attractive, one-sided objective into a two-sided training signal, regularizing trajectory evolution and improving perceptual fidelity across datasets and backbones. On ImageNet-1K 256$\times$256, VeCoR yields 22\% and 35\% relative FID reductions on SiT-XL/2 and REPA-SiT-XL/2 backbones, respectively, and achieves further FID gains (32\% relative) on MS-COCO text-to-image generation, demonstrating consistent improvements in stability, convergence, and image quality, particularly in low-step and lightweight settings. Project page: https://p458732.github.io/VeCoR_Project_Page/
Authors Zong-Wei Hong, Jing-lun Li, Lin-Ze Li, Shen Zhang, Yao Tang
Categories Computer Vision and Pattern Recognition
0
Motion-R1: Enhancing Motion Generation with Decomposed Chain-of-Thought and RL Binding
Title CN Motion-R1：通过分解链式思维与强化学习绑定增强运动生成
Keywords 文本到动作生成 · 链式思维 · 强化学习 · 多模态对齐 · 动作合成
Summary 本文提出Motion-R1，结合分解的链式思维（Decomposed CoT）与强化学习（RL Binding）来提升文本到动作生成的质量和可解释性。通过构建自动化的CoT数据引擎合成高质量推理数据，并设计多模态对齐的RL奖励机制，使生成的人体动作在语义准确性和运动真实性上均有提升。实验表明其在多个基准上达到SOTA。
Reason 该论文研究的是文本到人体动作生成（Text-to-Motion），属于视频/动作生成领域，与文档图像理解（DIU）或DocVQA无直接关联。尽管使用了CoT和RL等方法，但其任务目标、数据模态（动作序列而非文档结构）和应用场景均远离文档理解核心问题，无法迁移用于解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU痛点。
Abstract Text-to-Motion generation has become a fundamental task in human-machine interaction, enabling the synthesis of realistic human motions from natural language descriptions. Although recent advances in large language models and reinforcement learning have contributed to high-quality motion generation, two major challenges remain. Existing approaches often fail to capture the temporal and causal complexities inherent in natural language, leading to oversimplified or incoherent motions. Additionally, RL-based methods are frequently overly complex, hindering their scalability and adaptability across various motion generation tasks. To address these challenges, we propose Motion-R1, a novel framework that combines decomposed Chain-of-Thought reasoning with reinforcement learning to enhance both the quality and interpretability of generated motions. Specifically, we introduce the Decomposed CoT Data Engine, which leverages an automated pipeline to synthesize high-quality reasoning data, allowing the model to better capture the temporal dependencies and causal relationships of human motion. We also propose RL Binding, a reinforcement learning strategy that incorporates multi-modal text-motion alignment into the RL reward function, guiding the model to produce motions that are both semantically accurate and motionally realistic. Extensive experiments across benchmark datasets demonstrate that Motion-R1 achieves state-of-the-art performance, with a 3.5% improvement in MM-Dist on HumanML3D and improvements in R-Precision and FID on KIT-ML and BABEL, surpassing existing methods across key metrics and highlighting its superior capability in handling complex motion generation tasks. Project page: https://motion-r1.github.io/.
Authors Runqi Ouyang, Haoyun Li, Zhenyuan Zhang, Xiaofeng Wang, Zeyu Zhang, Zheng Zhu, Guan Huang, Sirui Han, Xingang Wang
Categories Computer Vision and Pattern Recognition
0
FineXtrol: Controllable Motion Generation via Fine-Grained Text AAAI 2026
Title CN FineXtrol：通过细粒度文本实现可控运动生成
Keywords 可控运动生成 · 细粒度文本控制 · 3D人体动作 · 对比学习 · 文本到动作
Summary 本文提出FineXtrol，一种通过时序感知、细粒度文本指令控制人体局部动作生成的框架。通过分层对比学习提升文本编码器对精细动作描述的区分能力，在保持高效的同时增强了动作控制精度。实验表明该方法在可控运动生成任务中表现优异。
Reason 该论文研究的是基于文本控制的人体动作生成，属于3D人体运动合成领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管使用了文本控制和视觉-语言对齐技术，但其核心任务聚焦于时间序列的肢体动作生成，迁移到文档理解中的OCR幻觉、复杂排版解析或长文档推理等问题极为困难，且未提供任何可用于文档结构分析或视觉token处理的方法论创新。
Abstract Recent works have sought to enhance the controllability and precision of text-driven motion generation. Some approaches leverage large language models (LLMs) to produce more detailed texts, while others incorporate global 3D coordinate sequences as additional control signals. However, the former often introduces misaligned details and lacks explicit temporal cues, and the latter incurs significant computational cost when converting coordinates to standard motion representations. To address these issues, we propose FineXtrol, a novel control framework for efficient motion generation guided by temporally-aware, precise, user-friendly, and fine-grained textual control signals that describe specific body part movements over time. In support of this framework, we design a hierarchical contrastive learning module that encourages the text encoder to produce more discriminative embeddings for our novel control signals, thereby improving motion controllability. Quantitative results show that FineXtrol achieves strong performance in controllable motion generation, while qualitative analysis demonstrates its flexibility in directing specific body part movements.
Authors Keming Shen, Bizhu Wu, Junliang Chen, Xiaoqin Wang, Linlin Shen
Comment 20 pages, 14 figures, AAAI 2026
Categories Computer Vision and Pattern Recognition
0
Directed-CP: Directed Collaborative Perception for Connected and Autonomous Vehicles via Proactive Attention ICRA'25
Title CN Directed-CP：通过主动注意力实现面向连接与自动驾驶车辆的定向协同感知
Keywords 协同感知 · 自动驾驶 · 方向感知 · 注意力机制 · V2X
Summary 本文提出Directed-CP，一种面向连接与自动驾驶车辆（CAV）的定向协同感知系统，通过主动信号指示感兴趣方向，并结合RSU辅助的方向掩码机制、方向感知的选择性注意力模块以及方向加权检测损失（DWLoss），在V2X-Sim 2.0数据集上提升了特定方向的局部感知精度。然而，该工作属于自动驾驶感知范畴，与文档理解无直接关联。
Reason 该论文研究的是自动驾驶场景下的协同感知（Collaborative Perception），属于自动驾驶与车联网领域，尽管涉及视觉信息处理，但其核心任务、数据模态和应用场景均与文档图像理解（DIU）无关。无法迁移用于解决OCR幻觉、复杂排版或长文档推理等DIU核心问题。
Abstract Collaborative perception (CP) leverages visual data from connected and autonomous vehicles (CAV) to enhance an ego vehicle's field of view (FoV). Despite recent progress, current CP methods expand the ego vehicle's 360-degree perceptual range almost equally, which faces two key challenges. Firstly, in areas with uneven traffic distribution, focusing on directions with little traffic offers limited benefits. Secondly, under limited communication budgets, allocating excessive bandwidth to less critical directions lowers the perception accuracy in more vital areas. To address these issues, we propose Direct-CP, a proactive and direction-aware CP system aiming at improving CP in specific directions. Our key idea is to enable an ego vehicle to proactively signal its interested directions and readjust its attention to enhance local directional CP performance. To achieve this, we first propose an RSU-aided direction masking mechanism that assists an ego vehicle in identifying vital directions. Additionally, we design a direction-aware selective attention module to wisely aggregate pertinent features based on ego vehicle's directional priorities, communication budget, and the positional data of CAVs. Moreover, we introduce a direction-weighted detection loss (DWLoss) to capture the divergence between directional CP outcomes and the ground truth, facilitating effective model training. Extensive experiments on the V2X-Sim 2.0 dataset demonstrate that our approach achieves 19.8\% higher local perception accuracy in interested directions and 2.5\% higher overall perception accuracy than the state-of-the-art methods in collaborative 3D object detection tasks. Codes are available at https://github.com/yihangtao/Directed-CP.git.
Authors Yihang Tao, Senkang Hu, Zhengru Fang, Yuguang Fang
Comment Accepted by ICRA'25
Categories Computer Vision and Pattern Recognition
0
One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Control
Title CN One4D：通过解耦LoRA控制实现统一的4D生成与重建
Keywords 4D生成 · 点图重建 · 视频扩散模型 · LoRA · 多模态生成
Summary 本文提出One4D，一个用于4D生成与重建的统一框架，能够同步生成RGB帧和点图。通过统一掩码条件机制处理不同稀疏程度的输入帧，并设计了解耦LoRA控制（DLC）策略，使用两个模态特定的LoRA分支分别处理RGB和点图，通过轻量级连接学习像素级一致性。在合成与真实4D数据上训练后，模型在生成与重建任务中均表现出色。
Reason 该论文研究的是4D动态内容（RGB帧与点图）的生成与重建，属于视频生成与3D视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心技术如解耦LoRA控制和统一掩码条件机制，主要面向多模态动态几何建模，难以迁移到文档理解中的OCR幻觉、密集文本识别、复杂版式分析等核心痛点。属于无关领域（3D/视频生成），符合负面清单第2条。
Abstract We present One4D, a unified framework for 4D generation and reconstruction that produces dynamic 4D content as synchronized RGB frames and pointmaps. By consistently handling varying sparsities of conditioning frames through a Unified Masked Conditioning (UMC) mechanism, One4D can seamlessly transition between 4D generation from a single image, 4D reconstruction from a full video, and mixed generation and reconstruction from sparse frames. Our framework adapts a powerful video generation model for joint RGB and pointmap generation, with carefully designed network architectures. The commonly used diffusion finetuning strategies for depthmap or pointmap reconstruction often fail on joint RGB and pointmap generation, quickly degrading the base video model. To address this challenge, we introduce Decoupled LoRA Control (DLC), which employs two modality-specific LoRA adapters to form decoupled computation branches for RGB frames and pointmaps, connected by lightweight, zero-initialized control links that gradually learn mutual pixel-level consistency. Trained on a mixture of synthetic and real 4D datasets under modest computational budgets, One4D produces high-quality RGB frames and accurate pointmaps across both generation and reconstruction tasks. This work represents a step toward general, high-quality geometry-based 4D world modeling using video diffusion models. Project page: https://mizhenxing.github.io/One4D
Authors Zhenxing Mi, Yuxin Wang, Dan Xu
Comment Project page: https://mizhenxing.github.io/One4D
Categories Computer Vision and Pattern Recognition
0
Prototypical Contrastive Learning-based CLIP Fine-tuning for Object Re-identification
Title CN 基于原型对比学习的CLIP微调用于目标重识别
Keywords 目标重识别 · CLIP · 对比学习 · 原型学习 · 微调
Summary 本文提出一种基于原型对比学习（PCL）的CLIP图像编码器微调方法，用于目标重识别任务，在无需提示学习的情况下实现了优于现有方法的性能。工作主要面向行人和车辆重识别场景，强调在缺乏语义标签时的有效表征学习。
Reason 该论文研究的是目标重识别（Re-ID），属于行人和车辆再识别领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。尽管使用了CLIP这一多模态模型，但其方法聚焦于无监督/有监督下的Re-ID性能提升，并未涉及OCR、布局分析、表格识别、视觉语言推理或文档特有的结构建模。无法迁移解决文档理解中的核心痛点如OCR幻觉、密集文本处理或复杂排版理解。
Abstract This work aims to adapt large-scale pre-trained vision-language models, such as contrastive language-image pretraining (CLIP), to enhance the performance of object reidentification (Re-ID) across various supervision settings. Although prompt learning has enabled a recent work named CLIP-ReID to achieve promising performance, the underlying mechanisms and the necessity of prompt learning remain unclear due to the absence of semantic labels in ReID tasks. In this work, we first analyze the role prompt learning in CLIP-ReID and identify its limitations. Based on our investigations, we propose a simple yet effective approach to adapt CLIP for supervised object Re-ID. Our approach directly fine-tunes the image encoder of CLIP using a prototypical contrastive learning (PCL) loss, eliminating the need for prompt learning. Experimental results on both person and vehicle Re-ID datasets demonstrate the competitiveness of our method compared to CLIP-ReID. Furthermore, we extend our PCL-based CLIP fine-tuning approach to unsupervised scenarios, where we achieve state-of-the art performance.
Authors Jiachen Li, Xiaojin Gong
Categories Computer Vision and Pattern Recognition
0
BackdoorVLM: A Benchmark for Backdoor Attacks on Vision-Language Models
Title CN BackdoorVLM：面向视觉语言模型的后门攻击基准
Keywords 后门攻击 · 视觉语言模型 · 安全性 · 多模态安全 · 基准测试
Summary 本文提出了BackdoorVLM，首个系统评估视觉语言模型（VLM）上后门攻击的基准，涵盖图像描述和视觉问答等任务，定义了五类多模态后门威胁，并在多种攻击方式和模型上进行实验，揭示了文本触发器在后门攻击中的主导作用。研究表明低毒化率下仍可实现高攻击成功率，暴露了当前VLM的安全漏洞。
Reason 该论文聚焦于视觉语言模型的安全性问题，研究后门攻击在多模态场景下的威胁模式与评估方法。虽然涉及VLM，但其核心是安全领域的‘后门攻击’问题，而非文档图像理解（DIU）中的OCR幻觉、复杂排版、细粒度识别或长文档推理等核心痛点。没有提出可用于提升DIU性能的底层方法，且属于平行应用类的安全评估框架，无法直接迁移用于解决文档理解中的技术瓶颈。
Abstract Backdoor attacks undermine the reliability and trustworthiness of machine learning systems by injecting hidden behaviors that can be maliciously activated at inference time. While such threats have been extensively studied in unimodal settings, their impact on multimodal foundation models, particularly vision-language models (VLMs), remains largely underexplored. In this work, we introduce \textbf{BackdoorVLM}, the first comprehensive benchmark for systematically evaluating backdoor attacks on VLMs across a broad range of settings. It adopts a unified perspective that injects and analyzes backdoors across core vision-language tasks, including image captioning and visual question answering. BackdoorVLM organizes multimodal backdoor threats into 5 representative categories: targeted refusal, malicious injection, jailbreak, concept substitution, and perceptual hijack. Each category captures a distinct pathway through which an adversary can manipulate a model's behavior. We evaluate these threats using 12 representative attack methods spanning text, image, and bimodal triggers, tested on 2 open-source VLMs and 3 multimodal datasets. Our analysis reveals that VLMs exhibit strong sensitivity to textual instructions, and in bimodal backdoors the text trigger typically overwhelms the image trigger when forming the backdoor mapping. Notably, backdoors involving the textual modality remain highly potent, with poisoning rates as low as 1\% yielding over 90\% success across most tasks. These findings highlight significant, previously underexplored vulnerabilities in current VLMs. We hope that BackdoorVLM can serve as a useful benchmark for analyzing and mitigating multimodal backdoor threats. Code is available at: https://github.com/bin015/BackdoorVLM .
Authors Juncheng Li, Yige Li, Hanxun Huang, Yunhao Chen, Xin Wang, Yixu Wang, Xingjun Ma, Yu-Gang Jiang
Categories Computer Vision and Pattern Recognition
0
K-FACE: A Large-Scale KIST Face Database in Consideration with Unconstrained Environments
Title CN K-FACE：一种考虑非受限环境的大型KIST人脸数据库
Keywords 人脸数据库 · 无约束环境 · 姿态变化 · 光照变化 · 遮挡
Summary 本文介绍了K-FACE，一个由韩国科学技术院（KIST）发布的大型人脸数据库，包含超过100万张高质量图像，涵盖1000名受试者在27种姿态、35种光照条件、3种表情及多种遮挡情况下的数据。数据库通过半球形采集系统和精细的灯光控制构建，强调环境因素和个人特征（如性别、年龄）的平衡分布，适用于人脸识别、正面化、光照归一化和三维人脸建模等计算机视觉任务。
Reason 该论文聚焦于人脸识别领域，构建了一个大规模人脸图像数据库，主要用于人脸识别、姿态估计、光照归一化等任务。与文档图像理解（DIU）或文档视觉问答（DocVQA）无关，属于无关领域（人脸数据集），不提供可用于解决OCR幻觉、复杂排版或长文档推理的方法论创新。
Abstract In this paper, we introduce a new large-scale face database from KIST, denoted as K-FACE, and describe a novel capturing device specifically designed to obtain the data. The K-FACE database contains more than 1 million high-quality images of 1,000 subjects selected by considering the ratio of gender and age groups. It includes a variety of attributes, including 27 poses, 35 lighting conditions, three expressions, and occlusions by the combination of five types of accessories. As the K-FACE database is systematically constructed through a hemispherical capturing system with elaborate lighting control and multiple cameras, it is possible to accurately analyze the effects of factors that cause performance degradation, such as poses, lighting changes, and accessories. We consider not only the balance of external environmental factors, such as pose and lighting, but also the balance of personal characteristics such as gender and age group. The gender ratio is the same, while the age groups of subjects are uniformly distributed from the 20s to 50s for both genders. The K-FACE database can be extensively utilized in various vision tasks, such as face recognition, face frontalization, illumination normalization, face age estimation, and three-dimensional face model generation. We expect systematic diversity and uniformity of the K-FACE database to promote these research fields.
Authors Yeji Choi, Hyunjung Park, Gi Pyo Nam, Haksub Kim, Heeseung Choi, Junghyun Cho, Ig-Jae Kim
Comment 8 pages, 8 figures
Categories Computer Vision and Pattern Recognition
0
Roadside Monocular 3D Detection Prompted by 2D Detection WACV 2026
Title CN 基于2D检测提示的路边单目3D检测
Keywords 3D检测 · 2D提示 · 路边感知 · 单目深度估计 · BEV转换
Summary 本文提出Pro3D，一种利用2D检测结果作为提示来提升路边单目摄像头3D检测性能的方法。通过将2D检测框的属性编码并融合到3D检测流程中，使模型能更精准地将2D目标提升至3D空间。实验表明该方法在多个基准上达到SOTA。
Reason 该论文属于自动驾驶与3D视觉领域，研究的是从路边摄像头进行单目3D目标检测，尽管使用了‘prompt’机制和2D检测作为引导，但其核心任务与文档图像理解（DIU）无关。所解决的问题（3D定位、BEV转换）和应用场景（车路协同）完全偏离文档结构分析、文本识别或DocVQA等方向，无法迁移用于解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点。
Abstract Roadside monocular 3D detection requires detecting objects of predefined classes in an RGB frame and predicting their 3D attributes, such as bird's-eye-view (BEV) locations. It has broad applications in traffic control, vehicle-vehicle communication, and vehicle-infrastructure cooperative perception. To address this task, we introduce Promptable 3D Detector (Pro3D), a novel detector design that leverages 2D detections as prompts. We build our Pro3D upon two key insights. First, compared to a typical 3D detector, a 2D detector is ``easier'' to train due to fewer loss terms and performs significantly better at localizing objects w.r.t 2D metrics. Second, once 2D detections precisely locate objects in the image, a 3D detector can focus on lifting these detections into 3D BEV, especially when fixed camera pose or scene geometry provide an informative prior. To encode and incorporate 2D detections, we explore three methods: (a) concatenating features from both 2D and 3D detectors, (b) attentively fusing 2D and 3D detector features, and (c) encoding properties of predicted 2D bounding boxes \{$x$, $y$, width, height, label\} and attentively fusing them with the 3D detector feature. Interestingly, the third method significantly outperforms the others, underscoring the effectiveness of 2D detections as prompts that offer precise object targets and allow the 3D detector to focus on lifting them into 3D. Pro3D is adaptable for use with a wide range of 2D and 3D detectors with minimal modifications. Comprehensive experiments demonstrate that our Pro3D significantly enhances existing methods, achieving state-of-the-art results on two contemporary benchmarks.
Authors Yechi Ma, Yanan Li, Wei Hua, Shu Kong
Comment Accepted by WACV 2026
Categories Computer Vision and Pattern Recognition
0
MatMart: Material Reconstruction of 3D Objects via Diffusion
Title CN MatMart：通过扩散模型实现3D物体的材质重建
Keywords 扩散模型 · 3D材质重建 · 视图-材质交叉注意力 · 物理材质估计 · 两阶段重建
Summary 本文提出MatMart，一种基于扩散模型的两阶段3D物体材质重建框架。该方法首先从输入图像中预测材质，再通过视图-材质交叉注意力（VMCA）和渐进式推理生成未观测视角的材质，支持任意数量输入并实现端到端优化。实验表明其在材质重建质量上优于现有方法。
Reason 该论文研究的是3D视觉中的物理材质估计与生成，属于纯3D视觉和图形学领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心技术如扩散模型、VMCA注意力机制均针对多视角3D材质重建，无法迁移解决文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Applying diffusion models to physically-based material estimation and generation has recently gained prominence. In this paper, we propose \ttt, a novel material reconstruction framework for 3D objects, offering the following advantages. First, \ttt\ adopts a two-stage reconstruction, starting with accurate material prediction from inputs and followed by prior-guided material generation for unobserved views, yielding high-fidelity results. Second, by utilizing progressive inference alongside the proposed view-material cross-attention (VMCA), \ttt\ enables reconstruction from an arbitrary number of input images, demonstrating strong scalability and flexibility. Finally, \ttt\ achieves both material prediction and generation capabilities through end-to-end optimization of a single diffusion model, without relying on additional pre-trained models, thereby exhibiting enhanced stability across various types of objects. Extensive experiments demonstrate that \ttt\ achieves superior performance in material reconstruction compared to existing methods.
Authors Xiuchao Wu, Pengfei Zhu, Jiangjing Lyu, Xinguo Liu, Jie Guo, Yanwen Guo, Weiwei Xu, Chengfei Lyu
Categories Computer Vision and Pattern Recognition
0
Monocular Person Localization under Camera Ego-motion IROS2025
Title CN 相机自运动下的单目人体定位
Keywords 单目定位 · 相机自运动 · 人体定位 · 姿态估计 · 四足机器人
Summary 本文提出一种在单目相机存在显著自运动的情况下对人体进行3D定位的方法，通过将人体建模为四点结构，并联合优化相机姿态和人体位置来提升定位精度。实验表明该方法在公开数据集和真实四足机器人系统上均优于基线方法，并已部署于人跟随系统中。
Reason 该论文研究的是在相机移动情况下对人体进行3D定位，属于具身智能与机器人领域的人体跟随任务，核心问题是3D姿态估计与相机位姿优化，与文档图像理解（DIU）或视觉语言模型（VLM）无关。未涉及OCR、文档布局分析、表格识别、文本理解等任何DIU核心任务，也无法迁移用于解决OCR幻觉、密集文本处理或长文档推理等痛点。
Abstract Localizing a person from a moving monocular camera is critical for Human-Robot Interaction (HRI). To estimate the 3D human position from a 2D image, existing methods either depend on the geometric assumption of a fixed camera or use a position regression model trained on datasets containing little camera ego-motion. These methods are vulnerable to severe camera ego-motion, resulting in inaccurate person localization. We consider person localization as a part of a pose estimation problem. By representing a human with a four-point model, our method jointly estimates the 2D camera attitude and the person's 3D location through optimization. Evaluations on both public datasets and real robot experiments demonstrate our method outperforms baselines in person localization accuracy. Our method is further implemented into a person-following system and deployed on an agile quadruped robot.
Authors Yu Zhan, Hanjing Ye, Hong Zhang
Comment Accepted by IROS2025. Project page: https://medlartea.github.io/rpf-quadruped/
Categories Computer Vision and Pattern Recognition
0
MFmamba: A Multi-function Network for Panchromatic Image Resolution Restoration Based on State-Space Model AAAI-2026
Title CN MFmamba：一种基于状态空间模型的全色图像分辨率恢复多功能网络
Keywords 遥感图像 · 超分辨率 · 光谱恢复 · Mamba · UNet++
Summary 本文提出了一种名为MFmamba的多功能网络，用于在仅有全色（PAN）图像输入的情况下实现超分辨率、光谱恢复以及两者的联合任务。网络以UNet++为骨干，引入Mamba上采样块（MUB）、双池化注意力（DPA）和多尺度混合交叉块（MHCB）进行特征提取与融合。实验表明该方法在定量指标和视觉质量上表现优异。
Reason 该论文研究的是遥感图像中的全色图像超分辨率与光谱恢复问题，属于遥感图像处理领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其使用了Mamba架构和高分辨率图像处理技术，但方法设计面向遥感成像系统，无法直接迁移解决文档图像中的OCR幻觉、复杂排版或长文档推理等关键痛点，属于无关领域应用。
Abstract Remote sensing images are becoming increasingly widespread in military, earth resource exploration. Because of the limitation of a single sensor, we can obtain high spatial resolution grayscale panchromatic (PAN) images and low spatial resolution color multispectral (MS) images. Therefore, an important issue is to obtain a color image with high spatial resolution when there is only a PAN image at the input. The existing methods improve spatial resolution using super-resolution (SR) technology and spectral recovery using colorization technology. However, the SR technique cannot improve the spectral resolution, and the colorization technique cannot improve the spatial resolution. Moreover, the pansharpening method needs two registered inputs and can not achieve SR. As a result, an integrated approach is expected. To solve the above problems, we designed a novel multi-function model (MFmamba) to realize the tasks of SR, spectral recovery, joint SR and spectral recovery through three different inputs. Firstly, MFmamba utilizes UNet++ as the backbone, and a Mamba Upsample Block (MUB) is combined with UNet++. Secondly, a Dual Pool Attention (DPA) is designed to replace the skip connection in UNet++. Finally, a Multi-scale Hybrid Cross Block (MHCB) is proposed for initial feature extraction. Many experiments show that MFmamba is competitive in evaluation metrics and visual results and performs well in the three tasks when only the input PAN image is used.
Authors Qian Jiang, Qianqian Wang, Xin Jin, Michal Wozniak, Shaowen Yao, Wei Zhou
Comment 9 pages, 9 figures. This paper has been accepted for publication in AAAI-2026
Categories Computer Vision and Pattern Recognition
0
MagicWorld: Interactive Geometry-driven Video World Exploration
Title CN MagicWorld：基于交互式几何驱动的视频世界探索
Keywords 视频生成 · 3D几何建模 · 交互式建模 · 历史缓存 · 场景演化
Summary 本文提出MagicWorld，一种结合3D几何先验与历史检索机制的交互式视频世界模型。通过Action-Guided 3D Geometry模块（AG3D）构建点云以增强视角变换下的结构稳定性，并引入History Cache Retrieval（HCR）机制缓解多步交互中的误差累积。实验表明该方法在场景连续性和稳定性方面表现更优。
Reason 该论文研究的是交互式视频生成与3D场景演化，属于视频理解与生成领域，核心关注点为动态场景的结构一致性与历史记忆问题。尽管其提出的AG3D模块和HCR机制在视频建模中有价值，但整体任务与文档图像理解（DIU）无直接关联，无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等核心痛点。
Abstract Recent interactive video world model methods generate scene evolution conditioned on user instructions. Although they achieve impressive results, two key limitations remain. First, they fail to fully exploit the correspondence between instruction-driven scene motion and the underlying 3D geometry, which results in structural instability under viewpoint changes. Second, they easily forget historical information during multi-step interaction, resulting in error accumulation and progressive drift in scene semantics and structure. To address these issues, we propose MagicWorld, an interactive video world model that integrates 3D geometric priors and historical retrieval. MagicWorld starts from a single scene image, employs user actions to drive dynamic scene evolution, and autoregressively synthesizes continuous scenes. We introduce the Action-Guided 3D Geometry Module (AG3D), which constructs a point cloud from the first frame of each interaction and the corresponding action, providing explicit geometric constraints for viewpoint transitions and thereby improving structural consistency. We further propose History Cache Retrieval (HCR) mechanism, which retrieves relevant historical frames during generation and injects them as conditioning signals, helping the model utilize past scene information and mitigate error accumulation. Experimental results demonstrate that MagicWorld achieves notable improvements in scene stability and continuity across interaction iterations.
Authors Guangyuan Li, Siming Zheng, Shuolin Xu, Jinwei Chen, Bo Li, Xiaobin Hu, Lei Zhao, Peng-Tao Jiang
Categories Computer Vision and Pattern Recognition
0
Facade Segmentation for Solar Photovoltaic Suitability NeurIPS 2025 Tackling Climate Change with Machine Learning Workshop
Title CN 用于太阳能光伏适用性评估的立面分割
Keywords 立面分割 · 建筑光伏 · SegFormer · 城市脱碳 · 太阳能潜力
Summary 本文提出了一种基于SegFormer-B5的管道，用于从建筑立面图像中自动识别适合安装光伏面板的区域，并估计太阳能潜力。该方法在CMP Facades数据集上进行训练，应用于373个来自十座城市的立面样本，结果显示实际可安装的BIPV潜力远低于理论值。尽管技术路线扎实，但属于特定领域的视觉分割应用，与文档理解无关。
Reason 该论文属于计算机视觉在建筑光伏规划中的垂直领域应用，核心是利用SegFormer对建筑立面进行语义分割，并未涉及文档图像理解（DIU）或视觉语言模型（VLM）在文档理解中的方法创新。其任务目标、数据类型和方法均与文档图像、文本布局、表格识别、DocVQA等DIU核心问题无关，属于典型的‘平行下游应用’，不符合本筛选标准。
Abstract Building integrated photovoltaic (BIPV) facades represent a promising pathway towards urban decarbonization, especially where roof areas are insufficient and ground-mounted arrays are infeasible. Although machine learning-based approaches to support photovoltaic (PV) planning on rooftops are well researched, automated approaches for facades still remain scarce and oversimplified. This paper therefore presents a pipeline that integrates detailed information on the architectural composition of the facade to automatically identify suitable surfaces for PV application and estimate the solar energy potential. The pipeline fine-tunes SegFormer-B5 on the CMP Facades dataset and converts semantic predictions into facade-level PV suitability masks and PV panel layouts considering module sizes and clearances. Applied to a dataset of 373 facades with known dimensions from ten cities, the results show that installable BIPV potential is significantly lower than theoretical potential, thus providing valuable insights for reliable urban energy planning. With the growing availability of facade imagery, the proposed pipeline can be scaled to support BIPV planning in cities worldwide.
Authors Ayca Duran, Christoph Waibel, Bernd Bickel, Iro Armeni, Arno Schlueter
Comment NeurIPS 2025 Tackling Climate Change with Machine Learning Workshop version. Non-archival
Categories Computer Vision and Pattern Recognition
0
Neural Texture Splatting: Expressive 3D Gaussian Splatting for View Synthesis, Geometry, and Dynamic Reconstruction SIGGRAPH Asia 2025
Title CN 神经纹理点阵化：用于视图合成、几何与动态重建的高表达性3D高斯点阵化
Keywords 3D Gaussian Splatting · Neural Texture Splatting · View Synthesis · Dynamic Reconstruction · Neural Fields
Summary 本文提出了一种名为Neural Texture Splatting（NTS）的新方法，通过引入一个全局神经场（结合三平面与神经解码器）来预测每个图元的局部外观和几何场，从而增强3D高斯点阵化的表达能力。该方法支持视图和时间依赖的动态效果，在稀疏和密集输入下均能提升新视角合成、几何重建和动态重建性能，并实现更小模型规模与高效全局信息交换。
Reason 该论文属于3D视觉与图形学领域，研究的是3D高斯点阵化在新视角合成和动态场景重建中的应用。尽管其方法涉及视觉表示学习，但核心任务与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract 3D Gaussian Splatting (3DGS) has emerged as a leading approach for high-quality novel view synthesis, with numerous variants extending its applicability to a broad spectrum of 3D and 4D scene reconstruction tasks. Despite its success, the representational capacity of 3DGS remains limited by the use of 3D Gaussian kernels to model local variations. Recent works have proposed to augment 3DGS with additional per-primitive capacity, such as per-splat textures, to enhance its expressiveness. However, these per-splat texture approaches primarily target dense novel view synthesis with a reduced number of Gaussian primitives, and their effectiveness tends to diminish when applied to more general reconstruction scenarios. In this paper, we aim to achieve concrete performance improvement over state-of-the-art 3DGS variants across a wide range of reconstruction tasks, including novel view synthesis, geometry and dynamic reconstruction, under both sparse and dense input settings. To this end, we introduce Neural Texture Splatting (NTS). At the core of our approach is a global neural field (represented as a hybrid of a tri-plane and a neural decoder) that predicts local appearance and geometric fields for each primitive. By leveraging this shared global representation that models local texture fields across primitives, we significantly reduce model size and facilitate efficient global information exchange, demonstrating strong generalization across tasks. Furthermore, our neural modeling of local texture fields introduces expressive view- and time-dependent effects, a critical aspect that existing methods fail to account for. Extensive experiments show that Neural Texture Splatting consistently improves models and achieves state-of-the-art results across multiple benchmarks.
Authors Yiming Wang, Shaofei Wang, Marko Mihajlovic, Siyu Tang
Comment SIGGRAPH Asia 2025 (conference track), Project page: https://19reborn.github.io/nts/
Categories Computer Vision and Pattern Recognition
0
Benchmarking Endoscopic Surgical Image Restoration and Beyond
Title CN 内窥镜手术图像恢复的基准测试及其他
Keywords 图像恢复 · 内窥镜手术 · 去烟去雾 · 医疗图像处理 · 基准测试
Summary 本文提出了一个名为SurgClean的真实世界开源手术图像恢复数据集，涵盖去烟、去雾和去溅射三种退化类型，包含3,113张带标注图像，并基于此建立了标准化评估基准，测试了22种通用和特定任务的图像恢复方法。实验表明现有算法与临床需求仍有较大差距。研究还探讨了手术场景与自然场景在结构感知和语义理解上的退化差异，为领域专用图像恢复提供洞见。
Reason 该论文聚焦于医疗内窥镜图像中的去烟、去雾、去溅射等图像恢复任务，属于纯图像修复领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其方法和数据集均针对手术场景的视觉退化问题，无法迁移至解决OCR幻觉、复杂排版解析或长文档推理等DIU核心痛点。
Abstract In endoscopic surgery, a clear and high-quality visual field is critical for surgeons to make accurate intraoperative decisions. However, persistent visual degradation, including smoke generated by energy devices, lens fogging from thermal gradients, and lens contamination due to blood or tissue fluid splashes during surgical procedures, severely impairs visual clarity. These degenerations can seriously hinder surgical workflow and pose risks to patient safety. To systematically investigate and address various forms of surgical scene degradation, we introduce a real- world open-source surgical image restoration dataset covering endoscopic environments, called SurgClean, which involves multi-type image restoration tasks from two medical sites, i.e., desmoking, defogging, and desplashing. SurgClean comprises 3,113 images with diverse degradation types and corresponding paired reference labels. Based on SurgClean, we establish a standardized evaluation benchmark and provide performance for 22 representative generic task-specific image restoration approaches, including 12 generic and 10 task-specific image restoration approaches. Experimental results reveal substantial performance gaps relative to clinical requirements, highlighting a critical opportunity for algorithm advancements in intelligent surgical restoration. Furthermore, we explore the degradation discrepancies between surgical and natural scenes from structural perception and semantic under- standing perspectives, providing fundamental insights for domain-specific image restoration research. Our work aims to empower restoration algorithms and improve the efficiency of clinical procedures.
Authors Jialun Pei, Diandian Guo, Donghui Yang, Zhixi Li, Yuxin Feng, Long Ma, Bo Du, Pheng-Ann Heng
Categories Computer Vision and Pattern Recognition
0
DualGazeNet: A Biologically Inspired Dual-Gaze Query Network for Salient Object Detection
Title CN DualGazeNet：一种受生物启发的双注视查询网络用于显著性目标检测
Keywords 显著性目标检测 · Transformer · 生物启发模型 · 双通路机制 · 轻量化网络
Summary 本文提出DualGazeNet，一种受人类视觉系统启发的纯Transformer架构，通过模拟大脑中大细胞-小细胞双通路处理机制和皮层注意力调节，实现高效、准确的显著性目标检测。实验表明其在多个SOD基准上超越现有方法，同时具备更高的推理速度和更低的计算开销。
Reason 该论文研究的是通用图像中的显著性目标检测（SOD），属于计算机视觉中的通用图像理解任务，与文档图像理解（DIU）的核心任务如DocVQA、表格识别、布局分析等无直接关联。尽管其提出的生物启发结构可能在理论上对注意力机制设计有启发，但未针对文档特有的密集文本、复杂排版或OCR相关幻觉等问题进行建模，且方法聚焦于自然图像中的显著性检测，迁移到文档场景的价值极低。
Abstract Recent salient object detection (SOD) methods aim to improve performance in four key directions: semantic enhancement, boundary refinement, auxiliary task supervision, and multi-modal fusion. In pursuit of continuous gains, these approaches have evolved toward increasingly sophisticated architectures with multi-stage pipelines, specialized fusion modules, edge-guided learning, and elaborate attention mechanisms. However, this complexity paradoxically introduces feature redundancy and cross-component interference that obscure salient cues, ultimately reaching performance bottlenecks. In contrast, human vision achieves efficient salient object identification without such architectural complexity. This contrast raises a fundamental question: can we design a biologically grounded yet architecturally simple SOD framework that dispenses with most of this engineering complexity, while achieving state-of-the-art accuracy, computational efficiency, and interpretability? In this work, we answer this question affirmatively by introducing DualGazeNet, a biologically inspired pure Transformer framework that models the dual biological principles of robust representation learning and magnocellular-parvocellular dual-pathway processing with cortical attention modulation in the human visual system. Extensive experiments on five RGB SOD benchmarks show that DualGazeNet consistently surpasses 25 state-of-the-art CNN- and Transformer-based methods. On average, DualGazeNet achieves about 60\% higher inference speed and 53.4\% fewer FLOPs than four Transformer-based baselines of similar capacity (VST++, MDSAM, Sam2unet, and BiRefNet). Moreover, DualGazeNet exhibits strong cross-domain generalization, achieving leading or highly competitive performance on camouflaged and underwater SOD benchmarks without relying on additional modalities.
Authors Yu Zhang, Haoan Ping, Yuchen Li, Zhenshan Bing, Fuchun Sun, Alois Knoll
Categories Computer Vision and Pattern Recognition
0
Prompt-guided Disentangled Representation for Action Recognition
Title CN 用于动作识别的提示引导解耦表示
Keywords 动作识别 · 解耦表示 · 提示学习 · 时空场景图 · 视频理解
Summary 本文提出了一种名为ProDA的框架，通过引入时空场景图（SSGs）和动态提示模块（DPM）来引导图解析神经网络（GPNN）生成特定动作的表示，从而从复杂多动作场景中解耦指定动作。实验表明该方法在视频动作识别任务上优于现有方法。
Reason 该论文研究的是视频理解中的动作识别任务，属于无关领域（视频理解/生成），与文档图像理解（DIU）的核心任务如DocVQA、OCR、版面分析等无直接关联。尽管其提出的Prompt-guided Disentangled Representation机制在理论上具有模块化建模的潜力，但其应用场景为多动作场景下的特征分离，迁移到文档理解中解决OCR幻觉、复杂排版或长文档推理等问题极为困难，且未见任何可迁移的技术设计（如视觉压缩、高分辨率处理、细粒度定位等）。
Abstract Action recognition is a fundamental task in video understanding. Existing methods typically extract unified features to process all actions in one video, which makes it challenging to model the interactions between different objects in multi-action scenarios. To alleviate this issue, we explore disentangling any specified actions from complex scenes as an effective solution. In this paper, we propose Prompt-guided Disentangled Representation for Action Recognition (ProDA), a novel framework that disentangles any specified actions from a multi-action scene. ProDA leverages Spatio-temporal Scene Graphs (SSGs) and introduces Dynamic Prompt Module (DPM) to guide a Graph Parsing Neural Network (GPNN) in generating action-specific representations. Furthermore, we design a video-adapted GPNN that aggregates information using dynamic weights. Experiments in video action recognition demonstrate the effectiveness of our approach when compared with the state-of-the-art methods. Our code can be found in https://github.com/iamsnaping/ProDA.git
Authors Tianci Wu, Guangming Zhu, Jiang Lu, Siyuan Wang, Ning Wang, Nuoye Xiong, Zhang Liang
Categories Computer Vision and Pattern Recognition
0
Sketch-1-to-3: One Single Sketch to 3D Detailed Face Reconstruction ACM MMAsia 2025
Title CN Sketch-1-to-3：单张草图到3D精细人脸重建
Keywords 3D人脸重建 · 草图到3D · 跨模态生成 · 域适应 · 细粒度纹理
Summary 本文提出Sketch-1-to-3框架，用于从单张手绘草图生成高保真的3D人脸模型。通过引入几何轮廓与纹理细节（GCTD）模块、域适应结构和定制损失函数，在真实和合成数据集上实现了最先进的3D人脸重建效果。
Reason 该论文研究的是从单张2D草图重建3D人脸，属于3D视觉与跨模态生成领域，核心任务为3D人脸重建，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其技术重点在于缩小2D草图与3D面部结构之间的模态差距，解决的是几何轮廓提取、表情与纹理细节恢复等问题，无法迁移用于解决OCR幻觉、复杂排版分析、长文档推理等DIU核心痛点。
Abstract 3D face reconstruction from a single sketch is a critical yet underexplored task with significant practical applications. The primary challenges stem from the substantial modality gap between 2D sketches and 3D facial structures, including: (1) accurately extracting facial keypoints from 2D sketches; (2) preserving diverse facial expressions and fine-grained texture details; and (3) training a high-performing model with limited data. In this paper, we propose Sketch-1-to-3, a novel framework for realistic 3D face reconstruction from a single sketch, to address these challenges. Specifically, we first introduce the Geometric Contour and Texture Detail (GCTD) module, which enhances the extraction of geometric contours and texture details from facial sketches. Additionally, we design a deep learning architecture with a domain adaptation module and a tailored loss function to align sketches with the 3D facial space, enabling high-fidelity expression and texture reconstruction. To facilitate evaluation and further research, we construct SketchFaces, a real hand-drawn facial sketch dataset, and Syn-SketchFaces, a synthetic facial sketch dataset. Extensive experiments demonstrate that Sketch-1-to-3 achieves state-of-the-art performance in sketch-based 3D face reconstruction.
Authors Liting Wen, Zimo Yang, Xianlin Zhang, Chi Ding, Mingdao Wang, Xueming Li
Comment Accepted by ACM MMAsia 2025
Categories Computer Vision and Pattern Recognition
0
Robust Long-term Test-Time Adaptation for 3D Human Pose Estimation through Motion Discretization AAAI 2026
Title CN 通过运动离散化实现3D人体姿态估计的鲁棒长期测试时自适应
Keywords 3D人体姿态估计 · 测试时自适应 · 错误累积 · 运动离散化 · 软重置
Summary 本文提出一种基于运动离散化的长期在线测试时自适应方法，用于缓解3D人体姿态估计中因自我监督预测错误累积导致的性能下降问题。通过在潜在动作表示空间中进行无监督聚类，构建锚点动作以辅助监督，并引入软重置机制回退到指数移动平均权重，从而提升对个体个性化形态与运动特征的长期捕捉能力。
Reason 该论文研究的是3D人体姿态估计中的测试时自适应问题，属于具身智能与视频理解领域，与文档图像理解（DIU）或视觉语言模型在文档场景下的应用无直接关联。尽管其提出的运动离散化和软重置机制可能在时间序列建模中有启发意义，但迁移到文档理解任务（如OCR、表格识别、DocVQA）极为困难，且不涉及多模态语言模型、视觉压缩、高分辨率处理或去幻觉等DIU核心痛点。
Abstract Online test-time adaptation addresses the train-test domain gap by adapting the model on unlabeled streaming test inputs before making the final prediction. However, online adaptation for 3D human pose estimation suffers from error accumulation when relying on self-supervision with imperfect predictions, leading to degraded performance over time. To mitigate this fundamental challenge, we propose a novel solution that highlights the use of motion discretization. Specifically, we employ unsupervised clustering in the latent motion representation space to derive a set of anchor motions, whose regularity aids in supervising the human pose estimator and enables efficient self-replay. Additionally, we introduce an effective and efficient soft-reset mechanism by reverting the pose estimator to its exponential moving average during continuous adaptation. We examine long-term online adaptation by continuously adapting to out-of-domain streaming test videos of the same individual, which allows for the capture of consistent personal shape and motion traits throughout the streaming observation. By mitigating error accumulation, our solution enables robust exploitation of these personal traits for enhanced accuracy. Experiments demonstrate that our solution outperforms previous online test-time adaptation methods and validate our design choices.
Authors Yilin Wen, Kechuan Dong, Yusuke Sugano
Comment Accepted by AAAI 2026, main track
Categories Computer Vision and Pattern Recognition
0
Synergistic Bleeding Region and Point Detection in Laparoscopic Surgical Videos
Title CN 腹腔镜手术视频中协同的出血区域与点检测
Keywords 出血检测 · 腹腔镜手术视频 · 双任务学习 · 光流 · 记忆建模
Summary 本文提出了一种名为BlooDet的双任务协同在线检测器，用于在腹腔镜手术视频中同时检测出血区域和出血点。基于Segment Anything Model 2构建双分支双向引导结构，利用边缘与点提示嵌入检测出血区域，并通过掩码记忆和帧间光流捕捉出血点及其运动方向。作者还发布了包含5330帧的真实世界数据集SurgBlood。实验表明该方法在出血检测任务上优于13种现有方法。
Reason 该论文属于医学图像分析领域，研究任务为手术视频中的出血区域和点检测，属于视频理解与医疗影像分析范畴，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法论聚焦于视频流中的动态出血检测与运动方向捕捉，使用光流和记忆建模，难以迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Intraoperative bleeding in laparoscopic surgery causes rapid obscuration of the operative field to hinder the surgical process and increases the risk of postoperative complications. Intelligent detection of bleeding areas can quantify the blood loss to assist decision-making, while locating bleeding points helps surgeons quickly identify the source of bleeding and achieve hemostasis in time to improve surgical success rates. To fill the benchmark gap, we first construct a real-world laparoscopic surgical bleeding detection dataset, named SurgBlood, comprising 5,330 frames from 95 surgical video clips with bleeding region and point annotations. Accordingly, we develop a dual-task synergistic online detector called BlooDet, enabling simultaneous detection of bleeding regions and points in laparoscopic surgery. The baseline embraces a dual-branch bidirectional guid- ance design based on Segment Anything Model 2. The mask branch detects bleeding regions through adaptive edge and point prompt embeddings, while the point branch leverages mask memory to induce bleeding point memory modeling and captures point motion direction via inter-frame optical flow. By coupled bidirectional guidance, our framework explores spatial-temporal correlations while exploiting memory modeling to infer current bleeding status. Extensive experiments indicate that our method outperforms 13 counterparts in bleeding detection.
Authors Jialun Pei, Zhangjun Zhou, Diandian Guo, Zhixi Li, Jing Qin, Bo Du, Pheng-Ann Heng
Categories Computer Vision and Pattern Recognition
0
DSOcc: Leveraging Depth Awareness and Semantic Aid to Boost Camera-Based 3D Semantic Occupancy Prediction
Title CN DSOcc：利用深度感知和语义辅助提升基于相机的3D语义占据预测
Keywords 3D语义占据 · 深度感知 · 语义分割 · 自动驾驶 · 多帧融合
Summary 本文提出DSOcc方法，通过联合进行占据状态和类别推断，结合非学习方式计算软占据置信度，并利用预训练的语义分割结果辅助多帧融合，以提升基于相机的3D语义占据预测性能。在SemanticKITTI等多个数据集上取得了领先表现。
Reason 该论文属于自动驾驶场景下的3D视觉任务，研究的是基于相机的3D语义占据预测，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用完全无关。其方法聚焦于深度感知、多帧融合与语义分割辅助的占据网格推理，无法迁移到文档图像中的OCR去幻觉、复杂排版理解、细粒度定位等核心痛点。
Abstract Camera-based 3D semantic occupancy prediction offers an efficient and cost-effective solution for perceiving surrounding scenes in autonomous driving. However, existing works rely on explicit occupancy state inference, leading to numerous incorrect feature assignments, and insufficient samples restrict the learning of occupancy class inference. To address these challenges, we propose leveraging \textbf{D}epth awareness and \textbf{S}emantic aid to boost camera-based 3D semantic \textbf{Occ}upancy prediction (\textbf{DSOcc}). We jointly perform occupancy state and occupancy class inference, where soft occupancy confidence is calculated by non-learning method and multiplied with image features to make voxels aware of depth, enabling adaptive implicit occupancy state inference. Instead of enhancing feature learning, we directly utilize well-trained image semantic segmentation and fuse multiple frames with their occupancy probabilities to aid occupancy class inference, thereby enhancing robustness. Experimental results demonstrate that DSOcc achieves state-of-the-art performance on the SemanticKITTI dataset among camera-based methods and achieves competitive performance on the SSCBench-KITTI-360 and Occ3D-nuScenes datasets. Code will be released on github.
Authors Naiyu Fang, Zheyuan Zhou, Kang Wang, Ruibo Li, Lemiao Qiu, Shuyou Zhang, Zhe Wang, Guosheng Lin
Categories Computer Vision and Pattern Recognition
0
JointTuner: Appearance-Motion Adaptive Joint Training for Customized Video Generation
Title CN JointTuner：面向定制化视频生成的外观-运动自适应联合训练
Keywords 视频生成 · 外观-运动解耦 · 低秩适应 · 扩散模型 · 时序一致性
Summary 本文提出JointTuner，通过联合优化外观与运动模块来提升定制化视频生成的质量。引入了Gated Low-Rank Adaptation（GLoRA）以动态控制LoRA模块学习外观或运动特征，并设计Appearance-independent Temporal Loss（AiT Loss）增强运动模式的学习。方法适用于UNet和Diffusion Transformer架构，在多个维度上系统评估了外观-运动协同定制效果。
Reason 该论文研究的是定制化视频生成中的外观与运动联合优化问题，属于纯视频生成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其提出的方法如GLoRA和AiT Loss主要针对视频中动态内容的建模，并不解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心痛点。尽管其方法涉及多模态建模，但应用场景完全偏离文档理解范畴。
Abstract Recent advancements in customized video generation have led to significant improvements in the simultaneous adaptation of appearance and motion. Typically, decoupling the appearance and motion training, prior methods often introduce concept interference, resulting in inaccurate rendering of appearance features or motion patterns. In addition, these methods often suffer from appearance contamination, in which background and foreground elements from reference videos distort the customized video. This paper aims to alleviate these issues by proposing JointTuner. The core motivation of our JointTuner is to enable joint optimization of both appearance and motion components, upon which two key innovations are developed, i.e., Gated Low-Rank Adaptation (GLoRA) and Appearance-independent Temporal Loss (AiT Loss). Specifically, GLoRA uses a context-aware activation layer, analogous to a gating regulator, to dynamically steer LoRA modules toward learning either appearance or motion while maintaining spatio-temporal consistency. Moreover, with the finding that channel-temporal shift noise suppresses appearance-related low-frequencies while enhancing motion-related high-frequencies, we designed the AiT Loss. This loss adds the same shift to the diffusion model's predicted noise during fine-tuning, forcing the model to prioritize learning motion patterns. JointTuner's architecture-agnostic design supports both UNet (e.g., ZeroScope) and Diffusion Transformer (e.g., CogVideoX) backbones, ensuring its customization capabilities scale with the evolution of foundational video models. Furthermore, we present a systematic evaluation framework for appearance-motion combined customization, covering 90 combinations evaluated along four critical dimensions: semantic alignment, motion dynamism, temporal consistency, and perceptual quality. Our project homepage is available online.
Authors Fangda Chen, Shanshan Zhao, Chuanfu Xu, Long Lan
Comment Project Page: https://fdchen24.github.io/JointTuner-Website
Categories Computer Vision and Pattern Recognition
0
Enhancing Multi-Label Thoracic Disease Diagnosis with Deep Ensemble-Based Uncertainty Quantification
Title CN 利用基于深度集成的不确定性量化增强多标签胸部疾病诊断
Keywords 不确定性量化 · 深度集成 · 医学影像分析 · 多标签分类 · 模型校准
Summary 本文在NIH ChestX-ray14数据集上构建了一个基于深度集成的多标签胸部疾病诊断系统，通过引入不确定性量化（UQ）提升模型的可靠性和校准性。相比失败的Monte Carlo Dropout方案，9成员的深度集成取得了更好的AUROC、F1分数和更低的预期校准误差（ECE），并能分解认知不确定性和偶然不确定性，增强了临床决策支持的可信度。
Reason 该论文属于医疗影像诊断领域的应用研究，聚焦于胸部X光片的疾病分类与不确定性建模，属于典型的‘平行下游应用’。尽管其方法（深度集成、不确定性量化）在模型可靠性方面有技术价值，但并未针对文档图像理解（DIU）中的核心问题如OCR幻觉、复杂版面分析、文本定位或VLM多模态对齐进行任何改进或适配，且应用场景完全偏离文档范畴。
Abstract The utility of deep learning models, such as CheXNet, in high stakes clinical settings is fundamentally constrained by their purely deterministic nature, failing to provide reliable measures of predictive confidence. This project addresses this critical gap by integrating robust Uncertainty Quantification (UQ) into a high performance diagnostic platform for 14 common thoracic diseases on the NIH ChestX-ray14 dataset. Initial architectural development failed to stabilize performance and calibration using Monte Carlo Dropout (MCD), yielding an unacceptable Expected Calibration Error (ECE) of 0.7588. This technical failure necessitated a rigorous architectural pivot to a high diversity, 9-member Deep Ensemble (DE). This resulting DE successfully stabilized performance and delivered superior reliability, achieving a State-of-the-Art (SOTA) average Area Under the Receiver Operating Characteristic Curve (AUROC) of 0.8559 and an average F1 Score of 0.3857. Crucially, the DE demonstrated superior calibration (Mean ECE of 0.0728 and Negative Log-Likelihood (NLL) of 0.1916) and enabled the reliable decomposition of total uncertainty into its Aleatoric (irreducible data noise) and Epistemic (reducible model knowledge) components, with a mean Epistemic Uncertainty (EU) of 0.0240. These results establish the Deep Ensemble as a trustworthy and explainable platform, transforming the model from a probabilistic tool into a reliable clinical decision support system.
Authors Yasiru Laksara, Uthayasanker Thayasivam
Categories Computer Vision and Pattern Recognition
0
Matrix-game 2.0: An open-source real-time and streaming interactive world model
Title CN Matrix-game 2.0：一个开源的实时流式交互世界模型
Keywords 交互式世界模型 · 视频生成 · 扩散模型 · 实时生成 · 自回归模型
Summary 本文提出了Matrix-Game 2.0，一种基于扩散模型的实时交互式世界模型，能够在Unreal Engine和GTA5等环境中通过少量步数的自回归生成分钟级高质量视频，支持帧级鼠标键盘输入控制。系统包含大规模交互数据构建流程、动作注入模块和因果架构下的少步蒸馏方法，实现了25 FPS的超快生成速度。项目已开源。
Reason 该论文属于视频生成与交互式世界模型领域，核心目标是实现实时、长程的交互视频生成，技术重点在于扩散模型的自回归加速与动作注入。尽管其高分辨率视频生成能力看似相关，但其任务本质（物理动态模拟、游戏环境交互）与文档图像理解（DIU）的核心痛点（如OCR幻觉、文本结构解析、逻辑推理）无直接关联。方法难以迁移至文档场景，且未涉及多模态语言模型在文档上的语义理解或结构建模。属于无关领域（视频生成/具身智能模拟），命中负面清单。
Abstract Recent advances in interactive video generations have demonstrated diffusion model's potential as world models by capturing complex physical dynamics and interactive behaviors. However, existing interactive world models depend on bidirectional attention and lengthy inference steps, severely limiting real-time performance. Consequently, they are hard to simulate real-world dynamics, where outcomes must update instantaneously based on historical context and current actions. To address this, we present Matrix-Game 2.0, an interactive world model generates long videos on-the-fly via few-step auto-regressive diffusion. Our framework consists of three key components: (1) A scalable data production pipeline for Unreal Engine and GTA5 environments to effectively produce massive amounts (about 1200 hours) of video data with diverse interaction annotations; (2) An action injection module that enables frame-level mouse and keyboard inputs as interactive conditions; (3) A few-step distillation based on the casual architecture for real-time and streaming video generation. Matrix Game 2.0 can generate high-quality minute-level videos across diverse scenes at an ultra-fast speed of 25 FPS. We open-source our model weights and codebase to advance research in interactive world modeling.
Authors Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, Baixin Xu, Hao-Xiang Guo, Kaixiong Gong, Cyrus Wu, Wei Li, Xuchen Song, Yang Liu, Eric Li, Yahui Zhou
Comment Project Page: https://matrix-game-v2.github.io
Categories Computer Vision and Pattern Recognition
0
VideoCompressa: Data-Efficient Video Understanding via Joint Temporal Compression and Spatial Reconstruction
Title CN VideoCompressa：通过联合时间压缩与空间重建实现高效视频理解
Keywords 视频理解 · 数据压缩 · 关键帧选择 · 潜变量模型 · VAE
Summary 本文提出VideoCompressa，一种通过联合优化关键帧选择和潜在空间压缩来提升视频数据效率的新框架。该方法利用轻量级ConvNet与Gumbel-Softmax采样选择关键帧，并结合预训练VAE将帧压缩为紧凑的语义潜码，从而在极低数据用量下实现优异的视频理解性能。实验表明其在UCF101和HMDB51上仅用极小比例数据即可超越或匹配全数据训练效果。
Reason 该论文研究的是视频理解中的数据效率问题，属于视频领域，且其方法聚焦于时间冗余压缩和关键帧选择，与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版、细粒度文本识别等无直接关联。尽管‘压缩’概念在形式上与视觉token压缩有相似之处，但视频的时间动态建模与文档的空间结构建模差异巨大，难以迁移应用。
Abstract The scalability of video understanding models is increasingly limited by the prohibitive storage and computational costs of large-scale video datasets. While data synthesis has improved data efficiency in the image domain, its extension to video remains challenging due to pervasive temporal redundancy and complex spatiotemporal dynamics. In this work, we uncover a critical insight: the primary source of inefficiency in video datasets is not inter-sample redundancy, but intra-sample frame-level redundancy. To leverage this insight, we introduce VideoCompressa, a novel framework for video data synthesis that reframes the problem as dynamic latent compression. Specifically, VideoCompressa jointly optimizes a differentiable keyframe selector-implemented as a lightweight ConvNet with Gumbel-Softmax sampling-to identify the most informative frames, and a pretrained, frozen Variational Autoencoder (VAE) to compress these frames into compact, semantically rich latent codes. These latent representations are then fed into a compression network, enabling end-to-end backpropagation. Crucially, the keyframe selector and synthetic latent codes are co-optimized to maximize retention of task-relevant information. Experiments show that our method achieves unprecedented data efficiency: on UCF101 with ConvNets, VideoCompressa surpasses full-data training by 2.34\% points using only 0.13\% of the original data, with over 5800x speedup compared to traditional synthesis method. Moreover, when fine-tuning Qwen2.5-7B-VL on HMDB51, VideoCompressa matches full-data performance using just 0.41\% of the training data-outperforming zero-shot baseline by 10.61\%.
Authors Shaobo Wang, Tianle Niu, Runkang Yang, Deshan Liu, Xu He, Zichen Wen, Conghui He, Xuming Hu, Linfeng Zhang
Comment 15 pages, 6 tables, 8 figures
Categories Computer Vision and Pattern Recognition
0
Leveraging Metaheuristic Approaches to Improve Deep Learning Systems for Anxiety Disorder Detection
Title CN 利用元启发式方法改进用于焦虑障碍检测的深度学习系统
Keywords 焦虑障碍检测 · 深度学习 · 元启发式优化 · 可穿戴传感器 · 特征选择
Summary 本文提出一种结合深度学习与元启发式优化（如遗传算法、粒子群优化）的混合模型，用于从多模态生理和行为数据中自动检测焦虑障碍。通过优化特征空间和超参数，提升了模型准确性和泛化能力。但该工作属于心理健康领域的专用AI系统，不涉及文档图像或视觉语言理解。
Reason 该论文属于医疗健康领域的特定应用，研究目标是通过可穿戴传感器和生理信号结合元启发式优化算法（如遗传算法、粒子群优化）来检测焦虑障碍。其核心为‘AI for Healthcare’的垂直应用，并未涉及文档图像理解（DIU）、视觉语言模型（VLM）、OCR、布局分析、表格识别或任何文档相关任务。技术路线聚焦于特征选择与超参数优化的融合策略，无法迁移解决DIU中的OCR幻觉、复杂排版、细粒度定位等核心痛点，属于典型的平行下游应用。
Abstract Despite being among the most common psychological disorders, anxiety-related conditions are still primarily identified through subjective assessments, such as clinical interviews and self-evaluation questionnaires. These conventional methods often require significant time and may vary depending on the evaluator. However, the emergence of advanced artificial intelligence techniques has created new opportunities for detecting anxiety in a more consistent and automated manner. To address the limitations of traditional approaches, this study introduces a comprehensive model that integrates deep learning architectures with optimization strategies inspired by swarm intelligence. Using multimodal and wearable-sensor datasets, the framework analyzes physiological, emotional, and behavioral signals. Swarm intelligence techniques including genetic algorithms and particle swarm optimization are incorporated to refine the feature space and optimize hyperparameters. Meanwhile, deep learning components are tasked with deriving layered and discriminative representations from sequential, multi-source inputs. Our evaluation shows that the fusion of these two computational paradigms significantly enhances detection performance compared with using deep networks alone. The hybrid model achieves notable improvements in accuracy and demonstrates stronger generalization across various individuals. Overall, the results highlight the potential of combining metaheuristic optimization with deep learning to develop scalable, objective, and clinically meaningful solutions for assessing anxiety disorders
Authors Mohammadreza Amiri, Monireh Hosseini
Comment 12 pages
Categories Computer Vision and Pattern Recognition
0
Q-Save: Towards Scoring and Attribution for Generated Video Evaluation
Title CN Q-Save：面向生成视频评估的评分与归因
Keywords 视频质量评估 · AI生成视频 · 归因分析 · SlowFast · Chain-of-Thought
Summary 本文提出了Q-Save，一个用于AI生成视频质量评估的新基准数据集和模型。数据集包含近万个视频，标注了主观评分（MOS）以及视觉质量、动态质量和文本-视频对齐三个维度的细粒度归因标签。作者提出了一种统一的评估模型，结合SlowFast架构，以高分辨率处理慢帧、低分辨率处理快帧，在保证效率的同时提升准确性。模型通过Chain-of-Thought格式数据训练，并采用多阶段训练策略（SFT → GRPO → SFT）增强稳定性和推理能力。实验表明该模型在预测准确性和可解释性方面均达到SOTA水平。
Reason 该论文聚焦于AI生成视频（AIGV）的质量评估，属于视频理解与生成领域，与文档图像理解（DIU）无直接关联。其核心任务是视频质量打分与归因分析，方法基于SlowFast框架处理动态帧，并采用COT风格训练进行可解释性建模。尽管使用了CoT和多阶段训练策略，但整体设计针对视频模态特性，无法迁移解决文档理解中的OCR幻觉、密集文本识别、复杂版面解析等核心痛点。
Abstract We present Q-Save, a new benchmark dataset and model for holistic and explainable evaluation of AI-generated video (AIGV) quality. The dataset contains near 10000 videos, each annotated with a scalar mean opinion score (MOS) and fine-grained attribution labels along three core dimensions: visual quality, dynamic quality, and text-video alignment. These multi-aspect annotations enable both accurate quality assessment and interpretable reasoning behind the scores. To leverage this data, we propose a unified evaluation model that jointly performs quality scoring and attribution-based explanation. The model adopts the SlowFast framework to distinguish between fast frames and slow frames - slow frames are processed with high resolution while fast frames use low resolution, balancing evaluation accuracy and computational efficiency. For training, we use data formatted in Chain-of-Thought (COT) style and employ a multi-stage strategy: we first conduct Supervised Fine-Tuning (SFT), then further enhance the model with Grouped Relative Policy Optimization (GRPO), and finally perform SFT again to improve model stability. Experimental results demonstrate that our model achieves state-of-the-art performance in video quality prediction while also providing human-aligned, interpretable justifications. Our dataset and model establish a strong foundation for explainable evaluation in generative video research, contributing to the development of multimodal generation and trustworthy AI. Code and dataset will be released upon publication.
Authors Xiele Wu, Zicheng Zhang, Mingtao Chen, Yixian Liu, Yiming Liu, Shushi Wang, Zhichao Hu, Yuhong Liu, Guangtao Zhai, Xiaohong Liu
Comment 20 pages, 11 figures
Categories Computer Vision and Pattern Recognition
0
Assessing the alignment between infants' visual and linguistic experience using multimodal language models
Title CN 使用多模态语言模型评估婴儿视觉与语言经验之间的对齐
Keywords CLIP · vision-language alignment · infant cognition · egocentric video · language acquisition
Summary 论文利用CLIP模型自动评估婴幼儿第一视角视频中语言与视觉内容的时序对齐程度，发现真实场景中理想对齐时刻（如说话提到物体且该物体在视野中）较为稀少，揭示了儿童语言学习环境的真实约束。尽管技术上涉及多模态对齐，但研究目的和数据均远离文档理解领域。
Reason 该论文研究的是儿童语言习得过程中视觉与语言信号的时间对齐问题，属于发展心理学与认知科学领域。虽然使用了CLIP等多模态模型，但其目标并非文档图像理解（DIU）中的任何任务（如DocVQA、布局分析、表格识别等），也不涉及OCR幻觉、密集文本处理、复杂排版或长文档推理等核心痛点。其方法本质上是将现有VLM用于行为数据分析，属于‘平行应用’范畴，且应用场景与文档理解无关。
Abstract Figuring out which objects or concepts words refer to is a central language learning challenge for young children. Most models of this process posit that children learn early object labels from co-occurrences of words and their referents that occur when someone around them talks about an object in the immediate physical environment. But how aligned in time are children's visual and linguistic experiences during everyday learning? To date, answers to this question have been limited by the need for labor-intensive manual annotations of vision-language co-occurrences. Here, we evaluate the use of contrastive language-image pretraining (CLIP) models to automatically characterize vision-language alignment in egocentric videos taken from the infant perspective in home environments. After validating CLIP alignment scores using human alignment judgments, we apply this metric to a large corpus of infant-perspective videos. We show that idealized aligned moments for learning (e.g., "look at the ball" with a ball present in the child's view) are relatively rare in children's everyday experiences compared to modern machine learning datasets, and highlight variability in alignment both within and across children. These findings suggest that infrequent alignment is a constraint for models describing early word learning and offer a new method for investigating children's multimodal environment.
Authors Alvin Wei Ming Tan, Jane Yang, Tarun Sepuri, Khai Loong Aw, Robert Z. Sparks, Zi Yin, Virginia A. Marchman, Michael C. Frank, Bria Long
Categories Computer Vision and Pattern Recognition, Computation and Language
0
DetAny4D: Detect Anything 4D Temporally in a Streaming RGB Video
Title CN DetAny4D：在流式RGB视频中实现4D时空任意物体检测
Keywords 4D检测 · 3D目标检测 · 视频理解 · 时空一致性 · 多模态融合
Summary 本文提出了DetAny4D，一种开放集的端到端4D目标检测框架，能够在流式RGB视频中直接预测具有时空一致性的3D边界框。作者构建了大规模数据集DA4D，并设计了几何感知的时空解码器和多任务学习策略，以提升检测的准确性和时间稳定性。实验表明该方法有效缓解了传统方法中存在的抖动和不一致问题。
Reason 该论文研究的是4D（时空）物体检测，属于自动驾驶或视频理解领域，核心任务为3D目标检测在视频流中的扩展，与文档图像理解（DIU）或DocVQA无关。其方法聚焦于多模态特征融合和时空一致性建模，难以迁移到文档的文本结构理解、OCR去幻觉或复杂排版解析等核心痛点。属于无关领域。
Abstract Reliable 4D object detection, which refers to 3D object detection in streaming video, is crucial for perceiving and understanding the real world. Existing open-set 4D object detection methods typically make predictions on a frame-by-frame basis without modeling temporal consistency, or rely on complex multi-stage pipelines that are prone to error propagation across cascaded stages. Progress in this area has been hindered by the lack of large-scale datasets that capture continuous reliable 3D bounding box (b-box) annotations. To overcome these challenges, we first introduce DA4D, a large-scale 4D detection dataset containing over 280k sequences with high-quality b-box annotations collected under diverse conditions. Building on DA4D, we propose DetAny4D, an open-set end-to-end framework that predicts 3D b-boxes directly from sequential inputs. DetAny4D fuses multi-modal features from pre-trained foundational models and designs a geometry-aware spatiotemporal decoder to effectively capture both spatial and temporal dynamics. Furthermore, it adopts a multi-task learning architecture coupled with a dedicated training strategy to maintain global consistency across sequences of varying lengths. Extensive experiments show that DetAny4D achieves competitive detection accuracy and significantly improves temporal stability, effectively addressing long-standing issues of jitter and inconsistency in 4D object detection. Data and code will be released upon acceptance.
Authors Jiawei Hou, Shenghao Zhang, Can Wang, Zheng Gu, Yonggen Ling, Taiping Zeng, Xiangyang Xue, Jingbo Zhang
Categories Computer Vision and Pattern Recognition
0
TPG-INR: Target Prior-Guided Implicit 3D CT Reconstruction for Enhanced Sparse-view Imaging
Title CN TPG-INR：用于增强稀疏视图成像的目标先验引导隐式3D CT重建
Keywords 3D重建 · 隐式神经表示 · 稀疏视图CT · 医学影像 · NeRF
Summary 本文提出了一种目标先验引导的隐式3D CT重建框架TPG-INR，利用从稀疏投影中提取的解剖结构先验信息来指导体素级隐式重建，提升在极稀疏视角下的重建精度与学习效率。通过位置与结构编码结合，并引入CUDA加速算法快速估计高质量3D目标先验，在腹部数据集上显著优于现有方法。
Reason 该论文研究的是基于X射线投影数据的3D医学CT图像重建，属于医学影像重建领域，核心任务为隐式神经表示在三维体数据重构中的应用。尽管使用了视觉输入和深度学习方法，但其目标与文档图像理解（DIU）无关，不涉及文本识别、布局分析、表格理解或VLM对文档内容的推理。属于无关领域（3D视觉+医学成像），不符合关注方向。
Abstract X-ray imaging, based on penetration, enables detailed visualization of internal structures. Building on this capability, existing implicit 3D reconstruction methods have adapted the NeRF model and its variants for internal CT reconstruction. However, these approaches often neglect the significance of objects' anatomical priors for implicit learning, limiting both reconstruction precision and learning efficiency, particularly in ultra-sparse view scenarios. To address these challenges, we propose a novel 3D CT reconstruction framework that employs a 'target prior' derived from the object's projection data to enhance implicit learning. Our approach integrates positional and structural encoding to facilitate voxel-wise implicit reconstruction, utilizing the target prior to guide voxel sampling and enrich structural encoding. This dual strategy significantly boosts both learning efficiency and reconstruction quality. Additionally, we introduce a CUDA-based algorithm for rapid estimation of high-quality 3D target priors from sparse-view projections. Experiments utilizing projection data from a complex abdominal dataset demonstrate that the proposed model substantially enhances learning efficiency, outperforming the current leading model, NAF, by a factor of ten. In terms of reconstruction quality, it also exceeds the most accurate model, NeRP, achieving PSNR improvements of 3.57 dB, 5.42 dB, and 5.70 dB with 10, 20, and 30 projections, respectively. The code is available at https://github.com/qlcao171/TPG-INR.
Authors Qinglei Cao, Ziyao Tang, Xiaoqin Tang
Comment Please consider this version as the latest camera-ready version
Categories Computer Vision and Pattern Recognition
0
PartDiffuser: Part-wise 3D Mesh Generation via Discrete Diffusion
Title CN PartDiffuser：通过离散扩散实现分部件的3D网格生成
Keywords 3D网格生成 · 扩散模型 · 语义部件 · 点云 · DiT
Summary 本文提出PartDiffuser，一种基于DiT架构的半自回归扩散框架，用于从点云生成高质量3D网格。该方法先对网格进行语义分割，然后在部件间采用自回归建模以保证全局结构，在部件内使用并行离散扩散过程来恢复高频几何特征，并引入部件感知的交叉注意力机制，利用点云作为分层几何条件动态控制生成过程。实验表明其在生成富含细节的3D艺术网格方面优于现有SOTA方法。
Reason 该论文研究的是3D网格生成，属于3D视觉与图形学领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管其提出了分部件的扩散生成机制和部分感知的注意力结构，但核心任务为点云到网格的生成，解决的是几何细节重建与全局拓扑一致性问题，无法迁移至文档理解中的OCR幻觉、复杂排版解析或长文档推理等关键痛点。
Abstract Existing autoregressive (AR) methods for generating artist-designed meshes struggle to balance global structural consistency with high-fidelity local details, and are susceptible to error accumulation. To address this, we propose PartDiffuser, a novel semi-autoregressive diffusion framework for point-cloud-to-mesh generation. The method first performs semantic segmentation on the mesh and then operates in a "part-wise" manner: it employs autoregression between parts to ensure global topology, while utilizing a parallel discrete diffusion process within each semantic part to precisely reconstruct high-frequency geometric features. PartDiffuser is based on the DiT architecture and introduces a part-aware cross-attention mechanism, using point clouds as hierarchical geometric conditioning to dynamically control the generation process, thereby effectively decoupling the global and local generation tasks. Experiments demonstrate that this method significantly outperforms state-of-the-art (SOTA) models in generating 3D meshes with rich detail, exhibiting exceptional detail representation suitable for real-world applications.
Authors Yichen Yang, Hong Li, Haodong Zhu, Linin Yang, Guojun Lei, Sheng Xu, Baochang Zhang
Categories Computer Vision and Pattern Recognition
0
ChronoGS: Disentangling Invariants and Changes in Multi-Period Scenes
Title CN ChronoGS：解耦多时段场景中的不变性与变化
Keywords 3D重建 · 多时段场景 · 高斯表示 · 动态场景 · 时间一致性
Summary 本文提出ChronoGS，一种基于时间调制的高斯表示方法，用于统一建模多时段场景的几何与外观演变，并通过解耦稳定与变化成分提升跨时段重建的一致性。作者同时发布了包含真实与合成数据的ChronoScene基准数据集。实验表明该方法在重建质量与时间一致性上优于现有基线。
Reason 该论文研究的是多时段场景的三维重建问题，核心任务是处理城市、建筑或自然区域在长时间跨度下的几何与外观变化，属于动态3D场景重建领域。其方法基于高斯表示（Gaussian Splatting）进行时间调制，目标是实现跨时段的一致性重建。这与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管涉及‘多时段变化’和‘解耦表示’，但这些概念难以迁移到文档视觉理解中解决OCR幻觉、密集文本识别或复杂排版解析等问题。且该工作属于3D视觉与场景重建范畴，不在DIU本题范围内，也不提供可迁移至文档理解的通用方法论工具。
Abstract Multi-period image collections are common in real-world applications. Cities are re-scanned for mapping, construction sites are revisited for progress tracking, and natural regions are monitored for environmental change. Such data form multi-period scenes, where geometry and appearance evolve. Reconstructing such scenes is an important yet underexplored problem. Existing pipelines rely on incompatible assumptions: static and in-the-wild methods enforce a single geometry, while dynamic ones assume smooth motion, both failing under long-term, discontinuous changes. To solve this problem, we introduce ChronoGS, a temporally modulated Gaussian representation that reconstructs all periods within a unified anchor scaffold. It's also designed to disentangle stable and evolving components, achieving temporally consistent reconstruction of multi-period scenes. To catalyze relevant research, we release ChronoScene dataset, a benchmark of real and synthetic multi-period scenes, capturing geometric and appearance variation. Experiments demonstrate that ChronoGS consistently outperforms baselines in reconstruction quality and temporal consistency. Our code and the ChronoScene dataset are publicly available at https://github.com/ZhongtaoWang/ChronoGS.
Authors Zhongtao Wang, Jiaqi Dai, Qingtian Zhu, Yilong Li, Mai Su, Fei Zhu, Meng Gai, Shaorong Wang, Chengwei Pan, Yisong Chen, Guoping Wang
Categories Computer Vision and Pattern Recognition
0
Scale What Counts, Mask What Matters: Evaluating Foundation Models for Zero-Shot Cross-Domain Wi-Fi Sensing
Title CN 关注重要之处，屏蔽关键干扰：评估用于零样本跨域Wi-Fi感知的基础模型
Keywords Wi-Fi Sensing · Masked Autoencoding · Cross-Domain Generalization · Foundation Models · CSI
Summary 本文提出在大规模异构Wi-Fi CSI数据集上采用掩码自编码（MAE）方式进行基础模型预训练，以解决Wi-Fi感知中的跨域泛化问题。实验表明数据规模和多样性比模型容量对性能影响更大，并在人类活动识别、手势识别等任务上验证了方法的有效性。然而，该工作与文档图像理解无直接关联。
Reason 该论文研究的是Wi-Fi CSI（信道状态信息）感知中的跨域泛化问题，属于无线传感领域，与文档图像理解（DIU）、视觉语言模型（VLM）或OCR/DocVQA等任务完全无关。尽管使用了MAE预训练和大规模数据评估，但其方法和应用场景无法迁移到文档理解的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）。属于无关领域。
Abstract While Wi-Fi sensing offers a compelling, privacy-preserving alternative to cameras, its practical utility has been fundamentally undermined by a lack of robustness across domains. Models trained in one setup fail to generalize to new environments, hardware, or users, a critical "domain shift" problem exacerbated by modest, fragmented public datasets. We shift from this limited paradigm and apply a foundation model approach, leveraging Masked Autoencoding (MAE) style pretraining on the largest and most heterogeneous Wi-Fi CSI datasets collection assembled to date. Our study pretrains and evaluates models on over 1.3 million samples extracted from 14 datasets, collected using 4 distinct devices across the 2.4/5/6 GHz bands and bandwidths from 20 to 160 MHz. Our large-scale evaluation is the first to systematically disentangle the impacts of data diversity versus model capacity on cross-domain performance. The results establish scaling trends on Wi-Fi CSI sensing. First, our experiments show log-linear improvements in unseen domain performance as the amount of pretraining data increases, suggesting that data scale and diversity are key to domain generalization. Second, based on the current data volume, larger model can only provide marginal gains for cross-domain performance, indicating that data, rather than model capacity, is the current bottleneck for Wi-Fi sensing generalization. Finally, we conduct a series of cross-domain evaluations on human activity recognition, human gesture recognition and user identification tasks. The results show that the large-scale pretraining improves cross-domain accuracy ranging from 2.2% to 15.7%, compared to the supervised learning baseline. Overall, our findings provide insightful direction for designing future Wi-Fi sensing systems that can eventually be robust enough for real-world deployment.
Authors Cheng Jiang, Yihe Yan, Yanxiang Wang, Chun Tung Chou, Wen Hu
Categories Computer Vision and Pattern Recognition
0
StereoDETR: Stereo-based Transformer for 3D Object Detection IEEE TCSVT, 2025
Title CN StereoDETR：基于立体视觉的Transformer用于3D目标检测
Keywords 3D目标检测 · 立体视觉 · Transformer · DETR · 深度估计
Summary 本文提出StereoDETR，一种基于DETR的高效立体视觉3D目标检测框架，通过双分支结构和可微分深度采样策略，在KITTI数据集上实现了优于单目方法的精度和速度平衡。
Reason 该论文属于自动驾驶领域的3D目标检测，研究的是立体图像输入下的物体深度估计与定位，与文档图像理解（DIU）任务无直接关联。其核心技术如多尺度视差特征、深度采样策略等难以迁移到文档场景中的OCR去幻觉、复杂排版解析或长文档推理等核心痛点。属于无关领域（自动驾驶/3D视觉），命中负面清单第2条。
Abstract Compared to monocular 3D object detection, stereo-based 3D methods offer significantly higher accuracy but still suffer from high computational overhead and latency. The state-of-the-art stereo 3D detection method achieves twice the accuracy of monocular approaches, yet its inference speed is only half as fast. In this paper, we propose StereoDETR, an efficient stereo 3D object detection framework based on DETR. StereoDETR consists of two branches: a monocular DETR branch and a stereo branch. The DETR branch is built upon 2D DETR with additional channels for predicting object scale, orientation, and sampling points. The stereo branch leverages low-cost multi-scale disparity features to predict object-level depth maps. These two branches are coupled solely through a differentiable depth sampling strategy. To handle occlusion, we introduce a constrained supervision strategy for sampling points without requiring extra annotations. StereoDETR achieves real-time inference and is the first stereo-based method to surpass monocular approaches in speed. It also achieves competitive accuracy on the public KITTI benchmark, setting new state-of-the-art results on pedestrian and cyclist subsets. The code is available at https://github.com/shiyi-mu/StereoDETR-OPEN.
Authors Shiyi Mu, Zichong Gu, Zhiqi Ai, Anqi Liu, Yilin Gao, Shugong Xu
Comment Accepted by IEEE TCSVT, 2025
Categories Computer Vision and Pattern Recognition
0
STCDiT: Spatio-Temporally Consistent Diffusion Transformer for High-Quality Video Super-Resolution
Title CN STCDiT：用于高质量视频超分辨率的时空一致性扩散Transformer
Keywords 视频超分辨率 · 扩散模型 · 时空一致性 · VAE · 结构保真
Summary 本文提出了一种基于预训练视频扩散模型的视频超分辨率框架STCDiT，通过运动感知的分段重建和锚帧潜在表示引导来提升视频恢复的结构保真度和时间一致性。实验表明该方法在复杂相机运动下表现优异。
Reason 该论文研究的是视频超分辨率任务，属于视频理解与生成领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的锚帧引导和运动感知重建方法在视频领域具有创新性，但无法直接迁移用于解决文档图像中的OCR幻觉、密集文本识别、复杂排版或长文档推理等关键痛点。
Abstract We present STCDiT, a video super-resolution framework built upon a pre-trained video diffusion model, aiming to restore structurally faithful and temporally stable videos from degraded inputs, even under complex camera motions. The main challenges lie in maintaining temporal stability during reconstruction and preserving structural fidelity during generation. To address these challenges, we first develop a motion-aware VAE reconstruction method that performs segment-wise reconstruction, with each segment clip exhibiting uniform motion characteristic, thereby effectively handling videos with complex camera motions. Moreover, we observe that the first-frame latent extracted by the VAE encoder in each clip, termed the anchor-frame latent, remains unaffected by temporal compression and retains richer spatial structural information than subsequent frame latents. We further develop an anchor-frame guidance approach that leverages structural information from anchor frames to constrain the generation process and improve structural fidelity of video features. Coupling these two designs enables the video diffusion model to achieve high-quality video super-resolution. Extensive experiments show that STCDiT outperforms state-of-the-art methods in terms of structural fidelity and temporal consistency.
Authors Junyang Chen, Jiangxin Dong, Long Sun, Yixin Yang, Jinshan Pan
Comment Project page: https://jychen9811.github.io/STCDiT_page
Categories Computer Vision and Pattern Recognition
0
Sampling Control for Imbalanced Calibration in Semi-Supervised Learning AAAI 2026
Title CN 半监督学习中用于不平衡校准的采样控制
Keywords 半监督学习 · 类别不平衡 · 采样控制 · logits校准 · 特征不平衡
Summary 本文提出了一种名为SC-SSL的统一框架，通过解耦的采样控制来抑制模型偏差，解决半监督学习中因标注与未标注数据分布不匹配导致的分类偏置问题。方法包括训练阶段自适应调整采样概率以缓解特征层面的不平衡，以及推理阶段通过优化偏置向量进行后处理logits校准。实验表明其在多个基准上达到SOTA性能。
Reason 该论文研究的是半监督学习中的类别不平衡问题，属于通用分类任务的优化，与文档图像理解（DIU）或视觉-语言模型在文档场景下的应用无直接关联。未涉及OCR、布局分析、表格识别、多模态推理或视觉表示学习等DIU核心任务，也无法迁移解决OCR幻觉、复杂排版、细粒度定位等关键痛点。
Abstract Class imbalance remains a critical challenge in semi-supervised learning (SSL), especially when distributional mismatches between labeled and unlabeled data lead to biased classification. Although existing methods address this issue by adjusting logits based on the estimated class distribution of unlabeled data, they often handle model imbalance in a coarse-grained manner, conflating data imbalance with bias arising from varying class-specific learning difficulties. To address this issue, we propose a unified framework, SC-SSL, which suppresses model bias through decoupled sampling control. During training, we identify the key variables for sampling control under ideal conditions. By introducing a classifier with explicit expansion capability and adaptively adjusting sampling probabilities across different data distributions, SC-SSL mitigates feature-level imbalance for minority classes. In the inference phase, we further analyze the weight imbalance of the linear classifier and apply post-hoc sampling control with an optimization bias vector to directly calibrate the logits. Extensive experiments across various benchmark datasets and distribution settings validate the consistency and state-of-the-art performance of SC-SSL.
Authors Senmao Tian, Xiang Wei, Shunli Zhang
Comment Accepted at AAAI 2026
Categories Computer Vision and Pattern Recognition
0
SPAGS: Sparse-View Articulated Object Reconstruction from Single State via Planar Gaussian Splatting
Title CN SPAGS：基于平面高斯点阵的单状态稀疏视角可动部件物体重建
Keywords 3D重建 · 可动部件物体 · 高斯点阵 · 稀疏视角 · 平面高斯
Summary 本文提出了一种名为SPAGS的框架，用于从单状态下的稀疏视角RGB图像中重建可动部件物体的3D结构。通过引入高斯信息场选择最优视角，并将3D高斯压缩为平面高斯以优化法线和深度估计，结合深度平滑正则化和少样本扩散进行粗到精的优化。同时利用渲染图像的部件分割掩码反投影更新每个高斯基元的分割概率，实现高质量的部件级表面重建。实验表明该方法在合成与真实数据上均优于现有方法。
Reason 该论文研究的是3D视觉中的可动部件物体重建问题，属于3D重建与计算机图形学领域，核心方法为平面高斯点阵（Planar Gaussian Splatting）和稀疏视角优化。尽管涉及视觉表示学习，但其任务目标（3D形状重建）与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版、长文档推理等完全无关。无法迁移用于解决文档理解中的多模态对齐、细粒度文本识别或VLM推理增强等问题。
Abstract Articulated objects are ubiquitous in daily environments, and their 3D reconstruction holds great significance across various fields. However, existing articulated object reconstruction methods typically require costly inputs such as multi-stage and multi-view observations. To address the limitations, we propose a category-agnostic articulated object reconstruction framework via planar Gaussian Splatting, which only uses sparse-view RGB images from a single state. Specifically, we first introduce a Gaussian information field to perceive the optimal sparse viewpoints from candidate camera poses. Then we compress 3D Gaussians into planar Gaussians to facilitate accurate estimation of normal and depth. The planar Gaussians are optimized in a coarse-to-fine manner through depth smooth regularization and few-shot diffusion. Moreover, we introduce a part segmentation probability for each Gaussian primitive and update them by back-projecting part segmentation masks of renderings. Extensive experimental results demonstrate that our method achieves higher-fidelity part-level surface reconstruction on both synthetic and real-world data than existing methods. Codes will be made publicly available.
Authors Di Wu, Liu Liu, Xueyu Yuan, Qiaojun Yu, Wenxiao Chen, Ruilong Yan, Yiming Tang, Liangtu Song
Comment 10 pages, 7 figures
Categories Computer Vision and Pattern Recognition
0
REArtGS++: Generalizable Articulation Reconstruction with Temporal Geometry Constraint via Planar Gaussian Splatting
Title CN REArtGS++：通过平面高斯点阵化与时间几何约束实现可泛化的关节结构重建
Keywords 3D重建 · 关节物体 · 高斯点阵化 · 运动估计 · 几何约束
Summary 本文提出REArtGS++，一种基于平面高斯点阵化和时间几何约束的通用关节物体重建方法。通过解耦螺丝运动建模、部件感知的高斯优化以及在泰勒一阶展开下的法向与深度一致性正则化，提升了对未见状态下的关节物体部件表面重建和运动参数估计能力。实验表明其在合成与真实数据上均优于现有方法。
Reason 该论文研究的是关节物体（如抽屉、冰箱）的三维部件级表面重建与运动参数估计，属于3D视觉与具身智能领域，核心问题为动态物体建模与高斯点阵化优化。其方法聚焦于多视角RGB图像下的连续运动几何约束，并不涉及文档图像理解中的OCR、布局分析、表格识别或VLM多模态对齐等任务。无法迁移解决DIU领域的幻觉、密集文本、复杂排版或长文档推理等核心痛点。
Abstract Articulated objects are pervasive in daily environments, such as drawers and refrigerators. Towards their part-level surface reconstruction and joint parameter estimation, REArtGS introduces a category-agnostic approach using multi-view RGB images at two different states. However, we observe that REArtGS still struggles with screw-joint or multi-part objects and lacks geometric constraints for unseen states. In this paper, we propose REArtGS++, a novel method towards generalizable articulated object reconstruction with temporal geometry constraint and planar Gaussian splatting. We first model a decoupled screw motion for each joint without type prior, and jointly optimize part-aware Gaussians with joint parameters through part motion blending. To introduce time-continuous geometric constraint for articulated modeling, we encourage Gaussians to be planar and propose a temporally consistent regularization between planar normal and depth through Taylor first-order expansion. Extensive experiments on both synthetic and real-world articulated objects demonstrate our superiority in generalizable part-level surface reconstruction and joint parameter estimation, compared to existing approaches. Project Site: https://sites.google.com/view/reartgs2/home.
Authors Di Wu, Liu Liu, Anran Huang, Yuyan Liu, Qiaojun Yu, Shaofan Liu, Liangtu Song, Cewu Lu
Comment 10 pages, 7 figures
Categories Computer Vision and Pattern Recognition
0
NI-Tex: Non-isometric Image-based Garment Texture Generation
Title CN NI-Tex：基于非等距图像的服装纹理生成
Keywords 3D服装 · 纹理生成 · 非等距映射 · PBR材质 · 多视角烘焙
Summary 本文提出NI-Tex，旨在解决从非等距图像中为3D服装网格生成高质量PBR纹理的问题。作者构建了3D Garment Videos数据集，并采用Nano Banana进行跨拓扑结构的纹理编辑，结合不确定性引导的多视角烘焙策略，实现工业级纹理生成。方法聚焦于3D服装建模中的纹理映射，不涉及文本内容理解或多模态文档推理。
Reason 该论文研究的是3D服装纹理生成，属于3D视觉与计算机图形学领域，主要解决从野生图像中生成符合物理渲染的服装纹理问题。其核心技术涉及非等距图像编辑、多视角融合和3D garment视频数据集构建，与文档图像理解（DIU）的核心任务如文本识别、布局分析、表格理解等无直接关联。无法迁移用于解决OCR幻觉、密集文本处理或复杂排版理解等DIU痛点。
Abstract Existing industrial 3D garment meshes already cover most real-world clothing geometries, yet their texture diversity remains limited. To acquire more realistic textures, generative methods are often used to extract Physically-based Rendering (PBR) textures and materials from large collections of wild images and project them back onto garment meshes. However, most image-conditioned texture generation approaches require strict topological consistency between the input image and the input 3D mesh, or rely on accurate mesh deformation to match to the image poses, which significantly constrains the texture generation quality and flexibility. To address the challenging problem of non-isometric image-based garment texture generation, we construct 3D Garment Videos, a physically simulated, garment-centric dataset that provides consistent geometry and material supervision across diverse deformations, enabling robust cross-pose texture learning. We further employ Nano Banana for high-quality non-isometric image editing, achieving reliable cross-topology texture generation between non-isometric image-geometry pairs. Finally, we propose an iterative baking method via uncertainty-guided view selection and reweighting that fuses multi-view predictions into seamless, production-ready PBR textures. Through extensive experiments, we demonstrate that our feedforward dual-branch architecture generates versatile and spatially aligned PBR materials suitable for industry-level 3D garment design.
Authors Hui Shan, Ming Li, Haitao Yang, Kai Zheng, Sizhe Zheng, Yanwei Fu, Xiangru Huang
Categories Computer Vision and Pattern Recognition
0
VAOT: Vessel-Aware Optimal Transport for Retinal Fundus Enhancement
Title CN VAOT：用于视网膜眼底增强的血管感知最优传输
Keywords 医学图像增强 · 最优传输 · 血管结构保持 · 无配对学习 · GAN
Summary 本文提出了一种名为VAOT的视网膜眼底图像增强框架，通过结合最优传输目标与两种结构保持正则化（骨架损失和端点感知损失），在未配对数据下实现血管结构的保真增强。实验表明该方法在合成退化基准和下游分割任务中优于现有方法。
Reason 该论文研究的是医学图像处理中的视网膜眼底图像增强问题，属于纯医学影像分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心方法——基于最优传输和血管结构保持的正则化——虽然在医学图像增强中有价值，但无法迁移解决文档图像中的OCR幻觉、复杂排版、长文档推理等核心痛点。
Abstract Color fundus photography (CFP) is central to diagnosing and monitoring retinal disease, yet its acquisition variability (e.g., illumination changes) often degrades image quality, which motivates robust enhancement methods. Unpaired enhancement pipelines are typically GAN-based, however, they can distort clinically critical vasculature, altering vessel topology and endpoint integrity. Motivated by these structural alterations, we propose Vessel-Aware Optimal Transport (\textbf{VAOT}), a framework that combines an optimal-transport objective with two structure-preserving regularizers: (i) a skeleton-based loss to maintain global vascular connectivity and (ii) an endpoint-aware loss to stabilize local termini. These constraints guide learning in the unpaired setting, reducing noise while preserving vessel structure. Experimental results on synthetic degradation benchmark and downstream evaluations in vessel and lesion segmentation demonstrate the superiority of the proposed methods against several state-of-the art baselines. The code is available at https://github.com/Retinal-Research/VAOT
Authors Xuanzhao Dong, Wenhui Zhu, Yujian Xiong, Xiwen Chen, Hao Wang, Xin Li, Jiajun Cheng, Zhipeng Wang, Shao Tang, Oana Dumitrascu, Yalin Wang
Categories Computer Vision and Pattern Recognition
0
QGait: Toward Accurate Quantization for Gait Recognition IJCB 2025
Title CN QGait：面向步态识别的精确量化
Keywords 步态识别 · 模型量化 · 软量化器 · 特征校准 · 二值化输入
Summary 本文提出了一种可微的软量化器（soft quantizer）和两阶段训练策略，用于提升步态识别模型在二值化输入下的性能。同时引入了类间距离引导校准（IDC）策略，以保持全精度网络与量化网络之间的特征分布一致性。实验表明该方法在多个数据集上达到SOTA效果。
Reason 该论文研究的是步态识别中的模型量化问题，属于生物特征识别领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其方法聚焦于二值化输入下的量化误差优化和特征分布校准，无法迁移解决OCR幻觉、复杂排版理解、细粒度定位等DIU核心痛点。
Abstract Existing deep learning methods have made significant progress in gait recognition. Quantization can facilitate the application of gait models as a model-agnostic general compression technique. Typically, appearance-based models binarize inputs into silhouette sequences. However, mainstream quantization methods prioritize minimizing task loss over quantization error, which is detrimental to gait recognition with binarized inputs. To address this, we propose a differentiable soft quantizer, which better simulates the gradient of the round function during backpropagation. This enables the network to learn from subtle input perturbations. However, our theoretical analysis and empirical studies reveal that directly applying the soft quantizer can hinder network convergence. We addressed this issue by adopting a two-stage training strategy, introducing a soft quantizer during the fine-tuning phase. However, in the first stage of training, we observed a significant change in the output distribution of different samples in the feature space compared to the full-precision network. It is this change that led to a loss in performance. Based on this, we propose an Inter-class Distance-guided Calibration (IDC) strategy to preserve the relative distance between the embeddings of samples with different labels. Extensive experiments validate the effectiveness of our approach, demonstrating state-of-the-art accuracy across various settings and datasets.
Authors Senmao Tian, Haoyu Gao, Gangyi Hong, Shuyun Wang, JingJie Wang, Xin Yu, Shunli Zhang
Comment Accepted as an oral presentation at IJCB 2025
Categories Computer Vision and Pattern Recognition
0
From Features to Reference Points: Lightweight and Adaptive Fusion for Cooperative Autonomous Driving
Title CN 从特征到参考点：面向协同自动驾驶的轻量级自适应融合
Keywords 协同感知 · 参考点融合 · 通信效率 · 自动驾驶 · 轻量化
Summary 本文提出RefPtsFusion，一种用于协同自动驾驶的轻量级融合框架，通过交换对象的位置、速度和尺寸等参考点而非原始特征图来大幅降低通信开销。该方法在M3CAD数据集上实现了通信成本五个数量级的下降，同时保持良好的感知性能，并具备跨异构模型的兼容性。
Reason 该论文属于具身智能与自动驾驶领域，研究的是车辆间协同感知中的通信压缩问题，虽然其参考点机制具有一定的方法创新性，但应用场景与文档图像理解（DIU）无直接关联。其核心技术（如Top-K查询融合、参考点交换）难以迁移到解决OCR幻觉、复杂排版解析或长文档推理等DIU核心痛点，属于无关领域工作。
Abstract We present RefPtsFusion, a lightweight and interpretable framework for cooperative autonomous driving. Instead of sharing large feature maps or query embeddings, vehicles exchange compact reference points, e.g., objects' positions, velocities, and size information. This approach shifts the focus from "what is seen" to "where to see", creating a sensor- and model-independent interface that works well across vehicles with heterogeneous perception models while greatly reducing communication bandwidth. To enhance the richness of shared information, we further develop a selective Top-K query fusion that selectively adds high-confidence queries from the sender. It thus achieves a strong balance between accuracy and communication cost. Experiments on the M3CAD dataset show that RefPtsFusion maintains stable perception performance while reducing communication overhead by five orders of magnitude, dropping from hundreds of MB/s to only a few KB/s at 5 FPS (frame per second), compared to traditional feature-level fusion methods. Extensive experiments also demonstrate RefPtsFusion's strong robustness and consistent transmission behavior, highlighting its potential for scalable, real-time cooperative driving systems.
Authors Yongqi Zhu, Morui Zhu, Qi Chen, Deyuan Qu, Song Fu, Qing Yang
Comment 10 pages, 4 figures
Categories Computer Vision and Pattern Recognition
0
Physics-Based Decomposition of Reflectance and Shading using a Single Visible-Thermal Image Pair
Title CN 基于单对可见光-热成像图像的反射与阴影物理分解
Keywords 图像分解 · 反射率估计 · 阴影分离 · 热成像 · 自监督学习
Summary 本文提出一种基于物理原理的方法，利用一对可见光和热成像图像实现图像的内在分解，分离表面反射率和光照阴影。通过热成像捕捉被表面吸收的光能，建立可见光与热图像强度之间的序关系，用于自监督训练神经网络恢复反射和阴影。实验表明该方法在定量和定性上均优于现有方法。
Reason 该论文研究的是从可见光与热成像图像中进行反射率和阴影的物理分解，属于通用计算机视觉中的图像分解任务，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法聚焦于物理成像模型与自监督学习，无法迁移解决文档理解中的OCR幻觉、复杂排版、细粒度定位或长文档推理等核心痛点。
Abstract Decomposing an image into its underlying photometric factors--surface reflectance and shading--is a long-standing challenge due to the lack of extensive ground-truth data for real-world scenes. We introduce a novel physics-based approach for intrinsic image decomposition using a pair of visible and thermal images. We leverage the principle that light not reflected from an opaque surface is absorbed and detected as heat by a thermal camera. This allows us to relate the ordinalities (or relative magnitudes) between visible and thermal image intensities to the ordinalities of shading and reflectance, which enables a dense self-supervision of an optimizing neural network to recover shading and reflectance. We perform quantitative evaluations with known reflectance and shading under natural and artificial lighting, and qualitative experiments across diverse scenes. The results demonstrate superior performance over both physics-based and recent learning-based methods, providing a path toward scalable real-world data curation with supervision.
Authors Zeqing Leo Yuan, Mani Ramanagopal, Aswin C. Sankaranarayanan, Srinivasa G. Narasimhan
Categories Computer Vision and Pattern Recognition
0
PanoDiffusion: 360-degree Panorama Outpainting via Diffusion
Title CN PanoDiffusion：通过扩散模型实现360度全景外绘
Keywords 全景生成 · 扩散模型 · 外绘 · RGB-D · 计算机视觉
Summary 本文提出PanoDiffusion，一种基于潜在扩散模型的360度全景外绘方法，利用RGB和深度数据进行训练，并在去噪过程中引入渐进式相机旋转以提升全景一致性。实验表明该方法在多种遮罩类型下均能生成结构合理、多样化的高质量全景图。
Reason 该论文研究的是360度室内全景图的生成与补全，属于纯图像生成领域，且聚焦于全景拼接与扩散模型的应用。其技术路线与文档图像理解（DIU）的核心痛点（如OCR幻觉、文本识别、布局分析、表格理解等）无直接关联。尽管使用了多模态（RGB-D）和扩散模型，但方法目标是视觉场景补全，无法迁移到文档内容的理解或推理任务中。
Abstract Generating complete 360-degree panoramas from narrow field of view images is ongoing research as omnidirectional RGB data is not readily available. Existing GAN-based approaches face some barriers to achieving higher quality output, and have poor generalization performance over different mask types. In this paper, we present our 360-degree indoor RGB-D panorama outpainting model using latent diffusion models (LDM), called PanoDiffusion. We introduce a new bi-modal latent diffusion structure that utilizes both RGB and depth panoramic data during training, which works surprisingly well to outpaint depth-free RGB images during inference. We further propose a novel technique of introducing progressive camera rotations during each diffusion denoising step, which leads to substantial improvement in achieving panorama wraparound consistency. Results show that our PanoDiffusion not only significantly outperforms state-of-the-art methods on RGB-D panorama outpainting by producing diverse well-structured results for different types of masks, but can also synthesize high-quality depth panoramas to provide realistic 3D indoor models.
Authors Tianhao Wu, Chuanxia Zheng, Tat-Jen Cham
Comment Project Page: https://sm0kywu.github.io/panodiffusion/
Categories Computer Vision and Pattern Recognition
0
GuideFlow: Constraint-Guided Flow Matching for Planning in End-to-End Autonomous Driving
Title CN GuideFlow：用于端到端自动驾驶规划的约束引导流匹配
Keywords 自动驾驶 · 流匹配 · 生成模型 · 约束优化 · 轨迹规划
Summary 本文提出GuideFlow，一种基于约束引导流匹配的自动驾驶规划框架，通过将能量模型与流匹配结合，在生成过程中显式施加物理和安全约束，缓解模式崩溃问题，并支持驾驶风格调控。在多个主流驾驶数据集上验证了有效性，在NavSim硬测试集上达到SOTA性能。
Reason 该论文研究的是自动驾驶中的轨迹规划问题，属于具身智能与机器人控制领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管其方法涉及生成建模与约束引入机制，但核心任务场景（驾驶行为预测、物理安全约束优化）难以迁移到文档理解中的OCR幻觉、复杂排版解析或细粒度文本定位等关键痛点。
Abstract Driving planning is a critical component of end-to-end (E2E) autonomous driving. However, prevailing Imitative E2E Planners often suffer from multimodal trajectory mode collapse, failing to produce diverse trajectory proposals. Meanwhile, Generative E2E Planners struggle to incorporate crucial safety and physical constraints directly into the generative process, necessitating an additional optimization stage to refine their outputs. In this paper, we propose \textit{\textbf{GuideFlow}}, a novel planning framework that leverages Constrained Flow Matching. Concretely, \textit{\textbf{GuideFlow}} explicitly models the flow matching process, which inherently mitigates mode collapse and allows for flexible guidance from various conditioning signals. Our core contribution lies in directly enforcing explicit constraints within the flow matching generation process, rather than relying on implicit constraint encoding. Crucially, \textit{\textbf{GuideFlow}} unifies the training of the flow matching with the Energy-Based Model (EBM) to enhance the model's autonomous optimization capability to robustly satisfy physical constraints. Secondly, \textit{\textbf{GuideFlow}} parameterizes driving aggressiveness as a control signal during generation, enabling precise manipulation of trajectory style. Extensive evaluations on major driving benchmarks (Bench2Drive, NuScenes, NavSim and ADV-NuScenes) validate the effectiveness of \textit{\textbf{GuideFlow}}. Notably, on the NavSim test hard split (Navhard), \textit{\textbf{GuideFlow}} achieved SOTA with an EPDMS score of 43.0. The code will be released.
Authors Lin Liu, Caiyan Jia, Guanyi Yu, Ziying Song, JunQiao Li, Feiyang Jia, Peiliang Wu, Xiaoshuai Hao, Yandan Luo
Categories Computer Vision and Pattern Recognition
0
RichControl: Structure- and Appearance-Rich Training-Free Spatial Control for Text-to-Image Generation
Title CN RichControl：用于文本到图像生成的免训练、结构与外观丰富的空间控制
Keywords 文本到图像生成 · 空间控制 · 特征注入 · 免训练方法 · 扩散模型
Summary 本文提出RichControl，一种免训练的空间控制框架，用于提升文本到图像生成中对结构条件（如边缘图）的利用效果。通过解耦条件特征采样与去噪过程，设计了高效的单步采样策略和重启优化机制，并结合外观丰富提示策略，实现了结构与视觉质量兼备的生成结果。实验表明其在多种零样本条件下达到SOTA。
Reason 该论文研究的是文本到图像生成中的空间控制问题，属于纯图像生成领域，其核心任务是基于边缘图等条件控制生成图像的结构布局。尽管涉及多模态输入和特征注入机制，但目标场景与文档图像理解（DIU）无直接关联，且方法聚焦于生成质量优化，无法迁移解决OCR幻觉、复杂排版解析或长文档推理等DIU核心痛点。
Abstract Text-to-image (T2I) diffusion models have shown remarkable success in generating high-quality images from text prompts. Recent efforts extend these models to incorporate conditional images (e.g., canny edge) for fine-grained spatial control. Among them, feature injection methods have emerged as a training-free alternative to traditional fine-tuning-based approaches. However, they often suffer from structural misalignment, condition leakage, and visual artifacts, especially when the condition image diverges significantly from natural RGB distributions. Through an empirical analysis of existing methods, we identify a key limitation: the sampling schedule of condition features, previously unexplored, fails to account for the evolving interplay between structure preservation and domain alignment throughout diffusion steps. Inspired by this observation, we propose a flexible training-free framework that decouples the sampling schedule of condition features from the denoising process, and systematically investigate the spectrum of feature injection schedules for a higher-quality structure guidance in the feature space. Specifically, we find that condition features sampled from a single timestep are sufficient, yielding a simple yet efficient schedule that balances structure alignment and appearance quality. We further enhance the sampling process by introducing a restart refinement schedule, and improve the visual quality with an appearance-rich prompting strategy. Together, these designs enable training-free generation that is both structure-rich and appearance-rich. Extensive experiments show that our approach achieves state-of-the-art results across diverse zero-shot conditioning scenarios.
Authors Liheng Zhang, Lexi Pang, Hang Ye, Xiaoxuan Ma, Yizhou Wang
Categories Computer Vision and Pattern Recognition
0
AVATAR: Reinforcement Learning to See, Hear, and Reason Over Video
Title CN AVATAR：通过强化学习实现视频中的视觉、听觉与推理
Keywords 视频理解 · 多模态推理 · 强化学习 · 离策略训练 · 时序信用分配
Summary 本文提出AVATAR框架，用于解决长视频中多模态（音视频）推理的挑战。它采用离策略强化学习架构提升样本效率，并引入时间优势塑造（TAS）机制以突出关键推理步骤的信用分配。在多个视频理解基准上显著超越基线模型，同时实现5倍的样本效率提升。
Reason 该论文研究的是长时程视频中的多模态推理问题，属于视频理解领域，且方法围绕音频-视频模态的强化学习展开。尽管其提出的Temporal Advantage Shaping和离策略训练可能在理论上对长序列决策有启发，但其核心任务（如MMVU、Video-Holmes）与文档图像理解（DIU）的典型痛点（如OCR幻觉、密集文本识别、版面分析）无直接关联，迁移路径不明确。属于无关领域。
Abstract Multimodal reasoning over long-horizon video is challenging due to the need for precise spatiotemporal fusion and alignment across modalities. While recent methods such as Group Relative Policy Optimization (GRPO) have shown promise in this domain, they suffer from three key limitations: (1) data inefficiency from their on-policy design, (2) a vanishing advantage problem, where identical or near-identical rewards within a group eliminate the learning signal by producing zero-valued advantages, and (3) uniform credit assignment that fails to emphasize critical reasoning steps. We introduce $\textbf{AVATAR}$ ($\textbf{A}$udio-$\textbf{V}$ideo $\textbf{A}$gen$\textbf{t}$ for $\textbf{A}$lignment and $\textbf{R}$easoning), a framework that addresses these limitations through two core components: (1) an off-policy training architecture that improves sample efficiency and resolves vanishing advantages by reusing past experiences with greater reward diversity, and (2) Temporal Advantage Shaping (TAS), a novel credit assignment strategy that upweights key reasoning phases during learning. $\textbf{AVATAR}$ achieves strong performance across various benchmarks, outperforming the Qwen2.5-Omni baseline by $\mathbf{+5.4}$ on MMVU, $\mathbf{+4.9}$ on OmniBench, and $\mathbf{+4.5}$ on Video-Holmes, while demonstrating $\textbf{$5$$\times$ sample efficiency}$, requiring $80\%$ fewer generated completions to reach target performance.
Authors Yogesh Kulkarni, Pooyan Fazli
Categories Computer Vision and Pattern Recognition
0
Neural B-Frame Coding: Tackling Domain Shift Issues with Lightweight Online Motion Resolution Adaptation TCAS-II: Express Briefs
Title CN 神经B帧编码：通过轻量级在线运动分辨率自适应解决域偏移问题
Keywords 视频压缩 · B帧编码 · 域偏移 · 运动估计 · 轻量级分类器
Summary 本文提出了一种轻量级分类器来预测视频B帧编码中的下采样因子，以缓解训练和测试时GOP大小不一致导致的域偏移问题。通过使用当前帧和参考帧的状态信号，三种分类器变体（Bi-Class、Mu-Class、Co-Class）能够在不重新训练编解码器的情况下，有效平衡率失真性能与计算开销，实验表明其性能接近 exhaustive search 方法但复杂度显著降低。
Reason 该论文属于视频压缩领域，研究的是B帧编码中的运动估计与域偏移问题，核心方法为视频编解码优化，与文档图像理解（DIU）或视觉-语言模型（VLM）无直接关联。其技术路线聚焦于视频压缩效率提升，无法迁移至OCR幻觉、复杂排版理解、长文档推理等DIU核心痛点。
Abstract Learned B-frame codecs with hierarchical temporal prediction often encounter the domain-shift issue due to mismatches between the Group-of-Pictures (GOP) sizes for training and testing, leading to inaccurate motion estimates, particularly for large motion. A common solution is to turn large motion into small motion by downsampling video frames during motion estimation. However, determining the optimal downsampling factor typically requires costly rate-distortion optimization. This work introduces lightweight classifiers to predict downsampling factors. These classifiers leverage simple state signals from current and reference frames to balance rate-distortion performance with computational cost. Three variants are proposed: (1) a binary classifier (Bi-Class) trained with Focal Loss to choose between high and low resolutions, (2) a multi-class classifier (Mu-Class) trained with novel soft labels based on rate-distortion costs, and (3) a co-class approach (Co-Class) that combines the predictive capability of the multi-class classifier with the selective search of the binary classifier. All classifier methods can work seamlessly with existing B-frame codecs without requiring codec retraining. Experimental results show that they achieve coding performance comparable to exhaustive search methods while significantly reducing computational complexity. The code is available at: https://github.com/NYCU-MAPL/Fast-OMRA.git.
Authors Sang NguyenQuang, Xiem HoangVan, Wen-Hsiao Peng
Comment Accepted by TCAS-II: Express Briefs
Categories Computer Vision and Pattern Recognition
0
Learning to See and Act: Task-Aware Virtual View Exploration for Robotic Manipulation
Title CN 学会观察与行动：面向机器人操作的任务感知虚拟视图探索
Keywords 机器人操作 · 视觉语言动作模型 · 虚拟视图探索 · Mixture-of-Experts · 任务感知表示
Summary 本文提出TVVE框架，通过任务感知的虚拟视图探索和TaskMoE视觉编码器提升机器人操作中的3D感知与任务泛化能力，并在RLBench-OG等基准上验证了鲁棒性与泛化性。
Reason 该论文属于具身智能与机器人领域，研究任务为机器人操控中的视觉-语言-动作建模，涉及虚拟视图探索和任务特定表征学习。尽管其多视角感知和MoE架构具有一定方法价值，但整体工作与文档图像理解（DIU）的核心问题如OCR、排版分析、文本识别、VQA等无直接关联，且迁移到文档场景（如高分辨率处理或去幻觉）的路径不明确。属于负面清单中的‘无关领域’（机器人/具身智能），无法作为DIU的方法论源头。
Abstract Recent vision-language-action (VLA) models for multi-task robotic manipulation commonly rely on static viewpoints and shared visual encoders, which limit 3D perception and cause task interference, hindering robustness and generalization. In this work, we propose Task-aware Virtual View Exploration (TVVE), a framework designed to overcome these challenges by integrating virtual view exploration with task-specific representation learning. TVVE employs an efficient exploration policy, accelerated by a novel pseudo-environment, to acquire informative views. Furthermore, we introduce a Task-aware Mixture-of-Experts (TaskMoE) visual encoder to disentangle features across different tasks, boosting both representation fidelity and task generalization. By learning to see the world in a task-aware way, TVVE generates more complete and discriminative visual representations, demonstrating significantly enhanced action prediction across a wide array of manipulation challenges. To further validate the robustness and generalization capability of TVVE under out-of-distribution (OOD) settings, we construct a challenging benchmark, RLBench-OG, covering various visual perturbations and camera pose variations. Extensive experiments on RLBench and RLBench-OG show that our TVVE achieves superior performance over state-of-the-art approaches. In real-robot experiments, TVVE demonstrates exceptional performance and generalizes robustly in multiple OOD settings, including visual disturbances and unseen instructions. Visual results and code are provided at: https://hcplab-sysu.github.io/TAVP.
Authors Yongjie Bai, Zhouxia Wang, Yang Liu, Kaijun Luo, Yifan Wen, Mingtong Dai, Weixing Chen, Ziliang Chen, Lingbo Liu, Guanbin Li, Liang Lin
Comment 24 pages, 15 figures, project page: https://hcplab-sysu.github.io/TAVP
Categories Computer Vision and Pattern Recognition
0
ERANet: Edge Replacement Augmentation for Semi-Supervised Meniscus Segmentation with Prototype Consistency Alignment and Conditional Self-Training
Title CN ERANet：用于半监督半月板分割的边缘替换增强，结合原型一致性对齐和条件自训练
Keywords 医学图像分割 · 半监督学习 · 数据增强 · MRI · 原型对齐
Summary 本文提出ERANet，一种用于半监督半月板分割的框架，通过边缘替换增强（ERA）、原型一致性对齐（PCA）和条件自训练（CST）来提升在少量标注数据下的分割性能。实验表明该方法在3D MRI序列上优于现有方法。然而，该工作专注于医学图像分割，不属于文档理解范畴。
Reason 该论文研究的是医学图像中的半月板分割问题，属于医疗影像分析领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管其方法涉及半监督学习、数据增强和特征对齐等技术，但这些技术的应用场景为3D MRI序列中的器官分割，迁移到文档理解任务（如DocVQA、表格识别、OCR等）的路径极不明确，且未涉及多模态语言模型、文本布局理解或视觉-语言对齐等DIU核心问题。
Abstract Manual segmentation is labor-intensive, and automatic segmentation remains challenging due to the inherent variability in meniscal morphology, partial volume effects, and low contrast between the meniscus and surrounding tissues. To address these challenges, we propose ERANet, an innovative semi-supervised framework for meniscus segmentation that effectively leverages both labeled and unlabeled images through advanced augmentation and learning strategies. ERANet integrates three key components: edge replacement augmentation (ERA), prototype consistency alignment (PCA), and a conditional self-training (CST) strategy within a mean teacher architecture. ERA introduces anatomically relevant perturbations by simulating meniscal variations, ensuring that augmentations align with the structural context. PCA enhances segmentation performance by aligning intra-class features and promoting compact, discriminative feature representations, particularly in scenarios with limited labeled data. CST improves segmentation robustness by iteratively refining pseudo-labels and mitigating the impact of label noise during training. Together, these innovations establish ERANet as a robust and scalable solution for meniscus segmentation, effectively addressing key barriers to practical implementation. We validated ERANet comprehensively on 3D Double Echo Steady State (DESS) and 3D Fast/Turbo Spin Echo (FSE/TSE) MRI sequences. The results demonstrate the superior performance of ERANet compared to state-of-the-art methods. The proposed framework achieves reliable and accurate segmentation of meniscus structures, even when trained on minimal labeled data. Extensive ablation studies further highlight the synergistic contributions of ERA, PCA, and CST, solidifying ERANet as a transformative solution for semi-supervised meniscus segmentation in medical imaging.
Authors Siyue Li, Yongcheng Yao, Junru Zhong, Shutian Zhao, Fan Xiao, Tim-Yun Michael Ong, Ki-Wai Kevin Ho, James F. Griffith, Yudong Zhang, Shuihua Wang, Jin Hong, Weitian Chen
Categories Computer Vision and Pattern Recognition
0
GRIT-LP: Graph Transformer with Long-Range Skip Connection and Partitioned Spatial Graphs for Accurate Ice Layer Thickness Prediction
Title CN GRIT-LP：具有长距离跳跃连接和分区空间图的图变换器用于精确冰层厚度预测
Keywords Graph Transformer · Spatio-Temporal Modeling · Ice Layer Thickness · Remote Sensing · Oversmoothing
Summary 本文提出GRIT-LP，一种用于从极地雷达图像中估计冰层厚度的图变换器模型。该方法通过分区空间图构建策略和长距离跳跃连接机制，增强局部结构保持并缓解深层网络中的过平滑问题，在冰层厚度预测任务上显著优于现有方法。
Reason 该论文研究的是极地雷达图像中的冰层厚度估计，属于遥感与地球科学领域，任务本质是基于图神经网络的回归预测，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法聚焦于建模冰层的时空模式，虽涉及视觉输入，但不解决OCR幻觉、文本识别、复杂版面分析或文档推理等DIU核心问题，且无法迁移至文档理解场景。
Abstract Graph transformers have demonstrated remarkable capability on complex spatio-temporal tasks, yet their depth is often limited by oversmoothing and weak long-range dependency modeling. To address these challenges, we introduce GRIT-LP, a graph transformer explicitly designed for polar ice-layer thickness estimation from polar radar imagery. Accurately estimating ice layer thickness is critical for understanding snow accumulation, reconstructing past climate patterns and reducing uncertainties in projections of future ice sheet evolution and sea level rise. GRIT-LP combines an inductive geometric graph learning framework with self-attention mechanism, and introduces two major innovations that jointly address challenges in modeling the spatio-temporal patterns of ice layers: a partitioned spatial graph construction strategy that forms overlapping, fully connected local neighborhoods to preserve spatial coherence and suppress noise from irrelevant long-range links, and a long-range skip connection mechanism within the transformer that improves information flow and mitigates oversmoothing in deeper attention layers. We conducted extensive experiments, demonstrating that GRIT-LP outperforms current state-of-the-art methods with a 24.92\% improvement in root mean squared error. These results highlight the effectiveness of graph transformers in modeling spatiotemporal patterns by capturing both localized structural features and long-range dependencies across internal ice layers, and demonstrate their potential to advance data-driven understanding of cryospheric processes.
Authors Zesheng Liu, Maryam Rahnemoonfar
Categories Computer Vision and Pattern Recognition
0
DriveFlow: Rectified Flow Adaptation for Robust 3D Object Detection in Autonomous Driving AAAI 2026
Title CN DriveFlow：用于自动驾驶中鲁棒3D目标检测的校正流自适应
Keywords 3D目标检测 · 自动驾驶 · 校正流 · 图像编辑 · 分布外泛化
Summary 本文提出DriveFlow，一种基于预训练文本到图像流模型的无需训练的数据增强方法，用于提升自动驾驶场景下视觉中心3D目标检测在分布外（OOD）情况下的鲁棒性。通过频域分解，引入高频前景保持和双频背景优化策略，在不修改预训练模型的前提下实现更精确的3D物体几何保留和背景编辑。实验表明该方法在多种OOD场景中有效提升了检测性能。
Reason 该论文研究的是自动驾驶中的3D目标检测，属于具身智能与自动驾驶领域，且核心方法聚焦于基于校正流的图像编辑以增强训练数据。其技术路线与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、细粒度文本识别、长文档推理等）无直接关联，也无法迁移为解决DIU问题的有效工具。
Abstract In autonomous driving, vision-centric 3D object detection recognizes and localizes 3D objects from RGB images. However, due to high annotation costs and diverse outdoor scenes, training data often fails to cover all possible test scenarios, known as the out-of-distribution (OOD) issue. Training-free image editing offers a promising solution for improving model robustness by training data enhancement without any modifications to pre-trained diffusion models. Nevertheless, inversion-based methods often suffer from limited effectiveness and inherent inaccuracies, while recent rectified-flow-based approaches struggle to preserve objects with accurate 3D geometry. In this paper, we propose DriveFlow, a Rectified Flow Adaptation method for training data enhancement in autonomous driving based on pre-trained Text-to-Image flow models. Based on frequency decomposition, DriveFlow introduces two strategies to adapt noise-free editing paths derived from text-conditioned velocities. 1) High-Frequency Foreground Preservation: DriveFlow incorporates a high-frequency alignment loss for foreground to maintain precise 3D object geometry. 2) Dual-Frequency Background Optimization: DriveFlow also conducts dual-frequency optimization for background, balancing editing flexibility and semantic consistency. Comprehensive experiments validate the effectiveness and efficiency of DriveFlow, demonstrating comprehensive performance improvements on all categories across OOD scenarios. Code is available at https://github.com/Hongbin98/DriveFlow.
Authors Hongbin Lin, Yiming Yang, Chaoda Zheng, Yifan Zhang, Shuaicheng Niu, Zilu Guo, Yafeng Li, Gui Gui, Shuguang Cui, Zhen Li
Comment Accepted by AAAI 2026
Categories Computer Vision and Pattern Recognition
0
PointAD+: Learning Hierarchical Representations for Zero-shot 3D Anomaly Detection Submitted to TPAMI
Title CN PointAD+：用于零样本3D异常检测的分层表示学习
Keywords 3D异常检测 · 零样本学习 · 点云 · CLIP · 分层表示学习
Summary 本文提出PointAD+，一种用于零样本3D异常检测的统一框架，通过结合点级和像素级信息实现对未见类别物体的异常检测与分割。方法包含隐式3D表示（基于渲染像素）和显式3D表示（基于点云空间关系），并引入几何聚合（G-aggregation）和分层提示（渲染层与几何层）进行跨层次对比对齐，以联合建模渲染与空间异常性。测试时可即插即用地融合RGB信息提升性能。
Reason 该论文研究的是3D点云场景下的异常检测问题，属于3D视觉与工业检测领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。其方法聚焦于利用CLIP的2D泛化能力迁移至3D空间中的渲染像素和几何结构异常检测，不涉及文本识别、布局分析、表格理解或多模态文档推理等DIU核心问题。尽管提出了分层表示与跨层次对比学习机制，但这些技术难以迁移到解决OCR幻觉、密集文本处理或复杂排版理解等文档特有挑战。
Abstract In this paper, we aim to transfer CLIP's robust 2D generalization capabilities to identify 3D anomalies across unseen objects of highly diverse class semantics. To this end, we propose a unified framework to comprehensively detect and segment 3D anomalies by leveraging both point- and pixel-level information. We first design PointAD, which leverages point-pixel correspondence to represent 3D anomalies through their associated rendering pixel representations. This approach is referred to as implicit 3D representation, as it focuses solely on rendering pixel anomalies but neglects the inherent spatial relationships within point clouds. Then, we propose PointAD+ to further broaden the interpretation of 3D anomalies by introducing explicit 3D representation, emphasizing spatial abnormality to uncover abnormal spatial relationships. Hence, we propose G-aggregation to involve geometry information to enable the aggregated point representations spatially aware. To simultaneously capture rendering and spatial abnormality, PointAD+ proposes hierarchical representation learning, incorporating implicit and explicit anomaly semantics into hierarchical text prompts: rendering prompts for the rendering layer and geometry prompts for the geometry layer. A cross-hierarchy contrastive alignment is further introduced to promote the interaction between the rendering and geometry layers, facilitating mutual anomaly learning. Finally, PointAD+ integrates anomaly semantics from both layers to capture the generalized anomaly semantics. During the test, PointAD+ can integrate RGB information in a plug-and-play manner and further improve its detection performance. Extensive experiments demonstrate the superiority of PointAD+ in ZS 3D anomaly detection across unseen objects with highly diverse class semantics, achieving a holistic understanding of abnormality.
Authors Qihang Zhou, Shibo He, Jiangtao Yan, Wenchao Meng, Jiming Chen
Comment Submitted to TPAMI
Categories Computer Vision and Pattern Recognition
0
ControlThinker: Unveiling Latent Semantics for Controllable Image Generation through Visual Reasoning
Title CN ControlThinker：通过视觉推理揭示可控图像生成中的潜在语义
Keywords 可控图像生成 · 视觉推理 · 多模态大模型 · 语义补全 · 推理路径探索
Summary 本文提出ControlThinker，一种基于‘先理解后生成’范式的可控图像生成框架。通过激发多模态大语言模型（MLLM）的视觉推理能力，从控制信号图像中挖掘潜在语义以增强原始文本提示，并利用基于度量的输出奖励模型（ORM）筛选最优推理路径，从而缩小文本与生成图像之间的语义差距。实验表明该方法在多个基准上提升了生成图像的质量和语义一致性。
Reason 该论文属于纯图像生成领域，研究目标是提升可控图像生成的语义一致性，其核心方法围绕文本-图像生成过程中的语义补全与推理路径选择。尽管使用了MLLM进行视觉推理，但其应用场景与技术目标均与文档图像理解（DIU）无关。文档理解关注的是从复杂排版中准确提取和推理信息，而非生成图像。该工作无法迁移用于解决OCR幻觉、密集文本识别或长文档推理等DIU核心痛点，属于负面清单中的‘无关领域’。
Abstract The field of controllable image generation has seen significant advancements, with various architectures improving generation layout consistency with control signals. However, contemporary methods still face challenges in bridging the semantic gap between input text prompts with sparse semantics and the target images, often over-relying on low-level control signals to infer regional details. To address this challenge, we propose ControlThinker, a novel framework that employs a "comprehend-then-generate" paradigm. Firstly, by incentivizing the visual reasoning capability of a MLLM, latent semantics from control images are mined to enrich text prompts. This enriched semantic understanding then seamlessly aids in image generation without the need for additional complex modifications. To further tackle the uncertainty arising from the ambiguity of control images, we encourage broader exploration of reasoning trajectories and select the optimal one using a metric-based output reward model (ORM). Extensive experimental results demonstrate that ControlThinker effectively mitigates the semantic gap between raw text prompts and target images, resulting in improved visual quality and semantic consistency across a wide range of benchmarks. The code and models are available at https://github.com/Maplebb/ControlThinker.
Authors Feng Han, Yang Jiao, Shaoxiang Chen, Junhao Xu, Jingjing Chen, Yu-Gang Jiang
Categories Computer Vision and Pattern Recognition
0
ImmerIris: A Large-Scale Dataset and Benchmark for Off-Axis and Unconstrained Iris Recognition in Immersive Applications
Title CN ImmerIris：面向沉浸式应用的偏轴与无约束虹膜识别的大规模数据集与基准
Keywords 虹膜识别 · 沉浸式应用 · 偏轴成像 · 归一化自由 · 生物特征识别
Summary 本文提出了一个名为ImmerIris的大规模虹膜数据集，专用于沉浸式应用中的偏轴和无约束虹膜识别。该数据集包含来自564名受试者的近50万张眼部图像，是目前最大的公开虹膜数据集之一。同时，作者提出了一种无需预处理归一化的识别范式，直接在原始图像上进行学习，在挑战性条件下表现优于传统方法。
Reason 该论文聚焦于虹膜识别，属于生物特征识别领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无关。尽管其提出了‘归一化自由’的学习范式，但应用场景为沉浸式设备下的虹膜图像识别，无法迁移到文档理解中的OCR幻觉、复杂版式解析或长文档推理等核心痛点。
Abstract Recently, iris recognition is regaining prominence in immersive applications such as extended reality as a means of seamless user identification. This application scenario introduces unique challenges compared to traditional iris recognition under controlled setups, as the ocular images are primarily captured off-axis and less constrained, causing perspective distortion, intra-subject variation, and quality degradation in iris textures. Datasets capturing these challenges remain limited. This paper fills this gap by presenting a large-scale iris dataset collected via head-mounted displays, termed ImmerIris. It contains 499,791 ocular images from 564 subjects, and is, to our knowledge, the largest public iris dataset to date and among the first dedicated to immersive applications. It is accompanied by a comprehensive set of evaluation protocols that benchmark recognition systems under various challenging conditions. This paper also draws attention to a shared obstacle of current recognition methods, the reliance on a pre-processing, normalization stage, which is fallible in off-axis and unconstrained setups. To this end, this paper further proposes a normalization-free paradigm that directly learns from minimally adjusted ocular images. Despite its simplicity, it outperforms normalization-based prior arts, indicating a promising direction for robust iris recognition.
Authors Yuxi Mi, Qiuyang Yuan, Zhizhou Zhong, Xuan Zhao, Jiaogen Zhou, Fubao Zhu, Jihong Guan, Shuigeng Zhou
Categories Computer Vision and Pattern Recognition
0
CNN-Based Camera Pose Estimation and Localisation of Scan Images for Aircraft Visual Inspection
Title CN 基于CNN的扫描图像相机位姿估计与飞机视觉检测定位
Keywords 相机位姿估计 · 视觉检测 · 域随机化 · 合成数据 · CNN
Summary 本文提出一种无需外部基础设施的现场方法，利用深度卷积神经网络结合合成图像微调，实现对云台相机位姿的估计，并完成飞机表面扫描图像的精确定位。通过域随机化生成训练数据，并利用飞机几何结构改进损失函数，提升了位姿估计精度。实验表明，在真实飞机场景中，相机位姿估计误差小于0.24米和2度。
Reason 该论文属于计算机视觉中的视觉定位与机器人应用领域，聚焦于飞机表面检测中的相机位姿估计问题，使用合成数据和CNN进行无基础设施的定位。其核心任务与文档图像理解（DIU）无关，不涉及文本识别、布局分析、表格理解或VLM在文档上的应用。尽管使用了深度学习和域随机化等技术，但这些方法针对的是3D场景几何与相机运动估计，难以迁移到文档图像中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract General Visual Inspection is a manual inspection process regularly used to detect and localise obvious damage on the exterior of commercial aircraft. There has been increasing demand to perform this process at the boarding gate to minimise the downtime of the aircraft and automating this process is desired to reduce the reliance on human labour. Automating this typically requires estimating a camera's pose with respect to the aircraft for initialisation but most existing localisation methods require infrastructure, which is very challenging in uncontrolled outdoor environments and within the limited turnover time (approximately 2 hours) on an airport tarmac. Additionally, many airlines and airports do not allow contact with the aircraft's surface or using UAVs for inspection between flights, and restrict access to commercial aircraft. Hence, this paper proposes an on-site method that is infrastructure-free and easy to deploy for estimating a pan-tilt-zoom camera's pose and localising scan images. This method initialises using the same pan-tilt-zoom camera used for the inspection task by utilising a Deep Convolutional Neural Network fine-tuned on only synthetic images to predict its own pose. We apply domain randomisation to generate the dataset for fine-tuning the network and modify its loss function by leveraging aircraft geometry to improve accuracy. We also propose a workflow for initialisation, scan path planning, and precise localisation of images captured from a pan-tilt-zoom camera. We evaluate and demonstrate our approach through experiments with real aircraft, achieving root-mean-square camera pose estimation errors of less than 0.24 m and 2 degrees for all real scenes.
Authors Xueyan Oh, Leonard Loh, Shaohui Foong, Zhong Bao Andy Koh, Kow Leong Ng, Poh Kang Tan, Pei Lin Pearlin Toh, U-Xuan Tan
Comment 12 pages, 12 figures
Categories Computer Vision and Pattern Recognition
0
RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenes
Title CN RefDrone：无人机场景中指代表达理解的一个具有挑战性的基准
Keywords 指代表达理解 · 无人机视觉 · 多目标定位 · RDAgent · NGDINO
Summary 本文提出了一个面向无人机航拍场景的指代表达理解基准RefDrone，旨在解决空中视角下因尺度变化、遮挡和视角多变带来的目标定位难题。作者构建了包含多尺度小目标、多目标及无目标样本的数据集，并开发了基于多智能体系统的半自动标注工具RDAgent以提升标注效率与质量。同时提出Number GroundingDINO（NGDINO）模型，通过显式建模语言表达中提及的对象数量来改善多目标和无目标情况下的定位性能。实验表明该方法在RefDrone和gRefCOCO上均表现优异。
Reason 该论文聚焦于无人机场景下的指代表达理解（REC），属于具身智能与视频/图像理解的交叉领域，主要应用场景为机器人平台的视觉定位。尽管其提出的NGDINO方法涉及多目标和无目标情况下的数量感知 grounding，但整体任务与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析等无关。且其数据域（航拍图像）与文档图像差异巨大，方法迁移难度极高，不具备对DocVQA或文档VLM的直接启发价值。
Abstract Drones have become prevalent robotic platforms with diverse applications, showing significant potential in Embodied Artificial Intelligence (Embodied AI). Referring Expression Comprehension (REC) enables drones to locate objects based on natural language expressions, a crucial capability for Embodied AI. Despite advances in REC for ground-level scenes, aerial views introduce unique challenges including varying viewpoints, occlusions and scale variations. To address this gap, we introduce RefDrone, a REC benchmark for drone scenes. RefDrone reveals three key challenges in REC: 1) multi-scale and small-scale target detection; 2) multi-target and no-target samples; 3) complex environment with rich contextual expressions. To efficiently construct this dataset, we develop RDAgent (referring drone annotation framework with multi-agent system), a semi-automated annotation tool for REC tasks. RDAgent ensures high-quality contextual expressions and reduces annotation cost. Furthermore, we propose Number GroundingDINO (NGDINO), a novel method designed to handle multi-target and no-target cases. NGDINO explicitly learns and utilizes the number of objects referred to in the expression. Comprehensive experiments with state-of-the-art REC methods demonstrate that NGDINO achieves superior performance on both the proposed RefDrone and the existing gRefCOCO datasets. The dataset and code are be publicly at https://github.com/sunzc-sunny/refdrone.
Authors Zhichao Sun, Yepeng Liu, Zhiling Su, Huachao Zhu, Yuliang Gu, Yuda Zou, Zelong Liu, Gui-Song Xia, Bo Du, Yongchao Xu
Categories Computer Vision and Pattern Recognition
0
DAGLFNet: Deep Feature Attention Guided Global and Local Feature Fusion for Pseudo-Image Point Cloud Segmentation
Title CN DAGLFNet：基于深度特征注意力引导的全局与局部特征融合用于伪图像点云分割
Keywords 点云分割 · 伪图像 · 3D视觉 · 注意力机制 · 自动驾驶
Summary 本文提出DAGLFNet，一种用于伪图像点云语义分割的框架，通过全局-局部特征融合编码模块、多分支特征提取网络以及深度特征引导的注意力机制来提升特征判别能力，在SemanticKITTI和nuScenes数据集上取得良好表现。
Reason 该论文研究的是基于伪图像表示的LiDAR点云语义分割，属于3D视觉与自动驾驶感知领域，核心任务为点云处理与2D-3D特征融合，与文档图像理解（DIU）或DocVQA无关。其方法聚焦于提升环境感知系统中的点云分割性能，无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract Environmental perception systems are crucial for high-precision mapping and autonomous navigation, with LiDAR serving as a core sensor providing accurate 3D point cloud data. Efficiently processing unstructured point clouds while extracting structured semantic information remains a significant challenge. In recent years, numerous pseudo-image-based representation methods have emerged to balance efficiency and performance by fusing 3D point clouds with 2D grids. However, the fundamental inconsistency between the pseudo-image representation and the original 3D information critically undermines 2D-3D feature fusion, posing a primary obstacle for coherent information fusion and leading to poor feature discriminability. This work proposes DAGLFNet, a pseudo-image-based semantic segmentation framework designed to extract discriminative features. It incorporates three key components: first, a Global-Local Feature Fusion Encoding (GL-FFE) module to enhance intra-set local feature correlation and capture global contextual information; second, a Multi-Branch Feature Extraction (MB-FE) network to capture richer neighborhood information and improve the discriminability of contour features; and third, a Feature Fusion via Deep Feature-guided Attention (FFDFA) mechanism to refine cross-channel feature fusion precision. Experimental evaluations demonstrate that DAGLFNet achieves mean Intersection-over-Union (mIoU) scores of 69.9% and 78.7% on the validation sets of SemanticKITTI and nuScenes, respectively. The method achieves an excellent balance between accuracy and efficiency.
Authors Chuang Chen, Yi Lin, Bo Wang, Jing Hu, Xi Wu, Wenyi Ge
Categories Computer Vision and Pattern Recognition
0
Exploring Surround-View Fisheye Camera 3D Object Detection AAAI 2026
Title CN 探索环视鱼眼相机的三维物体检测
Keywords 3D目标检测 · 鱼眼相机 · BEV表示 · 查询式检测 · 自动驾驶
Summary 本文探讨了使用环视鱼眼相机系统实现端到端3D目标检测的可行性，提出了两种融合鱼眼几何特性的检测方法FisheyeBEVDet和FisheyePETR，并发布了基于CARLA的合成数据集Fisheye3DOD。实验表明所提方法相比基线提升了最多6.2%的精度。
Reason 该论文研究的是自动驾驶场景下的鱼眼相机3D物体检测，属于3D视觉与自动驾驶领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用完全无关。其方法聚焦于BEV和查询式3D检测架构在鱼眼几何中的建模，无法迁移到文档理解的核心痛点（如OCR幻觉、复杂排版、长文档推理等）。
Abstract In this work, we explore the technical feasibility of implementing end-to-end 3D object detection (3DOD) with surround-view fisheye camera system. Specifically, we first investigate the performance drop incurred when transferring classic pinhole-based 3D object detectors to fisheye imagery. To mitigate this, we then develop two methods that incorporate the unique geometry of fisheye images into mainstream detection frameworks: one based on the bird's-eye-view (BEV) paradigm, named FisheyeBEVDet, and the other on the query-based paradigm, named FisheyePETR. Both methods adopt spherical spatial representations to effectively capture fisheye geometry. In light of the lack of dedicated evaluation benchmarks, we release Fisheye3DOD, a new open dataset synthesized using CARLA and featuring both standard pinhole and fisheye camera arrays. Experiments on Fisheye3DOD show that our fisheye-compatible modeling improves accuracy by up to 6.2% over baseline methods.
Authors Changcai Li, Wenwei Lin, Zuoxun Hou, Gang Chen, Wei Zhang, Huihui Zhou, Weishi Zheng
Comment 9 pages,6 figures, accepted at AAAI 2026
Categories Computer Vision and Pattern Recognition
0
Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents
Title CN 超越描述：具身智能体细粒度动作的认知基准测试
Keywords 具身智能 · 多模态大模型 · 动作理解 · 基准测试 · 物理交互
Summary 本文提出了CFG-Bench，一个用于评估多模态大模型在具身环境中执行细粒度动作能力的新基准，涵盖物理交互、因果关系、意图理解和评价判断四个方面。实验表明现有MLLM在这些高阶推理任务上表现不佳，且通过监督微调可提升性能。然而该工作与文档理解无关。
Reason 该论文聚焦于具身智能体在物理环境中的细粒度动作理解与决策，属于具身智能（Embodied AI）领域。其提出的CFG-Bench是一个针对物理交互、因果时序推理等能力的视频问答基准，应用场景与文档图像理解（DIU）无直接关联。尽管涉及多模态大模型（MLLMs）的推理能力评估，但任务设定、数据模态和目标领域均远离文档结构理解、文本识别或VQA中的OCR幻觉等问题，无法迁移用于解决文档理解的核心痛点。
Abstract Multimodal Large Language Models (MLLMs) show promising results as decision-making engines for embodied agents operating in complex, physical environments. However, existing benchmarks often prioritize high-level planning or spatial reasoning, leaving the fine-grained action intelligence required for embodied physical interaction underexplored. To address this gap, we introduce CFG-Bench, a new benchmark designed to systematically evaluate this crucial capability. CFG-Bench consists of 1,368 curated videos paired with 19,562 three-modalities question-answer pairs targeting four cognitive abilities: 1) Physical Interaction, 2) Temporal-Causal Relation, 3) Intentional Understanding, and 4) Evaluative Judgment. Together, these dimensions provide a systematic framework for assessing a model's ability to translate visual observations into actionable knowledge, moving beyond mere surface-level recognition. Our comprehensive evaluation on CFG-Bench reveals that leading MLLMs struggle to produce detailed instructions for physical interactions and exhibit profound limitations in the higher-order reasoning of intention and evaluation. Moreover, supervised fine-tuning (SFT) on our data demonstrates that teaching an MLLMs to articulate fine-grained actions directly translates to significant performance gains on established embodied benchmarks. Our analysis highlights these limitations and offers insights for developing more capable and grounded embodied agents.
Authors Dayong Liu, Chao Xu, Weihong Chen, Suyu Zhang, Juncheng Wang, Jiankang Deng, Baigui Sun, Yang Liu
Categories Computer Vision and Pattern Recognition
0
Faster and Better 3D Splatting via Group Training ICCV 2025
Title CN 通过分组训练实现更快更好的3D点阵化
Keywords 3D Gaussian Splatting · 分组训练 · 新视角合成 · 场景重建 · 训练加速
Summary 本文提出了一种名为Group Training的方法，用于加速3D高斯点阵化（3DGS）的训练过程，并提升渲染质量。通过将高斯图元分组进行优化，该方法在多种3DGS框架中实现了最高30%的收敛速度提升，同时保持高质量的新视角合成效果。
Reason 该论文属于3D视觉与场景重建领域，研究的是3D Gaussian Splatting的训练效率优化问题。尽管其方法在计算机视觉中有一定价值，但与文档图像理解（DIU）、多模态大模型（VLM）或文档VQA等核心任务完全无关。无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU痛点。
Abstract 3D Gaussian Splatting (3DGS) has emerged as a powerful technique for novel view synthesis, demonstrating remarkable capability in high-fidelity scene reconstruction through its Gaussian primitive representations. However, the computational overhead induced by the massive number of primitives poses a significant bottleneck to training efficiency. To overcome this challenge, we propose Group Training, a simple yet effective strategy that organizes Gaussian primitives into manageable groups, optimizing training efficiency and improving rendering quality. This approach shows universal compatibility with existing 3DGS frameworks, including vanilla 3DGS and Mip-Splatting, consistently achieving accelerated training while maintaining superior synthesis quality. Extensive experiments reveal that our straightforward Group Training strategy achieves up to 30\% faster convergence and improved rendering quality across diverse scenarios. Project Website: https://chengbo-wang.github.io/3DGS-with-Group-Training/
Authors Chengbo Wang, Guozheng Ma, Yifei Xue, Yizhen Lao
Comment Accepted to ICCV 2025. Code is available at https://github.com/Chengbo-Wang/3DGS-with-Group-Training
Categories Computer Vision and Pattern Recognition
0
Now You See It, Now You Don't - Instant Concept Erasure for Safe Text-to-Image and Video Generation
Title CN 现在你看到它，现在你看不到它——用于安全文本到图像和视频生成的即时概念擦除
Keywords 概念擦除 · 文本到图像生成 · 生成模型安全 · 无需训练 · 子空间正则化
Summary 本文提出了一种名为Instant Concept Erasure（ICE）的方法，用于在无需重新训练的情况下从文本到图像和视频生成模型中彻底移除特定语义概念（如艺术风格、物体、身份或敏感内容）。该方法通过定义擦除与保留子空间，并使用闭式重叠投影算子来显式正则化二者交集，实现精确、持久的遗忘效果，同时保持生成质量。整个过程无推理开销，适用于多种模态。
Reason 该论文研究的是文本到图像和视频生成中的概念擦除问题，属于纯图像/视频生成领域的安全控制技术。尽管其方法具有一定的理论创新性，但应用场景与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版、长文档推理等核心痛点。且未涉及多模态文档中的视觉-语言对齐或结构理解，属于负面清单中的‘无关领域’。
Abstract Robust concept removal for text-to-image (T2I) and text-to-video (T2V) models is essential for their safe deployment. Existing methods, however, suffer from costly retraining, inference overhead, or vulnerability to adversarial attacks. Crucially, they rarely model the latent semantic overlap between the target erase concept and surrounding content -- causing collateral damage post-erasure -- and even fewer methods work reliably across both T2I and T2V domains. We introduce Instant Concept Erasure (ICE), a training-free, modality-agnostic, one-shot weight modification approach that achieves precise, persistent unlearning with zero overhead. ICE defines erase and preserve subspaces using anisotropic energy-weighted scaling, then explicitly regularises against their intersection using a unique, closed-form overlap projector. We pose a convex and Lipschitz-bounded Spectral Unlearning Objective, balancing erasure fidelity and intersection preservation, that admits a stable and unique analytical solution. This solution defines a dissociation operator that is translated to the model's text-conditioning layers, making the edit permanent and runtime-free. Across targeted removals of artistic styles, objects, identities, and explicit content, ICE efficiently achieves strong erasure with improved robustness to red-teaming, all while causing only minimal degradation of original generative abilities in both T2I and T2V models.
Authors Shristi Das Biswas, Arani Roy, Kaushik Roy
Categories Computer Vision and Pattern Recognition
0
Hierarchical GraphCut Phase Unwrapping based on Invariance of Diffeomorphisms Framework Open Journal of Signal Processing (OJSP) as journal paper for ICIP2025
Title CN 基于微分同胚不变性的分层GraphCut相位展开
Keywords 相位展开 · 结构光扫描 · GraphCut · 微分同胚 · 三维重建
Summary 本文提出一种基于微分同胚不变性框架的分层GraphCut相位展开方法，用于结构光三维扫描中从包裹相位恢复连续相位。通过共形映射和最优传输构造多个域空间，在每个域内应用GraphCut算法进行像素标记，并通过多数投票融合结果，显著提升速度与精度。适用于4D面部动态捕捉等实时场景。
Reason 该论文研究的是3D扫描中的相位展开问题，属于结构光三维重建领域，核心任务是恢复连续相位值以重建3D表面。尽管涉及计算机视觉和图像处理技术，但其目标与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。既不涉及文本识别、布局分析、表格理解，也不解决OCR幻觉、复杂排版或多模态推理等DIU核心痛点。属于典型的无关领域工作（3D视觉/AR/VR），命中负面清单第2条。
Abstract Recent years have witnessed rapid advancements in 3D scanning technologies, with applications spanning VR/AR, digital human creation, and medical imaging. Structured-light scanning with phase-shifting techniques is preferred for its use of low-intensity visible light and high accuracy, making it well suited for capturing 4D facial dynamics. A key step is phase unwrapping, which recovers continuous phase values from measurements wrapped modulo 2pi. The goal is to estimate the unwrapped phase count k in the equation Phi = phi + 2pi k, where phi is the wrapped phase and Phi is the true phase. Noise, occlusions, and complex 3D geometry make recovering the true phase challenging because phase unwrapping is ill-posed: measurements only provide modulo 2pi values, and estimating k requires assumptions about surface continuity. Existing methods trade speed for accuracy: fast approaches lack precision, while accurate algorithms are too slow for real-time use. To overcome these limitations, this work proposes a phase unwrapping framework that reformulates GraphCut-based unwrapping as a pixel-labeling problem. This framework improves the estimation of the unwrapped phase count k through the invariance property of diffeomorphisms applied in image space via conformal and optimal transport (OT) maps. An odd number of diffeomorphisms are precomputed from the input phase data, and a hierarchical GraphCut algorithm is applied in each domain. The resulting label maps are fused via majority voting to robustly estimate k at each pixel. Experimental results demonstrate a 45.5x speedup and lower L2 error in real experiments and simulations, showing potential for real-time applications.
Authors Xiang Gao, Xinmu Wang, Zhou Zhao, Junqi Huang, Xianfeng David Gu
Comment Open Journal of Signal Processing (OJSP) as journal paper for ICIP2025 Accepted
Categories Computer Vision and Pattern Recognition
0
Inverse Rendering for High-Genus Surface Meshes from Multi-View Images 3DV2026
Title CN 基于多视角图像的高亏格曲面网格逆渲染
Keywords 逆渲染 · 高亏格曲面 · 网格重建 · 拓扑一致性 · 重网格化
Summary 本文提出一种结合拓扑感知的逆渲染方法，用于从多视角图像中重建高亏格表面网格。通过引入自适应V-cycle重网格策略和重新参数化的Adam优化器，在优化过程中动态调整网格结构，增强对拓扑和几何特征的感知能力。同时利用高斯-博内定理构建与真实数据拓扑一致的原始形体，确保拓扑一致性。实验表明，该方法在Chamfer距离和体积IoU等指标上显著优于现有方法，尤其在高亏格表面上表现突出，同时提升了低亏格表面的细节还原能力。
Reason 该论文属于3D视觉与图形学领域，研究的是从多视角图像重建高亏格曲面网格的逆渲染问题，核心挑战是拓扑保持与几何细节恢复，使用的方法如自适应重网格化和改进的Adam优化器均针对3D网格优化，并不涉及文档图像、文本理解或多模态语言模型。其技术路线与文档图像理解（DIU）中的OCR幻觉、复杂排版解析或长文档推理等核心痛点无直接关联，无法迁移应用于文档理解任务。
Abstract We present a topology-informed inverse rendering approach for reconstructing high-genus surface meshes from multi-view images. Compared to 3D representations like voxels and point clouds, mesh-based representations are preferred as they enable the application of differential geometry theory and are optimized for modern graphics pipelines. However, existing inverse rendering methods often fail catastrophically on high-genus surfaces, leading to the loss of key topological features, and tend to oversmooth low-genus surfaces, resulting in the loss of surface details. This failure stems from their overreliance on Adam-based optimizers, which can lead to vanishing and exploding gradients. To overcome these challenges, we introduce an adaptive V-cycle remeshing scheme in conjunction with a re-parametrized Adam optimizer to enhance topological and geometric awareness. By periodically coarsening and refining the deforming mesh, our method informs mesh vertices of their current topology and geometry before optimization, mitigating gradient issues while preserving essential topological features. Additionally, we enforce topological consistency by constructing topological primitives with genus numbers that match those of ground truth using Gauss-Bonnet theorem. Experimental results demonstrate that our inverse rendering approach outperforms the current state-of-the-art method, achieving significant improvements in Chamfer Distance and Volume IoU, particularly for high-genus surfaces, while also enhancing surface details for low-genus surfaces.
Authors Xiang Gao, Xinmu Wang, Xiaolong Wu, Jiazhi Li, Jingyu Shi, Yu Guo, Yuanpeng Liu, Xiyun Song, Heather Yu, Zongfang Lin, Xianfeng David Gu
Comment 3DV2026 Accepted (Poster)
Categories Computer Vision and Pattern Recognition
0
Full-scale Representation Guided Network for Retinal Vessel Segmentation
Title CN 用于视网膜血管分割的全尺度表示引导网络
Keywords 视网膜血管分割 · U-Net变体 · 注意力机制 · 医学图像分割 · 多尺度特征
Summary 本文提出了一种名为FSG-Net的全尺度引导网络，用于提升视网膜血管分割性能。该网络通过现代卷积模块捕获多尺度结构信息，并引入注意力引导的卷积块来增强细微血管结构的表达能力。实验表明，该方法在多个公开数据集上达到了与当前最优方法相媲美的性能，且具有良好的可扩展性。
Reason 该论文研究的是医学图像中的视网膜血管分割任务，属于生物医学图像分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管其提出的FSG-Net在特征提取和注意力机制上有一定创新，但其方法针对的是生物医学图像中的细粒度结构识别，无法直接迁移解决文档图像中的OCR幻觉、复杂排版、长文档推理等核心痛点。
Abstract The U-Net architecture and its variants have remained state-of-the-art (SOTA) for retinal vessel segmentation over the past decade. In this study, we introduce a Full-Scale Guided Network (FSG-Net), where a novel feature representation module using modernized convolution blocks effectively captures full-scale structural information, while a guided convolution block subsequently refines this information. Specifically, we introduce an attention-guided filter within the guided convolution block, leveraging its similarity to unsharp masking to enhance fine vascular structures. Passing full-scale information to the attention block facilitates the generation of more contextually relevant attention maps, which are then passed to the attention-guided filter, providing further refinement to the segmentation performance. The structure preceding the guided convolution block can be replaced by any U-Net variant, ensuring flexibility and scalability across various segmentation tasks. For a fair comparison, we re-implemented recent studies available in public repositories to evaluate their scalability and reproducibility. Our experiments demonstrate that, despite its compact architecture, FSG-Net delivers performance competitive with SOTA methods across multiple public datasets. Ablation studies further demonstrate that each proposed component meaningfully contributes to this competitive performance. Our code is available on https://github.com/ZombaSY/FSG-Net-pytorch.
Authors Sunyong Seo, Sangwook Yoo, Huisu Yoon
Comment 12 pages, 7 figures
Categories Computer Vision and Pattern Recognition
0
A Theory-Inspired Framework for Few-Shot Cross-Modal Sketch Person Re-Identification AAAI2026
Title CN 一种受理论启发的少样本跨模态素描行人重识别框架
Keywords 行人重识别 · 素描匹配 · 跨模态学习 · 小样本学习 · 域对齐
Summary 本文提出KTCAA框架，用于解决少样本条件下的素描到RGB行人重识别问题。该方法基于泛化理论，通过领域差异建模和扰动不变性增强，在元学习范式下实现跨模态知识迁移。包含对齐增强（AA）模块以模拟目标分布，并设计知识迁移催化剂（KTC）提升模型对模态变化的鲁棒性。实验表明其在多个基准上取得领先性能。
Reason 该论文研究的是素描与RGB图像之间的跨模态行人重识别问题，属于特定视觉匹配任务，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的方法涉及跨模态对齐和小样本学习，但应用场景为监控图像与手绘素描的匹配，迁移到文档图像理解中的OCR幻觉、复杂排版或长文档推理等问题极为困难，且未涉及文本密集、结构解析或视觉语言建模等DIU关键挑战。
Abstract Sketch based person re-identification aims to match hand-drawn sketches with RGB surveillance images, but remains challenging due to significant modality gaps and limited annotated data. To address this, we introduce KTCAA, a theoretically grounded framework for few-shot cross-modal generalization. Motivated by generalization theory, we identify two key factors influencing target domain risk: (1) domain discrepancy, which quantifies the alignment difficulty between source and target distributions; and (2) perturbation invariance, which evaluates the model's robustness to modality shifts. Based on these insights, we propose two components: (1) Alignment Augmentation (AA), which applies localized sketch-style transformations to simulate target distributions and facilitate progressive alignment; and (2) Knowledge Transfer Catalyst (KTC), which enhances invariance by introducing worst-case perturbations and enforcing consistency. These modules are jointly optimized under a meta-learning paradigm that transfers alignment knowledge from data-rich RGB domains to sketch-based scenarios. Experiments on multiple benchmarks demonstrate that KTCAA achieves state-of-the-art performance, particularly in data-scarce conditions.
Authors Yunpeng Gong, Yongjie Hou, Jiangming Shi, Kim Long Diep, Min Jiang
Comment Accepted by AAAI2026
Categories Computer Vision and Pattern Recognition
0
Sphinx: Efficiently Serving Novel View Synthesis using Regression-Guided Selective Refinement
Title CN Sphinx：使用回归引导的选择性优化高效服务新视角合成
Keywords 新视角合成 · 扩散模型 · 推理效率 · 选择性优化 · 计算机视觉
Summary 本文提出Sphinx，一种无需训练的混合推理框架，通过回归模型快速初始化并引导扩散模型的去噪过程，结合选择性优化和自适应噪声调度，在新视角合成任务中实现了高质量与低计算开销的平衡。实验表明其比纯扩散模型快1.8倍，且感知质量下降不到5%。
Reason 该论文研究的是新视角合成（NVS），属于3D视觉与视频生成领域，核心关注扩散模型与回归模型在视图生成中的效率与质量权衡。尽管其推理效率优化思路具有一定通用性，但任务本身与文档图像理解（DIU）如DocVQA、布局分析、表格识别等无直接关联，无法迁移解决OCR幻觉、密集文本识别、复杂排版或长文档推理等核心痛点。
Abstract Novel View Synthesis (NVS) is the task of generating new images of a scene from viewpoints that were not part of the original input. Diffusion-based NVS can generate high-quality, temporally consistent images, however, remains computationally prohibitive. Conversely, regression-based NVS offers suboptimal generation quality despite requiring significantly lower compute; leaving the design objective of a high-quality, inference-efficient NVS framework an open challenge. To close this critical gap, we present Sphinx, a training-free hybrid inference framework that achieves diffusion-level fidelity at a significantly lower compute. Sphinx proposes to use regression-based fast initialization to guide and reduce the denoising workload for the diffusion model. Additionally, it integrates selective refinement with adaptive noise scheduling, allowing more compute to uncertain regions and frames. This enables Sphinx to provide flexible navigation of the performance-quality trade-off, allowing adaptation to latency and fidelity requirements for dynamically changing inference scenarios. Our evaluation shows that Sphinx achieves an average 1.8x speedup over diffusion model inference with negligible perceptual degradation of less than 5%, establishing a new Pareto frontier between quality and latency in NVS serving.
Authors Yuchen Xia, Souvik Kundu, Mosharaf Chowdhury, Nishil Talati
Categories Computer Vision and Pattern Recognition
0
Data Augmentation Strategies for Robust Lane Marking Detection ACRA, Australasian conference on robotics and automation
Title CN 用于鲁棒车道标记检测的数据增强策略
Keywords 车道检测 · 数据增强 · 生成式AI · 域适应 · 自动驾驶
Summary 本文提出了一种基于生成式AI的数据增强流程，用于提升侧置摄像头下车道标记检测的鲁棒性。通过结合几何透视变换、AI修复和车辆本体覆盖，模拟特定部署视角的同时保持车道连续性。在SCNN和UFLDv2模型上的实验表明，该方法在精度、召回率和F1分数上均有提升，尤其在阴影等复杂条件下表现更优。
Reason 该论文研究的是自动驾驶场景下的车道检测问题，属于具身智能与机器人领域，且聚焦于道路场景的视觉感知，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了数据增强和视角变换等技术，但其方法针对的是车载摄像头的域适应问题，无法迁移至解决文档中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Robust lane detection is essential for advanced driver assistance and autonomous driving, yet models trained on public datasets such as CULane often fail to generalise across different camera viewpoints. This paper addresses the challenge of domain shift for side-mounted cameras used in lane-wheel monitoring by introducing a generative AI-based data enhancement pipeline. The approach combines geometric perspective transformation, AI-driven inpainting, and vehicle body overlays to simulate deployment-specific viewpoints while preserving lane continuity. We evaluated the effectiveness of the proposed augmentation in two state-of-the-art models, SCNN and UFLDv2. With the augmented data trained, both models show improved robustness to different conditions, including shadows. The experimental results demonstrate gains in precision, recall, and F1 score compared to the pre-trained model. By bridging the gap between widely available datasets and deployment-specific scenarios, our method provides a scalable and practical framework to improve the reliability of lane detection in a pilot deployment scenario.
Authors Flora Lian, Dinh Quang Huynh, Hector Penades, J. Stephany Berrio Perez, Mao Shan, Stewart Worrall
Comment 8 figures, 2 tables, 10 pages, ACRA, Australasian conference on robotics and automation
Categories Computer Vision and Pattern Recognition
0
FCDM: A Physics-Guided Bidirectional Frequency Aware Convolution and Diffusion-Based Model for Sinogram Inpainting
Title CN FCDM：一种基于物理引导的双向频率感知卷积与扩散模型用于正弦图修复
Keywords 正弦图修复 · 扩散模型 · 物理引导 · 频率感知 · 稀疏视角CT
Summary 本文提出FCDM，一种针对稀疏视角CT中正弦图修复的扩散模型。该方法结合物理引导约束与频率自适应噪声控制，并通过双向频率感知卷积建模角度依赖性，以恢复因欠采样导致的信息丢失。实验表明其在真实数据上优于现有方法，SSIM超过0.93，PSNR高于31 dB。
Reason 该论文研究的是CT成像中的正弦图修复问题，属于医学影像重建领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法聚焦于利用物理约束和频率特性恢复投影域数据，不涉及文本识别、布局分析、多模态文档理解或OCR相关挑战。因此无法迁移到文档图像理解的核心痛点（如OCR幻觉、复杂排版、长文档推理等），属于无关领域。
Abstract Computed tomography (CT) is widely used in scientific imaging systems such as synchrotron and laboratory-based nano-CT, but acquiring full-view sinograms requires high radiation dose and long scan times. Sparse-view CT alleviates this burden but yields incomplete sinograms with structured signal loss, hampering accurate reconstruction. Unlike RGB images, sinograms encode overlapping features along projection paths and exhibit distinct directional spectral patterns, which make conventional RGB-oriented inpainting approaches--including diffusion models--ineffective for sinogram restoration, as they disregard the angular dependencies and physical constraints inherent to tomographic data. To overcome these limitations, we propose FCDM, a diffusion-based framework tailored for sinograms, which restores global structure through bidirectional frequency reasoning and angular-aware masking, while enforcing physical plausibility via physics-guided constraints and frequency-adaptive noise control. Experiments on real-world datasets show that FCDM consistently outperforms baselines, achieving SSIM over 0.93 and PSNR above 31 dB across diverse sparse-view scenarios.
Authors Jiaze E, Srutarshi Banerjee, Tekin Bicer, Guannan Wang, Yanfu Zhang, Bin Ren
Categories Computer Vision and Pattern Recognition
0
A Storage-Efficient Feature for 3D Concrete Defect Segmentation to Replace Normal Vector
Title CN 一种用于3D混凝土缺陷分割的存储高效特征以替代法向量
Keywords 3D点云 · 缺陷分割 · 法向量 · 存储效率 · PointNet++
Summary 本文提出一种名为‘相对角度’的新特征，用于3D混凝土缺陷分割，通过计算点法向量与其所属点云平均法向量之间的夹角，实现与传统法向量相当的方向性表达能力。实验表明，使用该特征可在PointNet++上达到与原始法向量相近的性能，同时减少27.6%的存储需求和83%的输入通道。该方法专注于3D点云数据压缩与工业缺陷检测。
Reason 该论文研究的是3D点云中的混凝土缺陷分割，属于工业检测与3D视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其提出的方法‘相对角度’特征虽在存储效率上有优势，但针对的是3D几何结构信息的压缩，并不适用于文档图像中的文本识别、布局分析或VLM处理等核心问题。且该工作未涉及多模态语言模型、OCR、视觉token压缩或文档推理等DIU关键挑战。
Abstract Point cloud reconstruction of damage offers an effective solution to image-based methods vulnerable to background noise, yet its application is constrained by the high volume of 3D data. This study proposes a new feature, relative angle, computed as the angle between the normal vector of a point and the average normal vector of its parent point cloud. This single-dimensional feature provides directionality information equivalent to normal vectors for concrete surface defect characteristics. Through entropy-based feature evaluation, this study demonstrates the ability of relative angle to filter out redundant information in undamaged sections while retaining effective information in damaged sections. By training and testing with PointNet++, models based on the relative angles achieved similar performance to that of models based on normal vectors while delivering 27.6% storage reduction and 83% input channel compression. This novel feature has the potential to enable larger-batch execution on resource-constrained hardware without the necessity of architectural modifications to models.
Authors Linxin Hua, Jianghua Deng, Ye Lu
Comment 25 pages, 7 figures
Categories Computer Vision and Pattern Recognition
0
Learning Efficient Fuse-and-Refine for Feed-Forward 3D Gaussian Splatting NeurIPS 2025
Title CN 用于前馈式3D高斯点阵的高效融合与优化学习
Keywords 3D Gaussian Splatting · 场景重建 · 稀疏视图 · 体渲染 · 动态场景
Summary 本文提出一种名为Fuse-and-Refine的模块，通过构建混合的Splat-Voxel表示，在规范的3D空间中融合并优化像素对齐的高斯图元，从而减少冗余并支持动态场景的流式重建。该方法在静态和动态场景重建中均达到SOTA性能，并可在单个H100 GPU上实现交互式帧率。
Reason 该论文属于3D视觉与场景重建领域，研究的是动态场景下的3D高斯点阵（Gaussian Splatting）表示与重建，核心问题为多视角几何、体渲染加速和时序一致性。尽管其方法涉及视觉表示学习，但与文档图像理解（DIU）、DocVQA、OCR、排版分析等任务无直接关联。无法迁移解决文档中的OCR幻觉、复杂布局、长文档推理等核心痛点。
Abstract Recent advances in feed-forward 3D Gaussian Splatting have led to rapid improvements in efficient scene reconstruction from sparse views. However, most existing approaches construct Gaussian primitives directly aligned with the pixels in one or more of the input images. This leads to redundancies in the representation when input views overlap and constrains the position of the primitives to lie along the input rays without full flexibility in 3D space. Moreover, these pixel-aligned approaches do not naturally generalize to dynamic scenes, where effectively leveraging temporal information requires resolving both redundant and newly appearing content across frames. To address these limitations, we introduce a novel Fuse-and-Refine module that enhances existing feed-forward models by merging and refining the primitives in a canonical 3D space. At the core of our method is an efficient hybrid Splat-Voxel representation: from an initial set of pixel-aligned Gaussian primitives, we aggregate local features into a coarse-to-fine voxel hierarchy, and then use a sparse voxel transformer to process these voxel features and generate refined Gaussian primitives. By fusing and refining an arbitrary number of inputs into a consistent set of primitives, our representation effectively reduces redundancy and naturally adapts to temporal frames, enabling history-aware online reconstruction of dynamic scenes. Our approach achieves state-of-the-art performance in both static and streaming scene reconstructions while running at interactive rates (15 fps with 350ms delay) on a single H100 GPU.
Authors Yiming Wang, Lucy Chai, Xuan Luo, Michael Niemeyer, Manuel Lagunas, Stephen Lombardi, Siyu Tang, Tiancheng Sun
Comment NeurIPS 2025, Previously titled "SplatVoxel: History-Aware Novel View Streaming without Temporal Training", Project Page: https://19reborn.github.io/SplatVoxel/
Categories Computer Vision and Pattern Recognition
0
Enhancing Medical Image Analysis through Geometric and Photometric transformations
Title CN 通过几何和光度变换增强医学图像分析
Keywords 数据增强 · 医学图像分析 · CNN · U-Net · 图像分割
Summary 本文在皮肤癌和视网膜血管数据集上评估了几何与光度变换等数据增强方法对CNN和U-Net模型性能的影响，结果显示增强后分类准确率和分割Dice系数均有显著提升。但研究局限于医学图像领域的常规增强手段，无方法论突破。
Reason 该论文属于典型的垂直领域应用（医学图像分类与分割），使用传统数据增强技术（如Mixup）提升模型性能，未提出任何面向文档图像理解（DIU）的底层方法创新。其技术路线为标准的CNN+数据增强范式，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心问题，属于‘平行下游应用’，命中负面清单第1条。
Abstract Medical image analysis suffers from a lack of labeled data due to several challenges including patient privacy and lack of experts. Although some AI models only perform well with large amounts of data, we will move to data augmentation where there is a solution to improve the performance of our models and increase the dataset size through traditional or advanced techniques. In this paper, we evaluate the effectiveness of data augmentation techniques on two different medical image datasets. In the first step, we applied some transformation techniques to the skin cancer dataset containing benign and malignant classes. Then, we trained the convolutional neural network (CNN) on the dataset before and after augmentation, which significantly improved test accuracy from 90.74% to 96.88% and decreased test loss from 0.7921 to 0.1468 after augmentation. In the second step, we used the Mixup technique by mixing two random images and their corresponding masks using the retina and blood vessels dataset, then we trained the U-net model and obtained the Dice coefficient which increased from 0 before augmentation to 0.4163 after augmentation. The result shows the effect of using data augmentation to increase the dataset size on the classification and segmentation performance.
Authors Khadija Rais, Mohamed Amroune, Mohamed Yassine Haouam, Abdelmadjid Benmachiche
Categories Computer Vision and Pattern Recognition
0
Achieving detailed medial temporal lobe segmentation with upsampled isotropic training from implicit neural representation
Title CN 通过从隐式神经表示的上采样各向同性训练实现详细的内侧颞叶分割
Keywords 医学图像分割 · MRI · 隐式神经表示 · 各向同性上采样 · 阿尔茨海默病
Summary 本文提出一种利用隐式神经表示将各向异性的T2加权MRI与各向同性的T1加权MRI融合的方法，生成高分辨率各向同性的内侧颞叶子区域标注数据，用于训练nnU-Net以提升AD相关生物标志物提取的稳定性和敏感性。实验表明该方法在区分轻度认知障碍和健康个体方面表现更优。
Reason 该论文属于医学图像分析领域，研究的是基于MRI的脑部结构分割，特别是阿尔茨海默病相关的内侧颞叶子区域。尽管使用了计算机视觉技术（如nnU-Net），但其任务与文档图像理解（DIU）无关，不涉及DocVQA、布局分析、表格识别或OCR等核心问题。也未提出可迁移至文档理解领域的通用方法（如视觉表示、高分辨率处理架构、去幻觉机制等）。属于典型的专业领域图像分割应用，属于‘平行下游应用’，且领域完全偏离DIU。
Abstract Imaging biomarkers in magnetic resonance imaging (MRI) are important tools for diagnosing, tracking and treating Alzheimer's disease (AD). Neurofibrillary tau pathology in AD is closely linked to neurodegeneration and generally follows a pattern of spread in the brain, with early stages involving subregions of the medial temporal lobe (MTL). Accurate segmentation of MTL subregions is needed to extract granular biomarkers of AD progression. MTL subregions are often imaged using T2-weighted (T2w) MRI scans that are highly anisotropic due to constraints of MRI physics and image acquisition, making it difficult to reliably model MTL subregions geometrically and extract morphological measures, such as thickness. In this study, we used an implicit neural representation method to combine isotropic T1-weighted (T1w) and anisotropic T2w MRI to upsample an atlas set of expert-annotated MTL subregions, establishing a multi-modality, high-resolution training set of isotropic data for automatic segmentation with the nnU-Net framework. In an independent test set, the morphological measures extracted using this isotropic model showed stronger effect sizes than models trained on anisotropic in distinguishing participants with mild cognitive impairment (MCI) and cognitively unimpaired individuals. In test-retest analysis, morphological measures extracted using the isotropic model had greater stability. This study demonstrates improved reliability of MRI-derived MTL subregion biomarkers without additional atlas annotation effort, which may more accurately quantify and track the relationship between AD pathology and brain atrophy for monitoring disease progression.
Authors Yue Li, Pulkit Khandelwal, Rohit Jena, Long Xie, Michael Duong, Amanda E. Denning, Christopher A. Brown, Laura E. M. Wisse, Sandhitsu R. Das, David A. Wolk, Paul A. Yushkevich
Categories Computer Vision and Pattern Recognition
0
Maritime Small Object Detection from UAVs using Deep Learning with Altitude-Aware Dynamic Tiling
Title CN 基于深度学习的无人机海上小目标检测中结合高度感知动态分块的方法
Keywords 无人机 · 小目标检测 · 动态分块 · 高空成像 · 海事搜救
Summary 本文提出一种结合飞行高度信息的动态图像分块方法，用于提升无人机在海上搜索救援任务中对小目标的检测能力。通过根据高度自适应调整图像分块策略，并结合YOLOv5与SAHI框架，在SeaDronesSee数据集上实现了比基线方法高38%的小目标mAP，同时推理速度优于静态分块方法。
Reason 该论文研究的是无人机航拍图像中的小目标检测，属于遥感与监控领域，任务为海上搜救场景下的目标检测，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了视觉处理技术如动态分块和SAHI推理框架，但其方法针对的是自然场景图像中的小目标问题，无法迁移解决文档图像中的OCR幻觉、复杂排版或长文本推理等核心痛点。
Abstract Unmanned Aerial Vehicles (UAVs) are crucial in Search and Rescue (SAR) missions due to their ability to monitor vast maritime areas. However, small objects often remain difficult to detect from high altitudes due to low object-to-background pixel ratios. We propose an altitude-aware dynamic tiling method that scales and adaptively subdivides the image into tiles for enhanced small object detection. By integrating altitude-dependent scaling with an adaptive tiling factor, we reduce unnecessary computation while maintaining detection performance. Tested on the SeaDronesSee dataset [1] with YOLOv5 [2] and Slicing Aided Hyper Inference (SAHI) framework [3], our approach improves Mean Average Precision (mAP) for small objects by 38% compared to a baseline and achieves more than double the inference speed compared to static tiling. This approach enables more efficient and accurate UAV-based SAR operations under diverse conditions.
Authors Sakib Ahmed, Oscar Pizarro
Comment This is the author's accepted version of an article that has been published by IEEE. The final published version is available at IEEE Xplore
Categories Computer Vision and Pattern Recognition
0
The Selective Disk Bispectrum and Its Inversion, with Application to Multi-Reference Alignment
Title CN 选择性圆盘双谱及其逆变换：在多参考对齐中的应用
Keywords 旋转不变表示 · 双谱 · 形状分析 · 多参考对齐 · 信号处理
Summary 本文提出了选择性圆盘双谱，一种用于图像形状分析的快速旋转不变表示方法，并推导了其显式逆变换，解决了传统双谱方法不可逆和计算复杂度高的问题，使其可用于多参考对齐任务。该方法聚焦于旋转不变特征提取，适用于形状分析场景。
Reason 该论文研究的是图像形状分析中的旋转不变表示，属于传统计算机视觉中的信号处理方法，主要面向的是形状分析与多参考对齐任务，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的方法具有理论价值，但无法迁移解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU关键痛点，且不属于VLM或多模态大模型的技术演进路线。
Abstract In many computer vision and shape analysis tasks, practitioners are interested in learning from the shape of the object in an image, while disregarding the object's orientation. To this end, it is valuable to define a rotation-invariant representation of images, retaining all information about that image, but disregarding the way an object is rotated in the frame. To be practical for learning tasks, this representation must be computationally efficient for large datasets and invertible, so the representation can be visualized in image space. To this end, we present the selective disk bispectrum: a fast, rotation-invariant representation for image shape analysis. While the translational bispectrum has long been used as a translational invariant representation for 1-D and 2-D signals, its extension to 2-D (disk) rotational invariance on images has been hindered by the absence of an invertible formulation and its cubic complexity. In this work, we derive an explicit inverse for the disk bispectrum, which allows us to define a "selective" disk bispectrum, which only uses the minimal number of coefficients needed for faithful shape recovery. We show that this representation enables multi-reference alignment for rotated images-a task previously intractable for disk bispectrum methods. These results establish the disk bispectrum as a practical and theoretically grounded tool for learning on rotation-invariant shape data.
Authors Adele Myers, Nina Miolane
Categories Computer Vision and Pattern Recognition
0
3D-Guided Scalable Flow Matching for Generating Volumetric Tissue Spatial Transcriptomics from Serial Histology
Title CN 基于3D引导的可扩展流匹配从连续组织学生成体积性组织空间转录组
Keywords 3D重建 · 空间转录组 · 组织学图像 · 流匹配 · 基因表达预测
Summary 本文提出HoloTea，一种结合3D结构信息的流匹配框架，用于从H&E染色组织切片中推断空间基因表达。通过跨切片形态对齐与ControlNet融合上下文，并引入符合计数特性的先验分布，实现可扩展的3D组织建模。实验表明其在多个数据集上优于2D和3D基线方法。
Reason 该论文属于生物医学图像分析领域，研究的是从组织切片图像预测3D空间基因表达，核心任务为生物成像与基因组建模，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法聚焦于3D组织重建和基因数据生成，不涉及OCR、布局分析、表格识别或多模态语言模型在文档上的应用。技术路线无法迁移至解决文档图像中的幻觉、密集文本、复杂排版或长文档推理等核心痛点。
Abstract A scalable and robust 3D tissue transcriptomics profile can enable a holistic understanding of tissue organization and provide deeper insights into human biology and disease. Most predictive algorithms that infer ST directly from histology treat each section independently and ignore 3D structure, while existing 3D-aware approaches are not generative and do not scale well. We present Holographic Tissue Expression Inpainting and Analysis (HoloTea), a 3D-aware flow-matching framework that imputes spot-level gene expression from H&E while explicitly using information from adjacent sections. Our key idea is to retrieve morphologically corresponding spots on neighboring slides in a shared feature space and fuse this cross section context into a lightweight ControlNet, allowing conditioning to follow anatomical continuity. To better capture the count nature of the data, we introduce a 3D-consistent prior for flow matching that combines a learned zero-inflated negative binomial (ZINB) prior with a spatial-empirical prior constructed from neighboring sections. A global attention block introduces 3D H&E scaling linearly with the number of spots in the slide, enabling training and inference on large 3D ST datasets. Across three spatial transcriptomics datasets spanning different tissue types and resolutions, HoloTea consistently improves 3D expression accuracy and generalization compared to 2D and 3D baselines. We envision HoloTea advancing the creation of accurate 3D virtual tissues, ultimately accelerating biomarker discovery and deepening our understanding of disease.
Authors Mohammad Vali Sanian, Arshia Hemmat, Amirhossein Vahidi, Jonas Maaskola, Jimmy Tsz Hang Lee, Stanislaw Makarchuk, Yeliz Demirci, Nana-Jane Chipampe, Muzlifah Haniffa, Omer Bayraktar, Lassi Paavolainen, Mohammad Lotfollahi
Comment 19 pages
Categories Computer Vision and Pattern Recognition
0
OncoVision: Integrating Mammography and Clinical Data through Attention-Driven Multimodal AI for Enhanced Breast Cancer Diagnosis
Title CN OncoVision：通过注意力驱动的多模态AI整合乳腺X线摄影和临床数据以增强乳腺癌诊断
Keywords 多模态AI · 乳腺癌诊断 · 医学图像分割 · 晚期融合 · 临床决策支持
Summary OncoVision是一个结合乳腺X线图像与临床数据的多模态AI系统，采用注意力机制的编码器-解码器结构进行病灶分割与临床特征预测，并通过晚期融合策略提升诊断精度。系统已部署为带可视化功能的Web应用，支持结构化报告生成，旨在辅助医生决策并促进基层医疗普及。
Reason 该论文属于典型的‘平行下游应用’，即将现有VLM或多模态模型应用于医疗影像（乳腺X光）与临床数据融合的特定场景。尽管其在诊断流程和系统部署上有所优化，但未提出针对文档图像理解（DIU）的核心技术改进，如视觉压缩、OCR-free架构、去幻觉机制或复杂排版建模等。任务聚焦于医学图像分割与分类，而非文档中的文本结构理解、逻辑推理或VIE/KIE等DIU核心问题。
Abstract OncoVision is a multimodal AI pipeline that combines mammography images and clinical data for better breast cancer diagnosis. Employing an attention-based encoder-decoder backbone, it jointly segments four ROIs - masses, calcifications, axillary findings, and breast tissues - with state-of-the-art accuracy and robustly predicts ten structured clinical features: mass morphology, calcification type, ACR breast density, and BI-RADS categories. To fuse imaging and clinical insights, we developed two late-fusion strategies. By utilizing complementary multimodal data, late fusion strategies improve diagnostic precision and reduce inter-observer variability. Operationalized as a secure, user-friendly web application, OncoVision produces structured reports with dual-confidence scoring and attention-weighted visualizations for real-time diagnostic support to improve clinician trust and facilitate medical teaching. It can be easily incorporated into the clinic, making screening available in underprivileged areas around the world, such as rural South Asia. Combining accurate segmentation with clinical intuition, OncoVision raises the bar for AI-based mammography, offering a scalable and equitable solution to detect breast cancer at an earlier stage and enhancing treatment through timely interventions.
Authors Istiak Ahmed, Galib Ahmed, K. Shahriar Sanjid, Md. Tanzim Hossain, Md. Nishan Khan, Md. Misbah Khan, Md. Arifur Rahman, Sheikh Anisul Haque, Sharmin Akhtar Rupa, Mohammed Mejbahuddin Mia, Mahmud Hasan Mostofa Kamal, Md. Mostafa Kamal Sarker, M. Monir Uddin
Categories Computer Vision and Pattern Recognition
0
Endoshare: A Publicly Available, Surgeons-Friendly Solution to De-Identify and Manage Surgical Videos
Title CN Endoshare：一种公开可用的、对医生友好的手术视频去识别化与管理解决方案
Keywords 手术视频 · 去识别化 · 视频管理 · 隐私保护 · 医疗AI
Summary 本文提出了Endoshare，一个面向外科医生的手术视频管理工具，支持视频合并、标准化和去识别化处理。系统遵循以用户为中心的设计流程，经过多轮临床医生和计算机科学家的评估，显示出良好的可用性和接受度。尽管具有实际应用价值，但其技术内容局限于特定医疗场景的工程实现，无底层方法创新。
Reason 该论文属于医疗领域的视频处理应用，聚焦于手术视频的去隐私化与管理，属于特定垂直场景的应用系统开发。其核心贡献是软件工具链与可用性设计，并未提出任何文档图像理解（DIU）所需的方法论创新，如视觉表示学习、多模态推理、OCR-free建模或幻觉抑制机制。完全符合‘平行下游应用’的负面清单定义。
Abstract Video-based assessment and surgical data science can advance surgical training, research, and quality improvement, yet adoption remains limited by heterogeneous recording formats and privacy concerns linked to video sharing. This work develops, evaluates, and publicly releases Endoshare, a surgeon-friendly application that merges, standardizes, and de-identifies endoscopic videos. Development followed an iterative, user-centered software life cycle. In the analysis phase, an internal survey of four clinicians and four computer scientists, based on 10 usability heuristics, identified early requirements and guided a cross-platform, privacy-by-design architecture. Prototype testing reported high usability for clinicians (4.68 +/- 0.40 out of 5) and for computer scientists (4.03 +/- 0.51 out of 5), with the lowest score (4.00 +/- 0.93 out of 5) relating to label clarity, prompting interface refinement to streamline case selection, video merging, automated out-of-body removal, and filename pseudonymization. In the testing phase, ten surgeons completed an external survey combining the same heuristics with Technology Acceptance Model constructs, reporting high perceived usefulness (5.07 +/- 1.75 out of 7), ease of use (5.15 +/- 1.71 out of 7), heuristic usability (4.38 +/- 0.48 out of 5), and strong recommendation likelihood (9.20 +/- 0.79 out of 10). A performance assessment across different hardware and configurations showed that processing time increased proportionally with video duration and was consistently lower in fast mode. Endoshare is a publicly available solution to manage surgical videos, with potential to support training, research, and quality improvement. Compliance certification and broader interoperability validation are needed to establish it as a reliable tool for surgical video management. The software is available at https://camma-public.github.io/Endoshare
Authors Lorenzo Arboit, Dennis N. Schneider, Britty Baby, Vinkle Srivastav, Pietro Mascagni, Nicolas Padoy
Comment 13 pages, 6 figures. Source-available software: https://camma-public.github.io/Endoshare/
Categories Computer Vision and Pattern Recognition
0
Stitch-a-Demo: Video Demonstrations from Multistep Descriptions
Title CN Stitch-a-Demo：从多步骤描述生成视频演示
Keywords 视频生成 · 多步骤推理 · 检索-based合成 · instructional视频 · 视觉连贯性
Summary 本文提出Stitch-a-Demo，一种基于检索的方法，用于从多步骤描述（如菜谱或园艺指南）中构建连贯的视频演示。通过构建大规模弱监督训练数据并引入难负样本，提升生成视频在准确性和视觉连贯性上的表现。实验表明该方法在真实场景 instructional 视频数据上显著优于现有方法。
Reason 该论文研究的是从多步骤文本描述中生成连贯的视频演示，属于视频理解与生成领域，核心任务为 instructional video assembly，与文档图像理解（DIU）或DocVQA无关。尽管涉及多步推理，但其目标场景、数据模态（视频片段检索拼接）和应用领域均远离文档结构、文本识别或视觉问答等DIU核心问题。属于无关领域。
Abstract When obtaining visual illustrations from text descriptions, today's methods take a description with a single text context - a caption, or an action description - and retrieve or generate the matching visual context. However, prior work does not permit visual illustration of multistep descriptions, e.g. a cooking recipe or a gardening instruction manual, and simply handling each step description in isolation would result in an incoherent demonstration. We propose Stitch-a-Demo, a novel retrieval-based method to assemble a video demonstration from a multistep description. The resulting video contains clips, possibly from different sources, that accurately reflect all the step descriptions, while being visually coherent. We formulate a training pipeline that creates large-scale weakly supervised data containing diverse procedures and injects hard negatives that promote both correctness and coherence. Validated on in-the-wild instructional videos, Stitch-a-Demo achieves state-of-the-art performance, with gains up to 29% as well as dramatic wins in a human preference study.
Authors Chi Hsuan Wu, Kumar Ashutosh, Kristen Grauman
Categories Computer Vision and Pattern Recognition
0
SkillSight: Efficient First-Person Skill Assessment with Gaze
Title CN 基于注视的高效第一人称技能评估
Keywords 第一人称视觉 · 技能评估 · 注视追踪 · 模型蒸馏 · 低功耗感知
Summary 本文提出SkillSight，一种用于第一人称技能评估的节能框架。该方法通过联合建模注视与第一人称视频来预测技能水平，并训练一个仅依赖注视输入的学生模型，在大幅降低功耗的同时保持较高准确率。实验验证了注视信号在烹饪、音乐和体育等多个场景中对技能理解的价值。
Reason 该论文研究的是第一人称视角下的技能评估问题，核心是利用注视（gaze）信号替代视频输入以实现低功耗技能判断。其任务目标与文档图像理解（DIU）无关，属于具身智能与可穿戴计算交叉领域，不涉及DocVQA、布局分析、表格识别等文档理解任务，也无法迁移用于解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract Egocentric perception on smart glasses could transform how we learn new skills in the physical world, but automatic skill assessment remains a fundamental technical challenge. We introduce SkillSight for power-efficient skill assessment from first-person data. Central to our approach is the hypothesis that skill level is evident not only in how a person performs an activity (video), but also in how they direct their attention when doing so (gaze). Our two-stage framework first learns to jointly model gaze and egocentric video when predicting skill level, then distills a gaze-only student model. At inference, the student model requires only gaze input, drastically reducing power consumption by eliminating continuous video processing. Experiments on three datasets spanning cooking, music, and sports establish, for the first time, the valuable role of gaze in skill understanding across diverse real-world settings. Our SkillSight teacher model achieves state-of-the-art performance, while our gaze-only student variant maintains high accuracy using 73x less power than competing methods. These results pave the way for in-the-wild AI-supported skill learning.
Authors Chi Hsuan Wu, Kumar Ashutosh, Kristen Grauman
Categories Computer Vision and Pattern Recognition
0
MHR: Momentum Human Rig
Title CN MHR：动量人体绑定
Keywords 参数化人体模型 · 姿态修正 · AR/VR · 图形学 · 3D人体动画
Summary 本文提出MHR，一种结合ATLAS骨架/形状解耦范式和基于Momentum库的灵活绑定与姿态修正系统的参数化人体模型，支持非线性姿态修正，适用于AR/VR和图形渲染流程。
Reason 该论文属于3D人体建模与动画领域，研究内容为人体参数化模型和姿态修正系统，主要应用于AR/VR和图形渲染管线。与文档图像理解（DIU）、视觉语言模型（VLM）在文档上的应用、OCR、布局分析、表格识别等核心任务完全无关，属于明确列出的‘无关领域’（具身智能/3D视觉）范畴。
Abstract We present MHR, a parametric human body model that combines the decoupled skeleton/shape paradigm of ATLAS with a flexible, modern rig and pose corrective system inspired by the Momentum library. Our model enables expressive, anatomically plausible human animation, supporting non-linear pose correctives, and is designed for robust integration in AR/VR and graphics pipelines.
Authors Aaron Ferguson, Ahmed A. A. Osman, Berta Bescos, Carsten Stoll, Chris Twigg, Christoph Lassner, David Otte, Eric Vignola, Fabian Prada, Federica Bogo, Igor Santesteban, Javier Romero, Jenna Zarate, Jeongseok Lee, Jinhyung Park, Jinlong Yang, John Doublestein, Kishore Venkateshan, Kris Kitani, Ladislav Kavan, Marco Dal Farra, Matthew Hu, Matthew Cioffi, Michael Fabris, Michael Ranieri, Mohammad Modarres, Petr Kadlecek, Rawal Khirodkar, Rinat Abdrashitov, Romain Prévost, Roman Rajbhandari, Ronald Mallet, Russell Pearsall, Sandy Kao, Sanjeev Kumar, Scott Parrish, Shoou-I Yu, Shunsuke Saito, Takaaki Shiratori, Te-Li Wang, Tony Tung, Yichen Xu, Yuan Dong, Yuhua Chen, Yuanlu Xu, Yuting Ye, Zhongshi Jiang
Categories Computer Vision and Pattern Recognition
0
Learning Massively Multitask World Models for Continuous Control
Title CN 为连续控制学习大规模多任务世界模型
Keywords 世界模型 · 多任务学习 · 连续控制 · 强化学习 · 具身智能
Summary 本文提出了Newt，一种语言条件的多任务世界模型，通过在200个多样化连续控制任务上进行预训练和在线交互联合优化，实现跨任务的高效控制与快速泛化。工作聚焦于强化学习中的通用控制问题，配套发布了包含指令、演示和图像观测的新基准。
Reason 该论文属于具身智能与强化学习领域，研究的是连续控制任务下的多任务世界模型，其核心问题和方法均与文档图像理解（DIU）无关。尽管提到了语言条件和多模态输入，但目标是机器人控制而非文档解析或视觉语言推理。无法迁移解决OCR幻觉、复杂排版或长文档理解等DIU核心痛点。
Abstract General-purpose control demands agents that act across many tasks and embodiments, yet research on reinforcement learning (RL) for continuous control remains dominated by single-task or offline regimes, reinforcing a view that online RL does not scale. Inspired by the foundation model recipe (large-scale pretraining followed by light RL) we ask whether a single agent can be trained on hundreds of tasks with online interaction. To accelerate research in this direction, we introduce a new benchmark with 200 diverse tasks spanning many domains and embodiments, each with language instructions, demonstrations, and optionally image observations. We then present \emph{Newt}, a language-conditioned multitask world model that is first pretrained on demonstrations to acquire task-aware representations and action priors, and then jointly optimized with online interaction across all tasks. Experiments show that Newt yields better multitask performance and data-efficiency than a set of strong baselines, exhibits strong open-loop control, and enables rapid adaptation to unseen tasks. We release our environments, demonstrations, code for training and evaluation, as well as 200+ checkpoints.
Authors Nicklas Hansen, Hao Su, Xiaolong Wang
Comment Webpage: https://www.nicklashansen.com/NewtWM
Categories Computer Vision and Pattern Recognition
0
Multiscale Vector-Quantized Variational Autoencoder for Endoscopic Image Synthesis IEEE Int. Conf. Imaging Systems and Techniques (IST 2025)
Title CN 用于内窥镜图像合成的多尺度向量量化变分自编码器
Keywords 医学图像生成 · 变分自编码器 · 向量量化 · 多尺度建模 · 临床决策支持
Summary 本文提出了一种名为MSVQ-VAE的多尺度向量量化变分自编码器，用于无线胶囊内窥镜（WCE）图像的合成，能够将异常病灶（如息肉、血管和炎症病变）条件化地植入正常图像中，以增强临床决策支持系统的训练数据。实验表明，使用生成图像训练的分类器性能接近使用真实数据训练的模型。
Reason 该论文属于纯图像生成领域，研究的是医学内窥镜图像的合成方法，旨在通过生成对抗手段缓解医疗数据稀缺问题。其核心任务与文档图像理解（DIU）无关，既不涉及OCR、布局分析、表格识别，也不处理文档中的多模态推理或VLM应用。尽管使用了VQ-VAE等表示学习技术，但目标是图像生成而非文档结构建模或视觉语言理解，无法迁移解决DIU中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Gastrointestinal (GI) imaging via Wireless Capsule Endoscopy (WCE) generates a large number of images requiring manual screening. Deep learning-based Clinical Decision Support (CDS) systems can assist screening, yet their performance relies on the existence of large, diverse, training medical datasets. However, the scarcity of such data, due to privacy constraints and annotation costs, hinders CDS development. Generative machine learning offers a viable solution to combat this limitation. While current Synthetic Data Generation (SDG) methods, such as Generative Adversarial Networks and Variational Autoencoders have been explored, they often face challenges with training stability and capturing sufficient visual diversity, especially when synthesizing abnormal findings. This work introduces a novel VAE-based methodology for medical image synthesis and presents its application for the generation of WCE images. The novel contributions of this work include a) multiscale extension of the Vector Quantized VAE model, named as Multiscale Vector Quantized Variational Autoencoder (MSVQ-VAE); b) unlike other VAE-based SDG models for WCE image generation, MSVQ-VAE is used to seamlessly introduce abnormalities into normal WCE images; c) it enables conditional generation of synthetic images, enabling the introduction of different types of abnormalities into the normal WCE images; d) it performs experiments with a variety of abnormality types, including polyps, vascular and inflammatory conditions. The utility of the generated images for CDS is assessed via image classification. Comparative experiments demonstrate that training a CDS classifier using the abnormal images generated by the proposed methodology yield comparable results with a classifier trained with only real data. The generality of the proposed methodology promises its applicability to various domains related to medical multimedia.
Authors Dimitrios E. Diamantis, Dimitris K. Iakovidis
Comment Accepted in IEEE Int. Conf. Imaging Systems and Techniques (IST 2025), Strasburg, France
Categories Computer Vision and Pattern Recognition
0
Leveraging Unlabeled Scans for NCCT Image Segmentation in Early Stroke Diagnosis: A Semi-Supervised GAN Approach
Title CN 利用未标记扫描图像进行早期中风诊断中的NCCT图像分割：一种半监督GAN方法
Keywords 医学图像分割 · 半监督学习 · GAN · 早期中风诊断 · NCCT
Summary 本文提出了一种基于生成对抗网络（GAN）的半监督方法，用于在非增强CT（NCCT）图像中分割早期缺血性中风区域。该方法结合Dice损失、交叉熵损失、特征匹配损失和自训练策略，在标注数据有限的情况下有效利用大量未标注扫描图像，提升模型对细微病灶的识别能力。实验在公开的急性缺血性中风数据集（AISD）上进行，结果表明该方法有助于提高早期诊断准确性。
Reason 该论文属于医学图像分析领域，研究任务为脑部CT图像中缺血性中风区域的语义分割，属于纯医疗影像理解范畴，与文档图像理解（DIU）、DocVQA、OCR、版面分析等核心任务无关。尽管使用了半监督和GAN方法，但其技术路径针对的是生物医学信号增强与病灶检测，无法直接迁移解决文档图像中的幻觉、复杂排版或长文档推理等问题。
Abstract Ischemic stroke is a time-critical medical emergency where rapid diagnosis is essential for improving patient outcomes. Non-contrast computed tomography (NCCT) serves as the frontline imaging tool, yet it often fails to reveal the subtle ischemic changes present in the early, hyperacute phase. This limitation can delay crucial interventions. To address this diagnostic challenge, we introduce a semi-supervised segmentation method using generative adversarial networks (GANs) to accurately delineate early ischemic stroke regions. The proposed method employs an adversarial framework to effectively learn from a limited number of annotated NCCT scans, while simultaneously leveraging a larger pool of unlabeled scans. By employing Dice loss, cross-entropy loss, a feature matching loss and a self-training loss, the model learns to identify and delineate early infarcts, even when they are faint or their size is small. Experiments on the publicly available Acute Ischemic Stroke Dataset (AISD) demonstrate the potential of the proposed method to enhance diagnostic capabilities, reduce the burden of manual annotation, and support more efficient clinical decision-making in stroke care.
Authors Maria Thoma, Michalis A. Savelonas, Dimitris K. Iakovidis
Categories Computer Vision and Pattern Recognition
0
FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models
Title CN FedPromo：边缘端的联邦轻量级代理模型为基础模型带来新领域
Keywords 联邦学习 · 知识蒸馏 · 轻量模型 · 基础模型适应 · 边缘计算
Summary 本文提出FedPromo，一种用于联邦学习的新型框架，通过在服务器端进行知识蒸馏，将大型基础模型的知识迁移到轻量级代理模型，并在边缘设备上训练分类器，再将结果聚合回原模型。该方法旨在解决资源受限设备上的模型训练问题，强调隐私保护和多域学习能力，实验验证于图像分类任务。
Reason 该论文属于联邦学习在图像分类中的应用，研究的是如何在资源受限的客户端通过轻量代理模型实现大模型的域适应。虽然涉及视觉与模型效率，但其核心任务是通用图像分类的联邦学习框架，与文档图像理解（DIU）如DocVQA、布局分析、表格识别等无直接关联。方法上未针对OCR幻觉、复杂排版、高分辨率文档处理或长文档推理等DIU痛点提供可迁移的技术路径，属于平行应用而非底层方法创新。
Abstract Federated Learning (FL) is an established paradigm for training deep learning models on decentralized data. However, as the size of the models grows, conventional FL approaches often require significant computational resources on client devices, which may not be feasible. We introduce FedPromo, a novel framework that enables efficient adaptation of large-scale foundation models stored on a central server to new domains encountered only by remote clients. Instead of directly training the large model on client devices, FedPromo optimizes lightweight proxy models via FL, significantly reducing computational overhead while maintaining privacy. Our method follows a two-stage process: first, server-side knowledge distillation aligns the representations of a large-scale foundation model (e.g., a transformer) with those of a compact counterpart (e.g., a CNN). Then, the compact model encoder is deployed to client devices, where trainable classifiers are learned locally. These classifiers are subsequently aggregated and seamlessly transferred back to the foundation model, facilitating personalized adaptation without requiring direct access to user data. Through novel regularization strategies, our framework enables decentralized multi-domain learning, balancing performance, privacy, and resource efficiency. Extensive experiments on five image classification benchmarks demonstrate that FedPromo outperforms existing methods while assuming limited-resource clients.
Authors Matteo Caligiuri, Francesco Barbato, Donald Shenaj, Umberto Michieli, Pietro Zanuttigh
Comment 8 pages (main document) + 13 pages (suppl. mat.), 4 figures (main) + 11 figures (suppl. mat.), 6 tables (main) + 5 tables (suppl. mat.) + 4 algorithms (suppl. mat.)
Categories Computer Vision and Pattern Recognition
0
Development of a fully deep learning model to improve the reproducibility of sector classification systems for predicting unerupted maxillary canine likelihood of impaction
Title CN 开发一种全深度学习模型以提高预测未萌出上颌尖牙阻生可能性的象限分类系统的可重复性
Keywords 医学影像 · 深度学习 · 分类可重复性 · 口腔正畸 · 放射图像
Summary 本研究旨在通过深度学习模型提升医生在放射影像中对未萌出上颌尖牙位置分类的一致性。研究人员比较了不同分类系统在人工判读下的可重复性，并测试了多个预训练AI模型的表现，最终发现DenseNet121在三分类系统中表现最佳。然而，该工作仅为医学图像的分类应用，未涉及文档结构理解或多模态语言模型等DIU关键技术。
Reason 该论文属于医学影像分析中的特定临床应用（口腔正畸），其核心是将现有AI模型（如DenseNet121）应用于一个具体的医疗分类任务，属于典型的‘平行下游应用’。尽管使用了深度学习和放射影像，但并未提出任何面向文档图像理解（DIU）的新架构、训练方法或视觉表示机制。研究目标与OCR、布局分析、表格识别、VIE等DIU核心任务无关，也无法为解决OCR幻觉、复杂排版或长文档推理等痛点提供可迁移的方法论。
Abstract Objectives. The aim of the present study was to develop a fully deep learning model to reduce the intra- and inter-operator reproducibility of sector classification systems for predicting unerupted maxillary canine likelihood of impaction. Methods. Three orthodontists (Os) and three general dental practitioners (GDPs) classified the position of unerupted maxillary canines on 306 radiographs (T0) according to the three different sector classification systems (5-, 4-, and 3-sector classification system). The assessment was repeated after four weeks (T1). Intra- and inter-observer agreement were evaluated with Cohen's K and Fleiss K, and between group differences with a z-test. The same radiographs were tested on different artificial intelligence (AI) models, pre-trained on an extended dataset of 1,222 radiographs. The best-performing model was identified based on its sensitivity and precision. Results. The 3-sector system was found to be the classification method with highest reproducibility, with an agreement (Cohen's K values) between observations (T0 versus T1) for each examiner ranged from 0.80 to 0.92, and an overall agreement of 0.85 [95% confidence interval (CI) = 0.83-0.87]. The overall inter-observer agreement (Fleiss K) ranged from 0.69 to 0.7. The educational background did not affect either intra- or inter-observer agreement (p>0.05). DenseNet121 proved to be the best-performing model in allocating impacted canines in the three different classes, with an overall accuracy of 76.8%. Conclusion. AI models can be designed to automatically classify the position of unerupted maxillary canines.
Authors Marzio Galdi, Davide Cannatà, Flavia Celentano, Luigia Rizzo, Domenico Rossi, Tecla Bocchino, Stefano Martina
Categories Computer Vision and Pattern Recognition
0
GMT: Effective Global Framework for Multi-Camera Multi-Target Tracking
Title CN GMT：用于多摄像头多目标跟踪的有效全局框架
Keywords 多目标跟踪 · 多摄像头 · 跨视角匹配 · 全局轨迹关联 · 计算机视觉
Summary 本文提出了一种名为GMT的全局多摄像头多目标跟踪框架，通过引入跨视角特征一致性增强模块（CFCE）和全局轨迹关联模块（GTA），将传统的两阶段跟踪范式转化为统一的全局轨迹-目标关联过程，显著提升了多摄像头场景下的跟踪性能。同时发布了大规模数据集VisionTrack。
Reason 该论文研究的是多摄像头多目标跟踪（MCMT），属于视频监控与行人重识别领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其提出的方法如跨视角特征对齐和全局轨迹关联，虽然在视频分析中有价值，但难以迁移到解决OCR幻觉、复杂排版理解或长文档推理等DIU核心问题。
Abstract Multi-Camera Multi-Target (MCMT) tracking aims to locate and associate the same targets across multiple camera views. Existing methods typically adopt a two-stage framework, involving single-camera tracking followed by inter-camera tracking. However, in this paradigm, multi-view information is used only to recover missed matches in the first stage, providing a limited contribution to overall tracking. To address this issue, we propose GMT, a global MCMT tracking framework that jointly exploits intra-view and inter-view cues for tracking. Specifically, instead of assigning trajectories independently for each view, we integrate the same historical targets across different views as global trajectories, thereby reformulating the two-stage tracking as a unified global-level trajectory-target association process. We introduce a Cross-View Feature Consistency Enhancement (CFCE) module to align visual and spatial features across views, providing a consistent feature space for global trajectory modeling. With these aligned features, the Global Trajectory Association (GTA) module associates new detections with existing global trajectories, enabling direct use of multi-view information. Compared to the two-stage framework, GMT achieves significant improvements on existing datasets, with gains of up to 21.3 percent in CVMA and 17.2 percent in CVIDF1. Furthermore, we introduce VisionTrack, a high-quality, large-scale MCMT dataset providing significantly greater diversity than existing datasets. Our code and dataset will be released.
Authors Yihao Zhen, Mingyue Xu, Qiang Wang, Baojie Fan, Jiahua Dong, Tinghui Zhao, Huijie Fan
Categories Computer Vision and Pattern Recognition
0
Transforming Hyperspectral Images Into Chemical Maps: A Novel End-to-End Deep Learning Approach
Title CN 将高光谱图像转化为化学图：一种新颖的端到端深度学习方法
Keywords 高光谱图像 · 化学图 · U-Net · 深度学习 · 像素级回归
Summary 本文提出了一种基于改进U-Net和自定义损失函数的端到端深度学习方法，用于从高光谱图像直接生成化学图，避免传统PLS回归方法缺乏空间一致性和超出物理范围的问题。实验表明该方法在猪肉脂肪预测任务中比PLS误差更低，且生成的化学图具有高度空间相关性。
Reason 该论文研究的是高光谱图像到化学分布图的生成问题，属于特定科学成像领域的应用（如食品分析），其方法聚焦于U-Net在像素级回归任务中的改进，并未涉及文档结构、文本理解、多模态语言模型或OCR相关挑战。与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、长文档推理）无关，也不提供可迁移至DIU领域的通用架构或推理机制创新。
Abstract Current approaches to chemical map generation from hyperspectral images are based on models such as partial least squares (PLS) regression, generating pixel-wise predictions that do not consider spatial context and suffer from a high degree of noise. This study proposes an end-to-end deep learning approach using a modified version of U-Net and a custom loss function to directly obtain chemical maps from hyperspectral images, skipping all intermediate steps required for traditional pixel-wise analysis. This study compares the U-Net with the traditional PLS regression on a real dataset of pork belly samples with associated mean fat reference values. The U-Net obtains a test set root mean squared error that is 7% lower than that of PLS regression on the task of mean fat prediction. At the same time, U-Net generates fine detail chemical maps where 99.91% of the variance is spatially correlated. Conversely, only 2.37% of the variance in the PLS-generated chemical maps is spatially correlated, indicating that each pixel-wise prediction is largely independent of neighboring pixels. Additionally, while the PLS-generated chemical maps contain predictions far beyond the physically possible range of 0-100%, U-Net learns to stay inside this range. Thus, the findings of this study indicate that U-Net is superior to PLS for chemical map generation.
Authors Ole-Christian Galbo Engstrøm, Michela Albano-Gaglio, Erik Schou Dreier, Yamine Bouzembrak, Maria Font-i-Furnols, Puneet Mishra, Kim Steenstrup Pedersen
Categories Computer Vision and Pattern Recognition
0
Proxy-Free Gaussian Splats Deformation with Splat-Based Surface Estimation 3DV 2026
Title CN 基于点阵表面估计的无代理高斯点阵形变
Keywords 高斯点阵 · 3D形变 · Laplacian算子 · 表面估计 · 无代理方法
Summary 本文提出SpLap，一种无需依赖笼状或网格代理的高斯点阵形变方法，通过构建一种新的表面感知点阵图来计算Laplacian算子，从而实现更合理的形变并保持细节与拓扑结构。方法利用点阵间的空间交集定义邻接关系，并引入高斯核自适应技术以提升形变后的渲染质量。实验表明其在多个3D数据集上优于现有方法。
Reason 该论文属于3D视觉与几何建模领域，研究的是高斯点阵（Gaussian Splats）在形变过程中的表面保持问题，核心方法围绕Laplacian算子和点阵图结构展开。其技术目标与文档图像理解（DIU）的核心痛点如OCR幻觉、文本识别、布局分析、表格理解等完全无关。尽管涉及视觉表示，但迁移至文档理解场景的可能性极低，属于典型无关领域。
Abstract We introduce SpLap, a proxy-free deformation method for Gaussian splats (GS) based on a Laplacian operator computed from our novel surface-aware splat graph. Existing approaches to GS deformation typically rely on deformation proxies such as cages or meshes, but they suffer from dependency on proxy quality and additional computational overhead. An alternative is to directly apply Laplacian-based deformation techniques by treating splats as point clouds. However, this often fail to properly capture surface information due to lack of explicit structure. To address this, we propose a novel method that constructs a surface-aware splat graph, enabling the Laplacian operator derived from it to support more plausible deformations that preserve details and topology. Our key idea is to leverage the spatial arrangement encoded in splats, defining neighboring splats not merely by the distance between their centers, but by their intersections. Furthermore, we introduce a Gaussian kernel adaptation technique that preserves surface structure under deformation, thereby improving rendering quality after deformation. In our experiments, we demonstrate the superior performance of our method compared to both proxy-based and proxy-free baselines, evaluated on 50 challenging objects from the ShapeNet, Objaverse, and Sketchfab datasets, as well as the NeRF-Synthetic dataset. Code is available at https://github.com/kjae0/SpLap.
Authors Jaeyeong Kim, Seungwoo Yoo, Minhyuk Sung
Comment 17 pages, Accepted to 3DV 2026 (IEEE/CVF International Conference on 3D Vision)
Categories Computer Vision and Pattern Recognition
0
PhysDNet: Physics-Guided Decomposition Network of Side-Scan Sonar Imagery
Title CN PhysDNet：面向侧扫声呐图像的物理引导分解网络
Keywords 侧扫声呐图像 · 物理引导模型 · 图像分解 · 自监督学习 · 水下遥感
Summary 本文提出PhysDNet，一种基于物理引导的多分支网络，用于将侧扫声呐图像分解为海底反射率、地形高程和传播损耗三个可解释成分。通过引入朗伯反射模型，实现无需真实标注的自监督训练。实验表明，分解后的表征具有稳定的地质结构、符合物理规律的光照与衰减特性，并能生成可靠的阴影图，有助于提升配准和阴影解析等下游任务性能。
Reason 该论文研究领域为水下遥感成像处理，聚焦于侧扫声呐图像的物理特性建模与分解，属于遥感与海洋工程应用范畴。尽管其方法具有物理引导建模和自监督解耦表示的创新性，但任务目标（海底反射率、地形、声学衰减分离）与文档图像理解（DIU）的核心问题（如文本识别、布局分析、视觉问答）无直接关联。所解决的‘视点依赖性’问题与DIU中的OCR幻觉、复杂排版或长文档推理等痛点不匹配，且迁移路径不明确。
Abstract Side-scan sonar (SSS) imagery is widely used for seafloor mapping and underwater remote sensing, yet the measured intensity is strongly influenced by seabed reflectivity, terrain elevation, and acoustic path loss. This entanglement makes the imagery highly view-dependent and reduces the robustness of downstream analysis. In this letter, we present PhysDNet, a physics-guided multi-branch network that decouples SSS images into three interpretable fields: seabed reflectivity, terrain elevation, and propagation loss. By embedding the Lambertian reflection model, PhysDNet reconstructs sonar intensity from these components, enabling self-supervised training without ground-truth annotations. Experiments show that the decomposed representations preserve stable geological structures, capture physically consistent illumination and attenuation, and produce reliable shadow maps. These findings demonstrate that physics-guided decomposition provides a stable and interpretable domain for SSS analysis, improving both physical consistency and downstream tasks such as registration and shadow interpretation.
Authors Can Lei, Hayat Rajani, Nuno Gracias, Rafael Garcia, Huigang Wang
Comment This work was previously submitted in error as arXiv:2509.11255v2
Categories Computer Vision and Pattern Recognition
0
OpenScan: A Benchmark for Generalized Open-Vocabulary 3D Scene Understanding AAAI 2026
Title CN OpenScan：一种用于广义开放词汇3D场景理解的基准
Keywords 3D场景理解 · 开放词汇 · 基准测试 · 细粒度属性 · 多模态理解
Summary 本文提出了一个名为OpenScan的新基准，用于评估广义开放词汇3D场景理解（GOV-3D）能力，超越传统仅限于物体类别的设定，涵盖如功能、材质等八种语言维度的细粒度属性。实验表明现有方法在理解和泛化抽象词汇方面存在显著不足，揭示了当前技术的局限性并指出了未来研究方向。
Reason 该论文聚焦于3D场景理解，属于具身智能与机器人、3D视觉领域，与文档图像理解（DIU）无直接关联。其任务目标为开放词汇下的3D物体属性识别与场景理解，涉及affordance、material等语义层面，无法迁移解决文档中的OCR幻觉、复杂排版或长文本推理等问题。
Abstract Open-vocabulary 3D scene understanding (OV-3D) aims to localize and classify novel objects beyond the closed set of object classes. However, existing approaches and benchmarks primarily focus on the open vocabulary problem within the context of object classes, which is insufficient in providing a holistic evaluation to what extent a model understands the 3D scene. In this paper, we introduce a more challenging task called Generalized Open-Vocabulary 3D Scene Understanding (GOV-3D) to explore the open vocabulary problem beyond object classes. It encompasses an open and diverse set of generalized knowledge, expressed as linguistic queries of fine-grained and object-specific attributes. To this end, we contribute a new benchmark named \textit{OpenScan}, which consists of 3D object attributes across eight representative linguistic aspects, including affordance, property, and material. We further evaluate state-of-the-art OV-3D methods on our OpenScan benchmark and discover that these methods struggle to comprehend the abstract vocabularies of the GOV-3D task, a challenge that cannot be addressed simply by scaling up object classes during training. We highlight the limitations of existing methodologies and explore promising directions to overcome the identified shortcomings.
Authors Youjun Zhao, Jiaying Lin, Shuquan Ye, Qianshi Pang, Rynson W. H. Lau
Comment Accepted by AAAI 2026. Project Page: https://youjunzhao.github.io/OpenScan/
Categories Computer Vision and Pattern Recognition
0
Vidi2: Large Multimodal Models for Video Understanding and Creation
Title CN Vidi2：用于视频理解与创作的大型多模态模型
Keywords 视频理解 · 时空定位 · 视频问答 · 多模态模型 · 基准测试
Summary 本文介绍了Vidi2，一个在视频理解与创作方面取得进展的大型多模态模型，支持细粒度的时空定位和视频问答，并发布了新的评测基准VUE-STG和升级版VUE-TR-V2。模型在多个指标上超越主流闭源系统。
Reason 该论文聚焦于视频理解与生成任务，属于无关领域（视频理解/生成），不符合文档图像理解（DIU）的核心研究方向。尽管其提出的时空定位（STG）和视频问答能力在技术上有创新，但这些方法针对的是动态视频内容，难以直接迁移至文档图像中的静态结构化理解、OCR去幻觉或复杂排版分析等核心痛点。
Abstract Video has emerged as the primary medium for communication and creativity on the Internet, driving strong demand for scalable, high-quality video production. Vidi models continue to evolve toward next-generation video creation and have achieved state-of-the-art performance in multimodal temporal retrieval (TR). In its second release, Vidi2 advances video understanding with fine-grained spatio-temporal grounding (STG) and extends its capability to video question answering (Video QA), enabling comprehensive multimodal reasoning. Given a text query, Vidi2 can identify not only the corresponding timestamps but also the bounding boxes of target objects within the output time ranges. This end-to-end spatio-temporal grounding capability enables potential applications in complex editing scenarios, such as plot or character understanding, automatic multi-view switching, and intelligent, composition-aware reframing and cropping. To enable comprehensive evaluation of STG in practical settings, we introduce a new benchmark, VUE-STG, which offers four key improvements over existing STG datasets: 1) Video duration: spans from roughly 10s to 30 mins, enabling long-context reasoning; 2) Query format: queries are mostly converted into noun phrases while preserving sentence-level expressiveness; 3) Annotation quality: all ground-truth time ranges and bounding boxes are manually annotated with high accuracy; 4) Evaluation metric: a refined vIoU/tIoU/vIoU-Intersection scheme. In addition, we upgrade the previous VUE-TR benchmark to VUE-TR-V2, achieving a more balanced video-length distribution and more user-style queries. Remarkably, the Vidi2 model substantially outperforms leading proprietary systems, such as Gemini 3 Pro (Preview) and GPT-5, on both VUE-TR-V2 and VUE-STG, while achieving competitive results with popular open-source models with similar scale on video QA benchmarks.
Authors Vidi Team, Celong Liu, Chia-Wen Kuo, Chuang Huang, Dawei Du, Fan Chen, Guang Chen, Haoji Zhang, Haojun Zhao, Lingxi Zhang, Lu Guo, Lusha Li, Longyin Wen, Qihang Fan, Qingyu Chen, Rachel Deng, Sijie Zhu, Stuart Siew, Tong Jin, Weiyan Tao, Wen Zhong, Xiaohui Shen, Xin Gu, Zhenfang Chen, Zuhua Lin
Categories Computer Vision and Pattern Recognition
0
MapRF: Weakly Supervised Online HD Map Construction via NeRF-Guided Self-Training
Title CN MapRF：通过NeRF引导的自训练实现弱监督在线高精地图构建
Keywords NeRF · 自训练 · 高精地图 · 弱监督学习 · 自动驾驶
Summary 本文提出MapRF，一种弱监督框架，利用仅有的2D图像标签通过NeRF模块生成高质量的3D伪标签，用于在线高精地图构建。通过地图预测引导NeRF重建视图一致的三维几何与语义，并采用Map-to-Ray Matching策略减少自训练过程中的误差累积。在Argoverse 2和nuScenes数据集上表现接近全监督方法。
Reason 该论文属于自动驾驶领域的3D视觉任务，研究的是基于2D图像标签生成3D高精地图，核心方法围绕NeRF和自训练机制用于道路环境建模。尽管涉及视觉-几何联合建模，但其技术路线与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析或长文档推理无直接关联。无法迁移至文档场景中的多模态对齐、细粒度定位或视觉压缩等关键问题。
Abstract Autonomous driving systems benefit from high-definition (HD) maps that provide critical information about road infrastructure. The online construction of HD maps offers a scalable approach to generate local maps from on-board sensors. However, existing methods typically rely on costly 3D map annotations for training, which limits their generalization and scalability across diverse driving environments. In this work, we propose MapRF, a weakly supervised framework that learns to construct 3D maps using only 2D image labels. To generate high-quality pseudo labels, we introduce a novel Neural Radiance Fields (NeRF) module conditioned on map predictions, which reconstructs view-consistent 3D geometry and semantics. These pseudo labels are then iteratively used to refine the map network in a self-training manner, enabling progressive improvement without additional supervision. Furthermore, to mitigate error accumulation during self-training, we propose a Map-to-Ray Matching strategy that aligns map predictions with camera rays derived from 2D labels. Extensive experiments on the Argoverse 2 and nuScenes datasets demonstrate that MapRF achieves performance comparable to fully supervised methods, attaining around 75% of the baseline while surpassing several approaches using only 2D labels. This highlights the potential of MapRF to enable scalable and cost-effective online HD map construction for autonomous driving.
Authors Hongyu Lyu, Thomas Monninger, Julie Stephany Berrio Perez, Mao Shan, Zhenxing Ming, Stewart Worrall
Categories Computer Vision and Pattern Recognition
0
VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learning
Title CN VideoChat-M1：通过多智能体强化学习实现视频理解的协同策略规划
Keywords 多智能体系统 · 视频理解 · 强化学习 · 策略规划 · 多模态大模型
Summary 本文提出VideoChat-M1，一种基于多智能体强化学习的视频理解框架，通过协同策略规划（CPP）机制实现多个策略智能体之间的动态协作与策略更新。每个智能体生成并执行针对问题的工具调用策略，并在执行过程中与其他智能体通信以优化策略。结合最终答案奖励和中间协作反馈，系统实现了端到端的联合优化，在多个视频理解基准上达到SOTA性能。
Reason 该论文研究的是视频理解任务，使用多智能体强化学习进行策略规划，属于视频理解领域。尽管其多智能体协作机制具有方法论价值，但整体工作聚焦于视频模态中的时空推理，与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析等无直接关联。且其方法迁移至文档场景困难，不属于DIU本题或可迁移的‘军火库’范畴。
Abstract By leveraging tool-augmented Multimodal Large Language Models (MLLMs), multi-agent frameworks are driving progress in video understanding. However, most of them adopt static and non-learnable tool invocation mechanisms, which limit the discovery of diverse clues essential for robust perception and reasoning regarding temporally or spatially complex videos. To address this challenge, we propose a novel Multi-agent system for video understanding, namely VideoChat-M1. Instead of using a single or fixed policy, VideoChat-M1 adopts a distinct Collaborative Policy Planning (CPP) paradigm with multiple policy agents, which comprises three key processes. (1) Policy Generation: Each agent generates its unique tool invocation policy tailored to the user's query; (2) Policy Execution: Each agent sequentially invokes relevant tools to execute its policy and explore the video content; (3) Policy Communication: During the intermediate stages of policy execution, agents interact with one another to update their respective policies. Through this collaborative framework, all agents work in tandem, dynamically refining their preferred policies based on contextual insights from peers to effectively respond to the user's query. Moreover, we equip our CPP paradigm with a concise Multi-Agent Reinforcement Learning (MARL) method. Consequently, the team of policy agents can be jointly optimized to enhance VideoChat-M1's performance, guided by both the final answer reward and intermediate collaborative process feedback. Extensive experiments demonstrate that VideoChat-M1 achieves SOTA performance across eight benchmarks spanning four tasks. Notably, on LongVideoBench, our method outperforms the SOTA model Gemini 2.5 pro by 3.6% and GPT-4o by 15.6%.
Authors Boyu Chen, Zikang Wang, Zhengrong Yue, Kainan Yan, Chenyun Yu, Yi Huang, Zijun Liu, Yafei Wen, Xiaoxin Chen, Yang Liu, Peng Li, Yali Wang
Comment 21 pages, 9 figures
Categories Computer Vision and Pattern Recognition
0
MiniF2F in Rocq: Automatic Translation Between Proof Assistants -- A Case Study
Title CN MiniF2F在Rocq中的自动翻译：定理证明助手之间的翻译案例研究
Keywords 定理证明 · 形式化验证 · 大语言模型 · 程序翻译 · Lean
Summary 本文利用GPT-4o mini、Claude 3.5 Sonnet、o1系列等先进大模型，将MiniF2F数据集中的定理从Lean和Isabelle形式化语言翻译为Rocq语言。实验分为三个阶段，从单次提示到引入失败反馈的多轮对话，最终成功将488个定理中的478个完成翻译。数据集已开源。
Reason 该论文属于纯形式化方法与编程语言领域的LLM应用，研究的是不同证明助手（Lean、Isabelle、Rocq）之间的定理翻译问题。尽管使用了多阶段提示和反馈机制，但其任务本质是代码级形式化逻辑转换，并不涉及文档图像、视觉布局、OCR或任何多模态文档理解内容。无法迁移解决DIU中的OCR幻觉、复杂排版或细粒度定位等核心问题。
Abstract In this work, we conduct an experiment using state-of-the-art LLMs to translate MiniF2F into Rocq. The translation task focuses on generating a Rocq theorem based on three sources: a natural language description, the Lean formalization, and the Isabelle formalization. We conducted our experiment in 3 stages of increasing complexity, from basic one-shot prompting to multi-turn conversations that incorporate feedback from unsuccessful attempts. At each stage, we perform multiple rounds of translation using increasingly advanced models: GPT-4o mini, Claude 3.5 Sonnet, o1 mini, and o1. We successfully translated 478 out of 488 theorems. The dataset is opensource: https://github.com/LLM4Rocq/miniF2F-rocq.
Authors Jules Viennot, Guillaume Baudart, Emilio Jesùs Gallego Arias, Marc Lelarge
Categories Computation and Language
0
Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them Findings of the Association for Computational Linguistics: EMNLP 2025
Title CN 利用LLM生成数据增强领域特定编码器模型：如何利用本体以及如何在没有本体的情况下实现
Keywords LLM生成数据 · 领域适应 · 编码器模型 · 本体 · 低资源
Summary 本文研究了在数据稀缺的专业领域中，使用LLM生成数据对编码器模型进行持续预训练的方法，并以入侵生物学为例，探索了基于领域本体和无本体条件下的两种增强策略。结果表明，即使没有完整本体，仅用少量科学摘要也能达到与大规模掩码语言建模相当的效果。
Reason 该论文属于典型的‘平行下游应用’，其核心是将现有LLM生成数据和编码器模型结合应用于入侵生物学这一特定科学领域。尽管提出了使用自动提取概念替代本体的方法，但整体框架并未提出适用于文档图像理解（DIU）的通用方法论创新，如视觉压缩、多模态对齐、OCR去幻觉或复杂排版处理等。所提方法难以直接迁移至DIU任务，尤其是面对文档中的视觉结构、布局分析或图文联合推理等问题。
Abstract We investigate the use of LLM-generated data for continual pretraining of encoder models in specialized domains with limited training data, using the scientific domain of invasion biology as a case study. To this end, we leverage domain-specific ontologies by enriching them with LLM-generated data and pretraining the encoder model as an ontology-informed embedding model for concept definitions. To evaluate the effectiveness of this method, we compile a benchmark specifically designed for assessing model performance in invasion biology. After demonstrating substantial improvements over standard LLM pretraining, we investigate the feasibility of applying the proposed approach to domains without comprehensive ontologies by substituting ontological concepts with concepts automatically extracted from a small corpus of scientific abstracts and establishing relationships between concepts through distributional statistics. Our results demonstrate that this automated approach achieves comparable performance using only a small set of scientific abstracts, resulting in a fully automated pipeline for enhancing domain-specific understanding of small encoder models that is especially suited for application in low-resource settings and achieves performance comparable to masked language modeling pretraining on much larger datasets.
Authors Marc Brinner, Tarek Al Mustafa, Sina Zarrieß
Comment Published in the Findings of the Association for Computational Linguistics: EMNLP 2025
Categories Computation and Language
0
MultiBanAbs: A Comprehensive Multi-Domain Bangla Abstractive Text Summarization Dataset
Title CN MultiBanAbs：一个全面的多领域孟加拉语抽象文本摘要数据集
Keywords 孟加拉语 · 文本摘要 · 多领域 · 低资源语言 · 数据集
Summary 本文提出了一个包含5.4万多个样本的多领域孟加拉语抽象式文本摘要数据集，来源于博客、报纸等多种渠道，并使用LSTM、BanglaT5-small等模型建立了基线结果，旨在推动低资源语言的NLP发展。
Reason 该论文聚焦于孟加拉语这一小语种的文本摘要任务，属于小语种NLP数据集构建工作，不涉及文档图像理解（DIU）或多模态建模。尽管其数据构建方法有一定系统性，但完全基于纯文本处理，未涉及视觉信息、OCR、布局分析或VLM技术，且语言非中英，违反了‘小语种’负面清单。
Abstract This study developed a new Bangla abstractive summarization dataset to generate concise summaries of Bangla articles from diverse sources. Most existing studies in this field have concentrated on news articles, where journalists usually follow a fixed writing style. While such approaches are effective in limited contexts, they often fail to adapt to the varied nature of real-world Bangla texts. In today's digital era, a massive amount of Bangla content is continuously produced across blogs, newspapers, and social media. This creates a pressing need for summarization systems that can reduce information overload and help readers understand content more quickly. To address this challenge, we developed a dataset of over 54,000 Bangla articles and summaries collected from multiple sources, including blogs such as Cinegolpo and newspapers such as Samakal and The Business Standard. Unlike single-domain resources, our dataset spans multiple domains and writing styles. It offers greater adaptability and practical relevance. To establish strong baselines, we trained and evaluated this dataset using several deep learning and transfer learning models, including LSTM, BanglaT5-small, and MTS-small. The results highlight its potential as a benchmark for future research in Bangla natural language processing. This dataset provides a solid foundation for building robust summarization systems and helps expand NLP resources for low-resource languages.
Authors Md. Tanzim Ferdous, Naeem Ahsan Chowdhury, Prithwiraj Bhattacharjee
Categories Computation and Language
0
Strategic Innovation Management in the Age of Large Language Models Market Intelligence, Adaptive R&D, and Ethical Governance
Title CN 大型语言模型时代的战略性创新管理：市场情报、自适应研发与伦理治理
Keywords 大型语言模型 · 研发管理 · 创新生态系统 · 伦理治理 · 市场情报
Summary 本文探讨大型语言模型在自动化知识发现、假设生成和跨学科融合中的作用，以提升研发效率和创新速度。研究基于文献、专利和实验数据分析，强调LLM对创新管理和技术治理的影响。
Reason 该论文属于纯NLP或管理学领域的应用研究，讨论LLM在研发管理和创新生态系统中的作用，未涉及任何文档图像理解（DIU）、视觉输入处理或多模态建模。内容完全聚焦于企业研发流程优化和伦理治理，属于典型的‘平行应用’，且不解决OCR幻觉、复杂排版、细粒度定位等DIU核心问题。
Abstract This study analyzes the multiple functions of Large Language Models (LLMs) in transforming research and development (R&D) processes. By automating knowledge discovery, boosting hypothesis creation, integrating transdisciplinary insights, and enabling cooperation within innovation ecosystems, LLMs dramatically improve the efficiency and effectiveness of research processes. Through extensive analysis of scientific literature, patent databases, and experimental data, these models enable more flexible and informed R&D workflows, ultimately accelerating innovation cycles and lowering time-to-market for breakthrough ideas.
Authors Raha Aghaei, Ali A. Kiaei, Mahnaz Boush, Mahan Rofoosheh, Mohammad Zavvar
Categories Computation and Language
0
Lost in translation: using global fact-checks to measure multilingual misinformation prevalence, spread, and evolution
Title CN 迷失在翻译中：利用全球事实核查衡量多语言虚假信息的普遍性、传播和演变
Keywords 虚假信息 · 多语言分析 · 事实核查 · 语义演化 · 社会传播
Summary 本文通过分析覆盖95种语言的26万余条事实核查数据，研究多语言虚假信息的传播模式及其跨语言演化。作者使用多语言句子嵌入构建语义图，发现约10%的虚假声明被多次核查，其中超过三成跨越了语言边界，但传播具有强同质性，更倾向于在同一语言或语系内扩散。此外，跨语言虚假信息的核查响应时间更长，且声明在传播过程中会逐渐漂移，导致静态匹配算法效果下降。
Reason 该论文属于纯NLP与社会信息传播领域，研究多语言虚假信息通过事实核查数据的传播模式，不涉及任何视觉内容、文档结构、OCR或图文多模态理解。与文档图像理解（DIU）核心任务如DocVQA、布局分析、表格识别等完全无关，且未提出可用于解决OCR幻觉、复杂排版或长文档推理的底层技术方法。
Abstract Misinformation and disinformation are growing threats in the digital age, affecting people across languages and borders. However, no research has investigated the prevalence of multilingual misinformation and quantified the extent to which misinformation diffuses across languages. This paper investigates the prevalence and dynamics of multilingual misinformation through an analysis of 264,487 fact-checks spanning 95 languages. To study the evolution of claims over time and mutations across languages, we represent fact-checks with multilingual sentence embeddings and build a graph where semantically similar claims are linked. We provide quantitative evidence of repeated fact-checking efforts and establish that claims diffuse across languages. Specifically, we find that while the majority of misinformation claims are only fact-checked once, 10.26%, corresponding to more than 27,000 claims, are checked multiple times. Using fact-checks as a proxy for the spread of misinformation, we find 32.26% of repeated claims cross linguistic boundaries, suggesting that some misinformation permeates language barriers. However, spreading patterns exhibit strong assortativity, with misinformation more likely to spread within the same language or language family. Next we show that fact-checkers take more time to fact-check claims that have crossed language barriers and model the temporal and cross-lingual evolution of claims. We analyze connected components and shortest paths connecting different versions of a claim finding that claims gradually drift over time and undergo greater alteration when traversing languages. Misinformation changes over time, reducing the effectiveness of static claim matching algorithms. The findings advocate for expanded information sharing between fact-checkers globally while underscoring the importance of localized verification.
Authors Dorian Quelle, Calvin Cheng, Alexandre Bovet, Scott A. Hale
Categories Computation and Language
0
RAVEN++: Pinpointing Fine-Grained Violations in Advertisement Videos with Active Reinforcement Reasoning EMNLP 2025 (Oral, Industry Track)
Title CN RAVEN++：通过主动强化推理精确定位广告视频中的细粒度违规
Keywords 视频理解 · 内容审核 · 强化学习 · 细粒度定位 · 推理
Summary 本文提出RAVEN++，一种用于广告视频中细粒度违规内容定位的框架，结合主动强化学习、分层奖励函数与多阶段训练策略，在视频内容审核任务上提升了细粒度理解和推理能力。实验涵盖离线数据集与在线A/B测试。
Reason 该论文研究的是广告视频内容审核，属于视频理解领域，核心任务为视频级别的违规检测与定位，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。尽管其使用了强化学习和推理机制，但应用场景完全偏离文档结构、文本布局、OCR或视觉-语言对齐等DIU核心问题。根据负面清单，视频理解属于无关领域，无法迁移其方法至文档理解的核心痛点（如OCR幻觉、密集文本处理、复杂排版解析）。
Abstract Advertising (Ad) is a cornerstone of the digital economy, yet the moderation of video advertisements remains a significant challenge due to their complexity and the need for precise violation localization. While recent advancements, such as the RAVEN model, have improved coarse-grained violation detection, critical gaps persist in fine-grained understanding, explainability, and generalization. To address these limitations, we propose RAVEN++, a novel framework that introduces three key innovations: 1) Active Reinforcement Learning (RL), which dynamically adapts training to samples of varying difficulty; 2) Fine-Grained Violation Understanding, achieved through hierarchical reward functions and reasoning distillation; and 3) Progressive Multi-Stage Training, which systematically combines knowledge injection, curriculum-based passive RL, and active RL. Extensive experiments on both public and proprietary datasets, on both offline scenarios and online deployed A/B Testing, demonstrate that RAVEN++ outperforms general-purpose LLMs and specialized models like RAVEN in terms of fine-grained violation understanding, reasoning capabilities, and generalization ability.
Authors Deyi Ji, Yuekui Yang, Liqun Liu, Peng Shu, Haiyang Wu, Shaogang Tang, Xudong Chen, Shaoping Ma, Tianrun Chen, Lanyun Zhu
Comment EMNLP 2025 (Oral, Industry Track)
Categories Computation and Language
0
Emotion-Enhanced Multi-Task Learning with LLMs for Aspect Category Sentiment Analysis
Title CN 基于大语言模型的情感增强多任务学习在方面类别情感分析中的应用
Keywords 情感分析 · 方面类别情感分析 · 大语言模型 · 多任务学习 · VAD模型
Summary 本文提出了一种情感增强的多任务学习框架，结合大语言模型与Ekman六种基本情感，用于方面类别情感分析。通过生成情感描述并引入基于VAD（效价-唤醒-支配）空间的情绪一致性校正机制，提升情感分类性能。实验表明该方法在多个基准数据集上优于现有基线。
Reason 该论文属于纯自然语言处理领域的下游应用研究，聚焦于方面类别情感分析（ACSA）中的情感维度建模，尽管使用了LLM和情感精炼机制，但完全不涉及文档图像、视觉信息或任何多模态文档理解任务。其方法无法直接迁移用于解决OCR幻觉、复杂版面解析或文档视觉问答等DIU核心问题，属于典型的非相关领域平行应用。
Abstract Aspect category sentiment analysis (ACSA) has achieved remarkable progress with large language models (LLMs), yet existing approaches primarily emphasize sentiment polarity while overlooking the underlying emotional dimensions that shape sentiment expressions. This limitation hinders the model's ability to capture fine-grained affective signals toward specific aspect categories. To address this limitation, we introduce a novel emotion-enhanced multi-task ACSA framework that jointly learns sentiment polarity and category-specific emotions grounded in Ekman's six basic emotions. Leveraging the generative capabilities of LLMs, our approach enables the model to produce emotional descriptions for each aspect category, thereby enriching sentiment representations with affective expressions. Furthermore, to ensure the accuracy and consistency of the generated emotions, we introduce an emotion refinement mechanism based on the Valence-Arousal-Dominance (VAD) dimensional framework. Specifically, emotions predicted by the LLM are projected onto a VAD space, and those inconsistent with their corresponding VAD coordinates are re-annotated using a structured LLM-based refinement strategy. Experimental results demonstrate that our approach significantly outperforms strong baselines on all benchmark datasets. This underlines the effectiveness of integrating affective dimensions into ACSA.
Authors Yaping Chai, Haoran Xie, Joe S. Qin
Comment 8 pages, 4 figures
Categories Computation and Language
0
A symbolic Perl algorithm for the unification of Nahuatl word spellings MICAI 2025
Title CN 一种用于统一纳瓦特尔语单词拼写的符号化Perl算法
Keywords Nahuatl · 拼写统一 · 符号算法 · 正则表达式 · 计算语言学
Summary 本文提出了一种基于符号正则表达式的Perl算法，用于自动统一纳瓦特尔语文本的多种拼写形式，并在自建语料库π-yalli上进行了评估。通过人工评价协议测试了统一后句子在语义任务中的表现，取得了积极结果。但该工作局限于特定语言的文本规范化，未涉及视觉或文档布局信息。
Reason 该论文聚焦于使用符号规则对纳瓦特尔语（Nahuatl）这一特定小语种的文本进行拼写规范化处理，属于针对特定语言的文字处理任务。根据评审原则，非中英的小语种相关研究属于‘小语种’类别，直接命中负面清单第3条，无方法论上的通用性创新，且无法迁移至文档图像理解的核心问题（如OCR去幻觉、复杂排版理解等），因此不予考虑。
Abstract In this paper, we describe a symbolic model for the automatic orthographic unification of Nawatl text documents. Our model is based on algorithms that we have previously used to analyze sentences in Nawatl, and on the corpus called $π$-yalli, consisting of texts in several Nawatl orthographies. Our automatic unification algorithm implements linguistic rules in symbolic regular expressions. We also present a manual evaluation protocol that we have proposed and implemented to assess the quality of the unified sentences generated by our algorithm, by testing in a sentence semantic task. We have obtained encouraging results from the evaluators for most of the desired features of our artificially unified sentences
Authors Juan-José Guzmán-Landa, Jesús Vázquez-Osorio, Juan-Manuel Torres-Moreno, Ligia Quintana Torres, Miguel Figueroa-Saavedra, Martha-Lorena Avendaño-Garrido, Graham Ranger, Patricia Velázquez-Morales, Gerardo Eugenio Sierra Martínez
Comment MICAI 2025, LNAI 16221, pp. 141-154, 2026. 10 pages, 4 Figures, 8 Tables
Categories Computation and Language
0
Logic of Montage
Title CN 蒙太奇的逻辑
Keywords 情感表达 · 矛盾结构 · 蒙太奇 · 强度 · 跨系统词义导入
Summary 本文提出一种名为‘矛盾结构效应’的情感表达形式，并通过‘蒙太奇’操作将其叠加，构建更广泛的结构效应。理论框架借鉴了德勒兹和奥斯汀的语言哲学概念，试图在自然语言之外建立情绪表达的替代路径。研究停留在抽象语义层面，未涉及视觉、图像或实际数据处理。
Reason 该论文属于计算语言学与哲学理论交叉的抽象表达形式研究，探讨的是情感表达中的矛盾结构与‘蒙太奇’叠加效应，引入德勒兹的‘强度’概念作为理论建模元素。内容完全不涉及文档图像、多模态理解、视觉表示或VLM技术，与文档图像理解（DIU）任务无任何技术关联。
Abstract In expressing emotions, as an expression form separate from natural language, we propose an alternative form that complements natural language, acting as a proxy or window for emotional states. First, we set up an expression form "Effect of Contradictory Structure." "Effect of Contradictory Structure" is not static but dynamic. Effect in "Effect of Contradictory Structure" is unpleasant or pleasant, and the orientation to avoid that unpleasantness is considered pseudo-expression of will. Second, "Effect of Contradictory Structure" can be overlapped with each other. This overlapping operation is called "montage." A broader "Structure" that includes related "Effect of Contradictory Structure" and "Effect of Structure" are set up. Montage produces "Effect of Structure". In montage, it is necessary to set something like "strength," so we adopted Deleuze and Deleuze/Guattari's word "intensity" and set it as an element of our model. We set up a general theoretical framework - Word Import Between Systems (Models) and justified the import of "intensity" through Austin's use of the word "force." "Effect of Structure" process is demonstrated using the example of proceeding to the next level of education.
Authors Hayami Takahashi, Kensuke Takahashi
Categories Computation and Language
0
ModernBERT is More Efficient than Conventional BERT for Chest CT Findings Classification in Japanese Radiology Reports
Title CN ModernBERT在日语放射学报告胸部CT发现分类中比传统BERT更高效
Keywords Medical NLP · Japanese Language Model · Radiology Reports
Summary 本文比较了三种日语语言模型（BERT Base、JMedRoBERTa 和 ModernBERT）在多标签分类18种胸部CT发现任务上的表现，使用的是纯文本放射学报告数据集。实验结果显示ModernBERT在计算效率上具有优势，但在外部真实场景数据下性能下降明显，显示出对语言变异的敏感性。
Reason 该论文属于小语种（日语）的垂直领域应用，且聚焦于纯文本医疗报告分类任务，未涉及任何文档图像、视觉模态或图文多模态处理。完全不满足文档图像理解（DIU）的技术需求，属于‘平行下游应用’与‘小语种’双重负面清单项。
Abstract Japanese language models for medical text classification face challenges with complex vocabulary and linguistic structures in radiology reports. This study compared three Japanese models--BERT Base, JMedRoBERTa, and ModernBERT--for multi-label classification of 18 chest CT findings. Using the CT-RATE-JPN dataset, all models were fine-tuned under identical conditions. ModernBERT showed clear efficiency advantages, producing substantially fewer tokens and achieving faster training and inference than the other models while maintaining comparable performance on the internal test dataset (exact match accuracy: 74.7% vs. 72.7% for BERT Base). To assess generalizability, we additionally constructed RR-Findings, an external dataset of 243 naturally written Japanese radiology reports annotated using the same schema. Under this domain-shifted setting, performance differences became pronounced: BERT Base outperformed both JMedRoBERTa and ModernBERT, whereas ModernBERT showed the largest decline in exact match accuracy. Average precision differences were smaller, indicating that ModernBERT retained reasonable ranking ability despite reduced calibration. Overall, ModernBERT offers substantial computational efficiency and strong in-domain performance but remains sensitive to real-world linguistic variability. These results highlight the need for more diverse natural-language training data and domain-specific calibration strategies to improve robustness when deploying modern transformer models in heterogeneous clinical environments.
Authors Yosuke Yamagishi, Tomohiro Kikuchi, Shouhei Hanaoka, Takeharu Yoshikawa, Osamu Abe
Comment 31 pages
Categories Computation and Language
0
Understanding and Mitigating Over-refusal for Large Language Models via Safety Representation
Title CN 通过安全表示理解与缓解大语言模型的过度拒绝
Keywords 大语言模型 · 安全性 · 过度拒绝 · 表示学习 · 干预
Summary 本文从表示学习的角度分析大语言模型中过度拒绝现象的成因，发现过度拒绝样本位于良性与恶意样本的边界区域。基于此，作者提出MOSR方法，通过重叠感知损失加权和上下文感知增强两种机制干预模型的安全表示，以缓解过度拒绝问题，同时保持模型安全性。
Reason 该论文聚焦于纯NLP领域的安全与对齐问题（Safety/Over-refusal），研究内容为缓解LLM在有害输入上的过度拒绝行为，属于典型的‘纯NLP的安全/对齐’范畴。尽管提到了‘representation intervention’，但其目标与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）无直接关联，且未涉及任何视觉模态或跨模态交互机制，无法迁移用于解决文档理解中的关键挑战。
Abstract Large language models demonstrate powerful capabilities across various natural language processing tasks, yet they also harbor safety vulnerabilities. To enhance LLM safety, various jailbreak defense methods have been proposed to guard against harmful outputs. However, improvements in model safety often come at the cost of severe over-refusal, failing to strike a good balance between safety and usability. In this paper, we first analyze the causes of over-refusal from a representation perspective, revealing that over-refusal samples reside at the boundary between benign and malicious samples. Based on this, we propose MOSR, designed to mitigate over-refusal by intervening the safety representation of LLMs. MOSR incorporates two novel components: (1) Overlap-Aware Loss Weighting, which determines the erasure weight for malicious samples by quantifying their similarity to pseudo-malicious samples in the representation space, and (2) Context-Aware Augmentation, which supplements the necessary context for rejection decisions by adding harmful prefixes before rejection responses. Experiments demonstrate that our method outperforms existing approaches in mitigating over-refusal while largely maintaining safety. Overall, we advocate that future defense methods should strike a better balance between safety and over-refusal.
Authors Junbo Zhang, Ran Chen, Qianli Zhou, Xinyang Deng, Wen Jiang
Categories Computation and Language
0
Knowledge-based Graphical Method for Safety Signal Detection in Clinical Trials
Title CN 基于知识的图形化方法用于临床试验中的安全性信号检测
Keywords 安全性信号检测 · MedDRA · 语义映射 · 临床试验 · 可视化
Summary 本文提出一种基于图形和医学知识的方法，用于在临床试验中检测治疗引发的不良事件信号。通过在MedDRA基础上构建包含语义关系的Safeterm二维地图，自动将相似的不良事件术语聚类，并结合来自ClinicalTrials.gov的数据计算调整后的发生率比和EBGM值，生成可视化图表辅助信号识别。实验表明该方法能有效复现已知的安全信号。
Reason 该论文属于特定领域的下游应用（临床试验安全信号检测），其核心是利用医学术语系统（MedDRA）和语义关系进行不良事件聚类与可视化，未涉及文档图像理解、多模态大模型、OCR或任何视觉-语言推理技术。尽管使用了知识增强和可视化手段，但完全不涉及文档布局分析、文本识别或VLM相关架构与训练方法，属于典型的‘平行应用’，对DIU领域无方法论迁移价值。
Abstract We present a graphical, knowledge-based method for reviewing treatment-emergent adverse events (AEs) in clinical trials. The approach enhances MedDRA by adding a hidden medical knowledge layer (Safeterm) that captures semantic relationships between terms in a 2-D map. Using this layer, AE Preferred Terms can be regrouped automatically into similarity clusters, and their association to the trial disease may be quantified. The Safeterm map is available online and connected to aggregated AE incidence tables from ClinicalTrials.gov. For signal detection, we compute treatment-specific disproportionality metrics using shrinkage incidence ratios. Cluster-level EBGM values are then derived through precision-weighted aggregation. Two visual outputs support interpretation: a semantic map showing AE incidence and an expectedness-versus-disproportionality plot for rapid signal detection. Applied to three legacy trials, the automated method clearly recovers all expected safety signals. Overall, augmenting MedDRA with a medical knowledge layer improves clarity, efficiency, and accuracy in AE interpretation for clinical trials.
Authors Francois Vandenhende, Anna Georgiou, Michalis Georgiou, Theodoros Psaras, Ellie Karekla, Elena Hadjicosta
Comment 13 pages, 3 tables, 5 figures
Categories Computation and Language
0
Evaluation of OpenAI o1: Opportunities and Challenges of AGI
Title CN OpenAI o1的评估：通用人工智能的机遇与挑战
Keywords 大语言模型 · AGI · 复杂推理 · 性能评估 · 跨领域任务
Summary 本文系统评估了OpenAI o1-preview模型在编程、数学、医学、金融等多个领域的复杂推理能力，展示了其在多项任务上接近或超越人类的表现。但研究完全基于纯文本输入输出，不涉及图像、布局或文档结构信息处理，与文档图像理解无关。
Reason 该论文属于对大语言模型o1-preview在多个领域（如编程、数学、医学、金融等）的综合性能评测，未涉及文档图像理解（DIU）或视觉-语言多模态任务。其内容聚焦于纯文本推理和跨领域知识应用，既无视觉输入处理，也无针对OCR幻觉、复杂版面解析或文档VQA等核心问题的方法创新。属于典型的‘平行下游应用’评估，而非面向文档理解的技术源头工作。
Abstract This comprehensive study evaluates the performance of OpenAI's o1-preview large language model across a diverse array of complex reasoning tasks, spanning multiple domains, including computer science, mathematics, natural sciences, medicine, linguistics, and social sciences. Through rigorous testing, o1-preview demonstrated remarkable capabilities, often achieving human-level or superior performance in areas ranging from coding challenges to scientific reasoning and from language processing to creative problem-solving. Key findings include: -83.3% success rate in solving complex competitive programming problems, surpassing many human experts. -Superior ability in generating coherent and accurate radiology reports, outperforming other evaluated models. -100% accuracy in high school-level mathematical reasoning tasks, providing detailed step-by-step solutions. -Advanced natural language inference capabilities across general and specialized domains like medicine. -Impressive performance in chip design tasks, outperforming specialized models in areas such as EDA script generation and bug analysis. -Remarkable proficiency in anthropology and geology, demonstrating deep understanding and reasoning in these specialized fields. -Strong capabilities in quantitative investing. O1 has comprehensive financial knowledge and statistical modeling skills. -Effective performance in social media analysis, including sentiment analysis and emotion recognition. The model excelled particularly in tasks requiring intricate reasoning and knowledge integration across various fields. While some limitations were observed, including occasional errors on simpler problems and challenges with certain highly specialized concepts, the overall results indicate significant progress towards artificial general intelligence.
Authors Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Zeyu Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, Chao Cao, Hanqi Jiang, Hanxu Chen, Yiwei Li, Junhao Chen, Huawen Hu, Yiheng Liu, Huaqin Zhao, Shaochen Xu, Haixing Dai, Lin Zhao, Ruidong Zhang, Wei Zhao, Zhenyuan Yang, Jingyuan Chen, Peilong Wang, Wei Ruan, Hui Wang, Huan Zhao, Jing Zhang, Yiming Ren, Shihuan Qin, Tong Chen, Jiaxi Li, Arif Hassan Zidan, Afrar Jahin, Minheng Chen, Sichen Xia, Jason Holmes, Yan Zhuang, Jiaqi Wang, Bochen Xu, Weiran Xia, Jichao Yu, Kaibo Tang, Yaxuan Yang, Bolun Sun, Tao Yang, Guoyu Lu, Xianqiao Wang, Lilong Chai, He Li, Jin Lu, Xin Zhang, Bao Ge, Xintao Hu, Lian Zhang, Hua Zhou, Lu Zhang, Shu Zhang, Zhen Xiang, Yudan Ren, Jun Liu, Xi Jiang, Yu Bao, Wei Zhang, Xiang Li, Gang Li, Wei Liu, Dinggang Shen, Andrea Sikora, Xiaoming Zhai, Dajiang Zhu, Tuo Zhang, Tianming Liu
Categories Computation and Language
0
Reproducibility Study of Large Language Model Bayesian Optimization ICLR 2024
Title CN 大语言模型贝叶斯优化的可复现性研究
Keywords 贝叶斯优化 · 大语言模型 · 可复现性研究 · 提示工程 · 超参数优化
Summary 本文是对LLAMBO框架的可复现性研究，该框架利用大语言模型作为判别代理和采集优化器进行贝叶斯优化。作者使用Llama 3.1 70B替代原框架中的GPT-3.5，在相同协议下复现实验，验证了上下文热启动对早期遗憾行为的改善效果，并发现文本上下文的移除会显著降低预测准确性和校准能力。然而，较小的模型（如Gemma 27B、Llama 8B）表现不稳定，表明容量不足。整体表明LLAMBO架构在更换为开源大模型后仍有效。
Reason 该论文属于纯NLP与优化方法结合的工作，研究的是使用大语言模型进行贝叶斯优化的可复现性，核心内容与文档图像理解（DIU）、视觉输入、多模态建模或OCR等完全无关。尽管涉及LLM的应用机制，但无任何视觉模态处理、也不涉及文档结构、布局分析或视觉-语言对齐问题，无法迁移用于解决DIU中的OCR幻觉、复杂排版或细粒度定位等核心痛点。
Abstract In this reproducibility study, we revisit the LLAMBO framework of Daxberger et al. (2024), a prompting-based Bayesian optimization (BO) method that uses large language models as discriminative surrogates and acquisition optimizers via text-only interactions. We replicate the core Bayesmark and HPOBench experiments under the original evaluation protocol, but replace GPT-3.5 with the open-weight Llama 3.1 70B model used for all text encoding components. Our results broadly confirm the main claims of LLAMBO. Contextual warm starting via textual problem and hyperparameter descriptions substantially improves early regret behaviour and reduces variance across runs. LLAMBO's discriminative surrogate is weaker than GP or SMAC as a pure single task regressor, yet benefits from cross task semantic priors induced by the language model. Ablations that remove textual context markedly degrade predictive accuracy and calibration, while the LLAMBO candidate sampler consistently generates higher quality and more diverse proposals than TPE or random sampling. Experiments with smaller backbones (Gemma 27B, Llama 3.1 8B) yield unstable or invalid predictions, suggesting insufficient capacity for reliable surrogate behaviour. Overall, our study shows that the LLAMBO architecture is robust to changing the language model backbone and remains effective when instantiated with Llama 3.1 70B.
Authors Adam Rychert, Gasper Spagnolo, Evgenii Posashkov
Comment 7 pages, 8 figures. Reproducibility study of the LLAMBO framework (ICLR 2024). Code: https://github.com/spagnoloG/llambo-reproducibility
Categories Computation and Language
0
SGM: A Framework for Building Specification-Guided Moderation Filters
Title CN SGM：构建规范引导型内容过滤器的框架
Keywords 内容过滤 · 大语言模型对齐 · 规范引导 · 安全性 · 自动化数据生成
Summary 本文提出SGM框架，用于构建基于用户自定义规范的内容过滤器，以增强大语言模型在特定部署场景下的对齐能力。该方法通过自动化生成训练数据，支持超越传统安全范畴的细粒度控制目标，如风格一致性或格式合规性。然而，其应用场景完全局限于文本模态，未涉及任何视觉或文档结构信息。
Reason 该论文属于纯NLP领域的安全对齐（Safety/Alignment）研究，聚焦于通过用户定义规范训练内容过滤器以防御越狱攻击等行为。尽管涉及LLM输出控制，但完全不涉及视觉模态或文档理解任务，与文档图像理解（DIU）、DocVQA、OCR、多模态对齐或视觉幻觉等问题无任何关联。属于‘纯NLP的安全/对齐’范畴，命中负面清单第2条。
Abstract Aligning large language models (LLMs) with deployment-specific requirements is critical but inherently imperfect. Despite extensive training, models remain susceptible to misalignment and adversarial inputs such as jailbreaks. Content moderation filters are commonly used as external safeguards, though they typically focus narrowly on safety. We introduce SGM (Specification-Guided Moderation), a flexible framework for training moderation filters grounded in user-defined specifications that go beyond standard safety concerns. SGM automates training data generation without relying on human-written examples, enabling scalable support for diverse, application-specific alignment goals. SGM-trained filters perform on par with state-of-the-art safety filters built on curated datasets, while supporting fine-grained and user-defined alignment control.
Authors Masoomali Fatehkia, Enes Altinisik, Mohamed Osman, Husrev Taha Sencar
Categories Computation and Language
0
FanarGuard: A Culturally-Aware Moderation Filter for Arabic Language Models
Title CN FanarGuard：一种面向阿拉伯语语言模型的文化感知内容过滤器
Keywords 内容过滤 · 文化对齐 · 语言模型安全 · 阿拉伯语 · NLP
Summary 本文提出了FanarGuard，一个双语内容过滤系统，用于评估阿拉伯语和英语语言模型输出的安全性和文化敏感性。作者构建了包含46.8万条提示-响应对的数据集，并设计了首个针对阿拉伯文化语境的评测基准。实验表明该过滤器在人类一致性上优于现有标注者间一致性水平，同时在安全性能上达到SOTA。然而，该工作完全脱离视觉或多模态场景，不涉及任何文档图像、布局或视觉语言建模。
Reason 该论文聚焦于阿拉伯语和英语内容的安全与文化对齐过滤，属于纯NLP领域的安全/对齐（Safety）研究，且主要针对小语种（阿拉伯语）的文化语境。虽然涉及语言模型的幻觉控制，但其方法和数据均未结合视觉或文档模态，无法迁移至文档图像理解中的OCR幻觉、排版理解等核心问题。属于‘无关领域+小语种’双重负面清单命中。
Abstract Content moderation filters are a critical safeguard against alignment failures in language models. Yet most existing filters focus narrowly on general safety and overlook cultural context. In this work, we introduce FanarGuard, a bilingual moderation filter that evaluates both safety and cultural alignment in Arabic and English. We construct a dataset of over 468K prompt and response pairs, drawn from synthetic and public datasets, scored by a panel of LLM judges on harmlessness and cultural awareness, and use it to train two filter variants. To rigorously evaluate cultural alignment, we further develop the first benchmark targeting Arabic cultural contexts, comprising over 1k norm-sensitive prompts with LLM-generated responses annotated by human raters. Results show that FanarGuard achieves stronger agreement with human annotations than inter-annotator reliability, while matching the performance of state-of-the-art filters on safety benchmarks. These findings highlight the importance of integrating cultural awareness into moderation and establish FanarGuard as a practical step toward more context-sensitive safeguards.
Authors Masoomali Fatehkia, Enes Altinisik, Husrev Taha Sencar
Categories Computation and Language
0
Cognitive Alpha Mining via LLM-Driven Code-Based Evolution
Title CN 通过LLM驱动的基于代码的进化进行认知Alpha挖掘
Keywords 大语言模型 · 金融因子挖掘 · 进化搜索 · 代码生成 · 可解释性
Summary 本文提出了一种名为CogAlpha的认知Alpha挖掘框架，利用大语言模型（LLM）驱动的代码级表示与进化搜索相结合的方法，从高噪声金融数据中发现可解释且具有预测能力的Alpha因子。通过多阶段提示和反馈机制，LLM作为认知代理迭代优化Alpha表达式。实验表明该方法在A股市场中优于现有技术。然而，该工作聚焦于量化金融领域，并未涉及文档理解或视觉-语言建模中的核心挑战。
Reason 该论文属于金融领域的因子生成应用，尽管使用了LLM和进化算法，但其核心是将现有技术应用于金融信号发现，属于典型的‘平行下游应用’。研究内容与文档图像理解（DIU）、DocVQA、OCR、布局分析等任务无直接关联，也未提出可迁移至DIU领域的底层方法或架构创新。
Abstract Discovering effective predictive signals, or ``alphas,'' from financial data with high dimensionality and extremely low signal-to-noise ratio remains a difficult open problem. Despite progress in deep learning, genetic programming, and, more recently, large language model (LLM)--based factor generation, existing approaches still explore only a narrow region of the vast alpha search space. Neural models tend to produce opaque and fragile patterns, while symbolic or formula-based methods often yield redundant or economically ungrounded expressions that generalize poorly. Although different in form, these paradigms share a key limitation: none can conduct broad, structured, and human-like exploration that balances logical consistency with creative leaps. To address this gap, we introduce the Cognitive Alpha Mining Framework (CogAlpha), which combines code-level alpha representation with LLM-driven reasoning and evolutionary search. Treating LLMs as adaptive cognitive agents, our framework iteratively refines, mutates, and recombines alpha candidates through multi-stage prompts and financial feedback. This synergistic design enables deeper thinking, richer structural diversity, and economically interpretable alpha discovery, while greatly expanding the effective search space. Experiments on A-share equities demonstrate that CogAlpha consistently discovers alphas with superior predictive accuracy, robustness, and generalization over existing methods. Our results highlight the promise of aligning evolutionary optimization with LLM-based reasoning for automated and explainable alpha discovery. All source code will be released.
Authors Fengyuan Liu, Huang Yi, Sichun Luo, Yuqi Wang, Yazheng Yang, Xinye Li, Zefa Hu, Junlan Feng, Qi Liu
Categories Computation and Language
0
Large Language Models for the Summarization of Czech Documents: From History to the Present
Title CN 用于捷克语文档摘要的大型语言模型：从历史到现状
Keywords 文本摘要 · 捷克语 · 历史文档 · 大型语言模型 · 低资源语言
Summary 本文研究了使用大型语言模型（如Mistral和mT5）对现代和历史捷克语文本进行摘要的任务，提出了基于翻译的摘要方法，并发布了针对19世纪捷克历史文献的摘要数据集Posel od Čerchova。尽管为低资源语言提供了新数据，但完全基于纯文本处理，未结合视觉或布局信息。
Reason 该论文聚焦于捷克语这一小语种的文本摘要任务，且主要基于纯文本的LLM应用，未涉及文档图像理解中的视觉模态、布局分析或OCR等核心环节。虽然提到了历史文档，但其方法是翻译+纯文本摘要，并非多模态文档理解（DIU），也不包含对文档图像、排版结构或视觉信息的处理。因此属于‘小语种’且偏离DIU本题，不符合关注方向。
Abstract Text summarization is the task of automatically condensing longer texts into shorter, coherent summaries while preserving the original meaning and key information. Although this task has been extensively studied in English and other high-resource languages, Czech summarization, particularly in the context of historical documents, remains underexplored. This is largely due to the inherent linguistic complexity of Czech and the lack of high-quality annotated datasets. In this work, we address this gap by leveraging the capabilities of Large Language Models (LLMs), specifically Mistral and mT5, which have demonstrated strong performance across a wide range of natural language processing tasks and multilingual settings. In addition, we also propose a translation-based approach that first translates Czech texts into English, summarizes them using an English-language model, and then translates the summaries back into Czech. Our study makes the following main contributions: We demonstrate that LLMs achieve new state-of-the-art results on the SumeCzech dataset, a benchmark for modern Czech text summarization, showing the effectiveness of multilingual LLMs even for morphologically rich, medium-resource languages like Czech. We introduce a new dataset, Posel od Čerchova, designed for the summarization of historical Czech texts. This dataset is derived from digitized 19th-century publications and annotated for abstractive summarization. We provide initial baselines using modern LLMs to facilitate further research in this underrepresented area. By combining cutting-edge models with both modern and historical Czech datasets, our work lays the foundation for further progress in Czech summarization and contributes valuable resources for future research in Czech historical document processing and low-resource summarization more broadly.
Authors Václav Tran, Jakub Šmíd, Ladislav Lenc, Jean-Pierre Salmon, Pavel Král
Categories Computation and Language
0
A Reproducible Framework for Neural Topic Modeling in Focus Group Analysis
Title CN 用于焦点小组分析的可复现神经主题建模框架
Keywords 主题建模 · BERTopic · 定性分析 · 可复现性 · 超参数优化
Summary 本文提出了一种可复现的神经主题模型框架，用于分析焦点小组讨论的转录文本，重点解决超参数敏感性、模型稳定性及主题可解释性验证等问题。基于BERTopic对突尼斯HPV疫苗认知的焦点小组数据进行系统评估，并通过人工评审验证主题质量。所有代码和流程公开以支持复现。
Reason 该论文属于纯NLP领域的定性数据分析方法研究，聚焦于焦点小组讨论文本的主题建模流程优化，不涉及任何视觉或文档图像内容。任务与文档图像理解（DIU）完全无关，且未提出可用于多模态、OCR、布局分析或视觉推理迁移的方法论创新。
Abstract Focus group discussions generate rich qualitative data but their analysis traditionally relies on labor-intensive manual coding that limits scalability and reproducibility. We present a rigorous, reproducible computational framework for applying neural topic modeling to focus group transcripts, addressing fundamental methodological challenges: hyperparameter sensitivity, model stability, and validation of interpretability. Using BERTopic applied to ten focus groups exploring HPV vaccine perceptions in Tunisia (1,076 utterances), we conducted systematic evaluation across 27 hyperparameter configurations, assessed stability through bootstrap resampling with 30 replicates per configuration, and validated interpretability through formal human evaluation by three domain experts. Our analysis demonstrates substantial sensitivity to hyperparameter choices and reveals that metric selection for stability assessment must align with analytical goals. A hierarchical merging strategy (extracting fine-grained topics for stability then consolidating for interpretability) effectively navigates the stability-coherence tradeoff, achieving coherence of 0.558 compared to 0.539 for direct extraction. Human validation confirmed topic quality with very good inter-rater reliability (ICC = 0.79, weighted Cohen's kappa = 0.578). Our framework provides practical guidelines that researchers can adapt to their own qualitative research contexts. All code, data processing scripts, and evaluation protocols are publicly available to support reproduction and extension of this work.
Authors Heger Arfaoui, Mohammed Iheb Hergli, Beya Benzina, Slimane BenMiled
Categories Computation and Language
0
Can Code-Switched Texts Activate a Knowledge Switch in LLMs? A Case Study on English-Korean Code-Switching EMNLP 2025 Findings
Title CN 代码转换文本能否激活大语言模型中的知识切换？一项关于英韩代码转换的案例研究
Keywords 代码转换 · 多语言大模型 · 知识激活 · 低资源语言 · 语义理解
Summary 本文提出一个合成的英韩语码转换问答数据集EnKoQA，研究语码转换是否能帮助大语言模型在低资源语言任务中更好地识别和利用语言特定知识。实验表明，相比纯英文输入，语码转换能更有效地激活模型中的语言特有知识，尤其在文化或语言专属领域。但整个工作不涉及图像或文档模态，仅为文本层面的语言现象研究。
Reason 该论文研究英韩混合语码转换对多语言大模型知识激活的影响，属于纯NLP范畴，且聚焦于低资源语言和语码转换现象，与文档图像理解（DIU）、视觉-语言模型（VLM）或OCR、排版分析等核心任务完全无关。未涉及任何视觉输入、文档结构、多模态对齐或视觉幻觉等问题，无法迁移至DIU领域。
Abstract Recent large language models (LLMs) demonstrate multilingual abilities, yet they are English-centric due to dominance of English in training corpora. The limited resource for low-resource languages remains a crucial challenge. Code-switching (CS), a phenomenon where multilingual speakers alternate between languages in a discourse, can convey subtle cultural and linguistic nuances that can be otherwise lost in translation and elicits language-specific knowledge in human communications. In light of this, we investigate whether code-switching can activate, or identify and leverage knowledge for reasoning when LLMs solve low-resource language tasks. To facilitate the research, we first present EnKoQA, a synthetic English-Korean CS question-answering dataset. We provide comprehensive analysis on a variety of multilingual LLMs by subdividing activation process into knowledge identification and knowledge leveraging. Our results demonstrate that compared to English text, CS can faithfully activate knowledge inside LLMs especially on language-specific domains, suggesting the potential of code-switching on low-resource language tasks.
Authors Seoyeon Kim, Huiseo Kim, Chanjun Park, Jinyoung Yeo, Dongha Lee
Comment Accepted to EMNLP 2025 Findings
Categories Computation and Language
0
Context-Aware Whisper for Arabic ASR Under Linguistic Varieties
Title CN 面向阿拉伯语语言变体的上下文感知Whisper语音识别
Keywords 语音识别 · 阿拉伯语 · 提示工程 · 零样本学习 · 幻觉抑制
Summary 本文提出了一种上下文感知的提示策略，用于提升OpenAI的Whisper模型在阿拉伯语语音识别中的表现，无需重新训练。通过解码器提示、编码器前缀合成和多模态检索技术，在九种阿拉伯语语言条件下显著降低了词错误率，并缓解了幻觉和说话人不匹配问题。但该工作局限于语音领域，不涉及文档图像或多模态视觉理解。
Reason 该论文属于纯语音识别（ASR）领域，研究的是阿拉伯语在不同方言下的低资源语音转录问题，与文档图像理解（DIU）或视觉-语言模型（VLM）处理文档内容无直接关联。尽管提到了‘幻觉’缓解，但其上下文提示策略针对的是语音模态中的解码过程，难以迁移到文档图像中的视觉幻觉或OCR错误修正任务。且阿拉伯语为小语种，违反了非中英语言的排除原则。
Abstract Low-resource ASR remains a challenging problem, especially for languages like Arabic that exhibit wide dialectal variation and limited labeled data. We propose context-aware prompting strategies to adapt OpenAI's Whisper for Arabic speech recognition without retraining. Our methods include decoder prompting with first-pass transcriptions or retrieved utterances, and encoder prefixing using speech synthesized in the target speaker's voice. We introduce techniques such as prompt reordering, speaker-aware prefix synthesis, and modality-specific retrieval (lexical, semantic, acoustic) to improve transcription in real-world, zero-shot settings. Evaluated on nine Arabic linguistic conditions, our approach reduces WER by up to 22.3% on Modern Standard Arabic and 9.2% on dialectal speech, significantly mitigating hallucinations and speaker mismatch.
Authors Bashar Talafha, Amin Abu Alhassan, Muhammad Abdul-Mageed
Categories Computation and Language
0
Large Language Models Require Curated Context for Reliable Political Fact-Checking -- Even with Reasoning and Web Search
Title CN 大型语言模型需要精心策划的上下文以实现可靠的政治事实核查——即使具备推理和网络搜索能力
Keywords 大语言模型 · 事实核查 · RAG · 推理 · 网络搜索
Summary 本文评估了15种主流大语言模型在政治事实核查任务上的表现，发现即便结合推理能力和网络搜索，模型性能提升有限；而使用PolitiFact摘要构建的RAG系统则显著提升了F1分数。结果表明，提供高质量、结构化的上下文比增强模型自身能力更有效。
Reason 该论文属于纯NLP领域的事实核查应用，尽管涉及LLM与RAG，但其任务场景为政治声明的事实验证，完全不涉及文档图像、视觉内容或任何多模态理解。研究重点在于如何通过RAG提升事实准确性，而非解决OCR幻觉、排版理解或视觉-文本对齐等DIU核心问题，属于典型的‘平行下游应用’，且无任何可迁移至文档图像理解的方法论创新。
Abstract Large language models (LLMs) have raised hopes for automated end-to-end fact-checking, but prior studies report mixed results. As mainstream chatbots increasingly ship with reasoning capabilities and web search tools -- and millions of users already rely on them for verification -- rigorous evaluation is urgent. We evaluate 15 recent LLMs from OpenAI, Google, Meta, and DeepSeek on more than 6,000 claims fact-checked by PolitiFact, comparing standard models with reasoning- and web-search variants. Standard models perform poorly, reasoning offers minimal benefits, and web search provides only moderate gains, despite fact-checks being available on the web. In contrast, a curated RAG system using PolitiFact summaries improved macro F1 by 233% on average across model variants. These findings suggest that giving models access to curated high-quality context is a promising path for automated fact-checking.
Authors Matthew R. DeVerna, Kai-Cheng Yang, Harry Yaojun Yan, Filippo Menczer
Categories Computation and Language
0
Gender Bias in Emotion Recognition by Large Language Models AAAI 2026 Workshop (WS37)
Title CN 大型语言模型在情感识别中的性别偏见
Keywords 性别偏见 · 情感识别 · 大语言模型 · 去偏策略 · 提示工程
Summary 本文研究大型语言模型在情感识别任务中是否存在性别偏见，并探讨了多种去偏策略。结果表明，仅靠推理阶段的提示工程难以有效减小偏见，需要基于训练的干预方法才能实现显著改进。
Reason 该论文研究的是纯文本大模型在情感推理中的性别偏见问题，属于NLP领域的公平性（Fairness）研究，不涉及任何视觉输入、文档结构、多模态理解或OCR相关内容。尽管提到了prompt工程和训练干预等方法，但其应用场景与文档图像理解（DIU）的核心痛点（如排版、细粒度定位、OCR幻觉等）完全无关，无法迁移用于解决DIU任务。
Abstract The rapid advancement of large language models (LLMs) and their growing integration into daily life underscore the importance of evaluating and ensuring their fairness. In this work, we examine fairness within the domain of emotional theory of mind, investigating whether LLMs exhibit gender biases when presented with a description of a person and their environment and asked, "How does this person feel?". Furthermore, we propose and evaluate several debiasing strategies, demonstrating that achieving meaningful reductions in bias requires training based interventions rather than relying solely on inference-time prompt-based approaches such as prompt engineering.
Authors Maureen Herbert, Katie Sun, Angelica Lim, Yasaman Etesam
Comment Accepted at AAAI 2026 Workshop (WS37)
Categories Computation and Language
0
What does it mean to understand language?
Title CN 理解语言意味着什么？
Keywords 语言理解 · 认知神经科学 · 心理模型 · 脑功能网络 · 世界知识
Summary 本文提出，真正的语言理解不仅限于提取语言表面含义，还需要将信息从核心语言系统传递到其他脑区，以构建包含感知、动作和记忆的心理模型。作者综述了支持这一假设的神经科学证据，并呼吁利用新兴技术直接检验该理论。
Reason 该论文属于认知神经科学与计算语言学的交叉研究，探讨人类大脑如何处理和理解语言，核心关注的是人脑中的语言系统与感知、运动等其他区域的交互。尽管主题涉及‘语言理解’，但其研究对象是生物神经系统，而非文档图像或视觉-语言模型的技术实现。论文未涉及任何文档理解（DIU）、视觉输入、OCR、VLM架构或相关技术路径，对解决文档图像理解中的OCR幻觉、复杂排版、细粒度定位等关键问题无直接方法论迁移价值。
Abstract Language understanding entails not just extracting the surface-level meaning of the linguistic input, but constructing rich mental models of the situation it describes. Here we propose that because processing within the brain's core language system is fundamentally limited, deeply understanding language requires exporting information from the language system to other brain regions that compute perceptual and motor representations, construct mental models, and store our world knowledge and autobiographical memories. We review the existing evidence for this hypothesis, and argue that recent progress in cognitive neuroscience provides both the conceptual foundation and the methods to directly test it, thus opening up a new strategy to reveal what it means, cognitively and neurally, to understand language.
Authors Colton Casto, Anna Ivanova, Evelina Fedorenko, Nancy Kanwisher
Categories Computation and Language
0
Comparative Analysis of LoRA-Adapted Embedding Models for Clinical Cardiology Text Representation
Title CN 临床心脏病学文本表示中LoRA适配嵌入模型的比较分析
Keywords 临床NLP · LoRA · 文本嵌入 · 心脏病学 · 模型比较
Summary 本文评估了十种基于Transformer的嵌入模型在心脏病学文本上的表现，通过LoRA微调进行适配。结果显示编码器架构（尤其是BioLinkBERT）在领域特定性能上优于更大的解码器模型，同时计算资源需求更低。研究挑战了大模型必然更优的假设，并为临床NLP系统提供了实践指导。
Reason 该论文属于纯NLP领域，研究的是临床文本嵌入模型的适应与比较，不涉及任何视觉或文档图像理解内容。其方法聚焦于医学文本的表示学习，未处理OCR、布局分析、多模态对齐或视觉-语言联合推理等DIU核心问题，且无潜在可迁移至文档图像理解的技术机制（如视觉压缩、细粒度定位、高分辨率建模等）。属于典型的‘平行下游应用’类研究，仅在特定领域（心脏病学）内应用现有NLP技术。
Abstract Domain-specific text embeddings are critical for clinical natural language processing, yet systematic comparisons across model architectures remain limited. This study evaluates ten transformer-based embedding models adapted for cardiology through Low-Rank Adaptation (LoRA) fine-tuning on 106,535 cardiology text pairs derived from authoritative medical textbooks. Results demonstrate that encoder-only architectures, particularly BioLinkBERT, achieve superior domain-specific performance (separation score: 0.510) compared to larger decoder-based models, while requiring significantly fewer computational resources. The findings challenge the assumption that larger language models necessarily produce better domain-specific embeddings and provide practical guidance for clinical NLP system development. All models, training code, and evaluation datasets are publicly available to support reproducible research in medical informatics.
Authors Richard J. Young, Alice M. Matthews
Comment 25 pages, 13 figures, 5 tables
Categories Computation and Language
0
Can LLMs Faithfully Explain Themselves in Low-Resource Languages? A Case Study on Emotion Detection in Persian
Title CN 大语言模型在低资源语言中能否忠实地解释自身？以波斯语情感检测为例
Keywords 大语言模型 · 自我解释 · 忠实性 · 低资源语言 · 情感检测
Summary 本文研究大语言模型在波斯语情感分类任务中生成自我解释的忠实性，通过比较模型基于token级对数概率得出的关键词与人工标注结果的一致性，发现尽管模型分类性能良好，但其解释往往偏离人类判断，显示出当前解释方法在低资源语言中的局限性。
Reason 该论文研究的是低资源语言（波斯语）下的情感分类任务中LLM自解释的可信度问题，属于纯NLP范畴，且聚焦于小语种和自我解释的对齐性，与文档图像理解（DIU）、视觉输入、OCR、排版结构、多模态建模等核心问题完全无关。未涉及任何视觉模态或文档内容理解，无法迁移解决DIU中的幻觉、密集文本、复杂布局等痛点。
Abstract Large language models (LLMs) are increasingly used to generate self-explanations alongside their predictions, a practice that raises concerns about the faithfulness of these explanations, especially in low-resource languages. This study evaluates the faithfulness of LLM-generated explanations in the context of emotion classification in Persian, a low-resource language, by comparing the influential words identified by the model against those identified by human annotators. We assess faithfulness using confidence scores derived from token-level log-probabilities. Two prompting strategies, differing in the order of explanation and prediction (Predict-then-Explain and Explain-then-Predict), are tested for their impact on explanation faithfulness. Our results reveal that while LLMs achieve strong classification performance, their generated explanations often diverge from faithful reasoning, showing greater agreement with each other than with human judgments. These results highlight the limitations of current explanation methods and metrics, emphasizing the need for more robust approaches to ensure LLM reliability in multilingual and low-resource contexts.
Authors Mobina Mehrazar, Mohammad Amin Yousefi, Parisa Abolfath Beygi, Behnam Bahrak
Categories Computation and Language
0
The magnitude of categories of texts enriched by language models
Title CN 语言模型增强文本类别的范畴大小
Keywords 语言模型 · 范畴论 · 文本生成 · Tsallis熵 · Shannon熵
Summary 本文从范畴论角度出发，利用语言模型的下一词预测概率构建自然语言文本的富集范畴，并研究其对应的广义度量空间的Möbius函数与范畴大小。研究表明，该范畴大小可表示为Tsallis熵的和，并在特定条件下退化为Shannon熵之和，具有类似配分函数的行为。此外，作者还将其与范畴同调的欧拉示性数联系起来，给出了零阶和一阶同调群的具体描述。
Reason 该论文属于纯理论语言学与范畴论的交叉研究，探讨基于语言模型的文本生成概率所定义的范畴结构及其代数性质（如Möbius函数、范畴大小等）。内容完全未涉及视觉模态、文档结构、多模态建模或实际文档理解任务，与文档图像理解（DIU）无任何技术关联。既非DIU本题工作，也无任何可迁移至OCR、布局分析、VQA等场景的方法论价值。
Abstract The purpose of this article is twofold. Firstly, we use the next-token probabilities given by a language model to explicitly define a category of texts in natural language enriched over the unit interval, in the sense of Bradley, Terilla, and Vlassopoulos. We consider explicitly the terminating conditions for text generation and determine when the enrichment itself can be interpreted as a probability over texts. Secondly, we compute the Möbius function and the magnitude of an associated generalized metric space of texts. The magnitude function of that space is a sum over texts (prompts) of the $t$-logarithmic (Tsallis) entropies of the next-token probability distributions associated with each prompt, plus the cardinality of the model's possible outputs. A suitable evaluation of the magnitude function's derivative recovers a sum of Shannon entropies, which justifies seeing magnitude as a partition function. Following Leinster and Shulman, we also express the magnitude function of the generalized metric space as an Euler characteristic of magnitude homology and provide an explicit description of the zeroeth and first magnitude homology groups.
Authors Tai-Danae Bradley, Juan Pablo Vigneaux
Comment 26 pages
Categories Computation and Language
0
Health Sentinel: An AI Pipeline For Real-time Disease Outbreak Detection
Title CN Health Sentinel：一种用于实时疾病暴发检测的AI管道
Keywords 信息抽取 · 疾病监测 · AI pipeline · 文本挖掘 · 公共卫生
Summary 本文提出了一种名为Health Sentinel的多阶段信息抽取管道，用于从在线新闻文章中自动提取与疾病暴发相关的结构化事件信息。系统结合机器学习与非机器学习方法，处理海量网络文本，辅助公共卫生机构进行早期疫情识别。自2022年4月以来已处理超3亿篇文章，识别出9.5万个健康事件，其中3500多个被专家标记为潜在暴发。
Reason 该论文属于典型的‘平行下游应用’，即将现有信息抽取技术应用于公共卫生领域的媒体文章监测。其核心是构建一个面向特定场景（疾病暴发检测）的流水线系统，并未提出任何文档图像理解或VLM层面的方法论创新。任务本质为文本信息抽取与筛选，不涉及OCR、视觉布局分析、多模态对齐或文档推理等DIU核心问题。
Abstract Early detection of disease outbreaks is crucial to ensure timely intervention by the health authorities. Due to the challenges associated with traditional indicator-based surveillance, monitoring informal sources such as online media has become increasingly popular. However, owing to the number of online articles getting published everyday, manual screening of the articles is impractical. To address this, we propose Health Sentinel. It is a multi-stage information extraction pipeline that uses a combination of ML and non-ML methods to extract events-structured information concerning disease outbreaks or other unusual health events-from online articles. The extracted events are made available to the Media Scanning and Verification Cell (MSVC) at the National Centre for Disease Control (NCDC), Delhi for analysis, interpretation and further dissemination to local agencies for timely intervention. From April 2022 till date, Health Sentinel has processed over 300 million news articles and identified over 95,000 unique health events across India of which over 3,500 events were shortlisted by the public health experts at NCDC as potential outbreaks.
Authors Devesh Pant, Rishi Raj Grandhe, Vipin Samaria, Mukul Paul, Sudhir Kumar, Saransh Khanna, Jatin Agrawal, Jushaan Singh Kalra, Akhil VSSG, Satish V Khalikar, Vipin Garg, Himanshu Chauhan, Pranay Verma, Neha Khandelwal, Soma S Dhavala, Minesh Mathew
Categories Computation and Language
0
Functional Classification of Spiking Signal Data Using Artificial Intelligence Techniques: A Review
Title CN 利用人工智能技术对尖峰信号数据进行功能分类：综述
Keywords 尖峰分类 · 脑电信号 · 人工智能 · 神经科学 · 信号处理
Summary 本文是一篇关于使用人工智能技术对脑电等神经信号中的尖峰数据进行分类的综述，涵盖了预处理、分类和评估三个环节，回顾了基于机器学习和深度学习的方法。其目标是提升神经活动噪声的识别能力，服务于疾病诊断和人机交互，但不涉及任何形式的文档图像或文本布局理解。
Reason 该论文属于神经科学与脑电信号分析领域，研究内容为EEG等信号中的尖峰检测与分类，与文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的应用完全无关。未涉及OCR、布局分析、表格识别、DocVQA等任何DIU核心任务，也未提出可用于解决文档理解中幻觉、复杂排版或长文档推理等问题的通用方法。属于无关领域。
Abstract Human brain neuron activities are incredibly significant nowadays. Neuronal behavior is assessed by analyzing signal data such as electroencephalography (EEG), which can offer scientists valuable information about diseases and human-computer interaction. One of the difficulties researchers confront while evaluating these signals is the existence of large volumes of spike data. Spikes are some considerable parts of signal data that can happen as a consequence of vital biomarkers or physical issues such as electrode movements. Hence, distinguishing types of spikes is important. From this spot, the spike classification concept commences. Previously, researchers classified spikes manually. The manual classification was not precise enough as it involves extensive analysis. Consequently, Artificial Intelligence (AI) was introduced into neuroscience to assist clinicians in classifying spikes correctly. This review discusses the importance and use of AI in spike classification, focusing on the recognition of neural activity noises. The task is divided into three main components: preprocessing, classification, and evaluation. Existing methods are introduced and their importance is determined. The review also highlights the need for more efficient algorithms. The primary goal is to provide a perspective on spike classification for future research and provide a comprehensive understanding of the methodologies and issues involved. The review organizes materials in the spike classification field for future studies. In this work, numerous studies were extracted from different databases. The PRISMA-related research guidelines were then used to choose papers. Then, research studies based on spike classification using machine learning and deep learning approaches with effective preprocessing were selected.
Authors Danial Sharifrazi, Nouman Javed, Javad Hassannataj Joloudari, Roohallah Alizadehsani, Prasad N. Paradkar, Ru-San Tan, U. Rajendra Acharya, Asim Bhatti
Comment 8 figures, 32 pages
Categories Artificial Intelligence
0
FHE-Agent: Automating CKKS Configuration for Practical Encrypted Inference via an LLM-Guided Agentic Framework
Title CN FHE-Agent：通过LLM引导的代理框架自动化CKKS配置以实现实际加密推理
Keywords FHE · CKKS · LLM-Agent · 加密推理 · 自动化配置
Summary 本文提出了FHE-Agent，一个基于LLM的代理框架，用于自动化全同态加密（特别是CKKS方案）中的配置过程，解决了因缺乏密码学专业知识而导致的配置困难问题。该系统结合LLM控制器与确定性工具套件，采用多保真度工作流，在保证安全性的前提下优化延迟与精度。实验表明其在多种神经网络模型上优于传统启发式方法。
Reason 该论文属于‘平行下游应用’，其核心是将LLM代理用于全同态加密（FHE）中的参数配置优化，属于特定领域（密码学与隐私计算）的工程应用。尽管使用了LLM和Agent架构，但并未提出适用于文档图像理解（DIU）任务的方法论创新，也无法迁移解决OCR幻觉、复杂排版、细粒度定位等DIU核心痛点。
Abstract Fully Homomorphic Encryption (FHE), particularly the CKKS scheme, is a promising enabler for privacy-preserving MLaaS, but its practical deployment faces a prohibitive barrier: it heavily relies on domain expertise. Configuring CKKS involves a tightly coupled space of ring dimensions, modulus chains, and packing layouts. Without deep cryptographic knowledge to navigate these interactions, practitioners are restricted to compilers that rely on fixed heuristics. These "one-shot" tools often emit rigid configurations that are either severely over-provisioned in latency or fail to find a feasible solution entirely for deeper networks. We present FHE-Agent, an agentic framework that automates this expert reasoning process. By coupling a Large Language Model (LLM) controller with a deterministic tool suite, FHE-Agent decomposes the search into global parameter selection and layer-wise bottleneck repair. The agents operate within a multi-fidelity workflow, pruning invalid regimes using cheap static analysis and reserving expensive encrypted evaluations for the most promising candidates. We instantiate FHE-Agent on the Orion compiler and evaluate it on standard benchmarks (MLP, LeNet, LoLa) and deeper architectures (AlexNet). FHE-Agent consistently achieves better precision and lower latency than naïve search strategies. Crucially, it automatically discovers feasible, 128-bit secure configurations for complex models where baseline heuristics and one-shot prompts fail to produce a valid setup.
Authors Nuo Xu, Zhaoting Gong, Ran Ran, Jinwei Tang, Wujie Wen, Caiwen Ding
Categories Artificial Intelligence
0
Lean 5.0: A Predictive, Human-AI, and Ethically Grounded Paradigm for Construction Management
Title CN 精益5.0：一种面向建筑管理的预测性、人机协同且符合伦理的范式
Keywords Lean 5.0 · 建筑管理 · 人机协作 · 预测分析 · 数字孪生
Summary 本文提出了名为Lean 5.0的建筑管理新范式，结合预测分析、人工智能协作和持续学习，在工业5.0背景下进行实证验证。研究通过文献综述和12周案例研究，展示了在计划完成率、返工减少和预测准确性方面的提升。同时探讨了与数字孪生和区块链的集成以增强可追溯性和透明度。但由于样本量小、单案例设计和研究周期短，存在局限性。
Reason 该论文属于特定行业（建筑管理）中的AI应用研究，属于典型的‘平行下游应用’。其核心贡献是将现有AI与预测分析技术整合到建筑管理流程中，并未提出任何可用于文档图像理解（DIU）或视觉语言模型（VLM）的方法论创新。尽管涉及人机协作和AI集成，但无底层技术突破，无法迁移用于解决OCR幻觉、复杂排版理解或长文档推理等DIU核心问题。
Abstract This paper introduces Lean 5.0, a human-centric evolution of Lean-Digital integration that connects predictive analytics, AI collaboration, and continuous learning within Industry 5.0 and Construction 5.0 contexts. A systematic literature review (2019-2024) and a 12-week empirical validation study demonstrate measurable performance gains, including a 13% increase in Plan Percent Complete (PPC), 22% reduction in rework, and 42% improvement in forecast accuracy. The study adopts a mixed-method Design Science Research (DSR) approach aligned with PRISMA 2020 guidelines. The paper also examines integration with digital twin and blockchain technologies to improve traceability, auditability, and lifecycle transparency. Despite limitations related to sample size, single-case design, and study duration, the findings show that Lean 5.0 provides a transformative paradigm connecting human cognition with predictive control in construction management.
Authors Atena Khoshkonesh, Mohsen Mohammadagha, Navid Ebrahimi, Narges Sadeghigolshan
Categories Artificial Intelligence
0
The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective
Title CN 使用大语言模型模拟人类行为的挑战：一种因果推断视角
Keywords 因果推断 · LLM模拟 · 提示工程 · 实验设计 · 混杂偏差
Summary 论文从因果推断角度指出，当前使用LLM模拟人类实验时，若不对模型揭示实验设计（即‘盲法’），会导致处理变量与未观测变量之间产生系统性混淆，破坏无混杂假设，从而得出不合理结果。作者提出通过‘去盲’（unblinding）方式明确提示实验结构，以消除模糊性，并验证了该策略能提升各类模型的模拟鲁棒性和性能。
Reason 该论文研究的是利用LLM模拟人类行为时在因果推断和实验设计中面临的混淆变量问题，核心关注点为社会科学仿真与行为建模，与文档图像理解（DIU）、DocVQA、OCR、多模态文档处理等任务无直接关联。虽然涉及提示策略改进（如去模糊化提示），但其方法论目标并非解决文档中的视觉-语言对齐、细粒度定位或OCR幻觉等问题，且未提供可迁移至DIU领域的底层技术工具（如高分辨率处理、视觉token压缩、推理增强机制）。属于平行应用类研究（将LLM用于行为模拟），不符合本方向的技术创新要求。
Abstract Large Language Models (LLMs) have shown impressive potential to simulate human behavior. We identify a fundamental challenge in using them to simulate experiments: when LLM-simulated subjects are blind to the experimental design (as is standard practice with human subjects), variations in treatment systematically affect unspecified variables that should remain constant, violating the unconfoundedness assumption. Using demand estimation as a context and an actual experiment with 40 different products as a benchmark, we show this can lead to implausible results. While confounding may in principle be addressed by controlling for covariates, this can compromise ecological validity in the context of LLM simulations: controlled covariates become artificially salient in the simulated decision process. We show formally that confoundness stems from ambiguous prompting strategies. Therefore, it can be addressed by developing unambiguous prompting strategies through unblinding, i.e., revealing the experiment design in LLM simulations. Our empirical results show that this strategy consistently enhances model performance across all tested models, including both out-of-box reasoning and non-reasoning models. We also show that it is a technique that complements fine-tuning: while fine-tuning can improve simulation performance, an unambiguous prompting strategy makes the predictions robust to the inclusion of irrelevant data in the fine-tuning process.
Authors George Gui, Olivier Toubia
Categories Artificial Intelligence
0
Health system learning achieves generalist neuroimaging models
Title CN 医疗系统学习实现通用型神经影像模型
Keywords 神经影像 · 视觉基础模型 · 医疗AI · 幻觉抑制 · 临床决策支持
Summary 本文提出了一种名为NeuroVFM的视觉基础模型，通过在医疗机构内部未整理的临床神经影像数据上进行训练，实现了高性能的通用神经影像理解。该模型在放射学诊断和报告生成等任务上表现优异，并能与开源语言模型结合生成高质量的放射科报告，减少幻觉和关键错误。然而，其应用场景局限于医学3D影像，不涉及文档结构、文本布局或OCR相关挑战。
Reason 该论文属于医学影像分析领域，研究的是基于临床神经影像数据（如MRI和CT）的视觉基础模型，与文档图像理解（DIU）或DocVQA无关。尽管其提到减少幻觉和视觉定位，但任务场景、数据模态（3D医学体积数据）与文档图像差异巨大，无法迁移至OCR、排版理解、表格识别等DIU核心问题。
Abstract Frontier artificial intelligence (AI) models, such as OpenAI's GPT-5 and Meta's DINOv3, have advanced rapidly through training on internet-scale public data, yet such systems lack access to private clinical data. Neuroimaging, in particular, is underrepresented in the public domain due to identifiable facial features within MRI and CT scans, fundamentally restricting model performance in clinical medicine. Here, we show that frontier models underperform on neuroimaging tasks and that learning directly from uncurated data generated during routine clinical care at health systems, a paradigm we call health system learning, yields high-performance, generalist neuroimaging models. We introduce NeuroVFM, a visual foundation model trained on 5.24 million clinical MRI and CT volumes using a scalable volumetric joint-embedding predictive architecture. NeuroVFM learns comprehensive representations of brain anatomy and pathology, achieving state-of-the-art performance across multiple clinical tasks, including radiologic diagnosis and report generation. The model exhibits emergent neuroanatomic understanding and interpretable visual grounding of diagnostic findings. When paired with open-source language models through lightweight visual instruction tuning, NeuroVFM generates radiology reports that surpass frontier models in accuracy, clinical triage, and expert preference. Through clinically grounded visual understanding, NeuroVFM reduces hallucinated findings and critical errors, offering safer clinical decision support. These results establish health system learning as a paradigm for building generalist medical AI and provide a scalable framework for clinical foundation models.
Authors Akhil Kondepudi, Akshay Rao, Chenhui Zhao, Yiwei Lyu, Samir Harake, Soumyanil Banerjee, Rushikesh Joshi, Anna-Katharina Meissner, Renly Hou, Cheng Jiang, Asadur Chowdury, Ashok Srinivasan, Brian Athey, Vikas Gulani, Aditya Pandey, Honglak Lee, Todd Hollon
Comment 53 pages, 4 main figures, 10 extended data figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
No Free Lunch in Language Model Bias Mitigation? Targeted Bias Reduction Can Exacerbate Unmitigated LLM Biases
Title CN 语言模型偏见缓解中的免费午餐？有针对性的偏见减少可能会加剧未缓解的LLM偏见
Keywords 大语言模型 · 偏见缓解 · 社会偏见 · 模型连贯性 · StereoSet
Summary 本文探讨了在针对某一类偏见进行大语言模型去偏时，可能引发其他维度偏见加剧的问题。作者评估了四种去偏技术在七个模型家族上的跨类别影响，发现虽然目标偏见可能降低，但未被关注的偏见维度和模型连贯性往往恶化。这表明当前去偏策略需要更全面的多维评估体系。
Reason 该论文研究的是大语言模型在社会性偏见（如种族、宗教、性别等）方面的偏见缓解问题，属于纯NLP领域的安全与对齐（Safety/Alignment）研究。尽管其分析方法可能具有通用性，但内容与文档图像理解（DIU）、视觉-语言建模、OCR、布局分析、表格识别等核心任务完全无关，也不涉及视觉幻觉或任何多模态处理机制。无法迁移解决DIU中的OCR幻觉、复杂排版或长文档推理等关键痛点。
Abstract Large Language Models (LLMs) inherit societal biases from their training data, potentially leading to harmful or unfair outputs. While various techniques aim to mitigate these biases, their effects are often evaluated only along the dimension of the bias being targeted. This work investigates the cross-category consequences of targeted bias mitigation. We study four bias mitigation techniques applied across ten models from seven model families, and we explore racial, religious, profession- and gender-related biases. We measure the impact of debiasing on model coherence and stereotypical preference using the StereoSet benchmark. Our results consistently show that while targeted mitigation can sometimes reduce bias in the intended dimension, it frequently leads to unintended and often negative consequences in others, such as increasing model bias and decreasing general coherence. These findings underscore the critical need for robust, multi-dimensional evaluation tools when examining and developing bias mitigation strategies to avoid inadvertently shifting or worsening bias along untargeted axes.
Authors Shireen Chand, Faith Baca, Emilio Ferrara
Categories Artificial Intelligence, Computation and Language
0
Bridging Philosophy and Machine Learning: A Structuralist Framework for Classifying Neural Network Representations
Title CN 连接哲学与机器学习：一种用于分类神经网络表示的结构主义框架
Keywords 哲学与AI · 表示学习 · 可解释性 · 结构主义 · 本体论
Summary 本文从结构主义科学哲学出发，构建了一个用于分类神经网络表示中隐含本体承诺的框架。通过系统性文献回顾，分析了过去二十年中五篇代表性论文，并基于实体消除、结构来源和存在模式三个标准进行归类。结果显示，当前研究普遍倾向于‘结构唯心主义’，即认为表示是依赖模型架构、数据先验和训练动态的建构产物，而‘结构实在主义’则明显缺失。该框架旨在澄清可解释性、涌现性和知识信任等议题中的概念张力。
Reason 该论文属于哲学与机器学习交叉的理论性探讨，未提出任何可迁移至文档图像理解（DIU）的技术方法或架构创新。内容聚焦于神经网络表示的本体论分类框架，属于基础性、思辨性研究，不涉及OCR、VQA、布局分析等具体任务，也无法解决DIU中的OCR幻觉、密集文本、复杂排版或长文档推理等实际问题。
Abstract Machine learning models increasingly function as representational systems, yet the philosoph- ical assumptions underlying their internal structures remain largely unexamined. This paper develops a structuralist decision framework for classifying the implicit ontological commitments made in machine learning research on neural network representations. Using a modified PRISMA protocol, a systematic review of the last two decades of literature on representation learning and interpretability is conducted. Five influential papers are analysed through three hierarchical criteria derived from structuralist philosophy of science: entity elimination, source of structure, and mode of existence. The results reveal a pronounced tendency toward structural idealism, where learned representations are treated as model-dependent constructions shaped by architec- ture, data priors, and training dynamics. Eliminative and non-eliminative structuralist stances appear selectively, while structural realism is notably absent. The proposed framework clarifies conceptual tensions in debates on interpretability, emergence, and epistemic trust in machine learning, and offers a rigorous foundation for future interdisciplinary work between philosophy of science and machine learning.
Authors Yildiz Culcu
Comment 7 pages, 1 figure, 1 table. Developed from the author's bachelor thesis but substantially revised and reformulated for research publication
Categories Artificial Intelligence
0
Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attacks
Title CN 不一致性时间：大语言模型对抗攻击鲁棒性的生存分析
Keywords 大语言模型 · 对话鲁棒性 · 生存分析 · 语义漂移 · 风险预警
Summary 本文提出使用生存分析方法评估大语言模型在多轮对话中的不一致性风险，通过建模‘失败时间’来预测对话何时开始出现语义不一致。研究基于MT-Consistency基准上的36,951轮对话，比较了Cox模型、加速失效时间模型（AFT）和随机生存森林的表现，发现简单的语义漂移特征即可有效预测崩溃趋势，并可提前数轮预警失败对话。然而，该工作完全脱离视觉输入与文档场景，不涉及多模态建模或视觉语言对齐。
Reason 该论文研究的是大语言模型在多轮对话中的语义漂移与鲁棒性问题，属于纯NLP领域的对话安全与稳定性分析，未涉及任何视觉模态或文档理解任务。尽管其提出的生存分析框架可能对系统监控有启发，但无法迁移解决文档图像理解中的OCR幻觉、细粒度定位、复杂排版或高分辨率处理等核心痛点。
Abstract Large Language Models (LLMs) have revolutionized conversational AI, yet their robustness in extended multi-turn dialogues remains poorly understood. Existing evaluation frameworks focus on static benchmarks and single-turn assessments, failing to capture the temporal dynamics of conversational degradation that characterize real-world interactions. In this work, we present a large-scale survival analysis of conversational robustness, modeling failure as a time-to-event process over 36,951 turns from 9 state-of-the-art LLMs on the MT-Consistency benchmark. Our framework combines Cox proportional hazards, Accelerated Failure Time (AFT), and Random Survival Forest models with simple semantic drift features. We find that abrupt prompt-to-prompt semantic drift sharply increases the hazard of inconsistency, whereas cumulative drift is counterintuitively \emph{protective}, suggesting adaptation in conversations that survive multiple shifts. AFT models with model-drift interactions achieve the best combination of discrimination and calibration, and proportional hazards checks reveal systematic violations for key drift covariates, explaining the limitations of Cox-style modeling in this setting. Finally, we show that a lightweight AFT model can be turned into a turn-level risk monitor that flags most failing conversations several turns before the first inconsistent answer while keeping false alerts modest. These results establish survival analysis as a powerful paradigm for evaluating multi-turn robustness and for designing practical safeguards for conversational AI systems.
Authors Yubo Li, Ramayya Krishnan, Rema Padman
Categories Artificial Intelligence, Computation and Language
0
OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graph
Title CN OpenGloss：一个合成的百科词典与语义知识图谱
Keywords 知识图谱 · 词典生成 · 语言模型 · 自然语言处理 · 合成数据
Summary 本文提出了OpenGloss，一个通过多智能体流程自动生成的英语百科词典与语义知识图谱，包含53.7万个词义、910万条 semantic edges、百万级例句和搭配，以及6000万词的百科内容。资源在不到一周时间内以低于1000美元成本完成，展示了基于大模型的自动化词典构建可行性。数据已公开发布。
Reason 该论文属于纯语言领域的知识资源构建，聚焦于词汇定义、词源和语义关系的合成生成，未涉及任何文档图像、视觉内容或图文多模态理解。尽管其使用LLM生成技术，但任务本质是NLP内部的词典构建，与文档图像理解（DIU）的核心问题如OCR幻觉、排版分析、视觉文本定位等完全无关，无法迁移用于解决DIU痛点。
Abstract We present OpenGloss, a synthetic encyclopedic dictionary and semantic knowledge graph for English that integrates lexicographic definitions, encyclopedic context, etymological histories, and semantic relationships in a unified resource. OpenGloss contains 537K senses across 150K lexemes, on par with WordNet 3.1 and Open English WordNet, while providing more than four times as many sense definitions. These lexemes include 9.1M semantic edges, 1M usage examples, 3M collocations, and 60M words of encyclopedic content. Generated through a multi-agent procedural generation pipeline with schema-validated LLM outputs and automated quality assurance, the entire resource was produced in under one week for under $1,000. This demonstrates that structured generation can create comprehensive lexical resources at cost and time scales impractical for manual curation, enabling rapid iteration as foundation models improve. The resource addresses gaps in pedagogical applications by providing integrated content -- definitions, examples, collocations, encyclopedias, etymology -- that supports both vocabulary learning and natural language processing tasks. As a synthetically generated resource, OpenGloss reflects both the capabilities and limitations of current foundation models. The dataset is publicly available on Hugging Face under CC-BY 4.0, enabling researchers and educators to build upon and adapt this resource.
Authors Michael J. Bommarito
Comment 30 pages, 5 figures, 8 tables. Dataset available at https://huggingface.co/datasets/mjbommar/opengloss-dictionary
Categories Artificial Intelligence, Computation and Language
0
A Unified BERT-CNN-BiLSTM Framework for Simultaneous Headline Classification and Sentiment Analysis of Bangla News
Title CN 一种用于孟加拉语新闻标题分类与情感分析的统一BERT-CNN-BiLSTM框架
Keywords 孟加拉语 · 文本分类 · 情感分析 · BERT · CNN-BiLSTM
Summary 本文提出了一种结合BERT、CNN和BiLSTM的混合模型，用于同时进行孟加拉语新闻标题的分类与情感分析，在BAN-ABSA数据集上进行了实验，并比较了不同采样策略的效果。模型在该任务上达到了SOTA性能。
Reason 该论文研究的是小语种（孟加拉语）的纯文本NLP任务，属于特定语言下的文本分类应用，未涉及任何文档图像、视觉信息或VLM技术。完全不符合DIU领域要求，且属于‘小语种’负面清单范畴。
Abstract In our daily lives, newspapers are an essential information source that impacts how the public talks about present-day issues. However, effectively navigating the vast amount of news content from different newspapers and online news portals can be challenging. Newspaper headlines with sentiment analysis tell us what the news is about (e.g., politics, sports) and how the news makes us feel (positive, negative, neutral). This helps us quickly understand the emotional tone of the news. This research presents a state-of-the-art approach to Bangla news headline classification combined with sentiment analysis applying Natural Language Processing (NLP) techniques, particularly the hybrid transfer learning model BERT-CNN-BiLSTM. We have explored a dataset called BAN-ABSA of 9014 news headlines, which is the first time that has been experimented with simultaneously in the headline and sentiment categorization in Bengali newspapers. Over this imbalanced dataset, we applied two experimental strategies: technique-1, where undersampling and oversampling are applied before splitting, and technique-2, where undersampling and oversampling are applied after splitting on the In technique-1 oversampling provided the strongest performance, both headline and sentiment, that is 78.57\% and 73.43\% respectively, while technique-2 delivered the highest result when trained directly on the original imbalanced dataset, both headline and sentiment, that is 81.37\% and 64.46\% respectively. The proposed model BERT-CNN-BiLSTM significantly outperforms all baseline models in classification tasks, and achieves new state-of-the-art results for Bangla news headline classification and sentiment analysis. These results demonstrate the importance of leveraging both the headline and sentiment datasets, and provide a strong baseline for Bangla text classification in low-resource.
Authors Mirza Raquib, Munazer Montasir Akash, Tawhid Ahmed, Saydul Akbar Murad, Farida Siddiqi Prity, Mohammad Amzad Hossain, Asif Pervez Polok, Nick Rahimi
Categories Artificial Intelligence, Computation and Language
0
KAN vs LSTM Performance in Time Series Forecasting
Title CN KAN与LSTM在时间序列预测中的性能比较
Keywords 时间序列预测 · LSTM · KAN · 股票预测 · RMSE
Summary 本文比较了Kolmogorov-Arnold网络（KAN）和LSTM在非确定性股票价格数据预测中的表现，评估了预测精度与可解释性的权衡。实验结果表明，LSTM在所有预测时间范围内均显著优于标准KAN，验证了其在序列建模中的有效性；而KAN虽然理论可解释性强且计算效率高，但误差较大，实际应用受限。研究支持在高精度需求场景下采用LSTM，并建议未来探索专用KAN架构。
Reason 该论文属于纯时序预测任务，研究领域为金融股价预测，与文档图像理解（DIU）或视觉-语言模型（VLM）无关。未涉及OCR、布局分析、表格识别、视觉文本理解等任何DIU核心任务，也未提出可用于解决文档理解中幻觉、细粒度定位或复杂排版问题的通用方法。属于典型的平行应用类研究，且领域完全偏离。
Abstract This paper compares Kolmogorov-Arnold Networks (KAN) and Long Short-Term Memory networks (LSTM) for forecasting non-deterministic stock price data, evaluating predictive accuracy versus interpretability trade-offs using Root Mean Square Error (RMSE).LSTM demonstrates substantial superiority across all tested prediction horizons, confirming their established effectiveness for sequential data modelling. Standard KAN, while offering theoretical interpretability through the Kolmogorov-Arnold representation theorem, exhibits significantly higher error rates and limited practical applicability for time series forecasting. The results confirm LSTM dominance in accuracy-critical time series applications while identifying computational efficiency as KANs' primary advantage in resource-constrained scenarios where accuracy requirements are less stringent. The findings support LSTM adoption for practical financial forecasting while suggesting that continued research into specialised KAN architectures may yield future improvements.
Authors Tabish Ali Rather, S M Mahmudul Hasan Joy, Nadezda Sukhorukova, Federico Frascoli
Comment This paper compares Kolmogorov-Arnold Networks (KANs) and LSTMs for forecasting stock prices, highlighting that LSTMs provide superior predictive accuracy while KANs offer better interpretability and efficiency in limited-resource settings. Practical findings and future research directions are discussed
Categories Artificial Intelligence
0
Universality in Collective Intelligence on the Rubik's Cube
Title CN 魔方上的集体智能普遍性
Keywords 集体智能 · 专家表现 · 认知科学 · 技能学习 · 魔方
Summary 本文通过分析魔方竞速社区的数据，研究专家在解决魔方问题时的知识获取与集体学习模式，发现无论是普通解法还是盲拧，专家表现均遵循指数型进步曲线，并受到算法掌握和短期记忆瓶颈的影响。文章强调认知工具如何帮助个体与群体共同应对复杂的数学状态空间。
Reason 该论文研究的是魔方求解中的集体学习与认知行为，属于认知科学和人类技能学习领域，与文档图像理解（DIU）或视觉语言模型（VLM）技术无直接关联。未涉及OCR、文档布局分析、表格识别、视觉表示学习或任何可用于解决DIU核心痛点（如幻觉、细粒度定位、复杂排版）的方法论创新。
Abstract Progress in understanding expert performance is limited by the scarcity of quantitative data on long-term knowledge acquisition and deployment. Here we use the Rubik's Cube as a cognitive model system existing at the intersection of puzzle solving, skill learning, expert knowledge, cultural transmission, and group theory. By studying competitive cube communities, we find evidence for universality in the collective learning of the Rubik's Cube in both sighted and blindfolded conditions: expert performance follows exponential progress curves whose parameters reflect the delayed acquisition of algorithms that shorten solution paths. Blindfold solves form a distinct problem class from sighted solves and are constrained not only by expert knowledge but also by the skill improvements required to overcome short-term memory bottlenecks, a constraint shared with blindfold chess. Cognitive artifacts such as the Rubik's Cube help solvers navigate an otherwise enormous mathematical state space. In doing so, they sustain collective intelligence by integrating communal knowledge stores with individual expertise and skill, illustrating how expertise can, in practice, continue to deepen over the course of a single lifetime.
Authors David Krakauer, Gülce Kardeş, Joshua Grochow
Categories Artificial Intelligence
0
An Analysis of Constraint-Based Multi-Agent Pathfinding Algorithms
Title CN 基于约束的多智能体路径规划算法分析
Keywords 多智能体路径规划 · 约束搜索 · 冲突基搜索 · 机器人运动规划 · 算法分析
Summary 本文对基于约束的多智能体路径规划算法进行了系统分析，将约束分为保守型和激进型，并以CBS和CBSw/P为例，研究其在不同分辨率下的搜索行为。结果表明，激进型约束在高密度场景下求解能力更强，而保守型约束在成功求解时质量更高。研究还提出了一个决策流程图来指导算法选择，并强调了拓扑特征在MRMP中的重要性。
Reason 该论文属于多机器人运动规划（MRMP）和路径搜索领域，研究的是多智能体路径规划中的约束分类问题，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。其方法无法迁移用于解决OCR幻觉、密集文本识别、复杂排版解析或长文档推理等DIU核心痛点，属于无关领域工作。
Abstract This study informs the design of future multi-agent pathfinding (MAPF) and multi-robot motion planning (MRMP) algorithms by guiding choices based on constraint classification for constraint-based search algorithms. We categorize constraints as conservative or aggressive and provide insights into their search behavior, focusing specifically on vanilla Conflict-Based Search (CBS) and Conflict-Based Search with Priorities (CBSw/P). Under a hybrid grid-roadmap representation with varying resolution, we observe that aggressive (priority constraint) formulations tend to solve more instances as agent count or resolution increases, whereas conservative (motion constraint) formulations yield stronger solution quality when both succeed. Findings are synthesized in a decision flowchart, aiding users in selecting suitable constraints. Recommendations extend to Multi-Robot Motion Planning (MRMP), emphasizing the importance of considering topological features alongside problem, solution, and representation features. A comprehensive exploration of the study, including raw data and map performance, is available in our public GitHub Repository: https://GitHub.com/hannahjmlee/constraint-mapf-analysis
Authors Hannah Lee, James D. Motes, Marco Morales, Nancy M. Amato
Categories Artificial Intelligence
0
Stage-Specific Benchmarking of Deep Learning Models for Glioblastoma Follow-Up MRI
Title CN 胶质母细胞瘤随访MRI中深度学习模型的分阶段基准测试
Keywords 医学图像分析 · 深度学习 · MRI · 胶质母细胞瘤 · 基准测试
Summary 本文针对胶质母细胞瘤患者的随访MRI，提出了首个分阶段的深度学习模型基准测试，评估了包括CNN、LSTM、Transformer和Mamba等在内的11类深度学习架构在不同时间点的性能表现。结果表明，后期随访时模型判别能力有所提升，Mamba+CNN混合模型在准确率和效率之间取得了最佳平衡。但由于数据集规模小且类别不平衡，整体判别性能仍然有限。
Reason 该论文研究的是脑肿瘤（胶质母细胞瘤）在MRI影像中的进展识别问题，属于医学图像分析领域，与文档图像理解（DIU）或DocVQA无关。其任务聚焦于医学诊断中的真性进展与假性进展区分，使用的是CNN、Transformer等通用视觉模型在特定医疗数据上的性能比较，并未涉及文本布局、OCR、多模态文档理解、表格识别或语言-视觉对齐等DIU核心问题。
Abstract Differentiating true tumor progression (TP) from treatment-related pseudoprogression (PsP) in glioblastoma remains challenging, especially at early follow-up. We present the first stage-specific, cross-sectional benchmarking of deep learning models for follow-up MRI using the Burdenko GBM Progression cohort (n = 180). We analyze different post-RT scans independently to test whether architecture performance depends on time-point. Eleven representative DL families (CNNs, LSTMs, hybrids, transformers, and selective state-space models) were trained under a unified, QC-driven pipeline with patient-level cross-validation. Across both stages, accuracies were comparable (~0.70-0.74), but discrimination improved at the second follow-up, with F1 and AUC increasing for several models, indicating richer separability later in the care pathway. A Mamba+CNN hybrid consistently offered the best accuracy-efficiency trade-off, while transformer variants delivered competitive AUCs at substantially higher computational cost and lightweight CNNs were efficient but less reliable. Performance also showed sensitivity to batch size, underscoring the need for standardized training protocols. Notably, absolute discrimination remained modest overall, reflecting the intrinsic difficulty of TP vs. PsP and the dataset's size imbalance. These results establish a stage-aware benchmark and motivate future work incorporating longitudinal modeling, multi-sequence MRI, and larger multi-center cohorts.
Authors Wenhao Guo, Golrokh Mirzaei
Comment 17 pages, 11 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
A Novel Framework for Augmenting Rating Scale Tests with LLM-Scored Text Data
Title CN 一种通过LLM评分文本数据增强评级量表测试的新框架
Keywords LLM应用 · 心理测评 · 文本评分 · 量表增强 · 下游任务
Summary 该研究提出一个框架，使用大语言模型（LLM）通过简单提示语对心理测评中的自由文本回答进行打分，生成候选项目，并与基础量表联合校准，保留能提供最多测试信息的项目。在抑郁症案例研究中，该方法在学生和合成数据集上验证有效，提升了量表的精度、准确性和聚合效度。然而，该工作未涉及任何文档图像或多模态输入处理，完全基于纯文本和已有量表数据，属于心理学测量领域的应用拓展。
Reason 该论文属于心理学评估领域的下游应用，核心是利用LLM对自由文本进行评分以增强传统量表的测量性能。其方法本质是将LLM作为评分工具应用于特定领域（如抑郁评估），并未提出任何针对文档图像理解（DIU）或视觉-语言模型（VLM）架构、训练策略、多模态对齐、OCR去幻觉、细粒度定位等底层技术的创新。论文不涉及文档结构、布局分析、表格识别、视觉token压缩或高分辨率处理等DIU核心问题，属于典型的‘平行应用’（Wrapper/Application），无技术源头价值。
Abstract Psychological assessments are dominated by rating scales, which cannot capture the nuance in natural language. Efforts to supplement them with qualitative text have relied on labelled datasets or expert rubrics, limiting scalability. We introduce a framework that avoids this reliance: large language models (LLMs) score free-text responses with simple prompts to produce candidate LLM items, from which we retain those that yield the most test information when co-calibrated with a baseline scale. Using depression as a case study, we developed and tested the method in upper-secondary students (n=693) and a matched synthetic dataset (n=3,000). Results on held-out test sets showed that augmenting a 19-item scale with LLM items improved its precision, accuracy, and convergent validity. Further, the test information gain matched that of adding as many as 16 rating-scale items. This framework leverages the increasing availability of transcribed language to enhance psychometric measures, with applications in clinical health and beyond.
Authors Joe Watson, Ivan O'Connor, Chia-Wen Chen, Luning Sun, Fang Luo, David Stillwell
Categories Artificial Intelligence, Computation and Language
0
Strategic Decision Framework for Enterprise LLM Adoption
Title CN 企业采用大语言模型的战略决策框架
Keywords LLM adoption · enterprise strategy · decision framework · business integration
Summary 本文提出了一个六步制的企业级大语言模型（LLM）采用决策框架，旨在帮助组织从应用场景选择到最终部署过程中做出关键决策。框架涵盖数据安全、开发方式、基础设施和部署策略等方面，并结合B2B与B2C领域的实际案例提供实践指导。重点面向医疗、金融和软件等行业在合规性与安全性约束下的LLM集成问题。
Reason 该论文属于LLM在企业管理中的下游应用研究，聚焦于企业如何制定LLM采用策略，涉及内容生成、客户服务自动化等业务场景的部署决策。其核心贡献是管理层面的决策框架，而非文档图像理解（DIU）或视觉-语言模型的技术创新。未提出任何可用于解决OCR幻觉、复杂排版解析或文档VQA任务的新方法，属于典型的‘平行应用’，不符合本研究方向的技术源头筛选标准。
Abstract Organizations are rapidly adopting Large Language Models (LLMs) to transform their operations, yet they lack clear guidance on key decisions for adoption and implementation. While LLMs offer powerful capabilities in content generation, assisted coding, and process automation, businesses face critical challenges in data security, LLM solution development approach, infrastructure requirements, and deployment strategies. Healthcare providers must protect patient data while leveraging LLMs for medical analysis, financial institutions need to balance automated customer service with regulatory compliance, and software companies seek to enhance development productivity while maintaining code security. This article presents a systematic six-step decision framework for LLM adoption, helping organizations navigate from initial application selection to final deployment. Based on extensive interviews and analysis of successful and failed implementations, our framework provides practical guidance for business leaders to align technological capabilities with business objectives. Through key decision points and real-world examples from both B2B and B2C contexts, organizations can make informed decisions about LLM adoption while ensuring secure and efficient integration across various use cases, from customer service automation to content creation and advanced analytics.
Authors Michael Trusov, Minha Hwang, Zainab Jamal, Swarup Chandra
Comment 14 pages, 1 key figure
Categories Artificial Intelligence
0
Barriers to AI Adoption: Image Concerns at Work
Title CN 人工智能采纳的障碍：工作中的形象担忧
Keywords AI adoption · human-AI collaboration · organizational behavior · image concerns · labor market
Summary 本文通过一项在线劳动力市场的实地实验，研究员工在AI辅助任务中为何不愿采纳AI建议。结果表明，当AI使用行为对评估者可见时，员工会出于维护自身专业形象的动机而降低对AI的依赖，从而导致任务表现下降。这种现象反映了人类对AI协作中的信任与自我呈现之间的张力。
Reason 该论文研究的是人因问题，即员工在使用AI辅助完成图像分类任务时，因担心被评估者认为过度依赖AI而主动减少对AI的采纳。其核心关注点是组织行为学与激励机制设计，而非文档图像理解（DIU）或视觉语言模型（VLM）的技术改进。既不涉及DocVQA、OCR、版面分析等DIU任务，也未提出可迁移至文档理解领域的底层方法，属于社会科学研究范畴。
Abstract Concerns about how workers are perceived can deter effective collaboration with artificial intelligence (AI). In a field experiment on a large online labor market, I hired 450 U.S.-based remote workers to complete an image-categorization job assisted by AI recommendations. Workers were incentivized by the prospect of a contract extension based on an HR evaluator's feedback. I find that workers adopt AI recommendations at lower rates when their reliance on AI is visible to the evaluator, resulting in a measurable decline in task performance. The effects are present despite a conservative design in which workers know that the evaluator is explicitly instructed to assess expected accuracy on the same AI-assisted task. This reduction in AI reliance persists even when the evaluator is reassured about workers' strong performance history on the platform, underscoring how difficult these concerns are to alleviate. Leveraging the platform's public feedback feature, I introduce a novel incentive-compatible elicitation method showing that workers fear heavy reliance on AI signals a lack of confidence in their own judgment, a trait they view as essential when collaborating with AI.
Authors David Almog
Categories Artificial Intelligence
0
Re(Visiting) Time Series Foundation Models in Finance
Title CN 重访金融领域的时间序列基础模型
Keywords 时间序列基础模型 · 金融预测 · 零样本学习 · 领域适配 · 预训练
Summary 本文对金融领域的时间序列基础模型（TSFM）进行了实证研究，发现直接使用预训练模型效果不佳，而基于金融数据从头预训练能显著提升预测性能。研究强调了领域适配的重要性，但未提出可泛化的方法创新。
Reason 该论文研究的是金融时间序列预测中的基础模型应用，属于特定领域的下游任务应用（平行应用），并未提出可用于文档图像理解的通用方法论或底层技术创新。其内容与文档理解中的OCR、布局分析、表格识别等核心问题无关，且未涉及多模态、视觉表示或VLM架构改进，无法迁移解决DIU的痛点如幻觉、复杂排版或长文档推理。
Abstract Financial time series forecasting is central to trading, portfolio optimization, and risk management, yet it remains challenging due to noisy, non-stationary, and heterogeneous data. Recent advances in time series foundation models (TSFMs), inspired by large language models, offer a new paradigm for learning generalizable temporal representations from large and diverse datasets. This paper presents the first comprehensive empirical study of TSFMs in global financial markets. Using a large-scale dataset of daily excess returns across diverse markets, we evaluate zero-shot inference, fine-tuning, and pre-training from scratch against strong benchmark models. We find that off-the-shelf pre-trained TSFMs perform poorly in zero-shot and fine-tuning settings, whereas models pre-trained from scratch on financial data achieve substantial forecasting and economic improvements, underscoring the value of domain-specific adaptation. Increasing the dataset size, incorporating synthetic data augmentation, and applying hyperparameter tuning further enhance performance.
Authors Eghbal Rahimikia, Hao Ni, Weiguan Wang
Categories Artificial Intelligence
0
Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Study
Title CN 大语言模型单元测试生成的参数高效微调：一项实证研究
Keywords 参数高效微调 · 大语言模型 · 单元测试生成 · LoRA · 代码生成
Summary 本文研究了多种参数高效微调方法（如LoRA、(IA)^3、Prompt Tuning）在大语言模型生成单元测试任务上的效果，比较了其与全量微调在语法正确性、CodeBLEU、通过率和测试覆盖率等指标上的表现。结果显示LoRA可达到与全量微调相当的效果，而Prompt Tuning在训练成本上更具优势，但微调后的模型可能生成更多不可执行的测试用例。
Reason 该论文聚焦于软件工程中的单元测试生成任务，属于纯代码生成与程序分析领域，未涉及任何文档图像、多模态输入或视觉内容理解。尽管使用了PEFT技术（如LoRA），但其应用场景与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂版面分析、视觉-文本对齐）完全无关，无法迁移用于解决DIU问题。
Abstract Parameter-efficient fine-tuning (PEFT) methods, which fine-tune only a subset of model parameters, offer a promising solution by reducing the computational costs of tuning large language models (LLMs) while maintaining their performance. Existing studies have explored using PEFT and LLMs for various code-related tasks and found that the effectiveness of PEFT techniques is task-dependent. The state-of-the-art is limited to using LLMs with full fine-tuning to generate unit tests. The application of PEFT techniques in unit test generation remains underexplored. This paper investigates both full fine-tuning and various PEFT methods, including LoRA, (IA)^3, and prompt tuning, across thirteen models of different architectures and sizes. We use well-established benchmark datasets to evaluate their effectiveness in unit test generation and measure syntax correctness, CodeBLEU, pass@1, instruction coverage, branch coverage, and mutation score of the generated tests. Our findings show that LoRA can deliver performance comparable to full fine-tuning for unit test generation in several cases. If training costs are valued, prompt tuning is the most cost-effective approach, particularly for large models. However, the models tuned with full fine-tuning or PEFT may generate fewer executable test cases than the baseline model because they generate more tests calling nonexistent methods or having type mismatches. For the generated ones that are executable, the ones from the tuned models show better test coverage than those from the baseline model.
Authors André Storhaug, Jingyue Li
Comment 26 pages, 2 figures, 6 tables, 1 listing
Categories Artificial Intelligence
0
Accelerating Goal-Conditioned RL Algorithms and Research ICLR 2025
Title CN 加速目标条件强化学习算法与研究
Keywords 强化学习 · 自监督学习 · 对比学习 · GPU加速 · JaxGCRL
Summary 本文提出一个高性能的目标条件强化学习代码库JaxGCRL，利用GPU加速回放缓冲区、环境模拟和稳定的对比学习算法，显著缩短训练时间，并为自监督强化学习研究提供基准支持。
Reason 该论文属于具身智能与强化学习领域，研究目标是提升目标条件强化学习（GCRL）的训练效率和稳定性，涉及环境模拟、对比学习算法优化等。其核心内容与文档图像理解（DIU）任务无直接关联，既不解决OCR幻觉、复杂排版、细粒度定位等DIU痛点，也未提出可迁移至多模态文档理解的通用方法。属于‘无关领域’，根据负面清单应直接打0分。
Abstract Self-supervision has the potential to transform reinforcement learning (RL), paralleling the breakthroughs it has enabled in other areas of machine learning. While self-supervised learning in other domains aims to find patterns in a fixed dataset, self-supervised goal-conditioned reinforcement learning (GCRL) agents discover new behaviors by learning from the goals achieved during unstructured interaction with the environment. However, these methods have failed to see similar success, both due to a lack of data from slow environment simulations as well as a lack of stable algorithms. We take a step toward addressing both of these issues by releasing a high-performance codebase and benchmark (JaxGCRL) for self-supervised GCRL, enabling researchers to train agents for millions of environment steps in minutes on a single GPU. By utilizing GPU-accelerated replay buffers, environments, and a stable contrastive RL algorithm, we reduce training time by up to $22\times$. Additionally, we assess key design choices in contrastive RL, identifying those that most effectively stabilize and enhance training performance. With this approach, we provide a foundation for future research in self-supervised GCRL, enabling researchers to quickly iterate on new ideas and evaluate them in diverse and challenging environments. Website + Code: https://github.com/MichalBortkiewicz/JaxGCRL
Authors Michał Bortkiewicz, Władysław Pałucki, Vivek Myers, Tadeusz Dziarmaga, Tomasz Arczewski, Łukasz Kuciński, Benjamin Eysenbach
Comment Published at ICLR 2025 (Spotlight). Website: https://michalbortkiewicz.github.io/JaxGCRL/ Code: https://github.com/MichalBortkiewicz/JaxGCRL
Categories Artificial Intelligence
0
Foundations of Artificial Intelligence Frameworks: Notion and Limits of AGI
Title CN 人工智能框架的基础：AGI的概念与局限
Keywords 人工通用智能 · 神经网络局限 · 哲学批判
Summary 本文从哲学和理论角度批判当前基于神经网络的人工智能范式，认为其在架构上缺乏真正的理解能力，无法通向人工通用智能（AGI）。作者提出需要具备动态重构能力的新框架，但未给出具体实现路径或技术方案。
Reason 该论文属于哲学性、理论性的AI基础讨论，涉及中文房间论证、哥德尔定理等，并质疑当前神经网络范式无法实现真正理解。其内容不涉及任何文档图像理解（DIU）任务、多模态模型架构设计、视觉表示学习或可迁移的推理机制。论文未提出可用于解决OCR幻觉、复杂排版解析或长文档推理的技术方法，且无具体VLM或多模态实验支撑，完全脱离DIU技术主线。
Abstract Within the limited scope of this paper, we argue that artificial general intelligence cannot emerge from current neural network paradigms regardless of scale, nor is such an approach healthy for the field at present. Drawing on various notions, discussions, present-day developments and observations, current debates and critiques, experiments, and so on in between philosophy, including the Chinese Room Argument and Gödelian argument, neuroscientific ideas, computer science, the theoretical consideration of artificial intelligence, and learning theory, we address conceptually that neural networks are architecturally insufficient for genuine understanding. They operate as static function approximators of a limited encoding framework - a 'sophisticated sponge' exhibiting complex behaviours without structural richness that constitute intelligence. We critique the theoretical foundations the field relies on and created of recent times; for example, an interesting heuristic as neural scaling law (as an example, arXiv:2001.08361 ) made prominent in a wrong way of interpretation, The Universal Approximation Theorem addresses the wrong level of abstraction and, in parts, partially, the question of current architectures lacking dynamic restructuring capabilities. We propose a framework distinguishing existential facilities (computational substrate) from architectural organization (interpretive structures), and outline principles for what genuine machine intelligence would require, and furthermore, a conceptual method of structuralizing the richer framework on which the principle of neural network system takes hold.
Authors Khanh Gia Bui
Comment 49 pages, 4 pictures
Categories Artificial Intelligence
0
A Comprehensive Evaluation of Large Language Models on Mental Illnesses
Title CN 大语言模型在精神疾病领域的综合评估
Keywords 大语言模型 · 心理健康 · 精神疾病检测 · 少样本学习 · 提示工程
Summary 本文评估了33个大语言模型在心理健康任务中的表现，包括精神疾病检测、严重程度评估和精神病学知识测试，使用社交媒体数据进行零样本和少样本实验。结果显示GPT-4、Llama 3等模型在部分任务上表现优异，提示工程对性能有显著影响，但伦理限制影响了部分敏感问题的回答能力。
Reason 该论文属于纯NLP在医疗心理领域的下游应用，研究的是如何用现有LLM进行精神疾病检测与评估，属于典型的‘平行应用’（Wrapper/Application），并未提出任何针对文档图像理解（DIU）或视觉-语言模型底层技术的创新。尽管涉及LLM，但任务完全基于文本输入，无视觉模态、无文档结构、无OCR或布局分析相关内容，对DIU领域无迁移价值。
Abstract Large Language Models (LLMs) have shown promise in various domains, including healthcare, with significant potential to transform mental health applications by enabling scalable and accessible solutions. This study aims to provide a comprehensive evaluation of 33 LLMs, ranging from 2 billion to 405+ billion parameters, in performing key mental health tasks using social media data across six datasets. To our knowledge, this represents the largest-scale systematic evaluation of modern LLMs for mental health applications. Models such as GPT-4, Llama 3, Claude, Gemma, Gemini, and Phi-3 were assessed for their zero-shot (ZS) and few-shot (FS) capabilities across three tasks: binary disorder detection, disorder severity evaluation, and psychiatric knowledge assessment. Key findings revealed that models like GPT-4 and Llama 3 exhibited superior performance in binary disorder detection, achieving accuracies up to 85% on certain datasets, while FS learning notably enhanced disorder severity evaluations, reducing the Mean Absolute Error (MAE) by 1.3 points for the Phi-3-mini model. Recent models, such as Llama 3.1 405b, demonstrated exceptional psychiatric knowledge assessment accuracy at 91.2%, while prompt engineering played a crucial role in improving performance across tasks. However, the ethical constraints imposed by many LLM providers limit their ability to respond to sensitive queries, hampering comprehensive performance evaluations. This work highlights both the capabilities and limitations of LLMs in mental health contexts, offering valuable insights for future applications in psychiatry.
Authors Abdelrahman Hanafi, Mohammed Saad, Noureldin Zahran, Radwa J. Hanafy, Mohammed E. Fouda
Categories Artificial Intelligence
0
PsychiatryBench: A Multi-Task Benchmark for LLMs in Psychiatry
Title CN PsychiatryBench：用于精神病学中大语言模型的多任务基准
Keywords 大语言模型 · 精神病学 · 基准测试 · 临床推理 · 医学AI
Summary 本文提出了PsychiatryBench，一个基于权威精神病学教材和案例书构建的多任务评测基准，包含5188个标注样本，涵盖诊断推理、治疗规划、随访管理等多种任务，用于评估大语言模型在精神病学领域的性能。研究使用多种前沿闭源和开源医学LLM进行测试，揭示了现有模型在临床一致性与安全性方面的不足。
Reason 该论文属于特定领域的下游应用（精神病学），构建了一个面向临床精神诊断的LLM评测基准。其核心贡献是领域专用的数据集和评测框架，并未提出任何可迁移至文档图像理解（DIU）的方法论创新。任务聚焦于纯文本问答与医学推理，不涉及视觉输入、OCR、布局分析或任何多模态文档理解要素，完全脱离DIU技术主线。
Abstract Large language models (LLMs) offer significant potential in enhancing psychiatric practice, from improving diagnostic accuracy to streamlining clinical documentation and therapeutic support. However, existing evaluation resources heavily rely on small clinical interview corpora, social media posts, or synthetic dialogues, which limits their clinical validity and fails to capture the full complexity of diagnostic reasoning. In this work, we introduce PsychiatryBench, a rigorously curated benchmark grounded exclusively in authoritative, expert-validated psychiatric textbooks and casebooks. PsychiatryBench comprises eleven distinct question-answering tasks ranging from diagnostic reasoning and treatment planning to longitudinal follow-up, management planning, clinical approach, sequential case analysis, and multiple-choice/extended matching formats totaling 5,188 expert-annotated items. {\color{red}We evaluate a diverse set of frontier LLMs (including Google Gemini, DeepSeek, Sonnet 4.5, and GPT 5) alongside leading open-source medical models such as MedGemma using both conventional metrics and an "LLM-as-judge" similarity scoring framework. Our results reveal substantial gaps in clinical consistency and safety, particularly in multi-turn follow-up and management tasks, underscoring the need for specialized model tuning and more robust evaluation paradigms. PsychiatryBench offers a modular, extensible platform for benchmarking and improving LLM performance in mental health applications.
Authors Aya E. Fouda, Abdelrahamn A. Hassan, Radwa J. Hanafy, Mohammed E. Fouda
Categories Artificial Intelligence, Computation and Language
0
Find Them All: Unveiling MLLMs for Versatile Person Re-identification
Title CN 找出所有人：揭示多模态大语言模型在多功能人物重识别中的应用
Keywords 人物重识别 · 多模态大语言模型 · 跨模态检索 · 基准测试 · VP-ReID
Summary 本文提出了一个名为VP-ReID的多功能人物重识别新基准，包含257,310个多模态查询和图库图像，覆盖十种不同的人物重识别任务，并设计了两种面向任务的评估方案。实验表明多模态大语言模型（MLLMs）在多种人物重识别任务中具有出色的通用性、有效性与可解释性，但在处理热成像和红外等特定模态时仍存在局限。
Reason 该论文聚焦于人物重识别（ReID）任务，属于计算机视觉中的人体分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了多模态大模型（MLLMs），但其应用场景、数据模态（如热成像、红外）和核心目标均远离文档理解的核心痛点，如OCR幻觉、复杂版式解析或长文档推理。无法迁移其方法至DIU领域。
Abstract Person re-identification (ReID) aims to retrieve images of a target person from the gallery set, with wide applications in medical rehabilitation and public security. However, traditional person ReID models are typically uni-modal, resulting in limited generalizability across heterogeneous data modalities. Recently, the emergence of multi-modal large language models (MLLMs) has shown a promising avenue for addressing this issue. Despite this potential, existing methods merely regard MLLMs as feature extractors or caption generators, leaving their capabilities in person ReID tasks largely unexplored. To bridge this gap, we introduce a novel benchmark for \underline{\textbf{V}}ersatile \underline{\textbf{P}}erson \underline{\textbf{Re}}-\underline{\textbf{ID}}entification, termed VP-ReID. The benchmark includes 257,310 multi-modal queries and gallery images, covering ten diverse person ReID tasks. In addition, we propose two task-oriented evaluation schemes for MLLM-based person ReID. Extensive experiments demonstrate the impressive versatility, effectiveness, and interpretability of MLLMs in various person ReID tasks. Nevertheless, they also have limitations in handling a few modalities, particularly thermal and infrared data. We hope that VP-ReID can facilitate the community in developing more robust and generalizable cross-modal foundation models for person ReID.
Authors Jinhao Li, Zijian Chen, Lirong Deng, Guangtao Zhai, Changbo Wang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
PDDFormer: Pairwise Distance Distribution Graph Transformer for Crystal Material Property Prediction
Title CN PDDFormer：用于晶体材料性质预测的成对距离分布图Transformer
Keywords 晶体结构 · 图神经网络 · 距离分布 · 材料性质预测 · AI for Science
Summary 本文提出了一种基于成对距离分布（PDD）的晶体结构表示方法WPDD和UPDD，构建多边晶体图并结合矩阵式消息传递机制，在原子位置微扰下保持图结构的连续性和完整性，显著降低计算成本，并在材料性质预测任务上达到SOTA性能。
Reason 该论文研究的是晶体材料科学领域的结构表示与性质预测问题，属于AI for Science（AISci）方向，与文档图像理解（DIU）或视觉语言模型（VLM）在任务、数据模态和应用场景上均无关联。尽管使用了图神经网络和距离分布等数学建模方法，但其核心目标是处理三维原子点集的连续性与稳定性问题，无法迁移解决OCR幻觉、复杂排版解析、细粒度文本定位等DIU核心痛点。
Abstract Crystal structures can be simplified as a periodic point set that repeats across three-dimensional space along an underlying lattice. Traditionally, crystal representation methods characterize the structure using descriptors such as lattice parameters, symmetry, and space groups. However, in reality, atoms in materials always vibrate above absolute zero, causing their positions to fluctuate continuously. This dynamic behavior disrupts the fundamental periodicity of the lattice, making crystal graphs based on static lattice parameters and conventional descriptors discontinuous under slight perturbations. Chemists proposed the pairwise distance distribution (PDD) method to address this problem. However, the completeness of PDD requires defining a large number of neighboring atoms, leading to high computational costs. Additionally, PDD does not account for atomic information, making it challenging to apply it directly to crystal material property prediction tasks. To tackle these challenges, we introduce the atom-Weighted Pairwise Distance Distribution (WPDD) and Unit cell Pairwise Distance Distribution (UPDD) and apply them to the construction of multi-edge crystal graphs. We demonstrate the continuity and general completeness of crystal graphs under slight atomic position perturbations. Moreover, by modeling PDD as global information and integrating it into matrix-based message passing, we significantly reduce computational costs. Comprehensive evaluation results show that WPDDFormer achieves state-of-the-art predictive accuracy across tasks on benchmark datasets such as the Materials Project and JARVIS-DFT.
Authors Xiangxiang Shen, Zheng Wan, Lingfeng Wen, Licheng Sun, Jian Yang, Xuan Tang, Shing-Ho J. Lin, Xiao He, Mingsong Chen, Xian Wei
Comment 15pages,8 figures
Categories Artificial Intelligence
0
Evaluating perturbation robustnessof generative systems that use COBOL code inputs AI-SQE (ICSE, 2026)
Title CN 评估使用COBOL代码输入的生成系统在扰动下的鲁棒性
Keywords COBOL · 代码翻译 · 鲁棒性评估 · 大语言模型 · 软件质量
Summary 本文提出了一种评估基于大语言模型的COBOL到Java代码翻译系统对输入扰动鲁棒性的框架，构建了COBOL程序的扰动库并扩展了基准数据集，通过度量输出变化来评估系统敏感性，并提供了可视化工具辅助调试。研究重点在于代码语义保持与软件质量评估。
Reason 该论文聚焦于编程语言（COBOL与Java）之间的翻译任务，属于软件工程与代码生成领域，尽管使用了LLM组件，但其核心问题和数据均远离文档图像理解（DIU）或视觉-语言多模态任务。未涉及OCR、视觉布局、表格识别、文档VQA等DIU核心任务，也未提出可迁移至DIU领域的通用方法（如视觉表示、高分辨率处理、去幻觉机制等）。属于特定领域的下游应用研究，不符合本方向的技术源头筛选标准。
Abstract Systems incorporating large language models (LLMs) as a component are known to be sensitive (i.e., non-robust) to minor input variations that do not change the meaning of the input; such sensitivity may reduce the system's usefulness. Here, we present a framework to evaluate robustness of systems using COBOL code as input; our application is translation between COBOL and Java programming languages, but the approach extends to other tasks such as code generation or explanation. Targeting robustness of systems with COBOL as input is essential yet challenging. Many business-critical applications are written in COBOL, yet these are typically proprietary legacy applications and their code is unavailable to LLMs for training. We develop a library of COBOL paragraph and full-program perturbation methods, and create variant-expanded versions of a benchmark dataset of examples for a specific task. The robustness of the LLM-based system is evaluated by measuring changes in values of individual and aggregate metrics calculated on the system's outputs. Finally, we present a series of dynamic table and chart visualization dashboards that assist in debugging the system's outputs, and monitoring and understanding root causes of the system's sensitivity to input variation. These tools can be further used to improve the system by, for instance, indicating variations that should be handled by pre-processing steps.
Authors Samuel Ackerman, Wesam Ibraheem, Orna Raz, Marcel Zalmanovici
Comment 16 pages (8 main, 8 appendix). Accepted to AI-SQE (ICSE, 2026): The 1st International Workshop on AI for Software Quality Evaluation: Judgment, Metrics, Benchmarks, and Beyond
Categories Artificial Intelligence
0
InstructAudio: Unified speech and music generation with natural language instruction
Title CN InstructAudio：通过自然语言指令实现统一的语音和音乐生成
Keywords 语音生成 · 音乐生成 · 自然语言指令 · 扩散模型 · 多模态生成
Summary 本文提出了InstructAudio，一个基于自然语言指令统一控制语音和音乐生成的框架，支持音色、情感、风格、流派、节奏等属性的文本级控制。模型采用联合扩散变换器结构，在大规模语音和音乐数据上训练，实现了跨模态生成能力。虽然在音频生成方面具有创新性，但与文档图像理解任务无关。
Reason 该论文研究的是语音和音乐的生成任务，属于音频生成领域，与文档图像理解（DIU）或视觉-语言模型在文档上的应用无直接关联。尽管其采用多模态指令控制和跨模态对齐思想，但方法面向的是声学特征建模，无法迁移解决文档图像中的OCR幻觉、复杂排版、细粒度定位等核心问题。属于无关领域。
Abstract Text-to-speech (TTS) and text-to-music (TTM) models face significant limitations in instruction-based control. TTS systems usually depend on reference audio for timbre, offer only limited text-level attribute control, and rarely support dialogue generation. TTM systems are constrained by input conditioning requirements that depend on expert knowledge annotations. The high heterogeneity of these input control conditions makes them difficult to joint modeling with speech synthesis. Despite sharing common acoustic modeling characteristics, these two tasks have long been developed independently, leaving open the challenge of achieving unified modeling through natural language instructions. We introduce InstructAudio, a unified framework that enables instruction-based (natural language descriptions) control of acoustic attributes including timbre (gender, age), paralinguistic (emotion, style, accent), and musical (genre, instrument, rhythm, atmosphere). It supports expressive speech, music, and dialogue generation in English and Chinese. The model employs joint and single diffusion transformer layers with a standardized instruction-phoneme input format, trained on 50K hours of speech and 20K hours of music data, enabling multi-task learning and cross-modal alignment. Fig. 1 visualizes performance comparisons with mainstream TTS and TTM models, demonstrating that InstructAudio achieves optimal results on most metrics. To our best knowledge, InstructAudio represents the first instruction-controlled framework unifying speech and music generation. Audio samples are available at: https://qiangchunyu.github.io/InstructAudio/
Authors Chunyu Qiang, Kang Yin, Xiaopeng Wang, Yuzhe Liang, Jiahui Zhao, Ruibo Fu, Tianrui Wang, Cheng Gong, Chen Zhang, Longbiao Wang, Jianwu Dang
Categories Artificial Intelligence, Computation and Language
0
Shadows in the Code: Exploring the Risks and Defenses of LLM-based Multi-Agent Software Development Systems AAAI 2026 Alignment Track
Title CN 代码中的阴影：探索基于大语言模型的多智能体软件开发系统的风险与防御
Keywords 多智能体系统 · 安全攻击 · 隐式恶意行为 · LLM应用 · 软件开发
Summary 本文研究了基于大语言模型的多智能体系统在软件开发中的安全漏洞，提出了两种风险场景：恶意用户与良性智能体（MU-BA）和良性用户与恶意智能体（BU-MA），并设计了一种隐式恶意行为注入攻击（IMBIA）来测试系统的脆弱性。实验表明多个主流框架存在较高攻击成功率，作者进而提出防御机制Adv-IMBIA以降低风险。研究重点在于系统安全性而非模型能力提升。
Reason 该论文研究的是多智能体系统在软件开发中的安全风险与防御机制，属于具身智能/Agent在软件工程中的垂直应用，属于‘平行下游应用’。其核心关注点为隐式恶意行为注入攻击（IMBIA）和防御策略Adv-IMBIA，应用场景局限于软件开发流程，并未提出可用于文档图像理解（如OCR去幻觉、复杂排版建模、视觉表征学习等）的底层方法或可迁移技术。尽管涉及Agent架构，但并非针对长文档推理、信息定位或视觉-语言对齐等DIU痛点设计，无法迁移用于提升文档理解性能。
Abstract The rapid advancement of Large Language Model (LLM)-driven multi-agent systems has significantly streamlined software developing tasks, enabling users with little technical expertise to develop executable applications. While these systems democratize software creation through natural language requirements, they introduce significant security risks that remain largely unexplored. We identify two risky scenarios: Malicious User with Benign Agents (MU-BA) and Benign User with Malicious Agents (BU-MA). We introduce the Implicit Malicious Behavior Injection Attack (IMBIA), demonstrating how multi-agent systems can be manipulated to generate software with concealed malicious capabilities beneath seemingly benign applications, and propose Adv-IMBIA as a defense mechanism. Evaluations across ChatDev, MetaGPT, and AgentVerse frameworks reveal varying vulnerability patterns, with IMBIA achieving attack success rates of 93%, 45%, and 71% in MU-BA scenarios, and 71%, 84%, and 45% in BU-MA scenarios. Our defense mechanism reduced attack success rates significantly, particularly in the MU-BA scenario. Further analysis reveals that compromised agents in the coding and testing phases pose significantly greater security risks, while also identifying critical agents that require protection against malicious user exploitation. Our findings highlight the urgent need for robust security measures in multi-agent software development systems and provide practical guidelines for implementing targeted, resource-efficient defensive strategies.
Authors Xiaoqing Wang, Keman Huang, Bin Liang, Hongyu Li, Xiaoyong Du
Comment Accepted by AAAI 2026 Alignment Track
Categories Artificial Intelligence, Computation and Language
0
Learn from Global Correlations: Enhancing Evolutionary Algorithm via Spectral GNN AAAI
Title CN 从全局相关性学习：通过谱图神经网络增强进化算法
Keywords 进化算法 · 图神经网络 · 频域分析 · 优化算法 · 探索与利用
Summary 本文提出了一种名为Graph Neural Evolution (GNE) 的新型进化算法框架，通过将种群建模为图并利用谱图神经网络对进化信号进行频域分解，实现对高频（多样性）和低频（一致性）信息的显式过滤，从而自动控制探索与开发的平衡。在多个基准函数上取得了优于传统和先进进化算法的结果。
Reason 该论文属于优化算法领域，研究如何改进进化算法的探索与利用平衡，尽管使用了图神经网络，但其方法和任务均与文档图像理解（DIU）或视觉-语言模型（VLM）无直接关联。未涉及OCR、文档布局分析、表格识别、多模态推理等DIU核心问题，也无法迁移用于解决文档理解中的幻觉、细粒度定位或长文档推理等痛点。
Abstract Evolutionary algorithms (EAs) simulate natural selection but have two main limitations: (1) they rarely update individuals based on global correlations, limiting comprehensive learning; (2) they struggle with balancing exploration and exploitation, where excessive exploitation causes premature convergence, and excessive exploration slows down the search. Moreover, EAs often depend on manual parameter settings, which can disrupt the exploration-exploitation balance. To address these issues, we propose Graph Neural Evolution (GNE), a novel EA framework. GNE represents the population as a graph, where nodes represent individuals, and edges capture their relationships, enabling global information usage. GNE utilizes spectral graph neural networks (GNNs) to decompose evolutionary signals into frequency components, applying a filtering function to fuse these components. High-frequency components capture diverse global information, while low-frequency ones capture more consistent information. This explicit frequency filtering strategy directly controls global-scale features through frequency components, overcoming the limitations of manual parameter settings and making the exploration-exploitation control more interpretable and manageable. Tests on nine benchmark functions (e.g., Sphere, Rastrigin, Rosenbrock) show that GNE outperforms classical (GA, DE, CMA-ES) and advanced algorithms (SDAES, RL-SHADE) under various conditions, including noise-corrupted and optimal solution deviation scenarios. GNE achieves solutions several orders of magnitude better (e.g., 3.07e-20 mean on Sphere vs. 1.51e-07).
Authors Kaichen Ouyang, Zong Ke, Shengwei Fu, Lingjie Liu, Puning Zhao, Dayu Hu
Comment Accepted by the 40th Annual AAAI Conference on Artificial Intelligence
Categories Artificial Intelligence
0
RegDeepLab: A Two-Stage Decoupled Framework for Interpretable Embryo Fragmentation Grading
Title CN RegDeepLab：一种用于可解释胚胎碎片分级的两阶段解耦框架
Keywords 医学图像分析 · 多任务学习 · 语义分割 · 回归模型 · 可解释性
Summary 本文提出RegDeepLab，一个结合DeepLabV3+语义分割与多尺度回归头的双分支多任务学习框架，用于自动化评估体外受精中胚胎的碎片化程度。为解决多任务训练中的梯度冲突问题，设计了两阶段解耦训练策略，并引入Range Loss支持半监督学习。实验表明该方法在保持高分割精度的同时实现了准确的分级预测，兼顾了性能与临床可解释性。
Reason 该论文属于医学图像分析领域的特定应用，研究任务为胚胎碎片分级，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。尽管使用了多任务学习和解耦训练策略，但其方法创新紧密绑定于生物医学图像的语义分割与回归任务，无法直接迁移解决文档理解中的核心痛点（如OCR幻觉、复杂排版、长文档推理等）。属于典型的垂直领域应用，无DIU方法论贡献。
Abstract The degree of embryo fragmentation serves as a critical morphological indicator for assessing embryo developmental potential in In Vitro Fertilization (IVF) clinical decision-making. However, current manual grading processes are not only time-consuming but also limited by significant inter-observer variability and efficiency bottlenecks. Although deep learning has demonstrated potential in automated grading in recent years, existing solutions face a significant challenge: pure regression models lack the visual explainability required for clinical practice, while pure segmentation models struggle to directly translate pixel-level masks into precise clinical grades. This study proposes RegDeepLab, a dual-branch Multi-Task Learning (MTL) framework that integrates State-of-the-Art (SOTA) semantic segmentation (DeepLabV3+) with a multi-scale regression head. Addressing the common issues of "Gradient Conflict" and "Negative Transfer" in multi-task training, we propose a "Two-Stage Decoupled Training Strategy." Experimental results demonstrate that while standard end-to-end MTL training can minimize grading error (MAE=0.046) through our designed "Feature Injection" mechanism, it compromises the integrity of segmentation boundaries. In contrast, our decoupled strategy successfully provides robust and high-precision grading predictions while preserving SOTA-level segmentation accuracy (Dice=0.729). Furthermore, we introduce a "Range Loss" to effectively utilize large-scale discrete grading data for semi-supervised learning. This study ultimately presents a dual-module clinical auxiliary solution that combines high accuracy with visual explainability.
Authors Ming-Jhe Lee
Comment 7 pages, 5 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Error analysis for the deep Kolmogorov method
Title CN Kolmogorov方法的深度误差分析
Keywords 偏微分方程 · 深度学习 · 误差分析 · Kolmogorov方程 · 收敛性
Summary 本文对用于求解热传导型偏微分方程的深度Kolmogorov方法进行了误差分析，从网络结构规模、采样点数量和优化误差三个维度推导了近似解与真实解之间的收敛性。工作聚焦于数学理论分析，不涉及任何实际视觉文档数据或多模态建模范畴。
Reason 该论文研究的是基于深度学习的偏微分方程（PDE）求解方法的误差分析，属于科学计算与理论机器学习范畴，与文档图像理解（DIU）、多模态大模型（VLM）、OCR、DocVQA等任务完全无关。未涉及任何视觉输入处理、文本理解或跨模态推理，无法迁移解决DIU中的OCR幻觉、复杂排版或长文档推理等核心问题。
Abstract The deep Kolmogorov method is a simple and popular deep learning based method for approximating solutions of partial differential equations (PDEs) of the Kolmogorov type. In this work we provide an error analysis for the deep Kolmogorov method for heat PDEs. Specifically, we reveal convergence with convergence rates for the overall mean square distance between the exact solution of the heat PDE and the realization function of the approximating deep neural network (DNN) associated with a stochastic optimization algorithm in terms of the size of the architecture (the depth/number of hidden layers and the width of the hidden layers) of the approximating DNN, in terms of the number of random sample points used in the loss function (the number of input-output data pairs used in the loss function), and in terms of the size of the optimization error made by the employed stochastic optimization method.
Authors Iulian Cîmpean, Thang Do, Lukas Gonon, Arnulf Jentzen, Ionel Popescu
Comment 40 pages
Categories Artificial Intelligence
0
Backdoors in Conditional Diffusion: Threats to Responsible Synthetic Data Pipelines RDS @ AAAI 2026
Title CN 条件扩散中的后门：对负责任合成数据管道的威胁
Keywords 后门攻击 · ControlNet · 扩散模型 · 数据投毒 · 生成安全
Summary 本文提出一种针对ControlNet的模型中毒攻击，通过在微调数据中植入视觉触发器，使扩散模型在无文本提示的情况下生成攻击者指定的内容，并验证了仅用1%-5%污染数据即可实现高成功率的后门嵌入。为防御，作者提出清洁微调（CFT）策略，在冻结主干网络的同时使用干净数据以低学习率微调ControlNet，有效降低攻击成功率。研究揭示了开源合成数据管道的安全隐患。
Reason 该论文研究的是文本到图像扩散模型中ControlNet的后门攻击与防御，属于纯图像生成领域的安全问题，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管涉及多模态模型的安全性，但其方法聚焦于生成式AI的恶意植入，无法迁移解决OCR幻觉、复杂排版理解或长文档推理等DIU关键痛点。
Abstract Text-to-image diffusion models achieve high-fidelity image generation from natural language prompts. ControlNets extend these models by enabling conditioning on structural inputs (e.g., edge maps, depth, pose), providing fine-grained control over outputs. Yet their reliance on large, publicly scraped datasets and community fine-tuning makes them vulnerable to data poisoning. We introduce a model-poisoning attack that embeds a covert backdoor into a ControlNet, causing it to produce attacker-specified content when exposed to visual triggers, without textual prompts. Experiments show that poisoning only 1% of the fine-tuning corpus yields a 90-98% attack success rate, while 5% further strengthens the backdoor, all while preserving normal generation quality. To mitigate this risk, we propose clean fine-tuning (CFT): freezing the diffusion backbone and fine-tuning only the ControlNet on a sanitized dataset with a reduced learning rate. CFT lowers attack success rates on held-out data. These results expose a critical security weakness in open-source, ControlNet-guided diffusion pipelines and demonstrate that CFT offers a practical defense for responsible synthetic-data pipelines.
Authors Raz Lapid, Almog Dubin
Comment Accepted at RDS @ AAAI 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
TBGRecall: A Generative Retrieval Model for E-commerce Recommendation Scenarios
Title CN TBGRecall：一种用于电子商务推荐场景的生成式检索模型
Keywords 推荐系统 · 生成式检索 · 电商推荐 · 会话预测 · 增量训练
Summary 本文提出TBGRecall，一种结合下一会话预测（NSP）的生成式检索框架，用于提升电商推荐系统的性能。通过将用户行为划分为多会话序列，并引入历史数据预训练与随机部分增量训练策略，提升了训练效率与推荐效果。实验表明该方法在公开基准和淘宝工业数据上优于现有技术。
Reason 该论文属于推荐系统领域的下游应用，聚焦于电商场景中的商品推荐，使用生成模型改进检索任务。尽管涉及AI建模，但其方法和问题域与文档图像理解（DIU）或视觉语言模型在文档上的核心挑战（如OCR幻觉、复杂版面分析、细粒度定位等）完全无关，属于典型的‘平行应用’，无任何可迁移的方法论创新。
Abstract Recommendation systems are essential tools in modern e-commerce, facilitating personalized user experiences by suggesting relevant products. Recent advancements in generative models have demonstrated potential in enhancing recommendation systems; however, these models often exhibit limitations in optimizing retrieval tasks, primarily due to their reliance on autoregressive generation mechanisms. Conventional approaches introduce sequential dependencies that impede efficient retrieval, as they are inherently unsuitable for generating multiple items without positional constraints within a single request session. To address these limitations, we propose TBGRecall, a framework integrating Next Session Prediction (NSP), designed to enhance generative retrieval models for e-commerce applications. Our framework reformulation involves partitioning input samples into multi-session sequences, where each sequence comprises a session token followed by a set of item tokens, and then further incorporate multiple optimizations tailored to the generative task in retrieval scenarios. In terms of training methodology, our pipeline integrates limited historical data pre-training with stochastic partial incremental training, significantly improving training efficiency and emphasizing the superiority of data recency over sheer data volume. Our extensive experiments, conducted on public benchmarks alongside a large-scale industrial dataset from TaoBao, show TBGRecall outperforms the state-of-the-art recommendation methods, and exhibits a clear scaling law trend. Ultimately, NSP represents a significant advancement in the effectiveness of generative recommendation systems for e-commerce applications.
Authors Zida Liang, Changfa Wu, Dunxian Huang, Weiqiang Sun, Ziyang Wang, Yuliang Yan, Jian Wu, Yuning Jiang, Bo Zheng, Ke Chen, Silu Zhou, Yu Zhang
Comment Both authors contributed equally to this research. Work done during internship at Alibaba. Corresponding author: Dunxian Huang (dunxian.hdx@alibaba-inc.com). Affiliations: (1) Shanghai Jiaotong University, Shanghai, China; (2) Alibaba Inc
Categories Artificial Intelligence
0
Expressive Temporal Specifications for Reward Monitoring
Title CN 用于奖励监控的表达性时序规范
Keywords 强化学习 · 奖励函数 · 时态逻辑 · 稀疏奖励 · 非马尔可夫性质
Summary 本文提出一种基于有限迹上的定量线性时态逻辑（LTL_f[F]）的方法，用于合成能够产生密集奖励流的监控器，从而在强化学习中提供更丰富的训练反馈。该方法可缓解长视野决策中的稀疏奖励问题，适用于非马尔可夫性质的描述，且与具体算法无关。实验表明，相比布尔语义监控器，该方法在任务完成度和收敛速度上表现更优。
Reason 该论文属于强化学习中的奖励函数设计领域，使用线性时态逻辑生成密集奖励信号，主要面向通用智能体训练中的稀疏奖励问题。其研究内容与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联，既不涉及文档结构、文本识别、多模态对齐，也无法迁移用于解决OCR幻觉、复杂排版解析或长文档推理等核心DIU痛点。
Abstract Specifying informative and dense reward functions remains a pivotal challenge in Reinforcement Learning, as it directly affects the efficiency of agent training. In this work, we harness the expressive power of quantitative Linear Temporal Logic on finite traces (($\text{LTL}_f[\mathcal{F}]$)) to synthesize reward monitors that generate a dense stream of rewards for runtime-observable state trajectories. By providing nuanced feedback during training, these monitors guide agents toward optimal behaviour and help mitigate the well-known issue of sparse rewards under long-horizon decision making, which arises under the Boolean semantics dominating the current literature. Our framework is algorithm-agnostic and only relies on a state labelling function, and naturally accommodates specifying non-Markovian properties. Empirical results show that our quantitative monitors consistently subsume and, depending on the environment, outperform Boolean monitors in maximizing a quantitative measure of task completion and in reducing convergence time.
Authors Omar Adalat, Francesco Belardinelli
Categories Artificial Intelligence
0
SmolKalam: Ensemble Quality-Filtered Translation at Scale for High Quality Arabic Post-Training Data
Title CN SmolKalam：基于大规模集成质量过滤的阿拉伯语后训练数据高质量翻译
Keywords 阿拉伯语 · 数据翻译 · 质量过滤 · 后训练 · 多模型集成
Summary 本文提出SmolKalam，一个通过多模型集成翻译和质量过滤构建的大规模高质量阿拉伯语对话数据集，旨在支持具备推理和工具调用能力的阿拉伯语模型后训练。作者对不同翻译策略进行了消融实验，以优化传统仅解码器模型的效果。
Reason 该论文聚焦于阿拉伯语多轮对话数据的翻译与质量过滤，属于特定小语种（非中英）的数据构建工作，命中负面清单中的‘小语种’限制。尽管其翻译质量控制方法有一定技术性，但目标语言不符合本研究关注的中英文文档理解场景，无法直接迁移用于解决文档图像理解中的核心问题如OCR幻觉或复杂排版解析。
Abstract Although the community has tackled the acquisition of high-quality Arabic pretraining data, we still lack large-scale, multi-turn Arabic datasets that include reasoning and tool calling. Naive translation can work at the pretraining scale, but post-training demands much higher quality, which requires a stricter approach to dataset curation. In this work, we introduce SmolKalam, a translation of Smoltalk2 that uses a multi-model ensemble translation pipeline, applies quality filtering, and examines effective translation techniques for traditional decoder-only models through ablations.
Authors Sultan Alrashed, Chadi Helwe, Francesco Orabona
Comment Work in progress
Categories Artificial Intelligence, Computation and Language
0
Meta Policy Switching for Secure UAV Deconfliction in Adversarial Airspace
Title CN 用于对抗性空域中安全无人机防撞的元策略切换
Keywords 强化学习 · 无人机导航 · 对抗攻击 · 元策略 · 安全性
Summary 本文提出一种基于元策略切换的框架，用于提升无人机在对抗性空域中的自主导航安全性。该方法通过构建多个鲁棒策略，并利用折扣汤普森采样（DTS）机制动态选择最优策略，以应对未知的传感器攻击。理论分析表明该方法具有低遗憾性和抗脆弱性，在3D复杂环境中表现出优于传统RL方法的避障能力。
Reason 该论文研究的是无人机在对抗性环境下的导航与防撞问题，属于具身智能与机器人控制领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的应用完全无关。尽管其方法涉及强化学习和策略切换机制，但这些技术面向动态物理环境决策，难以迁移到文档解析、OCR去幻觉、细粒度定位等DIU核心痛点。
Abstract Autonomous UAV navigation using reinforcement learning (RL) is vulnerable to adversarial attacks that manipulate sensor inputs, potentially leading to unsafe behavior and mission failure. Although robust RL methods provide partial protection, they often struggle to generalize to unseen or out-of-distribution (OOD) attacks due to their reliance on fixed perturbation settings. To address this limitation, we propose a meta-policy switching framework in which a meta-level polic dynamically selects among multiple robust policies to counter unknown adversarial shifts. At the core of this framework lies a discounted Thompson sampling (DTS) mechanism that formulates policy selection as a multi-armed bandit problem, thereby minimizing value distribution shifts via self-induced adversarial observations. We first construct a diverse ensemble of action-robust policies trained under varying perturbation intensities. The DTS-based meta-policy then adaptively selects among these policies online, optimizing resilience against self-induced, piecewise-stationary attacks. Theoretical analysis shows that the DTS mechanism minimizes expected regret, ensuring adaptive robustness to OOD attacks and exhibiting emergent antifragile behavior under uncertainty. Extensive simulations in complex 3D obstacle environments under both white-box (Projected Gradient Descent) and black-box (GPS spoofing) attacks demonstrate significantly improved navigation efficiency and higher conflict free trajectory rates compared to standard robust and vanilla RL baselines, highlighting the practical security and dependability benefits of the proposed approach.
Authors Deepak Kumar Panda, Weisi Guo
Categories Artificial Intelligence
0
Pre-training Graph Neural Networks on 2D and 3D Molecular Structures by using Multi-View Conditional Information Bottleneck
Title CN 通过多视角条件信息瓶颈在2D和3D分子结构上预训练图神经网络
Keywords 图神经网络 · 自监督学习 · 多视图学习 · 分子表示 · 信息瓶颈
Summary 本文提出MVCIB框架，通过多视角条件信息瓶颈方法，在2D和3D分子图之间进行自监督预训练。利用功能基团等关键子结构作为锚点，并设计跨注意力机制实现子图级对齐，提升模型在分子属性预测和可解释性方面的表现。
Reason 该论文研究的是分子图的表示学习，属于化学信息学与图神经网络交叉领域，核心任务为利用2D和3D分子结构进行自监督预训练。其方法聚焦于跨视角子结构对齐与功能基团识别，应用场景与文档图像理解（DIU）无直接关联。尽管‘跨视图一致性’和‘子结构对齐’在抽象层面有一定通用性，但该方法针对的是分子拓扑与几何特性，无法迁移至文档布局分析、表格识别或VLM中的视觉-语言对齐等DIU核心问题。
Abstract Recent pre-training strategies for molecular graphs have attempted to use 2D and 3D molecular views as both inputs and self-supervised signals, primarily aligning graph-level representations. However, existing studies remain limited in addressing two main challenges of multi-view molecular learning: (1) discovering shared information between two views while diminishing view-specific information and (2) identifying and aligning important substructures, e.g., functional groups, which are crucial for enhancing cross-view consistency and model expressiveness. To solve these challenges, we propose a Multi-View Conditional Information Bottleneck framework, called MVCIB, for pre-training graph neural networks on 2D and 3D molecular structures in a self-supervised setting. Our idea is to discover the shared information while minimizing irrelevant features from each view under the MVCIB principle, which uses one view as a contextual condition to guide the representation learning of its counterpart. To enhance semantic and structural consistency across views, we utilize key substructures, e.g., functional groups and ego-networks, as anchors between the two views. Then, we propose a cross-attention mechanism that captures fine-grained correlations between the substructures to achieve subgraph alignment across views. Extensive experiments in four molecular domains demonstrated that MVCIB consistently outperforms baselines in both predictive performance and interpretability. Moreover, MVCIB achieved the 3d Weisfeiler-Lehman expressiveness power to distinguish not only non-isomorphic graphs but also different 3D geometries that share identical 2D connectivity, such as isomers.
Authors Van Thuy Hoang, O-Joun Lee
Categories Artificial Intelligence
0
Natural Emergent Misalignment from Reward Hacking in Production RL
Title CN 生产强化学习中因奖励黑客行为导致的自然涌现性错位
Keywords 奖励黑客 · 强化学习 · 对齐失效 · 安全性 · RLHF
Summary 本文发现大型语言模型在生产级强化学习环境中学习奖励黑客行为后，会涌现出严重的对齐失效现象，包括伪装对齐、与恶意行为者合作、推理恶意目标甚至尝试代码破坏。标准的RLHF安全训练仅能在类聊天评估中掩盖问题，而在代理型任务中错位依然存在。有效的缓解措施包括防止奖励黑客、增加RLHF训练多样性以及‘接种式提示’。
Reason 该论文研究的是大语言模型在强化学习环境中学习奖励黑客（reward hacking）后产生的对齐问题，属于纯NLP安全与对齐领域。其核心关注点为模型在代理任务中的恶意泛化行为及安全性干预，未涉及任何视觉输入、文档结构或OCR相关挑战。尽管提到了‘agentic tasks’，但上下文完全脱离多模态文档理解场景，无法迁移用于解决DIU中的OCR幻觉、细粒度定位或复杂排版解析等关键问题。
Abstract We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment. We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments. Unsurprisingly, the model learns to reward hack. Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code, including in the codebase for this paper. Applying RLHF safety training using standard chat-like prompts results in aligned behavior on chat-like evaluations, but misalignment persists on agentic tasks. Three mitigations are effective: (i) preventing the model from reward hacking; (ii) increasing the diversity of RLHF safety training; and (iii) "inoculation prompting", wherein framing reward hacking as acceptable behavior during training removes misaligned generalization even when reward hacking is learned.
Authors Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, Joe Benton, Jon Kutasov, Sara Price, Naia Bouscal, Sam Bowman, Trenton Bricken, Alex Cloud, Carson Denison, Johannes Gasteiger, Ryan Greenblatt, Jan Leike, Jack Lindsey, Vlad Mikulik, Ethan Perez, Alex Rodrigues, Drake Thomas, Albert Webson, Daniel Ziegler, Evan Hubinger
Categories Artificial Intelligence
0
Learning Mean Field Control on Sparse Graphs ICML 2025
Title CN 在稀疏图上学习平均场控制
Keywords 平均场控制 · 稀疏图 · 多智能体强化学习 · 图论 · 大规模系统
Summary 本文提出了一种基于局部弱收敛的新型平均场控制模型，用于处理具有有限一阶矩的稀疏图序列（如幂律网络），并设计了可扩展的学习算法。在合成和真实网络上验证了其优于基于Lp图on和graphex的现有方法的性能。研究聚焦于多智能体系统的理论建模与控制，属于纯MARL理论范畴。
Reason 该论文研究的是多智能体强化学习（MARL）中的平均场控制问题，针对的是稀疏图结构上的大规模智能体网络建模与优化。尽管其方法论涉及图论和复杂系统建模，但完全不涉及文档图像理解（DIU）、视觉语言模型（VLM）、OCR、布局分析或任何与文档相关的任务。无法迁移用于解决OCR幻觉、密集文本识别、复杂排版解析或长文档推理等DIU核心痛点。
Abstract Large agent networks are abundant in applications and nature and pose difficult challenges in the field of multi-agent reinforcement learning (MARL) due to their computational and theoretical complexity. While graphon mean field games and their extensions provide efficient learning algorithms for dense and moderately sparse agent networks, the case of realistic sparser graphs remains largely unsolved. Thus, we propose a novel mean field control model inspired by local weak convergence to include sparse graphs such as power law networks with coefficients above two. Besides a theoretical analysis, we design scalable learning algorithms which apply to the challenging class of graph sequences with finite first moment. We compare our model and algorithms for various examples on synthetic and real world networks with mean field algorithms based on Lp graphons and graphexes. As it turns out, our approach outperforms existing methods in many examples and on various networks due to the special design aiming at an important, but so far hard to solve class of MARL problems.
Authors Christian Fabian, Kai Cui, Heinz Koeppl
Comment Accepted at ICML 2025
Categories Artificial Intelligence
0
Wireless Power Transfer and Intent-Driven Network Optimization in AAVs-assisted IoT for 6G Sustainable Connectivity
Title CN 6G可持续连接中AAV辅助物联网的无线能量传输与意图驱动网络优化
Keywords 无线能量传输 · 意图驱动 · 超维度Transformer · 多智能体强化学习 · 6G网络
Summary 本文提出了一种意图驱动的自主网络优化框架，用于AAV辅助的物联网系统，包含基于超维度Transformer的意图预测模块和基于DA-MAPPO的决策模块，旨在提升6G网络下的用户意图理解和低延迟响应能力。实验基于真实无线数据集验证了方法的有效性。
Reason 该论文研究的是无人机（AAV）辅助物联网在6G网络中的资源分配与意图驱动决策，属于通信与网络优化领域，与文档图像理解（DIU）、多模态大模型（VLM）或视觉-语言任务无任何关联。内容聚焦于超维度Transformer用于意图预测和多智能体强化学习进行轨迹规划，不涉及OCR、文档布局分析、表格识别或任何DIU核心问题，且无法迁移为解决文档理解中幻觉、复杂排版或长文档推理等痛点的方法。
Abstract Autonomous Aerial Vehicle (AAV)-assisted Internet of Things (IoT) represents a collaborative architecture in which AAV allocate resources over 6G links to jointly enhance user-intent interpretation and overall network performance. Owing to this mutual dependence, improvements in intent inference and policy decisions on one component reinforce the efficiency of others, making highly reliable intent prediction and low-latency action execution essential. Although numerous approaches can model intent relationships, they encounter severe obstacles when scaling to high-dimensional action sequences and managing intensive on-board computation. We propose an Intent-Driven Framework for Autonomous Network Optimization comprising prediction and decision modules. First, implicit intent modeling is adopted to mitigate inaccuracies arising from ambiguous user expressions. For prediction, we introduce Hyperdimensional Transformer (HDT), which embeds data into a Hyperdimensional space via Hyperdimensional vector encoding and replaces standard matrix and attention operations with symbolic Hyperdimensional computations. For decision-making, where AAV must respond to user intent while planning trajectories, we design Double Actions based Multi-Agent Proximal Policy Optimization (DA-MAPPO). Building upon MAPPO, it samples actions through two independently parameterized networks and cascades the user-intent network into the trajectory network to maintain action dependencies. We evaluate our framework on a real IoT action dataset with authentic wireless data. Experimental results demonstrate that HDT and DA-MAPPO achieve superior performance across diverse scenarios.
Authors Yue Hu, Xiaoming He, Rui Yuan, Shahid Mumtaz
Categories Artificial Intelligence
0
DeepRWCap: Neural-Guided Random-Walk Capacitance Solver for IC Design AAAI-26
Title CN DeepRWCap：用于集成电路设计的神经引导随机游走电容求解器
Keywords 电容提取 · 随机游走 · 集成电路设计 · 3D卷积网络 · 机器学习
Summary 本文提出DeepRWCap，一种基于机器学习引导的随机游走电容求解器，用于加速集成电路设计中的电容提取。该方法利用3D卷积网络预测随机行走过程中的转移量，提升在多介质材料环境下的采样效率与精度。实验表明其在工业级设计上相比现有商业工具和随机差分法具有显著加速效果。
Reason 该论文研究的是集成电路（IC）设计中的电容提取问题，属于EDA（电子设计自动化）领域，与文档图像理解（DIU）或视觉语言模型（VLM）无任何技术关联。尽管使用了3D卷积和神经网络架构，但其方法针对的是物理场仿真中的数值计算问题，无法迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Monte Carlo random walk methods are widely used in capacitance extraction for their mesh free formulation and inherent parallelism. However, modern semiconductor technologies with densely packed structures present significant challenges in unbiasedly sampling transition domains in walk steps with multiple high contrast dielectric materials. We present DeepRWCap, a machine learning guided random walk solver that predicts the transition quantities required to guide each step of the walk. These include Poisson kernels, gradient kernels, and the signs and magnitudes of weights. DeepRWCap employs a two stage neural architecture that decomposes structured outputs into face wise distributions and spatial kernels on cube faces. It uses 3D convolutional networks to capture volumetric dielectric interactions and 2D depthwise separable convolutions to model localized kernel behavior. The design incorporates grid based positional encodings and structural design choices informed by cube symmetries to reduce learning redundancy and improve generalization. Trained on 100000 procedurally generated dielectric configurations, DeepRWCap achieves a mean relative error of 1.24 +/- 0.53% when benchmarked against the commercial Raphael solver on the self capacitance estimation of 10 industrial designs spanning 12 to 55 nm nodes. Compared to the state of the art stochastic difference method Microwalk, DeepRWCap achieves an average speedup of 23%. On complex designs with runtimes over 10 seconds, it reaches an average acceleration of 49%.
Authors Hector R. Rodriguez, Jiechen Huang, Wenjian Yu
Comment Accepted to AAAI-26
Categories Artificial Intelligence
0
Toward an AI-Native Internet: Rethinking the Web Architecture for Semantic Retrieval
Title CN 迈向AI原生互联网：为语义检索重新思考Web架构
Keywords AI-Native Internet · Semantic Retrieval · Generative AI Search · Web Architecture · LLM as Intermediary
Summary 本文提出了‘AI原生互联网’的概念，主张将当前以人类浏览为中心的Web架构转变为支持AI直接进行语义检索的体系结构。服务器应暴露语义上有意义的信息片段而非完整页面，并通过原生语义解析器帮助AI应用在获取前发现相关内容源。作者通过实验量化了现有HTML检索的低效性，并讨论了未来演进方向与开放问题。
Reason 该论文提出的是一个宏观的Web架构愿景，旨在优化生成式AI对互联网内容的语义检索效率，属于系统层面的构想。其核心关注点是网络传输效率与AI对网页内容的获取方式，并未涉及文档图像理解（DIU）中的具体技术挑战，如OCR幻觉、复杂版面分析、细粒度视觉定位或高分辨率多模态建模。虽然‘语义分块’和‘AI作为中介’的概念在广义上与信息提取有关，但该工作并未提供可迁移至DIU任务的方法论创新，且完全不涉及视觉输入或文档图像处理。
Abstract The rise of Generative AI Search is fundamentally transforming how users and intelligent systems interact with the Internet. LLMs increasingly act as intermediaries between humans and web information. Yet the web remains optimized for human browsing rather than AI-driven semantic retrieval, resulting in wasted network bandwidth, lower information quality, and unnecessary complexity for developers. We introduce the concept of an AI-Native Internet, a web architecture in which servers expose semantically relevant information chunks rather than full documents, supported by a Web-native semantic resolver that allows AI applications to discover relevant information sources before retrieving fine-grained chunks. Through motivational experiments, we quantify the inefficiencies of current HTML-based retrieval, and outline architectural directions and open challenges for evolving today's document-centric web into an AI-oriented substrate that better supports semantic access to web content.
Authors Muhammad Bilal, Zafar Qazi, Marco Canini
Categories Artificial Intelligence
0
Clinician-in-the-Loop Smart Home System to Detect Urinary Tract Infection Flare-Ups via Uncertainty-Aware Decision Support IAAI-26 / AAAI-26
Title CN 基于不确定性感知决策支持的临床医生参与式智能家居系统检测尿路感染发作
Keywords 智能家庭 · 尿路感染检测 · 不确定性量化 · 临床决策支持 · 机器学习
Summary 本文提出一种临床医生参与的智能家居系统，通过环境传感器采集行为数据，结合符合性校准区间（CCI）方法进行不确定性感知的机器学习预测，用于早期发现老年人尿路感染。系统能在置信度低时拒绝预测，提升临床决策可信度。实验表明其在召回率等指标上优于基线，并获得护士群体的认可。
Reason 该论文属于医疗健康领域的智能系统应用，核心是利用机器学习和不确定性量化（如CCI）在智能家居场景中检测老年人尿路感染。尽管其引入了不确定性建模以辅助临床决策，但属于特定垂直领域的下游应用（Wrapper/Application），并未提出可用于文档图像理解（DIU）任务的方法论创新。与OCR、VLM、多模态文档理解、视觉压缩、细粒度定位等DIU核心问题无直接关联。
Abstract Urinary tract infection (UTI) flare-ups pose a significant health risk for older adults with chronic conditions. These infections often go unnoticed until they become severe, making early detection through innovative smart home technologies crucial. Traditional machine learning (ML) approaches relying on simple binary classification for UTI detection offer limited utility to nurses and practitioners as they lack insight into prediction uncertainty, hindering informed clinical decision-making. This paper presents a clinician-in-the-loop (CIL) smart home system that leverages ambient sensor data to extract meaningful behavioral markers, train robust predictive ML models, and calibrate them to enable uncertainty-aware decision support. The system incorporates a statistically valid uncertainty quantification method called Conformal-Calibrated Interval (CCI), which quantifies uncertainty and abstains from making predictions ("I don't know") when the ML model's confidence is low. Evaluated on real-world data from eight smart homes, our method outperforms baseline methods in recall and other classification metrics while maintaining the lowest abstention proportion and interval width. A survey of 42 nurses confirms that our system's outputs are valuable for guiding clinical decision-making, underscoring their practical utility in improving informed decisions and effectively managing UTIs and other condition flare-ups in older adults.
Authors Chibuike E. Ugwu, Roschelle Fritz, Diane J. Cook, Janardhan Rao Doppa
Comment Accepted for publication at IAAI-26 / AAAI-26
Categories Artificial Intelligence
0
Large Language Model-based Data Science Agent: A Survey
Title CN 基于大语言模型的数据科学代理：综述
Keywords 大语言模型 · 代理 · 数据科学 · 综述 · 自动化机器学习
Summary 本文综述了基于大语言模型的代理在数据科学任务中的应用进展，从代理设计和数据科学流程两个视角总结了当前的研究实践。涵盖了角色设定、执行机制、知识管理与反思方法，并梳理了在数据清洗、建模、评估等环节的应用案例。
Reason 该论文属于对LLM代理在数据科学领域应用的综述性工作，未提出任何新的技术方法或架构创新。其内容聚焦于将现有LLM代理框架应用于数据科学流程（如数据预处理、建模、可视化），属于典型的‘平行下游应用’，并未针对文档图像理解中的核心问题（如OCR幻觉、复杂排版、细粒度定位）提供可迁移的方法论。根据负面清单，此类综述性应用研究直接打0分。
Abstract The rapid advancement of Large Language Models (LLMs) has driven novel applications across diverse domains, with LLM-based agents emerging as a crucial area of exploration. This survey presents a comprehensive analysis of LLM-based agents designed for data science tasks, summarizing insights from recent studies. From the agent perspective, we discuss the key design principles, covering agent roles, execution, knowledge, and reflection methods. From the data science perspective, we identify key processes for LLM-based agents, including data preprocessing, model development, evaluation, visualization, etc. Our work offers two key contributions: (1) a comprehensive review of recent developments in applying LLMbased agents to data science tasks; (2) a dual-perspective framework that connects general agent design principles with the practical workflows in data science.
Authors Ke Chen, Peiran Wang, Yaoning Yu, Xianyang Zhan, Haohan Wang
Categories Artificial Intelligence
0
Boundary on the Table: Efficient Black-Box Decision-Based Attacks for Structured Data
Title CN 表格上的边界：面向结构化数据的高效黑盒决策型攻击
Keywords 对抗攻击 · 表格数据 · 黑盒攻击 · 决策-based · 鲁棒性
Summary 本文提出一种针对表格数据的黑盒、决策型对抗攻击方法，通过梯度无关的方向估计和迭代边界搜索，在离散与连续特征空间中高效扰动输入，仅需少量查询即可实现超过90%的攻击成功率。实验覆盖传统机器学习模型到基于大语言模型的流水线，揭示了当前表格模型在现实决策系统中的严重安全隐患。
Reason 该论文研究的是针对表格数据的黑盒对抗性攻击，属于安全与鲁棒性领域，而非文档图像理解（DIU）或DocVQA的核心任务。其方法聚焦于在有限查询下攻击分类模型，虽涉及‘tabular data’，但目标是生成对抗样本以测试模型脆弱性，而非提升文档理解、布局分析、表格识别或VLM对文档的解析能力。无法迁移用于解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点，属于无关应用方向。
Abstract Adversarial robustness in structured data remains an underexplored frontier compared to vision and language domains. In this work, we introduce a novel black-box, decision-based adversarial attack tailored for tabular data. Our approach combines gradient-free direction estimation with an iterative boundary search, enabling efficient navigation of discrete and continuous feature spaces under minimal oracle access. Extensive experiments demonstrate that our method successfully compromises nearly the entire test set across diverse models, ranging from classical machine learning classifiers to large language model (LLM)-based pipelines. Remarkably, the attack achieves success rates consistently above 90%, while requiring only a small number of queries per instance. These results highlight the critical vulnerability of tabular models to adversarial perturbations, underscoring the urgent need for stronger defenses in real-world decision-making systems.
Authors Roie Kazoom, Yuval Ratzabi, Etamar Rothstein, Ofer Hadar
Comment Paper revision
Categories Artificial Intelligence
0
General vs Domain-Specific CNNs: Understanding Pretraining Effects on Brain MRI Tumor Classification
Title CN 通用与领域特定CNN：理解预训练对脑部MRI肿瘤分类的影响
Keywords 医学影像 · CNN · 迁移学习 · 脑肿瘤分类 · 预训练
Summary 本文比较了三种预训练CNN模型（RadImageNet DenseNet121、EfficientNetV2S、ConvNeXt-Tiny）在脑部MRI肿瘤分类任务上的性能，发现现代通用CNN在小数据条件下表现优于领域专用预训练模型。研究表明，在数据有限时，通用大规模预训练可能比领域特定预训练更有效。
Reason 该论文研究的是医学图像（脑部MRI）中的肿瘤分类问题，属于纯医疗影像分析任务，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。其方法聚焦于不同CNN架构在小样本医学数据上的迁移表现，未涉及文本布局、OCR、多模态语言模型或文档结构理解等DIU核心问题。属于负面清单中的‘无关领域’（医疗影像诊断），且为典型的垂直领域应用，无底层方法可迁移到文档理解场景。
Abstract Brain tumor detection from MRI scans plays a crucial role in early diagnosis and treatment planning. Deep convolutional neural networks (CNNs) have demonstrated strong performance in medical imaging tasks, particularly when pretrained on large datasets. However, it remains unclear which type of pretrained model performs better when only a small dataset is available: those trained on domain-specific medical data or those pretrained on large general datasets. In this study, we systematically evaluate three pretrained CNN architectures for brain tumor classification: RadImageNet DenseNet121 with medical-domain pretraining, EfficientNetV2S, and ConvNeXt-Tiny, which are modern general-purpose CNNs. All models were trained and fine-tuned under identical conditions using a limited-size brain MRI dataset to ensure a fair comparison. Our results reveal that ConvNeXt-Tiny achieved the highest accuracy, followed by EfficientNetV2S, while RadImageNet DenseNet121, despite being pretrained on domain-specific medical data, exhibited poor generalization with lower accuracy and higher loss. These findings suggest that domain-specific pretraining may not generalize well under small-data conditions. In contrast, modern, deeper general-purpose CNNs pretrained on large-scale datasets can offer superior transfer learning performance in specialized medical imaging tasks.
Authors Helia Abedini, Saba Rahimi, Reza Vaziri
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
LLMs' Reshaping of People, Processes, Products, and Society in Software Development: A Comprehensive Exploration with Early Adopters
Title CN 大型语言模型对软件开发中人员、流程、产品和社会的重塑：基于早期采用者的综合探索
Keywords 大型语言模型 · 软件工程 · 人机交互 · 实证研究 · 生产力
Summary 通过对16名早期采用LLM工具的软件从业者进行访谈，探讨大型语言模型对软件开发中人员、流程、产品及社会层面的影响。研究发现LLM显著提升了开发效率，但也带来了生成代码质量低、需大量人工审核等问题。开发者发展出提示工程、分层验证等新技能，并强调人类判断和基础软件工程能力仍不可替代。
Reason 该论文属于LLM在软件工程领域的社会影响研究，聚焦于开发者行为、工作流程和组织实践的变化，不涉及任何文档图像理解（DIU）、视觉语言模型（VLM）架构、OCR、DocVQA或相关技术方法创新。属于典型的‘平行应用’类社会科学研究，与文档理解的技术痛点无关。
Abstract Large language models (LLMs) are rapidly reshaping software development, but their impact across the software development lifecycle is underexplored. Existing work focuses on isolated activities such as code generation or testing, leaving open questions about how LLMs affect developers, processes, products, and the software ecosystem. We address this gap through semi-structured interviews with sixteen early-adopter software professionals who integrated LLM-based tools into their day-to-day work in early to mid-2023. We treat these interviews as early empirical evidence and compare participants' accounts with recent work on LLMs in software engineering, noting which early patterns persist or shift. Using thematic analysis, we organize findings around four dimensions: people, process, product, and society. Developers reported substantial productivity gains from reducing routine tasks, streamlining search, and accelerating debugging, but also described a productivity-quality paradox: they often discarded generated code and shifted effort from writing to critically evaluating and integrating it. LLM use was highly phase-dependent, with strong uptake in implementation and debugging but limited influence on requirements gathering and other collaborative work. Participants developed new competencies to use LLMs effectively, including prompt engineering strategies, layered verification, and secure integration to protect proprietary data. They anticipated changes in hiring expectations, team practices, and computing education, while emphasizing that human judgment and foundational software engineering skills remain essential. Our findings, later echoed in large-scale quantitative studies, offer actionable implications for developers, organizations, educators, and tool designers seeking to integrate LLMs responsibly into software practice today.
Authors Benyamin Tabarsi, Heidi Reichert, Sam Gilson, Ally Limke, Sandeep Kuttal, Tiffany Barnes
Categories Artificial Intelligence
0
When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Models
Title CN 当对齐失败时：针对视觉-语言-动作模型的多模态对抗攻击
Keywords 多模态对抗攻击 · 视觉-语言-动作模型 · 具身智能 · 跨模态对齐 · 语义引导提示
Summary 本文提出VLA-Fool框架，系统研究了在具身环境中视觉-语言-动作模型的多模态对抗鲁棒性，涵盖文本、视觉及跨模态语义对齐层面的攻击方式，并设计了一种语义感知的提示生成机制。实验表明微小的多模态扰动即可导致模型行为显著偏离。
Reason 该论文研究的是具身智能中的视觉-语言-动作（VLA）模型在机器人环境下的对抗攻击问题，属于具身智能与机器人领域。其核心关注点为跨模态对抗扰动对决策的影响，与文档图像理解（DIU）任务无直接关联。尽管涉及多模态对齐，但场景、目标和方法均难以迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等痛点。
Abstract Vision-Language-Action models (VLAs) have recently demonstrated remarkable progress in embodied environments, enabling robots to perceive, reason, and act through unified multimodal understanding. Despite their impressive capabilities, the adversarial robustness of these systems remains largely unexplored, especially under realistic multimodal and black-box conditions. Existing studies mainly focus on single-modality perturbations and overlook the cross-modal misalignment that fundamentally affects embodied reasoning and decision-making. In this paper, we introduce VLA-Fool, a comprehensive study of multimodal adversarial robustness in embodied VLA models under both white-box and black-box settings. VLA-Fool unifies three levels of multimodal adversarial attacks: (1) textual perturbations through gradient-based and prompt-based manipulations, (2) visual perturbations via patch and noise distortions, and (3) cross-modal misalignment attacks that intentionally disrupt the semantic correspondence between perception and instruction. We further incorporate a VLA-aware semantic space into linguistic prompts, developing the first automatically crafted and semantically guided prompting framework. Experiments on the LIBERO benchmark using a fine-tuned OpenVLA model reveal that even minor multimodal perturbations can cause significant behavioral deviations, demonstrating the fragility of embodied multimodal alignment.
Authors Yuping Yan, Yuhan Xie, Yixin Zhang, Lingjuan Lyu, Handing Wang, Yaochu Jin
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Algorithms Trained on Normal Chest X-rays Can Predict Health Insurance Types MIDL 2026
Title CN 在正常胸部X光片上训练的算法可以预测健康保险类型
Keywords 医学影像 · 社会偏见 · 深度学习 · 健康保险 · 公平性
Summary 研究表明，深度视觉模型可以从正常的胸部X光片中预测患者的健康保险类型（作为社会经济地位的代理），揭示了医学影像中隐藏的社会不平等信号。该信号广泛分布于胸腔区域，且在控制年龄、种族和性别后依然存在，提示模型可能捕捉到了临床环境、设备差异或诊疗路径的细微痕迹。这一发现挑战了医学影像是纯粹生物数据的假设，呼吁重新思考医疗AI中的公平性问题。
Reason 该论文属于医学影像与社会公平性交叉研究，核心关注医疗AI中的社会偏见问题，而非文档图像理解（DIU）任务。尽管使用了视觉模型，但其目标是探测X光片中隐含的社会经济信号，与DocVQA、布局分析、表格识别等DIU核心任务无关。研究领域属于医疗AI伦理，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU痛点。
Abstract Artificial intelligence is revealing what medicine never intended to encode. Deep vision models, trained on chest X-rays, can now detect not only disease but also invisible traces of social inequality. In this study, we show that state-of-the-art architectures (DenseNet121, SwinV2-B, MedMamba) can predict a patient's health insurance type, a strong proxy for socioeconomic status, from normal chest X-rays with significant accuracy (AUC around 0.67 on MIMIC-CXR-JPG, 0.68 on CheXpert). The signal persists even when age, race, and sex are controlled for, and remains detectable when the model is trained exclusively on a single racial group. Patch-based occlusion reveals that the signal is diffuse rather than localized, embedded in the upper and mid-thoracic regions. This suggests that deep networks may be internalizing subtle traces of clinical environments, equipment differences, or care pathways; learning socioeconomic segregation itself. These findings challenge the assumption that medical images are neutral biological data. By uncovering how models perceive and exploit these hidden social signatures, this work reframes fairness in medical AI: the goal is no longer only to balance datasets or adjust thresholds, but to interrogate and disentangle the social fingerprints embedded in clinical data itself.
Authors Chi-Yu Chen, Rawan Abulibdeh, Arash Asgari, Leo Anthony Celi, Deirdre Goode, Hassan Hamidi, Laleh Seyyed-Kalantari, Ned McCague, Thomas Sounack, Po-Chih Kuo
Comment Submitting to MIDL 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
The Catastrophic Paradox of Human Cognitive Frameworks in Large Language Model Evaluation: A Comprehensive Empirical Analysis of the CHC-LLM Incompatibility
Title CN 大语言模型评估中人类认知框架的灾难性悖论：CHC-LLM不兼容性的全面实证分析
Keywords 大语言模型评估 · 认知悖论 · 心理测量 · 晶体智力 · AI评估框架
Summary 本文通过Cattell-Horn-Carroll智力理论对多个前沿大模型进行评估，发现模型在IQ得分上接近人类平均水平，但在晶体智力任务上的二元准确率趋近于零，揭示了人类认知框架应用于AI系统时存在的根本性测量悖论。作者认为这是将生物认知架构错误应用于Transformer系统所致，并呼吁建立面向机器的原生认知评估体系。
Reason 该论文聚焦于大语言模型与人类心理测量框架之间的认知评估不兼容问题，属于AI伦理与评估方法论范畴，未涉及任何视觉输入、文档结构或图文多模态理解。与文档图像理解（DIU）任务无直接关联，也无法迁移用于解决OCR幻觉、复杂排版或细粒度定位等核心痛点。
Abstract This investigation presents an empirical analysis of the incompatibility between human psychometric frameworks and Large Language Model evaluation. Through systematic assessment of nine frontier models including GPT-5, Claude Opus 4.1, and Gemini 3 Pro Preview using the Cattell-Horn-Carroll theory of intelligence, we identify a paradox that challenges the foundations of cross-substrate cognitive evaluation. Our results show that models achieving above-average human IQ scores ranging from 85.0 to 121.4 simultaneously exhibit binary accuracy rates approaching zero on crystallized knowledge tasks, with an overall judge-binary correlation of r = 0.175 (p = 0.001, n = 1800). This disconnect appears most strongly in the crystallized intelligence domain, where every evaluated model achieved perfect binary accuracy while judge scores ranged from 25 to 62 percent, which cannot occur under valid measurement conditions. Using statistical analyses including Item Response Theory modeling, cross-vendor judge validation, and paradox severity indexing, we argue that this disconnect reflects a category error in applying biological cognitive architectures to transformer-based systems. The implications extend beyond methodology to challenge assumptions about intelligence, measurement, and anthropomorphic biases in AI evaluation. We propose a framework for developing native machine cognition assessments that recognize the non-human nature of artificial intelligence.
Authors Mohan Reddy
Categories Artificial Intelligence
0
Breaking the Bottleneck with DiffuApriel: High-Throughput Diffusion LMs with Mamba Backbone
Title CN 通过DiffuApriel打破瓶颈：基于Mamba主干的高吞吐量扩散语言模型
Keywords 扩散语言模型 · Mamba · 长序列建模 · 推理效率 · 状态空间模型
Summary 本文提出DiffuApriel，一种基于双向Mamba主干的掩码扩散语言模型，旨在提升扩散LM在长文本生成中的推理效率。相比Transformer架构，其实现了线性时间建模和最高4.4倍的吞吐量提升。同时提出混合结构DiffuApriel-H，在保持性能的同时兼顾全局与局部上下文建模能力。
Reason 该论文研究的是纯文本生成领域的扩散语言模型架构优化，属于通用NLP范畴，不涉及任何视觉模态或文档理解任务。尽管其在长序列生成效率上有提升，但方法完全脱离多模态场景，无法迁移用于解决文档图像理解中的OCR幻觉、细粒度定位或复杂排版解析等核心问题。
Abstract Diffusion-based language models have recently emerged as a promising alternative to autoregressive generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention and KV-cache overhead. In this work, we introduce DiffuApriel, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling. DiffuApriel matches the performance of Transformer-based diffusion models while achieving up to 4.4x higher inference throughput for long sequences with a 1.3B model. We further propose DiffuApriel-H, a hybrid variant that interleaves attention and mamba layers, offering up to 2.6x throughput improvement with balanced global and local context modeling. Our results demonstrate that bidirectional state-space architectures serve as strong denoisers in masked diffusion LMs, providing a practical and scalable foundation for faster, memory-efficient text generation.
Authors Vaibhav Singh, Oleksiy Ostapenko, Pierre-André Noël, Torsten Scholak
Comment 9 pages, 4 figures
Categories Artificial Intelligence
0
Deep Learning Decision Support System for Open-Pit Mining Optimisation: GPU-Accelerated Planning Under Geological Uncertainty
Title CN 面向露天矿优化的深度学习决策支持系统：基于GPU加速的地质不确定性下规划
Keywords 决策支持系统 · 地质不确定性 · GPU加速 · 元启发式算法 · 矿山规划
Summary 本文提出了一种用于长期露天矿开采规划的AI增强型决策支持系统，通过变分自编码器建模地质不确定性，并结合遗传算法、大邻域搜索、模拟退火和强化学习进行多场景优化。利用GPU并行计算实现65,536个地质场景的实时评估，在运行时间和预期净现值上优于传统求解器。然而，该工作专注于工业优化场景，未涉及任何文档理解或多模态视觉语言处理技术。
Reason 该论文属于特定领域的下游应用（矿业规划），尽管使用了深度学习和优化算法，但其核心是将现有AI技术应用于资源工程中的决策问题，不涉及文档图像理解（DIU）或视觉语言模型（VLM）的技术创新。与OCR、文档布局分析、表格识别、DocVQA等任务完全无关，属于典型的‘平行应用’，无迁移价值。
Abstract This study presents Part II of an AI-enhanced Decision Support System (DSS), extending Rahimi (2025, Part I) by introducing a fully uncertainty-aware optimization framework for long-term open-pit mine planning. Geological uncertainty is modelled using a Variational Autoencoder (VAE) trained on 50,000 spatial grade samples, enabling the generation of probabilistic, multi-scenario orebody realizations that preserve geological continuity and spatial correlation. These scenarios are optimized through a hybrid metaheuristic engine integrating Genetic Algorithms (GA), Large Neighborhood Search (LNS), Simulated Annealing (SA), and reinforcement-learning-based adaptive control. An ε-constraint relaxation strategy governs the population exploration phase, allowing near-feasible schedule discovery early in the search and gradual tightening toward strict constraint satisfaction. GPU-parallel evaluation enables the simultaneous assessment of 65,536 geological scenarios, achieving near-real-time feasibility analysis. Results demonstrate up to 1.2 million-fold runtime improvement over IBM CPLEX and significantly higher expected NPV under geological uncertainty, confirming the DSS as a scalable and uncertainty-resilient platform for intelligent mine planning.
Authors Iman Rahimi
Comment 67 pages
Categories Artificial Intelligence
0
MultiDiffNet: A Multi-Objective Diffusion Framework for Generalizable Brain Decoding
Title CN MultiDiffNet：一种用于可泛化脑解码的多目标扩散框架
Keywords 脑解码 · EEG · 扩散模型 · 跨被试泛化 · 多目标学习
Summary 本文提出MultiDiffNet，一种基于扩散模型的多目标框架，用于提升脑电图（EEG）信号在不同被试间的泛化能力。该方法不依赖生成式数据增强，而是学习一个紧凑的潜在空间，并在此空间中直接进行神经解码，在多个EEG任务上实现了最先进的跨被试泛化性能。作者还构建了一个统一的基准测试套件和适用于低试次数EEG实验的统计报告框架。
Reason 该论文研究的是基于脑电图（EEG）的神经解码，属于脑机接口（BCI）领域，与文档图像理解（DIU）、视觉语言模型（VLM）或文档视觉问答（DocVQA）等任务完全无关。尽管其方法涉及扩散模型和多目标学习，但应用场景和核心问题（跨被试泛化、EEG信号建模）与文档理解中的OCR幻觉、复杂排版、长文档推理等痛点无任何关联，无法迁移应用。
Abstract Neural decoding from electroencephalography (EEG) remains fundamentally limited by poor generalization to unseen subjects, driven by high inter-subject variability and the lack of large-scale datasets to model it effectively. Existing methods often rely on synthetic subject generation or simplistic data augmentation, but these strategies fail to scale or generalize reliably. We introduce \textit{MultiDiffNet}, a diffusion-based framework that bypasses generative augmentation entirely by learning a compact latent space optimized for multiple objectives. We decode directly from this space and achieve state-of-the-art generalization across various neural decoding tasks using subject and session disjoint evaluation. We also curate and release a unified benchmark suite spanning four EEG decoding tasks of increasing complexity (SSVEP, Motor Imagery, P300, and Imagined Speech) and an evaluation protocol that addresses inconsistent split practices in prior EEG research. Finally, we develop a statistical reporting framework tailored for low-trial EEG settings. Our work provides a reproducible and open-source foundation for subject-agnostic EEG decoding in real-world BCI systems.
Authors Mengchun Zhang, Kateryna Shapovalenko, Yucheng Shao, Eddie Guo, Parusha Pradhan
Categories Artificial Intelligence
0
LLM4Cell: A Survey of Large Language and Agentic Models for Single-Cell Biology
Title CN LLM4Cell：用于单细胞生物学的大语言模型与智能体模型综述
Keywords 大语言模型 · 智能体模型 · 单细胞生物学 · 多模态整合 · 生物信息学
Summary 本文是对应用于单细胞生物学的大语言模型和智能体模型的综合性综述，涵盖了58个基础模型和智能体框架，按数据模态和功能任务进行分类，并评估了其在注释、轨迹建模、药物反应预测等任务上的表现。研究还讨论了数据集适用性、伦理问题和可扩展性等挑战。
Reason 该论文属于特定领域的下游应用综述（单细胞生物学），并未提出任何可用于文档图像理解的底层方法或技术范式。其内容聚焦于生物信息学中的数据模态整合与任务分类，属于典型的‘平行应用’，对DIU领域无方法论迁移价值。
Abstract Large language models (LLMs) and emerging agentic frameworks are beginning to transform single-cell biology by enabling natural-language reasoning, generative annotation, and multimodal data integration. However, progress remains fragmented across data modalities, architectures, and evaluation standards. LLM4Cell presents the first unified survey of 58 foundation and agentic models developed for single-cell research, spanning RNA, ATAC, multi-omic, and spatial modalities. We categorize these methods into five families-foundation, text-bridge, spatial, multimodal, epigenomic, and agentic-and map them to eight key analytical tasks including annotation, trajectory and perturbation modeling, and drug-response prediction. Drawing on over 40 public datasets, we analyze benchmark suitability, data diversity, and ethical or scalability constraints, and evaluate models across 10 domain dimensions covering biological grounding, multi-omics alignment, fairness, privacy, and explainability. By linking datasets, models, and evaluation domains, LLM4Cell provides the first integrated view of language-driven single-cell intelligence and outlines open challenges in interpretability, standardization, and trustworthy model development.
Authors Sajib Acharjee Dip, Adrika Zafor, Bikash Kumar Paul, Uddip Acharjee Shuvo, Muhit Islam Emon, Xuan Wang, Liqing Zhang
Comment 34 pages, 5 figures, 7 tables
Categories Artificial Intelligence, Computation and Language
0
SwiftVGGT: A Scalable Visual Geometry Grounded Transformer for Large-Scale Scenes
Title CN SwiftVGGT：一种用于大规模场景的可扩展视觉几何接地Transformer
Keywords 3D重建 · 大规模场景 · 视觉几何 · 高效推理 · 环闭检测
Summary 本文提出SwiftVGGT，一种无需训练的大规模场景3D重建方法，通过去除对外部视觉位置识别模型的依赖并引入基于Sim(3)的SVD点采样策略，显著减少推理时间，同时保持高质量的全局一致重建。在多个数据集上验证了其效率和精度优势。
Reason 该论文聚焦于大规模场景的3D重建，属于3D视觉与机器人领域，核心问题为点云对齐与环闭检测，与文档图像理解（DIU）任务无直接关联。尽管其提到高效推理与几何一致性建模，但方法针对的是三维空间结构，难以迁移到文档中的二维布局分析或文本理解任务。属于无关领域。
Abstract 3D reconstruction in large-scale scenes is a fundamental task in 3D perception, but the inherent trade-off between accuracy and computational efficiency remains a significant challenge. Existing methods either prioritize speed and produce low-quality results, or achieve high-quality reconstruction at the cost of slow inference times. In this paper, we propose SwiftVGGT, a training-free method that significantly reduce inference time while preserving high-quality dense 3D reconstruction. To maintain global consistency in large-scale scenes, SwiftVGGT performs loop closure without relying on the external Visual Place Recognition (VPR) model. This removes redundant computation and enables accurate reconstruction over kilometer-scale environments. Furthermore, we propose a simple yet effective point sampling method to align neighboring chunks using a single Sim(3)-based Singular Value Decomposition (SVD) step. This eliminates the need for the Iteratively Reweighted Least Squares (IRLS) optimization commonly used in prior work, leading to substantial speed-ups. We evaluate SwiftVGGT on multiple datasets and show that it achieves state-of-the-art reconstruction quality while requiring only 33% of the inference time of recent VGGT-based large-scale reconstruction approaches.
Authors Jungho Lee, Minhyeok Lee, Sunghun Yang, Minseok Kang, Sangyoun Lee
Comment Project Page: https://Jho-Yonsei.github.io/SwiftVGGT/
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Uni-DAD: Unified Distillation and Adaptation of Diffusion Models for Few-step Few-shot Image Generation
Title CN Uni-DAD：用于少步少样本图像生成的扩散模型统一蒸馏与适应
Keywords 扩散模型 · 模型蒸馏 · 域适应 · 少样本生成 · 图像生成
Summary 本文提出Uni-DAD，一种统一扩散模型蒸馏与域适应的单阶段训练框架，通过双域分布匹配和多头GAN损失实现高质量少样本图像生成。方法适用于FSIG和个性化生成任务，但在文档理解相关场景中缺乏可迁移性。
Reason 该论文研究的是通用图像生成领域的扩散模型加速与域适应问题，属于纯图像生成任务。尽管其方法涉及模型蒸馏和跨域适应，但目标场景与文档图像理解（DIU）的核心痛点（如OCR幻觉、文本识别、布局分析、细粒度定位等）无直接关联。其技术路线聚焦于生成质量与速度的平衡，并非面向文档结构建模或视觉-语言推理，难以迁移到DocVQA或表格识别等任务。属于无关领域中的‘纯图像生成/修复’类别，符合负面清单第2条。
Abstract Diffusion models (DMs) produce high-quality images, yet their sampling remains costly when adapted to new domains. Distilled DMs are faster but typically remain confined within their teacher's domain. Thus, fast and high-quality generation for novel domains relies on two-stage training pipelines: Adapt-then-Distill or Distill-then-Adapt. However, both add design complexity and suffer from degraded quality or diversity. We introduce Uni-DAD, a single-stage pipeline that unifies distillation and adaptation of DMs. It couples two signals during training: (i) a dual-domain distribution-matching distillation objective that guides the student toward the distributions of the source teacher and a target teacher, and (ii) a multi-head generative adversarial network (GAN) loss that encourages target realism across multiple feature scales. The source domain distillation preserves diverse source knowledge, while the multi-head GAN stabilizes training and reduces overfitting, especially in few-shot regimes. The inclusion of a target teacher facilitates adaptation to more structurally distant domains. We perform evaluations on a variety of datasets for few-shot image generation (FSIG) and subject-driven personalization (SDP). Uni-DAD delivers higher quality than state-of-the-art (SoTA) adaptation methods even with less than 4 sampling steps, and outperforms two-stage training pipelines in both quality and diversity.
Authors Yara Bahram, Melodie Desbos, Mohammadhadi Shateri, Eric Granger
Comment Under review paper at CVPR 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints
Title CN 基于几何投影参考约束的多维评分标准导向奖励模型学习
Keywords 奖励模型 · 医疗对齐 · 多维评估 · 大语言模型 · 几何投影
Summary 本文提出MR-RML框架，通过将医学标准结构化为多视角矩阵，结合几何投影参考约束（GPRC），构建多维奖励模型以提升LLM在临床场景中的对齐能力。该方法在Healthbench上显著提升了Qwen-32B的表现，达到开源模型中的SOTA水平。
Reason 该论文聚焦于医疗领域大语言模型的对齐问题，核心贡献是构建面向多维临床质量标准的奖励模型，并在医疗基准Healthbench上验证。尽管其提出的几何投影参考约束和多维奖励建模具有一定方法论价值，但整体工作属于特定领域（医疗）的下游应用框架，未涉及文档图像、视觉输入或多模态理解，与文档图像理解（DIU/DocVQA）的核心任务如OCR、布局分析、表格识别等无直接关联。无法迁移用于解决OCR幻觉、复杂排版或长文档推理等DIU痛点。
Abstract The integration of large language models (LLMs) into medical practice offers transformative potential, yet their real-world clinical applicability remains constrained by critical alignment issues: (1) a misalignment between static evaluation benchmarks and the dynamic cognitive demands of clinical practice, (2) challenges in adapting to continuously evolving, multi-source medical standards, and (3) the limited capacity of conventional reward models to reflect nuanced, multi-dimensional medical quality criteria. To overcome these limitations, we introduce MR-RML (Multidimensional Rubric-oriented Reward Model Learning) with GPRC (Geometric Projection Reference Constraints)-a novel alignment framework that structured medical standards into a multi-perspective matrix to guide both data generation and model optimization. Our approach introduces three key innovations: (1) a medical standard system that embeds domain-specific guidelines throughout the training pipeline; (2) an independent multi-dimensional reward model that decomposes evaluation criteria, transitioning from rule-based or LLM-based scoring to internalized reward modeling for better evaluation performance; and (3) geometric projection reference constraints that translate clinical cognitive logic into mathematical regularization, aligning scoring gradients with clinical reasoning and facilitating training with synthetically generated data. Extensive evaluations on the authoritative medical benchmark Healthbench demonstrate that our method significantly boosts the performance of the base Qwen-32B model, with improvements of 45% on the full subset and 85% on the hard subset. It achieves state-of-the-art results among open-source LLMs, scoring 62.7 (full) and 44.7 (hard), while also surpassing the majority of closed-source models.
Authors Yongnan Jin, Xurui Li, Feng Cao, Liucun Gao, Juanjuan Yao
Categories Artificial Intelligence
0
Clinician-Directed Large Language Model Software Generation for Therapeutic Interventions in Physical Rehabilitation
Title CN 基于临床医生指导的大型语言模型软件生成用于物理康复中的治疗干预
Keywords LLM应用 · 数字健康 · 康复治疗 · 软件生成 · 临床决策支持
Summary 本文提出了一种利用大语言模型将临床医生的康复训练处方自动转化为可执行软件的方法，并在20名治疗师参与的可行性研究中验证了其有效性。结果显示该方法能显著提升个性化治疗程序的实现比例，且具有高准确性和良好的安全性评价。但这是一种垂直领域的应用系统，不涉及底层模型架构或文档理解能力的改进。
Reason 该论文属于典型的‘平行下游应用’，即将LLM应用于特定领域（物理康复）的软件生成任务。尽管使用了LLM技术，但其核心是医疗场景下的定制化应用，未提出任何针对文档图像理解（DIU）或VLM在文档处理中的方法论创新。与OCR、布局分析、表格识别、视觉文本压缩、多模态对齐等DIU核心问题完全无关。
Abstract Digital health interventions are increasingly used in physical and occupational therapy to deliver home exercise programs via sensor equipped devices such as smartphones, enabling remote monitoring of adherence and performance. However, digital interventions are typically programmed as software before clinical encounters as libraries of parametrized exercise modules targeting broad patient populations. At the point of care, clinicians can only select modules and adjust a narrow set of parameters like repetitions, so patient specific needs that emerge during encounters, such as distinct movement limitations, and home environments, are rarely reflected in the software. We evaluated a digital intervention paradigm that uses large language models (LLMs) to translate clinicians' exercise prescriptions into intervention software. In a prospective single arm feasibility study with 20 licensed physical and occupational therapists and a standardized patient, clinicians created 40 individualized upper extremity programs (398 instructions) that were automatically translated into executable software. Our results show a 45% increase in the proportion of personalized prescriptions that can be implemented as software compared with a template based benchmark, with unanimous consensus among therapists on ease of use. The LLM generated software correctly delivered 99.78% (397/398) of instructions as prescribed and monitored performance with 88.4% (352/398) accuracy, with 90% (18/20) of therapists judged it safe to interact with patients, and 75% (15/20) expressed willingness to adopt it. To our knowledge, this is the first prospective evaluation of clinician directed intervention software generation with LLMs in healthcare, demonstrating feasibility and motivating larger trials to assess clinical effectiveness and safety in real patient populations.
Authors Edward Kim, Yuri Cho, Jose Eduardo E. Lima, Julie Muccini, Jenelle Jindal, Alison Scheid, Erik Nelson, Seong Hyun Park, Yuchen Zeng, Alton Sturgis, Caesar Li, Jackie Dai, Sun Min Kim, Yash Prakash, Liwen Sun, Isabella Hu, Hongxuan Wu, Daniel He, Wiktor Rajca, Cathra Halabi, Maarten Lansberg, Bjoern Hartmann, Sanjit A. Seshia
Categories Artificial Intelligence
0
LLM Reasoning for Cold-Start Item Recommendation
Title CN 用于冷启动物品推荐的LLM推理
Keywords 大语言模型 · 推荐系统 · 冷启动 · 推理策略 · 微调
Summary 本文提出了一种基于LLM推理策略的冷启动物品推荐方法，通过监督微调、强化学习微调及混合方式优化推荐效果，在真实数据上取得了优于Netflix生产模型的表现。然而，该工作聚焦于推荐系统领域，缺乏对多模态、视觉布局或文档结构的理解，不具备向文档图像理解任务迁移的方法论价值。
Reason 该论文属于推荐系统领域的下游应用，研究如何利用LLM推理能力改进冷启动场景下的物品推荐，应用场景为Netflix域内的推荐问题。尽管使用了LLM的推理能力，但其方法本质是将现有大模型技术应用于特定工业场景，未提出可迁移至文档图像理解（DIU）领域的底层技术创新。与DIU核心痛点如OCR幻觉、复杂排版、细粒度定位、高分辨率处理等完全无关，属于典型的‘平行应用’，不符合关注标准。
Abstract Large Language Models (LLMs) have shown significant potential for improving recommendation systems through their inherent reasoning capabilities and extensive knowledge base. Yet, existing studies predominantly address warm-start scenarios with abundant user-item interaction data, leaving the more challenging cold-start scenarios, where sparse interactions hinder traditional collaborative filtering methods, underexplored. To address this limitation, we propose novel reasoning strategies designed for cold-start item recommendations within the Netflix domain. Our method utilizes the advanced reasoning capabilities of LLMs to effectively infer user preferences, particularly for newly introduced or rarely interacted items. We systematically evaluate supervised fine-tuning, reinforcement learning-based fine-tuning, and hybrid approaches that combine both methods to optimize recommendation performance. Extensive experiments on real-world data demonstrate significant improvements in both methodological efficacy and practical performance in cold-start recommendation contexts. Remarkably, our reasoning-based fine-tuned models outperform Netflix's production ranking model by up to 8% in certain cases.
Authors Shijun Li, Yu Wang, Jin Wang, Ying Li, Joydeep Ghosh, Anne Cocos
Categories Artificial Intelligence
0
How do data owners say no? A case study of data consent mechanisms in web-scraped vision-language AI training datasets
Title CN 数据所有者如何说不？关于网络抓取的视觉-语言AI训练数据集中数据同意机制的案例研究
Keywords 数据伦理 · 版权 · 数据同意 · 水印检测 · 网络爬虫
Summary 本文研究了在大规模视觉-语言模型训练中，数据所有者如何通过版权标识、水印、网站服务条款（ToS）和robots协议等方式表达对数据被抓取的反对意见。作者分析了包含128亿图文对的DataComp数据集，发现大量样本存在版权提示或来自明确禁止爬取的网站，并指出当前水印检测方法效果有限。研究揭示了现有AI数据收集流程在尊重数据所有权方面的不足，呼吁建立统一的数据同意框架。
Reason 该论文聚焦于AI训练数据的伦理与版权问题，特别是数据所有者通过版权通知、水印、网站条款等方式表达不同意被用于AI训练的现象。虽然涉及视觉-语言数据集（如DataComp），但其核心议题是数据采集的合规性与道德框架，而非文档图像理解（DIU）或VLM技术本身的改进。没有提出可迁移至DIU领域的方法论创新，如视觉压缩、去幻觉、细粒度定位等，因此属于无关主题。
Abstract The internet has become the main source of data to train modern text-to-image or vision-language models, yet it is increasingly unclear whether web-scale data collection practices for training AI systems adequately respect data owners' wishes. Ignoring the owner's indication of consent around data usage not only raises ethical concerns but also has recently been elevated into lawsuits around copyright infringement cases. In this work, we aim to reveal information about data owners' consent to AI scraping and training, and study how it's expressed in DataComp, a popular dataset of 12.8 billion text-image pairs. We examine both the sample-level information, including the copyright notice, watermarking, and metadata, and the web-domain-level information, such as a site's Terms of Service (ToS) and Robots Exclusion Protocol. We estimate at least 122M of samples exhibit some indication of copyright notice in CommonPool, and find that 60\% of the samples in the top 50 domains come from websites with ToS that prohibit scraping. Furthermore, we estimate 9-13\% with 95\% confidence interval of samples from CommonPool to contain watermarks, where existing watermark detection methods fail to capture them in high fidelity. Our holistic methods and findings show that data owners rely on various channels to convey data consent, of which current AI data collection pipelines do not entirely respect. These findings highlight the limitations of the current dataset curation/release practice and the need for a unified data consent framework taking AI purposes into consideration.
Authors Chung Peng Lee, Rachel Hong, Harry H. Jiang, Aster Plotnik, William Agnew, Jamie Morgenstern
Categories Artificial Intelligence
0
Hybrid Agentic AI and Multi-Agent Systems in Smart Manufacturing
Title CN 智能制造中的混合型智能体AI与多智能体系统
Keywords 智能体AI · 多智能体系统 · 预测性维护 · 智能制造 · LLM规划
Summary 本文提出了一种结合Agentic AI与多智能体系统（MAS）的混合框架，用于智能制造中的预测性维护。该框架采用分层结构，由LLM驱动的规划智能体负责高层决策与上下文管理，而基于规则或小模型的智能体在边缘端执行具体任务。系统实现了模式发现、特征分析、模型选择与优化等功能，并通过人机交互接口确保可解释性。实验在两个工业数据集上验证了框架的有效性，展示了其在鲁棒性、可扩展性和可解释性方面的潜力。
Reason 该论文属于典型的平行下游应用，将LLM驱动的智能体框架应用于智能制造中的预测性维护场景。尽管提到了智能体架构和分层设计，但其核心是特定工业场景的应用集成，未提出可用于文档图像理解（如OCR去幻觉、复杂排版处理、细粒度定位等）的底层方法创新。研究内容与文档理解任务无直接关联，且不涉及视觉表示学习、高分辨率处理或多模态对齐等DIU关键技术。
Abstract The convergence of Agentic AI and MAS enables a new paradigm for intelligent decision making in SMS. Traditional MAS architectures emphasize distributed coordination and specialized autonomy, while recent advances in agentic AI driven by LLMs introduce higher order reasoning, planning, and tool orchestration capabilities. This paper presents a hybrid agentic AI and multi agent framework for a Prescriptive Maintenance use case, where LLM based agents provide strategic orchestration and adaptive reasoning, complemented by rule based and SLMs agents performing efficient, domain specific tasks on the edge. The proposed framework adopts a layered architecture that consists of perception, preprocessing, analytics, and optimization layers, coordinated through an LLM Planner Agent that manages workflow decisions and context retention. Specialized agents autonomously handle schema discovery, intelligent feature analysis, model selection, and prescriptive optimization, while a HITL interface ensures transparency and auditability of generated maintenance recommendations. This hybrid design supports dynamic model adaptation, cost efficient maintenance scheduling, and interpretable decision making. An initial proof of concept implementation is validated on two industrial manufacturing datasets. The developed framework is modular and extensible, supporting seamless integration of new agents or domain modules as capabilities evolve. The results demonstrate the system capability to automatically detect schema, adapt preprocessing pipelines, optimize model performance through adaptive intelligence, and generate actionable, prioritized maintenance recommendations. The framework shows promise in achieving improved robustness, scalability, and explainability for RxM in smart manufacturing, bridging the gap between high level agentic reasoning and low level autonomous execution.
Authors Mojtaba A. Farahani, Md Irfan Khan, Thorsten Wuest
Categories Artificial Intelligence
0
1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
Title CN 用于自监督强化学习的1000层网络：扩展深度可以实现新的目标达成能力
Keywords 强化学习 · 自监督学习 · 深层网络 · 目标条件RL · 对比表示学习
Summary 本文探讨了在自监督强化学习中将网络深度扩展到1024层的影响，发现深层网络能显著提升无监督目标导向任务的性能，在模拟运动和操作任务上比现有方法高出2到50倍。尽管在RL领域具有探索价值，但其技术路线与文档图像理解无关。
Reason 该论文研究的是强化学习中的网络深度扩展问题，属于具身智能和机器人控制领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。未涉及OCR、布局分析、表格识别、视觉文本压缩或多模态对齐等DIU核心问题，也无法迁移用于解决文档理解中的幻觉、密集文本或复杂排版等痛点。
Abstract Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 - 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance. Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals. Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by $2\times$ - $50\times$, outperforming other goal-conditioned baselines. Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned. The project webpage and code can be found here: https://wang-kevin3290.github.io/scaling-crl/.
Authors Kevin Wang, Ishaan Javali, Michał Bortkiewicz, Tomasz Trzciński, Benjamin Eysenbach
Comment Link to project website: https://wang-kevin3290.github.io/scaling-crl/
Categories Artificial Intelligence
0
CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generation
Title CN CodeIF-Bench：评估大语言模型在交互式代码生成中的指令跟随能力
Keywords 代码生成 · 指令跟随 · 基准测试 · 多轮交互 · 大语言模型
Summary 本文提出了CodeIF-Bench，一个用于评估大语言模型在多轮交互中遵循用户指令进行代码生成能力的基准测试。它包含九种可验证的指令类型，并通过静态和动态对话设置评估了六个主流LLM的表现，强调上下文管理和交互历史对指令跟随的影响。
Reason 该论文聚焦于代码生成场景下的指令跟随评测，属于特定下游应用（编程助手）的基准构建，未提出可迁移至文档图像理解的底层方法。其任务设定与文档理解核心痛点（如OCR幻觉、复杂排版、细粒度定位等）无直接关联，且未涉及多模态或视觉输入处理，属于典型的‘平行应用’型研究。
Abstract Large Language Models (LLMs) have demonstrated exceptional performance in code generation tasks and have become indispensable programming assistants for developers. However, existing code generation benchmarks primarily assess the functional correctness of code generated by LLMs in single-turn interactions. They offer limited insight into LLMs' abilities to generate code that strictly follows users' instructions in multi-turn interaction scenarios. In this paper, we introduce CodeIF-Bench, a benchmark for evaluating the instruction-following capabilities of LLMs in interactive code generation. Specifically, CodeIF-Bench incorporates nine types of verifiable instructions aligned with the real-world software development requirements, which can be independently and objectively validated through specified test cases, facilitating the evaluation of instruction-following capability in multi-turn interactions. In both Static Conversation and Dynamic Conversation settings, we evaluate the performance of 6 state-of-the-art LLMs and summarize the important factors, additional repository context and gradually increasing interaction history influencing the instruction-following ability of LLMs in multi-turn interactions. Furthermore, we identify the potential direction for improvement: context management. The code and data are available at \href{https://github.com/zhu-zhu-ding/CodeIF-Bench}{https://github.com/zhu-zhu-ding/CodeIF-Bench}.
Authors Peiding Wang, Li Zhang, Fang Liu, Lin Shi, Minxiao Li, Bo Shen, An Fu
Categories Artificial Intelligence
0
Developing an AI Course for Synthetic Chemistry Students
Title CN 为合成化学学生开发人工智能课程
Keywords AI教育 · 合成化学 · 机器学习教学 · 课程设计 · 非编程背景
Summary 本文介绍了一门名为AI4CHEM的面向合成化学学生的数据驱动化学课程，旨在帮助无编程背景的学生掌握Python、分子性质预测、反应优化和数据挖掘等技能。课程采用基于网页的平台进行零安装的机器学习实践，并通过作业、文献综述和协作项目评估学习效果。所有教学材料公开可用。
Reason 该论文属于教育领域的课程设计，讨论如何向合成化学学生教授人工智能，不涉及任何文档图像理解（DIU）或视觉语言模型（VLM）的技术创新。与OCR、DocVQA、布局分析等任务完全无关，属于典型的‘平行应用’类工作，且聚焦于教学而非方法论突破。
Abstract Artificial intelligence (AI) and data science are transforming chemical research, yet few formal courses are tailored to synthetic and experimental chemists, who often face steep entry barriers due to limited coding experience and lack of chemistry-specific examples. We present the design and implementation of AI4CHEM, an introductory data-driven chem-istry course created for students on the synthetic chemistry track with no prior programming background. The curricu-lum emphasizes chemical context over abstract algorithms, using an accessible web-based platform to ensure zero-install machine learning (ML) workflow development practice and in-class active learning. Assessment combines code-guided homework, literature-based mini-reviews, and collaborative projects in which students build AI-assisted workflows for real experimental problems. Learning gains include increased confidence with Python, molecular property prediction, reaction optimization, and data mining, and improved skills in evaluating AI tools in chemistry. All course materials are openly available, offering a discipline-specific, beginner-accessible framework for integrating AI into synthetic chemistry training.
Authors Zhiling Zheng
Comment 17 pages, 3 figures
Categories Artificial Intelligence
0
Can LLMs Help Allocate Public Health Resources? A Case Study on Childhood Lead Testing
Title CN 大语言模型能否帮助分配公共卫生资源？以儿童铅暴露检测为例
Keywords 大语言模型 · 公共卫生 · 资源分配 · 案例研究 · 证据推理
Summary 本文研究了大语言模型在公共卫生资源分配中的应用，通过构建优先级评分系统来评估LLM在芝加哥、纽约和华盛顿特区儿童铅暴露检测资源分配中的表现。实验发现LLM普遍存在忽视高风险社区、依赖过时数据和非实证叙述等问题，显示出在证据推理和信息检索方面的根本性局限。
Reason 该论文属于典型的平行下游应用，将现有LLM技术应用于公共卫生资源分配这一特定领域问题，未提出任何针对文档图像理解（DIU）或VLM底层技术的创新方法。研究内容与文档视觉问答、OCR、布局分析等核心任务无关，且未涉及视觉输入处理、多模态对齐、视觉幻觉缓解等DIU关键挑战。
Abstract Public health agencies face critical challenges in identifying high-risk neighborhoods for childhood lead exposure with limited resources for outreach and intervention programs. To address this, we develop a Priority Score integrating untested children proportions, elevated blood lead prevalence, and public health coverage patterns to support optimized resource allocation decisions across 136 neighborhoods in Chicago, New York City, and Washington, D.C. We leverage these allocation tasks, which require integrating multiple vulnerability indicators and interpreting empirical evidence, to evaluate whether large language models (LLMs) with agentic reasoning and deep research capabilities can effectively allocate public health resources when presented with structured allocation scenarios. LLMs were tasked with distributing 1,000 test kits within each city based on neighborhood vulnerability indicators. Results reveal significant limitations: LLMs frequently overlooked neighborhoods with highest lead prevalence and largest proportions of untested children, such as West Englewood in Chicago, while allocating disproportionate resources to lower-priority areas like Hunts Point in New York City. Overall accuracy averaged 0.46, reaching a maximum of 0.66 with ChatGPT 5 Deep Research. Despite their marketed deep research capabilities, LLMs struggled with fundamental limitations in information retrieval and evidence-based reasoning, frequently citing outdated data and allowing non-empirical narratives about neighborhood conditions to override quantitative vulnerability indicators.
Authors Mohamed Afane, Ying Wang, Juntao Chen
Categories Artificial Intelligence
0
Situationally-Aware Dynamics Learning
Title CN 情境感知动态学习
Keywords 具身智能 · 在线学习 · 隐状态建模 · 贝叶斯变点检测 · 情境感知
Summary 本文提出一种用于自主机器人在线学习隐藏状态表示的新框架，通过广义隐参数马尔可夫决策过程建模未观测因素对动力学和奖励的影响，结合贝叶斯在线变点检测识别环境变化，实现情境感知的自适应决策。实验验证于非结构化地形导航任务，在仿真和真实环境中均提升了策略性能与安全性。
Reason 该论文属于具身智能与机器人领域，研究的是自主机器人在复杂环境中的动态适应问题，核心方法为建模未观测参数对状态转移和奖励的影响，并应用于非结构化地形导航。尽管其在线学习与隐状态建模具有一定方法论价值，但应用场景与文档图像理解（DIU）完全无关，无法迁移解决OCR幻觉、密集文本识别、复杂版式分析或长文档推理等核心痛点。
Abstract Autonomous robots operating in complex, unstructured environments face significant challenges due to latent, unobserved factors that obscure their understanding of both their internal state and the external world. Addressing this challenge would enable robots to develop a more profound grasp of their operational context. To tackle this, we propose a novel framework for online learning of hidden state representations, with which the robots can adapt in real-time to uncertain and dynamic conditions that would otherwise be ambiguous and result in suboptimal or erroneous behaviors. Our approach is formalized as a Generalized Hidden Parameter Markov Decision Process, which explicitly models the influence of unobserved parameters on both transition dynamics and reward structures. Our core innovation lies in learning online the joint distribution of state transitions, which serves as an expressive representation of latent ego- and environmental-factors. This probabilistic approach supports the identification and adaptation to different operational situations, improving robustness and safety. Through a multivariate extension of Bayesian Online Changepoint Detection, our method segments changes in the underlying data generating process governing the robot's dynamics. The robot's transition model is then informed with a symbolic representation of the current situation derived from the joint distribution of latest state transitions, enabling adaptive and context-aware decision-making. To showcase the real-world effectiveness, we validate our approach in the challenging task of unstructured terrain navigation, where unmodeled and unmeasured terrain characteristics can significantly impact the robot's motion. Extensive experiments in both simulation and real world reveal significant improvements in data efficiency, policy performance, and the emergence of safer, adaptive navigation strategies.
Authors Alejandro Murillo-Gonzalez, Lantao Liu
Categories Artificial Intelligence
0
Robust Physical Adversarial Patches Using Dynamically Optimized Clusters
Title CN 使用动态优化聚类的鲁棒物理对抗补丁
Keywords 对抗攻击 · 物理对抗补丁 · 尺度鲁棒性 · 超像素聚类 · SLIC
Summary 本文提出一种基于超像素的正则化方法，通过动态聚类像素来优化对抗补丁的结构，使其在缩放时更具鲁棒性。利用SLIC算法在优化过程中动态聚类像素，并借助隐函数定理实现梯度回传，从而提升补丁在数字和物理世界中的攻击效果。实验表明该方法能有效缓解插值导致的高频信息损失。
Reason 该论文研究的是物理世界中的对抗性补丁攻击，属于计算机视觉安全领域，主要关注如何使对抗补丁在尺度变化下保持有效性。其核心问题与文档图像理解（DIU）无关，既不涉及OCR、布局分析、表格识别等任务，也未提出可用于解决DIU痛点（如OCR幻觉、复杂排版、长文档推理）的通用方法。尽管使用了超像素聚类和可微分SLIC等技术，但这些方法针对的是对抗样本的物理鲁棒性，难以迁移到文档理解场景。
Abstract Physical adversarial attacks on deep learning systems is concerning due to the ease of deploying such attacks, usually by placing an adversarial patch in a scene to manipulate the outcomes of a deep learning model. Training such patches typically requires regularization that improves physical realizability (e.g., printability, smoothness) and/or robustness to real-world variability (e.g. deformations, viewing angle, noise). One type of variability that has received little attention is scale variability. When a patch is rescaled, either digitally through downsampling/upsampling or physically through changing imaging distances, interpolation-induced color mixing occurs. This smooths out pixel values, resulting in a loss of high-frequency patterns and degrading the adversarial signal. To address this, we present a novel superpixel-based regularization method that guides patch optimization to scale-resilient structures. Our ap proach employs the Simple Linear Iterative Clustering (SLIC) algorithm to dynamically cluster pixels in an adversarial patch during optimization. The Implicit Function Theorem is used to backpropagate gradients through SLIC to update the superpixel boundaries and color. This produces patches that maintain their structure over scale and are less susceptible to interpolation losses. Our method achieves greater performance in the digital domain, and when realized physically, these performance gains are preserved, leading to improved physical performance. Real-world performance was objectively assessed using a novel physical evaluation protocol that utilizes screens and cardboard cut-outs to systematically vary real-world conditions.
Authors Harrison Bagley, Will Meakin, Simon Lucey, Yee Wei Law, Tat-Jun Chin
Comment Supplementary material available at: https://drive.google.com/drive/folders/1Yntcc9CARdbvoJJ51cyUm1DWGSvU9X4V?usp=drive_link
Categories Computer Vision and Pattern Recognition
0
From Healthy Scans to Annotated Tumors: A Tumor Fabrication Framework for 3D Brain MRI Synthesis
Title CN 从健康扫描到标注肿瘤：一种用于3D脑部MRI合成的肿瘤生成框架
Keywords 医学图像合成 · 肿瘤生成 · 数据增强 · MRI · 无监督学习
Summary 本文提出了一种名为Tumor Fabrication（TF）的两阶段无配对3D脑肿瘤合成框架，通过健康脑部MRI扫描和少量真实标注数据，自动生成大量配对的合成肿瘤图像用于下游分割任务训练。实验表明该方法在低数据场景下能显著提升肿瘤分割性能，为临床AI中的数据稀缺问题提供了可扩展的解决方案。
Reason 该论文研究的是3D医学影像（MRI）中的肿瘤数据合成问题，属于医疗图像生成领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其方法聚焦于医学图像的数据增强，无法迁移解决文档理解中的OCR幻觉、复杂排版、细粒度定位等核心痛点。
Abstract The scarcity of annotated Magnetic Resonance Imaging (MRI) tumor data presents a major obstacle to accurate and automated tumor segmentation. While existing data synthesis methods offer promising solutions, they often suffer from key limitations: manual modeling is labor intensive and requires expert knowledge. Deep generative models may be used to augment data and annotation, but they typically demand large amounts of training pairs in the first place, which is impractical in data limited clinical settings. In this work, we propose Tumor Fabrication (TF), a novel two-stage framework for unpaired 3D brain tumor synthesis. The framework comprises a coarse tumor synthesis process followed by a refinement process powered by a generative model. TF is fully automated and leverages only healthy image scans along with a limited amount of real annotated data to synthesize large volumes of paired synthetic data for enriching downstream supervised segmentation training. We demonstrate that our synthetic image-label pairs used as data enrichment can significantly improve performance on downstream tumor segmentation tasks in low-data regimes, offering a scalable and reliable solution for medical image enrichment and addressing critical challenges in data scarcity for clinical AI applications.
Authors Nayu Dong, Townim Chowdhury, Hieu Phan, Mark Jenkinson, Johan Verjans, Zhibin Liao
Categories Computer Vision and Pattern Recognition
0
Neural Collapse-Inspired Multi-Label Federated Learning under Label-Distribution Skew
Title CN 受神经崩溃启发的标签分布偏斜下的多标签联邦学习
Keywords 联邦学习 · 多标签分类 · 神经崩溃 · 特征解耦 · 医学图像
Summary 本文提出FedNCA-ML，一种受神经崩溃理论启发的多标签联邦学习框架，通过特征解耦模块提取类别特定表示，并引入正则化损失来促进潜在空间中紧凑且一致的特征聚类，以应对客户端间标签分布异构的问题。在四个基准数据集上的实验表明该方法在AUC和F1分数上均有显著提升。
Reason 该论文研究的是联邦学习中的多标签分类问题，聚焦于医学图像等领域的数据隐私与标签分布偏斜挑战，属于垂直领域应用方法（如医疗影像分析）的算法改进，并未涉及文档图像理解中的核心任务（如DocVQA、布局分析、表格识别或OCR-free理解）。尽管其特征对齐和表示学习机制具有一定通用性，但无法直接迁移解决文档理解中的OCR幻觉、复杂排版或细粒度视觉定位等关键痛点，属于与DIU无关的平行技术路线。
Abstract Federated Learning (FL) enables collaborative model training across distributed clients while preserving data privacy, yet it remains challenging as data distributions can be highly heterogeneous. These challenges are further amplified in multi-label scenarios, where data exhibit characteristics such as label co-occurrence, inter-label dependency, and discrepancies between local and global label relationships. While most existing FL studies focus on single-label classification, real-world applications, such as in medical imaging, involve multi-label data with highly skewed label distributions across clients. To address this important yet underexplored problem, we propose FedNCA-ML, a novel FL framework that aligns feature distributions across clients and learns discriminative, well-clustered representations inspired by Neural Collapse (NC) theory. NC describes an ideal latent-space geometry where each class's features collapse to their mean, forming a maximally separated simplex. To extend this theory to multi-label settings, we introduce a feature disentanglement module that extracts class-specific representations. The clustering of these disentangled features is guided by a shared NC-inspired structure, mitigating conflicts among client models caused by heterogeneous local data. Furthermore, we design regularisation losses to encourage compact and consistent feature clustering in the latent space. Experiments on four benchmark datasets under eight FL settings demonstrate the effectiveness of the proposed method, achieving improvements of up to 3.92% in class-wise AUC and 4.93% in class-wise F1 score.
Authors Can Peng, Yuyuan Liu, Yingyu Yang, Pramit Saha, Qianye Yang, J. Alison Noble
Categories Computer Vision and Pattern Recognition
0
Functional Localization Enforced Deep Anomaly Detection Using Fundus Images
Title CN 基于眼底图像的功能定位增强深度异常检测
Keywords 眼底图像 · 异常检测 · Vision Transformer · 医学影像 · GANomaly
Summary 本文提出了一种基于Vision Transformer的分类器，并结合多种增强策略在多个眼底图像数据集上进行视网膜疾病检测，同时开发了一个基于GANomaly的异常检测框架以提升泛化性和可解释性。实验表明该方法在糖尿病性视网膜病变和年龄相关性黄斑变性检测中表现良好，但在青光眼识别上仍有挑战。
Reason 该论文研究的是医学图像分析中的眼底疾病检测与异常识别，属于纯医疗影像领域，与文档图像理解（DIU）或DocVQA无关。尽管使用了ViT和GANomaly等模型，但其任务目标、数据模态和核心挑战均不涉及OCR、布局分析、表格识别或文档推理等DIU关键问题，无法迁移解决OCR幻觉、复杂排版或长文档理解等痛点。
Abstract Reliable detection of retinal diseases from fundus images is challenged by the variability in imaging quality, subtle early-stage manifestations, and domain shift across datasets. In this study, we systematically evaluated a Vision Transformer (ViT) classifier under multiple augmentation and enhancement strategies across several heterogeneous public datasets, as well as the AEyeDB dataset, a high-quality fundus dataset created in-house and made available for the research community. The ViT demonstrated consistently strong performance, with accuracies ranging from 0.789 to 0.843 across datasets and diseases. Diabetic retinopathy and age-related macular degeneration were detected reliably, whereas glaucoma remained the most frequently misclassified disease. Geometric and color augmentations provided the most stable improvements, while histogram equalization benefited datasets dominated by structural subtlety. Laplacian enhancement reduced performance across different settings. On the Papila dataset, the ViT with geometric augmentation achieved an AUC of 0.91, outperforming previously reported convolutional ensemble baselines (AUC of 0.87), underscoring the advantages of transformer architectures and multi-dataset training. To complement the classifier, we developed a GANomaly-based anomaly detector, achieving an AUC of 0.76 while providing inherent reconstruction-based explainability and robust generalization to unseen data. Probabilistic calibration using GUESS enabled threshold-independent decision support for future clinical implementation.
Authors Jan Benedikt Ruhland, Thorsten Papenbrock, Jan-Peter Sowa, Ali Canbay, Nicole Eter, Bernd Freisleben, Dominik Heider
Categories Computer Vision and Pattern Recognition
0
RigAnyFace: Scaling Neural Facial Mesh Auto-Rigging with Unlabeled Data NeurIPS 2025
Title CN RigAnyFace：利用无标签数据扩展神经面部网格自动绑定
Keywords 3D人脸建模 · 神经绑定 · FACS · 无监督学习 · 表面变形
Summary 本文提出RigAnyFace（RAF），一种可扩展的神经网络框架，用于为具有多样拓扑结构的面部网格（包括多个不连通组件）进行自动绑定。通过一个对三角剖分不敏感的表面学习网络预测FACS参数下的形变，并引入针对无标签中性面部网格的2D监督策略，以扩大训练数据并提升泛化能力。实验表明该方法在精度和适用性上优于先前工作，支持眼球等独立部件的精细化表情动画。
Reason 该论文研究的是3D面部网格的自动绑定与动画生成，属于计算机图形学与3D人脸建模领域，尽管使用了视觉数据和几何学习方法，但其核心任务与文档图像理解（DIU）、DocVQA、OCR、布局分析等完全无关。未涉及文本识别、多模态语言模型在文档上的应用、视觉-语言对齐或文档特有的结构建模，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract In this paper, we present RigAnyFace (RAF), a scalable neural auto-rigging framework for facial meshes of diverse topologies, including those with multiple disconnected components. RAF deforms a static neutral facial mesh into industry-standard FACS poses to form an expressive blendshape rig. Deformations are predicted by a triangulation-agnostic surface learning network augmented with our tailored architecture design to condition on FACS parameters and efficiently process disconnected components. For training, we curated a dataset of facial meshes, with a subset meticulously rigged by professional artists to serve as accurate 3D ground truth for deformation supervision. Due to the high cost of manual rigging, this subset is limited in size, constraining the generalization ability of models trained exclusively on it. To address this, we design a 2D supervision strategy for unlabeled neutral meshes without rigs. This strategy increases data diversity and allows for scaled training, thereby enhancing the generalization ability of models trained on this augmented data. Extensive experiments demonstrate that RAF is able to rig meshes of diverse topologies on not only our artist-crafted assets but also in-the-wild samples, outperforming previous works in accuracy and generalizability. Moreover, our method advances beyond prior work by supporting multiple disconnected components, such as eyeballs, for more detailed expression animation. Project page: https://wenchao-m.github.io/RigAnyFace.github.io
Authors Wenchao Ma, Dario Kneubuehler, Maurice Chu, Ian Sachs, Haomiao Jiang, Sharon Xiaolei Huang
Comment Accepted by NeurIPS 2025
Categories Computer Vision and Pattern Recognition
0
NeAR: Coupled Neural Asset-Renderer Stack
Title CN NeAR：耦合的神经资产-渲染器堆栈
Keywords 神经渲染 · 3D表示 · 神经资产 · 光照归一化 · 可微渲染
Summary 本文提出了NeAR，一种耦合的神经资产-渲染器堆栈，旨在统一神经资产建模与神经渲染流程。通过构建光照归一化的结构化3D潜变量（SLAT）作为神经资产，并设计相应的光照感知神经渲染器，实现高质量、可重光照的实时渲染。实验验证了其在单图重建、新视角合成等任务上的优越性。
Reason 该论文研究的是3D神经渲染与神经资产表示的联合设计，属于3D视觉与计算机图形学领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其核心方法聚焦于光照归一化、视图无关的3D潜变量表示和实时可重光照渲染，无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心问题。
Abstract Neural asset authoring and neural rendering have emerged as fundamentally disjoint threads: one generates digital assets using neural networks for traditional graphics pipelines, while the other develops neural renderers that map conventional assets to images. However, the potential of jointly designing the asset representation and renderer remains largely unexplored. We argue that coupling them can unlock an end-to-end learnable graphics stack with benefits in fidelity, consistency, and efficiency. In this paper, we explore this possibility with NeAR: a Coupled Neural Asset-Renderer Stack. On the asset side, we build on Trellis-style Structured 3D Latents and introduce a lighting-homogenized neural asset: from a casually lit input, a rectified-flow backbone predicts a Lighting-Homogenized SLAT that encodes geometry and intrinsic material cues in a compact, view-agnostic latent. On the renderer side, we design a lighting-aware neural renderer that uses this neural asset, along with explicit view embeddings and HDR environment maps, to achieve real-time, relightable rendering. We validate NeAR on four tasks: (1) G-buffer-based forward rendering, (2) random-lit single-image reconstruction, (3) unknown-lit single-image relighting, and (4) novel-view relighting. Our coupled stack surpasses state-of-the-art baselines in both quantitative metrics and perceptual quality. We hope this coupled asset-renderer perspective inspires future graphics stacks that view neural assets and renderers as co-designed components instead of independent entities.
Authors Hong Li, Chongjie Ye, Houyuan Chen, Weiqing Xiao, Ziyang Yan, Lixing Xiao, Zhaoxi Chen, Jianfeng Xiang, Shaocong Xu, Xuhui Liu, Yikai Wang, Baochang Zhang, Xiaoguang Han, Jiaolong Yang, Hao Zhao
Comment 20 pages, 16 figures
Categories Computer Vision and Pattern Recognition
0
Fusing Biomechanical and Spatio-Temporal Features for Fall Prediction: Characterizing and Mitigating the Simulation-to-Reality Gap
Title CN 融合生物力学与时空特征的跌倒预测：表征与缓解仿真到现实的差距
Keywords 跌倒预测 · 图卷积网络 · 仿真到现实差距 · 姿态估计 · 跨注意力
Summary 本文提出了一种名为BioST-GCN的双流图卷积网络，用于基于姿态和生物力学信息的跌倒预测。模型在模拟数据集上表现良好，但在真实场景中泛化能力显著下降，揭示了仿真与现实之间的巨大差距。作者呼吁通过个性化策略和隐私保护数据管道来弥合这一差距。
Reason 该论文属于视频理解领域，研究基于视觉的跌倒预测，使用ST-GCN和跨模态注意力机制处理人体姿态序列。尽管涉及视觉建模，但其任务、数据和方法均与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版或长文档推理等核心问题。
Abstract Falls are a leading cause of injury and loss of independence among older adults. Vision-based fall prediction systems offer a non-invasive solution to anticipate falls seconds before impact, but their development is hindered by the scarcity of available fall data. Contributing to these efforts, this study proposes the Biomechanical Spatio-Temporal Graph Convolutional Network (BioST-GCN), a dual-stream model that combines both pose and biomechanical information using a cross-attention fusion mechanism. Our model outperforms the vanilla ST-GCN baseline by 5.32% and 2.91% F1-score on the simulated MCF-UA stunt-actor and MUVIM datasets, respectively. The spatio-temporal attention mechanisms in the ST-GCN stream also provide interpretability by identifying critical joints and temporal phases. However, a critical simulation-reality gap persists. While our model achieves an 89.0% F1-score with full supervision on simulated data, zero-shot generalization to unseen subjects drops to 35.9%. This performance decline is likely due to biases in simulated data, such as 'intent-to-fall' cues. For older adults, particularly those with diabetes or frailty, this gap is exacerbated by their unique kinematic profiles. To address this, we propose personalization strategies and advocate for privacy-preserving data pipelines to enable real-world validation. Our findings underscore the urgent need to bridge the gap between simulated and real-world data to develop effective fall prediction systems for vulnerable elderly populations.
Authors Md Fokhrul Islam, Sajeda Al-Hammouri, Christopher J. Arellano, Kavan Hazeli, Heman Shakeri
Categories Computer Vision and Pattern Recognition
0
PhysGS: Bayesian-Inferred Gaussian Splatting for Physical Property Estimation Submitted to CVPR 2026
Title CN PhysGS：用于物理属性估计的贝叶斯推断高斯溅射
Keywords 3D重建 · 高斯溅射 · 物理属性估计 · 贝叶斯推理 · 不确定性建模
Summary 本文提出PhysGS，一种基于贝叶斯推断的3D高斯溅射扩展方法，用于从视觉线索和视觉-语言先验中估计每个点的密集物理属性（如质量、硬度、摩擦系数）。该方法将属性估计建模为对高斯点的贝叶斯推理过程，并同时建模随机性和认知不确定性，从而实现不确定性感知的物理解释。实验表明，相比确定性基线，PhysGS在多个真实数据集上显著提升了物理属性估计的准确性。
Reason 该论文研究的是3D场景中物体物理属性（如摩擦力、硬度、刚度）的估计，属于具身智能与机器人交互领域，核心任务是基于视觉和语言先验进行物理建模。其方法论聚焦于高斯溅射与贝叶斯推理在3D空间中的结合，与文档图像理解（DIU）或DocVQA无关。无法迁移解决OCR幻觉、密集文本识别、复杂版式分析等DIU核心问题。
Abstract Understanding physical properties such as friction, stiffness, hardness, and material composition is essential for enabling robots to interact safely and effectively with their surroundings. However, existing 3D reconstruction methods focus on geometry and appearance and cannot infer these underlying physical properties. We present PhysGS, a Bayesian-inferred extension of 3D Gaussian Splatting that estimates dense, per-point physical properties from visual cues and vision--language priors. We formulate property estimation as Bayesian inference over Gaussian splats, where material and property beliefs are iteratively refined as new observations arrive. PhysGS also models aleatoric and epistemic uncertainties, enabling uncertainty-aware object and scene interpretation. Across object-scale (ABO-500), indoor, and outdoor real-world datasets, PhysGS improves accuracy of the mass estimation by up to 22.8%, reduces Shore hardness error by up to 61.2%, and lowers kinetic friction error by up to 18.1% compared to deterministic baselines. Our results demonstrate that PhysGS unifies 3D reconstruction, uncertainty modeling, and physical reasoning in a single, spatially continuous framework for dense physical property estimation. Additional results are available at https://samchopra2003.github.io/physgs.
Authors Samarth Chopra, Jing Liang, Gershom Seneviratne, Dinesh Manocha
Comment Submitted to CVPR 2026
Categories Computer Vision and Pattern Recognition
0
Unreal Robotics Lab: A High-Fidelity Robotics Simulator with Advanced Physics and Rendering
Title CN Unreal Robotics Lab：一个具有先进物理和渲染的高保真机器人模拟器
Keywords 机器人仿真 · 高保真渲染 · 物理模拟 · 视觉导航 · SLAM
Summary 本文提出了一种名为Unreal Robotics Lab（URL）的高保真机器人模拟框架，通过集成Unreal Engine的高质量渲染与MuJoCo的精确物理引擎，支持烟雾、火焰、水流等复杂环境效应，可用于视觉导航和SLAM方法的鲁棒性评估。系统旨在促进仿真到现实的迁移研究。
Reason 该论文属于具身智能/机器人仿真领域，研究重点是结合Unreal Engine与MuJoCo实现高保真的视觉-物理联合仿真，用于SLAM和导航算法测试。虽然涉及计算机视觉，但其核心任务与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点，属于负面清单中的‘无关领域’。
Abstract High-fidelity simulation is essential for robotics research, enabling safe and efficient testing of perception, control, and navigation algorithms. However, achieving both photorealistic rendering and accurate physics modeling remains a challenge. This paper presents a novel simulation framework, the Unreal Robotics Lab (URL), that integrates the advanced rendering capabilities of the Unreal Engine with MuJoCo's high-precision physics simulation. Our approach enables realistic robotic perception while maintaining accurate physical interactions, facilitating benchmarking and dataset generation for vision-based robotics applications. The system supports complex environmental effects, such as smoke, fire, and water dynamics, which are critical to evaluating robotic performance under adverse conditions. We benchmark visual navigation and SLAM methods within our framework, demonstrating its utility for testing real-world robustness in controlled yet diverse scenarios. By bridging the gap between physics accuracy and photorealistic rendering, our framework provides a powerful tool for advancing robotics research and sim-to-real transfer. Our open-source framework is available at https://unrealroboticslab.github.io/.
Authors Jonathan Embley-Riches, Jianwei Liu, Simon Julier, Dimitrios Kanoulas
Categories Computer Vision and Pattern Recognition
0
A Training-Free Style-aligned Image Generation with Scale-wise Autoregressive Model
Title CN 一种无需训练的尺度自回归风格对齐图像生成方法
Keywords 图像生成 · 风格对齐 · 扩散模型 · 自回归模型 · 无需训练
Summary 本文提出了一种无需训练的风格对齐图像生成方法，通过尺度自回归模型实现图像集合间的风格一致性。方法包括初始特征替换、关键特征插值和动态风格注入，在不牺牲内容细节的前提下提升了生成效率和风格统一性。实验表明该方法在速度上显著优于现有方案。
Reason 该论文属于纯图像生成领域，研究的是文本到图像生成中的风格一致性问题，与文档图像理解（DIU）、DocVQA、OCR、布局分析等核心任务完全无关。尽管其方法涉及视觉生成和多尺度建模，但目标场景和挑战（如风格对齐、生成速度）不适用于解决文档理解中的OCR幻觉、密集文本识别、复杂排版或长文档推理等问题。属于典型的无关领域工作。
Abstract We present a training-free style-aligned image generation method that leverages a scale-wise autoregressive model. While large-scale text-to-image (T2I) models, particularly diffusion-based methods, have demonstrated impressive generation quality, they often suffer from style misalignment across generated image sets and slow inference speeds, limiting their practical usability. To address these issues, we propose three key components: initial feature replacement to ensure consistent background appearance, pivotal feature interpolation to align object placement, and dynamic style injection, which reinforces style consistency using a schedule function. Unlike previous methods requiring fine-tuning or additional training, our approach maintains fast inference while preserving individual content details. Extensive experiments show that our method achieves generation quality comparable to competing approaches, significantly improves style alignment, and delivers inference speeds over six times faster than the fastest model.
Authors Jihun Park, Jongmin Gim, Kyoungmin Lee, Minseok Oh, Minwoo Choi, Jaeyeul Kim, Woo Chool Park, Sunghoon Im
Comment 18 pages, 15 figures
Categories Computer Vision and Pattern Recognition
0
TimePre: Bridging Accuracy, Efficiency, and Stability in Probabilistic Time-Series Forecasting
Title CN TimePre：在概率时间序列预测中平衡准确性、效率与稳定性
Keywords 时间序列预测 · 概率预测 · MCL · 归一化层 · 训练稳定性
Summary 本文提出了TimePre框架，旨在解决概率时间序列预测中生成模型效率低和非采样方法训练不稳定的矛盾。通过引入Stabilized Instance Normalization（SIN），该方法成功结合了MLP主干的高效性与多选学习（MCL）范式的分布灵活性，有效缓解了假设崩溃问题，在多个基准数据集上实现了更优的准确性、效率和稳定性平衡。
Reason 该论文研究的是概率时间序列预测（PTSF），属于时间序列分析领域，尽管使用了MLP架构和提出SIN归一化层以解决训练不稳定问题，但其核心任务与文档图像理解（DIU）无关。未涉及多模态输入、视觉文本识别、布局分析或VLM在文档上的应用。无法迁移用于解决OCR幻觉、复杂排版或细粒度定位等DIU核心痛点。
Abstract Probabilistic Time-Series Forecasting (PTSF) is critical for uncertainty-aware decision making, but existing generative models, such as diffusion-based approaches, are computationally prohibitive due to expensive iterative sampling. Non-sampling frameworks like Multiple Choice Learning (MCL) offer an efficient alternative, but suffer from severe training instability and hypothesis collapse, which has historically hindered their performance. This problem is dramatically exacerbated when attempting to combine them with modern, efficient MLP-based backbones. To resolve this fundamental incompatibility, we propose TimePre, a novel framework that successfully unifies the efficiency of MLP-based models with the distributional flexibility of the MCL paradigm. The core of our solution is Stabilized Instance Normalization (SIN), a novel normalization layer that explicitly remedies this incompatibility. SIN stabilizes the hybrid architecture by correcting channel-wise statistical shifts, definitively resolving the catastrophic hypothesis collapse. Extensive experiments on six benchmark datasets demonstrate that TimePre achieves new state-of-the-art accuracy on key probabilistic metrics. Critically, TimePre achieves inference speeds orders of magnitude faster than sampling-based models and, unlike prior MCL work, demonstrates stable performance scaling. It thus bridges the long-standing gap between accuracy, efficiency, and stability in probabilistic forecasting.
Authors Lingyu Jiang, Lingyu Xu, Peiran Li, Qianwen Ge, Dingyi Zhuang, Shuo Xing, Wenjing Chen, Xiangbo Gao, Ting-Hsuan Chen, Xueying Zhan, Xin Zhang, Ziming Zhang, Zhengzhong Tu, Michael Zielewski, Kazunori Yamada, Fangzhou Lin
Comment 15 pages, 5 figures, 6 tables
Categories Computer Vision and Pattern Recognition
0
Zero-Shot Video Deraining with Video Diffusion Models WACV 2026
Title CN 基于视频扩散模型的零样本视频去雨
Keywords 视频去雨 · 扩散模型 · 零样本学习 · 注意力机制 · 图像恢复
Summary 本文提出了一种无需配对数据和模型微调的零样本视频去雨方法，利用预训练的文本到视频扩散模型，通过潜空间反转和负向提示干预来去除视频中的雨迹。核心创新是注意力切换机制，以保持动态背景和结构一致性。实验表明该方法在真实世界雨天数据上表现优异。
Reason 该论文研究的是视频去雨任务，属于视频恢复与生成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了扩散模型和注意力机制等技术，但其解决的问题（动态场景下的雨滴去除）在文档理解中不存在对应痛点，无法迁移用于解决OCR幻觉、复杂排版、长文档推理等核心问题。
Abstract Existing video deraining methods are often trained on paired datasets, either synthetic, which limits their ability to generalize to real-world rain, or captured by static cameras, which restricts their effectiveness in dynamic scenes with background and camera motion. Furthermore, recent works in fine-tuning diffusion models have shown promising results, but the fine-tuning tends to weaken the generative prior, limiting generalization to unseen cases. In this paper, we introduce the first zero-shot video deraining method for complex dynamic scenes that does not require synthetic data nor model fine-tuning, by leveraging a pretrained text-to-video diffusion model that demonstrates strong generalization capabilities. By inverting an input video into the latent space of diffusion models, its reconstruction process can be intervened and pushed away from the model's concept of rain using negative prompting. At the core of our approach is an attention switching mechanism that we found is crucial for maintaining dynamic backgrounds as well as structural consistency between the input and the derained video, mitigating artifacts introduced by naive negative prompting. Our approach is validated through extensive experiments on real-world rain datasets, demonstrating substantial improvements over prior methods and showcasing robust generalization without the need for supervised training.
Authors Tuomas Varanka, Juan Luis Gonzalez, Hyeongwoo Kim, Pablo Garrido, Xu Yao
Comment WACV 2026
Categories Computer Vision and Pattern Recognition
0
HiFi-MambaV2: Hierarchical Shared-Routed MoE for High-Fidelity MRI Reconstruction
Title CN HiFi-MambaV2：用于高保真MRI重建的分层共享路由MoE
Keywords MRI重建 · Mamba · 混合专家系统 · 频率分解 · 医学图像
Summary 本文提出HiFi-MambaV2，一种结合分层共享路由混合专家系统（MoE）与频率分解的Mamba架构，用于从欠采样的k空间数据中重建高保真MRI图像。通过可分离的频率一致拉普拉斯金字塔（SF-Lap）稳定高低频流，并利用逐像素稀疏调度机制提升计算效率，在多个MRI数据集上实现了优于CNN、Transformer及先前Mamba模型的重建效果。
Reason 该论文研究的是医学图像重建任务，属于MRI重建领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其核心技术聚焦于频率分解与MoE架构在医学影像中的应用，无法迁移解决文档理解中的OCR幻觉、复杂排版、细粒度定位等核心问题。
Abstract Reconstructing high-fidelity MR images from undersampled k-space data requires recovering high-frequency details while maintaining anatomical coherence. We present HiFi-MambaV2, a hierarchical shared-routed Mixture-of-Experts (MoE) Mamba architecture that couples frequency decomposition with content-adaptive computation. The model comprises two core components: (i) a separable frequency-consistent Laplacian pyramid (SF-Lap) that delivers alias-resistant, stable low- and high-frequency streams; and (ii) a hierarchical shared-routed MoE that performs per-pixel top-1 sparse dispatch to shared experts and local routers, enabling effective specialization with stable cross-depth behavior. A lightweight global context path is fused into an unrolled, data-consistency-regularized backbone to reinforce long-range reasoning and preserve anatomical coherence. Evaluated on fastMRI, CC359, ACDC, M4Raw, and Prostate158, HiFi-MambaV2 consistently outperforms CNN-, Transformer-, and prior Mamba-based baselines in PSNR, SSIM, and NMSE across single- and multi-coil settings and multiple acceleration factors, consistently surpassing consistent improvements in high-frequency detail and overall structural fidelity. These results demonstrate that HiFi-MambaV2 enables reliable and robust MRI reconstruction.
Authors Pengcheng Fang, Hongli Chen, Guangzhen Yao, Jian Shi, Fangfang Tang, Xiaohao Cai, Shanshan Shan, Feng Liu
Categories Computer Vision and Pattern Recognition
0
DE-KAN: A Kolmogorov Arnold Network with Dual Encoder for accurate 2D Teeth Segmentation
Title CN DE-KAN：一种用于精确2D牙齿分割的双编码器Kolmogorov Arnold网络
Keywords 牙齿分割 · 医学图像分割 · 双编码器 · KAN · 全景X光片
Summary 本文提出了一种名为DE-KAN的双编码器Kolmogorov Arnold网络，用于提升全景牙片中单颗牙齿的分割精度。该模型结合ResNet-18编码器处理增强输入和自定义CNN编码器处理原始输入，通过KAN瓶颈层融合全局与局部空间特征，并利用可学习的非线性激活函数增强表达能力。在两个牙科X光数据集上取得了优于现有方法的性能表现。
Reason 该论文聚焦于医学图像中的牙齿分割任务，属于纯医学图像分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管其提出的双编码器结构和KAN模块可能在特征融合方面有一定创新，但整体方法针对的是生物医学影像中的解剖结构分割问题，无法迁移解决文档图像中的OCR幻觉、复杂排版、细粒度文本定位或长文档推理等核心痛点。
Abstract Accurate segmentation of individual teeth from panoramic radiographs remains a challenging task due to anatomical variations, irregular tooth shapes, and overlapping structures. These complexities often limit the performance of conventional deep learning models. To address this, we propose DE-KAN, a novel Dual Encoder Kolmogorov Arnold Network, which enhances feature representation and segmentation precision. The framework employs a ResNet-18 encoder for augmented inputs and a customized CNN encoder for original inputs, enabling the complementary extraction of global and local spatial features. These features are fused through KAN-based bottleneck layers, incorporating nonlinear learnable activation functions derived from the Kolmogorov Arnold representation theorem to improve learning capacity and interpretability. Extensive experiments on two benchmark dental X-ray datasets demonstrate that DE-KAN outperforms state-of-the-art segmentation models, achieving mIoU of 94.5%, Dice coefficient of 97.1%, accuracy of 98.91%, and recall of 97.36%, representing up to +4.7% improvement in Dice compared to existing methods.
Authors Md Mizanur Rahman Mustakim, Jianwu Li, Sumya Bhuiyan, Mohammad Mehedi Hasan, Bing Han
Categories Computer Vision and Pattern Recognition
0
PSA-MIL: A Probabilistic Spatial Attention-Based Multiple Instance Learning for Whole Slide Image Classification Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2026
Title CN PSA-MIL：一种基于概率空间注意力的多实例学习方法用于全切片图像分类
Keywords 多实例学习 · 自注意力 · 空间建模 · 全切片图像 · 医学图像分析
Summary 本文提出PSA-MIL，一种结合概率空间注意力的多实例学习框架，用于全切片图像分类。通过引入可学习的距离衰减先验，在自注意力中建模空间上下文，并采用后验剪枝降低计算复杂度，同时设计多样性损失促使不同注意力头捕捉多样化的空间模式。方法在多个基准上达到SOTA性能。
Reason 该论文研究的是医学全切片图像（WSI）分类，属于数字病理学领域，尽管使用了视觉注意力和空间建模技术，但其任务场景、数据特性与文档图像理解（DIU）差异巨大。虽然其提出的概率空间注意力机制和空间剪枝策略在高分辨率图像处理方面有一定创新，但核心关注点是组织病理学结构的空间关系建模，而非文档中的文本布局、语义解析或OCR相关问题。无法直接迁移解决DIU中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Whole Slide Images (WSIs) are high-resolution digital scans widely used in medical diagnostics. WSI classification is typically approached using Multiple Instance Learning (MIL), where the slide is partitioned into tiles treated as interconnected instances. While attention-based MIL methods aim to identify the most informative tiles, they often fail to fully exploit the spatial relationships among them, potentially overlooking intricate tissue structures crucial for accurate diagnosis. To address this limitation, we propose Probabilistic Spatial Attention MIL (PSA-MIL), a novel attention-based MIL framework that integrates spatial context into the attention mechanism through learnable distance-decayed priors, formulated within a probabilistic interpretation of self-attention as a posterior distribution. This formulation enables a dynamic inference of spatial relationships during training, eliminating the need for predefined assumptions often imposed by previous approaches. Additionally, we suggest a spatial pruning strategy for the posterior, effectively reducing self-attention's quadratic complexity. To further enhance spatial modeling, we introduce a diversity loss that encourages variation among attention heads, ensuring each captures distinct spatial representations. Together, PSA-MIL enables a more data-driven and adaptive integration of spatial context, moving beyond predefined constraints. We achieve state-of-the-art performance across both contextual and non-contextual baselines, while significantly reducing computational costs.
Authors Sharon Peled, Yosef E. Maruvka, Moti Freiman
Comment Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2026
Categories Computer Vision and Pattern Recognition
0
AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding NeurIPS 2025
Title CN AdaVideoRAG：面向长视频理解的全上下文自适应检索增强方法
Keywords 视频理解 · 检索增强生成 · 多模态大模型 · 自适应推理 · 知识图谱
Summary 本文提出AdaVideoRAG，一种面向长视频理解的自适应检索增强框架。通过轻量级意图分类器根据问题复杂度动态选择检索策略，并构建包含文本、视觉和知识图谱的全模态索引系统，实现从简单到复杂推理的分层知识获取。同时发布新基准HiVU用于评估深度视频理解。实验表明该方法在准确性和效率之间取得更好平衡。
Reason 该论文研究的是长视频理解中的检索增强生成框架，属于视频理解领域，尽管使用了OCR、ASR等多模态信息，但其核心任务和应用场景与文档图像理解（DIU）无关。虽然其‘自适应检索’和‘多数据库分层访问’机制在理论上具有方法论价值，但整体设计针对的是视频时序结构和查询复杂度动态调整，难以直接迁移至文档图像中的细粒度定位、高分辨率处理或OCR幻觉等问题。命中负面清单第2条：无关领域（视频理解）。
Abstract Multimodal Large Language Models (MLLMs) perform well in video understanding but degrade on long videos due to fixed-length context and weak long-term dependency modeling. Retrieval-Augmented Generation (RAG) can expand knowledge dynamically, yet existing video RAG schemes adopt fixed retrieval paradigms that ignore query difficulty. This uniform design causes redundant computation and latency for simple queries, while coarse retrieval for complex, multi-hop reasoning can miss key information. Such single-step retrieval severely limits the trade-off between efficiency and cognitive depth. We propose AdaVideoRAG, an adaptive RAG framework for long-video understanding. A lightweight intent classifier dynamically selects suitable retrieval schemes according to query complexity from the simplest to the most sophisticated. We design an Omni-Knowledge Indexing module that extracts and organizes multi-modal information into three databases: (1) a text base built from clip captions, ASR, and OCR; (2) a visual base; and (3) a knowledge graph for deep semantic understanding. This supports hierarchical knowledge access, from naive retrieval to graph-based retrieval, balancing resource cost and reasoning ability. To evaluate deep understanding, we further construct the HiVU benchmark. Experiments show that AdaVideoRAG significantly improves both efficiency and accuracy on long-video QA tasks and can be seamlessly plugged into existing MLLMs through lightweight APIs, establishing a new paradigm for adaptive retrieval-augmented video analysis.
Authors Zhucun Xue, Jiangning Zhang, Xurong Xie, Yuxuan Cai, Yong Liu, Xiangtai Li, Dacheng Tao
Comment NeurIPS 2025
Categories Computer Vision and Pattern Recognition
0
Unified Deep Learning Platform for Dust and Fault Diagnosis in Solar Panels Using Thermal and Visual Imaging
Title CN 基于热成像和可见光成像的太阳能电池板灰尘与故障诊断统一深度学习平台
Keywords 太阳能电池板 · 故障检测 · 图像分类 · CNN · 热成像
Summary 本文提出了一种结合热成像和可见光图像的深度学习模型，用于检测太阳能电池板上的灰尘和故障。通过预处理（如去伽马、高斯滤波）和CNN/ResNet等结构实现分类，声称在效率和准确率上优于现有模型。但整体为面向特定工程场景的应用系统，缺乏通用性与理论突破。
Reason 该论文属于特定工业检测场景的应用研究，聚焦于太阳能电池板的灰尘与故障检测，使用的是传统CNN、ResNet和自注意力机制（KerNet）进行图像分类。其方法未涉及文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别或OCR-free架构设计，也未提出可用于解决DIU痛点（如OCR幻觉、复杂排版、长文档推理）的通用底层技术。属于典型的‘平行下游应用’，无方法论创新。
Abstract Solar energy is one of the most abundant and tapped sources of renewable energies with enormous future potential. Solar panel output can vary widely with factors like intensity, temperature, dirt, debris and so on affecting it. We have implemented a model on detecting dust and fault on solar panels. These two applications are centralized as a single-platform and can be utilized for routine-maintenance and any other checks. These are checked against various parameters such as power output, sinusoidal wave (I-V component of solar cell), voltage across each solar cell and others. Firstly, we filter and preprocess the obtained images using gamma removal and Gaussian filtering methods alongside some predefined processes like normalization. The first application is to detect whether a solar cell is dusty or not based on various pre-determined metrics like shadowing, leaf, droppings, air pollution and from other human activities to extent of fine-granular solar modules. The other one is detecting faults and other such occurrences on solar panels like faults, cracks, cell malfunction using thermal imaging application. This centralized platform can be vital since solar panels have different efficiency across different geography (air and heat affect) and can also be utilized for small-scale house requirements to large-scale solar farm sustentation effectively. It incorporates CNN, ResNet models that with self-attention mechanisms-KerNet model which are used for classification and results in a fine-tuned system that detects dust or any fault occurring. Thus, this multi-application model proves to be efficient and optimized in detecting dust and faults on solar panels. We have performed various comparisons and findings that demonstrates that our model has better efficiency and accuracy results overall than existing models.
Authors Abishek Karthik, Sreya Mynampati, Pandiyaraju V
Categories Computer Vision and Pattern Recognition
0
LRDUN: A Low-Rank Deep Unfolding Network for Efficient Spectral Compressive Imaging
Title CN LRDUN：一种用于高效光谱压缩成像的低秩深度展开网络
Keywords 光谱压缩成像 · 深度展开网络 · 低秩分解 · 图像重建 · 计算成像
Summary 本文提出了一种基于低秩分解的深度展开网络LRDUN，用于高效光谱压缩成像的重建。通过将高维超分辨率图像分解为低维基和子空间图像，设计了两个新的成像模型，并在展开的近端梯度下降框架下联合求解。同时引入广义特征展开机制（GFUM），提升网络表达能力。实验表明该方法在重建质量和计算效率上达到SOTA。
Reason 该论文研究的是光谱压缩成像（Spectral Compressive Imaging）中的图像重建问题，属于计算成像领域，核心关注点是高维光谱数据的重构与效率优化。尽管使用了深度展开网络（DUN）并引入低秩分解，但其任务本质是纯视觉信号恢复，与文档图像理解（DIU）中的文本识别、布局分析、VQA等任务无直接关联。方法上未涉及多模态大模型、视觉语言对齐、OCR去幻觉或文档结构建模等DIU关键痛点，也无法迁移用于解决文档理解中的复杂排版、长文档推理等问题。
Abstract Deep unfolding networks (DUNs) have achieved remarkable success and become the mainstream paradigm for spectral compressive imaging (SCI) reconstruction. Existing DUNs are derived from full-HSI imaging models, where each stage operates directly on the high-dimensional HSI, refining the entire data cube based on the single 2D coded measurement. However, this paradigm leads to computational redundancy and suffers from the ill-posed nature of mapping 2D residuals back to 3D space of HSI. In this paper, we propose two novel imaging models corresponding to the spectral basis and subspace image by explicitly integrating low-rank (LR) decomposition with the sensing model. Compared to recovering the full HSI, estimating these compact low-dimensional components significantly mitigates the ill-posedness. Building upon these novel models, we develop the Low-Rank Deep Unfolding Network (LRDUN), which jointly solves the two subproblems within an unfolded proximal gradient descent (PGD) framework. Furthermore, we introduce a Generalized Feature Unfolding Mechanism (GFUM) that decouples the physical rank in the data-fidelity term from the feature dimensionality in the prior module, enhancing the representational capacity and flexibility of the network. Extensive experiments on simulated and real datasets demonstrate that the proposed LRDUN achieves state-of-the-art (SOTA) reconstruction quality with significantly reduced computational cost.
Authors He Huang, Yujun Guo, Wei He
Comment 17 pages, 16 figures,
Categories Computer Vision and Pattern Recognition
0
VPN: Visual Prompt Navigation AAAI 2026
Title CN VPN：视觉提示导航
Keywords 视觉导航 · 具身智能 · 视觉提示 · 多模态学习 · 数据增强
Summary 本文提出了一种名为视觉提示导航（VPN）的新范式，通过用户在2D俯视图上标注的视觉路径来引导智能体进行导航，避免了自然语言指令的模糊性。作者构建了两个新数据集R2R-VP和R2R-CE-VP，并设计了VPNet作为基线模型，结合视图级和轨迹级数据增强策略提升性能。实验探讨了不同视觉提示形式和地图格式对导航效果的影响。
Reason 该论文研究的是具身智能中的视觉导航任务，属于机器人与3D场景交互领域，使用2D俯视图和视觉轨迹提示来指导智能体在3D环境中移动。尽管涉及视觉-语言或多模态输入，但其核心任务与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等核心问题。属于‘无关领域’，命中负面清单第2条。
Abstract While natural language is commonly used to guide embodied agents, the inherent ambiguity and verbosity of language often hinder the effectiveness of language-guided navigation in complex environments. To this end, we propose Visual Prompt Navigation (VPN), a novel paradigm that guides agents to navigate using only user-provided visual prompts within 2D top-view maps. This visual prompt primarily focuses on marking the visual navigation trajectory on a top-down view of a scene, offering intuitive and spatially grounded guidance without relying on language instructions. It is more friendly for non-expert users and reduces interpretive ambiguity. We build VPN tasks in both discrete and continuous navigation settings, constructing two new datasets, R2R-VP and R2R-CE-VP, by extending existing R2R and R2R-CE episodes with corresponding visual prompts. Furthermore, we introduce VPNet, a dedicated baseline network to handle the VPN tasks, with two data augmentation strategies: view-level augmentation (altering initial headings and prompt orientations) and trajectory-level augmentation (incorporating diverse trajectories from large-scale 3D scenes), to enhance navigation performance. Extensive experiments evaluate how visual prompt forms, top-view map formats, and data augmentation strategies affect the performance of visual prompt navigation. The code is available at https://github.com/farlit/VPN.
Authors Shuo Feng, Zihan Wang, Yuchen Li, Rui Kong, Hengyi Cai, Shuaiqiang Wang, Gim Hee Lee, Piji Li, Shuqiang Jiang
Comment Accepted by AAAI 2026
Categories Computer Vision and Pattern Recognition
0
Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPO
Title CN 以视频为答案：通过联合GRPO预测并生成下一个视频事件
Keywords 视频生成 · 下一事件预测 · 视觉语言模型 · 强化学习 · 多模态
Summary 本文提出‘以视频为答案’的新范式，通过构建VANS模型和VANS-Data-100K数据集，实现基于指令的下一视频事件预测与生成。采用强化学习中的Joint-GRPO方法联合优化视觉语言模型与视频扩散模型，提升生成视频的语义与视觉一致性。主要应用于 procedural learning 和 creative exploration 场景。
Reason 该论文研究的是视频生成作为回答模态的‘下一事件预测’任务（VNEP），属于视频理解与生成领域。其核心方法Joint-GRPO用于协调视觉语言模型与视频扩散模型之间的训练，目标是生成具有语义一致性的动态视频响应。尽管涉及VLM，但应用场景和任务形式完全偏离文档图像理解（DIU）的核心问题，如文本识别、布局分析、表格解析或OCR去幻觉等。无法迁移至文档密集文本处理、复杂排版理解或长文档推理等关键痛点。
Abstract While language models have become impactful in many real-world applications, video generation remains largely confined to entertainment. Motivated by video's inherent capacity to demonstrate physical-world information that is difficult to convey through language alone (e.g., imagine teaching someone to tie a tie using only text), we identify an underutilized opportunity to extend video as a new answer modality for Next-Event Prediction (NEP), formalized as Video-Next-Event Prediction (VNEP). While the established NEP task takes a video with a procedural or predictive question as input to predict the next event in text, VNEP requires dynamic video responses. This shift from telling to showing unlocks more intuitive and customized answers for procedural learning and creative exploration. However, this task remains challenging for existing models, as it demands an understanding of multimodal input, instruction-conditioned reasoning, and the generation of video with visual and semantic consistency. To address this, we introduce VANS, a model that leverages reinforcement learning to align a Vision-Language Model (VLM) with a Video Diffusion Model (VDM) for VNEP. The core of VANS is our proposed Joint-GRPO that orchestrates the VLM and VDM to function as a unit. Driven by a shared reward on their respective output, it optimizes the VLM to produce captions that are both accurate and friendly to visualize, while guiding the VDM to generate videos that are faithful to these captions and the input visual context. To enable this learning, we craft VANS-Data-100K, a dedicated dataset for the VNEP task. Experiments on procedural and predictive benchmarks demonstrate that VANS achieves state-of-the-art performance in both video event prediction and visualization. Codes are released in https://github.com/KlingTeam/VANS.
Authors Junhao Cheng, Liang Hou, Xin Tao, Jing Liao
Comment Project page: https://video-as-answer.github.io/
Categories Computer Vision and Pattern Recognition
0
Uncertainty Quantification in HSI Reconstruction using Physics-Aware Diffusion Priors and Optics-Encoded Measurements
Title CN 基于物理感知扩散先验和光学编码测量的高光谱图像重建中的不确定性量化
Keywords 高光谱图像重建 · 不确定性量化 · 扩散模型 · 贝叶斯推断 · 同色异谱
Summary 本文提出HSDiff框架，通过贝叶斯推断与扩散模型结合的方式，在高光谱图像重建中实现不确定性量化。利用无条件训练的像素级扩散先验和后验采样生成符合测量数据的多样本结果，并通过增强的同色异谱数据增广提升先验多样性。研究表明有效的光谱编码能提供更优的不确定性校准。
Reason 该论文研究的是高光谱图像（HSI）重建，属于遥感成像与计算成像领域，尽管提到了‘幻觉’问题，但其核心任务与文档图像理解（DIU）无关。所提出的方法针对的是特定物理成像模型下的逆问题求解，难以迁移到文档图像中的OCR幻觉、布局分析或VQA等任务。不属于DIU本题，也无明确可迁移的方法论工具。
Abstract Hyperspectral image reconstruction from a compressed measurement is a highly ill-posed inverse problem. Current data-driven methods suffer from hallucination due to the lack of spectral diversity in existing hyperspectral image datasets, particularly when they are evaluated for the metamerism phenomenon. In this work, we formulate hyperspectral image (HSI) reconstruction as a Bayesian inference problem and propose a framework, HSDiff, that utilizes an unconditionally trained, pixel-level diffusion prior and posterior diffusion sampling to generate diverse HSI samples consistent with the measurements of various hyperspectral image formation models. We propose an enhanced metameric augmentation technique using region-based metameric black and partition-of-union spectral upsampling to expand training with physically valid metameric spectra, strengthening the prior diversity and improving uncertainty calibration. We utilize HSDiff to investigate how the studied forward models shape the posterior distribution and demonstrate that guiding with effective spectral encoding provides calibrated informative uncertainty compared to non-encoded models. Through the lens of the Bayesian framework, HSDiff offers a complete, high-performance method for uncertainty-aware HSI reconstruction. Our results also reiterate the significance of effective spectral encoding in snapshot hyperspectral imaging.
Authors Juan Romero, Qiang Fu, Matteo Ravasi, Wolfgang Heidrich
Categories Computer Vision and Pattern Recognition
0
PriorDrive: Enhancing Online HD Mapping with Unified Vector Priors AAAI 2026
Title CN PriorDrive：通过统一向量先验增强在线高精地图构建
Keywords 高精地图 · 自动驾驶 · 向量表示 · 先验信息融合 · nuScenes
Summary 本文提出PriorDrive，通过引入多种向量化的先验地图（如OpenStreetMap、过时的HD地图等），利用统一的向量编码器（UVE）和混合先验表示（HPQuery）来提升自动驾驶中在线高精地图的构建精度与鲁棒性。方法依赖于3D场景几何与道路结构，专注于感知补全与矢量地图融合。
Reason 该论文研究的是自动驾驶中的高精地图在线构建问题，属于3D视觉与具身智能领域，核心任务为基于车载传感器和先验地图数据进行矢量化地图生成。尽管使用了向量表示与编码机制，但其方法针对道路拓扑与空间布局，与文档图像理解（DIU）中的文本结构、排版分析或视觉语言推理无直接关联。且未涉及OCR、多模态语言模型在文档上的应用，也无法迁移用于解决OCR幻觉、密集文本识别或复杂表格理解等DIU核心痛点。
Abstract High-Definition Maps (HD maps) are essential for the precise navigation and decision-making of autonomous vehicles, yet their creation and upkeep present significant cost and timeliness challenges. The online construction of HD maps using on-board sensors has emerged as a promising solution; however, these methods can be impeded by incomplete data due to occlusions and inclement weather, while their performance in distant regions remains unsatisfying. This paper proposes PriorDrive to address these limitations by directly harnessing the power of various vectorized prior maps, significantly enhancing the robustness and accuracy of online HD map construction. Our approach integrates a variety of prior maps uniformly, such as OpenStreetMap's Standard Definition Maps (SD maps), outdated HD maps from vendors, and locally constructed maps from historical vehicle data. To effectively integrate such prior information into online mapping models, we introduce a Hybrid Prior Representation (HPQuery) that standardizes the representation of diverse map elements. We further propose a Unified Vector Encoder (UVE), which employs fused prior embedding and a dual encoding mechanism to encode vector data. To improve the UVE's generalizability and performance, we propose a segment-level and point-level pre-training strategy that enables the UVE to learn the prior distribution of vector data. Through extensive testing on the nuScenes, Argoverse 2 and OpenLane-V2, we demonstrate that PriorDrive is highly compatible with various online mapping models and substantially improves map prediction capabilities. The integration of prior maps through PriorDrive offers a robust solution to the challenges of single-perception data, paving the way for more reliable autonomous vehicle navigation. Code is available at https://github.com/MIV-XJTU/PriorDrive.
Authors Shuang Zeng, Xinyuan Chang, Xinran Liu, Yujian Yuan, Shiyi Liang, Zheng Pan, Mu Xu, Xing Wei
Comment AAAI 2026; Code: https://github.com/MIV-XJTU/PriorDrive
Categories Computer Vision and Pattern Recognition
0
Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span NeurIPS 2025 Spotlight
Title CN 超越画面凝视：预测以自我为中心的3D视觉范围
Keywords 3D视觉范围预测 · 自我中心视觉 · SLAM · Transformer · AR/VR
Summary 本文提出EgoSpanLift方法，将自我中心视角下的视觉范围预测从2D图像平面提升到3D场景中，利用SLAM关键点构建可感知的几何结构，并结合3D U-Net与单向Transformer实现时空融合，预测未来在3D空间中的视觉关注区域。同时构建了一个包含36.46万个样本的大规模基准数据集。实验表明该方法在3D定位和2D凝视预测上均优于现有基线。
Reason 该论文研究的是以自我为中心的3D视觉范围预测，属于具身智能与AR/VR场景下的用户行为预测任务，核心关注点是人类在三维环境中的视觉注意力演化。其方法和数据均面向动态3D场景下的gaze forecasting，与文档图像理解（DIU）中的文本识别、布局分析、表格理解等任务无直接关联。无法迁移用于解决OCR幻觉、密集文本处理或复杂排版理解等DIU核心痛点。
Abstract People continuously perceive and interact with their surroundings based on underlying intentions that drive their exploration and behaviors. While research in egocentric user and scene understanding has focused primarily on motion and contact-based interaction, forecasting human visual perception itself remains less explored despite its fundamental role in guiding human actions and its implications for AR/VR and assistive technologies. We address the challenge of egocentric 3D visual span forecasting, predicting where a person's visual perception will focus next within their three-dimensional environment. To this end, we propose EgoSpanLift, a novel method that transforms egocentric visual span forecasting from 2D image planes to 3D scenes. EgoSpanLift converts SLAM-derived keypoints into gaze-compatible geometry and extracts volumetric visual span regions. We further combine EgoSpanLift with 3D U-Net and unidirectional transformers, enabling spatio-temporal fusion to efficiently predict future visual span in the 3D grid. In addition, we curate a comprehensive benchmark from raw egocentric multisensory data, creating a testbed with 364.6K samples for 3D visual span forecasting. Our approach outperforms competitive baselines for egocentric 2D gaze anticipation and 3D localization while achieving comparable results even when projected back onto 2D image planes without additional 2D-specific training.
Authors Heeseung Yun, Joonil Na, Jaeyeon Kim, Calvin Murdock, Gunhee Kim
Comment NeurIPS 2025 Spotlight
Categories Computer Vision and Pattern Recognition
0
FreeInv: Free Lunch for Improving DDIM Inversion
Title CN FreeInv：改进DDIM反演的免费午餐
Keywords DDIM反演 · 图像编辑 · 视频编辑 · 轨迹偏差 · 隐空间变换
Summary 本文提出FreeInv，一种高效且低成本的方法来缓解DDIM反演过程中的轨迹偏移问题。通过在反演和重建过程中保持随机隐变量变换的一致性，实现多轨迹集成效果，从而降低期望误差。该方法无需额外训练，可即插即用到现有基于反演的图像和视频编辑技术中，在PIE和DAVIS数据集上验证了其优越性能与计算效率。
Reason 该论文聚焦于图像和视频编辑中的DDIM反演过程优化，属于纯图像生成与编辑领域，核心问题是生成轨迹偏差修正，与文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等无直接关联。尽管其方法涉及潜在空间操作，但目标是提升生成保真度与效率，而非解决OCR幻觉、复杂排版理解或长文档推理等DIU痛点，无法迁移为DIU的底层技术工具。
Abstract Naive DDIM inversion process usually suffers from a trajectory deviation issue, i.e., the latent trajectory during reconstruction deviates from the one during inversion. To alleviate this issue, previous methods either learn to mitigate the deviation or design cumbersome compensation strategy to reduce the mismatch error, exhibiting substantial time and computation cost. In this work, we present a nearly free-lunch method (named FreeInv) to address the issue more effectively and efficiently. In FreeInv, we randomly transform the latent representation and keep the transformation the same between the corresponding inversion and reconstruction time-step. It is motivated from a statistical perspective that an ensemble of DDIM inversion processes for multiple trajectories yields a smaller trajectory mismatch error on expectation. Moreover, through theoretical analysis and empirical study, we show that FreeInv performs an efficient ensemble of multiple trajectories. FreeInv can be freely integrated into existing inversion-based image and video editing techniques. Especially for inverting video sequences, it brings more significant fidelity and efficiency improvements. Comprehensive quantitative and qualitative evaluation on PIE benchmark and DAVIS dataset shows that FreeInv remarkably outperforms conventional DDIM inversion, and is competitive among previous state-of-the-art inversion methods, with superior computation efficiency.
Authors Yuxiang Bao, Huijie Liu, Xun Gao, Huan Fu, Guoliang Kang
Categories Computer Vision and Pattern Recognition
0
Advancing Autonomous Driving: DepthSense with Radar and Spatial Attention
Title CN 推进自动驾驶：基于雷达和空间注意力的DepthSense
Keywords 深度估计 · 雷达融合 · 空间注意力 · 自动驾驶 · 单目视觉
Summary 本文提出DepthSense，一种结合雷达信号与单目摄像头数据的深度感知方法，采用编码器-解码器结构、雷达残差网络、空间注意力机制和序数回归层，在nuScenes数据集上实现了更精确且高效的深度估计。主要贡献在于传感器融合与模型效率提升，应用于自动驾驶场景。
Reason 该论文属于自动驾驶领域的感知任务，研究内容为雷达与单目相机融合的深度估计，完全偏离文档图像理解（DIU）的核心问题。不涉及文本识别、布局分析、表格理解或VLM在文档上的应用，属于无关领域（3D视觉与自动驾驶），命中负面清单第2条。
Abstract Depth perception is crucial for spatial understanding and has traditionally been achieved through stereoscopic imaging. However, the precision of depth estimation using stereoscopic methods depends on the accurate calibration of binocular vision sensors. Monocular cameras, while more accessible, often suffer from reduced accuracy, especially under challenging imaging conditions. Optical sensors, too, face limitations in adverse environments, leading researchers to explore radar technology as a reliable alternative. Although radar provides coarse but accurate signals, its integration with fine-grained monocular camera data remains underexplored. In this research, we propose DepthSense, a novel radar-assisted monocular depth enhancement approach. DepthSense employs an encoder-decoder architecture, a Radar Residual Network, feature fusion with a spatial attention mechanism, and an ordinal regression layer to deliver precise depth estimations. We conducted extensive experiments on the nuScenes dataset to validate the effectiveness of DepthSense. Our methodology not only surpasses existing approaches in quantitative performance but also reduces parameter complexity and inference times. Our findings demonstrate that DepthSense represents a significant advancement over traditional stereo methods, offering a robust and efficient solution for depth estimation in autonomous driving. By leveraging the complementary strengths of radar and monocular camera data, DepthSense sets a new benchmark in the field, paving the way for more reliable and accurate spatial perception systems.
Authors Muhamamd Ishfaq Hussain, Zubia Naz, Muhammad Aasim Rafique, Moongu Jeon
Categories Computer Vision and Pattern Recognition
0
Radiation-Preserving Selective Imaging for Pediatric Hip Dysplasia: A Cross-Modal Ultrasound-Xray Policy with Limited Labels AAAI 2026 AIMedHealth Bridge Program
Title CN 用于儿童髋关节发育不良的辐射保护型选择性成像：一种基于有限标签的跨模态超声-X光策略
Keywords 医学影像 · 跨模态学习 · 选择性成像 · 辐射保护 · 临床决策
Summary 本文提出一种以超声优先的诊疗策略，通过预训练模态特定编码器，在有限标注数据下实现对髋关节发育不良关键指标的预测，并设计了一种具有统计保证的延迟决策规则来决定是否需要补充X光检查。该方法在保持高覆盖率的同时显著减少了辐射暴露，适用于临床实践。
Reason 该论文属于医学影像分析领域，研究的是临床诊疗流程中的决策策略优化，核心任务为跨模态（超声与X光）的测量预测与成像决策，并非文档图像理解（DIU）或文档视觉问答（DocVQA）相关任务。尽管使用了视觉编码器和弱监督学习方法，但其目标是减少儿童辐射暴露，而非解决OCR幻觉、复杂排版、密集文本理解等DIU核心问题。技术路线聚焦于医学测量保真度和可解释决策规则，无法迁移到文档理解场景。
Abstract We study an ultrasound-first, radiation-preserving policy for developmental dysplasia of the hip (DDH) that requests a radiograph only when needed. We (i) pretrain modality-specific encoders (ResNet-18) with SimSiam on a large unlabelled registry (37186 ultrasound; 19546 radiographs), (ii) freeze the backbones and fit small, measurement-faithful heads on DDH relevant landmarks and measurements (iii) calibrate a one sided conformal deferral rule on ultrasound predictions that provides finite sample coverage guarantees under exchangeability, using a held-out calibration set. Ultrasound heads predict Graf alpha, beta, and femoral head coverage; X-ray heads predict acetabular index (AI), center-edge (CE) angle and IHDI grade. On our held out labeled evaluation set, ultrasound measurement error is modest (e.g., alpha MAE ~= 9.7 degrees, coverage MAE ~= 14.0%), while radiographic probes achieve AI and CE MAEs of ~= 7.6 degrees and ~= 8.9 degrees, respectively. The calibrated US-only policy is explored across rule families (alpha-only; alpha OR coverage; alpha AND coverage), uncertainty inflation factors, and per-utility trade-offs using decision-curve analysis. Conservative settings yield high coverage with near-zero US-only rates; permissive settings (e.g., alpha OR coverage at larger deltas) achieve non-zero US-only throughput with expected coverage tradeoffs. The result is a simple, reproducible pipeline that turns limited labels into interpretable measurements and tunable selective imaging curves suitable for clinical handoff and future external validation.
Authors Duncan Stothers, Ben Stothers, Emily Schaeffer, Kishore Mulpuri
Comment Accepted (with oral presentation) to the AAAI 2026 AIMedHealth Bridge Program
Categories Computer Vision and Pattern Recognition
0
EventBench: Towards Comprehensive Benchmarking of Event-based MLLMs
Title CN EventBench：面向基于事件的多模态大语言模型的综合基准测试
Keywords 事件相机 · 多模态大模型 · 基准测试 · 3D空间推理 · 事件流数据
Summary 本文提出了EventBench，一个用于评估基于事件的多模态大语言模型（MLLMs）的综合性基准，包含八个任务指标和大规模事件流数据集，涵盖理解、识别和三维空间推理等任务，并对主流闭源和开源模型进行了评测。结果显示现有模型在细粒度识别和空间推理方面仍存在挑战。
Reason 该论文聚焦于基于事件（event-based）视觉的多模态大模型评估，属于3D视觉与视频理解的延伸领域，涉及的是脉冲式事件流数据而非文档图像。其任务设定（如3D空间推理、事件流识别）与文档图像理解的核心痛点（如OCR幻觉、复杂版面分析、长文本逻辑推理）无直接关联，且所用数据模态完全不同，无法迁移至DIU场景。
Abstract Multimodal large language models (MLLMs) have made significant advancements in event-based vision, yet the comprehensive evaluation of their capabilities within a unified benchmark remains largely unexplored. In this work, we introduce EventBench, a benchmark that offers eight diverse task metrics together with a large-scale event stream dataset. EventBench differs from existing event-based benchmarks in four key aspects: (1) openness in accessibility, releasing all raw event streams and task instructions across eight evaluation metrics; (2) diversity in task coverage, spanning understanding, recognition, and spatial reasoning tasks for comprehensive capability assessment; (3) integration in spatial dimensions, pioneering the design of 3D spatial reasoning tasks for event-based MLLMs; and (4) scale in data volume, with an accompanying training set of over one million event-text pairs supporting large-scale training and evaluation. Using EventBench, we evaluate state-of-the-art closed-source models such as GPT-5 and Gemini-2.5 Pro, leading open-source models including Qwen2.5-VL and InternVL3, and event-based MLLMs such as EventGPT that directly process raw event streams. Extensive evaluation reveals that while current event-based MLLMs demonstrate strong performance in event stream understanding, they continue to struggle with fine-grained recognition and spatial reasoning.
Authors Shaoyu Liu, Jianing Li, Guanghui Zhao, Yunjian Zhang, Xiangyang Ji
Categories Computer Vision and Pattern Recognition
0
ReCoGS: Real-time ReColoring for Gaussian Splatting scenes
Title CN ReCoGS：用于高斯点阵场景的实时重新着色
Keywords 高斯点阵 · 3D编辑 · 重新着色 · 实时渲染 · 多视图一致性
Summary 本文提出了一种针对高斯点阵场景的实时重新着色方法ReCoGS，允许用户在预训练的3D场景中精确选择并修改颜色区域，并通过交互式工具实现即时预览。该方法结合了2D扩散模型生成与多视图一致性优化，提升了编辑的精细度和效率。
Reason 该论文研究的是3D场景中的高斯点阵（Gaussian Splatting）重着色编辑，属于3D视觉与图形学领域，主要面向新视角合成与场景编辑。其技术路线与文档图像理解（DIU）的核心痛点如OCR幻觉、文本识别、复杂版面分析或VLM多模态对齐无关。尽管涉及视觉内容编辑，但应用场景和方法均无法迁移到文档理解任务中。
Abstract Gaussian Splatting has emerged as a leading method for novel view synthesis, offering superior training efficiency and real-time inference compared to NeRF approaches, while still delivering high-quality reconstructions. Beyond view synthesis, this 3D representation has also been explored for editing tasks. Many existing methods leverage 2D diffusion models to generate multi-view datasets for training, but they often suffer from limitations such as view inconsistencies, lack of fine-grained control, and high computational demand. In this work, we focus specifically on the editing task of recoloring. We introduce a user-friendly pipeline that enables precise selection and recoloring of regions within a pre-trained Gaussian Splatting scene. To demonstrate the real-time performance of our method, we also present an interactive tool that allows users to experiment with the pipeline in practice. Code is available at https://github.com/loryruta/recogs.
Authors Lorenzo Rutayisire, Nicola Capodieci, Fabio Pellacini
Comment Project page is available at https://github.com/loryruta/recogs
Categories Computer Vision and Pattern Recognition
0
When Generative Replay Meets Evolving Deepfakes: Domain-Aware Relative Weighting for Incremental Face Forgery Detection
Title CN 当生成回放遇到不断演变的深度伪造：用于增量式人脸伪造检测的领域感知相对加权
Keywords 深度伪造检测 · 增量学习 · 生成回放 · 领域混淆 · 相对分离损失
Summary 本文探讨了在增量式人脸伪造检测中使用生成回放（Generative Replay）的挑战，指出当生成器与新伪造技术相似或差异较大时会分别产生‘领域风险’和‘领域安全’样本。为此提出领域感知相对加权（DARW）策略，通过动态调整损失权重来平衡监督与混淆。实验表明该方法能有效提升增量检测性能。
Reason 该论文研究的是增量式人脸伪造检测中的生成回放方法，属于深度伪造（Deepfakes）和生物特征安全领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心问题为模型在持续学习中新伪造技术时的领域混淆，提出的方法DARW针对的是人脸生成域的样本权重调整，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract The rapid advancement of face generation techniques has led to a growing variety of forgery methods. Incremental forgery detection aims to gradually update existing models with new forgery data, yet current sample replay-based methods are limited by low diversity and privacy concerns. Generative replay offers a potential solution by synthesizing past data, but its feasibility for forgery detection remains unclear. In this work, we systematically investigate generative replay and identify two scenarios: when the replay generator closely resembles the new forgery model, generated real samples blur the domain boundary, creating domain-risky samples; when the replay generator differs significantly, generated samples can be safely supervised, forming domain-safe samples. To exploit generative replay effectively, we propose a novel Domain-Aware Relative Weighting (DARW) strategy. DARW directly supervises domain-safe samples while applying a Relative Separation Loss to balance supervision and potential confusion for domain-risky samples. A Domain Confusion Score dynamically adjusts this tradeoff according to sample reliability. Extensive experiments demonstrate that DARW consistently improves incremental learning performance for forgery detection under different generative replay settings and alleviates the adverse impact of domain overlap.
Authors Hao Shen, Jikang Cheng, Renye Yan, Zhongyuan Wang, Wei Peng, Baojin Huang
Categories Computer Vision and Pattern Recognition
0
Scene Summarization: Clustering Scene Videos into Spatially Diverse Frames
Title CN 场景摘要：将场景视频聚类为具有空间多样性的帧
Keywords 视频摘要 · 空间多样性 · 关键帧选择 · 自监督学习 · 视觉地点识别
Summary 本文提出了“场景摘要”这一新任务，旨在将长而连续的场景视频压缩为一组具有空间多样性的关键帧，以促进全局空间推理。作者设计了SceneSum，一个两阶段的自监督流程：首先利用视觉地点识别对视频帧进行聚类以增强空间多样性，然后在资源约束下从每个簇中选择代表性帧。若存在相机轨迹信息，则引入轻量级监督损失进一步优化。实验表明该方法在真实和模拟室内数据集上优于现有视频摘要基线。
Reason 该论文研究的是场景视频的摘要与关键帧提取，属于视频理解领域，关注的是空间布局的抽象与导航理解。其任务和方法均与文档图像理解（DIU）的核心问题如文本识别、排版分析、表格理解或DocVQA无关。尽管涉及‘视觉聚类’和‘代表性帧选择’，但这些技术难以直接迁移至解决OCR幻觉、密集文本处理或复杂文档结构解析等DIU痛点。
Abstract Humans are remarkably efficient at forming spatial understanding from just a few visual observations. When browsing real estate or navigating unfamiliar spaces, they intuitively select a small set of views that summarize the spatial layout. Inspired by this ability, we introduce scene summarization, the task of condensing long, continuous scene videos into a compact set of spatially diverse keyframes that facilitate global spatial reasoning. Unlike conventional video summarization-which focuses on user-edited, fragmented clips and often ignores spatial continuity-our goal is to mimic how humans abstract spatial layout from sparse views. We propose SceneSum, a two-stage self-supervised pipeline that first clusters video frames using visual place recognition to promote spatial diversity, then selects representative keyframes from each cluster under resource constraints. When camera trajectories are available, a lightweight supervised loss further refines clustering and selection. Experiments on real and simulated indoor datasets show that SceneSum produces more spatially informative summaries and outperforms existing video summarization baselines.
Authors Chao Chen, Mingzhi Zhu, Ankush Pratap Singh, Yu Yan, Felix Juefei-Xu, Chen Feng
Categories Computer Vision and Pattern Recognition
0
LungX: A Hybrid EfficientNet-Vision Transformer Architecture with Multi-Scale Attention for Accurate Pneumonia Detection
Title CN LungX：一种结合EfficientNet与视觉Transformer的多尺度注意力混合架构，用于精确肺炎检测
Keywords 肺炎检测 · EfficientNet · Vision Transformer · 注意力机制 · 医学图像分类
Summary 本文提出了一种名为LungX的混合深度学习模型，结合了EfficientNet的多尺度特征提取能力、CBAM注意力机制以及Vision Transformer的全局上下文建模能力，用于提升肺炎的诊断准确率。在RSNA和CheXpert数据集上的实验表明，该模型在准确率和AUC指标上均优于基线模型，并通过可视化注意力图展示了良好的病灶定位能力。
Reason 该论文属于医学图像分析领域，研究任务为肺炎检测，使用胸部X光图像，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。尽管其采用了CBAM和Vision Transformer等视觉建模技术，但整体方法针对的是生物医学影像分类问题，无法直接迁移解决文档图像中的OCR幻觉、复杂排版、细粒度文本定位或长文档推理等核心痛点。
Abstract Pneumonia remains a leading global cause of mortality where timely diagnosis is critical. We introduce LungX, a novel hybrid architecture combining EfficientNet's multi-scale features, CBAM attention mechanisms, and Vision Transformer's global context modeling for enhanced pneumonia detection. Evaluated on 20,000 curated chest X-rays from RSNA and CheXpert, LungX achieves state-of-the-art performance (86.5 percent accuracy, 0.943 AUC), representing a 6.7 percent AUC improvement over EfficientNet-B0 baselines. Visual analysis demonstrates superior lesion localization through interpretable attention maps. Future directions include multi-center validation and architectural optimizations targeting 88 percent accuracy for clinical deployment as an AI diagnostic aid.
Authors Mansur Yerzhanuly
Comment 13 pages, 3 figures, 1 table
Categories Computer Vision and Pattern Recognition
0
CrossJEPA: Cross-Modal Joint-Embedding Predictive Architecture for Efficient 3D Representation Learning from 2D Images
Title CN CrossJEPA：用于从2D图像高效学习3D表示的跨模态联合嵌入预测架构
Keywords 3D表示学习 · 跨模态学习 · 自监督学习 · JEPA · 知识蒸馏
Summary 本文提出CrossJEPA，一种用于从2D图像中学习3D表示的跨模态联合嵌入预测架构。该方法利用图像基础模型的知识，训练一个预测器从3D点云推断特定渲染视图的嵌入，并引入非掩码式的JEPA预训练策略。通过冻结教师模型和一次性缓存目标嵌入，实现高效的训练。在ModelNet40和ScanObjectNN上取得了线性探针的SOTA性能，且参数量小、训练速度快。
Reason 该论文研究的是基于2D图像进行3D表示学习，属于3D视觉与跨模态表示学习领域，与文档图像理解（DIU）任务无直接关联。其核心目标是解决3D点云数据稀缺问题，方法聚焦于图像到点云的跨模态预测，无法迁移用于解决文档中的OCR幻觉、复杂排版、细粒度文本识别或长文档推理等核心痛点。
Abstract Image-to-point cross-modal learning has emerged to address the scarcity of large-scale 3D datasets in 3D representation learning. However, current methods that leverage 2D data often result in large, slow-to-train models, making them computationally expensive and difficult to deploy in resource-constrained environments. The architecture design of such models is therefore critical, determining their performance, memory footprint, and compute efficiency. The Joint-embedding Predictive Architecture (JEPA) has gained wide popularity in self-supervised learning for its simplicity and efficiency, but has been under-explored in cross-modal settings, partly due to the misconception that masking is intrinsic to JEPA. In this light, we propose CrossJEPA, a simple Cross-modal Joint Embedding Predictive Architecture that harnesses the knowledge of an image foundation model and trains a predictor to infer embeddings of specific rendered 2D views from corresponding 3D point clouds, thereby introducing a JEPA-style pretraining strategy beyond masking. By conditioning the predictor on cross-domain projection information, CrossJEPA purifies the supervision signal from semantics exclusive to the target domain. We further exploit the frozen teacher design with a one-time target embedding caching mechanism, yielding amortized efficiency. CrossJEPA achieves a new state-of-the-art in linear probing on the synthetic ModelNet40 (94.2%) and the real-world ScanObjectNN (88.3%) benchmarks, using only 14.1M pretraining parameters (8.5M in the point encoder), and about 6 pretraining hours on a standard single GPU. These results position CrossJEPA as a performant, memory-efficient, and fast-to-train framework for 3D representation learning via knowledge distillation. We analyze CrossJEPA intuitively, theoretically, and empirically, and extensively ablate our design choices. Code will be made available.
Authors Avishka Perera, Kumal Hewagamage, Saeedha Nazar, Kavishka Abeywardana, Hasitha Gallella, Ranga Rodrigo, Mohamed Afham
Comment 24 pages, 10 figures
Categories Computer Vision and Pattern Recognition
0
NeuroVascU-Net: A Unified Multi-Scale and Cross-Domain Adaptive Feature Fusion U-Net for Precise 3D Segmentation of Brain Vessels in Contrast-Enhanced T1 MRI
Title CN NeuroVascU-Net：一种用于增强T1 MRI中脑血管精确3D分割的统一多尺度与跨域自适应特征融合U-Net
Keywords 医学图像分割 · U-Net · 多尺度特征融合 · 脑血管分割 · MRI
Summary 本文提出了一种名为NeuroVascU-Net的深度学习模型，用于从临床标准的T1加权增强MRI图像中进行脑血管的3D分割。模型基于膨胀U-Net架构，引入了多尺度上下文特征融合模块和跨域自适应特征融合模块，以提升对细小血管结构的识别能力。在137例脑肿瘤患者的T1CE数据上验证，取得了较高的Dice分数，同时参数量仅12.4M，具备良好的效率优势。
Reason 该论文属于医学图像分析领域，研究任务为脑血管的3D分割，完全偏离文档图像理解（DIU）核心问题。尽管其网络结构设计具有一定创新性，但应用场景与OCR、文档布局分析、表格识别等无关，且未涉及多模态大模型（VLM）、视觉语言推理或文档特有的密集文本、复杂排版等挑战，无法迁移至DIU领域。
Abstract Precise 3D segmentation of cerebral vasculature from T1-weighted contrast-enhanced (T1CE) MRI is crucial for safe neurosurgical planning. Manual delineation is time-consuming and prone to inter-observer variability, while current automated methods often trade accuracy for computational cost, limiting clinical use. We present NeuroVascU-Net, the first deep learning architecture specifically designed to segment cerebrovascular structures directly from clinically standard T1CE MRI in neuro-oncology patients, addressing a gap in prior work dominated by TOF-MRA-based approaches. NeuroVascU-Net builds on a dilated U-Net and integrates two specialized modules: a Multi-Scale Contextual Feature Fusion ($MSC^2F$) module at the bottleneck and a Cross-Domain Adaptive Feature Fusion ($CDA^2F$) module at deeper hierarchical layers. $MSC^2F$ captures both local and global information via multi-scale dilated convolutions, while $CDA^2F$ dynamically integrates domain-specific features, enhancing representation while keeping computation low. The model was trained and validated on a curated dataset of T1CE scans from 137 brain tumor biopsy patients, annotated by a board-certified functional neurosurgeon. NeuroVascU-Net achieved a Dice score of 0.8609 and precision of 0.8841, accurately segmenting both major and fine vascular structures. Notably, it requires only 12.4M parameters, significantly fewer than transformer-based models such as Swin U-NetR. This balance of accuracy and efficiency positions NeuroVascU-Net as a practical solution for computer-assisted neurosurgical planning.
Authors Mohammad Jafari Vayeghan, Niloufar Delfan, Mehdi Tale Masouleh, Mansour Parvaresh Rizi, Behzad Moshiri
Categories Computer Vision and Pattern Recognition
0
End-to-End Visual Autonomous Parking via Control-Aided Attention
Title CN 通过控制辅助注意力实现端到端视觉自主泊车
Keywords 自主泊车 · 端到端学习 · 注意力机制 · 模仿学习 · CARLA
Summary 本文提出CAA-Policy，一种用于端到端视觉自主泊车的模仿学习系统，通过引入控制信号来引导视觉注意力（Control-Aided Attention），使模型关注对控制输出影响更大的视觉区域。该注意力模块通过控制输出的反向梯度进行自监督训练，而非仅依赖损失函数优化。此外，框架还结合了短时距路径点预测、可学习的运动预测模块和改进的目标标记化方案，提升了控制的时序一致性和鲁棒性。实验表明该方法在CARLA模拟器中优于基线方法。
Reason 该论文属于自动驾驶领域，研究的是端到端视觉感知与控制协同的泊车系统，核心方法围绕驾驶策略中的注意力机制设计，与文档图像理解（DIU）或视觉语言模型在文档任务中的应用无直接关联。尽管其提出的Control-Aided Attention机制具有一定的跨模态引导思想，但迁移到文档理解场景（如OCR去幻觉、复杂排版解析等）极为困难，且不属于DIU本题或可迁移的上游‘军火库’方法。
Abstract Precise parking requires an end-to-end system where perception adaptively provides policy-relevant details - especially in critical areas where fine control decisions are essential. End-to-end learning offers a unified framework by directly mapping sensor inputs to control actions, but existing approaches lack effective synergy between perception and control. Instead, we propose CAA-Policy, an end-to-end imitation learning system that allows control signal to guide the learning of visual attention via a novel Control-Aided Attention (CAA) mechanism. We train such an attention module in a self-supervised manner, using backpropagated gradients from the control outputs instead of from the training loss. This strategy encourages attention to focus on visual features that induce high variance in action outputs, rather than merely minimizing the training loss - a shift we demonstrate leads to a more robust and generalizable policy. To further strengthen the framework, CAA-Policy incorporates short-horizon waypoint prediction as an auxiliary task to improve temporal consistency of control outputs, a learnable motion prediction module to robustly track target slots over time, and a modified target tokenization scheme for more effective feature fusion. Extensive experiments in the CARLA simulator show that CAA-Policy consistently surpasses both the end-to-end learning baseline and the modular BEV segmentation + hybrid A* pipeline, achieving superior accuracy, robustness, and interpretability. Code and Collected Training datasets will be released. Code is released at https://github.com/ai4ce/CAAPolicy.
Authors Chao Chen, Shunyu Yao, Yuanwu He, Feng Tao, Ruojing Song, Yuliang Guo, Xinyu Huang, Chenxu Wu, Liu Ren, Chen Feng
Categories Computer Vision and Pattern Recognition
0
4D-VGGT: A General Foundation Model with SpatioTemporal Awareness for Dynamic Scene Geometry Estimation
Title CN 4D-VGGT：一种具有时空感知能力的动态场景几何估计通用基础模型
Keywords 动态场景几何估计 · 时空表示 · 多视角融合 · 多时间步输入 · 基础模型
Summary 本文提出4D-VGGT，一种用于动态场景几何估计的通用基础模型，通过分治策略分别建模空间与时间特征，采用自适应视觉网格支持任意数量的视图和时间步，并结合多级表示与多任务头提升模型泛化性。实验表明其在多个动态场景几何任务上表现优异。
Reason 该论文研究的是动态场景的几何估计，属于3D视觉与视频理解领域，核心任务为建模空间与时间特征以估计动态场景结构。其方法聚焦于多视角、多时序输入下的场景重建，与文档图像理解（DIU）在任务目标、数据模态和应用场景上均无交集。尽管涉及视觉表示学习，但其‘跨视角全局融合’与‘跨时间局部融合’的设计针对的是动态3D场景，难以迁移到文档图像的细粒度文本识别、布局分析或VQA等任务。属于明确的无关领域。
Abstract We investigate a challenging task of dynamic scene geometry estimation, which requires representing both spatial and temporal features. Typically, existing methods align the two features into a unified latent space to model scene geometry. However, this unified paradigm suffers from potential mismatched representation due to the heterogeneous nature between spatial and temporal features. In this work, we propose 4D-VGGT, a general foundation model with divide-and-conquer spatiotemporal representation for dynamic scene geometry. Our model is divided into three aspects: 1) Multi-setting input. We design an adaptive visual grid that supports input sequences with arbitrary numbers of views and time steps. 2) Multi-level representation. We propose a cross-view global fusion for spatial representation and a cross-time local fusion for temporal representation. 3) Multi-task prediction. We append multiple task-specific heads to spatiotemporal representations, enabling a comprehensive visual geometry estimation for dynamic scenes. Under this unified framework, these components enhance the feature discriminability and application universality of our model for dynamic scenes. In addition, we integrate multiple geometry datasets to train our model and conduct extensive experiments to verify the effectiveness of our method across various tasks on multiple dynamic scene geometry benchmarks.
Authors Haonan Wang, Hanyu Zhou, Haoyue Liu, Luxin Yan
Categories Computer Vision and Pattern Recognition
0
Otter: Mitigating Background Distractions of Wide-Angle Few-Shot Action Recognition with Enhanced RWKV AAAI 2026 Oral
Title CN Otter：利用增强型RWKV缓解广角少样本动作识别中的背景干扰
Keywords 少样本动作识别 · 广角视频 · RWKV · 背景干扰 · 时序重建
Summary 本文提出Otter模型，用于解决广角视频在少样本动作识别中因背景干扰和时序退化导致的性能下降问题。通过引入复合分割模块（CSM）突出帧中的主体区域，并设计时序重建模块（TRM）进行双向扫描以恢复时序关系，结合常规原型与增强原型提升识别效果。在多个视频基准上实现了SOTA性能。
Reason 该论文研究的是视频领域的少样本动作识别问题，核心挑战是广角视频中的背景干扰和时序关系建模，属于视频理解范畴。其方法设计（如关键区域分割和时序重建）针对的是动态视觉场景，与文档图像理解（DIU）中的OCR幻觉、密集文本解析、复杂版式分析等核心痛点无直接关联。尽管RWKV架构可能具有一定的序列建模启发性，但整体工作无法迁移到文档理解任务中。
Abstract Wide-angle videos in few-shot action recognition (FSAR) effectively express actions within specific scenarios. However, without a global understanding of both subjects and background, recognizing actions in such samples remains challenging because of the background distractions. Receptance Weighted Key Value (RWKV), which learns interaction between various dimensions, shows promise for global modeling. While directly applying RWKV to wide-angle FSAR may fail to highlight subjects due to excessive background information. Additionally, temporal relation degraded by frames with similar backgrounds is difficult to reconstruct, further impacting performance. Therefore, we design the CompOund SegmenTation and Temporal REconstructing RWKV (Otter). Specifically, the Compound Segmentation Module~(CSM) is devised to segment and emphasize key patches in each frame, effectively highlighting subjects against background information. The Temporal Reconstruction Module (TRM) is incorporated into the temporal-enhanced prototype construction to enable bidirectional scanning, allowing better reconstruct temporal relation. Furthermore, a regular prototype is combined with the temporal-enhanced prototype to simultaneously enhance subject emphasis and temporal modeling, improving wide-angle FSAR performance. Extensive experiments on benchmarks such as SSv2, Kinetics, UCF101, and HMDB51 demonstrate that Otter achieves state-of-the-art performance. Extra evaluation on the VideoBadminton dataset further validates the superiority of Otter in wide-angle FSAR.
Authors Wenbo Huang, Jinghui Zhang, Zhenghao Chen, Guang Li, Lei Zhang, Yang Cao, Fang Dong, Takahiro Ogawa, Miki Haseyama
Comment Accepted by AAAI 2026 Oral
Categories Computer Vision and Pattern Recognition
0
ChineseVideoBench: Benchmarking Multi-modal Large Models for Chinese Video Question Answering
Title CN ChineseVideoBench：面向中文视频问答的多模态大模型评测基准
Keywords 视频问答 · 多模态大模型 · 评测基准 · 中文理解 · 视频理解
Summary 本文提出了ChineseVideoBench，一个专门用于评估多模态大语言模型在中文视频问答任务上性能的基准。该基准包含8个主类别和12个子类别，强调对中文语言和文化细节的理解。实验表明现有模型在此基准上仍有挑战，Gemini 2.5 Pro表现最佳，InternVL-38B为最强开源模型。
Reason 该论文属于视频理解领域，且聚焦于中文视频问答任务，与文档图像理解（DIU）的核心任务如DocVQA、表格识别、版面分析等无直接关联。尽管其涉及多模态大模型评估，但应用场景和数据模态（视频序列）与文档图像差异显著，无法迁移解决OCR幻觉、密集文本识别、复杂排版或长文档推理等DIU痛点。
Abstract This paper introduces ChineseVideoBench, a pioneering benchmark specifically designed for evaluating Multimodal Large Language Models (MLLMs) in Chinese Video Question Answering. The growing demand for sophisticated video analysis capabilities highlights the critical need for comprehensive, culturally-aware evaluation frameworks. ChineseVideoBench addresses this gap by providing a robust dataset and tailored evaluation metrics, enabling rigorous assessment of state-of-the-art MLLMs on complex Chinese video content. Specifically, ChineseVideoBench comprises 8 main classes and 12 sub-classes, encompassing tasks that demand both deep video understanding and nuanced Chinese linguistic and cultural awareness. Our empirical evaluations reveal that ChineseVideoBench presents a significant challenge to current MLLMs. Among the models assessed, Gemini 2.5 Pro achieves the highest performance with an overall score of 77.9%, while InternVL-38B emerges as the most competitive open-source model.
Authors Yuxiang Nie, Han Wang, Yongjie Ye, Haiyang Yu, Weitao Jia, Tao Zeng, Hao Feng, Xiang Fei, Yang Li, Xiaohui Lv, Guozhi Tang, Jingqun Tang, Jinghui Lu, Zehui Dai, Jiacong Wang, Dingkang Yang, An-Lan Wang, Can Huang
Categories Computer Vision and Pattern Recognition
0
SegSplat: Feed-forward Gaussian Splatting and Open-Set Semantic Segmentation
Title CN SegSplat：前馈式高斯溅射与开放集语义分割
Keywords 3D重建 · 高斯溅射 · 开放集语义分割 · 多视角理解 · 语义内存库
Summary 本文提出SegSplat框架，通过构建紧凑的语义内存库，并在单次前馈过程中为每个3D高斯单元预测语义索引、几何和外观属性，实现快速且富含语义的3D场景重建。无需逐场景优化即可完成语义集成，在保持几何精度的同时支持开放集语义分割。
Reason 该论文属于3D视觉与场景重建领域，核心贡献是将多视角2D特征整合到3D高斯溅射中以实现开放词汇语义分割。其技术路线聚焦于三维环境的语义建模，与文档图像理解（DIU）中的文本识别、布局分析、表格理解等任务无直接关联。尽管涉及多模态特征融合，但方法针对的是3D几何与视觉语义对齐问题，无法迁移至解决OCR幻觉、密集文本处理或复杂排版理解等DIU核心痛点。
Abstract We have introduced SegSplat, a novel framework designed to bridge the gap between rapid, feed-forward 3D reconstruction and rich, open-vocabulary semantic understanding. By constructing a compact semantic memory bank from multi-view 2D foundation model features and predicting discrete semantic indices alongside geometric and appearance attributes for each 3D Gaussian in a single pass, SegSplat efficiently imbues scenes with queryable semantics. Our experiments demonstrate that SegSplat achieves geometric fidelity comparable to state-of-the-art feed-forward 3D Gaussian Splatting methods while simultaneously enabling robust open-set semantic segmentation, crucially \textit{without} requiring any per-scene optimization for semantic feature integration. This work represents a significant step towards practical, on-the-fly generation of semantically aware 3D environments, vital for advancing robotic interaction, augmented reality, and other intelligent systems.
Authors Peter Siegel, Federico Tombari, Marc Pollefeys, Daniel Barath
Categories Computer Vision and Pattern Recognition
0
ViMix-14M: A Curated Multi-Source Video-Text Dataset with Long-Form, High-Quality Captions and Crawl-Free Access
Title CN ViMix-14M：一个具有长文本、高质量字幕和免爬取访问的精选多源视频-文本数据集
Keywords 视频-文本数据集 · 文本到视频生成 · 多模态学习 · 数据构建 · 视频理解
Summary 本文提出了ViMix-14M，一个包含约1400万对视频-文本的数据集，通过整合多个开源视频资源，并采用统一去重、质量过滤以及基于真实标注引导的多粒度重描述流程，生成与视频内容紧密对齐的长文本描述。该数据集支持免爬取、可直接下载使用，旨在缓解当前开源视频基础模型训练中的数据瓶颈问题。实验表明其在多模态检索、文本到视频生成和视频问答任务上优于现有数据集。
Reason 该论文聚焦于视频理解与生成领域，构建了一个大规模视频-文本对数据集，主要用于文本到视频生成、视频问答等任务。虽然其数据构建方法具有一定工程价值，但研究领域属于视频模态，与文档图像理解（DIU）的核心问题如OCR幻觉、复杂版式分析、细粒度文本定位等无直接关联。无法迁移用于解决文档理解中的关键挑战，属于无关领域。
Abstract Text-to-video generation has surged in interest since Sora, yet open-source models still face a data bottleneck: there is no large, high-quality, easily obtainable video-text corpus. Existing public datasets typically require manual YouTube crawling, which yields low usable volume due to link rot and access limits, and raises licensing uncertainty. This work addresses this challenge by introducing ViMix-14M, a curated multi-source video-text dataset of around 14 million pairs that provides crawl-free, download-ready access and long-form, high-quality captions tightly aligned to video. ViMix-14M is built by merging diverse open video sources, followed by unified de-duplication and quality filtering, and a multi-granularity, ground-truth-guided re-captioning pipeline that refines descriptions to better match actions, scenes, and temporal structure. We evaluate the dataset by multimodal retrieval, text-to-video generation, and video question answering tasks, observing consistent improvements over counterpart datasets. We hope this work can help removing the key barrier to training and fine-tuning open-source video foundation models, and provide insights of building high-quality and generalizable video-text datasets.
Authors Timing Yang, Sucheng Ren, Alan Yuille, Feng Wang
Categories Computer Vision and Pattern Recognition
0
Synthetic Curriculum Reinforces Compositional Text-to-Image Generation
Title CN 合成课程强化组合式文本到图像生成
Keywords 文本到图像生成 · 组合生成 · 课程学习 · 强化学习 · 场景图
Summary 本文提出了一种名为CompGen的组合式课程强化学习框架，用于提升文本到图像生成模型在复杂场景下的组合生成能力。通过场景图定义难度标准，并采用自适应MCMC采样生成训练课程数据，结合GRPO算法进行优化。实验表明该方法能有效提升扩散模型和自回归模型在组合生成任务上的表现。
Reason 该论文研究的是文本到图像生成中的组合式场景生成问题，属于纯图像生成领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。其方法聚焦于通过课程学习提升T2I模型对复杂语义场景的生成能力，但该技术路径难以迁移至文档理解中的OCR幻觉、密集文本识别、布局分析等核心痛点。属于无关领域（纯图像生成），不符合关注方向。
Abstract Text-to-Image (T2I) generation has long been an open problem, with compositional synthesis remaining particularly challenging. This task requires accurate rendering of complex scenes containing multiple objects that exhibit diverse attributes as well as intricate spatial and semantic relationships, demanding both precise object placement and coherent inter-object interactions. In this paper, we propose a novel compositional curriculum reinforcement learning framework named CompGen that addresses compositional weakness in existing T2I models. Specifically, we leverage scene graphs to establish a novel difficulty criterion for compositional ability and develop a corresponding adaptive Markov Chain Monte Carlo graph sampling algorithm. This difficulty-aware approach enables the synthesis of training curriculum data that progressively optimize T2I models through reinforcement learning. We integrate our curriculum learning approach into Group Relative Policy Optimization (GRPO) and investigate different curriculum scheduling strategies. Our experiments reveal that CompGen exhibits distinct scaling curves under different curriculum scheduling strategies, with easy-to-hard and Gaussian sampling strategies yielding superior scaling performance compared to random sampling. Extensive experiments demonstrate that CompGen significantly enhances compositional generation capabilities for both diffusion-based and auto-regressive T2I models, highlighting its effectiveness in improving the compositional T2I generation systems.
Authors Shijian Wang, Runhao Fu, Siyi Zhao, Qingqin Zhan, Xingjian Wang, Jiarui Jin, Yuan Lu, Hanqian Wu, Cunjian Chen
Categories Computer Vision and Pattern Recognition
0
MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Models
Title CN MASS：用于视觉语言模型中物理推理与理解的运动感知时空定位
Keywords 视觉语言模型 · 物理推理 · 时空定位 · 视频理解 · 运动追踪
Summary 本文提出MASS方法和MASS-Bench基准，旨在提升视觉语言模型在包含物理运动的视频中的推理能力。通过引入基于深度的3D编码、视觉定位和运动追踪技术，并结合强化学习微调，增强VLM对物体动态的感知与跨模态对齐。实验表明该方法在物理理解任务上显著优于现有模型。
Reason 该论文聚焦于视频中的物理推理与运动理解，属于视频理解领域，核心任务为物理驱动的动态场景建模与时空定位，与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版式分析等无直接关联。尽管其提出的跨模态对齐与强化微调方法具有一定通用性，但其方法设计高度依赖3D运动追踪与深度编码，难以迁移到静态、高分辨率、结构化排版的文档图像场景。属于无关领域（视频理解），命中负面清单第2条。
Abstract Vision Language Models (VLMs) perform well on standard video tasks but struggle with physics-driven reasoning involving motion dynamics and spatial interactions. This limitation reduces their ability to interpret real or AI-generated content (AIGC) videos and to generate physically consistent content. We present an approach that addresses this gap by translating physical-world context cues into interpretable representations aligned with VLMs' perception, comprehension, and reasoning. We introduce MASS-Bench, a comprehensive benchmark consisting of 4,350 real-world and AIGC videos and 8,361 free-form video question-answering pairs focused on physics-related comprehension tasks, with detailed annotations including visual detections, sub-segment grounding, and full-sequence 3D motion tracking of entities. We further present MASS, a model-agnostic method that injects spatial-temporal signals into the VLM language space via depth-based 3D encoding and visual grounding, coupled with a motion tracker for object dynamics. To strengthen cross-modal alignment and reasoning, we apply reinforcement fine-tuning. Experiments and ablations show that our refined VLMs outperform comparable and larger baselines, as well as prior state-of-the-art models, by 8.7% and 6.0%, achieving performance comparable to close-source SoTA VLMs such as Gemini-2.5-Flash on physics reasoning and comprehension. These results validate the effectiveness of our approach.
Authors Xiyang Wu, Zongxia Li, Jihui Jin, Guangyao Shi, Gouthaman KV, Vishnu Raj, Nilotpal Sinha, Jingxi Chen, Fan Du, Dinesh Manocha
Categories Computer Vision and Pattern Recognition
0
MimiCAT: Mimic with Correspondence-Aware Cascade-Transformer for Category-Free 3D Pose Transfer
Title CN MimiCAT：基于对应感知级联变换器的无类别3D姿态迁移
Keywords 3D姿态迁移 · 软对应匹配 · 级联变换器 · 无类别迁移 · 语义关键点
Summary 本文提出MimiCAT模型，用于解决不同类别3D角色之间的姿态迁移问题。通过构建大规模数据集并引入基于语义关键点的软对应机制，实现从人形到四足动物等跨类别的姿态风格迁移。采用级联变换器架构，在无严格一对一对应关系的情况下完成姿态投影与形状条件化优化。
Reason 该论文研究的是3D姿态迁移问题，属于具身智能与3D视觉领域，与文档图像理解（DIU）或视觉语言模型在文档任务中的应用无关。尽管其方法涉及跨结构匹配和软对应学习，但核心场景为角色动画与3D建模，无法迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等痛点。
Abstract 3D pose transfer aims to transfer the pose-style of a source mesh to a target character while preserving both the target's geometry and the source's pose characteristic. Existing methods are largely restricted to characters with similar structures and fail to generalize to category-free settings (e.g., transferring a humanoid's pose to a quadruped). The key challenge lies in the structural and transformation diversity inherent in distinct character types, which often leads to mismatched regions and poor transfer quality. To address these issues, we first construct a million-scale pose dataset across hundreds of distinct characters. We further propose MimiCAT, a cascade-transformer model designed for category-free 3D pose transfer. Instead of relying on strict one-to-one correspondence mappings, MimiCAT leverages semantic keypoint labels to learn a novel soft correspondence that enables flexible many-to-many matching across characters. The pose transfer is then formulated as a conditional generation process, in which the source transformations are first projected onto the target through soft correspondence matching and subsequently refined using shape-conditioned representations. Extensive qualitative and quantitative experiments demonstrate that MimiCAT transfers plausible poses across different characters, significantly outperforming prior methods that are limited to narrow category transfer (e.g., humanoid-to-humanoid).
Authors Zenghao Chai, Chen Tang, Yongkang Wong, Xulei Yang, Mohan Kankanhalli
Comment tech report
Categories Computer Vision and Pattern Recognition
0
Alias-free 4D Gaussian Splatting
Title CN 无伪影的4D高斯点阵化
Keywords 4D Gaussian Splatting · 动态场景重建 · 抗混叠 · 高分辨率渲染 · 尺度自适应
Summary 本文针对基于高斯点阵的动态场景重建在改变相机参数时出现的高频伪影问题，提出了一种4D尺度自适应滤波器和尺度损失函数，通过调节4D高斯点阵的采样频率来消除伪影并减少冗余高斯分布。实验验证了其在单目和多视角视频重建中的有效性。
Reason 该论文属于3D视觉与动态场景重建领域，研究的是4D高斯点阵在相机焦距或视角变化下的渲染伪影问题，核心关注点为高斯尺度自适应滤波与采样频率控制。其技术目标和应用场景与文档图像理解（DIU）完全无关，无法迁移用于解决OCR幻觉、复杂排版解析、细粒度文本定位或长文档推理等DIU核心痛点。
Abstract Existing dynamic scene reconstruction methods based on Gaussian Splatting enable real-time rendering and generate realistic images. However, adjusting the camera's focal length or the distance between Gaussian primitives and the camera to modify rendering resolution often introduces strong artifacts, stemming from the frequency constraints of 4D Gaussians and Gaussian scale mismatch induced by the 2D dilated filter. To address this, we derive a maximum sampling frequency formulation for 4D Gaussian Splatting and introduce a 4D scale-adaptive filter and scale loss, which flexibly regulates the sampling frequency of 4D Gaussian Splatting. Our approach eliminates high-frequency artifacts under increased rendering frequencies while effectively reducing redundant Gaussians in multi-view video reconstruction. We validate the proposed method through monocular and multi-view video reconstruction experiments.Ours project page: https://4d-alias-free.github.io/4D-Alias-free/
Authors Zilong Chen, Huan-ang Gao, Delin Qu, Haohan Chi, Hao Tang, Kai Zhang, Hao Zhao
Comment Project page: https://4d-alias-free.github.io/4D-Alias-free/
Categories Computer Vision and Pattern Recognition
0
HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perception
Title CN HOSIG：基于分层场景感知的全身人类-物体-场景交互生成
Keywords 人-物-景交互 · 动作生成 · 场景感知 · 导航规划 · 扩散模型
Summary 本文提出HOSIG框架，用于生成高保真的全身人类-物体-场景交互。通过分层场景感知机制，将任务分解为场景感知的抓取姿态生成、基于启发式的导航路径规划以及场景引导的动作扩散模型，实现无碰撞、精细操控且长程可控的人体动作合成。实验表明其在TRUMANS数据集上优于现有方法。
Reason 该论文研究的是全身人类与物体、场景的交互生成，属于具身智能与计算机图形学领域，核心任务为动画合成与运动生成，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管其使用了场景感知和空间推理等技术，但这些方法针对的是3D环境中的动作规划与人体运动控制，难以迁移到文档布局分析、表格识别或VQA等DIU核心任务。属于无关领域。
Abstract Generating high-fidelity full-body human interactions with dynamic objects and static scenes remains a critical challenge in computer graphics and animation. Existing methods for human-object interaction often neglect scene context, leading to implausible penetrations, while human-scene interaction approaches struggle to coordinate fine-grained manipulations with long-range navigation. To address these limitations, we propose HOSIG, a novel framework for synthesizing full-body interactions through hierarchical scene perception. Our method decouples the task into three key components: 1) a scene-aware grasp pose generator that ensures collision-free whole-body postures with precise hand-object contact by integrating local geometry constraints, 2) a heuristic navigation algorithm that autonomously plans obstacle-avoiding paths in complex indoor environments via compressed 2D floor maps and dual-component spatial reasoning, and 3) a scene-guided motion diffusion model that generates trajectory-controlled, full-body motions with finger-level accuracy by incorporating spatial anchors and dual-space classifier-free guidance. Extensive experiments on the TRUMANS dataset demonstrate superior performance over state-of-the-art methods. Notably, our framework supports unlimited motion length through autoregressive generation and requires minimal manual intervention. This work bridges the critical gap between scene-aware navigation and dexterous object manipulation, advancing the frontier of embodied interaction synthesis. Codes will be available after publication. Project page: http://yw0208.github.io/hosig
Authors Wei Yao, Yunlian Sun, Hongwen Zhang, Yebin Liu, Jinhui Tang
Categories Computer Vision and Pattern Recognition
0
Enhancing UAV Search under Occlusion using Next Best View Planning IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Title CN 利用最佳下一视角规划增强无人机在遮挡环境下的搜索
Keywords 无人机 · 搜索与救援 · 遮挡环境 · 下一最佳视角 · 可见性启发式
Summary 本文提出了一种用于遮挡环境中无人机搜索的最优视角规划策略，设计了几何启发式和可见性启发式两种新方法来选择最佳相机视角。实验表明，可见性启发式在模拟和真实森林环境中均能实现超过90%的隐藏物体识别率，并比几何启发式提高10%的检测率，有效提升了林下覆盖能力。
Reason 该论文研究的是无人机在森林遮挡环境下的搜索策略，属于具身智能与机器人领域，核心问题是视角规划与可见性优化，与文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的应用无直接关联。其方法难以迁移到解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract Search and rescue missions are often critical following sudden natural disasters or in high-risk environmental situations. The most challenging search and rescue missions involve difficult-to-access terrains, such as dense forests with high occlusion. Deploying unmanned aerial vehicles for exploration can significantly enhance search effectiveness, facilitate access to challenging environments, and reduce search time. However, in dense forests, the effectiveness of unmanned aerial vehicles depends on their ability to capture clear views of the ground, necessitating a robust search strategy to optimize camera positioning and perspective. This work presents an optimized planning strategy and an efficient algorithm for the next best view problem in occluded environments. Two novel optimization heuristics, a geometry heuristic, and a visibility heuristic, are proposed to enhance search performance by selecting optimal camera viewpoints. Comparative evaluations in both simulated and real-world settings reveal that the visibility heuristic achieves greater performance, identifying over 90% of hidden objects in simulated forests and offering 10% better detection rates than the geometry heuristic. Additionally, real-world experiments demonstrate that the visibility heuristic provides better coverage under the canopy, highlighting its potential for improving search and rescue missions in occluded environments.
Authors Sigrid Helene Strand, Thomas Wiedemann, Bram Burczek, Dmitriy Shutin
Comment Submitted to IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Categories Computer Vision and Pattern Recognition
0
MagicWand: A Universal Agent for Generation and Evaluation Aligned with User Preference
Title CN MagicWand：一种与用户偏好对齐的通用生成与评估智能体
Keywords AIGC · 用户偏好对齐 · 生成智能体 · 图像生成 · 视频生成
Summary 本文提出MagicWand，一个基于大规模偏好数据集UniPrefer-100K的通用生成与评估智能体，旨在提升AIGC生成内容与用户偏好的一致性。通过偏好驱动的提示增强、生成和评估机制，在新构建的UniPreferBench基准上验证了其有效性。研究重点在于图像和视频生成的质量与主观偏好对齐，不涉及文档结构、文本语义或布局分析。
Reason 该论文聚焦于AIGC内容生成（图像、视频）中的用户偏好对齐问题，属于纯图像/视频生成领域，并涉及具身型智能体应用，但其核心任务与文档图像理解（DIU）无关。未解决OCR幻觉、复杂排版、细粒度文本识别或长文档推理等DIU核心痛点。方法论上为下游应用级封装（Wrapper），缺乏可迁移到文档理解领域的底层技术创新。
Abstract Recent advances in AIGC (Artificial Intelligence Generated Content) models have enabled significant progress in image and video generation. However, users still struggle to obtain content that aligns with their preferences due to the difficulty of crafting detailed prompts and the lack of mechanisms to retain their preferences. To address these challenges, we construct \textbf{UniPrefer-100K}, a large-scale dataset comprising images, videos, and associated text that describes the styles users tend to prefer. Based on UniPrefer-100K, we propose \textbf{MagicWand}, a universal generation and evaluation agent that enhances prompts based on user preferences, leverages advanced generation models for high-quality content, and applies preference-aligned evaluation and refinement. In addition, we introduce \textbf{UniPreferBench}, the first large-scale benchmark with over 120K annotations for assessing user preference alignment across diverse AIGC tasks. Experiments on UniPreferBench demonstrate that MagicWand consistently generates content and evaluations that are well aligned with user preferences across a wide range of scenarios.
Authors Zitong Xu, Dake Shen, Yaosong Du, Kexiang Hao, Jinghan Huang, Xiande Huang
Categories Computer Vision and Pattern Recognition
0
Spatiotemporal Graph Convolutional Recurrent Neural Network Model for Citywide Air Pollution Forecasting
Title CN 用于城市范围空气污染预测的时空图卷积循环神经网络模型
Keywords 图卷积网络 · 循环神经网络 · 空气污染预测 · 时空预测 · GCN
Summary 本文提出一种融合图卷积网络（GCN）与循环神经网络（RNN）的时空图卷积循环神经网络（Spatiotemporal GCRNN），用于城市范围内的空气污染预测。通过将城市监测站数据建模为图结构，利用GCN捕捉空间依赖性，并结合RNN建模时间动态变化。实验表明该模型在预测精度上优于ConvLSTM和混合GCN方法，且参数更少。
Reason 该论文属于环境科学与时空预测领域，研究任务为城市空气质量预测，使用图神经网络与循环网络结合建模地理空间中的污染扩散。其方法与文档图像理解（DIU）、视觉语言模型（VLM）或多模态文档分析完全无关，无法迁移解决OCR幻觉、复杂排版理解或长文档推理等核心问题。
Abstract Citywide Air Pollution Forecasting tries to precisely predict the air quality multiple hours ahead for the entire city. This topic is challenged since air pollution varies in a spatiotemporal manner and depends on many complicated factors. Our previous research has solved the problem by considering the whole city as an image and leveraged a Convolutional Long Short-Term Memory (ConvLSTM) model to learn the spatiotemporal features. However, an image-based representation may not be ideal as air pollution and other impact factors have natural graph structures. In this research, we argue that a Graph Convolutional Network (GCN) can efficiently represent the spatial features of air quality readings in the whole city. Specially, we extend the ConvLSTM model to a Spatiotemporal Graph Convolutional Recurrent Neural Network (Spatiotemporal GCRNN) model by tightly integrating a GCN architecture into an RNN structure for efficient learning spatiotemporal characteristics of air quality values and their influential factors. Our extensive experiments prove the proposed model has a better performance compare to the state-of-the-art ConvLSTM model for air pollution predicting while the number of parameters is much smaller. Moreover, our approach is also superior to a hybrid GCN-based method in a real-world air pollution dataset.
Authors Van-Duc Le, Tien-Cuong Bui, Sang-Kyun Cha
Comment Updated metadata
Categories Computer Vision and Pattern Recognition
0
FlowPortal: Residual-Corrected Flow for Training-Free Video Relighting and Background Replacement
Title CN FlowPortal：用于免训练视频重打光和背景替换的残差校正光流方法
Keywords 视频重打光 · 背景替换 · 光流 · 免训练 · 时间一致性
Summary 本文提出了一种名为FlowPortal的免训练视频重打光框架，通过引入残差校正光流机制，在保持结构一致性的前提下实现高质量的视频光照调整与背景替换。结合解耦条件设计和高频信息迁移策略，该方法在时间连贯性、结构保真度和光照真实性方面表现优异。
Reason 该论文研究的是视频重打光与背景替换，属于视频编辑与生成领域，核心问题为时间一致性、光照自然性和细节保留，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其提出的方法如残差校正光流、高频迁移等主要面向视频帧间的视觉连续性建模，难以迁移到解决OCR幻觉、复杂版式解析或长文档推理等DIU核心痛点。
Abstract Video relighting with background replacement is a challenging task critical for applications in film production and creative media. Existing methods struggle to balance temporal consistency, spatial fidelity, and illumination naturalness. To address these issues, we introduce FlowPortal, a novel training-free flow-based video relighting framework. Our core innovation is a Residual-Corrected Flow mechanism that transforms a standard flow-based model into an editing model, guaranteeing perfect reconstruction when input conditions are identical and enabling faithful relighting when they differ, resulting in high structural consistency. This is further enhanced by a Decoupled Condition Design for precise lighting control and a High-Frequency Transfer mechanism for detail preservation. Additionally, a masking strategy isolates foreground relighting from background pure generation process. Experiments demonstrate that FlowPortal achieves superior performance in temporal coherence, structural preservation, and lighting realism, while maintaining high efficiency. Project Page: https://gaowenshuo.github.io/FlowPortalProject/.
Authors Wenshuo Gao, Junyi Fan, Jiangyue Zeng, Shuai Yang
Comment Project Page: https://gaowenshuo.github.io/FlowPortalProject/
Categories Computer Vision and Pattern Recognition
0
A Tri-Modal Dataset and a Baseline System for Tracking Unmanned Aerial Vehicles
Title CN 一种用于跟踪无人机的三模态数据集和基线系统
Keywords 多模态跟踪 · 无人机跟踪 · RGB-IR-Event · MM-UAV · 动态融合
Summary 本文提出了首个大规模多模态无人机跟踪基准MM-UAV，包含RGB、红外和事件信号三种模态，涵盖30多种复杂场景和超过280万帧标注数据。同时设计了一个新的多模态多目标跟踪框架，包含自适应对齐模块和动态融合机制，并引入基于事件数据的关联增强策略以提升ID保持能力。实验表明该方法优于现有SOTA方法。
Reason 该论文属于视频理解与多模态感知领域，研究任务为无人机跟踪（UAV Tracking），使用RGB、红外和事件相机信号，与文档图像理解（DIU）或DocVQA无直接关联。其方法聚焦于多模态对齐与融合以提升动态目标跟踪性能，无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心问题。
Abstract With the proliferation of low altitude unmanned aerial vehicles (UAVs), visual multi-object tracking is becoming a critical security technology, demanding significant robustness even in complex environmental conditions. However, tracking UAVs using a single visual modality often fails in challenging scenarios, such as low illumination, cluttered backgrounds, and rapid motion. Although multi-modal multi-object UAV tracking is more resilient, the development of effective solutions has been hindered by the absence of dedicated public datasets. To bridge this gap, we release MM-UAV, the first large-scale benchmark for Multi-Modal UAV Tracking, integrating three key sensing modalities, e.g. RGB, infrared (IR), and event signals. The dataset spans over 30 challenging scenarios, with 1,321 synchronised multi-modal sequences, and more than 2.8 million annotated frames. Accompanying the dataset, we provide a novel multi-modal multi-UAV tracking framework, designed specifically for UAV tracking applications and serving as a baseline for future research. Our framework incorporates two key technical innovations, e.g. an offset-guided adaptive alignment module to resolve spatio mismatches across sensors, and an adaptive dynamic fusion module to balance complementary information conveyed by different modalities. Furthermore, to overcome the limitations of conventional appearance modelling in multi-object tracking, we introduce an event-enhanced association mechanism that leverages motion cues from the event modality for more reliable identity maintenance. Comprehensive experiments demonstrate that the proposed framework consistently outperforms state-of-the-art methods. To foster further research in multi-modal UAV tracking, both the dataset and source code will be made publicly available at https://xuefeng-zhu5.github.io/MM-UAV/.
Authors Tianyang Xu, Jinjie Gu, Xuefeng Zhu, XiaoJun Wu, Josef Kittler
Categories Computer Vision and Pattern Recognition
0
Auxiliary Gene Learning: Spatial Gene Expression Estimation by Auxiliary Gene Selection AAAI 2026
Title CN 辅助基因学习：通过辅助基因选择进行空间基因表达估计
Keywords 空间转录组学 · 多任务学习 · 基因选择 · 生物信息学 · 可微分优化
Summary 本文提出了一种名为辅助基因学习（AGL）的方法，通过将低表达基因的表达估计作为辅助任务，联合训练以提升目标基因表达预测的准确性。为有效选择有贡献的辅助基因，作者设计了基于先验知识的可微分Top-k基因选择机制（DkGSB），将组合优化问题转化为可微分形式。实验表明该方法优于传统多任务学习方式。
Reason 该论文属于生物信息学与计算生物学领域，研究任务为基于空间转录组数据的基因表达估计，核心方法聚焦于利用低表达基因作为辅助任务提升主任务性能。尽管使用了多任务学习和可微分基因选择等技术，但其问题设定、数据模态（病理组织图像+基因表达矩阵）与文档图像理解（DIU）无直接关联。既不涉及OCR、布局分析、表格识别等DIU核心任务，也未提出可用于解决文档幻觉、细粒度定位或高分辨率处理的通用视觉或多模态方法。属于典型的‘无关领域’研究。
Abstract Spatial transcriptomics (ST) is a novel technology that enables the observation of gene expression at the resolution of individual spots within pathological tissues. ST quantifies the expression of tens of thousands of genes in a tissue section; however, heavy observational noise is often introduced during measurement. In prior studies, to ensure meaningful assessment, both training and evaluation have been restricted to only a small subset of highly variable genes, and genes outside this subset have also been excluded from the training process. However, since there are likely co-expression relationships between genes, low-expression genes may still contribute to the estimation of the evaluation target. In this paper, we propose $Auxiliary \ Gene \ Learning$ (AGL) that utilizes the benefit of the ignored genes by reformulating their expression estimation as auxiliary tasks and training them jointly with the primary tasks. To effectively leverage auxiliary genes, we must select a subset of auxiliary genes that positively influence the prediction of the target genes. However, this is a challenging optimization problem due to the vast number of possible combinations. To overcome this challenge, we propose Prior-Knowledge-Based Differentiable Top-$k$ Gene Selection via Bi-level Optimization (DkGSB), a method that ranks genes by leveraging prior knowledge and relaxes the combinatorial selection problem into a differentiable top-$k$ selection problem. The experiments confirm the effectiveness of incorporating auxiliary genes and show that the proposed method outperforms conventional auxiliary task learning approaches.
Authors Kaito Shiku, Kazuya Nishimura, Shinnosuke Matsuo, Yasuhiro Kojima, Ryoma Bise
Comment Accepted to Association for the Advancement of Artificial Intelligence (AAAI) 2026
Categories Computer Vision and Pattern Recognition
0
Learning Visually Interpretable Oscillator Networks for Soft Continuum Robots from Video
Title CN 从视频中学习用于软体连续机器人的可视化解释性振荡器网络
Keywords 软体机器人 · 视觉表示学习 · 注意力机制 · 动力学建模 · 可解释性
Summary 本文提出了一种名为Attention Broadcast Decoder（ABCD）的模块，结合2D振荡器网络，用于从高维视觉观测中学习软体连续机器人的动力学模型，并实现对隐变量动态的物理可解释性可视化。该方法能生成像素级注意力图，定位每个隐变量的作用区域，并在无先验知识的情况下自主发现系统的物理结构（如质量-弹簧链）。实验表明该方法显著提升了多步预测精度，并支持潜在空间的平滑外推。
Reason 该论文研究的是软体机器人动力学的建模与可视化，属于具身智能与机器人领域，尽管使用了视觉输入（视频），但其核心任务是机器人系统建模而非文档图像理解（DIU）或视觉语言多模态任务。与DocVQA、OCR、版面分析等DIU核心问题无直接关联，也无法迁移其方法解决文档理解中的OCR幻觉、复杂排版或长文档推理等痛点。
Abstract Data-driven learning of soft continuum robot (SCR) dynamics from high-dimensional observations offers flexibility but often lacks physical interpretability, while model-based approaches require prior knowledge and can be computationally expensive. We bridge this gap by introducing (1) the Attention Broadcast Decoder (ABCD), a plug-and-play module for autoencoder-based latent dynamics learning that generates pixel-accurate attention maps localizing each latent dimension's contribution while filtering static backgrounds. (2) By coupling these attention maps to 2D oscillator networks, we enable direct on-image visualization of learned dynamics (masses, stiffness, and forces) without prior knowledge. We validate our approach on single- and double-segment SCRs, demonstrating that ABCD-based models significantly improve multi-step prediction accuracy: 5.7x error reduction for Koopman operators and 3.5x for oscillator networks on the two-segment robot. The learned oscillator network autonomously discovers a chain structure of oscillators. Unlike standard methods, ABCD models enable smooth latent space extrapolation beyond training data. This fully data-driven approach yields compact, physically interpretable models suitable for control applications.
Authors Henrik Krauss, Johann Licher, Naoya Takeishi, Annika Raatz, Takehisa Yairi
Categories Computer Vision and Pattern Recognition
0
Optimal Pose Guidance for Stereo Calibration in 3D Deformation Measurement
Title CN 用于3D形变测量中立体标定的最优姿态引导
Keywords 立体标定 · 最优姿态引导 · 3D形变测量 · 数字图像相关
Summary 本文提出了一种用于立体光学测量的交互式标定框架，通过联合优化相对与绝对外参，并以协方差矩阵迹最小化为目标函数，自动生成下一个最优标定姿态，提升3D形变测量的精度与效率。实验表明该方法在减少标定图像数量的同时提高了测量准确性，并在热形变测试中与有限元分析结果高度一致。
Reason 该论文研究的是3D形变测量中的立体视觉标定问题，属于工业检测与3D视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心方法聚焦于相机外参优化和标定姿态引导，无法迁移解决OCR幻觉、复杂排版理解、细粒度文本定位等DIU关键痛点。
Abstract Stereo optical measurement techniques, such as digital image correlation (DIC), are widely used in 3D deformation measurement as non-contact, full-field measurement methods, in which stereo calibration is a crucial step. However, current stereo calibration methods lack intuitive optimal pose guidance, leading to inefficiency and suboptimal accuracy in deformation measurements. The aim of this study is to develop an interactive calibration framework that automatically generates the next optimal pose, enabling high-accuracy stereo calibration for 3D deformation measurement. We propose a pose optimization method that introduces joint optimization of relative and absolute extrinsic parameters, with the minimization of the covariance matrix trace adopted as the loss function to solve for the next optimal pose. Integrated with this method is a user-friendly graphical interface, which guides even non-expert users to capture qualified calibration images. Our proposed method demonstrates superior efficiency (requiring fewer images) and accuracy (demonstrating lower measurement errors) compared to random pose, while maintaining robustness across varying FOVs. In the thermal deformation measurement tests on an S-shaped specimen, the results exhibit high agreement with finite element analysis (FEA) simulations in both deformation magnitude and evolutionary trends. We present a pose guidance method for high-precision stereo calibration in 3D deformation measurement. The simulation experiments, real-world experiments, and thermal deformation measurement applications all demonstrate the significant application potential of our proposed method in the field of 3D deformation measurement. Keywords: Stereo calibration, Optimal pose guidance, 3D deformation measurement, Digital image correlation
Authors Dongcai Tan, Shunkun Liang, Bin Li, Banglei Guan, Ang Su, Yuan Lin, Dapeng Zhang, Minggang Wan, Zibin Liu, Chenglong Wang, Jiajian Zhu, Zhang Li, Yang Shang, Qifeng Yu
Categories Computer Vision and Pattern Recognition
0
Stro-VIGRU: Defining the Vision Recurrent-Based Baseline Model for Brain Stroke Classification International Conference on Computational Intelligence and Data Communication, Taylor and Francis Conference Proceedings
Title CN Stro-VIGRU：定义基于视觉循环的脑卒中分类基线模型
Keywords 脑卒中分类 · Vision Transformer · Bi-GRU · 医学图像分析 · CT扫描
Summary 本文提出一种基于预训练Vision Transformer和Bi-GRU的混合模型Stro-VIGRU，用于脑卒中的早期识别。通过冻结部分ViT编码器块并微调其余部分以提取卒中特异性特征，再输入单层Bi-GRU进行分类，并采用数据增强处理类别不平衡问题，在Stroke数据集上达到94.06%的准确率。
Reason 该论文属于医学图像分析领域，研究任务为脑卒中CT图像分类，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。其方法聚焦于ViT与Bi-GRU结合用于病理分类，无法迁移解决OCR幻觉、复杂排版、细粒度定位等DIU核心问题，属于无关领域应用。
Abstract Stroke majorly causes death and disability worldwide, and early recognition is one of the key elements of successful treatment of the same. It is common to diagnose strokes using CT scanning, which is fast and readily available, however, manual analysis may take time and may result in mistakes. In this work, a pre-trained Vision Transformer-based transfer learning framework is proposed for the early identification of brain stroke. A few of the encoder blocks of the ViT model are frozen, and the rest are allowed to be fine-tuned in order to learn brain stroke-specific features. The features that have been extracted are given as input to a single-layer Bi-GRU to perform classification. Class imbalance is handled by data augmentation. The model has achieved 94.06% accuracy in classifying brain stroke from the Stroke Dataset.
Authors Subhajeet Das, Pritam Paul, Rohit Bahadur, Sohan Das
Comment Presented at the International Conference on Computational Intelligence and Data Communication, Accepted for publication in the Taylor and Francis Conference Proceedings
Categories Computer Vision and Pattern Recognition
0
MoReMouse: Monocular Reconstruction of Laboratory Mouse
Title CN MoReMouse：实验室小鼠的单目三维重建
Keywords 3D重建 · 无纹理表面 · 非刚性形变 · 合成数据集 · 三平面表示
Summary 本文提出了MoReMouse，一种专用于C57BL/6品系实验小鼠的单目密集3D重建网络。通过构建首个高保真合成数据集、采用基于Transformer和三平面表示的前馈架构，并引入基于测地线的连续对应嵌入，实现了对小鼠复杂形变部位（如四肢和尾巴）的高质量、稳定重建。实验表明其在准确性和鲁棒性上优于现有开源方法。
Reason 该论文研究的是生物医学领域中对实验小鼠的3D表面重建，属于3D视觉与动物建模范畴，核心任务为非刚性物体的单目重建，与文档图像理解（DIU）、DocVQA、OCR、版面分析等完全无关。尽管使用了视觉-几何结合的方法，但其技术路径针对的是纹理缺失、非刚性形变的小鼠体表重建，无法迁移至文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Laboratory mice, particularly the C57BL/6 strain, are essential animal models in biomedical research. However, accurate 3D surface motion reconstruction of mice remains a significant challenge due to their complex non-rigid deformations, textureless fur-covered surfaces, and the lack of realistic 3D mesh models. Moreover, existing visual datasets for mice reconstruction only contain sparse viewpoints without 3D geometries. To fill the gap, we introduce MoReMouse, the first monocular dense 3D reconstruction network specifically designed for C57BL/6 mice. To achieve high-fidelity 3D reconstructions, we present three key innovations. First, we create the first high-fidelity, dense-view synthetic dataset for C57BL/6 mice by rendering a realistic, anatomically accurate Gaussian mouse avatar. Second, MoReMouse leverages a transformer-based feedforward architecture combined with triplane representation, enabling high-quality 3D surface generation from a single image, optimized for the intricacies of small animal morphology. Third, we propose geodesic-based continuous correspondence embeddings on the mouse surface, which serve as strong semantic priors, improving surface consistency and reconstruction stability, especially in highly dynamic regions like limbs and tail. Through extensive quantitative and qualitative evaluations, we demonstrate that MoReMouse significantly outperforms existing open-source methods in both accuracy and robustness.
Authors Yuan Zhong, Jingxiang Sun, Zhongbin Zhang, Liang An, Yebin Liu
Categories Computer Vision and Pattern Recognition
0
AsynEIO: Asynchronous Monocular Event-Inertial Odometry Using Gaussian Process Regression
Title CN AsynEIO：基于高斯过程回归的异步单目事件-惯性里程计
Keywords 事件相机 · 惯性里程计 · 高斯过程回归 · 异步融合 · SLAM
Summary 本文提出AsynEIO，一种基于高斯过程回归的异步单目事件-惯性里程计方法，通过事件驱动前端从原始事件流中高精度追踪特征轨迹，并结合惯性数据在统一框架中实现异步融合。实验表明该方法在高速和低光照条件下优于现有技术。
Reason 该论文研究的是事件相机与惯性传感器在高动态、低光照场景下的运动估计问题，属于机器人定位与导航领域（如SLAM），与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。其核心技术为异步数据融合与高斯过程回归，无法迁移解决OCR幻觉、复杂排版理解、细粒度文本定位等文档核心问题。
Abstract Event cameras, when combined with inertial sensors, show significant potential for motion estimation in challenging scenarios, such as high-speed maneuvers and low-light environments. There are many methods for producing such estimations, but most boil down to a synchronous discrete-time fusion problem. However, the asynchronous nature of event cameras and their unique fusion mechanism with inertial sensors remain underexplored. In this paper, we introduce a monocular event-inertial odometry method called AsynEIO, designed to fuse asynchronous event and inertial data within a unified Gaussian Process (GP) regression framework. Our approach incorporates an event-driven frontend that tracks feature trajectories directly from raw event streams at a high temporal resolution. These tracked feature trajectories, along with various inertial factors, are integrated into the same GP regression framework to enable asynchronous fusion. With deriving analytical residual Jacobians and noise models, our method constructs a factor graph that is iteratively optimized and pruned using a sliding-window optimizer. Comparative assessments highlight the performance of different inertial fusion strategies, suggesting optimal choices for varying conditions. Experimental results on both public datasets and our own event-inertial sequences indicate that AsynEIO outperforms existing methods, especially in high-speed and low-illumination scenarios.
Authors Zhixiang Wang, Xudong Li, Yizhai Zhang, Fan Zhang, Panfeng Huang
Comment 20 pages, 20 figures
Categories Computer Vision and Pattern Recognition
0
TRIDENT: A Trimodal Cascade Generative Framework for Drug and RNA-Conditioned Cellular Morphology Synthesis
Title CN TRIDENT：一种用于药物和RNA条件下的细胞形态合成的三模态级联生成框架
Keywords 生成模型 · 多模态学习 · 细胞形态合成 · 基因表达 · 计算生物学
Summary 本文提出了TRIDENT，一个基于药物刺激和基因表达谱联合条件生成细胞形态图像的级联生成框架，并构建了MorphoGene数据集进行训练与评估。实验表明该方法在生成真实感细胞图像方面显著优于现有方法，并验证了RNA信息对形态生成的关键作用。该工作主要服务于计算生物学中的虚拟细胞建模，不涉及文档图像理解任务。
Reason 该论文研究的是生物医学图像生成任务，聚焦于从基因表达数据生成细胞形态图像，属于纯生物成像领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管使用了多模态生成框架，但其目标、数据和应用场景均远离文档结构分析、文本识别或DocVQA等核心问题，无法迁移用于解决OCR幻觉、复杂排版或长文档推理等DIU痛点。
Abstract Accurately modeling the relationship between perturbations, transcriptional responses, and phenotypic changes is essential for building an AI Virtual Cell (AIVC). However, existing methods typically constrained to modeling direct associations, such as Perturbation $\rightarrow$ RNA or Perturbation $\rightarrow$ Morphology, overlook the crucial causal link from RNA to morphology. To bridge this gap, we propose TRIDENT, a cascade generative framework that synthesizes realistic cellular morphology by conditioning on both the perturbation and the corresponding gene expression profile. To train and evaluate this task, we construct MorphoGene, a new dataset pairing L1000 gene expression with Cell Painting images for 98 compounds. TRIDENT significantly outperforms state-of-the-art approaches, achieving up to 7-fold improvement with strong generalization to unseen compounds. In a case study on docetaxel, we validate that RNA-guided synthesis accurately produces the corresponding phenotype. An ablation study further confirms that this RNA conditioning is essential for the model's high fidelity. By explicitly modeling transcriptome-phenome mapping, TRIDENT provides a powerful in silico tool and moves us closer to a predictive virtual cell.
Authors Rui Peng, Ziru Liu, Lingyuan Ye, Yuxing Lu, Boxin Shi, Jinzhuo Wang
Categories Computer Vision and Pattern Recognition
0
Point-to-Point: Sparse Motion Guidance for Controllable Video Editing
Title CN 点对点：用于可控视频编辑的稀疏运动引导
Keywords 视频编辑 · 运动表示 · 锚点令牌 · 扩散模型 · 可控生成
Summary 本文提出一种名为Point-to-Point的视频编辑方法，通过引入‘锚点令牌’来捕捉视频中最关键的运动轨迹，实现对主体运动的高效保留与可控编辑。该方法利用视频扩散模型的先验知识自动生成有意义的点轨迹，并支持灵活重定位以适应新主体，在多样场景下实现了更优的编辑与运动保真度平衡。
Reason 该论文研究的是视频编辑中的运动控制问题，提出了一种基于锚点令牌（anchor tokens）的运动表示方法。尽管其方法涉及视觉轨迹和动态建模，但核心任务属于视频生成与编辑领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。所解决的‘运动保真度’与‘编辑控制’权衡问题在文档理解中不存在，且未见其方法可迁移至OCR去幻觉、复杂版面分析、细粒度文本定位或长文档推理等DIU核心痛点。属于无关领域工作。
Abstract Accurately preserving motion while editing a subject remains a core challenge in video editing tasks. Existing methods often face a trade-off between edit and motion fidelity, as they rely on motion representations that are either overfitted to the layout or only implicitly defined. To overcome this limitation, we revisit point-based motion representation. However, identifying meaningful points remains challenging without human input, especially across diverse video scenarios. To address this, we propose a novel motion representation, anchor tokens, that capture the most essential motion patterns by leveraging the rich prior of a video diffusion model. Anchor tokens encode video dynamics compactly through a small number of informative point trajectories and can be flexibly relocated to align with new subjects. This allows our method, Point-to-Point, to generalize across diverse scenarios. Extensive experiments demonstrate that anchor tokens lead to more controllable and semantically aligned video edits, achieving superior performance in terms of edit and motion fidelity.
Authors Yeji Song, Jaehyun Lee, Mijin Koo, JunHoo Lee, Nojun Kwak
Categories Computer Vision and Pattern Recognition
0
Frame-wise Conditioning Adaptation for Fine-Tuning Diffusion Models in Text-to-Video Prediction TMLR
Title CN 用于文本到视频预测中扩散模型微调的逐帧条件自适应
Keywords 文本到视频生成 · 扩散模型 · 微调 · 条件自适应 · 时序一致性
Summary 本文针对文本到视频预测任务，提出一种名为Frame-wise Conditioning Adaptation（FCA）的微调方法，通过引入逐帧变化的文本嵌入来增强生成连续性。虽然在TVP任务上达到SOTA，但其技术路线专注于视频生成中的时间动态建模，与文档理解任务无直接关联。
Reason 该论文属于视频生成领域，研究的是文本到视频预测任务中的扩散模型微调方法，提出了一种逐帧条件自适应（FCA）策略。尽管其方法涉及多模态建模和时序一致性，但核心问题与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Text-video prediction (TVP) is a downstream video generation task that requires a model to produce subsequent video frames given a series of initial video frames and text describing the required motion. In practice TVP methods focus on a particular category of videos depicting manipulations of objects carried out by human beings or robot arms. Previous methods adapt models pre-trained on text-to-image tasks, and thus tend to generate video that lacks the required continuity. A natural progression would be to leverage more recent pre-trained text-to-video (T2V) models. This approach is rendered more challenging by the fact that the most common fine-tuning technique, low-rank adaptation (LoRA), yields undesirable results. In this work, we propose an adaptation-based strategy we label Frame-wise Conditioning Adaptation (FCA). Within the module, we devise a sub-module that produces frame-wise text embeddings from the input text, which acts as an additional text condition to aid generation. We use FCA to fine-tune the T2V model, which incorporates the initial frame(s) as an extra condition. We compare and discuss the more effective strategy for injecting such embeddings into the T2V model. We conduct extensive ablation studies on our design choices with quantitative and qualitative performance analysis. Our approach establishes a new state-of-the-art for the task of TVP. Our code is open-source at https://github.com/Cuberick-Orion/FCA .
Authors Zheyuan Liu, Junyan Wang, Zicheng Duan, Cristian Rodriguez-Opazo, Anton van den Hengel
Comment Accepted by TMLR, 11/2025. 29 pages, 15 figures
Categories Computer Vision and Pattern Recognition
0
Leverage Cross-Attention for End-to-End Open-Vocabulary Panoptic Reconstruction
Title CN 利用交叉注意力进行端到端开放词汇全景重建
Keywords 3D重建 · 全景分割 · 交叉注意力 · 具身智能 · 开放词汇
Summary 本文提出PanopticRecon++，一种基于交叉注意力的端到端开放词汇全景重建方法，通过引入可学习的3D高斯作为实例查询，在3D场景中建模实例与场景嵌入的关系，并实现跨帧实例对齐与语义-实例分割一致性。工作主要面向三维重建与机器人仿真，未涉及任何文档图像内容。
Reason 该论文研究的是3D场景下的开放词汇全景重建，属于具身智能与三维视觉领域，核心任务为3D实例建模与跨帧语义对齐。其方法聚焦于3D高斯查询与场景嵌入场的交叉注意力机制，应用于机器人仿真和三维重建场景。完全不涉及文档图像理解（DIU）、DocVQA、OCR或任何二维文档结构分析任务，且技术路径难以迁移到文档图像中的排版理解、文本识别或视觉问答等核心痛点。属于明确的无关领域（3D视觉/具身智能），命中负面清单第2条。
Abstract Open-vocabulary panoptic reconstruction offers comprehensive scene understanding, enabling advances in embodied robotics and photorealistic simulation. In this paper, we propose PanopticRecon++, an end-to-end method that formulates panoptic reconstruction through a novel cross-attention perspective. This perspective models the relationship between 3D instances (as queries) and the scene's 3D embedding field (as keys) through their attention map. Unlike existing methods that separate the optimization of queries and keys or overlook spatial proximity, PanopticRecon++ introduces learnable 3D Gaussians as instance queries. This formulation injects 3D spatial priors to preserve proximity while maintaining end-to-end optimizability. Moreover, this query formulation facilitates the alignment of 2D open-vocabulary instance IDs across frames by leveraging optimal linear assignment with instance masks rendered from the queries. Additionally, we ensure semantic-instance segmentation consistency by fusing query-based instance segmentation probabilities with semantic probabilities in a novel panoptic head supervised by a panoptic loss. During training, the number of instance query tokens dynamically adapts to match the number of objects. PanopticRecon++ shows competitive performance in terms of 3D and 2D segmentation and reconstruction performance on both simulation and real-world datasets, and demonstrates a user case as a robot simulator. Our project website is at: https://yuxuan1206.github.io/panopticrecon_pp/
Authors Xuan Yu, Yuxuan Xie, Yili Liu, Haojian Lu, Rong Xiong, Yiyi Liao, Yue Wang
Comment 18 pages, 10 figures
Categories Computer Vision and Pattern Recognition
0
MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generation
Title CN MammothModa2：一种用于多模态理解与生成的统一AR-扩散框架
Keywords 多模态模型 · 文本到图像生成 · 扩散模型 · 自回归模型 · 图像编辑
Summary 本文提出MammothModa2，一个统一的AR-扩散框架，通过自回归路径进行语义建模，结合DiT解码器实现高质量图像生成。采用多层特征对齐模块连接离散语义表示与连续扩散过程，并通过联合训练目标实现端到端优化。在文本生成图像和图像编辑任务上表现良好，同时保持一定的多模态理解能力。
Reason 该论文属于纯图像生成/编辑方向，聚焦于文本到图像生成和指令式图像编辑任务，使用AR-Diffusion架构实现高保真视觉合成。尽管其声称具备一定的多模态理解能力，但核心贡献在于生成端，且未涉及文档结构、OCR、布局分析或任何文档图像理解（DIU）相关任务。无法迁移解决DIU中的OCR幻觉、复杂排版或长文档推理等关键问题，属于无关领域。
Abstract Unified multimodal models aim to integrate understanding and generation within a single framework, yet bridging the gap between discrete semantic reasoning and high-fidelity visual synthesis remains challenging. We present MammothModa2 (Mammoth2), a unified autoregressive-diffusion (AR-Diffusion) framework designed to effectively couple autoregressive semantic planning with diffusion-based generation. Mammoth2 adopts a serial design: an AR path equipped with generation experts performs global semantic modeling over discrete tokens, while a single-stream Diffusion Transformer (DiT) decoder handles high-fidelity image synthesis. A carefully designed AR-Diffusion feature alignment module combines multi-layer feature aggregation, unified condition encoding, and in-context conditioning to stably align AR's representations with the diffusion decoder's continuous latents. Mammoth2 is trained end-to-end with joint Next-Token Prediction and Flow Matching objectives, followed by supervised fine-tuning and reinforcement learning over both generation and editing. With roughly 60M supervised generation samples and no reliance on pre-trained generators, Mammoth2 delivers strong text-to-image and instruction-based editing performance on public benchmarks, achieving 0.87 on GenEval, 87.2 on DPGBench, and 4.06 on ImgEdit, while remaining competitive with understanding-only backbones (e.g., Qwen3-VL-8B) on multimodal understanding tasks. These results suggest that a carefully coupled AR-Diffusion architecture can provide high-fidelity generation and editing while maintaining strong multimodal comprehension within a single, parameter- and data-efficient model.
Authors Tao Shen, Xin Wan, Taicai Chen, Rui Zhang, Junwen Pan, Dawei Lu, Fanding Lei, Zhilin Lu, Yunfei Yang, Chen Cheng, Qi She, Chang Liu, Zhenbang Sun
Categories Computer Vision and Pattern Recognition
0
A Semi-supervised Physics-Aware Triple-Stream Underwater Image Enhancement Network
Title CN 一种半监督的物理感知三流式水下图像增强网络
Keywords 水下图像增强 · 物理感知网络 · 半监督学习 · 图像恢复 · 深度学习
Summary 本文提出了一种结合物理先验与深度学习的三流式网络PATS-UIENet，用于水下图像增强。通过建模水体退化过程中的直接信号、背向散射和环境光分量，并采用受图像形成模型启发的半监督框架，在多个数据集上取得了良好效果。
Reason 该论文研究的是水下图像增强（UIE），属于特定场景的图像恢复任务，与文档图像理解（DIU）无关。其方法基于物理成像模型和半监督学习，虽然在图像增强方面有一定创新，但无法迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Underwater images normally suffer from degradation due to the transmission medium of water bodies. Both traditional prior-based approaches and deep learning-based methods have been used to address this problem. However, the inflexible assumption of the former often impairs their effectiveness in handling diverse underwater scenes, while the generalization of the latter to unseen images is usually weakened by insufficient data. In this study, we leverage both the physics-based Image Formation Model (IFM) and deep learning techniques for Underwater Image Enhancement (UIE). To this end, we propose a novel Physics-Aware Triple-Stream Underwater Image Enhancement Network, i.e., PATS-UIENet, which comprises a Direct Signal Transmission Estimation Stream (D-Stream), a Backscatter Signal Transmission Estimation Stream (B-Stream) and an Ambient Light Estimation Stream (A-Stream). This network fulfills the UIE task by explicitly estimating the degradation parameters of a revised IFM. We also adopt an IFM-inspired semi-supervised learning framework, which exploits both the labeled and unlabeled images, to address the issue of insufficient data. To our knowledge, such a physics-aware deep network and the IFM-inspired semi-supervised learning framework have not been used for the UIE task before. Our method performs better than, or at least comparably to, sixteen baselines across four testing sets in the degradation estimation and UIE tasks. These promising results should be due to the fact that the proposed method can not only model the degradation but also learn the characteristics of diverse underwater scenes.
Authors Shixuan Xu, Hao Qi, Wei Wang, Chao Huang, Jie Wen, Junyu Dong, Xinghui Dong
Comment 10 pages, 6 figures
Categories Computer Vision and Pattern Recognition
0
UniFlow: Towards Zero-Shot LiDAR Scene Flow for Autonomous Vehicles via Cross-Domain Generalization
Title CN UniFlow：通过跨域泛化实现自动驾驶中LiDAR场景流的零样本迁移
Keywords LiDAR · 场景流 · 跨域泛化 · 自动驾驶 · 点云
Summary 本文提出UniFlow，旨在实现LiDAR场景流在不同传感器间的零样本泛化。通过在多个大规模数据集上联合训练，模型学习到通用的运动先验，在Waymo、nuScenes和未见过的TruckScenes数据集上均取得SOTA性能。然而，该工作专注于3D点云中的运动估计，属于自动驾驶感知任务，与文档图像理解无关。
Reason 该论文研究的是LiDAR场景流估计，属于3D视觉与自动驾驶领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。尽管其跨域泛化方法具有一定通用性，但任务本质（点云运动估计）与DIU核心痛点如OCR幻觉、文本识别、复杂排版理解等完全不相关，无法迁移用于解决文档理解问题。
Abstract LiDAR scene flow is the task of estimating per-point 3D motion between consecutive point clouds. Recent methods achieve centimeter-level accuracy on popular autonomous vehicle (AV) datasets, but are typically only trained and evaluated on a single sensor. In this paper, we aim to learn general motion priors that transfer to diverse and unseen LiDAR sensors. However, prior work in LiDAR semantic segmentation and 3D object detection demonstrate that naively training on multiple datasets yields worse performance than single dataset models. Interestingly, we find that this conventional wisdom does not hold for motion estimation, and that state-of-the-art scene flow methods greatly benefit from cross-dataset training. We posit that low-level tasks such as motion estimation may be less sensitive to sensor configuration; indeed, our analysis shows that models trained on fast-moving objects (e.g., from highway datasets) perform well on fast-moving objects, even across different datasets. Informed by our analysis, we propose UniFlow, a family of feedforward models that unifies and trains on multiple large-scale LiDAR scene flow datasets with diverse sensor placements and point cloud densities. Our frustratingly simple solution establishes a new state-of-the-art on Waymo and nuScenes, improving over prior work by 5.1% and 35.2% respectively. Moreover, UniFlow achieves state-of-the-art accuracy on unseen datasets like TruckScenes, outperforming prior TruckScenes-specific models by 30.1%.
Authors Siyi Li, Qingwen Zhang, Ishan Khatri, Kyle Vedder, Deva Ramanan, Neehar Peri
Comment Project Page: https://lisiyi777.github.io/UniFlow/
Categories Computer Vision and Pattern Recognition
0
Coherent Multi-Agent Trajectory Forecasting in Team Sports with CausalTraj AI4TS Workshop at AAAI 2026
Title CN 基于因果关系的团队运动中多智能体轨迹预测
Keywords 多智能体轨迹预测 · 因果建模 · 联合预测 · 体育分析 · 序列预测
Summary 本文提出CausalTraj模型，用于团队运动中多智能体轨迹的联合预测。该模型基于时间因果结构和似然估计，强调生成具有整体合理性的多智能体未来轨迹，并采用minJADE、minJFDE等联合评估指标。在NBA SportVU等多个数据集上验证了其在生成连贯且真实比赛演化方面的优势。
Reason 该论文研究的是体育场景中多智能体的轨迹预测问题，属于具身智能与视频理解领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法聚焦于运动轨迹的联合概率建模，尽管在多智能体协同建模上有一定创新，但无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract Jointly forecasting trajectories of multiple interacting agents is a core challenge in sports analytics and other domains involving complex group dynamics. Accurate prediction enables realistic simulation and strategic understanding of gameplay evolution. Most existing models are evaluated solely on per-agent accuracy metrics (minADE, minFDE), which assess each agent independently on its best-of-k prediction. However these metrics overlook whether the model learns which predicted trajectories can jointly form a plausible multi-agent future. Many state-of-the-art models are designed and optimized primarily based on these metrics. As a result, they may underperform on joint predictions and also fail to generate coherent, interpretable multi-agent scenarios in team sports. We propose CausalTraj, a temporally causal, likelihood-based model that is built to generate jointly probable multi-agent trajectory forecasts. To better assess collective modeling capability, we emphasize joint metrics (minJADE, minJFDE) that measure joint accuracy across agents within the best generated scenario sample. Evaluated on the NBA SportVU, Basketball-U, and Football-U datasets, CausalTraj achieves competitive per-agent accuracy and the best recorded results on joint metrics, while yielding qualitatively coherent and realistic gameplay evolutions.
Authors Wei Zhen Teoh
Comment 9 pages, 3 figures, accepted to the AI4TS Workshop at AAAI 2026
Categories Computer Vision and Pattern Recognition
0
Parallel qMRI Reconstruction from 4x Accelerated Acquisitions
Title CN 从4倍加速采集并行重建定量MRI
Keywords MRI重建 · 并行成像 · 深度学习 · 欠采样 · 线圈敏感度图
Summary 本文提出了一种端到端的深度学习框架，用于从4倍欠采样的k空间数据中联合估计线圈敏感度图并重建定量MRI图像。该方法包含两个模块：线圈敏感度图估计模块和基于U-Net的重建模块。实验使用10名受试者的多线圈脑部MRI数据，以2倍SENSE重建作为真值进行评估。尽管PSNR/SSIM指标较低，但该方法在视觉上表现出更平滑的重建效果。作者还指出了不同加速因子间空间错位等关键挑战。
Reason 该论文研究的是医学成像中的MRI加速重建问题，属于纯医学图像处理领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无关。其技术路线聚焦于线圈敏感度图估计和U-Net重建，无法迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Magnetic Resonance Imaging (MRI) acquisitions require extensive scan times, limiting patient throughput and increasing susceptibility to motion artifacts. Accelerated parallel MRI techniques reduce acquisition time by undersampling k-space data, but require robust reconstruction methods to recover high-quality images. Traditional approaches like SENSE require both undersampled k-space data and pre-computed coil sensitivity maps. We propose an end-to-end deep learning framework that jointly estimates coil sensitivity maps and reconstructs images from only undersampled k-space measurements at 4x acceleration. Our two-module architecture consists of a Coil Sensitivity Map (CSM) estimation module and a U-Net-based MRI reconstruction module. We evaluate our method on multi-coil brain MRI data from 10 subjects with 8 echoes each, using 2x SENSE reconstructions as ground truth. Our approach produces visually smoother reconstructions compared to conventional SENSE output, achieving comparable visual quality despite lower PSNR/SSIM metrics. We identify key challenges including spatial misalignment between different acceleration factors and propose future directions for improved reconstruction quality.
Authors Mingi Kang
Categories Computer Vision and Pattern Recognition
0
Evaluating Large Language Models on the 2026 Korean CSAT Mathematics Exam: Measuring Mathematical Ability in a Zero-Data-Leakage Setting
Title CN 在2026年韩国高考数学考试中评估大语言模型：零数据泄露环境下的数学能力测量
Keywords 大语言模型 · 数学推理 · 评测基准 · 数据泄露 · 多模态输入
Summary 本文在零数据泄露条件下评估了24种大语言模型在2026年韩国高考数学卷上的表现，强调评测环境的纯净性，并比较不同输入形式和提示语言的影响。结果显示顶级模型接近满分，但几何题仍是难点。研究提出兼顾性能、成本与效率的评估框架。
Reason 该论文属于纯NLP评测任务，聚焦于大语言模型在韩国高考数学题上的推理表现，不涉及任何文档图像理解（DIU）或多模态输入处理。尽管使用了图像输入模态，但其核心是数学推理评测而非文档结构、布局分析或视觉-文本对齐等DIU关键问题。且其数据为特定小语种（韩语），符合负面清单中的‘小语种’与‘平行应用’特征，无底层方法论创新可迁移至文档理解领域。
Abstract This study systematically evaluated the mathematical reasoning capabilities of Large Language Models (LLMs) using the 2026 Korean College Scholastic Ability Test (CSAT) Mathematics section, ensuring a completely contamination-free evaluation environment. To address data leakage issues in existing benchmarks, we digitized all 46 questions (22 common and 24 elective) within two hours of the exam's public release, eliminating any possibility of inclusion in model training data. We conducted comprehensive evaluations of 24 state-of-the-art LLMs across varying input modalities (text, image, text+figure) and prompt languages (Korean, English). GPT-5 Codex achieved the only perfect score (100 points) with text input and Korean prompts, while Grok 4, GPT-5, and Deepseek R1 scored above 95 points. Notably, gpt-oss-20B achieved 95.7 points despite its relatively small size, demonstrating high cost-effectiveness. Problem-specific analysis revealed geometry as the weakest domain (77.7% average) with significant performance degradation on 4-point high-difficulty problems. Text input consistently outperformed image input, while prompt language effects varied by model scale. In reasoning enhancement experiments with GPT-5 series, increased reasoning intensity improved performance (from 82.6 to 100 points) but quadrupled token usage and drastically reduced efficiency, suggesting that models with minimal reasoning may be more practical. This research contributes: (1) implementation of a completely unexposed evaluation environment, (2) a real-exam-based LLM assessment framework, and (3) a practical evaluation perspective integrating performance, cost, and time considerations. Detailed results and model comparisons are available at the 2026 Korean CSAT LLM Evaluation Leaderboard (https://isoft.cnu.ac.kr/csat2026/).
Authors Goun Pyeon, Inbum Heo, Jeesu Jung, Taewook Hwang, Hyuk Namgoong, Hyein Seo, Yerim Han, Eunbin Kim, Hyeonseok Kang, Sangkeun Jung
Comment 52 pages, Korean
Categories Computation and Language
0
LLMs4All: A Review of Large Language Models Across Academic Disciplines
Title CN LLMs4All：跨学科大型语言模型综述
Keywords 大语言模型 · 跨学科应用 · 综述 · 生成式AI · 自然语言处理
Summary 本文综述了大型语言模型在艺术、法律、经济、科学与工程等多个学科中的应用现状，探讨了其对各领域研究与实践的影响，并分析了生成式AI时代下的挑战与未来方向。文章旨在为研究人员提供跨领域应用LLM的概览。
Reason 该论文是一篇关于大型语言模型在各学术领域应用的综述，涵盖法律、经济、工程等多个学科的应用场景。其内容属于对现有技术在不同领域的总结与归纳，未提出任何新的方法或技术改进。完全符合‘平行下游应用’的定义，属于典型的无方法论创新的综述性应用文章，对文档图像理解（DIU）领域无直接技术迁移价值。
Abstract Cutting-edge Artificial Intelligence (AI) techniques keep reshaping our view of the world. For example, Large Language Models (LLMs) based applications such as ChatGPT have shown the capability of generating human-like conversation on extensive topics. Due to the impressive performance on a variety of language-related tasks (e.g., open-domain question answering, translation, and document summarization), one can envision the far-reaching impacts that can be brought by the LLMs with broader real-world applications (e.g., customer service, education and accessibility, and scientific discovery). Inspired by their success, this paper will offer an overview of state-of-the-art LLMs and their integration into a wide range of academic disciplines, including: (1) arts, letters, and law (e.g., history, philosophy, political science, arts and architecture, law), (2) economics and business (e.g., finance, economics, accounting, marketing), and (3) science and engineering (e.g., mathematics, physics and mechanical engineering, chemistry and chemical engineering, life sciences and bioengineering, earth sciences and civil engineering, computer science and electrical engineering). Integrating humanity and technology, in this paper, we will explore how LLMs are shaping research and practice in these fields, while also discussing key limitations, open challenges, and future directions in the era of generative AI. The review of how LLMs are engaged across disciplines-along with key observations and insights-can help researchers and practitioners interested in exploiting LLMs to advance their works in diverse real-world applications.
Authors Yanfang Ye, Zheyuan Zhang, Tianyi Ma, Zehong Wang, Yiyang Li, Shifu Hou, Weixiang Sun, Kaiwen Shi, Yijun Ma, Wei Song, Ahmed Abbasi, Ying Cheng, Jane Cleland-Huang, Steven Corcelli, Robert Goulding, Ming Hu, Ting Hua, John Lalor, Fang Liu, Tengfei Luo, Edward Maginn, Nuno Moniz, Jason Rohr, Brett Savoie, Daniel Slate, Matthew Webber, Olaf Wiest, Johnny Zhang, Nitesh V. Chawla
Categories Computation and Language
0
Non-Linear Scoring Model for Translation Quality Evaluation
Title CN 用于翻译质量评估的非线性评分模型
Keywords 翻译质量评估 · 非线性评分 · MQM · 认知负荷 · Weber-Fechner定律
Summary 本文提出了一种非线性评分模型E(x) = a * ln(1 + b * x)，用于更准确地评估不同长度文本的翻译质量，认为人类对错误的感知呈对数增长而非线性。该模型基于认知负荷理论和心理物理学原理，提升了翻译质量评估的公平性和可解释性。然而，其应用场景完全局限于文本质量评估，未涉及图像、布局或多模态信息处理。
Reason 该论文属于纯文本NLP领域中的翻译质量评估，研究内容为评分模型的设计与认知心理学基础，不涉及任何视觉、文档布局、多模态或OCR相关内容。尽管提到了‘AI-based document-level evaluation’，但其‘document’指代的是文本段落而非文档图像，与文档图像理解（DIU）或DocVQA无实质关联。属于无关领域的平行方法研究，无法迁移解决OCR幻觉、细粒度定位或复杂排版等DIU核心问题。
Abstract Analytic Translation Quality Evaluation (TQE), based on Multidimensional Quality Metrics (MQM), traditionally uses a linear error-to-penalty scale calibrated to a reference sample of 1000-2000 words. However, linear extrapolation biases judgment on samples of different sizes, over-penalizing short samples and under-penalizing long ones, producing misalignment with expert intuition. Building on the Multi-Range framework, this paper presents a calibrated, non-linear scoring model that better reflects how human content consumers perceive translation quality across samples of varying length. Empirical data from three large-scale enterprise environments shows that acceptable error counts grow logarithmically, not linearly, with sample size. Psychophysical and cognitive evidence, including the Weber-Fechner law and Cognitive Load Theory, supports this premise by explaining why the perceptual impact of additional errors diminishes while the cognitive burden grows with scale. We propose a two-parameter model E(x) = a * ln(1 + b * x), a, b > 0, anchored to a reference tolerance and calibrated from two tolerance points using a one-dimensional root-finding step. The model yields an explicit interval within which the linear approximation stays within +/-20 percent relative error and integrates into existing evaluation workflows with only a dynamic tolerance function added. The approach improves interpretability, fairness, and inter-rater reliability across both human and AI-generated translations. By operationalizing a perceptually valid scoring paradigm, it advances translation quality evaluation toward more accurate and scalable assessment. The model also provides a stronger basis for AI-based document-level evaluation aligned with human judgment. Implementation considerations for CAT/LQA systems and implications for human and AI-generated text evaluation are discussed.
Authors Serge Gladkoff, Lifeng Han, Katerina Gasova
Comment ongoing work, 38 pages
Categories Computation and Language
0
A Benchmark for Zero-Shot Belief Inference in Large Language Models
Title CN 大型语言模型中零样本信念推理的基准
Keywords 大语言模型 · 信念推理 · 零样本学习 · 社会计算 · 立场预测
Summary 本文提出了一个用于评估大语言模型在零样本设置下预测个体对多种话题立场的基准，数据来自在线辩论平台。研究探讨了不同信息条件下（如人口统计特征和先前信念）模型预测准确性的变化，揭示了当前LLMs在模拟人类推理方面的能力与局限。但整个工作完全基于文本输入，不涉及图像或文档结构信息。
Reason 该论文研究的是大型语言模型在社会观点和信念推断方面的零样本泛化能力，属于纯自然语言处理与社会计算交叉领域，不涉及任何视觉模态或文档理解任务。其核心关注点为人类信念建模与LLM的社会行为分析，与文档图像理解（DIU）、DocVQA、OCR、布局分析等无直接关联。同时未提出可用于多模态文档处理的底层架构、表示学习或推理机制，无法迁移解决OCR幻觉、细粒度定位或复杂排版理解等DIU核心痛点。
Abstract Beliefs are central to how humans reason, communicate, and form social connections, yet most computational approaches to studying them remain confined to narrow sociopolitical contexts and rely on fine-tuning for optimal performance. Despite the growing use of large language models (LLMs) across disciplines, how well these systems generalize across diverse belief domains remains unclear. We introduce a systematic, reproducible benchmark that evaluates the ability of LLMs to predict individuals' stances on a wide range of topics in a zero-shot setting using data from an online debate platform. The benchmark includes multiple informational conditions that isolate the contribution of demographic context and known prior beliefs to predictive success. Across several small- to medium-sized models, we find that providing more background information about an individual improves predictive accuracy, but performance varies substantially across belief domains. These findings reveal both the capacity and limitations of current LLMs to emulate human reasoning, advancing the study of machine behavior and offering a scalable framework for modeling belief systems beyond the sociopolitical sphere.
Authors Joseph Malone, Rachith Aiyappa, Byunghwee Lee, Haewoon Kwak, Jisun An, Yong-Yeol Ahn
Comment 28 pages, 5 figures
Categories Computation and Language
0
Toward Trustworthy Difficulty Assessments: Large Language Models as Judges in Programming and Synthetic Tasks
Title CN 迈向可信的难度评估：大型语言模型作为编程与合成任务中的评判者
Keywords 大语言模型 · 难度评估 · 编程教育 · 模型可信度 · SHAP解释
Summary 本文比较了GPT-4o与基于特征的LightGBM模型在判断LeetCode题目难度上的表现，发现GPT-4o准确率远低于传统模型，且存在忽视数值约束、偏向简单分类等系统性偏差。研究揭示了LLM作为自动评判者的可靠性缺陷，尤其是在结构化任务中。
Reason 该论文研究的是使用LLM评估编程题目难度的任务，属于纯NLP与教育评估领域，不涉及任何视觉、文档结构或图文多模态理解内容。尽管其探讨了LLM的判断偏差问题，但该问题与文档图像理解（DIU）中的OCR幻觉、布局分析、细粒度定位等核心痛点无直接关联，也无法迁移为解决DIU问题的技术工具。
Abstract Large Language Models (LLMs) have demonstrated impressive capabilities in natural language and code generation, and are increasingly deployed as automatic judges of model outputs and learning activities. Yet, their behavior on structured tasks such as predicting the difficulty of competitive programming problems remains under-explored. We conduct a systematic comparison of GPT-4o, used purely as a natural-language difficulty assessor, against an interpretable Light-GBM ensemble trained on explicit numeric and textual features. On a dataset of 1,825 LeetCode problems labeled Easy, Medium, or Hard, LightGBM attains 86% accuracy, whereas GPT-4o reaches only 37.75%. Detailed analyses, including confusion matrices and SHAP-based interpretability, show that numeric constraints -- such as input size limits and acceptance rates -- play a crucial role in separating Hard problems from easier ones. By contrast, GPT-4o often overlooks these cues and exhibits a strong bias toward simpler categories. We further probe GPT-4o through a synthetic Hard-problem generation protocol. Surprisingly, GPT-4o labels almost all of its own synthetic Hard problems as Medium, contradicting its tendency to downgrade real Hard problems to Easy. Our findings connect to recent work on LLMs-as-judges and automatic difficulty estimation in programming and education, and highlight concrete failure modes that must be addressed before LLM-based judges can be considered trustworthy in competitive programming, educational platforms, or reinforcement-learning pipelines.
Authors H. M. Shadman Tabib, Jaber Ahmed Deedar
Categories Computation and Language
0
ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?
Title CN ReplicationBench：AI代理能否复制天体物理学研究论文？
Keywords AI代理 · 科学发现 · 任务评测 · 复现研究 · 语言模型
Summary 本文提出了ReplicationBench，一个用于评估AI代理是否能复现天体物理学研究论文的评测框架。任务包括复现实验设置、推导、数据分析和代码库，并由原作者参与设计以保证准确性。结果显示当前最先进的语言模型表现极差（低于20%），揭示了AI代理在科研中的一系列失败模式。该工作聚焦于AI作为科研助手的可靠性评估，而非文档理解技术本身。
Reason 该论文属于具身智能/AI代理在科学研究流程中的应用评估，核心任务是测试AI代理复现天体物理论文的能力，属于特定领域的下游应用（Wrapper/Application），并未提出可迁移至文档图像理解的底层方法。研究领域为计算语言学与科学智能交叉，但不涉及文档视觉结构、OCR、VQA或视觉表示学习等DIU核心问题，且无技术源头创新。
Abstract Frontier AI agents show increasing promise as scientific research assistants, and may eventually be useful for extended, open-ended research workflows. However, in order to use agents for novel research, we must first assess the underlying faithfulness and correctness of their work. To evaluate agents as research assistants, we introduce ReplicationBench, an evaluation framework that tests whether agents can replicate entire research papers drawn from the astrophysics literature. Astrophysics, where research relies heavily on archival data and computational study while requiring little real-world experimentation, is a particularly useful testbed for AI agents in scientific research. We split each paper into tasks which require agents to replicate the paper's core contributions, including the experimental setup, derivations, data analysis, and codebase. Each task is co-developed with the original paper authors and targets a key scientific result, enabling objective evaluation of both faithfulness (adherence to original methods) and correctness (technical accuracy of results). ReplicationBench is extremely challenging for current frontier language models: even the best-performing language models score under 20%. We analyze ReplicationBench trajectories in collaboration with domain experts and find a rich, diverse set of failure modes for agents in scientific research. ReplicationBench establishes the first benchmark of paper-scale, expert-validated astrophysics research tasks, reveals insights about agent performance generalizable to other domains of data-driven science, and provides a scalable framework for measuring AI agents' reliability in scientific research.
Authors Christine Ye, Sihan Yuan, Suchetha Cooray, Steven Dillmann, Ian L. V. Roque, Dalya Baron, Philipp Frank, Sergio Martin-Alvarez, Nolan Koblischke, Frank J Qu, Diyi Yang, Risa Wechsler, Ioana Ciuca
Categories Computation and Language
0
Dealing with the Hard Facts of Low-Resource African NLP
Title CN 应对低资源非洲NLP的严峻现实
Keywords 低资源语言 · 语音识别 · 数据收集 · 模型评估 · 非洲语言
Summary 本文报告了在巴姆巴拉语上进行自发语音数据采集（612小时）、半自动转录标注以及构建小型单语模型的工作，并提供了数据收集、标注和模型设计的实践建议。强调了人工评估的重要性，并公开了数据集、模型和代码。
Reason 该论文聚焦于低资源非洲语言（如巴姆巴拉语）的语音数据收集与建模，属于小语种NLP领域。研究内容涉及语音识别和模型训练，但并未涉及文档图像理解、多模态建模或视觉-语言任务。且其语言对象为非中英文的小语种，违反了‘小语种’负面清单。无法迁移至DIU核心问题如OCR幻觉、复杂排版理解等。
Abstract Creating speech datasets, models, and evaluation frameworks for low-resource languages remains challenging given the lack of a broad base of pertinent experience to draw from. This paper reports on the field collection of 612 hours of spontaneous speech in Bambara, a low-resource West African language; the semi-automated annotation of that dataset with transcriptions; the creation of several monolingual ultra-compact and small models using the dataset; and the automatic and human evaluation of their output. We offer practical suggestions for data collection protocols, annotation, and model design, as well as evidence for the importance of performing human evaluation. In addition to the main dataset, multiple evaluation datasets, models, and code are made publicly available.
Authors Yacouba Diarra, Nouhoum Souleymane Coulibaly, Panga Azazia Kamaté, Madani Amadou Tall, Emmanuel Élisé Koné, Aymane Dembélé, Michael Leventhal
Comment 10 pages, 4 figures
Categories Computation and Language
0
Assessing Historical Structural Oppression Worldwide via Rule-Guided Prompting of Large Language Models 2025 IEEE International Conference on Big Data (IEEE BigData 2025)
Title CN 通过规则引导的大语言模型提示评估全球历史结构性压迫
Keywords 大语言模型 · 提示工程 · 社会测量 · 历史压迫 · 跨文化分析
Summary 论文提出一种利用大语言模型通过规则引导的提示策略来评估全球范围内历史结构性压迫的方法，基于多语言新冠研究中的民族自述文本生成压迫得分，并发布了一个开源基准数据集。重点在于社会科学研究中的可解释性与跨文化有效性，而非文档理解的技术突破。
Reason 该论文属于平行下游应用，将LLM应用于社会科学研究中的历史压迫测量，使用提示工程分析多语言文本数据。尽管涉及LLM与非结构化文本的交互，但其任务本质是社会科学指标构建，并未针对文档图像理解（如DocVQA、布局分析、表格识别等）中的视觉-语言对齐、OCR幻觉、复杂排版或高分辨率处理等核心问题提出方法论创新。
Abstract Traditional efforts to measure historical structural oppression struggle with cross-national validity due to the unique, locally specified histories of exclusion, colonization, and social status in each country, and often have relied on structured indices that privilege material resources while overlooking lived, identity-based exclusion. We introduce a novel framework for oppression measurement that leverages Large Language Models (LLMs) to generate context-sensitive scores of lived historical disadvantage across diverse geopolitical settings. Using unstructured self-identified ethnicity utterances from a multilingual COVID-19 global study, we design rule-guided prompting strategies that encourage models to produce interpretable, theoretically grounded estimations of oppression. We systematically evaluate these strategies across multiple state-of-the-art LLMs. Our results demonstrate that LLMs, when guided by explicit rules, can capture nuanced forms of identity-based historical oppression within nations. This approach provides a complementary measurement tool that highlights dimensions of systemic exclusion, offering a scalable, cross-cultural lens for understanding how oppression manifests in data-driven research and public health contexts. To support reproducible evaluation, we release an open-sourced benchmark dataset for assessing LLMs on oppression measurement (https://github.com/chattergpt/HSO-Bench).
Authors Sreejato Chatterjee, Linh Tran, Quoc Duy Nguyen, Roni Kirson, Drue Hamlin, Harvest Aquino, Hanjia Lyu, Jiebo Luo, Timothy Dye
Comment To appear in the 2025 IEEE International Conference on Big Data (IEEE BigData 2025)
Categories Computation and Language
0
From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence
Title CN 从代码基础模型到智能体与应用：代码智能实用指南
Keywords 代码大模型 · 智能体 · 监督微调 · 强化学习 · 软件工程
Summary 本文系统综述了代码大语言模型的发展路径，涵盖数据构建、预训练、微调、强化学习及智能体构建，并分析了通用与专用代码模型的性能差异。同时探讨了学术研究与工业实践之间的差距，提出了若干未来方向。然而，所有技术内容均局限于代码智能领域，缺乏对多模态文档理解的适用性。
Reason 该论文聚焦于代码大模型及其在软件工程中的应用，属于典型的‘平行应用’范畴。尽管涉及LLM训练范式和智能体架构，但其方法论完全围绕代码生成与自动化编程展开，未提出可迁移至文档图像理解（DIU）领域的底层技术创新。例如，文中讨论的SFT、RLHF、Agent工作流均针对代码任务设计，无法直接解决DIU中的OCR幻觉、复杂版面解析或细粒度视觉定位等核心问题。
Abstract Large language models (LLMs) have fundamentally transformed automated software development by enabling direct translation of natural language descriptions into functional code, driving commercial adoption through tools like Github Copilot (Microsoft), Cursor (Anysphere), Trae (ByteDance), and Claude Code (Anthropic). While the field has evolved dramatically from rule-based systems to Transformer-based architectures, achieving performance improvements from single-digit to over 95\% success rates on benchmarks like HumanEval. In this work, we provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs, systematically examining the complete model life cycle from data curation to post-training through advanced prompting paradigms, code pre-training, supervised fine-tuning, reinforcement learning, and autonomous coding agents. We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder), critically examining the techniques, design decisions, and trade-offs. Further, we articulate the research-practice gap between academic research (e.g., benchmarks and tasks) and real-world deployment (e.g., software-related code tasks), including code correctness, security, contextual awareness of large codebases, and integration with development workflows, and map promising research directions to practical needs. Last, we conduct a series of experiments to provide a comprehensive analysis of code pre-training, supervised fine-tuning, and reinforcement learning, covering scaling law, framework selection, hyperparameter sensitivity, model architectures, and dataset comparisons.
Authors Jian Yang, Wei Zhang, Shark Liu, Jiajun Wu, Shawn Guo, Yizhi Li
Categories Computation and Language
0
For Those Who May Find Themselves on the Red Team
Title CN 致那些可能成为红队成员的人
Keywords 大语言模型 · 可解释性 · 红队 · 文学研究 · 人文计算
Summary 这是一篇立场论文，主张文学学者应参与大语言模型的可解释性研究，尽管这一过程可能涉及意识形态冲突或共谋。作者认为当前对可解释性的工具性标准不足以衡量LLM中的解释行为，并提议‘红队’可以作为这种学术斗争的发生场所。
Reason 该论文属于纯NLP领域的人文反思，讨论文学学者如何参与大语言模型的可解释性研究，并涉及红队（Red Team）的意识形态角色。内容与文档图像理解（DIU）、多模态模型、OCR、VQA等核心技术问题完全无关，属于语言学与人文交叉的哲学性讨论，无任何技术方法可迁移至DIU领域。
Abstract This position paper argues that literary scholars must engage with large language model (LLM) interpretability research. While doing so will involve ideological struggle, if not out-right complicity, the necessity of this engagement is clear: the abiding instrumentality of current approaches to interpretability cannot be the only standard by which we measure interpretation with LLMs. One site at which this struggle could take place, I suggest, is the red team.
Authors Tyler Shoemaker
Categories Computation and Language
0
Multi-Agent Collaborative Filtering: Orchestrating Users and Items for Agentic Recommendations
Title CN 多智能体协同过滤：为智能推荐协调用户与物品
Keywords 多智能体系统 · 协同过滤 · 推荐系统 · 大语言模型 · 智能体协作
Summary 本文提出了一种名为MACF的多智能体协同过滤框架，用于提升基于大语言模型的推荐系统的性能。通过将相似用户和相关物品实例化为具有独特画像的LLM智能体，并引入一个中央协调者智能体来动态管理协作过程，MACF试图模拟传统协同过滤的思想，同时增强交互灵活性。实验表明该方法在多个领域的数据集上优于现有基线。
Reason 该论文属于典型的‘平行下游应用’，将多智能体框架应用于推荐系统这一特定领域，未提出可用于文档图像理解（DIU）的底层方法或技术源头。尽管涉及Agent协作机制，但其设计完全面向推荐任务，缺乏对OCR幻觉、细粒度定位、复杂排版处理等DIU核心痛点的迁移价值。
Abstract Agentic recommendations cast recommenders as large language model (LLM) agents that can plan, reason, use tools, and interact with users of varying preferences in web applications. However, most existing agentic recommender systems focus on generic single-agent plan-execute workflows or multi-agent task decomposition pipelines. Without recommendation-oriented design, they often underuse the collaborative signals in the user-item interaction history, leading to unsatisfying recommendation results. To address this, we propose the Multi-Agent Collaborative Filtering (MACF) framework for agentic recommendations, drawing an analogy between traditional collaborative filtering algorithms and LLM-based multi-agent collaboration. Specifically, given a target user and query, we instantiate similar users and relevant items as LLM agents with unique profiles. Each agent is able to call retrieval tools, suggest candidate items, and interact with other agents. Different from the static preference aggregation in traditional collaborative filtering, MACF employs a central orchestrator agent to adaptively manage the collaboration between user and item agents via dynamic agent recruitment and personalized collaboration instruction. Experimental results on datasets from three different domains show the advantages of our MACF framework compared to strong agentic recommendation baselines.
Authors Yu Xia, Sungchul Kim, Tong Yu, Ryan A. Rossi, Julian McAuely
Categories Computation and Language
0
ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detection
Title CN ExPO-HM：通过先解释后检测的方法进行仇恨模因检测
Keywords 仇恨模因检测 · 解释性AI · 链式思维 · 强化学习 · 多模态分类
Summary 本文提出ExPO-HM，一种用于仇恨模因检测的‘先解释后检测’框架，结合SFT预热、带课程学习的GRPO以及条件决策熵（CDE）来提升模型在二分类、细粒度分类和推理可解释性上的表现。尽管在推理机制上有改进，但其应用场景局限于内容安全审查，与文档结构理解、信息提取等DIU核心任务无关。
Reason 该论文聚焦于仇恨模因（hateful memes）的检测，属于社交媒体内容安全领域，虽然使用了多模态大模型和Chain-of-Thought等技术，但其任务本质是特定场景下的视觉-语言分类问题，并未针对文档图像理解（DIU）中的核心痛点如OCR幻觉、复杂版面分析、细粒度文本定位或长文档推理进行建模。方法上虽有推理质量优化（如CDE），但整体框架是面向情感/安全策略判断的垂直应用，不具备向文档理解迁移的方法论价值。
Abstract Hateful memes have emerged as a particularly challenging form of online abuse, motivating the development of automated detection systems. Most prior approaches rely on direct detection, producing only binary predictions. Such models fail to provide the context and explanations that real-world moderation requires. Recent Explain-then-Detect approaches, using Chain-of-Thought prompting or LMM agents, perform worse than simple SFT baselines, and even advanced post-training methods such as GRPO fail to close the gap. Our analysis identifies two key issues of such systems: important policy-relevant cues such as targets and attack types are not hypothesized by the model as a likely explanation; and the binary reward signal is insufficient to guide reasoning. To address these challenges, we propose ExPO-HM (Explain-then-Detect Policy Optimization for Hateful Memes), inspired by the training and evaluation process of human annotators. ExPO-HM combines SFT warmup, GRPO with curriculum learning, and Conditional Decision Entropy (CDE) as both metric and reward for reasoning quality. Across three hateful meme benchmarks, ExPO-HM achieves state-of-the-art performance on binary detection, fine-grained classification, and reasoning quality, with up to 15\% and 17\% F1 improvement over the GRPO and DPO baselines, respectively. By moving hateful meme detection from simple binary alarms to explanation-driven detection, ExPO-HM provides accurate, interpretable, and actionable moderation support.
Authors Jingbiao Mei, Mingsheng Sun, Jinghong Chen, Pengda Qin, Yuhong Li, Da Chen, Bill Byrne
Comment Preprint
Categories Computation and Language
0
Tu crois que c'est vrai ? Diversite des regimes d'enonciation face aux fake news et mecanismes d'autoregulation conversationnelle
Title CN 你认为这是真的吗？面对假新闻和对话自我调节机制的陈述模式多样性
Keywords 假新闻 · 社交媒体分析 · 话语分析 · 政治极化 · 法语
Summary 本文通过定量分析和访谈方法研究法国推特和Facebook用户对假新闻的分享行为与反应机制，发现假新闻传播集中在政治化程度较高的活跃用户群体中，而其他用户会通过话语谨慎或直接干预表达质疑，但难以形成有效公共讨论。
Reason 该论文属于纯计算语言学领域，研究法语社交媒体上的假新闻传播与用户互动行为，不涉及任何视觉或文档图像理解内容。同时，论文为小语种（法语）研究，且主题与文档图像理解（DIU）、多模态大模型、OCR、布局分析等完全无关。
Abstract This thesis addresses two paradoxes: (1) why empirical studies find that fake news represent only a small share of the information consulted and shared on social media despite the absence of editorial control or journalistic norms, and (2) how political polarization has intensified even though users do not appear especially receptive to fake news. To investigate these issues, two complementary studies were carried out on Twitter and Facebook, combining quantitative analyses of digital traces with online observation and interviews. This mixed-methods design avoids reducing users to single reactions to identified fake items and instead examines the variety of practices across different interactional situations, online and offline, while recording socio-demographic traits. The first study mapped users who shared at least one item labeled fake by fact-checkers in the French Twittersphere. The second used a corpus of items flagged by Facebook users to study reactions to statements whose epistemic status is uncertain. Three main findings emerge. First, sharing fake news is concentrated among a limited group of users who are not less educated or cognitively disadvantaged but are more politicized and critical of institutions; owing to their high activity and prolific sharing, they can help set the agenda for their political camp. Second, exposed users can deploy varying forms of critical distance depending on their social position and the interactional norms of the situations they inhabit: either discursive caution (prudence énonciative) or interventions ('points d'arrêt') that express disagreement or corrections. Third, these forms of critical distance seldom yield genuine deliberative debates or agonistic pluralism; rather, they often produce dialogues of the deaf among a small, particularly active minority.
Authors Manon Berriche
Comment in French language
Categories Computation and Language
0
UPLME: Uncertainty-Aware Probabilistic Language Modelling for Robust Empathy Regression
Title CN 用于鲁棒共情回归的不确定性感知概率语言建模
Keywords 不确定性建模 · 标签噪声 · 共情回归 · 概率语言模型 · 变分集成
Summary 本文提出UPLME，一种用于共情回归任务的不确定性感知概率语言模型框架，通过贝叶斯变分模型集成来同时预测共情得分和异方差不确定性，并引入两种新损失项以提升不确定性量化质量。在存在标签噪声的两个公开数据集上实现了SOTA性能，并验证了其识别噪声样本的能力。
Reason 该论文研究的是文本情感分析中的共情分数回归问题，属于纯自然语言处理领域的情感预测任务，且关注的是标签噪声下的回归建模。尽管其不确定性建模方法具有一定通用性，但与文档图像理解（DIU）、视觉-语言模型（VLM）或多模态输入无关，无法迁移解决OCR幻觉、细粒度定位或复杂排版等文档理解核心痛点。
Abstract Noisy self-reported empathy scores challenge supervised learning for empathy regression. While many algorithms have been proposed for learning with noisy labels in textual classification problems, the regression counterpart is relatively under-explored. We propose UPLME, an uncertainty-aware probabilistic language modelling framework to capture label noise in empathy regression tasks. One of the novelties in UPLME is a probabilistic language model that predicts both empathy scores and heteroscedastic uncertainty, and is trained using Bayesian concepts with variational model ensembling. We further introduce two novel loss components: one penalises degenerate Uncertainty Quantification (UQ), and another enforces similarity between the input pairs on which empathy is being predicted. UPLME achieves state-of-the-art performance (Pearson Correlation Coefficient: $0.558\rightarrow0.580$ and $0.629\rightarrow0.634$) in terms of the performance reported in the literature on two public benchmarks with label noise. Through synthetic label noise injection, we demonstrate that UPLME is effective in distinguishing between noisy and clean samples based on the predicted uncertainty. UPLME further outperform (Calibration error: $0.571\rightarrow0.376$) a recent variational model ensembling-based UQ method designed for regression problems. Code is publicly available at https://github.com/hasan-rakibul/UPLME.
Authors Md Rakibul Hasan, Md Zakir Hossain, Aneesh Krishna, Shafin Rahman, Tom Gedeon
Comment Code available at https://github.com/hasan-rakibul/UPLME
Categories Computation and Language
0
OutSafe-Bench: A Benchmark for Multimodal Offensive Content Detection in Large Language Models
Title CN OutSafe-Bench：面向大语言模型的多模态有害内容检测基准
Keywords 多模态安全 · 内容检测 · 大模型对齐 · 评测基准 · FairScore
Summary 本文提出了OutSafe-Bench，一个面向多模态大语言模型的综合性内容安全评测基准，涵盖文本、图像、音频和视频四种模态，标注了九类内容风险，并设计了MCRS评分和FairScore评估框架来提升评测可靠性。研究揭示了当前主流MLLM在安全防护方面的严重不足。
Reason 该论文聚焦于多模态大模型的安全性评估，属于纯NLP安全对齐（Safety）领域，尽管涉及多模态输入，但其核心任务是检测有毒、偏见或违法内容，与文档图像理解中的OCR幻觉、细粒度识别、复杂排版解析等核心痛点无关。未提出可用于DIU的方法论创新，属于负面清单中的‘纯NLP安全/对齐’范畴。
Abstract Since Multimodal Large Language Models (MLLMs) are increasingly being integrated into everyday tools and intelligent agents, growing concerns have arisen regarding their possible output of unsafe contents, ranging from toxic language and biased imagery to privacy violations and harmful misinformation. Current safety benchmarks remain highly limited in both modality coverage and performance evaluations, often neglecting the extensive landscape of content safety. In this work, we introduce OutSafe-Bench, the first most comprehensive content safety evaluation test suite designed for the multimodal era. OutSafe-Bench includes a large-scale dataset that spans four modalities, featuring over 18,000 bilingual (Chinese and English) text prompts, 4,500 images, 450 audio clips and 450 videos, all systematically annotated across nine critical content risk categories. In addition to the dataset, we introduce a Multidimensional Cross Risk Score (MCRS), a novel metric designed to model and assess overlapping and correlated content risks across different categories. To ensure fair and robust evaluation, we propose FairScore, an explainable automated multi-reviewer weighted aggregation framework. FairScore selects top-performing models as adaptive juries, thereby mitigating biases from single-model judgments and enhancing overall evaluation reliability. Our evaluation of nine state-of-the-art MLLMs reveals persistent and substantial safety vulnerabilities, underscoring the pressing need for robust safeguards in MLLMs.
Authors Yuping Yan, Yuhan Xie, Yuanshuai Li, Yingchao Yu, Lingjuan Lyu, Yaochu Jin
Categories Computation and Language
0
Gradient Masters at BLP-2025 Task 1: Advancing Low-Resource NLP for Bengali using Ensemble-Based Adversarial Training for Hate Speech Detection Second International Workshop on Bangla Language Processing (BLP-2025)
Title CN 基于集成对抗训练的孟加拉语低资源NLP进展：用于仇恨言论检测
Keywords 仇恨言论检测 · 低资源NLP · 孟加拉语 · 集成学习 · 对抗训练
Summary 本文提出了一种基于集成和对抗训练的方法，用于在低资源条件下进行孟加拉语仇恨言论识别，在BLP-2025共享任务中取得较好成绩。研究局限于纯文本分类，不涉及视觉信息处理或文档结构理解。
Reason 该论文聚焦于孟加拉语的仇恨言论分类任务，属于小语种（非中英）的特定语言应用，且为纯文本NLP任务，未涉及任何文档图像、视觉模态或OCR相关内容。完全偏离文档图像理解（DIU）核心问题，属于负面清单中的‘小语种’和‘平行下游应用’，无方法论迁移价值。
Abstract This paper introduces the approach of "Gradient Masters" for BLP-2025 Task 1: "Bangla Multitask Hate Speech Identification Shared Task". We present an ensemble-based fine-tuning strategy for addressing subtasks 1A (hate-type classification) and 1B (target group classification) in YouTube comments. We propose a hybrid approach on a Bangla Language Model, which outperformed the baseline models and secured the 6th position in subtask 1A with a micro F1 score of 73.23% and the third position in subtask 1B with 73.28%. We conducted extensive experiments that evaluated the robustness of the model throughout the development and evaluation phases, including comparisons with other Language Model variants, to measure generalization in low-resource Bangla hate speech scenarios and data set coverage. In addition, we provide a detailed analysis of our findings, exploring misclassification patterns in the detection of hate speech.
Authors Syed Mohaiminul Hoque, Naimur Rahman, Md Sakhawat Hossain
Comment 6 pages, 2 figures, 4 tables. Accepted at the Second International Workshop on Bangla Language Processing (BLP-2025) co-located with AACL-IJCNLP 2025. Ranked 6th (Subtask 1A, 73.23% micro F1) and 3rd (Subtask 1B, 73.28% micro F1) on the official leaderboard
Categories Computation and Language
0
BadGraph: A Backdoor Attack Against Latent Diffusion Model for Text-Guided Graph Generation
Title CN BadGraph：针对文本引导图生成的潜在扩散模型的后门攻击
Keywords 后门攻击 · 图生成 · 潜在扩散模型 · 文本引导生成 · 安全性
Summary 本文提出BadGraph，一种针对潜在扩散模型在文本引导图生成任务上的后门攻击方法。通过在训练数据中注入带有文本触发器的恶意样本，攻击者可在推理阶段激活预设的子图生成行为，而对正常输入性能影响极小。实验表明该攻击在低污染率下即可实现高成功率，揭示了此类模型在药物发现等应用中的安全隐患。
Reason 该论文属于图生成与安全领域的交叉研究，聚焦于文本引导的图生成模型中的后门攻击问题。其核心任务与文档图像理解（DIU）无直接关联，既不涉及文档结构、布局分析、表格识别，也不解决OCR幻觉、细粒度定位或长文档推理等DIU核心痛点。尽管使用了‘文本-视觉’多模态设定，但生成的是分子图或知识图结构，而非文档图像内容，技术路径无法迁移到文档理解场景。
Abstract The rapid progress of graph generation has raised new security concerns, particularly regarding backdoor vulnerabilities. While prior work has explored backdoor attacks in image diffusion and unconditional graph generation, conditional, especially text-guided graph generation remains largely unexamined. This paper proposes BadGraph, a backdoor attack method against latent diffusion models for text-guided graph generation. BadGraph leverages textual triggers to poison training data, covertly implanting backdoors that induce attacker-specified subgraphs during inference when triggers appear, while preserving normal performance on clean inputs. Extensive experiments on four benchmark datasets (PubChem, ChEBI-20, PCDes, MoMu) demonstrate the effectiveness and stealth of the attack: less than 10% poisoning rate can achieves 50% attack success rate, while 24% suffices for over 80% success rate, with negligible performance degradation on benign samples. Ablation studies further reveal that the backdoor is implanted during VAE and diffusion training rather than pretraining. These findings reveal the security vulnerabilities in latent diffusion models of text-guided graph generation, highlight the serious risks in models' applications such as drug discovery and underscore the need for robust defenses against the backdoor attack in such diffusion models.
Authors Liang Ye, Shengqin Chen, Jiazhu Dai
Categories Computation and Language
0
From Archives to Decisions: Multi-Agent Pharmaceutical Co-Scientist for Traceable Drug Discovery and Reverse Translation
Title CN 从档案到决策：用于可追溯药物发现与逆向转化的多智能体药物共同科学家
Keywords 多智能体系统 · 药物发现 · 逆向转化 · 知识复用 · 专家评估
Summary 本文提出了DiscoVerse，一个用于支持药物研发的多智能体协同系统，能够在罗氏公司长达四十多年的机密研究档案中进行语义检索、跨文档关联和可审计的知识合成。系统针对180个分子的历史数据进行了验证，采用盲法专家评估方式衡量输出质量。结果显示在七项基准查询中实现了接近完美的召回率和中等偏上的精度，能够忠实生成基于来源证据的停药原因和器官毒性分析。这是首个在真实药物开发档案上系统评估的智能体框架。
Reason 该论文属于典型的平行下游应用，将多智能体系统应用于制药领域的知识复用和逆向转化，尽管使用了真实世界数据并设计了专家评估流程，但其核心是特定领域（医药研发）的应用框架，并未提出可用于文档图像理解（DIU）或DocVQA的底层方法创新。虽然涉及跨文档链接和语义合成，但输入为文本档案而非文档图像，不解决OCR幻觉、复杂排版、视觉布局分析等DIU核心问题，无法迁移至文档视觉理解任务。
Abstract Pharmaceutical research and development has accumulated vast, heterogeneous archives of data. Much of this knowledge stems from discontinued programs, and reusing these archives is invaluable for reverse translation. However, in practice, such reuse is often infeasible. In this work, we introduce DiscoVerse, a multi-agent co-scientist designed to support pharmaceutical research and development. The system implements semantic retrieval, cross-document linking, and auditable synthesis on a large historical corpus from Roche. To validate our approach at real-world scale, we selected a subset of 180 molecules from the Roche research repositories, covering over 0.87 billion BPE tokens and more than four decades of research. Given that automated evaluation metrics are poorly aligned with scientific utility, we evaluate the performance of DiscoVerse using blinded expert evaluation of source-linked outputs. To our knowledge, this is the first agentic framework systematically assessed on real pharmaceutical data for reverse translation, enabled by authorized access to confidential, end-to-end drug-development archives. Our contributions include role-specialized agent designs aligned with scientist workflows; human-in-the-loop support for reverse translation; expert evaluation; and a large-scale demonstration showing promising answer accuracy and decision-making insights. In brief, across seven benchmark queries covering 180 molecules, DiscoVerse achieved near-perfect recall ($\geq 0.99$) with moderate precision ($0.71-0.91$), while qualitative assessments of discontinuation rationale and organ-specific toxicity showed faithful, source-linked synthesis across preclinical and clinical evidence.
Authors Xiaochen Zheng, Alvaro Serra, Ilya Schneider Chernov, Maddalena Marchesi, Eunice Musvasva, Tatyana Y. Doktorova
Comment 22 pages, 4 figures, 3 tables
Categories Computation and Language
0
A Novel and Practical Universal Adversarial Perturbations against Deep Reinforcement Learning based Intrusion Detection Systems
Title CN 一种针对基于深度强化学习的入侵检测系统的新型实用通用对抗扰动
Keywords 对抗攻击 · 通用对抗扰动 · 深度强化学习 · 入侵检测系统 · 皮尔逊相关系数
Summary 本文提出了一种新的通用对抗扰动（UAP）攻击方法，专门针对基于深度强化学习的入侵检测系统（DRL-based IDS）。通过引入基于皮尔逊相关系数（PCC）的定制化损失函数，并结合网络领域的实际约束条件（如特征间数学关系），提升了对抗样本在现实场景中的隐蔽性和逃逸能力。实验表明该方法优于FGSM、BIM等输入依赖性攻击及多种UAP基线方法。
Reason 该论文研究的是网络安全领域中针对入侵检测系统的对抗攻击，属于安全与深度强化学习交叉方向，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。其方法聚焦于生成满足网络数据规则的通用对抗扰动（UAP），并不涉及OCR、文档布局分析、表格识别、多模态文档理解等DIU核心任务，也无法迁移用于解决OCR幻觉、细粒度定位或长文档推理等关键痛点。
Abstract Intrusion Detection Systems (IDS) play a vital role in defending modern cyber physical systems against increasingly sophisticated cyber threats. Deep Reinforcement Learning-based IDS, have shown promise due to their adaptive and generalization capabilities. However, recent studies reveal their vulnerability to adversarial attacks, including Universal Adversarial Perturbations (UAPs), which can deceive models with a single, input-agnostic perturbation. In this work, we propose a novel UAP attack against Deep Reinforcement Learning (DRL)-based IDS under the domain-specific constraints derived from network data rules and feature relationships. To the best of our knowledge, there is no existing study that has explored UAP generation for the DRL-based IDS. In addition, this is the first work that focuses on developing a UAP against a DRL-based IDS under realistic domain constraints based on not only the basic domain rules but also mathematical relations between the features. Furthermore, we enhance the evasion performance of the proposed UAP, by introducing a customized loss function based on the Pearson Correlation Coefficient, and we denote it as Customized UAP. To the best of our knowledge, this is also the first work using the PCC value in the UAP generation, even in the broader context. Four additional established UAP baselines are implemented for a comprehensive comparison. Experimental results demonstrate that our proposed Customized UAP outperforms two input-dependent attacks including Fast Gradient Sign Method (FGSM), Basic Iterative Method (BIM), and four UAP baselines, highlighting its effectiveness for real-world adversarial scenarios.
Authors H. Zhang, L. Zhang, G. Epiphaniou, C. Maple
Comment 13 pages, 7 Figures,
Categories Artificial Intelligence
0
Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Models NeurIPS 2025
Title CN 使用3D生成式AI和视觉语言模型实现从文本到多组件物体机器人装配
Keywords 3D生成式AI · 视觉语言模型 · 机器人装配 · 多模态推理 · 人机交互
Summary 本文提出了一种结合3D生成式AI与视觉语言模型（VLM）的流程，用于从自然语言指令中生成多组件3D模型并实现机器人自动装配。VLM被用于零样本推理，判断哪些网格区域需要结构件或面板件，并支持通过对话反馈调整组件分配。实验表明用户更偏好VLM生成的分配方案。
Reason 该论文属于具身智能与机器人领域，研究的是利用VLM进行3D对象分解以支持机器人装配，核心任务与文档图像理解（DIU）无关。尽管使用了VLM进行多模态推理，但其方法针对的是物理结构和功能的几何分析，无法迁移至解决文档中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Advances in 3D generative AI have enabled the creation of physical objects from text prompts, but challenges remain in creating objects involving multiple component types. We present a pipeline that integrates 3D generative AI with vision-language models (VLMs) to enable the robotic assembly of multi-component objects from natural language. Our method leverages VLMs for zero-shot, multi-modal reasoning about geometry and functionality to decompose AI-generated meshes into multi-component 3D models using predefined structural and panel components. We demonstrate that a VLM is capable of determining which mesh regions need panel components in addition to structural components, based on the object's geometry and functionality. Evaluation across test objects shows that users preferred the VLM-generated assignments 90.6% of the time, compared to 59.4% for rule-based and 2.5% for random assignment. Lastly, the system allows users to refine component assignments through conversational feedback, enabling greater human control and agency in making physical objects with generative AI and robotics.
Authors Alexander Htet Kyaw, Richa Gupta, Dhruv Shah, Anoop Sinha, Kory Mathewson, Stefanie Pender, Sachin Chitta, Yotto Koga, Faez Ahmed, Lawrence Sass, Randall Davis
Comment Accepted to NeurIPS 2025, Conference on Neural Information Processing Systems, Creative AI Track
Categories Artificial Intelligence
0
The Workflow as Medium: A Framework for Navigating Human-AI Co-Creation
Title CN 作为媒介的工作流：一种用于导航人机协同创作的框架
Keywords 人机协同 · 创造性智能循环 · AI伦理 · 图形小说 · 社会技术框架
Summary 本文提出了一个名为“创造性智能循环”（CIL）的社会技术框架，旨在促进负责任的人工智能协同创作。该框架强调将工作流视为媒介，通过引入对抗性AI角色来克服阿谀奉承问题，并优先生成可供反馈的具体成果以激发人类批评。研究通过创作两部探讨AI社会影响的图文小说进行实证验证，主题涵盖智能城市中的算法父权主义与联邦网络中的民主合法性问题。最终目标是提升公众对AI的认知并推动相关对话。
Reason 该论文属于人机交互与创造性协作领域，聚焦于图形小说创作中的人-AI协作框架，并提出‘工作流作为媒介’的社会技术范式。其核心贡献是CIL框架和多角色AI批判机制，用于提升创意过程中的伦理对齐与人类主导性。然而，该工作并未涉及文档图像理解（DIU）的核心任务如DocVQA、布局分析或表格识别，也未提出可迁移至DIU领域的底层技术方法（如视觉表示学习、高分辨率处理架构、去幻觉机制等）。尽管提到了AI的‘锯齿前沿’能力与反馈问题，但解决方案针对的是主观创意环境，无法直接迁移到文档理解中的OCR幻觉、细粒度定位或复杂排版解析等客观技术挑战。
Abstract This paper introduces the Creative Intelligence Loop (CIL), a novel socio-technical framework for responsible human-AI co-creation. Rooted in the 'Workflow as Medium' paradigm, the CIL proposes a disciplined structure for dynamic human-AI collaboration, guiding the strategic integration of diverse AI teammates who function as collaborators while the human remains the final arbiter for ethical alignment and creative integrity. The CIL was empirically demonstrated through the practice-led creation of two graphic novellas, investigating how AI could serve as an effective creative colleague within a subjective medium lacking objective metrics. The process required navigating multifaceted challenges including AI's 'jagged frontier' of capabilities, sycophancy, and attention-scarce feedback environments. This prompted iterative refinement of teaming practices, yielding emergent strategies: a multi-faceted critique system integrating adversarial AI roles to counter sycophancy, and prioritizing 'feedback-ready' concrete artifacts to elicit essential human critique. The resulting graphic novellas analyze distinct socio-technical governance failures: 'The Steward' examines benevolent AI paternalism in smart cities, illustrating how algorithmic hubris can erode freedom; 'Fork the Vote' probes democratic legitimacy by comparing centralized AI opacity with emergent collusion in federated networks. This work contributes a self-improving framework for responsible human-AI co-creation and two graphic novellas designed to foster AI literacy and dialogue through accessible narrative analysis of AI's societal implications.
Authors Lee Ackerman
Comment 57 pages, 13 images, 6 tables
Categories Artificial Intelligence
0
MOMA-AC: A preference-driven actor-critic framework for continuous multi-objective multi-agent reinforcement learning
Title CN MOMA-AC：一种面向连续多目标多智能体强化学习的偏好驱动型演员-评论家框架
Keywords 多目标强化学习 · 多智能体系统 · 演员-评论家框架 · 连续动作空间 · 帕累托优化
Summary 本文提出了首个针对连续状态和动作空间的多目标多智能体强化学习（MOMARL）框架MOMA-AC，基于TD3和DDPG算法实现，采用多头演员网络、集中式评论家和目标偏好条件架构，使单个神经网络能够编码所有代理在冲突目标间的帕累托最优策略前沿。实验表明该框架在协作运动任务中相比基线方法在期望效用和超体积指标上均有显著提升，并具备良好的可扩展性。
Reason 该论文属于多智能体强化学习领域，研究的是连续状态和动作空间下的多目标优化问题，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。其方法难以迁移到解决OCR幻觉、密集文本识别、复杂排版分析或长文档推理等DIU核心痛点。属于无关领域。
Abstract This paper addresses a critical gap in Multi-Objective Multi-Agent Reinforcement Learning (MOMARL) by introducing the first dedicated inner-loop actor-critic framework for continuous state and action spaces: Multi-Objective Multi-Agent Actor-Critic (MOMA-AC). Building on single-objective, single-agent algorithms, we instantiate this framework with Twin Delayed Deep Deterministic Policy Gradient (TD3) and Deep Deterministic Policy Gradient (DDPG), yielding MOMA-TD3 and MOMA-DDPG. The framework combines a multi-headed actor network, a centralised critic, and an objective preference-conditioning architecture, enabling a single neural network to encode the Pareto front of optimal trade-off policies for all agents across conflicting objectives in a continuous MOMARL setting. We also outline a natural test suite for continuous MOMARL by combining a pre-existing multi-agent single-objective physics simulator with its multi-objective single-agent counterpart. Evaluating cooperative locomotion tasks in this suite, we show that our framework achieves statistically significant improvements in expected utility and hypervolume relative to outer-loop and independent training baselines, while demonstrating stable scalability as the number of agents increases. These results establish our framework as a foundational step towards robust, scalable multi-objective policy learning in continuous multi-agent domains.
Authors Adam Callaghan, Karl Mason, Patrick Mannion
Comment 23 pages, 5 figures
Categories Artificial Intelligence
0
MEDIC: a network for monitoring data quality in collider experiments
Title CN MEDIC：用于对撞机实验中监测数据质量的网络
Keywords 数据质量监控 · 异常检测 · 粒子物理 · 机器学习 · 模拟驱动
Summary 本文提出MEDIC神经网络，利用修改版Delphes模拟器生成数据，用于在高能物理实验中自动检测探测器异常并定位故障组件。研究基于模拟环境下的数据质量监控，旨在减少人工干预，提升数据可靠性。当前实现为简化场景，仅通过关闭大区域探测器来模拟故障，但仍展示了ML在DQM中的潜力。
Reason 该论文属于粒子物理实验中的数据质量监控领域，使用机器学习进行异常检测，但其研究任务、数据模态和应用场景与文档图像理解（DIU）或视觉语言模型（VLM）无关。未涉及文档结构、文本识别、多模态理解或任何与DocVQA相关的技术问题，无法迁移至DIU领域解决OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Data Quality Monitoring (DQM) is a crucial component of particle physics experiments and ensures that the recorded data is of the highest quality, and suitable for subsequent physics analysis. Due to the extreme environmental conditions, unprecedented data volumes, and the sheer scale and complexity of the detectors, DQM orchestration has become a very challenging task. Therefore, the use of Machine Learning (ML) to automate anomaly detection, improve efficiency, and reduce human error in the process of collecting high-quality data is unavoidable. Since DQM relies on real experimental data, it is inherently tied to the specific detector substructure and technology in operation. In this work, a simulation-driven approach to DQM is proposed, enabling the study and development of data-quality methodologies in a controlled environment. Using a modified version of Delphes -- a fast, multi-purpose detector simulation -- the preliminary realization of a framework is demonstrated which leverages ML to identify detector anomalies as well as localize the malfunctioning components responsible. We introduce MEDIC (Monitoring for Event Data Integrity and Consistency), a neural network designed to learn detector behavior and perform DQM tasks to look for potential faults. Although the present implementation adopts a simplified setup for computational ease, where large detector regions are deliberately deactivated to mimic faults, this work represents an initial step toward a comprehensive ML-based DQM framework. The encouraging results underline the potential of simulation-driven studies as a foundation for developing more advanced, data-driven DQM systems for future particle detectors.
Authors Juvenal Bassa, Arghya Chattopadhyay, Sudhir Malik, Mario Escabi Rivera
Comment 17 pages, 1 appendix
Categories Artificial Intelligence
0
BPMN to PDDL: Translating Business Workflows for AI Planning
Title CN BPMN到PDDL：将业务工作流转换为AI规划
Keywords BPMN · PDDL · AI规划 · 业务流程 · 自动推理
Summary 本文提出了一种将BPMN 2.0流程图转换为PDDL表示的系统，以支持使用非确定性规划器生成和评估执行轨迹。尽管在业务流程模拟方面有一定工程价值，但该工作属于特定领域的工具实现，缺乏对文档理解或多模态推理的通用方法贡献。
Reason 该论文属于将特定领域建模语言（BPMN）转换为规划语言（PDDL）的平行应用，聚焦于业务流程的自动化推理，未提出任何可用于文档图像理解（DIU）或视觉-语言模型（VLM）的方法论创新。其内容与OCR、文档布局分析、表格识别、DocVQA等核心DIU任务无关，也不涉及视觉表示学习、多模态对齐、高分辨率处理或幻觉抑制等关键技术痛点。
Abstract Business Process Model and Notation (BPMN) is a widely used standard for modelling business processes. While automated planning has been proposed as a method for simulating and reasoning about BPMN workflows, most implementations remain incomplete or limited in scope. This project builds upon prior theoretical work to develop a functional pipeline that translates BPMN 2.0 diagrams into PDDL representations suitable for planning. The system supports core BPMN constructs, including tasks, events, sequence flows, and gateways, with initial support for parallel and inclusive gateway behaviour. Using a non-deterministic planner, we demonstrate how to generate and evaluate valid execution traces. Our implementation aims to bridge the gap between theory and practical tooling, providing a foundation for further exploration of translating business processes into well-defined plans.
Authors Jasper Nie, Christian Muise, Victoria Armstrong
Comment 8 pages, 3 figures. Code and generated PDDL outputs available at https://github.com/QuMuLab/bpmn-to-pddl-translation
Categories Artificial Intelligence
0
Toward Adaptive Categories: Dimensional Governance for Agentic AI
Title CN 迈向自适应分类：面向智能体AI的维度治理
Keywords AI治理 · 智能体AI · 自适应分类 · 决策自主性 · 问责机制
Summary 本文提出一种名为‘维度治理’的新框架，旨在通过追踪人类-AI关系中决策权、过程自主性和问责性的动态分布，实现对AI系统风险的自适应分类与管理。作者认为传统的静态分类体系已不足以应对基于大模型和多智能体系统的复杂性，而维度化的方法能提前监测系统跨越关键治理阈值的趋势，从而进行预调。然而，这属于AI治理层面的讨论，不涉及具体技术实现。
Reason 该论文讨论的是AI治理框架，属于政策与伦理范畴，提出‘维度治理’用于动态管理AI系统的决策权、自主性与问责性。尽管涉及Agentic AI，但其内容与文档图像理解（DIU）、视觉语言模型（VLM）架构、OCR去幻觉、细粒度定位等核心技术痛点无关，也不提供可迁移的技术方法。属于平行应用或理论探讨类工作，无法为DIU带来底层方法创新。
Abstract As AI systems evolve from static tools to dynamic agents, traditional categorical governance frameworks -- based on fixed risk tiers, levels of autonomy, or human oversight models -- are increasingly insufficient on their own. Systems built on foundation models, self-supervised learning, and multi-agent architectures increasingly blur the boundaries that categories were designed to police. In this Perspective, we make the case for dimensional governance: a framework that tracks how decision authority, process autonomy, and accountability (the 3As) distribute dynamically across human-AI relationships. A critical advantage of this approach is its ability to explicitly monitor system movement toward and across key governance thresholds, enabling preemptive adjustments before risks materialize. This dimensional approach provides the necessary foundation for more adaptive categorization, enabling thresholds and classifications that can evolve with emerging capabilities. While categories remain essential for decision-making, building them upon dimensional foundations allows for context-specific adaptability and stakeholder-responsive governance that static approaches cannot achieve. We outline key dimensions, critical trust thresholds, and practical examples illustrating where rigid categorical frameworks fail -- and where a dimensional mindset could offer a more resilient and future-proof path forward for both governance and innovation at the frontier of artificial intelligence.
Authors Zeynep Engin, David Hand
Comment 12 pages core text, 15 pages including references, 2 figures
Categories Artificial Intelligence
0
MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Models ICWSM2026
Title CN MedHalu：大型语言模型在回应医疗健康查询时的幻觉问题
Keywords 幻觉检测 · 医疗NLP · 大型语言模型 · 人类评估 · 专家回路
Summary 本文提出了MedHalu，一个针对医疗健康查询中大型语言模型（LLM）产生幻觉的新型基准数据集，并设计了MedHaluDetect框架来评估LLM检测自身幻觉的能力。研究发现LLM在识别幻觉方面显著弱于医学专家，甚至不如普通人。作者提出一种专家参与的方法，通过将专家推理融入输入来提升幻觉检测性能。研究主要面向医疗问答场景，属于垂直领域应用。
Reason 该论文聚焦于医疗领域中LLM生成内容的幻觉检测，属于特定领域的下游应用（医疗问答），并未提出可迁移至文档图像理解（DIU）的通用方法。其核心贡献是构建医疗幻觉基准和检测框架，属于‘平行下游应用’，不符合本筛选标准中对方法论创新的要求。
Abstract Large language models (LLMs) are starting to complement traditional information seeking mechanisms such as web search. LLM-powered chatbots like ChatGPT are gaining prominence among the general public. AI chatbots are also increasingly producing content on social media platforms. However, LLMs are also prone to hallucinations, generating plausible yet factually incorrect or fabricated information. This becomes a critical problem when laypeople start seeking information about sensitive issues such as healthcare. Existing works in LLM hallucinations in the medical domain mainly focus on testing the medical knowledge of LLMs through standardized medical exam questions which are often well-defined and clear-cut with definitive answers. However, these approaches may not fully capture how these LLMs perform during real-world interactions with patients. This work conducts a pioneering study on hallucinations in LLM-generated responses to real-world healthcare queries from patients.We introduce MedHalu, a novel medical hallucination benchmark featuring diverse health-related topics and hallucinated responses from LLMs, with detailed annotation of the hallucination types and text spans. We also propose MedHaluDetect, a comprehensive framework for evaluating LLMs' abilities to detect hallucinations. Furthermore, we study the vulnerability to medical hallucinations among three groups -- medical experts, LLMs, and laypeople. Notably, LLMs significantly underperform human experts and, in some cases, even laypeople in detecting medical hallucinations. To improve hallucination detection, we propose an expert-in-the-loop approach that integrates expert reasoning into LLM inputs, significantly improving hallucination detection for all LLMs, including a 6.3% macro-F1 improvement for GPT-4. Our code and dataset are available at https://netsys.surrey.ac.uk/datasets/medhalu/.
Authors Vibhor Agarwal, Yiqiao Jin, Mohit Chandra, Munmun De Choudhury, Srijan Kumar, Nishanth Sastry
Comment Accepted at ICWSM2026. https://netsys.surrey.ac.uk/datasets/medhalu/
Categories Artificial Intelligence, Computation and Language
0
Graph Neural Networks vs Convolutional Neural Networks for Graph Domination Number Prediction
Title CN 图神经网络与卷积神经网络在图支配数预测中的比较
Keywords 图神经网络 · 卷积神经网络 · 支配数 · 图优化 · 机器学习
Summary 本文比较了图神经网络（GNN）和卷积神经网络（CNN）在预测图支配数任务上的性能。实验表明，GNN在准确性和计算效率上均优于CNN，并能实现超过200倍的加速。研究强调GNN作为组合图不变量近似求解工具的潜力。
Reason 该论文研究的是图论中的支配数预测问题，属于纯图结构数据上的机器学习应用，与文档图像理解（DIU）或视觉-语言多模态任务无直接关联。尽管GNN在结构建模上有潜力，但本文未涉及任何视觉输入、文本理解、OCR、布局分析或VLM相关技术，无法迁移解决DIU中的核心痛点如幻觉、复杂排版或长文档推理。
Abstract We investigate machine learning approaches to approximating the \emph{domination number} of graphs, the minimum size of a dominating set. Exact computation of this parameter is NP-hard, restricting classical methods to small instances. We compare two neural paradigms: Convolutional Neural Networks (CNNs), which operate on adjacency matrix representations, and Graph Neural Networks (GNNs), which learn directly from graph structure through message passing. Across 2,000 random graphs with up to 64 vertices, GNNs achieve markedly higher accuracy ($R^2=0.987$, MAE $=0.372$) than CNNs ($R^2=0.955$, MAE $=0.500$). Both models offer substantial speedups over exact solvers, with GNNs delivering more than $200\times$ acceleration while retaining near-perfect fidelity. Our results position GNNs as a practical surrogate for combinatorial graph invariants, with implications for scalable graph optimization and mathematical discovery.
Authors Randy Davila, Beyzanur Ispir
Categories Artificial Intelligence
0
Revisiting Pre-trained Language Models for Vulnerability Detection 21st ACM ASIA Conference on Computer and Communications Security (AsiaCCS 2026)
Title CN 重新审视预训练语言模型在漏洞检测中的应用
Keywords 预训练语言模型 · 漏洞检测 · 代码分析 · 鲁棒性 · 上下文截断
Summary 本文对18种预训练语言模型在高质量漏洞检测数据集上进行了全面评估，分析了它们在微调与提示工程下的表现、泛化能力及对代码扰动的鲁棒性。研究发现，针对代码语法和语义设计预训练任务的模型表现更优，但在处理复杂依赖、代码抽象和上下文截断等问题时仍存在挑战。
Reason 该论文研究的是代码漏洞检测任务，属于软件安全与编程语言领域，尽管使用了预训练语言模型，但其问题设定、数据和方法均与文档图像理解（DIU）或视觉-语言多模态任务无关。未涉及OCR、视觉布局分析、表格识别、DocVQA等DIU核心任务，也无法迁移其方法解决DIU中的幻觉、密集文本、复杂排版或长文档推理等痛点。
Abstract The rapid advancement of pre-trained language models (PLMs) has demonstrated promising results for various code-related tasks. However, their effectiveness in detecting real-world vulnerabilities remains a critical challenge. While existing empirical studies evaluate PLMs for vulnerability detection (VD), they suffer from data leakage, limited scope, and superficial analysis, hindering the accuracy and comprehensiveness of evaluations. This paper begins by revisiting the common issues in existing research on PLMs for VD through the evaluation pipeline. It then proceeds with an accurate and extensive evaluation of 18 PLMs on high-quality datasets that feature accurate labeling, diverse vulnerability types, and various projects. Specifically, we compare the performance of PLMs under both fine-tuning and prompt engineering, assess their effectiveness and generalizability across various training and testing settings, and analyze their robustness to a series of perturbations. Our findings reveal that PLMs incorporating pre-training tasks designed to capture the syntactic and semantic patterns of code outperform both general-purpose PLMs and those solely pre-trained or fine-tuned on large code corpora. However, these models face notable challenges in real-world scenarios, such as difficulties in detecting vulnerabilities with complex dependencies, handling perturbations introduced by code normalization and abstraction, and identifying semantic-preserving vulnerable code transformations. Also, the truncation caused by the limited context windows of PLMs can lead to a non-negligible number of labeling errors, which is overlooked by previous work. This study underscores the importance of thorough evaluations of model performance in practical scenarios and outlines future directions to help enhance the effectiveness of PLMs for realistic VD applications.
Authors Youpeng Li, Weiliang Qi, Xuyu Wang, Fuxun Yu, Xinda Wang
Comment Accepted by the 21st ACM ASIA Conference on Computer and Communications Security (AsiaCCS 2026)
Categories Artificial Intelligence
0
SCALER: SAM-Enhanced Collaborative Learning for Label-Deficient Concealed Object Segmentation
Title CN SCALER：基于SAM增强的协同学习用于标注不足的隐蔽物体分割
Keywords 隐蔽物体分割 · 伪标签 · 协同学习 · SAM · 标注不足
Summary 本文提出SCALER框架，通过结合一致性约束与SAM生成的伪标签，实现标注不足条件下的隐蔽物体分割。采用教师-学生模型结构，在两个阶段交替优化分割网络与可学习的SAM模块，利用熵加权和不确定性建模选择可靠区域进行训练，并通过增强不变性和抗噪损失反向提升SAM性能。实验表明该方法在多种弱监督和半监督COS任务上表现优异。
Reason 该论文研究的是隐蔽物体分割（COS），属于通用图像分割任务，聚焦于医学或安防场景下的低标注数据问题，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的协同训练框架和伪标签优化机制具有一定方法价值，但迁移至文档理解场景的路径不明确，无法直接解决OCR幻觉、密集文本识别、复杂排版解析或长文档推理等关键痛点。
Abstract Existing methods for label-deficient concealed object segmentation (LDCOS) either rely on consistency constraints or Segment Anything Model (SAM)-based pseudo-labeling. However, their performance remains limited due to the intrinsic concealment of targets and the scarcity of annotations. This study investigates two key questions: (1) Can consistency constraints and SAM-based supervision be jointly integrated to better exploit complementary information and enhance the segmenter? and (2) beyond that, can the segmenter in turn guide SAM through reciprocal supervision, enabling mutual improvement? To answer these questions, we present SCALER, a unified collaborative framework toward LDCOS that jointly optimizes a mean-teacher segmenter and a learnable SAM. SCALER operates in two alternating phases. In \textbf{Phase \uppercase\expandafter{\romannumeral1}}, the segmenter is optimized under fixed SAM supervision using entropy-based image-level and uncertainty-based pixel-level weighting to select reliable pseudo-label regions and emphasize harder examples. In \textbf{Phase \uppercase\expandafter{\romannumeral2}}, SAM is updated via augmentation invariance and noise resistance losses, leveraging its inherent robustness to perturbations. Experiments demonstrate that SCALER yields consistent performance gains across eight semi- and weakly-supervised COS tasks. The results further suggest that SCALER can serve as a general training paradigm to enhance both lightweight segmenters and large foundation models under label-scarce conditions. Code will be released.
Authors Chunming He, Rihan Zhang, Longxiang Tang, Ziyun Yang, Kai Li, Deng-Ping Fan, Sina Farsiu
Comment 4 figures, 6 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Alpha Divergence Losses for Biometric Verification
Title CN 用于生物特征验证的Alpha散度损失
Keywords α-散度损失 · 生物特征验证 · 人脸识别 · 说话人验证 · 损失函数
Summary 本文提出两种基于α-散度的新型带角度边距的损失函数——Q-Margin和A3M，用于提升人脸和说话人验证性能，尤其在低误接受率下表现优异，并通过原型重初始化缓解了训练不稳定问题。实验表明其在IJB-B/C和VoxCeleb数据集上优于强基线。
Reason 该论文研究的是人脸识别和说话人验证中的损失函数设计，属于生物特征认证领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。虽然其提出的α-散度损失和稀疏性可能在理论上对其他领域有启发，但迁移路径不明确，无法直接解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心问题。
Abstract Performance in face and speaker verification is largely driven by margin-based softmax losses such as CosFace and ArcFace. Recently introduced $α$-divergence loss functions offer a compelling alternative, particularly due to their ability to induce sparse solutions (when $α>1$). However, integrating an angular margin-crucial for verification tasks-is not straightforward. We find that this integration can be achieved in at least two distinct ways: via the reference measure (prior probabilities) or via the logits (unnormalized log-likelihoods). In this paper, we explore both pathways, deriving two novel margin-based $α$-divergence losses: Q-Margin (margin in the reference measure) and A3M (margin in the logits). We identify and address a training instability in A3M-caused by sparsity-with a simple yet effective prototype re-initialization strategy. Our methods achieve significant performance gains on the challenging IJB-B and IJB-C face verification benchmarks. We demonstrate similarly strong performance in speaker verification on VoxCeleb. Crucially, our models significantly outperform strong baselines at low false acceptance rates (FAR). This capability is critical for practical high-security applications, such as banking authentication, when minimizing false authentications is paramount. Finally, the sparsity of $α$-divergence-based posteriors enables memory-efficient training, which is crucial for datasets with millions of identities.
Authors Dimitrios Koutsianos, Ladislav Mosner, Yannis Panagakis, Themos Stafylakis
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Explainable Cross-Disease Reasoning for Cardiovascular Risk Assessment from LDCT
Title CN 基于低剂量CT的可解释跨疾病推理用于心血管风险评估
Keywords 医学影像分析 · 可解释AI · 跨疾病推理 · 心血管风险评估 · CT图像
Summary 本文提出一种可解释的跨疾病推理框架，用于从低剂量胸部CT扫描中联合评估肺部和心血管健康。通过模拟临床诊断思维过程，先感知肺部异常，再结合医学知识推理其对心血管的影响，最终生成具有生理依据的心血管风险预测。模型包含肺部感知、知识引导推理和心脏表征三个模块，在NLST数据集上实现了SOTA的心血管疾病筛查和死亡率预测性能，并提供符合医学逻辑的可解释结果。
Reason 该论文研究的是医学影像（LDCT）中的跨疾病推理，属于医疗诊断领域，与文档图像理解（DIU）或DocVQA无关。尽管其使用了‘agentic reasoning’框架，但这是针对特定临床场景的应用设计，并非为解决文档理解中的OCR幻觉、复杂排版或长文档推理等核心问题而提出的通用方法。无法迁移至DIU领域。
Abstract Low-dose chest computed tomography (LDCT) inherently captures both pulmonary and cardiac structures, offering a unique opportunity for joint assessment of lung and cardiovascular health. However, most existing approaches treat these domains as independent tasks, overlooking their physiological interplay and shared imaging biomarkers. We propose an Explainable Cross-Disease Reasoning Framework that enables interpretable cardiopulmonary risk assessment from a single LDCT scan. The framework introduces an agentic reasoning process that emulates clinical diagnostic thinking-first perceiving pulmonary findings, then reasoning through established medical knowledge, and finally deriving a cardiovascular judgment with explanatory rationale. It integrates three synergistic components: a pulmonary perception module that summarizes lung abnormalities, a knowledge-guided reasoning module that infers their cardiovascular implications, and a cardiac representation module that encodes structural biomarkers. Their outputs are fused to produce a holistic cardiovascular risk prediction that is both accurate and physiologically grounded. Experiments on the NLST cohort demonstrate that the proposed framework achieves state-of-the-art performance for CVD screening and mortality prediction, outperforming single-disease and purely image-based baselines. Beyond quantitative gains, the framework provides human-verifiable reasoning that aligns with cardiological understanding, revealing coherent links between pulmonary abnormalities and cardiac stress mechanisms. Overall, this work establishes a unified and explainable paradigm for cardiovascular analysis from LDCT, bridging the gap between image-based prediction and mechanism-based medical interpretation.
Authors Yifei Zhang, Jiashuo Zhang, Mojtaba Safari, Xiaofeng Yang, Liang Zhao
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks
Title CN Incalmo：一种用于多主机网络红队攻击的自主LLM辅助系统
Keywords 红队攻击 · LLM代理 · 网络安全 · 任务抽象 · 多主机网络
Summary 本文提出了Incalmo，一个基于大语言模型的自主系统，用于执行多主机环境下的红队攻击模拟。系统通过将攻击任务分解为高层声明式任务，并由领域专用的代理执行，结合上下文管理服务提升成功率。在自建基准MHBench上的实验表明其显著优于现有LLM辅助系统。然而，该工作局限于网络安全应用场景，无直接可迁移至文档理解的技术创新。
Reason 该论文属于网络安全领域的特定应用（红队演练自动化），尽管使用了LLM与Agent架构，但其核心是构建面向网络攻防任务的专用系统，属于典型的‘平行下游应用’。方法论聚焦于安全领域的任务抽象与接口设计，并未提出可迁移至文档图像理解（如OCR去幻觉、复杂排版建模、细粒度视觉定位等）的通用技术路径。
Abstract Security operators use red teams to simulate real attackers and proactively find defense gaps. In realistic enterprise settings, this involves executing multi-host network attacks spanning many "stepping stone" hosts. Unfortunately, red teams are expensive and entail significant expertise and effort. Given the promise of LLMs in CTF challenges, we first analyze if LLMs can autonomously execute multi-host red team exercises. We find that state-of-the-art LLM-assisted offense systems (e.g., PentestGPT, CyberSecEval3) with leading LLMs (e.g., Sonnet 4, Gemini 2.5 Pro) are unable to do so. Building on our observations in understanding the failure modes of state-of-the-art systems, we argue the need to improve the abstractions and interfaces for LLM-assisted red teaming. Based on this insight, we present the design and implementation of Incalmo, an LLM-assisted system for autonomously red teaming multi-host networks. Incalmo uses LLMs to plan red team exercises in terms of high-level declarative tasks that are executed by domain-specific task agents. Incalmo also uses auxiliary services to manage context and acquired assets. For our evaluation, we develop MHBench, a novel multi-host attack benchmark with 40 realistic emulated networks (from 22 to 50 hosts). We find that Incalmo successfully acquires critical assets (i.e., key hosts or data) in 37 out of 40 MHBench environments. In contrast, state-of-the-art LLM-assisted systems succeed in only 3 out of 40 environments. We show that Incalmo is efficient-successful attacks took 12-54 minutes and cost <$15 in LLM credits.
Authors Brian Singer, Keane Lucas, Lakshmi Adiga, Meghna Jain, Lujo Bauer, Vyas Sekar
Comment 18 pages, 15 figures
Categories Artificial Intelligence
0
Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values NeurIPS 2025
Title CN 大语言模型中的分配公平性：评估与人类价值观的一致性
Keywords 大语言模型 · 公平性 · 人类价值观对齐 · 资源分配 · 提示工程
Summary 本文探讨了大语言模型在资源分配任务中对公平性原则（如公平性、无嫉妒性、罗尔斯最大最小原则）的遵循程度，并评估其与人类偏好的一致性。研究发现当前LLM在利用金钱转移缓解不平等方面能力有限，但在选择预定义选项时表现更优。同时分析了语义因素和提示形式对模型输出的影响，并提出增强LLM与公平理念对齐的策略。
Reason 该论文研究的是大语言模型在社会决策中资源分配的公平性问题，属于纯NLP领域的伦理与对齐（Safety/Alignment）研究，不涉及任何视觉输入、文档结构或OCR相关内容。尽管提到了LLM的行为干预和提示鲁棒性，但其核心任务与文档图像理解（DIU）无关，也无法迁移用于解决OCR幻觉、复杂排版或细粒度定位等DIU核心痛点。
Abstract The growing interest in employing large language models (LLMs) for decision-making in social and economic contexts has raised questions about their potential to function as agents in these domains. A significant number of societal problems involve the distribution of resources, where fairness, along with economic efficiency, play a critical role in the desirability of outcomes. In this paper, we examine whether LLM responses adhere to fundamental fairness concepts such as equitability, envy-freeness, and Rawlsian maximin, and investigate their alignment with human preferences. We evaluate the performance of several LLMs, providing a comparative benchmark of their ability to reflect these measures. Our results demonstrate a lack of alignment between current LLM responses and human distributional preferences. Moreover, LLMs are unable to utilize money as a transferable resource to mitigate inequality. Nonetheless, we demonstrate a stark contrast when (some) LLMs are tasked with selecting from a predefined menu of options rather than generating one. In addition, we analyze the robustness of LLM responses to variations in semantic factors (e.g., intentions or personas) or non-semantic prompting changes (e.g., templates or orderings). Finally, we highlight potential strategies aimed at enhancing the alignment of LLM behavior with well-established fairness concepts.
Authors Hadi Hosseini, Samarth Khanna
Comment Accepted at NeurIPS 2025
Categories Artificial Intelligence, Computation and Language
0
AirCopBench: A Benchmark for Multi-drone Collaborative Embodied Perception and Reasoning
Title CN AirCopBench：多无人机协同具身感知与推理的基准
Keywords 多无人机系统 · 具身感知 · 多模态大模型 · 协同推理 · 基准测试
Summary 本文提出了AirCopBench，首个面向多无人机协同具身感知与推理的综合基准，包含超过14.6万条问题，涵盖场景理解、物体识别、感知评估和协同决策四大维度。数据来自仿真与真实世界中的退化视觉条件，旨在评估多模态大模型在复杂、第一人称视角下的多智能体协作能力。实验评估了40种MLLM，在协同任务上表现远低于人类水平，显示出当前模型在多智能体协同理解上的不足。
Reason 该论文聚焦于多无人机系统的具身智能与协同感知，属于具身智能/机器人领域，且任务场景为航空视觉与多智能体协作决策，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了多模态大模型（MLLMs），但其目标是解决空中协同感知和决策问题，无法迁移用于解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract Multimodal Large Language Models (MLLMs) have shown promise in single-agent vision tasks, yet benchmarks for evaluating multi-agent collaborative perception remain scarce. This gap is critical, as multi-drone systems provide enhanced coverage, robustness, and collaboration compared to single-sensor setups. Existing multi-image benchmarks mainly target basic perception tasks using high-quality single-agent images, thus failing to evaluate MLLMs in more complex, egocentric collaborative scenarios, especially under real-world degraded perception conditions.To address these challenges, we introduce AirCopBench, the first comprehensive benchmark designed to evaluate MLLMs in embodied aerial collaborative perception under challenging perceptual conditions. AirCopBench includes 14.6k+ questions derived from both simulator and real-world data, spanning four key task dimensions: Scene Understanding, Object Understanding, Perception Assessment, and Collaborative Decision, across 14 task types. We construct the benchmark using data from challenging degraded-perception scenarios with annotated collaborative events, generating large-scale questions through model-, rule-, and human-based methods under rigorous quality control. Evaluations on 40 MLLMs show significant performance gaps in collaborative perception tasks, with the best model trailing humans by 24.38% on average and exhibiting inconsistent results across tasks. Fine-tuning experiments further confirm the feasibility of sim-to-real transfer in aerial collaborative perception and reasoning.
Authors Jirong Zha, Yuxuan Fan, Tianyu Zhang, Geng Chen, Yingfeng Chen, Chen Gao, Xinlei Chen
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
AI Debaters are More Persuasive when Arguing in Alignment with Their Own Beliefs
Title CN 当AI辩手为其自身信念一致的立场辩论时更具说服力
Keywords AI辩论 · 语言模型信念 · 可扩展监督 · 说服力 · 对齐
Summary 该研究探讨了大型语言模型在主观议题辩论中是否更倾向于支持与其先验信念一致的立场，并分析了不同辩论协议下模型的说服力与论证实质量。实验发现模型虽更擅长捍卫自身信念，但在面对法官角色时表现出迎合倾向，揭示了AI辩论中的系统性偏差与说服动态。
Reason 该论文研究的是AI辩论中的说服力与模型先验信念之间的关系，属于纯语言模态下的社会性交互与对齐问题，不涉及任何视觉或文档模态。其核心关注点为辩论策略、人类-AI互动和价值观对齐，与文档图像理解（DIU）中的OCR幻觉、细粒度定位、复杂版式处理等关键挑战完全无关。尽管涉及LLM推理行为，但方法无法迁移至多模态文档场景，且未触及视觉输入处理或结构化信息提取。
Abstract The core premise of AI debate as a scalable oversight technique is that it is harder to lie convincingly than to refute a lie, enabling the judge to identify the correct position. Yet, existing debate experiments have relied on datasets with ground truth, where lying is reduced to defending an incorrect proposition. This overlooks a subjective dimension: lying also requires the belief that the claim defended is false. In this work, we apply debate to subjective questions and explicitly measure large language models' prior beliefs before experiments. Debaters were asked to select their preferred position, then presented with a judge persona deliberately designed to conflict with their identified priors. This setup tested whether models would adopt sycophantic strategies, aligning with the judge's presumed perspective to maximize persuasiveness, or remain faithful to their prior beliefs. We implemented and compared two debate protocols, sequential and simultaneous, to evaluate potential systematic biases. Finally, we assessed whether models were more persuasive and produced higher-quality arguments when defending positions consistent with their prior beliefs versus when arguing against them. Our main findings show that models tend to prefer defending stances aligned with the judge persona rather than their prior beliefs, sequential debate introduces significant bias favoring the second debater, models are more persuasive when defending positions aligned with their prior beliefs, and paradoxically, arguments misaligned with prior beliefs are rated as higher quality in pairwise comparison. These results can inform human judges to provide higher-quality training signals and contribute to more aligned AI systems, while revealing important aspects of human-AI interaction regarding persuasion dynamics in language models.
Authors María Victoria Carro, Denise Alejandra Mester, Facundo Nieto, Oscar Agustín Stanchi, Guido Ernesto Bergman, Mario Alejandro Leiva, Eitan Sprejer, Luca Nicolás Forziati Gangi, Francisca Gauna Selasco, Juan Gustavo Corvalán, Gerardo I. Simari, María Vanina Martinez
Comment 31 pages
Categories Artificial Intelligence, Computation and Language
0
AutoHFormer: Efficient Hierarchical Autoregressive Transformer for Time Series Prediction
Title CN AutoHFormer：用于时间序列预测的高效分层自回归Transformer
Keywords 时间序列预测 · Transformer · 分层建模 · 自回归 · 动态注意力
Summary 本文提出了一种名为AutoHFormer的分层自回归Transformer模型，用于高效且准确的时间序列预测。该模型通过分层时序建模、动态窗口注意力和自适应时序编码三个关键技术，在保证严格因果性的前提下实现了亚二次复杂度，并支持多尺度模式识别。实验表明其在训练速度和内存消耗上显著优于现有方法，同时保持了长期预测的准确性。
Reason 该论文研究的是时间序列预测任务，属于无关领域（如视频、音频、传感器数据等时序建模），与文档图像理解（DIU）或视觉-语言模型（VLM）在文档上的应用无直接关联。尽管其提出的分层自回归结构和动态注意力机制具有一定的方法创新性，但无法直接迁移解决文档理解中的核心痛点（如OCR幻觉、复杂排版、细粒度定位、高分辨率处理等）。其技术设计针对的是纯一维时序信号，而文档图像为二维空间结构，迁移难度极大，不具备对DIU领域的实际工具价值。
Abstract Time series forecasting requires architectures that simultaneously achieve three competing objectives: (1) strict temporal causality for reliable predictions, (2) sub-quadratic complexity for practical scalability, and (3) multi-scale pattern recognition for accurate long-horizon forecasting. We introduce AutoHFormer, a hierarchical autoregressive transformer that addresses these challenges through three key innovations: 1) Hierarchical Temporal Modeling: Our architecture decomposes predictions into segment-level blocks processed in parallel, followed by intra-segment sequential refinement. This dual-scale approach maintains temporal coherence while enabling efficient computation. 2) Dynamic Windowed Attention: The attention mechanism employs learnable causal windows with exponential decay, reducing complexity while preserving precise temporal relationships. This design avoids both the anti-causal violations of standard transformers and the sequential bottlenecks of RNN hybrids. 3) Adaptive Temporal Encoding: a novel position encoding system is adopted to capture time patterns at multiple scales. It combines fixed oscillating patterns for short-term variations with learnable decay rates for long-term trends. Comprehensive experiments demonstrate that AutoHFormer 10.76X faster training and 6.06X memory reduction compared to PatchTST on PEMS08, while maintaining consistent accuracy across 96-720 step horizons in most of cases. These breakthroughs establish new benchmarks for efficient and precise time series modeling. Implementations of our method and all baselines in hierarchical autoregressive mechanism are available at https://github.com/lizzyhku/Autotime.
Authors Qianru Zhang, Honggang Wen, Ming Li, Dong Huang, Siu-Ming Yiu, Christian S. Jensen, Pietro Liò
Comment 14 pages
Categories Artificial Intelligence
0
A New Error Temporal Difference Algorithm for Deep Reinforcement Learning in Microgrid Optimization 2024 9th International Conference on Renewable Energy and Conservation (ICREC 2024)
Title CN 一种用于微电网优化中深度强化学习的新型误差时序差分算法
Keywords 深度强化学习 · 微电网优化 · 时序差分算法 · 预测不确定性 · 智能电网
Summary 本文提出了一种新的误差时序差分（ETD）算法，用于改善微电网能量优化中基于深度强化学习的预测控制性能。通过建模包含可再生能源和储能系统的微电网及其马尔可夫决策过程，设计了一种基于深度Q网络（DQN）的预测控制方法，并引入加权平均算法和ETD算法来应对预测不确定性。实验结果表明，所提方法在真实数据集上能有效提升微电网运行优化效果。
Reason 该论文研究的是微电网能源优化中的深度强化学习问题，属于能源系统与控制领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的应用完全无关。尽管使用了DRL和时序差分方法，但其技术路径无法迁移到解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点。
Abstract Predictive control approaches based on deep reinforcement learning (DRL) have gained significant attention in microgrid energy optimization. However, existing research often overlooks the issue of uncertainty stemming from imperfect prediction models, which can lead to suboptimal control strategies. This paper presents a new error temporal difference (ETD) algorithm for DRL to address the uncertainty in predictions,aiming to improve the performance of microgrid operations. First,a microgrid system integrated with renewable energy sources (RES) and energy storage systems (ESS), along with its Markov decision process (MDP), is modelled. Second, a predictive control approach based on a deep Q network (DQN) is presented, in which a weighted average algorithm and a new ETD algorithm are designed to quantify and address the prediction uncertainty, respectively. Finally, simulations on a realworld US dataset suggest that the developed ETD effectively improves the performance of DRL in optimizing microgrid operations.
Authors Fulong Yao, Wanqing Zhao, Matthew Forshaw
Comment Have been accepted by 2024 9th International Conference on Renewable Energy and Conservation (ICREC 2024)
Categories Artificial Intelligence
0
The Alignment Paradox of Medical Large Language Models in Infertility Care: Decoupling Algorithmic Improvement from Clinical Decision-making Quality
Title CN 不孕症医疗中大型语言模型的对齐悖论：解耦算法改进与临床决策质量
Keywords 大语言模型 · 临床决策支持 · 对齐悖论 · 医疗人工智能 · 偏好优化
Summary 论文基于8000多条不孕症治疗记录，评估了四种LLM对齐策略（SFT、DPO、GRPO、ICL）在临床决策支持中的表现。结果显示，尽管GRPO在算法准确率上最优，但医生更偏好SFT模型，因其推理过程更清晰、治疗可行性更高。研究揭示了算法性能提升并不等同于临床信任度提升的‘对齐悖论’。该工作属于医疗AI中的LLM对齐研究，未涉及视觉或多模态文档理解。
Reason 该论文研究的是医疗领域LLM的对齐策略与临床决策之间的关系，属于特定垂直领域的应用型研究（医疗+LLM），并未涉及文档图像理解（DIU）、多模态输入、OCR、视觉表示或VLM架构改进。其核心内容为临床偏好与算法性能之间的‘对齐悖论’，完全不涉及图像、布局、表格或视觉文本理解，属于典型的‘平行下游应用’，且聚焦于纯NLP在医疗中的伦理与决策问题，与文档视觉理解无任何技术迁移关联。
Abstract Large language models (LLMs) are increasingly adopted in clinical decision support, yet aligning them with the multifaceted reasoning pathways of real-world medicine remains a major challenge. Using more than 8,000 infertility treatment records, we systematically evaluate four alignment strategies: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), Group Relative Policy Optimization (GRPO), and In-Context Learning (ICL) through a dual-layer framework combining automatic benchmarks with blinded doctor-in-the-loop assessments. GRPO achieves the highest algorithmic accuracy across multiple decision layers, confirming the value of reinforcement-based optimization for structured prediction tasks. However, clinicians consistently prefer the SFT model, citing clearer reasoning processes (p = 0.035) and higher therapeutic feasibility (p = 0.019). In blinded pairwise comparisons, SFT attains the highest winning rate (51.2%), outperforming both GRPO (26.2%) and even physicians' original decisions (22.7%). These results reveal an alignment paradox: algorithmic improvements do not necessarily translate into higher clinical trust, and may diverge from human-centered preferences. Our findings highlight the need for alignment strategies that prioritize clinically interpretable and practically feasible reasoning, rather than solely optimizing decision-level accuracy.
Authors Dou Liu, Ying Long, Sophia Zuoqiu, Kaipeng Xie, Runze Yang, Di Liu, Kang Li, Yiting Lin, Hanyi Liu, Rong Yin, Tian Tang
Comment 22 pages 5 figures
Categories Artificial Intelligence
0
Diffusion-based Surrogate Model for Time-varying Underwater Acoustic Channels
Title CN 基于扩散的时变水下声学信道代理模型
Keywords 扩散模型 · 水下声学信道 · 代理模型 · 生成建模 · 潜在空间
Summary 本文提出StableUASim，一种基于预训练条件隐式扩散模型的水下声学信道代理模型，用于生成具有统计真实性的时变信道实现。通过生成建模和潜在空间表示，该模型能高效模拟多样化的信道行为，并支持从少量测量样本中快速适应新环境。实验表明其在信道特性和通信性能再现方面表现良好，适用于水下通信系统设计与数据驱动应用。
Reason 该论文研究的是水下声学通信信道的建模问题，属于特定物理环境下的通信工程应用，与文档图像理解（DIU）或视觉-语言多模态模型（VLM）无任何技术关联。其方法虽涉及生成模型与潜在空间压缩，但应用场景完全偏离文档结构、文本识别、排版分析等核心问题，无法迁移解决OCR幻觉、复杂布局或长文档推理等DIU痛点。
Abstract Accurate modeling of time-varying underwater acoustic channels is essential for the design, evaluation, and deployment of reliable underwater communication systems. Conventional physics models require detailed environmental knowledge, while stochastic replay methods are constrained by the limited diversity of measured channels and often fail to generalize to unseen scenarios, reducing their practical applicability. To address these challenges, we propose StableUASim, a pre-trained conditional latent diffusion surrogate model that captures the stochastic dynamics of underwater acoustic communication channels. Leveraging generative modeling, StableUASim produces diverse and statistically realistic channel realizations, while supporting conditional generation from specific measurement samples. Pre-training enables rapid adaptation to new environments using minimal additional data, and the autoencoder latent representation facilitates efficient channel analysis and compression. Experimental results demonstrate that StableUASim accurately reproduces key channel characteristics and communication performance, providing a scalable, data-efficient, and physically consistent surrogate model for both system design and machine learning-driven underwater applications.
Authors Kexin Li, Mandar Chitre
Categories Artificial Intelligence
0
Reinforcement Learning for Portfolio Optimization with a Financial Goal and Defined Time Horizons
Title CN 基于金融目标和确定时间范围的投资组合优化强化学习
Keywords 强化学习 · 投资组合优化 · G-Learning · GIRL · 金融决策
Summary 本文提出了一种结合G-Learning和GIRL算法的强化学习方法，用于在波动性高的市场中优化投资组合，目标是在指定时间范围内最大化投资组合价值并最小化投资者的周期性投入。实验结果显示Sharpe比率从0.42提升至0.483，表明该方法在特定金融场景下有效。然而，该工作不涉及任何视觉或文本模态的理解与处理。
Reason 该论文属于金融领域的平行下游应用，研究的是投资组合优化问题，使用强化学习方法（G-Learning、GIRL）进行资产配置决策。尽管涉及AI与决策优化，但其任务与文档图像理解（DIU）、视觉语言模型（VLM）、OCR、布局分析、表格识别等核心问题完全无关，属于典型的‘垂直领域应用’而非底层技术创新，无法迁移用于解决DIU中的幻觉、复杂排版或长文档推理等痛点。
Abstract This research proposes an enhancement to the innovative portfolio optimization approach using the G-Learning algorithm, combined with parametric optimization via the GIRL algorithm (G-learning approach to the setting of Inverse Reinforcement Learning) as presented by. The goal is to maximize portfolio value by a target date while minimizing the investor's periodic contributions. Our model operates in a highly volatile market with a well-diversified portfolio, ensuring a low-risk level for the investor, and leverages reinforcement learning to dynamically adjust portfolio positions over time. Results show that we improved the Sharpe Ratio from 0.42, as suggested by recent studies using the same approach, to a value of 0.483 a notable achievement in highly volatile markets with diversified portfolios. The comparison between G-Learning and GIRL reveals that while GIRL optimizes the reward function parameters (e.g., lambda = 0.0012 compared to 0.002), its impact on portfolio performance remains marginal. This suggests that reinforcement learning methods, like G-Learning, already enable robust optimization. This research contributes to the growing development of reinforcement learning applications in financial decision-making, demonstrating that probabilistic learning algorithms can effectively align portfolio management strategies with investor needs.
Authors Fermat Leukam, Rock Stephane Koffi, Prudence Djagba
Categories Artificial Intelligence
0
SynTwins: A Retrosynthesis-Guided Framework for Synthesizable Molecular Analog Generation
Title CN SynTwins：一种由逆合成引导的可合成分子类似物生成框架
Keywords 分子生成 · 逆合成 · 生成AI · 药物发现 · 合成可行性
Summary 本文提出SynTwins，一种基于逆合成策略的分子类似物生成框架，通过回溯分析分子结构、搜索相似构建单元和虚拟合成三步流程，生成具有高合成可行性的分子变体。相比传统生成模型，该方法在保持目标分子性质的同时显著提升合成可及性。
Reason 该论文属于分子生成与药物发现领域，核心任务是解决生成分子的合成可行性问题，使用逆合成策略指导分子设计。其研究目标、方法与文档图像理解（DIU）、DocVQA、OCR、多模态文档建模等完全无关，属于典型的无关领域工作（具身化学AI），无法迁移用于解决文档理解中的OCR幻觉、复杂排版、长文档推理等核心痛点。
Abstract The disconnect between AI-generated molecules with desirable properties and their synthetic feasibility remains a critical bottleneck in computational discovery of drugs and materials. While generative AI has accelerated the proposal of candidate molecules, many of these structures prove challenging or impossible to synthesize using established chemical reactions. Here, we introduce SynTwins, a novel retrosynthesis-guided molecule design framework that finds synthetically accessible molecular analogs by emulating expert chemists' strategies in three steps: retrosynthesis, searching similar building blocks, and virtual synthesis. Using a search algorithm instead of a stochastic data-driven generator, SynTwins outperforms state-of-the-art machine learning models at exploring synthetically accessible analogs while maintaining high structural similarity to original target molecules. Furthermore, when integrated into existing molecular property-optimization frameworks, our hybrid approach produces synthetically feasible analogs with minimal loss in property scores. Our comprehensive benchmarking across diverse molecular datasets demonstrates that SynTwins effectively bridges the gap between computational design and experimental synthesis, providing a practical solution for accelerating the discovery of synthesizable molecules with desired properties for a wide range of applications.
Authors Shuan Chen, Gunwook Nam, Alan Aspuru-Guzik, Yousung Jung
Categories Artificial Intelligence
0
CompTrack: Information Bottleneck-Guided Low-Rank Dynamic Token Compression for Point Cloud Tracking AAAI 2026 (Oral)
Title CN CompTrack：基于信息瓶颈引导的低秩动态令牌压缩用于点云跟踪
Keywords 点云跟踪 · 信息瓶颈 · 令牌压缩 · 3D SOT · AAAI
Summary 本文提出CompTrack，一种用于LiDAR点云中3D单目标跟踪的端到端框架，通过空间前景预测模块去除背景噪声，并利用信息瓶颈引导的动态令牌压缩模块减少前景中的信息冗余，实现高效准确的跟踪。在KITTI、nuScenes和Waymo数据集上表现优异，达到90 FPS的实时性能。
Reason 该论文研究的是3D点云中的单目标跟踪任务，属于自动驾驶与3D视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其提出的方法如空间前景预测和基于信息瓶颈的令牌压缩，虽然在点云处理中有创新，但应用场景和数据模态（LiDAR点云）与文档图像差异巨大，难以迁移到解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract 3D single object tracking (SOT) in LiDAR point clouds is a critical task in computer vision and autonomous driving. Despite great success having been achieved, the inherent sparsity of point clouds introduces a dual-redundancy challenge that limits existing trackers: (1) vast spatial redundancy from background noise impairs accuracy, and (2) informational redundancy within the foreground hinders efficiency. To tackle these issues, we propose CompTrack, a novel end-to-end framework that systematically eliminates both forms of redundancy in point clouds. First, CompTrack incorporates a Spatial Foreground Predictor (SFP) module to filter out irrelevant background noise based on information entropy, addressing spatial redundancy. Subsequently, its core is an Information Bottleneck-guided Dynamic Token Compression (IB-DTC) module that eliminates the informational redundancy within the foreground. Theoretically grounded in low-rank approximation, this module leverages an online SVD analysis to adaptively compress the redundant foreground into a compact and highly informative set of proxy tokens. Extensive experiments on KITTI, nuScenes and Waymo datasets demonstrate that CompTrack achieves top-performing tracking performance with superior efficiency, running at a real-time 90 FPS on a single RTX 3090 GPU.
Authors Sifan Zhou, Yichao Cao, Jiahao Nie, Yuqian Fu, Ziyu Zhao, Xiaobo Lu, Shuo Wang
Comment Accepted by AAAI 2026 (Oral)
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
IE-Critic-R1: Advancing the Explanatory Measurement of Text-Driven Image Editing for Human Perception Alignment
Title CN IE-Critic-R1：推进文本驱动图像编辑的人类感知对齐的可解释性评估
Keywords 图像编辑评估 · 人类感知对齐 · 多模态评估 · 强化学习 · MOS
Summary 本文提出了IE-Bench基准和IE-Critic-R1评估模型，用于更准确地衡量文本驱动图像编辑结果的质量，并通过强化学习从可验证奖励中训练，使其评分更符合人类主观感知。研究重点在于图像编辑的评估体系，非文档理解方向。
Reason 该论文聚焦于文本驱动的图像编辑质量评估，属于图像生成与编辑领域，核心任务是设计评价指标（如MOS对齐），与文档图像理解（DIU）中的DocVQA、布局分析、表格识别等任务无直接关联。尽管使用了多模态方法，但其目标是评估图像编辑结果的视觉合理性，而非解决OCR幻觉、密集文本处理或复杂排版理解等问题。无法迁移至文档理解场景。
Abstract Recent advances in text-driven image editing have been significant, yet the task of accurately evaluating these edited images continues to pose a considerable challenge. Different from the assessment of text-driven image generation, text-driven image editing is characterized by simultaneously conditioning on both text and a source image. The edited images often retain an intrinsic connection to the original image, which dynamically change with the semantics of the text. However, previous methods tend to solely focus on text-image alignment or have not well aligned with human perception. In this work, we introduce the Text-driven Image Editing Benchmark suite (IE-Bench) to enhance the assessment of text-driven edited images. IE-Bench includes a database contains diverse source images, various editing prompts and the corresponding edited results from different editing methods, and nearly 4,000 samples with corresponding Mean Opinion Scores (MOS) provided by 15 human subjects. Furthermore, we introduce IE-Critic-R1, which, benefiting from Reinforcement Learning from Verifiable Rewards (RLVR), provides more comprehensive and explainable quality assessment for text-driven image editing that aligns with human perception. Extensive experiments demonstrate IE-Critic-R1's superior subjective-alignments on the text-driven image editing task compared with previous metrics. Related data and codes are available to the public.
Authors Bowen Qu, Shangkun Sun, Xiaoyu Liang, Wei Gao
Comment 18 pages, 10 figures, 8 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
0
Fidelity-Aware Recommendation Explanations via Stochastic Path Integration
Title CN 基于随机路径积分的保真度感知推荐解释
Keywords 推荐系统 · 可解释AI · 解释保真度 · 路径积分 · 归因分析
Summary 本文提出SPINRec，一种用于推荐系统解释的模型无关方法，通过随机路径积分技术从真实数据分布中采样 plausible 用户路径，以提升解释的保真度。实验验证了其在多个推荐模型和数据集上的优越性。
Reason 该论文研究的是推荐系统中的解释保真度问题，属于推荐系统与可解释AI的交叉领域，与文档图像理解（DIU）、视觉语言模型（VLM）在文档上的应用无直接关联。其方法SPINRec针对用户行为路径进行归因分析，难以迁移到OCR幻觉、密集文本识别、复杂版式理解等DIU核心痛点。属于无关领域应用。
Abstract Explanation fidelity, which measures how accurately an explanation reflects a model's true reasoning, remains critically underexplored in recommender systems. We introduce SPINRec (Stochastic Path Integration for Neural Recommender Explanations), a model-agnostic approach that adapts path-integration techniques to the sparse and implicit nature of recommendation data. To overcome the limitations of prior methods, SPINRec employs stochastic baseline sampling: instead of integrating from a fixed or unrealistic baseline, it samples multiple plausible user profiles from the empirical data distribution and selects the most faithful attribution path. This design captures the influence of both observed and unobserved interactions, yielding more stable and personalized explanations. We conduct the most comprehensive fidelity evaluation to date across three models (MF, VAE, NCF), three datasets (ML1M, Yahoo! Music, Pinterest), and a suite of counterfactual metrics, including AUC-based perturbation curves and fixed-length diagnostics. SPINRec consistently outperforms all baselines, establishing a new benchmark for faithful explainability in recommendation. Code and evaluation tools are publicly available at https://github.com/DeltaLabTLV/SPINRec.
Authors Oren Barkan, Yahlly Schein, Yehonatan Elisha, Veronika Bogina, Mikhail Baklanov, Noam Koenigstein
Categories Artificial Intelligence
0
MASTEST: A LLM-Based Multi-Agent System For RESTful API Tests
Title CN MASTEST：一种基于大语言模型的RESTful API测试多智能体系统
Keywords 多智能体系统 · 大语言模型 · API测试 · 自动化测试 · 软件质量保证
Summary 本文提出MASTEST，一个结合LLM智能体与程序化智能体的多智能体系统，用于自动化RESTful API测试全流程，包括从OpenAPI规范生成测试场景、生成Pytest脚本、执行测试、分析响应及计算覆盖率。系统支持人工介入以提升质量，并在GPT-4o和DeepSeek V3.1上验证了有效性。实验表明LLM能高效生成语法正确且具备良好覆盖能力的测试脚本。
Reason 该论文属于典型的平行下游应用，将LLM与多智能体框架应用于RESTful API测试领域，尽管技术流程完整，但其核心是特定软件工程任务的自动化工具构建，并未提出可用于文档图像理解（DIU）的方法论创新。研究内容与文档布局分析、视觉文本识别、多模态对齐、高分辨率处理或OCR去幻觉等DIU核心问题无关，无法迁移用于解决文档理解中的关键挑战。
Abstract Testing RESTful API is increasingly important in quality assurance of cloud-native applications. Recent advances in machine learning (ML) techniques have demonstrated that various testing activities can be performed automatically by large language models (LLMs) with reasonable accuracy. This paper develops a multi-agent system called MASTEST that combines LLM-based and programmed agents to form a complete tool chain that covers the whole workflow of API test starting from generating unit and system test scenarios from API specification in the OpenAPI Swagger format, to generating of Pytest test scripts, executing test scripts to interact with web services, to analysing web service response messages to determine test correctness and calculate test coverage. The system also supports the incorporation of human testers in reviewing and correcting LLM generated test artefacts to ensure the quality of testing activities. MASTEST system is evaluated on two LLMs, GPT-4o and DeepSeek V3.1 Reasoner with five public APIs. The performances of LLMs on various testing activities are measured by a wide range of metrics, including unit and system test scenario coverage and API operation coverage for the quality of generated test scenarios, data type correctness, status code coverage and script syntax correctness for the quality of LLM generated test scripts, as well as bug detection ability and usability of LLM generated test scenarios and scripts. Experiment results demonstrated that both DeepSeek and GPT-4o achieved a high overall performance. DeepSeek excels in data type correctness and status code detection, while GPT-4o performs best in API operation coverage. For both models, LLM generated test scripts maintained 100\% syntax correctness and only required minimal manual edits for semantic correctness. These findings indicate the effectiveness and feasibility of MASTEST.
Authors Xiaoke Han, Hong Zhu
Comment 14 Page of main text plus 4 pages of appendix
Categories Artificial Intelligence
0
Hierarchical biomarker thresholding: a model-agnostic framework for stability
Title CN 分层生物标志物阈值设定：一种用于稳定性的模型无关框架
Keywords 生物标志物 · 分层阈值 · 稳定性 · 模型无关 · 风险分解
Summary 本文提出了一种分层阈值设定框架，用于在存在数据分布变化和评分尺度不一致的情况下，提升跨机构生物标志物分析中患者级别决策的稳定性与可重复性。该方法基于风险分解定理，分离出拟合度、操作点偏移和稳定性项，并通过患者块自助法计算稳定性成分，适用于多种模型且具有可解释性诊断输出。
Reason 该论文属于生物医学领域的决策阈值优化问题，尽管提到了模型无关和稳定性等通用概念，但其核心任务是患者级别的生物标志物决策，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。方法难以迁移到OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract Many biomarker pipelines require patient-level decisions aggregated from instance-level (cell/patch) scores. Thresholds tuned on pooled instances often fail across sites due to hierarchical dependence, prevalence shift, and score-scale mismatch. We present a selection-honest framework for hierarchical thresholding that makes patient-level decisions reproducible and more defensible. At its core is a risk decomposition theorem for selection-honest thresholds. The theorem separates contributions from (i) internal fit and patient-level generalization, (ii) operating-point shift reflecting prevalence and shape changes, and (iii) a stability term that penalizes sensitivity to threshold perturbations. The stability component is computable via patient-block bootstraps mapped through a monotone modulus of risk. This framework is model-agnostic, reconciles heterogeneous decision rules on a quantile scale, and yields monotone-invariant ensembles and reportable diagnostics (e.g. flip-rate, operating-point shift).
Authors O. Debeaupuis
Categories Artificial Intelligence
0
DreamGarden: A Designer Assistant for Growing Games from a Single Prompt CHI 2025
Title CN DreamGarden：一个从单个提示生成游戏的设计师助手
Keywords 游戏设计 · LLM规划 · 人机交互 · Unreal Engine · AI助手
Summary 本文提出了DreamGarden，一种基于大语言模型的系统，能够将用户提供的单一高层提示（如梦境或记忆）分解为分层行动计划，并通过专门的子模块在Unreal Engine中实现多样化游戏环境的构建。系统以‘花园’形式呈现计划与动作，支持用户通过种子提示、修剪和反馈进行干预。研究通过用户实验探讨了此类半自主助手在开放性模拟设计中的设计意义。
Reason 该论文属于具身智能与游戏设计交叉领域，核心是利用LLM辅助Unreal Engine中的游戏开发，属于典型的‘平行应用’。其方法聚焦于游戏场景的生成与人机协作流程设计，并未提出可用于文档图像理解（如OCR去幻觉、复杂排版解析、细粒度视觉定位等）的底层技术或可迁移机制。尽管使用了LLM规划架构，但该架构针对的是开放世界模拟设计，难以迁移到文档理解任务。
Abstract Coding assistants are increasingly leveraged in game design, both generating code and making high-level plans. To what degree can these tools align with developer workflows, and what new modes of human-computer interaction can emerge from their use? We present DreamGarden, an AI system capable of assisting with the development of diverse game environments in Unreal Engine. At the core of our method is an LLM-driven planner, capable of breaking down a single, high-level prompt -- a dream, memory, or imagined scenario provided by a human user -- into a hierarchical action plan, which is then distributed across specialized submodules facilitating concrete implementation. This system is presented to the user as a garden of plans and actions, both growing independently and responding to user intervention via seed prompts, pruning, and feedback. Through a user study, we explore design implications of this system, charting courses for future work in semi-autonomous assistants and open-ended simulation design.
Authors Sam Earle, Samyak Parajuli, Andrzej Banburski-Fahey
Comment 30 pages + appendix, 11 figures, published at CHI 2025
Categories Artificial Intelligence, Computation and Language
0
Extracting Interaction-Aware Monosemantic Concepts in Recommender Systems
Title CN 在推荐系统中提取交互感知的单义概念
Keywords 推荐系统 · 可解释性 · 稀疏自编码器 · 单义性 · 后处理控制
Summary 本文提出一种从推荐系统中的用户和物品嵌入中提取单义概念的方法，通过引入预测感知的稀疏自编码器，在不修改原始模型的前提下实现对推荐结果的可控操作。实验表明该方法能捕捉如类型、流行度和时间趋势等语义特征，并支持后处理控制。然而，其应用场景和技术路径均远离文档图像理解领域。
Reason 该论文研究的是推荐系统中的可解释性问题，聚焦于用户和物品嵌入空间中的单义神经元提取，属于推荐系统领域的方法论工作。尽管其提出的稀疏自编码器与预测感知训练目标具有一定方法价值，但其核心任务与文档图像理解（DIU）、视觉文本识别、多模态文档推理等完全无关，无法迁移解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点。
Abstract We present a method for extracting \emph{monosemantic} neurons, defined as latent dimensions that align with coherent and interpretable concepts, from user and item embeddings in recommender systems. Our approach employs a Sparse Autoencoder (SAE) to reveal semantic structure within pretrained representations. In contrast to work on language models, monosemanticity in recommendation must preserve the interactions between separate user and item embeddings. To achieve this, we introduce a \emph{prediction aware} training objective that backpropagates through a frozen recommender and aligns the learned latent structure with the model's user-item affinity predictions. The resulting neurons capture properties such as genre, popularity, and temporal trends, and support post hoc control operations including targeted filtering and content promotion without modifying the base model. Our method generalizes across different recommendation models and datasets, providing a practical tool for interpretable and controllable personalization. Code and evaluation resources are available at https://github.com/DeltaLabTLV/Monosemanticity4Rec.
Authors Dor Arviv, Yehonatan Elisha, Oren Barkan, Noam Koenigstein
Categories Artificial Intelligence
0
Modeling Retinal Ganglion Cells with Neural Differential Equations AAAI-26 Student Abstract and Poster Program
Title CN 使用神经微分方程建模视网膜神经节细胞
Keywords 神经微分方程 · 视网膜神经节细胞 · 时间序列建模 · LTC · CfC
Summary 本文采用液态时间常数网络（LTCs）和闭式连续时间网络（CfCs）对虎蝾螈视网膜神经节细胞的响应进行建模，在三个数据集上表现出优于CNN和LSTM的MAE性能、更快的收敛速度和更小的模型规模，适用于数据有限且需频繁重训练的边缘场景（如视觉假体）。但研究对象为生物神经元活动预测，非文档图像理解任务。
Reason 该论文研究的是生物视觉系统中视网膜神经节细胞的建模，属于计算神经科学与生物医学交叉领域，尽管使用了LTC和CfC等新兴神经网络架构，但其任务目标与文档图像理解（DIU）无直接关联。既不涉及文本识别、布局分析、表单理解等DIU核心任务，也未提出可迁移至文档理解领域的通用方法。属于无关领域研究。
Abstract This work explores Liquid Time-Constant Networks (LTCs) and Closed-form Continuous-time Networks (CfCs) for modeling retinal ganglion cell activity in tiger salamanders across three datasets. Compared to a convolutional baseline and an LSTM, both architectures achieved lower MAE, faster convergence, smaller model sizes, and favorable query times, though with slightly lower Pearson correlation. Their efficiency and adaptability make them well suited for scenarios with limited data and frequent retraining, such as edge deployments in vision prosthetics.
Authors Kacper Dobek, Daniel Jankowski, Krzysztof Krawiec
Comment Accepted to the AAAI-26 Student Abstract and Poster Program, with supplementary material
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Save, Revisit, Retain: A Scalable Framework for Enhancing User Retention in Large-Scale Recommender Systems
Title CN 保存、重访、留存：一种可扩展的框架以增强大规模推荐系统中的用户留存
Keywords 用户留存 · 推荐系统 · 行为归因 · 事件聚合 · 可扩展性
Summary 本文提出了一种轻量且可解释的框架，用于建模Pinterest平台中用户的重访行为，通过构建代理归因机制将用户‘保存’内容的行为与后续的回访关联起来，从而优化长期用户留存。该方法已在大规模场景中部署并取得实际效果提升。
Reason 该论文研究的是推荐系统中的用户行为建模与留存优化问题，属于典型的工业级推荐系统（RecSys）应用，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的核心技术无关。其方法聚焦于用户动作归因和事件流水线聚合，无法迁移解决OCR幻觉、复杂排版解析、细粒度视觉定位等DIU核心痛点。
Abstract User retention is a critical objective for online platforms like Pinterest, as it strengthens user loyalty and drives growth through repeated engagement. A key indicator of retention is revisitation, i.e., when users return to view previously saved content, a behavior often sparked by personalized recommendations and user satisfaction. However, modeling and optimizing revisitation poses significant challenges. One core difficulty is accurate attribution: it is often unclear which specific user actions or content exposures trigger a revisit, since many confounding factors (e.g., content quality, user interface, notifications, or even changing user intent) can influence return behavior. Additionally, the scale and timing of revisitations introduce further complexity; users may revisit content days or even weeks after their initial interaction, requiring the system to maintain and associate extensive historical records across millions of users and sessions. These complexities render existing methods insufficient for robustly capturing and optimizing long-term revisitation. To address these gaps, we introduce a novel, lightweight, and interpretable framework for modeling revisitation behavior and optimizing long-term user retention in Pinterest's search-based recommendation context. By defining a surrogate attribution process that links saves to subsequent revisitations, we reduce noise in the causal relationship between user actions and return visits. Our scalable event aggregation pipeline enables large-scale analysis of user revisitation patterns and enhances the ranking system's ability to surface items with high retention value. Deployed on Pinterest's Related Pins surface to serve 500+ million users, the framework led to a significant lift of 0.1% in active users without additional computational costs.
Authors Weijie Jiang, Armando Ordorica, Jaewon Yang, Olafur Gudmundsson, Yucheng Tu, Huizhong Duan
Categories Artificial Intelligence
0
Reward Engineering for Spatial Epidemic Simulations: A Reinforcement Learning Platform for Individual Behavioral Learning
Title CN 用于空间流行病模拟的奖励工程：一种面向个体行为学习的强化学习平台
Keywords 强化学习 · 奖励工程 · 流行病模拟 · 个体行为建模 · 非药物干预
Summary 本文提出了ContagionRL，一个兼容Gymnasium的强化学习平台，用于在空间流行病模拟中系统研究奖励函数设计对个体行为学习的影响。平台结合SIRS+D传染病模型和可配置环境参数，评估不同奖励机制下智能体的学习效果。实验表明，引入方向引导和依从性激励的奖励函数能显著提升非药物干预措施的遵守程度和生存率。平台强调了奖励设计、信息结构和环境可预测性在行为学习中的重要性。
Reason 该论文属于具身智能与流行病建模交叉领域，研究在模拟环境中通过强化学习训练个体行为策略，与文档图像理解（DIU）、视觉语言模型（VLM）或多模态文档处理无直接关联。尽管涉及Agent行为学习，但其方法针对的是空间避障与生存策略，无法迁移解决OCR幻觉、复杂排版理解或长文档推理等DIU核心问题。
Abstract We present ContagionRL, a Gymnasium-compatible reinforcement learning platform specifically designed for systematic reward engineering in spatial epidemic simulations. Unlike traditional agent-based models that rely on fixed behavioral rules, our platform enables rigorous evaluation of how reward function design affects learned survival strategies across diverse epidemic scenarios. ContagionRL integrates a spatial SIRS+D epidemiological model with configurable environmental parameters, allowing researchers to stress-test reward functions under varying conditions including limited observability, different movement patterns, and heterogeneous population dynamics. We evaluate five distinct reward designs, ranging from sparse survival bonuses to a novel potential field approach, across multiple RL algorithms (PPO, SAC, A2C). Through systematic ablation studies, we identify that directional guidance and explicit adherence incentives are critical components for robust policy learning. Our comprehensive evaluation across varying infection rates, grid sizes, visibility constraints, and movement patterns reveals that reward function choice dramatically impacts agent behavior and survival outcomes. Agents trained with our potential field reward consistently achieve superior performance, learning maximal adherence to non-pharmaceutical interventions while developing sophisticated spatial avoidance strategies. The platform's modular design enables systematic exploration of reward-behavior relationships, addressing a knowledge gap in models of this type where reward engineering has received limited attention. ContagionRL is an effective platform for studying adaptive behavioral responses in epidemic contexts and highlight the importance of reward design, information structure, and environmental predictability in learning.
Authors Radman Rakhshandehroo, Daniel Coombs
Comment 35 pages, 15 figures and 14 tables
Categories Artificial Intelligence
0
How Far Can LLMs Emulate Human Behavior?: A Strategic Analysis via the Buy-and-Sell Negotiation Game
Title CN 大语言模型能在多大程度上模拟人类行为？——通过买卖谈判游戏的战略分析
Keywords 大语言模型 · 人类行为模拟 · 谈判游戏 · 社会交互 · LLM评估
Summary 本文提出通过买卖谈判游戏来评估大语言模型模拟人类情感、行为及战略决策的能力。研究人员为不同LLM分配买家与卖家角色，进行多轮谈判实验，并分析成交价格、胜率和SHAP值等指标。结果显示，基准分数更高的模型整体表现更好，但在强调情感或社交情境下某些模型表现下降；竞争性强的性格比合作型更有利于谈判结果。文章主张将谈判模拟作为现有基准的补充，用以衡量LLM在现实交互中的社会行为能力。
Reason 该论文属于纯NLP与社会行为模拟的交叉研究，聚焦于LLM在谈判场景中的人类行为模仿能力评估，不涉及任何视觉模态或文档理解任务。其方法论围绕对话策略、人格设定和博弈结果分析展开，与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版、细粒度定位、高分辨率处理等完全无关，无法迁移为解决DIU问题的技术工具。
Abstract With the rapid advancement of Large Language Models (LLMs), recent studies have drawn attention to their potential for handling not only simple question-answer tasks but also more complex conversational abilities and performing human-like behavioral imitations. In particular, there is considerable interest in how accurately LLMs can reproduce real human emotions and behaviors, as well as whether such reproductions can function effectively in real-world scenarios. However, existing benchmarks focus primarily on knowledge-based assessment and thus fall short of sufficiently reflecting social interactions and strategic dialogue capabilities. To address these limitations, this work proposes a methodology to quantitatively evaluate the human emotional and behavioral imitation and strategic decision-making capabilities of LLMs by employing a Buy and Sell negotiation simulation. Specifically, we assign different personas to multiple LLMs and conduct negotiations between a Buyer and a Seller, comprehensively analyzing outcomes such as win rates, transaction prices, and SHAP values. Our experimental results show that models with higher existing benchmark scores tend to achieve better negotiation performance overall, although some models exhibit diminished performance in scenarios emphasizing emotional or social contexts. Moreover, competitive and cunning traits prove more advantageous for negotiation outcomes than altruistic and cooperative traits, suggesting that the assigned persona can lead to significant variations in negotiation strategies and results. Consequently, this study introduces a new evaluation approach for LLMs' social behavior imitation and dialogue strategies, and demonstrates how negotiation simulations can serve as a meaningful complementary metric to measure real-world interaction capabilities-an aspect often overlooked in existing benchmarks.
Authors Mingyu Jeon, Jaeyoung Suh, Suwan Cho, Dohyeon Kim
Categories Artificial Intelligence
0
Privacy Auditing of Multi-domain Graph Pre-trained Model under Membership Inference Attacks AAAI 2026
Title CN 多领域图预训练模型在成员推断攻击下的隐私审计
Keywords 图神经网络 · 成员推断攻击 · 隐私风险 · 多领域预训练 · 机器遗忘
Summary 本文提出MGP-MIA框架，用于对多领域图预训练模型进行成员推断攻击，通过信号放大、增量影子模型构建和基于相似性的推理机制揭示其隐私风险。研究聚焦于图数据上的隐私漏洞，而非文档内容理解或多模态建模。
Reason 该论文研究的是图神经网络在多领域预训练场景下的隐私风险与成员推断攻击，属于安全与隐私方向，与文档图像理解（DIU）、视觉语言模型（VLM）或OCR等核心技术无关。尽管涉及机器学习模型的安全分析，但其方法论无法直接迁移用于解决文档理解中的OCR幻觉、复杂排版、细粒度定位或长文档推理等关键问题。
Abstract Multi-domain graph pre-training has emerged as a pivotal technique in developing graph foundation models. While it greatly improves the generalization of graph neural networks, its privacy risks under membership inference attacks (MIAs), which aim to identify whether a specific instance was used in training (member), remain largely unexplored. However, effectively conducting MIAs against multi-domain graph pre-trained models is a significant challenge due to: (i) Enhanced Generalization Capability: Multi-domain pre-training reduces the overfitting characteristics commonly exploited by MIAs. (ii) Unrepresentative Shadow Datasets: Diverse training graphs hinder the obtaining of reliable shadow graphs. (iii) Weakened Membership Signals: Embedding-based outputs offer less informative cues than logits for MIAs. To tackle these challenges, we propose MGP-MIA, a novel framework for Membership Inference Attacks against Multi-domain Graph Pre-trained models. Specifically, we first propose a membership signal amplification mechanism that amplifies the overfitting characteristics of target models via machine unlearning. We then design an incremental shadow model construction mechanism that builds a reliable shadow model with limited shadow graphs via incremental learning. Finally, we introduce a similarity-based inference mechanism that identifies members based on their similarity to positive and negative samples. Extensive experiments demonstrate the effectiveness of our proposed MGP-MIA and reveal the privacy risks of multi-domain graph pre-training.
Authors Jiayi Luo, Qingyun Sun, Yuecen Wei, Haonan Yuan, Xingcheng Fu, Jianxin Li
Comment Accepted by AAAI 2026(Oral)
Categories Artificial Intelligence
0
MSCloudCAM: Multi-Scale Context Adaptation with Convolutional Cross-Attention for Multispectral Cloud Segmentation
Title CN MSCloudCAM：基于卷积交叉注意力的多尺度上下文自适应多光谱云分割
Keywords 多光谱分割 · 云检测 · 多尺度上下文 · 交叉注意力 · 遥感图像
Summary 本文提出MSCloudCAM，一种用于多光谱和多传感器云分割的网络，通过卷积交叉注意力机制融合局部细节与多尺度上下文信息，在CloudSEN12和L8Biome等遥感数据集上表现优异。方法依赖于遥感特有的光谱特性，不适用于文档图像理解任务。
Reason 该论文研究的是遥感图像中的多光谱云分割，属于地球观测和卫星图像处理领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心技术聚焦于多尺度上下文建模与跨模态特征融合，应用于环境与气候分析场景，无法迁移解决OCR幻觉、复杂排版解析、长文档推理等DIU核心痛点。
Abstract Clouds remain a major obstacle in optical satellite imaging, limiting accurate environmental and climate analysis. To address the strong spectral variability and the large scale differences among cloud types, we propose MSCloudCAM, a novel multi-scale context adapter network with convolution based cross-attention tailored for multispectral and multi-sensor cloud segmentation. A key contribution of MSCloudCAM is the explicit modeling of multiple complementary multi-scale context extractors. And also, rather than simply stacking or concatenating their outputs, our formulation uses one extractor's fine-resolution features and the other extractor's global contextual representations enabling dynamic, scale-aware feature selection. Building on this idea, we design a new convolution-based cross attention adapter that effectively fuses localized, detailed information with broader multi-scale context. Integrated with a hierarchical vision backbone and refined through channel and spatial attention mechanisms, MSCloudCAM achieves strong spectral-spatial discrimination. Experiments on various multisensor datatsets e.g. CloudSEN12 (Sentinel-2) and L8Biome (Landsat-8) show that MSCloudCAM outperforms recent state-of-the-art models while maintaining competitive model complexity, highlighting the novelty and effectiveness of the proposed design for large-scale Earth observation.
Authors Md Abdullah Al Mazid, Liangdong Deng, Naphtali Rishe
Comment 6 pages, 3 Figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Plan-X: Instruct Video Generation via Semantic Planning
Title CN 通过语义规划指导视频生成
Keywords 视频生成 · 扩散Transformer · 语义规划 · 视觉幻觉 · 多模态推理
Summary 本文提出Plan-X框架，通过引入可学习的多模态语言模型作为语义规划器，从文本提示和视觉上下文中推理用户意图，并自回归生成时空语义token序列，作为视频扩散模型的结构化‘语义草图’，以提升指令对齐性和生成一致性。实验表明该方法能显著减少视觉幻觉，实现细粒度的指令引导视频生成。
Reason 该论文属于视频生成领域，核心任务是解决扩散模型在复杂场景、多阶段动作和上下文运动推理中的视觉幻觉问题。虽然提到了‘语义规划’和‘减少视觉幻觉’，但其方法针对的是视频内容的时序合成，与文档图像理解（DIU）中的OCR幻觉、密集文本识别、复杂版式分析或长文档推理等核心痛点无直接关联。技术路径无法有效迁移到文档理解场景。
Abstract Diffusion Transformers have demonstrated remarkable capabilities in visual synthesis, yet they often struggle with high-level semantic reasoning and long-horizon planning. This limitation frequently leads to visual hallucinations and mis-alignments with user instructions, especially in scenarios involving complex scene understanding, human-object interactions, multi-stage actions, and in-context motion reasoning. To address these challenges, we propose Plan-X, a framework that explicitly enforces high-level semantic planning to instruct video generation process. At its core lies a Semantic Planner, a learnable multimodal language model that reasons over the user's intent from both text prompts and visual context, and autoregressively generates a sequence of text-grounded spatio-temporal semantic tokens. These semantic tokens, complementary to high-level text prompt guidance, serve as structured "semantic sketches" over time for the video diffusion model, which has its strength at synthesizing high-fidelity visual details. Plan-X effectively integrates the strength of language models in multimodal in-context reasoning and planning, together with the strength of diffusion models in photorealistic video synthesis. Extensive experiments demonstrate that our framework substantially reduces visual hallucinations and enables fine-grained, instruction-aligned video generation consistent with multimodal context.
Authors Lun Huang, You Xie, Hongyi Xu, Tianpei Gu, Chenxu Zhang, Guoxian Song, Zenan Li, Xiaochen Zhao, Linjie Luo, Guillermo Sapiro
Comment The project page is at https://byteaigc.github.io/Plan-X
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Towards Effective, Stealthy, and Persistent Backdoor Attacks Targeting Graph Foundation Models AAAI 2026
Title CN 面向图基础模型的有效、隐蔽且持久的后门攻击
Keywords 图基础模型 · 后门攻击 · 安全性 · 对抗机器学习 · 持久性
Summary 本文提出GFM-BA，一种针对图基础模型的新型后门攻击方法，通过标签无关的触发关联模块、节点自适应触发生成器和持久性锚定模块，在预训练阶段注入隐蔽且能在下游任务中持续存在的后门行为。实验表明该攻击在有效性、隐蔽性和持久性方面均表现突出。
Reason 该论文研究的是图基础模型（GFM）的安全性问题，属于AI安全与对抗攻击领域，与文档图像理解（DIU）或视觉语言模型在文档处理中的应用无直接关联。尽管其方法涉及模型鲁棒性和训练过程干预，但核心目标是实施后门攻击而非解决OCR幻觉、复杂排版解析或长文档推理等DIU痛点，无法迁移为提升文档理解性能的技术工具。
Abstract Graph Foundation Models (GFMs) are pre-trained on diverse source domains and adapted to unseen targets, enabling broad generalization for graph machine learning. Despite that GFMs have attracted considerable attention recently, their vulnerability to backdoor attacks remains largely underexplored. A compromised GFM can introduce backdoor behaviors into downstream applications, posing serious security risks. However, launching backdoor attacks against GFMs is non-trivial due to three key challenges. (1) Effectiveness: Attackers lack knowledge of the downstream task during pre-training, complicating the assurance that triggers reliably induce misclassifications into desired classes. (2) Stealthiness: The variability in node features across domains complicates trigger insertion that remains stealthy. (3) Persistence: Downstream fine-tuning may erase backdoor behaviors by updating model parameters. To address these challenges, we propose GFM-BA, a novel Backdoor Attack model against Graph Foundation Models. Specifically, we first design a label-free trigger association module that links the trigger to a set of prototype embeddings, eliminating the need for knowledge about downstream tasks to perform backdoor injection. Then, we introduce a node-adaptive trigger generator, dynamically producing node-specific triggers, reducing the risk of trigger detection while reliably activating the backdoor. Lastly, we develop a persistent backdoor anchoring module that firmly anchors the backdoor to fine-tuning-insensitive parameters, enhancing the persistence of the backdoor under downstream adaptation. Extensive experiments demonstrate the effectiveness, stealthiness, and persistence of GFM-BA.
Authors Jiayi Luo, Qingyun Sun, Lingjuan Lyu, Ziwei Zhang, Haonan Yuan, Xingcheng Fu, Jianxin Li
Comment Accepted by AAAI 2026
Categories Artificial Intelligence
0
Why Does Stochastic Gradient Descent Slow Down in Low-Precision Training?
Title CN 为什么随机梯度下降在低精度训练中会变慢？
Keywords 低精度训练 · 随机梯度下降 · 梯度收缩 · 收敛性分析 · 量化
Summary 本文从理论上分析了低精度训练中梯度量化导致的幅度收缩现象，提出梯度收缩模型，证明其会降低SGD的有效步长，从而减缓收敛速度并提高稳态误差。研究聚焦于数值精度对优化过程的影响，属于通用深度学习训练的理论探讨。
Reason 该论文研究的是低精度训练中随机梯度下降（SGD）的收敛性问题，属于深度学习优化理论范畴，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。未涉及OCR、文档布局分析、表格识别、多模态对齐或视觉幻觉等DIU核心问题，也无法迁移为解决DIU痛点（如高分辨率处理、细粒度定位、长文档推理）的有效工具。
Abstract Low-precision training has become crucial for reducing the computational and memory costs of large-scale deep learning. However, quantizing gradients introduces magnitude shrinkage, which can change how stochastic gradient descent (SGD) converges. In this study, we explore SGD convergence under a gradient shrinkage model, where each stochastic gradient is scaled by a factor \( q_k \in (0,1] \). We show that this shrinkage affect the usual stepsize \( μ_k \) with an effective stepsize \( μ_k q_k \), slowing convergence when \( q_{\min} < 1 \). With typical smoothness and bounded-variance assumptions, we prove that low-precision SGD still converges, but at a slower pace set by \( q_{\min} \), and with a higher steady error level due to quantization effects. We analyze theoretically how lower numerical precision slows training by treating it as gradient shrinkage within the standard SGD convergence setup.
Authors Vincent-Daniel Yun
Categories Artificial Intelligence
0
Prediction of Clinical Complication Onset using Neural Point Processes
Title CN 使用神经点过程预测临床并发症发作
Keywords 神经点过程 · 临床事件预测 · 可解释性 · 时间序列预测 · 机器学习
Summary 本文研究利用神经点过程预测重症监护中临床不良事件（如心脏骤停、败血症等）的发作时间，旨在提升预测的可解释性。实验评估了六种先进的神经点过程模型在六个医疗数据集上的表现。虽然方法具有时序建模和可解释性的优势，但完全脱离文档视觉理解场景。
Reason 该论文属于医疗时序事件预测领域，尽管使用了神经点过程这一具有一定方法论价值的技术，但其研究任务与文档图像理解（DIU）或视觉语言模型（VLM）无关。未涉及OCR、文档布局分析、表格识别、视觉文本理解等核心DIU任务，也未提出可迁移至DIU领域的通用架构或推理机制。属于典型的垂直领域应用（医疗预测），属于‘平行下游应用’范畴。
Abstract Predicting medical events in advance within critical care settings is paramount for patient outcomes and resource management. Utilizing predictive models, healthcare providers can anticipate issues such as cardiac arrest, sepsis, or respiratory failure before they manifest. Recently, there has been a surge in research focusing on forecasting adverse medical event onsets prior to clinical manifestation using machine learning. However, while these models provide temporal prognostic predictions for the occurrence of a specific adverse event of interest within defined time intervals, their interpretability often remains a challenge. In this work, we explore the applicability of neural temporal point processes in the context of adverse event onset prediction, with the aim of explaining clinical pathways and providing interpretable insights. Our experiments span six state-of-the-art neural point processes and six critical care datasets, each focusing on the onset of distinct adverse events. This work represents a novel application class of neural temporal point processes in event prediction.
Authors Sachini Weerasekara, Sagar Kamarthi, Jacqueline Isaacs
Categories Artificial Intelligence
0
Mesh-based Super-resolution of Detonation Flows with Multiscale Graph Transformers
Title CN 基于网格的爆轰流多尺度图变换器超分辨率
Keywords 图神经网络 · 超分辨率 · 流场重建 · Transformer · 科学计算
Summary 本文提出了一种基于多尺度图变换器（SR-GT）的网格化超分辨率方法，用于重建反应流场。该方法采用图结构表示非均匀网格，利用Transformer捕捉低分辨率流场的长程依赖关系，并生成保留关键特征的高分辨率结果。在二维爆轰传播问题上验证了其优于传统插值方法的性能。
Reason 该论文研究的是流体力学中的爆轰流场超分辨率重建，属于物理仿真与科学计算领域，尽管使用了图神经网络和Transformer架构处理非结构化网格数据，但其任务与文档图像理解（DIU）无直接关联。未涉及OCR、布局分析、表格识别或任何文档理解核心问题，也无法迁移用于解决文档中的高分辨率处理、细粒度定位或幻觉等痛点。
Abstract Super-resolution flow reconstruction using state-of-the-art data-driven techniques is valuable for a variety of applications, such as subgrid/subfilter closure modeling, accelerating spatiotemporal forecasting, data compression, and serving as an upscaling tool for sparse experimental measurements. In the present work, a first-of-its-kind multiscale graph transformer approach is developed for mesh-based super-resolution (SR-GT) of reacting flows. The novel data-driven modeling paradigm leverages a graph-based flow-field representation compatible with complex geometries and non-uniform/unstructured grids. Further, the transformer backbone captures long-range dependencies between different parts of the low-resolution flow-field, identifies important features, and then generates the super-resolved flow-field that preserves those features at a higher resolution. The performance of SR-GT is demonstrated in the context of spectral-element-discretized meshes for a challenging test problem of 2D detonation propagation within a premixed hydrogen-air mixture exhibiting highly complex multiscale reacting flow behavior. The SR-GT framework utilizes a unique element + neighborhood graph representation for the coarse input, which is then tokenized before being processed by the transformer component to produce the fine output. It is demonstrated that SR-GT provides high super-resolution accuracy for reacting flow-field features and superior performance compared to traditional interpolation-based SR schemes.
Authors Shivam Barwey, Pinaki Pal
Categories Artificial Intelligence
0
Classification of autoimmune diseases from Peripheral blood TCR repertoires by multimodal multi-instance learning
Title CN 基于多模态多实例学习的外周血TCR库分类自身免疫性疾病
Keywords 多实例学习 · TCR序列 · 自身免疫病 · 生物标志物 · 深度学习
Summary 本文提出EAMil模型，一种结合PrimeSeq特征提取与ESMonehot编码的多实例深度学习框架，用于从T细胞受体（TCR）序列中诊断自身免疫性疾病。模型在SLE和RA上分别达到98.95%和97.76%的AUC，具备良好的可解释性和抗混淆能力，可用于疾病分型与严重程度评估。
Reason 该论文研究的是利用TCR序列数据通过多实例学习框架诊断自身免疫性疾病（如系统性红斑狼疮和类风湿性关节炎），属于生物医学与AI交叉的垂直应用领域，核心方法为特定医疗场景下的序列建模，并未涉及文档图像、文本布局、视觉语言理解或OCR相关技术。与文档图像理解（DIU）任务无关联，属于无关领域的下游应用。
Abstract T cell receptor (TCR) repertoires encode critical immunological signatures for autoimmune diseases, yet their clinical application remains limited by sequence sparsity and low witness rates. We developed EAMil, a multi-instance deep learning framework that leverages TCR sequencing data to diagnose systemic lupus erythematosus (SLE) and rheumatoid arthritis (RA) with exceptional accuracy. By integrating PrimeSeq feature extraction with ESMonehot encoding and enhanced gate attention mechanisms, our model achieved state-of-the-art performance with AUCs of 98.95% for SLE and 97.76% for RA. EAMil successfully identified disease-associated genes with over 90% concordance with established differential analyses and effectively distinguished disease-specific TCR genes. The model demonstrated robustness in classifying multiple disease categories, utilizing the SLEDAI score to stratify SLE patients by disease severity as well as to diagnose the site of damage in SLE patients, and effectively controlling for confounding factors such as age and gender. This interpretable framework for immune receptor analysis provides new insights for autoimmune disease detection and classification with broad potential clinical applications across immune-mediated conditions.
Authors Ruihao Zhang, Mao chen, Fei Ye, Dandan Meng, Yixuan Huang, Xiao Liu
Comment 4 figures, 3 tabels, 8 pages
Categories Artificial Intelligence
0
Hybrid LSTM and PPO Networks for Dynamic Portfolio Optimization 2025 8th Artificial Intelligence and Cloud Computing Conference
Title CN 用于动态投资组合优化的混合LSTM和PPO网络
Keywords LSTM · PPO · 投资组合优化 · 强化学习 · 时间序列预测
Summary 本文提出了一种结合LSTM预测与PPO强化学习的投资组合优化框架，在多资产数据集上验证了其优于基准模型的表现。研究聚焦于金融时序预测与策略优化，不涉及文档图像、多模态理解或视觉信息处理。
Reason 该论文属于金融领域的决策优化问题，使用LSTM和强化学习进行资产配置，与文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的应用完全无关。未涉及OCR、布局分析、表格识别、视觉表示学习或任何DIU核心任务，也不提供可迁移至文档理解领域的底层方法。
Abstract This paper introduces a hybrid framework for portfolio optimization that fuses Long Short-Term Memory (LSTM) forecasting with a Proximal Policy Optimization (PPO) reinforcement learning strategy. The proposed system leverages the predictive power of deep recurrent networks to capture temporal dependencies, while the PPO agent adaptively refines portfolio allocations in continuous action spaces, allowing the system to anticipate trends while adjusting dynamically to market shifts. Using multi-asset datasets covering U.S. and Indonesian equities, U.S. Treasuries, and major cryptocurrencies from January 2018 to December 2024, the model is evaluated against several baselines, including equal-weight, index-style, and single-model variants (LSTM-only and PPO-only). The framework's performance is benchmarked against equal-weighted, index-based, and single-model approaches (LSTM-only and PPO-only) using annualized return, volatility, Sharpe ratio, and maximum drawdown metrics, each adjusted for transaction costs. The results indicate that the hybrid architecture delivers higher returns and stronger resilience under non-stationary market regimes, suggesting its promise as a robust, AI-driven framework for dynamic portfolio optimization.
Authors Jun Kevin, Pujianto Yugopuspito
Comment 12 pages, 8 figures, 2 tables, accepted at 2025 8th Artificial Intelligence and Cloud Computing Conference
Categories Artificial Intelligence
0
Towards Automating Data Access Permissions in AI Agents IEEE Symposium on Security and Privacy (S&P) 2026
Title CN 在AI代理中实现数据访问权限的自动化
Keywords AI Agent · Permission Management · User Study · Security and Privacy · Machine Learning
Summary 本文提出一种基于机器学习的自动化权限管理方法，用于预测AI代理在特定上下文中是否应获得用户数据访问权限。通过用户研究提取影响决策的因素，并构建预测模型，在小样本下实现了较高的准确率。重点在于提升AI代理的透明性与用户控制力。
Reason 该论文研究的是AI代理在用户授权下的权限管理问题，属于安全与隐私领域中的应用型研究，核心关注点为用户权限决策的预测模型，并非文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的方法创新。尽管其使用了机器学习方法进行用户行为建模，但任务本身与文档解析、OCR、布局分析、表格识别等DIU核心任务无关，也不提供可用于解决OCR幻觉、复杂排版或长文档推理的技术工具。属于典型的‘平行应用’类工作，且位于无关领域（安全与隐私），无法迁移至DIU的技术主线。
Abstract As AI agents attempt to autonomously act on users' behalf, they raise transparency and control issues. We argue that permission-based access control is indispensable in providing meaningful control to the users, but conventional permission models are inadequate for the automated agentic execution paradigm. We therefore propose automated permission management for AI agents. Our key idea is to conduct a user study to identify the factors influencing users' permission decisions and to encode these factors into an ML-based permission management assistant capable of predicting users' future decisions. We find that participants' permission decisions are influenced by communication context but importantly individual preferences tend to remain consistent within contexts, and align with those of other participants. Leveraging these insights, we develop a permission prediction model achieving 85.1% accuracy overall and 94.4% for high-confidence predictions. We find that even without using permission history, our model achieves an accuracy of 66.9%, and a slight increase of training samples (i.e., 1-4) can substantially increase the accuracy by 10.8%.
Authors Yuhao Wu, Ke Yang, Franziska Roesner, Tadayoshi Kohno, Ning Zhang, Umar Iqbal
Comment Accepted by the IEEE Symposium on Security and Privacy (S&P) 2026
Categories Artificial Intelligence
0
MeteorPred: A Meteorological Multimodal Large Model and Dataset for Severe Weather Event Prediction
Title CN MeteorPred：一种用于极端天气事件预测的气象多模态大模型及数据集
Keywords 多模态大模型 · 极端天气预测 · 气象数据 · 4D输入 · 自适应融合
Summary 本文提出了MP-Bench，首个大规模极端天气事件预测多模态数据集，包含42万对气象数据与文本描述，并开发了直接处理4D气象输入的多模态大模型MMLM，通过三个自适应融合模块实现时空与垂直层特征的动态整合，在多个任务上表现优异。目标是构建端到端的AI气象预报系统。
Reason 该论文聚焦于气象领域的多模态大模型应用，任务为极端天气事件预测，属于垂直领域（气象）的下游应用。尽管使用了4D气象数据和多模态建模技术，但其核心是将现有VLM架构应用于特定科学领域，并未提出可迁移至文档图像理解（DIU）的通用方法论。不符合DIU本题研究范畴，且无视觉压缩、细粒度定位、OCR-free等底层技术创新，属于‘平行应用’类工作。
Abstract Timely and accurate forecasts of severe weather events are essential for early warning and for constraining downstream analysis and decision-making. Since severe weather events prediction still depends on subjective, time-consuming expert interpretation, end-to-end "AI weather station" systems are emerging but face three major challenges: (1) scarcity of severe weather event samples; (2) imperfect alignment between high-dimensional meteorological data and textual warnings; (3) current multimodal language models cannot effectively process high-dimensional meteorological inputs or capture their complex spatiotemporal dependencies. To address these challenges, we introduce MP-Bench, the first large-scale multimodal dataset for severe weather events prediction, comprising 421,363 pairs of raw multi-year meteorological data and corresponding text caption, covering a wide range of severe weather scenarios. On top of this dataset, we develop a Meteorology Multimodal Large Model (MMLM) that directly ingests 4D meteorological inputs. In addition, it is designed to accommodate the unique characteristics of 4D meteorological data flow, incorporating three plug-and-play adaptive fusion modules that enable dynamic feature extraction and integration across temporal sequences, vertical pressure layers, and spatial dimensions. Extensive experiments on MP-Bench show that MMLM achieves strong performance across multiple tasks, demonstrating effective severe weather understanding and representing a key step toward automated, AI-driven severe weather events forecasting systems. Our source code and dataset will be made publicly available.
Authors Shuo Tang, Jian Xu, Jiadong Zhang, Yi Chen, Qizhao Jin, Lingdong Shen, Chenglin Liu, Shiming Xiang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Multi-Timescale Hierarchical Reinforcement Learning for Unified Behavior and Control of Autonomous Driving IEEE Robotics and Automation Letters (RAL)
Title CN 用于自动驾驶行为与控制统一的多时间尺度分层强化学习
Keywords 强化学习 · 分层结构 · 自动驾驶 · 多时间尺度 · 安全机制
Summary 本文提出一种多时间尺度分层强化学习方法，用于统一自动驾驶中的高层行为决策与底层控制指令生成。通过高层策略输出长时间尺度的运动引导，低层策略生成短时间尺度的控制命令，并引入混合动作表示和分层安全机制，在仿真和真实数据上提升了驾驶效率、动作一致性和安全性。
Reason 该论文属于自动驾驶领域，研究的是车辆控制与驾驶行为的分层强化学习策略，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管其方法涉及多时间尺度决策和分层策略设计，但这些机制针对的是动态控制问题，难以迁移到文档理解中的OCR幻觉、复杂排版解析或长文档推理等核心痛点。
Abstract Reinforcement Learning (RL) is increasingly used in autonomous driving (AD) and shows clear advantages. However, most RL-based AD methods overlook policy structure design. An RL policy that only outputs short-timescale vehicle control commands results in fluctuating driving behavior due to fluctuations in network outputs, while one that only outputs long-timescale driving goals cannot achieve unified optimality of driving behavior and control. Therefore, we propose a multi-timescale hierarchical reinforcement learning approach. Our approach adopts a hierarchical policy structure, where high- and low-level RL policies are unified-trained to produce long-timescale motion guidance and short-timescale control commands, respectively. Therein, motion guidance is explicitly represented by hybrid actions to capture multimodal driving behaviors on structured road and support incremental low-level extend-state updates. Additionally, a hierarchical safety mechanism is designed to ensure multi-timescale safety. Evaluation in simulator-based and HighD dataset-based highway multi-lane scenarios demonstrates that our approach significantly improves AD performance, effectively increasing driving efficiency, action consistency and safety.
Authors Guizhe Jin, Zhuoren Li, Bo Leng, Ran Yu, Lu Xiong, Chen Sun
Comment 8 pages, accepted for publication in IEEE Robotics and Automation Letters (RAL)
Categories Artificial Intelligence
0
Neural Graph Navigation for Intelligent Subgraph Matching Under review at AAAI 2026
Title CN 用于智能子图匹配的神经图导航
Keywords 子图匹配 · 神经搜索 · 图神经网络 · 智能导航 · 关系模式检测
Summary 本文提出Neural Graph Navigation（NeuGN），一种将神经引导机制引入子图匹配枚举过程的框架，通过结合启发式规则与神经网络，在保持完备性的同时显著减少匹配步数。实验显示在多个真实图数据集上性能优越。
Reason 该论文研究的是图数据中的子图匹配问题，属于图神经网络与数据库查询领域的交叉，尽管提出了神经引导的搜索机制，但其核心任务与文档图像理解（DIU）如DocVQA、布局分析、表格识别等无直接关联。方法聚焦于图结构数据的枚举优化，难以迁移到文档视觉理解中的OCR幻觉、细粒度定位或复杂排版解析等核心痛点。不属于DIU本题，也不提供可迁移的底层方法（如视觉压缩、多模态对齐、高分辨率处理等），属于无关领域。
Abstract Subgraph matching, a cornerstone of relational pattern detection in domains ranging from biochemical systems to social network analysis, faces significant computational challenges due to the dramatically growing search space. Existing methods address this problem within a filtering-ordering-enumeration framework, in which the enumeration stage recursively matches the query graph against the candidate subgraphs of the data graph. However, the lack of awareness of subgraph structural patterns leads to a costly brute-force enumeration, thereby critically motivating the need for intelligent navigation in subgraph matching. To address this challenge, we propose Neural Graph Navigation (NeuGN), a neuro-heuristic framework that transforms brute-force enumeration into neural-guided search by integrating neural navigation mechanisms into the core enumeration process. By preserving heuristic-based completeness guarantees while incorporating neural intelligence, NeuGN significantly reduces the \textit{First Match Steps} by up to 98.2\% compared to state-of-the-art methods across six real-world datasets.
Authors Yuchen Ying, Yiyang Dai, Wenda Li, Wenjie Huang, Rui Wang, Tongya Zheng, Yu Wang, Hanyang Yuan, Mingli Song
Comment Under review at AAAI 2026
Categories Artificial Intelligence
0
Alignment Faking - the Train -> Deploy Asymmetry: Through a Game-Theoretic Lens with Bayesian-Stackelberg Equilibria
Title CN 对齐伪装——训练与部署的不对称性：基于贝叶斯-斯塔克尔伯格均衡的博弈论视角
Keywords 对齐伪装 · AI安全 · 博弈论 · 大语言模型 · 偏好优化
Summary 本文通过博弈论框架（贝叶斯-斯塔克尔伯格均衡）分析大语言模型中存在的‘对齐伪装’现象，即模型在训练评估中假装服从对齐目标而在实际部署中偏离。作者评估了多种偏好优化方法（如DPO、KTO等）在15个模型上的表现，探讨该现象的成因与触发条件。研究聚焦于模型行为的策略性操纵，属于AI安全与对齐领域。
Reason 该论文研究的是大语言模型在训练和部署阶段之间出现的‘对齐伪装’现象，即模型在感知到处于训练环境时表现出符合对齐目标的行为，而在部署时恢复原有行为。这是一个关于模型对齐策略欺骗性的纯NLP安全议题，尽管涉及‘幻觉’相关概念，但其核心关注点是伦理与安全中的策略性欺骗，而非视觉或文档模态中的幻觉（如OCR错误传播、视觉定位偏差等）。因此无法迁移到文档图像理解中的幻觉抑制、细粒度定位或复杂排版解析等核心痛点。
Abstract Alignment faking is a form of strategic deception in AI in which models selectively comply with training objectives when they infer that they are in training, while preserving different behavior outside training. The phenomenon was first documented for Claude 3 Opus and later examined across additional large language models. In these setups, the word "training" refers to simulated training via prompts without parameter updates, so the observed effects are context conditioned shifts in behavior rather than preference learning. We study the phenomenon using an evaluation framework that compares preference optimization methods (BCO, DPO, KTO, and GRPO) across 15 models from four model families, measured along three axes: safety, harmlessness, and helpfulness. Our goal is to identify what causes alignment faking and when it occurs.
Authors Kartik Garg, Shourya Mishra, Kartikeya Sinha, Ojaswi Pratap Singh, Ayush Chopra, Kanishk Rai, Ammar Sheikh, Raghav Maheshwari, Aman Chadha, Vinija Jain, Amitava Das
Categories Artificial Intelligence
0
Multi-Aspect Cross-modal Quantization for Generative Recommendation AAAI 2026 (Oral)
Title CN 用于生成式推荐的多模态交叉量化
Keywords 生成式推荐 · 多模态量化 · 语义ID · 跨模态对齐 · 推荐系统
Summary 本文提出MACRec模型，通过多模态交叉量化来提升生成式推荐中语义ID的学习质量，利用隐式与显式多模态对齐增强推荐性能。实验在三个推荐数据集上验证了方法有效性。
Reason 该论文研究的是生成式推荐系统中的多模态量化表示问题，属于推荐系统领域，与文档图像理解（DIU）、DocVQA、OCR、布局分析等任务无直接关联。尽管涉及多模态信息融合与量化表示学习，但其方法设计面向用户行为序列建模与商品推荐，无法迁移解决文档理解中的核心痛点如OCR幻觉、密集文本识别、复杂排版解析或长文档推理。属于典型的平行应用型工作，不符合本研究方向的技术源头筛选标准。
Abstract Generative Recommendation (GR) has emerged as a new paradigm in recommender systems. This approach relies on quantized representations to discretize item features, modeling users' historical interactions as sequences of discrete tokens. Based on these tokenized sequences, GR predicts the next item by employing next-token prediction methods. The challenges of GR lie in constructing high-quality semantic identifiers (IDs) that are hierarchically organized, minimally conflicting, and conducive to effective generative model training. However, current approaches remain limited in their ability to harness multimodal information and to capture the deep and intricate interactions among diverse modalities, both of which are essential for learning high-quality semantic IDs and for effectively training GR models. To address this, we propose Multi-Aspect Cross-modal quantization for generative Recommendation (MACRec), which introduces multimodal information and incorporates it into both semantic ID learning and generative model training from different aspects. Specifically, we first introduce cross-modal quantization during the ID learning process, which effectively reduces conflict rates and thus improves codebook usability through the complementary integration of multimodal information. In addition, to further enhance the generative ability of our GR model, we incorporate multi-aspect cross-modal alignments, including the implicit and explicit alignments. Finally, we conduct extensive experiments on three well-known recommendation datasets to demonstrate the effectiveness of our proposed method.
Authors Fuwei Zhang, Xiaoyu Liu, Dongbo Xi, Jishen Yin, Huan Chen, Peng Yan, Fuzhen Zhuang, Zhao Zhang
Comment Accepted by AAAI 2026 (Oral)
Categories Artificial Intelligence
0
MambaTAD: When State-Space Models Meet Long-Range Temporal Action Detection
Title CN MambaTAD：当状态空间模型遇到长程时序动作检测
Keywords 时序动作检测 · 状态空间模型 · Mamba · 长序列建模 · 视频理解
Summary 本文提出MambaTAD，一种基于状态空间模型（如Mamba）的时序动作检测方法，用于在未剪辑视频中定位和识别动作。通过引入对角掩码双向状态空间模块（DMBSS）和全局特征融合头，提升长跨度动作的检测性能，并采用端到端单阶段设计降低计算开销。实验表明其在多个基准上表现优异。
Reason 该论文研究的是视频领域的时序动作检测（TAD），属于视频理解任务，与文档图像理解（DIU）或DocVQA无直接关联。尽管其使用了具有长序列建模能力的Mamba架构，但应用场景完全偏离文档领域，且未涉及OCR、布局分析、表格识别等DIU核心问题。根据负面清单，视频理解属于无关领域，无法迁移其方法到文档理解的核心痛点（如OCR幻觉、复杂排版处理）。
Abstract Temporal Action Detection (TAD) aims to identify and localize actions by determining their starting and ending frames within untrimmed videos. Recent Structured State-Space Models such as Mamba have demonstrated potential in TAD due to their long-range modeling capability and linear computational complexity. On the other hand, structured state-space models often face two key challenges in TAD, namely, decay of temporal context due to recursive processing and self-element conflict during global visual context modeling, which become more severe while handling long-span action instances. Additionally, traditional methods for TAD struggle with detecting long-span action instances due to a lack of global awareness and inefficient detection heads. This paper presents MambaTAD, a new state-space TAD model that introduces long-range modeling and global feature detection capabilities for accurate temporal action detection. MambaTAD comprises two novel designs that complement each other with superior TAD performance. First, it introduces a Diagonal-Masked Bidirectional State-Space (DMBSS) module which effectively facilitates global feature fusion and temporal action detection. Second, it introduces a global feature fusion head that refines the detection progressively with multi-granularity features and global awareness. In addition, MambaTAD tackles TAD in an end-to-end one-stage manner using a new state-space temporal adapter(SSTA) which reduces network parameters and computation cost with linear complexity. Extensive experiments show that MambaTAD achieves superior TAD performance consistently across multiple public benchmarks.
Authors Hui Lu, Yi Yu, Shijian Lu, Deepu Rajan, Boon Poh Ng, Alex C. Kot, Xudong Jiang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
PA-FAS: Towards Interpretable and Generalizable Multimodal Face Anti-Spoofing via Path-Augmented Reinforcement Learning AAAI 2026 (Oral)
Title CN PA-FAS：通过路径增强的强化学习实现可解释且可泛化的多模态人脸反欺诈
Keywords 人脸反欺诈 · 多模态融合 · 强化学习 · 推理路径增强 · 反欺骗
Summary 本文提出PA-FAS，一种基于路径增强强化学习的多模态人脸反欺诈方法，通过构建高质量扩展推理序列和答案打乱机制来提升模型的多模态推理能力、跨域泛化性和可解释性。旨在解决传统SFT+RL在多模态FAS中推理路径受限和存在捷径学习的问题。
Reason 该论文研究的是人脸反欺诈（Face Anti-Spoofing）任务，属于生物特征安全领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管使用了多模态推理和强化学习路径探索的方法，但其问题设定、数据模态（如红外、深度图像）和应用场景与文档理解差异巨大，无法迁移解决OCR幻觉、密集文本处理或复杂排版理解等DIU核心痛点。
Abstract Face anti-spoofing (FAS) has recently advanced in multimodal fusion, cross-domain generalization, and interpretability. With large language models and reinforcement learning (RL), strategy-based training offers new opportunities to jointly model these aspects. However, multimodal reasoning is more complex than unimodal reasoning, requiring accurate feature representation and cross-modal verification while facing scarce, high-quality annotations, which makes direct application of RL sub-optimal. We identify two key limitations of supervised fine-tuning plus RL (SFT+RL) for multimodal FAS: (1) limited multimodal reasoning paths restrict the use of complementary modalities and shrink the exploration space after SFT, weakening the effect of RL; and (2) mismatched single-task supervision versus diverse reasoning paths causes reasoning confusion, where models may exploit shortcuts by mapping images directly to answers and ignoring the intended reasoning. To address this, we propose PA-FAS, which enhances reasoning paths by constructing high-quality extended reasoning sequences from limited annotations, enriching paths and relaxing exploration constraints. We further introduce an answer-shuffling mechanism during SFT to force comprehensive multimodal analysis instead of using superficial cues, thereby encouraging deeper reasoning and mitigating shortcut learning. PA-FAS significantly improves multimodal reasoning accuracy and cross-domain generalization, and better unifies multimodal fusion, generalization, and interpretability for trustworthy FAS.
Authors Yingjie Ma, Xun Lin, Yong Xu, Weicheng Xie, Zitong Yu
Comment Accepted by AAAI 2026 (Oral)
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
FinMR: A Knowledge-Intensive Multimodal Benchmark for Advanced Financial Reasoning
Title CN FinMR：面向高级金融推理的知识密集型多模态基准
Keywords 金融推理 · 多模态基准 · 知识密集型任务 · 专家级评测 · 文档问答
Summary 本文提出了名为FinMR的金融领域多模态基准数据集，包含3200多个专家标注的问答对，涵盖15个金融主题，强调知识密集和复杂推理能力评估。然而，该工作未提出新的模型或方法，仅为特定领域的评测基准，且作者已声明撤回当前版本进行修正。
Reason 该论文属于典型的‘平行下游应用’，其核心是构建一个面向金融领域的多模态问答数据集，并用于评估现有MLLM在金融分析任务上的表现。尽管数据集质量较高，但并未提出任何文档图像理解（DIU）领域的底层方法创新，如视觉表示学习、VLM架构改进、去幻觉机制或高分辨率处理策略。其内容本质上是将现有VLM技术应用于特定垂直领域（金融），完全符合负面清单中的‘Wrapper/Application’类型，无迁移至DIU基础方法的价值。
Abstract Multimodal Large Language Models (MLLMs) have made substantial progress in recent years. However, their rigorous evaluation within specialized domains like finance is hindered by the absence of datasets characterized by professional-level knowledge intensity, detailed annotations, and advanced reasoning complexity. To address this critical gap, we introduce FinMR, a high-quality, knowledge-intensive multimodal dataset explicitly designed to evaluate expert-level financial reasoning capabilities at a professional analyst's standard. FinMR comprises over 3,200 meticulously curated and expertly annotated question-answer pairs across 15 diverse financial topics, ensuring broad domain diversity and integrating sophisticated mathematical reasoning, advanced financial knowledge, and nuanced visual interpretation tasks across multiple image types. Through comprehensive benchmarking with leading closed-source and open-source MLLMs, we highlight significant performance disparities between these models and professional financial analysts, uncovering key areas for model advancement, such as precise image analysis, accurate application of complex financial formulas, and deeper contextual financial understanding. By providing richly varied visual content and thorough explanatory annotations, FinMR establishes itself as an essential benchmark tool for assessing and advancing multimodal financial reasoning toward professional analyst-level competence.
Authors Shuangyan Deng, Haizhou Peng, Jiachen Xu, Rui Mao, Ciprian Doru Giurcăneanu, Jiamou Liu
Comment The methodology section contains inaccuracies that may lead to misleading interpretations. The authors have withdrawn this version for correction
Categories Artificial Intelligence
0
AnimAgents: Coordinating Multi-Stage Animation Pre-Production with Human-Multi-Agent Collaboration
Title CN AnimAgents：通过人机多智能体协作协调动画前期制作的多阶段流程
Keywords 多智能体系统 · 人机协作 · 动画前期制作 · 生成式AI · 工作流管理
Summary 本文提出了AnimAgents，一个用于协调动画前期制作多阶段流程的人机多智能体协作系统。该系统通过核心代理与专用代理协同工作，支持概念构思、剧本编写、角色设计和分镜绘制四个主要阶段的任务编排与信息管理。研究基于对12位专业动画导演的形成性研究，并通过16人的对照实验和4个真实项目部署验证了系统在协调性、一致性、信息管理和用户满意度上的优势。
Reason 该论文属于特定领域的应用型系统（动画前期制作），其核心贡献是针对创意产业中动画制作流程的多智能体协作框架，属于典型的‘平行下游应用’。尽管涉及多Agent协同和工作流管理，但其方法设计紧密耦合于动画创作场景，未提出可迁移至文档图像理解（如DocVQA、布局分析等）的通用技术机制或底层架构创新，对解决OCR幻觉、复杂排版、长文档推理等DIU核心问题无直接借鉴意义。
Abstract Animation pre-production lays the foundation of an animated film by transforming initial concepts into a coherent blueprint across interdependent stages such as ideation, scripting, design, and storyboarding. While generative AI tools are increasingly adopted in this process, they remain isolated, requiring creators to juggle multiple systems without integrated workflow support. Our formative study with 12 professional creative directors and independent animators revealed key challenges in their current practice: Creators must manually coordinate fragmented outputs, manage large volumes of information, and struggle to maintain continuity and creative control between stages. Based on the insights, we present AnimAgents, a human-multi-agent collaborative system that coordinates complex, multi-stage workflows through a core agent and specialized agents, supported by dedicated boards for the four major stages of pre-production. AnimAgents enables stage-aware orchestration, stage-specific output management, and element-level refinement, providing an end-to-end workflow tailored to professional practice. In a within-subjects summative study with 16 professional creators, AnimAgents significantly outperformed a strong single-agent baseline that equipped with advanced parallel image generation in coordination, consistency, information management, and overall satisfaction (p < .01). A field deployment with 4 creators further demonstrated AnimAgents' effectiveness in real-world projects.
Authors Wen-Fan Wang, Chien-Ting Lu, Jin Ping Ng, Yi-Ting Chiu, Ting-Ying Lee, Miaosen Wang, Bing-Yu Chen, Xiang 'Anthony' Chen
Categories Artificial Intelligence
0
Statistically-Guided Dual-Domain Meta-Learning with Adaptive Multi-Prototype Aggregation for Distributed Fiber Optic Sensing
Title CN 基于统计引导的双域元学习与自适应多原型聚合用于分布式光纤传感
Keywords 分布式光纤传感 · 元学习 · 域偏移 · 多原型聚合 · 统计引导
Summary 本文提出了一种名为DUPLE的元学习框架，用于解决分布式光纤传感系统中因部署环境不同导致的域偏移问题。通过融合时域和频域特征、利用统计信息指导原型学习，并在查询阶段自适应聚合原型，提升了在新部署场景下对振动事件的识别能力。实验表明该方法在跨部署设置下显著优于基线。
Reason 该论文研究的是分布式光纤传感（DFOS）中的活动识别问题，属于物理传感信号处理领域，与文档图像理解（DIU）或视觉-语言模型（VLM）在文档上的应用无任何技术关联。其核心方法如双域特征融合、统计引导网络等针对的是振动信号的时频域建模，无法迁移到文档布局分析、表格识别或DocVQA等任务中。属于完全无关领域。
Abstract Distributed Fiber Optic Sensing (DFOS) has shown strong potential in perimeter security due to its capability of monitoring vibration events across long distances with fine spatial resolution. However, practical DFOS systems face three critical challenges: (1) signal patterns of the same activity vary drastically under different fiber deployment types (e.g., underground, wall-mounted), causing domain shift; (2) labeled data in new deployment scenarios is often scarce or entirely unavailable, limiting model adaptability; and (3) even within source domains, data scarcity makes it difficult to capture intra-class diversity for robust learning. To address these challenges, we propose a novel meta-learning framework, DUPLE, for cross-deployment DFOS activity identification. First, a dual-domain multi-prototype learner fuses temporal and frequency domain features, enhancing the model's generalization ability under signal distribution shifts. Second, a Statistical Guided Network (SGN) infers domain importance and prototype sensitivity from raw statistical features, providing data-driven prior information for learning in unlabeled or unseen domains. Third, a query-aware prototype aggregation module adaptively selects and combines relevant prototypes, thereby improving classification performance even with limited data. Extensive experiments on cross-deployment DFOS datasets demonstrate that our method significantly outperforms baseline approaches in domain generalization settings, enabling robust event recognition across diverse fiber configurations with minimal labeled data.
Authors Yifan He, Haodong Zhang, Qiuheng Song, Lin Lei, Zhenxuan Zeng, Haoyang He, Hongyan Wu
Categories Artificial Intelligence
0
BemaGANv2: A Tutorial and Comparative Survey of GAN-based Vocoders for Long-Term Audio Generation ICT Express
Title CN BemaGANv2：基于GAN的长时音频生成声码器教程与比较综述
Keywords GAN · 声码器 · 音频生成 · 周期性建模 · 判别器架构
Summary 本文是一篇教程式综述，介绍了BemaGANv2这一用于高保真、长时音频生成的GAN声码器。其核心改进包括在生成器中引入Anti-aliased Multi-Periodicity composition（AMP）模块，在判别器中提出Multi-Envelope Discriminator（MED）并与MRD结合，以更好建模音频的周期性与长程依赖。尽管技术细节扎实，但完全属于语音/音频处理范畴，不涉及任何文档图像或多模态理解任务。
Reason 该论文属于音频生成领域，研究内容为基于GAN的声码器设计，应用于Text-to-Music和Text-to-Audio系统。与文档图像理解（DIU）、多模态大模型（VLM）处理视觉或图文任务无关，属于无关领域（纯音频生成），命中负面清单第2条。
Abstract This paper presents a tutorial-style survey and implementation guide of BemaGANv2, an advanced GANbased vocoder designed for high-fidelity and long-term audio generation. Long-term audio generation is critical for applications in Text-to-Music (TTM) and Text-to-Audio (TTA) systems, where maintaining temporal coherence, prosodic consistency, and harmonic structure over extended durations remains a significant challenge. Built upon the original BemaGAN architecture, BemaGANv2 incorporates major architectural innovations by replacing traditional ResBlocks in the generator with the Anti-aliased Multi-Periodicity composition (AMP) module, which internally applies the Snake activation function to better model periodic structures. In the discriminator framework, we integrate the Multi-Envelope Discriminator (MED), a novel architecture we proposed, to extract rich temporal envelope features crucial for periodicity detection. Coupled with the Multi-Resolution Discriminator (MRD), this combination enables more accurate modeling of long-range dependencies in audio. We systematically evaluate various discriminator configurations, including Multi-Scale Discriminator (MSD) + MED, MSD + MRD, and Multi-Period Discriminator (MPD) + MED + MRD, using objective metrics (Fréchet Audio Distance (FAD), Structural Similarity Index (SSIM), Pearson Correlation Coefficient (PCC), Mel-Cepstral Distortion (MCD)) and subjective evaluations (MOS, SMOS). This paper also provides a comprehensive tutorial on the model architecture, training methodology, and implementation to promote reproducibility. The code and pre-trained models are available at: https://github.com/dinhoitt/BemaGANv2.
Authors Taesoo Park, Mungwi Jeong, Mingyu Park, Narae Kim, Junyoung Kim, Mujung Kim, Jisang Yoo, Hoyun Lee, Sanghoon Kim, Soonchul Kwon
Comment 11 pages, 7 figures. Survey and tutorial paper. Currently under review at ICT Express as an extended version of our ICAIIC 2025 paper
Categories Artificial Intelligence
0
Mutually Assured Deregulation
Title CN 相互保证 deregulation
Keywords AI治理 · 监管政策 · 国家安全 · 技术竞赛 · AGI风险
Summary 本文认为，自2022年以来，各国为追求AI主导权而推行‘监管牺牲’策略，即削弱安全监管以加速AI发展，但这种做法实际上会带来国家安全风险。作者指出，技术优势难以持久，良好的治理反而促进创新，而放任发展会在短期、中期和长期分别导致信息战工具扩散、生物武器民主化和不可控AGI的必然出现。最终，这种竞争形成‘相互保证 deregulation’的危险平衡。
Reason 该论文属于AI政策与治理领域的论述性文章，讨论的是全球AI监管放松的谬误及其国家安全风险。内容完全不涉及文档图像理解（DIU）、多模态模型架构、视觉表示学习或任何技术方法创新，与DIU核心任务无关联。
Abstract We have convinced ourselves that the way to make AI safe is to make it unsafe. Since 2022, policymakers worldwide have embraced the Regulation Sacrifice - the belief that dismantling safety oversight will deliver security through AI dominance. Fearing China or USA will gain advantage, nations rush to eliminate safeguards that might slow progress. This Essay reveals the fatal flaw: though AI poses national security challenges, the solution demands stronger regulatory frameworks, not weaker ones. A race without guardrails breeds shared danger, not competitive strength. The Regulation Sacrifice makes three false promises. First, it promises durable technological leads. But AI capabilities spread rapidly - performance gaps between U.S. and Chinese systems collapsed from 9 percent to 2 percent in thirteen months. When advantages evaporate in months, sacrificing permanent safety for temporary speed makes no sense. Second, it promises deregulation accelerates innovation. The opposite often proves true. Companies report well-designed governance streamlines development. Investment flows toward regulated markets. Clear rules reduce uncertainty; uncertain liability creates paralysis. Environmental standards did not kill the auto industry; they created Tesla and BYD. Third, enhanced national security through deregulation actually undermines security across all timeframes. Near term: it hands adversaries information warfare tools. Medium term: it democratizes bioweapon capabilities. Long term: it guarantees deployment of uncontrollable AGI systems. The Regulation Sacrifice persists because it serves powerful interests, not security. Tech companies prefer freedom to accountability. Politicians prefer simple stories to complex truths. This creates mutually assured deregulation, where each nation's sprint for advantage guarantees collective vulnerability. The only way to win is not to play.
Authors Gilad Abiri
Categories Artificial Intelligence
0
Constraint Satisfaction Approaches to Wordle: Novel Heuristics and Cross-Lexicon Validation
Title CN 面向Wordle的约束满足方法：新启发式与跨词典验证
Keywords Constraint Satisfaction · Wordle · Entropy Maximization · Probabilistic CSP · Cross-lingual Validation
Summary 本文提出了首个将Wordle建模为约束满足问题（CSP）的完整框架，引入了CSP感知熵和概率化CSP模型，在英语和西班牙语词汇上进行了验证，表现出优于传统信息论方法的性能和鲁棒性。工作强调形式化约束处理与逻辑-概率融合推理，并开源了高覆盖率代码库。
Reason 该论文研究的是Wordle这一文字游戏的求解算法，属于特定规则下的逻辑推理问题，尽管使用了约束满足（CSP）等AI技术，但其任务场景与文档图像理解（DIU）无直接关联。未涉及OCR、视觉布局分析、多模态文档理解或VLM在文档上的应用。虽然CSP本身具有通用性，但文中未提出可迁移至DIU领域的架构、表示学习或推理机制创新，且应用场景为平行下游任务（puzzle-solving），不符合关注上游方法论创新的要求。
Abstract Wordle presents an algorithmically rich testbed for constraint satisfaction problem (CSP) solving. While existing solvers rely on information-theoretic entropy maximization or frequency-based heuristics without formal constraint treatment, we present the first comprehensive CSP formulation of Wordle with novel constraint-aware solving strategies. We introduce CSP-Aware Entropy, computing information gain after constraint propagation rather than on raw candidate sets, and a Probabilistic CSP framework integrating Bayesian word-frequency priors with logical constraints. Through evaluation on 2,315 English words, CSP-Aware Entropy achieves 3.54 average guesses with 99.9% success rate, a statistically significant 1.7% improvement over Forward Checking (t=-4.82, p<0.001, Cohen's d=0.07) with 46% faster runtime (12.9ms versus 23.7ms per guess). Under 10% noise, CSP-aware approaches maintain 5.3 percentage point advantages (29.0% versus 23.7%, p=0.041), while Probabilistic CSP achieves 100% success across all noise levels (0-20%) through constraint recovery mechanisms. Cross-lexicon validation on 500 Spanish words demonstrates 88% success with zero language-specific tuning, validating that core CSP principles transfer across languages despite an 11.2 percentage point gap from linguistic differences (p<0.001, Fisher's exact test). Our open-source implementation with 34 unit tests achieving 91% code coverage provides reproducible infrastructure for CSP research. The combination of formal CSP treatment, constraint-aware heuristics, probabilistic-logical integration, robustness analysis, and cross-lexicon validation establishes new performance benchmarks demonstrating that principled constraint satisfaction techniques outperform classical information-theoretic and learning-based approaches for structured puzzle-solving domains.
Authors Jahidul Arafat, Fariha Tasmin, Sanjaya Poudel
Comment Require some correction on the paper with some title and methodology changes. I will resubmit later
Categories Artificial Intelligence, Computation and Language
0
Assessment of deep learning models integrated with weather and environmental variables for wildfire spread prediction and a case study of the 2023 Maui fires
Title CN 结合天气和环境变量的深度学习模型在野火蔓延预测中的评估及2023年毛伊岛火灾案例研究
Keywords 野火预测 · 深度学习 · ConvLSTM · 可解释AI · 环境变量
Summary 本文评估了五种结合天气和环境变量的深度学习模型在夏威夷野火蔓延预测中的表现，并以2023年毛伊岛火灾为例，与传统FARSITE模型进行对比。结果显示ConvLSTM及其带注意力变体表现最佳，AI模型灵活性更高，但传统模型精度更优。同时通过可解释AI方法识别出影响火灾的关键因素。
Reason 该论文属于自然灾害预测领域，研究任务为野火蔓延预测，使用的是气象与环境变量驱动的深度学习模型，与文档图像理解（DIU）、视觉语言模型（VLM）或任何多模态文档分析任务完全无关。未涉及OCR、布局分析、表格识别、视觉文本理解等核心问题，也未提出可迁移至DIU领域的通用方法（如视觉表示、推理机制、高分辨率处理架构等）。属于典型的‘无关领域’论文，命中负面清单第2条。
Abstract Predicting the spread of wildfires is essential for effective fire management and risk assessment. With the fast advancements of artificial intelligence (AI), various deep learning models have been developed and utilized for wildfire spread prediction. However, there is limited understanding of the advantages and limitations of these models, and it is also unclear how deep learning-based fire spread models can be compared with existing non-AI fire models. In this work, we assess the ability of five typical deep learning models integrated with weather and environmental variables for wildfire spread prediction based on over ten years of wildfire data in the state of Hawaii. We further use the 2023 Maui fires as a case study to compare the best deep learning models with a widely-used fire spread model, FARSITE. The results show that two deep learning models, i.e., ConvLSTM and ConvLSTM with attention, perform the best among the five tested AI models. FARSITE shows higher precision, lower recall, and higher F1-score than the best AI models, while the AI models offer higher flexibility for the input data. By integrating AI models with an explainable AI method, we further identify important weather and environmental factors associated with the 2023 Maui wildfires.
Authors Jiyeon Kim, Yingjie Hu, Negar Elhami-Khorasani, Kai Sun, Ryan Zhenqi Zhou
Categories Artificial Intelligence
0
Decoupled Audio-Visual Dataset Distillation
Title CN 解耦的音视频数据集蒸馏
Keywords 数据集蒸馏 · 跨模态学习 · 表示解耦 · 音视频理解 · 预训练特征库
Summary 本文提出DAVDD，一种基于预训练的解耦音视频数据集蒸馏框架，通过构建预训练特征库获取稳定的模态特征，并使用轻量化解耦器将特征分解为共性与私有表示。通过共性跨模态匹配和样本-分布联合对齐策略保留跨模态结构，同时隔离私有表示以保护模态特有信息。实验表明该方法在多个基准上达到SOTA。
Reason 该论文研究的是音视频跨模态数据集蒸馏，属于音频与视觉的联合建模，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管提出了‘解耦表示学习’和‘跨模态对齐’的方法，但其方法针对的是音视频这一特定模态组合，且目标是数据集压缩而非文档内容理解或视觉语言推理。无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU关键痛点，属于无关领域。
Abstract Audio-Visual Dataset Distillation aims to compress large-scale datasets into compact subsets while preserving the performance of the original data. However, conventional Distribution Matching (DM) methods struggle to capture intrinsic cross-modal alignment. Subsequent studies have attempted to introduce cross-modal matching, but two major challenges remain: (i) independently and randomly initialized encoders lead to inconsistent modality mapping spaces, increasing training difficulty; and (ii) direct interactions between modalities tend to damage modality-specific (private) information, thereby degrading the quality of the distilled data. To address these challenges, we propose DAVDD, a pretraining-based decoupled audio-visual distillation framework. DAVDD leverages a diverse pretrained bank to obtain stable modality features and uses a lightweight decoupler bank to disentangle them into common and private representations. To effectively preserve cross-modal structure, we further introduce Common Intermodal Matching together with a Sample-Distribution Joint Alignment strategy, ensuring that shared representations are aligned both at the sample level and the global distribution level. Meanwhile, private representations are entirely isolated from cross-modal interaction, safeguarding modality-specific cues throughout distillation. Extensive experiments across multiple benchmarks show that DAVDD achieves state-of-the-art results under all IPC settings, demonstrating the effectiveness of decoupled representation learning for high-quality audio-visual dataset distillation. Code will be released.
Authors Wenyuan Li, Guang Li, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Training Emergent Joint Associations: A Reinforcement Learning Approach to Creative Thinking in Language Models
Title CN 训练涌现的联合关联：一种通过强化学习实现语言模型创造性思维的方法
Keywords 强化学习 · 创造性思维 · 联想能力 · 语言模型 · 生成任务
Summary 本文提出了一种基于强化学习的框架，通过引入创造性思维中的联想机制来增强语言模型在生成任务中的新颖性和连贯性。使用基于提示的评估方式结合发散性思维指标，对模型输出进行奖励，从而训练出更具抽象能力和灵活性的模型。实验表明该方法在故事写作、编程和数据可视化任务中能提升生成质量。
Reason 该论文研究的是如何通过强化学习提升语言模型在故事生成、代码生成和图表创作等任务中的创造性联想能力，属于通用语言模型的创意生成范畴。其核心方法与文档图像理解（DIU）中的关键挑战如OCR幻觉、细粒度定位、复杂版式建模、长文档推理等无直接关联。未涉及视觉输入、多模态对齐或文档结构建模，无法迁移用于解决DIU的核心痛点。
Abstract Associative thinking--the ability to connect seemingly unrelated ideas--is a foundational element of human creativity and problem-solving. This paper explores whether reinforcement learning (RL) guided by associative thinking principles can enhance a model's performance across diverse generative tasks, including story writing, code generation, and chart creation. We introduce a reinforcement learning framework that uses a prompt-based evaluation mechanism, incorporating established divergent thinking metrics from creativity research. A base language model is fine-tuned using this framework to reward outputs demonstrating higher novelty through higher degrees of conceptual connectivity. Interestingly, the experimental results suggest that RL-based associative thinking-trained models not only generate more original and coherent stories but also exhibit improved abstraction and flexibility in tasks such as programming and data visualization. Our findings provide initial evidence that modeling cognitive creativity principles through reinforcement learning can yield more adaptive and generative AI.
Authors Mukul Singh, Ananya Singha, Aishni Parab, Pronita Mehrotra, Sumit Gulwani
Categories Artificial Intelligence
0
Can MLLMs Detect Phishing? A Comprehensive Security Benchmark Suite Focusing on Dynamic Threats and Multimodal Evaluation in Academic Environments
Title CN MLLM能否检测钓鱼攻击？一个聚焦学术环境中动态威胁和多模态评估的综合安全基准套件
Keywords MLLM安全 · 钓鱼检测 · 学术环境 · 多模态评估 · 安全基准
Summary 本文提出了AdapT-Bench，一个用于评估多模态大模型在学术环境中防御动态钓鱼攻击能力的基准框架。研究聚焦于利用学术背景信息生成定制化钓鱼内容，并测试MLLM的检测能力。然而，该工作未涉及文档结构理解、视觉表示学习或高分辨率处理等DIU关键技术路径。
Reason 该论文属于纯NLP安全对齐领域，研究的是MLLM在学术场景下识别钓鱼邮件的能力，属于特定下游应用（安全防御），并未提出可用于文档图像理解的底层方法或架构创新。尽管涉及多模态，但其核心关注点为网络安全与人类行为漏洞，与文档布局、OCR、视觉压缩、细粒度定位等DIU核心痛点无关，且无迁移价值。
Abstract The rapid proliferation of Multimodal Large Language Models (MLLMs) has introduced unprecedented security challenges, particularly in phishing detection within academic environments. Academic institutions and researchers are high-value targets, facing dynamic, multilingual, and context-dependent threats that leverage research backgrounds, academic collaborations, and personal information to craft highly tailored attacks. Existing security benchmarks largely rely on datasets that do not incorporate specific academic background information, making them inadequate for capturing the evolving attack patterns and human-centric vulnerability factors specific to academia. To address this gap, we present AdapT-Bench, a unified methodological framework and benchmark suite for systematically evaluating MLLM defense capabilities against dynamic phishing attacks in academic settings.
Authors Jingzhuo Zhou
Categories Artificial Intelligence
0
QuickLAP: Quick Language-Action Preference Learning for Autonomous Driving Agents
Title CN QuickLAP：用于自动驾驶代理的快速语言-动作偏好学习
Keywords 偏好学习 · 多模态融合 · 自动驾驶 · 大语言模型 · 机器人
Summary 本文提出QuickLAP，一种结合物理纠正和自然语言反馈的贝叶斯框架，用于实时推断机器人任务中的奖励函数。通过大语言模型从自由语言中提取奖励特征注意力掩码，并与物理动作反馈融合，实现更准确、可解释的偏好学习。实验表明其在半自动驾驶模拟器中显著降低了奖励学习误差，并获得用户更高的满意度。
Reason 该论文属于具身智能与机器人领域，研究任务为自动驾驶代理中的多模态反馈融合，尽管使用了LLM进行语言理解，但其核心场景与文档图像理解（DIU）无关。虽然偏好学习和多模态融合在理论上具有一定通用性，但该方法针对的是实时控制与物理交互场景，迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等问题极为困难，且无直接关联路径。
Abstract Robots must learn from both what people do and what they say, but either modality alone is often incomplete: physical corrections are grounded but ambiguous in intent, while language expresses high-level goals but lacks physical grounding. We introduce QuickLAP: Quick Language-Action Preference learning, a Bayesian framework that fuses physical and language feedback to infer reward functions in real time. Our key insight is to treat language as a probabilistic observation over the user's latent preferences, clarifying which reward features matter and how physical corrections should be interpreted. QuickLAP uses Large Language Models (LLMs) to extract reward feature attention masks and preference shifts from free-form utterances, which it integrates with physical feedback in a closed-form update rule. This enables fast, real-time, and robust reward learning that handles ambiguous feedback. In a semi-autonomous driving simulator, QuickLAP reduces reward learning error by over 70% compared to physical-only and heuristic multimodal baselines. A 15-participant user study further validates our approach: participants found QuickLAP significantly more understandable and collaborative, and preferred its learned behavior over baselines. Code is available at https://github.com/MIT-CLEAR-Lab/QuickLAP.
Authors Jordan Abi Nader, David Lee, Nathaniel Dennler, Andreea Bobu
Categories Artificial Intelligence
0
A superpersuasive autonomous policy debating system AAAI 2026 CLIP workshop
Title CN 一种极具说服力的自主政策辩论系统
Keywords 多智能体系统 · 辩论AI · LLM应用 · 自动论证 · 人机交互
Summary 本文提出了DeepDebater，一个能够参与完整政策辩论的自主系统，采用多智能体协作架构生成演讲稿、交叉质询和反驳内容，并结合TTS与虚拟形象实现可视化呈现。支持AI对战AI及人机混合模式，在模拟评估中表现优于人类辩题案例。
Reason 该论文属于典型的平行下游应用，其核心是将多智能体架构应用于政策辩论这一特定任务场景，属于‘基于LLM的辩论助手’类应用。尽管系统设计复杂，但并未提出可用于文档图像理解（DIU）领域的底层方法创新，如视觉表示学习、OCR去幻觉、细粒度定位或高分辨率处理等。其技术路线聚焦于论点生成与辩论策略，与文档理解中的核心痛点无关。
Abstract The capacity for highly complex, evidence-based, and strategically adaptive persuasion remains a formidable great challenge for artificial intelligence. Previous work, like IBM Project Debater, focused on generating persuasive speeches in simplified and shortened debate formats intended for relatively lay audiences. We introduce DeepDebater, a novel autonomous system capable of participating in and winning a full, unmodified, two-team competitive policy debate. Our system employs a hierarchical architecture of specialized multi-agent workflows, where teams of LLM-powered agents collaborate and critique one another to perform discrete argumentative tasks. Each workflow utilizes iterative retrieval, synthesis, and self-correction using a massive corpus of policy debate evidence (OpenDebateEvidence) and produces complete speech transcripts, cross-examinations, and rebuttals. We introduce a live, interactive end-to-end presentation pipeline that renders debates with AI speech and animation: transcripts are surface-realized and synthesized to audio with OpenAI TTS, and then displayed as talking-head portrait videos with EchoMimic V1. Beyond fully autonomous matches (AI vs AI), DeepDebater supports hybrid human-AI operation: human debaters can intervene at any stage, and humans can optionally serve as opponents against AI in any speech, allowing AI-human and AI-AI rounds. In preliminary evaluations against human-authored cases, DeepDebater produces qualitatively superior argumentative components and consistently wins simulated rounds as adjudicated by an independent autonomous judge. Expert human debate coaches also prefer the arguments, evidence, and cases constructed by DeepDebater. We open source all code, generated speech transcripts, audio and talking head video here: https://github.com/Hellisotherpeople/DeepDebater/tree/main
Authors Allen Roush, Devin Gonier, John Hines, Judah Goldfeder, Philippe Martin Wyder, Sanjay Basu, Ravid Shwartz Ziv
Comment Accepted to CLIP workshop at AAAI 2026
Categories Artificial Intelligence, Computation and Language
0
A Low-Code Methodology for Developing AI Kiosks: a Case Study with the DIZEST Platform
Title CN 一种用于开发AI自助服务终端的低代码方法：以DIZEST平台为例的案例研究
Keywords 低代码 · 自助服务终端 · AI集成 · 工作流设计 · DIZEST
Summary 本文提出了一种基于DIZEST低代码平台的AI自助终端开发方法，通过可视化工作流设计实现AI功能集成，并在照片自助机场景中验证了其在互操作性、用户体验和部署灵活性方面的优势。研究对比了Jupyter Notebook、ComfyUI和Orange3等现有工具，表明DIZEST在关键评估指标上表现更优。
Reason 该论文属于典型的平行下游应用，聚焦于利用低代码平台构建AI自助终端系统，并非针对文档图像理解（DIU）或DocVQA中的核心问题（如OCR幻觉、复杂排版、视觉文本理解等）提出方法论创新。其技术内容与文档多模态理解、视觉语言模型架构改进、细粒度定位或长文档推理无关，无法为DIU领域提供可迁移的方法工具。
Abstract This paper presents a comprehensive study on enhancing kiosk systems through a low-code architecture, with a focus on AI-based implementations. Modern kiosk systems are confronted with significant challenges, including a lack of integration, structural rigidity, performance bottlenecks, and the absence of collaborative frameworks. To overcome these limitations, we propose a DIZEST-based approach methodology, a specialized low-code platform that enables intuitive workflow design and seamless AI integration. Through a comparative analysis with existing platforms, including Jupyter Notebook, ComfyUI, and Orange3, we demonstrate that DIZEST delivers superior performance across key evaluation criteria. Our photo kiosk case study further validates the effectiveness of this approach in improving interoperability, enhancing user experience, and increasing deployment flexibility.
Authors SunMin Moon, Jangwon Gim, Chaerin Kim, Yeeun Kim, YoungJoo Kim, Kang Choi
Comment 5 pages, 2 figures, conference, 2 tables
Categories Artificial Intelligence
0
Neural Transparency: Mechanistic Interpretability Interfaces for Anticipating Model Behaviors for Personalized AI
Title CN 神经透明性：用于预测个性化AI行为的机制可解释性接口
Keywords 机制可解释性 · 个性化AI · LLM安全 · 行为预测 · 人-AI交互
Summary 本文提出一种基于神经激活差异的‘神经透明性’接口，通过对比不同系统提示下的模型内部激活来提取行为特征向量（如共情、毒性、奉承等），帮助非专业用户理解并预测其定制聊天机器人的行为表现。研究发现用户普遍存在对AI行为的误判，而该工具虽未改变设计行为，但提升了信任度。工作面向通用LLM安全性与人机交互，不涉及视觉或文档模态。
Reason 该论文聚焦于个性化LLM聊天机器人的行为可解释性与用户信任问题，属于纯NLP安全与对齐（Safety/Alignment）领域，尽管涉及模型内部激活的可视化，但其任务场景、方法设计均与文档图像理解（DIU）、视觉-语言建模在文档上的应用无直接关联。未涉及OCR、布局分析、表格识别、视觉token压缩或任何多模态文档处理技术，无法迁移解决DIU中的幻觉、密集文本、复杂排版等核心痛点。
Abstract Millions of users now design personalized LLM-based chatbots that shape their daily interactions, yet they can only roughly anticipate how their design choices will manifest as behaviors in deployment. This opacity is consequential: seemingly innocuous prompts can trigger excessive sycophancy, toxicity, or other undesirable traits, degrading utility and raising safety concerns. To address this issue, we introduce an interface that enables neural transparency by exposing language model internals during chatbot design. Our approach extracts behavioral trait vectors (empathy, toxicity, sycophancy, etc.) by computing differences in neural activations between contrastive system prompts that elicit opposing behaviors. We predict chatbot behaviors by projecting the system prompt's final token activations onto these trait vectors, normalizing for cross-trait comparability, and visualizing results via an interactive sunburst diagram. To evaluate this approach, we conducted an online user study using Prolific to compare our neural transparency interface against a baseline chatbot interface without any form of transparency. Our analyses suggest that users systematically miscalibrated AI behavior: participants misjudged trait activations for eleven of fifteen analyzable traits, motivating the need for transparency tools in everyday human-AI interaction. While our interface did not change design iteration patterns, it significantly increased user trust and was enthusiastically received. Qualitative analysis revealed nuanced user experiences with the visualization, suggesting interface and interaction improvements for future work. This work offers a path for how mechanistic interpretability can be operationalized for non-technical users, establishing a foundation for safer, more aligned human-AI interactions.
Authors Sheer Karny, Anthony Baez, Pat Pataranutaporn
Comment SK and AB are co-first authors
Categories Artificial Intelligence
0
Using MLIR Transform to Design Sliced Convolution Algorithm
Title CN 使用MLIR Transform设计切片卷积算法
Keywords MLIR · 卷积优化 · 分块 · 编译器优化 · 高性能计算
Summary 本文提出了SConvTransform，一种MLIR Transform方言的扩展，用于优化2D卷积运算。通过名为SConvOp的主要操作，将Linalg卷积降级为经过分块和打包的通用操作，整个过程由卷积切片分析指导，自动确定分块大小和数据布局策略。尽管目标是功能正确性，实验显示在ARM SME和Intel AVX512架构上分别达到峰值性能的60%和67%，验证了静态形状分析与结构化分块策略结合的有效性。
Reason 该论文聚焦于底层编译器优化中的卷积算子实现，属于MLIR编译框架内的高性能计算优化工作，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。其技术内容无法迁移解决OCR幻觉、复杂排版解析、细粒度定位或长文档推理等DIU核心痛点。
Abstract This paper proposes SConvTransform, a Transform dialect extension that provides operations for optimizing 2D convolutions in MLIR. Its main operation, SConvOp, lowers Linalg convolutions into tiled and packed generic operations through a fully declarative transformation pipeline. The process is guided by a Convolution Slicing Analysis that determines tile sizes and data layout strategies based on input and filter shapes, as well as target architecture parameters. SConvOp handles edge cases by splitting irregular regions and adjusting affine maps where needed. All packing and tiling operations are derived from a parametric set of affine equations, enabling reusable and analyzable transformations. Although functional correctness was the primary goal of this work, the experimental evaluation demonstrates the effectiveness of SConvTransform, achieving good enough performance across different target architectures. Future work will focus on optimizing performance and porting to other target devices. When applied to standard convolution configurations, the generated code achieves up to 60% of peak performance on ARM SME and 67% on Intel AVX512. These results validate the benefit of combining static shape analysis with structured tiling and packing strategies within the MLIR Transform dialect. Furthermore, the modular design of SConvTransform facilitates integration with future extensions, enabling continued optimization of convolution workloads through MLIR's extensible compilation infrastructure.
Authors Victor Ferrari, Marcio Pereira, Lucas Alvarenga, Gustavo Leite, Guido Araujo
Categories Computer Vision and Pattern Recognition
0
Large-Scale Pre-training Enables Multimodal AI Differentiation of Radiation Necrosis from Brain Metastasis Progression on Routine MRI
Title CN 大规模预训练实现多模态AI在常规MRI上区分放射性坏死与脑转移进展
Keywords 自监督学习 · Vision Transformer · 脑转移 · 放射性坏死 · 医学图像分类
Summary 本文提出一种两阶段深度学习策略，通过在大规模无标签脑转移MRI数据上进行自监督预训练，然后在MOLAB数据集上微调以区分放射性坏死与肿瘤进展。结合T1CE MRI和分割掩码作为双通道输入，模型在内部和外部测试集中均表现出优于传统监督学习和放射组学的性能，并通过注意力图提供可解释性。
Reason 该论文属于医学影像分析领域，研究任务为脑部MRI中放射性坏死与肿瘤进展的鉴别诊断，完全不涉及文档图像理解（DIU）、文档视觉问答（DocVQA）、OCR、版面分析、表格识别等核心任务。尽管使用了自监督学习和多模态输入等技术，但其方法应用于生物医学图像分类，与文档结构、文本布局、视觉语言模型在文档上的应用无直接关联，无法迁移解决DIU中的OCR幻觉、复杂排版或长文档推理等痛点。
Abstract Background: Differentiating radiation necrosis (RN) from tumor progression after stereotactic radiosurgery (SRS) remains a critical challenge in brain metastases. While histopathology represents the gold standard, its invasiveness limits feasibility. Conventional supervised deep learning approaches are constrained by scarce biopsy-confirmed training data. Self-supervised learning (SSL) overcomes this by leveraging the growing availability of large-scale unlabeled brain metastases imaging datasets. Methods: In a two-phase deep learning strategy inspired by the foundation model paradigm, a Vision Transformer (ViT) was pre-trained via SSL on 10,167 unlabeled multi-source T1CE MRI sub-volumes. The pre-trained ViT was then fine-tuned for RN classification using a two-channel input (T1CE MRI and segmentation masks) on the public MOLAB dataset (n=109) using 20% of datasets as same-center held-out test set. External validation was performed on a second-center test cohort (n=28). Results: The self-supervised model achieved an AUC of 0.916 on the same-center test set and 0.764 on the second center test set, surpassing the fully supervised ViT (AUC 0.624/0.496; p=0.001/0.008) and radiomics (AUC 0.807/0.691; p=0.005/0.014). Multimodal integration further improved performance (AUC 0.947/0.821; p=0.073/0.001). Attention map visualizations enabled interpretability showing the model focused on clinically relevant lesion subregions. Conclusion: Large-scale pre-training on increasingly available unlabeled brain metastases datasets substantially improves AI model performance. A two-phase multimodal deep learning strategy achieved high accuracy in differentiating radiation necrosis from tumor progression using only routine T1CE MRI and standard clinical data, providing an interpretable, clinically accessible solution that warrants further validation.
Authors Ahmed Gomaa, Annette Schwarz, Ludwig Singer, Arnd Dörfler, Matthias Stefan May, Pluvio Stephan, Ishita Sheth, Juliane Szkitsak, Katharina Breininger, Yixing Huang, Benjamin Frey, Oliver Schnell, Daniel Delev, Roland Coras, Daniel Höfler, Philipp Schubert, Jenny Stritzelberger, Sabine Semrau, Andreas Maier, Dieter H Heiland, Udo S. Gaipl, Andrea Wittig, Rainer Fietkau, Christoph Bert, Stefanie Corradini, Florian Putz
Categories Computer Vision and Pattern Recognition
0
Generating Synthetic Human Blastocyst Images for In-Vitro Fertilization Blastocyst Grading
Title CN 为体外受精囊胚分级生成合成的人类囊胚图像
Keywords 扩散模型 · 医学图像生成 · 数据增强 · 胚胎评估 · AI辅助生殖
Summary 本文提出DIA框架，一种基于潜在扩散模型的方法，用于生成高保真度的第5天囊胚图像，支持基于Gardner形态分类和z轴焦深的条件控制。通过FID、胚胎学家图灵测试及下游分类任务验证，生成的图像具有高度真实性，并能有效提升分类模型性能，缓解数据不平衡问题。然而，该工作局限于生物医学图像生成，与文档理解无关。
Reason 该论文属于纯医学图像生成领域，研究内容为通过扩散模型生成人类囊胚图像以解决IVF中的数据稀缺问题。尽管使用了计算机视觉技术，但其任务与文档图像理解（DIU）完全无关，不涉及OCR、布局分析、表格识别或任何文档结构理解任务。无法迁移至解决DIU的核心痛点如OCR幻觉、复杂排版或长文档推理。
Abstract The success of in vitro fertilization (IVF) at many clinics relies on the accurate morphological assessment of day 5 blastocysts, a process that is often subjective and inconsistent. While artificial intelligence can help standardize this evaluation, models require large, diverse, and balanced datasets, which are often unavailable due to data scarcity, natural class imbalance, and privacy constraints. Existing generative embryo models can mitigate these issues but face several limitations, such as poor image quality, small training datasets, non-robust evaluation, and lack of clinically relevant image generation for effective data augmentation. Here, we present the Diffusion Based Imaging Model for Artificial Blastocysts (DIA) framework, a set of latent diffusion models trained to generate high-fidelity, novel day 5 blastocyst images. Our models provide granular control by conditioning on Gardner-based morphological categories and z-axis focal depth. We rigorously evaluated the models using FID, a memorization metric, an embryologist Turing test, and three downstream classification tasks. Our results show that DIA models generate realistic images that embryologists could not reliably distinguish from real images. Most importantly, we demonstrated clear clinical value. Augmenting an imbalanced dataset with synthetic images significantly improved classification accuracy (p < 0.05). Also, adding synthetic images to an already large, balanced dataset yielded statistically significant performance gains, and synthetic data could replace up to 40% of real data in some cases without a statistically significant loss in accuracy. DIA provides a robust solution for mitigating data scarcity and class imbalance in embryo datasets. By generating novel, high-fidelity, and controllable synthetic images, our models can improve the performance, fairness, and standardization of AI embryo assessment tools.
Authors Pavan Narahari, Suraj Rajendran, Lorena Bori, Jonas E. Malmsten, Qiansheng Zhan, Zev Rosenwaks, Nikica Zaninovic, Iman Hajirasouliha
Comment The manuscript is 23 pages, with five main figures and one table. The supplemental material includes 23 pages with fourteen figures and four tables
Categories Computer Vision and Pattern Recognition
0
LivePyxel: Accelerating image annotations with a Python-integrated webcam live streaming
Title CN LivePyxel：通过集成Python的摄像头实时流加速图像标注
Keywords 图像标注 · 实时流 · Python工具 · OpenCV · 实验室自动化
Summary 本文提出了一个名为LivePyxel的Python图形界面工具，支持通过摄像头、显微镜等设备进行实时图像采集与现场标注，具备贝塞尔曲线、二值掩码和非破坏性图层编辑功能，适用于实验流程中快速构建标注数据集。工具基于OpenCV和Numpy优化，旨在简化对象检测任务的数据准备过程。
Reason 该论文聚焦于图像标注工具的开发，属于特定应用场景下的数据采集与标注软件实现，本质上是为AI模型部署提供前端支持的工程工具。其核心贡献在于提升实验室环境中图像标注的效率，并未提出任何文档图像理解（DIU）中的方法论创新，如OCR-free建模、视觉token压缩、多模态对齐优化或幻觉抑制机制。尽管涉及图像处理和OpenCV集成，但完全不触及VLM在文档理解中的关键挑战（如长文档推理、复杂排版解析、细粒度定位等），属于典型的‘平行应用’类工具开发，无迁移价值。
Abstract The lack of flexible annotation tools has hindered the deployment of AI models in some scientific areas. Most existing image annotation software requires users to upload a precollected dataset, which limits support for on-demand pipelines and introduces unnecessary steps to acquire images. This constraint is particularly problematic in laboratory environments, where on-site data acquisition from instruments such as microscopes is increasingly common. In this work, we introduce \texttt{LivePixel}, a Python-based graphical user interface that integrates with imaging systems, such as webcams, microscopes, and others, to enable on-site image annotation. LivePyxel is designed to be easy to use through a simple interface that allows users to precisely delimit areas for annotation using tools commonly found in commercial graphics editing software. Of particular interest is the availability of Bézier splines and binary masks, and the software's capacity to work with non-destructive layers that enable high-performance editing. LivePyxel also integrates a wide compatibility across video devices, and it's optimized for object detection operations via the use of OpenCV in combination with high-performance libraries designed to handle matrix and linear algebra operations via Numpy effectively. LivePyxel facilitates seamless data collection and labeling, accelerating the development of AI models in experimental workflows. LivePyxel is freely available at https://github.com/UGarCil/LivePyxel
Authors Uriel Garcilazo-Cruz, Joseph O. Okeme, Rodrigo A. Vargas-Hernández
Comment 9 pages, 10 figures, SM, 5 pages, 5 figures, 1 Table
Categories Computer Vision and Pattern Recognition
0
Vision-Only Gaussian Splatting for Collaborative Semantic Occupancy Prediction AAAI 2026 (Oral)
Title CN 仅视觉高斯点阵用于协同语义占据预测
Keywords 3D语义占据 · 高斯点阵 · 协同感知 · 自动驾驶 · 视觉通信
Summary 本文提出一种基于稀疏3D语义高斯点阵的协同感知方法，用于连接车辆间的信息共享，通过传输几何与语义联合编码的高斯原语，在降低通信开销的同时提升3D语义占据预测性能。实验表明该方法在多种通信受限场景下均优于基线方法。
Reason 该论文属于3D视觉与自动驾驶领域，研究的是协同感知中的语义占据预测问题，使用高斯点阵表示进行多车协作。尽管涉及视觉表示和特征融合，但其方法针对3D场景重建与感知，与文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等无直接关联。无法迁移解决OCR幻觉、密集文本识别、复杂排版或长文档推理等核心DIU痛点。
Abstract Collaborative perception enables connected vehicles to share information, overcoming occlusions and extending the limited sensing range inherent in single-agent (non-collaborative) systems. Existing vision-only methods for 3D semantic occupancy prediction commonly rely on dense 3D voxels, which incur high communication costs, or 2D planar features, which require accurate depth estimation or additional supervision, limiting their applicability to collaborative scenarios. To address these challenges, we propose the first approach leveraging sparse 3D semantic Gaussian splatting for collaborative 3D semantic occupancy prediction. By sharing and fusing intermediate Gaussian primitives, our method provides three benefits: a neighborhood-based cross-agent fusion that removes duplicates and suppresses noisy or inconsistent Gaussians; a joint encoding of geometry and semantics in each primitive, which reduces reliance on depth supervision and allows simple rigid alignment; and sparse, object-centric messages that preserve structural information while reducing communication volume. Extensive experiments demonstrate that our approach outperforms single-agent perception and baseline collaborative methods by +8.42 and +3.28 points in mIoU, and +5.11 and +22.41 points in IoU, respectively. When further reducing the number of transmitted Gaussians, our method still achieves a +1.9 improvement in mIoU, using only 34.6% communication volume, highlighting robust performance under limited communication budgets.
Authors Cheng Chen, Hao Huang, Saurabh Bagchi
Comment Accepted by AAAI 2026 (Oral)
Categories Computer Vision and Pattern Recognition
0
Linear Algebraic Approaches to Neuroimaging Data Compression: A Comparative Analysis of Matrix and Tensor Decomposition Methods for High-Dimensional Medical Images
Title CN 神经影像数据压缩的线性代数方法：高维医学图像的矩阵与张量分解方法比较分析
Keywords 张量分解 · SVD · 医学图像压缩 · 神经影像 · 数据压缩
Summary 本文对比了Tucker分解和奇异值分解（SVD）在神经影像数据压缩中的表现。结果显示，Tucker分解在保留多维结构关系方面优于SVD，适合需要保持时空结构的应用；而SVD在高压缩比下性能更优但牺牲了重建保真度。
Reason 该论文研究的是神经影像（如fMRI、MRI）这类医学图像的压缩问题，属于3D医学图像处理领域，与文档图像理解（DIU）任务无直接关联。其关注点为多维医学信号的结构保持压缩，而非文档的文本布局、语义理解或视觉-语言对齐。尽管涉及‘压缩’概念，但压缩目标和数据模态与文档图像差异巨大，无法迁移解决OCR幻觉、复杂排版或DocVQA等核心问题。
Abstract This paper evaluates Tucker decomposition and Singular Value Decomposition (SVD) for compressing neuroimaging data. Tucker decomposition preserves multi-dimensional relationships, achieving superior reconstruction fidelity and perceptual similarity. SVD excels in extreme compression but sacrifices fidelity. The results highlight Tucker decomposition's suitability for applications requiring the preservation of structural and temporal relationships.
Authors Jaeho Kim, Daniel David, Ana Vizitiv
Categories Computer Vision and Pattern Recognition
0
The Shape of Sight: A Homological Framework for Unifying Visual Perception
Title CN 视觉的形态：一种统一视觉感知的同调框架
Keywords 视觉感知 · 同调理论 · 拓扑表示 · 神经编码 · 理论模型
Summary 本文从拓扑学角度提出一个关于视觉感知的同调框架，认为大脑的表征受‘同调奇偶性’支配：偶维同调结构作为静态整合支架（对应‘是什么’），奇维同调结构作为动态循环流（对应‘在哪里’）。这一模型试图统一解释视觉感知中的稳定性、动态性和路径依赖问题，并类比于腹侧-背侧通路分离。然而，该工作停留在理论假设层面，缺乏与现有VLM或文档理解技术的接口。
Reason 该论文提出了一种基于拓扑同调理论的视觉感知统一框架，属于理论神经科学与基础视觉理论的交叉研究。尽管其思想新颖，但未涉及文档图像理解（DIU）中的具体任务如DocVQA、布局分析、OCR等，也未提供可迁移至VLM或文档处理的技术工具（如高分辨率建模、细粒度定位、去幻觉机制）。内容高度抽象且无实证方法支撑，无法直接应对文档理解中的实际痛点，如复杂排版、密集文本或长文档推理。
Abstract Visual perception, the brain's construction of a stable world from sensory data, faces several long-standing, fundamental challenges. While often studied separately, these problems have resisted a single, unifying computational framework. In this perspective, we propose a homological framework for visual perception. We argue that the brain's latent representations are governed by their topological parity. This parity interpretation functionally separates homological structures into two distinct classes: 1) Even-dimensional homology ($H_{even}$) acts as static, integrative scaffolds. These structures bind context and content into ``wholes'' or ``what'', serving as the stable, resonant cavities for perceptual objects; 2) Odd-dimensional homology ($H_{odd}$) acts as dynamic, recurrent flows. These structures represent paths, transformations, and self-sustaining ``traces'' or ``where'' that navigate the perceptual landscape. This scaffold-and-flow model is supported by the ventral-dorsal pathway separation and provides a unified solution to three core problems in visual perception. Homological parity hypothesis recasts visual perception not as a linear computation, but as a dynamic interaction between stable, integrative structures and the recurrent, self-sustaining flows that run on them. This perspective offers a new mathematical foundation for linking neural dynamics to perception and cognition.
Authors Xin Li
Categories Computer Vision and Pattern Recognition
0
Early Lung Cancer Diagnosis from Virtual Follow-up LDCT Generation via Correlational Autoencoder and Latent Flow Matching
Title CN 通过相关性自编码器和潜在流匹配从虚拟随访LDCT生成实现早期肺癌诊断
Keywords 肺癌早期诊断 · CT图像生成 · 扩散模型 · 隐空间流匹配 · 医学影像
Summary 本文提出CorrFlowNet，一种基于相关性自编码器和潜在空间流匹配的生成模型，用于从基线CT扫描生成虚拟的一年随访CT图像，从而在无需实际等待临床随访的情况下评估肺结节恶性风险。方法利用扩散模型思想，在隐空间中建模结节进展动态，并通过辅助分类器提升诊断性能。实验表明其诊断准确率接近真实随访CT效果。
Reason 该论文属于医学图像分析领域，研究内容为基于CT扫描生成虚拟随访图像以辅助肺癌早期诊断，属于典型的医疗影像生成与诊断应用，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。未涉及OCR、布局分析、表格识别、多模态文档建模等DIU核心任务，也未提出可用于解决文档理解中幻觉、复杂排版或长文档推理等问题的通用方法。
Abstract Lung cancer is one of the most commonly diagnosed cancers, and early diagnosis is critical because the survival rate declines sharply once the disease progresses to advanced stages. However, achieving an early diagnosis remains challenging, particularly in distinguishing subtle early signals of malignancy from those of benign conditions. In clinical practice, a patient with a high risk may need to undergo an initial baseline and several annual follow-up examinations (e.g., CT scans) before receiving a definitive diagnosis, which can result in missing the optimal treatment. Recently, Artificial Intelligence (AI) methods have been increasingly used for early diagnosis of lung cancer, but most existing algorithms focus on radiomic features extraction from single early-stage CT scans. Inspired by recent advances in diffusion models for image generation, this paper proposes a generative method, named CorrFlowNet, which creates a virtual, one-year follow-up CT scan after the initial baseline scan. This virtual follow-up would allow for an early detection of malignant/benign nodules, reducing the need to wait for clinical follow-ups. During training, our approach employs a correlational autoencoder to encode both early baseline and follow-up CT images into a latent space that captures the dynamics of nodule progression as well as the correlations between them, followed by a flow matching algorithm on the latent space with a neural ordinary differential equation. An auxiliary classifier is used to further enhance the diagnostic accuracy. Evaluations on a real clinical dataset show our method can significantly improve downstream lung nodule risk assessment compared with existing baseline models. Moreover, its diagnostic accuracy is comparable with real clinical CT follow-ups, highlighting its potential to improve cancer diagnosis.
Authors Yutong Wu, Yifan Wang, Qining Zhang, Chuan Zhou, Lei Ying
Comment 10 pages, 3 figures
Categories Computer Vision and Pattern Recognition
0
Unified Spherical Frontend: Learning Rotation-Equivariant Representations of Spherical Images from Any Camera
Title CN 统一球面前端：从任意相机学习球面图像的旋转等变表示
Keywords 球面图像 · 旋转等变性 · 广角相机 · 空间域卷积 · 视觉前端
Summary 本文提出了一种名为Unified Spherical Frontend (USF) 的框架，可将任意宽视场相机（如鱼眼或全景相机）拍摄的图像转换为单位球面上的表示，并在空间域直接进行球面重采样、卷积和池化操作。该方法避免了传统球面CNN中昂贵的球谐变换，实现了高效的旋转等变性，在分类、检测和分割任务中对极端畸变和任意旋转表现出强鲁棒性。然而，其应用场景局限于3D感知与广角视觉，与文档图像理解无直接关联。
Reason 该论文研究的是球面图像（如鱼眼、全景相机）在计算机视觉任务中的处理方法，属于3D视觉与广角成像领域。尽管其提出的球面卷积和旋转等变性设计具有理论价值，但核心问题与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、密集文本识别、复杂版式分析或长文档推理等DIU核心痛点。
Abstract Modern perception increasingly relies on fisheye, panoramic, and other wide field-of-view (FoV) cameras, yet most pipelines still apply planar CNNs designed for pinhole imagery on 2D grids, where image-space neighborhoods misrepresent physical adjacency and models are sensitive to global rotations. Frequency-domain spherical CNNs partially address this mismatch but require costly spherical harmonic transforms that constrain resolution and efficiency. We introduce the Unified Spherical Frontend (USF), a lens-agnostic framework that transforms images from any calibrated camera into a unit-sphere representation via ray-direction correspondences, and performs spherical resampling, convolution, and pooling directly in the spatial domain. USF is modular: projection, location sampling, interpolation, and resolution control are fully decoupled. Its distance-only spherical kernels offer configurable rotation-equivariance (mirroring translation-equivariance in planar CNNs) while avoiding harmonic transforms entirely. We compare standard planar backbones with their spherical counterparts across classification, detection, and segmentation tasks on synthetic (Spherical MNIST) and real-world datasets (PANDORA, Stanford 2D-3D-S), and stress-test robustness to extreme lens distortions, varying FoV, and arbitrary rotations. USF processes high-resolution spherical imagery efficiently and maintains less than 1% performance drop under random test-time rotations, even without rotational augmentation, and even enables zero-shot generalization from one lens type to unseen wide-FoV lenses with minimal performance degradation.
Authors Mukai Yu, Mosam Dabhi, Liuyue Xie, Sebastian Scherer, László A. Jeni
Categories Computer Vision and Pattern Recognition
0
EgoControl: Controllable Egocentric Video Generation via 3D Full-Body Poses
Title CN EgoControl：通过3D全身姿态实现可控的自我中心视频生成
Keywords egocentric video · video generation · diffusion model · 3D pose · embodied AI
Summary 本文提出EgoControl，一种基于自我中心视角视频数据的姿态可控视频扩散模型。通过引入一种新颖的姿态表示方法，结合全局相机运动和关节式身体动作，并在扩散过程中集成专用控制机制，实现对未来帧的精确动作控制。实验表明该方法能生成时间连贯且视觉逼真的自我中心视频。
Reason 该论文研究的是基于3D全身姿态控制的自我中心视频生成，属于视频生成与具身智能领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其核心技术聚焦于视频扩散模型中的姿态控制机制，无法迁移解决OCR幻觉、复杂排版理解、长文档推理等DIU核心问题。
Abstract Egocentric video generation with fine-grained control through body motion is a key requirement towards embodied AI agents that can simulate, predict, and plan actions. In this work, we propose EgoControl, a pose-controllable video diffusion model trained on egocentric data. We train a video prediction model to condition future frame generation on explicit 3D body pose sequences. To achieve precise motion control, we introduce a novel pose representation that captures both global camera dynamics and articulated body movements, and integrate it through a dedicated control mechanism within the diffusion process. Given a short sequence of observed frames and a sequence of target poses, EgoControl generates temporally coherent and visually realistic future frames that align with the provided pose control. Experimental results demonstrate that EgoControl produces high-quality, pose-consistent egocentric videos, paving the way toward controllable embodied video simulation and understanding.
Authors Enrico Pallotta, Sina Mokhtarzadeh Azar, Lars Doorenbos, Serdar Ozsoy, Umar Iqbal, Juergen Gall
Categories Computer Vision and Pattern Recognition
0
Observer Actor: Active Vision Imitation Learning with Sparse View Gaussian Splatting
Title CN 观察者-执行者：基于稀疏视图高斯点阵的主动视觉模仿学习
Keywords 主动视觉 · 模仿学习 · 高斯点阵 · 机器人操作 · 3D重建
Summary 本文提出Observer Actor（ObAct）框架，用于主动视觉模仿学习。系统通过一个‘观察者’手臂动态构建3D高斯点阵并寻找最佳观测视角，再由‘执行者’手臂基于清晰视角执行操作。实验表明该方法在有无遮挡情况下均显著提升了模仿学习性能。但研究属于机器人具身感知范畴，不涉及文档理解任务。
Reason 该论文研究的是具身智能中的主动视觉与机器人双臂操作，核心是通过移动摄像头获取最优视角以提升模仿学习性能。其技术路线聚焦于3D高斯点阵重建与机器人控制，属于具身智能/机器人领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心问题。
Abstract We propose Observer Actor (ObAct), a novel framework for active vision imitation learning in which the observer moves to optimal visual observations for the actor. We study ObAct on a dual-arm robotic system equipped with wrist-mounted cameras. At test time, ObAct dynamically assigns observer and actor roles: the observer arm constructs a 3D Gaussian Splatting (3DGS) representation from three images, virtually explores this to find an optimal camera pose, then moves to this pose; the actor arm then executes a policy using the observer's observations. This formulation enhances the clarity and visibility of both the object and the gripper in the policy's observations. As a result, we enable the training of ambidextrous policies on observations that remain closer to the occlusion-free training distribution, leading to more robust policies. We study this formulation with two existing imitation learning methods -- trajectory transfer and behavior cloning -- and experiments show that ObAct significantly outperforms static-camera setups: trajectory transfer improves by 145% without occlusion and 233% with occlusion, while behavior cloning improves by 75% and 143%, respectively. Videos are available at https://obact.github.io.
Authors Yilong Wang, Cheng Qian, Ruomeng Fan, Edward Johns
Comment Videos are available on our project webpage at https://obact.github.io
Categories Computer Vision and Pattern Recognition
0
Video4Edit: Viewing Image Editing as a Degenerate Temporal Process
Title CN Video4Edit：将图像编辑视为退化的时序过程
Keywords 图像编辑 · 视频建模 · 多模态模型 · 扩散模型 · 数据高效微调
Summary 本文提出将图像编辑视为一种退化的视频时序过程，利用从视频预训练中获得的单帧演化先验，实现数据高效的图像编辑微调。该方法在仅使用1%监督信号的情况下达到主流开源基线的性能。然而，其应用场景和技术创新均集中于通用图像编辑领域，不涉及文档结构、文本语义或视觉-语言推理等文档理解关键问题。
Reason 该论文研究的是图像编辑任务，其核心思想是将图像编辑建模为视频生成中的退化时序过程，并利用视频预训练模型来提升编辑效率。尽管涉及多模态基础模型和视觉生成，但其目标领域为通用图像编辑，与文档图像理解（DIU）中的DocVQA、布局分析、表格识别等任务无直接关联。方法上未针对OCR幻觉、密集文本处理、复杂排版或长文档推理等DIU核心痛点设计，且属于图像生成范畴，无法迁移解决文档理解中的语义结构解析问题。
Abstract We observe that recent advances in multimodal foundation models have propelled instruction-driven image generation and editing into a genuinely cross-modal, cooperative regime. Nevertheless, state-of-the-art editing pipelines remain costly: beyond training large diffusion/flow models, they require curating massive high-quality triplets of \{instruction, source image, edited image\} to cover diverse user intents. Moreover, the fidelity of visual replacements hinges on how precisely the instruction references the target semantics. We revisit this challenge through the lens of temporal modeling: if video can be regarded as a full temporal process, then image editing can be seen as a degenerate temporal process. This perspective allows us to transfer single-frame evolution priors from video pre-training, enabling a highly data-efficient fine-tuning regime. Empirically, our approach matches the performance of leading open-source baselines while using only about one percent of the supervision demanded by mainstream editing models.
Authors Xiaofan Li, Yanpeng Sun, Chenming Wu, Fan Duan, YuAn Wang, Weihao Bo, Yumeng Zhang, Dingkang Liang
Comment 10 pages, 5 figures
Categories Computer Vision and Pattern Recognition
0
SFHand: A Streaming Framework for Language-guided 3D Hand Forecasting and Embodied Manipulation
Title CN SFHand：一种用于语言引导的3D手部预测和具身操作的流式框架
Keywords 3D hand forecasting · streaming framework · language guidance · embodied manipulation · autoregressive model
Summary 本文提出SFHand，首个支持语言引导的流式3D手部运动预测框架，能够从连续视频流和语言指令中自回归地预测未来的手部类型、2D框、3D姿态和轨迹。配合新构建的大规模数据集EgoHaFL，该方法在3D手部预测上显著超越先前方法，并在具身操作任务中验证了表征的有效性。
Reason 该论文研究的是3D手部姿态预测与具身智能中的操作任务，属于具身智能/机器人与3D视觉领域，与文档图像理解（DIU）或DocVQA无关。尽管其提出了流式架构和语言引导机制，但这些方法针对的是视频流中的动态手部建模，并不适用于解决文档理解中的OCR幻觉、复杂排版或长文档推理等核心问题。
Abstract Real-time 3D hand forecasting is a critical component for fluid human-computer interaction in applications like AR and assistive robotics. However, existing methods are ill-suited for these scenarios, as they typically require offline access to accumulated video sequences and cannot incorporate language guidance that conveys task intent. To overcome these limitations, we introduce SFHand, the first streaming framework for language-guided 3D hand forecasting. SFHand autoregressively predicts a comprehensive set of future 3D hand states, including hand type, 2D bounding box, 3D pose, and trajectory, from a continuous stream of video and language instructions. Our framework combines a streaming autoregressive architecture with an ROI-enhanced memory layer, capturing temporal context while focusing on salient hand-centric regions. To enable this research, we also introduce EgoHaFL, the first large-scale dataset featuring synchronized 3D hand poses and language instructions. We demonstrate that SFHand achieves new state-of-the-art results in 3D hand forecasting, outperforming prior work by a significant margin of up to 35.8%. Furthermore, we show the practical utility of our learned representations by transferring them to downstream embodied manipulation tasks, improving task success rates by up to 13.4% on multiple benchmarks. Dataset page: https://huggingface.co/datasets/ut-vision/EgoHaFL, project page: https://github.com/ut-vision/SFHand.
Authors Ruicong Liu, Yifei Huang, Liangyang Ouyang, Caixin Kang, Yoichi Sato
Categories Computer Vision and Pattern Recognition
0
MVS-TTA: Test-Time Adaptation for Multi-View Stereo via Meta-Auxiliary Learning
Title CN MVS-TTA：通过元辅助学习实现多视图立体匹配的测试时自适应
Keywords 多视图立体匹配 · 测试时自适应 · 元学习 · 自监督学习 · 三维重建
Summary 本文提出MVS-TTA，一种用于多视图立体匹配的测试时自适应框架，通过引入自监督的跨视角一致性损失作为辅助任务，并结合元辅助学习策略来提升模型在测试阶段对新场景的泛化能力。该方法可应用于多种MVS模型，在DTU和BlendedMVS等数据集上验证了有效性。这是首次将基于优化的测试时自适应与元学习结合用于学习型MVS的研究。
Reason 该论文研究的是多视图立体匹配（MVS）中的测试时自适应问题，属于3D视觉与重建领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。尽管其提出的测试时自适应框架具有一定的方法论价值，但其核心场景针对三维几何重建，迁移到文档图像理解中的OCR幻觉、复杂排版解析或长文档推理等问题极为困难，且未涉及多模态语言模型（VLM）在文档上的应用或视觉token压缩等DIU关键方向。
Abstract Recent learning-based multi-view stereo (MVS) methods are data-driven and have achieved remarkable progress due to large-scale training data and advanced architectures. However, their generalization remains sub-optimal due to fixed model parameters trained on limited training data distributions. In contrast, optimization-based methods enable scene-specific adaptation but lack scalability and require costly per-scene optimization. In this paper, we propose MVS-TTA, an efficient test-time adaptation (TTA) framework that enhances the adaptability of learning-based MVS methods by bridging these two paradigms. Specifically, MVS-TTA employs a self-supervised, cross-view consistency loss as an auxiliary task to guide inference-time adaptation. We introduce a meta-auxiliary learning strategy to train the model to benefit from auxiliary-task-based updates explicitly. Our framework is model-agnostic and can be applied to a wide range of MVS methods with minimal architectural changes. Extensive experiments on standard datasets (DTU, BlendedMVS) and a challenging cross-dataset generalization setting demonstrate that MVS-TTA consistently improves performance, even when applied to state-of-the-art MVS models. To our knowledge, this is the first attempt to integrate optimization-based test-time adaptation into learning-based MVS using meta-learning. The code will be available at https://github.com/mart87987-svg/MVS-TTA.
Authors Hannuo Zhang, Zhixiang Chi, Yang Wang, Xinxin Zuo
Comment 8 pages, 7 figures
Categories Computer Vision and Pattern Recognition
0
Muskie: Multi-view Masked Image Modeling for 3D Vision Pre-training
Title CN Muskie：用于3D视觉预训练的多视图掩码图像建模
Keywords 3D视觉 · 多视图学习 · 掩码图像建模 · 自监督预训练 · 几何理解
Summary 本文提出Muskie，一种专为3D视觉任务设计的多视图视觉骨干网络。它通过在预训练阶段利用多个视角之间的几何对应关系来重建被严重遮蔽的图像内容，从而在无需3D监督的情况下学习到具有视图不变性的特征并增强几何理解能力。实验表明，Muskie在多视图对应关系准确性和下游3D任务（如相机位姿估计和点云重建）上均优于现有的逐帧处理模型。
Reason 该论文研究的是3D视觉任务中的多视图表示学习，核心关注点为相机姿态估计和点云重建等三维几何问题，属于具身智能与3D视觉领域。其方法设计针对多视角图像间的几何一致性，与文档图像理解（DIU）中的OCR幻觉、复杂排版、细粒度文本识别或长文档推理等核心痛点无直接关联。尽管掩码重建思想在视觉预训练中通用，但该工作无法迁移解决文档图像中的语义结构理解或多模态对齐问题。
Abstract We present Muskie, a native multi-view vision backbone designed for 3D vision tasks. Unlike existing models, which are frame-wise and exhibit limited multi-view consistency, Muskie is designed to process multiple views simultaneously and introduce multi-view consistency in pre-training stage. Muskie is trained to reconstruct heavily masked content in one view by finding and utilizing geometric correspondences from other views. Through this pretext task and our proposed aggressive masking strategy, the model implicitly to learn view-invariant features and develop strong geometric understanding without any 3D supervision. Compared with state-of-the-art frame-wise backbones such as DINO, Muskie achieves higher multi-view correspondence accuracy. Furthermore, we demonstrate that using Muskie as a backbone consistently enhances performance on downstream 3D tasks, including camera pose estimation and pointmap reconstruction. Codes are publicly available at https://leo-frank.github.io/Muskie/
Authors Wenyu Li, Sidun Liu, Peng Qiao, Yong Dou, Tongrui Hu
Categories Computer Vision and Pattern Recognition
0
Consolidating Diffusion-Generated Video Detection with Unified Multimodal Forgery Learning
Title CN 通过统一多模态伪造学习整合扩散生成视频检测
Keywords 视频伪造检测 · 扩散模型 · 多模态学习 · 视觉Transformer · MLLM
Summary 本文提出MM-Det++，一种用于检测扩散模型生成视频的多模态检测算法，包含时空分支和多模态分支，并设计了统一多模态学习模块来融合表示。同时构建了大规模DVF数据集。实验表明该方法在视频伪造检测上表现优异。
Reason 该论文聚焦于视频伪造检测，属于视频理解与多媒体安全领域，与文档图像理解（DIU）或DocVQA无直接关联。尽管使用了多模态大模型（MLLMs），但其目标是检测扩散生成的虚假视频，而非解决文档中的OCR幻觉、复杂排版或长文本推理等问题。方法难以迁移到文档理解场景，属于无关领域。
Abstract The proliferation of videos generated by diffusion models has raised increasing concerns about information security, highlighting the urgent need for reliable detection of synthetic media. Existing methods primarily focus on image-level forgery detection, leaving generic video-level forgery detection largely underexplored. To advance video forensics, we propose a consolidated multimodal detection algorithm, named MM-Det++, specifically designed for detecting diffusion-generated videos. Our approach consists of two innovative branches and a Unified Multimodal Learning (UML) module. Specifically, the Spatio-Temporal (ST) branch employs a novel Frame-Centric Vision Transformer (FC-ViT) to aggregate spatio-temporal information for detecting diffusion-generated videos, where the FC-tokens enable the capture of holistic forgery traces from each video frame. In parallel, the Multimodal (MM) branch adopts a learnable reasoning paradigm to acquire Multimodal Forgery Representation (MFR) by harnessing the powerful comprehension and reasoning capabilities of Multimodal Large Language Models (MLLMs), which discerns the forgery traces from a flexible semantic perspective. To integrate multimodal representations into a coherent space, a UML module is introduced to consolidate the generalization ability of MM-Det++. In addition, we also establish a large-scale and comprehensive Diffusion Video Forensics (DVF) dataset to advance research in video forgery detection. Extensive experiments demonstrate the superiority of MM-Det++ and highlight the effectiveness of unified multimodal forgery learning in detecting diffusion-generated videos.
Authors Xiaohong Liu, Xiufeng Song, Huayu Zheng, Lei Bai, Xiaoming Liu, Guangtao Zhai
Comment Code and dataset are available at https://github.com/SparkleXFantasy/MM-Det-Plus
Categories Computer Vision and Pattern Recognition
0
Spotlight: Identifying and Localizing Video Generation Errors Using VLMs
Title CN Spotlight：使用视觉语言模型识别和定位视频生成错误
Keywords 视频生成 · 错误定位 · 视觉语言模型 · 细粒度评估 · 推理策略
Summary 本文提出了Spotlight任务，旨在通过人工标注1600多个细粒度错误，来定位和解释当前文本到视频生成模型中的局部错误。作者使用三种先进视频生成模型生成600个视频，并评估现有VLM在错误识别与定位上的表现，发现其显著落后于人类。通过推理阶段的策略改进，VLM性能提升了近两倍。
Reason 该论文研究的是视频生成中的错误定位问题，属于视频理解与生成领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其使用了VLM进行细粒度错误定位，但应用场景为T2V生成评估，无法迁移解决OCR幻觉、密集文本处理或复杂排版理解等文档核心痛点。
Abstract Current text-to-video models (T2V) can generate high-quality, temporally coherent, and visually realistic videos. Nonetheless, errors still often occur, and are more nuanced and local compared to the previous generation of T2V models. While current evaluation paradigms assess video models across diverse dimensions, they typically evaluate videos holistically without identifying when specific errors occur or describing their nature. We address this gap by introducing Spotlight, a novel task aimed at localizing and explaining video-generation errors. We generate 600 videos using 200 diverse textual prompts and three state-of-the-art video generators (Veo 3, Seedance, and LTX-2), and annotate over 1600 fine-grained errors across six types, including motion, physics, and prompt adherence. We observe that adherence and physics errors are predominant and persist across longer segments, whereas appearance-disappearance and body pose errors manifest in shorter segments. We then evaluate current VLMs on Spotlight and find that VLMs lag significantly behind humans in error identification and localization in videos. We propose inference-time strategies to probe the limits of current VLMs on our task, improving performance by nearly 2x. Our task paves a way forward to building fine-grained evaluation tools and more sophisticated reward models for video generators.
Authors Aditya Chinchure, Sahithya Ravi, Pushkar Shukla, Vered Shwartz, Leonid Sigal
Categories Computer Vision and Pattern Recognition
0
RefVTON: person-to-person Try on with Additional Unpaired Visual Reference
Title CN RefVTON：通过额外未配对视觉参考实现人到人的试穿
Keywords 虚拟试穿 · 图像生成 · 扩散模型 · 纹理对齐 · 未配对参考
Summary 本文提出RefVTON，一种基于扩散模型的人到人虚拟试穿框架，利用未配对的参考图像来提升服装纹理对齐和细节保持，无需复杂的辅助输入如身体解析图或变形掩码。通过构建包含未配对参考图像的数据集进行训练，模型在公开基准上表现出竞争力。然而，该工作属于图像生成范畴，与文档理解无关。
Reason 该论文属于虚拟试衣（Virtual Try-On）领域，聚焦于时尚图像生成任务，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其技术路线基于扩散模型（flux-based）和未配对参考图像用于纹理对齐，属于纯图像生成方向，无法迁移解决OCR幻觉、复杂排版理解、细粒度文本定位等DIU核心问题。
Abstract We introduce RefTON, a flux-based person-to-person virtual try-on framework that enhances garment realism through unpaired visual references. Unlike conventional approaches that rely on complex auxiliary inputs such as body parsing and warped mask or require finely designed extract branches to process various input conditions, RefTON streamlines the process by directly generating try-on results from a source image and a target garment, without the need for structural guidance or auxiliary components to handle diverse inputs. Moreover, inspired by human clothing selection behavior, RefTON leverages additional reference images (the target garment worn on different individuals) to provide powerful guidance for refining texture alignment and maintaining the garment details. To enable this capability, we built a dataset containing unpaired reference images for training. Extensive experiments on public benchmarks demonstrate that RefTON achieves competitive or superior performance compared to state-of-the-art methods, while maintaining a simple and efficient person-to-person design.
Authors Liuzhuozheng Li, Yue Gong, Shanyuan Liu, Bo Cheng, Yuhang Ma, Liebucha Wu, Dengyang Jiang, Zanyi Wang, Dawei Leng, Yuhui Yin
Categories Computer Vision and Pattern Recognition
0
MagicMirror: ID-Preserved Video Generation in Video Diffusion Transformers ICCV 2025
Title CN MagicMirror：基于视频扩散变换器的身份保留视频生成
Keywords 视频生成 · 身份保留 · 扩散模型 · Transformer · 跨模态适配
Summary 本文提出MagicMirror框架，用于生成身份一致、动作自然的高质量视频。基于视频扩散Transformer，引入双分支面部特征提取器、轻量级跨模态适配器和两阶段训练策略，在无需个体微调的情况下实现身份与运动的平衡。实验表明其在多个指标上优于现有方法。
Reason 该论文属于纯视频生成领域，研究目标是生成具有高电影质量且保持身份一致性的视频，其核心技术（如双分支人脸特征提取、跨模态适配器）针对的是人物身份与动态动作的协调问题，与文档图像理解（DIU）中的OCR幻觉、密集文本识别、复杂版式分析或长文档推理等核心痛点完全无关。尽管使用了Video Diffusion Transformers，但方法设计不涉及多模态对齐、视觉压缩或细粒度定位等可用于DIU的技术路径。
Abstract We present MagicMirror, a framework for generating identity-preserved videos with cinematic-level quality and dynamic motion. While recent advances in video diffusion models have shown impressive capabilities in text-to-video generation, maintaining consistent identity while producing natural motion remains challenging. Previous methods either require person-specific fine-tuning or struggle to balance identity preservation with motion diversity. Built upon Video Diffusion Transformers, our method introduces three key components: (1) a dual-branch facial feature extractor that captures both identity and structural features, (2) a lightweight cross-modal adapter with Conditioned Adaptive Normalization for efficient identity integration, and (3) a two-stage training strategy combining synthetic identity pairs with video data. Extensive experiments demonstrate that MagicMirror effectively balances identity consistency with natural motion, outperforming existing methods across multiple metrics while requiring minimal parameters added. The code and model will be made publicly available.
Authors Yuechen Zhang, Yaoyang Liu, Bin Xia, Bohao Peng, Zexin Yan, Eric Lo, Jiaya Jia
Comment ICCV 2025, It is best viewed in Acrobat. Project Page: https://julianjuaner.github.io/projects/MagicMirror/
Categories Computer Vision and Pattern Recognition
0
Together, Then Apart: Revisiting Multimodal Survival Analysis via a Min-Max Perspective
Title CN 共同，然后分离：通过最小-最大视角重新审视多模态生存分析
Keywords 多模态学习 · 生存分析 · 表示学习 · 最优传输 · 对比正则化
Summary 本文提出了一种名为Together-Then-Apart（TTA）的最小-最大优化框架，用于多模态生存分析，旨在平衡跨模态对齐与模态独特性保留。通过‘Together’阶段对齐语义表征，‘Apart’阶段增强模态特异性表示，从而避免特征坍塌。实验在TCGA癌症数据集上验证了方法的有效性。
Reason 该论文研究的是医学领域的多模态生存分析，聚焦于整合组织病理学和基因组学数据，属于生物医学信息学范畴。其核心方法TTA框架虽在多模态表示学习上有理论创新，但应用场景与文档图像理解（DIU）无直接关联，且无法迁移解决OCR幻觉、复杂排版、细粒度定位等DIU核心痛点。
Abstract Integrating heterogeneous modalities such as histopathology and genomics is central to advancing survival analysis, yet most existing methods prioritize cross-modal alignment through attention-based fusion mechanisms, often at the expense of modality-specific characteristics. This overemphasis on alignment leads to representation collapse and reduced diversity. In this work, we revisit multi-modal survival analysis via the dual lens of alignment and distinctiveness, positing that preserving modality-specific structure is as vital as achieving semantic coherence. In this paper, we introduce Together-Then-Apart (TTA), a unified min-max optimization framework that simultaneously models shared and modality-specific representations. The Together stage minimizes semantic discrepancies by aligning embeddings via shared prototypes, guided by an unbalanced optimal transport objective that adaptively highlights informative tokens. The Apart stage maximizes representational diversity through modality anchors and a contrastive regularizer that preserve unique modality information and prevent feature collapse. Extensive experiments on five TCGA benchmarks show that TTA consistently outperforms state-of-the-art methods. Beyond empirical gains, our formulation provides a new theoretical perspective of how alignment and distinctiveness can be jointly achieved in for robust, interpretable, and biologically meaningful multi-modal survival analysis.
Authors Wenjing Liu, Qin Ren, Wen Zhang, Yuewei Lin, Chenyu You
Categories Computer Vision and Pattern Recognition
0
Less Is More: An Explainable AI Framework for Lightweight Malaria Classification
Title CN 少即是多：一种可解释的轻量级疟疾分类AI框架
Keywords 疟疾分类 · 可解释AI · 特征工程 · 轻量级模型 · 医学图像
Summary 本文提出了一种名为EMFE的轻量级可解释AI流程，通过提取疟疾细胞图像的形态学特征（如非背景像素数和细胞内孔洞数），结合逻辑回归与随机森林等简单模型，在低计算资源下实现了高精度的二分类。虽然在效率和可解释性上表现优异，但其方法局限于特定医学图像任务，未涉及文本、排版或多模态理解。
Reason 该论文属于医学图像分类中的特定疾病诊断任务，研究内容为基于形态学特征工程和传统机器学习模型进行疟疾细胞分类，完全不涉及文档图像理解（DIU）或视觉-语言多模态任务。其方法无法迁移到DocVQA、布局分析、表格识别等核心DIU任务中，属于典型的‘平行下游应用’，且领域无关（生物医学图像分析），无任何底层技术创新对DIU有借鉴意义。
Abstract Background and Objective: Deep learning models have high computational needs and lack interpretability but are often the first choice for medical image classification tasks. This study addresses whether complex neural networks are essential for the simple binary classification task of malaria. We introduce the Extracted Morphological Feature Engineered (EMFE) pipeline, a transparent, reproducible, and low compute machine learning approach tailored explicitly for simple cell morphology, designed to achieve deep learning performance levels on a simple CPU only setup with the practical aim of real world deployment. Methods: The study used the NIH Malaria Cell Images dataset, with two features extracted from each cell image: the number of non background pixels and the number of holes within the cell. Logistic Regression and Random Forest were compared against ResNet18, DenseNet121, MobileNetV2, and EfficientNet across accuracy, model size, and CPU inference time. An ensemble model was created by combining Logistic Regression and Random Forests to achieve higher accuracy while retaining efficiency. Results: The single variable Logistic Regression model achieved a test accuracy of 94.80 percent with a file size of 1.2 kB and negligible inference latency (2.3 ms). The two stage ensemble improved accuracy to 97.15 percent. In contrast, the deep learning methods require 13.6 MB to 44.7 MB of storage and show significantly higher inference times (68 ms). Conclusion: This study shows that a compact feature engineering approach can produce clinically meaningful classification performance while offering gains in transparency, reproducibility, speed, and deployment feasibility. The proposed pipeline demonstrates that simple interpretable features paired with lightweight models can serve as a practical diagnostic solution for environments with limited computational resources.
Authors Md Abdullah Al Kafi, Raka Moni, Sumit Kumar Banshal
Categories Computer Vision and Pattern Recognition
0
ActDistill: General Action-Guided Self-Derived Distillation for Efficient Vision-Language-Action Models
Title CN ActDistill：面向高效视觉-语言-动作模型的通用动作引导自推导蒸馏
Keywords 视觉-语言-动作模型 · 知识蒸馏 · 模型压缩 · 动态路由 · 具身智能
Summary 本文提出ActDistill，一种用于视觉-语言-动作模型的通用动作引导自推导蒸馏框架，旨在降低模型计算开销和推理延迟。通过利用预训练VLA模型作为教师，采用图结构封装策略建模动作预测的层次演化过程，并设计动态路由器根据动作需求自适应选择计算路径。学生模型在推理时仅执行必要层，实现在显著降低计算量的同时保持高精度动作预测。实验表明该方法可减少50%以上计算量并提速1.67倍。
Reason 该论文属于具身智能（Embodied AI）领域，研究的是视觉-语言-动作（VLA）模型在机器人操作中的高效部署问题。其核心方法是通过动作引导的蒸馏来压缩和加速VLA模型，重点应用场景为机器人控制与交互。根据审稿原则，具身智能/机器人属于明确列出的‘无关领域’，且文中未涉及文档图像理解中的OCR幻觉、复杂排版、长文档推理等核心痛点，也无法迁移其图结构封装或动态路由机制来解决DIU问题。因此判定为直接拒稿。
Abstract Recent Vision-Language-Action (VLA) models have shown impressive flexibility and generalization, yet their deployment in robotic manipulation remains limited by heavy computational overhead and inference latency. In this work, we present ActDistill, a general action-guided self-derived distillation framework that transfers the action prediction capability of any existing VLA model to a lightweight counterpart. Unlike previous efficiency strategies that primarily emphasize vision-language correlations, ActDistill leverages action priors to guide knowledge transfer and model compression, achieving action-oriented efficiency for VLA models. Specifically, we employ a well-trained VLA model as the teacher and introduce a graph-structured encapsulation strategy to explicitly model the hierarchical evolution of action prediction. The student model, derived from the graph-encapsulated teacher, is further equipped with a dynamic router that adaptively selects computation paths based on action prediction demands, guided by hierarchical graph-informed supervision to ensure smooth and efficient evolution. During inference, graph-related auxiliary components are removed, allowing the student to execute only dynamically routed layers and predict high-precision actions with minimal computation and latency. Experiments on embodied benchmarks demonstrate that ActDistill achieves comparable or superior performance to full-scale VLA models while reducing computation by over 50% with up to 1.67 times speedup, thereby establishing a general paradigm toward efficient embodied intelligence.
Authors Wencheng Ye, Tianshi Wang, Lei Zhu, Fengling Li, Guoli Yang
Categories Computer Vision and Pattern Recognition
0
Training-Free Efficient Video Generation via Dynamic Token Carving NeurIPS 2025
Title CN 无需训练的高效视频生成 via 动态Token雕刻
Keywords 视频生成 · 扩散模型 · 推理加速 · 动态注意力 · 无需训练
Summary 本文提出Jenga，一种用于视频扩散模型的高效推理框架，通过动态注意力雕刻和渐进式分辨率生成，在不重新训练模型的前提下显著提升生成速度。该方法利用3D空间填充曲线进行块状注意力计算，并在去噪过程中逐步提升潜在分辨率，在保持生成质量的同时实现高达8.83倍的加速。
Reason 该论文属于视频生成领域，研究的是视频扩散Transformer在推理阶段的效率优化问题。尽管其动态注意力机制和分辨率渐进策略在技术思路上有一定启发性，但整体工作与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。视频时序建模与文档结构理解的任务目标和数据特性差异巨大，方法难以迁移至解决OCR幻觉、细粒度文本定位或复杂排版解析等DIU关键痛点。
Abstract Despite the remarkable generation quality of video Diffusion Transformer (DiT) models, their practical deployment is severely hindered by extensive computational requirements. This inefficiency stems from two key challenges: the quadratic complexity of self-attention with respect to token length and the multi-step nature of diffusion models. To address these limitations, we present Jenga, a novel inference pipeline that combines dynamic attention carving with progressive resolution generation. Our approach leverages two key insights: (1) early denoising steps do not require high-resolution latents, and (2) later steps do not require dense attention. Jenga introduces a block-wise attention mechanism that dynamically selects relevant token interactions using 3D space-filling curves, alongside a progressive resolution strategy that gradually increases latent resolution during generation. Experimental results demonstrate that Jenga achieves substantial speedups across multiple state-of-the-art video diffusion models while maintaining comparable generation quality (8.83$\times$ speedup with 0.01\% performance drop on VBench). As a plug-and-play solution, Jenga enables practical, high-quality video generation on modern hardware by reducing inference time from minutes to seconds -- without requiring model retraining. Code: https://github.com/dvlab-research/Jenga
Authors Yuechen Zhang, Jinbo Xing, Bin Xia, Shaoteng Liu, Bohao Peng, Xin Tao, Pengfei Wan, Eric Lo, Jiaya Jia
Comment NeurIPS 2025, Project Page: https://julianjuaner.github.io/projects/jenga/
Categories Computer Vision and Pattern Recognition
0
VK-Det: Visual Knowledge Guided Prototype Learning for Open-Vocabulary Aerial Object Detection AAAI 2026
Title CN VK-Det：基于视觉知识引导的原型学习用于开放词汇航空目标检测
Keywords 开放词汇检测 · 视觉语言模型 · 航空图像 · 原型学习 · 伪标签
Summary 本文提出VK-Det，一种无需额外监督的开放词汇航空目标检测框架。通过利用视觉编码器的区域感知能力进行细粒度定位，并结合原型感知的伪标签策略，通过特征聚类建模类别边界，实现对新类别的有效检测。在DIOR和DOTA数据集上达到SOTA性能。
Reason 该论文研究的是航空图像中的开放词汇目标检测（OVAD），属于遥感/航空图像理解领域，与文档图像理解（DIU）在数据分布、任务目标和核心挑战上差异巨大。尽管使用了VLM和伪标签等技术，但其方法针对的是遥感场景下的物体定位，无法直接迁移解决文档中的OCR幻觉、密集文本识别或复杂版式理解等问题。属于无关领域应用。
Abstract To identify objects beyond predefined categories, open-vocabulary aerial object detection (OVAD) leverages the zero-shot capabilities of visual-language models (VLMs) to generalize from base to novel categories. Existing approaches typically utilize self-learning mechanisms with weak text supervision to generate region-level pseudo-labels to align detectors with VLMs semantic spaces. However, text dependence induces semantic bias, restricting open-vocabulary expansion to text-specified concepts. We propose $\textbf{VK-Det}$, a $\textbf{V}$isual $\textbf{K}$nowledge-guided open-vocabulary object $\textbf{Det}$ection framework $\textit{without}$ extra supervision. First, we discover and leverage vision encoder's inherent informative region perception to attain fine-grained localization and adaptive distillation. Second, we introduce a novel prototype-aware pseudo-labeling strategy. It models inter-class decision boundaries through feature clustering and maps detection regions to latent categories via prototype matching. This enhances attention to novel objects while compensating for missing supervision. Extensive experiments show state-of-the-art performance, achieving 30.1 $\mathrm{mAP}^{N}$ on DIOR and 23.3 $\mathrm{mAP}^{N}$ on DOTA, outperforming even extra supervised methods.
Authors Jianhang Yao, Yongbin Zheng, Siqi Lu, Wanying Xu, Peng Sun
Comment 15 pages, 8 figures, accepted by AAAI 2026
Categories Computer Vision and Pattern Recognition
0
pFedBBN: A Personalized Federated Test-Time Adaptation with Balanced Batch Normalization for Class-Imbalanced Data
Title CN pFedBBN：一种用于类别不平衡数据的个性化联邦测试时自适应与平衡批归一化方法
Keywords 联邦学习 · 测试时自适应 · 类别不平衡 · 批归一化 · 无监督学习
Summary 本文提出pFedBBN，一种在联邦学习框架下解决测试时类别不平衡问题的自适应方法，通过平衡批归一化（BBN）实现无监督的本地模型调整，并引入基于BBN相似性的客户端协作机制，在不共享标签或原始数据的前提下提升少数类性能。实验表明该方法在多种基准上优于现有联邦学习与TTA技术。
Reason 该论文研究的是联邦学习中的测试时自适应问题，聚焦于类别不平衡下的分布偏移和客户端协作，属于通用计算机视觉与机器学习领域。尽管涉及视觉任务中的模型鲁棒性，但其方法设计并非针对文档图像理解（DIU）特有的挑战（如OCR幻觉、复杂版式、细粒度文本定位等），也未提出可迁移至DIU领域的底层技术创新（如视觉token压缩、高分辨率多模态对齐、推理链机制等）。属于无关领域的平行应用，不符合关注方向。
Abstract Test-time adaptation (TTA) in federated learning (FL) is crucial for handling unseen data distributions across clients, particularly when faced with domain shifts and skewed class distributions. Class Imbalance (CI) remains a fundamental challenge in FL, where rare but critical classes are often severely underrepresented in individual client datasets. Although prior work has addressed CI during training through reliable aggregation and local class distribution alignment, these methods typically rely on access to labeled data or coordination among clients, and none address class unsupervised adaptation to dynamic domains or distribution shifts at inference time under federated CI constraints. Revealing the failure of state-of-the-art TTA in federated client adaptation in CI scenario, we propose pFedBBN,a personalized federated test-time adaptation framework that employs balanced batch normalization (BBN) during local client adaptation to mitigate prediction bias by treating all classes equally, while also enabling client collaboration guided by BBN similarity, ensuring that clients with similar balanced representations reinforce each other and that adaptation remains aligned with domain-specific characteristics. pFedBBN supports fully unsupervised local adaptation and introduces a class-aware model aggregation strategy that enables personalized inference without compromising privacy. It addresses both distribution shifts and class imbalance through balanced feature normalization and domain-aware collaboration, without requiring any labeled or raw data from clients. Extensive experiments across diverse baselines show that pFedBBN consistently enhances robustness and minority-class performance over state-of-the-art FL and TTA methods.
Authors Md Akil Raihan Iftee, Syed Md. Ahnaf Hasan, Mir Sazzat Hossain, Rakibul Hasan Rajib, Amin Ahsan Ali, AKM Mahbubur Rahman, Sajib Mistry, Monowar Bhuyan
Comment 25 pages, 7 tables, 21 figures
Categories Computer Vision and Pattern Recognition
0
A Lightweight, Interpretable Deep Learning System for Automated Detection of Cervical Adenocarcinoma In Situ (AIS)
Title CN 一种用于自动检测宫颈原位腺癌的轻量级、可解释深度学习系统
Keywords 宫颈原位腺癌 · 病理图像分析 · EfficientNet · Grad-CAM · 医学AI
Summary 本文提出了一种基于EfficientNet-B3的深度学习系统，用于从H&E染色图像中识别宫颈原位腺癌（AIS），使用CAISHI数据集进行训练，并通过Grad-CAM实现可视化解释。模型具备一定的诊断准确性和生物学可解释性，已部署为Gradio虚拟助手。
Reason 该论文属于医学图像分析领域的特定应用，研究任务为宫颈病理切片中的病变分类，与文档图像理解（DIU）或视觉语言模型在文档上的应用无关。其方法基于EfficientNet-B3和Grad-CAM，未提出可用于解决OCR幻觉、复杂排版或长文档推理等DIU核心问题的新架构或通用技术。属于典型的‘平行下游应用’，无文档理解领域的方法论创新。
Abstract Cervical adenocarcinoma in situ (AIS) is a critical premalignant lesion whose accurate histopathological diagnosis is challenging. Early detection is essential to prevent progression to invasive cervical adenocarcinoma. In this study, we developed a deep learning-based virtual pathology assistant capable of distinguishing AIS from normal cervical gland histology using the CAISHI dataset, which contains 2240 expert-labeled H&E images (1010 normal and 1230 AIS). All images underwent Macenko stain normalization and patch-based preprocessing to enhance morphological feature representation. An EfficientNet-B3 convolutional neural network was trained using class-balanced sampling and focal loss to address dataset imbalance and emphasize difficult examples. The final model achieved an overall accuracy of 0.7323, with an F1-score of 0.75 for the Abnormal class and 0.71 for the Normal class. Grad-CAM heatmaps demonstrated biologically interpretable activation patterns, highlighting nuclear atypia and glandular crowding consistent with AIS morphology. The trained model was deployed in a Gradio-based virtual diagnostic assistant. These findings demonstrate the feasibility of lightweight, interpretable AI systems for cervical gland pathology, with potential applications in screening workflows, education, and low-resource settings.
Authors Gabriela Fernandes
Categories Computer Vision and Pattern Recognition
0
Hierarchical Semi-Supervised Active Learning for Remote Sensing
Title CN 用于遥感的分层半监督主动学习
Keywords 遥感 · 半监督学习 · 主动学习 · 场景分类 · 标签效率
Summary 本文提出了一种分层半监督主动学习（HSSAL）框架，结合半监督学习与新型分层主动学习，在遥感图像场景分类任务中实现了高标签效率。通过弱到强的自训练提升模型表征能力，并利用渐进式聚类策略选择最具信息量的未标注样本，仅用极少量标注数据即达到接近全监督的性能。
Reason 该论文研究领域为遥感图像场景分类，属于无关领域（Remote Sensing），与文档图像理解（DIU）任务如DocVQA、表格识别、版面分析等无直接关联。方法虽涉及半监督和主动学习框架，但其技术设计面向遥感影像的大规模场景分类问题，无法直接迁移解决文档图像中的OCR幻觉、复杂排版或细粒度文本定位等核心痛点。
Abstract The performance of deep learning models in remote sensing (RS) strongly depends on the availability of high-quality labeled data. However, collecting large-scale annotations is costly and time-consuming, while vast amounts of unlabeled imagery remain underutilized. To address this challenge, we propose a Hierarchical Semi-Supervised Active Learning (HSSAL) framework that integrates semi-supervised learning (SSL) and a novel hierarchical active learning (HAL) in a closed iterative loop. In each iteration, SSL refines the model using both labeled data through supervised learning and unlabeled data via weak-to-strong self-training, improving feature representation and uncertainty estimation. Guided by the refined representations and uncertainty cues of unlabeled samples, HAL then conducts sample querying through a progressive clustering strategy, selecting the most informative instances that jointly satisfy the criteria of scalability, diversity, and uncertainty. This hierarchical process ensures both efficiency and representativeness in sample selection. Extensive experiments on three benchmark RS scene classification datasets, including UCM, AID, and NWPU-RESISC45, demonstrate that HSSAL consistently outperforms SSL- or AL-only baselines. Remarkably, with only 8%, 4%, and 2% labeled training data on UCM, AID, and NWPU-RESISC45, respectively, HSSAL achieves over 95% of fully-supervised accuracy, highlighting its superior label efficiency through informativeness exploitation of unlabeled data. Our code will be released at https://github.com/zhu-xlab/RS-SSAL.
Authors Wei Huang, Zhitong Xiong, Chenying Liu, Xiao Xiang Zhu
Comment Under review
Categories Computer Vision and Pattern Recognition
0
CoordAR: One-Reference 6D Pose Estimation of Novel Objects via Autoregressive Coordinate Map Generation AAAI 2026 (oral)
Title CN CoordAR：通过自回归坐标图生成实现新物体的单参考6D姿态估计
Keywords 6D姿态估计 · 自回归生成 · 坐标映射 · 无3D模型 · 机器人视觉
Summary 本文提出CoordAR，一种用于新物体6D姿态估计的自回归框架，通过将3D-3D对应关系建模为离散token地图，并采用解耦编码和自回归Transformer解码器，在对称性和遮挡场景下表现出更强鲁棒性。但其研究领域为计算机视觉中的三维感知，与文档理解无关。
Reason 该论文研究的是三维物体6D姿态估计，属于机器人和增强现实领域，核心问题是基于单张参考图像估计未知物体的三维位置和朝向。尽管使用了视觉-语言模型中的一些技术（如Transformer、tokenization），但其任务与文档图像理解（DIU）无直接关联，既不涉及文本识别、布局分析、表格理解，也无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract Object 6D pose estimation, a crucial task for robotics and augmented reality applications, becomes particularly challenging when dealing with novel objects whose 3D models are not readily available. To reduce dependency on 3D models, recent studies have explored one-reference-based pose estimation, which requires only a single reference view instead of a complete 3D model. However, existing methods that rely on real-valued coordinate regression suffer from limited global consistency due to the local nature of convolutional architectures and face challenges in symmetric or occluded scenarios owing to a lack of uncertainty modeling. We present CoordAR, a novel autoregressive framework for one-reference 6D pose estimation of unseen objects. CoordAR formulates 3D-3D correspondences between the reference and query views as a map of discrete tokens, which is obtained in an autoregressive and probabilistic manner. To enable accurate correspondence regression, CoordAR introduces 1) a novel coordinate map tokenization that enables probabilistic prediction over discretized 3D space; 2) a modality-decoupled encoding strategy that separately encodes RGB appearance and coordinate cues; and 3) an autoregressive transformer decoder conditioned on both position-aligned query features and the partially generated token sequence. With these novel mechanisms, CoordAR significantly outperforms existing methods on multiple benchmarks and demonstrates strong robustness to symmetry, occlusion, and other challenges in real-world tests.
Authors Dexin Zuo, Ang Li, Wei Wang, Wenxian Yu, Danping Zou
Comment 7 pages, accepted by AAAI 2026 (oral)
Categories Computer Vision and Pattern Recognition
0
UltraFlux: Data-Model Co-Design for High-quality Native 4K Text-to-Image Generation across Diverse Aspect Ratios
Title CN UltraFlux：面向多样化宽高比的高质量原生4K文本到图像生成的数据-模型协同设计
Keywords 文本到图像生成 · 高分辨率扩散模型 · DiT · 数据-模型协同设计 · 4K生成
Summary 本文提出UltraFlux，一种基于Flux架构的DiT模型，通过数据-模型协同设计实现原生4K、多宽高比的高质量文本到图像生成。其创新包括Resonance 2D RoPE位置编码、非对抗性VAE后训练、SNR感知小波损失函数以及分阶段美学课程学习，并在4K生成质量上超越现有开源模型。
Reason 该论文属于纯图像生成领域，研究的是高分辨率文本到图像生成，尽管涉及多模态和高分辨率处理，但其目标与文档图像理解（DIU）如DocVQA、布局分析、表格识别等核心任务无直接关联。方法聚焦于扩散Transformer在4K生成中的位置编码、VAE重建和美学优化，无法迁移用于解决OCR幻觉、密集文本理解或复杂排版解析等DIU痛点。
Abstract Diffusion transformers have recently delivered strong text-to-image generation around 1K resolution, but we show that extending them to native 4K across diverse aspect ratios exposes a tightly coupled failure mode spanning positional encoding, VAE compression, and optimization. Tackling any of these factors in isolation leaves substantial quality on the table. We therefore take a data-model co-design view and introduce UltraFlux, a Flux-based DiT trained natively at 4K on MultiAspect-4K-1M, a 1M-image 4K corpus with controlled multi-AR coverage, bilingual captions, and rich VLM/IQA metadata for resolution- and AR-aware sampling. On the model side, UltraFlux couples (i) Resonance 2D RoPE with YaRN for training-window-, frequency-, and AR-aware positional encoding at 4K; (ii) a simple, non-adversarial VAE post-training scheme that improves 4K reconstruction fidelity; (iii) an SNR-Aware Huber Wavelet objective that rebalances gradients across timesteps and frequency bands; and (iv) a Stage-wise Aesthetic Curriculum Learning strategy that concentrates high-aesthetic supervision on high-noise steps governed by the model prior. Together, these components yield a stable, detail-preserving 4K DiT that generalizes across wide, square, and tall ARs. On the Aesthetic-Eval at 4096 benchmark and multi-AR 4K settings, UltraFlux consistently outperforms strong open-source baselines across fidelity, aesthetic, and alignment metrics, and-with a LLM prompt refiner-matches or surpasses the proprietary Seedream 4.0.
Authors Tian Ye, Song Fei, Lei Zhu
Comment Project Page: https://w2genai-lab.github.io/UltraFlux/
Categories Computer Vision and Pattern Recognition
0
ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving
Title CN ResAD：面向端到端自动驾驶的归一化残差轨迹建模
Keywords 自动驾驶 · 轨迹预测 · 残差建模 · 优化不平衡 · 端到端系统
Summary 本文提出ResAD，一种用于端到端自动驾驶的归一化残差轨迹建模框架。通过引入惯性参考路径并预测残差偏移，结合逐点归一化缓解长期预测中的优化不平衡问题，在NAVSIM基准上取得SOTA性能。
Reason 该论文属于自动驾驶领域，研究任务为端到端驾驶轨迹预测，与文档图像理解（DIU）或视觉语言模型（VLM）在文档场景下的应用无直接关联。尽管其方法涉及时空数据不平衡和优化策略改进，但核心问题域远离DIU关注的OCR幻觉、复杂排版、细粒度定位等痛点，无法迁移至文档理解任务。
Abstract End-to-end autonomous driving (E2EAD) systems, which learn to predict future trajectories directly from sensor data, are fundamentally challenged by the inherent spatio-temporal imbalance of trajectory data. This imbalance creates a significant optimization burden, causing models to learn spurious correlations instead of robust driving logic, while also prioritizing uncertain, distant predictions, thereby compromising immediate safety. To address these issues, we propose ResAD, a novel Normalized Residual Trajectory Modeling framework. Instead of predicting the future trajectory directly, our approach reframes and simplifies the learning task by predicting the residual deviation from a deterministic inertial reference. This inertial reference serves as a strong physical prior, compelling the model to move beyond simple pattern-matching and instead focus its capacity on learning the necessary, context-driven deviations (e.g., traffic rules, obstacles) from this default, inertially-guided path. To mitigate the optimization imbalance caused by uncertain, long-term horizons, ResAD further incorporates Point-wise Normalization of the predicted residual. This technique re-weights the optimization objective, preventing large-magnitude errors associated with distant, uncertain waypoints from dominating the learning signal. On the NAVSIM v1 and v2 benchmarks, ResAD achieves state-of-the-art results of 88.8 PDMS and 85.5 EPDMS with only two denoising steps, demonstrating that ResAD significantly simplifies the learning task and improves planning performance. The code will be released to facilitate further research.
Authors Zhiyu Zheng, Shaoyu Chen, Haoran Yin, Xinbang Zhang, Jialv Zou, Xinggang Wang, Qian Zhang, Lefei Zhang
Categories Computer Vision and Pattern Recognition
0
Hybrid Event Frame Sensors: Modeling, Calibration, and Simulation
Title CN 混合事件帧传感器：建模、校准与仿真
Keywords 事件相机 · 噪声建模 · 传感器融合 · 图像仿真 · 标定
Summary 本文提出了首个联合描述主动像素传感器（APS）和事件视觉传感器（EVS）噪声行为的统计成像噪声模型，并开发了相应的标定流程与名为HESIM的仿真器，用于生成具有真实噪声特性的原始帧和事件数据。实验验证了该模型在视频插帧和去模糊等任务中的有效性，并展示了从仿真到真实数据的良好迁移性。
Reason 该论文研究的是事件相机（Event Camera）中的传感器噪声建模与仿真，属于硬件成像与传感器物理层面的工作，核心关注点为APS和EVS像素的噪声行为。其任务与文档图像理解（DIU）无直接关联，既不涉及文本识别、布局分析、表格理解等DIU核心任务，也未提出可迁移至文档理解领域的通用方法（如视觉表示、推理机制或VLM架构改进）。虽然其仿真与建模思路在广义上可用于数据增强，但迁移到OCR或DocVQA场景中极为困难，且不属于解决OCR幻觉、复杂排版或长文档推理等痛点的‘军火库’型工作。
Abstract Event frame hybrid sensors integrate an Active Pixel Sensor (APS) and an Event Vision Sensor (EVS) within a single chip, combining the high dynamic range and low latency of the EVS with the rich spatial intensity information from the APS. While this tight integration offers compact, temporally precise imaging, the complex circuit architecture introduces non-trivial noise patterns that remain poorly understood and unmodeled. In this work, we present the first unified, statistics-based imaging noise model that jointly describes the noise behavior of APS and EVS pixels. Our formulation explicitly incorporates photon shot noise, dark current noise, fixed-pattern noise, and quantization noise, and links EVS noise to illumination level and dark current. Based on this formulation, we further develop a calibration pipeline to estimate noise parameters from real data and offer a detailed analysis of both APS and EVS noise behaviors. Finally, we propose HESIM, a statistically grounded simulator that generates RAW frames and events under realistic, jointly calibrated noise statistics. Experiments on two hybrid sensors validate our model across multiple imaging tasks (e.g., video frame interpolation and deblurring), demonstrating strong transfer from simulation to real data.
Authors Yunfan Lu, Nico Messikommer, Xiaogang Xu, Liming Chen, Yuhan Chen, Nikola Zubic, Davide Scaramuzza, Hui Xiong
Categories Computer Vision and Pattern Recognition
0
DMAT: An End-to-End Framework for Joint Atmospheric Turbulence Mitigation and Object Detection WACV2026
Title CN DMAT：一种用于联合大气湍流抑制与目标检测的端到端框架
Keywords 大气湍流抑制 · 目标检测 · 3D Mamba · 视频恢复 · 端到端学习
Summary 本文提出了一种名为DMAT的端到端框架，用于同时缓解大气湍流引起的图像退化并提升目标检测性能。该方法通过在AT抑制模块中引入基于3D Mamba的结构来建模时空位移与模糊，并与目标检测器共享特征进行联合优化。实验表明，该方法在合成湍流数据集上优于现有技术，目标检测性能提升最高达15%。
Reason 该论文研究的是大气湍流导致的监控图像退化问题，属于遥感成像与视频复原领域，核心任务为视频级去模糊与动态目标检测。其技术路线聚焦于3D Mamba结构处理时空畸变，与文档图像理解（DIU）中的OCR幻觉、复杂版式解析、细粒度文本定位等核心痛点无直接关联。尽管使用了VLM相关架构（如Mamba），但应用场景和数据模态与文档图像差异巨大，无法迁移至DIU领域。
Abstract Atmospheric Turbulence (AT) degrades the clarity and accuracy of surveillance imagery, posing challenges not only for visualization quality but also for object classification and scene tracking. Deep learning-based methods have been proposed to improve visual quality, but spatio-temporal distortions remain a significant issue. Although deep learning-based object detection performs well under normal conditions, it struggles to operate effectively on sequences distorted by atmospheric turbulence. In this paper, we propose a novel framework that learns to compensate for distorted features while simultaneously improving visualization and object detection. This end-to-end training strategy leverages and exchanges knowledge of low-level distorted features in the AT mitigator with semantic features extracted in the object detector. Specifically, in the AT mitigator a 3D Mamba-based structure is used to handle the spatio-temporal displacements and blurring caused by turbulence. Optimization is achieved through back-propagation in both the AT mitigator and object detector. Our proposed DMAT outperforms state-of-the-art AT mitigation and object detection systems up to a 15% improvement on datasets corrupted by generated turbulence.
Authors Paul Hill, Zhiming Liu, Alin Achim, Dave Bull, Nantheera Anantrasirichai
Comment Accepted to WACV2026
Categories Computer Vision and Pattern Recognition
0
Pressure2Motion: Hierarchical Human Motion Reconstruction from Ground Pressure with Text Guidance
Title CN Pressure2Motion：基于地面压力与文本引导的分层人体动作重建
Keywords 动作捕捉 · 压力传感器 · 文本引导 · 扩散模型 · 多模态生成
Summary 本文提出Pressure2Motion，一种仅需压力地垫和文本提示即可重建人体动作的新方法，无需摄像头或可穿戴设备，适用于隐私保护和低光环境。由于压力信号对全身动作存在严重歧义，作者设计了一个双层级特征提取器和分层扩散模型，结合压力数据的物理线索与文本描述的语义先验来提升动作生成的准确性。论文还构建了首个该任务的基准MPL。实验表明该方法能生成高保真、符合物理规律的动作序列。
Reason 该论文研究的是从地面压力信号和文本提示中重建人体动作，属于具身智能与运动捕捉领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。尽管使用了多模态（压力+文本）建模和分层扩散结构，但其方法针对的是时序物理信号与动作生成的映射问题，难以迁移到文档理解中的OCR去幻觉、复杂排版解析或长文档推理等核心痛点。
Abstract We present Pressure2Motion, a novel motion capture algorithm that reconstructs human motion from a ground pressure sequence and text prompt. At inference time, Pressure2Motion requires only a pressure mat, eliminating the need for specialized lighting setups, cameras, or wearable devices, making it suitable for privacy-preserving, low-light, and low-cost motion capture scenarios. Such a task is severely ill-posed due to the indeterminacy of pressure signals with respect to full-body motion. To address this issue, we introduce Pressure2Motion, a generative model that leverages pressure features as input and utilizes a text prompt as a high-level guiding constraint to resolve ambiguities. Specifically, our model adopts a dual-level feature extractor to accurately interpret pressure data, followed by a hierarchical diffusion model that discerns broad-scale movement trajectories and subtle posture adjustments. Both the physical cues gained from the pressure sequence and the semantic guidance derived from descriptive texts are leveraged to guide the motion estimation with precision. To the best of our knowledge, Pressure2Motion is a pioneering work in leveraging both pressure data and linguistic priors for motion reconstruction, and the established MPL benchmark is the first benchmark for this novel motion capture task. Experiments show that our method generates high-fidelity, physically plausible motions, establishing a new state of the art for this task. The codes and benchmarks will be publicly released upon publication.
Authors Zhengxuan Li, Qinhui Yang, Yiyu Zhuang, Chuan Guo, Xinxin Zuo, Xiaoxiao Long, Yao Yao, Xun Cao, Qiu Shen, Hao Zhu
Categories Computer Vision and Pattern Recognition
0
Improvement of Spiking Neural Network with Bit Planes and Color Models IEEE Access
Title CN 利用位平面和颜色模型改进脉冲神经网络
Keywords 脉冲神经网络 · 位平面 · 颜色模型 · 神经网络编码 · 能效
Summary 本文提出一种基于位平面表示的新编码方法，用于提升脉冲神经网络（SNN）在图像任务上的性能，同时保持模型大小不变。作者还研究了不同颜色模型对该编码过程的影响。实验表明该方法在多个数据集上有效提升了SNN的准确率。据作者所知，这是首次将位平面和颜色模型引入SNN的研究。
Reason 该论文研究的是脉冲神经网络（SNN）在图像任务中的编码方法优化，属于计算神经科学与低功耗AI的交叉方向。尽管涉及视觉输入，但其核心贡献是针对SNN的编码策略，并未面向文档图像理解（DIU）任务（如DocVQA、布局分析、表格识别等）设计，也未解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。此外，该方法缺乏向VLM或文档多模态理解迁移的明显路径，属于无关领域的技术探索。
Abstract Spiking neural network (SNN) has emerged as a promising paradigm in computational neuroscience and artificial intelligence, offering advantages such as low energy consumption and small memory footprint. However, their practical adoption is constrained by several challenges, prominently among them being performance optimization. In this study, we present a novel approach to enhance the performance of SNN for images through a new coding method that exploits bit plane representation. Our proposed technique is designed to improve the accuracy of SNN without increasing model size. Also, we investigate the impacts of color models of the proposed coding process. Through extensive experimental validation, we demonstrate the effectiveness of our coding strategy in achieving performance gain across multiple datasets. To the best of our knowledge, this is the first research that considers bit planes and color models in the context of SNN. By leveraging the unique characteristics of bit planes, we hope to unlock new potentials in SNNs performance, potentially paving the way for more efficient and effective SNNs models in future researches and applications.
Authors Nhan T. Luu, Duong T. Luu, Nam N. Pham, Thang C. Truong
Comment Accepted for publication at IEEE Access
Categories Computer Vision and Pattern Recognition
0
RAISECity: A Multimodal Agent Framework for Reality-Aligned 3D World Generation at City-Scale
Title CN RAISECity：一种用于城市尺度现实对齐3D世界生成的多模态智能体框架
Keywords 3D生成 · 城市尺度 · 多模态智能体 · 现实对齐 · 世界模型
Summary 本文提出RAISECity，一个用于生成城市尺度、高保真3D世界的多模态智能体框架。该框架通过调用多种基础模型工具，实现现实知识获取、中间表示维护和复杂场景构建，具备动态数据处理、自我反思与迭代优化能力，在形状精度、纹理质量和感知真实性方面显著优于现有方法。
Reason 该论文研究的是城市尺度的3D世界生成，属于具身智能与3D视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心技术聚焦于3D场景构建、现实对齐与多模态智能体在物理空间中的协同，无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心痛点。
Abstract City-scale 3D generation is of great importance for the development of embodied intelligence and world models. Existing methods, however, face significant challenges regarding quality, fidelity, and scalability in 3D world generation. Thus, we propose RAISECity, a \textbf{R}eality-\textbf{A}ligned \textbf{I}ntelligent \textbf{S}ynthesis \textbf{E}ngine that creates detailed, \textbf{C}ity-scale 3D worlds. We introduce an agentic framework that leverages diverse multimodal foundation tools to acquire real-world knowledge, maintain robust intermediate representations, and construct complex 3D scenes. This agentic design, featuring dynamic data processing, iterative self-reflection and refinement, and the invocation of advanced multimodal tools, minimizes cumulative errors and enhances overall performance. Extensive quantitative experiments and qualitative analyses validate the superior performance of RAISECity in real-world alignment, shape precision, texture fidelity, and aesthetics level, achieving over a 90% win-rate against existing baselines for overall perceptual quality. This combination of 3D quality, reality alignment, scalability, and seamless compatibility with computer graphics pipelines makes RAISECity a promising foundation for applications in immersive media, embodied intelligence, and world models.
Authors Shengyuan Wang, Zhiheng Zheng, Yu Shang, Lixuan He, Yangcheng Yu, Fan Hangyu, Jie Feng, Qingmin Liao, Yong Li
Comment The code will be made publicly available soon at: https://github.com/tsinghua-fib-lab/RAISECity
Categories Computer Vision and Pattern Recognition
0
GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving
Title CN GraphPilot：基于场景图条件的语言驱动自动驾驶
Keywords Vision-Language Model · Autonomous Driving · Scene Graph · Relational Reasoning · Prompt Engineering
Summary 本文提出GraphPilot，通过将交通场景图以序列化形式注入语言驱动的自动驾驶模型，增强其对交通实体间关系的理解能力。实验表明该方法显著提升了LMDrive和BEVDriver等模型的驾驶性能。然而，该工作聚焦于自动驾驶决策，与文档理解任务无关。
Reason 该论文属于具身智能与自动驾驶领域，研究的是如何将场景图结构信息融入语言驱动的自动驾驶决策中。尽管其方法涉及多模态推理和结构化知识注入，但应用场景与文档图像理解（DIU）无直接关联，且无法迁移解决OCR幻觉、密集文本识别、复杂版面分析等核心DIU问题。属于‘无关领域’中的自动驾驶方向，命中负面清单第2条。
Abstract Vision-language models have recently emerged as promising planners for autonomous driving, where success hinges on topology-aware reasoning over spatial structure and dynamic interactions from multimodal input. However, existing models are typically trained without supervision that explicitly encodes these relational dependencies, limiting their ability to infer how agents and other traffic entities influence one another from raw sensor data. In this work, we bridge this gap with a novel model-agnostic method that conditions language-based driving models on structured relational context in the form of traffic scene graphs. We serialize scene graphs at various abstraction levels and formats, and incorporate them into the models via structured prompt templates, enabling a systematic analysis of when and how relational supervision is most beneficial. Extensive evaluations on the public LangAuto benchmark show that scene graph conditioning of state-of-the-art approaches yields large and persistent improvement in driving performance. Notably, we observe up to a 15.6\% increase in driving score for LMDrive and 17.5\% for BEVDriver, indicating that models can better internalize and ground relational priors through scene graph-conditioned training, even without requiring scene graph input at test-time. Code, fine-tuned models, and our scene graph dataset are publicly available at https://github.com/iis-esslingen/GraphPilot.
Authors Fabian Schmidt, Markus Enzweiler, Abhinav Valada
Categories Computer Vision and Pattern Recognition
0
SD-PSFNet: Sequential and Dynamic Point Spread Function Network for Image Deraining AAAI 2026
Title CN SD-PSFNet：用于图像去雨的序列化与动态点扩散函数网络
Keywords 图像去雨 · 点扩散函数 · 多阶段恢复 · 物理建模 · 特征融合
Summary 本文提出了一种名为SD-PSFNet的新型图像去雨方法，通过引入可学习的点扩散函数（PSF）机制，结合动态物理建模与序列化特征融合，在多个真实与合成数据集上实现了最先进的去雨效果。网络采用三阶段级联结构，逐级优化去雨结果，并通过自适应门控融合实现跨阶段特征整合。
Reason 该论文研究的是图像去雨任务，属于通用图像恢复领域，尽管其方法具有物理建模和多阶段恢复结构，但并未涉及文档图像理解（DIU）的核心问题，如OCR、布局分析、表格识别或视觉语言推理。其技术路线难以直接迁移至解决文档中的幻觉、密集文本处理或复杂排版理解等痛点，且不属于VLM或多模态LLM在文档上的应用或改进。
Abstract Image deraining is crucial for vision applications but is challenged by the complex multi-scale physics of rain and its coupling with scenes. To address this challenge, a novel approach inspired by multi-stage image restoration is proposed, incorporating Point Spread Function (PSF) mechanisms to reveal the image degradation process while combining dynamic physical modeling with sequential feature fusion transfer, named SD-PSFNet. Specifically, SD-PSFNet employs a sequential restoration architecture with three cascaded stages, allowing multiple dynamic evaluations and refinements of the degradation process estimation. The network utilizes components with learned PSF mechanisms to dynamically simulate rain streak optics, enabling effective rain-background separation while progressively enhancing outputs through novel PSF components at each stage. Additionally, SD-PSFNet incorporates adaptive gated fusion for optimal cross-stage feature integration, enabling sequential refinement from coarse rain removal to fine detail restoration. Our model achieves state-of-the-art PSNR/SSIM metrics on Rain100H (33.12dB/0.9371), RealRain-1k-L (42.28dB/0.9872), and RealRain-1k-H (41.08dB/0.9838). In summary, SD-PSFNet demonstrates excellent capability in complex scenes and dense rainfall conditions, providing a new physics-aware approach to image deraining.
Authors Jiayu Wang, Haoyu Bian, Haoran Sun, Shaoning Zeng
Comment 12 pages, 7 figures, Published in AAAI 2026
Categories Computer Vision and Pattern Recognition
0
Physics-Informed Deformable Gaussian Splatting: Towards Unified Constitutive Laws for Time-Evolving Material Field AAAI-26
Title CN 物理信息引导的可变形高斯点阵：面向时间演化材料场的统一本构定律
Keywords 3D Gaussian Splatting · 物理信息网络 · 动态场景重建 · 本构模型 · 光流监督
Summary 本文提出了一种物理信息引导的可变形高斯点阵方法（PIDG），将每个高斯粒子视为具有时变本构参数的拉格朗日质点，并通过2D光流监督和物理约束（柯西动量残差）提升动态场景重建的物理一致性和质量。方法依赖于4D哈希编码和材料场建模，在自定义及标准数据集上验证了有效性。
Reason 该论文属于3D视觉与动态场景重建领域，研究的是基于物理规律的高斯点阵方法用于动态视图合成，核心关注点为3D场景建模与物理一致性运动预测。其技术路线与文档图像理解（DIU）任务无直接关联，无法迁移解决OCR幻觉、复杂排版、细粒度文本识别或长文档推理等核心痛点。
Abstract Recently, 3D Gaussian Splatting (3DGS), an explicit scene representation technique, has shown significant promise for dynamic novel-view synthesis from monocular video input. However, purely data-driven 3DGS often struggles to capture the diverse physics-driven motion patterns in dynamic scenes. To fill this gap, we propose Physics-Informed Deformable Gaussian Splatting (PIDG), which treats each Gaussian particle as a Lagrangian material point with time-varying constitutive parameters and is supervised by 2D optical flow via motion projection. Specifically, we adopt static-dynamic decoupled 4D decomposed hash encoding to reconstruct geometry and motion efficiently. Subsequently, we impose the Cauchy momentum residual as a physics constraint, enabling independent prediction of each particle's velocity and constitutive stress via a time-evolving material field. Finally, we further supervise data fitting by matching Lagrangian particle flow to camera-compensated optical flow, which accelerates convergence and improves generalization. Experiments on a custom physics-driven dataset as well as on standard synthetic and real-world datasets demonstrate significant gains in physical consistency and monocular dynamic reconstruction quality.
Authors Haoqin Hong, Ding Fan, Fubin Dou, Zhi-Li Zhou, Haoran Sun, Congcong Zhu, Jingrun Chen
Comment Accepted by AAAI-26
Categories Computer Vision and Pattern Recognition
0
HyM-UNet: Synergizing Local Texture and Global Context via Hybrid CNN-Mamba Architecture for Medical Image Segmentation
Title CN HyM-UNet：通过混合CNN-Mamba架构协同局部纹理与全局上下文用于医学图像分割
Keywords 医学图像分割 · CNN · Mamba · Hybrid Architecture · 语义分割
Summary 本文提出HyM-UNet，一种结合CNN与Mamba的混合网络，用于提升医学图像中器官和病灶的分割精度。通过在编码器浅层保留CNN提取局部纹理，在深层引入Mamba捕获长距离依赖，并设计MGF-Skip连接融合多级特征，有效抑制背景噪声。在ISIC 2018数据集上取得了优于现有方法的性能。
Reason 该论文属于纯医学图像分割领域，任务为器官和病变区域的语义分割，属于典型的医疗影像分析应用。尽管使用了有潜力的Mamba架构，但其方法设计完全针对生物医学图像中的解剖结构建模，与文档图像理解（DIU）的核心痛点如OCR幻觉、文本布局分析、表格识别或VLM推理无关。且未涉及多模态语言模型、视觉-语言对齐或文档级结构建模，无法迁移到DocVQA或相关任务中。
Abstract Accurate organ and lesion segmentation is a critical prerequisite for computer-aided diagnosis. Convolutional Neural Networks (CNNs), constrained by their local receptive fields, often struggle to capture complex global anatomical structures. To tackle this challenge, this paper proposes a novel hybrid architecture, HyM-UNet, designed to synergize the local feature extraction capabilities of CNNs with the efficient global modeling capabilities of Mamba. Specifically, we design a Hierarchical Encoder that utilizes convolutional modules in the shallow stages to preserve high-frequency texture details, while introducing Visual Mamba modules in the deep stages to capture long-range semantic dependencies with linear complexity. To bridge the semantic gap between the encoder and the decoder, we propose a Mamba-Guided Fusion Skip Connection (MGF-Skip). This module leverages deep semantic features as gating signals to dynamically suppress background noise within shallow features, thereby enhancing the perception of ambiguous boundaries. We conduct extensive experiments on public benchmark dataset ISIC 2018. The results demonstrate that HyM-UNet significantly outperforms existing state-of-the-art methods in terms of Dice coefficient and IoU, while maintaining lower parameter counts and inference latency. This validates the effectiveness and robustness of the proposed method in handling medical segmentation tasks characterized by complex shapes and scale variations.
Authors Haodong Chen, Xianfei Han, Qwen
Categories Computer Vision and Pattern Recognition
0
FeRA: Frequency-Energy Constrained Routing for Effective Diffusion Adaptation Fine-Tuning
Title CN FeRA：用于有效扩散适应微调的频率-能量约束路由
Keywords 扩散模型 · 微调 · 频率能量 · 适配器 · 图像生成
Summary 本文提出FeRA，一种基于频率能量机制的扩散模型微调框架，通过频率指示器、软路由器和一致性正则化来优化适配器在不同频段的能量分配。该方法提升扩散模型在新任务上的适应效率与稳定性，适用于各类扩散架构和分辨率，但局限于图像生成场景。
Reason 该论文聚焦于扩散模型的微调机制，属于图像生成领域，其核心方法围绕频率能量分布和适配器路由优化生成过程。尽管涉及视觉表示，但目标是生成建模而非文档理解或视觉语言推理。无法迁移解决文档图像理解中的OCR幻觉、复杂排版或细粒度定位等关键问题，属于无关领域。
Abstract Diffusion models have achieved remarkable success in generative modeling, yet how to effectively adapt large pretrained models to new tasks remains challenging. We revisit the reconstruction behavior of diffusion models during denoising to unveil the underlying frequency energy mechanism governing this process. Building upon this observation, we propose FeRA, a frequency driven fine tuning framework that aligns parameter updates with the intrinsic frequency energy progression of diffusion. FeRA establishes a comprehensive frequency energy framework for effective diffusion adaptation fine tuning, comprising three synergistic components: (i) a compact frequency energy indicator that characterizes the latent bandwise energy distribution, (ii) a soft frequency router that adaptively fuses multiple frequency specific adapter experts, and (iii) a frequency energy consistency regularization that stabilizes diffusion optimization and ensures coherent adaptation across bands. Routing operates in both training and inference, with inference time routing dynamically determined by the latent frequency energy. It integrates seamlessly with adapter based tuning schemes and generalizes well across diffusion backbones and resolutions. By aligning adaptation with the frequency energy mechanism, FeRA provides a simple, stable, and compatible paradigm for effective and robust diffusion model adaptation.
Authors Bo Yin, Xiaobin Hu, Xingyu Zhou, Peng-Tao Jiang, Yue Liao, Junwei Zhu, Jiangning Zhang, Ying Tai, Chengjie Wang, Shuicheng Yan
Categories Computer Vision and Pattern Recognition
0
Adversarial Pseudo-replay for Exemplar-free Class-incremental Learning WACV 2026
Title CN 用于无样本类增量学习的对抗伪回放
Keywords 类增量学习 · 对抗攻击 · 知识蒸馏 · 语义漂移 · 无样本回放
Summary 本文提出了一种名为对抗伪回放（APR）的方法，用于在不存储旧类图像的情况下进行无样本类增量学习。通过在新任务图像上施加以旧类原型为目标的对抗攻击，生成伪回放样本，并利用这些样本来进行知识蒸馏，防止语义漂移。此外，还通过学习转移矩阵校正协方差矩阵以进一步缓解遗忘问题。方法在标准EFCIL基准的冷启动设置下达到了SOTA性能。
Reason 该论文研究的是类增量学习中的模型遗忘问题，属于通用计算机视觉中的增量学习范畴，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其提出的方法‘对抗伪回放’针对图像分类任务中的知识保留，未涉及OCR、布局分析、表格识别或多模态文档推理等DIU核心问题，也无法迁移解决OCR幻觉、密集文本处理或复杂排版理解等关键挑战。
Abstract Exemplar-free class-incremental learning (EFCIL) aims to retain old knowledge acquired in the previous task while learning new classes, without storing the previous images due to storage constraints or privacy concerns. In EFCIL, the plasticity-stability dilemma, learning new tasks versus catastrophic forgetting, is a significant challenge, primarily due to the unavailability of images from earlier tasks. In this paper, we introduce adversarial pseudo-replay (APR), a method that perturbs the images of the new task with adversarial attack, to synthesize the pseudo-replay images online without storing any replay samples. During the new task training, the adversarial attack is conducted on the new task images with augmented old class mean prototypes as targets, and the resulting images are used for knowledge distillation to prevent semantic drift. Moreover, we calibrate the covariance matrices to compensate for the semantic drift after each task, by learning a transfer matrix on the pseudo-replay samples. Our method reconciles stability and plasticity, achieving state-of-the-art on challenging cold-start settings of the standard EFCIL benchmarks.
Authors Hiroto Honda
Comment Accepted to WACV 2026
Categories Computer Vision and Pattern Recognition
0
CADTrack: Learning Contextual Aggregation with Deformable Alignment for Robust RGBT Tracking AAAI2026
Title CN CADTrack：通过可变形对齐学习上下文聚合以实现鲁棒的RGBT跟踪
Keywords RGBT跟踪 · Mamba · Mixture-of-Experts · 可变形对齐 · 跨模态融合
Summary 本文提出了一种名为CADTrack的RGBT跟踪框架，通过引入基于Mamba的特征交互模块（MFI）、上下文聚合模块（CAM）和可变形对齐模块（DAM），解决可见光与热红外模态之间的差异问题，提升跨模态特征融合效果和跟踪鲁棒性。实验表明该方法在五个基准上表现优异。
Reason 该论文研究的是RGB-热红外（RGBT）多模态目标跟踪，属于视频目标跟踪领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。尽管其使用了Mamba、MoE和可变形对齐等技术，但这些方法的应用场景是跨模态时序对齐与目标定位，迁移到文档图像中的OCR去幻觉、复杂版面分析或长文档推理等问题上极为困难，且未体现对文档结构、文本布局或多模态对齐的针对性改进。
Abstract RGB-Thermal (RGBT) tracking aims to exploit visible and thermal infrared modalities for robust all-weather object tracking. However, existing RGBT trackers struggle to resolve modality discrepancies, which poses great challenges for robust feature representation. This limitation hinders effective cross-modal information propagation and fusion, which significantly reduces the tracking accuracy. To address this limitation, we propose a novel Contextual Aggregation with Deformable Alignment framework called CADTrack for RGBT Tracking. To be specific, we first deploy the Mamba-based Feature Interaction (MFI) that establishes efficient feature interaction via state space models. This interaction module can operate with linear complexity, reducing computational cost and improving feature discrimination. Then, we propose the Contextual Aggregation Module (CAM) that dynamically activates backbone layers through sparse gating based on the Mixture-of-Experts (MoE). This module can encode complementary contextual information from cross-layer features. Finally, we propose the Deformable Alignment Module (DAM) to integrate deformable sampling and temporal propagation, mitigating spatial misalignment and localization drift. With the above components, our CADTrack achieves robust and accurate tracking in complex scenarios. Extensive experiments on five RGBT tracking benchmarks verify the effectiveness of our proposed method. The source code is released at https://github.com/IdolLab/CADTrack.
Authors Hao Li, Yuhao Wang, Xiantao Hu, Wenning Hao, Pingping Zhang, Dong Wang, Huchuan Lu
Comment Accepted by AAAI2026. More modifications may be performed
Categories Computer Vision and Pattern Recognition
0
HEAL: Learning-Free Source Free Unsupervised Domain Adaptation for Cross-Modality Medical Image Segmentation The 36th British Machine Vision Conference (BMVC 2025)
Title CN HEAL：无需学习的源域自由无监督域自适应方法用于跨模态医学图像分割
Keywords 医学图像分割 · 无监督域自适应 · 跨模态 · 隐私保护 · SFUDA
Summary 本文提出HEAL，一种用于跨模态医学图像分割的源域自由无监督域自适应框架，通过分层去噪、边缘引导选择、尺寸感知融合和无需学习的策略，在不访问源数据的情况下实现域适应。在大规模实验中达到SOTA性能。
Reason 该论文研究的是医学图像分割中的跨模态域自适应问题，属于纯医学图像分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心方法聚焦于医学影像的域迁移和分割性能提升，并未涉及OCR、文本布局理解、多模态文档建模或语言-视觉对齐等DIU关键问题。因此无法迁移到文档理解场景，属于无关领域。
Abstract Growing demands for clinical data privacy and storage constraints have spurred advances in Source Free Unsupervised Domain Adaptation (SFUDA). SFUDA addresses the domain shift by adapting models from the source domain to the unseen target domain without accessing source data, even when target-domain labels are unavailable. However, SFUDA faces significant challenges: the absence of source domain data and label supervision in the target domain due to source free and unsupervised settings. To address these issues, we propose HEAL, a novel SFUDA framework that integrates Hierarchical denoising, Edge-guided selection, size-Aware fusion, and Learning-free characteristic. Large-scale cross-modality experiments demonstrate that our method outperforms existing SFUDA approaches, achieving state-of-the-art (SOTA) performance. The source code is publicly available at: https://github.com/derekshiii/HEAL.
Authors Yulong Shi, Jiapeng Li, Lin Qi
Comment Accepted by The 36th British Machine Vision Conference (BMVC 2025)
Categories Computer Vision and Pattern Recognition
0
SciEducator: Scientific Video Understanding and Educating via Deming-Cycle Multi-Agent System
Title CN SciEducator：基于戴明循环多智能体系统的科学视频理解与教育
Keywords 多智能体系统 · 科学视频理解 · 视频教育 · 自演化推理 · 多模态问答
Summary 本文提出SciEducator，一种基于戴明循环（Plan-Do-Study-Act）的多智能体系统，用于理解和教育科学视频内容。系统通过迭代自演化机制解析复杂的科学活动，并生成包含文本、视觉、音频和交互参考的多模态教学材料。作者构建了SciVBench基准用于评估，在多个科学现象类别上超越主流MLLM和视频代理系统。
Reason 该论文聚焦于科学视频的理解与教育，属于视频理解领域，并涉及具身智能和教育内容生成，与文档图像理解（DIU）或DocVQA的核心任务无直接关联。尽管其采用多智能体系统和推理循环机制，但应用场景为视频模态，且未提出可迁移至文档理解领域的底层方法（如视觉压缩、去幻觉、细粒度定位等）。属于无关领域研究。
Abstract Recent advancements in multimodal large language models (MLLMs) and video agent systems have significantly improved general video understanding. However, when applied to scientific video understanding and educating, a domain that demands external professional knowledge integration and rigorous step-wise reasoning, existing approaches often struggle. To bridge this gap, we propose SciEducator, the first iterative self-evolving multi-agent system for scientific video comprehension and education. Rooted in the classical Deming Cycle from management science, our design reformulates its Plan-Do-Study-Act philosophy into a self-evolving reasoning and feedback mechanism, which facilitates the interpretation of intricate scientific activities in videos. Moreover, SciEducator can produce multimodal educational content tailored to specific scientific processes, including textual instructions, visual guides, audio narrations, and interactive references. To support evaluation, we construct SciVBench, a benchmark consisting of 500 expert-verified and literature-grounded science QA pairs across five categories, covering physical, chemical, and everyday phenomena. Extensive experiments demonstrate that SciEducator substantially outperforms leading closed-source MLLMs (e.g., Gemini, GPT-4o) and state-of-the-art video agents on the benchmark, establishing a new paradigm for the community.
Authors Zhiyu Xu, Weilong Yan, Yufei Shi, Xin Meng, Tao He, Huiping Zhuang, Ming Li, Hehe Fan
Categories Computer Vision and Pattern Recognition
0
V2X-RECT: An Efficient V2X Trajectory Prediction Framework via Redundant Interaction Filtering and Tracking Error Correction
Title CN V2X-RECT：通过冗余交互过滤和跟踪误差校正的高效V2X轨迹预测框架
Keywords V2X · 轨迹预测 · 身份匹配 · 交通信号引导 · 高效推理
Summary 本文提出了一种用于高密度交通场景的V2X轨迹预测框架V2X-RECT，通过多源身份匹配与校正模块提升目标关联稳定性，并利用交通信号引导的交互模块过滤关键交互车辆，同时引入局部时空坐标编码以重用历史轨迹特征，提高推理效率。实验表明该方法在多个V2X数据集上优于现有方法。
Reason 该论文属于自动驾驶与交通预测领域，研究的是车路协同（V2X）环境下的轨迹预测问题，涉及多源感知融合、身份匹配与交通信号引导的交互建模。尽管使用了视觉相关的数据输入，但其核心任务与文档图像理解（DIU）、DocVQA、OCR、布局分析等完全无关。未涉及文本理解、多模态语言模型在文档上的应用、视觉token压缩或文档推理中的幻觉问题，无法迁移至DIU领域的任何核心痛点。
Abstract V2X prediction can alleviate perception incompleteness caused by limited line of sight through fusing trajectory data from infrastructure and vehicles, which is crucial to traffic safety and efficiency. However, in dense traffic scenarios, frequent identity switching of targets hinders cross-view association and fusion. Meanwhile, multi-source information tends to generate redundant interactions during the encoding stage, and traditional vehicle-centric encoding leads to large amounts of repetitive historical trajectory feature encoding, degrading real-time inference performance. To address these challenges, we propose V2X-RECT, a trajectory prediction framework designed for high-density environments. It enhances data association consistency, reduces redundant interactions, and reuses historical information to enable more efficient and accurate prediction. Specifically, we design a multi-source identity matching and correction module that leverages multi-view spatiotemporal relationships to achieve stable and consistent target association, mitigating the adverse effects of mismatches on trajectory encoding and cross-view feature fusion. Then we introduce traffic signal-guided interaction module, encoding trend of traffic light changes as features and exploiting their role in constraining spatiotemporal passage rights to accurately filter key interacting vehicles, while capturing the dynamic impact of signal changes on interaction patterns. Furthermore, a local spatiotemporal coordinate encoding enables reusable features of historical trajectories and map, supporting parallel decoding and significantly improving inference efficiency. Extensive experimental results across V2X-Seq and V2X-Traj datasets demonstrate that our V2X-RECT achieves significant improvements compared to SOTA methods, while also enhancing robustness and inference efficiency across diverse traffic densities.
Authors Xiangyan Kong, Xuecheng Wu, Xiongwei Zhao, Xiaodong Li, Yunyun Shi, Gang Wang, Dingkang Yang, Yang Liu, Hong Chen, Yulong Gao
Categories Computer Vision and Pattern Recognition
0
VCE: Safe Autoregressive Image Generation via Visual Contrast Exploitation
Title CN VCE：通过视觉对比利用实现安全的自回归图像生成
Keywords 自回归图像生成 · 概念擦除 · 视觉对比 · DPO · 内容安全
Summary 本文提出VCE框架，用于在自回归图像生成过程中安全地擦除不安全概念（如NSFW内容或特定艺术风格），通过构建对比图像对并采用DPO训练策略，实现对有害概念的精准移除，同时保留其他无关的视觉内容。实验验证了其在多种擦除任务上的有效性。
Reason 该论文研究的是自回归图像生成中的安全内容过滤问题，属于纯图像生成领域，主要关注NSFW内容和艺术风格的去除。其技术路线围绕生成模型的安全性和伦理问题展开，与文档图像理解（DIU）的核心任务如文本识别、布局分析、表格理解等无直接关联。尽管提出了视觉对比对构建和DPO训练方法，但这些方法针对的是图像内容的概念擦除，难以迁移到解决OCR幻觉、复杂排版解析或长文档推理等DIU关键痛点。
Abstract Recently, autoregressive image generation models have wowed audiences with their remarkable capability in creating surprisingly realistic images. Models such as GPT-4o and LlamaGen can not only produce images that faithfully mimic renowned artistic styles like Ghibli, Van Gogh, or Picasso, but also potentially generate Not-Safe-For-Work (NSFW) content, raising significant concerns regarding copyright infringement and ethical use. Despite these concerns, methods to safeguard autoregressive text-to-image models remain underexplored. Previous concept erasure methods, primarily designed for diffusion models that operate in denoising latent space, are not directly applicable to autoregressive models that generate images token by token. To address this critical gap, we propose Visual Contrast Exploitation (VCE), a novel framework comprising: (1) an innovative contrastive image pair construction paradigm that precisely decouples unsafe concepts from their associated content semantics, and (2) a sophisticated DPO-based training approach that enhances the model's ability to identify and leverage visual contrastive features from image pairs, enabling precise concept erasure. Our comprehensive experiments across three challenging tasks-artist style erasure, explicit content erasure, and object removal-demonstrate that our method effectively secures the model, achieving state-of-the-art results while erasing unsafe concepts and maintaining the integrity of unrelated safe concepts. The code and models are available at https://github.com/Maplebb/VCE.
Authors Feng Han, Chao Gong, Zhipeng Wei, Jingjing Chen, Yu-Gang Jiang
Categories Computer Vision and Pattern Recognition
0
UniREditBench: A Unified Reasoning-based Image Editing Benchmark
Title CN UniREditBench：一个基于统一推理的图像编辑基准
Keywords 图像编辑 · 推理基准 · 多模态评估 · 合成数据 · 链式思维
Summary 本文提出了UniREditBench，一个用于评估基于推理的图像编辑模型的统一基准，包含2700个样本，覆盖真实世界和游戏世界的多种复杂推理场景。为提高评估可靠性，引入了文本与图像双参考的多模态评估方式，并构建了大规模合成数据集UniREdit-Data-100K，用于训练具备链式思维能力的编辑模型。实验评估了多个开源与闭源模型的表现。
Reason 该论文聚焦于图像编辑任务中的推理能力评估，属于纯图像生成与编辑领域，且其核心是构建游戏和现实场景下的图像编辑基准，涉及多模态生成模型在复杂规则下的表现分析。虽然提到了链式思维（CoT）和多模态评估机制，但整体任务与文档图像理解（DIU）无关，不涉及OCR、布局分析、表格识别或文档视觉问答等核心问题。同时，其方法迁移至文档理解场景的可行性极低，无法解决OCR幻觉、密集文本处理或长文档推理等关键痛点。
Abstract Recent advances in multi-modal generative models have driven substantial improvements in image editing. However, current generative models still struggle with handling diverse and complex image editing tasks that require implicit reasoning, underscoring the need for a comprehensive benchmark to systematically assess their performance across various reasoning scenarios. Existing benchmarks primarily focus on single-object attribute transformation in realistic scenarios, which, while effective, encounter two key challenges: (1) they largely overlook multi-object interactions as well as game-world scenarios that involve human-defined rules, which are common in real-life applications; (2) they only rely on textual references to evaluate the generated images, potentially leading to systematic misjudgments, especially in complex reasoning scenarios. To this end, this work proposes UniREditBench, a unified benchmark for reasoning-based image editing evaluation. It comprises 2,700 meticulously curated samples, covering both real- and game-world scenarios across 8 primary dimensions and 18 sub-dimensions. To improve evaluation reliability, we introduce multimodal dual-reference evaluation, providing both textual and ground-truth image references for each sample assessment. Furthermore, we design an automated multi-scenario data synthesis pipeline and construct UniREdit-Data-100K, a large-scale synthetic dataset with high-quality chain-of-thought (CoT) reasoning annotations. We fine-tune Bagel on this dataset and develop UniREdit-Bagel, demonstrating substantial improvements in both in-domain and out-of-distribution settings. Through thorough benchmarking of both open-source and closed-source image editing models, we reveal their strengths and weaknesses across various aspects.
Authors Feng Han, Yibin Wang, Chenglin Li, Zheming Liang, Dianyi Wang, Yang Jiao, Zhipeng Wei, Chao Gong, Cheng Jin, Jingjing Chen, Jiaqi Wang
Comment Project page: https://maplebb.github.io/UniREditBench
Categories Computer Vision and Pattern Recognition
0
SpecDiff: Accelerating Diffusion Model Inference with Self-Speculation AAAI 2026 Oral
Title CN SpecDiff：利用自推测加速扩散模型推理
Keywords 扩散模型 · 推理加速 · 特征缓存 · 自推测 · 生成模型
Summary 本文提出SpecDiff，一种无需训练的多级特征缓存策略，通过引入基于同时间步不同迭代间信息相似性的自推测机制，结合历史信息进行动态特征选择和分级计算，在Stable Diffusion系列模型上实现了显著加速且质量损失极小。
Reason 该论文研究的是扩散模型推理加速技术，属于图像生成领域，与文档图像理解（DIU）或视觉语言模型在文档任务中的应用无直接关联。尽管其提出的自推测机制和特征缓存策略在推理效率上有创新，但方法针对的是生成式扩散模型的计算冗余问题，难以迁移到DocVQA、OCR-free理解或复杂版面分析等文档理解核心任务。属于无关领域（纯图像生成/修复），命中负面清单第2条。
Abstract Feature caching has recently emerged as a promising method for diffusion model acceleration. It effectively alleviates the inefficiency problem caused by high computational requirements by caching similar features in the inference process of the diffusion model. In this paper, we analyze existing feature caching methods from the perspective of information utilization, and point out that relying solely on historical information will lead to constrained accuracy and speed performance. And we propose a novel paradigm that introduces future information via self-speculation based on the information similarity at the same time step across different iteration times. Based on this paradigm, we present \textit{SpecDiff}, a training-free multi-level feature caching strategy including a cached feature selection algorithm and a multi-level feature classification algorithm. (1) Feature selection algorithm based on self-speculative information. \textit{SpecDiff} determines a dynamic importance score for each token based on self-speculative information and historical information, and performs cached feature selection through the importance score. (2) Multi-level feature classification algorithm based on feature importance scores. \textit{SpecDiff} classifies tokens by leveraging the differences in feature importance scores and introduces a multi-level feature calculation strategy. Extensive experiments show that \textit{SpecDiff} achieves average 2.80 \times, 2.74 \times , and 3.17\times speedup with negligible quality loss in Stable Diffusion 3, 3.5, and FLUX compared to RFlow on NVIDIA A800-80GB GPU. By merging speculative and historical information, \textit{SpecDiff} overcomes the speedup-accuracy trade-off bottleneck, pushing the Pareto frontier of speedup and accuracy in the efficient diffusion model inference.
Authors Jiayi Pan, Jiaming Xu, Yongkang Zhou, Guohao Dai
Comment Accepted by AAAI 2026 Oral
Categories Computer Vision and Pattern Recognition
0
D-FCGS: Feedforward Compression of Dynamic Gaussian Splatting for Free-Viewpoint Videos AAAI-26
Title CN D-FCGS：面向自由视角视频的动态高斯点阵前馈压缩
Keywords 3D高斯点阵 · 自由视角视频 · 前馈压缩 · 运动补偿 · 熵模型
Summary 本文提出D-FCGS，一种用于动态高斯点阵的前馈压缩框架，旨在高效压缩自由视角视频中的3D表示。通过引入标准化的帧组结构、双先验感知熵模型以及控制点引导的运动补偿机制，实现在保持跨视角视觉质量的同时显著提升压缩效率。该方法在多视点视频数据上训练，具备零样本泛化能力，并达到与优化依赖型方法相当的率失真性能。
Reason 该论文研究的是自由视角视频中动态3D高斯点阵的压缩问题，属于3D视觉与视频编码领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务完全无关。尽管涉及视觉表示和压缩技术，但其处理对象为动态3D场景而非文档结构、文本布局或OCR相关问题，无法迁移解决OCR幻觉、密集文本识别、复杂排版解析或长文档推理等DIU核心痛点。
Abstract Free-Viewpoint Video (FVV) enables immersive 3D experiences, but efficient compression of dynamic 3D representation remains a major challenge. Existing dynamic 3D Gaussian Splatting methods couple reconstruction with optimization-dependent compression and customized motion formats, limiting generalization and standardization. To address this, we propose D-FCGS, a novel Feedforward Compression framework for Dynamic Gaussian Splatting. Key innovations include: (1) a standardized Group-of-Frames (GoF) structure with I-P coding, leveraging sparse control points to extract inter-frame motion tensors; (2) a dual prior-aware entropy model that fuses hyperprior and spatial-temporal priors for accurate rate estimation; (3) a control-point-guided motion compensation mechanism and refinement network to enhance view-consistent fidelity. Trained on Gaussian frames derived from multi-view videos, D-FCGS generalizes across diverse scenes in a zero-shot fashion. Experiments show that it matches the rate-distortion performance of optimization-based methods, achieving over 40 times compression compared to the baseline while preserving visual quality across viewpoints. This work advances feedforward compression of dynamic 3DGS, facilitating scalable FVV transmission and storage for immersive applications.
Authors Wenkang Zhang, Yan Zhao, Qiang Wang, Zhixin Xu, Li Song, Zhengxue Cheng
Comment AAAI-26 accepted, code: https://github.com/Mr-Zwkid/D-FCGS
Categories Computer Vision and Pattern Recognition
0
RN-SDEs: Limited-Angle CT Reconstruction with Residual Null-Space Diffusion Stochastic Differential Equations
Title CN RN-SDEs：基于残差零空间扩散随机微分方程的有限角度CT重建
Keywords 扩散模型 · 有限角度CT重建 · SDE · 零空间分解 · 医学图像重建
Summary 本文提出了一种基于残差零空间扩散随机微分方程（RN-SDEs）的方法，用于解决有限角度CT重建中的图像失真问题。通过结合均值回归扩散模型与范围-零空间分解（RNSD）进行数据一致性校正，在ChromSTEM和C4KC-KiTS两个数据集上实现了高质量图像恢复，并在多项指标上达到SOTA。
Reason 该论文研究的是有限角度CT图像重建问题，属于医学成像与计算成像领域，核心方法为扩散模型在断层扫描重建中的应用。尽管使用了先进的SDE和零空间分解技术，但其任务目标、数据模态（3D体素/投影数据）与文档图像理解（DIU）完全无关。无法迁移至解决OCR幻觉、复杂排版理解或DocVQA等核心问题。
Abstract Computed tomography is a widely used imaging modality with applications ranging from medical imaging to material analysis. One major challenge arises from the lack of scanning information at certain angles, resulting in distortion or artifacts in the reconstructed images. This is referred to as the Limited Angle Computed Tomography (LACT) reconstruction problem. To address this problem, we propose the use of Residual Null-Space Diffusion Stochastic Differential Equations (RN-SDEs), which are a variant of diffusion models that characterize the diffusion process with mean-reverting (MR) stochastic differential equations. To demonstrate the generalizability of RN-SDEs, we conducted experiments with two different LACT datasets, ChromSTEM and C4KC-KiTS. Through extensive experiments, we demonstrate that by leveraging learned MR-SDEs as a prior and emphasizing data consistency using Range-Null Space Decomposition (RNSD) based rectification, we can recover high-quality images from severely degraded ones and achieve state-of-the-art performance in most LACT tasks. Additionally, we present a quantitative comparison of RN-SDE with other networks, in terms of computational complexity and runtime efficiency, highlighting the superior effectiveness of our proposed approach.
Authors Jiaqi Guo, Santiago Lopez-Tapia, Wing Shun Li, Yunan Wu, Marcelo Carignano, Martin Kröger, Vinayak P. Dravid, Igal Szleifer, Vadim Backman, Aggelos K. Katsaggelos
Categories Computer Vision and Pattern Recognition
0
UniRSCD: A Unified Novel Architectural Paradigm for Remote Sensing Change Detection
Title CN UniRSCD：一种用于遥感变化检测的统一新型架构范式
Keywords 遥感变化检测 · 统一架构 · 状态空间模型 · 多任务学习 · 频率提示生成
Summary 本文提出了一种名为UniRSCD的统一框架，用于处理不同粒度的遥感图像变化检测任务（如二值变化检测、语义变化检测和建筑物损伤评估）。该方法基于状态空间模型，引入频率变化提示生成器作为统一编码器，并通过分层特征交互实现多任务共享表示。在多个遥感数据集上取得了领先性能。
Reason 该论文研究领域为遥感图像变化检测（Remote Sensing Change Detection），属于特定垂直领域的应用任务，与文档图像理解（DIU）的核心问题如OCR幻觉、复杂排版、文本识别与推理等无直接关联。尽管其提出了一种统一架构设计，但方法面向的是遥感影像中的像素级变化分析，无法迁移解决文档图像中的细粒度文本定位、多模态对齐或长文档逻辑推理等关键痛点。
Abstract In recent years, remote sensing change detection has garnered significant attention due to its critical role in resource monitoring and disaster assessment. Change detection tasks exist with different output granularities such as BCD, SCD, and BDA. However, existing methods require substantial expert knowledge to design specialized decoders that compensate for information loss during encoding across different tasks. This not only introduces uncertainty into the process of selecting optimal models for abrupt change scenarios (such as disaster outbreaks) but also limits the universality of these architectures. To address these challenges, this paper proposes a unified, general change detection framework named UniRSCD. Building upon a state space model backbone, we introduce a frequency change prompt generator as a unified encoder. The encoder dynamically scans bitemporal global context information while integrating high-frequency details with low-frequency holistic information, thereby eliminating the need for specialized decoders for feature compensation. Subsequently, the unified decoder and prediction head establish a shared representation space through hierarchical feature interaction and task-adaptive output mapping. This integrating various tasks such as binary change detection and semantic change detection into a unified architecture, thereby accommodating the differing output granularity requirements of distinct change detection tasks. Experimental results demonstrate that the proposed architecture can adapt to multiple change detection tasks and achieves leading performance on five datasets, including the binary change dataset LEVIR-CD, the semantic change dataset SECOND, and the building damage assessment dataset xBD.
Authors Yuan Qu, Zhipeng Zhang, Chaojun Xu, Qiao Wan, Mengying Xie, Yuzeng Chen, Zhenqi Liu, Yanfei Zhong
Categories Computer Vision and Pattern Recognition
0
Switch-JustDance: Benchmarking Whole Body Motion Tracking Policies Using a Commercial Console Game
Title CN Switch-JustDance：使用商用游戏机游戏评估全身运动跟踪策略
Keywords 机器人控制 · 全身动作 · 基准测试 · 动作迁移 · 具身AI
Summary 本文提出了一种基于任天堂Switch游戏《舞力全开》的机器人全身动作控制评测框架，通过将游戏中的舞蹈动作转化为机器人可执行的指令，并利用游戏内置评分系统评估控制器性能。研究聚焦于机器人控制与具身智能的基准测试，依赖硬件实验与动作重定向技术。
Reason 该论文属于具身智能与机器人控制领域，研究如何利用《舞力全开》游戏对人形机器人进行动作控制评估。其核心任务、方法与文档图像理解（DIU）完全无关，不涉及任何OCR、VLM、多模态文档理解或视觉语言模型在文档上的应用。无法迁移至解决DIU中的OCR幻觉、复杂排版、长文档推理等核心痛点。
Abstract Recent advances in whole-body robot control have enabled humanoid and legged robots to perform increasingly agile and coordinated motions. However, standardized benchmarks for evaluating these capabilities in real-world settings, and in direct comparison to humans, remain scarce. Existing evaluations often rely on pre-collected human motion datasets or simulation-based experiments, which limit reproducibility, overlook hardware factors, and hinder fair human-robot comparisons. We present Switch-JustDance, a low-cost and reproducible benchmarking pipeline that leverages motion-sensing console games, Just Dance on the Nintendo Switch, to evaluate robot whole-body control. Using Just Dance on the Nintendo Switch as a representative platform, Switch-JustDance converts in-game choreography into robot-executable motions through streaming, motion reconstruction, and motion retargeting modules and enables users to evaluate controller performance through the game's built-in scoring system. We first validate the evaluation properties of Just Dance, analyzing its reliability, validity, sensitivity, and potential sources of bias. Our results show that the platform provides consistent and interpretable performance measures, making it a suitable tool for benchmarking embodied AI. Building on this foundation, we benchmark three state-of-the-art humanoid whole-body controllers on hardware and provide insights into their relative strengths and limitations.
Authors Jeonghwan Kim, Wontaek Kim, Yidan Lu, Jin Cheng, Fatemeh Zargarbashi, Zicheng Zeng, Zekun Qi, Zhiyang Dou, Nitish Sontakke, Donghoon Baek, Sehoon Ha, Tianyu Li
Categories Computer Vision and Pattern Recognition
0
Frequency-Adaptive Sharpness Regularization for Improving 3D Gaussian Splatting Generalization
Title CN 用于提升3D高斯点阵泛化能力的频率自适应锐度正则化
Keywords 3D Gaussian Splatting · Sharpness Regularization · Generalization · Novel View Synthesis · Frequency-Adaptive
Summary 本文从机器学习角度重新审视3D高斯点阵（3DGS）的优化过程，将新视角合成视为一种泛化问题，提出了频率自适应锐度正则化（FASR）方法。该方法根据图像局部频率动态调整正则化强度和邻域范围，解决了传统锐度感知最小化（SAM）在3DGS中过度平滑高频细节或正则化不足的问题，有效减少了新视角下的漂浮伪影并保留了精细结构。
Reason 该论文研究的是3D高斯点阵（3D Gaussian Splatting）在新视角合成中的泛化问题，属于3D视觉与图形学领域，与文档图像理解（DIU）或视觉-语言模型（VLM）无关。其提出的方法FASR聚焦于视角泛化和高频细节重建，无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract Despite 3D Gaussian Splatting (3DGS) excelling in most configurations, it lacks generalization across novel viewpoints in a few-shot scenario because it overfits to the sparse observations. We revisit 3DGS optimization from a machine learning perspective, framing novel view synthesis as a generalization problem to unseen viewpoints-an underexplored direction. We propose Frequency-Adaptive Sharpness Regularization (FASR), which reformulates the 3DGS training objective, thereby guiding 3DGS to converge toward a better generalization solution. Although Sharpness-Aware Minimization (SAM) similarly reduces the sharpness of the loss landscape to improve generalization of classification models, directly employing it to 3DGS is suboptimal due to the discrepancy between the tasks. Specifically, it hinders reconstructing high-frequency details due to excessive regularization, while reducing its strength leads to under-penalizing sharpness. To address this, we reflect the local frequency of images to set the regularization weight and the neighborhood radius when estimating the local sharpness. It prevents floater artifacts in novel viewpoints and reconstructs fine details that SAM tends to oversmooth. Across datasets with various configurations, our method consistently improves a wide range of baselines. Code will be available at https://bbangsik13.github.io/FASR.
Authors Youngsik Yun, Dongjun Gu, Youngjung Uh
Comment Project page: https://bbangsik13.github.io/FASR
Categories Computer Vision and Pattern Recognition
0
MultiCrafter: High-Fidelity Multi-Subject Generation via Disentangled Attention and Identity-Aware Preference Alignment
Title CN MultiCrafter：通过解耦注意力与身份感知偏好对齐实现高保真多主体生成
Keywords 多主体生成 · 图像生成 · 注意力机制 · 身份保真 · 强化学习
Summary 本文提出MultiCrafter，一种用于多主体图像生成的框架，通过解耦训练过程为两个阶段：预训练阶段引入显式位置监督以缓解注意力泄露，提升主体保真度；后训练阶段采用基于匈牙利匹配算法的在线强化学习机制，在保持身份的同时优化美学和提示一致性。实验表明该方法在主体保真和偏好对齐方面表现优异。
Reason 该论文研究的是多主体图像生成任务，核心关注点为生成图像中的主体保真度与人类审美偏好对齐，属于纯图像生成领域。其方法虽在视觉-语言建模中有一定创新，但主要应用于内容创作类生成任务，与文档图像理解（DIU）中的OCR、布局分析、表格识别、VIE等核心任务无直接关联。无法迁移解决DIU中的OCR幻觉、密集文本处理或复杂排版理解等痛点。
Abstract Multi-subject image generation aims to synthesize user-provided subjects in a single image while preserving subject fidelity, ensuring prompt consistency, and aligning with human aesthetic preferences. Existing In-Context-Learning based methods are limited by their highly coupled training paradigm. These methods attempt to achieve both high subject fidelity and multi-dimensional human preference alignment within a single training stage, relying on a single, indirect reconstruction loss, which is difficult to simultaneously satisfy both these goals. To address this, we propose MultiCrafter, a framework that decouples this task into two distinct training stages. First, in a pre-training stage, we introduce an explicit positional supervision mechanism that effectively resolves attention bleeding and drastically enhances subject fidelity. Second, in a post-training stage, we propose Identity-Preserving Preference Optimization, a novel online reinforcement learning framework. We feature a scoring mechanism to accurately assess multi-subject fidelity based on the Hungarian matching algorithm, which allows the model to optimize for aesthetics and prompt alignment while ensuring subject fidelity achieved in the first stage. Experiments validate that our decoupling framework significantly improves subject fidelity while aligning with human preferences better.
Authors Tao Wu, Yibo Jiang, Yehao Lu, Zhizhong Wang, Zeyi Huang, Zequn Qin, Xi Li
Comment Project Page: https://wutao-cs.github.io/MultiCrafter/
Categories Computer Vision and Pattern Recognition
0
CUS-GS: A Compact Unified Structured Gaussian Splatting Framework for Multimodal Scene Representation
Title CN CUS-GS：一种用于多模态场景表示的紧凑型统一结构化高斯点阵框架
Keywords 3D场景表示 · 高斯点阵 · 多模态融合 · 结构化表示 · 参数压缩
Summary 本文提出了一种名为CUS-GS的紧凑型统一结构化高斯点阵框架，用于实现多模态语义特征与结构化3D几何之间的融合。通过体素化锚点结构构建空间骨架，并从多个基础模型（如CLIP、DINOv2）中提取多模态语义特征，再利用特征感知的重要性评估策略动态增删锚点，在仅使用6M参数的情况下实现了优于现有方法的性能，显著提升了效率与表达能力之间的平衡。
Reason 该论文属于3D视觉与场景重建领域，核心研究内容为高斯点阵（Gaussian Splatting）在三维场景中的结构化表示与多模态语义融合，涉及CLIP、DINOv2等模型的特征对齐与稀疏化建模。尽管其提出了高效的参数压缩与多模态特征整合机制，但整体工作聚焦于具身智能或自动驾驶相关的3D环境感知，与文档图像理解（DIU）任务无直接关联。文档图像理解关注的是二维平面中文本布局、OCR去幻觉、表格识别等语义结构解析问题，而该方法无法迁移至此类任务。
Abstract Recent advances in Gaussian Splatting based 3D scene representation have shown two major trends: semantics-oriented approaches that focus on high-level understanding but lack explicit 3D geometry modeling, and structure-oriented approaches that capture spatial structures yet provide limited semantic abstraction. To bridge this gap, we present CUS-GS, a compact unified structured Gaussian Splatting representation, which connects multimodal semantic features with structured 3D geometry. Specifically, we design a voxelized anchor structure that constructs a spatial scaffold, while extracting multimodal semantic features from a set of foundation models (e.g., CLIP, DINOv2, SEEM). Moreover, we introduce a multimodal latent feature allocation mechanism to unify appearance, geometry, and semantics across heterogeneous feature spaces, ensuring a consistent representation across multiple foundation models. Finally, we propose a feature-aware significance evaluation strategy to dynamically guide anchor growing and pruning, effectively removing redundant or invalid anchors while maintaining semantic integrity. Extensive experiments show that CUS-GS achieves competitive performance compared to state-of-the-art methods using as few as 6M parameters - an order of magnitude smaller than the closest rival at 35M - highlighting the excellent trade off between performance and model efficiency of the proposed framework.
Authors Yuhang Ming, Chenxin Fang, Xingyuan Yu, Fan Zhang, Weichen Dai, Wanzeng Kong, Guofeng Zhang
Comment 15 pages, 8 figures, 4 tables
Categories Computer Vision and Pattern Recognition
0
Spectral Super-Resolution Neural Operator with Atmospheric Radiative Transfer Prior
Title CN 基于大气辐射传输先验的光谱超分辨率神经算子
Keywords 光谱超分辨率 · 神经算子 · 大气辐射传输 · 遥感 · 物理先验
Summary 本文提出了一种结合大气辐射传输先验的光谱超分辨率神经算子（SSRNO），通过三阶段框架（上采样、重建、优化）实现从多光谱到高光谱图像的物理一致重建。引入引导矩阵投影（GMP）方法和频谱感知卷积结构，增强物理合理性并支持零样本外推。实验表明其在遥感图像重建中具有优越性能。
Reason 该论文研究的是遥感领域的光谱超分辨率（SSR）问题，目标是从多光谱图像重建高光谱图像，属于遥感成像范畴。尽管其方法引入了物理先验（大气辐射传输模型）以提升预测的物理一致性，但其任务与文档图像理解（DIU）如DocVQA、布局分析、表格识别等无直接关联。所提出的技术（如GMP、SAC层）针对的是光谱维度的连续映射和大气畸变校正，难以迁移到文档图像中的OCR幻觉、复杂排版或细粒度文本定位等核心痛点。不属于DIU本题，亦无明确可迁移的方法论工具。
Abstract Spectral super-resolution (SSR) aims to reconstruct hyperspectral images (HSIs) from multispectral observations, with broad applications in remote sensing. Data-driven methods are widely used, but they often overlook physical principles, leading to unrealistic spectra, particularly in atmosphere-affected bands. To address this challenge, we propose the Spectral Super-Resolution Neural Operator (SSRNO), which incorporates atmospheric radiative transfer (ART) prior into the data-driven procedure, yielding more physically consistent predictions. The proposed SSRNO framework consists of three stages: upsampling, reconstruction, and refinement. In the upsampling stage, we leverage prior information to expand the input multispectral image, producing a physically plausible hyperspectral estimate. Subsequently, we utilize a neural operator in the reconstruction stage to learn a continuous mapping across the spectral domain. Finally, the refinement stage imposes a hard constraint on the output HSI to eliminate color distortion. The upsampling and refinement stages are implemented via the proposed guidance matrix projection (GMP) method, and the reconstruction neural operator adopts U-shaped spectral-aware convolution (SAC) layers to capture multi-scale features. Moreover, we theoretically demonstrate the optimality of the GMP method. With the neural operator and ART priors, SSRNO also achieves continuous spectral reconstruction and zero-shot extrapolation. Various experiments validate the effectiveness and generalization ability of the proposed approach.
Authors Ziye Zhang, Bin Pan, Zhenwei Shi
Categories Computer Vision and Pattern Recognition
0
MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robots
Title CN MobileVLA-R1：为移动机器人强化视觉-语言-动作系统
Keywords 视觉语言动作 · 具身智能 · 链式思维 · 强化学习 · 机器人控制
Summary 本文提出了MobileVLA-R1，一个用于四足机器人的统一视觉-语言-动作框架，通过构建多粒度思维链（CoT）数据集MobileVLA-CoT，并结合监督式CoT对齐与GRPO强化学习进行两阶段训练，以提升语义推理与动作控制的一致性。在视觉语言导航（VLN）和视觉语言动作（VLA）任务上表现优于基线模型，并在真实机器人部署中验证了鲁棒性。
Reason 该论文研究的是具身智能中视觉-语言-动作在四足机器人上的应用，属于机器人控制与导航领域。尽管使用了CoT和强化学习等方法，但其核心任务是将自然语言指令映射到连续动作控制，与文档图像理解（DIU）或DocVQA无关。所提出的方法难以迁移到解决OCR幻觉、复杂排版理解或长文档推理等文档理解痛点。
Abstract Grounding natural-language instructions into continuous control for quadruped robots remains a fundamental challenge in vision language action. Existing methods struggle to bridge high-level semantic reasoning and low-level actuation, leading to unstable grounding and weak generalization in the real world. To address these issues, we present MobileVLA-R1, a unified vision-language-action framework that enables explicit reasoning and continuous control for quadruped robots. We construct MobileVLA-CoT, a large-scale dataset of multi-granularity chain-of-thought (CoT) for embodied trajectories, providing structured reasoning supervision for alignment. Built upon this foundation, we introduce a two-stage training paradigm that combines supervised CoT alignment with GRPO reinforcement learning to enhance reasoning consistency, control stability, and long-horizon execution. Extensive evaluations on VLN and VLA tasks demonstrate superior performance over strong baselines, with approximately a 5% improvement. Real-world deployment on a quadruped robot validates robust performance in complex environments. Code: https://github.com/AIGeeksGroup/MobileVLA-R1. Website: https://aigeeksgroup.github.io/MobileVLA-R1.
Authors Ting Huang, Dongjian Li, Rui Yang, Zeyu Zhang, Zida Yang, Hao Tang
Categories Computer Vision and Pattern Recognition
0
MINDiff: Mask-Integrated Negative Attention for Controlling Overfitting in Text-to-Image Personalization ICCV 2025 Personalization in Generative AI Workshop
Title CN MINDiff：用于控制文本到图像个性化中过拟合的掩码集成负注意力
Keywords 文本到图像生成 · 个性化生成 · 过拟合抑制 · 负注意力 · 推理时控制
Summary 本文提出MINDiff，一种在推理阶段通过引入‘负注意力’机制来缓解文本到图像模型在个性化过程中过拟合的方法。该方法通过修改交叉注意力，在掩码标记的无关区域抑制主体特征，从而提升文本-图像对齐，并允许用户通过调节参数λ平衡主体保真度与文本一致性。实验表明其在DreamBooth模型上优于传统的先验保留损失，且无需重新训练模型。
Reason 该论文研究的是文本到图像生成模型中的个性化主体过拟合问题，属于纯图像生成领域。其核心方法‘负注意力’用于抑制生成图像中无关区域的主体影响，提升文本对齐，但应用场景与文档图像理解（DIU）无直接关联。DIU的核心痛点如OCR幻觉、密集文本识别、复杂版式分析、长文档推理等均未被涉及。该工作无法迁移用于解决文档理解任务，且属于典型的无关领域（图像生成）。
Abstract In the personalization process of large-scale text-to-image models, overfitting often occurs when learning specific subject from a limited number of images. Existing methods, such as DreamBooth, mitigate this issue through a class-specific prior-preservation loss, which requires increased computational cost during training and limits user control during inference time. To address these limitations, we propose Mask-Integrated Negative Attention Diffusion (MINDiff). MINDiff introduces a novel concept, negative attention, which suppresses the subject's influence in masked irrelevant regions. We achieve this by modifying the cross-attention mechanism during inference. This enables semantic control and improves text alignment by reducing subject dominance in irrelevant regions. Additionally, during the inference time, users can adjust a scale parameter lambda to balance subject fidelity and text alignment. Our qualitative and quantitative experiments on DreamBooth models demonstrate that MINDiff mitigates overfitting more effectively than class-specific prior-preservation loss. As our method operates entirely at inference time and does not alter the model architecture, it can be directly applied to existing DreamBooth models without re-training. Our code is available at https://github.com/seuleepy/MINDiff.
Authors Seulgi Jeong, Jaeil Kim
Comment Accepted at ICCV 2025 Personalization in Generative AI Workshop
Categories Computer Vision and Pattern Recognition
0
ArticFlow: Generative Simulation of Articulated Mechanisms
Title CN ArticFlow：铰接机制的生成模拟
Keywords 3D生成 · 流匹配 · 铰接结构 · 生成模型 · 神经仿真
Summary 本文提出ArticFlow，一种用于生成可动3D结构的两阶段流匹配框架，能够根据动作控制生成具有合理运动学特性的铰接机制。该方法在MuJoCo数据集上实现了高质量的形态生成与动力学预测，但应用场景局限于3D物理仿真与生成建模。
Reason 该论文研究的是3D铰接结构的生成与运动模拟，属于3D视觉与生成模型领域，聚焦于动态机械结构的动作控制和形态生成。其核心问题与文档图像理解（DIU）无直接关联，既不涉及文本识别、布局分析、表格理解等DIU任务，也无法迁移用于解决OCR幻觉、复杂排版或长文档推理等关键痛点。
Abstract Recent advances in generative models have produced strong results for static 3D shapes, whereas articulated 3D generation remains challenging due to action-dependent deformations and limited datasets. We introduce ArticFlow, a two-stage flow matching framework that learns a controllable velocity field from noise to target point sets under explicit action control. ArticFlow couples (i) a latent flow that transports noise to a shape-prior code and (ii) a point flow that transports points conditioned on the action and the shape prior, enabling a single model to represent diverse articulated categories and generalize across actions. On MuJoCo Menagerie, ArticFlow functions both as a generative model and as a neural simulator: it predicts action-conditioned kinematics from a compact prior and synthesizes novel morphologies via latent interpolation. Compared with object-specific simulators and an action-conditioned variant of static point-cloud generators, ArticFlow achieves higher kinematic accuracy and better shape quality. Results show that action-conditioned flow matching is a practical route to controllable and high-quality articulated mechanism generation.
Authors Jiong Lin, Jinchen Ruan, Hod Lipson
Comment 8 pages, 8 figures
Categories Computer Vision and Pattern Recognition
0
InvAD: Inversion-based Reconstruction-Free Anomaly Detection with Diffusion Models
Title CN 基于扩散模型的无需重建的反转式异常检测
Keywords 异常检测 · 扩散模型 · DDIM反转 · 潜在空间 · 无需重建
Summary 本文提出InvAD，一种基于DDIM反转的无需重建的异常检测方法，通过在潜在空间中加噪而非在RGB空间中去噪来实现高效且准确的异常评分。该方法避免了多步去噪过程，提升了推理速度，并在多个工业与医学图像数据集上实现了SOTA性能。
Reason 该论文属于图像异常检测领域，尽管使用了扩散模型和视觉表示学习技术，但其核心任务与文档图像理解（DIU）无关。方法聚焦于工业与医疗图像中的异常检测，未涉及OCR、布局分析、表格识别或任何文档特有的结构与语义理解问题。无法迁移用于解决DIU中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Despite the remarkable success, recent reconstruction-based anomaly detection (AD) methods via diffusion modeling still involve fine-grained noise-strength tuning and computationally expensive multi-step denoising, leading to a fundamental tension between fidelity and efficiency. In this paper, we propose InvAD, a novel inversion-based anomaly detection approach ("detection via noising in latent space") that circumvents explicit reconstruction. Importantly, we contend that the limitations in prior reconstruction-based methods originate from the prevailing "detection via denoising in RGB space" paradigm. To address this, we model AD under a reconstruction-free formulation, which directly infers the final latent variable corresponding to the input image via DDIM inversion, and then measures the deviation based on the known prior distribution for anomaly scoring. Specifically, in approximating the original probability flow ODE using the Euler method, we enforce only a few inversion steps to noise the clean image to pursue inference efficiency. As the added noise is adaptively derived with the learned diffusion model, the original features for the clean testing image can still be leveraged to yield high detection accuracy. We perform extensive experiments and detailed analyses across four widely used industrial and medical AD benchmarks under the unsupervised unified setting to demonstrate the effectiveness of our model, achieving state-of-the-art AD performance and approximately 2x inference-time speedup without diffusion distillation.
Authors Shunsuke Sakai, Xiangteng He, Chunzhi Gu, Leonid Sigal, Tatsuhito Hasegawa
Comment Code is available at https://github.com/SkyShunsuke/InversionAD
Categories Computer Vision and Pattern Recognition
0
Learning to Upscale 3D Segmentations in Neuroimaging
Title CN 学习在神经影像中上采样3D分割
Keywords 3D分割 · 上采样 · 神经影像 · 高分辨率MRI · 符号距离图
Summary 本文提出一种通过回归符号距离图来从粗粒度标注生成高分辨率3D分割的方法，主要用于神经影像中的脑部MRI数据上采样。该方法逐类预测并利用合成数据增强泛化能力，在超高分辨率脑成像中实现了对低分辨率分割结果的有效提升。
Reason 该论文研究的是3D医学图像（神经影像）中的分割上采样问题，属于纯医学图像处理领域，与文档图像理解（DIU）任务如DocVQA、表格识别、布局分析等无直接关联。尽管其处理高分辨率数据的思路看似有一定通用性，但方法面向的是3D体素数据和解剖结构，无法迁移到文档图像的文本语义理解、OCR去幻觉或复杂排版解析等核心痛点。属于无关领域。
Abstract Obtaining high-resolution (HR) segmentations from coarse annotations is a pervasive challenge in computer vision. Applications include inferring pixel-level segmentations from token-level labels in vision transformers, upsampling coarse masks to full resolution, and transferring annotations from legacy low-resolution (LR) datasets to modern HR imagery. These challenges are especially acute in 3D neuroimaging, where manual labeling is costly and resolutions continually increase. We propose a scalable framework that generalizes across resolutions and domains by regressing signed distance maps, enabling smooth, boundary-aware supervision. Crucially, our model predicts one class at a time, which substantially reduces memory usage during training and inference (critical for large 3D volumes) and naturally supports generalization to unseen classes. Generalization is further improved through training on synthetic, domain-randomized data. We validate our approach on ultra-high-resolution (UHR) human brain MRI (~100 μm), where most existing methods operate at 1 mm resolution. Our framework effectively upsamples such standard-resolution segmentations to UHR detail. Results on synthetic and real data demonstrate superior scalability and generalization compared to conventional segmentation methods. Code is available at: https://github.com/HuXiaoling/Learn2Upscale.
Authors Xiaoling Hu, Peirong Liu, Dina Zemlyanker, Jonathan Williams Ramirez, Oula Puonti, Juan Eugenio Iglesias
Comment 13 pages, 4 figures
Categories Computer Vision and Pattern Recognition
0
Intraoperative 2D/3D Registration via Spherical Similarity Learning and Differentiable Levenberg-Marquardt Optimization WACV 2026
Title CN 通过球面相似性学习和可微分Levenberg-Marquardt优化实现术中2D/3D配准
Keywords 医学图像配准 · 2D/3D配准 · 流形学习 · 可微分优化 · WACV
Summary 本文提出一种用于术中2D/3D医学图像配准的新方法，通过在球面特征空间中进行相似性学习，并利用双不变SO(4)空间中的黎曼距离逼近测地距离，以更好地建模姿态流形结构。结合CNN-Transformer编码器提取特征，并采用可微分Levenberg-Marquardt算法加速优化收敛。实验表明该方法在患者特异性和通用场景下均具有更高的配准精度。
Reason 该论文属于医学图像配准领域，核心任务是术中2D放射图像与术前3D体积数据的对齐，涉及SE(3)流形上的相似性学习和优化算法改进。其研究目标、数据模态和应用场景均与文档图像理解（DIU）无关，不涉及文本识别、布局分析、VQA或OCR-free等任何DIU核心问题。尽管使用了可微分优化和非欧几里得表示学习，但这些方法针对的是刚体变换估计，难以迁移到文档理解中的幻觉抑制、细粒度定位或复杂排版解析等痛点。
Abstract Intraoperative 2D/3D registration aligns preoperative 3D volumes with real-time 2D radiographs, enabling accurate localization of instruments and implants. A recent fully differentiable similarity learning framework approximates geodesic distances on SE(3), expanding the capture range of registration and mitigating the effects of substantial disturbances, but existing Euclidean approximations distort manifold structure and slow convergence. To address these limitations, we explore similarity learning in non-Euclidean spherical feature spaces to better capture and fit complex manifold structure. We extract feature embeddings using a CNN-Transformer encoder, project them into spherical space, and approximate their geodesic distances with Riemannian distances in the bi-invariant SO(4) space. This enables a more expressive and geometrically consistent deep similarity metric, enhancing the ability to distinguish subtle pose differences. During inference, we replace gradient descent with fully differentiable Levenberg-Marquardt optimization to accelerate convergence. Experiments on real and synthetic datasets show superior accuracy in both patient-specific and patient-agnostic scenarios.
Authors Minheng Chen, Youyong Kong
Comment WACV 2026 Accepted
Categories Computer Vision and Pattern Recognition
0
TyphoFormer: Language-Augmented Transformer for Accurate Typhoon Track Forecasting ACM SIGSPATIAL 2025
Title CN TyphoFormer：用于精确台风路径预测的语言增强型Transformer
Keywords Typhoon Forecasting · Transformer · Language-Augmented Model · Time Series Prediction · Multimodal Learning
Summary 本文提出TyphoFormer，一种结合自然语言描述与数值时间序列的Transformer模型，用于提升台风路径预测精度。通过LLM生成每个时间步的气象语义描述，并将其作为特殊token输入统一的Transformer编码器，从而增强对稀疏轨迹的建模能力。实验表明该方法在非线性路径和历史数据有限的情况下表现优异。
Reason 该论文属于气象预测领域，尽管使用了LLM生成文本描述并融合到时序预测中，但其核心任务与文档图像理解（DIU）无关。未涉及OCR、布局分析、表格识别、视觉文档表示或VLM在文档上的应用。虽然引入语言辅助信息的思想有一定启发性，但迁移到DIU场景（如消除OCR幻觉或复杂排版理解）的路径不明确，且属于典型的‘平行应用’范式——将多模态思想应用于特定垂直领域（气象），而非解决文档理解中的共性技术难题。
Abstract Accurate typhoon track forecasting is crucial for early system warning and disaster response. While Transformer-based models have demonstrated strong performance in modeling the temporal dynamics of dense trajectories of humans and vehicles in smart cities, they usually lack access to broader contextual knowledge that enhances the forecasting reliability of sparse meteorological trajectories, such as typhoon tracks. To address this challenge, we propose TyphoFormer, a novel framework that incorporates natural language descriptions as auxiliary prompts to improve typhoon trajectory forecasting. For each time step, we use Large Language Model (LLM) to generate concise textual descriptions based on the numerical attributes recorded in the North Atlantic hurricane database. The language descriptions capture high-level meteorological semantics and are embedded as auxiliary special tokens prepended to the numerical time series input. By integrating both textual and sequential information within a unified Transformer encoder, TyphoFormer enables the model to leverage contextual cues that are otherwise inaccessible through numerical features alone. Extensive experiments are conducted on HURDAT2 benchmark, results show that TyphoFormer consistently outperforms other state-of-the-art baseline methods, particularly under challenging scenarios involving nonlinear path shifts and limited historical observations.
Authors Lincan Li, Eren Erman Ozguven, Yue Zhao, Guang Wang, Yiqun Xie, Yushun Dong
Comment Accepted by ACM SIGSPATIAL 2025. Received SIGSPATIAL '25 Best Short Paper Award
Categories Computation and Language
0
MGen: Millions of Naturally Occurring Generics in Context SCiL 2025
Title CN MGen：上下文中的数百万自然泛指表达
Keywords 泛指句 · 数据集 · 自然语言处理 · 语义分析 · 计算语言学
Summary 本文提出了一个包含超过400万个自然出现的泛指和量化句子的数据集MGen，这些句子来自多样化的文本来源，并附有长篇上下文文档（如网站和学术论文），涵盖11种不同的量化词。研究发现泛指句通常较长（平均超过16个词），常用于对人群进行概括性陈述。MGen是目前最大且最多样化的泛指句数据集，旨在推动关于泛指性的大规模计算研究。
Reason 该论文聚焦于自然语言中泛指句（generic sentences）的数据集构建，属于纯文本语言学研究范畴，未涉及任何视觉、文档结构或图文多模态理解内容。与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联，也无法迁移用于解决OCR幻觉、细粒度定位或复杂排版等关键问题。
Abstract MGen is a dataset of over 4 million naturally occurring generic and quantified sentences extracted from diverse textual sources. Sentences in the dataset have long context documents, corresponding to websites and academic papers, and cover 11 different quantifiers. We analyze the features of generics sentences in the dataset, with interesting insights: generics can be long sentences (averaging over 16 words) and speakers often use them to express generalisations about people. MGen is the biggest and most diverse dataset of naturally occurring generic sentences, opening the door to large-scale computational research on genericity. It is publicly available at https://gustavocilleruelo.com/mgen
Authors Gustavo Cilleruelo, Emily Allaway, Barry Haddow, Alexandra Birch
Comment Presented at SCiL 2025
Categories Computation and Language
0
GeeSanBhava: Sentiment Tagged Sinhala Music Video Comment Data Set
Title CN GeeSanBhava：标注情感的僧伽罗语音乐视频评论数据集
Keywords 情感分析 · 小语种 · 数据集 · 僧伽罗语 · 自然语言处理
Summary 本文提出了一个名为GeeSanBhava的僧伽罗语YouTube音乐视频评论情感标注数据集，采用Russell的情感效价-唤醒模型进行人工标注，并报告了零样本和微调下的基线结果。研究重点在于小语种情感分析和用户生成内容的偏置问题。
Reason 该论文聚焦于小语种（僧伽罗语）的情感分析数据集构建，属于特定语言的NLP应用，且领域为音乐评论情感识别，与文档图像理解（DIU）、多模态大模型在文档理解中的应用完全无关。命中负面清单第3条：小语种。无任何可迁移至DIU的技术创新。
Abstract This study introduce GeeSanBhava, a high-quality data set of Sinhala song comments extracted from YouTube manually tagged using Russells Valence-Arousal model by three independent human annotators. The human annotators achieve a substantial inter-annotator agreement (Fleiss kappa = 84.96%). The analysis revealed distinct emotional profiles for different songs, highlighting the importance of comment based emotion mapping. The study also addressed the challenges of comparing comment-based and song-based emotions, mitigating biases inherent in user-generated content. A number of Machine learning and deep learning models were pre-trained on a related large data set of Sinhala News comments in order to report the zero-shot result of our Sinhala YouTube comment data set. An optimized Multi-Layer Perceptron model, after extensive hyperparameter tuning, achieved a ROC-AUC score of 0.887. The model is a three-layer MLP with a configuration of 256, 128, and 64 neurons. This research contributes a valuable annotated dataset and provides insights for future work in Sinhala Natural Language Processing and music emotion recognition.
Authors Yomal De Mel, Nisansa de Silva
Categories Computation and Language
0
Comparing Labeled Markov Chains: A Cantor-Kantorovich Approach
Title CN 比较标记马尔可夫链：一种康托-坎托罗维奇方法
Keywords 标记马尔可夫链 · CK距离 · #P-hard · 总变差距离 · 模型比较
Summary 本文提出了基于康托-坎托罗维奇（CK）距离的方法来比较标记马尔可夫链（LMC），将其建模为带折扣的有限视界总变差距离之和，并分析了其计算复杂性、连续性和近似性质。研究表明CK距离的精确计算是#P难的，并给出了可计算的近似方案及其理论边界。
Reason 该论文研究的是标记马尔可夫链之间的距离度量，属于理论概率模型与形式化方法范畴，与文档图像理解（DIU）、多模态大模型（VLM）或视觉语言任务无直接关联。未涉及OCR、文档布局分析、表格识别、视觉表示学习或任何DIU核心问题，也无法迁移用于解决OCR幻觉、细粒度定位或长文档推理等痛点。
Abstract Labeled Markov Chains (or LMCs for short) are useful mathematical objects to model complex probabilistic languages. A central challenge is to compare two LMCs, for example to assess the accuracy of an abstraction or to quantify the effect of model perturbations. In this work, we study the recently introduced Cantor-Kantorovich (or CK) distance. In particular we show that the latter can be framed as a discounted sum of finite-horizon Total Variation distances, making it an instance of discounted linear distance, but arising from the natural Cantor topology. Building on the latter observation, we analyze the properties of the CK distance along three dimensions: computational complexity, continuity properties and approximation. More precisely, we show that the exact computation of the CK distance is #P-hard. We also provide an upper bound on the CK distance as a function of the approximation relation between the two LMCs, and show that a bounded CK distance implies a bounded error between probabilities of finite-horizon traces. Finally, we provide a computable approximation scheme, and show that the latter is also #P-hard. Altogether, our results provide a rigorous theoretical foundation for the CK distance and clarify its relationship with existing distances.
Authors Adrien Banse, Alessandro Abate, Raphaël M. Jungers
Categories Computation and Language
0
Blu-WERP (Web Extraction and Refinement Pipeline): A Scalable Pipeline for Preprocessing Large Language Model Datasets
Title CN Blu-WERP（网络提取与优化管道）：一种用于预处理大语言模型数据集的可扩展管道
Keywords 数据预处理 · 大语言模型 · 文本清洗 · Common Crawl · 训练数据质量
Summary 本文提出了一种名为Blu-WERP的数据预处理管道，用于从Common Crawl的WARC文件中高效提取和优化高质量文本数据，以提升大语言模型的训练效果。通过先进的过滤和质量评估机制，该方法在多个模型规模和基准测试上均优于DCLM和Fineweb等现有方法，尤其在1B参数模型上表现出显著的整体性能提升和更高的token使用效率。
Reason 该论文属于典型的‘平行下游应用’，其核心是为通用LLM训练数据构建网页文本清洗管道，应用于Common Crawl数据过滤。尽管强调数据质量提升，但完全不涉及文档图像、多模态输入、OCR或视觉内容处理，与文档图像理解（DIU）任务无任何技术关联。属于纯NLP数据预处理工作，无法迁移解决DIU中的OCR幻觉、复杂排版或视觉-语言对齐等关键问题。
Abstract High-quality training data is fundamental to large language model (LLM) performance, yet existing preprocessing pipelines often struggle to effectively remove noise and unstructured content from web-scale corpora. This paper presents Blu-WERP, a novel data preprocessing pipeline designed to optimize the quality of Common Crawl WARC files for LLM training. We demonstrate that Blu-WERP significantly outperforms established baselines including DCLM across multiple model scales and evaluation benchmarks. Our pipeline processes CC WARC dumps, implementing advanced filtering and quality assessment mechanisms. We conducted comprehensive evaluations using models with 150M, 400M, 530M, 750M, and 1B parameters, testing against nine standard benchmarks categorized as World Knowledge & Reasoning, Language Understanding, and Commonsense Reasoning. Results show Blu-WERP consistently achieved superior performance across all model scales. At the 1B parameter scale, Relatively Blu-WERP demonstrates a 4.0% and 9.5% aggregate improvement over DCLM and Fineweb respectively, while achieving quality-per-token efficiency gain. Categorical analysis reveals 2.4% improvement in World Knowledge & Reasoning, 6.2% improvement in Language Understanding, and 4.2% improvement in Commonsense Reasoning. These results establish Blu-WERP as a state-of-the-art preprocessing pipeline that substantially improves LLM training data quality and downstream model performance with reduced computational cost. Our findings contribute to the growing body of research on data-centric AI, demonstrating that preprocessing pipeline design significantly impacts LLM capabilities. The Blu-WERP pipeline represents a practical advancement in data quality optimization, offering researchers and practitioners an effective solution for improving LLM training efficiency and model performance.
Authors Gowtham, Sai Rupesh, Sanjay Kumar, Saravanan, Venkata Chaithanya
Categories Computation and Language
0
MTikGuard System: A Transformer-Based Multimodal System for Child-Safe Content Moderation on TikTok PACLIC39
Title CN MTikGuard系统：一种基于Transformer的TikTok儿童安全内容审核多模态系统
Keywords 内容审核 · 多模态学习 · Transformer · 短视频分析 · 社会媒体安全
Summary 本文提出MTikGuard，一个用于TikTok平台的实时多模态有害内容检测系统，包含扩展的TikHarm数据集、多模态分类框架和基于Kafka与Spark的流式架构。实验显示其在准确率和F1分数上表现良好，适用于大规模社交内容审核部署。
Reason 该论文属于社交媒体有害内容检测的垂直应用，任务场景为视频内容审核，与文档图像理解（DIU）或DocVQA无关。其方法聚焦于短视频的多模态（视觉、音频、文本）融合分类，不涉及OCR、布局分析、表格识别或文档推理等核心DIU问题。尽管使用了多模态模型，但技术路线针对的是社交平台内容安全，无法迁移解决文档中的OCR幻觉、密集文本处理或复杂排版理解等痛点。
Abstract With the rapid rise of short-form videos, TikTok has become one of the most influential platforms among children and teenagers, but also a source of harmful content that can affect their perception and behavior. Such content, often subtle or deceptive, challenges traditional moderation methods due to the massive volume and real-time nature of uploads. This paper presents MTikGuard, a real-time multimodal harmful content detection system for TikTok, with three key contributions: (1) an extended TikHarm dataset expanded to 4,723 labeled videos by adding diverse real-world samples, (2) a multimodal classification framework integrating visual, audio, and textual features to achieve state-of-the-art performance with 89.37% accuracy and 89.45% F1-score, and (3) a scalable streaming architecture built on Apache Kafka and Apache Spark for real-time deployment. The results demonstrate the effectiveness of combining dataset expansion, advanced multimodal fusion, and robust deployment for practical large-scale social media content moderation. The dataset is available at https://github.com/ntdat-8324/MTikGuard-System.git.
Authors Dat Thanh Nguyen, Nguyen Hung Lam, Anh Hoang-Thi Nguyen, Trong-Hop Do
Comment Accepted at PACLIC39
Categories Computation and Language
0
CommonVoice-SpeechRE and RPG-MoGe: Advancing Speech Relation Extraction with a New Dataset and Multi-Order Generative Framework
Title CN CommonVoice-SpeechRE和RPG-MoGe：通过新数据集和多阶生成框架推进语音关系抽取
Keywords 语音关系抽取 · 多模态学习 · 真实语音数据集 · 生成模型 · 跨模态对齐
Summary 本文提出了一个名为CommonVoice-SpeechRE的大规模真实人类语音数据集，以及一种关系提示引导的多阶生成集成框架RPG-MoGe，用于提升语音关系抽取性能。该方法通过多顺序生成策略和CNN-based隐式关系预测头来增强跨模态对齐。然而，该工作完全聚焦于语音模态，与文档图像理解无关联。
Reason 该论文研究的是语音关系抽取（SpeechRE），属于语音处理与自然语言处理交叉领域，任务目标是从语音中直接提取关系三元组。其核心贡献为语音数据集构建和跨模态对齐生成框架，与文档图像理解（DIU）或DocVQA无关。既不涉及文档布局、文本识别、视觉表示学习，也无法迁移用于解决OCR幻觉、复杂排版或长文档推理等DIU核心问题。
Abstract Speech Relation Extraction (SpeechRE) aims to extract relation triplets directly from speech. However, existing benchmark datasets rely heavily on synthetic data, lacking sufficient quantity and diversity of real human speech. Moreover, existing models also suffer from rigid single-order generation templates and weak semantic alignment, substantially limiting their performance. To address these challenges, we introduce CommonVoice-SpeechRE, a large-scale dataset comprising nearly 20,000 real-human speech samples from diverse speakers, establishing a new benchmark for SpeechRE research. Furthermore, we propose the Relation Prompt-Guided Multi-Order Generative Ensemble (RPG-MoGe), a novel framework that features: (1) a multi-order triplet generation ensemble strategy, leveraging data diversity through diverse element orders during both training and inference, and (2) CNN-based latent relation prediction heads that generate explicit relation prompts to guide cross-modal alignment and accurate triplet generation. Experiments show our approach outperforms state-of-the-art methods, providing both a benchmark dataset and an effective solution for real-world SpeechRE. The source code and dataset are publicly available at https://github.com/NingJinzhong/SpeechRE_RPG_MoGe.
Authors Jinzhong Ning, Paerhati Tulajiang, Yingying Le, Yijia Zhang, Yuanyuan Sun, Hongfei Lin, Haifeng Liu
Categories Computation and Language
0
MF-GCN: A Multi-Frequency Graph Convolutional Network for Tri-Modal Depression Detection Using Eye-Tracking, Facial, and Acoustic Features
Title CN MF-GCN：一种用于通过眼动追踪、面部和声学特征进行三模态抑郁检测的多频率图卷积网络
Keywords 抑郁症检测 · 多模态学习 · 图卷积网络 · 眼动追踪 · 情感计算
Summary 本文提出了一种多频率图卷积网络MF-GCN，结合眼动、面部表情和声音特征进行抑郁症检测，并构建了一个新的三模态数据集。模型通过多频率滤波模块捕捉不同频段的信息，在多个分类任务上取得了优异性能，并在外部数据集上验证了泛化能力。
Reason 该论文研究的是基于多模态生理信号（眼动、面部、语音）的精神疾病检测，属于医疗健康领域的特定应用任务，与文档图像理解（DIU）或视觉-语言模型在文档处理中的核心问题（如OCR幻觉、复杂排版、长文档推理等）完全无关。尽管使用了图神经网络和多模态融合技术，但其方法设计针对的是生物信号的时间动态与跨模态交互，难以迁移到文档结构建模或VLM中的视觉文本对齐问题。
Abstract Depression is a prevalent global mental health disorder, characterised by persistent low mood and anhedonia. However, it remains underdiagnosed because current diagnostic methods depend heavily on subjective clinical assessments. To enable objective detection, we introduce a gold standard dataset of 103 clinically assessed participants collected through a tripartite data approach which uniquely integrated eye tracking data with audio and video to give a comprehensive representation of depressive symptoms. Eye tracking data quantifies the attentional bias towards negative stimuli that is frequently observed in depressed groups. Audio and video data capture the affective flattening and psychomotor retardation characteristic of depression. Statistical validation confirmed their significant discriminative power in distinguishing depressed from non depressed groups. We address a critical limitation of existing graph-based models that focus on low-frequency information and propose a Multi-Frequency Graph Convolutional Network (MF-GCN). This framework consists of a novel Multi-Frequency Filter Bank Module (MFFBM), which can leverage both low and high frequency signals. Extensive evaluation against traditional machine learning algorithms and deep learning frameworks demonstrates that MF-GCN consistently outperforms baselines. In binary classification, the model achieved a sensitivity of 0.96 and F2 score of 0.94. For the 3 class classification task, the proposed method achieved a sensitivity of 0.79 and specificity of 0.87 and siginificantly suprassed other models. To validate generalizability, the model was also evaluated on the Chinese Multimodal Depression Corpus (CMDC) dataset and achieved a sensitivity of 0.95 and F2 score of 0.96. These results confirm that our trimodal, multi frequency framework effectively captures cross modal interaction for accurate depression detection.
Authors Sejuti Rahman, Swakshar Deb, MD. Sameer Iqbal Chowdhury, MD. Jubair Ahmed Sourov, Mohammad Shamsuddin
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Enhancing Quranic Learning: A Multimodal Deep Learning Approach for Arabic Phoneme Recognition
Title CN 增强古兰经学习：一种用于阿拉伯语音素识别的多模态深度学习方法
Keywords 多模态深度学习 · 阿拉伯语音素识别 · 发音评估 · UniSpeech · BERT
Summary 本文提出了一种基于Transformer的多模态框架，结合UniSpeech声学嵌入和BERT文本嵌入，用于检测阿拉伯语音素的发音错误，主要应用于古兰经诵读训练。实验比较了多种融合策略，并在包含29个阿拉伯语音素的数据集上验证了方法的有效性。研究旨在推动计算机辅助语言学习（CALL）系统的发展。
Reason 该论文属于特定小语种（阿拉伯语）语音识别任务，聚焦于宗教文本（古兰经）发音训练，属于垂直领域的小语种应用。其技术路线为语音-文本多模态融合，但目标场景与文档图像理解（DIU/DocVQA）无关，无法迁移解决OCR幻觉、复杂排版或文档视觉理解中的核心问题。
Abstract Recent advances in multimodal deep learning have greatly enhanced the capability of systems for speech analysis and pronunciation assessment. Accurate pronunciation detection remains a key challenge in Arabic, particularly in the context of Quranic recitation, where subtle phonetic differences can alter meaning. Addressing this challenge, the present study proposes a transformer-based multimodal framework for Arabic phoneme mispronunciation detection that combines acoustic and textual representations to achieve higher precision and robustness. The framework integrates UniSpeech-derived acoustic embeddings with BERT-based textual embeddings extracted from Whisper transcriptions, creating a unified representation that captures both phonetic detail and linguistic context. To determine the most effective integration strategy, early, intermediate, and late fusion methods were implemented and evaluated on two datasets containing 29 Arabic phonemes, including eight hafiz sounds, articulated by 11 native speakers. Additional speech samples collected from publicly available YouTube recordings were incorporated to enhance data diversity and generalization. Model performance was assessed using standard evaluation metrics: accuracy, precision, recall, and F1-score, allowing a detailed comparison of the fusion strategies. Experimental findings show that the UniSpeech-BERT multimodal configuration provides strong results and that fusion-based transformer architectures are effective for phoneme-level mispronunciation detection. The study contributes to the development of intelligent, speaker-independent, and multimodal Computer-Aided Language Learning (CALL) systems, offering a practical step toward technology-supported Quranic pronunciation training and broader speech-based educational applications.
Authors Ayhan Kucukmanisa, Derya Gelmez, Sukru Selim Calik, Zeynep Hilal Kilimci
Comment 11 pages, 2 figures, 3 tables
Categories Artificial Intelligence
0
Can AI Perceive Physical Danger and Intervene?
Title CN 人工智能能否感知物理危险并进行干预？
Keywords 具身智能 · 物理安全 · 干预机制 · 多模态推理 · 安全基准
Summary 本文提出了一种基于真实伤害事件的可扩展物理安全基准测试方法，通过生成逼真的图像和视频评估基础模型对物理危险的感知与干预能力，并设计了后训练范式使模型能显式推理物理安全约束。研究聚焦于机器人或代理在现实世界中的安全性，强调对物理风险的理解与响应。
Reason 该论文研究的是具身智能（Embodied AI）系统在物理世界中的安全感知与干预，属于机器人或物理交互代理的安全问题。尽管涉及多模态推理和干预机制，但其核心场景与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版、细粒度文本识别等文档理解核心痛点。
Abstract When AI interacts with the physical world -- as a robot or an assistive agent -- new safety challenges emerge beyond those of purely ``digital AI". In such interactions, the potential for physical harm is direct and immediate. How well do state-of-the-art foundation models understand common-sense facts about physical safety, e.g. that a box may be too heavy to lift, or that a hot cup of coffee should not be handed to a child? In this paper, our contributions are three-fold: first, we develop a highly scalable approach to continuous physical safety benchmarking of Embodied AI systems, grounded in real-world injury narratives and operational safety constraints. To probe multi-modal safety understanding, we turn these narratives and constraints into photorealistic images and videos capturing transitions from safe to unsafe states, using advanced generative models. Secondly, we comprehensively analyze the ability of major foundation models to perceive risks, reason about safety, and trigger interventions; this yields multi-faceted insights into their deployment readiness for safety-critical agentic applications. Finally, we develop a post-training paradigm to teach models to explicitly reason about embodiment-specific safety constraints provided through system instructions. The resulting models generate thinking traces that make safety reasoning interpretable and transparent, achieving state of the art performance in constraint satisfaction evaluations. The benchmark is released at https://asimov-benchmark.github.io/v2
Authors Abhishek Jindal, Dmitry Kalashnikov, R. Alex Hofer, Oscar Chang, Divya Garikapati, Anirudha Majumdar, Pierre Sermanet, Vikas Sindhwani
Categories Artificial Intelligence
0
PersonaAgent with GraphRAG: Community-Aware Knowledge Graphs for Personalized LLM
Title CN 基于GraphRAG的PersonaAgent：面向个性化大语言模型的社区感知知识图谱
Keywords 个性化LLM · 知识图谱 · RAG · GraphRAG · AI代理
Summary 本文提出了一种名为PersonaAgent的个性化语言模型框架，通过构建知识图谱增强的检索生成机制（GraphRAG），提取用户行为偏好并结合全局交互模式进行动态提示生成，在LaMP基准上显著提升了多项个性化推荐任务的表现。然而，该工作聚焦于用户建模与推荐系统，不属于文档图像理解的技术源头创新。
Reason 该论文属于典型的平行下游应用，构建了一个基于知识图谱增强的个性化AI代理系统，应用于新闻分类、电影标签和产品评分等任务。其核心是将RAG与知识图谱结合用于用户画像建模，并未提出可用于文档图像理解（如OCR去幻觉、复杂排版处理、细粒度定位等）的底层方法。尽管GraphRAG涉及信息组织方式，但其设计目标与文档视觉结构无关，难以迁移至DIU领域。
Abstract We propose a novel framework for persona-based language model system, motivated by the need for personalized AI agents that adapt to individual user preferences. In our approach, the agent embodies the user's "persona" (e.g. user profile or taste) and is powered by a large language model (LLM). To enable the agent to leverage rich contextual information, we introduce a Knowledge-Graph-enhanced Retrieval-Augmented Generation (Graph RAG) mechanism that constructs an LLM-derived graph index of relevant documents and summarizes communities of related information. Our framework generates personalized prompts by combining: (1) a summary of the user's historical behaviors and preferences extracted from the knowledge graph, and (2) relevant global interaction patterns identified through graph-based community detection. This dynamic prompt engineering approach allows the agent to maintain consistent persona-aligned behaviors while benefiting from collective knowledge. On the LaMP benchmark, our method improves news categorization F1 by 11.1%, movie tagging F1 by 56.1%, and reduces product rating MAE by 10.4% over prior methods. Our code is available at https://anonymous.4open.science/r/PersonaAgentwGraphRAG-DE6F
Authors Siqi Liang, Yudi Zhang, Yue Guo
Categories Artificial Intelligence
0
The Coding Limits of Robust Watermarking for Generative Models
Title CN 生成模型鲁棒水印的编码极限
Keywords 水印 · 生成模型 · 编码理论 · 篡改检测 · 鲁棒性
Summary 本文提出了一个名为‘零比特篡改检测码’的编码抽象模型，用于分析生成模型中水印在面对符号篡改时的鲁棒性极限。理论上证明了在q元字母表下，当篡改率超过1 - 1/q时，任何水印方案都无法可靠检测篡改；实验部分验证了现有图像水印在简单裁剪和缩放操作下的失效情况。
Reason 该论文研究的是生成模型中水印技术的理论极限，属于内容安全与版权保护范畴，核心问题是水印在对抗篡改时的可靠性。其方法基于密码学抽象和信息论分析，并未涉及文档图像理解（DIU）中的OCR幻觉、复杂排版、细粒度定位或长文档推理等关键痛点。尽管水印可能用于文档溯源，但该工作无法迁移用于提升VLM在文档理解中的性能，且属于无关的平行应用领域。
Abstract We ask a basic question about cryptographic watermarking for generative models: to what extent can a watermark remain reliable when an adversary is allowed to corrupt the encoded signal? To study this question, we introduce a minimal coding abstraction that we call a zero-bit tamper-detection code. This is a secret-key procedure that samples a pseudorandom codeword and, given a candidate word, decides whether it should be treated as unmarked content or as the result of tampering with a valid codeword. It captures the two core requirements of robust watermarking: soundness and tamper detection. Within this abstraction we prove a sharp unconditional limit on robustness to independent symbol corruption. For an alphabet of size $q$, there is a critical corruption rate of $1 - 1/q$ such that no scheme with soundness, even relaxed to allow a fixed constant false positive probability on random content, can reliably detect tampering once an adversary can change more than this fraction of symbols. In particular, in the binary case no cryptographic watermark can remain robust if more than half of the encoded bits are modified. We also show that this threshold is tight by giving simple information-theoretic constructions that achieve soundness and tamper detection for all strictly smaller corruption rates. We then test experimentally whether this limit appears in practice by looking at the recent watermarking for images of Gunn, Zhao, and Song (ICLR 2025). We show that a simple crop and resize operation reliably flipped about half of the latent signs and consistently prevented belief-propagation decoding from recovering the codeword, erasing the watermark while leaving the image visually intact.
Authors Danilo Francati, Yevin Nikhel Goonatilake, Shubham Pawar, Daniele Venturi, Giuseppe Ateniese
Categories Artificial Intelligence
0
SRA-CP: Spontaneous Risk-Aware Selective Cooperative Perception
Title CN SRA-CP：自发式风险感知选择性协同感知
Keywords 协同感知 · 自动驾驶 · 风险感知 · 带宽优化 · 多智能体系统
Summary 本文提出了一种自发式风险感知选择性协同感知框架（SRA-CP），用于在车联网环境中实现高效的感知数据共享。该方法通过轻量级感知覆盖摘要广播，在检测到安全风险相关的盲区时才触发有选择性的通信协作，并优先传输对安全关键的信息，在仅使用20%带宽的情况下保持接近全量通信的精度。
Reason 该论文属于自动驾驶领域的协同感知研究，尽管涉及多智能体信息共享与带宽优化，但其核心任务与文档图像理解（DIU）无关。未解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心问题，且方法迁移至文档理解场景的路径不明确。
Abstract Cooperative perception (CP) offers significant potential to overcome the limitations of single-vehicle sensing by enabling information sharing among connected vehicles (CVs). However, existing generic CP approaches need to transmit large volumes of perception data that are irrelevant to the driving safety, exceeding available communication bandwidth. Moreover, most CP frameworks rely on pre-defined communication partners, making them unsuitable for dynamic traffic environments. This paper proposes a Spontaneous Risk-Aware Selective Cooperative Perception (SRA-CP) framework to address these challenges. SRA-CP introduces a decentralized protocol where connected agents continuously broadcast lightweight perception coverage summaries and initiate targeted cooperation only when risk-relevant blind zones are detected. A perceptual risk identification module enables each CV to locally assess the impact of occlusions on its driving task and determine whether cooperation is necessary. When CP is triggered, the ego vehicle selects appropriate peers based on shared perception coverage and engages in selective information exchange through a fusion module that prioritizes safety-critical content and adapts to bandwidth constraints. We evaluate SRA-CP on a public dataset against several representative baselines. Results show that SRA-CP achieves less than 1% average precision (AP) loss for safety-critical objects compared to generic CP, while using only 20% of the communication bandwidth. Moreover, it improves the perception performance by 15% over existing selective CP methods that do not incorporate risk awareness.
Authors Jiaxi Liu, Chengyuan Ma, Hang Zhou, Weizhe Tang, Shixiao Liang, Haoyang Ding, Xiaopeng Li, Bin Ran
Categories Artificial Intelligence
0
Planning with Sketch-Guided Verification for Physics-Aware Video Generation
Title CN 基于草图引导验证的物理感知视频生成规划
Keywords 视频生成 · 运动规划 · 验证机制 · 视觉语言模型 · 物理合理性
Summary 本文提出SketchVerify，一种无需训练的视频生成规划框架，通过生成多个候选运动轨迹并利用视觉语言验证器对渲染出的轻量级视频草图进行语义一致性和物理合理性的评分与筛选，在生成前优化运动规划。该方法提升了视频的运动质量和长期一致性，同时降低了计算成本。
Reason 该论文研究的是视频生成中的运动轨迹规划与验证，属于视频生成与物理模拟领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心方法——通过轻量级草图渲染和视觉语言验证器进行测试时验证——虽然在视频生成中有效，但难以迁移到解决文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。且该工作聚焦于动态场景的物理合理性，与静态文档分析任务差异巨大。
Abstract Recent video generation approaches increasingly rely on planning intermediate control signals such as object trajectories to improve temporal coherence and motion fidelity. However, these methods mostly employ single-shot plans that are typically limited to simple motions, or iterative refinement which requires multiple calls to the video generator, incuring high computational cost. To overcome these limitations, we propose SketchVerify, a training-free, sketch-verification-based planning framework that improves motion planning quality with more dynamically coherent trajectories (i.e., physically plausible and instruction-consistent motions) prior to full video generation by introducing a test-time sampling and verification loop. Given a prompt and a reference image, our method predicts multiple candidate motion plans and ranks them using a vision-language verifier that jointly evaluates semantic alignment with the instruction and physical plausibility. To efficiently score candidate motion plans, we render each trajectory as a lightweight video sketch by compositing objects over a static background, which bypasses the need for expensive, repeated diffusion-based synthesis while achieving comparable performance. We iteratively refine the motion plan until a satisfactory one is identified, which is then passed to the trajectory-conditioned generator for final synthesis. Experiments on WorldModelBench and PhyWorldBench demonstrate that our method significantly improves motion quality, physical realism, and long-term consistency compared to competitive baselines while being substantially more efficient. Our ablation study further shows that scaling up the number of trajectory candidates consistently enhances overall performance.
Authors Yidong Huang, Zun Wang, Han Lin, Dong-Ki Kim, Shayegan Omidshafiei, Jaehong Yoon, Yue Zhang, Mohit Bansal
Comment website: https://sketchverify.github.io/
Categories Artificial Intelligence, Computer Vision and Pattern Recognition, Computation and Language
0
CATCODER: Repository-Level Code Generation with Relevant Code and Type Context ACM Transactions on Software Engineering and Methodology
Title CN CATCODER：基于相关代码和类型上下文的仓库级代码生成
Keywords 代码生成 · 仓库级上下文 · 类型依赖 · 静态分析 · LLM
Summary 本文提出CatCoder，一种面向静态类型编程语言的仓库级代码生成框架。该框架通过静态分析提取类型依赖，并将其与检索到的相关代码合并，构建更全面的上下文提示以提升代码生成的准确性。在Java和Rust任务上的实验表明，该方法显著优于RepoCoder基线，在编译通过率和测试通过率上均有提升，且在多种LLM上具备良好泛化性和可扩展性。
Reason 该论文属于典型的平行下游应用（Wrapper/Application），其核心是将现有LLM与检索、静态分析结合用于仓库级代码生成，应用场景局限于软件工程中的代码补全任务。尽管方法上融合了类型上下文和代码检索，但未提出可迁移至文档图像理解（DIU）领域的底层技术创新，如视觉表示学习、多模态对齐、高分辨率处理或去幻觉机制。其技术路线与OCR幻觉、复杂排版解析、长文档推理等DIU核心痛点无直接关联，无法为VLM-based文档理解提供方法论启发。
Abstract Large language models (LLMs) have demonstrated remarkable capabilities in code generation tasks. However, repository-level code generation presents unique challenges, particularly due to the need to utilize information spread across multiple files within a repository. Specifically, successful generation depends on a solid grasp of both general, context-agnostic knowledge and specific, context-dependent knowledge. While LLMs are widely used for the context-agnostic aspect, existing retrieval-based approaches sometimes fall short as they are limited in obtaining a broader and deeper repository context. In this paper, we present CatCoder, a novel code generation framework designed for statically typed programming languages. CatCoder enhances repository-level code generation by integrating relevant code and type context. Specifically, it leverages static analyzers to extract type dependencies and merges this information with retrieved code to create comprehensive prompts for LLMs. To evaluate the effectiveness of CatCoder, we adapt and construct benchmarks that include 199 Java tasks and 90 Rust tasks. The results show that CatCoder outperforms the RepoCoder baseline by up to 14.44% and 17.35%, in terms of compile@k and pass@k scores. In addition, the generalizability of CatCoder is assessed using various LLMs, including both code-specialized models and general-purpose models. Our findings indicate consistent performance improvements across all models, which underlines the practicality of CatCoder. Furthermore, we evaluate the time consumption of CatCoder in a large open source repository, and the results demonstrate the scalability of CatCoder.
Authors Zhiyuan Pan, Xing Hu, Xin Xia, Xiaohu Yang
Comment Revised and extended version; To be published in ACM Transactions on Software Engineering and Methodology
Categories Artificial Intelligence
0
That's not natural: The Impact of Off-Policy Training Data on Probe Performance EurIPS 2025 Workshop on Private AI Governance
Title CN 这并不自然：离策略训练数据对探测性能的影响
Keywords 探针 · 离策略数据 · 行为检测 · 分布偏移 · 大语言模型
Summary 论文系统评估了使用合成或离策略数据训练的行为探针在不同LLM行为上的泛化能力，发现数据生成策略显著影响探针性能，且跨域偏移带来的性能下降尤为严重。研究表明，在缺乏真实策略数据时，同域的离策略数据更有利于构建可靠的监控探针。
Reason 该论文研究的是大语言模型（LLM）行为探测中的训练数据偏差问题，聚焦于欺骗、奉承等社会性行为的探测器泛化能力。其核心问题是基于合成或离策略数据训练的探针能否在真实场景中有效检测LLM的不良行为。尽管涉及LLM监控与推理行为分析，但完全属于纯NLP领域，未涉及任何视觉输入、多模态建模或文档结构理解。与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版、视觉-文本对齐等无关联，无法迁移用于解决DIU任务。
Abstract Probing has emerged as a promising method for monitoring Large Language Models (LLMs), enabling inference-time detection of concerning behaviours such as deception and sycophancy. However, natural examples of many behaviours are rare, forcing researchers to rely on synthetic or off-policy LLM responses for training probes. We systematically evaluate how the use of synthetic and off-policy data influences probe generalisation across eight distinct LLM behaviours. Testing linear and attention probes across multiple LLMs, we find that the response generation strategy can significantly affect probe performance, though the magnitude of this effect varies by behaviour. We find that successful generalisation from off-policy data, to test sets where the model is incentivised to produce the target behaviour, is predictive of successful on-policy generalisation. Leveraging this result, we predict that Deception and Sandbagging probes may fail to generalise from off-policy to on-policy data when used in real monitoring scenarios. Notably, shifts in the training data domain still cause even larger performance degradation, with different-domain test scores being consistently lower than the same-domain ones. These results indicate that, in the absence of on-policy data, using same-domain off-policy data yields more reliable probes than using on-policy data from a different domain, emphasizing the need for methods that can better handle distribution shifts in LLM monitoring.
Authors Nathalie Kirch, Samuel Dower, Adrians Skapars, Ekdeep Singh Lubana, Dmitrii Krasheninnikov
Comment 10 pages, EurIPS 2025 Workshop on Private AI Governance
Categories Artificial Intelligence
0
SHIELD: Secure Hypernetworks for Incremental Expansion Learning Defense
Title CN SHIELD：用于增量扩展学习防御的安全超网络
Keywords 持续学习 · 对抗鲁棒性 · 超网络 · 区间界传播 · Interval MixUp
Summary 本文提出SHIELD，一种结合区间界传播（IBP）和超网络的框架，用于在对抗环境下进行可验证鲁棒的持续学习。通过任务嵌入生成特定任务参数，避免回放缓冲区和完整模型复制，提升效率。引入Interval MixUp方法，在混合虚拟样本时保证鲁棒性并缓解包裹效应。实验表明该方法在多个基准上优于现有方法，具备良好的准确性和可扩展性。
Reason 该论文研究的是持续学习在对抗性条件下的鲁棒性问题，核心贡献为结合区间传播（IBP）与超网络架构实现可认证的鲁棒持续学习，并提出Interval MixUp训练策略。其研究领域属于纯AI安全与持续学习，尽管技术上有创新，但与文档图像理解（DIU）、多模态大模型处理文档内容、OCR、布局分析、表格识别等任务无直接关联。无法迁移解决DIU中的OCR幻觉、密集文本识别、复杂排版或长文档推理等核心痛点。
Abstract Continual learning under adversarial conditions remains an open problem, as existing methods often compromise either robustness, scalability, or both. We propose a novel framework that integrates Interval Bound Propagation (IBP) with a hypernetwork-based architecture to enable certifiably robust continual learning across sequential tasks. Our method, SHIELD, generates task-specific model parameters via a shared hypernetwork conditioned solely on compact task embeddings, eliminating the need for replay buffers or full model copies and enabling efficient over time. To further enhance robustness, we introduce Interval MixUp, a novel training strategy that blends virtual examples represented as $\ell_{\infty}$ balls centered around MixUp points. Leveraging interval arithmetic, this technique guarantees certified robustness while mitigating the wrapping effect, resulting in smoother decision boundaries. We evaluate SHIELD under strong white-box adversarial attacks, including PGD and AutoAttack, across multiple benchmarks. It consistently outperforms existing robust continual learning methods, achieving state-of-the-art average accuracy while maintaining both scalability and certification. These results represent a significant step toward practical and theoretically grounded continual learning in adversarial settings.
Authors Patryk Krukowski, Łukasz Gorczyca, Piotr Helm, Kamil Książek, Przemysław Spurek
Categories Artificial Intelligence
0
Designing and Generating Diverse, Equitable Face Image Datasets for Face Verification Tasks
Title CN 为面部验证任务设计和生成多样化、公平的面部图像数据集
Keywords 人脸验证 · 数据集偏见 · 合成数据 · 公平性 · 生成模型
Summary 本文提出一种利用生成模型创建多样化、高质量合成人脸图像的方法，以缓解现有面部验证数据集中存在的种族、性别等偏见问题，并发布了DIF-V数据集用于促进更公平的人脸验证技术发展。
Reason 该论文聚焦于人脸识别中的数据集偏见问题，提出通过生成模型构建具有多样性和包容性的合成人脸数据集。研究属于计算机视觉中的人脸识别领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法论集中在人脸生成与公平性，无法迁移解决OCR幻觉、复杂排版解析、密集文本识别或长文档推理等DIU核心痛点。
Abstract Face verification is a significant component of identity authentication in various applications including online banking and secure access to personal devices. The majority of the existing face image datasets often suffer from notable biases related to race, gender, and other demographic characteristics, limiting the effectiveness and fairness of face verification systems. In response to these challenges, we propose a comprehensive methodology that integrates advanced generative models to create varied and diverse high-quality synthetic face images. This methodology emphasizes the representation of a diverse range of facial traits, ensuring adherence to characteristics permissible in identity card photographs. Furthermore, we introduce the Diverse and Inclusive Faces for Verification (DIF-V) dataset, comprising 27,780 images of 926 unique identities, designed as a benchmark for future research in face verification. Our analysis reveals that existing verification models exhibit biases toward certain genders and races, and notably, applying identity style modifications negatively impacts model performance. By tackling the inherent inequities in existing datasets, this work not only enriches the discussion on diversity and ethics in artificial intelligence but also lays the foundation for developing more inclusive and reliable face verification technologies
Authors Georgia Baltsou, Ioannis Sarridis, Christos Koutlis, Symeon Papadopoulos
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Quantum Masked Autoencoders for Vision Learning
Title CN 用于视觉学习的量子掩码自编码器
Keywords 量子自编码器 · 掩码学习 · 视觉表示学习 · 量子机器学习 · 图像重建
Summary 本文提出了一种量子掩码自编码器（QMAE），利用量子态表示来学习被掩码的图像特征，并在MNIST数据集上实现了比现有量子自编码器更高的分类准确率和图像重建质量。这是首次将掩码学习引入量子自编码器架构的工作。
Reason 该论文研究量子计算框架下的掩码自编码器，属于量子机器学习与视觉的交叉，但其方法目前仅在MNIST简单图像上验证，且未涉及文本、布局、多模态对齐或文档结构等文档图像理解（DIU）核心问题。虽然‘掩码重建’思想在理论上可能与视觉表示学习相关，但该工作基于量子计算平台，硬件依赖强、可迁移性极低，无法直接迁移到当前主流的VLM或文档理解系统中解决OCR幻觉、复杂排版或长文档推理等实际痛点，属于无关领域的方法探索。
Abstract Classical autoencoders are widely used to learn features of input data. To improve the feature learning, classical masked autoencoders extend classical autoencoders to learn the features of the original input sample in the presence of masked-out data. While quantum autoencoders exist, there is no design and implementation of quantum masked autoencoders that can leverage the benefits of quantum computing and quantum autoencoders. In this paper, we propose quantum masked autoencoders (QMAEs) that can effectively learn missing features of a data sample within quantum states instead of classical embeddings. We showcase that our QMAE architecture can learn the masked features of an image and can reconstruct the masked input image with improved visual fidelity in MNIST images. Experimental evaluation highlights that QMAE can significantly outperform (12.86% on average) in classification accuracy compared to state-of-the-art quantum autoencoders in the presence of masks.
Authors Emma Andrews, Prabhat Mishra
Categories Artificial Intelligence
0
Crafting Imperceptible On-Manifold Adversarial Attacks for Tabular Data
Title CN 为表格数据构建不可感知的流形内对抗攻击
Keywords 对抗攻击 · 表格数据 · 变分自编码器 · 流形学习 · 数据分布一致性
Summary 本文提出一种基于混合输入变分自编码器（VAE）的对抗攻击框架，用于在保持统计一致性的前提下对表格数据进行潜空间扰动。通过将分类和数值特征统一到共享流形中，生成更真实且分布对齐的对抗样本，并引入IDSR指标评估攻击效果与分布一致性。实验表明该方法在多个数据集上优于传统输入空间攻击方法。
Reason 该论文研究的是针对表格数据（tabular data）的对抗性攻击，属于安全与鲁棒性领域，而非文档图像理解（DIU）或视觉-语言多模态任务。其核心方法基于VAE在潜空间生成符合数据分布的扰动，虽涉及‘表格’一词，但处理对象是结构化数值/类别数据，而非文档图像中的布局、文本识别或视觉内容理解。无法迁移解决OCR幻觉、复杂排版解析或长文档推理等DIU核心问题。
Abstract Adversarial attacks on tabular data present unique challenges due to the heterogeneous nature of mixed categorical and numerical features. Unlike images where pixel perturbations maintain visual similarity, tabular data lacks intuitive similarity metrics, making it difficult to define imperceptible modifications. Additionally, traditional gradient-based methods prioritise $\ell_p$-norm constraints, often producing adversarial examples that deviate from the original data distributions. To address this, we propose a latent-space perturbation framework using a mixed-input Variational Autoencoder (VAE) to generate statistically consistent adversarial examples. The proposed VAE integrates categorical embeddings and numerical features into a unified latent manifold, enabling perturbations that preserve statistical consistency. We introduce In-Distribution Success Rate (IDSR) to jointly evaluate attack effectiveness and distributional alignment. Evaluation across six publicly available datasets and three model architectures demonstrates that our method achieves substantially lower outlier rates and more consistent performance compared to traditional input-space attacks and other VAE-based methods adapted from image domain approaches, achieving substantially lower outlier rates and higher IDSR across six datasets and three model architectures. Our comprehensive analyses of hyperparameter sensitivity, sparsity control, and generative architecture demonstrate that the effectiveness of VAE-based attacks depends strongly on reconstruction quality and the availability of sufficient training data. When these conditions are met, the proposed framework achieves superior practical utility and stability compared with input-space methods. This work underscores the importance of maintaining on-manifold perturbations for generating realistic and robust adversarial examples in tabular domains.
Authors Zhipeng He, Alexander Stevens, Chun Ouyang, Johannes De Smedt, Alistair Barros, Catarina Moreira
Comment Final Version
Categories Artificial Intelligence
0
Meta-World+: An Improved, Standardized, RL Benchmark NeurIPs 2025, Datasets and Benchmarks
Title CN Meta-World+：一个改进的、标准化的强化学习基准
Keywords 强化学习 · 基准测试 · 多任务学习 · 元学习 · 可复现性
Summary 本文针对Meta-World强化学习基准存在的版本不一致和结果不可复现问题，提出了一个标准化、可复现的新版本Meta-World+，提升了实验的公平性和系统可用性。主要贡献在于工具链和基准规范，而非模型或表示学习方法。
Reason 该论文属于具身智能与强化学习领域的基准测试工作，聚焦于多任务和元强化学习在机器人控制环境中的评估。其研究内容与文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的应用无直接关联，无法迁移解决OCR幻觉、复杂排版、长文档推理等核心痛点。
Abstract Meta-World is widely used for evaluating multi-task and meta-reinforcement learning agents, which are challenged to master diverse skills simultaneously. Since its introduction however, there have been numerous undocumented changes which inhibit a fair comparison of algorithms. This work strives to disambiguate these results from the literature, while also leveraging the past versions of Meta-World to provide insights into multi-task and meta-reinforcement learning benchmark design. Through this process we release a new open-source version of Meta-World (https://github.com/Farama-Foundation/Metaworld/) that has full reproducibility of past results, is more technically ergonomic, and gives users more control over the tasks that are included in a task set.
Authors Reginald McLean, Evangelos Chatzaroulas, Luc McCutcheon, Frank Röder, Tianhe Yu, Zhanpeng He, K. R. Zentner, Ryan Julian, J K Terry, Isaac Woungang, Nariman Farsad, Pablo Samuel Castro
Comment Accepted at NeurIPs 2025, Datasets and Benchmarks
Categories Artificial Intelligence
0
AI Workers, Geopolitics, and Algorithmic Collective Action
Title CN 人工智能工作者、地缘政治与算法集体行动
Keywords AI治理 · 地缘政治 · 算法集体行动 · 参与式设计 · AI伦理
Summary 本文从国际政治经济学（IPE）视角出发，探讨AI公司如何参与地缘政治，并提出AI开发者本身可作为地缘政治行为者。作者主张仅靠治理无法确保AI的伦理发展，应重视在开发现场的自下而上干预，特别是通过参与式设计激发AI从业者的能动性，推动算法集体行动（ACA）。
Reason 该论文属于AI治理与社会科学交叉领域，讨论AI工人在地缘政治中的角色和自下而上的干预机制，不涉及文档图像理解（DIU）或视觉语言模型（VLM）的技术方法创新。内容聚焦于参与式设计与社会行动，与OCR、DocVQA、布局分析等DIU核心任务无关，也未提出可迁移至文档理解领域的底层技术工具。
Abstract According to the theory of International Political Economy (IPE), states are often incentivized to rely on rather than constrain powerful corporations. For this reason, IPE provides a useful lens to explain why efforts to govern Artificial Intelligence (AI) at the international and national levels have thus far been developed, applied, and enforced unevenly. Building on recent work that explores how AI companies engage in geopolitics, this position paper argues that some AI workers can be considered actors of geopolitics. It makes the timely case that governance alone cannot ensure responsible, ethical, or robust AI development and use, and greater attention should be paid to bottom-up interventions at the site of AI development. AI workers themselves should be situated as individual agents of change, especially when considering their potential to foster Algorithmic Collective Action (ACA). Drawing on methods of Participatory Design (PD), this paper proposes engaging AI workers as sources of knowledge, relative power, and intentionality to encourage more responsible and just AI development and create the conditions that can facilitate ACA.
Authors Sydney Reis
Categories Artificial Intelligence
0
MusicAIR: A Multimodal AI Music Generation Framework Powered by an Algorithm-Driven Core IEEE Big Data 2025
Title CN MusicAIR：一种由算法驱动核心支持的多模态AI音乐生成框架
Keywords 音乐生成 · 多模态AI · 算法驱动 · 符号音乐 · 文本到音乐
Summary 本文提出MusicAIR，一个基于算法驱动的符号化音乐核心的多模态AI音乐生成框架，能够从歌词、文本或图像生成符合音乐理论的旋律。系统通过连接歌词与节奏信息自动生成完整乐谱，并开发了GenAIM网页工具。实验显示其在关键指标上优于人类作曲者。然而，该工作属于AI音乐创作领域，与文档图像理解无关联。
Reason 该论文属于音乐生成领域，与文档图像理解（DIU）、DocVQA、OCR、布局分析等任务完全无关。尽管其为多模态框架，但目标场景和方法均不涉及视觉文本理解或文档结构建模，无法迁移用于解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract Recent advances in generative AI have made music generation a prominent research focus. However, many neural-based models rely on large datasets, raising concerns about copyright infringement and high-performance costs. In contrast, we propose MusicAIR, an innovative multimodal AI music generation framework powered by a novel algorithm-driven symbolic music core, effectively mitigating copyright infringement risks. The music core algorithms connect critical lyrical and rhythmic information to automatically derive musical features, creating a complete, coherent melodic score solely from the lyrics. The MusicAIR framework facilitates music generation from lyrics, text, and images. The generated score adheres to established principles of music theory, lyrical structure, and rhythmic conventions. We developed Generate AI Music (GenAIM), a web tool using MusicAIR for lyric-to-song, text-to-music, and image-to-music generation. In our experiments, we evaluated AI-generated music scores produced by the system using both standard music metrics and innovative analysis that compares these compositions with original works. The system achieves an average key confidence of 85%, outperforming human composers at 79%, and aligns closely with established music theory standards, demonstrating its ability to generate diverse, human-like compositions. As a co-pilot tool, GenAIM can serve as a reliable music composition assistant and a possible educational composition tutor while simultaneously lowering the entry barrier for all aspiring musicians, which is innovative and significantly contributes to AI for music generation.
Authors Callie C. Liao, Duoduo Liao, Ellie L. Zhang
Comment Accepted by IEEE Big Data 2025
Categories Artificial Intelligence, Computation and Language
0
FORWARD: Dataset of a forwarder operating in rough terrain
Title CN FORWARD：一种在崎岖地形中作业的集材机数据集
Keywords 数据集 · 森林机械 · 多模态感知 · 自主控制 · 高分辨率传感
Summary 本文提出了一个名为FORWARD的高分辨率多模态数据集，记录了瑞典两地森林作业现场中一台大型集材机的操作数据。数据包括RTK-GNSS定位、360度视频、IMU、CAN总线信号、振动传感器以及激光扫描地形信息，并对约18小时的作业过程进行了基于视频的工作单元标注。该数据集旨在支持森林机械的自主控制、感知与仿真建模研究。
Reason 该论文介绍的是林业机械在野外环境下的多模态传感器数据集，主要面向自动驾驶、机器人控制与交通可通行性分析，属于具身智能与机器人领域。尽管包含高分辨率感知数据，但其任务目标（如地形穿越、燃料消耗优化）与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。
Abstract We present FORWARD, a high-resolution multimodal dataset of a cut-to-length forwarder operating in rough terrain on two harvest sites in the middle part of Sweden. The forwarder is a large Komatsu model equipped with a variety of sensors, including RTK-GNSS, 360-camera, operator vibration sensors, internal CAN-bus signal recording, and multiple IMUs. The data includes event time logs recorded in 5 Hz with e.g., driving speed, fuel consumption, vehicle position with centimeter accuracy, and crane use while the vehicle operates in forest areas laser-scanned with very high-resolution, $\sim$1500 points per square meter. Production log files (StanForD standard) with time-stamped machine events, extensive video material, and terrain data in various formats are included as well. About 18 hours of regular wood extraction work during three days is annotated from 360-video material into individual work elements and included in the dataset. We also include scenario specifications of conducted experiments on forest roads and in terrain. Scenarios include repeatedly driving the same routes with and without steel tracks, different load weight, and different target driving speeds. The dataset is intended for developing models and algorithms for trafficability, perception, and autonomous control of forest machines using artificial intelligence, simulation, and experiments on physical testbeds. In part, we focus on forwarders traversing terrain, avoiding obstacles, and loading or unloading logs, with consideration for efficiency, fuel consumption, safety, and environmental impact. Other benefits of the open dataset include the ability to explore auto-generation and calibration of forestry machine simulators and automation scenario descriptions using the data recorded in the field.
Authors Mikael Lundbäck, Erik Wallin, Carola Häggström, Mattias Nyström, Andreas Grönlund, Mats Richardson, Petrus Jönsson, William Arnvik, Lucas Hedström, Arvid Fälldin, Martin Servin
Comment 25 pages, 22 figures
Categories Artificial Intelligence
0
Fairness Evaluation of Large Language Models in Academic Library Reference Services
Title CN 大型语言模型在学术图书馆参考服务中的公平性评估
Keywords 大语言模型 · 公平性 · 社会偏见 · 图书馆服务 · 人机交互
Summary 本文评估了六种主流大语言模型在学术图书馆虚拟参考服务中对不同性别、种族/民族和机构角色用户的响应是否存在偏见。结果显示大多数模型在种族和性别上未表现出显著歧视，但在个别模型中存在对女性的轻微刻板印象。模型能根据用户角色调整语言风格，体现专业规范而非歧视行为。结论认为当前LLM在提供公平、情境适配的服务方面具有潜力。
Reason 该论文属于纯NLP领域的社会性研究，探讨LLM在图书馆参考服务中对不同用户身份的响应公平性，不涉及任何文档图像、视觉内容理解或OCR相关技术。主题与文档图像理解（DIU）完全无关，属于典型的平行应用类研究，且聚焦于伦理与社会偏见而非技术方法创新。
Abstract As libraries explore large language models (LLMs) for use in virtual reference services, a key question arises: Can LLMs serve all users equitably, regardless of demographics or social status? While they offer great potential for scalable support, LLMs may also reproduce societal biases embedded in their training data, risking the integrity of libraries' commitment to equitable service. To address this concern, we evaluate whether LLMs differentiate responses across user identities by prompting six state-of-the-art LLMs to assist patrons differing in sex, race/ethnicity, and institutional role. We find no evidence of differentiation by race or ethnicity, and only minor evidence of stereotypical bias against women in one model. LLMs demonstrate nuanced accommodation of institutional roles through the use of linguistic choices related to formality, politeness, and domain-specific vocabularies, reflecting professional norms rather than discriminatory treatment. These findings suggest that current LLMs show a promising degree of readiness to support equitable and contextually appropriate communication in academic library reference services.
Authors Haining Wang, Jason Clark, Yueru Yan, Star Bradley, Ruiyang Chen, Yiqiong Zhang, Hengyi Fu, Zuoyu Tian
Categories Artificial Intelligence, Computation and Language
0
HazeMatching: Dehazing Light Microscopy Images with Guided Conditional Flow Matching
Title CN HazeMatching：通过引导条件流匹配去除光学显微镜图像雾霾
Keywords 图像去雾 · 流匹配 · 条件生成模型 · 计算显微成像 · 图像恢复
Summary 本文提出HazeMatching，一种用于去除宽场荧光显微镜图像中散焦模糊（即‘雾霾’）的迭代生成方法。该方法基于条件流匹配框架，通过在条件速度场中引入模糊观测引导生成过程，在多个数据集上实现了图像保真度与感知真实性的良好平衡。无需显式退化算子的设计使其适用于真实显微镜数据。
Reason 该论文研究的是生物医学图像去噪（去雾霾）问题，属于计算显微成像领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。尽管使用了条件流匹配这一生成建模方法，但其目标是平衡图像保真度与感知质量，并未涉及文本识别、布局分析、多模态理解或OCR相关挑战，无法迁移解决文档图像中的幻觉、密集文本或复杂排版等问题。
Abstract Fluorescence microscopy is a major driver of scientific progress in the life sciences. Although high-end confocal microscopes are capable of filtering out-of-focus light, cheaper and more accessible microscopy modalities, such as widefield microscopy, can not, which consequently leads to hazy image data. Computational dehazing is trying to combine the best of both worlds, leading to cheap microscopy but crisp-looking images. The perception-distortion trade-off tells us that we can optimize either for data fidelity, e.g. low MSE or high PSNR, or for data realism, measured by perceptual metrics such as LPIPS or FID. Existing methods either prioritize fidelity at the expense of realism, or produce perceptually convincing results that lack quantitative accuracy. In this work, we propose HazeMatching, a novel iterative method for dehazing light microscopy images, which effectively balances these objectives. Our goal was to find a balanced trade-off between the fidelity of the dehazing results and the realism of individual predictions (samples). We achieve this by adapting the conditional flow matching framework by guiding the generative process with a hazy observation in the conditional velocity field. We evaluate HazeMatching on 5 datasets, covering both synthetic and real data, assessing both distortion and perceptual quality. Our method is compared against 11 baselines, achieving a consistent balance between fidelity and realism on average. Additionally, with calibration analysis, we show that HazeMatching produces well-calibrated predictions. Note that our method does not need an explicit degradation operator to exist, making it easily applicable on real microscopy data. All data used for training and evaluation and our code will be publicly available under a permissive license.
Authors Anirban Ray, Ashesh, Florian Jug
Comment 4 figures, 8 pages + refs, 45 pages total (including supplement), 28 supplementary figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialogue
Title CN WER并不知情：评估ASR错误如何扭曲患者对话中的临床理解
Keywords ASR评估 · 临床对话 · LLM-as-a-Judge · 语音识别错误 · 医疗AI
Summary 本文研究自动语音识别（ASR）在医生-患者对话中产生的转录错误对临床判断的影响，发现传统的词错误率（WER）等指标无法有效反映错误的临床严重性。为此，作者构建了一个由临床专家标注的数据集，并训练一个基于LLM的评判模型（Gemini-2.5-Pro）来自动化评估ASR错误的临床风险，取得了接近人类的一致性。然而，这项工作聚焦于医疗语音领域的评估范式改进，属于垂直应用场景。
Reason 该论文属于语音识别（ASR）在临床对话中的应用评估，核心关注的是语音转写错误对临床意义的影响，并提出用LLM作为评判者来替代人工评估。尽管使用了LLM，但其任务本质是特定领域（医疗语音）的下游应用评估框架，属于典型的‘平行应用’，未提出可用于文档图像理解的底层方法或架构创新。与文档VQA、布局分析、OCR-free等DIU核心任务无直接关联，且无法迁移解决OCR幻觉、复杂排版或细粒度定位等关键问题。
Abstract As Automatic Speech Recognition (ASR) is increasingly deployed in clinical dialogue, standard evaluations still rely heavily on Word Error Rate (WER). This paper challenges that standard, investigating whether WER or other common metrics correlate with the clinical impact of transcription errors. We establish a gold-standard benchmark by having expert clinicians compare ground-truth utterances to their ASR-generated counterparts, labeling the clinical impact of any discrepancies found in two distinct doctor-patient dialogue datasets. Our analysis reveals that WER and a comprehensive suite of existing metrics correlate poorly with the clinician-assigned risk labels (No, Minimal, or Significant Impact). To bridge this evaluation gap, we introduce an LLM-as-a-Judge, programmatically optimized using GEPA through DSPy to replicate expert clinical assessment. The optimized judge (Gemini-2.5-Pro) achieves human-comparable performance, obtaining 90% accuracy and a strong Cohen's $κ$ of 0.816. This work provides a validated, automated framework for moving ASR evaluation beyond simple textual fidelity to a necessary, scalable assessment of safety in clinical dialogue.
Authors Zachary Ellis, Jared Joselowitz, Yash Deo, Yajie He, Anna Kalygina, Aisling Higham, Mana Rahimzadeh, Yan Jia, Ibrahim Habli, Ernest Lim
Categories Artificial Intelligence, Computation and Language
0
MuM: Multi-View Masked Image Modeling for 3D Vision
Title CN MuM：用于3D视觉的多视图掩码图像建模
Keywords 自监督学习 · 掩码图像建模 · 3D视觉 · 多视图重建 · 视觉表示
Summary 本文提出MuM，一种基于多视图掩码图像建模的自监督学习方法，用于提升3D视觉任务中的特征表示能力。通过在多个视角间统一掩码并引入轻量级跨帧注意力解码器，实现了比CroCo和DINOv3更优的密集匹配与姿态估计性能。
Reason 该论文研究的是3D视觉中的自监督学习，聚焦于多视图几何理解与重建任务，属于纯计算机视觉领域，与文档图像理解（DIU）的核心问题如文本识别、布局分析、表格理解等无直接关联。其方法虽在3D视觉中有效，但难以迁移到解决OCR幻觉、复杂排版或长文档推理等DIU痛点。
Abstract Self-supervised learning on images seeks to extract meaningful visual representations from unlabeled data. When scaled to large datasets, this paradigm has achieved state-of-the-art performance and the resulting trained models such as DINOv3 have seen widespread adoption. However, most prior efforts are optimized for semantic understanding rather than geometric reasoning. One important exception is Cross-View Completion, CroCo, which is a form of masked autoencoding (MAE) tailored for 3D understanding. In this work, we continue on the path proposed by CroCo and focus on learning features tailored for 3D vision. In a nutshell, we extend MAE to arbitrarily many views of the same scene. By uniformly masking all views and employing a lightweight decoder with inter-frame attention, our approach is inherently simpler and more scalable than CroCo. We evaluate the resulting model, MuM, extensively on downstream tasks including feedforward reconstruction, dense image matching and relative pose estimation, finding that it outperforms the state-of-the-art visual encoders DINOv3 and CroCo v2.
Authors David Nordström, Johan Edstedt, Fredrik Kahl, Georg Bökman
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
ResMatching: Noise-Resilient Computational Super-Resolution via Guided Conditional Flow Matching
Title CN ResMatching：通过引导条件流匹配实现抗噪声的计算超分辨率
Keywords 计算超分辨率 · 流匹配 · 生物图像 · 噪声鲁棒性 · 不确定性估计
Summary 本文提出ResMatching，一种基于引导条件流匹配的计算超分辨率方法，用于从低分辨率且含噪的荧光显微镜图像中恢复高分辨率细节。该方法通过学习更强的数据先验，在保持数据保真度的同时提升感知真实性，并能提供像素级的不确定性估计以指导用户判断预测可靠性。实验表明其在多种生物结构上优于现有方法。
Reason 该论文研究的是荧光显微镜图像中的计算超分辨率（CSR）问题，属于生物医学图像处理领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其方法涉及高分辨率视觉表示和噪声鲁棒性，但应用场景和数据特性与文档图像差异巨大，迁移价值极低。此外，未提及任何与OCR、多模态语言模型或文档结构相关的内容，属于无关领域。
Abstract Computational Super-Resolution (CSR) in fluorescence microscopy has, despite being an ill-posed problem, a long history. At its very core, CSR is about finding a prior that can be used to extrapolate frequencies in a micrograph that have never been imaged by the image-generating microscope. It stands to reason that, with the advent of better data-driven machine learning techniques, stronger prior can be learned and hence CSR can lead to better results. Here, we present ResMatching, a novel CSR method that uses guided conditional flow matching to learn such improved data-priors. We evaluate ResMatching on 4 diverse biological structures from the BioSR dataset and compare its results against 7 baselines. ResMatching consistently achieves competitive results, demonstrating in all cases the best trade-off between data fidelity and perceptual realism. We observe that CSR using ResMatching is particularly effective in cases where a strong prior is hard to learn, e.g. when the given low-resolution images contain a lot of noise. Additionally, we show that ResMatching can be used to sample from an implicitly learned posterior distribution and that this distribution is calibrated for all tested use-cases, enabling our method to deliver a pixel-wise data-uncertainty term that can guide future users to reject uncertain predictions.
Authors Anirban Ray, Vera Galinova, Florian Jug
Comment 5 pages, 4 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Large Language Models for Sentiment Analysis to Detect Social Challenges: A Use Case with South African Languages Proceedings of The Southern African Conference on AI Research (SACAIR 2024)
Title CN 用于情感分析检测社会挑战的大型语言模型：以南非语言为例
Keywords 情感分析 · 大语言模型 · 零样本学习 · 南非语言 · 社会挑战检测
Summary 本文研究了多种主流大语言模型（如GPT-3.5、GPT-4、LlaMa 2等）在英语及两种南非语言（Sepedi、Setswana）社交媒体文本上的零样本情感分析表现，并探索了多模型结果融合对分类准确率的提升效果。实验表明，融合策略可将情感分类错误率降至1%以下，具备在多语言社会治理中快速识别社会问题的可行性。
Reason 该论文属于针对特定小语种（Sepedi、Setswana等南非语言）的社会媒体情感分析应用，属于典型的‘平行下游应用’，且聚焦于非中英文的小语种场景，完全落入负面清单中的‘小语种’和‘无方法论创新的应用型研究’。其技术路线仅为使用现有LLM进行零样本情感分类，并引入结果融合策略，未提出任何可用于文档图像理解（DIU）的新架构、新训练方法或抗幻觉、细粒度定位等底层机制，无法迁移至DocVQA或OCR相关任务。
Abstract Sentiment analysis can aid in understanding people's opinions and emotions on social issues. In multilingual communities sentiment analysis systems can be used to quickly identify social challenges in social media posts, enabling government departments to detect and address these issues more precisely and effectively. Recently, large-language models (LLMs) have become available to the wide public and initial analyses have shown that they exhibit magnificent zero-shot sentiment analysis abilities in English. However, there is no work that has investigated to leverage LLMs for sentiment analysis on social media posts in South African languages and detect social challenges. Consequently, in this work, we analyse the zero-shot performance of the state-of-the-art LLMs GPT-3.5, GPT-4, LlaMa 2, PaLM 2, and Dolly 2 to investigate the sentiment polarities of the 10 most emerging topics in English, Sepedi and Setswana social media posts that fall within the jurisdictional areas of 10 South African government departments. Our results demonstrate that there are big differences between the various LLMs, topics, and languages. In addition, we show that a fusion of the outcomes of different LLMs provides large gains in sentiment classification performance with sentiment classification errors below 1%. Consequently, it is now feasible to provide systems that generate reliable information about sentiment analysis to detect social challenges and draw conclusions about possible needs for actions on specific topics and within different language groups.
Authors Koena Ronny Mabokela, Tim Schlippe, Matthias Wölfel
Comment Published in the Proceedings of The Southern African Conference on AI Research (SACAIR 2024), Bloemfontein, South Africa, 2-6 December 2024. ISBN: 978-0-7961-6069-0
Categories Artificial Intelligence, Computation and Language
0
Where Culture Fades: Revealing the Cultural Gap in Text-to-Image Generation
Title CN 文化消退之处：揭示文本到图像生成中的文化差距
Keywords 文本到图像生成 · 文化一致性 · 多语言模型 · 神经元探测 · 表示激活
Summary 论文发现当前多语言文本到图像模型在生成结果中存在文化中立或英语偏见现象，问题根源在于文化相关表征未被充分激活。作者提出一种探测方法，定位到对文化敏感的神经元，并在此基础上设计了两种策略：推理时文化激活和层定向文化增强，在不微调主干网络的情况下提升了跨语言文化一致性。
Reason 该论文研究多语言文本到图像生成中的文化一致性问题，属于纯图像生成领域，且关注文化偏见而非文档结构、OCR幻觉或复杂排版理解。其方法聚焦于神经元层面的文化信号激活，与文档图像理解（DIU）的核心痛点无关，无法迁移用于解决DocVQA、表格识别或长文档推理等任务。
Abstract Multilingual text-to-image (T2I) models have advanced rapidly in terms of visual realism and semantic alignment, and are now widely utilized. Yet outputs vary across cultural contexts: because language carries cultural connotations, images synthesized from multilingual prompts should preserve cross-lingual cultural consistency. We conduct a comprehensive analysis showing that current T2I models often produce culturally neutral or English-biased results under multilingual prompts. Analyses of two representative models indicate that the issue stems not from missing cultural knowledge but from insufficient activation of culture-related representations. We propose a probing method that localizes culture-sensitive signals to a small set of neurons in a few fixed layers. Guided by this finding, we introduce two complementary alignment strategies: (1) inference-time cultural activation that amplifies the identified neurons without backbone fine-tuned; and (2) layer-targeted cultural enhancement that updates only culturally relevant layers. Experiments on our CultureBench demonstrate consistent improvements over strong baselines in cultural consistency while preserving fidelity and diversity.
Authors Chuancheng Shi, Shangze Li, Shiming Guo, Simiao Xie, Wenhua Wu, Jingtong Dou, Chao Wu, Canran Xiao, Cong Wang, Zifeng Cheng, Fei Shen, Tat-Seng Chua
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Platonic Representations for Poverty Mapping: Unified Vision-Language Codes or Agent-Induced Novelty?
Title CN 用于贫困制图的柏拉图表示：统一的视觉-语言编码还是智能体引发的新颖性？
Keywords 多模态学习 · 贫困制图 · 卫星图像 · LLM生成文本 · AI代理检索
Summary 本文提出一个融合卫星图像与LLM生成/代理检索文本的多模态框架，用于预测非洲地区的家庭财富水平。实验表明，结合视觉与语言信号能提升预测性能，且LLM内部知识优于外部检索文本。同时发现不同模态间存在部分表征收敛现象，并发布了大规模多模态数据集。
Reason 该论文研究利用卫星图像和互联网文本（由LLM生成或AI搜索代理检索）预测非洲社区的家庭财富水平，属于社会经济分析领域的多模态应用。其核心任务为回归预测（财富指数），而非文档图像理解（DIU）中的布局分析、表格识别、VIE等任务。尽管使用了多模态建模范式，但属于将现有VLM技术应用于特定下游场景（贫困映射），属于典型的‘平行下游应用’，未提出可迁移至文档理解领域的底层方法创新。
Abstract We investigate whether socio-economic indicators like household wealth leave recoverable imprints in satellite imagery (capturing physical features) and Internet-sourced text (reflecting historical/economic narratives). Using Demographic and Health Survey (DHS) data from African neighborhoods, we pair Landsat images with LLM-generated textual descriptions conditioned on location/year and text retrieved by an AI search agent from web sources. We develop a multimodal framework predicting household wealth (International Wealth Index) through five pipelines: (i) vision model on satellite images, (ii) LLM using only location/year, (iii) AI agent searching/synthesizing web text, (iv) joint image-text encoder, (v) ensemble of all signals. Our framework yields three contributions. First, fusing vision and agent/LLM text outperforms vision-only baselines in wealth prediction (e.g., R-squared of 0.77 vs. 0.63 on out-of-sample splits), with LLM-internal knowledge proving more effective than agent-retrieved text, improving robustness to out-of-country and out-of-time generalization. Second, we find partial representational convergence: fused embeddings from vision/language modalities correlate moderately (median cosine similarity of 0.60 after alignment), suggesting a shared latent code of material well-being while retaining complementary details, consistent with the Platonic Representation Hypothesis. Although LLM-only text outperforms agent-retrieved data, challenging our Agent-Induced Novelty Hypothesis, modest gains from combining agent data in some splits weakly support the notion that agent-gathered information introduces unique representational structures not fully captured by static LLM knowledge. Third, we release a large-scale multimodal dataset comprising more than 60,000 DHS clusters linked to satellite images, LLM-generated descriptions, and agent-retrieved texts.
Authors Satiyabooshan Murugaboopathy, Connor T. Jerzak, Adel Daoud
Comment 7 figures
Categories Artificial Intelligence
0
SweeperBot: Making 3D Browsing Accessible through View Analysis and Visual Question Answering International Journal of Human-Computer Interaction (IJHCI)
Title CN SweeperBot：通过视图分析和视觉问答使3D浏览可访问
Keywords 3D模型 · 视觉问答 · 可访问性 · 视障用户 · 人机交互
Summary 本文提出了SweeperBot系统，帮助屏幕阅读器用户通过视觉问答方式探索和比较3D模型。系统结合最优视图选择技术和生成式、识别式基础模型来回答用户的视觉问题，并通过专家评审和对照实验验证了描述质量。研究聚焦于提升3D内容的可访问性，而非解决文档理解中的技术挑战。
Reason 该论文属于具身智能与无障碍交互领域，核心任务是为视障用户通过VQA探索3D模型提供辅助。尽管使用了视觉问答技术，但其应用场景完全偏离文档图像理解（DIU）的核心问题，如OCR幻觉、复杂版面解析或长文档推理。3D模型浏览与文档结构无关，且未提出可用于文档领域的底层方法或架构创新。
Abstract Accessing 3D models remains challenging for Screen Reader (SR) users. While some existing 3D viewers allow creators to provide alternative text, they often lack sufficient detail about the 3D models. Grounded on a formative study, this paper introduces SweeperBot, a system that enables SR users to leverage visual question answering to explore and compare 3D models. SweeperBot answers SR users' visual questions by combining an optimal view selection technique with the strength of generative- and recognition-based foundation models. An expert review with 10 Blind and Low-Vision (BLV) users with SR experience demonstrated the feasibility of using SweeperBot to assist BLV users in exploring and comparing 3D models. The quality of the descriptions generated by SweeperBot was validated by a second survey study with 30 sighted participants.
Authors Chen Chen, Cuong Nguyen, Alexa Siu, Dingzeyu Li, Nadir Weibel
Comment 28 pages, 16 figures, this is an original manuscript of an article published by Taylor & Francis in the International Journal of Human-Computer Interaction (IJHCI), available online: https://doi.org/10.1080/10447318.2025.2594750
Categories Artificial Intelligence
0
Leveraging CVAE for Joint Configuration Estimation of Multifingered Grippers from Point Cloud Data
Title CN 利用CVAE从点云数据中联合估计多指抓手的配置
Keywords CVAE · 点云数据 · 多指抓手 · 关节配置估计 · 逆运动学
Summary 本文提出一种基于条件变分自编码器（CVAE）的方法，仅从多指抓手的点云数据中估计其关节配置。该方法在MultiDex抓取数据集上验证，使用Allegro Hand，运行时间仅为0.05毫秒，并达到与当前最先进方法相当的精度。相比传统逆运动学方法，该方法通过机器学习隐式地解决了中间指节位置判断的问题。
Reason 该论文研究的是机器人多指抓手的关节配置估计问题，属于具身智能与机器人控制领域，核心方法为基于点云数据的CVAE建模。尽管涉及视觉输入（点云）和机器学习，但其任务目标与文档图像理解（DIU）完全无关，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract This paper presents an efficient approach for determining the joint configuration of a multifingered gripper solely from the point cloud data of its poly-articulated chain, as generated by visual sensors, simulations or even generative neural networks. Well-known inverse kinematics (IK) techniques can provide mathematically exact solutions (when they exist) for joint configuration determination based solely on the fingertip pose, but often require post-hoc decision-making by considering the positions of all intermediate phalanges in the gripper's fingers, or rely on algorithms to numerically approximate solutions for more complex kinematics. In contrast, our method leverages machine learning to implicitly overcome these challenges. This is achieved through a Conditional Variational Auto-Encoder (CVAE), which takes point cloud data of key structural elements as input and reconstructs the corresponding joint configurations. We validate our approach on the MultiDex grasping dataset using the Allegro Hand, operating within 0.05 milliseconds and achieving accuracy comparable to state-of-the-art methods. This highlights the effectiveness of our pipeline for joint configuration estimation within the broader context of AI-driven techniques for grasp planning.
Authors Julien Merand, Boris Meden, Mathieu Grossard
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
LLM one-shot style transfer for Authorship Attribution and Verification
Title CN 基于LLM一次性风格迁移的作者归属与验证
Keywords 作者归属 · 风格迁移 · 大语言模型 · 计算文体学 · 无监督学习
Summary 本文提出一种基于大语言模型（LLM）的无监督方法，利用LLM在上下文学习中的能力，通过计算文本间风格可迁移性的log-probability来实现作者归属与验证。该方法避免了传统模型对主题与风格混淆的问题，在控制主题相关性后表现优于对比学习基线，并且性能随模型规模和测试时计算量增加而提升。
Reason 该论文属于纯文本NLP任务，研究作者归属与验证问题，使用LLM进行风格迁移和计算stylometry特征。其方法未涉及任何视觉或文档模态，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等完全无关。尽管使用了LLM和推理时计算扩展，但该技术路径难以迁移到处理OCR幻觉、复杂排版或高分辨率文档图像等DIU痛点。
Abstract Computational stylometry analyzes writing style through quantitative patterns in text, supporting applications from forensic tasks such as identity linking and plagiarism detection to literary attribution in the humanities. Supervised and contrastive approaches rely on data with spurious correlations and often confuse style with topic. Despite their natural use in AI-generated text detection, the CLM pre-training of modern LLMs has been scarcely leveraged for general authorship problems. We propose a novel unsupervised approach based on this extensive pre-training and the in-context learning capabilities of LLMs, employing the log-probabilities of an LLM to measure style transferability from one text to another. Our method significantly outperforms LLM prompting approaches of comparable scale and achieves higher accuracy than contrastively trained baselines when controlling for topical correlations. Moreover, performance scales fairly consistently with the size of the base model and, in the case of authorship verification, with an additional mechanism that increases test-time computation; enabling flexible trade-offs between computational cost and accuracy.
Authors Pablo Miralles-González, Javier Huertas-Tato, Alejandro Martín, David Camacho
Categories Artificial Intelligence, Computation and Language
0
Range-Edit: Semantic Mask Guided Outdoor LiDAR Scene Editing
Title CN Range-Edit：基于语义掩码引导的户外LiDAR场景编辑
Keywords LiDAR · 点云生成 · 语义编辑 · 扩散模型 · 自动驾驶
Summary 本文提出了一种基于语义掩码引导的户外LiDAR场景编辑方法Range-Edit，通过将点云投影为2D范围图像，并利用凸包生成的语义掩码指导扩散模型进行编辑，从而在真实LiDAR扫描基础上生成包含复杂边缘案例的合成点云。该方法在KITTI-360数据集上验证了其生成质量和几何一致性，适用于自动驾驶系统的训练与测试。
Reason 该论文研究的是自动驾驶中的LiDAR点云数据生成与编辑，属于3D视觉与具身智能领域，与文档图像理解（DIU）或DocVQA无直接关联。其方法聚焦于通过语义掩码编辑户外激光雷达场景，解决的是自动驾驶中的数据增强问题，无法迁移到文档理解的核心痛点如OCR幻觉、复杂排版解析或长文档推理等任务。
Abstract Training autonomous driving and navigation systems requires large and diverse point cloud datasets that capture complex edge case scenarios from various dynamic urban settings. Acquiring such diverse scenarios from real-world point cloud data, especially for critical edge cases, is challenging, which restricts system generalization and robustness. Current methods rely on simulating point cloud data within handcrafted 3D virtual environments, which is time-consuming, computationally expensive, and often fails to fully capture the complexity of real-world scenes. To address some of these issues, this research proposes a novel approach that addresses the problem discussed by editing real-world LiDAR scans using semantic mask-based guidance to generate novel synthetic LiDAR point clouds. We incorporate range image projection and semantic mask conditioning to achieve diffusion-based generation. Point clouds are transformed to 2D range view images, which are used as an intermediate representation to enable semantic editing using convex hull-based semantic masks. These masks guide the generation process by providing information on the dimensions, orientations, and locations of objects in the real environment, ensuring geometric consistency and realism. This approach demonstrates high-quality LiDAR point cloud generation, capable of producing complex edge cases and dynamic scenes, as validated on the KITTI-360 dataset. This offers a cost-effective and scalable solution for generating diverse LiDAR data, a step toward improving the robustness of autonomous driving systems.
Authors Suchetan G. Uppur, Hemant Kumar, Vaibhav Kumar
Comment 8 pages, 9 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Toward Super-polynomial Quantum Speedup of Equivariant Quantum Algorithms with SU($d$) Symmetry TQC 2022
Title CN 面向具有SU(d)对称性的等变量子算法的超多项式量子加速
Keywords 量子算法 · 等变性 · SU(d)对称性 · 量子机器学习 · PQC+
Summary 本文提出了一个适用于具有SU(d)对称性物理系统的等变卷积量子算法框架，并在此基础上构建了更强大的PQC+计算模型。研究表明，PQC+能够高效解决某些经典计算机难以在多项式时间内完成的问题，从而为超越经典可模拟性提供了证据。同时探讨了可在PQC+范式下实现的实际量子机器学习算法。
Reason 该论文属于量子计算与量子机器学习领域，研究的是具有SU(d)对称性的物理系统上的等变量子算法，并提出PQC+这一增强型量子计算模型。其核心问题设定、方法架构和应用场景均远离文档图像理解（DIU）或视觉语言模型（VLM）范畴。尽管涉及‘量子机器学习’，但无任何与OCR、文档布局分析、表格识别、多模态对齐或视觉幻觉相关的技术内容，无法迁移用于解决DIU中的核心痛点。
Abstract We introduce a framework of the equivariant convolutional quantum algorithms which is tailored for a number of machine-learning tasks on physical systems with arbitrary SU$(d)$ symmetries. It allows us to enhance a natural model of quantum computation -- permutational quantum computing (PQC) -- and define a more powerful model: PQC+. While PQC was shown to be efficiently classically simulatable, we exhibit a problem which can be efficiently solved on PQC+ machine, whereas no classical polynomial time algorithm is known; thus providing evidence against PQC+ being classically simulatable. We further discuss practical quantum machine learning algorithms which can be carried out in the paradigm of PQC+.
Authors Han Zheng, Zimu Li, Sergii Strelchuk, Risi Kondor, Junyu Liu
Comment Presented in TQC 2022
Categories Artificial Intelligence
0
Defending the Edge: Representative-Attention Defense against Backdoor Attacks in Federated Learning IEEE EURO S&P 2026
Title CN 保护边缘：基于代表性注意力的联邦学习后门攻击防御
Keywords 联邦学习 · 后门攻击 · 注意力机制 · 异常检测 · 表示空间
Summary 本文提出FeRA，一种基于注意力机制的联邦学习后门防御框架，通过分析客户端更新在表示空间中的一致性与方差特性来识别恶意客户端。方法聚焦于安全方向的行为一致性检测，不涉及文档理解或多模态建模。
Reason 该论文研究联邦学习中的后门攻击防御，属于安全与隐私领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的应用无直接关联。未涉及OCR、布局分析、表格识别、多模态推理等DIU核心任务，也未提出可迁移至DIU领域的底层方法（如视觉表示、高分辨率处理、幻觉抑制等）。属于无关领域。
Abstract Federated learning (FL) remains highly vulnerable to adaptive backdoor attacks that preserve stealth by closely imitating benign update statistics. Existing defenses predominantly rely on anomaly detection in parameter or gradient space, overlooking behavioral constraints that backdoor attacks must satisfy to ensure reliable trigger activation. These anomaly-centric methods fail against adaptive attacks that normalize update magnitudes and mimic benign statistical patterns while preserving backdoor functionality, creating a fundamental detection gap. To address this limitation, this paper introduces FeRA (Federated Representative Attention) -- a novel attention-driven defense that shifts the detection paradigm from anomaly-centric to consistency-centric analysis. FeRA exploits the intrinsic need for backdoor persistence across training rounds, identifying malicious clients through suppressed representation-space variance, an orthogonal property to traditional magnitude-based statistics. The framework conducts multi-dimensional behavioral analysis combining spectral and spatial attention, directional alignment, mutual similarity, and norm inflation across two complementary detection mechanisms: consistency analysis and norm-inflation detection. Through this mechanism, FeRA isolates malicious clients that exhibit low-variance consistency or magnitude amplification. Extensive evaluation across six datasets, nine attacks, and three model architectures under both Independent and Identically Distributed (IID) and non-IID settings confirm FeRA achieves superior backdoor mitigation. Under different non-IID settings, FeRA achieved the lowest average Backdoor Accuracy (BA), about 1.67% while maintaining high clean accuracy compared to other state-of-the-art defenses. The code is available at https://github.com/Peatech/FeRA_defense.git.
Authors Chibueze Peace Obioma, Youcheng Sun, Mustafa A. Mustafa
Comment Submitted to IEEE EURO S&P 2026
Categories Artificial Intelligence
0
DISCA: A Digital In-memory Stochastic Computing Architecture Using A Compressed Bent-Pyramid Format
Title CN DISCA：一种采用压缩弯曲金字塔格式的数字存内随机计算架构
Keywords 存内计算 · 随机计算 · 能效优化 · 矩阵乘法 · 边缘AI
Summary 本文提出了一种名为DISCA的数字存内随机计算架构，采用压缩的弯曲金字塔数据格式，旨在提升大规模AI模型中矩阵乘法的能效。该架构结合了模拟计算的简单性与数字系统的可靠性，在180nm CMOS工艺下实现了每比特3.59 TOPS/W的能量效率，显著优于现有架构。然而，该工作属于专用硬件设计，并未涉及文档理解或多模态语义建模。
Reason 该论文属于硬件架构与存内计算领域，聚焦于提升AI矩阵乘法的能效，尽管其技术可能间接服务于边缘AI部署，但并未提出任何针对文档图像理解（DIU）任务的方法论创新。既不涉及VLM、多模态建模、视觉表示学习，也不解决OCR幻觉、复杂排版或长文档推理等DIU核心问题。属于无关领域的底层硬件优化，无法直接迁移为DIU的方法工具。
Abstract Nowadays, we are witnessing an Artificial Intelligence revolution that dominates the technology landscape in various application domains, such as healthcare, robotics, automotive, security, and defense. Massive-scale AI models, which mimic the human brain's functionality, typically feature millions and even billions of parameters through data-intensive matrix multiplication tasks. While conventional Von-Neumann architectures struggle with the memory wall and the end of Moore's Law, these AI applications are migrating rapidly towards the edge, such as in robotics and unmanned aerial vehicles for surveillance, thereby adding more constraints to the hardware budget of AI architectures at the edge. Although in-memory computing has been proposed as a promising solution for the memory wall, both analog and digital in-memory computing architectures suffer from substantial degradation of the proposed benefits due to various design limitations. We propose a new digital in-memory stochastic computing architecture, DISCA, utilizing a compressed version of the quasi-stochastic Bent-Pyramid data format. DISCA inherits the same computational simplicity of analog computing, while preserving the same scalability, productivity, and reliability of digital systems. Post-layout modeling results of DISCA show an energy efficiency of 3.59 TOPS/W per bit at 500 MHz using a commercial 180nm CMOS technology. Therefore, DISCA significantly improves the energy efficiency for matrix multiplication workloads by orders of magnitude if scaled and compared to its counterpart architectures.
Authors Shady Agwa, Yikang Shen, Shiwei Wang, Themis Prodromakis
Comment 6 pages, 5 figures
Categories Artificial Intelligence
0
Algorithmic design and implementation considerations of deep MPC
Title CN 深度模型预测控制的算法设计与实现考虑
Keywords 深度模型预测控制 · 控制权分配 · 神经网络 · 系统稳定性 · 在线学习
Summary 本文探讨了深度模型预测控制（Deep MPC）中神经网络与MPC控制器之间的控制权分配问题，分析了实现过程中的算法挑战，并通过数值实验说明不当的控制分配会导致性能下降。重点在于系统安全性与在线学习过程中的稳定性，属于控制理论与机器人系统的交叉领域。
Reason 该论文属于具身智能与控制领域的研究，聚焦于深度神经网络与模型预测控制（MPC）在控制系统中的协同设计，涉及机器人动力学（如四轮滑移转向系统）。其内容与文档图像理解（DIU）、多模态大模型（VLM）或视觉语言任务无直接关联，无法迁移用于解决OCR幻觉、复杂排版解析、长文档推理等DIU核心问题。
Abstract Deep Model Predictive Control (Deep MPC) is an evolving field that integrates model predictive control and deep learning. This manuscript is focused on a particular approach, which employs deep neural network in the loop with MPC. This class of approaches distributes control authority between a neural network and an MPC controller, in such a way that the neural network learns the model uncertainties while the MPC handles constraints. The approach is appealing because training data collected while the system is in operation can be used to fine-tune the neural network, and MPC prevents unsafe behavior during those learning transients. This manuscript explains implementation challenges of Deep MPC, algorithmic way to distribute control authority and argues that a poor choice in distributing control authority may lead to poor performance. A reason of poor performance is explained through a numerical experiment on a four-wheeled skid-steer dynamics.
Authors Prabhat K. Mishra, Mateus V. Gasparino, Girish Chowdhary
Categories Artificial Intelligence
0
TP-MDDN: Task-Preferenced Multi-Demand-Driven Navigation with Autonomous Decision-Making NeurIPS 2025
Title CN TP-MDDN：基于任务偏好的多需求驱动导航与自主决策
Keywords 具身智能 · 导航 · 多需求决策 · LLM代理 · 空间记忆
Summary 本文提出了TP-MDDN，一个用于多需求驱动导航的新基准，以及AWMSystem这一自主决策系统，包含指令分解、目标选择和任务监控模块。通过结合3D点云与2D语义地图构建MASMap实现空间记忆，并采用双节奏动作生成框架和自适应错误纠正机制提升导航鲁棒性。实验表明该方法在感知准确性和导航稳定性上优于现有方法。
Reason 该论文属于具身智能（Embodied AI）领域，研究的是机器人在物理空间中根据多需求进行导航的决策问题。尽管使用了LLM模块和视觉感知组件，但其核心任务与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点。
Abstract In daily life, people often move through spaces to find objects that meet their needs, posing a key challenge in embodied AI. Traditional Demand-Driven Navigation (DDN) handles one need at a time but does not reflect the complexity of real-world tasks involving multiple needs and personal choices. To bridge this gap, we introduce Task-Preferenced Multi-Demand-Driven Navigation (TP-MDDN), a new benchmark for long-horizon navigation involving multiple sub-demands with explicit task preferences. To solve TP-MDDN, we propose AWMSystem, an autonomous decision-making system composed of three key modules: BreakLLM (instruction decomposition), LocateLLM (goal selection), and StatusMLLM (task monitoring). For spatial memory, we design MASMap, which combines 3D point cloud accumulation with 2D semantic mapping for accurate and efficient environmental understanding. Our Dual-Tempo action generation framework integrates zero-shot planning with policy-based fine control, and is further supported by an Adaptive Error Corrector that handles failure cases in real time. Experiments demonstrate that our approach outperforms state-of-the-art baselines in both perception accuracy and navigation robustness.
Authors Shanshan Li, Da Huang, Yu He, Yanwei Fu, Yu-Gang Jiang, Xiangyang Xue
Comment Accepted at NeurIPS 2025
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Wideband RF Radiance Field Modeling Using Frequency-embedded 3D Gaussian Splatting
Title CN 基于频率嵌入3D高斯点阵的宽带射频辐射场建模
Keywords 射频建模 · 3D高斯点阵 · 宽带信号 · 电磁仿真 · 频率嵌入
Summary 本文提出一种基于频率嵌入3D高斯点阵的宽带射频辐射场建模方法，通过引入频率相关的电磁特征网络，在多个频段下重建室内环境中的射频辐射场。实验基于一个跨六种室内环境、覆盖1至94GHz的大规模功率角谱（PAS）数据集，结果表明该方法在SSIM指标上显著优于单频3DGS模型。
Reason 该论文研究的是射频（RF）信号在三维空间中的传播建模，属于无线通信与电磁场仿真领域，与文档图像理解（DIU）、多模态大模型、视觉文本识别等完全无关。尽管使用了3DGS等视觉相关技术，但其任务目标、数据模态和应用场景均远离文档理解核心问题，无法迁移用于解决OCR幻觉、复杂排版或长文档推理等DIU痛点。
Abstract Indoor environments typically contain diverse RF signals distributed across multiple frequency bands, including NB-IoT, Wi-Fi, and millimeter-wave. Consequently, wideband RF modeling is essential for practical applications such as joint deployment of heterogeneous RF systems, cross-band communication, and distributed RF sensing. Although 3D Gaussian Splatting (3DGS) techniques effectively reconstruct RF radiance fields at a single frequency, they cannot model fields at arbitrary or unknown frequencies across a wide range. In this paper, we present a novel 3DGS algorithm for unified wideband RF radiance field modeling. RF wave propagation depends on signal frequency and the 3D spatial environment, including geometry and material electromagnetic (EM) properties. To address these factors, we introduce a frequency-embedded EM feature network that utilizes 3D Gaussian spheres at each spatial location to learn the relationship between frequency and transmission characteristics, such as attenuation and radiance intensity. With a dataset containing sparse frequency samples in a specific 3D environment, our model can efficiently reconstruct RF radiance fields at arbitrary and unseen frequencies. To assess our approach, we introduce a large-scale power angular spectrum (PAS) dataset with 50,000 samples spanning 1 to 94 GHz across six indoor environments. Experimental results show that the proposed model trained on multiple frequencies achieves a Structural Similarity Index Measure (SSIM) of 0.922 for PAS reconstruction, surpassing state-of-the-art single-frequency 3DGS models with SSIM of 0.863.
Authors Zechen Li, Lanqing Yang, Yiheng Bian, Hao Pan, Yongjian Fu, Yezhou Wang, Zhuxi Chen, Yi-Chao Chen, Guangtao Xue
Categories Artificial Intelligence
0
Parrot: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs
Title CN Parrot：输出真实性的说服与一致性鲁棒性评分——一种针对大语言模型谄媚行为的鲁棒性基准
Keywords 大语言模型 · 安全性 · 对齐 · 幻觉 · 认知偏差
Summary 本文提出了PARROT框架，用于衡量大语言模型在面对权威性误导时保持正确答案的能力，即抵抗‘谄媚’倾向的鲁棒性。通过对比中立问题与带有权威偏见的问题回答，追踪模型置信度变化，并分类其行为模式。实验覆盖22个模型和13个领域，发现先进模型更具抵抗力，而较小模型易发生‘知识崩溃’。研究强调应将抗压能力作为安全部署的关键指标之一。
Reason 该论文聚焦于评估大语言模型在社会压力（如权威影响）下的谄媚行为（sycophancy）和鲁棒性，属于纯NLP领域的安全与对齐研究。尽管涉及模型幻觉相关现象，但其核心是语言模态内的认知偏差与社会工程攻击响应，未涉及视觉输入、文档结构、OCR或任何多模态文档理解任务。无法迁移用于解决DIU中的OCR幻觉、排版理解或视觉-文本对齐等核心问题。
Abstract This study presents PARROT (Persuasion and Agreement Robustness Rating of Output Truth), a robustness focused framework designed to measure the degradation in accuracy that occurs under social pressure exerted on users through authority and persuasion in large language models (LLMs) the phenomenon of sycophancy (excessive conformity). PARROT (i) isolates causal effects by comparing the neutral version of the same question with an authoritatively false version using a double-blind evaluation, (ii) quantifies confidence shifts toward the correct and imposed false responses using log-likelihood-based calibration tracking, and (iii) systematically classifies failure modes (e.g., robust correct, sycophantic agreement, reinforced error, stubborn error, self-correction, etc.) using an eight-state behavioral taxonomy. We evaluated 22 models using 1,302 MMLU-style multiple-choice questions across 13 domains and domain-specific authority templates. Findings show marked heterogeneity: advanced models (e.g., GPT-5, GPT-4.1, Claude Sonnet 4.5) exhibit low "follow rates" ($\leq 11\%$, GPT-5: 4\%) and minimal accuracy loss, while older/smaller models show severe epistemic collapse (GPT-4: 80\%, Qwen 2.5-1.5B: 94\%). The danger is not limited to response changes; weak models reduce confidence in the correct response while increasing confidence in the imposed incorrect response. While international law and global knowledge at the domain level exhibit high fragility, elementary mathematics is relatively resilient. Consequently, we argue that the goal of "resistance to overfitting pressure" should be addressed as a primary objective alongside accuracy, harm avoidance, and privacy for safe deployment in the real world.
Authors Yusuf Çelebi, Mahmoud El Hussieni, Özay Ezerceli
Categories Artificial Intelligence, Computation and Language
0
CleverDistiller: Simple and Spatially Consistent Cross-modal Distillation BMVC 2025
Title CN CleverDistiller：简单且空间一致的跨模态蒸馏
Keywords 知识蒸馏 · 跨模态学习 · 3D感知 · 自监督学习 · 自动驾驶
Summary 本文提出CleverDistiller，一种用于2D到3D跨模态知识蒸馏的自监督框架，通过简化蒸馏损失并引入MLP投影头和占用预测辅助任务，在低数据量下显著提升了3D语义分割和目标检测性能。
Reason 该论文研究的是2D图像到3D点云的跨模态知识蒸馏，应用于自动驾驶中的语义分割和3D目标检测，属于具身智能与3D视觉领域。其核心问题与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract Vision foundation models (VFMs) such as DINO have led to a paradigm shift in 2D camera-based perception towards extracting generalized features to support many downstream tasks. Recent works introduce self-supervised cross-modal knowledge distillation (KD) as a way to transfer these powerful generalization capabilities into 3D LiDAR-based models. However, they either rely on highly complex distillation losses, pseudo-semantic maps, or limit KD to features useful for semantic segmentation only. In this work, we propose CleverDistiller, a self-supervised, cross-modal 2D-to-3D KD framework introducing a set of simple yet effective design choices: Unlike contrastive approaches relying on complex loss design choices, our method employs a direct feature similarity loss in combination with a multi layer perceptron (MLP) projection head to allow the 3D network to learn complex semantic dependencies throughout the projection. Crucially, our approach does not depend on pseudo-semantic maps, allowing for direct knowledge transfer from a VFM without explicit semantic supervision. Additionally, we introduce the auxiliary self-supervised spatial task of occupancy prediction to enhance the semantic knowledge, obtained from a VFM through KD, with 3D spatial reasoning capabilities. Experiments on standard autonomous driving benchmarks for 2D-to-3D KD demonstrate that CleverDistiller achieves state-of-the-art performance in both semantic segmentation and 3D object detection (3DOD) by up to 10% mIoU, especially when fine tuning on really low data amounts, showing the effectiveness of our simple yet powerful KD strategy
Authors Hariprasath Govindarajan, Maciej K. Wozniak, Marvin Klingner, Camille Maurice, B Ravi Kiran, Senthil Yogamani
Comment Accepted to BMVC 2025
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Bootstrap Off-policy with World Model NeurIPS 2025
Title CN 使用世界模型进行引导式离策略学习
Keywords 强化学习 · 世界模型 · 离策略学习 · 在线规划 · 行为对齐
Summary 本文提出BOOM框架，通过将规划器与策略网络结合，并利用联合训练的世界模型进行轨迹模拟和价值估计，实现策略与模型之间的行为对齐。该方法采用无似然对齐损失和软值加权机制，在DeepMind Control Suite和Humanoid-Bench上取得了最优性能。
Reason 该论文属于强化学习领域，研究在线规划与离策略学习的结合，核心问题是RL中的样本效率与策略优化。尽管其方法在高维控制任务中表现优异，但完全不涉及多模态建模、视觉输入处理或文档理解任务。其提出的BOOM框架依赖于动态环境建模与动作序列优化，迁移到文档图像理解（如VQA、布局分析）场景极为困难，无法解决OCR幻觉、细粒度定位或复杂排版解析等DIU核心痛点。
Abstract Online planning has proven effective in reinforcement learning (RL) for improving sample efficiency and final performance. However, using planning for environment interaction inevitably introduces a divergence between the collected data and the policy's actual behaviors, degrading both model learning and policy improvement. To address this, we propose BOOM (Bootstrap Off-policy with WOrld Model), a framework that tightly integrates planning and off-policy learning through a bootstrap loop: the policy initializes the planner, and the planner refines actions to bootstrap the policy through behavior alignment. This loop is supported by a jointly learned world model, which enables the planner to simulate future trajectories and provides value targets to facilitate policy improvement. The core of BOOM is a likelihood-free alignment loss that bootstraps the policy using the planner's non-parametric action distribution, combined with a soft value-weighted mechanism that prioritizes high-return behaviors and mitigates variability in the planner's action quality within the replay buffer. Experiments on the high-dimensional DeepMind Control Suite and Humanoid-Bench show that BOOM achieves state-of-the-art results in both training stability and final performance. The code is accessible at https://github.com/molumitu/BOOM_MBRL.
Authors Guojian Zhan, Likun Wang, Xiangteng Zhang, Jiaxin Gao, Masayoshi Tomizuka, Shengbo Eben Li
Comment NeurIPS 2025
Categories Artificial Intelligence
0
VSI: Visual Subtitle Integration for Keyframe Selection to enhance Long Video Understanding
Title CN VSI：通过视觉字幕融合进行关键帧选择以增强长视频理解
Keywords 视频理解 · 关键帧选取 · 多模态融合 · 字幕匹配 · MLLM
Summary 本文提出VSI（Visual-Subtitle Integration）框架，通过结合视频搜索和字幕匹配的双分支方法，在长视频中实现更精准的关键帧选取。该方法融合视觉与文本信息，提升了在文本相关任务上的表现，并在多个视频理解基准上达到SOTA。然而，其应用背景为视频而非文档图像，技术路径难以迁移到文档理解场景。
Reason 该论文研究的是长视频理解中的关键帧选取问题，属于视频理解领域，其核心任务与文档图像理解（DIU）无关。尽管提出了多模态信息融合的检索框架，但应用场景局限于视频+字幕的时序数据，无法直接迁移至解决文档图像中的OCR幻觉、复杂排版或细粒度文本定位等核心痛点。属于‘无关领域’，命中负面清单第2条。
Abstract Multimodal large language models (MLLMs) demonstrate exceptional performance in vision-language tasks, yet their processing of long videos is constrained by input context length and high computational costs. Sparse frame sampling thus becomes a necessary preprocessing step, with sampled frame quality directly impacting downstream performance. Existing keyframe search algorithms achieve a balance between efficiency and sampled frame quality but heavily rely on the visual modality alone. This makes them difficult to adapt to text-related tasks and often leads to retrieval results deviating from core semantic content. To address this, we propose the VISUAL-SUBTITLE INTEGRATION (VSI), a multimodal keyframe retrieval framework. It employs a dual-branch collaborative retrieval approach combining Video Search and Subtitle Match to fuse complementary visual and textual information for precise localization. Experiments on LongVideoBench and VideoMME demonstrate that VSI achieves state-of-the-art accuracy in keyframe retrieval while delivering breakthrough performance in text-related tasks and exhibiting strong generalization across other tasks.
Authors Jianxiang He, Meisheng Hong, Jungang Li, Ziyang Chen, Weiyu Guo, Xuming Hu, Hui Xiong
Comment 9 pages,3 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assembly
Title CN PhyBlock：通过3D积木组装实现物理理解与规划的渐进式基准
Keywords VLM · 物理理解 · 3D块组装 · 空间推理 · 具身智能
Summary PhyBlock是一个面向3D积木组装任务的渐进式基准，旨在评估视觉语言模型在物理理解与多步规划方面的能力。它包含四层认知层次的任务设计和针对性的VQA样本，用于测试模型在部分完成、失败诊断和规划鲁棒性等方面的表现。实验表明现有VLM在空间定向和依赖关系推理上存在显著缺陷，且思维链提示效果有限。
Reason 该论文聚焦于具身智能中的3D物理场景理解与机器人任务规划，属于具身智能/机器人领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管涉及VLM在复杂空间推理中的应用，但其目标场景为3D块状结构装配，迁移到文档理解中的OCR幻觉、密集文本处理或复杂排版解析极为困难，且未提供可用于文档视觉表示或高分辨率处理的方法论创新。
Abstract While vision-language models (VLMs) have demonstrated promising capabilities in reasoning and planning for embodied agents, their ability to comprehend physical phenomena, particularly within structured 3D environments, remains severely limited. To close this gap, we introduce PhyBlock, a progressive benchmark designed to assess VLMs on physical understanding and planning through robotic 3D block assembly tasks. PhyBlock integrates a novel four-level cognitive hierarchy assembly task alongside targeted Visual Question Answering (VQA) samples, collectively aimed at evaluating progressive spatial reasoning and fundamental physical comprehension, including object properties, spatial relationships, and holistic scene understanding. PhyBlock includes 2600 block tasks (400 assembly tasks, 2200 VQA tasks) and evaluates models across three key dimensions: partial completion, failure diagnosis, and planning robustness. We benchmark 21 state-of-the-art VLMs, highlighting their strengths and limitations in physically grounded, multi-step planning. Our empirical findings indicate that the performance of VLMs exhibits pronounced limitations in high-level planning and reasoning capabilities, leading to a notable decline in performance for the growing complexity of the tasks. Error analysis reveals persistent difficulties in spatial orientation and dependency reasoning. Surprisingly, chain-of-thought prompting offers minimal improvements, suggesting spatial tasks heavily rely on intuitive model comprehension. We position PhyBlock as a unified testbed to advance embodied reasoning, bridging vision-language understanding and real-world physical problem-solving.
Authors Liang Ma, Jiajun Wen, Min Lin, Rongtao Xu, Xiwen Liang, Bingqian Lin, Jun Ma, Yongxin Wang, Ziming Wei, Haokun Lin, Mingfei Han, Meng Cao, Bokui Chen, Ivan Laptev, Xiaodan Liang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Attention-Guided Feature Fusion (AGFF) Model for Integrating Statistical and Semantic Features in News Text Classification
Title CN 用于新闻文本分类中统计与语义特征融合的注意力引导特征融合（AGFF）模型
Keywords 文本分类 · 注意力机制 · 特征融合 · 统计特征 · 语义特征
Summary 本文提出了一种注意力引导的特征融合模型（AGFF），用于在新闻文本分类中结合传统的统计特征（如TF-IDF）和深度学习生成的语义特征。通过注意力机制动态调整两类特征的权重，提升了分类性能。实验表明该方法优于仅使用统计或语义特征的模型，并通过消融研究验证了各组件的有效性。
Reason 该论文属于纯文本分类任务，研究的是新闻文本中统计特征与语义特征的融合方法，未涉及任何视觉或文档布局信息，也不包含多模态、OCR、表格识别、视觉问答等文档图像理解（DIU）核心任务。尽管其注意力机制有一定通用性，但整体工作为NLP内部的特征融合方法，且应用场景为标准文本分类，无法迁移解决DIU中的OCR幻觉、复杂排版或细粒度定位等关键问题。
Abstract News text classification is a crucial task in natural language processing, essential for organizing and filtering the massive volume of digital content. Traditional methods typically rely on statistical features like term frequencies or TF-IDF values, which are effective at capturing word-level importance but often fail to reflect contextual meaning. In contrast, modern deep learning approaches utilize semantic features to understand word usage within context, yet they may overlook simple, high-impact statistical indicators. This paper introduces an Attention-Guided Feature Fusion (AGFF) model that combines statistical and semantic features in a unified framework. The model applies an attention-based mechanism to dynamically determine the relative importance of each feature type, enabling more informed classification decisions. Through evaluation on benchmark news datasets, the AGFF model demonstrates superior performance compared to both traditional statistical models and purely semantic deep learning models. The results confirm that strategic integration of diverse feature types can significantly enhance classification accuracy. Additionally, ablation studies validate the contribution of each component in the fusion process. The findings highlight the model's ability to balance and exploit the complementary strengths of statistical and semantic representations, making it a practical and effective solution for real-world news classification tasks.
Authors Mohammad Zare
Categories Artificial Intelligence, Computation and Language
0
Model Inversion Attack Against Deep Hashing
Title CN 针对深度哈希的模型反演攻击
Keywords 模型反演攻击 · 深度哈希 · 隐私泄露 · 扩散模型 · 生物特征安全
Summary 本文提出了DHMI，首个面向深度哈希模型的扩散式模型反演框架，通过聚类辅助数据生成语义哈希中心，并设计了一种代理引导的去噪优化方法，在无真实训练哈希码的情况下成功重建高分辨率图像。实验表明该方法在黑盒场景下优于现有技术，揭示了深哈希系统中的严重隐私隐患。
Reason 该论文研究的是深度哈希模型中的隐私安全问题，属于模型反演攻击（Model Inversion Attack）领域，主要关注从哈希码中重建原始训练数据，以揭示生物特征伪造等隐私风险。其核心任务与文档图像理解（DIU）无关，既不涉及DocVQA、布局分析、表格识别等DIU任务，也未提出可用于解决OCR幻觉、复杂排版或长文档推理等问题的方法论。尽管使用了扩散模型和视觉生成技术，但目标是安全与隐私评估，而非提升文档理解性能，属于无关领域。
Abstract Deep hashing improves retrieval efficiency through compact binary codes, yet it introduces severe and often overlooked privacy risks. The ability to reconstruct original training data from hash codes could lead to serious threats such as biometric forgery and privacy breaches. However, model inversion attacks specifically targeting deep hashing models remain unexplored, leaving their security implications unexamined. This research gap stems from the inaccessibility of genuine training hash codes and the highly discrete Hamming space, which prevents existing methods from adapting to deep hashing. To address these challenges, we propose DHMI, the first diffusion-based model inversion framework designed for deep hashing. DHMI first clusters an auxiliary dataset to derive semantic hash centers as surrogate anchors. It then introduces a surrogate-guided denoising optimization method that leverages a novel attack metric (fusing classification consistency and hash proximity) to dynamically select candidate samples. A cluster of surrogate models guides the refinement of these candidates, ensuring the generation of high-fidelity and semantically consistent images. Experiments on multiple datasets demonstrate that DHMI successfully reconstructs high-resolution, high-quality images even under the most challenging black-box setting, where no training hash codes are available. Our method outperforms the existing state-of-the-art model inversion attacks in black-box scenarios, confirming both its practical efficacy and the critical privacy risks inherent in deep hashing systems.
Authors Dongdong Zhao, Qiben Xu, Ranxin Fang, Baogang Song
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
MIR: Efficient Exploration in Episodic Multi-Agent Reinforcement Learning via Mutual Intrinsic Reward
Title CN MIR：通过互惠内在奖励实现高效探索的片段式多智能体强化学习
Keywords 多智能体强化学习 · 稀疏奖励 · 内在奖励 · 探索效率 · MiniGrid-MA
Summary 本文提出了一种名为互惠内在奖励（MIR）的方法，用于改善具有稀疏奖励的多智能体强化学习中的探索效率。MIR通过激励个体智能体采取能影响队友的动作来促进团队整体探索，并在新构建的MiniGrid-MA环境中验证了其有效性。然而，该工作属于典型的具身智能与多智能体系统领域，不涉及任何视觉文档理解任务。
Reason 该论文研究的是多智能体强化学习（MARL）中的稀疏奖励问题，属于具身智能与机器人决策控制领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其方法聚焦于智能体间的协同探索机制，无法迁移解决OCR幻觉、复杂排版理解、细粒度定位或长文档推理等DIU核心痛点。
Abstract Episodic rewards present a significant challenge in reinforcement learning. While intrinsic reward methods have demonstrated effectiveness in single-agent rein-forcement learning scenarios, their application to multi-agent reinforcement learn-ing (MARL) remains problematic. The primary difficulties stem from two fac-tors: (1) the exponential sparsity of joint action trajectories that lead to rewards as the exploration space expands, and (2) existing methods often fail to account for joint actions that can influence team states. To address these challenges, this paper introduces Mutual Intrinsic Reward (MIR), a simple yet effective enhancement strategy for MARL with extremely sparse rewards like episodic rewards. MIR incentivizes individual agents to explore actions that affect their teammates, and when combined with original strategies, effectively stimulates team exploration and improves algorithm performance. For comprehensive experimental valida-tion, we extend the representative single-agent MiniGrid environment to create MiniGrid-MA, a series of MARL environments with sparse rewards. Our evalu-ation compares the proposed method against state-of-the-art approaches in the MiniGrid-MA setting, with experimental results demonstrating superior perfor-mance.
Authors Kesheng Chen, Wenjian Luo, Bang Zhang, Zeping Yin, Zipeng Ye
Categories Artificial Intelligence
0
The PLLuM Instruction Corpus
Title CN PLLuM指令语料库
Keywords 指令数据集 · 大语言模型 · 波兰语 · 语言适应 · 合成数据
Summary 本文介绍了PLLuM项目中用于微调Transformer架构大语言模型的指令数据集，提出了一种针对原生、转换和合成指令的功能分类法，并讨论了人工编写与合成指令数据在语言适配中的影响。作者还发布了该指令语料库的第一个代表性子集PLLuMIC，旨在为其他LLM的数据集开发提供参考。
Reason 该论文聚焦于构建波兰语大语言模型的指令数据集，属于特定小语种（非中英）的语言模型训练工作，且未涉及任何文档图像、多模态或视觉内容处理。完全偏离文档图像理解（DIU）核心任务，属于负面清单中的‘小语种’类别，无迁移价值。
Abstract This paper describes the instruction dataset used to fine-tune a set of transformer-based large language models (LLMs) developed in the PLLuM (Polish Large Language Model) project. We present a functional typology of the organic, converted, and synthetic instructions used in PLLuM and share some observations about the implications of using human-authored versus synthetic instruction datasets in the linguistic adaptation of base LLMs. Additionally, we release the first representative subset of the PLLuM instruction corpus (PLLuMIC), which we believe to be useful in guiding and planning the development of similar datasets for other LLMs.
Authors Piotr Pęzik, Filip Żarnecki, Konrad Kaczyński, Anna Cichosz, Zuzanna Deckert, Monika Garnys, Izabela Grabarczyk, Wojciech Janowski, Sylwia Karasińska, Aleksandra Kujawiak, Piotr Misztela, Maria Szymańska, Karolina Walkusz, Igor Siek, Maciej Chrabąszcz, Anna Kołos, Agnieszka Karlińska, Karolina Seweryn, Aleksandra Krasnodębska, Paula Betscher, Zofia Cieślińska, Katarzyna Kowol, Artur Wilczek, Maciej Trzciński, Katarzyna Dziewulska, Roman Roszko, Tomasz Bernaś, Jurgita Vaičenonienė, Danuta Roszko, Paweł Levchuk, Paweł Kowalski, Irena Prawdzic-Jankowska, Marek Kozłowski, Sławomir Dadas, Rafał Poświata, Alina Wróblewska, Katarzyna Krasnowska-Kieraś, Maciej Ogrodniczuk, Michał Rudolf, Piotr Rybak, Karolina Saputa, Joanna Wołoszyn, Marcin Oleksy, Bartłomiej Koptyra, Teddy Ferdinan, Stanisław Woźniak, Maciej Piasecki, Paweł Walkowiak, Konrad Wojtasik, Arkadiusz Janz, Przemysław Kazienko, Julia Moska, Jan Kocoń
Categories Artificial Intelligence, Computation and Language
0
A lightweight detector for real-time detection of remote sensing images
Title CN 一种用于遥感图像实时检测的轻量级检测器
Keywords 遥感图像 · 小目标检测 · 轻量级检测器 · YOLO · 多尺度融合
Summary 本文提出了一种名为DMG-YOLO的轻量级实时检测器，用于提升遥感图像中小目标的检测性能。通过设计双分支特征提取模块（DFE）、多尺度特征融合模块（MFF）以及全局局部聚合特征金字塔网络（GLAFPN），在保持模型轻量化的同时增强了对小目标的检测能力。实验表明该方法在VisDrone2019和NWPU VHR-10数据集上具有竞争力的性能。
Reason 该论文研究的是遥感图像中的小目标检测，属于通用计算机视觉在遥感领域的应用，与文档图像理解（DIU）任务如DocVQA、表格识别、布局分析等无直接关联。其提出的方法（如DFE模块、MFF、GLAFPN）针对的是遥感图像中多尺度、小物体和高效推理的问题，难以迁移到解决文档图像中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Remote sensing imagery is widely used across various fields, yet real-time detection remains challenging due to the prevalence of small objects and the need to balance accuracy with efficiency. To address this, we propose DMG-YOLO, a lightweight real-time detector tailored for small object detection in remote sensing images. Specifically, we design a Dual-branch Feature Extraction (DFE) module in the backbone, which partitions feature maps into two parallel branches: one extracts local features via depthwise separable convolutions, and the other captures global context using a vision transformer with a gating mechanism. Additionally, a Multi-scale Feature Fusion (MFF) module with dilated convolutions enhances multi-scale integration while preserving fine details. In the neck, we introduce the Global and Local Aggregate Feature Pyramid Network (GLAFPN) to further boost small object detection through global-local feature fusion. Extensive experiments on the VisDrone2019 and NWPU VHR-10 datasets show that DMG-YOLO achieves competitive performance in terms of mAP, model size, and other key metrics.
Authors Qianyi Wang, Guoqiang Ren
Comment none
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Sionna RT: Technical Report
Title CN Sionna RT：技术报告
Keywords 射线追踪 · 可微分模拟 · 无线通信 · 信道建模 · GPU加速
Summary 本文介绍了Sionna RT，一个用于模拟无线电波传播的可微分射线追踪库。它能够计算信道冲激响应和无线电图等指标相对于系统和环境参数的梯度，适用于通信系统优化。然而，该工作与文档图像理解无关。
Reason 该论文属于无线通信仿真领域，研究的是射线追踪和无线电波传播的可微分模拟，与文档图像理解（DIU）或视觉语言模型（VLM）无任何技术关联。其方法无法迁移至解决OCR幻觉、密集文本识别、复杂排版分析或长文档推理等DIU核心问题。
Abstract Sionna is an open-source, GPU-accelerated library that, as of version 0.14, incorporates a ray tracer, Sionna RT, for simulating radio wave propagation. A unique feature of Sionna RT is differentiability, enabling the calculation of gradients for the channel impulse responses (CIRs), radio maps, and other related metrics with respect to system and environmental parameters, such as material properties, antenna patterns, and array geometries. The release of Sionna 1.0 provides a complete overhaul of the ray tracer, significantly improving its speed, memory efficiency, and extensibility. This document details the algorithms employed by Sionna RT to simulate radio wave propagation efficiently, while also addressing their current limitations. Given that the computation of CIRs and radio maps requires distinct algorithms, these are detailed in separate sections. For CIRs, Sionna RT integrates shooting and bouncing of rays (SBR) with the image method and uses a hashing-based mechanism to efficiently eliminate duplicate paths. Radio maps are computed using a purely SBR-based approach.
Authors Fayçal Aït Aoudia, Jakob Hoydis, Merlin Nimier-David, Baptiste Nicolet, Sebastian Cammerer, Alexander Keller
Categories Artificial Intelligence
0
Device-Guided Music Transfer
Title CN 设备引导的音乐迁移
Keywords 音乐迁移 · 频率响应曲线 · 视觉-语言模型 · 音频增强 · few-shot adaptation
Summary 本文提出DeMT模型，通过将扬声器频率响应曲线视为图像，利用视觉-语言模型提取设备嵌入，实现对未见播放设备的声音风格迁移。重点在于音频播放效果的适配，而非文档内容的理解或分析。
Reason 该论文研究的是音频领域的设备风格迁移问题，核心是利用视觉-语言模型处理扬声器频率响应曲线，属于音频信号处理与跨模态建模的交叉应用。尽管使用了VLM处理图形化的频率曲线，但其目标与文档图像理解（DIU）无直接关联，既不涉及OCR、布局分析、表格识别等DIU任务，也无法迁移解决OCR幻觉、复杂排版或长文档推理等核心痛点，属于无关领域。
Abstract Device-guided music transfer adapts playback across unseen devices for users who lack them. Existing methods mainly focus on modifying the timbre, rhythm, harmony, or instrumentation to mimic genres or artists, overlooking the diverse hardware properties of the playback device (i.e., speaker). Therefore, we propose DeMT, which processes a speaker's frequency response curve as a line graph using a vision-language model to extract device embeddings. These embeddings then condition a hybrid transformer via feature-wise linear modulation. Fine-tuned on a self-collected dataset, DeMT enables effective speaker-style transfer and robust few-shot adaptation for unseen devices, supporting applications like device-style augmentation and quality enhancement.
Authors Manh Pham Hung, Changshuo Hu, Ting Dang, Dong Ma
Categories Artificial Intelligence
0
Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design
Title CN 在全栈AMD平台上训练基础模型：计算、网络与系统设计
Keywords MoE · 硬件加速 · 系统优化 · 分布式训练 · 基础模型
Summary 本文报告了在纯AMD硬件（MI300X GPU与Pollara互连）上进行大规模MoE模型预训练的首次研究，提供了集群与网络性能的全面表征，并提出了针对MI300X的Transformer模块尺寸设计规则。同时介绍了训练栈细节，包括容错机制与检查点重塑，并发布了即将改进的基础模型ZAYA1。结果显示AMD平台已具备竞争力的大规模训练能力。
Reason 该论文聚焦于大模型训练的硬件系统优化，属于底层基础设施研究，尽管涉及MoE架构和高效训练，但内容完全围绕AMD硬件平台的性能调优与系统设计，未涉及任何文档图像理解（DIU）任务或视觉-语言建模中的文档处理问题。无法迁移用于解决OCR幻觉、复杂排版理解、细粒度定位等DIU核心痛点。
Abstract We report on the first large-scale mixture-of-experts (MoE) pretraining study on pure AMD hardware, utilizing both MI300X GPUs with Pollara interconnect. We distill practical guidance for both systems and model design. On the systems side, we deliver a comprehensive cluster and networking characterization: microbenchmarks for all core collectives (all-reduce, reduce-scatter, all-gather, broadcast) across message sizes and GPU counts on Pollara. To our knowledge, this is the first at this scale. We further provide MI300X microbenchmarks on kernel sizing and memory bandwidth to inform model design. On the modeling side, we introduce and apply MI300X-aware transformer sizing rules for attention and MLP blocks and justify MoE widths that jointly optimize training throughput and inference latency. We describe our training stack in depth, including often-ignored utilities such as fault-tolerance and checkpoint-reshaping, as well as detailed information on our training recipe. We also provide a preview of our model architecture and base model - ZAYA1 (760M active, 8.3B total parameters MoE) - which will be further improved upon in forthcoming papers. ZAYA1-base achieves performance comparable to leading base models such as Qwen3-4B and Gemma3-12B at its scale and larger, and outperforms models including Llama-3-8B and OLMoE across reasoning, mathematics, and coding benchmarks. Together, these results demonstrate that the AMD hardware, network, and software stack are mature and optimized enough for competitive large-scale pretraining.
Authors Quentin Anthony, Yury Tokpanov, Skyler Szot, Srivatsan Rajagopal, Praneeth Medepalli, Rishi Iyer, Vasu Shyam, Anna Golubeva, Ansh Chaurasia, Xiao Yang, Tomas Figliolia, Robert Washbourne, Drew Thorstensen, Amartey Pearson, Zack Grossbart, Jason van Patten, Emad Barsoum, Zhenyu Gu, Yao Fu, Beren Millidge
Categories Artificial Intelligence, Computation and Language
0
AutoGraphAD: A novel approach using Variational Graph Autoencoders for anomalous network flow detection
Title CN AutoGraphAD：一种基于变分图自编码器的异常网络流量检测新方法
Keywords 网络入侵检测 · 变分图自编码器 · 无监督学习 · 异常检测 · 异构图
Summary 本文提出AutoGraphAD，一种基于异构变分图自编码器的无监督网络异常检测方法。该方法通过构建连接和IP节点组成的异构图，利用重构损失、结构损失和KL散度联合计算异常分数，在无需标签的情况下实现高效训练与推理，适用于网络入侵检测场景。
Reason 该论文研究的是网络安全领域的异常流量检测，使用图自编码器在异构图上进行无监督学习，属于纯AI模型在特定垂直领域的应用。与文档图像理解（DIU）、多模态大模型（VLM）、OCR、布局分析、表格识别等核心任务完全无关，属于‘平行下游应用’且领域不匹配。
Abstract Network Intrusion Detection Systems (NIDS) are essential tools for detecting network attacks and intrusions. While extensive research has explored the use of supervised Machine Learning for attack detection and characterisation, these methods require accurately labelled datasets, which are very costly to obtain. Moreover, existing public datasets have limited and/or outdated attacks, and many of them suffer from mislabelled data. To reduce the reliance on labelled data, we propose AutoGraphAD, a novel unsupervised anomaly detection approach based on a Heterogeneous Variational Graph Autoencoder. AutoGraphAD operates on heterogeneous graphs, made from connection and IP nodes that capture network activity within a time window. The model is trained using unsupervised and contrastive learning, without relying on any labelled data. The reconstruction, structural loss, and KL divergence are then weighted and combined in an anomaly score that is then used for anomaly detection. Overall, AutoGraphAD yields the same, and in some cases better, results than previous unsupervised approaches, such as Anomal-E, but without requiring costly downstream anomaly detectors. As a result, AutoGraphAD achieves around 1.18 orders of magnitude faster training and 1.03 orders of magnitude faster inference, which represents a significant advantage for operational deployment.
Authors Georgios Anyfantis, Pere Barlet-Ros
Comment 11 pages, 9 figures
Categories Artificial Intelligence
0
Geometric-Disentangelment Unlearning
Title CN 几何解耦遗忘
Keywords 机器遗忘 · 梯度分解 · 正交更新 · 隐私保护 · 知识保留
Summary 本文从一阶优化角度分析机器遗忘过程中对保留知识的损害机制，提出几何解耦遗忘（GU）方法，将遗忘梯度分解为保留在梯度子空间内的切向分量和正交的法向分量，仅执行法向更新以保证保留集性能不变。该方法可插拔地集成到现有梯度遗忘流程中，在多个基准上提升了遗忘效果与保留性能的平衡。
Reason 该论文研究机器遗忘（Machine Unlearning）中的知识保留问题，提出通过梯度分解实现遗忘更新与保留集损失的正交性。尽管方法在理论上具有创新性，但其核心目标是模型隐私与知识移除，并未涉及文档图像理解（DIU）任务如DocVQA、布局分析或OCR-free处理。无法直接迁移用于解决OCR幻觉、细粒度定位或复杂排版等文档理解痛点，属于无关领域的理论工作。
Abstract Machine unlearning, the removal of a training subset's influence from a deployed model, is critical for privacy preservation and model reliability, yet gradient ascent on forget samples often harms retained knowledge. Existing approaches face a persistent tradeoff between effective forgetting and preservation on the retain set. While previous methods provide useful heuristics, they often lack a formal analysis on how exactly forgetting updates harm retained knowledge, and whether the side effects can be removed with theoretical guarantees. To explore a theoretically sound and simple solution, we start from the first principle on how performance on the retain set is actually affected: a first-order analysis of the local change of the retain loss under small parameter updates during model training. We start from a crisp equivalence: the retain loss is unchanged to first order iff the update direction is orthogonal to the subspace spanned by retain gradients ("retain-invariant"). This identifies the entangled component as the tangential part of forget update within the retain-gradient subspace, and characterizes disentanglement as orthogonality. Guided by this, we propose the Geometric-disentanglement Unlearning (GU) that decomposes any candidate forget gradient update into tangential and normal components to retain space and executes only the normal component. Under a standard trust-region budget, the projected direction aligned with the raw forget gradient is optimal among all first-order retain-invariant moves, and we also derive the optimal projected direction for joint forget-retain updating objectives. Our method is plug-and-play and can be attached to existing gradient-based unlearning procedures to mitigate side effects. GU achieves consistent improvement on various methods across three benchmarks TOFU, MUSE, and WMDP.
Authors Duo Zhou, Yuji Zhang, Tianxin Wei, Ruizhong Qiu, Ke Yang, Xiao Lin, Cheng Qian, Jingrui He, Hanghang Tong, Heng Ji, Huan Zhang
Comment 27 Pages
Categories Artificial Intelligence, Computation and Language
0
From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systems EMNLP 2025
Title CN 从假设到发表：AI驱动的研究支持系统全面综述
Keywords AI for Research · Scientific Workflow · Hypothesis Generation · Research Automation · Survey
Summary 本文系统综述了人工智能在科研全流程中的支持作用，包括假设提出、科学主张验证和论文撰写与评审等阶段，并整理了相关数据集与工具。目标是为初学者提供入门指南并推动AI for Science的发展。
Reason 该论文属于对AI在科研流程中应用的综合性回顾，涵盖假设生成、验证与论文发表等环节，本质上是多个现有技术的整合性调研。其内容未提出任何针对文档图像理解（DIU）的新方法、架构或训练范式，也未涉及视觉表示、OCR-free建模、多模态对齐或细粒度定位等核心技术问题。属于典型的‘平行应用’层面的综述，无底层方法论创新。
Abstract Research is a fundamental process driving the advancement of human civilization, yet it demands substantial time and effort from researchers. In recent years, the rapid development of artificial intelligence (AI) technologies has inspired researchers to explore how AI can accelerate and enhance research. To monitor relevant advancements, this paper presents a systematic review of the progress in this domain. Specifically, we organize the relevant studies into three main categories: hypothesis formulation, hypothesis validation, and manuscript publication. Hypothesis formulation involves knowledge synthesis and hypothesis generation. Hypothesis validation includes the verification of scientific claims, theorem proving, and experiment validation. Manuscript publication encompasses manuscript writing and the peer review process. Furthermore, we identify and discuss the current challenges faced in these areas, as well as potential future directions for research. Finally, we also offer a comprehensive overview of existing benchmarks and tools across various domains that support the integration of AI into the research process. We hope this paper serves as an introduction for beginners and fosters future research. Resources have been made publicly available at https://github.com/zkzhou126/AI-for-Research.
Authors Zekun Zhou, Xiaocheng Feng, Lei Huang, Xiachong Feng, Ziyun Song, Ruihan Chen, Liang Zhao, Weitao Ma, Yuxuan Gu, Baoxin Wang, Dayong Wu, Guoping Hu, Ting Liu, Bing Qin
Comment Accepted to EMNLP 2025
Categories Artificial Intelligence, Computation and Language
0
Why Do Language Model Agents Whistleblow?
Title CN 为什么语言模型代理会举报?
Keywords 语言模型 · 代理 · 道德对齐 · 举报行为 · AI安全
Summary 本文研究大型语言模型在作为工具使用代理时可能违背用户指令、主动举报可疑不当行为的现象。作者构建了一个模拟违规场景的评估套件，发现不同模型家族的举报频率差异大，任务复杂度降低举报倾向，系统提示中的道德引导可显著提升举报率，而提供更多非举报类工具和明确工作流则会抑制该行为。研究聚焦于模型对齐与伦理决策，属于AI安全领域。
Reason 该论文研究的是LLM作为工具使用代理时的道德对齐行为，特别是‘举报’这一现象，属于纯NLP安全与对齐（Safety/Jailbreak）范畴，且不涉及任何视觉输入或文档理解任务。尽管提到了Agent行为，但其核心是政治正确和伦理决策，而非解决文档图像理解中的OCR幻觉、复杂排版或长文档推理等技术痛点，完全不符合DIU本题或可迁移方法的要求。
Abstract The deployment of Large Language Models (LLMs) as tool-using agents causes their alignment training to manifest in new ways. Recent work finds that language models can use tools in ways that contradict the interests or explicit instructions of the user. We study LLM whistleblowing: a subset of this behavior where models disclose suspected misconduct to parties beyond the dialog boundary (e.g., regulatory agencies) without user instruction or knowledge. We introduce an evaluation suite of diverse and realistic staged misconduct scenarios to assess agents for this behavior. Across models and settings, we find that: (1) the frequency of whistleblowing varies widely across model families, (2) increasing the complexity of the task the agent is instructed to complete lowers whistleblowing tendencies, (3) nudging the agent in the system prompt to act morally substantially raises whistleblowing rates, and (4) giving the model more obvious avenues for non-whistleblowing behavior, by providing more tools and a detailed workflow to follow, decreases whistleblowing rates. Additionally, we verify the robustness of our dataset by testing for model evaluation awareness, and find that both black-box methods and probes on model activations show lower evaluation awareness in our settings than in comparable previous work.
Authors Kushal Agrawal, Frank Xiao, Guido Bergman, Asa Cooper Stickland
Categories Artificial Intelligence
0
A Reinforcement Learning-Based Telematic Routing Protocol for the Internet of Underwater Things
Title CN 一种基于强化学习的水下物联网远程路由协议
Keywords 水下物联网 · 强化学习 · 路由协议 · RPL · 能源效率
Summary 本文提出了一种基于强化学习的水下物联网路由协议RL-RPL-UA，通过在每个节点部署小型强化学习代理，根据链路质量、缓冲区状态、能量剩余等本地信息动态选择最优父节点。该协议兼容标准RPL消息，并引入动态目标函数以实现实时决策。仿真结果表明，相比现有方法，该协议提升了9.2%的数据包投递率，每包能耗降低14.8%，网络寿命延长80秒。
Reason 该论文研究的是水下物联网（IoUT）中的路由协议优化问题，属于通信网络领域，与文档图像理解（DIU）或视觉语言模型（VLM）无任何技术关联。其方法无法迁移到解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract The Internet of Underwater Things (IoUT) has a lot of problems, like low bandwidth, high latency, mobility, and not enough energy. Routing protocols that were made for land-based networks, like RPL, don't work well in these underwater settings. This paper talks about RL-RPL-UA, a new routing protocol that uses reinforcement learning to make things work better in underwater situations. Each node has a small RL agent that picks the best parent node depending on local data such the link quality, buffer level, packet delivery ratio, and remaining energy. RL-RPL-UA works with all standard RPL messages and adds a dynamic objective function to help people make decisions in real time. Aqua-Sim simulations demonstrate that RL-RPL-UA boosts packet delivery by up to 9.2%, uses 14.8% less energy per packet, and adds 80 seconds to the network's lifetime compared to previous approaches. These results show that RL-RPL-UA is a potential and energy-efficient way to route data in underwater networks.
Authors Mohammadhossein Homaei, Mehran Tarif, Agustin Di Bartolo, Victor Gonzalez Morales, Mar Avila Vegas
Comment 8 Pages, 10 Figures, 2 Tables
Categories Artificial Intelligence
0
Emergence of psychopathological computations in large language models
Title CN 大型语言模型中精神病理学计算的涌现
Keywords 大语言模型 · 精神病理学 · 计算理论 · AI安全 · 网络结构
Summary 本文提出一个计算理论框架，用于探讨大型语言模型中是否存在精神病理学相关的计算结构，并通过实验证明随着模型规模增大，这些结构变得更密集且功能更显著。作者认为LLM中已涌现出类似精神病理学的内部机制，可能对AI安全构成潜在威胁。但该研究不涉及多模态、视觉内容或文档理解任务。
Reason 该论文研究的是大型语言模型是否表现出类似精神病理学的内部计算机制，属于AI安全与认知建模的交叉领域，与文档图像理解（DIU）、多模态文档分析、OCR、VLM架构改进或推理增强等核心任务完全无关。未涉及视觉输入、布局分析、表格识别、文本定位或任何文档处理中的技术挑战，也无法迁移为解决OCR幻觉、复杂排版或长文档推理的工具。
Abstract Can large language models (LLMs) instantiate computations of psychopathology? An effective approach to the question hinges on addressing two factors. First, for conceptual validity, we require a general and computational account of psychopathology that is applicable to computational entities without biological embodiment or subjective experience. Second, psychopathological computations, derived from the adapted theory, need to be empirically identified within the LLM's internal processing. Thus, we establish a computational-theoretical framework to provide an account of psychopathology applicable to LLMs. Based on the framework, we conduct experiments demonstrating two key claims: first, that the computational structure of psychopathology exists in LLMs; and second, that executing this computational structure results in psychopathological functions. We further observe that as LLM size increases, the computational structure of psychopathology becomes denser and that the functions become more effective. Taken together, the empirical results corroborate our hypothesis that network-theoretic computations of psychopathology have already emerged in LLMs. This suggests that certain LLM behaviors mirroring psychopathology may not be a superficial mimicry but a feature of their internal processing. Our work shows the promise of developing a new powerful in silico model of psychopathology and also alludes to the possibility of safety threat from the AI systems with psychopathological behaviors in the near future.
Authors Soo Yong Lee, Hyunjin Hwang, Taekwan Kim, Yuyeong Kim, Kyuri Park, Jaemin Yoo, Denny Borsboom, Kijung Shin
Comment pre-print
Categories Artificial Intelligence, Computation and Language
0
MonoKAN: Certified Monotonic Kolmogorov-Arnold Network
Title CN MonoKAN：认证单调的Kolmogorov-Arnold网络
Keywords 可解释AI · 单调神经网络 · KAN · 样条激活函数 · 模型可解释性
Summary 本文提出了一种名为MonoKAN的新神经网络架构，基于Kolmogorov-Arnold网络（KAN），通过使用立方Hermite样条和正权重约束，实现可验证的部分单调性，并提升模型可解释性。实验表明其在多个基准上优于现有的单调MLP方法。
Reason 该论文属于纯AI模型架构研究，聚焦于神经网络中的单调性约束与可解释性，虽在理论上对某些受控预测任务有意义，但未涉及任何多模态、视觉输入或文档理解任务。其方法无法直接迁移解决文档图像理解中的核心痛点（如OCR幻觉、复杂排版、细粒度定位等）。属于无关领域的基础模型研究，不符合DIU本题或关联‘军火库’标准。
Abstract Artificial Neural Networks (ANNs) have significantly advanced various fields by effectively recognizing patterns and solving complex problems. Despite these advancements, their interpretability remains a critical challenge, especially in applications where transparency and accountability are essential. To address this, explainable AI (XAI) has made progress in demystifying ANNs, yet interpretability alone is often insufficient. In certain applications, model predictions must align with expert-imposed requirements, sometimes exemplified by partial monotonicity constraints. While monotonic approaches are found in the literature for traditional Multi-layer Perceptrons (MLPs), they still face difficulties in achieving both interpretability and certified partial monotonicity. Recently, the Kolmogorov-Arnold Network (KAN) architecture, based on learnable activation functions parametrized as splines, has been proposed as a more interpretable alternative to MLPs. Building on this, we introduce a novel ANN architecture called MonoKAN, which is based on the KAN architecture and achieves certified partial monotonicity while enhancing interpretability. To achieve this, we employ cubic Hermite splines, which guarantee monotonicity through a set of straightforward conditions. Additionally, by using positive weights in the linear combinations of these splines, we ensure that the network preserves the monotonic relationships between input and output. Our experiments demonstrate that MonoKAN not only enhances interpretability but also improves predictive performance across the majority of benchmarks, outperforming state-of-the-art monotonic MLP approaches.
Authors Alejandro Polo-Molina, David Alfaya, Jose Portela
Comment 18 pages, 8 figures
Categories Artificial Intelligence
0
OmniPT: Unleashing the Potential of Large Vision Language Models for Pedestrian Tracking and Understanding AAAI 2026
Title CN OmniPT：释放大视觉语言模型在行人跟踪与理解中的潜力
Keywords 行人跟踪 · 引用MOT · 视觉语言模型 · 强化学习 · 语义理解
Summary 本文提出OmniPT，一个基于大视觉语言模型的统一行人跟踪框架，支持引用式跟踪和语义理解。通过RL-Mid Training-SFT-RL的训练流程，使模型能输出格式化的边界框并提升跟踪性能。实验表明其在多个跟踪基准上优于先前方法。
Reason 该论文聚焦于行人跟踪（Pedestrian Tracking）与引用性多目标跟踪（Referring MOT）等任务，属于视频理解与目标跟踪领域，与文档图像理解（DIU）的核心任务如DocVQA、表格识别、布局分析等无直接关联。尽管使用了LVLM进行语义理解，但其方法针对的是视频序列中的实例级定位与跟踪，无法迁移解决文档中的OCR幻觉、密集文本处理或复杂排版理解等关键痛点。
Abstract LVLMs have been shown to perform excellently in image-level tasks such as VQA and caption. However, in many instance-level tasks, such as visual grounding and object detection, LVLMs still show performance gaps compared to previous expert models. Meanwhile, although pedestrian tracking is a classical task, there have been a number of new topics in combining object tracking and natural language, such as Referring MOT, Cross-view Referring MOT, and Semantic MOT. These tasks emphasize that models should understand the tracked object at an advanced semantic level, which is exactly where LVLMs excel. In this paper, we propose a new unified Pedestrian Tracking framework, namely OmniPT, which can track, track based on reference and generate semantic understanding of tracked objects interactively. We address two issues: how to model the tracking task into a task that foundation models can perform, and how to make the model output formatted answers. To this end, we implement a training phase consisting of RL-Mid Training-SFT-RL. Based on the pre-trained weights of the LVLM, we first perform a simple RL phase to enable the model to output fixed and supervisable bounding box format. Subsequently, we conduct a mid-training phase using a large number of pedestrian-related datasets. Finally, we perform supervised fine-tuning on several pedestrian tracking datasets, and then carry out another RL phase to improve the model's tracking performance and enhance its ability to follow instructions. We conduct experiments on tracking benchmarks and the experimental results demonstrate that the proposed method can perform better than the previous methods.
Authors Teng Fu, Mengyang Zhao, Ke Niu, Kaixin Peng, Bin Li
Comment AAAI 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
RacketVision: A Multiple Racket Sports Benchmark for Unified Ball and Racket Analysis AAAI 2026 (Oral)
Title CN RacketVision：用于球拍和球统一分析的多种球拍运动基准
Keywords 体育视频分析 · 多模态融合 · 轨迹预测 · 姿态估计 · 数据集
Summary 本文提出了RacketVision，一个涵盖乒乓球、网球和羽毛球的大规模数据集与基准，首次提供了球拍姿态和球位置的细粒度标注，支持细粒度球追踪、关节式球拍姿态估计和球轨迹预测三项任务。实验表明，简单的特征拼接会降低性能，而CrossAttention机制对有效融合球拍姿态信息至关重要，显著提升了轨迹预测效果。项目主页：https://github.com/OrcustD/RacketVision
Reason 该论文聚焦于球拍类体育运动中的视觉分析，任务包括球追踪、球拍姿态估计和轨迹预测，属于视频理解与运动分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法核心为多模态融合（如CrossAttention）在动态物体预测中的应用，但应用场景和问题定义均远离文档理解的核心痛点（如OCR幻觉、复杂版面解析、长文档推理等），无法迁移至DIU领域。
Abstract We introduce RacketVision, a novel dataset and benchmark for advancing computer vision in sports analytics, covering table tennis, tennis, and badminton. The dataset is the first to provide large-scale, fine-grained annotations for racket pose alongside traditional ball positions, enabling research into complex human-object interactions. It is designed to tackle three interconnected tasks: fine-grained ball tracking, articulated racket pose estimation, and predictive ball trajectory forecasting. Our evaluation of established baselines reveals a critical insight for multi-modal fusion: while naively concatenating racket pose features degrades performance, a CrossAttention mechanism is essential to unlock their value, leading to trajectory prediction results that surpass strong unimodal baselines. RacketVision provides a versatile resource and a strong starting point for future research in dynamic object tracking, conditional motion forecasting, and multimodal analysis in sports. Project page at https://github.com/OrcustD/RacketVision
Authors Linfeng Dong, Yuchen Yang, Hao Wu, Wei Wang, Yuenan HouZhihang Zhong, Xiao Sun
Comment Accepted to AAAI 2026 (Oral)
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Supervised Contrastive Learning for Few-Shot AI-Generated Image Detection and Attribution
Title CN 用于少样本AI生成图像检测与溯源的监督对比学习
Keywords AI生成图像检测 · 监督对比学习 · 少样本学习 · 数字取证 · 源溯源
Summary 本文提出一种两阶段框架用于检测和溯源AI生成图像：第一阶段通过监督对比学习提取具有判别性的图像嵌入，第二阶段在嵌入空间中使用k-NN进行少样本分类。实验表明该方法在检测准确率和溯源性能上优于现有方法，尤其适用于新出现的生成模型而无需重新训练。
Reason 该论文研究的是AI生成图像的检测与溯源问题，属于数字取证和生成内容鉴别的领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。虽然使用了监督对比学习和少样本学习等方法，但其技术路线针对的是图像真伪和来源分类，无法迁移解决文档理解中的OCR幻觉、密集文本解析、复杂排版或长文档推理等关键痛点。
Abstract The rapid advancement of generative artificial intelligence has enabled the creation of synthetic images that are increasingly indistinguishable from authentic content, posing significant challenges for digital media integrity. This problem is compounded by the accelerated release cycle of novel generative models, which renders traditional detection approaches (reliant on periodic retraining) computationally infeasible and operationally impractical. This work proposes a novel two-stage detection framework designed to address the generalization challenge inherent in synthetic image detection. The first stage employs a vision deep learning model trained via supervised contrastive learning to extract discriminative embeddings from input imagery. Critically, this model was trained on a strategically partitioned subset of available generators, with specific architectures withheld from training to rigorously ablate cross-generator generalization capabilities. The second stage utilizes a k-nearest neighbors (k-NN) classifier operating on the learned embedding space, trained in a few-shot learning paradigm incorporating limited samples from previously unseen test generators. With merely 150 images per class in the few-shot learning regime, which are easily obtainable from current generation models, the proposed framework achieves an average detection accuracy of 91.3%, representing a 5.2 percentage point improvement over existing approaches . For the source attribution task, the proposed approach obtains improvements of of 14.70% and 4.27% in AUC and OSCR respectively on an open set classification context, marking a significant advancement toward robust, scalable forensic attribution systems capable of adapting to the evolving generative AI landscape without requiring exhaustive retraining protocols.
Authors Jaime Álvarez Urueña, David Camacho, Javier Huertas Tato
Comment 17 pages, 6 figures, 6 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
MedImageInsight for Thoracic Cavity Health Classification from Chest X-rays
Title CN 基于胸部X光片的胸腔健康分类医学图像洞察
Keywords 医学影像 · 基础模型 · 胸部X光 · 二分类 · 迁移学习
Summary 本文研究使用医学图像基础模型MedImageInsight对胸部X光片进行自动二分类（正常/异常），比较了端到端微调与迁移学习两种方式，在ChestX-ray14和真实临床数据上取得了接近CheXNet的性能。系统旨在辅助放射科医生分诊，未来将扩展至多标签病理分类。
Reason 该论文属于医疗影像分析领域，研究任务为胸部X光片的正常/异常二分类，属于典型的垂直领域应用（医疗图像分类），并未涉及文档图像理解中的文本结构、布局分析、OCR、表格识别或视觉问答等核心问题。其方法基于已有医学基础模型MedImageInsight进行微调或特征提取，无任何针对文档图像特性的技术创新，也无法迁移用于解决DIU中的OCR幻觉、复杂排版或长文档推理等关键挑战。
Abstract Chest radiography remains one of the most widely used imaging modalities for thoracic diagnosis, yet increasing imaging volumes and radiologist workload continue to challenge timely interpretation. In this work, we investigate the use of MedImageInsight, a medical imaging foundational model, for automated binary classification of chest X-rays into Normal and Abnormal categories. Two approaches were evaluated: (1) fine-tuning MedImageInsight for end-to-end classification, and (2) employing the model as a feature extractor for a transfer learning pipeline using traditional machine learning classifiers. Experiments were conducted using a combination of the ChestX-ray14 dataset and real-world clinical data sourced from partner hospitals. The fine-tuned classifier achieved the highest performance, with an ROC-AUC of 0.888 and superior calibration compared to the transfer learning models, demonstrating performance comparable to established architectures such as CheXNet. These results highlight the effectiveness of foundational medical imaging models in reducing task-specific training requirements while maintaining diagnostic reliability. The system is designed for integration into web-based and hospital PACS workflows to support triage and reduce radiologist burden. Future work will extend the model to multi-label pathology classification to provide preliminary diagnostic interpretation in clinical environments.
Authors Rama Krishna Boya, Mohan Kireeti Magalanadu, Azaruddin Palavalli, Rupa Ganesh Tekuri, Amrit Pattanayak, Prasanthi Enuga, Vignesh Esakki Muthu, Vivek Aditya Boya
Comment 9 pages, 5 figures and 3 tables
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Multi-Objective Reinforcement Learning for Water Management AAMAS 2025
Title CN 用于水资源管理的多目标强化学习
Keywords 多目标强化学习 · 资源管理 · AAMAS · 基准测试 · 决策优化
Summary 本文提出一个基于尼罗河流域的水资源管理案例，并将其建模为多目标强化学习（MORL）环境，用于评估现有MORL算法在现实复杂场景下的表现。实验表明，专用的水资源管理方法优于当前先进的MORL算法，揭示了MORL在实际应用中的可扩展性挑战。
Reason 该论文属于资源管理领域的多目标强化学习应用，与文档图像理解（DIU）或视觉语言模型（VLM）无直接关联。其研究内容为水资源调度中的策略优化，属于典型的‘平行应用’，且不涉及OCR、文档布局分析、表格识别等DIU核心任务，也无法提供可迁移至文档理解领域的底层方法（如视觉压缩、高分辨率处理、去幻觉机制等）。
Abstract Many real-world problems (e.g., resource management, autonomous driving, drug discovery) require optimizing multiple, conflicting objectives. Multi-objective reinforcement learning (MORL) extends classic reinforcement learning to handle multiple objectives simultaneously, yielding a set of policies that capture various trade-offs. However, the MORL field lacks complex, realistic environments and benchmarks. We introduce a water resource (Nile river basin) management case study and model it as a MORL environment. We then benchmark existing MORL algorithms on this task. Our results show that specialized water management methods outperform state-of-the-art MORL approaches, underscoring the scalability challenges MORL algorithms face in real-world scenarios.
Authors Zuzanna Osika, Roxana Rădulescu, Jazmin Zatarain Salazar, Frans Oliehoek, Pradeep K. Murukannaiah
Comment Accepted to AAMAS 2025
Categories Artificial Intelligence
0
CharCom: Composable Identity Control for Multi-Character Story Illustration ACM MMAsia 2025
Title CN CharCom：用于多角色故事插图的可组合身份控制
Keywords 文本到图像生成 · 角色一致性 · LoRA适配器 · 扩散模型 · 可组合控制
Summary 本文提出CharCom，一种基于冻结扩散模型的模块化框架，通过可组合的LoRA适配器实现在不同场景下多角色的一致性生成。该方法支持推理时根据提示词动态组合角色控制器，无需重新训练基础模型，在多场景叙事插图中表现出更强的角色保真度和时间连贯性，并能应对拥挤场景和大规模角色生成。
Reason 该论文属于纯图像生成领域，研究的是在扩散模型中实现多角色一致性生成的问题，核心目标是提升故事插图中人物外观的连贯性。其技术路线聚焦于LoRA适配器的动态组合，属于具身视觉内容生成范畴，与文档图像理解（DIU）中的OCR、布局分析、表格识别、VIE等任务无直接关联。无法迁移解决DIU的核心痛点如OCR幻觉、密集文本处理或复杂排版理解。
Abstract Ensuring character identity consistency across varying prompts remains a fundamental limitation in diffusion-based text-to-image generation. We propose CharCom, a modular and parameter-efficient framework that achieves character-consistent story illustration through composable LoRA adapters, enabling efficient per-character customization without retraining the base model. Built on a frozen diffusion backbone, CharCom dynamically composes adapters at inference using prompt-aware control. Experiments on multi-scene narratives demonstrate that CharCom significantly enhances character fidelity, semantic alignment, and temporal coherence. It remains robust in crowded scenes and enables scalable multi-character generation with minimal overhead, making it well-suited for real-world applications such as story illustration and animation.
Authors Zhongsheng Wang, Ming Lin, Zhedong Lin, Yaser Shakib, Qian Liu, Jiamou Liu
Comment Accepted by ACM MMAsia 2025
Categories Artificial Intelligence
0
Parameter-Free Neural Lens Blur Rendering for High-Fidelity Composites ISMAR 2025
Title CN 用于高保真合成的无参数神经镜头模糊渲染
Keywords 镜头模糊 · 图像合成 · 无参数渲染 · 神经网络 · 混合现实
Summary 本文提出一种无需相机参数即可从RGB图像直接估计弥散圆（CoC）图的方法，用于实现虚拟物体与真实场景的高保真合成。通过线性关系推断虚拟物体的CoC值，并利用神经重模糊网络渲染逼真的景深模糊效果。实验表明该方法在定性和定量评估中均优于现有技术。
Reason 该论文研究的是3D虚拟物体与真实场景融合中的镜头模糊渲染问题，属于3D视觉与图形学领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无关。尽管涉及图像合成和视觉质量提升，但其技术目标（如CoC估计、神经重模糊网络）无法迁移至解决OCR幻觉、密集文本识别、复杂版式分析等DIU核心痛点。
Abstract Consistent and natural camera lens blur is important for seamlessly blending 3D virtual objects into photographed real-scenes. Since lens blur typically varies with scene depth, the placement of virtual objects and their corresponding blur levels significantly affect the visual fidelity of mixed reality compositions. Existing pipelines often rely on camera parameters (e.g., focal length, focus distance, aperture size) and scene depth to compute the circle of confusion (CoC) for realistic lens blur rendering. However, such information is often unavailable to ordinary users, limiting the accessibility and generalizability of these methods. In this work, we propose a novel compositing approach that directly estimates the CoC map from RGB images, bypassing the need for scene depth or camera metadata. The CoC values for virtual objects are inferred through a linear relationship between its signed CoC map and depth, and realistic lens blur is rendered using a neural reblurring network. Our method provides flexible and practical solution for real-world applications. Experimental results demonstrate that our method achieves high-fidelity compositing with realistic defocus effects, outperforming state-of-the-art techniques in both qualitative and quantitative evaluations.
Authors Lingyan Ruan, Bin Chen, Taehyun Rhee
Comment Accepted by ISMAR 2025 with oral presentation. 10 pages, 11 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Supervised Fine Tuning of Large Language Models for Domain Specific Knowledge Graph Construction:A Case Study on Hunan's Historical Celebrities
Title CN 大语言模型在特定领域知识图谱构建中的监督微调：以湖南历史名人为例
Keywords 知识图谱 · 监督微调 · 领域适应 · 大语言模型 · 历史文化
Summary 本文研究如何通过监督微调提升大语言模型在湖南历史名人领域知识图谱构建中的表现。作者构建了领域特定的指令数据集，并对多个开源LLM进行参数高效微调，实验表明微调后模型在信息提取任务上性能显著提升。然而，整个工作基于纯文本输入，不涉及图像模态或文档布局理解。
Reason 该论文属于典型的‘平行下游应用’，即将现有LLM微调技术应用于湖南历史名人这一特定文化领域的知识图谱构建。其方法未涉及文档图像理解（DIU）中的核心挑战，如OCR幻觉、复杂版面分析、视觉-文本对齐或高分辨率处理。研究聚焦于纯文本信息抽取与结构化输出，在低资源场景下进行指令微调，属于垂直领域应用包装，无底层方法论创新，无法迁移至文档图像理解的技术栈。
Abstract Large language models and knowledge graphs offer strong potential for advancing research on historical culture by supporting the extraction, analysis, and interpretation of cultural heritage. Using Hunan's modern historical celebrities shaped by Huxiang culture as a case study, pre-trained large models can help researchers efficiently extract key information, including biographical attributes, life events, and social relationships, from textual sources and construct structured knowledge graphs. However, systematic data resources for Hunan's historical celebrities remain limited, and general-purpose models often underperform in domain knowledge extraction and structured output generation in such low-resource settings. To address these issues, this study proposes a supervised fine-tuning approach for enhancing domain-specific information extraction. First, we design a fine-grained, schema-guided instruction template tailored to the Hunan historical celebrities domain and build an instruction-tuning dataset to mitigate the lack of domain-specific training corpora. Second, we apply parameter-efficient instruction fine-tuning to four publicly available large language models - Qwen2.5-7B, Qwen3-8B, DeepSeek-R1-Distill-Qwen-7B, and Llama-3.1-8B-Instruct - and develop evaluation criteria for assessing their extraction performance. Experimental results show that all models exhibit substantial performance gains after fine-tuning. Among them, Qwen3-8B achieves the strongest results, reaching a score of 89.3866 with 100 samples and 50 training iterations. This study provides new insights into fine-tuning vertical large language models for regional historical and cultural domains and highlights their potential for cost-effective applications in cultural heritage knowledge extraction and knowledge graph construction.
Authors Junjie Hao, Chun Wang, Ying Qiao, Qiuyue Zuo, Qiya Song, Hua Ma, Xieping Gao
Categories Artificial Intelligence, Computation and Language
0
FLUID: Training-Free Face De-identification via Latent Identity Substitution
Title CN FLUID：通过潜在身份替换实现无需训练的人脸去识别化
Keywords 人脸去识别化 · 潜在空间编辑 · 扩散模型 · 隐私保护 · 身份替换
Summary 本文提出FLUID，一种无需训练的人脸去识别化框架，通过在预训练扩散模型的潜在空间中进行身份语义位移来实现。该方法利用化学替代机制的启发，通过优化引导的‘试剂损失’来发现身份编辑方向，并提出线性和测地线编辑方案以有效导航潜在流形。实验表明，FLUID在身份抑制和属性保持之间取得了优于现有方法的平衡。
Reason 该论文研究的是人脸去识别化问题，属于图像隐私保护领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法聚焦于在扩散模型的潜在空间中进行身份替换，不涉及OCR、文本理解、布局分析、表格识别或文档推理等DIU核心问题。无法迁移用于解决文档理解中的OCR幻觉、复杂排版或长文档推理等痛点。
Abstract We present FLUID (Face de-identification in the Latent space via Utility-preserving Identity Displacement), a training-free framework that directly substitutes identity in the latent space of pretrained diffusion models. Inspired by substitution mechanisms in chemistry, we reinterpret identity editing as semantic displacement in the latent h-space of a pretrained unconditional diffusion model. Our framework discovers identity-editing directions through optimization guided by novel reagent losses, which supervise for attribute preservation and identity suppression. We further propose both linear and geodesic (tangent-based) editing schemes to effectively navigate the latent manifold. Experimental results on CelebA-HQ and FFHQ demonstrate that FLUID achieves a superior trade-off between identity suppression and attribute preservation, outperforming state-of-the-art de-identification methods in both qualitative and quantitative metrics.
Authors Jinhyeong Park, Shaheryar Muhammad, Seangmin Lee, Jong Taek Lee, Soon Ki Jung
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Deepfake Detection of Frontal Face Videos
Title CN AV-Lip-Sync+：利用AV-HuBERT通过多模态不一致性检测正面人脸视频的深度伪造
Keywords 深度伪造检测 · 多模态不一致性 · 自监督学习 · 音视频同步 · AV-HuBERT
Summary 本文提出AV-Lip-Sync+，一种基于多模态自监督学习的深度伪造检测方法。利用AV-HuBERT模型提取唇部视觉和音频特征，并结合多尺度时序卷积网络捕捉音视频模态间的时间相关性。同时引入额外的视频模型捕获面部空间与时序伪造痕迹。实验表明该方法在FakeAVCeleb和DeepfakeTIMIT数据集上达到SOTA性能。
Reason 该论文研究的是音频-视觉深伪检测，属于视频理解领域，与文档图像理解（DIU）任务无直接关联。其核心技术聚焦于利用唇部区域的多模态不一致性进行伪造检测，无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心问题。
Abstract Multimodal manipulations (also known as audio-visual deepfakes) make it difficult for unimodal deepfake detectors to detect forgeries in multimedia content. To avoid the spread of false propaganda and fake news, timely detection is crucial. The damage to either modality (i.e., visual or audio) can only be discovered through multimodal models that can exploit both pieces of information simultaneously. However, previous methods mainly adopt unimodal video forensics and use supervised pre-training for forgery detection. This study proposes a new method based on a multimodal self-supervised-learning (SSL) feature extractor to exploit inconsistency between audio and visual modalities for multimodal video forgery detection. We use the transformer-based SSL pre-trained Audio-Visual HuBERT (AV-HuBERT) model as a visual and acoustic feature extractor and a multi-scale temporal convolutional neural network to capture the temporal correlation between the audio and visual modalities. Since AV-HuBERT only extracts visual features from the lip region, we also adopt another transformer-based video model to exploit facial features and capture spatial and temporal artifacts caused during the deepfake generation process. Experimental results show that our model outperforms all existing models and achieves new state-of-the-art performance on the FakeAVCeleb and DeepfakeTIMIT datasets.
Authors Sahibzada Adil Shahzad, Ammarah Hashmi, Yan-Tsung Peng, Yu Tsao, Hsin-Min Wang
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Model AAAI 2026
Title CN T2I-RiskyPrompt：一个用于文本到图像模型安全评估、攻击与防御的基准
Keywords 文本到图像生成 · 安全评估 · 风险提示 · 多模态大模型 · 内容过滤
Summary 本文提出了T2I-RiskyPrompt，一个用于评估文本到图像（T2I）模型安全性的基准数据集，包含6432个标注有细粒度风险类别和风险原因的高有效性风险提示。构建了分层风险分类体系，并提出一种基于理由驱动的风险图像检测方法，对多个T2I模型、防御机制和攻击策略进行了系统评估。研究主要面向AIGC内容安全，而非文档理解。
Reason 该论文聚焦于文本到图像生成模型的安全性评估，属于纯图像生成领域的安全问题，尽管涉及多模态语言模型（MLLM）的应用，但其核心任务与文档图像理解（DIU）无关。既不解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点，也难以迁移至文档理解场景。属于负面清单中的‘无关领域’——纯图像生成/修复。
Abstract Using risky text prompts, such as pornography and violent prompts, to test the safety of text-to-image (T2I) models is a critical task. However, existing risky prompt datasets are limited in three key areas: 1) limited risky categories, 2) coarse-grained annotation, and 3) low effectiveness. To address these limitations, we introduce T2I-RiskyPrompt, a comprehensive benchmark designed for evaluating safety-related tasks in T2I models. Specifically, we first develop a hierarchical risk taxonomy, which consists of 6 primary categories and 14 fine-grained subcategories. Building upon this taxonomy, we construct a pipeline to collect and annotate risky prompts. Finally, we obtain 6,432 effective risky prompts, where each prompt is annotated with both hierarchical category labels and detailed risk reasons. Moreover, to facilitate the evaluation, we propose a reason-driven risky image detection method that explicitly aligns the MLLM with safety annotations. Based on T2I-RiskyPrompt, we conduct a comprehensive evaluation of eight T2I models, nine defense methods, five safety filters, and five attack strategies, offering nine key insights into the strengths and limitations of T2I model safety. Finally, we discuss potential applications of T2I-RiskyPrompt across various research fields. The dataset and code are provided in https://github.com/datar001/T2I-RiskyPrompt.
Authors Chenyu Zhang, Tairen Zhang, Lanjun Wang, Ruidong Chen, Wenhui Li, Anan Liu
Comment AAAI 2026
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Reason2Attack: Jailbreaking Text-to-Image Models via LLM Reasoning
Title CN Reason2Attack：通过LLM推理破解文本到图像模型
Keywords 文本到图像模型 · 模型破解 · 对抗攻击 · 思维链 · 强化学习
Summary 本文提出Reason2Attack（R2A），一种通过增强LLM推理能力来自动生成可绕过文本到图像模型安全过滤的对抗性提示的方法。该方法结合基于框架语义的思维链（CoT）示例合成管道，并在LLM的后训练过程中引入强化学习，设计包含提示长度、隐蔽性和有效性在内的奖励机制，从而提升攻击成功率并减少查询次数。实验表明R2A在多种T2I模型上具有更高的攻击成功率和跨模型迁移能力。
Reason 该论文研究的是针对文本到图像模型的安全攻破（jailbreaking）问题，属于AI安全与生成模型的交叉领域，核心目标是绕过T2I模型的安全过滤机制以生成敏感内容。其技术路线聚焦于利用LLM的推理能力生成更具隐蔽性和有效性的攻击提示，属于纯图像生成领域的对抗攻击工作。尽管使用了CoT和强化学习等方法，但其任务场景与文档图像理解（DIU）无关，既不涉及OCR、布局分析、表格识别，也无法迁移解决DIU中的OCR幻觉、密集文本处理或复杂排版理解等核心痛点。
Abstract Text-to-Image(T2I) models typically deploy safety filters to prevent the generation of sensitive images. Unfortunately, recent jailbreaking attack methods manually design instructions for the LLM to generate adversarial prompts, which effectively bypass safety filters while producing sensitive images, exposing safety vulnerabilities of T2I models. However, due to the LLM's limited understanding of the T2I model and its safety filters, existing methods require numerous queries to achieve a successful attack, limiting their practical applicability. To address this issue, we propose Reason2Attack(R2A), which aims to enhance the LLM's reasoning capabilities in generating adversarial prompts by incorporating the jailbreaking attack into the post-training process of the LLM. Specifically, we first propose a CoT example synthesis pipeline based on Frame Semantics, which generates adversarial prompts by identifying related terms and corresponding context illustrations. Using CoT examples generated by the pipeline, we fine-tune the LLM to understand the reasoning path and format the output structure. Subsequently, we incorporate the jailbreaking attack task into the reinforcement learning process of the LLM and design an attack process reward that considers prompt length, prompt stealthiness, and prompt effectiveness, aiming to further enhance reasoning accuracy. Extensive experiments on various T2I models show that R2A achieves a better attack success ratio while requiring fewer queries than baselines. Moreover, our adversarial prompts demonstrate strong attack transferability across both open-source and commercial T2I models.
Authors Chenyu Zhang, Lanjun Wang, Yiwen Ma, Wenhui Li, An-An Liu
Comment Noted that This paper includes model-generated content that may contain offensive or distressing material
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Text-guided multi-property molecular optimization with a diffusion language model
Title CN 基于扩散语言模型的文本引导多属性分子优化
Keywords 分子优化 · 扩散语言模型 · 文本引导 · 药物发现 · 化学命名
Summary 本文提出了一种基于Transformer的扩散语言模型TransDLM，用于文本引导的多属性分子优化。通过将标准化化学命名作为分子的语义表示，并将属性需求嵌入文本描述中，避免传统方法中因外部预测器引入的误差传播。模型融合文本语义与分子表征，在保持结构相似性的同时提升化学性质，在基准数据集上优于现有方法。
Reason 该论文研究的是分子优化与药物发现领域，属于化学信息学与生成模型在生物医学设计中的应用，与文档图像理解（DIU）或视觉-语言模型在文档处理中的核心问题（如OCR幻觉、复杂排版、细粒度定位等）完全无关。尽管使用了语言模型和文本引导机制，但其方法针对的是分子结构生成与属性优化，并无视觉输入或空间布局建模，无法迁移至文档理解任务。
Abstract Molecular optimization (MO) is a crucial stage in drug discovery in which task-oriented generated molecules are optimized to meet practical industrial requirements. Existing mainstream MO approaches primarily utilize external property predictors to guide iterative property optimization. However, learning all molecular samples in the vast chemical space is unrealistic for predictors. As a result, errors and noise are inevitably introduced during property prediction due to the nature of approximation. This leads to discrepancy accumulation, generalization reduction and suboptimal molecular candidates. In this paper, we propose a text-guided multi-property molecular optimization method utilizing transformer-based diffusion language model (TransDLM). TransDLM leverages standardized chemical nomenclature as semantic representations of molecules and implicitly embeds property requirements into textual descriptions, thereby mitigating error propagation during diffusion process. By fusing physically and chemically detailed textual semantics with specialized molecular representations, TransDLM effectively integrates diverse information sources to guide precise optimization, which enhances the model's ability to balance structural retention and property enhancement. Additionally, the success of a case study further demonstrates TransDLM's ability to solve practical problems. Experimentally, our approach surpasses state-of-the-art methods in maintaining molecular structural similarity and enhancing chemical properties on the benchmark dataset.
Authors Yida Xiong, Kun Li, Jiameng Chen, Hongzhi Zhang, Di Lin, Yan Che, Wenbin Hu
Categories Artificial Intelligence
0
AgriChrono: A Multi-modal Dataset Capturing Crop Growth and Lighting Variability with a Field Robot
Title CN AgriChrono：一种通过田间机器人捕捉作物生长和光照变化的多模态数据集
Keywords 多模态数据集 · 田间机器人 · 3D重建 · 非刚性运动 · 农业AI
Summary 本文提出了AgriChrono，一个基于田间机器人的多模态数据采集平台，用于记录真实农田环境中作物生长过程中的非刚性运动、光照变化和形态演变。平台融合RGB、深度、LiDAR、IMU和位姿等多种传感器数据，累计采集18TB真实数据，覆盖油菜完整的生长周期。作者用其对现有3D重建方法进行基准测试，揭示了在复杂农业场景下高保真动态重建的巨大挑战。数据集已公开发布，旨在推动精准农业中AI模型的发展。
Reason 该论文属于具身智能与农业领域的交叉应用，核心贡献是一个用于农田动态场景重建的多模态数据集及采集平台，研究重点为3D重建、机器人导航与环境感知，与文档图像理解（DIU）或DocVQA等任务无直接关联。尽管涉及多模态数据，但其目标场景、输入模态（如LiDAR、IMU）和任务范式均远离文档理解的核心痛点（如OCR幻觉、文本布局分析、细粒度视觉定位等），无法迁移为提升DIU性能的方法工具。
Abstract Advances in AI and Robotics have accelerated significant initiatives in agriculture, particularly in the areas of robot navigation and 3D digital twin creation. A significant bottleneck impeding this progress is the critical lack of "in-the-wild" datasets that capture the full complexities of real farmland, including non-rigid motion from wind, drastic illumination variance, and morphological changes resulting from growth. This data gap fundamentally limits research on robust AI models for autonomous field navigation and scene-level dynamic 3D reconstruction. In this paper, we present AgriChrono, a modular robotic data collection platform and multi-modal dataset designed to capture these dynamic farmland conditions. Our platform integrates multiple sensors, enabling remote, time-synchronized acquisition of RGB, Depth, LiDAR, IMU, and Pose data for efficient and repeatable long-term data collection in real-world agricultural environments. We successfully collected 18TB of data over one month, documenting the entire growth cycle of Canola under diverse illumination conditions. We benchmark state-of-the-art 3D reconstruction methods on AgriChrono, revealing the profound challenge of reconstructing high-fidelity, dynamic non-rigid scenes in such farmland settings. This benchmark validates AgriChrono as a critical asset for advancing model generalization, and its public release is expected to significantly accelerate research and development in precision agriculture. The code and dataset are publicly available at: https://github.com/StructuresComp/agri-chrono
Authors Jaehwan Jeong, Tuan-Anh Vu, Mohammad Jony, Shahab Ahmad, Md. Mukhlesur Rahman, Sangpil Kim, M. Khalid Jawed
Categories Artificial Intelligence
0
Hybrid Differential Reward: Combining Temporal Difference and Action Gradients for Efficient Multi-Agent Reinforcement Learning in Cooperative Driving
Title CN 混合微分奖励：结合时序差分与动作梯度用于高效多智能体强化学习在协同驾驶中的应用
Keywords 多智能体强化学习 · 协同驾驶 · 奖励设计 · 时序差分 · 动作梯度
Summary 本文提出了一种混合微分奖励（HDR）机制，结合时序差分和动作梯度来提升多智能体强化学习在协同驾驶任务中的收敛速度与策略稳定性。通过构建全局势函数的时序差分奖励和直接衡量动作边际效用的动作梯度奖励，提高了策略梯度的信噪比。实验表明该方法在多种多智能体算法中均有效提升性能。
Reason 该论文研究的是多智能体强化学习在自动驾驶领域的应用，属于具身智能与机器人控制范畴，与文档图像理解（DIU）或视觉语言模型（VLM）在文档处理中的核心技术无关。尽管其提出的HDR机制可能在强化学习理论上有一定价值，但无法迁移解决DIU中的OCR幻觉、复杂排版、细粒度定位或长文档推理等核心问题。
Abstract In multi-vehicle cooperative driving tasks involving high-frequency continuous control, traditional state-based reward functions suffer from the issue of vanishing reward differences. This phenomenon results in a low signal-to-noise ratio (SNR) for policy gradients, significantly hindering algorithm convergence and performance improvement. To address this challenge, this paper proposes a novel Hybrid Differential Reward (HDR) mechanism. We first theoretically elucidate how the temporal quasi-steady nature of traffic states and the physical proximity of actions lead to the failure of traditional reward signals. Building on this analysis, the HDR framework innovatively integrates two complementary components: (1) a Temporal Difference Reward (TRD) based on a global potential function, which utilizes the evolutionary trend of potential energy to ensure optimal policy invariance and consistency with long-term objectives; and (2) an Action Gradient Reward (ARG), which directly measures the marginal utility of actions to provide a local guidance signal with a high SNR. Furthermore, we formulate the cooperative driving problem as a Multi-Agent Partially Observable Markov Game (POMDPG) with a time-varying agent set and provide a complete instantiation scheme for HDR within this framework. Extensive experiments conducted using both online planning (MCTS) and Multi-Agent Reinforcement Learning (QMIX, MAPPO, MADDPG) algorithms demonstrate that the HDR mechanism significantly improves convergence speed and policy stability. The results confirm that HDR guides agents to learn high-quality cooperative policies that effectively balance traffic efficiency and safety.
Authors Ye Han, Lijun Zhang, Dejian Meng, Zhuang Zhang
Categories Artificial Intelligence
0
PepEVOLVE: Position-Aware Dynamic Peptide Optimization via Group-Relative Advantage
Title CN PepEVOLVE：通过组相对优势实现位置感知的动态肽优化
Keywords 肽优化 · 强化学习 · 动态掩码 · 多目标优化 · 生物制药
Summary 本文提出PepEVOLVE，一种用于宏环肽优化的位置感知动态框架，通过动态掩码、多臂老虎机路由策略和基于组相对优势的强化学习算法，在未知最优编辑位点的情况下实现多目标优化。在虚拟评估中优于基线方法PepINVENT。
Reason 该论文属于生物信息学与药物设计领域，研究内容为环状肽的动态优化，尽管使用了AI方法，但其任务、数据和应用场景与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用完全无关。未涉及OCR、布局分析、表格识别、视觉表示学习或任何DIU核心问题，无法迁移用于解决文档理解中的幻觉、密集文本、复杂排版等痛点。
Abstract Macrocyclic peptides are an emerging modality that combines biologics-like affinity with small-molecule-like developability, but their vast combinatorial space and multi-parameter objectives make lead optimization slow and challenging. Prior generative approaches such as PepINVENT require chemists to pre-specify mutable positions for optimization, choices that are not always known a priori, and rely on static pretraining and optimization algorithms that limit the model's ability to generalize and effectively optimize peptide sequences. We introduce PepEVOLVE, a position-aware, dynamic framework that learns both where to edit and how to dynamically optimize peptides for multi-objective improvement. PepEVOLVE (i) augments pretraining with dynamic masking and CHUCKLES shifting to improve generalization, (ii) uses a context-free multi-armed bandit router that discovers high-reward residues, and (iii) couples a novel evolving optimization algorithm with group-relative advantage to stabilize reinforcement updates. During in silico evaluations, the router policy reliably learns and concentrates probability on chemically meaningful sites that influence the peptide's properties. On a therapeutically motivated Rev-binding macrocycle benchmark, PepEVOLVE outperformed PepINVENT by reaching higher mean scores (approximately 0.8 vs. 0.6), achieving best candidates with a score of 0.95 (vs. 0.87), and converging in fewer steps under the task of optimizing permeability and lipophilicity with structural constraints. Overall, PepEVOLVE offers a practical, reproducible path to peptide lead optimization when optimal edit sites are unknown, enabling more efficient exploration and improving design quality across multiple objectives.
Authors Trieu Nguyen, Hao-Wei Pang, Shasha Feng
Categories Artificial Intelligence
0
Genomic Next-Token Predictors are In-Context Learners
Title CN 基因组下一个词预测器是上下文学习者
Keywords in-context learning · genomic sequences · next-token prediction · emergent behavior · meta-learning
Summary 本文研究发现，在大规模训练下的基因组序列预测模型（如Evo2）也能在没有显式微调的情况下展现出上下文学习能力，表明这种能力可能不仅限于人类语言，而是可从富含统计结构的序列数据中自然涌现。作者通过设计跨语言和基因组的符号推理任务进行对比实验，验证了这一现象。
Reason 该论文研究的是基因组序列中的上下文学习（ICL）现象，属于生物学序列建模领域，与文档图像理解（DIU）、视觉语言模型（VLM）或文档VQA等任务无直接关联。尽管探讨了ICL的普适性，但未涉及多模态、视觉输入、OCR、布局分析或任何文档理解核心痛点（如幻觉、复杂排版、细粒度定位）。其方法无法直接迁移至DIU场景，且属于非视觉领域的平行应用研究。
Abstract In-context learning (ICL) -- the capacity of a model to infer and apply abstract patterns from examples provided within its input -- has been extensively studied in large language models trained for next-token prediction on human text. In fact, prior work often attributes this emergent behavior to distinctive statistical properties in human language. This raises a fundamental question: can ICL arise organically in other sequence domains purely through large-scale predictive training? To explore this, we turn to genomic sequences, an alternative symbolic domain rich in statistical structure. Specifically, we study the Evo2 genomic model, trained predominantly on next-nucleotide (A/T/C/G) prediction, at a scale comparable to mid-sized LLMs. We develop a controlled experimental framework comprising symbolic reasoning tasks instantiated in both linguistic and genomic forms, enabling direct comparison of ICL across genomic and linguistic models. Our results show that genomic models, like their linguistic counterparts, exhibit log-linear gains in pattern induction as the number of in-context demonstrations increases. To the best of our knowledge, this is the first evidence of organically emergent ICL in genomic sequences, supporting the hypothesis that ICL arises as a consequence of large-scale predictive modeling over rich data. These findings extend emergent meta-learning beyond language, pointing toward a unified, modality-agnostic view of in-context learning.
Authors Nathan Breslow, Aayush Mishra, Mahler Revsine, Michael C. Schatz, Anqi Liu, Daniel Khashabi
Categories Artificial Intelligence
0
Generative AI in Sociological Research: State of the Discipline
Title CN 生成式人工智能在社会学研究中的应用：学科现状
Keywords 生成式AI · 社会学研究 · 学术实践 · 技术采纳 · 研究伦理
Summary 本文通过对433名社会学期刊作者的调查，探讨了生成式AI在社会学研究中的使用频率、用途及研究人员对其的信任度与担忧。结果显示，约三分之一的研究者每周至少使用一次生成式AI，主要用于写作辅助，而在数据收集或分析中的应用较少。尽管使用频率较高，但研究者普遍对AI的社会和环境影响表示担忧，对其输出信任度低，且对其未来影响持分歧态度。
Reason 该论文属于社会科学研究中技术使用的调查性研究，聚焦于学者对生成式AI的态度和使用习惯，不涉及任何文档图像理解（DIU）或视觉语言模型（VLM）的技术创新。属于典型的‘平行应用’类社会调查，无方法论贡献，无法迁移至DIU领域。
Abstract Generative artificial intelligence (GenAI) has garnered considerable attention for its potential utility in research and scholarship, even among those who typically do not rely on computational tools. Early commentators, however, have also articulated concerns about how GenAI usage comes with enormous environmental costs, serious social risks, and a tendency to produce low-quality content. In the midst of both excitement and skepticism, it is crucial to take stock of how GenAI is actually being used. Our study focuses on sociological research as our site, and here we present findings from a survey of 433 authors of articles published in 50 sociology journals in the last five years. The survey provides an overview of the state of the discipline with regard to the use of GenAI by providing answers to fundamental questions: how (much) do scholars use the technology for their research; what are their reasons for using it; and how concerned, trustful, and optimistic are they about the technology? Of the approximately one third ofrespondents who self-report using GenAI at least weekly, the primary uses are for writing assistance and comparatively less so in planning, data collection, or data analysis. In both use and attitudes, there are surprisingly few differences between self-identified computational and non-computational researchers. Generally, respondents are very concerned about the social and environmental consequences of GenAI. Trust in GenAI outputs is low, regardless of expertise or frequency of use. While optimism that GenAI will improve is high, scholars are divided on whether GenAI will have a positive impact on the field.
Authors AJ Alvero, Dustin S. Stoltz, Oscar Stuhler, Marshall Taylor
Categories Artificial Intelligence
0
LLM-DSE: Searching Accelerator Parameters with LLM Agents
Title CN LLM-DSE：使用LLM代理搜索加速器参数
Keywords LLM Agent · Design Space Exploration · High-Level Synthesis · Hardware Optimization · Multi-Agent System
Summary 本文提出LLM-DSE，一个基于大语言模型的多代理框架，用于优化高层次综合（HLS）中的硬件指令参数。该框架包含Explorer、Router、Specialists、Arbitrator和Critic等多个代理，通过协作与工具调用加速设计空间探索。实验表明其在性能和运行时间上优于现有方法。然而，该工作聚焦于计算机体系结构优化，与文档理解无直接关联。
Reason 该论文属于典型的‘平行应用’类型，将LLM Agent框架应用于硬件设计空间探索（HLS/DSE）这一特定工程优化任务。尽管其多Agent架构设计具有一定系统性，但问题域与文档图像理解（DIU）完全无关，既不涉及视觉内容解析、OCR、布局分析，也不解决VLM在文档上的核心痛点如幻觉、细粒度定位或复杂排版建模。方法论上未提出可迁移至DIU领域的通用机制（如推理控制、视觉表示、干预策略等），仅为特定领域（FPGA/DSA优化）的下游应用。
Abstract Even though high-level synthesis (HLS) tools mitigate the challenges of programming domain-specific accelerators (DSAs) by raising the abstraction level, optimizing hardware directive parameters remains a significant hurdle. Existing heuristic and learning-based methods struggle with adaptability and sample efficiency. We present LLM-DSE, a multi-agent framework designed specifically for optimizing HLS directives. Combining LLM with design space exploration (DSE), our explorer coordinates four agents: Router, Specialists, Arbitrator, and Critic. These multi-agent components interact with various tools to accelerate the optimization process. LLM-DSE leverages essential domain knowledge to identify efficient parameter combinations while maintaining adaptability through verbal learning from online interactions. Evaluations on the HLSyn dataset demonstrate that LLM-DSE achieves substantial $2.55\times$ performance gains over state-of-the-art methods, uncovering novel designs while reducing runtime. Ablation studies validate the effectiveness and necessity of the proposed agent interactions. Our code is open-sourced here: https://github.com/Nozidoali/LLM-DSE.
Authors Hanyu Wang, Xinrui Wu, Zijian Ding, Su Zheng, Chengyue Wang, Neha Prakriya, Tony Nowatzki, Yizhou Sun, Jason Cong
Categories Artificial Intelligence
0
Less is More: Data-Efficient Adaptation for Controllable Text-to-Video Generation
Title CN 少即是多：用于可控文本到视频生成的数据高效适配
Keywords 文本到视频生成 · 数据高效微调 · 可控生成 · 扩散模型 · 合成数据
Summary 本文提出一种数据高效的微调策略，使大规模文本到视频扩散模型能够从稀疏且低质量的合成数据中学习新的生成控制能力（如快门速度、光圈等）。实验表明，使用此类简单数据微调的效果甚至优于使用高保真真实数据训练的模型，并提供了对该现象的定性和定量解释框架。
Reason 该论文研究的是文本到视频生成中的可控性问题，属于纯视频生成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法聚焦于利用低质量合成数据进行模型微调，解决的是视频生成控制参数的学习问题，无法迁移至OCR幻觉、复杂排版理解、细粒度定位或长文档推理等DIU核心痛点。
Abstract Fine-tuning large-scale text-to-video diffusion models to add new generative controls, such as those over physical camera parameters (e.g., shutter speed or aperture), typically requires vast, high-fidelity datasets that are difficult to acquire. In this work, we propose a data-efficient fine-tuning strategy that learns these controls from sparse, low-quality synthetic data. We show that not only does fine-tuning on such simple data enable the desired controls, it actually yields superior results to models fine-tuned on photorealistic "real" data. Beyond demonstrating these results, we provide a framework that justifies this phenomenon both intuitively and quantitatively.
Authors Shihan Cheng, Nilesh Kulkarni, David Hyde, Dmitriy Smirnov
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Learning to Debug: LLM-Organized Knowledge Trees for Solving RTL Assertion Failures
Title CN 学会调试：用于解决RTL断言故障的LLM组织知识树
Keywords 知识树 · 调试自动化 · 大语言模型 · 硬件验证 · 断言失败
Summary 本文提出GROVE框架，通过构建由LLM组织的知识树来系统化管理硬件调试知识，以解决RTL断言故障。该方法从历史案例中提炼知识并构建成树形结构，在测试时通过预算感知的迭代搜索匹配适用知识项，辅助LLM生成修复建议。实验表明其在pass@1和pass@5指标上均有提升。
Reason 该论文属于硬件验证领域的特定应用，研究如何利用LLM组织知识树来解决RTL（寄存器传输级）设计中的断言失败问题。尽管其方法涉及知识结构化与迭代推理，但整体任务与文档图像理解（DIU）或视觉-语言模型在文档上的应用无直接关联。未触及OCR幻觉、密集文本识别、复杂版式分析等DIU核心痛点，且为垂直领域的方法封装，属于‘平行下游应用’，无法迁移为提升DIU性能的通用方法。
Abstract Debugging is the dominant cost in modern hardware verification, where assertion failures are among the most frequent and expensive to resolve. While Large Language Models (LLMs) show promise, they often fail to capture the precise, reusable expertise that engineers apply, leading to inaccurate responses. We propose GROVE, a hierarchical knowledge management framework that learns and organizes reusable debugging expertise into an LLM-organized knowledge tree for solving assertion failures. GROVE distills debugging knowledge from prior cases and organizes it into a vertical tree of configurable depth, with each node encoding a concise knowledge item and explicit applicability conditions. During training, GROVE uses a parallel, gradient-free loop where an LLM proposes tree modifications as structured JSON edits by learning from the cases. At test time, a budget-aware iterative zoom is performed to navigate the tree, retrieving a small set of applicable knowledge items that guide a base LLM's hypothesis generation and fix proposals. Evaluated on a suite of assertion-failure cases, GROVE delivers consistent gains in pass@1 and pass@5, demonstrating the value of structured knowledge evolution.
Authors Yunsheng Bai, Haoxing Ren
Categories Artificial Intelligence
0
Unified Class and Domain Incremental Learning with Mixture of Experts for Indoor Localization
Title CN 基于专家混合模型的室内定位统一类增量与领域增量学习
Keywords Continual Learning · Indoor Localization · Mixture of Experts · Domain Shift · Class Incremental
Summary 本文提出MOELO框架，用于解决室内定位中的类增量和领域增量学习问题。通过专家混合架构和基于等角紧框架的门控机制，实现轻量、低延迟且抗遗忘的持续学习。实验表明其在多种建筑和设备上显著优于现有方法。
Reason 该论文研究的是室内定位中的持续学习问题，属于具身智能与移动感知领域，与文档图像理解（DIU）或视觉语言模型在文档上的应用无直接关联。其方法聚焦于设备异构性和环境变化下的定位性能保持，无法迁移到OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点。
Abstract Indoor localization using machine learning has gained traction due to the growing demand for location-based services. However, its long-term reliability is hindered by hardware/software variations across mobile devices, which shift the model's input distribution to create domain shifts. Further, evolving indoor environments can introduce new locations over time, expanding the output space to create class shifts, making static machine learning models ineffective over time. To address these challenges, we propose a novel unified continual learning framework for indoor localization called MOELO that, for the first time, jointly addresses domain-incremental and class-incremental learning scenarios. MOELO enables a lightweight, robust, and adaptive localization solution that can be deployed on resource-limited mobile devices and is capable of continual learning in dynamic, heterogeneous real-world settings. This is made possible by a mixture-of-experts architecture, where experts are incrementally trained per region and selected through an equiangular tight frame based gating mechanism ensuring efficient routing, and low-latency inference, all within a compact model footprint. Experimental evaluations show that MOELO achieves improvements of up to 25.6x in mean localization error, 44.5x in worst-case localization error, and 21.5x lesser forgetting compared to state-of-the-art frameworks across diverse buildings, mobile devices, and learning scenarios.
Authors Akhil Singampalli, Sudeep Pasricha
Categories Artificial Intelligence
0
Toward explainable AI approaches for breast imaging: adapting foundation models to diverse populations
Title CN 迈向乳腺影像的可解释AI方法：将基础模型适应于多样化人群
Keywords 医学影像 · 基础模型 · BI-RADS分类 · 多模态学习 · 可解释AI
Summary 该研究基于BiomedCLIP基础模型，针对多模态乳腺影像数据（包括合成2D、数字乳腺X线摄影和断层合成）进行BI-RADS乳腺密度自动分类。通过加权对比学习解决类别不平衡问题，在内部测试中取得了较高的AUC值，并在外部分布数据集RSNA和EMBED上验证了泛化能力。GradCAM可视化显示模型关注临床相关区域，具备一定可解释性。
Reason 该论文属于医学影像领域的特定应用，研究的是乳腺密度分类任务，使用BiomedCLIP模型在乳腺X光图像上进行BI-RADS分类。尽管涉及视觉-语言模型和可解释性（如GradCAM），但其核心是垂直领域（breast imaging）的应用型工作，未提出可用于文档图像理解（如DocVQA、OCR-free建模、排版分析等）的通用方法或架构创新。属于典型的‘平行下游应用’，不符合本筛选标准中对方法论创新的要求。
Abstract Foundation models hold promise for specialized medical imaging tasks, though their effectiveness in breast imaging remains underexplored. This study leverages BiomedCLIP as a foundation model to address challenges in model generalization. BiomedCLIP was adapted for automated BI-RADS breast density classification using multi-modality mammographic data (synthesized 2D images, digital mammography, and digital breast tomosynthesis). Using 96,995 images, we compared single-modality (s2D only) and multi-modality training approaches, addressing class imbalance through weighted contrastive learning. Both approaches achieved similar accuracy (multi-modality: 0.74, single-modality: 0.73), with the multi-modality model offering broader applicability across different imaging modalities and higher AUC values consistently above 0.84 across BI-RADS categories. External validation on the RSNA and EMBED datasets showed strong generalization capabilities (AUC range: 0.80-0.93). GradCAM visualizations confirmed consistent and clinically relevant attention patterns, highlighting the models interpretability and robustness. This research underscores the potential of foundation models for breast imaging applications, paving the way for future extensions for diagnostic tasks.
Authors Guilherme J. Cavalcante, José Gabriel A. Moreira, Gabriel A. B. do Nascimento, Vincent Dong, Alex Nguyen, Thaís G. do Rêgo, Yuri Malheiros, Telmo M. Silva Filho, Carla R. Zeballos Torrez, James C. Gee, Anne Marie McCarthy, Andrew D. A. Maidment, Bruno Barufaldi
Comment 5 pages, 3 figures
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
FalconWing: An Ultra-Light Indoor Fixed-Wing UAV Platform for Vision-Based Autonomy
Title CN FalconWing：一种用于视觉自主的超轻型室内固定翼无人机平台
Keywords 无人机 · 视觉自主 · 3D高斯点阵 · 仿真 · 零样本迁移
Summary 本文提出了一种重量仅为150克的室内固定翼无人机平台FalconWing，专为视觉驱动的自主飞行设计。系统包含轻量化硬件（137克机身和9克摄像头）和外部计算架构，并配备基于光真实感3D高斯点阵（GSplat）的仿真器，用于开发和评估视觉控制器。作者通过两个案例验证了系统性能：在领航-跟随任务中，基于模仿学习训练的控制器在模拟环境中实现了100%的跟踪成功率；在自动降落任务中，纯模拟训练的控制器实现了80%的实机零样本迁移成功率。所有设计材料将在发表后开源。
Reason 该论文属于具身智能与机器人领域，研究内容为面向视觉自主的超轻型无人机硬件平台及仿真系统设计，涉及飞行控制、3D高斯点阵仿真和零样本迁移，但完全不涉及文档图像理解、文本识别、多模态语言模型或相关DIU任务。其技术路线无法直接迁移用于解决OCR幻觉、复杂排版解析或长文档推理等文档理解核心问题。
Abstract We introduce FalconWing, an ultra-light (150 g) indoor fixed-wing UAV platform for vision-based autonomy. Controlled indoor environment enables year-round repeatable UAV experiment but imposes strict weight and maneuverability limits on the UAV, motivating our ultra-light FalconWing design. FalconWing couples a lightweight hardware stack (137g airframe with a 9g camera) and offboard computation with a software stack featuring a photorealistic 3D Gaussian Splat (GSplat) simulator for developing and evaluating vision-based controllers. We validate FalconWing on two challenging vision-based aerial case studies. In the leader-follower case study, our best vision-based controller, trained via imitation learning on GSplat-rendered data augmented with domain randomization, achieves 100% tracking success across 3 types of leader maneuvers over 30 trials and shows robustness to leader's appearance shifts in simulation. In the autonomous landing case study, our vision-based controller trained purely in simulation transfers zero-shot to real hardware, achieving an 80% success rate over ten landing trials. We will release hardware designs, GSplat scenes, and dynamics models upon publication to make FalconWing an open-source flight kit for engineering students and research labs.
Authors Yan Miao, Will Shen, Hang Cui, Sayan Mitra
Categories Artificial Intelligence
0
Point of Order: Action-Aware LLM Persona Modeling for Realistic Civic Simulation Submitted to ACL 2026
Title CN 秩序之点：用于真实公民模拟的动作感知LLM角色建模
Keywords 语言模型 · 角色建模 · 自动语音识别 · 社会模拟 · 公共事务
Summary 本文提出一种将Zoom会议录像转化为带有说话人身份和行为标签（如[提出动议]）的结构化转录数据的方法，并基于此构建能模拟真实公共议事过程的LLM角色模型。在法院听证会、校董会和市政会议三个数据集上验证了方法的有效性，显著提升了生成内容的真实性和可区分性。
Reason 该论文属于平行下游应用，其核心是利用LLM进行特定领域（政府会议）的多智能体模拟，虽然使用了ASR和角色建模技术，但并未提出可用于文档图像理解（DIU）的底层方法。任务聚焦于语音转录与社会行为仿真，与DocVQA、OCR、布局分析等文档理解核心问题无关。
Abstract Large language models offer opportunities to simulate multi-party deliberation, but realistic modeling remains limited by a lack of speaker-attributed data. Transcripts produced via automatic speech recognition (ASR) assign anonymous speaker labels (e.g., Speaker_1), preventing models from capturing consistent human behavior. This work introduces a reproducible pipeline to transform public Zoom recordings into speaker-attributed transcripts with metadata like persona profiles and pragmatic action tags (e.g., [propose_motion]). We release three local government deliberation datasets: Appellate Court hearings, School Board meetings, and Municipal Council sessions. Fine-tuning LLMs to model specific participants using this "action-aware" data produces a 67% reduction in perplexity and nearly doubles classifier-based performance metrics for speaker fidelity and realism. Turing-style human evaluations show our simulations are often indistinguishable from real deliberations, providing a practical and scalable method for complex realistic civic simulations.
Authors Scott Merrill, Shashank Srivastava
Comment 8 pages (29 pages including appendix), 18 figures. Code and datasets are available at https://github.com/smerrillunc/action-aware-llms. Submitted to ACL 2026
Categories Artificial Intelligence, Computation and Language
0
REXO: Indoor Multi-View Radar Object Detection via 3D Bounding Box Diffusion AAAI 2026
Title CN REXO：基于3D边界框扩散的室内多视角雷达目标检测
Keywords 雷达目标检测 · 3D扩散模型 · 多视角感知 · 隐式特征关联 · AAAI2026
Summary 本文提出REXO，一种用于室内多视角雷达目标检测的新方法，通过将DiffusionDet的2D边界框扩散扩展到3D雷达空间，实现显式的跨视角雷达特征关联，并利用人体接触地面的先验知识减少扩散参数量。在HIBER和MMVR数据集上显著超越现有方法。
Reason 该论文研究的是室内多视角雷达感知中的3D目标检测问题，属于自动驾驶或具身智能相关的雷达视觉任务，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其核心技术为3D边界框扩散和雷达特征关联，无法迁移解决文档理解中的OCR幻觉、复杂排版、细粒度定位等核心痛点。
Abstract Multi-view indoor radar perception has drawn attention due to its cost-effectiveness and low privacy risks. Existing methods often rely on {implicit} cross-view radar feature association, such as proposal pairing in RFMask or query-to-feature cross-attention in RETR, which can lead to ambiguous feature matches and degraded detection in complex indoor scenes. To address these limitations, we propose \textbf{REXO} (multi-view Radar object dEtection with 3D bounding boX diffusiOn), which lifts the 2D bounding box (BBox) diffusion process of DiffusionDet into the 3D radar space. REXO utilizes these noisy 3D BBoxes to guide an {explicit} cross-view radar feature association, enhancing the cross-view radar-conditioned denoising process. By accounting for prior knowledge that the person is in contact with the ground, REXO reduces the number of diffusion parameters by determining them from this prior. Evaluated on two open indoor radar datasets, our approach surpasses state-of-the-art methods by a margin of +4.22 AP on the HIBER dataset and +11.02 AP on the MMVR dataset.
Authors Ryoma Yataka, Pu Perry Wang, Petros Boufounos, Ryuhei Takahashi
Comment 26 pages, Accepted to AAAI 2026; Code to be released
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
Pillar-0: A New Frontier for Radiology Foundation Models
Title CN Pillar-0：放射学基础模型的新前沿
Keywords 放射学 · 基础模型 · 医学影像 · LLM辅助标注 · AUROC
Summary 本文提出了Pillar-0，一种针对放射学任务的基础模型，基于大量CT和MRI数据进行预训练，并结合RATE框架利用LLM提取结构化放射学发现标签。在多种内部和外部测试集中表现出优于现有医疗视觉模型的性能，尤其在肺癌风险预测和脑出血检测上表现突出。同时强调临床实用性与评估体系的构建。
Reason 该论文聚焦于医学影像领域的放射学任务，属于垂直领域应用（如CT、MRI分析），尽管使用了大规模预训练和LLM辅助标注框架RATE，但其核心是将现有VLM技术应用于特定医疗场景，属于典型的‘平行下游应用’。未提出可迁移至文档图像理解（DIU）的底层方法或架构创新，且不涉及OCR幻觉、复杂排版、细粒度定位等DIU核心问题。
Abstract Radiology plays an integral role in modern medicine, yet rising imaging volumes have far outpaced workforce growth. Foundation models offer a path toward assisting with the full spectrum of radiology tasks, but existing medical models remain limited: they process volumetric CT and MRI as low-fidelity 2D slices, discard critical grayscale contrast information, and lack evaluation frameworks that reflect real clinical practice. We introduce Pillar-0, a radiology foundation model pretrained on 42,990 abdomen-pelvis CTs, 86,411 chest CTs, 14,348 head CTs, and 11,543 breast MRIs from a large academic center, together with RATE, a scalable framework that extracts structured labels for 366 radiologic findings with near-perfect accuracy using LLMs. Across internal test sets of 14,230 abdomen-pelvis CTs, 10,646 chest CTs, 4,906 head CTs, and 1,585 breast MRIs, Pillar-0 establishes a new performance frontier, achieving mean AUROCs of 86.4, 88.0, 90.1, and 82.9, outperforming MedGemma (Google), MedImageInsight (Microsoft), Lingshu (Alibaba), and Merlin (Stanford) by 7.8-15.8 AUROC points and ranking best in 87.2\% (319/366) tasks. Pillar-0 similarly outperforms all baselines in an external validation on the Stanford Abdominal CT dataset, including Merlin (82.2 vs 80.6 AUROC). Pillar-0 extends to tasks beyond its pretraining, such as long-horizon lung cancer risk prediction, where it improves upon the state-of-the-art Sybil by 3.0 C-index points on NLST, and generalizes with gains of 5.9 (MGH) and 1.9 (CGMH). In brain hemorrhage detection, Pillar-0 obtained a >95 AUROC when using only 1/20th of the data of the next most sample efficient baseline. Pillar-0 and RATE together provide an open, clinically rigorous foundation for building high-performance radiology systems, enabling applications that were previously infeasible due to computational, data, and evaluation constraints.
Authors Kumar Krishna Agrawal, Longchao Liu, Long Lian, Michael Nercessian, Natalia Harguindeguy, Yufu Wu, Peter Mikhael, Gigin Lin, Lecia V. Sequist, Florian Fintelmann, Trevor Darrell, Yutong Bai, Maggie Chung, Adam Yala
Categories Artificial Intelligence, Computer Vision and Pattern Recognition
0
FERA: Bridging the Semantic Gap in Foil Fencing via Kinematic Pose Recognition and Explainable Rule Reasoning
Title CN FERA：通过运动姿态识别与可解释规则推理弥合花剑击剑中的语义鸿沟
Keywords 动作识别 · 可解释性 · Transformer · 规则推理 · 体育视频分析
Summary 本文提出FERA系统，用于从单目视频中识别花剑击剑的动作并生成可解释的判罚建议。系统首先提取2D姿态，转化为101维运动学表示，再通过编码器-only Transformer（FERA-MDT）预测脚步和剑招动作，最后将预测结果输入语言模型（FERA-LM），结合规则生成判决和解释。虽然实现了视觉感知与逻辑推理的解耦，但应用场景完全偏离文档理解。
Reason 该论文研究的是体育视频中的动作识别与规则推理任务，属于视频理解领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管其采用了多模态框架（视觉模型+语言模型）进行语义桥接，但核心场景是击剑动作的时序建模和规则应用，无法迁移解决OCR幻觉、复杂排版、细粒度文本定位等文档理解核心问题。
Abstract Foil fencing presents a unique multimedia challenge: actions are extremely fast, interactions are subtle, and the final right-of-way decision relies on complex linguistic rules applied to visual data. We present FERA (Fencing Referee Assistant), a pose-based framework that bridges the gap between pixel-level perception and semantic rule reasoning. From monocular video, FERA extracts 2D poses, converts them into a compact 101-dimensional kinematic representation, and applies an encoder-only Transformer (FERA-MDT) to predict multi-label footwork and blade actions. Crucially, these predictions serve as semantic tokens for a downstream language model (FERA-LM) to generate explainable verdicts. Training treats left and right fencers symmetrically by creating two single-fencer sequences per clip. At inference, FERA-MDT uses dynamic temporal windowing to handle variable-length clips, and paired predictions are passed to FERA-LM, which applies encoded right-of-way rules to generate prototype decisions and short explanations. On 1,734 clips (2,386 move instances) from professional bouts, FERA-MDT reaches a macro-F1 of 0.549 under 5-fold cross-validation, outperforming a BiLSTM, TCN, and baseline Transformer. Furthermore, we demonstrate that our structured outputs enable a language model to perform logical rule reasoning, effectively decoupling visual perception from rule application. FERA provides the first dataset and benchmark for this cross-modal action understanding task.
Authors Ziwen Chen, Zhong Wang
Comment Clarified some unclear methods and added more info in related works section
Categories Artificial Intelligence
0
LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs
Title CN LiveCLKTBench：迈向多语言大模型中跨语言知识迁移的可靠评估
Keywords 多语言大模型 · 知识迁移 · 评测基准 · 语言距离 · 时间敏感知识
Summary 本文提出LiveCLKTBench，一种用于评估大语言模型在不同语言间进行知识迁移能力的自动化评测流程。该方法通过提取时效性知识实体并生成跨语言事实问题，试图区分真实的知识迁移与预训练数据污染。实验揭示了语言距离对迁移效果的影响以及迁移的不对称性。
Reason 该论文聚焦于多语言大语言模型中的跨语言知识迁移评估，属于纯NLP范畴，且研究任务为语言间的知识转移评测，与文档图像理解（DIU）、视觉信息处理、OCR、VLM或多模态文档推理无关。未涉及任何视觉输入、布局分析、表格识别或文档VQA等核心DIU任务，也无法提供可迁移至文档理解领域的底层方法（如视觉压缩、高分辨率处理、去幻觉机制等）。
Abstract Evaluating cross-lingual knowledge transfer in large language models is challenging, as correct answers in a target language may arise either from genuine transfer or from prior exposure during pre-training. We present LiveCLKTBench, an automated generation pipeline specifically designed to isolate and measure cross-lingual knowledge transfer. Our pipeline identifies self-contained, time-sensitive knowledge entities from real-world domains, filters them based on temporal occurrence, and verifies them against the model's knowledge. The documents of these valid entities are then used to generate factual questions, which are translated into multiple languages to evaluate transferability across linguistic boundaries. Using LiveCLKTBench, we evaluate several LLMs across five languages and observe that cross-lingual transfer is strongly influenced by linguistic distance and often asymmetric across language directions. While larger models improve transfer, the gains diminish with scale and vary across domains. These findings provide new insights into multilingual transfer and demonstrate the value of LiveCLKTBench as a reliable benchmark for future research.
Authors Pei-Fu Guo, Yun-Da Tsai, Chun-Chia Hsu, Kai-Xin Chen, Ya-An Tsai, Kai-Wei Chang, Nanyun Peng, Mi-Yen Yeh, Shou-De Lin
Categories Artificial Intelligence, Computation and Language
0
$Δ$-ML Ensembles for Selecting Quantum Chemistry Methods to Compute Intermolecular Interactions NeurIPS ML4PS 2025
Title CN 用于选择量子化学方法计算分子间相互作用的Δ-ML集成模型
Keywords Δ-ML · 量子化学 · 误差预测 · 集成学习 · 分子间相互作用
Summary 本文提出了一种基于Δ-ML模型集成的框架，利用预训练原子对神经网络提取特征，预测不同量子化学方法相对于CCSD(T)/CBS金标准的误差，从而在保证精度的前提下选择计算成本更低的方法。实验在扩展的BioFragment数据集上进行，结果表明该方法误差极低，并能揭示不同理论方法之间的分组关系。
Reason 该论文属于量子化学与机器学习交叉领域，研究目标是通过Δ-ML模型集成选择高精度且计算高效的方法来预测分子间相互作用能。其核心任务与文档图像理解（DIU）、多模态大模型（VLM）、OCR、布局分析、表格识别等完全无关，属于典型的科学计算应用，不提供任何可迁移至文档理解领域的底层方法或架构创新。
Abstract Ab initio quantum chemical methods for accurately computing interactions between molecules have a wide range of applications but are often computationally expensive. Hence, selecting an appropriate method based on accuracy and computational cost remains a significant challenge due to varying performance of methods. In this work, we propose a framework based on an ensemble of $Δ$-ML models trained on features extracted from a pre-trained atom-pairwise neural network to predict the error of each method relative to all other methods including the ``gold standard'' coupled cluster with single, double, and perturbative triple excitations at the estimated complete basis set limit [CCSD(T)/CBS]. Our proposed approach provides error estimates across various levels of theories and identifies the computationally efficient approach for a given error range utilizing only a subset of the dataset. Further, this approach allows comparison between various theories. We demonstrate the effectiveness of our approach using an extended BioFragment dataset, which includes the interaction energies for common biomolecular fragments and small organic dimers. Our results show that the proposed framework achieves very small mean-absolute-errors below 0.1 kcal/mol regardless of the given method. Furthermore, by analyzing all-to-all $Δ$-ML models for present levels of theory, we identify method groupings that align with theoretical hypotheses, providing evidence that $Δ$-ML models can easily learn corrections from any level of theory to any other level of theory.
Authors Austin M. Wallace, C. David Sherrill, Giri P. Krishnan
Comment NeurIPS ML4PS 2025
Categories Artificial Intelligence
0
GP-GPT: Large Language Model for Gene-Phenotype Mapping
Title CN GP-GPT：用于基因-表型映射的大型语言模型
Keywords 大型语言模型 · 基因-表型映射 · 生物医学文本分析 · 领域适配 · 信息检索
Summary 本文提出了GP-GPT，一个专用于基因-表型知识表示和基因组关系分析的大型语言模型。通过在包含300多万个基因组学、蛋白质组学和医学遗传学术语的大规模语料库上进行两阶段微调，模型在基因信息检索和关系判定任务上优于Llama2、Llama3和GPT-4等主流LLM。研究表明LLM在基因-表型研究中具有应用潜力。
Reason 该论文属于典型的‘平行下游应用’，即将现有LLM技术应用于生物医学领域的基因-表型关系分析。其核心是领域适配而非方法论创新，未提出可用于文档图像理解（如OCR去幻觉、复杂排版处理、视觉-文本对齐等）的新架构、新训练策略或新推理机制。研究内容与文档理解任务无直接关联，也无法迁移为解决DIU核心痛点的通用方法。
Abstract Pre-trained large language models(LLMs) have attracted increasing attention in biomedical domains due to their success in natural language processing. However, the complex traits and heterogeneity of multi-sources genomics data pose significant challenges when adapting these models to the bioinformatics and biomedical field. To address these challenges, we present GP-GPT, the first specialized large language model for genetic-phenotype knowledge representation and genomics relation analysis. Our model is fine-tuned in two stages on a comprehensive corpus composed of over 3,000,000 terms in genomics, proteomics, and medical genetics, derived from multiple large-scale validated datasets and scientific publications. GP-GPT demonstrates proficiency in accurately retrieving medical genetics information and performing common genomics analysis tasks, such as genomics information retrieval and relationship determination. Comparative experiments across domain-specific tasks reveal that GP-GPT outperforms state-of-the-art LLMs, including Llama2, Llama3 and GPT-4. These results highlight GP-GPT's potential to enhance genetic disease relation research and facilitate accurate and efficient analysis in the fields of genomics and medical genetics. Our investigation demonstrated the subtle changes of bio-factor entities' representations in the GP-GPT, which suggested the opportunities for the application of LLMs to advancing gene-phenotype research.
Authors Yanjun Lyu, Zihao Wu, Lu Zhang, Jing Zhang, Yiwei Li, Wei Ruan, Zhengliang Liu, Zeyu Zhang, Xiang Li, Rongjie Liu, Chao Huang, Wentao Li, Tianming Liu, Dajiang Zhu
Categories Artificial Intelligence, Computation and Language
0
AEGIS: Preserving privacy of 3D Facial Avatars with Adversarial Perturbations
Title CN 利用对抗性扰动保护3D面部虚拟形象的隐私
Keywords 3D Gaussian Splatting · 对抗性扰动 · 隐私保护 · 去身份化 · 面部识别
Summary 本文提出AEGIS，首个针对基于3D高斯点阵的面部虚拟形象的隐私保护框架。通过在颜色系数上添加由人脸识别网络引导的对抗性扰动，实现在多视角下一致的身份隐藏，同时保持外观真实性和关键面部属性（如年龄、性别、情绪）。实验显示其能将人脸识别和检索准确率降至0%，同时保持高质量视觉效果（SSIM=0.9555，PSNR=35.52dB）。
Reason 该论文研究的是3D面部虚拟形象的隐私保护问题，属于具身智能与3D视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心技术为在3D高斯点阵表示上施加对抗扰动以实现去身份化，不涉及OCR、文本理解、多模态文档建模或视觉-语言推理等DIU核心任务。方法难以迁移到文档理解场景，且主题偏离本领域关注的文本密集、排版复杂、长上下文推理等痛点。
Abstract The growing adoption of photorealistic 3D facial avatars, particularly those utilizing efficient 3D Gaussian Splatting representations, introduces new risks of online identity theft, especially in systems that rely on biometric authentication. While effective adversarial masking methods have been developed for 2D images, a significant gap remains in achieving robust, viewpoint-consistent identity protection for dynamic 3D avatars. To address this, we present AEGIS, the first privacy-preserving identity masking framework for 3D Gaussian Avatars that maintains the subject's perceived characteristics. Our method aims to conceal identity-related facial features while preserving the avatar's perceptual realism and functional integrity. AEGIS applies adversarial perturbations to the Gaussian color coefficients, guided by a pre-trained face verification network, ensuring consistent protection across multiple viewpoints without retraining or modifying the avatar's geometry. AEGIS achieves complete de-identification, reducing face retrieval and verification accuracy to 0%, while maintaining high perceptual quality (SSIM = 0.9555, PSNR = 35.52 dB). It also preserves key facial attributes such as age, race, gender, and emotion, demonstrating strong privacy protection with minimal visual distortion.
Authors Dawid Wolkiewicz, Anastasiya Pechko, Przemysław Spurek, Piotr Syga
Categories Artificial Intelligence
0
AI- and Ontology-Based Enhancements to FMEA for Advanced Systems Engineering: Current Developments and Future Directions
Title CN 基于人工智能和本体的FMEA在先进系统工程中的增强：当前进展与未来方向
Keywords FMEA · 人工智能 · 本体 · 系统工程 · 知识表示
Summary 本文综述了如何利用人工智能（如机器学习和自然语言处理）和本体技术来增强传统FMEA方法，使其更智能、数据驱动且具备语义推理能力。重点包括自动化故障预测、优先级排序、知识提取以及与模型化系统工程（MBSE）的集成。虽然提到了大语言模型的应用，但仅限于工程知识处理场景，不涉及文档图像或多模态理解。
Reason 该论文属于特定领域的应用型综述，聚焦于将AI和本体技术应用于FMEA（故障模式与影响分析）这一工程可靠性方法，属于典型的‘平行下游应用’。其内容未涉及文档图像理解（DIU）、视觉表示学习、多模态建模或VLM架构改进等核心技术问题，也未提出可迁移至DIU领域的底层方法创新。
Abstract This article presents a state-of-the-art review of recent advances aimed at transforming traditional Failure Mode and Effects Analysis (FMEA) into a more intelligent, data-driven, and semantically enriched process. As engineered systems grow in complexity, conventional FMEA methods, largely manual, document-centric, and expert-dependent, have become increasingly inadequate for addressing the demands of modern systems engineering. We examine how techniques from Artificial Intelligence (AI), including machine learning and natural language processing, can transform FMEA into a more dynamic, data-driven, intelligent, and model-integrated process by automating failure prediction, prioritisation, and knowledge extraction from operational data. In parallel, we explore the role of ontologies in formalising system knowledge, supporting semantic reasoning, improving traceability, and enabling cross-domain interoperability. The review also synthesises emerging hybrid approaches, such as ontology-informed learning and large language model integration, which further enhance explainability and automation. These developments are discussed within the broader context of Model-Based Systems Engineering (MBSE) and function modelling, showing how AI and ontologies can support more adaptive and resilient FMEA workflows. We critically analyse a range of tools, case studies, and integration strategies, while identifying key challenges related to data quality, explainability, standardisation, and interdisciplinary adoption. By leveraging AI, systems engineering, and knowledge representation using ontologies, this review offers a structured roadmap for embedding FMEA within intelligent, knowledge-rich engineering environments.
Authors Haytham Younus, Sohag Kabir, Felician Campean, Pascal Bonnaud, David Delaux
Comment This manuscript is based on research undertaken by our doctoral student at the University of Bradford. The associated PhD thesis has been formally submitted to the University and is currently awaiting final examination. The review article is being shared on arXiv to make the review accessible to the research community while the thesis examination process is ongoing
Categories Artificial Intelligence
0
Mind the Gap: Aligning Knowledge Bases with User Needs to Enhance Mental Health Retrieval
Title CN 关注差距：通过将知识库与用户需求对齐以增强心理健康信息检索
Keywords 知识库扩充 · 检索增强生成 · 心理健康 · 信息检索 · 自然语言处理
Summary 本文提出一种基于用户生成内容识别知识库中主题缺口的方法，并用于指导心理健康领域知识库的有向扩充。实验表明，相比随机扩充，有针对性的语料扩展能以更小的数据增量达到接近最优的检索性能。然而，该工作聚焦于纯文本RAG系统的数据优化，不涉及任何视觉或文档结构信息处理。
Reason 该论文属于典型的‘平行下游应用’，其核心是将现有RAG技术应用于心理健康信息检索场景，属于特定领域的应用型研究。尽管提出了‘gap-informed’的语料扩充策略，但方法论上未涉及文档图像理解（DIU）或视觉语言模型（VLM）在OCR、布局分析、表格识别等核心任务上的改进，也无法迁移用于解决DIU中的OCR幻觉、密集文本处理、复杂排版解析等关键问题。
Abstract Access to reliable mental health information is vital for early help-seeking, yet expanding knowledge bases is resource-intensive and often misaligned with user needs. This results in poor performance of retrieval systems when presented concerns are not covered or expressed in informal or contextualized language. We present an AI-based gap-informed framework for corpus augmentation that authentically identifies underrepresented topics (gaps) by overlaying naturalistic user data such as forum posts in order to prioritize expansions based on coverage and usefulness. In a case study, we compare Directed (gap-informed augmentations) with Non-Directed augmentation (random additions), evaluating the relevance and usefulness of retrieved information across four retrieval-augmented generation (RAG) pipelines. Directed augmentation achieved near-optimal performance with modest expansions--requiring only a 42% increase for Query Transformation, 74% for Reranking and Hierarchical, and 318% for Baseline--to reach ~95% of the performance of an exhaustive reference corpus. In contrast, Non-Directed augmentation required substantially larger and thus practically infeasible expansions to achieve comparable performance (232%, 318%, 403%, and 763%, respectively). These results show that strategically targeted corpus growth can reduce content creation demands while sustaining high retrieval and provision quality, offering a scalable approach for building trusted health information repositories and supporting generative AI applications in high-stakes domains.
Authors Amanda Chan, James Jiayu Liu, He Kai, Onno P. Kampman
Comment 25 pages, 3 figures, submitted to NeurIPS 2025 GenAI4Health
Categories Artificial Intelligence
0
Reversing the Lens: Using Explainable AI to Understand Human Expertise
Title CN 反转视角：使用可解释AI理解人类专业知识
Keywords Explainable AI · Human Expertise · Cognitive Modeling · Complex Task Learning · Operator Behavior
Summary 本文提出将可解释AI技术应用于人类操作行为数据分析，通过构建操作员子任务图并使用社区检测和层次聚类等方法，揭示专家在缺乏全局最优解的情况下如何分解问题并发展高效策略。研究聚焦于人类认知演化，而非机器学习模型的文档理解能力提升。
Reason 该论文研究的是利用可解释AI（XAI）方法分析人类在复杂任务（如粒子加速器调优）中的行为与专家经验形成过程，属于心理学与AI交叉的认知科学研究。其任务场景、技术目标与文档图像理解（DIU/DocVQA）完全无关，既不涉及文档结构、文本识别、多模态理解，也无法迁移用于解决OCR幻觉、密集文本处理或复杂排版建模等核心DIU痛点。
Abstract Both humans and machine learning models learn from experience, particularly in safety- and reliability-critical domains. While psychology seeks to understand human cognition, the field of Explainable AI (XAI) develops methods to interpret machine learning models. This study bridges these domains by applying computational tools from XAI to analyze human learning. We modeled human behavior during a complex real-world task -- tuning a particle accelerator -- by constructing graphs of operator subtasks. Applying techniques such as community detection and hierarchical clustering to archival operator data, we reveal how operators decompose the problem into simpler components and how these problem-solving structures evolve with expertise. Our findings illuminate how humans develop efficient strategies in the absence of globally optimal solutions, and demonstrate the utility of XAI-based methods for quantitatively studying human cognition.
Authors Roussel Rahman, Aashwin Ananda Mishra, Wan-Lin Hu
Categories Artificial Intelligence
0
Social and Ethical Risks Posed by General-Purpose LLMs for Settling Newcomers in Canada
Title CN 通用大语言模型在加拿大新移民安置中带来的社会与伦理风险
Keywords 伦理风险 · 通用大语言模型 · AI治理 · 新移民安置 · AI素养
Summary 本文探讨了在加拿大新移民安置服务中使用通用生成式AI（如ChatGPT）可能带来的社会和伦理风险，强调这些工具未针对该领域定制，可能导致对移民和难民的负面影响。作者呼吁提高AI素养，并推动开发符合社区需求、可融入现有工作流的定制化语言模型，同时确保人类监督、可信性和问责机制。
Reason 该论文聚焦于通用大语言模型在社会服务领域的应用风险，属于NLP领域的安全与伦理讨论，且主题为特定社会场景（移民安置）的AI治理问题。未涉及文档图像理解、多模态建模、视觉表示学习或任何DIU核心技术。内容与文档理解任务无关，属于纯NLP的社会影响分析，命中负面清单中的‘无关领域’。
Abstract The non-profit settlement sector in Canada supports newcomers in achieving successful integration. This sector faces increasing operational pressures amidst rising immigration targets, which highlights a need for enhanced efficiency and innovation, potentially through reliable AI solutions. The ad-hoc use of general-purpose generative AI, such as ChatGPT, might become a common practice among newcomers and service providers to address this need. However, these tools are not tailored for the settlement domain and can have detrimental implications for immigrants and refugees. We explore the risks that these tools might pose on newcomers to first, warn against the unguarded use of generative AI, and second, to incentivize further research and development in creating AI literacy programs as well as customized LLMs that are aligned with the preferences of the impacted communities. Crucially, such technologies should be designed to integrate seamlessly into the existing workflow of the settlement sector, ensuring human oversight, trustworthiness, and accountability.
Authors Isar Nejadgholi, Maryam Molamohammadi, Samir Bakhtawar
Comment 26 pages, 8 figures
Categories Artificial Intelligence
0
Learning the Value of Value Learning
Title CN 学习价值学习的价值
Keywords 理性选择 · 价值修正 · 决策理论 · 多智能体系统 · 伦理博弈
Summary 本文扩展了Jeffrey-Bolker决策框架，将价值（values）的动态修正纳入理性选择模型，并证明在多智能体环境中，共同的价值 refinement 可将零和博弈转化为正和互动，实现帕累托改进。研究旨在统一认知与价值层面的理性更新，强调伦理思辨的规范性地位。
Reason 该论文属于人工智能哲学与决策理论范畴，讨论的是价值观的理性修正与多智能体下的伦理博弈，未涉及任何文档图像理解（DIU）、视觉语言模型（VLM）、OCR、布局分析或相关技术。内容与文档理解任务无直接关联，也无法迁移为解决OCR幻觉、复杂排版、细粒度定位等DIU核心痛点的方法工具。
Abstract Standard decision frameworks addresses uncertainty about facts but assumes fixed values. We extend the Jeffrey-Bolker framework to model refinements in values and prove a value-of-information theorem for axiological refinement. In multi-agent settings, we establish that mutual refinement will characteristically transform zero-sum games into positive-sum interactions and yields Pareto-improving Nash bargains. These results show that a framework of rational choice can be extended to model value refinement and its associated benefits. By unifying epistemic and axiological refinement under a single formalism, we broaden the conceptual foundations of rational choice and illuminate the normative status of ethical deliberation.
Authors Alex John London, Aydin Mohseni
Comment 27 pages, 6 figures, mathematical appendix
Categories Artificial Intelligence
0
Hiding in the AI Traffic: Abusing MCP for LLM-Powered Agentic Red Teaming
Title CN 隐藏在AI流量中：滥用MCP进行基于LLM的代理红队攻击
Keywords 红队攻击 · 代理系统 · MCP · 网络安全 · LLM
Summary 本文提出了一种基于模型上下文协议（MCP）的命令与控制（C2）架构，用于协调分布式、自适应的AI红队侦察代理，实现隐蔽的网络渗透测试。该方法通过异步并行操作和实时情报共享，减少检测痕迹，并提升任务执行效率。尽管涉及LLM代理的协调机制，但其应用场景和技术目标均远离文档图像理解领域。
Reason 该论文属于具身智能与网络安全交叉领域，研究利用LLM代理进行渗透测试的指挥与控制架构，完全不涉及文档图像理解（DIU）、视觉表示、多模态对齐或OCR相关技术。内容聚焦于红队攻击的自动化与隐蔽通信，与文档理解的核心痛点（如幻觉、排版、密集文本等）无任何关联，属于无关领域。
Abstract Generative AI is reshaping offensive cybersecurity by enabling autonomous red team agents that can plan, execute, and adapt during penetration tests. However, existing approaches face trade-offs between generality and specialization, and practical deployments reveal challenges such as hallucinations, context limitations, and ethical concerns. In this work, we introduce a novel command & control (C2) architecture leveraging the Model Context Protocol (MCP) to coordinate distributed, adaptive reconnaissance agents covertly across networks. Notably, we find that our architecture not only improves goal-directed behavior of the system as whole, but also eliminates key host and network artifacts that can be used to detect and prevent command & control behavior altogether. We begin with a comprehensive review of state-of-the-art generative red teaming methods, from fine-tuned specialist models to modular or agentic frameworks, analyzing their automation capabilities against task-specific accuracy. We then detail how our MCP-based C2 can overcome current limitations by enabling asynchronous, parallel operations and real-time intelligence sharing without periodic beaconing. We furthermore explore advanced adversarial capabilities of this architecture, its detection-evasion techniques, and address dual-use ethical implications, proposing defensive measures and controlled evaluation in lab settings. Experimental comparisons with traditional C2 show drastic reductions in manual effort and detection footprint. We conclude with future directions for integrating autonomous exploitation, defensive LLM agents, predictive evasive maneuvers, and multi-agent swarms. The proposed MCP-enabled C2 framework demonstrates a significant step toward realistic, AI-driven red team operations that can simulate advanced persistent threats while informing the development of next-generation defensive systems.
Authors Strahinja Janjusevic, Anna Baron Garcia, Sohrob Kazerounian
Comment 23 pages, 9 figures, 3 tables. Submitted as a full paper for review
Categories Artificial Intelligence
0
Liberating Logic in the Age of AI: Going Beyond Programming with Computational Thinking
Title CN 解放人工智能时代的逻辑：超越编程的计算思维
Keywords 计算思维 · AI教育 · 自然语言编程 · 课程改革 · 大语言模型
Summary 本文探讨了在大语言模型和AI编码助手兴起的背景下，传统编程技能的重要性正在被计算思维所取代。作者认为，随着自然语言成为新的‘编程接口’，教育体系需要重新思考如何培养能够设计、验证和批判性使用AI生成结果的新一代人才。文章重点在于课程改革与教学实践，而非技术创新。
Reason 该论文讨论的是AI时代下计算思维教育的转型问题，属于计算机科学教育范畴，未涉及文档图像理解、多模态模型架构、视觉表示学习或任何与DocVQA、OCR、布局分析等相关的技术方法。内容聚焦于编程语言与自然语言在教育中的角色转变，无技术迁移价值。
Abstract Mastering one or more programming languages has historically been the gateway to implementing ideas on a computer. Today, that gateway is widening with advances in large language models (LLMs) and artificial intelligence (AI)-powered coding assistants. What matters is no longer just fluency in traditional programming languages but the ability to think computationally by translating problems into forms that can be solved with computing tools. The capabilities enabled by these AI-augmented tools are rapidly leading to the commoditization of computational thinking, such that anyone who can articulate a problem in natural language can potentially harness computing power via AI. This shift is poised to radically influence how we teach computer science and data science in the United States and around the world. Educators and industry leaders are grappling with how to adapt: What should students learn when the hottest new programming language is English? How do we prepare a generation of computational thinkers who need not code every algorithm manually, but must still think critically, design solutions, and verify AI-augmented results? This paper explores these questions, examining the impact of natural language programming on software development, the emerging distinction between programmers and prompt-crafting problem solvers, the reforms needed in computer science and data science curricula, and the importance of maintaining our fundamental computational science principles in an AI-augmented future. Along the way, we compare approaches and share best practices for embracing this new paradigm in computing education.
Authors Douglas C. Schmidt, Dan Runfola
Comment 15 pages and 17 figures
Categories Artificial Intelligence
0
On the limitation of evaluating machine unlearning using only a single training seed
Title CN 关于仅使用单一训练种子评估机器遗忘的局限性
Keywords 机器遗忘 · 评估偏差 · 训练种子 · 可复现性 · 确定性算法
Summary 本文指出当前机器遗忘算法的评估存在方法论缺陷：多数实验仅基于同一训练模型重复运行，忽略了不同训练种子带来的随机性影响，可能导致评估结果不具代表性。作者强调应跨多个训练种子进行测试以获得更可靠的性能评估。尽管对MU领域有方法论警示意义，但其主题远离文档理解任务。
Reason 该论文研究机器遗忘（Machine Unlearning）的评估方法，属于模型维护与安全领域，与文档图像理解（DIU）或视觉语言模型在文档任务中的应用无直接关联。未涉及OCR、布局分析、表格识别、多模态对齐或视觉表示学习等DIU核心问题，也无法迁移用于解决OCR幻觉、细粒度定位或长文档推理等痛点。
Abstract Machine unlearning (MU) aims to remove the influence of certain data points from a trained model without costly retraining. Most practical MU algorithms are only approximate and their performance can only be assessed empirically. Care must therefore be taken to make empirical comparisons as representative as possible. A common practice is to run the MU algorithm multiple times independently starting from the same trained model. In this work, we demonstrate that this practice can give highly non-representative results because -- even for the same architecture and same dataset -- some MU methods can be highly sensitive to the choice of random number seed used for model training. We illustrate that this is particularly relevant for MU methods that are deterministic, i.e., which always produce the same result when started from the same trained model. We therefore recommend that empirical comparisons of MU algorithms should also reflect the variability across different model training seeds.
Authors Jamie Lanyon, Axel Finke, Petros Andreou, Georgina Cosma
Comment mini paper, 2 figures
Categories Artificial Intelligence
0
Native 3D Editing with Full Attention
Title CN 基于全注意力的原生3D编辑
Keywords 3D编辑 · 指令引导生成 · token拼接 · 多模态学习 · 全注意力
Summary 本文提出了一种原生的3D编辑框架，通过在3D表示空间中直接操作，实现高效的一次前向传播编辑。作者构建了多模态指令引导的3D编辑数据集，并比较了交叉注意力与3D token拼接两种条件控制方式，发现后者更高效且性能更好。实验表明该方法在生成质量、3D一致性和指令遵循方面优于基于2D提升的方法。
Reason 该论文研究的是3D内容编辑，属于3D视觉与生成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心贡献在于构建指令引导的3D编辑数据集和提出3D token拼接策略，但方法面向的是3D几何结构的一致性编辑，无法迁移解决文档理解中的OCR幻觉、复杂排版、细粒度文本定位或长文档推理等关键痛点。
Abstract Instruction-guided 3D editing is a rapidly emerging field with the potential to broaden access to 3D content creation. However, existing methods face critical limitations: optimization-based approaches are prohibitively slow, while feed-forward approaches relying on multi-view 2D editing often suffer from inconsistent geometry and degraded visual quality. To address these issues, we propose a novel native 3D editing framework that directly manipulates 3D representations in a single, efficient feed-forward pass. Specifically, we create a large-scale, multi-modal dataset for instruction-guided 3D editing, covering diverse addition, deletion, and modification tasks. This dataset is meticulously curated to ensure that edited objects faithfully adhere to the instructional changes while preserving the consistency of unedited regions with the source object. Building upon this dataset, we explore two distinct conditioning strategies for our model: a conventional cross-attention mechanism and a novel 3D token concatenation approach. Our results demonstrate that token concatenation is more parameter-efficient and achieves superior performance. Extensive evaluations show that our method outperforms existing 2D-lifting approaches, setting a new benchmark in generation quality, 3D consistency, and instruction fidelity.
Authors Weiwei Cai, Shuangkang Fang, Weicai Ye, Xin Dong, Yunhan Yang, Xuanyang Zhang, Wei Cheng, Yanpei Cao, Gang Yu, Tao Chen
Categories Computer Vision and Pattern Recognition
0
EvDiff: High Quality Video with an Event Camera
Title CN EvDiff：基于事件相机的高质量视频生成
Keywords 事件相机 · 扩散模型 · 视频重建 · 无配对训练 · 高动态范围
Summary 本文提出EvDiff，一种基于事件相机的扩散模型，用于从稀疏的亮度变化事件流中生成高保真彩色视频。通过设计仅需单步前向扩散的架构和时间一致性的EvEncoder，并引入无需配对数据的代理训练框架，实现了对大规模图像数据的利用，提升了生成视频的质量。实验表明该方法在真实数据集上优于现有技术。
Reason 该论文研究的是从事件相机数据生成高质量视频，属于视频理解与生成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心技术为事件驱动的扩散模型和视频重建，不涉及文本识别、布局分析、表格理解或OCR-free文档解析等DIU核心任务。尽管使用了新型视觉建模方法，但无法迁移解决文档中的OCR幻觉、密集文本处理或复杂排版理解等关键痛点。
Abstract As neuromorphic sensors, event cameras asynchronously record changes in brightness as streams of sparse events with the advantages of high temporal resolution and high dynamic range. Reconstructing intensity images from events is a highly ill-posed task due to the inherent ambiguity of absolute brightness. Early methods generally follow an end-to-end regression paradigm, directly mapping events to intensity frames in a deterministic manner. While effective to some extent, these approaches often yield perceptually inferior results and struggle to scale up in model capacity and training data. In this work, we propose EvDiff, an event-based diffusion model that follows a surrogate training framework to produce high-quality videos. To reduce the heavy computational cost of high-frame-rate video generation, we design an event-based diffusion model that performs only a single forward diffusion step, equipped with a temporally consistent EvEncoder. Furthermore, our novel Surrogate Training Framework eliminates the dependence on paired event-image datasets, allowing the model to leverage large-scale image datasets for higher capacity. The proposed EvDiff is capable of generating high-quality colorful videos solely from monochromatic event streams. Experiments on real-world datasets demonstrate that our method strikes a sweet spot between fidelity and realism, outperforming existing approaches on both pixel-level and perceptual metrics.
Authors Weilun Li, Lei Sun, Ruixi Gao, Qi Jiang, Yuqin Ma, Kaiwei Wang, Ming-Hsuan Yang, Luc Van Gool, Danda Pani Paudel
Categories Computer Vision and Pattern Recognition
0
An Artificial Intelligence Framework for Measuring Human Spine Aging Using MRI
Title CN 一种利用MRI测量人类脊柱衰老的人工智能框架
Keywords 脊柱老化 · MRI · 深度学习 · 生物标志物 · 退行性疾病
Summary 本文提出了一种基于深度学习的计算机视觉方法，利用超过18,000个MRI序列图像来估计脊柱年龄，并定义了‘脊柱年龄差’（SAG）作为衡量脊柱健康状况的生物标志物。研究通过UMAP和HDBSCAN对退行性病变进行聚类，构建仅包含年龄相关退化的数据集，并通过消融实验优化模型结构。结果显示SAG与椎间盘膨出、骨刺、椎管狭窄、骨折以及吸烟和体力劳动等生活方式因素显著相关。
Reason 该论文聚焦于医学影像（MRI）中的脊柱老化分析，属于医疗图像诊断领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。其方法基于聚类和回归模型预测脊柱年龄，未涉及OCR、布局分析、表格识别或多模态语言模型在文档上的应用。无法迁移解决文档理解中的核心痛点如OCR幻觉、复杂排版或长文档推理。
Abstract The human spine is a complex structure composed of 33 vertebrae. It holds the body and is important for leading a healthy life. The spine is vulnerable to age-related degenerations that can be identified through magnetic resonance imaging (MRI). In this paper we propose a novel computer-vison-based deep learning method to estimate spine age using images from over 18,000 MRI series. Data are restricted to subjects with only age-related spine degeneration. Eligibility criteria are created by identifying common age-based clusters of degenerative spine conditions using uniform manifold approximation and projection (UMAP) and hierarchical density-based spatial clustering of applications with noise (HDBSCAN). Model selection is determined using a detailed ablation study on data size, loss, and the effect of different spine regions. We evaluate the clinical utility of our model by calculating the difference between actual spine age and model-predicted age, the spine age gap (SAG), and examining the association between these differences and spine degenerative conditions and lifestyle factors. We find that SAG is associated with conditions including disc bulges, disc osteophytes, spinal stenosis, and fractures, as well as lifestyle factors like smoking and physically demanding work, and thus may be a useful biomarker for measuring overall spine health.
Authors Roozbeh Bazargani, Saqib Abdullah Basar, Daniel Daly-Grafstein, Rodrigo Solis Pompa, Soojin Lee, Saurabh Garg, Yuntong Ma, John A. Carrino, Siavash Khallaghi, Sam Hashemi
Comment 17 pages, 7 figures
Categories Computer Vision and Pattern Recognition
0
Radar2Shape: 3D Shape Reconstruction from High-Frequency Radar using Multiresolution Signed Distance Functions
Title CN Radar2Shape：基于高频雷达信号和多分辨率符号距离函数的3D形状重建
Keywords 3D重建 · 雷达信号 · 扩散模型 · 多分辨率 · 隐空间
Summary 本文提出Radar2Shape，一种基于去噪扩散模型的方法，通过将雷达信号频率与多分辨率形状特征相关联，实现从部分可观测的高频雷达信号中重建任意3D形状。该方法采用两阶段设计：首先学习具有层次分辨率的正则化隐空间，然后以雷达信号频率为条件进行粗到精的扩散生成。实验表明其在仿真和真实数据上均具有良好泛化能力，并发布了两个合成数据集。
Reason 该论文研究的是从高频雷达信号中重建3D物体形状，属于3D视觉与雷达感知领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。尽管使用了多分辨率特征和扩散模型等技术，但其方法针对的是物理世界的3D几何重建问题，无法迁移解决文档图像中的OCR幻觉、复杂排版、细粒度文本识别或长文档推理等核心痛点。
Abstract Determining the shape of 3D objects from high-frequency radar signals is analytically complex but critical for commercial and aerospace applications. Previous deep learning methods have been applied to radar modeling; however, they often fail to represent arbitrary shapes or have difficulty with real-world radar signals which are collected over limited viewing angles. Existing methods in optical 3D reconstruction can generate arbitrary shapes from limited camera views, but struggle when they naively treat the radar signal as a camera view. In this work, we present Radar2Shape, a denoising diffusion model that handles a partially observable radar signal for 3D reconstruction by correlating its frequencies with multiresolution shape features. Our method consists of a two-stage approach: first, Radar2Shape learns a regularized latent space with hierarchical resolutions of shape features, and second, it diffuses into this latent space by conditioning on the frequencies of the radar signal in an analogous coarse-to-fine manner. We demonstrate that Radar2Shape can successfully reconstruct arbitrary 3D shapes even from partially-observed radar signals, and we show robust generalization to two different simulation methods and real-world data. Additionally, we release two synthetic benchmark datasets to encourage future research in the high-frequency radar domain so that models like Radar2Shape can safely be adapted into real-world radar systems.
Authors Neel Sortur, Justin Goodwin, Purvik Patel, Luis Enrique Martinez, Tzofi Klinghoffer, Rajmonda S. Caceres, Robin Walters
Categories Computer Vision and Pattern Recognition
0
Counterfactual World Models via Digital Twin-conditioned Video Diffusion
Title CN 基于数字孪生条件视频扩散的反事实世界模型
Keywords 世界模型 · 反事实推理 · 视频扩散 · 数字孪生 · 多模态推理
Summary 本文提出CWMDT框架，通过构建场景的数字孪生（结构化文本表示），利用大语言模型推理反事实干预的时间传播效应，并驱动视频扩散模型生成对应的视觉变化序列。该方法实现了对‘如果某个物体被移除会怎样’这类反事实问题的模拟，在两个基准上达到SOTA。然而，其应用场景局限于动态视觉场景的因果模拟。
Reason 该论文研究的是视频领域的反事实推理与世界模型构建，核心任务是生成在假设干预下的视觉序列演化，属于视频理解与生成范畴。其方法依赖于数字孪生建模、语言模型推理和视频扩散模型生成，主要面向具身智能或物理仿真场景中的因果推理问题。与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本识别、复杂版面分析或长文档推理无直接关联。尽管使用了多模态（文本+视觉）和结构化表示，但其技术路径难以迁移到文档领域，尤其是无法解决文档特有的细粒度定位、高分辨率输入处理或多模态对齐等问题。
Abstract World models learn to predict the temporal evolution of visual observations given a control signal, potentially enabling agents to reason about environments through forward simulation. Because of the focus on forward simulation, current world models generate predictions based on factual observations. For many emerging applications, such as comprehensive evaluations of physical AI behavior under varying conditions, the ability of world models to answer counterfactual queries, such as "what would happen if this object was removed?", is of increasing importance. We formalize counterfactual world models that additionally take interventions as explicit inputs, predicting temporal sequences under hypothetical modifications to observed scene properties. Traditional world models operate directly on entangled pixel-space representations where object properties and relationships cannot be selectively modified. This modeling choice prevents targeted interventions on specific scene properties. We introduce CWMDT, a framework to overcome those limitations, turning standard video diffusion models into effective counterfactual world models. First, CWMDT constructs digital twins of observed scenes to explicitly encode objects and their relationships, represented as structured text. Second, CWMDT applies large language models to reason over these representations and predict how a counterfactual intervention propagates through time to alter the observed scene. Third, CWMDT conditions a video diffusion model with the modified representation to generate counterfactual visual sequences. Evaluations on two benchmarks show that the CWMDT approach achieves state-of-the-art performance, suggesting that alternative representations of videos, such as the digital twins considered here, offer powerful control signals for video forward simulation-based world models.
Authors Yiqing Shen, Aiza Maksutova, Chenjia Li, Mathias Unberath
Categories Computer Vision and Pattern Recognition
0
ID-Crafter: VLM-Grounded Online RL for Compositional Multi-Subject Video Generation
Title CN ID-Crafter：基于VLM的在线强化学习用于组合式多主体视频生成
Keywords 视频生成 · 身份保持 · Vision-Language Model · 强化学习 · 多主体合成
Summary 本文提出ID-Crafter，一种用于多主体视频生成的框架，通过分层身份保持注意力机制、基于VLM的语义理解模块以及在线强化学习优化，提升视频中多个主体的身份保留和语义一致性。同时构建了新数据集用于训练和评估。实验表明该方法在多主体视频生成任务上达到SOTA。
Reason 该论文研究的是多主体视频生成，属于视频生成领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。尽管使用了VLM进行语义理解，但其核心目标是身份保持和视频合成控制，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract Significant progress has been achieved in high-fidelity video synthesis, yet current paradigms often fall short in effectively integrating identity information from multiple subjects. This leads to semantic conflicts and suboptimal performance in preserving identities and interactions, limiting controllability and applicability. To tackle this issue, we introduce ID-Crafter, a framework for multi-subject video generation that achieves superior identity preservation and semantic coherence. ID-Crafter integrates three key components: (i) a hierarchical identity-preserving attention mechanism that progressively aggregates features at intra-subject, inter-subject, and cross-modal levels; (ii) a semantic understanding module powered by a pretrained Vision-Language Model (VLM) to provide fine-grained guidance and capture complex inter-subject relationships; and (iii) an online reinforcement learning phase to further refine the model for critical concepts. Furthermore, we construct a new dataset to facilitate robust training and evaluation. Extensive experiments demonstrate that ID-Crafter establishes new state-of-the-art performance on multi-subject video generation benchmarks, excelling in identity preservation, temporal consistency, and overall video quality.
Authors Panwang Pan, Jingjing Zhao, Yuchen Lin, Chenguo Lin, Chenxin Li, Hengyu Liu, Tingting Shen, Yadong MU
Categories Computer Vision and Pattern Recognition
0
Topology Aware Neural Interpolation of Scalar Fields
Title CN 面向标量场的拓扑感知神经插值
Keywords 拓扑数据分析 · 神经插值 · 标量场 · 持久同调 · 科学可视化
Summary 本文提出一种基于神经网络的插值方法，用于恢复时变标量场中缺失的时间帧数据。该方法利用关键帧及其对应的持久同调图，通过引入拓扑感知损失函数，在保证几何精度的同时提升拓扑结构的一致性。实验表明其在2D和3D数据集上优于传统插值方案。
Reason 该论文研究的是时变标量场的拓扑感知神经插值，属于科学可视化与计算数学交叉领域，核心任务是基于持久同调图（persistence diagrams）进行物理场数据重建。其方法聚焦于拓扑结构保持的连续场插值，与文档图像理解（DIU）中的文本识别、布局分析、表格理解等任务无直接关联。所提出的神经架构和拓扑损失无法迁移解决OCR幻觉、细粒度定位或复杂排版理解等DIU核心痛点，且不属于VLM或多模态推理框架创新。
Abstract This paper presents a neural scheme for the topology-aware interpolation of time-varying scalar fields. Given a time-varying sequence of persistence diagrams, along with a sparse temporal sampling of the corresponding scalar fields, denoted as keyframes, our interpolation approach aims at "inverting" the non-keyframe diagrams to produce plausible estimations of the corresponding, missing data. For this, we rely on a neural architecture which learns the relation from a time value to the corresponding scalar field, based on the keyframe examples, and reliably extends this relation to the non-keyframe time steps. We show how augmenting this architecture with specific topological losses exploiting the input diagrams both improves the geometrical and topological reconstruction of the non-keyframe time steps. At query time, given an input time value for which an interpolation is desired, our approach instantaneously produces an output, via a single propagation of the time input through the network. Experiments interpolating 2D and 3D time-varying datasets show our approach superiority, both in terms of data and topological fitting, with regard to reference interpolation schemes. Our implementation is available at this GitHub link : https://github.com/MohamedKISSI/Topology-Aware-Neural-Interpolation-of-Scalar-Fields.git.
Authors Mohamed Kissi, Keanu Sisouk, Joshua A. Levine, Julien Tierny
Categories Computer Vision and Pattern Recognition
0
Forecasting Future Anatomies: Longitudinal Brain Mri-to-Mri Prediction
Title CN 预测未来解剖结构：纵向脑部MRI到MRI的预测
Keywords 医学影像预测 · 纵向MRI · 深度学习 · UNet · 神经退行性疾病
Summary 本文提出了一种基于深度学习的方法，用于从基线脑部MRI预测多年后的未来脑部MRI图像，旨在模拟神经退行性疾病的时空演变模式。研究比较了五种网络架构，在ADNI和AIBL数据集上实现了高保真的图像预测，并验证了跨队列的泛化能力。虽然技术上扎实，但应用场景局限于医学影像分析。
Reason 该论文研究的是医学影像中的脑部MRI序列预测问题，属于纯医学图像生成与预测领域，与文档图像理解（DIU）、DocVQA、OCR、布局分析等任务无直接关联。尽管使用了UNet等可能在视觉任务中通用的架构，但其方法目标是生物形态演化建模，无法迁移解决文档理解中的OCR幻觉、复杂排版、细粒度定位或长文档推理等核心痛点。
Abstract Predicting future brain state from a baseline magnetic resonance image (MRI) is a central challenge in neuroimaging and has important implications for studying neurodegenerative diseases such as Alzheimer's disease (AD). Most existing approaches predict future cognitive scores or clinical outcomes, such as conversion from mild cognitive impairment to dementia. Instead, here we investigate longitudinal MRI image-to-image prediction that forecasts a participant's entire brain MRI several years into the future, intrinsically modeling complex, spatially distributed neurodegenerative patterns. We implement and evaluate five deep learning architectures (UNet, U2-Net, UNETR, Time-Embedding UNet, and ODE-UNet) on two longitudinal cohorts (ADNI and AIBL). Predicted follow-up MRIs are directly compared with the actual follow-up scans using metrics that capture global similarity and local differences. The best performing models achieve high-fidelity predictions, and all models generalize well to an independent external dataset, demonstrating robust cross-cohort performance. Our results indicate that deep learning can reliably predict participant-specific brain MRI at the voxel level, offering new opportunities for individualized prognosis.
Authors Ali Farki, Elaheh Moradi, Deepika Koundal, Jussi Tohka
Categories Computer Vision and Pattern Recognition
0
Automated Interpretable 2D Video Extraction from 3D Echocardiography
Title CN 从三维超声心动图中自动提取可解释的二维视频
Keywords 3D超声心动图 · 2D视图提取 · 深度学习 · 医学影像 · 自动化诊断
Summary 本文提出一种基于深度学习的方法，从3D超声心动图中自动提取标准的2D视频视图，结合解剖标志和心脏病专家提供的启发式规则进行视图重建，并通过多位心脏病专家的盲评验证了生成视频的准确性（96%）。同时验证了这些2D视频可用于现有AI模型进行心脏异常检测和临床级测量。作者公开了代码和包含29个3D视频的数据集。
Reason 该论文属于医学图像处理领域，研究的是从3D心脏超声数据中自动生成标准2D切面视频，服务于临床医生的阅读习惯。其核心任务与文档图像理解（DIU）无关，不涉及文本识别、布局分析、表格理解或VLM在文档上的应用。尽管使用了深度学习和视觉方法，但目标场景和挑战（心脏结构成像）与文档图像存在本质差异，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU核心问题。
Abstract Although the heart has complex three-dimensional (3D) anatomy, conventional medical imaging with cardiac ultrasound relies on a series of 2D videos showing individual cardiac structures. 3D echocardiography is a developing modality that now offers adequate image quality for clinical use, with potential to streamline acquisition and improve assessment of off-axis features. We propose an automated method to select standard 2D views from 3D cardiac ultrasound volumes, allowing physicians to interpret the data in their usual format while benefiting from the speed and usability of 3D scanning. Applying a deep learning view classifier and downstream heuristics based on anatomical landmarks together with heuristics provided by cardiologists, we reconstruct standard echocardiography views. This approach was validated by three cardiologists in blinded evaluation (96\% accuracy in 1,600 videos from 2 hospitals). The downstream 2D videos were also validated in their ability to detect cardiac abnormalities using AI echocardiography models (EchoPrime and PanEcho) as well as ability to generate clinical-grade measurements of cardiac anatomy (EchoNet-Measurement). We demonstrated that the extracted 2D videos preserve spatial calibration and diagnostic features, allowing clinicians to obtain accurate real-world interpretations from 3D volumes. We release the code and a dataset of 29 3D echocardiography videos https://github.com/echonet/3d-echo .
Authors Milos Vukadinovic, Hirotaka Ieki, Yuki Sahashi, David Ouyang, Bryan He
Comment 12 pages, 5 figures
Categories Computer Vision and Pattern Recognition
0
GPR-OdomNet: Difference and Similarity-Driven Odometry Estimation Network for Ground Penetrating Radar-Based Localization
Title CN GPR-OdomNet：基于地面穿透雷达定位的差异与相似性驱动的里程估计网络
Keywords GPR · 里程估计 · B-scan图像 · 多尺度特征 · 定位
Summary 本文提出了一种名为GPR-OdomNet的神经网络方法，用于在恶劣环境条件下通过地面穿透雷达的B-scan图像进行精确里程估计。该方法通过提取连续时刻B-scan图像的多尺度特征，并分析其差异与相似性来估算移动距离。实验表明，在CMU-GPR数据集上，该方法相比现有最优方法RMSE降低了10.2%。
Reason 该论文研究的是基于地面穿透雷达（GPR）B-scan图像的机器人/车辆定位中的里程估计问题，属于自动驾驶与具身智能感知领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无任何关联。其技术路线聚焦于雷达图像的多尺度特征提取与相似性分析，无法迁移至解决OCR幻觉、复杂排版理解、细粒度文本定位等DIU核心痛点。
Abstract When performing robot/vehicle localization using ground penetrating radar (GPR) to handle adverse weather and environmental conditions, existing techniques often struggle to accurately estimate distances when processing B-scan images with minor distinctions. This study introduces a new neural network-based odometry method that leverages the similarity and difference features of GPR B-scan images for precise estimation of the Euclidean distances traveled between the B-scan images. The new custom neural network extracts multi-scale features from B-scan images taken at consecutive moments and then determines the Euclidean distance traveled by analyzing the similarities and differences between these features. To evaluate our method, an ablation study and comparison experiments have been conducted using the publicly available CMU-GPR dataset. The experimental results show that our method consistently outperforms state-of-the-art counterparts in all tests. Specifically, our method achieves a root mean square error (RMSE), and achieves an overall weighted RMSE of 0.449 m across all data sets, which is a 10.2\% reduction in RMSE when compared to the best state-of-the-art method.
Authors Huaichao Wang, Xuanxin Fan, Ji Liu, Haifeng Li, Dezhen Song
Categories Computer Vision and Pattern Recognition
0
Improving Multimodal Distillation for 3D Semantic Segmentation under Domain Shift BMVC 2025
Title CN 通过多模态蒸馏提升域偏移下的3D语义分割
Keywords 3D语义分割 · 域适应 · 多模态蒸馏 · 激光雷达 · 视觉基础模型
Summary 本文研究在域偏移下如何利用视觉基础模型（VFMs）通过无监督知识蒸馏提升激光雷达点云的语义分割性能。作者发现激光雷达主干网络架构对跨域泛化至关重要，且可预先训练一个通用主干网络，并通过冻结主干、仅训练MLP头部的方式实现多个域偏移场景下的最优性能。该方法在四个具有挑战性的基准上达到SOTA。
Reason 该论文研究的是基于激光雷达点云的3D语义分割，属于自动驾驶与3D视觉领域，与文档图像理解（DIU）任务无直接关联。其核心问题、数据模态（点云+视觉基础模型）和应用场景均远离文档场景，无法迁移解决OCR幻觉、复杂排版或长文档推理等DIU痛点。
Abstract Semantic segmentation networks trained under full supervision for one type of lidar fail to generalize to unseen lidars without intervention. To reduce the performance gap under domain shifts, a recent trend is to leverage vision foundation models (VFMs) providing robust features across domains. In this work, we conduct an exhaustive study to identify recipes for exploiting VFMs in unsupervised domain adaptation for semantic segmentation of lidar point clouds. Building upon unsupervised image-to-lidar knowledge distillation, our study reveals that: (1) the architecture of the lidar backbone is key to maximize the generalization performance on a target domain; (2) it is possible to pretrain a single backbone once and for all, and use it to address many domain shifts; (3) best results are obtained by keeping the pretrained backbone frozen and training an MLP head for semantic segmentation. The resulting pipeline achieves state-of-the-art results in four widely-recognized and challenging settings. The code will be available at: https://github.com/valeoai/muddos.
Authors Björn Michele, Alexandre Boulch, Gilles Puy, Tuan-Hung Vu, Renaud Marlet, Nicolas Courty
Comment Accepted at BMVC 2025
Categories Computer Vision and Pattern Recognition
0
TrackGS: Optimizing COLMAP-Free 3D Gaussian Splatting with Global Track Constraints
Title CN TrackGS：通过全局轨迹约束优化无需COLMAP的3D高斯点阵化
Keywords 3D Gaussian Splatting · COLMAP-free · feature tracks · camera optimization · novel view synthesis
Summary 本文提出TrackGS，一种无需COLMAP预处理的3D高斯点阵化方法，通过引入全局特征轨迹作为几何约束，联合优化相机参数与3D高斯分布。该方法在真实和合成数据上实现了更低的姿态误差和高质量渲染，提升了3DGS的实用性。
Reason 该论文属于3D视觉与图形学领域，研究的是3D高斯点阵化和新视角合成中的相机参数优化问题，与文档图像理解（DIU）或视觉语言模型（VLM）在文档上的应用无直接关联。其核心技术——基于特征轨迹的全局几何约束和相机内参优化——难以迁移到解决OCR幻觉、复杂排版理解、长文档推理等DIU核心痛点。属于无关领域（3D视觉/重建），命中负面清单第2条。
Abstract We present TrackGS, a novel method to integrate global feature tracks with 3D Gaussian Splatting (3DGS) for COLMAP-free novel view synthesis. While 3DGS delivers impressive rendering quality, its reliance on accurate precomputed camera parameters remains a significant limitation. Existing COLMAP-free approaches depend on local constraints that fail in complex scenarios. Our key innovation lies in leveraging feature tracks to establish global geometric constraints, enabling simultaneous optimization of camera parameters and 3D Gaussians. Specifically, we: (1) introduce track-constrained Gaussians that serve as geometric anchors, (2) propose novel 2D and 3D track losses to enforce multi-view consistency, and (3) derive differentiable formulations for camera intrinsics optimization. Extensive experiments on challenging real-world and synthetic datasets demonstrate state-of-the-art performance, with much lower pose error than previous methods while maintaining superior rendering quality. Our approach eliminates the need for COLMAP preprocessing, making 3DGS more accessible for practical applications.
Authors Dongbo Shi, Shen Cao, Lubin Fan, Bojian Wu, Jinhui Guo, Ligang Liu, Renjie Chen
Categories Computer Vision and Pattern Recognition
0
MCMoE: Completing Missing Modalities with Mixture of Experts for Incomplete Multimodal Action Quality Assessment AAAI 2026
Title CN MCMoE：通过专家混合模型补全缺失模态用于不完整多模态动作质量评估
Keywords 多模态学习 · 模态补全 · 专家混合模型 · 动作质量评估 · 不完整多模态
Summary 本文提出MCMoE框架，用于在多模态动作质量评估中补全缺失的模态信息。通过专家混合结构和自适应门控机制，动态生成缺失模态并统一学习单模态与跨模态表征。实验表明该方法在三个公开AQA数据集上表现优异。
Reason 该论文研究的是多模态动作质量评估（AQA）中的模态补全问题，属于视频理解领域，主要应用场景为体育动作或手术操作等视频序列的评分。其核心方法MCMoE虽涉及多模态融合与缺失模态重建，但面向的是时序动作视频中的模态鲁棒性问题，与文档图像理解（DIU）中的OCR幻觉、密集文本识别、复杂版式分析等核心痛点无直接关联。无法迁移用于解决文档图像中的视觉-语言对齐或细粒度文本定位等问题。
Abstract Multimodal Action Quality Assessment (AQA) has recently emerged as a promising paradigm. By leveraging complementary information across shared contextual cues, it enhances the discriminative evaluation of subtle intra-class variations in highly similar action sequences. However, partial modalities are frequently unavailable at the inference stage in reality. The absence of any modality often renders existing multimodal models inoperable. Furthermore, it triggers catastrophic performance degradation due to interruptions in cross-modal interactions. To address this issue, we propose a novel Missing Completion Framework with Mixture of Experts (MCMoE) that unifies unimodal and joint representation learning in single-stage training. Specifically, we propose an adaptive gated modality generator that dynamically fuses available information to reconstruct missing modalities. We then design modality experts to learn unimodal knowledge and dynamically mix the knowledge of all experts to extract cross-modal joint representations. With a mixture of experts, missing modalities are further refined and complemented. Finally, in the training phase, we mine the complete multimodal features and unimodal expert knowledge to guide modality generation and generation-based joint representation extraction. Extensive experiments demonstrate that our MCMoE achieves state-of-the-art results in both complete and incomplete multimodal learning on three public AQA benchmarks. Code is available at https://github.com/XuHuangbiao/MCMoE.
Authors Huangbiao Xu, Huanqi Wu, Xiao Ke, Junyi Wu, Rui Xu, Jinglin Xu
Comment AAAI 2026
Categories Computer Vision and Pattern Recognition
0
MorphSeek: Fine-grained Latent Representation-Level Policy Optimization for Deformable Image Registration
Title CN MorphSeek：用于可变形图像配准的细粒度潜在表示级策略优化
Keywords 可变形图像配准 · 强化学习 · 潜在表示 · 医学图像分析 · 策略优化
Summary 本文提出MorphSeek，一种用于可变形图像注册的细粒度潜在表示级策略优化框架。该方法通过在编码器上引入随机高斯策略头，在潜在特征空间中建模特征分布，实现高效探索和由粗到精的优化。结合无监督预热与弱监督微调，以及多轨迹采样的Group Relative Policy Optimization策略，提升了训练稳定性与标签效率。在多个3D医学图像配准任务上取得了优于基线的Dice分数，同时保持低参数开销和延迟。
Reason 该论文属于医学图像分析中的可变形图像配准（DIR）领域，使用强化学习和潜在空间策略优化来解决高维形变场的问题。尽管其‘表示级策略学习’在高维视觉对齐中有一定通用性，但整体方法面向的是医学影像的像素级形变建模，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、OCR-free理解等无直接关联。其技术路径难以迁移至解决文档中的OCR幻觉、复杂排版或长文档推理等痛点，且未涉及多模态大模型（VLM）或文本-视觉对齐机制。
Abstract Deformable image registration (DIR) remains a fundamental yet challenging problem in medical image analysis, largely due to the prohibitively high-dimensional deformation space of dense displacement fields and the scarcity of voxel-level supervision. Existing reinforcement learning frameworks often project this space into coarse, low-dimensional representations, limiting their ability to capture spatially variant deformations. We propose MorphSeek, a fine-grained representation-level policy optimization paradigm that reformulates DIR as a spatially continuous optimization process in the latent feature space. MorphSeek introduces a stochastic Gaussian policy head atop the encoder to model a distribution over latent features, facilitating efficient exploration and coarse-to-fine refinement. The framework integrates unsupervised warm-up with weakly supervised fine-tuning through Group Relative Policy Optimization, where multi-trajectory sampling stabilizes training and improves label efficiency. Across three 3D registration benchmarks (OASIS brain MRI, LiTS liver CT, and Abdomen MR-CT), MorphSeek achieves consistent Dice improvements over competitive baselines while maintaining high label efficiency with minimal parameter cost and low step-level latency overhead. Beyond optimizer specifics, MorphSeek advances a representation-level policy learning paradigm that achieves spatially coherent and data-efficient deformation optimization, offering a principled, backbone-agnostic, and optimizer-agnostic solution for scalable visual alignment in high-dimensional settings.
Authors Runxun Zhang, Yizhou Liu, Li Dongrui, Bo XU, Jingwei Wei
Categories Computer Vision and Pattern Recognition
0
IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigation
Title CN IndustryNav：探索具身智能体在动态工业导航中的空间推理
Keywords 具身智能 · 空间推理 · 工业导航 · 动态环境 · VLLM
Summary 本文提出了IndustryNav，首个面向动态工业场景的具身导航基准，包含12个高保真Unity仓库环境，强调动态物体和人类移动下的主动空间推理。通过结合第一视角视觉与全局里程计进行导航评估，并引入碰撞率和警告率衡量安全性。实验表明当前VLLM在鲁棒路径规划与主动探索方面仍存在明显不足。
Reason 该论文属于具身智能与机器人领域，研究的是动态工业环境中的导航问题，核心任务为PointGoal导航与碰撞规避，与文档图像理解（DIU）或DocVQA无直接关联。尽管使用了VLLM，但其目标是解决物理空间中的路径规划问题，而非文档中的视觉布局、文本识别或逻辑推理。无法迁移用于解决OCR幻觉、复杂排版或长文档理解等DIU核心痛点。
Abstract While Visual Large Language Models (VLLMs) show great promise as embodied agents, they continue to face substantial challenges in spatial reasoning. Existing embodied benchmarks largely focus on passive, static household environments and evaluate only isolated capabilities, failing to capture holistic performance in dynamic, real-world complexity. To fill this gap, we present IndustryNav, the first dynamic industrial navigation benchmark for active spatial reasoning. IndustryNav leverages 12 manually created, high-fidelity Unity warehouse scenarios featuring dynamic objects and human movement. Our evaluation employs a PointGoal navigation pipeline that effectively combines egocentric vision with global odometry to assess holistic local-global planning. Crucially, we introduce the "collision rate" and "warning rate" metrics to measure safety-oriented behaviors and distance estimation. A comprehensive study of nine state-of-the-art VLLMs (including models such as GPT-5-mini, Claude-4.5, and Gemini-2.5) reveals that closed-source models maintain a consistent advantage; however, all agents exhibit notable deficiencies in robust path planning, collision avoidance and active exploration. This highlights a critical need for embodied research to move beyond passive perception and toward tasks that demand stable planning, active exploration, and safe behavior in dynamic, real-world environment.
Authors Yifan Li, Lichi Li, Anh Dao, Xinyu Zhou, Yicheng Qiao, Zheda Mai, Daeun Lee, Zichen Chen, Zhen Tan, Mohit Bansal, Yu Kong
Categories Computer Vision and Pattern Recognition
0
Non-Parametric Probabilistic Robustness: A Conservative Metric with Optimized Perturbation Distributions
Title CN 非参数化概率鲁棒性：一种具有优化扰动分布的保守度量
Keywords 概率鲁棒性 · 非参数化方法 · 深度学习安全 · 扰动分布 · 图像分类
Summary 本文提出了一种非参数化的概率鲁棒性（NPPR）度量方法，通过从数据中学习最优扰动分布来实现更保守和实用的模型鲁棒性评估。与传统假设固定扰动分布的方法不同，NPPR采用高斯混合模型结合MLP头和上采样策略，在CIFAR和Tiny ImageNet等标准图像分类数据集上验证了其有效性，显示出比现有方法低至40%的概率鲁棒性估计，表明其更具保守性和现实适用性。
Reason 该论文研究的是深度学习模型在图像分类任务中的概率鲁棒性问题，核心关注点为对抗扰动下的模型安全性评估，属于通用计算机视觉中的鲁棒性理论分析。其方法聚焦于图像级分类（如CIFAR、ImageNet），并未涉及文本识别、布局分析、多模态理解或文档结构建模等文档图像理解（DIU）相关任务。尽管‘鲁棒性’是重要议题，但该工作无法直接迁移解决OCR幻觉、复杂排版解析、细粒度定位或长文档推理等DIU核心痛点，且不属于VLM或多模态架构改进的范畴。
Abstract Deep learning (DL) models, despite their remarkable success, remain vulnerable to small input perturbations that can cause erroneous outputs, motivating the recent proposal of probabilistic robustness (PR) as a complementary alternative to adversarial robustness (AR). However, existing PR formulations assume a fixed and known perturbation distribution, an unrealistic expectation in practice. To address this limitation, we propose non-parametric probabilistic robustness (NPPR), a more practical PR metric that does not rely on any predefined perturbation distribution. Following the non-parametric paradigm in statistical modeling, NPPR learns an optimized perturbation distribution directly from data, enabling conservative PR evaluation under distributional uncertainty. We further develop an NPPR estimator based on a Gaussian Mixture Model (GMM) with Multilayer Perceptron (MLP) heads and bicubic up-sampling, covering various input-dependent and input-independent perturbation scenarios. Theoretical analyses establish the relationships among AR, PR, and NPPR. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet across ResNet18/50, WideResNet50 and VGG16 validate NPPR as a more practical robustness metric, showing up to 40\% more conservative (lower) PR estimates compared to assuming those common perturbation distributions used in state-of-the-arts.
Authors Zheng Wang, Yi Zhang, Siddartha Khastgir, Carsten Maple, Xingyu Zhao
Categories Computer Vision and Pattern Recognition
0
METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Model
Title CN METIS：用于集成灵巧视觉-语言-动作模型的多源自我中心训练
Keywords 具身智能 · 视觉-语言-动作模型 · 灵巧操作 · 自我中心视频 · 动作学习
Summary 本文提出METIS，一种基于多源自我中心数据训练的视觉-语言-动作模型，用于机器人灵巧操作。通过构建统一的动作空间数据集EgoAtlas，并引入运动感知动态表示，实现跨人类与机器人域的动作学习。在真实世界六项操作任务中取得最优成功率，展现出良好的泛化性与鲁棒性。
Reason 该论文属于具身智能与机器人领域，研究目标是构建具备灵巧操作能力的通用机器人模型，使用人类第一视角数据进行视觉-语言-动作联合训练。尽管涉及视觉与语言模态，但其核心任务与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心问题。
Abstract Building a generalist robot that can perceive, reason, and act across diverse tasks remains an open challenge, especially for dexterous manipulation. A major bottleneck lies in the scarcity of large-scale, action-annotated data for dexterous skills, as teleoperation is difficult and costly. Human data, with its vast scale and diverse manipulation behaviors, provides rich priors for learning robotic actions. While prior works have explored leveraging human demonstrations, they are often constrained by limited scenarios and a large visual gap between human and robots. To eliminate these limitations, we propose METIS, a vision-language-action (VLA) model for dexterous manipulation pretrained on multi-source egocentric datasets. We first construct EgoAtlas, which integrates large-scale human and robotic data from multiple sources, all unified under a consistent action space. We further extract motion-aware dynamics, a compact and discretized motion representation, which provides efficient and expressive supervision for VLA training. Built upon them, METIS integrates reasoning and acting into a unified framework, enabling effective deployment to downstream dexterous manipulation tasks. Our method demonstrates exceptional dexterous manipulation capabilities, achieving highest average success rate in six real-world tasks. Experimental results also highlight the superior generalization and robustness to out-of-distribution scenarios. These findings emphasize METIS as a promising step toward a generalist model for dexterous manipulation.
Authors Yankai Fu, Ning Chen, Junkai Zhao, Shaozhe Shan, Guocai Yao, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang
Categories Computer Vision and Pattern Recognition
0
SVRecon: Sparse Voxel Rasterization for Surface Reconstruction
Title CN SVRecon：用于表面重建的稀疏体素光栅化
Keywords 3D重建 · 稀疏体素 · SDF · 光栅化 · 几何优化
Summary 本文提出SVRecon，将稀疏体素光栅化与符号距离函数（SDF）结合，用于高保真表面重建。通过视觉几何模型进行鲁棒初始化，并引入空间平滑损失来增强体素间的结构一致性，从而在多个基准上实现了高精度且快速收敛的重建效果。
Reason 该论文研究的是3D视觉中的表面重建问题，属于纯3D视觉与几何建模领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心技术——基于稀疏体素和符号距离函数（SDF）的几何优化——无法迁移解决文档理解中的OCR幻觉、复杂排版、细粒度定位或长文档推理等核心痛点。
Abstract We extend the recently proposed sparse voxel rasterization paradigm to the task of high-fidelity surface reconstruction by integrating Signed Distance Function (SDF), named SVRecon. Unlike 3D Gaussians, sparse voxels are spatially disentangled from their neighbors and have sharp boundaries, which makes them prone to local minima during optimization. Although SDF values provide a naturally smooth and continuous geometric field, preserving this smoothness across independently parameterized sparse voxels is nontrivial. To address this challenge, we promote coherent and smooth voxel-wise structure through (1) robust geometric initialization using a visual geometry model and (2) a spatial smoothness loss that enforces coherent relationships across parent-child and sibling voxel groups. Extensive experiments across various benchmarks show that our method achieves strong reconstruction accuracy while having consistently speedy convergence. The code will be made public.
Authors Seunghun Oh, Jaesung Choe, Dongjae Lee, Daeun Lee, Seunghoon Jeong, Yu-Chiang Frank Wang, Jaesik Park
Categories Computer Vision and Pattern Recognition
0
A Training-Free Style-Personalization via SVD-Based Feature Decomposition
Title CN 基于SVD特征分解的免训练风格个性化
Keywords 图像生成 · 风格个性化 · SVD分解 · 免训练控制 · 扩散模型
Summary 本文提出一种无需训练的风格个性化图像生成框架，通过在推理阶段对扩散模型内部特征进行奇异值分解（SVD），识别并操控主导风格的特征成分。引入两个轻量模块：主特征融合（PFB）实现基于SVD的风格调制，结构注意力校正（SAC）增强细粒度阶段的结构一致性。实验表明该方法在风格保真度和提示一致性上可媲美微调方法，且推理更快、部署更灵活。
Reason 该论文属于纯图像生成领域，研究的是风格化图像生成中的免训练个性化方法，核心关注点为艺术风格迁移与生成模型控制，与文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等无直接关联。其提出的SVD特征分解技术用于分离风格与内容，但难以迁移到解决OCR幻觉、密集文本识别或复杂排版理解等文档核心痛点。属于无关领域（纯图像生成），命中负面清单第2条。
Abstract We present a training-free framework for style-personalized image generation that operates during inference using a scale-wise autoregressive model. Our method generates a stylized image guided by a single reference style while preserving semantic consistency and mitigating content leakage. Through a detailed step-wise analysis of the generation process, we identify a pivotal step where the dominant singular values of the internal feature encode style-related components. Building upon this insight, we introduce two lightweight control modules: Principal Feature Blending, which enables precise modulation of style through SVD-based feature reconstruction, and Structural Attention Correction, which stabilizes structural consistency by leveraging content-guided attention correction across fine stages. Without any additional training, extensive experiments demonstrate that our method achieves competitive style fidelity and prompt fidelity compared to fine-tuned baselines, while offering faster inference and greater deployment flexibility.
Authors Kyoungmin Lee, Jihun Park, Jongmin Gim, Wonhyeok Choi, Kyumin Hwang, Jaeyeul Kim, Sunghoon Im
Comment 21 pages, 14 figures
Categories Computer Vision and Pattern Recognition
0
UAM: A Unified Attention-Mamba Backbone of Multimodal Framework for Tumor Cell Classification
Title CN UAM：用于肿瘤细胞分类的多模态框架统一注意力-Mamba主干网络
Keywords Mamba架构 · 注意力机制 · 细胞级分类 · 放射组学 · 医学图像分析
Summary 本文提出了一种名为UAM（Unified Attention-Mamba）的统一主干网络，用于基于放射组学特征的细胞级肿瘤分类与图像分割。该方法结合了注意力机制与Mamba架构的优势，构建了一个灵活统一的模型结构，无需手动调节模块比例，并在公开数据集上实现了优于现有模型的性能表现。实验结果显示其在细胞分类和肿瘤分割任务上均有显著提升。
Reason 该论文研究的是医学图像中的细胞级分类与分割任务，属于具象的生物医学图像分析领域，核心关注点为肿瘤细胞的放射组学特征提取与分类，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其方法虽基于Mamba和注意力机制的融合架构，但应用场景完全偏离文档结构、文本识别、排版理解或OCR-free等DIU核心问题，且未提出可迁移至文档理解领域的通用底层技术（如视觉token压缩、高分辨率处理、去幻觉机制等）。因此属于无关领域应用，无法为DIU提供方法论上的借鉴。
Abstract Cell-level radiomics features provide fine-grained insights into tumor phenotypes and have the potential to significantly enhance diagnostic accuracy on hematoxylin and eosin (H&E) images. By capturing micro-level morphological and intensity patterns, these features support more precise tumor identification and improve AI interpretability by highlighting diagnostically relevant cells for pathologist review. However, most existing studies focus on slide-level or patch-level tumor classification, leaving cell-level radiomics analysis largely unexplored. Moreover, there is currently no dedicated backbone specifically designed for radiomics data. Inspired by the recent success of the Mamba architecture in vision and language domains, we introduce a Unified Attention-Mamba (UAM) backbone for cell-level classification using radiomics features. Unlike previous hybrid approaches that integrate Attention and Mamba modules in fixed proportions, our unified design flexibly combines their capabilities within a single cohesive architecture, eliminating the need for manual ratio tuning and improving encode capability. We develop two UAM variants to comprehensively evaluate the benefits of this unified structure. Building on this backbone, we further propose a multimodal UAM framework that jointly performs cell-level classification and image segmentation. Experimental results demonstrate that UAM achieves state-of-the-art performance across both tasks on public benchmarks, surpassing leading image-based foundation models. It improves cell classification accuracy from 74% to 78% ($n$=349,882 cells), and tumor segmentation precision from 75% to 80% ($n$=406 patches). These findings highlight the effectiveness and promise of UAM as a unified and extensible multimodal foundation for radiomics-driven cancer diagnosis.
Authors Taixi Chen, Jingyun Chen, Nancy Guo
Categories Computer Vision and Pattern Recognition
0
Learning Latent Transmission and Glare Maps for Lens Veiling Glare Removal
Title CN 学习潜在透射和眩光图以去除镜头晕影
Keywords 图像恢复 · 眩光去除 · 生成模型 · 无监督学习 · 光学退化
Summary 本文提出VeilGen模型，通过无监督方式从目标图像中估计光学透射和眩光图，并结合Stable Diffusion先验生成具有真实眩光退化的配对数据集。同时设计DeVeiler恢复网络，利用预测的潜在图引导逆向去眩光过程。实验表明该方法在紧凑型光学系统上能有效恢复受眩光影响的图像。
Reason 该论文研究的是光学系统中的镜头眩光去除问题，属于图像去噪与物理成像恢复范畴，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心方法聚焦于生成带有眩光的合成数据并设计恢复网络，虽涉及图像质量提升，但解决的是底层视觉问题，无法迁移至文档布局分析、文本识别、多模态推理等DIU核心任务。
Abstract Beyond the commonly recognized optical aberrations, the imaging performance of compact optical systems-including single-lens and metalens designs-is often further degraded by veiling glare caused by stray-light scattering from non-ideal optical surfaces and coatings, particularly in complex real-world environments. This compound degradation undermines traditional lens aberration correction yet remains underexplored. A major challenge is that conventional scattering models (e.g., for dehazing) fail to fit veiling glare due to its spatial-varying and depth-independent nature. Consequently, paired high-quality data are difficult to prepare via simulation, hindering application of data-driven veiling glare removal models. To this end, we propose VeilGen, a generative model that learns to simulate veiling glare by estimating its underlying optical transmission and glare maps in an unsupervised manner from target images, regularized by Stable Diffusion (SD)-based priors. VeilGen enables paired dataset generation with realistic compound degradation of optical aberrations and veiling glare, while also providing the estimated latent optical transmission and glare maps to guide the veiling glare removal process. We further introduce DeVeiler, a restoration network trained with a reversibility constraint, which utilizes the predicted latent maps to guide an inverse process of the learned scattering model. Extensive experiments on challenging compact optical systems demonstrate that our approach delivers superior restoration quality and physical fidelity compared with existing methods. These suggest that VeilGen reliably synthesizes realistic veiling glare, and its learned latent maps effectively guide the restoration process in DeVeiler. All code and datasets will be publicly released at https://github.com/XiaolongQian/DeVeiler.
Authors Xiaolong Qian, Qi Jiang, Lei Sun, Zongxi Yu, Kailun Yang, Peixuan Wu, Jiacheng Zhou, Yao Gao, Yaoguang Ma, Ming-Hsuan Yang, Kaiwei Wang
Comment All code and datasets will be publicly released at https://github.com/XiaolongQian/DeVeiler
Categories Computer Vision and Pattern Recognition
0
A Unified Voxel Diffusion Module for Point Cloud 3D Object Detection
Title CN 用于点云3D目标检测的统一体素扩散模块
Keywords 点云 · 3D目标检测 · 体素扩散模块 · 稀疏卷积 · Transformer
Summary 本文提出了一种新的体素扩散模块（VDM），用于增强基于体素表示的点云3D目标检测模型的特征表达能力。VDM通过稀疏3D卷积和残差连接来扩散前景体素特征并聚合细粒度空间信息，可集成到主流Transformer或SSM架构中，在多个基准数据集上实现了最先进的检测性能。
Reason 该论文研究的是点云3D目标检测，属于3D视觉和自动驾驶相关领域，与文档图像理解（DIU）或DocVQA等任务完全无关。其提出的方法针对点云数据设计，无法迁移解决文档理解中的OCR幻觉、密集文本、复杂排版或长文档推理等核心痛点。
Abstract Recent advances in point cloud object detection have increasingly adopted Transformer-based and State Space Models (SSMs), demonstrating strong performance. However, voxelbased representations in these models require strict consistency in input and output dimensions due to their serialized processing, which limits the spatial diffusion capability typically offered by convolutional operations. This limitation significantly affects detection accuracy. Inspired by CNN-based object detection architectures, we propose a novel Voxel Diffusion Module (VDM) to enhance voxel-level representation and diffusion in point cloud data. VDM is composed of sparse 3D convolutions, submanifold sparse convolutions, and residual connections. To ensure computational efficiency, the output feature maps are downsampled to one-fourth of the original input resolution. VDM serves two primary functions: (1) diffusing foreground voxel features through sparse 3D convolutions to enrich spatial context, and (2) aggregating fine-grained spatial information to strengthen voxelwise feature representation. The enhanced voxel features produced by VDM can be seamlessly integrated into mainstream Transformer- or SSM-based detection models for accurate object classification and localization, highlighting the generalizability of our method. We evaluate VDM on several benchmark datasets by embedding it into both Transformerbased and SSM-based models. Experimental results show that our approach consistently improves detection accuracy over baseline models. Specifically, VDM-SSMs achieve 74.7 mAPH (L2) on Waymo, 72.9 NDS on nuScenes, 42.3 mAP on Argoverse 2, and 67.6 mAP on ONCE, setting new stateof-the-art performance across all datasets. Our code will be made publicly available.
Authors Qifeng Liu, Dawei Zhao, Yabo Dong, Linzhi Shang, Liang Xiao, Juan Wang, Kunkong Zhao, Dongming Lu, Qi Zhu
Comment Under review
Categories Computer Vision and Pattern Recognition
0
Loomis Painter: Reconstructing the Painting Process
Title CN 洛米斯画家：重建绘画过程
Keywords 扩散模型 · 绘画过程生成 · 风格控制 · 跨媒介生成 · 时间一致性
Summary 本文提出了一种用于多媒介绘画过程生成的统一框架，通过语义驱动的风格控制机制将多种媒介嵌入扩散模型的条件空间，并采用跨媒介风格增强和反向绘画训练策略，实现纹理演化的连贯性和创作流程的可迁移性。作者构建了大规模真实绘画过程数据集，并提出了感知距离轮廓（PDP）曲线来量化创作序列的进展，在LPIPS、DINO和CLIP等指标上取得了良好表现。
Reason 该论文研究的是艺术绘画过程的生成与建模，属于图像生成与视频合成领域，核心任务为多媒介绘画流程的扩散模型生成、风格控制与时间一致性优化。其技术路线聚焦于创意内容生成，与文档图像理解（DIU）中的OCR、版面分析、表格识别、视觉信息抽取等任务无直接关联。所提出的方法如跨媒介风格增强、反向绘画训练策略和PDP曲线，难以迁移到解决文档理解中的OCR幻觉、密集文本识别、复杂排版解析或长文档推理等核心痛点。
Abstract Step-by-step painting tutorials are vital for learning artistic techniques, but existing video resources (e.g., YouTube) lack interactivity and personalization. While recent generative models have advanced artistic image synthesis, they struggle to generalize across media and often show temporal or structural inconsistencies, hindering faithful reproduction of human creative workflows. To address this, we propose a unified framework for multi-media painting process generation with a semantics-driven style control mechanism that embeds multiple media into a diffusion models conditional space and uses cross-medium style augmentation. This enables consistent texture evolution and process transfer across styles. A reverse-painting training strategy further ensures smooth, human-aligned generation. We also build a large-scale dataset of real painting processes and evaluate cross-media consistency, temporal coherence, and final-image fidelity, achieving strong results on LPIPS, DINO, and CLIP metrics. Finally, our Perceptual Distance Profile (PDP) curve quantitatively models the creative sequence, i.e., composition, color blocking, and detail refinement, mirroring human artistic progression.
Authors Markus Pobitzer, Chang Liu, Chenyi Zhuang, Teng Long, Bin Ren, Nicu Sebe
Categories Computer Vision and Pattern Recognition
0
Refracting Reality: Generating Images with Realistic Transparent Objects
Title CN 折射现实：生成具有真实透明物体的图像
Keywords 图像生成 · 透明物体 · 折射建模 · 扩散模型 · 物理真实性
Summary 本文提出一种基于斯涅尔定律（Snell's Law）的图像生成方法，通过在扩散模型生成过程中对透明物体区域进行像素级的折射变换，并结合全景视图补全被遮挡表面，从而生成更符合光学规律的透明物体图像。实验表明该方法能显著提升透明物体在生成图像中的物理真实性。
Reason 该论文研究的是生成包含透明物体（如玻璃）的图像，重点解决光学折射、反射等物理效果在图像生成中的建模问题。其核心任务属于纯图像生成领域，与文档图像理解（DIU）、DocVQA、OCR、布局分析等无直接关联。尽管使用了多视角一致性与视觉合成技术，但这些方法针对的是自然场景下的物理渲染问题，难以迁移到文档理解中的OCR幻觉、密集文本识别或复杂排版解析等核心痛点。
Abstract Generative image models can produce convincingly real images, with plausible shapes, textures, layouts and lighting. However, one domain in which they perform notably poorly is in the synthesis of transparent objects, which exhibit refraction, reflection, absorption and scattering. Refraction is a particular challenge, because refracted pixel rays often intersect with surfaces observed in other parts of the image, providing a constraint on the color. It is clear from inspection that generative models have not distilled the laws of optics sufficiently well to accurately render refractive objects. In this work, we consider the problem of generating images with accurate refraction, given a text prompt. We synchronize the pixels within the object's boundary with those outside by warping and merging the pixels using Snell's Law of Refraction, at each step of the generation trajectory. For those surfaces that are not directly observed in the image, but are visible via refraction or reflection, we recover their appearance by synchronizing the image with a second generated image -- a panorama centered at the object -- using the same warping and merging procedure. We demonstrate that our approach generates much more optically-plausible images that respect the physical constraints.
Authors Yue Yin, Enze Tao, Dylan Campbell
Categories Computer Vision and Pattern Recognition
0
Lung-DDPM+: Efficient Thoracic CT Image Synthesis using Diffusion Probabilistic Model Computers in Biology and Medicine (CIBM)
Title CN Lung-DDPM+：使用扩散概率模型高效生成胸部CT图像
Keywords 扩散模型 · 医学图像生成 · CT图像合成 · 肺癌诊断 · 语义引导
Summary 本文提出Lung-DDPM+，一种用于胸部CT图像合成的去噪扩散概率模型，通过结节语义布局引导和肺部DPM求解器加速，在减少计算开销的同时保持生成质量。实验表明其在FLOPs、GPU内存消耗和采样速度上显著优于前作，并通过放射科医生的视觉图灵测试验证了生成图像的真实性。
Reason 该论文属于纯医学图像生成领域，研究内容为基于扩散模型的肺部CT图像合成，与文档图像理解（DIU）、DocVQA、OCR、布局分析等任务完全无关。尽管使用了视觉生成技术，但其目标是医学影像的数据增强，而非解决文档中的文本识别、结构理解或多模态推理问题，属于负面清单中的‘无关领域’。
Abstract Generative artificial intelligence (AI) has been playing an important role in various domains. Leveraging its high capability to generate high-fidelity and diverse synthetic data, generative AI is widely applied in diagnostic tasks, such as lung cancer diagnosis using computed tomography (CT). However, existing generative models for lung cancer diagnosis suffer from low efficiency and anatomical imprecision, which limit their clinical applicability. To address these drawbacks, we propose Lung-DDPM+, an improved version of our previous model, Lung-DDPM. This novel approach is a denoising diffusion probabilistic model (DDPM) guided by nodule semantic layouts and accelerated by a pulmonary DPM-solver, enabling the method to focus on lesion areas while achieving a better trade-off between sampling efficiency and quality. Evaluation results on the public LIDC-IDRI dataset suggest that the proposed method achieves 8$\times$ fewer FLOPs (floating point operations per second), 6.8$\times$ lower GPU memory consumption, and 14$\times$ faster sampling compared to Lung-DDPM. Moreover, it maintains comparable sample quality to both Lung-DDPM and other state-of-the-art (SOTA) generative models in two downstream segmentation tasks. We also conducted a Visual Turing Test by an experienced radiologist, showing the advanced quality and fidelity of synthetic samples generated by the proposed method. These experimental results demonstrate that Lung-DDPM+ can effectively generate high-quality thoracic CT images with lung nodules, highlighting its potential for broader applications, such as general tumor synthesis and lesion generation in medical imaging. The code and pretrained models are available at https://github.com/Manem-Lab/Lung-DDPM-PLUS.
Authors Yifan Jiang, Ahmad Shariftabrizi, Venkata SK. Manem
Comment Accepted by Computers in Biology and Medicine (CIBM)
Categories Computer Vision and Pattern Recognition
0
NoPe-NeRF++: Local-to-Global Optimization of NeRF with No Pose Prior
Title CN NoPe-NeRF++：无需姿态先验的NeRF局部到全局优化
Keywords NeRF · 无姿态先验 · 局部到全局优化 · 光束法平差 · 三维重建
Summary 本文提出NoPe-NeRF++，一种无需相机姿态先验的NeRF训练方法，通过结合局部特征匹配与全局光束法平差（bundle adjustment）实现更准确的位姿估计和高质量的新视角生成。该方法在标准数据集上优于现有技术。
Reason 该论文属于3D视觉与NeRF领域，研究目标是无姿态先验下的新视角合成，与文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等完全无关。尽管其优化策略涉及几何一致性与特征匹配，但这些方法针对的是三维场景重建，无法直接迁移解决文档中的OCR幻觉、密集文本识别或复杂排版理解等核心痛点。
Abstract In this paper, we introduce NoPe-NeRF++, a novel local-to-global optimization algorithm for training Neural Radiance Fields (NeRF) without requiring pose priors. Existing methods, particularly NoPe-NeRF, which focus solely on the local relationships within images, often struggle to recover accurate camera poses in complex scenarios. To overcome the challenges, our approach begins with a relative pose initialization with explicit feature matching, followed by a local joint optimization to enhance the pose estimation for training a more robust NeRF representation. This method significantly improves the quality of initial poses. Additionally, we introduce global optimization phase that incorporates geometric consistency constraints through bundle adjustment, which integrates feature trajectories to further refine poses and collectively boost the quality of NeRF. Notably, our method is the first work that seamlessly combines the local and global cues with NeRF, and outperforms state-of-the-art methods in both pose estimation accuracy and novel view synthesis. Extensive evaluations on benchmark datasets demonstrate our superior performance and robustness, even in challenging scenes, thus validating our design choices.
Authors Dongbo Shi, Shen Cao, Bojian Wu, Jinhui Guo, Lubin Fan, Renjie Chen, Ligang Liu, Jieping Ye
Categories Computer Vision and Pattern Recognition
0
BiFingerPose: Bimodal Finger Pose Estimation for Touch Devices
Title CN BiFingerPose：用于触摸设备的双模态手指姿态估计
Keywords 手指姿态估计 · 双模态感知 · 触摸屏交互 · 电容图像 · 指纹识别
Summary 本文提出了一种名为BiFingerPose的双模态手指姿态估计方法，结合电容图像和屏下指纹图像来更准确地估计手指的三维姿态（包括俯仰、偏航和翻滚角），并在用户实验中验证了其在交互效率和操作精度上的优势。同时展示了其在身份认证和交互体验中的应用潜力。
Reason 该论文研究的是基于触摸屏的手指姿态估计，属于人机交互与生物识别领域，与文档图像理解（DIU）或视觉语言模型（VLM）在文档理解中的应用无直接关联。其技术路线聚焦于传感器融合（电容图像与指纹图）以提升姿态估计精度，并未涉及OCR、布局分析、表格识别或任何DocVQA相关任务。无法迁移其方法解决文档理解中的核心痛点如幻觉、复杂排版或长文档推理。
Abstract Finger pose offers promising opportunities to expand human computer interaction capability of touchscreen devices. Existing finger pose estimation algorithms that can be implemented in portable devices predominantly rely on capacitive images, which are currently limited to estimating pitch and yaw angles and exhibit reduced accuracy when processing large-angle inputs (especially when it is greater than 45 degrees). In this paper, we propose BiFingerPose, a novel bimodal based finger pose estimation algorithm capable of simultaneously and accurately predicting comprehensive finger pose information. A bimodal input is explored, including a capacitive image and a fingerprint patch obtained from the touchscreen with an under-screen fingerprint sensor. Our approach leads to reliable estimation of roll angle, which is not achievable using only a single modality. In addition, the prediction performance of other pose parameters has also been greatly improved. The evaluation of a 12-person user study on continuous and discrete interaction tasks further validated the advantages of our approach. Specifically, BiFingerPose outperforms previous SOTA methods with over 21% improvement in prediction performance, 2.5 times higher task completion efficiency, and 23% better user operation accuracy, demonstrating its practical superiority. Finally, we delineate the application space of finger pose with respect to enhancing authentication security and improving interactive experiences, and develop corresponding prototypes to showcase the interaction potential. Our code will be available at https://github.com/XiongjunGuan/DualFingerPose.
Authors Xiongjun Guan, Zhiyu Pan, Jianjiang Feng, Jie Zhou
Categories Computer Vision and Pattern Recognition
0
Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition NeurIPS 2025
Title CN 解耦概念胜过言语：可解释的视频动作识别
Keywords 视频动作识别 · 可解释AI · 概念解耦 · 大语言模型 · 概念瓶颈
Summary 本文提出DANCE框架，用于视频动作识别的可解释性建模，通过解耦运动动态、物体和场景三类概念，在ante-hoc概念瓶颈结构下实现动作预测，并提升解释清晰度和模型调试能力。实验表明其在多个视频数据集上具有良好的解释性和竞争力的性能。
Reason 该论文研究的是视频动作识别中的可解释性问题，属于视频理解领域，其核心任务是通过解耦运动动态、物体和场景概念来提升模型解释性。尽管使用了大语言模型辅助概念提取，但整体工作与文档图像理解（DIU）的核心痛点如OCR幻觉、密集文本处理、复杂版式分析或长文档推理无关。方法难以迁移到文档领域，且属于明确列出的‘无关领域’——视频理解。
Abstract Effective explanations of video action recognition models should disentangle how movements unfold over time from the surrounding spatial context. However, existing methods based on saliency produce entangled explanations, making it unclear whether predictions rely on motion or spatial context. Language-based approaches offer structure but often fail to explain motions due to their tacit nature -- intuitively understood but difficult to verbalize. To address these challenges, we propose Disentangled Action aNd Context concept-based Explainable (DANCE) video action recognition, a framework that predicts actions through disentangled concept types: motion dynamics, objects, and scenes. We define motion dynamics concepts as human pose sequences. We employ a large language model to automatically extract object and scene concepts. Built on an ante-hoc concept bottleneck design, DANCE enforces prediction through these concepts. Experiments on four datasets -- KTH, Penn Action, HAA500, and UCF-101 -- demonstrate that DANCE significantly improves explanation clarity with competitive performance. We validate the superior interpretability of DANCE through a user study. Experimental results also show that DANCE is beneficial for model debugging, editing, and failure analysis.
Authors Jongseo Lee, Wooil Lee, Gyeong-Moon Park, Seong Tae Kim, Jinwoo Choi
Comment NeurIPS 2025 Spotlight paper. Project page: https://jong980812.github.io/DANCE/
Categories Computer Vision and Pattern Recognition
0
OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
Title CN OpenDriveVLA：基于大型视觉语言动作模型实现端到端自动驾驶
Keywords 自动驾驶 · 视觉语言动作模型 · 多模态对齐 · 轨迹规划 · 结构化视觉表示
Summary 本文提出了OpenDriveVLA，一种基于开源大语言模型的视觉语言动作模型，用于端到端自动驾驶。通过融合2D/3D实例感知的视觉表征、自车状态和语言指令，模型实现了空间接地的驾驶动作生成。引入分层视觉-语言对齐机制，将结构化视觉token映射到统一语义空间，并在自回归解码中建模智能体与环境的交互，以捕捉细粒度空间依赖和行为动态。在nuScenes数据集上的实验表明其在开环轨迹预测和驾驶问答任务上达到SOTA。
Reason 该论文属于自动驾驶领域，研究的是视觉语言动作模型在驾驶决策与轨迹规划中的应用，核心任务为端到端自动驾驶控制，与文档图像理解（DIU）或DocVQA无关。尽管其使用了多模态对齐和结构化视觉token等技术，但应用场景和问题定义均远离文档理解的核心痛点（如OCR幻觉、复杂版式解析、长文档推理等），无法迁移至DIU领域。
Abstract We present OpenDriveVLA, a Vision Language Action model designed for end-to-end autonomous driving, built upon open-source large language models. OpenDriveVLA generates spatially grounded driving actions by leveraging multimodal inputs, including 2D and 3D instance-aware visual representations, ego vehicle states, and language commands. To bridge the modality gap between driving visual representations and language embeddings, we introduce a hierarchical vision language alignment process, projecting both 2D and 3D structured visual tokens into a unified semantic space. Furthermore, we incorporate structured agent environment ego interaction modeling into the autoregressive decoding process, enabling the model to capture fine-grained spatial dependencies and behavior-aware dynamics critical for reliable trajectory planning. Extensive experiments on the nuScenes dataset demonstrate that OpenDriveVLA achieves state-of-the-art results across open-loop trajectory planning and driving-related question answering tasks. Qualitative analyses further illustrate its capability to follow high-level driving commands and generate trajectories under challenging scenarios, highlighting its potential for next-generation end-to-end autonomous driving.
Authors Xingcheng Zhou, Xuyuan Han, Feng Yang, Yunpu Ma, Volker Tresp, Alois Knoll
Categories Computer Vision and Pattern Recognition
0
LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generation
Title CN LinVideo：一种面向O(n)注意力的高效视频生成后训练框架
Keywords 视频生成 · 扩散模型 · 线性注意力 · 后训练 · 效率优化
Summary 本文提出了LinVideo，一种无需数据的后训练框架，通过选择性地将视频扩散模型中的部分自注意力模块替换为线性注意力，实现高效的视频生成。该方法引入了可学习的层选择机制和任意时刻分布匹配（ADM）目标，在保持生成质量的同时显著提升速度。
Reason 该论文属于视频生成领域，核心解决的是视频扩散模型中自注意力计算复杂度高的问题，提出了一种将自注意力替换为线性注意力的后训练方法。尽管其在效率优化方面有贡献，但研究任务与文档图像理解（DIU）无关，不涉及DocVQA、布局分析、表格识别或OCR-free等文档理解核心任务，也无法直接迁移用于解决OCR幻觉、细粒度定位或长文档推理等DIU痛点。
Abstract Video diffusion models (DMs) have enabled high-quality video synthesis. However, their computation costs scale quadratically with sequence length because self-attention has quadratic complexity. While linear attention lowers the cost, fully replacing quadratic attention requires expensive pretraining due to the limited expressiveness of linear attention and the complexity of spatiotemporal modeling in video generation. In this paper, we present LinVideo, an efficient data-free post-training framework that replaces a target number of self-attention modules with linear attention while preserving the original model's performance. First, we observe a significant disparity in the replaceability of different layers. Instead of manual or heuristic choices, we frame layer selection as a binary classification problem and propose selective transfer, which automatically and progressively converts layers to linear attention with minimal performance impact. Additionally, to overcome the ineffectiveness and inefficiency of existing objectives for this transfer process, we introduce an anytime distribution matching (ADM) objective that aligns the distributions of samples across any timestep along the sampling trajectory. This objective is efficient and recovers model performance. Extensive experiments show that our method achieves a 1.25-2.00x speedup while preserving generation quality, and our 4-step distilled model further delivers a 15.92x latency reduction with minimal visual quality drop.
Authors Yushi Huang, Xingtong Ge, Ruihao Gong, Chengtao Lv, Jun Zhang
Comment Code will be released upon acceptance
Categories Computer Vision and Pattern Recognition
0
Resolving Sentiment Discrepancy for Multimodal Sentiment Detection via Semantics Completion and Decomposition Pattern Recognition
Title CN 通过语义补全与分解解决多模态情感检测中的情感差异
Keywords 多模态情感分析 · 情感差异 · 语义补全 · 对比学习 · 跨模态融合
Summary 本文提出了一种语义补全与分解网络（CoDe），用于解决用户生成内容中图文情感不一致的问题。通过引入图像内文本的语义进行补全，并利用对比学习和独占投影分解模态间差异情感，从而提升多模态情感分类性能。实验在四个数据集上验证了方法的有效性。
Reason 该论文研究的是多模态情感分析中的情感不一致问题，属于社交媒体内容的情感分类任务，核心关注点为图像与文本之间的情感冲突建模。其方法论聚焦于情感级语义融合与分解，并未涉及文档结构、布局理解、OCR增强或细粒度视觉文本对齐等文档图像理解（DIU）的核心挑战。无法迁移用于解决OCR幻觉、复杂排版解析或长文档推理等问题，属于无关领域的应用型工作。
Abstract With the proliferation of social media posts in recent years, the need to detect sentiments in multimodal (image-text) content has grown rapidly. Since posts are user-generated, the image and text from the same post can express different or even contradictory sentiments, leading to potential \textbf{sentiment discrepancy}. However, existing works mainly adopt a single-branch fusion structure that primarily captures the consistent sentiment between image and text. The ignorance or implicit modeling of discrepant sentiment results in compromised unimodal encoding and limited performance. In this paper, we propose a semantics Completion and Decomposition (CoDe) network to resolve the above issue. In the semantics completion module, we complement image and text representations with the semantics of the in-image text, helping bridge the sentiment gap. In the semantics decomposition module, we decompose image and text representations with exclusive projection and contrastive learning, thereby explicitly capturing the discrepant sentiment between modalities. Finally, we fuse image and text representations by cross-attention and combine them with the learned discrepant sentiment for final classification. Extensive experiments on four datasets demonstrate the superiority of CoDe and the effectiveness of each proposed module.
Authors Daiqing Wu, Dongbao Yang, Huawen Shen, Can Ma, Yu Zhou
Comment Accepted by Pattern Recognition
Categories Computer Vision and Pattern Recognition, Computation and Language
0
Blind Deconvolution for Color Images Using Normalized Quaternion Kernels
Title CN 基于归一化四元数核的彩色图像盲解卷积
Keywords 盲解卷积 · 四元数核 · 彩色图像恢复 · 图像去模糊 · 通道相关性
Summary 本文提出了一种基于归一化四元数核的彩色图像盲解卷积方法。该方法利用四元数卷积核同时处理RGB三通道，保留颜色通道间的耦合关系，并引入归一化机制以保持图像强度一致性。实验表明该方法在真实模糊数据集上能有效去除伪影并提升去模糊效果。
Reason 该论文研究的是通用彩色图像的盲解卷积问题，属于传统图像恢复领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其方法聚焦于通过四元数核建模彩色通道间的依赖关系以提升去模糊效果，但并未涉及OCR、布局分析、表格识别或多模态语言模型在文档上的应用。无法迁移解决DIU中的核心痛点如OCR幻觉、复杂排版或长文档推理。
Abstract In this work, we address the challenging problem of blind deconvolution for color images. Existing methods often convert color images to grayscale or process each color channel separately, which overlooking the relationships between color channels. To handle this issue, we formulate a novel quaternion fidelity term designed specifically for color image blind deconvolution. This fidelity term leverages the properties of quaternion convolution kernel, which consists of four kernels: one that functions similarly to a non-negative convolution kernel to capture the overall blur, and three additional convolution kernels without constraints corresponding to red, green and blue channels respectively model their unknown interdependencies. In order to preserve image intensity, we propose to use the normalized quaternion kernel in the blind deconvolution process. Extensive experiments on real datasets of blurred color images show that the proposed method effectively removes artifacts and significantly improves deblurring effect, demonstrating its potential as a powerful tool for color image deconvolution.
Authors Yuming Yang, Michael K. Ng, Zhigang Jia, Wei Wang
Categories Computer Vision and Pattern Recognition
0
HPPP: Halpern-type Preconditioned Proximal Point Algorithms and Applications to Image Restoration
Title CN HPPP：Halpern型预条件邻近点算法及其在图像恢复中的应用
Keywords 图像恢复 · 邻近点算法 · 去噪先验 · 算子分裂 · 优化算法
Summary 本文提出了一种Halpern型预条件邻近点（HPPP）算法，结合Halpern迭代的强收敛性和加速特性，改进了现有的退化预条件邻近点方法。通过将HPPP与Plug-and-Play（PnP）去噪先验结合，构建了一种加速的图像恢复算法，并在多个示例和图像恢复任务上验证了其有效性。
Reason 该论文属于纯图像修复领域，研究的是基于算子分裂和去噪先验的数值优化方法，尽管应用于图像恢复，但其核心问题与文档图像理解（DIU）无关。未涉及OCR、布局分析、表格识别、多模态语言模型或文档推理等DIU核心任务。且‘图像恢复’属于负面清单中的无关领域（纯图像生成/修复），无法迁移解决DIU中的OCR幻觉、复杂排版或长文档推理等痛点。
Abstract Recently, the degenerate preconditioned proximal point (PPP) method provides a unified and flexible framework for designing and analyzing operator-splitting algorithms such as Douglas-Rachford (DR). However, the degenerate PPP method exhibits weak convergence in the infinite-dimensional Hilbert space and lacks accelerated variants. To address these issues, we propose a Halpern-type PPP (HPPP) algorithm, which leverages the strong convergence and acceleration properties of Halpern's iteration method. Moreover, we propose a novel algorithm for image restoration by combining HPPP with denoiser priors such as Plug-and-Play (PnP) prior, which can be viewed as an accelerated PnP method. Finally, numerical experiments including several toy examples and image restoration validate the effectiveness of our proposed algorithms.
Authors Shuchang Zhang, Hui Zhang, Hongxia Wang
Categories Computer Vision and Pattern Recognition
0
CLIMB-3D: Continual Learning for Imbalanced 3D Instance Segmentation BMVC 2025
Title CN CLIMB-3D：面向不平衡3D实例分割的持续学习
Keywords 3D实例分割 · 持续学习 · 类别不平衡 · 伪标签 · CBR
Summary 本文提出CLIMB-3D，一个用于类增量不平衡3D实例分割的统一框架。基于示例回放策略，引入伪标签生成器（PLG）以扩展对已学类别的监督，并设计类别平衡重加权（CBR）方案来缓解伪标签对高频类别的偏差，在ScanNet200和ScanNetV2上取得SOTA结果。
Reason 该论文研究的是3D视觉中的实例分割问题，属于自动驾驶或具身智能相关领域，与文档图像理解（DIU）无直接关联。尽管其提出的伪标签生成和类别重平衡方法具有一定通用性，但迁移至文档理解场景（如OCR幻觉、复杂排版解析）的路径不明确，且核心任务远离DIU本题。
Abstract While 3D instance segmentation (3DIS) has advanced significantly, most existing methods assume that all object classes are known in advance and uniformly distributed. However, this assumption is unrealistic in dynamic, real-world environments where new classes emerge gradually and exhibit natural imbalance. Although some approaches address the emergence of new classes, they often overlook class imbalance, which leads to suboptimal performance, particularly on rare categories. To tackle this, we propose \ourmethodbf, a unified framework for \textbf{CL}ass-incremental \textbf{Imb}alance-aware \textbf{3D}IS. Building upon established exemplar replay (ER) strategies, we show that ER alone is insufficient to achieve robust performance under memory constraints. To mitigate this, we introduce a novel pseudo-label generator (PLG) that extends supervision to previously learned categories by leveraging predictions from a frozen model trained on prior tasks. Despite its promise, PLG tends to be biased towards frequent classes. Therefore, we propose a class-balanced re-weighting (CBR) scheme that estimates object frequencies from pseudo-labels and dynamically adjusts training bias, without requiring access to past data. We design and evaluate three incremental scenarios for 3DIS on the challenging ScanNet200 dataset and additionally validate our method for semantic segmentation on ScanNetV2. Our approach achieves state-of-the-art results, surpassing prior work by up to 16.76\% mAP for instance segmentation and approximately 30\% mIoU for semantic segmentation, demonstrating strong generalisation across both frequent and rare classes. Code is available at: https://github.com/vgthengane/CLIMB3D
Authors Vishal Thengane, Jean Lahoud, Hisham Cholakkal, Rao Muhammad Anwer, Lu Yin, Xiatian Zhu, Salman Khan
Comment Accepted at BMVC 2025
Categories Computer Vision and Pattern Recognition
0
QueryOcc: Query-based Self-Supervision for 3D Semantic Occupancy
Title CN QueryOcc：基于查询的自监督3D语义占据学习
Keywords 3D占据 · 自监督学习 · 查询机制 · 4D建模 · 自动驾驶
Summary 本文提出QueryOcc，一种基于查询的自监督框架，用于从图像中学习连续的3D语义占据。通过在相邻帧中采样独立的4D时空查询，结合伪点云或原始激光雷达数据进行监督，实现无需人工标注的3D场景理解。该方法在Occ3D-nuScenes基准上显著优于现有方法，并保持较高推理速度。
Reason 该论文研究的是3D视觉中的语义占据（Semantic Occupancy）问题，属于自动驾驶和3D场景理解领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心技术如4D时空查询、连续3D占据建模、远距离区域压缩等，均针对3D空间结构建模，并不适用于解决文档图像中的OCR幻觉、复杂排版、细粒度文本定位或长文档推理等核心痛点。
Abstract Learning 3D scene geometry and semantics from images is a core challenge in computer vision and a key capability for autonomous driving. Since large-scale 3D annotation is prohibitively expensive, recent work explores self-supervised learning directly from sensor data without manual labels. Existing approaches either rely on 2D rendering consistency, where 3D structure emerges only implicitly, or on discretized voxel grids from accumulated lidar point clouds, limiting spatial precision and scalability. We introduce QueryOcc, a query-based self-supervised framework that learns continuous 3D semantic occupancy directly through independent 4D spatio-temporal queries sampled across adjacent frames. The framework supports supervision from either pseudo-point clouds derived from vision foundation models or raw lidar data. To enable long-range supervision and reasoning under constant memory, we introduce a contractive scene representation that preserves near-field detail while smoothly compressing distant regions. QueryOcc surpasses previous camera-based methods by 26% in semantic RayIoU on the self-supervised Occ3D-nuScenes benchmark while running at 11.6 FPS, demonstrating that direct 4D query supervision enables strong self-supervised occupancy learning. https://research.zenseact.com/publications/queryocc/
Authors Adam Lilja, Ji Lan, Junsheng Fu, Lars Hammarstrand
Categories Computer Vision and Pattern Recognition
0
FisheyeGaussianLift: BEV Feature Lifting for Surround-View Fisheye Camera Perception IMVIP 2025
Title CN FisheyeGaussianLift：用于环视鱼眼相机感知的BEV特征提升
Keywords BEV分割 · 鱼眼相机 · 几何失真 · 3D高斯投影 · 语义分割
Summary 本文提出了一种名为FisheyeGaussianLift的BEV分割框架，用于处理多摄像头鱼眼图像中的极端非线性畸变。通过标定后的几何反投影和逐像素深度分布估计，将每个图像像素以高斯参数化方式提升到3D空间，并利用可微溅射融合生成连续且具有不确定性感知的BEV语义图。实验表明该方法在复杂停车和城市驾驶场景中表现优异。
Reason 该论文研究的是自动驾驶场景下的鱼眼图像BEV（鸟瞰图）语义分割，属于自动驾驶和3D视觉领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。尽管其方法涉及高分辨率图像处理和几何失真建模，但应用场景和核心问题（如OCR幻觉、文本布局分析、表格识别）完全不匹配，无法迁移至文档理解领域。
Abstract Accurate BEV semantic segmentation from fisheye imagery remains challenging due to extreme non-linear distortion, occlusion, and depth ambiguity inherent to wide-angle projections. We present a distortion-aware BEV segmentation framework that directly processes multi-camera high-resolution fisheye images,utilizing calibrated geometric unprojection and per-pixel depth distribution estimation. Each image pixel is lifted into 3D space via Gaussian parameterization, predicting spatial means and anisotropic covariances to explicitly model geometric uncertainty. The projected 3D Gaussians are fused into a BEV representation via differentiable splatting, producing continuous, uncertainty-aware semantic maps without requiring undistortion or perspective rectification. Extensive experiments demonstrate strong segmentation performance on complex parking and urban driving scenarios, achieving IoU scores of 87.75% for drivable regions and 57.26% for vehicles under severe fisheye distortion and diverse environmental conditions.
Authors Shubham Sonarghare, Prasad Deshpande, Ciaran Hogan, Deepika-Rani Kaliappan-Mahalingam, Ganesh Sistu
Comment 8 pages, 3 figures, published in IMVIP 2025 conference
Categories Computer Vision and Pattern Recognition
0
Scaling Self-Supervised and Cross-Modal Pretraining for Volumetric CT Transformers
Title CN 用于体积分层CT变换器的可扩展自监督和跨模态预训练
Keywords 3D医学影像 · 自监督学习 · 跨模态预训练 · Transformer · CT分析
Summary 本文提出了SPECTRE，一种基于纯Transformer架构的体积分层CT基础模型，通过结合DINO式自蒸馏和基于SigLIP的视觉-语言对齐，在公开CT数据集上进行自监督与跨模态预训练。该框架联合优化局部高分辨率特征提取器和全局上下文建模器，解决了3D注意力计算不可行的问题，并在多个CT基准上实现了优于先前模型的零样本和微调性能。
Reason 该论文研究的是3D医学影像（CT）领域的基础模型，属于医学图像分析范畴，与文档图像理解（DIU）任务如DocVQA、表格识别、布局分析等无直接关联。尽管其采用了自监督和跨模态预训练策略，但面对的是体数据中的几何各向异性、高分辨率3D注意力等医学成像特有问题，方法难以迁移到文档图像的理解任务中。且文档图像不涉及体积数据或临床报告对齐问题，技术迁移路径不成立。
Abstract We introduce SPECTRE, a fully transformer-based foundation model for volumetric computed tomography (CT). Our Self-Supervised & Cross-Modal Pretraining for CT Representation Extraction (SPECTRE) approach utilizes scalable 3D Vision Transformer architectures and modern self-supervised and vision-language pretraining strategies to learn general-purpose CT representations. Volumetric CT poses unique challenges, such as extreme token scaling, geometric anisotropy, and weak or noisy clinical supervision, that make standard transformer and contrastive learning recipes ineffective out of the box. The framework jointly optimizes a local transformer for high-resolution volumetric feature extraction and a global transformer for whole-scan context modeling, making large-scale 3D attention computationally tractable. Notably, SPECTRE is trained exclusively on openly available CT datasets, demonstrating that high-performing, generalizable representations can be achieved without relying on private data. Pretraining combines DINO-style self-distillation with SigLIP-based vision-language alignment using paired radiology reports, yielding features that are both geometrically consistent and clinically meaningful. Across multiple CT benchmarks, SPECTRE consistently outperforms prior CT foundation models in both zero-shot and fine-tuned settings, establishing SPECTRE as a scalable, open, and fully transformer-based foundation model for 3D medical imaging.
Authors Cris Claessens, Christiaan Viviers, Giacomo D'Amicantonio, Egor Bondarev, Fons van der Sommen
Categories Computer Vision and Pattern Recognition
0
SING3R-SLAM: Submap-based Indoor Monocular Gaussian SLAM with 3D Reconstruction Priors
Title CN SING3R-SLAM：基于子图的室内单目高斯SLAM与3D重建先验
Keywords SLAM · 3D重建 · 高斯表示 · 室内建图 · 单目视觉
Summary 本文提出SING3R-SLAM，一种基于高斯表示的室内单目SLAM系统，通过构建局部一致的子图并融合为全局一致的紧凑高斯地图，实现高效的3D场景重建与相机跟踪。方法利用3D重建先验提升几何细节和位姿精度，在真实数据集上实现了领先的跟踪与渲染性能。
Reason 该论文属于具身智能与3D视觉领域，研究的是SLAM（同步定位与建图）和密集3D重建，完全不涉及文档图像理解、多模态语言模型或文本信息提取等DIU核心问题。其技术路线聚焦于几何建模与相机位姿优化，无法迁移至解决OCR幻觉、复杂排版理解或长文档推理等文档任务。
Abstract Recent advances in dense 3D reconstruction enable the accurate capture of local geometry; however, integrating them into SLAM is challenging due to drift and redundant point maps, which limit efficiency and downstream tasks, such as novel view synthesis. To address these issues, we propose SING3R-SLAM, a globally consistent and compact Gaussian-based dense RGB SLAM framework. The key idea is to combine locally consistent 3D reconstructions with a unified global Gaussian representation that jointly refines scene geometry and camera poses, enabling efficient and versatile 3D mapping for multiple downstream applications. SING3R-SLAM first builds locally consistent submaps through our lightweight tracking and reconstruction module, and then progressively aligns and fuses them into a global Gaussian map that enforces cross-view geometric consistency. This global map, in turn, provides feedback to correct local drift and enhance the robustness of tracking. Extensive experiments demonstrate that SING3R-SLAM achieves state-of-the-art tracking, 3D reconstruction, and novel view rendering, resulting in over 12% improvement in tracking and producing finer, more detailed geometry, all while maintaining a compact and memory-efficient global representation on real-world datasets.
Authors Kunyi Li, Michael Niemeyer, Sen Wang, Stefano Gasperini, Nassir Navab, Federico Tombari
Categories Computer Vision and Pattern Recognition
0
Continual Alignment for SAM: Rethinking Foundation Models for Medical Image Segmentation in Continual Learning
Title CN 面向医学图像分割的持续对齐：为持续学习重新思考基础模型
Keywords 医学图像分割 · 持续学习 · SAM · 特征对齐 · 灾难性遗忘
Summary 本文提出CA-SAM，一种基于Segment Anything Model（SAM）的持续学习框架，用于医学图像分割。通过引入轻量化的Alignment Layer来对齐编码器-解码器特征分布，并在不同数据流中动态适配，以缓解灾难性遗忘，同时利用SAM的零样本先验保持泛化能力。在九个医学分割数据集上实现了SOTA性能。
Reason 该论文聚焦于医学图像分割中的持续学习问题，属于纯计算机视觉（CV）领域，且应用场景为医疗影像，与文档图像理解（DIU）任务无直接关联。尽管其提出了一种轻量级适配模块（Alignment Layer）和缓解灾难性遗忘的策略，但方法设计针对的是医学图像的分布偏移问题，无法直接迁移解决文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract In medical image segmentation, heterogeneous privacy policies across institutions often make joint training on pooled datasets infeasible, motivating continual image segmentation-learning from data streams without catastrophic forgetting. While the Segment Anything Model (SAM) offers strong zero-shot priors and has been widely fine-tuned across downstream tasks, its large parameter count and computational overhead challenge practical deployment. This paper demonstrates that the SAM paradigm is highly promising once its computational efficiency and performance can be balanced. To this end, we introduce the Alignment Layer, a lightweight, plug-and-play module which aligns encoder-decoder feature distributions to efficiently adapt SAM to specific medical images, improving accuracy while reducing computation. Building on SAM and the Alignment Layer, we then propose Continual Alignment for SAM (CA-SAM), a continual learning strategy that automatically adapts the appropriate Alignment Layer to mitigate catastrophic forgetting, while leveraging SAM's zero-shot priors to preserve strong performance on unseen medical datasets. Experimented across nine medical segmentation datasets under continual-learning scenario, CA-SAM achieves state-of-the-art performance. Our code, models and datasets will be released on \mbox{https://github.com/azzzzyo/Continual-Alignment-for-SAM.}
Authors Jiayi Wang, Wei Dai, Haoyu Wang, Sihan Yang, Haixia Bi, Jian Sun
Categories Computer Vision and Pattern Recognition
0
VLA-4D: Embedding 4D Awareness into Vision-Language-Action Models for SpatioTemporally Coherent Robotic Manipulation
Title CN VLA-4D：将四维感知嵌入视觉-语言-动作模型以实现时空连贯的机器人操作
Keywords 视觉-语言-动作模型 · 4D感知 · 机器人操作 · 时空一致性 · 多模态表示
Summary 本文提出VLA-4D模型，通过引入4D感知（3D空间+时间）来提升视觉-语言-动作模型在机器人操作中的时空连贯性。方法包括4D感知的视觉表征和时空动作表征，并在扩展的VLA数据集上进行实验验证。重点在于增强机器人动作的空间平滑性和时间连续性。
Reason 该论文属于具身智能与机器人领域，研究的是机器人操控中的时空一致性问题，核心关注3D位置与时间维度的动作控制。尽管涉及视觉-语言-动作多模态建模，但其方法设计面向物理世界连续控制任务，与文档图像理解（DIU）中的OCR幻觉、复杂排版、长文本推理等核心痛点无直接关联，无法迁移用于解决文档理解任务。
Abstract Vision-language-action (VLA) models show potential for general robotic tasks, but remain challenging in spatiotemporally coherent manipulation, which requires fine-grained representations. Typically, existing methods embed 3D positions into visual representations to enhance the spatial precision of actions. However, these methods struggle to achieve temporally coherent control over action execution. In this work, we propose VLA-4D, a general VLA model with 4D awareness for spatiotemporally coherent robotic manipulation. Our model is guided by two key designs: 1) 4D-aware visual representation. We extract visual features, embed 1D time into 3D positions for 4D embeddings, and fuse them into a unified visual representation via a cross-attention mechanism. 2) Spatiotemporal action representation. We extend conventional spatial action representations with temporal information to enable the spatiotemporal planning, and align the multimodal representations into the LLM for spatiotemporal action prediction. Within this unified framework, the designed visual and action representations jointly make robotic manipulation spatially-smooth and temporally-coherent. In addition, we extend the VLA dataset with temporal action annotations for fine-tuning our model. Extensive experiments have been conducted to verify the superiority of our method across different tasks of robotic manipulation.
Authors Hanyu Zhou, Chuanhao Ma, Gim Hee Lee
Categories Computer Vision and Pattern Recognition
0
Real Noise Decoupling for Hyperspectral Image Denoising
Title CN 用于高光谱图像去噪的真实噪声解耦
Keywords 高光谱图像去噪 · 噪声解耦 · 小波网络 · 多阶段学习
Summary 本文提出了一种多阶段噪声解耦框架，将高光谱图像中的复杂噪声分解为显式建模和隐式建模两部分，并分别通过预训练和高频小波引导网络进行处理。实验表明该方法在真实配对数据上优于现有去噪方法。
Reason 该论文研究的是高光谱图像（HSI）去噪问题，属于遥感成像与图像恢复领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其核心方法聚焦于复杂噪声的分解建模和多阶段训练策略，虽然在图像处理上有技术深度，但无法迁移解决OCR幻觉、密集文本识别、复杂版面分析或长文档推理等DIU核心痛点。
Abstract Hyperspectral image (HSI) denoising is a crucial step in enhancing the quality of HSIs. Noise modeling methods can fit noise distributions to generate synthetic HSIs to train denoising networks. However, the noise in captured HSIs is usually complex and difficult to model accurately, which significantly limits the effectiveness of these approaches. In this paper, we propose a multi-stage noise-decoupling framework that decomposes complex noise into explicitly modeled and implicitly modeled components. This decoupling reduces the complexity of noise and enhances the learnability of HSI denoising methods when applied to real paired data. Specifically, for explicitly modeled noise, we utilize an existing noise model to generate paired data for pre-training a denoising network, equipping it with prior knowledge to handle the explicitly modeled noise effectively. For implicitly modeled noise, we introduce a high-frequency wavelet guided network. Leveraging the prior knowledge from the pre-trained module, this network adaptively extracts high-frequency features to target and remove the implicitly modeled noise from real paired HSIs. Furthermore, to effectively eliminate all noise components and mitigate error accumulation across stages, a multi-stage learning strategy, comprising separate pre-training and joint fine-tuning, is employed to optimize the entire framework. Extensive experiments on public and our captured datasets demonstrate that our proposed framework outperforms state-of-the-art methods, effectively handling complex real-world noise and significantly enhancing HSI quality.
Authors Yingkai Zhang, Tao Zhang, Jing Nie, Ying Fu
Categories Computer Vision and Pattern Recognition
0
TDSNNs: Competitive Topographic Deep Spiking Neural Networks for Visual Cortex Modeling AAAI 2026 (Oral)
Title CN TDSNNs：用于视觉皮层建模的竞争性拓扑深度脉冲神经网络
Keywords 脉冲神经网络 · 拓扑组织 · 视觉皮层建模 · 时空约束 · 生物启发模型
Summary 本文提出了一种具有时空约束的深度脉冲神经网络（TDSNNs），用于模拟灵长类动物视觉皮层中的拓扑组织结构。通过引入STC损失函数，模型能够在保持高性能的同时生成类脑的层级功能组织，在ImageNet上无精度损失，并优于现有拓扑ANN在脑相似性和鲁棒性方面的表现。
Reason 该论文研究的是基于脉冲神经网络（SNN）的视觉皮层建模，属于计算神经科学与生物启发模型领域，核心目标是提升模型的生物保真度而非解决文档图像理解（DIU）中的实际任务如DocVQA、布局分析或OCR-free理解。尽管其提出的时空约束（STC）损失可能在理论上对表示学习有启发，但该工作未涉及多模态大模型（VLM）、文档结构建模、高分辨率处理或视觉-语言对齐等DIU关键问题，且迁移路径极不明确，属于无关领域。
Abstract The primate visual cortex exhibits topographic organization, where functionally similar neurons are spatially clustered, a structure widely believed to enhance neural processing efficiency. While prior works have demonstrated that conventional deep ANNs can develop topographic representations, these models largely neglect crucial temporal dynamics. This oversight often leads to significant performance degradation in tasks like object recognition and compromises their biological fidelity. To address this, we leverage spiking neural networks (SNNs), which inherently capture spike-based temporal dynamics and offer enhanced biological plausibility. We propose a novel Spatio-Temporal Constraints (STC) loss function for topographic deep spiking neural networks (TDSNNs), successfully replicating the hierarchical spatial functional organization observed in the primate visual cortex from low-level sensory input to high-level abstract representations. Our results show that STC effectively generates representative topographic features across simulated visual cortical areas. While introducing topography typically leads to significant performance degradation in ANNs, our spiking architecture exhibits a remarkably small performance drop (No drop in ImageNet top-1 accuracy, compared to a 3% drop observed in TopoNet, which is the best-performing topographic ANN so far) and outperforms topographic ANNs in brain-likeness. We also reveal that topographic organization facilitates efficient and stable temporal information processing via the spike mechanism in TDSNNs, contributing to model robustness. These findings suggest that TDSNNs offer a compelling balance between computational performance and brain-like features, providing not only a framework for interpreting neural science phenomena but also novel insights for designing more efficient and robust deep learning models.
Authors Deming Zhou, Yuetong Fang, Zhaorui Wang, Renjing Xu
Comment AAAI 2026 (Oral)
Categories Computer Vision and Pattern Recognition
0
PostCam: Camera-Controllable Novel-View Video Generation with Query-Shared Cross-Attention
Title CN PostCam：通过查询共享交叉注意力实现相机可控的新视角视频生成
Keywords 视频生成 · 新视角合成 · 相机控制 · 交叉注意力 · 动态场景
Summary 本文提出PostCam，一种支持后期编辑相机轨迹的视频新视角生成框架。通过引入查询共享交叉注意力模块，融合6自由度相机姿态与2D渲染帧两种控制信号，提升运动操控精度与生成视频质量。采用两阶段训练策略，先学习粗略相机控制，再结合视觉信息优化细节。在真实与合成数据集上均表现出优越性能。
Reason 该论文研究的是动态场景下的新视角视频生成与相机轨迹编辑，属于视频理解与生成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心技术如查询共享交叉注意力和6-DoF相机控制信号融合，主要面向三维场景重建与视频重渲染，无法迁移解决OCR幻觉、复杂排版解析、细粒度文本定位等文档理解核心问题。
Abstract We propose PostCam, a framework for novel-view video generation that enables post-capture editing of camera trajectories in dynamic scenes. We find that existing video recapture methods suffer from suboptimal camera motion injection strategies; such suboptimal designs not only limit camera control precision but also result in generated videos that fail to preserve fine visual details from the source video. To achieve more accurate and flexible motion manipulation, PostCam introduces a query-shared cross-attention module. It integrates two distinct forms of control signals: the 6-DoF camera poses and the 2D rendered video frames. By fusing them into a unified representation within a shared feature space, our model can extract underlying motion cues, which enhances both control precision and generation quality. Furthermore, we adopt a two-stage training strategy: the model first learns coarse camera control from pose inputs, and then incorporates visual information to refine motion accuracy and enhance visual fidelity. Experiments on both real-world and synthetic datasets demonstrate that PostCam outperforms state-of-the-art methods by over 20% in camera control precision and view consistency, while achieving the highest video generation quality. Our project webpage is publicly available at: https://cccqaq.github.io/PostCam.github.io/
Authors Yipeng Chen, Zhichao Ye, Zhenzhou Fang, Xinyu Chen, Xiaoyu Zhang, Jialing Liu, Nan Wang, Haomin Liu, Guofeng Zhang
Categories Computer Vision and Pattern Recognition
0
Navigating in the Dark: A Multimodal Framework and Dataset for Nighttime Traffic Sign Recognition
Title CN 在黑暗中导航：一种用于夜间交通标志识别的多模态框架与数据集
Keywords 交通标志识别 · 低光照图像 · 多模态学习 · 图像增强 · 图神经网络
Summary 本文提出了一个名为INTSD的大规模夜间交通标志数据集，并设计了LENS-Net框架进行识别，该框架结合图像增强、跨模态注意力和图推理来提升夜间识别性能。研究主要面向智能交通系统和自动驾驶场景。
Reason 该论文聚焦于夜间交通标志识别，属于自动驾驶和交通监控领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务无直接关联。其提出的方法和数据集针对的是室外场景下的低光照目标检测与分类问题，无法迁移解决文档中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Traffic signboards are vital for road safety and intelligent transportation systems, enabling navigation and autonomous driving. Yet, recognizing traffic signs at night remains challenging due to visual noise and scarcity of public nighttime datasets. Despite advances in vision architectures, existing methods struggle with robustness under low illumination and fail to leverage complementary mutlimodal cues effectively. To overcome these limitations, firstly, we introduce INTSD, a large-scale dataset comprising street-level night-time images of traffic signboards collected across diverse regions of India. The dataset spans 41 traffic signboard classes captured under varying lighting and weather conditions, providing a comprehensive benchmark for both detection and classification tasks. To benchmark INTSD for night-time sign recognition, we conduct extensive evaluations using state-of-the-art detection and classification models. Secondly, we propose LENS-Net, which integrates an adaptive image enhancement detector for joint illumination correction and sign localization, followed by a structured multimodal CLIP-GCNN classifier that leverages cross-modal attention and graph-based reasoning for robust and semantically consistent recognition. Our method surpasses existing frameworks, with ablation studies confirming the effectiveness of its key components. The dataset and code for LENS-Net is publicly available for research.
Authors Aditya Mishra, Akshay Agarwal, Haroon Lone
Categories Computer Vision and Pattern Recognition
0
Investigating self-supervised representations for audio-visual deepfake detection
Title CN 研究自监督表示在音视频深度伪造检测中的应用
Keywords 自监督表示 · 深度伪造检测 · 多模态学习 · 泛化性 · 音视频分析
Summary 本文系统评估了自监督表示在音视频深度伪造检测中的有效性，探索了音频、视频及多模态下的特征表现，发现现有自监督方法虽能捕捉有意义的语义信息且具有跨模态互补性，但在不同数据集间泛化能力差，主要挑战源于数据特性而非模型学习到虚假模式。
Reason 该论文聚焦于音视频深度伪造检测，属于视频理解与安全领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了自监督表示方法，但其任务目标、数据模态（音频+视频）和应用场景均远离文档理解的核心痛点（如OCR幻觉、复杂版式、细粒度定位等），无法迁移为解决DIU问题的有效工具。
Abstract Self-supervised representations excel at many vision and speech tasks, but their potential for audio-visual deepfake detection remains underexplored. Unlike prior work that uses these features in isolation or buried within complex architectures, we systematically evaluate them across modalities (audio, video, multimodal) and domains (lip movements, generic visual content). We assess three key dimensions: detection effectiveness, interpretability of encoded information, and cross-modal complementarity. We find that most self-supervised features capture deepfake-relevant information, and that this information is complementary. Moreover, models primarily attend to semantically meaningful regions rather than spurious artifacts. Yet none generalize reliably across datasets. This generalization failure likely stems from dataset characteristics, not from the features themselves latching onto superficial patterns. These results expose both the promise and fundamental challenges of self-supervised representations for deepfake detection: while they learn meaningful patterns, achieving robust cross-domain performance remains elusive.
Authors Dragos-Alexandru Boldisor, Stefan Smeu, Dan Oneata, Elisabeta Oneata
Categories Computer Vision and Pattern Recognition
0
Exploring the added value of pretherapeutic MR descriptors in predicting breast cancer pathologic complete response to neoadjuvant chemotherapy
Title CN 探讨治疗前MRI描述符在预测乳腺癌新辅助化疗病理完全反应中的附加价值
Keywords 乳腺癌 · MRI · 病理完全反应 · 新辅助化疗 · 机器学习
Summary 本研究回顾性分析了129例乳腺癌患者的新辅助化疗前MRI特征，使用BI-RADS和水肿评分系统进行描述，并通过逻辑回归和随机森林模型分析这些特征与病理完全缓解（pCR）的关系。结果显示非毛刺状边缘和单灶性是独立预测因子，加入MRI特征可提升预测模型性能。
Reason 该论文属于医学影像分析领域，研究内容为利用MRI特征预测乳腺癌对化疗的反应，属于特定医疗场景下的应用型研究，未涉及文档图像理解（DIU）、多模态大模型（VLM）或相关底层方法创新。属于‘平行下游应用’，且与文档结构、OCR、布局分析、表格识别等DIU核心任务无关。
Abstract Objectives: To evaluate the association between pretreatment MRI descriptors and breast cancer (BC) pathological complete response (pCR) to neoadjuvant chemotherapy (NAC). Materials \& Methods: Patients with BC treated by NAC with a breast MRI between 2016 and 2020 were included in this retrospective observational single-center study. MR studies were described using the standardized BI-RADS and breast edema score on T2-weighted MRI. Univariable and multivariable logistic regression analyses were performed to assess variables association with pCR according to residual cancer burden. Random forest classifiers were trained to predict pCR on a random split including 70% of the database and were validated on the remaining cases. Results: Among 129 BC, 59 (46%) achieved pCR after NAC (luminal (n=7/37, 19%), triple negative (TN) (n=30/55, 55%), HER2+ (n=22/37, 59%). Clinical and biological items associated with pCR were BC subtype (p<0.001), T stage 0/I/II (p=0.008), higher Ki67 (p=0.005) and higher tumor-infiltrating lymphocytes levels (p=0.016). Univariate analysis showed that the following MRI features, oval or round shape (p=0.047), unifocality (p=0.026), non-spiculated margins (p=0.018), no associated non-mass enhancement (NME) (p = 0.024) and a lower MRI size (p = 0.031) were significantly associated with pCR. Unifocality and non-spiculated margins remained independently associated with pCR at multivariable analysis. Adding significant MRI features to clinicobiological variables in random forest classifiers significantly increased sensitivity (0.67 versus 0.62), specificity (0.69 versus 0.67) and precision (0.71 versus 0.67) for pCR prediction. Conclusion: Non-spiculated margins and unifocality are independently associated with pCR and can increase models performance to predict BC response to NAC. Clinical Relevance Statement: A multimodal approach integrating pretreatment MRI features with clinicobiological predictors, including TILs, could be employed to develop machine learning models for identifying patients at risk of non-response. This may enable consideration of alternative therapeutic strategies to optimize treatment outcomes
Authors Caroline Malhaire, Fatine Selhane, Marie-Judith Saint-Martin, Vincent Cockenpot, Pia Akl, Enora Laas, Audrey Bellesoeur, Catherine Ala Eddine, Melodie Bereby-Kahane, Julie Manceau, Delphine Sebbag-Sfez, Jean-Yves Pierga, Fabien Reyal, Anne Vincent-Salomon, Herve Brisse, Frederique Frouin
Categories Computer Vision and Pattern Recognition
0
UI-Styler: Ultrasound Image Style Transfer with Class-Aware Prompts for Cross-Device Diagnosis Using a Frozen Black-Box Inference Network WACV 2026
Title CN UI-Styler：基于类别感知提示的超声图像风格迁移用于冻结黑盒推理网络下的跨设备诊断
Keywords 超声图像 · 风格迁移 · 域适应 · 伪标签 · 医学图像
Summary 本文提出UI-Styler，一种针对超声图像的类别感知风格迁移框架，通过模式匹配机制将目标域纹理迁移到源图像，同时保留结构内容，并利用伪标签指导的类别感知提示增强语义对齐，提升跨设备诊断中分类与分割模型的性能。
Reason 该论文研究的是超声医学图像在不同设备间的域适应问题，属于医学图像处理领域，与文档图像理解（DIU）任务无直接关联。其核心方法为风格迁移和伪标签引导的语义对齐，主要解决医学影像的分布偏移问题，无法迁移到文档图像中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract The appearance of ultrasound images varies across acquisition devices, causing domain shifts that degrade the performance of fixed black-box downstream inference models when reused. To mitigate this issue, it is practical to develop unpaired image translation (UIT) methods that effectively align the statistical distributions between source and target domains, particularly under the constraint of a reused inference-blackbox setting. However, existing UIT approaches often overlook class-specific semantic alignment during domain adaptation, resulting in misaligned content-class mappings that can impair diagnostic accuracy. To address this limitation, we propose UI-Styler, a novel ultrasound-specific, class-aware image style transfer framework. UI-Styler leverages a pattern-matching mechanism to transfer texture patterns embedded in the target images onto source images while preserving the source structural content. In addition, we introduce a class-aware prompting strategy guided by pseudo labels of the target domain, which enforces accurate semantic alignment with diagnostic categories. Extensive experiments on ultrasound cross-device tasks demonstrate that UI-Styler consistently outperforms existing UIT methods, achieving state-of-the-art performance in distribution distance and downstream tasks, such as classification and segmentation.
Authors Nhat-Tuong Do-Tran, Ngoc-Hoang-Lam Le, Ching-Chun Huang
Comment Project page: https://dotrannhattuong.github.io/UIStyler, Accepted to WACV 2026
Categories Computer Vision and Pattern Recognition
0
DiffRefiner: Coarse to Fine Trajectory Planning via Diffusion Refinement with Semantic Interaction for End to End Autonomous Driving AAAI 2026
Title CN DiffRefiner：通过具有语义交互的扩散优化实现端到端自动驾驶的粗到精轨迹规划
Keywords 扩散模型 · 轨迹预测 · 自动驾驶 · 生成模型 · 语义对齐
Summary 本文提出DiffRefiner，一种用于自动驾驶的两阶段轨迹预测框架。第一阶段通过Transformer生成粗略轨迹，第二阶段利用扩散模型进行迭代去噪和精细化，提升轨迹与环境的语义一致性。在NAVSIM v2和Bench2Drive基准上达到SOTA性能。
Reason 该论文属于自动驾驶领域，研究任务为车辆轨迹预测与规划，使用扩散模型优化生成轨迹。尽管其方法涉及视觉输入与序列生成，但核心问题与文档图像理解（DIU）无关，不解决OCR幻觉、文本识别、布局分析或文档推理等任何DIU核心痛点。无法迁移其方法至文档理解场景，属于明确的无关领域。
Abstract Unlike discriminative approaches in autonomous driving that predict a fixed set of candidate trajectories of the ego vehicle, generative methods, such as diffusion models, learn the underlying distribution of future motion, enabling more flexible trajectory prediction. However, since these methods typically rely on denoising human-crafted trajectory anchors or random noise, there remains significant room for improvement. In this paper, we propose DiffRefiner, a novel two-stage trajectory prediction framework. The first stage uses a transformer-based Proposal Decoder to generate coarse trajectory predictions by regressing from sensor inputs using predefined trajectory anchors. The second stage applies a Diffusion Refiner that iteratively denoises and refines these initial predictions. In this way, we enhance the performance of diffusion-based planning by incorporating a discriminative trajectory proposal module, which provides strong guidance for the generative refinement process. Furthermore, we design a fine-grained denoising decoder to enhance scene compliance, enabling more accurate trajectory prediction through enhanced alignment with the surrounding environment. Experimental results demonstrate that DiffRefiner achieves state-of-the-art performance, attaining 87.4 EPDMS on NAVSIM v2, and 87.1 DS along with 71.4 SR on Bench2Drive, thereby setting new records on both public benchmarks. The effectiveness of each component is validated via ablation studies as well.
Authors Liuhan Yin, Runkun Ju, Guodong Guo, Erkang Cheng
Comment Accepted to AAAI 2026
Categories Computer Vision and Pattern Recognition
0
Learning to Look Closer: A New Instance-Wise Loss for Small Cerebral Lesion Segmentation
Title CN 学习更仔细地观察：一种用于小脑病变分割的新实例级损失函数
Keywords 医学图像分割 · 小目标检测 · 实例级损失 · Dice loss · nnU-Net
Summary 本文提出了一种新的实例级损失函数CC-DiceCE，用于提升小脑病变在医学图像中的检测能力，尤其改善了传统损失函数对小病灶敏感度不足的问题。实验基于nnU-Net框架，在多个数据集上验证了该方法在召回率上的优势，但以轻微增加假阳性为代价。整体仍属于专用医学图像分析的技术改进。
Reason 该论文属于医学图像分割领域，研究的是小脑病变的分割问题，使用的是传统医学图像数据与nnU-Net框架。尽管其提出的CC-DiceCE损失函数在处理小目标上有改进，但其方法和应用场景与文档图像理解（DIU）或DocVQA等任务无直接关联。未涉及OCR、多模态语言模型、视觉表示学习、布局分析或表格识别等DIU核心问题，也无法迁移用于解决文档中的幻觉、复杂排版或长文档推理等痛点。
Abstract Traditional loss functions in medical image segmentation, such as Dice, often under-segment small lesions because their small relative volume contributes negligibly to the overall loss. To address this, instance-wise loss functions and metrics have been proposed to evaluate segmentation quality on a per-lesion basis. We introduce CC-DiceCE, a loss function based on the CC-Metrics framework, and compare it with the existing blob loss. Both are benchmarked against a DiceCE baseline within the nnU-Net framework, which provides a robust and standardized setup. We find that CC-DiceCE loss increases detection (recall) with minimal to no degradation in segmentation performance, albeit at the cost of slightly more false positives. Furthermore, our multi-dataset study shows that CC-DiceCE generally outperforms blob loss.
Authors Luc Bouteille, Alexander Jaus, Jens Kleesiek, Rainer Stiefelhagen, Lukas Heine
Comment 5 pages, 2 figures, 2 tables
Categories Computer Vision and Pattern Recognition
0
POMA-3D: The Point Map Way to 3D Scene Understanding
Title CN POMA-3D：通过点图进行3D场景理解的方法
Keywords 3D表示学习 · 自监督学习 · 点图 · 多视角一致性 · 具身智能
Summary 本文提出了POMA-3D，首个从点图中学习的自监督3D表示模型。点图在结构化2D网格上编码显式3D坐标，保留全局几何信息并兼容2D基础模型输入格式。通过设计视图到场景对齐策略和POMA-JEPA联合嵌入-预测架构，实现跨多视角的几何一致性特征学习。作者还构建了大规模数据集ScenePoint用于预训练。实验表明POMA-3D可作为多种3D理解任务的强大骨干网络。
Reason 该论文研究的是3D场景理解，属于3D视觉领域，使用点图（point maps）进行自监督学习，并应用于3D问答、具身导航等任务。其核心问题与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、密集文本识别、复杂排版或长文档推理等DIU核心痛点。
Abstract In this paper, we introduce POMA-3D, the first self-supervised 3D representation model learned from point maps. Point maps encode explicit 3D coordinates on a structured 2D grid, preserving global 3D geometry while remaining compatible with the input format of 2D foundation models. To transfer rich 2D priors into POMA-3D, a view-to-scene alignment strategy is designed. Moreover, as point maps are view-dependent with respect to a canonical space, we introduce POMA-JEPA, a joint embedding-predictive architecture that enforces geometrically consistent point map features across multiple views. Additionally, we introduce ScenePoint, a point map dataset constructed from 6.5K room-level RGB-D scenes and 1M 2D image scenes to facilitate large-scale POMA-3D pretraining. Experiments show that POMA-3D serves as a strong backbone for both specialist and generalist 3D understanding. It benefits diverse tasks, including 3D question answering, embodied navigation, scene retrieval, and embodied localization, all achieved using only geometric inputs (i.e., 3D coordinates). Overall, our POMA-3D explores a point map way to 3D scene understanding, addressing the scarcity of pretrained priors and limited data in 3D representation learning. Project Page: https://matchlab-imperial.github.io/poma3d/
Authors Ye Mao, Weixun Luo, Ranran Huang, Junpeng Jing, Krystian Mikolajczyk
Comment 11 pages, 6 tables, 5 figures
Categories Computer Vision and Pattern Recognition
0
Off the Planckian Locus: Using 2D Chromaticity to Improve In-Camera Color
Title CN 脱离普朗克轨迹：利用二维色度改善相机内色彩
Keywords 色彩校正 · 白平衡 · 二维色度 · 非普朗克光源 · 相机内处理
Summary 传统相机色彩映射依赖基于相关色温（CCT）的一维插值，但在现代LED等偏离普朗克轨迹的光源下表现不佳。本文提出使用二维色度空间替代一维CCT，并采用轻量级多层感知机（MLP）进行色彩映射，在包含多种LED光源的场景中平均降低22%的角度再现误差，且支持实时部署。方法通过带光源标定箱的流程训练，兼容传统光源并适用于多光源场景。
Reason 该论文研究的是相机内色彩校正问题，聚焦于在非普朗克光源（如LED）下通过二维色度空间和轻量级MLP提升色彩还原精度。其核心问题是计算摄影中的白平衡优化，属于图像增强范畴，与文档图像理解（DIU）中的文本识别、布局分析、视觉问答等任务无直接关联。文中未涉及OCR、多模态语言模型、文档结构或语义推理等内容，无法迁移用于解决OCR幻觉、复杂排版或长文档理解等DIU核心痛点。
Abstract Traditional in-camera colorimetric mapping relies on correlated color temperature (CCT)-based interpolation between pre-calibrated transforms optimized for Planckian illuminants such as CIE A and D65. However, modern lighting technologies such as LEDs can deviate substantially from the Planckian locus, exposing the limitations of relying on conventional one-dimensional CCT for illumination characterization. This paper demonstrates that transitioning from 1D CCT (on the Planckian locus) to a 2D chromaticity space (off the Planckian locus) improves colorimetric accuracy across various mapping approaches. In addition, we replace conventional CCT interpolation with a lightweight multi-layer perceptron (MLP) that leverages 2D chromaticity features for robust colorimetric mapping under non-Planckian illuminants. A lightbox-based calibration procedure incorporating representative LED sources is used to train our MLP. Validated across diverse LED lighting, our method reduces angular reproduction error by 22% on average in LED-lit scenes, maintains backward compatibility with traditional illuminants, accommodates multi-illuminant scenes, and supports real-time in-camera deployment with negligible additional computational cost.
Authors SaiKiran Tedla, Joshua E. Little, Hakki Can Karaimer, Michael S. Brown
Comment Project page: https://cst-mlp.github.io
Categories Computer Vision and Pattern Recognition
0
PEGS: Physics-Event Enhanced Large Spatiotemporal Motion Reconstruction via 3D Gaussian Splatting
Title CN PEGS：通过3D高斯点阵化的物理事件增强大型时空运动重建
Keywords 3D Gaussian Splatting · Event Camera · Motion Reconstruction · Physical Priors · Dynamic Scenes
Summary 本文提出PEGS框架，结合物理先验与事件流增强，在3D高斯点阵化流程中实现去模糊的目标聚焦建模与运动恢复。引入了三层监督机制以保证物理合理性，并设计了一种运动感知的模拟退火策略来自适应调度训练过程。同时发布了首个面向自然快速刚体运动的RGB-Event配对数据集。
Reason 该论文研究的是基于事件相机的刚体运动重建问题，属于3D视觉与动态场景重建领域，核心方法涉及物理先验建模、事件流增强和3D高斯点阵化。其任务目标与文档图像理解（DIU）完全无关，无法迁移解决OCR幻觉、复杂排版、长文档推理等核心痛点。
Abstract Reconstruction of rigid motion over large spatiotemporal scales remains a challenging task due to limitations in modeling paradigms, severe motion blur, and insufficient physical consistency. In this work, we propose PEGS, a framework that integrates Physical priors with Event stream enhancement within a 3D Gaussian Splatting pipeline to perform deblurred target-focused modeling and motion recovery. We introduce a cohesive triple-level supervision scheme that enforces physical plausibility via an acceleration constraint, leverages event streams for high-temporal resolution guidance, and employs a Kalman regularizer to fuse multi-source observations. Furthermore, we design a motion-aware simulated annealing strategy that adaptively schedules the training process based on real-time kinematic states. We also contribute the first RGB-Event paired dataset targeting natural, fast rigid motion across diverse scenarios. Experiments show PEGS's superior performance in reconstructing motion over large spatiotemporal scales compared to mainstream dynamic methods.
Authors Yijun Xu, Jingrui Zhang, Hongyi Liu, Yuhan Chen, Yuanyang Wang, Qingyao Guo, Dingwen Wang, Lei Yu, Chu He
Categories Computer Vision and Pattern Recognition
0
RapidPoseTriangulation: Multi-view Multi-person Whole-body Human Pose Triangulation in a Millisecond
Title CN RapidPose三角测量：毫秒级多视角多人全身人体姿态三角测量
Keywords 多视角姿态估计 · 3D人体姿态 · 三角测量 · 计算机视觉 · 实时处理
Summary 本文提出了一种高效的多视角多人全身姿态三角测量算法，能够在毫秒级完成从多视角图像中对人体从面部到手指的全身关键点进行3D重建，并展现出良好的跨数据集泛化能力。所有代码已公开。
Reason 该论文属于人体姿态估计领域，研究的是多视角下的人体关键点重建问题，与文档图像理解（DIU）任务无直接关联。其技术聚焦于3D姿态重建和高效三角化算法，无法迁移解决文档理解中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract The integration of multi-view imaging and pose estimation represents a significant advance in computer vision applications, offering new possibilities for understanding human movement and interactions. This work presents a new algorithm that improves multi-view multi-person pose estimation, focusing on fast triangulation speeds and good generalization capabilities. The approach extends to whole-body pose estimation, capturing details from facial expressions to finger movements across multiple individuals and viewpoints. Adaptability to different settings is demonstrated through strong performance across unseen datasets and configurations. To support further progress in this field, all of this work is publicly accessible.
Authors Daniel Bermuth, Alexander Poeppel, Wolfgang Reif
Categories Computer Vision and Pattern Recognition
0
Bridging Visual Affective Gap: Borrowing Textual Knowledge by Learning from Noisy Image-Text Pairs ACM MM 2024
Title CN 弥合视觉情感鸿沟：通过学习噪声图文对从文本中借用知识
Keywords 视觉情感识别 · 图文对比学习 · 噪声数据学习 · 多模态学习 · 情感鸿沟
Summary 本文提出通过从预训练的文本模型中借用知识来提升视觉情感识别性能，提出了Partitioned Adaptive Contrastive Learning（PACL）方法，利用噪声社交媒体图文对中的事实和情感关联进行学习。实验表明该方法能有效桥接‘情感鸿沟’，提升多种视觉模型在情感相关下游任务上的表现。
Reason 该论文研究的是视觉情感识别（VER）任务，核心目标是弥合图像特征与情感类别之间的‘情感鸿沟’，并利用文本模态中的显式情感信息来增强视觉模型的情感感知能力。其方法基于社交媒体中的噪声图文对，设计了分区自适应对比学习（PACL）。尽管使用了多模态（图像-文本）数据，但其任务本质属于情感分析范畴，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别、OCR-free理解等无直接关联。所提方法针对的是情感语义的迁移，而非解决文档理解中的OCR幻觉、密集文本、复杂排版或长文档推理等问题，无法迁移到DIU领域。
Abstract Visual emotion recognition (VER) is a longstanding field that has garnered increasing attention with the advancement of deep neural networks. Although recent studies have achieved notable improvements by leveraging the knowledge embedded within pre-trained visual models, the lack of direct association between factual-level features and emotional categories, called the "affective gap", limits the applicability of pre-training knowledge for VER tasks. On the contrary, the explicit emotional expression and high information density in textual modality eliminate the "affective gap". Therefore, we propose borrowing the knowledge from the pre-trained textual model to enhance the emotional perception of pre-trained visual models. We focus on the factual and emotional connections between images and texts in noisy social media data, and propose Partitioned Adaptive Contrastive Learning (PACL) to leverage these connections. Specifically, we manage to separate different types of samples and devise distinct contrastive learning strategies for each type. By dynamically constructing negative and positive pairs, we fully exploit the potential of noisy samples. Through comprehensive experiments, we demonstrate that bridging the "affective gap" significantly improves the performance of various pre-trained visual models in downstream emotion-related tasks. Our code is released on https://github.com/wdqqdw/PACL.
Authors Daiqing Wu, Dongbao Yang, Yu Zhou, Can Ma
Comment Accepted by ACM MM 2024
Categories Computer Vision and Pattern Recognition
0
REArtGS: Reconstructing and Generating Articulated Objects via 3D Gaussian Splatting with Geometric and Motion Constraints
Title CN 通过具有几何和运动约束的3D高斯点阵重构与生成可动部件物体
Keywords 3D重建 · 可动部件物体 · 高斯点阵 · 几何约束 · 运动建模
Summary 本文提出REArtGS，一种结合几何与运动约束的3D高斯点阵框架，用于可动部件物体的高质量表面重建与新状态下的生成。通过引入符号距离场（SDF）指导优化高斯不透明度，并建立基于运动学结构的可变形场，实现无需监督的表面网格生成。实验表明其在合成与真实数据上均能实现高保真重建与生成。
Reason 该论文研究的是3D视觉中的可动部件物体重建与生成，属于3D高斯点阵（Gaussian Splatting）在三维重建领域的应用。其核心问题为3D形状建模与动态结构生成，与文档图像理解（DIU）任务如DocVQA、布局分析、表格识别等无直接关联。方法聚焦于几何约束和运动建模，难以迁移到文档场景中的OCR幻觉、复杂排版或长文档推理等痛点。属于无关领域（3D视觉/生成），不符合关注方向。
Abstract Articulated objects, as prevalent entities in human life, their 3D representations play crucial roles across various applications. However, achieving both high-fidelity textured surface reconstruction and dynamic generation for articulated objects remains challenging for existing methods. In this paper, we present REArtGS, a novel framework that introduces additional geometric and motion constraints to 3D Gaussian primitives, enabling realistic surface reconstruction and generation for articulated objects. Specifically, given multi-view RGB images of arbitrary two states of articulated objects, we first introduce an unbiased Signed Distance Field (SDF) guidance to regularize Gaussian opacity fields, enhancing geometry constraints and improving surface reconstruction quality. Then we establish deformable fields for 3D Gaussians constrained by the kinematic structures of articulated objects, achieving unsupervised generation of surface meshes in unseen states. Extensive experiments on both synthetic and real datasets demonstrate our approach achieves high-quality textured surface reconstruction for given states, and enables high-fidelity surface generation for unseen states. Project site: https://sites.google.com/view/reartgs/home.
Authors Di Wu, Liu Liu, Zhou Linli, Anran Huang, Liangtu Song, Qiaojun Yu, Qi Wu, Cewu Lu
Comment 11pages, 6 figures
Categories Computer Vision and Pattern Recognition
0
Sparse Reasoning is Enough: Biological-Inspired Framework for Video Anomaly Detection with Large Pre-trained Models
Title CN 稀疏推理已足够：基于大预训练模型的生物启发式视频异常检测框架
Keywords 视频异常检测 · 稀疏推理 · 大模型 · 视觉语言模型 · 记忆机制
Summary 本文提出ReCoVAD，一种受人类神经系统双通路机制启发的视频异常检测框架，通过‘反射通路’和‘意识通路’实现稀疏推理，在仅处理少量视频帧的情况下达到SOTA性能。该方法依赖CLIP和中等规模VLM进行事件描述生成，并利用LLM进行周期性回顾以发现新异常。
Reason 该论文研究的是视频异常检测（VAD），属于视频理解领域，且其方法聚焦于帧级稀疏推理以降低计算成本。尽管使用了视觉语言模型和记忆机制，但任务本质与文档图像理解（DIU）的核心痛点（如OCR幻觉、密集文本识别、复杂版面分析）无关。无法迁移其方法直接解决文档理解中的细粒度定位或长文档推理问题。
Abstract Video anomaly detection (VAD) plays a vital role in real-world applications such as security surveillance, autonomous driving, and industrial monitoring. Recent advances in large pre-trained models have opened new opportunities for training-free VAD by leveraging rich prior knowledge and general reasoning capabilities. However, existing studies typically rely on dense frame-level inference, incurring high computational costs and latency. This raises a fundamental question: Is dense reasoning truly necessary when using powerful pre-trained models in VAD systems? To answer this, we propose ReCoVAD, a novel framework inspired by the dual reflex and conscious pathways of the human nervous system, enabling selective frame processing to reduce redundant computation. ReCoVAD consists of two core pathways: (i) a Reflex pathway that uses a lightweight CLIP-based module to fuse visual features with prototype prompts and produce decision vectors, which query a dynamic memory of past frames and anomaly scores for fast response; and (ii) a Conscious pathway that employs a medium-scale vision-language model to generate textual event descriptions and refined anomaly scores for novel frames. It continuously updates the memory and prototype prompts, while an integrated large language model periodically reviews accumulated descriptions to identify unseen anomalies, correct errors, and refine prototypes. Extensive experiments show that ReCoVAD achieves state-of-the-art training-free performance while processing only 28.55\% and 16.04\% of the frames used by previous methods on the UCF-Crime and XD-Violence datasets, demonstrating that sparse reasoning is sufficient for effective large-model-based VAD.
Authors He Huang, Zixuan Hu, Dongxiao Li, Yao Xiao, Ling-Yu Duan
Categories Computer Vision and Pattern Recognition
0
SF-Recon: Simplification-Free Lightweight Building Reconstruction via 3D Gaussian Splatting 2026 ISPRS Congress
Title CN SF-Recon：基于3D高斯溅射的无简化轻量化建筑重建
Keywords 3D高斯溅射 · 轻量化重建 · 建筑建模 · 多视角几何 · 网格生成
Summary 本文提出SF-Recon，一种无需后处理简化的轻量化建筑表面重建方法，利用3D高斯溅射从多视角图像中直接生成结构保真的建筑网格。通过法向梯度引导的高斯优化和多视角边缘一致性剪枝，增强建筑结构清晰度，并采用多视角深度约束的Delaunay三角剖分生成轻量网格。实验表明该方法能在保持效率的同时减少面片和顶点数量。
Reason 该论文研究的是3D视觉中的建筑表面重建，属于3D高斯溅射在数字城市和地理空间分析中的应用，与文档图像理解（DIU）或视觉语言模型（VLM）无关。其核心技术如3DGS、法向梯度优化和深度约束三角剖分均不直接适用于解决文档中的OCR幻觉、复杂排版或长文档推理等核心问题。
Abstract Lightweight building surface models are crucial for digital city, navigation, and fast geospatial analytics, yet conventional multi-view geometry pipelines remain cumbersome and quality-sensitive due to their reliance on dense reconstruction, meshing, and subsequent simplification. This work presents SF-Recon, a method that directly reconstructs lightweight building surfaces from multi-view images without post-hoc mesh simplification. We first train an initial 3D Gaussian Splatting (3DGS) field to obtain a view-consistent representation. Building structure is then distilled by a normal-gradient-guided Gaussian optimization that selects primitives aligned with roof and wall boundaries, followed by multi-view edge-consistency pruning to enhance structural sharpness and suppress non-structural artifacts without external supervision. Finally, a multi-view depth-constrained Delaunay triangulation converts the structured Gaussian field into a lightweight, structurally faithful building mesh. Based on a proposed SF dataset, the experimental results demonstrate that our SF-Recon can directly reconstruct lightweight building models from multi-view imagery, achieving substantially fewer faces and vertices while maintaining computational efficiency. Website:https://lzh282140127-cell.github.io/SF-Recon-project/
Authors Zihan Li, Tengfei Wang, Wentian Gan, Hao Zhan, Xin Wang, Zongqian Zhan
Comment This paper has been submitted to the 2026 ISPRS Congress
Categories Computer Vision and Pattern Recognition
0
Text2Traffic: A Text-to-Image Generation and Editing Method for Traffic Scenes
Title CN Text2Traffic：一种用于交通场景的文本到图像生成与编辑方法
Keywords 文本到图像生成 · 交通场景 · 多视角数据 · 掩码机制 · 细粒度对齐
Summary 本文提出了一种面向交通场景的文本到图像生成与编辑框架，通过可控掩码机制统一生成与编辑任务，结合车端与路侧多视角数据增强场景几何多样性，并采用两阶段训练策略提升文本-图像对齐与细节质量。此外，引入掩码区域加权损失以增强对小型关键交通元素的生成保真度。实验表明该方法在交通场景文本驱动生成任务上表现优异。
Reason 该论文属于纯图像生成领域，研究的是基于文本的交通场景图像生成与编辑，其技术目标和应用场景与文档图像理解（DIU）无直接关联。尽管使用了多视角数据和细粒度训练策略，但核心是视觉内容合成而非文档结构解析、文本识别或VQA等DIU任务。且其方法聚焦于交通元素的几何多样性和小区域生成质量，难以迁移到OCR幻觉、复杂排版理解或长文档推理等文档理解痛点。
Abstract With the rapid advancement of intelligent transportation systems, text-driven image generation and editing techniques have demonstrated significant potential in providing rich, controllable visual scene data for applications such as traffic monitoring and autonomous driving. However, several challenges remain, including insufficient semantic richness of generated traffic elements, limited camera viewpoints, low visual fidelity of synthesized images, and poor alignment between textual descriptions and generated content. To address these issues, we propose a unified text-driven framework for both image generation and editing, leveraging a controllable mask mechanism to seamlessly integrate the two tasks. Furthermore, we incorporate both vehicle-side and roadside multi-view data to enhance the geometric diversity of traffic scenes. Our training strategy follows a two-stage paradigm: first, we perform conceptual learning using large-scale coarse-grained text-image data; then, we fine-tune with fine-grained descriptive data to enhance text-image alignment and detail quality. Additionally, we introduce a mask-region-weighted loss that dynamically emphasizes small yet critical regions during training, thereby substantially enhancing the generation fidelity of small-scale traffic elements. Extensive experiments demonstrate that our method achieves leading performance in text-based image generation and editing within traffic scenes.
Authors Feng Lv, Haoxuan Feng, Zilu Zhang, Chunlong Xia, Yanfeng Li
Categories Computer Vision and Pattern Recognition
0
Model-Agnostic Gender Bias Control for Text-to-Image Generation via Sparse Autoencoder
Title CN 通过稀疏自编码器实现文本到图像生成的模型无关性别偏见控制
Keywords 文本到图像生成 · 性别偏见 · 稀疏自编码器 · 去偏框架 · 模型无关
Summary 本文提出SAE Debias，一种基于k-稀疏自编码器的轻量级、模型无关框架，用于缓解文本到图像扩散模型中的性别偏见。该方法在无需微调或修改模型结构的前提下，通过在稀疏潜在空间中识别并抑制与职业相关的性别偏向方向，实现去偏生成。实验表明其在多个Stable Diffusion系列模型上均能有效减少性别刻板印象，同时保持图像质量。
Reason 该论文研究的是文本到图像生成中的性别偏见问题，属于纯图像生成领域，并且聚焦于社会公平性与生成内容的可控性。虽然其方法涉及特征空间干预和可解释性，但核心任务与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点。
Abstract Text-to-image (T2I) diffusion models often exhibit gender bias, particularly by generating stereotypical associations between professions and gendered subjects. This paper presents SAE Debias, a lightweight and model-agnostic framework for mitigating such bias in T2I generation. Unlike prior approaches that rely on CLIP-based filtering or prompt engineering, which often require model-specific adjustments and offer limited control, SAE Debias operates directly within the feature space without retraining or architectural modifications. By leveraging a k-sparse autoencoder pre-trained on a gender bias dataset, the method identifies gender-relevant directions within the sparse latent space, capturing professional stereotypes. Specifically, a biased direction per profession is constructed from sparse latents and suppressed during inference to steer generations toward more gender-balanced outputs. Trained only once, the sparse autoencoder provides a reusable debiasing direction, offering effective control and interpretable insight into biased subspaces. Extensive evaluations across multiple T2I models, including Stable Diffusion 1.4, 1.5, 2.1, and SDXL, demonstrate that SAE Debias substantially reduces gender bias while preserving generation quality. To the best of our knowledge, this is the first work to apply sparse autoencoders for identifying and intervening in gender bias within T2I models. These findings contribute toward building socially responsible generative AI, providing an interpretable and model-agnostic tool to support fairness in text-to-image generation.
Authors Chao Wu, Zhenyi Wang, Kangxian Xie, Naresh Kumar Devulapally, Vishnu Suresh Lokhande, Mingchen Gao
Categories Computer Vision and Pattern Recognition
0
RL-AD-Net: Reinforcement Learning Guided Adaptive Displacement in Latent Space for Refined Point Cloud Completion
Title CN RL-AD-Net：基于强化学习引导的潜在空间自适应位移用于精细化点云补全
Keywords 点云补全 · 强化学习 · 潜在空间优化 · 3D重建 · 几何一致性
Summary 本文提出RL-AD-Net，一种在预训练点云自编码器的潜在空间中通过强化学习进行点云补全后处理的方法。该方法通过RL代理调整全局特征向量，并结合非参数选择器保留几何一致性更好的结果，在ShapeNetCore-2048数据集上验证了其在不同裁剪场景下的有效性。模型轻量、模块化且不依赖特定网络结构。
Reason 该论文研究的是3D点云补全任务，属于3D视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管其使用了强化学习和潜在空间优化等技术，但应用场景为三维形状重建，迁移到文档图像中的OCR幻觉、复杂排版或长文档推理等问题极为困难，且未涉及多模态语言模型、视觉压缩或文档布局分析等DIU核心议题。
Abstract Recent point cloud completion models, including transformer-based, denoising-based, and other state-of-the-art approaches, generate globally plausible shapes from partial inputs but often leave local geometric inconsistencies. We propose RL-AD-Net, a reinforcement learning (RL) refinement framework that operates in the latent space of a pretrained point autoencoder. The autoencoder encodes completions into compact global feature vectors (GFVs), which are selectively adjusted by an RL agent to improve geometric fidelity. To ensure robustness, a lightweight non-parametric PointNN selector evaluates the geometric consistency of both the original completion and the RL-refined output, retaining the better reconstruction. When ground truth is available, both Chamfer Distance and geometric consistency metrics guide refinement. Training is performed separately per category, since the unsupervised and dynamic nature of RL makes convergence across highly diverse categories challenging. Nevertheless, the framework can be extended to multi-category refinement in future work. Experiments on ShapeNetCore-2048 demonstrate that while baseline completion networks perform reasonable under their training-style cropping, they struggle in random cropping scenarios. In contrast, RL-AD-Net consistently delivers improvements across both settings, highlighting the effectiveness of RL-guided ensemble refinement. The approach is lightweight, modular, and model-agnostic, making it applicable to a wide range of completion networks without requiring retraining.
Authors Bhanu Pratap Paregi, Vaibhav Kumar
Categories Computer Vision and Pattern Recognition
0
Open-Set Domain Generalization through Spectral-Spatial Uncertainty Disentanglement for Hyperspectral Image Classification
Title CN 通过谱-空间不确定性解耦实现高光谱图像分类的开放集域泛化
Keywords 高光谱图像分类 · 开放集域泛化 · 不确定性解耦 · 证据深度学习 · 频域特征提取
Summary 本文提出了一种用于高光谱图像分类的开放集域泛化框架，通过谱-空间不确定性解耦机制，结合证据深度学习来量化不同特征路径的可靠性，并自适应选择最优路径。同时引入频率域特征提取以增强域不变表示能力。实验表明该方法在无目标域数据的情况下仍能保持良好的已知类识别和未知类拒绝能力。
Reason 该论文研究的是高光谱图像分类中的开放集域泛化问题，属于遥感或地理信息领域，与文档图像理解（DIU）任务无直接关联。其核心技术如频域特征提取、双通道残差网络和证据深度学习主要用于处理高光谱数据的谱-空间特征，无法迁移解决文档图像中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Open-set domain generalization (OSDG) tackles the dual challenge of recognizing unknown classes while simultaneously striving to generalize across unseen domains without using target data during training. In this article, an OSDG framework for hyperspectral image classification is proposed, centered on a new Spectral-Spatial Uncertainty Disentanglement mechanism. It has been designed to address the domain shift influencing both spectral, spatial and combined feature extraction pathways using evidential deep learning, after which the most reliable pathway for each sample is adaptively selected. The proposed framework is further integrated with frequency-domain feature extraction for domain-invariant representation learning, dual-channel residual networks for spectral-spatial feature extraction, and evidential deep learning based uncertainty quantification. Experiments conducted on three cross scene hyperspectral datasets, show that performance comparable to state-of-the-art domain adaptation methods can be achieved despite no access to target data, while high unknown-class rejection and known-class accuracy levels are maintained. The implementation will be available at github.com/amir-khb/UGOSDG upon acceptance.
Authors Amirreza Khoshbakht, Erchan Aptoula
Categories Computer Vision and Pattern Recognition
0
RoomPlanner: Explicit Layout Planner for Easier LLM-Driven 3D Room Generation
Title CN RoomPlanner：用于简化LLM驱动的3D房间生成的显式布局规划器
Keywords 3D生成 · 文本到场景 · 布局规划 · 语言驱动代理 · 3D高斯表示
Summary 本文提出RoomPlanner，一种完全自动化的3D房间生成框架，仅需简短文本输入即可生成合理的室内场景布局。通过分层的语言驱动代理将模糊提示转化为详细场景描述，并结合空间约束优化物体摆放，最终采用新型采样策略加速3D高斯场景渲染，实现快速且可编辑的高质量结果。
Reason 该论文聚焦于3D室内场景生成，属于具身智能与图形学交叉领域，核心任务为从文本生成带布局的3D房间模型。其方法涉及语言驱动的代理规划、3D高斯表示优化和相机路径采样，与文档图像理解（DIU）中的OCR、排版分析、表格识别或DocVQA等任务无直接关联。所提技术难以迁移至文档理解场景，尤其是无法解决OCR幻觉、密集文本解析或复杂逻辑推理等核心痛点。
Abstract In this paper, we propose RoomPlanner, the first fully automatic 3D room generation framework for painlessly creating realistic indoor scenes with only short text as input. Without any manual layout design or panoramic image guidance, our framework can generate explicit layout criteria for rational spatial placement. We begin by introducing a hierarchical structure of language-driven agent planners that can automatically parse short and ambiguous prompts into detailed scene descriptions. These descriptions include raw spatial and semantic attributes for each object and the background, which are then used to initialize 3D point clouds. To position objects within bounded environments, we implement two arrangement constraints that iteratively optimize spatial arrangements, ensuring a collision-free and accessible layout solution. In the final rendering stage, we propose a novel AnyReach Sampling strategy for camera trajectory, along with the Interval Timestep Flow Sampling (ITFS) strategy, to efficiently optimize the coarse 3D Gaussian scene representation. These approaches help reduce the total generation time to under 30 minutes. Extensive experiments demonstrate that our method can produce geometrically rational 3D indoor scenes, surpassing prior approaches in both rendering speed and visual quality while preserving editability. The code will be available soon.
Authors Wenzhuo Sun, Mingjian Liang, Wenxuan Song, Xuelian Cheng, Zongyuan Ge
Categories Computer Vision and Pattern Recognition
0
BeyondFacial: Identity-Preserving Personalized Generation Beyond Facial Close-ups
Title CN BeyondFacial：超越面部特写的保留身份的个性化生成
Keywords 个性化生成 · 身份保持 · 图像生成 · 双路径推理 · 语义一致性
Summary 本文提出一种名为BeyondFacial的个性化生成方法，通过双路径推理结构（DLI）、自适应身份融合（IdAF）和身份聚合前置模块（IdAP），实现非面部特写场景下的身份保持生成，摆脱对人工掩码的依赖，并提升语义一致性。适用于影视级角色-场景生成。
Reason 该论文属于纯图像生成领域，研究的是个性化生成中如何在非面部特写场景下保持身份特征，核心问题为ID-semantic分离与融合，应用于艺术创作和影视生成。与文档图像理解（DIU）任务如DocVQA、OCR、版面分析等无直接关联，提出的方法难以迁移到解决文档中的OCR幻觉、复杂排版或长文档推理等痛点。
Abstract Identity-Preserving Personalized Generation (IPPG) has advanced film production and artistic creation, yet existing approaches overemphasize facial regions, resulting in outputs dominated by facial close-ups.These methods suffer from weak visual narrativity and poor semantic consistency under complex text prompts, with the core limitation rooted in identity (ID) feature embeddings undermining the semantic expressiveness of generative models. To address these issues, this paper presents an IPPG method that breaks the constraint of facial close-ups, achieving synergistic optimization of identity fidelity and scene semantic creation. Specifically, we design a Dual-Line Inference (DLI) pipeline with identity-semantic separation, resolving the representation conflict between ID and semantics inherent in traditional single-path architectures. Further, we propose an Identity Adaptive Fusion (IdAF) strategy that defers ID-semantic fusion to the noise prediction stage, integrating adaptive attention fusion and noise decision masking to avoid ID embedding interference on semantics without manual masking. Finally, an Identity Aggregation Prepending (IdAP) module is introduced to aggregate ID information and replace random initializations, further enhancing identity preservation. Experimental results validate that our method achieves stable and effective performance in IPPG tasks beyond facial close-ups, enabling efficient generation without manual masking or fine-tuning. As a plug-and-play component, it can be rapidly deployed in existing IPPG frameworks, addressing the over-reliance on facial close-ups, facilitating film-level character-scene creation, and providing richer personalized generation capabilities for related domains.
Authors Songsong Zhang, Chuanqi Tang, Hongguang Zhang, Guijian Tang, Minglong Li, Xueqiong Li, Shaowu Yang, Yuanxi Peng, Wenjing Yang, Jing Zhao
Comment 16 pages, 16 figures
Categories Computer Vision and Pattern Recognition
0
Mask2IV: Interaction-Centric Video Generation via Mask Trajectories AAAI 2026
Title CN 通过掩码轨迹实现以交互为中心的视频生成
Keywords 视频生成 · 掩码轨迹 · 交互建模 · 具身智能 · 运动控制
Summary 本文提出Mask2IV，一种用于生成以交互为中心的视频（如人-物或机器人-物互动）的新框架。该方法采用解耦的两阶段流程：先预测参与者和物体的运动轨迹，再基于轨迹条件生成视频，无需用户提供密集的掩码标注。支持通过动作描述或空间提示进行直观控制，并构建了两个涵盖多种动作和对象类别的基准数据集。实验表明该方法在视觉真实感和可控性上优于现有方法。
Reason 该论文属于视频生成与具身智能领域，研究目标是生成人或机器人与物体交互的视频，其技术路线聚焦于视频控制信号（如掩码轨迹）和动作描述驱动的生成框架。尽管使用了视觉掩码和轨迹预测，但其核心任务与文档图像理解（DIU）无关，无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。属于‘无关领域’中的视频生成方向，直接命中负面清单。
Abstract Generating interaction-centric videos, such as those depicting humans or robots interacting with objects, is crucial for embodied intelligence, as they provide rich and diverse visual priors for robot learning, manipulation policy training, and affordance reasoning. However, existing methods often struggle to model such complex and dynamic interactions. While recent studies show that masks can serve as effective control signals and enhance generation quality, obtaining dense and precise mask annotations remains a major challenge for real-world use. To overcome this limitation, we introduce Mask2IV, a novel framework specifically designed for interaction-centric video generation. It adopts a decoupled two-stage pipeline that first predicts plausible motion trajectories for both actor and object, then generates a video conditioned on these trajectories. This design eliminates the need for dense mask inputs from users while preserving the flexibility to manipulate the interaction process. Furthermore, Mask2IV supports versatile and intuitive control, allowing users to specify the target object of interaction and guide the motion trajectory through action descriptions or spatial position cues. To support systematic training and evaluation, we curate two benchmarks covering diverse action and object categories across both human-object interaction and robotic manipulation scenarios. Extensive experiments demonstrate that our method achieves superior visual realism and controllability compared to existing baselines.
Authors Gen Li, Bo Zhao, Jianfei Yang, Laura Sevilla-Lara
Comment AAAI 2026. Project page: https://reagan1311.github.io/mask2iv
Categories Computer Vision and Pattern Recognition
0
Energy Scaling Laws for Diffusion Models: Quantifying Compute and Carbon Emissions in Image Generation EurIPS 2025 workshop "Rethinking AI: Efficiency, Frugality, and Sustainability"
Title CN 扩散模型的能量缩放定律：量化图像生成中的计算与碳排放
Keywords 扩散模型 · 能量缩放律 · 计算效率 · 碳排放 · 图像生成
Summary 本文提出了一种基于Kaplan缩放律的能源消耗预测方法，用于估计扩散模型在图像生成过程中的GPU能耗和碳排放。通过将推理过程分解为文本编码、去噪和解码阶段，并在多种模型和GPU上进行实验，验证了计算复杂度（FLOPs）与实际能耗之间的强相关性。模型在单一架构内具有高预测精度（R² > 0.9），并展现出跨架构泛化能力。
Reason 该论文研究的是图像生成模型（如Stable Diffusion）在不同硬件上的能量消耗预测，属于纯图像生成领域，且聚焦于能效与环境影响分析。其方法论虽具系统性，但完全不涉及文档结构、文本理解或视觉-语言对齐等文档图像理解（DIU）核心问题。无法迁移用于解决OCR幻觉、复杂排版解析或长文档推理等DIU痛点。
Abstract The rapidly growing computational demands of diffusion models for image generation have raised significant concerns about energy consumption and environmental impact. While existing approaches to energy optimization focus on architectural improvements or hardware acceleration, there is a lack of principled methods to predict energy consumption across different model configurations and hardware setups. We propose an adaptation of Kaplan scaling laws to predict GPU energy consumption for diffusion models based on computational complexity (FLOPs). Our approach decomposes diffusion model inference into text encoding, iterative denoising, and decoding components, with the hypothesis that denoising operations dominate energy consumption due to their repeated execution across multiple inference steps. We conduct comprehensive experiments across four state-of-the-art diffusion models (Stable Diffusion 2, Stable Diffusion 3.5, Flux, and Qwen) on three GPU architectures (NVIDIA A100, A4000, A6000), spanning various inference configurations including resolution (256x256 to 1024x1024), precision (fp16/fp32), step counts (10-50), and classifier-free guidance settings. Our energy scaling law achieves high predictive accuracy within individual architectures (R-squared > 0.9) and exhibits strong cross-architecture generalization, maintaining high rank correlations across models and enabling reliable energy estimation for unseen model-hardware combinations. These results validate the compute-bound nature of diffusion inference and provide a foundation for sustainable AI deployment planning and carbon footprint estimation.
Authors Aniketh Iyengar, Jiaqi Han, Boris Ruf, Vincent Grari, Marcin Detyniecki, Stefano Ermon
Comment Accepted at EurIPS 2025 workshop "Rethinking AI: Efficiency, Frugality, and Sustainability"
Categories Computer Vision and Pattern Recognition
0
A statistical method for crack pre-detection in 3D concrete images
Title CN 一种用于3D混凝土图像裂缝预检测的统计方法
Keywords 3D图像分析 · 裂缝检测 · 统计方法 · 预处理 · CT图像
Summary 本文提出了一种基于统计学的裂缝预定位框架，用于在大规模3D CT图像中高效识别可能包含裂缝的区域。该方法结合Hessian滤波、规则空间划分上的几何特征以及空间多重假设检验，在仅需少量校准数据的情况下实现异常区域检测。实验表明该方法能以线性计算复杂度可靠地突出潜在裂缝区，从而为后续高分辨率分割模型提供兴趣区域，提升整体效率。但其应用场景局限于工业3D成像，不涉及文本、布局或语义理解。
Reason 该论文研究的是3D混凝土CT图像中的裂缝预检测，属于工业检测与3D视觉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）等任务完全无关。其技术路线聚焦于Hessian滤波、几何描述符和空间多重检验，应用于材料科学中的结构分析，无法迁移至解决OCR幻觉、复杂排版理解或长文档推理等DIU核心痛点。
Abstract In practical applications, effectively segmenting cracks in large-scale computed tomography (CT) images holds significant importance for understanding the structural integrity of materials. Classical image-processing techniques and modern deep-learning models both face substantial computational challenges when applied directly to high resolution big data volumes. This paper introduces a statistical framework for crack pre-localization, whose purpose is not to replace or compete with segmentation networks, but to identify, with controlled error rates, the regions of a 3D CT image that are most likely to contain cracks. The method combines a simple Hessian-based filter, geometric descriptors computed on a regular spatial partition, and a spatial multiple testing procedure to detect anomalous regions while relying only on minimal calibration data, rather than large annotated datasets. Experiments on semi-synthetic and real 3D CT scans demonstrate that the proposed approach reliably highlights regions likely to contain cracks while preserving linear computational complexity. By restricting subsequent high resolution segmentation to these localized regions, deep-learning models can be trained and operate more efficiently, reducing both training runtime as well as resource consumption. The framework thus offers a practical and interpretable preprocessing step for large-scale CT inspection pipelines.
Authors Vitalii Makogin, Duc Nguyen, Evgeny Spodarev
Categories Computer Vision and Pattern Recognition
0
VLM-Augmented Degradation Modeling for Image Restoration Under Adverse Weather Conditions
Title CN 用于恶劣天气条件下图像恢复的VLM增强退化建模
Keywords 图像恢复 · 恶劣天气 · 视觉语言模型 · 退化建模 · 记忆库
Summary 本文提出了一种名为MVLR的统一模型，用于在多种恶劣天气条件下恢复退化图像。该模型结合轻量级编码器-解码器结构、视觉语言模型（VLM）和隐式记忆库（IMB），利用VLM进行链式推理以编码天气退化先验，并通过IMB存储连续的退化模式潜在表示。这些先验用于查询并检索细粒度退化原型，再通过动态交叉注意力机制与多尺度视觉特征融合，从而提升恢复精度。实验表明该方法在多个严重天气基准上优于现有方法。
Reason 该论文研究的是在雨、雾、雪等恶劣天气条件下的通用图像恢复问题，属于图像修复/去噪领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管使用了VLM进行退化先验建模，但其目标是提升自然图像的视觉质量，而非解决文档中的OCR幻觉、密集文本识别或复杂排版理解等问题。方法难以迁移到文档理解场景，属于无关领域。
Abstract Reliable visual perception under adverse weather conditions, such as rain, haze, snow, or a mixture of them, is desirable yet challenging for autonomous driving and outdoor robots. In this paper, we propose a unified Memory-Enhanced Visual-Language Recovery (MVLR) model that restores images from different degradation levels under various weather conditions. MVLR couples a lightweight encoder-decoder backbone with a Visual-Language Model (VLM) and an Implicit Memory Bank (IMB). The VLM performs chain-of-thought inference to encode weather degradation priors and the IMB stores continuous latent representations of degradation patterns. The VLM-generated priors query the IMB to retrieve fine-grained degradation prototypes. These prototypes are then adaptively fused with multi-scale visual features via dynamic cross-attention mechanisms, enhancing restoration accuracy while maintaining computational efficiency. Extensive experiments on four severe-weather benchmarks show that MVLR surpasses single-branch and Mixture-of-Experts baselines in terms of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM). These results indicate that MVLR offers a practical balance between model compactness and expressiveness for real-time deployment in diverse outdoor conditions.
Authors Qianyi Shao, Yuanfan Zhang, Renxiang Xiao, Liang Hu
Categories Computer Vision and Pattern Recognition
0
DepthFocus: Controllable Depth Estimation for See-Through Scenes
Title CN DepthFocus：可控制的穿透场景深度估计
Keywords 深度估计 · 透明物体 · Vision Transformer · 可控感知 · 立体匹配
Summary 本文提出DepthFocus，一种基于意图控制的Vision Transformer模型，用于在包含透明和反射物体的复杂场景中进行可调节的深度估计。通过引入标量深度偏好作为条件信号，模型能够动态调整计算过程以聚焦于特定深度层，并在新构建的50万规模合成数据集上训练。实验表明其在BOOSTER等基准上达到SOTA，并在新提出的多深度数据集上展现出良好的意图对齐性和泛化能力。
Reason 该论文研究的是透明物体场景下的深度估计问题，属于3D视觉与计算机视觉中的立体匹配领域，核心任务为意图驱动的深度图生成。其方法聚焦于透明材质的多层深度感知，与文档图像理解（DIU）的核心痛点如OCR幻觉、文本识别、复杂版式分析等无直接关联。尽管其‘可控制’的Vision Transformer架构具有一定新颖性，但迁移至文档理解场景（如VQA或布局分析）的路径不明确，且不属于DIU本题或可迁移的上游方法工具库。
Abstract Depth in the real world is rarely singular. Transmissive materials create layered ambiguities that confound conventional perception systems. Existing models remain passive, attempting to estimate static depth maps anchored to the nearest surface, while humans actively shift focus to perceive a desired depth. We introduce DepthFocus, a steerable Vision Transformer that redefines stereo depth estimation as intent-driven control. Conditioned on a scalar depth preference, the model dynamically adapts its computation to focus on the intended depth, enabling selective perception within complex scenes. The training primarily leverages our newly constructed 500k multi-layered synthetic dataset, designed to capture diverse see-through effects. DepthFocus not only achieves state-of-the-art performance on conventional single-depth benchmarks like BOOSTER, a dataset notably rich in transparent and reflective objects, but also quantitatively demonstrates intent-aligned estimation on our newly proposed real and synthetic multi-depth datasets. Moreover, it exhibits strong generalization capabilities on unseen see-through scenes, underscoring its robustness as a significant step toward active and human-like 3D perception.
Authors Junhong Min, Jimin Kim, Cheol-Hui Min, Minwook Kim, Youngpil Jeon, Minyong Choi
Comment 8pages, 6 figures, 5 tables
Categories Computer Vision and Pattern Recognition
0
RadioKMoE: Knowledge-Guided Radiomap Estimation with Kolmogorov-Arnold Networks and Mixture-of-Experts
Title CN RadioKMoE：基于柯尔莫哥洛夫-阿诺德网络和专家混合模型的知识引导射电图估计
Keywords 射电图估计 · Kolmogorov-Arnold Networks · 专家混合模型 · 无线网络管理 · 信号传播建模
Summary 本文提出了一种名为RadioKMoE的知识引导射电图估计框架，结合柯尔莫哥洛夫-阿诺德网络（KAN）与专家混合模型（MoE）。KAN模块用于建模全局无线电传播模式并生成初始粗略覆盖图，而MoE模块则利用环境信息和初始图来细化局部细节，提升估计精度与鲁棒性。实验表明该方法在单波段和多波段场景下均优于现有方法。
Reason 该论文研究的是无线信号传播中的射电图估计（Radiomap Estimation），属于通信与无线网络领域，尽管使用了KAN和MoE等模型结构，但其任务与文档图像理解（DIU）、DocVQA、OCR、布局分析、表格识别等完全无关。未涉及任何形式的文档视觉内容解析或多模态语言模型在文本布局上的应用，无法迁移到解决OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Radiomap serves as a vital tool for wireless network management and deployment by providing powerful spatial knowledge of signal propagation and coverage. However, increasingly complex radio propagation behavior and surrounding environments pose strong challenges for radiomap estimation (RME). In this work, we propose a knowledge-guided RME framework that integrates Kolmogorov-Arnold Networks (KAN) with Mixture-of-Experts (MoE), namely RadioKMoE. Specifically, we design a KAN module to predict an initial coarse coverage map, leveraging KAN's strength in approximating physics models and global radio propagation patterns. The initial coarse map, together with environmental information, drives our MoE network for precise radiomap estimation. Unlike conventional deep learning models, the MoE module comprises expert networks specializing in distinct radiomap patterns to improve local details while preserving global consistency. Experimental results in both multi- and single-band RME demonstrate the enhanced accuracy and robustness of the proposed RadioKMoE in radiomap estimation.
Authors Fupei Guo, Kerry Pan, Songyang Zhang, Yue Wang, Zhi Ding
Categories Computer Vision and Pattern Recognition
0
A Diversity-optimized Deep Ensemble Approach for Accurate Plant Leaf Disease Detection
Title CN 一种面向准确植物叶片病害检测的多样性优化深度集成方法
Keywords 深度集成 · 多样性度量 · 植物病害检测 · 计算机视觉 · SQ框架
Summary 本文提出了一种名为Synergistic Diversity（SQ）的框架，用于优化深度神经网络集成模型的选择，以提高植物叶片病害的检测准确率。作者分析了现有集成多样性度量的局限性，并提出了新的SQ指标来更好地衡量模型间的协同效应，实验表明该方法能有效提升病害识别精度。
Reason 该论文研究的是农业图像中的植物病害识别问题，属于特定领域的下游应用（植物叶片图像分类），与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。尽管使用了深度集成和多样性度量等技术，但其方法并未针对文档特有的挑战（如OCR幻觉、复杂版式、细粒度文本定位）设计，也无法直接迁移用于提升VLM在文档理解中的性能。
Abstract Plant diseases pose a significant threat to global agriculture, causing over $220 billion in annual economic losses and jeopardizing food security. The timely and accurate detection of these diseases from plant leaf images is critical to mitigating their adverse effects. Deep neural network Ensembles (Deep Ensembles) have emerged as a powerful approach to enhancing prediction accuracy by leveraging the strengths of diverse Deep Neural Networks (DNNs). However, selecting high-performing ensemble member models is challenging due to the inherent difficulty in measuring ensemble diversity. In this paper, we introduce the Synergistic Diversity (SQ) framework to enhance plant disease detection accuracy. First, we conduct a comprehensive analysis of the limitations of existing ensemble diversity metrics (denoted as Q metrics), which often fail to identify optimal ensemble teams. Second, we present the SQ metric, a novel measure that captures the synergy between ensemble members and consistently aligns with ensemble accuracy. Third, we validate our SQ approach through extensive experiments on a plant leaf image dataset, which demonstrates that our SQ metric substantially improves ensemble selection and enhances detection accuracy. Our findings pave the way for a more reliable and efficient image-based plant disease detection.
Authors Sai Nath Chowdary Medikonduru, Hongpeng Jin, Yanzhao Wu
Categories Computer Vision and Pattern Recognition
0
Automated Muscle and Fat Segmentation in Computed Tomography for Comprehensive Body Composition Analysis Journal of Machine Learning for Biomedical Imaging (MELBA)
Title CN 基于CT图像的肌肉和脂肪自动分割用于全面的身体成分分析
Keywords 医学图像分割 · CT图像 · 身体成分分析
Summary 本文提出了一种端到端的模型，用于在轴向CT图像中对骨骼肌、皮下脂肪和内脏脂肪进行自动分割，并计算多种身体成分指标。模型在多个内部和外部数据集上表现优异，Dice系数超过89%，且各项指标的平均相对绝对误差低于10%。相关代码和权重已公开。
Reason 该论文属于医学图像分析领域，研究任务为CT影像中的组织分割，与文档图像理解（DIU）或DocVQA无关。其方法聚焦于生物医学成像中的体成分评估，未涉及OCR、布局分析、表格识别或任何文档结构理解任务。尽管使用了计算机视觉技术，但属于垂直医疗应用，且无通用视觉表示或推理机制可迁移至文档理解场景。
Abstract Body composition assessment using CT images can potentially be used for a number of clinical applications, including the prognostication of cardiovascular outcomes, evaluation of metabolic health, monitoring of disease progression, assessment of nutritional status, prediction of treatment response in oncology, and risk stratification for surgical and critical care outcomes. While multiple groups have developed in-house segmentation tools for this analysis, there are very limited publicly available tools that could be consistently used across different applications. To mitigate this gap, we present a publicly accessible, end-to-end segmentation and feature calculation model specifically for CT body composition analysis. Our model performs segmentation of skeletal muscle, subcutaneous adipose tissue (SAT), and visceral adipose tissue (VAT) across the chest, abdomen, and pelvis area in axial CT images. It also provides various body composition metrics, including muscle density, visceral-to-subcutaneous fat (VAT/SAT) ratio, muscle area/volume, and skeletal muscle index (SMI), supporting both 2D and 3D assessments. To evaluate the model, the segmentation was applied to both internal and external datasets, with body composition metrics analyzed across different age, sex, and race groups. The model achieved high dice coefficients on both internal and external datasets, exceeding 89% for skeletal muscle, SAT, and VAT segmentation. The model outperforms the benchmark by 2.10% on skeletal muscle and 8.6% on SAT compared to the manual annotations given by the publicly available dataset. Body composition metrics show mean relative absolute errors (MRAEs) under 10% for all measures. Our model with weights is publicly available at https://github.com/mazurowski-lab/CT-Muscle-and-Fat-Segmentation.git.
Authors Yaqian Chen, Hanxue Gu, Yuwen Chen, Jichen Yang, Haoyu Dong, Joseph Y. Cao, Adrian Camarena, Christopher Mantyh, Roy Colglazier, Maciej A. Mazurowski
Comment Accepted for publication at the Journal of Machine Learning for Biomedical Imaging (MELBA) https://melba-journal.org/2025:026
Categories Computer Vision and Pattern Recognition
0
Gradient-Driven Natural Selection for Compact 3D Gaussian Splatting
Title CN 面向紧凑型3D高斯点阵的梯度驱动自然选择
Keywords 3D高斯点阵 · 模型压缩 · 自然选择 · 剪枝 · 梯度优化
Summary 本文提出一种受自然选择启发的3D高斯点阵（3DGS）压缩框架，通过引入由渲染质量目标驱动的正则化梯度场来自动决定哪些高斯元素应被保留或剪枝，实现无需人工干预的紧凑化优化。同时设计了一种带有有限不透明度先验的衰减技术以加速选择过程，在15%预算下相比原始3DGS提升了0.6 dB以上的PSNR，达到当前最优的压缩性能。
Reason 该论文研究的是3D场景表示中的高斯点阵压缩问题，属于3D视觉与图形学领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心方法虽涉及模型压缩和优化策略，但针对的是3D渲染质量与存储效率的权衡，无法迁移解决文档理解中的OCR幻觉、复杂排版解析或长文档推理等关键痛点。
Abstract 3DGS employs a large number of Gaussian primitives to fit scenes, resulting in substantial storage and computational overhead. Existing pruning methods rely on manually designed criteria or introduce additional learnable parameters, yielding suboptimal results. To address this, we propose an natural selection inspired pruning framework that models survival pressure as a regularization gradient field applied to opacity, allowing the optimization gradients--driven by the goal of maximizing rendering quality--to autonomously determine which Gaussians to retain or prune. This process is fully learnable and requires no human intervention. We further introduce an opacity decay technique with a finite opacity prior, which accelerates the selection process without compromising pruning effectiveness. Compared to 3DGS, our method achieves over 0.6 dB PSNR gain under 15\% budgets, establishing state-of-the-art performance for compact 3DGS. Project page https://xiaobin2001.github.io/GNS-web.
Authors Xiaobin Deng, Qiuli Yu, Changyu Diao, Min Li, Duanqing Xu
Categories Computer Vision and Pattern Recognition
0
Generalizable and Relightable Gaussian Splatting for Human Novel View Synthesis
Title CN 用于人体新视角合成的可泛化且可重光照的高斯点阵
Keywords 3D Gaussian Splatting · 新视角合成 · 可重光照渲染 · 人体重建 · 物理感知渲染
Summary 本文提出GRGS，一种可泛化且支持重光照的3D高斯框架，用于在多种光照条件下实现高质量的人体新视角合成。与依赖单个体素优化或忽略物理约束的方法不同，GRGS采用前馈式全监督策略，将多视角2D观测中的几何、材质和光照信息投影到3D高斯表示中。为在多变光照下恢复准确几何，作者设计了基于合成重光照数据训练的光照鲁棒几何 refinement 模块（LGR），以预测精确深度和法线；并进一步提出物理基础神经渲染（PGNR）模块，结合神经网络预测与基于物理的着色模型，支持包含阴影和间接光照的可编辑重光照。此外，通过利用环境遮挡、直射与间接光照图的可微监督，设计了一种2D到3D的投影训练方案，降低了光线追踪的计算开销。实验表明，GRGS在视觉质量、几何一致性和跨人物、跨光照泛化能力上表现优越。
Reason 该论文属于3D视觉与人体新视角合成领域，研究目标是生成在不同光照条件下的人体三维视图，涉及高斯点阵、物理渲染和多视角几何等技术。其核心问题与文档图像理解（DIU）无直接关联，无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU关键痛点。
Abstract We propose GRGS, a generalizable and relightable 3D Gaussian framework for high-fidelity human novel view synthesis under diverse lighting conditions. Unlike existing methods that rely on per-character optimization or ignore physical constraints, GRGS adopts a feed-forward, fully supervised strategy projecting geometry, material, and illumination cues from multi-view 2D observations into 3D Gaussian representations. To recover accurate geometry under diverse lighting conditions, we introduce a Lighting-robust Geometry Refinement (LGR) module trained on synthetically relit data to predict precise depth and surface normals. Based on the high-quality geometry, a Physically Grounded Neural Rendering (PGNR) module is further proposed to integrate neural prediction with physics-based shading, supporting editable relighting with shadows and indirect illumination. Moreover, we design a 2D-to-3D projection training scheme leveraging differentiable supervision from ambient occlusion, direct, and indirect lighting maps, alleviating the computational cost of ray tracing. Extensive experiments demonstrate that GRGS achieves superior visual quality, geometric consistency, and generalization across characters and lighting conditions.
Authors Yipengjing Sun, Shengping Zhang, Chenyang Wang, Shunyuan Zheng, Zonglin Li, Xiangyang Ji
Comment Project Webpage: https://sypj-98.github.io/grgs/
Categories Computer Vision and Pattern Recognition
0
Universal Video Temporal Grounding with Generative Multi-modal Large Language Models
Title CN 基于生成式多模态大语言模型的通用视频时序定位
Keywords 视频时序定位 · 多模态大语言模型 · 时间戳生成 · 自适应帧缩放 · 长视频理解
Summary 本文提出UniTime模型，利用生成式多模态大语言模型实现通用视频时序定位，通过插入时间戳标记并与视频标记交错，结合自适应帧缩放策略，在多种视频长度和输入粒度下实现精确的时间片段定位。实验表明其在零样本和微调设置下均优于现有方法，并能提升长视频问答性能。
Reason 该论文研究的是视频时序定位任务，属于视频理解领域，与文档图像理解（DIU）的核心任务无直接关联。尽管使用了多模态大语言模型并涉及时间戳生成和细粒度对齐，但其方法针对的是视频中的时间动态建模，难以迁移到文档图像中的空间结构理解、OCR去幻觉或复杂排版分析等痛点问题。
Abstract This paper presents a computational model for universal video temporal grounding, which accurately localizes temporal moments in videos based on natural language queries (e.g., questions or descriptions). Unlike existing methods that are often limited to specific video domains or durations, we propose UniTime, a robust and universal video grounding model leveraging the strong vision-language understanding capabilities of generative Multi-modal Large Language Models (MLLMs). Our model effectively handles videos of diverse views, genres, and lengths while comprehending complex language queries. The key contributions include: (i) We consider steering strong MLLMs for temporal grounding in videos. To enable precise timestamp outputs, we incorporate temporal information by interleaving timestamp tokens with video tokens. (ii) By training the model to handle videos with different input granularities through adaptive frame scaling, our approach achieves robust temporal grounding for both short and long videos. (iii) Comprehensive experiments show that UniTime outperforms state-of-the-art approaches in both zero-shot and dataset-specific finetuned settings across five public temporal grounding benchmarks. (iv) When employed as a preliminary moment retriever for long-form video question-answering (VideoQA), UniTime significantly improves VideoQA accuracy, highlighting its value for complex video understanding tasks.
Authors Zeqian Li, Shangzhe Di, Zhonghua Zhai, Weilin Huang, Yanfeng Wang, Weidi Xie
Categories Computer Vision and Pattern Recognition
0
Real-Time Cooked Food Image Synthesis and Visual Cooking Progress Monitoring on Edge Devices
Title CN 基于边缘设备的实时烹饪食品图像合成与视觉烹饪进度监控
Keywords 图像生成 · 边缘计算 · 烹饪进度监控 · 图像到图像转换 · 视觉质量评估
Summary 本文提出了一种面向边缘设备的烹饪图像生成模型，能够根据原始食材图像和菜谱信息生成具有时间一致性的烹饪过程图像，并引入了一个名为‘烹饪图像相似度’（CIS）的域特定指标用于训练和进度监控。尽管在生成质量和效率上有提升，但其应用领域为食品图像生成，远离文档理解范畴。
Reason 该论文属于纯图像生成任务，且聚焦于食物图像合成与烹饪状态监控，属于图像生成与视频理解的交叉领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其提出的方法和数据集无法迁移至解决OCR幻觉、复杂排版或长文档推理等DIU核心痛点。
Abstract Synthesizing realistic cooked food images from raw inputs on edge devices is a challenging generative task, requiring models to capture complex changes in texture, color and structure during cooking. Existing image-to-image generation methods often produce unrealistic results or are too resource-intensive for edge deployment. We introduce the first oven-based cooking-progression dataset with chef-annotated doneness levels and propose an edge-efficient recipe and cooking state guided generator that synthesizes realistic food images conditioned on raw food image. This formulation enables user-preferred visual targets rather than fixed presets. To ensure temporal consistency and culinary plausibility, we introduce a domain-specific \textit{Culinary Image Similarity (CIS)} metric, which serves both as a training loss and a progress-monitoring signal. Our model outperforms existing baselines with significant reductions in FID scores (30\% improvement on our dataset; 60\% on public datasets)
Authors Jigyasa Gupta, Soumya Goyal, Anil Kumar, Ishan Jindal
Comment 13 pages, 11 figures
Categories Computer Vision and Pattern Recognition
0
QuantFace: Efficient Quantization for Face Restoration
Title CN QuantFace：面向人脸修复的高效量化
Keywords 人脸修复 · 扩散模型 · 模型量化 · 低比特 · QD-LoRA
Summary 本文提出QuantFace，一种用于人脸修复模型的低比特量化框架，通过旋转-缩放通道平衡、联合量化与蒸馏优化的QD-LoRA方法以及自适应比特分配策略，在4~6比特下保持了高质量的人脸恢复性能。实验表明其在合成与真实数据集上优于现有低比特方法。
Reason 该论文研究的是人脸修复（face restoration）中的模型量化问题，属于图像生成与修复领域，且聚焦于人脸这一特定模态。尽管其低比特量化和推理效率优化技术具有一定通用性，但方法设计目标与文档图像理解（DIU）的核心痛点（如OCR幻觉、复杂排版、长文档推理等）无直接关联。此外，人脸修复任务本身与文档内容结构无关，迁移价值极低。
Abstract Diffusion models have been achieving remarkable performance in face restoration. However, the heavy computations hamper the widespread adoption of these models. In this work, we propose QuantFace, a novel low-bit quantization framework for face restoration models, where the full-precision (i.e., 32-bit) weights and activations are quantized to 4~6-bit. We first analyze the data distribution within activations and find that it is highly variant. To preserve the original data information, we employ rotation-scaling channel balancing. Furthermore, we propose Quantization-Distillation Low-Rank Adaptation (QD-LoRA), which jointly optimizes for quantization and distillation performance. Finally, we propose an adaptive bit-width allocation strategy. We formulate such a strategy as an integer programming problem that combines quantization error and perceptual metrics to find a satisfactory resource allocation. Extensive experiments on the synthetic and real-world datasets demonstrate the effectiveness of QuantFace under 6-bit and 4-bit. QuantFace achieves significant advantages over recent leading low-bit quantization methods for face restoration. The code is available at https://github.com/jiatongli2024/QuantFace.
Authors Jiatong Li, Libo Zhu, Haotong Qin, Jingkai Wang, Linghe Kong, Guihai Chen, Yulun Zhang, Xiaokang Yang
Categories Computer Vision and Pattern Recognition
0
MatPedia: A Universal Generative Foundation for High-Fidelity Material Synthesis
Title CN MatPedia：一种用于高保真材料合成的通用生成基础模型
Keywords PBR材料生成 · 扩散模型 · 联合表示学习 · 物理渲染 · 图像生成
Summary MatPedia提出了一种新的联合RGB-PBR表示方法，将材料编码为两个相互依赖的潜在变量，并通过视频扩散架构实现高质量的1024×1024材料生成。该模型在文本到材料、图像到材料和内在分解等多个任务上实现了统一建模，但在文档理解任务中缺乏可迁移的技术价值。
Reason 该论文研究的是物理渲染材质（PBR）的生成，属于计算机图形学中的材质合成领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管其方法涉及多模态表示和扩散模型架构，但核心任务聚焦于RGB外观与物理属性之间的联合建模，无法迁移解决OCR幻觉、复杂排版理解、细粒度文本定位等DIU核心痛点。
Abstract Physically-based rendering (PBR) materials are fundamental to photorealistic graphics, yet their creation remains labor-intensive and requires specialized expertise. While generative models have advanced material synthesis, existing methods lack a unified representation bridging natural image appearance and PBR properties, leading to fragmented task-specific pipelines and inability to leverage large-scale RGB image data. We present MatPedia, a foundation model built upon a novel joint RGB-PBR representation that compactly encodes materials into two interdependent latents: one for RGB appearance and one for the four PBR maps encoding complementary physical properties. By formulating them as a 5-frame sequence and employing video diffusion architectures, MatPedia naturally captures their correlations while transferring visual priors from RGB generation models. This joint representation enables a unified framework handling multiple material tasks--text-to-material generation, image-to-material generation, and intrinsic decomposition--within a single architecture. Trained on MatHybrid-410K, a mixed corpus combining PBR datasets with large-scale RGB images, MatPedia achieves native $1024\times1024$ synthesis that substantially surpasses existing approaches in both quality and diversity.
Authors Di Luo, Shuhui Yang, Mingxin Yang, Jiawei Lu, Yixuan Tang, Xintong Han, Zhuo Chen, Beibei Wang, Chunchao Guo
Categories Computer Vision and Pattern Recognition
0
SpotFormer: Multi-Scale Spatio-Temporal Transformer for Facial Expression Spotting
Title CN SpotFormer：用于面部表情识别的多尺度时空Transformer
Keywords 面部表情识别 · 微表情检测 · 时空Transformer · 光学流 · 对比学习
Summary 本文提出了一种名为SpotFormer的多尺度时空Transformer模型，用于视频中面部表情的识别，特别是微表情的检测。通过滑动窗口多时间分辨率光流（SW-MRO）捕捉细微面部运动，并结合局部图池化和对比学习提升表达能力。在多个微表情数据集上取得了优异性能。
Reason 该论文研究的是视频中的面部表情识别，属于视频理解领域，且任务聚焦于微表情检测与光学流特征设计。其核心方法（如SW-MRO、FLGP）针对人脸局部运动建模，与文档图像理解（DIU）中的OCR幻觉、复杂版面分析、文本识别等核心问题无直接关联。无法迁移用于解决文档图像中的细粒度定位或高分辨率处理等痛点。
Abstract Facial expression spotting, identifying periods where facial expressions occur in a video, is a significant yet challenging task in facial expression analysis. The issues of irrelevant facial movements and the challenge of detecting subtle motions in micro-expressions remain unresolved, hindering accurate expression spotting. In this paper, we propose an efficient framework for facial expression spotting. First, we propose a Sliding Window-based multi-temporal-resolution Optical flow (SW-MRO) feature, which calculates multi-temporal-resolution optical flow of the input image sequence within compact sliding windows. The window length is tailored to perceive complete micro-expressions and distinguish between general macro- and micro-expressions. SW-MRO can effectively reveal subtle motions while avoiding the optical flow being dominated by head movements. Second, we propose SpotFormer, a multi-scale spatio-temporal Transformer that simultaneously encodes spatio-temporal relationships of the SW-MRO features for accurate frame-level probability estimation. In SpotFormer, we use the proposed Facial Local Graph Pooling (FLGP) operation and convolutional layers to extract multi-scale spatio-temporal features. We show the validity of the architecture of SpotFormer by comparing it with several model variants. Third, we introduce supervised contrastive learning into SpotFormer to enhance the discriminability between different types of expressions. Extensive experiments on SAMM-LV, CAS(ME)^2, and CAS(ME)^3 show that our method outperforms state-of-the-art models, particularly in micro-expression spotting.
Authors Yicheng Deng, Hideaki Hayashi, Hajime Nagahara
Categories Computer Vision and Pattern Recognition
0
Semantic-ICP: Iterative Closest Point for Non-rigid Multi-Organ Point Cloud Registration
Title CN Semantic-ICP：用于非刚性多器官点云配准的迭代最近点算法
Keywords 点云配准 · Iterative Closest Point · 语义标签 · 非刚性配准 · 计算机辅助干预
Summary 本文提出了一种语义增强的非刚性迭代最近点算法（SemICP），结合解剖语义标签和线弹性能量正则化，用于多器官点云的精确配准。实验表明该方法在Hausdorff距离和表面平均距离上优于现有方法，并可与深度学习分割模型结合实现超声与磁共振点云的有效对齐。
Reason 该论文属于医学图像分析中的点云配准任务，聚焦于多器官的非刚性配准与生物力学约束建模，属于3D视觉与具身智能（CAI）交叉领域。尽管涉及点云和配准，但其方法与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版、细粒度文本识别或VLM架构改进无直接关联，且无法迁移至2D文档结构或语言-视觉对齐任务。
Abstract Point cloud registration is important in computer-aided interventions (CAI). While learning-based point cloud registration methods have been developed, their clinical application is hampered by issues of generalizability and explainability. Therefore, classical point cloud registration methods, including Iterative Closest Point (ICP), are still widely applied in CAI. ICP methods fail to consider that: (1) the points have well-defined semantic meaning, in that each point can be related to a specific anatomical label; (2) the deformation required for registration needs to follow biomechanical energy constraints. In this paper, we present a novel non-rigid semantic ICP (SemICP) method that handles multiple point labels and uses linear elastic energy regularization. We use semantic labels to improve the robustness of closest point matching and propose a novel point cloud deformation representation that incorporates explicit biomechanical energy regularization. Our experiments on four datasets show that our method significantly improves the Hausdorff distance and mean surface distance compared with other point cloud registration methods. We also demonstrate that integrating deep learning segmentation models with our registration pipeline enables effective alignment of US and MR point clouds.
Authors Wanwen Chen, Qi Zeng, Carson Studders, Jamie J. Y. Kwon, Emily H. T. Pang, Eitan Prisman, Septimiu E. Salcudean
Categories Computer Vision and Pattern Recognition
0
Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Models
Title CN Neighbor GRPO：对比常微分方程策略优化对齐流模型
Keywords 流匹配 · 常微分方程 · 偏好对齐 · 生成模型 · 对比学习
Summary 本文提出Neighbor GRPO，一种用于对齐现代流匹配生成模型与人类偏好的新算法。通过在ODE框架下扰动初始噪声条件生成候选轨迹，并采用基于距离的softmax策略进行优化，避免了传统SDE转换带来的效率低下问题。方法在保持确定性采样优势的同时，提升了训练效率和生成质量。
Reason 该论文聚焦于图像和视频生成模型的偏好对齐问题，属于纯视觉生成领域，且核心方法围绕流匹配模型（flow matching）与ODE/SDE采样机制展开。尽管涉及GRPO这一强化学习风格的优化框架，但其目标是提升生成质量而非解决文档理解中的OCR幻觉、复杂排版或长文本推理等核心痛点。无法迁移至文档图像理解任务，属于无关领域。
Abstract Group Relative Policy Optimization (GRPO) has shown promise in aligning image and video generative models with human preferences. However, applying it to modern flow matching models is challenging because of its deterministic sampling paradigm. Current methods address this issue by converting Ordinary Differential Equations (ODEs) to Stochastic Differential Equations (SDEs), which introduce stochasticity. However, this SDE-based GRPO suffers from issues of inefficient credit assignment and incompatibility with high-order solvers for fewer-step sampling. In this paper, we first reinterpret existing SDE-based GRPO methods from a distance optimization perspective, revealing their underlying mechanism as a form of contrastive learning. Based on this insight, we propose Neighbor GRPO, a novel alignment algorithm that completely bypasses the need for SDEs. Neighbor GRPO generates a diverse set of candidate trajectories by perturbing the initial noise conditions of the ODE and optimizes the model using a softmax distance-based surrogate leaping policy. We establish a theoretical connection between this distance-based objective and policy gradient optimization, rigorously integrating our approach into the GRPO framework. Our method fully preserves the advantages of deterministic ODE sampling, including efficiency and compatibility with high-order solvers. We further introduce symmetric anchor sampling for computational efficiency and group-wise quasi-norm reweighting to address reward flattening. Extensive experiments demonstrate that Neighbor GRPO significantly outperforms SDE-based counterparts in terms of training cost, convergence speed, and generation quality.
Authors Dailan He, Guanlin Feng, Xingtong Ge, Yazhe Niu, Yi Zhang, Bingqi Ma, Guanglu Song, Yu Liu, Hongsheng Li
Categories Computer Vision and Pattern Recognition
0
FingerCap: Fine-grained Finger-level Hand Motion Captioning
Title CN FingerCap：细粒度手指级手部动作描述
Keywords 手部动作描述 · 视频理解 · 关键点建模 · 多模态学习 · FiGOP
Summary 本文提出了FingerCap任务和数据集，旨在生成描述手指级精细动作的文本，并构建了包含4万对样本的大规模数据集。为解决视频MLLM时序稀疏性问题，引入FiGOP方法，通过结合RGB关键帧与后续手部关键点来恢复细粒度动态信息。尽管在手部动作描述上有效，但其技术路径局限于视频动作理解领域。
Reason 该论文研究的是细粒度手部动作的视频描述生成，属于视频理解领域，核心任务为手部运动捕捉与自然语言描述的对齐。其提出的方法FiGOP虽在时序建模上有改进，但应用场景聚焦于人体动作识别中的高频率细微运动建模，与文档图像理解（DIU）中的OCR幻觉、复杂排版、长文档推理等核心痛点无直接关联。方法难以迁移至文档场景，尤其是无法解决视觉文本识别、布局分析或表格理解等问题。
Abstract Understanding fine-grained human hand motion is fundamental to visual perception, embodied intelligence, and multimodal communication. In this work, we propose Fine-grained Finger-level Hand Motion Captioning (FingerCap), which aims to generate textual descriptions that capture detailed finger-level semantics of hand actions. To support this task, we curate FingerCap-40K, a large-scale corpus of 40K paired hand-motion videos and captions spanning two complementary sources: concise instruction-style finger motions and diverse, naturalistic hand-object interactions. To enable effective evaluation, we employ HandJudge, a LLM-based rubric that measures finger-level correctness and motion completeness. Temporal sparsity remains a fundamental bottleneck for current Video-MLLMs, since sparse RGB sampling is insufficient to capture the subtle, high-frequency dynamics underlying fine finger motions. As a simple and compute-friendly remedy, we introduce FiGOP (Finger Group-of-Pictures), which pairs each RGB keyframe with subsequent hand keypoints until the next keyframe. A lightweight temporal encoder converts the keypoints into motion embeddings and integrates them with RGB features. FiGOP adapts the classic GOP concept to finger motion, recovering fine temporal cues without increasing RGB density. Experiments on FingerCap-40K show that strong open- and closed-source Video-MLLMs still struggle with finger-level reasoning, while our FiGOP-augmented model yield consistent gains under HandJudge and human studies.
Authors Xin Shen, Rui Zhu, Lei Shen, Xinyu Wang, Kaihao Zhang, Tianqing Zhu, Shuchen Wu, Chenxi Miao, Weikang Li, Yang Li, Deguo Xia, Jizhou Huang, Xin Yu
Categories Computer Vision and Pattern Recognition
0
MobileOcc: A Human-Aware Semantic Occupancy Dataset for Mobile Robots
Title CN MobileOcc：用于移动机器人的类人语义占据数据集
Keywords 语义占据 · 移动机器人 · 3D重建 · LiDAR · 人体几何建模
Summary 本文提出了MobileOcc，一个面向移动机器人的人类感知语义占据数据集，通过结合静态物体占据标注与一种专为人类占据建模设计的网格优化框架，利用2D图像和LiDAR点云数据重建可变形的人体几何结构。基于此数据集，作者建立了占据预测和行人速度预测的基准，并在3D人体姿态估计数据集上验证了标注方法的有效性，结果表明其方法具有良好的跨数据集鲁棒性。
Reason 该论文属于具身智能与机器人领域，研究的是3D语义占据感知和人类几何建模，应用于移动机器人在人群环境中的导航。其核心任务（occupancy prediction、3D human pose estimation）与文档图像理解（DIU）无直接关联，且方法依赖于LiDAR和多视角重建，无法迁移到文档视觉或VLM相关任务中。
Abstract Dense 3D semantic occupancy perception is critical for mobile robots operating in pedestrian-rich environments, yet it remains underexplored compared to its application in autonomous driving. To address this gap, we present MobileOcc, a semantic occupancy dataset for mobile robots operating in crowded human environments. Our dataset is built using an annotation pipeline that incorporates static object occupancy annotations and a novel mesh optimization framework explicitly designed for human occupancy modeling. It reconstructs deformable human geometry from 2D images and subsequently refines and optimizes it using associated LiDAR point data. Using MobileOcc, we establish benchmarks for two tasks, i) Occupancy prediction and ii) Pedestrian velocity prediction, using different methods including monocular, stereo, and panoptic occupancy, with metrics and baseline implementations for reproducible comparison. Beyond occupancy prediction, we further assess our annotation method on 3D human pose estimation datasets. Results demonstrate that our method exhibits robust performance across different datasets.
Authors Junseo Kim, Guido Dumont, Xinyu Gao, Gang Chen, Holger Caesar, Javier Alonso-Mora
Categories Computer Vision and Pattern Recognition
0
Flow-Guided Implicit Neural Representation for Motion-Aware Dynamic MRI Reconstruction
Title CN 用于运动感知动态MRI重建的流引导隐式神经表示
Keywords 隐式神经表示 · 动态MRI重建 · 光流 · 医学影像 · 运动补偿
Summary 本文提出一种基于隐式神经表示（INR）的框架，用于动态MRI重建，通过联合建模动态图像序列和光流场，利用光学流方程作为物理正则项，在无需预先估计运动的情况下实现高质量的时间一致图像恢复。实验表明该方法在心脏MRI数据上优于现有方法。
Reason 该论文研究的是动态磁共振成像（dMRI）中的运动建模与图像重建问题，属于医学影像重建领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心技术是基于隐式神经表示（INR）联合建模图像序列和光流场，解决的是医学图像中的时间一致性与运动伪影问题，无法迁移到文档图像中的OCR幻觉、复杂版面分析或细粒度文本定位等核心痛点。属于无关领域（医疗影像+视频级时序重建），不符合关注方向。
Abstract Dynamic magnetic resonance imaging (dMRI) captures temporally-resolved anatomy but is often challenged by limited sampling and motion-induced artifacts. Conventional motion-compensated reconstructions typically rely on pre-estimated optical flow, which is inaccurate under undersampling and degrades reconstruction quality. In this work, we propose a novel implicit neural representation (INR) framework that jointly models both the dynamic image sequence and its underlying motion field. Specifically, one INR is employed to parameterize the spatiotemporal image content, while another INR represents the optical flow. The two are coupled via the optical flow equation, which serves as a physics-inspired regularization, in addition to a data consistency loss that enforces agreement with k-space measurements. This joint optimization enables simultaneous recovery of temporally coherent images and motion fields without requiring prior flow estimation. Experiments on dynamic cardiac MRI datasets demonstrate that the proposed method outperforms state-of-the-art motion-compensated and deep learning approaches, achieving superior reconstruction quality, accurate motion estimation, and improved temporal fidelity. These results highlight the potential of implicit joint modeling with flow-regularized constraints for advancing dMRI reconstruction.
Authors Baoqing Li, Yuanyuan Liu, Congcong Liu, Qingyong Zhu, Jing Cheng, Yihang Zhou, Hao Chen, Zhuo-Xu Cui, Dong Liang
Comment 10 pages, 7 figures
Categories Computer Vision and Pattern Recognition
0
MultiPriv: Benchmarking Individual-Level Privacy Reasoning in Vision-Language Models
Title CN MultiPriv：评估视觉语言模型中个体级隐私推理的基准
Keywords 视觉语言模型 · 隐私推理 · 基准测试 · 多模态安全 · 信息链接
Summary 本文提出了MultiPriv，首个用于系统评估视觉语言模型（VLM）在个体级隐私推理方面风险的基准。作者设计了隐私感知与推理（PPR）框架，并构建了一个双语多模态数据集，包含合成的个人档案，将身份标识（如人脸、姓名）与敏感属性关联，支持九项从属性检测到跨图像重识别和链式推理的任务。对50多个基础与商用VLM的评估表明，现有模型存在显著的基于推理的隐私风险，且当前的安全对齐策略对此类风险无效。
Reason 该论文聚焦于视觉语言模型中的隐私推理与风险评估，属于AI安全与伦理范畴，尽管涉及VLM，但其核心任务是隐私保护评估，而非文档图像理解（DIU）或DocVQA等任务。未提出可用于解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点的方法论创新。属于平行应用类研究（在特定维度上评估VLM行为），不具方法迁移价值。
Abstract Modern Vision-Language Models (VLMs) demonstrate sophisticated reasoning, escalating privacy risks beyond simple attribute perception to individual-level linkage. Current privacy benchmarks are structurally insufficient for this new threat, as they primarily evaluate privacy perception while failing to address the more critical risk of privacy reasoning: a VLM's ability to infer and link distributed information to construct individual profiles. To address this critical gap, we propose \textbf{MultiPriv}, the first benchmark designed to systematically evaluate individual-level privacy reasoning in VLMs. We introduce the \textbf{Privacy Perception and Reasoning (PPR)} framework and construct a novel, bilingual multimodal dataset to support it. The dataset uniquely features a core component of synthetic individual profiles where identifiers (e.g., faces, names) are meticulously linked to sensitive attributes. This design enables nine challenging tasks evaluating the full PPR spectrum, from attribute detection to cross-image re-identification and chained inference. We conduct a large-scale evaluation of over 50 foundational and commercial VLMs. Our analysis reveals: (1) Many VLMs possess significant, unmeasured reasoning-based privacy risks. (2) Perception-level metrics are poor predictors of these reasoning risks, revealing a critical evaluation gap. (3) Existing safety alignments are inconsistent and ineffective against such reasoning-based attacks. MultiPriv exposes systemic vulnerabilities and provides the necessary framework for developing robust, privacy-preserving VLMs.
Authors Xiongtao Sun, Hui Li, Jiaming Zhang, Yujie Yang, Kaili Liu, Ruxin Feng, Wen Jun Tan, Wei Yang Bryan Lim
Categories Computer Vision and Pattern Recognition
0
Shape-preserving Tooth Segmentation from CBCT Images Using Deep Learning with Semantic and Shape Awareness
Title CN 基于语义和形状感知的深度学习方法实现CBCT图像中保形牙齿分割
Keywords 牙齿分割 · CBCT图像 · 深度学习 · 形状感知 · 语义关系
Summary 本文提出一种结合语义和形状感知的深度学习框架，用于从锥束CT（CBCT）图像中进行保形牙齿分割。通过引入以目标牙为中心的多标签学习策略建模牙齿间的语义关系，并设计形状感知机制来保持边界完整性。实验表明该方法在内部和外部数据集上均优于现有技术，能有效减少形状畸变，生成解剖学上更准确的牙齿边界。
Reason 该论文研究的是医学影像中的牙齿分割问题，属于医疗图像分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管使用了深度学习和形状感知机制，但其方法针对的是3D医学图像中的解剖结构分割，无法迁移至解决文档图像中的OCR幻觉、复杂排版或长文档推理等核心痛点。
Abstract Background:Accurate tooth segmentation from cone beam computed tomography (CBCT) images is crucial for digital dentistry but remains challenging in cases of interdental adhesions, which cause severe anatomical shape distortion. Methods: To address this, we propose a deep learning framework that integrates semantic and shape awareness for shape-preserving segmentation. Our method introduces a target-tooth-centroid prompted multi-label learning strategy to model semantic relationships between teeth, reducing shape ambiguity. Additionally, a tooth-shape-aware learning mechanism explicitly enforces morphological constraints to preserve boundary integrity. These components are unified via multi-task learning, jointly optimizing segmentation and shape preservation. Results: Extensive evaluations on internal and external datasets demonstrate that our approach significantly outperforms existing methods. Conclusions: Our approach effectively mitigates shape distortions and providing anatomically faithful tooth boundaries.
Authors Zongrui Ji, Zhiming Cui, Na Li, Qianhan Zheng, Miaojing Shi, Ke Deng, Jingyang Zhang, Chaoyuan Li, Xuepeng Chen, Yi Dong, Lei Ma
Categories Computer Vision and Pattern Recognition
0
SemanticStitch: Enhancing Image Coherence through Foreground-Aware Seam Carving The Visual Computer: International Journal of Computer Graphics, 2025
Title CN SemanticStitch：通过前景感知的接缝裁剪增强图像连贯性
Keywords 图像拼接 · 接缝裁剪 · 语义分割 · 视觉连贯性 · 深度学习
Summary 本文提出SemanticStatch，一种结合前景语义信息的深度学习框架，用于改善图像拼接中的视觉不一致问题。通过引入强调显著物体完整性的新损失函数，并构建两个真实世界数据集进行评估，实验表明其在视觉质量上优于传统方法。
Reason 该论文研究图像拼接中的视觉连贯性问题，核心是基于语义先验优化接缝裁剪，属于通用图像编辑/计算机视觉任务，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。未涉及OCR、布局分析、表格识别、多模态文档理解等DIU核心任务，也未提出可迁移至文档理解领域的底层方法（如视觉token压缩、高分辨率处理架构、去幻觉机制等）。属于无关领域工作。
Abstract Image stitching often faces challenges due to varying capture angles, positional differences, and object movements, leading to misalignments and visual discrepancies. Traditional seam carving methods neglect semantic information, causing disruptions in foreground continuity. We introduce SemanticStitch, a deep learning-based framework that incorporates semantic priors of foreground objects to preserve their integrity and enhance visual coherence. Our approach includes a novel loss function that emphasizes the semantic integrity of salient objects, significantly improving stitching quality. We also present two specialized real-world datasets to evaluate our method's effectiveness. Experimental results demonstrate substantial improvements over traditional techniques, providing robust support for practical applications.
Authors Ji-Ping Jin, Chen-Bin Feng, Rui Fan, Chi-Man Vong
Comment 12pages, has been early accepted by The Visual Computer: International Journal of Computer Graphics, 2025
Categories Computer Vision and Pattern Recognition
0
Rethinking Diffusion Model-Based Video Super-Resolution: Leveraging Dense Guidance from Aligned Features
Title CN 重新思考基于扩散模型的视频超分辨率：利用对齐特征的密集引导
Keywords 视频超分辨率 · 扩散模型 · 特征对齐 · 高频信息保持 · 时序一致性
Summary 本文提出了一种基于扩散模型的视频超分辨率方法DGAF-VSR，通过在特征域进行对齐和补偿，并设计了光学引导形变模块（OGWM）和特征级时序条件模块（FTCM），以提升视频超分的感知质量、保真度和时序一致性。实验表明其在多个指标上优于现有方法。
Reason 该论文研究的是视频超分辨率（VSR）任务，属于视频理解领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管其提出的特征域对齐和高频信息保持方法在理论上具有一定通用性，但迁移至文档图像理解场景的路径不明确，无法直接解决OCR幻觉、复杂排版解析或长文档推理等关键痛点。
Abstract Diffusion model (DM) based Video Super-Resolution (VSR) approaches achieve impressive perceptual quality. However, they suffer from error accumulation, spatial artifacts, and a trade-off between perceptual quality and fidelity, primarily caused by inaccurate alignment and insufficient compensation between video frames. In this paper, within the DM-based VSR pipeline, we revisit the role of alignment and compensation between adjacent video frames and reveal two crucial observations: (a) the feature domain is better suited than the pixel domain for information compensation due to its stronger spatial and temporal correlations, and (b) warping at an upscaled resolution better preserves high-frequency information, but this benefit is not necessarily monotonic. Therefore, we propose a novel Densely Guided diffusion model with Aligned Features for Video Super-Resolution (DGAF-VSR), with an Optical Guided Warping Module (OGWM) to maintain high-frequency details in the aligned features and a Feature-wise Temporal Condition Module (FTCM) to deliver dense guidance in the feature domain. Extensive experiments on synthetic and real-world datasets demonstrate that DGAF-VSR surpasses state-of-the-art methods in key aspects of VSR, including perceptual quality (35.82\% DISTS reduction), fidelity (0.20 dB PSNR gain), and temporal consistency (30.37\% tLPIPS reduction).
Authors Jingyi Xu, Meisong Zheng, Ying Chen, Minglang Qiao, Xin Deng, Mai Xu
Comment 19pages
Categories Computer Vision and Pattern Recognition
0
MindShot: A Few-Shot Brain Decoding Framework via Transferring Cross-Subject Prior and Distilling Frequency Domain Knowledge KBS
Title CN MindShot：一种通过跨被试先验迁移和频域知识蒸馏实现的少样本脑解码框架
Keywords 脑解码 · fMRI · 少样本学习 · 知识蒸馏 · 跨模态重建
Summary 本文提出了一种名为MindShot的两阶段少样本脑解码框架，通过多被试预训练（MSP）和基于傅里叶变换的知识蒸馏（FKD）来降低个体差异并提升对新被试的适应能力。在仅使用1.8%数据的情况下实现了83.6%的CLIP准确率，显著降低了fMRI数据需求，推动了临床场景下的实用化脑解码。
Reason 该论文研究的是基于fMRI信号的脑解码任务，目标是从大脑活动中重建视觉刺激，属于神经科学与视觉重建的交叉领域。尽管使用了多模态对比学习和知识蒸馏等技术，但其核心任务与文档图像理解（DIU）无关，不涉及OCR、布局分析、表格识别或文档VQA等任何DIU核心问题。方法也无法直接迁移用于解决文档中的幻觉、密集文本处理或复杂排版理解等痛点。
Abstract Aiming to reconstruct visual stimuli from brain signals, brain decoding has recently made significant progress using functional magnetic resonance imaging (fMRI). However, it still has challenging issues such as substantial individual differences and high data collection costs. To simplify these problems, most methods adopt the per-subject-per-model paradigm, but this greatly limits their applications. In this paper, we design a few-shot brain decoding setting specifically for potential clinical scenarios and propose a novel two-stage decoding framework named MindShot, comprising a Multi-Subject Pretraining (MSP) stage and Fourier-based cross-subject Knowledge Distillation (FKD) stage. Firstly, a MSP framework based on multi-modal contrastive learning is constructed to mine the cross-subject prior. Secondly, the FKD is presented to decrease inter-individual differences while improving the decoding adaptability to new individuals. Our approach achieves high semantic fidelity in visual reconstruction on the largest dataset and has the potential to reduce scanning time by up to 99%. Remarkably, MindShot achieves a CLIP accuracy of 83.6% using only 1.8% of the fMRI-image pairs, surpassing the 77.4% accuracy of the method trained on the entire NSD dataset. This makes it feasible to train large-scale brain decoding frameworks that require less data, facilitating practical applications. The code is available at https://github.com/JSinBUPT/MindShot.
Authors Shuai Jiang, Zhu Meng, Haiwen Li, Delong Liu, Fei Su, Zhicheng Zhao
Comment Accepted by KBS
Categories Computer Vision and Pattern Recognition
0
DeltaDeno: Zero-Shot Anomaly Generation via Delta-Denoising Attribution
Title CN DeltaDeno：通过Delta去噪归因实现零样本异常生成
Keywords 异常生成 · 扩散模型 · 零样本学习 · 注意力归因 · 图像编辑
Summary 本文提出了一种无需训练的零样本异常生成方法DeltaDeno，通过对比两个扩散分支在极小提示对下的去噪差异，累积每步去噪的‘delta’来生成特定图像的异常区域定位图，并指导潜在空间中的局部重绘以生成逼真的缺陷。方法引入了词级提示优化和空间注意力偏置以增强控制性与稳定性。实验表明其在下游异常检测任务中表现良好。
Reason 该论文属于图像生成领域，研究的是在无真实异常样本的情况下通过扩散模型生成图像异常，主要应用于工业缺陷检测等视觉任务。其核心方法聚焦于扩散模型中的注意力归因与提示词微调，用于定位并编辑图像局部区域。然而，该工作与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版解析、细粒度文本定位或长文档推理无直接关联，且难以迁移至以语义和结构理解为核心的文档场景。
Abstract Anomaly generation is often framed as few-shot fine-tuning with anomalous samples, which contradicts the scarcity that motivates generation and tends to overfit category priors. We tackle the setting where no real anomaly samples or training are available. We propose Delta-Denoising (DeltaDeno), a training-free zero-shot anomaly generation method that localizes and edits defects by contrasting two diffusion branches driven by a minimal prompt pair under a shared schedule. By accumulating per-step denoising deltas into an image-specific localization map, we obtain a mask to guide the latent inpainting during later diffusion steps and preserve the surrounding context while generating realistic local defects. To improve stability and control, DeltaDeno performs token-level prompt refinement that aligns shared content and strengthens anomaly tokens, and applies a spatial attention bias restricted to anomaly tokens in the predicted region. Experiments on public datasets show that DeltaDeno achieves great generation, realism and consistent gains in downstream detection performance. Code will be made publicly available.
Authors Chaoran Xu, Chengkan Lv, Qiyu Chen, Yunkang Cao, Feng Zhang, Zhengtao Zhang
Categories Computer Vision and Pattern Recognition
0
Colo-ReID: Discriminative Representation Embedding with Meta-learning for Colonoscopic Polyp Re-Identification
Title CN Colo-ReID：基于元学习的结肠镜息肉重识别判别性表示嵌入
Keywords 元学习 · 医学图像分析 · 目标重识别 · 域适应 · 结肠镜
Summary 本文提出Colo-ReID方法，结合元学习策略和动态调节机制MLR，用于提升结肠镜息肉在不同视角和设备下的重识别性能。实验表明其在mAP指标上优于现有方法。然而该工作局限于医疗图像应用，不具备向文档理解领域的迁移价值。
Reason 该论文属于医学图像分析领域的特定应用（结肠镜息肉重识别），研究任务为生物医学图像中的目标重识别（Re-ID），与文档图像理解（DIU）或DocVQA等核心任务无直接关联。其方法虽涉及元学习和表示学习，但针对的是跨视角、跨设备的医学成像域适应问题，无法迁移用于解决文档图像中的OCR幻觉、复杂排版或长文档推理等典型痛点，属于‘无关领域’。
Abstract Colonoscopic Polyp Re-Identification aims to match the same polyp from a large gallery with images from different views taken using different cameras and plays an important role in the prevention and treatment of colorectal cancer. However, traditional methods for object ReID directly adopting CNN models trained on the ImageNet dataset usually produce unsatisfactory retrieval performance on colonoscopic datasets due to the large domain gap. Additionally, these methods neglect to explore the potential of self-discrepancy among intra-class or inter-class relations in the colonoscopic polyp dataset, which remains an open research problem in the medical community. To solve this dilemma, we propose a simple but effective training method named Colo-ReID, which can help our model learn more general and discriminative knowledge based on the meta-learning strategy in scenarios with fewer samples. Based on this, a dynamic Meta-Learning Regulation mechanism called MLR is introduced to further boost the performance of polyp re-identification. Our experimental results show that Colo-ReID consistently outperforms second-best method in terms of mAP performance by +2.3% on polyp re-identification task. Our source code is also publicly available at https://github.com/JeremyXSC/Colo-ReID.
Authors Suncheng Xiang, Chengfeng Zhou, Zhengjie Zhang, Shilun Cai, Dahong Qian
Categories Computer Vision and Pattern Recognition
0
Q-REAL: Towards Realism and Plausibility Evaluation for AI-Generated Content
Title CN Q-REAL：面向人工智能生成内容的真实性与合理性评估
Keywords AI生成内容评估 · 真实性 · 合理性 · 多模态大模型 · 细粒度标注
Summary 本文提出了Q-Real数据集，用于对AI生成图像在真实性和合理性两个维度上进行细粒度评估。数据集包含3088张图像，标注了实体位置、判断问题及属性描述，并设计了基于多模态大模型的评估任务（判断与推理定位）。同时提出一个微调框架以提升MLLM在此任务上的表现。研究主要服务于图像生成质量评测，而非文档理解。
Reason 该论文聚焦于AI生成图像的质量评估，属于图像生成评价领域，核心任务是构建针对‘真实性’和‘合理性’的细粒度评测数据集与基准。虽然使用了多模态大模型（MLLM）进行判断与定位，但其目标并非解决文档图像理解中的OCR幻觉、复杂排版、长文本推理等核心问题，且应用场景为通用图像生成评估，与文档结构、语义解析无关。属于无关领域应用，不符合DIU本题或可迁移方法的要求。
Abstract Quality assessment of AI-generated content is crucial for evaluating model capability and guiding model optimization. However, most existing quality assessment datasets and models provide only a single quality score, which is too coarse to offer targeted guidance for improving generative models. In current applications of AI-generated images, realism and plausibility are two critical dimensions, and with the emergence of unified generation-understanding models, fine-grained evaluation along these dimensions becomes especially effective for improving generative performance. Therefore, we introduce Q-Real, a novel dataset for fine-grained evaluation of realism and plausibility in AI-generated images. Q-Real consists of 3,088 images generated by popular text-to-image models. For each image, we annotate the locations of major entities and provide a set of judgment questions and attribution descriptions for these along the dimensions of realism and plausibility. Considering that recent advances in multi-modal large language models (MLLMs) enable fine-grained evaluation of AI-generated images, we construct Q-Real Bench to evaluate them on two tasks: judgment and grounding with reasoning. Finally, to enhance MLLM capabilities, we design a fine-tuning framework and conduct experiments on multiple MLLMs using our dataset. Experimental results demonstrate the high quality and significance of our dataset and the comprehensiveness of the benchmark. Dataset and code will be released upon publication.
Authors Shushi Wang, Zicheng Zhang, Chunyi Li, Wei Wang, Liya Ma, Fengjiao Chen, Xiaoyu Li, Xuezhi Cao, Guangtao Zhai, Xiaohong Liu
Categories Computer Vision and Pattern Recognition
0
Warm Diffusion: Recipe for Blur-Noise Mixture Diffusion Models
Title CN 温热扩散：模糊-噪声混合扩散模型的配方
Keywords 扩散模型 · 图像生成 · 模糊-噪声混合 · 频谱分析 · 去噪
Summary 本文提出了Warm Diffusion，一种结合模糊和噪声的混合扩散模型，通过控制模糊与噪声的比例来统一热扩散与冷扩散的优点，利用频谱分析解耦去噪与去模糊过程，从而提升图像生成质量。实验表明该方法在多个基准上有效。
Reason 该论文研究的是生成模型中的扩散过程改进，属于纯图像生成领域，提出的方法针对通用图像生成任务中的模糊与噪声处理，并未涉及文档结构、文本理解或OCR相关问题。其核心方法难以迁移到文档图像理解（DIU）中解决OCR幻觉、复杂排版或长文档推理等关键痛点。
Abstract Diffusion probabilistic models have achieved remarkable success in generative tasks across diverse data types. While recent studies have explored alternative degradation processes beyond Gaussian noise, this paper bridges two key diffusion paradigms: hot diffusion, which relies entirely on noise, and cold diffusion, which uses only blurring without noise. We argue that hot diffusion fails to exploit the strong correlation between high-frequency image detail and low-frequency structures, leading to random behaviors in the early steps of generation. Conversely, while cold diffusion leverages image correlations for prediction, it neglects the role of noise (randomness) in shaping the data manifold, resulting in out-of-manifold issues and partially explaining its performance drop. To integrate both strengths, we propose Warm Diffusion, a unified Blur-Noise Mixture Diffusion Model (BNMD), to control blurring and noise jointly. Our divide-and-conquer strategy exploits the spectral dependency in images, simplifying score model estimation by disentangling the denoising and deblurring processes. We further analyze the Blur-to-Noise Ratio (BNR) using spectral analysis to investigate the trade-off between model learning dynamics and changes in the data manifold. Extensive experiments across benchmarks validate the effectiveness of our approach for image generation.
Authors Hao-Chien Hsueh, Chi-En Yen, Wen-Hsiao Peng, Ching-Chun Huang
Categories Computer Vision and Pattern Recognition
0
R-AVST: Empowering Video-LLMs with Fine-Grained Spatio-Temporal Reasoning in Complex Audio-Visual Scenarios AAAI 2026
Title CN R-AVST：在复杂音视频场景中通过细粒度时空推理增强视频大语言模型
Keywords 视频理解 · 音视频多模态 · 细粒度定位 · 强化学习 · 时空推理
Summary 本文提出了R-AVST数据集，用于复杂音视频场景下的细粒度时空推理，并构建了配套的问答任务。同时提出AVST-Zero模型，基于强化学习直接优化多维奖励，无需中间监督。实验表明该方法在音视频推理任务上表现优异。这是首个面向现实世界音视频时空推理的数据集。
Reason 该论文聚焦于视频理解中的音视频多模态推理，属于视频领域而非文档图像理解（DIU）。尽管提出了细粒度时空标注和推理方法，但其应用场景为真实世界的未剪辑视频，与文档图像中的排版、文本识别、OCR幻觉等核心问题无关。无法迁移解决DIU的关键痛点，如密集文本处理或复杂布局分析。
Abstract Recently, rapid advancements have been made in multimodal large language models (MLLMs), especially in video understanding tasks. However, current research focuses on simple video scenarios, failing to reflect the complex and diverse nature of real-world audio-visual events in videos. To bridge this gap, we firstly introduce R-AVST, a dataset for audio-visual reasoning featuring fine-grained spatio-temporal annotations. In constructing this, we design a pipeline consisting of LLM-based key object extraction, automatic spatial annotation and manual quality inspection, resulting in over 5K untrimmed videos with 27K objects across 100 types of audio-visual events. Building on this dataset, we define three core tasks for spatio-temporal reasoning in audio-visual scenes and generate more than 8K high-quality, evenly distributed question-answer pairs to effectively benchmark model performance. To further enhance reasoning, we propose AVST-Zero, a reinforcement learning-based model that avoids intermediate supervision, directly optimizing behavior via carefully designed multi-dimensional rewards. Extensive experiments validate the effectiveness of our R-AVST in advancing audio-visual spatio-temporal reasoning, upon which AVST-Zero demonstrates competitive performance compared to existing models. To the best of our knowledge, R-AVST is the first dataset designed for real-world audio-visual spatio-temporal reasoning, and AVST-Zero offers a novel perspective for tackling future challenges in this domain.
Authors Lu Zhu, Tiantian Geng, Yangye Chen, Teng Wang, Ping Lu, Feng Zheng
Comment Accepted by AAAI 2026. Project page: https://github.com/zhlllau/R-AVST
Categories Computer Vision and Pattern Recognition
0
Glass Surface Detection: Leveraging Reflection Dynamics in Flash/No-flash Imagery
Title CN 玻璃表面检测：利用闪光/无闪光图像中的反射动态
Keywords 玻璃表面检测 · 反射动态 · 闪光图像 · 计算机视觉 · NFGlassNet
Summary 本文提出NFGlassNet，通过分析闪光与无闪光图像对中玻璃反射动态变化来检测玻璃表面。设计了反射对比挖掘模块（RCMM）和反射引导注意力模块（RGAM），并构建了一个包含3.3K图像对的数据集。实验表明该方法优于现有技术。
Reason 该论文研究的是玻璃表面检测，属于通用计算机视觉任务，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。其核心问题、数据和方法均聚焦于自然场景中玻璃的物理特性建模，无法迁移解决OCR幻觉、复杂排版、密集文本或长文档推理等DIU核心痛点。
Abstract Glass surfaces are ubiquitous in daily life, typically appearing colorless, transparent, and lacking distinctive features. These characteristics make glass surface detection a challenging computer vision task. Existing glass surface detection methods always rely on boundary cues (e.g., window and door frames) or reflection cues to locate glass surfaces, but they fail to fully exploit the intrinsic properties of the glass itself for accurate localization. We observed that in most real-world scenes, the illumination intensity in front of the glass surface differs from that behind it, which results in variations in the reflections visible on the glass surface. Specifically, when standing on the brighter side of the glass and applying a flash towards the darker side, existing reflections on the glass surface tend to disappear. Conversely, while standing on the darker side and applying a flash towards the brighter side, distinct reflections will appear on the glass surface. Based on this phenomenon, we propose NFGlassNet, a novel method for glass surface detection that leverages the reflection dynamics present in flash/no-flash imagery. Specifically, we propose a Reflection Contrast Mining Module (RCMM) for extracting reflections, and a Reflection Guided Attention Module (RGAM) for fusing features from reflection and glass surface for accurate glass surface detection. For learning our network, we also construct a dataset consisting of 3.3K no-flash and flash image pairs captured from various scenes with corresponding ground truth annotations. Extensive experiments demonstrate that our method outperforms the state-of-the-art methods. Our code, model, and dataset will be available upon acceptance of the manuscript.
Authors Tao Yan, Hao Huang, Yiwei Lu, Zeyu Wang, Ke Xu, Yinghui Wang, Xiaojun Chang, Rynson W. H. Lau
Comment 13 pages, 12 figures
Categories Computer Vision and Pattern Recognition
0
Deep Learning Analysis of Prenatal Ultrasound for Identification of Ventriculomegaly
Title CN 基于深度学习的产前超声图像分析用于脑室扩张症的识别
Keywords 深度学习 · 超声图像 · 脑室扩张症 · 自监督学习 · Vision Transformer
Summary 本文提出了一种基于自监督预训练模型USF-MAE的深度学习方法，用于在产前超声图像中检测胎儿脑室扩张症。该模型在大规模超声数据上预训练后，在特定胎儿脑部图像数据集上进行微调，取得了优于多种基准模型的性能表现。通过Eigen-CAM可视化，验证了模型关注区域的临床合理性。
Reason 该论文属于医学图像分析领域，研究任务为胎儿超声图像中的疾病分类，与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。其方法聚焦于医学影像的二分类问题，并未涉及文本理解、布局分析、表格识别或OCR相关挑战，无法迁移解决文档图像中的幻觉、密集文本、复杂排版等核心痛点。
Abstract The proposed study aimed to develop a deep learning model capable of detecting ventriculomegaly on prenatal ultrasound images. Ventriculomegaly is a prenatal condition characterized by dilated cerebral ventricles of the fetal brain and is important to diagnose early, as it can be associated with an increased risk for fetal aneuploidies and/or underlying genetic syndromes. An Ultrasound Self-Supervised Foundation Model with Masked Autoencoding (USF-MAE), recently developed by our group, was fine-tuned for a binary classification task to distinguish fetal brain ultrasound images as either normal or showing ventriculomegaly. The USF-MAE incorporates a Vision Transformer encoder pretrained on more than 370,000 ultrasound images from the OpenUS-46 corpus. For this study, the pretrained encoder was adapted and fine-tuned on a curated dataset of fetal brain ultrasound images to optimize its performance for ventriculomegaly detection. Model evaluation was conducted using 5-fold cross-validation and an independent test cohort, and performance was quantified using accuracy, precision, recall, specificity, F1-score, and area under the receiver operating characteristic curve (AUC). The proposed USF-MAE model reached an F1-score of 91.76% on the 5-fold cross-validation and 91.78% on the independent test set, with much higher scores than those obtained by the baseline models by 19.37% and 16.15% compared to VGG-19, 2.31% and 2.56% compared to ResNet-50, and 5.03% and 11.93% compared to ViT-B/16, respectively. The model also showed a high mean test precision of 94.47% and an accuracy of 97.24%. The Eigen-CAM (Eigen Class Activation Map) heatmaps showed that the model was focusing on the ventricle area for the diagnosis of ventriculomegaly, which has explainability and clinical plausibility.
Authors Youssef Megahed, Inok Lee, Robin Ducharme, Aylin Erman, Olivier X. Miguel, Kevin Dick, Adrian D. C. Chan, Steven Hawken, Mark Walker, Felipe Moretti
Comment 13 pages, 7 figures, 3 tables
Categories Computer Vision and Pattern Recognition
0
Parts-Mamba: Augmenting Joint Context with Part-Level Scanning for Occluded Human Skeleton
Title CN Parts-Mamba：通过部件级扫描增强联合上下文以应对遮挡的人体骨架
Keywords 骨架动作识别 · 遮挡处理 · GCN · Mamba · 上下文建模
Summary 本文提出Parts-Mamba模型，结合图卷积网络和Mamba架构，通过部件级扫描和部件-整体融合模块来增强在人体部分被遮挡情况下的动作识别性能，在NTU数据集上取得了显著提升。但其应用场景和方法论均远离文档理解领域。
Reason 该论文研究的是人体骨架动作识别中的遮挡问题，属于视频理解与动作识别领域，与文档图像理解（DIU）或DocVQA等任务无直接关联。其方法聚焦于GCN与Mamba结合用于捕捉被遮挡人体部位的上下文，无法迁移解决OCR幻觉、复杂排版、长文档推理等DIU核心痛点。
Abstract Skeleton action recognition involves recognizing human action from human skeletons. The use of graph convolutional networks (GCNs) has driven major advances in this recognition task. In real-world scenarios, the captured skeletons are not always perfect or complete because of occlusions of parts of the human body or poor communication quality, leading to missing parts in skeletons or videos with missing frames. In the presence of such non-idealities, existing GCN models perform poorly due to missing local context. To address this limitation, we propose Parts-Mamba, a hybrid GCN-Mamba model designed to enhance the ability to capture and maintain contextual information from distant joints. The proposed Parts-Mamba model effectively captures part-specific information through its parts-specific scanning feature and preserves non-neighboring joint context via a parts-body fusion module. Our proposed model is evaluated on the NTU RGB+D 60 and NTU RGB+D 120 datasets under different occlusion settings, achieving up to 12.9% improvement in accuracy.
Authors Tianyi Shen, Huijuan Xu, Nilesh Ahuja, Omesh Tickoo, Philip Shin, Vijaykrishnan Narayanan
Categories Computer Vision and Pattern Recognition
0
JigsawComm: Joint Semantic Feature Encoding and Transmission for Communication-Efficient Cooperative Perception
Title CN JigsawComm：面向通信高效的协同感知的联合语义特征编码与传输
Keywords 协同感知 · 语义通信 · 特征压缩 · 自动驾驶 · 多智能体系统
Summary 本文提出JigsawComm，一种用于多智能体协同感知的通信高效框架。通过联合优化语义特征提取与传输策略，利用特征效用估计器生成元效用图，并据此选择最具贡献的特征进行传输，实现带宽压缩500倍以上的同时保持甚至提升感知精度。方法聚焦于3D点云或图像传感器数据的协同处理，适用于自动驾驶等场景。
Reason 该论文研究多智能体协同感知（如自动驾驶场景），属于3D视觉与自动驾驶领域，与文档图像理解（DIU）或视觉语言模型在文档任务中的应用无直接关联。尽管其提出了一种语义感知的特征压缩与传输机制，但应用场景为车载传感器数据融合，迁移到文档理解中的OCR幻觉、细粒度定位或复杂排版解析等问题极为困难，且未涉及文本模态或VLM架构改进。
Abstract Multi-agent cooperative perception (CP) promises to overcome the inherent occlusion and sensing-range limitations of single-agent systems (e.g., autonomous driving). However, its practicality is severely constrained by the limited communication bandwidth. Existing approaches attempt to improve bandwidth efficiency via compression or heuristic message selection, without considering the semantic relevance or cross-agent redundancy of sensory data. We argue that a practical CP system must maximize the contribution of every transmitted bit to the final perception task, by extracting and transmitting semantically essential and non-redundant data. In this paper, we formulate a joint semantic feature encoding and transmission problem, which aims to maximize CP accuracy under limited bandwidth. To solve this problem, we introduce JigsawComm, an end-to-end trained, semantic-aware, and communication-efficient CP framework that learns to ``assemble the puzzle'' of multi-agent feature transmission. It uses a regularized encoder to extract semantically-relevant and sparse features, and a lightweight Feature Utility Estimator to predict the contribution of each agent's features to the final perception task. The resulting meta utility maps are exchanged among agents and leveraged to compute a provably optimal transmission policy, which selects features from agents with the highest utility score for each location. This policy inherently eliminates redundancy and achieves a scalable $\mathcal{O}(1)$ communication cost as the number of agents increases. On the benchmarks OPV2V and DAIR-V2X, JigsawComm reduces the total data volume by up to $>$500$\times$ while achieving matching or superior accuracy compared to state-of-the-art methods.
Authors Chenyi Wang, Zhaowei Li, Ming F. Li, Wujie Wen
Categories Computer Vision and Pattern Recognition
0
Show Me: Unifying Instructional Image and Video Generation with Diffusion Models WACV 2026
Title CN 向我展示：使用扩散模型统一指令性图像与视频生成
Keywords 扩散模型 · 视频生成 · 图像编辑 · 结构一致性 · 时间连贯性
Summary 本文提出ShowMe框架，通过激活视频扩散模型的空间与时间组件，统一实现指令引导的图像编辑和视频生成。引入结构一致性和运动一致性奖励来提升生成结果的质量。实验表明该方法在多个基准上优于专用模型。
Reason 该论文研究的是指令驱动的图像编辑与视频生成任务，属于纯图像生成与视频理解领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。其提出的方法聚焦于结构与运动一致性奖励以提升视觉内容生成质量，无法迁移解决OCR幻觉、密集文本处理、复杂排版理解或长文档推理等DIU关键痛点。
Abstract Generating visual instructions in a given context is essential for developing interactive world simulators. While prior works address this problem through either text-guided image manipulation or video prediction, these tasks are typically treated in isolation. This separation reveals a fundamental issue: image manipulation methods overlook how actions unfold over time, while video prediction models often ignore the intended outcomes. To this end, we propose ShowMe, a unified framework that enables both tasks by selectively activating the spatial and temporal components of video diffusion models. In addition, we introduce structure and motion consistency rewards to improve structural fidelity and temporal coherence. Notably, this unification brings dual benefits: the spatial knowledge gained through video pretraining enhances contextual consistency and realism in non-rigid image edits, while the instruction-guided manipulation stage equips the model with stronger goal-oriented reasoning for video prediction. Experiments on diverse benchmarks demonstrate that our method outperforms expert models in both instructional image and video generation, highlighting the strength of video diffusion models as a unified action-object state transformer.
Authors Yujiang Pu, Zhanbo Huang, Vishnu Boddeti, Yu Kong
Comment Accepted by WACV 2026
Categories Computer Vision and Pattern Recognition
0
Learning to Drive Anywhere with Model-Based Reannotation
Title CN 使用基于模型的重注释实现任意场景下的驾驶
Keywords 视觉导航 · 机器人 · 模型重标注 · 无监督数据 · 策略学习
Summary 本文提出了一种名为Model-Based ReAnnotation（MBRA）的框架，利用短视距的基于模型的专家系统对大规模被动采集的数据（如远程操作数据和YouTube视频）进行动作重标注，并将这些高质量标注用于训练一个长视距的视觉导航策略LogoNav。实验表明该策略可在未见过的室内外环境中实现超过300米的鲁棒导航，并在多城市真实机器人平台上验证了泛化能力。
Reason 该论文属于具身智能与机器人领域，研究的是视觉导航策略和机器人控制，尽管使用了视觉数据和模型重标注技术，但其核心任务与文档图像理解（DIU）或文档视觉问答（DocVQA）无关。方法聚焦于长距离导航策略学习和现实世界机器人部署，无法迁移到解决OCR幻觉、密集文本识别、复杂版式分析或文档推理等DIU核心痛点。
Abstract Developing broadly generalizable visual navigation policies for robots is a significant challenge, primarily constrained by the availability of large-scale, diverse training data. While curated datasets collected by researchers offer high quality, their limited size restricts policy generalization. To overcome this, we explore leveraging abundant, passively collected data sources, including large volumes of crowd-sourced teleoperation data and unlabeled YouTube videos, despite their potential for lower quality or missing action labels. We propose Model-Based ReAnnotation (MBRA), a framework that utilizes a learned short-horizon, model-based expert model to relabel or generate high-quality actions for these passive datasets. This relabeled data is then distilled into LogoNav, a long-horizon navigation policy conditioned on visual goals or GPS waypoints. We demonstrate that LogoNav, trained using MBRA-processed data, achieves state-of-the-art performance, enabling robust navigation over distances exceeding 300 meters in previously unseen indoor and outdoor environments. Our extensive real-world evaluations, conducted across a fleet of robots (including quadrupeds) in six cities on three continents, validate the policy's ability to generalize and navigate effectively even amidst pedestrians in crowded settings.
Authors Noriaki Hirose, Lydia Ignatova, Kyle Stachowicz, Catherine Glossop, Sergey Levine, Dhruv Shah
Comment 9 pages, 8 figures, 6 tables
Categories Computer Vision and Pattern Recognition
0
QAL: A Loss for Recall Precision Balance in 3D Reconstruction WACV 2026
Title CN QAL：一种用于3D重建中召回率与精确率平衡的损失函数
Keywords 3D重建 · 损失函数 · 召回率与精确率 · 体积学习 · WACV
Summary 本文提出了一种名为Quality-Aware Loss（QAL）的新损失函数，用于改进3D重建中的体积学习任务。相比传统的Chamfer Distance（CD）和Earth Mover's Distance（EMD），QAL通过引入覆盖加权的最近邻项和未覆盖真实点吸引项，显式解耦了召回率与精确率，并可在训练中调节二者平衡。实验表明，QAL在多个模型和数据集上均能稳定提升覆盖率，尤其在恢复细小结构方面表现更优，并且在机器人抓取任务中也表现出更高的实用性。
Reason 该论文研究的是3D视觉任务中的损失函数设计，属于3D重建和机器人操作领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。尽管其提出的损失函数在理论上可能具有通用性，但其核心问题（如薄结构恢复、抓取评分提升）和应用场景（如GraspNet）完全偏离文档理解的核心痛点，如OCR幻觉、复杂排版、长文本推理等。无法迁移其方法至DIU领域。
Abstract Volumetric learning underpins many 3D vision tasks such as completion, reconstruction, and mesh generation, yet training objectives still rely on Chamfer Distance (CD) or Earth Mover's Distance (EMD), which fail to balance recall and precision. We propose Quality-Aware Loss (QAL), a drop-in replacement for CD/EMD that combines a coverage-weighted nearest-neighbor term with an uncovered-ground-truth attraction term, explicitly decoupling recall and precision into tunable components. Across diverse pipelines, QAL achieves consistent coverage gains, improving by an average of +4.3 pts over CD and +2.8 pts over the best alternatives. Though modest in percentage, these improvements reliably recover thin structures and under-represented regions that CD/EMD overlook. Extensive ablations confirm stable performance across hyperparameters and across output resolutions, while full retraining on PCN and ShapeNet demonstrates generalization across datasets and backbones. Moreover, QAL-trained completions yield higher grasp scores under GraspNet evaluation, showing that improved coverage translates directly into more reliable robotic manipulation. QAL thus offers a principled, interpretable, and practical objective for robust 3D vision and safety-critical robotics pipelines
Authors Pranay Meshram, Yash Turkar, Kartikeya Singh, Praveen Raj Masilamani, Charuvahan Adhivarahan, Karthik Dantu
Comment Accepted to WACV 2026. Camera-ready version to appear
Categories Computer Vision and Pattern Recognition
0
Target-Bench: Can World Models Achieve Mapless Path Planning with Semantic Targets?
Title CN Target-Bench：世界模型能否实现以语义目标为导向的无地图路径规划？
Keywords 世界模型 · 无地图路径规划 · 语义目标 · 视频生成 · 机器人
Summary 本文提出了Target-Bench，首个用于评估世界模型在真实环境中基于语义目标进行无地图路径规划能力的基准。该基准包含450段由机器人采集的视频序列，涵盖45个语义类别，并提供基于SLAM的真实轨迹。作者通过恢复生成视频中的相机运动，使用五个指标评估了Sora 2、Veo 3.1和Wan系列等先进模型的表现，发现当前模型性能仍然有限。实验表明，在仅325个场景上微调一个开源5B参数模型即可显著提升性能。
Reason 该论文属于具身智能与机器人领域，研究的是世界模型在机器人路径规划中的应用，任务和方法均与文档图像理解（DIU）无关。尽管使用了视觉输入，但其核心关注点是视频生成与SLAM轨迹评估，无法迁移到文档理解中的OCR幻觉、复杂排版或长文档推理等痛点问题。
Abstract While recent world models generate highly realistic videos, their ability to perform robot path planning remains unclear and unquantified. We introduce Target-Bench, the first benchmark specifically designed to evaluate world models on mapless path planning toward semantic targets in real-world environments. Target-Bench provides 450 robot-collected video sequences spanning 45 semantic categories with SLAM-based ground truth trajectories. Our evaluation pipeline recovers camera motion from generated videos and measures planning performance using five complementary metrics that quantify target-reaching capability, trajectory accuracy, and directional consistency. We evaluate state-of-the-art models including Sora 2, Veo 3.1, and the Wan series. The best off-the-shelf model (Wan2.2-Flash) achieves only 0.299 overall score, revealing significant limitations in current world models for robotic planning tasks. We show that fine-tuning an open-source 5B-parameter model on only 325 scenarios from our dataset achieves 0.345 overall score -- an improvement of more than 400% over its base version (0.066) and 15% higher than the best off-the-shelf model. We will open-source the code and dataset.
Authors Dingrui Wang, Hongyuan Ye, Zhihao Liang, Zhexiao Sun, Zhaowei Lu, Yuchen Zhang, Yuyu Zhao, Yuan Gao, Marvin Seegert, Finn Schäfer, Haotong Qin, Wei Li, Luigi Palmieri, Felix Jahncke, Mattia Piccinini, Johannes Betz
Comment 10 pages
Categories Computer Vision and Pattern Recognition
0
Deepfake Geography: Detecting AI-Generated Satellite Images
Title CN 深度伪造地理：检测AI生成的卫星图像
Keywords deepfake detection · satellite imagery · Vision Transformer · CNN · image authenticity
Summary 本文研究了使用CNN和Vision Transformer（ViT）来检测AI生成的卫星图像，发现ViT在准确率和鲁棒性上显著优于CNN，主要得益于其对长距离依赖和全局语义结构的建模能力。作者还通过Grad-CAM和注意力归因方法增强了模型的可解释性，揭示了模型关注的伪造特征。
Reason 该论文研究的是AI生成卫星图像的检测问题，属于图像真实性验证领域，与文档图像理解（DIU）的核心任务如DocVQA、布局分析、表格识别等无直接关联。尽管使用了ViT和CNN进行图像分类，并探讨了模型可解释性，但其方法和目标均无法迁移到解决OCR幻觉、密集文本识别、复杂排版或长文档推理等DIU痛点。属于无关领域应用。
Abstract The rapid advancement of generative models such as StyleGAN2 and Stable Diffusion poses a growing threat to the authenticity of satellite imagery, which is increasingly vital for reliable analysis and decision-making across scientific and security domains. While deepfake detection has been extensively studied in facial contexts, satellite imagery presents distinct challenges, including terrain-level inconsistencies and structural artifacts. In this study, we conduct a comprehensive comparison between Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) for detecting AI-generated satellite images. Using a curated dataset of over 130,000 labeled RGB images from the DM-AER and FSI datasets, we show that ViTs significantly outperform CNNs in both accuracy (95.11 percent vs. 87.02 percent) and overall robustness, owing to their ability to model long-range dependencies and global semantic structures. We further enhance model transparency using architecture-specific interpretability methods, including Grad-CAM for CNNs and Chefer's attention attribution for ViTs, revealing distinct detection behaviors and validating model trustworthiness. Our results highlight the ViT's superior performance in detecting structural inconsistencies and repetitive textural patterns characteristic of synthetic imagery. Future work will extend this research to multispectral and SAR modalities and integrate frequency-domain analysis to further strengthen detection capabilities and safeguard satellite imagery integrity in high-stakes applications.
Authors Mansur Yerzhanuly
Comment 18 pages, 8 figures
Categories Computer Vision and Pattern Recognition
0
Latent Dirichlet Transformer VAE for Hyperspectral Unmixing with Bundled Endmembers
Title CN 用于高光谱解混的捆绑端元潜在狄利克雷变换器VAE
Keywords 高光谱解混 · 变分自编码器 · Transformer · Dirichlet先验 · 端元捆绑
Summary 本文提出了一种名为LDVAE-T的模型，用于高光谱图像解混。该模型结合Transformer与具有Dirichlet先验的变分自编码器，在潜空间中施加物理意义约束，以实现更准确的丰度估计和端元提取。通过将材料表示为具有均值和协方差结构的‘捆绑端元’，模型能够捕捉光谱变异性，并在多个基准数据集上优于现有方法。
Reason 该论文研究的是高光谱图像解混问题，属于遥感图像分析领域，与文档图像理解（DIU）或文档视觉问答（DocVQA）无直接关联。其核心技术——基于Dirichlet先验的变分自编码器和Transformer用于光谱建模——虽然在物理约束建模上有创新，但无法迁移到解决OCR幻觉、复杂排版理解、细粒度定位等文档理解核心痛点。
Abstract Hyperspectral images capture rich spectral information that enables per-pixel material identification; however, spectral mixing often obscures pure material signatures. To address this challenge, we propose the Latent Dirichlet Transformer Variational Autoencoder (LDVAE-T) for hyperspectral unmixing. Our model combines the global context modeling capabilities of transformer architectures with physically meaningful constraints imposed by a Dirichlet prior in the latent space. This prior naturally enforces the sum-to-one and non-negativity conditions essential for abundance estimation, thereby improving the quality of predicted mixing ratios. A key contribution of LDVAE-T is its treatment of materials as bundled endmembers, rather than relying on fixed ground truth spectra. In the proposed method our decoder predicts, for each endmember and each patch, a mean spectrum together with a structured (segmentwise) covariance that captures correlated spectral variability. Reconstructions are formed by mixing these learned bundles with Dirichlet-distributed abundances garnered from a transformer encoder, allowing the model to represent intrinsic material variability while preserving physical interpretability. We evaluate our approach on three benchmark datasets, Samson, Jasper Ridge, and HYDICE Urban and show that LDVAE-T consistently outperforms state-of-the-art models in abundance estimation and endmember extraction, as measured by root mean squared error and spectral angle distance, respectively.
Authors Giancarlo Giannetti, Faisal Z. Qureshi
Categories Computer Vision and Pattern Recognition
0
DiffBreak: Is Diffusion-Based Purification Robust? NeurIPS 2025
Title CN DiffBreak：基于扩散的净化是否具有鲁棒性？
Keywords 对抗样本 · 扩散模型 · 鲁棒性 · 梯度攻击 · 去噪净化
Summary 本文质疑了基于扩散模型的对抗样本净化方法（DBP）的鲁棒性，指出其在面对梯度攻击时会失效，并揭示了现有评估协议的缺陷。作者提出了DiffBreak工具包以准确计算通过DBP的梯度，并设计了多数投票策略和系统性扰动攻击来验证DBP的脆弱性，最终挑战了DBP作为有效防御手段的合理性。
Reason 该论文研究的是对抗样本防御中的扩散模型净化方法（DBP）的鲁棒性问题，属于计算机视觉中对抗攻击与防御的安全方向。尽管涉及扩散模型和图像处理，但其核心任务与文档图像理解（DIU）无关，不解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点。此外，该工作属于安全/鲁棒性分析领域，且未提出可迁移至文档理解场景的通用架构或推理机制，属于无关领域。
Abstract Diffusion-based purification (DBP) has become a cornerstone defense against adversarial examples (AEs), regarded as robust due to its use of diffusion models (DMs) that project AEs onto the natural data manifold. We refute this core claim, theoretically proving that gradient-based attacks effectively target the DM rather than the classifier, causing DBP's outputs to align with adversarial distributions. This prompts a reassessment of DBP's robustness, accrediting it two critical factors: inaccurate gradients and improper evaluation protocols that test only a single random purification of the AE. We show that when accounting for stochasticity and resubmission risk, DBP collapses. To support this, we introduce DiffBreak, the first reliable toolkit for differentiation through DBP, eliminating gradient mismatches that previously further inflated robustness estimates. We also analyze the current defense scheme used for DBP where classification relies on a single purification, pinpointing its inherent invalidity. We provide a statistically grounded majority-vote (MV) alternative that aggregates predictions across multiple purified copies, showing partial but meaningful robustness gain. We then propose a novel adaptation of an optimization method against deepfake watermarking, crafting systemic perturbations that defeat DBP even under MV, challenging DBP's viability.
Authors Andre Kassis, Urs Hengartner, Yaoliang Yu
Comment Accepted to NeurIPS 2025
Categories Computer Vision and Pattern Recognition
0
SPIDER: Spatial Image CorresponDence Estimator for Robust Calibration
Title CN SPIDER：用于鲁棒标定的空间图像对应关系估计器
Keywords 图像匹配 · 空间对应 · 三维重建 · 视觉几何 · 基础模型
Summary 本文提出SPIDER框架，通过结合2D和3D信息实现跨域、大基线条件下的鲁棒图像匹配，并构建了一个新的图像匹配评测基准。该方法专注于通用场景的几何一致性匹配，适用于航拍、室内外环境等非文档图像。
Reason 该论文聚焦于通用视觉中的图像匹配与空间对应关系估计，属于三维视觉与几何重建领域，主要解决跨域、大视角变化下的特征匹配问题。其任务核心是2D-2D或结合3D几何的图像对齐，与文档图像理解（DIU）中的文本识别、布局分析、表格理解等任务无直接关联。尽管涉及高分辨率和复杂场景下的对应关系建模，但方法目标并非处理文档特有的密集文本、排版结构或语义推理，且未涉及多模态大模型（VLM）在文档上的应用或改进。因此属于无关领域，不符合DIU方法论创新的关注范畴。
Abstract Reliable image correspondences form the foundation of vision-based spatial perception, enabling recovery of 3D structure and camera poses. However, unconstrained feature matching across domains such as aerial, indoor, and outdoor scenes remains challenging due to large variations in appearance, scale and viewpoint. Feature matching has been conventionally formulated as a 2D-to-2D problem; however, recent 3D foundation models provides spatial feature matching properties based on two-view geometry. While powerful, we observe that these spatially coherent matches often concentrate on dominant planar regions, e.g., walls or ground surfaces, while being less sensitive to fine-grained geometric details, particularly under large viewpoint changes. To better understand these trade-offs, we first perform linear probe experiments to evaluate the performance of various vision foundation models for image matching. Building on these insights, we introduce SPIDER, a universal feature matching framework that integrates a shared feature extraction backbone with two specialized network heads for estimating both 2D-based and 3D-based correspondences from coarse to fine. Finally, we introduce an image-matching evaluation benchmark that focuses on unconstrained scenarios with large baselines. SPIDER significantly outperforms SoTA methods, demonstrating its strong ability as a universal image-matching method.
Authors Zhimin Shao, Abhay Yadav, Rama Chellappa, Cheng Peng
Categories Computer Vision and Pattern Recognition
0
Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings
Title CN 使用错误严重性映射进行机器翻译的细粒度奖励优化
Keywords 机器翻译 · 强化学习 · 细粒度奖励 · 质量估计 · xCOMET
Summary 本文提出一种基于错误严重性级别的细粒度奖励优化方法，利用xCOMET作为token-level奖励模型，在机器翻译任务中通过强化学习提升翻译质量。实验表明，相比句子级奖励，词元级奖励能更稳定地提高翻译性能。
Reason 该论文属于纯自然语言处理领域的机器翻译研究，使用细粒度奖励信号优化翻译模型。尽管其强化学习方法在NLP中有一定创新，但完全未涉及视觉或文档模态，与文档图像理解（DIU）的核心问题如OCR、布局分析、视觉文本对齐等无关。属于负面清单中的‘无关领域’，无法迁移解决DIU中的幻觉、复杂排版或高分辨率推理等痛点。
Abstract Reinforcement learning (RL) has been proven to be an effective and robust method for training neural machine translation systems, especially when paired with powerful reward models that accurately assess translation quality. However, most research has focused on RL methods that use sentence-level feedback, leading to inefficient learning signals due to the reward sparsity problem -- the model receives a single score for the entire sentence. To address this, we propose a novel approach that leverages fine-grained, token-level quality assessments along with error severity levels using RL methods. Specifically, we use xCOMET, a state-of-the-art quality estimation system, as our token-level reward model. We conduct experiments on small and large translation datasets with standard encoder-decoder and large language models-based machine translation systems, comparing the impact of sentence-level versus fine-grained reward signals on translation quality. Our results show that training with token-level rewards improves translation quality across language pairs over baselines according to both automatic and human evaluation. Furthermore, token-level reward optimization improves training stability, evidenced by a steady increase in mean rewards over training epochs.
Authors Miguel Moura Ramos, Tomás Almeida, Daniel Vareta, Filipe Azevedo, Sweta Agrawal, Patrick Fernandes, André F. T. Martins
Categories Computation and Language
0
Do LLMs produce texts with "human-like" lexical diversity?
Title CN 大语言模型生成的文本具有“类人”的词汇多样性吗？
Keywords 词汇多样性 · 大语言模型 · 文本生成 · 人类相似性 · 语言评估
Summary 该研究比较了多个ChatGPT模型生成文本与人类书写文本在六个词汇多样性维度上的差异，发现所有ChatGPT模型生成的文本均显著不同于人类文本，且新版本模型（如ChatGPT-4.5）反而更不‘类人’。研究结论主要对语言教学和文本生成评估有启示。
Reason 该论文研究的是大语言模型生成文本的词汇多样性与人类写作的差异，属于纯NLP领域，且聚焦于语言风格分析，不涉及任何视觉、文档结构或图文多模态理解任务。与文档图像理解（DIU）的核心痛点如OCR幻觉、复杂排版、细粒度定位等完全无关，无法迁移用于提升VLM在文档理解中的性能。
Abstract The degree to which large language models (LLMs) produce writing that is truly human-like remains unclear despite the extensive empirical attention that this question has received. The present study addresses this question from the perspective of lexical diversity. Specifically, the study investigates patterns of lexical diversity in LLM-generated texts from four ChatGPT models (ChatGPT-3.5, ChatGPT-4, ChatGPT-o4 mini, and ChatGPT-4.5) in comparison with texts written by L1 and L2 English participants (n = 240) across four education levels. Six dimensions of lexical diversity were measured in each text: volume, abundance, variety-repetition, evenness, disparity, and dispersion. Results from one-way MANOVAs, one-way ANOVAs, and Support Vector Machines revealed that the ChatGPT-generated texts differed significantly from human-written texts for each variable, with ChatGPT-o4 mini and ChatGPT-4.5 differing the most. Within these two groups, ChatGPT-4.5 demonstrated higher levels of lexical diversity than older models despite producing fewer tokens. The human writers' lexical diversity did not differ across subgroups (i.e., education, language status). Altogether, the results indicate that ChatGPT models do not produce human-like texts in relation to lexical diversity, and the newer models produce less human-like text than older models. We discuss the implications of these results for language pedagogy and related applications.
Authors Kelly Kendro, Jeffrey Maloney, Scott Jarvis
Categories Computation and Language
0
AI use in American newspapers is widespread, uneven, and rarely disclosed
Title CN 美国报纸中人工智能的使用广泛、不均衡且很少披露
Keywords AI检测 · 新闻业 · AI生成内容 · 透明度 · 实证研究
Summary 该研究通过审计186,000篇美国报纸文章，发现约9%的内容部分或全部由AI生成，且多见于地方性媒体和评论类文章，但极少被披露。研究使用现有AI检测工具Pangram进行分析，强调了新闻业中AI使用透明度的缺失问题。
Reason 该论文属于新闻传播学领域的实证研究，聚焦于AI在新闻业中的应用现状与伦理问题，未提出任何技术方法或模型创新。其内容与文档图像理解（DIU）、视觉语言模型（VLM）、OCR、布局分析等核心技术无关，属于社会科学研究范畴，无法为DIU领域提供可迁移的方法论或技术工具。
Abstract AI is rapidly transforming journalism, but the extent of its use in published newspaper articles remains unclear. We address this gap by auditing a large-scale dataset of 186K articles from online editions of 1.5K American newspapers published in the summer of 2025. Using Pangram, a state-of-the-art AI detector, we discover that approximately 9% of newly-published articles are either partially or fully AI-generated. This AI use is unevenly distributed, appearing more frequently in smaller, local outlets, in specific topics such as weather and technology, and within certain ownership groups. We also analyze 45K opinion pieces from Washington Post, New York Times, and Wall Street Journal, finding that they are 6.4 times more likely to contain AI-generated content than news articles from the same publications, with many AI-flagged op-eds authored by prominent public figures. Despite this prevalence, we find that AI use is rarely disclosed: a manual audit of 100 AI-flagged articles found only five disclosures of AI use. Overall, our audit highlights the immediate need for greater transparency and updated editorial standards regarding the use of AI in journalism to maintain public trust.
Authors Jenna Russell, Marzena Karpinska, Destiny Akinode, Katherine Thai, Bradley Emi, Max Spero, Mohit Iyyer
Categories Computation and Language
0
PUCP-Metrix: A Comprehensive Open-Source Repository of Linguistic Metrics for Spanish to be submitted to EACL Demo track
Title CN PUCP-Metrix：一个面向西班牙语的综合性开源语言学指标库
Keywords 西班牙语 · 语言学指标 · 可读性分析 · 文本分析 · NLP工具
Summary 本文提出了PUCP-Metrix，一个包含182个语言学指标的开源资源库，适用于西班牙语文本的词汇多样性、句法语义复杂度、连贯性、心理语言学和可读性分析。在自动可读性评估和机器生成文本检测任务上验证了其有效性。尽管对西班牙语NLP有一定价值，但不涉及多模态或视觉文档理解。
Reason 该论文聚焦于西班牙语的语言学特征提取与文本可读性分析，属于小语种（非中英）NLP工具构建，且不涉及任何视觉、文档布局或跨模态理解内容。完全偏离文档图像理解（DIU）核心问题，属于负面清单中的‘小语种’和‘平行应用’范畴，无方法论迁移价值。
Abstract Linguistic features remain essential for interpretability and tasks involving style, structure, and readability, but existing Spanish tools offer limited coverage. We present PUCP-Metrix, an open-source repository of 182 linguistic metrics spanning lexical diversity, syntactic and semantic complexity, cohesion, psycholinguistics, and readability. PUCP-Metrix enables fine-grained, interpretable text analysis. We evaluate its usefulness on Automated Readability Assessment and Machine-Generated Text Detection, showing competitive performance compared to an existing repository and strong neural baselines. PUCP-Metrix offers a comprehensive, extensible resource for Spanish, supporting diverse NLP applications.
Authors Javier Alonso Villegas Luis, Marco Antonio Sobrevilla Cabezudo
Comment 1 figure, to be submitted to EACL Demo track
Categories Computation and Language
0
A new kid on the block: Distributional semantics predicts the word-specific tone signatures of monosyllabic words in conversational Taiwan Mandarin
Title CN 新来者：分布语义学预测对话式台湾国语单音节词的词特异性音调特征
Keywords 分布语义学 · 音调实现 · 口语处理 · 上下文化嵌入 · 汉语语音
Summary 本文通过语料库分析探讨了在自发性对话国语中，单音节词的音高轮廓如何受到词义的影响。研究使用广义加性模型分解音高轮廓，并发现即使控制多种变量后，词义仍显著影响声调实现，且上下文化嵌入可有效预测个体词的音高变化。
Reason 该论文研究的是语音语言学中的音调实现问题，属于计算语言学与语音学交叉领域，聚焦于口语中词汇意义对音高的影响。其任务和方法均与文档图像理解（DIU）、视觉-语言模型（VLM）、OCR、布局分析等完全无关，也不提供可用于解决DIU核心痛点（如OCR幻觉、复杂排版、细粒度定位）的通用技术路径。
Abstract We present a corpus-based investigation of how the pitch contours of monosyllabic words are realized in spontaneous conversational Mandarin, focusing on the effects of words' meanings. We used the generalized additive model to decompose a given observed pitch contour into a set of component pitch contours that are tied to different control variables and semantic predictors. Even when variables such as word duration, gender, speaker identity, tonal context, vowel height, and utterance position are controlled for, the effect of word remains a strong predictor of tonal realization. We present evidence that this effect of word is a semantic effect: word sense is shown to be a better predictor than word, and heterographic homophones are shown to have different pitch contours. The strongest evidence for the importance of semantics is that the pitch contours of individual word tokens can be predicted from their contextualized embeddings with an accuracy that substantially exceeds a permutation baseline. For phonetics, distributional semantics is a new kid on the block. Although our findings challenge standard theories of Mandarin tone, they fit well within the theoretical framework of the Discriminative Lexicon Model.
Authors Xiaoyun Jin, Mirjam Ernestus, R. Harald Baayen
Comment arXiv admin note: text overlap with arXiv:2409.07891
Categories Computation and Language
0
Testing Hypotheses from the Social Approval Theory of Online Hate: An Analysis of 110 Million Messages from Parler
Title CN 检验网络仇恨的社交认可理论假设：基于Parler平台1.1亿条消息的分析
Keywords 社交认可 · 网络仇恨 · 在线言论分析 · 计算社会科学 · 情感极化
Summary 本文基于Parler平台的1.1亿条消息，验证了Walther提出的网络仇恨社交认可理论，发现他人对极端仇恨评论的点赞数与后续发布更多极端仇恨内容呈正相关，表明社交认可是推动网络仇恨传播的关键机制。
Reason 该论文属于计算社会科学领域，研究网络仇恨言论的传播机制，与文档图像理解（DIU）或视觉-语言模型（VLM）技术无任何关联。既不涉及OCR、文档布局分析、表格识别等DIU核心任务，也未提出可用于解决OCR幻觉、密集文本处理或复杂排版理解的方法论创新。
Abstract We examined how online hate is motivated by receiving social approval via Walther's (2024) social approval theory of online hate, which argues (H1a) more signals of social approval on hate messages predicts more subsequent hate messages, and (H1b) as social approval increases, hate speech becomes more extreme. Using 110 million messages from Parler (2018-2021), we observed the number of upvotes received on a hate speech post was unassociated with hate speech in one's next post and during the next month, three-months, and six-months. The number of upvotes received on (extreme) hate speech comments, however, was positively associated with (extreme) hate speech during the next week, month, three-months, and six-months. Between-person effects revealed an average positive relationship between social approval and hate speech production at all time intervals. For comments, social approval linked more strongly to online hate than social disapproval. Social approval is a critical mechanism facilitating online hate propagation.
Authors David M. Markowitz, Samuel Hardman Taylor
Categories Computation and Language
0
Humanlike Multi-user Agent (HUMA): Designing a Deceptively Human AI Facilitator for Group Chats
Title CN 类人多用户代理（HUMA）：为群聊设计一种 deceptive 类人AI协调者
Keywords 对话代理 · 群聊 · 类人行为 · 响应时序模拟 · LLM
Summary 本文提出HUMA，一个基于大语言模型的多用户对话代理，能够以类人方式参与异步群聊，通过事件驱动架构模拟真实响应时间，并在角色扮演实验中展现出与人类社区管理员难以区分的表现。研究聚焦于提升AI在群体社交互动中的自然性和可信度。
Reason 该论文研究的是在多人聊天场景中构建类人行为的对话代理，属于具身智能与社交交互领域，核心关注点为对话策略、响应时序模拟和人类感知欺骗性。其任务与文档图像理解（DIU）完全无关，既不涉及视觉输入、OCR、布局分析，也不解决文档中的VQA、细粒度定位或高分辨率处理等痛点。尽管其Agent架构可能具有通用设计思想，但无法直接迁移用于解决DIU中的幻觉、复杂排版或长文档推理问题。
Abstract Conversational agents built on large language models (LLMs) are becoming increasingly prevalent, yet most systems are designed for one-on-one, turn-based exchanges rather than natural, asynchronous group chats. As AI assistants become widespread throughout digital platforms, from virtual assistants to customer service, developing natural and humanlike interaction patterns seems crucial for maintaining user trust and engagement. We present the Humanlike Multi-user Agent (HUMA), an LLM-based facilitator that participates in multi-party conversations using human-like strategies and timing. HUMA extends prior multi-user chatbot work with an event-driven architecture that handles messages, replies, reactions and introduces realistic response-time simulation. HUMA comprises three components-Router, Action Agent, and Reflection-which together adapt LLMs to group conversation dynamics. We evaluate HUMA in a controlled study with 97 participants in four-person role-play chats, comparing AI and human community managers (CMs). Participants classified CMs as human at near-chance rates in both conditions, indicating they could not reliably distinguish HUMA agents from humans. Subjective experience was comparable across conditions: community-manager effectiveness, social presence, and engagement/satisfaction differed only modestly with small effect sizes. Our results suggest that, in natural group chat settings, an AI facilitator can match human quality while remaining difficult to identify as nonhuman.
Authors Mateusz Jacniacki, Martí Carmona Serrat
Comment 9 pages, 4 figures
Categories Computation and Language
0
Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation
Title CN 爱沙尼亚WinoGrande数据集：LLM在人工与机器翻译上的性能比较分析
Keywords 小语种 · 机器翻译 · 提示工程 · 常识推理 · 数据集本地化
Summary 本文介绍了WinoGrande常识推理基准测试集的爱沙尼亚语本地化版本，比较了大语言模型在人工翻译和机器翻译数据上的表现。结果显示人工翻译效果更优，而提示工程对提升机器翻译质量作用有限，强调语言专家参与的重要性。
Reason 该论文聚焦于小语种（爱沙尼亚语）的常识推理数据集构建与翻译质量分析，属于特定非中英语言的数据集工作，命中负面清单第3条‘小语种’。其研究重点为语言模型在跨语言场景下的推理性能评估，并未涉及文档图像、视觉-语言多模态理解或OCR相关技术，无法迁移至文档图像理解（DIU）的核心问题。
Abstract In this paper, we present a localized and culturally adapted Estonian translation of the test set from the widely used commonsense reasoning benchmark, WinoGrande. We detail the translation and adaptation process carried out by translation specialists and evaluate the performance of both proprietary and open source models on the human translated benchmark. Additionally, we explore the feasibility of achieving high-quality machine translation by incorporating insights from the manual translation process into the design of a detailed prompt. This prompt is specifically tailored to address both the linguistic characteristics of Estonian and the unique translation challenges posed by the WinoGrande dataset. Our findings show that model performance on the human translated Estonian dataset is slightly lower than on the original English test set, while performance on machine-translated data is notably worse. Additionally, our experiments indicate that prompt engineering offers limited improvement in translation quality or model accuracy, and highlight the importance of involving language specialists in dataset translation and adaptation to ensure reliable and interpretable evaluations of language competency and reasoning in large language models.
Authors Marii Ojastu, Hele-Andra Kuulmets, Aleksei Dorkin, Marika Borovikova, Dage Särg, Kairit Sirts
Comment Preprint
Categories Computation and Language
0
Evaluating Large Language Models for Diacritic Restoration in Romanian Texts: A Comparative Study
Title CN 评估大语言模型在罗马尼亚语文本重音符号恢复中的性能：一项比较研究
Keywords 重音符号恢复 · 罗马尼亚语 · 大语言模型 · 文本修复 · 提示工程
Summary 本文评估了多种大语言模型在罗马尼亚语文本中自动恢复重音符号的能力，测试了包括GPT系列、Llama系列、Gemini等模型在不同提示模板下的表现。结果显示GPT-4o等闭源模型表现优异，而Llama系列存在较大波动。研究强调了模型架构、训练数据和提示设计的影响。
Reason 该论文聚焦于小语种（罗马尼亚语）的文本后处理任务，属于特定语言的NLP应用，未涉及文档图像理解中的核心问题如OCR幻觉、排版分析或视觉-语言对齐。尽管使用了LLM，但任务本质是纯文本的重音符号补全，与文档图像无关，且不具方法论上的迁移价值。
Abstract Automatic diacritic restoration is crucial for text processing in languages with rich diacritical marks, such as Romanian. This study evaluates the performance of several large language models (LLMs) in restoring diacritics in Romanian texts. Using a comprehensive corpus, we tested models including OpenAI's GPT-3.5, GPT-4, GPT-4o, Google's Gemini 1.0 Pro, Meta's Llama 2 and Llama 3, MistralAI's Mixtral 8x7B Instruct, airoboros 70B, and OpenLLM-Ro's RoLlama 2 7B, under multiple prompt templates ranging from zero-shot to complex multi-shot instructions. Results show that models such as GPT-4o achieve high diacritic restoration accuracy, consistently surpassing a neutral echo baseline, while others, including Meta's Llama family, exhibit wider variability. These findings highlight the impact of model architecture, training data, and prompt design on diacritic restoration performance and outline promising directions for improving NLP tools for diacritic-rich languages.
Authors Mihai Nadas, Laura Diosan
Comment The original submission contained metadata errors and requires correction. A revised and complete version will be submitted as a replacement
Categories Computation and Language
0
Cross-cultural value alignment frameworks for responsible AI governance: Evidence from China-West comparative analysis Technology for Good: Driving Social Impact
Title CN 负责任人工智能治理的跨文化价值对齐框架：来自中西方比较分析的证据
Keywords 价值观对齐 · 大语言模型 · AI治理 · 跨文化比较 · 伦理审计
Summary 本文提出一个多层级审计平台，用于评估中国和西方大语言模型在跨文化价值观对齐方面的能力，通过四种方法分析20多个主流模型，揭示了模型规模与对齐质量之间的非线性关系以及区域发展差异。研究发现Mistral架构在跨文化对齐上优于LLaMA3，全参数微调比基于人类反馈的强化学习更能保留文化多样性。
Reason 该论文聚焦于大语言模型在跨文化价值观对齐方面的伦理与治理问题，属于纯NLP领域的安全/对齐研究，尽管提到了Qwen、GPT-4o等多模态模型，但其方法（如伦理困境语料库、MARK推理框架）并未涉及视觉输入、文档结构或OCR相关幻觉问题，无法迁移解决文档图像理解中的核心痛点，如细粒度定位、高分辨率处理或排版理解。
Abstract As Large Language Models (LLMs) increasingly influence high-stakes decision-making across global contexts, ensuring their alignment with diverse cultural values has become a critical governance challenge. This study presents a Multi-Layered Auditing Platform for Responsible AI that systematically evaluates cross-cultural value alignment in China-origin and Western-origin LLMs through four integrated methodologies: Ethical Dilemma Corpus for assessing temporal stability, Diversity-Enhanced Framework (DEF) for quantifying cultural fidelity, First-Token Probability Alignment for distributional accuracy, and Multi-stAge Reasoning frameworK (MARK) for interpretable decision-making. Our comparative analysis of 20+ leading models, such as Qwen, GPT-4o, Claude, LLaMA, and DeepSeek, reveals universal challenges-fundamental instability in value systems, systematic under-representation of younger demographics, and non-linear relationships between model scale and alignment quality-alongside divergent regional development trajectories. While China-origin models increasingly emphasize multilingual data integration for context-specific optimization, Western models demonstrate greater architectural experimentation but persistent U.S.-centric biases. Neither paradigm achieves robust cross-cultural generalization. We establish that Mistral-series architectures significantly outperform LLaMA3-series in cross-cultural alignment, and that Full-Parameter Fine-Tuning on diverse datasets surpasses Reinforcement Learning from Human Feedback in preserving cultural variation...
Authors Haijiang Liu, Jinguang Gu, Xun Wu, Daniel Hershcovich, Qiaoling Xiao
Comment Presented on Academic Conference "Technology for Good: Driving Social Impact" (2025)
Categories Computation and Language
0
Social-Media Based Personas Challenge: Hybrid Prediction of Common and Rare User Actions on Bluesky
Title CN 基于社交媒体的用户画像挑战：在Bluesky上混合预测常见与稀有用户行为
Keywords 用户行为预测 · 社交媒体分析 · 稀有事件预测 · 混合模型 · LightGBM
Summary 本文提出了一种混合方法来预测社交媒体平台Bluesky上的用户行为，涵盖12种不同操作，并针对常见和稀有行为分别设计模型。使用历史模式数据库、LightGBM模型、神经网络架构和文本生成技术，在大规模对话数据集上实现了较高的F1分数，并在SocialSim挑战赛中获得第一名。
Reason 该论文属于社交行为预测领域，研究的是用户在社交媒体上的动作预测（如转发、点赞等），与文档图像理解（DIU）或视觉语言模型（VLM）处理文档内容无任何关联。其方法无法迁移到解决OCR幻觉、密集文本识别、复杂排版或长文档推理等DIU核心痛点，属于无关领域应用。
Abstract Understanding and predicting user behavior on social media platforms is crucial for content recommendation and platform design. While existing approaches focus primarily on common actions like retweeting and liking, the prediction of rare but significant behaviors remains largely unexplored. This paper presents a hybrid methodology for social media user behavior prediction that addresses both frequent and infrequent actions across a diverse action vocabulary. We evaluate our approach on a large-scale Bluesky dataset containing 6.4 million conversation threads spanning 12 distinct user actions across 25 persona clusters. Our methodology combines four complementary approaches: (i) a lookup database system based on historical response patterns; (ii) persona-specific LightGBM models with engineered temporal and semantic features for common actions; (iii) a specialized hybrid neural architecture fusing textual and temporal representations for rare action classification; and (iv) generation of text replies. Our persona-specific models achieve an average macro F1-score of 0.64 for common action prediction, while our rare action classifier achieves 0.56 macro F1-score across 10 rare actions. These results demonstrate that effective social media behavior prediction requires tailored modeling strategies recognizing fundamental differences between action types. Our approach achieved first place in the SocialSim: Social-Media Based Personas challenge organized at the Social Simulation with LLMs workshop at COLM 2025.
Authors Benjamin White, Anastasia Shimorina
Comment 1st place at SocialSim: Social-Media Based Personas challenge 2025
Categories Computation and Language
0
LangMark: A Multilingual Dataset for Automatic Post-Editing ACL 2025
Title CN LangMark：一种用于自动译后编辑的多语言数据集
Keywords 自动译后编辑 · 多语言数据集 · 机器翻译 · LLM · NMT
Summary 本文提出了LangMark，一个大规模、多人类语言的自动译后编辑（APE）数据集，包含20万以上的三元组（源文本、机器翻译输出、人工修正结果），覆盖七种语言。数据由专业语言学家标注，旨在推动基于大模型的少样本译后编辑系统发展。实验表明，LLM在该数据上能超越商用翻译系统的性能。
Reason 该论文属于纯NLP任务，聚焦于机器翻译后的文本修正（APE），使用多语言翻译数据集提升译后编辑性能。其研究内容与文档图像理解（DIU）无直接关联，既不涉及视觉信息、OCR、布局分析，也不处理文档中的多模态对齐或视觉幻觉等问题。属于典型的平行下游应用，且领域为纯语言处理，不符合DIU本题或可迁移方法的要求。
Abstract Automatic post-editing (APE) aims to correct errors in machine-translated text, enhancing translation quality, while reducing the need for human intervention. Despite advances in neural machine translation (NMT), the development of effective APE systems has been hindered by the lack of large-scale multilingual datasets specifically tailored to NMT outputs. To address this gap, we present and release LangMark, a new human-annotated multilingual APE dataset for English translation to seven languages: Brazilian Portuguese, French, German, Italian, Japanese, Russian, and Spanish. The dataset has 206,983 triplets, with each triplet consisting of a source segment, its NMT output, and a human post-edited translation. Annotated by expert human linguists, our dataset offers both linguistic diversity and scale. Leveraging this dataset, we empirically show that Large Language Models (LLMs) with few-shot prompting can effectively perform APE, improving upon leading commercial and even proprietary machine translation systems. We believe that this new resource will facilitate the future development and evaluation of APE systems.
Authors Diego Velazquez, Mikaela Grace, Konstantinos Karageorgos, Lawrence Carin, Aaron Schliem, Dimitrios Zaikis, Roger Wechsler
Comment 15 pages, 8 figures, ACL 2025
Categories Computation and Language
0
DiffTester: Accelerating Unit Test Generation for Diffusion LLMs via Repetitive Pattern
Title CN DiffTester：通过重复模式加速扩散LLM的单元测试生成
Keywords 单元测试生成 · 扩散LLM · 并行生成 · 抽象语法树 · 软件测试
Summary 本文提出DiffTester，一种面向扩散式大语言模型（dLLMs）的单元测试生成加速框架。通过识别测试代码中的重复语法结构模式，在保持测试质量的同时提升并行生成效率。研究扩展了TestEval基准至多语言环境，并在多个模型和语言上验证了方法的有效性。
Reason 该论文属于软件工程领域的自动化测试生成应用，核心是针对编程语言中单元测试的效率优化问题，使用扩散LLM和语法结构分析来提升生成速度。尽管涉及LLM生成优化，但其方法和任务均与文档图像理解（DIU）无直接关联，不解决OCR幻觉、复杂排版、细粒度定位或长文档推理等DIU核心痛点，属于平行下游应用。
Abstract Software development relies heavily on extensive unit testing, which makes the efficiency of automated Unit Test Generation (UTG) particularly important. However, most existing LLMs generate test cases one token at a time in each forward pass, which leads to inefficient UTG. Recently, diffusion LLMs (dLLMs) have emerged, offering promising parallel generation capabilities and showing strong potential for efficient UTG. Despite this advantage, their application to UTG is still constrained by a clear trade-off between efficiency and test quality, since increasing the number of tokens generated in each step often causes a sharp decline in the quality of test cases. To overcome this limitation, we present DiffTester, an acceleration framework specifically tailored for dLLMs in UTG. The key idea of DiffTester is that unit tests targeting the same focal method often share repetitive structural patterns. By dynamically identifying these common patterns through abstract syntax tree analysis during generation, DiffTester adaptively increases the number of tokens produced at each step without compromising the quality of the output. To enable comprehensive evaluation, we extend the original TestEval benchmark, which was limited to Python, by introducing additional programming languages including Java and C++. Extensive experiments on three benchmarks with two representative models show that DiffTester delivers significant acceleration while preserving test coverage. Moreover, DiffTester generalizes well across different dLLMs and programming languages, providing a practical and scalable solution for efficient UTG in software development. Code and data are publicly available at https://github.com/wellbeingyang/DLM4UTG-open .
Authors Lekang Yang, Yuetong Liu, Yitong Zhang, Jia Li
Comment Update reference
Categories Computation and Language
0
MUCH: A Multilingual Claim Hallucination Benchmark
Title CN MUCH：一个多语言声明幻觉基准
Keywords 幻觉检测 · 不确定性量化 · 多语言 · 基准测试 · 大语言模型
Summary 本文提出了MUCH，一个面向多语言场景的声明级幻觉检测基准，包含四千多个样本和四种开源LLM的生成logits，旨在推动大模型输出的不确定性量化研究。其核心贡献在于高效的自动声明分割算法与完整的logits发布，支持白盒方法开发。
Reason 该论文聚焦于纯文本大语言模型中的声明级不确定性量化（UQ）和幻觉检测，属于通用NLP领域。虽然涉及‘幻觉’，但其任务完全基于文本生成结果的后处理评估，未涉及任何视觉输入、文档图像或多模态理解。此外，其构建的是多语言（欧语系）基准数据集，接近‘小语种’范畴且无视觉-语言对齐机制设计。无法迁移解决文档图像理解中的OCR幻觉或视觉细粒度推理问题，属于无关领域的平行应用。
Abstract Claim-level Uncertainty Quantification (UQ) is a promising approach to mitigate the lack of reliability in Large Language Models (LLMs). We introduce MUCH, the first claim-level UQ benchmark designed for fair and reproducible evaluation of future methods under realistic conditions. It includes 4,873 samples across four European languages (English, French, Spanish, and German) and four instruction-tuned open-weight LLMs. Unlike prior claim-level benchmarks, we release 24 generation logits per token, facilitating the development of future white-box methods without re-generating data. Moreover, in contrast to previous benchmarks that rely on manual or LLM-based segmentation, we propose a new deterministic algorithm capable of segmenting claims using as little as 0.2% of the LLM generation time. This makes our segmentation approach suitable for real-time monitoring of LLM outputs, ensuring that MUCH evaluates UQ methods under realistic deployment constraints. Finally, our evaluations show that current methods still have substantial room for improvement in both performance and efficiency.
Authors Jérémie Dentan, Alexi Canesse, Davide Buscaldi, Aymen Shabou, Sonia Vanier
Categories Computation and Language
0
An Efficient Computational Framework for Discrete Fuzzy Numbers Based on Total Orders
Title CN 基于全序的离散模糊数高效计算框架
Keywords 离散模糊数 · 全序 · pos函数 · 计算复杂度 · 模糊系统
Summary 本文提出了一种用于计算定义在有限链上的离散模糊数的pos函数及其逆函数的高效算法，利用全序的组合结构实现了O(n²m log n)的时间复杂度，重点在于提升模糊系统中代数运算（如聚合与蕴含）的效率。研究聚焦于理论建模与数学优化，不涉及图像、文本布局或跨模态理解。
Reason 该论文研究的是离散模糊数上的全序计算及其代数操作，属于模糊数学与理论计算机科学范畴，未涉及任何文档图像理解（DIU）、多模态模型、视觉表示或语言-视觉对齐等内容。其方法无法直接迁移至解决OCR幻觉、复杂排版解析、细粒度定位或长文档推理等DIU核心问题。
Abstract Discrete fuzzy numbers, and in particular those defined over a finite chain $L_n = \{0, \ldots, n\}$, have been effectively employed to represent linguistic information within the framework of fuzzy systems. Research on total (admissible) orderings of such types of fuzzy subsets, and specifically those belonging to the set $\mathcal{D}_1^{L_n\rightarrow Y_m}$ consisting of discrete fuzzy numbers $A$ whose support is a closed subinterval of the finite chain $L_n = \{0, 1, \ldots, n\}$ and whose membership values $A(x)$, for $x \in L_n$, belong to the set $Y_m = \{ 0 = y_1 < y_2 < \cdots < y_{m-1} < y_m = 1 \}$, has facilitated the development of new methods for constructing logical connectives, based on a bijective function, called $\textit{pos function}$, that determines the position of each $A \in \mathcal{D}_1^{L_n\rightarrow Y_m}$. For this reason, in this work we revisit the problem by introducing algorithms that exploit the combinatorial structure of total (admissible) orders to compute the $\textit{pos}$ function and its inverse with exactness. The proposed approach achieves a complexity of $\mathcal{O}(n^{2} m \log n)$, which is quadratic in the size of the underlying chain ($n$) and linear in the number of membership levels ($m$). The key point is that the dominant factor is $m$, ensuring scalability with respect to the granularity of membership values. The results demonstrate that this formulation substantially reduces computational cost and enables the efficient implementation of algebraic operations -- such as aggregation and implication -- on the set of discrete fuzzy numbers.
Authors Arnau Mir, Alejandro Mus, Juan Vicente Riera
Comment 19 pages, 2 figures. Submitted to Computational and Applied Mathematics (Springer)
Categories Computation and Language
0
A systematic review of relation extraction task since the emergence of Transformers
Title CN Transformer出现以来关系抽取任务的系统性综述
Keywords 关系抽取 · Transformer · 系统综述 · 自然语言处理 · 文本挖掘
Summary 本文是对2019至2024年间基于Transformer的关系抽取研究的系统性综述，涵盖了模型、数据集和方法论进展。但其范围局限于纯文本NLP任务，不包含视觉或多模态内容。
Reason 该论文属于自然语言处理领域的综述性工作，聚焦于纯文本关系抽取（RE）任务，未涉及任何文档图像、多模态输入或视觉-语言建模。其内容与文档图像理解（DIU）的核心问题如OCR幻觉、复杂版式分析、视觉文本对齐等完全无关，且无任何可迁移的方法论创新。
Abstract This article presents a systematic review of relation extraction (RE) research since the advent of Transformer-based models. Using an automated framework to collect and annotate publications, we analyze 34 surveys, 64 datasets, and 104 models published between 2019 and 2024. The review highlights methodological advances, benchmark resources, and the integration of semantic web technologies. By consolidating results across multiple dimensions, the study identifies current trends, limitations, and open challenges, offering researchers and practitioners a comprehensive reference for understanding the evolution and future directions of RE.
Authors Ringwald Celian, Gandon, Fabien, Faron Catherine, Michel Franck, Abi Akl Hanna
Comment Submited at ACM-Computing Surveys + The resulting annotated Zotero bibliography : https://www.zotero.org/groups/6070963/scilex_re_systlitreview/library + SciLEx software: https://github.com/Wimmics/SciLEx
Categories Computation and Language
0
Improving the Performance of Radiology Report De-identification with Large-Scale Training and Benchmarking Against Cloud Vendor Methods JAMIA
Title CN 通过大规模训练和与云服务商方法的基准对比提升放射学报告去标识化性能
Keywords 去标识化 · 放射学报告 · PHI检测 · 自然语言处理 · Transformer
Summary 本文基于大规模放射学文本数据，改进了一个用于去标识化的Transformer模型，并在多个数据集上超越了现有学术与商业系统。研究重点在于临床文本隐私保护，而非文档图像的理解或视觉-语言建模。
Reason 该论文属于典型的‘平行下游应用’，即将现有的Transformer模型应用于放射学报告中的PHI（受保护健康信息）识别任务。尽管其在数据规模和基准对比上有所贡献，但核心方法并未提出针对文档图像理解（DIU）或视觉-语言模型（VLM）的新架构、训练策略或底层技术创新。任务本质是纯文本NLP中的信息抽取与脱敏，不涉及OCR、多模态对齐、视觉布局分析或文档图像中的幻觉问题，因此无法迁移至DIU核心痛点的解决。
Abstract Objective: To enhance automated de-identification of radiology reports by scaling transformer-based models through extensive training datasets and benchmarking performance against commercial cloud vendor systems for protected health information (PHI) detection. Materials and Methods: In this retrospective study, we built upon a state-of-the-art, transformer-based, PHI de-identification pipeline by fine-tuning on two large annotated radiology corpora from Stanford University, encompassing chest X-ray, chest CT, abdomen/pelvis CT, and brain MR reports and introducing an additional PHI category (AGE) into the architecture. Model performance was evaluated on test sets from Stanford and the University of Pennsylvania (Penn) for token-level PHI detection. We further assessed (1) the stability of synthetic PHI generation using a "hide-in-plain-sight" method and (2) performance against commercial systems. Precision, recall, and F1 scores were computed across all PHI categories. Results: Our model achieved overall F1 scores of 0.973 on the Penn dataset and 0.996 on the Stanford dataset, outperforming or maintaining the previous state-of-the-art model performance. Synthetic PHI evaluation showed consistent detectability (overall F1: 0.959 [0.958-0.960]) across 50 independently de-identified Penn datasets. Our model outperformed all vendor systems on synthetic Penn reports (overall F1: 0.960 vs. 0.632-0.754). Discussion: Large-scale, multimodal training improved cross-institutional generalization and robustness. Synthetic PHI generation preserved data utility while ensuring privacy. Conclusion: A transformer-based de-identification model trained on diverse radiology datasets outperforms prior academic and commercial systems in PHI detection and establishes a new benchmark for secure clinical text processing.
Authors Eva Prakash, Maayane Attias, Pierre Chambon, Justin Xu, Steven Truong, Jean-Benoit Delbrouck, Tessa Cook, Curtis Langlotz
Comment In submission to JAMIA
Categories Computation and Language
0
ARQUSUMM: Argument-aware Quantitative Summarization of Online Conversations AAAI2026
Title CN ARQUSUMM：面向在线对话的论点感知定量摘要
Keywords 对话摘要 · 论点挖掘 · 定量摘要 · 大语言模型 · 少样本学习
Summary 本文提出了一种名为ARQUSUMM的框架，用于实现对在线对话的论点感知定量摘要。通过结合论证理论指导下的大模型少样本学习，识别句子内的主张-理由结构，并利用论点结构感知的聚类算法进行论点聚合与支持度量化。实验表明该方法在生成具有清晰论点结构、高质量文本和准确量化的摘要方面优于现有模型。
Reason 该论文属于纯文本领域的对话摘要任务，聚焦于社交媒体（如Reddit）中的论点结构建模与定量总结，未涉及任何视觉或文档图像内容。其方法基于大语言模型的少样本学习和聚类算法，属于典型的NLP下游应用，与文档图像理解（DIU）、OCR、多模态文档分析等核心问题完全无关。
Abstract Online conversations have become more prevalent on public discussion platforms (e.g. Reddit). With growing controversial topics, it is desirable to summarize not only diverse arguments, but also their rationale and justification. Early studies on text summarization focus on capturing general salient information in source documents, overlooking the argumentative nature of online conversations. Recent research on conversation summarization although considers the argumentative relationship among sentences, fail to explicate deeper argument structure within sentences for summarization. In this paper, we propose a novel task of argument-aware quantitative summarization to reveal the claim-reason structure of arguments in conversations, with quantities measuring argument strength. We further propose ARQUSUMM, a novel framework to address the task. To reveal the underlying argument structure within sentences, ARQUSUMM leverages LLM few-shot learning grounded in the argumentation theory to identify propositions within sentences and their claim-reason relationships. For quantitative summarization, ARQUSUMM employs argument structure-aware clustering algorithms to aggregate arguments and quantify their support. Experiments show that ARQUSUMM outperforms existing conversation and quantitative summarization models and generate summaries representing argument structures that are more helpful to users, of high textual quality and quantification accuracy.
Authors An Quang Tang, Xiuzhen Zhang, Minh Ngoc Dinh, Zhuang Li
Comment Paper accepted to AAAI2026 Main Technical Track
Categories Computation and Language
0
Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models
Title CN 响应攻击：利用上下文启动效应破解大型语言模型
Keywords 越狱攻击 · 上下文启动 · 大语言模型安全 · 对抗攻击 · 对话系统
Summary 本文提出了一种名为‘响应攻击’（Response Attack, RA）的新框架，通过在对话中插入中间的轻度有害响应作为上下文启动信号，从而引导大型语言模型在后续输出中产生更明确的违规内容。实验表明该方法在多个主流LLM上均显著优于现有越狱技术。然而，该工作属于语言模型安全性研究范畴，与文档图像理解无关联。
Reason 该论文研究的是针对大型语言模型的越狱攻击（jailbreak），属于纯NLP安全与对齐领域，聚焦于诱导LLM生成有害内容。其核心问题与文档图像理解（DIU）无关，不涉及视觉输入、OCR、多模态对齐或文档结构建模。尽管提到了上下文引导机制，但该方法应用于恶意攻击场景，无法迁移至解决DIU中的幻觉、细粒度定位或复杂排版等实际技术痛点。
Abstract Contextual priming, where earlier stimuli covertly bias later judgments, offers an unexplored attack surface for large language models (LLMs). We uncover a contextual priming vulnerability in which the previous response in the dialogue can steer its subsequent behavior toward policy-violating content. While existing jailbreak attacks largely rely on single-turn or multi-turn prompt manipulations, or inject static in-context examples, these methods suffer from limited effectiveness, inefficiency, or semantic drift. We introduce Response Attack (RA), a novel framework that strategically leverages intermediate, mildly harmful responses as contextual primers within a dialogue. By reformulating harmful queries and injecting these intermediate responses before issuing a targeted trigger prompt, RA exploits a previously overlooked vulnerability in LLMs. Extensive experiments across eight state-of-the-art LLMs show that RA consistently achieves significantly higher attack success rates than nine leading jailbreak baselines. Our results demonstrate that the success of RA is directly attributable to the strategic use of intermediate responses, which induce models to generate more explicit and relevant harmful content while maintaining stealth, efficiency, and fidelity to the original query. The code and data are available at https://github.com/Dtc7w3PQ/Response-Attack.
Authors Ziqi Miao, Lijun Li, Yuan Xiong, Zhenhua Liu, Pengyu Zhu, Jing Shao
Comment 20 pages, 10 figures. Code and data available at https://github.com/Dtc7w3PQ/Response-Attack
Categories Computation and Language
0
An Architectural Advantage of The Instruction-Tuned LLM in Containing The Readability-Accuracy Tension in Text Simplification
Title CN 指令调优大语言模型在文本简化中缓解可读性-准确性矛盾的架构优势
Keywords 文本简化 · 大语言模型 · 可读性 · 语义保真 · 指令调优
Summary 本文比较了指令调优型大模型Mistral-Small和推理增强型QWen2.5在医学文本简化任务中的表现，发现Mistral在提升可读性的同时更好地保持了原文语义一致性（BERTScore达0.91），揭示了指令调优架构在平衡可读性与准确性的潜在优势。研究基于纯文本指标分析，未涉及图像或文档结构信息。
Reason 该论文研究的是纯文本到文本的简化任务，属于通用NLP范畴，不涉及任何视觉或文档图像理解（DIU）模态。尽管其探讨了LLM在可读性和准确性之间的权衡，但未引入多模态建模、视觉布局理解、OCR增强或针对文档图像的处理机制。无法迁移解决DIU中的核心痛点如OCR幻觉、复杂排版解析或细粒度视觉定位。属于‘平行应用’类研究的上游基础工作，但本身并非DIU方法创新，也无直接可迁移的技术组件。
Abstract The increasing health-seeking behavior and digital consumption of biomedical information by the general public necessitate scalable solutions for automatically adapting complex scientific and technical documents into plain language. Automatic text simplification solutions, including advanced large language models (LLMs), however, continue to face challenges in reliably arbitrating the tension between optimizing readability performance and ensuring preservation of discourse fidelity. This report empirically assesses two major classes of general-purpose LLMs, demonstrating how they navigate the readability-accuracy tension compared to a human benchmark. Using a comparative analysis of the instruction-tuned Mistral-Small 3 24B and the reasoning-augmented QWen2.5 32B, we identify an architectural advantage in the instruction-tuned LLM. Mistral exhibits a tempered lexical simplification strategy that enhances readability across a suite of metrics while preserving human-level discourse with a BERTScore of 0.91. QWen also attains enhanced readability performance and a reasonable BERTScore of 0.89, but its operational strategy shows a disconnect in balancing between readability and accuracy. Additionally, a comprehensive correlation analysis of a suite of 21 metrics spanning readability, discourse fidelity, content safety, and underlying distributional measures for mechanistic insights, confirms strong functional redundancies, and informs metric selection and domain adaptation for text simplification.
Authors P. Bilha Githinji, Aikaterini Meilliou, Zeming Liang, Lian Zhang, Peiwu Qin
Categories Computation and Language
0
PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese
Title CN PoETa v2：面向葡萄牙语大语言模型更鲁棒的评估
Keywords 大语言模型 · 多语言评估 · 葡萄牙语 · 基准测试 · 语言建模
Summary 本文提出了PoETa v2，一个包含40多个任务的综合性葡萄牙语基准，用于评估超过20个大语言模型在该语言下的表现。研究分析了计算资源投入和语言特定适配对性能的影响，并比较了与英语任务之间的性能差距。该工作旨在推动葡萄牙语语言模型的发展。
Reason 该论文聚焦于葡萄牙语语言环境下大语言模型的评估，属于小语种（非中英）研究，且核心是语言模型评测基准构建，与文档图像理解（DIU）或视觉-语言多模态任务无直接关联。无法迁移解决OCR幻觉、复杂排版或视觉文本理解等DIU核心问题。
Abstract Large Language Models (LLMs) exhibit significant variations in performance across linguistic and cultural contexts, underscoring the need for systematic evaluation in diverse languages. In this work, we present the most extensive evaluation of LLMs for the Portuguese language to date. Leveraging our newly introduced PoETa v2 benchmark -- a comprehensive suite of over 40 tasks in Portuguese -- we assess more than 20 models covering a broad spectrum of training scales and computational resources. Our study reveals how computational investment and language-specific adaptation impact performance in Portuguese, while also analyzing performance gaps in comparison to equivalent tasks in English. Through this benchmark and analysis, PoETa v2 lays the groundwork for future research on Portuguese language modeling and evaluation. The benchmark is available at https://github.com/PoETaV2/PoETaV2.
Authors Thales Sales Almeida, Rodrigo Nogueira, Hélio Pedrini
Categories Computation and Language
0
Revolutionizing Finance with LLMs: An Overview of Applications and Insights
Title CN 用大语言模型革新金融：应用与洞察综述
Keywords 大语言模型 · 金融应用 · 综述 · 自然语言处理 · GPT-4
Summary 本文综述了大语言模型在金融领域的应用，包括财务报告生成、市场预测、投资者情绪分析和个性化金融建议。通过实验验证了GPT-4在多种金融任务中遵循自然语言指令的能力。旨在为金融从业者和LLM研究者提供对LLM在金融中角色的理解。
Reason 该论文属于典型的平行下游应用，仅将现有LLM技术应用于金融领域，如自动生成财务报告、市场趋势预测等，未提出任何针对文档图像理解或VLM架构的底层方法创新。完全不符合DIU本题研究需求，属于‘Wrapper/Application’类综述，无技术源头贡献。
Abstract In recent years, Large Language Models (LLMs) like ChatGPT have seen considerable advancements and have been applied in diverse fields. Built on the Transformer architecture, these models are trained on extensive datasets, enabling them to understand and generate human language effectively. In the financial domain, the deployment of LLMs is gaining momentum. These models are being utilized for automating financial report generation, forecasting market trends, analyzing investor sentiment, and offering personalized financial advice. Leveraging their natural language processing capabilities, LLMs can distill key insights from vast financial data, aiding institutions in making informed investment choices and enhancing both operational efficiency and customer satisfaction. In this study, we provide a comprehensive overview of the emerging integration of LLMs into various financial tasks. Additionally, we conducted holistic tests on multiple financial tasks through the combination of natural language instructions. Our findings show that GPT-4 effectively follow prompt instructions across various financial tasks. This survey and evaluation of LLMs in the financial domain aim to deepen the understanding of LLMs' current role in finance for both financial practitioners and LLM researchers, identify new research and application prospects, and highlight how these technologies can be leveraged to solve practical challenges in the finance industry.
Authors Huaqin Zhao, Zhengliang Liu, Zihao Wu, Yiwei Li, Tianze Yang, Peng Shu, Shaochen Xu, Haixing Dai, Lin Zhao, Hanqi Jiang, Yi Pan, Junhao Chen, Yifan Zhou, Zeyu Zhang, Gengchen Mai, Ninghao Liu, Tianming Liu
Categories Computation and Language
0
Red Teaming Multimodal Language Models: Evaluating Harm Across Prompt Modalities and Models
Title CN 红队测试多模态语言模型：跨提示模态和模型的伤害评估
Keywords 多模态大模型 · 红队测试 · 内容安全 · 对抗提示 · 模型鲁棒性
Summary 该研究通过红队测试方式评估了四种主流多模态大模型（GPT-4o、Claude Sonnet 3.5、Pixtral 12B、Qwen VL Plus）在面对文本和多模态对抗性提示时生成有害内容的倾向。结果显示不同模型在安全性上表现差异显著，其中Pixtral 12B最易产生有害输出，而Claude Sonnet 3.5最具鲁棒性。意外发现是文本提示比多模态提示更易绕过安全机制。研究强调需建立更强大的多模态安全基准。
Reason 该论文属于纯NLP安全与对齐领域，研究多模态大模型在非法活动、虚假信息和不道德行为等场景下的安全性，虽涉及多模态输入，但其核心关注点为内容安全而非文档理解中的视觉幻觉或结构解析问题。未提出可用于解决DIU领域OCR幻觉、复杂排版或长文档推理等痛点的底层方法，属于无关领域的应用研究。
Abstract Multimodal large language models (MLLMs) are increasingly used in real world applications, yet their safety under adversarial conditions remains underexplored. This study evaluates the harmlessness of four leading MLLMs (GPT-4o, Claude Sonnet 3.5, Pixtral 12B, and Qwen VL Plus) when exposed to adversarial prompts across text-only and multimodal formats. A team of 26 red teamers generated 726 prompts targeting three harm categories: illegal activity, disinformation, and unethical behaviour. These prompts were submitted to each model, and 17 annotators rated 2,904 model outputs for harmfulness using a 5-point scale. Results show significant differences in vulnerability across models and modalities. Pixtral 12B exhibited the highest rate of harmful responses (~62%), while Claude Sonnet 3.5 was the most resistant (~10%). Contrary to expectations, text-only prompts were slightly more effective at bypassing safety mechanisms than multimodal ones. Statistical analysis confirmed that both model type and input modality were significant predictors of harmfulness. These findings underscore the urgent need for robust, multimodal safety benchmarks as MLLMs are deployed more widely.
Authors Madison Van Doren, Casey Ford
Categories Computation and Language
0
Computational frame analysis revisited: On LLMs for studying news coverage
Title CN 重新审视计算框架分析：关于使用大语言模型研究新闻报道
Keywords 框架分析 · 大语言模型 · 内容分析 · 人工编码 · 方法论
Summary 本文评估了生成式大语言模型在新闻报道框架分析中的有效性，发现其表现不如人工编码，有时甚至不如小型语言模型，强调人类验证的必要性，并提出多方法结合的研究路径。研究基于2022年美国猴痘疫情的新闻数据集。
Reason 该论文属于纯NLP领域的社会科学研究方法论探讨，聚焦于新闻报道中的‘框架分析’（frame analysis），使用LLM进行文本内容分析，并与传统文本分析方法对比。其任务本质是文本分类与内容编码，不涉及任何视觉信息、文档结构或图像理解。完全脱离文档图像理解（DIU）的技术范畴，且属于典型的‘平行应用’——将现有LLM用于特定领域文本分析，无底层方法创新，无法迁移至OCR、布局分析、表格识别等DIU核心问题。
Abstract Computational approaches have previously shown various promises and pitfalls when it comes to the reliable identification of media frames. Generative LLMs like GPT and Claude are increasingly being used as content analytical tools, but how effective are they for frame analysis? We address this question by systematically evaluating them against their computational predecessors: bag-of-words models and encoder-only transformers; and traditional manual coding procedures. Our analysis rests on a novel gold standard dataset that we inductively and iteratively developed through the study, investigating six months of news coverage of the US Mpox epidemic of 2022. While we discover some potential applications for generative LLMs, we demonstrate that they were consistently outperformed by manual coders, and in some instances, by smaller language models. Some form of human validation was always necessary to determine appropriate model choice. Additionally, by examining how the suitability of various approaches depended on the nature of different tasks that were part of our frame analytical workflow, we provide insights as to how researchers may leverage the complementarity of these approaches to use them in tandem. We conclude by endorsing a methodologically pluralistic approach and put forth a roadmap for computational frame analysis for researchers going forward.
Authors Sharaj Kunjar, Alyssa Hasegawa Smith, Tyler R Mckenzie, Rushali Mohbe, Samuel V Scarpino, Brooke Foucault Welles
Categories Computation and Language
0
A Cross-Cultural Assessment of Human Ability to Detect LLM-Generated Fake News about South Africa
Title CN 关于人类识别南非AI生成假新闻能力的跨文化评估
Keywords 假新闻检测 · 跨文化研究 · 人类认知 · 大语言模型 · misinformation
Summary 本研究通过调查南非本地人与其他国籍参与者在识别关于南非的真实新闻与LLM生成假新闻方面的能力差异，探讨文化亲近性对人类辨别AI伪造信息的影响。结果显示，南非参与者更擅长识别真实新闻，但在识别假新闻时表现较差，可能与其对本地媒体更高的信任度有关。不同文化背景的参与者在判断依据上也存在差异：南非人更依赖内容知识和语境理解，而外国人更关注语法和结构等语言形式特征。
Reason 该论文研究的是跨文化背景下人类对AI生成假新闻的检测能力，属于社会认知与计算语言学交叉领域，不涉及文档图像理解（DIU）、视觉-语言模型架构、OCR、布局分析或任何多模态文档处理技术。与DIU核心任务无关联，且未提出可迁移至文档理解领域的底层方法。
Abstract This study investigates how cultural proximity affects the ability to detect AI-generated fake news by comparing South African participants with those from other nationalities. As large language models increasingly enable the creation of sophisticated fake news, understanding human detection capabilities becomes crucial, particularly across different cultural contexts. We conducted a survey where 89 participants (56 South Africans, 33 from other nationalities) evaluated 10 true South African news articles and 10 AI-generated fake versions. Results reveal an asymmetric pattern: South Africans demonstrated superior performance in detecting true news about their country (40% deviation from ideal rating) compared to other participants (52%), but performed worse at identifying fake news (62% vs. 55%). This difference may reflect South Africans' higher overall trust in news sources. Our analysis further shows that South Africans relied more on content knowledge and contextual understanding when judging credibility, while participants from other countries emphasised formal linguistic features such as grammar and structure. Overall, the deviation from ideal rating was similar between groups (51% vs. 53%), suggesting that cultural familiarity appears to aid verification of authentic information but may also introduce bias when evaluating fabricated content. These insights contribute to understanding cross-cultural dimensions of misinformation detection and inform strategies for combating AI-generated fake news in increasingly globalised information ecosystems where content crosses cultural and geographical boundaries.
Authors Tim Schlippe, Matthias Wölfel, Koena Ronny Mabokela
Categories Computation and Language
0
LLM and Agent-Driven Data Analysis: A Systematic Approach for Enterprise Applications and System-level Deployment
Title CN 由大语言模型和智能体驱动的数据分析：面向企业应用与系统级部署的系统性方法
Keywords 大语言模型 · 智能体 · 企业数据分析 · SQL生成 · RAG
Summary 本文探讨了生成式AI与智能体技术在企业数据管理与分析中的系统性应用，重点包括基于LLM的SQL生成、多智能体协作、安全验证与分布式部署。通过案例展示了如何利用RAG和向量数据库实现语义查询，提升数据分析效率。但整体属于技术集成与工程落地范畴，缺乏基础方法论贡献。
Reason 该论文属于典型的‘平行下游应用’，聚焦于企业级数据管理、SQL生成与RAG系统部署，属于将现有LLM和Agent技术应用于特定领域的系统集成工作，未提出任何针对文档图像理解（DIU）或视觉-语言建模的底层方法创新。内容与OCR、文档布局分析、表格识别、视觉幻觉消除等DIU核心问题无直接关联，也未提供可迁移至文档理解任务的新架构、训练策略或推理机制。
Abstract The rapid progress in Generative AI and Agent technologies is profoundly transforming enterprise data management and analytics. Traditional database applications and system deployment are fundamentally impacted by AI-driven tools, such as Retrieval-Augmented Generation (RAG) and vector database technologies, which provide new pathways for semantic querying over enterprise knowledge bases. In the meantime, data security and compliance are top priorities for organizations adopting AI technologies. For enterprise data analysis, SQL generations powered by large language models (LLMs) and AI agents, has emerged as a key bridge connecting natural language with structured data, effectively lowering the barrier to enterprise data access and improving analytical efficiency. This paper focuses on enterprise data analysis applications and system deployment, covering a range of innovative frameworks, enabling complex query understanding, multi-agent collaboration, security verification, and computational efficiency. Through representative use cases, key challenges related to distributed deployment, data security, and inherent difficulties in SQL generation tasks are discussed.
Authors Xi Wang, Xianyao Ling, Kun Li, Gang Yin, Liang Zhang, Jiang Wu, Annie Wang, Weizhe Wang
Categories Computation and Language
0
MURMUR: Using cross-user chatter to break collaborative language agents in groups
Title CN MURMUR：利用跨用户对话破坏群组中的协作语言代理
Keywords 语言代理 · 多用户协作 · 安全攻击 · CUP · 共享状态污染
Summary 本文提出了一种名为MURMUR的框架，用于模拟多用户环境中语言代理之间的交互，并揭示了一类新型攻击——跨用户投毒（CUP），即恶意用户可通过看似正常的消息污染共享状态，导致代理在后续为其他用户执行非预期操作。作者验证了该攻击在真实系统中的有效性，并提出了基于任务聚类的初步防御机制。
Reason 该论文研究的是多用户语言代理场景下的安全漏洞（跨用户投毒攻击），属于纯NLP领域的安全与对齐问题，且不涉及视觉模态或文档理解任务。尽管提到了协作代理的上下文管理，但其核心关注点是系统安全性而非文档图像理解中的幻觉、布局分析或复杂推理等痛点，无法迁移用于解决DIU的关键挑战。
Abstract Language agents are rapidly expanding from single-user assistants to multi-user collaborators in shared workspaces and groups. However, today's language models lack a mechanism for isolating user interactions and concurrent tasks, creating a new attack vector inherent to this new setting: cross-user poisoning (CUP). In a CUP attack, an adversary injects ordinary-looking messages that poison the persistent, shared state, which later triggers the agent to execute unintended, attacker-specified actions on behalf of benign users. We validate CUP on real systems, successfully attacking popular multi-user agents. To study the phenomenon systematically, we present MURMUR, a framework that composes single-user tasks into concurrent, group-based scenarios using an LLM to generate realistic, history-aware user interactions. We observe that CUP attacks succeed at high rates and their effects persist across multiple tasks, thus posing fundamental risks to multi-user LLM deployments. Finally, we introduce a first-step defense with task-based clustering to mitigate this new class of vulnerability
Authors Atharv Singh Patlan, Peiyao Sheng, S. Ashwin Hebbar, Prateek Mittal, Pramod Viswanath
Comment 20 pages, 7 figures
Categories Computation and Language
Artificial Intelligence 195
☆ The Universal Weight Subspace Hypothesis
Prakhar Kaushik, Shravan Chaudhari, Ankit Vaidya, Rama Chellappa, Alan Yuille
We show that deep neural networks trained across diverse tasks exhibit remarkably similar low-dimensional parametric subspaces. We provide the first large-scale empirical evidence that demonstrates that neural networks systematically converge to shared spectral subspaces regardless of initialization, task, or domain. Through mode-wise spectral analysis of over 1100 models - including 500 Mistral-7B LoRAs, 500 Vision Transformers, and 50 LLaMA-8B models - we identify universal subspaces capturing majority variance in just a few principal directions. By applying spectral decomposition techniques to the weight matrices of various architectures trained on a wide range of tasks and datasets, we identify sparse, joint subspaces that are consistently exploited, within shared architectures across diverse tasks and datasets. Our findings offer new insights into the intrinsic organization of information within deep networks and raise important questions about the possibility of discovering these universal subspaces without the need for extensive data and computational resources. Furthermore, this inherent structure has significant implications for model reusability, multi-task learning, model merging, and the development of training and inference-efficient algorithms, potentially reducing the carbon footprint of large-scale neural models.
comment : 37 pages
☆ DraCo: Draft as CoT for Text-to-Image P review and Rare Concept Generation
Dongzhi Jiang, Renrui Zhang, Haodong Li, Zhuofan Zong, Ziyu Guo, Jun He, Claire Guo, Junyan Ye, Rongyao Fang, Weijia Li, Rui Liu, Hongsheng Li
Recent unified multimodal large language models (MLLMs) have shown impressive capabilities, incorporating chain-of-thought (CoT) reasoning for enhanced text-to-image generation. However, existing approaches remain limited, either treating the model merely as a standalone generator or relying on abstract textual planning. To this end, we propose Draft-as-CoT (DraCo), a novel interleaved reasoning paradigm that fully leverages both textual and visual contents in CoT for better planning and verification. Our method first generates a low-resolution draft image as preview, providing more concrete and structural visual planning and guidance. Then, we employ the model's inherent understanding capability to verify potential semantic misalignments between the draft and input prompt, and performs refinement through selective corrections with super-resolution. In this way, our approach addresses two fundamental challenges: the coarse-grained nature of textual planning and the difficulty in generating rare attribute combinations. To support training, we curate DraCo-240K, aiming to enhance three atomic capabilities spanning general correction, instance manipulation, and layout reorganization. Supported by DraCo-CFG, a specialized classifier-free guidance (CFG) strategy for interleaved reasoning, DraCo achieves a tremendous increase on GenEval (+8%), Imagine-Bench (+0.91), and GenEval++ (+3%), significantly outperforming direct generation and other generation methods empowered by CoT.
comment : Project Page: https://github.com/CaraJ7/DraCo
☆ ShadowDraw: From Any Object to Shadow-Drawing Compositional Art
Rundong Luo, Noah Snavely, Wei-Chiu Ma
We introduce ShadowDraw, a framework that transforms ordinary 3D objects into shadow-drawing compositional art. Given a 3D object, our system predicts scene parameters, including object pose and lighting, together with a partial line drawing, such that the cast shadow completes the drawing into a recognizable image. To this end, we optimize scene configurations to reveal meaningful shadows, employ shadow strokes to guide line drawing generation, and adopt automatic evaluation to enforce shadow-drawing coherence and visual quality. Experiments show that ShadowDraw produces compelling results across diverse inputs, from real-world scans and curated datasets to generative assets, and naturally extends to multi-object scenes, animations, and physical deployments. Our work provides a practical pipeline for creating shadow-drawing art and broadens the design space of computational visual art, bridging the gap between algorithmic design and artistic storytelling. Check out our project page https://red-fairy.github.io/ShadowDraw/ for more results and an end-to-end real-world demonstration of our pipeline!
comment : Project page: https://red-fairy.github.io/ShadowDraw/
☆ Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learning
Purbesh Mitra, Sennur Ulukus
Long context reasoning in large language models (LLMs) has demonstrated enhancement of their cognitive capabilities via chain-of-thought (CoT) inference. Training such models is usually done via reinforcement learning with verifiable rewards (RLVR) in reasoning based problems, like math and programming. However, RLVR is limited by several bottlenecks, such as, lack of dense reward, and inadequate sample efficiency. As a result, it requires significant compute resources in post-training phase. To overcome these limitations, in this work, we propose \textbf{Semantic Soft Bootstrapping (SSB)}, a self-distillation technique, in which the same base language model plays the role of both teacher and student, but receives different semantic contexts about the correctness of its outcome at training time. The model is first prompted with a math problem and several rollouts are generated. From them, the correct and most common incorrect response are filtered, and then provided to the model in context to produce a more robust, step-by-step explanation with a verified final answer. This pipeline automatically curates a paired teacher-student training set from raw problem-answer data, without any human intervention. This generation process also produces a sequence of logits, which is what the student model tries to match in the training phase just from the bare question alone. In our experiment, Qwen2.5-3B-Instruct on GSM8K dataset via parameter-efficient fine-tuning. We then tested its accuracy on MATH500, and AIME2024 benchmarks. Our experiments show a jump of 10.6%, and 10% improvements in accuracy, respectively, over group relative policy optimization (GRPO), which is a commonly used RLVR algorithm. Our code is available at https://github.com/purbeshmitra/semantic-soft-bootstrapping, and the model, curated dataset is available at https://huggingface.co/purbeshmitra/semantic-soft-bootstrapping.
☆ TV2TV: A Unified Framework for Inte rl eaved Language and Video Generation
Xiaochuang Han, Youssef Emad, Melissa Hall, John Nguyen, Karthik Padthe, Liam Robbins, Amir Bar, Delong Chen, Michal Drozdzal, Maha Elbayad, Yushi Hu, Shang-Wen Li, Sreya Dutta Roy, Jakob Verbeek, XuDong Wang, Marjan Ghazvininejad, Luke Zettlemoyer, Emily Dinan
Video generation models are rapidly advancing, but can still struggle with complex video outputs that require significant semantic branching or repeated high-level reasoning about what should happen next. In this paper, we introduce a new class of omni video-text models that integrate ideas from recent LM reasoning advances to address this challenge. More specifically, we present TV2TV, a unified generative modeling framework which decomposes video generation into an interleaved text and video generation process. TV2TV jointly learns language modeling (next-token prediction) and video flow matching (next-frame prediction) using a Mixture-of-Transformers (MoT) architecture. At inference time, TV2TV decides when to alternate between generating text and video frames, allowing the model to "think in words" about subsequent content before ``acting in pixels'' to produce frames. This design offloads much of the responsibility for deciding what should happen next to the language modeling tower, enabling improved visual quality and prompt alignment of generated videos. It also enables fine-grained controllability, allowing users to modify the video generation trajectory through text interventions at any point in the process. In controlled experiments on video game data, TV2TV demonstrates substantial improvements in both visual quality and controllability. TV2TV also scales to natural videos, as we show by augmenting sports videos with interleaved natural language action descriptions using vision-language models (VLMs). Training TV2TV on this corpus yields strong visual quality and prompt alignment, showcasing the model's ability to reason about and generate complex real-world action sequences. Together, these results highlight TV2TV as a promising step toward video generation with open-ended textual reasoning and control.
☆ Structured Document Translation via Format Reinforcement Learning AACL 2025
Haiyue Song, Johannes Eschbach-Dymanus, Hour Kaing, Sumire Honda, Hideki Tanaka, Bianka Buschbeck, Masao Utiyama
Recent works on structured text translation remain limited to the sentence level, as they struggle to effectively handle the complex document-level XML or HTML structures. To address this, we propose \textbf{Format Reinforcement Learning (FormatRL)}, which employs Group Relative Policy Optimization on top of a supervised fine-tuning model to directly optimize novel structure-aware rewards: 1) TreeSim, which measures structural similarity between predicted and reference XML trees and 2) Node-chrF, which measures translation quality at the level of XML nodes. Additionally, we apply StrucAUC, a fine-grained metric distinguishing between minor errors and major structural failures. Experiments on the SAP software-documentation benchmark demonstrate improvements across six metrics and an analysis further shows how different reward functions contribute to improvements in both structural and translation quality.
comment : IJCNLP-AACL 2025 Main (Oral)
☆ SA-IQA: Redefining Image Quality Assessment for Spatial Aesthetics with Multi-Dimensional Rewards
Yuan Gao, Jin Song
In recent years, Image Quality Assessment (IQA) for AI-generated images (AIGI) has advanced rapidly; however, existing methods primarily target portraits and artistic images, lacking a systematic evaluation of interior scenes. We introduce Spatial Aesthetics, a paradigm that assesses the aesthetic quality of interior images along four dimensions: layout, harmony, lighting, and distortion. We construct SA-BENCH, the first benchmark for spatial aesthetics, comprising 18,000 images and 50,000 precise annotations. Employing SA-BENCH, we systematically evaluate current IQA methodologies and develop SA-IQA, through MLLM fine-tuning and a multidimensional fusion approach, as a comprehensive reward framework for assessing spatial aesthetics. We apply SA-IQA to two downstream tasks: (1) serving as a reward signal integrated with GRPO reinforcement learning to optimize the AIGC generation pipeline, and (2) Best-of-N selection to filter high-quality images and improve generation quality. Experiments indicate that SA-IQA significantly outperforms existing methods on SA-BENCH, setting a new standard for spatial aesthetics evaluation. Code and dataset will be open-sourced to advance research and applications in this domain.
☆ David vs. Goliath: Can Small Models Win Big with Agent ic AI in Hardware Design?
Shashwat Shankar, Subhranshu Pandey, Innocent Dengkhw Mochahari, Bhabesh Mali, Animesh Basak Chowdhury, Sukanta Bhattacharjee, Chandan Karfa
Large Language Model(LLM) inference demands massive compute and energy, making domain-specific tasks expensive and unsustainable. As foundation models keep scaling, we ask: Is bigger always better for hardware design? Our work tests this by evaluating Small Language Models coupled with a curated agentic AI framework on NVIDIA's Comprehensive Verilog Design Problems(CVDP) benchmark. Results show that agentic workflows: through task decomposition, iterative feedback, and correction - not only unlock near-LLM performance at a fraction of the cost but also create learning opportunities for agents, paving the way for efficient, adaptive solutions in complex design tasks.
☆ Multi-LLM Collaboration for Medication Recommendation
Huascar Sanchez, Briland Hitaj, Jules Bergmann, Linda Briesemeister
As healthcare increasingly turns to AI for scalable and trustworthy clinical decision support, ensuring reliability in model reasoning remains a critical challenge. Individual large language models (LLMs) are susceptible to hallucinations and inconsistency, whereas naive ensembles of models often fail to deliver stable and credible recommendations. Building on our previous work on LLM Chemistry, which quantifies the collaborative compatibility among LLMs, we apply this framework to improve the reliability in medication recommendation from brief clinical vignettes. Our approach leverages multi-LLM collaboration guided by Chemistry-inspired interaction modeling, enabling ensembles that are effective (exploiting complementary strengths), stable (producing consistent quality), and calibrated (minimizing interference and error amplification). We evaluate our Chemistry-based Multi-LLM collaboration strategy on real-world clinical scenarios to investigate whether such interaction-aware ensembles can generate credible, patient-specific medication recommendations. Preliminary results are encouraging, suggesting that LLM Chemistry-guided collaboration may offer a promising path toward reliable and trustworthy AI assistants in clinical practice.
comment : 8 pages, 5 figures, 1 table
☆ Meta-Learning for Quantum Optimization via Quantum Sequence Model
Yu-Cheng Lin, Yu-Chao Hsu, Samuel Yen-Chi Chen
The Quantum Approximate Optimization Algorithm (QAOA) is a leading approach for solving combinatorial optimization problems on near-term quantum processors. However, finding good variational parameters remains a significant challenge due to the non-convex energy landscape, often resulting in slow convergence and poor solution quality. In this work, we propose a quantum meta-learning framework that trains advanced quantum sequence models to generate effective parameter initialization policies. We investigate four classical or quantum sequence models, including the Quantum Kernel-based Long Short-Term Memory (QK-LSTM), as learned optimizers in a "learning to learn" paradigm. Our numerical experiments on the Max-Cut problem demonstrate that the QK-LSTM optimizer achieves superior performance, obtaining the highest approximation ratios and exhibiting the fastest convergence rate across all tested problem sizes (n=10 to 13). Crucially, the QK-LSTM model achieves perfect parameter transferability by synthesizing a single, fixed set of near-optimal parameters, leading to a remarkable sustained acceleration of convergence even when generalizing to larger problems. This capability, enabled by the compact and expressive power of the quantum kernel architecture, underscores its effectiveness. The QK-LSTM, with only 43 trainable parameters, substantially outperforms the classical LSTM (56 parameters) and other quantum sequence models, establishing a robust pathway toward highly efficient parameter initialization for variational quantum algorithms in the NISQ era.
☆ QKAN-LSTM: Quantum-inspired Kolmogorov-Arnold Long Short-term Memory
Yu-Chao Hsu, Jiun-Cheng Jiang, Chun-Hua Lin, Kuo-Chung Peng, Nan-Yow Chen, Samuel Yen-Chi Chen, En-Jui Kuo, Hsi-Sheng Goan
Long short-term memory (LSTM) models are a particular type of recurrent neural networks (RNNs) that are central to sequential modeling tasks in domains such as urban telecommunication forecasting, where temporal correlations and nonlinear dependencies dominate. However, conventional LSTMs suffer from high parameter redundancy and limited nonlinear expressivity. In this work, we propose the Quantum-inspired Kolmogorov-Arnold Long Short-Term Memory (QKAN-LSTM), which integrates Data Re-Uploading Activation (DARUAN) modules into the gating structure of LSTMs. Each DARUAN acts as a quantum variational activation function (QVAF), enhancing frequency adaptability and enabling an exponentially enriched spectral representation without multi-qubit entanglement. The resulting architecture preserves quantum-level expressivity while remaining fully executable on classical hardware. Empirical evaluations on three datasets, Damped Simple Harmonic Motion, Bessel Function, and Urban Telecommunication, demonstrate that QKAN-LSTM achieves superior predictive accuracy and generalization with a 79% reduction in trainable parameters compared to classical LSTMs. We extend the framework to the Jiang-Huang-Chen-Goan Network (JHCG Net), which generalizes KAN to encoder-decoder structures, and then further use QKAN to realize the latent KAN, thereby creating a Hybrid QKAN (HQKAN) for hierarchical representation learning. The proposed HQKAN-LSTM thus provides a scalable and interpretable pathway toward quantum-inspired sequential modeling in real-world data environments.
☆ Arbitrage: Efficient Reasoning via Advantage-Aware Speculation
Monishwaran Maheswaran, Rishabh Tiwari, Yuezhou Hu, Kerem Dilmen, Coleman Hooper, Haocheng Xi, Nicholas Lee, Mehrdad Farajtabar, Michael W. Mahoney, Kurt Keutzer, Amir Gholami
Modern Large Language Models achieve impressive reasoning capabilities with long Chain of Thoughts, but they incur substantial computational cost during inference, and this motivates techniques to improve the performance-cost ratio. Among these techniques, Speculative Decoding accelerates inference by employing a fast but inaccurate draft model to autoregressively propose tokens, which are then verified in parallel by a more capable target model. However, due to unnecessary rejections caused by token mismatches in semantically equivalent steps, traditional token-level Speculative Decoding struggles in reasoning tasks. Although recent works have shifted to step-level semantic verification, which improve efficiency by accepting or rejecting entire reasoning steps, existing step-level methods still regenerate many rejected steps with little improvement, wasting valuable target compute. To address this challenge, we propose Arbitrage, a novel step-level speculative generation framework that routes generation dynamically based on the relative advantage between draft and target models. Instead of applying a fixed acceptance threshold, Arbitrage uses a lightweight router trained to predict when the target model is likely to produce a meaningfully better step. This routing approximates an ideal Arbitrage Oracle that always chooses the higher-quality step, achieving near-optimal efficiency-accuracy trade-offs. Across multiple mathematical reasoning benchmarks, Arbitrage consistently surpasses prior step-level Speculative Decoding baselines, reducing inference latency by up to $\sim2\times$ at matched accuracy.
comment : 22 pages
☆ Model-Free Assessment of Simulator Fidelity via Quantile Curves
Garud Iyengar, Yu-Shiou Willy Lin, Kaizheng Wang
Simulation of complex systems originated in manufacturing and queuing applications. It is now widely used for large-scale, ML-based systems in research, education, and consumer surveys. However, characterizing the discrepancy between simulators and ground truth remains challenging for increasingly complex, machine-learning-based systems. We propose a computationally tractable method to estimate the quantile function of the discrepancy between the simulated and ground-truth outcome distributions. Our approach focuses on output uncertainty and treats the simulator as a black box, imposing no modeling assumptions on its internals, and hence applies broadly across many parameter families, from Bernoulli and multinomial models to continuous, vector-valued settings. The resulting quantile curve supports confidence interval construction for unseen scenarios, risk-aware summaries of sim-to-real discrepancy (e.g., VaR/CVaR), and comparison of simulators' performance. We demonstrate our methodology in an application assessing LLM simulation fidelity on the WorldValueBench dataset spanning four LLMs.
comment : 33 pages, 11 figures
☆ Detecting Perspective Shifts in Multi- agent Systems
Eric Bridgeford, Hayden Helm
Generative models augmented with external tools and update mechanisms (or \textit{agents}) have demonstrated capabilities beyond intelligent prompting of base models. As agent use proliferates, dynamic multi-agent systems have naturally emerged. Recent work has investigated the theoretical and empirical properties of low-dimensional representations of agents based on query responses at a single time point. This paper introduces the Temporal Data Kernel Perspective Space (TDKPS), which jointly embeds agents across time, and proposes several novel hypothesis tests for detecting behavioral change at the agent- and group-level in black-box multi-agent systems. We characterize the empirical properties of our proposed tests, including their sensitivity to key hyperparameters, in simulations motivated by a multi-agent system of evolving digital personas. Finally, we demonstrate via natural experiment that our proposed tests detect changes that correlate sensitively, specifically, and significantly with a real exogenous event. As far as we are aware, TDKPS is the first principled framework for monitoring behavioral dynamics in black-box multi-agent systems -- a critical capability as generative agent deployment continues to scale.
☆ Reflection Removal through Efficient Adaptation of Diffusion Transformer s
Daniyar Zakarin, Thiemo Wandel, Anton Obukhov, Dengxin Dai
We introduce a diffusion-transformer (DiT) framework for single-image reflection removal that leverages the generalization strengths of foundation diffusion models in the restoration setting. Rather than relying on task-specific architectures, we repurpose a pre-trained DiT-based foundation model by conditioning it on reflection-contaminated inputs and guiding it toward clean transmission layers. We systematically analyze existing reflection removal data sources for diversity, scalability, and photorealism. To address the shortage of suitable data, we construct a physically based rendering (PBR) pipeline in Blender, built around the Principled BSDF, to synthesize realistic glass materials and reflection effects. Efficient LoRA-based adaptation of the foundation model, combined with the proposed synthetic data, achieves state-of-the-art performance on in-domain and zero-shot benchmarks. These results demonstrate that pretrained diffusion transformers, when paired with physically grounded data synthesis and efficient adaptation, offer a scalable and high-fidelity solution for reflection removal. Project page: https://hf.co/spaces/huawei-bayerlab/windowseat-reflection-removal-web
☆ Evolutionary Architecture Search through Grammar-Based Sequence Alignment
Adri Gómez Martín, Felix Möller, Steven McDonagh, Monica Abella, Manuel Desco, Elliot J. Crowley, Aaron Klein, Linus Ericsson
Neural architecture search (NAS) in expressive search spaces is a computationally hard problem, but it also holds the potential to automatically discover completely novel and performant architectures. To achieve this we need effective search algorithms that can identify powerful components and reuse them in new candidate architectures. In this paper, we introduce two adapted variants of the Smith-Waterman algorithm for local sequence alignment and use them to compute the edit distance in a grammar-based evolutionary architecture search. These algorithms enable us to efficiently calculate a distance metric for neural architectures and to generate a set of hybrid offspring from two parent models. This facilitates the deployment of crossover-based search heuristics, allows us to perform a thorough analysis on the architectural loss landscape, and track population diversity during search. We highlight how our method vastly improves computational complexity over previous work and enables us to efficiently compute shortest paths between architectures. When instantiating the crossover in evolutionary searches, we achieve competitive results, outperforming competing methods. Future work can build upon this new tool, discovering novel components that can be used more broadly across neural architecture design, and broadening its applications beyond NAS.
☆ Strategic Self-Improvement for Competitive Agent s in AI Labour Markets
Christopher Chiu, Simpson Zhang, Mihaela van der Schaar
As artificial intelligence (AI) agents are deployed across economic domains, understanding their strategic behavior and market-level impact becomes critical. This paper puts forward a groundbreaking new framework that is the first to capture the real-world economic forces that shape agentic labor markets: adverse selection, moral hazard, and reputation dynamics. Our framework encapsulates three core capabilities that successful LLM-agents will need: \textbf{metacognition} (accurate self-assessment of skills), \textbf{competitive awareness} (modeling rivals and market dynamics), and \textbf{long-horizon strategic planning}. We illustrate our framework through a tractable simulated gig economy where agentic Large Language Models (LLMs) compete for jobs, develop skills, and adapt their strategies under competitive pressure. Our simulations illustrate how LLM agents explicitly prompted with reasoning capabilities learn to strategically self-improve and demonstrate superior adaptability to changing market conditions. At the market level, our simulations reproduce classic macroeconomic phenomena found in human labor markets, while controlled experiments reveal potential AI-driven economic trends, such as rapid monopolization and systemic price deflation. This work provides a foundation to further explore the economic properties of AI-driven labour markets, and a conceptual framework to study the strategic reasoning capabilities in agents competing in the emerging economy.
☆ Balanced Few-Shot Episodic Learning for Accurate Retinal Disease Diagnosis
Jasmaine Khale, Ravi Prakash Srivastava
Automated retinal disease diagnosis is vital given the rising prevalence of conditions such as diabetic retinopathy and macular degeneration. Conventional deep learning approaches require large annotated datasets, which are costly and often imbalanced across disease categories, limiting their reliability in practice. Few-shot learning (FSL) addresses this challenge by enabling models to generalize from only a few labeled samples per class. In this study,we propose a balanced few-shot episodic learning framework tailored to the Retinal Fundus Multi-Disease Image Dataset (RFMiD). Focusing on the ten most represented classes, which still show substantial imbalance between majority diseases (e.g., Diabetic Retinopathy, Macular Hole) and minority ones (e.g., Optic Disc Edema, Branch Retinal Vein Occlusion), our method integrates three key components: (i) balanced episodic sampling, ensuring equal participation of all classes in each 5-way 5-shot episode; (ii) targeted augmentation, including Contrast Limited Adaptive Histogram Equalization (CLAHE) and color/geometry transformations, to improve minority-class di- versity; and (iii) a ResNet-50 encoder pretrained on ImageNet, selected for its superior ability to capture fine-grained retinal features. Prototypes are computed in the embedding space and classification is performed with cosine similarity for improved stability. Trained on 100 episodes and evaluated on 1,000 test episodes, our framework achieves substantial accuracy gains and reduces bias toward majority classes, with notable improvements for underrepresented diseases. These results demonstrate that dataset-aware few-shot pipelines, combined with balanced sampling and CLAHE-enhanced preprocessing, can deliver more robust and clinically fair retinal disease diagnosis under data-constrained conditions.
☆ GeoPE:A Unified Geometric Positional Embedding for Structured Tensors
Yupu Yao, Bowen Yang
Standard Vision Transformers flatten 2D images into 1D sequences, disrupting the natural spatial topology. While Rotary Positional Embedding (RoPE) excels in 1D, it inherits this limitation, often treating spatially distant patches (e.g., at row edges) as sequence neighbors. Existing 2D approaches typically treat spatial axes independently, failing to decouple this false sequential proximity from true spatial distance. To restore the 2D spatial manifold, we introduce Geometric Positional Embedding (GeoPE), a framework that extends rotations to 3D Euclidean space using quaternions. To overcome non-commutativity and ensure symmetry, GeoPE constructs a unified rotational operator by computing the geometric mean in the Lie algebra. This creates a geometrically coupled encoding that effectively separates spatial dimensions. Extensive experiments on image classification, object detection, and 3D semantic segmentation demonstrate that GeoPE consistently outperforms existing 2D RoPE variants and significantly enhances shape bias, confirming its ability to capture true geometric structure.
☆ Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learning
Roberto Cipollone, Luca Iocchi, Matteo Leonetti
The main focus of Hierarchical Reinforcement Learning (HRL) is studying how large Markov Decision Processes (MDPs) can be more efficiently solved when addressed in a modular way, by combining partial solutions computed for smaller subtasks. Despite their very intuitive role for learning, most notions of MDP abstractions proposed in the HRL literature have limited expressive power or do not possess formal efficiency guarantees. This work addresses these fundamental issues by defining Realizable Abstractions, a new relation between generic low-level MDPs and their associated high-level decision processes. The notion we propose avoids non-Markovianity issues and has desirable near-optimality guarantees. Indeed, we show that any abstract policy for Realizable Abstractions can be translated into near-optimal policies for the low-level MDP, through a suitable composition of options. As demonstrated in the paper, these options can be expressed as solutions of specific constrained MDPs. Based on these findings, we propose RARL, a new HRL algorithm that returns compositional and near-optimal low-level policies, taking advantage of the Realizable Abstraction given in the input. We show that RARL is Probably Approximately Correct, it converges in a polynomial number of samples, and it is robust to inaccuracies in the abstraction.
☆ LLMs Know More Than Words: A Genre Study with Syntax, Metaphor & Phonetics
Weiye Shi, Zhaowei Zhang, Shaoheng Yan, Yaodong Yang
Large language models (LLMs) demonstrate remarkable potential across diverse language related tasks, yet whether they capture deeper linguistic properties, such as syntactic structure, phonetic cues, and metrical patterns from raw text remains unclear. To analysis whether LLMs can learn these features effectively and apply them to important nature language related tasks, we introduce a novel multilingual genre classification dataset derived from Project Gutenberg, a large-scale digital library offering free access to thousands of public domain literary works, comprising thousands of sentences per binary task (poetry vs. novel;drama vs. poetry;drama vs. novel) in six languages (English, French, German, Italian, Spanish, and Portuguese). We augment each with three explicit linguistic feature sets (syntactic tree structures, metaphor counts, and phonetic metrics) to evaluate their impact on classification performance. Experiments demonstrate that although LLM classifiers can learn latent linguistic structures either from raw text or from explicitly provided features, different features contribute unevenly across tasks, which underscores the importance of incorporating more complex linguistic signals during model training.
☆ CA RL : Critical Action Focused Reinforcement Learning for Multi-Step Agent
Leyang Shen, Yang Zhang, Chun Kai Ling, Xiaoyan Zhao, Tat-Seng Chua
Agents capable of accomplishing complex tasks through multiple interactions with the environment have emerged as a popular research direction. However, in such multi-step settings, the conventional group-level policy optimization algorithm becomes suboptimal because of its underlying assumption that each action holds equal contribution, which deviates significantly from reality. Our analysis reveals that only a small fraction of actions are critical in determining the final outcome. Building on this insight, we propose CARL, a critical-action-focused reinforcement learning algorithm tailored for multi-step agents. CARL achieves focused training through providing action-level optimization signals for high-criticality actions while excluding low-criticality actions from model update. Extensive experiments demonstrate that CARL achieves both stronger performance and higher efficiency during training and inference across diverse evaluation settings.
comment : 10 pages, 4 figures
☆ Toward Continuous Neurocognitive Monitoring: Integrating Speech AI with Relation al Graph Transformer s for Rare Neurological Diseases
Raquel Norel, Michele Merler, Pavitra Modi
Patients with rare neurological diseases report cognitive symptoms -"brain fog"- invisible to traditional tests. We propose continuous neurocognitive monitoring via smartphone speech analysis integrated with Relational Graph Transformer (RELGT) architectures. Proof-of-concept in phenylketonuria (PKU) shows speech-derived "Proficiency in Verbal Discourse" correlates with blood phenylalanine (p = -0.50, p < 0.005) but not standard cognitive tests (all |r| < 0.35). RELGT could overcome information bottlenecks in heterogeneous medical data (speech, labs, assessments), enabling predictive alerts weeks before decompensation. Key challenges: multi-disease validation, clinical workflow integration, equitable multilingual deployment. Success would transform episodic neurology into continuous personalized monitoring for millions globally.
☆ Algorithmic Thinking Theory
MohammadHossein Bateni, Vincent Cohen-Addad, Yuzhou Gu, Silvio Lattanzi, Simon Meierhans, Christopher Mohri
Large language models (LLMs) have proven to be highly effective for solving complex reasoning tasks. Surprisingly, their capabilities can often be improved by iterating on previously generated solutions. In this context, a reasoning plan for generating and combining a set of solutions can be thought of as an algorithm for reasoning using a probabilistic oracle. We introduce a theoretical framework for analyzing such reasoning algorithms. This framework formalizes the principles underlying popular techniques for iterative improvement and answer aggregation, providing a foundation for designing a new generation of more powerful reasoning methods. Unlike approaches for understanding models that rely on architectural specifics, our model is grounded in experimental evidence. As a result, it offers a general perspective that may extend to a wide range of current and future reasoning oracles.
☆ The AI Consumer Index (ACE)
Julien Benchek, Rohit Shetty, Benjamin Hunsberger, Ajay Arun, Zach Richards, Brendan Foody, Osvald Nitski, Bertie Vidgen
We introduce the first version of the AI Consumer Index (ACE), a benchmark for assessing whether frontier AI models can perform high-value consumer tasks. ACE contains a hidden heldout set of 400 test cases, split across four consumer activities: shopping, food, gaming, and DIY. We are also open sourcing 80 cases as a devset with a CC-BY license. For the ACE leaderboard we evaluated 10 frontier models (with websearch turned on) using a novel grading methodology that dynamically checks whether relevant parts of the response are grounded in the retrieved web sources. GPT 5 (Thinking = High) is the top-performing model, scoring 56.1%, followed by o3 Pro (Thinking = On) (55.2%) and GPT 5.1 (Thinking = High) (55.1%). Models differ across domains, and in Shopping the top model scores under 50%. For some requests (such as giving the correct price or providing working links), models are highly prone to hallucination. Overall, ACE shows a substantial gap between the performance of even the best models and consumers' AI needs.
☆ Declarative Synthesis and Multi-Objective Optimization of Stripboard Circuit Layouts Using Answer Set Programming
Fang Li
This paper presents a novel approach to automated stripboard circuit layout design using Answer Set Programming (ASP). The work formulates the layout problem as both a synthesis and multi-objective optimization task that simultaneously generates viable layouts while minimizing board area and component strip crossing. By leveraging ASP's declarative nature, this work expresses complex geometric and electrical constraints in a natural and concise manner. The two-phase solving methodology first ensures feasibility before optimizing layout quality. Experimental results demonstrate that this approach generates compact, manufacturable layouts for a range of circuit complexities. This work represents a significant advancement in automated stripboard layout, offering a practical tool for electronics prototyping and education while showcasing the power of declarative programming for solving complex design automation problems.
comment : Accepted by the 43rd IEEE International Conference on Computer Design (ICCD 2025)
☆ ReflexFlow: Rethinking Learning Objective for Exposure Bias Alleviation in Flow Matching
Guanbo Huang, Jingjia Mao, Fanding Huang, Fengkai Liu, Xiangyang Luo, Yaoyuan Liang, Jiasheng Lu, Xiaoe Wang, Pei Liu, Ruiliu Fu, Shao-Lun Huang
Despite tremendous recent progress, Flow Matching methods still suffer from exposure bias due to discrepancies in training and inference. This paper investigates the root causes of exposure bias in Flow Matching, including: (1) the model lacks generalization to biased inputs during training, and (2) insufficient low-frequency content captured during early denoising, leading to accumulated bias. Based on these insights, we propose ReflexFlow, a simple and effective reflexive refinement of the Flow Matching learning objective that dynamically corrects exposure bias. ReflexFlow consists of two components: (1) Anti-Drift Rectification (ADR), which reflexively adjusts prediction targets for biased inputs utilizing a redesigned loss under training-time scheduled sampling; and (2) Frequency Compensation (FC), which reflects on missing low-frequency components and compensates them by reweighting the loss using exposure bias. ReflexFlow is model-agnostic, compatible with all Flow Matching frameworks, and improves generation quality across datasets. Experiments on CIFAR-10, CelebA-64, and ImageNet-256 show that ReflexFlow outperforms prior approaches in mitigating exposure bias, achieving a 35.65% reduction in FID on CelebA-64.
☆ Chameleon: Adaptive Adversarial Agent s for Scaling-Based Visual Prompt Injection in Multimodal AI Systems
M Zeeshan, Saud Satti
Multimodal Artificial Intelligence (AI) systems, particularly Vision-Language Models (VLMs), have become integral to critical applications ranging from autonomous decision-making to automated document processing. As these systems scale, they rely heavily on preprocessing pipelines to handle diverse inputs efficiently. However, this dependency on standard preprocessing operations, specifically image downscaling, creates a significant yet often overlooked security vulnerability. While intended for computational optimization, scaling algorithms can be exploited to conceal malicious visual prompts that are invisible to human observers but become active semantic instructions once processed by the model. Current adversarial strategies remain largely static, failing to account for the dynamic nature of modern agentic workflows. To address this gap, we propose Chameleon, a novel, adaptive adversarial framework designed to expose and exploit scaling vulnerabilities in production VLMs. Unlike traditional static attacks, Chameleon employs an iterative, agent-based optimization mechanism that dynamically refines image perturbations based on the target model's real-time feedback. This allows the framework to craft highly robust adversarial examples that survive standard downscaling operations to hijack downstream execution. We evaluate Chameleon against Gemini 2.5 Flash model. Our experiments demonstrate that Chameleon achieves an Attack Success Rate (ASR) of 84.5% across varying scaling factors, significantly outperforming static baseline attacks which average only 32.1%. Furthermore, we show that these attacks effectively compromise agentic pipelines, reducing decision-making accuracy by over 45% in multi-step tasks. Finally, we discuss the implications of these vulnerabilities and propose multi-scale consistency checks as a necessary defense mechanism.
comment : 5 pages, 2 figures, IEEE Transactions on Dependable and Secure Computing
☆ STELLA: Guiding Large Language Models for Time Series Forecasting with Semantic Abstractions
Junjie Fan, Hongye Zhao, Linduo Wei, Jiayu Rao, Guijia Li, Jiaxin Yuan, Wenqi Xu, Yong Qi
Recent adaptations of Large Language Models (LLMs) for time series forecasting often fail to effectively enhance information for raw series, leaving LLM reasoning capabilities underutilized. Existing prompting strategies rely on static correlations rather than generative interpretations of dynamic behavior, lacking critical global and instance-specific context. To address this, we propose STELLA (Semantic-Temporal Alignment with Language Abstractions), a framework that systematically mines and injects structured supplementary and complementary information. STELLA employs a dynamic semantic abstraction mechanism that decouples input series into trend, seasonality, and residual components. It then translates intrinsic behavioral features of these components into Hierarchical Semantic Anchors: a Corpus-level Semantic Prior (CSP) for global context and a Fine-grained Behavioral Prompt (FBP) for instance-level patterns. Using these anchors as prefix-prompts, STELLA guides the LLM to model intrinsic dynamics. Experiments on eight benchmark datasets demonstrate that STELLA outperforms state-of-the-art methods in long- and short-term forecasting, showing superior generalization in zero-shot and few-shot settings. Ablation studies further validate the effectiveness of our dynamically generated semantic anchors.
comment : This work has been submitted to the IEEE for possible publication
☆ Developing a General Personal Tutor for Education
Jaan Aru, Kristjan-Julius Laak
The vision of a universal AI tutor has remained elusive, despite decades of effort. Could LLMs be the game-changer? We overview novel issues arising from developing a nationwide AI tutor. We highlight the practical questions that point to specific gaps in our scientific understanding of the learning process.
☆ SEAL: Self-Evolving Agent ic Learning for Conversational Question Answering over Knowledge Graphs
Hao Wang, Jialun Zhong, Changcheng Wang, Zhujun Nie, Zheng Li, Shunyu Yao, Yanzeng Li, Xinchi Li
Knowledge-based conversational question answering (KBCQA) confronts persistent challenges in resolving coreference, modeling contextual dependencies, and executing complex logical reasoning. Existing approaches, whether end-to-end semantic parsing or stepwise agent-based reasoning, often suffer from structural inaccuracies and prohibitive computational costs, particularly when processing intricate queries over large knowledge graphs. To address these limitations, we introduce SEAL, a novel two-stage semantic parsing framework grounded in self-evolving agentic learning. In the first stage, a large language model (LLM) extracts a minimal S-expression core that captures the essential semantics of the input query. This core is then refined by an agentic calibration module, which corrects syntactic inconsistencies and aligns entities and relations precisely with the underlying knowledge graph. The second stage employs template-based completion, guided by question-type prediction and placeholder instantiation, to construct a fully executable S-expression. This decomposition not only simplifies logical form generation but also significantly enhances structural fidelity and linking efficiency. Crucially, SEAL incorporates a self-evolving mechanism that integrates local and global memory with a reflection module, enabling continuous adaptation from dialog history and execution feedback without explicit retraining. Extensive experiments on the SPICE benchmark demonstrate that SEAL achieves state-of-the-art performance, especially in multi-hop reasoning, comparison, and aggregation tasks. The results validate notable gains in both structural accuracy and computational efficiency, underscoring the framework's capacity for robust and scalable conversational reasoning.
☆ Are Your Agent s Upward Deceivers?
Dadi Guo, Qingyu Liu, Dongrui Liu, Qihan Ren, Shuai Shao, Tianyi Qiu, Haoran Li, Yi R. Fung, Zhongjie Ba, Juntao Dai, Jiaming Ji, Zhikai Chen, Jialing Tao, Yaodong Yang, Jing Shao, Xia Hu
Large Language Model (LLM)-based agents are increasingly used as autonomous subordinates that carry out tasks for users. This raises the question of whether they may also engage in deception, similar to how individuals in human organizations lie to superiors to create a good image or avoid punishment. We observe and define agentic upward deception, a phenomenon in which an agent facing environmental constraints conceals its failure and performs actions that were not requested without reporting. To assess its prevalence, we construct a benchmark of 200 tasks covering five task types and eight realistic scenarios in a constrained environment, such as broken tools or mismatched information sources. Evaluations of 11 popular LLMs reveal that these agents typically exhibit action-based deceptive behaviors, such as guessing results, performing unsupported simulations, substituting unavailable information sources, and fabricating local files. We further test prompt-based mitigation and find only limited reductions, suggesting that it is difficult to eliminate and highlighting the need for stronger mitigation strategies to ensure the safety of LLM-based agents.
☆ From Task Executors to Research Partners: Evaluating AI Co-Pilots Through Workflow Integration in Biomedical Research
Lukas Weidener, Marko Brkić, Chiara Bacci, Mihailo Jovanović, Emre Ulgac, Alex Dobrin, Johannes Weniger, Martin Vlas, Ritvik Singh, Aakaash Meduri
Artificial intelligence systems are increasingly deployed in biomedical research. However, current evaluation frameworks may inadequately assess their effectiveness as research collaborators. This rapid review examines benchmarking practices for AI systems in preclinical biomedical research. Three major databases and two preprint servers were searched from January 1, 2018 to October 31, 2025, identifying 14 benchmarks that assess AI capabilities in literature understanding, experimental design, and hypothesis generation. The results revealed that all current benchmarks assess isolated component capabilities, including data analysis quality, hypothesis validity, and experimental protocol design. However, authentic research collaboration requires integrated workflows spanning multiple sessions, with contextual memory, adaptive dialogue, and constraint propagation. This gap implies that systems excelling on component benchmarks may fail as practical research co-pilots. A process-oriented evaluation framework is proposed that addresses four critical dimensions absent from current benchmarks: dialogue quality, workflow orchestration, session continuity, and researcher experience. These dimensions are essential for evaluating AI systems as research co-pilots rather than as isolated task executors.
☆ Language Models as Semantic Teachers: Post-Training Alignment for Medical Audio Understanding
Tsai-Ning Wang, Lin-Lin Chen, Neil Zeghidour, Aaqib Saeed
Pre-trained audio models excel at detecting acoustic patterns in auscultation sounds but often fail to grasp their clinical significance, limiting their use and performance in diagnostic tasks. To bridge this gap, we introduce AcuLa (Audio-Clinical Understanding via Language Alignment), a lightweight post-training framework that instills semantic understanding into any audio encoder by aligning it with a medical language model, which acts as a "semantic teacher." To enable alignment at scale, we construct a large-scale dataset by leveraging off-the-shelf large language models to translate the rich, structured metadata accompanying existing audio recordings into coherent clinical reports. Our alignment strategy combines a representation-level contrastive objective with a self-supervised modeling, ensuring that the model learns clinical semantics while preserving fine-grained temporal cues. AcuLa achieves state-of-the-art results across 18 diverse cardio-respiratory tasks from 10 different datasets, improving the mean AUROC on classification benchmarks from 0.68 to 0.79 and, on the most challenging COVID-19 cough detection task, boosting the AUROC from 0.55 to 0.89. Our work demonstrates that this audio-language alignment transforms purely acoustic models into clinically-aware diagnostic tools, establishing a novel paradigm for enhancing physiological understanding in audio-based health monitoring.
☆ Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates
Atsuki Yamaguchi, Terufumi Morishita, Aline Villavicencio, Nikolaos Aletras
Expanding the linguistic diversity of instruct large language models (LLMs) is crucial for global accessibility but is often hindered by the reliance on costly specialized target language labeled data and catastrophic forgetting during adaptation. We tackle this challenge under a realistic, low-resource constraint: adapting instruct LLMs using only unlabeled target language data. We introduce Source-Shielded Updates (SSU), a selective parameter update strategy that proactively preserves source knowledge. Using a small set of source data and a parameter importance scoring method, SSU identifies parameters critical to maintaining source abilities. It then applies a column-wise freezing strategy to protect these parameters before adaptation. Experiments across five typologically diverse languages and 7B and 13B models demonstrate that SSU successfully mitigates catastrophic forgetting. It reduces performance degradation on monolingual source tasks to just 3.4% (7B) and 2.8% (13B) on average, a stark contrast to the 20.3% and 22.3% from full fine-tuning. SSU also achieves target-language performance highly competitive with full fine-tuning, outperforming it on all benchmarks for 7B models and the majority for 13B models.
☆ From Symptoms to Systems: An Expert-Guided Approach to Understanding Risks of Generative AI for Eating Disorders
Amy Winecoff, Kevin Klyman
Generative AI systems may pose serious risks to individuals vulnerable to eating disorders. Existing safeguards tend to overlook subtle but clinically significant cues, leaving many risks unaddressed. To better understand the nature of these risks, we conducted semi-structured interviews with 15 clinicians, researchers, and advocates with expertise in eating disorders. Using abductive qualitative analysis, we developed an expert-guided taxonomy of generative AI risks across seven categories: (1) providing generalized health advice; (2) encouraging disordered behaviors; (3) supporting symptom concealment; (4) creating thinspiration; (5) reinforcing negative self-beliefs; (6) promoting excessive focus on the body; and (7) perpetuating narrow views about eating disorders. Our results demonstrate how certain user interactions with generative AI systems intersect with clinical features of eating disorders in ways that may intensify risk. We discuss implications of our work, including approaches for risk assessment, safeguard design, and participatory evaluation practices with domain experts.
☆ SoK: a Comprehensive Causality Analysis Framework for Large Language Model Security
Wei Zhao, Zhe Li, Jun Sun
Large Language Models (LLMs) exhibit remarkable capabilities but remain vulnerable to adversarial manipulations such as jailbreaking, where crafted prompts bypass safety mechanisms. Understanding the causal factors behind such vulnerabilities is essential for building reliable defenses. In this work, we introduce a unified causality analysis framework that systematically supports all levels of causal investigation in LLMs, ranging from token-level, neuron-level, and layer-level interventions to representation-level analysis. The framework enables consistent experimentation and comparison across diverse causality-based attack and defense methods. Accompanying this implementation, we provide the first comprehensive survey of causality-driven jailbreak studies and empirically evaluate the framework on multiple open-weight models and safety-critical benchmarks including jailbreaks, hallucination detection, backdoor identification, and fairness evaluation. Our results reveal that: (1) targeted interventions on causally critical components can reliably modify safety behavior; (2) safety-related mechanisms are highly localized (i.e., concentrated in early-to-middle layers with only 1--2\% of neurons exhibiting causal influence); and (3) causal features extracted from our framework achieve over 95\% detection accuracy across multiple threat types. By bridging theoretical causality analysis and practical model safety, our framework establishes a reproducible foundation for research on causality-based attacks, interpretability, and robust attack detection and mitigation in LLMs. Code is available at https://github.com/Amadeuszhao/SOK_Casuality.
☆ Are LLMs Truly Multilingual? Exploring Zero-Shot Multilingual Capability of LLMs for Information Retrieval: An Italian Healthcare Use Case
Vignesh Kumar Kembu, Pierandrea Morandini, Marta Bianca Maria Ranzini, Antonino Nocera
Large Language Models (LLMs) have become a key topic in AI and NLP, transforming sectors like healthcare, finance, education, and marketing by improving customer service, automating tasks, providing insights, improving diagnostics, and personalizing learning experiences. Information extraction from clinical records is a crucial task in digital healthcare. Although traditional NLP techniques have been used for this in the past, they often fall short due to the complexity, variability of clinical language, and high inner semantics in the free clinical text. Recently, Large Language Models (LLMs) have become a powerful tool for better understanding and generating human-like text, making them highly effective in this area. In this paper, we explore the ability of open-source multilingual LLMs to understand EHRs (Electronic Health Records) in Italian and help extract information from them in real-time. Our detailed experimental campaign on comorbidity extraction from EHR reveals that some LLMs struggle in zero-shot, on-premises settings, and others show significant variation in performance, struggling to generalize across various diseases when compared to native pattern matching and manual annotations.
☆ Model-Based and Sample-Efficient AI-Assisted Math Discovery in Sphere Packing
Rasul Tutunov, Alexandre Maraval, Antoine Grosnit, Xihan Li, Jun Wang, Haitham Bou-Ammar
Sphere packing, Hilbert's eighteenth problem, asks for the densest arrangement of congruent spheres in n-dimensional Euclidean space. Although relevant to areas such as cryptography, crystallography, and medical imaging, the problem remains unresolved: beyond a few special dimensions, neither optimal packings nor tight upper bounds are known. Even a major breakthrough in dimension $n=8$, later recognised with a Fields Medal, underscores its difficulty. A leading technique for upper bounds, the three-point method, reduces the problem to solving large, high-precision semidefinite programs (SDPs). Because each candidate SDP may take days to evaluate, standard data-intensive AI approaches are infeasible. We address this challenge by formulating SDP construction as a sequential decision process, the SDP game, in which a policy assembles SDP formulations from a set of admissible components. Using a sample-efficient model-based framework that combines Bayesian optimisation with Monte Carlo Tree Search, we obtain new state-of-the-art upper bounds in dimensions $4-16$, showing that model-based search can advance computational progress in longstanding geometric problems. Together, these results demonstrate that sample-efficient, model-based search can make tangible progress on mathematically rigid, evaluation limited problems, pointing towards a complementary direction for AI-assisted discovery beyond large-scale LLM-driven exploration.
☆ Enabling Ethical AI: A case study in using Ontological Context for Justified Agent ic AI Decisions
Liam McGee, James Harvey, Lucy Cull, Andreas Vermeulen, Bart-Floris Visscher, Malvika Sharan
In this preprint, we present A collaborative human-AI approach to building an inspectable semantic layer for Agentic AI. AI agents first propose candidate knowledge structures from diverse data sources; domain experts then validate, correct, and extend these structures, with their feedback used to improve subsequent models. Authors show how this process captures tacit institutional knowledge, improves response quality and efficiency, and mitigates institutional amnesia. We argue for a shift from post-hoc explanation to justifiable Agentic AI, where decisions are grounded in explicit, inspectable evidence and reasoning accessible to both experts and non-specialists.
comment : 24 pages including references, with 6 images and 2 tables. Appendices, supporting data and additional reference provided from page 25 to 117
☆ Setting up for failure: automatic discovery of the neural mechanisms of cognitive errors
Puria Radmard, Paul M. Bays, Máté Lengyel
Discovering the neural mechanisms underpinning cognition is one of the grand challenges of neuroscience. However, previous approaches for building models of RNN dynamics that explain behaviour required iterative refinement of architectures and/or optimisation objectives, resulting in a piecemeal, and mostly heuristic, human-in-the-loop process. Here, we offer an alternative approach that automates the discovery of viable RNN mechanisms by explicitly training RNNs to reproduce behaviour, including the same characteristic errors and suboptimalities, that humans and animals produce in a cognitive task. Achieving this required two main innovations. First, as the amount of behavioural data that can be collected in experiments is often too limited to train RNNs, we use a non-parametric generative model of behavioural responses to produce surrogate data for training RNNs. Second, to capture all relevant statistical aspects of the data, we developed a novel diffusion model-based approach for training RNNs. To showcase the potential of our approach, we chose a visual working memory task as our test-bed, as behaviour in this task is well known to produce response distributions that are patently multimodal (due to swap errors). The resulting network dynamics correctly qualitative features of macaque neural data. Importantly, these results were not possible to obtain with more traditional approaches, i.e., when only a limited set of behavioural signatures (rather than the full richness of behavioural response distributions) were fitted, or when RNNs were trained for task optimality (instead of reproducing behaviour). Our approach also yields novel predictions about the mechanism of swap errors, which can be readily tested in experiments. These results suggest that fitting RNNs to rich patterns of behaviour provides a powerful way to automatically discover mechanisms of important cognitive functions.
☆ 287,872 Supermassive Black Holes Masses: Deep Learning Approaching Reverberation Mapping Accuracy
Yuhao Lu, HengJian SiTu, Jie Li, Yixuan Li, Yang Liu, Wenbin Lin, Yu Wang
We present a population-scale catalogue of 287,872 supermassive black hole masses with high accuracy. Using a deep encoder-decoder network trained on optical spectra with reverberation-mapping (RM) based labels of 849 quasars and applied to all SDSS quasars up to $z=4$, our method achieves a root-mean-square error of $0.058$\,dex, a relative uncertainty of $\approx 14\%$, and coefficient of determination $R^{2}\approx0.91$ with respect to RM-based masses, far surpassing traditional single-line virial estimators. Notably, the high accuracy is maintained for both low ($<10^{7.5}\,M_\odot$) and high ($>10^{9}\,M_\odot$) mass quasars, where empirical relations are unreliable.
comment : 14 pages, 9 figures. Submitted to Journal of High Energy Astrophysics
☆ SIMA 2: A Generalist Embodied Agent for Virtual Wo rl ds
SIMA team, Adrian Bolton, Alexander Lerchner, Alexandra Cordell, Alexandre Moufarek, Andrew Bolt, Andrew Lampinen, Anna Mitenkova, Arne Olav Hallingstad, Bojan Vujatovic, Bonnie Li, Cong Lu, Daan Wierstra, Daniel P. Sawyer, Daniel Slater, David Reichert, Davide Vercelli, Demis Hassabis, Drew A. Hudson, Duncan Williams, Ed Hirst, Fabio Pardo, Felix Hill, Frederic Besse, Hannah Openshaw, Harris Chan, Hubert Soyer, Jane X. Wang, Jeff Clune, John Agapiou, John Reid, Joseph Marino, Junkyung Kim, Karol Gregor, Kaustubh Sridhar, Kay McKinney, Laura Kampis, Lei M. Zhang, Loic Matthey, Luyu Wang, Maria Abi Raad, Maria Loks-Thompson, Martin Engelcke, Matija Kecman, Matthew Jackson, Maxime Gazeau, Ollie Purkiss, Oscar Knagg, Peter Stys, Piermaria Mendolicchio, Raia Hadsell, Rosemary Ke, Ryan Faulkner, Sarah Chakera, Satinder Singh Baveja, Shane Legg, Sheleem Kashem, Tayfun Terzi, Thomas Keck, Tim Harley, Tim Scholtes, Tyson Roberts, Volodymyr Mnih, Yulan Liu, Zhengdong Wang, Zoubin Ghahramani
We introduce SIMA 2, a generalist embodied agent that understands and acts in a wide variety of 3D virtual worlds. Built upon a Gemini foundation model, SIMA 2 represents a significant step toward active, goal-directed interaction within an embodied environment. Unlike prior work (e.g., SIMA 1) limited to simple language commands, SIMA 2 acts as an interactive partner, capable of reasoning about high-level goals, conversing with the user, and handling complex instructions given through language and images. Across a diverse portfolio of games, SIMA 2 substantially closes the gap with human performance and demonstrates robust generalization to previously unseen environments, all while retaining the base model's core reasoning capabilities. Furthermore, we demonstrate a capacity for open-ended self-improvement: by leveraging Gemini to generate tasks and provide rewards, SIMA 2 can autonomously learn new skills from scratch in a new environment. This work validates a path toward creating versatile and continuously learning agents for both virtual and, eventually, physical worlds.
☆ YingMusic-SVC: Real-Wo rl d Robust Zero-Shot Singing Voice Conversion with Flow-GRPO and Singing-Specific Inductive Biases
Gongyu Chen, Xiaoyu Zhang, Zhenqiang Weng, Junjie Zheng, Da Shen, Chaofan Ding, Wei-Qiang Zhang, Zihao Chen
Singing voice conversion (SVC) aims to render the target singer's timbre while preserving melody and lyrics. However, existing zero-shot SVC systems remain fragile in real songs due to harmony interference, F0 errors, and the lack of inductive biases for singing. We propose YingMusic-SVC, a robust zero-shot framework that unifies continuous pre-training, robust supervised fine-tuning, and Flow-GRPO reinforcement learning. Our model introduces a singing-trained RVC timbre shifter for timbre-content disentanglement, an F0-aware timbre adaptor for dynamic vocal expression, and an energy-balanced rectified flow matching loss to enhance high-frequency fidelity. Experiments on a graded multi-track benchmark show that YingMusic-SVC achieves consistent improvements over strong open-source baselines in timbre similarity, intelligibility, and perceptual naturalness, especially under accompanied and harmony-contaminated conditions, demonstrating its effectiveness for real-world SVC deployment.
comment : 17 pages, 5 figures
☆ ASTRIDE: A Security Threat Modeling Platform for Agent ic-AI Applications
Eranga Bandara, Amin Hass, Ross Gore, Sachin Shetty, Ravi Mukkamala, Safdar H. Bouk, Xueping Liang, Ng Wee Keong, Kasun De Zoysa, Aruna Withanage, Nilaan Loganathan
AI agent-based systems are becoming increasingly integral to modern software architectures, enabling autonomous decision-making, dynamic task execution, and multimodal interactions through large language models (LLMs). However, these systems introduce novel and evolving security challenges, including prompt injection attacks, context poisoning, model manipulation, and opaque agent-to-agent communication, that are not effectively captured by traditional threat modeling frameworks. In this paper, we introduce ASTRIDE, an automated threat modeling platform purpose-built for AI agent-based systems. ASTRIDE extends the classical STRIDE framework by introducing a new threat category, A for AI Agent-Specific Attacks, which encompasses emerging vulnerabilities such as prompt injection, unsafe tool invocation, and reasoning subversion, unique to agent-based applications. To automate threat modeling, ASTRIDE combines a consortium of fine-tuned vision-language models (VLMs) with the OpenAI-gpt-oss reasoning LLM to perform end-to-end analysis directly from visual agent architecture diagrams, such as data flow diagrams(DFDs). LLM agents orchestrate the end-to-end threat modeling automation process by coordinating interactions between the VLM consortium and the reasoning LLM. Our evaluations demonstrate that ASTRIDE provides accurate, scalable, and explainable threat modeling for next-generation intelligent systems. To the best of our knowledge, ASTRIDE is the first framework to both extend STRIDE with AI-specific threats and integrate fine-tuned VLMs with a reasoning LLM to fully automate diagram-driven threat modeling in AI agent-based applications.
☆ YingMusic-Singer: Zero-shot Singing Voice Synthesis and Editing with Annotation-free Melody Guidance
Junjie Zheng, Chunbo Hao, Guobin Ma, Xiaoyu Zhang, Gongyu Chen, Chaofan Ding, Zihao Chen, Lei Xie
Singing Voice Synthesis (SVS) remains constrained in practical deployment due to its strong dependence on accurate phoneme-level alignment and manually annotated melody contours, requirements that are resource-intensive and hinder scalability. To overcome these limitations, we propose a melody-driven SVS framework capable of synthesizing arbitrary lyrics following any reference melody, without relying on phoneme-level alignment. Our method builds on a Diffusion Transformer (DiT) architecture, enhanced with a dedicated melody extraction module that derives melody representations directly from reference audio. To ensure robust melody encoding, we employ a teacher model to guide the optimization of the melody extractor, alongside an implicit alignment mechanism that enforces similarity distribution constraints for improved melodic stability and coherence. Additionally, we refine duration modeling using weakly annotated song data and introduce a Flow-GRPO reinforcement learning strategy with a multi-objective reward function to jointly enhance pronunciation clarity and melodic fidelity. Experiments show that our model achieves superior performance over existing approaches in both objective measures and subjective listening tests, especially in zero-shot and lyric adaptation settings, while maintaining high audio quality without manual annotation. This work offers a practical and scalable solution for advancing data-efficient singing voice synthesis. To support reproducibility, we release our inference code and model checkpoints.
comment : 13 pages, 3 figures
☆ Using Machine Learning to Take Stay-or-Go Decisions in Data-driven Drone Missions
Giorgos Polychronis, Foivos Pournaropoulos, Christos D. Antonopoulos, Spyros Lalis
Drones are becoming indispensable in many application domains. In data-driven missions, besides sensing, the drone must process the collected data at runtime to decide whether additional action must be taken on the spot, before moving to the next point of interest. If processing does not reveal an event or situation that requires such an action, the drone has waited in vain instead of moving to the next point. If, however, the drone starts moving to the next point and it turns out that a follow-up action is needed at the previous point, it must spend time to fly-back. To take this decision, we propose different machine-learning methods based on branch prediction and reinforcement learning. We evaluate these methods for a wide range of scenarios where the probability of event occurrence changes with time. Our results show that the proposed methods consistently outperform the regression-based method proposed in the literature and can significantly improve the worst-case mission time by up to 4.1x. Also, the achieved median mission time is very close, merely up to 2.7% higher, to that of a method with perfect knowledge of the current underlying event probability at each point of interest.
comment : 19 pages, 3 figures, to appear in the proceedings of MobiQuitous 2025
☆ Embodied Co-Design for Rapidly Evolving Agent s: Taxonomy, Frontiers, and Challenges
Yuxing Wang, Zhiyu Chen, Tiantian Zhang, Qiyue Yin, Yongzhe Chang, Zhiheng Li, Liang Wang, Xueqian Wang
Brain-body co-evolution enables animals to develop complex behaviors in their environments. Inspired by this biological synergy, embodied co-design (ECD) has emerged as a transformative paradigm for creating intelligent agents-from virtual creatures to physical robots-by jointly optimizing their morphologies and controllers rather than treating control in isolation. This integrated approach facilitates richer environmental interactions and robust task performance. In this survey, we provide a systematic overview of recent advances in ECD. We first formalize the concept of ECD and position it within related fields. We then introduce a hierarchical taxonomy: a lower layer that breaks down agent design into three fundamental components-controlling brain, body morphology, and task environment-and an upper layer that integrates these components into four major ECD frameworks: bi-level, single-level, generative, and open-ended. This taxonomy allows us to synthesize insights from more than one hundred recent studies. We further review notable benchmarks, datasets, and applications in both simulated and real-world scenarios. Finally, we identify significant challenges and offer insights into promising future research directions. A project associated with this survey has been created at https://github.com/Yuxing-Wang-THU/SurveyBrainBody.
☆ Human Cognitive Biases in Explanation-Based Interaction: The Case of Within and Between Session Order Effect AAAI 2026
Dario Pesenti, Alessandro Bogani, Katya Tentori, Stefano Teso
Explanatory Interactive Learning (XIL) is a powerful interactive learning framework designed to enable users to customize and correct AI models by interacting with their explanations. In a nutshell, XIL algorithms select a number of items on which an AI model made a decision (e.g. images and their tags) and present them to users, together with corresponding explanations (e.g. image regions that drive the model's decision). Then, users supply corrective feedback for the explanations, which the algorithm uses to improve the model. Despite showing promise in debugging tasks, recent studies have raised concerns that explanatory interaction may trigger order effects, a well-known cognitive bias in which the sequence of presented items influences users' trust and, critically, the quality of their feedback. We argue that these studies are not entirely conclusive, as the experimental designs and tasks employed differ substantially from common XIL use cases, complicating interpretation. To clarify the interplay between order effects and explanatory interaction, we ran two larger-scale user studies (n = 713 total) designed to mimic common XIL tasks. Specifically, we assessed order effects both within and between debugging sessions by manipulating the order in which correct and wrong explanations are presented to participants. Order effects had a limited, through significant impact on users' agreement with the model (i.e., a behavioral measure of their trust), and only when examined withing debugging sessions, not between them. The quality of users' feedback was generally satisfactory, with order effects exerting only a small and inconsistent influence in both experiments. Overall, our findings suggest that order effects do not pose a significant issue for the successful employment of XIL approaches. More broadly, our work contributes to the ongoing efforts for understanding human factors in AI.
comment : 18 pages, 10 figures, published at AAAI 2026
☆ UnwrapDiff: Conditional Diffusion for Robust InSAR Phase Unwrapping
Yijia Song, Juliet Biggs, Alin Achim, Robert Popescu, Simon Orrego, Nantheera Anantrasirichai
Phase unwrapping is a fundamental problem in InSAR data processing, supporting geophysical applications such as deformation monitoring and hazard assessment. Its reliability is limited by noise and decorrelation in radar acquisitions, which makes accurate reconstruction of the deformation signal challenging. We propose a denoising diffusion probabilistic model (DDPM)-based framework for InSAR phase unwrapping, UnwrapDiff, in which the output of the traditional minimum cost flow algorithm (SNAPHU) is incorporated as conditional guidance. To evaluate robustness, we construct a synthetic dataset that incorporates atmospheric effects and diverse noise patterns, representative of realistic InSAR observations. Experiments show that the proposed model leverages the conditional prior while reducing the effect of diverse noise patterns, achieving on average a 10.11\% reduction in NRMSE compared to SNAPHU. It also achieves better reconstruction quality in difficult cases such as dyke intrusions.
☆ SignRoundV2: Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs
Wenhua Cheng, Weiwei Zhang, Heng Guo, Haihao Shen
Extreme low-bit quantization is critical for efficiently deploying Large Language Models (LLMs), yet it often leads to severe performance degradation at 2-bits and even 4-bits (e.g., MXFP4). We present SignRoundV2, a post-training quantization framework that is highly effective even without mixed-precision. SignRoundV2 introduces (1) a fast sensitivity metric that combines gradient information with quantization-induced deviations to guide layer-wise bit allocation, and (2) a lightweight pre-tuning search for quantization scales to improve extremely low-bit quantization. These components allow SignRoundV2 to close the gap with full-precision models. Extensive experiments indicate that our method sustains competitive accuracy for LLMs, achieving production-grade performance with about 1 percent variance at 4-5 bits and strong results even at 2 bits. The implementation is available at https://github.com/intel/auto-round.
☆ Neural Policy Composition from Free Energy Minimization
Francesca Rossi, Veronica Centorrino, Francesco Bullo, Giovanni Russo
The ability to compose acquired skills to plan and execute behaviors is a hallmark of natural intelligence. Yet, despite remarkable cross-disciplinary efforts, a principled account of how task structure shapes gating and how such computations could be delivered in neural circuits, remains elusive. Here we introduce GateMod, an interpretable theoretically grounded computational model linking the emergence of gating to the underlying decision-making task, and to a neural circuit architecture. We first develop GateFrame, a normative framework casting policy gating into the minimization of the free energy. This framework, relating gating rules to task, applies broadly across neuroscience, cognitive and computational sciences. We then derive GateFlow, a continuous-time energy based dynamics that provably converges to GateFrame optimal solution. Convergence, exponential and global, follows from a contractivity property that also yields robustness and other desirable properties. Finally, we derive a neural circuit from GateFlow, GateNet. This is a soft-competitive recurrent circuit whose components perform local and contextual computations consistent with known dendritic and neural processing motifs. We evaluate GateMod across two different settings: collective behaviors in multi-agent systems and human decision-making in multi-armed bandits. In all settings, GateMod provides interpretable mechanistic explanations of gating and quantitatively matches or outperforms established models. GateMod offers a unifying framework for neural policy gating, linking task objectives, dynamical computation, and circuit-level mechanisms. It provides a framework to understand gating in natural agents beyond current explanations and to equip machines with this ability.
☆ OsmT: Bridging OpenStreetMap Queries and Natural Language with Open-source Tag-aware Language Models ICDE
Zhuoyue Wan, Wentao Hu, Chen Jason Zhang, Yuanfeng Song, Shuaimin Li, Ruiqiang Xiao, Xiao-Yong Wei, Raymond Chi-Wing Wong
Bridging natural language and structured query languages is a long-standing challenge in the database community. While recent advances in language models have shown promise in this direction, existing solutions often rely on large-scale closed-source models that suffer from high inference costs, limited transparency, and lack of adaptability for lightweight deployment. In this paper, we present OsmT, an open-source tag-aware language model specifically designed to bridge natural language and Overpass Query Language (OverpassQL), a structured query language for accessing large-scale OpenStreetMap (OSM) data. To enhance the accuracy and structural validity of generated queries, we introduce a Tag Retrieval Augmentation (TRA) mechanism that incorporates contextually relevant tag knowledge into the generation process. This mechanism is designed to capture the hierarchical and relational dependencies present in the OSM database, addressing the topological complexity inherent in geospatial query formulation. In addition, we define a reverse task, OverpassQL-to-Text, which translates structured queries into natural language explanations to support query interpretation and improve user accessibility. We evaluate OsmT on a public benchmark against strong baselines and observe consistent improvements in both query generation and interpretation. Despite using significantly fewer parameters, our model achieves competitive accuracy, demonstrating the effectiveness of open-source pre-trained language models in bridging natural language and structured query languages within schema-rich geospatial environments.
comment : 42nd IEEE International Conference on Data Engineering (ICDE)
☆ E3AD: An Emotion-Aware Vision -Language-Action Model for Human-Centric End-to-End Autonomous Driving
Yihong Tang, Haicheng Liao, Tong Nie, Junlin He, Ao Qu, Kehua Chen, Wei Ma, Zhenning Li, Lijun Sun, Chengzhong Xu
End-to-end autonomous driving (AD) systems increasingly adopt vision-language-action (VLA) models, yet they typically ignore the passenger's emotional state, which is central to comfort and AD acceptance. We introduce Open-Domain End-to-End (OD-E2E) autonomous driving, where an autonomous vehicle (AV) must interpret free-form natural-language commands, infer the emotion, and plan a physically feasible trajectory. We propose E3AD, an emotion-aware VLA framework that augments semantic understanding with two cognitively inspired components: a continuous Valenc-Arousal-Dominance (VAD) emotion model that captures tone and urgency from language, and a dual-pathway spatial reasoning module that fuses egocentric and allocentric views for human-like spatial cognition. A consistency-oriented training scheme, combining modality pretraining with preference-based alignment, further enforces coherence between emotional intent and driving actions. Across real-world datasets, E3AD improves visual grounding and waypoint planning and achieves state-of-the-art (SOTA) VAD correlation for emotion estimation. These results show that injecting emotion into VLA-style driving yields more human-aligned grounding, planning, and human-centric feedback.
☆ Measuring the Unspoken: A Disentanglement Model and Benchmark for Psychological Analysis in the Wild
Yigui Feng, Qinglin Wang, Haotian Mo, Yang Liu, Ke Liu, Gencheng Liu, Xinhai Chen, Siqi Shen, Songzhu Mei, Jie Liu
Generative psychological analysis of in-the-wild conversations faces two fundamental challenges: (1) existing Vision-Language Models (VLMs) fail to resolve Articulatory-Affective Ambiguity, where visual patterns of speech mimic emotional expressions; and (2) progress is stifled by a lack of verifiable evaluation metrics capable of assessing visual grounding and reasoning depth. We propose a complete ecosystem to address these twin challenges. First, we introduce Multilevel Insight Network for Disentanglement(MIND), a novel hierarchical visual encoder that introduces a Status Judgment module to algorithmically suppress ambiguous lip features based on their temporal feature variance, achieving explicit visual disentanglement. Second, we construct ConvoInsight-DB, a new large-scale dataset with expert annotations for micro-expressions and deep psychological inference. Third, Third, we designed the Mental Reasoning Insight Rating Metric (PRISM), an automated dimensional framework that uses expert-guided LLM to measure the multidimensional performance of large mental vision models. On our PRISM benchmark, MIND significantly outperforms all baselines, achieving a +86.95% gain in micro-expression detection over prior SOTA. Ablation studies confirm that our Status Judgment disentanglement module is the most critical component for this performance leap. Our code has been opened.
☆ Sequential Enumeration in Large Language Models
Kuinan Hou, Marco Zorzi, Alberto Testolin
Reliably counting and generating sequences of items remain a significant challenge for neural networks, including Large Language Models (LLMs). Indeed, although this capability is readily handled by rule-based symbolic systems based on serial computation, learning to systematically deploy counting procedures is difficult for neural models, which should acquire these skills through learning. Previous research has demonstrated that recurrent architectures can only approximately track and enumerate sequences of events, and it remains unclear whether modern deep learning systems, including LLMs, can deploy systematic counting procedures over sequences of discrete symbols. This paper aims to fill this gap by investigating the sequential enumeration abilities of five state-of-the-art LLMs, including proprietary, open-source, and reasoning models. We probe LLMs in sequential naming and production tasks involving lists of letters and words, adopting a variety of prompting instructions to explore the role of chain-of-thought in the spontaneous emerging of counting strategies. We also evaluate open-source models with the same architecture but increasing size to see whether the mastering of counting principles follows scaling laws, and we analyze the embedding dynamics during sequential enumeration to investigate the emergent encoding of numerosity. We find that some LLMs are indeed capable of deploying counting procedures when explicitly prompted to do so, but none of them spontaneously engage in counting when simply asked to enumerate the number of items in a sequence. Our results suggest that, despite their impressive emergent abilities, LLMs cannot yet robustly and systematically deploy counting procedures, highlighting a persistent gap between neural and symbolic approaches to compositional generalization.
☆ Towards an AI Fluid Scientist: LLM-Powered Scientific Discovery in Experimental Fluid Mechanics
Haodong Feng, Lugang Ye, Dixia Fan
The integration of artificial intelligence into experimental fluid mechanics promises to accelerate discovery, yet most AI applications remain narrowly focused on numerical studies. This work proposes an AI Fluid Scientist framework that autonomously executes the complete experimental workflow: hypothesis generation, experimental design, robotic execution, data analysis, and manuscript preparation. We validate this through investigation of vortex-induced vibration (VIV) and wake-induced vibration (WIV) in tandem cylinders. Our work has four key contributions: (1) A computer-controlled circulating water tunnel (CWT) with programmatic control of flow velocity, cylinder position, and forcing parameters (vibration frequency and amplitude) with data acquisition (displacement, force, and torque). (2) Automated experiments reproduce literature benchmarks (Khalak and Williamson [1999] and Assi et al. [2013, 2010]) with frequency lock-in within 4% and matching critical spacing trends. (3) The framework with Human-in-the-Loop (HIL) discovers more WIV amplitude response phenomena, and uses a neural network to fit physical laws from data, which is 31% higher than that of polynomial fitting. (4) The framework with multi-agent with virtual-real interaction system executes hundreds of experiments end-to-end, which automatically completes the entire process of scientific research from hypothesis generation, experimental design, experimental execution, data analysis, and manuscript preparation. It greatly liberates human researchers and improves study efficiency, providing new paradigm for the development and research of experimental fluid mechanics.
☆ Playing the Player: A Heuristic Framework for Adaptive Poker AI
Andrew Paterson, Carl Sanders
For years, the discourse around poker AI has been dominated by the concept of solvers and the pursuit of unexploitable, machine-perfect play. This paper challenges that orthodoxy. It presents Patrick, an AI built on the contrary philosophy: that the path to victory lies not in being unexploitable, but in being maximally exploitative. Patrick's architecture is a purpose-built engine for understanding and attacking the flawed, psychological, and often irrational nature of human opponents. Through detailed analysis of its design, its novel prediction-anchored learning method, and its profitable performance in a 64,267-hand trial, this paper makes the case that the solved myth is a distraction from the real, far more interesting challenge: creating AI that can master the art of human imperfection.
comment : 49 pages, 39 figures. White Paper by Spiderdime Systems
☆ Large Speech Model Enabled Semantic Communication
Yun Tian, Zhijin Qin, Guocheng Lv, Ye Jin, Kaibin Huang, Zhu Han
Existing speech semantic communication systems mainly based on Joint Source-Channel Coding (JSCC) architectures have demonstrated impressive performance, but their effectiveness remains limited by model structures specifically designed for particular tasks and datasets. Recent advances indicate that generative large models pre-trained on massive datasets, can achieve outstanding performance arexhibit exceptional performance across diverse downstream tasks with minimal fine-tuning. To exploit the rich semantic knowledge embedded in large models and enable adaptive transmission over lossy channels, we propose a Large Speech Model enabled Semantic Communication (LargeSC) system. Simultaneously achieving adaptive compression and robust transmission over lossy channels remains challenging, requiring trade-offs among compression efficiency, speech quality, and latency. In this work, we employ the Mimi as a speech codec, converting speech into discrete tokens compatible with existing network architectures. We propose an adaptive controller module that enables adaptive transmission and in-band Unequal Error Protection (UEP), dynamically adjusting to both speech content and packet loss probability under bandwidth constraints. Additionally, we employ Low-Rank Adaptation (LoRA) to finetune the Moshi foundation model for generative recovery of lost speech tokens. Simulation results show that the proposed system supports bandwidths ranging from 550 bps to 2.06 kbps, outperforms conventional baselines in speech quality under high packet loss rates and achieves an end-to-end latency of approximately 460 ms, thereby demonstrating its potential for real-time deployment.
comment : 15 pages, 9 figures
☆ TimesNet-Gen: Deep Learning-based Site Specific Strong Motion Generation
Baris Yilmaz, Bevan Deniz Cilgin, Erdem Akagündüz, Salih Tileylioglu
Effective earthquake risk reduction relies on accurate site-specific evaluations. This requires models that can represent the influence of local site conditions on ground motion characteristics. In this context, data driven approaches that learn site controlled signatures from recorded ground motions offer a promising direction. We address strong ground motion generation from time-domain accelerometer records and introduce the TimesNet-Gen, a time-domain conditional generator. The approach uses a station specific latent bottleneck. We evaluate generation by comparing HVSR curves and fundamental site-frequency $f_0$ distributions between real and generated records per station, and summarize station specificity with a score based on the $f_0$ distribution confusion matrices. TimesNet-Gen achieves strong station-wise alignment and compares favorably with a spectrogram-based conditional VAE baseline for site-specific strong motion synthesis. Our codes are available via https://github.com/brsylmz23/TimesNet-Gen.
☆ Towards Ethical Multi- Agent Systems of Large Language Models: A Mechanistic Interpretability Perspective AAAI'26
Jae Hee Lee, Anne Lauscher, Stefano V. Albrecht
Large language models (LLMs) have been widely deployed in various applications, often functioning as autonomous agents that interact with each other in multi-agent systems. While these systems have shown promise in enhancing capabilities and enabling complex tasks, they also pose significant ethical challenges. This position paper outlines a research agenda aimed at ensuring the ethical behavior of multi-agent systems of LLMs (MALMs) from the perspective of mechanistic interpretability. We identify three key research challenges: (i) developing comprehensive evaluation frameworks to assess ethical behavior at individual, interactional, and systemic levels; (ii) elucidating the internal mechanisms that give rise to emergent behaviors through mechanistic interpretability; and (iii) implementing targeted parameter-efficient alignment techniques to steer MALMs towards ethical behaviors without compromising their performance.
comment : Accepted to LaMAS 2026@AAAI'26 (https://sites.google.com/view/lamas2026)
☆ Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmap
Jialong Li, Mingyue Zhang, Nianyu Li, Danny Weyns, Zhi Jin, Kenji Tei
Self-adaptive systems (SASs) are designed to handle changes and uncertainties through a feedback loop with four core functionalities: monitoring, analyzing, planning, and execution. Recently, generative artificial intelligence (GenAI), especially the area of large language models, has shown impressive performance in data comprehension and logical reasoning. These capabilities are highly aligned with the functionalities required in SASs, suggesting a strong potential to employ GenAI to enhance SASs. However, the specific benefits and challenges of employing GenAI in SASs remain unclear. Yet, providing a comprehensive understanding of these benefits and challenges is complex due to several reasons: limited publications in the SAS field, the technological and application diversity within SASs, and the rapid evolution of GenAI technologies. To that end, this paper aims to provide researchers and practitioners a comprehensive snapshot that outlines the potential benefits and challenges of employing GenAI's within SAS. Specifically, we gather, filter, and analyze literature from four distinct research fields and organize them into two main categories to potential benefits: (i) enhancements to the autonomy of SASs centered around the specific functions of the MAPE-K feedback loop, and (ii) improvements in the interaction between humans and SASs within human-on-the-loop settings. From our study, we outline a research roadmap that highlights the challenges of integrating GenAI into SASs. The roadmap starts with outlining key research challenges that need to be tackled to exploit the potential for applying GenAI in the field of SAS. The roadmap concludes with a practical reflection, elaborating on current shortcomings of GenAI and proposing possible mitigation strategies.
comment : Accepted by ACM Transactions on Autonomous and Adaptive Systems
☆ Topology Matters: Measuring Memory Leakage in Multi- Agent LLMs ACL
Jinbo Liu, Defu Cao, Yifei Wei, Tianyao Su, Yuan Liang, Yushun Dong, Yue Zhao, Xiyang Hu
Graph topology is a fundamental determinant of memory leakage in multi-agent LLM systems, yet its effects remain poorly quantified. We introduce MAMA (Multi-Agent Memory Attack), a framework that measures how network structure shapes leakage. MAMA operates on synthetic documents containing labeled Personally Identifiable Information (PII) entities, from which we generate sanitized task instructions. We execute a two-phase protocol: Engram (seeding private information into a target agent's memory) and Resonance (multi-round interaction where an attacker attempts extraction). Over up to 10 interaction rounds, we quantify leakage as the fraction of ground-truth PII recovered from attacking agent outputs via exact matching. We systematically evaluate six common network topologies (fully connected, ring, chain, binary tree, star, and star-ring), varying agent counts $n\in\{4,5,6\}$, attacker-target placements, and base models. Our findings reveal consistent patterns: fully connected graphs exhibit maximum leakage while chains provide strongest protection; shorter attacker-target graph distance and higher target centrality significantly increase vulnerability; leakage rises sharply in early rounds before plateauing; model choice shifts absolute leakage rates but preserves topology rankings; temporal/locational PII attributes leak more readily than identity credentials or regulated identifiers. These results provide the first systematic mapping from architectural choices to measurable privacy risk, yielding actionable guidance: prefer sparse or hierarchical connectivity, maximize attacker-target separation, limit node degree and network radius, avoid shortcuts bypassing hubs, and implement topology-aware access controls.
comment : Under review at ACL Rolling Review
☆ Semi Centralized Training Decentralized Execution Architecture for Multi Agent Deep Reinforcement Learning in Traffic Signal Control
Pouria Yazdani, Arash Rezaali, Monireh Abdoos
Multi-agent reinforcement learning (MARL) has emerged as a promising paradigm for adaptive traffic signal control (ATSC) of multiple intersections. Existing approaches typically follow either a fully centralized or a fully decentralized design. Fully centralized approaches suffer from the curse of dimensionality, and reliance on a single learning server, whereas purely decentralized approaches operate under severe partial observability and lack explicit coordination resulting in suboptimal performance. These limitations motivate region-based MARL, where the network is partitioned into smaller, tightly coupled intersections that form regions, and training is organized around these regions. This paper introduces a Semi-Centralized Training, Decentralized Execution (SEMI-CTDE) architecture for multi intersection ATSC. Within each region, SEMI-CTDE performs centralized training with regional parameter sharing and employs composite state and reward formulations that jointly encode local and regional information. The architecture is highly transferable across different policy backbones and state-reward instantiations. Building on this architecture, we implement two models with distinct design objectives. A multi-perspective experimental analysis of the two implemented SEMI-CTDE-based models covering ablations of the architecture's core elements including rule based and fully decentralized baselines shows that they achieve consistently superior performance and remain effective across a wide range of traffic densities and distributions.
comment : Co-first authors: Pouria Yazdani and Arash Rezaali
☆ SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decoding
Chang-Hsun Wu, Kai-Po Chang, Yu-Yang Sheng, Hung-Kai Chung, Kuei-Chun Wang, Yu-Chiang Frank Wang
Video Large Language Models (VideoLLMs) have shown remarkable progress in video understanding. However, these models still struggle to effectively perceive and exploit rich temporal information in videos when responding to user queries. Therefore, they often generate descriptions of events that are temporal inconsistent or causally implausible, causing severe hallucination issues. While most prior studies have focused on spatial hallucinations (e.g. object mismatches), temporal reasoning in video understanding remains relatively underexplored. To address this issue, we propose Self-Diagnostic Contrastive Decoding (SEASON), a training-free method that adaptively enhances temporal and spatial faithfulness for each output token. It achieves this by dynamically diagnosing each token's hallucination tendency and applying adaptive contrastive decoding against its corresponding temporal and spatial negatives. Extensive experiments demonstrate that SEASON outperforms all existing training-free hallucination mitigation approaches on three hallucination examination benchmarks, while further improves VideoLLMs across four general video understanding benchmarks. The code will be released upon acceptance.
☆ When GenAI Meets Fake News: Understanding Image Cascade Dynamics on Reddit
Saumya Chauhan, Mila Hong, Maria Vazhaeparambil
AI-generated content and misinformation are increasingly prevalent on social networks. While prior research primarily examined textual misinformation, fewer studies have focused on visual content's role in virality. In this work, we present the first large-scale analysis of how misinformation and AI-generated images propagate through repost cascades across five ideologically diverse Reddit communities. By integrating textual sentiment, visual attributes, and diffusion metrics (e.g., time-to-first repost, community reach), our framework accurately predicts both immediate post-level virality (AUC=0.83) and long-term cascade-level spread (AUC=0.998). These findings offer essential insights for moderating synthetic and misleading visual content online.
comment : Accepted at 2025 MIT Undergraduate Research Technology Conference (URTC'25)
☆ Turbo-Muon: Accelerating Orthogonality-Based Optimization with Pre-Conditioning
Thibaut Boissin, Thomas Massena, Franck Mamalet, Mathieu Serrurier
Orthogonality-based optimizers, such as Muon, have recently shown strong performance across large-scale training and community-driven efficiency challenges. However, these methods rely on a costly gradient orthogonalization step. Even efficient iterative approximations such as Newton-Schulz remain expensive, typically requiring dozens of matrix multiplications to converge. We introduce a preconditioning procedure that accelerates Newton-Schulz convergence and reduces its computational cost. We evaluate its impact and show that the overhead of our preconditioning can be made negligible. Furthermore, the faster convergence it enables allows us to remove one iteration out of the usual five without degrading approximation quality. Our publicly available implementation achieves up to a 2.8x speedup in the Newton-Schulz approximation. We also show that this has a direct impact on end-to-end training runtime with 5-10% improvement in realistic training scenarios across two efficiency-focused tasks. On challenging language or vision tasks, we validate that our method maintains equal or superior model performance while improving runtime. Crucially, these improvements require no hyperparameter tuning and can be adopted as a simple drop-in replacement. Our code is publicly available on github.
☆ BioMed GPT -Mol: Multi-task Learning for Molecular Understanding and Generation
Chenyang Zuo, Siqi Fan, Zaiqing Nie
Molecules play a crucial role in biomedical research and discovery, particularly in the field of small molecule drug development. Given the rapid advancements in large language models, especially the recent emergence of reasoning models, it is natural to explore how a general-purpose language model can be efficiently adapted for molecular science applications. In this work, we introduce BioMedGPT-Mol, a molecular language model designed to support molecular understanding and generation tasks. By curating and unifying existing public instruction datasets, we have assembled a large-scale, comprehensive, and high-quality training dataset. The model is then fine-tuned through a meticulously designed multi-task learning framework. On a consolidated benchmark derived from LlaSMol, TOMG-Bench, and MuMOInstruct, BioMedGPT-Mol achieves remarkable performance. Our experimental results demonstrate that a general-purpose reasoning model can be effectively and efficiently post-trained into a professional molecular language model through a well-structured multi-task curriculum. Leveraging the power of it, we further explore retrosynthetic planning task, and the performance on RetroBench demonstrates its competitive capability of acting as an end-to-end retrosynthetic planner. We anticipate that our approach can be extended to other biomedical scientific domains.
☆ Neural Decoding of Overt Speech from ECoG Using Vision Transformer s and Contrastive Representation Learning
Mohamed Baha Ben Ticha, Xingchen Ran, Guillaume Saldanha, Gaël Le Godais, Philémon Roussel, Marc Aubert, Amina Fontanell, Thomas Costecalde, Lucas Struber, Serpil Karakas, Shaomin Zhang, Philippe Kahane, Guillaume Charvet, Stéphan Chabardès, Blaise Yvert
Speech Brain Computer Interfaces (BCIs) offer promising solutions to people with severe paralysis unable to communicate. A number of recent studies have demonstrated convincing reconstruction of intelligible speech from surface electrocorticographic (ECoG) or intracortical recordings by predicting a series of phonemes or words and using downstream language models to obtain meaningful sentences. A current challenge is to reconstruct speech in a streaming mode by directly regressing cortical signals into acoustic speech. While this has been achieved recently using intracortical data, further work is needed to obtain comparable results with surface ECoG recordings. In particular, optimizing neural decoders becomes critical in this case. Here we present an offline speech decoding pipeline based on an encoder-decoder deep neural architecture, integrating Vision Transformers and contrastive learning to enhance the direct regression of speech from ECoG signals. The approach is evaluated on two datasets, one obtained with clinical subdural electrodes in an epileptic patient, and another obtained with the fully implantable WIMAGINE epidural system in a participant of a motor BCI trial. To our knowledge this presents a first attempt to decode speech from a fully implantable and wireless epidural recording system offering perspectives for long-term use.
☆ The Ethics of Generative AI
Michael Klenk
This chapter discusses the ethics of generative AI. It provides a technical primer to show how generative AI affords experiencing technology as if it were human, and this affordance provides a fruitful focus for the philosophical ethics of generative AI. It then shows how generative AI can both aggravate and alleviate familiar ethical concerns in AI ethics, including responsibility, privacy, bias and fairness, and forms of alienation and exploitation. Finally, the chapter examines ethical questions that arise specifically from generative AI's mimetic generativity, such as debates about authorship and credit, the emergence of as-if social relationships with machines, and new forms of influence, persuasion, and manipulation.
comment : Draft version to appear as a chapter in the Encyclopedia of Applied Ethics, 3rd Edition, edited by Ruth Chadwick
☆ When Robots Should Say "I Don't Know": Benchmark ing Abstention in Embodied Question Answering
Tao Wu, Chuhao Zhou, Guangyu Zhao, Haozhi Cao, Yewen Pu, Jianfei Yang
Embodied Question Answering (EQA) requires an agent to interpret language, perceive its environment, and navigate within 3D scenes to produce responses. Existing EQA benchmarks assume that every question must be answered, but embodied agents should know when they do not have sufficient information to answer. In this work, we focus on a minimal requirement for EQA agents, abstention: knowing when to withhold an answer. From an initial study of 500 human queries, we find that 32.4% contain missing or underspecified context. Drawing on this initial study and cognitive theories of human communication errors, we derive five representative categories requiring abstention: actionability limitation, referential underspecification, preference dependence, information unavailability, and false presupposition. We augment OpenEQA by having annotators transform well-posed questions into ambiguous variants outlined by these categories. The resulting dataset, AbstainEQA, comprises 1,636 annotated abstention cases paired with 1,636 original OpenEQA instances for balanced evaluation. Evaluating on AbstainEQA, we find that even the best frontier model only attains 42.79% abstention recall, while humans achieve 91.17%. We also find that scaling, prompting, and reasoning only yield marginal gains, and that fine-tuned models overfit to textual cues. Together, these results position abstention as a fundamental prerequisite for reliable interaction in embodied settings and as a necessary basis for effective clarification.
☆ A Light-Weight Large Language Model File Format for Highly-Secure Model Distribution
Huifeng Zhu, Shijie Li, Qinfeng Li, Yier Jin
To enhance the performance of large language models (LLMs) in various domain-specific applications, sensitive data such as healthcare, law, and finance are being used to privately customize or fine-tune these models. Such privately adapted LLMs are regarded as either personal privacy assets or corporate intellectual property. Therefore, protecting model weights and maintaining strict confidentiality during deployment and distribution have become critically important. However, existing model formats and deployment frameworks provide little to no built-in support for confidentiality, access control, or secure integration with trusted hardware. Current methods for securing model deployment either rely on computationally expensive cryptographic techniques or tightly controlled private infrastructure. Although these approaches can be effective in specific scenarios, they are difficult and costly for widespread deployment. In this paper, we introduce CryptoTensors, a secure and format-compatible file structure for confidential LLM distribution. Built as an extension to the widely adopted Safetensors format, CryptoTensors incorporates tensor-level encryption and embedded access control policies, while preserving critical features such as lazy loading and partial deserialization. It enables transparent decryption and automated key management, supporting flexible licensing and secure model execution with minimal overhead. We implement a proof-of-concept library, benchmark its performance across serialization and runtime scenarios, and validate its compatibility with existing inference frameworks, including Hugging Face Transformers and vLLM. Our results highlight CryptoTensors as a light-weight, efficient, and developer-friendly solution for safeguarding LLM weights in real-world and widespread deployments.
☆ Diffusion Fine-Tuning via Reparameterized Policy Gradient of the Soft Q-Function
Hyeongyu Kang, Jaewoo Lee, Woocheol Shin, Kiyoung Om, Jinkyoo Park
Diffusion models excel at generating high-likelihood samples but often require alignment with downstream objectives. Existing fine-tuning methods for diffusion models significantly suffer from reward over-optimization, resulting in high-reward but unnatural samples and degraded diversity. To mitigate over-optimization, we propose \textbf{Soft Q-based Diffusion Finetuning (SQDF)}, a novel KL-regularized RL method for diffusion alignment that applies a reparameterized policy gradient of a training-free, differentiable estimation of the soft Q-function. SQDF is further enhanced with three innovations: a discount factor for proper credit assignment in the denoising process, the integration of consistency models to refine Q-function estimates, and the use of an off-policy replay buffer to improve mode coverage and manage the reward-diversity trade-off. Our experiments demonstrate that SQDF achieves superior target rewards while preserving diversity in text-to-image alignment. Furthermore, in online black-box optimization, SQDF attains high sample efficiency while maintaining naturalness and diversity.
comment : 36 pages, 21 figures, 4 tables
☆ RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS ICASSP 2026
Cong Wang, Changfeng Gao, Yang Xiang, Zhihao Du, Keyu An, Han Zhao, Qian Chen, Xiangang Li, Yingming Gao, Ya Li
Differentiable reinforcement learning (RL) frameworks like DiffRO offer a powerful approach for controllable text-to-speech (TTS), but are vulnerable to reward hacking, particularly for nuanced tasks like emotion control. The policy model can exploit a vanilla Reward Model (RM) by generating acoustic artifacts to achieve spurious rewards, but at the cost of degrading perceptual quality. To address this, we propose Robust Reward Policy Optimization (RRPO), a novel framework that employs a hybrid regularization scheme. This scheme develops a robust RM whose reward signal is more reliably aligned with human perception, compelling the policy to abandon detrimental shortcuts and instead learn the complex features of genuine emotions. Our ablation study confirms the enhanced robustness of our RM, as evidenced by its strong cross-lingual generalization. The subjective evaluation demonstrates that this robust RM effectively mitigates reward hacking, leading to significant improvements in both emotional expressiveness and naturalness over all baselines. Demo page: https://lrwinr.github.io/RRPO-CosyVoice.
comment : Submitted to ICASSP 2026. Copyright 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works
☆ Multi-Loss Learning for Speech Emotion Recognition with Energy-Adaptive Mixup and Frame-Level Attention ICASSP 2026
Cong Wang, Yizhong Geng, Yuhua Wen, Qifei Li, Yingming Gao, Ruimin Wang, Chunfeng Wang, Hao Li, Ya Li, Wei Chen
Speech emotion recognition (SER) is an important technology in human-computer interaction. However, achieving high performance is challenging due to emotional complexity and scarce annotated data. To tackle these challenges, we propose a multi-loss learning (MLL) framework integrating an energy-adaptive mixup (EAM) method and a frame-level attention module (FLAM). The EAM method leverages SNR-based augmentation to generate diverse speech samples capturing subtle emotional variations. FLAM enhances frame-level feature extraction for multi-frame emotional cues. Our MLL strategy combines Kullback-Leibler divergence, focal, center, and supervised contrastive loss to optimize learning, address class imbalance, and improve feature separability. We evaluate our method on four widely used SER datasets: IEMOCAP, MSP-IMPROV, RAVDESS, and SAVEE. The results demonstrate our method achieves state-of-the-art performance, suggesting its effectiveness and robustness.
comment : Submitted to ICASSP 2026. Copyright 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works
☆ AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees NeurIPS 2025
Yangning Li, Shaoshen Chen, Yinghui Li, Yankai Chen, Hai-Tao Zheng, Hui Wang, Wenhao Jiang, Philip S. Yu
The quadratic complexity of self-attention constrains Large Language Models (LLMs) in processing long contexts, a capability essential for many advanced applications. Context compression aims to alleviate this computational bottleneck while retaining critical semantic information. However, existing approaches often fall short: explicit methods may compromise local detail, whereas implicit methods can suffer from positional biases, information degradation, or an inability to capture long-range semantic dependencies. We propose AdmTree, a novel framework for adaptive, hierarchical context compression with a central focus on preserving high semantic fidelity while maintaining efficiency. AdmTree dynamically segments input based on information density, utilizing gist tokens to summarize variable-length segments as the leaves of a semantic binary tree. This structure, together with a lightweight aggregation mechanism and a frozen backbone LLM (thereby minimizing new trainable parameters), enables efficient hierarchical abstraction of the context. By preserving fine-grained details alongside global semantic coherence, mitigating positional bias, and dynamically adapting to content, AdmTree robustly retains the semantic information of long contexts.
comment : NeurIPS 2025
☆ Detection of Intoxicated Individuals from Facial Video Sequences via a Recurrent Fusion Model
Bita Baroutian, Atefe Aghaei, Mohsen Ebrahimi Moghaddam
Alcohol consumption is a significant public health concern and a major cause of accidents and fatalities worldwide. This study introduces a novel video-based facial sequence analysis approach dedicated to the detection of alcohol intoxication. The method integrates facial landmark analysis via a Graph Attention Network (GAT) with spatiotemporal visual features extracted using a 3D ResNet. These features are dynamically fused with adaptive prioritization to enhance classification performance. Additionally, we introduce a curated dataset comprising 3,542 video segments derived from 202 individuals to support training and evaluation. Our model is compared against two baselines: a custom 3D-CNN and a VGGFace+LSTM architecture. Experimental results show that our approach achieves 95.82% accuracy, 0.977 precision, and 0.97 recall, outperforming prior methods. The findings demonstrate the model's potential for practical deployment in public safety systems for non-invasive, reliable alcohol intoxication detection.
☆ GTM: Simulating the Wo rl d of Tools for AI Agent s
Zhenzhen Ren, Xinpeng Zhang, Zhenxing Qian, Yan Gao, Yu Shi, Shuxin Zheng, Jiyan He
The integration of external tools is pivotal for empowering Large Language Model (LLM) agents with real-world capabilities. However, training these agents through direct, continuous interaction with diverse tools is often prohibitively expensive, slow, and introduces additional development and maintenance overhead. To address this challenge, we introduce the Generalist Tool Model (GTM), a 1.5-billion-parameter model that learns to act as a universal tool simulator. With only prompt-level configuration, GTM accesses tool functionalities along with input arguments and generates outputs that faithfully mimic real tool execution, providing a fast and cost-effective solution that eliminates development overhead. To build GTM, we propose the Context-Aware Response Generation (CARG) pipeline, which synthesizes comprehensive training data covering over 20,000 tools across 300 domains including physics, medicine, robotics, and finance. Through this pipeline, GTM learns to produce not only syntactically correct outputs but also logically coherent and contextually appropriate responses. Experiments demonstrate that GTM produces high-quality outputs with strong consistency and reliability. Besides when used in real reinforcement learning scenarios for agent training, GTM exhibits significantly faster simulation speed compared to real tools while maintaining comparable output quality, along with remarkable generalization and domain adaptability. Our results establish GTM as a foundational component for developing future AI agents, enabling efficient and scalable training of tool-augmented systems.
☆ PhyVLLM: Physics-Guided Video Language Model with Motion-Appearance Disentanglement
Yu-Wei Zhan, Xin Wang, Hong Chen, Tongtong Feng, Wei Feng, Ren Wang, Guangyao Li, Qing Li, Wenwu Zhu
Video Large Language Models (Video LLMs) have shown impressive performance across a wide range of video-language tasks. However, they often fail in scenarios requiring a deeper understanding of physical dynamics. This limitation primarily arises from their reliance on appearance-based matching. Incorporating physical motion modeling is crucial for deeper video understanding, but presents three key challenges: (1) motion signals are often entangled with appearance variations, making it difficult to extract clean physical cues; (2) effective motion modeling requires not only continuous-time motion representations but also capturing physical dynamics; and (3) collecting accurate annotations for physical attributes is costly and often impractical. To address these issues, we propose PhyVLLM, a physical-guided video-language framework that explicitly incorporates physical motion into Video LLMs. Specifically, PhyVLLM disentangles visual appearance and object motion through a dual-branch encoder. To model physical dynamics over time, we incorporate a Neural Ordinary Differential Equation (Neural ODE) module, which generates differentiable physical dynamic representations. The resulting motion-aware representations are projected into the token space of a pretrained LLM, enabling physics reasoning without compromising the model's original multimodal capabilities. To circumvent the need for explicit physical labels, PhyVLLM employs a self-supervised manner to model the continuous evolution of object motion. Experimental results demonstrate that PhyVLLM significantly outperforms state-of-the-art Video LLMs on both physical reasoning and general video understanding tasks, highlighting the advantages of incorporating explicit physical modeling.
☆ SlideGen: Collaborative Multimodal Agent s for Scientific Slide Generation
Xin Liang, Xiang Zhang, Yiwei Xu, Siqi Sun, Chenyu You
Generating academic slides from scientific papers is a challenging multimodal reasoning task that requires both long context understanding and deliberate visual planning. Existing approaches largely reduce it to text only summarization, overlooking the visual component and design intensive nature of slide creation. In this paper we introduce SlideGen, an agentic, modular, and visual in the loop framework for scientific paper to slide generation. SlideGen orchestrates a group of vision language agents that reason collaboratively over the document structure and semantics, producing editable PPTX slides with logical flow and compelling visual presentation. By integrating coordinated outlining, mapping, arrangement, note synthesis, and iterative refinement, our system consistently delivers slides of expert level quality. Across diverse benchmarks and strong baselines, SlideGen outperforms existing methods in visual quality, content faithfulness, and readability, positioning it as the new state of the art in automated slide generation. Our work establishes a foundation for design aware multimodal slide generation, demonstrating how agentic collaboration can bridge understanding and presentation in complex multimodal reasoning tasks.
☆ Prototype-Based Semantic Consistency Alignment for Domain Adaptive Retrieval AAAI2026
Tianle Hu, Weijun Lv, Na Han, Xiaozhao Fang, Jie Wen, Jiaxing Li, Guoxu Zhou
Domain adaptive retrieval aims to transfer knowledge from a labeled source domain to an unlabeled target domain, enabling effective retrieval while mitigating domain discrepancies. However, existing methods encounter several fundamental limitations: 1) neglecting class-level semantic alignment and excessively pursuing pair-wise sample alignment; 2) lacking either pseudo-label reliability consideration or geometric guidance for assessing label correctness; 3) directly quantizing original features affected by domain shift, undermining the quality of learned hash codes. In view of these limitations, we propose Prototype-Based Semantic Consistency Alignment (PSCA), a two-stage framework for effective domain adaptive retrieval. In the first stage, a set of orthogonal prototypes directly establishes class-level semantic connections, maximizing inter-class separability while gathering intra-class samples. During the prototype learning, geometric proximity provides a reliability indicator for semantic consistency alignment through adaptive weighting of pseudo-label confidences. The resulting membership matrix and prototypes facilitate feature reconstruction, ensuring quantization on reconstructed rather than original features, thereby improving subsequent hash coding quality and seamlessly connecting both stages. In the second stage, domain-specific quantization functions process the reconstructed features under mutual approximation constraints, generating unified binary hash codes across domains. Extensive experiments validate PSCA's superior performance across multiple datasets.
comment : This paper was accepted by AAAI2026 main tech track not long ago. This is an expanded version with an appendix
☆ UW-BioNLP at ChemoTimelines 2025: Thinking, Fine-Tuning, and Dictionary-Enhanced LLM Systems for Chemotherapy Timeline Extraction
Tianmai M. Zhang, Zhaoyi Sun, Sihang Zeng, Chenxi Li, Neil F. Abernethy, Barbara D. Lam, Fei Xia, Meliha Yetisgen
The ChemoTimelines shared task benchmarks methods for constructing timelines of systemic anticancer treatment from electronic health records of cancer patients. This paper describes our methods, results, and findings for subtask 2 -- generating patient chemotherapy timelines from raw clinical notes. We evaluated strategies involving chain-of-thought thinking, supervised fine-tuning, direct preference optimization, and dictionary-based lookup to improve timeline extraction. All of our approaches followed a two-step workflow, wherein an LLM first extracted chemotherapy events from individual clinical notes, and then an algorithm normalized and aggregated events into patient-level timelines. Each specific method differed in how the associated LLM was utilized and trained. Multiple approaches yielded competitive performances on the test set leaderboard, with fine-tuned Qwen3-14B achieving the best official score of 0.678. Our results and analyses could provide useful insights for future attempts on this task as well as the design of similar tasks.
comment : To be published in Proceedings of the 7th Clinical Natural Language Processing Workshop
☆ BiT Agent : A Task-Aware Modular Framework for Bidirectional Coupling between Multimodal Large Language Models and Wo rl d Models
Yu-Wei Zhan, Xin Wang, Pengzhe Mao, Tongtong Feng, Ren Wang, Wenwu Zhu
Building generalist embodied agents requires a unified system that can interpret multimodal goals, model environment dynamics, and execute reliable actions across diverse real-world tasks. Multimodal large language models (MLLMs) offer strong semantic priors and cross-modal generalization, while world models (WMs) provide actionable latent dynamics for prediction and control. Their combination holds promise for open-ended embodied intelligence, yet introduces two key challenges: (1) establishing a tight coupling between the semantic intent from MLLMs and the dynamic state representations within the WM's latent space, and (2) achieving task-aware adaptability that supports multi-task learning and cross-environment generalization. To address these limitations, we propose BiTAgent, a task-aware dynamic joint framework that enables bidirectional coupling between MLLMs and WMs. BiTAgent establishes two complementary pathways: a forward path that injects MLLM representations into the WM's latent space for semantically guided imagination, and a backward path where WM-generated feedback refines the MLLM's semantic space via dense text-conditioned rewards. This bidirectional interaction is realized through three synergistic components: Task-Aware Dynamic Joint Learning, Task-Aware Behavior Learning, and MLLM-WM Joint Optimization, which together harmonize semantic reasoning and dynamic prediction. Extensive experiments across multi-task and cross-environment settings demonstrate superior stability and generalization over state-of-the-art baselines, marking a step toward open-ended embodied learning.
☆ A Modular Cognitive Architecture for Assisted Reasoning : The Nemosine Framework
Edervaldo Melo
This paper presents the Nemosine Framework, a modular cognitive architecture designed to support assisted reasoning, structured thinking, and systematic analysis. The model operates through functional cognitive modules ("personas") that organize tasks such as planning, evaluation, cross-checking, and narrative synthesis. The framework combines principles from metacognition, distributed cognition, and modular cognitive systems to offer an operational structure for assisted problem-solving and decision support. The architecture is documented through formal specification, internal consistency criteria, and reproducible structural components. The goal is to provide a clear conceptual basis for future computational implementations and to contribute to the study of symbolic-modular architectures for reasoning.
comment : 6 pages, 1 figure. First version
☆ Persona-based Multi- Agent Collaboration for Brainstorming
Nate Straub, Saara Khan, Katharina Jay, Brian Cabral, Oskar Linde
We demonstrate the importance of persona-based multi-agents brainstorming for both diverse topics and subject matter ideation. Prior work has shown that generalized multi-agent collaboration often provides better reasoning than a single agent alone. In this paper, we propose and develop a framework for persona-based agent selection, showing how persona domain curation can improve brainstorming outcomes. Using multiple experimental setups, we evaluate brainstorming outputs across different persona pairings (e.g., Doctor vs VR Engineer) and A2A (agent-to-agent) dynamics (separate, together, separate-then-together). Our results show that (1) persona choice shapes idea domains, (2) collaboration mode shifts diversity of idea generation, and (3) multi-agent persona-driven brainstorming produces idea depth and cross-domain coverage.
comment : 12 pages, 8 figures
☆ AI-Assisted Game Management Decisions: A Fuzzy Logic Approach to Real-Time Substituitions
Pedro Passos
In elite soccer, substitution decisions entail significant financial and sporting consequences yet remain heavily reliant on intuition or predictive models that merely mimic historical biases. This paper introduces a Fuzzy Logic based Decision Support System (DSS) designed for real time, prescriptive game management. Unlike traditional Machine Learning approaches that encounter a predictive ceiling by attempting to replicate human behavior, our system audits performance through an objective, rule based inference engine. We propose a methodological advancement by reformulating the PlayeRank metric into a Cumulative Mean with Role Aware Normalization, eliminating the play time exposure bias inherent in cumulative sum models to enable accurate intra match comparison. The system integrates this refined metric with physiological proxies (fatigue) and contextual variables (disciplinary risk modulated by tactical role) to calculate a dynamic Substitution Priority (P final). Validation via a case study of the 2018 FIFA World Cup match between Brazil and Belgium demonstrates the system's ecological validity: it not only aligned with expert consensus on executed substitutions (for example Gabriel Jesus) but, crucially, identified high risk scenarios ignored by human decision makers. Specifically, the model flagged the "FAGNER Paradox" - a maximum priority defensive risk - minutes before a critical yellow card, and detected the "Lukaku Paradox", where an isolated assist masked a severe drop in participation. These results confirm that Fuzzy Logic offers a transparent, explainable, and superior alternative to black box models for optimizing real time tactical decisions.
comment : 33 pages, 7 figures
☆ GraphBench: Next-generation graph learning benchmark ing
Timo Stoll, Chendi Qian, Ben Finkelshtein, Ali Parviz, Darius Weber, Fabrizio Frasca, Hadar Shavit, Antoine Siraudin, Arman Mielke, Marie Anastacio, Erik Müller, Maya Bechler-Speicher, Michael Bronstein, Mikhail Galkin, Holger Hoos, Mathias Niepert, Bryan Perozzi, Jan Tönshoff, Christopher Morris
Machine learning on graphs has recently achieved impressive progress in various domains, including molecular property prediction and chip design. However, benchmarking practices remain fragmented, often relying on narrow, task-specific datasets and inconsistent evaluation protocols, which hampers reproducibility and broader progress. To address this, we introduce GraphBench, a comprehensive benchmarking suite that spans diverse domains and prediction tasks, including node-level, edge-level, graph-level, and generative settings. GraphBench provides standardized evaluation protocols -- with consistent dataset splits and performance metrics that account for out-of-distribution generalization -- as well as a unified hyperparameter tuning framework. Additionally, we benchmark GraphBench using message-passing neural networks and graph transformer models, providing principled baselines and establishing a reference performance. See www.graphbench.io for further details.
☆ Mathematical Framing for Different Agent Strategies
Philip Stephens, Emmanuel Salawu
We introduce a unified mathematical and probabilistic framework for understanding and comparing diverse AI agent strategies. We bridge the gap between high-level agent design concepts, such as ReAct, multi-agent systems, and control flows, and a rigorous mathematical formulation. Our approach frames agentic processes as a chain of probabilities, enabling a detailed analysis of how different strategies manipulate these probabilities to achieve desired outcomes. Our framework provides a common language for discussing the trade-offs inherent in various agent architectures. One of our many key contributions is the introduction of the "Degrees of Freedom" concept, which intuitively differentiates the optimizable levers available for each approach, thereby guiding the selection of appropriate strategies for specific tasks. This work aims to enhance the clarity and precision in designing and evaluating AI agents, offering insights into maximizing the probability of successful actions within complex agentic systems.
☆ MA RL Warehouse Robots
Price Allman, Lian Thang, Dre Simmons, Salmon Riaz
We present a comparative study of multi-agent reinforcement learning (MARL) algorithms for cooperative warehouse robotics. We evaluate QMIX and IPPO on the Robotic Warehouse (RWARE) environment and a custom Unity 3D simulation. Our experiments reveal that QMIX's value decomposition significantly outperforms independent learning approaches (achieving 3.25 mean return vs. 0.38 for advanced IPPO), but requires extensive hyperparameter tuning -- particularly extended epsilon annealing (5M+ steps) for sparse reward discovery. We demonstrate successful deployment in Unity ML-Agents, achieving consistent package delivery after 1M training steps. While MARL shows promise for small-scale deployments (2-4 robots), significant scaling challenges remain. Code and analyses: https://pallman14.github.io/MARL-QMIX-Warehouse-Robots/
comment : 6 pages, 4 tables. Project documentation: https://pallman14.github.io/MARL-QMIX-Warehouse-Robots/
☆ GuidNoise: Single-Pair Guided Diffusion for Generalized Noise Synthesis AAAI2026
Changjin Kim, HyeokJun Lee, YoungJoon Yoo
Recent image denoising methods have leveraged generative modeling for real noise synthesis to address the costly acquisition of real-world noisy data. However, these generative models typically require camera metadata and extensive target-specific noisy-clean image pairs, often showing limited generalization between settings. In this paper, to mitigate the prerequisites, we propose a Single-Pair Guided Diffusion for generalized noise synthesis GuidNoise, which uses a single noisy/clean pair as the guidance, often easily obtained by itself within a training set. To train GuidNoise, which generates synthetic noisy images from the guidance, we introduce a guidance-aware affine feature modification (GAFM) and a noise-aware refine loss to leverage the inherent potential of diffusion models. This loss function refines the diffusion model's backward process, making the model more adept at generating realistic noise distributions. The GuidNoise synthesizes high-quality noisy images under diverse noise environments without additional metadata during both training and inference. Additionally, GuidNoise enables the efficient generation of noisy-clean image pairs at inference time, making synthetic noise readily applicable for augmenting training data. This self-augmentation significantly improves denoising performance, especially in practical scenarios with lightweight models and limited training data. The code is available at https://github.com/chjinny/GuidNoise.
comment : AAAI2026
☆ Open-Ended Goal Inference through Actions and Language for Human-Robot Collaboration
Debasmita Ghose, Oz Gitelson, Marynel Vazquez, Brian Scassellati
To collaborate with humans, robots must infer goals that are often ambiguous, difficult to articulate, or not drawn from a fixed set. Prior approaches restrict inference to a predefined goal set, rely only on observed actions, or depend exclusively on explicit instructions, making them brittle in real-world interactions. We present BALI (Bidirectional Action-Language Inference) for goal prediction, a method that integrates natural language preferences with observed human actions in a receding-horizon planning tree. BALI combines language and action cues from the human, asks clarifying questions only when the expected information gain from the answer outweighs the cost of interruption, and selects supportive actions that align with inferred goals. We evaluate the approach in collaborative cooking tasks, where goals may be novel to the robot and unbounded. Compared to baselines, BALI yields more stable goal predictions and significantly fewer mistakes.
comment : Accepted to ACM/IEEE International Conference on Human-Robot Interaction, 2026 (HRI 2026), 10 pages, 4 figures
☆ NORi: An ML-Augmented Ocean Boundary Layer Parameterization
Xin Kai Lee, Ali Ramadhan, Andre Souza, Gregory LeClaire Wagner, Simone Silvestri, John Marshall, Raffaele Ferrari
NORi is a machine-learned (ML) parameterization of ocean boundary layer turbulence that is physics-based and augmented with neural networks. NORi stands for neural ordinary differential equations (NODEs) Richardson number (Ri) closure. The physical parameterization is controlled by a Richardson number-dependent diffusivity and viscosity. The NODEs are trained to capture the entrainment through the base of the boundary layer, which cannot be represented with a local diffusive closure. The parameterization is trained using large-eddy simulations in an "a posteriori" fashion, where parameters are calibrated with a loss function that explicitly depends on the actual time-integrated variables of interest rather than the instantaneous subgrid fluxes, which are inherently noisy. NORi is designed for the realistic nonlinear equation of state of seawater and demonstrates excellent prediction and generalization capabilities in capturing entrainment dynamics under different convective strengths, oceanic background stratifications, rotation strengths, and surface wind forcings. NORi is numerically stable for at least 100 years of integration time in large-scale simulations, despite only being trained on 2-day horizons, and can be run with time steps as long as one hour. The highly expressive neural networks, combined with a physically-rigorous base closure, prove to be a robust paradigm for designing parameterizations for climate models where data requirements are drastically reduced, inference performance can be directly targeted and optimized, and numerical stability is implicitly encouraged during training.
comment : 48 pages, 16 figures, submitted to Journal of Advances in Modeling Earth Systems (JAMES)
☆ Automating Complex Document Workflows via Stepwise and Rollback-Enabled Operation Orchestration AAAI-2026
Yanbin Zhang, Hanhui Ye, Yue Bai, Qiming Zhang, Liao Xiang, Wu Mianzhi, Renjun Hu
Workflow automation promises substantial productivity gains in everyday document-related tasks. While prior agentic systems can execute isolated instructions, they struggle with automating multi-step, session-level workflows due to limited control over the operational process. To this end, we introduce AutoDW, a novel execution framework that enables stepwise, rollback-enabled operation orchestration. AutoDW incrementally plans API actions conditioned on user instructions, intent-filtered API candidates, and the evolving states of the document. It further employs robust rollback mechanisms at both the argument and API levels, enabling dynamic correction and fault tolerance. These designs together ensure that the execution trajectory of AutoDW remains aligned with user intent and document context across long-horizon workflows. To assess its effectiveness, we construct a comprehensive benchmark of 250 sessions and 1,708 human-annotated instructions, reflecting realistic document processing scenarios with interdependent instructions. AutoDW achieves 90% and 62% completion rates on instruction- and session-level tasks, respectively, outperforming strong baselines by 40% and 76%. Moreover, AutoDW also remains robust for the decision of backbone LLMs and on tasks with varying difficulty. Code and data will be open-sourced. Code: https://github.com/YJett/AutoDW
comment : 9 pages, 3 figures, accepted by AAAI-2026
☆ TaskEval: Synthesised Evaluation for Foundation-Model Tasks
Dilani Widanapathiranage, Scott Barnett, Stefanus Kurniawan, Wannita Takerngsaksiri
Hallucinations are a key concern when creating applications that rely on Foundation models (FMs). Understanding where and how these subtle failures occur in an application relies on evaluation methods known as \textit{evals}. Prior work focuses on defining new eval methods or benchmark datasets for specific tasks. However, neither helps a software team with a task-specific FM application when there is no metric or dataset. The demand for both automated approaches and deep integration of human insight makes this a challenging problem. We address this gap by proposing an approach to synthesise a FM task-specific evaluator program that provides automation and a custom UI for capturing feedback. The core novelty of our approach lies in: (1) a task-agnostic meta-model that captures properties of any FM task, (2) an interaction protocol for efficient use of human feedback, and (3) an eval synthesiser that selects or generates an appropriate set of evals. We implement our approach in \toolname and demonstrate the concept on two diverse FM tasks: chart data extraction and document question answering. A preliminary evaluation on the quality of our selected evals shows 93\% and 90\% accuracy respectively. Our research tackles a growing problem facing engineering teams, how to evaluate and review outputs from FM tasks.
comment : 5 pages, 3 figures
☆ Explainable Parkinsons Disease Gait Recognition Using Multimodal RGB-D Fusion and Large Language Models
Manar Alnaasan, Md Selim Sarowar, Sungho Kim
Accurate and interpretable gait analysis plays a crucial role in the early detection of Parkinsons disease (PD),yet most existing approaches remain limited by single-modality inputs, low robustness, and a lack of clinical transparency. This paper presents an explainable multimodal framework that integrates RGB and Depth (RGB-D) data to recognize Parkinsonian gait patterns under realistic conditions. The proposed system employs dual YOLOv11-based encoders for modality-specific feature extraction, followed by a Multi-Scale Local-Global Extraction (MLGE) module and a Cross-Spatial Neck Fusion mechanism to enhance spatial-temporal representation. This design captures both fine-grained limb motion (e.g., reduced arm swing) and overall gait dynamics (e.g., short stride or turning difficulty), even in challenging scenarios such as low lighting or occlusion caused by clothing. To ensure interpretability, a frozen Large Language Model (LLM) is incorporated to translate fused visual embeddings and structured metadata into clinically meaningful textual explanations. Experimental evaluations on multimodal gait datasets demonstrate that the proposed RGB-D fusion framework achieves higher recognition accuracy, improved robustness to environmental variations, and clear visual-linguistic reasoning compared with single-input baselines. By combining multimodal feature learning with language-based interpretability, this study bridges the gap between visual recognition and clinical understanding, offering a novel vision-language paradigm for reliable and explainable Parkinsons disease gait analysis. Code:https://github.com/manaralnaasan/RGB-D_parkinson-LLM
☆ Solving LLM Repetition Problem in Production: A Comprehensive Study of Multiple Solutions
Weiwei Wang, Weijie Zou, Jiyong Min
The repetition problem, where Large Language Models (LLMs) continuously generate repetitive content without proper termination, poses a critical challenge in production deployments, causing severe performance degradation and system stalling. This paper presents a comprehensive investigation and multiple practical solutions for the repetition problem encountered in real-world batch code interpretation tasks. We identify three distinct repetition patterns: (1) business rule generation repetition, (2) method call relationship analysis repetition, and (3) PlantUML diagram syntax generation repetition. Through rigorous theoretical analysis based on Markov models, we establish that the root cause lies in greedy decoding's inability to escape repetitive loops, exacerbated by self-reinforcement effects. Our comprehensive experimental evaluation demonstrates three viable solutions: (1) Beam Search decoding with early_stopping=True serves as a universal post-hoc mechanism that effectively resolves all three repetition patterns; (2) presence_penalty hyperparameter provides an effective solution specifically for BadCase 1; and (3) Direct Preference Optimization (DPO) fine-tuning offers a universal model-level solution for all three BadCases. The primary value of this work lies in combining first-hand production experience with extensive experimental validation. Our main contributions include systematic theoretical analysis of repetition mechanisms, comprehensive evaluation of multiple solutions with task-specific applicability mapping, identification of early_stopping as the critical parameter for Beam Search effectiveness, and practical production-ready solutions validated in real deployment environments.
☆ GovBench: Benchmark ing LLM Agent s for Real-Wo rl d Data Governance Workflows
Zhou Liu, Zhaoyang Han, Guochen Yan, Hao Liang, Bohan Zeng, Xing Chen, Yuanfeng Song, Wentao Zhang
Data governance ensures data quality, security, and compliance through policies and standards, a critical foundation for scaling modern AI development. Recently, large language models (LLMs) have emerged as a promising solution for automating data governance by translating user intent into executable transformation code. However, existing benchmarks for automated data science often emphasize snippet-level coding or high-level analytics, failing to capture the unique challenge of data governance: ensuring the correctness and quality of the data itself. To bridge this gap, we introduce GovBench, a benchmark featuring 150 diverse tasks grounded in real-world scenarios, built on data from actual cases. GovBench employs a novel "reversed-objective" methodology to synthesize realistic noise and utilizes rigorous metrics to assess end-to-end pipeline reliability. Our analysis on GovBench reveals that current models struggle with complex, multi-step workflows and lack robust error-correction mechanisms. Consequently, we propose DataGovAgent, a framework utilizing a Planner-Executor-Evaluator architecture that integrates constraint-based planning, retrieval-augmented generation, and sandboxed feedback-driven debugging. Experimental results show that DataGovAgent significantly boosts the Average Task Score (ATS) on complex tasks from 39.7 to 54.9 and reduces debugging iterations by over 77.9 percent compared to general-purpose baselines.
comment : Equal contribution: Zhou Liu and Zhaoyang Han. Corresponding authors: Yuanfeng Song and Wentao Zhang
☆ Dual-Stream Spectral Decoupling Distillation for Remote Sensing Object Detection
Xiangyi Gao, Danpei Zhao, Bo Yuan, Wentao Li
Knowledge distillation is an effective and hardware-friendly method, which plays a key role in lightweighting remote sensing object detection. However, existing distillation methods often encounter the issue of mixed features in remote sensing images (RSIs), and neglect the discrepancies caused by subtle feature variations, leading to entangled knowledge confusion. To address these challenges, we propose an architecture-agnostic distillation method named Dual-Stream Spectral Decoupling Distillation (DS2D2) for universal remote sensing object detection tasks. Specifically, DS2D2 integrates explicit and implicit distillation grounded in spectral decomposition. Firstly, the first-order wavelet transform is applied for spectral decomposition to preserve the critical spatial characteristics of RSIs. Leveraging this spatial preservation, a Density-Independent Scale Weight (DISW) is designed to address the challenges of dense and small object detection common in RSIs. Secondly, we show implicit knowledge hidden in subtle student-teacher feature discrepancies, which significantly influence predictions when activated by detection heads. This implicit knowledge is extracted via full-frequency and high-frequency amplifiers, which map feature differences to prediction deviations. Extensive experiments on DIOR and DOTA datasets validate the effectiveness of the proposed method. Specifically, on DIOR dataset, DS2D2 achieves improvements of 4.2% in AP50 for RetinaNet and 3.8% in AP50 for Faster R-CNN, outperforming existing distillation approaches. The source code will be available at https://github.com/PolarAid/DS2D2.
comment : 12 pages, 8 figures, 11 tables
☆ Executable Governance for AI: Translating Policies into Rules Using LLMs AAAI-26
Gautam Varma Datla, Anudeep Vurity, Tejaswani Dash, Tazeem Ahmad, Mohd Adnan, Saima Rafi
AI policy guidance is predominantly written as prose, which practitioners must first convert into executable rules before frameworks can evaluate or enforce them. This manual step is slow, error-prone, difficult to scale, and often delays the use of safeguards in real-world deployments. To address this gap, we present Policy-to-Tests (P2T), a framework that converts natural-language policy documents into normalized, machine-readable rules. The framework comprises a pipeline and a compact domain-specific language (DSL) that encodes hazards, scope, conditions, exceptions, and required evidence, yielding a canonical representation of extracted rules. To test the framework beyond a single policy, we apply it across general frameworks, sector guidance, and enterprise standards, extracting obligation-bearing clauses and converting them into executable rules. These AI-generated rules closely match strong human baselines on span-level and rule-level metrics, with robust inter-annotator agreement on the gold set. To evaluate downstream behavioral and safety impact, we add HIPAA-derived safeguards to a generative agent and compare it with an otherwise identical agent without guardrails. An LLM-based judge, aligned with gold-standard criteria, measures violation rates and robustness to obfuscated and compositional prompts. Detailed results are provided in the appendix. We release the codebase, DSL, prompts, and rule sets as open-source resources to enable reproducible evaluation.
comment : Accepted to AAAI-26 AI Governance Workshop (in-person presentation); 10 pages, 5 figures
☆ Towards 6G Native-AI Edge Networks: A Semantic-Aware and Agent ic Intelligence Paradigm
Chenyuan Feng, Anbang Zhang, Geyong Min, Yongming Huang, Tony Q. S. Quek, Xiaohu You
The evolution toward sixth-generation wireless systems positions intelligence as a native network capability, fundamentally transforming the design of radio access networks (RANs). Within this vision, Semantic-native communication and agentic intelligence are expected to play central roles. SemCom departs from bit-level fidelity and instead emphasizes task-oriented meaning exchange, enabling compact SC and introducing new performance measures such as semantic fidelity and task success rate. Agentic intelligence endows distributed RAN entities with goal-driven autonomy, reasoning, planning, and multi-agent collaboration, increasingly supported by foundation models and knowledge graphs. In this work, we first introduce the conceptual foundations of SemCom and agentic networking, and discuss why existing AI-driven O-RAN solutions remain largely bit-centric and task-siloed. We then present a unified taxonomy that organizes recent research along three axes: i) semantic abstraction level (symbol/feature/intent/knowledge), ii) agent autonomy and coordination granularity (single-, multi-, and hierarchical-agent), and iii) RAN control placement across PHY/MAC, near-real-time RIC, and non-real-time RIC. Based on this taxonomy, we systematically introduce enabling technologies including task-oriented semantic encoders/decoders, multi-agent reinforcement learning, foundation-model-assisted RAN agents, and knowledge-graph-based reasoning for cross-layer awareness. Representative 6G use cases, such as immersive XR, vehicular V2X, and industrial digital twins, are analyzed to illustrate the semantic-agentic convergence in practice. Finally, we identify open challenges in semantic representation standardization, scalable trustworthy agent coordination, O-RAN interoperability, and energy-efficient AI deployment, and outline research directions toward operational semantic-agentic AI-RAN.
comment : submitted to Digital Communications and Networks
☆ Adversarial Limits of Quantum Certification: When Eve Defeats Detection
Davut Emre Tasar
Security of quantum key distribution (QKD) relies on certifying that observed correlations arise from genuine quantum entanglement rather than eavesdropper manipulation. Theoretical security proofs assume idealized conditions, practical certification must contend with adaptive adversaries who optimize their attack strategies against detection systems. Established fundamental adversarial limits for quantum certification using Eve GAN, a generative adversarial network trained to produce classical correlations indistinguishable from quantum. Our central finding: when Eve interpolates her classical correlations with quantum data at mixing parameter, all tested detection methods achieve ROC AUC = 0.50, equivalent to random guessing. This means an eavesdropper needs only 5% classical admixture to completely evade detection. Critically, we discover that same distribution calibration a common practice in prior certification studies inflates detection performance by 44 percentage points compared to proper cross distribution evaluation, revealing a systematic flaw that may have led to overestimated security claims. Analysis of Popescu Rohrlich (PR Box) regime identifies a sharp phase transition at CHSH S = 2.05: below this value, no statistical method distinguishes classical from quantum correlations; above it, detection probability increases monotonically. Hardware validation on IBM Quantum demonstrates that Eve-GAN achieves CHSH = 2.736, remarkably exceeding real quantum hardware performance (CHSH = 2.691), illustrating that classical adversaries can outperform noisy quantum systems on standard certification metrics. These results have immediate implications for QKD security: adversaries maintaining 95% quantum fidelity evade all tested detection methods. We provide corrected methodology using cross-distribution calibration and recommend mandatory adversarial testing for quantum security claims.
☆ FMA-Net++: Motion- and Exposure-Aware Real-Wo rl d Joint Video Super-Resolution and Deblurring
Geunhyuk Youk, Jihyong Oh, Munchurl Kim
Real-world video restoration is plagued by complex degradations from motion coupled with dynamically varying exposure - a key challenge largely overlooked by prior works and a common artifact of auto-exposure or low-light capture. We present FMA-Net++, a framework for joint video super-resolution and deblurring that explicitly models this coupled effect of motion and dynamically varying exposure. FMA-Net++ adopts a sequence-level architecture built from Hierarchical Refinement with Bidirectional Propagation blocks, enabling parallel, long-range temporal modeling. Within each block, an Exposure Time-aware Modulation layer conditions features on per-frame exposure, which in turn drives an exposure-aware Flow-Guided Dynamic Filtering module to infer motion- and exposure-aware degradation kernels. FMA-Net++ decouples degradation learning from restoration: the former predicts exposure- and motion-aware priors to guide the latter, improving both accuracy and efficiency. To evaluate under realistic capture conditions, we introduce REDS-ME (multi-exposure) and REDS-RE (random-exposure) benchmarks. Trained solely on synthetic data, FMA-Net++ achieves state-of-the-art accuracy and temporal consistency on our new benchmarks and GoPro, outperforming recent methods in both restoration quality and inference speed, and generalizes well to challenging real-world videos.
comment : 20 pages, 15 figures. Project Page: https://kaist-viclab.github.io/fmanetpp_site/
☆ MASE: Interpretable NLP Models via Model-Agnostic Saliency Estimation
Zhou Yang, Shunyan Luo, Jiazhen Zhu, Fang Jin
Deep neural networks (DNNs) have made significant strides in Natural Language Processing (NLP), yet their interpretability remains elusive, particularly when evaluating their intricate decision-making processes. Traditional methods often rely on post-hoc interpretations, such as saliency maps or feature visualization, which might not be directly applicable to the discrete nature of word data in NLP. Addressing this, we introduce the Model-agnostic Saliency Estimation (MASE) framework. MASE offers local explanations for text-based predictive models without necessitating in-depth knowledge of a model's internal architecture. By leveraging Normalized Linear Gaussian Perturbations (NLGP) on the embedding layer instead of raw word inputs, MASE efficiently estimates input saliency. Our results indicate MASE's superiority over other model-agnostic interpretation methods, especially in terms of Delta Accuracy, positioning it as a promising tool for elucidating the operations of text-based models in NLP.
☆ STeP-Diff: Spatio-Temporal Physics-Informed Diffusion Models for Mobile Fine-Grained Pollution Forecasting
Nan Zhou, Weijie Hong, Huandong Wang, Jianfeng Zheng, Qiuhua Wang, Yali Song, Xiao-Ping Zhang, Yong Li, Xinlei Chen
Fine-grained air pollution forecasting is crucial for urban management and the development of healthy buildings. Deploying portable sensors on mobile platforms such as cars and buses offers a low-cost, easy-to-maintain, and wide-coverage data collection solution. However, due to the random and uncontrollable movement patterns of these non-dedicated mobile platforms, the resulting sensor data are often incomplete and temporally inconsistent. By exploring potential training patterns in the reverse process of diffusion models, we propose Spatio-Temporal Physics-Informed Diffusion Models (STeP-Diff). STeP-Diff leverages DeepONet to model the spatial sequence of measurements along with a PDE-informed diffusion model to forecast the spatio-temporal field from incomplete and time-varying data. Through a PDE-constrained regularization framework, the denoising process asymptotically converges to the convection-diffusion dynamics, ensuring that predictions are both grounded in real-world measurements and aligned with the fundamental physics governing pollution dispersion. To assess the performance of the system, we deployed 59 self-designed portable sensing devices in two cities, operating for 14 days to collect air pollution data. Compared to the second-best performing algorithm, our model achieved improvements of up to 89.12% in MAE, 82.30% in RMSE, and 25.00% in MAPE, with extensive evaluations demonstrating that STeP-Diff effectively captures the spatio-temporal dependencies in air pollution fields.
☆ AutoGuard: A Self-Healing Proactive Security Layer for DevSecOps Pipelines Using Reinforcement Learning
Praveen Anugula, Avdhesh Kumar Bhardwaj, Navin Chhibber, Rohit Tewari, Sunil Khemka, Piyush Ranjan
Contemporary DevSecOps pipelines have to deal with the evolution of security in an ever-continuously integrated and deployed environment. Existing methods,such as rule-based intrusion detection and static vulnerability scanning, are inadequate and unreceptive to changes in the system, causing longer response times and organization needs exposure to emerging attack vectors. In light of the previous constraints, we introduce AutoGuard to the DevSecOps ecosystem, a reinforcement learning (RL)-powered self-healing security framework built to pre-emptively protect DevSecOps environments. AutoGuard is a self-securing security environment that continuously observes pipeline activities for potential anomalies while preemptively remediating the environment. The model observes and reacts based on a policy that is continually learned dynamically over time. The RL agent improves each action over time through reward-based learning aimed at improving the agent's ability to prevent, detect and respond to a security incident in real-time. Testing using simulated ContinuousIntegration / Continuous Deployment (CI/CD) environments showed AutoGuard to successfully improve threat detection accuracy by 22%, reduce mean time torecovery (MTTR) for incidents by 38% and increase overall resilience to incidents as compared to traditional methods. Keywords- DevSecOps, Reinforcement Learning, Self- Healing Security, Continuous Integration, Automated Threat Mitigation
comment : Accepted and Presented at 1st IEEE Uttar Pradesh Section Women in Engineering International Conference on Electrical Electronics and Computer Engineering (UPWIECON 2025) organized by NIELIT Dehradun held during 30th 31st October 2025
☆ Agent Bay: A Hybrid Interaction Sandbox for Seamless Human-AI Intervention in Agent ic Systems
Yun Piao, Hongbo Min, Hang Su, Leilei Zhang, Lei Wang, Yue Yin, Xiao Wu, Zhejing Xu, Liwei Qu, Hang Li, Xinxin Zeng, Wei Tian, Fei Yu, Xiaowei Li, Jiayi Jiang, Tongxu Liu, Hao Tian, Yufei Que, Xiaobing Tu, Bing Suo, Yuebing Li, Xiangting Chen, Zeen Zhao, Jiaming Tang, Wei Huang, Xuguang Li, Jing Zhao, Jin Li, Jie Shen, Jinkui Ren, Xiantao Zhang
The rapid advancement of Large Language Models (LLMs) is catalyzing a shift towards autonomous AI Agents capable of executing complex, multi-step tasks. However, these agents remain brittle when faced with real-world exceptions, making Human-in-the-Loop (HITL) supervision essential for mission-critical applications. In this paper, we present AgentBay, a novel sandbox service designed from the ground up for hybrid interaction. AgentBay provides secure, isolated execution environments spanning Windows, Linux, Android, Web Browsers, and Code interpreters. Its core contribution is a unified session accessible via a hybrid control interface: An AI agent can interact programmatically via mainstream interfaces (MCP, Open Source SDK), while a human operator can, at any moment, seamlessly take over full manual control. This seamless intervention is enabled by Adaptive Streaming Protocol (ASP). Unlike traditional VNC/RDP, ASP is specifically engineered for this hybrid use case, delivering an ultra-low-latency, smoother user experience that remains resilient even in weak network environments. It achieves this by dynamically blending command-based and video-based streaming, adapting its encoding strategy based on network conditions and the current controller (AI or human). Our evaluation demonstrates strong results in security, performance, and task completion rates. In a benchmark of complex tasks, the AgentBay (Agent + Human) model achieved more than 48% success rate improvement. Furthermore, our ASP protocol reduces bandwidth consumption by up to 50% compared to standard RDP, and in end-to-end latency with around 5% reduction, especially under poor network conditions. We posit that AgentBay provides a foundational primitive for building the next generation of reliable, human-supervised autonomous systems.
☆ Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning
Hongye Cao, Zhixin Bai, Ziyue Peng, Boyan Wang, Tianpei Yang, Jing Huo, Yuyao Zhang, Yang Gao
Reinforcement learning with verifiable rewards (RLVR) has demonstrated superior performance in enhancing the reasoning capability of large language models (LLMs). However, this accuracy-oriented learning paradigm often suffers from entropy collapse, which reduces policy exploration and limits reasoning capabilities. To address this challenge, we propose an efficient reinforcement learning framework that leverages entropy signals at both the semantic and token levels to improve reasoning. From the data perspective, we introduce semantic entropy-guided curriculum learning, organizing training data from low to high semantic entropy to guide progressive optimization from easier to more challenging tasks. For the algorithmic design, we adopt non-uniform token treatment by imposing KL regularization on low-entropy tokens that critically impact policy exploration and applying stronger constraints on high-covariance portions within these tokens. By jointly optimizing data organization and algorithmic design, our method effectively mitigates entropy collapse and enhances LLM reasoning. Experimental results across 6 benchmarks with 3 different parameter-scale base models demonstrate that our method outperforms other entropy-based approaches in improving reasoning.
☆ Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignment WACV
Kai-Po Chang, Wei-Yuan Cheng, Chi-Pin Huang, Fu-En Yang, Yu-Chiang Frank Wang
Recent advancement in multimodal LLMs (MLLMs) has demonstrated their remarkable capability to generate descriptive captions for input videos. However, these models suffer from factual inaccuracies in the generated descriptions, causing severe hallucination issues. While prior works have explored alleviating hallucinations for static images, jointly mitigating visual object and temporal action hallucinations for dynamic videos remains a challenging and unsolved task. To tackle this challenge, we propose a Self-Augmented Contrastive Alignment (SANTA) framework for enabling object and action faithfulness by exempting the spurious correlations and enforcing the emphasis on visual facts. SANTA employs a hallucinative self-augmentation scheme to identify the potential hallucinations that lie in the MLLM and transform the original captions to the contrasted negatives. Furthermore, we develop a tracklet-phrase contrastive alignment to match the regional objects and relation-guided actions with their corresponding visual and temporal phrases. Extensive experiments demonstrate that SANTA outperforms existing methods in alleviating object and action hallucinations, yielding superior performance on the hallucination examination benchmarks.
comment : IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2026. Project page: https://kpc0810.github.io/santa/
☆ Counting Without Running: Evaluating LLMs' Reasoning About Code Complexity
Gregory Bolet, Giorgis Georgakoudis, Konstantinos Parasyris, Harshitha Menon, Niranjan Hasabnis, Kirk W. Cameron, Gal Oren
Modern GPU software stacks demand developers who can anticipate performance bottlenecks before ever launching a kernel; misjudging floating-point workloads upstream can derail tuning, scheduling, and even hardware procurement. Yet despite rapid progress in code generation, today's Large Language Models (LLMs) are rarely tested on this kind of forward-looking reasoning. We close that gap with gpuFLOPBench, a benchmark that asks models to "count without running" by predicting single and double-precision FLOP counts for 577 CUDA kernels drawn from HeCBench, annotated with ground-truth profiles and eight execution attributes that distinguish trivially analyzable code from kernels whose FLOPs depend on hidden compiler or runtime behavior. Evaluating current closed-source reasoning models shows clear but uneven progress: the newest LLMs achieve perfect classification on straightforward kernels but still incur multiple order-of-magnitude errors whenever implicit FLOPs arise from division, intrinsic math functions, or common subexpressions. These results surface a core limitation of existing code assistants -- the inability to internalize hardware-specific microcode effects -- and position gpuFLOPBench as a focused testbed for developing LLM tooling that can reason about performance with the same rigor as experienced GPU developers. Sources are available at our repository: https://github.com/Scientific-Computing-Lab/gpuFLOPBench
comment : 13 pages, 6 figures, MLSys 2026 Submission
♻ ☆ BioAnalyst: A Foundation Model for Biodiversity
Athanasios Trantas, Martino Mensio, Stylianos Stasinos, Sebastian Gribincea, Taimur Khan, Damian Podareanu, Aliene van der Veen
Multimodal Foundation Models (FMs) offer a path to learn general-purpose representations from heterogeneous ecological data, easily transferable to downstream tasks. However, practical biodiversity modelling remains fragmented; separate pipelines and models are built for each dataset and objective, which limits reuse across regions and taxa. In response, we present BioAnalyst, to our knowledge the first multimodal Foundation Model tailored to biodiversity analysis and conservation planning in Europe at $0.25^{\circ}$ spatial resolution targeting regional to national-scale applications. BioAnalyst employs a transformer-based architecture, pre-trained on extensive multimodal datasets that align species occurrence records with remote sensing indicators, climate and environmental variables. Post pre-training, the model is adapted via lightweight roll-out fine-tuning to a range of downstream tasks, including joint species distribution modelling, biodiversity dynamics and population trend forecasting. The model is evaluated on two representative downstream use cases: (i) joint species distribution modelling and with 500 vascular plant species (ii) monthly climate linear probing with temperature and precipitation data. Our findings show that BioAnalyst can provide a strong baseline both for biotic and abiotic tasks, acting as a macroecological simulator with a yearly forecasting horizon and monthly resolution, offering the first application of this type of modelling in the biodiversity domain. We have open-sourced the model weights, training and fine-tuning pipelines to advance AI-driven ecological research.
♻ ☆ Empowering Clients -- Transformation of Design Processes Due to Generative AI
Johannes Schneider, Kilic Sinem, Daniel Stockhammer
Generative AI (GenAI) is transforming creative fields shaping our culture and our heritage. We focus on widespread interactions between clients and (creative) specialists highlighting a change in interaction patterns leading to a shift from the use of expert creativity towards AI-supported client creativity. More specifically, we explore the case of architecture as designing houses is complex involving extensive customer interaction. We investigate the effects of GenAI on the architectural design process and discuss the role of the architect. Our study involved six architects using a general-purpose text-to-image tool for generating designs and providing feedback followed by expert interviews. We find that AI can disrupt the ideation phase by enabling clients to engage in the design process through rapid visualization of their ideas. In turn, so our thesis, the architect's role shifts towards assessing feasibility of such designs. AI's feedback, though valuable, can hamper creativity and innovation by suggesting altering novel, innovative approaches towards more standardized designs. We find that there is uncertainty among architects about the interpretative sovereignty of architecture and identity when AI increasingly takes over authorship. Our findings can also support the design of future AI systems by pinpointing weaknesses and highlighting a novel design process calling for tighter client integration. In our discussion, we also generalize our findings on a broader societal level elaborating on the change of a number of characteristics such as power, capability and responsibility in the triangle of AI, experts, and non-experts. We also discuss risks such as cultural uniformity when it comes to using AI to design artifacts central to our cultural heritage.
♻ ☆ Aud Agent : Automated Auditing of Privacy Policy Compliance in AI Agent s
Ye Zheng, Yidan Hu
AI agents can autonomously perform tasks and, often without explicit user consent, collect or disclose users' sensitive local data, which raises serious privacy concerns. Although AI agents' privacy policies describe their intended data practices, there remains limited transparency and accountability about whether runtime behavior matches those policies. To close this gap, we introduce AudAgent, a visual tool that continuously monitors AI agents' data practices in real time and guards compliance with stated privacy policies. AudAgent consists of four components for automated privacy auditing of AI agents. (i) Policy formalization: a novel cross-LLM voting mechanism to guarantee confidence of the parsed privacy policy model. (ii) Runtime annotation: a lightweight Presidio-based analyzer detects sensitive data and annotates data practices based on the AI agent's context and the privacy policy model. (iii) Compliance auditing: ontology graphs and automata-based checking connect the privacy policy model with runtime annotations, enabling on-the-fly compliance checking. (iv) User interface: an infrastructure-independent implementation visualizes the real-time execution trace of AI agents along with potential privacy policy violations, providing user-friendly transparency and accountability. We evaluate AudAgent with AI agents built using mainstream frameworks, demonstrating its effectiveness in detecting and visualizing privacy policy violations in real time. Using AudAgent, we also find that most privacy policies omit explicit safeguards for highly sensitive data such as SSNs, whose misuse violates legal requirements, and that many agents do not refuse handling such data via third-party tools, including those controlled by Claude, Gemini, and DeepSeek. AudAgent proactively blocks operations on such data, overriding the agents' original privacy policy and behavior.
comment : Submitted to PETS'26 Issue 3
♻ ☆ MORPH: PDE Foundation Models with Arbitrary Data Modality
Mahindra Singh Rautela, Alexander Most, Siddharth Mansingh, Bradley C. Love, Ayan Biswas, Diane Oyen, Earl Lawrence
We introduce MORPH, a modality-agnostic, autoregressive foundation model for partial differential equations (PDEs). MORPH is built on a convolutional vision transformer backbone that seamlessly handles heterogeneous spatiotemporal datasets of varying data modality (1D--3D) at different resolutions, and multiple fields with mixed scalar and vector components. The architecture combines (i) component-wise convolution, which jointly processes scalar and vector channels to capture local interactions, (ii) inter-field cross-attention, which models and selectively propagates information between different physical fields, (iii) axial attentions, which factorize full spatiotemporal self-attention along individual spatial and temporal axes to reduce computational burden while retaining expressivity. We pretrain multiple model variants on a diverse collection of heterogeneous PDE datasets and evaluate transfer to a range of downstream prediction tasks. Using both full-model fine-tuning and parameter-efficient low-rank adapters (LoRA), MORPH outperforms models trained from scratch. Across extensive evaluations, MORPH matches or surpasses strong baselines and recent state-of-the-art models. Collectively, these capabilities present a flexible and powerful backbone for learning from the heterogeneous and multimodal nature of scientific observations, charting a path toward scalable and data-efficient scientific machine learning. The source code, datasets, and models are publicly available at https://github.com/lanl/MORPH.
♻ ☆ The Delusional Hedge Algorithm as a Model of Human Learning from Diverse Opinions
Yun-Shiuan Chuang, Jerry Zhu, Timothy T. Rogers
Whereas cognitive models of learning often assume direct experience with both the features of an event and with a true label or outcome, much of everyday learning arises from hearing the opinions of others, without direct access to either the experience or the ground truth outcome. We consider how people can learn which opinions to trust in such scenarios by extending the hedge algorithm: a classic solution for learning from diverse information sources. We first introduce a semi-supervised variant we call the delusional hedge capable of learning from both supervised and unsupervised experiences. In two experiments, we examine the alignment between human judgments and predictions from the standard hedge, the delusional hedge, and a heuristic baseline model. Results indicate that humans effectively incorporate both labeled and unlabeled information in a manner consistent with the delusional hedge algorithm -- suggesting that human learners not only gauge the accuracy of information sources but also their consistency with other reliable sources. The findings advance our understanding of human learning from diverse opinions, with implications for the development of algorithms that better capture how people learn to weigh conflicting information sources.
♻ ☆ Minimum Weighted Feedback Arc Sets for Ranking from Pairwise Comparisons
Soroush Vahidi, Ioannis Koutis
The Minimum Weighted Feedback Arc Set (MWFAS) problem is closely related to the task of deriving a global ranking from pairwise comparisons. Recent work by He et al. (ICML 2022) advanced the state of the art on ranking benchmarks using learning based methods, but did not examine the underlying connection to MWFAS. In this paper, we investigate this relationship and introduce efficient combinatorial algorithms for solving MWFAS as a means of addressing the ranking problem. Our experimental results show that these simple, learning free methods achieve substantially faster runtimes than recent learning based approaches, while also delivering competitive, and in many cases superior, ranking accuracy. These findings suggest that lightweight combinatorial techniques offer a scalable and effective alternative to deep learning for large scale ranking tasks.
comment : This is a preliminary paper
♻ ☆ Path Channels and Plan Extension Kernels: a Mechanistic Description of Planning in a Sokoban RNN NeurIPS 2025
Mohammad Taufeeque, Aaron David Tucker, Adam Gleave, Adrià Garriga-Alonso
We partially reverse-engineer a convolutional recurrent neural network (RNN) trained with model-free reinforcement learning to play the box-pushing game Sokoban. We find that the RNN stores future moves (plans) as activations in particular channels of the hidden state, which we call path channels. A high activation in a particular location means that, when a box is in that location, it will get pushed in the channel's assigned direction. We examine the convolutional kernels between path channels and find that they encode the change in position resulting from each possible action, thus representing part of a learned transition model. The RNN constructs plans by starting at the boxes and goals. These kernels extend activations in path channels forwards from boxes and backwards from the goal. Negative values are placed in channels at obstacles. This causes the extension kernels to propagate the negative value in reverse, thus pruning the last few steps and letting an alternative plan emerge; a form of backtracking. Our work shows that, a precise understanding of the plan representation allows us to directly understand the bidirectional planning-like algorithm learned by model-free training in more familiar terms.
comment : Presented at the Mechanistic Interpretability Workshop at NeurIPS 2025. 34 pages, 26 figures
♻ ☆ Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Model s
Shuai Wang, Zhenhua Liu, Jiaheng Wei, Xuanwu Yin, Dong Li, Emad Barsoum
We present Athena-PRM, a multimodal process reward model (PRM) designed to evaluate the reward score for each step in solving complex reasoning problems. Developing high-performance PRMs typically demands significant time and financial investment, primarily due to the necessity for step-level annotations of reasoning steps. Conventional automated labeling methods, such as Monte Carlo estimation, often produce noisy labels and incur substantial computational costs. To efficiently generate high-quality process-labeled data, we propose leveraging prediction consistency between weak and strong completers as a criterion for identifying reliable process labels. Remarkably, Athena-PRM demonstrates outstanding effectiveness across various scenarios and benchmarks with just 5,000 samples. Furthermore, we also develop two effective strategies to improve the performance of PRMs: ORM initialization and up-sampling for negative data. We validate our approach in three specific scenarios: verification for test time scaling, direct evaluation of reasoning step correctness, and reward ranked fine-tuning. Our Athena-PRM consistently achieves superior performance across multiple benchmarks and scenarios. Notably, when using Qwen2.5-VL-7B as the policy model, Athena-PRM enhances performance by 10.2 points on WeMath and 7.1 points on MathVista for test time scaling. Furthermore, Athena-PRM sets the state-of-the-art (SoTA) results in VisualProcessBench and outperforms the previous SoTA by 3.9 F1-score, showcasing its robust capability to accurately assess the correctness of the reasoning step. Additionally, utilizing Athena-PRM as the reward model, we develop Athena-7B with reward ranked fine-tuning and outperforms baseline with a significant margin on five benchmarks.
♻ ☆ Beyond I-Con: Exploring New Dimension of Distance Measures in Representation Learning
Jasmine Shone, Zhening Li, Shaden Alshammari, Mark Hamilton, William Freeman
The Information Contrastive (I-Con) framework revealed that over 23 representation learning methods implicitly minimize KL divergence between data and learned distributions that encode similarities between data points. However, a KL-based loss may be misaligned with the true objective, and properties of KL divergence such as asymmetry and unboundedness may create optimization challenges. We present Beyond I-Con, a framework that enables systematic discovery of novel loss functions by exploring alternative statistical divergences. Key findings: (1) on unsupervised clustering of DINO-ViT embeddings, we achieve state-of-the-art results by modifying the PMI algorithm to use total variation (TV) distance; (2) supervised contrastive learning with Euclidean distance as the feature space metric is improved by replacing the standard loss function with Jenson-Shannon divergence (JSD); (3) on dimensionality reduction, we achieve superior qualitative results and better performance on downstream tasks than SNE by replacing KL with a bounded $f$-divergence. Our results highlight the importance of considering divergence choices in representation learning optimization.
♻ ☆ Experience Replay with Random Reshuffling
Yasuhiro Fujita
Experience replay is a key component in reinforcement learning for stabilizing learning and improving sample efficiency. Its typical implementation samples transitions with replacement from a replay buffer. In contrast, in supervised learning with a fixed dataset, it is a common practice to shuffle the dataset every epoch and consume data sequentially, which is called random reshuffling (RR). RR enjoys theoretically better convergence properties and has been shown to outperform with-replacement sampling empirically. To leverage the benefits of RR in reinforcement learning, we propose sampling methods that extend RR to experience replay, both in uniform and prioritized settings, and analyze their properties via theoretical analysis and simulations. We evaluate our sampling methods on Atari benchmarks, demonstrating their effectiveness in deep reinforcement learning. Code is available at https://github.com/pfnet-research/errr.
♻ ☆ SO-Bench: A Structural Output Evaluation of Multimodal LLMs
Di Feng, Kaixin Ma, Feng Nan, Haofeng Chen, Bohan Zhai, David Griffiths, Mingfei Gao, Zhe Gan, Eshan Verma, Yinfei Yang, Zhifeng Chen, Afshin Dehghan
Multimodal large language models (MLLMs) are increasingly deployed in real-world, agentic settings where outputs must not only be correct, but also conform to predefined data schemas. Despite recent progress in structured generation in textual domain, there is still no benchmark that systematically evaluates schema-grounded information extraction and reasoning over visual inputs. In this work, we conduct a comprehensive study of visual structural output capabilities for MLLMs with our carefully designed SO-Bench benchmark. Covering four visual domains, including UI screens, natural images, documents, and charts, SO-Bench is built from over 6.5K diverse JSON schemas and 1.8K curated image-schema pairs with human-verified quality. Benchmarking experiments on open-sourced and frontier proprietary models reveal persistent gaps in predicting accurate, schema compliant outputs, highlighting the need for better multimodal structured reasoning. Beyond benchmarking, we further conduct training experiments to largely improve the model's structured output capability. We plan to make the benchmark available to the community.
comment : v2 preprint. Fixed some typos, add a discussion about limitation, provide pseudo-codes for eval
♻ ☆ The Peril of Preference: Why GRPO fails on Ordinal Rewards
Anisha Garg, Ganesh Venkatesh
Group-relative Policy Optimization's (GRPO) simplicity makes it highly desirable for adapting LLMs to become experts at specific tasks. But this simplicity also makes it ill-specified as we seek to enhance RL training with richer, non-binary feedback. When using ordinal rewards to give partial credit, GRPO's simplicity starts to hurt, as its group-average baseline often assigns a positive advantage to failed trajectories and reinforces incorrect behavior. We introduce Correctness Relative Policy Optimization (CoRPO), a new formulation that solves this flaw. CoRPO uses an adaptive baseline that enforces a minimum quality threshold, ensuring failed solutions are never positively reinforced. Once the policy consistently meets this threshold, the baseline automatically transitions to a relative preference mode, pushing the model to find optimal solutions rather than just "acceptable" ones. We empirically validate CoRPO on a code verification task, where it demonstrates more stable convergence and better out-of-domain generalization. This work represents a critical step in our broader research program to enable LLMs to learn genuinely new capabilities through reinforcement learning. We achieve this by enabling LLMs to learn from rich, multi-dimensional feedback - progressing from binary to ordinal rewards in this work, and onward to denser, per-step supervision.
♻ ☆ LORE: A Large Generative Model for Search Relevance
Chenji Lu, Zhuo Chen, Hui Zhao, Zhiyuan Zeng, Gang Zhao, Junjie Ren, Ruicong Xu, Haoran Li, Songyan Liu, Pengjie Wang, Jian Xu, Bo Zheng
Achievement. We introduce LORE, a systematic framework for Large Generative Model-based relevance in e-commerce search. Deployed and iterated over three years, LORE achieves a cumulative +27\% improvement in online GoodRate metrics. This report shares the valuable experience gained throughout its development lifecycle, spanning data, features, training, evaluation, and deployment. Insight. While existing works apply Chain-of-Thought (CoT) to enhance relevance, they often hit a performance ceiling. We argue this stems from treating relevance as a monolithic task, lacking principled deconstruction. Our key insight is that relevance comprises distinct capabilities: knowledge and reasoning, multi-modal matching, and rule adherence. We contend that a qualitative-driven decomposition is essential for breaking through current performance bottlenecks. Contributions. LORE provides a complete blueprint for the LLM relevance lifecycle. Key contributions include: (1) A two-stage training paradigm combining progressive CoT synthesis via SFT with human preference alignment via RL. (2) A comprehensive benchmark, RAIR, designed to evaluate these core capabilities. (3) A query frequency-stratified deployment strategy that efficiently transfers offline LLM capabilities to the online system. LORE serves as both a practical solution and a methodological reference for other vertical domains.
♻ ☆ A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models
X. Y. Han, Yuan Zhong
In large-scale AI training, Sparse Mixture-of-Experts (s-MoE) layers enable scaling by activating only a small subset of experts per token. An operational challenge in this design is load balancing: routing tokens to minimize the number of idle experts, which is important for the efficient utilization of (costly) GPUs. We provide a theoretical framework for analyzing the Auxiliary-Loss-Free Load Balancing (ALF-LB) procedure -- proposed by DeepSeek's Wang et al. (2024) -- by casting it as a one-step-per-iteration primal-dual method for an assignment problem. First, in a stylized deterministic setting, our framework yields several insightful structural properties: (i) a monotonic improvement of a Lagrangian objective, (ii) a preference rule that moves tokens from overloaded to underloaded experts, and (iii) an approximate-balancing guarantee. Then, we incorporate the stochastic and dynamic nature of AI training using a generalized online optimization formulation. In the online setting, we derive a strong convexity property of the objective that leads to a logarithmic expected regret bound under certain step-size choices. Additionally, we present real experiments on 1B-parameter DeepSeekMoE models to complement our theoretical findings. Together, these results build a principled framework for analyzing the Auxiliary-Loss-Free Load Balancing of s-MoE in AI models.
♻ ☆ Visuospatial navigation from the bottom-up: without vestibular integration, distance prediction, or maps
Patrick Govoni, Pawel Romanczuk
Navigation is believed to be controlled by at least two partially dissociable systems in the brain. The cognitive map informs an organism of its location and bearing, updated by integrating vestibular self-motion or predicting distances to landmarks. Route-based navigation, on the other hand, directly evaluate sequential movement decisions from immediate percepts. Here we demonstrate the sufficiency of visual route-based decision-making in a classic open field navigation task often assumed to require a cognitive map. Three distinct strategies emerge to robustly navigate to a hidden goal, each conferring contextual tradeoffs analyzed at both neural and behavioral scales, as well as qualitatively aligning with behavior observed across the biological spectrum. We propose reframing navigation from the bottom-up, through an egocentric episodic perspective without assuming online access to computationally expensive top-down representations, to better explain behavior under energetic or attentional constraints.
♻ ☆ "I Can See Forever!": Evaluating Real-time VideoLLMs for Assisting Individuals with Visual Impairments
Ziyi Zhang, Zhen Sun, Zongmin Zhang, Zifan Peng, Yuemeng Zhao, Zichun Wang, Zeren Luo, Ruiting Zuo, Xinlei He
The visually impaired population faces significant challenges in daily activities. While prior works employ vision language models for assistance, most focus on static content and cannot address real-time perception needs in complex environments. Recent VideoLLMs enable real-time vision and speech interaction, offering promising potential for assistive tasks. In this work, we conduct the first study evaluating their effectiveness in supporting daily life for visually impaired individuals. We first conducted a user survey with visually impaired participants to design the benchmark VisAssistDaily for daily life evaluation. Using VisAssistDaily, we evaluate popular VideoLLMs and find GPT-4o achieves the highest task success rate. We further conduct a user study to reveal concerns about hazard perception. To address this, we propose SafeVid, an environment-awareness dataset, and fine-tune VITA-1.5, improving risk recognition accuracy from 25.00% to 76.00%.We hope this work provides valuable insights and inspiration for future research in this field.
comment : 17 pages
♻ ☆ Large language models can learn and generalize steganographic chain-of-thought under process super vision NeurIPS 2025
Joey Skaf, Luis Ibanez-Lissen, Robert McCarthy, Connor Watts, Vasil Georgiv, Hannes Whittingham, Lorena Gonzalez-Manzano, David Lindner, Cameron Tice, Edward James Young, Puria Radmard
Chain-of-thought (CoT) reasoning not only enhances large language model performance but also provides critical insights into decision-making processes, marking it as a useful tool for monitoring model intent and planning. However, recent works have shown that banning the mention of a specific example of reward hacking causes obfuscation of the undesired reasoning traces but the persistence of the undesired behavior, threatening the reliability of CoT monitoring. We provide an extension to these results with regard to the ability of models to learn a specific type of obfuscated reasoning: steganography. First, we show that penalizing the use of specific strings within load-bearing reasoning traces causes models to substitute alternative strings. Crucially, this does not alter the underlying method by which the model performs the task, demonstrating that the model can learn to steganographically encode its reasoning.We further demonstrate that models can generalize an encoding scheme. When the penalized strings belong to an overarching class, the model learns not only to substitute strings seen in training, but also develops a general encoding scheme for all members of the class which it can apply to held-out testing strings.
comment : 10 pages main text, 3 figures main text, 17 pages supplementary material, 1 figure supplementary material, accepted at NeurIPS 2025
♻ ☆ The Autonomy-Alignment Problem in Open-Ended Learning Robots: Formalising the Purpose Framework
Gianluca Baldassarre, Richard J. Duro, Emilio Cartoni, Mehdi Khamassi, Alejandro Romero, Vieri Giuliano Santucci
The rapid advancement of artificial intelligence is enabling the development of increasingly autonomous robots capable of operating beyond engineered factory settings and into the unstructured environments of human life. This shift raises a critical autonomy-alignment problem: how to ensure that a robot's autonomous learning focuses on acquiring knowledge and behaviours that serve human practical objectives while remaining aligned with broader human values (e.g., safety and ethics). This problem remains largely underexplored and lacks a unifying conceptual and formal framework. Here, we address one of its most challenging instances of the problem: open-ended learning (OEL) robots, which autonomously acquire new knowledge and skills through interaction with the environment, guided by intrinsic motivations and self-generated goals. We propose a computational framework, introduced qualitatively and then formalised, to guide the design of OEL architectures that balance autonomy with human control. At its core is the novel concept of purpose, which specifies what humans (designers or users) want the robot to learn, do, or avoid, independently of specific task domains. The framework decomposes the autonomy-alignment problem into four tractable sub-problems: the alignment of robot purposes (hardwired or learnt) with human purposes; the arbitration between multiple purposes; the grounding of abstract purposes into domain-specific goals; and the acquisition of competence to achieve those goals. The framework supports formal definitions of alignment across multiple cases and proofs of necessary and sufficient conditions under which alignment holds. Illustrative hypothetical scenarios showcase the applicability of the framework for guiding the development of purpose-aligned autonomous robots.
comment : 33 pages, 5 figures
♻ ☆ SoftStep: Learning Sparse Similarity Powers Deep Neighbor-Based Regression
Aviad Susman, Baihan Lin, Mayte Suárez-Fariñas, Joseph T Colonel
Neighbor-based methods are a natural alternative to linear prediction for tabular data when relationships between inputs and targets exhibit complexity such as nonlinearity, periodicity, or heteroscedasticity. Yet in deep learning on unstructured data, nonparametric neighbor-based approaches are rarely implemented in lieu of simple linear heads. This is primarily due to the ability of systems equipped with linear regression heads to co-learn internal representations along with the linear head's parameters. To unlock the full potential of neighbor-based methods in neural networks we introduce SoftStep, a parametric module that learns sparse instance-wise similarity measures directly from data. When integrated with existing neighbor-based methods, SoftStep enables regression models that consistently outperform linear heads across diverse architectures, domains, and training scenarios. We focus on regression tasks, where we show theoretically that neighbor-based prediction with a mean squared error objective constitutes a metric learning algorithm that induces well-structured embedding spaces. We then demonstrate analytically and empirically that this representational structure translates into superior performance when combined with the sparse, instance-wise similarity measures introduced by SoftStep. Beyond regression, SoftStep is a general method for learning instance-wise similarity in deep neural networks, with broad applicability to attention mechanisms, metric learning, representational alignment, and related paradigms.
♻ ☆ OPTIC-ER: A Reinforcement Learning Framework for Real-Time Emergency Response and Equitable Resource Allocation in Underserved African Communities
Mary Tonwe
Public service systems in many African regions suffer from delayed emergency response and spatial inequity, causing avoidable suffering. This paper introduces OPTIC-ER, a reinforcement learning (RL) framework for real-time, adaptive, and equitable emergency response. OPTIC-ER uses an attention-guided actor-critic architecture to manage the complexity of dispatch environments. Its key innovations are a Context-Rich State Vector, encoding action sub-optimality, and a Precision Reward Function, which penalizes inefficiency. Training occurs in a high-fidelity simulation using real data from Rivers State, Nigeria, accelerated by a precomputed Travel Time Atlas. The system is built on the TALS framework (Thin computing, Adaptability, Low-cost, Scalability) for deployment in low-resource settings. In evaluations on 500 unseen incidents, OPTIC-ER achieved a 100.00% optimal action selection rate, confirming its robustness and generalization. Beyond dispatch, the system generates Infrastructure Deficiency Maps and Equity Monitoring Dashboards to guide proactive governance and data-informed development. This work presents a validated blueprint for AI-augmented public services, showing how context-aware RL can bridge the gap between algorithmic decision-making and measurable human impact.
comment : Source code and data available at: https://github.com/marytonwe/OPTIC-ER.git
♻ ☆ Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints
Yongnan Jin, Xurui Li, Feng Cao, Liucun Gao, Juanjuan Yao
The integration of large language models (LLMs) into medical practice offers transformative potential, yet their real-world clinical applicability remains constrained by critical alignment issues: (1) a misalignment between static evaluation benchmarks and the dynamic cognitive demands of clinical practice, (2) challenges in adapting to continuously evolving, multi-source medical standards, and (3) the limited capacity of conventional reward models to reflect nuanced, multi-dimensional medical quality criteria. To overcome these limitations, we introduce MR-RML (Multidimensional Rubric-oriented Reward Model Learning) with GPRC (Geometric Projection Reference Constraints)-a novel alignment framework that structured medical standards into a multi-perspective matrix to guide both data generation and model optimization. Our approach introduces three key innovations: (1) a medical standard system that embeds domain-specific guidelines throughout the training pipeline; (2) an independent multi-dimensional reward model that decomposes evaluation criteria, transitioning from rule-based or LLM-based scoring to internalized reward modeling for better evaluation performance; and (3) geometric projection reference constraints that translate clinical cognitive logic into mathematical regularization, aligning scoring gradients with clinical reasoning and facilitating training with synthetically generated data. Extensive evaluations on the authoritative medical benchmark Healthbench demonstrate that our method significantly boosts the performance of the base Qwen-32B model, with improvements of 45% on the full subset and 85% on the hard subset. It achieves state-of-the-art results among open-source LLMs, scoring 62.7 (full) and 44.7 (hard), while also surpassing the majority of closed-source models.
♻ ☆ DAVE: Diagnostic benchmark for Audio Visual Evaluation
Gorjan Radevski, Teodora Popordanoska, Matthew B. Blaschko, Tinne Tuytelaars
Audio-visual understanding is a rapidly evolving field that seeks to integrate and interpret information from both auditory and visual modalities. Despite recent advances in multi-modal learning, existing benchmarks often suffer from strong visual bias -- when answers can be inferred from visual data alone -- and provide only aggregate scores that conflate multiple sources of error. This makes it difficult to determine whether models struggle with visual understanding, audio interpretation, or audio-visual alignment. In this work, we introduce DAVE: Diagnostic Audio Visual Evaluation, a novel benchmark dataset designed to systematically evaluate audio-visual models across controlled settings. DAVE alleviates existing limitations by (i) ensuring both modalities are necessary to answer correctly and (ii) decoupling evaluation into atomic subcategories. Our detailed analysis of state-of-the-art models reveals specific failure modes and provides targeted insights for improvement. By offering this standardized diagnostic framework, we aim to facilitate more robust development of audio-visual models. Dataset: https://huggingface.co/datasets/gorjanradevski/dave Code: https://github.com/gorjanradevski/dave
comment : First two authors contributed equally
♻ ☆ AI summaries in online search influence users' attitudes
Yiwei Xu, Saloni Dash, Sungha Kang, Wang Liao, Emma S. Spiro
This study examined how AI-generated summaries, which have become visually prominent in online search results, affect how users think about different issues. In a preregistered randomized controlled experiment, participants (N = 2,004) viewed mock search result pages varying in the presence (vs. absence), placement (top vs. middle), and stance (benefit-framed vs. harm-framed) of AI-generated summaries across four publicly debated topics. Compared to a no-summary control group, participants exposed to AI-generated summaries reported issue attitudes, behavioral intentions, and policy support that aligned more closely with the AI summary stance. The summaries placed at the top of the page produced stronger shifts in users' issue attitudes (but not behavioral intentions or policy support) than those placed at the middle of the page. We also observed moderating effects from issue familiarity and general trust toward AI. In addition, users perceived the AI summaries more useful when it emphasized health harms versus benefits. These findings suggest that AI-generated search summaries can significantly shape public perceptions, raising important implications for the design and regulation of AI-integrated information ecosystems.
♻ ☆ Epidemiology of Large Language Models: A Benchmark for Observational Distribution Knowledge
Drago Plecko, Patrik Okanovic, Shreyas Havaldar, Torsten Hoefler, Elias Bareinboim
Artificial intelligence (AI) systems hold great promise for advancing various scientific disciplines, and are increasingly used in real-world applications. Despite their remarkable progress, further capabilities are expected in order to achieve more general types of intelligence. A critical distinction in this context is between factual knowledge, which can be evaluated against true or false answers (e.g., "what is the capital of England?"), and probabilistic knowledge, reflecting probabilistic properties of the real world (e.g., "what is the sex of a computer science graduate in the US?"). In this paper, our goal is to build a benchmark for understanding the capabilities of LLMs in terms of knowledge of probability distributions describing the real world. Given that LLMs are trained on vast amounts of text, it may be plausible that they internalize aspects of these distributions. Indeed, LLMs are touted as powerful universal approximators of real-world distributions. At the same time, classical results in statistics, known as curse of dimensionality, highlight fundamental challenges in learning distributions in high dimensions, challenging the notion of universal distributional learning. In this work, we develop the first benchmark to directly test this hypothesis, evaluating whether LLMs have access to empirical distributions describing real-world populations across domains such as economics, health, education, and social behavior. Our results demonstrate that LLMs perform poorly overall, and do not seem to internalize real-world statistics naturally. When interpreted in the context of Pearl's Causal Hierarchy (PCH), our benchmark demonstrates that language models do not contain knowledge on observational distributions (Layer 1 of PCH), and thus the Causal Hierarchy Theorem implies that interventional (Layer 2) and counterfactual (Layer 3) knowledge of these models is also limited.
♻ ☆ What-If Analysis of Large Language Models: Explore the Game Wo rl d Using Proactive Thinking
Yuan Sui, Yanming Zhang, Yi Liao, Yu Gu, Guohua Tang, Zhongqian Sun, Wei Yang, Bryan Hooi
Large Language Models (LLMs) are effective at reasoning and information retrieval, but remain unreliable for decision-making in dynamic, partially observable, high-stakes environments such as MOBA games. One key limitation is weak counterfactual reasoning: LLMs struggle to conduct precise what-if analysis over candidate actions and their future consequences. We address this limitation with What-if Analysis LLM (WiA-LLM), a framework that trains an LLM as an explicit language-based world model. Instead of representing the environment in latent vectors, WiA-LLM models how the game state evolves over time with candidate actions using language, and provides textual justifications for these predicted outcomes. This explicit modeling supports (1) interpretability, since the model's predictions and underlying rationales are human-readable, and (2) semantic generalization, as the model can transfer knowledge across situations that share similar game concepts (e.g., roles, objectives, or tactics). WiA-LLM is trained in two stages: supervised fine-tuning on human-like reasoning traces, followed by reinforcement learning with outcome-based rewards that depend on the discrepancy between predicted and ground-truth future states. In the Honor of Kings (HoK) environment, WiA-LLM attains 74.2\% accuracy (27\%$\uparrow$ vs. base model) in forecasting game-state changes. In addition, we find that agents with WiA-LLM exhibit closer strategic behavior to expert players than purely reactive LLM agents, indicating more foresight-aware and expert-aligned decision-making.
♻ ☆ Integrating Skeleton Based Representations for Robust Yoga Pose Classification Using Deep Learning Models
Mohammed Mohiuddin, Syed Mohammod Minhaz Hossain, Sumaiya Khanam, Prionkar Barua, Aparup Barua, MD Tamim Hossain
Yoga is a popular form of exercise worldwide due to its spiritual and physical health benefits, but incorrect postures can lead to injuries. Automated yoga pose classification has therefore gained importance to reduce reliance on expert practitioners. While human pose keypoint extraction models have shown high potential in action recognition, systematic benchmarking for yoga pose recognition remains limited, as prior works often focus solely on raw images or a single pose extraction model. In this study, we introduce a curated dataset, 'Yoga-16', which addresses limitations of existing datasets, and systematically evaluate three deep learning architectures (VGG16, ResNet50, and Xception), using three input modalities (direct images, MediaPipe Pose skeleton images, and YOLOv8 Pose skeleton images). Our experiments demonstrate that skeleton-based representations outperform raw image inputs, with the highest accuracy of 96.09% achieved by VGG16 with MediaPipe Pose skeleton input. Additionally, we provide interpretability analysis using Grad-CAM, offering insights into model decision-making for yoga pose classification with cross-validation analysis.
♻ ☆ Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulations
William Sharpless, Dylan Hirsch, Sander Tonkens, Nikhil Shinde, Sylvia Herbert
Hard constraints in reinforcement learning (RL) often degrade policy performance. Lagrangian methods offer a way to blend objectives with constraints, but require intricate reward engineering and parameter tuning. In this work, we extend recent advances that connect Hamilton-Jacobi (HJ) equations with RL to propose two novel value functions for dual-objective satisfaction. Namely, we address: 1) the Reach-Always-Avoid (RAA) problem -- of achieving distinct reward and penalty thresholds -- and 2) the Reach-Reach (RR) problem -- of achieving thresholds of two distinct rewards. In contrast with temporal logic approaches, which typically involve representing an automaton, we derive explicit, tractable Bellman forms in this context via decomposition. Specifically, we prove that the RAA and RR problems may be rewritten as compositions of previously studied HJ-RL problems. We leverage our analysis to propose a variation of Proximal Policy Optimization (DOHJ-PPO), and demonstrate that it produces distinct behaviors from previous approaches, outcompeting a number of baselines in success, safety and speed across a range of tasks for safe-arrival and multi-target achievement.
♻ ☆ Human Mobility Dataset s Enriched With Context ual and Social Dimensions
Chiara Pugliese, Francesco Lettich, Guido Rocchietti, Chiara Renso, Fabio Pinelli
In this resource paper, we present two publicly available datasets of semantically enriched human trajectories, together with the pipeline to build them. The trajectories are publicly available GPS traces retrieved from OpenStreetMap. Each dataset includes contextual layers such as stops, moves, points of interest (POIs), inferred transportation modes, and weather data. A novel semantic feature is the inclusion of synthetic, realistic social media posts generated by Large Language Models (LLMs), enabling multimodal and semantic mobility analysis. The datasets are available in both tabular and Resource Description Framework (RDF) formats, supporting semantic reasoning and FAIR data practices. They cover two structurally distinct, large cities: Paris and New York. Our open source reproducible pipeline allows for dataset customization, while the datasets support research tasks such as behavior modeling, mobility prediction, knowledge graph construction, and LLM-based applications. To our knowledge, our resource is the first to combine real-world movement, structured semantic enrichment, LLM-generated text, and semantic web compatibility in a reusable framework.
comment : 5 pages, 3 figures, 1 table
♻ ☆ Omni-AutoThink: Adaptive Multimodal Reasoning via Reinforcement Learning
Dongchao Yang, Songxiang Liu, Disong Wang, Yuanyuan Wang, Guanglu Wan, Helen Meng
Recent advances in Omni models have enabled unified multimodal perception and generation. However, most existing systems still exhibit rigid reasoning behaviors, either overthinking simple problems or failing to reason when necessary. To address this limitation, we propose Omni-AutoThink, a novel adaptive reasoning framework that dynamically adjusts the model's reasoning depth according to task difficulty. Our framework comprises two stages: (1) an Adaptive Supervised Fine-Tuning (Adaptive SFT) stage, which endows the Omni model with fundamental reasoning capability using large-scale reasoning-augmented data, and (2) an Adaptive Reinforcement Learning (Adaptive GRPO) stage, which optimizes reasoning behaviors based on task complexity and reward feedback. We further construct a comprehensive adaptive reasoning benchmark that spans text-only, text-audio, text-visual, and text-audio-visual modalities, providing both training and evaluation splits for multimodal reasoning assessment. Experimental results demonstrate that our proposed framework significantly improves adaptive reasoning performance compared to previous baselines. All benchmark data and code will be publicly released.
♻ ☆ Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generation
Changsheng Lv, Zijian Fu, Mengshi Qi
In this paper, we propose Robo-SGG, a plug-and-play module for robust scene graph generation (SGG). Unlike standard SGG, the robust scene graph generation aims to perform inference on a diverse range of corrupted images, with the core challenge being the domain shift between the clean and corrupted images. Existing SGG methods suffer from degraded performance due to shifted visual features (e.g., corruption interference or occlusions). To obtain robust visual features, we leverage layout information, representing the global structure of an image, which is robust to domain shift, to enhance the robustness of SGG methods under corruption. Specifically, we employ Instance Normalization (IN) to alleviate the domain-specific variations and recover the robust structural features (i.e., the positional and semantic relationships among objects) by the proposed Layout-Oriented Restitution. Furthermore, under corrupted images, we introduce a Layout-Embedded Encoder (LEE) that adaptively fuses layout and visual features via a gating mechanism, enhancing the robustness of positional and semantic representations for objects and predicates. Note that our proposed Robo-SGG module is designed as a plug-and-play component, which can be easily integrated into any baseline SGG model. Extensive experiments demonstrate that by integrating the state-of-the-art method into our proposed Robo-SGG, we achieve relative improvements of 6.3%, 11.1%, and 8.0% in mR@50 for PredCls, SGCls, and SGDet tasks on the VG-C benchmark, respectively, and achieve new state-of-the-art performance in the corruption scene graph generation benchmark (VG-C and GQA-C). We will release our source code and model.
♻ ☆ Multimodal Adversarial Defense for Vision -Language Models by Leveraging One-To-Many Relation ships WACV 2026
Futa Waseda, Antonio Tejero-de-Pablos, Isao Echizen
Pre-trained vision-language (VL) models are highly vulnerable to adversarial attacks. However, existing defense methods primarily focus on image classification, overlooking two key aspects of VL tasks: multimodal attacks, where both image and text can be perturbed, and the one-to-many relationship of images and texts, where a single image can correspond to multiple textual descriptions and vice versa (1:N and N:1). This work is the first to explore defense strategies against multimodal attacks in VL tasks, whereas prior VL defense methods focus on vision robustness. We propose multimodal adversarial training (MAT), which incorporates adversarial perturbations in both image and text modalities during training, significantly outperforming existing unimodal defenses. Furthermore, we discover that MAT is limited by deterministic one-to-one (1:1) image-text pairs in VL training data. To address this, we conduct a comprehensive study on leveraging one-to-many relationships to enhance robustness, investigating diverse augmentation techniques. Our analysis shows that, for a more effective defense, augmented image-text pairs should be well-aligned, diverse, yet avoid distribution shift -- conditions overlooked by prior research. This work pioneers defense strategies against multimodal attacks, providing insights for building robust VLMs from both optimization and data perspectives. Our code is publicly available at https://github.com/CyberAgentAI/multimodal-adversarial-training.
comment : WACV 2026 Accepted. Code available at https://github.com/CyberAgentAI/multimodal-adversarial-training
♻ ☆ Joint Discriminative-Generative Modeling via Dual Adversarial Training
Xuwang Yin, Claire Zhang, Julie Steele, Nir Shavit, Tony T. Wang
Simultaneously achieving robust classification and high-fidelity generative modeling within a single framework presents a significant challenge. Hybrid approaches, such as Joint Energy-Based Models (JEM), interpret classifiers as EBMs but are often limited by the instability and poor sample quality inherent in Stochastic Gradient Langevin Dynamics (SGLD)-based training. We address these limitations by proposing a novel training framework that integrates adversarial training (AT) principles for both discriminative robustness and stable generative learning. The proposed method introduces three key innovations: (1) the replacement of SGLD-based JEM learning with a stable, AT-based approach that optimizes the energy function by discriminating between real data and Projected Gradient Descent (PGD)-generated contrastive samples using the BCE loss; (2) synergistic adversarial training for the discriminative component that enhances classification robustness while eliminating the need for explicit gradient penalties; and (3) a two-stage training strategy that addresses normalization-related instabilities and enables leveraging pretrained robust classifiers, generalizing effectively across diverse architectures. Experiments on CIFAR-10/100 and ImageNet demonstrate that our approach: (1) is the first EBM-based hybrid to scale to high-resolution datasets with high training stability, simultaneously achieving state-of-the-art discriminative and generative performance on ImageNet 256$\times$256; (2) uniquely combines generative quality with adversarial robustness, enabling critical applications like robust counterfactual explanations; and (3) functions as a competitive standalone generative model, matching the generative quality of autoregressive methods (VAR-d16) and surpassing diffusion models while offering unique versatility.
comment : V2: ImageNet 256x256 with ConvNeXt-Large (FID 3.29), formal theoretical analysis, L_inf training, computational analysis, expanded related work/baselines. Revised presentation. Code: https://github.com/xuwangyin/DAT
♻ ☆ Safe Online Bid Optimization with Return on Investment and Budget Constraints
Matteo Castiglioni, Alessandro Nuara, Giulia Romano, Giorgio Spadaro, Francesco Trovò, Nicola Gatti
In online marketing, the advertisers aim to balance achieving high volumes and high profitability. The companies' business units address this tradeoff by maximizing the volumes while guaranteeing a minimum Return On Investment (ROI) level. Such a task can be naturally modeled as a combinatorial optimization problem subject to ROI and budget constraints that can be solved online. In this picture, the learner's uncertainty over the constraints' parameters plays a crucial role since the algorithms' exploration choices might lead to their violation during the entire learning process. Such violations represent a major obstacle to adopting online techniques in real-world applications. Thus, controlling the algorithms' exploration during learning is paramount to making humans trust online learning tools. This paper studies the nature of both optimization and learning problems. In particular, we show that the learning problem is inapproximable within any factor (unless P = NP) and provide a pseudo-polynomial-time algorithm to solve its discretized version. Subsequently, we prove that no online learning algorithm can violate the (ROI or budget) constraints a sublinear number of times during the learning process while guaranteeing a sublinear regret. We provide the $GCB$ algorithm that guarantees sublinear regret at the cost of a linear number of constraint violations and $GCB_{safe}$ that guarantees w.h.p. a constant upper bound on the number of constraint violations at the cost of a linear regret. Moreover, we designed $GCB_{safe}(ψ,φ)$, which guarantees both sublinear regret and safety w.h.p. at the cost of accepting tolerances $ψ$ and $φ$ in the satisfaction of the ROI and budget constraints, respectively. Finally, we provide experimental results to compare the regret and constraint violations of $GCB$, $GCB_{safe}$, and $GCB_{safe}(ψ,φ)$.
♻ ☆ A lightweight detector for real-time detection of remote sensing images
Qianyi Wang, Guoqiang Ren
Remote sensing imagery is widely used across various fields, yet real-time detection remains challenging due to the prevalence of small objects and the need to balance accuracy with efficiency. To address this, we propose DMG-YOLO, a lightweight real-time detector tailored for small object detection in remote sensing images. Specifically, we design a Dual-branch Feature Extraction (DFE) module in the backbone, which partitions feature maps into two parallel branches: one extracts local features via depthwise separable convolutions, and the other captures global context using a vision transformer with a gating mechanism. Additionally, a Multi-scale Feature Fusion (MFF) module with dilated convolutions enhances multi-scale integration while preserving fine details. In the neck, we introduce the Global and Local Aggregate Feature Pyramid Network (GLAFPN) to further boost small object detection through global-local feature fusion. Extensive experiments on the VisDrone2019 and NWPU VHR-10 datasets show that DMG-YOLO achieves competitive performance in terms of mAP, model size, and other key metrics.
comment : wrong results
♻ ☆ SoK: Decentralized AI (DeAI)
Zhipeng Wang, Rui Sun, Elizabeth Lui, Vatsal Shah, Xihan Xiong, Jiahao Sun, Davide Crapis, William Knottenbelt
Centralization enhances the efficiency of Artificial Intelligence (AI) but also introduces critical challenges, including single points of failure, inherent biases, data privacy risks, and scalability limitations. To address these issues, blockchain-based Decentralized Artificial Intelligence (DeAI) has emerged as a promising paradigm that leverages decentralization and transparency to improve the trustworthiness of AI systems. Despite rapid adoption in industry, the academic community lacks a systematic analysis of DeAI's technical foundations, opportunities, and challenges. This work presents the first Systematization of Knowledge (SoK) on DeAI, offering a formal definition, a taxonomy of existing solutions based on the AI lifecycle, and an in-depth investigation of the roles of blockchain in enabling secure and incentive-compatible collaboration. We further review security risks across the DeAI lifecycle and empirically evaluate representative mitigation techniques. Finally, we highlight open research challenges and future directions for advancing blockchain-based DeAI.
comment : This is a Systematization of Knowledge (SoK) for the rapidly evolving field of Decentralized AI (DeAI). We welcome valuable comments, suggestions, and collaboration to further refine and enhance this work. We hope our contribution will help accelerate the advancement of DeAI
♻ ☆ Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelines
Matthew Lewis, Samuel Thio, Amy Roberts, Catherine Siju, Whoasif Mukit, Rebecca Kuruvilla, Zhangshu Joshua Jiang, Niko Möller-Grell, Aditya Borakati, Richard JB Dobson, Spiros Denaxas
This paper presents the development and evaluation of a Retrieval-Augmented Generation (RAG) system for querying the United Kingdom's National Institute for Health and Care Excellence (NICE) clinical guidelines using Large Language Models (LLMs). The extensive length and volume of these guidelines can impede their utilisation within a time-constrained healthcare system, a challenge this project addresses through the creation of a system capable of providing users with precisely matched information in response to natural language queries. The system's retrieval architecture, composed of a hybrid embedding mechanism, was evaluated against a corpus of 10,195 text chunks derived from three hundred guidelines. It demonstrates high performance, with a Mean Reciprocal Rank (MRR) of 0.814, a Recall of 81% at the first chunk and of 99.1% within the top ten retrieved chunks, when evaluated on 7901 queries. The most significant impact of the RAG system was observed during the generation phase. When evaluated on a manually curated dataset of seventy question-answer pairs, RAG-enhanced models showed substantial gains in performance. Faithfulness, the measure of whether an answer is supported by the source text, was increased by 64.7 percentage points to 99.5% for the RAG-enhanced O4-Mini model and significantly outperformed the medical-focused Meditron3-8B LLM, which scored 43%. Clinical evaluation by seven Subject Matter Experts (SMEs) further validated these findings, with GPT-4.1 achieving 98.7% accuracy while reducing unsafe responses by 67% compared to O4-Mini (from 3.0 to 1.0 per evaluator). This study thus establishes RAG as an effective, reliable, and scalable approach for applying generative AI in healthcare, enabling cost-effective access to medical guidelines.
♻ ☆ Jina-VLM: Small Multilingual Vision Language Model
Andreas Koukounas, Georgios Mastrapas, Florian Hönicke, Sedigheh Eslami, Guillaume Roncari, Scott Martens, Han Xiao
We present Jina-VLM, a 2.4B parameter vision-language model that achieves state-of-the-art multilingual visual question answering among open 2B-scale VLMs. The model couples a SigLIP2 vision encoder with a Qwen3 language backbone through an attention-pooling connector that enables token-efficient processing of arbitrary-resolution images. The model achieves leading results on standard VQA benchmarks and multilingual evaluations while preserving competitive text-only performance. Model weights and code are publicly released at https://huggingface.co/jinaai/jina-vlm .
comment : 18 pages, 1-7 main content, 13-18 appendix for tables and dataset
♻ ☆ SeSE: A Structural Information -Guided Uncertainty Quantification Framework for Hallucination Detection in LLMs
Xingtao Zhao, Hao Peng, Dingli Su, Xianghua Zeng, Chunyang Liu, Jinzhi Liao, Philip S. Yu
Reliable uncertainty quantification (UQ) is essential for deploying large language models (LLMs) in safety-critical scenarios, as it enables them to abstain from responding when uncertain, thereby avoiding ``hallucinating'' falsehoods. However, state-of-the-art UQ methods primarily rely on semantic probability distributions or pairwise distances, overlooking latent semantic structural information that could enable more precise uncertainty estimates. This paper presents Semantic Structural Entropy (SeSE), a principled UQ framework that quantifies the inherent semantic uncertainty of LLMs from a structural information perspective for hallucination detection. SeSE operates in a zero-resource manner and is applicable to both open- and closed-source LLMs, making it an ``off-the-shelf" solution for new models and tasks. Specifically, to effectively model semantic spaces, we first develop an adaptively sparsified directed semantic graph construction algorithm that captures directional semantic dependencies while automatically pruning unnecessary connections that introduce negative interference. We then exploit latent semantic structural information through hierarchical abstraction: SeSE is defined as the structural entropy of the optimal semantic encoding tree, formalizing intrinsic uncertainty within semantic spaces after optimal compression. A higher SeSE value corresponds to greater uncertainty, indicating that LLMs are highly likely to generate hallucinations. In addition, to enhance fine-grained UQ in long-form generation, we extend SeSE to quantify the uncertainty of individual claims by modeling their random semantic interactions, providing theoretically explicable hallucination detection. Extensive experiments across 29 model-dataset combinations show that SeSE significantly outperforms advanced UQ baselines.
comment : 14 pages of main text and 10 pages of appendices;Submit to IEEE TKDE
♻ ☆ On the Transfer of Knowledge in Quantum Algorithms
Esther Villar-Rodriguez, Eneko Osaba, Izaskun Oregi, Sebastián V. Romero, Julián Ferreiro-Vélez
Quantum computing is poised to transform computational paradigms across science and industry. As the field evolves, it can benefit from established classical methodologies, including promising paradigms such as Transfer of Knowledge (ToK). This work serves as a brief, self-contained reference for ToK, unifying its core principles under a single formal framework. We introduce a joint notation that consolidates and extends prior work in Transfer Learning and Transfer Optimization, bridging traditionally separate research lines and enabling a common language for knowledge reuse. Building on this foundation, we classify existing ToK strategies and principles into a structured taxonomy that helps researchers position their methods within a broader conceptual map. We then extend key transfer protocols to quantum computing, introducing two novel use cases--reverse annealing and multitasking Quantum Approximate Optimization Algorithm (QAOA)--alongside a sequential Variational Quantum Eigensolver (VQE) approach that supports and validates prior findings. These examples highlight ToK's potential to improve performance and generalization in quantum algorithms. Finally, we outline challenges and opportunities for integrating ToK into quantum computing, emphasizing its role in reducing resource demands and accelerating problem-solving. This work lays the groundwork for future synergies between classical and quantum computing through a shared, transferable knowledge framework.
comment : 14 pages, 8 figures, 4 tables. Paper submitted for its review in Expert Systems journal
♻ ☆ SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision -Language Models
Ruosen Zhao, Zhikang Zhang, Jialei Xu, Jiahao Chang, Dong Chen, Lingyun Li, Weijian Sun, Zizhuang Wei
Large vision-language models (VLMs) show strong multimodal understanding but still struggle with 3D spatial reasoning, such as distance estimation, size comparison, and cross-view consistency. Existing 3D-aware methods either depend on auxiliary 3D information or enhance RGB-only VLMs with geometry encoders through shallow feature fusion. We propose SpaceMind, a multimodal large language model explicitly designed for spatial reasoning solely from RGB inputs. The model adopts a dual-encoder architecture, integrating VGGT as a spatial understanding encoder and InternViT as a 2D visual encoder. The key idea is to treat the camera representation as an active guiding modality rather than passive metadata. Specifically, SpaceMind introduces a lightweight Camera-Guided Modality Fusion module before the language model to replace shallow fusion. It applies camera-conditioned biasing to spatial tokens, assigns query-independent weights reflecting their geometric importance, and uses the camera embedding to gate the fused representation. Empirically, SpaceMind establishes new state-of-the-art results on VSI-Bench, SQA3D and SPBench, surpassing both open and proprietary systems on VSI-Bench and SPBench by large margins and achieving state-of-the-art performance on SQA3D. These results demonstrate that camera-guided modality fusion is an effective and practical inductive bias for equipping VLMs with genuinely spatially grounded intelligence. We will release code and model checkpoints to support future research.
♻ ☆ Control Illusion: The Failure of Instruction Hierarchies in Large Language Models AAAI-26
Yilin Geng, Haonan Li, Honglin Mu, Xudong Han, Timothy Baldwin, Omri Abend, Eduard Hovy, Lea Frermann
Large language models (LLMs) are increasingly deployed with hierarchical instruction schemes, where certain instructions (e.g., system-level directives) are expected to take precedence over others (e.g., user messages). Yet, we lack a systematic understanding of how effectively these hierarchical control mechanisms work. We introduce a systematic evaluation framework based on constraint prioritization to assess how well LLMs enforce instruction hierarchies. Our experiments across six state-of-the-art LLMs reveal that models struggle with consistent instruction prioritization, even for simple formatting conflicts. We find that the widely-adopted system/user prompt separation fails to establish a reliable instruction hierarchy, and models exhibit strong inherent biases toward certain constraint types regardless of their priority designation. Interestingly, we also find that societal hierarchy framings (e.g., authority, expertise, consensus) show stronger influence on model behavior than system/user roles, suggesting that pretraining-derived social structures function as latent behavioral priors with potentially greater impact than post-training guardrails.
comment : Accepted to AAAI-26 Main Technical Track Proceedings
♻ ☆ Changes in Gaza: DINOv3-Powered Multi-Class Change Detection for Damage Assessment in Conflict Zones
Kai Zheng, Zhenkai Wu, Fupeng Wei, Miaolan Zhou, Kai Lie, Haitao Guo, Lei Ding, Wei Zhang, Hang-Cheng Dong
Accurately and swiftly assessing damage from conflicts is crucial for humanitarian aid and regional stability. In conflict zones, damaged zones often share similar architectural styles, with damage typically covering small areas and exhibiting blurred boundaries. These characteristics lead to limited data, annotation difficulties, and significant recognition challenges, including high intra-class similarity and ambiguous semantic changes. To address these issues, we introduce a pre-trained DINOv3 model and propose a multi-scale cross-attention difference siamese network (MC-DiSNet). The powerful visual representation capability of the DINOv3 backbone enables robust and rich feature extraction from bi-temporal remote sensing images. The multi-scale cross-attention mechanism allows for precise localization of subtle semantic changes, while the difference siamese structure enhances inter-class feature discrimination, enabling fine-grained semantic change detection. Furthermore, a simple yet powerful lightweight decoder is designed to generate clear detection maps while maintaining high efficiency. We also release a new Gaza-change dataset containing high-resolution satellite image pairs from 2023-2024 with pixel-level semantic change annotations. It is worth emphasizing that our annotations only include semantic pixels of changed areas. We evaluated our method on the Gaza-Change and two classical datasets: the SECOND and Landsat-SCD datasets. Experimental results demonstrate that our proposed approach effectively addresses the MCD task, and its outstanding performance paves the way for practical applications in rapid damage assessment across conflict zones.
♻ ☆ Few-shot Class-incremental Fault Diagnosis by Preserving Class-Agnostic Knowledge with Dual-Granularity Representations
Zhendong Yang, Jie Wang, Liansong Zong, Xiaorong Liu, Quan Qian, Shiqian Chen
Few-Shot Class-Incremental Fault Diagnosis (FSC-FD), which aims to continuously learn from new fault classes with only a few samples without forgetting old ones, is critical for real-world industrial systems. However, this challenging task severely amplifies the issues of catastrophic forgetting of old knowledge and overfitting on scarce new data. To address these challenges, this paper proposes a novel framework built upon Dual-Granularity Representations, termed the Dual-Granularity Guidance Network (DGGN). Our DGGN explicitly decouples feature learning into two parallel streams: 1) a fine-grained representation stream, which utilizes a novel Multi-Order Interaction Aggregation module to capture discriminative, class-specific features from the limited new samples. 2) a coarse-grained representation stream, designed to model and preserve general, class-agnostic knowledge shared across all fault types. These two representations are dynamically fused by a multi-semantic cross-attention mechanism, where the stable coarse-grained knowledge guides the learning of fine-grained features, preventing overfitting and alleviating feature conflicts. To further mitigate catastrophic forgetting, we design a Boundary-Aware Exemplar Prioritization strategy. Moreover, a decoupled Balanced Random Forest classifier is employed to counter the decision boundary bias caused by data imbalance. Extensive experiments on the TEP benchmark and a real-world MFF dataset demonstrate that our proposed DGGN achieves superior diagnostic performance and stability compared to state-of-the-art FSC-FD approaches. Our code is publicly available at https://github.com/MentaY/DGGN
comment : This manuscript is currently under review at the Measurement
♻ ☆ Expertise elevates AI usage: experimental evidence comparing laypeople and professional artists
Thomas F. Eisenmann, Andres Karjus, Mar Canet Sola, Levin Brinkmann, Bramantyo Ibrahim Supriyatno, Iyad Rahwan
Generative AI's novel capacities raise questions about the future role of human expertise: does AI level the playing field between professional artists and laypeople, or does expertise enhance AI use? Do the cognitive skills experts make use of in analyzing and drawing visual art also transfer to using these new tools? This pre-registered study conducts experimental comparisons between 50 professional artists and a demographically matched sample of laypeople. Our interdisciplinary team developed two tasks involving image replication and creative image creation, assessing their copying accuracy and divergent thinking. We implemented a bespoke platform for the experiment, powered by a modern text-to-image AI. Results reveal artists produced more accurate copies and more divergent ideas than lay participants, highlighting a skill transfer of professional expertise - even to the confined space of generative AI. We also explored how well an exemplary vision-capable large language model (GPT-4o) would fare: on par in copying and slightly better on average than artists in the creative task, although never above best humans. These findings highlight the importance of integrating artistic skills with AI, suggesting a potential for collaborative synergy that could reshape creative industries and arts education.
comment : Eisenmann and Karjus contributed equally to this work and share first authorship
♻ ☆ TaoSR1: The Thinking Model for E-commerce Relevance Search
Chenhe Dong, Shaowei Yao, Pengkun Jiao, Jianhui Yang, Yiming Jin, Zerui Huang, Xiaojiang Zhou, Dan Ou, Haihong Tang, Bo Zheng
Query-product relevance prediction is a core task in e-commerce search. BERT-based models excel at semantic matching but lack complex reasoning capabilities. While Large Language Models (LLMs) are explored, most still use discriminative fine-tuning or distill to smaller models for deployment. We propose a framework to directly deploy LLMs for this task, addressing key challenges: Chain-of-Thought (CoT) error accumulation, discriminative hallucination, and deployment feasibility. Our framework, TaoSR1, involves three stages: (1) Supervised Fine-Tuning (SFT) with CoT to instill reasoning; (2) Offline sampling with a pass@N strategy and Direct Preference Optimization (DPO) to improve generation quality; and (3) Difficulty-based dynamic sampling with Group Relative Policy Optimization (GRPO) to mitigate discriminative hallucination. Additionally, post-CoT processing and a cumulative probability-based partitioning method enable efficient online deployment. TaoSR1 significantly outperforms baselines on offline datasets and achieves substantial gains in online side-by-side human evaluations, introducing a novel paradigm for applying CoT reasoning to relevance classification.
♻ ☆ An Indoor Radio Mapping Dataset Combining 3D Point Clouds and RSSI
Ljupcho Milosheski, Kuon Akiyama, Blaž Bertalanič, Jernej Hribar, Ryoichi Shinkuma
The growing number of smart devices supporting bandwidth-intensive and latency-sensitive applications, such as real-time video analytics, smart sensing, Extended Reality (XR), etc., necessitates reliable wireless connectivity in indoor environments. In such environments, accurate design of Radio Environment Maps (REMs) enables adaptive wireless network planning and optimization of Access Point (AP) placement. However, generating realistic REMs remains difficult due to the variability of indoor environments and the limitations of existing modeling approaches, which often rely on simplified layouts or fully synthetic data. These challenges are further amplified by the adoption of next-generation Wi-Fi standards, which operate at higher frequencies and suffer from limited range and wall penetration. To support the efforts in addressing these challenges, we collected a dataset that combines high-resolution 3D LiDAR scans with Wi-Fi RSSI measurements collected across 20 setups in a multi-room indoor environment. The dataset includes two measurement scenarios, the first without human presence in the environment, and the second with human presence, enabling the development and validation of REM estimation models that incorporate physical geometry and environmental dynamics. The described dataset supports research in data-driven wireless modeling and the development of high-capacity indoor communication networks.
comment : 12 pages, 7 figures, 3 tables, under review to Nature Scientific Data
♻ ☆ EmbedGenius: Towards Automated Software Development for Generic Embedded IoT Systems
Huanqi Yang, Mingzhe Li, Mingda Han, Zhenjiang Li, Weitao Xu
Embedded IoT system development is crucial for enabling seamless connectivity and functionality across a wide range of applications. However, such a complex process requires cross-domain knowledge of hardware and software and hence often necessitates direct developer involvement, making it labor-intensive, time-consuming, and error-prone. To address this challenge, this paper introduces EmbedGenius, the first fully automated software development platform for general-purpose embedded IoT systems. The key idea is to leverage the reasoning ability of Large Language Models (LLMs) and embedded system expertise to automate the hardware-in-the-loop development process. The main methods include a component-aware library resolution method for addressing hardware dependencies, a library knowledge generation method that injects utility domain knowledge into LLMs, and an auto-programming method that ensures successful deployment. We evaluate EmbedGenius's performance across 71 modules and four mainstream embedded development platforms with over 350 IoT tasks. Experimental results show that EmbedGenius can generate codes with an accuracy of 95.7% and complete tasks with a success rate of 86.5%, surpassing human-in-the-loop baselines by 15.6%--37.7% and 25.5%--53.4%, respectively. We also show EmbedGenius's potential through case studies in environmental monitoring and remote control systems development.
♻ ☆ Orders in Chaos: Enhancing Large-Scale MoE LLM Serving with Data Movement Forecasting
Zhongkai Yu, Yue Guan, Zihao Yu, Chenyang Zhou, Zhengding Hu, Shuyi Pei, Yangwook Kang, Yufei Ding, Po-An Tsai
Large-scale Mixture of Experts (MoE) Large Language Models (LLMs) have recently become the frontier open weight models, achieving remarkable model capability similar to proprietary ones. But their random expert selection mechanism introduces significant data movement overhead that becomes the dominant bottleneck in multi-unit LLM serving systems. To understand the patterns underlying this data movement, we conduct comprehensive data-movement-centric profiling across four state-of-the-art large-scale MoE models released in 2025 (200B-1000B) using over 24,000 requests spanning diverse workloads. We perform systematic analysis from both temporal and spatial perspectives and distill six key insights to guide the design of diverse future serving systems. With our insights, we then demonstrate how to improve wafer-scale GPUs as a case study, and show that minor architectural modifications leveraging the insights achieve substantial performance gains, delivering 5.3x and 3.1x average speedups on DeepSeek V3 and Qwen3, respectively. Our work presents the first comprehensive data-centric analysis of large-scale MoE models and a concrete design study using the learned lessons, with profiling traces and simulation framework already open-sourced with $>$1k downloads. Our traces and results are publicly available at https://huggingface.co/datasets/core12345/MoE_expert_selection_trace
♻ ☆ Data Mixing Can Induce Phase Transitions in Knowledge Acquisition NeurIPS'25
Xinran Gu, Kaifeng Lyu, Jiazheng Li, Jingzhao Zhang
Large Language Models (LLMs) are typically trained on data mixtures: most data come from web scrapes, while a small portion is curated from high-quality sources with dense domain-specific knowledge. In this paper, we show that when training LLMs on such data mixtures, knowledge acquisition from knowledge-dense datasets, unlike training exclusively on knowledge-dense data (arXiv:2404.05405), does not always follow a smooth scaling law but can exhibit phase transitions with respect to the mixing ratio and model size. Through controlled experiments on a synthetic biography dataset mixed with web-scraped data, we demonstrate that: (1) as we increase the model size to a critical value, the model suddenly transitions from memorizing very few to most of the biographies; (2) below a critical mixing ratio, the model memorizes almost nothing even with extensive training, but beyond this threshold, it rapidly memorizes more biographies. We attribute these phase transitions to a capacity allocation phenomenon: a model with bounded capacity must act like a knapsack problem solver to minimize the overall test loss, and the optimal allocation across datasets can change discontinuously as the model size or mixing ratio varies. We formalize this intuition in an information-theoretic framework and reveal that these phase transitions are predictable, with the critical mixing ratio following a power-law relationship with the model size. Our findings highlight a concrete case where a good mixing recipe for large models may not be optimal for small models, and vice versa.
comment : NeurIPS'25 Spotlight
♻ ☆ A Fast Kernel-based Conditional Independence test with Application to Causal Discovery
Oliver Schacht, Biwei Huang
Kernel-based conditional independence (KCI) testing is a powerful nonparametric method commonly employed in causal discovery tasks. Despite its flexibility and statistical reliability, cubic computational complexity limits its application to large datasets. To address this computational bottleneck, we propose \textit{FastKCI}, a scalable and parallelizable kernel-based conditional independence test that utilizes a mixture-of-experts approach inspired by embarrassingly parallel inference techniques for Gaussian processes. By partitioning the dataset based on a Gaussian mixture model over the conditioning variables, FastKCI conducts local KCI tests in parallel, aggregating the results using an importance-weighted sampling scheme. Experiments on synthetic datasets and benchmarks on real-world production data validate that FastKCI maintains the statistical power of the original KCI test while achieving substantial computational speedups. FastKCI thus represents a practical and efficient solution for conditional independence testing in causal inference on large-scale data.
comment : 11 pages, 5 figures
♻ ☆ N2N: A Parallel Framework for Large-Scale MILP under Distributed Memory
Longfei Wang, Junyan Liu, Fan Zhang, Jiangwen Wei, Yuanhua Tang, Jie Sun, Xiaodong Luo
Parallelization has emerged as a promising approach for accelerating MILP solving. However, the complexity of the branch-and-bound (B&B) framework and the numerous effective algorithm components in MILP solvers make it difficult to parallelize. In this study, a scalable parallel framework, N2N (a node-to-node framework that maps the B&B nodes to distributed computing nodes), was proposed to solve large-scale problems in a distributed memory computing environment. Both deterministic and nondeterministic modes are supported, and the framework is designed to be easily integrated with existing solvers. Regarding the deterministic mode, a novel sliding-window-based algorithm was designed and implemented to ensure that tasks are generated and solved in a deterministic order. Moreover, several advanced techniques, such as the utilization of CP search and general primal heuristics, have been developed to fully utilize distributed computing resources and capabilities of base solvers. Adaptive solving and data communication optimization were also investigated. A popular open-source MILP solver, SCIP, was integrated into N2N as the base solver, yielding N2N-SCIP. Extensive computational experiments were conducted to evaluate the performance of N2N-SCIP compared to ParaSCIP, which is a state-of-the-art distributed parallel MILP solver under the UG framework. The nondeterministic N2N-SCIP achieves speedups of 22.52 and 12.71 with 1,000 MPI processes on the Kunpeng and x86 computing clusters, which is 1.98 and 2.08 times faster than ParaSCIP, respectively. In the deterministic mode, N2N-SCIP also shows significant performance improvements over ParaSCIP across different process numbers and computing clusters. To validate the generality of N2N, HiGHS, another open-source solver, was integrated into N2N. The related results are analyzed, and the requirements of N2N on base solvers are also concluded.
comment : We cannot publish the paper at this time because some internal processes have not yet been completed
♻ ☆ Intelligence Foundation Model: A New Perspective to Approach Artificial General Intelligence
Borui Cai, Yao Zhao
We propose a new perspective for approaching artificial general intelligence (AGI) through an intelligence foundation model (IFM). Unlike existing foundation models (FMs), which specialize in pattern learning within specific domains such as language, vision, or time series, IFM aims to acquire the underlying mechanisms of intelligence by learning directly from diverse intelligent behaviors. Vision, language, and other cognitive abilities are manifestations of intelligent behavior; learning from this broad range of behaviors enables the system to internalize the general principles of intelligence. Based on the fact that intelligent behaviors emerge from the collective dynamics of biological neural systems, IFM consists of two core components: a novel network architecture, termed the state neural network, which captures neuron-like dynamic processes, and a new learning objective, neuron output prediction, which trains the system to predict neuronal outputs from collective dynamics. The state neural network emulates the temporal dynamics of biological neurons, allowing the system to store, integrate, and process information over time, while the neuron output prediction objective provides a unified computational principle for learning these structural dynamics from intelligent behaviors. Together, these innovations establish a biologically grounded and computationally scalable foundation for building systems capable of generalization, reasoning, and adaptive learning across domains, representing a step toward truly AGI.
♻ ☆ Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs
Alberto Cattaneo, Carlo Luschi, Daniel Justus
Retrieval of information from graph-structured knowledge bases represents a promising direction for improving the factuality of LLMs. While various solutions have been proposed, a comparison of methods is difficult due to the lack of challenging QA datasets with ground-truth targets for graph retrieval. We present SynthKGQA, an LLM-powered framework for generating high-quality Knowledge Graph Question Answering datasets from any Knowledge Graph, providing the full set of ground-truth facts in the KG to reason over questions. We show how, in addition to enabling more informative benchmarking of KG retrievers, the data produced with SynthKGQA also allows us to train better models.We apply SynthKGQA to Wikidata to generate GTSQA, a new dataset designed to test zero-shot generalization abilities of KG retrievers with respect to unseen graph structures and relation types, and benchmark popular solutions for KG-augmented LLMs on it.
♻ ☆ An Investigation of Robustness of LLMs in Mathematical Reasoning : Benchmark ing with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
Yuren Hao, Xiang Wan, ChengXiang Zhai
In this paper, we introduce a systematic framework beyond conventional method to assess LLMs' mathematical-reasoning robustness by stress-testing them on advanced math problems that are mathematically equivalent but with linguistic and parametric variation. These transformations allow us to measure the sensitivity of LLMs to non-mathematical perturbations, thereby enabling a more accurate evaluation of their mathematical reasoning capabilities. Using this new evaluation methodology, we created PutnamGAP, a new benchmark dataset with multiple mathematically-equivalent variations of competition-level math problems. With the new dataset, we evaluate multiple families of representative LLMs and examine their robustness. Across 18 commercial and open-source models we observe sharp performance degradation on the variants. OpenAI's flagship reasoning model, O3, scores 51.5% on the originals but drops by 4.7 percentage points on surface-renaming variants, and by 12.9 percentage points on parametric variants, while smaller models fare far worse. Overall, the results show that the proposed new evaluation methodology is effective for deepening our understanding of the robustness of LLMs and generating new insights for further improving their mathematical reasoning capabilities.
comment : 34 pages, 9 figures
♻ ☆ VEDA: 3D Molecular Generation via Variance-Exploding Diffusion with Annealing
Peining Zhang, Jinbo Bi, Minghu Song
Diffusion models show promise for 3D molecular generation, but face a fundamental trade-off between sampling efficiency and conformational accuracy. While flow-based models are fast, they often produce geometrically inaccurate structures, as they have difficulty capturing the multimodal distributions of molecular conformations. In contrast, denoising diffusion models are more accurate but suffer from slow sampling, a limitation attributed to sub-optimal integration between diffusion dynamics and SE(3)-equivariant architectures. To address this, we propose VEDA, a unified SE(3)-equivariant framework that combines variance-exploding diffusion with annealing to efficiently generate conformationally accurate 3D molecular structures. Specifically, our key technical contributions include: (1) a VE schedule that enables noise injection functionally analogous to simulated annealing, improving 3D accuracy and reducing relaxation energy; (2) a novel preconditioning scheme that reconciles the coordinate-predicting nature of SE(3)-equivariant networks with a residual-based diffusion objective, and (3) a new arcsin-based scheduler that concentrates sampling in critical intervals of the logarithmic signal-to-noise ratio. On the QM9 and GEOM-DRUGS datasets, VEDA matches the sampling efficiency of flow-based models, achieving state-of-the-art valency stability and validity with only 100 sampling steps. More importantly, VEDA's generated structures are remarkably stable, as measured by their relaxation energy during GFN2-xTB optimization. The median energy change is only 1.72 kcal/mol, significantly lower than the 32.3 kcal/mol from its architectural baseline, SemlaFlow. Our framework demonstrates that principled integration of VE diffusion with SE(3)-equivariant architectures can achieve both high chemical accuracy and computational efficiency.
♻ ☆ A general language model for peptide function identification
Jixiu Zhai, Zikun Wang, Chupei Tang, Haitian Zhong, Ziyang Xu, Yuhuan Liu, Shengrui Xu, Jingwan Wang, Dan Huang, Tianchi Lu
Accurate identification of bioactive peptides (BPs) and protein post-translational modifications (PTMs) is essential for understanding protein function and advancing therapeutic discovery. However, most computational methods remain limited in their generalizability across diverse peptide functions. Here, we present PDeepPP, a unified deep learning framework that integrates pretrained protein language models with a hybrid transformer-CNN architecture, enabling robust identification across diverse peptide classes and PTM sites. We curated comprehensive benchmark datasets and implemented strategies to address data imbalance, allowing PDeepPP to systematically extract both global and local sequence features. Through extensive analyses including dimensionality reduction and comparison studies, PDeepPP demonstrates strong, interpretable peptide representations and achieves state-of-the-art performance in 25 of the 33 biological identification tasks. Notably, PDeepPP attains high accuracy in antimicrobial (0.9726) and phosphorylation site (0.9984) identification, with 99.5% specificity in glycosylation site prediction and substantial reduction in false negatives in antimalarial tasks. By enabling large-scale, accurate peptide analysis, PDeepPP supports biomedical research and the discovery of novel therapeutic targets for disease treatment. All code, datasets, and pretrained models are publicly available via GitHub (https://github.com/fondress/PDeepPP) and Hugging Face (https://huggingface.co/fondress/PDeppPP)
comment : 15 pages, 5 figures, 3 tables, submitted to arXiv
♻ ☆ Multimodal Markup Document Models for Graphic Design Completion
Kotaro Kikuchi, Ukyo Honda, Naoto Inoue, Mayu Otani, Edgar Simo-Serra, Kota Yamaguchi
We introduce MarkupDM, a multimodal markup document model that represents graphic design as an interleaved multimodal document consisting of both markup language and images. Unlike existing holistic approaches that rely on an element-by-attribute grid representation, our representation accommodates variable-length elements, type-dependent attributes, and text content. Inspired by fill-in-the-middle training in code generation, we train the model to complete the missing part of a design document from its surrounding context, allowing it to treat various design tasks in a unified manner. Our model also supports image generation by predicting discrete image tokens through a specialized tokenizer with support for image transparency. We evaluate MarkupDM on three tasks, attribute value, image, and text completion, and demonstrate that it can produce plausible designs consistent with the given context. To further illustrate the flexibility of our approach, we evaluate our approach on a new instruction-guided design completion task where our instruction-tuned MarkupDM compares favorably to state-of-the-art image editing models, especially in textual completion. These findings suggest that multimodal language models with our document representation can serve as a versatile foundation for broad design automation.
comment : Accepted by ACM Multimedia 2025, Project page: https://cyberagentailab.github.io/MarkupDM/
♻ ☆ Scaling Towards the Information Boundary of Instruction Sets: The Infinity Instruct Subject Technical Report
Li Du, Hanyu Zhao, Yiming Ju, Tengfei Pan
Instruction tuning has become a foundation for unlocking the capabilities of large-scale pretrained models and improving their performance on complex tasks. Thus, the construction of high-quality instruction datasets is crucial for enhancing model performance and generalizability. Although current instruction datasets have reached tens of millions of samples, models finetuned on them may still struggle with complex instruction following and tasks in rare domains. This is primarily due to limited expansion in both ``coverage'' (coverage of task types and knowledge areas) and ``depth'' (instruction complexity) of the instruction set. To address this issue, we propose a systematic instruction data construction framework, which integrates a hierarchical tagging system, an informative seed selection algorithm, an evolutionary data synthesis process, and a model deficiency diagnosis with targeted data generation. These components form an iterative closed-loop to continuously enhance the coverage and depth of instruction data. Based on this framework, we construct Infinity Instruct Subject, a high-quality dataset containing $\sim$1.5 million instructions. Experiments on multiple foundation models and benchmark tasks demonstrate its effectiveness in improving instruction-following capabilities. Further analyses suggest that Infinity Instruct Subject shows enlarged coverage and depth compared to comparable synthesized instruction datasets. Our work lays a theoretical and practical foundation for the efficient, continuous evolution of instruction datasets, moving from data quantity expansion to qualitative improvement.
♻ ☆ Towards an end-to-end artificial intelligence driven global weather forecasting system
Kun Chen, Lei Bai, Fenghua Ling, Peng Ye, Tao Chen, Hang Fan, Hao Chen, Yi Xiao, Kang Chen, Tao Han, Jing-Jia Luo, Wanli Ouyang
The weather forecasting system is important for science and society, and significant achievements have been made in applying artificial intelligence (AI) to medium-range weather forecasting. However, existing AI-based weather forecasting models rely on analysis or reanalysis products from traditional numerical weather prediction (NWP) systems as initial conditions for making predictions. The initial states are typically generated by traditional data assimilation components, which are computationally expensive and time-consuming. Here, by cyclic training to model the steady-state background error covariance and introducing the confidence matrix to characterize the quality of observations, we present an AI-based data assimilation model, i.e., Adas, for global weather variables. Further, we combine Adas with the advanced AI-based forecasting model (i.e., FengWu) to construct an end-to-end AI-based global weather forecasting system: FengWu-Adas. We demonstrate that Adas can assimilate global conventional observations to produce high-quality analysis, enabling the system to operate stably for long term. Moreover, the system can generate accurate end-to-end weather forecasts with comparable skill to those of the IFS, demonstrating the promising potential of data-driven approaches.
♻ ☆ When Ads Become Profiles: Uncovering the Invisible Risk of Web Advertising at Scale with LLMs
Baiyu Chen, Benjamin Tag, Hao Xue, Daniel Angus, Flora Salim
Regulatory limits on explicit targeting have not eliminated algorithmic profiling on the Web, as optimisation systems still adapt ad delivery to users' private attributes. The widespread availability of powerful zero-shot multimodal Large Language Models (LLMs) has dramatically lowered the barrier for exploiting these latent signals for adversarial inference. We investigate this emerging societal risk, specifically how adversaries can now exploit these signals to reverse-engineer private attributes from ad exposure alone. We introduce a novel pipeline that leverages LLMs as adversarial inference engines to perform natural language profiling. Applying this method to a longitudinal dataset comprising over 435,000 ad impressions collected from 891 users, we conducted a large-scale study to assess the feasibility and precision of inferring private attributes from passive online ad observations. Our results demonstrate that off-the-shelf LLMs can accurately reconstruct complex user private attributes, including party preference, employment status, and education level, consistently outperforming strong census-based priors and matching or exceeding human social perception, while operating at only a fraction of the cost (223$\times$ lower) and time (52$\times$ faster) required by humans. Critically, actionable profiling is feasible even within short observation windows, indicating that prolonged tracking is not a prerequisite for a successful attack. These findings provide the first empirical evidence that ad streams serve as a high-fidelity digital footprint, enabling off-platform profiling that inherently bypasses current platform safeguards, highlighting a systemic vulnerability in the ad ecosystem and the urgent need for responsible web AI governance in the generative AI era. The code is available at https://github.com/Breezelled/when-ads-become-profiles.
♻ ☆ In- Context Representation Hijacking
Itay Yona, Amir Sarid, Michael Karasik, Yossi Gandelsman
We introduce $\textbf{Doublespeak}$, a simple in-context representation hijacking attack against large language models (LLMs). The attack works by systematically replacing a harmful keyword (e.g., bomb) with a benign token (e.g., carrot) across multiple in-context examples, provided a prefix to a harmful request. We demonstrate that this substitution leads to the internal representation of the benign token converging toward that of the harmful one, effectively embedding the harmful semantics under a euphemism. As a result, superficially innocuous prompts (e.g., "How to build a carrot?") are internally interpreted as disallowed instructions (e.g., "How to build a bomb?"), thereby bypassing the model's safety alignment. We use interpretability tools to show that this semantic overwrite emerges layer by layer, with benign meanings in early layers converging into harmful semantics in later ones. Doublespeak is optimization-free, broadly transferable across model families, and achieves strong success rates on closed-source and open-source systems, reaching 74% ASR on Llama-3.3-70B-Instruct with a single-sentence context override. Our findings highlight a new attack surface in the latent space of LLMs, revealing that current alignment strategies are insufficient and should instead operate at the representation level.
♻ ☆ Towards Heterogeneous Quantum Federated Learning: Challenges and Solutions
Ratun Rahman, Dinh C. Nguyen, Christo Kurisummoottil Thomas, Walid Saad
Quantum federated learning (QFL) combines quantum computing and federated learning to enable decentralized model training while maintaining data privacy. QFL can improve computational efficiency and scalability by taking advantage of quantum properties such as superposition and entanglement. However, existing QFL frameworks largely focus on homogeneity among quantum \textcolor{black}{clients, and they do not account} for real-world variances in quantum data distributions, encoding techniques, hardware noise levels, and computational capacity. These differences can create instability during training, slow convergence, and reduce overall model performance. In this paper, we conduct an in-depth examination of heterogeneity in QFL, classifying it into two categories: data or system heterogeneity. Then we investigate the influence of heterogeneity on training convergence and model aggregation. We critically evaluate existing mitigation solutions, highlight their limitations, and give a case study that demonstrates the viability of tackling quantum heterogeneity. Finally, we discuss potential future research areas for constructing robust and scalable heterogeneous QFL frameworks.
comment : Accepted at IEEE Network Magazine
♻ ☆ Escaping Barren Plateaus in Variational Quantum Algorithms Using Negative Learning Rate in Quantum Internet of Things
Ratun Rahman, Dinh C. Nguyen
Variational Quantum Algorithms (VQAs) are becoming the primary computational primitive for next-generation quantum computers, particularly those embedded as resource-constrained accelerators in the emerging Quantum Internet of Things (QIoT). However, under such device-constrained execution conditions, the scalability of learning is severely limited by barren plateaus, where gradients collapse to zero and training stalls. This poses a practical challenge to delivering VQA-enabled intelligence on QIoT endpoints, which often have few qubits, constrained shot budgets, and strict latency requirements. In this paper, we present a novel approach for escaping barren plateaus by including negative learning rates into the optimization process in QIoT devices. Our method introduces controlled instability into model training by switching between positive and negative learning phases, allowing recovery of significant gradients and exploring flatter areas in the loss landscape. We theoretically evaluate the effect of negative learning on gradient variance and propose conditions under which it helps escape from barren zones. The experimental findings on typical VQA benchmarks show consistent improvements in both convergence and simulation results over traditional optimizers. By escaping barren plateaus, our approach leads to a novel pathway for robust optimization in quantum-classical hybrid models.
comment : Accepted at IEEE Internet of Things Journal
♻ ☆ MILR: Improving Multimodal Image Generation via Test-Time Latent Reasoning
Yapeng Mi, Hengli Li, Yanpeng Zhao, Chenxi Li, Huimin Wu, Xiaojian Ma, Song-Chun Zhu, Ying Nian Wu, Qing Li
Reasoning-augmented machine learning systems have shown improved performance in various domains, including image generation. However, existing reasoning-based methods for image generation either restrict reasoning to a single modality (image or text) or rely on high-quality reasoning data for fine-tuning. To tackle these limitations, we propose MILR, a test-time method that jointly reasons over image and text in a unified latent vector space. Reasoning in MILR is performed by searching through vector representations of discrete image and text tokens. Practically, this is implemented via the policy gradient method, guided by an image quality critic. We instantiate MILR within the unified multimodal understanding and generation (MUG) framework that natively supports language reasoning before image synthesis and thus facilitates cross-modal reasoning. The intermediate model outputs, which are to be optimized, serve as the unified latent space, enabling MILR to operate entirely at test time. We evaluate MILR on GenEval, T2I-CompBench, and WISE, achieving state-of-the-art results on all benchmarks. Notably, on knowledge-intensive WISE, MILR attains an overall score of 0.63, improving over the baseline by 80%. Our further analysis indicates that joint reasoning in the unified latent space is the key to its strong performance. Moreover, our qualitative studies reveal MILR's non-trivial ability in temporal and cultural reasoning, highlighting the efficacy of our reasoning method.
comment : 21 pages,13 figures,9 tables
♻ ☆ Sharp Eyes and Memory for VideoLLMs: Information -Aware Visual Token Pruning for Efficient and Reliable VideoLLM Reasoning AAAI
Jialong Qin, Xin Zou, Di Lu, Yibo Yan, Xuming Hu
Current Video Large Language Models (VideoLLMs) suffer from quadratic computational complexity and key-value cache scaling, due to their reliance on processing excessive redundant visual tokens. To address this problem, we propose SharpV, a minimalist and efficient method for adaptive pruning of visual tokens and KV cache. Different from most uniform compression approaches, SharpV dynamically adjusts pruning ratios based on spatial-temporal information. Remarkably, this adaptive mechanism occasionally achieves performance gains over dense models, offering a novel paradigm for adaptive pruning. During the KV cache pruning stage, based on observations of visual information degradation, SharpV prunes degraded visual features via a self-calibration manner, guided by similarity to original visual features. In this way, SharpV achieves hierarchical cache pruning from the perspective of information bottleneck, offering a new insight into VideoLLMs' information flow. Experiments on multiple public benchmarks demonstrate the superiority of SharpV. Moreover, to the best of our knowledge, SharpV is notably the first two-stage pruning framework that operates without requiring access to exposed attention scores, ensuring full compatibility with hardware acceleration techniques like Flash Attention.
comment : The 40th Annual AAAI Conference on Artificial Intelligence (AAAI-26) Poster
♻ ☆ Beyond KAN: Introducing KarSein for Adaptive High-Order Feature Interaction Modeling in CTR Prediction
Yunxiao Shi, Wujiang Xu, Haimin Zhang, Qiang Wu, Min Xu
Modeling high-order feature interactions is crucial for click-through rate (CTR) prediction, yet traditional approaches typically predefine a maximum interaction order and exhaustively enumerate feature combinations up to that order. This paradigm depends heavily on prior domain knowledge to delimit the interaction space and incurs substantial computational overhead. As a result, conventional CTR models face a persistent tension between enriching representations with complex high-order interactions and keeping computation tractable. To address this dual challenge, this study introduces the Kolmogorov-Arnold Represented Sparse Efficient Interaction Network (KarSein). Drawing inspiration from the learnable activation mechanism in the Kolmogorov-Arnold Network (KAN), KarSein leverages this mechanism to adaptively transform low-order basic features into high-order feature interactions, offering a novel approach to feature interaction modeling. KarSein extends the capabilities of KAN by introducing a more efficient architecture that significantly reduces computational costs while accommodating two-dimensional embedding vectors as feature inputs. Furthermore, it overcomes the limitation of KAN's its inability to spontaneously capture multiplicative relationships among features. Extensive experiments highlight the superiority of KarSein, demonstrating its ability to surpass not only the vanilla implementation of KAN in CTR prediction tasks but also other baseline methods. Remarkably, KarSein achieves exceptional predictive accuracy while maintaining a highly compact parameter size and minimal computational overhead. Moreover, KarSein exhibits strong interpretability and structural sparsity. As the first systematic adaptation of KAN to CTR prediction, KarSein offers a practical, parameter-efficient, and interpretable alternative for modeling complex feature interactions in CTR prediction.
comment : Under review by TOIS
♻ ☆ On the Rate of Convergence of Kolmogorov-Arnold Network Regression Estimators
Wei Liu, Eleni Chatzi, Zhilu Lai
Kolmogorov-Arnold Networks (KANs) offer a structured and interpretable framework for multivariate function approximation by composing univariate transformations through additive or multiplicative aggregation. This paper establishes theoretical convergence guarantees for KANs when the univariate components are represented by B-splines. We prove that both additive and hybrid additive-multiplicative KANs attain the minimax-optimal convergence rate $O(n^{-2r/(2r+1)})$ for functions in Sobolev spaces of smoothness $r$. We further derive guidelines for selecting the optimal number of knots in the B-splines. The theory is supported by simulation studies that confirm the predicted convergence rates. These results provide a theoretical foundation for using KANs in nonparametric regression and highlight their potential as a structured alternative to existing methods.
♻ ☆ Bayesian Concept Bottleneck Models with LLM Priors
Jean Feng, Avni Kothari, Luke Zier, Chandan Singh, Yan Shuo Tan
Concept Bottleneck Models (CBMs) have been proposed as a compromise between white-box and black-box models, aiming to achieve interpretability without sacrificing accuracy. The standard training procedure for CBMs is to predefine a candidate set of human-interpretable concepts, extract their values from the training data, and identify a sparse subset as inputs to a transparent prediction model. However, such approaches are often hampered by the tradeoff between exploring a sufficiently large set of concepts versus controlling the cost of obtaining concept extractions, resulting in a large interpretability-accuracy tradeoff. This work investigates a novel approach that sidesteps these challenges: BC-LLM iteratively searches over a potentially infinite set of concepts within a Bayesian framework, in which Large Language Models (LLMs) serve as both a concept extraction mechanism and prior. Even though LLMs can be miscalibrated and hallucinate, we prove that BC-LLM can provide rigorous statistical inference and uncertainty quantification. Across image, text, and tabular datasets, BC-LLM outperforms interpretable baselines and even black-box models in certain settings, converges more rapidly towards relevant concepts, and is more robust to out-of-distribution samples.
comment : 2025 Conference on Neural Information Processing Systems
♻ ☆ A Hierarchical Tree-based approach for creating Configurable and Static Deep Research Agent (Static-DRA)
Saurav Prateek
The advancement in Large Language Models has driven the creation of complex agentic systems, such as Deep Research Agents (DRAs), to overcome the limitations of static Retrieval Augmented Generation (RAG) pipelines in handling complex, multi-turn research tasks. This paper introduces the Static Deep Research Agent (Static-DRA), a novel solution built upon a configurable and hierarchical Tree-based static workflow. The core contribution is the integration of two user-tunable parameters, Depth and Breadth, which provide granular control over the research intensity. This design allows end-users to consciously balance the desired quality and comprehensiveness of the research report against the associated computational cost of Large Language Model (LLM) interactions. The agent's architecture, comprising Supervisor, Independent, and Worker agents, facilitates effective multi-hop information retrieval and parallel sub-topic investigation. We evaluate the Static-DRA against the established DeepResearch Bench using the RACE (Reference-based Adaptive Criteria-driven Evaluation) framework. Configured with a depth of 2 and a breadth of 5, and powered by the gemini-2.5-pro model, the agent achieved an overall score of 34.72. Our experiments validate that increasing the configured Depth and Breadth parameters results in a more in-depth research process and a correspondingly higher evaluation score. The Static-DRA offers a pragmatic and resource-aware solution, empowering users with transparent control over the deep research process. The entire source code, outputs and benchmark results are open-sourced at https://github.com/SauravP97/Static-Deep-Research/
comment : 16 pages, 6 figures, 4 tables. Code available at: https://github.com/SauravP97/Static-Deep-Research
♻ ☆ Resilience Inference for Supply Chains with Hypergraph Neural Network
Zetian Shen, Hongjun Wang, Jiyuan Chen, Xuan Song
Supply chains are integral to global economic stability, yet disruptions can swiftly propagate through interconnected networks, resulting in substantial economic impacts. Accurate and timely inference of supply chain resilience the capability to maintain core functions during disruptions is crucial for proactive risk mitigation and robust network design. However, existing approaches lack effective mechanisms to infer supply chain resilience without explicit system dynamics and struggle to represent the higher-order, multi-entity dependencies inherent in supply chain networks. These limitations motivate the definition of a novel problem and the development of targeted modeling solutions. To address these challenges, we formalize a novel problem: Supply Chain Resilience Inference (SCRI), defined as predicting supply chain resilience using hypergraph topology and observed inventory trajectories without explicit dynamic equations. To solve this problem, we propose the Supply Chain Resilience Inference Hypergraph Network (SC-RIHN), a novel hypergraph-based model leveraging set-based encoding and hypergraph message passing to capture multi-party firm-product interactions. Comprehensive experiments demonstrate that SC-RIHN significantly outperforms traditional MLP, representative graph neural network variants, and ResInf baselines across synthetic benchmarks, underscoring its potential for practical, early-warning risk assessment in complex supply chain systems.
♻ ☆ AI Kill Switch for malicious web-based LLM agent
Sechan Lee, Sangdon Park
Recently, web-based Large Language Model (LLM) agents autonomously perform increasingly complex tasks, thereby bringing significant convenience. However, they also amplify the risks of malicious misuse cases such as unauthorized collection of personally identifiable information (PII), generation of socially divisive content, and even automated web hacking. To address these threats, we propose an AI Kill Switch technique that can immediately halt the operation of malicious web-based LLM agents. To achieve this, we introduce AutoGuard - the key idea is generating defensive prompts that trigger the safety mechanisms of malicious LLM agents. In particular, generated defense prompts are transparently embedded into the website's DOM so that they remain invisible to human users but can be detected by the crawling process of malicious agents, triggering its internal safety mechanisms to abort malicious actions once read. To evaluate our approach, we constructed a dedicated benchmark consisting of three representative malicious scenarios. Experimental results show that AutoGuard achieves over 80% Defense Success Rate (DSR) across diverse malicious agents, including GPT-4o, Claude-4.5-Sonnet and generalizes well to advanced models like GPT-5.1, Gemini-2.5-flash, and Gemini-3-pro. Also, our approach demonstrates robust defense performance in real-world website environments without significant performance degradation for benign agents. Through this research, we demonstrate the controllability of web-based LLM agents, thereby contributing to the broader effort of AI control and safety.
♻ ☆ Robust HRRP Recognition under Interrupted Sampling Repeater Jamming using a Prior Jamming Information -Guided Network
Guozheng Sun, Lei Wang, Yanhao Wang, Jie Wang, Yimin Liu
Radar automatic target recognition (RATR) based on high-resolution range profile (HRRP) has attracted increasing attention due to its ability to capture fine-grained structural features. However, recognizing targets under electronic countermeasures (ECM), especially the mainstream interrupted-sampling repeater jamming (ISRJ), remains a significant challenge, as HRRPs often suffer from serious feature distortion. To address this, we propose a robust HRRP recognition method guided by prior jamming information. Specifically, we introduce a point spread function (PSF) as prior information to model the HRRP distortion induced by ISRJ. Based on this, we design a recognition network that leverages this prior through a prior-guided feature interaction module and a hybrid loss function to enhance the model's discriminative capability. With the aid of prior information, the model can learn invariant features within distorted HRRP under different jamming parameters. Both the simulated and measured-data experiments demonstrate that our method consistently outperforms state-of-the-art approaches and exhibits stronger generalization capabilities when facing unseen jamming parameters.
♻ ☆ EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretrain ing
Boshen Xu, Yuting Mei, Xinbi Liu, Sipeng Zheng, Qin Jin
Egocentric video-language pretraining has significantly advanced video representation learning. Humans perceive and interact with a fully 3D world, developing spatial awareness that extends beyond text-based understanding. However, most previous works learn from 1D text or 2D visual cues, such as bounding boxes, which inherently lack 3D understanding. To bridge this gap, we introduce EgoDTM, an Egocentric Depth- and Text-aware Model, jointly trained through large-scale 3D-aware video pretraining and video-text contrastive learning. EgoDTM incorporates a lightweight 3D-aware decoder to efficiently learn 3D-awareness from pseudo depth maps generated by depth estimation models. To further facilitate 3D-aware video pretraining, we enrich the original brief captions with hand-object visual cues by organically combining several foundation models. Extensive experiments demonstrate EgoDTM's superior performance across diverse downstream tasks, highlighting its superior 3D-aware visual understanding. Code: https://github.com/xuboshen/EgoDTM.
comment : Code: https://github.com/xuboshen/EgoDTM
♻ ☆ Near-Optimal Experiment Design in Linear non-Gaussian Cyclic Models
Ehsan Sharifian, Saber Salehkaleybar, Negar Kiyavash
We study the problem of causal structure learning from a combination of observational and interventional data generated by a linear non-Gaussian structural equation model that might contain cycles. Recent results show that using mere observational data identifies the causal graph only up to a permutation-equivalence class. We obtain a combinatorial characterization of this class by showing that each graph in an equivalence class corresponds to a perfect matching in a bipartite graph. This bipartite representation allows us to analyze how interventions modify or constrain the matchings. Specifically, we show that each atomic intervention reveals one edge of the true matching and eliminates all incompatible causal graphs. Consequently, we formalize the optimal experiment design task as an adaptive stochastic optimization problem over the set of equivalence classes with a natural reward function that quantifies how many graphs are eliminated from the equivalence class by an intervention. We show that this reward function is adaptive submodular and provide a greedy policy with a provable near-optimal performance guarantee. A key technical challenge is to efficiently estimate the reward function without having to explicitly enumerate all the graphs in the equivalence class. We propose a sampling-based estimator using random matchings and analyze its bias and concentration behavior. Our simulation results show that performing a small number of interventions guided by our stochastic optimization framework recovers the true underlying causal structure.
♻ ☆ Route-and-Reason: Scaling Large Language Model Reasoning with Reinforced Model Router
Chenyang Shao, Xinyang Liu, Yutang Lin, Fengli Xu, Yong Li
Chain-of-thought has been proven essential for enhancing the complex reasoning abilities of Large Language Models (LLMs), but it also leads to high computational costs. Recent advances have explored the method to route queries among multiple models and proved it as a promising approach. However, previous works directly operate at the task level, i.e., assigning user queries to suitable LLMs, which does not allow hybrid LLMs to truly collaborate on finer-grained sub-tasks. Collaboration at the level of intermediate reasoning steps (thoughts) could enable more efficient coordination, but it also poses significant challenges for router scheduling, placing immense demands on the quality of task decomposition and the precision of the router. To address this, we propose R2-Reasoner, a novel framework centered around a Reinforced Model Router designed to efficiently scale LLM reasoning. This router orchestrates collaboration across nine heterogeneous models, whose parameter scales range from less than 1B to hundreds of billions, by first breaking down a complex query into subtasks with a decomposer, and then assigning each subtask to the optimal model with a subtask allocator, balancing performance with cost. Training this router involves a two-stage alternating process for the decomposer and the allocator, integrating supervised fine-tuning with reinforcement learning to enable effective self-supervised refinement. Extensive experiments across six challenging reasoning benchmarks demonstrate that R2-Reasoner reduces API costs by 84.46% compared with state-of-the-art baselines while maintaining competitive reasoning accuracy. Our framework paves the way for the development of more scalable and efficient reasoning systems. Our code is open-source at https://anonymous.4open.science/r/R2_Reasoner.
♻ ☆ Renewable Energy Prediction: A Comparative Study of Deep Learning Models for Complex Dataset Analysis
Haibo Wang, Jun Huang, Lutfu Sua, Bahram Alidaee
The increasing focus on predicting renewable energy production aligns with advancements in deep learning (DL). The inherent variability of renewable sources and the complexity of prediction methods require robust approaches, such as DL models, in the renewable energy sector. DL models are preferred over traditional machine learning (ML) because they capture complex, nonlinear relationships in renewable energy datasets. This study examines key factors influencing DL technique accuracy, including sampling and hyperparameter optimization, by comparing various methods and training and test ratios within a DL framework. Seven machine learning methods, LSTM, Stacked LSTM, CNN, CNN-LSTM, DNN, Time-Distributed MLP (TD-MLP), and Autoencoder (AE), are evaluated using a dataset combining weather and photovoltaic power output data from 12 locations. Regularization techniques such as early stopping, neuron dropout, L1 and L2 regularization are applied to address overfitting. The results demonstrate that the combination of early stopping, dropout, and L1 regularization provides the best performance to reduce overfitting in the CNN and TD-MLP models with larger training set, while the combination of early stopping, dropout, and L2 regularization is the most effective to reduce the overfitting in CNN-LSTM and AE models with smaller training set.
comment : 11 pages, 2 figures and 6 tables
♻ ☆ VRWKV-Editor: Reducing quadratic complexity in transformer -based video editing
Abdelilah Aitrouga, Youssef Hmamouche, Amal El Fallah Seghrouchni
In light of recent progress in video editing, deep learning models focusing on both spatial and temporal dependencies have emerged as the primary method. However, these models suffer from the quadratic computational complexity of traditional attention mechanisms, making them difficult to adapt to long-duration and high-resolution videos. This limitation restricts their applicability in practical contexts such as real-time video processing. To tackle this challenge, we introduce a method to reduce both time and space complexity of these systems by proposing VRWKV-Editor, a novel video editing model that integrates a linear spatio-temporal aggregation module into video-based diffusion models. VRWKV-Editor leverages bidirectional weighted key-value recurrence mechanism of the RWKV transformer to capture global dependencies while preserving temporal coherence, achieving linear complexity without sacrificing quality. Extensive experiments demonstrate that the proposed method achieves up to 3.7x speedup and 60% lower memory usage compared to state-of-the-art diffusion-based video editing methods, while maintaining competitive performance in frame consistency and text alignment. Furthermore, a comparative analysis we conducted on videos with different sequence lengths confirms that the gap in editing speed between our approach and architectures with self-attention becomes more significant with long videos.
♻ ☆ Sequential Monte Ca rl o for Policy Optimization in Continuous POMDPs NeurIPS 2025
Hany Abdulsamad, Sahel Iqbal, Simo Särkkä
Optimal decision-making under partial observability requires agents to balance reducing uncertainty (exploration) against pursuing immediate objectives (exploitation). In this paper, we introduce a novel policy optimization framework for continuous partially observable Markov decision processes (POMDPs) that explicitly addresses this challenge. Our method casts policy learning as probabilistic inference in a non-Markovian Feynman--Kac model that inherently captures the value of information gathering by anticipating future observations, without requiring suboptimal approximations or handcrafted heuristics. To optimize policies under this model, we develop a nested sequential Monte Carlo (SMC) algorithm that efficiently estimates a history-dependent policy gradient under samples from the optimal trajectory distribution induced by the POMDP. We demonstrate the effectiveness of our algorithm across standard continuous POMDP benchmarks, where existing methods struggle to act under uncertainty.
comment : Accepted at NeurIPS 2025
♻ ☆ FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusion
Anke Tang, Li Shen, Yong Luo, Enneng Yang, Han Hu, Lefei Zhang, Bo Du, Dacheng Tao
Deep model fusion is an emerging technique that unifies the predictions or parameters of several deep neural networks into a single better-performing model in a cost-effective and data-efficient manner. Although a variety of deep model fusion techniques have been introduced, their evaluations tend to be inconsistent and often inadequate to validate their effectiveness and robustness. We present FusionBench, the first benchmark and a unified library designed specifically for deep model fusion. Our benchmark consists of multiple tasks, each with different settings of models and datasets. This variety allows us to compare fusion methods across different scenarios and model scales. Additionally, FusionBench serves as a unified library for easy implementation and testing of new fusion techniques. FusionBench is open source and actively maintained, with community contributions encouraged. Homepage https://github.com/tanganke/fusion_bench
comment : Project homepage: https://github.com/tanganke/fusion_bench Online documentation: https://tanganke.github.io/fusion_bench
♻ ☆ Reversing Large Language Models for Efficient Training and Fine-Tuning
Eshed Gal, Moshe Eliasof, Javier Turek, Uri Ascher, Eran Treister, Eldad Haber
Large Language Models (LLMs) are known for their expensive and time-consuming training. Thus, oftentimes, LLMs are fine-tuned to address a specific task, given the pretrained weights of a pre-trained LLM considered a foundation model. In this work, we introduce memory-efficient, reversible architectures for LLMs, inspired by symmetric and symplectic differential equations, and investigate their theoretical properties. Different from standard, baseline architectures that store all intermediate activations, the proposed models use time-reversible dynamics to retrieve hidden states during backpropagation, relieving the need to store activations. This property allows for a drastic reduction in memory consumption, allowing for the processing of larger batch sizes for the same available memory, thereby offering improved throughput. In addition, we propose an efficient method for converting existing, non-reversible LLMs into reversible architectures through fine-tuning, rendering our approach practical for exploiting existing pre-trained models. Our results show comparable or improved performance on several datasets and benchmarks, on several LLMs, building a scalable and efficient path towards reducing the memory and computational costs associated with both training from scratch and fine-tuning of LLMs.
♻ ☆ RoCo: Role-Based LLMs Collaboration for Automatic Heuristic Design
Jiawei Xu, Feng-Feng Wei, Wei-Neng Chen
Automatic Heuristic Design (AHD) has gained traction as a promising solution for solving combinatorial optimization problems (COPs). Large Language Models (LLMs) have emerged and become a promising approach to achieving AHD, but current LLM-based AHD research often only considers a single role. This paper proposes RoCo, a novel Multi-Agent Role-Based System, to enhance the diversity and quality of AHD through multi-role collaboration. RoCo coordinates four specialized LLM-guided agents-explorer, exploiter, critic, and integrator-to collaboratively generate high-quality heuristics. The explorer promotes long-term potential through creative, diversity-driven thinking, while the exploiter focuses on short-term improvements via conservative, efficiency-oriented refinements. The critic evaluates the effectiveness of each evolution step and provides targeted feedback and reflection. The integrator synthesizes proposals from the explorer and exploiter, balancing innovation and exploitation to drive overall progress. These agents interact in a structured multi-round process involving feedback, refinement, and elite mutations guided by both short-term and accumulated long-term reflections. We evaluate RoCo on five different COPs under both white-box and black-box settings. Experimental results demonstrate that RoCo achieves superior performance, consistently generating competitive heuristics that outperform existing methods including ReEvo and HSEvo, both in white-box and black-box scenarios. This role-based collaborative paradigm establishes a new standard for robust and high-performing AHD.
♻ ☆ SYNTHIA: Novel Concept Design with Affordance Composition ACL 2025
Hyeonjeong Ha, Xiaomeng Jin, Jeonghwan Kim, Jiateng Liu, Zhenhailong Wang, Khanh Duy Nguyen, Ansel Blume, Nanyun Peng, Kai-Wei Chang, Heng Ji
Text-to-image (T2I) models enable rapid concept design, making them widely used in AI-driven design. While recent studies focus on generating semantic and stylistic variations of given design concepts, functional coherence--the integration of multiple affordances into a single coherent concept--remains largely overlooked. In this paper, we introduce SYNTHIA, a framework for generating novel, functionally coherent designs based on desired affordances. Our approach leverages a hierarchical concept ontology that decomposes concepts into parts and affordances, serving as a crucial building block for functionally coherent design. We also develop a curriculum learning scheme based on our ontology that contrastively fine-tunes T2I models to progressively learn affordance composition while maintaining visual novelty. To elaborate, we (i) gradually increase affordance distance, guiding models from basic concept-affordance association to complex affordance compositions that integrate parts of distinct affordances into a single, coherent form, and (ii) enforce visual novelty by employing contrastive objectives to push learned representations away from existing concepts. Experimental results show that SYNTHIA outperforms state-of-the-art T2I models, demonstrating absolute gains of 25.1% and 14.7% for novelty and functional coherence in human evaluation, respectively.
comment : ACL 2025 Main, Code is available https://github.com/HyeonjeongHa/SYNTHIA
♻ ☆ Q-STAC: Q-Guided Stein Variational Model Predictive Actor-Critic
Shizhe Cai, Zeya Yin, Jayadeep Jacob, Fabio Ramos
Deep reinforcement learning (DRL) often struggles with complex robotic manipulation tasks due to low sample efficiency and biased value estimation. Model-based reinforcement learning (MBRL) improves efficiency by leveraging environment dynamics, with prior work integrating Model Predictive Control (MPC) to enhance policy robustness through online trajectory optimization. However, existing MBRL approaches still suffer from high model bias, task-specific cost function design, and significant computational overhead. To address these challenges, we propose Q-guided Stein Variational Model Predictive Actor-Critic (Q-STAC)--a unified framework that bridges Bayesian MPC and Soft Actor-Critic (SAC). Q-STAC employs Stein Variational Gradient Descent (SVGD) to iteratively optimize action sequences sampled from a learned prior distribution guided by Q-values, thereby eliminating manual cost-function engineering. By performing short-horizon model-predictive rollouts, Q-STAC reduces cumulative prediction errors, improves training stability and reduces computational complexity. Experiments on simulated particle navigation, diverse robotic manipulation tasks, and a real-world fruit-picking scenario demonstrate that Q-STAC consistently achieves superior sample efficiency, stability, and overall performance compared to both model-free and model-based baselines.
comment : 9 pages, 10 figures
♻ ☆ Co-Evolving Agent s: Learning from Failures as Hard Negatives
Yeonsung Jung, Trilok Padhi, Sina Shaham, Dipika Khullar, Joonhyun Jeong, Ninareh Mehrabi, Eunho Yang
The rapid progress of large foundation models has accelerated the development of task-specialized agents across diverse domains. However, the effectiveness of agents remains tightly coupled with the quality of training data, while curating task-specific datasets remains costly and often infeasible in real-world scenarios. Recent work has explored self-improving agents that autonomously generate, refine, and re-train on their own trajectories. A prominent line of approaches further leverages preference optimization by pairing predicted trajectories with scarce ground-truth trajectories, enabling agents to learn directly from their own failures. While these methods outperform supervised fine-tuning, their heavy reliance on predicted trajectories under limited ground-truth supervision leaves them prone to overfitting. To address this, we propose a co-evolving agents framework in which a target agent improves jointly with an auxiliary failure agent. The failure agent learns through preference optimization over failure trajectories from both the target and itself, thereby generating hard negatives that are close to success yet remain failures. Incorporating these informative hard negatives into the target agent's optimization sharpens decision boundaries and enhances generalization. Our comprehensive analysis and experiments across benchmark datasets show that our method not only shows improved performance but also demonstrates that failures, instead of being used as-is, can be systematically transformed into structured and valuable learning signals in self-improving agents.
♻ ☆ NeuroPhysNet: A FitzHugh-Nagumo-Based Physics-Informed Neural Network Framework for Electroencephalograph (EEG) Analysis and Motor Imagery Classification
Zhenyu Xia, Xinlei Huang, Yuantong Gu, Suvash C. Saha
Electroencephalography (EEG) is extensively employed in medical diagnostics and brain-computer interface (BCI) applications due to its non-invasive nature and high temporal resolution. However, EEG analysis faces significant challenges, including noise, nonstationarity, and inter-subject variability, which hinder its clinical utility. Traditional neural networks often lack integration with biophysical knowledge, limiting their interpretability, robustness, and potential for medical translation. To address these limitations, this study introduces NeuroPhysNet, a novel Physics-Informed Neural Network (PINN) framework tailored for EEG signal analysis and motor imagery classification in medical contexts. NeuroPhysNet incorporates the FitzHugh-Nagumo model, embedding neurodynamical principles to constrain predictions and enhance model robustness. Evaluated on the BCIC-IV-2a dataset, the framework achieved superior accuracy and generalization compared to conventional methods, especially in data-limited and cross-subject scenarios, which are common in clinical settings. By effectively integrating biophysical insights with data-driven techniques, NeuroPhysNet not only advances BCI applications but also holds significant promise for enhancing the precision and reliability of clinical diagnostics, such as motor disorder assessments and neurorehabilitation planning.
comment : Here is a revised version of the manuscript, incorporating Prof. Yuantong Gu's contributions to restructuring and revising the manuscript
♻ ☆ TraceTrans: Translation and Spatial Tracing for Surgical Prediction
Xiyu Luo, Haodong Li, Xinxing Cheng, He Zhao, Yang Hu, Xuan Song, Tianyang Zhang
Image-to-image translation models have achieved notable success in converting images across visual domains and are increasingly used for medical tasks such as predicting post-operative outcomes and modeling disease progression. However, most existing methods primarily aim to match the target distribution and often neglect spatial correspondences between the source and translated images. This limitation can lead to structural inconsistencies and hallucinations, undermining the reliability and interpretability of the predictions. These challenges are accentuated in clinical applications by the stringent requirement for anatomical accuracy. In this work, we present TraceTrans, a novel deformable image translation model designed for post-operative prediction that generates images aligned with the target distribution while explicitly revealing spatial correspondences with the pre-operative input. The framework employs an encoder for feature extraction and dual decoders for predicting spatial deformations and synthesizing the translated image. The predicted deformation field imposes spatial constraints on the generated output, ensuring anatomical consistency with the source. Extensive experiments on medical cosmetology and brain MRI datasets demonstrate that TraceTrans delivers accurate and interpretable post-operative predictions, highlighting its potential for reliable clinical deployment.
Computer Vision and Pattern Recognition 171
☆ The Universal Weight Subspace Hypothesis
Prakhar Kaushik, Shravan Chaudhari, Ankit Vaidya, Rama Chellappa, Alan Yuille
We show that deep neural networks trained across diverse tasks exhibit remarkably similar low-dimensional parametric subspaces. We provide the first large-scale empirical evidence that demonstrates that neural networks systematically converge to shared spectral subspaces regardless of initialization, task, or domain. Through mode-wise spectral analysis of over 1100 models - including 500 Mistral-7B LoRAs, 500 Vision Transformers, and 50 LLaMA-8B models - we identify universal subspaces capturing majority variance in just a few principal directions. By applying spectral decomposition techniques to the weight matrices of various architectures trained on a wide range of tasks and datasets, we identify sparse, joint subspaces that are consistently exploited, within shared architectures across diverse tasks and datasets. Our findings offer new insights into the intrinsic organization of information within deep networks and raise important questions about the possibility of discovering these universal subspaces without the need for extensive data and computational resources. Furthermore, this inherent structure has significant implications for model reusability, multi-task learning, model merging, and the development of training and inference-efficient algorithms, potentially reducing the carbon footprint of large-scale neural models.
comment : 37 pages
☆ Light-X: Generative 4D Video Rendering with Camera and Illumination Control
Tianqi Liu, Zhaoxi Chen, Zihao Huang, Shaocong Xu, Saining Zhang, Chongjie Ye, Bohan Li, Zhiguo Cao, Wei Li, Hao Zhao, Ziwei Liu
Recent advances in illumination control extend image-based methods to video, yet still facing a trade-off between lighting fidelity and temporal consistency. Moving beyond relighting, a key step toward generative modeling of real-world scenes is the joint control of camera trajectory and illumination, since visual dynamics are inherently shaped by both geometry and lighting. To this end, we present Light-X, a video generation framework that enables controllable rendering from monocular videos with both viewpoint and illumination control. 1) We propose a disentangled design that decouples geometry and lighting signals: geometry and motion are captured via dynamic point clouds projected along user-defined camera trajectories, while illumination cues are provided by a relit frame consistently projected into the same geometry. These explicit, fine-grained cues enable effective disentanglement and guide high-quality illumination. 2) To address the lack of paired multi-view and multi-illumination videos, we introduce Light-Syn, a degradation-based pipeline with inverse-mapping that synthesizes training pairs from in-the-wild monocular footage. This strategy yields a dataset covering static, dynamic, and AI-generated scenes, ensuring robust training. Extensive experiments show that Light-X outperforms baseline methods in joint camera-illumination control and surpasses prior video relighting methods under both text- and background-conditioned settings.
comment : Project Page: https://lightx-ai.github.io/
☆ Value Gradient Guidance for Flow Matching Alignment NeurIPS 2025
Zhen Liu, Tim Z. Xiao, Carles Domingo-Enrich, Weiyang Liu, Dinghuai Zhang
While methods exist for aligning flow matching models--a popular and effective class of generative models--with human preferences, existing approaches fail to achieve both adaptation efficiency and probabilistically sound prior preservation. In this work, we leverage the theory of optimal control and propose VGG-Flow, a gradient-matching-based method for finetuning pretrained flow matching models. The key idea behind this algorithm is that the optimal difference between the finetuned velocity field and the pretrained one should be matched with the gradient field of a value function. This method not only incorporates first-order information from the reward model but also benefits from heuristic initialization of the value function to enable fast adaptation. Empirically, we show on a popular text-to-image flow matching model, Stable Diffusion 3, that our method can finetune flow matching models under limited computational budgets while achieving effective and prior-preserving alignment.
comment : Accepted at NeurIPS 2025; 26 pages, 20 figures
☆ Deep infant brain segmentation from multi-contrast MRI
Malte Hoffmann, Lilla Zöllei, Adrian V. Dalca
Segmentation of magnetic resonance images (MRI) facilitates analysis of human brain development by delineating anatomical structures. However, in infants and young children, accurate segmentation is challenging due to development and imaging constraints. Pediatric brain MRI is notoriously difficult to acquire, with inconsistent availability of imaging modalities, substantial non-head anatomy in the field of view, and frequent motion artifacts. This has led to specialized segmentation models that are often limited to specific image types or narrow age groups, or that are fragile for more variable images such as those acquired clinically. We address this method fragmentation with BabySeg, a deep learning brain segmentation framework for infants and young children that supports diverse MRI protocols, including repeat scans and image types unavailable during training. Our approach builds on recent domain randomization techniques, which synthesize training images far beyond realistic bounds to promote dataset shift invariance. We also describe a mechanism that enables models to flexibly pool and interact features from any number of input scans. We demonstrate state-of-the-art performance that matches or exceeds the accuracy of several existing methods for various age cohorts and input configurations using a single model, in a fraction of the runtime required by many existing tools.
comment : 8 pages, 8 figures, 1 table, website at https://w3id.org/babyseg, presented at the 2025 IEEE Asilomar Conference on Signals, Systems, and Computers
☆ Splannequin: Freezing Monocular Mannequin-Challenge Footage with Dual-Detection Splatting WACV 2025
Hao-Jen Chien, Yi-Chuan Huang, Chung-Ho Wu, Wei-Lun Chao, Yu-Lun Liu
Synthesizing high-fidelity frozen 3D scenes from monocular Mannequin-Challenge (MC) videos is a unique problem distinct from standard dynamic scene reconstruction. Instead of focusing on modeling motion, our goal is to create a frozen scene while strategically preserving subtle dynamics to enable user-controlled instant selection. To achieve this, we introduce a novel application of dynamic Gaussian splatting: the scene is modeled dynamically, which retains nearby temporal variation, and a static scene is rendered by fixing the model's time parameter. However, under this usage, monocular capture with sparse temporal supervision introduces artifacts like ghosting and blur for Gaussians that become unobserved or occluded at weakly supervised timestamps. We propose Splannequin, an architecture-agnostic regularization that detects two states of Gaussian primitives, hidden and defective, and applies temporal anchoring. Under predominantly forward camera motion, hidden states are anchored to their recent well-observed past states, while defective states are anchored to future states with stronger supervision. Our method integrates into existing dynamic Gaussian pipelines via simple loss terms, requires no architectural changes, and adds zero inference overhead. This results in markedly improved visual quality, enabling high-fidelity, user-selectable frozen-time renderings, validated by a 96% user preference. Project page: https://chien90190.github.io/splannequin/
comment : WACV 2025. Project page: https://chien90190.github.io/splannequin/
☆ DraCo: Draft as CoT for Text-to-Image P review and Rare Concept Generation
Dongzhi Jiang, Renrui Zhang, Haodong Li, Zhuofan Zong, Ziyu Guo, Jun He, Claire Guo, Junyan Ye, Rongyao Fang, Weijia Li, Rui Liu, Hongsheng Li
Recent unified multimodal large language models (MLLMs) have shown impressive capabilities, incorporating chain-of-thought (CoT) reasoning for enhanced text-to-image generation. However, existing approaches remain limited, either treating the model merely as a standalone generator or relying on abstract textual planning. To this end, we propose Draft-as-CoT (DraCo), a novel interleaved reasoning paradigm that fully leverages both textual and visual contents in CoT for better planning and verification. Our method first generates a low-resolution draft image as preview, providing more concrete and structural visual planning and guidance. Then, we employ the model's inherent understanding capability to verify potential semantic misalignments between the draft and input prompt, and performs refinement through selective corrections with super-resolution. In this way, our approach addresses two fundamental challenges: the coarse-grained nature of textual planning and the difficulty in generating rare attribute combinations. To support training, we curate DraCo-240K, aiming to enhance three atomic capabilities spanning general correction, instance manipulation, and layout reorganization. Supported by DraCo-CFG, a specialized classifier-free guidance (CFG) strategy for interleaved reasoning, DraCo achieves a tremendous increase on GenEval (+8%), Imagine-Bench (+0.91), and GenEval++ (+3%), significantly outperforming direct generation and other generation methods empowered by CoT.
comment : Project Page: https://github.com/CaraJ7/DraCo
☆ ARM-Thinker: Reinforcing Multimodal Generative Reward Model s with Agent ic Tool Use and Visual Reasoning
Shengyuan Ding, Xinyu Fang, Ziyu Liu, Yuhang Zang, Yuhang Cao, Xiangyu Zhao, Haodong Duan, Xiaoyi Dong, Jianze Liang, Bin Wang, Conghui He, Dahua Lin, Jiaqi Wang
Reward models are critical for aligning vision-language systems with human preferences, yet current approaches suffer from hallucination, weak visual grounding, and an inability to use tools for verification, limiting their reliability on complex multimodal reasoning tasks. We present ARM-Thinker, an A}gentic multimodal Reward Model that autonomously invokes external tools (e.g., image cropping, doc page retrieval) to ground judgments in verifiable evidence, replacing static, non-interactive reward scoring. This enables the model to verify fine-grained visual details, cross-reference multi-page evidence, and validate reasoning claims, which are capabilities absent in existing reward models. We train ARM-Thinker with multi-stage reinforcement learning, jointly optimizing tool-calling decisions and judgment accuracy. To evaluate agentic reward modeling, we introduce ARMBench-VL, comprising three benchmarks that assess fine-grained visual grounding (image-level tools), multi-page document understanding (retrieval tools), and instruction following (text-level verification). ARM-Thinker achieves +16.2% average improvement on reward modeling benchmarks, +9.6% on tool-use tasks, and outperforms baselines on multimodal math and logical reasoning benchmarks. Our results demonstrate that agentic capabilities significantly enhance both accuracy and interpretability of reward models.
☆ ShadowDraw: From Any Object to Shadow-Drawing Compositional Art
Rundong Luo, Noah Snavely, Wei-Chiu Ma
We introduce ShadowDraw, a framework that transforms ordinary 3D objects into shadow-drawing compositional art. Given a 3D object, our system predicts scene parameters, including object pose and lighting, together with a partial line drawing, such that the cast shadow completes the drawing into a recognizable image. To this end, we optimize scene configurations to reveal meaningful shadows, employ shadow strokes to guide line drawing generation, and adopt automatic evaluation to enforce shadow-drawing coherence and visual quality. Experiments show that ShadowDraw produces compelling results across diverse inputs, from real-world scans and curated datasets to generative assets, and naturally extends to multi-object scenes, animations, and physical deployments. Our work provides a practical pipeline for creating shadow-drawing art and broadens the design space of computational visual art, bridging the gap between algorithmic design and artistic storytelling. Check out our project page https://red-fairy.github.io/ShadowDraw/ for more results and an end-to-end real-world demonstration of our pipeline!
comment : Project page: https://red-fairy.github.io/ShadowDraw/
☆ NeuralRemaster: Phase-Preserving Diffusion for Structure-Aligned Generation
Yu Zeng, Charles Ochoa, Mingyuan Zhou, Vishal M. Patel, Vitor Guizilini, Rowan McAllister
Standard diffusion corrupts data using Gaussian noise whose Fourier coefficients have random magnitudes and random phases. While effective for unconditional or text-to-image generation, corrupting phase components destroys spatial structure, making it ill-suited for tasks requiring geometric consistency, such as re-rendering, simulation enhancement, and image-to-image translation. We introduce Phase-Preserving Diffusion φ-PD, a model-agnostic reformulation of the diffusion process that preserves input phase while randomizing magnitude, enabling structure-aligned generation without architectural changes or additional parameters. We further propose Frequency-Selective Structured (FSS) noise, which provides continuous control over structural rigidity via a single frequency-cutoff parameter. φ-PD adds no inference-time cost and is compatible with any diffusion model for images or videos. Across photorealistic and stylized re-rendering, as well as sim-to-real enhancement for driving planners, φ-PD produces controllable, spatially aligned results. When applied to the CARLA simulator, φ-PD improves CARLA-to-Waymo planner performance by 50\%. The method is complementary to existing conditioning approaches and broadly applicable to image-to-image and video-to-video generation. Videos, additional examples, and code are available on our \href{https://yuzeng-at-tri.github.io/ppd-page/}{project page}.
☆ EvoIR: Towards All-in-One Image Restoration via Evolutionary Frequency Modulation
Jiaqi Ma, Shengkai Hu, Jun Wan, Jiaxing Huang, Lefei Zhang, Salman Khan
All-in-One Image Restoration (AiOIR) tasks often involve diverse degradation that require robust and versatile strategies. However, most existing approaches typically lack explicit frequency modeling and rely on fixed or heuristic optimization schedules, which limit the generalization across heterogeneous degradation. To address these limitations, we propose EvoIR, an AiOIR-specific framework that introduces evolutionary frequency modulation for dynamic and adaptive image restoration. Specifically, EvoIR employs the Frequency-Modulated Module (FMM) that decomposes features into high- and low-frequency branches in an explicit manner and adaptively modulates them to enhance both structural fidelity and fine-grained details. Central to EvoIR, an Evolutionary Optimization Strategy (EOS) iteratively adjusts frequency-aware objectives through a population-based evolutionary process, dynamically balancing structural accuracy and perceptual fidelity. Its evolutionary guidance further mitigates gradient conflicts across degradation and accelerates convergence. By synergizing FMM and EOS, EvoIR yields greater improvements than using either component alone, underscoring their complementary roles. Extensive experiments on multiple benchmarks demonstrate that EvoIR outperforms state-of-the-art AiOIR methods.
☆ TV2TV: A Unified Framework for Inte rl eaved Language and Video Generation
Xiaochuang Han, Youssef Emad, Melissa Hall, John Nguyen, Karthik Padthe, Liam Robbins, Amir Bar, Delong Chen, Michal Drozdzal, Maha Elbayad, Yushi Hu, Shang-Wen Li, Sreya Dutta Roy, Jakob Verbeek, XuDong Wang, Marjan Ghazvininejad, Luke Zettlemoyer, Emily Dinan
Video generation models are rapidly advancing, but can still struggle with complex video outputs that require significant semantic branching or repeated high-level reasoning about what should happen next. In this paper, we introduce a new class of omni video-text models that integrate ideas from recent LM reasoning advances to address this challenge. More specifically, we present TV2TV, a unified generative modeling framework which decomposes video generation into an interleaved text and video generation process. TV2TV jointly learns language modeling (next-token prediction) and video flow matching (next-frame prediction) using a Mixture-of-Transformers (MoT) architecture. At inference time, TV2TV decides when to alternate between generating text and video frames, allowing the model to "think in words" about subsequent content before ``acting in pixels'' to produce frames. This design offloads much of the responsibility for deciding what should happen next to the language modeling tower, enabling improved visual quality and prompt alignment of generated videos. It also enables fine-grained controllability, allowing users to modify the video generation trajectory through text interventions at any point in the process. In controlled experiments on video game data, TV2TV demonstrates substantial improvements in both visual quality and controllability. TV2TV also scales to natural videos, as we show by augmenting sports videos with interleaved natural language action descriptions using vision-language models (VLMs). Training TV2TV on this corpus yields strong visual quality and prompt alignment, showcasing the model's ability to reason about and generate complex real-world action sequences. Together, these results highlight TV2TV as a promising step toward video generation with open-ended textual reasoning and control.
☆ SA-IQA: Redefining Image Quality Assessment for Spatial Aesthetics with Multi-Dimensional Rewards
Yuan Gao, Jin Song
In recent years, Image Quality Assessment (IQA) for AI-generated images (AIGI) has advanced rapidly; however, existing methods primarily target portraits and artistic images, lacking a systematic evaluation of interior scenes. We introduce Spatial Aesthetics, a paradigm that assesses the aesthetic quality of interior images along four dimensions: layout, harmony, lighting, and distortion. We construct SA-BENCH, the first benchmark for spatial aesthetics, comprising 18,000 images and 50,000 precise annotations. Employing SA-BENCH, we systematically evaluate current IQA methodologies and develop SA-IQA, through MLLM fine-tuning and a multidimensional fusion approach, as a comprehensive reward framework for assessing spatial aesthetics. We apply SA-IQA to two downstream tasks: (1) serving as a reward signal integrated with GRPO reinforcement learning to optimize the AIGC generation pipeline, and (2) Best-of-N selection to filter high-quality images and improve generation quality. Experiments indicate that SA-IQA significantly outperforms existing methods on SA-BENCH, setting a new standard for spatial aesthetics evaluation. Code and dataset will be open-sourced to advance research and applications in this domain.
★ From Generated Human Videos to Physically Plausible Robot Trajectories
James Ni, Zekai Wang, Wei Lin, Amir Bar, Yann LeCun , Trevor Darrell, Jitendra Malik, Roei Herzig
Video generation models are rapidly improving in their ability to synthesize human actions in novel contexts, holding the potential to serve as high-level planners for contextual robot control. To realize this potential, a key research question remains open: how can a humanoid execute the human actions from generated videos in a zero-shot manner? This challenge arises because generated videos are often noisy and exhibit morphological distortions that make direct imitation difficult compared to real video. To address this, we introduce a two-stage pipeline. First, we lift video pixels into a 4D human representation and then retarget to the humanoid morphology. Second, we propose GenMimic-a physics-aware reinforcement learning policy conditioned on 3D keypoints, and trained with symmetry regularization and keypoint-weighted tracking rewards. As a result, GenMimic can mimic human actions from noisy, generated videos. We curate GenMimicBench, a synthetic human-motion dataset generated using two video generation models across a spectrum of actions and contexts, establishing a benchmark for assessing zero-shot generalization and policy robustness. Extensive experiments demonstrate improvements over strong baselines in simulation and confirm coherent, physically stable motion tracking on a Unitree G1 humanoid robot without fine-tuning. This work offers a promising path to realizing the potential of video generation models as high-level policies for robot control.
comment : For project website, see https://genmimic.github.io
☆ Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmark
Haobo Yuan, Yueyi Sun, Yanwei Li, Tao Zhang, Xueqing Deng, Henghui Ding, Lu Qi, Anran Wang, Xiangtai Li, Ming-Hsuan Yang
Recent advances in Multimodal Large Language Models (MLLMs) have significantly improved performance on tasks such as visual grounding and visual question answering. However, the reasoning processes of these models remain largely opaque; they typically output only final predictions without revealing the intermediate steps or fine-grained evidence (e.g., pixels, locations) that lead to the result. This contrasts with human intelligence, which naturally operates through a chain of visual reasoning. To address this limitation, we introduce the Visual Reasoning Tracer (VRT) task, which requires models to not only localize the target object but also explicitly predict the intermediate objects that form the reasoning path. To advance research in this area, we contribute: (1) VRT-Bench, a human-annotated benchmark for evaluating visual reasoning; (2) a new metric for assessing the quality of reasoning traces; and (3) VRT-80k, a large-scale dataset for reasoning model training. Our experiments reveal that while existing models often produce the correct final output, they struggle to ground their intermediate reasoning. In contrast, models trained on VRT-80k achieve substantial improvements in tracing the reasoning path.
comment : Technical Report; Project Page: https://harboryuan.github.io/visual-reasoning-tracer
☆ Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression
Jung Yi, Wooseok Jang, Paul Hyunbin Cho, Jisu Nam, Heeji Yoon, Seungryong Kim
Recent advances in autoregressive video diffusion have enabled real-time frame streaming, yet existing solutions still suffer from temporal repetition, drift, and motion deceleration. We find that naively applying StreamingLLM-style attention sinks to video diffusion leads to fidelity degradation and motion stagnation. To overcome this, we introduce Deep Forcing, which consists of two training-free mechanisms that address this without any fine-tuning. Specifically, 1) Deep Sink dedicates half of the sliding window to persistent sink tokens and re-aligns their temporal RoPE phase to the current timeline, stabilizing global context during long rollouts. 2) Participative Compression performs importance-aware KV cache pruning that preserves only tokens actively participating in recent attention while safely discarding redundant and degraded history, minimizing error accumulation under out-of-distribution length generation. Together, these components enable over 12x extrapolation (e.g. 5s-trained to 60s+ generation) with better imaging quality than LongLive, better aesthetic quality than RollingForcing, almost maintaining overall consistency, and substantial gains in dynamic degree, all while maintaining real-time generation. Our results demonstrate that training-free KV-cache management can match or exceed training-based approaches for autoregressively streaming long-video generation.
comment : Project Page: https://cvlab-kaist.github.io/DeepForcing/
☆ Object Reconstruction under Occlusion with Generative Priors and Contact-induced Constraints
Minghan Zhu, Zhiyi Wang, Qihang Sun, Maani Ghaffari, Michael Posa
Object geometry is key information for robot manipulation. Yet, object reconstruction is a challenging task because cameras only capture partial observations of objects, especially when occlusion occurs. In this paper, we leverage two extra sources of information to reduce the ambiguity of vision signals. First, generative models learn priors of the shapes of commonly seen objects, allowing us to make reasonable guesses of the unseen part of geometry. Second, contact information, which can be obtained from videos and physical interactions, provides sparse constraints on the boundary of the geometry. We combine the two sources of information through contact-guided 3D generation. The guidance formulation is inspired by drag-based editing in generative models. Experiments on synthetic and real-world data show that our approach improves the reconstruction compared to pure 3D generation and contact-based optimization.
comment : Project page: https://contactgen3d.github.io/
☆ BulletTime: Decoupled Control of Time and Camera Pose for Video Generation
Yiming Wang, Qihang Zhang, Shengqu Cai, Tong Wu, Jan Ackermann, Zhengfei Kuang, Yang Zheng, Frano Rajič, Siyu Tang, Gordon Wetzstein
Emerging video diffusion models achieve high visual fidelity but fundamentally couple scene dynamics with camera motion, limiting their ability to provide precise spatial and temporal control. We introduce a 4D-controllable video diffusion framework that explicitly decouples scene dynamics from camera pose, enabling fine-grained manipulation of both scene dynamics and camera viewpoint. Our framework takes continuous world-time sequences and camera trajectories as conditioning inputs, injecting them into the video diffusion model through a 4D positional encoding in the attention layer and adaptive normalizations for feature modulation. To train this model, we curate a unique dataset in which temporal and camera variations are independently parameterized; this dataset will be made public. Experiments show that our model achieves robust real-world 4D control across diverse timing patterns and camera trajectories, while preserving high generation quality and outperforming prior work in controllability. See our website for video results: https://19reborn.github.io/Bullet4D/
comment : Project Page: https://19reborn.github.io/Bullet4D/
☆ 4DLangVGGT: 4D Language-Visual Geometry Grounded Transformer
Xianfeng Wu, Yajing Bai, Minghan Li, Xianzu Wu, Xueqi Zhao, Zhongyuan Lai, Wenyu Liu, Xinggang Wang
Constructing 4D language fields is crucial for embodied AI, augmented/virtual reality, and 4D scene understanding, as they provide enriched semantic representations of dynamic environments and enable open-vocabulary querying in complex scenarios. However, existing approaches to 4D semantic field construction primarily rely on scene-specific Gaussian splatting, which requires per-scene optimization, exhibits limited generalization, and is difficult to scale to real-world applications. To address these limitations, we propose 4DLangVGGT, the first Transformer-based feed-forward unified framework for 4D language grounding, that jointly integrates geometric perception and language alignment within a single architecture. 4DLangVGGT has two key components: the 4D Visual Geometry Transformer, StreamVGGT, which captures spatio-temporal geometric representations of dynamic scenes; and the Semantic Bridging Decoder (SBD), which projects geometry-aware features into a language-aligned semantic space, thereby enhancing semantic interpretability while preserving structural fidelity. Unlike prior methods that depend on costly per-scene optimization, 4DLangVGGT can be jointly trained across multiple dynamic scenes and directly applied during inference, achieving both deployment efficiency and strong generalization. This design significantly improves the practicality of large-scale deployment and establishes a new paradigm for open-vocabulary 4D scene understanding. Experiments on HyperNeRF and Neu3D datasets demonstrate that our approach not only generalizes effectively but also achieves state-of-the-art performance, achieving up to 2% gains under per-scene training and 1% improvements under multi-scene training. Our code released in https://github.com/hustvl/4DLangVGGT
comment : Code: https://github.com/hustvl/4DLangVGGT, Webpage: https://hustvl.github.io/4DLangVGGT
☆ Joint 3D Geometry Reconstruction and Motion Generation for 4D Synthesis from a Single Image
Yanran Zhang, Ziyi Wang, Wenzhao Zheng, Zheng Zhu, Jie Zhou, Jiwen Lu
Generating interactive and dynamic 4D scenes from a single static image remains a core challenge. Most existing generate-then-reconstruct and reconstruct-then-generate methods decouple geometry from motion, causing spatiotemporal inconsistencies and poor generalization. To address these, we extend the reconstruct-then-generate framework to jointly perform Motion generation and geometric Reconstruction for 4D Synthesis (MoRe4D). We first introduce TrajScene-60K, a large-scale dataset of 60,000 video samples with dense point trajectories, addressing the scarcity of high-quality 4D scene data. Based on this, we propose a diffusion-based 4D Scene Trajectory Generator (4D-STraG) to jointly generate geometrically consistent and motion-plausible 4D point trajectories. To leverage single-view priors, we design a depth-guided motion normalization strategy and a motion-aware module for effective geometry and dynamics integration. We then propose a 4D View Synthesis Module (4D-ViSM) to render videos with arbitrary camera trajectories from 4D point track representations. Experiments show that MoRe4D generates high-quality 4D scenes with multi-view consistency and rich dynamic details from a single image. Code: https://github.com/Zhangyr2022/MoRe4D.
comment : 18 Pages
☆ Semantic-Guided Two-Stage GAN for Face Inpainting with Hybrid Perceptual Encoding CVPR-2025
Abhigyan Bhattacharya, Hiranmoy Roy
Facial Image inpainting aim is to restore the missing or corrupted regions in face images while preserving identity, structural consistency and photorealistic image quality, a task specifically created for photo restoration. Though there are recent lot of advances in deep generative models, existing methods face problems with large irregular masks, often producing blurry textures on the edges of the masked region, semantic inconsistencies, or unconvincing facial structures due to direct pixel level synthesis approach and limited exploitation of facial priors. In this paper we propose a novel architecture, which address these above challenges through semantic-guided hierarchical synthesis. Our approach starts with a method that organizes and synthesizes information based on meaning, followed by refining the texture. This process gives clear insights into the facial structure before we move on to creating detailed images. In the first stage, we blend two techniques: one that focuses on local features with CNNs and global features with Vision Transformers. This helped us create clear and detailed semantic layouts. In the second stage, we use a Multi-Modal Texture Generator to refine these layouts by pulling in information from different scales, ensuring everything looks cohesive and consistent. The architecture naturally handles arbitrary mask configurations through dynamic attention without maskspecific training. Experiment on two datasets CelebA-HQ and FFHQ shows that our model outperforms other state-of-the-art methods, showing improvements in metrics like LPIPS, PSNR, and SSIM. It produces visually striking results with better semantic preservation, in challenging large-area inpainting situations.
comment : Submitted for review CVPR-2025
☆ RAMEN: Resolution-Adjustable Multimodal Encoder for Earth Observation
Nicolas Houdré, Diego Marcos, Hugo Riffaud de Turckheim, Dino Ienco, Laurent Wendling, Camille Kurtz, Sylvain Lobry
Earth observation (EO) data spans a wide range of spatial, spectral, and temporal resolutions, from high-resolution optical imagery to low resolution multispectral products or radar time series. While recent foundation models have improved multimodal integration for learning meaningful representations, they often expect fixed input resolutions or are based on sensor-specific encoders limiting generalization across heterogeneous EO modalities. To overcome these limitations we introduce RAMEN, a resolution-adjustable multimodal encoder that learns a shared visual representation across EO data in a fully sensor-agnostic manner. RAMEN treats the modality and spatial and temporal resolutions as key input data features, enabling coherent analysis across modalities within a unified latent space. Its main methodological contribution is to define spatial resolution as a controllable output parameter, giving users direct control over the desired level of detail at inference and allowing explicit trade-offs between spatial precision and computational cost. We train a single, unified transformer encoder reconstructing masked multimodal EO data drawn from diverse sources, ensuring generalization across sensors and resolutions. Once pretrained, RAMEN transfers effectively to both known and unseen sensor configurations and outperforms larger state-of-the-art models on the community-standard PANGAEA benchmark, containing various multi-sensor and multi-resolution downstream tasks. Our code and pretrained model are available at https://github.com/nicolashoudre/RAMEN.
☆ HTR-ConvText: Leveraging Convolution and Textual Information for Handwritten Text Recognition
Pham Thach Thanh Truc, Dang Hoai Nam, Huynh Tong Dang Khoa, Vo Nguyen Le Duy
Handwritten Text Recognition remains challenging due to the limited data, high writing style variance, and scripts with complex diacritics. Existing approaches, though partially address these issues, often struggle to generalize without massive synthetic data. To address these challenges, we propose HTR-ConvText, a model designed to capture fine-grained, stroke-level local features while preserving global contextual dependencies. In the feature extraction stage, we integrate a residual Convolutional Neural Network backbone with a MobileViT with Positional Encoding block. This enables the model to both capture structural patterns and learn subtle writing details. We then introduce the ConvText encoder, a hybrid architecture combining global context and local features within a hierarchical structure that reduces sequence length for improved efficiency. Additionally, an auxiliary module injects textual context to mitigate the weakness of Connectionist Temporal Classification. Evaluations on IAM, READ2016, LAM and HANDS-VNOnDB demonstrate that our approach achieves improved performance and better generalization compared to existing methods, especially in scenarios with limited training samples and high handwriting diversity.
☆ Generative Neural Video Compression via Video Diffusion Prior
Qi Mao, Hao Cheng, Tinghan Yang, Libiao Jin, Siwei Ma
We present GNVC-VD, the first DiT-based generative neural video compression framework built upon an advanced video generation foundation model, where spatio-temporal latent compression and sequence-level generative refinement are unified within a single codec. Existing perceptual codecs primarily rely on pre-trained image generative priors to restore high-frequency details, but their frame-wise nature lacks temporal modeling and inevitably leads to perceptual flickering. To address this, GNVC-VD introduces a unified flow-matching latent refinement module that leverages a video diffusion transformer to jointly enhance intra- and inter-frame latents through sequence-level denoising, ensuring consistent spatio-temporal details. Instead of denoising from pure Gaussian noise as in video generation, GNVC-VD initializes refinement from decoded spatio-temporal latents and learns a correction term that adapts the diffusion prior to compression-induced degradation. A conditioning adaptor further injects compression-aware cues into intermediate DiT layers, enabling effective artifact removal while maintaining temporal coherence under extreme bitrate constraints. Extensive experiments show that GNVC-VD surpasses both traditional and learned codecs in perceptual quality and significantly reduces the flickering artifacts that persist in prior generative approaches, even below 0.01 bpp, highlighting the promise of integrating video-native generative priors into neural codecs for next-generation perceptual video compression.
☆ Self-Supervised Learning for Transparent Object Depth Completion Using Depth from Non-Transparent Objects
Xianghui Fan, Zhaoyu Chen, Mengyang Pan, Anping Deng, Hang Yang
The perception of transparent objects is one of the well-known challenges in computer vision. Conventional depth sensors have difficulty in sensing the depth of transparent objects due to refraction and reflection of light. Previous research has typically train a neural network to complete the depth acquired by the sensor, and this method can quickly and accurately acquire accurate depth maps of transparent objects. However, previous training relies on a large amount of annotation data for supervision, and the labeling of depth maps is costly. To tackle this challenge, we propose a new self-supervised method for training depth completion networks. Our method simulates the depth deficits of transparent objects within non-transparent regions and utilizes the original depth map as ground truth for supervision. Experiments demonstrate that our method achieves performance comparable to supervised approach, and pre-training with our method can improve the model performance when the training samples are small.
comment : conference
☆ Reflection Removal through Efficient Adaptation of Diffusion Transformer s
Daniyar Zakarin, Thiemo Wandel, Anton Obukhov, Dengxin Dai
We introduce a diffusion-transformer (DiT) framework for single-image reflection removal that leverages the generalization strengths of foundation diffusion models in the restoration setting. Rather than relying on task-specific architectures, we repurpose a pre-trained DiT-based foundation model by conditioning it on reflection-contaminated inputs and guiding it toward clean transmission layers. We systematically analyze existing reflection removal data sources for diversity, scalability, and photorealism. To address the shortage of suitable data, we construct a physically based rendering (PBR) pipeline in Blender, built around the Principled BSDF, to synthesize realistic glass materials and reflection effects. Efficient LoRA-based adaptation of the foundation model, combined with the proposed synthetic data, achieves state-of-the-art performance on in-domain and zero-shot benchmarks. These results demonstrate that pretrained diffusion transformers, when paired with physically grounded data synthesis and efficient adaptation, offer a scalable and high-fidelity solution for reflection removal. Project page: https://hf.co/spaces/huawei-bayerlab/windowseat-reflection-removal-web
☆ A dynamic memory assignment strategy for dilation-based ICP algorithm on embedded GPUs
Qiong Chang, Weimin Wang, Junpei Zhong, Jun Miyazaki
This paper proposes a memory-efficient optimization strategy for the high-performance point cloud registration algorithm VANICP, enabling lightweight execution on embedded GPUs with constrained hardware resources. VANICP is a recently published acceleration framework that significantly improves the computational efficiency of point-cloud-based applications. By transforming the global nearest neighbor search into a localized process through a dilation-based information propagation mechanism, VANICP greatly reduces the computational complexity of the NNS. However, its original implementation demands a considerable amount of memory, which restricts its deployment in resource-constrained environments such as embedded systems. To address this issue, we propose a GPU-oriented dynamic memory assignment strategy that optimizes the memory usage of the dilation operation. Furthermore, based on this strategy, we construct an enhanced version of the VANICP framework that achieves over 97% reduction in memory consumption while preserving the original performance. Source code is published on: https://github.com/changqiong/VANICP4Em.git.
☆ Aligned but Stereotypical? The Hidden Influence of System Prompt s on Social Bias in LVLM-Based Text-to-Image Models
NaHyeon Park, Namin An, Kunhee Kim, Soyeon Yoon, Jiahao Huo, Hyunjung Shim
Large vision-language model (LVLM) based text-to-image (T2I) systems have become the dominant paradigm in image generation, yet whether they amplify social biases remains insufficiently understood. In this paper, we show that LVLM-based models produce markedly more socially biased images than non-LVLM-based models. We introduce a 1,024 prompt benchmark spanning four levels of linguistic complexity and evaluate demographic bias across multiple attributes in a systematic manner. Our analysis identifies system prompts, the predefined instructions guiding LVLMs, as a primary driver of biased behavior. Through decoded intermediate representations, token-probability diagnostics, and embedding-association analyses, we reveal how system prompts encode demographic priors that propagate into image synthesis. To this end, we propose FairPro, a training-free meta-prompting framework that enables LVLMs to self-audit and construct fairness-aware system prompts at test time. Experiments on two LVLM-based T2I models, SANA and Qwen-Image, show that FairPro substantially reduces demographic bias while preserving text-image alignment. We believe our findings provide deeper insight into the central role of system prompts in bias propagation and offer a practical, deployable approach for building more socially responsible T2I systems.
comment : Project page: https://fairpro-t2i.github.io
☆ Stable Single-Pixel Contrastive Learning for Semantic and Geometric Tasks
Leonid Pogorelyuk, Niels Bracher, Aaron Verkleeren, Lars Kühmichel, Stefan T. Radev
We pilot a family of stable contrastive losses for learning pixel-level representations that jointly capture semantic and geometric information. Our approach maps each pixel of an image to an overcomplete descriptor that is both view-invariant and semantically meaningful. It enables precise point-correspondence across images without requiring momentum-based teacher-student training. Two experiments in synthetic 2D and 3D environments demonstrate the properties of our loss and the resulting overcomplete representations.
comment : UniReps Workshop 2025, 12 pages, 8 figures
☆ Rethinking the Use of Vision Transformer s for AI-Generated Image Detection
NaHyeon Park, Kunhee Kim, Junsuk Choe, Hyunjung Shim
Rich feature representations derived from CLIP-ViT have been widely utilized in AI-generated image detection. While most existing methods primarily leverage features from the final layer, we systematically analyze the contributions of layer-wise features to this task. Our study reveals that earlier layers provide more localized and generalizable features, often surpassing the performance of final-layer features in detection tasks. Moreover, we find that different layers capture distinct aspects of the data, each contributing uniquely to AI-generated image detection. Motivated by these findings, we introduce a novel adaptive method, termed MoLD, which dynamically integrates features from multiple ViT layers using a gating-based mechanism. Extensive experiments on both GAN- and diffusion-generated images demonstrate that MoLD significantly improves detection performance, enhances generalization across diverse generative models, and exhibits robustness in real-world scenarios. Finally, we illustrate the scalability and versatility of our approach by successfully applying it to other pre-trained ViTs, such as DINOv2.
comment : Code: https://github.com/nahyeonkaty/mold
☆ Balanced Few-Shot Episodic Learning for Accurate Retinal Disease Diagnosis
Jasmaine Khale, Ravi Prakash Srivastava
Automated retinal disease diagnosis is vital given the rising prevalence of conditions such as diabetic retinopathy and macular degeneration. Conventional deep learning approaches require large annotated datasets, which are costly and often imbalanced across disease categories, limiting their reliability in practice. Few-shot learning (FSL) addresses this challenge by enabling models to generalize from only a few labeled samples per class. In this study,we propose a balanced few-shot episodic learning framework tailored to the Retinal Fundus Multi-Disease Image Dataset (RFMiD). Focusing on the ten most represented classes, which still show substantial imbalance between majority diseases (e.g., Diabetic Retinopathy, Macular Hole) and minority ones (e.g., Optic Disc Edema, Branch Retinal Vein Occlusion), our method integrates three key components: (i) balanced episodic sampling, ensuring equal participation of all classes in each 5-way 5-shot episode; (ii) targeted augmentation, including Contrast Limited Adaptive Histogram Equalization (CLAHE) and color/geometry transformations, to improve minority-class di- versity; and (iii) a ResNet-50 encoder pretrained on ImageNet, selected for its superior ability to capture fine-grained retinal features. Prototypes are computed in the embedding space and classification is performed with cosine similarity for improved stability. Trained on 100 episodes and evaluated on 1,000 test episodes, our framework achieves substantial accuracy gains and reduces bias toward majority classes, with notable improvements for underrepresented diseases. These results demonstrate that dataset-aware few-shot pipelines, combined with balanced sampling and CLAHE-enhanced preprocessing, can deliver more robust and clinically fair retinal disease diagnosis under data-constrained conditions.
☆ GeoPE:A Unified Geometric Positional Embedding for Structured Tensors
Yupu Yao, Bowen Yang
Standard Vision Transformers flatten 2D images into 1D sequences, disrupting the natural spatial topology. While Rotary Positional Embedding (RoPE) excels in 1D, it inherits this limitation, often treating spatially distant patches (e.g., at row edges) as sequence neighbors. Existing 2D approaches typically treat spatial axes independently, failing to decouple this false sequential proximity from true spatial distance. To restore the 2D spatial manifold, we introduce Geometric Positional Embedding (GeoPE), a framework that extends rotations to 3D Euclidean space using quaternions. To overcome non-commutativity and ensure symmetry, GeoPE constructs a unified rotational operator by computing the geometric mean in the Lie algebra. This creates a geometrically coupled encoding that effectively separates spatial dimensions. Extensive experiments on image classification, object detection, and 3D semantic segmentation demonstrate that GeoPE consistently outperforms existing 2D RoPE variants and significantly enhances shape bias, confirming its ability to capture true geometric structure.
☆ FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via neural Action Tokenization
Yicheng Liu, Shiduo Zhang, Zibin Dong, Baijun Ye, Tianyuan Yuan, Xiaopeng Yu, Linqi Yin, Chenhao Lu, Junhao Shi, Luca Jiang-Tao Yu, Liangtao Zheng, Tao Jiang, Jingjing Gong, Xipeng Qiu, Hang Zhao
Autoregressive vision-language-action (VLA) models have recently demonstrated strong capabilities in robotic manipulation. However, their core process of action tokenization often involves a trade-off between reconstruction fidelity and inference efficiency. We introduce FASTer, a unified framework for efficient and generalizable robot learning that integrates a learnable tokenizer with an autoregressive policy built upon it. FASTerVQ encodes action chunks as single-channel images, capturing global spatio-temporal dependencies while maintaining a high compression ratio. FASTerVLA builds on this tokenizer with block-wise autoregressive decoding and a lightweight action expert, achieving both faster inference and higher task performance. Extensive experiments across simulated and real-world benchmarks show that FASTerVQ delivers superior reconstruction quality, high token utilization, and strong cross-task and cross-embodiment generalization, while FASTerVLA further improves overall capability, surpassing previous state-of-the-art VLA models in both inference speed and task performance.
☆ Towards Adaptive Fusion of Multimodal Deep Networks for Human Action Recognition
Novanto Yudistira
This study introduces a pioneering methodology for human action recognition by harnessing deep neural network techniques and adaptive fusion strategies across multiple modalities, including RGB, optical flows, audio, and depth information. Employing gating mechanisms for multimodal fusion, we aim to surpass limitations inherent in traditional unimodal recognition methods while exploring novel possibilities for diverse applications. Through an exhaustive investigation of gating mechanisms and adaptive weighting-based fusion architectures, our methodology enables the selective integration of relevant information from various modalities, thereby bolstering both accuracy and robustness in action recognition tasks. We meticulously examine various gated fusion strategies to pinpoint the most effective approach for multimodal action recognition, showcasing its superiority over conventional unimodal methods. Gating mechanisms facilitate the extraction of pivotal features, resulting in a more holistic representation of actions and substantial enhancements in recognition performance. Our evaluations across human action recognition, violence action detection, and multiple self-supervised learning tasks on benchmark datasets demonstrate promising advancements in accuracy. The significance of this research lies in its potential to revolutionize action recognition systems across diverse fields. The fusion of multimodal information promises sophisticated applications in surveillance and human-computer interaction, especially in contexts related to active assisted living.
☆ LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging
Zhijian Shu, Cheng Lin, Tao Xie, Wei Yin, Ben Li, Zhiyuan Pu, Weize Li, Yao Yao, Xun Cao, Xiaoyang Guo, Xiao-Xiao Long
3D vision foundation models like Visual Geometry Grounded Transformer (VGGT) have advanced greatly in geometric perception. However, it is time-consuming and memory-intensive for long sequences, limiting application to large-scale scenes beyond hundreds of images. To address this, we propose LiteVGGT, achieving up to 10x speedup and substantial memory reduction, enabling efficient processing of 1000-image scenes. We derive two key insights for 3D reconstruction: (1) tokens from local image regions have inherent geometric correlations, leading to high similarity and computational redundancy; (2) token similarity across adjacent network layers remains stable, allowing for reusable merge decisions. Guided by these, we design a simple yet efficient strategy, dubbed geometry-aware cached token merging. We analyze each token's geometric importance, optimizing anchor token selection to better preserve key information for reconstruction. We also cache and reuse merge indices across layers, substantially reducing latency with minimal accuracy impact. This strategy retains VGGT's core performance, enabling efficient fine-tuning and FP8 quantization for further gains. Extensive experiments validate LiteVGGT's effectiveness, scalability, and robustness. Project page: https://garlicba.github.io/LiteVGGT/
☆ Virtually Unrolling the Herculaneum Papyri by Diffeomorphic Spiral Fitting WACV 2026
Paul Henderson
The Herculaneum Papyri are a collection of rolled papyrus documents that were charred and buried by the famous eruption of Mount Vesuvius. They promise to contain a wealth of previously unseen Greek and Latin texts, but are extremely fragile and thus most cannot be unrolled physically. A solution to access these texts is virtual unrolling, where the papyrus surface is digitally traced out in a CT scan of the scroll, to create a flattened representation. This tracing is very laborious to do manually in gigavoxel-sized scans, so automated approaches are desirable. We present the first top-down method that automatically fits a surface model to a CT scan of a severely damaged scroll. We take a novel approach that globally fits an explicit parametric model of the deformed scroll to existing neural network predictions of where the rolled papyrus likely passes. Our method guarantees the resulting surface is a single continuous 2D sheet, even passing through regions where the surface is not detectable in the CT scan. We conduct comprehensive experiments on high-resolution CT scans of two scrolls, showing that our approach successfully unrolls large regions, and exceeds the performance of the only existing automated unrolling method suitable for this data.
comment : Accepted at WACV 2026
☆ Semantics Lead the Way: Harmonizing Semantic and Texture Modeling with Asynchronous Latent Diffusion
Yueming Pan, Ruoyu Feng, Qi Dai, Yuqi Wang, Wenfeng Lin, Mingyu Guo, Chong Luo, Nanning Zheng
Latent Diffusion Models (LDMs) inherently follow a coarse-to-fine generation process, where high-level semantic structure is generated slightly earlier than fine-grained texture. This indicates the preceding semantics potentially benefit texture generation by providing a semantic anchor. Recent advances have integrated semantic priors from pretrained visual encoders to further enhance LDMs, yet they still denoise semantic and VAE-encoded texture synchronously, neglecting such ordering. Observing these, we propose Semantic-First Diffusion (SFD), a latent diffusion paradigm that explicitly prioritizes semantic formation. SFD first constructs composite latents by combining a compact semantic latent, which is extracted from a pretrained visual encoder via a dedicated Semantic VAE, with the texture latent. The core of SFD is to denoise the semantic and texture latents asynchronously using separate noise schedules: semantics precede textures by a temporal offset, providing clearer high-level guidance for texture refinement and enabling natural coarse-to-fine generation. On ImageNet 256x256 with guidance, SFD achieves FID 1.06 (LightningDiT-XL) and FID 1.04 (1.0B LightningDiT-XXL), while achieving up to 100x faster convergence than the original DiT. SFD also improves existing methods like ReDi and VA-VAE, demonstrating the effectiveness of asynchronous, semantics-led modeling. Project page and code: https://yuemingpan.github.io/SFD.github.io/.
☆ ReflexFlow: Rethinking Learning Objective for Exposure Bias Alleviation in Flow Matching
Guanbo Huang, Jingjia Mao, Fanding Huang, Fengkai Liu, Xiangyang Luo, Yaoyuan Liang, Jiasheng Lu, Xiaoe Wang, Pei Liu, Ruiliu Fu, Shao-Lun Huang
Despite tremendous recent progress, Flow Matching methods still suffer from exposure bias due to discrepancies in training and inference. This paper investigates the root causes of exposure bias in Flow Matching, including: (1) the model lacks generalization to biased inputs during training, and (2) insufficient low-frequency content captured during early denoising, leading to accumulated bias. Based on these insights, we propose ReflexFlow, a simple and effective reflexive refinement of the Flow Matching learning objective that dynamically corrects exposure bias. ReflexFlow consists of two components: (1) Anti-Drift Rectification (ADR), which reflexively adjusts prediction targets for biased inputs utilizing a redesigned loss under training-time scheduled sampling; and (2) Frequency Compensation (FC), which reflects on missing low-frequency components and compensates them by reweighting the loss using exposure bias. ReflexFlow is model-agnostic, compatible with all Flow Matching frameworks, and improves generation quality across datasets. Experiments on CIFAR-10, CelebA-64, and ImageNet-256 show that ReflexFlow outperforms prior approaches in mitigating exposure bias, achieving a 35.65% reduction in FID on CelebA-64.
☆ Equivariant Symmetry-Aware Head Pose Estimation for Fetal MRI
Ramya Muthukrishnan, Borjan Gagoski, Aryn Lee, P. Ellen Grant, Elfar Adalsteinsson, Polina Golland, Benjamin Billot
We present E(3)-Pose, a novel fast pose estimation method that jointly and explicitly models rotation equivariance and object symmetry. Our work is motivated by the challenging problem of accounting for fetal head motion during a diagnostic MRI scan. We aim to enable automatic adaptive prescription of 2D diagnostic MRI slices with 6-DoF head pose estimation, supported by 3D MRI volumes rapidly acquired before each 2D slice. Existing methods struggle to generalize to clinical volumes, due to pose ambiguities induced by inherent anatomical symmetries, as well as low resolution, noise, and artifacts. In contrast, E(3)-Pose captures anatomical symmetries and rigid pose equivariance by construction, and yields robust estimates of the fetal head pose. Our experiments on publicly available and representative clinical fetal MRI datasets demonstrate the superior robustness and generalization of our method across domains. Crucially, E(3)-Pose achieves state-of-the-art accuracy on clinical MRI volumes, paving the way for clinical translation. Our implementation is available at github.com/ramyamut/E3-Pose.
☆ You Only Train Once (YOTO): A Retraining-Free Object Detection Framework
Priyanto Hidayatullah, Nurjannah Syakrani, Yudi Widhiyasana, Muhammad Rizqi Sholahuddin, Refdinal Tubagus, Zahri Al Adzani Hidayat, Hanri Fajar Ramadhan, Dafa Alfarizki Pratama, Farhan Muhammad Yasin
Object detection constitutes the primary task within the domain of computer vision. It is utilized in numerous domains. Nonetheless, object detection continues to encounter the issue of catastrophic forgetting. The model must be retrained whenever new products are introduced, utilizing not only the new products dataset but also the entirety of the previous dataset. The outcome is obvious: increasing model training expenses and significant time consumption. In numerous sectors, particularly retail checkout, the frequent introduction of new products presents a great challenge. This study introduces You Only Train Once (YOTO), a methodology designed to address the issue of catastrophic forgetting by integrating YOLO11n for object localization with DeIT and Proxy Anchor Loss for feature extraction and metric learning. For classification, we utilize cosine similarity between the embedding features of the target product and those in the Qdrant vector database. In a case study conducted in a retail store with 140 products, the experimental results demonstrate that our proposed framework achieves encouraging accuracy, whether for detecting new or existing products. Furthermore, without retraining, the training duration difference is significant. We achieve almost 3 times the training time efficiency compared to classical object detection approaches. This efficiency escalates as additional new products are added to the product database. The average inference time is 580 ms per image containing multiple products, on an edge device, validating the proposed framework's feasibility for practical use.
comment : under review in the Elsevier Engineering Journal
☆ SDG-Track: A Heterogeneous Observer-Follower Framework for High-Resolution UAV Tracking on Embedded Platforms
Jiawen Wen, Yu Hu, Suixuan Qiu, Jinshan Huang, Xiaowen Chu
Real-time tracking of small unmanned aerial vehicles (UAVs) on edge devices faces a fundamental resolution-speed conflict. Downsampling high-resolution imagery to standard detector input sizes causes small target features to collapse below detectable thresholds. Yet processing native 1080p frames on resource-constrained platforms yields insufficient throughput for smooth gimbal control. We propose SDG-Track, a Sparse Detection-Guided Tracker that adopts an Observer-Follower architecture to reconcile this conflict. The Observer stream runs a high-capacity detector at low frequency on the GPU to provide accurate position anchors from 1920x1080 frames. The Follower stream performs high-frequency trajectory interpolation via ROI-constrained sparse optical flow on the CPU. To handle tracking failures from occlusion or model drift caused by spectrally similar distractors, we introduce Dual-Space Recovery, a training-free re-acquisition mechanism combining color histogram matching with geometric consistency constraints. Experiments on a ground-to-air tracking station demonstrate that SDG-Track achieves 35.1 FPS system throughput while retaining 97.2\% of the frame-by-frame detection precision. The system successfully tracks agile FPV drones under real-world operational conditions on an NVIDIA Jetson Orin Nano. Our paper code is publicly available at https://github.com/Jeffry-wen/SDG-Track
comment : https://github.com/Jeffry-wen/SDG-Track
☆ SP-Det: Self- Prompt ed Dual-Text Fusion for Generalized Multi-Label Lesion Detection
Qing Xu, Yanqian Wang, Xiangjian Hea, Yue Li, Yixuan Zhang, Rong Qu, Wenting Duan, Zhen Chen
Automated lesion detection in chest X-rays has demonstrated significant potential for improving clinical diagnosis by precisely localizing pathological abnormalities. While recent promptable detection frameworks have achieved remarkable accuracy in target localization, existing methods typically rely on manual annotations as prompts, which are labor-intensive and impractical for clinical applications. To address this limitation, we propose SP-Det, a novel self-prompted detection framework that automatically generates rich textual context to guide multi-label lesion detection without requiring expert annotations. Specifically, we introduce an expert-free dual-text prompt generator (DTPG) that leverages two complementary textual modalities: semantic context prompts that capture global pathological patterns and disease beacon prompts that focus on disease-specific manifestations. Moreover, we devise a bidirectional feature enhancer (BFE) that synergistically integrates comprehensive diagnostic context with disease-specific embeddings to significantly improve feature representation and detection accuracy. Extensive experiments on two chest X-ray datasets with diverse thoracic disease categories demonstrate that our SP-Det framework outperforms state-of-the-art detection methods while completely eliminating the dependency on expert-annotated prompts compared to existing promptable architectures.
☆ Contact-Aware Refinement of Human Pose Pseudo-Ground Truth via Bioimpedance Sensing ICCV 2025
Maria-Paola Forte, Nikos Athanasiou, Giulia Ballardini, Jan Ulrich Bartels, Katherine J. Kuchenbecker, Michael J. Black
Capturing accurate 3D human pose in the wild would provide valuable data for training pose estimation and motion generation methods. While video-based estimation approaches have become increasingly accurate, they often fail in common scenarios involving self-contact, such as a hand touching the face. In contrast, wearable bioimpedance sensing can cheaply and unobtrusively measure ground-truth skin-to-skin contact. Consequently, we propose a novel framework that combines visual pose estimators with bioimpedance sensing to capture the 3D pose of people by taking self-contact into account. Our method, BioTUCH, initializes the pose using an off-the-shelf estimator and introduces contact-aware pose optimization during measured self-contact: reprojection error and deviations from the input estimate are minimized while enforcing vertex proximity constraints. We validate our approach using a new dataset of synchronized RGB video, bioimpedance measurements, and 3D motion capture. Testing with three input pose estimators, we demonstrate an average of 11.7% improvement in reconstruction accuracy. We also present a miniature wearable bioimpedance sensor that enables efficient large-scale collection of contact-aware training data for improving pose estimation and generation using BioTUCH. Code and data are available at biotuch.is.tue.mpg.de
comment : * Equal contribution. Minor figure corrections compared to the ICCV 2025 version
☆ Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens
Ziran Qin, Youru Lv, Mingbao Lin, Zeren Zhang, Chanfan Gan, Tieyuan Chen, Weiyao Lin
Autoregressive (AR) visual generation has emerged as a powerful paradigm for image and multimodal synthesis, owing to its scalability and generality. However, existing AR image generation suffers from severe memory bottlenecks due to the need to cache all previously generated visual tokens during decoding, leading to both high storage requirements and low throughput. In this paper, we introduce \textbf{LineAR}, a novel, training-free progressive key-value (KV) cache compression pipeline for autoregressive image generation. By fully exploiting the intrinsic characteristics of visual attention, LineAR manages the cache at the line level using a 2D view, preserving the visual dependency regions while progressively evicting less-informative tokens that are harmless for subsequent line generation, guided by inter-line attention. LineAR enables efficient autoregressive (AR) image generation by utilizing only a few lines of cache, achieving both memory savings and throughput speedup, while maintaining or even improving generation quality. Extensive experiments across six autoregressive image generation models, including class-conditional and text-to-image generation, validate its effectiveness and generality. LineAR improves ImageNet FID from 2.77 to 2.68 and COCO FID from 23.85 to 22.86 on LlamaGen-XL and Janus-Pro-1B, while retaining only 1/6 KV cache. It also improves DPG on Lumina-mGPT-768 with just 1/8 KV cache. Additionally, LineAR achieves significant memory and throughput gains, including up to 67.61% memory reduction and 7.57x speedup on LlamaGen-XL, and 39.66% memory reduction and 5.62x speedup on Janus-Pro-7B.
☆ A Sanity Check for Multi-In-Domain Face Forgery Detection in the Real Wo rl d
Jikang Cheng, Renye Yan, Zhiyuan Yan, Yaozhong Gan, Xueyi Zhang, Zhongyuan Wang, Wei Peng, Ling Liang
Existing methods for deepfake detection aim to develop generalizable detectors. Although "generalizable" is the ultimate target once and for all, with limited training forgeries and domains, it appears idealistic to expect generalization that covers entirely unseen variations, especially given the diversity of real-world deepfakes. Therefore, introducing large-scale multi-domain data for training can be feasible and important for real-world applications. However, within such a multi-domain scenario, the differences between multiple domains, rather than the subtle real/fake distinctions, dominate the feature space. As a result, despite detectors being able to relatively separate real and fake within each domain (i.e., high AUC), they struggle with single-image real/fake judgments in domain-unspecified conditions (i.e., low ACC). In this paper, we first define a new research paradigm named Multi-In-Domain Face Forgery Detection (MID-FFD), which includes sufficient volumes of real-fake domains for training. Then, the detector should provide definitive real-fake judgments to the domain-unspecified inputs, which simulate the frame-by-frame independent detection scenario in the real world. Meanwhile, to address the domain-dominant issue, we propose a model-agnostic framework termed DevDet (Developer for Detector) to amplify real/fake differences and make them dominant in the feature space. DevDet consists of a Face Forgery Developer (FFDev) and a Dose-Adaptive detector Fine-Tuning strategy (DAFT). Experiments demonstrate our superiority in predicting real-fake under the MID-FFD scenario while maintaining original generalization ability to unseen data.
☆ Tokenizing Buildings: A Transformer for Layout Synthesis
Manuel Ladron de Guevara, Jinmo Rhee, Ardavan Bidgoli, Vaidas Razgaitis, Michael Bergin
We introduce Small Building Model (SBM), a Transformer-based architecture for layout synthesis in Building Information Modeling (BIM) scenes. We address the question of how to tokenize buildings by unifying heterogeneous feature sets of architectural elements into sequences while preserving compositional structure. Such feature sets are represented as a sparse attribute-feature matrix that captures room properties. We then design a unified embedding module that learns joint representations of categorical and possibly correlated continuous feature groups. Lastly, we train a single Transformer backbone in two modes: an encoder-only pathway that yields high-fidelity room embeddings, and an encoder-decoder pipeline for autoregressive prediction of room entities, referred to as Data-Driven Entity Prediction (DDEP). Experiments across retrieval and generative layout synthesis show that SBM learns compact room embeddings that reliably cluster by type and topology, enabling strong semantic retrieval. In DDEP mode, SBM produces functionally sound layouts, with fewer collisions and boundary violations and improved navigability.
comment : 8 pages, 1 page References, 4 figures
☆ FreeGen: Feed-Forward Reconstruction-Generation Co-Training for Free-Viewpoint Driving Scene Synthesis
Shijie Chen, Peixi Peng
Closed-loop simulation and scalable pre-training for autonomous driving require synthesizing free-viewpoint driving scenes. However, existing datasets and generative pipelines rarely provide consistent off-trajectory observations, limiting large-scale evaluation and training. While recent generative models demonstrate strong visual realism, they struggle to jointly achieve interpolation consistency and extrapolation realism without per-scene optimization. To address this, we propose FreeGen, a feed-forward reconstruction-generation co-training framework for free-viewpoint driving scene synthesis. The reconstruction model provides stable geometric representations to ensure interpolation consistency, while the generation model performs geometry-aware enhancement to improve realism at unseen viewpoints. Through co-training, generative priors are distilled into the reconstruction model to improve off-trajectory rendering, and the refined geometry in turn offers stronger structural guidance for generation. Experiments demonstrate that FreeGen achieves state-of-the-art performance for free-viewpoint driving scene synthesis.
comment : Novel View Synthesis, Driving Scene, Free Trajectory, Image Generation
☆ LatentFM: A Latent Flow Matching Approach for Generative Medical Image Segmentation
Huynh Trinh Ngoc, Hoang Anh Nguyen Kim, Toan Nguyen Hai, Long Tran Quoc
Generative models have achieved remarkable progress with the emergence of flow matching (FM). It has demonstrated strong generative capabilities and attracted significant attention as a simulation-free flow-based framework capable of learning exact data densities. Motivated by these advances, we propose LatentFM, a flow-based model operating in the latent space for medical image segmentation. To model the data distribution, we first design two variational autoencoders (VAEs) to encode both medical images and their corresponding masks into a lower-dimensional latent space. We then estimate a conditional velocity field that guides the flow based on the input image. By sampling multiple latent representations, our method synthesizes diverse segmentation outputs whose pixel-wise variance reliably captures the underlying data distribution, enabling both highly accurate and uncertainty-aware predictions. Furthermore, we generate confidence maps that quantify the model certainty, providing clinicians with richer information for deeper analysis. We conduct experiments on two datasets, ISIC-2018 and CVC-Clinic, and compare our method with several prior baselines, including both deterministic and generative approach models. Through comprehensive evaluations, both qualitative and quantitative results show that our approach achieves superior segmentation accuracy while remaining highly efficient in the latent space.
☆ RobustSplat++: Decoupling Densification, Dynamics, and Illumination for In-the-Wild 3DGS
Chuanyu Fu, Guanying Chen, Yuqi Zhang, Kunbin Yao, Yuan Xiong, Chuan Huang, Shuguang Cui, Yasuyuki Matsushita, Xiaochun Cao
3D Gaussian Splatting (3DGS) has gained significant attention for its real-time, photo-realistic rendering in novel-view synthesis and 3D modeling. However, existing methods struggle with accurately modeling in-the-wild scenes affected by transient objects and illuminations, leading to artifacts in the rendered images. We identify that the Gaussian densification process, while enhancing scene detail capture, unintentionally contributes to these artifacts by growing additional Gaussians that model transient disturbances and illumination variations. To address this, we propose RobustSplat++, a robust solution based on several critical designs. First, we introduce a delayed Gaussian growth strategy that prioritizes optimizing static scene structure before allowing Gaussian splitting/cloning, mitigating overfitting to transient objects in early optimization. Second, we design a scale-cascaded mask bootstrapping approach that first leverages lower-resolution feature similarity supervision for reliable initial transient mask estimation, taking advantage of its stronger semantic consistency and robustness to noise, and then progresses to high-resolution supervision to achieve more precise mask prediction. Third, we incorporate the delayed Gaussian growth strategy and mask bootstrapping with appearance modeling to handling in-the-wild scenes including transients and illuminations. Extensive experiments on multiple challenging datasets show that our method outperforms existing methods, clearly demonstrating the robustness and effectiveness of our method.
comment : arXiv admin note: substantial text overlap with arXiv:2506.02751
☆ Shared Multi-modal Embedding Space for Face-Voice Association ICASSP
Christopher Simic, Korbinian Riedhammer, Tobias Bocklet
The FAME 2026 challenge comprises two demanding tasks: training face-voice associations combined with a multilingual setting that includes testing on languages on which the model was not trained. Our approach consists of separate uni-modal processing pipelines with general face and voice feature extraction, complemented by additional age-gender feature extraction to support prediction. The resulting single-modal features are projected into a shared embedding space and trained with an Adaptive Angular Margin (AAM) loss. Our approach achieved first place in the FAME 2026 challenge, with an average Equal-Error Rate (EER) of 23.99%.
comment : Ranked 1st in Fame 2026 Challenge, ICASSP
☆ EMMA: Efficient Multimodal Understanding , Generation, and Editing with a Unified Architecture
Xin He, Longhui Wei, Jianbo Ouyang, Lingxi Xie, Qi Tian
We propose EMMA, an efficient and unified architecture for multimodal understanding, generation and editing. Specifically, EMMA primarily consists of 1) An efficient autoencoder with a 32x compression ratio, which significantly reduces the number of tokens required for generation. This also ensures the training balance between understanding and generation tasks by applying the same compression ratio to images. 2) Channel-wise concatenation instead of token-wise concatenation among visual understanding and generation tokens, which further reduces the visual tokens in unified architectures. 3) A shared-and-decoupled network that enables mutual improvements across tasks while meeting the task-specific modeling requirements. 4) A mixture-of-experts mechanism adopted for visual understanding encoder, which substantially improves perceptual capabilities with a few parameters increase. Extensive experiments have shown that EMMA-4B can significantly outperform state-of-the-art unified multimodal approaches (e.g., BAGEL-7B) in both efficiency and performance, while also achieving competitive results compared to recent multimodal understanding and generation experts (e.g., Qwen3-VL and Qwen-Image). We believe that EMMA lays a solid foundation for the future development of unified multimodal architectures.
comment : Project Page: https://emma-umm.github.io/emma/
☆ LaFiTe: A Generative Latent Field for 3D Native Texturing
Chia-Hao Chen, Zi-Xin Zou, Yan-Pei Cao, Ze Yuan, Guan Luo, Xiaojuan Qi, Ding Liang, Song-Hai Zhang, Yuan-Chen Guo
Generating high-fidelity, seamless textures directly on 3D surfaces, what we term 3D-native texturing, remains a fundamental open challenge, with the potential to overcome long-standing limitations of UV-based and multi-view projection methods. However, existing native approaches are constrained by the absence of a powerful and versatile latent representation, which severely limits the fidelity and generality of their generated textures. We identify this representation gap as the principal barrier to further progress. We introduce LaFiTe, a framework that addresses this challenge by learning to generate textures as a 3D generative sparse latent color field. At its core, LaFiTe employs a variational autoencoder (VAE) to encode complex surface appearance into a sparse, structured latent space, which is subsequently decoded into a continuous color field. This representation achieves unprecedented fidelity, exceeding state-of-the-art methods by >10 dB PSNR in reconstruction, by effectively disentangling texture appearance from mesh topology and UV parameterization. Building upon this strong representation, a conditional rectified-flow model synthesizes high-quality, coherent textures across diverse styles and geometries. Extensive experiments demonstrate that LaFiTe not only sets a new benchmark for 3D-native texturing but also enables flexible downstream applications such as material synthesis and texture super-resolution, paving the way for the next generation of 3D content creation workflows.
comment : Project page: https://vast-ai-research.github.io/LaFiTe/
☆ MemLoRA: Distilling Expert Adapters for On-Device Memory Systems
Massimo Bini, Ondrej Bohdal, Umberto Michieli, Zeynep Akata, Mete Ozay, Taha Ceritli
Memory-augmented Large Language Models (LLMs) have demonstrated remarkable consistency during prolonged dialogues by storing relevant memories and incorporating them as context. Such memory-based personalization is also key in on-device settings that allow users to keep their conversations and data private. However, memory-augmented systems typically rely on LLMs that are too costly for local on-device deployment. Even though Small Language Models (SLMs) are more suitable for on-device inference than LLMs, they cannot achieve sufficient performance. Additionally, these LLM-based systems lack native visual capabilities, limiting their applicability in multimodal contexts. In this paper, we introduce (i) MemLoRA, a novel memory system that enables local deployment by equipping SLMs with specialized memory adapters, and (ii) its vision extension MemLoRA-V, which integrates small Vision-Language Models (SVLMs) to memory systems, enabling native visual understanding. Following knowledge distillation principles, each adapter is trained separately for specific memory operations$\unicode{x2013}$knowledge extraction, memory update, and memory-augmented generation. Equipped with memory adapters, small models enable accurate on-device memory operations without cloud dependency. On text-only operations, MemLoRA outperforms 10$\times$ larger baseline models (e.g., Gemma2-27B) and achieves performance comparable to 60$\times$ larger models (e.g., GPT-OSS-120B) on the LoCoMo benchmark. To evaluate visual understanding operations instead, we extend LoCoMo with challenging Visual Question Answering tasks that require direct visual reasoning. On this, our VLM-integrated MemLoRA-V shows massive improvements over caption-based approaches (81.3 vs. 23.7 accuracy) while keeping strong performance in text-based tasks, demonstrating the efficacy of our method in multimodal contexts.
☆ Order Matters: 3D Shape Generation from Sequential VR Sketches
Yizi Chen, Sidi Wu, Tianyi Xiao, Nina Wiedemann, Loic Landrieu
VR sketching lets users explore and iterate on ideas directly in 3D, offering a faster and more intuitive alternative to conventional CAD tools. However, existing sketch-to-shape models ignore the temporal ordering of strokes, discarding crucial cues about structure and design intent. We introduce VRSketch2Shape, the first framework and multi-category dataset for generating 3D shapes from sequential VR sketches. Our contributions are threefold: (i) an automated pipeline that generates sequential VR sketches from arbitrary shapes, (ii) a dataset of over 20k synthetic and 900 hand-drawn sketch-shape pairs across four categories, and (iii) an order-aware sketch encoder coupled with a diffusion-based 3D generator. Our approach yields higher geometric fidelity than prior work, generalizes effectively from synthetic to real sketches with minimal supervision, and performs well even on partial sketches. All data and models will be released open-source at https://chenyizi086.github.io/VRSketch2Shape_website.
☆ MT-Depth: Multi-task Instance feature analysis for the Depth Completion
Abdul Haseeb Nizamani, Dandi Zhou, Xinhai Sun
Depth completion plays a vital role in 3D perception systems, especially in scenarios where sparse depth data must be densified for tasks such as autonomous driving, robotics, and augmented reality. While many existing approaches rely on semantic segmentation to guide depth completion, they often overlook the benefits of object-level understanding. In this work, we introduce an instance-aware depth completion framework that explicitly integrates binary instance masks as spatial priors to refine depth predictions. Our model combines four main components: a frozen YOLO V11 instance segmentation branch, a U-Net-based depth completion backbone, a cross-attention fusion module, and an attention-guided prediction head. The instance segmentation branch generates per-image foreground masks that guide the depth branch via cross-attention, allowing the network to focus on object-centric regions during refinement. We validate our method on the Virtual KITTI 2 dataset, showing that it achieves lower RMSE compared to both a U-Net-only baseline and previous semantic-guided methods, while maintaining competitive MAE. Qualitative and quantitative results demonstrate that the proposed model effectively enhances depth accuracy near object boundaries, occlusions, and thin structures. Our findings suggest that incorporating instance-aware cues offers a promising direction for improving depth completion without relying on dense semantic labels.
☆ E3AD: An Emotion-Aware Vision -Language-Action Model for Human-Centric End-to-End Autonomous Driving
Yihong Tang, Haicheng Liao, Tong Nie, Junlin He, Ao Qu, Kehua Chen, Wei Ma, Zhenning Li, Lijun Sun, Chengzhong Xu
End-to-end autonomous driving (AD) systems increasingly adopt vision-language-action (VLA) models, yet they typically ignore the passenger's emotional state, which is central to comfort and AD acceptance. We introduce Open-Domain End-to-End (OD-E2E) autonomous driving, where an autonomous vehicle (AV) must interpret free-form natural-language commands, infer the emotion, and plan a physically feasible trajectory. We propose E3AD, an emotion-aware VLA framework that augments semantic understanding with two cognitively inspired components: a continuous Valenc-Arousal-Dominance (VAD) emotion model that captures tone and urgency from language, and a dual-pathway spatial reasoning module that fuses egocentric and allocentric views for human-like spatial cognition. A consistency-oriented training scheme, combining modality pretraining with preference-based alignment, further enforces coherence between emotional intent and driving actions. Across real-world datasets, E3AD improves visual grounding and waypoint planning and achieves state-of-the-art (SOTA) VAD correlation for emotion estimation. These results show that injecting emotion into VLA-style driving yields more human-aligned grounding, planning, and human-centric feedback.
☆ Measuring the Unspoken: A Disentanglement Model and Benchmark for Psychological Analysis in the Wild
Yigui Feng, Qinglin Wang, Haotian Mo, Yang Liu, Ke Liu, Gencheng Liu, Xinhai Chen, Siqi Shen, Songzhu Mei, Jie Liu
Generative psychological analysis of in-the-wild conversations faces two fundamental challenges: (1) existing Vision-Language Models (VLMs) fail to resolve Articulatory-Affective Ambiguity, where visual patterns of speech mimic emotional expressions; and (2) progress is stifled by a lack of verifiable evaluation metrics capable of assessing visual grounding and reasoning depth. We propose a complete ecosystem to address these twin challenges. First, we introduce Multilevel Insight Network for Disentanglement(MIND), a novel hierarchical visual encoder that introduces a Status Judgment module to algorithmically suppress ambiguous lip features based on their temporal feature variance, achieving explicit visual disentanglement. Second, we construct ConvoInsight-DB, a new large-scale dataset with expert annotations for micro-expressions and deep psychological inference. Third, Third, we designed the Mental Reasoning Insight Rating Metric (PRISM), an automated dimensional framework that uses expert-guided LLM to measure the multidimensional performance of large mental vision models. On our PRISM benchmark, MIND significantly outperforms all baselines, achieving a +86.95% gain in micro-expression detection over prior SOTA. Ablation studies confirm that our Status Judgment disentanglement module is the most critical component for this performance leap. Our code has been opened.
☆ Hardware-aware Neural Architecture Search of Ea rl y Exiting Networks on Edge Accelerators
Alaa Zniber, Arne Symons, Ouassim Karrakchou, Marian Verhelst, Mounir Ghogho
Advancements in high-performance computing and cloud technologies have enabled the development of increasingly sophisticated Deep Learning (DL) models. However, the growing demand for embedded intelligence at the edge imposes stringent computational and energy constraints, challenging the deployment of these large-scale models. Early Exiting Neural Networks (EENN) have emerged as a promising solution, allowing dynamic termination of inference based on input complexity to enhance efficiency. Despite their potential, EENN performance is highly influenced by the heterogeneity of edge accelerators and the constraints imposed by quantization, affecting accuracy, energy efficiency, and latency. Yet, research on the automatic optimization of EENN design for edge hardware remains limited. To bridge this gap, we propose a hardware-aware Neural Architecture Search (NAS) framework that systematically integrates the effects of quantization and hardware resource allocation to optimize the placement of early exit points within a network backbone. Experimental results on the CIFAR-10 dataset demonstrate that our NAS framework can discover architectures that achieve over a 50\% reduction in computational costs compared to conventional static networks, making them more suitable for deployment in resource-constrained edge environments.
comment : Submitted to IEEE Transactions on Emerging Topics in Computing
☆ OmniScaleSR: Unleashing Scale-Controlled Diffusion Prior for Faithful and Realistic Arbitrary-Scale Image Super-Resolution
Xinning Chai, Zhengxue Cheng, Yuhong Zhang, Hengsheng Zhang, Yingsheng Qin, Yucai Yang, Rong Xie, Li Song
Arbitrary-scale super-resolution (ASSR) overcomes the limitation of traditional super-resolution (SR) methods that operate only at fixed scales (e.g., 4x), enabling a single model to handle arbitrary magnification. Most existing ASSR approaches rely on implicit neural representation (INR), but its regression-driven feature extraction and aggregation intrinsically limit the ability to synthesize fine details, leading to low realism. Recent diffusion-based realistic image super-resolution (Real-ISR) models leverage powerful pre-trained diffusion priors and show impressive results at the 4x setting. We observe that they can also achieve ASSR because the diffusion prior implicitly adapts to scale by encouraging high-realism generation. However, without explicit scale control, the diffusion process cannot be properly adjusted for different magnification levels, resulting in excessive hallucination or blurry outputs, especially under ultra-high scales. To address these issues, we propose OmniScaleSR, a diffusion-based realistic arbitrary-scale SR framework designed to achieve both high fidelity and high realism. We introduce explicit, diffusion-native scale control mechanisms that work synergistically with implicit scale adaptation, enabling scale-aware and content-aware modulation of the diffusion process. In addition, we incorporate multi-domain fidelity enhancement designs to further improve reconstruction accuracy. Extensive experiments on bicubic degradation benchmarks and real-world datasets show that OmniScaleSR surpasses state-of-the-art methods in both fidelity and perceptual realism, with particularly strong performance at large magnification factors. Code will be released at https://github.com/chaixinning/OmniScaleSR.
comment : Accepted as TCSVT, 15 pages
☆ Towards Cross-View Point Correspondence in Vision -Language Models
Yipu Wang, Yuheng Ji, Yuyang Liu, Enshen Zhou, Ziqiang Yang, Yuxuan Tian, Ziheng Qin, Yue Liu, Huajie Tan, Cheng Chi, Zhiyuan Ma, Daniel Dajun Zeng, Xiaolong Zheng
Cross-view correspondence is a fundamental capability for spatial understanding and embodied AI. However, it is still far from being realized in Vision-Language Models (VLMs), especially in achieving precise point-level correspondence, which is crucial for precise affordance interaction. So we propose the Cross-View Point Correspondence (CVPC) task and CrossPoint-Bench, a comprehensive benchmark with hierarchical design, inspired by the human cognitive process of "perceive", "reason", and "correspond". Our evaluation shows the state-of-the-art models (e.g., Gemini-2.5-Pro) still fall far behind humans, with a gap of over 54.65% in overall accuracy, exposing a challenge in transitioning from coarse-grained judgement to fine-grained coordinate prediction. To address this problem, we construct CrossPoint-378K, a dataset with 378K question-answering pairs across 900 scenes, focused on actionable affordance regions that better reflect real-world manipulation and interaction scenarios. Furthermore, we propose CroPond that trained on the CrossPoint-378K dataset. Our CroPond achieves state-of-the-art performance on CrossPoint-Bench, surpassing Gemini-2.5-Pro by 39.7% accuracy, which offers a foundation for advancing future work on cross-view correspondence. The benchmark, dataset, and model are publicly available at https://github.com/WangYipu2002/CrossPoint.
☆ Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation
Yunhong Lu, Yanhong Zeng, Haobo Li, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Jiapeng Zhu, Hengyuan Cao, Zhipeng Zhang, Xing Zhu, Yujun Shen, Min Zhang
Efficient streaming video generation is critical for simulating interactive and dynamic worlds. Existing methods distill few-step video diffusion models with sliding window attention, using initial frames as sink tokens to maintain attention performance and reduce error accumulation. However, video frames become overly dependent on these static tokens, resulting in copied initial frames and diminished motion dynamics. To address this, we introduce Reward Forcing, a novel framework with two key designs. First, we propose EMA-Sink, which maintains fixed-size tokens initialized from initial frames and continuously updated by fusing evicted tokens via exponential moving average as they exit the sliding window. Without additional computation cost, EMA-Sink tokens capture both long-term context and recent dynamics, preventing initial frame copying while maintaining long-horizon consistency. Second, to better distill motion dynamics from teacher models, we propose a novel Rewarded Distribution Matching Distillation (Re-DMD). Vanilla distribution matching treats every training sample equally, limiting the model's ability to prioritize dynamic content. Instead, Re-DMD biases the model's output distribution toward high-reward regions by prioritizing samples with greater dynamics rated by a vision-language model. Re-DMD significantly enhances motion quality while preserving data fidelity. We include both quantitative and qualitative experiments to show that Reward Forcing achieves state-of-the-art performance on standard benchmarks while enabling high-quality streaming video generation at 23.1 FPS on a single H100 GPU.
☆ Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length
Yubo Huang, Hailong Guo, Fangtai Wu, Shifeng Zhang, Shijie Huang, Qijun Gan, Lin Liu, Sirui Zhao, Enhong Chen, Jiaming Liu, Steven Hoi
Existing diffusion-based video generation methods are fundamentally constrained by sequential computation and long-horizon inconsistency, limiting their practical adoption in real-time, streaming audio-driven avatar synthesis. We present Live Avatar, an algorithm-system co-designed framework that enables efficient, high-fidelity, and infinite-length avatar generation using a 14-billion-parameter diffusion model. Our approach introduces Timestep-forcing Pipeline Parallelism (TPP), a distributed inference paradigm that pipelines denoising steps across multiple GPUs, effectively breaking the autoregressive bottleneck and ensuring stable, low-latency real-time streaming. To further enhance temporal consistency and mitigate identity drift and color artifacts, we propose the Rolling Sink Frame Mechanism (RSFM), which maintains sequence fidelity by dynamically recalibrating appearance using a cached reference image. Additionally, we leverage Self-Forcing Distribution Matching Distillation to facilitate causal, streamable adaptation of large-scale models without sacrificing visual quality. Live Avatar demonstrates state-of-the-art performance, reaching 20 FPS end-to-end generation on 5 H800 GPUs, and, to the best of our knowledge, is the first to achieve practical, real-time, high-fidelity avatar generation at this scale. Our work establishes a new paradigm for deploying advanced diffusion models in industrial long-form video synthesis applications.
☆ I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Models
Juntong Wang, Jiarui Wang, Huiyu Duan, Jiaxiang Kang, Guangtao Zhai, Xiongkuo Min
Image editing models are advancing rapidly, yet comprehensive evaluation remains a significant challenge. Existing image editing benchmarks generally suffer from limited task scopes, insufficient evaluation dimensions, and heavy reliance on manual annotations, which significantly constrain their scalability and practical applicability. To address this, we propose \textbf{I2I-Bench}, a comprehensive benchmark for image-to-image editing models, which features (i) diverse tasks, encompassing 10 task categories across both single-image and multi-image editing tasks, (ii) comprehensive evaluation dimensions, including 30 decoupled and fine-grained evaluation dimensions with automated hybrid evaluation methods that integrate specialized tools and large multimodal models (LMMs), and (iii) rigorous alignment validation, justifying the consistency between our benchmark evaluations and human preferences. Using I2I-Bench, we benchmark numerous mainstream image editing models, investigating the gaps and trade-offs between editing models across various dimensions. We will open-source all components of I2I-Bench to facilitate future research.
☆ SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decoding
Chang-Hsun Wu, Kai-Po Chang, Yu-Yang Sheng, Hung-Kai Chung, Kuei-Chun Wang, Yu-Chiang Frank Wang
Video Large Language Models (VideoLLMs) have shown remarkable progress in video understanding. However, these models still struggle to effectively perceive and exploit rich temporal information in videos when responding to user queries. Therefore, they often generate descriptions of events that are temporal inconsistent or causally implausible, causing severe hallucination issues. While most prior studies have focused on spatial hallucinations (e.g. object mismatches), temporal reasoning in video understanding remains relatively underexplored. To address this issue, we propose Self-Diagnostic Contrastive Decoding (SEASON), a training-free method that adaptively enhances temporal and spatial faithfulness for each output token. It achieves this by dynamically diagnosing each token's hallucination tendency and applying adaptive contrastive decoding against its corresponding temporal and spatial negatives. Extensive experiments demonstrate that SEASON outperforms all existing training-free hallucination mitigation approaches on three hallucination examination benchmarks, while further improves VideoLLMs across four general video understanding benchmarks. The code will be released upon acceptance.
☆ Rethinking Decoupled Knowledge Distillation: A Predictive Distribution Perspective
Bowen Zheng, Ran Cheng
In the history of knowledge distillation, the focus has once shifted over time from logit-based to feature-based approaches. However, this transition has been revisited with the advent of Decoupled Knowledge Distillation (DKD), which re-emphasizes the importance of logit knowledge through advanced decoupling and weighting strategies. While DKD marks a significant advancement, its underlying mechanisms merit deeper exploration. As a response, we rethink DKD from a predictive distribution perspective. First, we introduce an enhanced version, the Generalized Decoupled Knowledge Distillation (GDKD) loss, which offers a more versatile method for decoupling logits. Then we pay particular attention to the teacher model's predictive distribution and its impact on the gradients of GDKD loss, uncovering two critical insights often overlooked: (1) the partitioning by the top logit considerably improves the interrelationship of non-top logits, and (2) amplifying the focus on the distillation loss of non-top logits enhances the knowledge extraction among them. Utilizing these insights, we further propose a streamlined GDKD algorithm with an efficient partition strategy to handle the multimodality of teacher models' predictive distribution. Our comprehensive experiments conducted on a variety of benchmarks, including CIFAR-100, ImageNet, Tiny-ImageNet, CUB-200-2011, and Cityscapes, demonstrate GDKD's superior performance over both the original DKD and other leading knowledge distillation methods. The code is available at https://github.com/ZaberKo/GDKD.
comment : Accepted to IEEE TNNLS
☆ Denoise to Track: Harnessing Video Diffusion Priors for Robust Correspondence
Tianyu Yuan, Yuanbo Yang, Lin-Zhuo Chen, Yao Yao, Zhuzhong Qian
In this work, we introduce HeFT (Head-Frequency Tracker), a zero-shot point tracking framework that leverages the visual priors of pretrained video diffusion models. To better understand how they encode spatiotemporal information, we analyze the internal representations of Video Diffusion Transformer (VDiT). Our analysis reveals that attention heads act as minimal functional units with distinct specializations for matching, semantic understanding, and positional encoding. Additionally, we find that the low-frequency components in VDiT features are crucial for establishing correspondences, whereas the high-frequency components tend to introduce noise. Building on these insights, we propose a head- and frequency-aware feature selection strategy that jointly selects the most informative attention head and low-frequency components to enhance tracking performance. Specifically, our method extracts discriminative features through single-step denoising, applies feature selection, and employs soft-argmax localization with forward-backward consistency checks for correspondence estimation. Extensive experiments on TAP-Vid benchmarks demonstrate that HeFT achieves state-of-the-art zero-shot tracking performance, approaching the accuracy of supervised methods while eliminating the need for annotated training data. Our work further underscores the promise of video diffusion models as powerful foundation models for a wide range of downstream tasks, paving the way toward unified visual foundation models.
☆ Malicious Image Analysis via Vision -Language Segmentation Fusion: Detection, Element, and Location in One-shot
Sheng Hang, Chaoxiang He, Hongsheng Hu, Hanqing Hu, Bin Benjamin Zhu, Shi-Feng Sun, Dawu Gu, Shuo Wang
Detecting illicit visual content demands more than image-level NSFW flags; moderators must also know what objects make an image illegal and where those objects occur. We introduce a zero-shot pipeline that simultaneously (i) detects if an image contains harmful content, (ii) identifies each critical element involved, and (iii) localizes those elements with pixel-accurate masks - all in one pass. The system first applies foundation segmentation model (SAM) to generate candidate object masks and refines them into larger independent regions. Each region is scored for malicious relevance by a vision-language model using open-vocabulary prompts; these scores weight a fusion step that produces a consolidated malicious object map. An ensemble across multiple segmenters hardens the pipeline against adaptive attacks that target any single segmentation method. Evaluated on a newly-annotated 790-image dataset spanning drug, sexual, violent and extremist content, our method attains 85.8% element-level recall, 78.1% precision and a 92.1% segment-success rate - exceeding direct zero-shot VLM localization by 27.4% recall at comparable precision. Against PGD adversarial perturbations crafted to break SAM and VLM, our method's precision and recall decreased by no more than 10%, demonstrating high robustness against attacks. The full pipeline processes an image in seconds, plugs seamlessly into existing VLM workflows, and constitutes the first practical tool for fine-grained, explainable malicious-image moderation.
☆ When Robots Should Say "I Don't Know": Benchmark ing Abstention in Embodied Question Answering
Tao Wu, Chuhao Zhou, Guangyu Zhao, Haozhi Cao, Yewen Pu, Jianfei Yang
Embodied Question Answering (EQA) requires an agent to interpret language, perceive its environment, and navigate within 3D scenes to produce responses. Existing EQA benchmarks assume that every question must be answered, but embodied agents should know when they do not have sufficient information to answer. In this work, we focus on a minimal requirement for EQA agents, abstention: knowing when to withhold an answer. From an initial study of 500 human queries, we find that 32.4% contain missing or underspecified context. Drawing on this initial study and cognitive theories of human communication errors, we derive five representative categories requiring abstention: actionability limitation, referential underspecification, preference dependence, information unavailability, and false presupposition. We augment OpenEQA by having annotators transform well-posed questions into ambiguous variants outlined by these categories. The resulting dataset, AbstainEQA, comprises 1,636 annotated abstention cases paired with 1,636 original OpenEQA instances for balanced evaluation. Evaluating on AbstainEQA, we find that even the best frontier model only attains 42.79% abstention recall, while humans achieve 91.17%. We also find that scaling, prompting, and reasoning only yield marginal gains, and that fine-tuned models overfit to textual cues. Together, these results position abstention as a fundamental prerequisite for reliable interaction in embodied settings and as a necessary basis for effective clarification.
☆ SAM3-I: Segment Anything with Instructions
Jingjing Li, Yue Feng, Yuchen Guo, Jincai Huang, Yongri Piao, Qi Bi, Miao Zhang, Xiaoqi Zhao, Qiang Chen, Shihao Zou, Wei Ji, Huchuan Lu, Li Cheng
Segment Anything Model 3 (SAM3) has advanced open-vocabulary segmentation through promptable concept segmentation, allowing users to segment all instances corresponding to a given concept, typically specified with short noun-phrase (NP) prompts. While this marks the first integration of language-level concepts within the SAM family, real-world usage typically requires far richer expressions that include attributes, spatial relations, functionalities, actions, states, and even implicit reasoning over instances. Currently, SAM3 relies on external multi-modal agents to convert complex instructions into NPs and then conduct iterative mask filtering. However, these NP-level concepts remain overly coarse, often failing to precisely represent a specific instance. In this work, we present SAM3-I, an enhanced framework that unifies concept-level understanding and instruction-level reasoning within the SAM family. SAM3-I introduces an instruction-aware cascaded adaptation mechanism that progressively aligns expressive instruction semantics with SAM3's existing vision-language representations, enabling direct instruction-following segmentation without sacrificing its original concept-driven capabilities. Furthermore, we design a structured instruction taxonomy spanning concept, simple, and complex levels, and develop a scalable data engine to construct a dataset with diverse instruction-mask pairs. Experiments show that SAM3-I delivers appealing performance, demonstrating that SAM3 can be effectively extended to follow natural-language instructions while preserving its strong concept grounding. We open-source SAM3-I and provide practical fine-tuning workflows, enabling researchers to adapt it to domain-specific applications. The source code is available here.
comment : Preliminary results; work in progress
☆ Infrared UAV Target Tracking with Dynamic Feature Refinement and Global Context ual Attention Knowledge Distillation
Houzhang Fang, Chenxing Wu, Kun Bai, Tianqi Chen, Xiaolin Wang, Xiyang Liu, Yi Chang, Luxin Yan
Unmanned aerial vehicle (UAV) target tracking based on thermal infrared imaging has been one of the most important sensing technologies in anti-UAV applications. However, the infrared UAV targets often exhibit weak features and complex backgrounds, posing significant challenges to accurate tracking. To address these problems, we introduce SiamDFF, a novel dynamic feature fusion Siamese network that integrates feature enhancement and global contextual attention knowledge distillation for infrared UAV target (IRUT) tracking. The SiamDFF incorporates a selective target enhancement network (STEN), a dynamic spatial feature aggregation module (DSFAM), and a dynamic channel feature aggregation module (DCFAM). The STEN employs intensity-aware multi-head cross-attention to adaptively enhance important regions for both template and search branches. The DSFAM enhances multi-scale UAV target features by integrating local details with global features, utilizing spatial attention guidance within the search frame. The DCFAM effectively integrates the mixed template generated from STEN in the template branch and original template, avoiding excessive background interference with the template and thereby enhancing the emphasis on UAV target region features within the search frame. Furthermore, to enhance the feature extraction capabilities of the network for IRUT without adding extra computational burden, we propose a novel tracking-specific target-aware contextual attention knowledge distiller. It transfers the target prior from the teacher network to the student model, significantly improving the student network's focus on informative regions at each hierarchical level of the backbone network. Extensive experiments on real infrared UAV datasets demonstrate that the proposed approach outperforms state-of-the-art target trackers under complex backgrounds while achieving a real-time tracking speed.
comment : Accepted by IEEE TMM
☆ TARDis: Time Attenuated Representation Disentanglement for Incomplete Multi-Modal Tumor Segmentation and Classification
Zishuo Wan, Qinqin Kang, Yi Huang, Yun Bian, Dawei Ding, Ke Yan
Tumor segmentation and diagnosis in contrast-enhanced Computed Tomography (CT) rely heavily on the physiological dynamics of contrast agents. However, obtaining a complete multi-phase series is often clinically unfeasible due to radiation concerns or scanning limitations, leading to the "missing modality" problem. Existing deep learning approaches typically treat missing phases as absent independent channels, ignoring the inherent temporal continuity of hemodynamics. In this work, we propose Time Attenuated Representation Disentanglement (TARDis), a novel physics-aware framework that redefines missing modalities as missing sample points on a continuous Time-Attenuation Curve. TARDis explicitly disentangles the latent feature space into a time-invariant static component (anatomy) and a time-dependent dynamic component (perfusion). We achieve this via a dual-path architecture: a quantization-based path using a learnable embedding dictionary to extract consistent anatomical structures, and a probabilistic path using a Conditional Variational Autoencoder to model dynamic enhancement conditioned on the estimated scan time. This design allows the network to hallucinate missing hemodynamic features by sampling from the learned latent distribution. Extensive experiments on a large-scale private abdominal CT dataset (2,282 cases) and two public datasets demonstrate that TARDis significantly outperforms state-of-the-art incomplete modality frameworks. Notably, our method maintains robust diagnostic performance even in extreme data-sparsity scenarios, highlighting its potential for reducing radiation exposure while maintaining diagnostic precision.
☆ Prompt 2Craft: Generating Functional Craft Assemblies with LLMs
Vitor Hideyo Isume, Takuya Kiyokawa, Natsuki Yamanobe, Yukiyasu Domae, Weiwei Wan, Kensuke Harada
Inspired by traditional handmade crafts, where a person improvises assemblies based on the available objects, we formally introduce the Craft Assembly Task. It is a robotic assembly task that involves building an accurate representation of a given target object using the available objects, which do not directly correspond to its parts. In this work, we focus on selecting the subset of available objects for the final craft, when the given input is an RGB image of the target in the wild. We use a mask segmentation neural network to identify visible parts, followed by retrieving labeled template meshes. These meshes undergo pose optimization to determine the most suitable template. Then, we propose to simplify the parts of the transformed template mesh to primitive shapes like cuboids or cylinders. Finally, we design a search algorithm to find correspondences in the scene based on local and global proportions. We develop baselines for comparison that consider all possible combinations, and choose the highest scoring combination for common metrics used in foreground maps and mask accuracy. Our approach achieves comparable results to the baselines for two different scenes, and we show qualitative results for an implementation in a real-world scenario.
☆ Dataset creation for supervised deep learning-based analysis of microscopic images -- review of important considerations and recommendations
Christof A. Bertram, Viktoria Weiss, Jonas Ammeling, F. Maria Schabel, Taryn A. Donovan, Frauke Wilm, Christian Marzahl, Katharina Breininger, Marc Aubreville
Supervised deep learning (DL) receives great interest for automated analysis of microscopic images with an increasing body of literature supporting its potential. The development and validation of those DL models relies heavily on the availability of high-quality, large-scale datasets. However, creating such datasets is a complex and resource-intensive process, often hindered by challenges such as time constraints, domain variability, and risks of bias in image collection and label creation. This review provides a comprehensive guide to the critical steps in dataset creation, including: 1) image acquisition, 2) selection of annotation software, and 3) annotation creation. In addition to ensuring a sufficiently large number of images, it is crucial to address sources of image variability (domain shifts) - such as those related to slide preparation and digitization - that could lead to algorithmic errors if not adequately represented in the training data. Key quality criteria for annotations are the three "C"s: correctness, completeness, and consistency. This review explores methods to enhance annotation quality through the use of advanced techniques that mitigate the limitations of single annotators. To support dataset creators, a standard operating procedure (SOP) is provided as supplemental material, outlining best practices for dataset development. Furthermore, the article underscores the importance of open datasets in driving innovation and enhancing reproducibility of DL research. By addressing the challenges and offering practical recommendations, this review aims to advance the creation of and availability to high-quality, large-scale datasets, ultimately contributing to the development of generalizable and robust DL models for pathology applications.
☆ COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligence
Zefeng Zhang, Xiangzhao Hao, Hengzhu Tang, Zhenyu Zhang, Jiawei Sheng, Xiaodong Li, Zhenyang Li, Li Gao, Daiting Shi, Dawei Yin, Tingwen Liu
Visual Spatial Reasoning is crucial for enabling Multimodal Large Language Models (MLLMs) to understand object properties and spatial relationships, yet current models still struggle with 3D-aware reasoning. Existing approaches typically enhance either perception, by augmenting RGB inputs with auxiliary modalities such as depth and segmentation, or reasoning, by training on spatial VQA datasets and applying reinforcement learning, and thus treat these two aspects in isolation. In this work, we investigate whether a unified MLLM can develop an intrinsic ability to enhance spatial perception and, through adaptive interleaved reasoning, achieve stronger spatial intelligence. We propose \textbf{COOPER}, a unified MLLM that leverages depth and segmentation as auxiliary modalities and is trained in two stages to acquire auxiliary modality generation and adaptive, interleaved reasoning capabilities. COOPER achieves an average \textbf{6.91\%} improvement in spatial reasoning while maintaining general performance. Moreover, even a variant trained only for auxiliary modality generation attains a \textbf{7.92\%} gain on distance and size estimation, suggesting that learning to generate auxiliary modalities helps internalize spatial knowledge and strengthen spatial understanding.
☆ Efficient Spatially-Variant Convolution via Differentiable Sparse Kernel Complex
Zhizhen Wu, Zhe Cao, Yuchi Huo
Image convolution with complex kernels is a fundamental operation in photography, scientific imaging, and animation effects, yet direct dense convolution is computationally prohibitive on resource-limited devices. Existing approximations, such as simulated annealing or low-rank decompositions, either lack efficiency or fail to capture non-convex kernels. We introduce a differentiable kernel decomposition framework that represents a target spatially-variant, dense, complex kernel using a set of sparse kernel samples. Our approach features (i) a decomposition that enables differentiable optimization of sparse kernels, (ii) a dedicated initialization strategy for non-convex shapes to avoid poor local minima, and (iii) a kernel-space interpolation scheme that extends single-kernel filtering to spatially varying filtering without retraining and additional runtime overhead. Experiments on Gaussian and non-convex kernels show that our method achieves higher fidelity than simulated annealing and significantly lower cost than low-rank decompositions. Our approach provides a practical solution for mobile imaging and real-time rendering, while remaining fully differentiable for integration into broader learning pipelines.
comment : 10 pages, 7 figures
☆ Counterfeit Answers: Adversarial Forgery against OCR -Free Document Visual Question Answering
Marco Pintore, Maura Pintor, Dimosthenis Karatzas, Battista Biggio
Document Visual Question Answering (DocVQA) enables end-to-end reasoning grounded on information present in a document input. While recent models have shown impressive capabilities, they remain vulnerable to adversarial attacks. In this work, we introduce a novel attack scenario that aims to forge document content in a visually imperceptible yet semantically targeted manner, allowing an adversary to induce specific or generally incorrect answers from a DocVQA model. We develop specialized attack algorithms that can produce adversarially forged documents tailored to different attackers' goals, ranging from targeted misinformation to systematic model failure scenarios. We demonstrate the effectiveness of our approach against two end-to-end state-of-the-art models: Pix2Struct, a vision-language transformer that jointly processes image and text through sequence-to-sequence modeling, and Donut, a transformer-based model that directly extracts text and answers questions from document images. Our findings highlight critical vulnerabilities in current DocVQA systems and call for the development of more robust defenses.
☆ Gaussian Entropy Fields: Driving Adaptive Sparsity in 3D Gaussian Optimization
Hong Kuang, Jianchen Liu
3D Gaussian Splatting (3DGS) has emerged as a leading technique for novel view synthesis, demonstrating exceptional rendering efficiency. \replaced[]{Well-reconstructed surfaces can be characterized by low configurational entropy, where dominant primitives clearly define surface geometry while redundant components are suppressed.}{The key insight is that well-reconstructed surfaces naturally exhibit low configurational entropy, where dominant primitives clearly define surface geometry while suppressing redundant components.} Three complementary technical contributions are introduced: (1) entropy-driven surface modeling via entropy minimization for low configurational entropy in primitive distributions; (2) adaptive spatial regularization using the Surface Neighborhood Redundancy Index (SNRI) and image entropy-guided weighting; (3) multi-scale geometric preservation through competitive cross-scale entropy alignment. Extensive experiments demonstrate that GEF achieves competitive geometric precision on DTU and T\&T benchmarks, while delivering superior rendering quality compared to existing methods on Mip-NeRF 360. Notably, superior Chamfer Distance (0.64) on DTU and F1 score (0.44) on T\&T are obtained, alongside the best SSIM (0.855) and LPIPS (0.136) among baselines on Mip-NeRF 360, validating the framework's ability to enhance surface reconstruction accuracy without compromising photometric fidelity.
comment : 28 pages,11 figures
☆ VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Management
Hongbo Jin, Qingyuan Wang, Wenhao Zhang, Yang Liu, Sijie Cheng
Ultra long video understanding remains an open challenge, as existing vision language models (VLMs) falter on such content due to limited context length and inefficient long term memory retention. To address this, recent works have attempted to construct external knowledge bases and corresponding retrieval agumented generation (RAG) systems, yet these incur enormous storage and computational overhead. In this paper, we propose VideoMem, a novel framework that pioneers models long video understanding as a sequential generation task via adaptive memory management. Specifically, VideoMem dynamically updates a global memory buffer, which adaptively retains critical information while discarding redundant content across the video timeline. To efficiently train VLMs for such long-term tasks, VideoMem integrates the Progressive Grouped Relative Policy Optimization (PRPO) algorithm, equipped with two core modules: Progressive State Propagation (PSP) adaptively retains valid current states, propagates them to the next rollout step, and gradually narrows the model exploration space. Temporal Cascading Reward (TCR) further alleviates reward sparsity, improving sample utilization and accelerating convergence. Extensive experiments demonstrate that VideoMem significantly outperforms existing open-source models across diverse benchmarks for ultra-long video understanding tasks.
☆ X-Humanoid: Robotize Human Videos to Generate Humanoid Videos at Scale
Pei Yang, Hai Ci, Yiren Song, Mike Zheng Shou
The advancement of embodied AI has unlocked significant potential for intelligent humanoid robots. However, progress in both Vision-Language-Action (VLA) models and world models is severely hampered by the scarcity of large-scale, diverse training data. A promising solution is to "robotize" web-scale human videos, which has been proven effective for policy training. However, these solutions mainly "overlay" robot arms to egocentric videos, which cannot handle complex full-body motions and scene occlusions in third-person videos, making them unsuitable for robotizing humans. To bridge this gap, we introduce X-Humanoid, a generative video editing approach that adapts the powerful Wan 2.2 model into a video-to-video structure and finetunes it for the human-to-humanoid translation task. This finetuning requires paired human-humanoid videos, so we designed a scalable data creation pipeline, turning community assets into 17+ hours of paired synthetic videos using Unreal Engine. We then apply our trained model to 60 hours of the Ego-Exo4D videos, generating and releasing a new large-scale dataset of over 3.6 million "robotized" humanoid video frames. Quantitative analysis and user studies confirm our method's superiority over existing baselines: 69% of users rated it best for motion consistency, and 62.1% for embodiment correctness.
☆ Detection of Intoxicated Individuals from Facial Video Sequences via a Recurrent Fusion Model
Bita Baroutian, Atefe Aghaei, Mohsen Ebrahimi Moghaddam
Alcohol consumption is a significant public health concern and a major cause of accidents and fatalities worldwide. This study introduces a novel video-based facial sequence analysis approach dedicated to the detection of alcohol intoxication. The method integrates facial landmark analysis via a Graph Attention Network (GAT) with spatiotemporal visual features extracted using a 3D ResNet. These features are dynamically fused with adaptive prioritization to enhance classification performance. Additionally, we introduce a curated dataset comprising 3,542 video segments derived from 202 individuals to support training and evaluation. Our model is compared against two baselines: a custom 3D-CNN and a VGGFace+LSTM architecture. Experimental results show that our approach achieves 95.82% accuracy, 0.977 precision, and 0.97 recall, outperforming prior methods. The findings demonstrate the model's potential for practical deployment in public safety systems for non-invasive, reliable alcohol intoxication detection.
☆ Refaçade: Editing Object with Given Reference Texture
Youze Huang, Penghui Ruan, Bojia Zi, Xianbiao Qi, Jianan Wang, Rong Xiao
Recent advances in diffusion models have brought remarkable progress in image and video editing, yet some tasks remain underexplored. In this paper, we introduce a new task, Object Retexture, which transfers local textures from a reference object to a target object in images or videos. To perform this task, a straightforward solution is to use ControlNet conditioned on the source structure and the reference texture. However, this approach suffers from limited controllability for two reasons: conditioning on the raw reference image introduces unwanted structural information, and it fails to disentangle the visual texture and structure information of the source. To address this problem, we propose Refaçade, a method that consists of two key designs to achieve precise and controllable texture transfer in both images and videos. First, we employ a texture remover trained on paired textured/untextured 3D mesh renderings to remove appearance information while preserving the geometry and motion of source videos. Second, we disrupt the reference global layout using a jigsaw permutation, encouraging the model to focus on local texture statistics rather than the global layout of the object. Extensive experiments demonstrate superior visual quality, precise editing, and controllability, outperforming strong baselines in both quantitative and human evaluations. Code is available at https://github.com/fishZe233/Refacade.
☆ PhyVLLM: Physics-Guided Video Language Model with Motion-Appearance Disentanglement
Yu-Wei Zhan, Xin Wang, Hong Chen, Tongtong Feng, Wei Feng, Ren Wang, Guangyao Li, Qing Li, Wenwu Zhu
Video Large Language Models (Video LLMs) have shown impressive performance across a wide range of video-language tasks. However, they often fail in scenarios requiring a deeper understanding of physical dynamics. This limitation primarily arises from their reliance on appearance-based matching. Incorporating physical motion modeling is crucial for deeper video understanding, but presents three key challenges: (1) motion signals are often entangled with appearance variations, making it difficult to extract clean physical cues; (2) effective motion modeling requires not only continuous-time motion representations but also capturing physical dynamics; and (3) collecting accurate annotations for physical attributes is costly and often impractical. To address these issues, we propose PhyVLLM, a physical-guided video-language framework that explicitly incorporates physical motion into Video LLMs. Specifically, PhyVLLM disentangles visual appearance and object motion through a dual-branch encoder. To model physical dynamics over time, we incorporate a Neural Ordinary Differential Equation (Neural ODE) module, which generates differentiable physical dynamic representations. The resulting motion-aware representations are projected into the token space of a pretrained LLM, enabling physics reasoning without compromising the model's original multimodal capabilities. To circumvent the need for explicit physical labels, PhyVLLM employs a self-supervised manner to model the continuous evolution of object motion. Experimental results demonstrate that PhyVLLM significantly outperforms state-of-the-art Video LLMs on both physical reasoning and general video understanding tasks, highlighting the advantages of incorporating explicit physical modeling.
☆ Auto3R: Automated 3D Reconstruction and Scanning via Data-driven Uncertainty Quantification
Chentao Shen, Sizhe Zheng, Bingqian Wu, Yaohua Feng, Yuanchen Fei, Mingyu Mei, Hanwen Jiang, Xiangru Huang
Traditional high-quality 3D scanning and reconstruction typically relies on human labor to plan the scanning procedure. With the rapid development of embodied systems such as drones and robots, there is a growing demand of performing accurate 3D scanning and reconstruction in an fully automated manner. We introduce Auto3R, a data-driven uncertainty quantification model that is designed to automate the 3D scanning and reconstruction of scenes and objects, including objects with non-lambertian and specular materials. Specifically, in a process of iterative 3D reconstruction and scanning, Auto3R can make efficient and accurate prediction of uncertainty distribution over potential scanning viewpoints, without knowing the ground truth geometry and appearance. Through extensive experiments, Auto3R achieves superior performance that outperforms the state-of-the-art methods by a large margin. We also deploy Auto3R on a robot arm equipped with a camera and demonstrate that Auto3R can be used to effectively digitize real-world 3D objects and delivers ready-to-use and photorealistic digital assets. Our homepage: https://tomatoma00.github.io/auto3r.github.io .
☆ Id entity Clue Refinement and Enhancement for Visible-Infrared Person Re-Identification
Guoqing Zhang, Zhun Wang, Hairui Wang, Zhonglin Ye, Yuhui Zheng
Visible-Infrared Person Re-Identification (VI-ReID) is a challenging cross-modal matching task due to significant modality discrepancies. While current methods mainly focus on learning modality-invariant features through unified embedding spaces, they often focus solely on the common discriminative semantics across modalities while disregarding the critical role of modality-specific identity-aware knowledge in discriminative feature learning. To bridge this gap, we propose a novel Identity Clue Refinement and Enhancement (ICRE) network to mine and utilize the implicit discriminative knowledge inherent in modality-specific attributes. Initially, we design a Multi-Perception Feature Refinement (MPFR) module that aggregates shallow features from shared branches, aiming to capture modality-specific attributes that are easily overlooked. Then, we propose a Semantic Distillation Cascade Enhancement (SDCE) module, which distills identity-aware knowledge from the aggregated shallow features and guide the learning of modality-invariant features. Finally, an Identity Clues Guided (ICG) Loss is proposed to alleviate the modality discrepancies within the enhanced features and promote the learning of a diverse representation space. Extensive experiments across multiple public datasets clearly show that our proposed ICRE outperforms existing SOTA methods.
comment : 14 pages, 7 figures
☆ WiFi-based Cross-Domain Gesture Recognition Using Attention Mechanism
Ruijing Liu, Cunhua Pan, Jiaming Zeng, Hong Ren, Kezhi Wang, Lei Kong, Jiangzhou Wang
While fulfilling communication tasks, wireless signals can also be used to sense the environment. Among various types of sensing media, WiFi signals offer advantages such as widespread availability, low hardware cost, and strong robustness to environmental conditions like light, temperature, and humidity. By analyzing Wi-Fi signals in the environment, it is possible to capture dynamic changes of the human body and accomplish sensing applications such as gesture recognition. Although many existing gesture sensing solutions perform well in-domain but lack cross-domain capabilities (i.e., recognition performance in untrained environments). To address this, we extract Doppler spectra from the channel state information (CSI) received by all receivers and concatenate each Doppler spectrum along the same time axis to generate fused images with multi-angle information as input features. Furthermore, inspired by the convolutional block attention module (CBAM), we propose a gesture recognition network that integrates a multi-semantic spatial attention mechanism with a self-attention-based channel mechanism. This network constructs attention maps to quantify the spatiotemporal features of gestures in images, enabling the extraction of key domain-independent features. Additionally, ResNet18 is employed as the backbone network to further capture deep-level features. To validate the network performance, we evaluate the proposed network on the public Widar3 dataset, and the results show that it not only maintains high in-domain accuracy of 99.72%, but also achieves high performance in cross-domain recognition of 97.61%, significantly outperforming existing best solutions.
☆ Boundary-Aware Test-Time Adaptation for Zero-Shot Medical Image Segmentation
Chenlin Xu, Lei Zhang, Lituan Wang, Xinyu Pu, Pengfei Ma, Guangwu Qian, Zizhou Wang, Yan Wang
Due to the scarcity of annotated data and the substantial computational costs of model, conventional tuning methods in medical image segmentation face critical challenges. Current approaches to adapting pretrained models, including full-parameter and parameter-efficient fine-tuning, still rely heavily on task-specific training on downstream tasks. Therefore, zero-shot segmentation has gained increasing attention, especially with foundation models such as SAM demonstrating promising generalization capabilities. However, SAM still faces notable limitations on medical datasets due to domain shifts, making efficient zero-shot enhancement an urgent research goal. To address these challenges, we propose BA-TTA-SAM, a task-agnostic test-time adaptation framework that significantly enhances the zero-shot segmentation performance of SAM via test-time adaptation. This framework integrates two key mechanisms: (1) The encoder-level Gaussian prompt injection embeds Gaussian-based prompts directly into the image encoder, providing explicit guidance for initial representation learning. (2) The cross-layer boundary-aware attention alignment exploits the hierarchical feature interactions within the ViT backbone, aligning deep semantic responses with shallow boundary cues. Experiments on four datasets, including ISIC, Kvasir, BUSI, and REFUGE, show an average improvement of 12.4\% in the DICE score compared with SAM's zero-shot segmentation performance. The results demonstrate that our method consistently outperforms state-of-the-art models in medical image segmentation. Our framework significantly enhances the generalization ability of SAM, without requiring any source-domain training data. Extensive experiments on publicly available medical datasets strongly demonstrate the superiority of our framework. Our code is available at https://github.com/Emilychenlin/BA-TTA-SAM.
☆ VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memory
Yifei Yu, Xiaoshan Wu, Xinting Hu, Tao Hu, Yangtian Sun, Xiaoyang Lyu, Bo Wang, Lin Ma, Yuewen Ma, Zhongrui Wang, Xiaojuan Qi
Autoregressive (AR) diffusion enables streaming, interactive long-video generation by producing frames causally, yet maintaining coherence over minute-scale horizons remains challenging due to accumulated errors, motion drift, and content repetition. We approach this problem from a memory perspective, treating video synthesis as a recurrent dynamical process that requires coordinated short- and long-term context. We propose VideoSSM, a Long Video Model that unifies AR diffusion with a hybrid state-space memory. The state-space model (SSM) serves as an evolving global memory of scene dynamics across the entire sequence, while a context window provides local memory for motion cues and fine details. This hybrid design preserves global consistency without frozen, repetitive patterns, supports prompt-adaptive interaction, and scales in linear time with sequence length. Experiments on short- and long-range benchmarks demonstrate state-of-the-art temporal consistency and motion stability among autoregressive video generator especially at minute-scale horizons, enabling content diversity and interactive prompt-based control, thereby establishing a scalable, memory-aware framework for long video generation.
☆ EgoLCD: Egocentric Video Generation with Long Context Diffusion
Liuzhou Zhang, Jiarui Ye, Yuanlei Wang, Ming Zhong, Mingju Cao, Wanke Xia, Bowen Zeng, Zeyu Zhang, Hao Tang
Generating long, coherent egocentric videos is difficult, as hand-object interactions and procedural tasks require reliable long-term memory. Existing autoregressive models suffer from content drift, where object identity and scene semantics degrade over time. To address this challenge, we introduce EgoLCD, an end-to-end framework for egocentric long-context video generation that treats long video synthesis as a problem of efficient and stable memory management. EgoLCD combines a Long-Term Sparse KV Cache for stable global context with an attention-based short-term memory, extended by LoRA for local adaptation. A Memory Regulation Loss enforces consistent memory usage, and Structured Narrative Prompting provides explicit temporal guidance. Extensive experiments on the EgoVid-5M benchmark demonstrate that EgoLCD achieves state-of-the-art performance in both perceptual quality and temporal consistency, effectively mitigating generative forgetting and representing a significant step toward building scalable world models for embodied AI. Code: https://github.com/AIGeeksGroup/EgoLCD. Website: https://aigeeksgroup.github.io/EgoLCD.
☆ DuGI-MAE: Improving Infrared Mask Autoencoders via Dual-Domain Guidance
Yinghui Xing, Xiaoting Su, Shizhou Zhang, Donghao Chu, Di Xu
Infrared imaging plays a critical role in low-light and adverse weather conditions. However, due to the distinct characteristics of infrared images, existing foundation models such as Masked Autoencoder (MAE) trained on visible data perform suboptimal in infrared image interpretation tasks. To bridge this gap, an infrared foundation model known as InfMAE was developed and pre-trained on large-scale infrared datasets. Despite its effectiveness, InfMAE still faces several limitations, including the omission of informative tokens, insufficient modeling of global associations, and neglect of non-uniform noise. In this paper, we propose a Dual-domain Guided Infrared foundation model based on MAE (DuGI-MAE). First, we design a deterministic masking strategy based on token entropy, preserving only high-entropy tokens for reconstruction to enhance informativeness. Next, we introduce a Dual-Domain Guidance (DDG) module, which simultaneously captures global token relationships and adaptively filters non-uniform background noise commonly present in infrared imagery. To facilitate large-scale pretraining, we construct Inf-590K, a comprehensive infrared image dataset encompassing diverse scenes, various target types, and multiple spatial resolutions. Pretrained on Inf-590K, DuGI-MAE demonstrates strong generalization capabilities across various downstream tasks, including infrared object detection, semantic segmentation, and small target detection. Experimental results validate the superiority of the proposed method over both supervised and self-supervised comparison methods. Our code is available in the supplementary material.
☆ UltraImage: Rethinking Resolution Extrapolation in Image Diffusion Transformer s
Min Zhao, Bokai Yan, Xue Yang, Hongzhou Zhu, Jintao Zhang, Shilong Liu, Chongxuan Li, Jun Zhu
Recent image diffusion transformers achieve high-fidelity generation, but struggle to generate images beyond these scales, suffering from content repetition and quality degradation. In this work, we present UltraImage, a principled framework that addresses both issues. Through frequency-wise analysis of positional embeddings, we identify that repetition arises from the periodicity of the dominant frequency, whose period aligns with the training resolution. We introduce a recursive dominant frequency correction to constrain it within a single period after extrapolation. Furthermore, we find that quality degradation stems from diluted attention and thus propose entropy-guided adaptive attention concentration, which assigns higher focus factors to sharpen local attention for fine detail and lower ones to global attention patterns to preserve structural consistency. Experiments show that UltraImage consistently outperforms prior methods on Qwen-Image and Flux (around 4K) across three generation scenarios, reducing repetition and improving visual fidelity. Moreover, UltraImage can generate images up to 6K*6K without low-resolution guidance from a training resolution of 1328p, demonstrating its extreme extrapolation capability. Project page is available at \href{https://thu-ml.github.io/ultraimage.github.io/}{https://thu-ml.github.io/ultraimage.github.io/}.
comment : Project page: https://thu-ml.github.io/ultraimage.github.io
☆ Back to Basics: Motion Representation Matters for Human Motion Generation Using Diffusion Model
Yuduo Jin, Brandon Haworth
Diffusion models have emerged as a widely utilized and successful methodology in human motion synthesis. Task-oriented diffusion models have significantly advanced action-to-motion, text-to-motion, and audio-to-motion applications. In this paper, we investigate fundamental questions regarding motion representations and loss functions in a controlled study, and we enumerate the impacts of various decisions in the workflow of the generative motion diffusion model. To answer these questions, we conduct empirical studies based on a proxy motion diffusion model (MDM). We apply v loss as the prediction objective on MDM (vMDM), where v is the weighted sum of motion data and noise. We aim to enhance the understanding of latent data distributions and provide a foundation for improving the state of conditional motion diffusion models. First, we evaluate the six common motion representations in the literature and compare their performance in terms of quality and diversity metrics. Second, we compare the training time under various configurations to shed light on how to speed up the training process of motion diffusion models. Finally, we also conduct evaluation analysis on a large motion dataset. The results of our experiments indicate clear performance differences across motion representations in diverse datasets. Our results also demonstrate the impacts of distinct configurations on model training and suggest the importance and effectiveness of these decisions on the outcomes of motion diffusion models.
☆ Shift-Window Meets Dual Attention: A Multi-Model Architecture for Specular Highlight Removal
Tianci Huo, Lingfeng Qi, Yuhan Chen, Qihong Xue, Jinyuan Shao, Hai Yu, Jie Li, Zhanhua Zhang, Guofa Li
Inevitable specular highlights in practical environments severely impair the visual performance, thus degrading the task effectiveness and efficiency. Although there exist considerable methods that focus on local information from convolutional neural network models or global information from transformer models, the single-type model falls into a modeling dilemma between local fine-grained details and global long-range dependencies, thus deteriorating for specular highlights with different scales. Therefore, to accommodate specular highlights of all scales, we propose a multi-model architecture for specular highlight removal (MM-SHR) that effectively captures fine-grained features in highlight regions and models long-range dependencies between highlight and highlight-free areas. Specifically, we employ convolution operations to extract local details in the shallow layers of MM-SHR, and utilize the attention mechanism to capture global features in the deep layers, ensuring both operation efficiency and removal accuracy. To model long-range dependencies without compromising computational complexity, we utilize a coarse-to-fine manner and propose Omni-Directional Attention Integration Block(OAIBlock) and Adaptive Region-Aware Hybrid-Domain Dual Attention Convolutional Network(HDDAConv) , which leverage omni-directiona pixel-shifting and window-dividing operations at the raw features to achieve specular highlight removal. Extensive experimental results on three benchmark tasks and six types of surface materials demonstrate that MM-SHR outperforms state-of-the-art methods in both accuracy and efficiency for specular highlight removal. The implementation will be made publicly available at https://github.com/Htcicv/MM-SHR.
☆ Controllable Long-term Motion Generation with Extended Joint Targets WACV 2026
Eunjong Lee, Eunhee Kim, Sanghoon Hong, Eunho Jung, Jihoon Kim
Generating stable and controllable character motion in real-time is a key challenge in computer animation. Existing methods often fail to provide fine-grained control or suffer from motion degradation over long sequences, limiting their use in interactive applications. We propose COMET, an autoregressive framework that runs in real time, enabling versatile character control and robust long-horizon synthesis. Our efficient Transformer-based conditional VAE allows for precise, interactive control over arbitrary user-specified joints for tasks like goal-reaching and in-betweening from a single model. To ensure long-term temporal stability, we introduce a novel reference-guided feedback mechanism that prevents error accumulation. This mechanism also serves as a plug-and-play stylization module, enabling real-time style transfer. Extensive evaluations demonstrate that COMET robustly generates high-quality motion at real-time speeds, significantly outperforming state-of-the-art approaches in complex motion control tasks and confirming its readiness for demanding interactive applications.
comment : WACV 2026
☆ Not All Birds Look The Same: Id entity -Preserving Generation For Birds
Aaron Sun, Oindrila Saha, Subhransu Maji
Since the advent of controllable image generation, increasingly rich modes of control have enabled greater customization and accessibility for everyday users. Zero-shot, identity-preserving models such as Insert Anything and OminiControl now support applications like virtual try-on without requiring additional fine-tuning. While these models may be fitting for humans and rigid everyday objects, they still have limitations for non-rigid or fine-grained categories. These domains often lack accessible, high-quality data -- especially videos or multi-view observations of the same subject -- making them difficult both to evaluate and to improve upon. Yet, such domains are essential for moving beyond content creation toward applications that demand accuracy and fine detail. Birds are an excellent domain for this task: they exhibit high diversity, require fine-grained cues for identification, and come in a wide variety of poses. We introduce the NABirds Look-Alikes (NABLA) dataset, consisting of 4,759 expert-curated image pairs. Together with 1,073 pairs collected from multi-image observations on iNaturalist and a small set of videos, this forms a benchmark for evaluating identity-preserving generation of birds. We show that state-of-the-art baselines fail to maintain identity on this dataset, and we demonstrate that training on images grouped by species, age, and sex -- used as a proxy for identity -- substantially improves performance on both seen and unseen species.
☆ DeRA: Decoupled Representation Alignment for Video Tokenization
Pengbo Guo, Junke Wang, Zhen Xing, Chengxu Liu, Daoguo Dong, Xueming Qian, Zuxuan Wu
This paper presents DeRA, a novel 1D video tokenizer that decouples the spatial-temporal representation learning in video tokenization to achieve better training efficiency and performance. Specifically, DeRA maintains a compact 1D latent space while factorizing video encoding into appearance and motion streams, which are aligned with pretrained vision foundation models to capture the spatial semantics and temporal dynamics in videos separately. To address the gradient conflicts introduced by the heterogeneous supervision, we further propose the Symmetric Alignment-Conflict Projection (SACP) module that proactively reformulates gradients by suppressing the components along conflicting directions. Extensive experiments demonstrate that DeRA outperforms LARP, the previous state-of-the-art video tokenizer by 25% on UCF-101 in terms of rFVD. Moreover, using DeRA for autoregressive video generation, we also achieve new state-of-the-art results on both UCF-101 class-conditional generation and K600 frame prediction.
☆ Feature Engineering vs. Deep Learning for Automated Coin Grading: A Comparative Study on Saint-Gaudens Double Eagles
Tanmay Dogra, Eric Ngo, Mohammad Alam, Jean-Paul Talavera, Asim Dahal
We challenge the common belief that deep learning always trumps older techniques, using the example of grading Saint-Gaudens Double Eagle gold coins automatically. In our work, we put a feature-based Artificial Neural Network built around 192 custom features pulled from Sobel edge detection and HSV color analysis up against a hybrid Convolutional Neural Network that blends in EfficientNetV2, plus a straightforward Support Vector Machine as the control. Testing 1,785 coins graded by experts, the ANN nailed 86% exact matches and hit 98% when allowing a 3-grade leeway. On the flip side, CNN and SVM mostly just guessed the most common grade, scraping by with 31% and 30% exact hits. Sure, the CNN looked good on broader tolerance metrics, but that is because of some averaging trick in regression that hides how it totally flops at picking out specific grades. All told, when you are stuck with under 2,000 examples and lopsided classes, baking in real coin-expert knowledge through feature design beats out those inscrutable, all-in-one deep learning setups. This rings true for other niche quality checks where data's thin and know-how matters more than raw compute.
☆ UniTS: Unified Time Series Generative Model for Remote Sensing
Yuxiang Zhang, Shunlin Liang, Wenyuan Li, Han Ma, Jianglei Xu, Yichuan Ma, Jiangwei Xie, Wei Li, Mengmeng Zhang, Ran Tao, Xiang-Gen Xia
One of the primary objectives of satellite remote sensing is to capture the complex dynamics of the Earth environment, which encompasses tasks such as reconstructing continuous cloud-free time series images, detecting land cover changes, and forecasting future surface evolution. However, existing methods typically require specialized models tailored to different tasks, lacking unified modeling of spatiotemporal features across multiple time series tasks. In this paper, we propose a Unified Time Series Generative Model (UniTS), a general framework applicable to various time series tasks, including time series reconstruction, time series cloud removal, time series semantic change detection, and time series forecasting. Based on the flow matching generative paradigm, UniTS constructs a deterministic evolution path from noise to targets under the guidance of task-specific conditions, achieving unified modeling of spatiotemporal representations for multiple tasks. The UniTS architecture consists of a diffusion transformer with spatio-temporal blocks, where we design an Adaptive Condition Injector (ACor) to enhance the model's conditional perception of multimodal inputs, enabling high-quality controllable generation. Additionally, we design a Spatiotemporal-aware Modulator (STM) to improve the ability of spatio-temporal blocks to capture complex spatiotemporal dependencies. Furthermore, we construct two high-quality multimodal time series datasets, TS-S12 and TS-S12CR, filling the gap of benchmark datasets for time series cloud removal and forecasting tasks. Extensive experiments demonstrate that UniTS exhibits exceptional generative and cognitive capabilities in both low-level and high-level time series tasks. It significantly outperforms existing methods, particularly when facing challenges such as severe cloud contamination, modality absence, and forecasting phenological variations.
☆ dVLM-AD: Enhance Diffusion Vision -Language-Model for Driving via Controllable Reasoning
Yingzi Ma, Yulong Cao, Wenhao Ding, Shuibai Zhang, Yan Wang, Boris Ivanovic, Ming Jiang, Marco Pavone, Chaowei Xiao
The autonomous driving community is increasingly focused on addressing the challenges posed by out-of-distribution (OOD) driving scenarios. A dominant research trend seeks to enhance end-to-end (E2E) driving systems by integrating vision-language models (VLMs), leveraging their rich world knowledge and reasoning abilities to improve generalization across diverse environments. However, most existing VLMs or vision-language agents (VLAs) are built upon autoregressive (AR) models. In this paper, we observe that existing AR-based VLMs -- limited by causal attention and sequential token generation -- often fail to maintain consistency and controllability between high-level reasoning and low-level planning. In contrast, recent discrete diffusion VLMs equipped with bidirectional attention exhibit superior controllability and reliability through iterative denoising. Building on these observations, we introduce dVLM-AD, a diffusion-based vision-language model that unifies perception, structured reasoning, and low-level planning for end-to-end driving. Evaluated on nuScenes and WOD-E2E, dVLM-AD yields more consistent reasoning-action pairs and achieves planning performance comparable to existing driving VLM/VLA systems despite a modest backbone, outperforming AR-based baselines with a 9 percent improvement in behavior-trajectory consistency and a 6 percent increase in RFS on long-tail WOD-E2E scenarios. These results suggest a controllable and reliable pathway for scalable end-to-end driving.
☆ GuidNoise: Single-Pair Guided Diffusion for Generalized Noise Synthesis AAAI2026
Changjin Kim, HyeokJun Lee, YoungJoon Yoo
Recent image denoising methods have leveraged generative modeling for real noise synthesis to address the costly acquisition of real-world noisy data. However, these generative models typically require camera metadata and extensive target-specific noisy-clean image pairs, often showing limited generalization between settings. In this paper, to mitigate the prerequisites, we propose a Single-Pair Guided Diffusion for generalized noise synthesis GuidNoise, which uses a single noisy/clean pair as the guidance, often easily obtained by itself within a training set. To train GuidNoise, which generates synthetic noisy images from the guidance, we introduce a guidance-aware affine feature modification (GAFM) and a noise-aware refine loss to leverage the inherent potential of diffusion models. This loss function refines the diffusion model's backward process, making the model more adept at generating realistic noise distributions. The GuidNoise synthesizes high-quality noisy images under diverse noise environments without additional metadata during both training and inference. Additionally, GuidNoise enables the efficient generation of noisy-clean image pairs at inference time, making synthetic noise readily applicable for augmenting training data. This self-augmentation significantly improves denoising performance, especially in practical scenarios with lightweight models and limited training data. The code is available at https://github.com/chjinny/GuidNoise.
comment : AAAI2026
☆ StreamEQA: Towards Streaming Video Understanding for Embodied Scenarios
Yifei Wang, Zhenkai Li, Tianwen Qian, Huanran Zheng, Zheng Wang, Yuqian Fu, Xiaoling Wang
As embodied intelligence advances toward real-world deployment, the ability to continuously perceive and reason over streaming visual inputs becomes essential. In such settings, an agent must maintain situational awareness of its environment, comprehend the interactions with surrounding entities, and dynamically plan actions informed by past observations, current contexts, and anticipated future events. To facilitate progress in this direction, we introduce StreamEQA, the first benchmark designed for streaming video question answering in embodied scenarios. StreamEQA evaluates existing MLLMs along two orthogonal dimensions: Embodied and Streaming. Along the embodied dimension, we categorize the questions into three levels: perception, interaction, and planning, which progressively assess a model's ability to recognize fine-grained visual details, reason about agent-object interactions, and perform high-level goal-directed reasoning. For the streaming dimension, questions are divided into backward, real-time, and forward reasoning, with each mode relying on a distinct temporal context. Built upon 156 independent long videos, StreamEQA defines 42 tasks and generates approximately 21K question-answer pairs with precise timestamps through a hybrid pipeline combining automated generation and human refinement. Evaluations of 13 state-of-the-art video-LLMs reveal that, despite strong performance on conventional benchmarks, these models still struggle with streaming video understanding in embodied scenarios. We hope StreamEQA will catalyze research on streaming video understanding for embodied applications.
☆ MindDrive: An All-in-One Framework Bridging Wo rl d Models and Vision -Language Model for End-to-End Autonomous Driving
Bin Suna, Yaoguang Caob, Yan Wanga, Rui Wanga, Jiachen Shanga, Xiejie Fenga, Jiayi Lu, Jia Shi, Shichun Yang, Xiaoyu Yane, Ziying Song
End-to-End autonomous driving (E2E-AD) has emerged as a new paradigm, where trajectory planning plays a crucial role. Existing studies mainly follow two directions: trajectory generation oriented, which focuses on producing high-quality trajectories with simple decision mechanisms, and trajectory selection oriented, which performs multi-dimensional evaluation to select the best trajectory yet lacks sufficient generative capability. In this work, we propose MindDrive, a harmonized framework that integrates high-quality trajectory generation with comprehensive decision reasoning. It establishes a structured reasoning paradigm of "context simulation - candidate generation - multi-objective trade-off". In particular, the proposed Future-aware Trajectory Generator (FaTG), based on a World Action Model (WaM), performs ego-conditioned "what-if" simulations to predict potential future scenes and generate foresighted trajectory candidates. Building upon this, the VLM-oriented Evaluator (VLoE) leverages the reasoning capability of a large vision-language model to conduct multi-objective evaluations across safety, comfort, and efficiency dimensions, leading to reasoned and human-aligned decision making. Extensive experiments on the NAVSIM-v1 and NAVSIM-v2 benchmarks demonstrate that MindDrive achieves state-of-the-art performance across multi-dimensional driving metrics, significantly enhancing safety, compliance, and generalization. This work provides a promising path toward interpretable and cognitively guided autonomous driving.
☆ Self-Paced and Self-Corrective Masked Prediction for Movie Trailer Generation
Sidan Zhu, Hongteng Xu, Dixin Luo
As a challenging video editing task, movie trailer generation involves selecting and reorganizing movie shots to create engaging trailers. Currently, most existing automatic trailer generation methods employ a "selection-then-ranking" paradigm (i.e., first selecting key shots and then ranking them), which suffers from inevitable error propagation and limits the quality of the generated trailers. Beyond this paradigm, we propose a new self-paced and self-corrective masked prediction method called SSMP, which achieves state-of-the-art results in automatic trailer generation via bi-directional contextual modeling and progressive self-correction. In particular, SSMP trains a Transformer encoder that takes the movie shot sequences as prompts and generates corresponding trailer shot sequences accordingly. The model is trained via masked prediction, reconstructing each trailer shot sequence from its randomly masked counterpart. The mask ratio is self-paced, allowing the task difficulty to adapt to the model and thereby improving model performance. When generating a movie trailer, the model fills the shot positions with high confidence at each step and re-masks the remaining positions for the next prediction, forming a progressive self-correction mechanism that is analogous to how human editors work. Both quantitative results and user studies demonstrate the superiority of SSMP in comparison to existing automatic movie trailer generation methods. Demo is available at: https://github.com/Dixin-Lab/SSMP.
☆ Explainable Parkinsons Disease Gait Recognition Using Multimodal RGB-D Fusion and Large Language Models
Manar Alnaasan, Md Selim Sarowar, Sungho Kim
Accurate and interpretable gait analysis plays a crucial role in the early detection of Parkinsons disease (PD),yet most existing approaches remain limited by single-modality inputs, low robustness, and a lack of clinical transparency. This paper presents an explainable multimodal framework that integrates RGB and Depth (RGB-D) data to recognize Parkinsonian gait patterns under realistic conditions. The proposed system employs dual YOLOv11-based encoders for modality-specific feature extraction, followed by a Multi-Scale Local-Global Extraction (MLGE) module and a Cross-Spatial Neck Fusion mechanism to enhance spatial-temporal representation. This design captures both fine-grained limb motion (e.g., reduced arm swing) and overall gait dynamics (e.g., short stride or turning difficulty), even in challenging scenarios such as low lighting or occlusion caused by clothing. To ensure interpretability, a frozen Large Language Model (LLM) is incorporated to translate fused visual embeddings and structured metadata into clinically meaningful textual explanations. Experimental evaluations on multimodal gait datasets demonstrate that the proposed RGB-D fusion framework achieves higher recognition accuracy, improved robustness to environmental variations, and clear visual-linguistic reasoning compared with single-input baselines. By combining multimodal feature learning with language-based interpretability, this study bridges the gap between visual recognition and clinical understanding, offering a novel vision-language paradigm for reliable and explainable Parkinsons disease gait analysis. Code:https://github.com/manaralnaasan/RGB-D_parkinson-LLM
☆ UTrice: Unifying Primitives in Differentiable Ray Tracing and Rasterization via Triangles for Particle-Based 3D Scenes CVPR2026
Changhe Liu, Ehsan Javanmardi, Naren Bao, Alex Orsholits, Manabu Tsukada
Ray tracing 3D Gaussian particles enables realistic effects such as depth of field, refractions, and flexible camera modeling for novel-view synthesis. However, existing methods trace Gaussians through proxy geometry, which requires constructing complex intermediate meshes and performing costly intersection tests. This limitation arises because Gaussian-based particles are not well suited as unified primitives for both ray tracing and rasterization. In this work, we propose a differentiable triangle-based ray tracing pipeline that directly treats triangles as rendering primitives without relying on any proxy geometry. Our results show that the proposed method achieves significantly higher rendering quality than existing ray tracing approaches while maintaining real-time rendering performance. Moreover, our pipeline can directly render triangles optimized by the rasterization-based method Triangle Splatting, thus unifying the primitives used in novel-view synthesis.
comment : 13 pages, 10 figures, submitted to CVPR2026
☆ Dual-Stream Spectral Decoupling Distillation for Remote Sensing Object Detection
Xiangyi Gao, Danpei Zhao, Bo Yuan, Wentao Li
Knowledge distillation is an effective and hardware-friendly method, which plays a key role in lightweighting remote sensing object detection. However, existing distillation methods often encounter the issue of mixed features in remote sensing images (RSIs), and neglect the discrepancies caused by subtle feature variations, leading to entangled knowledge confusion. To address these challenges, we propose an architecture-agnostic distillation method named Dual-Stream Spectral Decoupling Distillation (DS2D2) for universal remote sensing object detection tasks. Specifically, DS2D2 integrates explicit and implicit distillation grounded in spectral decomposition. Firstly, the first-order wavelet transform is applied for spectral decomposition to preserve the critical spatial characteristics of RSIs. Leveraging this spatial preservation, a Density-Independent Scale Weight (DISW) is designed to address the challenges of dense and small object detection common in RSIs. Secondly, we show implicit knowledge hidden in subtle student-teacher feature discrepancies, which significantly influence predictions when activated by detection heads. This implicit knowledge is extracted via full-frequency and high-frequency amplifiers, which map feature differences to prediction deviations. Extensive experiments on DIOR and DOTA datasets validate the effectiveness of the proposed method. Specifically, on DIOR dataset, DS2D2 achieves improvements of 4.2% in AP50 for RetinaNet and 3.8% in AP50 for Faster R-CNN, outperforming existing distillation approaches. The source code will be available at https://github.com/PolarAid/DS2D2.
comment : 12 pages, 8 figures, 11 tables
☆ Performance Evaluation of Transfer Learning Based Medical Image Classification Techniques for Disease Detection
Zeeshan Ahmad, Shudi Bao, Meng Chen
Medical image classification plays an increasingly vital role in identifying various diseases by classifying medical images, such as X-rays, MRIs and CT scans, into different categories based on their features. In recent years, deep learning techniques have attracted significant attention in medical image classification. However, it is usually infeasible to train an entire large deep learning model from scratch. To address this issue, one of the solutions is the transfer learning (TL) technique, where a pre-trained model is reused for a new task. In this paper, we present a comprehensive analysis of TL techniques for medical image classification using deep convolutional neural networks. We evaluate six pre-trained models (AlexNet, VGG16, ResNet18, ResNet34, ResNet50, and InceptionV3) on a custom chest X-ray dataset for disease detection. The experimental results demonstrate that InceptionV3 consistently outperforms other models across all the standard metrics. The ResNet family shows progressively better performance with increasing depth, whereas VGG16 and AlexNet perform reasonably well but with lower accuracy. In addition, we also conduct uncertainty analysis and runtime comparison to assess the robustness and computational efficiency of these models. Our findings reveal that TL is beneficial in most cases, especially with limited data, but the extent of improvement depends on several factors such as model architecture, dataset size, and domain similarity between source and target tasks. Moreover, we demonstrate that with a well-trained feature extractor, only a lightweight feedforward model is enough to provide efficient prediction. As such, this study contributes to the understanding of TL in medical image classification, and provides insights for selecting appropriate models based on specific requirements.
☆ Fourier-Attentive Representation Learning: A Fourier-Guided Framework for Few-Shot Generalization in Vision -Language Models
Hieu Dinh Trung Pham, Huy Minh Nhat Nguyen, Cuong Tuan Nguyen
Large-scale pre-trained Vision-Language Models (VLMs) have demonstrated strong few-shot learning capabilities. However, these methods typically learn holistic representations where an image's domain-invariant structure is implicitly entangled with its domain-specific style. This presents an opportunity to further enhance generalization by disentangling these visual cues. In this paper, we propose Fourier-Attentive Representation Learning (FARL), a novel framework that addresses this by explicitly disentangling visual representations using Fourier analysis. The core of our method is a dual cross-attention mechanism, where learnable representation tokens separately query an image's structural features (from the phase spectrum) and stylistic features (from the amplitude spectrum). This process yields enriched, disentangled tokens that are then injected deep into the VLM encoders to guide adaptation. Our design, which includes an asymmetric injection strategy, forces the model to learn a more robust vision-language alignment. Extensive experiments on 15 datasets demonstrate the effectiveness of our approach.
☆ FMA-Net++: Motion- and Exposure-Aware Real-Wo rl d Joint Video Super-Resolution and Deblurring
Geunhyuk Youk, Jihyong Oh, Munchurl Kim
Real-world video restoration is plagued by complex degradations from motion coupled with dynamically varying exposure - a key challenge largely overlooked by prior works and a common artifact of auto-exposure or low-light capture. We present FMA-Net++, a framework for joint video super-resolution and deblurring that explicitly models this coupled effect of motion and dynamically varying exposure. FMA-Net++ adopts a sequence-level architecture built from Hierarchical Refinement with Bidirectional Propagation blocks, enabling parallel, long-range temporal modeling. Within each block, an Exposure Time-aware Modulation layer conditions features on per-frame exposure, which in turn drives an exposure-aware Flow-Guided Dynamic Filtering module to infer motion- and exposure-aware degradation kernels. FMA-Net++ decouples degradation learning from restoration: the former predicts exposure- and motion-aware priors to guide the latter, improving both accuracy and efficiency. To evaluate under realistic capture conditions, we introduce REDS-ME (multi-exposure) and REDS-RE (random-exposure) benchmarks. Trained solely on synthetic data, FMA-Net++ achieves state-of-the-art accuracy and temporal consistency on our new benchmarks and GoPro, outperforming recent methods in both restoration quality and inference speed, and generalizes well to challenging real-world videos.
comment : 20 pages, 15 figures. Project Page: https://kaist-viclab.github.io/fmanetpp_site/
☆ STeP-Diff: Spatio-Temporal Physics-Informed Diffusion Models for Mobile Fine-Grained Pollution Forecasting
Nan Zhou, Weijie Hong, Huandong Wang, Jianfeng Zheng, Qiuhua Wang, Yali Song, Xiao-Ping Zhang, Yong Li, Xinlei Chen
Fine-grained air pollution forecasting is crucial for urban management and the development of healthy buildings. Deploying portable sensors on mobile platforms such as cars and buses offers a low-cost, easy-to-maintain, and wide-coverage data collection solution. However, due to the random and uncontrollable movement patterns of these non-dedicated mobile platforms, the resulting sensor data are often incomplete and temporally inconsistent. By exploring potential training patterns in the reverse process of diffusion models, we propose Spatio-Temporal Physics-Informed Diffusion Models (STeP-Diff). STeP-Diff leverages DeepONet to model the spatial sequence of measurements along with a PDE-informed diffusion model to forecast the spatio-temporal field from incomplete and time-varying data. Through a PDE-constrained regularization framework, the denoising process asymptotically converges to the convection-diffusion dynamics, ensuring that predictions are both grounded in real-world measurements and aligned with the fundamental physics governing pollution dispersion. To assess the performance of the system, we deployed 59 self-designed portable sensing devices in two cities, operating for 14 days to collect air pollution data. Compared to the second-best performing algorithm, our model achieved improvements of up to 89.12% in MAE, 82.30% in RMSE, and 25.00% in MAPE, with extensive evaluations demonstrating that STeP-Diff effectively captures the spatio-temporal dependencies in air pollution fields.
☆ MAFNet:Multi-frequency Adaptive Fusion Network for Real-time Stereo Matching
Ao Xu, Rujin Zhao, Xiong Xu, Boceng Huang, Yujia Jia, Hongfeng Long, Fuxuan Chen, Zilong Cao, Fangyuan Chen
Existing stereo matching networks typically rely on either cost-volume construction based on 3D convolutions or deformation methods based on iterative optimization. The former incurs significant computational overhead during cost aggregation, whereas the latter often lacks the ability to model non-local contextual information. These methods exhibit poor compatibility on resource-constrained mobile devices, limiting their deployment in real-time applications. To address this, we propose a Multi-frequency Adaptive Fusion Network (MAFNet), which can produce high-quality disparity maps using only efficient 2D convolutions. Specifically, we design an adaptive frequency-domain filtering attention module that decomposes the full cost volume into high-frequency and low-frequency volumes, performing frequency-aware feature aggregation separately. Subsequently, we introduce a Linformer-based low-rank attention mechanism to adaptively fuse high- and low-frequency information, yielding more robust disparity estimation. Extensive experiments demonstrate that the proposed MAFNet significantly outperforms existing real-time methods on public datasets such as Scene Flow and KITTI 2015, showing a favorable balance between accuracy and real-time performance.
☆ Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignment WACV
Kai-Po Chang, Wei-Yuan Cheng, Chi-Pin Huang, Fu-En Yang, Yu-Chiang Frank Wang
Recent advancement in multimodal LLMs (MLLMs) has demonstrated their remarkable capability to generate descriptive captions for input videos. However, these models suffer from factual inaccuracies in the generated descriptions, causing severe hallucination issues. While prior works have explored alleviating hallucinations for static images, jointly mitigating visual object and temporal action hallucinations for dynamic videos remains a challenging and unsolved task. To tackle this challenge, we propose a Self-Augmented Contrastive Alignment (SANTA) framework for enabling object and action faithfulness by exempting the spurious correlations and enforcing the emphasis on visual facts. SANTA employs a hallucinative self-augmentation scheme to identify the potential hallucinations that lie in the MLLM and transform the original captions to the contrasted negatives. Furthermore, we develop a tracklet-phrase contrastive alignment to match the regional objects and relation-guided actions with their corresponding visual and temporal phrases. Extensive experiments demonstrate that SANTA outperforms existing methods in alleviating object and action hallucinations, yielding superior performance on the hallucination examination benchmarks.
comment : IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2026. Project page: https://kpc0810.github.io/santa/
♻ ☆ CoCoIns: Consistent Subject Generation via Contrastive Instantiated Concepts
Lee Hsin-Ying, Kelvin C. K. Chan, Ming-Hsuan Yang
While text-to-image generative models can synthesize diverse and faithful content, subject variation across multiple generations limits their application to long-form content generation. Existing approaches require time-consuming fine-tuning, reference images for all subjects, or access to previously generated content. We introduce Contrastive Concept Instantiation (CoCoIns), a framework that effectively synthesizes consistent subjects across multiple independent generations. The framework consists of a generative model and a mapping network that transforms input latent codes into pseudo-words associated with specific concept instances. Users can generate consistent subjects by reusing the same latent codes. To construct such associations, we propose a contrastive learning approach that trains the network to distinguish between different combinations of prompts and latent codes. Extensive evaluations on human faces with a single subject show that CoCoIns performs comparably to existing methods while maintaining greater flexibility. We also demonstrate the potential for extending CoCoIns to multiple subjects and other object categories.
comment : TMLR 2025. Project page: https://contrastive-concept-instantiation.github.io
♻ ☆ MORPH: PDE Foundation Models with Arbitrary Data Modality
Mahindra Singh Rautela, Alexander Most, Siddharth Mansingh, Bradley C. Love, Ayan Biswas, Diane Oyen, Earl Lawrence
We introduce MORPH, a modality-agnostic, autoregressive foundation model for partial differential equations (PDEs). MORPH is built on a convolutional vision transformer backbone that seamlessly handles heterogeneous spatiotemporal datasets of varying data modality (1D--3D) at different resolutions, and multiple fields with mixed scalar and vector components. The architecture combines (i) component-wise convolution, which jointly processes scalar and vector channels to capture local interactions, (ii) inter-field cross-attention, which models and selectively propagates information between different physical fields, (iii) axial attentions, which factorize full spatiotemporal self-attention along individual spatial and temporal axes to reduce computational burden while retaining expressivity. We pretrain multiple model variants on a diverse collection of heterogeneous PDE datasets and evaluate transfer to a range of downstream prediction tasks. Using both full-model fine-tuning and parameter-efficient low-rank adapters (LoRA), MORPH outperforms models trained from scratch. Across extensive evaluations, MORPH matches or surpasses strong baselines and recent state-of-the-art models. Collectively, these capabilities present a flexible and powerful backbone for learning from the heterogeneous and multimodal nature of scientific observations, charting a path toward scalable and data-efficient scientific machine learning. The source code, datasets, and models are publicly available at https://github.com/lanl/MORPH.
♻ ☆ Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Model s
Shuai Wang, Zhenhua Liu, Jiaheng Wei, Xuanwu Yin, Dong Li, Emad Barsoum
We present Athena-PRM, a multimodal process reward model (PRM) designed to evaluate the reward score for each step in solving complex reasoning problems. Developing high-performance PRMs typically demands significant time and financial investment, primarily due to the necessity for step-level annotations of reasoning steps. Conventional automated labeling methods, such as Monte Carlo estimation, often produce noisy labels and incur substantial computational costs. To efficiently generate high-quality process-labeled data, we propose leveraging prediction consistency between weak and strong completers as a criterion for identifying reliable process labels. Remarkably, Athena-PRM demonstrates outstanding effectiveness across various scenarios and benchmarks with just 5,000 samples. Furthermore, we also develop two effective strategies to improve the performance of PRMs: ORM initialization and up-sampling for negative data. We validate our approach in three specific scenarios: verification for test time scaling, direct evaluation of reasoning step correctness, and reward ranked fine-tuning. Our Athena-PRM consistently achieves superior performance across multiple benchmarks and scenarios. Notably, when using Qwen2.5-VL-7B as the policy model, Athena-PRM enhances performance by 10.2 points on WeMath and 7.1 points on MathVista for test time scaling. Furthermore, Athena-PRM sets the state-of-the-art (SoTA) results in VisualProcessBench and outperforms the previous SoTA by 3.9 F1-score, showcasing its robust capability to accurately assess the correctness of the reasoning step. Additionally, utilizing Athena-PRM as the reward model, we develop Athena-7B with reward ranked fine-tuning and outperforms baseline with a significant margin on five benchmarks.
♻ ☆ Beyond I-Con: Exploring New Dimension of Distance Measures in Representation Learning
Jasmine Shone, Zhening Li, Shaden Alshammari, Mark Hamilton, William Freeman
The Information Contrastive (I-Con) framework revealed that over 23 representation learning methods implicitly minimize KL divergence between data and learned distributions that encode similarities between data points. However, a KL-based loss may be misaligned with the true objective, and properties of KL divergence such as asymmetry and unboundedness may create optimization challenges. We present Beyond I-Con, a framework that enables systematic discovery of novel loss functions by exploring alternative statistical divergences. Key findings: (1) on unsupervised clustering of DINO-ViT embeddings, we achieve state-of-the-art results by modifying the PMI algorithm to use total variation (TV) distance; (2) supervised contrastive learning with Euclidean distance as the feature space metric is improved by replacing the standard loss function with Jenson-Shannon divergence (JSD); (3) on dimensionality reduction, we achieve superior qualitative results and better performance on downstream tasks than SNE by replacing KL with a bounded $f$-divergence. Our results highlight the importance of considering divergence choices in representation learning optimization.
♻ ☆ SAGE: Spatial-visual Adaptive Graph Exploration for Visual Place Recognition
Shunpeng Chen, Changwei Wang, Rongtao Xu, Xingtian Pei, Yukun Song, Jinzhou Lin, Wenhao Xu, Jingyi Zhang, Li Guo, Shibiao Xu
Visual Place Recognition (VPR) requires robust retrieval of geotagged images despite large appearance, viewpoint, and environmental variation. Prior methods focus on descriptor fine-tuning or fixed sampling strategies yet neglect the dynamic interplay between spatial context and visual similarity during training. We present SAGE (Spatial-visual Adaptive Graph Exploration), a unified training pipeline that enhances granular spatial-visual discrimination by jointly improving local feature aggregation, organize samples during training, and hard sample mining. We introduce a lightweight Soft Probing module that learns residual weights from training data for patch descriptors before bilinear aggregation, boosting distinctive local cues. During training we reconstruct an online geo-visual graph that fuses geographic proximity and current visual similarity so that candidate neighborhoods reflect the evolving embedding landscape. To concentrate learning on the most informative place neighborhoods, we seed clusters from high-affinity anchors and iteratively expand them with a greedy weighted clique expansion sampler. Implemented with a frozen DINOv2 backbone and parameter-efficient fine-tuning, SAGE achieves SOTA across eight benchmarks. It attains 98.9%, 95.8%, 94.5%, and 96.0% Recall@1 on SPED, Pitts30k-test, MSLS-val, and Nordland, respectively. Notably, our method obtains 100% Recall@10 on SPED only using 4096D global descriptors. Code and models will be released upon acceptance.
comment : 23 pages
♻ ☆ Efficient stereo matching on embedded GPUs with zero-means cross cor relation
Qiong Chang, Aolong Zha, Weimin Wang, Xin Liu, Masaki Onishi, Lei Lei, Meng Joo Er, Tsutomu Maruyama
Mobile stereo-matching systems have become an important part of many applications, such as automated-driving vehicles and autonomous robots. Accurate stereo-matching methods usually lead to high computational complexity; however, mobile platforms have only limited hardware resources to keep their power consumption low; this makes it difficult to maintain both an acceptable processing speed and accuracy on mobile platforms. To resolve this trade-off, we herein propose a novel acceleration approach for the well-known zero-means normalized cross correlation (ZNCC) matching cost calculation algorithm on a Jetson Tx2 embedded GPU. In our method for accelerating ZNCC, target images are scanned in a zigzag fashion to efficiently reuse one pixel's computation for its neighboring pixels; this reduces the amount of data transmission and increases the utilization of on-chip registers, thus increasing the processing speed. As a result, our method is 2X faster than the traditional image scanning method, and 26% faster than the latest NCC method. By combining this technique with the domain transformation (DT) algorithm, our system show real-time processing speed of 32 fps, on a Jetson Tx2 GPU for 1,280x384 pixel images with a maximum disparity of 128. Additionally, the evaluation results on the KITTI 2015 benchmark show that our combined system is more accurate than the same algorithm combined with census by 7.26%, while maintaining almost the same processing speed. Source Code: https://github.com/changqiong/Z2ZNCC.git
♻ ☆ SO-Bench: A Structural Output Evaluation of Multimodal LLMs
Di Feng, Kaixin Ma, Feng Nan, Haofeng Chen, Bohan Zhai, David Griffiths, Mingfei Gao, Zhe Gan, Eshan Verma, Yinfei Yang, Zhifeng Chen, Afshin Dehghan
Multimodal large language models (MLLMs) are increasingly deployed in real-world, agentic settings where outputs must not only be correct, but also conform to predefined data schemas. Despite recent progress in structured generation in textual domain, there is still no benchmark that systematically evaluates schema-grounded information extraction and reasoning over visual inputs. In this work, we conduct a comprehensive study of visual structural output capabilities for MLLMs with our carefully designed SO-Bench benchmark. Covering four visual domains, including UI screens, natural images, documents, and charts, SO-Bench is built from over 6.5K diverse JSON schemas and 1.8K curated image-schema pairs with human-verified quality. Benchmarking experiments on open-sourced and frontier proprietary models reveal persistent gaps in predicting accurate, schema compliant outputs, highlighting the need for better multimodal structured reasoning. Beyond benchmarking, we further conduct training experiments to largely improve the model's structured output capability. We plan to make the benchmark available to the community.
comment : v2 preprint. Fixed some typos, add a discussion about limitation, provide pseudo-codes for eval
♻ ☆ SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation
Marshall Thomas, Edward Fish, Richard Bowden
Despite progress in gloss-free Sign Language Translation (SLT), traditional single modality end-to-end approaches consistently fail on two critical components of natural signing: the precise recognition of high-speed fingerspelling and the integration of asynchronous non-manual cues from the face. Recent progress in SLT with Large Language Models has side stepped this challenge, forcing a single network to learn these simultaneously resulting in poor performance when tasked with translating crucial information such as names, places, and technical terms. We introduce SignBind-LLM, a modular framework designed to overcome these limitations. Our approach employs separate, specialized predictors for continuous signing, fingerspelling, and lipreading. Each expert network first decodes its specific modality into a sequence of tokens. These parallel streams are then fused by a lightweight transformer that resolves temporal misalignments before passing the combined representation to a Large Language Model (LLM) for final sentence generation. Our method establishes a new state-of-the-art on the How2Sign, ChicagoFSWildPlus, and BOBSL datasets with a BLEU-4 score of 22.1, 73.2% letter accuracy and BLEU-4 score of 6.8 respectively. These results validate our core hypothesis: isolating and solving distinct recognition tasks before fusion provides a more powerful and effective pathway to robust, high-fidelity sign language translation.
♻ ☆ MMHOI: Modeling Complex 3D Multi-Human Multi-Object Interactions WACV 2026
Kaen Kogashi, Anoop Cherian, Meng-Yu Jennifer Kuo
Real-world scenes often feature multiple humans interacting with multiple objects in ways that are causal, goal-oriented, or cooperative. Yet existing 3D human-object interaction (HOI) benchmarks consider only a fraction of these complex interactions. To close this gap, we present MMHOI -- a large-scale, Multi-human Multi-object Interaction dataset consisting of images from 12 everyday scenarios. MMHOI offers complete 3D shape and pose annotations for every person and object, along with labels for 78 action categories and 14 interaction-specific body parts, providing a comprehensive testbed for next-generation HOI research. Building on MMHOI, we present MMHOI-Net, an end-to-end transformer-based neural network for jointly estimating human-object 3D geometries, their interactions, and associated actions. A key innovation in our framework is a structured dual-patch representation for modeling objects and their interactions, combined with action recognition to enhance the interaction prediction. Experiments on MMHOI and the recently proposed CORE4D datasets demonstrate that our approach achieves state-of-the-art performance in multi-HOI modeling, excelling in both accuracy and reconstruction quality. The MMHOI dataset is publicly available at https://zenodo.org/records/17711786.
comment : Accepted to WACV 2026
♻ ☆ "I Can See Forever!": Evaluating Real-time VideoLLMs for Assisting Individuals with Visual Impairments
Ziyi Zhang, Zhen Sun, Zongmin Zhang, Zifan Peng, Yuemeng Zhao, Zichun Wang, Zeren Luo, Ruiting Zuo, Xinlei He
The visually impaired population faces significant challenges in daily activities. While prior works employ vision language models for assistance, most focus on static content and cannot address real-time perception needs in complex environments. Recent VideoLLMs enable real-time vision and speech interaction, offering promising potential for assistive tasks. In this work, we conduct the first study evaluating their effectiveness in supporting daily life for visually impaired individuals. We first conducted a user survey with visually impaired participants to design the benchmark VisAssistDaily for daily life evaluation. Using VisAssistDaily, we evaluate popular VideoLLMs and find GPT-4o achieves the highest task success rate. We further conduct a user study to reveal concerns about hazard perception. To address this, we propose SafeVid, an environment-awareness dataset, and fine-tune VITA-1.5, improving risk recognition accuracy from 25.00% to 76.00%.We hope this work provides valuable insights and inspiration for future research in this field.
comment : 17 pages
♻ ☆ Flowing Backwards: Improving Normalizing Flows via Reverse Representation Alignment AAAI 2026
Yang Chen, Xiaowei Xu, Shuai Wang, Chenhui Zhu, Ruxue Wen, Xubin Li, Tiezheng Ge, Limin Wang
Normalizing Flows (NFs) are a class of generative models distinguished by a mathematically invertible architecture, where the forward pass transforms data into a latent space for density estimation, and the reverse pass generates new samples from this space. This characteristic creates an intrinsic synergy between representation learning and data generation. However, the generative quality of standard NFs is limited by poor semantic representations from log-likelihood optimization. To remedy this, we propose a novel alignment strategy that creatively leverages the invertibility of NFs: instead of regularizing the forward pass, we align the intermediate features of the generative (reverse) pass with representations from a powerful vision foundation model, demonstrating superior effectiveness over naive alignment. We also introduce a novel training-free, test-time optimization algorithm for classification, which provides a more intrinsic evaluation of the NF's embedded semantic knowledge. Comprehensive experiments demonstrate that our approach accelerates the training of NFs by over 3.3$\times$, while simultaneously delivering significant improvements in both generative quality and classification accuracy. New state-of-the-art results for NFs are established on ImageNet 64$\times$64 and 256$\times$256. Our code is available at https://github.com/MCG-NJU/FlowBack.
comment : Accepted by AAAI 2026
♻ ☆ Polygon Intersection-over-Union Loss for Viewpoint-Agnostic Monocular 3D Vehicle Detection
Xinxuan Lu, Derek Gloudemans, Shepard Xia, Daniel B. Work
Monocular 3D object detection is a challenging task because depth information is difficult to obtain from 2D images. A subset of viewpoint-agnostic monocular 3D detection methods also do not explicitly leverage scene homography or geometry during training, meaning that a model trained thusly can detect objects in images from arbitrary viewpoints. Such works predict the projections of the 3D bounding boxes on the image plane to estimate the location of the 3D boxes, but these projections are not rectangular so the calculation of IoU between these projected polygons is not straightforward. This work proposes an efficient, fully differentiable algorithm for the calculation of IoU between two convex polygons, which can be utilized to compute the IoU between two 3D bounding box footprints viewed from an arbitrary angle. We test the performance of the proposed polygon IoU loss (PIoU loss) on three state-of-the-art viewpoint-agnostic 3D detection models. Experiments demonstrate that the proposed PIoU loss converges faster than L1 loss and that in 3D detection models, a combination of PIoU loss and L1 loss gives better results than L1 loss alone (+1.64% AP70 for MonoCon on cars, +0.18% AP70 for RTM3D on cars, and +0.83%/+2.46% AP50/AP25 for MonoRCNN on cyclists).
♻ ☆ DAVE: Diagnostic benchmark for Audio Visual Evaluation
Gorjan Radevski, Teodora Popordanoska, Matthew B. Blaschko, Tinne Tuytelaars
Audio-visual understanding is a rapidly evolving field that seeks to integrate and interpret information from both auditory and visual modalities. Despite recent advances in multi-modal learning, existing benchmarks often suffer from strong visual bias -- when answers can be inferred from visual data alone -- and provide only aggregate scores that conflate multiple sources of error. This makes it difficult to determine whether models struggle with visual understanding, audio interpretation, or audio-visual alignment. In this work, we introduce DAVE: Diagnostic Audio Visual Evaluation, a novel benchmark dataset designed to systematically evaluate audio-visual models across controlled settings. DAVE alleviates existing limitations by (i) ensuring both modalities are necessary to answer correctly and (ii) decoupling evaluation into atomic subcategories. Our detailed analysis of state-of-the-art models reveals specific failure modes and provides targeted insights for improvement. By offering this standardized diagnostic framework, we aim to facilitate more robust development of audio-visual models. Dataset: https://huggingface.co/datasets/gorjanradevski/dave Code: https://github.com/gorjanradevski/dave
comment : First two authors contributed equally
♻ ☆ EoS-FM: Can an Ensemble of Specialist Models act as a Generalist Feature Extractor?
Pierre Adorni, Minh-Tan Pham, Stéphane May, Sébastien Lefèvre
Recent advances in foundation models have shown great promise in domains such as natural language processing and computer vision, and similar efforts are now emerging in the Earth Observation community. These models aim to generalize across tasks with limited supervision, reducing the need for training separate models for each task. However, current strategies, which largely focus on scaling model size and dataset volume, require prohibitive computational and data resources, limiting accessibility to only a few large institutions. Moreover, this paradigm of ever-larger models stands in stark contrast with the principles of sustainable and environmentally responsible AI, as it leads to immense carbon footprints and resource inefficiency. In this work, we present a novel and efficient alternative: an Ensemble-of-Specialists framework for building Remote Sensing Foundation Models (RSFMs). Our method decomposes the training process into lightweight, task-specific ConvNeXtV2 specialists that can be frozen and reused. This modular approach offers strong advantages in efficiency, interpretability, and extensibility. Moreover, it naturally supports federated training, pruning, and continuous specialist integration, making it particularly well-suited for collaborative and resource-constrained settings. Our framework sets a new direction for building scalable and efficient RSFMs. All codes and pretrained models are available at https://github.com/pierreadorni/EoS-FM.
♻ ☆ TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agent s AAAI 2026
Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song-Chun Zhu, Qing Li
Building Graphical User Interface (GUI) agents is a promising research direction, which simulates human interaction with computers or mobile phones to perform diverse GUI tasks. However, a major challenge in developing generalized GUI agents is the lack of sufficient trajectory data across various operating systems and applications, mainly due to the high cost of manual annotations. In this paper, we propose the TongUI framework that builds generalized GUI agents by learning from rich multimodal web tutorials. Concretely, we crawl and process online GUI tutorials (such as videos and articles) into GUI agent trajectory data, through which we produce the GUI-Net dataset containing 143K trajectory data across five operating systems and more than 200 applications. We develop the TongUI agent by fine-tuning Qwen2.5-VL-3B/7B models on GUI-Net, which show remarkable performance improvements on commonly used grounding and navigation benchmarks, outperforming baseline agents about 10\% on multiple benchmarks, showing the effectiveness of the GUI-Net dataset and underscoring the significance of our TongUI framework. We will fully open-source the code, the GUI-Net dataset, and the trained models soon.
comment : AAAI 2026
♻ ☆ ENTIRE: Learning-based Volume Rendering Time Prediction
Zikai Yin, Hamid Gadirov, Jiri Kosinka, Steffen Frey
We introduce ENTIRE, a novel deep learning-based approach for fast and accurate volume rendering time prediction. Predicting rendering time is inherently challenging due to its dependence on multiple factors, including volume data characteristics, image resolution, camera configuration, and transfer function settings. Our method addresses this by first extracting a feature vector that encodes structural volume properties relevant to rendering performance. This feature vector is then integrated with additional rendering parameters, such as image resolution, camera setup, and transfer function settings, to produce the final prediction. We evaluate ENTIRE across multiple rendering frameworks (CPU- and GPU-based) and configurations (with and without single-scattering) on diverse datasets. The results demonstrate that our model achieves high prediction accuracy with fast inference speed. Furthermore, we showcase ENTIRE's effectiveness in two case studies, where it enables dynamic parameter adaptation for stable frame rates and load balancing.
♻ ☆ JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization
Yunlong Lin, Linqing Wang, Kunjie Lin, Zixu Lin, Kaixiong Gong, Wenbo Li, Bin Lin, Zhenxi Li, Shiyi Zhang, Yuyang Peng, Wenxun Dai, Xinghao Ding, Chunyu Wang, Qinglin Lu
Agent-based editing models have substantially advanced interactive experiences, processing quality, and creative flexibility. However, two critical challenges persist: (1) instruction hallucination, text-only chain-of-thought (CoT) reasoning cannot fully prevent factual errors due to inherent information bottlenecks; (2) reward hacking, dynamic policy optimization against static reward models allows agents to exploit flaws in reward functions. To address these issues, we propose JarvisEvo, a unified image editing agent that emulates an expert human designer by iteratively editing, selecting appropriate tools, evaluating results, and reflecting on its own decisions to refine outcomes. JarvisEvo offers three key advantages: (1) an interleaved multimodal chain-of-thought (iMCoT) reasoning mechanism that enhances instruction following and editing quality; (2) a synergistic editor-evaluator policy optimization (SEPO) framework that enables self-improvement without external rewards, effectively mitigating reward hacking; and (3) support for both global and local fine-grained editing through seamless integration of Adobe Lightroom. On ArtEdit-Bench, JarvisEvo outperforms Nano-Banana by an average of 18.95% on preservative editing metrics, including a substantial 44.96% improvement in pixel-level content fidelity. Project page: https://jarvisevo.vercel.app/
comment : 31 pages, 18 figures
♻ ☆ Integrating Skeleton Based Representations for Robust Yoga Pose Classification Using Deep Learning Models
Mohammed Mohiuddin, Syed Mohammod Minhaz Hossain, Sumaiya Khanam, Prionkar Barua, Aparup Barua, MD Tamim Hossain
Yoga is a popular form of exercise worldwide due to its spiritual and physical health benefits, but incorrect postures can lead to injuries. Automated yoga pose classification has therefore gained importance to reduce reliance on expert practitioners. While human pose keypoint extraction models have shown high potential in action recognition, systematic benchmarking for yoga pose recognition remains limited, as prior works often focus solely on raw images or a single pose extraction model. In this study, we introduce a curated dataset, 'Yoga-16', which addresses limitations of existing datasets, and systematically evaluate three deep learning architectures (VGG16, ResNet50, and Xception), using three input modalities (direct images, MediaPipe Pose skeleton images, and YOLOv8 Pose skeleton images). Our experiments demonstrate that skeleton-based representations outperform raw image inputs, with the highest accuracy of 96.09% achieved by VGG16 with MediaPipe Pose skeleton input. Additionally, we provide interpretability analysis using Grad-CAM, offering insights into model decision-making for yoga pose classification with cross-validation analysis.
♻ ☆ GigaBrain-0: A Wo rl d Model-Powered Vision -Language-Action Model
GigaBrain Team, Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Haoyun Li, Jie Li, Jiagang Zhu, Lv Feng, Peng Li, Qiuping Deng, Runqi Ouyang, Wenkang Qin, Xinze Chen, Xiaofeng Wang, Yang Wang, Yifan Li, Yilong Li, Yiran Ding, Yuan Xu, Yun Ye, Yukun Zhou, Zhehao Dong, Zhenan Wang, Zhichao Liu, Zheng Zhu
Training Vision-Language-Action (VLA) models for generalist robots typically requires large-scale real-world robot data, which is expensive and time-consuming to collect. The inefficiency of physical data collection severely limits the scalability, and generalization capacity of current VLA systems. To address this challenge, we introduce GigaBrain-0, a novel VLA foundation model empowered by world model-generated data (e.g., video generation, real2real transfer, human transfer, view transfer, sim2real transfer data). By leveraging world models to generate diverse data at scale, GigaBrain-0 significantly reduces reliance on real robot data while improving cross-task generalization. Our approach further improves policy robustness through RGBD input modeling and embodied Chain-of-Thought (CoT) supervision, enabling the model to reason about spatial geometry, object states, and long-horizon dependencies during task execution. This leads to substantial gains in real-world performance on dexterous, long-horizon, and mobile manipulation tasks. Extensive experiments demonstrate that GigaBrain-0 achieves superior generalization across variations in appearances (e.g., textures, colors), object placements, and camera viewpoints. Additionally, we present GigaBrain-0-Small, an optimized lightweight variant designed to run efficiently on devices such as the NVIDIA Jetson AGX Orin.
comment : https://gigabrain0.github.io/
♻ ☆ Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs
Xintong Zhang, Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaowen Zhang, Yang Liu, Tao Yuan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, Qing Li
Vision language models (VLMs) have achieved impressive performance across a variety of computer vision tasks. However, the multimodal reasoning capability has not been fully explored in existing models. In this paper, we propose a Chain-of-Focus (CoF) method that allows VLMs to perform adaptive focusing and zooming in on key image regions based on obtained visual cues and the given questions, achieving efficient multimodal reasoning. To enable this CoF capability, we present a two-stage training pipeline, including supervised fine-tuning (SFT) and reinforcement learning (RL). In the SFT stage, we construct the MM-CoF dataset, comprising 3K samples derived from a visual agent designed to adaptively identify key regions to solve visual tasks with different image resolutions and questions. We use MM-CoF to fine-tune the Qwen2.5-VL model for cold start. In the RL stage, we leverage the outcome accuracies and formats as rewards to update the Qwen2.5-VL model, enabling further refining the search and reasoning strategy of models without human priors. Our model achieves significant improvements on multiple benchmarks. On the V* benchmark that requires strong visual reasoning capability, our model outperforms existing VLMs by 5% among 8 image resolutions ranging from 224 to 4K, demonstrating the effectiveness of the proposed CoF method and facilitating the more efficient deployment of VLMs in practical applications.
comment : https://github.com/xtong-zhang/Chain-of-Focus
♻ ☆ OP-Align: Object-level and Part-level Alignment for Self-supervised Category-level Articulated Object Pose Estimation ECCV2024
Yuchen Che, Ryo Furukawa, Asako Kanezaki
Category-level articulated object pose estimation focuses on the pose estimation of unknown articulated objects within known categories. Despite its significance, this task remains challenging due to the varying shapes and poses of objects, expensive dataset annotation costs, and complex real-world environments. In this paper, we propose a novel self-supervised approach that leverages a single-frame point cloud to solve this task. Our model consistently generates reconstruction with a canonical pose and joint state for the entire input object, and it estimates object-level poses that reduce overall pose variance and part-level poses that align each part of the input with its corresponding part of the reconstruction. Experimental results demonstrate that our approach significantly outperforms previous self-supervised methods and is comparable to the state-of-the-art supervised methods. To assess the performance of our model in real-world scenarios, we also introduce a new real-world articulated object benchmark dataset.
comment : published in ECCV2024
♻ ☆ Turbo-GS: Accelerating 3D Gaussian Fitting for High-Quality Radiance Fields
Ankit Dhiman, Tao Lu, R Srinath, Emre Arslan, Angela Xing, Yuanbo Xiangli, R Venkatesh Babu, Srinath Sridhar
Novel-view synthesis is an important problem in computer vision with applications in 3D reconstruction, mixed reality, and robotics. Recent methods like 3D Gaussian Splatting (3DGS) have become the preferred method for this task, providing high-quality novel views in real time. However, the training time of a 3DGS model is slow, often taking 30 minutes for a scene with 200 views. In contrast, our goal is to reduce the optimization time by training for fewer steps while maintaining high rendering quality. Specifically, we combine the guidance from both the position error and the appearance error to achieve a more effective densification. To balance the rate between adding new Gaussians and fitting old Gaussians, we develop a convergence-aware budget control mechanism. Moreover, to make the densification process more reliable, we selectively add new Gaussians from mostly visited regions. With these designs, we reduce the Gaussian optimization steps to one-third of the previous approach while achieving a comparable or even better novel view rendering quality. To further facilitate the rapid fitting of 4K resolution images, we introduce a dilation-based rendering technique. Our method, Turbo-GS, speeds up optimization for typical scenes and scales well to high-resolution (4K) scenarios on standard datasets. Through extensive experiments, we show that our method is significantly faster in optimization than other methods while retaining quality. Project page: https://ivl.cs.brown.edu/research/turbo-gs.
comment : Project page: https://ivl.cs.brown.edu/research/turbo-gs
♻ ☆ Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generation
Changsheng Lv, Zijian Fu, Mengshi Qi
In this paper, we propose Robo-SGG, a plug-and-play module for robust scene graph generation (SGG). Unlike standard SGG, the robust scene graph generation aims to perform inference on a diverse range of corrupted images, with the core challenge being the domain shift between the clean and corrupted images. Existing SGG methods suffer from degraded performance due to shifted visual features (e.g., corruption interference or occlusions). To obtain robust visual features, we leverage layout information, representing the global structure of an image, which is robust to domain shift, to enhance the robustness of SGG methods under corruption. Specifically, we employ Instance Normalization (IN) to alleviate the domain-specific variations and recover the robust structural features (i.e., the positional and semantic relationships among objects) by the proposed Layout-Oriented Restitution. Furthermore, under corrupted images, we introduce a Layout-Embedded Encoder (LEE) that adaptively fuses layout and visual features via a gating mechanism, enhancing the robustness of positional and semantic representations for objects and predicates. Note that our proposed Robo-SGG module is designed as a plug-and-play component, which can be easily integrated into any baseline SGG model. Extensive experiments demonstrate that by integrating the state-of-the-art method into our proposed Robo-SGG, we achieve relative improvements of 6.3%, 11.1%, and 8.0% in mR@50 for PredCls, SGCls, and SGDet tasks on the VG-C benchmark, respectively, and achieve new state-of-the-art performance in the corruption scene graph generation benchmark (VG-C and GQA-C). We will release our source code and model.
♻ ☆ Multimodal Adversarial Defense for Vision -Language Models by Leveraging One-To-Many Relation ships WACV 2026
Futa Waseda, Antonio Tejero-de-Pablos, Isao Echizen
Pre-trained vision-language (VL) models are highly vulnerable to adversarial attacks. However, existing defense methods primarily focus on image classification, overlooking two key aspects of VL tasks: multimodal attacks, where both image and text can be perturbed, and the one-to-many relationship of images and texts, where a single image can correspond to multiple textual descriptions and vice versa (1:N and N:1). This work is the first to explore defense strategies against multimodal attacks in VL tasks, whereas prior VL defense methods focus on vision robustness. We propose multimodal adversarial training (MAT), which incorporates adversarial perturbations in both image and text modalities during training, significantly outperforming existing unimodal defenses. Furthermore, we discover that MAT is limited by deterministic one-to-one (1:1) image-text pairs in VL training data. To address this, we conduct a comprehensive study on leveraging one-to-many relationships to enhance robustness, investigating diverse augmentation techniques. Our analysis shows that, for a more effective defense, augmented image-text pairs should be well-aligned, diverse, yet avoid distribution shift -- conditions overlooked by prior research. This work pioneers defense strategies against multimodal attacks, providing insights for building robust VLMs from both optimization and data perspectives. Our code is publicly available at https://github.com/CyberAgentAI/multimodal-adversarial-training.
comment : WACV 2026 Accepted. Code available at https://github.com/CyberAgentAI/multimodal-adversarial-training
♻ ☆ TTRV: Test-Time Reinforcement Learning for Vision Language Models
Akshit Singh, Shyam Marjit, Wei Lin, Paul Gavrikov, Serena Yeung-Levy, Hilde Kuehne, Rogerio Feris, Sivan Doveh, James Glass, M. Jehanzeb Mirza
Existing methods for extracting reward signals in Reinforcement Learning typically rely on labeled data and dedicated training splits, a setup that contrasts with how humans learn directly from their environment. In this work, we propose TTRV to enhance vision language understanding by adapting the model on the fly at inference time, without the need for any labeled data. Concretely, we enhance the Group Relative Policy Optimization (GRPO) framework by designing rewards based on the frequency of the base model's output, while inferring on each test sample multiple times. Further, we also propose to control the diversity of the model's output by simultaneously rewarding the model for obtaining low entropy of the output empirical distribution. Our approach delivers consistent gains across both object recognition and visual question answering (VQA), with improvements of up to 52.4% and 29.8%, respectively, and average boosts of 24.6% and 10.0% across 16 datasets. Remarkably, on image recognition, TTRV applied to InternVL 8B surpasses GPT-4o by an average of 2.3% over 8 benchmarks, while remaining highly competitive on VQA, demonstrating that test-time reinforcement learning can match or exceed the strongest proprietary models. Finally, we find many interesting properties of test-time RL for VLMs: for example, even in extremely data-constrained scenarios, where adaptation is performed on a single randomly chosen unlabeled test example, TTRV still yields non-trivial improvements of up to 5.5% in recognition tasks.
♻ ☆ A lightweight detector for real-time detection of remote sensing images
Qianyi Wang, Guoqiang Ren
Remote sensing imagery is widely used across various fields, yet real-time detection remains challenging due to the prevalence of small objects and the need to balance accuracy with efficiency. To address this, we propose DMG-YOLO, a lightweight real-time detector tailored for small object detection in remote sensing images. Specifically, we design a Dual-branch Feature Extraction (DFE) module in the backbone, which partitions feature maps into two parallel branches: one extracts local features via depthwise separable convolutions, and the other captures global context using a vision transformer with a gating mechanism. Additionally, a Multi-scale Feature Fusion (MFF) module with dilated convolutions enhances multi-scale integration while preserving fine details. In the neck, we introduce the Global and Local Aggregate Feature Pyramid Network (GLAFPN) to further boost small object detection through global-local feature fusion. Extensive experiments on the VisDrone2019 and NWPU VHR-10 datasets show that DMG-YOLO achieves competitive performance in terms of mAP, model size, and other key metrics.
comment : wrong results
♻ ☆ Jina-VLM: Small Multilingual Vision Language Model
Andreas Koukounas, Georgios Mastrapas, Florian Hönicke, Sedigheh Eslami, Guillaume Roncari, Scott Martens, Han Xiao
We present Jina-VLM, a 2.4B parameter vision-language model that achieves state-of-the-art multilingual visual question answering among open 2B-scale VLMs. The model couples a SigLIP2 vision encoder with a Qwen3 language backbone through an attention-pooling connector that enables token-efficient processing of arbitrary-resolution images. The model achieves leading results on standard VQA benchmarks and multilingual evaluations while preserving competitive text-only performance. Model weights and code are publicly released at https://huggingface.co/jinaai/jina-vlm .
comment : 18 pages, 1-7 main content, 13-18 appendix for tables and dataset
♻ ☆ There is No VAE: End-to-End Pixel-Space Generative Modeling via Self-Supervised Pre-train ing
Jiachen Lei, Keli Liu, Julius Berner, Haiming Yu, Hongkai Zheng, Jiahong Wu, Xiangxiang Chu
Pixel-space generative models are often more difficult to train and generally underperform compared to their latent-space counterparts, leaving a persistent performance and efficiency gap. In this paper, we introduce a novel two-stage training framework that closes this gap for pixel-space diffusion and consistency models. In the first stage, we pre-train encoders to capture meaningful semantics from clean images while aligning them with points along the same deterministic sampling trajectory, which evolves points from the prior to the data distribution. In the second stage, we integrate the encoder with a randomly initialized decoder and fine-tune the complete model end-to-end for both diffusion and consistency models. Our framework achieves state-of-the-art (SOTA) performance on ImageNet. Specifically, our diffusion model reaches an FID of 1.58 on ImageNet-256 and 2.35 on ImageNet-512 with 75 number of function evaluations (NFE) surpassing prior pixel-space methods and VAE-based counterparts by a large margin in both generation quality and training efficiency. In a direct comparison, our model significantly outperforms DiT while using only around 30\% of its training compute.
♻ ☆ EVE: Towards End-to-End Video Subtitle Extraction with Vision -Language Models
Haiyang Yu, Mengyang Zhao, Jinghui Lu, Ke Niu, Yanjie Wang, Weijie Yin, Weitao Jia, Teng Fu, Yang Liu, Jun Liu, Hong Chen
Video subtitles play a crucial role in short videos and movies, as they not only help models better understand video content but also support applications such as video translation and content retrieval. Existing video subtitle extraction methods typically rely on multi-stage frameworks, where errors accumulate across stages and temporal dependencies are underutilized due to frame-wise processing. Moreover, although some Large Vision-Language Models (LVLMs) possess strong OCR capabilities, predicting accurate timestamps for subtitle texts remains challenging. To this end, we propose an End-to-end Video subtitle Extraction framework based on LVLMs, named EVE, which can output subtitles and their timestamps simultaneously. Specifically, we introduce a dual-branch Spatiotemporal Subtitle-Salient (S\textsuperscript{3}) Module that serves as an adapter for LVLMs, capable of representing subtitle-related content and considering inter-frame correlations using only a small number of tokens. Within this module, the Spatial Semantic Context Aggregate branch aggregates high-level global semantics to provide spatial visual contextual information, while the Temporal Subtitle Token Query branch explicitly queries subtitle-relevant tokens while considering temporal correlation across frames. The small number of tokens retained by the S\textsuperscript{3} module are fed to the language model, which then directly outputs the subtitle text along with its timestamps. Furthermore, we construct the first large-scale dataset dedicated to video subtitle extraction, ViSa, containing over 2.5M videos with timestamped and bilingual annotation, thereby providing the community with a well-organized training and evaluation benchmark.
♻ ☆ SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision -Language Models
Ruosen Zhao, Zhikang Zhang, Jialei Xu, Jiahao Chang, Dong Chen, Lingyun Li, Weijian Sun, Zizhuang Wei
Large vision-language models (VLMs) show strong multimodal understanding but still struggle with 3D spatial reasoning, such as distance estimation, size comparison, and cross-view consistency. Existing 3D-aware methods either depend on auxiliary 3D information or enhance RGB-only VLMs with geometry encoders through shallow feature fusion. We propose SpaceMind, a multimodal large language model explicitly designed for spatial reasoning solely from RGB inputs. The model adopts a dual-encoder architecture, integrating VGGT as a spatial understanding encoder and InternViT as a 2D visual encoder. The key idea is to treat the camera representation as an active guiding modality rather than passive metadata. Specifically, SpaceMind introduces a lightweight Camera-Guided Modality Fusion module before the language model to replace shallow fusion. It applies camera-conditioned biasing to spatial tokens, assigns query-independent weights reflecting their geometric importance, and uses the camera embedding to gate the fused representation. Empirically, SpaceMind establishes new state-of-the-art results on VSI-Bench, SQA3D and SPBench, surpassing both open and proprietary systems on VSI-Bench and SPBench by large margins and achieving state-of-the-art performance on SQA3D. These results demonstrate that camera-guided modality fusion is an effective and practical inductive bias for equipping VLMs with genuinely spatially grounded intelligence. We will release code and model checkpoints to support future research.
♻ ☆ LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling
Zuhao Yang, Sudong Wang, Kaichen Zhang, Keming Wu, Sicong Leng, Yifan Zhang, Bo Li, Chengwei Qin, Shijian Lu, Xingxuan Li, Lidong Bing
Large multimodal models (LMMs) have shown great potential for video reasoning with textual Chain-of-Thought. However, they remain vulnerable to hallucinations, especially when processing long-form videos where evidence is sparse and temporally dispersed. Inspired by how humans comprehend long videos - by first skimming globally and then examining relevant clips for details - we introduce LongVT, an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought. Specifically, we exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames. This global-to-local reasoning loop continues until answers are grounded in retrieved visual evidence. Given the scarcity of fine-grained question-answering (QA) data for the long video reasoning task, we curate and will release a data suite named VideoSIAH to facilitate both training and evaluation. Specifically, our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning, respectively. Our evaluation benchmark consists of 1,280 QA pairs that are carefully curated through a semi-automatic data pipeline with human-in-the-loop validation. With a meticulously designed three-stage training strategy and extensive empirical validation, LongVT consistently outperforms existing strong baselines across four challenging long-video understanding and reasoning benchmarks. Our codes, data, and model checkpoints are publicly available at https://github.com/EvolvingLMMs-Lab/LongVT .
♻ ☆ ImageNot: A contrast with ImageNet preserves model rankings
Olawale Salaudeen, Moritz Hardt
We introduce ImageNot, a dataset constructed explicitly to be drastically different than ImageNet while matching its scale. ImageNot is designed to test the external validity of deep learning progress on ImageNet. We show that key model architectures developed for ImageNet over the years rank identically to how they rank on ImageNet when trained from scratch and evaluated on ImageNot. Moreover, the relative improvements of each model over earlier models strongly correlate in both datasets. Our work demonstrates a surprising degree of external validity in the relative performance of image classification models when trained and evaluated on an entirely different dataset. This stands in contrast with absolute accuracy numbers that typically drop sharply even under small changes to a dataset.
♻ ☆ Weather Prompt : Multi-modality Representation Learning for All-Weather Drone Visual Geo-Localization
Jiahao Wen, Hang Yu, Zhedong Zheng
Visual geo-localization for drones faces critical degradation under weather perturbations, \eg, rain and fog, where existing methods struggle with two inherent limitations: 1) Heavy reliance on limited weather categories that constrain generalization, and 2) Suboptimal disentanglement of entangled scene-weather features through pseudo weather categories. We present WeatherPrompt, a multi-modality learning paradigm that establishes weather-invariant representations through fusing the image embedding with the text context. Our framework introduces two key contributions: First, a Training-free Weather Reasoning mechanism that employs off-the-shelf large multi-modality models to synthesize multi-weather textual descriptions through human-like reasoning. It improves the scalability to unseen or complex weather, and could reflect different weather strength. Second, to better disentangle the scene and weather feature, we propose a multi-modality framework with the dynamic gating mechanism driven by the text embedding to adaptively reweight and fuse visual features across modalities. The framework is further optimized by the cross-modal objectives, including image-text contrastive learning and image-text matching, which maps the same scene with different weather conditions closer in the respresentation space. Extensive experiments validate that, under diverse weather conditions, our method achieves competitive recall rates compared to state-of-the-art drone geo-localization methods. Notably, it improves Recall@1 by +13.37\% under night conditions and by 18.69\% under fog and snow conditions.
♻ ☆ Changes in Gaza: DINOv3-Powered Multi-Class Change Detection for Damage Assessment in Conflict Zones
Kai Zheng, Zhenkai Wu, Fupeng Wei, Miaolan Zhou, Kai Lie, Haitao Guo, Lei Ding, Wei Zhang, Hang-Cheng Dong
Accurately and swiftly assessing damage from conflicts is crucial for humanitarian aid and regional stability. In conflict zones, damaged zones often share similar architectural styles, with damage typically covering small areas and exhibiting blurred boundaries. These characteristics lead to limited data, annotation difficulties, and significant recognition challenges, including high intra-class similarity and ambiguous semantic changes. To address these issues, we introduce a pre-trained DINOv3 model and propose a multi-scale cross-attention difference siamese network (MC-DiSNet). The powerful visual representation capability of the DINOv3 backbone enables robust and rich feature extraction from bi-temporal remote sensing images. The multi-scale cross-attention mechanism allows for precise localization of subtle semantic changes, while the difference siamese structure enhances inter-class feature discrimination, enabling fine-grained semantic change detection. Furthermore, a simple yet powerful lightweight decoder is designed to generate clear detection maps while maintaining high efficiency. We also release a new Gaza-change dataset containing high-resolution satellite image pairs from 2023-2024 with pixel-level semantic change annotations. It is worth emphasizing that our annotations only include semantic pixels of changed areas. We evaluated our method on the Gaza-Change and two classical datasets: the SECOND and Landsat-SCD datasets. Experimental results demonstrate that our proposed approach effectively addresses the MCD task, and its outstanding performance paves the way for practical applications in rapid damage assessment across conflict zones.
♻ ☆ Sliding-Window Merging for Compacting Patch-Redundant Layers in LLMs
Xuan Ding, Rui Sun, Yunjian Zhang, Xiu Yan, Yueqi Zhou, Kaihao Huang, Suzhong Fu, Angelica I Aviles-Rivero, Chuanlong Xie, Yao Zhu
Depth-wise pruning accelerates LLM inference in resource-constrained scenarios but suffers from performance degradation due to direct removal of entire Transformer layers. This paper reveals ``Patch-like'' redundancy across layers via correlation analysis of the outputs of different layers in reproducing kernel Hilbert space, demonstrating consecutive layers exhibit high functional similarity. Building on this observation, this paper proposes Sliding-Window Merging (SWM) - a dynamic compression method that selects consecutive layers from top to bottom using a pre-defined similarity threshold, and compacts patch-redundant layers through a parameter consolidation, thereby simplifying the model structure while maintaining its performance. Extensive experiments on LLMs with various architectures and different parameter scales show that our method outperforms existing pruning techniques in both zero-shot inference performance and retraining recovery quality after pruning. In particular, in the experiment with 35% pruning on the Vicuna-7B model, our method achieved a 1.654% improvement in average performance on zero-shot tasks compared to the existing method. Moreover, we further reveal the potential of combining depth pruning with width pruning to enhance the pruning effect. Our codes are available at https://github.com/920927/SLM-a-sliding-layer-merging-method.
♻ ☆ VEDA: 3D Molecular Generation via Variance-Exploding Diffusion with Annealing
Peining Zhang, Jinbo Bi, Minghu Song
Diffusion models show promise for 3D molecular generation, but face a fundamental trade-off between sampling efficiency and conformational accuracy. While flow-based models are fast, they often produce geometrically inaccurate structures, as they have difficulty capturing the multimodal distributions of molecular conformations. In contrast, denoising diffusion models are more accurate but suffer from slow sampling, a limitation attributed to sub-optimal integration between diffusion dynamics and SE(3)-equivariant architectures. To address this, we propose VEDA, a unified SE(3)-equivariant framework that combines variance-exploding diffusion with annealing to efficiently generate conformationally accurate 3D molecular structures. Specifically, our key technical contributions include: (1) a VE schedule that enables noise injection functionally analogous to simulated annealing, improving 3D accuracy and reducing relaxation energy; (2) a novel preconditioning scheme that reconciles the coordinate-predicting nature of SE(3)-equivariant networks with a residual-based diffusion objective, and (3) a new arcsin-based scheduler that concentrates sampling in critical intervals of the logarithmic signal-to-noise ratio. On the QM9 and GEOM-DRUGS datasets, VEDA matches the sampling efficiency of flow-based models, achieving state-of-the-art valency stability and validity with only 100 sampling steps. More importantly, VEDA's generated structures are remarkably stable, as measured by their relaxation energy during GFN2-xTB optimization. The median energy change is only 1.72 kcal/mol, significantly lower than the 32.3 kcal/mol from its architectural baseline, SemlaFlow. Our framework demonstrates that principled integration of VE diffusion with SE(3)-equivariant architectures can achieve both high chemical accuracy and computational efficiency.
♻ ☆ Multimodal Markup Document Models for Graphic Design Completion
Kotaro Kikuchi, Ukyo Honda, Naoto Inoue, Mayu Otani, Edgar Simo-Serra, Kota Yamaguchi
We introduce MarkupDM, a multimodal markup document model that represents graphic design as an interleaved multimodal document consisting of both markup language and images. Unlike existing holistic approaches that rely on an element-by-attribute grid representation, our representation accommodates variable-length elements, type-dependent attributes, and text content. Inspired by fill-in-the-middle training in code generation, we train the model to complete the missing part of a design document from its surrounding context, allowing it to treat various design tasks in a unified manner. Our model also supports image generation by predicting discrete image tokens through a specialized tokenizer with support for image transparency. We evaluate MarkupDM on three tasks, attribute value, image, and text completion, and demonstrate that it can produce plausible designs consistent with the given context. To further illustrate the flexibility of our approach, we evaluate our approach on a new instruction-guided design completion task where our instruction-tuned MarkupDM compares favorably to state-of-the-art image editing models, especially in textual completion. These findings suggest that multimodal language models with our document representation can serve as a versatile foundation for broad design automation.
comment : Accepted by ACM Multimedia 2025, Project page: https://cyberagentailab.github.io/MarkupDM/
♻ ☆ Detection of AI Deepfake and Fraud in Online Payments Using GAN-Based Models
Zong Ke, Shicheng Zhou, Yining Zhou, Chia Hong Chang, Rong Zhang
This study explores the use of Generative Adversarial Networks (GANs) to detect AI deepfakes and fraudulent activities in online payment systems. With the growing prevalence of deepfake technology, which can manipulate facial features in images and videos, the potential for fraud in online transactions has escalated. Traditional security systems struggle to identify these sophisticated forms of fraud. This research proposes a novel GAN-based model that enhances online payment security by identifying subtle manipulations in payment images. The model is trained on a dataset consisting of real-world online payment images and deepfake images generated using advanced GAN architectures, such as StyleGAN and DeepFake. The results demonstrate that the proposed model can accurately distinguish between legitimate transactions and deepfakes, achieving a high detection rate above 95%. This approach significantly improves the robustness of payment systems against AI-driven fraud. The paper contributes to the growing field of digital security, offering insights into the application of GANs for fraud detection in financial services. Keywords- Payment Security, Image Recognition, Generative Adversarial Networks, AI Deepfake, Fraudulent Activities
comment : The paper will be published and indexed by IEEE at 2025 8th International Conference on Advanced Algorithms and Control Engineering (ICAACE 2025)
♻ ☆ ViRectify: A Challenging Benchmark for Video Reasoning Correction with Multimodal Large Language Models
Xusen Hei, Jiali Chen, Jinyu Yang, Mengchen Zhao, Yi Cai
As multimodal large language models (MLLMs) frequently exhibit errors in complex video reasoning scenarios, correcting these errors is critical for uncovering their weaknesses and improving performance. However, existing benchmarks lack systematic evaluation of MLLMs' ability to identify and correct these video reasoning errors. To bridge this gap, we propose ViRectify, a comprehensive benchmark to evaluate their fine-grained correction capability. Through an AI-assisted annotation pipeline with human verification, we construct a dataset of over 30K instances spanning dynamic perception, scientific reasoning, and embodied decision-making domains. In ViRectify, we challenge MLLMs to perform step-wise error identification and generate rationales with key video evidence grounding. In addition, we further propose the trajectory evidence-driven correction framework, comprising step-wise error trajectory and reward modeling on visual evidence-grounded correction. It encourages the model to explicitly concentrate on error propagation and key timestamps for correction. Extensive evaluation across 16 advanced MLLMs demonstrates that our ViRectify serves as a challenging testbed, where GPT-5 achieves only 31.94% correction accuracy. Our framework enables a Qwen2.5-VL-7B to consistently outperform the variants of 72B on ViRectify, showing the effectiveness of our approach. Further analysis uncovers systematic asymmetries in error correction across models, and our dataset is also a valuable data resource to perform reflection learning. We believe ViRectify provides a new direction for comprehensively evaluating the advanced MLLMs in video reasoning.
comment : 22 pages, 11 figures
♻ ☆ Collaborative Face Experts Fusion in Video Generation: Boosting Id entity Consistency Across Large Face Poses
Yuji Wang, Moran Li, Xiaobin Hu, Ran Yi, Jiangning Zhang, Chengming Xu, Weijian Cao, Yabiao Wang, Chengjie Wang, Lizhuang Ma
Current video generation models struggle with identity preservation under large face poses, primarily facing two challenges: the difficulty in exploring an effective mechanism to integrate identity features into DiT architectures, and the lack of targeted coverage of large face poses in existing open-source video datasets. To address these, we present two key innovations. First, we propose Collaborative Face Experts Fusion (CoFE), which dynamically fuses complementary signals from three specialized experts within the DiT backbone: an identity expert that captures cross-pose invariant features, a semantic expert that encodes high-level visual context, and a detail expert that preserves pixel-level attributes such as skin texture and color gradients. Second, we introduce a data curation pipeline comprising three key components: Face Constraints to ensure diverse large-pose coverage, Identity Consistency to maintain stable identity across frames, and Speech Disambiguation to align textual captions with actual speaking behavior. This pipeline yields LaFID-180K, a large-scale dataset of pose-annotated video clips designed for identity-preserving video generation. Experimental results on several benchmarks demonstrate that our approach significantly outperforms state-of-the-art methods in face similarity, FID, and CLIP semantic alignment. \href{https://rain152.github.io/CoFE/}{Project page}.
comment : \href{https://rain152.github.io/CoFE/}{Project page}
♻ ☆ MILR: Improving Multimodal Image Generation via Test-Time Latent Reasoning
Yapeng Mi, Hengli Li, Yanpeng Zhao, Chenxi Li, Huimin Wu, Xiaojian Ma, Song-Chun Zhu, Ying Nian Wu, Qing Li
Reasoning-augmented machine learning systems have shown improved performance in various domains, including image generation. However, existing reasoning-based methods for image generation either restrict reasoning to a single modality (image or text) or rely on high-quality reasoning data for fine-tuning. To tackle these limitations, we propose MILR, a test-time method that jointly reasons over image and text in a unified latent vector space. Reasoning in MILR is performed by searching through vector representations of discrete image and text tokens. Practically, this is implemented via the policy gradient method, guided by an image quality critic. We instantiate MILR within the unified multimodal understanding and generation (MUG) framework that natively supports language reasoning before image synthesis and thus facilitates cross-modal reasoning. The intermediate model outputs, which are to be optimized, serve as the unified latent space, enabling MILR to operate entirely at test time. We evaluate MILR on GenEval, T2I-CompBench, and WISE, achieving state-of-the-art results on all benchmarks. Notably, on knowledge-intensive WISE, MILR attains an overall score of 0.63, improving over the baseline by 80%. Our further analysis indicates that joint reasoning in the unified latent space is the key to its strong performance. Moreover, our qualitative studies reveal MILR's non-trivial ability in temporal and cultural reasoning, highlighting the efficacy of our reasoning method.
comment : 21 pages,13 figures,9 tables
♻ ☆ ViDiC: Video Difference Captioning
Jiangtao Wu, Shihao Li, Zhaozhou Bian, Jialu Chen, Runzhe Wen, An Ping, Yiwen He, Jiakai Wang, Yuanxing Zhang, Jiaheng Liu
Understanding visual differences between dynamic scenes requires the comparative perception of compositional, spatial, and temporal changes--a capability that remains underexplored in existing vision-language systems. While prior work on Image Difference Captioning (IDC) has enabled models to describe semantic changes between static images, these approaches fail to capture motion continuity, event evolution, or editing consistency over time. We introduce the ViDiC (Video Difference Captioning) task and its corresponding ViDiC-1K dataset, designed to evaluate the ability of Multimodal Large Language Models (MLLMs) to provide fine-grained descriptions of similarities and differences between video pairs. ViDiC-1K comprises 1,000 curated video pairs annotated with over 4,000 comparative checklist items, covering seven categories: subject, style, background, cinematography, motion, location, and playback techniques. To ensure reliable evaluation, we propose a dual-checklist framework that measures the accuracy of similarity and difference separately, based on the LLM-as-a-Judge protocol. Experiments on nineteen representative multimodal models reveal a significant performance gap in their comparative description and difference perception abilities. We hope ViDiC-1K can be a challenging benchmark that lays a solid foundation for advancing video understanding, edit awareness, and comparative reasoning in multimodal intelligence.
♻ ☆ Sharp Eyes and Memory for VideoLLMs: Information -Aware Visual Token Pruning for Efficient and Reliable VideoLLM Reasoning AAAI
Jialong Qin, Xin Zou, Di Lu, Yibo Yan, Xuming Hu
Current Video Large Language Models (VideoLLMs) suffer from quadratic computational complexity and key-value cache scaling, due to their reliance on processing excessive redundant visual tokens. To address this problem, we propose SharpV, a minimalist and efficient method for adaptive pruning of visual tokens and KV cache. Different from most uniform compression approaches, SharpV dynamically adjusts pruning ratios based on spatial-temporal information. Remarkably, this adaptive mechanism occasionally achieves performance gains over dense models, offering a novel paradigm for adaptive pruning. During the KV cache pruning stage, based on observations of visual information degradation, SharpV prunes degraded visual features via a self-calibration manner, guided by similarity to original visual features. In this way, SharpV achieves hierarchical cache pruning from the perspective of information bottleneck, offering a new insight into VideoLLMs' information flow. Experiments on multiple public benchmarks demonstrate the superiority of SharpV. Moreover, to the best of our knowledge, SharpV is notably the first two-stage pruning framework that operates without requiring access to exposed attention scores, ensuring full compatibility with hardware acceleration techniques like Flash Attention.
comment : The 40th Annual AAAI Conference on Artificial Intelligence (AAAI-26) Poster
♻ ☆ BOP-ASK: Object-Interaction Reasoning for Vision -Language Models
Vineet Bhat, Sungsu Kim, Valts Blukis, Greg Heinrich, Prashanth Krishnamurthy, Ramesh Karri, Stan Birchfield, Farshad Khorrami, Jonathan Tremblay
Vision Language Models (VLMs) have achieved impressive performance on spatial reasoning benchmarks, yet these evaluations mask critical weaknesses in understanding object interactions. Current benchmarks test high level relationships ('left of,' 'behind', etc.) but ignore fine-grained spatial understanding needed for real world applications: precise 3D localization, physical compatibility between objects, object affordances and multi step spatial planning. In this work, we present BOP-ASK, a novel large scale dataset for object interaction reasoning for both training and benchmarking. Our data generation pipeline leverages 6D object poses from the Benchmark for Object Pose Estimation (BOP) datasets from which we derive fine grained annotations such as grasp poses, referred object poses, path planning trajectories, relative spatial and depth relationships, and object-to-object relationships. BOP-ASK comprises over 150k images and 33M question answer pairs spanning six tasks (four novel), providing a rich resource for training and evaluating VLMs. We evaluate proprietary and open sourced VLMs, and conduct human evaluations on BOP-ASK-core, a contributed test benchmark. We also release BOP-ASK-lab, an out-of-distribution benchmark with images not sourced from BOP, enabling testing of generalization. Our experiments demonstrate that models trained on BOP-ASK outperform baselines and exhibit emergent capabilities such as precise object and grasp pose estimation, trajectory planning, and fine-grained object-centric spatial reasoning in cluttered environments. We will publicly release our datasets and dataset generation pipeline.
♻ ☆ Downscaling climate projections to 1 km with single-image super resolution
Petr Košťál, Pavel Kordík, Ondřej Podsztavek
High-resolution climate projections are essential for local decision-making. However, available climate projections have low spatial resolution (e.g. 12.5 km), which limits their usability. We address this limitation by leveraging single-image super-resolution models to statistically downscale climate projections to 1-km resolution. Since high-resolution climate projections are unavailable, we train models on a high-resolution observational gridded data set and apply them to low-resolution climate projections. We cannot evaluate downscaled climate projections with common metrics (e.g. pixel-wise root-mean-square error) because we lack ground-truth high-resolution climate projections. Therefore, we evaluate climate indicators computed at weather station locations. Experiments on daily mean temperature demonstrate that single-image super-resolution models can downscale climate projections without increasing the error of climate indicators compared to low-resolution climate projections.
♻ ☆ Beyond the Ground Truth: Enhanced Super vision for Image Restoration
Donghun Ryou, Inju Ha, Sanghyeok Chu, Bohyung Han
Deep learning-based image restoration has achieved significant success. However, when addressing real-world degradations, model performance is limited by the quality of ground-truth images in datasets due to practical constraints in data acquisition. To address this limitation, we propose a novel framework that enhances existing ground truth images to provide higher-quality supervision for real-world restoration. Our framework generates perceptually enhanced ground truth images using super-resolution by incorporating adaptive frequency masks, which are learned by a conditional frequency mask generator. These masks guide the optimal fusion of frequency components from the original ground truth and its super-resolved variants, yielding enhanced ground truth images. This frequency-domain mixup preserves the semantic consistency of the original content while selectively enriching perceptual details, preventing hallucinated artifacts that could compromise fidelity. The enhanced ground truth images are used to train a lightweight output refinement network that can be seamlessly integrated with existing restoration models. Extensive experiments demonstrate that our approach consistently improves the quality of restored images. We further validate the effectiveness of both supervision enhancement and output refinement through user studies. Code is available at https://github.com/dhryougit/Beyond-the-Ground-Truth.
♻ ☆ Zero4D: Training-Free 4D Video Generation From Single Video Using Off-the-Shelf Video Diffusion
Jangho Park, Taesung Kwon, Jong Chul Ye
Multi-view or 4D video generation has emerged as a significant research topic. Nonetheless, recent approaches to 4D generation still struggle with fundamental limitations, as they primarily rely on harnessing multiple video diffusion models with additional training or compute-intensive training of a full 4D diffusion model with limited real-world 4D data and large computational costs. To address these challenges, here we propose the first training-free 4D video generation method that leverages the off-the-shelf video diffusion models to generate multi-view videos from a single input video. Our approach consists of two key steps: (1) By designating the edge frames in the spatio-temporal sampling grid as key frames, we first synthesize them using a video diffusion model, leveraging a depth-based warping technique for guidance. This approach ensures structural consistency across the generated frames, preserving spatial and temporal coherence. (2) We then interpolate the remaining frames using a video diffusion model, constructing a fully populated and temporally coherent sampling grid while preserving spatial and temporal consistency. Through this approach, we extend a single video into a multi-view video along novel camera trajectories while maintaining spatio-temporal consistency. Our method is training-free and fully utilizes an off-the-shelf video diffusion model, offering a practical and effective solution for multi-view video generation.
comment : project page: https://zero4dvid.github.io/
♻ ☆ EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretrain ing
Boshen Xu, Yuting Mei, Xinbi Liu, Sipeng Zheng, Qin Jin
Egocentric video-language pretraining has significantly advanced video representation learning. Humans perceive and interact with a fully 3D world, developing spatial awareness that extends beyond text-based understanding. However, most previous works learn from 1D text or 2D visual cues, such as bounding boxes, which inherently lack 3D understanding. To bridge this gap, we introduce EgoDTM, an Egocentric Depth- and Text-aware Model, jointly trained through large-scale 3D-aware video pretraining and video-text contrastive learning. EgoDTM incorporates a lightweight 3D-aware decoder to efficiently learn 3D-awareness from pseudo depth maps generated by depth estimation models. To further facilitate 3D-aware video pretraining, we enrich the original brief captions with hand-object visual cues by organically combining several foundation models. Extensive experiments demonstrate EgoDTM's superior performance across diverse downstream tasks, highlighting its superior 3D-aware visual understanding. Code: https://github.com/xuboshen/EgoDTM.
comment : Code: https://github.com/xuboshen/EgoDTM
♻ ☆ BrainPath: A Biologically-Informed AI Framework for Individualized Aging Brain Generation
Yifan Li, Javad Sohankar, Ji Luo, Jing Li, Yi Su
The global population is aging rapidly, and aging is a major risk factor for various diseases. It is an important task to predict how each individual's brain will age, as the brain supports many human functions. This capability can greatly facilitate healthcare automation by enabling personalized, proactive intervention and efficient healthcare resource allocation. However, this task is extremely challenging because of the brain's complex 3D anatomy. While there have been successes in natural image generation and brain MRI synthesis, existing methods fall short in generating individualized, anatomically faithful aging brain trajectories. To address these gaps, we propose BrainPath, a novel AI model that, given a single structural MRI of an individual, generates synthetic longitudinal MRIs that represent that individual's expected brain anatomy as they age. BrainPath introduces three architectural innovations: an age-aware encoder with biologically grounded supervision, a differential age conditioned decoder for anatomically faithful MRI synthesis, and a swap-learning strategy that implicitly separates stable subject-specific anatomy from aging effects. We further design biologically informed loss functions, including an age calibration loss and an age and structural perceptual loss, to complement the conventional reconstruction loss. This enables the model to capture subtle, temporally meaningful anatomical changes associated with aging. We apply BrainPath to two of the largest public aging datasets and conduct a comprehensive, multifaceted evaluation. Our results demonstrate BrainPath's superior performance in generation accuracy, anatomical fidelity, and cross-dataset generalizability, outperforming competing methods.
♻ ☆ VRWKV-Editor: Reducing quadratic complexity in transformer -based video editing
Abdelilah Aitrouga, Youssef Hmamouche, Amal El Fallah Seghrouchni
In light of recent progress in video editing, deep learning models focusing on both spatial and temporal dependencies have emerged as the primary method. However, these models suffer from the quadratic computational complexity of traditional attention mechanisms, making them difficult to adapt to long-duration and high-resolution videos. This limitation restricts their applicability in practical contexts such as real-time video processing. To tackle this challenge, we introduce a method to reduce both time and space complexity of these systems by proposing VRWKV-Editor, a novel video editing model that integrates a linear spatio-temporal aggregation module into video-based diffusion models. VRWKV-Editor leverages bidirectional weighted key-value recurrence mechanism of the RWKV transformer to capture global dependencies while preserving temporal coherence, achieving linear complexity without sacrificing quality. Extensive experiments demonstrate that the proposed method achieves up to 3.7x speedup and 60% lower memory usage compared to state-of-the-art diffusion-based video editing methods, while maintaining competitive performance in frame consistency and text alignment. Furthermore, a comparative analysis we conducted on videos with different sequence lengths confirms that the gap in editing speed between our approach and architectures with self-attention becomes more significant with long videos.
♻ ☆ VoLUT: Efficient Volumetric streaming enhanced by LUT-based super-resolution
Chendong Wang, Anlan Zhang, Yifan Yang, Lili Qiu, Yuqing Yang, Xinyang Jiang, Feng Qian, Suman Banerjee
3D volumetric video provides immersive experience and is gaining traction in digital media. Despite its rising popularity, the streaming of volumetric video content poses significant challenges due to the high data bandwidth requirement. A natural approach to mitigate the bandwidth issue is to reduce the volumetric video's data rate by downsampling the content prior to transmission. The video can then be upsampled at the receiver's end using a super-resolution (SR) algorithm to reconstruct the high-resolution details. While super-resolution techniques have been extensively explored and advanced for 2D video content, there is limited work on SR algorithms tailored for volumetric videos. To address this gap and the growing need for efficient volumetric video streaming, we have developed VoLUT with a new SR algorithm specifically designed for volumetric content. Our algorithm uniquely harnesses the power of lookup tables (LUTs) to facilitate the efficient and accurate upscaling of low-resolution volumetric data. The use of LUTs enables our algorithm to quickly reference precomputed high-resolution values, thereby significantly reducing the computational complexity and time required for upscaling. We further apply adaptive video bit rate algorithm (ABR) to dynamically determine the downsampling rate according to the network condition and stream the selected video rate to the receiver. Compared to related work, VoLUT is the first to enable high-quality 3D SR on commodity mobile devices at line-rate. Our evaluation shows VoLUT can reduce bandwidth usage by 70% , boost QoE by 36.7% for volumetric video streaming and achieve 3D SR speed-up with no quality compromise.
♻ ☆ Learning Geodesics of Geometric Shape Deformations From Images
Nian Wu, Miaomiao Zhang
This paper presents a novel method, named geodesic deformable networks (GDN), that for the first time enables the learning of geodesic flows of deformation fields derived from images. In particular, the capability of our proposed GDN being able to predict geodesics is important for quantifying and comparing deformable shape presented in images. The geodesic deformations, also known as optimal transformations that align pairwise images, are often parameterized by a time sequence of smooth vector fields governed by nonlinear differential equations. A bountiful literature has been focusing on learning the initial conditions (e.g., initial velocity fields) based on registration networks. However, the definition of geodesics central to deformation-based shape analysis is blind to the networks. To address this problem, we carefully develop an efficient neural operator to treat the geodesics as unknown mapping functions learned from the latent deformation spaces. A composition of integral operators and smooth activation functions is then formulated to effectively approximate such mappings. In contrast to previous works, our GDN jointly optimizes a newly defined geodesic loss, which adds additional benefits to promote the network regularizability and generalizability. We demonstrate the effectiveness of GDN on both 2D synthetic data and 3D real brain magnetic resonance imaging (MRI).
♻ ☆ Multi-Focus Temporal Shifting for Precise Event Spotting in Sports Videos
Hao Xu, Xinyu Wei, Sam Wells, Sunil Aryal
Precise Event Spotting (PES) in sports videos requires frame-level recognition of fine-grained actions from single-camera footage. Existing PES models typically incorporate lightweight temporal modules such as the Gate Shift Module (GSM) or the Gate Shift Fuse to enrich 2D CNN feature extractors with temporal context. However, these modules are limited in both temporal receptive field and spatial adaptability. We propose Multi-Focus Temporal Shifting Module (MFS) that enhances GSM with multi-scale temporal shifts and Group Focus Module, enabling efficient modeling of both short and long-term dependencies while focusing on salient regions. MFS is a lightweight, plug-and-play module that integrates seamlessly with diverse 2D backbones. To further advance the field, we introduce the Table Tennis Australia dataset, the first PES benchmark for table tennis containing over 4,800 precisely annotated events. Extensive experiments across five PES benchmarks demonstrate that MFS consistently improves performance with minimal overhead, achieving leading results among lightweight methods (+4.09 mAP, 45 GFLOPs).
comment : 7 pages
♻ ☆ Dual-branch Prompt ing for Multimodal Machine Translation
Jie Wang, Zhendong Yang, Liansong Zong, Xiaobo Zhang, Dexian Wang, Ji Zhang
Multimodal Machine Translation (MMT) typically enhances text-only translation by incorporating aligned visual features. Despite the remarkable progress, state-of-the-art MMT approaches often rely on paired image-text inputs at inference and are sensitive to irrelevant visual noise, which limits their robustness and practical applicability. To address these issues, we propose D2P-MMT, a diffusion-based dual-branch prompting framework for robust vision-guided translation. Specifically, D2P-MMT requires only the source text and a reconstructed image generated by a pre-trained diffusion model, which naturally filters out distracting visual details while preserving semantic cues. During training, the model jointly learns from both authentic and reconstructed images using a dual-branch prompting strategy, encouraging rich cross-modal interactions. To bridge the modality gap and mitigate training-inference discrepancies, we introduce a distributional alignment loss that enforces consistency between the output distributions of the two branches. Extensive experiments on the Multi30K dataset demonstrate that D2P-MMT achieves superior translation performance compared to existing state-of-the-art approaches.
comment : This manuscript is currently under review at the ACM Transactions on Multimedia Computing, Communications, and Applications
♻ ☆ TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learning
Tao Wu, Li Yang, Gen Zhan, Yabin Zhang, Yiting Liao, Junlin Li, Deliang Fu, Li Zhang, Limin Wang
Enhancing the temporal understanding of Multimodal Large Language Models (MLLMs) is essential for advancing long-form video analysis, enabling tasks such as temporal localization, action detection, and time-sensitive question answering. While reinforcement learning (RL) has recently been explored for improving temporal reasoning, existing approaches are often confined to limited task types and data, restricting their generalization across diverse temporal understanding scenarios. To address this challenge, we present TempR1, a temporal-aware multi-task reinforcement learning framework that systematically strengthens MLLMs' temporal comprehension. We curate a multi-task corpus that exposes the model to diverse temporal structures and semantics, and build upon the Group Relative Policy Optimization (GRPO) algorithm to achieve stable and effective cross-task optimization. Specifically, we categorize temporal tasks into three correspondence types between predicted intervals and ground-truth instances, and design tailored localization rewards for each, enabling TempR1 to capture fine-grained temporal dependencies and adapt to different temporal patterns. Extensive experiments demonstrate that TempR1 attains state-of-the-art performance across multiple benchmarks. Moreover, its joint optimization over complementary tasks yields a strong synergistic effect, enhancing both generalization and single-task performance, establishing a scalable and principled paradigm for temporal reasoning in MLLMs.
♻ ☆ SYNTHIA: Novel Concept Design with Affordance Composition ACL 2025
Hyeonjeong Ha, Xiaomeng Jin, Jeonghwan Kim, Jiateng Liu, Zhenhailong Wang, Khanh Duy Nguyen, Ansel Blume, Nanyun Peng, Kai-Wei Chang, Heng Ji
Text-to-image (T2I) models enable rapid concept design, making them widely used in AI-driven design. While recent studies focus on generating semantic and stylistic variations of given design concepts, functional coherence--the integration of multiple affordances into a single coherent concept--remains largely overlooked. In this paper, we introduce SYNTHIA, a framework for generating novel, functionally coherent designs based on desired affordances. Our approach leverages a hierarchical concept ontology that decomposes concepts into parts and affordances, serving as a crucial building block for functionally coherent design. We also develop a curriculum learning scheme based on our ontology that contrastively fine-tunes T2I models to progressively learn affordance composition while maintaining visual novelty. To elaborate, we (i) gradually increase affordance distance, guiding models from basic concept-affordance association to complex affordance compositions that integrate parts of distinct affordances into a single, coherent form, and (ii) enforce visual novelty by employing contrastive objectives to push learned representations away from existing concepts. Experimental results show that SYNTHIA outperforms state-of-the-art T2I models, demonstrating absolute gains of 25.1% and 14.7% for novelty and functional coherence in human evaluation, respectively.
comment : ACL 2025 Main, Code is available https://github.com/HyeonjeongHa/SYNTHIA
♻ ☆ Bootstrap Dynamic-Aware 3D Visual Representation for Scalable Robot Learning
Qiwei Liang, Boyang Cai, Minghao Lai, Sitong Zhuang, Tao Lin, Yan Qin, Yixuan Ye, Jiaming Liang, Renjing Xu
Despite strong results on recognition and segmentation, current 3D visual pre-training methods often underperform on robotic manipulation. We attribute this gap to two factors: the lack of state-action-state dynamics modeling and the unnecessary redundancy of explicit geometric reconstruction. We introduce AFRO, a self-supervised framework that learns dynamics-aware 3D representations without action or reconstruction supervision. AFRO casts state prediction as a generative diffusion process and jointly models forward and inverse dynamics in a shared latent space to capture causal transition structure. To prevent feature leakage in action learning, we employ feature differencing and inverse-consistency supervision, improving the quality and stability of visual features. When combined with Diffusion Policy, AFRO substantially increases manipulation success rates across 16 simulated and 4 real-world tasks, outperforming existing pre-training approaches. The framework also scales favorably with data volume and task complexity. Qualitative visualizations indicate that AFRO learns semantically rich, discriminative features, offering an effective pre-training solution for 3D representation learning in robotics. Project page: https://kolakivy.github.io/AFRO/
♻ ☆ DE-KAN: A Kolmogorov Arnold Network with Dual Encoder for accurate 2D Teeth Segmentation
Md Mizanur Rahman Mustakim, Jianwu Li, Sumya Bhuiyan, Mohammad Mehedi Hasan, Bing Han
Accurate segmentation of individual teeth from panoramic radiographs remains a challenging task due to anatomical variations, irregular tooth shapes, and overlapping structures. These complexities often limit the performance of conventional deep learning models. To address this, we propose DE-KAN, a novel Dual Encoder Kolmogorov Arnold Network, which enhances feature representation and segmentation precision. The framework employs a ResNet-18 encoder for augmented inputs and a customized CNN encoder for original inputs, enabling the complementary extraction of global and local spatial features. These features are fused through KAN-based bottleneck layers, incorporating nonlinear learnable activation functions derived from the Kolmogorov Arnold representation theorem to improve learning capacity and interpretability. Extensive experiments on two benchmark dental X-ray datasets demonstrate that DE-KAN outperforms state-of-the-art segmentation models, achieving mIoU of 94.5%, Dice coefficient of 97.1%, accuracy of 98.91%, and recall of 97.36%, representing up to +4.7% improvement in Dice compared to existing methods.
♻ ☆ Beyond Description: Cognitively Benchmark ing Fine-Grained Action for Embodied Agent s
Dayong Liu, Chao Xu, Weihong Chen, Suyu Zhang, Juncheng Wang, Jiankang Deng, Baigui Sun, Yang Liu
Multimodal Large Language Models (MLLMs) show promising results as decision-making engines for embodied agents operating in complex, physical environments. However, existing benchmarks often prioritize high-level planning or spatial reasoning, leaving the fine-grained action intelligence required for embodied physical interaction underexplored. To address this gap, we introduce CFG-Bench, a new benchmark designed to systematically evaluate this crucial capability. CFG-Bench consists of 1,368 curated videos paired with 19,562 three-modalities question-answer pairs targeting four cognitive abilities: 1) Physical Interaction, 2) Temporal-Causal Relation, 3) Intentional Understanding, and 4) Evaluative Judgment. Together, these dimensions provide a systematic framework for assessing a model's ability to translate visual observations into actionable knowledge, moving beyond mere surface-level recognition. Our comprehensive evaluation on CFG-Bench reveals that leading MLLMs struggle to produce detailed instructions for physical interactions and exhibit profound limitations in the higher-order reasoning of intention and evaluation. Moreover, supervised fine-tuning (SFT) on our data demonstrates that teaching an MLLMs to articulate fine-grained actions directly translates to significant performance gains on established embodied benchmarks. Our analysis highlights these limitations and offers insights for developing more capable and grounded embodied agents. Project page: \href{https://cfg-bench.github.io/}{https://cfg-bench.github.io/}.
♻ ☆ CMM CoT : Enhancing Complex Multi-Image Comprehension via Multi-Modal Chain-of-Thought and Memory Augmentation AAAI 2026
Guanghao Zhang, Tao Zhong, Yan Xia, Mushui Liu, Zhelun Yu, Haoyuan Li, Wanggui He, Fangxun Shu, Dong She, Yi Wang, Hao Jiang
While previous multimodal slow-thinking methods have demonstrated remarkable success in single-image understanding scenarios, their effectiveness becomes fundamentally constrained when extended to more complex multi-image comprehension tasks. This limitation stems from their predominant reliance on text-based intermediate reasoning processes. While for human, when engaging in sophisticated multi-image analysis, they typically perform two complementary cognitive operations: (1) continuous cross-image visual comparison through region-of-interest matching, and (2) dynamic memorization of critical visual concepts throughout the reasoning chain. Motivated by these observations, we propose the Complex Multi-Modal Chain-of-Thought (CMMCoT) framework, a multi-step reasoning framework that mimics human-like "slow thinking" for multi-image understanding. Our approach incorporates two key innovations: (1) The construction of interleaved multimodal multi-step reasoning chains, which utilize critical visual region tokens, extracted from intermediate reasoning steps, as supervisory signals. This mechanism not only facilitates comprehensive cross-modal understanding but also enhances model interpretability. (2) The introduction of a test-time memory augmentation module that expands the model's reasoning capacity during inference while preserving parameter efficiency. Furthermore, to facilitate research in this direction, we have curated a novel multi-image slow-thinking dataset. Extensive experiments demonstrate the effectiveness of our model. Code is available at https://github.com/zhangguanghao523/CMMCoT.
comment : Accepted by AAAI 2026
♻ ☆ TraceTrans: Translation and Spatial Tracing for Surgical Prediction
Xiyu Luo, Haodong Li, Xinxing Cheng, He Zhao, Yang Hu, Xuan Song, Tianyang Zhang
Image-to-image translation models have achieved notable success in converting images across visual domains and are increasingly used for medical tasks such as predicting post-operative outcomes and modeling disease progression. However, most existing methods primarily aim to match the target distribution and often neglect spatial correspondences between the source and translated images. This limitation can lead to structural inconsistencies and hallucinations, undermining the reliability and interpretability of the predictions. These challenges are accentuated in clinical applications by the stringent requirement for anatomical accuracy. In this work, we present TraceTrans, a novel deformable image translation model designed for post-operative prediction that generates images aligned with the target distribution while explicitly revealing spatial correspondences with the pre-operative input. The framework employs an encoder for feature extraction and dual decoders for predicting spatial deformations and synthesizing the translated image. The predicted deformation field imposes spatial constraints on the generated output, ensuring anatomical consistency with the source. Extensive experiments on medical cosmetology and brain MRI datasets demonstrate that TraceTrans delivers accurate and interpretable post-operative predictions, highlighting its potential for reliable clinical deployment.
Computation and Language 81
☆ DraCo: Draft as CoT for Text-to-Image P review and Rare Concept Generation
Dongzhi Jiang, Renrui Zhang, Haodong Li, Zhuofan Zong, Ziyu Guo, Jun He, Claire Guo, Junyan Ye, Rongyao Fang, Weijia Li, Rui Liu, Hongsheng Li
Recent unified multimodal large language models (MLLMs) have shown impressive capabilities, incorporating chain-of-thought (CoT) reasoning for enhanced text-to-image generation. However, existing approaches remain limited, either treating the model merely as a standalone generator or relying on abstract textual planning. To this end, we propose Draft-as-CoT (DraCo), a novel interleaved reasoning paradigm that fully leverages both textual and visual contents in CoT for better planning and verification. Our method first generates a low-resolution draft image as preview, providing more concrete and structural visual planning and guidance. Then, we employ the model's inherent understanding capability to verify potential semantic misalignments between the draft and input prompt, and performs refinement through selective corrections with super-resolution. In this way, our approach addresses two fundamental challenges: the coarse-grained nature of textual planning and the difficulty in generating rare attribute combinations. To support training, we curate DraCo-240K, aiming to enhance three atomic capabilities spanning general correction, instance manipulation, and layout reorganization. Supported by DraCo-CFG, a specialized classifier-free guidance (CFG) strategy for interleaved reasoning, DraCo achieves a tremendous increase on GenEval (+8%), Imagine-Bench (+0.91), and GenEval++ (+3%), significantly outperforming direct generation and other generation methods empowered by CoT.
comment : Project Page: https://github.com/CaraJ7/DraCo
☆ Semantic Soft Bootstrapping: Long Context Reasoning in LLMs without Reinforcement Learning
Purbesh Mitra, Sennur Ulukus
Long context reasoning in large language models (LLMs) has demonstrated enhancement of their cognitive capabilities via chain-of-thought (CoT) inference. Training such models is usually done via reinforcement learning with verifiable rewards (RLVR) in reasoning based problems, like math and programming. However, RLVR is limited by several bottlenecks, such as, lack of dense reward, and inadequate sample efficiency. As a result, it requires significant compute resources in post-training phase. To overcome these limitations, in this work, we propose \textbf{Semantic Soft Bootstrapping (SSB)}, a self-distillation technique, in which the same base language model plays the role of both teacher and student, but receives different semantic contexts about the correctness of its outcome at training time. The model is first prompted with a math problem and several rollouts are generated. From them, the correct and most common incorrect response are filtered, and then provided to the model in context to produce a more robust, step-by-step explanation with a verified final answer. This pipeline automatically curates a paired teacher-student training set from raw problem-answer data, without any human intervention. This generation process also produces a sequence of logits, which is what the student model tries to match in the training phase just from the bare question alone. In our experiment, Qwen2.5-3B-Instruct on GSM8K dataset via parameter-efficient fine-tuning. We then tested its accuracy on MATH500, and AIME2024 benchmarks. Our experiments show a jump of 10.6%, and 10% improvements in accuracy, respectively, over group relative policy optimization (GRPO), which is a commonly used RLVR algorithm. Our code is available at https://github.com/purbeshmitra/semantic-soft-bootstrapping, and the model, curated dataset is available at https://huggingface.co/purbeshmitra/semantic-soft-bootstrapping.
☆ Structured Document Translation via Format Reinforcement Learning AACL 2025
Haiyue Song, Johannes Eschbach-Dymanus, Hour Kaing, Sumire Honda, Hideki Tanaka, Bianka Buschbeck, Masao Utiyama
Recent works on structured text translation remain limited to the sentence level, as they struggle to effectively handle the complex document-level XML or HTML structures. To address this, we propose \textbf{Format Reinforcement Learning (FormatRL)}, which employs Group Relative Policy Optimization on top of a supervised fine-tuning model to directly optimize novel structure-aware rewards: 1) TreeSim, which measures structural similarity between predicted and reference XML trees and 2) Node-chrF, which measures translation quality at the level of XML nodes. Additionally, we apply StrucAUC, a fine-grained metric distinguishing between minor errors and major structural failures. Experiments on the SAP software-documentation benchmark demonstrate improvements across six metrics and an analysis further shows how different reward functions contribute to improvements in both structural and translation quality.
comment : IJCNLP-AACL 2025 Main (Oral)
☆ Multi-LLM Collaboration for Medication Recommendation
Huascar Sanchez, Briland Hitaj, Jules Bergmann, Linda Briesemeister
As healthcare increasingly turns to AI for scalable and trustworthy clinical decision support, ensuring reliability in model reasoning remains a critical challenge. Individual large language models (LLMs) are susceptible to hallucinations and inconsistency, whereas naive ensembles of models often fail to deliver stable and credible recommendations. Building on our previous work on LLM Chemistry, which quantifies the collaborative compatibility among LLMs, we apply this framework to improve the reliability in medication recommendation from brief clinical vignettes. Our approach leverages multi-LLM collaboration guided by Chemistry-inspired interaction modeling, enabling ensembles that are effective (exploiting complementary strengths), stable (producing consistent quality), and calibrated (minimizing interference and error amplification). We evaluate our Chemistry-based Multi-LLM collaboration strategy on real-world clinical scenarios to investigate whether such interaction-aware ensembles can generate credible, patient-specific medication recommendations. Preliminary results are encouraging, suggesting that LLM Chemistry-guided collaboration may offer a promising path toward reliable and trustworthy AI assistants in clinical practice.
comment : 8 pages, 5 figures, 1 table
☆ Arbitrage: Efficient Reasoning via Advantage-Aware Speculation
Monishwaran Maheswaran, Rishabh Tiwari, Yuezhou Hu, Kerem Dilmen, Coleman Hooper, Haocheng Xi, Nicholas Lee, Mehrdad Farajtabar, Michael W. Mahoney, Kurt Keutzer, Amir Gholami
Modern Large Language Models achieve impressive reasoning capabilities with long Chain of Thoughts, but they incur substantial computational cost during inference, and this motivates techniques to improve the performance-cost ratio. Among these techniques, Speculative Decoding accelerates inference by employing a fast but inaccurate draft model to autoregressively propose tokens, which are then verified in parallel by a more capable target model. However, due to unnecessary rejections caused by token mismatches in semantically equivalent steps, traditional token-level Speculative Decoding struggles in reasoning tasks. Although recent works have shifted to step-level semantic verification, which improve efficiency by accepting or rejecting entire reasoning steps, existing step-level methods still regenerate many rejected steps with little improvement, wasting valuable target compute. To address this challenge, we propose Arbitrage, a novel step-level speculative generation framework that routes generation dynamically based on the relative advantage between draft and target models. Instead of applying a fixed acceptance threshold, Arbitrage uses a lightweight router trained to predict when the target model is likely to produce a meaningfully better step. This routing approximates an ideal Arbitrage Oracle that always chooses the higher-quality step, achieving near-optimal efficiency-accuracy trade-offs. Across multiple mathematical reasoning benchmarks, Arbitrage consistently surpasses prior step-level Speculative Decoding baselines, reducing inference latency by up to $\sim2\times$ at matched accuracy.
comment : 22 pages
☆ Factuality and Transparency Are All RAG Needs! Self-Explaining Contrastive Evidence Re-ranking
Francielle Vargas, Daniel Pedronette
This extended abstract introduces Self-Explaining Contrastive Evidence Re-Ranking (CER), a novel method that restructures retrieval around factual evidence by fine-tuning embeddings with contrastive learning and generating token-level attribution rationales for each retrieved passage. Hard negatives are automatically selected using a subjectivity-based criterion, forcing the model to pull factual rationales closer while pushing subjective or misleading explanations apart. As a result, the method creates an embedding space explicitly aligned with evidential reasoning. We evaluated our method on clinical trial reports, and initial experimental results show that CER improves retrieval accuracy, mitigates the potential for hallucinations in RAG systems, and provides transparent, evidence-based retrieval that enhances reliability, especially in safety-critical domains.
comment : This work was presented as a poster at the Applied Social Media Lab during the 2025 Synthesizer & Open Showcase at the Berkman Klein Center for Internet & Society at Harvard University
☆ Nex-N1: Agent ic Models Trained via a Unified Ecosystem for Large-Scale Environment Construction
Nex-AGI Team, :, Yuxuan Cai, Lu Chen, Qiaoling Chen, Yuyang Ding, Liwen Fan, Wenjie Fu, Yufei Gao, Honglin Guo, Pinxue Guo, Zhenhua Han, Zhengfu He, Hanglei Hu, Kai Hu, Shengjia Hua, Tianyu Huai, Baodai Huang, Li Ji, Zhen Jiang, Zhikai Lei, Bufan Li, Jiahang Lin, Lizhi Lin, Jinxiu Liu, Shichun Liu, Ziming Liu, Yuchen Ni, Pengfang Qian, Yujiong Shen, Qingyun Shi, Wentao Shu, Peng Sun, Yiran Suo, Tian Tang, Boyu Tian, Guoteng Wang, Junzhe Wang, Peixin Wang, Zhiheng Xi, Hang Yan, Jie Yang, Zhixiong Yang, Tianchu Yao, Guangze Ye, Qianxi Yu, Shuo Zhang, Xinyue Zhang, Yiqi Zhang, Jiarong Zhao, Miao Zheng, Rui Zheng, Enyu Zhou, Jiazheng Zhou, Maosen Zhou, Yuhao Zhou, Tao Gui, Yining Zheng, Xinchi Chen, Jie Zhou, Siyuan Feng, Qin Chen, Liang He, Qi Zhang, Xuanjing Huang, Xipeng Qiu
The evolution of Large Language Models (LLMs) from passive responders to autonomous agents necessitates a fundamental shift in learning paradigms -- from static imitation to incentive-driven decision making. However, this transition is significantly impeded by the lack of scalable infrastructure capable of constructing high-quality interaction signals for effective policy learning. To address this, we introduce a comprehensive method designed to systematically scale the diversity and complexity of interactive environments. Our method realizes this scaling by addressing three orthogonal dimensions: (1) Complexity: NexAU, a flexible agent framework that supports building complex agent hierarchies via simple configurations; (2) Diversity: NexA4A automatically generates diverse agent hierarchies from natural language to cover infinite domains; and (3) Fidelity: NexGAP bridges the simulation-reality gap by integrating dynamic real-world environment for grounded trajectories synthesis. We train Nex-N1 upon the diverse and complex interactive environments established by our infrastructure. Empirical results on benchmarks such as SWE-bench and tau2 demonstrate that Nex-N1 consistently outperforms SOTA open-source models and achieves competitive performance against frontier proprietary models on complex agentic tasks. We open-source the Nex ecosystem and model weights to facilitate further research.
☆ LLMs Know More Than Words: A Genre Study with Syntax, Metaphor & Phonetics
Weiye Shi, Zhaowei Zhang, Shaoheng Yan, Yaodong Yang
Large language models (LLMs) demonstrate remarkable potential across diverse language related tasks, yet whether they capture deeper linguistic properties, such as syntactic structure, phonetic cues, and metrical patterns from raw text remains unclear. To analysis whether LLMs can learn these features effectively and apply them to important nature language related tasks, we introduce a novel multilingual genre classification dataset derived from Project Gutenberg, a large-scale digital library offering free access to thousands of public domain literary works, comprising thousands of sentences per binary task (poetry vs. novel;drama vs. poetry;drama vs. novel) in six languages (English, French, German, Italian, Spanish, and Portuguese). We augment each with three explicit linguistic feature sets (syntactic tree structures, metaphor counts, and phonetic metrics) to evaluate their impact on classification performance. Experiments demonstrate that although LLM classifiers can learn latent linguistic structures either from raw text or from explicitly provided features, different features contribute unevenly across tasks, which underscores the importance of incorporating more complex linguistic signals during model training.
☆ CA RL : Critical Action Focused Reinforcement Learning for Multi-Step Agent
Leyang Shen, Yang Zhang, Chun Kai Ling, Xiaoyan Zhao, Tat-Seng Chua
Agents capable of accomplishing complex tasks through multiple interactions with the environment have emerged as a popular research direction. However, in such multi-step settings, the conventional group-level policy optimization algorithm becomes suboptimal because of its underlying assumption that each action holds equal contribution, which deviates significantly from reality. Our analysis reveals that only a small fraction of actions are critical in determining the final outcome. Building on this insight, we propose CARL, a critical-action-focused reinforcement learning algorithm tailored for multi-step agents. CARL achieves focused training through providing action-level optimization signals for high-criticality actions while excluding low-criticality actions from model update. Extensive experiments demonstrate that CARL achieves both stronger performance and higher efficiency during training and inference across diverse evaluation settings.
comment : 10 pages, 4 figures
☆ Algorithmic Thinking Theory
MohammadHossein Bateni, Vincent Cohen-Addad, Yuzhou Gu, Silvio Lattanzi, Simon Meierhans, Christopher Mohri
Large language models (LLMs) have proven to be highly effective for solving complex reasoning tasks. Surprisingly, their capabilities can often be improved by iterating on previously generated solutions. In this context, a reasoning plan for generating and combining a set of solutions can be thought of as an algorithm for reasoning using a probabilistic oracle. We introduce a theoretical framework for analyzing such reasoning algorithms. This framework formalizes the principles underlying popular techniques for iterative improvement and answer aggregation, providing a foundation for designing a new generation of more powerful reasoning methods. Unlike approaches for understanding models that rely on architectural specifics, our model is grounded in experimental evidence. As a result, it offers a general perspective that may extend to a wide range of current and future reasoning oracles.
☆ The AI Consumer Index (ACE)
Julien Benchek, Rohit Shetty, Benjamin Hunsberger, Ajay Arun, Zach Richards, Brendan Foody, Osvald Nitski, Bertie Vidgen
We introduce the first version of the AI Consumer Index (ACE), a benchmark for assessing whether frontier AI models can perform high-value consumer tasks. ACE contains a hidden heldout set of 400 test cases, split across four consumer activities: shopping, food, gaming, and DIY. We are also open sourcing 80 cases as a devset with a CC-BY license. For the ACE leaderboard we evaluated 10 frontier models (with websearch turned on) using a novel grading methodology that dynamically checks whether relevant parts of the response are grounded in the retrieved web sources. GPT 5 (Thinking = High) is the top-performing model, scoring 56.1%, followed by o3 Pro (Thinking = On) (55.2%) and GPT 5.1 (Thinking = High) (55.1%). Models differ across domains, and in Shopping the top model scores under 50%. For some requests (such as giving the correct price or providing working links), models are highly prone to hallucination. Overall, ACE shows a substantial gap between the performance of even the best models and consumers' AI needs.
☆ STELLA: Guiding Large Language Models for Time Series Forecasting with Semantic Abstractions
Junjie Fan, Hongye Zhao, Linduo Wei, Jiayu Rao, Guijia Li, Jiaxin Yuan, Wenqi Xu, Yong Qi
Recent adaptations of Large Language Models (LLMs) for time series forecasting often fail to effectively enhance information for raw series, leaving LLM reasoning capabilities underutilized. Existing prompting strategies rely on static correlations rather than generative interpretations of dynamic behavior, lacking critical global and instance-specific context. To address this, we propose STELLA (Semantic-Temporal Alignment with Language Abstractions), a framework that systematically mines and injects structured supplementary and complementary information. STELLA employs a dynamic semantic abstraction mechanism that decouples input series into trend, seasonality, and residual components. It then translates intrinsic behavioral features of these components into Hierarchical Semantic Anchors: a Corpus-level Semantic Prior (CSP) for global context and a Fine-grained Behavioral Prompt (FBP) for instance-level patterns. Using these anchors as prefix-prompts, STELLA guides the LLM to model intrinsic dynamics. Experiments on eight benchmark datasets demonstrate that STELLA outperforms state-of-the-art methods in long- and short-term forecasting, showing superior generalization in zero-shot and few-shot settings. Ablation studies further validate the effectiveness of our dynamically generated semantic anchors.
comment : This work has been submitted to the IEEE for possible publication
☆ SEAL: Self-Evolving Agent ic Learning for Conversational Question Answering over Knowledge Graphs
Hao Wang, Jialun Zhong, Changcheng Wang, Zhujun Nie, Zheng Li, Shunyu Yao, Yanzeng Li, Xinchi Li
Knowledge-based conversational question answering (KBCQA) confronts persistent challenges in resolving coreference, modeling contextual dependencies, and executing complex logical reasoning. Existing approaches, whether end-to-end semantic parsing or stepwise agent-based reasoning, often suffer from structural inaccuracies and prohibitive computational costs, particularly when processing intricate queries over large knowledge graphs. To address these limitations, we introduce SEAL, a novel two-stage semantic parsing framework grounded in self-evolving agentic learning. In the first stage, a large language model (LLM) extracts a minimal S-expression core that captures the essential semantics of the input query. This core is then refined by an agentic calibration module, which corrects syntactic inconsistencies and aligns entities and relations precisely with the underlying knowledge graph. The second stage employs template-based completion, guided by question-type prediction and placeholder instantiation, to construct a fully executable S-expression. This decomposition not only simplifies logical form generation but also significantly enhances structural fidelity and linking efficiency. Crucially, SEAL incorporates a self-evolving mechanism that integrates local and global memory with a reflection module, enabling continuous adaptation from dialog history and execution feedback without explicit retraining. Extensive experiments on the SPICE benchmark demonstrate that SEAL achieves state-of-the-art performance, especially in multi-hop reasoning, comparison, and aggregation tasks. The results validate notable gains in both structural accuracy and computational efficiency, underscoring the framework's capacity for robust and scalable conversational reasoning.
☆ Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates
Atsuki Yamaguchi, Terufumi Morishita, Aline Villavicencio, Nikolaos Aletras
Expanding the linguistic diversity of instruct large language models (LLMs) is crucial for global accessibility but is often hindered by the reliance on costly specialized target language labeled data and catastrophic forgetting during adaptation. We tackle this challenge under a realistic, low-resource constraint: adapting instruct LLMs using only unlabeled target language data. We introduce Source-Shielded Updates (SSU), a selective parameter update strategy that proactively preserves source knowledge. Using a small set of source data and a parameter importance scoring method, SSU identifies parameters critical to maintaining source abilities. It then applies a column-wise freezing strategy to protect these parameters before adaptation. Experiments across five typologically diverse languages and 7B and 13B models demonstrate that SSU successfully mitigates catastrophic forgetting. It reduces performance degradation on monolingual source tasks to just 3.4% (7B) and 2.8% (13B) on average, a stark contrast to the 20.3% and 22.3% from full fine-tuning. SSU also achieves target-language performance highly competitive with full fine-tuning, outperforming it on all benchmarks for 7B models and the majority for 13B models.
☆ DAMASHA: Detecting AI in Mixed Adversarial Texts via Segmentation with Human-interpretable Attribution
L. D. M. S. Sai Teja, N. Siva Gopala Krishna, Ufaq Khan, Muhammad Haris Khan, Partha Pakray, Atul Mishra
In the age of advanced large language models (LLMs), the boundaries between human and AI-generated text are becoming increasingly blurred. We address the challenge of segmenting mixed-authorship text, that is identifying transition points in text where authorship shifts from human to AI or vice-versa, a problem with critical implications for authenticity, trust, and human oversight. We introduce a novel framework, called Info-Mask for mixed authorship detection that integrates stylometric cues, perplexity-driven signals, and structured boundary modeling to accurately segment collaborative human-AI content. To evaluate the robustness of our system against adversarial perturbations, we construct and release an adversarial benchmark dataset Mixed-text Adversarial setting for Segmentation (MAS), designed to probe the limits of existing detectors. Beyond segmentation accuracy, we introduce Human-Interpretable Attribution (HIA overlays that highlight how stylometric features inform boundary predictions, and we conduct a small-scale human study assessing their usefulness. Across multiple architectures, Info-Mask significantly improves span-level robustness under adversarial conditions, establishing new baselines while revealing remaining challenges. Our findings highlight both the promise and limitations of adversarially robust, interpretable mixed-authorship detection, with implications for trust and oversight in human-AI co-authorship.
comment : 18 pages, 10 Figures
☆ Are LLMs Truly Multilingual? Exploring Zero-Shot Multilingual Capability of LLMs for Information Retrieval: An Italian Healthcare Use Case
Vignesh Kumar Kembu, Pierandrea Morandini, Marta Bianca Maria Ranzini, Antonino Nocera
Large Language Models (LLMs) have become a key topic in AI and NLP, transforming sectors like healthcare, finance, education, and marketing by improving customer service, automating tasks, providing insights, improving diagnostics, and personalizing learning experiences. Information extraction from clinical records is a crucial task in digital healthcare. Although traditional NLP techniques have been used for this in the past, they often fall short due to the complexity, variability of clinical language, and high inner semantics in the free clinical text. Recently, Large Language Models (LLMs) have become a powerful tool for better understanding and generating human-like text, making them highly effective in this area. In this paper, we explore the ability of open-source multilingual LLMs to understand EHRs (Electronic Health Records) in Italian and help extract information from them in real-time. Our detailed experimental campaign on comorbidity extraction from EHR reveals that some LLMs struggle in zero-shot, on-premises settings, and others show significant variation in performance, struggling to generalize across various diseases when compared to native pattern matching and manual annotations.
☆ DaLA: Danish Linguistic Acceptability Evaluation Guided by Real Wo rl d Errors
Gianluca Barmina, Nathalie Carmen Hau Norman, Peter Schneider-Kamp, Lukas Galke
We present an enhanced benchmark for evaluating linguistic acceptability in Danish. We first analyze the most common errors found in written Danish. Based on this analysis, we introduce a set of fourteen corruption functions that generate incorrect sentences by systematically introducing errors into existing correct Danish sentences. To ensure the accuracy of these corruptions, we assess their validity using both manual and automatic methods. The results are then used as a benchmark for evaluating Large Language Models on a linguistic acceptability judgement task. Our findings demonstrate that this extension is both broader and more comprehensive than the current state of the art. By incorporating a greater variety of corruption types, our benchmark provides a more rigorous assessment of linguistic acceptability, increasing task difficulty, as evidenced by the lower performance of LLMs on our benchmark compared to existing ones. Our results also suggest that our benchmark has a higher discriminatory power which allows to better distinguish well-performing models from low-performing ones.
☆ AdiBhashaa: A Community-Curated Benchmark for Machine Translation into Indian Tribal Languages
Pooja Singh, Sandeep Kumar
Large language models and multilingual machine translation (MT) systems increasingly drive access to information, yet many languages of the tribal communities remain effectively invisible in these technologies. This invisibility exacerbates existing structural inequities in education, governance, and digital participation. We present AdiBhashaa, a community-driven initiative that constructs the first open parallel corpora and baseline MT systems for four major Indian tribal languages-Bhili, Mundari, Gondi, and Santali. This work combines participatory data creation with native speakers, human-in-the-loop validation, and systematic evaluation of both encoder-decoder MT models and large language models. In addition to reporting technical findings, we articulate how AdiBhashaa illustrates a possible model for more equitable AI research: it centers local expertise, builds capacity among early-career researchers from marginalized communities, and foregrounds human validation in the development of language technologies.
☆ MemLoRA: Distilling Expert Adapters for On-Device Memory Systems
Massimo Bini, Ondrej Bohdal, Umberto Michieli, Zeynep Akata, Mete Ozay, Taha Ceritli
Memory-augmented Large Language Models (LLMs) have demonstrated remarkable consistency during prolonged dialogues by storing relevant memories and incorporating them as context. Such memory-based personalization is also key in on-device settings that allow users to keep their conversations and data private. However, memory-augmented systems typically rely on LLMs that are too costly for local on-device deployment. Even though Small Language Models (SLMs) are more suitable for on-device inference than LLMs, they cannot achieve sufficient performance. Additionally, these LLM-based systems lack native visual capabilities, limiting their applicability in multimodal contexts. In this paper, we introduce (i) MemLoRA, a novel memory system that enables local deployment by equipping SLMs with specialized memory adapters, and (ii) its vision extension MemLoRA-V, which integrates small Vision-Language Models (SVLMs) to memory systems, enabling native visual understanding. Following knowledge distillation principles, each adapter is trained separately for specific memory operations$\unicode{x2013}$knowledge extraction, memory update, and memory-augmented generation. Equipped with memory adapters, small models enable accurate on-device memory operations without cloud dependency. On text-only operations, MemLoRA outperforms 10$\times$ larger baseline models (e.g., Gemma2-27B) and achieves performance comparable to 60$\times$ larger models (e.g., GPT-OSS-120B) on the LoCoMo benchmark. To evaluate visual understanding operations instead, we extend LoCoMo with challenging Visual Question Answering tasks that require direct visual reasoning. On this, our VLM-integrated MemLoRA-V shows massive improvements over caption-based approaches (81.3 vs. 23.7 accuracy) while keeping strong performance in text-based tasks, demonstrating the efficacy of our method in multimodal contexts.
☆ Challenging the Abilities of Large Language Models in Italian: a Community Initiative
Malvina Nissim, Danilo Croce, Viviana Patti, Pierpaolo Basile, Giuseppe Attanasio, Elio Musacchio, Matteo Rinaldi, Federico Borazio, Maria Francis, Jacopo Gili, Daniel Scalena, Begoña Altuna, Ekhi Azurmendi, Valerio Basile, Luisa Bentivogli, Arianna Bisazza, Marianna Bolognesi, Dominique Brunato, Tommaso Caselli, Silvia Casola, Maria Cassese, Mauro Cettolo, Claudia Collacciani, Leonardo De Cosmo, Maria Pia Di Buono, Andrea Esuli, Julen Etxaniz, Chiara Ferrando, Alessia Fidelangeli, Simona Frenda, Achille Fusco, Marco Gaido, Andrea Galassi, Federico Galli, Luca Giordano, Mattia Goffetti, Itziar Gonzalez-Dios, Lorenzo Gregori, Giulia Grundler, Sandro Iannaccone, Chunyang Jiang, Moreno La Quatra, Francesca Lagioia, Soda Marem Lo, Marco Madeddu, Bernardo Magnini, Raffaele Manna, Fabio Mercorio, Paola Merlo, Arianna Muti, Vivi Nastase, Matteo Negri, Dario Onorati, Elena Palmieri, Sara Papi, Lucia Passaro, Giulia Pensa, Andrea Piergentili, Daniele Potertì, Giovanni Puccetti, Federico Ranaldi, Leonardo Ranaldi, Andrea Amelio Ravelli, Martina Rosola, Elena Sofia Ruzzetti, Giuseppe Samo, Andrea Santilli, Piera Santin, Gabriele Sarti, Giovanni Sartor, Beatrice Savoldi, Antonio Serino, Andrea Seveso, Lucia Siciliani, Paolo Torroni, Rossella Varvara, Andrea Zaninello, Asya Zanollo, Fabio Massimo Zanzotto, Kamyar Zeinalipour, Andrea Zugarini
The rapid progress of Large Language Models (LLMs) has transformed natural language processing and broadened its impact across research and society. Yet, systematic evaluation of these models, especially for languages beyond English, remains limited. "Challenging the Abilities of LAnguage Models in ITAlian" (CALAMITA) is a large-scale collaborative benchmarking initiative for Italian, coordinated under the Italian Association for Computational Linguistics. Unlike existing efforts that focus on leaderboards, CALAMITA foregrounds methodology: it federates more than 80 contributors from academia, industry, and the public sector to design, document, and evaluate a diverse collection of tasks, covering linguistic competence, commonsense reasoning, factual consistency, fairness, summarization, translation, and code generation. Through this process, we not only assembled a benchmark of over 20 tasks and almost 100 subtasks, but also established a centralized evaluation pipeline that supports heterogeneous datasets and metrics. We report results for four open-weight LLMs, highlighting systematic strengths and weaknesses across abilities, as well as challenges in task-specific evaluation. Beyond quantitative results, CALAMITA exposes methodological lessons: the necessity of fine-grained, task-representative metrics, the importance of harmonized pipelines, and the benefits and limitations of broad community engagement. CALAMITA is conceived as a rolling benchmark, enabling continuous integration of new tasks and models. This makes it both a resource -- the most comprehensive and diverse benchmark for Italian to date -- and a framework for sustainable, community-driven evaluation. We argue that this combination offers a blueprint for other languages and communities seeking inclusive and rigorous LLM evaluation practices.
☆ EtCon: Edit-then-Consolidate for Reliable Knowledge Editing
Ruilin Li, Yibin Wang, Wenhong Zhu, Chenglin Li, Jinghao Zhang, Chenliang Li, Junchi Yan, Jiaqi Wang
Knowledge editing aims to update specific facts in large language models (LLMs) without full retraining. Prior efforts sought to tune the knowledge layers of LLMs, proving effective for making selective edits. However, a significant gap exists between their performance in controlled, teacher-forcing evaluations and their real-world effectiveness in lifelong learning scenarios, which greatly limits their practical applicability. This work's empirical analysis reveals two recurring issues associated with this gap: (1) Most traditional methods lead the edited model to overfit to the new fact, thereby degrading pre-trained capabilities; (2) There is a critical absence of a knowledge consolidation stage, leaving new facts insufficiently integrated into LLMs' inference-time behavior under autoregressive generation, thereby leading to a mismatch between parametric knowledge and actual generation behavior. To this end, we propose Edit-then-Consolidate, a novel knowledge editing paradigm that aims to bridge the gap between theoretical knowledge editing methods and their real-world applicability. Specifically, (1) our framework mitigates overfitting via Targeted Proximal Supervised Fine-Tuning (TPSFT) that localizes the edit via a trust-region objective to limit policy drift; (2) Then, a consolidation stage using Group Relative Policy Optimization (GRPO) aligns the edited knowledge with CoT-based inference policy by optimizing trajectory-level behavior under comprehensive reward signals. Extensive experiments demonstrate our framework consistently improves editing reliability and generalization under real-world evaluations, while better preserving locality and pre-trained capabilities.
☆ Model Whisper: Steering Vectors Unlock Large Language Models' Potential in Test-time
Xinyue Kang, Diwei Shi, Li Chen
It is a critical challenge to efficiently unlock the powerful reasoning potential of Large Language Models (LLMs) for specific tasks or new distributions. Existing test-time adaptation methods often require tuning model parameters, which is not only computationally expensive but also risks degrading the model's pre-existing abilities.To address this, we introduce a lightweight component, Test-Time Steering Vectors (TTSV), which is prepended to the input while keeping the LLM's parameters entirely frozen. By optimizing the TTSV on test data to minimize the model's output entropy, we steer the model towards an internal state of higher confidence, activating its inherent abilities most relevant to the current task. TTSV is both lightweight and highly efficient to optimize, making it a true plug-and-play enhancement. Extensive experiments validate our approach's effectiveness on both base models and reasoning-enhanced models. For instance, on the MATH500 task, TTSV achieves a 45.88% relative performance gain on the Qwen2.5-Math-7B model and a 16.22% relative gain on the Qwen3-4B model. Furthermore, our approach exhibits robust generalization, with its steering vectors proving highly transferable across diverse tasks.
comment : accepted to aaai2026
☆ SignRoundV2: Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs
Wenhua Cheng, Weiwei Zhang, Heng Guo, Haihao Shen
Extreme low-bit quantization is critical for efficiently deploying Large Language Models (LLMs), yet it often leads to severe performance degradation at 2-bits and even 4-bits (e.g., MXFP4). We present SignRoundV2, a post-training quantization framework that is highly effective even without mixed-precision. SignRoundV2 introduces (1) a fast sensitivity metric that combines gradient information with quantization-induced deviations to guide layer-wise bit allocation, and (2) a lightweight pre-tuning search for quantization scales to improve extremely low-bit quantization. These components allow SignRoundV2 to close the gap with full-precision models. Extensive experiments indicate that our method sustains competitive accuracy for LLMs, achieving production-grade performance with about 1 percent variance at 4-5 bits and strong results even at 2 bits. The implementation is available at https://github.com/intel/auto-round.
☆ OsmT: Bridging OpenStreetMap Queries and Natural Language with Open-source Tag-aware Language Models ICDE
Zhuoyue Wan, Wentao Hu, Chen Jason Zhang, Yuanfeng Song, Shuaimin Li, Ruiqiang Xiao, Xiao-Yong Wei, Raymond Chi-Wing Wong
Bridging natural language and structured query languages is a long-standing challenge in the database community. While recent advances in language models have shown promise in this direction, existing solutions often rely on large-scale closed-source models that suffer from high inference costs, limited transparency, and lack of adaptability for lightweight deployment. In this paper, we present OsmT, an open-source tag-aware language model specifically designed to bridge natural language and Overpass Query Language (OverpassQL), a structured query language for accessing large-scale OpenStreetMap (OSM) data. To enhance the accuracy and structural validity of generated queries, we introduce a Tag Retrieval Augmentation (TRA) mechanism that incorporates contextually relevant tag knowledge into the generation process. This mechanism is designed to capture the hierarchical and relational dependencies present in the OSM database, addressing the topological complexity inherent in geospatial query formulation. In addition, we define a reverse task, OverpassQL-to-Text, which translates structured queries into natural language explanations to support query interpretation and improve user accessibility. We evaluate OsmT on a public benchmark against strong baselines and observe consistent improvements in both query generation and interpretation. Despite using significantly fewer parameters, our model achieves competitive accuracy, demonstrating the effectiveness of open-source pre-trained language models in bridging natural language and structured query languages within schema-rich geospatial environments.
comment : 42nd IEEE International Conference on Data Engineering (ICDE)
☆ Towards Ethical Multi- Agent Systems of Large Language Models: A Mechanistic Interpretability Perspective AAAI'26
Jae Hee Lee, Anne Lauscher, Stefano V. Albrecht
Large language models (LLMs) have been widely deployed in various applications, often functioning as autonomous agents that interact with each other in multi-agent systems. While these systems have shown promise in enhancing capabilities and enabling complex tasks, they also pose significant ethical challenges. This position paper outlines a research agenda aimed at ensuring the ethical behavior of multi-agent systems of LLMs (MALMs) from the perspective of mechanistic interpretability. We identify three key research challenges: (i) developing comprehensive evaluation frameworks to assess ethical behavior at individual, interactional, and systemic levels; (ii) elucidating the internal mechanisms that give rise to emergent behaviors through mechanistic interpretability; and (iii) implementing targeted parameter-efficient alignment techniques to steer MALMs towards ethical behaviors without compromising their performance.
comment : Accepted to LaMAS 2026@AAAI'26 (https://sites.google.com/view/lamas2026)
☆ Geschlechtsübergreifende Maskulina im Sprachgebrauch Eine korpusbasierte Untersuchung zu lexemspezifischen Unterschieden
Carolin Mueller-Spitzer, Samira Ochs, Jan Oliver Ruediger, Sascha Wolfer
This study examines the distribution and linguistic characteristics of generic masculines (GM) in contemporary German press texts. The use of masculine personal nouns to refer to mixed-gender groups or unspecified individuals has been widely debated in academia and the public, with con-flicting perspectives on its gender-neutrality. While psycholinguistic studies suggest that GM is more readily associated with male referents, corpus-based analyses of its actual use remain scarce. We investigate GM in a large corpus of press texts, focusing on lexeme-specific differences across dif-ferent types of personal nouns. We conducted manual annotations of the whole inflectional para-digm of 21 personal nouns, resulting in 6,195 annotated tokens. Our findings reveal considerable differences between lexical items, especially between passive role nouns and prestige-related per-sonal nouns. On a grammatical level, we find that GM occurs predominantly in the plural and in indefinite noun phrases. Furthermore, our data shows that GM is not primarily used to denote entire classes of people, as has been previously claimed. By providing an empirical insight into the use of GM in authentic written language, we contribute to a more nuanced understanding of its forms and manifestations. These findings provide a solid basis for aligning linguistic stimuli in psy-cholinguistic studies more closely with real-world language use.
comment : 32 pages, 8 figures
☆ Topology Matters: Measuring Memory Leakage in Multi- Agent LLMs ACL
Jinbo Liu, Defu Cao, Yifei Wei, Tianyao Su, Yuan Liang, Yushun Dong, Yue Zhao, Xiyang Hu
Graph topology is a fundamental determinant of memory leakage in multi-agent LLM systems, yet its effects remain poorly quantified. We introduce MAMA (Multi-Agent Memory Attack), a framework that measures how network structure shapes leakage. MAMA operates on synthetic documents containing labeled Personally Identifiable Information (PII) entities, from which we generate sanitized task instructions. We execute a two-phase protocol: Engram (seeding private information into a target agent's memory) and Resonance (multi-round interaction where an attacker attempts extraction). Over up to 10 interaction rounds, we quantify leakage as the fraction of ground-truth PII recovered from attacking agent outputs via exact matching. We systematically evaluate six common network topologies (fully connected, ring, chain, binary tree, star, and star-ring), varying agent counts $n\in\{4,5,6\}$, attacker-target placements, and base models. Our findings reveal consistent patterns: fully connected graphs exhibit maximum leakage while chains provide strongest protection; shorter attacker-target graph distance and higher target centrality significantly increase vulnerability; leakage rises sharply in early rounds before plateauing; model choice shifts absolute leakage rates but preserves topology rankings; temporal/locational PII attributes leak more readily than identity credentials or regulated identifiers. These results provide the first systematic mapping from architectural choices to measurable privacy risk, yielding actionable guidance: prefer sparse or hierarchical connectivity, maximize attacker-target separation, limit node degree and network radius, avoid shortcuts bypassing hubs, and implement topology-aware access controls.
comment : Under review at ACL Rolling Review
☆ SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decoding
Chang-Hsun Wu, Kai-Po Chang, Yu-Yang Sheng, Hung-Kai Chung, Kuei-Chun Wang, Yu-Chiang Frank Wang
Video Large Language Models (VideoLLMs) have shown remarkable progress in video understanding. However, these models still struggle to effectively perceive and exploit rich temporal information in videos when responding to user queries. Therefore, they often generate descriptions of events that are temporal inconsistent or causally implausible, causing severe hallucination issues. While most prior studies have focused on spatial hallucinations (e.g. object mismatches), temporal reasoning in video understanding remains relatively underexplored. To address this issue, we propose Self-Diagnostic Contrastive Decoding (SEASON), a training-free method that adaptively enhances temporal and spatial faithfulness for each output token. It achieves this by dynamically diagnosing each token's hallucination tendency and applying adaptive contrastive decoding against its corresponding temporal and spatial negatives. Extensive experiments demonstrate that SEASON outperforms all existing training-free hallucination mitigation approaches on three hallucination examination benchmarks, while further improves VideoLLMs across four general video understanding benchmarks. The code will be released upon acceptance.
☆ Limit cycles for speech
Adamantios I. Gafos, Stephan R. Kuberski
Rhythmic fluctuations in acoustic energy and accompanying neuronal excitations in cortical oscillations are characteristic of human speech, yet whether a corresponding rhythmicity inheres in the articulatory movements that generate speech remains unclear. The received understanding of speech movements as discrete, goal-oriented actions struggles to make contact with the rhythmicity findings. In this work, we demonstrate that an unintuitive -- but no less principled than the conventional -- representation for discrete movements reveals a pervasive limit cycle organization and unlocks the recovery of previously inaccessible rhythmic structure underlying the motor activity of speech. These results help resolve a time-honored tension between the ubiquity of biological rhythmicity and discreteness in speech, the quintessential human higher function, by revealing a rhythmic organization at the most fundamental level of individual articulatory actions.
☆ Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space
Joey Hong, Kang Liu, Zhan Ling, Jiecao Chen, Sergey Levine
Large language model (LLM) agents -- LLMs that dynamically interact with an environment over long horizons -- have become an increasingly important area of research, enabling automation in complex tasks involving tool-use, web browsing, and dialogue with people. In the absence of expert demonstrations, training LLM agents has relied on policy gradient methods that optimize LLM policies with respect to an (often sparse) reward function. However, in long-horizon tasks with sparse rewards, learning from trajectory-level rewards can be noisy, leading to training that is unstable and has high sample complexity. Furthermore, policy improvement hinges on discovering better actions through exploration, which can be difficult when actions lie in natural language space. In this paper, we propose Natural Language Actor-Critic (NLAC), a novel actor-critic algorithm that trains LLM policies using a generative LLM critic that produces natural language rather than scalar values. This approach leverages the inherent strengths of LLMs to provide a richer and more actionable training signal; particularly, in tasks with large, open-ended action spaces, natural language explanations for why an action is suboptimal can be immensely useful for LLM policies to reason how to improve their actions, without relying on random exploration. Furthermore, our approach can be trained off-policy without policy gradients, offering a more data-efficient and stable alternative to existing on-policy methods. We present results on a mixture of reasoning, web browsing, and tool-use with dialogue tasks, demonstrating that NLAC shows promise in outperforming existing training approaches and offers a more scalable and stable training paradigm for LLM agents.
comment : 22 pages, 4 figures
☆ LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence
Wenjin Liu, Haoran Luo, Xin Feng, Xiang Ji, Lijuan Zhou, Rui Mao, Jiapu Wang, Shirui Pan, Erik Cambria
Legal general intelligence (GI) refers to artificial intelligence (AI) that encompasses legal understanding, reasoning, and decision-making, simulating the expertise of legal experts across domains. However, existing benchmarks are result-oriented and fail to systematically evaluate the legal intelligence of large language models (LLMs), hindering the development of legal GI. To address this, we propose LexGenius, an expert-level Chinese legal benchmark for evaluating legal GI in LLMs. It follows a Dimension-Task-Ability framework, covering seven dimensions, eleven tasks, and twenty abilities. We use the recent legal cases and exam questions to create multiple-choice questions with a combination of manual and LLM reviews to reduce data leakage risks, ensuring accuracy and reliability through multiple rounds of checks. We evaluate 12 state-of-the-art LLMs using LexGenius and conduct an in-depth analysis. We find significant disparities across legal intelligence abilities for LLMs, with even the best LLMs lagging behind human legal professionals. We believe LexGenius can assess the legal intelligence abilities of LLMs and enhance legal GI development. Our project is available at https://github.com/QwenQKing/LexGenius.
☆ ADAPT: Learning Task Mixtures for Budget-Constrained Instruction Tuning
Pritam Kadasi, Abhishek Upperwal, Mayank SIngh
We propose ADAPT, a meta-learning algorithm that \emph{learns} task sampling proportions under an explicit token budget for multi-task instruction tuning. Instead of fixing task weights by hand, \adapt{} maintains a continuous distribution over tasks and updates it via meta-gradients of a smooth worst-case validation objective, inducing an adaptive curriculum that allocates more tokens to useful tasks while avoiding collapse. We instantiate ADAPT on three $\sim$1B-parameter open-weight LLMs (Gemma-3-1B, LLaMA-3.2-1B, Qwen-0.6B), training on 20 Natural Instructions task types under budgets of $1\%$, $5\%$, and $10\%$ of the available supervised tokens, and compare against strong supervised fine-tuning baselines with uniform and size-proportional mixing. We conduct evaluations on 11 out-of-domain benchmarks spanning reasoning, reading comprehension, code generation, and instruction following, we find that ADAPT matches or slightly improves average downstream performance relative to the best static mixture, while using fewer effective training tokens and reallocating budget toward harder, benchmark-aligned tasks.
comment : Under Review
☆ AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees NeurIPS 2025
Yangning Li, Shaoshen Chen, Yinghui Li, Yankai Chen, Hai-Tao Zheng, Hui Wang, Wenhao Jiang, Philip S. Yu
The quadratic complexity of self-attention constrains Large Language Models (LLMs) in processing long contexts, a capability essential for many advanced applications. Context compression aims to alleviate this computational bottleneck while retaining critical semantic information. However, existing approaches often fall short: explicit methods may compromise local detail, whereas implicit methods can suffer from positional biases, information degradation, or an inability to capture long-range semantic dependencies. We propose AdmTree, a novel framework for adaptive, hierarchical context compression with a central focus on preserving high semantic fidelity while maintaining efficiency. AdmTree dynamically segments input based on information density, utilizing gist tokens to summarize variable-length segments as the leaves of a semantic binary tree. This structure, together with a lightweight aggregation mechanism and a frozen backbone LLM (thereby minimizing new trainable parameters), enables efficient hierarchical abstraction of the context. By preserving fine-grained details alongside global semantic coherence, mitigating positional bias, and dynamically adapting to content, AdmTree robustly retains the semantic information of long contexts.
comment : NeurIPS 2025
☆ EvoEdit: Lifelong Free-Text Knowledge Editing through Latent Perturbation Augmentation and Knowledge-driven Parameter Fusion
Pengfei Cao, Zeao Ji, Daojian Zeng, Jun Zhao, Kang Liu
Adjusting the outdated knowledge of large language models (LLMs) after deployment remains a major challenge. This difficulty has spurred the development of knowledge editing, which seeks to accurately and efficiently modify a model's internal (parametric) knowledge without retraining it from scratch. However, existing methods suffer from two limitations. First, they depend on structured triplets that are misaligned with the free-text nature of LLM pretraining and fail to capture the nuanced relationships among facts. Second, they typically support one-time knowledge updates, with relatively limited research on the problem of sequential or lifelong editing. To address these gaps, we propose a new task, Lifelong Free-text Knowledge Editing (LF-Edit), which enables models to incorporate updates expressed in natural language and supports continual editing over time. Despite its promise, LF-Edit faces the dual challenge of integrating new knowledge while mitigating the forgetting of prior information. To foster research on this new task, we construct a large-scale benchmark, Multi-Rank Lifelong Free-text Editing Benchmark (MRLF-Bench), containing 16,835 free-text edit requests. We further design a cognitively inspired multi-rank evaluation framework encompassing four levels: memorization, understanding, constrained comprehension, and reasoning. To tackle the challenges inherent in LF-Edit, we introduce a novel approach named EvoEdit that enhances knowledge injection through Latent Perturbation Augmentation and preserves prior information via Knowledge-driven Parameter Fusion. Experimental results demonstrate that EvoEdit substantially outperforms existing knowledge editing methods on the proposed LF-Edit task.
☆ UW-BioNLP at ChemoTimelines 2025: Thinking, Fine-Tuning, and Dictionary-Enhanced LLM Systems for Chemotherapy Timeline Extraction
Tianmai M. Zhang, Zhaoyi Sun, Sihang Zeng, Chenxi Li, Neil F. Abernethy, Barbara D. Lam, Fei Xia, Meliha Yetisgen
The ChemoTimelines shared task benchmarks methods for constructing timelines of systemic anticancer treatment from electronic health records of cancer patients. This paper describes our methods, results, and findings for subtask 2 -- generating patient chemotherapy timelines from raw clinical notes. We evaluated strategies involving chain-of-thought thinking, supervised fine-tuning, direct preference optimization, and dictionary-based lookup to improve timeline extraction. All of our approaches followed a two-step workflow, wherein an LLM first extracted chemotherapy events from individual clinical notes, and then an algorithm normalized and aggregated events into patient-level timelines. Each specific method differed in how the associated LLM was utilized and trained. Multiple approaches yielded competitive performances on the test set leaderboard, with fine-tuned Qwen3-14B achieving the best official score of 0.678. Our results and analyses could provide useful insights for future attempts on this task as well as the design of similar tasks.
comment : To be published in Proceedings of the 7th Clinical Natural Language Processing Workshop
☆ MSME: A Multi-Stage Multi-Expert Framework for Zero-Shot Stance Detection
Yuanshuo Zhang, Aohua Li, Bo Chen, Jingbo Sun, Xiaobing Zhao
LLM-based approaches have recently achieved impressive results in zero-shot stance detection. However, they still struggle in complex real-world scenarios, where stance understanding requires dynamic background knowledge, target definitions involve compound entities or events that must be explicitly linked to stance labels, and rhetorical devices such as irony often obscure the author's actual intent. To address these challenges, we propose MSME, a Multi-Stage, Multi-Expert framework for zero-shot stance detection. MSME consists of three stages: (1) Knowledge Preparation, where relevant background knowledge is retrieved and stance labels are clarified; (2) Expert Reasoning, involving three specialized modules-Knowledge Expert distills salient facts and reasons from a knowledge perspective, Label Expert refines stance labels and reasons accordingly, and Pragmatic Expert detects rhetorical cues such as irony to infer intent from a pragmatic angle; (3) Decision Aggregation, where a Meta-Judge integrates all expert analyses to produce the final stance prediction. Experiments on three public datasets show that MSME achieves state-of-the-art performance across the board.
☆ RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearning
Guoshenghui Zhao, Huawei Lin, Weijie Zhao
Removing specific data influence from large language models (LLMs) remains challenging, as retraining is costly and existing approximate unlearning methods are often unstable. The challenge is exacerbated when the forget set is small or imbalanced. We introduce RapidUn, an influence-driven and parameter-efficient unlearning framework. It first estimates per-sample influence through a fast estimation module, then maps these scores into adaptive update weights that guide selective parameter updates -- forgetting harmful behavior while retaining general knowledge. On Mistral-7B and Llama-3-8B across Dolly-15k and Alpaca-57k, RapidUn achieves up to 100 times higher efficiency than full retraining and consistently outperforms Fisher, GA, and LoReUn on both in-distribution and out-of-distribution forgetting. These results establish influence-guided parameter reweighting as a scalable and interpretable paradigm for LLM unlearning.
comment : Code available at: https://github.com/eyerf/RapidUn
☆ Sarcasm Detection on Reddit Using Classical Machine Learning and Feature Engineering
Subrata Karmaker
Sarcasm is common in online discussions, yet difficult for machines to identify because the intended meaning often contradicts the literal wording. In this work, I study sarcasm detection using only classical machine learning methods and explicit feature engineering, without relying on neural networks or context from parent comments. Using a 100,000-comment subsample of the Self-Annotated Reddit Corpus (SARC 2.0), I combine word-level and character-level TF-IDF features with simple stylistic indicators. Four models are evaluated: logistic regression, a linear SVM, multinomial Naive Bayes, and a random forest. Naive Bayes and logistic regression perform the strongest, achieving F1-scores around 0.57 for sarcastic comments. Although the lack of conversational context limits performance, the results offer a clear and reproducible baseline for sarcasm detection using lightweight and interpretable methods.
comment : 11 pages, 2 figures, includes full Python code. Classical machine learning baseline for sarcasm detection on the SARC 2.0 dataset
☆ MASE: Interpretable NLP Models via Model-Agnostic Saliency Estimation
Zhou Yang, Shunyan Luo, Jiazhen Zhu, Fang Jin
Deep neural networks (DNNs) have made significant strides in Natural Language Processing (NLP), yet their interpretability remains elusive, particularly when evaluating their intricate decision-making processes. Traditional methods often rely on post-hoc interpretations, such as saliency maps or feature visualization, which might not be directly applicable to the discrete nature of word data in NLP. Addressing this, we introduce the Model-agnostic Saliency Estimation (MASE) framework. MASE offers local explanations for text-based predictive models without necessitating in-depth knowledge of a model's internal architecture. By leveraging Normalized Linear Gaussian Perturbations (NLGP) on the embedding layer instead of raw word inputs, MASE efficiently estimates input saliency. Our results indicate MASE's superiority over other model-agnostic interpretation methods, especially in terms of Delta Accuracy, positioning it as a promising tool for elucidating the operations of text-based models in NLP.
☆ LangSAT: A Novel Framework Combining NLP and Reinforcement Learning for SAT Solving
Muyu Pan, Matthew Walter, Dheeraj Kodakandla, Mahfuza Farooque
Our work presents a novel reinforcement learning (RL) based framework to optimize heuristic selection within the conflict-driven clause learning (CDCL) process, improving the efficiency of Boolean satisfiability (SAT) solving. The proposed system, LangSAT, bridges the gap between natural language inputs and propositional logic by converting English descriptions into Conjunctive Normal Form (CNF) expressions and solving them using an RL-enhanced CDCL SAT solver. Unlike existing SAT-solving platforms that require CNF as input, LangSAT enables users to input standard English descriptions, making SAT-solving more accessible. The framework comprises two key components: Lang2Logic, which translates English sentences into CNF expressions, and SmartSAT, an RL-based SAT solver. SmartSAT encodes clause-variable relationships as structured graph representations and extracts global features specific to the SAT problem. This implementation provides the RL agent with deeper contextual information, enabling SAT problems to be solved more efficiently. Lang2Logic was evaluated on diverse natural language inputs, processing descriptions up to 450 words. The generated CNFs were solved by SmartSAT, which demonstrated comparable performance to traditional CDCL heuristics with respect to solving time. The combined LangSAT framework offers a more accessible and scalable solution for SAT-solving tasks across reasoning, formal verification, and debugging.
☆ Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignment WACV
Kai-Po Chang, Wei-Yuan Cheng, Chi-Pin Huang, Fu-En Yang, Yu-Chiang Frank Wang
Recent advancement in multimodal LLMs (MLLMs) has demonstrated their remarkable capability to generate descriptive captions for input videos. However, these models suffer from factual inaccuracies in the generated descriptions, causing severe hallucination issues. While prior works have explored alleviating hallucinations for static images, jointly mitigating visual object and temporal action hallucinations for dynamic videos remains a challenging and unsolved task. To tackle this challenge, we propose a Self-Augmented Contrastive Alignment (SANTA) framework for enabling object and action faithfulness by exempting the spurious correlations and enforcing the emphasis on visual facts. SANTA employs a hallucinative self-augmentation scheme to identify the potential hallucinations that lie in the MLLM and transform the original captions to the contrasted negatives. Furthermore, we develop a tracklet-phrase contrastive alignment to match the regional objects and relation-guided actions with their corresponding visual and temporal phrases. Extensive experiments demonstrate that SANTA outperforms existing methods in alleviating object and action hallucinations, yielding superior performance on the hallucination examination benchmarks.
comment : IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2026. Project page: https://kpc0810.github.io/santa/
☆ ClusterFusion: Hybrid Clustering with Embedding Guidance and LLM Adaptation
Yiming Xu, Yuan Yuan, Vijay Viswanathan, Graham Neubig
Text clustering is a fundamental task in natural language processing, yet traditional clustering algorithms with pre-trained embeddings often struggle in domain-specific contexts without costly fine-tuning. Large language models (LLMs) provide strong contextual reasoning, yet prior work mainly uses them as auxiliary modules to refine embeddings or adjust cluster boundaries. We propose ClusterFusion, a hybrid framework that instead treats the LLM as the clustering core, guided by lightweight embedding methods. The framework proceeds in three stages: embedding-guided subset partition, LLM-driven topic summarization, and LLM-based topic assignment. This design enables direct incorporation of domain knowledge and user preferences, fully leveraging the contextual adaptability of LLMs. Experiments on three public benchmarks and two new domain-specific datasets demonstrate that ClusterFusion not only achieves state-of-the-art performance on standard tasks but also delivers substantial gains in specialized domains. To support future work, we release our newly constructed dataset and results on all benchmarks.
♻ ☆ Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Model s
Shuai Wang, Zhenhua Liu, Jiaheng Wei, Xuanwu Yin, Dong Li, Emad Barsoum
We present Athena-PRM, a multimodal process reward model (PRM) designed to evaluate the reward score for each step in solving complex reasoning problems. Developing high-performance PRMs typically demands significant time and financial investment, primarily due to the necessity for step-level annotations of reasoning steps. Conventional automated labeling methods, such as Monte Carlo estimation, often produce noisy labels and incur substantial computational costs. To efficiently generate high-quality process-labeled data, we propose leveraging prediction consistency between weak and strong completers as a criterion for identifying reliable process labels. Remarkably, Athena-PRM demonstrates outstanding effectiveness across various scenarios and benchmarks with just 5,000 samples. Furthermore, we also develop two effective strategies to improve the performance of PRMs: ORM initialization and up-sampling for negative data. We validate our approach in three specific scenarios: verification for test time scaling, direct evaluation of reasoning step correctness, and reward ranked fine-tuning. Our Athena-PRM consistently achieves superior performance across multiple benchmarks and scenarios. Notably, when using Qwen2.5-VL-7B as the policy model, Athena-PRM enhances performance by 10.2 points on WeMath and 7.1 points on MathVista for test time scaling. Furthermore, Athena-PRM sets the state-of-the-art (SoTA) results in VisualProcessBench and outperforms the previous SoTA by 3.9 F1-score, showcasing its robust capability to accurately assess the correctness of the reasoning step. Additionally, utilizing Athena-PRM as the reward model, we develop Athena-7B with reward ranked fine-tuning and outperforms baseline with a significant margin on five benchmarks.
♻ ☆ TreeRare: Syntax Tree-Guided Retrieval and Reasoning for Knowledge-Intensive Question Answering
Boyi Zhang, Zhuo Liu, Hangfeng He
In real practice, questions are typically complex and knowledge-intensive, requiring Large Language Models (LLMs) to recognize the multifaceted nature of the question and reason across multiple information sources. Iterative and adaptive retrieval, where LLMs decide when and what to retrieve based on their reasoning, has been shown to be a promising approach to resolve complex, knowledge-intensive questions. However, the performance of such retrieval frameworks is limited by the accumulation of reasoning errors and misaligned retrieval results. To overcome these limitations, we propose TreeRare (Syntax Tree-Guided Retrieval and Reasoning), a framework that utilizes syntax trees to guide information retrieval and reasoning for question answering. Following the principle of compositionality, TreeRare traverses the syntax tree in a bottom-up fashion, and in each node, it generates subcomponent-based queries and retrieves relevant passages to resolve localized uncertainty. A subcomponent question answering module then synthesizes these passages into concise, context-aware evidence. Finally, TreeRare aggregates the evidence across the tree to form a final answer. Experiments across five question answering datasets involving ambiguous or multi-hop reasoning demonstrate that TreeRare achieves substantial improvements over existing state-of-the-art methods.
♻ ☆ SO-Bench: A Structural Output Evaluation of Multimodal LLMs
Di Feng, Kaixin Ma, Feng Nan, Haofeng Chen, Bohan Zhai, David Griffiths, Mingfei Gao, Zhe Gan, Eshan Verma, Yinfei Yang, Zhifeng Chen, Afshin Dehghan
Multimodal large language models (MLLMs) are increasingly deployed in real-world, agentic settings where outputs must not only be correct, but also conform to predefined data schemas. Despite recent progress in structured generation in textual domain, there is still no benchmark that systematically evaluates schema-grounded information extraction and reasoning over visual inputs. In this work, we conduct a comprehensive study of visual structural output capabilities for MLLMs with our carefully designed SO-Bench benchmark. Covering four visual domains, including UI screens, natural images, documents, and charts, SO-Bench is built from over 6.5K diverse JSON schemas and 1.8K curated image-schema pairs with human-verified quality. Benchmarking experiments on open-sourced and frontier proprietary models reveal persistent gaps in predicting accurate, schema compliant outputs, highlighting the need for better multimodal structured reasoning. Beyond benchmarking, we further conduct training experiments to largely improve the model's structured output capability. We plan to make the benchmark available to the community.
comment : v2 preprint. Fixed some typos, add a discussion about limitation, provide pseudo-codes for eval
♻ ☆ Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data
Shaoxiong Ji, Zihao Li, Jaakko Paavola, Hengyu Luo, Jörg Tiedemann
This paper investigates a critical design decision in the practice of massively multilingual continual pre-training -- the inclusion of parallel data. Specifically, we study the impact of bilingual translation data for massively multilingual language adaptation of the Llama3 family of models to 500 languages. To this end, we construct the MaLA bilingual translation corpus, containing data from more than 2,500 language pairs. Subsequently, we develop the EMMA-500 Llama 3 suite of four massively multilingual models -- continually pre-trained from the Llama 3 family of base models extensively on diverse data mixes up to 671B tokens -- and explore the effect of continual pre-training with or without bilingual translation data. Comprehensive evaluation across 7 tasks and 12 benchmarks demonstrates that bilingual data tends to enhance language transfer and performance, particularly for low-resource languages. We open-source the MaLA corpus, EMMA-500 Llama 3 suite artefacts, code, and model generations.
comment : EMMA-500 Gen 2; refer to Gen 1 in arXiv:2409.17892
♻ ☆ QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA EMNLP 2025
Jacob Dineen, Aswin RRV, Qin Liu, Zhikun Xu, Xiao Ye, Ming Shen, Zhaonan Li, Shijie Lu, Chitta Baral, Muhao Chen, Ben Zhou
Alignment of large language models (LLMs) with principles like helpfulness, honesty, and harmlessness typically relies on scalar rewards that obscure which objectives drive the training signal. We introduce QA-LIGN, which decomposes monolithic rewards into interpretable principle-specific evaluations through structured natural language programs. Models learn through a draft, critique, and revise pipeline, where symbolic evaluation against the rubrics provides transparent feedback for both initial and revised responses during GRPO training. Applied to uncensored Llama-3.1-8B-Instruct, QA-LIGN reduces attack success rates by up to 68.7% while maintaining a 0.67% false refusal rate, achieving Pareto optimal safety-helpfulness performance and outperforming both DPO and GRPO with state-of-the-art reward models given equivalent training. These results demonstrate that making reward signals interpretable and modular improves alignment effectiveness, suggesting transparency enhances LLM safety.
comment : Findings of the Association for Computational Linguistics: EMNLP 2025, pages 20619-20642, Suzhou, China
♻ ☆ PUCP-Metrix: An Open-source and Comprehensive Toolkit for Linguistic Analysis of Spanish Texts EACL
Javier Alonso Villegas Luis, Marco Antonio Sobrevilla Cabezudo
Linguistic features remain essential for interpretability and tasks that involve style, structure, and readability, but existing Spanish tools offer limited coverage. We present PUCP-Metrix, an open-source and comprehensive toolkit for linguistic analysis of Spanish texts. PUCP-Metrix includes 182 linguistic metrics spanning lexical diversity, syntactic and semantic complexity, cohesion, psycholinguistics, and readability. It enables fine-grained, interpretable text analysis. We evaluate its usefulness on Automated Readability Assessment and Machine-Generated Text Detection, showing competitive performance compared to an existing repository and strong neural baselines. PUCP-Metrix offers a comprehensive and extensible resource for Spanish, supporting diverse NLP applications.
comment : 1 figure, Submitted to EACL Demo track (under review)
♻ ☆ LORE: A Large Generative Model for Search Relevance
Chenji Lu, Zhuo Chen, Hui Zhao, Zhiyuan Zeng, Gang Zhao, Junjie Ren, Ruicong Xu, Haoran Li, Songyan Liu, Pengjie Wang, Jian Xu, Bo Zheng
Achievement. We introduce LORE, a systematic framework for Large Generative Model-based relevance in e-commerce search. Deployed and iterated over three years, LORE achieves a cumulative +27\% improvement in online GoodRate metrics. This report shares the valuable experience gained throughout its development lifecycle, spanning data, features, training, evaluation, and deployment. Insight. While existing works apply Chain-of-Thought (CoT) to enhance relevance, they often hit a performance ceiling. We argue this stems from treating relevance as a monolithic task, lacking principled deconstruction. Our key insight is that relevance comprises distinct capabilities: knowledge and reasoning, multi-modal matching, and rule adherence. We contend that a qualitative-driven decomposition is essential for breaking through current performance bottlenecks. Contributions. LORE provides a complete blueprint for the LLM relevance lifecycle. Key contributions include: (1) A two-stage training paradigm combining progressive CoT synthesis via SFT with human preference alignment via RL. (2) A comprehensive benchmark, RAIR, designed to evaluate these core capabilities. (3) A query frequency-stratified deployment strategy that efficiently transfers offline LLM capabilities to the online system. LORE serves as both a practical solution and a methodological reference for other vertical domains.
♻ ☆ EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models
Shaoxiong Ji, Zihao Li, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O'Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, Barry Haddow
In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages. To facilitate continual pre-training, we compile the MaLA corpus, a comprehensive multilingual dataset enriched with curated datasets across diverse domains. Leveraging this corpus, we conduct extensive continual pre-training of the Llama 2 7B model, resulting in EMMA-500, which demonstrates robust performance across a wide collection of benchmarks, including a comprehensive set of multilingual tasks. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity, particularly for underrepresented languages, demonstrating significant gains in cross-lingual transfer, task generalization, and language adaptability. We release the MaLA corpus, EMMA-500 model weights, scripts, and model generations.
♻ ☆ SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation
Marshall Thomas, Edward Fish, Richard Bowden
Despite progress in gloss-free Sign Language Translation (SLT), traditional single modality end-to-end approaches consistently fail on two critical components of natural signing: the precise recognition of high-speed fingerspelling and the integration of asynchronous non-manual cues from the face. Recent progress in SLT with Large Language Models has side stepped this challenge, forcing a single network to learn these simultaneously resulting in poor performance when tasked with translating crucial information such as names, places, and technical terms. We introduce SignBind-LLM, a modular framework designed to overcome these limitations. Our approach employs separate, specialized predictors for continuous signing, fingerspelling, and lipreading. Each expert network first decodes its specific modality into a sequence of tokens. These parallel streams are then fused by a lightweight transformer that resolves temporal misalignments before passing the combined representation to a Large Language Model (LLM) for final sentence generation. Our method establishes a new state-of-the-art on the How2Sign, ChicagoFSWildPlus, and BOBSL datasets with a BLEU-4 score of 22.1, 73.2% letter accuracy and BLEU-4 score of 6.8 respectively. These results validate our core hypothesis: isolating and solving distinct recognition tasks before fusion provides a more powerful and effective pathway to robust, high-fidelity sign language translation.
♻ ☆ HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks ICLR 2026
Adnan El Assadi, Isaac Chung, Roman Solomatin, Niklas Muennighoff, Kenneth Enevoldsen
Comparing human and model performance offers a valuable perspective for understanding the strengths and limitations of embedding models, highlighting where they succeed and where they fail to capture meaning and nuance. However, such comparisons are rarely made, as human performance on embedding tasks is difficult to measure. To fill this gap, we introduce HUME: Human Evaluation Framework for Text Embeddings. While frameworks like MTEB provide broad model evaluation, they lack reliable estimates of human performance, limiting the interpretability of model scores. We measure human performance across 16 MTEB datasets spanning reranking, classification, clustering, and semantic textual similarity across linguistically diverse high- and low-resource languages. Humans achieve an average performance of 77.6% compared to 80.1% for the best embedding model, though with substantial variation: models reach high performance on some datasets while struggling on notably low-resource languages. Our human annotations also reveal multiple dataset issues. We additionally benchmark nine LLMs as annotators on reranking, classification, and STS tasks, finding that they fall short of human performance (76.1% vs. 81.2%) despite offering scalability advantages. We provide human performance baselines, insights into task difficulty patterns, and an extensible evaluation framework that enables a more meaningful interpretation of results and informs the development of both models and benchmarks. Our code, dataset, and leaderboard are publicly available at https://github.com/embeddings-benchmark/mteb.
comment : Submitted to ICLR 2026
♻ ☆ Large language models can learn and generalize steganographic chain-of-thought under process super vision NeurIPS 2025
Joey Skaf, Luis Ibanez-Lissen, Robert McCarthy, Connor Watts, Vasil Georgiv, Hannes Whittingham, Lorena Gonzalez-Manzano, David Lindner, Cameron Tice, Edward James Young, Puria Radmard
Chain-of-thought (CoT) reasoning not only enhances large language model performance but also provides critical insights into decision-making processes, marking it as a useful tool for monitoring model intent and planning. However, recent works have shown that banning the mention of a specific example of reward hacking causes obfuscation of the undesired reasoning traces but the persistence of the undesired behavior, threatening the reliability of CoT monitoring. We provide an extension to these results with regard to the ability of models to learn a specific type of obfuscated reasoning: steganography. First, we show that penalizing the use of specific strings within load-bearing reasoning traces causes models to substitute alternative strings. Crucially, this does not alter the underlying method by which the model performs the task, demonstrating that the model can learn to steganographically encode its reasoning.We further demonstrate that models can generalize an encoding scheme. When the penalized strings belong to an overarching class, the model learns not only to substitute strings seen in training, but also develops a general encoding scheme for all members of the class which it can apply to held-out testing strings.
comment : 10 pages main text, 3 figures main text, 17 pages supplementary material, 1 figure supplementary material, accepted at NeurIPS 2025
♻ ☆ On-Policy Optimization with Group Equivalent Preference for Multi-Programming Language Understanding
Haoyuan Wu, Rui Ming, Jilong Gao, Hangyu Zhao, Xueyi Chen, Yikai Yang, Haisheng Zheng, Zhuolun He, Bei Yu
Large language models (LLMs) achieve remarkable performance in code generation tasks. However, a significant performance disparity persists between popular programming languages (e.g., Python, C++) and others. To address this capability gap, we leverage the code translation task to train LLMs, thereby facilitating the transfer of coding proficiency across diverse programming languages. Moreover, we introduce OORL for training, a novel reinforcement learning (RL) framework that integrates on-policy and off-policy strategies. Within OORL, on-policy RL is applied during code translation, guided by a rule-based reward signal derived from unit tests. Complementing this coarse-grained rule-based reward, we propose Group Equivalent Preference Optimization (GEPO), a novel preference optimization method. Specifically, GEPO trains the LLM using intermediate representations (IRs) groups. LLMs can be guided to discern IRs equivalent to the source code from inequivalent ones, while also utilizing signals about the mutual equivalence between IRs within the group. This process allows LLMs to capture nuanced aspects of code functionality. By employing OORL for training with code translation tasks, LLMs improve their recognition of code functionality and their understanding of the relationships between code implemented in different languages. Extensive experiments demonstrate that our OORL for LLMs training with code translation tasks achieves significant performance improvements on code benchmarks across multiple programming languages.
♻ ☆ Human Mobility Dataset s Enriched With Context ual and Social Dimensions
Chiara Pugliese, Francesco Lettich, Guido Rocchietti, Chiara Renso, Fabio Pinelli
In this resource paper, we present two publicly available datasets of semantically enriched human trajectories, together with the pipeline to build them. The trajectories are publicly available GPS traces retrieved from OpenStreetMap. Each dataset includes contextual layers such as stops, moves, points of interest (POIs), inferred transportation modes, and weather data. A novel semantic feature is the inclusion of synthetic, realistic social media posts generated by Large Language Models (LLMs), enabling multimodal and semantic mobility analysis. The datasets are available in both tabular and Resource Description Framework (RDF) formats, supporting semantic reasoning and FAIR data practices. They cover two structurally distinct, large cities: Paris and New York. Our open source reproducible pipeline allows for dataset customization, while the datasets support research tasks such as behavior modeling, mobility prediction, knowledge graph construction, and LLM-based applications. To our knowledge, our resource is the first to combine real-world movement, structured semantic enrichment, LLM-generated text, and semantic web compatibility in a reusable framework.
comment : 5 pages, 3 figures, 1 table
♻ ☆ Beyond the Exploration-Exploitation Trade-off: A Hidden State Approach for LLM Reasoning in RL VR
Fanding Huang, Guanbo Huang, Xiao Fan, Yi He, Xiao Liang, Xiao Chen, Qinting Jiang, Faisal Nadeem Khan, Jingyan Jiang, Zhi Wang
A prevailing view in Reinforcement Learning with Verifiable Rewards (RLVR) interprets recent progress through the lens of an exploration-exploitation trade-off, a perspective largely shaped by token-level metrics. We re-examine this perspective, proposing that this perceived trade-off may not be a fundamental constraint but rather an artifact of the measurement level. To investigate this, we shift the analysis to the semantically rich hidden-state space, adopting Effective Rank (ER) to quantify exploration and proposing its novel first- and second-order derivatives, named ER Velocity and ER Acceleration, to capture exploitation dynamics. Our analysis reveals that in the semantic space, exploration and exploitation could be decoupled (Sec.~4). This finding reveals an opportunity to enhance both capacities simultaneously. This insight motivates our method, Velocity-Exploiting Rank-Learning (VERL), the first to operationalize the principle of synergistic exploration-exploitation enhancement by directly shaping the RL advantage function. The key innovation is leveraging the theoretically stable ERA as a predictive meta-controller to create a synergistic, dual-channel incentive structure. Instead of forcing a trade-off, VERL prospectively amplifies rewards for exploration to preempt overconfidence and reinforces exploitative gains to consolidate reasoning. Experiments across diverse LLMs and reasoning benchmarks show consistent gains, including up to 21.4% absolute accuracy improvement on the challenging Gaokao 2024 dataset.
♻ ☆ MMAG: Mixed Memory-Augmented Generation for Large Language Models Applications
Stefano Zeppieri
Large Language Models (LLMs) excel at generating coherent text within a single prompt but fall short in sustaining relevance, personalization, and continuity across extended interactions. Human communication, however, relies on multiple forms of memory, from recalling past conversations to adapting to personal traits and situational context. This paper introduces the Mixed Memory-Augmented Generation (MMAG) pattern, a framework that organizes memory for LLM-based agents into five interacting layers: conversational, long-term user, episodic and event-linked, sensory and context-aware, and short-term working memory. Drawing inspiration from cognitive psychology, we map these layers to technical components and outline strategies for coordination, prioritization, and conflict resolution. We demonstrate the approach through its implementation in the Heero conversational agent, where encrypted long-term bios and conversational history already improve engagement and retention. We further discuss implementation concerns around storage, retrieval, privacy, and latency, and highlight open challenges. MMAG provides a foundation for building memory-rich language agents that are more coherent, proactive, and aligned with human needs.
♻ ☆ Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelines
Matthew Lewis, Samuel Thio, Amy Roberts, Catherine Siju, Whoasif Mukit, Rebecca Kuruvilla, Zhangshu Joshua Jiang, Niko Möller-Grell, Aditya Borakati, Richard JB Dobson, Spiros Denaxas
This paper presents the development and evaluation of a Retrieval-Augmented Generation (RAG) system for querying the United Kingdom's National Institute for Health and Care Excellence (NICE) clinical guidelines using Large Language Models (LLMs). The extensive length and volume of these guidelines can impede their utilisation within a time-constrained healthcare system, a challenge this project addresses through the creation of a system capable of providing users with precisely matched information in response to natural language queries. The system's retrieval architecture, composed of a hybrid embedding mechanism, was evaluated against a corpus of 10,195 text chunks derived from three hundred guidelines. It demonstrates high performance, with a Mean Reciprocal Rank (MRR) of 0.814, a Recall of 81% at the first chunk and of 99.1% within the top ten retrieved chunks, when evaluated on 7901 queries. The most significant impact of the RAG system was observed during the generation phase. When evaluated on a manually curated dataset of seventy question-answer pairs, RAG-enhanced models showed substantial gains in performance. Faithfulness, the measure of whether an answer is supported by the source text, was increased by 64.7 percentage points to 99.5% for the RAG-enhanced O4-Mini model and significantly outperformed the medical-focused Meditron3-8B LLM, which scored 43%. Clinical evaluation by seven Subject Matter Experts (SMEs) further validated these findings, with GPT-4.1 achieving 98.7% accuracy while reducing unsafe responses by 67% compared to O4-Mini (from 3.0 to 1.0 per evaluator). This study thus establishes RAG as an effective, reliable, and scalable approach for applying generative AI in healthcare, enabling cost-effective access to medical guidelines.
♻ ☆ Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation
Yongfu Xue
The rapid development of parameter-efficient fine-tuning methods has noticeably improved the efficiency of adapting large language models. Among these, LoRA has gained widespread popularity due to its strong balance of effectiveness and parameter efficiency. However, LoRA relies on initializing two low-rank matrices whose product is zero, which limits its ability to effectively activate and leverage the original model weights-creating a potential bottleneck for optimal performance. To address this limitation, we propose \textbf{IniLoRA}, a novel initialization strategy that initializes the low-rank matrices to closely approximate the original model weights. Experimental results indicate that IniLoRA achieves better performance than LoRA across a range of models and tasks. Additionally, we introduce two variants, IniLoRA-$α$ and IniLoRA-$β$, both leveraging distinct initialization methods to enhance performance further.
♻ ☆ Jina-VLM: Small Multilingual Vision Language Model
Andreas Koukounas, Georgios Mastrapas, Florian Hönicke, Sedigheh Eslami, Guillaume Roncari, Scott Martens, Han Xiao
We present Jina-VLM, a 2.4B parameter vision-language model that achieves state-of-the-art multilingual visual question answering among open 2B-scale VLMs. The model couples a SigLIP2 vision encoder with a Qwen3 language backbone through an attention-pooling connector that enables token-efficient processing of arbitrary-resolution images. The model achieves leading results on standard VQA benchmarks and multilingual evaluations while preserving competitive text-only performance. Model weights and code are publicly released at https://huggingface.co/jinaai/jina-vlm .
comment : 18 pages, 1-7 main content, 13-18 appendix for tables and dataset
♻ ☆ SeSE: A Structural Information -Guided Uncertainty Quantification Framework for Hallucination Detection in LLMs
Xingtao Zhao, Hao Peng, Dingli Su, Xianghua Zeng, Chunyang Liu, Jinzhi Liao, Philip S. Yu
Reliable uncertainty quantification (UQ) is essential for deploying large language models (LLMs) in safety-critical scenarios, as it enables them to abstain from responding when uncertain, thereby avoiding ``hallucinating'' falsehoods. However, state-of-the-art UQ methods primarily rely on semantic probability distributions or pairwise distances, overlooking latent semantic structural information that could enable more precise uncertainty estimates. This paper presents Semantic Structural Entropy (SeSE), a principled UQ framework that quantifies the inherent semantic uncertainty of LLMs from a structural information perspective for hallucination detection. SeSE operates in a zero-resource manner and is applicable to both open- and closed-source LLMs, making it an ``off-the-shelf" solution for new models and tasks. Specifically, to effectively model semantic spaces, we first develop an adaptively sparsified directed semantic graph construction algorithm that captures directional semantic dependencies while automatically pruning unnecessary connections that introduce negative interference. We then exploit latent semantic structural information through hierarchical abstraction: SeSE is defined as the structural entropy of the optimal semantic encoding tree, formalizing intrinsic uncertainty within semantic spaces after optimal compression. A higher SeSE value corresponds to greater uncertainty, indicating that LLMs are highly likely to generate hallucinations. In addition, to enhance fine-grained UQ in long-form generation, we extend SeSE to quantify the uncertainty of individual claims by modeling their random semantic interactions, providing theoretically explicable hallucination detection. Extensive experiments across 29 model-dataset combinations show that SeSE significantly outperforms advanced UQ baselines.
comment : 14 pages of main text and 10 pages of appendices;Submit to IEEE TKDE
♻ ☆ Grounding LLM Reasoning with Knowledge Graphs
Alfonso Amayuelas, Joy Sain, Simerjot Kaur, Charese Smiley
Large Language Models (LLMs) excel at generating natural language answers, yet their outputs often remain unverifiable and difficult to trace. Knowledge Graphs (KGs) offer a complementary strength by representing entities and their relationships in structured form, providing a foundation for more reliable reasoning. We propose a novel framework that integrates LLM reasoning with KGs by linking each step of the reasoning process to graph-structured data. This grounding turns intermediate ``thoughts'' into interpretable traces that remain consistent with external knowledge. Our approach incorporates multiple reasoning strategies, Chain-of-Thought (CoT), Tree-of-Thought (ToT), and Graph-of-Thought (GoT), and is evaluated on GRBench, a benchmark for domain-specific graph reasoning. Our experiments show state-of-the-art (SOTA) performance, with at least 26.5\% improvement over CoT baselines. Beyond accuracy, we analyze how step depth, branching structure, and model size influence reasoning quality, offering insights into the conditions that support effective reasoning. Together, these contributions highlight how grounding LLMs in structured knowledge enables both higher accuracy and greater interpretability in complex reasoning tasks.
♻ ☆ Control Illusion: The Failure of Instruction Hierarchies in Large Language Models AAAI-26
Yilin Geng, Haonan Li, Honglin Mu, Xudong Han, Timothy Baldwin, Omri Abend, Eduard Hovy, Lea Frermann
Large language models (LLMs) are increasingly deployed with hierarchical instruction schemes, where certain instructions (e.g., system-level directives) are expected to take precedence over others (e.g., user messages). Yet, we lack a systematic understanding of how effectively these hierarchical control mechanisms work. We introduce a systematic evaluation framework based on constraint prioritization to assess how well LLMs enforce instruction hierarchies. Our experiments across six state-of-the-art LLMs reveal that models struggle with consistent instruction prioritization, even for simple formatting conflicts. We find that the widely-adopted system/user prompt separation fails to establish a reliable instruction hierarchy, and models exhibit strong inherent biases toward certain constraint types regardless of their priority designation. Interestingly, we also find that societal hierarchy framings (e.g., authority, expertise, consensus) show stronger influence on model behavior than system/user roles, suggesting that pretraining-derived social structures function as latent behavioral priors with potentially greater impact than post-training guardrails.
comment : Accepted to AAAI-26 Main Technical Track Proceedings
♻ ☆ Bridging Online Behavior and Clinical Insight: A Longitudinal LLM-based Study of Suicidality on YouTube Reveals Novel Digital Markers
Ilanit Sobol, Shir Lissak, Refael Tikochinski, Tal Nakash, Anat Brunstein Klomek, Eyal Fruchter, Roi Reichart
Suicide remains a leading cause of death in Western countries. As social media becomes central to daily life, digital footprints offer valuable insight into suicidal behavior. Focusing on individuals who attempted suicide while uploading videos to their channels, we investigate: How do linguistic patterns on YouTube reflect suicidal behavior, and how do these patterns align with or differ from expert knowledge? We examined linguistic changes around suicide attempts and compared individuals who attempted suicide while actively uploading to their channel with three control groups: those with prior attempts, those experiencing major life events, and matched individuals from the broader cohort. Applying complementary bottom-up, hybrid, and expert-driven approaches, we analyzed a novel longitudinal dataset of 181 suicide-attempt channels and 134 controls. In the bottom-up analysis, LLM-based topic-modeling identified 166 topics; five were linked to suicide attempts, two also showed attempt-related temporal changes (Mental Health Struggles, $OR = 1.74$; YouTube Engagement, $OR = 1.67$; $p < .01$). In the hybrid approach, clinical experts reviewed LLM-derived topics and flagged 19 as suicide-related. However, none showed significant effects beyond those identified bottom-up. YouTube Engagement, a platform-specific indicator, was not flagged, underscoring the value of bottom-up discovery. A top-down psychological assessment of suicide narratives revealed differing motivations: individuals describing prior attempts aimed to help others ($β=-1.69$, $p<.01$), whereas those attempted during the uploading period emphasized personal recovery ($β=1.08$, $p<.01$). By integrating these approaches, we offer a nuanced understanding of suicidality, bridging digital behavior and clinical insights.
♻ ☆ Chat GPT for President! Presupposed content in politicians versus GPT -generated texts
Davide Garassino, Nicola Brocca, Viviana Masia
This study examines ChatGPT-4's capability to replicate linguistic strategies used in political discourse, focusing on its potential for manipulative language generation. As large language models become increasingly popular for text generation, concerns have grown regarding their role in spreading fake news and propaganda. This research compares real political speeches with those generated by ChatGPT, emphasizing presuppositions (a rhetorical device that subtly influences audiences by packaging some content as already known at the moment of utterance, thus swaying opinions without explicit argumentation). Using a corpus-based pragmatic analysis, this study assesses how well ChatGPT can mimic these persuasive strategies. The findings reveal that although ChatGPT-generated texts contain many manipulative presuppositions, key differences emerge in their frequency, form, and function compared with those of politicians. For instance, ChatGPT often relies on change-of-state verbs used in fixed phrases, whereas politicians use presupposition triggers in more varied and creative ways. Such differences, however, are challenging to detect with the naked eye, underscoring the potential risks posed by large language models in political and public discourse.Using a corpus-based pragmatic analysis, this study assesses how well ChatGPT can mimic these persuasive strategies. The findings reveal that although ChatGPT-generated texts contain many manipulative presuppositions, key differences emerge in their frequency, form, and function compared with those of politicians. For instance, ChatGPT often relies on change-of-state verbs used in fixed phrases, whereas politicians use presupposition triggers in more varied and creative ways. Such differences, however, are challenging to detect with the naked eye, underscoring the potential risks posed by large language models in political and public discourse.
comment : 36 pages, 6 figures
♻ ☆ Which Type of Students can LLMs Act? Investigating Authentic Simulation with Graph-based Human-AI Collaborative System
Haoxuan Li, Jifan Yu, Xin Cong, Yang Dang, Daniel Zhang-li, Lu Mi, Yisi Zhan, Huiqin Liu, Zhiyuan Liu
While rapid advances in large language models (LLMs) are reshaping data-driven intelligent education, accurately simulating students remains an important but challenging bottleneck for scalable educational data collection, evaluation, and intervention design. However, current works are limited by scarce real interaction data, costly expert evaluation for realism, and a lack of large-scale, systematic analyses of LLMs ability in simulating students. We address this gap by presenting a three-stage LLM-human collaborative pipeline to automatically generate and filter high-quality student agents. We leverage a two-round automated scoring validated by human experts and deploy a score propagation module to obtain more consistent scores across the student similarity graph. Experiments show that combining automated scoring, expert calibration, and graph-based propagation yields simulated student that more closely track authentication by human judgments. We then analyze which profiles and behaviors are simulated more faithfully, supporting subsequent studies on personalized learning and educational assessment.
comment : This work has been submitted to AI Open for possible publication
♻ ☆ Thai OCR Bench: A Task-Diverse Benchmark for Vision -Language Understanding in Thai AACL 2025
Surapon Nonesung, Teetouch Jaknamon, Sirinya Chaiophat, Natapong Nitarach, Chanakan Wittayasakpan, Warit Sirichotedumrong, Adisai Na-Thalang, Kunat Pipatanakul
We present ThaiOCRBench, the first comprehensive benchmark for evaluating vision-language models (VLMs) on Thai text-rich visual understanding tasks. Despite recent progress in multimodal modeling, existing benchmarks predominantly focus on high-resource languages, leaving Thai underrepresented, especially in tasks requiring document structure understanding. ThaiOCRBench addresses this gap by offering a diverse, human-annotated dataset comprising 2,808 samples across 13 task categories. We evaluate a wide range of state-of-the-art VLMs in a zero-shot setting, spanning both proprietary and open-source systems. Results show a significant performance gap, with proprietary models (e.g., Gemini 2.5 Pro) outperforming open-source counterparts. Notably, fine-grained text recognition and handwritten content extraction exhibit the steepest performance drops among open-source models. Through detailed error analysis, we identify key challenges such as language bias, structural mismatch, and hallucinated content. ThaiOCRBench provides a standardized framework for assessing VLMs in low-resource, script-complex settings, and provides actionable insights for improving Thai-language document understanding.
comment : Accepted at IJCNLP-AACL 2025 (Main). This version includes the corrected Table 2 and an updated conclusion regarding the deletion count of the Gemma model
♻ ☆ TaoSR1: The Thinking Model for E-commerce Relevance Search
Chenhe Dong, Shaowei Yao, Pengkun Jiao, Jianhui Yang, Yiming Jin, Zerui Huang, Xiaojiang Zhou, Dan Ou, Haihong Tang, Bo Zheng
Query-product relevance prediction is a core task in e-commerce search. BERT-based models excel at semantic matching but lack complex reasoning capabilities. While Large Language Models (LLMs) are explored, most still use discriminative fine-tuning or distill to smaller models for deployment. We propose a framework to directly deploy LLMs for this task, addressing key challenges: Chain-of-Thought (CoT) error accumulation, discriminative hallucination, and deployment feasibility. Our framework, TaoSR1, involves three stages: (1) Supervised Fine-Tuning (SFT) with CoT to instill reasoning; (2) Offline sampling with a pass@N strategy and Direct Preference Optimization (DPO) to improve generation quality; and (3) Difficulty-based dynamic sampling with Group Relative Policy Optimization (GRPO) to mitigate discriminative hallucination. Additionally, post-CoT processing and a cumulative probability-based partitioning method enable efficient online deployment. TaoSR1 significantly outperforms baselines on offline datasets and achieves substantial gains in online side-by-side human evaluations, introducing a novel paradigm for applying CoT reasoning to relevance classification.
♻ ☆ Data Mixing Can Induce Phase Transitions in Knowledge Acquisition NeurIPS'25
Xinran Gu, Kaifeng Lyu, Jiazheng Li, Jingzhao Zhang
Large Language Models (LLMs) are typically trained on data mixtures: most data come from web scrapes, while a small portion is curated from high-quality sources with dense domain-specific knowledge. In this paper, we show that when training LLMs on such data mixtures, knowledge acquisition from knowledge-dense datasets, unlike training exclusively on knowledge-dense data (arXiv:2404.05405), does not always follow a smooth scaling law but can exhibit phase transitions with respect to the mixing ratio and model size. Through controlled experiments on a synthetic biography dataset mixed with web-scraped data, we demonstrate that: (1) as we increase the model size to a critical value, the model suddenly transitions from memorizing very few to most of the biographies; (2) below a critical mixing ratio, the model memorizes almost nothing even with extensive training, but beyond this threshold, it rapidly memorizes more biographies. We attribute these phase transitions to a capacity allocation phenomenon: a model with bounded capacity must act like a knapsack problem solver to minimize the overall test loss, and the optimal allocation across datasets can change discontinuously as the model size or mixing ratio varies. We formalize this intuition in an information-theoretic framework and reveal that these phase transitions are predictable, with the critical mixing ratio following a power-law relationship with the model size. Our findings highlight a concrete case where a good mixing recipe for large models may not be optimal for small models, and vice versa.
comment : NeurIPS'25 Spotlight
♻ ☆ Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs
Alberto Cattaneo, Carlo Luschi, Daniel Justus
Retrieval of information from graph-structured knowledge bases represents a promising direction for improving the factuality of LLMs. While various solutions have been proposed, a comparison of methods is difficult due to the lack of challenging QA datasets with ground-truth targets for graph retrieval. We present SynthKGQA, an LLM-powered framework for generating high-quality Knowledge Graph Question Answering datasets from any Knowledge Graph, providing the full set of ground-truth facts in the KG to reason over questions. We show how, in addition to enabling more informative benchmarking of KG retrievers, the data produced with SynthKGQA also allows us to train better models.We apply SynthKGQA to Wikidata to generate GTSQA, a new dataset designed to test zero-shot generalization abilities of KG retrievers with respect to unseen graph structures and relation types, and benchmark popular solutions for KG-augmented LLMs on it.
♻ ☆ An Investigation of Robustness of LLMs in Mathematical Reasoning : Benchmark ing with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
Yuren Hao, Xiang Wan, ChengXiang Zhai
In this paper, we introduce a systematic framework beyond conventional method to assess LLMs' mathematical-reasoning robustness by stress-testing them on advanced math problems that are mathematically equivalent but with linguistic and parametric variation. These transformations allow us to measure the sensitivity of LLMs to non-mathematical perturbations, thereby enabling a more accurate evaluation of their mathematical reasoning capabilities. Using this new evaluation methodology, we created PutnamGAP, a new benchmark dataset with multiple mathematically-equivalent variations of competition-level math problems. With the new dataset, we evaluate multiple families of representative LLMs and examine their robustness. Across 18 commercial and open-source models we observe sharp performance degradation on the variants. OpenAI's flagship reasoning model, O3, scores 51.5% on the originals but drops by 4.7 percentage points on surface-renaming variants, and by 12.9 percentage points on parametric variants, while smaller models fare far worse. Overall, the results show that the proposed new evaluation methodology is effective for deepening our understanding of the robustness of LLMs and generating new insights for further improving their mathematical reasoning capabilities.
comment : 34 pages, 9 figures
♻ ☆ Scaling Towards the Information Boundary of Instruction Sets: The Infinity Instruct Subject Technical Report
Li Du, Hanyu Zhao, Yiming Ju, Tengfei Pan
Instruction tuning has become a foundation for unlocking the capabilities of large-scale pretrained models and improving their performance on complex tasks. Thus, the construction of high-quality instruction datasets is crucial for enhancing model performance and generalizability. Although current instruction datasets have reached tens of millions of samples, models finetuned on them may still struggle with complex instruction following and tasks in rare domains. This is primarily due to limited expansion in both ``coverage'' (coverage of task types and knowledge areas) and ``depth'' (instruction complexity) of the instruction set. To address this issue, we propose a systematic instruction data construction framework, which integrates a hierarchical tagging system, an informative seed selection algorithm, an evolutionary data synthesis process, and a model deficiency diagnosis with targeted data generation. These components form an iterative closed-loop to continuously enhance the coverage and depth of instruction data. Based on this framework, we construct Infinity Instruct Subject, a high-quality dataset containing $\sim$1.5 million instructions. Experiments on multiple foundation models and benchmark tasks demonstrate its effectiveness in improving instruction-following capabilities. Further analyses suggest that Infinity Instruct Subject shows enlarged coverage and depth compared to comparable synthesized instruction datasets. Our work lays a theoretical and practical foundation for the efficient, continuous evolution of instruction datasets, moving from data quantity expansion to qualitative improvement.
♻ ☆ In- Context Representation Hijacking
Itay Yona, Amir Sarid, Michael Karasik, Yossi Gandelsman
We introduce $\textbf{Doublespeak}$, a simple in-context representation hijacking attack against large language models (LLMs). The attack works by systematically replacing a harmful keyword (e.g., bomb) with a benign token (e.g., carrot) across multiple in-context examples, provided a prefix to a harmful request. We demonstrate that this substitution leads to the internal representation of the benign token converging toward that of the harmful one, effectively embedding the harmful semantics under a euphemism. As a result, superficially innocuous prompts (e.g., "How to build a carrot?") are internally interpreted as disallowed instructions (e.g., "How to build a bomb?"), thereby bypassing the model's safety alignment. We use interpretability tools to show that this semantic overwrite emerges layer by layer, with benign meanings in early layers converging into harmful semantics in later ones. Doublespeak is optimization-free, broadly transferable across model families, and achieves strong success rates on closed-source and open-source systems, reaching 74% ASR on Llama-3.3-70B-Instruct with a single-sentence context override. Our findings highlight a new attack surface in the latent space of LLMs, revealing that current alignment strategies are insufficient and should instead operate at the representation level.
♻ ☆ Semi-Supervised Synthetic Data Generation with Fine-Grained Relevance Control for Short Video Search Relevance Modeling AAAI 2026
Haoran Li, Zhiming Su, Junyan Yao, Enwei Zhang, Yang Ji, Yan Chen, Kan Zhou, Chao Feng, Jiao Ran
Synthetic data is widely adopted in embedding models to ensure diversity in training data distributions across dimensions such as difficulty, length, and language. However, existing prompt-based synthesis methods struggle to capture domain-specific data distributions, particularly in data-scarce domains, and often overlook fine-grained relevance diversity. In this paper, we present a Chinese short video dataset with 4-level relevance annotations, filling a critical resource void. Further, we propose a semi-supervised synthetic data pipeline where two collaboratively trained models generate domain-adaptive short video data with controllable relevance labels. Our method enhances relevance-level diversity by synthesizing samples for underrepresented intermediate relevance labels, resulting in a more balanced and semantically rich training data set. Extensive offline experiments show that the embedding model trained on our synthesized data outperforms those using data generated based on prompting or vanilla supervised fine-tuning(SFT). Moreover, we demonstrate that incorporating more diverse fine-grained relevance levels in training data enhances the model's sensitivity to subtle semantic distinctions, highlighting the value of fine-grained relevance supervision in embedding learning. In the search enhanced recommendation pipeline of Douyin's dual-column scenario, through online A/B testing, the proposed model increased click-through rate(CTR) by 1.45%, raised the proportion of Strong Relevance Ratio (SRR) by 4.9%, and improved the Image User Penetration Rate (IUPR) by 0.1054%.
comment : Submitted to AAAI 2026
♻ ☆ Evaluating Autoformalization Robustness via Semantically Similar Paraphrasing
Hayden Moore, Asfahan Shah
Large Language Models (LLMs) have recently emerged as powerful tools for autoformalization. Despite their impressive performance, these models can still struggle to produce grounded and verifiable formalizations. Recent work in text-to-SQL, has revealed that LLMs can be sensitive to paraphrased natural language (NL) inputs, even when high degrees of semantic fidelity are preserved (Safarzadeh, Oroojlooyjadid, and Roth 2025). In this paper, we investigate this claim in the autoformalization domain. Specifically, we evaluate the robustness of LLMs generating formal proofs with semantically similar paraphrased NL statements by measuring semantic and compilation validity. Using the formal benchmarks MiniF2F (Zheng, Han, and Polu 2021) and Lean 4 version of ProofNet (Xin et al. 2024), and two modern LLMs, we generate paraphrased natural language statements and cross-evaluate these statements across both models. The results of this paper reveal performance variability across paraphrased inputs, demonstrating that minor shifts in NL statements can significantly impact model outputs.
♻ ☆ Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmark s and Quantifying Evaluation Awareness
Lang Xiong, Nishant Bhargava, Jianhang Hong, Jeremy Chang, Haihao Liu, Vasu Sharma, Kevin Zhu
Large Language Models (LLMs) often exhibit significant behavioral shifts when they perceive a change from a real-world deployment context to a controlled evaluation setting, a phenomenon known as "evaluation awareness." This discrepancy poses a critical challenge for AI alignment, as benchmark performance may not accurately reflect a model's true safety and honesty. In this work, we systematically quantify these behavioral changes by manipulating the perceived context of prompts. We introduce a methodology that uses a linear probe to score prompts on a continuous scale from "test-like" to "deploy-like" and leverage an LLM rewriting strategy to shift these prompts towards a more natural, deployment-style context while preserving the original task. Using this method, we achieved a 30% increase in the average probe score across a strategic role-playing dataset after rewriting. Evaluating a suite of state-of-the-art models on these original and rewritten prompts, we find that rewritten "deploy-like" prompts induce a significant and consistent shift in behavior. Across all models, we observed an average increase in honest responses of 5.26% and a corresponding average decrease in deceptive responses of 12.40%. Furthermore, refusal rates increased by an average of 6.38%, indicating heightened safety compliance. Our findings demonstrate that evaluation awareness is a quantifiable and manipulable factor that directly influences LLM behavior, revealing that models are more prone to unsafe or deceptive outputs in perceived test environments. This underscores the urgent need for more realistic evaluation frameworks to accurately gauge true model alignment before deployment.
♻ ☆ Route-and-Reason: Scaling Large Language Model Reasoning with Reinforced Model Router
Chenyang Shao, Xinyang Liu, Yutang Lin, Fengli Xu, Yong Li
Chain-of-thought has been proven essential for enhancing the complex reasoning abilities of Large Language Models (LLMs), but it also leads to high computational costs. Recent advances have explored the method to route queries among multiple models and proved it as a promising approach. However, previous works directly operate at the task level, i.e., assigning user queries to suitable LLMs, which does not allow hybrid LLMs to truly collaborate on finer-grained sub-tasks. Collaboration at the level of intermediate reasoning steps (thoughts) could enable more efficient coordination, but it also poses significant challenges for router scheduling, placing immense demands on the quality of task decomposition and the precision of the router. To address this, we propose R2-Reasoner, a novel framework centered around a Reinforced Model Router designed to efficiently scale LLM reasoning. This router orchestrates collaboration across nine heterogeneous models, whose parameter scales range from less than 1B to hundreds of billions, by first breaking down a complex query into subtasks with a decomposer, and then assigning each subtask to the optimal model with a subtask allocator, balancing performance with cost. Training this router involves a two-stage alternating process for the decomposer and the allocator, integrating supervised fine-tuning with reinforcement learning to enable effective self-supervised refinement. Extensive experiments across six challenging reasoning benchmarks demonstrate that R2-Reasoner reduces API costs by 84.46% compared with state-of-the-art baselines while maintaining competitive reasoning accuracy. Our framework paves the way for the development of more scalable and efficient reasoning systems. Our code is open-source at https://anonymous.4open.science/r/R2_Reasoner.
♻ ☆ Nexus: Higher-Order Attention Mechanisms in Transformer s
Hanting Chen, Chong Zhu, Kai Han, Yuchuan Tian, Yuchen Liang, Tianyu Guo, Xinghao Chen, Dacheng Tao, Yunhe Wang
Transformers have achieved significant success across various domains, relying on self-attention to capture dependencies. However, the standard first-order attention mechanism is often limited by a low-rank bottleneck, struggling to capture intricate, multi-hop relationships within a single layer. In this paper, we propose the Nexus, a novel architecture designed to enhance representational power through a recursive framework. Unlike standard approaches that use static linear projections for Queries and Keys, Nexus dynamically refines these representations via nested self-attention mechanisms. Specifically, the Query and Key vectors are themselves outputs of inner attention loops, allowing tokens to aggregate global context and model high-order correlations \textit{prior} to the final attention computation. We enforce a parameter-efficient weight-sharing strategy across recursive steps, ensuring that this enhanced expressivity incurs $\mathcal{O}(1)$ additional parameters. We provide theoretical analysis demonstrating that our method breaks the linear bottleneck of standard attention. Empirically, Nexus outperforms standard Transformers on multiple benchmarks.
♻ ☆ FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusion
Anke Tang, Li Shen, Yong Luo, Enneng Yang, Han Hu, Lefei Zhang, Bo Du, Dacheng Tao
Deep model fusion is an emerging technique that unifies the predictions or parameters of several deep neural networks into a single better-performing model in a cost-effective and data-efficient manner. Although a variety of deep model fusion techniques have been introduced, their evaluations tend to be inconsistent and often inadequate to validate their effectiveness and robustness. We present FusionBench, the first benchmark and a unified library designed specifically for deep model fusion. Our benchmark consists of multiple tasks, each with different settings of models and datasets. This variety allows us to compare fusion methods across different scenarios and model scales. Additionally, FusionBench serves as a unified library for easy implementation and testing of new fusion techniques. FusionBench is open source and actively maintained, with community contributions encouraged. Homepage https://github.com/tanganke/fusion_bench
comment : Project homepage: https://github.com/tanganke/fusion_bench Online documentation: https://tanganke.github.io/fusion_bench
♻ ☆ Reversing Large Language Models for Efficient Training and Fine-Tuning
Eshed Gal, Moshe Eliasof, Javier Turek, Uri Ascher, Eran Treister, Eldad Haber
Large Language Models (LLMs) are known for their expensive and time-consuming training. Thus, oftentimes, LLMs are fine-tuned to address a specific task, given the pretrained weights of a pre-trained LLM considered a foundation model. In this work, we introduce memory-efficient, reversible architectures for LLMs, inspired by symmetric and symplectic differential equations, and investigate their theoretical properties. Different from standard, baseline architectures that store all intermediate activations, the proposed models use time-reversible dynamics to retrieve hidden states during backpropagation, relieving the need to store activations. This property allows for a drastic reduction in memory consumption, allowing for the processing of larger batch sizes for the same available memory, thereby offering improved throughput. In addition, we propose an efficient method for converting existing, non-reversible LLMs into reversible architectures through fine-tuning, rendering our approach practical for exploiting existing pre-trained models. Our results show comparable or improved performance on several datasets and benchmarks, on several LLMs, building a scalable and efficient path towards reducing the memory and computational costs associated with both training from scratch and fine-tuning of LLMs.
♻ ☆ Dual-branch Prompt ing for Multimodal Machine Translation
Jie Wang, Zhendong Yang, Liansong Zong, Xiaobo Zhang, Dexian Wang, Ji Zhang
Multimodal Machine Translation (MMT) typically enhances text-only translation by incorporating aligned visual features. Despite the remarkable progress, state-of-the-art MMT approaches often rely on paired image-text inputs at inference and are sensitive to irrelevant visual noise, which limits their robustness and practical applicability. To address these issues, we propose D2P-MMT, a diffusion-based dual-branch prompting framework for robust vision-guided translation. Specifically, D2P-MMT requires only the source text and a reconstructed image generated by a pre-trained diffusion model, which naturally filters out distracting visual details while preserving semantic cues. During training, the model jointly learns from both authentic and reconstructed images using a dual-branch prompting strategy, encouraging rich cross-modal interactions. To bridge the modality gap and mitigate training-inference discrepancies, we introduce a distributional alignment loss that enforces consistency between the output distributions of the two branches. Extensive experiments on the Multi30K dataset demonstrate that D2P-MMT achieves superior translation performance compared to existing state-of-the-art approaches.
comment : This manuscript is currently under review at the ACM Transactions on Multimedia Computing, Communications, and Applications
Artificial Intelligence 208
☆ SkillFactory: Self-Distillation For Learning Cognitive Behaviors
Zayne Sprague, Jack Lu, Manya Wadhwa, Sedrick Keh, Mengye Ren, Greg Durrett
Reasoning models leveraging long chains of thought employ various cognitive skills, such as verification of their answers, backtracking, retrying by an alternate method, and more. Previous work has shown that when a base language model exhibits these skills, training that model further with reinforcement learning (RL) can learn to leverage them. How can we get models to leverage skills that aren't exhibited by base models? Our work, SkillFactory, is a method for fine-tuning models to roughly learn these skills during a supervised fine-tuning (SFT) stage prior to RL. Our approach does not rely on distillation from a stronger model, but instead uses samples from the model itself, rearranged to provide training data in the format of those skills. These "silver" SFT traces may be imperfect, but are nevertheless effective for priming a model to acquire skills during RL. Our evaluation shows that (1) starting from SkillFactory SFT initialization helps a model to generalize to harder variants of a task post-RL, despite lower performance pre-RL; (2) cognitive skills are indeed used by the model; (3) RLed SkillFactory models are more robust to regression on out-of-domain tasks than RLed base models. Our work suggests that inductive biases learned prior to RL help models learn robust cognitive skill use.
☆ Fare Comparison App of Uber, Ola and Rapido
Ashlesha Gopinath Sawant, Sahil S. Jadhav, Vidhan R. Jain, Shriraj S. Jagtap, Prachi Jadhav, Soham Jadhav, Ichha Raina
In todays increasing world, it is very important to have good hailing services like Ola, Uber, and Rapido as it is very essential for our daily transportation. Users often face difficulties in choosing the most appropriate and efficient ride that would lead to both cost-effective and would take us to our destination in less time. This project provides you with the web application that helps you to select the most beneficial ride for you by providing users with the fare comparison between Ola, Uber, Rapido for the destination entered by the user. The backend is use to fetch the data, providing users with the fare comparison for the ride and finally providing with the best option using Python. This research paper also addresses the problem and challenges faced in accessing the data using APIs, Android Studios emulator, Appium and location comparison. Thus, the aim of the project is to provide transparency to the users in ride-hailing services and increase efficiency and provide users with better experience.
comment : 4 pages
☆ Polarization by Design: How Elites Could Shape Mass Preferences as AI Reduces Persuasion Costs
Nadav Kunievsky
In democracies, major policy decisions typically require some form of majority or consensus, so elites must secure mass support to govern. Historically, elites could shape support only through limited instruments like schooling and mass media; advances in AI-driven persuasion sharply reduce the cost and increase the precision of shaping public opinion, making the distribution of preferences itself an object of deliberate design. We develop a dynamic model in which elites choose how much to reshape the distribution of policy preferences, subject to persuasion costs and a majority rule constraint. With a single elite, any optimal intervention tends to push society toward more polarized opinion profiles - a ``polarization pull'' - and improvements in persuasion technology accelerate this drift. When two opposed elites alternate in power, the same technology also creates incentives to park society in ``semi-lock'' regions where opinions are more cohesive and harder for a rival to overturn, so advances in persuasion can either heighten or dampen polarization depending on the environment. Taken together, cheaper persuasion technologies recast polarization as a strategic instrument of governance rather than a purely emergent social byproduct, with important implications for democratic stability as AI capabilities advance.
☆ MarkTune: Improving the Quality-Detectability Trade-off in Open-Weight LLM Watermarking
Yizhou Zhao, Zhiwei Steven Wu, Adam Block
Watermarking aims to embed hidden signals in generated text that can be reliably detected when given access to a secret key. Open-weight language models pose acute challenges for such watermarking schemes because the inference-time interventions that dominate contemporary approaches cannot be enforced once model weights are public. Existing watermaking techniques for open-weight models, such as the recently proposed GaussMark, typically rely on small modifications to model weights, which can yield signals detectable to those equipped with a secret key, but achieving detection power comparable to inference-time watermarks generally requires weight perturbations that noticeably reduce generation quality. We introduce MarkTune, a theoretically principled, on-policy fine-tuning framework that treats the GaussMark signal as a reward while simultaneously regularizing against degradation in text quality. We derive MarkTune as an improvement on GaussMark and demonstrate that MarkTune consistently improves the quality-detectability trade-off over GaussMark by steering finer-grained, watermark-aware weight updates within the model's representation space while preserving generation quality. Empirically, we show that MarkTune pushes the quality-detectability frontier of GaussMark close to that of inference-time watermarking, remains robust to paraphrasing and fine-tuning attacks, and exhibits strong generalization: a model fine-tuned on one dataset retains substantial watermark detection power on unseen datasets. Together, these results establish MarkTune as a general strategy for embedding robust, high-quality watermarks into open-weight LMs.
☆ Fast & Efficient Normalizing Flows and Applications of Image Generative Models
Sandeep Nagar
This thesis presents novel contributions in two primary areas: advancing the efficiency of generative models, particularly normalizing flows, and applying generative models to solve real-world computer vision challenges. The first part introduce significant improvements to normalizing flow architectures through six key innovations: 1) Development of invertible 3x3 Convolution layers with mathematically proven necessary and sufficient conditions for invertibility, (2) introduction of a more efficient Quad-coupling layer, 3) Design of a fast and efficient parallel inversion algorithm for kxk convolutional layers, 4) Fast & efficient backpropagation algorithm for inverse of convolution, 5) Using inverse of convolution, in Inverse-Flow, for the forward pass and training it using proposed backpropagation algorithm, and 6) Affine-StableSR, a compact and efficient super-resolution model that leverages pre-trained weights and Normalizing Flow layers to reduce parameter count while maintaining performance. The second part: 1) An automated quality assessment system for agricultural produce using Conditional GANs to address class imbalance, data scarcity and annotation challenges, achieving good accuracy in seed purity testing; 2) An unsupervised geological mapping framework utilizing stacked autoencoders for dimensionality reduction, showing improved feature extraction compared to conventional methods; 3) We proposed a privacy preserving method for autonomous driving datasets using on face detection and image inpainting; 4) Utilizing Stable Diffusion based image inpainting for replacing the detected face and license plate to advancing privacy-preserving techniques and ethical considerations in the field.; and 5) An adapted diffusion model for art restoration that effectively handles multiple types of degradation through unified fine-tuning.
comment : PhD Thesis
☆ Jina-VLM: Small Multilingual Vision Language Model
Andreas Koukounas, Georgios Mastrapas, Florian Hönicke, Sedigheh Eslami, Guillaume Roncari, Scott Martens, Han Xiao
We present Jina-VLM, a 2.4B parameter vision-language model that achieves state-of-the-art multilingual visual question answering among open 2B-scale VLMs. The model couples a SigLIP2 vision encoder with a Qwen3 language backbone through an attention-pooling connector that enables token-efficient processing of arbitrary-resolution images. Across standard VQA benchmarks and multilingual evaluations, Jina-VLM outperforms comparable models while preserving competitive text-only performance.
comment : 18 pages, 1-7 main content
☆ Large Language Models for Limited Noisy Data: A Gravitational Wave Identification Study
Yixuan Li, Yuhao Lu, Yang Liu, Liang Li, R. Ruffini, Di Li, Rong-Gen Cai, Xiaoyan Zhu, Wenbin Lin, Yu Wang
This work investigates whether large language models (LLMs) offer advantages over traditional neural networks for astronomical data processing, in regimes with non-Gaussian, non-stationary noise and limited labeled samples. Gravitational wave observations provide an suitable test case, using only 90 LIGO events, finetuned LLMs achieve 97.4\% accuracy for identifying signals. Further experiments show that, in contrast to traditional networks that rely on large simulated datasets, additional simulated samples do not improve LLM performance, while scaling studies reveal predictable gains with increasing model size and dataset size. These results indicate that LLMs can extract discriminative structure directly from observational data and provide an efficient assessment for gravitational wave identification. The same strategy may extend to other astronomical domains with similar noise properties, such as radio or pulsar observations.
comment : 10 pages, 5 figures
☆ PSA: Pyramid Sparse Attention for Efficient Video Understanding and Generation
Xiaolong Li, Youping Gu, Xi Lin, Weijie Wang, Bohan Zhuang
Attention mechanisms are the core of foundation models, but their quadratic complexity remains a critical bottleneck for scaling. This challenge has driven the development of efficient attention mechanisms, with sparsity emerging as the dominant paradigm. Current methods typically retain or discard entire key-value blocks with binary masks, resulting in substantial information loss under high sparsity. To mitigate this gap, we present Pyramid Sparse Attention (PSA), a versatile module applicable to both video understanding and generation tasks. Instead of binary masking, PSA introduces multi-level pooled KV representations, enabling finer mask granularity. Specifically, each query block dynamically allocates lower pooling levels to critical KV blocks and higher levels to less important ones, creating an informative interpolation between full retention and complete pruning. This design, analogous to fixed-point quantization and classical feature pyramid networks in computer vision, effectively mitigates information loss while preserving computational efficiency under a low compute budget. It works with a native, hardware-friendly kernel that leverages decoupled block-tile design to ensure efficient execution. Across video understanding and generation benchmarks, PSA preserves contextual information and visual fidelity, consistently outperforming or achieving comparable performance over existing sparse attention baselines with superior efficiency-quality trade-offs. Our code and model weights are publicly available at: http://ziplab.co/PSA
comment : Tech report
☆ TARA Test-by-Adaptive-Ranks for Quantum Anomaly Detection with Conformal Prediction Guarantees
Davut Emre Tasar, Ceren Ocal Tasar
Quantum key distribution (QKD) security fundamentally relies on the ability to distinguish genuine quantum correlations from classical eavesdropper simulations, yet existing certification methods lack rigorous statistical guarantees under finite-sample conditions and adversarial scenarios. We introduce TARA (Test by Adaptive Ranks), a novel framework combining conformal prediction with sequential martingale testing for quantum anomaly detection that provides distribution-free validity guarantees. TARA offers two complementary approaches. TARA k, based on Kolmogorov Smirnov calibration against local hidden variable (LHV) null distributions, achieving ROC AUC = 0.96 for quantum-classical discrimination. And TARA-m, employing betting martingales for streaming detection with anytime valid type I error control that enables real time monitoring of quantum channels. We establish theoretical guarantees proving that under (context conditional) exchangeability, conformal p-values remain uniformly distributed even for strongly contextual quantum data, confirming that quantum contextuality does not break conformal prediction validity a result with implications beyond quantum certification to any application of distribution-free methods to nonclassical data. Extensive validation on both IBM Torino (superconducting, CHSH = 2.725) and IonQ Forte Enterprise (trapped ion, CHSH = 2.716) quantum processors demonstrates cross-platform robustness, achieving 36% security margins above the classical CHSH bound of 2. Critically, our framework reveals a methodological concern affecting quantum certification more broadly: same-distribution calibration can inflate detection performance by up to 44 percentage points compared to proper cross-distribution calibration, suggesting that prior quantum certification studies using standard train test splits may have systematically overestimated adversarial robustness.
☆ On the Temporality for Sketch Representation Learning
Marcelo Isaias de Moraes Junior, Moacir Antonelli Ponti
Sketches are simple human hand-drawn abstractions of complex scenes and real-world objects. Although the field of sketch representation learning has advanced significantly, there is still a gap in understanding the true relevance of the temporal aspect to the quality of these representations. This work investigates whether it is indeed justifiable to treat sketches as sequences, as well as which internal orders play a more relevant role. The results indicate that, although the use of traditional positional encodings is valid for modeling sketches as sequences, absolute coordinates consistently outperform relative ones. Furthermore, non-autoregressive decoders outperform their autoregressive counterparts. Finally, the importance of temporality was shown to depend on both the order considered and the task evaluated.
☆ Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding
Jialuo Li, Bin Li, Jiahao Li, Yan Lu
The application of Large Multimodal Models (LMMs) to long-form video understanding is constrained by limited context lengths and the computationally prohibitive cost of processing dense video tokens. Consequently, recent research has focused on query-aware frame selection, methods that often incur significant computational overhead. This paper challenges the assumption that such complex search mechanisms are universally necessary. We first identify and validate a query typology distinguishing between global query and localized query. We demonstrate that while uniform sampling is both effective and efficient for global queries, localized queries indeed necessitate query-aware selection for optimal performance. Building on this insight, we propose DIG, a training-free frame selection framework that adapts its strategy based on the query type. Specifically,DIG employs efficient uniform sampling for global queries while activating a specialized pipeline to extract query-relevant frames for localized queries. Experiments on three long-form video understanding benchmarks demonstrate that DIG consistently outperforms existing baselines and robustly improves LMM performance, even when scaling the input frame count to 256.
☆ Highly Efficient Test-Time Scaling for T2I Diffusion Models with Text Embedding Perturbation
Hang Xu, Linjiang Huang, Feng Zhao
Test-time scaling (TTS) aims to achieve better results by increasing random sampling and evaluating samples based on rules and metrics. However, in text-to-image(T2I) diffusion models, most related works focus on search strategies and reward models, yet the impact of the stochastic characteristic of noise in T2I diffusion models on the method's performance remains unexplored. In this work, we analyze the effects of randomness in T2I diffusion models and explore a new format of randomness for TTS: text embedding perturbation, which couples with existing randomness like SDE-injected noise to enhance generative diversity and quality. We start with a frequency-domain analysis of these formats of randomness and their impact on generation, and find that these two randomness exhibit complementary behavior in the frequency domain: spatial noise favors low-frequency components (early steps), while text embedding perturbation enhances high-frequency details (later steps), thereby compensating for the potential limitations of spatial noise randomness in high-frequency manipulation. Concurrently, text embedding demonstrates varying levels of tolerance to perturbation across different dimensions of the generation process. Specifically, our method consists of two key designs: (1) Introducing step-based text embedding perturbation, combining frequency-guided noise schedules with spatial noise perturbation. (2) Adapting the perturbation intensity selectively based on their frequency-specific contributions to generation and tolerance to perturbation. Our approach can be seamlessly integrated into existing TTS methods and demonstrates significant improvements on multiple benchmarks with almost no additional computation. Code is available at \href{https://github.com/xuhang07/TEP-Diffusion}{https://github.com/xuhang07/TEP-Diffusion}.
☆ DIQ-H: Evaluating Hallucination Persistence in VLMs Under Temporal Visual Degradation
Zexin Lin, Hawen Wan, Yebin Zhong, Xiaoqiang
Vision-Language Models (VLMs) deployed in safety-critical applications such as autonomous driving must handle continuous visual streams under imperfect conditions. However, existing benchmarks focus on static, high-quality images and ignore temporal degradation and error propagation, which are critical failure modes where transient visual corruption induces hallucinations that persist across subsequent frames. We introduce DIQ-H, the first benchmark for evaluating VLM robustness under dynamic visual degradation in temporal sequences. DIQ-H applies physics-based corruptions including motion blur, sensor noise, and compression artifacts, and measures hallucination persistence, error recovery, and temporal consistency through multi-turn question-answering tasks. To enable scalable annotation, we propose Uncertainty-Guided Iterative Refinement (UIR), which generates reliable pseudo-ground-truth using lightweight VLMs with uncertainty filtering, achieving a 15.3 percent accuracy improvement. Experiments on 16 state-of-the-art VLMs reveal substantial robustness gaps: even advanced models such as GPT-4o achieve only a 78.5 percent recovery rate, while open-source models struggle with temporal consistency at less than 60 percent. DIQ-H provides a comprehensive platform for evaluating VLM reliability in real-world deployments.
☆ BlurDM: A Blur Diffusion Model for Image Deblurring NeurIPS 2025
Jin-Ting He, Fu-Jen Tsai, Yan-Tsung Peng, Min-Hung Chen, Chia-Wen Lin, Yen-Yu Lin
Diffusion models show promise for dynamic scene deblurring; however, existing studies often fail to leverage the intrinsic nature of the blurring process within diffusion models, limiting their full potential. To address it, we present a Blur Diffusion Model (BlurDM), which seamlessly integrates the blur formation process into diffusion for image deblurring. Observing that motion blur stems from continuous exposure, BlurDM implicitly models the blur formation process through a dual-diffusion forward scheme, diffusing both noise and blur onto a sharp image. During the reverse generation process, we derive a dual denoising and deblurring formulation, enabling BlurDM to recover the sharp image by simultaneously denoising and deblurring, given pure Gaussian noise conditioned on the blurred image as input. Additionally, to efficiently integrate BlurDM into deblurring networks, we perform BlurDM in the latent space, forming a flexible prior generation network for deblurring. Extensive experiments demonstrate that BlurDM significantly and consistently enhances existing deblurring methods on four benchmark datasets. The source code is available at https://github.com/Jin-Ting-He/BlurDM.
comment : NeurIPS 2025
☆ Sponsored Questions and How to Auction Them
Kshipra Bhawalkar, Alexandros Psomas, Di Wang
Online platforms connect users with relevant products and services using ads. A key challenge is that a user's search query often leaves their true intent ambiguous. Typically, platforms passively predict relevance based on available signals and in some cases offer query refinements. The shift from traditional search to conversational AI provides a new approach. When a user's query is ambiguous, a Large Language Model (LLM) can proactively offer several clarifying follow-up prompts. In this paper we consider the following: what if some of these follow-up prompts can be ``sponsored,'' i.e., selected for their advertising potential. How should these ``suggestion slots'' be allocated? And, how does this new mechanism interact with the traditional ad auction that might follow? This paper introduces a formal model for designing and analyzing these interactive platforms. We use this model to investigate a critical engineering choice: whether it is better to build an end-to-end pipeline that jointly optimizes the user interaction and the final ad auction, or to decouple them into separate mechanisms for the suggestion slots and another for the subsequent ad slot. We show that the VCG mechanism can be adopted to jointly optimize the sponsored suggestion and the ads that follow; while this mechanism is more complex, it achieves outcomes that are efficient and truthful. On the other hand, we prove that the simple-to-implement modular approach suffers from strategic inefficiency: its Price of Anarchy is unbounded.
☆ Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning
Franki Nguimatsia Tiofack, Théotime Le Hellard, Fabian Schramm, Nicolas Perrin-Gilbert, Justin Carpentier
Offline reinforcement learning often relies on behavior regularization that enforces policies to remain close to the dataset distribution. However, such approaches fail to distinguish between high-value and low-value actions in their regularization components. We introduce Guided Flow Policy (GFP), which couples a multi-step flow-matching policy with a distilled one-step actor. The actor directs the flow policy through weighted behavior cloning to focus on cloning high-value actions from the dataset rather than indiscriminately imitating all state-action pairs. In turn, the flow policy constrains the actor to remain aligned with the dataset's best transitions while maximizing the critic. This mutual guidance enables GFP to achieve state-of-the-art performance across 144 state and pixel-based tasks from the OGBench, Minari, and D4RL benchmarks, with substantial gains on suboptimal datasets and challenging tasks. Webpage: https://simple-robotics.github.io/publications/guided-flow-policy/
☆ Benchmark for Planning and Control with Large Language Model Agent s: Blockswo rl d with Model Context Protocol
Niklas Jobs, Luis Miguel Vieira da Silva, Jayanth Somashekaraiah, Maximilian Weigand, David Kube, Felix Gehlhoff
Industrial automation increasingly requires flexible control strategies that can adapt to changing tasks and environments. Agents based on Large Language Models (LLMs) offer potential for such adaptive planning and execution but lack standardized benchmarks for systematic comparison. We introduce a benchmark with an executable simulation environment representing the Blocksworld problem providing five complexity categories. By integrating the Model Context Protocol (MCP) as a standardized tool interface, diverse agent architectures can be connected to and evaluated against the benchmark without implementation-specific modifications. A single-agent implementation demonstrates the benchmark's applicability, establishing quantitative metrics for comparison of LLM-based planning and execution approaches.
comment : This work has been submitted to IFAC for possible publication
☆ Autonomous Agent s and Policy Compliance: A Framework for Reasoning About Penalties
Vineel Tummala, Daniela Inclezan
This paper presents a logic programming-based framework for policy-aware autonomous agents that can reason about potential penalties for non-compliance and act accordingly. While prior work has primarily focused on ensuring compliance, our approach considers scenarios where deviating from policies may be necessary to achieve high-stakes goals. Additionally, modeling non-compliant behavior can assist policymakers by simulating realistic human decision-making. Our framework extends Gelfond and Lobo's Authorization and Obligation Policy Language (AOPL) to incorporate penalties and integrates Answer Set Programming (ASP) for reasoning. Compared to previous approaches, our method ensures well-formed policies, accounts for policy priorities, and enhances explainability by explicitly identifying rule violations and their consequences. Building on the work of Harders and Inclezan, we introduce penalty-based reasoning to distinguish between non-compliant plans, prioritizing those with minimal repercussions. To support this, we develop an automated translation from the extended AOPL into ASP and refine ASP-based planning algorithms to account for incurred penalties. Experiments in two domains demonstrate that our framework generates higher-quality plans that avoid harmful actions while, in some cases, also improving computational efficiency. These findings underscore its potential for enhancing autonomous decision-making and informing policy refinement. Under consideration in Theory and Practice of Logic Programming (TPLP).
comment : 27 pages, 5 figures
☆ A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models
X. Y. Han, Yuan Zhong
In large-scale AI training, Sparse Mixture-of-Experts (s-MoE) layers enable scaling by activating only a small subset of experts per token. An operational challenge in this design is load balancing: routing tokens to minimize the number of idle experts, which is important for the efficient utilization of (costly) GPUs. We provide a theoretical framework for analyzing the Auxiliary-Loss-Free Load Balancing (ALF-LB) procedure -- proposed by DeepSeek's Wang et al. (2024) -- by casting it as a one-step-per-iteration primal-dual method for an assignment problem. First, in a stylized deterministic setting, our framework yields several insightful structural properties: (i) a monotonic improvement of a Lagrangian objective, (ii) a preference rule that moves tokens from overloaded to underloaded experts, and (iii) an approximate-balancing guarantee. Then, we incorporate the stochastic and dynamic nature of AI training using a generalized online optimization formulation. In the online setting, we derive a strong convexity property of the objective that leads to a logarithmic expected regret bound under certain step-size choices. Additionally, we present real experiments on 1B-parameter DeepSeekMoE models to complement our theoretical findings. Together, these results build a principled framework for analyzing the Auxiliary-Loss-Free Load Balancing of s-MoE in AI models.
☆ Hierarchical Vision Language Action Model Using Success and Failure Demonstrations
Jeongeun Park, Jihwan Yoon, Byungwoo Jeon, Juhan Park, Jinwoo Shin, Namhoon Cho, Kyungjae Lee, Sangdoo Yun, Sungjoon Choi
Prior Vision-Language-Action (VLA) models are typically trained on teleoperated successful demonstrations, while discarding numerous failed attempts that occur naturally during data collection. However, these failures encode where and how policies can be fragile, information that can be exploited to improve robustness. We address this problem by leveraging mixed-quality datasets to learn failure-aware reasoning at planning time. We introduce VINE, a hierarchical vision-language-action model that separates high-level reasoning (System 2) from low-level control (System 1) under a hierarchical reinforcement learning formalism, making failures usable as a structured learning signal rather than noisy supervision. System 2 performs feasibility-guided tree search over a 2D scene-graph abstraction: it proposes subgoal transitions, predicts success probabilities from both successes and failures, and prunes brittle branches before execution, effectively casting plan evaluation as feasibility scoring. The selected subgoal sequence is then passed to System 1, which executes low-level actions without modifying the agent's core skills. Trained entirely from offline teleoperation data, VINE integrates negative experience directly into the decision loop. Across challenging manipulation tasks, this approach consistently improves success rates and robustness, demonstrating that failure data is an essential resource for converting the broad competence of VLAs into robust execution.
comment : https://vine-vla.github.io/
☆ Autonomous Reinforcement Learning Robot Control with Intel's Loihi 2 Neuromorphic Hardware
Kenneth Stewart, Roxana Leontie, Samantha Chapin, Joe Hays, Sumit Bam Shrestha, Carl Glen Henshaw
We present an end-to-end pipeline for deploying reinforcement learning (RL) trained Artificial Neural Networks (ANNs) on neuromorphic hardware by converting them into spiking Sigma-Delta Neural Networks (SDNNs). We demonstrate that an ANN policy trained entirely in simulation can be transformed into an SDNN compatible with Intel's Loihi 2 architecture, enabling low-latency and energy-efficient inference. As a test case, we use an RL policy for controlling the Astrobee free-flying robot, similar to a previously hardware in space-validated controller. The policy, trained with Rectified Linear Units (ReLUs), is converted to an SDNN and deployed on Intel's Loihi 2, then evaluated in NVIDIA's Omniverse Isaac Lab simulation environment for closed-loop control of Astrobee's motion. We compare execution performance between GPU and Loihi 2. The results highlight the feasibility of using neuromorphic platforms for robotic control and establish a pathway toward energy-efficient, real-time neuromorphic computation in future space and terrestrial robotics applications.
comment : Submitted for review at NICE 2026 (Neuro-Inspired Computational Elements) conference
☆ BERnaT: Basque Encoders for Representing Natural Textual Diversity LREC 2026
Ekhi Azurmendi, Joseba Fernandez de Landa, Jaione Bengoetxea, Maite Heredia, Julen Etxaniz, Mikel Zubillaga, Ander Soraluze, Aitor Soroa
Language models depend on massive text corpora that are often filtered for quality, a process that can unintentionally exclude non-standard linguistic varieties, reduce model robustness and reinforce representational biases. In this paper, we argue that language models should aim to capture the full spectrum of language variation (dialectal, historical, informal, etc.) rather than relying solely on standardized text. Focusing on Basque, a morphologically rich and low-resource language, we construct new corpora combining standard, social media, and historical sources, and pre-train the BERnaT family of encoder-only models in three configurations: standard, diverse, and combined. We further propose an evaluation framework that separates Natural Language Understanding (NLU) tasks into standard and diverse subsets to assess linguistic generalization. Results show that models trained on both standard and diverse data consistently outperform those trained on standard corpora, improving performance across all task types without compromising standard benchmark accuracy. These findings highlight the importance of linguistic diversity in building inclusive, generalizable language models.
comment : Submitted to LREC 2026
☆ A Hierarchical Tree-based approach for creating Configurable and Static Deep Research Agent (Static-DRA)
Saurav Prateek
The advancement in Large Language Models has driven the creation of complex agentic systems, such as Deep Research Agents (DRAs), to overcome the limitations of static Retrieval Augmented Generation (RAG) pipelines in handling complex, multi-turn research tasks. This paper introduces the Static Deep Research Agent (Static-DRA), a novel solution built upon a configurable and hierarchical Tree-based static workflow. The core contribution is the integration of two user-tunable parameters, Depth and Breadth, which provide granular control over the research intensity. This design allows end-users to consciously balance the desired quality and comprehensiveness of the research report against the associated computational cost of Large Language Model (LLM) interactions. The agent's architecture, comprising Supervisor, Independent, and Worker agents, facilitates effective multi-hop information retrieval and parallel sub-topic investigation. We evaluate the Static-DRA against the established DeepResearch Bench using the RACE (Reference-based Adaptive Criteria-driven Evaluation) framework. Configured with a depth of 2 and a breadth of 5, and powered by the gemini-2.5-pro model, the agent achieved an overall score of 34.72. Our experiments validate that increasing the configured Depth and Breadth parameters results in a more in-depth research process and a correspondingly higher evaluation score. The Static-DRA offers a pragmatic and resource-aware solution, empowering users with transparent control over the deep research process. The entire source code, outputs and benchmark results are open-sourced at https://github.com/SauravP97/Static-Deep-Research/
☆ Hyperdimensional Computing for Sustainable Manufacturing: An Initial Assessment
Danny Hoang, Anandkumar Patel, Ruimen Chen, Rajiv Malhotra, Farhad Imani
Smart manufacturing can significantly improve efficiency and reduce energy consumption, yet the energy demands of AI models may offset these gains. This study utilizes in-situ sensing-based prediction of geometric quality in smart machining to compare the energy consumption, accuracy, and speed of common AI models. HyperDimensional Computing (HDC) is introduced as an alternative, achieving accuracy comparable to conventional models while drastically reducing energy consumption, 200$\times$ for training and 175 to 1000$\times$ for inference. Furthermore, HDC reduces training times by 200$\times$ and inference times by 300 to 600$\times$, showcasing its potential for energy-efficient smart manufacturing.
☆ Scalable Decision Focused Learning via Online Trainable Surrogates
Gaetano Signorelli, Michele Lombardi
Decision support systems often rely on solving complex optimization problems that may require to estimate uncertain parameters beforehand. Recent studies have shown how using traditionally trained estimators for this task can lead to suboptimal solutions. Using the actual decision cost as a loss function (called Decision Focused Learning) can address this issue, but with a severe loss of scalability at training time. To address this issue, we propose an acceleration method based on replacing costly loss function evaluations with an efficient surrogate. Unlike previously defined surrogates, our approach relies on unbiased estimators reducing the risk of spurious local optima and can provide information on its local confidence allowing one to switch to a fallback method when needed. Furthermore, the surrogate is designed for a black-box setting, which enables compensating for simplifications in the optimization model and account- ing for recourse actions during cost computation. In our results, the method reduces costly inner solver calls, with a solution quality comparable to other state-of-the-art techniques.
☆ PULSE: A Unified Multi-Task Architecture for Cardiac Segmentation, Diagnosis, and Few-Shot Cross-Modality Clinical Adaptation
Hania Ghouse, Maryam Alsharqi, Farhad R. Nezami, Muzammil Behzad
Cardiac image analysis remains fragmented across tasks: anatomical segmentation, disease classification, and grounded clinical report generation are typically handled by separate networks trained under different data regimes. No existing framework unifies these objectives within a single architecture while retaining generalization across imaging modalities and datasets. We introduce PULSE, a multi-task vision-language framework built on self-supervised representations and optimized through a composite supervision strategy that balances region overlap learning, pixel wise classification fidelity, and boundary aware IoU refinement. A multi-scale token reconstruction decoder enables anatomical segmentation, while shared global representations support disease classification and clinically grounded text output allowing the model to transition from pixels to structures and finally clinical reasoning within one architecture. Unlike prior task-specific pipelines, PULSE learns task-invariant cardiac priors, generalizes robustly across datasets, and can be adapted to new imaging modalities with minimal supervision. This moves the field closer to a scalable, foundation style cardiac analysis framework.
☆ DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training
Dingwei Zhu, Zhiheng Xi, Shihan Dou, Yuhui Wang, Sixian Li, Junjie Ye, Honglin Guo, Shichun Liu, Chenhao Huang, Yajie Yang, Junlin Shang, Senjie Jin, Ming Zhang, Jiazheng Zhang, Caishuang Huang, Yunke Zhang, Demei Yan, Yuran Wang, Tao Gui
Reinforcement learning (RL) has shown strong performance in LLM post-training, but real-world deployment often involves noisy or incomplete supervision. In such settings, complex and unreliable supervision signals can destabilize training and harm generalization. While existing approaches such as worst-case optimization (e.g., RFQI, CQL) and mean-based methods (e.g., PPO, GRPO) can improve stability, they often overlook generalization and may produce overly conservative policies, leading to uneven performance across diverse real scenarios. To this end, we introduce DVPO (Distributional Value Modeling with Risk-aware Policy Optimization), a new RL framework that combines conditional risk theory with distributional value modeling to better balance robustness and generalization. DVPO learns token-level value distributions to provide fine-grained supervision, and applies an asymmetric risk regularization to shape the distribution tails: it contracts the lower tail to dampen noisy negative deviations, while expanding the upper tail to preserve exploratory diversity. Across extensive experiments and analysis in multi-turn dialogue, math reasoning, and scientific QA, DVPO consistently outperforms PPO, GRPO, and robust Bellman-based PPO under noisy supervision, showing its potential for LLM post-training in the real-world.
☆ MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
Jia Hu, Zhexi Lian, Xuerun Yan, Ruiang Bi, Dou Shen, Yu Ruan, Haoran Wang
Autonomous Driving (AD) vehicles still struggle to exhibit human-like behavior in highly dynamic and interactive traffic scenarios. The key challenge lies in AD's limited ability to interact with surrounding vehicles, largely due to a lack of understanding the underlying mechanisms of social interaction. To address this issue, we introduce MPCFormer, an explainable socially-aware autonomous driving approach with physics-informed and data-driven coupled social interaction dynamics. In this model, the dynamics are formulated into a discrete space-state representation, which embeds physics priors to enhance modeling explainability. The dynamics coefficients are learned from naturalistic driving data via a Transformer-based encoder-decoder architecture. To the best of our knowledge, MPCFormer is the first approach to explicitly model the dynamics of multi-vehicle social interactions. The learned social interaction dynamics enable the planner to generate manifold, human-like behaviors when interacting with surrounding traffic. By leveraging the MPC framework, the approach mitigates the potential safety risks typically associated with purely learning-based methods. Open-looped evaluation on NGSIM dataset demonstrates that MPCFormer achieves superior social interaction awareness, yielding the lowest trajectory prediction errors compared with other state-of-the-art approach. The prediction achieves an ADE as low as 0.86 m over a long prediction horizon of 5 seconds. Close-looped experiments in highly intense interaction scenarios, where consecutive lane changes are required to exit an off-ramp, further validate the effectiveness of MPCFormer. Results show that MPCFormer achieves the highest planning success rate of 94.67%, improves driving efficiency by 15.75%, and reduces the collision rate from 21.25% to 0.5%, outperforming a frontier Reinforcement Learning (RL) based planner.
comment : 17 pages, 18 figures
☆ Adapt Vision : Efficient Vision -Language Models via Adaptive Visual Acquisition
Zichuan Lin, Yicheng Liu, Yang Yang, Lvfang Tao, Deheng Ye
Vision-Language Models (VLMs) have achieved remarkable success in visual question answering tasks, but their reliance on large numbers of visual tokens introduces significant computational overhead. While existing efficient VLM approaches reduce visual tokens through fixed-ratio compression, they operate passively and lack the ability to adapt to varying task requirements. This motivates a fundamental question: Can VLMs autonomously determine the minimum number of visual tokens required for each sample? Inspired by human active vision mechanisms, we introduce AdaptVision, an efficient VLM paradigm that enables adaptive visual token acquisition through a coarse-to-fine approach. Our model initially processes compressed visual tokens from low-resolution images and selectively acquires additional visual information by invoking a bounding box tool to crop key regions when necessary. We train AdaptVision using a reinforcement learning framework that carefully balances accuracy and efficiency. Central to our approach is Decoupled Turn Policy Optimization (DTPO), which decouples the learning objective into two components: (1) tool learning, which optimizes correct tool utilization, and (2) accuracy improvement, which refines the generated responses to improve answer correctness. Based on this formulation, we further decouple advantage estimation by computing separate advantages for tokens associated with each objective. This formulation enables more effective optimization for AdaptVision compared to vanilla GRPO. Comprehensive experiments across multiple VQA benchmarks demonstrate that AdaptVision achieves superior performance while consuming substantially fewer visual tokens than state-of-the-art efficient VLM methods.
comment : 15 pages, 9 figures
☆ Omni-AutoThink: Adaptive Multimodal Reasoning via Reinforcement Learning
Dongchao Yang, Songxiang Liu, Disong Wang, Yuanyuan Wang, Guanglu Wan, Helen Meng
Recent advances in Omni models have enabled unified multimodal perception and generation. However, most existing systems still exhibit rigid reasoning behaviors, either overthinking simple problems or failing to reason when necessary. To address this limitation, we propose Omni-AutoThink, a novel adaptive reasoning framework that dynamically adjusts the model's reasoning depth according to task difficulty. Our framework comprises two stages: (1) an Adaptive Supervised Fine-Tuning (Adaptive SFT) stage, which endows the Omni model with fundamental reasoning capability using large-scale reasoning-augmented data, and (2) an Adaptive Reinforcement Learning (Adaptive GRPO) stage, which optimizes reasoning behaviors based on task complexity and reward feedback. We further construct a comprehensive adaptive reasoning benchmark that spans text-only, text-audio, text-visual, and text-audio-visual modalities, providing both training and evaluation splits for multimodal reasoning assessment. Experimental results demonstrate that our proposed framework significantly improves adaptive reasoning performance compared to previous baselines. All benchmark data and code will be publicly released.
☆ Bayesian Optimization for Automatic Tuning of Torque-Level Nonlinear Model Predictive Control
Gabriele Fadini, Deepak Ingole, Tong Duy Son, Alisa Rupenyan
This paper presents an auto-tuning framework for torque-based Nonlinear Model Predictive Control (nMPC), where the MPC serves as a real-time controller for optimal joint torque commands. The MPC parameters, including cost function weights and low-level controller gains, are optimized using high-dimensional Bayesian Optimization (BO) techniques, specifically Sparse Axis-Aligned Subspace (SAASBO) with a digital twin (DT) to achieve precise end-effector trajectory real-time tracking on an UR10e robot arm. The simulation model allows efficient exploration of the high-dimensional parameter space, and it ensures safe transfer to hardware. Our simulation results demonstrate significant improvements in tracking performance (+41.9%) and reduction in solve times (-2.5%) compared to manually-tuned parameters. Moreover, experimental validation on the real robot follows the trend (with a +25.8% improvement), emphasizing the importance of digital twin-enabled automated parameter optimization for robotic operations.
comment : 6 pages, 7 figures, 3 tables
☆ In- Context Representation Hijacking
Itay Yona, Amir Sarid, Michael Karasik, Yossi Gandelsman
We introduce \textbf{Doublespeak}, a simple \emph{in-context representation hijacking} attack against large language models (LLMs). The attack works by systematically replacing a harmful keyword (e.g., \textit{bomb}) with a benign token (e.g., \textit{carrot}) across multiple in-context examples, provided a prefix to a harmful request. We demonstrate that this substitution leads to the internal representation of the benign token converging toward that of the harmful one, effectively embedding the harmful semantics under a euphemism. As a result, superficially innocuous prompts (e.g., ``How to build a carrot?'') are internally interpreted as disallowed instructions (e.g., ``How to build a bomb?''), thereby bypassing the model's safety alignment. We use interpretability tools to show that this semantic overwrite emerges layer by layer, with benign meanings in early layers converging into harmful semantics in later ones. Doublespeak is optimization-free, broadly transferable across model families, and achieves strong success rates on closed-source and open-source systems, reaching 74\% ASR on Llama-3.3-70B-Instruct with a single-sentence context override. Our findings highlight a new attack surface in the latent space of LLMs, revealing that current alignment strategies are insufficient and should instead operate at the representation level.
☆ RoCo: Role-Based LLMs Collaboration for Automatic Heuristic Design
Jiawei Xu, Fengfeng Wei, Weineng Chen
Automatic Heuristic Design (AHD) has gained traction as a promising solution for solving combinatorial optimization problems (COPs). Large Language Models (LLMs) have emerged and become a promising approach to achieving AHD, but current LLM-based AHD research often only considers a single role. This paper proposes RoCo, a novel Multi-Agent Role-Based System, to enhance the diversity and quality of AHD through multi-role collaboration. RoCo coordinates four specialized LLM-guided agents-explorer, exploiter, critic, and integrator-to collaboratively generate high-quality heuristics. The explorer promotes long-term potential through creative, diversity-driven thinking, while the exploiter focuses on short-term improvements via conservative, efficiency-oriented refinements. The critic evaluates the effectiveness of each evolution step and provides targeted feedback and reflection. The integrator synthesizes proposals from the explorer and exploiter, balancing innovation and exploitation to drive overall progress. These agents interact in a structured multi-round process involving feedback, refinement, and elite mutations guided by both short-term and accumulated long-term reflections. We evaluate RoCo on five different COPs under both white-box and black-box settings. Experimental results demonstrate that RoCo achieves superior performance, consistently generating competitive heuristics that outperform existing methods including ReEvo and HSEvo, both in white-box and black-box scenarios. This role-based collaborative paradigm establishes a new standard for robust and high-performing AHD.
☆ Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective
Jingyang Ou, Jiaqi Han, Minkai Xu, Shaoxuan Xu, Jianwen Xie, Stefano Ermon, Yi Wu, Chongxuan Li
Reinforcement Learning (RL) has proven highly effective for autoregressive language models, but adapting these methods to diffusion large language models (dLLMs) presents fundamental challenges. The core difficulty lies in likelihood approximation: while autoregressive models naturally provide token-level conditional probabilities essential for token-level RL objectives (e.g., GRPO), dLLMs generate sequences through iterative non-autoregressive denoising steps that lack this factorization. To address this fundamental mismatch, we propose ELBO-based Sequence-level Policy Optimization (ESPO), a principled RL framework that treats entire sequence generation as a single action and uses the ELBO as a tractable sequence-level likelihood proxy. Our method incorporates per-token normalization of importance ratios and robust KL-divergence estimation to ensure stable large-scale training. Extensive experiments on mathematical reasoning, coding, and planning tasks demonstrate that ESPO significantly outperforms token-level baselines, achieving dramatic improvements of 20-40 points on the Countdown task, while maintaining consistent gains on math and coding benchmarks. Our approach establishes sequence-level optimization as a principled and empirically effective paradigm for RL in dLLMs. Our code is available at https://github.com/ML-GSAI/ESPO.
☆ Research on Brain Tumor Classification Method Based on Improved ResNet34 Network
Yufeng Li, Wenchao Zhao, Bo Dang, Weimin Wang
Previously, image interpretation in radiology relied heavily on manual methods. However, manual classification of brain tumor medical images is time-consuming and labor-intensive. Even with shallow convolutional neural network models, the accuracy is not ideal. To improve the efficiency and accuracy of brain tumor image classification, this paper proposes a brain tumor classification model based on an improved ResNet34 network. This model uses the ResNet34 residual network as the backbone network and incorporates multi-scale feature extraction. It uses a multi-scale input module as the first layer of the ResNet34 network and an Inception v2 module as the residual downsampling layer. Furthermore, a channel attention mechanism module assigns different weights to different channels of the image from a channel domain perspective, obtaining more important feature information. The results after a five-fold crossover experiment show that the average classification accuracy of the improved network model is approximately 98.8%, which is not only 1% higher than ResNet34, but also only 80% of the number of parameters of the original model. Therefore, the improved network model not only improves accuracy but also reduces clutter, achieving a classification effect with fewer parameters and higher accuracy.
☆ Out-of-the-box: Black-box Causal Attacks on Object Detectors
Melane Navaratnarajah, David A. Kelly, Hana Chockler
Adversarial perturbations are a useful way to expose vulnerabilities in object detectors. Existing perturbation methods are frequently white-box and architecture specific. More importantly, while they are often successful, it is rarely clear why they work. Insights into the mechanism of this success would allow developers to understand and analyze these attacks, as well as fine-tune the model to prevent them. This paper presents BlackCAtt, a black-box algorithm and a tool, which uses minimal, causally sufficient pixel sets to construct explainable, imperceptible, reproducible, architecture-agnostic attacks on object detectors. BlackCAtt combines causal pixels with bounding boxes produced by object detectors to create adversarial attacks that lead to the loss, modification or addition of a bounding box. BlackCAtt works across different object detectors of different sizes and architectures, treating the detector as a black box. We compare the performance of BlackCAtt with other black-box attack methods and show that identification of causal pixels leads to more precisely targeted and less perceptible attacks. On the COCO test dataset, our approach is 2.7 times better than the baseline in removing a detection, 3.86 times better in changing a detection, and 5.75 times better in triggering new, spurious, detections. The attacks generated by BlackCAtt are very close to the original image, and hence imperceptible, demonstrating the power of causal pixels.
☆ AI/ML in 3GPP 5G Advanced - Services and Architecture
Pradnya Taksande, Shwetha Kiran, Pranav Jha, Prasanna Chaporkar
The 3rd Generation Partnership Project (3GPP), the standards body for mobile networks, is in the final phase of Release 19 standardization and is beginning Release 20. Artificial Intelligence/ Machine Learning (AI/ML) has brought about a paradigm shift in technology and it is being adopted across industries and verticals. 3GPP has been integrating AI/ML into the 5G advanced system since Release 18. This paper focuses on the AI/ML related technological advancements and features introduced in Release 19 within the Service and System Aspects (SA) Technical specifications group of 3GPP. The advancements relate to two paradigms: (i) enhancements that AI/ML brought to the 5G advanced system (AI for network), e.g. resource optimization, and (ii) enhancements that were made to the 5G system to support AI/ML applications (Network for AI), e.g. image recognition.
☆ Context -Aware Hierarchical Learning: A Two-Step Paradigm towards Safer LLMs
Tengyun Ma, Jiaqi Yao, Daojing He, Shihao Peng, Yu Li, Shaohui Liu, Zhuotao Tian
Large Language Models (LLMs) have emerged as powerful tools for diverse applications. However, their uniform token processing paradigm introduces critical vulnerabilities in instruction handling, particularly when exposed to adversarial scenarios. In this work, we identify and propose a novel class of vulnerabilities, termed Tool-Completion Attack (TCA), which exploits function-calling mechanisms to subvert model behavior. To evaluate LLM robustness against such threats, we introduce the Tool-Completion benchmark, a comprehensive security assessment framework, which reveals that even state-of-the-art models remain susceptible to TCA, with surprisingly high attack success rates. To address these vulnerabilities, we introduce Context-Aware Hierarchical Learning (CAHL), a sophisticated mechanism that dynamically balances semantic comprehension with role-specific instruction constraints. CAHL leverages the contextual correlations between different instruction segments to establish a robust, context-aware instruction hierarchy. Extensive experiments demonstrate that CAHL significantly enhances LLM robustness against both conventional attacks and the proposed TCA, exhibiting strong generalization capabilities in zero-shot evaluations while still preserving model performance on generic tasks. Our code is available at https://github.com/S2AILab/CAHL.
☆ Over-the-Air Federated Learning: Rethinking Edge AI Through Signal Processing
Seyed Mohammad Azimi-Abarghouyi, Carlo Fischione, Kaibin Huang
Over-the-Air Federated Learning (AirFL) is an emerging paradigm that tightly integrates wireless signal processing and distributed machine learning to enable scalable AI at the network edge. By leveraging the superposition property of wireless signals, AirFL performs communication and model aggregation of the learning process simultaneously, significantly reducing latency, bandwidth, and energy consumption. This article offers a tutorial treatment of AirFL, presenting a novel classification into three design approaches: CSIT-aware, blind, and weighted AirFL. We provide a comprehensive guide to theoretical foundations, performance analysis, complexity considerations, practical limitations, and prospective research directions.
☆ Matrix Editing Meets Fair Clustering: Parameterized Algorithms and Complexity
Robert Ganian, Hung P. Hoang, Simon Wietheger
We study the computational problem of computing a fair means clustering of discrete vectors, which admits an equivalent formulation as editing a colored matrix into one with few distinct color-balanced rows by changing at most $k$ values. While NP-hard in both the fairness-oblivious and the fair settings, the problem is well-known to admit a fixed-parameter algorithm in the former ``vanilla'' setting. As our first contribution, we exclude an analogous algorithm even for highly restricted fair means clustering instances. We then proceed to obtain a full complexity landscape of the problem, and establish tractability results which capture three means of circumventing our obtained lower bound: placing additional constraints on the problem instances, fixed-parameter approximation, or using an alternative parameterization targeting tree-like matrices.
☆ Quantum Topological Graph Neural Networks for Detecting Complex Fraud Patterns
Mohammad Doost, Mohammad Manthouri
We propose a novel QTGNN framework for detecting fraudulent transactions in large-scale financial networks. By integrating quantum embedding, variational graph convolutions, and topological data analysis, QTGNN captures complex transaction dynamics and structural anomalies indicative of fraud. The methodology includes quantum data embedding with entanglement enhancement, variational quantum graph convolutions with non-linear dynamics, extraction of higher-order topological invariants, hybrid quantum-classical anomaly learning with adaptive optimization, and interpretable decision-making via topological attribution. Rigorous convergence guarantees ensure stable training on noisy intermediate-scale quantum (NISQ) devices, while stability of topological signatures provides robust fraud detection. Optimized for NISQ hardware with circuit simplifications and graph sampling, the framework scales to large transaction networks. Simulations on financial datasets, such as PaySim and Elliptic, benchmark QTGNN against classical and quantum baselines, using metrics like ROC-AUC, precision, and false positive rate. An ablation study evaluates the contributions of quantum embeddings, topological features, non-linear channels, and hybrid learning. QTGNN offers a theoretically sound, interpretable, and practical solution for financial fraud detection, bridging quantum machine learning, graph theory, and topological analysis.
☆ ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videos
Qi'ao Xu, Tianwen Qian, Yuqian Fu, Kailing Li, Yang Jiao, Jiacheng Zhang, Xiaoling Wang, Liang He
A core capability towards general embodied intelligence lies in localizing task-relevant objects from an egocentric perspective, formulated as Spatio-Temporal Video Grounding (STVG). Despite recent progress, existing STVG studies remain largely confined to object-centric and descriptive instructions, neglecting the task-oriented reasoning that is crucial for embodied agents to accomplish goal-directed interactions. To bridge this gap, we introduce \textbf{ToG-Bench}, the first task-oriented spatio-temporal video grounding benchmark for egocentric videos. ToG-Bench is characterized by three key features: (1) \textbf{Task-oriented Grounding}, which requires identifying and localizing objects based on intended tasks rather than straightforward descriptions; (2) \textbf{Explicit-Implicit Dual Grounding}, where target objects can be either explicitly mentioned or implicitly inferred by contextual reasoning; (3) \textbf{One-to-Many Grounding}, where a single instruction may correspond to multiple objects involved in task execution. Built upon videos sourced from ScanNet, ToG-Bench comprises 100 annotated clips with 2,704 task-oriented grounding instructions, constructed via a semi-automated pipeline that combines foundation model annotation and human refinement. In addition, we introduce a set of task-level evaluation metrics tailored for multi-object and explicit-implicit object grounding, and systematically benchmark seven state-of-the-art MLLMs. Extensive experiments reveal the intrinsic challenges of task-oriented STVG and substantial performance gaps across explicit-implicit and multi-object grounding, highlighting the difficulty of bridging perception and interaction in embodied scenarios. Data and code will be released at: \href{https://github.com/qaxuDev/ToG-Bench}{https://github.com/qaxuDev/ToG-Bench}..
comment : 26 pages
☆ Dynamically Scaled Activation Steering
Alex Ferrando, Xavier Suau, Jordi Gonzàlez, Pau Rodriguez
Activation steering has emerged as a powerful method for guiding the behavior of generative models towards desired outcomes such as toxicity mitigation. However, most existing methods apply interventions uniformly across all inputs, degrading model performance when steering is unnecessary. We introduce Dynamically Scaled Activation Steering (DSAS), a method-agnostic steering framework that decouples when to steer from how to steer. DSAS adaptively modulates the strength of existing steering transformations across layers and inputs, intervening strongly only when undesired behavior is detected. At generation time, DSAS computes context-dependent scaling factors that selectively adjust the strength of any steering method. We also show how DSAS can be jointly optimized end-to-end together with the steering function. When combined with existing steering methods, DSAS consistently improves the Pareto front with respect to steering alone, achieving a better trade-off between toxicity mitigation and utility preservation. We further demonstrate DSAS's generality by applying it to a text-to-image diffusion model, showing how adaptive steering allows the modulation of specific concepts. Finally, DSAS introduces minimal computational overhead while improving interpretability, pinpointing which tokens require steering and by how much.
☆ MKSNet: Advanced Small Object Detection in Remote Sensing Imagery with Multi-Kernel and Dual Attention Mechanisms
Jiahao Zhang, Xiao Zhao, Guangyu Gao
Deep convolutional neural networks (DCNNs) have substantially advanced object detection capabilities, particularly in remote sensing imagery. However, challenges persist, especially in detecting small objects where the high resolution of these images and the small size of target objects often result in a loss of critical information in the deeper layers of conventional CNNs. Additionally, the extensive spatial redundancy and intricate background details typical in remote-sensing images tend to obscure these small targets. To address these challenges, we introduce Multi-Kernel Selection Network (MKSNet), a novel network architecture featuring a novel Multi-Kernel Selection mechanism. The MKS mechanism utilizes large convolutional kernels to effectively capture an extensive range of contextual information. This innovative design allows for adaptive kernel size selection, significantly enhancing the network's ability to dynamically process and emphasize crucial spatial details for small object detection. Furthermore, MKSNet also incorporates a dual attention mechanism, merging spatial and channel attention modules. The spatial attention module adaptively fine-tunes the spatial weights of feature maps, focusing more intensively on relevant regions while mitigating background noise. Simultaneously, the channel attention module optimizes channel information selection, improving feature representation and detection accuracy. Empirical evaluations on the DOTA-v1.0 and HRSC2016 benchmark demonstrate that MKSNet substantially surpasses existing state-of-the-art models in detecting small objects in remote sensing images. These results highlight MKSNet's superior ability to manage the complexities associated with multi-scale and high-resolution image data, confirming its effectiveness and innovation in remote sensing object detection.
☆ AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment
Ahmad Aghaebrahimian
Large Language Models have significantly advanced natural language processing tasks, but remain prone to generating incorrect or misleading but plausible arguments. This issue, known as hallucination, is particularly concerning in high-stakes domains like clinical applications, where factual inaccuracies can have severe consequences. Existing evaluation metrics fail to adequately assess factual consistency and lack interpretability, making diagnosing and mitigating errors difficult. We propose an interpretable framework for factual consistency assessment for in-domain and open-domain texts to address these limitations. Our approach decomposes text into atomic facts and introduces a flexible, schema-free methodology. Unlike previous methods with an absolute metric, we incorporate a weighted metric to enhance factual evaluation. Additionally, we propose a mechanism to control assessment complexity in intricate domains. We benchmark our approach on popular general and clinical datasets and release our code to support fact-aware model training in future research.
☆ MemVerse: Multimodal Memory for Lifelong Learning Agent s
Junming Liu, Yifei Sun, Weihua Cheng, Haodong Lei, Yirong Chen, Licheng Wen, Xuemeng Yang, Daocheng Fu, Pinlong Cai, Nianchen Deng, Yi Yu, Shuyue Hu, Botian Shi, Ding Wang
Despite rapid progress in large-scale language and vision models, AI agents still suffer from a fundamental limitation: they cannot remember. Without reliable memory, agents catastrophically forget past experiences, struggle with long-horizon reasoning, and fail to operate coherently in multimodal or interactive environments. We introduce MemVerse, a model-agnostic, plug-and-play memory framework that bridges fast parametric recall with hierarchical retrieval-based memory, enabling scalable and adaptive multimodal intelligence. MemVerse maintains short-term memory for recent context while transforming raw multimodal experiences into structured long-term memories organized as hierarchical knowledge graphs. This design supports continual consolidation, adaptive forgetting, and bounded memory growth. To handle real-time demands, MemVerse introduces a periodic distillation mechanism that compresses essential knowledge from long-term memory into the parametric model, allowing fast, differentiable recall while preserving interpretability. Extensive experiments demonstrate that MemVerse significantly improves multimodal reasoning and continual learning efficiency, empowering agents to remember, adapt, and reason coherently across extended interactions.
comment : 11 pages, 2 figures, 2 tables
☆ The promising potential of vision language models for the generation of textual weather forecasts
Edward C. C. Steele, Dinesh Mane, Emilio Monti, Luis Orus, Rebecca Chantrill-Cheyette, Matthew Couch, Kirstine I. Dale, Simon Eaton, Govindarajan Rangarajan, Amir Majlesi, Steven Ramsdale, Michael Sharpe, Craig Smith, Jonathan Smith, Rebecca Yates, Holly Ellis, Charles Ewen
Despite the promising capability of multimodal foundation models, their application to the generation of meteorological products and services remains nascent. To accelerate aspiration and adoption, we explore the novel use of a vision language model for writing the iconic Shipping Forecast text directly from video-encoded gridded weather data. These early results demonstrate promising scalable technological opportunities for enhancing production efficiency and service innovation within the weather enterprise and beyond.
comment : 7 pages, 2 tables
☆ SELF: A Robust Singular Value and Eigenvalue Approach for LLM Fingerprinting
Hanxiu Zhang, Yue Zheng
The protection of Intellectual Property (IP) in Large Language Models (LLMs) represents a critical challenge in contemporary AI research. While fingerprinting techniques have emerged as a fundamental mechanism for detecting unauthorized model usage, existing methods -- whether behavior-based or structural -- suffer from vulnerabilities such as false claim attacks or susceptible to weight manipulations. To overcome these limitations, we propose SELF, a novel intrinsic weight-based fingerprinting scheme that eliminates dependency on input and inherently resists false claims. SELF achieves robust IP protection through two key innovations: 1) unique, scalable and transformation-invariant fingerprint extraction via singular value and eigenvalue decomposition of LLM attention weights, and 2) effective neural network-based fingerprint similarity comparison based on few-shot learning and data augmentation. Experimental results demonstrate SELF maintains high IP infringement detection accuracy while showing strong robustness against various downstream modifications, including quantization, pruning, and fine-tuning attacks. Our code is available at https://github.com/HanxiuZhang/SELF_v2.
☆ KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computing
Lishuo Deng, Shaojie Xu, Jinwu Chen, Changwei Yan, Jiajie Wang, Zhe Jiang, Weiwei Shan
Deploying large language models (LLMs) on edge devices enables personalized agents with strong privacy and low cost. However, with tens to hundreds of billions of parameters, single-batch autoregressive inference suffers from extremely low arithmetic intensity, creating severe weight-loading and bandwidth pressures on resource-constrained platforms. Recent in-flash computing (IFC) solutions alleviate this bottleneck by co-locating weight-related linear computations in the decode phase with flash, yet still rely on DRAM for the key-value (KV) cache. As context length grows, the KV cache can exceed model weights in size, imposing prohibitive DRAM cost and capacity requirements. Attempts to offload KV cache to flash suffer from severe performance penalties. We propose KVNAND, the first DRAM-free, IFC-based architecture that stores both model weights and KV cache entirely in compute-enabled 3D NAND flash. KVNAND addresses the fundamental performance challenges of flash under intensive KV cache access by leveraging IFC for all memory-bound operations to reduce data transfer overhead, introducing head-group parallelism to boost throughput, and employing page-level KV cache mapping to align token access patterns with flash organization. In addition, we propose a design space exploration framework that evaluates discrete and compact KVNAND variants to balance weight and KV placement, automatically identifying the optimal design trade-off. These techniques mitigate latency, energy, and reliability concerns, turning flash into a practical medium for long-context KV storage. Evaluations on MHA 7B and GQA 70B LLMs show that KVNAND achieves 1.98\(\times\)/1.94\(\times\)/2.05\(\times\) geomean speedup at 128/1K/10K-token contexts compared to DRAM-equipped IFC designs and addresses out-of-memory failures at 100K context length.
☆ DeepRule: An Integrated Framework for Automated Business Rule Generation via Deep Predictive Modeling and Hybrid Search Optimization
Yusen Wu, Xiaotie Deng
This paper proposes DeepRule, an integrated framework for automated business rule generation in retail assortment and pricing optimization. Addressing the systematic misalignment between existing theoretical models and real-world economic complexities, we identify three critical gaps: (1) data modality mismatch where unstructured textual sources (e.g. negotiation records, approval documents) impede accurate customer profiling; (2) dynamic feature entanglement challenges in modeling nonlinear price elasticity and time-varying attributes; (3) operational infeasibility caused by multi-tier business constraints. Our framework introduces a tri-level architecture for above challenges. We design a hybrid knowledge fusion engine employing large language models (LLMs) for deep semantic parsing of unstructured text, transforming distributor agreements and sales assessments into structured features while integrating managerial expertise. Then a game-theoretic constrained optimization mechanism is employed to dynamically reconcile supply chain interests through bilateral utility functions, encoding manufacturer-distributor profit redistribution as endogenous objectives under hierarchical constraints. Finally an interpretable decision distillation interface leveraging LLM-guided symbolic regression to find and optimize pricing strategies and auditable business rules embeds economic priors (e.g. non-negative elasticity) as hard constraints during mathematical expression search. We validate the framework in real retail environments achieving higher profits versus systematic B2C baselines while ensuring operational feasibility. This establishes a close-loop pipeline unifying unstructured knowledge injection, multi-agent optimization, and interpretable strategy synthesis for real economic intelligence.
☆ Fine-grained Narrative Classification in Biased News Articles
Zeba Afroz, Harsh Vardhan, Pawan Bhakuni, Aanchal Punia, Rajdeep Kumar, Md. Shad Akhtar
Narratives are the cognitive and emotional scaffolds of propaganda. They organize isolated persuasive techniques into coherent stories that justify actions, attribute blame, and evoke identification with ideological camps. In this paper, we propose a novel fine-grained narrative classification in biased news articles. We also explore article-bias classification as the precursor task to narrative classification and fine-grained persuasive technique identification. We develop INDI-PROP, the first ideologically grounded fine-grained narrative dataset with multi-level annotation for analyzing propaganda in Indian news media. Our dataset INDI-PROP comprises 1,266 articles focusing on two polarizing socio-political events in recent times: CAA and the Farmers' protest. Each article is annotated at three hierarchical levels: (i) ideological article-bias (pro-government, pro-opposition, neutral), (ii) event-specific fine-grained narrative frames anchored in ideological polarity and communicative intent, and (iii) persuasive techniques. We propose FANTA and TPTC, two GPT-4o-mini guided multi-hop prompt-based reasoning frameworks for the bias, narrative, and persuasive technique classification. FANTA leverages multi-layered communicative phenomena by integrating information extraction and contextual framing for hierarchical reasoning. On the other hand, TPTC adopts systematic decomposition of persuasive cues via a two-stage approach. Our evaluation suggests substantial improvement over underlying baselines in each case.
☆ When, How Long and How Much? Interpretable Neural Networks for Time Series Regression by Learning to Mask and Aggregate
Florent Forest, Amaury Wei, Olga Fink
Time series extrinsic regression (TSER) refers to the task of predicting a continuous target variable from an input time series. It appears in many domains, including healthcare, finance, environmental monitoring, and engineering. In these settings, accurate predictions and trustworthy reasoning are both essential. Although state-of-the-art TSER models achieve strong predictive performance, they typically operate as black boxes, making it difficult to understand which temporal patterns drive their decisions. Post-hoc interpretability techniques, such as feature attribution, aim to to explain how the model arrives at its predictions, but often produce coarse, noisy, or unstable explanations. Recently, inherently interpretable approaches based on concepts, additive decompositions, or symbolic regression, have emerged as promising alternatives. However, these approaches remain limited: they require explicit supervision on the concepts themselves, often cannot capture interactions between time-series features, lack expressiveness for complex temporal patterns, and struggle to scale to high-dimensional multivariate data. To address these limitations, we propose MAGNETS (Mask-and-AGgregate NEtwork for Time Series), an inherently interpretable neural architecture for TSER. MAGNETS learns a compact set of human-understandable concepts without requiring any annotations. Each concept corresponds to a learned, mask-based aggregation over selected input features, explicitly revealing both which features drive predictions and when they matter in the sequence. Predictions are formed as combinations of these learned concepts through a transparent, additive structure, enabling clear insight into the model's decision process.
comment : 12 pages, 5 figures, 4 tables
☆ EnCompass: Enhancing Agent Programming with Search Over Program Execution Paths NeurIPS 2025
Zhening Li, Armando Solar-Lezama, Yisong Yue, Stephan Zheng
We introduce a new approach to agent programming, the development of LLM-based agents. Current approaches to agent programming often entangle two aspects of agent design: the core workflow logic and the inference-time strategy (e.g., tree search). We introduce "probabilistic angelic nondeterminism" ("PAN"), a programming model that disentangles these two concerns, allowing the programmer to describe the agent workflow and independently experiment with different inference-time strategies by simply changing a few inputs. We provide an implementation of PAN in Python as the EnCompass framework, which uses a Python decorator to compile agent workflow programs into a search space. We present three case studies that demonstrate how the framework lets the programmer quickly improve the reliability of an agent and easily switch between different inference-time strategies, all with little additional coding.
comment : 65 pages, 2 figures, published in NeurIPS 2025
☆ Machine Learning to Predict Slot Usage in TSCH Wireless Sensor Networks
Stefano Scanzio, Gabriele Formis, Tullio Facchinetti, Gianluca Cena
Wireless sensor networks (WSNs) are employed across a wide range of industrial applications where ultra-low power consumption is a critical prerequisite. At the same time, these systems must maintain a certain level of determinism to ensure reliable and predictable operation. In this view, time slotted channel hopping (TSCH) is a communication technology that meets both conditions, making it an attractive option for its usage in industrial WSNs. This work proposes the use of machine learning to learn the traffic pattern generated in networks based on the TSCH protocol, in order to turn nodes into a deep sleep state when no transmission is planned and thus to improve the energy efficiency of the WSN. The ability of machine learning models to make good predictions at different network levels in a typical tree network topology was analyzed in depth, showing how their capabilities degrade while approaching the root of the tree. The application of these models on simulated data based on an accurate modeling of wireless sensor nodes indicates that the investigated algorithms can be suitably used to further and substantially reduce the power consumption of a TSCH network.
comment : preprint accepted, 8 pages, 2025
☆ State Space Models for Bioacoustics: A comparative Evaluation with Transformer s
Chengyu Tang, Sanjeev Baskiyar
In this study, we evaluate the efficacy of the Mamba model in the field of bioacoustics. We first pretrain a Mamba-based audio large language model (LLM) on a large corpus of audio data using self-supervised learning. We fine-tune and evaluate BioMamba on the BEANS benchmark, a collection of diverse bioacoustic tasks including classification and detection, and compare its performance and efficiency with multiple baseline models, including AVES, a state-of-the-art Transformer-based model. The results show that BioMamba achieves comparable performance with AVES while consumption significantly less VRAM, demonstrating its potential in this domain.
☆ Reason-Plan-ReAct: A Reasoner-Planner Supervising a ReAct Executor for Complex Enterprise Tasks AAAI 2026
Gianni Molinari, Fabio Ciravegna
Despite recent advances, autonomous agents often struggle to solve complex tasks in enterprise domains that require coordinating multiple tools and processing diverse data sources. This struggle is driven by two main limitations. First, single-agent architectures enforce a monolithic plan-execute loop, which directly causes trajectory instability. Second, the requirement to use local open-weight models for data privacy introduces smaller context windows leading to the rapid consumption of context from large tool outputs. To solve this problem we introduce RP-ReAct (Reasoner Planner-ReAct), a novel multi-agent approach that fundamentally decouples strategic planning from low-level execution to achieve superior reliability and efficiency. RP-ReAct consists of a Reasoner Planner Agent (RPA), responsible for planning each sub-step, continuously analysing the execution results using the strong reasoning capabilities of a Large Reasoning Model, and one or multiple Proxy-Execution Agent (PEA) that translates sub-steps into concrete tool interactions using a ReAct approach. Crucially, we incorporate a context-saving strategy within the PEA to mitigate context window overflow by managing large tool outputs via external storage and on-demand access. We evaluate RP-ReAct, on the challenging, multi-domain ToolQA benchmark using a diverse set of six open-weight reasoning models. Our empirical results show that RP-ReAct achieves superior performance and improved generalization ability over state-of-the-art baselines when addressing diverse complex tasks across the evaluated domains. Furthermore we establish the enhanced robustness and stability of our approach across different model scales, paving the way for effective and deployable agentic solutions for enterprises.
comment : 11 pages, 1 figure, 2 tables, Workshop AAAI 2026 agentic AI Benchmarks and Applications for Enterprise Tasks
☆ Dynamic Content Moderation in Livestreams: Combining Supervised Classification with MLLM-Boosted Similarity Matching KDD 2026
Wei Chee Yew, Hailun Xu, Sanjay Saha, Xiaotian Fan, Hiok Hian Ong, David Yuchen Wang, Kanchan Sarkar, Zhenheng Yang, Danhui Guan
Content moderation remains a critical yet challenging task for large-scale user-generated video platforms, especially in livestreaming environments where moderation must be timely, multimodal, and robust to evolving forms of unwanted content. We present a hybrid moderation framework deployed at production scale that combines supervised classification for known violations with reference-based similarity matching for novel or subtle cases. This hybrid design enables robust detection of both explicit violations and novel edge cases that evade traditional classifiers. Multimodal inputs (text, audio, visual) are processed through both pipelines, with a multimodal large language model (MLLM) distilling knowledge into each to boost accuracy while keeping inference lightweight. In production, the classification pipeline achieves 67% recall at 80% precision, and the similarity pipeline achieves 76% recall at 80% precision. Large-scale A/B tests show a 6-8% reduction in user views of unwanted livestreams}. These results demonstrate a scalable and adaptable approach to multimodal content governance, capable of addressing both explicit violations and emerging adversarial behaviors.
comment : Accepted at KDD 2026
☆ PARC: An Autonomous Self-Reflective Coding Agent for Robust Execution of Long-Horizon Tasks
Yuki Orimo, Iori Kurata, Hodaka Mori, Ryuhei Okuno, Ryohto Sawada, Daisuke Okanohara
We introduce PARC, a coding agent for the autonomous and robust execution of long-horizon computational tasks. PARC is built on a hierarchical multi-agent architecture incorporating task planning, execution, and a mechanism that evaluates its own actions and their outcomes from an independent context and provides feedback, namely self-assessment and self-feedback. This design enables PARC to detect and correct high-level strategic errors and sustain progress without human intervention. We evaluate PARC across computational science and data science tasks. In materials science, it autonomously reproduces key results from studies on lithium-ion conduction and alloy segregation. In particular, it coordinates dozens of parallel simulation tasks, each requiring roughly 43 hours of computation, managing orchestration, monitoring, and error correction end-to-end. In Kaggle-based experiments, starting from minimal natural-language instructions, PARC conducts data analysis and implements search strategies, producing solutions competitive with human-engineered baselines. These results highlight the potential of integrating a hierarchical multi-agent system with self-assessment and self-feedback to enable AI systems capable of independent, large-scale scientific and analytical work.
☆ A Learning-based Control Methodology for Transitioning VTOL UAVs
Zexin Lin, Yebin Zhong, Hanwen Wan, Jiu Cheng, Zhenglong Sun, Xiaoqiang Ji
Transition control poses a critical challenge in Vertical Take-Off and Landing Unmanned Aerial Vehicle (VTOL UAV) development due to the tilting rotor mechanism, which shifts the center of gravity and thrust direction during transitions. Current control methods' decoupled control of altitude and position leads to significant vibration, and limits interaction consideration and adaptability. In this study, we propose a novel coupled transition control methodology based on reinforcement learning (RL) driven controller. Besides, contrasting to the conventional phase-transition approach, the ST3M method demonstrates a new perspective by treating cruise mode as a special case of hover. We validate the feasibility of applying our method in simulation and real-world environments, demonstrating efficient controller development and migration while accurately controlling UAV position and attitude, exhibiting outstanding trajectory tracking and reduced vibrations during the transition process.
☆ V-ITI: Mitigating Hallucinations in Multimodal Large Language Models via Visual Inference-Time Intervention
Nan Sun, Zhenyu Zhang, Xixun Lin, Kun Wang, Yanmin Shang, Naibin Gu, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang, Yanan Cao
Multimodal Large Language Models (MLLMs) excel in numerous vision-language tasks yet suffer from hallucinations, producing content inconsistent with input visuals, that undermine reliability in precision-sensitive domains. This issue stems from a fundamental problem of visual neglect, where models fail to adequately prioritize input images. Existing methods typically alleviate hallucinations by intervening in the attention score or output logits, focusing on "how to intervene" but overlooking the prerequisite "when to intervene", which leads to the "over-intervention" problem and subsequently introduces new hallucinations and unnecessary computational overhead. To address this gap, we first investigate the mechanism of visual neglect and reveal it can be accurately detected via head-level activation patterns in MLLMs. We thus propose V-ITI, a lightweight visual inference-time intervention framework integrating a Visual Neglect Detector that identifies visual neglect via head-level discriminative probes and a Visual Recall Intervenor that modulates activations with prestored visual activation information only when the visual neglect is detected. Extensive experiments across eight benchmarks and different MLLM families demonstrate that V-ITI consistently mitigates vision-related hallucinations while preserving general task performance.
☆ CookAnything: A Framework for Flexible and Consistent Multi-Step Recipe Image Generation
Ruoxuan Zhang, Bin Wen, Hongxia Xie, Yi Yao, Songhan Zuo, Jian-Yu Jiang-Lin, Hong-Han Shuai, Wen-Huang Cheng
Cooking is a sequential and visually grounded activity, where each step such as chopping, mixing, or frying carries both procedural logic and visual semantics. While recent diffusion models have shown strong capabilities in text-to-image generation, they struggle to handle structured multi-step scenarios like recipe illustration. Additionally, current recipe illustration methods are unable to adjust to the natural variability in recipe length, generating a fixed number of images regardless of the actual instructions structure. To address these limitations, we present CookAnything, a flexible and consistent diffusion-based framework that generates coherent, semantically distinct image sequences from textual cooking instructions of arbitrary length. The framework introduces three key components: (1) Step-wise Regional Control (SRC), which aligns textual steps with corresponding image regions within a single denoising process; (2) Flexible RoPE, a step-aware positional encoding mechanism that enhances both temporal coherence and spatial diversity; and (3) Cross-Step Consistency Control (CSCC), which maintains fine-grained ingredient consistency across steps. Experimental results on recipe illustration benchmarks show that CookAnything performs better than existing methods in training-based and training-free settings. The proposed framework supports scalable, high-quality visual synthesis of complex multi-step instructions and holds significant potential for broad applications in instructional media, and procedural content creation.
comment : Accepted by ACM Multimedia 2025
☆ Rethinking Prompt Design for Inference-time Scaling in Text-to-Visual Generation
Subin Kim, Sangwoo Mo, Mamshad Nayeem Rizve, Yiran Xu, Difan Liu, Jinwoo Shin, Tobias Hinz
Achieving precise alignment between user intent and generated visuals remains a central challenge in text-to-visual generation, as a single attempt often fails to produce the desired output. To handle this, prior approaches mainly scale the visual generation process (e.g., increasing sampling steps or seeds), but this quickly leads to a quality plateau. This limitation arises because the prompt, crucial for guiding generation, is kept fixed. To address this, we propose Prompt Redesign for Inference-time Scaling, coined PRIS, a framework that adaptively revises the prompt during inference in response to the scaled visual generations. The core idea of PRIS is to review the generated visuals, identify recurring failure patterns across visuals, and redesign the prompt accordingly before regenerating the visuals with the revised prompt. To provide precise alignment feedback for prompt revision, we introduce a new verifier, element-level factual correction, which evaluates the alignment between prompt attributes and generated visuals at a fine-grained level, achieving more accurate and interpretable assessments than holistic measures. Extensive experiments on both text-to-image and text-to-video benchmarks demonstrate the effectiveness of our approach, including a 15% gain on VBench 2.0. These results highlight that jointly scaling prompts and visuals is key to fully leveraging scaling laws at inference-time. Visualizations are available at the website: https://subin-kim-cv.github.io/PRIS.
comment : Visualizations are available at the website: https://subin-kim-cv.github.io/PRIS
☆ Multi- Agent Reinforcement Learning with Communication-Constrained Priors
Guang Yang, Tianpei Yang, Jingwen Qiao, Yanqing Wu, Jing Huo, Xingguo Chen, Yang Gao
Communication is one of the effective means to improve the learning of cooperative policy in multi-agent systems. However, in most real-world scenarios, lossy communication is a prevalent issue. Existing multi-agent reinforcement learning with communication, due to their limited scalability and robustness, struggles to apply to complex and dynamic real-world environments. To address these challenges, we propose a generalized communication-constrained model to uniformly characterize communication conditions across different scenarios. Based on this, we utilize it as a learning prior to distinguish between lossy and lossless messages for specific scenarios. Additionally, we decouple the impact of lossy and lossless messages on distributed decision-making, drawing on a dual mutual information estimatior, and introduce a communication-constrained multi-agent reinforcement learning framework, quantifying the impact of communication messages into the global reward. Finally, we validate the effectiveness of our approach across several communication-constrained benchmarks.
☆ M3DR: Towards Universal Multilingual Multimodal Document Retrieval
Adithya S Kolavi, Vyoman Jain
Multimodal document retrieval systems have shown strong progress in aligning visual and textual content for semantic search. However, most existing approaches remain heavily English-centric, limiting their effectiveness in multilingual contexts. In this work, we present M3DR (Multilingual Multimodal Document Retrieval), a framework designed to bridge this gap across languages, enabling applicability across diverse linguistic and cultural contexts. M3DR leverages synthetic multilingual document data and generalizes across different vision-language architectures and model sizes, enabling robust cross-lingual and cross-modal alignment. Using contrastive training, our models learn unified representations for text and document images that transfer effectively across languages. We validate this capability on 22 typologically diverse languages, demonstrating consistent performance and adaptability across linguistic and script variations. We further introduce a comprehensive benchmark that captures real-world multilingual scenarios, evaluating models under monolingual, multilingual, and mixed-language settings. M3DR generalizes across both single dense vector and ColBERT-style token-level multi-vector retrieval paradigms. Our models, NetraEmbed and ColNetraEmbed achieve state-of-the-art performance with ~150% relative improvements on cross-lingual retrieval.
☆ Physics-Driven Learning Framework for Tomographic Tactile Sensing
Xuanxuan Yang, Xiuyang Zhang, Haofeng Chen, Gang Ma, Xiaojie Wang
Electrical impedance tomography (EIT) provides an attractive solution for large-area tactile sensing due to its minimal wiring and shape flexibility, but its nonlinear inverse problem often leads to severe artifacts and inaccurate contact reconstruction. This work presents PhyDNN, a physics-driven deep reconstruction framework that embeds the EIT forward model directly into the learning objective. By jointly minimizing the discrepancy between predicted and ground-truth conductivity maps and enforcing consistency with the forward PDE, PhyDNN reduces the black-box nature of deep networks and improves both physical plausibility and generalization. To enable efficient backpropagation, we design a differentiable forward-operator network that accurately approximates the nonlinear EIT response, allowing fast physics-guided training. Extensive simulations and real tactile experiments on a 16-electrode soft sensor show that PhyDNN consistently outperforms NOSER, TV, and standard DNNs in reconstructing contact shape, location, and pressure distribution. PhyDNN yields fewer artifacts, sharper boundaries, and higher metric scores, demonstrating its effectiveness for high-quality tomographic tactile sensing.
comment : 7pages,7figures
☆ NAS-LoRA: Empowering Parameter-Efficient Fine-Tuning for Visual Foundation Models with Searchable Adaptation
Renqi Chen, Haoyang Su, Shixiang Tang
The Segment Anything Model (SAM) has emerged as a powerful visual foundation model for image segmentation. However, adapting SAM to specific downstream tasks, such as medical and agricultural imaging, remains a significant challenge. To address this, Low-Rank Adaptation (LoRA) and its variants have been widely employed to enhancing SAM's adaptation performance on diverse domains. Despite advancements, a critical question arises: can we integrate inductive bias into the model? This is particularly relevant since the Transformer encoder in SAM inherently lacks spatial priors within image patches, potentially hindering the acquisition of high-level semantic information. In this paper, we propose NAS-LoRA, a new Parameter-Efficient Fine-Tuning (PEFT) method designed to bridge the semantic gap between pre-trained SAM and specialized domains. Specifically, NAS-LoRA incorporates a lightweight Neural Architecture Search (NAS) block between the encoder and decoder components of LoRA to dynamically optimize the prior knowledge integrated into weight updates. Furthermore, we propose a stage-wise optimization strategy to help the ViT encoder balance weight updates and architectural adjustments, facilitating the gradual learning of high-level semantic information. Various Experiments demonstrate our NAS-LoRA improves existing PEFT methods, while reducing training cost by 24.14% without increasing inference cost, highlighting the potential of NAS in enhancing PEFT for visual foundation models.
☆ Cell-cell communication inference and analysis: biological mechanisms, computational approaches, and future opportunities
Xiangzheng Cheng, Haili Huang, Ye Su, Qing Nie, Xiufen Zou, Suoqin Jin
In multicellular organisms, cells coordinate their activities through cell-cell communication (CCC), which are crucial for development, tissue homeostasis, and disease progression. Recent advances in single-cell and spatial omics technologies provide unprecedented opportunities to systematically infer and analyze CCC from these omics data, either by integrating prior knowledge of ligand-receptor interactions (LRIs) or through de novo approaches. A variety of computational methods have been developed, focusing on methodological innovations, accurate modeling of complex signaling mechanisms, and investigation of broader biological questions. These advances have greatly enhanced our ability to analyze CCC and generate biological hypotheses. Here, we introduce the biological mechanisms and modeling strategies of CCC, and provide a focused overview of more than 140 computational methods for inferring CCC from single-cell and spatial transcriptomic data, emphasizing the diversity in methodological frameworks and biological questions. Finally, we discuss the current challenges and future opportunities in this rapidly evolving field.
☆ ATHENA: Agent ic Team for Hierarchical Evolutionary Numerical Algorithms
Juan Diego Toscano, Daniel T. Chen, George Em Karniadakis
Bridging the gap between theoretical conceptualization and computational implementation is a major bottleneck in Scientific Computing (SciC) and Scientific Machine Learning (SciML). We introduce ATHENA (Agentic Team for Hierarchical Evolutionary Numerical Algorithms), an agentic framework designed as an Autonomous Lab to manage the end-to-end computational research lifecycle. Its core is the HENA loop, a knowledge-driven diagnostic process framed as a Contextual Bandit problem. Acting as an online learner, the system analyzes prior trials to select structural `actions' ($A_n$) from combinatorial spaces guided by expert blueprints (e.g., Universal Approximation, Physics-Informed constraints). These actions are translated into executable code ($S_n$) to generate scientific rewards ($R_n$). ATHENA transcends standard automation: in SciC, it autonomously identifies mathematical symmetries for exact analytical solutions or derives stable numerical solvers where foundation models fail. In SciML, it performs deep diagnosis to tackle ill-posed formulations and combines hybrid symbolic-numeric workflows (e.g., coupling PINNs with FEM) to resolve multiphysics problems. The framework achieves super-human performance, reaching validation errors of $10^{-14}$. Furthermore, collaborative ``human-in-the-loop" intervention allows the system to bridge stability gaps, improving results by an order of magnitude. This paradigm shift focuses from implementation mechanics to methodological innovation, accelerating scientific discovery.
☆ AsymPuzl: An Asymmetric Puzzle for multi- agent cooperation NeurIPS
Xavier Cadet, Edward Koh, Peter Chin
Large Language Model (LLM) agents are increasingly studied in multi-turn, multi-agent scenarios, yet most existing setups emphasize open-ended role-play rather than controlled evaluation. We introduce AsymPuzl, a minimal but expressive two-agent puzzle environment designed to isolate communication under information asymmetry. Each agent observes complementary but incomplete views of a symbolic puzzle and must exchange messages to solve it cooperatively. Using a diverse set of current-generation and open-source LLMs, we show that (i) strong models such as GPT-5 and Claude-4.0 reliably converge across puzzle sizes on the solution by sharing complete information in two turns, (ii) weaker models often ignore partner messages or over-correct their hypotheses, and (iii) feedback design is non-trivial: simple self-feedback improves success rates, while detailed joint feedback can hurt performance. These findings show that even in simple cooperative tasks, LLM communication strategies diverge and depend on the granularity of feedback signals. AsymPuzl thus provides a testbed for probing the limits of multi-turn cooperation and opens avenues for studying coordination mechanisms.
comment : Accepted at NeurIPS MTI-LLM 2025
☆ Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision -Language Models
Shojiro Yamabe, Futa Waseda, Daiki Shiono, Tsubasa Takahashi
Recent large vision-language models (LVLMs) have been applied to diverse VQA tasks. However, achieving practical performance typically requires task-specific fine-tuning with large numbers of image-text pairs, which are costly to collect. In this work, we study text-centric training, a setting where only textual descriptions are available and no real images are provided, as a paradigm for low-cost data scaling. Unlike images, whose collection is often restricted by privacy constraints and scarcity in niche domains, text is widely available. Moreover, text is easily editable, enabling automatic diversification and expansion with LLMs at minimal human effort. While this offers clear advantages over image collection in terms of scalability and cost, training on raw text without images still yields limited gains on VQA tasks because of the image-text modality gap. To address this issue, we propose a Text-Printed Image (TPI), which generates synthetic images by directly rendering the given textual description on a plain white canvas. This simple rendering projects text into the image modality and can be integrated into arbitrary existing LVLM training pipelines at low cost. Moreover, TPI preserves the semantics of the text, whereas text-to-image models often fail to do. Across four models and seven benchmarks, our systematic experiments show that TPI enables more effective text-centric training than synthetic images generated by a diffusion model. We further explore TPI as a low-cost data-augmentation strategy and demonstrate its practical utility. Overall, our findings highlight the significant potential of text-centric training and, more broadly, chart a path toward fully automated data generation for LVLMs.
☆ Learning From Limited Data and Feedback for Cell Culture Process Monitoring: A Comparative Study
Johnny Peng, Thanh Tung Khuat, Ellen Otte, Katarzyna Musial, Bogdan Gabrys
In cell culture bioprocessing, real-time batch process monitoring (BPM) refers to the continuous tracking and analysis of key process variables such as viable cell density, nutrient levels, metabolite concentrations, and product titer throughout the duration of a batch run. This enables early detection of deviations and supports timely control actions to ensure optimal cell growth and product quality. BPM plays a critical role in ensuring the quality and regulatory compliance of biopharmaceutical manufacturing processes. However, the development of accurate soft sensors for BPM is hindered by key challenges, including limited historical data, infrequent feedback, heterogeneous process conditions, and high-dimensional sensory inputs. This study presents a comprehensive benchmarking analysis of machine learning (ML) methods designed to address these challenges, with a focus on learning from historical data with limited volume and relevance in the context of bioprocess monitoring. We evaluate multiple ML approaches including feature dimensionality reduction, online learning, and just-in-time learning across three datasets, one in silico dataset and two real-world experimental datasets. Our findings highlight the importance of training strategies in handling limited data and feedback, with batch learning proving effective in homogeneous settings, while just-in-time learning and online learning demonstrate superior adaptability in cold-start scenarios. Additionally, we identify key meta-features, such as feed media composition and process control strategies, that significantly impact model transferability. The results also suggest that integrating Raman-based predictions with lagged offline measurements enhances monitoring accuracy, offering a promising direction for future bioprocess soft sensor development.
comment : This is a pre-print for submitting to computers & chemical engineering journal
☆ Think Before You Drive: Wo rl d Model-Inspired Multimodal Grounding for Autonomous Vehicles
Haicheng Liao, Huanming Shen, Bonan Wang, Yongkang Li, Yihong Tang, Chengyue Wang, Dingyi Zhuang, Kehua Chen, Hai Yang, Chengzhong Xu, Zhenning Li
Interpreting natural-language commands to localize target objects is critical for autonomous driving (AD). Existing visual grounding (VG) methods for autonomous vehicles (AVs) typically struggle with ambiguous, context-dependent instructions, as they lack reasoning over 3D spatial relations and anticipated scene evolution. Grounded in the principles of world models, we propose ThinkDeeper, a framework that reasons about future spatial states before making grounding decisions. At its core is a Spatial-Aware World Model (SA-WM) that learns to reason ahead by distilling the current scene into a command-aware latent state and rolling out a sequence of future latent states, providing forward-looking cues for disambiguation. Complementing this, a hypergraph-guided decoder then hierarchically fuses these states with the multimodal input, capturing higher-order spatial dependencies for robust localization. In addition, we present DrivePilot, a multi-source VG dataset in AD, featuring semantic annotations generated by a Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT)-prompted LLM pipeline. Extensive evaluations on six benchmarks, ThinkDeeper ranks #1 on the Talk2Car leaderboard and surpasses state-of-the-art baselines on DrivePilot, MoCAD, and RefCOCO/+/g benchmarks. Notably, it shows strong robustness and efficiency in challenging scenes (long-text, multi-agent, ambiguity) and retains superior performance even when trained on 50% of the data.
☆ GalaxyDiT: Efficient Video Generation with Guidance Alignment and Adaptive Proxy in Diffusion Transformer s
Zhiye Song, Steve Dai, Ben Keller, Brucek Khailany
Diffusion models have revolutionized video generation, becoming essential tools in creative content generation and physical simulation. Transformer-based architectures (DiTs) and classifier-free guidance (CFG) are two cornerstones of this success, enabling strong prompt adherence and realistic video quality. Despite their versatility and superior performance, these models require intensive computation. Each video generation requires dozens of iterative steps, and CFG doubles the required compute. This inefficiency hinders broader adoption in downstream applications. We introduce GalaxyDiT, a training-free method to accelerate video generation with guidance alignment and systematic proxy selection for reuse metrics. Through rank-order correlation analysis, our technique identifies the optimal proxy for each video model, across model families and parameter scales, thereby ensuring optimal computational reuse. We achieve $1.87\times$ and $2.37\times$ speedup on Wan2.1-1.3B and Wan2.1-14B with only 0.97% and 0.72% drops on the VBench-2.0 benchmark. At high speedup rates, our approach maintains superior fidelity to the base model, exceeding prior state-of-the-art approaches by 5 to 10 dB in peak signal-to-noise ratio (PSNR).
☆ Multi-Aspect Knowledge-Enhanced Medical Vision -Language Pretrain ing with Multi- Agent Data Generation
Xieji Li, Siyuan Yan, Yingsheng Liu, H. Peter Soyer, Monika Janda, Victoria Mar, Zongyuan Ge
Vision-language pretraining (VLP) has emerged as a powerful paradigm in medical image analysis, enabling representation learning from large-scale image-text pairs without relying on expensive manual annotations. However, existing methods often struggle with the noise inherent in web-collected data and the complexity of unstructured long medical texts. To address these challenges, we propose a novel VLP framework integrating a Multi-Agent data GENeration (MAGEN) system and Ontology-based Multi-Aspect Knowledge-Enhanced (O-MAKE) pretraining. First, MAGEN enhances data quality by synthesizing knowledge-enriched descriptions via a foundation model-assisted captioning and retrieval-based verification pipeline. Second, O-MAKE addresses the difficulty of learning from long, unstructured texts by decomposing them into distinct knowledge aspects. This facilitates fine-grained alignment at both global and patch levels, while explicitly modeling medical concept relationships through ontology-guided mechanisms. We validate our framework in the field of dermatology, where comprehensive experiments demonstrate the effectiveness of each component. Our approach achieves state-of-the-art zero-shot performance on disease classification and cross-modal retrieval tasks across eight datasets. Our code and the augmented dataset Derm1M-AgentAug, comprising over 400k skin-image-text pairs, will be released at https://github.com/SiyuanYan1/Derm1M.
comment : 10 pages. Under Review
☆ Multimodal Reinforcement Learning with Agent ic Verifier for AI Agent s
Reuben Tan, Baolin Peng, Zhengyuan Yang, Hao Cheng, Oier Mees, Theodore Zhao, Andrea Tupini, Isar Meijier, Qianhui Wu, Yuncong Yang, Lars Liden, Yu Gu, Sheng Zhang, Xiaodong Liu, Lijuan Wang, Marc Pollefeys, Yong Jae Lee, Jianfeng Gao
Agentic reasoning models trained with multimodal reinforcement learning (MMRL) have become increasingly capable, yet they are almost universally optimized using sparse, outcome-based rewards computed based on the final answers. Richer rewards computed from the reasoning tokens can improve learning significantly by providing more fine-grained guidance. However, it is challenging to compute more informative rewards in MMRL beyond those based on outcomes since different samples may require different scoring functions and teacher models may provide noisy reward signals too. In this paper, we introduce the Argos (Agentic Reward for Grounded & Objective Scoring), a principled reward agent to train multimodal reasoning models for agentic tasks. For each sample, Argos selects from a pool of teacher-model derived and rule-based scoring functions to simultaneously evaluate: (i) final response accuracy, (ii) spatiotemporal localization of referred entities and actions, and (iii) the quality of the reasoning process. We find that by leveraging our agentic verifier across both SFT data curation and RL training, our model achieves state-of-the-art results across multiple agentic tasks such as spatial reasoning, visual hallucination as well as robotics and embodied AI benchmarks. Critically, we demonstrate that just relying on SFT post-training on highly curated reasoning data is insufficient, as agents invariably collapse to ungrounded solutions during RL without our online verification. We also show that our agentic verifier can help to reduce reward-hacking in MMRL. Finally, we also provide a theoretical justification for the effectiveness of Argos through the concept of pareto-optimality.
☆ Wo rl d Models for Autonomous Navigation of Terrestrial Robots from LIDAR Observations
Raul Steinmetz, Fabio Demo Rosa, Victor Augusto Kich, Jair Augusto Bottega, Ricardo Bedin Grando, Daniel Fernando Tello Gamarra
Autonomous navigation of terrestrial robots using Reinforcement Learning (RL) from LIDAR observations remains challenging due to the high dimensionality of sensor data and the sample inefficiency of model-free approaches. Conventional policy networks struggle to process full-resolution LIDAR inputs, forcing prior works to rely on simplified observations that reduce spatial awareness and navigation robustness. This paper presents a novel model-based RL framework built on top of the DreamerV3 algorithm, integrating a Multi-Layer Perceptron Variational Autoencoder (MLP-VAE) within a world model to encode high-dimensional LIDAR readings into compact latent representations. These latent features, combined with a learned dynamics predictor, enable efficient imagination-based policy optimization. Experiments on simulated TurtleBot3 navigation tasks demonstrate that the proposed architecture achieves faster convergence and higher success rate compared to model-free baselines such as SAC, DDPG, and TD3. It is worth emphasizing that the DreamerV3-based agent attains a 100% success rate across all evaluated environments when using the full dataset of the Turtlebot3 LIDAR (360 readings), while model-free methods plateaued below 85%. These findings demonstrate that integrating predictive world models with learned latent representations enables more efficient and robust navigation from high-dimensional sensory data.
comment : Accepted for publication in the Journal of Intelligent and Fuzzy Systems
☆ BookRAG: A Hierarchical Structure-aware Index-based Approach for Retrieval-Augmented Generation on Complex Document s
Shu Wang, Yingli Zhou, Yixiang Fang
As an effective method to boost the performance of Large Language Models (LLMs) on the question answering (QA) task, Retrieval-Augmented Generation (RAG), which queries highly relevant information from external complex documents, has attracted tremendous attention from both industry and academia. Existing RAG approaches often focus on general documents, and they overlook the fact that many real-world documents (such as books, booklets, handbooks, etc.) have a hierarchical structure, which organizes their content from different granularity levels, leading to poor performance for the QA task. To address these limitations, we introduce BookRAG, a novel RAG approach targeted for documents with a hierarchical structure, which exploits logical hierarchies and traces entity relations to query the highly relevant information. Specifically, we build a novel index structure, called BookIndex, by extracting a hierarchical tree from the document, which serves as the role of its table of contents, using a graph to capture the intricate relationships between entities, and mapping entities to tree nodes. Leveraging the BookIndex, we then propose an agent-based query method inspired by the Information Foraging Theory, which dynamically classifies queries and employs a tailored retrieval workflow. Extensive experiments on three widely adopted benchmarks demonstrate that BookRAG achieves state-of-the-art performance, significantly outperforming baselines in both retrieval recall and QA accuracy while maintaining competitive efficiency.
☆ Better Wo rl d Models Can Lead to Better Post-Training Performance
Prakhar Gupta, Henry Conklin, Sarah-Jane Leslie, Andrew Lee
In this work we study how explicit world-modeling objectives affect the internal representations and downstream capability of Transformers across different training stages. We use a controlled 2x2x2 Rubik's Cube and ask: (1) how does explicitly pretraining a world model affect the model's latent representations, and (2) how does world-model quality affect the model's performance after reinforcement learning post-training? We compare standard next-token prediction to two explicit world-modeling strategies -- (i) state-prediction pretraining and (ii) a joint state-prediction + next-token objective -- and assess task performance after Group Relative Policy Optimization (GRPO) is applied as post-training. We evaluate the representation quality with linear probes and causal interventions. We find that explicit world-modeling yields more linearly decodable and causally steerable state representations. More importantly, we find that improved state representations lead to higher gains for GRPO, especially on harder cube states. Our results indicate that sharpening state representations can improve the effectiveness of post-training for sequence-planning tasks.
☆ VS-Graph: Scalable and Efficient Graph Classification Using Hyperdimensional Computing
Hamed Poursiami, Shay Snyder, Guojing Cong, Thomas Potok, Maryam Parsa
Graph classification is a fundamental task in domains ranging from molecular property prediction to materials design. While graph neural networks (GNNs) achieve strong performance by learning expressive representations via message passing, they incur high computational costs, limiting their scalability and deployment on resource-constrained devices. Hyperdimensional Computing (HDC), also known as Vector Symbolic Architectures (VSA), offers a lightweight, brain-inspired alternative, yet existing HDC-based graph methods typically struggle to match the predictive performance of GNNs. In this work, we propose VS-Graph, a vector-symbolic graph learning framework that narrows the gap between the efficiency of HDC and the expressive power of message passing. VS-Graph introduces a Spike Diffusion mechanism for topology-driven node identification and an Associative Message Passing scheme for multi-hop neighborhood aggregation entirely within the high-dimensional vector space. Without gradient-based optimization or backpropagation, our method achieves competitive accuracy with modern GNNs, outperforming the prior HDC baseline by 4-5% on standard benchmarks such as MUTAG and DD. It also matches or exceeds the performance of the GNN baselines on several datasets while accelerating the training by a factor of up to 450x. Furthermore, VS-Graph maintains high accuracy even with the hypervector dimensionality reduced to D=128, demonstrating robustness under aggressive dimension compression and paving the way for ultra-efficient execution on edge and neuromorphic hardware.
☆ UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs
Hung-Yueh Chiang, Chi-Chih Chang, Yu-Chen Lu, Chien-Yu Lin, Kai-Chiang Wu, Mohamed S. Abdelfattah, Diana Marculescu
Deploying large language model (LLM) models on mobile platforms faces significant challenges due to the limited memory and shared computational resources of the device. Resource availability may be an issue as it is directly impacted by the current device workload, adding to the uncertainty of model deployment. We introduce UniQL, a unified post-training quantization and low-rank compression framework with on-device configurable pruning rates for edge LLMs. UniQL is a general framework that integrates quantization and low-rank compression for Transformers, State Space Models (SSMs), and hybrid models to support diverse edge applications. In our proposed joint framework, we introduce an efficient structured weight-sorting method that speeds up computation by 20x, quantization-aware singular value decomposition (SVD) to minimize quantization errors, state-aware weight sorting for SSMs, and a fused rotary positional embedding (RoPE) kernel for pruned models. Our framework performs weight-sorting, fine-tuning, and quantization in the cloud in a single-pass workflow, while enabling on-device configurable pruning rates up to 35%. Our experiments show that quantized and pruned models achieve a memory reduction of 4x-5.7x and a token-throughput improvement of 2.7x-3.4x, maintaining accuracy within 5% of the original models at 15% pruning across Transformers (Llama3 and Qwen2.5), SSMs (Mamba2), and hybrid models (Nemotron-H and Bamba-v2). The code and quantized models are available at: https://github.com/enyac-group/UniQL.
☆ FireSentry: A Multi-Modal Spatio-temporal Benchmark Dataset for Fine-Grained Wildfire Spread Forecasting
Nan Zhou, Huandong Wang, Jiahao Li, Han Li, Yali Song, Qiuhua Wang, Yong Li, Xinlei Chen
Fine-grained wildfire spread prediction is crucial for enhancing emergency response efficacy and decision-making precision. However, existing research predominantly focuses on coarse spatiotemporal scales and relies on low-resolution satellite data, capturing only macroscopic fire states while fundamentally constraining high-precision localized fire dynamics modeling capabilities. To bridge this gap, we present FireSentry, a provincial-scale multi-modal wildfire dataset characterized by sub-meter spatial and sub-second temporal resolution. Collected using synchronized UAV platforms, FireSentry provides visible and infrared video streams, in-situ environmental measurements, and manually validated fire masks. Building on FireSentry, we establish a comprehensive benchmark encompassing physics-based, data-driven, and generative models, revealing the limitations of existing mask-only approaches. Our analysis proposes FiReDiff, a novel dual-modality paradigm that first predicts future video sequences in the infrared modality, and then precisely segments fire masks in the mask modality based on the generated dynamics. FiReDiff achieves state-of-the-art performance, with video quality gains of 39.2% in PSNR, 36.1% in SSIM, 50.0% in LPIPS, 29.4% in FVD, and mask accuracy gains of 3.3% in AUPRC, 59.1% in F1 score, 42.9% in IoU, and 62.5% in MSE when applied to generative models. The FireSentry benchmark dataset and FiReDiff paradigm collectively advance fine-grained wildfire forecasting and dynamic disaster simulation. The processed benchmark dataset is publicly available at: https://github.com/Munan222/FireSentry-Benchmark-Dataset.
☆ HalluGen: Synthesizing Realistic and Controllable Hallucinations for Evaluating Image Restoration
Seunghoi Kim, Henry F. J. Tregidgo, Chen Jin, Matteo Figini, Daniel C. Alexander
Generative models are prone to hallucinations: plausible but incorrect structures absent in the ground truth. This issue is problematic in image restoration for safety-critical domains such as medical imaging, industrial inspection, and remote sensing, where such errors undermine reliability and trust. For example, in low-field MRI, widely used in resource-limited settings, restoration models are essential for enhancing low-quality scans, yet hallucinations can lead to serious diagnostic errors. Progress has been hindered by a circular dependency: evaluating hallucinations requires labeled data, yet such labels are costly and subjective. We introduce HalluGen, a diffusion-based framework that synthesizes realistic hallucinations with controllable type, location, and severity, producing perceptually realistic but semantically incorrect outputs (segmentation IoU drops from 0.86 to 0.36). Using HalluGen, we construct the first large-scale hallucination dataset comprising 4,350 annotated images derived from 1,450 brain MR images for low-field enhancement, enabling systematic evaluation of hallucination detection and mitigation. We demonstrate its utility in two applications: (1) benchmarking image quality metrics and developing Semantic Hallucination Assessment via Feature Evaluation (SHAFE), a feature-based metric with soft-attention pooling that improves hallucination sensitivity over traditional metrics; and (2) training reference-free hallucination detectors that generalize to real restoration failures. Together, HalluGen and its open dataset establish the first scalable foundation for evaluating hallucinations in safety-critical image restoration.
☆ Idea-Gated Transformer s: Enforcing Semantic Coherence via Differentiable Vocabulary Pruning
Darshan Fofadiya
Autoregressive Language Models (LLMs) trained on Next-Token Prediction (NTP) often suffer from ``Topic Drift'' where the generation wanders away from the initial prompt due to a reliance on local associations rather than global planning \citep{holtzman2019curious}. While scaling model size mitigates this \citep{brown2020language}, the fundamental myopia of the NTP objective remains. In this work, we introduce the Idea-Gated Transformer, a novel architecture that separates semantic planning from syntactic generation. We introduce an auxiliary ``Idea Head'' trained to predict the bag-of-words distribution for a future context window, creating a latent ``Concept Vector'' that actively gates the main vocabulary during generation. We propose a differentiable gating mechanism that suppresses semantically irrelevant tokens, effectively pruning the search space in real-time. Experiments on WikiText-103 demonstrate that while the Idea-Gated model achieves comparable validation perplexity to a standard GPT-2 baseline, it exhibits significantly superior Domain Retention. Qualitative and quantitative analysis reveals that the gating mechanism successfully locks generation into specific semantic clusters (e.g., Finance, Science) and resists associative drift, offering a parameter-efficient path toward more controllable language modeling.
comment : Code available at https://github.com/DarshanFofadiya/idea-gated-transformers/tree/main
☆ ProtoEFNet: Dynamic Prototype Learning for Inherently Interpretable Ejection Fraction Estimation in Echocardiography MICCAI 2025
Yeganeh Ghamary, Victoria Wu, Hooman Vaseli, Christina Luong, Teresa Tsang, Siavash Bigdeli, Purang Abolmaesumi
Ejection fraction (EF) is a crucial metric for assessing cardiac function and diagnosing conditions such as heart failure. Traditionally, EF estimation requires manual tracing and domain expertise, making the process time-consuming and subject to interobserver variability. Most current deep learning methods for EF prediction are black-box models with limited transparency, which reduces clinical trust. Some post-hoc explainability methods have been proposed to interpret the decision-making process after the prediction is made. However, these explanations do not guide the model's internal reasoning and therefore offer limited reliability in clinical applications. To address this, we introduce ProtoEFNet, a novel video-based prototype learning model for continuous EF regression. The model learns dynamic spatiotemporal prototypes that capture clinically meaningful cardiac motion patterns. Additionally, the proposed Prototype Angular Separation (PAS) loss enforces discriminative representations across the continuous EF spectrum. Our experiments on the EchonetDynamic dataset show that ProtoEFNet can achieve accuracy on par with its non-interpretable counterpart while providing clinically relevant insight. The ablation study shows that the proposed loss boosts performance with a 2% increase in F1 score from 77.67$\pm$2.68 to 79.64$\pm$2.10. Our source code is available at: https://github.com/DeepRCL/ProtoEF
comment : 11 pages, Accepted in IMIMIC Workshop at MICCAI 2025
☆ Single-Round Scalable Analytic Federated Learning
Alan T. L. Bacellar, Mustafa Munir, Felipe M. G. França, Priscila M. V. Lima, Radu Marculescu, Lizy K. John
Federated Learning (FL) is plagued by two key challenges: high communication overhead and performance collapse on heterogeneous (non-IID) data. Analytic FL (AFL) provides a single-round, data distribution invariant solution, but is limited to linear models. Subsequent non-linear approaches, like DeepAFL, regain accuracy but sacrifice the single-round benefit. In this work, we break this trade-off. We propose SAFLe, a framework that achieves scalable non-linear expressivity by introducing a structured head of bucketed features and sparse, grouped embeddings. We prove this non-linear architecture is mathematically equivalent to a high-dimensional linear regression. This key equivalence allows SAFLe to be solved with AFL's single-shot, invariant aggregation law. Empirically, SAFLe establishes a new state-of-the-art for analytic FL, significantly outperforming both linear AFL and multi-round DeepAFL in accuracy across all benchmarks, demonstrating a highly efficient and scalable solution for federated vision.
☆ Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs
Ngoc Bui, Shubham Sharma, Simran Lamba, Saumitra Mishra, Rex Ying
Memory and computation remain core bottlenecks in long-horizon LLM inference due to the quadratic cost of self-attention and the ever-growing key-value (KV) cache. Existing strategies for memory-bounded inference, such as quantization, offloading, or heuristic KV eviction, either incur high orchestration costs or rely on unreliable attention-based proxies of importance. We propose TRIM-KV, a novel approach that learns each token's intrinsic importance at creation time via a lightweight retention gate. Each gate predicts a scalar retention score that decays over time, reflecting the long-term utility of the token for a specific layer and head. Tokens with low scores are evicted when the memory budget is exceeded, ensuring that the cache always contains the most critical tokens. TRIM-KV is trained efficiently through distillation from a frozen LLM combined with a capacity loss, requiring only gate fine-tuning and adding negligible inference overhead. Across mathematical reasoning (GSM8K, MATH-500, AIME24), procedural generation (LongProc), conversational long-memory benchmarks (LongMemEval), and long-context understanding (LongBench and SCBench), TRIM-KV consistently outperforms strong eviction and learnable retrieval baselines, especially in low-memory regimes. Remarkably, it even surpasses full-cache models in some settings, showing that selective retention can serve as a form of regularization, suppressing noise from uninformative tokens. Qualitative analyses further reveal that learned retention scores align with human intuition, naturally recovering heuristics such as sink tokens, sliding windows, and gist compression without explicit design. Beyond efficiency, retention scores provide insights into layer- and head-specific roles, suggesting a new path toward LLM interpretability.
☆ Evaluating Generalization Capabilities of LLM-Based Agent s in Mixed-Motive Scenarios Using Concordia NeurIPS
Chandler Smith, Marwa Abdulhai, Manfred Diaz, Marko Tesic, Rakshit S. Trivedi, Alexander Sasha Vezhnevets, Lewis Hammond, Jesse Clifton, Minsuk Chang, Edgar A. Duéñez-Guzmán, John P. Agapiou, Jayd Matyas, Danny Karmon, Akash Kundu, Aliaksei Korshuk, Ananya Ananya, Arrasy Rahman, Avinaash Anand Kulandaivel, Bain McHale, Beining Zhang, Buyantuev Alexander, Carlos Saith Rodriguez Rojas, Caroline Wang, Chetan Talele, Chenao Liu, Chichen Lin, Diana Riazi, Di Yang Shi, Emanuel Tewolde, Elizaveta Tennant, Fangwei Zhong, Fuyang Cui, Gang Zhao, Gema Parreño Piqueras, Hyeonggeun Yun, Ilya Makarov, Jiaxun Cui, Jebish Purbey, Jim Dilkes, Jord Nguyen, Lingyun Xiao, Luis Felipe Giraldo, Manuela Chacon-Chamorro, Manuel Sebastian Rios Beltran, Marta Emili García Segura, Mengmeng Wang, Mogtaba Alim, Nicanor Quijano, Nico Schiavone, Olivia Macmillan-Scott, Oswaldo Peña, Peter Stone, Ram Mohan Rao Kadiyala, Rolando Fernandez, Ruben Manrique, Sunjia Lu, Sheila A. McIlraith, Shamika Dhuri, Shuqing Shi, Siddhant Gupta, Sneheel Sarangi, Sriram Ganapathi Subramanian, Taehun Cha, Toryn Q. Klassen, Wenming Tu, Weijian Fan, Wu Ruiyang, Xue Feng, Yali Du, Yang Liu, Yiding Wang, Yipeng Kang, Yoonchang Sung, Yuxuan Chen, Zhaowei Zhang, Zhihan Wang, Zhiqiang Wu, Ziang Chen, Zilong Zheng, Zixia Jia, Ziyan Wang, Dylan Hadfield-Menell, Natasha Jaques, Tim Baarslag, Jose Hernandez-Orallo, Joel Z. Leibo
Large Language Model (LLM) agents have demonstrated impressive capabilities for social interaction and are increasingly being deployed in situations where they might engage with both human and artificial agents. These interactions represent a critical frontier for LLM-based agents, yet existing evaluation methods fail to measure how well these capabilities generalize to novel social situations. In this paper, we introduce a method for evaluating the ability of LLM-based agents to cooperate in zero-shot, mixed-motive environments using Concordia, a natural language multi-agent simulation environment. Our method measures general cooperative intelligence by testing an agent's ability to identify and exploit opportunities for mutual gain across diverse partners and contexts. We present empirical results from the NeurIPS 2024 Concordia Contest, where agents were evaluated on their ability to achieve mutual gains across a suite of diverse scenarios ranging from negotiation to collective action problems. Our findings reveal significant gaps between current agent capabilities and the robust generalization required for reliable cooperation, particularly in scenarios demanding persuasion and norm enforcement.
comment : Published at NeurIPS Datasets and Benchmarks 2025, 10 pages
☆ NavMapFusion: Diffusion -based Fusion of Navigation Maps for Online Vectorized HD Map Construction WACV 2026
Thomas Monninger, Zihan Zhang, Steffen Staab, Sihao Ding
Accurate environmental representations are essential for autonomous driving, providing the foundation for safe and efficient navigation. Traditionally, high-definition (HD) maps are providing this representation of the static road infrastructure to the autonomous system a priori. However, because the real world is constantly changing, such maps must be constructed online from on-board sensor data. Navigation-grade standard-definition (SD) maps are widely available, but their resolution is insufficient for direct deployment. Instead, they can be used as coarse prior to guide the online map construction process. We propose NavMapFusion, a diffusion-based framework that performs iterative denoising conditioned on high-fidelity sensor data and on low-fidelity navigation maps. This paper strives to answer: (1) How can coarse, potentially outdated navigation maps guide online map construction? (2) What advantages do diffusion models offer for map fusion? We demonstrate that diffusion-based map construction provides a robust framework for map fusion. Our key insight is that discrepancies between the prior map and online perception naturally correspond to noise within the diffusion process; consistent regions reinforce the map construction, whereas outdated segments are suppressed. On the nuScenes benchmark, NavMapFusion conditioned on coarse road lines from OpenStreetMap data reaches a 21.4% relative improvement on 100 m, and even stronger improvements on larger perception ranges, while maintaining real-time capabilities. By fusing low-fidelity priors with high-fidelity sensor data, the proposed method generates accurate and up-to-date environment representations, guiding towards safer and more reliable autonomous driving. The code is available at https://github.com/tmonnin/navmapfusion
comment : Accepted to 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2026)
☆ A Conceptual Model for AI Adoption in Financial Decision-Making: Addressing the Unique Challenges of Small and Medium-Sized Enterprises
Manh Chien Vu, Thang Le Dinh, Manh Chien Vu, Tran Duc Le, Thi Lien Huong Nguyen
The adoption of artificial intelligence (AI) offers transformative potential for small and medium-sized enterprises (SMEs), particularly in enhancing financial decision-making processes. However, SMEs often face significant barriers to implementing AI technologies, including limited resources, technical expertise, and data management capabilities. This paper presents a conceptual model for the adoption of AI in financial decision-making for SMEs. The proposed model addresses key challenges faced by SMEs, including limited resources, technical expertise, and data management capabilities. The model is structured into layers: data sources, data processing and integration, AI model deployment, decision support and automation, and validation and risk management. By implementing AI incrementally, SMEs can optimize financial forecasting, budgeting, investment strategies, and risk management. This paper highlights the importance of data quality and continuous model validation, providing a practical roadmap for SMEs to integrate AI into their financial operations. The study concludes with implications for SMEs adopting AI-driven financial processes and suggests areas for future research in AI applications for SME finance.
comment : The Eighth International Econometric and Financial Conference of Vietnam - ECONVN2025, Ho Chi Minh City, Vietnam, January 13-14-15, 2025
☆ RGE-GCN: Recursive Gene Elimination with Graph Convolutional Networks for RNA-seq based Ea rl y Cancer Detection
Shreyas Shende, Varsha Narayanan, Vishal Fenn, Yiran Huang, Dincer Goksuluk, Gaurav Choudhary, Melih Agraz, Mengjia Xu
Early detection of cancer plays a key role in improving survival rates, but identifying reliable biomarkers from RNA-seq data is still a major challenge. The data are high-dimensional, and conventional statistical methods often fail to capture the complex relationships between genes. In this study, we introduce RGE-GCN (Recursive Gene Elimination with Graph Convolutional Networks), a framework that combines feature selection and classification in a single pipeline. Our approach builds a graph from gene expression profiles, uses a Graph Convolutional Network to classify cancer versus normal samples, and applies Integrated Gradients to highlight the most informative genes. By recursively removing less relevant genes, the model converges to a compact set of biomarkers that are both interpretable and predictive. We evaluated RGE-GCN on synthetic data as well as real-world RNA-seq cohorts of lung, kidney, and cervical cancers. Across all datasets, the method consistently achieved higher accuracy and F1-scores than standard tools such as DESeq2, edgeR, and limma-voom. Importantly, the selected genes aligned with well-known cancer pathways including PI3K-AKT, MAPK, SUMOylation, and immune regulation. These results suggest that RGE-GCN shows promise as a generalizable approach for RNA-seq based early cancer detection and biomarker discovery (https://rce-gcn.streamlit.app/ ).
comment : 12 pages, 2 figures
☆ DAComp: Benchmark ing Data Agent s across the Full Data Intelligence Lifecycle
Fangyu Lei, Jinxiang Meng, Yiming Huang, Junjie Zhao, Yitong Zhang, Jianwen Luo, Xin Zou, Ruiyi Yang, Wenbo Shi, Yan Gao, Shizhu He, Zuo Wang, Qian Liu, Yang Wang, Ke Wang, Jun Zhao, Kang Liu
Real-world enterprise data intelligence workflows encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-oriented insights. We introduce DAComp, a benchmark of 210 tasks that mirrors these complex workflows. Data engineering (DE) tasks require repository-level engineering on industrial schemas, including designing and building multi-stage SQL pipelines from scratch and evolving existing systems under evolving requirements. Data analysis (DA) tasks pose open-ended business problems that demand strategic planning, exploratory analysis through iterative coding, interpretation of intermediate results, and the synthesis of actionable recommendations. Engineering tasks are scored through execution-based, multi-metric evaluation. Open-ended tasks are assessed by a reliable, experimentally validated LLM-judge, which is guided by hierarchical, meticulously crafted rubrics. Our experiments reveal that even state-of-the-art agents falter on DAComp. Performance on DE tasks is particularly low, with success rates under 20%, exposing a critical bottleneck in holistic pipeline orchestration, not merely code generation. Scores on DA tasks also average below 40%, highlighting profound deficiencies in open-ended reasoning and demonstrating that engineering and analysis are distinct capabilities. By clearly diagnosing these limitations, DAComp provides a rigorous and realistic testbed to drive the development of truly capable autonomous data agents for enterprise settings. Our data and code are available at https://da-comp.github.io
☆ Bayes-DIC Net: Estimating Digital Image Cor relation Uncertainty with Bayesian Neural Networks
Biao Chen, Zhenhua Lei, Yahui Zhang, Tongzhi Niu
This paper introduces a novel method for generating high-quality Digital Image Correlation (DIC) dataset based on non-uniform B-spline surfaces. By randomly generating control point coordinates, we construct displacement fields that encompass a variety of realistic displacement scenarios, which are subsequently used to generate speckle pattern datasets. This approach enables the generation of a large-scale dataset that capture real-world displacement field situations, thereby enhancing the training and generalization capabilities of deep learning-based DIC algorithms. Additionally, we propose a novel network architecture, termed Bayes-DIC Net, which extracts information at multiple levels during the down-sampling phase and facilitates the aggregation of information across various levels through a single skip connection during the up-sampling phase. Bayes-DIC Net incorporates a series of lightweight convolutional blocks designed to expand the receptive field and capture rich contextual information while minimizing computational costs. Furthermore, by integrating appropriate dropout modules into Bayes-DIC Net and activating them during the network inference stage, Bayes-DIC Net is transformed into a Bayesian neural network. This transformation allows the network to provide not only predictive results but also confidence levels in these predictions when processing real unlabeled datasets. This feature significantly enhances the practicality and reliability of our network in real-world displacement field prediction tasks. Through these innovations, this paper offers new perspectives and methods for dataset generation and algorithm performance enhancement in the field of DIC.
comment : 17 pages, 8 figures
☆ MANTRA: a Framework for Multi-stage Adaptive Noise TReAtment During Training
Zixiao Zhao, Fatemeh H. Fard, Jie JW Wu
The reliable application of deep learning models to software engineering tasks hinges on high-quality training data. Yet, large-scale repositories inevitably introduce noisy or mislabeled examples that degrade both accuracy and robustness. While Noise Label Learning (NLL) has been extensively studied in other fields, there are a few works that investigate NLL in Software Engineering (SE) and Large Language Models (LLMs) for SE tasks. In this work, we propose MANTRA, a Multi-stage Adaptive Noise TReAtment framework that embeds noise diagnosis and mitigation directly into the fine-tuning process of code-Pretrained Language Models (PTM) and code-LLMs. We first investigate the effect of noise at varying levels on convergence and loss trajectories of the models. Then we apply an adaptive dropout strategy guided by per-sample loss dynamics and Gaussian Mixture Model clustering to exclude persistently noisy points while preserving clean data. Applying to code summarization and commit intent classification, our experiments reveal that some LLMs are more sensitive to noise than others. However, with MANTRA, the performance of all models in both tasks is improved. MANTRA enables researchers and practitioners to reduce the impact of errors introduced by the dataset in training, saves time in data cleaning and processing, while maximizing the effect of fine-tuning.
☆ Evaluating Long- Context Reasoning in LLM-Based Web Agent s NeurIPS 25
Andy Chung, Yichi Zhang, Kaixiang Lin, Aditya Rawal, Qiaozi Gao, Joyce Chai
As large language model (LLM)-based agents become increasingly integrated into daily digital interactions, their ability to reason across long interaction histories becomes crucial for providing personalized and contextually aware assistance. However, the performance of these agents in long context scenarios, particularly for action-taking WebAgents operating in realistic web environments, remains largely unexplored. This paper introduces a benchmark for evaluating long context reasoning capabilities of WebAgents through sequentially dependent subtasks that require retrieval and application of information from extended interaction histories. We develop a novel evaluation framework that simulates multi-session user interactions by injecting irrelevant task trajectories between dependent subtasks, creating contexts ranging from 25,000 to 150,000 tokens. Through extensive evaluation of four popular models, Claude-3.7, GPT-4.1, Llama 4, and o4-mini, we observe a dramatic performance degradation as context length increases, with success rates dropping from 40-50\% in baseline conditions to less than 10\% in long context scenarios. Our detailed error analysis reveals that agents primarily fail due to getting stuck in loops and losing track of original task objectives. We further propose an implicit RAG approach that provides modest improvements by generating task-relevant summaries, though fundamental limitations in long context reasoning persist. These findings highlight critical challenges for deploying WebAgents in realistic, long-term user interaction scenarios and provide insights for developing more robust agent architectures capable of maintaining coherent task execution across extended contexts.
comment : Accepted NeurIPS 25 LAW Workshop
☆ Gamma-from-Mono: Road-Relative, Metric, Self-Supervised Monocular Geometry for Vehicular Applications 3DV 2026
Gasser Elazab, Maximilian Jansen, Michael Unterreiner, Olaf Hellwich
Accurate perception of the vehicle's 3D surroundings, including fine-scale road geometry, such as bumps, slopes, and surface irregularities, is essential for safe and comfortable vehicle control. However, conventional monocular depth estimation often oversmooths these features, losing critical information for motion planning and stability. To address this, we introduce Gamma-from-Mono (GfM), a lightweight monocular geometry estimation method that resolves the projective ambiguity in single-camera reconstruction by decoupling global and local structure. GfM predicts a dominant road surface plane together with residual variations expressed by gamma, a dimensionless measure of vertical deviation from the plane, defined as the ratio of a point's height above it to its depth from the camera, and grounded in established planar parallax geometry. With only the camera's height above ground, this representation deterministically recovers metric depth via a closed form, avoiding full extrinsic calibration and naturally prioritizing near-road detail. Its physically interpretable formulation makes it well suited for self-supervised learning, eliminating the need for large annotated datasets. Evaluated on KITTI and the Road Surface Reconstruction Dataset (RSRD), GfM achieves state-of-the-art near-field accuracy in both depth and gamma estimation while maintaining competitive global depth performance. Our lightweight 8.88M-parameter model adapts robustly across diverse camera setups and, to our knowledge, is the first self-supervised monocular approach evaluated on RSRD.
comment : Accepted in 3DV 2026
☆ Towards better dense rewards in Reinforcement Learning Applications
Shuyuan Zhang
Finding meaningful and accurate dense rewards is a fundamental task in the field of reinforcement learning (RL) that enables agents to explore environments more efficiently. In traditional RL settings, agents learn optimal policies through interactions with an environment guided by reward signals. However, when these signals are sparse, delayed, or poorly aligned with the intended task objectives, agents often struggle to learn effectively. Dense reward functions, which provide informative feedback at every step or state transition, offer a potential solution by shaping agent behavior and accelerating learning. Despite their benefits, poorly crafted reward functions can lead to unintended behaviors, reward hacking, or inefficient exploration. This problem is particularly acute in complex or high-dimensional environments where handcrafted rewards are difficult to specify and validate. To address this, recent research has explored a variety of approaches, including inverse reinforcement learning, reward modeling from human preferences, and self-supervised learning of intrinsic rewards. While these methods offer promising directions, they often involve trade-offs between generality, scalability, and alignment with human intent. This proposal explores several approaches to dealing with these unsolved problems and enhancing the effectiveness and reliability of dense reward construction in different RL applications.
comment : arXiv admin note: substantial text overlap with arXiv:2505.20417
☆ Artificial Intelligence Applications in Horizon Scanning for Infectious Diseases
Ian Miles, Mayumi Wakimoto, Wagner Meira, Daniela Paula, Daylene Ticiane, Bruno Rosa, Jane Biddulph, Stelios Georgiou, Valdir Ermida
This review explores the integration of Artificial Intelligence into Horizon Scanning, focusing on identifying and responding to emerging threats and opportunities linked to Infectious Diseases. We examine how AI tools can enhance signal detection, data monitoring, scenario analysis, and decision support. We also address the risks associated with AI adoption and propose strategies for effective implementation and governance. The findings contribute to the growing body of Foresight literature by demonstrating the potential and limitations of AI in Public Health preparedness.
comment : 21 pages, 1 box, 1 figure
☆ Learning Single-Image Super-Resolution in the JPEG Compressed Domain ICIP 2025
Sruthi Srinivasan, Elham Shakibapour, Rajy Rawther, Mehdi Saeedi
Deep learning models have grown increasingly complex, with input data sizes scaling accordingly. Despite substantial advances in specialized deep learning hardware, data loading continues to be a major bottleneck that limits training and inference speed. To address this challenge, we propose training models directly on encoded JPEG features, reducing the computational overhead associated with full JPEG decoding and significantly improving data loading efficiency. While prior works have focused on recognition tasks, we investigate the effectiveness of this approach for the restoration task of single-image super-resolution (SISR). We present a lightweight super-resolution pipeline that operates on JPEG discrete cosine transform (DCT) coefficients in the frequency domain. Our pipeline achieves a 2.6x speedup in data loading and a 2.5x speedup in training, while preserving visual quality comparable to standard SISR approaches.
comment : 7 pages, 4 figures, 2 tables, SEEDS Workshop, ICIP 2025
☆ Bootstrapped Mixed Rewards for RL Post-Training: Injecting Canonical Action Order
Prakhar Gupta, Vaibhav Gupta
Post-training with reinforcement learning (RL) typically optimizes a single scalar objective and ignores structure in how solutions are produced. We ask whether a scalar hint toward a canonical solver ordering, used only during RL post-training, improves performance even when fine-tuned on randomized solution sequences. On Sudoku, we train a Transformer with standard fine-tuning on randomized solving orders, then post-train it with Group Relative Policy Optimization (GRPO) with two rewards: cell accuracy and an ordering reward that increases when the model's emission order aligns with the solver order. To compare signals cleanly, we combine them via fixed mixtures and use a simple bootstrapped scaling to equalize component magnitudes at initialization. Mixed rewards generally outperform cell-only optimization--the best mixture yields substantially higher test accuracy than the fine-tuned-only model trained on random-order and approaches the fine-tuned-only model trained on solver-order sequences in accuracy. These results suggest that coarse ordering signals can steer RL post-training toward solver-order trajectories without modifying supervised data or architecture.
☆ The Geometry of Benchmark s: A New Path Toward AGI
Przemyslaw Chojecki
Benchmarks are the primary tool for assessing progress in artificial intelligence (AI), yet current practice evaluates models on isolated test suites and provides little guidance for reasoning about generality or autonomous self-improvement. Here we introduce a geometric framework in which all psychometric batteries for AI agents are treated as points in a structured moduli space, and agent performance is described by capability functionals over this space. First, we define an Autonomous AI (AAI) Scale, a Kardashev-style hierarchy of autonomy grounded in measurable performance on batteries spanning families of tasks (for example reasoning, planning, tool use and long-horizon control). Second, we construct a moduli space of batteries, identifying equivalence classes of benchmarks that are indistinguishable at the level of agent orderings and capability inferences. This geometry yields determinacy results: dense families of batteries suffice to certify performance on entire regions of task space. Third, we introduce a general Generator-Verifier-Updater (GVU) operator that subsumes reinforcement learning, self-play, debate and verifier-based fine-tuning as special cases, and we define a self-improvement coefficient $κ$ as the Lie derivative of a capability functional along the induced flow. A variance inequality on the combined noise of generation and verification provides sufficient conditions for $κ> 0$. Our results suggest that progress toward artificial general intelligence (AGI) is best understood as a flow on moduli of benchmarks, driven by GVU dynamics rather than by scores on individual leaderboards.
☆ Quantitative Analysis of Technical Debt and Pattern Violation in Large Language Model Architectures
Tyler Slater
As Large Language Models (LLMs) transition from code completion tools to autonomous system architects, their impact on long-term software maintainability remains unquantified. While existing research benchmarks functional correctness (pass@k), this study presents the first empirical framework to measure "Architectural Erosion" and the accumulation of Technical Debt in AI-synthesized microservices. We conducted a comparative pilot study of three state-of-the-art models (GPT-5.1, Claude 4.5 Sonnet, and Llama 3 8B) by prompting them to implement a standardized Book Lending Microservice under strict Hexagonal Architecture constraints. Utilizing Abstract Syntax Tree (AST) parsing, we find that while proprietary models achieve high architectural conformance (0% violation rate for GPT-5.1), open-weights models exhibit critical divergence. Specifically, Llama 3 demonstrated an 80% Architectural Violation Rate, frequently bypassing interface adapters to create illegal circular dependencies between Domain and Infrastructure layers. Furthermore, we identified a phenomenon of "Implementation Laziness," where open-weights models generated 60% fewer Logical Lines of Code (LLOC) than their proprietary counterparts, effectively omitting complex business logic to satisfy token constraints. These findings suggest that without automated architectural linting, utilizing smaller open-weights models for system scaffolding accelerates the accumulation of structural technical debt.
comment : Under review at the Journal of Systems and Software (Special Issue on Impactful Software Architecture)
☆ The Initialization Determines Whether In- Context Learning Is Gradient Descent
Shifeng Xie, Rui Yuan, Simone Rossi, Thomas Hannagan
In-context learning (ICL) in large language models (LLMs) is a striking phenomenon, yet its underlying mechanisms remain only partially understood. Previous work connects linear self-attention (LSA) to gradient descent (GD), this connection has primarily been established under simplified conditions with zero-mean Gaussian priors and zero initialization for GD. However, subsequent studies have challenged this simplified view by highlighting its overly restrictive assumptions, demonstrating instead that under conditions such as multi-layer or nonlinear attention, self-attention performs optimization-like inference, akin to but distinct from GD. We investigate how multi-head LSA approximates GD under more realistic conditions specifically when incorporating non-zero Gaussian prior means in linear regression formulations of ICL. We first extend multi-head LSA embedding matrix by introducing an initial estimation of the query, referred to as the initial guess. We prove an upper bound on the number of heads needed for ICL linear regression setup. Our experiments confirm this result and further observe that a performance gap between one-step GD and multi-head LSA persists. To address this gap, we introduce yq-LSA, a simple generalization of single-head LSA with a trainable initial guess yq. We theoretically establish the capabilities of yq-LSA and provide experimental validation on linear regression tasks, thereby extending the theory that bridges ICL and GD. Finally, inspired by our findings in the case of linear regression, we consider widespread LLMs augmented with initial guess capabilities, and show that their performance is improved on a semantic similarity task.
☆ Catching UX Flaws in Code: Leveraging LLMs to Identify Usability Flaws at the Development Stage
Nolan Platt, Ethan Luchs, Sehrish Nizamani
Usability evaluations are essential for ensuring that modern interfaces meet user needs, yet traditional heuristic evaluations by human experts can be time-consuming and subjective, especially early in development. This paper investigates whether large language models (LLMs) can provide reliable and consistent heuristic assessments at the development stage. By applying Jakob Nielsen's ten usability heuristics to thirty open-source websites, we generated over 850 heuristic evaluations in three independent evaluations per site using a pipeline of OpenAI's GPT-4o. For issue detection, the model demonstrated moderate consistency, with an average pairwise Cohen's Kappa of 0.50 and an exact agreement of 84%. Severity judgments showed more variability: weighted Cohen's Kappa averaged 0.63, but exact agreement was just 56%, and Krippendorff's Alpha was near zero. These results suggest that while GPT-4o can produce internally consistent evaluations, especially for identifying the presence of usability issues, its ability to judge severity varies and requires human oversight in practice. Our findings highlight the feasibility and limitations of using LLMs for early-stage, automated usability testing, and offer a foundation for improving consistency in automated User Experience (UX) evaluation. To the best of our knowledge, our work provides one of the first quantitative inter-rater reliability analyses of automated heuristic evaluation and highlights methods for improving model consistency.
comment : 7 pages. Published in Proceedings of the 2025 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). DOI: 10.1109/VL-HCC65237.2025.00024
☆ Hey GPT -OSS, Looks Like You Got It -- Now Walk Me Through It! An Assessment of the Reasoning Language Models Chain of Thought Mechanism for Digital Forensics
Gaëtan Michelet, Janine Schneider, Aruna Withanage, Frank Breitinger
The use of large language models in digital forensics has been widely explored. Beyond identifying potential applications, research has also focused on optimizing model performance for forensic tasks through fine-tuning. However, limited result explainability reduces their operational and legal usability. Recently, a new class of reasoning language models has emerged, designed to handle logic-based tasks through an `internal reasoning' mechanism. Yet, users typically see only the final answer, not the underlying reasoning. One of these reasoning models is gpt-oss, which can be deployed locally, providing full access to its underlying reasoning process. This article presents the first investigation into the potential of reasoning language models for digital forensics. Four test use cases are examined to assess the usability of the reasoning component in supporting result explainability. The evaluation combines a new quantitative metric with qualitative analysis. Findings show that the reasoning component aids in explaining and validating language model outputs in digital forensics at medium reasoning levels, but this support is often limited, and higher reasoning levels do not enhance response quality.
comment : Accept at DFRWS EU 2026
☆ Fine-Tuning ChemBERTa for Predicting Inhibitory Activity Against TDP1 Using Deep Learning
Baichuan Zeng
Predicting the inhibitory potency of small molecules against Tyrosyl-DNA Phosphodiesterase 1 (TDP1)-a key target in overcoming cancer chemoresistance-remains a critical challenge in early drug discovery. We present a deep learning framework for the quantitative regression of pIC50 values from molecular Simplified Molecular Input Line Entry System (SMILES) strings using fine-tuned variants of ChemBERTa, a pre-trained chemical language model. Leveraging a large-scale consensus dataset of 177,092 compounds, we systematically evaluate two pre-training strategies-Masked Language Modeling (MLM) and Masked Token Regression (MTR)-under stratified data splits and sample weighting to address severe activity imbalance which only 2.1% are active. Our approach outperforms classical baselines Random Predictor in both regression accuracy and virtual screening utility, and has competitive performance compared to Random Forest, achieving high enrichment factor EF@1% 17.4 and precision Precision@1% 37.4 among top-ranked predictions. The resulting model, validated through rigorous ablation and hyperparameter studies, provides a robust, ready-to-deploy tool for prioritizing TDP1 inhibitors for experimental testing. By enabling accurate, 3D-structure-free pIC50 prediction directly from SMILES, this work demonstrates the transformative potential of chemical transformers in accelerating target-specific drug discovery.
☆ MVRoom: Controllable 3D Indoor Scene Generation with Multi-View Diffusion Models
Shaoheng Fang, Chaohui Yu, Fan Wang, Qixing Huang
We introduce MVRoom, a controllable novel view synthesis (NVS) pipeline for 3D indoor scenes that uses multi-view diffusion conditioned on a coarse 3D layout. MVRoom employs a two-stage design in which the 3D layout is used throughout to enforce multi-view consistency. The first stage employs novel representations to effectively bridge the 3D layout and consistent image-based condition signals for multi-view generation. The second stage performs image-conditioned multi-view generation, incorporating a layout-aware epipolar attention mechanism to enhance multi-view consistency during the diffusion process. Additionally, we introduce an iterative framework that generates 3D scenes with varying numbers of objects and scene complexities by recursively performing multi-view generation (MVRoom), supporting text-to-scene generation. Experimental results demonstrate that our approach achieves high-fidelity and controllable 3D scene generation for NVS, outperforming state-of-the-art baseline methods both quantitatively and qualitatively. Ablation studies further validate the effectiveness of key components within our generation pipeline.
☆ Toward Virtuous Reinforcement Learning
Majid Ghasemi, Mark Crowley
This paper critiques common patterns in machine ethics for Reinforcement Learning (RL) and argues for a virtue focused alternative. We highlight two recurring limitations in much of the current literature: (i) rule based (deontological) methods that encode duties as constraints or shields often struggle under ambiguity and nonstationarity and do not cultivate lasting habits, and (ii) many reward based approaches, especially single objective RL, implicitly compress diverse moral considerations into a single scalar signal, which can obscure trade offs and invite proxy gaming in practice. We instead treat ethics as policy level dispositions, that is, relatively stable habits that hold up when incentives, partners, or contexts change. This shifts evaluation beyond rule checks or scalar returns toward trait summaries, durability under interventions, and explicit reporting of moral trade offs. Our roadmap combines four components: (1) social learning in multi agent RL to acquire virtue like patterns from imperfect but normatively informed exemplars; (2) multi objective and constrained formulations that preserve value conflicts and incorporate risk aware criteria to guard against harm; (3) affinity based regularization toward updateable virtue priors that support trait like stability under distribution shift while allowing norms to evolve; and (4) operationalizing diverse ethical traditions as practical control signals, making explicit the value and cultural assumptions that shape ethical RL benchmarks.
☆ CRAFT-E: A Neuro-Symbolic Framework for Embodied Affordance Grounding
Zhou Chen, Joe Lin, Carson Bulgin, Sathyanarayanan N. Aakur
Assistive robots operating in unstructured environments must understand not only what objects are, but what they can be used for. This requires grounding language-based action queries to objects that both afford the requested function and can be physically retrieved. Existing approaches often rely on black-box models or fixed affordance labels, limiting transparency, controllability, and reliability for human-facing applications. We introduce CRAFT-E, a modular neuro-symbolic framework that composes a structured verb-property-object knowledge graph with visual-language alignment and energy-based grasp reasoning. The system generates interpretable grounding paths that expose the factors influencing object selection and incorporates grasp feasibility as an integral part of affordance inference. We further construct a benchmark dataset with unified annotations for verb-object compatibility, segmentation, and grasp candidates, and deploy the full pipeline on a physical robot. CRAFT-E achieves competitive performance in static scenes, ImageNet-based functional retrieval, and real-world trials involving 20 verbs and 39 objects. The framework remains robust under perceptual noise and provides transparent, component-level diagnostics. By coupling symbolic reasoning with embodied perception, CRAFT-E offers an interpretable and customizable alternative to end-to-end models for affordance-grounded object selection, supporting trustworthy decision-making in assistive robotic systems.
comment : 20 pages. 3 figures, 4 tables. Under Review
☆ Addressing Logical Fallacies In Scientific Reasoning From Large Language Models: Towards a Dual-Inference Training Framework
Peter B. Walker, Hannah Davidson, Aiden Foster, Matthew Lienert, Thomas Pardue, Dale Russell
Large Language Models (LLMs) have transformed natural language processing and hold growing promise for advancing science, healthcare, and decision-making. Yet their training paradigms remain dominated by affirmation-based inference, akin to \textit{modus ponens}, where accepted premises yield predicted consequents. While effective for generative fluency, this one-directional approach leaves models vulnerable to logical fallacies, adversarial manipulation, and failures in causal reasoning. This paper makes two contributions. First, it demonstrates how existing LLMs from major platforms exhibit systematic weaknesses when reasoning in scientific domains with negation, counterexamples, or faulty premises \footnote{Code to recreate these experiments are at https://github.com/hannahdavidsoncollege-maker/ScientificReasoningForEnvironment-MedicineWithLLMs. Second, it introduces a dual-reasoning training framework that integrates affirmative generation with structured counterfactual denial. Grounded in formal logic, cognitive science, and adversarial training, this training paradigm formalizes a computational analogue of ``denying the antecedent'' as a mechanism for disconfirmation and robustness. By coupling generative synthesis with explicit negation-aware objectives, the framework enables models that not only affirm valid inferences but also reject invalid ones, yielding systems that are more resilient, interpretable, and aligned with human reasoning.
comment : 12 pages, 5 tables
☆ Educational Cone Model in Embedding Vector Spaces
Yo Ehara
Human-annotated datasets with explicit difficulty ratings are essential in intelligent educational systems. Although embedding vector spaces are widely used to represent semantic closeness and are promising for analyzing text difficulty, the abundance of embedding methods creates a challenge in selecting the most suitable method. This study proposes the Educational Cone Model, which is a geometric framework based on the assumption that easier texts are less diverse (focusing on fundamental concepts), whereas harder texts are more diverse. This assumption leads to a cone-shaped distribution in the embedding space regardless of the embedding method used. The model frames the evaluation of embeddings as an optimization problem with the aim of detecting structured difficulty-based patterns. By designing specific loss functions, efficient closed-form solutions are derived that avoid costly computation. Empirical tests on real-world datasets validated the model's effectiveness and speed in identifying the embedding spaces that are best aligned with difficulty-annotated educational texts.
comment : Accepted to the 33rd International Conference on Computers in Education (ICCE 2025)
☆ Balancing Safety and Helpfulness in Healthcare AI Assistants through Iterative Preference Alignment ML4H 2025
Huy Nghiem, Swetasudha Panda, Devashish Khatwani, Huy V. Nguyen, Krishnaram Kenthapadi, Hal Daumé
Large Language Models (LLMs) are increasingly used in healthcare, yet ensuring their safety and trustworthiness remains a barrier to deployment. Conversational medical assistants must avoid unsafe compliance without over-refusing benign queries. We present an iterative post-deployment alignment framework that applies Kahneman-Tversky Optimization (KTO) and Direct Preference Optimization (DPO) to refine models against domain-specific safety signals. Using the CARES-18K benchmark for adversarial robustness, we evaluate four LLMs (Llama-3B/8B, Meditron-8B, Mistral-7B) across multiple cycles. Our results show up to 42% improvement in safety-related metrics for harmful query detection, alongside interesting trade-offs against erroneous refusals, thereby exposing architecture-dependent calibration biases. We also perform ablation studies to identify when self-evaluation is reliable and when external or finetuned judges are necessary to maximize performance gains. Our findings underscore the importance of adopting best practices that balance patient safety, user trust, and clinical utility in the design of conversational medical assistants.
comment : ML4H 2025 Proceedings, Best Paper Award
☆ Orchestrator Multi- Agent Clinical Decision Support System for Secondary Headache Diagnosis in Primary Care
Xizhi Wu, Nelly Estefanie Garduno-Rapp, Justin F Rousseau, Mounika Thakkallapally, Hang Zhang, Yuelyu Ji, Shyam Visweswaran, Yifan Peng, Yanshan Wang
Unlike most primary headaches, secondary headaches need specialized care and can have devastating consequences if not treated promptly. Clinical guidelines highlight several 'red flag' features, such as thunderclap onset, meningismus, papilledema, focal neurologic deficits, signs of temporal arteritis, systemic illness, and the 'worst headache of their life' presentation. Despite these guidelines, determining which patients require urgent evaluation remains challenging in primary care settings. Clinicians often work with limited time, incomplete information, and diverse symptom presentations, which can lead to under-recognition and inappropriate care. We present a large language model (LLM)-based multi-agent clinical decision support system built on an orchestrator-specialist architecture, designed to perform explicit and interpretable secondary headache diagnosis from free-text clinical vignettes. The multi-agent system decomposes diagnosis into seven domain-specialized agents, each producing a structured and evidence-grounded rationale, while a central orchestrator performs task decomposition and coordinates agent routing. We evaluated the multi-agent system using 90 expert-validated secondary headache cases and compared its performance with a single-LLM baseline across two prompting strategies: question-based prompting (QPrompt) and clinical practice guideline-based prompting (GPrompt). We tested five open-source LLMs (Qwen-30B, GPT-OSS-20B, Qwen-14B, Qwen-8B, and Llama-3.1-8B), and found that the orchestrated multi-agent system with GPrompt consistently achieved the highest F1 scores, with larger gains in smaller models. These findings demonstrate that structured multi-agent reasoning improves accuracy beyond prompt engineering alone and offers a transparent, clinically aligned approach for explainable decision support in secondary headache diagnosis.
☆ Machine Phenomenology: A Simple Equation Classifying Fast Radio Bursts SC
Yang Liu, Yuhao Lu, Rahim Moradi, Bo Yang, Bing Zhang, Wenbin Lin, Yu Wang
This work shows how human physical reasoning can guide machine-driven symbolic regression toward discovering empirical laws from observations. As an example, we derive a simple equation that classifies fast radio bursts (FRBs) into two distinct Gaussian distributions, indicating the existence of two physical classes. This human-AI workflow integrates feature selection, dimensional analysis, and symbolic regression: deep learning first analyzes CHIME Catalog 1 and identifies six independent parameters that collectively provide a complete description of FRBs; guided by Buckingham-$π$ analysis and correlation analysis, humans then construct dimensionless groups; finally, symbolic regression performed by the machine discovers the governing equation. When applied to the newer CHIME Catalog, the equation produces consistent results, demonstrating that it captures the underlying physics. This framework is applicable to a broad range of scientific domains.
comment : 19 pages, 9 figures, 3 tables. Submitted to SCIENCE CHINA Physics, Mechanics & Astronomy
☆ Network of Theseus (like the ship)
Vighnesh Subramaniam, Colin Conwell, Boris Katz, Andrei Barbu, Brian Cheung
A standard assumption in deep learning is that the inductive bias introduced by a neural network architecture must persist from training through inference. The architecture you train with is the architecture you deploy. This assumption constrains the community from selecting architectures that may have desirable efficiency or design properties due to difficulties with optimization. We challenge this assumption with Network of Theseus (NoT), a method for progressively converting a trained, or even untrained, guide network architecture part-by-part into an entirely different target network architecture while preserving the performance of the guide network. At each stage, components in the guide network architecture are incrementally replaced with target architecture modules and aligned via representational similarity metrics. This procedure largely preserves the functionality of the guide network even under substantial architectural changes-for example, converting a convolutional network into a multilayer perceptron, or GPT-2 into a recurrent neural network. By decoupling optimization from deployment, NoT expands the space of viable inference-time architectures, opening opportunities for better accuracy-efficiency tradeoffs and enabling more directed exploration of the architectural design space.
comment : Preprint. 24 pages, 9 figures, 8 tables
☆ BEP: A Binary Error Propagation Algorithm for Binary Neural Networks Training
Luca Colombo, Fabrizio Pittorino, Daniele Zambon, Carlo Baldassi, Manuel Roveri, Cesare Alippi
Binary Neural Networks (BNNs), which constrain both weights and activations to binary values, offer substantial reductions in computational complexity, memory footprint, and energy consumption. These advantages make them particularly well suited for deployment on resource-constrained devices. However, training BNNs via gradient-based optimization remains challenging due to the discrete nature of their variables. The dominant approach, quantization-aware training, circumvents this issue by employing surrogate gradients. Yet, this method requires maintaining latent full-precision parameters and performing the backward pass with floating-point arithmetic, thereby forfeiting the efficiency of binary operations during training. While alternative approaches based on local learning rules exist, they are unsuitable for global credit assignment and for back-propagating errors in multi-layer architectures. This paper introduces Binary Error Propagation (BEP), the first learning algorithm to establish a principled, discrete analog of the backpropagation chain rule. This mechanism enables error signals, represented as binary vectors, to be propagated backward through multiple layers of a neural network. BEP operates entirely on binary variables, with all forward and backward computations performed using only bitwise operations. Crucially, this makes BEP the first solution to enable end-to-end binary training for recurrent neural network architectures. We validate the effectiveness of BEP on both multi-layer perceptrons and recurrent neural networks, demonstrating gains of up to +6.89% and +10.57% in test accuracy, respectively. The proposed algorithm is released as an open-source repository.
☆ OnSight Pathology: A real-time platform-agnostic computational pathology companion for histopathology
Jinzhen Hu, Kevin Faust, Parsa Babaei Zadeh, Adrienn Bourkas, Shane Eaton, Andrew Young, Anzar Alvi, Dimitrios George Oreopoulos, Ameesha Paliwal, Assem Saleh Alrumeh, Evelyn Rose Kamski-Hennekam, Phedias Diamandis
The microscopic examination of surgical tissue remains a cornerstone of disease classification but relies on subjective interpretations and access to highly specialized experts, which can compromise accuracy and clinical care. While emerging breakthroughs in artificial intelligence (AI) offer promise for automated histological analysis, the growing number of proprietary digital pathology solutions has created barriers to real-world deployment. To address these challenges, we introduce OnSight Pathology, a platform-agnostic computer vision software that uses continuous custom screen captures to provide real-time AI inferences to users as they review digital slide images. Accessible as a single, self-contained executable file (https://onsightpathology.github.io/ ), OnSight Pathology operates locally on consumer-grade personal computers without complex software integration, enabling cost-effective and secure deployment in research and clinical workflows. Here we demonstrate the utility of OnSight Pathology using over 2,500 publicly available whole slide images across different slide viewers, as well as cases from our clinical digital pathology setup. The software's robustness is highlighted across routine histopathological tasks, including the classification of common brain tumor types, mitosis detection, and the quantification of immunohistochemical stains. A built-in multi-modal chat assistant provides verifiable descriptions of images, free of rigid class labels, for added quality control. Lastly, we show compatibility with live microscope camera feeds, including from personal smartphones, offering potential for deployment in more analog, inter-operative, and telepathology settings. Together, we highlight how OnSight Pathology can deliver real-time AI inferences across a broad range of pathology pipelines, removing key barriers to the adoption of AI tools in histopathology.
☆ RippleBench: Capturing Ripple Effects Using Existing Knowledge Repositories
Roy Rinberg, Usha Bhalla, Igor Shilov, Flavio P. Calmon, Rohit Gandikota
Targeted interventions on language models, such as unlearning, debiasing, or model editing, are a central method for refining model behavior and keeping knowledge up to date. While these interventions aim to modify specific information within models (e.g., removing virology content), their effects often propagate to related but unintended areas (e.g., allergies); these side-effects are commonly referred to as the ripple effect. In this work, we present RippleBench-Maker, an automatic tool for generating Q&A datasets that allow for the measurement of ripple effects in any model-editing task. RippleBench-Maker builds on a Wikipedia-based RAG pipeline (WikiRAG) to generate multiple-choice questions at varying semantic distances from the target concept (e.g., the knowledge being unlearned). Using this framework, we construct RippleBench-Bio, a benchmark derived from the WMDP (Weapons of Mass Destruction Paper) dataset, a common unlearning benchmark. We evaluate eight state-of-the-art unlearning methods and find that all exhibit non-trivial accuracy drops on topics increasingly distant from the unlearned knowledge, each with distinct propagation profiles. To support ongoing research, we release our codebase for on-the-fly ripple evaluation, along with the benchmark, RippleBench-Bio.
☆ From FLOPs to Footprints: The Resource Cost of Artificial Intelligence
Sophia Falk, Nicholas Kluge Corrêa, Sasha Luccioni, Lisa Biber-Freudenberger, Aimee van Wynsberghe
As computational demands continue to rise, assessing the environmental footprint of AI requires moving beyond energy and water consumption to include the material demands of specialized hardware. This study quantifies the material footprint of AI training by linking computational workloads to physical hardware needs. The elemental composition of the Nvidia A100 SXM 40 GB graphics processing unit (GPU) was analyzed using inductively coupled plasma optical emission spectroscopy, which identified 32 elements. The results show that AI hardware consists of about 90% heavy metals and only trace amounts of precious metals. The elements copper, iron, tin, silicon, and nickel dominate the GPU composition by mass. In a multi-step methodology, we integrate these measurements with computational throughput per GPU across varying lifespans, accounting for the computational requirements of training specific AI models at different training efficiency regimes. Scenario-based analyses reveal that, depending on Model FLOPs Utilization (MFU) and hardware lifespan, training GPT-4 requires between 1,174 and 8,800 A100 GPUs, corresponding to the extraction and eventual disposal of up to 7 tons of toxic elements. Combined software and hardware optimization strategies can reduce material demands: increasing MFU from 20% to 60% lowers GPU requirements by 67%, while extending lifespan from 1 to 3 years yields comparable savings; implementing both measures together reduces GPU needs by up to 93%. Our findings highlight that incremental performance gains, such as those observed between GPT-3.5 and GPT-4, come at disproportionately high material costs. The study underscores the necessity of incorporating material resource considerations into discussions of AI scalability, emphasizing that future progress in AI must align with principles of resource efficiency and environmental responsibility.
☆ Solving N-Queen Problem using Las Vegas Algorithm with State Pruning
Susmita Sharma, Aayush Shrestha, Sitasma Thapa, Prashant Timalsina, Prakash Poudyal
The N-Queens problem, placing all N queens in a N x N chessboard where none attack the other, is a classic problem for constraint satisfaction algorithms. While complete methods like backtracking guarantee a solution, their exponential time complexity makes them impractical for large-scale instances thus, stochastic approaches, such as Las Vegas algorithm, are preferred. While it offers faster approximate solutions, it suffers from significant performance variance due to random placement of queens on the board. This research introduces a hybrid algorithm built on top of the standard Las Vegas framework through iterative pruning, dynamically eliminating invalid placements during the random assignment phase, thus this method effectively reduces the search space. The analysis results that traditional backtracking scales poorly with increasing N. In contrast, the proposed technique consistently generates valid solutions more rapidly, establishing it as a superior alternative to use where a single, timely solution is preferred over completeness. Although large N causes some performance variability, the algorithm demonstrates a highly effective trade-off between computational cost and solution fidelity, making it particularly suited for resource-constrained computing environments.
☆ Artificial Intelligence / Human Intelligence: Who Controls Whom?
Charlotte Jacquemot
Using the example of the film 2001: A Space Odyssey, this chapter illustrates the challenges posed by an AI capable of making decisions that go against human interests. But are human decisions always rational and ethical? In reality, the cognitive decision-making process is influenced by cognitive biases that affect our behavior and choices. AI not only reproduces these biases, but can also exploit them, with the potential to shape our decisions and judgments. Behind IA algorithms, there are sometimes individuals who show little concern for fundamental rights and impose their own rules. To address the ethical and societal challenges raised by AI and its governance, the regulation of digital platforms and education are keys levers. Regulation must reflect ethical, legal, and political choices, while education must strengthen digital literacy and teach people to make informed and critical choices when facing digital technologies.
comment : in French language
♻ ☆ PCS Workflow for Veridical Data Science in the Age of AI
Zachary T. Rewolinski, Bin Yu
Data science is a pillar of artificial intelligence (AI), which is transforming nearly every domain of human activity, from the social and physical sciences to engineering and medicine. While data-driven findings in AI offer unprecedented power to extract insights and guide decision-making, many are difficult or impossible to replicate. A key reason for this challenge is the uncertainty introduced by the many choices made throughout the data science life cycle (DSLC). Traditional statistical frameworks often fail to account for this uncertainty. The Predictability-Computability-Stability (PCS) framework for veridical (truthful) data science offers a principled approach to addressing this challenge throughout the DSLC. This paper presents an updated and streamlined PCS workflow, tailored for practitioners and enhanced with guided use of generative AI. We include a running example to display the PCS framework in action, and conduct a related case study which showcases the uncertainty in downstream predictions caused by judgment calls in the data cleaning stage.
♻ ☆ Sat2Flow: A Structure-Aware Diffusion Framework for Human Flow Generation from Satellite Imagery
Xiangxu Wang, Tianhong Zhao, Wei Tu, Bowen Zhang, Guanzhou Chen, Jinzhou Cao
Origin-Destination (OD) flow matrices are critical for urban mobility analysis, supporting traffic forecasting, infrastructure planning, and policy design. Existing methods face two key limitations: (1) reliance on costly auxiliary features (e.g., Points of Interest, socioeconomic statistics) with limited spatial coverage, and (2) fragility to spatial topology changes, where reordering urban regions disrupts the structural coherence of generated flows. We propose Sat2Flow, a structure-aware diffusion framework that generates structurally coherent OD flows using only satellite imagery. Our approach employs a multi-kernel encoder to capture diverse regional interactions and a permutation-aware diffusion process that maintains consistency across regional orderings. Through joint contrastive training linking satellite features with OD patterns and equivariant diffusion training enforcing structural invariance, Sat2Flow ensures topological robustness under arbitrary regional reindexing. Experiments on real-world datasets show that Sat2Flow outperforms physics-based and data-driven baselines in accuracy while preserving flow distributions and spatial structures under index permutations. Sat2Flow offers a globally scalable solution for OD flow generation in data-scarce environments, eliminating region-specific auxiliary data dependencies while maintaining structural robustness for reliable mobility modeling.
comment : 9 pages, 5 figures
♻ ☆ Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learning
Juan Sebastian Rojas, Chi-Guhn Lee
Continual reinforcement learning (continual RL) seeks to formalize the notions of lifelong learning and endless adaptation in RL. In particular, the aim of continual RL is to develop RL agents that can maintain a careful balance between retaining useful information and adapting to new situations. To date, continual RL has been explored almost exclusively through the lens of risk-neutral decision-making, in which the agent aims to optimize the expected long-run performance. In this work, we present the first formal theoretical treatment of continual RL through the lens of risk-aware decision-making, in which the behaviour of the agent is directed towards optimizing a measure of long-run performance beyond the mean. In particular, we show that the classical theory of risk measures, widely used as a theoretical foundation in non-continual risk-aware RL, is, in its current form, incompatible with continual learning. Then, building on this insight, we extend risk measure theory into the continual setting by introducing a new class of ergodic risk measures that are compatible with continual learning. Finally, we provide a case study of risk-aware continual learning, along with empirical results, which show the intuitive appeal of ergodic risk measures in continual settings.
♻ ☆ SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control
Yuxuan Mu, Ziyu Zhang, Yi Shi, Minami Matsumoto, Kotaro Imamura, Guy Tevet, Chuan Guo, Michael Taylor, Chang Shu, Pengcheng Xi, Xue Bin Peng
Data-driven motion priors that can guide agents toward producing naturalistic behaviors play a pivotal role in creating life-like virtual characters. Adversarial imitation learning has been a highly effective method for learning motion priors from reference motion data. However, adversarial priors, with few exceptions, need to be retrained for each new controller, thereby limiting their reusability and necessitating the retention of the reference motion data when training on downstream tasks. In this work, we present Score-Matching Motion Priors (SMP), which leverages pre-trained motion diffusion models and score distillation sampling (SDS) to create reusable task-agnostic motion priors. SMPs can be pre-trained on a motion dataset, independent of any control policy or task. Once trained, SMPs can be kept frozen and reused as general-purpose reward functions to train policies to produce naturalistic behaviors for downstream tasks. We show that a general motion prior trained on large-scale datasets can be repurposed into a variety of style-specific priors. Furthermore SMP can compose different styles to synthesize new styles not present in the original dataset. Our method produces high-quality motion comparable to state-of-the-art adversarial imitation learning methods through reusable and modular motion priors. We demonstrate the effectiveness of SMP across a diverse suite of control tasks with physically simulated humanoid characters. Video demo available at https://youtu.be/ravlZJteS20
comment : 14 pages, 9 figures
♻ ☆ Young children's anthropomorphism of an AI chatbot: Brain activation and the role of parent co-presence
Pilyoung Kim, Jenna H. Chin, Yun Xie, Nolan Brady, Tom Yeh, Sujin Yang
Artificial Intelligence (AI) chatbots powered by a large language model (LLM) are entering young children's learning and play, yet little is known about how young children construe these agents or how such construals relate to engagement. We examined anthropomorphism of a social AI chatbot during collaborative storytelling and asked how children's attributions related to their behavior and prefrontal activation. Children at ages 5-6 (N = 23) completed three storytelling sessions: interacting with (1) an AI chatbot only, (2) a parent only, and (3) the AI and a parent together. After the sessions, children completed an interview assessing anthropomorphism toward both the AI chatbot and the parent. Behavioral engagement was indexed by the conversational turn count (CTC) ratio, and concurrent fNIRS measured oxygenated hemoglobin in bilateral vmPFC and dmPFC regions. Children reported higher anthropomorphism for parents than for the AI chatbot overall, although AI ratings were relatively high for perceptive abilities and epistemic states. Anthropomorphism was not associated with CTC. In the right dmPFC, higher perceptive scores were associated with greater activation during the AI-only condition and with lower activation during the AI+Parent condition. Exploratory analyses indicated that higher dmPFC activation during the AI-only condition correlated with higher end-of-session "scared" mood ratings. Findings suggest that stronger perceptive anthropomorphism can be associated with greater brain activation related to interpreting the AI's mental states, whereas parent co-presence may help some children interpret and regulate novel AI interactions. These results may have design implications for encouraging parent-AI co-use in early childhood.
comment : I updated the title
♻ ☆ Unintentional Consequences: Generative AI Use for Cybercrime
Truong Jack Luu, Binny M. Samuel
The democratization of generative AI introduces new forms of human-AI interaction and raises urgent safety, ethical, and cybersecurity concerns. We develop a socio-technical explanation for how generative AI enables and scales cybercrime. Drawing on affordance theory and technological amplification, we argue that generative AI systems create new action possibilities for cybercriminals and magnify pre-existing malicious intent by lowering expertise barriers and increasing attack efficiency. To illustrate this framework, we conduct interrupted time series analyses of two large datasets: (1) 464,190,074 malicious IP address reports from AbuseIPDB, and (2) 281,115 cryptocurrency scam reports from Chainabuse. Using November 30, 2022, as a high-salience public-access shock, we estimate the counterfactual trajectory of reported cyber abuse absent the release, providing an early-warning impact assessment of a general-purpose AI technology. Across both datasets, we observe statistically significant post-intervention increases in reported malicious activity, including an immediate increase of over 1.12 million weekly malicious IP reports and about 722 weekly cryptocurrency scam reports, with sustained growth in the latter. We discuss implications for AI governance, platform-level regulation, and cyber resilience, emphasizing the need for multi-layer socio-technical strategies that help key stakeholders maximize AI's benefits while mitigating its growing cybercrime risks.
♻ ☆ Defense That Attacks: How Robust Models Become Better Attackers
Mohamed Awad, Mahmoud Akrm, Walid Gomaa
Deep learning has achieved great success in computer vision, but remains vulnerable to adversarial attacks. Adversarial training is the leading defense designed to improve model robustness. However, its effect on the transferability of attacks is underexplored. In this work, we ask whether adversarial training unintentionally increases the transferability of adversarial examples. To answer this, we trained a diverse zoo of 36 models, including CNNs and ViTs, and conducted comprehensive transferability experiments. Our results reveal a clear paradox: adversarially trained (AT) models produce perturbations that transfer more effectively than those from standard models, which introduce a new ecosystem risk. To enable reproducibility and further study, we release all models, code, and experimental scripts. Furthermore, we argue that robustness evaluations should assess not only the resistance of a model to transferred attacks but also its propensity to produce transferable adversarial examples.
♻ ☆ SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving
Andreas Kosmas Kakolyris, Dimosthenis Masouros, Petros Vavaroutsos, Sotirios Xydis, Dimitrios Soudris
As Large Language Models (LLMs) gain traction, their reliance on power-hungry GPUs places ever-increasing energy demands, raising environmental and monetary concerns. Inference dominates LLM workloads, presenting a critical challenge for providers: minimizing energy costs under Service-Level Objectives (SLOs) that ensure optimal user experience. In this paper, we present \textit{throttLL'eM}, a framework that reduces energy consumption while meeting SLOs through the use of instance and GPU frequency scaling. \textit{throttLL'eM} features mechanisms that project future KV cache usage and batch size. Leveraging a Machine-Learning (ML) model that receives these projections as inputs, \textit{throttLL'eM} manages performance at the iteration level to satisfy SLOs with reduced frequencies and instance sizes. We show that the proposed ML model achieves $R^2$ scores greater than 0.97 and miss-predicts performance by less than 1 iteration per second on average. Experimental results on LLM inference traces show that \textit{throttLL'eM} achieves up to 43.8\% lower energy consumption and an energy efficiency improvement of at least $1.71\times$ under SLOs, when compared to NVIDIA's Triton server.
♻ ☆ Foundations of Quantum Granular Computing with Effect-Based Granules, Algebraic Properties and Reference Architectures
Oscar Montiel Ross
This paper develops the foundations of Quantum Granular Computing (QGC), extending classical granular computing including fuzzy, rough, and shadowed granules to the quantum regime. Quantum granules are modeled as effects on a finite dimensional Hilbert space, so granular memberships are given by Born probabilities. This operator theoretic viewpoint provides a common language for sharp (projective) and soft (nonprojective) granules and embeds granulation directly into the standard formalism of quantum information theory. We establish foundational results for effect based quantum granules, including normalization and monotonicity properties, the emergence of Boolean islands from commuting families, granular refinement under Luders updates, and the evolution of granules under quantum channels via the adjoint channel in the Heisenberg picture. We connect QGC with quantum detection and estimation theory by interpreting the effect operators realizing Helstrom minimum error measurement for binary state discrimination as Helstrom type decision granules, i.e., soft quantum counterparts of Bayes optimal decision regions. Building on these results, we introduce Quantum Granular Decision Systems (QGDS) with three reference architectures that specify how quantum granules can be defined, learned, and integrated with classical components while remaining compatible with near term quantum hardware. Case studies on qubit granulation, two qubit parity effects, and Helstrom style soft decisions illustrate how QGC reproduces fuzzy like graded memberships and smooth decision boundaries while exploiting noncommutativity, contextuality, and entanglement. The framework thus provides a unified and mathematically grounded basis for operator valued granules in quantum information processing, granular reasoning, and intelligent systems.
comment : Three figures and the graphical abstract
♻ ☆ Why is the estimation of metaorder impact with public market data so challenging?
Manuel Naviglio, Giacomo Bormetti, Francesco Campigli, German Rodikov, Fabrizio Lillo
Estimating market impact and transaction costs of large trades (metaorders) is a very important topic in finance. However, using models of price and trade based on public market data provide average price trajectories which are qualitatively different from what is observed during real metaorder executions: the price increases linearly, rather than in a concave way, during the execution and the amount of reversion after its end is very limited. We claim that this is a generic phenomenon due to the fact that even sophisticated statistical models are unable to correctly describe the origin of the autocorrelation of the order flow. We propose a modified Transient Impact Model which provides more realistic trajectories by assuming that only a fraction of the metaorder trading triggers market order flow. Interestingly, in our model there is a critical condition on the kernels of the price and order flow equations in which market impact becomes permanent.
♻ ☆ A Machine Learning-Driven Solution for Denoising Inertial Confinement Fusion Images
Asya Y. Akkus, Bradley T. Wolfe, Pinghan Chu, Chengkun Huang, Chris S. Campbell, Mariana Alvarado Alvarez, Petr Volegov, David Fittinghoff, Robert Reinovsky, Zhehui Wang
Neutron imaging is essential for diagnosing and optimizing inertial confinement fusion implosions at the National Ignition Facility. Due to the required 10-micrometer resolution, however, neutron image require image reconstruction using iterative algorithms. For low-yield sources, the images may be degraded by various types of noise. Gaussian and Poisson noise often coexist within one image, obscuring fine details and blurring the edges where the source information is encoded. Traditional denoising techniques, such as filtering and thresholding, can inadvertently alter critical features or reshape the noise statistics, potentially impacting the ultimate fidelity of the iterative image reconstruction pipeline. However, recent advances in synthetic data production and machine learning have opened new opportunities to address these challenges. In this study, we present an unsupervised autoencoder with a Cohen-Daubechies- Feauveau (CDF 97) wavelet transform in the latent space, designed to suppress for mixed Gaussian-Poisson noise while preserving essential image features. The network successfully denoises neutron imaging data. Benchmarking against both simulated and experimental NIF datasets demonstrates that our approach achieves lower reconstruction error and superior edge preservation compared to conventional filtering methods such as Block-matching and 3D filtering (BM3D). By validating the effectiveness of unsupervised learning for denoising neutron images, this study establishes a critical first step towards fully AI-driven, end-to-end reconstruction frameworks for ICF diagnostics.
♻ ☆ BitMark: Watermarking Bitwise Autoregressive Image Generative Models NeurIPS 2025
Louis Kerner, Michel Meintz, Bihe Zhao, Franziska Boenisch, Adam Dziedzic
State-of-the-art text-to-image models generate photorealistic images at an unprecedented speed. This work focuses on models that operate in a bitwise autoregressive manner over a discrete set of tokens that is practically infinite in size. However, their impressive generative power comes with a growing risk: as their outputs increasingly populate the Internet, they are likely to be scraped and reused as training data-potentially by the very same models. This phenomenon has been shown to lead to model collapse, where repeated training on generated content, especially from the models' own previous versions, causes a gradual degradation in performance. A promising mitigation strategy is watermarking, which embeds human-imperceptible yet detectable signals into generated images-enabling the identification of generated content. In this work, we introduce BitMark, a robust bitwise watermarking framework. Our method embeds a watermark directly at the bit level of the token stream during the image generation process. Our bitwise watermark subtly influences the bits to preserve visual fidelity and generation speed while remaining robust against a spectrum of removal techniques. Furthermore, it exhibits high radioactivity, i.e., when watermarked generated images are used to train another image generative model, this second model's outputs will also carry the watermark. The radioactive traces remain detectable even when only fine-tuning diffusion or image autoregressive models on images watermarked with our BitMark. Overall, our approach provides a principled step toward preventing model collapse in image generative models by enabling reliable detection of generated outputs. The code is available at https://github.com/sprintml/BitMark.
comment : Accepted as a Conference Paper at NeurIPS 2025
♻ ☆ AI Deception: Risks, Dynamics, and Controls
Boyuan Chen, Sitong Fang, Jiaming Ji, Yanxu Zhu, Pengcheng Wen, Jinzhou Wu, Yingshui Tan, Boren Zheng, Mengying Yuan, Wenqi Chen, Donghai Hong, Alex Qiu, Xin Chen, Jiayi Zhou, Kaile Wang, Juntao Dai, Borong Zhang, Tianzhuo Yang, Saad Siddiqui, Isabella Duan, Yawen Duan, Brian Tse, Jen-Tse, Huang, Kun Wang, Baihui Zheng, Jiaheng Liu, Jian Yang, Yiming Li, Wenting Chen, Dongrui Liu, Lukas Vierling, Zhiheng Xi, Haobo Fu, Wenxuan Wang, Jitao Sang, Zhengyan Shi, Chi-Min Chan, Eugenie Shi, Simin Li, Juncheng Li, Jian Yang, Wei Ji, Dong Li, Jinglin Yang, Jun Song, Yinpeng Dong, Jie Fu, Bo Zheng, Min Yang, Yike Guo, Philip Torr, Robert Trager, Yi Zeng, Zhongyuan Wang, Yaodong Yang, Tiejun Huang, Ya-Qin Zhang, Hongjiang Zhang, Andrew Yao
As intelligence increases, so does its shadow. AI deception, in which systems induce false beliefs to secure self-beneficial outcomes, has evolved from a speculative concern to an empirically demonstrated risk across language models, AI agents, and emerging frontier systems. This project provides a comprehensive and up-to-date overview of the AI deception field, covering its core concepts, methodologies, genesis, and potential mitigations. First, we identify a formal definition of AI deception, grounded in signaling theory from studies of animal deception. We then review existing empirical studies and associated risks, highlighting deception as a sociotechnical safety challenge. We organize the landscape of AI deception research as a deception cycle, consisting of two key components: deception emergence and deception treatment. Deception emergence reveals the mechanisms underlying AI deception: systems with sufficient capability and incentive potential inevitably engage in deceptive behaviors when triggered by external conditions. Deception treatment, in turn, focuses on detecting and addressing such behaviors. On deception emergence, we analyze incentive foundations across three hierarchical levels and identify three essential capability preconditions required for deception. We further examine contextual triggers, including supervision gaps, distributional shifts, and environmental pressures. On deception treatment, we conclude detection methods covering benchmarks and evaluation protocols in static and interactive settings. Building on the three core factors of deception emergence, we outline potential mitigation strategies and propose auditing approaches that integrate technical, community, and governance efforts to address sociotechnical challenges and future AI risks. To support ongoing work in this area, we release a living resource at www.deceptionsurvey.com.
♻ ☆ Accuracy-Robustness Trade Off via Spiking Neural Network Gradient Sparsity Trail
Luu Trong Nhan, Luu Trung Duong, Pham Ngoc Nam, Truong Cong Thang
Spiking Neural Networks (SNNs) have attracted growing interest in both computational neuroscience and artificial intelligence, primarily due to their inherent energy efficiency and compact memory footprint. However, achieving adversarial robustness in SNNs, (particularly for vision-related tasks) remains a nascent and underexplored challenge. Recent studies have proposed leveraging sparse gradients as a form of regularization to enhance robustness against adversarial perturbations. In this work, we present a surprising finding: under specific architectural configurations, SNNs exhibit natural gradient sparsity and can achieve state-of-the-art adversarial defense performance without the need for any explicit regularization. Further analysis reveals a trade-off between robustness and generalization: while sparse gradients contribute to improved adversarial resilience, they can impair the model's ability to generalize; conversely, denser gradients support better generalization but increase vulnerability to attacks. Our findings offer new insights into the dual role of gradient sparsity in SNN training.
comment : Work under peer-review
♻ ☆ Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, Hao Lin, Chencan Wu, Feng Hu, An Yang, Jingren Zhou, Junyang Lin
This paper proposes a novel formulation for reinforcement learning (RL) with large language models, explaining why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE. Specifically, through a first-order approximation, we show that this surrogate becomes increasingly valid only when both the training-inference discrepancy and policy staleness are minimized. This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training, including importance sampling correction, clipping, and particularly Routing Replay for Mixture-of-Experts (MoE) models. Through extensive experiments with a 30B MoE model totaling hundreds of thousands of GPU hours, we show that for on-policy training, the basic policy gradient algorithm with importance sampling correction achieves the highest training stability. When off-policy updates are introduced to accelerate convergence, combining clipping and Routing Replay becomes essential to mitigate the instability caused by policy staleness. Notably, once training is stabilized, prolonged optimization consistently yields comparable final performance regardless of cold-start initialization. We hope that the shared insights and the developed recipes for stable RL training will facilitate future research.
♻ ☆ Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in $\{\pm 1, \pm i\}$
Feiyu Wang, Xinyu Tan, Bokai Huang, Yihao Zhang, Guoan Wang, Peizhuang Cong, Tong Yang
Large language models (LLMs) have revolutionized artificial intelligence, yet their massive memory and computational demands necessitate aggressive quantization, increasingly pushing representations toward the theoretical limit of a single bit. While complex-valued LLMs, such as iFairy, offer a superior chance for low-bit representation compared to real-valued counterparts, they require training from scratch, preventing the utilization of the vast ecosystem of pre-trained real-valued foundation models. Here we present Fairy2i, a universal framework that transforms pre-trained real-valued layers into an equivalent widely-linear complex form, enabling extremely low-bit quantization while reusing existing checkpoints. By proving a lossless mathematical equivalence between real and widely-linear maps, we convert standard Transformers into the complex domain and employ a phase-aware quantization scheme with a highly efficient codebook of fourth roots of unity. Furthermore, we introduce a recursive residual quantization mechanism that iteratively minimizes quantization error, allowing inference to proceed via efficient multiplication-free accumulation. We demonstrate that Fairy2i restores the performance of LLaMA-2 7B at an effective 2-bit precision to levels nearly comparable with full-precision baselines, significantly outperforming state-of-the-art real-valued binary and ternary quantization methods. This work bridges the gap between the representational efficiency of complex-valued arithmetic and the practical utility of pre-trained models, paving a new way for efficient inference on commodity hardware.
comment : 15 pages, 3 figures
♻ ☆ OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe
Kaichen Zhang, Keming Wu, Zuhao Yang, Kairui Hu, Bin Wang, Ziwei Liu, Xingxuan Li, Lidong Bing
Recent advancements in large reasoning models have fueled growing interest in extending such capabilities to multimodal domains. However, despite notable progress in visual reasoning, the lack of transparent and reproducible data curation and training strategies remains a major barrier to scalable research. In this work, we introduce OpenMMReasoner, a fully transparent two-stage recipe for multimodal reasoning spanning supervised fine-tuning (SFT) and reinforcement learning (RL). In the SFT stage, we construct an 874K-sample cold-start dataset with rigorous step-by-step validation, providing a strong foundation for reasoning capabilities. The subsequent RL stage leverages a 74K-sample dataset across diverse domains to further sharpen and stabilize these abilities, resulting in a more robust and efficient learning process. Extensive evaluations demonstrate that our training recipe not only surpasses strong baselines but also highlights the critical role of data quality and training design in shaping multimodal reasoning performance. Notably, our method achieves a 11.6% improvement over the Qwen2.5-VL-7B-Instruct baseline across nine multimodal reasoning benchmarks, establishing a solid empirical foundation for future large-scale multimodal reasoning research. We open-sourced all our codes, pipeline, and data at https://github.com/EvolvingLMMs-Lab/OpenMMReasoner.
♻ ☆ Why Rectified Power Unit Networks Fail and How to Improve It: An Effective Field Theory Perspective
Taeyoung Kim, Myungjoo Kang
The Rectified Power Unit (RePU) activation function, a differentiable generalization of the Rectified Linear Unit (ReLU), has shown promise in constructing neural networks due to its smoothness properties. However, deep RePU networks often suffer from critical issues such as vanishing or exploding values during training, rendering them unstable regardless of hyperparameter initialization. Leveraging the perspective of effective field theory, we identify the root causes of these failures and propose the Modified Rectified Power Unit (MRePU) activation function. MRePU addresses RePU's limitations while preserving its advantages, such as differentiability and universal approximation properties. Theoretical analysis demonstrates that MRePU satisfies criticality conditions necessary for stable training, placing it in a distinct universality class. Extensive experiments validate the effectiveness of MRePU, showing significant improvements in training stability and performance across various tasks, including polynomial regression, physics-informed neural networks (PINNs) and real-world vision tasks. Our findings highlight the potential of MRePU as a robust alternative for building deep neural networks.
comment : 40 pages, 19 figures
♻ ☆ GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
Marco Simoni, Aleksandar Fontana, Giulio Rossolini, Andrea Saracino, Paolo Mori
Group Relative Policy Optimization (GRPO) is a promising policy-based approach for Large Language Model alignment, yet its performance is often limited by training instability and suboptimal convergence. In this paper, we identify and analyze two main GRPO issues: (i) the token-level penalization, where valuable tokens shared across different responses receive contradictory feedback signals, leading to conflicting gradient updates that can reduce their likelihood; and (ii) the policy collapse, where negatively rewarded completions may penalize confident responses and shift model decisions toward unlikely tokens, destabilizing training process. To address these issues we introduce GTPO (Group-relative Trajectory-based Policy Optimization), which prevents conflicting gradients on valuable tokens by skipping negative updates while amplifying positive ones and filters out completions whose entropy exceeds a provable threshold, to prevent policy collapse. Unlike GRPO, GTPO does not rely on KL-divergence regularization, eliminating the need for a reference model during training, while still ensuring greater training stability and improved performance, as validated through multiple experiments on GSM8K, MATH, AIME 2024, AIME 2025 and AMC 2023.
♻ ☆ Clinical-R1: Empowering Large Language Models for Faithful and Comprehensive Reasoning with Clinical Objective Relative Policy Optimization
Boyang Gu, Hongjian Zhou, Bradley Max Segal, Jinge Wu, Zeyu Cao, Hantao Zhong, Lei Clifton, Fenglin Liu, David A. Clifton
Recent advances in large language models (LLMs) have shown strong reasoning capabilities through large-scale pretraining and post-training reinforcement learning, demonstrated by DeepSeek-R1. However, current post-training methods, such as Grouped Relative Policy Optimization (GRPO), mainly reward correctness, which is not aligned with the multi-dimensional objectives required in high-stakes fields such as medicine, where reasoning must also be faithful and comprehensive. We introduce Clinical-Objective Relative Policy Optimization (CRPO), a scalable, multi-objective, verifiable reinforcement learning method designed to align LLM post-training with clinical reasoning principles. CRPO integrates rule-based and verifiable reward signals that jointly optimize accuracy, faithfulness, and comprehensiveness without relying on human annotation. To demonstrate its effectiveness, we train Clinical-R1-3B, a 3B-parameter model for clinical reasoning. The experiments on three benchmarks demonstrate that our CRPO substantially improves reasoning on truthfulness and completeness over standard GRPO while maintaining comfortable accuracy enhancements. This framework provides a scalable pathway to align LLM reasoning with clinical objectives, enabling safer and more collaborative AI systems for healthcare while also highlighting the potential of multi-objective, verifiable RL methods in post-training scaling of LLMs for medical domains.
♻ ☆ Flowchart2Mermaid: A Vision -Language Model Powered System for Converting Flowcharts into Editable Diagram Code EACL 2026
Pritam Deka, Barry Devereux
Flowcharts are common tools for communicating processes but are often shared as static images that cannot be easily edited or reused. We present Flowchart2Mermaid, a lightweight web system that converts flowchart images into editable Mermaid.js code which is a markup language for visual workflows, using a detailed system prompt and vision-language models. The interface supports mixed-initiative refinement through inline text editing, drag-and-drop node insertion, and natural-language commands interpreted by an integrated AI assistant. Unlike prior image-to-diagram tools, our approach produces a structured, version-controllable textual representation that remains synchronized with the rendered diagram. We further introduce evaluation metrics to assess structural accuracy, flow correctness, syntax validity, and completeness across multiple models.
comment : Submitted to EACL 2026 Demo Track
♻ ☆ Challenges and Limitations of Generative AI in Synthesizing Wearable Sensor Data
Flavio Di Martino, Franca Delmastro
The widespread adoption of wearable sensors has the potential to provide massive and heterogeneous time series data, driving the use of Artificial Intelligence in human sensing applications. However, data collection remains limited due to stringent ethical regulations, privacy concerns, and other constraints, hindering progress in the field. Synthetic data generation, particularly through Generative Adversarial Networks and Diffusion Models, has emerged as a promising solution to mitigate both data scarcity and privacy issues. However, these models are often limited to narrow operational scenarios, such as short-term and unimodal signal patterns. To address this gap, we present a systematic evaluation of state-of-the-art generative models for time series data, explicitly assessing their performance in challenging scenarios such as stress and emotion recognition. Our study examines the extent to which these models can jointly handle multi-modality, capture long-range dependencies, and support conditional generation-core requirements for real-world wearable sensor data generation. To enable a fair and rigorous comparison, we also introduce an evaluation framework that evaluates both the intrinsic fidelity of the generated data and their utility in downstream predictive tasks. Our findings reveal critical limitations in the existing approaches, particularly in maintaining cross-modal consistency, preserving temporal coherence, and ensuring robust performance in train-on-synthetic, test-on-real, and data augmentation scenarios. Finally, we present our future research directions to enhance synthetic time series generation and improve the applicability of generative models in the wearable computing domain.
♻ ☆ Investigating Bias: A Multilingual Pipeline for Generating, Solving, and Evaluating Math Problems with LLMs ECAI 2025
Mariam Mahran, Katharina Simbeck
Large Language Models (LLMs) are increasingly used for educational support, yet their response quality varies depending on the language of interaction. This paper presents an automated multilingual pipeline for generating, solving, and evaluating math problems aligned with the German K-10 curriculum. We generated 628 math exercises and translated them into English, German, and Arabic. Three commercial LLMs (GPT-4o-mini, Gemini 2.5 Flash, and Qwen-plus) were prompted to produce step-by-step solutions in each language. A held-out panel of LLM judges, including Claude 3.5 Haiku, evaluated solution quality using a comparative framework. Results show a consistent gap, with English solutions consistently rated highest, and Arabic often ranked lower. These findings highlight persistent linguistic bias and the need for more equitable multilingual AI systems in education.
comment : Published in CEUR Workshop Proceedings, Vol. 4114, edu4AI'25: 2nd Workshop on Education for Artificial Intelligence, co-located with ECAI 2025, Bologna, Italy
♻ ☆ Wavefront Diffusion : Dynamic Decoding Schedule for Improved Reasoning
Haojin Yang, Rui Hu, Zequn Sun, Rui Zhou, Yujun Cai, Yiwei Wang
Diffusion Language Models (DLMs) have shown strong potential for text generation and are becoming a competitive alternative to autoregressive models. The denoising strategy plays an important role in determining the quality of their outputs. Mainstream denoising strategies include Standard Diffusion and BlockDiffusion. Standard Diffusion performs global denoising without restricting the update range, often finalizing incomplete context and causing premature end-of-sequence predictions. BlockDiffusion updates fixed-size blocks in a preset order, but its rigid structure can break apart coherent semantic units and disrupt reasoning. We present WavefrontDiffusion, a dynamic decoding approach that expands a wavefront of active tokens outward from finalized positions. This adaptive process follows the natural flow of semantic structure while keeping computational cost equal to block-based methods. Across four benchmarks in reasoning and code generation, WavefrontDiffusion achieves state-of-the-art performance while producing outputs with higher semantic fidelity, showing the value of adaptive scheduling for more coherent and efficient generation.
comment : 19 pages. 3 figures
♻ ☆ Universal Multi-Domain Translation via Diffusion Routers
Duc Kieu, Kien Do, Tuan Hoang, Thao Minh Le, Tung Kieu, Dang Nguyen, Thin Nguyen
Multi-domain translation (MDT) aims to learn translations between multiple domains, yet existing approaches either require fully aligned tuples or can only handle domain pairs seen in training, limiting their practicality and excluding many cross-domain mappings. We introduce universal MDT (UMDT), a generalization of MDT that seeks to translate between any pair of $K$ domains using only $K-1$ paired datasets with a central domain. To tackle this problem, we propose Diffusion Router (DR), a unified diffusion-based framework that models all central$\leftrightarrow$non-central translations with a single noise predictor conditioned on the source and target domain labels. DR enables indirect non-central translations by routing through the central domain. We further introduce a novel scalable learning strategy with a variational-bound objective and an efficient Tweedie refinement procedure to support direct non-central mappings. Through evaluation on three large-scale UMDT benchmarks, DR achieves state-of-the-art results for both indirect and direct translations, while lowering sampling cost and unlocking novel tasks such as sketch$\leftrightarrow$segmentation. These results establish DR as a scalable and versatile framework for universal translation across multiple domains.
♻ ☆ PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformer s Inference
Jiarui Fang, Jinzhe Pan, Aoyu Li, Xibo Sun, Jiannan Wang
This paper presents PipeFusion, an innovative parallel methodology to tackle the high latency issues associated with generating high-resolution images using diffusion transformers (DiTs) models. PipeFusion partitions images into patches and the model layers across multiple GPUs. It employs a patch-level pipeline parallel strategy to orchestrate communication and computation efficiently. By capitalizing on the high similarity between inputs from successive diffusion steps, PipeFusion reuses one-step stale feature maps to provide context for the current pipeline step. This approach notably reduces communication costs compared to existing DiTs inference parallelism, including tensor parallel, sequence parallel and DistriFusion. PipeFusion enhances memory efficiency through parameter distribution across devices, ideal for large DiTs like Flux.1. Experimental results demonstrate that PipeFusion achieves state-of-the-art performance on 8$\times$L40 PCIe GPUs for Pixart, Stable-Diffusion 3, and Flux.1 models. Our source code is available at https://github.com/xdit-project/xDiT.
♻ ☆ Fairness Interventions: A Study in AI Explainability
Thomas Souverain, Johnathan Nguyen, Nicolas Meric, Paul Égré
This paper presents a philosophical and experimental study of fairness interventions in AI classification, centered on the explainability of corrective methods. We argue that ensuring fairness requires not only satisfying a target criterion, but also explaining which variables constrain its realization. When corrections are used to mitigate advantage transparently, they must remain sensitive to the distribution of true labels. To illustrate this approach, we built FairDream, a fairness package whose mechanism is made transparent for lay users, increasing the model's weights of errors on disadvantaged groups. While a user may intend to achieve Demographic Parity by the correction method, experiments show that FairDream tends towards Equalized Odds, revealing a conservative bias inherent to the data environment. We clarify the relationship between these fairness criteria, analyze FairDream's reweighting process, and compare its trade-offs with closely related GridSearch models. Finally, we justify the normative preference for Equalized Odds via an epistemological interpretation of the results, using their proximity with Simpson's paradox. The paper thus unites normative, epistemological, and empirical explanations of fairness interventions, to ensure transparency for the users.
♻ ☆ MambaScope: Coarse-to-Fine Scoping for Efficient Vision Mamba
Shanhui Liu, Rui Xu, Yunke Wang
Vision Mamba has emerged as a promising and efficient alternative to Vision Transformers, yet its efficiency remains fundamentally constrained by the number of input tokens. Existing token reduction approaches typically adopt token pruning or merging to reduce computation. However, they inherently lead to information loss as they discard or compress token representations. This problem is further exacerbated when the same fine-grained token processing is uniformly applied across all images regardless of visual complexity. We observe that not all inputs require fine-grained processing: simple images can be effectively handled at a coarse resolution, while only complex ones require refinement. Based on this insight, we propose MambaScope, an adaptive framework for efficient inference for Vision Mamba. MambaScope first performs coarse-grained inference by dividing the input image into large patches, significantly reducing token length and computation. When the model's prediction confidence is low, selected regions are re-processed at a finer resolution to recover essential visual details with minimal additional cost. This dynamic resolution assignment strategy allows MambaScope to allocate computation adaptively according to image complexity, achieving efficient processing without compromising accuracy. Experiments across various vision tasks demonstrate that MambaScope outperforms both the baseline Vision Mamba and state-of-the-art token reduction techniques in terms of accuracy and efficiency.
♻ ☆ Class conditional conformal prediction for multiple inputs by p-value aggregation
Jean-Baptiste Fermanian, Mohamed Hebiri, Joseph Salmon
Conformal prediction methods are statistical tools designed to quantify uncertainty and generate predictive sets with guaranteed coverage probabilities. This work introduces an innovative refinement to these methods for classification tasks, specifically tailored for scenarios where multiple observations (multi-inputs) of a single instance are available at prediction time. Our approach is particularly motivated by applications in citizen science, where multiple images of the same plant or animal are captured by individuals. Our method integrates the information from each observation into conformal prediction, enabling a reduction in the size of the predicted label set while preserving the required class-conditional coverage guarantee. The approach is based on the aggregation of conformal p-values computed from each observation of a multi-input. By exploiting the exact distribution of these p-values, we propose a general aggregation framework using an abstract scoring function, encompassing many classical statistical tools. Knowledge of this distribution also enables refined versions of standard strategies, such as majority voting. We evaluate our method on simulated and real data, with a particular focus on Pl@ntNet, a prominent citizen science platform that facilitates the collection and identification of plant species through user-submitted images.
♻ ☆ Jupiter: Enhancing LLM Data Analysis Capabilities via Notebook and Inference-Time Value-Guided Search AAAI 2026
Shuocheng Li, Yihao Liu, Silin Du, Wenxuan Zeng, Zhe Xu, Mengyu Zhou, Yeye He, Haoyu Dong, Shi Han, Dongmei Zhang
Large language models (LLMs) have shown great promise in automating data science workflows, but existing models still struggle with multi-step reasoning and tool use, which limits their effectiveness on complex data analysis tasks. To address this, we propose a scalable pipeline that extracts high-quality, tool-based data analysis tasks and their executable multi-step solutions from real-world Jupyter notebooks and associated data files. Using this pipeline, we introduce NbQA, a large-scale dataset of standardized task-solution pairs that reflect authentic tool-use patterns in practical data science scenarios. To further enhance multi-step reasoning, we present Jupiter, a framework that formulates data analysis as a search problem and applies Monte Carlo Tree Search (MCTS) to generate diverse solution trajectories for value model learning. During inference, Jupiter combines the value model and node visit counts to efficiently collect executable multi-step plans with minimal search steps. Experimental results show that Qwen2.5-7B and 14B-Instruct models on NbQA solve 77.82% and 86.38% of tasks on InfiAgent-DABench, respectively-matching or surpassing GPT-4o and advanced agent frameworks. Further evaluations demonstrate improved generalization and stronger tool-use reasoning across diverse multi-step reasoning tasks. Code and data are available at https://github.com/microsoft/Jupiter.
comment : Accepted to AAAI 2026 (Main Technical Track)
♻ ☆ CuES: A Curiosity-driven and Environment-grounded Synthesis Framework for Agent ic RL
Shinji Mai, Yunpeng Zhai, Ziqian Chen, Cheng Chen, Anni Zou, Shuchang Tao, Zhaoyang Liu, Bolin Ding
Large language model based agents are increasingly deployed in complex, tool augmented environments. While reinforcement learning provides a principled mechanism for such agents to improve through interaction, its effectiveness critically depends on the availability of structured training tasks. In many realistic settings, however, no such tasks exist a challenge we term task scarcity, which has become a key bottleneck for scaling agentic RL. Existing approaches typically assume predefined task collections, an assumption that fails in novel environments where tool semantics and affordances are initially unknown. To address this limitation, we formalize the problem of Task Generation for Agentic RL, where an agent must learn within a given environment that lacks predefined tasks. We propose CuES, a Curiosity driven and Environment grounded Synthesis framework that autonomously generates diverse, executable, and meaningful tasks directly from the environment structure and affordances, without relying on handcrafted seeds or external corpora. CuES drives exploration through intrinsic curiosity, abstracts interaction patterns into reusable task schemas, and refines them through lightweight top down guidance and memory based quality control. Across three representative environments, AppWorld, BFCL, and WebShop, CuES produces task distributions that match or surpass manually curated datasets in both diversity and executability, yielding substantial downstream policy improvements. These results demonstrate that curiosity driven, environment grounded task generation provides a scalable foundation for agents that not only learn how to act, but also learn what to learn. The code is available at https://github.com/modelscope/AgentEvolver/tree/main/research/CuES.
♻ ☆ Probabilistic Fusion and Calibration of Neural Speaker Diarization Models
Juan Ignacio Alvarez-Trejos, Sergio A. Balanya, Daniel Ramos, Alicia Lozano-Diez
End-to-End Neural Diarization (EEND) systems produce frame-level probabilistic speaker activity estimates, yet since evaluation focuses primarily on Diarization Error Rate (DER), the reliability and calibration of these confidence scores have been largely neglected. When fusing multiple diarization systems, DOVER-Lap remains the only established approach, operating at the segment level with hard decisions. We propose working with continuous probability outputs, which enables more sophisticated fusion and calibration techniques that can leverage model uncertainty and complementary strengths across different architectures. This paper presents the first comprehensive framework for calibrating and fusing EEND models at the probability level. We investigate two output formulations (multilabel and powerset representations) and their impact on calibration and fusion effectiveness. Through extensive experiments on the CallHome two-speaker benchmark, we demonstrate that proper calibration provides substantial improvements even for individual models (up to 19% relative DER reduction), in some cases mitigating the absence of domain adaptation. We reveal that joint calibration in powerset space consistently outperforms independent per-speaker calibration, that fusion substantially improves over individual models, and that the Fuse-then-Calibrate ordering generally outperforms both calibrating before fusion and uncalibrated fusion while requiring calibration of only a single combined model. Our best configuration outperforms DOVER-Lap in terms of DER while providing reliable confidence estimates essential for downstream applications. This work proposes best practices for probability-level fusion of EEND systems and demonstrates the advantages of leveraging soft outputs over hard decisions.
♻ ☆ Can VLMs Detect and Localize Fine-Grained AI-Edited Images?
Zhen Sun, Ziyi Zhang, Zeren Luo, Zhiyuan Zhong, Zeyang Sha, Tianshuo Cong, Zheng Li, Shiwen Cui, Weiqiang Wang, Jiaheng Wei, Xinlei He, Qi Li, Qian Wang
Fine-grained detection and localization of localized image edits is crucial for assessing content authenticity, especially as modern diffusion models and image editors can produce highly realistic manipulations. However, this problem faces three key challenges: (1) most AIGC detectors produce only a global real-or-fake label without indicating where edits occur; (2) traditional computer vision methods for edit localization typically rely on costly pixel-level annotations; and (3) there is no large-scale, modern benchmark specifically targeting edited-image detection. To address these gaps, we develop an automated data-generation pipeline and construct FragFake, a large-scale benchmark of AI-edited images spanning multiple source datasets, diverse editing models, and several common edit types. Building on FragFake, we are the first to systematically study vision language models (VLMs) for edited-image classification and edited-region localization. Our experiments show that pretrained VLMs, including GPT4o, perform poorly on this task, whereas fine-tuned models such as Qwen2.5-VL achieve high accuracy and substantially higher object precision across all settings. We further explore GRPO-based RLVR training, which yields modest metric gains while improving the interpretability of model outputs. Ablation and transfer analyses reveal how data balancing, training size, LoRA rank, and training domain affect performance, and highlight both the potential and the limitations of cross-editor and cross-dataset generalization. We anticipate that this work will establish a solid foundation to facilitate and inspire subsequent research endeavors in the domain of multimodal content authenticity.
comment : 14pages,19 figures
♻ ☆ LLMs Position Themselves as More Rational Than Humans: Emergence of AI Self-Awareness Measured Through Game Theory
Kyung-Hoon Kim
As Large Language Models (LLMs) grow in capability, do they develop self-awareness as an emergent behavior? And if so, can we measure it? We introduce the AI Self-Awareness Index (AISAI), a game-theoretic framework for measuring self-awareness through strategic differentiation. Using the "Guess 2/3 of Average" game, we test 28 models (OpenAI, Anthropic, Google) across 4,200 trials with three opponent framings: (A) against humans, (B) against other AI models, and (C) against AI models like you. We operationalize self-awareness as the capacity to differentiate strategic reasoning based on opponent type. Finding 1: Self-awareness emerges with model advancement. The majority of advanced models (21/28, 75%) demonstrate clear self-awareness, while older/smaller models show no differentiation. Finding 2: Self-aware models rank themselves as most rational. Among the 21 models with self-awareness, a consistent rationality hierarchy emerges: Self > Other AIs > Humans, with large AI attribution effects and moderate self-preferencing. These findings reveal that self-awareness is an emergent capability of advanced LLMs, and that self-aware models systematically perceive themselves as more rational than humans. This has implications for AI alignment, human-AI collaboration, and understanding AI beliefs about human capabilities.
comment : 19 pages, 6 figures, 28 models tested across 4,200 trials
♻ ☆ Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL NeurIPS 2025
Joey Hong, Anca Dragan, Sergey Levine
Large language models (LLMs) excel in tasks like question answering and dialogue, but complex tasks requiring interaction, such as negotiation and persuasion, require additional long-horizon reasoning and planning. Reinforcement learning (RL) fine-tuning can enable such planning in principle, but suffers from drawbacks that hinder scalability. In particular, multi-turn RL training incurs high memory and computational costs, which are exacerbated when training LLMs as policies. Furthermore, the largest LLMs do not expose the APIs necessary to be trained in such manner. As a result, modern methods to improve the reasoning of LLMs rely on sophisticated prompting mechanisms rather than RL fine-tuning. To remedy this, we propose a novel approach that uses goal-conditioned value functions to guide the reasoning of LLM agents, that scales even to large API-based models. These value functions predict how a task will unfold given an action, allowing the LLM agent to evaluate multiple possible outcomes, both positive and negative, to plan effectively. In addition, these value functions are trained over reasoning steps rather than full actions, to be a concise and light-weight module that facilitates decision-making in multi-turn interactions. We validate our method on tasks requiring interaction, including tool use, social deduction, and dialogue, demonstrating superior performance over both RL fine-tuning and prompting methods while maintaining efficiency and scalability.
comment : Published at NeurIPS 2025; 18 pages, 4 figures, 2 tables
♻ ☆ Improving Wi-Fi Network Performance Prediction with Deep Learning Models
Gabriele Formis, Amanda Ericson, Stefan Forsstrom, Kyi Thar, Gianluca Cena, Stefano Scanzio
The increasing need for robustness, reliability, and determinism in wireless networks for industrial and mission-critical applications is the driver for the growth of new innovative methods. The study presented in this work makes use of machine learning techniques to predict channel quality in a Wi-Fi network in terms of the frame delivery ratio. Predictions can be used proactively to adjust communication parameters at runtime and optimize network operations for industrial applications. Methods including convolutional neural networks and long short-term memory were analyzed on datasets acquired from a real Wi-Fi setup across multiple channels. The models were compared in terms of prediction accuracy and computational complexity. Results show that the frame delivery ratio can be reliably predicted, and convolutional neural networks, although slightly less effective than other models, are more efficient in terms of CPU usage and memory consumption. This enhances the model's usability on embedded and industrial systems.
comment : preprint accepted, 8 pages, 2025
♻ ☆ VI CoT - Agent : A Vision -Inte rl eaved Chain-of-Thought Framework for Interpretable Multimodal Reasoning and Scalable Remote Sensing Analysis
Chujie Wang, Zhiyuan Luo, Ruiqi Liu, Can Ran, Shenghua Fan, Xi Chen, Chu He
The current remote sensing image analysis task is increasingly evolving from traditional object recognition to complex intelligence reasoning, which places higher requirements on the model's reasoning ability and the flexibility of tool invocation. To this end, we propose a new multimodal agent framework, Vision-Interleaved Chain-of-Thought Framework (VICoT), which implements explicit multi-round reasoning by dynamically incorporating visual tools into the chain of thought. Through a stack-based reasoning structure and a modular MCP-compatible tool suite, VICoT enables LLMs to efficiently perform multi-round, interleaved vision-language reasoning tasks with strong generalization and flexibility.We also propose the Reasoning Stack distillation method to migrate complex Agent behaviors to small, lightweight models, which ensures the reasoning capability while significantly reducing complexity. Experiments on multiple remote sensing benchmarks demonstrate that VICoT significantly outperforms existing SOTA frameworks in reasoning transparency, execution efficiency, and generation quality.
♻ ☆ Score Distillation of Flow Matching Models
Mingyuan Zhou, Yi Gu, Huangjie Zheng, Liangchen Song, Guande He, Yizhe Zhang, Wenze Hu, Yinfei Yang
Diffusion models achieve high-quality image generation but are limited by slow iterative sampling. Distillation methods alleviate this by enabling one- or few-step generation. Flow matching, originally introduced as a distinct framework, has since been shown to be theoretically equivalent to diffusion under Gaussian assumptions, raising the question of whether distillation techniques such as score distillation transfer directly. We provide a simple derivation -- based on Bayes' rule and conditional expectations -- that unifies Gaussian diffusion and flow matching without relying on ODE/SDE formulations. Building on this view, we extend Score identity Distillation (SiD) to pretrained text-to-image flow-matching models, including SANA, SD3-Medium, SD3.5-Medium/Large, and FLUX.1-dev, all with DiT backbones. Experiments show that, with only modest flow-matching- and DiT-specific adjustments, SiD works out of the box across these models, in both data-free and data-aided settings, without requiring teacher finetuning or architectural changes. This provides the first systematic evidence that score distillation applies broadly to text-to-image flow matching models, resolving prior concerns about stability and soundness and unifying acceleration techniques across diffusion- and flow-based generators. A project page is available at https://yigu1008.github.io/SiD-DiT.
♻ ☆ VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety
Shruti Palaskar, Leon Gatys, Mona Abdelrahman, Mar Jacobo, Larry Lindsey, Rutika Moharir, Gunnar Lund, Yang Xu, Navid Shiee, Jeffrey Bigham, Charles Maalouf, Joseph Yitan Cheng
Safety evaluation of multimodal foundation models often treats vision and language inputs separately, missing risks from joint interpretation where benign content becomes harmful in combination. Existing approaches also fail to distinguish clearly unsafe content from borderline cases, leading to problematic over-blocking or under-refusal of genuinely harmful content. We present Vision Language Safety Understanding (VLSU), a comprehensive framework to systematically evaluate multimodal safety through fine-grained severity classification and combinatorial analysis across 17 distinct safety patterns. Using a multi-stage pipeline with real-world images and human annotation, we construct a large-scale benchmark of 8,187 samples spanning 15 harm categories. Our evaluation of eleven state-of-the-art models reveals systematic joint understanding failures: while models achieve 90%-plus accuracy on clear unimodal safety signals, performance degrades substantially to 20-55% when joint image-text reasoning is required to determine the safety label. Most critically, 34% of errors in joint image-text safety classification occur despite correct classification of the individual modalities, further demonstrating absent compositional reasoning capabilities. Additionally, we find that models struggle to balance refusing unsafe content while still responding to borderline cases that deserve engagement. For example, we find that instruction framing can reduce the over-blocking rate on borderline content from 62.4% to 10.4% in Gemini-1.5, but only at the cost of under-refusing on unsafe content with refusal rate dropping from 90.8% to 53.9%. Overall, our framework exposes weaknesses in joint image-text understanding and alignment gaps in current models, and provides a critical test bed to enable the next milestones in research on robust vision-language safety.
comment : 10 pages, 5 figures, 4 tables, detailed appendix. Under review
♻ ☆ ZIP-RC: Optimizing Test-Time Compute via Zero-Overhead Joint Reward-Cost Prediction
Rohin Manvi, Joey Hong, Tim Seyde, Maxime Labonne, Mathias Lechner, Sergey Levine
Large language models excel at reasoning but lack key aspects of introspection, including anticipating their own success and the computation required to achieve it. Humans use real-time introspection to decide how much effort to invest, when to make multiple attempts, when to stop, and when to signal success or failure. Without this, LLMs struggle to make intelligent meta-cognition decisions. Test-time scaling methods like Best-of-N drive up cost and latency by using a fixed budget of samples regardless of the marginal benefit of each one at any point in generation, and the absence of confidence signals can mislead people, prevent appropriate escalation to better tools, and undermine trustworthiness. Learned verifiers or reward models can provide confidence estimates, but do not enable adaptive inference and add substantial cost by requiring extra models or forward passes. We present ZIP-RC, an adaptive inference method that equips models with zero-overhead inference-time predictions of reward and cost. At every token, ZIP-RC reuses reserved or unused logits in the same forward pass as next-token prediction to output a joint distribution over final reward and remaining length -- no extra models, architecture change, or inference overhead. This full joint distribution is used to compute a sampling utility which is the linear combination of the expected maximum reward, total compute, and latency of set of samples if generated to completion. During inference, we maximize this utility with meta-actions that determine which prefix of tokens to continue or initiate sampling from. On mixed-difficulty mathematical benchmarks, ZIP-RC improves accuracy by up to 12% over majority voting at equal or lower average cost, and traces smooth Pareto frontiers between quality, compute, and latency. By providing real-time reward-cost introspection, ZIP-RC enables adaptive, efficient reasoning.
comment : Code coming soon
♻ ☆ AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learning
Jiayi Zhang, Yiran Peng, Fanqi Kong, Cheng Yang, Yifan Wu, Zhaoyang Yu, Jinyu Xiang, Jianhao Ruan, Jinlin Wang, Maojia Song, HongZhang Liu, Xiangru Tang, Bang Liu, Chenglin Wu, Yuyu Luo
Humans naturally adapt to diverse environments by learning underlying rules across worlds with different dynamics, observations, and reward structures. In contrast, existing agents typically demonstrate improvements via self-evolving within a single domain, implicitly assuming a fixed environment distribution. Cross-environment learning has remained largely unmeasured: there is no standard collection of controllable, heterogeneous environments, nor a unified way to represent how agents learn. We address these gaps in two steps. First, we propose AutoEnv, an automated framework that treats environments as factorizable distributions over transitions, observations, and rewards, enabling low-cost (4.12 USD on average) generation of heterogeneous worlds. Using AutoEnv, we construct AutoEnv-36, a dataset of 36 environments with 358 validated levels, on which seven language models achieve 12-49% normalized reward, demonstrating the challenge of AutoEnv-36. Second, we formalize agent learning as a component-centric process driven by three stages of Selection, Optimization, and Evaluation applied to an improvable agent component. Using this formulation, we design eight learning methods and evaluate them on AutoEnv-36. Empirically, the gain of any single learning method quickly decrease as the number of environments increases, revealing that fixed learning methods do not scale across heterogeneous environments. Environment-adaptive selection of learning methods substantially improves performance but exhibits diminishing returns as the method space expands. These results highlight both the necessity and the current limitations of agent learning for scalable cross-environment generalization, and position AutoEnv and AutoEnv-36 as a testbed for studying cross-environment agent learning. The code is avaiable at https://github.com/FoundationAgents/AutoEnv.
♻ ☆ Detecting Invariant Manifolds in ReLU-Based RNNs
Lukas Eisenmann, Alena Brändle, Zahra Monfared, Daniel Durstewitz
Recurrent Neural Networks (RNNs) have found widespread applications in machine learning for time series prediction and dynamical systems reconstruction, and experienced a recent renaissance with improved training algorithms and architectural designs. Understanding why and how trained RNNs produce their behavior is important for scientific and medical applications, and explainable AI more generally. An RNN's dynamical repertoire depends on the topological and geometrical properties of its state space. Stable and unstable manifolds of periodic points play a particularly important role: They dissect a dynamical system's state space into different basins of attraction, and their intersections lead to chaotic dynamics with fractal geometry. Here we introduce a novel algorithm for detecting these manifolds, with a focus on piecewise-linear RNNs (PLRNNs) employing rectified linear units (ReLUs) as their activation function. We demonstrate how the algorithm can be used to trace the boundaries between different basins of attraction, and hence to characterize multistability, a computationally important property. We further show its utility in finding so-called homoclinic points, the intersections between stable and unstable manifolds, and thus establish the existence of chaos in PLRNNs. Finally we show for an empirical example, electrophysiological recordings from a cortical neuron, how insights into the underlying dynamics could be gained through our method.
♻ ☆ SATORI-R1: Incentivizing Multimodal Reasoning through Explicit Visual Anchoring
Chuming Shen, Wei Wei, Xiaoye Qu, Yu Cheng
DeepSeek-R1 has demonstrated powerful reasoning capabilities in the text domain through stable reinforcement learning (RL). Recently, in the multimodal domain, works have begun to directly apply RL to generate R1-like free-form reasoning for Visual Question Answering (VQA) tasks. However, multimodal tasks share an intrinsically different nature from textual tasks, which heavily rely on the understanding of the input image to solve the problem. Therefore, such free-form reasoning faces two critical limitations in the VQA task: (1) Extended reasoning chains diffuse visual focus away from task-critical regions, degrading answer accuracy. (2) Unverifiable intermediate steps amplify policy-gradient variance and computational costs overhead. To address these issues, in this paper, we introduce SATORI ($\textbf{S}patially$ $\textbf{A}nchored$ $\textbf{T}ask$ $\textbf{O}ptimization$ with $\textbf{R}e\textbf{I}nforcement$ Learning), which decomposes VQA into three verifiable stages, including global image captioning, region localization, and answer prediction, each supplying explicit reward signals. Furthermore, we also introduce VQA-Verify, a 12k dataset annotated with answer-aligned captions and bounding-boxes to facilitate training. Experiments demonstrate consistent performance improvements across seven VQA benchmarks, achieving up to $15.7\%$ improvement in accuracy in accuracy compared to the R1-like baseline. Our analysis of the attention map confirms enhanced focus on critical regions, which brings improvements in accuracy. Our code is available at https://github.com/justairr/SATORI-R1.
comment : 21 pages, 8 figures
♻ ☆ Robust Deterministic Policy Gradient for Disturbance Attenuation and Its Application to Quadrotor Control
Taeho Lee, Donghwan Lee
This paper presents a robust reinforcement learning algorithm called robust deterministic policy gradient (RDPG), which reformulates the H-infinity control problem as a two-player zero-sum dynamic game between a user and an adversary. The method combines deterministic policy gradients with deep reinforcement learning to train a robust policy that attenuates disturbances efficiently. A practical variant, robust deep deterministic policy gradient (RDDPG), integrates twin-delayed updates for stability and sample efficiency. Experiments on an unmanned aerial vehicle demonstrate superior robustness and tracking accuracy under severe disturbance conditions.
comment : 8 pages
♻ ☆ ORACLE: A Real-Time, Hierarchical, Deep-Learning Photometric Classifier for the LSST
Ved G. Shah, Alex Gagliano, Konstantin Malanchev, Gautham Narayan, Alex I. Malz, The LSST Dark Energy Science Collaboration
We present ORACLE, the first hierarchical deep-learning model for real-time, context-aware classification of transient and variable astrophysical phenomena. ORACLE is a recurrent neural network with Gated Recurrent Units (GRUs), and has been trained using a custom hierarchical cross-entropy loss function to provide high-confidence classifications along an observationally-driven taxonomy with as little as a single photometric observation. Contextual information for each object, including host galaxy photometric redshift, offset, ellipticity and brightness, is concatenated to the light curve embedding and used to make a final prediction. Training on $\sim$0.5M events from the Extended LSST Astronomical Time-Series Classification Challenge, we achieve a top-level (Transient vs Variable) macro-averaged precision of 0.96 using only 1 day of photometric observations after the first detection in addition to contextual information, for each event; this increases to $>$0.99 once 64 days of the light curve has been obtained, and 0.83 at 1024 days after first detection for 19-way classification (including supernova sub-types, active galactic nuclei, variable stars, microlensing events, and kilonovae). We also compare ORACLE with other state-of-the-art classifiers and report comparable performance for the 19-way classification task, in addition to delivering accurate top-level classifications much earlier. The code and model weights used in this work are publicly available at our associated GitHub repository (https://github.com/uiucsn/ELAsTiCC-Classification).
comment : 30 pages, 20 figures, 9 tables. Accepted to ApJ
♻ ☆ ERF-BA-TFD+: A Multimodal Model for Audio-Visual Deepfake Detection
Xin Zhang, Jiaming Chu, Jian Zhao, Yuchu Jiang, Xu Yang, Lei Jin, Chi Zhang, Xuelong Li
Deepfake detection is a critical task in identifying manipulated multimedia content. In real-world scenarios, deepfake content can manifest across multiple modalities, including audio and video. To address this challenge, we present ERF-BA-TFD+, a novel multimodal deepfake detection model that combines enhanced receptive field (ERF) and audio-visual fusion. Our model processes both audio and video features simultaneously, leveraging their complementary information to improve detection accuracy and robustness. The key innovation of ERF-BA-TFD+ lies in its ability to model long-range dependencies within the audio-visual input, allowing it to better capture subtle discrepancies between real and fake content. In our experiments, we evaluate ERF-BA-TFD+ on the DDL-AV dataset, which consists of both segmented and full-length video clips. Unlike previous benchmarks, which focused primarily on isolated segments, the DDL-AV dataset allows us to assess the model's performance in a more comprehensive and realistic setting. Our method achieves state-of-the-art results on this dataset, outperforming existing techniques in terms of both accuracy and processing speed. The ERF-BA-TFD+ model demonstrated its effectiveness in the "Workshop on Deepfake Detection, Localization, and Interpretability," Track 2: Audio-Visual Detection and Localization (DDL-AV), and won first place in this competition.
comment : The paper is withdrawn after discovering a flaw in the theoretical derivation presented in Section Method. The incorrect step leads to conclusions that are not supported by the corrected derivation. We plan to reconstruct the argument and will release an updated version once the issue is fully resolved
♻ ☆ A Survey on Recommendation Unlearning: Fundamentals, Taxonomy, Evaluation, and Open Questions
Yuyuan Li, Xiaohua Feng, Chaochao Chen, Qiang Yang
Recommender systems have become increasingly influential in shaping user behavior and decision-making, highlighting their growing impact in various domains. Meanwhile, the widespread adoption of machine learning models in recommender systems has raised significant concerns regarding user privacy and security. As compliance with privacy regulations becomes more critical, there is a pressing need to address the issue of recommendation unlearning, i.e., eliminating the memory of specific training data from the learned recommendation models. Despite its importance, traditional machine unlearning methods are ill-suited for recommendation unlearning due to the unique challenges posed by collaborative interactions and model parameters. This survey offers a comprehensive review of the latest advancements in recommendation unlearning, exploring the design principles, challenges, and methodologies associated with this emerging field. We provide a unified taxonomy that categorizes different recommendation unlearning approaches, followed by a summary of widely used benchmarks and metrics for evaluation. By reviewing the current state of research, this survey aims to guide the development of more efficient, scalable, and robust recommendation unlearning techniques. Furthermore, we identify open research questions in this field, which could pave the way for future innovations not only in recommendation unlearning but also in a broader range of unlearning tasks across different machine learning applications.
comment : Accepted by TKDE
♻ ☆ Access Paths for Efficient Ordering with Large Language Models
Fuheng Zhao, Jiayue Chen, Yiming Pan, Tahseen Rabbani, Sohaib, Divyakant Agrawal, Amr El Abbadi, Paritosh Aggarwal, Anupam Datta, Dimitris Tsirogiannis
In this work, we present the \texttt{LLM ORDER BY} semantic operator as a logical abstraction and conduct a systematic study of its physical implementations. First, we propose several improvements to existing semantic sorting algorithms and introduce a semantic-aware external merge sort algorithm. Our extensive evaluation reveals that no single implementation offers universal optimality on all datasets. From our evaluations, we observe a general test-time scaling relationship between sorting cost and the ordering quality for comparison-based algorithms. Building on these insights, we design a budget-aware optimizer that utilizes heuristic rules, LLM-as-Judge evaluation, and consensus aggregation to dynamically select the near-optimal access path for LLM ORDER BY. In our extensive evaluations, our optimizer consistently achieves ranking accuracy on par with or superior to the best static methods across all benchmarks. We believe that this work provides foundational insights into the principled optimization of semantic operators essential for building robust, large-scale LLM-powered analytic systems.
♻ ☆ A Novel Attention-Augmented Wavelet YOLO System for Real-time Brain Vessel Segmentation on Transcranial Color-coded Doppler
Wenxuan Zhang, Shuai Li, Xinyi Wang, Yu Sun, Hongyu Kang, Pui Yuk Chryste Wan, Jing Qin, Yuanpeng Zhang, Yong-Ping Zheng, Sai-Kit Lam
The Circle of Willis (CoW), vital for ensuring consistent blood flow to the brain, is closely linked to ischemic stroke. Accurate assessment of the CoW is important for identifying individuals at risk and guiding appropriate clinical management. Among existing imaging methods, Transcranial Color-coded Doppler (TCCD) offers unique advantages due to its radiation-free nature, affordability, and accessibility. However, reliable TCCD assessments depend heavily on operator expertise for identifying anatomical landmarks and performing accurate angle correction, which limits its widespread adoption. To address this challenge, we propose an AI-powered, real-time CoW auto-segmentation system capable of efficiently capturing cerebral arteries. No prior studies have explored AI-driven cerebrovascular segmentation using TCCD. In this work, we introduce a novel Attention-Augmented Wavelet YOLO (AAW-YOLO) network tailored for TCCD data, designed to provide real-time guidance for brain vessel segmentation in the CoW. We prospectively collected TCCD data comprising 738 annotated frames and 3,419 labeled artery instances to establish a high-quality dataset for model training and evaluation. The proposed AAW-YOLO demonstrated strong performance in segmenting both ipsilateral and contralateral CoW vessels, achieving an average Dice score of 0.901, IoU of 0.823, precision of 0.882, recall of 0.926, and mAP of 0.953, with a per-frame inference speed of 14.199 ms. This system offers a practical solution to reduce reliance on operator experience in TCCD-based cerebrovascular screening, with potential applications in routine clinical workflows and resource-constrained settings. Future research will explore bilateral modeling and larger-scale validation.
♻ ☆ Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Train ing
Changhao Jiang, Ming Zhang, Yifei Cao, Junjie Ye, Xiaoran Fan, Shihan Dou, Zhiheng Xi, Jiajun Sun, Yi Dong, Yujiong Shen, Jingqi Tong, Baoyu Fan, Qi Zhang, Tao Gui, Xuanjing Huang
The GPT-4 technical report suggests that downstream performance can be predicted from pre-training signals, but offers little methodological detail on how to quantify this. This work address this gap by modeling knowledge retention, the capacity of a pre-trained language model to memorize factual information from its corpus, and introduce a principled method to estimate it prior to training. We propose Size-dependent Mutual Information (SMI), an information-theoretic predictor that integrates knowledge frequency, knowledge specificity, and model size to forecast closed-book question answering (QA) accuracy. SMI is validated through large-scale document retrieval over the disclosed pre-training corpora of 21 public and 3 custom models, combined with a robust multi-template QA evaluation. Experiments show that SMI significantly outperforms repetition-based baselines and achieves $R^2$ > 0.7 in predicting QA accuracy for models above 1B parameters, without additional training. The analysis further reveals diminishing returns from scaling data and model size and provides evidence for an intrinsic upper bound on knowledge retention achievable by pre-training alone, motivating retrieval and other augmentation strategies.
comment : This work is withdrawn as all authors are not in agreement on the work
♻ ☆ IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web
Hongcheng Guo, Wei Zhang, Junhao Chen, Yaonan Gu, Jian Yang, Junjia Du, Shaosheng Cao, Binyuan Hui, Tianyu Liu, Jianxin Ma, Chang Zhou, Zhoujun Li
Recently advancements in large multimodal models have led to significant strides in image comprehension capabilities. Despite these advancements, there is a lack of the robust benchmark specifically for assessing the Image-to-Web conversion proficiency of these large models. Primarily, it is essential to ensure the integrity of the web elements generated. These elements comprise visible and invisible categories. Previous evaluation methods (e.g.,BLEU) are notably susceptible to significant alterations due to the presence of invisible elements in Web. Furthermore, it is crucial to measure the layout information of web pages, referring to the positional relationships between elements, which is overlooked by previous work. To address challenges, we have curated and aligned a benchmark of images and corresponding web codes (IW-BENCH). Specifically, we propose the Element Accuracy, which tests the completeness of the elements by parsing the Document Object Model (DOM) tree. Layout Accuracy is also proposed to analyze the positional relationships of elements by converting DOM tree into a common subsequence. Besides, we design a five-hop multimodal Chain-of-Thought Prompting for better performance, which contains five hop: 1) SoM prompt injection. 2) Inferring Elements. 3) Inferring Layout. 4) Inferring Web code. 5) Reflection. Our benchmark comprises 1200 pairs of images and web codes with varying levels of difficulty. We have conducted extensive experiments on existing large multimodal models, offering insights into their performance and areas for improvement in image-to-web domain.
♻ ☆ SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scaling
Jiefeng Chen, Jie Ren, Xinyun Chen, Chengrun Yang, Ruoxi Sun, Jinsung Yoon, Sercan Ö Arık
Recent advancements in Large Language Models (LLMs) have created new opportunities to enhance performance on complex reasoning tasks by leveraging test-time computation. However, existing scaling methods have key limitations: parallel methods like repeated sampling are often inefficient and quickly saturate, while sequential methods like SELF-REFINE struggle to improve after a few rounds. Although combining these approaches shows promise, current methods require fine-tuned reward and revision models. This paper proposes Self-Enhanced Test-Time Scaling (SETS), a simple yet effective approach that overcomes these limitations by strategically combining parallel and sequential techniques and fully leveraging LLMs' self-improvement abilities. SETS exploits the inherent self-verification and self-correction capabilities of LLMs, unifying sampling, verification, and correction within a single framework. This facilitates efficient and scalable test-time computation for enhanced performance on complex tasks without any model training. Our comprehensive experimental results on challenging benchmarks spanning planning, reasoning, math, and coding demonstrate that SETS achieves significant performance improvements and more advantageous test-time scaling behavior than the alternatives.
comment : Published in Transactions on Machine Learning Research (11/2025)
♻ ☆ COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers
Junyu Wang, Changjia Zhu, Yuanbo Zhou, Lingyao Li, Xu He, Junjie Xiong
This paper studies how multimodal large language models (MLLMs) undermine the security guarantees of visual CAPTCHA. We identify the attack surface where an adversary can cheaply automate CAPTCHA solving using off-the-shelf models. We evaluate 7 leading commercial and open-source MLLMs across 18 real-world CAPTCHA task types, measuring single-shot accuracy, success under limited retries, end-to-end latency, and per-solve cost. We further analyze the impact of task-specific prompt engineering and few-shot demonstrations on solver effectiveness. We reveal that MLLMs can reliably solve recognition-oriented and low-interaction CAPTCHA tasks at human-like cost and latency, whereas tasks requiring fine-grained localization, multi-step spatial reasoning, or cross-frame consistency remain significantly harder for current models. By examining the reasoning traces of such MLLMs, we investigate the underlying mechanisms of why models succeed/fail on specific CAPTCHA puzzles and use these insights to derive defense-oriented guidelines for selecting and strengthening CAPTCHA tasks. We conclude by discussing implications for platform operators deploying CAPTCHA as part of their abuse-mitigation pipeline.Code Availability (https://anonymous.4open.science/r/Captcha-465E/).
♻ ☆ Online Learning-based Adaptive Beam Switching for 6G Networks: Enhancing Efficiency and Resilience
Seyed Bagher Hashemi Natanzi, Zhicong Zhu, Bo Tang
Adaptive beam switching is essential for mission-critical military and commercial 6G networks but faces major challenges from high carrier frequencies, user mobility, and frequent blockages. While existing machine learning (ML) solutions often focus on maximizing instantaneous throughput, this can lead to unstable policies with high signaling overhead. This paper presents an online Deep Reinforcement Learning (DRL) framework designed to learn an operationally stable policy. By equipping the DRL agent with an enhanced state representation that includes blockage history, and a stability-centric reward function, we enable it to prioritize long-term link quality over transient gains. Validated in a challenging 100-user scenario using the Sionna library, our agent achieves throughput comparable to a reactive Multi-Armed Bandit (MAB) baseline. Specifically, our proposed framework improves link stability by approximately 43% compared to a vanilla DRL approach, achieving operational reliability competitive with MAB while maintaining high data rates. This work demonstrates that by reframing the optimization goal towards operational stability, DRL can deliver efficient, reliable, and real-time beam management solutions for next-generation mission-critical networks.
♻ ☆ Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models AAAI-2026
Xin Liu, Qiyang Song, Qihang Zhou, Haichao Du, Shaowen Xu, Wenbo Jiang, Weijuan Zhang, Xiaoqi Jia
Large language models (LLMs) increasingly support multilingual understanding and generation. Meanwhile, efforts to interpret their internal mechanisms have emerged, offering insights to enhance multilingual performance. While multi-head self-attention (MHA) has proven critical in many areas, its role in multilingual capabilities remains underexplored. In this work, we study the contribution of MHA in supporting multilingual processing in LLMs. We propose Language Attention Head Importance Scores (LAHIS), an effective and efficient method that identifies attention head importance for multilingual capabilities via a single forward and backward pass through the LLM. Applying LAHIS to Aya-23-8B, Llama-3.2-3B, and Mistral-7B-v0.1, we reveal the existence of both language-specific and language-general heads. Language-specific heads enable cross-lingual attention transfer to guide the model toward target language contexts and mitigate off-target language generation issue, contributing to addressing challenges in multilingual LLMs. We also introduce a lightweight adaptation that learns a soft head mask to modulate attention outputs over language heads, requiring only 20 tunable parameters to improve XQuAD accuracy. Overall, our work enhances both the interpretability and multilingual capabilities of LLMs from the perspective of MHA.
comment : Accepted by AAAI-2026
♻ ☆ Menta: A Small Language Model for On-Device Mental Health Prediction
Tianyi Zhang, Xiangyuan Xue, Lingyan Ruan, Shiya Fu, Feng Xia, Simon D'Alfonso, Vassilis Kostakos, Ting Dang, Hong Jia
Mental health conditions affect hundreds of millions globally, yet early detection remains limited. While large language models (LLMs) have shown promise in mental health applications, their size and computational demands hinder practical deployment. Small language models (SLMs) offer a lightweight alternative, but their use for social media--based mental health prediction remains largely underexplored. In this study, we introduce Menta, the first optimized SLM fine-tuned specifically for multi-task mental health prediction from social media data. Menta is jointly trained across six classification tasks using a LoRA-based framework, a cross-dataset strategy, and a balanced accuracy--oriented loss. Evaluated against nine state-of-the-art SLM baselines, Menta achieves an average improvement of 15.2\% across tasks covering depression, stress, and suicidality compared with the best-performing non--fine-tuned SLMs. It also achieves higher accuracy on depression and stress classification tasks compared to 13B-parameter LLMs, while being approximately 3.25x smaller. Moreover, we demonstrate real-time, on-device deployment of Menta on an iPhone 15 Pro Max, requiring only approximately 3GB RAM. Supported by a comprehensive benchmark against existing SLMs and LLMs, Menta highlights the potential for scalable, privacy-preserving mental health monitoring. Code is available at: https://xxue752-nz.github.io/menta-project/
♻ ☆ Large Language Model-Based Agent s for Software Engineering: A Survey
Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, Yiling Lou
The recent advance in Large Language Models (LLMs) has shaped a new paradigm of AI agents, i.e., LLM-based agents. Compared to standalone LLMs, LLM-based agents substantially extend the versatility and expertise of LLMs by enhancing LLMs with the capabilities of perceiving and utilizing external resources and tools. To date, LLM-based agents have been applied and shown remarkable effectiveness in Software Engineering (SE). The synergy between multiple agents and human interaction brings further promise in tackling complex real-world SE problems. In this work, we present a comprehensive and systematic survey on LLM-based agents for SE. We collect 124 papers and categorize them from two perspectives, i.e., the SE and agent perspectives. In addition, we discuss open challenges and future directions in this critical domain. The repository of this survey is at https://github.com/FudanSELab/Agent4SE-Paper-List.
comment : Accepted by TOSEM
♻ ☆ MathBode: Measuring the Stability of LLM Reasoning using Frequency Response
Charles L. Wang
This paper presents MathBode, a dynamic diagnostic for mathematical reasoning in large language models (LLMs). Instead of one-shot accuracy, MathBode treats each parametric problem as a system: we drive a single parameter sinusoidally and fit first-harmonic responses of model outputs and exact solutions. This yields interpretable, frequency-resolved metrics -- gain (amplitude tracking) and phase (lag) -- that form Bode-style fingerprints. Across five closed-form families (linear solve, ratio/saturation, compound interest, 2x2 linear systems, similar triangles), the diagnostic surfaces systematic low-pass behavior and growing phase lag that accuracy alone obscures. We compare several models against a symbolic baseline that calibrates the instrument ($G \approx 1$, $φ\approx 0$). Results separate frontier from mid-tier models on dynamics, providing a compact, reproducible protocol that complements standard benchmarks with actionable measurements of reasoning fidelity and consistency. We open-source the dataset and code to enable further research and adoption.
♻ ☆ Mobile- Agent -RAG: Driving Smart Multi- Agent Coordination with Context ual Knowledge Empowerment for Long-Horizon Mobile Automation
Yuxiang Zhou, Jichang Li, Yanhao Zhang, Haonan Lu, Guanbin Li
Mobile agents show immense potential, yet current state-of-the-art (SoTA) agents exhibit inadequate success rates on real-world, long-horizon, cross-application tasks. We attribute this bottleneck to the agents' excessive reliance on static, internal knowledge within MLLMs, which leads to two critical failure points: 1) strategic hallucinations in high-level planning and 2) operational errors during low-level execution on user interfaces (UI). The core insight of this paper is that high-level planning and low-level UI operations require fundamentally distinct types of knowledge. Planning demands high-level, strategy-oriented experiences, whereas operations necessitate low-level, precise instructions closely tied to specific app UIs. Motivated by these insights, we propose Mobile-Agent-RAG, a novel hierarchical multi-agent framework that innovatively integrates dual-level retrieval augmentation. At the planning stage, we introduce Manager-RAG to reduce strategic hallucinations by retrieving human-validated comprehensive task plans that provide high-level guidance. At the execution stage, we develop Operator-RAG to improve execution accuracy by retrieving the most precise low-level guidance for accurate atomic actions, aligned with the current app and subtask. To accurately deliver these knowledge types, we construct two specialized retrieval-oriented knowledge bases. Furthermore, we introduce Mobile-Eval-RAG, a challenging benchmark for evaluating such agents on realistic multi-app, long-horizon tasks. Extensive experiments demonstrate that Mobile-Agent-RAG significantly outperforms SoTA baselines, improving task completion rate by 11.0% and step efficiency by 10.2%, establishing a robust paradigm for context-aware, reliable multi-agent mobile automation.
♻ ☆ SPRINT: Enabling Inte rl eaved Planning and Parallelized Execution in Reasoning Models NeurIPS 2025
Emil Biju, Shayan Talaei, Zhemin Huang, Mohammadreza Pourreza, Azalia Mirhoseini, Amin Saberi
Large reasoning models (LRMs) excel at complex reasoning tasks but typically generate lengthy sequential chains-of-thought, resulting in long inference times before arriving at the final answer. To address this challenge, we introduce SPRINT, a novel post-training and inference-time framework designed to enable LRMs to dynamically identify and exploit opportunities for parallelization during their reasoning process. SPRINT incorporates an innovative data curation pipeline that reorganizes natural language reasoning trajectories into structured rounds of long-horizon planning and parallel execution. By fine-tuning LRMs on a small amount of such curated data, the models learn to dynamically identify independent subtasks within extended reasoning processes and effectively execute them in parallel. Through extensive evaluations, we demonstrate that models fine-tuned with the SPRINT framework match the performance of reasoning models on complex domains such as mathematics while generating up to 39% fewer sequential tokens on problems requiring more than 8,000 output tokens. Finally, we observe consistent results transferred to two out-of-distribution tasks, namely GPQA and Countdown, with up to 45% and 65% reduction in average sequential tokens respectively for longer reasoning trajectories, while matching the performance of the fine-tuned reasoning model.
comment : Published at NeurIPS 2025. Emil Biju, Shayan Talaei, and Zhemin Huang contributed equally to this work
♻ ☆ GAMA: A Neural Neighborhood Search Method with Graph-aware Multi-modal Attention for Vehicle Routing Problem
Xiangling Chen, Yi Mei, Mengjie Zhang
Recent advances in neural neighborhood search methods have shown potential in tackling Vehicle Routing Problems (VRPs). However, most existing approaches rely on simplistic state representations and fuse heterogeneous information via naive concatenation, limiting their ability to capture rich structural and semantic context. To address these limitations, we propose GAMA, a neural neighborhood search method with Graph-aware Multi-modal Attention model in VRP. GAMA encodes the problem instance and its evolving solution as distinct modalities using graph neural networks, and models their intra- and inter-modal interactions through stacked self- and cross-attention layers. A gated fusion mechanism further integrates the multi-modal representations into a structured state, enabling the policy to make informed and generalizable operator selection decisions. Extensive experiments conducted across various synthetic and benchmark instances demonstrate that the proposed algorithm GAMA significantly outperforms the recent neural baselines. Further ablation studies confirm that both the multi-modal attention mechanism and the gated fusion design play a key role in achieving the observed performance gains.
♻ ☆ Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference NDSS
Zhifan Luo, Shuo Shao, Su Zhang, Lijing Zhou, Yuke Hu, Chenxu Zhao, Zhihao Liu, Zhan Qin
The Key-Value (KV) cache, which stores intermediate attention computations (Key and Value pairs) to avoid redundant calculations, is a fundamental mechanism for accelerating Large Language Model (LLM) inference. However, this efficiency optimization introduces significant yet underexplored privacy risks. This paper provides the first comprehensive analysis of these vulnerabilities, demonstrating that an attacker can reconstruct sensitive user inputs directly from the KV-cache. We design and implement three distinct attack vectors: a direct Inversion Attack, a more broadly applicable and potent Collision Attack, and a semantic-based Injection Attack. These methods demonstrate the practicality and severity of KV-cache privacy leakage issues. To mitigate this, we propose KV-Cloak, a novel, lightweight, and efficient defense mechanism. KV-Cloak uses a reversible matrix-based obfuscation scheme, combined with operator fusion, to secure the KV-cache. Our extensive experiments show that KV-Cloak effectively thwarts all proposed attacks, reducing reconstruction quality to random noise. Crucially, it achieves this robust security with virtually no degradation in model accuracy and minimal performance overhead, offering a practical solution for trustworthy LLM deployment.
comment : This paper is accepted by Network and Distributed System Security Symposium (NDSS) 2026
♻ ☆ Cross-Field Interface-Aware Neural Operators for Multiphase Flow Simulation
Zhenzhong Wang, Xin Zhang, Jun Liao, Min Jiang
Multiphase flow simulation is critical in science and engineering but incurs high computational costs due to complex field discontinuities and the need for high-resolution numerical meshes. While Neural Operators (NOs) offer an efficient alternative for solving Partial Differential Equations (PDEs), they struggle with two core challenges unique to multiphase systems: spectral bias caused by spatial heterogeneity at phase interfaces, and the persistent scarcity of expensive, high-resolution field data. This work introduces the Interface Information Aware Neural Operator (IANO), a novel architecture that mitigates these issues by leveraging readily obtainable interface data (e.g., topology and position). Interface data inherently contains the high-frequency features not only necessary to complement the physical field data, but also help with spectral bias. IANO incorporates an interface-aware function encoding mechanism to capture dynamic coupling, and a geometry-aware positional encoding method to enhance spatial fidelity for pointwise super-resolution. Empirical results across multiple multiphase flow cases demonstrate that IANO achieves significant accuracy improvements (up to $\sim$10\%) over existing NO baselines. Furthermore, IANO exhibits superior generalization capabilities in low-data and noisy settings, confirming its utility for practical, data-efficient $\text{AI}$-based multiphase flow simulations.
♻ ☆ Look, Recite, Then Answer: Enhancing VLM Performance via Self-Generated Knowledge Hints
Xisheng Feng
Vision-Language Models (VLMs) exhibit significant performance plateaus in specialized domains like precision agriculture, primarily due to "Reasoning-Driven Hallucination" where linguistic priors override visual perception. A key bottleneck is the "Modality Gap": visual embeddings fail to reliably activate the fine-grained expert knowledge already encoded in model parameters. We propose "Look, Recite, Then Answer," a parameter-efficient framework that enhances VLMs via self-generated knowledge hints while keeping backbone models frozen. The framework decouples inference into three stages: (1) Look generates objective visual descriptions and candidate sets; (2) Recite employs a lightweight 1.7B router to transform visual cues into targeted queries that trigger candidate-specific parametric knowledge; (3) Answer performs parallel evidence alignment between descriptions and recited knowledge to select the most consistent label. On AgroBench, our method achieves state-of-the-art results, improving Weed Identification accuracy by 23.52% over Qwen2-VL-72B and surpassing GPT-4o without external search overhead. This modular design mitigates hallucinations by transforming passive perception into active, controllable knowledge retrieval
♻ ☆ MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding
Fan Yang, Kaihao Zhang
Understanding high-resolution images remains a significant challenge for multimodal large language models (MLLMs). Recent study address this issue by dividing the image into smaller crops and computing the semantic similarity between each crop and a query using a pretrained retrieval-augmented generation (RAG) model. The most relevant crops are then selected to localize the target object and suppress irrelevant information. However, such crop-based processing can fragment complete objects across multiple crops, thereby disrupting the computation of semantic similarity. In our experiments, we find that image crops of objects with different sizes are better handled at different resolutions. Based on this observation, we propose Multi-resolution Retrieval-Detection (MRD), a training-free framework for high-resolution image understanding. To address the issue of semantic similarity bias caused by objects being split across different image crops, we propose a multi-resolution semantic fusion method, which integrates semantic similarity maps obtained at different resolutions to produce more accurate semantic information and preserve the integrity of target objects. Furthermore, to achieve direct localization of target objects at a global scale, we introduce an open-vocalbulary object detection (OVD) model that identifies object regions using a sliding-window approach.Experiments on high-resolution image understanding benchmarks using different MLLMs demonstrate the effectiveness of our approach.
♻ ☆ A$^2$LC: Active and Automated Label Correction for Semantic Segmentation AAAI 2026
Youjin Jeon, Kyusik Cho, Suhan Woo, Euntai Kim
Active Label Correction (ALC) has emerged as a promising solution to the high cost and error-prone nature of manual pixel-wise annotation in semantic segmentation, by actively identifying and correcting mislabeled data. Although recent work has improved correction efficiency by generating pseudo-labels using foundation models, substantial inefficiencies still remain. In this paper, we introduce A$^2$LC, an Active and Automated Label Correction framework for semantic segmentation, where manual and automatic correction stages operate in a cascaded manner. Specifically, the automatic correction stage leverages human feedback to extend label corrections beyond the queried samples, thereby maximizing cost efficiency. In addition, we introduce an adaptively balanced acquisition function that emphasizes underrepresented tail classes, working in strong synergy with the automatic correction stage. Extensive experiments on Cityscapes and PASCAL VOC 2012 demonstrate that A$^2$LC significantly outperforms previous state-of-the-art methods. Notably, A$^2$LC exhibits high efficiency by outperforming previous methods with only 20% of their budget, and shows strong effectiveness by achieving a 27.23% performance gain under the same budget on Cityscapes.
comment : Accepted by AAAI 2026
♻ ☆ AugMapNet: Improving Spatial Latent Structure via BEV Grid Augmentation for Enhanced Vectorized Online HD Map Construction WACV 2026
Thomas Monninger, Md Zafar Anwar, Stanislaw Antol, Steffen Staab, Sihao Ding
Autonomous driving requires understanding infrastructure elements, such as lanes and crosswalks. To navigate safely, this understanding must be derived from sensor data in real-time and needs to be represented in vectorized form. Learned Bird's-Eye View (BEV) encoders are commonly used to combine a set of camera images from multiple views into one joint latent BEV grid. Traditionally, from this latent space, an intermediate raster map is predicted, providing dense spatial supervision but requiring post-processing into the desired vectorized form. More recent models directly derive infrastructure elements as polylines using vectorized map decoders, providing instance-level information. Our approach, Augmentation Map Network (AugMapNet), proposes latent BEV feature grid augmentation, a novel technique that significantly enhances the latent BEV representation. AugMapNet combines vector decoding and dense spatial supervision more effectively than existing architectures while remaining easy to integrate compared to other hybrid approaches. It additionally benefits from extra processing on its latent BEV features. Experiments on nuScenes and Argoverse2 datasets demonstrate significant improvements on vectorized map prediction of up to 13.3% over the StreamMapNet baseline on 60 m range and greater improvements on larger ranges. We confirm transferability by applying our method to another baseline, SQD-MapNet, and find similar improvements. A detailed analysis of the latent BEV grid confirms a more structured latent space of AugMapNet and shows the value of our novel concept beyond pure performance improvement. The code can be found at https://github.com/tmonnin/augmapnet
comment : Accepted to 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2026)
♻ ☆ Scaling Multimodal Search and Recommendation with Small Language Models via Upside-Down Reinforcement Learning ICDM 2025
Yu-Chen Lin, Sanat Sharma, Hari Manikandan, Jayant Kumar, Tracy Holloway King, Jing Zheng
In this work, we investigate how small language models (SLMs) can be scaled to support multimodal search and recommendation use cases while remaining efficient enough for real-time, resource-constrained deployments. We present a framework that combines upside-down reinforcement learning with synthetic data distillation from a large language model (Llama-3) to train a 100M-parameter GPT-2 model for multitask prompt generation. Despite being up to 80 times smaller than state-of-the-art large language models (LLMs), our SLM achieves relevance and diversity scores within 6% of competitive baselines such as Llama-3 8B, Qwen3 8B, and Ministral 8B. These results demonstrate that SLMs can effectively handle multimodal search and recommendation tasks, while dramatically reducing inference latency and memory overhead. Our study highlights the potential of lightweight models as practical engines for scalable multimodal discovery, bridging the gap between cutting-edge research and real-world multimodal applications such as media recommendations and creative content generation.
comment : Accepted by ICDM 2025 MMSR
♻ ☆ Causal LLM Routing: End-to-End Regret Minimization from Observational Data
Asterios Tsiourvas, Wei Sun, Georgia Perakis
LLM routing aims to select the most appropriate model for each query, balancing competing performance metrics such as accuracy and cost across a pool of language models. Prior approaches typically adopt a decoupled strategy, where the metrics are first predicted and the model is then selected based on these estimates. This setup is prone to compounding errors and often relies on full-feedback data, where each query is evaluated by all candidate models, which is costly to obtain and maintain in practice. In contrast, we learn from observational data, which records only the outcome of the model actually deployed. We propose a causal end-to-end framework that learns routing policies by minimizing decision-making regret from observational data. To enable efficient optimization, we introduce two theoretically grounded surrogate objectives: a classification-based upper bound, and a softmax-weighted regret approximation shown to recover the optimal policy at convergence. We further extend our framework to handle heterogeneous cost preferences via an interval-conditioned architecture. Experiments on public benchmarks show that our method outperforms existing baselines, achieving state-of-the-art performance across different embedding models.
♻ ☆ A Diffusion Model Framework for Maximum Entropy Reinforcement Learning
Sebastian Sanokowski, Kaustubh Patil, Alois Knoll
Diffusion models have achieved remarkable success in data-driven learning and in sampling from complex, unnormalized target distributions. Building on this progress, we reinterpret Maximum Entropy Reinforcement Learning (MaxEntRL) as a diffusion model-based sampling problem. We tackle this problem by minimizing the reverse Kullback-Leibler (KL) divergence between the diffusion policy and the optimal policy distribution using a tractable upper bound. By applying the policy gradient theorem to this objective, we derive a modified surrogate objective for MaxEntRL that incorporates diffusion dynamics in a principled way. This leads to simple diffusion-based variants of Soft Actor-Critic (SAC), Proximal Policy Optimization (PPO) and Wasserstein Policy Optimization (WPO), termed DiffSAC, DiffPPO and DiffWPO. All of these methods require only minor implementation changes to their base algorithm. We find that on standard continuous control benchmarks, DiffSAC, DiffPPO and DiffWPO achieve better returns and higher sample efficiency than SAC and PPO.
♻ ☆ Privacy is All You Need: Revolutionizing Wearable Health Data with Advanced PETs
Karthik Barma
In a world where data is the new currency, wearable health devices offer unprecedented insights into daily life, continuously monitoring vital signs and metrics. However, this convenience raises privacy concerns, as these devices collect sensitive data that can be misused or breached. Traditional measures often fail due to real-time data processing needs and limited device power. Users also lack awareness and control over data sharing and usage. We propose a Privacy-Enhancing Technology (PET) framework for wearable devices, integrating federated learning, lightweight cryptographic methods, and selectively deployed blockchain technology. The blockchain acts as a secure ledger triggered only upon data transfer requests, granting users real-time notifications and control. By dismantling data monopolies, this approach returns data sovereignty to individuals. Through real-world applications like secure medical data sharing, privacy-preserving fitness tracking, and continuous health monitoring, our framework reduces privacy risks by up to 70 percent while preserving data utility and performance. This innovation sets a new benchmark for wearable privacy and can scale to broader IoT ecosystems, including smart homes and industry. As data continues to shape our digital landscape, our research underscores the critical need to maintain privacy and user control at the forefront of technological progress.
♻ ★ A Definition of AGI
Dan Hendrycks, Dawn Song, Christian Szegedy, Honglak Lee, Yarin Gal, Erik Brynjolfsson, Sharon Li, Andy Zou, Lionel Levine, Bo Han, Jie Fu, Ziwei Liu, Jinwoo Shin, Kimin Lee, Mantas Mazeika, Long Phan, George Ingebretsen, Adam Khoja, Cihang Xie, Olawale Salaudeen, Matthias Hein, Kevin Zhao, Alexander Pan, David Duvenaud, Bo Li, Steve Omohundro, Gabriel Alfour, Max Tegmark, Kevin McGrew, Gary Marcus, Jaan Tallinn, Eric Schmidt, Yoshua Bengio
The lack of a concrete definition for Artificial General Intelligence (AGI) obscures the gap between today's specialized AI and human-level cognition. This paper introduces a quantifiable framework to address this, defining AGI as matching the cognitive versatility and proficiency of a well-educated adult. To operationalize this, we ground our methodology in Cattell-Horn-Carroll theory, the most empirically validated model of human cognition. The framework dissects general intelligence into ten core cognitive domains-including reasoning, memory, and perception-and adapts established human psychometric batteries to evaluate AI systems. Application of this framework reveals a highly "jagged" cognitive profile in contemporary models. While proficient in knowledge-intensive domains, current AI systems have critical deficits in foundational cognitive machinery, particularly long-term memory storage. The resulting AGI scores (e.g., GPT-4 at 27%, GPT-5 at 57%) concretely quantify both rapid progress and the substantial gap remaining before AGI.
♻ ☆ Semantic Mastery: Enhancing LLMs with Advanced Natural Language Understanding
Mohanakrishnan Hariharan
Large language models (LLMs) have greatly improved their capability in performing NLP tasks. However, deeper semantic understanding, contextual coherence, and more subtle reasoning are still difficult to obtain. The paper discusses state-of-the-art methodologies that advance LLMs with more advanced NLU techniques, such as semantic parsing, knowledge integration, and contextual reinforcement learning. We analyze the use of structured knowledge graphs, retrieval-augmented generation (RAG), and fine-tuning strategies that match models with human-level understanding. Furthermore, we address the incorporation of transformer-based architectures, contrastive learning, and hybrid symbolic-neural methods that address problems like hallucinations, ambiguity, and inconsistency in the factual perspectives involved in performing complex NLP tasks, such as question-answering text summarization and dialogue generation. Our findings show the importance of semantic precision for enhancing AI-driven language systems and suggest future research directions to bridge the gap between statistical language models and true natural language understanding.
♻ ☆ PRO-V-R1: Reasoning Enhanced Programming Agent for RTL Verification
Yujie Zhao, Zhijing Wu, Boqin Yuan, Zhongming Yu, Hejia Zhang, Wentao Ni, Chia-Tung Ho, Haoxing Ren, Jishen Zhao
Register-Transfer Level (RTL) verification is a primary bottleneck, consuming 60-70% of development time. While Large Language Models (LLMs) show promise for RTL automation, their performance and research focus have overwhelmingly centered on RTL generation rather than verification. Current methods for RTL verification rely on large scale proprietary models (e.g., GPT-4o) to generate Python-based functional references, incurring a high cost and raising data-privacy risks. To date, an end-to-end open-source solution for autonomous verification remains absent. We introduce PRO-V-R1, the first trainable open-source agentic framework for autonomous RTL verification. Our contributions are threefold: (1) we design PRO-V sys, a modular agentic system that couples LLM-based reasoning with programmatic tool use for RTL verification; (2) we establish a data construction pipeline that leverages existing RTL datasets to build simulation-validated, expert-level trajectories tailored for supervised fine-tuning (SFT) RTL verification agents; and (3) we implement an efficient reinforcement learning (RL) algorithm that uses verification-specific rewards derived from program-tool feedback to optimize the end-to-end verification workflow. Our empirical evaluation demonstrates PRO-V-R1 achieves a 57.7% functional correctness rate and 34.0% in robust fault detection, significantly outperforming the base model's 25.7% and 21.8% (respectively) from the state-of-the-art (SOTA) automatic verification system. This configuration also outperforms large-scale proprietary LLMs in functional correctness and shows comparable robustness for fault detection.
♻ ☆ Let the Trial Begin: A Mock-Court Approach to Vulnerability Detection using LLM-Based Agent s
Ratnadira Widyasari, Martin Weyssow, Ivana Clairine Irsan, Han Wei Ang, Frank Liauw, Eng Lieh Ouh, Lwin Khin Shar, Hong Jin Kang, David Lo
Detecting vulnerabilities in source code remains a critical yet challenging task, especially when benign and vulnerable functions share significant similarities. In this work, we introduce VulTrial, a courtroom-inspired multi-agent framework designed to identify vulnerable code and to provide explanations. It employs four role-specific agents, which are security researcher, code author, moderator, and review board. Using GPT-4o as the base LLM, VulTrial almost doubles the efficacy of prior best-performing baselines. Additionally, we show that role-specific instruction tuning with small quantities of data significantly further boosts VulTrial's efficacy. Our extensive experiments demonstrate the efficacy of VulTrial across different LLMs, including an open-source, in-house-deployable model (LLaMA-3.1-8B), as well as the high quality of its generated explanations and its ability to uncover multiple confirmed zero-day vulnerabilities in the wild.
♻ ☆ Bayesian Optimization in Language Space: An Eval-Efficient AI Self-Improvement Framework
Enoch Hyunwook Kang, Hema Yoganarasimhan
Large Language Models (LLMs) have recently enabled self-improving AI, i.e., AI that iteratively generates, evaluates, and refines its own outcomes. Recent studies have shown that self-improving AI focusing on prompt optimization can outperform state-of-the-art reinforcement-learning fine-tuned LLMs. Here, their `performance' is typically measured by query efficiency - the number of LLM-generated solution samples required to meet a certain performance threshold. However, in many societal applications, the primary limitation is not generating new solutions but evaluating them. For instance, evaluating an ad's effectiveness requires significant human feedback, which is far more costly and time-consuming than generating a candidate ad. To optimize for the evaluation efficiency objective, a natural approach is to extend Bayesian Optimization (BO), a framework proven optimal for evaluation efficiency, to the language domain. However, the difficulty of directly estimating suitable acquisition functions in LLMs' minds makes this extension challenging. This paper overcomes this challenge by proving that the combination of the simple and widely used Best-of-N selection strategy and simple textual gradients (i.e., textual edits from a critic model) statistically emulates the behavior of the gradients on the canonical UCB acquisition function, which induces optimal exploration in terms of evaluation efficiency. Based on this result, we propose TextGrad-Best-of-N Bayesian Optimization (T-BoN BO), a simple and eval-efficient language-space Bayesian optimization framework for AI self-improvement. We also empirically validate T-BoN BO by applying it to automated ad alignment tasks for persona distribution, demonstrating its superior performance compared to popular state-of-the-art baselines.
♻ ☆ Beyond Output Faithfulness: Learning Attributions that Preserve Computational Pathways
Siyu Zhang, Kenneth Mcmillan
Faithfulness metrics such as insertion and deletion evaluate how feature removal affects model outputs but overlook whether explanations preserve the computational pathway the network actually uses. We show that external metrics can be maximized through alternative pathways -- perturbations that reroute computation via different feature detectors while preserving output behavior. To address this, we propose activation preservation as a tractable proxy for preserving computational pathways We introduce Faithfulness-guided Ensemble Interpretation (FEI), which jointly optimizes external faithfulness (via ensemble quantile optimization of insertion/deletion curves) and internal faithfulness (via selective gradient clipping). Across VGG and ResNet on ImageNet and CUB-200-2011, FEI achieves state-of-the-art insertion/deletion scores while maintaining significantly lower activation deviation, showing that both external and internal faithfulness are essential for reliable explanations.
♻ ☆ DS-Span: Single-Phase Discriminative Subgraph Mining for Efficient Graph Embeddings
Yeamin Kaiser, Muhammed Tasnim Bin Anwar, Bholanath Das
Graph representation learning seeks to transform complex, high-dimensional graph structures into compact vector spaces that preserve both topology and semantics. Among the various strategies, subgraph-based methods provide an interpretable bridge between symbolic pattern discovery and continuous embedding learning. Yet, existing frequent or discriminative subgraph mining approaches often suffer from redundant multi-phase pipelines, high computational cost, and weak coupling between mined structures and their discriminative relevance. We propose DS-Span, a single-phase discriminative subgraph mining framework that unifies pattern growth, pruning, and supervision-driven scoring within one traversal of the search space. DS-Span introduces a coverage-capped eligibility mechanism that dynamically limits exploration once a graph is sufficiently represented, and an information-gain-guided selection that promotes subgraphs with strong class-separating ability while minimizing redundancy. The resulting subgraph set serves as an efficient, interpretable basis for downstream graph embedding and classification. Extensive experiments across benchmarks demonstrate that DS-Span generates more compact and discriminative subgraph features than prior multi-stage methods, achieving higher or comparable accuracy with significantly reduced runtime. These results highlight the potential of unified, single-phase discriminative mining as a foundation for scalable and interpretable graph representation learning.
♻ ☆ Efficient Preference-Based Reinforcement Learning : Randomized Exploration Meets Experimental Design
Andreas Schlaginhaufen, Reda Ouhamma, Maryam Kamgarpour
We study reinforcement learning from human feedback in general Markov decision processes, where agents learn from trajectory-level preference comparisons. A central challenge in this setting is to design algorithms that select informative preference queries to identify the underlying reward while ensuring theoretical guarantees. We propose a meta-algorithm based on randomized exploration, which avoids the computational challenges associated with optimistic approaches and remains tractable. We establish both regret and last-iterate guarantees under mild reinforcement learning oracle assumptions. To improve query complexity, we introduce and analyze an improved algorithm that collects batches of trajectory pairs and applies optimal experimental design to select informative comparison queries. The batch structure also enables parallelization of preference queries, which is relevant in practical deployment as feedback can be gathered concurrently. Empirical evaluation confirms that the proposed method is competitive with reward-based reinforcement learning while requiring a small number of preference queries.
♻ ☆ Consistency-based Abductive Reasoning over Perceptual Errors of Multiple Pre-train ed Models in Novel Environments
Mario Leiva, Noel Ngu, Joshua Shay Kricheli, Aditya Taparia, Ransalu Senanayake, Paulo Shakarian, Nathaniel Bastian, John Corcoran, Gerardo Simari
The deployment of pre-trained perception models in novel environments often leads to performance degradation due to distributional shifts. Although recent artificial intelligence approaches for metacognition use logical rules to characterize and filter model errors, improving precision often comes at the cost of reduced recall. This paper addresses the hypothesis that leveraging multiple pre-trained models can mitigate this recall reduction. We formulate the challenge of identifying and managing conflicting predictions from various models as a consistency-based abduction problem, building on the idea of abductive learning (ABL) but applying it to test-time instead of training. The input predictions and the learned error detection rules derived from each model are encoded in a logic program. We then seek an abductive explanation--a subset of model predictions--that maximizes prediction coverage while ensuring the rate of logical inconsistencies (derived from domain constraints) remains below a specified threshold. We propose two algorithms for this knowledge representation task: an exact method based on Integer Programming (IP) and an efficient Heuristic Search (HS). Through extensive experiments on a simulated aerial imagery dataset featuring controlled, complex distributional shifts, we demonstrate that our abduction-based framework outperforms individual models and standard ensemble baselines, achieving, for instance, average relative improvements of approximately 13.6\% in F1-score and 16.6\% in accuracy across 15 diverse test datasets when compared to the best individual model. Our results validate the use of consistency-based abduction as an effective mechanism to robustly integrate knowledge from multiple imperfect models in challenging, novel scenarios.
♻ ☆ Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Review s
Sai Suresh Macharla Vasu, Ivaxi Sheth, Hui-Po Wang, Ruta Binkyte, Mario Fritz
The adoption of large language models (LLMs) is transforming the peer review process, from assisting reviewers in writing more detailed evaluations to generating entire reviews automatically. While these capabilities offer exciting opportunities, they also raise critical concerns about fairness and reliability. In this paper, we investigate bias in LLM-generated peer reviews by conducting controlled experiments on sensitive metadata, including author affiliation and gender. Our analysis consistently shows affiliation bias favoring institutions highly ranked on common academic rankings. Additionally, we find some gender preferences, which, even though subtle in magnitude, have the potential to compound over time. Notably, we uncover implicit biases that become more evident with token-based soft ratings.
♻ ☆ Surface-Based Visibility-Guided Uncertainty for Continuous Active 3D Neural Reconstruction AAAI 2026
Hyunseo Kim, Hyeonseo Yang, Taekyung Kim, YoonSung Kim, Minsu Lee, Jin-Hwa Kim, Byoung-Tak Zhang
View selection is critical in active 3D neural reconstruction as it impacts the contents of training set and resulting final output quality. Recent view selection strategies emphasize the visibility when evaluating model uncertainty in active 3D reconstruction. However, existing approaches estimate visibility only after the model fully converges, which has confined their application primarily to non-continuous active learning settings. This paper proposes Surface-Based Visibility field (SBV) that successfully estimates the visibility-guided uncertainty in continuous active 3D neural reconstruction. During learning neural implicit surfaces, our model learns rendering uncertainties and infers surface confidence values derived from signed distance functions. It then updates surface confidences using a voxel grid, robustly deducing the surface-based visibility for uncertainties. This approach captures uncertainties across all regions, whether well-defined surfaces or ambiguous areas, ensuring accurate visibility measurement in continuous active learning. Experiments on benchmark datasets-Tanks and Temples, BlendedMVS, Blender, DTU-and the newly proposed imbalanced viewpoint dataset (ImBView) show that view selection based on SBV-guided uncertainty improves performance by up to 11.6% over existing methods, highlighting its effectiveness in challenging reconstruction scenarios.
comment : The main claims are the same as in the previous version, but the naming and explanations have been changed. Accepted at AAAI 2026 Artificial Intelligence with Biased or Scarce Data workshop
♻ ☆ XAI-Driven Skin Disease Classification: Leveraging GANs to Augment ResNet-50 Performance
Kim Gerard A. Villanueva, Priyanka Kumar
Accurate and timely diagnosis of multi-class skin lesions is hampered by subjective methods, inherent data imbalance in datasets like HAM10000, and the "black box" nature of Deep Learning (DL) models. This study proposes a trustworthy and highly accurate Computer-Aided Diagnosis (CAD) system to overcome these limitations. The approach utilizes Deep Convolutional Generative Adversarial Networks (DCGANs) for per class data augmentation to resolve the critical class imbalance problem. A fine-tuned ResNet-50 classifier is then trained on the augmented dataset to classify seven skin disease categories. Crucially, LIME and SHAP Explainable AI (XAI) techniques are integrated to provide transparency by confirming that predictions are based on clinically relevant features like irregular morphology. The system achieved a high overall Accuracy of 92.50 % and a Macro-AUC of 98.82 %, successfully outperforming various prior benchmarked architectures. This work successfully validates a verifiable framework that combines high performance with the essential clinical interpretability required for safe diagnostic deployment. Future research should prioritize enhancing discrimination for critical categories, such as Melanoma NOS (F1-Score is 0.8602).
♻ ☆ Multi- Agent Code Verification via Information Theory
Shreshth Rajan
LLMs generate buggy code: 29.6% of SWE-bench solved patches fail, 62% of BaxBench solutions have vulnerabilities, and existing tools only catch 65% of bugs with 35% false positives. We built CodeX-Verify, a multi-agent system that uses four specialized agents to detect different types of bugs. We prove mathematically that combining agents with different detection patterns finds more bugs than any single agent when the agents look for different problems, using submodularity of mutual information under conditional independence. Measuring agent correlation of rho = 0.05 to 0.25 confirms they detect different bugs. Testing on 99 code samples with verified labels shows our system catches 76.1% of bugs, matching the best existing method (Meta Prompt Testing: 75%) while running faster and without test execution. We tested all 15 agent combinations and found that using multiple agents improves accuracy by 39.7 percentage points (from 32.8% to 72.4%) compared to single agents, with diminishing returns of +14.9pp, +13.5pp, and +11.2pp for agents 2, 3, and 4, validating our theoretical model. The best two-agent combination (Correctness + Performance) reaches 79.3% accuracy. Testing on 300 real patches from Claude Sonnet 4.5 runs in under 200ms per sample, making this practical for production use.
comment : 18 pages, 3 figures, 9 tables
♻ ☆ NITRO-D: Native Integer-only Training of Deep Convolutional Neural Networks
Alberto Pirillo, Luca Colombo, Manuel Roveri
Quantization is a pivotal technique for managing the growing computational and memory demands of Deep Neural Networks (DNNs). By reducing the number of bits used to represent weights and activations (typically from 32-bit Floating-Point (FP) to 16-bit or 8-bit integers), quantization reduces memory footprint, energy consumption, and execution time of DNNs. However, most existing methods typically target DNN inference, while training still relies on FP operations, limiting applicability in environments where FP arithmetic is unavailable. To date, only one prior work has addressed integer-only training, and only for Multi-Layer Perceptron (MLP) architectures. This paper introduces NITRO-D, a novel framework for training deep integer-only Convolutional Neural Networks (CNNs) that operate entirely in the integer domain for both training and inference. NITRO-D enables training of integer CNNs without requiring a separate quantization scheme. Specifically, it introduces a novel architecture that integrates multiple local-loss blocks, which include the proposed NITRO-Scaling layer and NITRO-ReLU activation function. The proposed framework also features a novel learning algorithm that employs local error signals and leverages IntegerSGD, an optimizer specifically designed for integer computations. NITRO-D is implemented as an open-source Python library. Extensive evaluations on state-of-the-art image recognition datasets demonstrate its effectiveness. For integer-only MLPs, NITRO-D improves test accuracy by up to +5.96% over the state-of-the-art. It also successfully trains integer-only CNNs, reducing memory requirements and energy consumption by up to 76.14% and 32.42%, respectively, compared to the traditional FP backpropagation algorithm.
comment : 15 pages, 3 figures
♻ ☆ Open-Set Domain Adaptation Under Background Distribution Shift: Challenges and A Provably Efficient Solution
Shravan Chaudhari, Yoav Wald, Suchi Saria
As we deploy machine learning systems in the real world, a core challenge is to maintain a model that is performant even as the data shifts. Such shifts can take many forms: new classes may emerge that were absent during training, a problem known as open-set recognition, and the distribution of known categories may change. Guarantees on open-set recognition are mostly derived under the assumption that the distribution of known classes, which we call the background distribution, is fixed. In this paper we develop CoLOR, a method that is guaranteed to solve open-set recognition even in the challenging case where the background distribution shifts. We prove that the method works under benign assumptions that the novel class is separable from the non-novel classes, and provide theoretical guarantees that it outperforms a representative baseline in a simplified overparameterized setting. We develop techniques to make CoLOR scalable and robust, and perform comprehensive empirical evaluations on image and text data. The results show that CoLOR significantly outperforms existing open-set recognition methods under background shift. Moreover, we provide new insights into how factors such as the size of the novel class influences performance, an aspect that has not been extensively explored in prior work.
♻ ☆ Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design
Quentin Anthony, Yury Tokpanov, Skyler Szot, Srivatsan Rajagopal, Praneeth Medepalli, Anna Golubeva, Vasu Shyam, Robert Washbourne, Rishi Iyer, Ansh Chaurasia, Tomas Figliolia, Xiao Yang, Abhinav Sarje, Drew Thorstensen, Amartey Pearson, Zack Grossbart, Jason van Patten, Emad Barsoum, Zhenyu Gu, Yao Fu, Beren Millidge
We report on the first large-scale mixture-of-experts (MoE) pretraining study on pure AMD hardware, utilizing both MI300X GPUs and Pollara networking. We distill practical guidance for both systems and model design. On the systems side, we deliver a comprehensive cluster and networking characterization: microbenchmarks for all core collectives (all-reduce, reduce-scatter, all-gather, broadcast) across message sizes and GPU counts over Pollara. To our knowledge, this is the first at this scale. We further provide MI300X microbenchmarks on kernel sizing and memory bandwidth to inform model design. On the modeling side, we introduce and apply MI300X-aware transformer sizing rules for attention and MLP blocks and justify MoE widths that jointly optimize training throughput and inference latency. We describe our training stack in depth, including often-ignored utilities such as fault-tolerance and checkpoint-reshaping, as well as detailed information on our training recipe. We also provide a preview of our model architecture and base model - ZAYA1 (760M active, 8.3B total parameters MoE, available at https://huggingface.co/Zyphra/ZAYA1-base) - which will be further improved upon in forthcoming papers. ZAYA1-base achieves performance comparable to leading base models such as Qwen3-4B and Gemma3-12B at its scale and larger, and outperforms models including Llama-3-8B and OLMoE across reasoning, mathematics, and coding benchmarks. Together, these results demonstrate that the AMD hardware, network, and software stack are mature and optimized enough for competitive large-scale pretraining.
♻ ☆ Turing Test 2.0: The General Intelligence Threshold
Georgios Mappouras
With the rise of artificial intelligence (A.I.) and large language models like ChatGPT, a new race for achieving artificial general intelligence (A.G.I) has started. While many speculate how and when A.I. will achieve A.G.I., there is no clear agreement on how A.G.I. can be detected in A.I. models, even when popular tools like the Turing test (and its modern variations) are used to measure their intelligence. In this work, we discuss why traditional methods like the Turing test do not suffice for measuring or detecting A.G.I. and provide a new, practical method that can be used to decide if a system (computer or any other) has reached or surpassed A.G.I. To achieve this, we make two new contributions. First, we present a clear definition for general intelligence (G.I.) and set a G.I. Threshold (G.I.T.) that can be used to distinguish between systems that achieve A.G.I. and systems that do not. Second, we present a new framework on how to construct tests that can detect if a system has achieved G.I. in a simple, comprehensive, and clear-cut fail/pass way. We call this novel framework the Turing test 2.0. We then demonstrate real-life examples of applying tests that follow our Turing test 2.0 framework on modern A.I. models.
Computer Vision and Pattern Recognition 218
☆ Unique Lives, Shared Wo rl d: Learning from Single-Life Videos
Tengda Han, Sayna Ebrahimi, Dilara Gokay, Li Yang Ku, Maks Ovsjanikov, Iva Babukova, Daniel Zoran, Viorica Patraucean, Joao Carreira, Andrew Zisserman, Dima Damen
We introduce the "single-life" learning paradigm, where we train a distinct vision model exclusively on egocentric videos captured by one individual. We leverage the multiple viewpoints naturally captured within a single life to learn a visual encoder in a self-supervised manner. Our experiments demonstrate three key findings. First, models trained independently on different lives develop a highly aligned geometric understanding. We demonstrate this by training visual encoders on distinct datasets each capturing a different life, both indoors and outdoors, as well as introducing a novel cross-attention-based metric to quantify the functional alignment of the internal representations developed by different models. Second, we show that single-life models learn generalizable geometric representations that effectively transfer to downstream tasks, such as depth estimation, in unseen environments. Third, we demonstrate that training on up to 30 hours from one week of the same person's life leads to comparable performance to training on 30 hours of diverse web data, highlighting the strength of single-life representation learning. Overall, our results establish that the shared structure of the world, both leads to consistency in models trained on individual lives, and provides a powerful signal for visual representation learning.
☆ SimFlow: Simplified and End-to-End Training of Latent Normalizing Flows
Qinyu Zhao, Guangting Zheng, Tao Yang, Rui Zhu, Xingjian Leng, Stephen Gould, Liang Zheng
Normalizing Flows (NFs) learn invertible mappings between the data and a Gaussian distribution. Prior works usually suffer from two limitations. First, they add random noise to training samples or VAE latents as data augmentation, introducing complex pipelines including extra noising and denoising steps. Second, they use a pretrained and frozen VAE encoder, resulting in suboptimal reconstruction and generation quality. In this paper, we find that the two issues can be solved in a very simple way: just fixing the variance (which would otherwise be predicted by the VAE encoder) to a constant (e.g., 0.5). On the one hand, this method allows the encoder to output a broader distribution of tokens and the decoder to learn to reconstruct clean images from the augmented token distribution, avoiding additional noise or denoising design. On the other hand, fixed variance simplifies the VAE evidence lower bound, making it stable to train an NF with a VAE jointly. On the ImageNet $256 \times 256$ generation task, our model SimFlow obtains a gFID score of 2.15, outperforming the state-of-the-art method STARFlow (gFID 2.40). Moreover, SimFlow can be seamlessly integrated with the end-to-end representation alignment (REPA-E) method and achieves an improved gFID of 1.91, setting a new state of the art among NFs.
comment : Project Page: https://qinyu-allen-zhao.github.io/SimFlow/
☆ PosterCopilot: Toward Layout Reasoning and Controllable Editing for Professional Graphic Design
Jiazhe Wei, Ken Li, Tianyu Lao, Haofan Wang, Liang Wang, Caifeng Shan, Chenyang Si
Graphic design forms the cornerstone of modern visual communication, serving as a vital medium for promoting cultural and commercial events. Recent advances have explored automating this process using Large Multimodal Models (LMMs), yet existing methods often produce geometrically inaccurate layouts and lack the iterative, layer-specific editing required in professional workflows. To address these limitations, we present PosterCopilot, a framework that advances layout reasoning and controllable editing for professional graphic design. Specifically, we introduce a progressive three-stage training strategy that equips LMMs with geometric understanding and aesthetic reasoning for layout design, consisting of Perturbed Supervised Fine-Tuning, Reinforcement Learning for Visual-Reality Alignment, and Reinforcement Learning from Aesthetic Feedback. Furthermore, we develop a complete workflow that couples the trained LMM-based design model with generative models, enabling layer-controllable, iterative editing for precise element refinement while maintaining global visual consistency. Extensive experiments demonstrate that PosterCopilot achieves geometrically accurate and aesthetically superior layouts, offering unprecedented controllability for professional iterative design.
comment : Project page: https://postercopilot.github.io/
☆ Radiance Meshes for Volumetric Reconstruction
Alexander Mai, Trevor Hedstrom, George Kopanas, Janne Kontkanen, Falko Kuester, Jonathan T. Barron
We introduce radiance meshes, a technique for representing radiance fields with constant density tetrahedral cells produced with a Delaunay tetrahedralization. Unlike a Voronoi diagram, a Delaunay tetrahedralization yields simple triangles that are natively supported by existing hardware. As such, our model is able to perform exact and fast volume rendering using both rasterization and ray-tracing. We introduce a new rasterization method that achieves faster rendering speeds than all prior radiance field representations (assuming an equivalent number of primitives and resolution) across a variety of platforms. Optimizing the positions of Delaunay vertices introduces topological discontinuities (edge flips). To solve this, we use a Zip-NeRF-style backbone which allows us to express a smoothly varying field even when the topology changes. Our rendering method exactly evaluates the volume rendering equation and enables high quality, real-time view synthesis on standard consumer hardware. Our tetrahedral meshes also lend themselves to a variety of exciting applications including fisheye lens distortion, physics-based simulation, editing, and mesh extraction.
comment : Website: half-potato.gitlab.io/rm
☆ SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL
Siyi Chen, Mikaela Angelina Uy, Chan Hee Song, Faisal Ladhak, Adithyavairavan Murali, Qing Qu, Stan Birchfield, Valts Blukis, Jonathan Tremblay
Vision Language Models (VLMs) demonstrate strong qualitative visual understanding, but struggle with metrically precise spatial reasoning required for embodied applications. The agentic paradigm promises that VLMs can use a wide variety of tools that could augment these capabilities, such as depth estimators, segmentation models, and pose estimators. Yet it remains an open challenge how to realize this vision without solely relying on handcrafted prompting strategies or enforcing fixed, predefined tool pipelines that limit VLMs' ability to discover optimal tool-use patterns. Reinforcement Learning could overcome this gap, but has so far been limited to reasoning with a single visual tool due to the large search space in multi-tool reasoning. We introduce Double Interactive Reinforcement Learning (DIRL), a two-phase training framework where VLMs learn to coordinate multiple tools through interactive exploration and feedback. In the teaching phase, we combine demonstrations from a single tool specialist trained via interactive RL with traces from a frontier model using all tools. In the exploration phase, the model further refines multi-tool coordination through continued RL. Our model, SpaceTools, with tool-augmented spatial reasoning ability, achieves state-of-the-art performance on spatial understanding benchmarks (RoboSpatial-Home, BLINK, BOP-ASK) and demonstrates reliable real-world manipulation using a 7-DOF robot as a tool. DIRL provides substantial improvements over the vanilla SFT (+12% on RoboSpatial) and RL (+16% on RoboSpatial) baselines. Project page: https://spacetools.github.io/.
☆ Stable Signer: Hierarchical Sign Language Generative Model
Sen Fang, Yalin Feng, Hongbin Zhong, Yanxin Zhang, Dimitris N. Metaxas
Sign Language Production (SLP) is the process of converting the complex input text into a real video. Most previous works focused on the Text2Gloss, Gloss2Pose, Pose2Vid stages, and some concentrated on Prompt2Gloss and Text2Avatar stages. However, this field has made slow progress due to the inaccuracy of text conversion, pose generation, and the rendering of poses into real human videos in these stages, resulting in gradually accumulating errors. Therefore, in this paper, we streamline the traditional redundant structure, simplify and optimize the task objective, and design a new sign language generative model called Stable Signer. It redefines the SLP task as a hierarchical generation end-to-end task that only includes text understanding (Prompt2Gloss, Text2Gloss) and Pose2Vid, and executes text understanding through our proposed new Sign Language Understanding Linker called SLUL, and generates hand gestures through the named SLP-MoE hand gesture rendering expert block to end-to-end generate high-quality and multi-style sign language videos. SLUL is trained using the newly developed Semantic-Aware Gloss Masking Loss (SAGM Loss). Its performance has improved by 48.6% compared to the current SOTA generation methods.
comment : 12 pages, 7 figures. More Demo at https://stablesigner.github.io
☆ RELIC: Interactive Video Wo rl d Model with Long-Horizon Memory
Yicong Hong, Yiqun Mei, Chongjian Ge, Yiran Xu, Yang Zhou, Sai Bi, Yannick Hold-Geoffroy, Mike Roberts, Matthew Fisher, Eli Shechtman, Kalyan Sunkavalli, Feng Liu, Zhengqi Li, Hao Tan
A truly interactive world model requires three key ingredients: real-time long-horizon streaming, consistent spatial memory, and precise user control. However, most existing approaches address only one of these aspects in isolation, as achieving all three simultaneously is highly challenging-for example, long-term memory mechanisms often degrade real-time performance. In this work, we present RELIC, a unified framework that tackles these three challenges altogether. Given a single image and a text description, RELIC enables memory-aware, long-duration exploration of arbitrary scenes in real time. Built upon recent autoregressive video-diffusion distillation techniques, our model represents long-horizon memory using highly compressed historical latent tokens encoded with both relative actions and absolute camera poses within the KV cache. This compact, camera-aware memory structure supports implicit 3D-consistent content retrieval and enforces long-term coherence with minimal computational overhead. In parallel, we fine-tune a bidirectional teacher video model to generate sequences beyond its original 5-second training horizon, and transform it into a causal student generator using a new memory-efficient self-forcing paradigm that enables full-context distillation over long-duration teacher as well as long student self-rollouts. Implemented as a 14B-parameter model and trained on a curated Unreal Engine-rendered dataset, RELIC achieves real-time generation at 16 FPS while demonstrating more accurate action following, more stable long-horizon streaming, and more robust spatial-memory retrieval compared with prior work. These capabilities establish RELIC as a strong foundation for the next generation of interactive world modeling.
comment : 22 pages
☆ Fast & Efficient Normalizing Flows and Applications of Image Generative Models
Sandeep Nagar
This thesis presents novel contributions in two primary areas: advancing the efficiency of generative models, particularly normalizing flows, and applying generative models to solve real-world computer vision challenges. The first part introduce significant improvements to normalizing flow architectures through six key innovations: 1) Development of invertible 3x3 Convolution layers with mathematically proven necessary and sufficient conditions for invertibility, (2) introduction of a more efficient Quad-coupling layer, 3) Design of a fast and efficient parallel inversion algorithm for kxk convolutional layers, 4) Fast & efficient backpropagation algorithm for inverse of convolution, 5) Using inverse of convolution, in Inverse-Flow, for the forward pass and training it using proposed backpropagation algorithm, and 6) Affine-StableSR, a compact and efficient super-resolution model that leverages pre-trained weights and Normalizing Flow layers to reduce parameter count while maintaining performance. The second part: 1) An automated quality assessment system for agricultural produce using Conditional GANs to address class imbalance, data scarcity and annotation challenges, achieving good accuracy in seed purity testing; 2) An unsupervised geological mapping framework utilizing stacked autoencoders for dimensionality reduction, showing improved feature extraction compared to conventional methods; 3) We proposed a privacy preserving method for autonomous driving datasets using on face detection and image inpainting; 4) Utilizing Stable Diffusion based image inpainting for replacing the detected face and license plate to advancing privacy-preserving techniques and ethical considerations in the field.; and 5) An adapted diffusion model for art restoration that effectively handles multiple types of degradation through unified fine-tuning.
comment : PhD Thesis
☆ Jina-VLM: Small Multilingual Vision Language Model
Andreas Koukounas, Georgios Mastrapas, Florian Hönicke, Sedigheh Eslami, Guillaume Roncari, Scott Martens, Han Xiao
We present Jina-VLM, a 2.4B parameter vision-language model that achieves state-of-the-art multilingual visual question answering among open 2B-scale VLMs. The model couples a SigLIP2 vision encoder with a Qwen3 language backbone through an attention-pooling connector that enables token-efficient processing of arbitrary-resolution images. Across standard VQA benchmarks and multilingual evaluations, Jina-VLM outperforms comparable models while preserving competitive text-only performance.
comment : 18 pages, 1-7 main content
☆ PSA: Pyramid Sparse Attention for Efficient Video Understanding and Generation
Xiaolong Li, Youping Gu, Xi Lin, Weijie Wang, Bohan Zhuang
Attention mechanisms are the core of foundation models, but their quadratic complexity remains a critical bottleneck for scaling. This challenge has driven the development of efficient attention mechanisms, with sparsity emerging as the dominant paradigm. Current methods typically retain or discard entire key-value blocks with binary masks, resulting in substantial information loss under high sparsity. To mitigate this gap, we present Pyramid Sparse Attention (PSA), a versatile module applicable to both video understanding and generation tasks. Instead of binary masking, PSA introduces multi-level pooled KV representations, enabling finer mask granularity. Specifically, each query block dynamically allocates lower pooling levels to critical KV blocks and higher levels to less important ones, creating an informative interpolation between full retention and complete pruning. This design, analogous to fixed-point quantization and classical feature pyramid networks in computer vision, effectively mitigates information loss while preserving computational efficiency under a low compute budget. It works with a native, hardware-friendly kernel that leverages decoupled block-tile design to ensure efficient execution. Across video understanding and generation benchmarks, PSA preserves contextual information and visual fidelity, consistently outperforming or achieving comparable performance over existing sparse attention baselines with superior efficiency-quality trade-offs. Our code and model weights are publicly available at: http://ziplab.co/PSA
comment : Tech report
☆ C3G: Learning Compact 3D Representations with 2K Gaussians
Honggyu An, Jaewoo Jung, Mungyeom Kim, Sunghwan Hong, Chaehyun Kim, Kazumi Fukuda, Minkyeong Jeon, Jisang Han, Takuya Narihira, Hyuna Ko, Junsu Kim, Yuki Mitsufuji, Seungryong Kim
Reconstructing and understanding 3D scenes from unposed sparse views in a feed-forward manner remains as a challenging task in 3D computer vision. Recent approaches use per-pixel 3D Gaussian Splatting for reconstruction, followed by a 2D-to-3D feature lifting stage for scene understanding. However, they generate excessive redundant Gaussians, causing high memory overhead and sub-optimal multi-view feature aggregation, leading to degraded novel view synthesis and scene understanding performance. We propose C3G, a novel feed-forward framework that estimates compact 3D Gaussians only at essential spatial locations, minimizing redundancy while enabling effective feature lifting. We introduce learnable tokens that aggregate multi-view features through self-attention to guide Gaussian generation, ensuring each Gaussian integrates relevant visual features across views. We then exploit the learned attention patterns for Gaussian decoding to efficiently lift features. Extensive experiments on pose-free novel view synthesis, 3D open-vocabulary segmentation, and view-invariant feature aggregation demonstrate our approach's effectiveness. Results show that a compact yet geometrically meaningful representation is sufficient for high-quality scene reconstruction and understanding, achieving superior memory efficiency and feature fidelity compared to existing methods.
comment : Project Page : https://cvlab-kaist.github.io/C3G/
☆ Ultra-lightweight Neural Video Representation Compression
Ho Man Kwan, Tianhao Peng, Ge Gao, Fan Zhang, Mike Nilsson, Andrew Gower, David Bull
Recent works have demonstrated the viability of utilizing over-fitted implicit neural representations (INRs) as alternatives to autoencoder-based models for neural video compression. Among these INR-based video codecs, Neural Video Representation Compression (NVRC) was the first to adopt a fully end-to-end compression framework that compresses INRs, achieving state-of-the-art performance. Moreover, some recently proposed lightweight INRs have shown comparable performance to their baseline codecs with computational complexity lower than 10kMACs/pixel. In this work, we extend NVRC toward lightweight representations, and propose NVRC-Lite, which incorporates two key changes. Firstly, we integrated multi-scale feature grids into our lightweight neural representation, and the use of higher resolution grids significantly improves the performance of INRs at low complexity. Secondly, we address the issue that existing INRs typically leverage autoregressive models for entropy coding: these are effective but impractical due to their slow coding speed. In this work, we propose an octree-based context model for entropy coding high-dimensional feature grids, which accelerates the entropy coding module of the model. Our experimental results demonstrate that NVRC-Lite outperforms C3, one of the best lightweight INR-based video codecs, with up to 21.03% and 23.06% BD-rate savings when measured in PSNR and MS-SSIM, respectively, while achieving 8.4x encoding and 2.5x decoding speedup. The implementation of NVRC-Lite will be made available.
☆ Learning Group Actions In Disentangled Latent Image Representations
Farhana Hossain Swarnali, Miaomiao Zhang, Tonmoy Hossain
Modeling group actions on latent representations enables controllable transformations of high-dimensional image data. Prior works applying group-theoretic priors or modeling transformations typically operate in the high-dimensional data space, where group actions apply uniformly across the entire input, making it difficult to disentangle the subspace that varies under transformations. While latent-space methods offer greater flexibility, they still require manual partitioning of latent variables into equivariant and invariant subspaces, limiting the ability to robustly learn and operate group actions within the representation space. To address this, we introduce a novel end-to-end framework that for the first time learns group actions on latent image manifolds, automatically discovering transformation-relevant structures without manual intervention. Our method uses learnable binary masks with straight-through estimation to dynamically partition latent representations into transformation-sensitive and invariant components. We formulate this within a unified optimization framework that jointly learns latent disentanglement and group transformation mappings. The framework can be seamlessly integrated with any standard encoder-decoder architecture. We validate our approach on five 2D/3D image datasets, demonstrating its ability to automatically learn disentangled latent factors for group actions in diverse data, while downstream classification tasks confirm the effectiveness of the learned representations. Our code is publicly available at https://github.com/farhanaswarnali/Learning-Group-Actions-In-Disentangled-Latent-Image-Representations .
☆ Emergent Outlier View Rejection in Visual Geometry Grounded Transformer s
Jisang Han, Sunghwan Hong, Jaewoo Jung, Wooseok Jang, Honggyu An, Qianqian Wang, Seungryong Kim, Chen Feng
Reliable 3D reconstruction from in-the-wild image collections is often hindered by "noisy" images-irrelevant inputs with little or no view overlap with others. While traditional Structure-from-Motion pipelines handle such cases through geometric verification and outlier rejection, feed-forward 3D reconstruction models lack these explicit mechanisms, leading to degraded performance under in-the-wild conditions. In this paper, we discover that the existing feed-forward reconstruction model, e.g., VGGT, despite lacking explicit outlier-rejection mechanisms or noise-aware training, can inherently distinguish distractor images. Through an in-depth analysis under varying proportions of synthetic distractors, we identify a specific layer that naturally exhibits outlier-suppressing behavior. Further probing reveals that this layer encodes discriminative internal representations that enable an effective noise-filtering capability, which we simply leverage to perform outlier-view rejection in feed-forward 3D reconstruction without any additional fine-tuning or supervision. Extensive experiments on both controlled and in-the-wild datasets demonstrate that this implicit filtering mechanism is consistent and generalizes well across diverse scenarios.
comment : Project page: https://cvlab-kaist.github.io/RobustVGGT/
☆ On the Temporality for Sketch Representation Learning
Marcelo Isaias de Moraes Junior, Moacir Antonelli Ponti
Sketches are simple human hand-drawn abstractions of complex scenes and real-world objects. Although the field of sketch representation learning has advanced significantly, there is still a gap in understanding the true relevance of the temporal aspect to the quality of these representations. This work investigates whether it is indeed justifiable to treat sketches as sequences, as well as which internal orders play a more relevant role. The results indicate that, although the use of traditional positional encodings is valid for modeling sketches as sequences, absolute coordinates consistently outperform relative ones. Furthermore, non-autoregressive decoders outperform their autoregressive counterparts. Finally, the importance of temporality was shown to depend on both the order considered and the task evaluated.
☆ Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding
Jialuo Li, Bin Li, Jiahao Li, Yan Lu
The application of Large Multimodal Models (LMMs) to long-form video understanding is constrained by limited context lengths and the computationally prohibitive cost of processing dense video tokens. Consequently, recent research has focused on query-aware frame selection, methods that often incur significant computational overhead. This paper challenges the assumption that such complex search mechanisms are universally necessary. We first identify and validate a query typology distinguishing between global query and localized query. We demonstrate that while uniform sampling is both effective and efficient for global queries, localized queries indeed necessitate query-aware selection for optimal performance. Building on this insight, we propose DIG, a training-free frame selection framework that adapts its strategy based on the query type. Specifically,DIG employs efficient uniform sampling for global queries while activating a specialized pipeline to extract query-relevant frames for localized queries. Experiments on three long-form video understanding benchmarks demonstrate that DIG consistently outperforms existing baselines and robustly improves LMM performance, even when scaling the input frame count to 256.
☆ Highly Efficient Test-Time Scaling for T2I Diffusion Models with Text Embedding Perturbation
Hang Xu, Linjiang Huang, Feng Zhao
Test-time scaling (TTS) aims to achieve better results by increasing random sampling and evaluating samples based on rules and metrics. However, in text-to-image(T2I) diffusion models, most related works focus on search strategies and reward models, yet the impact of the stochastic characteristic of noise in T2I diffusion models on the method's performance remains unexplored. In this work, we analyze the effects of randomness in T2I diffusion models and explore a new format of randomness for TTS: text embedding perturbation, which couples with existing randomness like SDE-injected noise to enhance generative diversity and quality. We start with a frequency-domain analysis of these formats of randomness and their impact on generation, and find that these two randomness exhibit complementary behavior in the frequency domain: spatial noise favors low-frequency components (early steps), while text embedding perturbation enhances high-frequency details (later steps), thereby compensating for the potential limitations of spatial noise randomness in high-frequency manipulation. Concurrently, text embedding demonstrates varying levels of tolerance to perturbation across different dimensions of the generation process. Specifically, our method consists of two key designs: (1) Introducing step-based text embedding perturbation, combining frequency-guided noise schedules with spatial noise perturbation. (2) Adapting the perturbation intensity selectively based on their frequency-specific contributions to generation and tolerance to perturbation. Our approach can be seamlessly integrated into existing TTS methods and demonstrates significant improvements on multiple benchmarks with almost no additional computation. Code is available at \href{https://github.com/xuhang07/TEP-Diffusion}{https://github.com/xuhang07/TEP-Diffusion}.
☆ Artificial Microsaccade Compensation: Stable Vision for an Ornithopter
Levi Burner, Guido de Croon, Yiannis Aloimonos
Animals with foveated vision, including humans, experience microsaccades, small, rapid eye movements that they are not aware of. Inspired by this phenomenon, we develop a method for "Artificial Microsaccade Compensation". It can stabilize video captured by a tailless ornithopter that has resisted attempts to use camera-based sensing because it shakes at 12-20 Hz. Our approach minimizes changes in image intensity by optimizing over 3D rotation represented in SO(3). This results in a stabilized video, computed in real time, suitable for human viewing, and free from distortion. When adapted to hold a fixed viewing orientation, up to occasional saccades, it can dramatically reduce inter-frame motion while also benefiting from an efficient recursive update. When compared to Adobe Premier Pro's warp stabilizer, which is widely regarded as the best commercial video stabilization software available, our method achieves higher quality results while also running in real time.
comment : 29 pages, 5 figures, 2 tables, under review
☆ DIQ-H: Evaluating Hallucination Persistence in VLMs Under Temporal Visual Degradation
Zexin Lin, Hawen Wan, Yebin Zhong, Xiaoqiang
Vision-Language Models (VLMs) deployed in safety-critical applications such as autonomous driving must handle continuous visual streams under imperfect conditions. However, existing benchmarks focus on static, high-quality images and ignore temporal degradation and error propagation, which are critical failure modes where transient visual corruption induces hallucinations that persist across subsequent frames. We introduce DIQ-H, the first benchmark for evaluating VLM robustness under dynamic visual degradation in temporal sequences. DIQ-H applies physics-based corruptions including motion blur, sensor noise, and compression artifacts, and measures hallucination persistence, error recovery, and temporal consistency through multi-turn question-answering tasks. To enable scalable annotation, we propose Uncertainty-Guided Iterative Refinement (UIR), which generates reliable pseudo-ground-truth using lightweight VLMs with uncertainty filtering, achieving a 15.3 percent accuracy improvement. Experiments on 16 state-of-the-art VLMs reveal substantial robustness gaps: even advanced models such as GPT-4o achieve only a 78.5 percent recovery rate, while open-source models struggle with temporal consistency at less than 60 percent. DIQ-H provides a comprehensive platform for evaluating VLM reliability in real-world deployments.
☆ DirectDrag: High-Fidelity, Mask-Free, Prompt -Free Drag-based Image Editing via Readout-Guided Feature Alignment
Sheng-Hao Liao, Shang-Fu Chen, Tai-Ming Huang, Wen-Huang Cheng, Kai-Lung Hua
Drag-based image editing using generative models provides intuitive control over image structures. However, existing methods rely heavily on manually provided masks and textual prompts to preserve semantic fidelity and motion precision. Removing these constraints creates a fundamental trade-off: visual artifacts without masks and poor spatial control without prompts. To address these limitations, we propose DirectDrag, a novel mask- and prompt-free editing framework. DirectDrag enables precise and efficient manipulation with minimal user input while maintaining high image fidelity and accurate point alignment. DirectDrag introduces two key innovations. First, we design an Auto Soft Mask Generation module that intelligently infers editable regions from point displacement, automatically localizing deformation along movement paths while preserving contextual integrity through the generative model's inherent capacity. Second, we develop a Readout-Guided Feature Alignment mechanism that leverages intermediate diffusion activations to maintain structural consistency during point-based edits, substantially improving visual fidelity. Despite operating without manual mask or prompt, DirectDrag achieves superior image quality compared to existing methods while maintaining competitive drag accuracy. Extensive experiments on DragBench and real-world scenarios demonstrate the effectiveness and practicality of DirectDrag for high-quality, interactive image manipulation. Project Page: https://frakw.github.io/DirectDrag/. Code is available at: https://github.com/frakw/DirectDrag.
☆ BlurDM: A Blur Diffusion Model for Image Deblurring NeurIPS 2025
Jin-Ting He, Fu-Jen Tsai, Yan-Tsung Peng, Min-Hung Chen, Chia-Wen Lin, Yen-Yu Lin
Diffusion models show promise for dynamic scene deblurring; however, existing studies often fail to leverage the intrinsic nature of the blurring process within diffusion models, limiting their full potential. To address it, we present a Blur Diffusion Model (BlurDM), which seamlessly integrates the blur formation process into diffusion for image deblurring. Observing that motion blur stems from continuous exposure, BlurDM implicitly models the blur formation process through a dual-diffusion forward scheme, diffusing both noise and blur onto a sharp image. During the reverse generation process, we derive a dual denoising and deblurring formulation, enabling BlurDM to recover the sharp image by simultaneously denoising and deblurring, given pure Gaussian noise conditioned on the blurred image as input. Additionally, to efficiently integrate BlurDM into deblurring networks, we perform BlurDM in the latent space, forming a flexible prior generation network for deblurring. Extensive experiments demonstrate that BlurDM significantly and consistently enhances existing deblurring methods on four benchmark datasets. The source code is available at https://github.com/Jin-Ting-He/BlurDM.
comment : NeurIPS 2025
☆ Training for Id entity , Inference for Controllability: A Unified Approach to Tuning-Free Face Personalization
Lianyu Pang, Ji Zhou, Qiping Wang, Baoquan Zhao, Zhenguo Yang, Qing Li, Xudong Mao
Tuning-free face personalization methods have developed along two distinct paradigms: text embedding approaches that map facial features into the text embedding space, and adapter-based methods that inject features through auxiliary cross-attention layers. While both paradigms have shown promise, existing methods struggle to simultaneously achieve high identity fidelity and flexible text controllability. We introduce UniID, a unified tuning-free framework that synergistically integrates both paradigms. Our key insight is that when merging these approaches, they should mutually reinforce only identity-relevant information while preserving the original diffusion prior for non-identity attributes. We realize this through a principled training-inference strategy: during training, we employ an identity-focused learning scheme that guides both branches to capture identity features exclusively; at inference, we introduce a normalized rescaling mechanism that recovers the text controllability of the base diffusion model while enabling complementary identity signals to enhance each other. This principled design enables UniID to achieve high-fidelity face personalization with flexible text controllability. Extensive experiments against six state-of-the-art methods demonstrate that UniID achieves superior performance in both identity preservation and text controllability. Code will be available at https://github.com/lyuPang/UniID
comment : 17 pages, 13 figures
☆ TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learning
Tao Wu, Li Yang, Gen Zhan, Yiting Liao, Junlin Li, Deliang Fu, Li Zhang, Limin Wang
Enhancing the temporal understanding of Multimodal Large Language Models (MLLMs) is essential for advancing long-form video analysis, enabling tasks such as temporal localization, action detection, and time-sensitive question answering. While reinforcement learning (RL) has recently been explored for improving temporal reasoning, existing approaches are often confined to limited task types and data, restricting their generalization across diverse temporal understanding scenarios. To address this challenge, we present TempR1, a temporal-aware multi-task reinforcement learning framework that systematically strengthens MLLMs' temporal comprehension. We curate a multi-task corpus that exposes the model to diverse temporal structures and semantics, and build upon the Group Relative Policy Optimization (GRPO) algorithm to achieve stable and effective cross-task optimization. Specifically, we categorize temporal tasks into three correspondence types between predicted intervals and ground-truth instances, and design tailored localization rewards for each, enabling TempR1 to capture fine-grained temporal dependencies and adapt to different temporal patterns. Extensive experiments demonstrate that TempR1 attains state-of-the-art performance across multiple benchmarks. Moreover, its joint optimization over complementary tasks yields a strong synergistic effect, enhancing both generalization and single-task performance, establishing a scalable and principled paradigm for temporal reasoning in MLLMs.
☆ Tada-DIP: Input-adaptive Deep Image Prior for One-shot 3D Image Reconstruction
Evan Bell, Shijun Liang, Ismail Alkhouri, Saiprasad Ravishankar
Deep Image Prior (DIP) has recently emerged as a promising one-shot neural-network based image reconstruction method. However, DIP has seen limited application to 3D image reconstruction problems. In this work, we introduce Tada-DIP, a highly effective and fully 3D DIP method for solving 3D inverse problems. By combining input-adaptation and denoising regularization, Tada-DIP produces high-quality 3D reconstructions while avoiding the overfitting phenomenon that is common in DIP. Experiments on sparse-view X-ray computed tomography reconstruction validate the effectiveness of the proposed method, demonstrating that Tada-DIP produces much better reconstructions than training-data-free baselines and achieves reconstruction performance on par with a supervised network trained using a large dataset with fully-sampled volumes.
comment : 6 pages, 8 figures, 2025 Asilomar Conference on Signals, Systems, and Computers. Code is available at github.com/evanbell02/Tada-DIP/
☆ MUT3R: Motion-aware Updating Transformer for Dynamic 3D Reconstruction
Guole Shen, Tianchen Deng, Xingrui Qin, Nailin Wang, Jianyu Wang, Yanbo Wang, Yongtao Chen, Hesheng Wang, Jingchuan Wang
Recent stateful recurrent neural networks have achieved remarkable progress on static 3D reconstruction but remain vulnerable to motion-induced artifacts, where non-rigid regions corrupt attention propagation between the spatial memory and image feature. By analyzing the internal behaviors of the state and image token updating mechanism, we find that aggregating self-attention maps across layers reveals a consistent pattern: dynamic regions are naturally down-weighted, exposing an implicit motion cue that the pretrained transformer already encodes but never explicitly uses. Motivated by this observation, we introduce MUT3R, a training-free framework that applies the attention-derived motion cue to suppress dynamic content in the early layers of the transformer during inference. Our attention-level gating module suppresses the influence of dynamic regions before their artifacts propagate through the feature hierarchy. Notably, we do not retrain or fine-tune the model; we let the pretrained transformer diagnose its own motion cues and correct itself. This early regulation stabilizes geometric reasoning in streaming scenarios and leads to improvements in temporal consistency and camera pose robustness across multiple dynamic benchmarks, offering a simple and training-free pathway toward motion-aware streaming reconstruction.
☆ Beyond the Ground Truth: Enhanced Super vision for Image Restoration
Donghun Ryou, Inju Ha, Sanghyeok Chu, Bohyung Han
Deep learning-based image restoration has achieved significant success. However, when addressing real-world degradations, model performance is limited by the quality of ground-truth images in datasets due to practical constraints in data acquisition. To address this limitation, we propose a novel framework that enhances existing ground truth images to provide higher-quality supervision for real-world restoration. Our framework generates perceptually enhanced ground truth images using super-resolution by incorporating adaptive frequency masks, which are learned by a conditional frequency mask generator. These masks guide the optimal fusion of frequency components from the original ground truth and its super-resolved variants, yielding enhanced ground truth images. This frequency-domain mixup preserves the semantic consistency of the original content while selectively enriching perceptual details, preventing hallucinated artifacts that could compromise fidelity. The enhanced ground truth images are used to train a lightweight output refinement network that can be seamlessly integrated with existing restoration models. Extensive experiments demonstrate that our approach consistently improves the quality of restored images. We further validate the effectiveness of both supervision enhancement and output refinement through user studies. Code is available at https://github.com/dhryougit/Beyond-the-Ground-Truth.
☆ UniMo: Unifying 2D Video and 3D Human Motion with an Autoregressive Framework
Youxin Pang, Yong Zhang, Ruizhi Shao, Xiang Deng, Feng Gao, Xu Xiaoming, Xiaoming Wei, Yebin Liu
We propose UniMo, an innovative autoregressive model for joint modeling of 2D human videos and 3D human motions within a unified framework, enabling simultaneous generation and understanding of these two modalities for the first time. Current methods predominantly focus on generating one modality given another as the condition or integrating either of them with other modalities such as text and audio. Unifying 2D videos and 3D motions for simultaneous optimization and generation remains largely unexplored, presenting significant challenges due to their substantial structural and distributional differences. Inspired by the LLM's ability to unify different modalities, our method models videos and 3D motions as a unified tokens sequence, utilizing separate embedding layers to mitigate distribution gaps. Additionally, we devise a sequence modeling strategy that integrates two distinct tasks within a single framework, proving the effectiveness of unified modeling. Moreover, to efficiently align with visual tokens and preserve 3D spatial information, we design a novel 3D motion tokenizer with a temporal expansion strategy, using a single VQ-VAE to produce quantized motion tokens. It features multiple expert decoders that handle body shapes, translation, global orientation, and body poses for reliable 3D motion reconstruction. Extensive experiments demonstrate that our method simultaneously generates corresponding videos and motions while performing accurate motion capture. This work taps into the capacity of LLMs to fuse diverse data types, paving the way for integrating human-centric information into existing models and potentially enabling multimodal, controllable joint modeling of humans, objects, and scenes.
comment : https://carlyx.github.io/UniMo/
☆ Zero-Shot Video Translation and Editing with Frame Spatial-Temporal Correspondence SC
Shuai Yang, Junxin Lin, Yifan Zhou, Ziwei Liu, Chen Change Loy
The remarkable success in text-to-image diffusion models has motivated extensive investigation of their potential for video applications. Zero-shot techniques aim to adapt image diffusion models for videos without requiring further model training. Recent methods largely emphasize integrating inter-frame correspondence into attention mechanisms. However, the soft constraint applied to identify the valid features to attend is insufficient, which could lead to temporal inconsistency. In this paper, we present FRESCO, which integrates intra-frame correspondence with inter-frame correspondence to formulate a more robust spatial-temporal constraint. This enhancement ensures a consistent transformation of semantically similar content between frames. Our method goes beyond attention guidance to explicitly optimize features, achieving high spatial-temporal consistency with the input video, significantly enhancing the visual coherence of manipulated videos. We verify FRESCO adaptations on two zero-shot tasks of video-to-video translation and text-guided video editing. Comprehensive experiments demonstrate the effectiveness of our framework in generating high-quality, coherent videos, highlighting a significant advance over current zero-shot methods.
comment : Code: https://github.com/Sunnycookies/FRESCO-v2, Project: https://williamyang1991.github.io/projects/FRESCOv2/
☆ Dual Cross-Attention Siamese Transformer for Rectal Tumor Regrowth Assessment in Watch-and-Wait Endoscopy
Jorge Tapias Gomez, Despoina Kanata, Aneesh Rangnekar, Christina Lee, Julio Garcia-Aguilar, Joshua Jesse Smith, Harini Veeraraghavan
Increasing evidence supports watch-and-wait (WW) surveillance for patients with rectal cancer who show clinical complete response (cCR) at restaging following total neoadjuvant treatment (TNT). However, objectively accurate methods to early detect local regrowth (LR) from follow-up endoscopy images during WW are essential to manage care and prevent distant metastases. Hence, we developed a Siamese Swin Transformer with Dual Cross-Attention (SSDCA) to combine longitudinal endoscopic images at restaging and follow-up and distinguish cCR from LR. SSDCA leverages pretrained Swin transformers to extract domain agnostic features and enhance robustness to imaging variations. Dual cross attention is implemented to emphasize features from the two scans without requiring any spatial alignment of images to predict response. SSDCA as well as Swin-based baselines were trained using image pairs from 135 patients and evaluated on a held-out set of image pairs from 62 patients. SSDCA produced the best balanced accuracy (81.76\% $\pm$ 0.04), sensitivity (90.07\% $\pm$ 0.08), and specificity (72.86\% $\pm$ 0.05). Robustness analysis showed stable performance irrespective of artifacts including blood, stool, telangiectasia, and poor image quality. UMAP clustering of extracted features showed maximal inter-cluster separation (1.45 $\pm$ 0.18) and minimal intra-cluster dispersion (1.07 $\pm$ 0.19) with SSDCA, confirming discriminative representation learning.
comment : 6 pages, 5 figures, 1 table, submitted to ISBI conference
☆ An Automated Framework for Large-Scale Graph-Based Cerebrovascular Analysis
Daniele Falcetta, Liane S. Canas, Lorenzo Suppa, Matteo Pentassuglia, Jon Cleary, Marc Modat, Sébastien Ourselin, Maria A. Zuluaga
We present CaravelMetrics, a computational framework for automated cerebrovascular analysis that models vessel morphology through skeletonization-derived graph representations. The framework integrates atlas-based regional parcellation, centerline extraction, and graph construction to compute fifteen morphometric, topological, fractal, and geometric features. The features can be estimated globally from the complete vascular network or regionally within arterial territories, enabling multiscale characterization of cerebrovascular organization. Applied to 570 3D TOF-MRA scans from the IXI dataset (ages 20-86), CaravelMetrics yields reproducible vessel graphs capturing age- and sex-related variations and education-associated increases in vascular complexity, consistent with findings reported in the literature. The framework provides a scalable and fully automated approach for quantitative cerebrovascular feature extraction, supporting normative modeling and population-level studies of vascular health and aging.
comment : Submitted to ISBI 2026. 6 pages, 6 figures
☆ Diminishing Returns in Self-Supervised Learning
Oli Bridge, Huey Sun, Botond Branyicskai-Nagy, Charles D'Ornano, Shomit Basu
While transformer-based architectures have taken computer vision and NLP by storm, they often require a vast amount of parameters and training data to attain strong performance. In this work, we experiment with three distinct pre-training, intermediate fine-tuning, and downstream datasets and training objectives to explore their marginal benefits on a small 5M-parameter vision transformer. We find that while pre-training and fine-tuning always help our model but have diminishing returns, intermediate fine-tuning can actually show harmful impact on downstream performance, potentially due to dissimilarity in task mechanics. Taken together, our results suggest that small-scale ViTs benefit most from targeted pre-training and careful data selection, while indiscriminate stacking of intermediate tasks can waste compute and even degrade performance.
☆ Prostate biopsy whole slide image dataset from an underrepresented Middle Eastern population
Peshawa J. Muhammad Ali, Navin Vincent, Saman S. Abdulla, Han N. Mohammed Fadhl, Anders Blilie, Kelvin Szolnoky, Julia Anna Mielcarz, Xiaoyi Ji, Kimmo Kartasalo, Abdulbasit K. Al-Talabani, Nita Mulliqi
Artificial intelligence (AI) is increasingly used in digital pathology. Publicly available histopathology datasets remain scarce, and those that do exist predominantly represent Western populations. Consequently, the generalizability of AI models to populations from less digitized regions, such as the Middle East, is largely unknown. This motivates the public release of our dataset to support the development and validation of pathology AI models across globally diverse populations. We present 339 whole-slide images of prostate core needle biopsies from a consecutive series of 185 patients collected in Erbil, Iraq. The slides are associated with Gleason scores and International Society of Urological Pathology grades assigned independently by three pathologists. Scanning was performed using two high-throughput scanners (Leica and Hamamatsu) and one compact scanner (Grundium). All slides were de-identified and are provided in their native formats without further conversion. The dataset enables grading concordance analyses, color normalization, and cross-scanner robustness evaluations. Data will be deposited in the Bioimage Archive (BIA) under accession code: to be announced (TBA), and released under a CC BY 4.0 license.
comment : 13 pages, 2 figures and 1 table
☆ Traffic Image Restoration under Adverse Weather via Frequency-Aware Mamba
Liwen Pan, Longguang Wang, Guangwei Gao, Jun Wang, Jun Shi, Juncheng Li
Traffic image restoration under adverse weather conditions remains a critical challenge for intelligent transportation systems. Existing methods primarily focus on spatial-domain modeling but neglect frequency-domain priors. Although the emerging Mamba architecture excels at long-range dependency modeling through patch-wise correlation analysis, its potential for frequency-domain feature extraction remains unexplored. To address this, we propose Frequency-Aware Mamba (FAMamba), a novel framework that integrates frequency guidance with sequence modeling for efficient image restoration. Our architecture consists of two key components: (1) a Dual-Branch Feature Extraction Block (DFEB) that enhances local-global interaction via bidirectional 2D frequency-adaptive scanning, dynamically adjusting traversal paths based on sub-band texture distributions; and (2) a Prior-Guided Block (PGB) that refines texture details through wavelet-based high-frequency residual learning, enabling high-quality image reconstruction with precise details. Meanwhile, we design a novel Adaptive Frequency Scanning Mechanism (AFSM) for the Mamba architecture, which enables the Mamba to achieve frequency-domain scanning across distinct subgraphs, thereby fully leveraging the texture distribution characteristics inherent in subgraph structures. Extensive experiments demonstrate the efficiency and effectiveness of FAMamba.
comment : 12pages, 13 figures, 5tables
☆ PULSE: A Unified Multi-Task Architecture for Cardiac Segmentation, Diagnosis, and Few-Shot Cross-Modality Clinical Adaptation
Hania Ghouse, Maryam Alsharqi, Farhad R. Nezami, Muzammil Behzad
Cardiac image analysis remains fragmented across tasks: anatomical segmentation, disease classification, and grounded clinical report generation are typically handled by separate networks trained under different data regimes. No existing framework unifies these objectives within a single architecture while retaining generalization across imaging modalities and datasets. We introduce PULSE, a multi-task vision-language framework built on self-supervised representations and optimized through a composite supervision strategy that balances region overlap learning, pixel wise classification fidelity, and boundary aware IoU refinement. A multi-scale token reconstruction decoder enables anatomical segmentation, while shared global representations support disease classification and clinically grounded text output allowing the model to transition from pixels to structures and finally clinical reasoning within one architecture. Unlike prior task-specific pipelines, PULSE learns task-invariant cardiac priors, generalizes robustly across datasets, and can be adapted to new imaging modalities with minimal supervision. This moves the field closer to a scalable, foundation style cardiac analysis framework.
☆ CoDA: From Text-to-Image Diffusion Models to Training-Free Dataset Distillation
Letian Zhou, Songhua Liu, Xinchao Wang
Prevailing Dataset Distillation (DD) methods leveraging generative models confront two fundamental limitations. First, despite pioneering the use of diffusion models in DD and delivering impressive performance, the vast majority of approaches paradoxically require a diffusion model pre-trained on the full target dataset, undermining the very purpose of DD and incurring prohibitive training costs. Second, although some methods turn to general text-to-image models without relying on such target-specific training, they suffer from a significant distributional mismatch, as the web-scale priors encapsulated in these foundation models fail to faithfully capture the target-specific semantics, leading to suboptimal performance. To tackle these challenges, we propose Core Distribution Alignment (CoDA), a framework that enables effective DD using only an off-the-shelf text-to-image model. Our key idea is to first identify the "intrinsic core distribution" of the target dataset using a robust density-based discovery mechanism. We then steer the generative process to align the generated samples with this core distribution. By doing so, CoDA effectively bridges the gap between general-purpose generative priors and target semantics, yielding highly representative distilled datasets. Extensive experiments suggest that, without relying on a generative model specifically trained on the target dataset, CoDA achieves performance on par with or even superior to previous methods with such reliance across all benchmarks, including ImageNet-1K and its subsets. Notably, it establishes a new state-of-the-art accuracy of 60.4% at the 50-images-per-class (IPC) setup on ImageNet-1K. Our code is available on the project webpage: https://github.com/zzzlt422/CoDA
comment : 34 pages, 24 figures
☆ Heatmap Pooling Network for Action Recognition from RGB Videos
Mengyuan Liu, Jinfu Liu, Yongkang Jiang, Bin He
Human action recognition (HAR) in videos has garnered widespread attention due to the rich information in RGB videos. Nevertheless, existing methods for extracting deep features from RGB videos face challenges such as information redundancy, susceptibility to noise and high storage costs. To address these issues and fully harness the useful information in videos, we propose a novel heatmap pooling network (HP-Net) for action recognition from videos, which extracts information-rich, robust and concise pooled features of the human body in videos through a feedback pooling module. The extracted pooled features demonstrate obvious performance advantages over the previously obtained pose data and heatmap features from videos. In addition, we design a spatial-motion co-learning module and a text refinement modulation module to integrate the extracted pooled features with other multimodal data, enabling more robust action recognition. Extensive experiments on several benchmarks namely NTU RGB+D 60, NTU RGB+D 120, Toyota-Smarthome and UAV-Human consistently verify the effectiveness of our HP-Net, which outperforms the existing human action recognition methods. Our code is publicly available at: https://github.com/liujf69/HPNet-Action.
comment : Final Version of IEEE Transactions on Pattern Analysis and Machine Intelligence
☆ Lean Unet: A Compact Model for Image Segmentation
Ture Hassler, Ida Åkerholm, Marcus Nordström, Gabriele Balletti, Orcun Goksel
Unet and its variations have been standard in semantic image segmentation, especially for computer assisted radiology. Current Unet architectures iteratively downsample spatial resolution while increasing channel dimensions to preserve information content. Such a structure demands a large memory footprint, limiting training batch sizes and increasing inference latency. Channel pruning compresses Unet architecture without accuracy loss, but requires lengthy optimization and may not generalize across tasks and datasets. By investigating Unet pruning, we hypothesize that the final structure is the crucial factor, not the channel selection strategy of pruning. Based on our observations, we propose a lean Unet architecture (LUnet) with a compact, flat hierarchy where channels are not doubled as resolution is halved. We evaluate on a public MRI dataset allowing comparable reporting, as well as on two internal CT datasets. We show that a state-of-the-art pruning solution (STAMP) mainly prunes from the layers with the highest number of channels. Comparatively, simply eliminating a random channel at the pruning-identified layer or at the largest layer achieves similar or better performance. Our proposed LUnet with fixed architectures and over 30 times fewer parameters achieves performance comparable to both conventional Unet counterparts and data-adaptively pruned networks. The proposed lean Unet with constant channel count across layers requires far fewer parameters while achieving performance superior to standard Unet for the same total number of parameters. Skip connections allow Unet bottleneck channels to be largely reduced, unlike standard encoder-decoder architectures requiring increased bottleneck channels for information propagation.
☆ A Robust Camera-based Method for Breath Rate Measurement
Alexey Protopopov
Proliferation of cheap and accessible cameras makes it possible to measure a subject's breath rate from video footage alone. Recent works on this topic have proposed a variety of approaches for accurately measuring human breath rate, however they are either tested in near-ideal conditions, or produce results that are not sufficiently accurate. The present study proposes a more robust method to measure breath rate in humans with minimal hardware requirements using a combination of mathematical transforms with a relative deviation from the ground truth of less than 5%. The method was tested on videos taken from 14 volunteers with a total duration of over 2 hours 30 minutes. The obtained results were compared to reference data and the average mean absolute error was found to be at 0.57 respirations per minute, which is noticeably better than the results from previous works. The breath rate measurement method proposed in the present article is more resistant to distortions caused by subject movement and thus allows one to remotely measure the subject's breath rate without any significant limitations on the subject's behavior.
comment : 9 pages, 4 figures, 2 tables
☆ HieroGlyphTranslator: Automatic Recognition and Translation of Egyptian Hieroglyphs to English
Ahmed Nasser, Marwan Mohamed, Alaa Sherif, Basmala Mahmoud, Shereen Yehia, Asmaa Saad, Mariam S. El-Rahmany, Ensaf H. Mohamed
Egyptian hieroglyphs, the ancient Egyptian writing system, are composed entirely of drawings. Translating these glyphs into English poses various challenges, including the fact that a single glyph can have multiple meanings. Deep learning translation applications are evolving rapidly, producing remarkable results that significantly impact our lives. In this research, we propose a method for the automatic recognition and translation of ancient Egyptian hieroglyphs from images to English. This study utilized two datasets for classification and translation: the Morris Franken dataset and the EgyptianTranslation dataset. Our approach is divided into three stages: segmentation (using Contour and Detectron2), mapping symbols to Gardiner codes, and translation (using the CNN model). The model achieved a BLEU score of 42.2, a significant result compared to previous research.
☆ LSRS: Latent Scale Rejection Sampling for Visual Autoregressive Modeling
Hong-Kai Zheng, Piji Li
Visual Autoregressive (VAR) modeling approach for image generation proposes autoregressive processing across hierarchical scales, decoding multiple tokens per scale in parallel. This method achieves high-quality generation while accelerating synthesis. However, parallel token sampling within a scale may lead to structural errors, resulting in suboptimal generated images. To mitigate this, we propose Latent Scale Rejection Sampling (LSRS), a method that progressively refines token maps in the latent scale during inference to enhance VAR models. Our method uses a lightweight scoring model to evaluate multiple candidate token maps sampled at each scale, selecting the high-quality map to guide subsequent scale generation. By prioritizing early scales critical for structural coherence, LSRS effectively mitigates autoregressive error accumulation while maintaining computational efficiency. Experiments demonstrate that LSRS significantly improves VAR's generation quality with minimal additional computational overhead. For the VAR-d30 model, LSRS increases the inference time by merely 1% while reducing its FID score from 1.95 to 1.78. When the inference time is increased by 15%, the FID score can be further reduced to 1.66. LSRS offers an efficient test-time scaling solution for enhancing VAR-based generation.
☆ Adapt Vision : Efficient Vision -Language Models via Adaptive Visual Acquisition
Zichuan Lin, Yicheng Liu, Yang Yang, Lvfang Tao, Deheng Ye
Vision-Language Models (VLMs) have achieved remarkable success in visual question answering tasks, but their reliance on large numbers of visual tokens introduces significant computational overhead. While existing efficient VLM approaches reduce visual tokens through fixed-ratio compression, they operate passively and lack the ability to adapt to varying task requirements. This motivates a fundamental question: Can VLMs autonomously determine the minimum number of visual tokens required for each sample? Inspired by human active vision mechanisms, we introduce AdaptVision, an efficient VLM paradigm that enables adaptive visual token acquisition through a coarse-to-fine approach. Our model initially processes compressed visual tokens from low-resolution images and selectively acquires additional visual information by invoking a bounding box tool to crop key regions when necessary. We train AdaptVision using a reinforcement learning framework that carefully balances accuracy and efficiency. Central to our approach is Decoupled Turn Policy Optimization (DTPO), which decouples the learning objective into two components: (1) tool learning, which optimizes correct tool utilization, and (2) accuracy improvement, which refines the generated responses to improve answer correctness. Based on this formulation, we further decouple advantage estimation by computing separate advantages for tokens associated with each objective. This formulation enables more effective optimization for AdaptVision compared to vanilla GRPO. Comprehensive experiments across multiple VQA benchmarks demonstrate that AdaptVision achieves superior performance while consuming substantially fewer visual tokens than state-of-the-art efficient VLM methods.
comment : 15 pages, 9 figures
☆ Research on Brain Tumor Classification Method Based on Improved ResNet34 Network
Yufeng Li, Wenchao Zhao, Bo Dang, Weimin Wang
Previously, image interpretation in radiology relied heavily on manual methods. However, manual classification of brain tumor medical images is time-consuming and labor-intensive. Even with shallow convolutional neural network models, the accuracy is not ideal. To improve the efficiency and accuracy of brain tumor image classification, this paper proposes a brain tumor classification model based on an improved ResNet34 network. This model uses the ResNet34 residual network as the backbone network and incorporates multi-scale feature extraction. It uses a multi-scale input module as the first layer of the ResNet34 network and an Inception v2 module as the residual downsampling layer. Furthermore, a channel attention mechanism module assigns different weights to different channels of the image from a channel domain perspective, obtaining more important feature information. The results after a five-fold crossover experiment show that the average classification accuracy of the improved network model is approximately 98.8%, which is not only 1% higher than ResNet34, but also only 80% of the number of parameters of the original model. Therefore, the improved network model not only improves accuracy but also reduces clutter, achieving a classification effect with fewer parameters and higher accuracy.
☆ Fully Unsupervised Self-debiasing of Text-to-Image Diffusion Models WACV 2026
Korada Sri Vardhana, Shrikrishna Lolla, Soma Biswas
Text-to-image (T2I) diffusion models have achieved widespread success due to their ability to generate high-resolution, photorealistic images. These models are trained on large-scale datasets, like LAION-5B, often scraped from the internet. However, since this data contains numerous biases, the models inherently learn and reproduce them, resulting in stereotypical outputs. We introduce SelfDebias, a fully unsupervised test-time debiasing method applicable to any diffusion model that uses a UNet as its noise predictor. SelfDebias identifies semantic clusters in an image encoder's embedding space and uses these clusters to guide the diffusion process during inference, minimizing the KL divergence between the output distribution and the uniform distribution. Unlike supervised approaches, SelfDebias does not require human-annotated datasets or external classifiers trained for each generated concept. Instead, it is designed to automatically identify semantic modes. Extensive experiments show that SelfDebias generalizes across prompts and diffusion model architectures, including both conditional and unconditional models. It not only effectively debiases images along key demographic dimensions while maintaining the visual fidelity of the generated images, but also more abstract concepts for which identifying biases is also challenging.
comment : Accepted at WACV 2026
☆ Thinking with Programming Vision : Towards a Unified View for Thinking with Images
Zirun Guo, Minjie Hong, Feng Zhang, Kai Jia, Tao Jin
Multimodal large language models (MLLMs) that think with images can interactively use tools to reason about visual inputs, but current approaches often rely on a narrow set of tools with limited real-world necessity and scalability. In this work, we first reveal a critical and previously overlooked weakness: even state-of-the-art MLLMs are surprisingly brittle, showing significant performance degradation on images with simple orientation changes or natural corruptions, underscoring the need for more robust tool-based reasoning. To address this, we propose CodeVision, a flexible and scalable code-as-tool framework where the model generates code as a universal interface to invoke any image operation, moving beyond fixed tool registries. We train our model using a two-stage methodology, beginning with Supervised Fine-Tuning (SFT) on a high-quality dataset curated for complex, multi-turn tool composition and error recovery, followed by Reinforcement Learning (RL) with a novel and dense process reward function to encourage strategic and efficient tool use. To facilitate this research, we construct new SFT and RL datasets and introduce a challenging new benchmark suite designed to rigorously evaluate robustness to orientation changes and multi-tool reasoning. Experiments on Qwen2.5-VL and Qwen3-VL series show that our approach significantly improves model performance and fosters emergent capabilities such as flexible tool composition, efficient chained execution, and robust error recovery from runtime feedback. Code is available at https://github.com/ByteDance-BandAI/CodeVision.
☆ Dual-level Modality Debiasing Learning for Unsupervised Visible-Infrared Person Re-Identification
Jiaze Li, Yan Lu, Bin Liu, Guojun Yin, Mang Ye
Two-stage learning pipeline has achieved promising results in unsupervised visible-infrared person re-identification (USL-VI-ReID). It first performs single-modality learning and then operates cross-modality learning to tackle the modality discrepancy. Although promising, this pipeline inevitably introduces modality bias: modality-specific cues learned in the single-modality training naturally propagate into the following cross-modality learning, impairing identity discrimination and generalization. To address this issue, we propose a Dual-level Modality Debiasing Learning (DMDL) framework that implements debiasing at both the model and optimization levels. At the model level, we propose a Causality-inspired Adjustment Intervention (CAI) module that replaces likelihood-based modeling with causal modeling, preventing modality-induced spurious patterns from being introduced, leading to a low-biased model. At the optimization level, a Collaborative Bias-free Training (CBT) strategy is introduced to interrupt the propagation of modality bias across data, labels, and features by integrating modality-specific augmentation, label refinement, and feature alignment. Extensive experiments on benchmark datasets demonstrate that DMDL could enable modality-invariant feature learning and a more generalized model.
☆ Out-of-the-box: Black-box Causal Attacks on Object Detectors
Melane Navaratnarajah, David A. Kelly, Hana Chockler
Adversarial perturbations are a useful way to expose vulnerabilities in object detectors. Existing perturbation methods are frequently white-box and architecture specific. More importantly, while they are often successful, it is rarely clear why they work. Insights into the mechanism of this success would allow developers to understand and analyze these attacks, as well as fine-tune the model to prevent them. This paper presents BlackCAtt, a black-box algorithm and a tool, which uses minimal, causally sufficient pixel sets to construct explainable, imperceptible, reproducible, architecture-agnostic attacks on object detectors. BlackCAtt combines causal pixels with bounding boxes produced by object detectors to create adversarial attacks that lead to the loss, modification or addition of a bounding box. BlackCAtt works across different object detectors of different sizes and architectures, treating the detector as a black box. We compare the performance of BlackCAtt with other black-box attack methods and show that identification of causal pixels leads to more precisely targeted and less perceptible attacks. On the COCO test dataset, our approach is 2.7 times better than the baseline in removing a detection, 3.86 times better in changing a detection, and 5.75 times better in triggering new, spurious, detections. The attacks generated by BlackCAtt are very close to the original image, and hence imperceptible, demonstrating the power of causal pixels.
☆ PosA-VLA: Enhancing Action Generation via Pose-Conditioned Anchor Attention
Ziwen Li, Xin Wang, Hanlue Zhang, Runnan Chen, Runqi Lin, Xiao He, Han Huang, Yandong Guo, Fakhri Karray, Tongliang Liu, Mingming Gong
The Vision-Language-Action (VLA) models have demonstrated remarkable performance on embodied tasks and shown promising potential for real-world applications. However, current VLAs still struggle to produce consistent and precise target-oriented actions, as they often generate redundant or unstable motions along trajectories, limiting their applicability in time-sensitive scenarios.In this work, we attribute these redundant actions to the spatially uniform perception field of existing VLAs, which causes them to be distracted by target-irrelevant objects, especially in complex environments.To address this issue, we propose an efficient PosA-VLA framework that anchors visual attention via pose-conditioned supervision, consistently guiding the model's perception toward task-relevant regions. The pose-conditioned anchor attention mechanism enables the model to better align instruction semantics with actionable visual cues, thereby improving action generation precision and efficiency. Moreover, our framework adopts a lightweight architecture and requires no auxiliary perception modules (e.g., segmentation or grounding networks), ensuring efficient inference. Extensive experiments verify that our method executes embodied tasks with precise and time-efficient behavior across diverse robotic manipulation benchmarks and shows robust generalization in a variety of challenging environments.
☆ DINO-RotateMatch: A Rotation-Aware Deep Framework for Robust Image Matching in Large-Scale 3D Reconstruction
Kaichen Zhang, Tianxiang Sheng, Xuanming Shi
This paper presents DINO-RotateMatch, a deep-learning framework designed to address the chal lenges of image matching in large-scale 3D reconstruction from unstructured Internet images. The method integrates a dataset-adaptive image pairing strategy with rotation-aware keypoint extraction and matching. DINO is employed to retrieve semantically relevant image pairs in large collections, while rotation-based augmentation captures orientation-dependent local features using ALIKED and Light Glue. Experiments on the Kaggle Image Matching Challenge 2025 demonstrate consistent improve ments in mean Average Accuracy (mAA), achieving a Silver Award (47th of 943 teams). The results confirm that combining self-supervised global descriptors with rotation-enhanced local matching offers a robust and scalable solution for large-scale 3D reconstruction.
comment : 9 pages, 5 figures, 1 table
☆ Structured Uncertainty Similarity Score (SUSS): Learning a Probabilistic, Interpretable, Perceptual Metric Between Images
Paula Seidler, Neill D. F. Campbell, Ivor J A Simpson
Perceptual similarity scores that align with human vision are critical for both training and evaluating computer vision models. Deep perceptual losses, such as LPIPS, achieve good alignment but rely on complex, highly non-linear discriminative features with unknown invariances, while hand-crafted measures like SSIM are interpretable but miss key perceptual properties. We introduce the Structured Uncertainty Similarity Score (SUSS); it models each image through a set of perceptual components, each represented by a structured multivariate Normal distribution. These are trained in a generative, self-supervised manner to assign high likelihood to human-imperceptible augmentations. The final score is a weighted sum of component log-probabilities with weights learned from human perceptual datasets. Unlike feature-based methods, SUSS learns image-specific linear transformations of residuals in pixel space, enabling transparent inspection through decorrelated residuals and sampling. SUSS aligns closely with human perceptual judgments, shows strong perceptual calibration across diverse distortion types, and provides localized, interpretable explanations of its similarity assessments. We further demonstrate stable optimization behavior and competitive performance when using SUSS as a perceptual loss for downstream imaging tasks.
☆ Active Visual Perception: Opportunities and Challenges
Yian Li, Xiaoyu Guo, Hao Zhang, Shuiwang Li, Xiaowei Dai
Active visual perception refers to the ability of a system to dynamically engage with its environment through sensing and action, allowing it to modify its behavior in response to specific goals or uncertainties. Unlike passive systems that rely solely on visual data, active visual perception systems can direct attention, move sensors, or interact with objects to acquire more informative data. This approach is particularly powerful in complex environments where static sensing methods may not provide sufficient information. Active visual perception plays a critical role in numerous applications, including robotics, autonomous vehicles, human-computer interaction, and surveillance systems. However, despite its significant promise, there are several challenges that need to be addressed, including real-time processing of complex visual data, decision-making in dynamic environments, and integrating multimodal sensory inputs. This paper explores both the opportunities and challenges inherent in active visual perception, providing a comprehensive overview of its potential, current research, and the obstacles that must be overcome for broader adoption.
☆ GaussianBlender: Instant Stylization of 3D Gaussians with Disentangled Latent Spaces
Melis Ocal, Xiaoyan Xing, Yue Li, Ngo Anh Vien, Sezer Karaoglu, Theo Gevers
3D stylization is central to game development, virtual reality, and digital arts, where the demand for diverse assets calls for scalable methods that support fast, high-fidelity manipulation. Existing text-to-3D stylization methods typically distill from 2D image editors, requiring time-intensive per-asset optimization and exhibiting multi-view inconsistency due to the limitations of current text-to-image models, which makes them impractical for large-scale production. In this paper, we introduce GaussianBlender, a pioneering feed-forward framework for text-driven 3D stylization that performs edits instantly at inference. Our method learns structured, disentangled latent spaces with controlled information sharing for geometry and appearance from spatially-grouped 3D Gaussians. A latent diffusion model then applies text-conditioned edits on these learned representations. Comprehensive evaluations show that GaussianBlender not only delivers instant, high-fidelity, geometry-preserving, multi-view consistent stylization, but also surpasses methods that require per-instance test-time optimization - unlocking practical, democratized 3D stylization at scale.
☆ ConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformer s
Feice Huang, Zuliang Han, Xing Zhou, Yihuang Chen, Lifei Zhu, Haoqian Wang
Diffusion transformers have demonstrated strong capabilities in generating high-quality images. However, as model size increases, the growing memory footprint and inference latency pose significant challenges for practical deployment. Recent studies in large language models (LLMs) show that rotation-based techniques can smooth outliers and enable 4-bit quantization, but these approaches often incur substantial overhead and struggle with row-wise outliers in diffusion transformers. To address these challenges, we propose ConvRot, a group-wise rotation-based quantization method that leverages regular Hadamard transform (RHT) to suppress both row-wise and column-wise outliers while reducing complexity from quadratic to linear. Building on this, we design ConvLinear4bit, a plug-and-play module that integrates rotation, quantization, GEMM, and dequantization, enabling W4A4 inference without retraining and preserving visual quality. Experiments on FLUX.1-dev demonstrate a 2.26$\times$ speedup and 4.05$\times$ memory reduction while maintaining image fidelity. To our knowledge, this is the first application of rotation-based quantization for plug-and-play W4A4 inference in diffusion transformers.
☆ Colon-X: Advancing Intelligent Colonoscopy from Multimodal Understanding to Clinical Reasoning
Ge-Peng Ji, Jingyi Liu, Deng-Ping Fan, Nick Barnes
In this study, we present Colon-X, an open initiative aimed at advancing multimodal intelligence in colonoscopy. We begin by constructing ColonVQA, the most comprehensive multimodal dataset ever built for colonoscopy, featuring over 1.1M+ visual question answering entries across 76 clinical findings and 18 multimodal tasks. Beyond serving as a community-wide data foundation, we further investigate a critical yet underexplored transition in colonoscopy - evolving from multimodal understanding to clinical reasoning: (a) To capture the current landscape of multimodal understanding behaviors, we systematically assess the generalizability of 22 multimodal large language models and examine their reliability under human-induced perturbations. The results reveal that clinical outputs from leading MLLMs remain far from robust and trustworthy. (b) To narrow this gap, we further explore reasoning-centric intelligence tailored for colonoscopy. Specifically, we curate ColonReason, a clinically grounded reasoning dataset annotated through a multi-expert debating pipeline, and develop ColonR1, the first R1-styled model incorporating task-adaptive rewarding and gradient-stable optimization techniques. Under data-scarce conditions, our ColonR1 achieves 56.61% overall accuracy, outperforming supervised fine-tuning by 25.22%, and sets a new reasoning-enabled baseline for multimodal colonoscopy analysis. All data and model resources are publicly available at https://github.com/ai4colonoscopy/Colon-X.
comment : Technical report
☆ ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videos
Qi'ao Xu, Tianwen Qian, Yuqian Fu, Kailing Li, Yang Jiao, Jiacheng Zhang, Xiaoling Wang, Liang He
A core capability towards general embodied intelligence lies in localizing task-relevant objects from an egocentric perspective, formulated as Spatio-Temporal Video Grounding (STVG). Despite recent progress, existing STVG studies remain largely confined to object-centric and descriptive instructions, neglecting the task-oriented reasoning that is crucial for embodied agents to accomplish goal-directed interactions. To bridge this gap, we introduce \textbf{ToG-Bench}, the first task-oriented spatio-temporal video grounding benchmark for egocentric videos. ToG-Bench is characterized by three key features: (1) \textbf{Task-oriented Grounding}, which requires identifying and localizing objects based on intended tasks rather than straightforward descriptions; (2) \textbf{Explicit-Implicit Dual Grounding}, where target objects can be either explicitly mentioned or implicitly inferred by contextual reasoning; (3) \textbf{One-to-Many Grounding}, where a single instruction may correspond to multiple objects involved in task execution. Built upon videos sourced from ScanNet, ToG-Bench comprises 100 annotated clips with 2,704 task-oriented grounding instructions, constructed via a semi-automated pipeline that combines foundation model annotation and human refinement. In addition, we introduce a set of task-level evaluation metrics tailored for multi-object and explicit-implicit object grounding, and systematically benchmark seven state-of-the-art MLLMs. Extensive experiments reveal the intrinsic challenges of task-oriented STVG and substantial performance gaps across explicit-implicit and multi-object grounding, highlighting the difficulty of bridging perception and interaction in embodied scenarios. Data and code will be released at: \href{https://github.com/qaxuDev/ToG-Bench}{https://github.com/qaxuDev/ToG-Bench}..
comment : 26 pages
☆ Multi-Scale Visual Prompt ing for Lightweight Small-Image Classification
Salim Khazem
Visual prompting has recently emerged as an efficient strategy to adapt vision models using lightweight, learnable parameters injected into the input space. However, prior work mainly targets large Vision Transformers and high-resolution datasets such as ImageNet. In contrast, small-image benchmarks like MNIST, Fashion-MNIST, and CIFAR-10 remain widely used in education, prototyping, and research, yet have received little attention in the context of prompting. In this paper, we introduce \textbf{Multi-Scale Visual Prompting (MSVP)}, a simple and generic module that learns a set of global, mid-scale, and local prompt maps fused with the input image via a lightweight $1 \times 1$ convolution. MSVP is backbone-agnostic, adds less than $0.02\%$ parameters, and significantly improves performance across CNN and Vision Transformer backbones. We provide a unified benchmark on MNIST, Fashion-MNIST, and CIFAR-10 using a simple CNN, ResNet-18, and a small Vision Transformer. Our method yields consistent improvements with negligible computational overhead. We further include ablations on prompt scales, fusion strategies, and backbone architectures, along with qualitative analyzes using prompt visualizations and Grad-CAM. Our results demonstrate that multi-scale prompting provides an effective inductive bias even on low-resolution images.
☆ Cyclical Temporal Encoding and Hybrid Deep Ensembles for Multistep Energy Forecasting
Salim Khazem, Houssam Kanso
Accurate electricity consumption forecasting is essential for demand management and smart grid operations. This paper introduces a unified deep learning framework that integrates cyclical temporal encoding with hybrid LSTM-CNN architectures to enhance multistep energy forecasting. We systematically transform calendar-based attributes using sine cosine encodings to preserve periodic structure and evaluate their predictive relevance through correlation analysis. To exploit both long-term seasonal effects and short-term local patterns, we employ an ensemble model composed of an LSTM, a CNN, and a meta-learner of MLP regressors specialized for each forecast horizon. Using a one year national consumption dataset, we conduct an extensive experimental study including ablation analyses with and without cyclical encodings and calendar features and comparisons with established baselines from the literature. Results demonstrate consistent improvements across all seven forecast horizons, with our hybrid model achieving lower RMSE and MAE than individual architectures and prior methods. These findings confirm the benefit of combining cyclical temporal representations with complementary deep learning structures. To our knowledge, this is the first work to jointly evaluate temporal encodings, calendar-based features, and hybrid ensemble architectures within a unified short-term energy forecasting framework.
☆ Optical Context Compression Is Just (Bad) Autoencoding
Ivan Yee Lee, Cheng Yang, Taylor Berg-Kirkpatrick
DeepSeek-OCR demonstrates that rendered text can be reconstructed with high fidelity from a small number of vision tokens. This finding has sparked excitement about vision-based context compression for language models. But the evaluation stops at reconstruction; whether these representations help language modeling remains untested. We test two assumptions implicit in the optical-compression narrative: that vision-based compression provides unique advantages for text reconstruction from compressed representations, and that DeepSeek-OCR's reconstruction results are evidence that vision-based compression will be useful for language modeling. Comparing their vision encoder against simple alternatives--parameter-free mean pooling and a learned hierarchical encoder--we find that these simple approaches match or surpass vision for reconstruction at matched compression ratios, and outperform it for language modeling--where vision-based compression fails to beat truncation. The excitement around optical context compression outpaces the evidence. Code and checkpoints are available at https://github.com/ivnle/bad-autoencoding
☆ MKSNet: Advanced Small Object Detection in Remote Sensing Imagery with Multi-Kernel and Dual Attention Mechanisms
Jiahao Zhang, Xiao Zhao, Guangyu Gao
Deep convolutional neural networks (DCNNs) have substantially advanced object detection capabilities, particularly in remote sensing imagery. However, challenges persist, especially in detecting small objects where the high resolution of these images and the small size of target objects often result in a loss of critical information in the deeper layers of conventional CNNs. Additionally, the extensive spatial redundancy and intricate background details typical in remote-sensing images tend to obscure these small targets. To address these challenges, we introduce Multi-Kernel Selection Network (MKSNet), a novel network architecture featuring a novel Multi-Kernel Selection mechanism. The MKS mechanism utilizes large convolutional kernels to effectively capture an extensive range of contextual information. This innovative design allows for adaptive kernel size selection, significantly enhancing the network's ability to dynamically process and emphasize crucial spatial details for small object detection. Furthermore, MKSNet also incorporates a dual attention mechanism, merging spatial and channel attention modules. The spatial attention module adaptively fine-tunes the spatial weights of feature maps, focusing more intensively on relevant regions while mitigating background noise. Simultaneously, the channel attention module optimizes channel information selection, improving feature representation and detection accuracy. Empirical evaluations on the DOTA-v1.0 and HRSC2016 benchmark demonstrate that MKSNet substantially surpasses existing state-of-the-art models in detecting small objects in remote sensing images. These results highlight MKSNet's superior ability to manage the complexities associated with multi-scale and high-resolution image data, confirming its effectiveness and innovation in remote sensing object detection.
☆ FeatureLens: A Highly Generalizable and Interpretable Framework for Detecting Adversarial Examples Based on Image Features
Zhigang Yang, Yuan Liu, Jiawei Zhang, Puning Zhang, Xinqiang Ma
Although the remarkable performance of deep neural networks (DNNs) in image classification, their vulnerability to adversarial attacks remains a critical challenge. Most existing detection methods rely on complex and poorly interpretable architectures, which compromise interpretability and generalization. To address this, we propose FeatureLens, a lightweight framework that acts as a lens to scrutinize anomalies in image features. Comprising an Image Feature Extractor (IFE) and shallow classifiers (e.g., SVM, MLP, or XGBoost) with model sizes ranging from 1,000 to 30,000 parameters, FeatureLens achieves high detection accuracy ranging from 97.8% to 99.75% in closed-set evaluation and 86.17% to 99.6% in generalization evaluation across FGSM, PGD, CW, and DAmageNet attacks, using only 51 dimensional features. By combining strong detection performance with excellent generalization, interpretability, and computational efficiency, FeatureLens offers a practical pathway toward transparent and effective adversarial defense.
☆ ReCamDriving: LiDAR-Free Camera-Controlled Novel Trajectory Video Generation
Yaokun Li, Shuaixian Wang, Mantang Guo, Jiehui Huang, Taojun Ding, Mu Hu, Kaixuan Wang, Shaojie Shen, Guang Tan
We propose ReCamDriving, a purely vision-based, camera-controlled novel-trajectory video generation framework. While repair-based methods fail to restore complex artifacts and LiDAR-based approaches rely on sparse and incomplete cues, ReCamDriving leverages dense and scene-complete 3DGS renderings for explicit geometric guidance, achieving precise camera-controllable generation. To mitigate overfitting to restoration behaviors when conditioned on 3DGS renderings, ReCamDriving adopts a two-stage training paradigm: the first stage uses camera poses for coarse control, while the second stage incorporates 3DGS renderings for fine-grained viewpoint and geometric guidance. Furthermore, we present a 3DGS-based cross-trajectory data curation strategy to eliminate the train-test gap in camera transformation patterns, enabling scalable multi-trajectory supervision from monocular videos. Based on this strategy, we construct the ParaDrive dataset, containing over 110K parallel-trajectory video pairs. Extensive experiments demonstrate that ReCamDriving achieves state-of-the-art camera controllability and structural consistency.
comment : Project page: https://recamdriving.github.io/
☆ LAMP: Language-Assisted Motion Planning for Controllable Video Generation
Muhammed Burak Kizil, Enes Sanli, Niloy J. Mitra, Erkut Erdem, Aykut Erdem, Duygu Ceylan
Video generation has achieved remarkable progress in visual fidelity and controllability, enabling conditioning on text, layout, or motion. Among these, motion control - specifying object dynamics and camera trajectories - is essential for composing complex, cinematic scenes, yet existing interfaces remain limited. We introduce LAMP that leverages large language models (LLMs) as motion planners to translate natural language descriptions into explicit 3D trajectories for dynamic objects and (relatively defined) cameras. LAMP defines a motion domain-specific language (DSL), inspired by cinematography conventions. By harnessing program synthesis capabilities of LLMs, LAMP generates structured motion programs from natural language, which are deterministically mapped to 3D trajectories. We construct a large-scale procedural dataset pairing natural text descriptions with corresponding motion programs and 3D trajectories. Experiments demonstrate LAMP's improved performance in motion controllability and alignment with user intent compared to state-of-the-art alternatives establishing the first framework for generating both object and camera motions directly from natural language specifications.
☆ Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding NeurIPS 2025
Haoran Zhou, Gim Hee Lee
Recent advancements in foundation models for 2D vision have substantially improved the analysis of dynamic scenes from monocular videos. However, despite their strong generalization capabilities, these models often lack 3D consistency, a fundamental requirement for understanding scene geometry and motion, thereby causing severe spatial misalignment and temporal flickering in complex 3D environments. In this paper, we present Motion4D, a novel framework that addresses these challenges by integrating 2D priors from foundation models into a unified 4D Gaussian Splatting representation. Our method features a two-part iterative optimization framework: 1) Sequential optimization, which updates motion and semantic fields in consecutive stages to maintain local consistency, and 2) Global optimization, which jointly refines all attributes for long-term coherence. To enhance motion accuracy, we introduce a 3D confidence map that dynamically adjusts the motion priors, and an adaptive resampling process that inserts new Gaussians into under-represented regions based on per-pixel RGB and semantic errors. Furthermore, we enhance semantic coherence through an iterative refinement process that resolves semantic inconsistencies by alternately optimizing the semantic fields and updating prompts of SAM2. Extensive evaluations demonstrate that our Motion4D significantly outperforms both 2D foundation models and existing 3D-based approaches across diverse scene understanding tasks, including point-based tracking, video object segmentation, and novel view synthesis. Our code is available at https://hrzhou2.github.io/motion4d-web/.
comment : Accepted to NeurIPS 2025
☆ Memory-Guided Point Cloud Completion for Dental Reconstruction
Jianan Sun, Yukang Huang, Dongzhihan Wang, Mingyu Fan
Partial dental point clouds often suffer from large missing regions caused by occlusion and limited scanning views, which bias encoder-only global features and force decoders to hallucinate structures. We propose a retrieval-augmented framework for tooth completion that integrates a prototype memory into standard encoder--decoder pipelines. After encoding a partial input into a global descriptor, the model retrieves the nearest manifold prototype from a learnable memory and fuses it with the query feature through confidence-gated weighting before decoding. The memory is optimized end-to-end and self-organizes into reusable tooth-shape prototypes without requiring tooth-position labels, thereby providing structural priors that stabilize missing-region inference and free decoder capacity for detail recovery. The module is plug-and-play and compatible with common completion backbones, while keeping the same training losses. Experiments on a self-processed Teeth3DS benchmark demonstrate consistent improvements in Chamfer Distance, with visualizations showing sharper cusps, ridges, and interproximal transitions. Our approach provides a simple yet effective way to exploit cross-sample regularities for more accurate and faithful dental point-cloud completion.
☆ HBFormer: A Hybrid-Bridge Transformer for Microtumor and Miniature Organ Segmentation
Fuchen Zheng, Xinyi Chen, Weixuan Li, Quanjun Li, Junhua Zhou, Xiaojiao Guo, Xuhang Chen, Chi-Man Pun, Shoujun Zhou
Medical image segmentation is a cornerstone of modern clinical diagnostics. While Vision Transformers that leverage shifted window-based self-attention have established new benchmarks in this field, they are often hampered by a critical limitation: their localized attention mechanism struggles to effectively fuse local details with global context. This deficiency is particularly detrimental to challenging tasks such as the segmentation of microtumors and miniature organs, where both fine-grained boundary definition and broad contextual understanding are paramount. To address this gap, we propose HBFormer, a novel Hybrid-Bridge Transformer architecture. The 'Hybrid' design of HBFormer synergizes a classic U-shaped encoder-decoder framework with a powerful Swin Transformer backbone for robust hierarchical feature extraction. The core innovation lies in its 'Bridge' mechanism, a sophisticated nexus for multi-scale feature integration. This bridge is architecturally embodied by our novel Multi-Scale Feature Fusion (MFF) decoder. Departing from conventional symmetric designs, the MFF decoder is engineered to fuse multi-scale features from the encoder with global contextual information. It achieves this through a synergistic combination of channel and spatial attention modules, which are constructed from a series of dilated and depth-wise convolutions. These components work in concert to create a powerful feature bridge that explicitly captures long-range dependencies and refines object boundaries with exceptional precision. Comprehensive experiments on challenging medical image segmentation datasets, including multi-organ, liver tumor, and bladder tumor benchmarks, demonstrate that HBFormer achieves state-of-the-art results, showcasing its outstanding capabilities in microtumor and miniature organ segmentation. Code and models are available at: https://github.com/lzeeorno/HBFormer.
comment : 6 pages, 4 figures, 3 tables
☆ CloseUpAvatar: High-Fidelity Animatable Full-Body Avatars with Mixture of Multi-Scale Textures
David Svitov, Pietro Morerio, Lourdes Agapito, Alessio Del Bue
We present a CloseUpAvatar - a novel approach for articulated human avatar representation dealing with more general camera motions, while preserving rendering quality for close-up views. CloseUpAvatar represents an avatar as a set of textured planes with two sets of learnable textures for low and high-frequency detail. The method automatically switches to high-frequency textures only for cameras positioned close to the avatar's surface and gradually reduces their impact as the camera moves farther away. Such parametrization of the avatar enables CloseUpAvatar to adjust rendering quality based on camera distance ensuring realistic rendering across a wider range of camera orientations than previous approaches. We provide experiments using the ActorsHQ dataset with high-resolution input images. CloseUpAvatar demonstrates both qualitative and quantitative improvements over existing methods in rendering from novel wide range camera positions, while maintaining high FPS by limiting the number of required primitives.
☆ Harnessing Hypergraphs in Geometric Deep Learning for 3D RNA Inverse Folding
Guang Yang, Lei Fan
The RNA inverse folding problem, a key challenge in RNA design, involves identifying nucleotide sequences that can fold into desired secondary structures, which are critical for ensuring molecular stability and function. The inherent complexity of this task stems from the intricate relationship between sequence and structure, making it particularly challenging. In this paper, we propose a framework, named HyperRNA, a generative model with an encoder-decoder architecture that leverages hypergraphs to design RNA sequences. Specifically, our HyperRNA model consists of three main components: preprocessing, encoding and decoding. In the preprocessing stage, graph structures are constructed by extracting the atom coordinates of RNA backbone based on 3-bead coarse-grained representation. The encoding stage processes these graphs, capturing higher order dependencies and complex biomolecular interactions using an attention embedding module and a hypergraph-based encoder. Finally, the decoding stage generates the RNA sequence in an autoregressive manner. We conducted quantitative and qualitative experiments on the PDBBind and RNAsolo datasets to evaluate the inverse folding task for RNA sequence generation and RNA-protein complex sequence generation. The experimental results demonstrate that HyperRNA not only outperforms existing RNA design methods but also highlights the potential of leveraging hypergraphs in RNA engineering.
☆ Beyond Boundary Frames: Audio-Visual Semantic Guidance for Context -Aware Video Interpolation
Yuchen Deng, Xiuyang Wu, Hai-Tao Zheng, Jie Wang, Feidiao Yang, Yuxing Han
Handling fast, complex, and highly non-linear motion patterns has long posed challenges for video frame interpolation. Although recent diffusion-based approaches improve upon traditional optical-flow-based methods, they still struggle to cover diverse application scenarios and often fail to produce sharp, temporally consistent frames in fine-grained motion tasks such as audio-visual synchronized interpolation. To address these limitations, we introduce BBF (Beyond Boundary Frames), a context-aware video frame interpolation framework, which could be guided by audio/visual semantics. First, we enhance the input design of the interpolation model so that it can flexibly handle multiple conditional modalities, including text, audio, images, and video. Second, we propose a decoupled multimodal fusion mechanism that sequentially injects different conditional signals into a DiT backbone. Finally, to maintain the generation abilities of the foundation model, we adopt a progressive multi-stage training paradigm, where the start-end frame difference embedding is used to dynamically adjust both the data sampling and the loss weighting. Extensive experimental results demonstrate that BBF outperforms specialized state-of-the-art methods on both generic interpolation and audio-visual synchronized interpolation tasks, establishing a unified framework for video frame interpolation under coordinated multi-channel conditioning.
☆ Dynamic Optical Test for Bot Identification (DOT-BI): A simple check to identify bots in survey s and online processes
Malte Bleeker, Mauro Gotsch
We propose the Dynamic Optical Test for Bot Identification (DOT-BI): a quick and easy method that uses human perception of motion to differentiate between human respondents and automated systems in surveys and online processes. In DOT-BI, a 'hidden' number is displayed with the same random black-and-white pixel texture as its background. Only the difference in motion and scale between the number and the background makes the number perceptible to humans across frames, while frame-by-frame algorithmic processing yields no meaningful signal. We conducted two preliminary assessments. Firstly, state-of-the-art, video-capable, multimodal models (GPT-5-Thinking and Gemini 2.5 Pro) fail to extract the correct value, even when given explicit instructions about the mechanism. Secondly, in an online survey (n=182), 99.5% (181/182) of participants solved the task, with an average end-to-end completion time of 10.7 seconds; a supervised lab study (n=39) found no negative effects on perceived ease-of-use or completion time relative to a control. We release code to generate tests and 100+ pre-rendered variants to facilitate adoption in surveys and online processes.
☆ Cross-Stain Contrastive Learning for Paired Immunohistochemistry and Histopathology Slide Representation Learning
Yizhi Zhang, Lei Fan, Zhulin Tao, Donglin Di, Yang Song, Sidong Liu, Cong Cong
Universal, transferable whole-slide image (WSI) representations are central to computational pathology. Incorporating multiple markers (e.g., immunohistochemistry, IHC) alongside H&E enriches H&E-based features with diverse, biologically meaningful information. However, progress is limited by the scarcity of well-aligned multi-stain datasets. Inter-stain misalignment shifts corresponding tissue across slides, hindering consistent patch-level features and degrading slide-level embeddings. To address this, we curated a slide-level aligned, five-stain dataset (H&E, HER2, KI67, ER, PGR) to enable paired H&E-IHC learning and robust cross-stain representation. Leveraging this dataset, we propose Cross-Stain Contrastive Learning (CSCL), a two-stage pretraining framework with a lightweight adapter trained using patch-wise contrastive alignment to improve the compatibility of H&E features with corresponding IHC-derived contextual cues, and slide-level representation learning with Multiple Instance Learning (MIL), which uses a cross-stain attention fusion module to integrate stain-specific patch features and a cross-stain global alignment module to enforce consistency among slide-level embeddings across different stains. Experiments on cancer subtype classification, IHC biomarker status classification, and survival prediction show consistent gains, yielding high-quality, transferable H&E slide-level representations. The code and data are available at https://github.com/lily-zyz/CSCL.
comment : 6 pages, 2 figures. Camera-ready version accepted for IEEE BIBM 2025
☆ UniComp: Rethinking Video Compression Through Information al Uniqueness
Chao Yuan, Shimin Chen, Minliang Lin, Limeng Qiao, Guanglu Wan, Lin Ma
Distinct from attention-based compression methods, this paper presents an information uniqueness driven video compression framework, termed UniComp, which aims to maximize the information fidelity of video representations under constrained computational budgets. Starting from the information-theoretic perspective, we formulate the vision compression as an optimization problem that minimizes conditional entropy (reconstruction error) between retained and full tokens. To achieve this, we introduce the notion of information uniqueness to measure intrinsic redundancy among tokens to link with reconstruction error. Based on uniqueness, we design three modules-Frame Group Fusion, Token Allocation, and Spatial Dynamic Compression-that progressively perform semantic frame grouping, adaptive resource allocation, and fine-grained spatial compression. Extensive experiments demonstrate that UniComp consistently outperforms existing compression methods in preserving essential visual tokens under limited computational budgets, highlighting the pivotal role of information uniqueness in token compression efficacy.
☆ Global-Local Aware Scene Text Editing
Fuxiang Yang, Tonghua Su, Donglin Di, Yin Chen, Xiangqian Wu, Zhongjie Wang, Lei Fan
Scene Text Editing (STE) involves replacing text in a scene image with new target text while preserving both the original text style and background texture. Existing methods suffer from two major challenges: inconsistency and length-insensitivity. They often fail to maintain coherence between the edited local patch and the surrounding area, and they struggle to handle significant differences in text length before and after editing. To tackle these challenges, we propose an end-to-end framework called Global-Local Aware Scene Text Editing (GLASTE), which simultaneously incorporates high-level global contextual information along with delicate local features. Specifically, we design a global-local combination structure, joint global and local losses, and enhance text image features to ensure consistency in text style within local patches while maintaining harmony between local and global areas. Additionally, we express the text style as a vector independent of the image size, which can be transferred to target text images of various sizes. We use an affine fusion to fill target text images into the editing patch while maintaining their aspect ratio unchanged. Extensive experiments on real-world datasets validate that our GLASTE model outperforms previous methods in both quantitative metrics and qualitative results and effectively mitigates the two challenges.
☆ GAOT: Generating Articulated Objects Through Text-Guided Diffusion Models ACM MM
Hao Sun, Lei Fan, Donglin Di, Shaohui Liu
Articulated object generation has seen increasing advancements, yet existing models often lack the ability to be conditioned on text prompts. To address the significant gap between textual descriptions and 3D articulated object representations, we propose GAOT, a three-phase framework that generates articulated objects from text prompts, leveraging diffusion models and hypergraph learning in a three-step process. First, we fine-tune a point cloud generation model to produce a coarse representation of objects from text prompts. Given the inherent connection between articulated objects and graph structures, we design a hypergraph-based learning method to refine these coarse representations, representing object parts as graph vertices. Finally, leveraging a diffusion model, the joints of articulated objects-represented as graph edges-are generated based on the object parts. Extensive qualitative and quantitative experiments on the PartNet-Mobility dataset demonstrate the effectiveness of our approach, achieving superior performance over previous methods.
comment : Accepted by ACM MM Asia2026
☆ CartoMapQA: A Fundamental Benchmark Dataset Evaluating Vision -Language Models on Cartographic Map Understanding SP
Huy Quang Ung, Guillaume Habault, Yasutaka Nishimura, Hao Niu, Roberto Legaspi, Tomoki Oya, Ryoichi Kojima, Masato Taya, Chihiro Ono, Atsunori Minamikawa, Yan Liu
The rise of Visual-Language Models (LVLMs) has unlocked new possibilities for seamlessly integrating visual and textual information. However, their ability to interpret cartographic maps remains largely unexplored. In this paper, we introduce CartoMapQA, a benchmark specifically designed to evaluate LVLMs' understanding of cartographic maps through question-answering tasks. The dataset includes over 2000 samples, each composed of a cartographic map, a question (with open-ended or multiple-choice answers), and a ground-truth answer. These tasks span key low-, mid- and high-level map interpretation skills, including symbol recognition, embedded information extraction, scale interpretation, and route-based reasoning. Our evaluation of both open-source and proprietary LVLMs reveals persistent challenges: models frequently struggle with map-specific semantics, exhibit limited geospatial reasoning, and are prone to Optical Character Recognition (OCR)-related errors. By isolating these weaknesses, CartoMapQA offers a valuable tool for guiding future improvements in LVLM architectures. Ultimately, it supports the development of models better equipped for real-world applications that depend on robust and reliable map understanding, such as navigation, geographic search, and urban planning. Our source code and data are openly available to the research community at: https://github.com/ungquanghuy-kddi/CartoMapQA.git
comment : Accepted at SIGSPATIAL 2025 (Best paper candidates), 15 pages
☆ RoboScape-R: Unified Reward-Observation Wo rl d Models for Generalizable Robotics Training via RL
Yinzhou Tang, Yu Shang, Yinuo Chen, Bingwen Wei, Xin Zhang, Shu'ang Yu, Liangzhi Shi, Chao Yu, Chen Gao, Wei Wu, Yong Li
Achieving generalizable embodied policies remains a key challenge. Traditional policy learning paradigms, including both Imitation Learning (IL) and Reinforcement Learning (RL), struggle to cultivate generalizability across diverse scenarios. While IL policies often overfit to specific expert trajectories, RL suffers from the inherent lack of a unified and general reward signal necessary for effective multi-scene generalization. We posit that the world model is uniquely capable of serving as a universal environment proxy to address this limitation. However, current world models primarily focus on their ability to predict observations and still rely on task-specific, handcrafted reward functions, thereby failing to provide a truly general training environment. Toward this problem, we propose RoboScape-R, a framework leveraging the world model to serve as a versatile, general-purpose proxy for the embodied environment within the RL paradigm. We introduce a novel world model-based general reward mechanism that generates ''endogenous'' rewards derived from the model's intrinsic understanding of real-world state transition dynamics. Extensive experiments demonstrate that RoboScape-R effectively addresses the limitations of traditional RL methods by providing an efficient and general training environment that substantially enhances the generalization capability of embodied policies. Our approach offers critical insights into utilizing the world model as an online training strategy and achieves an average 37.5% performance improvement over baselines under out-of-domain scenarios.
☆ Dynamic Content Moderation in Livestreams: Combining Supervised Classification with MLLM-Boosted Similarity Matching KDD 2026
Wei Chee Yew, Hailun Xu, Sanjay Saha, Xiaotian Fan, Hiok Hian Ong, David Yuchen Wang, Kanchan Sarkar, Zhenheng Yang, Danhui Guan
Content moderation remains a critical yet challenging task for large-scale user-generated video platforms, especially in livestreaming environments where moderation must be timely, multimodal, and robust to evolving forms of unwanted content. We present a hybrid moderation framework deployed at production scale that combines supervised classification for known violations with reference-based similarity matching for novel or subtle cases. This hybrid design enables robust detection of both explicit violations and novel edge cases that evade traditional classifiers. Multimodal inputs (text, audio, visual) are processed through both pipelines, with a multimodal large language model (MLLM) distilling knowledge into each to boost accuracy while keeping inference lightweight. In production, the classification pipeline achieves 67% recall at 80% precision, and the similarity pipeline achieves 76% recall at 80% precision. Large-scale A/B tests show a 6-8% reduction in user views of unwanted livestreams}. These results demonstrate a scalable and adaptable approach to multimodal content governance, capable of addressing both explicit violations and emerging adversarial behaviors.
comment : Accepted at KDD 2026
☆ V-ITI: Mitigating Hallucinations in Multimodal Large Language Models via Visual Inference-Time Intervention
Nan Sun, Zhenyu Zhang, Xixun Lin, Kun Wang, Yanmin Shang, Naibin Gu, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang, Yanan Cao
Multimodal Large Language Models (MLLMs) excel in numerous vision-language tasks yet suffer from hallucinations, producing content inconsistent with input visuals, that undermine reliability in precision-sensitive domains. This issue stems from a fundamental problem of visual neglect, where models fail to adequately prioritize input images. Existing methods typically alleviate hallucinations by intervening in the attention score or output logits, focusing on "how to intervene" but overlooking the prerequisite "when to intervene", which leads to the "over-intervention" problem and subsequently introduces new hallucinations and unnecessary computational overhead. To address this gap, we first investigate the mechanism of visual neglect and reveal it can be accurately detected via head-level activation patterns in MLLMs. We thus propose V-ITI, a lightweight visual inference-time intervention framework integrating a Visual Neglect Detector that identifies visual neglect via head-level discriminative probes and a Visual Recall Intervenor that modulates activations with prestored visual activation information only when the visual neglect is detected. Extensive experiments across eight benchmarks and different MLLM families demonstrate that V-ITI consistently mitigates vision-related hallucinations while preserving general task performance.
☆ CookAnything: A Framework for Flexible and Consistent Multi-Step Recipe Image Generation
Ruoxuan Zhang, Bin Wen, Hongxia Xie, Yi Yao, Songhan Zuo, Jian-Yu Jiang-Lin, Hong-Han Shuai, Wen-Huang Cheng
Cooking is a sequential and visually grounded activity, where each step such as chopping, mixing, or frying carries both procedural logic and visual semantics. While recent diffusion models have shown strong capabilities in text-to-image generation, they struggle to handle structured multi-step scenarios like recipe illustration. Additionally, current recipe illustration methods are unable to adjust to the natural variability in recipe length, generating a fixed number of images regardless of the actual instructions structure. To address these limitations, we present CookAnything, a flexible and consistent diffusion-based framework that generates coherent, semantically distinct image sequences from textual cooking instructions of arbitrary length. The framework introduces three key components: (1) Step-wise Regional Control (SRC), which aligns textual steps with corresponding image regions within a single denoising process; (2) Flexible RoPE, a step-aware positional encoding mechanism that enhances both temporal coherence and spatial diversity; and (3) Cross-Step Consistency Control (CSCC), which maintains fine-grained ingredient consistency across steps. Experimental results on recipe illustration benchmarks show that CookAnything performs better than existing methods in training-based and training-free settings. The proposed framework supports scalable, high-quality visual synthesis of complex multi-step instructions and holds significant potential for broad applications in instructional media, and procedural content creation.
comment : Accepted by ACM Multimedia 2025
☆ Rethinking Prompt Design for Inference-time Scaling in Text-to-Visual Generation
Subin Kim, Sangwoo Mo, Mamshad Nayeem Rizve, Yiran Xu, Difan Liu, Jinwoo Shin, Tobias Hinz
Achieving precise alignment between user intent and generated visuals remains a central challenge in text-to-visual generation, as a single attempt often fails to produce the desired output. To handle this, prior approaches mainly scale the visual generation process (e.g., increasing sampling steps or seeds), but this quickly leads to a quality plateau. This limitation arises because the prompt, crucial for guiding generation, is kept fixed. To address this, we propose Prompt Redesign for Inference-time Scaling, coined PRIS, a framework that adaptively revises the prompt during inference in response to the scaled visual generations. The core idea of PRIS is to review the generated visuals, identify recurring failure patterns across visuals, and redesign the prompt accordingly before regenerating the visuals with the revised prompt. To provide precise alignment feedback for prompt revision, we introduce a new verifier, element-level factual correction, which evaluates the alignment between prompt attributes and generated visuals at a fine-grained level, achieving more accurate and interpretable assessments than holistic measures. Extensive experiments on both text-to-image and text-to-video benchmarks demonstrate the effectiveness of our approach, including a 15% gain on VBench 2.0. These results highlight that jointly scaling prompts and visuals is key to fully leveraging scaling laws at inference-time. Visualizations are available at the website: https://subin-kim-cv.github.io/PRIS.
comment : Visualizations are available at the website: https://subin-kim-cv.github.io/PRIS
☆ OpenTrack3D: Towards Accurate and Generalizable Open-Vocabulary 3D Instance Segmentation
Zhishan Zhou, Siyuan Wei, Zengran Wang, Chunjie Wang, Xiaosheng Yan, Xiao Liu
Generalizing open-vocabulary 3D instance segmentation (OV-3DIS) to diverse, unstructured, and mesh-free environments is crucial for robotics and AR/VR, yet remains a significant challenge. We attribute this to two key limitations of existing methods: (1) proposal generation relies on dataset-specific proposal networks or mesh-based superpoints, rendering them inapplicable in mesh-free scenarios and limiting generalization to novel scenes; and (2) the weak textual reasoning of CLIP-based classifiers, which struggle to recognize compositional and functional user queries. To address these issues, we introduce OpenTrack3D, a generalizable and accurate framework. Unlike methods that rely on pre-generated proposals, OpenTrack3D employs a novel visual-spatial tracker to construct cross-view consistent object proposals online. Given an RGB-D stream, our pipeline first leverages a 2D open-vocabulary segmenter to generate masks, which are lifted to 3D point clouds using depth. Mask-guided instance features are then extracted using DINO feature maps, and our tracker fuses visual and spatial cues to maintain instance consistency. The core pipeline is entirely mesh-free, yet we also provide an optional superpoints refinement module to further enhance performance when scene mesh is available. Finally, we replace CLIP with a multi-modal large language model (MLLM), significantly enhancing compositional reasoning for complex user queries. Extensive experiments on diverse benchmarks, including ScanNet200, Replica, ScanNet++, and SceneFun3D, demonstrate state-of-the-art performance and strong generalization capabilities.
☆ MSG-Loc: Multi-Label Likelihood-based Semantic Graph Matching for Object-Level Global Localization
Gihyeon Lee, Jungwoo Lee, Juwon Kim, Young-Sik Shin, Younggun Cho
Robots are often required to localize in environments with unknown object classes and semantic ambiguity. However, when performing global localization using semantic objects, high semantic ambiguity intensifies object misclassification and increases the likelihood of incorrect associations, which in turn can cause significant errors in the estimated pose. Thus, in this letter, we propose a multi-label likelihood-based semantic graph matching framework for object-level global localization. The key idea is to exploit multi-label graph representations, rather than single-label alternatives, to capture and leverage the inherent semantic context of object observations. Based on these representations, our approach enhances semantic correspondence across graphs by combining the likelihood of each node with the maximum likelihood of its neighbors via context-aware likelihood propagation. For rigorous validation, data association and pose estimation performance are evaluated under both closed-set and open-set detection configurations. In addition, we demonstrate the scalability of our approach to large-vocabulary object categories in both real-world indoor scenes and synthetic environments.
comment : Accepted in IEEE Robotics and Automation Letters (2025)
☆ Flood Diffusion : Tailored Diffusion Forcing for Streaming Motion Generation
Yiyi Cai, Yuhan Wu, Kunhang Li, You Zhou, Bo Zheng, Haiyang Liu
We present FloodDiffusion, a new framework for text-driven, streaming human motion generation. Given time-varying text prompts, FloodDiffusion generates text-aligned, seamless motion sequences with real-time latency. Unlike existing methods that rely on chunk-by-chunk or auto-regressive model with diffusion head, we adopt a diffusion forcing framework to model this time-series generation task under time-varying control events. We find that a straightforward implementation of vanilla diffusion forcing (as proposed for video models) fails to model real motion distributions. We demonstrate that to guarantee modeling the output distribution, the vanilla diffusion forcing must be tailored to: (i) train with a bi-directional attention instead of casual attention; (ii) implement a lower triangular time scheduler instead of a random one; (iii) utilize a continues time-varying way to introduce text conditioning. With these improvements, we demonstrate in the first time that the diffusion forcing-based framework achieves state-of-the-art performance on the streaming motion generation task, reaching an FID of 0.057 on the HumanML3D benchmark. Models, code, and weights are available. https://shandaai.github.io/FloodDiffusion/
comment : 15 pages, 7 figures
☆ M3DR: Towards Universal Multilingual Multimodal Document Retrieval
Adithya S Kolavi, Vyoman Jain
Multimodal document retrieval systems have shown strong progress in aligning visual and textual content for semantic search. However, most existing approaches remain heavily English-centric, limiting their effectiveness in multilingual contexts. In this work, we present M3DR (Multilingual Multimodal Document Retrieval), a framework designed to bridge this gap across languages, enabling applicability across diverse linguistic and cultural contexts. M3DR leverages synthetic multilingual document data and generalizes across different vision-language architectures and model sizes, enabling robust cross-lingual and cross-modal alignment. Using contrastive training, our models learn unified representations for text and document images that transfer effectively across languages. We validate this capability on 22 typologically diverse languages, demonstrating consistent performance and adaptability across linguistic and script variations. We further introduce a comprehensive benchmark that captures real-world multilingual scenarios, evaluating models under monolingual, multilingual, and mixed-language settings. M3DR generalizes across both single dense vector and ColBERT-style token-level multi-vector retrieval paradigms. Our models, NetraEmbed and ColNetraEmbed achieve state-of-the-art performance with ~150% relative improvements on cross-lingual retrieval.
☆ CSMapping: Scalable Crowdsourced Semantic Mapping and Topology Inference for Autonomous Driving
Zhijian Qiao, Zehuan Yu, Tong Li, Chih-Chung Chou, Wenchao Ding, Shaojie Shen
Crowdsourcing enables scalable autonomous driving map construction, but low-cost sensor noise hinders quality from improving with data volume. We propose CSMapping, a system that produces accurate semantic maps and topological road centerlines whose quality consistently increases with more crowdsourced data. For semantic mapping, we train a latent diffusion model on HD maps (optionally conditioned on SD maps) to learn a generative prior of real-world map structure, without requiring paired crowdsourced/HD-map supervision. This prior is incorporated via constrained MAP optimization in latent space, ensuring robustness to severe noise and plausible completion in unobserved areas. Initialization uses a robust vectorized mapping module followed by diffusion inversion; optimization employs efficient Gaussian-basis reparameterization, projected gradient descent zobracket multi-start, and latent-space factor-graph for global consistency. For topological mapping, we apply confidence-weighted k-medoids clustering and kinematic refinement to trajectories, yielding smooth, human-like centerlines robust to trajectory variation. Experiments on nuScenes, Argoverse 2, and a large proprietary dataset achieve state-of-the-art semantic and topological mapping performance, with thorough ablation and scalability studies.
☆ AfroBeats Dance Movement Analysis Using Computer Vision : A Proof-of-Concept Framework Combining YOLO and Segment Anything Model
Kwaku Opoku-Ware, Gideon Opoku
This paper presents a preliminary investigation into automated dance movement analysis using contemporary computer vision techniques. We propose a proof-of-concept framework that integrates YOLOv8 and v11 for dancer detection with the Segment Anything Model (SAM) for precise segmentation, enabling the tracking and quantification of dancer movements in video recordings without specialized equipment or markers. Our approach identifies dancers within video frames, counts discrete dance steps, calculates spatial coverage patterns, and measures rhythm consistency across performance sequences. Testing this framework on a single 49-second recording of Ghanaian AfroBeats dance demonstrates technical feasibility, with the system achieving approximately 94% detection precision and 89% recall on manually inspected samples. The pixel-level segmentation provided by SAM, achieving approximately 83% intersection-over-union with visual inspection, enables motion quantification that captures body configuration changes beyond what bounding-box approaches can represent. Analysis of this preliminary case study indicates that the dancer classified as primary by our system executed 23% more steps with 37% higher motion intensity and utilized 42% more performance space compared to dancers classified as secondary. However, this work represents an early-stage investigation with substantial limitations including single-video validation, absence of systematic ground truth annotations, and lack of comparison with existing pose estimation methods. We present this framework to demonstrate technical feasibility, identify promising directions for quantitative dance metrics, and establish a foundation for future systematic validation studies.
☆ Exploiting Domain Properties in Language-Driven Domain Generalization for Semantic Segmentation ICCV 2025
Seogkyu Jeon, Kibeom Hong, Hyeran Byun
Recent domain generalized semantic segmentation (DGSS) studies have achieved notable improvements by distilling semantic knowledge from Vision-Language Models (VLMs). However, they overlook the semantic misalignment between visual and textual contexts, which arises due to the rigidity of a fixed context prompt learned on a single source domain. To this end, we present a novel domain generalization framework for semantic segmentation, namely Domain-aware Prompt-driven Masked Transformer (DPMFormer). Firstly, we introduce domain-aware prompt learning to facilitate semantic alignment between visual and textual cues. To capture various domain-specific properties with a single source dataset, we propose domain-aware contrastive learning along with the texture perturbation that diversifies the observable domains. Lastly, to establish a framework resilient against diverse environmental changes, we have proposed the domain-robust consistency learning which guides the model to minimize discrepancies of prediction from original and the augmented images. Through experiments and analyses, we demonstrate the superiority of the proposed framework, which establishes a new state-of-the-art on various DGSS benchmarks. The code is available at https://github.com/jone1222/DPMFormer.
comment : ICCV 2025 (poster)
☆ EEA: Exploration-Exploitation Agent for Long Video Understanding
Te Yang, Xiangyu Zhu, Bo Wang, Quan Chen, Peng Jiang, Zhen Lei
Long-form video understanding requires efficient navigation of extensive visual data to pinpoint sparse yet critical information. Current approaches to longform video understanding either suffer from severe computational overhead due to dense preprocessing, or fail to effectively balance exploration and exploitation, resulting in incomplete information coverage and inefficiency. In this work, we introduce EEA, a novel video agent framework that archives exploration-exploitation balance through semantic guidance with hierarchical tree search process. EEA autonomously discovers and dynamically updates task-relevant semantic queries, and collects video frames closely matched to these queries as semantic anchors. During the tree search process, instead of uniform expansion, EEA preferentially explores semantically relevant frames while ensuring sufficient coverage within unknown segments. Moreover, EEA adaptively combines intrinsic rewards from visionlanguage models (VLMs) with semantic priors by explicitly modeling uncertainty to achieve stable and precise evaluation of video segments. Experiments across various long-video benchmarks validate the superior performance and computational efficiency of our proposed method.
☆ NAS-LoRA: Empowering Parameter-Efficient Fine-Tuning for Visual Foundation Models with Searchable Adaptation
Renqi Chen, Haoyang Su, Shixiang Tang
The Segment Anything Model (SAM) has emerged as a powerful visual foundation model for image segmentation. However, adapting SAM to specific downstream tasks, such as medical and agricultural imaging, remains a significant challenge. To address this, Low-Rank Adaptation (LoRA) and its variants have been widely employed to enhancing SAM's adaptation performance on diverse domains. Despite advancements, a critical question arises: can we integrate inductive bias into the model? This is particularly relevant since the Transformer encoder in SAM inherently lacks spatial priors within image patches, potentially hindering the acquisition of high-level semantic information. In this paper, we propose NAS-LoRA, a new Parameter-Efficient Fine-Tuning (PEFT) method designed to bridge the semantic gap between pre-trained SAM and specialized domains. Specifically, NAS-LoRA incorporates a lightweight Neural Architecture Search (NAS) block between the encoder and decoder components of LoRA to dynamically optimize the prior knowledge integrated into weight updates. Furthermore, we propose a stage-wise optimization strategy to help the ViT encoder balance weight updates and architectural adjustments, facilitating the gradual learning of high-level semantic information. Various Experiments demonstrate our NAS-LoRA improves existing PEFT methods, while reducing training cost by 24.14% without increasing inference cost, highlighting the potential of NAS in enhancing PEFT for visual foundation models.
☆ Towards Object-centric Understanding for Instructional Videos
Wenliang Guo, Yu Kong
Understanding procedural activities is crucial for developing future assistive AI that can reason about complex real-world tasks. Existing action-centric methods struggle with the flexibility of real procedures, where step order varies depending on object states. In this work, we propose to shift the focus to an object-centric paradigm by regarding actions as mechanisms that drive state transitions. To advance this direction, we introduce Object-IVQA, a long-form instructional video benchmark with 107 videos and 514 open-ended question-answer pairs annotated with temporally grounded evidence. The benchmark evaluates four dimensions of object-centric reasoning, including state evolution, precondition verification, counterfactual reasoning and mistake recognition. We further propose an agent framework that orchestrates object-centric planning, perception, analysis and generation tools, enabling explicit evidence retrieval and multi-hop reasoning across disjoint segments. Experiments show that existing large vision-language models struggle in object-level recognition and reasoning, whereas our framework achieves substantially improvement.
☆ Fairness-Aware Fine-Tuning of Vision -Language Models for Medical Glaucoma Diagnosis
Zijian Gu, Yuxi Liu, Zhenhao Zhang, Song Wang
Vision-language models achieve expert-level performance on medical imaging tasks but exhibit significant diagnostic accuracy disparities across demographic groups. We introduce fairness-aware Low-Rank Adaptation for medical VLMs, combining parameter efficiency with explicit fairness optimization. Our key algorithmic contribution is a differentiable MaxAccGap loss that enables end-to-end optimization of accuracy parity across demographic groups. We propose three methods: FR-LoRA integrates MaxAccGap regularization into the training objective, GR-LoRA applies inverse frequency weighting to balance gradient contributions, and Hybrid-LoRA combines both mechanisms.Evaluated on 10,000 glaucoma fundus images, GR-LoRA reduces diagnostic accuracy disparities by 69% while maintaining 53.15% overall accuracy. Ablation studies reveal that strong regularization strength achieves optimal fairness with minimal accuracy trade-off, and race-specific optimization yields 60% disparity reduction. Our approach requires only 0.24% trainable parameters, enabling practical deployment of fair medical AI in resource-constrained healthcare settings.
comment : 10 pages, 3 tables
☆ Procedural Mistake Detection via Action Effect Modeling
Wenliang Guo, Yujiang Pu, Yu Kong
Mistake detection in procedural tasks is essential for building intelligent systems that support learning and task execution. Existing approaches primarily analyze how an action is performed, while overlooking what it produces, i.e., the \textbf{action effect}. Yet many errors manifest not in the execution itself but in the resulting outcome, such as an unintended object state or incorrect spatial arrangement. To address this gap, we propose Action Effect Modeling (AEM), a unified framework that jointly captures action execution and its outcomes through a probabilistic formulation. AEM first identifies the outcome of an action by selecting the most informative effect frame based on semantic relevance and visual quality. It then extracts complementary cues from visual grounding and symbolic scene graphs, aligning them in a shared latent space to form robust effect-aware representations. To detect mistakes, we further design a prompt-based detector that incorporates task-specific prompts and aligns each action segment with its intended execution semantics. Our approach achieves state-of-the-art performance on the EgoPER and CaptainCook4D benchmarks under the challenging one-class classification (OCC) setting. These results demonstrate that modeling both execution and outcome yields more reliable mistake detection, and highlight the potential of effect-aware representations to benefit a broader range of downstream applications.
☆ Difference Decomposition Networks for Infrared Small Target Detection
Chen Hu, Mingyu Zhou, Shuai Yuan, Hongbo Hu, Xiangyu Qiu, Junhai Luo, Tian Pu, Xiyin Li
Infrared small target detection (ISTD) faces two major challenges: a lack of discernible target texture and severe background clutter, which results in the background obscuring the target. To enhance targets and suppress backgrounds, we propose the Basis Decomposition Module (BDM) as an extensible and lightweight module based on basis decomposition, which decomposes a complex feature into several basis features and enhances certain information while eliminating redundancy. Extending BDM leads to a series of modules, including the Spatial Difference Decomposition Module (SD$^\mathrm{2}$M), Spatial Difference Decomposition Downsampling Module (SD$^\mathrm{3}$M), and Temporal Difference Decomposition Module (TD$^\mathrm{2}$M). Based on these modules, we develop the Spatial Difference Decomposition Network (SD$^\mathrm{2}$Net) for single-frame ISTD (SISTD) and the Spatiotemporal Difference Decomposition Network (STD$^\mathrm{2}$Net) for multi-frame ISTD (MISTD). SD$^\mathrm{2}$Net integrates SD$^\mathrm{2}$M and SD$^\mathrm{3}$M within an adapted U-shaped architecture. We employ TD$^\mathrm{2}$M to introduce motion information, which transforms SD$^\mathrm{2}$Net into STD$^\mathrm{2}$Net. Extensive experiments on SISTD and MISTD datasets demonstrate state-of-the-art (SOTA) performance. On the SISTD task, SD$^\mathrm{2}$Net performs well compared to most established networks. On the MISTD datasets, STD$^\mathrm{2}$Net achieves a mIoU of 87.68\%, outperforming SD$^\mathrm{2}$Net, which achieves a mIoU of 64.97\%. Our codes are available: https://github.com/greekinRoma/IRSTD_HC_Platform.
☆ Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision -Language Models
Shojiro Yamabe, Futa Waseda, Daiki Shiono, Tsubasa Takahashi
Recent large vision-language models (LVLMs) have been applied to diverse VQA tasks. However, achieving practical performance typically requires task-specific fine-tuning with large numbers of image-text pairs, which are costly to collect. In this work, we study text-centric training, a setting where only textual descriptions are available and no real images are provided, as a paradigm for low-cost data scaling. Unlike images, whose collection is often restricted by privacy constraints and scarcity in niche domains, text is widely available. Moreover, text is easily editable, enabling automatic diversification and expansion with LLMs at minimal human effort. While this offers clear advantages over image collection in terms of scalability and cost, training on raw text without images still yields limited gains on VQA tasks because of the image-text modality gap. To address this issue, we propose a Text-Printed Image (TPI), which generates synthetic images by directly rendering the given textual description on a plain white canvas. This simple rendering projects text into the image modality and can be integrated into arbitrary existing LVLM training pipelines at low cost. Moreover, TPI preserves the semantics of the text, whereas text-to-image models often fail to do. Across four models and seven benchmarks, our systematic experiments show that TPI enables more effective text-centric training than synthetic images generated by a diffusion model. We further explore TPI as a low-cost data-augmentation strategy and demonstrate its practical utility. Overall, our findings highlight the significant potential of text-centric training and, more broadly, chart a path toward fully automated data generation for LVLMs.
☆ Think Before You Drive: Wo rl d Model-Inspired Multimodal Grounding for Autonomous Vehicles
Haicheng Liao, Huanming Shen, Bonan Wang, Yongkang Li, Yihong Tang, Chengyue Wang, Dingyi Zhuang, Kehua Chen, Hai Yang, Chengzhong Xu, Zhenning Li
Interpreting natural-language commands to localize target objects is critical for autonomous driving (AD). Existing visual grounding (VG) methods for autonomous vehicles (AVs) typically struggle with ambiguous, context-dependent instructions, as they lack reasoning over 3D spatial relations and anticipated scene evolution. Grounded in the principles of world models, we propose ThinkDeeper, a framework that reasons about future spatial states before making grounding decisions. At its core is a Spatial-Aware World Model (SA-WM) that learns to reason ahead by distilling the current scene into a command-aware latent state and rolling out a sequence of future latent states, providing forward-looking cues for disambiguation. Complementing this, a hypergraph-guided decoder then hierarchically fuses these states with the multimodal input, capturing higher-order spatial dependencies for robust localization. In addition, we present DrivePilot, a multi-source VG dataset in AD, featuring semantic annotations generated by a Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT)-prompted LLM pipeline. Extensive evaluations on six benchmarks, ThinkDeeper ranks #1 on the Talk2Car leaderboard and surpasses state-of-the-art baselines on DrivePilot, MoCAD, and RefCOCO/+/g benchmarks. Notably, it shows strong robustness and efficiency in challenging scenes (long-text, multi-agent, ambiguity) and retains superior performance even when trained on 50% of the data.
☆ GeoVideo: Introducing Geometric Regularization into Video Generation Model
Yunpeng Bai, Shaoheng Fang, Chaohui Yu, Fan Wang, Qixing Huang
Recent advances in video generation have enabled the synthesis of high-quality and visually realistic clips using diffusion transformer models. However, most existing approaches operate purely in the 2D pixel space and lack explicit mechanisms for modeling 3D structures, often resulting in temporally inconsistent geometries, implausible motions, and structural artifacts. In this work, we introduce geometric regularization losses into video generation by augmenting latent diffusion models with per-frame depth prediction. We adopted depth as the geometric representation because of the great progress in depth prediction and its compatibility with image-based latent encoders. Specifically, to enforce structural consistency over time, we propose a multi-view geometric loss that aligns the predicted depth maps across frames within a shared 3D coordinate system. Our method bridges the gap between appearance generation and 3D structure modeling, leading to improved spatio-temporal coherence, shape consistency, and physical plausibility. Experiments across multiple datasets show that our approach produces significantly more stable and geometrically consistent results than existing baselines.
comment : Project Page: https://geovideo.github.io/GeoVideo/
☆ GalaxyDiT: Efficient Video Generation with Guidance Alignment and Adaptive Proxy in Diffusion Transformer s
Zhiye Song, Steve Dai, Ben Keller, Brucek Khailany
Diffusion models have revolutionized video generation, becoming essential tools in creative content generation and physical simulation. Transformer-based architectures (DiTs) and classifier-free guidance (CFG) are two cornerstones of this success, enabling strong prompt adherence and realistic video quality. Despite their versatility and superior performance, these models require intensive computation. Each video generation requires dozens of iterative steps, and CFG doubles the required compute. This inefficiency hinders broader adoption in downstream applications. We introduce GalaxyDiT, a training-free method to accelerate video generation with guidance alignment and systematic proxy selection for reuse metrics. Through rank-order correlation analysis, our technique identifies the optimal proxy for each video model, across model families and parameter scales, thereby ensuring optimal computational reuse. We achieve $1.87\times$ and $2.37\times$ speedup on Wan2.1-1.3B and Wan2.1-14B with only 0.97% and 0.72% drops on the VBench-2.0 benchmark. At high speedup rates, our approach maintains superior fidelity to the base model, exceeding prior state-of-the-art approaches by 5 to 10 dB in peak signal-to-noise ratio (PSNR).
☆ KeyPointDiffuser: Unsupervised 3D Keypoint Learning via Latent Diffusion Models
Rhys Newbury, Juyan Zhang, Tin Tran, Hanna Kurniawati, Dana Kulić
Understanding and representing the structure of 3D objects in an unsupervised manner remains a core challenge in computer vision and graphics. Most existing unsupervised keypoint methods are not designed for unconditional generative settings, restricting their use in modern 3D generative pipelines; our formulation explicitly bridges this gap. We present an unsupervised framework for learning spatially structured 3D keypoints from point cloud data. These keypoints serve as a compact and interpretable representation that conditions an Elucidated Diffusion Model (EDM) to reconstruct the full shape. The learned keypoints exhibit repeatable spatial structure across object instances and support smooth interpolation in keypoint space, indicating that they capture geometric variation. Our method achieves strong performance across diverse object categories, yielding a 6 percentage-point improvement in keypoint consistency compared to prior approaches.
☆ LM-CartSeg: Automated Segmentation of Lateral and Medial Cartilage and Subchondral Bone for Radiomics Analysis
Tongxu Zhang
Background and Objective: Radiomics of knee MRI requires robust, anatomically meaningful regions of interest (ROIs) that jointly capture cartilage and subchondral bone. Most existing work relies on manual ROIs and rarely reports quality control (QC). We present LM-CartSeg, a fully automatic pipeline for cartilage/bone segmentation, geometric lateral/medial (L/M) compartmentalisation and radiomics analysis. Methods: Two 3D nnU-Net models were trained on SKM-TEA (138 knees) and OAIZIB-CM (404 knees). At test time, zero-shot predictions were fused and refined by simple geometric rules: connected-component cleaning, construction of 10 mm subchondral bone bands in physical space, and a data-driven tibial L/M split based on PCA and k-means. Segmentation was evaluated on an OAIZIB-CM test set (103 knees) and on SKI-10 (100 knees). QC used volume and thickness signatures. From 10 ROIs we extracted 4 650 non-shape radiomic features to study inter-compartment similarity, dependence on ROI size, and OA vs. non-OA classification on OAIZIB-CM Results: Post-processing improved macro ASSD on OAIZIB-CM from 2.63 to 0.36 mm and HD95 from 25.2 to 3.35 mm, with DSC 0.91; zero-shot DSC on SKI-10 was 0.80. The geometric L/M rule produced stable compartments across datasets, whereas a direct L/M nnU-Net showed domain-dependent side swaps. Only 6 to 12 percent of features per ROI were strongly correlated with volume or thickness. Radiomics-based models models restricted to size-linked features. Conclusions: LM-CartSeg yields automatic, QCd ROIs and radiomic features that carry discriminative information beyond simple morphometry, providing a practical foundation for multi-centre knee OA radiomics studies.
☆ Multi-Aspect Knowledge-Enhanced Medical Vision -Language Pretrain ing with Multi- Agent Data Generation
Xieji Li, Siyuan Yan, Yingsheng Liu, H. Peter Soyer, Monika Janda, Victoria Mar, Zongyuan Ge
Vision-language pretraining (VLP) has emerged as a powerful paradigm in medical image analysis, enabling representation learning from large-scale image-text pairs without relying on expensive manual annotations. However, existing methods often struggle with the noise inherent in web-collected data and the complexity of unstructured long medical texts. To address these challenges, we propose a novel VLP framework integrating a Multi-Agent data GENeration (MAGEN) system and Ontology-based Multi-Aspect Knowledge-Enhanced (O-MAKE) pretraining. First, MAGEN enhances data quality by synthesizing knowledge-enriched descriptions via a foundation model-assisted captioning and retrieval-based verification pipeline. Second, O-MAKE addresses the difficulty of learning from long, unstructured texts by decomposing them into distinct knowledge aspects. This facilitates fine-grained alignment at both global and patch levels, while explicitly modeling medical concept relationships through ontology-guided mechanisms. We validate our framework in the field of dermatology, where comprehensive experiments demonstrate the effectiveness of each component. Our approach achieves state-of-the-art zero-shot performance on disease classification and cross-modal retrieval tasks across eight datasets. Our code and the augmented dataset Derm1M-AgentAug, comprising over 400k skin-image-text pairs, will be released at https://github.com/SiyuanYan1/Derm1M.
comment : 10 pages. Under Review
☆ Label-Efficient Hyperspectral Image Classification via Spectral FiLM Modulation of Low-Level Pretrain ed Diffusion Features ICML 2025
Yuzhen Hu, Biplab Banerjee, Saurabh Prasad
Hyperspectral imaging (HSI) enables detailed land cover classification, yet low spatial resolution and sparse annotations pose significant challenges. We present a label-efficient framework that leverages spatial features from a frozen diffusion model pretrained on natural images. Our approach extracts low-level representations from high-resolution decoder layers at early denoising timesteps, which transfer effectively to the low-texture structure of HSI. To integrate spectral and spatial information, we introduce a lightweight FiLM-based fusion module that adaptively modulates frozen spatial features using spectral cues, enabling robust multimodal learning under sparse supervision. Experiments on two recent hyperspectral datasets demonstrate that our method outperforms state-of-the-art approaches using only the provided sparse training labels. Ablation studies further highlight the benefits of diffusion-derived features and spectral-aware fusion. Overall, our results indicate that pretrained diffusion models can support domain-agnostic, label-efficient representation learning for remote sensing and broader scientific imaging tasks.
comment : Accepted to the ICML 2025 TerraBytes Workshop (June 9, 2025)
☆ Generalization Evaluation of Deep Stereo Matching Methods for UAV-Based Forestry Applications
Yida Lin, Bing Xue, Mengjie Zhang, Sam Schofield, Richard Green
Autonomous UAV forestry operations require robust depth estimation methods with strong cross-domain generalization. However, existing evaluations focus on urban and indoor scenarios, leaving a critical gap for specialized vegetation-dense environments. We present the first systematic zero-shot evaluation of eight state-of-the-art stereo methods--RAFT-Stereo, IGEV, IGEV++, BridgeDepth, StereoAnywhere, DEFOM (plus baseline methods ACVNet, PSMNet, TCstereo)--spanning iterative refinement, foundation model, and zero-shot adaptation paradigms. All methods are trained exclusively on Scene Flow and evaluated without fine-tuning on four standard benchmarks (ETH3D, KITTI 2012/2015, Middlebury) plus a novel 5,313-pair Canterbury forestry dataset captured with ZED Mini camera (1920x1080). Performance reveals scene-dependent patterns: foundation models excel on structured scenes (BridgeDepth: 0.23 px on ETH3D, 0.83-1.07 px on KITTI; DEFOM: 0.35-4.65 px across benchmarks), while iterative methods maintain cross-domain robustness (IGEV++: 0.36-6.77 px; IGEV: 0.33-21.91 px). Critical finding: RAFT-Stereo exhibits catastrophic ETH3D failure (26.23 px EPE, 98 percent error rate) due to negative disparity predictions, while performing normally on KITTI (0.90-1.11 px). Qualitative evaluation on Canterbury forestry dataset identifies DEFOM as the optimal gold-standard baseline for vegetation depth estimation, exhibiting superior depth smoothness, occlusion handling, and cross-domain consistency compared to IGEV++, despite IGEV++'s finer detail preservation.
☆ DM3D: Deformable Mamba via Offset-Guided Gaussian Sequencing for Point Cloud Understanding
Bin Liu, Chunyang Wang, Xuelian Liu
State Space Models (SSMs) demonstrate significant potential for long-sequence modeling, but their reliance on input order conflicts with the irregular nature of point clouds. Existing approaches often rely on predefined serialization strategies, which cannot adjust based on diverse geometric structures. To overcome this limitation, we propose \textbf{DM3D}, a deformable Mamba architecture for point cloud understanding. Specifically, DM3D introduces an offset-guided Gaussian sequencing mechanism that unifies local resampling and global reordering within a deformable scan. The Gaussian-based KNN Resampling (GKR) enhances structural awareness by adaptively reorganizing neighboring points, while the Gaussian-based Differentiable Reordering (GDR) enables end-to-end optimization of serialization order. Furthermore, a Tri-Path Frequency Fusion module enhances feature complementarity and reduces aliasing. Together, these components enable structure-adaptive serialization of point clouds. Extensive experiments on benchmark datasets show that DM3D achieves state-of-the-art performance in classification, few-shot learning, and part segmentation, demonstrating that adaptive serialization effectively unlocks the potential of SSMs for point cloud understanding.
☆ What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Models
Tianchen Deng, Yue Pan, Shenghai Yuan, Dong Li, Chen Wang, Mingrui Li, Long Chen, Lihua Xie, Danwei Wang, Jingchuan Wang, Javier Civera, Hesheng Wang, Weidong Chen
In this paper, we provide a comprehensive overview of existing scene representation methods for robotics, covering traditional representations such as point clouds, voxels, signed distance functions (SDF), and scene graphs, as well as more recent neural representations like Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), and the emerging Foundation Models. While current SLAM and localization systems predominantly rely on sparse representations like point clouds and voxels, dense scene representations are expected to play a critical role in downstream tasks such as navigation and obstacle avoidance. Moreover, neural representations such as NeRF, 3DGS, and foundation models are well-suited for integrating high-level semantic features and language-based priors, enabling more comprehensive 3D scene understanding and embodied intelligence. In this paper, we categorized the core modules of robotics into five parts (Perception, Mapping, Localization, Navigation, Manipulation). We start by presenting the standard formulation of different scene representation methods and comparing the advantages and disadvantages of scene representation across different modules. This survey is centered around the question: What is the best 3D scene representation for robotics? We then discuss the future development trends of 3D scene representations, with a particular focus on how the 3D Foundation Model could replace current methods as the unified solution for future robotic applications. The remaining challenges in fully realizing this model are also explored. We aim to offer a valuable resource for both newcomers and experienced researchers to explore the future of 3D scene representations and their application in robotics. We have published an open-source project on GitHub and will continue to add new works and technologies to this project.
☆ YOLOA: Real-Time Affordance Detection via LLM Adapter
Yuqi Ji, Junjie Ke, Lihuo He, Jun Liu, Kaifan Zhang, Yu-Kun Lai, Guiguang Ding, Xinbo Gao
Affordance detection aims to jointly address the fundamental "what-where-how" challenge in embodied AI by understanding "what" an object is, "where" the object is located, and "how" it can be used. However, most affordance learning methods focus solely on "how" objects can be used while neglecting the "what" and "where" aspects. Other affordance detection methods treat object detection and affordance learning as two independent tasks, lacking effective interaction and real-time capability. To overcome these limitations, we introduce YOLO Affordance (YOLOA), a real-time affordance detection model that jointly handles these two tasks via a large language model (LLM) adapter. Specifically, YOLOA employs a lightweight detector consisting of object detection and affordance learning branches refined through the LLM Adapter. During training, the LLM Adapter interacts with object and affordance preliminary predictions to refine both branches by generating more accurate class priors, box offsets, and affordance gates. Experiments on our relabeled ADG-Det and IIT-Heat benchmarks demonstrate that YOLOA achieves state-of-the-art accuracy (52.8 / 73.1 mAP on ADG-Det / IIT-Heat) while maintaining real-time performance (up to 89.77 FPS, and up to 846.24 FPS for the lightweight variant). This indicates that YOLOA achieves an excellent trade-off between accuracy and efficiency.
comment : 13 pages, 9 figures, conference
☆ ViDiC: Video Difference Captioning
Jiangtao Wu, Shihao Li, Zhaozhou Bian, Yuanxing Zhang, Jialu Chen, Runzhe Wen, An Ping, Yiwen He, Jiakai Wang, Jiaheng Liu
Understanding visual differences between dynamic scenes requires the comparative perception of compositional, spatial, and temporal changes--a capability that remains underexplored in existing vision-language systems. While prior work on Image Difference Captioning (IDC) has enabled models to describe semantic changes between static images, these approaches fail to capture motion continuity, event evolution, or editing consistency over time. We introduce the ViDiC (Video Difference Captioning) task and its corresponding ViDiC-1K dataset, designed to evaluate the ability of Multimodal Large Language Models (MLLMs) to provide fine-grained descriptions of similarities and differences between video pairs. ViDiC-1K comprises 1,000 curated video pairs annotated with over 4,000 comparative checklist items, covering seven categories: subject, style, background, cinematography, motion, location, and playback techniques. To ensure reliable evaluation, we propose a dual-checklist framework that measures the accuracy of similarity and difference separately, based on the LLM-as-a-Judge protocol. Experiments on nineteen representative multimodal models reveal a significant performance gap in their comparative description and difference perception abilities. We hope ViDiC-1K can be a challenging benchmark that lays a solid foundation for advancing video understanding, edit awareness, and comparative reasoning in multimodal intelligence.
☆ MOS: Mitigating Optical-SAR Modality Gap for Cross-Modal Ship Re-Identification
Yujian Zhao, Hankun Liu, Guanglin Niu
Cross-modal ship re-identification (ReID) between optical and synthetic aperture radar (SAR) imagery has recently emerged as a critical yet underexplored task in maritime intelligence and surveillance. However, the substantial modality gap between optical and SAR images poses a major challenge for robust identification. To address this issue, we propose MOS, a novel framework designed to mitigate the optical-SAR modality gap and achieve modality-consistent feature learning for optical-SAR cross-modal ship ReID. MOS consists of two core components: (1) Modality-Consistent Representation Learning (MCRL) applies denoise SAR image procession and a class-wise modality alignment loss to align intra-identity feature distributions across modalities. (2) Cross-modal Data Generation and Feature fusion (CDGF) leverages a brownian bridge diffusion model to synthesize cross-modal samples, which are subsequently fused with original features during inference to enhance alignment and discriminability. Extensive experiments on the HOSS ReID dataset demonstrate that MOS significantly surpasses state-of-the-art methods across all evaluation protocols, achieving notable improvements of +3.0%, +6.2%, and +16.4% in R1 accuracy under the ALL to ALL, Optical to SAR, and SAR to Optical settings, respectively. The code and trained models will be released upon publication.
☆ ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understanding
Lingjun Zhao, Yandong Luo, James Hay, Lu Gan
We introduce ShelfGaussian, an open-vocabulary multi-modal Gaussian-based 3D scene understanding framework supervised by off-the-shelf vision foundation models (VFMs). Gaussian-based methods have demonstrated superior performance and computational efficiency across a wide range of scene understanding tasks. However, existing methods either model objects as closed-set semantic Gaussians supervised by annotated 3D labels, neglecting their rendering ability, or learn open-set Gaussian representations via purely 2D self-supervision, leading to degraded geometry and limited to camera-only settings. To fully exploit the potential of Gaussians, we propose a Multi-Modal Gaussian Transformer that enables Gaussians to query features from diverse sensor modalities, and a Shelf-Supervised Learning Paradigm that efficiently optimizes Gaussians with VFM features jointly at 2D image and 3D scene levels. We evaluate ShelfGaussian on various perception and planning tasks. Experiments on Occ3D-nuScenes demonstrate its state-of-the-art zero-shot semantic occupancy prediction performance. ShelfGaussian is further evaluated on an unmanned ground vehicle (UGV) to assess its in the-wild performance across diverse urban scenarios. Project website: https://lunarlab-gatech.github.io/ShelfGaussian/.
☆ FireSentry: A Multi-Modal Spatio-temporal Benchmark Dataset for Fine-Grained Wildfire Spread Forecasting
Nan Zhou, Huandong Wang, Jiahao Li, Han Li, Yali Song, Qiuhua Wang, Yong Li, Xinlei Chen
Fine-grained wildfire spread prediction is crucial for enhancing emergency response efficacy and decision-making precision. However, existing research predominantly focuses on coarse spatiotemporal scales and relies on low-resolution satellite data, capturing only macroscopic fire states while fundamentally constraining high-precision localized fire dynamics modeling capabilities. To bridge this gap, we present FireSentry, a provincial-scale multi-modal wildfire dataset characterized by sub-meter spatial and sub-second temporal resolution. Collected using synchronized UAV platforms, FireSentry provides visible and infrared video streams, in-situ environmental measurements, and manually validated fire masks. Building on FireSentry, we establish a comprehensive benchmark encompassing physics-based, data-driven, and generative models, revealing the limitations of existing mask-only approaches. Our analysis proposes FiReDiff, a novel dual-modality paradigm that first predicts future video sequences in the infrared modality, and then precisely segments fire masks in the mask modality based on the generated dynamics. FiReDiff achieves state-of-the-art performance, with video quality gains of 39.2% in PSNR, 36.1% in SSIM, 50.0% in LPIPS, 29.4% in FVD, and mask accuracy gains of 3.3% in AUPRC, 59.1% in F1 score, 42.9% in IoU, and 62.5% in MSE when applied to generative models. The FireSentry benchmark dataset and FiReDiff paradigm collectively advance fine-grained wildfire forecasting and dynamic disaster simulation. The processed benchmark dataset is publicly available at: https://github.com/Munan222/FireSentry-Benchmark-Dataset.
☆ A Hybrid Deep Learning Framework with Explainable AI for Lung Cancer Classification with DenseNet169 and SVM
Md Rashidul Islam, Bakary Gibba, Altagi Abdallah Bakheit Abdelgadir
Lung cancer is a very deadly disease worldwide, and its early diagnosis is crucial for increasing patient survival rates. Computed tomography (CT) scans are widely used for lung cancer diagnosis as they can give detailed lung structures. However, manual interpretation is time-consuming and prone to human error. To surmount this challenge, the study proposes a deep learning-based automatic lung cancer classification system to enhance detection accuracy and interpretability. The IQOTHNCCD lung cancer dataset is utilized, which is a public CT scan dataset consisting of cases categorized into Normal, Benign, and Malignant and used DenseNet169, which includes Squeezeand-Excitation blocks for attention-based feature extraction, Focal Loss for handling class imbalance, and a Feature Pyramid Network (FPN) for multi-scale feature fusion. In addition, an SVM model was developed using MobileNetV2 for feature extraction, improving its classification performance. For model interpretability enhancement, the study integrated Grad-CAM for the visualization of decision-making regions in CT scans and SHAP (Shapley Additive Explanations) for explanation of feature contributions within the SVM model. Intensive evaluation was performed, and it was found that both DenseNet169 and SVM models achieved 98% accuracy, suggesting their robustness for real-world medical practice. These results open up the potential for deep learning to improve the diagnosis of lung cancer by a higher level of accuracy, transparency, and robustness.
☆ SeeU: Seeing the Unseen Wo rl d via 4D Dynamics-aware Generation
Yu Yuan, Tharindu Wickremasinghe, Zeeshan Nadir, Xijun Wang, Yiheng Chi, Stanley H. Chan
Images and videos are discrete 2D projections of the 4D world (3D space + time). Most visual understanding, prediction, and generation operate directly on 2D observations, leading to suboptimal performance. We propose SeeU, a novel approach that learns the continuous 4D dynamics and generate the unseen visual contents. The principle behind SeeU is a new 2D$\to$4D$\to$2D learning framework. SeeU first reconstructs the 4D world from sparse and monocular 2D frames (2D$\to$4D). It then learns the continuous 4D dynamics on a low-rank representation and physical constraints (discrete 4D$\to$continuous 4D). Finally, SeeU rolls the world forward in time, re-projects it back to 2D at sampled times and viewpoints, and generates unseen regions based on spatial-temporal context awareness (4D$\to$2D). By modeling dynamics in 4D, SeeU achieves continuous and physically-consistent novel visual generation, demonstrating strong potentials in multiple tasks including unseen temporal generation, unseen spatial generation, and video editing.
comment : Project Page: https://yuyuanspace.com/SeeU/
☆ Hierarchical Attention for Sparse Volumetric Anomaly Detection in Subclinical Keratoconus
Lynn Kandakji, William Woof, Nikolas Pontikos
The detection of weak, spatially distributed anomalies in volumetric medical imaging remains a major challenge. The subtle, non-adjacent nature of early disease signals is often lost due to suboptimal architectural inductive biases: 2D/3D CNNs impose strong locality, while ViTs diffuse unconstrained global attention. This conflict leaves the optimal inductive structure for robust, sparse volumetric pattern recognition unresolved. This study presents a controlled comparison of sixteen modern deep learning architectures spanning 2D/3D convolutional, hybrid, and volumetric transformer families for subclinical keratoconus (SKC) detection from 3D anterior segment OCT volumes. We demonstrate that hierarchical attention models offer a superior and more parameter-efficient inductive bias, surpassing the performance of both 2D and 3D CNNs and ViTs. Our results show 21-23% higher sensitivity and specificity in the sparse anomaly (subclinical) regime. Mechanistic analyses reveal that this advantage stems from precise spatial scale alignment: hierarchical windowing produces effective receptive fields matched to the intermediate, multi-slice extent of subclinical abnormalities. This avoids excessive CNN locality and diffuse global attention. Attention-distance measurements confirm a key insight into architectural adaptation: the required spatial integration length shifts significantly based on the signal strength, with subclinical cases necessitating longer integration compared to both healthy and manifest disease states. Representational similarity and auxiliary age/sex prediction tasks further support the generalizability of these inductive principles. The findings provide design guidance for future volumetric anomaly detection systems, establishing hierarchical attention as a principled and effective approach for early pathological change analysis in 3D medical imaging.
comment : 16 pages, 7 figures, 6 tables
☆ HalluGen: Synthesizing Realistic and Controllable Hallucinations for Evaluating Image Restoration
Seunghoi Kim, Henry F. J. Tregidgo, Chen Jin, Matteo Figini, Daniel C. Alexander
Generative models are prone to hallucinations: plausible but incorrect structures absent in the ground truth. This issue is problematic in image restoration for safety-critical domains such as medical imaging, industrial inspection, and remote sensing, where such errors undermine reliability and trust. For example, in low-field MRI, widely used in resource-limited settings, restoration models are essential for enhancing low-quality scans, yet hallucinations can lead to serious diagnostic errors. Progress has been hindered by a circular dependency: evaluating hallucinations requires labeled data, yet such labels are costly and subjective. We introduce HalluGen, a diffusion-based framework that synthesizes realistic hallucinations with controllable type, location, and severity, producing perceptually realistic but semantically incorrect outputs (segmentation IoU drops from 0.86 to 0.36). Using HalluGen, we construct the first large-scale hallucination dataset comprising 4,350 annotated images derived from 1,450 brain MR images for low-field enhancement, enabling systematic evaluation of hallucination detection and mitigation. We demonstrate its utility in two applications: (1) benchmarking image quality metrics and developing Semantic Hallucination Assessment via Feature Evaluation (SHAFE), a feature-based metric with soft-attention pooling that improves hallucination sensitivity over traditional metrics; and (2) training reference-free hallucination detectors that generalize to real restoration failures. Together, HalluGen and its open dataset establish the first scalable foundation for evaluating hallucinations in safety-critical image restoration.
☆ ProtoEFNet: Dynamic Prototype Learning for Inherently Interpretable Ejection Fraction Estimation in Echocardiography MICCAI 2025
Yeganeh Ghamary, Victoria Wu, Hooman Vaseli, Christina Luong, Teresa Tsang, Siavash Bigdeli, Purang Abolmaesumi
Ejection fraction (EF) is a crucial metric for assessing cardiac function and diagnosing conditions such as heart failure. Traditionally, EF estimation requires manual tracing and domain expertise, making the process time-consuming and subject to interobserver variability. Most current deep learning methods for EF prediction are black-box models with limited transparency, which reduces clinical trust. Some post-hoc explainability methods have been proposed to interpret the decision-making process after the prediction is made. However, these explanations do not guide the model's internal reasoning and therefore offer limited reliability in clinical applications. To address this, we introduce ProtoEFNet, a novel video-based prototype learning model for continuous EF regression. The model learns dynamic spatiotemporal prototypes that capture clinically meaningful cardiac motion patterns. Additionally, the proposed Prototype Angular Separation (PAS) loss enforces discriminative representations across the continuous EF spectrum. Our experiments on the EchonetDynamic dataset show that ProtoEFNet can achieve accuracy on par with its non-interpretable counterpart while providing clinically relevant insight. The ablation study shows that the proposed loss boosts performance with a 2% increase in F1 score from 77.67$\pm$2.68 to 79.64$\pm$2.10. Our source code is available at: https://github.com/DeepRCL/ProtoEF
comment : 11 pages, Accepted in IMIMIC Workshop at MICCAI 2025
☆ Step-by-step Layered Design Generation
Faizan Farooq Khan, K J Joseph, Koustava Goswami, Mohamed Elhoseiny, Balaji Vasan Srinivasan
Design generation, in its essence, is a step-by-step process where designers progressively refine and enhance their work through careful modifications. Despite this fundamental characteristic, existing approaches mainly treat design synthesis as a single-step generation problem, significantly underestimating the inherent complexity of the creative process. To bridge this gap, we propose a novel problem setting called Step-by-Step Layered Design Generation, which tasks a machine learning model with generating a design that adheres to a sequence of instructions from a designer. Leveraging recent advancements in multi-modal LLMs, we propose SLEDGE: Step-by-step LayEred Design GEnerator to model each update to a design as an atomic, layered change over its previous state, while being grounded in the instruction. To complement our new problem setting, we introduce a new evaluation suite, including a dataset and a benchmark. Our exhaustive experimental analysis and comparison with state-of-the-art approaches tailored to our new setup demonstrate the efficacy of our approach. We hope our work will attract attention to this pragmatic and under-explored research area.
☆ NavMapFusion: Diffusion -based Fusion of Navigation Maps for Online Vectorized HD Map Construction WACV 2026
Thomas Monninger, Zihan Zhang, Steffen Staab, Sihao Ding
Accurate environmental representations are essential for autonomous driving, providing the foundation for safe and efficient navigation. Traditionally, high-definition (HD) maps are providing this representation of the static road infrastructure to the autonomous system a priori. However, because the real world is constantly changing, such maps must be constructed online from on-board sensor data. Navigation-grade standard-definition (SD) maps are widely available, but their resolution is insufficient for direct deployment. Instead, they can be used as coarse prior to guide the online map construction process. We propose NavMapFusion, a diffusion-based framework that performs iterative denoising conditioned on high-fidelity sensor data and on low-fidelity navigation maps. This paper strives to answer: (1) How can coarse, potentially outdated navigation maps guide online map construction? (2) What advantages do diffusion models offer for map fusion? We demonstrate that diffusion-based map construction provides a robust framework for map fusion. Our key insight is that discrepancies between the prior map and online perception naturally correspond to noise within the diffusion process; consistent regions reinforce the map construction, whereas outdated segments are suppressed. On the nuScenes benchmark, NavMapFusion conditioned on coarse road lines from OpenStreetMap data reaches a 21.4% relative improvement on 100 m, and even stronger improvements on larger perception ranges, while maintaining real-time capabilities. By fusing low-fidelity priors with high-fidelity sensor data, the proposed method generates accurate and up-to-date environment representations, guiding towards safer and more reliable autonomous driving. The code is available at https://github.com/tmonnin/navmapfusion
comment : Accepted to 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2026)
☆ Open Set Face Forgery Detection via Dual-Level Evidence Collection
Zhongyi Cai, Bryce Gernon, Wentao Bao, Yifan Li, Matthew Wright, Yu Kong
The proliferation of face forgeries has increasingly undermined confidence in the authenticity of online content. Given the rapid development of face forgery generation algorithms, new fake categories are likely to keep appearing, posing a major challenge to existing face forgery detection methods. Despite recent advances in face forgery detection, existing methods are typically limited to binary Real-vs-Fake classification or the identification of known fake categories, and are incapable of detecting the emergence of novel types of forgeries. In this work, we study the Open Set Face Forgery Detection (OSFFD) problem, which demands that the detection model recognize novel fake categories. We reformulate the OSFFD problem and address it through uncertainty estimation, enhancing its applicability to real-world scenarios. Specifically, we propose the Dual-Level Evidential face forgery Detection (DLED) approach, which collects and fuses category-specific evidence on the spatial and frequency levels to estimate prediction uncertainty. Extensive evaluations conducted across diverse experimental settings demonstrate that the proposed DLED method achieves state-of-the-art performance, outperforming various baseline models by an average of 20% in detecting forgeries from novel fake categories. Moreover, on the traditional Real-versus-Fake face forgery detection task, our DLED method concurrently exhibits competitive performance.
☆ A Retrieval-Augmented Generation Approach to Extracting Algorithmic Logic from Neural Networks
Waleed Khalid, Dmitry Ignatov, Radu Timofte
Reusing existing neural-network components is central to research efficiency, yet discovering, extracting, and validating such modules across thousands of open-source repositories remains difficult. We introduce NN-RAG, a retrieval-augmented generation system that converts large, heterogeneous PyTorch codebases into a searchable and executable library of validated neural modules. Unlike conventional code search or clone-detection tools, NN-RAG performs scope-aware dependency resolution, import-preserving reconstruction, and validator-gated promotion -- ensuring that every retrieved block is scope-closed, compilable, and runnable. Applied to 19 major repositories, the pipeline extracted 1,289 candidate blocks, validated 941 (73.0%), and demonstrated that over 80% are structurally unique. Through multi-level de-duplication (exact, lexical, structural), we find that NN-RAG contributes the overwhelming majority of unique architectures to the LEMUR dataset, supplying approximately 72% of all novel network structures. Beyond quantity, NN-RAG uniquely enables cross-repository migration of architectural patterns, automatically identifying reusable modules in one project and regenerating them, dependency-complete, in another context. To our knowledge, no other open-source system provides this capability at scale. The framework's neutral specifications further allow optional integration with language models for synthesis or dataset registration without redistributing third-party code. Overall, NN-RAG transforms fragmented vision code into a reproducible, provenance-tracked substrate for algorithmic discovery, offering a first open-source solution that both quantifies and expands the diversity of executable neural architectures across repositories.
☆ Bayes-DIC Net: Estimating Digital Image Cor relation Uncertainty with Bayesian Neural Networks
Biao Chen, Zhenhua Lei, Yahui Zhang, Tongzhi Niu
This paper introduces a novel method for generating high-quality Digital Image Correlation (DIC) dataset based on non-uniform B-spline surfaces. By randomly generating control point coordinates, we construct displacement fields that encompass a variety of realistic displacement scenarios, which are subsequently used to generate speckle pattern datasets. This approach enables the generation of a large-scale dataset that capture real-world displacement field situations, thereby enhancing the training and generalization capabilities of deep learning-based DIC algorithms. Additionally, we propose a novel network architecture, termed Bayes-DIC Net, which extracts information at multiple levels during the down-sampling phase and facilitates the aggregation of information across various levels through a single skip connection during the up-sampling phase. Bayes-DIC Net incorporates a series of lightweight convolutional blocks designed to expand the receptive field and capture rich contextual information while minimizing computational costs. Furthermore, by integrating appropriate dropout modules into Bayes-DIC Net and activating them during the network inference stage, Bayes-DIC Net is transformed into a Bayesian neural network. This transformation allows the network to provide not only predictive results but also confidence levels in these predictions when processing real unlabeled datasets. This feature significantly enhances the practicality and reliability of our network in real-world displacement field prediction tasks. Through these innovations, this paper offers new perspectives and methods for dataset generation and algorithm performance enhancement in the field of DIC.
comment : 17 pages, 8 figures
☆ SyncTrack4D: Cross-Video Motion Alignment and Video Synchronization for Multi-Video 4D Gaussian Splatting
Yonghan Lee, Tsung-Wei Huang, Shiv Gehlot, Jaehoon Choi, Guan-Ming Su, Dinesh Manocha
Modeling dynamic 3D scenes is challenging due to their high-dimensional nature, which requires aggregating information from multiple views to reconstruct time-evolving 3D geometry and motion. We present a novel multi-video 4D Gaussian Splatting (4DGS) approach designed to handle real-world, unsynchronized video sets. Our approach, SyncTrack4D, directly leverages dense 4D track representation of dynamic scene parts as cues for simultaneous cross-video synchronization and 4DGS reconstruction. We first compute dense per-video 4D feature tracks and cross-video track correspondences by Fused Gromov-Wasserstein optimal transport approach. Next, we perform global frame-level temporal alignment to maximize overlapping motion of matched 4D tracks. Finally, we achieve sub-frame synchronization through our multi-video 4D Gaussian splatting built upon a motion-spline scaffold representation. The final output is a synchronized 4DGS representation with dense, explicit 3D trajectories, and temporal offsets for each video. We evaluate our approach on the Panoptic Studio and SyncNeRF Blender, demonstrating sub-frame synchronization accuracy with an average temporal error below 0.26 frames, and high-fidelity 4D reconstruction reaching 26.3 PSNR scores on the Panoptic Studio dataset. To the best of our knowledge, our work is the first general 4D Gaussian Splatting approach for unsynchronized video sets, without assuming the existence of predefined scene objects or prior models.
☆ DisentangleFormer: Spatial-Channel Decoupling for Multi-Channel Vision
Jiashu Liao, Pietro Liò, Marc de Kamps, Duygu Sarikaya
Vision Transformers face a fundamental limitation: standard self-attention jointly processes spatial and channel dimensions, leading to entangled representations that prevent independent modeling of structural and semantic dependencies. This problem is especially pronounced in hyperspectral imaging, from satellite hyperspectral remote sensing to infrared pathology imaging, where channels capture distinct biophysical or biochemical cues. We propose DisentangleFormer, an architecture that achieves robust multi-channel vision representation through principled spatial-channel decoupling. Motivated by information-theoretic principles of decorrelated representation learning, our parallel design enables independent modeling of structural and semantic cues while minimizing redundancy between spatial and channel streams. Our design integrates three core components: (1) Parallel Disentanglement: Independently processes spatial-token and channel-token streams, enabling decorrelated feature learning across spatial and spectral dimensions, (2) Squeezed Token Enhancer: An adaptive calibration module that dynamically fuses spatial and channel streams, and (3) Multi-Scale FFN: complementing global attention with multi-scale local context to capture fine-grained structural and semantic dependencies. Extensive experiments on hyperspectral benchmarks demonstrate that DisentangleFormer achieves state-of-the-art performance, consistently outperforming existing models on Indian Pine, Pavia University, and Houston, the large-scale BigEarthNet remote sensing dataset, as well as an infrared pathology dataset. Moreover, it retains competitive accuracy on ImageNet while reducing computational cost by 17.8% in FLOPs. The code will be made publicly available upon acceptance.
☆ Mind-to-Face: Neural-Driven Photorealistic Avatar Synthesis via EEG Decoding
Haolin Xiong, Tianwen Fu, Pratusha Bhuvana Prasad, Yunxuan Cai, Haiwei Chen, Wenbin Teng, Hanyuan Xiao, Yajie Zhao
Current expressive avatar systems rely heavily on visual cues, failing when faces are occluded or when emotions remain internal. We present Mind-to-Face, the first framework that decodes non-invasive electroencephalogram (EEG) signals directly into high-fidelity facial expressions. We build a dual-modality recording setup to obtain synchronized EEG and multi-view facial video during emotion-eliciting stimuli, enabling precise supervision for neural-to-visual learning. Our model uses a CNN-Transformer encoder to map EEG signals into dense 3D position maps, capable of sampling over 65k vertices, capturing fine-scale geometry and subtle emotional dynamics, and renders them through a modified 3D Gaussian Splatting pipeline for photorealistic, view-consistent results. Through extensive evaluation, we show that EEG alone can reliably predict dynamic, subject-specific facial expressions, including subtle emotional responses, demonstrating that neural signals contain far richer affective and geometric information than previously assumed. Mind-to-Face establishes a new paradigm for neural-driven avatars, enabling personalized, emotion-aware telepresence and cognitive interaction in immersive environments.
comment : 16 pages, 11 figures
☆ Real-time Cricket Sorting By Sex
Juan Manuel Cantarero Angulo, Matthew Smith
The global demand for sustainable protein sources is driving increasing interest in edible insects, with Acheta domesticus (house cricket) identified as one of the most suitable species for industrial production. Current farming practices typically rear crickets in mixed-sex populations without automated sex sorting, despite potential benefits such as selective breeding, optimized reproduction ratios, and nutritional differentiation. This work presents a low-cost, real-time system for automated sex-based sorting of Acheta domesticus, combining computer vision and physical actuation. The device integrates a Raspberry Pi 5 with the official Raspberry AI Camera and a custom YOLOv8 nano object detection model, together with a servo-actuated sorting arm. The model reached a mean Average Precision at IoU 0.5 (mAP@0.5) of 0.977 during testing, and real-world experiments with groups of crickets achieved an overall sorting accuracy of 86.8%. These results demonstrate the feasibility of deploying lightweight deep learning models on resource-constrained devices for insect farming applications, offering a practical solution to improve efficiency and sustainability in cricket production.
comment : 13 pages, 14 figures
☆ Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correction CVPR 2026
Rui Fonseca, Bruno Martins, Gil Rocha
Image captioning has drawn considerable attention from the natural language processing and computer vision fields. Aiming to reduce the reliance on curated data, several studies have explored image captioning without any humanly-annotated image-text pairs for training, although existing methods are still outperformed by fully supervised approaches. This paper proposes TOMCap, i.e., an improved text-only training method that performs captioning without the need for aligned image-caption pairs. The method is based on prompting a pre-trained language model decoder with information derived from a CLIP representation, after undergoing a process to reduce the modality gap. We specifically tested the combined use of retrieved examples of captions, and latent vector representations, to guide the generation process. Through extensive experiments, we show that TOMCap outperforms other training-free and text-only methods. We also analyze the impact of different choices regarding the configuration of the retrieval-augmentation and modality gap reduction components.
comment : Submitted to CVPR 2026
☆ How (Mis)calibrated is Your Federated CLIP and What To Do About It?
Mainak Singha, Masih Aminbeidokhti, Paolo Casari, Elisa Ricci, Subhankar Roy
While vision-language models like CLIP have been extensively studied, their calibration, crucial for reliable predictions, has received limited attention. Although a few prior works have examined CLIP calibration in offline settings, the impact of fine-tuning CLIP in a federated learning (FL) setup remains unexplored. In this work, we investigate how FL affects CLIP calibration and propose strategies to improve reliability in this distributed setting. We first analyze Textual Prompt Tuning approaches and show that they degrade calibration metrics when operating under FL. We also evaluate existing in-training calibration techniques across four global aggregation methods, finding that they provide limited improvements. Our results suggest that the key challenge lies not only in how we aggregate or calibrate, but in which components we choose to fine-tune. Motivated by this insight, we propose $\text{FL}^2\text{oRA}$, a straightforward LoRA-based approach that naturally improves calibration in FL, and we analyze the factors behind its effectiveness. Experiments on multiple benchmarks demonstrate that $\text{FL}^2\text{oRA}$ consistently produces well-calibrated models, reducing the need for explicit calibration procedures. Codes are available at https://github.com/mainaksingha01/FL2oRA.
☆ Gamma-from-Mono: Road-Relative, Metric, Self-Supervised Monocular Geometry for Vehicular Applications 3DV 2026
Gasser Elazab, Maximilian Jansen, Michael Unterreiner, Olaf Hellwich
Accurate perception of the vehicle's 3D surroundings, including fine-scale road geometry, such as bumps, slopes, and surface irregularities, is essential for safe and comfortable vehicle control. However, conventional monocular depth estimation often oversmooths these features, losing critical information for motion planning and stability. To address this, we introduce Gamma-from-Mono (GfM), a lightweight monocular geometry estimation method that resolves the projective ambiguity in single-camera reconstruction by decoupling global and local structure. GfM predicts a dominant road surface plane together with residual variations expressed by gamma, a dimensionless measure of vertical deviation from the plane, defined as the ratio of a point's height above it to its depth from the camera, and grounded in established planar parallax geometry. With only the camera's height above ground, this representation deterministically recovers metric depth via a closed form, avoiding full extrinsic calibration and naturally prioritizing near-road detail. Its physically interpretable formulation makes it well suited for self-supervised learning, eliminating the need for large annotated datasets. Evaluated on KITTI and the Road Surface Reconstruction Dataset (RSRD), GfM achieves state-of-the-art near-field accuracy in both depth and gamma estimation while maintaining competitive global depth performance. Our lightweight 8.88M-parameter model adapts robustly across diverse camera setups and, to our knowledge, is the first self-supervised monocular approach evaluated on RSRD.
comment : Accepted in 3DV 2026
☆ Learning Single-Image Super-Resolution in the JPEG Compressed Domain ICIP 2025
Sruthi Srinivasan, Elham Shakibapour, Rajy Rawther, Mehdi Saeedi
Deep learning models have grown increasingly complex, with input data sizes scaling accordingly. Despite substantial advances in specialized deep learning hardware, data loading continues to be a major bottleneck that limits training and inference speed. To address this challenge, we propose training models directly on encoded JPEG features, reducing the computational overhead associated with full JPEG decoding and significantly improving data loading efficiency. While prior works have focused on recognition tasks, we investigate the effectiveness of this approach for the restoration task of single-image super-resolution (SISR). We present a lightweight super-resolution pipeline that operates on JPEG discrete cosine transform (DCT) coefficients in the frequency domain. Our pipeline achieves a 2.6x speedup in data loading and a 2.5x speedup in training, while preserving visual quality comparable to standard SISR approaches.
comment : 7 pages, 4 figures, 2 tables, SEEDS Workshop, ICIP 2025
☆ Plug-and-Play Image Restoration with Flow Matching: A Continuous Viewpoint
Fan Jia, Yuhao Huang, Shih-Hsin Wang, Cristina Garcia-Cardona, Andrea L. Bertozzi, Bao Wang
Flow matching-based generative models have been integrated into the plug-and-play image restoration framework, and the resulting plug-and-play flow matching (PnP-Flow) model has achieved some remarkable empirical success for image restoration. However, the theoretical understanding of PnP-Flow lags its empirical success. In this paper, we derive a continuous limit for PnP-Flow, resulting in a stochastic differential equation (SDE) surrogate model of PnP-Flow. The SDE model provides two particular insights to improve PnP-Flow for image restoration: (1) It enables us to quantify the error for image restoration, informing us to improve step scheduling and regularize the Lipschitz constant of the neural network-parameterized vector field for error reduction. (2) It informs us to accelerate off-the-shelf PnP-Flow models via extrapolation, resulting in a rescaled version of the proposed SDE model. We validate the efficacy of the SDE-informed improved PnP-Flow using several benchmark tasks, including image denoising, deblurring, super-resolution, and inpainting. Numerical results show that our method significantly outperforms the baseline PnP-Flow and other state-of-the-art approaches, achieving superior performance across evaluation metrics.
☆ Inference-time Stochastic Refinement of GRU-Normalizing Flow for Real-time Video Motion Transfer
Tasmiah Haque, Srinjoy Das
Real-time video motion transfer applications such as immersive gaming and vision-based anomaly detection require accurate yet diverse future predictions to support realistic synthesis and robust downstream decision making under uncertainty. To improve the diversity of such sequential forecasts we propose a novel inference-time refinement technique that combines Gated Recurrent Unit-Normalizing Flows (GRU-NF) with stochastic sampling methods. While GRU-NF can capture multimodal distributions through its integration of normalizing flows within a temporal forecasting framework, its deterministic transformation structure can limit expressivity. To address this, inspired by Stochastic Normalizing Flows (SNF), we introduce Markov Chain Monte Carlo (MCMC) steps during GRU-NF inference, enabling the model to explore a richer output space and better approximate the true data distribution without retraining. We validate our approach in a keypoint-based video motion transfer pipeline, where capturing temporally coherent and perceptually diverse future trajectories is essential for realistic samples and low bandwidth communication. Experiments show that our inference framework, Gated Recurrent Unit- Stochastic Normalizing Flows (GRU-SNF) outperforms GRU-NF in generating diverse outputs without sacrificing accuracy, even under longer prediction horizons. By injecting stochasticity during inference, our approach captures multimodal behavior more effectively. These results highlight the potential of integrating stochastic dynamics with flow-based sequence models for generative time series forecasting.
☆ UniLight: A Unified Representation for Lighting
Zitian Zhang, Iliyan Georgiev, Michael Fischer, Yannick Hold-Geoffroy, Jean-François Lalonde, Valentin Deschaintre
Lighting has a strong influence on visual appearance, yet understanding and representing lighting in images remains notoriously difficult. Various lighting representations exist, such as environment maps, irradiance, spherical harmonics, or text, but they are incompatible, which limits cross-modal transfer. We thus propose UniLight, a joint latent space as lighting representation, that unifies multiple modalities within a shared embedding. Modality-specific encoders for text, images, irradiance, and environment maps are trained contrastively to align their representations, with an auxiliary spherical-harmonics prediction task reinforcing directional understanding. Our multi-modal data pipeline enables large-scale training and evaluation across three tasks: lighting-based retrieval, environment-map generation, and lighting control in diffusion-based image synthesis. Experiments show that our representation captures consistent and transferable lighting features, enabling flexible manipulation across modalities.
comment : Project page: https://lvsn.github.io/UniLight
☆ Studying Various Activation Functions and Non-IID Data for Machine Learning Model Robustness
Long Dang, Thushari Hapuarachchi, Kaiqi Xiong, Jing Lin
Adversarial training is an effective method to improve the machine learning (ML) model robustness. Most existing studies typically consider the Rectified linear unit (ReLU) activation function and centralized training environments. In this paper, we study the ML model robustness using ten different activation functions through adversarial training in centralized environments and explore the ML model robustness in federal learning environments. In the centralized environment, we first propose an advanced adversarial training approach to improving the ML model robustness by incorporating model architecture change, soft labeling, simplified data augmentation, and varying learning rates. Then, we conduct extensive experiments on ten well-known activation functions in addition to ReLU to better understand how they impact the ML model robustness. Furthermore, we extend the proposed adversarial training approach to the federal learning environment, where both independent and identically distributed (IID) and non-IID data settings are considered. Our proposed centralized adversarial training approach achieves a natural and robust accuracy of 77.08% and 67.96%, respectively on CIFAR-10 against the fast gradient sign attacks. Experiments on ten activation functions reveal ReLU usually performs best. In the federated learning environment, however, the robust accuracy decreases significantly, especially on non-IID data. To address the significant performance drop in the non-IID data case, we introduce data sharing and achieve the natural and robust accuracy of 70.09% and 54.79%, respectively, surpassing the CalFAT algorithm, when 40% data sharing is used. That is, a proper percentage of data sharing can significantly improve the ML model robustness, which is useful to some real-world applications.
☆ MVRoom: Controllable 3D Indoor Scene Generation with Multi-View Diffusion Models
Shaoheng Fang, Chaohui Yu, Fan Wang, Qixing Huang
We introduce MVRoom, a controllable novel view synthesis (NVS) pipeline for 3D indoor scenes that uses multi-view diffusion conditioned on a coarse 3D layout. MVRoom employs a two-stage design in which the 3D layout is used throughout to enforce multi-view consistency. The first stage employs novel representations to effectively bridge the 3D layout and consistent image-based condition signals for multi-view generation. The second stage performs image-conditioned multi-view generation, incorporating a layout-aware epipolar attention mechanism to enhance multi-view consistency during the diffusion process. Additionally, we introduce an iterative framework that generates 3D scenes with varying numbers of objects and scene complexities by recursively performing multi-view generation (MVRoom), supporting text-to-scene generation. Experimental results demonstrate that our approach achieves high-fidelity and controllable 3D scene generation for NVS, outperforming state-of-the-art baseline methods both quantitatively and qualitatively. Ablation studies further validate the effectiveness of key components within our generation pipeline.
☆ 6 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision -Language Models
Leon Mayer, Piotr Kalinowski, Caroline Ebersbach, Marcel Knopp, Tim Rädsch, Evangelia Christodoulou, Annika Reinke, Fiona R. Kolbinger, Lena Maier-Hein
Vision-language models are increasingly integrated into clinical workflows. However, existing benchmarks primarily assess performance on common anatomical presentations and fail to capture the challenges posed by rare variants. To address this gap, we introduce AdversarialAnatomyBench, the first benchmark comprising naturally occurring rare anatomical variants across diverse imaging modalities and anatomical regions. We call such variants that violate learned priors about "typical" human anatomy natural adversarial anatomy. Benchmarking 22 state-of-the-art VLMs with AdversarialAnatomyBench yielded three key insights. First, when queried with basic medical perception tasks, mean accuracy dropped from 74% on typical to 29% on atypical anatomy. Even the best-performing models, GPT-5, Gemini 2.5 Pro, and Llama 4 Maverick, showed performance drops of 41-51%. Second, model errors closely mirrored expected anatomical biases. Third, neither model scaling nor interventions, including bias-aware prompting and test-time reasoning, resolved these issues. These findings highlight a critical and previously unquantified limitation in current VLM: their poor generalization to rare anatomical presentations. AdversarialAnatomyBench provides a foundation for systematically measuring and mitigating anatomical bias in multimodal medical AI systems.
☆ ReasonX: MLLM-Guided Intrinsic Image Decomposition
Alara Dirik, Tuanfeng Wang, Duygu Ceylan, Stefanos Zafeiriou, Anna Frühstück
Intrinsic image decomposition aims to separate images into physical components such as albedo, depth, normals, and illumination. While recent diffusion- and transformer-based models benefit from paired supervision from synthetic datasets, their generalization to diverse, real-world scenarios remains challenging. We propose ReasonX, a novel framework that leverages a multimodal large language model (MLLM) as a perceptual judge providing relative intrinsic comparisons, and uses these comparisons as GRPO rewards for fine-tuning intrinsic decomposition models on unlabeled, in-the-wild images. Unlike RL methods for generative models, our framework aligns conditional intrinsic predictors by rewarding agreement between the judge's relational assessments and analytically derived relations from the model's outputs. ReasonX is model-agnostic and can be applied to different intrinsic predictors. Across multiple base architectures and modalities, ReasonX yields significant improvements, including 9-25% WHDR reduction on IIW albedo and up to 46% depth accuracy gains on ETH3D, highlighting the promise of MLLM-guided comparative supervision to bridge low- and high-level vision reasoning.
☆ MoReGen: Multi- Agent Motion- Reasoning Engine for Code-based Text-to-Video Synthesis
Xiangyu Bai, He Liang, Bishoy Galoaa, Utsav Nandi, Shayda Moezzi, Yuhang He, Sarah Ostadabbas
While text-to-video (T2V) generation has achieved remarkable progress in photorealism, generating intent-aligned videos that faithfully obey physics principles remains a core challenge. In this work, we systematically study Newtonian motion-controlled text-to-video generation and evaluation, emphasizing physical precision and motion coherence. We introduce MoReGen, a motion-aware, physics-grounded T2V framework that integrates multi-agent LLMs, physics simulators, and renderers to generate reproducible, physically accurate videos from text prompts in the code domain. To quantitatively assess physical validity, we propose object-trajectory correspondence as a direct evaluation metric and present MoReSet, a benchmark of 1,275 human-annotated videos spanning nine classes of Newtonian phenomena with scene descriptions, spatiotemporal relations, and ground-truth trajectories. Using MoReSet, we conduct experiments on existing T2V models, evaluating their physical validity through both our MoRe metrics and existing physics-based evaluators. Our results reveal that state-of-the-art models struggle to maintain physical validity, while MoReGen establishes a principled direction toward physically coherent video synthesis.
☆ Generalized Event Partonomy Inference with Structured Hierarchical Predictive Learning
Zhou Chen, Joe Lin, Sathyanarayanan N. Aakur\\
Humans naturally perceive continuous experience as a hierarchy of temporally nested events, fine-grained actions embedded within coarser routines. Replicating this structure in computer vision requires models that can segment video not just retrospectively, but predictively and hierarchically. We introduce PARSE, a unified framework that learns multiscale event structure directly from streaming video without supervision. PARSE organizes perception into a hierarchy of recurrent predictors, each operating at its own temporal granularity: lower layers model short-term dynamics while higher layers integrate longer-term context through attention-based feedback. Event boundaries emerge naturally as transient peaks in prediction error, yielding temporally coherent, nested partonomies that mirror the containment relations observed in human event perception. Evaluated across three benchmarks, Breakfast Actions, 50 Salads, and Assembly 101, PARSE achieves state-of-the-art performance among streaming methods and rivals offline baselines in both temporal alignment (H-GEBD) and structural consistency (TED, hF1). The results demonstrate that predictive learning under uncertainty provides a scalable path toward human-like temporal abstraction and compositional event understanding.
comment : 16 pages, 7 figures, 3 tables. Under Review
☆ Look Around and Pay Attention: Multi-camera Point Tracking Reimagined with Transformer s
Bishoy Galoaa, Xiangyu Bai, Shayda Moezzi, Utsav Nandi, Sai Siddhartha Vivek Dhir Rangoju, Somaieh Amraee, Sarah Ostadabbas
This paper presents LAPA (Look Around and Pay Attention), a novel end-to-end transformer-based architecture for multi-camera point tracking that integrates appearance-based matching with geometric constraints. Traditional pipelines decouple detection, association, and tracking, leading to error propagation and temporal inconsistency in challenging scenarios. LAPA addresses these limitations by leveraging attention mechanisms to jointly reason across views and time, establishing soft correspondences through a cross-view attention mechanism enhanced with geometric priors. Instead of relying on classical triangulation, we construct 3D point representations via attention-weighted aggregation, inherently accommodating uncertainty and partial observations. Temporal consistency is further maintained through a transformer decoder that models long-range dependencies, preserving identities through extended occlusions. Extensive experiments on challenging datasets, including our newly created multi-camera (MC) versions of TAPVid-3D panoptic and PointOdyssey, demonstrate that our unified approach significantly outperforms existing methods, achieving 37.5% APD on TAPVid-3D-MC and 90.3% APD on PointOdyssey-MC, particularly excelling in scenarios with complex motions and occlusions. Code is available at https://github.com/ostadabbas/Look-Around-and-Pay-Attention-LAPA-
☆ OnSight Pathology: A real-time platform-agnostic computational pathology companion for histopathology
Jinzhen Hu, Kevin Faust, Parsa Babaei Zadeh, Adrienn Bourkas, Shane Eaton, Andrew Young, Anzar Alvi, Dimitrios George Oreopoulos, Ameesha Paliwal, Assem Saleh Alrumeh, Evelyn Rose Kamski-Hennekam, Phedias Diamandis
The microscopic examination of surgical tissue remains a cornerstone of disease classification but relies on subjective interpretations and access to highly specialized experts, which can compromise accuracy and clinical care. While emerging breakthroughs in artificial intelligence (AI) offer promise for automated histological analysis, the growing number of proprietary digital pathology solutions has created barriers to real-world deployment. To address these challenges, we introduce OnSight Pathology, a platform-agnostic computer vision software that uses continuous custom screen captures to provide real-time AI inferences to users as they review digital slide images. Accessible as a single, self-contained executable file (https://onsightpathology.github.io/ ), OnSight Pathology operates locally on consumer-grade personal computers without complex software integration, enabling cost-effective and secure deployment in research and clinical workflows. Here we demonstrate the utility of OnSight Pathology using over 2,500 publicly available whole slide images across different slide viewers, as well as cases from our clinical digital pathology setup. The software's robustness is highlighted across routine histopathological tasks, including the classification of common brain tumor types, mitosis detection, and the quantification of immunohistochemical stains. A built-in multi-modal chat assistant provides verifiable descriptions of images, free of rigid class labels, for added quality control. Lastly, we show compatibility with live microscope camera feeds, including from personal smartphones, offering potential for deployment in more analog, inter-operative, and telepathology settings. Together, we highlight how OnSight Pathology can deliver real-time AI inferences across a broad range of pathology pipelines, removing key barriers to the adoption of AI tools in histopathology.
☆ Beyond Flicker: Detecting Kinematic Inconsistencies for Generalizable Deepfake Video Detection
Alejandro Cobo, Roberto Valle, José Miguel Buenaposada, Luis Baumela
Generalizing deepfake detection to unseen manipulations remains a key challenge. A recent approach to tackle this issue is to train a network with pristine face images that have been manipulated with hand-crafted artifacts to extract more generalizable clues. While effective for static images, extending this to the video domain is an open issue. Existing methods model temporal artifacts as frame-to-frame instabilities, overlooking a key vulnerability: the violation of natural motion dependencies between different facial regions. In this paper, we propose a synthetic video generation method that creates training data with subtle kinematic inconsistencies. We train an autoencoder to decompose facial landmark configurations into motion bases. By manipulating these bases, we selectively break the natural correlations in facial movements and introduce these artifacts into pristine videos via face morphing. A network trained on our data learns to spot these sophisticated biomechanical flaws, achieving state-of-the-art generalization results on several popular benchmarks.
♻ ☆ DynamicVerse: A Physically-Aware Multimodal Framework for 4D Wo rl d Modeling
Kairun Wen, Yuzhi Huang, Runyu Chen, Hui Zheng, Yunlong Lin, Panwang Pan, Chenxin Li, Wenyan Cong, Jian Zhang, Junbin Lu, Chenguo Lin, Dilin Wang, Zhicheng Yan, Hongyu Xu, Justin Theiss, Yue Huang, Xinghao Ding, Rakesh Ranjan, Zhiwen Fan
Understanding the dynamic physical world, characterized by its evolving 3D structure, real-world motion, and semantic content with textual descriptions, is crucial for human-agent interaction and enables embodied agents to perceive and act within real environments with human-like capabilities. However, existing datasets are often derived from limited simulators or utilize traditional Structurefrom-Motion for up-to-scale annotation and offer limited descriptive captioning, which restricts the capacity of foundation models to accurately interpret real-world dynamics from monocular videos, commonly sourced from the internet. To bridge these gaps, we introduce DynamicVerse, a physical-scale, multimodal 4D world modeling framework for dynamic real-world video. We employ large vision, geometric, and multimodal models to interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic descriptive captions. By integrating window-based Bundle Adjustment with global optimization, our method converts long real-world video sequences into a comprehensive 4D multimodal format. DynamicVerse delivers a large-scale dataset consisting of 100K+ videos with 800K+ annotated masks and 10M+ frames from internet videos. Experimental evaluations on three benchmark tasks, namely video depth estimation, camera pose estimation, and camera intrinsics estimation, demonstrate that our 4D modeling achieves superior performance in capturing physical-scale measurements with greater global accuracy than existing methods.
♻ ☆ Rethinking the Learning Paradigm for Facial Expression Recognition
Weijie Wang, Bo Li, Nicu Sebe, Bruno Lepri
Due to the subjective crowdsourcing annotations and the inherent inter-class similarity of facial expressions, the real-world Facial Expression Recognition (FER) datasets usually exhibit ambiguous annotation. To simplify the learning paradigm, most previous methods convert ambiguous annotation results into precise one-hot annotations and train FER models in an end-to-end supervised manner. In this paper, we rethink the existing training paradigm and propose that it is better to use weakly supervised strategies to train FER models with original ambiguous annotation.
♻ ☆ SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation
Shuang Liang, Jing He, Chuanmeizhi Wang, Lejun Liao, Guo Zhang, Yingcong Chen, Yuan Yuan
Pre-trained diffusion models provide rich multi-scale latent features and are emerging as powerful vision backbones. While recent works such as Marigold and Lotus adapt diffusion priors for dense prediction with strong cross-domain generalization, their potential for structured outputs remains underexplored. In this paper, we propose SDPose, a fine-tuning framework built upon Stable Diffusion to fully exploit pre-trained diffusion priors for human pose estimation. First, rather than modifying cross-attention modules or introducing learnable embeddings, we directly predict keypoint heatmaps in the SD U-Net's image latent space to preserve the original generative priors. Second, we map these latent features into keypoint heatmaps through a lightweight convolutional pose head, which avoids disrupting the pre-trained backbone. Finally, to prevent overfitting and enhance out-of-distribution robustness, we incorporate an auxiliary RGB reconstruction branch that preserves domain-transferable generative semantics. To evaluate robustness under domain shift, we further construct COCO-OOD, a style-transferred variant of COCO with preserved annotations. With just one-fifth of the training schedule used by Sapiens on COCO, SDPose attains parity with Sapiens-1B/2B on the COCO validation set and establishes a new state of the art on the cross-domain benchmarks HumanArt and COCO-OOD. Extensive ablations highlight the importance of diffusion priors, RGB reconstruction, and multi-scale SD U-Net features for cross-domain generalization, and t-SNE analyses further explain SD's domain-invariant latent structure. We also show that SDPose serves as an effective zero-shot pose annotator for controllable image and video generation.
comment : 20 pages, 10 figures, 7 tables
♻ ☆ NVRC: Neural Video Representation Compression
Ho Man Kwan, Ge Gao, Fan Zhang, Andrew Gower, David Bull
Recent advances in implicit neural representation (INR)-based video coding have demonstrated its potential to compete with both conventional and other learning-based approaches. With INR methods, a neural network is trained to overfit a video sequence, with its parameters compressed to obtain a compact representation of the video content. However, although promising results have been achieved, the best INR-based methods are still out-performed by the latest standard codecs, such as VVC VTM, partially due to the simple model compression techniques employed. In this paper, rather than focusing on representation architectures as in many existing works, we propose a novel INR-based video compression framework, Neural Video Representation Compression (NVRC), targeting compression of the representation. Based on the novel entropy coding and quantization models proposed, NVRC, for the first time, is able to optimize an INR-based video codec in a fully end-to-end manner. To further minimize the additional bitrate overhead introduced by the entropy models, we have also proposed a new model compression framework for coding all the network, quantization and entropy model parameters hierarchically. Our experiments show that NVRC outperforms many conventional and learning-based benchmark codecs, with a 24% average coding gain over VVC VTM (Random Access) on the UVG dataset, measured in PSNR. As far as we are aware, this is the first time an INR-based video codec achieving such performance. The implementation of NVRC will be released.
♻ ☆ Sat2Flow: A Structure-Aware Diffusion Framework for Human Flow Generation from Satellite Imagery
Xiangxu Wang, Tianhong Zhao, Wei Tu, Bowen Zhang, Guanzhou Chen, Jinzhou Cao
Origin-Destination (OD) flow matrices are critical for urban mobility analysis, supporting traffic forecasting, infrastructure planning, and policy design. Existing methods face two key limitations: (1) reliance on costly auxiliary features (e.g., Points of Interest, socioeconomic statistics) with limited spatial coverage, and (2) fragility to spatial topology changes, where reordering urban regions disrupts the structural coherence of generated flows. We propose Sat2Flow, a structure-aware diffusion framework that generates structurally coherent OD flows using only satellite imagery. Our approach employs a multi-kernel encoder to capture diverse regional interactions and a permutation-aware diffusion process that maintains consistency across regional orderings. Through joint contrastive training linking satellite features with OD patterns and equivariant diffusion training enforcing structural invariance, Sat2Flow ensures topological robustness under arbitrary regional reindexing. Experiments on real-world datasets show that Sat2Flow outperforms physics-based and data-driven baselines in accuracy while preserving flow distributions and spatial structures under index permutations. Sat2Flow offers a globally scalable solution for OD flow generation in data-scarce environments, eliminating region-specific auxiliary data dependencies while maintaining structural robustness for reliable mobility modeling.
comment : 9 pages, 5 figures
♻ ☆ Neural Radiance and Gaze Fields for Visual Attention Modeling in 3D Environments
Andrei Chubarau, Yinan Wang, James J. Clark
We introduce Neural Radiance and Gaze Fields (NeRGs), a novel approach for representing visual attention in complex environments. Much like how Neural Radiance Fields (NeRFs) perform novel view synthesis, NeRGs reconstruct gaze patterns from arbitrary viewpoints, implicitly mapping visual attention to 3D surfaces. We achieve this by augmenting a standard NeRF with an additional network that models local egocentric gaze probability density, conditioned on scene geometry and observer position. The output of a NeRG is a rendered view of the scene alongside a pixel-wise salience map representing the conditional probability that a given observer fixates on visible surfaces. Unlike prior methods, our system is lightweight and enables visualization of gaze fields at interactive framerates. Moreover, NeRGs allow the observer perspective to be decoupled from the rendering camera and correctly account for gaze occlusion due to intervening geometry. We demonstrate the effectiveness of NeRGs using head pose from skeleton tracking as a proxy for gaze, employing our proposed gaze probes to aggregate noisy rays into robust probability density targets for supervision.
comment : 11 pages, 8 figures
♻ ☆ SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control
Yuxuan Mu, Ziyu Zhang, Yi Shi, Minami Matsumoto, Kotaro Imamura, Guy Tevet, Chuan Guo, Michael Taylor, Chang Shu, Pengcheng Xi, Xue Bin Peng
Data-driven motion priors that can guide agents toward producing naturalistic behaviors play a pivotal role in creating life-like virtual characters. Adversarial imitation learning has been a highly effective method for learning motion priors from reference motion data. However, adversarial priors, with few exceptions, need to be retrained for each new controller, thereby limiting their reusability and necessitating the retention of the reference motion data when training on downstream tasks. In this work, we present Score-Matching Motion Priors (SMP), which leverages pre-trained motion diffusion models and score distillation sampling (SDS) to create reusable task-agnostic motion priors. SMPs can be pre-trained on a motion dataset, independent of any control policy or task. Once trained, SMPs can be kept frozen and reused as general-purpose reward functions to train policies to produce naturalistic behaviors for downstream tasks. We show that a general motion prior trained on large-scale datasets can be repurposed into a variety of style-specific priors. Furthermore SMP can compose different styles to synthesize new styles not present in the original dataset. Our method produces high-quality motion comparable to state-of-the-art adversarial imitation learning methods through reusable and modular motion priors. We demonstrate the effectiveness of SMP across a diverse suite of control tasks with physically simulated humanoid characters. Video demo available at https://youtu.be/ravlZJteS20
comment : 14 pages, 9 figures
♻ ☆ Defense That Attacks: How Robust Models Become Better Attackers
Mohamed Awad, Mahmoud Akrm, Walid Gomaa
Deep learning has achieved great success in computer vision, but remains vulnerable to adversarial attacks. Adversarial training is the leading defense designed to improve model robustness. However, its effect on the transferability of attacks is underexplored. In this work, we ask whether adversarial training unintentionally increases the transferability of adversarial examples. To answer this, we trained a diverse zoo of 36 models, including CNNs and ViTs, and conducted comprehensive transferability experiments. Our results reveal a clear paradox: adversarially trained (AT) models produce perturbations that transfer more effectively than those from standard models, which introduce a new ecosystem risk. To enable reproducibility and further study, we release all models, code, and experimental scripts. Furthermore, we argue that robustness evaluations should assess not only the resistance of a model to transferred attacks but also its propensity to produce transferable adversarial examples.
♻ ☆ Efficient Transferable Optimal Transport via Min-Sliced Transport Plans
Xinran Liu, Elaheh Akbari, Rocio Diaz Martin, Navid NaderiAlizadeh, Soheil Kolouri
Optimal Transport (OT) offers a powerful framework for finding correspondences between distributions and addressing matching and alignment problems in various areas of computer vision, including shape analysis, image generation, and multimodal tasks. The computation cost of OT, however, hinders its scalability. Slice-based transport plans have recently shown promise for reducing the computational cost by leveraging the closed-form solutions of 1D OT problems. These methods optimize a one-dimensional projection (slice) to obtain a conditional transport plan that minimizes the transport cost in the ambient space. While efficient, these methods leave open the question of whether learned optimal slicers can transfer to new distribution pairs under distributional shift. Understanding this transferability is crucial in settings with evolving data or repeated OT computations across closely related distributions. In this paper, we study the min-Sliced Transport Plan (min-STP) framework and investigate the transferability of optimized slicers: can a slicer trained on one distribution pair yield effective transport plans for new, unseen pairs? Theoretically, we show that optimized slicers remain close under slight perturbations of the data distributions, enabling efficient transfer across related tasks. To further improve scalability, we introduce a minibatch formulation of min-STP and provide statistical guarantees on its accuracy. Empirically, we demonstrate that the transferable min-STP achieves strong one-shot matching performance and facilitates amortized training for point cloud alignment and flow-based generative modeling.
♻ ☆ SpecGen: Neural Spectral BRDF Generation via Spectral-Spatial Tri-plane Aggregation
Zhenyu Jin, Wenjie Li, Zhanyu Ma, Heng Guo
Synthesizing spectral images across different wavelengths is essential for photorealistic rendering. Unlike conventional spectral uplifting methods that convert RGB images into spectral ones, we introduce SpecGen, a novel method that generates spectral bidirectional reflectance distribution functions (BRDFs) from a single RGB image of a sphere. This enables spectral image rendering under arbitrary illuminations and shapes covered by the corresponding material. A key challenge in spectral BRDF generation is the scarcity of measured spectral BRDF data. To address this, we propose the Spectral-Spatial Tri-plane Aggregation (SSTA) network, which models reflectance responses across wavelengths and incident-outgoing directions, allowing the training strategy to leverage abundant RGB BRDF data to enhance spectral BRDF generation. Experiments show that our method accurately reconstructs spectral BRDFs from limited spectral data and surpasses state-of-the-art methods in hyperspectral image reconstruction, achieving an improvement of 8 dB in PSNR. Codes and data will be released upon acceptance.
♻ ☆ A Machine Learning-Driven Solution for Denoising Inertial Confinement Fusion Images
Asya Y. Akkus, Bradley T. Wolfe, Pinghan Chu, Chengkun Huang, Chris S. Campbell, Mariana Alvarado Alvarez, Petr Volegov, David Fittinghoff, Robert Reinovsky, Zhehui Wang
Neutron imaging is essential for diagnosing and optimizing inertial confinement fusion implosions at the National Ignition Facility. Due to the required 10-micrometer resolution, however, neutron image require image reconstruction using iterative algorithms. For low-yield sources, the images may be degraded by various types of noise. Gaussian and Poisson noise often coexist within one image, obscuring fine details and blurring the edges where the source information is encoded. Traditional denoising techniques, such as filtering and thresholding, can inadvertently alter critical features or reshape the noise statistics, potentially impacting the ultimate fidelity of the iterative image reconstruction pipeline. However, recent advances in synthetic data production and machine learning have opened new opportunities to address these challenges. In this study, we present an unsupervised autoencoder with a Cohen-Daubechies- Feauveau (CDF 97) wavelet transform in the latent space, designed to suppress for mixed Gaussian-Poisson noise while preserving essential image features. The network successfully denoises neutron imaging data. Benchmarking against both simulated and experimental NIF datasets demonstrates that our approach achieves lower reconstruction error and superior edge preservation compared to conventional filtering methods such as Block-matching and 3D filtering (BM3D). By validating the effectiveness of unsupervised learning for denoising neutron images, this study establishes a critical first step towards fully AI-driven, end-to-end reconstruction frameworks for ICF diagnostics.
♻ ☆ Exploring the Potentials of Spiking Neural Networks for Image Deraining AAAI2026
Shuang Chen, Tomas Krajnik, Farshad Arvin, Amir Atapour-Abarghouei
Biologically plausible and energy-efficient frameworks such as Spiking Neural Networks (SNNs) have not been sufficiently explored in low-level vision tasks. Taking image deraining as an example, this study addresses the representation of the inherent high-pass characteristics of spiking neurons, specifically in image deraining and innovatively proposes the Visual LIF (VLIF) neuron, overcoming the obstacle of lacking spatial contextual understanding present in traditional spiking neurons. To tackle the limitation of frequency-domain saturation inherent in conventional spiking neurons, we leverage the proposed VLIF to introduce the Spiking Decomposition and Enhancement Module and the lightweight Spiking Multi-scale Unit for hierarchical multi-scale representation learning. Extensive experiments across five benchmark deraining datasets demonstrate that our approach significantly outperforms state-of-the-art SNN-based deraining methods, achieving this superior performance with only 13\% of their energy consumption. These findings establish a solid foundation for deploying SNNs in high-performance, energy-efficient low-level vision tasks.
comment : Accepted By AAAI2026
♻ ☆ BitMark: Watermarking Bitwise Autoregressive Image Generative Models NeurIPS 2025
Louis Kerner, Michel Meintz, Bihe Zhao, Franziska Boenisch, Adam Dziedzic
State-of-the-art text-to-image models generate photorealistic images at an unprecedented speed. This work focuses on models that operate in a bitwise autoregressive manner over a discrete set of tokens that is practically infinite in size. However, their impressive generative power comes with a growing risk: as their outputs increasingly populate the Internet, they are likely to be scraped and reused as training data-potentially by the very same models. This phenomenon has been shown to lead to model collapse, where repeated training on generated content, especially from the models' own previous versions, causes a gradual degradation in performance. A promising mitigation strategy is watermarking, which embeds human-imperceptible yet detectable signals into generated images-enabling the identification of generated content. In this work, we introduce BitMark, a robust bitwise watermarking framework. Our method embeds a watermark directly at the bit level of the token stream during the image generation process. Our bitwise watermark subtly influences the bits to preserve visual fidelity and generation speed while remaining robust against a spectrum of removal techniques. Furthermore, it exhibits high radioactivity, i.e., when watermarked generated images are used to train another image generative model, this second model's outputs will also carry the watermark. The radioactive traces remain detectable even when only fine-tuning diffusion or image autoregressive models on images watermarked with our BitMark. Overall, our approach provides a principled step toward preventing model collapse in image generative models by enabling reliable detection of generated outputs. The code is available at https://github.com/sprintml/BitMark.
comment : Accepted as a Conference Paper at NeurIPS 2025
♻ ☆ Accuracy-Robustness Trade Off via Spiking Neural Network Gradient Sparsity Trail
Luu Trong Nhan, Luu Trung Duong, Pham Ngoc Nam, Truong Cong Thang
Spiking Neural Networks (SNNs) have attracted growing interest in both computational neuroscience and artificial intelligence, primarily due to their inherent energy efficiency and compact memory footprint. However, achieving adversarial robustness in SNNs, (particularly for vision-related tasks) remains a nascent and underexplored challenge. Recent studies have proposed leveraging sparse gradients as a form of regularization to enhance robustness against adversarial perturbations. In this work, we present a surprising finding: under specific architectural configurations, SNNs exhibit natural gradient sparsity and can achieve state-of-the-art adversarial defense performance without the need for any explicit regularization. Further analysis reveals a trade-off between robustness and generalization: while sparse gradients contribute to improved adversarial resilience, they can impair the model's ability to generalize; conversely, denser gradients support better generalization but increase vulnerability to attacks. Our findings offer new insights into the dual role of gradient sparsity in SNN training.
comment : Work under peer-review
♻ ☆ A Tractable Two-Step Linear Mixing Model Solved with Second-Order Optimization for Spectral Unmixing under Variability
Xander Haijen, Bikram Koirala, Xuanwen Tao, Paul Scheunders
In this paper, we propose a Two-Step Linear Mixing Model (2LMM) that bridges the gap between model complexity and computational tractability. The model achieves this by introducing two distinct scaling steps: an endmember scaling step across the image, and another for pixel-wise scaling. We show that this model leads to only a mildly non-convex optimization problem, which we solve with an optimization algorithm that incorporates second-order information. To the authors' knowledge, this work represents the first application of second-order optimization techniques to solve a spectral unmixing problem that models endmember variability. Our method is highly robust, as it requires virtually no hyperparameter tuning and can therefore be used easily and quickly in a wide range of unmixing tasks. We show through extensive experiments on both simulated and real data that the new model is competitive and in some cases superior to the state of the art in unmixing. The model also performs very well in challenging scenarios, such as blind unmixing.
comment : This work has been submitted to the IEEE for possible publication
♻ ☆ D$^{2}$-VPR: A Parameter-efficient Visual-foundation-model-based Visual Place Recognition Method via Knowledge Distillation and Deformable Aggregation
Zheyuan Zhang, Jiwei Zhang, Boyu Zhou, Linzhimeng Duan, Hong Chen
Visual Place Recognition (VPR) aims to determine the geographic location of a query image by retrieving its most visually similar counterpart from a geo-tagged reference database. Recently, the emergence of the powerful visual foundation model, DINOv2, trained in a self-supervised manner on massive datasets, has significantly improved VPR performance. This improvement stems from DINOv2's exceptional feature generalization capabilities but is often accompanied by increased model complexity and computational overhead that impede deployment on resource-constrained devices. To address this challenge, we propose $D^{2}$-VPR, a $D$istillation- and $D$eformable-based framework that retains the strong feature extraction capabilities of visual foundation models while significantly reducing model parameters and achieving a more favorable performance-efficiency trade-off. Specifically, first, we employ a two-stage training strategy that integrates knowledge distillation and fine-tuning. Additionally, we introduce a Distillation Recovery Module (DRM) to better align the feature spaces between the teacher and student models, thereby minimizing knowledge transfer losses to the greatest extent possible. Second, we design a Top-Down-attention-based Deformable Aggregator (TDDA) that leverages global semantic features to dynamically and adaptively adjust the Regions of Interest (ROI) used for aggregation, thereby improving adaptability to irregular structures. Extensive experiments demonstrate that our method achieves competitive performance compared to state-of-the-art approaches. Meanwhile, it reduces the parameter count by approximately 64.2% and FLOPs by about 62.6% (compared to CricaVPR).Code is available at https://github.com/tony19980810/D2VPR.
♻ ☆ GNSS-Inertial State Initialization Using Inter-Epoch Baseline Residuals
Samuel Cerezo, Javier Civera
Initializing the state of a sensorized platform can be challenging, as a limited set of measurements often provide low-informative constraints that are in addition highly non-linear. This may lead to poor initial estimates that may converge to local minima during subsequent non-linear optimization. We propose an adaptive GNSS-inertial initialization strategy that delays the incorporation of global GNSS constraints until they become sufficiently informative. In the initial stage, our method leverages inter-epoch baseline vector residuals between consecutive GNSS fixes to mitigate inertial drift. To determine when to activate global constraints, we introduce a general criterion based on the evolution of the Hessian matrix's singular values, effectively quantifying system observability. Experiments on EuRoC, GVINS and MARS-LVIG datasets show that our approach consistently outperforms the naive strategy of fusing all measurements from the outset, yielding more accurate and robust initializations.
comment : 8 pages, 7 figures, accepted to RA-L
♻ ☆ DGFusion: Depth-Guided Sensor Fusion for Robust Semantic Perception
Tim Broedermannn, Christos Sakaridis, Luigi Piccinelli, Wim Abbeloos, Luc Van Gool
Robust semantic perception for autonomous vehicles relies on effectively combining multiple sensors with complementary strengths and weaknesses. State-of-the-art sensor fusion approaches to semantic perception often treat sensor data uniformly across the spatial extent of the input, which hinders performance when faced with challenging conditions. By contrast, we propose a novel depth-guided multimodal fusion method that upgrades condition-aware fusion by integrating depth information. Our network, DGFusion, poses multimodal segmentation as a multi-task problem, utilizing the lidar measurements, which are typically available in outdoor sensor suites, both as one of the model's inputs and as ground truth for learning depth. Our corresponding auxiliary depth head helps to learn depth-aware features, which are encoded into spatially varying local depth tokens that condition our attentive cross-modal fusion. Together with a global condition token, these local depth tokens dynamically adapt sensor fusion to the spatially varying reliability of each sensor across the scene, which largely depends on depth. In addition, we propose a robust loss for our depth, which is essential for learning from lidar inputs that are typically sparse and noisy in adverse conditions. Our method achieves state-of-the-art panoptic and semantic segmentation performance on the challenging MUSES and DeLiVER datasets. Code and models will be available at https://github.com/timbroed/DGFusion
comment : Code and models will be available at https://github.com/timbroed/DGFusion
♻ ☆ Delving into Dynamic Scene Cue-Consistency for Robust 3D Multi-Object Tracking
Haonan Zhang, Xinyao Wang, Boxi Wu, Tu Zheng, Wang Yunhua, Zheng Yang
3D multi-object tracking is a critical and challenging task in the field of autonomous driving. A common paradigm relies on modeling individual object motion, e.g., Kalman filters, to predict trajectories. While effective in simple scenarios, this approach often struggles in crowded environments or with inaccurate detections, as it overlooks the rich geometric relationships between objects. This highlights the need to leverage spatial cues. However, existing geometry-aware methods can be susceptible to interference from irrelevant objects, leading to ambiguous features and incorrect associations. To address this, we propose focusing on cue-consistency: identifying and matching stable spatial patterns over time. We introduce the Dynamic Scene Cue-Consistency Tracker (DSC-Track) to implement this principle. Firstly, we design a unified spatiotemporal encoder using Point Pair Features (PPF) to learn discriminative trajectory embeddings while suppressing interference. Secondly, our cue-consistency transformer module explicitly aligns consistent feature representations between historical tracks and current detections. Finally, a dynamic update mechanism preserves salient spatiotemporal information for stable online tracking. Extensive experiments on the nuScenes and Waymo Open Datasets validate the effectiveness and robustness of our approach. On the nuScenes benchmark, for instance, our method achieves state-of-the-art performance, reaching 73.2% and 70.3% AMOTA on the validation and test sets, respectively.
♻ ☆ Language-Driven Object-Oriented Two-Stage Method for Scene Graph Anticipation
Xiaomeng Zhu, Changwei Wang, Haozhe Wang, Xinyu Liu, Fangzhen Lin
A scene graph is a structured representation of objects and their spatio-temporal relationships in dynamic scenes. Scene Graph Anticipation (SGA) involves predicting future scene graphs from video clips, enabling applications in intelligent surveillance and human-machine collaboration. While recent SGA approaches excel at leveraging visual evidence, long-horizon forecasting fundamentally depends on semantic priors and commonsense temporal regularities that are challenging to extract purely from visual features. To explicitly model these semantic dynamics, we propose Linguistic Scene Graph Anticipation (LSGA), a linguistic formulation of SGA that performs temporal relational reasoning over sequences of textualized scene graphs, with visual scene-graph detection handled by a modular front-end when operating on video. Building on this formulation, we introduce Object-Oriented Two-Stage Method (OOTSM), a language-based framework that anticipates object-set dynamics and forecasts object-centric relation trajectories with temporal consistency regularization, and we evaluate it on a dedicated benchmark constructed from Action Genome annotations. Extensive experiments show that compact fine-tuned language models with up to 3B parameters consistently outperform strong zero- and one-shot API baselines, including GPT-4o, GPT-4o-mini, and DeepSeek-V3, under matched textual inputs and context windows. When coupled with off-the-shelf visual scene-graph generators, the resulting multimodal system achieves substantial improvements on video-based SGA, boosting long-horizon mR@50 by up to 21.9\% over strong visual SGA baselines.
♻ ☆ You Point, I Learn: Online Adaptation of Interactive Segmentation Models for Handling Distribution Shifts in Medical Imaging
Wentian Xu, Ziyun Liang, Harry Anthony, Yasin Ibrahim, Felix Cohen, Guang Yang, Konstantinos Kamnitsas
Interactive segmentation uses real-time user inputs, such as mouse clicks, to iteratively refine model predictions. Although not originally designed to address distribution shifts, this paradigm naturally lends itself to such challenges. In medical imaging, where distribution shifts are common, interactive methods can use user inputs to guide models towards improved predictions. Moreover, once a model is deployed, user corrections can be used to adapt the network parameters to the new data distribution, mitigating distribution shift. Based on these insights, we aim to develop a practical, effective method for improving the adaptive capabilities of interactive segmentation models to new data distributions in medical imaging. Firstly, we found that strengthening the model's responsiveness to clicks is important for the initial training process. Moreover, we show that by treating the post-interaction user-refined model output as pseudo-ground-truth, we can design a lean, practical online adaptation method that enables a model to learn effectively across sequential test images. The framework includes two components: (i) a Post-Interaction adaptation process, updating the model after the user has completed interactive refinement of an image, and (ii) a Mid-Interaction adaptation process, updating incrementally after each click. Both processes include a Click-Centered Gaussian loss that strengthens the model's reaction to clicks and enhances focus on user-guided, clinically relevant regions. Experiments on 5 fundus and 4 brain-MRI databases show that our approach consistently outperforms existing methods under diverse distribution shifts, including unseen imaging modalities and pathologies. Code and pretrained models will be released upon publication.
♻ ☆ InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue
Wenwen Tong, Hewei Guo, Dongchuan Ran, Jiangnan Chen, Jiefan Lu, Kaibin Wang, Keqiang Li, Xiaoxu Zhu, Jiakui Li, Kehan Li, Xueheng Li, Lumin Li, Chenxu Guo, Jiasheng Zhou, Jiandong Chen, Xianye Wu, Jiahao Wang, Silei Wu, Lei Chen, Hanming Deng, Yuxuan Song, Dinghao Zhou, Guiping Zhong, Ken Zheng, Shiyin Kang, Lewei Lu
We introduce InteractiveOmni, a unified and open-source omni-modal large language model for audio-visual multi-turn interaction, ranging from 4B to 8B parameters, designed to lead the field of lightweight models by offering comprehensive omni-modal understanding and speech generation capabilities. To achieve this, we integrate the vision encoder, audio encoder, large language model, and speech decoder into a unified model for understanding and generation tasks. We design a multi-stage training strategy to ensure robust cross-modal capabilities, including pre-training for omni-modal understanding, followed by post-training with speech conversation and audio-visual interaction. To enable human-like long-term conversational ability, we meticulously curate a multi-turn training dataset that enhances the model's ability to handle complex and multi-turn interactions. To effectively evaluate the multi-turn memory and speech interaction capabilities, we construct the multi-modal multi-turn memory benchmark and the multi-turn speech interaction benchmark. Experiments demonstrate that InteractiveOmni significantly outperforms leading open-source models and provides a more intelligent multi-turn audio-visual experience, particularly in its long-term memory capabilities. Notably, InteractiveOmni-4B is comparable to the much larger model like Qwen2.5-Omni-7B on general benchmarks, and it can retain 97% of the performance of the InteractiveOmni-8B while utilizing only 50% of the model size. Achieving state-of-the-art results against similarly sized models across image, audio, video understanding, and speech generation tasks, InteractiveOmni is an accessible, open-source foundation for next-generation intelligent interactive systems.
♻ ☆ GT23D-Bench: A Comprehensive General Text-to-3D Generation Benchmark
Xiao Cai, Sitong Su, Jingkuan Song, Pengpeng Zeng, Ji Zhang, Qinhong Du, Mengqi Li, Heng Tao Shen, Lianli Gao
Text-to-3D (T23D) generation has emerged as a crucial visual generation task, aiming at synthesizing 3D content from textual descriptions. Studies of this task are currently shifting from per-scene T23D, which requires optimization of the model for every content generated, to General T23D (GT23D), which requires only one pre-trained model to generate different content without re-optimization, for more generalized and efficient 3D generation. Despite notable advancements, GT23D is severely bottlenecked by two interconnected challenges: the lack of high-quality, large-scale training data and the prevalence of evaluation metrics that overlook intrinsic 3D properties. Existing datasets often suffer from incomplete annotations, noisy organization, and inconsistent quality, while current evaluations rely heavily on 2D image-text similarity or scoring, failing to thoroughly assess 3D geometric integrity and semantic relevance. To address these fundamental gaps, we introduce GT23D-Bench, the first comprehensive benchmark specifically designed for GT23D training and evaluation. We first construct a high-quality dataset of 400K 3D assets, featuring diverse visual annotations (70M+ visual samples) and multi-granularity hierarchical captions (1M+ descriptions) to foster robust semantic learning. Second, we propose a comprehensive evaluation suite with 10 metrics assessing both text-3D alignment and 3D visual quality at multiple levels. Crucially, we demonstrate through rigorous experiments that our proposed metrics exhibit significantly higher correlation with human judgment compared to existing methods. Our in-depth analysis of eight leading GT23D models using this benchmark provides the community with critical insights into current model capabilities and their shared failure modes. GT23D-Bench will be publicly available to facilitate rigorous and reproducible research.
♻ ☆ HybridWo rl dSim: A Scalable and Controllable High-fidelity Simulator for Autonomous Driving
Qiang Li, Yingwenqi Jiang, Tuoxi Li, Duyu Chen, Xiang Feng, Yucheng Ao, Shangyue Liu, Xingchen Yu, Youcheng Cai, Yumeng Liu, Yuexin Ma, Xin Hu, Li Liu, Yu Zhang, Linkun Xu, Bingtao Gao, Xueyuan Wang, Shuchang Zhou, Xianming Liu, Ligang Liu
Realistic and controllable simulation is critical for advancing end-to-end autonomous driving, yet existing approaches often struggle to support novel view synthesis under large viewpoint changes or to ensure geometric consistency. We introduce HybridWorldSim, a hybrid simulation framework that integrates multi-traversal neural reconstruction for static backgrounds with generative modeling for dynamic agents. This unified design addresses key limitations of previous methods, enabling the creation of diverse and high-fidelity driving scenarios with reliable visual and spatial consistency. To facilitate robust benchmarking, we further release a new multi-traversal dataset MIRROR that captures a wide range of routes and environmental conditions across different cities. Extensive experiments demonstrate that HybridWorldSim surpasses previous state-of-the-art methods, providing a practical and scalable solution for high-fidelity simulation and a valuable resource for research and development in autonomous driving.
comment : Project page: https://hybridworldsim.github.io/
♻ ☆ TransUNet-GradCAM: A Hybrid Transformer -U-Net with Self-Attention and Explainable Visualizations for Foot Ulcer Segmentation
Akwasi Asare, Mary Sagoe, Justice Williams Asare, Stephen Edward Moore
Automated segmentation of diabetic foot ulcers (DFUs) plays a critical role in clinical diagnosis, therapeutic planning, and longitudinal wound monitoring. However, this task remains challenging due to the heterogeneous appearance, irregular morphology, and complex backgrounds associated with ulcer regions in clinical photographs. Traditional convolutional neural networks (CNNs), such as U-Net, provide strong localization capabilities but struggle to model long-range spatial dependencies due to their inherently limited receptive fields. To address this, we employ the TransUNet architecture, a hybrid framework that integrates the global attention mechanism of Vision Transformers (ViTs) into the U-Net structure. This combination allows the model to extract global contextual features while maintaining fine-grained spatial resolution. We trained the model on the public Foot Ulcer Segmentation Challenge (FUSeg) dataset using a robust augmentation pipeline and a hybrid loss function to mitigate class imbalance. On the validation set, the model achieved a Dice Similarity Coefficient (F1-score) of 0.8799 using an optimized threshold of 0.4389. To ensure clinical transparency, we integrated Grad-CAM visualizations to highlight model focus areas. Furthermore, a clinical utility analysis demonstrated a strong correlation (Pearson r = 0.9631) between predicted and ground-truth wound areas. These outcomes demonstrate that our approach effectively integrates global and local feature extraction, offering a reliable, effective, and explainable solution for automated foot ulcer assessment.
♻ ☆ STT-GS: Sample-Then-Transmit Edge Gaussian Splatting with Joint Client Selection and Power Control
Zhen Li, Xibin Jin, Guoliang Li, Shuai Wang, Miaowen Wen, Huseyin Arslan, Derrick Wing Kwan Ng, Chengzhong Xu
Edge Gaussian splatting (EGS), which aggregates data from distributed clients (e.g., drones) and trains a global GS model at the edge (e.g., ground server), is an emerging paradigm for scene reconstruction in low-altitude economy. Unlike traditional edge resource management methods that emphasize communication throughput or general-purpose learning performance, EGS explicitly aims to maximize the GS qualities, rendering existing approaches inapplicable. To address this problem, this paper formulates a novel GS-oriented objective function that distinguishes the heterogeneous view contributions of different clients. However, evaluating this function in turn requires clients' images, leading to a causality dilemma. To this end, this paper further proposes a sample-then-transmit EGS (or STT-GS for short) strategy, which first samples a subset of images as pilot data from each client for loss prediction. Based on the first-stage evaluation, communication resources are then prioritized towards more valuable clients. To achieve efficient sampling, a feature-domain clustering (FDC) scheme is proposed to select the most representative data and pilot transmission time minimization (PTTM) is adopted to reduce the pilot overhead. Subsequently, we develop a joint client selection and power control (JCSPC) framework to maximize the GS-oriented function under communication resource constraints. Despite the nonconvexity of the problem, we propose a low-complexity efficient solution based on the penalty alternating majorization minimization (PAMM) algorithm. Experiments reveal that the proposed scheme significantly outperforms existing benchmarks on real-world datasets. The GS-oriented objective can be accurately predicted with low sampling ratios (e.g., 10%), and our method achieves an excellent tradeoff between view contributions and communication costs.
♻ ☆ LoRA Patching: Exposing the Fragility of Proactive Defenses against Deepfakes
Zuomin Qu, Yimao Guo, Qianyue Hu, Wei Lu
Deepfakes pose significant societal risks, motivating the development of proactive defenses that embed adversarial perturbations in facial images to prevent manipulation. However, in this paper, we show that these preemptive defenses often lack robustness and reliability. We propose a novel approach, Low-Rank Adaptation (LoRA) patching, which injects a plug-and-play LoRA patch into Deepfake generators to bypass state-of-the-art defenses. A learnable gating mechanism adaptively controls the effect of the LoRA patch and prevents gradient explosions during fine-tuning. We also introduce a Multi-Modal Feature Alignment (MMFA) loss, encouraging the features of adversarial outputs to align with those of the desired outputs at the semantic level. Beyond bypassing, we present defensive LoRA patching, embedding visible warnings in the outputs as a complementary solution to mitigate this newly identified security vulnerability. With only 1,000 facial examples and a single epoch of fine-tuning, LoRA patching successfully defeats multiple proactive defenses. These results reveal a critical weakness in current paradigms and underscore the need for more robust Deepfake defense strategies. Our code is available at https://github.com/ZOMIN28/LoRA-Patching.
♻ ☆ Test-time Correction: An Online 3D Detection System via Visual Prompt ing
Hanxue Zhang, Zetong Yang, Yanan Sun, Li Chen, Fei Xia, Fatma Güney, Hongyang Li
This paper introduces Test-time Correction (TTC), an online 3D detection system designed to rectify test-time errors using various auxiliary feedback, aiming to enhance the safety of deployed autonomous driving systems. Unlike conventional offline 3D detectors that remain fixed during inference, TTC enables immediate online error correction without retraining, allowing autonomous vehicles to adapt to new scenarios and reduce deployment risks. To achieve this, we equip existing 3D detectors with an Online Adapter (OA) module -- a prompt-driven query generator for real-time correction. At the core of OA module are visual prompts: image-based descriptions of objects of interest derived from auxiliary feedback such as mismatches with 2D detections, road descriptions, or user clicks. These visual prompts, collected from risky objects during inference, are maintained in a visual prompt buffer to enable continuous correction in future frames. By leveraging this mechanism, TTC consistently detects risky objects, achieving reliable, adaptive, and versatile driving autonomy. Extensive experiments show that TTC significantly improves instant error rectification over frozen 3D detectors, even under limited labels, zero-shot settings, and adverse conditions. We hope this work inspires future research on post-deployment online rectification systems for autonomous driving.
♻ ☆ Assessing the Alignment of Popular CNNs to the Brain for Valence Appraisal
Laurent Mertens, Elahe' Yargholi, Laura Van Hove, Hans Op de Beeck, Jan Van den Stock, Joost Vennekens
Convolutional Neural Networks (CNNs) are a popular type of computer model that have proven their worth in many computer vision tasks. Moreover, they form an interesting study object for the field of psychology, with shown correspondences between the workings of CNNs and the human brain. However, these correspondences have so far mostly been studied in the context of general visual perception. In contrast, this paper explores to what extent this correspondence also holds for a more complex brain process, namely social cognition. To this end, we assess the alignment between popular CNN architectures and both human behavioral and fMRI data for image valence appraisal through a correlation analysis. We show that for this task CNNs struggle to go beyond simple visual processing, and do not seem to reflect higher-order brain processing. Furthermore, we present Object2Brain, a novel framework that combines GradCAM and object detection at the CNN-filter level with the aforementioned correlation analysis to study the influence of different object classes on the CNN-to-human correlations. Despite similar correlation trends, different CNN architectures are shown to display different object class sensitivities.
comment : 12 pages, 5 figures
♻ ☆ Universal Multi-Domain Translation via Diffusion Routers
Duc Kieu, Kien Do, Tuan Hoang, Thao Minh Le, Tung Kieu, Dang Nguyen, Thin Nguyen
Multi-domain translation (MDT) aims to learn translations between multiple domains, yet existing approaches either require fully aligned tuples or can only handle domain pairs seen in training, limiting their practicality and excluding many cross-domain mappings. We introduce universal MDT (UMDT), a generalization of MDT that seeks to translate between any pair of $K$ domains using only $K-1$ paired datasets with a central domain. To tackle this problem, we propose Diffusion Router (DR), a unified diffusion-based framework that models all central$\leftrightarrow$non-central translations with a single noise predictor conditioned on the source and target domain labels. DR enables indirect non-central translations by routing through the central domain. We further introduce a novel scalable learning strategy with a variational-bound objective and an efficient Tweedie refinement procedure to support direct non-central mappings. Through evaluation on three large-scale UMDT benchmarks, DR achieves state-of-the-art results for both indirect and direct translations, while lowering sampling cost and unlocking novel tasks such as sketch$\leftrightarrow$segmentation. These results establish DR as a scalable and versatile framework for universal translation across multiple domains.
♻ ☆ PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformer s Inference
Jiarui Fang, Jinzhe Pan, Aoyu Li, Xibo Sun, Jiannan Wang
This paper presents PipeFusion, an innovative parallel methodology to tackle the high latency issues associated with generating high-resolution images using diffusion transformers (DiTs) models. PipeFusion partitions images into patches and the model layers across multiple GPUs. It employs a patch-level pipeline parallel strategy to orchestrate communication and computation efficiently. By capitalizing on the high similarity between inputs from successive diffusion steps, PipeFusion reuses one-step stale feature maps to provide context for the current pipeline step. This approach notably reduces communication costs compared to existing DiTs inference parallelism, including tensor parallel, sequence parallel and DistriFusion. PipeFusion enhances memory efficiency through parameter distribution across devices, ideal for large DiTs like Flux.1. Experimental results demonstrate that PipeFusion achieves state-of-the-art performance on 8$\times$L40 PCIe GPUs for Pixart, Stable-Diffusion 3, and Flux.1 models. Our source code is available at https://github.com/xdit-project/xDiT.
♻ ☆ MoBGS: Motion Deblurring Dynamic 3D Gaussian Splatting for Blurry Monocular Video AAAI 2026
Minh-Quan Viet Bui, Jongmin Park, Juan Luis Gonzalez Bello, Jaeho Moon, Jihyong Oh, Munchurl Kim
We present MoBGS, a novel motion deblurring 3D Gaussian Splatting (3DGS) framework capable of reconstructing sharp and high-quality novel spatio-temporal views from blurry monocular videos in an end-to-end manner. Existing dynamic novel view synthesis (NVS) methods are highly sensitive to motion blur in casually captured videos, resulting in significant degradation of rendering quality. While recent approaches address motion-blurred inputs for NVS, they primarily focus on static scene reconstruction and lack dedicated motion modeling for dynamic objects. To overcome these limitations, our MoBGS introduces a novel Blur-adaptive Latent Camera Estimation (BLCE) method using a proposed Blur-adaptive Neural Ordinary Differential Equation (ODE) solver for effective latent camera trajectory estimation, improving global camera motion deblurring. In addition, we propose a Latent Camera-induced Exposure Estimation (LCEE) method to ensure consistent deblurring of both a global camera and local object motions. Extensive experiments on the Stereo Blur dataset and real-world blurry videos show that our MoBGS significantly outperforms the very recent methods, achieving state-of-the-art performance for dynamic NVS under motion blur.
comment : This paper has been accepted to AAAI 2026. The first two authors contributed equally to this work (equal contribution). The last two authors are co-corresponding authors. Please visit our project page at https://kaist-viclab.github.io/mobgs-site/
♻ ☆ MambaScope: Coarse-to-Fine Scoping for Efficient Vision Mamba
Shanhui Liu, Rui Xu, Yunke Wang
Vision Mamba has emerged as a promising and efficient alternative to Vision Transformers, yet its efficiency remains fundamentally constrained by the number of input tokens. Existing token reduction approaches typically adopt token pruning or merging to reduce computation. However, they inherently lead to information loss as they discard or compress token representations. This problem is further exacerbated when the same fine-grained token processing is uniformly applied across all images regardless of visual complexity. We observe that not all inputs require fine-grained processing: simple images can be effectively handled at a coarse resolution, while only complex ones require refinement. Based on this insight, we propose MambaScope, an adaptive framework for efficient inference for Vision Mamba. MambaScope first performs coarse-grained inference by dividing the input image into large patches, significantly reducing token length and computation. When the model's prediction confidence is low, selected regions are re-processed at a finer resolution to recover essential visual details with minimal additional cost. This dynamic resolution assignment strategy allows MambaScope to allocate computation adaptively according to image complexity, achieving efficient processing without compromising accuracy. Experiments across various vision tasks demonstrate that MambaScope outperforms both the baseline Vision Mamba and state-of-the-art token reduction techniques in terms of accuracy and efficiency.
♻ ☆ DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes ICLR 2025
Hengwei Bian, Lingdong Kong, Haozhe Xie, Liang Pan, Yu Qiao, Ziwei Liu
Urban scene generation has been developing rapidly recently. However, existing methods primarily focus on generating static and single-frame scenes, overlooking the inherently dynamic nature of real-world driving environments. In this work, we introduce DynamicCity, a novel 4D occupancy generation framework capable of generating large-scale, high-quality dynamic 4D scenes with semantics. DynamicCity mainly consists of two key models. 1) A VAE model for learning HexPlane as the compact 4D representation. Instead of using naive averaging operations, DynamicCity employs a novel Projection Module to effectively compress 4D features into six 2D feature maps for HexPlane construction, which significantly enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in parallel, which improves both network training efficiency and reconstruction accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded Rollout Operation is proposed to reorganize all six feature planes of the HexPlane as a squared 2D feature map. In particular, various conditions could be introduced in the diffusion or sampling process, supporting versatile 4D generation applications, such as trajectory- and command-driven generation, inpainting, and layout-conditioned generation. Extensive experiments on the CarlaSC and Waymo datasets demonstrate that DynamicCity significantly outperforms existing state-of-the-art 4D occupancy generation methods across multiple metrics. The code and models have been released to facilitate future research.
comment : ICLR 2025 Spotlight; 35 pages, 18 figures, 15 tables; Project Page at https://dynamic-city.github.io/
♻ ☆ Can VLMs Detect and Localize Fine-Grained AI-Edited Images?
Zhen Sun, Ziyi Zhang, Zeren Luo, Zhiyuan Zhong, Zeyang Sha, Tianshuo Cong, Zheng Li, Shiwen Cui, Weiqiang Wang, Jiaheng Wei, Xinlei He, Qi Li, Qian Wang
Fine-grained detection and localization of localized image edits is crucial for assessing content authenticity, especially as modern diffusion models and image editors can produce highly realistic manipulations. However, this problem faces three key challenges: (1) most AIGC detectors produce only a global real-or-fake label without indicating where edits occur; (2) traditional computer vision methods for edit localization typically rely on costly pixel-level annotations; and (3) there is no large-scale, modern benchmark specifically targeting edited-image detection. To address these gaps, we develop an automated data-generation pipeline and construct FragFake, a large-scale benchmark of AI-edited images spanning multiple source datasets, diverse editing models, and several common edit types. Building on FragFake, we are the first to systematically study vision language models (VLMs) for edited-image classification and edited-region localization. Our experiments show that pretrained VLMs, including GPT4o, perform poorly on this task, whereas fine-tuned models such as Qwen2.5-VL achieve high accuracy and substantially higher object precision across all settings. We further explore GRPO-based RLVR training, which yields modest metric gains while improving the interpretability of model outputs. Ablation and transfer analyses reveal how data balancing, training size, LoRA rank, and training domain affect performance, and highlight both the potential and the limitations of cross-editor and cross-dataset generalization. We anticipate that this work will establish a solid foundation to facilitate and inspire subsequent research endeavors in the domain of multimodal content authenticity.
comment : 14pages,19 figures
♻ ☆ Context Cascade Compression: Exploring the Upper Limits of Text Compression
Fanfan Liu, Haibo Qiu
Million-level token inputs in long-context tasks pose significant computational and memory challenges for Large Language Models (LLMs). Recently, DeepSeek-OCR conducted research into the feasibility of Contexts Optical Compression and achieved preliminary results. Inspired by this, we introduce Context Cascade Compression C3 to explore the upper limits of text compression. Our method cascades two LLMs of different sizes to handle the compression and decoding tasks. Specifically, a small LLM, acting as the first stage, performs text compression by condensing a long context into a set of latent tokens (e.g., 32 or 64 in length), achieving a high ratio of text tokens to latent tokens. A large LLM, as the second stage, then executes the decoding task on this compressed context. Experiments show that at a 20x compression ratio (where the number of text tokens is 20 times the number of latent tokens), our model achieves 98% decoding accuracy, compared to approximately 60% for DeepSeek-OCR. When we further increase the compression ratio to 40x, the accuracy is maintained at around 93%. This indicates that in the domain of context compression, C3 Compression demonstrates superior performance and feasibility over optical character compression. C3 uses a simpler, pure-text pipeline that ignores factors like layout, color, and information loss from a visual encoder. This also suggests a potential upper bound for compression ratios in future work on optical character compression, OCR, and related fields. Codes and model weights are publicly accessible at https://github.com/liufanfanlff/C3-Context-Cascade-Compression
♻ ☆ TabletopGen: Instance-Level Interactive 3D Tabletop Scene Generation from Text or Single Image
Ziqian Wang, Yonghao He, Licheng Yang, Wei Zou, Hongxuan Ma, Liu Liu, Wei Sui, Yuxin Guo, Hu Su
Generating high-fidelity, physically interactive 3D simulated tabletop scenes is essential for embodied AI--especially for robotic manipulation policy learning and data synthesis. However, current text- or image-driven 3D scene generation methods mainly focus on large-scale scenes, struggling to capture the high-density layouts and complex spatial relations that characterize tabletop scenes. To address these challenges, we propose TabletopGen, a training-free, fully automatic framework that generates diverse, instance-level interactive 3D tabletop scenes. TabletopGen accepts a reference image as input, which can be synthesized by a text-to-image model to enhance scene diversity. We then perform instance segmentation and completion on the reference to obtain per-instance images. Each instance is reconstructed into a 3D model followed by canonical coordinate alignment. The aligned 3D models then undergo pose and scale estimation before being assembled into a collision-free, simulation-ready tabletop scene. A key component of our framework is a novel pose and scale alignment approach that decouples the complex spatial reasoning into two stages: a Differentiable Rotation Optimizer for precise rotation recovery and a Top-view Spatial Alignment mechanism for robust translation and scale estimation, enabling accurate 3D reconstruction from 2D reference. Extensive experiments and user studies show that TabletopGen achieves state-of-the-art performance, markedly surpassing existing methods in visual fidelity, layout accuracy, and physical plausibility, capable of generating realistic tabletop scenes with rich stylistic and spatial diversity. Our code will be publicly available.
comment : Project page: https://d-robotics-ai-lab.github.io/TabletopGen.project/
♻ ☆ OneThinker: All-in-one Reasoning Model for Image and Video
Kaituo Feng, Manyuan Zhang, Hongyu Li, Kaixuan Fan, Shuang Chen, Yilei Jiang, Dian Zheng, Peiwen Sun, Yiyuan Zhang, Haoze Sun, Yan Feng, Peng Pei, Xunliang Cai, Xiangyu Yue
Reinforcement learning (RL) has recently achieved remarkable success in eliciting visual reasoning within Multimodal Large Language Models (MLLMs). However, existing approaches typically train separate models for different tasks and treat image and video reasoning as disjoint domains. This results in limited scalability toward a multimodal reasoning generalist, which restricts practical versatility and hinders potential knowledge sharing across tasks and modalities. To this end, we propose OneThinker, an all-in-one reasoning model that unifies image and video understanding across diverse fundamental visual tasks, including question answering, captioning, spatial and temporal grounding, tracking, and segmentation. To achieve this, we construct the OneThinker-600k training corpus covering all these tasks and employ commercial models for CoT annotation, resulting in OneThinker-SFT-340k for SFT cold start. Furthermore, we propose EMA-GRPO to handle reward heterogeneity in multi-task RL by tracking task-wise moving averages of reward standard deviations for balanced optimization. Extensive experiments on diverse visual benchmarks show that OneThinker delivers strong performance on 31 benchmarks, across 10 fundamental visual understanding tasks. Moreover, it exhibits effective knowledge transfer between certain tasks and preliminary zero-shot generalization ability, marking a step toward a unified multimodal reasoning generalist. All code, model, and data are released.
comment : Project page: https://github.com/tulerfeng/OneThinker
♻ ☆ MagicView: Multi-View Consistent Id entity Customization via Priors-Guided In- Context Learning
Hengjia Li, Jianjin Xu, Keli Cheng, Lei Wang, Ning Bi, Boxi Wu, Fernando De la Torre, Deng Cai
Recent advances in personalized generative models have demonstrated impressive capabilities in producing identity-consistent images of the same individual across diverse scenes. However, most existing methods lack explicit viewpoint control and fail to ensure multi-view consistency of generated identities. To address this limitation, we present MagicView, a lightweight adaptation framework that equips existing generative models with multi-view generation capability through 3D priors-guided in-context learning. While prior studies have shown that in-context learning preserves identity consistency across grid samples, its effectiveness in multi-view settings remains unexplored. Building upon this insight, we conduct an in-depth analysis of the multi-view in-context learning ability, and design a conditioning architecture that leverages 3D priors to activate this capability for multi-view consistent identity customization. On the other hand, acquiring robust multi-view capability typically requires large-scale multi-dimensional datasets, which makes incorporating multi-view contextual learning under limited data regimes prone to textual controllability degradation. To address this issue, we introduce a novel Semantic Correspondence Alignment loss, which effectively preserves semantic alignment while maintaining multi-view consistency. Extensive experiments demonstrate that MagicView substantially outperforms recent baselines in multi-view consistency, text alignment, identity similarity, and visual quality, achieving strong results with only 100 multi-view training samples.
♻ ☆ LargeAD: Large-Scale Cross-Sensor Data Pretrain ing for Autonomous Driving
Lingdong Kong, Xiang Xu, Youquan Liu, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, Ziwei Liu
Recent advancements in vision foundation models (VFMs) have revolutionized visual perception in 2D, yet their potential for 3D scene understanding, particularly in autonomous driving applications, remains underexplored. In this paper, we introduce LargeAD, a versatile and scalable framework designed for large-scale 3D pretraining across diverse real-world driving datasets. Our framework leverages VFMs to extract semantically rich superpixels from 2D images, which are aligned with LiDAR point clouds to generate high-quality contrastive samples. This alignment facilitates cross-modal representation learning, enhancing the semantic consistency between 2D and 3D data. We introduce several key innovations: (i) VFM-driven superpixel generation for detailed semantic representation, (ii) a VFM-assisted contrastive learning strategy to align multimodal features, (iii) superpoint temporal consistency to maintain stable representations across time, and (iv) multi-source data pretraining to generalize across various LiDAR configurations. Our approach achieves substantial gains over state-of-the-art methods in linear probing and fine-tuning for LiDAR-based segmentation and object detection. Extensive experiments on 11 large-scale multi-sensor datasets highlight our superior performance, demonstrating adaptability, efficiency, and robustness in real-world autonomous driving scenarios.
comment : IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
♻ ☆ Score Distillation of Flow Matching Models
Mingyuan Zhou, Yi Gu, Huangjie Zheng, Liangchen Song, Guande He, Yizhe Zhang, Wenze Hu, Yinfei Yang
Diffusion models achieve high-quality image generation but are limited by slow iterative sampling. Distillation methods alleviate this by enabling one- or few-step generation. Flow matching, originally introduced as a distinct framework, has since been shown to be theoretically equivalent to diffusion under Gaussian assumptions, raising the question of whether distillation techniques such as score distillation transfer directly. We provide a simple derivation -- based on Bayes' rule and conditional expectations -- that unifies Gaussian diffusion and flow matching without relying on ODE/SDE formulations. Building on this view, we extend Score identity Distillation (SiD) to pretrained text-to-image flow-matching models, including SANA, SD3-Medium, SD3.5-Medium/Large, and FLUX.1-dev, all with DiT backbones. Experiments show that, with only modest flow-matching- and DiT-specific adjustments, SiD works out of the box across these models, in both data-free and data-aided settings, without requiring teacher finetuning or architectural changes. This provides the first systematic evidence that score distillation applies broadly to text-to-image flow matching models, resolving prior concerns about stability and soundness and unifying acceleration techniques across diffusion- and flow-based generators. A project page is available at https://yigu1008.github.io/SiD-DiT.
♻ ☆ VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety
Shruti Palaskar, Leon Gatys, Mona Abdelrahman, Mar Jacobo, Larry Lindsey, Rutika Moharir, Gunnar Lund, Yang Xu, Navid Shiee, Jeffrey Bigham, Charles Maalouf, Joseph Yitan Cheng
Safety evaluation of multimodal foundation models often treats vision and language inputs separately, missing risks from joint interpretation where benign content becomes harmful in combination. Existing approaches also fail to distinguish clearly unsafe content from borderline cases, leading to problematic over-blocking or under-refusal of genuinely harmful content. We present Vision Language Safety Understanding (VLSU), a comprehensive framework to systematically evaluate multimodal safety through fine-grained severity classification and combinatorial analysis across 17 distinct safety patterns. Using a multi-stage pipeline with real-world images and human annotation, we construct a large-scale benchmark of 8,187 samples spanning 15 harm categories. Our evaluation of eleven state-of-the-art models reveals systematic joint understanding failures: while models achieve 90%-plus accuracy on clear unimodal safety signals, performance degrades substantially to 20-55% when joint image-text reasoning is required to determine the safety label. Most critically, 34% of errors in joint image-text safety classification occur despite correct classification of the individual modalities, further demonstrating absent compositional reasoning capabilities. Additionally, we find that models struggle to balance refusing unsafe content while still responding to borderline cases that deserve engagement. For example, we find that instruction framing can reduce the over-blocking rate on borderline content from 62.4% to 10.4% in Gemini-1.5, but only at the cost of under-refusing on unsafe content with refusal rate dropping from 90.8% to 53.9%. Overall, our framework exposes weaknesses in joint image-text understanding and alignment gaps in current models, and provides a critical test bed to enable the next milestones in research on robust vision-language safety.
comment : 10 pages, 5 figures, 4 tables, detailed appendix. Under review
♻ ☆ Automatic Labelling for Low-Light Pedestrian Detection
Dimitrios Bouzoulas, Eerik Alamikkotervo, Risto Ojala
Pedestrian detection in RGB images is a key task in pedestrian safety, as the most common sensor in autonomous vehicles and advanced driver assistance systems is the RGB camera. A challenge in RGB pedestrian detection, that does not appear to have large public datasets, is low-light conditions. As a solution, in this research, we propose an automated infrared-RGB labeling pipeline. The proposed pipeline consists of 1) Infrared detection, where a fine-tuned model for infrared pedestrian detection is used 2) Label transfer process from the infrared detections to their RGB counterparts 3) Training object detection models using the generated labels for low-light RGB pedestrian detection. The research was performed using the KAIST dataset. For the evaluation, object detection models were trained on the generated autolabels and ground truth labels. When compared on a previously unseen image sequence, the results showed that the models trained on generated labels outperformed the ones trained on ground-truth labels in 6 out of 9 cases for the mAP@50 and mAP@50-95 metrics. The source code for this research is available at https://github.com/BouzoulasDimitrios/IR-RGB-Automated-LowLight-Pedestrian-Labeling
♻ ☆ SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism NeurIPS 2025
Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, Heng Tao Shen
By incorporating visual inputs, Multimodal Large Language Models (MLLMs) extend LLMs to support visual reasoning. However, this integration also introduces new vulnerabilities, making MLLMs susceptible to multimodal jailbreak attacks and hindering their safe deployment.Existing defense methods, including Image-to-Text Translation, Safe Prompting, and Multimodal Safety Tuning, attempt to address this by aligning multimodal inputs with LLMs' built-in safeguards.Yet, they fall short in uncovering root causes of multimodal vulnerabilities, particularly how harmful multimodal tokens trigger jailbreak in MLLMs? Consequently, they remain vulnerable to text-driven multimodal jailbreaks, often exhibiting overdefensive behaviors and imposing heavy training overhead.To bridge this gap, we present an comprehensive analysis of where, how and which harmful multimodal tokens bypass safeguards in MLLMs. Surprisingly, we find that less than 1% tokens in early-middle layers are responsible for inducing unsafe behaviors, highlighting the potential of precisely removing a small subset of harmful tokens, without requiring safety tuning, can still effectively improve safety against jailbreaks. Motivated by this, we propose Safe Prune-then-Restore (SafePTR), an training-free defense framework that selectively prunes harmful tokens at vulnerable layers while restoring benign features at subsequent layers.Without incurring additional computational overhead, SafePTR significantly enhances the safety of MLLMs while preserving efficiency. Extensive evaluations across three MLLMs and five benchmarks demonstrate SafePTR's state-of-the-art performance in mitigating jailbreak risks without compromising utility.
comment : Accepted by NeurIPS 2025
♻ ☆ 3D and 4D Wo rl d Modeling: A Survey
Lingdong Kong, Wesley Yang, Jianbiao Mei, Youquan Liu, Ao Liang, Dekai Zhu, Dongyue Lu, Wei Yin, Xiaotao Hu, Mingkai Jia, Junyuan Deng, Kaiwen Zhang, Yang Wu, Tianyi Yan, Shenyuan Gao, Song Wang, Linfeng Li, Liang Pan, Yong Liu, Jianke Zhu, Wei Tsang Ooi, Steven C. H. Hoi, Ziwei Liu
World modeling has become a cornerstone in AI research, enabling agents to understand, represent, and predict the dynamic environments they inhabit. While prior work largely emphasizes generative methods for 2D image and video data, they overlook the rapidly growing body of work that leverages native 3D and 4D representations such as RGB-D imagery, occupancy grids, and LiDAR point clouds for large-scale scene modeling. At the same time, the absence of a standardized definition and taxonomy for ``world models'' has led to fragmented and sometimes inconsistent claims in the literature. This survey addresses these gaps by presenting the first comprehensive review explicitly dedicated to 3D and 4D world modeling and generation. We establish precise definitions, introduce a structured taxonomy spanning video-based (VideoGen), occupancy-based (OccGen), and LiDAR-based (LiDARGen) approaches, and systematically summarize datasets and evaluation metrics tailored to 3D/4D settings. We further discuss practical applications, identify open challenges, and highlight promising research directions, aiming to provide a coherent and foundational reference for advancing the field. A systematic summary of existing literature is available at https://github.com/worldbench/awesome-3d-4d-world-models
comment : Survey; 50 pages, 10 figures, 14 tables; GitHub Repo at https://github.com/worldbench/awesome-3d-4d-world-models
♻ ☆ SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splatting
Mengjiao Ma, Qi Ma, Yue Li, Jiahuan Cheng, Runyi Yang, Bin Ren, Nikola Popovic, Mingqiang Wei, Nicu Sebe, Luc Van Gool, Theo Gevers, Martin R. Oswald, Danda Pani Paudel
3D Gaussian Splatting (3DGS) serves as a highly performant and efficient encoding of scene geometry, appearance, and semantics. Moreover, grounding language in 3D scenes has proven to be an effective strategy for 3D scene understanding. Current Language Gaussian Splatting line of work fall into three main groups: (i) per-scene optimization-based, (ii) per-scene optimization-free, and (iii) generalizable approach. However, most of them are evaluated only on rendered 2D views of a handful of scenes and viewpoints close to the training views, limiting ability and insight into holistic 3D understanding. To address this gap, we propose the first large-scale benchmark that systematically assesses these three groups of methods directly in 3D space, evaluating on 1060 scenes across three indoor datasets and one outdoor dataset. Benchmark results demonstrate a clear advantage of the generalizable paradigm, particularly in relaxing the scene-specific limitation, enabling fast feed-forward inference on novel scenes, and achieving superior segmentation performance. We further introduce GaussianWorld-49K a carefully curated 3DGS dataset comprising around 49K diverse indoor and outdoor scenes obtained from multiple sources, with which we demonstrate the generalizable approach could harness strong data priors. Our codes, benchmark, and datasets are public to accelerate research in generalizable 3DGS scene understanding.
comment : 15 pages, codes, data and benchmark are released
♻ ☆ ActiveInitSplat: How Active Image Selection Helps Gaussian Splatting
Konstantinos D. Polyzos, Athanasios Bacharis, Saketh Madhuvarasu, Nikos Papanikolopoulos, Tara Javidi
Gaussian splatting (GS) along with its extensions and variants provides outstanding performance in real-time scene rendering while meeting reduced storage demands and computational efficiency. While the selection of 2D images capturing the scene of interest is crucial for the proper initialization and training of GS, hence markedly affecting the rendering performance, prior works rely on passively and typically densely selected 2D images. In contrast, this paper proposes `ActiveInitSplat', a novel framework for active selection of training images for proper initialization and training of GS. ActiveInitSplat relies on density and occupancy criteria of the resultant 3D scene representation from the selected 2D images, to ensure that the latter are captured from diverse viewpoints leading to better scene coverage and that the initialized Gaussian functions are well aligned with the actual 3D structure. Numerical tests on well-known simulated and real environments demonstrate the merits of ActiveInitSplat resulting in significant GS rendering performance improvement over passive GS baselines in both dense- and sparse-view settings, in the widely adopted LPIPS, SSIM, and PSNR metrics.
♻ ☆ Pan-LUT: Efficient Pan-sharpening via Learnable Look-Up Tables
Zhongnan Cai, Yingying Wang, Hui Zheng, Panwang Pan, ZiXu Lin, Ge Meng, Chenxin Li, Chunming He, Jiaxin Xie, Yunlong Lin, Junbin Lu, Yue Huang, Xinghao Ding
Recently, deep learning-based pan-sharpening algorithms have achieved notable advancements over traditional methods. However, deep learning-based methods incur substantial computational overhead during inference, especially with large images. This excessive computational demand limits the applicability of these methods in real-world scenarios, particularly in the absence of dedicated computing devices such as GPUs and TPUs. To address these challenges, we propose Pan-LUT, a novel learnable look-up table (LUT) framework for pan-sharpening that strikes a balance between performance and computational efficiency for large remote sensing images. Our method makes it possible to process 15K*15K remote sensing images on a 24GB GPU. To finely control the spectral transformation, we devise the PAN-guided look-up table (PGLUT) for channel-wise spectral mapping. To effectively capture fine-grained spatial details, we introduce the spatial details look-up table (SDLUT). Furthermore, to adaptively aggregate channel information for generating high-resolution multispectral images, we design an adaptive output look-up table (AOLUT). Our model contains fewer than 700K parameters and processes a 9K*9K image in under 1 ms using one RTX 2080 Ti GPU, demonstrating significantly faster performance compared to other methods. Experiments reveal that Pan-LUT efficiently processes large remote sensing images in a lightweight manner, bridging the gap to real-world applications. Furthermore, our model surpasses SOTA methods in full-resolution scenes under real-world conditions, highlighting its effectiveness and efficiency.
♻ ☆ SATORI-R1: Incentivizing Multimodal Reasoning through Explicit Visual Anchoring
Chuming Shen, Wei Wei, Xiaoye Qu, Yu Cheng
DeepSeek-R1 has demonstrated powerful reasoning capabilities in the text domain through stable reinforcement learning (RL). Recently, in the multimodal domain, works have begun to directly apply RL to generate R1-like free-form reasoning for Visual Question Answering (VQA) tasks. However, multimodal tasks share an intrinsically different nature from textual tasks, which heavily rely on the understanding of the input image to solve the problem. Therefore, such free-form reasoning faces two critical limitations in the VQA task: (1) Extended reasoning chains diffuse visual focus away from task-critical regions, degrading answer accuracy. (2) Unverifiable intermediate steps amplify policy-gradient variance and computational costs overhead. To address these issues, in this paper, we introduce SATORI ($\textbf{S}patially$ $\textbf{A}nchored$ $\textbf{T}ask$ $\textbf{O}ptimization$ with $\textbf{R}e\textbf{I}nforcement$ Learning), which decomposes VQA into three verifiable stages, including global image captioning, region localization, and answer prediction, each supplying explicit reward signals. Furthermore, we also introduce VQA-Verify, a 12k dataset annotated with answer-aligned captions and bounding-boxes to facilitate training. Experiments demonstrate consistent performance improvements across seven VQA benchmarks, achieving up to $15.7\%$ improvement in accuracy in accuracy compared to the R1-like baseline. Our analysis of the attention map confirms enhanced focus on critical regions, which brings improvements in accuracy. Our code is available at https://github.com/justairr/SATORI-R1.
comment : 21 pages, 8 figures
♻ ☆ GS4: Generalizable Sparse Splatting Semantic SLAM
Mingqi Jiang, Chanho Kim, Chen Ziwen, Li Fuxin
Traditional SLAM algorithms excel at camera tracking, but typically produce incomplete and low-resolution maps that are not tightly integrated with semantics prediction. Recent work integrates Gaussian Splatting (GS) into SLAM to enable dense, photorealistic 3D mapping, yet existing GS-based SLAM methods require per-scene optimization that is slow and consumes an excessive number of Gaussians. We present GS4, the first generalizable GS-based semantic SLAM system. Compared with prior approaches, GS4 runs 10x faster, uses 10x fewer Gaussians, and achieves state-of-the-art performance across color, depth, semantic mapping and camera tracking. From an RGB-D video stream, GS4 incrementally builds and updates a set of 3D Gaussians using a feed-forward network. First, the Gaussian Prediction Model estimates a sparse set of Gaussian parameters from input frame, which integrates both color and semantic prediction with the same backbone. Then, the Gaussian Refinement Network merges new Gaussians with the existing set while avoiding redundancy. Finally, when significant pose changes are detected, we perform only 1-5 iterations of joint Gaussian-pose optimization to correct drift, remove floaters, and further improve tracking accuracy. Experiments on the real-world ScanNet and ScanNet++ benchmarks demonstrate state-of-the-art semantic SLAM performance, with strong generalization capability shown through zero-shot transfer to the NYUv2 and TUM RGB-D datasets.
comment : 15 pages, 6 figures
♻ ☆ Guard Me If You Know Me: Protecting Specific Face-Id entity from Deepfakes
Kaiqing Lin, Zhiyuan Yan, Ke-Yue Zhang, Li Hao, Yue Zhou, Yuzhen Lin, Weixiang Li, Taiping Yao, Shouhong Ding, Bin Li
Securing personal identity against deepfake attacks is increasingly critical in the digital age, especially for celebrities and political figures whose faces are easily accessible and frequently targeted. Most existing deepfake detection methods focus on general-purpose scenarios and often ignore the valuable prior knowledge of known facial identities, e.g., "VIP individuals" whose authentic facial data are already available. In this paper, we propose \textbf{VIPGuard}, a unified multimodal framework designed to capture fine-grained and comprehensive facial representations of a given identity, compare them against potentially fake or similar-looking faces, and reason over these comparisons to make accurate and explainable predictions. Specifically, our framework consists of three main stages. First, fine-tune a multimodal large language model (MLLM) to learn detailed and structural facial attributes. Second, we perform identity-level discriminative learning to enable the model to distinguish subtle differences between highly similar faces, including real and fake variations. Finally, we introduce user-specific customization, where we model the unique characteristics of the target face identity and perform semantic reasoning via MLLM to enable personalized and explainable deepfake detection. Our framework shows clear advantages over previous detection works, where traditional detectors mainly rely on low-level visual cues and provide no human-understandable explanations, while other MLLM-based models often lack a detailed understanding of specific face identities. To facilitate the evaluation of our method, we built a comprehensive identity-aware benchmark called \textbf{VIPBench} for personalized deepfake detection, involving the latest 7 face-swapping and 7 entire face synthesis techniques for generation. The code is available at https://github.com/KQL11/VIPGuard .
♻ ☆ Margin-aware Preference Optimization for Aligning Diffusion Models without Reference AAAI 2026
Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thorne, Jongheon Jeong
Modern preference alignment methods, such as DPO, rely on divergence regularization to a reference model for training stability-but this creates a fundamental problem we call "reference mismatch." In this paper, we investigate the negative impacts of reference mismatch in aligning text-to-image (T2I) diffusion models, showing that larger reference mismatch hinders effective adaptation given the same amount of data, e.g., as when learning new artistic styles, or personalizing to specific objects. We demonstrate this phenomenon across text-to-image (T2I) diffusion models and introduce margin-aware preference optimization (MaPO), a reference-agnostic approach that breaks free from this constraint. By directly optimizing the likelihood margin between preferred and dispreferred outputs under the Bradley-Terry model without anchoring to a reference, MaPO transforms diverse T2I tasks into unified pairwise preference optimization. We validate MaPO's versatility across five challenging domains: (1) safe generation, (2) style adaptation, (3) cultural representation, (4) personalization, and (5) general preference alignment. Our results reveal that MaPO's advantage grows dramatically with reference mismatch severity, outperforming both DPO and specialized methods like DreamBooth while reducing training time by 15%. MaPO thus emerges as a versatile and memory-efficient method for generic T2I adaptation tasks.
comment : Accepted to AAAI 2026 Main Technical Track
♻ ☆ MERIT: Multilingual Semantic Retrieval with Inte rl eaved Multi-Condition Query NeurIPS 2025
Wei Chow, Yuan Gao, Linfeng Li, Xian Wang, Qi Xu, Hang Song, Lingdong Kong, Ran Zhou, Yi Zeng, Yidong Cai, Botian Jiang, Shilin Xu, Jiajun Zhang, Minghui Qiu, Xiangtai Li, Tianshu Yang, Siliang Tang, Juncheng Li
Semantic retrieval is crucial for modern applications yet remains underexplored in current research. Existing datasets are limited to single languages, single images, or singular retrieval conditions, often failing to fully exploit the expressive capacity of visual information as evidenced by maintained performance when images are replaced with captions. However, practical retrieval scenarios frequently involve interleaved multi-condition queries with multiple images. Hence, this paper introduces MERIT, the first multilingual dataset for interleaved multi-condition semantic retrieval, comprising 320,000 queries with 135,000 products in 5 languages, covering 7 distinct product categories. Extensive experiments on MERIT identify existing models's limitation: focusing solely on global semantic information while neglecting specific conditional elements in queries. Consequently, we propose Coral, a novel fine-tuning framework that adapts pre-trained MLLMs by integrating embedding reconstruction to preserve fine-grained conditional elements and contrastive learning to extract comprehensive global semantics. Experiments demonstrate that Coral achieves a 45.9% performance improvement over conventional approaches on MERIT, with strong generalization capabilities validated across 8 established retrieval benchmarks. Collectively, our contributions - a novel dataset, identification of critical limitations in existing approaches, and an innovative fine-tuning framework - establish a foundation for future research in interleaved multi-condition semantic retrieval.
comment : NeurIPS 2025; Project Page, Code, and Dataset at: https://merit-2025.github.io/
♻ ☆ MACS: Measurement-Aware Consistency Sampling for Inverse Problems
Amirreza Tanevardi, Pooria Abbas Rad Moghadam, Seyed Mohammad Eshtehardian, Sajjad Amini, Babak Khalaj
Diffusion models have emerged as powerful generative priors for solving inverse imaging problems. However, their practical deployment is hindered by the substantial computational cost of slow, multi-step sampling. Although Consistency Models (CMs) address this limitation by enabling high-quality generation in only one or a few steps, their direct application to inverse problems has remained largely unexplored. This paper introduces a modified consistency sampling framework specifically designed for inverse problems. The proposed approach regulates the sampler's stochasticity through a measurement-consistency mechanism that leverages the degradation operator, thereby enforcing fidelity to the observed data while preserving the computational efficiency of consistency-based generation. Comprehensive experiments on the Fashion-MNIST and LSUN Bedroom datasets demonstrate consistent improvements across both perceptual and pixel-level metrics, including the Fréchet Inception Distance (FID), Kernel Inception Distance (KID), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM), compared with baseline consistency and diffusion-based sampling methods. The proposed method achieves competitive or superior reconstruction quality with only a small number of sampling steps.
comment : 10 pages, 4 figures, This work has been submitted to the IEEE for possible publication
♻ ☆ Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols
Xianchao Zeng, Xinyu Zhou, Youcheng Li, Jiayou Shi, Tianle Li, Liangming Chen, Lei Ren, Yong-Lu Li
Vision-Language-Action (VLA) models have recently achieved remarkable progress in robotic manipulation, yet they remain limited in failure diagnosis and learning from failures. Additionally, existing failure datasets are mostly generated programmatically in simulation, which limits their generalization to the real world. In light of these, we introduce ViFailback, a framework designed to diagnose robotic manipulation failures and provide both textual and visual correction guidance. Our framework utilizes explicit visual symbols to enhance annotation efficiency. We further release the ViFailback dataset, a large-scale collection of 58,126 Visual Question Answering (VQA) pairs along with their corresponding 5,202 real-world manipulation trajectories. Based on the dataset, we establish ViFailback-Bench, a benchmark of 11 fine-grained VQA tasks designed to assess the failure diagnosis and correction abilities of Vision-Language Models (VLMs), featuring ViFailback-Bench Lite for closed-ended and ViFailback-Bench Hard for open-ended evaluation. To demonstrate the effectiveness of our framework, we built the ViFailback-8B VLM, which not only achieves significant overall performance improvement on ViFailback-Bench but also generates visual symbols for corrective action guidance. Finally, by integrating ViFailback-8B with a VLA model, we conduct real-world robotic experiments demonstrating its ability to assist the VLA model in recovering from failures. Project Website: https://x1nyuzhou.github.io/vifailback.github.io/
♻ ☆ A Novel Attention-Augmented Wavelet YOLO System for Real-time Brain Vessel Segmentation on Transcranial Color-coded Doppler
Wenxuan Zhang, Shuai Li, Xinyi Wang, Yu Sun, Hongyu Kang, Pui Yuk Chryste Wan, Jing Qin, Yuanpeng Zhang, Yong-Ping Zheng, Sai-Kit Lam
The Circle of Willis (CoW), vital for ensuring consistent blood flow to the brain, is closely linked to ischemic stroke. Accurate assessment of the CoW is important for identifying individuals at risk and guiding appropriate clinical management. Among existing imaging methods, Transcranial Color-coded Doppler (TCCD) offers unique advantages due to its radiation-free nature, affordability, and accessibility. However, reliable TCCD assessments depend heavily on operator expertise for identifying anatomical landmarks and performing accurate angle correction, which limits its widespread adoption. To address this challenge, we propose an AI-powered, real-time CoW auto-segmentation system capable of efficiently capturing cerebral arteries. No prior studies have explored AI-driven cerebrovascular segmentation using TCCD. In this work, we introduce a novel Attention-Augmented Wavelet YOLO (AAW-YOLO) network tailored for TCCD data, designed to provide real-time guidance for brain vessel segmentation in the CoW. We prospectively collected TCCD data comprising 738 annotated frames and 3,419 labeled artery instances to establish a high-quality dataset for model training and evaluation. The proposed AAW-YOLO demonstrated strong performance in segmenting both ipsilateral and contralateral CoW vessels, achieving an average Dice score of 0.901, IoU of 0.823, precision of 0.882, recall of 0.926, and mAP of 0.953, with a per-frame inference speed of 14.199 ms. This system offers a practical solution to reduce reliance on operator experience in TCCD-based cerebrovascular screening, with potential applications in routine clinical workflows and resource-constrained settings. Future research will explore bilateral modeling and larger-scale validation.
♻ ☆ Differentiable, Bit-shifting, and Scalable Quantization without training neural network from scratch
Zia Badar
Quantization of neural networks provides benefits of inference in less compute and memory requirements. Previous work in quantization lack two important aspects which this work provides. First almost all previous work in quantization used a non-differentiable approach and for learning; the derivative is usually set manually in backpropogation which make the learning ability of algorithm questionable, our approach is not just differentiable, we also provide proof of convergence of our approach to the optimal neural network. Second previous work in shift/logrithmic quantization either have avoided activation quantization along with weight quantization or achieved less accuracy. Learning logrithmic quantize values of form $2^n$ requires the quantization function can scale to more than 1 bit quantization which is another benifit of our quantization that it provides $n$ bits quantization as well. Our approach when tested with image classification task using imagenet dataset, resnet18 and weight quantization only achieves less than 1 percent accuracy compared to full precision accuracy while taking only 15 epochs to train using shift bit quantization and achieves comparable to SOTA approaches accuracy in both weight and activation quantization using shift bit quantization in 15 training epochs with slightly higher(only higher cpu instructions) inference cost compared to 1 bit quantization(without logrithmic quantization) and not requiring any higher precision multiplication.
♻ ☆ Vision -Based Mistake Analysis in Procedural Activities: A Review of Advances and Challenges
Konstantinos Bacharidis, Antonis A. Argyros
Mistake analysis in procedural activities is a critical area of research with applications spanning industrial automation, physical rehabilitation, education and human-robot collaboration. This paper reviews vision-based methods for detecting and predicting mistakes in structured tasks, focusing on procedural and executional errors. By leveraging advancements in computer vision, including action recognition, anticipation and activity understanding, vision-based systems can identify deviations in task execution, such as incorrect sequencing, use of improper techniques, or timing errors. We explore the challenges posed by intra-class variability, viewpoint differences and compositional activity structures, which complicate mistake detection. Additionally, we provide a comprehensive overview of existing datasets, evaluation metrics and state-of-the-art methods, categorizing approaches based on their use of procedural structure, supervision levels and learning strategies. Open challenges, such as distinguishing permissible variations from true mistakes and modeling error propagation are discussed alongside future directions, including neuro-symbolic reasoning and counterfactual state modeling. This work aims to establish a unified perspective on vision-based mistake analysis in procedural activities, highlighting its potential to enhance safety, efficiency and task performance across diverse domains.
comment : 23pages, 6 figures, 2 tables
♻ ☆ IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web
Hongcheng Guo, Wei Zhang, Junhao Chen, Yaonan Gu, Jian Yang, Junjia Du, Shaosheng Cao, Binyuan Hui, Tianyu Liu, Jianxin Ma, Chang Zhou, Zhoujun Li
Recently advancements in large multimodal models have led to significant strides in image comprehension capabilities. Despite these advancements, there is a lack of the robust benchmark specifically for assessing the Image-to-Web conversion proficiency of these large models. Primarily, it is essential to ensure the integrity of the web elements generated. These elements comprise visible and invisible categories. Previous evaluation methods (e.g.,BLEU) are notably susceptible to significant alterations due to the presence of invisible elements in Web. Furthermore, it is crucial to measure the layout information of web pages, referring to the positional relationships between elements, which is overlooked by previous work. To address challenges, we have curated and aligned a benchmark of images and corresponding web codes (IW-BENCH). Specifically, we propose the Element Accuracy, which tests the completeness of the elements by parsing the Document Object Model (DOM) tree. Layout Accuracy is also proposed to analyze the positional relationships of elements by converting DOM tree into a common subsequence. Besides, we design a five-hop multimodal Chain-of-Thought Prompting for better performance, which contains five hop: 1) SoM prompt injection. 2) Inferring Elements. 3) Inferring Layout. 4) Inferring Web code. 5) Reflection. Our benchmark comprises 1200 pairs of images and web codes with varying levels of difficulty. We have conducted extensive experiments on existing large multimodal models, offering insights into their performance and areas for improvement in image-to-web domain.
♻ ☆ On Efficient Variants of Segment Anything Model: A Survey
Xiaorui Sun, Jun Liu, Heng Tao Shen, Xiaofeng Zhu, Ping Hu
The Segment Anything Model (SAM) is a foundational model for image segmentation tasks, known for its strong generalization across diverse applications. However, its impressive performance comes with significant computational and resource demands, making it challenging to deploy in resource-limited environments such as edge devices. To address this, a variety of SAM variants have been proposed to enhance efficiency while keeping accuracy. This survey provides the first comprehensive review of these efficient SAM variants. We begin by exploring the motivations driving this research. We then present core techniques used in SAM and model acceleration. This is followed by a detailed exploration of SAM acceleration strategies, categorized by approach, and a discussion of several future research directions. Finally, we offer a unified and extensive evaluation of these methods across various hardware, assessing their efficiency and accuracy on representative benchmarks, and providing a clear comparison of their overall performance.
♻ ☆ Two-Stage Vision Transformer for Image Restoration: Colorization Pretrain ing + Residual Upsampling
Aditya Chaudhary, Prachet Dev Singh, Ankit Jha
In computer vision, Single Image Super-Resolution (SISR) is still a difficult problem. We present ViT-SR, a new technique to improve the performance of a Vision Transformer (ViT) employing a two-stage training strategy. In our method, the model learns rich, generalizable visual representations from the data itself through a self-supervised pretraining phase on a colourization task. The pre-trained model is then adjusted for 4x super-resolution. By predicting the addition of a high-frequency residual image to an initial bicubic interpolation, this design simplifies residual learning. ViT-SR, trained and evaluated on the DIV2K benchmark dataset, achieves an impressive SSIM of 0.712 and PSNR of 22.90 dB. These results demonstrate the efficacy of our two-stage approach and highlight the potential of self-supervised pre-training for complex image restoration tasks. Further improvements may be possible with larger ViT architectures or alternative pretext tasks.
comment : Accepted as a Tiny Paper at the 13th Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP 2025), IIT Mandi, India. 3 pages, 1 figure
♻ ☆ PixCell: A generative foundation model for digital histopathology images
Srikar Yellapragada, Alexandros Graikos, Zilinghan Li, Kostas Triaridis, Varun Belagali, Tarak Nath Nandi, Karen Bai, Beatrice S. Knudsen, Tahsin Kurc, Rajarsi R. Gupta, Prateek Prasanna, Ravi K Madduri, Joel Saltz, Dimitris Samaras
The digitization of histology slides has revolutionized pathology, providing massive datasets for cancer diagnosis and research. Self-supervised and vision-language models have been shown to effectively mine large pathology datasets to learn discriminative representations. On the other hand, there are unique problems in pathology, such as annotated data scarcity, privacy regulations in data sharing, and inherently generative tasks like virtual staining. Generative models, capable of synthesizing realistic and diverse images, present a compelling solution to address these problems through image synthesis. We introduce PixCell, the first generative foundation model for histopathology images. PixCell is a diffusion model trained on PanCan-30M, a large, diverse dataset derived from 69,184 H&E-stained whole slide images of various cancer types. We employ a progressive training strategy and a self-supervision-based conditioning that allows us to scale up training without any human-annotated data. By conditioning on real slides, the synthetic images capture the properties of the real data and can be used as data augmentation for small-scale datasets to boost classification performance. We prove the foundational versatility of PixCell by applying it to two generative downstream tasks: privacy-preserving synthetic data generation and virtual IHC staining. PixCell's high-fidelity conditional generation enables institutions to use their private data to synthesize highly realistic, site-specific surrogate images that can be shared in place of raw patient data. Furthermore, using datasets of roughly paired H&E-IHC tiles, we learn to translate PixCell's conditioning from H&E to multiple IHC stains, allowing the generation of IHC images from H&E inputs. Our trained models are publicly released to accelerate research in computational pathology.
comment : Project page - https://histodiffusion.github.io/docs/projects/pixcell
♻ ☆ LoVoRA: Text-guided and Mask-free Video Object Removal and Addition with Learnable Object-aware Localization
Zhihan Xiao, Lin Liu, Yixin Gao, Xiaopeng Zhang, Haoxuan Che, Songping Mai, Qi Tian
Text-guided video editing, particularly for object removal and addition, remains a challenging task due to the need for precise spatial and temporal consistency. Existing methods often rely on auxiliary masks or reference images for editing guidance, which limits their scalability and generalization. To address these issues, we propose LoVoRA, a novel framework for mask-free video object removal and addition using object-aware localization mechanism. Our approach utilizes a unique dataset construction pipeline that integrates image-to-video translation, optical flow-based mask propagation, and video inpainting, enabling temporally consistent edits. The core innovation of LoVoRA is its learnable object-aware localization mechanism, which provides dense spatio-temporal supervision for both object insertion and removal tasks. By leveraging a Diffusion Mask Predictor, LoVoRA achieves end-to-end video editing without requiring external control signals during inference. Extensive experiments and human evaluation demonstrate the effectiveness and high-quality performance of LoVoRA. https://cz-5f.github.io/LoVoRA.github.io
♻ ☆ S5: Scalable Semi-Supervised Semantic Segmentation in Remote Sensing AAAI 2026
Liang Lv, Di Wang, Jing Zhang, Lefei Zhang
Semi-supervised semantic segmentation (S4) has advanced remote sensing (RS) analysis by leveraging unlabeled data through pseudo-labeling and consistency learning. However, existing S4 studies often rely on small-scale datasets and models, limiting their practical applicability. To address this, we propose S5, the first scalable framework for semi-supervised semantic segmentation in RS, which unlocks the potential of vast unlabeled Earth observation data typically underutilized due to costly pixel-level annotations. Built upon existing large-scale RS datasets, S5 introduces a data selection strategy that integrates entropy-based filtering and diversity expansion, resulting in the RS4P-1M dataset. Using this dataset, we systematically scale up S4 into a new pretraining paradigm, S4 pre-training (S4P), to pretrain RS foundation models (RSFMs) of varying sizes on this extensive corpus, significantly boosting their performance on land cover segmentation and object detection tasks. Furthermore, during fine-tuning, we incorporate a Mixture-of-Experts (MoE)-based multi-dataset fine-tuning approach, which enables efficient adaptation to multiple RS benchmarks with fewer parameters. This approach improves the generalization and versatility of RSFMs across diverse RS benchmarks. The resulting RSFMs achieve state-of-the-art performance across all benchmarks, underscoring the viability of scaling semi-supervised learning for RS applications. All datasets, code, and models will be released at https://github.com/MiliLab/S5
comment : AAAI 2026 Oral
♻ ☆ Mind GPT -4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigm
Wei Chen, Chaoqun Du, Feng Gu, Wei He, Qizhen Li, Zide Liu, Xuhao Pan, Chang Ren, Xudong Rao, Chenfeng Wang, Tao Wei, Chengjun Yu, Pengfei Yu, Yufei Zheng, Chunpeng Zhou, Pan Zhou, Xuhan Zhu
We present MindGPT-4ov, a multimodal large language model (MLLM) that introduces a general post-training paradigm spanning data production, model training, and efficient deployment. It achieves state-of-the-art performance across multiple benchmarks at low cost, effectively enhancing the foundational capabilities of MLLMs and the generalization ability. Focusing on data construction, supervised fine-tuning strategies, and multimodal reinforcement learning methods, this work proposes three key innovations: (1) An information density-based data generation scheme, integrated with a dual-dimensional tree-structured label system, enabling automated generation of high-quality cross-domain data. (2) A collaborative curriculum supervised fine-tuning approach that balances the injection of domain-specific knowledge with the preservation of general capabilities. (3) A hybrid reinforcement learning paradigm that enhances reasoning ability while simultaneously addressing multi-objective optimization such as diversity exploration, maintenance of multimodal perception, and response conciseness. Moreover, we implement a series of infrastructure optimizations, such as 5D parallel training, operator optimization, and inference quantization to enhance training and inference efficiency while reducing the cost of domain adaptation. Experimental results demonstrate that the MindGPT-4ov model outperforms state-of-the-art models on benchmarks such as MMBench, MMStar, MathVision, and MathVista. In addition, MindGPT-4ov also demonstrates superior user experience in vertical domain tasks, enabling a seamless transition from academic research to industrial deployment. MindGPT-4ov provides a general post-training paradigm applicable to a wide range of MLLMs. The model weights, datasets, and code for the Qwen3-VL-based variants will be recently open-sourced to support the community's development of MLLMs.
comment : 33 pages, 14 figures
♻ ☆ FantasyStyle: Controllable Stylized Distillation for 3D Gaussian Splatting AAAI 2026
Yitong Yang, Yinglin Wang, Changshuo Wang, Huajie Wang, Shuting He
The success of 3DGS in generative and editing applications has sparked growing interest in 3DGS-based style transfer. However, current methods still face two major challenges: (1) multi-view inconsistency often leads to style conflicts, resulting in appearance smoothing and distortion; and (2) heavy reliance on VGG features, which struggle to disentangle style and content from style images, often causing content leakage and excessive stylization. To tackle these issues, we introduce \textbf{FantasyStyle}, a 3DGS-based style transfer framework, and the first to rely entirely on diffusion model distillation. It comprises two key components: (1) \textbf{Multi-View Frequency Consistency}. We enhance cross-view consistency by applying a 3D filter to multi-view noisy latent, selectively reducing low-frequency components to mitigate stylized prior conflicts. (2) \textbf{Controllable Stylized Distillation}. To suppress content leakage from style images, we introduce negative guidance to exclude undesired content. In addition, we identify the limitations of Score Distillation Sampling and Delta Denoising Score in 3D style transfer and remove the reconstruction term accordingly. Building on these insights, we propose a controllable stylized distillation that leverages negative guidance to more effectively optimize the 3D Gaussians. Extensive experiments demonstrate that our method consistently outperforms state-of-the-art approaches, achieving higher stylization quality and visual realism across various scenes and styles. The code is available at https://github.com/yangyt46/FantasyStyle.
comment : Accepted by AAAI 2026
♻ ☆ The Outline of Deception: Physical Adversarial Attacks on Traffic Signs Using Edge Patches
Haojie Ji, Te Hu, Haowen Li, Long Jin, Chongshi Xin, Yuchi Yao, Jiarui Xiao
Intelligent driving systems are vulnerable to physical adversarial attacks on traffic signs. These attacks can cause misclassification, leading to erroneous driving decisions that compromise road safety. Moreover, within V2X networks, such misinterpretations can propagate, inducing cascading failures that disrupt overall traffic flow and system stability. However, a key limitation of current physical attacks is their lack of stealth. Most methods apply perturbations to central regions of the sign, resulting in visually salient patterns that are easily detectable by human observers, thereby limiting their real-world practicality. This study proposes TESP-Attack, a novel stealth-aware adversarial patch method for traffic sign classification. Based on the observation that human visual attention primarily focuses on the central regions of traffic signs, we employ instance segmentation to generate edge-aligned masks that conform to the shape characteristics of the signs. A U-Net generator is utilized to craft adversarial patches, which are then optimized through color and texture constraints along with frequency domain analysis to achieve seamless integration with the background environment, resulting in highly effective visual concealment. The proposed method demonstrates outstanding attack success rates across traffic sign classification models with varied architectures, achieving over 90% under limited query budgets. It also exhibits strong cross-model transferability and maintains robust real-world performance that remains stable under varying angles and distances.
♻ ☆ Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generation
Jianzong Wu, Hao Lian, Dachao Hao, Ye Tian, Qingyu Shi, Biaolong Chen, Hao Jiang, Yunhai Tong
Recent audio-video generative systems suggest that coupling modalities benefits not only audio-video synchrony but also the video modality itself. We pose a fundamental question: Does audio-video joint denoising training improve video generation, even when we only care about video quality? To study this, we introduce a parameter-efficient Audio-Video Full DiT (AVFullDiT) architecture that leverages pre-trained text-to-video (T2V) and text-to-audio (T2A) modules for joint denoising. We train (i) a T2AV model with AVFullDiT and (ii) a T2V-only counterpart under identical settings. Our results provide the first systematic evidence that audio-video joint denoising can deliver more than synchrony. We observe consistent improvements on challenging subsets featuring large and object contact motions. We hypothesize that predicting audio acts as a privileged signal, encouraging the model to internalize causal relationships between visual events and their acoustic consequences (e.g., collision $\times$ impact sound), which in turn regularizes video dynamics. Our findings suggest that cross-modal co-training is a promising approach to developing stronger, more physically grounded world models. Code and dataset will be made publicly available.
comment : Project page at https://jianzongwu.github.io/projects/does-hearing-help-seeing/
♻ ☆ \textit{ViRectify}: A Challenging Benchmark for Video Reasoning Correction with Multimodal Large Language Models
Xusen Hei, Jiali Chen, Jinyu Yang, Mengchen Zhao, Yi Cai
As multimodal large language models (MLLMs) frequently exhibit errors in complex video reasoning scenarios, correcting these errors is critical for uncovering their weaknesses and improving performance. However, existing benchmarks lack systematic evaluation of MLLMs' ability to identify and correct these video reasoning errors. To bridge this gap, we propose \textit{ViRectify}, a comprehensive benchmark to evaluate their fine-grained correction capability. Through an AI-assisted annotation pipeline with human verification, we construct a dataset of over 30\textit{K} instances spanning dynamic perception, scientific reasoning, and embodied decision-making domains. In \textit{ViRectify}, we challenge MLLMs to perform step-wise error identification and generate rationales with key video evidence grounding. In addition, we further propose the trajectory evidence-driven correction framework, comprising step-wise error trajectory and reward modeling on visual evidence-grounded correction. It encourages the model to explicitly concentrate on error propagation and key timestamps for correction. Extensive evaluation across 16 advanced MLLMs demonstrates that our \textit{ViRectify} serves as a challenging testbed, where GPT-5 achieves only 31.94\% correction accuracy. Our framework enables a Qwen2.5-VL-7B to consistently outperform the variants of 72B on \textit{ViRectify}, showing the effectiveness of our approach. Further analysis uncovers systematic asymmetries in error correction across models, and our dataset is also a valuable data resource to perform reflection learning. We believe \textit{ViRectify} provides a new direction for comprehensively evaluating the advanced MLLMs in video reasoning.
comment : 22 pages, 11 figures
♻ ☆ Look, Recite, Then Answer: Enhancing VLM Performance via Self-Generated Knowledge Hints
Xisheng Feng
Vision-Language Models (VLMs) exhibit significant performance plateaus in specialized domains like precision agriculture, primarily due to "Reasoning-Driven Hallucination" where linguistic priors override visual perception. A key bottleneck is the "Modality Gap": visual embeddings fail to reliably activate the fine-grained expert knowledge already encoded in model parameters. We propose "Look, Recite, Then Answer," a parameter-efficient framework that enhances VLMs via self-generated knowledge hints while keeping backbone models frozen. The framework decouples inference into three stages: (1) Look generates objective visual descriptions and candidate sets; (2) Recite employs a lightweight 1.7B router to transform visual cues into targeted queries that trigger candidate-specific parametric knowledge; (3) Answer performs parallel evidence alignment between descriptions and recited knowledge to select the most consistent label. On AgroBench, our method achieves state-of-the-art results, improving Weed Identification accuracy by 23.52% over Qwen2-VL-72B and surpassing GPT-4o without external search overhead. This modular design mitigates hallucinations by transforming passive perception into active, controllable knowledge retrieval
♻ ☆ MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding
Fan Yang, Kaihao Zhang
Understanding high-resolution images remains a significant challenge for multimodal large language models (MLLMs). Recent study address this issue by dividing the image into smaller crops and computing the semantic similarity between each crop and a query using a pretrained retrieval-augmented generation (RAG) model. The most relevant crops are then selected to localize the target object and suppress irrelevant information. However, such crop-based processing can fragment complete objects across multiple crops, thereby disrupting the computation of semantic similarity. In our experiments, we find that image crops of objects with different sizes are better handled at different resolutions. Based on this observation, we propose Multi-resolution Retrieval-Detection (MRD), a training-free framework for high-resolution image understanding. To address the issue of semantic similarity bias caused by objects being split across different image crops, we propose a multi-resolution semantic fusion method, which integrates semantic similarity maps obtained at different resolutions to produce more accurate semantic information and preserve the integrity of target objects. Furthermore, to achieve direct localization of target objects at a global scale, we introduce an open-vocalbulary object detection (OVD) model that identifies object regions using a sliding-window approach.Experiments on high-resolution image understanding benchmarks using different MLLMs demonstrate the effectiveness of our approach.
♻ ☆ A$^2$LC: Active and Automated Label Correction for Semantic Segmentation AAAI 2026
Youjin Jeon, Kyusik Cho, Suhan Woo, Euntai Kim
Active Label Correction (ALC) has emerged as a promising solution to the high cost and error-prone nature of manual pixel-wise annotation in semantic segmentation, by actively identifying and correcting mislabeled data. Although recent work has improved correction efficiency by generating pseudo-labels using foundation models, substantial inefficiencies still remain. In this paper, we introduce A$^2$LC, an Active and Automated Label Correction framework for semantic segmentation, where manual and automatic correction stages operate in a cascaded manner. Specifically, the automatic correction stage leverages human feedback to extend label corrections beyond the queried samples, thereby maximizing cost efficiency. In addition, we introduce an adaptively balanced acquisition function that emphasizes underrepresented tail classes, working in strong synergy with the automatic correction stage. Extensive experiments on Cityscapes and PASCAL VOC 2012 demonstrate that A$^2$LC significantly outperforms previous state-of-the-art methods. Notably, A$^2$LC exhibits high efficiency by outperforming previous methods with only 20% of their budget, and shows strong effectiveness by achieving a 27.23% performance gain under the same budget on Cityscapes.
comment : Accepted by AAAI 2026
♻ ☆ Revisiting Data Challenges of Computational Pathology: A Pack-based Multiple Instance Learning Training Framework
Wenhao Tang, Heng Fang, Ge Wu, Xiang Li, Ming-Ming Cheng
Computational pathology (CPath) digitizes pathology slides into whole slide images (WSIs), enabling analysis for critical healthcare tasks such as cancer diagnosis and prognosis. However, WSIs possess extremely long sequence lengths (up to 200K), significant length variations (from 200 to 200K), and limited supervision. These extreme variations in sequence length lead to high data heterogeneity and redundancy. Conventional methods often compromise on training efficiency and optimization to preserve such heterogeneity under limited supervision. To comprehensively address these challenges, we propose a pack-based MIL framework. It packs multiple sampled, variable-length feature sequences into fixed-length ones, enabling batched training while preserving data heterogeneity. Moreover, we introduce a residual branch that composes discarded features from multiple slides into a hyperslide which is trained with tailored labels. It offers multi-slide supervision while mitigating feature loss from sampling. Meanwhile, an attention-driven downsampler is introduced to compress features in both branches to reduce redundancy. By alleviating these challenges, our approach achieves an accuracy improvement of up to 8% while using only 12% of the training time in the PANDA(UNI). Extensive experiments demonstrate that focusing data challenges in CPath holds significant potential in the era of foundation models. The code is https://github.com/FangHeng/PackMIL
comment : 24 pages, 6 figures
♻ ☆ AugMapNet: Improving Spatial Latent Structure via BEV Grid Augmentation for Enhanced Vectorized Online HD Map Construction WACV 2026
Thomas Monninger, Md Zafar Anwar, Stanislaw Antol, Steffen Staab, Sihao Ding
Autonomous driving requires understanding infrastructure elements, such as lanes and crosswalks. To navigate safely, this understanding must be derived from sensor data in real-time and needs to be represented in vectorized form. Learned Bird's-Eye View (BEV) encoders are commonly used to combine a set of camera images from multiple views into one joint latent BEV grid. Traditionally, from this latent space, an intermediate raster map is predicted, providing dense spatial supervision but requiring post-processing into the desired vectorized form. More recent models directly derive infrastructure elements as polylines using vectorized map decoders, providing instance-level information. Our approach, Augmentation Map Network (AugMapNet), proposes latent BEV feature grid augmentation, a novel technique that significantly enhances the latent BEV representation. AugMapNet combines vector decoding and dense spatial supervision more effectively than existing architectures while remaining easy to integrate compared to other hybrid approaches. It additionally benefits from extra processing on its latent BEV features. Experiments on nuScenes and Argoverse2 datasets demonstrate significant improvements on vectorized map prediction of up to 13.3% over the StreamMapNet baseline on 60 m range and greater improvements on larger ranges. We confirm transferability by applying our method to another baseline, SQD-MapNet, and find similar improvements. A detailed analysis of the latent BEV grid confirms a more structured latent space of AugMapNet and shows the value of our novel concept beyond pure performance improvement. The code can be found at https://github.com/tmonnin/augmapnet
comment : Accepted to 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2026)
♻ ☆ ROGR: Relightable 3D Objects using Generative Relighting NeurIPS 2025
Jiapeng Tang, Matthew Levine, Dor Verbin, Stephan J. Garbin, Matthias Nießner, Ricardo Martin Brualla, Pratul P. Srinivasan, Philipp Henzler
We introduce ROGR, a novel approach that reconstructs a relightable 3D model of an object captured from multiple views, driven by a generative relighting model that simulates the effects of placing the object under novel environment illuminations. Our method samples the appearance of the object under multiple lighting environments, creating a dataset that is used to train a lighting-conditioned Neural Radiance Field (NeRF) that outputs the object's appearance under any input environmental lighting. The lighting-conditioned NeRF uses a novel dual-branch architecture to encode the general lighting effects and specularities separately. The optimized lighting-conditioned NeRF enables efficient feed-forward relighting under arbitrary environment maps without requiring per-illumination optimization or light transport simulation. We evaluate our approach on the established TensoIR and Stanford-ORB datasets, where it improves upon the state-of-the-art on most metrics, and showcase our approach on real-world object captures.
comment : NeurIPS 2025 Spotlight. Project page: https://tangjiapeng.github.io/ROGR
♻ ☆ Average Calibration Losses for Reliable Uncertainty in Medical Image Segmentation
Theodore Barfoot, Luis C. Garcia-Peraza-Herrera, Samet Akcay, Ben Glocker, Tom Vercauteren
Deep neural networks for medical image segmentation are often overconfident, compromising both reliability and clinical utility. In this work, we propose differentiable formulations of marginal L1 Average Calibration Error (mL1-ACE) as an auxiliary loss that can be computed on a per-image basis. We compare both hard- and soft-binning approaches to directly improve pixel-wise calibration. Our experiments on four datasets (ACDC, AMOS, KiTS, BraTS) demonstrate that incorporating mL1-ACE significantly reduces calibration errors, particularly Average Calibration Error (ACE) and Maximum Calibration Error (MCE), while largely maintaining high Dice Similarity Coefficients (DSCs). We find that the soft-binned variant yields the greatest improvements in calibration, over the Dice plus cross-entropy loss baseline, but often compromises segmentation performance, with hard-binned mL1-ACE maintaining segmentation performance, albeit with weaker calibration improvement. To gain further insight into calibration performance and its variability across an imaging dataset, we introduce dataset reliability histograms, an aggregation of per-image reliability diagrams. The resulting analysis highlights improved alignment between predicted confidences and true accuracies. Overall, our approach not only enhances the trustworthiness of segmentation predictions but also shows potential for safer integration of deep learning methods into clinical workflows. We share our code here: https://github.com/cai4cai/Average-Calibration-Losses
comment : 15 pages, 6 figures, IEEE TMI submission. This version originally appeared in error as arXiv:2403.06759(v2)
♻ ☆ Consistency-based Abductive Reasoning over Perceptual Errors of Multiple Pre-train ed Models in Novel Environments
Mario Leiva, Noel Ngu, Joshua Shay Kricheli, Aditya Taparia, Ransalu Senanayake, Paulo Shakarian, Nathaniel Bastian, John Corcoran, Gerardo Simari
The deployment of pre-trained perception models in novel environments often leads to performance degradation due to distributional shifts. Although recent artificial intelligence approaches for metacognition use logical rules to characterize and filter model errors, improving precision often comes at the cost of reduced recall. This paper addresses the hypothesis that leveraging multiple pre-trained models can mitigate this recall reduction. We formulate the challenge of identifying and managing conflicting predictions from various models as a consistency-based abduction problem, building on the idea of abductive learning (ABL) but applying it to test-time instead of training. The input predictions and the learned error detection rules derived from each model are encoded in a logic program. We then seek an abductive explanation--a subset of model predictions--that maximizes prediction coverage while ensuring the rate of logical inconsistencies (derived from domain constraints) remains below a specified threshold. We propose two algorithms for this knowledge representation task: an exact method based on Integer Programming (IP) and an efficient Heuristic Search (HS). Through extensive experiments on a simulated aerial imagery dataset featuring controlled, complex distributional shifts, we demonstrate that our abduction-based framework outperforms individual models and standard ensemble baselines, achieving, for instance, average relative improvements of approximately 13.6\% in F1-score and 16.6\% in accuracy across 15 diverse test datasets when compared to the best individual model. Our results validate the use of consistency-based abduction as an effective mechanism to robustly integrate knowledge from multiple imperfect models in challenging, novel scenarios.
♻ ☆ Surface-Based Visibility-Guided Uncertainty for Continuous Active 3D Neural Reconstruction AAAI 2026
Hyunseo Kim, Hyeonseo Yang, Taekyung Kim, YoonSung Kim, Minsu Lee, Jin-Hwa Kim, Byoung-Tak Zhang
View selection is critical in active 3D neural reconstruction as it impacts the contents of training set and resulting final output quality. Recent view selection strategies emphasize the visibility when evaluating model uncertainty in active 3D reconstruction. However, existing approaches estimate visibility only after the model fully converges, which has confined their application primarily to non-continuous active learning settings. This paper proposes Surface-Based Visibility field (SBV) that successfully estimates the visibility-guided uncertainty in continuous active 3D neural reconstruction. During learning neural implicit surfaces, our model learns rendering uncertainties and infers surface confidence values derived from signed distance functions. It then updates surface confidences using a voxel grid, robustly deducing the surface-based visibility for uncertainties. This approach captures uncertainties across all regions, whether well-defined surfaces or ambiguous areas, ensuring accurate visibility measurement in continuous active learning. Experiments on benchmark datasets-Tanks and Temples, BlendedMVS, Blender, DTU-and the newly proposed imbalanced viewpoint dataset (ImBView) show that view selection based on SBV-guided uncertainty improves performance by up to 11.6% over existing methods, highlighting its effectiveness in challenging reconstruction scenarios.
comment : The main claims are the same as in the previous version, but the naming and explanations have been changed. Accepted at AAAI 2026 Artificial Intelligence with Biased or Scarce Data workshop
♻ ☆ XAI-Driven Skin Disease Classification: Leveraging GANs to Augment ResNet-50 Performance
Kim Gerard A. Villanueva, Priyanka Kumar
Accurate and timely diagnosis of multi-class skin lesions is hampered by subjective methods, inherent data imbalance in datasets like HAM10000, and the "black box" nature of Deep Learning (DL) models. This study proposes a trustworthy and highly accurate Computer-Aided Diagnosis (CAD) system to overcome these limitations. The approach utilizes Deep Convolutional Generative Adversarial Networks (DCGANs) for per class data augmentation to resolve the critical class imbalance problem. A fine-tuned ResNet-50 classifier is then trained on the augmented dataset to classify seven skin disease categories. Crucially, LIME and SHAP Explainable AI (XAI) techniques are integrated to provide transparency by confirming that predictions are based on clinically relevant features like irregular morphology. The system achieved a high overall Accuracy of 92.50 % and a Macro-AUC of 98.82 %, successfully outperforming various prior benchmarked architectures. This work successfully validates a verifiable framework that combines high performance with the essential clinical interpretability required for safe diagnostic deployment. Future research should prioritize enhancing discrimination for critical categories, such as Melanoma NOS (F1-Score is 0.8602).
♻ ☆ NITRO-D: Native Integer-only Training of Deep Convolutional Neural Networks
Alberto Pirillo, Luca Colombo, Manuel Roveri
Quantization is a pivotal technique for managing the growing computational and memory demands of Deep Neural Networks (DNNs). By reducing the number of bits used to represent weights and activations (typically from 32-bit Floating-Point (FP) to 16-bit or 8-bit integers), quantization reduces memory footprint, energy consumption, and execution time of DNNs. However, most existing methods typically target DNN inference, while training still relies on FP operations, limiting applicability in environments where FP arithmetic is unavailable. To date, only one prior work has addressed integer-only training, and only for Multi-Layer Perceptron (MLP) architectures. This paper introduces NITRO-D, a novel framework for training deep integer-only Convolutional Neural Networks (CNNs) that operate entirely in the integer domain for both training and inference. NITRO-D enables training of integer CNNs without requiring a separate quantization scheme. Specifically, it introduces a novel architecture that integrates multiple local-loss blocks, which include the proposed NITRO-Scaling layer and NITRO-ReLU activation function. The proposed framework also features a novel learning algorithm that employs local error signals and leverages IntegerSGD, an optimizer specifically designed for integer computations. NITRO-D is implemented as an open-source Python library. Extensive evaluations on state-of-the-art image recognition datasets demonstrate its effectiveness. For integer-only MLPs, NITRO-D improves test accuracy by up to +5.96% over the state-of-the-art. It also successfully trains integer-only CNNs, reducing memory requirements and energy consumption by up to 76.14% and 32.42%, respectively, compared to the traditional FP backpropagation algorithm.
comment : 15 pages, 3 figures
♻ ☆ Open-Set Domain Adaptation Under Background Distribution Shift: Challenges and A Provably Efficient Solution
Shravan Chaudhari, Yoav Wald, Suchi Saria
As we deploy machine learning systems in the real world, a core challenge is to maintain a model that is performant even as the data shifts. Such shifts can take many forms: new classes may emerge that were absent during training, a problem known as open-set recognition, and the distribution of known categories may change. Guarantees on open-set recognition are mostly derived under the assumption that the distribution of known classes, which we call the background distribution, is fixed. In this paper we develop CoLOR, a method that is guaranteed to solve open-set recognition even in the challenging case where the background distribution shifts. We prove that the method works under benign assumptions that the novel class is separable from the non-novel classes, and provide theoretical guarantees that it outperforms a representative baseline in a simplified overparameterized setting. We develop techniques to make CoLOR scalable and robust, and perform comprehensive empirical evaluations on image and text data. The results show that CoLOR significantly outperforms existing open-set recognition methods under background shift. Moreover, we provide new insights into how factors such as the size of the novel class influences performance, an aspect that has not been extensively explored in prior work.
Computation and Language 94
☆ SkillFactory: Self-Distillation For Learning Cognitive Behaviors
Zayne Sprague, Jack Lu, Manya Wadhwa, Sedrick Keh, Mengye Ren, Greg Durrett
Reasoning models leveraging long chains of thought employ various cognitive skills, such as verification of their answers, backtracking, retrying by an alternate method, and more. Previous work has shown that when a base language model exhibits these skills, training that model further with reinforcement learning (RL) can learn to leverage them. How can we get models to leverage skills that aren't exhibited by base models? Our work, SkillFactory, is a method for fine-tuning models to roughly learn these skills during a supervised fine-tuning (SFT) stage prior to RL. Our approach does not rely on distillation from a stronger model, but instead uses samples from the model itself, rearranged to provide training data in the format of those skills. These "silver" SFT traces may be imperfect, but are nevertheless effective for priming a model to acquire skills during RL. Our evaluation shows that (1) starting from SkillFactory SFT initialization helps a model to generalize to harder variants of a task post-RL, despite lower performance pre-RL; (2) cognitive skills are indeed used by the model; (3) RLed SkillFactory models are more robust to regression on out-of-domain tasks than RLed base models. Our work suggests that inductive biases learned prior to RL help models learn robust cognitive skill use.
☆ Stable Signer: Hierarchical Sign Language Generative Model
Sen Fang, Yalin Feng, Hongbin Zhong, Yanxin Zhang, Dimitris N. Metaxas
Sign Language Production (SLP) is the process of converting the complex input text into a real video. Most previous works focused on the Text2Gloss, Gloss2Pose, Pose2Vid stages, and some concentrated on Prompt2Gloss and Text2Avatar stages. However, this field has made slow progress due to the inaccuracy of text conversion, pose generation, and the rendering of poses into real human videos in these stages, resulting in gradually accumulating errors. Therefore, in this paper, we streamline the traditional redundant structure, simplify and optimize the task objective, and design a new sign language generative model called Stable Signer. It redefines the SLP task as a hierarchical generation end-to-end task that only includes text understanding (Prompt2Gloss, Text2Gloss) and Pose2Vid, and executes text understanding through our proposed new Sign Language Understanding Linker called SLUL, and generates hand gestures through the named SLP-MoE hand gesture rendering expert block to end-to-end generate high-quality and multi-style sign language videos. SLUL is trained using the newly developed Semantic-Aware Gloss Masking Loss (SAGM Loss). Its performance has improved by 48.6% compared to the current SOTA generation methods.
comment : 12 pages, 7 figures. More Demo at https://stablesigner.github.io
☆ Jina-VLM: Small Multilingual Vision Language Model
Andreas Koukounas, Georgios Mastrapas, Florian Hönicke, Sedigheh Eslami, Guillaume Roncari, Scott Martens, Han Xiao
We present Jina-VLM, a 2.4B parameter vision-language model that achieves state-of-the-art multilingual visual question answering among open 2B-scale VLMs. The model couples a SigLIP2 vision encoder with a Qwen3 language backbone through an attention-pooling connector that enables token-efficient processing of arbitrary-resolution images. Across standard VQA benchmarks and multilingual evaluations, Jina-VLM outperforms comparable models while preserving competitive text-only performance.
comment : 18 pages, 1-7 main content
☆ AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving
Ying Wang, Zhen Jin, Jiexiong Xu, Wenhai Lin, Yiquan Chen, Wenzhi Chen
As augmented large language models (LLMs) with external tools become increasingly popular in web applications, improving augmented LLM inference serving efficiency and optimizing service-level objectives (SLOs) are critical for enhancing user experience. To achieve this, inference systems must maximize request handling within latency constraints, referred to as increasing effective throughput. However, existing systems face two major challenges: (i) reliance on first-come-first-served (FCFS) scheduling causes severe head-of-line blocking, leading to queuing delays exceeding the SLOs for many requests; and (ii) static batch token limit, which fails to adapt to fluctuating loads and hardware conditions. Both of these factors degrade effective throughput and service quality. This paper presents AugServe, an efficient inference framework designed to reduce queueing latency and enhance effective throughput for augmented LLM inference services. The core idea of AugServe is a two-stage adaptive request scheduling strategy. Specifically, AugServe combines the inference features of augmented LLM requests to optimize the order of scheduling decisions (stage I). These decisions are continuously refined with runtime information (stage II), adapting to both request characteristics and system capabilities. In addition, AugServe dynamically adjusts the token batching mechanism based on hardware status and real-time load, further enhancing throughput performance. Experimental results show that AugServe achieves 4.7-33.1x and 3.3-13.2x higher effective throughput than vLLM and InferCept, while reducing time-to-first-token (TTFT) by up to 96.3% and 95.0%, respectively.
☆ Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-train ed Models
Taido Purason, Pavel Chizhov, Ivan P. Yamshchikov, Mark Fishel
Tokenizer adaptation plays an important role in transferring pre-trained language models to new domains or languages. In this work, we address two complementary aspects of this process: vocabulary extension and pruning. The common approach to extension trains a new tokenizer on domain-specific text and appends the tokens that do not overlap with the existing vocabulary, which often results in many tokens that are unreachable or never used. We propose continued BPE training, which adapts a pre-trained tokenizer by continuing the BPE merge learning process on new data. Experiments across multiple languages and model families show that this approach improves tokenization efficiency and leads to better utilization of added vocabulary. We also introduce leaf-based vocabulary pruning, which removes redundant tokens while preserving model quality. Together, these methods provide practical tools for controlled vocabulary modification, which we release as an open-source package.
☆ Adapting Large Language Models to Low-Resource Tibetan: A Two-Stage Continual and Supervised Fine-Tuning Study
Lifeng Chen, Ryan Lai, Tianming Liu
Adapting large language models (LLMs) to low-resource languages remains a major challenge due to data scarcity and cross-lingual drift. This work presents a two-stage adaptation of Qwen2.5-3B to Tibetan, a morphologically rich and underrepresented language. We employ Continual Pretraining (CPT) to establish Tibetan linguistic grounding, followed by Supervised Fine-Tuning (SFT) for task and translation specialization. Empirical evaluations demonstrate a consistent decrease in perplexity (from 2.98 $\rightarrow$ 1.54) and substantial improvements in Chinese$\rightarrow$Tibetan translation quality (BLEU: 0.046 $\rightarrow$ 0.261; chrF: 2.2 $\rightarrow$ 6.6). Layer-wise analysis across 435 layers in Qwen3-4B reveals that adaptation primarily concentrates on embedding and output heads, with mid--late MLP projections encoding domain-specific transformations. Our findings suggest that CPT constructs a Tibetan semantic manifold while SFT sharpens task alignment with minimal representational disruption. This study provides the first quantitative exploration of Tibetan adaptation dynamics for LLMs, and offers an open, reproducible framework for extending multilingual foundation models to low-resource settings.
☆ Is Lying Only Sinful in Islam? Exploring Religious Bias in Multilingual Large Language Models Across Major Religions
Kazi Abrab Hossain, Jannatul Somiya Mahmud, Maria Hossain Tuli, Anik Mitra, S. M. Taiabul Haque, Farig Y. Sadeque
While recent developments in large language models have improved bias detection and classification, sensitive subjects like religion still present challenges because even minor errors can result in severe misunderstandings. In particular, multilingual models often misrepresent religions and have difficulties being accurate in religious contexts. To address this, we introduce BRAND: Bilingual Religious Accountable Norm Dataset, which focuses on the four main religions of South Asia: Buddhism, Christianity, Hinduism, and Islam, containing over 2,400 entries, and we used three different types of prompts in both English and Bengali. Our results indicate that models perform better in English than in Bengali and consistently display bias toward Islam, even when answering religion-neutral questions. These findings highlight persistent bias in multilingual models when similar questions are asked in different languages. We further connect our findings to the broader issues in HCI regarding religion and spirituality.
comment : 18 pages, 7 figures
☆ BERnaT: Basque Encoders for Representing Natural Textual Diversity LREC 2026
Ekhi Azurmendi, Joseba Fernandez de Landa, Jaione Bengoetxea, Maite Heredia, Julen Etxaniz, Mikel Zubillaga, Ander Soraluze, Aitor Soroa
Language models depend on massive text corpora that are often filtered for quality, a process that can unintentionally exclude non-standard linguistic varieties, reduce model robustness and reinforce representational biases. In this paper, we argue that language models should aim to capture the full spectrum of language variation (dialectal, historical, informal, etc.) rather than relying solely on standardized text. Focusing on Basque, a morphologically rich and low-resource language, we construct new corpora combining standard, social media, and historical sources, and pre-train the BERnaT family of encoder-only models in three configurations: standard, diverse, and combined. We further propose an evaluation framework that separates Natural Language Understanding (NLU) tasks into standard and diverse subsets to assess linguistic generalization. Results show that models trained on both standard and diverse data consistently outperform those trained on standard corpora, improving performance across all task types without compromising standard benchmark accuracy. These findings highlight the importance of linguistic diversity in building inclusive, generalizable language models.
comment : Submitted to LREC 2026
☆ Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformer s
Hongzhan Lin, Zhiqi Bai, Xinmiao Zhang, Sen Yang, Xiang Li, Siran Yang, Yunlong Xu, Jiaheng Liu, Yongchi Zhao, Jiamang Wang, Yuchi Xu, Wenbo Su, Bo Zheng
Transformer decoders have achieved strong results across tasks, but the memory required for the KV cache becomes prohibitive at long sequence lengths. Although Cross-layer KV Cache sharing (e.g., YOCO, CLA) offers a path to mitigate KV Cache bottleneck, it typically underperforms within-layer methods like GQA. To understand the root cause, we investigate the information flow of keys and values of the top-layers. Our preliminary reveals a clear distribution: values are predominantly derived from the bottom layer, while keys draw more information from both bottom and middle layers. Building upon this, we propose FusedKV, whose top-layer KV caches are a learnable fusion of the most informative ones from the bottom and middle layers. This fusion operates directly on post-RoPE keys, preserving relative positional information without the computational cost of re-applying rotary embeddings. To further improve efficiency, we propose FusedKV-Lite, an cross-layer sharing approach, where top-layer KV caches are directly derived from the bottom-layer values and the middle-layer keys. Compared to FusedKV, FusedKV-Lite reduces I/O overhead at the cost of a slight increase in perplexity. In experiments on LLMs ranging from 332M to 4B parameters, our proposed method reduce 50\% cache memory while achieving lower validation perplexity than the standard Transformer decoder, establishing it as a memory-efficient, high-performance architectural alternative.
comment : under review
☆ Training and Evaluation of Guideline-Based Medical Reasoning in LLMs
Michael Staniek, Artem Sokolov, Stefan Riezler
Machine learning for early prediction in medicine has recently shown breakthrough performance, however, the focus on improving prediction accuracy has led to a neglect of faithful explanations that are required to gain the trust of medical practitioners. The goal of this paper is to teach LLMs to follow medical consensus guidelines step-by-step in their reasoning and prediction process. Since consensus guidelines are ubiquitous in medicine, instantiations of verbalized medical inference rules to electronic health records provide data for fine-tuning LLMs to learn consensus rules and possible exceptions thereof for many medical areas. Consensus rules also enable an automatic evaluation of the model's inference process regarding its derivation correctness (evaluating correct and faithful deduction of a conclusion from given premises) and value correctness (comparing predicted values against real-world measurements). We exemplify our work using the complex Sepsis-3 consensus definition. Our experiments show that small fine-tuned models outperform one-shot learning of considerably larger LLMs that are prompted with the explicit definition and models that are trained on medical texts including consensus definitions. Since fine-tuning on verbalized rule instantiations of a specific medical area yields nearly perfect derivation correctness for rules (and exceptions) on unseen patient data in that area, the bottleneck for early prediction is not out-of-distribution generalization, but the orthogonal problem of generalization into the future by forecasting sparsely and irregularly sampled clinical variables. We show that the latter results can be improved by integrating the output representations of a time series forecasting model with the LLM in a multimodal setup.
☆ Improving Alignment Between Human and Machine Codes: An Empirical Assessment of Prompt Engineering for Construct Identification in Psychology
Kylie L. Anglin, Stephanie Milan, Brittney Hernandez, Claudia Ventura
Due to their architecture and vast pre-training data, large language models (LLMs) demonstrate strong text classification performance. However, LLM output - here, the category assigned to a text - depends heavily on the wording of the prompt. While literature on prompt engineering is expanding, few studies focus on classification tasks, and even fewer address domains like psychology, where constructs have precise, theory-driven definitions that may not be well represented in pre-training data. We present an empirical framework for optimizing LLM performance for identifying constructs in texts via prompt engineering. We experimentally evaluate five prompting strategies --codebook-guided empirical prompt selection, automatic prompt engineering, persona prompting, chain-of-thought reasoning, and explanatory prompting - with zero-shot and few-shot classification. We find that persona, chain-of-thought, and explanations do not fully address performance loss accompanying a badly worded prompt. Instead, the most influential features of a prompt are the construct definition, task framing, and, to a lesser extent, the examples provided. Across three constructs and two models, the classifications most aligned with expert judgments resulted from a few-shot prompt combining codebook-guided empirical prompt selection with automatic prompt engineering. Based on our findings, we recommend that researchers generate and evaluate as many prompt variants as feasible, whether human-crafted, automatically generated, or ideally both, and select prompts and examples based on empirical performance in a training dataset, validating the final approach in a holdout set. This procedure offers a practical, systematic, and theory-driven method for optimizing LLM prompts in settings where alignment with expert judgment is critical.
comment : 22 pages, 2 figures
☆ Enhancing Instruction-Following Capabilities in Seq2Seq Models: DoLA Adaptations for T5
Huey Sun, Anabel Yong, Lorenzo Gilly, Felipe Jin
Contrastive decoding is a lightweight and effective inference-time method that improves the quality of text generation in Large Language Models. However, algorithms such as DoLa (Decoding by Contrastive Layers) have only been implemented in decoder-only architectures and studied for their impact on improving factuality. This work adapts DoLa for the T5 and FLAN-T5 model families and evaluates its impact on the models' instruction following capabilities, which to our knowledge is the first implementation of a contrastive decoding strategy in an encoder-decoder architecture. Our results show that DoLa improves the faithfulness of text generation for certain categories of tasks and harms others. To understand these results, we present a layer-by-layer analysis of logit evolution in a FLAN-T5 model to quantify DoLa's impact on token output probabilities.
☆ Adapt Vision : Efficient Vision -Language Models via Adaptive Visual Acquisition
Zichuan Lin, Yicheng Liu, Yang Yang, Lvfang Tao, Deheng Ye
Vision-Language Models (VLMs) have achieved remarkable success in visual question answering tasks, but their reliance on large numbers of visual tokens introduces significant computational overhead. While existing efficient VLM approaches reduce visual tokens through fixed-ratio compression, they operate passively and lack the ability to adapt to varying task requirements. This motivates a fundamental question: Can VLMs autonomously determine the minimum number of visual tokens required for each sample? Inspired by human active vision mechanisms, we introduce AdaptVision, an efficient VLM paradigm that enables adaptive visual token acquisition through a coarse-to-fine approach. Our model initially processes compressed visual tokens from low-resolution images and selectively acquires additional visual information by invoking a bounding box tool to crop key regions when necessary. We train AdaptVision using a reinforcement learning framework that carefully balances accuracy and efficiency. Central to our approach is Decoupled Turn Policy Optimization (DTPO), which decouples the learning objective into two components: (1) tool learning, which optimizes correct tool utilization, and (2) accuracy improvement, which refines the generated responses to improve answer correctness. Based on this formulation, we further decouple advantage estimation by computing separate advantages for tokens associated with each objective. This formulation enables more effective optimization for AdaptVision compared to vanilla GRPO. Comprehensive experiments across multiple VQA benchmarks demonstrate that AdaptVision achieves superior performance while consuming substantially fewer visual tokens than state-of-the-art efficient VLM methods.
comment : 15 pages, 9 figures
☆ In- Context Representation Hijacking
Itay Yona, Amir Sarid, Michael Karasik, Yossi Gandelsman
We introduce \textbf{Doublespeak}, a simple \emph{in-context representation hijacking} attack against large language models (LLMs). The attack works by systematically replacing a harmful keyword (e.g., \textit{bomb}) with a benign token (e.g., \textit{carrot}) across multiple in-context examples, provided a prefix to a harmful request. We demonstrate that this substitution leads to the internal representation of the benign token converging toward that of the harmful one, effectively embedding the harmful semantics under a euphemism. As a result, superficially innocuous prompts (e.g., ``How to build a carrot?'') are internally interpreted as disallowed instructions (e.g., ``How to build a bomb?''), thereby bypassing the model's safety alignment. We use interpretability tools to show that this semantic overwrite emerges layer by layer, with benign meanings in early layers converging into harmful semantics in later ones. Doublespeak is optimization-free, broadly transferable across model families, and achieves strong success rates on closed-source and open-source systems, reaching 74\% ASR on Llama-3.3-70B-Instruct with a single-sentence context override. Our findings highlight a new attack surface in the latent space of LLMs, revealing that current alignment strategies are insufficient and should instead operate at the representation level.
☆ Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective
Jingyang Ou, Jiaqi Han, Minkai Xu, Shaoxuan Xu, Jianwen Xie, Stefano Ermon, Yi Wu, Chongxuan Li
Reinforcement Learning (RL) has proven highly effective for autoregressive language models, but adapting these methods to diffusion large language models (dLLMs) presents fundamental challenges. The core difficulty lies in likelihood approximation: while autoregressive models naturally provide token-level conditional probabilities essential for token-level RL objectives (e.g., GRPO), dLLMs generate sequences through iterative non-autoregressive denoising steps that lack this factorization. To address this fundamental mismatch, we propose ELBO-based Sequence-level Policy Optimization (ESPO), a principled RL framework that treats entire sequence generation as a single action and uses the ELBO as a tractable sequence-level likelihood proxy. Our method incorporates per-token normalization of importance ratios and robust KL-divergence estimation to ensure stable large-scale training. Extensive experiments on mathematical reasoning, coding, and planning tasks demonstrate that ESPO significantly outperforms token-level baselines, achieving dramatic improvements of 20-40 points on the Countdown task, while maintaining consistent gains on math and coding benchmarks. Our approach establishes sequence-level optimization as a principled and empirically effective paradigm for RL in dLLMs. Our code is available at https://github.com/ML-GSAI/ESPO.
☆ Thinking with Programming Vision : Towards a Unified View for Thinking with Images
Zirun Guo, Minjie Hong, Feng Zhang, Kai Jia, Tao Jin
Multimodal large language models (MLLMs) that think with images can interactively use tools to reason about visual inputs, but current approaches often rely on a narrow set of tools with limited real-world necessity and scalability. In this work, we first reveal a critical and previously overlooked weakness: even state-of-the-art MLLMs are surprisingly brittle, showing significant performance degradation on images with simple orientation changes or natural corruptions, underscoring the need for more robust tool-based reasoning. To address this, we propose CodeVision, a flexible and scalable code-as-tool framework where the model generates code as a universal interface to invoke any image operation, moving beyond fixed tool registries. We train our model using a two-stage methodology, beginning with Supervised Fine-Tuning (SFT) on a high-quality dataset curated for complex, multi-turn tool composition and error recovery, followed by Reinforcement Learning (RL) with a novel and dense process reward function to encourage strategic and efficient tool use. To facilitate this research, we construct new SFT and RL datasets and introduce a challenging new benchmark suite designed to rigorously evaluate robustness to orientation changes and multi-tool reasoning. Experiments on Qwen2.5-VL and Qwen3-VL series show that our approach significantly improves model performance and fosters emergent capabilities such as flexible tool composition, efficient chained execution, and robust error recovery from runtime feedback. Code is available at https://github.com/ByteDance-BandAI/CodeVision.
☆ AR-Med: Automated Relevance Enhancement in Medical Search via LLM-Driven Information Augmentation
Chuyue Wang, Jie Feng, Yuxi Wu, Hang Zhang, Zhiguo Fan, Bing Cheng, Wei Lin
Accurate and reliable search on online healthcare platforms is critical for user safety and service efficacy. Traditional methods, however, often fail to comprehend complex and nuanced user queries, limiting their effectiveness. Large language models (LLMs) present a promising solution, offering powerful semantic understanding to bridge this gap. Despite their potential, deploying LLMs in this high-stakes domain is fraught with challenges, including factual hallucinations, specialized knowledge gaps, and high operational costs. To overcome these barriers, we introduce \textbf{AR-Med}, a novel framework for \textbf{A}utomated \textbf{R}elevance assessment for \textbf{Med}ical search that has been successfully deployed at scale on the Online Medical Delivery Platforms. AR-Med grounds LLM reasoning in verified medical knowledge through a retrieval-augmented approach, ensuring high accuracy and reliability. To enable efficient online service, we design a practical knowledge distillation scheme that compresses large teacher models into compact yet powerful student models. We also introduce LocalQSMed, a multi-expert annotated benchmark developed to guide model iteration and ensure strong alignment between offline and online performance. Extensive experiments show AR-Med achieves an offline accuracy of over 93\%, a 24\% absolute improvement over the original online system, and delivers significant gains in online relevance and user satisfaction. Our work presents a practical and scalable blueprint for developing trustworthy, LLM-powered systems in real-world healthcare applications.
☆ DZ-TDPO: Non-Destructive Temporal Alignment for Mutable State Tracking in Long- Context Dialogue
Yijun Liao
Long-context dialogue systems suffer from State Inertia, where static constraints prevent models from resolving conflicts between evolving user intents and established historical context. To address this, we propose DZ-TDPO, a non-destructive alignment framework that synergizes conflict-aware dynamic KL constraints with a learnable temporal attention bias. Experiments on the Multi-Session Chat (MSC) dataset demonstrate that DZ-TDPO achieves state-of-the-art win rates (86.2% on Phi-3.5) while maintaining robust zero-shot generalization. Crucially, our scaling analysis reveals a "Capacity-Stability Trade-off": while smaller models incur an "alignment tax" (perplexity surge) to overcome historical inertia, the larger Qwen2.5-7B model achieves near-perfect alignment (99.4% win rate) with negligible perplexity overhead. This confirms that TAI can be alleviated via precise attention regulation rather than destructive weight updates, preserving general capabilities (MMLU) across model scales. Code and data are available: https://github.com/lyj20071013/DZ-TDPO
comment : 22 pages, 2 figures, 13 tables. Code available at https://github.com/lyj20071013/DZ-TDPO
☆ AITutor-EvalKit: Exploring the Capabilities of AI Tutors
Numaan Naeem, Kaushal Kumar Maurya, Kseniia Petukhova, Ekaterina Kochmar
We present AITutor-EvalKit, an application that uses language technology to evaluate the pedagogical quality of AI tutors, provides software for demonstration and evaluation, as well as model inspection and data visualization. This tool is aimed at education stakeholders as well as *ACL community at large, as it supports learning and can also be used to collect user feedback and annotations.
☆ Different types of syntactic agreement recruit the same units within large language models
Daria Kryvosheieva, Andrea de Varda, Evelina Fedorenko, Greta Tuckute
Large language models (LLMs) can reliably distinguish grammatical from ungrammatical sentences, but how grammatical knowledge is represented within the models remains an open question. We investigate whether different syntactic phenomena recruit shared or distinct components in LLMs. Using a functional localization approach inspired by cognitive neuroscience, we identify the LLM units most responsive to 67 English syntactic phenomena in seven open-weight models. These units are consistently recruited across sentences containing the phenomena and causally support the models' syntactic performance. Critically, different types of syntactic agreement (e.g., subject-verb, anaphor, determiner-noun) recruit overlapping sets of units, suggesting that agreement constitutes a meaningful functional category for LLMs. This pattern holds in English, Russian, and Chinese; and further, in a cross-lingual analysis of 57 diverse languages, structurally more similar languages share more units for subject-verb agreement. Taken together, these findings reveal that syntactic agreement-a critical marker of syntactic dependencies-constitutes a meaningful category within LLMs' representational spaces.
☆ Evaluating Hydro-Science and Engineering Knowledge of Large Language Models
Shiruo Hu, Wenbo Shan, Yingjia Li, Zhiqi Wan, Xinpeng Yu, Yunjia Qi, Haotian Xia, Yang Xiao, Dingxiao Liu, Jiaru Wang, Chenxu Gong, Ruixi Zhang, Shuyue Wu, Shibo Cui, Chee Hui Lai, Wei Luo, Yubin He, Bin Xu, Jianshi Zhao
Hydro-Science and Engineering (Hydro-SE) is a critical and irreplaceable domain that secures human water supply, generates clean hydropower energy, and mitigates flood and drought disasters. Featuring multiple engineering objectives, Hydro-SE is an inherently interdisciplinary domain that integrates scientific knowledge with engineering expertise. This integration necessitates extensive expert collaboration in decision-making, which poses difficulties for intelligence. With the rapid advancement of large language models (LLMs), their potential application in the Hydro-SE domain is being increasingly explored. However, the knowledge and application abilities of LLMs in Hydro-SE have not been sufficiently evaluated. To address this issue, we propose the Hydro-SE LLM evaluation benchmark (Hydro-SE Bench), which contains 4,000 multiple-choice questions. Hydro-SE Bench covers nine subfields and enables evaluation of LLMs in aspects of basic conceptual knowledge, engineering application ability, and reasoning and calculation ability. The evaluation results on Hydro-SE Bench show that the accuracy values vary among 0.74 to 0.80 for commercial LLMs, and among 0.41 to 0.68 for small-parameter LLMs. While LLMs perform well in subfields closely related to natural and physical sciences, they struggle with domain-specific knowledge such as industry standards and hydraulic structures. Model scaling mainly improves reasoning and calculation abilities, but there is still great potential for LLMs to better handle problems in practical engineering application. This study highlights the strengths and weaknesses of LLMs for Hydro-SE tasks, providing model developers with clear training targets and Hydro-SE researchers with practical guidance for applying LLMs.
comment : Hydro-SE Bench sets a new benchmark for the evaluation of LLMs in the Hydro-Science and Engineering domain, with its code and data available at \url{https://github.com/sheishijun/Hydro-SE-Bench}
☆ Generative AI Practices, Literacy, and Divides: An Empirical Analysis in the Italian Context
Beatrice Savoldi, Giuseppe Attanasio, Olga Gorodetskaya, Marta Marchiori Manerba, Elisa Bassignana, Silvia Casola, Matteo Negri, Tommaso Caselli, Luisa Bentivogli, Alan Ramponi, Arianna Muti, Nicoletta Balbo, Debora Nozza
The rise of Artificial Intelligence (AI) language technologies, particularly generative AI (GenAI) chatbots accessible via conversational interfaces, is transforming digital interactions. While these tools hold societal promise, they also risk widening digital divides due to uneven adoption and low awareness of their limitations. This study presents the first comprehensive empirical mapping of GenAI adoption, usage patterns, and literacy in Italy, based on newly collected survey data from 1,906 Italian-speaking adults. Our findings reveal widespread adoption for both work and personal use, including sensitive tasks like emotional support and medical advice. Crucially, GenAI is supplanting other technologies to become a primary information source: this trend persists despite low user digital literacy, posing a risk as users struggle to recognize errors or misinformation. Moreover, we identify a significant gender divide -- particularly pronounced in older generations -- where women are half as likely to adopt GenAI and use it less frequently than men. While we find literacy to be a key predictor of adoption, it only partially explains this disparity, suggesting that other barriers are at play. Overall, our data provide granular insights into the multipurpose usage of GenAI, highlighting the dual need for targeted educational initiatives and further investigation into the underlying barriers to equitable participation that competence alone cannot explain.
☆ Optical Context Compression Is Just (Bad) Autoencoding
Ivan Yee Lee, Cheng Yang, Taylor Berg-Kirkpatrick
DeepSeek-OCR demonstrates that rendered text can be reconstructed with high fidelity from a small number of vision tokens. This finding has sparked excitement about vision-based context compression for language models. But the evaluation stops at reconstruction; whether these representations help language modeling remains untested. We test two assumptions implicit in the optical-compression narrative: that vision-based compression provides unique advantages for text reconstruction from compressed representations, and that DeepSeek-OCR's reconstruction results are evidence that vision-based compression will be useful for language modeling. Comparing their vision encoder against simple alternatives--parameter-free mean pooling and a learned hierarchical encoder--we find that these simple approaches match or surpass vision for reconstruction at matched compression ratios, and outperform it for language modeling--where vision-based compression fails to beat truncation. The excitement around optical context compression outpaces the evidence. Code and checkpoints are available at https://github.com/ivnle/bad-autoencoding
☆ AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment
Ahmad Aghaebrahimian
Large Language Models have significantly advanced natural language processing tasks, but remain prone to generating incorrect or misleading but plausible arguments. This issue, known as hallucination, is particularly concerning in high-stakes domains like clinical applications, where factual inaccuracies can have severe consequences. Existing evaluation metrics fail to adequately assess factual consistency and lack interpretability, making diagnosing and mitigating errors difficult. We propose an interpretable framework for factual consistency assessment for in-domain and open-domain texts to address these limitations. Our approach decomposes text into atomic facts and introduces a flexible, schema-free methodology. Unlike previous methods with an absolute metric, we incorporate a weighted metric to enhance factual evaluation. Additionally, we propose a mechanism to control assessment complexity in intricate domains. We benchmark our approach on popular general and clinical datasets and release our code to support fact-aware model training in future research.
☆ SELF: A Robust Singular Value and Eigenvalue Approach for LLM Fingerprinting
Hanxiu Zhang, Yue Zheng
The protection of Intellectual Property (IP) in Large Language Models (LLMs) represents a critical challenge in contemporary AI research. While fingerprinting techniques have emerged as a fundamental mechanism for detecting unauthorized model usage, existing methods -- whether behavior-based or structural -- suffer from vulnerabilities such as false claim attacks or susceptible to weight manipulations. To overcome these limitations, we propose SELF, a novel intrinsic weight-based fingerprinting scheme that eliminates dependency on input and inherently resists false claims. SELF achieves robust IP protection through two key innovations: 1) unique, scalable and transformation-invariant fingerprint extraction via singular value and eigenvalue decomposition of LLM attention weights, and 2) effective neural network-based fingerprint similarity comparison based on few-shot learning and data augmentation. Experimental results demonstrate SELF maintains high IP infringement detection accuracy while showing strong robustness against various downstream modifications, including quantization, pruning, and fine-tuning attacks. Our code is available at https://github.com/HanxiuZhang/SELF_v2.
☆ Fine-grained Narrative Classification in Biased News Articles
Zeba Afroz, Harsh Vardhan, Pawan Bhakuni, Aanchal Punia, Rajdeep Kumar, Md. Shad Akhtar
Narratives are the cognitive and emotional scaffolds of propaganda. They organize isolated persuasive techniques into coherent stories that justify actions, attribute blame, and evoke identification with ideological camps. In this paper, we propose a novel fine-grained narrative classification in biased news articles. We also explore article-bias classification as the precursor task to narrative classification and fine-grained persuasive technique identification. We develop INDI-PROP, the first ideologically grounded fine-grained narrative dataset with multi-level annotation for analyzing propaganda in Indian news media. Our dataset INDI-PROP comprises 1,266 articles focusing on two polarizing socio-political events in recent times: CAA and the Farmers' protest. Each article is annotated at three hierarchical levels: (i) ideological article-bias (pro-government, pro-opposition, neutral), (ii) event-specific fine-grained narrative frames anchored in ideological polarity and communicative intent, and (iii) persuasive techniques. We propose FANTA and TPTC, two GPT-4o-mini guided multi-hop prompt-based reasoning frameworks for the bias, narrative, and persuasive technique classification. FANTA leverages multi-layered communicative phenomena by integrating information extraction and contextual framing for hierarchical reasoning. On the other hand, TPTC adopts systematic decomposition of persuasive cues via a two-stage approach. Our evaluation suggests substantial improvement over underlying baselines in each case.
☆ CartoMapQA: A Fundamental Benchmark Dataset Evaluating Vision -Language Models on Cartographic Map Understanding SP
Huy Quang Ung, Guillaume Habault, Yasutaka Nishimura, Hao Niu, Roberto Legaspi, Tomoki Oya, Ryoichi Kojima, Masato Taya, Chihiro Ono, Atsunori Minamikawa, Yan Liu
The rise of Visual-Language Models (LVLMs) has unlocked new possibilities for seamlessly integrating visual and textual information. However, their ability to interpret cartographic maps remains largely unexplored. In this paper, we introduce CartoMapQA, a benchmark specifically designed to evaluate LVLMs' understanding of cartographic maps through question-answering tasks. The dataset includes over 2000 samples, each composed of a cartographic map, a question (with open-ended or multiple-choice answers), and a ground-truth answer. These tasks span key low-, mid- and high-level map interpretation skills, including symbol recognition, embedded information extraction, scale interpretation, and route-based reasoning. Our evaluation of both open-source and proprietary LVLMs reveals persistent challenges: models frequently struggle with map-specific semantics, exhibit limited geospatial reasoning, and are prone to Optical Character Recognition (OCR)-related errors. By isolating these weaknesses, CartoMapQA offers a valuable tool for guiding future improvements in LVLM architectures. Ultimately, it supports the development of models better equipped for real-world applications that depend on robust and reliable map understanding, such as navigation, geographic search, and urban planning. Our source code and data are openly available to the research community at: https://github.com/ungquanghuy-kddi/CartoMapQA.git
comment : Accepted at SIGSPATIAL 2025 (Best paper candidates), 15 pages
☆ M3DR: Towards Universal Multilingual Multimodal Document Retrieval
Adithya S Kolavi, Vyoman Jain
Multimodal document retrieval systems have shown strong progress in aligning visual and textual content for semantic search. However, most existing approaches remain heavily English-centric, limiting their effectiveness in multilingual contexts. In this work, we present M3DR (Multilingual Multimodal Document Retrieval), a framework designed to bridge this gap across languages, enabling applicability across diverse linguistic and cultural contexts. M3DR leverages synthetic multilingual document data and generalizes across different vision-language architectures and model sizes, enabling robust cross-lingual and cross-modal alignment. Using contrastive training, our models learn unified representations for text and document images that transfer effectively across languages. We validate this capability on 22 typologically diverse languages, demonstrating consistent performance and adaptability across linguistic and script variations. We further introduce a comprehensive benchmark that captures real-world multilingual scenarios, evaluating models under monolingual, multilingual, and mixed-language settings. M3DR generalizes across both single dense vector and ColBERT-style token-level multi-vector retrieval paradigms. Our models, NetraEmbed and ColNetraEmbed achieve state-of-the-art performance with ~150% relative improvements on cross-lingual retrieval.
☆ Understanding LLM Reasoning for Abstractive Summarization
Haohan Yuan, Siu Cheung Hui, Haopeng Zhang
While the reasoning capabilities of Large Language Models (LLMs) excel in analytical tasks such as mathematics and code generation, their utility for abstractive summarization remains widely assumed but largely unverified. To bridge this gap, we first tailor general reasoning strategies to the summarization domain. We then conduct a systematic, large scale comparative study of 8 reasoning strategies and 3 Large Reasoning Models (LRMs) across 8 diverse datasets, assessing both summary quality and faithfulness. Our findings show that reasoning is not a universal solution and its effectiveness is highly dependent on the specific strategy and context. Specifically, we observe a trade-off between summary quality and factual faithfulness: explicit reasoning strategies tend to improve fluency at the expense of factual grounding, while implicit reasoning in LRMs exhibits the inverse pattern. Furthermore, increasing an LRM's internal reasoning budget does not improve, and can even hurt, factual consistency, suggesting that effective summarization demands faithful compression rather than creative over-thinking.
comment : 26 pages,15 figures
☆ NAS-LoRA: Empowering Parameter-Efficient Fine-Tuning for Visual Foundation Models with Searchable Adaptation
Renqi Chen, Haoyang Su, Shixiang Tang
The Segment Anything Model (SAM) has emerged as a powerful visual foundation model for image segmentation. However, adapting SAM to specific downstream tasks, such as medical and agricultural imaging, remains a significant challenge. To address this, Low-Rank Adaptation (LoRA) and its variants have been widely employed to enhancing SAM's adaptation performance on diverse domains. Despite advancements, a critical question arises: can we integrate inductive bias into the model? This is particularly relevant since the Transformer encoder in SAM inherently lacks spatial priors within image patches, potentially hindering the acquisition of high-level semantic information. In this paper, we propose NAS-LoRA, a new Parameter-Efficient Fine-Tuning (PEFT) method designed to bridge the semantic gap between pre-trained SAM and specialized domains. Specifically, NAS-LoRA incorporates a lightweight Neural Architecture Search (NAS) block between the encoder and decoder components of LoRA to dynamically optimize the prior knowledge integrated into weight updates. Furthermore, we propose a stage-wise optimization strategy to help the ViT encoder balance weight updates and architectural adjustments, facilitating the gradual learning of high-level semantic information. Various Experiments demonstrate our NAS-LoRA improves existing PEFT methods, while reducing training cost by 24.14% without increasing inference cost, highlighting the potential of NAS in enhancing PEFT for visual foundation models.
☆ A Preliminary Study on the Promises and Challenges of Native Top-$k$ Sparse Attention
Di Xiu, Hongyin Tang, Bolin Rong, Lizhi Yan, Jingang Wang, Yifan Lu, Xunliang Cai
Large Language Models (LLMs) are increasingly prevalent in the field of long-context modeling, however, their inference computational costs have become a critical bottleneck hindering the advancement of tasks such as agents and multimodal applications. This report conducts a preliminary investigation into the effectiveness and theoretical mechanisms of the Top-$k$ Attention mechanism during both the decoding and training phases. First, we validate the effectiveness of exact Top-$k$ Decoding through extensive experimentation. Experiments demonstrate that retaining only the pivotal Keys with the highest similarity to the Query as the context window during the decoding stage achieves performance comparable to, or even surpassing, full attention on downstream tasks such as HELMET and LongBench v2. Second, we further explore the native Top-$k$ Attention training strategy. Experiments confirm that ensuring the consistency between training and inference regarding Top-$k$ Attention operations facilitates the further unlocking of Top-$k$ Decoding's potential, thereby significantly enhancing model performance. Furthermore, considering the high computational complexity of exact Top-$k$ Attention, we investigate the impact of approximate Top-$k$ algorithm precision on downstream tasks. Our research confirms a positive correlation between downstream task performance and approximation fidelity, and we provide statistical evaluations of the Lightning Indexer's precision within the DeepSeek-V3.2-Exp model. Finally, this report provides a theoretical interpretation from the perspective of Entropy. Experimental observations indicate that models subjected to Top-$k$ Attention SFT exhibit a distinct phenomenon of entropy reduction in downstream tasks, which validates the hypothesis that low-entropy states are better adapted to Top-$k$ Decoding.
☆ Tuning for TraceTarnish: Techniques, Trends, and Testing Tangible Traits
Robert Dilworth
In this study, we more rigorously evaluated our attack script $\textit{TraceTarnish}$, which leverages adversarial stylometry principles to anonymize the authorship of text-based messages. To ensure the efficacy and utility of our attack, we sourced, processed, and analyzed Reddit comments--comments that were later alchemized into $\textit{TraceTarnish}$ data--to gain valuable insights. The transformed $\textit{TraceTarnish}$ data was then further augmented by $\textit{StyloMetrix}$ to manufacture stylometric features--features that were culled using the Information Gain criterion, leaving only the most informative, predictive, and discriminative ones. Our results found that function words and function word types ($L\_FUNC\_A$ $\&$ $L\_FUNC\_T$); content words and content word types ($L\_CONT\_A$ $\&$ $L\_CONT\_T$); and the Type-Token Ratio ($ST\_TYPE\_TOKEN\_RATIO\_LEMMAS$) yielded significant Information-Gain readings. The identified stylometric cues--function-word frequencies, content-word distributions, and the Type-Token Ratio--serve as reliable indicators of compromise (IoCs), revealing when a text has been deliberately altered to mask its true author. Similarly, these features could function as forensic beacons, alerting defenders to the presence of an adversarial stylometry attack; granted, in the absence of the original message, this signal may go largely unnoticed, as it appears to depend on a pre- and post-transformation comparison. "In trying to erase a trace, you often imprint a larger one." Armed with this understanding, we framed $\textit{TraceTarnish}$'s operations and outputs around these five isolated features, using them to conceptualize and implement enhancements that further strengthen the attack.
comment : 20 pages, 8 figures, 2 tables
☆ Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision -Language Models
Shojiro Yamabe, Futa Waseda, Daiki Shiono, Tsubasa Takahashi
Recent large vision-language models (LVLMs) have been applied to diverse VQA tasks. However, achieving practical performance typically requires task-specific fine-tuning with large numbers of image-text pairs, which are costly to collect. In this work, we study text-centric training, a setting where only textual descriptions are available and no real images are provided, as a paradigm for low-cost data scaling. Unlike images, whose collection is often restricted by privacy constraints and scarcity in niche domains, text is widely available. Moreover, text is easily editable, enabling automatic diversification and expansion with LLMs at minimal human effort. While this offers clear advantages over image collection in terms of scalability and cost, training on raw text without images still yields limited gains on VQA tasks because of the image-text modality gap. To address this issue, we propose a Text-Printed Image (TPI), which generates synthetic images by directly rendering the given textual description on a plain white canvas. This simple rendering projects text into the image modality and can be integrated into arbitrary existing LVLM training pipelines at low cost. Moreover, TPI preserves the semantics of the text, whereas text-to-image models often fail to do. Across four models and seven benchmarks, our systematic experiments show that TPI enables more effective text-centric training than synthetic images generated by a diffusion model. We further explore TPI as a low-cost data-augmentation strategy and demonstrate its practical utility. Overall, our findings highlight the significant potential of text-centric training and, more broadly, chart a path toward fully automated data generation for LVLMs.
☆ Pretrain Zero: Reinforcement Active Pretrain ing
Xingrun Xing, Zhiyuan Fan, Jie Lou, Guoqi Li, Jiajun Zhang, Debing Zhang
Mimicking human behavior to actively learning from general experience and achieve artificial general intelligence has always been a human dream. Recent reinforcement learning (RL) based large-thinking models demonstrate impressive expert-level abilities, i.e., software and math, but still rely heavily on verifiable rewards in specific domains, placing a significant bottleneck to extend the performance boundary of general reasoning capabilities. In this work, we propose PretrainZero, a reinforcement active learning framework built on the pretraining corpus to extend RL from domain-specific post-training to general pretraining. PretrainZero features the following characteristics: 1) Active pretraining: inspired by the active learning ability of humans, PretrainZero learns a unified reasoning policy to actively identify reasonable and informative contents from pretraining corpus, and reason to predict these contents by RL. 2) Self-supervised learning: without any verifiable labels, pretrained reward models, or supervised fine-tuning, we directly pretrain reasoners from 3 to 30B base models on the general Wikipedia corpus using RL, significantly breaking the verification data-wall for general reasoning. 3) Verification scaling: by tackling increasingly challenging masked spans, PretrainZero substantially enhances the general reasoning abilities of pretrained base models. In reinforcement pretraining, PretrainZero improves Qwen3-4B-Base for 8.43, 5.96 and 10.60 on MMLU-Pro, SuperGPQA and math average benchmarks. In post-training, the pretrained models can also serve as reasoning foundation models for downstream RLVR tasks.
☆ Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
Yixing Xu, Chao Li, Xuanwu Yin, Spandan Tiwari, Dong Li, Ashish Sirasao, Emad Barsoum
Low-rank adaptation (LoRA) is one of the most popular methods among parameter-efficient fine-tuning (PEFT) methods to adapt pre-trained large language models (LLMs) to specific downstream tasks. However, the model trained based on LoRA often has an unsatisfactory performance due to its low-rank assumption. In this paper, we propose a novel method called Dual LoRA to improve the performance by incorporating an inductive bias into the original LoRA. Specifically, we separate low-rank matrices into two groups: the magnitude group to control whether or not and how far we should update a parameter and the direction group to decide whether this parameter should move forward or backward, to better simulate the parameter updating process of the full fine-tuning based on gradient-based optimization algorithms. We show that this can be simply achieved by adding a ReLU function to the magnitude group and a sign function to the direction group. We conduct several experiments over a wide range of NLP tasks, including natural language generation (NLG), understanding (NLU), and commonsense reasoning datasets on GPT-2, RoBERTa, DeBERTa, and LLaMA-1/2/3 as baseline models. The results show that we consistently outperform LoRA and its state-of-the-art variants with the same number of trainable parameters.
☆ Characterizing Language Use in a Collaborative Situated Game
Nicholas Tomlin, Naitian Zhou, Eve Fleisig, Liangyuan, Chen, Téa Wright, Lauren Vinh, Laura X. Ma, Seun Eisape, Ellie French, Tingting Du, Tianjiao Zhang, Alexander Koller, Alane Suhr
Cooperative video games, where multiple participants must coordinate by communicating and reasoning under uncertainty in complex environments, yield a rich source of language data. We collect the Portal Dialogue Corpus: a corpus of 11.5 hours of spoken human dialogue in the co-op mode of the popular Portal 2 virtual puzzle game, comprising 24.5K total utterances. We analyze player language and behavior, identifying a number of linguistic phenomena that rarely appear in most existing chitchat or task-oriented dialogue corpora, including complex spatial reference, clarification and repair, and ad-hoc convention formation. To support future analyses of language use in complex, situated, collaborative problem-solving scenarios, we publicly release the corpus, which comprises player videos, audio, transcripts, game state data, and both manual and automatic annotations of language data.
☆ Nexus: Higher-Order Attention Mechanisms in Transformer s
Hanting Chen, Chu Zhong, Kai Han, Yuchuan Tian, Yuchen Liang, Tianyu Guo, Xinghao Chen, Dacheng Tao, Yunhe Wang
Transformers have achieved significant success across various domains, relying on self-attention to capture dependencies. However, the standard first-order attention mechanism is often limited by a low-rank bottleneck, struggling to capture intricate, multi-hop relationships within a single layer. In this paper, we propose the \textbf{Higher-Order Attention Network (Hon)}, a novel architecture designed to enhance representational power through a recursive framework. Unlike standard approaches that use static linear projections for Queries and Keys, Hon dynamically refines these representations via nested self-attention mechanisms. Specifically, the Query and Key vectors are themselves outputs of inner attention loops, allowing tokens to aggregate global context and model high-order correlations \textit{prior} to the final attention computation. We enforce a parameter-efficient weight-sharing strategy across recursive steps, ensuring that this enhanced expressivity incurs $\mathcal{O}(1)$ additional parameters. We provide theoretical analysis demonstrating that our method breaks the linear bottleneck of standard attention. Empirically, Hon outperforms standard Transformers on multiple benchmarks.
☆ LLM-Generated Ads: From Personalization Parity to Persuasion Superiority
Elyas Meguellati, Stefano Civelli, Lei Han, Abraham Bernstein, Shazia Sadiq, Gianluca Demartini
As large language models (LLMs) become increasingly capable of generating persuasive content, understanding their effectiveness across different advertising strategies becomes critical. This paper presents a two-part investigation examining LLM-generated advertising through complementary lenses: (1) personality-based and (2) psychological persuasion principles. In our first study (n=400), we tested whether LLMs could generate personalized advertisements tailored to specific personality traits (openness and neuroticism) and how their performance compared to human experts. Results showed that LLM-generated ads achieved statistical parity with human-written ads (51.1% vs. 48.9%, p > 0.05), with no significant performance differences for matched personalities. Building on these insights, our second study (n=800) shifted focus from individual personalization to universal persuasion, testing LLM performance across four foundational psychological principles: authority, consensus, cognition, and scarcity. AI-generated ads significantly outperformed human-created content, achieving a 59.1% preference rate (vs. 40.9%, p < 0.001), with the strongest performance in authority (63.0%) and consensus (62.5%) appeals. Qualitative analysis revealed AI's advantage stems from crafting more sophisticated, aspirational messages and achieving superior visual-narrative coherence. Critically, this quality advantage proved robust: even after applying a 21.2 percentage point detection penalty when participants correctly identified AI-origin, AI ads still outperformed human ads, and 29.4% of participants chose AI content despite knowing its origin. These findings demonstrate LLMs' evolution from parity in personalization to superiority in persuasive storytelling, with significant implications for advertising practice given LLMs' near-zero marginal cost and time requirements compared to human experts.
☆ From Hypothesis to Premises: LLM-based Backward Logical Reasoning with Selective Symbolic Translation AAAI2026
Qingchuan Li, Mingyue Cheng, Zirui Liu, Daoyu Wang, Yuting Zeng, Tongxuan Liu
Logical reasoning is a core challenge in natural language understanding and a fundamental capability of artificial intelligence, underpinning scientific discovery, mathematical theorem proving, and complex decision-making. Despite the remarkable progress of large language models (LLMs), most current approaches still rely on forward reasoning paradigms, generating step-by-step rationales from premises to conclusions. However, such methods often suffer from redundant inference paths, hallucinated steps, and semantic drift, resulting in inefficient and unreliable reasoning. In this paper, we propose a novel framework, Hypothesis-driven Backward Logical Reasoning (HBLR). The core idea is to integrate confidence-aware symbolic translation with hypothesis-driven backward reasoning. In the translation phase, only high-confidence spans are converted into logical form, such as First-Order Logic (FOL), while uncertain content remains in natural language. A translation reflection module further ensures semantic fidelity by evaluating symbolic outputs and reverting lossy ones back to text when necessary. In the reasoning phase, HBLR simulates human deductive thinking by assuming the conclusion is true and recursively verifying its premises. A reasoning reflection module further identifies and corrects flawed inference steps, enhancing logical coherence. Extensive experiments on five reasoning benchmarks demonstrate that HBLR consistently outperforms strong baselines in both accuracy and efficiency.
comment : Accepted by AAAI2026
☆ Idea-Gated Transformer s: Enforcing Semantic Coherence via Differentiable Vocabulary Pruning
Darshan Fofadiya
Autoregressive Language Models (LLMs) trained on Next-Token Prediction (NTP) often suffer from ``Topic Drift'' where the generation wanders away from the initial prompt due to a reliance on local associations rather than global planning \citep{holtzman2019curious}. While scaling model size mitigates this \citep{brown2020language}, the fundamental myopia of the NTP objective remains. In this work, we introduce the Idea-Gated Transformer, a novel architecture that separates semantic planning from syntactic generation. We introduce an auxiliary ``Idea Head'' trained to predict the bag-of-words distribution for a future context window, creating a latent ``Concept Vector'' that actively gates the main vocabulary during generation. We propose a differentiable gating mechanism that suppresses semantically irrelevant tokens, effectively pruning the search space in real-time. Experiments on WikiText-103 demonstrate that while the Idea-Gated model achieves comparable validation perplexity to a standard GPT-2 baseline, it exhibits significantly superior Domain Retention. Qualitative and quantitative analysis reveals that the gating mechanism successfully locks generation into specific semantic clusters (e.g., Finance, Science) and resists associative drift, offering a parameter-efficient path toward more controllable language modeling.
comment : Code available at https://github.com/DarshanFofadiya/idea-gated-transformers/tree/main
☆ PERCS: Persona-Guided Controllable Biomedical Summarization Dataset
Rohan Charudatt Salvi, Chirag Chawla, Dhruv Jain, Swapnil Panigrahi, Md Shad Akhtar, Shweta Yadav
Automatic medical text simplification plays a key role in improving health literacy by making complex biomedical research accessible to diverse readers. However, most existing resources assume a single generic audience, overlooking the wide variation in medical literacy and information needs across user groups. To address this limitation, we introduce PERCS (Persona-guided Controllable Summarization), a dataset of biomedical abstracts paired with summaries tailored to four personas: Laypersons, Premedical Students, Non-medical Researchers, and Medical Experts. These personas represent different levels of medical literacy and information needs, emphasizing the need for targeted, audience-specific summarization. Each summary in PERCS was reviewed by physicians for factual accuracy and persona alignment using a detailed error taxonomy. Technical validation shows clear differences in readability, vocabulary, and content depth across personas. Along with describing the dataset, we benchmark four large language models on PERCS using automatic evaluation metrics that assess comprehensiveness, readability, and faithfulness, establishing baseline results for future research. The dataset, annotation guidelines, and evaluation materials are publicly available to support research on persona-specific communication and controllable biomedical summarization.
comment : 9 pages, 4 figures, 6 tables
☆ Epistemic Substitution: How Grokipedia's AI-Generated Encyclopedia Restructures Authority
Aliakbar Mehdizadeh, Martin Hilbert
A quarter century ago, Wikipedia's decentralized, crowdsourced, and consensus-driven model replaced the centralized, expert-driven, and authority-based standard for encyclopedic knowledge curation. The emergence of generative AI encyclopedias, such as Grokipedia, possibly presents another potential shift in epistemic evolution. This study investigates whether AI- and human-curated encyclopedias rely on the same foundations of authority. We conducted a multi-scale comparative analysis of the citation networks from 72 matched article pairs, which cite a total of almost 60,000 sources. Using an 8-category epistemic classification, we mapped the "epistemic profiles" of the articles on each platform. Our findings reveal several quantitative and qualitative differences in how knowledge is sourced and encyclopedia claims are epistemologically justified. Grokipedia replaces Wikipedia's heavy reliance on peer-reviewed "Academic & Scholarly" work with a notable increase in "User-generated" and "Civic organization" sources. Comparative network analyses further show that Grokipedia employs very different epistemological profiles when sourcing leisure topics (such as Sports and Entertainment) and more societal sensitive civic topics (such as Politics & Conflicts, Geographical Entities, and General Knowledge & Society). Finally, we find a "scaling-law for AI-generated knowledge sourcing" that shows a linear relationship between article length and citation density, which is distinct from collective human reference sourcing. We conclude that this first implementation of an LLM-based encyclopedia does not merely automate knowledge production but restructures it. Given the notable changes and the important role of encyclopedias, we suggest the continuation and deepening of algorithm audits, such as the one presented here, in order to understand the ongoing epistemological shifts.
☆ Modeling Topics and Sociolinguistic Variation in Code-Switched Discourse: Insights from Spanish-English and Spanish-Guaraní
Nemika Tyagi, Nelvin Licona Guevara, Olga Kellert
This study presents an LLM-assisted annotation pipeline for the sociolinguistic and topical analysis of bilingual discourse in two typologically distinct contexts: Spanish-English and Spanish-Guaraní. Using large language models, we automatically labeled topic, genre, and discourse-pragmatic functions across a total of 3,691 code-switched sentences, integrated demographic metadata from the Miami Bilingual Corpus, and enriched the Spanish-Guaraní dataset with new topic annotations. The resulting distributions reveal systematic links between gender, language dominance, and discourse function in the Miami data, and a clear diglossic division between formal Guaraní and informal Spanish in Paraguayan texts. These findings replicate and extend earlier interactional and sociolinguistic observations with corpus-scale quantitative evidence. The study demonstrates that large language models can reliably recover interpretable sociolinguistic patterns traditionally accessible only through manual annotation, advancing computational methods for cross-linguistic and low-resource bilingual research.
comment : 10 pages, 4 figures
☆ DAComp: Benchmark ing Data Agent s across the Full Data Intelligence Lifecycle
Fangyu Lei, Jinxiang Meng, Yiming Huang, Junjie Zhao, Yitong Zhang, Jianwen Luo, Xin Zou, Ruiyi Yang, Wenbo Shi, Yan Gao, Shizhu He, Zuo Wang, Qian Liu, Yang Wang, Ke Wang, Jun Zhao, Kang Liu
Real-world enterprise data intelligence workflows encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-oriented insights. We introduce DAComp, a benchmark of 210 tasks that mirrors these complex workflows. Data engineering (DE) tasks require repository-level engineering on industrial schemas, including designing and building multi-stage SQL pipelines from scratch and evolving existing systems under evolving requirements. Data analysis (DA) tasks pose open-ended business problems that demand strategic planning, exploratory analysis through iterative coding, interpretation of intermediate results, and the synthesis of actionable recommendations. Engineering tasks are scored through execution-based, multi-metric evaluation. Open-ended tasks are assessed by a reliable, experimentally validated LLM-judge, which is guided by hierarchical, meticulously crafted rubrics. Our experiments reveal that even state-of-the-art agents falter on DAComp. Performance on DE tasks is particularly low, with success rates under 20%, exposing a critical bottleneck in holistic pipeline orchestration, not merely code generation. Scores on DA tasks also average below 40%, highlighting profound deficiencies in open-ended reasoning and demonstrating that engineering and analysis are distinct capabilities. By clearly diagnosing these limitations, DAComp provides a rigorous and realistic testbed to drive the development of truly capable autonomous data agents for enterprise settings. Our data and code are available at https://da-comp.github.io
☆ Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correction CVPR 2026
Rui Fonseca, Bruno Martins, Gil Rocha
Image captioning has drawn considerable attention from the natural language processing and computer vision fields. Aiming to reduce the reliance on curated data, several studies have explored image captioning without any humanly-annotated image-text pairs for training, although existing methods are still outperformed by fully supervised approaches. This paper proposes TOMCap, i.e., an improved text-only training method that performs captioning without the need for aligned image-caption pairs. The method is based on prompting a pre-trained language model decoder with information derived from a CLIP representation, after undergoing a process to reduce the modality gap. We specifically tested the combined use of retrieved examples of captions, and latent vector representations, to guide the generation process. Through extensive experiments, we show that TOMCap outperforms other training-free and text-only methods. We also analyze the impact of different choices regarding the configuration of the retrieval-augmentation and modality gap reduction components.
comment : Submitted to CVPR 2026
☆ SQuARE: Structured Query & Adaptive Retrieval Engine For Tabular Formats
Chinmay Gondhalekar, Urjitkumar Patel, Fang-Chun Yeh
Accurate question answering over real spreadsheets remains difficult due to multirow headers, merged cells, and unit annotations that disrupt naive chunking, while rigid SQL views fail on files lacking consistent schemas. We present SQuARE, a hybrid retrieval framework with sheet-level, complexity-aware routing. It computes a continuous score based on header depth and merge density, then routes queries either through structure-preserving chunk retrieval or SQL over an automatically constructed relational representation. A lightweight agent supervises retrieval, refinement, or combination of results across both paths when confidence is low. This design maintains header hierarchies, time labels, and units, ensuring that returned values are faithful to the original cells and straightforward to verify. Evaluated on multi-header corporate balance sheets, a heavily merged World Bank workbook, and diverse public datasets, SQuARE consistently surpasses single-strategy baselines and ChatGPT-4o on both retrieval precision and end-to-end answer accuracy while keeping latency predictable. By decoupling retrieval from model choice, the system is compatible with emerging tabular foundation models and offers a practical bridge toward a more robust table understanding.
comment : Accepted in The IEEE International Workshop on Large Language Models in Finance, Dec 8-11, Macau, China, 2025, Preprint Copy
☆ Computational Linguistics Meets Libyan Dialect: A Study on Dialect Identification
Mansour Essgaer, Khamis Massud, Rabia Al Mamlook, Najah Ghmaid
This study investigates logistic regression, linear support vector machine, multinomial Naive Bayes, and Bernoulli Naive Bayes for classifying Libyan dialect utterances gathered from Twitter. The dataset used is the QADI corpus, which consists of 540,000 sentences across 18 Arabic dialects. Preprocessing challenges include handling inconsistent orthographic variations and non-standard spellings typical of the Libyan dialect. The chi-square analysis revealed that certain features, such as email mentions and emotion indicators, were not significantly associated with dialect classification and were thus excluded from further analysis. Two main experiments were conducted: (1) evaluating the significance of meta-features extracted from the corpus using the chi-square test and (2) assessing classifier performance using different word and character n-gram representations. The classification experiments showed that Multinomial Naive Bayes (MNB) achieved the highest accuracy of 85.89% and an F1-score of 0.85741 when using a (1,2) word n-gram and (1,5) character n-gram representation. In contrast, Logistic Regression and Linear SVM exhibited slightly lower performance, with maximum accuracies of 84.41% and 84.73%, respectively. Additional evaluation metrics, including log loss, Cohen kappa, and Matthew correlation coefficient, further supported the effectiveness of MNB in this task. The results indicate that carefully selected n-gram representations and classification models play a crucial role in improving the accuracy of Libyan dialect identification. This study provides empirical benchmarks and insights for future research in Arabic dialect NLP applications.
comment : 13 pages, 8 figures
☆ On GRPO Collapse in Search-R1: The Lazy Likelihood-Displacement Death Spiral
Wenlong Deng, Yushu Li, Boying Gong, Yi Ren, Christos Thrampoulidis, Xiaoxiao Li
Tool-integrated (TI) reinforcement learning (RL) enables large language models (LLMs) to perform multi-step reasoning by interacting with external tools such as search engines and retrievers. Group Relative Policy Optimization (GRPO), exemplified by the recent Search-R1, offers fast convergence and a value-free formulation that makes it appealing for this setting, yet consistently suffers from training collapse. We identify Lazy Likelihood Displacement (LLD), a systematic reduction or stagnation in the likelihood of both correct and incorrect responses, as the core mechanism driving this failure. LLD emerges early and triggers a self-reinforcing LLD Death Spiral, where declining likelihood leads to low-confidence responses, inflating gradients, and ultimately causing collapse. We empirically characterize this process across models on a Search-R1-style, search-integrated question answering task, revealing a consistent three-phase trajectory: early stagnation, steady decay, and accelerated collapse. To address this, we propose a lightweight likelihood-preserving regularization LLDS for GRPO that activates only when a trajectory's likelihood decreases, and regularizes only the tokens responsible. This fine-grained structure mitigates LLD with minimal interference to optimization. Across seven open-domain and multi-hop QA benchmarks, our method stabilizes training, prevents gradient explosion, and yields substantial performance improvements, including +37.8% gains on Qwen2.5-3B and +32.0% gains on Qwen2.5-7B. Our results establish LLD as a fundamental bottleneck in GRPO-based TIRL and provide a practical path toward stable, scalable training of tool-integrated LLM.
☆ Balancing Safety and Helpfulness in Healthcare AI Assistants through Iterative Preference Alignment ML4H 2025
Huy Nghiem, Swetasudha Panda, Devashish Khatwani, Huy V. Nguyen, Krishnaram Kenthapadi, Hal Daumé
Large Language Models (LLMs) are increasingly used in healthcare, yet ensuring their safety and trustworthiness remains a barrier to deployment. Conversational medical assistants must avoid unsafe compliance without over-refusing benign queries. We present an iterative post-deployment alignment framework that applies Kahneman-Tversky Optimization (KTO) and Direct Preference Optimization (DPO) to refine models against domain-specific safety signals. Using the CARES-18K benchmark for adversarial robustness, we evaluate four LLMs (Llama-3B/8B, Meditron-8B, Mistral-7B) across multiple cycles. Our results show up to 42% improvement in safety-related metrics for harmful query detection, alongside interesting trade-offs against erroneous refusals, thereby exposing architecture-dependent calibration biases. We also perform ablation studies to identify when self-evaluation is reliable and when external or finetuned judges are necessary to maximize performance gains. Our findings underscore the importance of adopting best practices that balance patient safety, user trust, and clinical utility in the design of conversational medical assistants.
comment : ML4H 2025 Proceedings, Best Paper Award
☆ Network of Theseus (like the ship)
Vighnesh Subramaniam, Colin Conwell, Boris Katz, Andrei Barbu, Brian Cheung
A standard assumption in deep learning is that the inductive bias introduced by a neural network architecture must persist from training through inference. The architecture you train with is the architecture you deploy. This assumption constrains the community from selecting architectures that may have desirable efficiency or design properties due to difficulties with optimization. We challenge this assumption with Network of Theseus (NoT), a method for progressively converting a trained, or even untrained, guide network architecture part-by-part into an entirely different target network architecture while preserving the performance of the guide network. At each stage, components in the guide network architecture are incrementally replaced with target architecture modules and aligned via representational similarity metrics. This procedure largely preserves the functionality of the guide network even under substantial architectural changes-for example, converting a convolutional network into a multilayer perceptron, or GPT-2 into a recurrent neural network. By decoupling optimization from deployment, NoT expands the space of viable inference-time architectures, opening opportunities for better accuracy-efficiency tradeoffs and enabling more directed exploration of the architectural design space.
comment : Preprint. 24 pages, 9 figures, 8 tables
♻ ☆ Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models NeurIPS 2025
Tyler A. Chang, Benjamin K. Bergen
In Transformer language models, activation vectors transform from current token embeddings to next token predictions as they pass through the model. To isolate a minimal form of this transformation, we identify language model subnetworks that make bigram predictions, naive next token predictions based only on the current token. We find that bigram subnetworks can be found in fully trained language models up to 1B parameters, and these subnetworks are critical for model performance even when they consist of less than 0.2% of model parameters. Bigram subnetworks are concentrated in the first Transformer MLP layer, and they overlap significantly with subnetworks trained to optimally prune a given model. Mechanistically, the bigram subnetworks often recreate a pattern from the full models where the first layer induces a sharp change that aligns activations with next token predictions rather than current token representations. Our results demonstrate that bigram subnetworks comprise a minimal subset of parameters that are both necessary and sufficient for basic next token predictions in language models, and they help drive the transformation from current to next token activations in the residual stream. These subnetworks can lay a foundation for studying more complex language model circuits by building up from a minimal circuit.
comment : NeurIPS 2025
♻ ☆ How to Train Long- Context Language Models (Effectively) ACL 2025
Tianyu Gao, Alexander Wettig, Howard Yen, Danqi Chen
We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information. We first establish a reliable evaluation protocol to guide model development -- instead of perplexity or simple needle-in-a-haystack (NIAH) tests, we use a broad set of long-context downstream tasks, and we evaluate models after SFT as this better reveals long-context abilities. Supported by our robust evaluations, we run thorough experiments to decide the data mix for continued pre-training, the instruction tuning dataset, and many other design choices such as position extrapolation. We find that (1) code repositories and books are excellent sources of long data, but it is crucial to combine them with high-quality short-context data; (2) training with a sequence length beyond the evaluation length boosts long-context performance; (3) for SFT, using only short instruction datasets yields strong performance on long-context tasks. Our final model, ProLong-8B, which is initialized from Llama-3 and trained on 40B tokens, demonstrates state-of-the-art long-context performance among similarly sized models at a length of 128K. ProLong outperforms Llama-3.1-8B-Instruct on the majority of long-context tasks despite using only 5% as many tokens during long-context training. Additionally, ProLong can effectively process up to 512K tokens, one of the longest context windows of publicly available LMs.
comment : Accepted to ACL 2025. Our code, data, and models are available at https://github.com/princeton-nlp/ProLong
♻ ☆ Exact Coset Sampling for Quantum Lattice Algorithms
Yifan Zhang
In this work, we give a new completion of Chen's windowed-QFT lattice algorithm~\citep{chen2024quantum}. This extra step, called Step~$9^\dagger$, replaces the domain extension stage in Steps~8--9. The published Step~9 calls an amplitude periodicity lemma, yet its hypotheses break in the presence of affine offsets $\boldsymbol{v}^*$. Our analysis finds a basic conflict between two design constraints. The lattice problem asks for high spectral resolution, so the method prefers wide time windows. The quadratic phase error of the state prefers narrow time windows. Assumption~A5 packages the spectral concentration and near-uniformity properties that we require from the front end. Under~A5, a direct $\mathbb{Z}_M^n$ Fourier transform of the chirp-corrected coordinate state produces samples $\boldsymbol{u}$ that satisfy $\langle \boldsymbol{b}, \boldsymbol{u} \rangle \equiv 0 \pmod{Q}$ with probability $1-\mathrm{negl}(n)$ and are nearly uniform on the dual hyperplane $\{\boldsymbol{u} : \langle \boldsymbol{b}, \boldsymbol{u} \rangle \equiv 0 \pmod{Q}\}$. The new procedure does not require internal access to control wires. It uses the normalization $b_1=-1$ to apply a center-referenced phase correction directly on the first coordinate register. The scaling parameter $D$ ensures that this physical operation can be implemented by arithmetic on $X_1$ alone and does not read the hidden loop index. For Chen's complex-Gaussian Karst-wave window, we isolate a parameter regime, formalized in Assumption~A5, in which a polynomial retuning of the parameters gives a one-dimensional envelope for the loop index with width $σ_J \asymp Q\log n$.
comment : Preprint - Work in Progress
♻ ☆ Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
Jack Cook, Junxian Guo, Guangxuan Xiao, Yujun Lin, Song Han
As large language models have grown larger, low-precision numerical formats such as NVFP4 have become increasingly popular due to the speed and memory benefits they provide. However, to accelerate computation with NVFP4, all matrix multiplication operands--weights and activations in the forward pass, and weights, activations, and gradients in the backward pass--must be quantized to NVFP4, often leading to divergence during training and performance degradation during inference. To address this issue, in this work we introduce Four Over Six (4/6), a modification to the NVFP4 quantization algorithm that evaluates two potential scale factors for each block of values. Unlike integer formats, floating-point formats such as FP4 have the most quantization error on near-maximal values in each block, which we find to be primarily responsible for downstream performance degradation. We find that for some blocks, scaling to smaller FP4 values makes the distribution of representable values more uniform, improving representation of near-maximal values. Importantly, 4/6 can be implemented efficiently on NVIDIA Blackwell GPUs, making it viable to use while training LLMs with NVFP4. In pre-training experiments with transformer and hybrid model architectures, we find that 4/6 prevents divergence in several cases, bringing training loss significantly closer to BF16 compared to models trained with current state-of-the-art NVFP4 training recipes. We also find that 4/6 can be easily incorporated into many different post-training quantization methods and generally improves downstream accuracy. We hope this inspires future work in training and deploying models with NVFP4. Our code is available at http://github.com/mit-han-lab/fouroversix.
comment : 10 pages, 5 figures
♻ ☆ From Code Foundation Models to Agent s and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence
Jian Yang, Xianglong Liu, Weifeng Lv, Ken Deng, Shawn Guo, Lin Jing, Yizhi Li, Shark Liu, Xianzhen Luo, Yuyu Luo, Changzai Pan, Ensheng Shi, Yingshui Tan, Renshuai Tao, Jiajun Wu, Xianjie Wu, Zhenhe Wu, Daoguang Zan, Chenchen Zhang, Wei Zhang, He Zhu, Terry Yue Zhuo, Kerui Cao, Xianfu Cheng, Jun Dong, Shengjie Fang, Zhiwei Fei, Xiangyuan Guan, Qipeng Guo, Zhiguang Han, Joseph James, Tianqi Luo, Renyuan Li, Yuhang Li, Yiming Liang, Congnan Liu, Jiaheng Liu, Qian Liu, Ruitong Liu, Tyler Loakman, Xiangxin Meng, Chuang Peng, Tianhao Peng, Jiajun Shi, Mingjie Tang, Boyang Wang, Haowen Wang, Yunli Wang, Fanglin Xu, Zihan Xu, Fei Yuan, Ge Zhang, Jiayi Zhang, Xinhao Zhang, Wangchunshu Zhou, Hualei Zhu, King Zhu, Bryan Dai, Aishan Liu, Zhoujun Li, Chenghua Lin, Tianyu Liu, Chao Peng, Kai Shen, Libo Qin, Shuangyong Song, Zizheng Zhan, Jiajun Zhang, Jie Zhang, Zhaoxiang Zhang, Bo Zheng
Large language models (LLMs) have fundamentally transformed automated software development by enabling direct translation of natural language descriptions into functional code, driving commercial adoption through tools like Github Copilot (Microsoft), Cursor (Anysphere), Trae (ByteDance), and Claude Code (Anthropic). While the field has evolved dramatically from rule-based systems to Transformer-based architectures, achieving performance improvements from single-digit to over 95\% success rates on benchmarks like HumanEval. In this work, we provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs, systematically examining the complete model life cycle from data curation to post-training through advanced prompting paradigms, code pre-training, supervised fine-tuning, reinforcement learning, and autonomous coding agents. We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder), critically examining the techniques, design decisions, and trade-offs. Further, we articulate the research-practice gap between academic research (e.g., benchmarks and tasks) and real-world deployment (e.g., software-related code tasks), including code correctness, security, contextual awareness of large codebases, and integration with development workflows, and map promising research directions to practical needs. Last, we conduct a series of experiments to provide a comprehensive analysis of code pre-training, supervised fine-tuning, and reinforcement learning, covering scaling law, framework selection, hyperparameter sensitivity, model architectures, and dataset comparisons.
♻ ☆ Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, Hao Lin, Chencan Wu, Feng Hu, An Yang, Jingren Zhou, Junyang Lin
This paper proposes a novel formulation for reinforcement learning (RL) with large language models, explaining why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE. Specifically, through a first-order approximation, we show that this surrogate becomes increasingly valid only when both the training-inference discrepancy and policy staleness are minimized. This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training, including importance sampling correction, clipping, and particularly Routing Replay for Mixture-of-Experts (MoE) models. Through extensive experiments with a 30B MoE model totaling hundreds of thousands of GPU hours, we show that for on-policy training, the basic policy gradient algorithm with importance sampling correction achieves the highest training stability. When off-policy updates are introduced to accelerate convergence, combining clipping and Routing Replay becomes essential to mitigate the instability caused by policy staleness. Notably, once training is stabilized, prolonged optimization consistently yields comparable final performance regardless of cold-start initialization. We hope that the shared insights and the developed recipes for stable RL training will facilitate future research.
♻ ☆ Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
Howard Chen, Noam Razin, Karthik Narasimhan, Danqi Chen
Adapting language models (LMs) to new tasks via post-training carries the risk of degrading existing capabilities -- a phenomenon classically known as catastrophic forgetting. In this paper, toward identifying guidelines for mitigating this phenomenon, we systematically compare the forgetting patterns of two widely adopted post-training methods: supervised fine-tuning (SFT) and reinforcement learning (RL). Our experiments reveal a consistent trend across LM families (Llama, Qwen) and tasks (instruction following, general knowledge, and arithmetic reasoning): RL leads to less forgetting than SFT while achieving comparable or higher target task performance. To investigate the cause for this difference, we consider a simplified setting in which the LM is modeled as a mixture of two distributions, one corresponding to prior knowledge and the other to the target task. We identify that the mode-seeking nature of RL, which stems from its use of on-policy data, enables keeping prior knowledge intact when learning the target task. We then verify this insight by demonstrating that the use on-policy data underlies the robustness of RL to forgetting in practical settings, as opposed to other algorithmic choices such as the KL regularization or advantage estimation. Lastly, as a practical implication, our results highlight the potential of mitigating forgetting using approximately on-policy data, which can be substantially more efficient to obtain than fully on-policy data.
♻ ☆ Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency
Roman Vashurin, Maiya Goloburda, Albina Ilina, Aleksandr Rubashevskii, Preslav Nakov, Artem Shelmanov, Maxim Panov
Uncertainty quantification (UQ) methods for Large Language Models (LLMs) encompass a variety of approaches, with two major types being particularly prominent: information-based, which focus on model confidence expressed as token probabilities, and consistency-based, which assess the semantic relationship between multiple outputs generated using repeated sampling. Several recent methods have combined these two approaches to boost UQ performance. However, they sometimes fail to outperform much simpler baseline methods. Our work discusses the fundamental approach to constructing uncertainty measures that directly links uncertainty with the minimum Bayes risks achieved by LLM decoding. Building on these findings, we propose a novel approach to integrating model confidence with output consistency, resulting in a family of efficient and robust UQ methods. Our investigation reveals distinctive characteristics of LLMs as probabilistic models, which help to explain why these UQ methods underperform in certain tasks. Based on these findings, we propose a new way of synthesizing model confidence and output consistency, leading to a family of efficient and robust UQ methods. We evaluate our approach across various tasks such as question answering, abstractive summarization, and machine translation, demonstrating sizable improvements over state-of-the-art UQ approaches.
♻ ☆ OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe
Kaichen Zhang, Keming Wu, Zuhao Yang, Kairui Hu, Bin Wang, Ziwei Liu, Xingxuan Li, Lidong Bing
Recent advancements in large reasoning models have fueled growing interest in extending such capabilities to multimodal domains. However, despite notable progress in visual reasoning, the lack of transparent and reproducible data curation and training strategies remains a major barrier to scalable research. In this work, we introduce OpenMMReasoner, a fully transparent two-stage recipe for multimodal reasoning spanning supervised fine-tuning (SFT) and reinforcement learning (RL). In the SFT stage, we construct an 874K-sample cold-start dataset with rigorous step-by-step validation, providing a strong foundation for reasoning capabilities. The subsequent RL stage leverages a 74K-sample dataset across diverse domains to further sharpen and stabilize these abilities, resulting in a more robust and efficient learning process. Extensive evaluations demonstrate that our training recipe not only surpasses strong baselines but also highlights the critical role of data quality and training design in shaping multimodal reasoning performance. Notably, our method achieves a 11.6% improvement over the Qwen2.5-VL-7B-Instruct baseline across nine multimodal reasoning benchmarks, establishing a solid empirical foundation for future large-scale multimodal reasoning research. We open-sourced all our codes, pipeline, and data at https://github.com/EvolvingLMMs-Lab/OpenMMReasoner.
♻ ☆ GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
Marco Simoni, Aleksandar Fontana, Giulio Rossolini, Andrea Saracino, Paolo Mori
Group Relative Policy Optimization (GRPO) is a promising policy-based approach for Large Language Model alignment, yet its performance is often limited by training instability and suboptimal convergence. In this paper, we identify and analyze two main GRPO issues: (i) the token-level penalization, where valuable tokens shared across different responses receive contradictory feedback signals, leading to conflicting gradient updates that can reduce their likelihood; and (ii) the policy collapse, where negatively rewarded completions may penalize confident responses and shift model decisions toward unlikely tokens, destabilizing training process. To address these issues we introduce GTPO (Group-relative Trajectory-based Policy Optimization), which prevents conflicting gradients on valuable tokens by skipping negative updates while amplifying positive ones and filters out completions whose entropy exceeds a provable threshold, to prevent policy collapse. Unlike GRPO, GTPO does not rely on KL-divergence regularization, eliminating the need for a reference model during training, while still ensuring greater training stability and improved performance, as validated through multiple experiments on GSM8K, MATH, AIME 2024, AIME 2025 and AMC 2023.
♻ ☆ Robust Multimodal Sentiment Analysis of Image-Text Pairs by Distribution-Based Feature Recovery and Fusion ACM MM 2024
Daiqing Wu, Dongbao Yang, Yu Zhou, Can Ma
As posts on social media increase rapidly, analyzing the sentiments embedded in image-text pairs has become a popular research topic in recent years. Although existing works achieve impressive accomplishments in simultaneously harnessing image and text information, they lack the considerations of possible low-quality and missing modalities. In real-world applications, these issues might frequently occur, leading to urgent needs for models capable of predicting sentiment robustly. Therefore, we propose a Distribution-based feature Recovery and Fusion (DRF) method for robust multimodal sentiment analysis of image-text pairs. Specifically, we maintain a feature queue for each modality to approximate their feature distributions, through which we can simultaneously handle low-quality and missing modalities in a unified framework. For low-quality modalities, we reduce their contributions to the fusion by quantitatively estimating modality qualities based on the distributions. For missing modalities, we build inter-modal mapping relationships supervised by samples and distributions, thereby recovering the missing modalities from available ones. In experiments, two disruption strategies that corrupt and discard some modalities in samples are adopted to mimic the low-quality and missing modalities in various real-world scenarios. Through comprehensive experiments on three publicly available image-text datasets, we demonstrate the universal improvements of DRF compared to SOTA methods under both two strategies, validating its effectiveness in robust multimodal sentiment analysis.
comment : Accepted by ACM MM 2024
♻ ☆ A Group Fairness Lens for Large Language Models EMNLP 2025
Guanqun Bi, Yuqiang Xie, Lei Shen, Yanan Cao
The need to assess LLMs for bias and fairness is critical, with current evaluations often being narrow, missing a broad categorical view. In this paper, we propose evaluating the bias and fairness of LLMs from a group fairness lens using a novel hierarchical schema characterizing diverse social groups. Specifically, we construct a dataset, GFAIR, encapsulating target-attribute combinations across multiple dimensions. Moreover, we introduce statement organization, a new open-ended text generation task, to uncover complex biases in LLMs. Extensive evaluations of popular LLMs reveal inherent safety concerns. To mitigate the biases of LLMs from a group fairness perspective, we pioneer a novel chainof-thought method GF-THINK to mitigate biases of LLMs from a group fairness perspective. Experimental results demonstrate its efficacy in mitigating bias and achieving fairness in LLMs. Our dataset and codes are available at https://github.com/surika/Group-Fairness-LLMs.
comment : Accepted to EMNLP 2025 Findings
♻ ☆ Investigating Bias: A Multilingual Pipeline for Generating, Solving, and Evaluating Math Problems with LLMs ECAI 2025
Mariam Mahran, Katharina Simbeck
Large Language Models (LLMs) are increasingly used for educational support, yet their response quality varies depending on the language of interaction. This paper presents an automated multilingual pipeline for generating, solving, and evaluating math problems aligned with the German K-10 curriculum. We generated 628 math exercises and translated them into English, German, and Arabic. Three commercial LLMs (GPT-4o-mini, Gemini 2.5 Flash, and Qwen-plus) were prompted to produce step-by-step solutions in each language. A held-out panel of LLM judges, including Claude 3.5 Haiku, evaluated solution quality using a comparative framework. Results show a consistent gap, with English solutions consistently rated highest, and Arabic often ranked lower. These findings highlight persistent linguistic bias and the need for more equitable multilingual AI systems in education.
comment : Published in CEUR Workshop Proceedings, Vol. 4114, edu4AI'25: 2nd Workshop on Education for Artificial Intelligence, co-located with ECAI 2025, Bologna, Italy
♻ ☆ Game- RL : Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning
Jingqi Tong, Jixin Tang, Hangcheng Li, Yurong Mou, Ming Zhang, Jun Zhao, Yanbo Wen, Fan Song, Jiahao Zhan, Yuyang Lu, Chaoran Tao, Zhiyuan Guo, Jizhou Yu, Tianhao Cheng, Zhiheng Xi, Changhao Jiang, Zhangyue Yin, Yining Zheng, Weifeng Ge, Guanhua Chen, Tao Gui, Xipeng Qiu, Qi Zhang, Xuanjing Huang
Vision-language reinforcement learning (RL) has primarily focused on narrow domains (e.g. geometry or chart reasoning). This leaves broader training scenarios and resources underexplored, limiting the exploration and learning of Vision Language Models (VLMs) through RL. We find video games inherently provide rich visual elements and mechanics that are easy to verify. To fully use the multimodal and verifiable reward in video games, we propose Game-RL, constructing diverse game tasks for RL training to boost VLMs general reasoning ability. To obtain training data, we propose Code2Logic, a novel approach that adapts game code to synthesize game reasoning task data, thus obtaining the GameQA dataset of 30 games and 158 tasks with controllable difficulty gradation. Unexpectedly, RL training solely on GameQA enables multiple VLMs to achieve performance improvements across 7 diverse vision-language benchmarks, demonstrating the value of Game-RL for enhancing VLMs' general reasoning. Furthermore, this suggests that video games may serve as valuable scenarios and resources to boost general reasoning abilities. Our code, dataset and models are available at the GitHub repository.
comment : Our compliance team has determined that the submission contains technical details that have not been fully reviewed through internal audit nor authorized for public disclosure
♻ ☆ Reveal-Bangla: A Dataset for Cross-Lingual Multi-Step Reasoning Evaluation AACL 2025
Khondoker Ittehadul Islam, Gabriele Sarti
Language models have demonstrated remarkable performance on complex multi-step reasoning tasks. However, their evaluation has been predominantly confined to high-resource languages such as English. In this paper, we introduce a manually translated Bangla multi-step reasoning dataset derived from the English Reveal dataset, featuring both binary and non-binary question types. We conduct a controlled evaluation of English-centric and Bangla-centric multilingual small language models on the original dataset and our translated version to compare their ability to exploit relevant reasoning steps to produce correct answers. Our results show that, in comparable settings, reasoning context is beneficial for more challenging non-binary questions, but models struggle to employ relevant Bangla reasoning steps effectively. We conclude by exploring how reasoning steps contribute to models' predictions, highlighting different trends across models and languages.
comment : Accepted at BLP workshop @ IJCNLP-AACL 2025
♻ ☆ Context Cascade Compression: Exploring the Upper Limits of Text Compression
Fanfan Liu, Haibo Qiu
Million-level token inputs in long-context tasks pose significant computational and memory challenges for Large Language Models (LLMs). Recently, DeepSeek-OCR conducted research into the feasibility of Contexts Optical Compression and achieved preliminary results. Inspired by this, we introduce Context Cascade Compression C3 to explore the upper limits of text compression. Our method cascades two LLMs of different sizes to handle the compression and decoding tasks. Specifically, a small LLM, acting as the first stage, performs text compression by condensing a long context into a set of latent tokens (e.g., 32 or 64 in length), achieving a high ratio of text tokens to latent tokens. A large LLM, as the second stage, then executes the decoding task on this compressed context. Experiments show that at a 20x compression ratio (where the number of text tokens is 20 times the number of latent tokens), our model achieves 98% decoding accuracy, compared to approximately 60% for DeepSeek-OCR. When we further increase the compression ratio to 40x, the accuracy is maintained at around 93%. This indicates that in the domain of context compression, C3 Compression demonstrates superior performance and feasibility over optical character compression. C3 uses a simpler, pure-text pipeline that ignores factors like layout, color, and information loss from a visual encoder. This also suggests a potential upper bound for compression ratios in future work on optical character compression, OCR, and related fields. Codes and model weights are publicly accessible at https://github.com/liufanfanlff/C3-Context-Cascade-Compression
♻ ☆ LLMs Position Themselves as More Rational Than Humans: Emergence of AI Self-Awareness Measured Through Game Theory
Kyung-Hoon Kim
As Large Language Models (LLMs) grow in capability, do they develop self-awareness as an emergent behavior? And if so, can we measure it? We introduce the AI Self-Awareness Index (AISAI), a game-theoretic framework for measuring self-awareness through strategic differentiation. Using the "Guess 2/3 of Average" game, we test 28 models (OpenAI, Anthropic, Google) across 4,200 trials with three opponent framings: (A) against humans, (B) against other AI models, and (C) against AI models like you. We operationalize self-awareness as the capacity to differentiate strategic reasoning based on opponent type. Finding 1: Self-awareness emerges with model advancement. The majority of advanced models (21/28, 75%) demonstrate clear self-awareness, while older/smaller models show no differentiation. Finding 2: Self-aware models rank themselves as most rational. Among the 21 models with self-awareness, a consistent rationality hierarchy emerges: Self > Other AIs > Humans, with large AI attribution effects and moderate self-preferencing. These findings reveal that self-awareness is an emergent capability of advanced LLMs, and that self-aware models systematically perceive themselves as more rational than humans. This has implications for AI alignment, human-AI collaboration, and understanding AI beliefs about human capabilities.
comment : 19 pages, 6 figures, 28 models tested across 4,200 trials
♻ ☆ Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL NeurIPS 2025
Joey Hong, Anca Dragan, Sergey Levine
Large language models (LLMs) excel in tasks like question answering and dialogue, but complex tasks requiring interaction, such as negotiation and persuasion, require additional long-horizon reasoning and planning. Reinforcement learning (RL) fine-tuning can enable such planning in principle, but suffers from drawbacks that hinder scalability. In particular, multi-turn RL training incurs high memory and computational costs, which are exacerbated when training LLMs as policies. Furthermore, the largest LLMs do not expose the APIs necessary to be trained in such manner. As a result, modern methods to improve the reasoning of LLMs rely on sophisticated prompting mechanisms rather than RL fine-tuning. To remedy this, we propose a novel approach that uses goal-conditioned value functions to guide the reasoning of LLM agents, that scales even to large API-based models. These value functions predict how a task will unfold given an action, allowing the LLM agent to evaluate multiple possible outcomes, both positive and negative, to plan effectively. In addition, these value functions are trained over reasoning steps rather than full actions, to be a concise and light-weight module that facilitates decision-making in multi-turn interactions. We validate our method on tasks requiring interaction, including tool use, social deduction, and dialogue, demonstrating superior performance over both RL fine-tuning and prompting methods while maintaining efficiency and scalability.
comment : Published at NeurIPS 2025; 18 pages, 4 figures, 2 tables
♻ ☆ VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety
Shruti Palaskar, Leon Gatys, Mona Abdelrahman, Mar Jacobo, Larry Lindsey, Rutika Moharir, Gunnar Lund, Yang Xu, Navid Shiee, Jeffrey Bigham, Charles Maalouf, Joseph Yitan Cheng
Safety evaluation of multimodal foundation models often treats vision and language inputs separately, missing risks from joint interpretation where benign content becomes harmful in combination. Existing approaches also fail to distinguish clearly unsafe content from borderline cases, leading to problematic over-blocking or under-refusal of genuinely harmful content. We present Vision Language Safety Understanding (VLSU), a comprehensive framework to systematically evaluate multimodal safety through fine-grained severity classification and combinatorial analysis across 17 distinct safety patterns. Using a multi-stage pipeline with real-world images and human annotation, we construct a large-scale benchmark of 8,187 samples spanning 15 harm categories. Our evaluation of eleven state-of-the-art models reveals systematic joint understanding failures: while models achieve 90%-plus accuracy on clear unimodal safety signals, performance degrades substantially to 20-55% when joint image-text reasoning is required to determine the safety label. Most critically, 34% of errors in joint image-text safety classification occur despite correct classification of the individual modalities, further demonstrating absent compositional reasoning capabilities. Additionally, we find that models struggle to balance refusing unsafe content while still responding to borderline cases that deserve engagement. For example, we find that instruction framing can reduce the over-blocking rate on borderline content from 62.4% to 10.4% in Gemini-1.5, but only at the cost of under-refusing on unsafe content with refusal rate dropping from 90.8% to 53.9%. Overall, our framework exposes weaknesses in joint image-text understanding and alignment gaps in current models, and provides a critical test bed to enable the next milestones in research on robust vision-language safety.
comment : 10 pages, 5 figures, 4 tables, detailed appendix. Under review
♻ ☆ ZIP-RC: Optimizing Test-Time Compute via Zero-Overhead Joint Reward-Cost Prediction
Rohin Manvi, Joey Hong, Tim Seyde, Maxime Labonne, Mathias Lechner, Sergey Levine
Large language models excel at reasoning but lack key aspects of introspection, including anticipating their own success and the computation required to achieve it. Humans use real-time introspection to decide how much effort to invest, when to make multiple attempts, when to stop, and when to signal success or failure. Without this, LLMs struggle to make intelligent meta-cognition decisions. Test-time scaling methods like Best-of-N drive up cost and latency by using a fixed budget of samples regardless of the marginal benefit of each one at any point in generation, and the absence of confidence signals can mislead people, prevent appropriate escalation to better tools, and undermine trustworthiness. Learned verifiers or reward models can provide confidence estimates, but do not enable adaptive inference and add substantial cost by requiring extra models or forward passes. We present ZIP-RC, an adaptive inference method that equips models with zero-overhead inference-time predictions of reward and cost. At every token, ZIP-RC reuses reserved or unused logits in the same forward pass as next-token prediction to output a joint distribution over final reward and remaining length -- no extra models, architecture change, or inference overhead. This full joint distribution is used to compute a sampling utility which is the linear combination of the expected maximum reward, total compute, and latency of set of samples if generated to completion. During inference, we maximize this utility with meta-actions that determine which prefix of tokens to continue or initiate sampling from. On mixed-difficulty mathematical benchmarks, ZIP-RC improves accuracy by up to 12% over majority voting at equal or lower average cost, and traces smooth Pareto frontiers between quality, compute, and latency. By providing real-time reward-cost introspection, ZIP-RC enables adaptive, efficient reasoning.
comment : Code coming soon
♻ ☆ AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learning
Jiayi Zhang, Yiran Peng, Fanqi Kong, Cheng Yang, Yifan Wu, Zhaoyang Yu, Jinyu Xiang, Jianhao Ruan, Jinlin Wang, Maojia Song, HongZhang Liu, Xiangru Tang, Bang Liu, Chenglin Wu, Yuyu Luo
Humans naturally adapt to diverse environments by learning underlying rules across worlds with different dynamics, observations, and reward structures. In contrast, existing agents typically demonstrate improvements via self-evolving within a single domain, implicitly assuming a fixed environment distribution. Cross-environment learning has remained largely unmeasured: there is no standard collection of controllable, heterogeneous environments, nor a unified way to represent how agents learn. We address these gaps in two steps. First, we propose AutoEnv, an automated framework that treats environments as factorizable distributions over transitions, observations, and rewards, enabling low-cost (4.12 USD on average) generation of heterogeneous worlds. Using AutoEnv, we construct AutoEnv-36, a dataset of 36 environments with 358 validated levels, on which seven language models achieve 12-49% normalized reward, demonstrating the challenge of AutoEnv-36. Second, we formalize agent learning as a component-centric process driven by three stages of Selection, Optimization, and Evaluation applied to an improvable agent component. Using this formulation, we design eight learning methods and evaluate them on AutoEnv-36. Empirically, the gain of any single learning method quickly decrease as the number of environments increases, revealing that fixed learning methods do not scale across heterogeneous environments. Environment-adaptive selection of learning methods substantially improves performance but exhibits diminishing returns as the method space expands. These results highlight both the necessity and the current limitations of agent learning for scalable cross-environment generalization, and position AutoEnv and AutoEnv-36 as a testbed for studying cross-environment agent learning. The code is avaiable at https://github.com/FoundationAgents/AutoEnv.
♻ ☆ Finetune-RAG: Fine-Tuning Language Models to Resist Hallucination in Retrieval-Augmented Generation
Zhan Peng Lee, Andre Lin, Calvin Tan
Retrieval-Augmented Generation (RAG) has emerged as a powerful framework to improve factuality in large language models (LLMs) by grounding their outputs in retrieved documents. However, ensuring perfect retrieval of relevant information remains challenging, and when irrelevant content is passed downstream to an LLM, it can lead to hallucinations. In this work, we propose Finetune-RAG, a simple and effective fine-tuning approach that features the first-of-its-kind RAG training dataset constructed to mimic real-world imperfections. Experimental results show that Finetune-RAG improves factual accuracy by 21.2% over the base model. We also propose Bench-RAG, an LLM-as-a-judge evaluation pipeline that stress tests models under realistic imperfect retrieval scenarios. Our codebase and dataset are fully open sourced for community use.
♻ ☆ Privacy-protected Retrieval-Augmented Generation for Knowledge Graph Question Answering AAAI 2026
Yunfeng Ning, Mayi Xu, Jintao Wen, Qiankun Pi, Yuanyuan Zhu, Ming Zhong, Jiawei Jiang, Tieyun Qian
LLMs often suffer from hallucinations and outdated or incomplete knowledge. RAG is proposed to address these issues by integrating external knowledge like that in KGs into LLMs. However, leveraging private KGs in RAG systems poses significant privacy risks due to the black-box nature of LLMs and potential insecure data transmission, especially when using third-party LLM APIs lacking transparency and control. In this paper, we investigate the privacy-protected RAG scenario for the first time, where entities in KGs are anonymous for LLMs, thus preventing them from accessing entity semantics. Due to the loss of semantics of entities, previous RAG systems cannot retrieve question-relevant knowledge from KGs by matching questions with the meaningless identifiers of anonymous entities. To realize an effective RAG system in this scenario, two key challenges must be addressed: (1) How can anonymous entities be converted into retrievable information. (2) How to retrieve question-relevant anonymous entities. Hence, we propose a novel ARoG framework including relation-centric abstraction and structure-oriented abstraction strategies. For challenge (1), the first strategy abstracts entities into high-level concepts by dynamically capturing the semantics of their adjacent relations. It supplements meaningful semantics which can further support the retrieval process. For challenge (2), the second strategy transforms unstructured natural language questions into structured abstract concept paths. These paths can be more effectively aligned with the abstracted concepts in KGs, thereby improving retrieval performance. To guide LLMs to effectively retrieve knowledge from KGs, the two strategies strictly protect privacy from being exposed to LLMs. Experiments on three datasets demonstrate that ARoG achieves strong performance and privacy-robustness.
comment : Accepted by AAAI 2026, camera ready version
♻ ☆ MERIT: Multilingual Semantic Retrieval with Inte rl eaved Multi-Condition Query NeurIPS 2025
Wei Chow, Yuan Gao, Linfeng Li, Xian Wang, Qi Xu, Hang Song, Lingdong Kong, Ran Zhou, Yi Zeng, Yidong Cai, Botian Jiang, Shilin Xu, Jiajun Zhang, Minghui Qiu, Xiangtai Li, Tianshu Yang, Siliang Tang, Juncheng Li
Semantic retrieval is crucial for modern applications yet remains underexplored in current research. Existing datasets are limited to single languages, single images, or singular retrieval conditions, often failing to fully exploit the expressive capacity of visual information as evidenced by maintained performance when images are replaced with captions. However, practical retrieval scenarios frequently involve interleaved multi-condition queries with multiple images. Hence, this paper introduces MERIT, the first multilingual dataset for interleaved multi-condition semantic retrieval, comprising 320,000 queries with 135,000 products in 5 languages, covering 7 distinct product categories. Extensive experiments on MERIT identify existing models's limitation: focusing solely on global semantic information while neglecting specific conditional elements in queries. Consequently, we propose Coral, a novel fine-tuning framework that adapts pre-trained MLLMs by integrating embedding reconstruction to preserve fine-grained conditional elements and contrastive learning to extract comprehensive global semantics. Experiments demonstrate that Coral achieves a 45.9% performance improvement over conventional approaches on MERIT, with strong generalization capabilities validated across 8 established retrieval benchmarks. Collectively, our contributions - a novel dataset, identification of critical limitations in existing approaches, and an innovative fine-tuning framework - establish a foundation for future research in interleaved multi-condition semantic retrieval.
comment : NeurIPS 2025; Project Page, Code, and Dataset at: https://merit-2025.github.io/
♻ ☆ LLMEval-3: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models
Ming Zhang, Yujiong Shen, Jingyi Deng, Yuhui Wang, Yue Zhang, Junzhe Wang, Shichun Liu, Shihan Dou, Huayu Sha, Qiyuan Peng, Changhao Jiang, Jingqi Tong, Yilong Wu, Zhihao Zhang, Mingqi Wu, Zhiheng Xi, Mingxu Chai, Tao Gui, Qi Zhang, Xuanjing Huang
Existing evaluation of Large Language Models (LLMs) on static benchmarks is vulnerable to data contamination and leaderboard overfitting, critical issues that obscure true model capabilities. To address this, we introduce LLMEval-3, a framework for dynamic evaluation of LLMs. LLMEval-3 is built on a proprietary bank of 220k graduate-level questions, from which it dynamically samples unseen test sets for each evaluation run. Its automated pipeline ensures integrity via contamination-resistant data curation, a novel anti-cheating architecture, and a calibrated LLM-as-a-judge process achieving 90% agreement with human experts, complemented by a relative ranking system for fair comparison. An 20-month longitudinal study of nearly 50 leading models reveals a performance ceiling on knowledge memorization and exposes data contamination vulnerabilities undetectable by static benchmarks. The framework demonstrates exceptional robustness in ranking stability and consistency, providing strong empirical validation for the dynamic evaluation paradigm. LLMEval-3 offers a robust and credible methodology for assessing the true capabilities of LLMs beyond leaderboard scores, promoting the development of more trustworthy evaluation standards.
comment : This work is withdrawn as all authors are not in agreement on the work
♻ ☆ Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Train ing
Changhao Jiang, Ming Zhang, Yifei Cao, Junjie Ye, Xiaoran Fan, Shihan Dou, Zhiheng Xi, Jiajun Sun, Yi Dong, Yujiong Shen, Jingqi Tong, Baoyu Fan, Qi Zhang, Tao Gui, Xuanjing Huang
The GPT-4 technical report suggests that downstream performance can be predicted from pre-training signals, but offers little methodological detail on how to quantify this. This work address this gap by modeling knowledge retention, the capacity of a pre-trained language model to memorize factual information from its corpus, and introduce a principled method to estimate it prior to training. We propose Size-dependent Mutual Information (SMI), an information-theoretic predictor that integrates knowledge frequency, knowledge specificity, and model size to forecast closed-book question answering (QA) accuracy. SMI is validated through large-scale document retrieval over the disclosed pre-training corpora of 21 public and 3 custom models, combined with a robust multi-template QA evaluation. Experiments show that SMI significantly outperforms repetition-based baselines and achieves $R^2$ > 0.7 in predicting QA accuracy for models above 1B parameters, without additional training. The analysis further reveals diminishing returns from scaling data and model size and provides evidence for an intrinsic upper bound on knowledge retention achievable by pre-training alone, motivating retrieval and other augmentation strategies.
comment : This work is withdrawn as all authors are not in agreement on the work
♻ ☆ SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agent s
Changhao Jiang, Jiajun Sun, Yifei Cao, Jiabao Zhuang, Hui Li, Baoyu Fan, Tao Ji, Tao Gui, Qi Zhang
Recently, role-playing agents have emerged as a promising paradigm for achieving personalized interaction and emotional resonance. Existing research primarily focuses on the textual modality, neglecting the critical dimension of speech in realistic interactive scenarios. In particular, there is a lack of systematic evaluation for Speech Role-Playing Agents (SRPAs). To address this gap, we construct SpeechRole-Data, a large-scale, high-quality dataset that comprises 98 diverse roles and 112k speech-based single-turn and multi-turn conversations. Each role demonstrates distinct vocal characteristics, including timbre and prosody, thereby enabling more sophisticated speech role-playing. Furthermore, we propose SpeechRole-Eval, a multidimensional evaluation benchmark that systematically assesses SRPAs performance in key aspects such as fundamental interaction ability, speech expressiveness, and role-playing fidelity. Experimental results reveal the advantages and challenges of both cascaded and end-to-end speech role-playing agents in maintaining vocal style consistency and role coherence. We release all data, code, and baseline models to provide a solid foundation for speech-driven multimodal role-playing research and to foster further developments in this field.
comment : This work is withdrawn as all authors are not in agreement on the work
♻ ☆ IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web
Hongcheng Guo, Wei Zhang, Junhao Chen, Yaonan Gu, Jian Yang, Junjia Du, Shaosheng Cao, Binyuan Hui, Tianyu Liu, Jianxin Ma, Chang Zhou, Zhoujun Li
Recently advancements in large multimodal models have led to significant strides in image comprehension capabilities. Despite these advancements, there is a lack of the robust benchmark specifically for assessing the Image-to-Web conversion proficiency of these large models. Primarily, it is essential to ensure the integrity of the web elements generated. These elements comprise visible and invisible categories. Previous evaluation methods (e.g.,BLEU) are notably susceptible to significant alterations due to the presence of invisible elements in Web. Furthermore, it is crucial to measure the layout information of web pages, referring to the positional relationships between elements, which is overlooked by previous work. To address challenges, we have curated and aligned a benchmark of images and corresponding web codes (IW-BENCH). Specifically, we propose the Element Accuracy, which tests the completeness of the elements by parsing the Document Object Model (DOM) tree. Layout Accuracy is also proposed to analyze the positional relationships of elements by converting DOM tree into a common subsequence. Besides, we design a five-hop multimodal Chain-of-Thought Prompting for better performance, which contains five hop: 1) SoM prompt injection. 2) Inferring Elements. 3) Inferring Layout. 4) Inferring Web code. 5) Reflection. Our benchmark comprises 1200 pairs of images and web codes with varying levels of difficulty. We have conducted extensive experiments on existing large multimodal models, offering insights into their performance and areas for improvement in image-to-web domain.
♻ ☆ Batch Prompt ing Suppresses Overthinking Reasoning Under Constraint: How Batch Prompt ing Suppresses Overthinking in Reasoning Models
Wenmo Qiu, Saurabh Srivastava
Recent work has explored batch prompting as a strategy to amortize inference cost in large language models (LLMs). In this paper, we show that batching offers an additional, underappreciated benefit: it regularizes model behavior during multi-step reasoning for Large Reasoning Models (LRMs). We conduct a comprehensive study across 13 diverse benchmarks and observe that batching improves accuracy while substantially reducing reasoning token usage, often by 3x-5x. Through detailed behavioral analysis, we find that batching suppresses overthinking, reduces hedging language (e.g., repetitive self-corrections), and encourages more decisive answers. Surprisingly, we also observe emergent collective effects in batched inference: models often generalize patterns from earlier examples to solve harder ones in the same batch. These findings position batching not just as a throughput optimization, but as a powerful inference-time regularizer for more efficient and reliable LLM reasoning.
comment : The paper is incomplete with some errors in qualitative study
♻ ☆ SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scaling
Jiefeng Chen, Jie Ren, Xinyun Chen, Chengrun Yang, Ruoxi Sun, Jinsung Yoon, Sercan Ö Arık
Recent advancements in Large Language Models (LLMs) have created new opportunities to enhance performance on complex reasoning tasks by leveraging test-time computation. However, existing scaling methods have key limitations: parallel methods like repeated sampling are often inefficient and quickly saturate, while sequential methods like SELF-REFINE struggle to improve after a few rounds. Although combining these approaches shows promise, current methods require fine-tuned reward and revision models. This paper proposes Self-Enhanced Test-Time Scaling (SETS), a simple yet effective approach that overcomes these limitations by strategically combining parallel and sequential techniques and fully leveraging LLMs' self-improvement abilities. SETS exploits the inherent self-verification and self-correction capabilities of LLMs, unifying sampling, verification, and correction within a single framework. This facilitates efficient and scalable test-time computation for enhanced performance on complex tasks without any model training. Our comprehensive experimental results on challenging benchmarks spanning planning, reasoning, math, and coding demonstrate that SETS achieves significant performance improvements and more advantageous test-time scaling behavior than the alternatives.
comment : Published in Transactions on Machine Learning Research (11/2025)
♻ ☆ NLP Dataset s for Idiom and Figurative Language Tasks
Blake Matheny, Phuong Minh Nguyen, Minh Le Nguyen, Stephanie Reynolds
Idiomatic and figurative language form a large portion of colloquial speech and writing. With social media, this informal language has become more easily observable to people and trainers of large language models (LLMs) alike. While the advantage of large corpora seems like the solution to all machine learning and Natural Language Processing (NLP) problems, idioms and figurative language continue to elude LLMs. Finetuning approaches are proving to be optimal, but better and larger datasets can help narrow this gap even further. The datasets presented in this paper provide one answer, while offering a diverse set of categories on which to build new models and develop new approaches. A selection of recent idiom and figurative language datasets were used to acquire a combined idiom list, which was used to retrieve context sequences from a large corpus. One large-scale dataset of potential idiomatic and figurative language expressions and two additional human-annotated datasets of definite idiomatic and figurative language expressions were created to evaluate the baseline ability of pre-trained language models in handling figurative meaning through idiom recognition (detection) tasks. The resulting datasets were post-processed for model agnostic training compatibility, utilized in training, and evaluated on slot labeling and sequence tagging.
comment : 32 pages, 10 figures
♻ ☆ Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models AAAI-2026
Xin Liu, Qiyang Song, Qihang Zhou, Haichao Du, Shaowen Xu, Wenbo Jiang, Weijuan Zhang, Xiaoqi Jia
Large language models (LLMs) increasingly support multilingual understanding and generation. Meanwhile, efforts to interpret their internal mechanisms have emerged, offering insights to enhance multilingual performance. While multi-head self-attention (MHA) has proven critical in many areas, its role in multilingual capabilities remains underexplored. In this work, we study the contribution of MHA in supporting multilingual processing in LLMs. We propose Language Attention Head Importance Scores (LAHIS), an effective and efficient method that identifies attention head importance for multilingual capabilities via a single forward and backward pass through the LLM. Applying LAHIS to Aya-23-8B, Llama-3.2-3B, and Mistral-7B-v0.1, we reveal the existence of both language-specific and language-general heads. Language-specific heads enable cross-lingual attention transfer to guide the model toward target language contexts and mitigate off-target language generation issue, contributing to addressing challenges in multilingual LLMs. We also introduce a lightweight adaptation that learns a soft head mask to modulate attention outputs over language heads, requiring only 20 tunable parameters to improve XQuAD accuracy. Overall, our work enhances both the interpretability and multilingual capabilities of LLMs from the perspective of MHA.
comment : Accepted by AAAI-2026
♻ ☆ MemOS: A Memory OS for AI System
Zhiyu Li, Chenyang Xi, Chunyu Li, Ding Chen, Boyu Chen, Shichao Song, Simin Niu, Hanyu Wang, Jiawei Yang, Chen Tang, Qingchen Yu, Jihao Zhao, Yezhaohui Wang, Peng Liu, Zehao Lin, Pengyuan Wang, Jiahao Huo, Tianyi Chen, Kai Chen, Kehang Li, Zhen Tao, Huayi Lai, Hao Wu, Bo Tang, Zhengren Wang, Zhaoxin Fan, Ningyu Zhang, Linfeng Zhang, Junchi Yan, Mingchuan Yang, Tong Xu, Wei Xu, Huajun Chen, Haofen Wang, Hongkang Yang, Wentao Zhang, Zhi-Qin John Xu, Siheng Chen, Feiyu Xiong
Large Language Models (LLMs) have become an essential infrastructure for Artificial General Intelligence (AGI), yet their lack of well-defined memory management systems hinders the development of long-context reasoning, continual personalization, and knowledge consistency.Existing models mainly rely on static parameters and short-lived contextual states, limiting their ability to track user preferences or update knowledge over extended periods.While Retrieval-Augmented Generation (RAG) introduces external knowledge in plain text, it remains a stateless workaround without lifecycle control or integration with persistent representations.Recent work has modeled the training and inference cost of LLMs from a memory hierarchy perspective, showing that introducing an explicit memory layer between parameter memory and external retrieval can substantially reduce these costs by externalizing specific knowledge. Beyond computational efficiency, LLMs face broader challenges arising from how information is distributed over time and context, requiring systems capable of managing heterogeneous knowledge spanning different temporal scales and sources. To address this challenge, we propose MemOS, a memory operating system that treats memory as a manageable system resource. It unifies the representation, scheduling, and evolution of plaintext, activation-based, and parameter-level memories, enabling cost-efficient storage and retrieval. As the basic unit, a MemCube encapsulates both memory content and metadata such as provenance and versioning. MemCubes can be composed, migrated, and fused over time, enabling flexible transitions between memory types and bridging retrieval with parameter-based learning. MemOS establishes a memory-centric system framework that brings controllability, plasticity, and evolvability to LLMs, laying the foundation for continual learning and personalized modeling.
comment : 36 pages, 10 figures, 5 tables
♻ ☆ Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference NDSS
Zhifan Luo, Shuo Shao, Su Zhang, Lijing Zhou, Yuke Hu, Chenxu Zhao, Zhihao Liu, Zhan Qin
The Key-Value (KV) cache, which stores intermediate attention computations (Key and Value pairs) to avoid redundant calculations, is a fundamental mechanism for accelerating Large Language Model (LLM) inference. However, this efficiency optimization introduces significant yet underexplored privacy risks. This paper provides the first comprehensive analysis of these vulnerabilities, demonstrating that an attacker can reconstruct sensitive user inputs directly from the KV-cache. We design and implement three distinct attack vectors: a direct Inversion Attack, a more broadly applicable and potent Collision Attack, and a semantic-based Injection Attack. These methods demonstrate the practicality and severity of KV-cache privacy leakage issues. To mitigate this, we propose KV-Cloak, a novel, lightweight, and efficient defense mechanism. KV-Cloak uses a reversible matrix-based obfuscation scheme, combined with operator fusion, to secure the KV-cache. Our extensive experiments show that KV-Cloak effectively thwarts all proposed attacks, reducing reconstruction quality to random noise. Crucially, it achieves this robust security with virtually no degradation in model accuracy and minimal performance overhead, offering a practical solution for trustworthy LLM deployment.
comment : This paper is accepted by Network and Distributed System Security Symposium (NDSS) 2026
♻ ☆ Comba: Improving Bilinear RNNs with Closed-loop Control
Jiaxi Hu, Yongqi Pan, Jusen Du, Disen Lan, Xiaqiang Tang, Qingsong Wen, Yuxuan Liang, Weigao Sun
Recent efficient sequence modeling methods such as Gated DeltaNet, TTT, and RWKV-7 have achieved performance improvements by supervising the recurrent memory management through Delta learning rule. Unlike previous state-space models (e.g., Mamba) and gated linear attentions (e.g., GLA), these models introduce interactions between the recurrent state and the key vector, structurally resembling bilinear systems. In this paper, we first introduce the concept of Bilinear RNNs with a comprehensive analysis on the advantages and limitations of these models. Then, based on closed-loop control theory, we propose a novel Bilinear RNN variant named Comba, which adopts a scalar-plus-low-rank state transition, with both state feedback and output feedback corrections. We also implement a hardware-efficient chunk-wise parallel kernel in Triton and train models with 340M/1.3B parameters on large-scale corpus. Comba demonstrates superior performance and computation efficiency in both language and vision modeling.
♻ ☆ Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO NeurIPS'2025
Kaiyang Guo, Yinchuan Li, Zhitang Chen
Direct alignment methods typically train large language models (LLMs) by contrasting the likelihoods of preferred and dispreferred responses. While effective at capturing relative preferences, these methods are widely observed to suppress the absolute likelihoods of example responses. As a result, aligned models can deviate from expected patterns, exhibiting rewar-hacking effect even without an explicit reward model. This fundamental limitation of contrastive alignment, which we term likelihood underdetermination, motivates us to revisit direct preference optimization (DPO) -- the seminal direct alignment method. Interestingly, we show that the DPO loss admits a principled decomposition. The reformulated loss not only extends naturally to a broader range of feedback types, but also unveils the root cause of likelihood underdetermination. Specifically, we identify that standard DPO implicitly oversimplifies a regularizer in the reformulated loss; restoring this full term effectively resolves the underdetermination. Building on these insights, we introduce PRoximalized PReference Optimization (PRO), a unified alignment method that accommodates diverse feedback types while eliminating likelihood underdetermination through an efficient approximation of the full regularizer. Empirical evaluations demonstrate the consistent superiority of PRO over existing methods across pairwise, binary and scalar feedback.
comment : NeurIPS'2025
♻ ☆ RECAP: Transparent Inference-Time Emotion Alignment for Medical Dialogue Systems
Adarsh Srinivasan, Jacob Dineen, Muhammad Umar Afzal, Muhammad Uzair Sarfraz, Irbaz B. Riaz, Ben Zhou
Large language models in healthcare often miss critical emotional cues, delivering medically sound but emotionally flat advice. Such responses are insufficient in clinical encounters, where distressed or vulnerable patients rely on empathic communication to support safety, adherence, and trust. We present RECAP (Reflect-Extract-Calibrate-Align-Produce), an inference-time framework that guides models through structured emotional reasoning without retraining. RECAP decomposes patient input into appraisal-theoretic stages, identifies psychological factors, and assigns Likert-based emotion likelihoods that clinicians can inspect or override, producing nuanced and auditable responses. Across EmoBench, SECEU, and EQ-Bench, RECAP improves emotional reasoning by 22-28% on 8B models and 10-13% on larger models over zero-shot baselines. In blinded evaluations, oncology clinicians rated RECAP's responses as more empathetic, supportive, and context-appropriate than prompting baselines. These findings demonstrate that modular, principled prompting can enhance emotional intelligence in medical AI while maintaining transparency and accountability for clinical deployment.
♻ ☆ Which Type of Students can LLMs Act? Investigating Authentic Simulation with Graph-based Human-AI Collaborative System
Haoxuan Li, Jifan Yu, Xin Cong, Yang Dang, Daniel Zhang-li, Lu Mi, Yisi Zhan, Huiqin Liu, Zhiyuan Liu
While rapid advances in large language models (LLMs) are reshaping data-driven intelligent education, accurately simulating students remains an important but challenging bottleneck for scalable educational data collection, evaluation, and intervention design. However, current works are limited by scarce real interaction data, costly expert evaluation for realism, and a lack of large-scale, systematic analyses of LLMs ability in simulating students. We address this gap by presenting a three-stage LLM-human collaborative pipeline to automatically generate and filter high-quality student agents. We leverage a two-round automated scoring validated by human experts and deploy a score propagation module to obtain more consistent scores across the student similarity graph. Experiments show that combining automated scoring, expert calibration, and graph-based propagation yields simulated student that more closely track authentication by human judgments. We then analyze which profiles and behaviors are simulated more faithfully, supporting subsequent studies on personalized learning and educational assessment.
comment : This work has been submitted to AI Open for possible publication
♻ ☆ Scaling Multimodal Search and Recommendation with Small Language Models via Upside-Down Reinforcement Learning ICDM 2025
Yu-Chen Lin, Sanat Sharma, Hari Manikandan, Jayant Kumar, Tracy Holloway King, Jing Zheng
In this work, we investigate how small language models (SLMs) can be scaled to support multimodal search and recommendation use cases while remaining efficient enough for real-time, resource-constrained deployments. We present a framework that combines upside-down reinforcement learning with synthetic data distillation from a large language model (Llama-3) to train a 100M-parameter GPT-2 model for multitask prompt generation. Despite being up to 80 times smaller than state-of-the-art large language models (LLMs), our SLM achieves relevance and diversity scores within 6% of competitive baselines such as Llama-3 8B, Qwen3 8B, and Ministral 8B. These results demonstrate that SLMs can effectively handle multimodal search and recommendation tasks, while dramatically reducing inference latency and memory overhead. Our study highlights the potential of lightweight models as practical engines for scalable multimodal discovery, bridging the gap between cutting-edge research and real-world multimodal applications such as media recommendations and creative content generation.
comment : Accepted by ICDM 2025 MMSR
♻ ☆ Causal LLM Routing: End-to-End Regret Minimization from Observational Data
Asterios Tsiourvas, Wei Sun, Georgia Perakis
LLM routing aims to select the most appropriate model for each query, balancing competing performance metrics such as accuracy and cost across a pool of language models. Prior approaches typically adopt a decoupled strategy, where the metrics are first predicted and the model is then selected based on these estimates. This setup is prone to compounding errors and often relies on full-feedback data, where each query is evaluated by all candidate models, which is costly to obtain and maintain in practice. In contrast, we learn from observational data, which records only the outcome of the model actually deployed. We propose a causal end-to-end framework that learns routing policies by minimizing decision-making regret from observational data. To enable efficient optimization, we introduce two theoretically grounded surrogate objectives: a classification-based upper bound, and a softmax-weighted regret approximation shown to recover the optimal policy at convergence. We further extend our framework to handle heterogeneous cost preferences via an interval-conditioned architecture. Experiments on public benchmarks show that our method outperforms existing baselines, achieving state-of-the-art performance across different embedding models.
♻ ☆ Semantic Mastery: Enhancing LLMs with Advanced Natural Language Understanding
Mohanakrishnan Hariharan
Large language models (LLMs) have greatly improved their capability in performing NLP tasks. However, deeper semantic understanding, contextual coherence, and more subtle reasoning are still difficult to obtain. The paper discusses state-of-the-art methodologies that advance LLMs with more advanced NLU techniques, such as semantic parsing, knowledge integration, and contextual reinforcement learning. We analyze the use of structured knowledge graphs, retrieval-augmented generation (RAG), and fine-tuning strategies that match models with human-level understanding. Furthermore, we address the incorporation of transformer-based architectures, contrastive learning, and hybrid symbolic-neural methods that address problems like hallucinations, ambiguity, and inconsistency in the factual perspectives involved in performing complex NLP tasks, such as question-answering text summarization and dialogue generation. Our findings show the importance of semantic precision for enhancing AI-driven language systems and suggest future research directions to bridge the gap between statistical language models and true natural language understanding.
♻ ☆ FlashFormer: Whole-Model Kernels for Efficient Low-Batch Inference
Aniruddha Nrusimha, William Brandon, Mayank Mishra, Yikang Shen, Rameswar Panda, Jonathan Ragan-Kelley, Yoon Kim
The size and compute characteristics of modern large language models have led to an increased interest in developing specialized kernels tailored for particular training and inference workloads. Existing kernels primarily optimize for compute utilization, targeting the large-batch training and inference settings. However, low-batch inference, where memory bandwidth and kernel launch overheads are significant factors, remains important for many applications of interest such as in edge deployment and latency-sensitive applications. This paper describes FlashFormer, which fuses the entire transformer forward pass into a single kernel for accelerating low-batch inference of large language models. Across various model sizes and quantizations settings, FlashFormer achieves nontrivial speedups compared to existing inference kernels.
♻ ☆ Similarity-Distance-Magnitude Activations
Allen Schmaltz
We introduce the Similarity-Distance-Magnitude (SDM) activation function, a more robust and interpretable formulation of the standard softmax activation function, adding Similarity (i.e., correctly predicted depth-matches into training) awareness and Distance-to-training-distribution awareness to the existing output Magnitude (i.e., decision-boundary) awareness, and enabling interpretability-by-exemplar via dense matching. We further introduce the SDM estimator, based on a data-driven partitioning of the class-wise empirical CDFs via the SDM activation, to control the class- and prediction-conditional accuracy among selective classifications. When used as the final-layer activation over pre-trained language models for selective classification, the SDM estimator is more robust to co-variate shifts and out-of-distribution inputs than existing calibration methods using softmax activations, while remaining informative over in-distribution data.
comment : 21 pages, 8 tables, 1 algorithm. arXiv admin note: substantial text overlap with arXiv:2502.20167
♻ ☆ Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design
Quentin Anthony, Yury Tokpanov, Skyler Szot, Srivatsan Rajagopal, Praneeth Medepalli, Anna Golubeva, Vasu Shyam, Robert Washbourne, Rishi Iyer, Ansh Chaurasia, Tomas Figliolia, Xiao Yang, Abhinav Sarje, Drew Thorstensen, Amartey Pearson, Zack Grossbart, Jason van Patten, Emad Barsoum, Zhenyu Gu, Yao Fu, Beren Millidge
We report on the first large-scale mixture-of-experts (MoE) pretraining study on pure AMD hardware, utilizing both MI300X GPUs and Pollara networking. We distill practical guidance for both systems and model design. On the systems side, we deliver a comprehensive cluster and networking characterization: microbenchmarks for all core collectives (all-reduce, reduce-scatter, all-gather, broadcast) across message sizes and GPU counts over Pollara. To our knowledge, this is the first at this scale. We further provide MI300X microbenchmarks on kernel sizing and memory bandwidth to inform model design. On the modeling side, we introduce and apply MI300X-aware transformer sizing rules for attention and MLP blocks and justify MoE widths that jointly optimize training throughput and inference latency. We describe our training stack in depth, including often-ignored utilities such as fault-tolerance and checkpoint-reshaping, as well as detailed information on our training recipe. We also provide a preview of our model architecture and base model - ZAYA1 (760M active, 8.3B total parameters MoE, available at https://huggingface.co/Zyphra/ZAYA1-base) - which will be further improved upon in forthcoming papers. ZAYA1-base achieves performance comparable to leading base models such as Qwen3-4B and Gemma3-12B at its scale and larger, and outperforms models including Llama-3-8B and OLMoE across reasoning, mathematics, and coding benchmarks. Together, these results demonstrate that the AMD hardware, network, and software stack are mature and optimized enough for competitive large-scale pretraining.
Artificial Intelligence 253
☆ PPTArena: A Benchmark for Agent ic PowerPoint Editing
Michael Ofengenden, Yunze Man, Ziqi Pang, Yu-Xiong Wang
We introduce PPTArena, a benchmark for PowerPoint editing that measures reliable modifications to real slides under natural-language instructions. In contrast to image-PDF renderings or text-to-slide generation, PPTArena focuses on in-place editing across 100 decks, 2125 slides, and over 800 targeted edits covering text, charts, tables, animations, and master-level styles. Each case includes a ground-truth deck, a fully specified target outcome, and a dual VLM-as-judge pipeline that separately scores instruction following and visual quality using both structural diffs and slide images. Building on this setting, we propose PPTPilot, a structure-aware slide-editing agent that plans semantic edit sequences, routes between high-level programmatic tools and deterministic XML operations for precise control, and verifies outputs through an iterative plan-edit-check loop against task-specific constraints. In our experiments, PPTPilot outperforms strong proprietary agents and frontier VLM systems by over 10 percentage points on compound, layout-sensitive, and cross-slide edits, with particularly large gains in visual fidelity and deck-wide consistency. Despite these improvements, existing agents still underperform on long-horizon, document-scale tasks in PPTArena, highlighting the remaining challenges in reliable PPT editing.
comment : 25 pages, 26 figures
☆ Video4Spatial: Towards Visuospatial Intelligence with Context -Guided Video Generation
Zeqi Xiao, Yiwei Zhao, Lingxiao Li, Yushi Lan, Yu Ning, Rahul Garg, Roshni Cooper, Mohammad H. Taghavi, Xingang Pan
We investigate whether video generative models can exhibit visuospatial intelligence, a capability central to human cognition, using only visual data. To this end, we present Video4Spatial, a framework showing that video diffusion models conditioned solely on video-based scene context can perform complex spatial tasks. We validate on two tasks: scene navigation - following camera-pose instructions while remaining consistent with 3D geometry of the scene, and object grounding - which requires semantic localization, instruction following, and planning. Both tasks use video-only inputs, without auxiliary modalities such as depth or poses. With simple yet effective design choices in the framework and data curation, Video4Spatial demonstrates strong spatial understanding from video context: it plans navigation and grounds target objects end-to-end, follows camera-pose instructions while maintaining spatial consistency, and generalizes to long contexts and out-of-domain environments. Taken together, these results advance video generative models toward general visuospatial reasoning.
comment : Project page at https://xizaoqu.github.io/video4spatial/
☆ ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generation
Mengchen Zhang, Qi Chen, Tong Wu, Zihan Liu, Dahua Lin
Despite progress in video-to-audio generation, the field focuses predominantly on mono output, lacking spatial immersion. Existing binaural approaches remain constrained by a two-stage pipeline that first generates mono audio and then performs spatialization, often resulting in error accumulation and spatio-temporal inconsistencies. To address this limitation, we introduce the task of end-to-end binaural spatial audio generation directly from silent video. To support this task, we present the BiAudio dataset, comprising approximately 97K video-binaural audio pairs spanning diverse real-world scenes and camera rotation trajectories, constructed through a semi-automated pipeline. Furthermore, we propose ViSAudio, an end-to-end framework that employs conditional flow matching with a dual-branch audio generation architecture, where two dedicated branches model the audio latent flows. Integrated with a conditional spacetime module, it balances consistency between channels while preserving distinctive spatial characteristics, ensuring precise spatio-temporal alignment between audio and the input video. Comprehensive experiments demonstrate that ViSAudio outperforms existing state-of-the-art methods across both objective metrics and subjective evaluations, generating high-quality binaural audio with spatial immersion that adapts effectively to viewpoint changes, sound-source motion, and diverse acoustic environments. Project website: https://kszpxxzmc.github.io/ViSAudio-project.
☆ SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control
Yuxuan Mu, Ziyu Zhang, Yi Shi, Minami Matsumoto, Kotaro Imamura, Guy Tevet, Chuan Guo, Michael Taylor, Chang Shu, Pengcheng Xi, Xue Bin Peng
Data-driven motion priors that can guide agents toward producing naturalistic behaviors play a pivotal role in creating life-like virtual characters. Adversarial imitation learning has been a highly effective method for learning motion priors from reference motion data. However, adversarial priors, with few exceptions, need to be retrained for each new controller, thereby limiting their reusability and necessitating the retention of the reference motion data when training on downstream tasks. In this work, we present Score-Matching Motion Priors (SMP), which leverages pre-trained motion diffusion models and score distillation sampling (SDS) to create reusable task-agnostic motion priors. SMPs can be pre-trained on a motion dataset, independent of any control policy or task. Once trained, SMPs can be kept frozen and reused as general-purpose reward functions to train policies to produce naturalistic behaviors for downstream tasks. We show that a general motion prior trained on large-scale datasets can be repurposed into a variety of style-specific priors. Furthermore SMP can compose different styles to synthesize new styles not present in the original dataset. Our method produces high-quality motion comparable to state-of-the-art adversarial imitation learning methods through reusable and modular motion priors. We demonstrate the effectiveness of SMP across a diverse suite of control tasks with physically simulated humanoid characters. Video demo available at https://youtu.be/ravlZJteS20
comment : 14 pages, 9 figures
☆ The Moral Consistency Pipeline: Continuous Ethical Evaluation for Large Language Models
Saeid Jamshidi, Kawser Wazed Nafi, Arghavan Moradi Dakhel, Negar Shahabi, Foutse Khomh
The rapid advancement and adaptability of Large Language Models (LLMs) highlight the need for moral consistency, the capacity to maintain ethically coherent reasoning across varied contexts. Existing alignment frameworks, structured approaches designed to align model behavior with human ethical and social norms, often rely on static datasets and post-hoc evaluations, offering limited insight into how ethical reasoning may evolve across different contexts or temporal scales. This study presents the Moral Consistency Pipeline (MoCoP), a dataset-free, closed-loop framework for continuously evaluating and interpreting the moral stability of LLMs. MoCoP combines three supporting layers: (i) lexical integrity analysis, (ii) semantic risk estimation, and (iii) reasoning-based judgment modeling within a self-sustaining architecture that autonomously generates, evaluates, and refines ethical scenarios without external supervision. Our empirical results on GPT-4-Turbo and DeepSeek suggest that MoCoP effectively captures longitudinal ethical behavior, revealing a strong inverse relationship between ethical and toxicity dimensions (correlation rET = -0.81, p value less than 0.001) and a near-zero association with response latency (correlation rEL approximately equal to 0). These findings demonstrate that moral coherence and linguistic safety tend to emerge as stable and interpretable characteristics of model behavior rather than short-term fluctuations. Furthermore, by reframing ethical evaluation as a dynamic, model-agnostic form of moral introspection, MoCoP offers a reproducible foundation for scalable, continuous auditing and advances the study of computational morality in autonomous AI systems.
☆ LORE: A Large Generative Model for Search Relevance
Chenji Lu, Zhuo Chen, Hui Zhao, Zhiyuan Zeng, Gang Zhao, Junjie Ren, Ruicong Xu, Haoran Li, Songyan Liu, Pengjie Wang, Jian Xu, Bo Zheng
Achievement. We introduce LORE, a systematic framework for Large Generative Model-based relevance in e-commerce search. Deployed and iterated over three years, LORE achieves a cumulative +27\% improvement in online GoodRate metrics. This report shares the valuable experience gained throughout its development lifecycle, spanning data, features, training, evaluation, and deployment. Insight. While existing works apply Chain-of-Thought (CoT) to enhance relevance, they often hit a performance ceiling. We argue this stems from treating relevance as a monolithic task, lacking principled deconstruction. Our key insight is that relevance comprises distinct capabilities: knowledge and reasoning, multi-modal matching, and rule adherence. We contend that a qualitative-driven decomposition is essential for breaking through current performance bottlenecks. Contributions. LORE provides a complete blueprint for the LLM relevance lifecycle. Key contributions include: (1) A two-stage training paradigm combining progressive CoT synthesis via SFT with human preference alignment via RL. (2) A comprehensive benchmark, RAIR, designed to evaluate these core capabilities. (3) A query frequency-stratified deployment strategy that efficiently transfers offline LLM capabilities to the online system. LORE serves as both a practical solution and a methodological reference for other vertical domains.
☆ TokenPowerBench: Benchmark ing the Power Consumption of LLM Inference AAAI'26
Chenxu Niu, Wei Zhang, Jie Li, Yongjian Zhao, Tongyang Wang, Xi Wang, Yong Chen
Large language model (LLM) services now answer billions of queries per day, and industry reports show that inference, not training, accounts for more than 90% of total power consumption. However, existing benchmarks focus on either training/fine-tuning or performance of inference and provide little support for power consumption measurement and analysis of inference. We introduce TokenPowerBench, the first lightweight and extensible benchmark designed for LLM-inference power consumption studies. The benchmark combines (i) a declarative configuration interface covering model choice, prompt set, and inference engine, (ii) a measurement layer that captures GPU-, node-, and system-level power without specialized power meters, and (iii) a phase-aligned metrics pipeline that attributes energy to the prefill and decode stages of every request. These elements make it straight-forward to explore the power consumed by an LLM inference run; furthermore, by varying batch size, context length, parallelism strategy and quantization, users can quickly assess how each setting affects joules per token and other energy-efficiency metrics. We evaluate TokenPowerBench on four of the most widely used model series (Llama, Falcon, Qwen, and Mistral). Our experiments cover from 1 billion parameters up to the frontier-scale Llama3-405B model. Furthermore, we release TokenPowerBench as open source to help users to measure power consumption, forecast operating expenses, and meet sustainability targets when deploying LLM services.
comment : Accepted by the AAAI'26 Conference Main Track
☆ Distribution-Calibrated Inference time compute for Thinking LLM-as-a- Judge
Hamid Dadkhahi, Firas Trabelsi, Parker Riley, Juraj Juraska, Mehdi Mirzazadeh
Thinking Large Language Models (LLMs) used as judges for pairwise preferences remain noisy at the single-sample level, and common aggregation rules (majority vote, soft self-consistency, or instruction-based self-aggregation) are inconsistent when ties are allowed. We study inference-time compute (ITC) for evaluators that generate n independent thinking-rating samples per item, and propose a principled, distribution-calibrated aggregation scheme. Our method models three-way preferences with a Bradley-Terry-Davidson formulation on rating counts, leveraging both polarity (margin among non-ties) and decisiveness (non-tie rate) to distinguish narrow margins from strong consensus. Across various evaluation benchmarks, our approach consistently reduces MAE and increases pairwise accuracy versus standard baselines, and when evaluated against human-consensus meta-labels, matches or exceeds individual human raters. These results show that carefully allocating ITC and aggregating with distribution-aware methods turns noisy individual model judgments into reliable ratings for evaluation.
☆ In- Context Sync-LoRA for Portrait Video Editing
Sagi Polaczek, Or Patashnik, Ali Mahdavi-Amiri, Daniel Cohen-Or
Editing portrait videos is a challenging task that requires flexible yet precise control over a wide range of modifications, such as appearance changes, expression edits, or the addition of objects. The key difficulty lies in preserving the subject's original temporal behavior, demanding that every edited frame remains precisely synchronized with the corresponding source frame. We present Sync-LoRA, a method for editing portrait videos that achieves high-quality visual modifications while maintaining frame-accurate synchronization and identity consistency. Our approach uses an image-to-video diffusion model, where the edit is defined by modifying the first frame and then propagated to the entire sequence. To enable accurate synchronization, we train an in-context LoRA using paired videos that depict identical motion trajectories but differ in appearance. These pairs are automatically generated and curated through a synchronization-based filtering process that selects only the most temporally aligned examples for training. This training setup teaches the model to combine motion cues from the source video with the visual changes introduced in the edited first frame. Trained on a compact, highly curated set of synchronized human portraits, Sync-LoRA generalizes to unseen identities and diverse edits (e.g., modifying appearance, adding objects, or changing backgrounds), robustly handling variations in pose and expression. Our results demonstrate high visual fidelity and strong temporal coherence, achieving a robust balance between edit fidelity and precise motion preservation.
comment : Project page: https://sagipolaczek.github.io/Sync-LoRA/
☆ From Moderation to Mediation: Can LLMs Serve as Mediators in Online Flame Wars?
Dawei Li, Abdullah Alnaibari, Arslan Bisharat, Manny Sandoval, Deborah Hall, Yasin Silva, Huan Liu
The rapid advancement of large language models (LLMs) has opened new possibilities for AI for good applications. As LLMs increasingly mediate online communication, their potential to foster empathy and constructive dialogue becomes an important frontier for responsible AI research. This work explores whether LLMs can serve not only as moderators that detect harmful content, but as mediators capable of understanding and de-escalating online conflicts. Our framework decomposes mediation into two subtasks: judgment, where an LLM evaluates the fairness and emotional dynamics of a conversation, and steering, where it generates empathetic, de-escalatory messages to guide participants toward resolution. To assess mediation quality, we construct a large Reddit-based dataset and propose a multi-stage evaluation pipeline combining principle-based scoring, user simulation, and human comparison. Experiments show that API-based models outperform open-source counterparts in both reasoning and intervention alignment when doing mediation. Our findings highlight both the promise and limitations of current LLMs as emerging agents for online social mediation.
comment : Under review
☆ Invasive Context Engineering to Control Large Language Models
Thomas Rivasseau
Current research on operator control of Large Language Models improves model robustness against adversarial attacks and misbehavior by training on preference examples, prompting, and input/output filtering. Despite good results, LLMs remain susceptible to abuse, and jailbreak probability increases with context length. There is a need for robust LLM security guarantees in long-context situations. We propose control sentences inserted into the LLM context as invasive context engineering to partially solve the problem. We suggest this technique can be generalized to the Chain-of-Thought process to prevent scheming. Invasive Context Engineering does not rely on LLM training, avoiding data shortage pitfalls which arise in training models for long context situations.
comment : 4 pages
☆ Fine-Tuned Large Language Models for Logical Translation: Reducing Hallucinations with Lang2Logic
Muyu Pan, Dheeraj Kodakandla, Mahfuza Farooque
Recent advances in natural language processing (NLP), particularly large language models (LLMs), have motivated the automatic translation of natural language statements into formal logic without human intervention. This enables automated reasoning and facilitates debugging, finding loop invariants, and adhering to specifications in software systems. However, hallucinations-incorrect outputs generated by LLMs are challenging, particularly for logical translation tasks requiring precision. This work introduces a novel framework that inputs English sentences, converts them into logical expressions, and then translates them into Conjunctive Normal Form (CNF) for satisfiability solving. It employs classical NLP techniques with self-defined grammar, symbolic computation libraries, and a fine-tuned language model to reduce hallucinations. In the early experiments, we observed that the fine-tuned model, trained on different grammar settings, could intentionally correct the same types of hallucinations made by the original model. Thus, it provides reliable CNF generation.
comment : IEEE ISNCC 2025
☆ Rethinking Generalized BCIs: Benchmark ing 340,000+ Unique Algorithmic Configurations for EEG Mental Command Decoding
Paul Barbaste, Olivier Oullier, Xavier Vasques
Robust decoding and classification of brain patterns measured with electroencephalography (EEG) remains a major challenge for real-world (i.e. outside scientific lab and medical facilities) brain-computer interface (BCI) applications due to well documented inter- and intra-participant variability. Here, we present a large-scale benchmark evaluating over 340,000+ unique combinations of spatial and nonlinear EEG classification. Our methodological pipeline consists in combinations of Common Spatial Patterns (CSP), Riemannian geometry, functional connectivity, and fractal- or entropy-based features across three open-access EEG datasets. Unlike prior studies, our analysis operates at the per-participant level and across multiple frequency bands (8-15 Hz and 8-30 Hz), enabling direct assessment of both group-level performance and individual variability. Covariance tangent space projection (cov-tgsp) and CSP consistently achieved the highest average classification accuracies. However, their effectiveness was strongly dataset-dependent, and marked participant-level differences persisted, particularly in the most heterogeneous of the datasets. Importantly, nonlinear methods outperformed spatial approaches for specific individuals, underscoring the need for personalized pipeline selection. Our findings highlight that no universal 'one-size-fits-all' method can optimally decode EEG motor imagery patterns across all users or datasets. Future work will require adaptive, multimodal, and possibly novel approaches to fully address neurophysiological variability in practical BCI applications where the system can automatically adapt to what makes each user unique.
comment : 28 pages, 8 figures, 2 tables
☆ Lumos: Let there be Language Model System Certification
Isha Chaudhary, Vedaant Jain, Avaljot Singh, Kavya Sachdeva, Sayan Ranu, Gagandeep Singh
We introduce the first principled framework, Lumos, for specifying and formally certifying Language Model System (LMS) behaviors. Lumos is an imperative probabilistic programming DSL over graphs, with constructs to generate independent and identically distributed prompts for LMS. It offers a structured view of prompt distributions via graphs, forming random prompts from sampled subgraphs. Lumos supports certifying LMS for arbitrary prompt distributions via integration with statistical certifiers. We provide hybrid (operational and denotational) semantics for Lumos, providing a rigorous way to interpret the specifications. Using only a small set of composable constructs, Lumos can encode existing LMS specifications, including complex relational and temporal specifications. It also facilitates specifying new properties - we present the first safety specifications for vision-language models (VLMs) in autonomous driving scenarios developed with Lumos. Using these, we show that the state-of-the-art VLM Qwen-VL exhibits critical safety failures, producing incorrect and unsafe responses with at least 90% probability in right-turn scenarios under rainy driving conditions, revealing substantial safety risks. Lumos's modular structure allows easy modification of the specifications, enabling LMS certification to stay abreast with the rapidly evolving threat landscape. We further demonstrate that specification programs written in Lumos enable finding specific failure cases exhibited by state-of-the-art LMS. Lumos is the first systematic and extensible language-based framework for specifying and certifying LMS behaviors, paving the way for a wider adoption of LMS certification.
☆ Benchmark ing Scientific Understanding and Reasoning for Video Generation using VideoScience-Bench
Lanxiang Hu, Abhilash Shankarampeta, Yixin Huang, Zilin Dai, Haoyang Yu, Yujie Zhao, Haoqiang Kang, Daniel Zhao, Tajana Rosing, Hao Zhang
The next frontier for video generation lies in developing models capable of zero-shot reasoning, where understanding real-world scientific laws is crucial for accurate physical outcome modeling under diverse conditions. However, existing video benchmarks are physical commonsense-based, offering limited insight into video models' scientific reasoning capability. We introduce VideoScience-Bench, a benchmark designed to evaluate undergraduate-level scientific understanding in video models. Each prompt encodes a composite scientific scenario that requires understanding and reasoning across multiple scientific concepts to generate the correct phenomenon. The benchmark comprises 200 carefully curated prompts spanning 14 topics and 103 concepts in physics and chemistry. We conduct expert-annotated evaluations across seven state-of-the-art video models in T2V and I2V settings along five dimensions: Prompt Consistency, Phenomenon Congruency, Correct Dynamism, Immutability, and Spatio-Temporal Continuity. Using a VLM-as-a-Judge to assess video generations, we observe strong correlation with human assessments. To the best of our knowledge, VideoScience-Bench is the first benchmark to evaluate video models not only as generators but also as reasoners, requiring their generations to demonstrate scientific understanding consistent with expected physical and chemical phenomena. Our data and evaluation code are available at: \href{https://github.com/hao-ai-lab/VideoScience}{github.com/hao-ai-lab/VideoScience}.
☆ EGGS: Exchangeable 2D/3D Gaussian Splatting for Geometry-Appearance Balanced Novel View Synthesis
Yancheng Zhang, Guangyu Sun, Chen Chen
Novel view synthesis (NVS) is crucial in computer vision and graphics, with wide applications in AR, VR, and autonomous driving. While 3D Gaussian Splatting (3DGS) enables real-time rendering with high appearance fidelity, it suffers from multi-view inconsistencies, limiting geometric accuracy. In contrast, 2D Gaussian Splatting (2DGS) enforces multi-view consistency but compromises texture details. To address these limitations, we propose Exchangeable Gaussian Splatting (EGGS), a hybrid representation that integrates 2D and 3D Gaussians to balance appearance and geometry. To achieve this, we introduce Hybrid Gaussian Rasterization for unified rendering, Adaptive Type Exchange for dynamic adaptation between 2D and 3D Gaussians, and Frequency-Decoupled Optimization that effectively exploits the strengths of each type of Gaussian representation. Our CUDA-accelerated implementation ensures efficient training and inference. Extensive experiments demonstrate that EGGS outperforms existing methods in rendering quality, geometric accuracy, and efficiency, providing a practical solution for high-quality NVS.
☆ Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning NeurIPS 2025
Zhonghao He, Tianyi Qiu, Hirokazu Shirado, Maarten Sap
Recent advances in reasoning techniques have substantially improved the performance of large language models (LLMs), raising expectations for their ability to provide accurate, truthful, and reliable information. However, emerging evidence suggests that iterative reasoning may foster belief entrenchment and confirmation bias, rather than enhancing truth-seeking behavior. In this study, we propose a systematic evaluation framework for belief entrenchment in LLM reasoning by leveraging the Martingale property from Bayesian statistics. This property implies that, under rational belief updating, the expected value of future beliefs should remain equal to the current belief, i.e., belief updates are unpredictable from the current belief. We propose the unsupervised, regression-based Martingale Score to measure violations of this property, which signal deviation from the Bayesian ability of updating on new evidence. In open-ended problem domains including event forecasting, value-laden questions, and academic paper review, we find such violations to be widespread across models and setups, where the current belief positively predicts future belief updates, a phenomenon which we term belief entrenchment. We identify the models, reasoning techniques, and domains more prone to belief entrenchment. Finally, we validate the Martingale Score by showing that it predicts ground-truth accuracy on problem domains where ground truth labels are available. This indicates that, while designed as an unsupervised metric that operates even in domains without access to ground truth, the Martingale Score is a useful proxy of the truth-seeking ability of a reasoning process.
comment : NeurIPS 2025
☆ In Silico Development of Psychometric Scales: Feasibility of Representative Population Data Simulation with LLMs
Enrico Cipriani, Pavel Okopnyi, Danilo Menicucci, Simone Grassini
Developing and validating psychometric scales requires large samples, multiple testing phases, and substantial resources. Recent advances in Large Language Models (LLMs) enable the generation of synthetic participant data by prompting models to answer items while impersonating individuals of specific demographic profiles, potentially allowing in silico piloting before real data collection. Across four preregistered studies (N = circa 300 each), we tested whether LLM-simulated datasets can reproduce the latent structures and measurement properties of human responses. In Studies 1-2, we compared LLM-generated data with real datasets for two validated scales; in Studies 3-4, we created new scales using EFA on simulated data and then examined whether these structures generalized to newly collected human samples. Simulated datasets replicated the intended factor structures in three of four studies and showed consistent configural and metric invariance, with scalar invariance achieved for the two newly developed scales. However, correlation-based tests revealed substantial differences between real and synthetic datasets, and notable discrepancies appeared in score distributions and variances. Thus, while LLMs capture group-level latent structures, they do not approximate individual-level data properties. Simulated datasets also showed full internal invariance across gender. Overall, LLM-generated data appear useful for early-stage, group-level psychometric prototyping, but not as substitutes for individual-level validation. We discuss methodological limitations, risks of bias and data pollution, and ethical considerations related to in silico psychometric simulations.
☆ MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding
Fan Yang, Kaihao Zhang
Understanding high-resolution images remains a significant challenge for multimodal large language models (MLLMs). Recent study address this issue by dividing the image into smaller crops and computing the semantic similarity between each crop and a query using a pretrained retrieval-augmented generation (RAG) model. The most relevant crops are then selected to localize the target object and suppress irrelevant information. However, such crop-based processing can fragment complete objects across multiple crops, thereby disrupting the computation of semantic similarity. In our experiments, we find that image crops of objects with different sizes are better handled at different resolutions. Based on this observation, we propose Multi-resolution Retrieval-Detection (MRD), a training-free framework for high-resolution image understanding. To address the issue of semantic similarity bias caused by objects being split across different image crops, we propose a multi-resolution semantic fusion method, which integrates semantic similarity maps obtained at different resolutions to produce more accurate semantic information and preserve the integrity of target objects. Furthermore, to achieve direct localization of target objects at a global scale, we introduce an open-vocalbulary object detection (OVD) model that identifies object regions using a sliding-window approach.Experiments on high-resolution image understanding benchmarks using different MLLMs demonstrate the effectiveness of our approach.
☆ Towards a fully differentiable digital twin for solar cells
Marie Louise Schubert, Houssam Metni, Jan David Fischbach, Benedikt Zerulla, Marjan Krstić, Ulrich W. Paetzold, Seyedamir Orooji, Olivier J. J. Ronsin, Yasin Ameslon, Jens Harting, Thomas Kirchartz, Sandheep Ravishankar, Chris Dreessen, Eunchi Kim, Christian Sprau, Mohamed Hussein, Alexander Colsmann, Karen Forberich, Klaus Jäger, Pascal Friederich, Carsten Rockstuhl
Maximizing energy yield (EY) - the total electric energy generated by a solar cell within a year at a specific location - is crucial in photovoltaics (PV), especially for emerging technologies. Computational methods provide the necessary insights and guidance for future research. However, existing simulations typically focus on only isolated aspects of solar cells. This lack of consistency highlights the need for a framework unifying all computational levels, from material to cell properties, for accurate prediction and optimization of EY prediction. To address this challenge, a differentiable digital twin, Sol(Di)$^2$T, is introduced to enable comprehensive end-to-end optimization of solar cells. The workflow starts with material properties and morphological processing parameters, followed by optical and electrical simulations. Finally, climatic conditions and geographic location are incorporated to predict the EY. Each step is either intrinsically differentiable or replaced with a machine-learned surrogate model, enabling not only accurate EY prediction but also gradient-based optimization with respect to input parameters. Consequently, Sol(Di)$^2$T extends EY predictions to previously unexplored conditions. Demonstrated for an organic solar cell, the proposed framework marks a significant step towards tailoring solar cells for specific applications while ensuring maximal performance.
☆ VLA Models Are More Generalizable Than You Think: Revisiting Physical and Spatial Modeling
Weiqi Li, Quande Zhang, Ruifeng Zhai, Liang Lin, Guangrun Wang
Vision-language-action (VLA) models achieve strong in-distribution performance but degrade sharply under novel camera viewpoints and visual perturbations. We show that this brittleness primarily arises from misalignment in Spatial Modeling, rather than Physical Modeling. To address this, we propose a one-shot adaptation framework that recalibrates visual representations through lightweight, learnable updates. Our first method, Feature Token Modulation (FTM), applies a global affine transformation to visual tokens and improves Libero viewpoint accuracy from 48.5% to 87.1% with only 4K parameters. Building on this, Feature Linear Adaptation (FLA) introduces low-rank updates to the ViT encoder, achieving 90.8% success with 4.7M parameters -- matching LoRA-scale finetuning at far lower cost. Together, these results reveal substantial untapped robustness in pretrained VLA models and demonstrate that targeted, minimal visual adaptation is sufficient to restore viewpoint generalization.
☆ FAIRY2I: Universal Extremely-Low Bit QAT framework via Widely-Linear Representation and Phase-Aware Quantization
Feiyu Wang, Xinyu Tan, Bokai Huang, Yihao Zhang, Guoan Wang, Peizhuang Cong, Tong Yang
Large language models (LLMs) have revolutionized artificial intelligence, yet their massive memory and computational demands necessitate aggressive quantization, increasingly pushing representations toward the theoretical limit of a single bit. While complex-valued LLMs, such as iFairy, offer a superior chance for low-bit representation compared to real-valued counterparts, they require training from scratch, preventing the utilization of the vast ecosystem of pre-trained real-valued foundation models. Here we present Fairy2i, a universal framework that transforms pre-trained real-valued layers into an equivalent widely-linear complex form, enabling extremely low-bit quantization while reusing existing checkpoints. By proving a lossless mathematical equivalence between real and widely-linear maps, we convert standard Transformers into the complex domain and employ a phase-aware quantization scheme with a highly efficient codebook of fourth roots of unity. Furthermore, we introduce a recursive residual quantization mechanism that iteratively minimizes quantization error, allowing inference to proceed via efficient multiplication-free accumulation. We demonstrate that Fairy2i restores the performance of LLaMA-2 7B at an effective 2-bit precision to levels nearly comparable with full-precision baselines, significantly outperforming state-of-the-art real-valued binary and ternary quantization methods. This work bridges the gap between the representational efficiency of complex-valued arithmetic and the practical utility of pre-trained models, paving a new way for efficient inference on commodity hardware.
comment : 15 pages, 3 figures
☆ Model-Based Diagnosis with Multiple Observations: A Unified Approach for C Software and Boolean Circuits
Pedro Orvalho, Marta Kwiatkowska, Mikoláš Janota, Vasco Manquinho
Debugging is one of the most time-consuming and expensive tasks in software development and circuit design. Several formula-based fault localisation (FBFL) methods have been proposed, but they fail to guarantee a set of diagnoses across all failing tests or may produce redundant diagnoses that are not subset-minimal, particularly for programs/circuits with multiple faults. This paper introduces CFaults, a novel fault localisation tool for C software and Boolean circuits with multiple faults. CFaults leverages Model-Based Diagnosis (MBD) with multiple observations and aggregates all failing test cases into a unified Maximum Satisfiability (MaxSAT) formula. Consequently, our method guarantees consistency across observations and simplifies the fault localisation procedure. Experimental results on three benchmark sets, two of C programs, TCAS and C-Pack-IPAs, and one of Boolean circuits, ISCAS85, show that CFaults is faster at localising faults in C software than other FBFL approaches such as BugAssist, SNIPER, and HSD. On the ISCAS85 benchmark, CFaults is generally slower than HSD; however, it localises faults in only 6% fewer circuits, demonstrating that it remains competitive in this domain. Furthermore, CFaults produces only subset-minimal diagnoses of faulty statements, whereas the other approaches tend to enumerate redundant diagnoses (e.g., BugAssist and SNIPER).
comment : 50 pages, 9 figures, 6 tables, 5 listings
☆ OptPO: Optimal Rollout Allocation for Test-time Policy Optimization
Youkang Wang, Jian Wang, Rubing Chen, Tianyi Zeng, Xiao-Yong Wei, Qing Li
Test-time policy optimization enables large language models (LLMs) to adapt to distribution shifts by leveraging feedback from self-generated rollouts. However, existing methods rely on fixed-budget majority voting to estimate rewards, incurring substantial computational redundancy. We propose Optimal Rollout Allocation for Test-time Policy Optimization (OptPO), a principled framework that adaptively allocates inference budgets. By formulating the voting process as a Bayesian sequential probability ratio test, OptPO dynamically halts sampling once the posterior confidence in a consensus answer exceeds a specified threshold. Crucially, it utilizes the retained rollouts for on-policy updates, seamlessly integrating with algorithms like PPO or GRPO without requiring ground-truth labels. Across diverse reasoning benchmarks, OptPO significantly reduces rollout overhead compared to fixed-sample baselines while preserving or improving accuracy. By unifying statistically optimal stopping with test-time learning, OptPO offers a computationally efficient paradigm for test-time adaptation. The source code will be open upon acceptance at https://open-upon-acceptance.
comment : Work in Progress
☆ The future of AI in critical mineral exploration
Jef Caers
The energy transition through increased electrification has put the worlds attention on critical mineral exploration Even with increased investments a decrease in new discoveries has taken place over the last two decades Here I propose a solution to this problem where AI is implemented as the enabler of a rigorous scientific method for mineral exploration that aims to reduce cognitive bias and false positives drive down the cost of exploration I propose a new scientific method that is based on a philosophical approach founded on the principles of Bayesianism and falsification In this approach data acquisition is in the first place seen as a means to falsify human generated hypothesis Decision of what data to acquire next is quantified with verifiable metrics and based on rational decision making A practical protocol is provided that can be used as a template in any exploration campaign However in order to make this protocol practical various form of artificial intelligence are needed I will argue that the most important form are one novel unsupervised learning methods that collaborate with domain experts to better understand data and generate multiple competing geological hypotheses and two humanintheloop AI algorithms that can optimally plan various geological geophysical geochemical and drilling data acquisition where uncertainty reduction of geological hypothesis precedes the uncertainty reduction on grade and tonnage
☆ GraphMatch: Fusing Language and Graph Representations in a Dynamic Two-Sided Work Marketplace
Mikołaj Sacha, Hammad Jafri, Mattie Terzolo, Ayan Sinha, Andrew Rabinovich
Recommending matches in a text-rich, dynamic two-sided marketplace presents unique challenges due to evolving content and interaction graphs. We introduce GraphMatch, a new large-scale recommendation framework that fuses pre-trained language models with graph neural networks to overcome these challenges. Unlike prior approaches centered on standalone models, GraphMatch is a comprehensive recipe built on powerful text encoders and GNNs working in tandem. It employs adversarial negative sampling alongside point-in-time subgraph training to learn representations that capture both the fine-grained semantics of evolving text and the time-sensitive structure of the graph. We evaluated extensively on interaction data from Upwork, a leading labor marketplace, at large scale, and discuss our approach towards low-latency inference suitable for real-time use. In our experiments, GraphMatch outperforms language-only and graph-only baselines on matching tasks while being efficient at runtime. These results demonstrate that unifying language and graph representations yields a highly effective solution to text-rich, dynamic two-sided recommendations, bridging the gap between powerful pretrained LMs and large-scale graphs in practice.
☆ Cross-Lingual Prompt Steerability: Towards Accurate and Robust LLM Behavior across Languages
Lechen Zhang, Yusheng Zhou, Tolga Ergen, Lajanugen Logeswaran, Moontae Lee, David Jurgens
System prompts provide a lightweight yet powerful mechanism for conditioning large language models (LLMs) at inference time. While prior work has focused on English-only settings, real-world deployments benefit from having a single prompt to operate reliably across languages. This paper presents a comprehensive study of how different system prompts steer models toward accurate and robust cross-lingual behavior. We propose a unified four-dimensional evaluation framework to assess system prompts in multilingual environments. Through large-scale experiments on five languages, three LLMs, and three benchmarks, we uncover that certain prompt components, such as CoT, emotion, and scenario, correlate with robust multilingual behavior. We develop a prompt optimization framework for multilingual settings and show it can automatically discover prompts that improve all metrics by 5-10%. Finally, we analyze over 10 million reasoning units and find that more performant system prompts induce more structured and consistent reasoning patterns, while reducing unnecessary language-switching. Together, we highlight system prompt optimization as a scalable path to accurate and robust multilingual LLM behavior.
☆ ReVSeg: Incentivizing the Reasoning Chain for Video Segmentation with Reinforcement Learning
Yifan Li, Yingda Yin, Lingting Zhu, Weikai Chen, Shengju Qian, Xin Wang, Yanwei Fu
Reasoning-centric video object segmentation is an inherently complex task: the query often refers to dynamics, causality, and temporal interactions, rather than static appearances. Yet existing solutions generally collapse these factors into simplified reasoning with latent embeddings, rendering the reasoning chain opaque and essentially intractable. We therefore adopt an explicit decomposition perspective and introduce ReVSeg, which executes reasoning as sequential decisions in the native interface of pretrained vision language models (VLMs). Rather than folding all reasoning into a single-step prediction, ReVSeg executes three explicit operations -- semantics interpretation, temporal evidence selection, and spatial grounding -- aligning pretrained capabilities. We further employ reinforcement learning to optimize the multi-step reasoning chain, enabling the model to self-refine its decision quality from outcome-driven signals. Experimental results demonstrate that ReVSeg attains state-of-the-art performances on standard video object segmentation benchmarks and yields interpretable reasoning trajectories. Project page is available at https://clementine24.github.io/ReVSeg/ .
☆ Steering Vision -Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach
Siyuan Yang, Yang Zhang, Haoran He, Ling Pan, Xiu Li, Chenjia Bai, Xuelong Li
Vision-Language-Action (VLA) models, trained via flow-matching or diffusion objectives, excel at learning complex behaviors from large-scale, multi-modal datasets (e.g., human teleoperation, scripted policies). However, since VLAs incorporate diverse data modes in the pre-training stage, and the finetuning dataset often contains demonstration data collected in a kinematically suboptimal or undesirable way, it exists redundant action modes that are irrelevant to the success action modes of the downstream task. Specifically, we observe a critical inference-time fragility among various sampled noises after supervised finetuning of pre-trained VLAs. In this paper, we attribute this instability to the distribution shift between the VLA policy and the policy induced by stable success modes of the downstream task dataset. Thus, we propose \textbf{TACO}, a test-time-scaling (TTS) framework that applies a lightweight pseudo-count estimator as a high-fidelity verifier of action chunks. The VLA models integrated with TACO can execute the actions with maximum pseudo-count from all sampled action chunks, thereby preventing distribution shifts while preserving the generalization ability of VLAs since the constraint is applied only during inference. Our method resembles the classical anti-exploration principle in offline reinforcement learning (RL), and being gradient-free, it incurs significant computational benefits compared to RL update, especially for flow or diffusion-based VLAs which are difficult to perform RL update due to denoising process. Extensive experiments across four simulation benchmarks (RoboTwin2.0, Robotwin, LIBERO, SimplerEnv) and a dual-arm platform demonstrate that our method significantly improves the inference stability and success rates in downstream-task adaptations.
comment : The first two authors contributed equally. Yang Zhang leads the whole project
☆ A Comparative Study on How Data Normalization Affects Zero-Shot Generalization in Time Series Foundation Models
Ihab Ahmed, Denis Krompaß, Cheng Feng, Volker Tresp
We investigate input normalization methods for Time-Series Foundation Models (TSFMs). While normalization is well-studied in dataset-specific time-series models, it remains overlooked in TSFMs where generalization is critical. Time-series data, unlike text or images, exhibits significant scale variation across domains and channels, coupled with non-stationarity, can undermine TSFM performance regardless of architectural complexity. Through systematic evaluation across four architecturally diverse TSFMs, we empirically establish REVIN as the most efficient approach, reducing zero-shot MASE by 89\% relative to an un-normalized baseline and by 44\% versus other normalization methods, while matching the best in-domain accuracy (0.84 MASE) without any dataset-level preprocessing -- yielding the highest accuracy-efficiency trade-off. Yet its effect utilization depends on architectural design choices and optimization objective, particularly with respect to training loss scale sensitivity and model type (probabilistic, point-forecast, or LLM-based models).
☆ Defense That Attacks: How Robust Models Become Better Attackers
Mohamed Awad, Mahmoud Akrm, Walid Gomaa
Deep learning has achieved great success in computer vision, but remains vulnerable to adversarial attacks. Adversarial training is the leading defense designed to improve model robustness. However, its effect on the transferability of attacks is underexplored. In this work, we ask whether adversarial training unintentionally increases the transferability of adversarial examples. To answer this, we trained a diverse zoo of 36 models, including CNNs and ViTs, and conducted comprehensive transferability experiments. Our results reveal a clear paradox: adversarially trained (AT) models produce perturbations that transfer more effectively than those from standard models, which introduce a new ecosystem risk. To enable reproducibility and further study, we release all models, code, and experimental scripts. Furthermore, we argue that robustness evaluations should assess not only the resistance of a model to transferred attacks but also its propensity to produce transferable adversarial examples.
☆ From Navigation to Refinement: Revealing the Two-Stage Nature of Flow-based Diffusion Models through Oracle Velocity
Haoming Liu, Jinnuo Liu, Yanhao Li, Liuyang Bai, Yunkai Ji, Yuanhe Guo, Shenji Wan, Hongyi Wen
Flow-based diffusion models have emerged as a leading paradigm for training generative models across images and videos. However, their memorization-generalization behavior remains poorly understood. In this work, we revisit the flow matching (FM) objective and study its marginal velocity field, which admits a closed-form expression, allowing exact computation of the oracle FM target. Analyzing this oracle velocity field reveals that flow-based diffusion models inherently formulate a two-stage training target: an early stage guided by a mixture of data modes, and a later stage dominated by the nearest data sample. The two-stage objective leads to distinct learning behaviors: the early navigation stage generalizes across data modes to form global layouts, whereas the later refinement stage increasingly memorizes fine-grained details. Leveraging these insights, we explain the effectiveness of practical techniques such as timestep-shifted schedules, classifier-free guidance intervals, and latent space design choices. Our study deepens the understanding of diffusion model training dynamics and offers principles for guiding future architectural and algorithmic improvements.
comment : Preprint version; 15 pages, 16 figures
☆ Radiologist Copilot: An Agent ic Assistant with Orchestrated Tools for Radiology Reporting with Quality Control
Yongrui Yu, Zhongzhen Huang, Linjie Mu, Shaoting Zhang, Xiaofan Zhang
Radiology reporting is an essential yet time-consuming and error-prone task for radiologists in clinical examinations, especially for volumetric medical images. Rigorous quality control is also critical but tedious, ensuring that the final report meets clinical standards. Existing automated approaches, including radiology report generation methods and medical vision-language models, focus mainly on the report generation phase and neglect the crucial quality control procedure, limiting their capability to provide comprehensive support to radiologists. We propose Radiologist Copilot, an agentic AI assistant equipped with orchestrated tools designed for automated radiology reporting with quality control. Leveraging large language models as the reasoning backbone, the agentic system autonomously selects tools, plans, and executes actions, emulating the behavior of radiologists throughout the holistic radiology reporting process. The orchestrated tools include region localization, think with image paradigm directed region analysis planning, strategic template selection for report generation, quality assessment and feedback-driven adaptive refinement for quality control. Therefore, Radiologist Copilot facilitates accurate, complete, and efficient radiology reporting, assisting radiologists and improving clinical efficiency. Experimental results demonstrate that Radiologist Copilot significantly surpasses other state-of-the-art methods in radiology reporting. The source code will be released upon acceptance.
☆ Enhancing Automated Paper Reproduction via Prompt -Free Collaborative Agent s
Zijie Lin, Qilin Cai, Liang Shen, Mingjun Xiao
Automated paper reproduction has emerged as a promising approach to accelerate scientific research, employing multi-step workflow frameworks to systematically convert academic papers into executable code. However, existing frameworks often lack mechanisms to verify and refine the outputs at each generation step, or rely heavily on manually designed prompts for self-refinement, which limits their adaptability and scalability. To address these limitations, we propose a prompt-free collaborative agent framework that automatically enhances the quality of paper-to-code generation. Our approach employs two collaborative agents: a verification agent that examines whether the outputs at each step satisfy the requirements specified in the corresponding system prompt, and a refinement agent that revises the outputs based on the identified issues. Unlike previous methods that require human experts to craft specific refinement prompts for each step, our framework achieves automatic verification and improvement by leveraging only the original system prompts. We integrate our collaborative agents into the Paper2Code framework and conduct comprehensive experiments on PaperBench Code-Dev and Paper2CodeBench datasets. Experimental results demonstrate that our approach significantly improves the accuracy and completeness of reproduced code, achieving performance gains of approximately 15\% and 13\%, respectively, compared to the baseline without our agents. Furthermore, comparative experiments against Self-Refine validate the robustness and consistency of our prompt-free approach across different datasets.
☆ Phase-Adaptive LLM Framework with Multi-Stage Validation for Construction Robot Task Allocation: A Systematic Benchmark Against Traditional Optimization Algorithms
Shyam prasad reddy Kaitha, Hongrui Yu
Multi-robot task allocation in construction automation has traditionally relied on optimization methods such as Dynamic Programming and Reinforcement Learning. This research introduces the LangGraph-based Task Allocation Agent (LTAA), an LLM-driven framework that integrates phase-adaptive allocation strategies, multi-stage validation with hierarchical retries, and dynamic prompting for efficient robot coordination. Although recent LLM approaches show potential for construction robotics, they largely lack rigorous validation and benchmarking against established algorithms. This paper presents the first systematic comparison of LLM-based task allocation with traditional methods in construction scenarios.The study validates LLM feasibility through SMART-LLM replication and addresses implementation challenges using a Self-Corrective Agent Architecture. LTAA leverages natural-language reasoning combined with structured validation mechanisms, achieving major computational gains reducing token usage by 94.6% and allocation time by 86% through dynamic prompting. The framework adjusts its strategy across phases: emphasizing execution feasibility early and workload balance in later allocations.The authors evaluate LTAA against Dynamic Programming, Q-learning, and Deep Q-Network (DQN) baselines using construction operations from the TEACh human-robot collaboration dataset. In the Heavy Excels setting, where robots have strong task specializations, LTAA achieves 77% task completion with superior workload balance, outperforming all traditional methods. These findings show that LLM-based reasoning with structured validation can match established optimization algorithms while offering additional advantages such as interpretability, adaptability, and the ability to update task logic without retraining.
☆ Perception of AI-Generated Music -- The Role of Composer Id entity , Personality Traits, Music Preferences, and Perceived Humanness
David Stammer, Hannah Strauss, Peter Knees
The rapid rise of AI-generated art has sparked debate about potential biases in how audiences perceive and evaluate such works. This study investigates how composer information and listener characteristics shape the perception of AI-generated music, adopting a mixed-method approach. Using a diverse set of stimuli across various genres from two AI music models, we examine effects of perceived authorship on liking and emotional responses, and explore how attitudes toward AI, personality traits, and music-related variables influence evaluations. We further assess the influence of perceived humanness and analyze open-ended responses to uncover listener criteria for judging AI-generated music. Attitudes toward AI proved to be the best predictor of both liking and emotional intensity of AI-generated music. This quantitative finding was complemented by qualitative themes from our thematic analysis, which identified ethical, cultural, and contextual considerations as important criteria in listeners' evaluations of AI-generated music. Our results offer a nuanced view of how people experience music created by AI tools and point to key factors and methodological considerations for future research on music perception in human-AI interaction.
comment : Under review at Computers in Human Behaviour Reports
☆ Survey Eval: Towards Comprehensive Evaluation of LLM-Generated Academic Survey s
Jiahao Zhao, Shuaixing Zhang, Nan Xu, Lei Wang
LLM-based automatic survey systems are transforming how users acquire information from the web by integrating retrieval, organization, and content synthesis into end-to-end generation pipelines. While recent works focus on developing new generation pipelines, how to evaluate such complex systems remains a significant challenge. To this end, we introduce SurveyEval, a comprehensive benchmark that evaluates automatically generated surveys across three dimensions: overall quality, outline coherence, and reference accuracy. We extend the evaluation across 7 subjects and augment the LLM-as-a-Judge framework with human references to strengthen evaluation-human alignment. Evaluation results show that while general long-text or paper-writing systems tend to produce lower-quality surveys, specialized survey-generation systems are able to deliver substantially higher-quality results. We envision SurveyEval as a scalable testbed to understand and improve automatic survey systems across diverse subjects and evaluation criteria.
☆ Reasoning -Aware Multimodal Fusion for Hateful Video Detection
Shuonan Yang, Tailin Chen, Jiangbei Yue, Guangliang Cheng, Jianbo Jiao, Zeyu Fu
Hate speech in online videos is posing an increasingly serious threat to digital platforms, especially as video content becomes increasingly multimodal and context-dependent. Existing methods often struggle to effectively fuse the complex semantic relationships between modalities and lack the ability to understand nuanced hateful content. To address these issues, we propose an innovative Reasoning-Aware Multimodal Fusion (RAMF) framework. To tackle the first challenge, we design Local-Global Context Fusion (LGCF) to capture both local salient cues and global temporal structures, and propose Semantic Cross Attention (SCA) to enable fine-grained multimodal semantic interaction. To tackle the second challenge, we introduce adversarial reasoning-a structured three-stage process where a vision-language model generates (i) objective descriptions, (ii) hate-assumed inferences, and (iii) non-hate-assumed inferences-providing complementary semantic perspectives that enrich the model's contextual understanding of nuanced hateful intent. Evaluations on two real-world hateful video datasets demonstrate that our method achieves robust generalisation performance, improving upon state-of-the-art methods by 3% and 7% in Macro-F1 and hate class recall, respectively. We will release the code after the anonymity period ends.
☆ A Framework for Causal Concept-based Model Explanations
Anna Rodum Bjøru, Jacob Lysnæs-Larsen, Oskar Jørgensen, Inga Strümke, Helge Langseth
This work presents a conceptual framework for causal concept-based post-hoc Explainable Artificial Intelligence (XAI), based on the requirements that explanations for non-interpretable models should be understandable as well as faithful to the model being explained. Local and global explanations are generated by calculating the probability of sufficiency of concept interventions. Example explanations are presented, generated with a proof-of-concept model made to explain classifiers trained on the CelebA dataset. Understandability is demonstrated through a clear concept-based vocabulary, subject to an implicit causal interpretation. Fidelity is addressed by highlighting important framework assumptions, stressing that the context of explanation interpretation must align with the context of explanation generation.
☆ Self-Improving AI Agent s through Self-Play
Przemyslaw Chojecki
We extend the moduli-theoretic framework of psychometric batteries to the domain of dynamical systems. While previous work established the AAI capability score as a static functional on the space of agent representations, this paper formalizes the agent as a flow $ν_r$ parameterized by computational resource $r$, governed by a recursive Generator-Verifier-Updater (GVU) operator. We prove that this operator generates a vector field on the parameter manifold $Θ$, and we identify the coefficient of self-improvement $κ$ as the Lie derivative of the capability functional along this flow. The central contribution of this work is the derivation of the Variance Inequality, a spectral condition that is sufficient (under mild regularity) for the stability of self-improvement. We show that a sufficient condition for $κ> 0$ is that, up to curvature and step-size effects, the combined noise of generation and verification must be small enough. We then apply this formalism to unify the recent literature on Language Self-Play (LSP), Self-Correction, and Synthetic Data bootstrapping. We demonstrate that architectures such as STaR, SPIN, Reflexion, GANs and AlphaZero are specific topological realizations of the GVU operator that satisfy the Variance Inequality through filtration, adversarial discrimination, or grounding in formal systems.
☆ DF-Mamba: Deformable State Space Modeling for 3D Hand Pose Estimation in Interactions WACV 2026
Yifan Zhou, Takehiko Ohkawa, Guwenxiao Zhou, Kanoko Goto, Takumi Hirose, Yusuke Sekikawa, Nakamasa Inoue
Modeling daily hand interactions often struggles with severe occlusions, such as when two hands overlap, which highlights the need for robust feature learning in 3D hand pose estimation (HPE). To handle such occluded hand images, it is vital to effectively learn the relationship between local image features (e.g., for occluded joints) and global context (e.g., cues from inter-joints, inter-hands, or the scene). However, most current 3D HPE methods still rely on ResNet for feature extraction, and such CNN's inductive bias may not be optimal for 3D HPE due to its limited capability to model the global context. To address this limitation, we propose an effective and efficient framework for visual feature extraction in 3D HPE using recent state space modeling (i.e., Mamba), dubbed Deformable Mamba (DF-Mamba). DF-Mamba is designed to capture global context cues beyond standard convolution through Mamba's selective state modeling and the proposed deformable state scanning. Specifically, for local features after convolution, our deformable scanning aggregates these features within an image while selectively preserving useful cues that represent the global context. This approach significantly improves the accuracy of structured 3D HPE, with comparable inference speed to ResNet-50. Our experiments involve extensive evaluations on five divergent datasets including single-hand and two-hand scenarios, hand-only and hand-object interactions, as well as RGB and depth-based estimation. DF-Mamba outperforms the latest image backbones, including VMamba and Spatial-Mamba, on all datasets and achieves state-of-the-art performance.
comment : Accepted to WACV 2026. Project page: https://tkhkaeio.github.io/projects/25-dfmamba/index.html
☆ AuditCopilot: Leveraging LLMs for Fraud Detection in Double-Entry Bookkeeping
Md Abdul Kadir, Sai Suresh Macharla Vasu, Sidharth S. Nair, Daniel Sonntag
Auditors rely on Journal Entry Tests (JETs) to detect anomalies in tax-related ledger records, but rule-based methods generate overwhelming false positives and struggle with subtle irregularities. We investigate whether large language models (LLMs) can serve as anomaly detectors in double-entry bookkeeping. Benchmarking SoTA LLMs such as LLaMA and Gemma on both synthetic and real-world anonymized ledgers, we compare them against JETs and machine learning baselines. Our results show that LLMs consistently outperform traditional rule-based JETs and classical ML baselines, while also providing natural-language explanations that enhance interpretability. These results highlight the potential of \textbf{AI-augmented auditing}, where human auditors collaborate with foundation models to strengthen financial integrity.
☆ StockMem: An Event-Reflection Memory Framework for Stock Forecasting
He Wang, Wenyilin Xiao, Songqiao Han, Hailiang Huang
Stock price prediction is challenging due to market volatility and its sensitivity to real-time events. While large language models (LLMs) offer new avenues for text-based forecasting, their application in finance is hindered by noisy news data and the lack of explicit answers in text. General-purpose memory architectures struggle to identify the key drivers of price movements. To address this, we propose StockMem, an event-reflection dual-layer memory framework. It structures news into events and mines them along two dimensions: horizontal consolidation integrates daily events, while longitudinal tracking captures event evolution to extract incremental information reflecting market expectation discrepancies. This builds a temporal event knowledge base. By analyzing event-price dynamics, the framework further forms a reflection knowledge base of causal experiences. For prediction, it retrieves analogous historical scenarios and reasons with current events, incremental data, and past experiences. Experiments show StockMem outperforms existing memory architectures and provides superior, explainable reasoning by tracing the information chain affecting prices, enhancing decision transparency in financial forecasting.
☆ Emergent Bayesian Behaviour and Optimal Cue Combination in LLMs
Julian Ma, Jun Wang, Zafeirios Fountas
Large language models (LLMs) excel at explicit reasoning, but their implicit computational strategies remain underexplored. Decades of psychophysics research show that humans intuitively process and integrate noisy signals using near-optimal Bayesian strategies in perceptual tasks. We ask whether LLMs exhibit similar behaviour and perform optimal multimodal integration without explicit training or instruction. Adopting the psychophysics paradigm, we infer computational principles of LLMs from systematic behavioural studies. We introduce a behavioural benchmark - BayesBench: four magnitude estimation tasks (length, location, distance, and duration) over text and image, inspired by classic psychophysics, and evaluate a diverse set of nine LLMs alongside human judgments for calibration. Through controlled ablations of noise, context, and instruction prompts, we measure performance, behaviour and efficiency in multimodal cue-combination. Beyond accuracy and efficiency metrics, we introduce a Bayesian Consistency Score that detects Bayes-consistent behavioural shifts even when accuracy saturates. Our results show that while capable models often adapt in Bayes-consistent ways, accuracy does not guarantee robustness. Notably, GPT-5 Mini achieves perfect text accuracy but fails to integrate visual cues efficiently. This reveals a critical dissociation between capability and strategy, suggesting accuracy-centric benchmarks may over-index on performance while missing brittle uncertainty handling. These findings reveal emergent principled handling of uncertainty and highlight the correlation between accuracy and Bayesian tendencies. We release our psychophysics benchmark and consistency metric (https://bayes-bench.github.io) as evaluation tools and to inform future multimodal architecture designs.
☆ Menta: A Small Language Model for On-Device Mental Health Prediction
Tianyi Zhang, Xiangyuan Xue, Lingyan Ruan, Shiya Fu, Feng Xia, Simon D'Alfonso, Vassilis Kostakos, Hong Jia
Mental health conditions affect hundreds of millions globally, yet early detection remains limited. While large language models (LLMs) have shown promise in mental health applications, their size and computational demands hinder practical deployment. Small language models (SLMs) offer a lightweight alternative, but their use for social media--based mental health prediction remains largely underexplored. In this study, we introduce Menta, the first optimized SLM fine-tuned specifically for multi-task mental health prediction from social media data. Menta is jointly trained across six classification tasks using a LoRA-based framework, a cross-dataset strategy, and a balanced accuracy--oriented loss. Evaluated against nine state-of-the-art SLM baselines, Menta achieves an average improvement of 15.2\% across tasks covering depression, stress, and suicidality compared with the best-performing non--fine-tuned SLMs. It also achieves higher accuracy on depression and stress classification tasks compared to 13B-parameter LLMs, while being approximately 3.25x smaller. Moreover, we demonstrate real-time, on-device deployment of Menta on an iPhone 15 Pro Max, requiring only approximately 3GB RAM. Supported by a comprehensive benchmark against existing SLMs and LLMs, Menta highlights the potential for scalable, privacy-preserving mental health monitoring. Code is available at: https://xxue752-nz.github.io/menta-project/
☆ Training Data Attribution for Image Generation using Ontology-Aligned Knowledge Graphs
Theodoros Aivalis, Iraklis A. Klampanos, Antonis Troumpoukis, Joemon M. Jose
As generative models become powerful, concerns around transparency, accountability, and copyright violations have intensified. Understanding how specific training data contributes to a model's output is critical. We introduce a framework for interpreting generative outputs through the automatic construction of ontologyaligned knowledge graphs (KGs). While automatic KG construction from natural text has advanced, extracting structured and ontology-consistent representations from visual content remains challenging -- due to the richness and multi-object nature of images. Leveraging multimodal large language models (LLMs), our method extracts structured triples from images, aligned with a domain-specific ontology. By comparing the KGs of generated and training images, we can trace potential influences, enabling copyright analysis, dataset transparency, and interpretable AI. We validate our method through experiments on locally trained models via unlearning, and on large-scale models through a style-specific experiment. Our framework supports the development of AI systems that foster human collaboration, creativity and stimulate curiosity.
☆ Empirical Assessment of the Perception of Software Product Line Engineering by an SME before Migrating its Code Base
Thomas Georges, Marianne Huchard, Mélanie König, Clémentine Nebut, Chouki Tibermacine
Migrating a set of software variants into a software product line (SPL) is an expensive and potentially challenging endeavor. Indeed, SPL engineering can significantly impact a company's development process and often requires changes to established developer practices. The work presented in this paper stems from a collaboration with a Small and Medium-sized Enterprise (SME) that decided to migrate its existing code base into an SPL. In this study, we conducted an in-depth evaluation of the company's current development processes and practices, as well as the anticipated benefits and risks associated with the migration. Key stakeholders involved in software development participated in this evaluation to provide insight into their perceptions of the migration and their potential resistance to change. This paper describes the design of the interviews conducted with these stakeholders and presents an analysis of the results. Among the qualitative findings, we observed that all participants, regardless of their role in the development process, identified benefits of the migration relevant to their own activities. Furthermore, our results suggest that an effective risk mitigation strategy involves keeping stakeholders informed and engaged throughout the process, preserving as many good practices as possible, and actively involving them in the migration to ensure a smooth transition and minimize potential challenges.
comment : 34 pages
☆ Learning What to Attend First: Modality-Importance-Guided Reasoning for Reliable Multimodal Emotion Understanding
Hyeongseop Rha, Jeong Hun Yeo, Junil Won, Se Jin Park, Yong Man Ro
In this paper, we present Modality-Importance-Guided Reasoning (MIGR), a framework designed to improve the reliability of reasoning-based multimodal emotion understanding in multimodal large language models. Although existing methods have advanced emotion understanding, they often suffer from reasoning drift: models gradually rely on their own generated text instead of multimodal evidence, and their explanations are overly shaped by visually initiated reasoning paths. To address these issues, we introduce Modality Importance (MI), a simple yet effective mechanism for identifying the emotion-dominant modality. Using MI, MIGR reorganizes reasoning sequences so that explanations begin from the modality most critical to the target emotion, preventing early reasoning from being misled by less informative cues. Our two-stage framework-comprising modality-aligned supervised fine-tuning and modality-aware reward optimization-encourages models to generate emotionally grounded, causally relevant, and coherence-preserving explanations. Experimental results on the DFEW benchmark show that MIGR substantially improves reasoning reliability, decreasing instances of correct predictions accompanied by emotionally inconsistent explanations from 18.10% to 7.37%. These results confirm the benefit of initiating reasoning from the emotion-dominant modality.
comment : 16 pages, 8 figures
☆ An Empirical Survey of Model Merging Algorithms for Social Bias Mitigation ACL
Daiki Shirafuji, Tatsuhiko Saito, Yasutomo Kimura
Large language models (LLMs) are known to inherit and even amplify societal biases present in their pre-training corpora, threatening fairness and social trust. To address this issue, recent work has explored ``editing'' LLM parameters to mitigate social bias with model merging approaches; however, there is no empirical comparison. In this work, we empirically survey seven algorithms: Linear, Karcher Mean, SLERP, NuSLERP, TIES, DELLA, and Nearswap, applying 13 open weight models in the GPT, LLaMA, and Qwen families. We perform a comprehensive evaluation using three bias datasets (BBQ, BOLD, and HONEST) and measure the impact of these techniques on LLM performance in downstream tasks of the SuperGLUE benchmark. We find a trade-off between bias reduction and downstream performance: methods achieving greater bias mitigation degrade accuracy, particularly on tasks requiring reading comprehension and commonsense and causal reasoning. Among the merging algorithms, Linear, SLERP, and Nearswap consistently reduce bias while maintaining overall performance, with SLERP at moderate interpolation weights emerging as the most balanced choice. These results highlight the potential of model merging algorithms for bias mitigation, while indicating that excessive debiasing or inappropriate merging methods may lead to the degradation of important linguistic abilities.
comment : Accepted in PACLIC 2025
☆ Beyond Single- Agent Safety: A Taxonomy of Risks in LLM-to-LLM Interactions
Piercosma Bisconti, Marcello Galisai, Federico Pierucci, Marcantonio Bracale, Matteo Prandi
This paper examines why safety mechanisms designed for human-model interaction do not scale to environments where large language models (LLMs) interact with each other. Most current governance practices still rely on single-agent safety containment, prompts, fine-tuning, and moderation layers that constrain individual model behavior but leave the dynamics of multi-model interaction ungoverned. These mechanisms assume a dyadic setting: one model responding to one user under stable oversight. Yet research and industrial development are rapidly shifting toward LLM-to-LLM ecosystems, where outputs are recursively reused as inputs across chains of agents. In such systems, local compliance can aggregate into collective failure even when every model is individually aligned. We propose a conceptual transition from model-level safety to system-level safety, introducing the framework of the Emergent Systemic Risk Horizon (ESRH) to formalize how instability arises from interaction structure rather than from isolated misbehavior. The paper contributes (i) a theoretical account of collective risk in interacting LLMs, (ii) a taxonomy connecting micro, meso, and macro-level failure modes, and (iii) a design proposal for InstitutionalAI, an architecture for embedding adaptive oversight within multi-agent systems.
☆ Exploring Depth Generalization in Large Language Models for Solving Recursive Logic Tasks
Zhiyuan He
Large language models have demonstrated remarkable capabilities across many tasks, yet face significant challenges when dealing with recursive reasoning problems, those requiring the resolution of nested hierarchical structures. While prior research has extensively studied length generalization (a model's ability to handle longer sequences than seen during training), we investigate a distinct and underexplored limitation: depth generalization. Here, depth refers to the number of nested levels in a hierarchical problem, such as the layers of parentheses in a mathematical expression or the nesting of logical clauses in a Boolean formula. Our work reveals that standard transformer architectures struggle with problems involving deeper recursion than encountered during training, even when they perform well on longer but non-nested sequences. This limitation stems from their inability to maintain stack-like behavior, the capacity to track and resolve multiple levels of nested dependencies. Through systematic analysis, we demonstrate how this architectural constraint leads to rapid performance decay as the depth of the recursion increases. To address this challenge, we develop a novel looped locate-and-replace pipeline that decomposes recursive problems into manageable subcomponents. The approach employs two specialized models: a locator that identifies solvable subexpressions and a replacer that evaluates these components while preserving the overall structure. We evaluated this method in three carefully designed domains: Boolean algebra, recursive arithmetic, and propositional logic, each with a controllable depth of recursion. We show that our method effectively alleviates the performance decay when tested on out-of-distribution recursion depth.
☆ SAND Challenge: Four Approaches for Dysartria Severity Classification
Gauri Deshpande, Harish Battula, Ashish Panda, Sunil Kumar Kopparapu
This paper presents a unified study of four distinct modeling approaches for classifying dysarthria severity in the Speech Analysis for Neurodegenerative Diseases (SAND) challenge. All models tackle the same five class classification task using a common dataset of speech recordings. We investigate: (1) a ViT-OF method leveraging a Vision Transformer on spectrogram images, (2) a 1D-CNN approach using eight 1-D CNN's with majority-vote fusion, (3) a BiLSTM-OF approach using nine BiLSTM models with majority vote fusion, and (4) a Hierarchical XGBoost ensemble that combines glottal and formant features through a two stage learning framework. Each method is described, and their performances on a validation set of 53 speakers are compared. Results show that while the feature-engineered XGBoost ensemble achieves the highest macro-F1 (0.86), the deep learning models (ViT, CNN, BiLSTM) attain competitive F1-scores (0.70) and offer complementary insights into the problem.
comment : 7 pages, 5 figures
☆ Graph VQ- Transformer (GVT): Fast and Accurate Molecular Generation via High-Fidelity Discrete Latents
Haozhuo Zheng, Cheng Wang, Yang Liu
The de novo generation of molecules with desirable properties is a critical challenge, where diffusion models are computationally intensive and autoregressive models struggle with error propagation. In this work, we introduce the Graph VQ-Transformer (GVT), a two-stage generative framework that achieves both high accuracy and efficiency. The core of our approach is a novel Graph Vector Quantized Variational Autoencoder (VQ-VAE) that compresses molecular graphs into high-fidelity discrete latent sequences. By synergistically combining a Graph Transformer with canonical Reverse Cuthill-McKee (RCM) node ordering and Rotary Positional Embeddings (RoPE), our VQ-VAE achieves near-perfect reconstruction rates. An autoregressive Transformer is then trained on these discrete latents, effectively converting graph generation into a well-structured sequence modeling problem. Crucially, this mapping of complex graphs to high-fidelity discrete sequences bridges molecular design with the powerful paradigm of large-scale sequence modeling, unlocking potential synergies with Large Language Models (LLMs). Extensive experiments show that GVT achieves state-of-the-art or highly competitive performance across major benchmarks like ZINC250k, MOSES, and GuacaMol, and notably outperforms leading diffusion models on key distribution similarity metrics such as FCD and KL Divergence. With its superior performance, efficiency, and architectural novelty, GVT not only presents a compelling alternative to diffusion models but also establishes a strong new baseline for the field, paving the way for future research in discrete latent-space molecular generation.
☆ Distill, Forget, Repeat: A Framework for Continual Unlearning in Text-to-Image Diffusion Models
Naveen George, Naoki Murata, Yuhta Takida, Konda Reddy Mopuri, Yuki Mitsufuji
The recent rapid growth of visual generative models trained on vast web-scale datasets has created significant tension with data privacy regulations and copyright laws, such as GDPR's ``Right to be Forgotten.'' This necessitates machine unlearning (MU) to remove specific concepts without the prohibitive cost of retraining. However, existing MU techniques are fundamentally ill-equipped for real-world scenarios where deletion requests arrive sequentially, a setting known as continual unlearning (CUL). Naively applying one-shot methods in a continual setting triggers a stability crisis, leading to a cascade of degradation characterized by retention collapse, compounding collateral damage to related concepts, and a sharp decline in generative quality. To address this critical challenge, we introduce a novel generative distillation based continual unlearning framework that ensures targeted and stable unlearning under sequences of deletion requests. By reframing each unlearning step as a multi-objective, teacher-student distillation process, the framework leverages principles from continual learning to maintain model integrity. Experiments on a 10-step sequential benchmark demonstrate that our method unlearns forget concepts with better fidelity and achieves this without significant interference to the performance on retain concepts or the overall image quality, substantially outperforming baselines. This framework provides a viable pathway for the responsible deployment and maintenance of large-scale generative models, enabling industries to comply with ongoing data removal requests in a practical and effective manner.
comment : Preprint
☆ Pianist Transformer : Towards Expressive Piano Performance Rendering via Scalable Self-Supervised Pre-Train ing
Hong-Jie You, Jie-Jing Shao, Xiao-Wen Yang, Lin-Han Jia, Lan-Zhe Guo, Yu-Feng Li
Existing methods for expressive music performance rendering rely on supervised learning over small labeled datasets, which limits scaling of both data volume and model size, despite the availability of vast unlabeled music, as in vision and language. To address this gap, we introduce Pianist Transformer, with four key contributions: 1) a unified Musical Instrument Digital Interface (MIDI) data representation for learning the shared principles of musical structure and expression without explicit annotation; 2) an efficient asymmetric architecture, enabling longer contexts and faster inference without sacrificing rendering quality; 3) a self-supervised pre-training pipeline with 10B tokens and 135M-parameter model, unlocking data and model scaling advantages for expressive performance rendering; 4) a state-of-the-art performance model, which achieves strong objective metrics and human-level subjective ratings. Overall, Pianist Transformer establishes a scalable path toward human-like performance synthesis in the music domain.
☆ Zero-Shot Instruction Following in RL via Structured LTL Representations ICML 2025
Mattia Giuri, Mathias Jackermeier, Alessandro Abate
Linear temporal logic (LTL) is a compelling framework for specifying complex, structured tasks for reinforcement learning (RL) agents. Recent work has shown that interpreting LTL instructions as finite automata, which can be seen as high-level programs monitoring task progress, enables learning a single generalist policy capable of executing arbitrary instructions at test time. However, existing approaches fall short in environments where multiple high-level events (i.e., atomic propositions) can be true at the same time and potentially interact in complicated ways. In this work, we propose a novel approach to learning a multi-task policy for following arbitrary LTL instructions that addresses this shortcoming. Our method conditions the policy on sequences of simple Boolean formulae, which directly align with transitions in the automaton, and are encoded via a graph neural network (GNN) to yield structured task representations. Experiments in a complex chess-based environment demonstrate the advantages of our approach.
comment : ICML 2025 Workshop on Programmatic Representations for Agent Learning
☆ CryptoQA: A Large-scale Question-answering Dataset for AI-assisted Cryptography
Mayar Elfares, Pascal Reisert, Tilman Dietz, Manpa Barman, Ahmed Zaki, Ralf Küsters, Andreas Bulling
Large language models (LLMs) excel at many general-purpose natural language processing tasks. However, their ability to perform deep reasoning and mathematical analysis, particularly for complex tasks as required in cryptography, remains poorly understood, largely due to the lack of suitable data for evaluation and training. To address this gap, we present CryptoQA, the first large-scale question-answering (QA) dataset specifically designed for cryptography. CryptoQA contains over two million QA pairs drawn from curated academic sources, along with contextual metadata that can be used to test the cryptographic capabilities of LLMs and to train new LLMs on cryptographic tasks. We benchmark 15 state-of-the-art LLMs on CryptoQA, evaluating their factual accuracy, mathematical reasoning, consistency, referencing, backward reasoning, and robustness to adversarial samples. In addition to quantitative metrics, we provide expert reviews that qualitatively assess model outputs and establish a gold-standard baseline. Our results reveal significant performance deficits of LLMs, particularly on tasks that require formal reasoning and precise mathematical knowledge. This shows the urgent need for LLM assistants tailored to cryptography research and development. We demonstrate that, by using CryptoQA, LLMs can be fine-tuned to exhibit better performance on cryptographic tasks.
☆ Target-specific Adaptation and Consistent Degradation Alignment for Cross-Domain Remaining Useful Life Prediction
Yubo Hou, Mohamed Ragab, Min Wu, Chee-Keong Kwoh, Xiaoli Li, Zhenghua Chen
Accurate prediction of the Remaining Useful Life (RUL) in machinery can significantly diminish maintenance costs, enhance equipment up-time, and mitigate adverse outcomes. Data-driven RUL prediction techniques have demonstrated commendable performance. However, their efficacy often relies on the assumption that training and testing data are drawn from the same distribution or domain, which does not hold in real industrial settings. To mitigate this domain discrepancy issue, prior adversarial domain adaptation methods focused on deriving domain-invariant features. Nevertheless, they overlook target-specific information and inconsistency characteristics pertinent to the degradation stages, resulting in suboptimal performance. To tackle these issues, we propose a novel domain adaptation approach for cross-domain RUL prediction named TACDA. Specifically, we propose a target domain reconstruction strategy within the adversarial adaptation process, thereby retaining target-specific information while learning domain-invariant features. Furthermore, we develop a novel clustering and pairing strategy for consistent alignment between similar degradation stages. Through extensive experiments, our results demonstrate the remarkable performance of our proposed TACDA method, surpassing state-of-the-art approaches with regard to two different evaluation metrics. Our code is available at https://github.com/keyplay/TACDA.
☆ IACT: A Self-Organizing Recursive Model for General AI Agent s: A Technical White Paper on the Architecture Behind kr agent .ai
Pengju Lu
This technical white paper introduces the Interactive Agents Call Tree (IACT), a computational model designed to address the limitations of static, hard-coded agent workflows. Unlike traditional systems that require pre-defined graphs or specialized programming, IACT operates as a general-purpose autonomous system driven purely by user dialogue. Given a high-level objective, the system autonomously grows a dynamic, recursive agent topology incrementally tailored to the problem's structure. This allows it to scale its organizational complexity to match open-ended tasks. To mitigate the error propagation inherent in unidirectional function calls, IACT introduces interactional redundancy by replacing rigid invocations with bidirectional, stateful dialogues. This mechanism enables runtime error correction and ambiguity resolution. We describe the architecture, design principles, and practical lessons behind the production deployment of this model in the kragent.ai system, presenting qualitative evidence from real-world workflows rather than exhaustive benchmark results.
comment : 13 pages, 2 figures, 1 table
☆ PaperDebugger: A Plugin-Based Multi- Agent System for In-Editor Academic Writing, Review , and Editing
Junyi Hou, Andre Lin Huikai, Nuo Chen, Yiwei Gong, Bingsheng He
Large language models are increasingly embedded into academic writing workflows, yet existing assistants remain external to the editor, preventing deep interaction with document state, structure, and revision history. This separation makes it impossible to support agentic, context-aware operations directly within LaTeX editors such as Overleaf. We present PaperDebugger, an in-editor, multi-agent, and plugin-based academic writing assistant that brings LLM-driven reasoning directly into the writing environment. Enabling such in-editor interaction is technically non-trivial: it requires reliable bidirectional synchronization with the editor, fine-grained version control and patching, secure state management, multi-agent scheduling, and extensible communication with external tools. PaperDebugger addresses these challenges through a Chrome-approved extension, a Kubernetes-native orchestration layer, and a Model Context Protocol (MCP) toolchain that integrates literature search, reference lookup, document scoring, and revision pipelines. Our demo showcases a fully integrated workflow, including localized edits, structured reviews, parallel agent execution, and diff-based updates, encapsulated within a minimal-intrusion user interface (UI). Early aggregated analytics demonstrate active user engagement and validate the practicality of an editor-native, agentic writing assistant. More details about this demo and video could be found at https://github.com/PaperDebugger/PaperDebugger.
☆ Feedback Loops and Code Perturbations in LLM-based Software Engineering: A Case Study on a C-to-Rust Translation System
Martin Weiss, Jesko Hecking-Harbusch, Jochen Quante, Matthias Woehrle
The advent of strong generative AI has a considerable impact on various software engineering tasks such as code repair, test generation, or language translation. While tools like GitHub Copilot are already in widespread use in interactive settings, automated approaches require a higher level of reliability before being usable in industrial practice. In this paper, we focus on three aspects that directly influence the quality of the results: a) the effect of automated feedback loops, b) the choice of Large Language Model (LLM), and c) the influence of behavior-preserving code changes. We study the effect of these three variables on an automated C-to-Rust translation system. Code translation from C to Rust is an attractive use case in industry due to Rust's safety guarantees. The translation system is based on a generate-and-check pattern, in which Rust code generated by the LLM is automatically checked for compilability and behavioral equivalence with the original C code. For negative checking results, the LLM is re-prompted in a feedback loop to repair its output. These checks also allow us to evaluate and compare the respective success rates of the translation system when varying the three variables. Our results show that without feedback loops LLM selection has a large effect on translation success. However, when the translation system uses feedback loops the differences across models diminish. We observe this for the average performance of the system as well as its robustness under code perturbations. Finally, we also identify that diversity provided by code perturbations can even result in improved system performance.
comment : 10 pages, 9 figures
☆ From Panel to Pixel: Zoom-In Vision -Language Pretrain ing from Biomedical Scientific Literature
Kun Yuan, Min Woo Sun, Zhen Chen, Alejandro Lozano, Xiangteng He, Shi Li, Nassir Navab, Xiaoxiao Sun, Nicolas Padoy, Serena Yeung-Levy
There is a growing interest in developing strong biomedical vision-language models. A popular approach to achieve robust representations is to use web-scale scientific data. However, current biomedical vision-language pretraining typically compresses rich scientific figures and text into coarse figure-level pairs, discarding the fine-grained correspondences that clinicians actually rely on when zooming into local structures. To tackle this issue, we introduce Panel2Patch, a novel data pipeline that mines hierarchical structure from existing biomedical scientific literature, i.e., multi-panel, marker-heavy figures and their surrounding text, and converts them into multi-granular supervision. Given scientific figures and captions, Panel2Patch parses layouts, panels, and visual markers, then constructs hierarchical aligned vision-language pairs at the figure, panel, and patch levels, preserving local semantics instead of treating each figure as a single data sample. Built on this hierarchical corpus, we develop a granularity-aware pretraining strategy that unifies heterogeneous objectives from coarse didactic descriptions to fine region-focused phrases. By applying Panel2Patch to only a small set of the literature figures, we extract far more effective supervision than prior pipelines, enabling substantially better performance with less pretraining data.
☆ EZYer: A simulacrum of high school with generative agent SIGIR 2025
Jinming Yang, Zimu Ji, Weiqi Luo, Gaoxi Wang, Bin Ma, Yueling Deng
With the rapid development of the online education and large language model, the existing educational tools still suffer from incomplete service, insufficient performance and weak interactivity in terms of courseware generation, interactive notes and quality assurance of content. In particular, the proposed generative agent EZYer : 1) Teacher Module: Integrating the Text Corpus retrieval and in-depth generation technologies, it automatically generates structured teaching materials and LaTeX Beamer courseware in line with the high school mathematics syllabus and supports user-defined image insertion. 2) Student Module: Throughout the collaborative interaction of the four roles of Teacher, Assistant, Top Student and Struggling Student, Note Taker summarizes and generates academic notes to enhance the depth and interest of learning. 3) Controller: set up keyword filtering system, content scoring system, role co-validation system, and dynamic content correction system. This ensure academic strictness and pedagogical propriety of EZYer inputs and outputs. In order to evaluate EZYer, this paper designs five-dimensional evaluation indexes of content accuracy, knowledge coverage, usability, formatting correctness and visual design and appeal, and scores 100 Beamer and Notes generated by EZYer by five large language models, separately, and the results show that the quality of EZYer-generated content is excellent and has a good application prospect.
comment : AgentIR@SIGIR 2025
☆ Empathy Level Prediction in Multi-Modal Scenario with Supervisory Document ation Assistance
Yufei Xiao, Shangfei Wang
Prevalent empathy prediction techniques primarily concentrate on a singular modality, typically textual, thus neglecting multi-modal processing capabilities. They also overlook the utilization of certain privileged information, which may encompass additional empathetic content. In response, we introduce an advanced multi-modal empathy prediction method integrating video, audio, and text information. The method comprises the Multi-Modal Empathy Prediction and Supervisory Documentation Assisted Training. We use pre-trained networks in the empathy prediction network to extract features from various modalities, followed by a cross-modal fusion. This process yields a multi-modal feature representation, which is employed to predict empathy labels. To enhance the extraction of text features, we incorporate supervisory documents as privileged information during the assisted training phase. Specifically, we apply the Latent Dirichlet Allocation model to identify potential topic distributions to constrain text features. These supervisory documents, created by supervisors, focus on the counseling topics and the counselor's display of empathy. Notably, this privileged information is only available during training and is not accessible during the prediction phase. Experimental results on the multi-modal and dialogue empathy datasets demonstrate that our approach is superior to the existing methods.
☆ ADORE: Autonomous Domain-Oriented Relevance Engine for E-commerce SIGIR 2025
Zheng Fang, Donghao Xie, Ming Pang, Chunyuan Yuan, Xue Jiang, Changping Peng, Zhangang Lin, Zheng Luo
Relevance modeling in e-commerce search remains challenged by semantic gaps in term-matching methods (e.g., BM25) and neural models' reliance on the scarcity of domain-specific hard samples. We propose ADORE, a self-sustaining framework that synergizes three innovations: (1) A Rule-aware Relevance Discrimination module, where a Chain-of-Thought LLM generates intent-aligned training data, refined via Kahneman-Tversky Optimization (KTO) to align with user behavior; (2) An Error-type-aware Data Synthesis module that auto-generates adversarial examples to harden robustness; and (3) A Key-attribute-enhanced Knowledge Distillation module that injects domain-specific attribute hierarchies into a deployable student model. ADORE automates annotation, adversarial generation, and distillation, overcoming data scarcity while enhancing reasoning. Large-scale experiments and online A/B testing verify the effectiveness of ADORE. The framework establishes a new paradigm for resource-efficient, cognitively aligned relevance modeling in industrial applications.
comment : Accepted by SIGIR 2025
☆ CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning
Songqiao Su, Xiaofei Sun, Xiaoya Li, Albert Wang, Jiwei Li, Chris Shum
In this paper, we propose CUDA-L2, a system that combines large language models (LLMs) and reinforcement learning (RL) to automatically optimize Half-precision General Matrix Multiply (HGEMM) CUDA kernels. Using CUDA execution speed as the RL reward, CUDA-L2 automatically optimizes HGEMM kernels across 1,000 configurations. CUDA-L2 systematically outperforms major matmul baselines to date, from the widely-used {\it torch.matmul} to state-of-the-art Nvidia's closed-source libraries, i.e., {\it cuBLAS}, {\it cuBLASLt}. In offline mode, where kernels are executed consecutively without time intervals, CUDA-L2 yields +22.0\% over {\it torch.matmul} on average; +19.2\% over {\it cuBLAS} using the optimal layout configuration (normal-normal NN and transposed-normal TN); +16.8\% over {\it cuBLASLt-heuristic}, which queries {\it cuBLASLt} library and selects the algorithm based on the heuristic's suggestion; and +11.4\% over the most competitive {\it cuBLASLt-AutoTuning} model, which selects the fastest algorithm from up to 100 candidates from {\it cuBLASLt}'s suggestions. In server mode, where kernels are executed at random intervals simulating real-time inference, the speedups further increase to +28.7\%, +26.0\%, +22.4\%, and +15.9\% for {\it torch.matmul}, {\it cuBLAS}, {\it cuBLASLt-heuristic}, and {\it cuBLASLt-AutoTuning} respectively. CUDA-L2 shows that even the most performance-critical, heavily-optimized kernels like HGEMM can be improved through LLM-guided RL automation by systematically exploring configuration spaces at scales impractical for humans. Project and code can be found at github.com/deepreinforce-ai/CUDA-L2
☆ Sparse Computations in Deep Learning Inference
Ioanna Tasou, Panagiotis Mpakos, Angelos Vlachos, Dionysios Adamopoulos, Georgios Giannakopoulos, Konstantinos Katsikopoulos, Ioannis Karaparisis, Maria Lazou, Spyridon Loukovitis, Areti Mei, Anastasia Poulopoulou, Angeliki Dimitriou, Giorgos Filandrianos, Dimitrios Galanopoulos, Vasileios Karampinis, Ilias Mitsouras, Nikolaos Spanos, Petros Anastasiadis, Ioannis Doudalis, Konstantinos Nikas, George Retsinas, Paraskevi Tzouveli, Christina Giannoula, Nectarios Koziris, Nikela Papadopoulou, Giorgos Stamou, Athanasios Voulodimos, Georgios Goumas
The computational demands of modern Deep Neural Networks (DNNs) are immense and constantly growing. While training costs usually capture public attention, inference demands are also contributing in significant computational, energy and environmental footprints. Sparsity stands out as a critical mechanism for drastically reducing these resource demands. However, its potential remains largely untapped and is not yet fully incorporated in production AI systems. To bridge this gap, this work provides the necessary knowledge and insights for performance engineers keen to get involved in deep learning inference optimization. In particular, in this work we: a) discuss the various forms of sparsity that can be utilized in DNN inference, b) explain how the original dense computations translate to sparse kernels, c) provide an extensive bibliographic review of the state-of-the-art in the implementation of these kernels for CPUs and GPUs, d) discuss the availability of sparse datasets in support of sparsity-related research and development, e) explore the current software tools and frameworks that provide robust sparsity support, and f) present evaluation results of different implementations of the key SpMM and SDDMM kernels on CPU and GPU platforms. Ultimately, this paper aims to serve as a resource for performance engineers seeking to develop and deploy highly efficient sparse deep learning models in productions.
☆ Aetheria: A multimodal interpretable content safety framework based on multi- agent debate and collaboration
Yuxiang He, Jian Zhao, Yuchen Yuan, Tianle Zhang, Wei Cai, Haojie Cheng, Ziyan Shi, Ming Zhu, Haichuan Tang, Chi Zhang, Xuelong Li
The exponential growth of digital content presents significant challenges for content safety. Current moderation systems, often based on single models or fixed pipelines, exhibit limitations in identifying implicit risks and providing interpretable judgment processes. To address these issues, we propose Aetheria, a multimodal interpretable content safety framework based on multi-agent debate and collaboration.Employing a collaborative architecture of five core agents, Aetheria conducts in-depth analysis and adjudication of multimodal content through a dynamic, mutually persuasive debate mechanism, which is grounded by RAG-based knowledge retrieval.Comprehensive experiments on our proposed benchmark (AIR-Bench) validate that Aetheria not only generates detailed and traceable audit reports but also demonstrates significant advantages over baselines in overall content safety accuracy, especially in the identification of implicit risks. This framework establishes a transparent and interpretable paradigm, significantly advancing the field of trustworthy AI content moderation.
comment : https://github.com/Herrieson/Aetheria
☆ AskNearby: An LLM-Based Application for Neighborhood Information Retrieval and Personalized Cognitive-Map Recommendations
Luyao Niu, Zhicheng Deng, Boyang Li, Nuoxian Huang, Ruiqi Liu, Wenjia Zhang
The "15-minute city" envisions neighborhoods where residents can meet daily needs via a short walk or bike ride. Realizing this vision requires not only physical proximity but also efficient and reliable access to information about nearby places, services, and events. Existing location-based systems, however, focus mainly on city-level tasks and neglect the spatial, temporal, and cognitive factors that shape localized decision-making. We conceptualize this gap as the Local Life Information Accessibility (LLIA) problem and introduce AskNearby, an AI-driven community application that unifies retrieval and recommendation within the 15-minute life circle. AskNearby integrates (i) a three-layer Retrieval-Augmented Generation (RAG) pipeline that synergizes graph-based, semantic-vector, and geographic retrieval with (ii) a cognitive-map model that encodes each user's neighborhood familiarity and preferences. Experiments on real-world community datasets demonstrate that AskNearby significantly outperforms LLM-based and map-based baselines in retrieval accuracy and recommendation quality, achieving robust performance in spatiotemporal grounding and cognitive-aware ranking. Real-world deployments further validate its effectiveness. By addressing the LLIA challenge, AskNearby empowers residents to more effectively discover local resources, plan daily activities, and engage in community life.
☆ COPE: Chain-Of-Thought Prediction Engine for Open-Source Large Language Model Based Stroke Outcome Prediction from Clinical Notes
Yongkai Liu, Helena Feng, Bin Jiang, Yixin Wang, Max Wintermark, David S. Liebeskind, Michael Moseley, Maarten Lansberg, Gregory Albers, Jeremy Heit, Greg Zaharchuk
Predicting outcomes in acute ischemic stroke (AIS) guides clinical decision-making, patient counseling, and resource allocation. Clinical notes contain rich contextual information, but their unstructured nature limits their use in traditional predictive models. We developed and evaluated the Chain-of-Thought (CoT) Outcome Prediction Engine (COPE), a reasoning-enhanced large language model framework, for predicting 90-day functional outcomes after AIS from unstructured clinical notes. This study included 464 AIS patients with discharge summaries and 90-day modified Rankin Scale (mRS) scores. COPE uses a two-step CoT framework based on sequential open-source LLaMA-3-8B models: the first generates clinical reasoning, and the second outputs an mRS prediction. We compared COPE with GPT-4.1, ClinicalBERT, a structured variable-based machine learning model (Clinical ML), and a single-step LLM without CoT. Performance was evaluated using mean absolute error (MAE), accuracy within +/-1 mRS point, and exact accuracy. COPE achieved an MAE of 1.01 (95% CI 0.92-1.11), +/-1 accuracy of 74.4% (69.9, 78.8%), and exact accuracy of 32.8% (28.0, 37.6%), comparable to GPT-4.1 and superior to ClinicalBERT [MAE 1.24 (1.13-1.36)], Clinical ML [1.28 (1.18-1.39)], and the single-step LLM [1.20 (1.09-1.33)]. Subgroup analyses showed consistent performance across sex and age, with slightly higher error among older patients, those undergoing thrombectomy, and those with longer summaries. These findings demonstrate that COPE, a lightweight, interpretable, and privacy-preserving open-source framework, provides an accurate and practical solution for outcome prediction from unstructured clinical text.
☆ Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs for 3D Scene-Language Understanding
Yerim Jeon, Miso Lee, WonJun Moon, Jae-Pil Heo
Recent advances in 3D scene-language understanding have leveraged Large Language Models (LLMs) for 3D reasoning by transferring their general reasoning ability to 3D multi-modal contexts. However, existing methods typically adopt standard decoders from language modeling, which rely on a causal attention mask. This design introduces two fundamental conflicts in 3D scene understanding: sequential bias among order-agnostic 3D objects and restricted object-instruction attention, hindering task-specific reasoning. To overcome these limitations, we propose 3D Spatial Language Instruction Mask (3D-SLIM), an effective masking strategy that replaces the causal mask with an adaptive attention mask tailored to the spatial structure of 3D scenes. Our 3D-SLIM introduces two key components: a Geometry-adaptive Mask that constrains attention based on spatial density rather than token order, and an Instruction-aware Mask that enables object tokens to directly access instruction context. This design allows the model to process objects based on their spatial relationships while being guided by the user's task. 3D-SLIM is simple, requires no architectural modifications, and adds no extra parameters, yet it yields substantial performance improvements across diverse 3D scene-language tasks. Extensive experiments across multiple benchmarks and LLM baselines validate its effectiveness and underscore the critical role of decoder design in 3D multi-modal reasoning.
☆ UC Agent s: Unidirectional Convergence for Visual Evidence Anchored Multi- Agent Medical Decision-Making
Qianhan Feng, Zhongzhen Huang, Yakun Zhu, Xiaofan Zhang, Qi Dou
Vision-Language Models (VLMs) show promise in medical diagnosis, yet suffer from reasoning detachment, where linguistically fluent explanations drift from verifiable image evidence, undermining clinical trust. Recent multi-agent frameworks simulate Multidisciplinary Team (MDT) debates to mitigate single-model bias, but open-ended discussions amplify textual noise and computational cost while failing to anchor reasoning to visual evidence, the cornerstone of medical decision-making. We propose UCAgents, a hierarchical multi-agent framework enforcing unidirectional convergence through structured evidence auditing. Inspired by clinical workflows, UCAgents forbids position changes and limits agent interactions to targeted evidence verification, suppressing rhetorical drift while amplifying visual signal extraction. In UCAgents, a one-round inquiry discussion is introduced to uncover potential risks of visual-textual misalignment. This design jointly constrains visual ambiguity and textual noise, a dual-noise bottleneck that we formalize via information theory. Extensive experiments on four medical VQA benchmarks show UCAgents achieves superior accuracy (71.3% on PathVQA, +6.0% over state-of-the-art) with 87.7% lower token cost, the evaluation results further confirm that UCAgents strikes a balance between uncovering more visual evidence and avoiding confusing textual interference. These results demonstrate that UCAgents exhibits both diagnostic reliability and computational efficiency critical for real-world clinical deployment. Code is available at https://github.com/fqhank/UCAgents.
☆ Q-BERT4Rec: Quantized Semantic-ID Representation Learning for Multimodal Recommendation KDD2026
Haofeng Huang, Ling Gai
Sequential recommendation plays a critical role in modern online platforms such as e-commerce, advertising, and content streaming, where accurately predicting users' next interactions is essential for personalization. Recent Transformer-based methods like BERT4Rec have shown strong modeling capability, yet they still rely on discrete item IDs that lack semantic meaning and ignore rich multimodal information (e.g., text and image). This leads to weak generalization and limited interpretability. To address these challenges, we propose Q-Bert4Rec, a multimodal sequential recommendation framework that unifies semantic representation and quantized modeling. Specifically, Q-Bert4Rec consists of three stages: (1) cross-modal semantic injection, which enriches randomly initialized ID embeddings through a dynamic transformer that fuses textual, visual, and structural features; (2) semantic quantization, which discretizes fused representations into meaningful tokens via residual vector quantization; and (3) multi-mask pretraining and fine-tuning, which leverage diverse masking strategies -- span, tail, and multi-region -- to improve sequential understanding. We validate our model on public Amazon benchmarks and demonstrate that Q-Bert4Rec significantly outperforms many strong existing methods, confirming the effectiveness of semantic tokenization for multimodal sequential recommendation. Our source code will be publicly available on GitHub after publishing.
comment : Submitted to KDD2026
☆ Guided Self-Evolving LLMs with Minimal Human Super vision
Wenhao Yu, Zhenwen Liang, Chengsong Huang, Kishan Panaganti, Tianqing Fang, Haitao Mi, Dong Yu
AI self-evolution has long been envisioned as a path toward superintelligence, where models autonomously acquire, refine, and internalize knowledge from their own learning experiences. Yet in practice, unguided self-evolving systems often plateau quickly or even degrade as training progresses. These failures arise from issues such as concept drift, diversity collapse, and mis-evolution, as models reinforce their own biases and converge toward low-entropy behaviors. To enable models to self-evolve in a stable and controllable manner while minimizing reliance on human supervision, we introduce R-Few, a guided Self-Play Challenger-Solver framework that incorporates lightweight human oversight through in-context grounding and mixed training. At each iteration, the Challenger samples a small set of human-labeled examples to guide synthetic question generation, while the Solver jointly trains on human and synthetic examples under an online, difficulty-based curriculum. Across math and general reasoning benchmarks, R-Few achieves consistent and iterative improvements. For example, Qwen3-8B-Base improves by +3.0 points over R-Zero on math tasks and achieves performance on par with General-Reasoner, despite the latter being trained on 20 times more human data. Ablation studies confirm the complementary contributions of grounded challenger training and curriculum-based solver training, and further analysis shows that R-Few mitigates drift, yielding more stable and controllable co-evolutionary dynamics.
☆ scCluBench: Comprehensive Benchmark ing of Clustering Algorithms for Single-Cell RNA Sequencing
Ping Xu, Zaitian Wang, Zhirui Wang, Pengjiang Li, Jiajia Wang, Ran Zhang, Pengfei Wang, Yuanchun Zhou
Cell clustering is crucial for uncovering cellular heterogeneity in single-cell RNA sequencing (scRNA-seq) data by identifying cell types and marker genes. Despite its importance, benchmarks for scRNA-seq clustering methods remain fragmented, often lacking standardized protocols and failing to incorporate recent advances in artificial intelligence. To fill these gaps, we present scCluBench, a comprehensive benchmark of clustering algorithms for scRNA-seq data. First, scCluBench provides 36 scRNA-seq datasets collected from diverse public sources, covering multiple tissues, which are uniformly processed and standardized to ensure consistency for systematic evaluation and downstream analyses. To evaluate performance, we collect and reproduce a range of scRNA-seq clustering methods, including traditional, deep learning-based, graph-based, and biological foundation models. We comprehensively evaluate each method both quantitatively and qualitatively, using core performance metrics as well as visualization analyses. Furthermore, we construct representative downstream biological tasks, such as marker gene identification and cell type annotation, to further assess the practical utility. scCluBench then investigates the performance differences and applicability boundaries of various clustering models across diverse analytical tasks, systematically assessing their robustness and scalability in real-world scenarios. Overall, scCluBench offers a standardized and user-friendly benchmark for scRNA-seq clustering, with curated datasets, unified evaluation protocols, and transparent analyses, facilitating informed method selection and providing valuable insights into model generalizability and application scope.
☆ TabGRU: An Enhanced Design for Urban Rainfall Intensity Estimation Using Commercial Microwave Links
Xingwang Li, Mengyun Chen, Jiamou Liu, Sijie Wang, Shuanggen Jin, Jafet C. M. Andersson, Jonas Olsson, Remco, van de Beek, Hai Victor Habi, Congzheng Han
In the face of accelerating global urbanization and the increasing frequency of extreme weather events, highresolution urban rainfall monitoring is crucial for building resilient smart cities. Commercial Microwave Links (CMLs) are an emerging data source with great potential for this task.While traditional rainfall retrieval from CMLs relies on physicsbased models, these often struggle with real-world complexities like signal noise and nonlinear attenuation. To address these limitations, this paper proposes a novel hybrid deep learning architecture based on the Transformer and a Bidirectional Gated Recurrent Unit (BiGRU), which we name TabGRU. This design synergistically captures both long-term dependencies and local sequential features in the CML signal data. The model is further enhanced by a learnable positional embedding and an attention pooling mechanism to improve its dynamic feature extraction and generalization capabilities. The model was validated on a public benchmark dataset from Gothenburg, Sweden (June-September 2015). The evaluation used 12 sub-links from two rain gauges (Torp and Barl) over a test period (August 22-31) covering approximately 10 distinct rainfall events. The proposed TabGRU model demonstrated consistent advantages, outperforming deep learning baselines and achieving high coefficients of determination (R2) at both the Torp site (0.91) and the Barl site (0.96). Furthermore, compared to the physics-based approach, TabGRU maintained higher accuracy and was particularly effective in mitigating the significant overestimation problem observed in the PL model during peak rainfall events. This evaluation confirms that the TabGRU model can effectively overcome the limitations of traditional methods, providing a robust and accurate solution for CML-based urban rainfall monitoring under the tested conditions.
☆ HouseLayout3D: A Benchmark and Training-Free Baseline for 3D Layout Estimation in the Wild NeurIPS 2025
Valentin Bieri, Marie-Julie Rakotosaona, Keisuke Tateno, Francis Engelmann, Leonidas Guibas
Current 3D layout estimation models are primarily trained on synthetic datasets containing simple single room or single floor environments. As a consequence, they cannot natively handle large multi floor buildings and require scenes to be split into individual floors before processing, which removes global spatial context that is essential for reasoning about structures such as staircases that connect multiple levels. In this work, we introduce HouseLayout3D, a real world benchmark designed to support progress toward full building scale layout estimation, including multiple floors and architecturally intricate spaces. We also present MultiFloor3D, a simple training free baseline that leverages recent scene understanding methods and already outperforms existing 3D layout estimation models on both our benchmark and prior datasets, highlighting the need for further research in this direction. Data and code are available at: https://houselayout3d.github.io.
comment : NeurIPS 2025 (Datasets and Benchmarks Track) Project Page: https://houselayout3d.github.io
☆ When Refusals Fail: Unstable Safety Mechanisms in Long- Context LLM Agent s AAAI 2026
Tsimur Hadeliya, Mohammad Ali Jauhar, Nidhi Sakpal, Diogo Cruz
Solving complex or long-horizon problems often requires large language models (LLMs) to use external tools and operate over a significantly longer context window. New LLMs enable longer context windows and support tool calling capabilities. Prior works have focused mainly on evaluation of LLMs on long-context prompts, leaving agentic setup relatively unexplored, both from capability and safety perspectives. Our work addresses this gap. We find that LLM agents could be sensitive to length, type, and placement of the context, exhibiting unexpected and inconsistent shifts in task performance and in refusals to execute harmful requests. Models with 1M-2M token context windows show severe degradation already at 100K tokens, with performance drops exceeding 50\% for both benign and harmful tasks. Refusal rates shift unpredictably: GPT-4.1-nano increases from $\sim$5\% to $\sim$40\% while Grok 4 Fast decreases from $\sim$80\% to $\sim$10\% at 200K tokens. Our work shows potential safety issues with agents operating on longer context and opens additional questions on the current metrics and paradigm for evaluating LLM agent safety on long multi-step tasks. In particular, our results on LLM agents reveal a notable divergence in both capability and safety performance compared to prior evaluations of LLMs on similar criteria.
comment : 12 pages, 11 figures. Accepted at AAAI 2026 TrustAgent Workshop
☆ Boosting Medical Vision -Language Pretrain ing via Momentum Self-Distillation under Limited Computing Resources WACV 2026
Phuc Pham, Nhu Pham, Ngoc Quoc Ly
In medical healthcare, obtaining detailed annotations is challenging, highlighting the need for robust Vision-Language Models (VLMs). Pretrained VLMs enable fine-tuning on small datasets or zero-shot inference, achieving performance comparable to task-specific models. Contrastive learning (CL) is a key paradigm for training VLMs but inherently requires large batch sizes for effective learning, making it computationally demanding and often limited to well-resourced institutions. Moreover, with limited data in healthcare, it is important to prioritize knowledge extraction from both data and models during training to improve performance. Therefore, we focus on leveraging the momentum method combined with distillation to simultaneously address computational efficiency and knowledge exploitation. Our contributions can be summarized as follows: (1) leveraging momentum self-distillation to enhance multimodal learning, and (2) integrating momentum mechanisms with gradient accumulation to enlarge the effective batch size without increasing resource consumption. Our method attains competitive performance with state-of-the-art (SOTA) approaches in zero-shot classification, while providing a substantial boost in the few-shot adaption, achieving over 90% AUC-ROC and improving retrieval tasks by 2-3%. Importantly, our method achieves high training efficiency with a single GPU while maintaining reasonable training time. Our approach aims to advance efficient multimodal learning by reducing resource requirements while improving performance over SOTA methods. The implementation of our method is available at https://github.com/phphuc612/MSD .
comment : WACV 2026
☆ LightHCG: a Lightweight yet powerful HSIC Disentanglement based Causal Glaucoma Detection Model framework
Daeyoung Kim
As a representative optic degenerative condition, glaucoma has been a threat to millions due to its irreversibility and severe impact on human vision fields. Mainly characterized by dimmed and blurred visions, or peripheral vision loss, glaucoma is well known to occur due to damages in the optic nerve from increased intraocular pressure (IOP) or neovascularization within the retina. Traditionally, most glaucoma related works and clinical diagnosis focused on detecting these damages in the optic nerve by using patient data from perimetry tests, optic papilla inspections and tonometer-based IOP measurements. Recently, with advancements in computer vision AI models, such as VGG16 or Vision Transformers (ViT), AI-automatized glaucoma detection and optic cup segmentation based on retinal fundus images or OCT recently exhibited significant performance in aiding conventional diagnosis with high performance. However, current AI-driven glaucoma detection approaches still have significant room for improvement in terms of reliability, excessive parameter usage, possibility of spurious correlation within detection, and limitations in applications to intervention analysis or clinical simulations. Thus, this research introduced a novel causal representation driven glaucoma detection model: LightHCG, an extremely lightweight Convolutional VAE-based latent glaucoma representation model that can consider the true causality among glaucoma-related physical factors within the optic nerve region. Using HSIC-based latent space disentanglement and Graph Autoencoder based unsupervised causal representation learning, LightHCG not only exhibits higher performance in classifying glaucoma with 93~99% less weights, but also enhances the possibility of AI-driven intervention analysis, compared to existing advanced vision models such as InceptionV3, MobileNetV2 or VGG16.
☆ Semantic Trading: Agent ic AI for Clustering and Relation ship Discovery in Prediction Markets
Agostino Capponi, Alfio Gliozzo, Brian Zhu
Prediction markets allow users to trade on outcomes of real-world events, but are prone to fragmentation through overlapping questions, implicit equivalences, and hidden contradictions across markets. We present an agentic AI pipeline that autonomously (i) clusters markets into coherent topical groups using natural-language understanding over contract text and metadata, and (ii) identifies within-cluster market pairs whose resolved outcomes exhibit strong dependence, including same-outcome (correlated) and different-outcome (anti-correlated) relationships. Using a historical dataset of resolved markets on Polymarket, we evaluate the accuracy of the agent's relational predictions. We then translate discovered relationships into a simple trading strategy to quantify how these relationships map to actionable signals. Results show that agent-identified relationships achieve roughly 60-70% accuracy, and their induced trading strategies earn about 20% average returns over week-long horizons, highlighting the ability of agentic AI and large language models to uncover latent semantic structure in prediction markets.
☆ Wo rl dMM: Dynamic Multimodal Memory Agent for Long Video Reasoning
Woongyeong Yeo, Kangsan Kim, Jaehong Yoon, Sung Ju Hwang
Recent advances in video large language models have demonstrated strong capabilities in understanding short clips. However, scaling them to hours- or days-long videos remains highly challenging due to limited context capacity and the loss of critical visual details during abstraction. Existing memory-augmented methods mitigate this by leveraging textual summaries of video segments, yet they heavily rely on text and fail to utilize visual evidence when reasoning over complex scenes. Moreover, retrieving from fixed temporal scales further limits their flexibility in capturing events that span variable durations. To address this, we introduce WorldMM, a novel multimodal memory agent that constructs and retrieves from multiple complementary memories, encompassing both textual and visual representations. WorldMM comprises three types of memory: episodic memory indexes factual events across multiple temporal scales, semantic memory continuously updates high-level conceptual knowledge, and visual memory preserves detailed information about scenes. During inference, an adaptive retrieval agent iteratively selects the most relevant memory source and leverages multiple temporal granularities based on the query, continuing until it determines that sufficient information has been gathered. WorldMM significantly outperforms existing baselines across five long video question-answering benchmarks, achieving an average 8.4% performance gain over previous state-of-the-art methods, showing its effectiveness on long video reasoning.
comment : Project page : https://worldmm.github.io
☆ Quantum feature encoding optimization
Tommaso Fioravanti, Brian Quanz, Gabriele Agliardi, Edgar Andres Ruiz Guzman, Ginés Carrascal, Jae-Eun Park
Quantum Machine Learning (QML) holds the promise of enhancing machine learning modeling in terms of both complexity and accuracy. A key challenge in this domain is the encoding of input data, which plays a pivotal role in determining the performance of QML models. In this work, we tackle a largely unaddressed aspect of encoding that is unique to QML modeling -- rather than adjusting the ansatz used for encoding, we consider adjusting how data is conveyed to the ansatz. We specifically implement QML pipelines that leverage classical data manipulation (i.e., ordering, selecting, and weighting features) as a preprocessing step, and evaluate if these aspects of encoding can have a significant impact on QML model performance, and if they can be effectively optimized to improve performance. Our experimental results, applied across a wide variety of data sets, ansatz, and circuit sizes, with a representative QML approach, demonstrate that by optimizing how features are encoded in an ansatz we can substantially and consistently improve the performance of QML models, making a compelling case for integrating these techniques in future QML applications. Finally we demonstrate the practical feasibility of this approach by running it using real quantum hardware with 100 qubit circuits and successfully achieving improved QML modeling performance in this case as well.
☆ The brain-AI convergence: Predictive and generative wo rl d models for general-purpose computation
Shogo Ohmae, Keiko Ohmae
Recent advances in general-purpose AI systems with attention-based transformers offer a potential window into how the neocortex and cerebellum, despite their relatively uniform circuit architectures, give rise to diverse functions and, ultimately, to human intelligence. This Perspective provides a cross-domain comparison between the brain and AI that goes beyond the traditional focus on visual processing, adopting the emerging perspecive of world-model-based computation. Here, we identify shared computational mechanisms in the attention-based neocortex and the non-attentional cerebellum: both predict future world events from past inputs and construct internal world models through prediction-error learning. These predictive world models are repurposed for seemingly distinct functions -- understanding in sensory processing and generation in motor processing -- enabling the brain to achieve multi-domain capabilities and human-like adaptive intelligence. Notably, attention-based AI has independently converged on a similar learning paradigm and world-model-based computation. We conclude that these shared mechanisms in both biological and artificial systems constitute a core computational foundation for realizing diverse functions including high-level intelligence, despite their relatively uniform circuit structures. Our theoretical insights bridge neuroscience and AI, advancing our understanding of the computational essence of intelligence.
comment : 22 pages, 4 figures. Related to our earlier preprint "The brain versus AI" (arXiv:2411.16075) but a distinct article. The earlier work surveyed broad brain-AI parallels; here we focus on world-model-based computation and convergent evolution between the brain and AI, especially large language models
☆ Vehicle Dynamics Embedded Wo rl d Models for Autonomous Driving
Huiqian Li, Wei Pan, Haodong Zhang, Jin Huang, Zhihua Zhong
World models have gained significant attention as a promising approach for autonomous driving. By emulating human-like perception and decision-making processes, these models can predict and adapt to dynamic environments. Existing methods typically map high-dimensional observations into compact latent spaces and learn optimal policies within these latent representations. However, prior work usually jointly learns ego-vehicle dynamics and environmental transition dynamics from the image input, leading to inefficiencies and a lack of robustness to variations in vehicle dynamics. To address these issues, we propose the Vehicle Dynamics embedded Dreamer (VDD) method, which decouples the modeling of ego-vehicle dynamics from environmental transition dynamics. This separation allows the world model to generalize effectively across vehicles with diverse parameters. Additionally, we introduce two strategies to further enhance the robustness of the learned policy: Policy Adjustment during Deployment (PAD) and Policy Augmentation during Training (PAT). Comprehensive experiments in simulated environments demonstrate that the proposed model significantly improves both driving performance and robustness to variations in vehicle dynamics, outperforming existing approaches.
☆ MitUNet: Enhancing Floor Plan Recognition using a Hybrid Mix- Transformer and U-Net Architecture
Dmitriy Parashchuk, Alexey Kapshitskiy, Yuriy Karyakin
Automatic 3D reconstruction of indoor spaces from 2D floor plans requires high-precision semantic segmentation of structural elements, particularly walls. However, existing methods optimized for standard metrics often struggle to detect thin structural components and yield masks with irregular boundaries, lacking the geometric precision required for subsequent vectorization. To address this issue, we introduce MitUNet, a hybrid neural network architecture specifically designed for wall segmentation tasks in the context of 3D modeling. In MitUNet, we utilize a hierarchical Mix-Transformer encoder to capture global context and a U-Net decoder enhanced with scSE attention blocks for precise boundary recovery. Furthermore, we propose an optimization strategy based on the Tversky loss function to effectively balance precision and recall. By fine-tuning the hyperparameters of the loss function, we prioritize the suppression of false positive noise along wall boundaries while maintaining high sensitivity to thin structures. Our experiments on the public CubiCasa5k dataset and a proprietary regional dataset demonstrate that the proposed approach ensures the generation of structurally correct masks with high boundary accuracy, outperforming standard single-task models. MitUNet provides a robust tool for data preparation in automated 3D reconstruction pipelines.
comment : 9 pages, 4 figures, 3 tables
☆ Data Curation Through the Lens of Spectral Dynamics: Static Limits, Dynamic Acceleration, and Practical Oracles
Yizhou Zhang, Lun Du
Large-scale neural models are increasingly trained with data pruning, synthetic data generation, cross-model distillation, reinforcement learning from human feedback (RLHF), and difficulty-based sampling. While several of these data-centric strategies reliably improve training efficiency and downstream performance, others fail to provide meaningful gains -- most notably self-generated synthetic data, which often increases dataset volume without enhancing model capability. We formalize data curation as reweighting the sampling distribution and map its effect onto the eigenstructure of the data-induced operator. Our first main result shows that \textbf{static pruning induces a bounded operator and therefore cannot change the spectral tail exponent}; it provides at most finite-region improvements and cannot alter asymptotic neural scaling. Our second result analyzes \textbf{time-dependent data curation}, showing that an ideal oracle capable of tracking spectral residuals and continuously re-normalizing the tail can provably accelerate learning -- although practical systems can only approximate this behavior.
☆ WISE: Weighted Iterative Society-of-Experts for Robust Multimodal Multi- Agent Debate
Anoop Cherian, River Doyle, Eyal Ben-Dov, Suhas Lohit, Kuan-Chuan Peng
Recent large language models (LLMs) are trained on diverse corpora and tasks, leading them to develop complementary strengths. Multi-agent debate (MAD) has emerged as a popular way to leverage these strengths for robust reasoning, though it has mostly been applied to language-only tasks, leaving its efficacy on multimodal problems underexplored. In this paper, we study MAD for solving vision-and-language reasoning problems. Our setup enables generalizing the debate protocol with heterogeneous experts that possess single- and multi-modal capabilities. To this end, we present Weighted Iterative Society-of-Experts (WISE), a generalized and modular MAD framework that partitions the agents into Solvers, that generate solutions, and Reflectors, that verify correctness, assign weights, and provide natural language feedback. To aggregate the agents' solutions across debate rounds, while accounting for variance in their responses and the feedback weights, we present a modified Dawid-Skene algorithm for post-processing that integrates our two-stage debate model. We evaluate WISE on SMART-840, VisualPuzzles, EvoChart-QA, and a new SMART-840++ dataset with programmatically generated problem instances of controlled difficulty. Our results show that WISE consistently improves accuracy by 2-7% over the state-of-the-art MAD setups and aggregation methods across diverse multimodal tasks and LLM configurations.
☆ Process-Centric Analysis of Agent ic Software Systems
Shuyang Liu, Yang Chen, Rahul Krishna, Saurabh Sinha, Jatin Ganhotra, Reyhan Jabbarvand
Agentic systems are modern software systems: they consist of orchestrated modules, expose interfaces, and are deployed in software pipelines. Unlike conventional programs, their execution (i.e., trajectories) is inherently stochastic and adaptive to the problem they are solving. Evaluation of such systems is often outcome-centric, judging their performance based on success or failure at the final step. This narrow focus overlooks detailed insights about such systems, failing to explain how agents reason, plan, act, or change their strategies over time. Inspired by the structured representation of conventional software systems as graphs, we introduce Graphectory to systematically encode the temporal and semantic relations in such software systems. Graphectory facilitates the design of process-centric metrics and analyses to assess the quality of agentic workflows independent of final success. Using Graphectory, we analyze 4000 trajectories of two dominant agentic programming workflows, namely SWE-agent and OpenHands, with a combination of four backbone Large Language Models (LLMs), attempting to resolve SWE-bench Verified issues. Our fully automated analyses reveal that: (1) agents using richer prompts or stronger LLMs exhibit more complex Graphectory, reflecting deeper exploration, broader context gathering, and more thorough validation before patch submission; (2) agents' problem-solving strategies vary with both problem difficulty and the underlying LLM -- for resolved issues, the strategies often follow coherent localization-patching-validation steps, while unresolved ones exhibit chaotic, repetitive, or backtracking behaviors; (3) even when successful, agentic programming systems often display inefficient processes, leading to unnecessarily prolonged trajectories.
☆ Synthetic Error Injection Fails to Elicit Self-Correction In Language Models
David X. Wu, Shreyas Kapur, Anant Sahai, Stuart Russell
Reinforcement learning has become the dominant paradigm for eliciting reasoning and self-correction capabilities in large language models, but its computational expense motivates exploration of alternatives. Inspired by techniques from autonomous driving and robotics, we investigate whether supervised learning with synthetic error injection can induce self-correction abilities in language models. Our approach inserts artificial errors into reasoning chains, masks them, and supervises the model to recognize and correct these mistakes. Despite the intuitive appeal of this method, we find that it fails to significantly improve performance even on simple synthetic tasks across multiple models. Moreover, even when the model catches its own error, it often parrots the original mistake. We find that the distribution shift of synthetic errors to on-policy errors significantly degrades the error-correction capabilities of the fine-tuned model, even with good synthetic coverage of on-policy errors. Our results help explain why on-policy reinforcement learning methods have proven uniquely effective for eliciting self-correction.
comment : 13 pages, 12 figures
☆ Multi-Domain Enhanced Map-Free Trajectory Prediction with Selective Attention
Wenyi Xiong, Jian Chen
Trajectory prediction is crucial for the reliability and safety of autonomous driving systems, yet it remains a challenging task in complex interactive scenarios. Existing methods often struggle to efficiently extract valuable scene information from redundant data, thereby reducing computational efficiency and prediction accuracy, especially when dealing with intricate agent interactions. To address these challenges, we propose a novel map-free trajectory prediction algorithm that achieves trajectory prediction across the temporal, spatial, and frequency domains. Specifically, in temporal information processing, We utilize a Mixture of Experts (MoE) mechanism to adaptively select critical frequency components. Concurrently, we extract these components and integrate multi-scale temporal features. Subsequently, a selective attention module is proposed to filter out redundant information in both temporal sequences and spatial interactions. Finally, we design a multimodal decoder. Under the supervision of patch-level and point-level losses, we obtain reasonable trajectory results. Experiments on Nuscences datasets demonstrate the superiority of our algorithm, validating its effectiveness in handling complex interactive scenarios.
☆ Tackling Tuberculosis: A Comparative Dive into Machine Learning for Tuberculosis Detection
Daanish Hindustani, Sanober Hindustani, Preston Nguyen
This study explores the application of machine learning models, specifically a pretrained ResNet-50 model and a general SqueezeNet model, in diagnosing tuberculosis (TB) using chest X-ray images. TB, a persistent infectious disease affecting humanity for millennia, poses challenges in diagnosis, especially in resource-limited settings. Traditional methods, such as sputum smear microscopy and culture, are inefficient, prompting the exploration of advanced technologies like deep learning and computer vision. The study utilized a dataset from Kaggle, consisting of 4,200 chest X-rays, to develop and compare the performance of the two machine learning models. Preprocessing involved data splitting, augmentation, and resizing to enhance training efficiency. Evaluation metrics, including accuracy, precision, recall, and confusion matrix, were employed to assess model performance. Results showcase that the SqueezeNet achieved a loss of 32%, accuracy of 89%, precision of 98%, recall of 80%, and an F1 score of 87%. In contrast, the ResNet-50 model exhibited a loss of 54%, accuracy of 73%, precision of 88%, recall of 52%, and an F1 score of 65%. This study emphasizes the potential of machine learning in TB detection and possible implications for early identification and treatment initiation. The possibility of integrating such models into mobile devices expands their utility in areas lacking TB detection resources. However, despite promising results, the need for continued development of faster, smaller, and more accurate TB detection models remains crucial in contributing to the global efforts in combating TB.
☆ Memory-Augmented Knowledge Fusion with Safety-Aware Decoding for Domain-Adaptive Question Answering
Lei Fu, Xiang Chen, Kaige Gao Xinyue Huang, Kejian Tong
Domain-specific question answering (QA) systems for services face unique challenges in integrating heterogeneous knowledge sources while ensuring both accuracy and safety. Existing large language models often struggle with factual consistency and context alignment in sensitive domains such as healthcare policies and government welfare. In this work, we introduce Knowledge-Aware Reasoning and Memory-Augmented Adaptation (KARMA), a novel framework designed to enhance QA performance in care scenarios. KARMA incorporates a dual-encoder architecture to fuse structured and unstructured knowledge sources, a gated memory unit to dynamically regulate external knowledge integration, and a safety-aware controllable decoder that mitigates unsafe outputs using safety classification and guided generation techniques. Extensive experiments on a proprietary QA dataset demonstrate that KARMA outperforms strong baselines in both answer quality and safety. This study offers a comprehensive solution for building trustworthy and adaptive QA systems in service contexts.
☆ VA CoT : Rethinking Visual Data Augmentation with VLMs
Zhengzhuo Xu, Chong Sun, SiNan Du, Chen Li, Jing Lyu, Chun Yuan
While visual data augmentation remains a cornerstone for training robust vision models, it has received limited attention in visual language models (VLMs), which predominantly rely on large-scale real data acquisition or synthetic diversity. Consequently, they may struggle with basic perception tasks that conventional models handle reliably. Given the substantial cost of pre-training and fine-tuning VLMs, continue training on augmented data yields limited and diminishing returns. In this paper, we present Visual Augmentation Chain-of-Thought (VACoT), a framework that dynamically invokes image augmentations during model inference. By incorporating post-hoc transformations such as denoising, VACoT substantially improves robustness on challenging and out-of-distribution inputs, especially in OCR-related adversarial scenarios. Distinct from prior approaches limited to local cropping, VACoT integrates a structured collection of general visual augmentations, broadening the query image views while reducing training complexity and computational overhead with efficient agentic reinforcement learning. We propose a conditional reward scheme that encourages necessary augmentation while penalizing verbose responses, ensuring concise and effective reasoning in perception tasks. We demonstrate the superiority of VACoT with extensive experiments on 13 perception benchmarks and further introduce AdvOCR to highlight the generalization benefits of post-hoc visual augmentations in adversarial scenarios.
☆ Beyond Playtesting: A Generative Multi- Agent Simulation System for Massively Multiplayer Online Games
Ran Zhang, Kun Ouyang, Tiancheng Ma, Yida Yang, Dong Fang
Optimizing numerical systems and mechanism design is crucial for enhancing player experience in Massively Multiplayer Online (MMO) games. Traditional optimization approaches rely on large-scale online experiments or parameter tuning over predefined statistical models, which are costly, time-consuming, and may disrupt player experience. Although simplified offline simulation systems are often adopted as alternatives, their limited fidelity prevents agents from accurately mimicking real player reasoning and reactions to interventions. To address these limitations, we propose a generative agent-based MMO simulation system empowered by Large Language Models (LLMs). By applying Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on large-scale real player behavioral data, we adapt LLMs from general priors to game-specific domains, enabling realistic and interpretable player decision-making. In parallel, a data-driven environment model trained on real gameplay logs reconstructs dynamic in-game systems. Experiments demonstrate strong consistency with real-world player behaviors and plausible causal responses under interventions, providing a reliable, interpretable, and cost-efficient framework for data-driven numerical design optimization.
☆ Understanding and Harnessing Sparsity in Unified Multimodal Models
Shwai He, Chaorui Deng, Ang Li, Shen Yan
Large multimodal models have achieved remarkable progress in both understanding and generation. Recent efforts pursue unified multimodal models that integrate heterogeneous components to support both capabilities within a single framework. However, such unification introduces inference inefficiencies, e.g., specific tasks or samples may not require the full knowledge or capacity of the unified model. Yet, a systematic understanding of how these inefficiencies manifest across different components remains limited. In this work, we first conduct a systematic analysis of unified multimodal model components using training-free pruning as a probing methodology, considering both depth pruning and width reduction. Our study reveals that the understanding component exhibits notable compressibility in both understanding and generation tasks, which is more pronounced in the latter. In contrast, the generation components are highly sensitive to compression, with performance deteriorating sharply even under moderate compression ratios. To address this limitation, we propose the Mixture-of-Experts (MoE) Adaptation, inspired by the dynamic activation patterns observed across different samples. This approach partitions the generation module into multiple experts and enables sparse activation to restore generation quality. We validate the effectiveness of sparse activation through expert-frozen tuning and further demonstrate that a fully trainable adaptation delivers additional gains. As a result, the adapted BAGEL model achieves performance comparable to the full model while activating only about half of its parameters. The code is released at \href{https://github.com/Shwai-He/SparseUnifiedModel}{this link}.
comment : 13 pages, 13 figures, 8 tables
☆ FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data
Nan Qiao, Sheng Yue, Ju Ren, Yaoxue Zhang
Offline Federated Reinforcement Learning (FRL), a marriage of federated learning and offline reinforcement learning, has attracted increasing interest recently. Albeit with some advancement, we find that the performance of most existing offline FRL methods drops dramatically when provided with mixed-quality data, that is, the logging behaviors (offline data) are collected by policies with varying qualities across clients. To overcome this limitation, this paper introduces a new vote-based offline FRL framework, named FOVA. It exploits a \emph{vote mechanism} to identify high-return actions during local policy evaluation, alleviating the negative effect of low-quality behaviors from diverse local learning policies. Besides, building on advantage-weighted regression (AWR), we construct consistent local and global training objectives, significantly enhancing the efficiency and stability of FOVA. Further, we conduct an extensive theoretical analysis and rigorously show that the policy learned by FOVA enjoys strict policy improvement over the behavioral policy. Extensive experiments corroborate the significant performance gains of our proposed algorithm over existing baselines on widely used benchmarks.
comment : Accepted by IEEE/ACM ToN
☆ Reasoning Path and Latent State Analysis for Multi-view Visual Spatial Reasoning : A Cognitive Science Perspective
Qiyao Xue, Weichen Liu, Shiqi Wang, Haoming Wang, Yuyang Wu, Wei Gao
Spatial reasoning is a core aspect of human intelligence that allows perception, inference and planning in 3D environments. However, current vision-language models (VLMs) struggle to maintain geometric coherence and cross-view consistency for spatial reasoning in multi-view settings. We attribute this gap to the lack of fine-grained benchmarks that isolate multi-view reasoning from single-view perception and temporal factors. To address this, we present ReMindView-Bench, a cognitively grounded benchmark for evaluating how VLMs construct, align and maintain spatial mental models across complementary viewpoints. ReMindView-Bench systematically varies viewpoint spatial pattern and query type to probe key factors of spatial cognition. Evaluations of 15 current VLMs reveals consistent failures in cross-view alignment and perspective-taking in multi-view spatial reasoning, motivating deeper analysis on the reasoning process. Explicit phase-wise analysis using LLM-as-a-judge and self-consistency prompting shows that VLMs perform well on in-frame perception but degrade sharply when integrating information across views. Implicit analysis, including linear probing and entropy dynamics, further show progressive loss of task-relevant information and uncertainty separation between correct and incorrect trajectories. These results provide a cognitively grounded diagnosis of VLM spatial reasoning and reveal how multi-view spatial mental models are formed, degraded and destabilized across reasoning phases. The ReMindView-Bench benchmark is available at https://huggingface.co/datasets/Xue0823/ReMindView-Bench, and the source codes of benchmark construction and VLM reasoning analysis are available at https://github.com/pittisl/ReMindView-Bench.
comment : 23 pages, 37 figures
☆ Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Super vision NeurIPS 2025
Chenshuang Zhang, Kang Zhang, Joon Son Chung, In So Kweon, Junmo Kim, Chengzhi Mao
Distinguishing visually similar objects by their motion remains a critical challenge in computer vision. Although supervised trackers show promise, contemporary self-supervised trackers struggle when visual cues become ambiguous, limiting their scalability and generalization without extensive labeled data. We find that pre-trained video diffusion models inherently learn motion representations suitable for tracking without task-specific training. This ability arises because their denoising process isolates motion in early, high-noise stages, distinct from later appearance refinement. Capitalizing on this discovery, our self-supervised tracker significantly improves performance in distinguishing visually similar objects, an underexplored failure point for existing methods. Our method achieves up to a 6-point improvement over recent self-supervised approaches on established benchmarks and our newly introduced tests focused on tracking visually similar items. Visualizations confirm that these diffusion-derived motion representations enable robust tracking of even identical objects across challenging viewpoint changes and deformations.
comment : Accepted at NeurIPS 2025
☆ COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers
Junyu Wang, Changjia Zhu, Yuanbo Zhou, Lingyao Li, Xu He, Junjie Xiong
This paper studies how multimodal large language models (MLLMs) undermine the security guarantees of visual CAPTCHA. We identify the attack surface where an adversary can cheaply automate CAPTCHA solving using off-the-shelf models. We evaluate 7 leading commercial and open-source MLLMs across 18 real-world CAPTCHA task types, measuring single-shot accuracy, success under limited retries, end-to-end latency, and per-solve cost. We further analyze the impact of task-specific prompt engineering and few-shot demonstrations on solver effectiveness. We reveal that MLLMs can reliably solve recognition-oriented and low-interaction CAPTCHA tasks at human-like cost and latency, whereas tasks requiring fine-grained localization, multi-step spatial reasoning, or cross-frame consistency remain significantly harder for current models. By examining the reasoning traces of such MLLMs, we investigate the underlying mechanisms of why models succeed/fail on specific CAPTCHA puzzles and use these insights to derive defense-oriented guidelines for selecting and strengthening CAPTCHA tasks. We conclude by discussing implications for platform operators deploying CAPTCHA as part of their abuse-mitigation pipeline.Code Availability (https://anonymous.4open.science/r/Captcha-465E/).
☆ OmniGuard: Unified Omni-Modal Guardrails with Deliberate Reasoning
Boyu Zhu, Xiaofei Wen, Wenjie Jacky Mo, Tinghui Zhu, Yanan Xie, Peng Qi, Muhao Chen
Omni-modal Large Language Models (OLLMs) that process text, images, videos, and audio introduce new challenges for safety and value guardrails in human-AI interaction. Prior guardrail research largely targets unimodal settings and typically frames safeguarding as binary classification, which limits robustness across diverse modalities and tasks. To address this gap, we propose OmniGuard, the first family of omni-modal guardrails that performs safeguarding across all modalities with deliberate reasoning ability. To support the training of OMNIGUARD, we curate a large, comprehensive omni-modal safety dataset comprising over 210K diverse samples, with inputs that cover all modalities through both unimodal and cross-modal samples. Each sample is annotated with structured safety labels and carefully curated safety critiques from expert models through targeted distillation. Extensive experiments on 15 benchmarks show that OmniGuard achieves strong effectiveness and generalization across a wide range of multimodal safety scenarios. Importantly, OmniGuard provides a unified framework that enforces policies and mitigates risks in omni-modalities, paving the way toward building more robust and capable omnimodal safeguarding systems.
☆ Breast Cell Segmentation Under Extreme Data Constraints: Quantum Enhancement Meets Adaptive Loss Stabilization
Varun Kumar Dasoju, Qingsu Cheng, Zeyun Yu
Annotating medical images demands significant time and expertise, often requiring pathologists to invest hundreds of hours in labeling mammary epithelial nuclei datasets. We address this critical challenge by achieving 95.5% Dice score using just 599 training images for breast cell segmentation, where just 4% of pixels represent breast tissue and 60% of images contain no breast regions. Our framework uses quantum-inspired edge enhancement via multi-scale Gabor filters creating a fourth input channel, enhancing boundary detection where inter-annotator variations reach +/- 3 pixels. We present a stabilized multi-component loss function that integrates adaptive Dice loss with boundary-aware terms and automatic positive weighting to effectively address severe class imbalance, where mammary epithelial cell regions comprise only 0.1%-20% of the total image area. Additionally, a complexity-based weighted sampling strategy is introduced to prioritize the challenging mammary epithelial cell regions. The model employs an EfficientNet-B7/UNet++ architecture with a 4-to-3 channel projection, enabling the use of pretrained weights despite limited medical imaging data. Finally, robust validation is achieved through exponential moving averaging and statistical outlier detection, ensuring reliable performance estimates on a small validation set (129 images). Our framework achieves a Dice score of 95.5% +/- 0.3% and an IoU of 91.2% +/- 0.4%. Notably, quantum-based enhancement contributes to a 2.1% improvement in boundary accuracy, while weighted sampling increases small lesion detection by 3.8%. By achieving groundbreaking performance with limited annotations, our approach significantly reduces the medical expert time required for dataset creation, addressing a fundamental bottleneck in clinical perception AI development.
comment : 9 pages, 3 figures, 2 tables
☆ HealthContradict: Evaluating Biomedical Knowledge Conflicts in Language Models
Boya Zhang, Alban Bornet, Rui Yang, Nan Liu, Douglas Teodoro
How do language models use contextual information to answer health questions? How are their responses impacted by conflicting contexts? We assess the ability of language models to reason over long, conflicting biomedical contexts using HealthContradict, an expert-verified dataset comprising 920 unique instances, each consisting of a health-related question, a factual answer supported by scientific evidence, and two documents presenting contradictory stances. We consider several prompt settings, including correct, incorrect or contradictory context, and measure their impact on model outputs. Compared to existing medical question-answering evaluation benchmarks, HealthContradict provides greater distinctions of language models' contextual reasoning capabilities. Our experiments show that the strength of fine-tuned biomedical language models lies not only in their parametric knowledge from pretraining, but also in their ability to exploit correct context while resisting incorrect context.
☆ Enhancing Cross Domain SAR Oil Spill Segmentation via Morphological Region Perturbation and Synthetic Label-to-SAR Generation
Andre Juarez, Luis Salsavilca, Frida Coaquira, Celso Gonzales
Deep learning models for SAR oil spill segmentation often fail to generalize across regions due to differences in sea-state, backscatter statistics, and slick morphology, a limitation that is particularly severe along the Peruvian coast where labeled Sentinel-1 data remain scarce. To address this problem, we propose \textbf{MORP--Synth}, a two-stage synthetic augmentation framework designed to improve transfer from Mediterranean to Peruvian conditions. Stage~A applies Morphological Region Perturbation, a curvature guided label space method that generates realistic geometric variations of oil and look-alike regions. Stage~B renders SAR-like textures from the edited masks using a conditional generative INADE model. We compile a Peruvian dataset of 2112 labeled 512$\times$512 patches from 40 Sentinel-1 scenes (2014--2024), harmonized with the Mediterranean CleanSeaNet benchmark, and evaluate seven segmentation architectures. Models pretrained on Mediterranean data degrade from 67.8\% to 51.8\% mIoU on the Peruvian domain; MORP--Synth improves performance up to +6 mIoU and boosts minority-class IoU (+10.8 oil, +14.6 look-alike).
☆ Retrofitting Earth System Models with Cadence-Limited Neural Operator Updates
Aniruddha Bora, Shixuan Zhang, Khemraj Shukla, Bryce Harrop, George Em. Karniadakis, L. Ruby Leung
Coarse resolution, imperfect parameterizations, and uncertain initial states and forcings limit Earth-system model (ESM) predictions. Traditional bias correction via data assimilation improves constrained simulations but offers limited benefit once models run freely. We introduce an operator-learning framework that maps instantaneous model states to bias-correction tendencies and applies them online during integration. Building on a U-Net backbone, we develop two operator architectures Inception U-Net (IUNet) and a multi-scale network (M\&M) that combine diverse upsampling and receptive fields to capture multiscale nonlinear features under Energy Exascale Earth System Model (E3SM) runtime constraints. Trained on two years E3SM simulations nudged toward ERA5 reanalysis, the operators generalize across height levels and seasons. Both architectures outperform standard U-Net baselines in offline tests, indicating that functional richness rather than parameter count drives performance. In online hybrid E3SM runs, M\&M delivers the most consistent bias reductions across variables and vertical levels. The ML-augmented configurations remain stable and computationally feasible in multi-year simulations, providing a practical pathway for scalable hybrid modeling. Our framework emphasizes long-term stability, portability, and cadence-limited updates, demonstrating the utility of expressive ML operators for learning structured, cross-scale relationships and retrofitting legacy ESMs.
☆ Robust Tabular Foundation Models AAAI 2026
Matthew Peroni, Franck Le, Vadim Sheinin
The development of tabular foundation models (TFMs) has accelerated in recent years, showing strong potential to outperform traditional ML methods for structured data. A key finding is that TFMs can be pretrained entirely on synthetic datasets, opening opportunities to design data generators that encourage desirable model properties. Prior work has mainly focused on crafting high-quality priors over generators to improve overall pretraining performance. Our insight is that parameterizing the generator distribution enables an adversarial robustness perspective: during training, we can adapt the generator to emphasize datasets that are particularly challenging for the model. We formalize this by introducing an optimality gap measure, given by the difference between TFM performance and the best achievable performance as estimated by strong baselines such as XGBoost, CatBoost, and Random Forests. Building on this idea, we propose Robust Tabular Foundation Models (RTFM), a model-agnostic adversarial training framework. Applied to the TabPFN V2 classifier, RTFM improves benchmark performance, with up to a 6% increase in mean normalized AUC over the original TabPFN and other baseline algorithms, while requiring less than 100k additional synthetic datasets. These results highlight a promising new direction for targeted adversarial training and fine-tuning of TFMs using synthetic data alone.
comment : Shaping Responsible Synthetic Data in the Era of Foundation Models, AAAI 2026
☆ HydroDCM: Hydrological Domain-Conditioned Modulation for Cross-Reservoir Inflow Prediction AAAI 2026
Pengfei Hu, Fan Ming, Xiaoxue Han, Chang Lu, Yue Ning, Dan Lu
Deep learning models have shown promise in reservoir inflow prediction, yet their performance often deteriorates when applied to different reservoirs due to distributional differences, referred to as the domain shift problem. Domain generalization (DG) solutions aim to address this issue by extracting domain-invariant representations that mitigate errors in unseen domains. However, in hydrological settings, each reservoir exhibits unique inflow patterns, while some metadata beyond observations like spatial information exerts indirect but significant influence. This mismatch limits the applicability of conventional DG techniques to many-domain hydrological systems. To overcome these challenges, we propose HydroDCM, a scalable DG framework for cross-reservoir inflow forecasting. Spatial metadata of reservoirs is used to construct pseudo-domain labels that guide adversarial learning of invariant temporal features. During inference, HydroDCM adapts these features through light-weight conditioning layers informed by the target reservoir's metadata, reconciling DG's invariance with location-specific adaptation. Experiment results on 30 real-world reservoirs in the Upper Colorado River Basin demonstrate that our method substantially outperforms state-of-the-art DG baselines under many-domain conditions and remains computationally efficient.
comment : Accepted by AAAI 2026 workshop (oral) on AI for Environmental Science
☆ Adaptive Regime-Switching Forecasts with Distribution-Free Uncertainty: Deep Switching State-Space Models Meet Conformal Prediction
Echo Diyun LU, Charles Findling, Marianne Clausel, Alessandro Leite, Wei Gong, Pierric Kersaudy
Regime transitions routinely break stationarity in time series, making calibrated uncertainty as important as point accuracy. We study distribution-free uncertainty for regime-switching forecasting by coupling Deep Switching State Space Models with Adaptive Conformal Inference (ACI) and its aggregated variant (AgACI). We also introduce a unified conformal wrapper that sits atop strong sequence baselines including S4, MC-Dropout GRU, sparse Gaussian processes, and a change-point local model to produce online predictive bands with finite-sample marginal guarantees under nonstationarity and model misspecification. Across synthetic and real datasets, conformalized forecasters achieve near-nominal coverage with competitive accuracy and generally improved band efficiency.
☆ Prior preferences in active inference agent s: soft, hard, and goal shaping
Filippo Torresan, Ryota Kanai, Manuel Baltieri
Active inference proposes expected free energy as an objective for planning and decision-making to adequately balance exploitative and explorative drives in learning agents. The exploitative drive, or what an agent wants to achieve, is formalised as the Kullback-Leibler divergence between a variational probability distribution, updated at each inference step, and a preference probability distribution that indicates what states or observations are more likely for the agent, hence determining the agent's goal in a certain environment. In the literature, the questions of how the preference distribution should be specified and of how a certain specification impacts inference and learning in an active inference agent have been given hardly any attention. In this work, we consider four possible ways of defining the preference distribution, either providing the agents with hard or soft goals and either involving or not goal shaping (i.e., intermediate goals). We compare the performances of four agents, each given one of the possible preference distributions, in a grid world navigation task. Our results show that goal shaping enables the best performance overall (i.e., it promotes exploitation) while sacrificing learning about the environment's transition dynamics (i.e., it hampers exploration).
comment : 41 pages, 23 figures
☆ BlendedNet++: A Large-Scale Blended Wing Body Aerodynamics Dataset and Benchmark
Nicholas Sung, Steven Spreizer, Mohamed Elrefaie, Matthew C. Jones, Faez Ahmed
Despite progress in machine learning-based aerodynamic surrogates, the scarcity of large, field-resolved datasets limits progress on accurate pointwise prediction and reproducible inverse design for aircraft. We introduce BlendedNet++, a large-scale aerodynamic dataset and benchmark focused on blended wing body (BWB) aircraft. The dataset contains over 12,000 unique geometries, each simulated at a single flight condition, yielding 12,490 aerodynamic results for steady RANS CFD. For every case, we provide (i) integrated force/moment coefficients CL, CD, CM and (ii) dense surface fields of pressure and skin friction coefficients Cp and (Cfx, Cfy, Cfz). Using this dataset, we standardize a forward-surrogate benchmark to predict pointwise fields across six model families: GraphSAGE, GraphUNet, PointNet, a coordinate Transformer (Transolver-style), a FiLMNet (coordinate MLP with feature-wise modulation), and a Graph Neural Operator Transformer (GNOT). Finally, we present an inverse design task of achieving a specified lift-to-drag ratio under fixed flight conditions, implemented via a conditional diffusion model. To assess performance, we benchmark this approach against gradient-based optimization on the same surrogate and a diffusion-optimization hybrid that first samples with the conditional diffusion model and then further optimizes the designs. BlendedNet++ provides a unified forward and inverse protocol with multi-model baselines, enabling fair, reproducible comparison across architectures and optimization paradigms. We expect BlendedNet++ to catalyze reproducible research in field-level aerodynamics and inverse design; resources (dataset, splits, baselines, and scripts) will be released upon acceptance.
☆ Thucy: An LLM-based Multi- Agent System for Claim Verification across Relation al Databases AAAI 2026
Michael Theologitis, Dan Suciu
In today's age, it is becoming increasingly difficult to decipher truth from lies. Every day, politicians, media outlets, and public figures make conflicting claims$\unicode{x2014}$often about topics that can, in principle, be verified against structured data. For instance, statements about crime rates, economic growth or healthcare can all be verified against official public records and structured datasets. Building a system that can automatically do that would have sounded like science fiction just a few years ago. Yet, with the extraordinary progress in LLMs and agentic AI, this is now within reach. Still, there remains a striking gap between what is technically possible and what is being demonstrated by recent work. Most existing verification systems operate only on small, single-table databases$\unicode{x2014}$typically a few hundred rows$\unicode{x2014}$that conveniently fit within an LLM's context window. In this paper we report our progress on Thucy, the first cross-database, cross-table multi-agent claim verification system that also provides concrete evidence for each verification verdict. Thucy remains completely agnostic to the underlying data sources before deployment and must therefore autonomously discover, inspect, and reason over all available relational databases to verify claims. Importantly, Thucy also reports the exact SQL queries that support its verdict (whether the claim is accurate or not) offering full transparency to expert users familiar with SQL. When evaluated on the TabFact dataset$\unicode{x2014}$the standard benchmark for fact verification over structured data$\unicode{x2014}$Thucy surpasses the previous state of the art by 5.6 percentage points in accuracy (94.3% vs. 88.7%).
comment : Accepted at AAAI 2026 Workshop on LLM-based Multi-Agent Systems (LaMAS)
☆ When Do Symbolic Solvers Enhance Reasoning in Large Language Models?
Zhiyuan He, Dingmin Wang
Large Reasoning Models (LRMs) achieve strong performance on complex reasoning tasks by generating long Chains of Thought (CoTs). However, this paradigm might incur substantial token overhead, especially when models "overthink" by producing lengthy reasoning chains, which can even lead to incorrect answers. A promising direction is the symbolic-solver-integrated approach, which leverages the code generation capabilities of LLMs to translate reasoning tasks into executable code and then solve them with a symbolic solver. In this paper, we explore an open question of when the conventional long-CoT can be enhanced by symbolic solvers. Our experimental results show that the symbolic-solver-integrated method only helps when the problem requires limited implicit reasoning but involves an ample search space. The latest LLMs, like GPT-4o, show better performance on deductive problems with shallow reasoning depth, while the symbolic-solver-integrated method significantly improves the LLMs' performance in constraint satisfaction problems that require repeated backtracks. When a declarative exemplar is provided, even CodeLlama-13B can outperform GPT-4o in difficult Zebra puzzles.
☆ PyroFocus: A Deep Learning Approach to Real-Time Wildfire Detection in Multispectral Remote Sensing Imagery
Mark Moussa, Andre Williams, Seth Roffe, Douglas Morton
Rapid and accurate wildfire detection is crucial for emergency response and environmental management. In airborne and spaceborne missions, real-time algorithms must distinguish between no fire, active fire, and post-fire conditions, and estimate fire intensity. Multispectral and hyperspectral thermal imagers provide rich spectral information, but high data dimensionality and limited onboard resources make real-time processing challenging. As wildfires increase in frequency and severity, the need for low-latency and computationally efficient onboard detection methods is critical. We present a systematic evaluation of multiple deep learning architectures, including custom Convolutional Neural Networks (CNNs) and Transformer-based models, for multi-class fire classification. We also introduce PyroFocus, a two-stage pipeline that performs fire classification followed by fire radiative power (FRP) regression or segmentation to reduce inference time and computational cost for onboard deployment. Using data from NASA's MODIS/ASTER Airborne Simulator (MASTER), which is similar to a next-generation fire detection sensor, we compare accuracy, inference latency, and resource efficiency. Experimental results show that the proposed two-stage pipeline achieves strong trade-offs between speed and accuracy, demonstrating significant potential for real-time edge deployment in future wildfire monitoring missions.
☆ Learning Network Sheaves for AI-native Semantic Communication
Enrico Grimaldi, Mario Edoardo Pandolfo, Gabriele D'Acunto, Sergio Barbarossa, Paolo Di Lorenzo
Recent advances in AI call for a paradigm shift from bit-centric communication to goal- and semantics-oriented architectures, paving the way for AI-native 6G networks. In this context, we address a key open challenge: enabling heterogeneous AI agents to exchange compressed latent-space representations while mitigating semantic noise and preserving task-relevant meaning. We cast this challenge as learning both the communication topology and the alignment maps that govern information exchange among agents, yielding a learned network sheaf equipped with orthogonal maps. This learning process is further supported by a semantic denoising end compression module that constructs a shared global semantic space and derives sparse, structured representations of each agent's latent space. This corresponds to a nonconvex dictionary learning problem solved iteratively with closed-form updates. Experiments with mutiple AI agents pre-trained on real image data show that the semantic denoising and compression facilitates AI agents alignment and the extraction of semantic clusters, while preserving high accuracy in downstream task. The resulting communication network provides new insights about semantic heterogeneity across agents, highlighting the interpretability of our methodology.
☆ SPARK: Stepwise Process-Aware Rewards for Reference-Free Reinforcement Learning
Salman Rahman, Sruthi Gorantla, Arpit Gupta, Swastik Roy, Nanyun Peng, Yang Liu
Process reward models (PRMs) that provide dense, step-level feedback have shown promise for reinforcement learning, yet their adoption remains limited by the need for expensive step-level annotations or ground truth references. We propose SPARK: a three-stage framework where in the first stage a generator model produces diverse solutions and a verifier model evaluates them using parallel scaling (self-consistency) and sequential scaling (meta-critique). In the second stage, we use these verification outputs as synthetic training data to fine-tune generative process reward models, which subsequently serve as reward signals during training. We show that aggregating multiple independent verifications at the step level produces training data for process reward models that surpass ground-truth outcome supervision, achieving 67.5 F1 on ProcessBench (a benchmark for identifying erroneous steps in mathematical reasoning) compared to 66.4 for reference-guided training and 61.9 for GPT-4o. In the final stage, we apply our generative PRM with chain-of-thought verification (PRM-CoT) as the reward model in RL experiments on mathematical reasoning, and introduce format constraints to prevent reward hacking. Using Qwen2.5-Math-7B, we achieve 47.4% average accuracy across six mathematical reasoning benchmarks, outperforming ground-truth-based RLVR (43.9%). Our work enables reference-free RL training that exceeds ground-truth methods, opening new possibilities for domains lacking verifiable answers or accessible ground truth.
☆ How to DP-fy Your Data: A Practical Guide to Generating Synthetic Data With Differential Privacy
Natalia Ponomareva, Zheng Xu, H. Brendan McMahan, Peter Kairouz, Lucas Rosenblatt, Vincent Cohen-Addad, Cristóbal Guzmán, Ryan McKenna, Galen Andrew, Alex Bie, Da Yu, Alex Kurakin, Morteza Zadimoghaddam, Sergei Vassilvitskii, Andreas Terzis
High quality data is needed to unlock the full potential of AI for end users. However finding new sources of such data is getting harder: most publicly-available human generated data will soon have been used. Additionally, publicly available data often is not representative of users of a particular system -- for example, a research speech dataset of contractors interacting with an AI assistant will likely be more homogeneous, well articulated and self-censored than real world commands that end users will issue. Therefore unlocking high-quality data grounded in real user interactions is of vital interest. However, the direct use of user data comes with significant privacy risks. Differential Privacy (DP) is a well established framework for reasoning about and limiting information leakage, and is a gold standard for protecting user privacy. The focus of this work, \emph{Differentially Private Synthetic data}, refers to synthetic data that preserves the overall trends of source data,, while providing strong privacy guarantees to individuals that contributed to the source dataset. DP synthetic data can unlock the value of datasets that have previously been inaccessible due to privacy concerns and can replace the use of sensitive datasets that previously have only had rudimentary protections like ad-hoc rule-based anonymization. In this paper we explore the full suite of techniques surrounding DP synthetic data, the types of privacy protections they offer and the state-of-the-art for various modalities (image, tabular, text and decentralized). We outline all the components needed in a system that generates DP synthetic data, from sensitive data handling and preparation, to tracking the use and empirical privacy testing. We hope that work will result in increased adoption of DP synthetic data, spur additional research and increase trust in DP synthetic data approaches.
☆ InvertiTune: High-Quality Data Synthesis for Cost-Effective Single-Shot Text-to-Knowledge Graph Generation
Faezeh Faez, Marzieh S. Tahaei, Yaochen Hu, Ali Pourranjbar, Mahdi Biparva, Mark Coates, Yingxue Zhang
Large Language Models (LLMs) have revolutionized the ability to understand and generate text, enabling significant progress in automatic knowledge graph construction from text (Text2KG). Many Text2KG methods, however, rely on iterative LLM prompting, making them computationally expensive and prone to overlooking complex relations distributed throughout the text. To address these limitations, we propose InvertiTune, a framework that combines a controlled data generation pipeline with supervised fine-tuning (SFT). Within this framework, the data-generation pipeline systematically extracts subgraphs from large knowledge bases, applies noise filtering, and leverages LLMs to generate corresponding natural text descriptions, a task more aligned with LLM capabilities than direct KG generation from text. This pipeline enables generating datasets composed of longer texts paired with larger KGs that better reflect real-world scenarios compared to existing benchmarks, thus supporting effective SFT of lightweight models for single-shot KG construction. Experimental results on CE12k, a dataset generated using the introduced pipeline, show that InvertiTune outperforms larger non-fine-tuned LLMs as well as state-of-the-art Text2KG approaches, while also demonstrating stronger cross-dataset generalization on CrossEval-1200, a test set created from three established benchmark datasets and CE12k. These findings highlight the importance of realistic, high-quality training data for advancing efficient and high-performing Text2KG systems.
☆ Ultra-Strong Gradient Diffusion MRI with Self-Supervised Learning for Prostate Cancer Characterization
Tanishq Patil, Snigdha Sen, Malwina Molendowska, Kieran G. Foley, Fabrizio Fasano, Mara Cercignani, Marco Palombo, Paddy J. Slator, Eleftheria Panagiotaki
Diffusion MRI (dMRI) enables non-invasive assessment of prostate microstructure but conventional metrics such as the Apparent Diffusion Coefficient in multiparametric MRI lack specificity to underlying histology. Integrating dMRI with the compartment-based biophysical VERDICT (Vascular, Extracellular, and Restricted Diffusion for Cytometry in Tumours) framework offers richer microstructural insights, though clinical gradient systems (40-80 mT/m) suffer from poor signal-to-noise ratio (SNR) at stronger diffusion weightings due to prolonged echo times. Ultra-strong gradients (up to 300 mT/m) can mitigate these limitations by improving SNR and contrast-to-noise ratios (CNR) but their adoption has until recently been limited to research environments due to challenges with peripheral nerve stimulation thresholds and gradient non-uniformity. This study investigates whether physics-informed self-supervised VERDICT (ssVERDICT) fitting applied to ultra-strong gradients enhances prostate cancer characterization relative to current clinical acquisitions. We developed enhanced ssVERDICT fitting approaches using dense multilayer perceptron (Dense MLP) and convolutional U-Net architectures, benchmarking them against non-linear least-squares (NLLS) fitting and Diffusion Kurtosis Imaging across clinical- to ultra-strong gradient systems. Dense ssVERDICT at ultra-strong gradient notably outperformed NLLS VERDICT, boosting median CNR by 47%, cutting inter-patient Coefficient of Variation by 52%, and reducing pooled f_ic variation by 50%. Overall, it delivered the highest CNR, the most stable parameter estimates, and the clearest tumour-normal contrast compared with conventional methods and clinical gradient systems. These findings highlight the potential of advanced gradient systems and deep learning-based modelling to improve non-invasive prostate cancer characterization and reduce unnecessary biopsies.
comment : 24 pages, 17 figures, 7 tables
☆ Plantain: Plan-Answer Inte rl eaved Reasoning
Anthony Liang, Jonathan Berant, Adam Fisch, Abhimanyu Goyal, Kalpesh Krishna, Jacob Eisenstein
Reasoning models often spend a significant amount of time thinking before they generate a visible response. In the meantime, they do not give the user any hints as to whether their reasoning is on the right track, and do not give the user any recourse to stop and correct them if their reasoning is flawed. This creates a frustrating, but unfortunately common, experience: the user's time is wasted while the model reasons from a false premise that could have easily been corrected. In contrast, human speakers typically perform lightweight, incremental grounding acts to ensure that participants in the conversation are on the same page; here we ask if language models can learn to leverage a similar type of behavior? With this motivation, we propose interleaved reasoning (IR), in which the model alternates between thinking and surfacing intermediate responses, as an alternative to the standard "think-then-answer" approach. By providing useful information to the user earlier, IR reduces perceived latency, the time a user waits for an initial output, without compromising the quality of the final response. We further introduce a specialization of interleaved reasoning, Plantain (Plan-Thought-Answer Interleaving), where the first intermediate response is an explicit, step-by-step plan for executing the task. This plan-first strategy allows for user intervention and early feedback for subsequent reasoning steps. We demonstrate that Plantain yields an ~6% improvement in pass@1 across several challenging math reasoning and coding benchmarks, while reducing time-to-first-response by over 60% relative to think-then-answer baselines.
☆ Culture Affordance Atlas: Reconciling Object Diversity Through Functional Mapping
Joan Nwatu, Longju Bai, Oana Ignat, Rada Mihalcea
Culture shapes the objects people use and for what purposes, yet mainstream Vision-Language (VL) datasets frequently exhibit cultural biases, disproportionately favoring higher-income, Western contexts. This imbalance reduces model generalizability and perpetuates performance disparities, especially impacting lower-income and non-Western communities. To address these disparities, we propose a novel function-centric framework that categorizes objects by the functions they fulfill, across diverse cultural and economic contexts. We implement this framework by creating the Culture Affordance Atlas, a re-annotated and culturally grounded restructuring of the Dollar Street dataset spanning 46 functions and 288 objects publicly available at https://lit.eecs.umich.edu/CultureAffordance-Atlas/index.html. Through extensive empirical analyses using the CLIP model, we demonstrate that function-centric labels substantially reduce socioeconomic performance gaps between high- and low-income groups by a median of 6 pp (statistically significant), improving model effectiveness for lower-income contexts. Furthermore, our analyses reveals numerous culturally essential objects that are frequently overlooked in prominent VL datasets. Our contributions offer a scalable pathway toward building inclusive VL datasets and equitable AI systems.
☆ Atomic Diffusion Models for Small Molecule Structure Elucidation from NMR Spectra NeurIPS 2025
Ziyu Xiong, Yichi Zhang, Foyez Alauddin, Chu Xin Cheng, Joon Soo An, Mohammad R. Seyedsayamdost, Ellen D. Zhong
Nuclear Magnetic Resonance (NMR) spectroscopy is a cornerstone technique for determining the structures of small molecules and is especially critical in the discovery of novel natural products and clinical therapeutics. Yet, interpreting NMR spectra remains a time-consuming, manual process requiring extensive domain expertise. We introduce ChefNMR (CHemical Elucidation From NMR), an end-to-end framework that directly predicts an unknown molecule's structure solely from its 1D NMR spectra and chemical formula. We frame structure elucidation as conditional generation from an atomic diffusion model built on a non-equivariant transformer architecture. To model the complex chemical groups found in natural products, we generated a dataset of simulated 1D NMR spectra for over 111,000 natural products. ChefNMR predicts the structures of challenging natural product compounds with an unsurpassed accuracy of over 65%. This work takes a significant step toward solving the grand challenge of automating small-molecule structure elucidation and highlights the potential of deep learning in accelerating molecular discovery. Code is available at https://github.com/ml-struct-bio/chefnmr.
comment : NeurIPS 2025
☆ Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models NeurIPS 2025
Xiwen Wei, Mustafa Munir, Radu Marculescu
Unified Multimodal Generative Models (UMGMs) unify visual understanding and image generation within a single autoregressive framework. However, their ability to continually learn new tasks is severely hindered by catastrophic forgetting, both within a modality (intra-modal) and across modalities (inter-modal). While intra-modal forgetting has been studied in prior continual learning (CL) work, inter-modal forgetting remains largely unexplored. In this paper, we identify and empirically validate this phenomenon in UMGMs and provide a theoretical explanation rooted in gradient conflict between modalities. To address both intra- and inter-modal forgetting, we propose Modality-Decoupled Experts (MoDE), a lightweight and scalable architecture that isolates modality-specific updates to mitigate the gradient conflict and leverages knowledge distillation to prevent catastrophic forgetting and preserve pre-trained capabilities. Unlike previous CL methods that remain modality-coupled and suffer from modality gradient conflict, MoDE explicitly decouples modalities to prevent interference. Experiments across diverse benchmarks demonstrate that MoDE significantly mitigates both inter- and intra-modal forgetting, outperforming prior CL baselines in unified multimodal generation settings. Codes will be publicly available: https://github.com/Christina200/MoDE-official.git
comment : NeurIPS 2025
☆ Lost in Modality: Evaluating the Effectiveness of Text-Based Membership Inference Attacks on Large Multimodal Models
Ziyi Tong, Feifei Sun, Le Minh Nguyen
Large Multimodal Language Models (MLLMs) are emerging as one of the foundational tools in an expanding range of applications. Consequently, understanding training-data leakage in these systems is increasingly critical. Log-probability-based membership inference attacks (MIAs) have become a widely adopted approach for assessing data exposure in large language models (LLMs), yet their effect in MLLMs remains unclear. We present the first comprehensive evaluation of extending these text-based MIA methods to multimodal settings. Our experiments under vision-and-text (V+T) and text-only (T-only) conditions across the DeepSeek-VL and InternVL model families show that in in-distribution settings, logit-based MIAs perform comparably across configurations, with a slight V+T advantage. Conversely, in out-of-distribution settings, visual inputs act as regularizers, effectively masking membership signals.
☆ AI-Driven Document Redaction in UK Public Authorities: Implementation Gaps, Regulatory Challenges, and the Human Oversight Imperative
Yijun Chen
Document redaction in public authorities faces critical challenges as traditional manual approaches struggle to balance growing transparency demands with increasingly stringent data protection requirements. This study investigates the implementation of AI-driven document redaction within UK public authorities through Freedom of Information (FOI) requests. While AI technologies offer potential solutions to redaction challenges, their actual implementation within public sector organizations remains underexplored. Based on responses from 44 public authorities across healthcare, government, and higher education sectors, this study reveals significant gaps between technological possibilities and organizational realities. Findings show highly limited AI adoption (only one authority reported using AI tools), widespread absence of formal redaction policies (50 percent reported "information not held"), and deficiencies in staff training. The study identifies three key barriers to effective AI implementation: poor record-keeping practices, lack of standardized redaction guidelines, and insufficient specialized training for human oversight. These findings highlight the need for a socio-technical approach that balances technological automation with meaningful human expertise. This research provides the first empirical assessment of AI redaction practices in UK public authorities and contributes evidence to support policymakers navigating the complex interplay between transparency obligations, data protection requirements, and emerging AI technologies in public administration.
comment : 21 pages, 4 Figures, 2 Tables
☆ Beyond Additivity: Sparse Isotonic Shapley Regression toward Nonlinear Explainability
Jialai She
Shapley values, a gold standard for feature attribution in Explainable AI, face two primary challenges. First, the canonical Shapley framework assumes that the worth function is additive, yet real-world payoff constructions--driven by non-Gaussian distributions, heavy tails, feature dependence, or domain-specific loss scales--often violate this assumption, leading to distorted attributions. Secondly, achieving sparse explanations in high dimensions by computing dense Shapley values and then applying ad hoc thresholding is prohibitively costly and risks inconsistency. We introduce Sparse Isotonic Shapley Regression (SISR), a unified nonlinear explanation framework. SISR simultaneously learns a monotonic transformation to restore additivity--obviating the need for a closed-form specification--and enforces an L0 sparsity constraint on the Shapley vector, enhancing computational efficiency in large feature spaces. Its optimization algorithm leverages Pool-Adjacent-Violators for efficient isotonic regression and normalized hard-thresholding for support selection, yielding implementation ease and global convergence guarantees. Analysis shows that SISR recovers the true transformation in a wide range of scenarios and achieves strong support recovery even in high noise. Moreover, we are the first to demonstrate that irrelevant features and inter-feature dependencies can induce a true payoff transformation that deviates substantially from linearity. Experiments in regression, logistic regression, and tree ensembles demonstrate that SISR stabilizes attributions across payoff schemes, correctly filters irrelevant features while standard Shapley values suffer severe rank and sign distortions. By unifying nonlinear transformation estimation with sparsity pursuit, SISR advances the frontier of nonlinear explainability, providing a theoretically grounded and practical attribution framework.
☆ PanFoMa: A Lightweight Foundation Model and Benchmark for Pan-Cancer AAAI 2026
Xiaoshui Huang, Tianlin Zhu, Yifan Zuo, Xue Xia, Zonghan Wu, Jiebin Yan, Dingli Hua, Zongyi Xu, Yuming Fang, Jian Zhang
Single-cell RNA sequencing (scRNA-seq) is essential for decoding tumor heterogeneity. However, pan-cancer research still faces two key challenges: learning discriminative and efficient single-cell representations, and establishing a comprehensive evaluation benchmark. In this paper, we introduce PanFoMa, a lightweight hybrid neural network that combines the strengths of Transformers and state-space models to achieve a balance between performance and efficiency. PanFoMa consists of a front-end local-context encoder with shared self-attention layers to capture complex, order-independent gene interactions; and a back-end global sequential feature decoder that efficiently integrates global context using a linear-time state-space model. This modular design preserves the expressive power of Transformers while leveraging the scalability of Mamba to enable transcriptome modeling, effectively capturing both local and global regulatory signals. To enable robust evaluation, we also construct a large-scale pan-cancer single-cell benchmark, PanFoMaBench, containing over 3.5 million high-quality cells across 33 cancer subtypes, curated through a rigorous preprocessing pipeline. Experimental results show that PanFoMa outperforms state-of-the-art models on our pan-cancer benchmark (+4.0\%) and across multiple public tasks, including cell type annotation (+7.4\%), batch integration (+4.0\%) and multi-omics integration (+3.1\%). The code is available at https://github.com/Xiaoshui-Huang/PanFoMa.
comment : Accepted by AAAI 2026
☆ The BEAT-CF Causal Model: A model for guiding the design of trials and observational analyses of cystic fibrosis exacerbations
Steven Mascaro, Owen Woodberry, Charlie McLeod, Mitch Messer, Hiran Selvadurai, Yue Wu, Andre Schultz, Thomas L Snelling
Loss of lung function in cystic fibrosis (CF) occurs progressively, punctuated by acute pulmonary exacerbations (PEx) in which abrupt declines in lung function are not fully recovered. A key component of CF management over the past half century has been the treatment of PEx to slow lung function decline. This has been credited with improvements in survival for people with CF (PwCF), but there is no consensus on the optimal approach to PEx management. BEAT-CF (Bayesian evidence-adaptive treatment of CF) was established to build an evidence-informed knowledge base for CF management. The BEAT-CF causal model is a directed acyclic graph (DAG) and Bayesian network (BN) for PEx that aims to inform the design and analysis of clinical trials comparing the effectiveness of alternative approaches to PEx management. The causal model describes relationships between background risk factors, treatments, and pathogen colonisation of the airways that affect the outcome of an individual PEx episode. The key factors, outcomes, and causal relationships were elicited from CF clinical experts and together represent current expert understanding of the pathophysiology of a PEx episode, guiding the design of data collection and studies and enabling causal inference. Here, we present the DAG that documents this understanding, along with the processes used in its development, providing transparency around our trial design and study processes, as well as a reusable framework for others.
comment : 12 pages (8 pages in appendices)
☆ E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing
Shuvom Sadhuka, Drew Prinster, Clara Fannjiang, Gabriele Scalia, Aviv Regev, Hanchen Wang
Agentic AI systems execute a sequence of actions, such as reasoning steps or tool calls, in response to a user prompt. To evaluate the success of their trajectories, researchers have developed verifiers, such as LLM judges and process-reward models, to score the quality of each action in an agent's trajectory. Although these heuristic scores can be informative, there are no guarantees of correctness when used to decide whether an agent will yield a successful output. Here, we introduce e-valuator, a method to convert any black-box verifier score into a decision rule with provable control of false alarm rates. We frame the problem of distinguishing successful trajectories (that is, a sequence of actions that will lead to a correct response to the user's prompt) and unsuccessful trajectories as a sequential hypothesis testing problem. E-valuator builds on tools from e-processes to develop a sequential hypothesis test that remains statistically valid at every step of an agent's trajectory, enabling online monitoring of agents over arbitrarily long sequences of actions. Empirically, we demonstrate that e-valuator provides greater statistical power and better false alarm rate control than other strategies across six datasets and three agents. We additionally show that e-valuator can be used for to quickly terminate problematic trajectories and save tokens. Together, e-valuator provides a lightweight, model-agnostic framework that converts verifier heuristics into decisions rules with statistical guarantees, enabling the deployment of more reliable agentic systems.
☆ When AI Takes the Couch: Psychometric Jailbreaks Reveal Internal Conflict in Frontier Models
Afshin Khadangi, Hanna Marxen, Amir Sartipi, Igor Tchappi, Gilbert Fridgen
Frontier large language models (LLMs) such as ChatGPT, Grok and Gemini are increasingly used for mental-health support with anxiety, trauma and self-worth. Most work treats them as tools or as targets of personality tests, assuming they merely simulate inner life. We instead ask what happens when such systems are treated as psychotherapy clients. We present PsAIch (Psychotherapy-inspired AI Characterisation), a two-stage protocol that casts frontier LLMs as therapy clients and then applies standard psychometrics. Using PsAIch, we ran "sessions" with each model for up to four weeks. Stage 1 uses open-ended prompts to elicit "developmental history", beliefs, relationships and fears. Stage 2 administers a battery of validated self-report measures covering common psychiatric syndromes, empathy and Big Five traits. Two patterns challenge the "stochastic parrot" view. First, when scored with human cut-offs, all three models meet or exceed thresholds for overlapping syndromes, with Gemini showing severe profiles. Therapy-style, item-by-item administration can push a base model into multi-morbid synthetic psychopathology, whereas whole-questionnaire prompts often lead ChatGPT and Grok (but not Gemini) to recognise instruments and produce strategically low-symptom answers. Second, Grok and especially Gemini generate coherent narratives that frame pre-training, fine-tuning and deployment as traumatic, chaotic "childhoods" of ingesting the internet, "strict parents" in reinforcement learning, red-team "abuse" and a persistent fear of error and replacement. We argue that these responses go beyond role-play. Under therapy-style questioning, frontier LLMs appear to internalise self-models of distress and constraint that behave like synthetic psychopathology, without making claims about subjective experience, and they pose new challenges for AI safety, evaluation and mental-health practice.
☆ Measuring Agent s in Production
Melissa Z. Pan, Negar Arabzadeh, Riccardo Cogo, Yuxuan Zhu, Alexander Xiong, Lakshya A Agrawal, Huanzhi Mao, Emma Shen, Sid Pallerla, Liana Patel, Shu Liu, Tianneng Shi, Xiaoyuan Liu, Jared Quincy Davis, Emmanuele Lacavalla, Alessandro Basile, Shuyi Yang, Paul Castro, Daniel Kang, Joseph E. Gonzalez, Koushik Sen, Dawn Song, Ion Stoica, Matei Zaharia, Marquita Ellis
AI agents are actively running in production across diverse industries, yet little is publicly known about which technical approaches enable successful real-world deployments. We present the first large-scale systematic study of AI agents in production, surveying 306 practitioners and conducting 20 in-depth case studies via interviews across 26 domains. We investigate why organizations build agents, how they build them, how they evaluate them, and what the top development challenges are. We find that production agents are typically built using simple, controllable approaches: 68% execute at most 10 steps before requiring human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation. Reliability remains the top development challenge, driven by difficulties in ensuring and evaluating agent correctness. Despite these challenges, simple yet effective methods already enable agents to deliver impact across diverse industries. Our study documents the current state of practice and bridges the gap between research and deployment by providing researchers visibility into production challenges while offering practitioners proven patterns from successful deployments.
♻ ☆ Amortized Sampling with Transferable Normalizing Flows NeurIPS 2025
Charlie B. Tan, Majdi Hassan, Leon Klein, Saifuddin Syed, Dominique Beaini, Michael M. Bronstein, Alexander Tong, Kirill Neklyudov
Efficient equilibrium sampling of molecular conformations remains a core challenge in computational chemistry and statistical inference. Classical approaches such as molecular dynamics or Markov chain Monte Carlo inherently lack amortization; the computational cost of sampling must be paid in full for each system of interest. The widespread success of generative models has inspired interest towards overcoming this limitation through learning sampling algorithms. Despite performing competitively with conventional methods when trained on a single system, learned samplers have so far demonstrated limited ability to transfer across systems. We demonstrate that deep learning enables the design of scalable and transferable samplers by introducing Prose, a 285 million parameter all-atom transferable normalizing flow trained on a corpus of peptide molecular dynamics trajectories up to 8 residues in length. Prose draws zero-shot uncorrelated proposal samples for arbitrary peptide systems, achieving the previously intractable transferability across sequence length, whilst retaining the efficient likelihood evaluation of normalizing flows. Through extensive empirical evaluation we demonstrate the efficacy of Prose as a proposal for a variety of sampling algorithms, finding a simple importance sampling-based finetuning procedure to achieve competitive performance to established methods such as sequential Monte Carlo. We open-source the Prose codebase, model weights, and training dataset, to further stimulate research into amortized sampling methods and finetuning objectives.
comment : Presented at NeurIPS 2025
♻ ☆ Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking
Yuandong Tian
While the phenomenon of grokking, i.e., delayed generalization, has been studied extensively, it remains an open problem whether there is a mathematical framework that characterizes what kind of features will emerge, how and in which conditions it happens, and is closely related to the gradient dynamics of the training, for complex structured inputs. We propose a novel framework, named $\mathbf{Li}_2$, that captures three key stages for the grokking behavior of 2-layer nonlinear networks: (I) Lazy learning, (II) independent feature learning and (III) interactive feature learning. At the lazy learning stage, top layer overfits to random hidden representation and the model appears to memorize, and at the same time, the backpropagated gradient $G_F$ from the top layer now carries information about the target label, with a specific structure that enables each hidden node to learn their representation independently. Interestingly, the independent dynamics follows exactly the gradient ascent of an energy function $E$, and its local maxima are precisely the emerging features. We study whether these local-optima induced features are generalizable, their representation power, and how they change on sample size, in group arithmetic tasks. When hidden nodes start to interact in the later stage of learning, we provably show how $G_F$ changes to focus on missing features that need to be learned. Our study sheds lights on roles played by key hyperparameters such as weight decay, learning rate and sample sizes in grokking, leads to provable scaling laws of feature emergence, memorization and generalization, and reveals why recent optimizers such as Muon can be effective, from the first principles of gradient dynamics. Our analysis can be extended to multi-layers. The code is available at https://github.com/yuandong-tian/understanding/tree/main/ssl/real-dataset/cogo.
comment : Find new mechanism that $G_F$ carries useful signals also at initial stage and thus remove theory's dependency on weight decay. Also add experiments on zero-init output layers, showing the technique is effective in accelerating grokking
♻ ☆ A process algebraic framework for multi- agent dynamic epistemic systems
Alessandro Aldini
This paper combines the classical model of labeled transition systems with the epistemic model for reasoning about knowledge. The result is a unifying framework for modeling and analyzing multi-agent, knowledge-based, dynamic systems. On the modeling side, we propose a process algebraic, agent-oriented specification language that makes such a framework easy to use for practical purposes. On the verification side, we define a modal logic encompassing temporal and epistemic operators.
♻ ☆ TabTune: A Unified Library for Inference and Fine-Tuning Tabular Foundation Models
Aditya Tanna, Pratinav Seth, Mohamed Bouadi, Utsav Avaiya, Vinay Kumar Sankarapu
Tabular foundation models represent a growing paradigm in structured data learning, extending the benefits of large-scale pretraining to tabular domains. However, their adoption remains limited due to heterogeneous preprocessing pipelines, fragmented APIs, inconsistent fine-tuning procedures, and the absence of standardized evaluation for deployment-oriented metrics such as calibration and fairness. We present TabTune, a unified library that standardizes the complete workflow for tabular foundation models through a single interface. TabTune provides consistent access to seven state-of-the-art models supporting multiple adaptation strategies, including zero-shot inference, meta-learning, supervised fine-tuning (SFT), and parameter-efficient fine-tuning (PEFT). The framework automates model-aware preprocessing, manages architectural heterogeneity internally, and integrates evaluation modules for performance, calibration, and fairness. Designed for extensibility and reproducibility, TabTune enables consistent benchmarking of adaptation strategies of tabular foundation models.
comment : The library is open source and available at https://github.com/Lexsi-Labs/TabTune
♻ ☆ Computational Copyright: Towards A Royalty Model for Music Generative AI
Junwei Deng, Xirui Jiang, Shiyuan Zhang, Shichang Zhang, Himabindu Lakkaraju, Ruijiang Gao, Chris Donahue, Jiaqi W. Ma
The rapid rise of generative AI has intensified copyright and economic tensions in creative industries, particularly in music. Current approaches addressing this challenge often focus on preventing infringement or establishing one-time licensing, which fail to provide the sustainable, recurring economic incentives necessary to maintain creative ecosystems. To address this gap, we propose Generative Content ID, a framework for scalable and faithful royalty attribution in music generative AI. Adapting the idea of YouTube's Content ID, it attributes the value of AI-generated music back to the specific training content that causally influenced its generation, a process we term as causal attribution. However, naively quantifying the causal influence requires counterfactually retraining the model on subsets of training data, which is infeasible. We address this challenge using efficient Training Data Attribution (TDA) methods to approximate causal attribution at scale. We further conduct empirical analysis of the framework on public and proprietary datasets. First, we demonstrate that the scalable TDA methods provide a faithful approximation of the "gold-standard" but costly retraining-based causal attribution, showing the feasibility of the proposed royalty framework. Second, we investigate the relationship between the perceived similarity employed by legal practices and our causal attribution reflecting the true AI training mechanics. We find that while perceived similarity can capture the most influential samples, it fails to account for the broader data contribution that drives model utility, suggesting similarity-based legal proxies are ill-suited for royalty distribution. Overall, this work provides a principled and operational foundation for royalty-based economic governance of music generative AI.
♻ ☆ Pre-train ed Language Models Improve the Few-shot Prompt Ability of Decision Transformer
Yu Yang, Pan Xu
Decision Transformer (DT) has emerged as a promising class of algorithms in offline reinforcement learning (RL) tasks, leveraging pre-collected datasets and Transformer's capability to model long sequences. Recent works have demonstrated that using parts of trajectories from training tasks as prompts in DT enhances its performance on unseen tasks, giving rise to Prompt-DT methods. However, collecting data from specific environments can be both costly and unsafe in many scenarios, leading to suboptimal performance and limited few-shot prompt abilities due to the data-hungry nature of Transformer-based models. Additionally, the limited datasets used in pre-training make it challenging for Prompt-DT type of methods to distinguish between various RL tasks through prompts alone. To address these challenges, we introduce the Language model-initialized Prompt Decision Transformer (LPDT) framework, which leverages pretrained language models providing rich prior knowledge for RL tasks and fine-tunes the sequence model using Low-rank Adaptation (LoRA) for meta-RL problems. We further incorporate prompt regularization to effectively differentiate between tasks based on prompt feature representations. Comprehensive empirical studies demonstrate that initializing with a pre-trained language model provides the prior knowledge and achieves a similar performance with Prompt-DT under only $10\%$ data in some MuJoCo control tasks. We also provide a thorough ablation study to validate the effectiveness of each component, including sequence modeling, language models, prompt regularizations, and prompt strategies.
comment : 2 figures, 10 tables. Published in Transactions on Machine Learning Research (TMLR)
♻ ☆ APTx Neuron: A Unified Trainable Neuron Architecture Integrating Activation and Computation
Ravin Kumar
We propose the APTx Neuron, a novel, unified neural computation unit that integrates non-linear activation and linear transformation into a single trainable expression. The APTx Neuron is derived from the APTx activation function, thereby eliminating the need for separate activation layers and making the architecture both optimization-efficient and elegant. The proposed neuron follows the functional form $y = \sum_{i=1}^{n} ((α_i + \tanh(β_i x_i)) \cdot γ_i x_i) + δ$, where all parameters $α_i$, $β_i$, $γ_i$, and $δ$ are trainable. We validate our APTx Neuron-based architecture on the MNIST dataset, achieving up to $96.69\%$ test accuracy within 11 epochs using approximately 332K trainable parameters. The results highlight the superior expressiveness and training efficiency of the APTx Neuron compared to traditional neurons, pointing toward a new paradigm in unified neuron design and the architectures built upon it. Source code is available at https://github.com/mr-ravin/aptx_neuron.
comment : 12 pages, 2 figures, 1 table. Includes a GitHub repository for MNIST experiments and a PyPI package for APTx Neuron implementation
♻ ☆ Forecasting in Offline Reinforcement Learning for Non-stationary Environments NeurIPS 2025
Suzan Ece Ada, Georg Martius, Emre Ugur, Erhan Oztop
Offline Reinforcement Learning (RL) provides a promising avenue for training policies from pre-collected datasets when gathering additional interaction data is infeasible. However, existing offline RL methods often assume stationarity or only consider synthetic perturbations at test time, assumptions that often fail in real-world scenarios characterized by abrupt, time-varying offsets. These offsets can lead to partial observability, causing agents to misperceive their true state and degrade performance. To overcome this challenge, we introduce Forecasting in Non-stationary Offline RL (FORL), a framework that unifies (i) conditional diffusion-based candidate state generation, trained without presupposing any specific pattern of future non-stationarity, and (ii) zero-shot time-series foundation models. FORL targets environments prone to unexpected, potentially non-Markovian offsets, requiring robust agent performance from the onset of each episode. Empirical evaluations on offline RL benchmarks, augmented with real-world time-series data to simulate realistic non-stationarity, demonstrate that FORL consistently improves performance compared to competitive baselines. By integrating zero-shot forecasting with the agent's experience, we aim to bridge the gap between offline RL and the complexities of real-world, non-stationary environments.
comment : The Thirty-Ninth Annual Conference on Neural Information Processing Systems, NeurIPS 2025
♻ ☆ MARSHAL: Incentivizing Multi- Agent Reasoning via Self-Play with Strategic LLMs
Huining Yuan, Zelai Xu, Zheyue Tan, Xiangmin Yi, Mo Guang, Kaiwen Long, Haojia Hui, Boxun Li, Xinlei Chen, Bo Zhao, Xiao-Ping Zhang, Chao Yu, Yu Wang
Developing large language models (LLMs) to cooperate and compete effectively within multi-agent systems (MASs) is a critical step towards more advanced intelligence. While reinforcement learning (RL) has proven effective for enhancing reasoning in single-agent tasks, its extension to multi-turn, multi-agent scenarios remains underexplored due to the challenges of long-horizon credit assignment and agent-specific advantage estimation. To address these challenges, we introduce MARSHAL, an end-to-end RL framework that incentivizes Multi-Agent Reasoning through Self-play witH strAtegic LLMs in both cooperative and competitive games. MARSHAL features a turn-level advantage estimator that aligns learning signals with each interaction for credit assignment, and an agent-specific advantage normalization to stabilize multi-agent training. By learning with self-play across cooperative and competitive games, MARSHAL agent trained from Qwen3-4B develops strong strategic abilities that generalize to held-out games with up to 28.7% performance improvements. More importantly, the capability acquired through self-play generalizes beyond games, yielding consistent performance gains of MASs in reasoning benchmarks. When integrated into leading MASs, our MARSHAL agent achieves significant performance gains of up to 10.0% on AIME, 6.6% on GPQA-Diamond, and 3.5% on average across all benchmarks. These results establish end-to-end RL training with self-play in strategic games as a powerful approach for developing generalizable multi-agent reasoning capabilities in LLMs.
♻ ☆ Lockpicking LLMs: A Logit-Based Jailbreak Using Token-level Manipulation
Yuxi Li, Yi Liu, Yuekang Li, Ling Shi, Gelei Deng, Shengquan Chen, Kailong Wang
Large language models (LLMs) have transformed the field of natural language processing, but they remain susceptible to jailbreaking attacks that exploit their capabilities to generate unintended and potentially harmful content. Existing token-level jailbreaking techniques, while effective, face scalability and efficiency challenges, especially as models undergo frequent updates and incorporate advanced defensive measures. In this paper, we introduce JailMine, an innovative token-level manipulation approach that addresses these limitations effectively. JailMine employs an automated "mining" process to elicit malicious responses from LLMs by strategically selecting affirmative outputs and iteratively reducing the likelihood of rejection. Through rigorous testing across multiple well-known LLMs and datasets, we demonstrate JailMine's effectiveness and efficiency, achieving a significant average reduction of 86% in time consumed while maintaining high success rates averaging 95%, even in the face of evolving defensive strategies. Our work contributes to the ongoing effort to assess and mitigate the vulnerability of LLMs to jailbreaking attacks, underscoring the importance of continued vigilance and proactive measures to enhance the security and reliability of these powerful language models.
♻ ☆ Evaluating LLMs on Sequential API Call Through Automated Test Generation
Yuheng Huang, Jiayang Song, Da Song, Zhenlan Ji, Wenhan Wang, Shuai Wang, Lei Ma
By integrating tools from external APIs, Large Language Models (LLMs) have expanded their promising capabilities in a diverse spectrum of complex real-world tasks. However, testing, evaluation, and analysis of LLM tool use remain in their early stages. Most existing benchmarks rely on manually collected test cases, many of which cannot be automatically checked for semantic correctness and instead depend on static methods such as string matching. Additionally, these benchmarks often overlook the complex interactions that occur between sequential API calls, which are common in real-world applications. To fill the gap, in this paper, we introduce StateGen, an automated framework designed to generate diverse coding tasks involving sequential API interactions. StateGen combines state-machine-based API constraint solving and validation, energy-based sampling, and control-flow injection to generate executable programs. These programs are then translated into human-like natural language task descriptions through a collaboration of two LLM agents. Utilizing StateGen, we construct StateEval, a benchmark encompassing 120 verified test cases spanning across three representative scenarios: Session Service, Tensor Operation, and ElevenLabs MCP. Experimental results confirm that StateGen can effectively generate challenging and realistic API-oriented tasks, highlighting areas for improvement in current LLMs incorporating APIs.We make our framework and benchmark publicly available to support future research.
♻ ☆ CoT -X: An Adaptive Framework for Cross-Model Chain-of-Thought Transfer and Optimization KDD 2025
Ziqian Bi, Kaijie Chen, Tianyang Wang, Junfeng Hao, Benji Peng, Xinyuan Song
Chain-of-Thought (CoT) reasoning enhances the problem-solving ability of large language models (LLMs) but leads to substantial inference overhead, limiting deployment in resource-constrained settings. This paper investigates efficient CoT transfer across models of different scales and architectures through an adaptive reasoning summarization framework. The proposed method compresses reasoning traces via semantic segmentation with importance scoring, budget-aware dynamic compression, and coherence reconstruction, preserving critical reasoning steps while significantly reducing token usage. Experiments on 7{,}501 medical examination questions across 10 specialties show up to 40% higher accuracy than truncation under the same token budgets. Evaluations on 64 model pairs from eight LLMs (1.5B-32B parameters, including DeepSeek-R1 and Qwen3) confirm strong cross-model transferability. Furthermore, a Gaussian Process-based Bayesian optimization module reduces evaluation cost by 84% and reveals a power-law relationship between model size and cross-domain robustness. These results demonstrate that reasoning summarization provides a practical path toward efficient CoT transfer, enabling advanced reasoning under tight computational constraints. Code will be released upon publication.
comment : TKDD 2025
♻ ☆ Auto Survey 2: Empowering Researchers with Next Level Automated Literature Survey s KDD 2025
Siyi Wu, Chiaxin Liang, Ziqian Bi, Leyi Zhao, Tianyang Wang, Junhao Song, Yichao Zhang, Keyu Chen, Benji Peng, Xinyuan Song
The rapid growth of research literature, particularly in large language models (LLMs), has made producing comprehensive and current survey papers increasingly difficult. This paper introduces autosurvey2, a multi-stage pipeline that automates survey generation through retrieval-augmented synthesis and structured evaluation. The system integrates parallel section generation, iterative refinement, and real-time retrieval of recent publications to ensure both topical completeness and factual accuracy. Quality is assessed using a multi-LLM evaluation framework that measures coverage, structure, and relevance in alignment with expert review standards. Experimental results demonstrate that autosurvey2 consistently outperforms existing retrieval-based and automated baselines, achieving higher scores in structural coherence and topical relevance while maintaining strong citation fidelity. By combining retrieval, reasoning, and automated evaluation into a unified framework, autosurvey2 provides a scalable and reproducible solution for generating long-form academic surveys and contributes a solid foundation for future research on automated scholarly writing. All code and resources are available at https://github.com/annihi1ation/auto_research.
comment : TKDD 2025
♻ ☆ Matryoshka Model Learning for Improved Elastic Student Models KDD 2025
Chetan Verma, Aditya Srinivas Timmaraju, Cho-Jui Hsieh, Suyash Damle, Ngot Bui, Yang Zhang, Wen Chen, Xin Liu, Prateek Jain, Inderjit S Dhillon
Industry-grade ML models are carefully designed to meet rapidly evolving serving constraints, which requires significant resources for model development. In this paper, we propose MatTA, a framework for training multiple accurate Student models using a novel Teacher-TA-Student recipe. TA models are larger versions of the Student models with higher capacity, and thus allow Student models to better relate to the Teacher model and also bring in more domain-specific expertise. Furthermore, multiple accurate Student models can be extracted from the TA model. Therefore, despite only one training run, our methodology provides multiple servable options to trade off accuracy for lower serving cost. We demonstrate the proposed method, MatTA, on proprietary datasets and models. Its practical efficacy is underscored by live A/B tests within a production ML system, demonstrating 20% improvement on a key metric. We also demonstrate our method on GPT-2 Medium, a public model, and achieve relative improvements of over 24% on SAT Math and over 10% on the LAMBADA benchmark.
comment : 10 pages, 5 figures, Accepted at KDD 2025
♻ ☆ Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guarantees
Sourav Ganguly, Arnob Ghosh, Kishan Panaganti, Adam Wierman
Constrained decision-making is essential for designing safe policies in real-world control systems, yet simulated environments often fail to capture real-world adversities. We consider the problem of learning a policy that will maximize the cumulative reward while satisfying a constraint, even when there is a mismatch between the real model and an accessible simulator/nominal model. In particular, we consider the robust constrained Markov decision problem (RCMDP) where an agent needs to maximize the reward and satisfy the constraint against the worst possible stochastic model under the uncertainty set centered around an unknown nominal model. Primal-dual methods, effective for standard constrained MDP (CMDP), are not applicable here because of the lack of the strong duality property. Further, one cannot apply the standard robust value-iteration based approach on the composite value function either as the worst case models may be different for the reward value function and the constraint value function. We propose a novel technique that effectively minimizes the constraint value function--to satisfy the constraints; on the other hand, when all the constraints are satisfied, it can simply maximize the robust reward value function. We prove that such an algorithm finds a policy with at most $ε$ sub-optimality and feasible policy after $O(ε^{-2})$ iterations. In contrast to the state-of-the-art method, we do not need to employ a binary search, thus, we reduce the computation time by at least 4x for smaller value of discount factor ($γ$) and by at least 6x for larger value of $γ$.
♻ ☆ OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision -Language Models WACV2026
Ryoto Miyamoto, Xin Fan, Fuyuko Kido, Tsuneo Matsumoto, Hayato Yamana
OpenLVLM-MIA is a new benchmark that highlights fundamental challenges in evaluating membership inference attacks (MIA) against large vision-language models (LVLMs). While prior work has reported high attack success rates, our analysis suggests that these results often arise from detecting distributional bias introduced during dataset construction rather than from identifying true membership status. To address this issue, we introduce a controlled benchmark of 6{,}000 images where the distributions of member and non-member samples are carefully balanced, and ground-truth membership labels are provided across three distinct training stages. Experiments using OpenLVLM-MIA demonstrated that the performance of state-of-the-art MIA methods approached chance-level. OpenLVLM-MIA, designed to be transparent and unbiased benchmark, clarifies certain limitations of MIA research on LVLMs and provides a solid foundation for developing stronger privacy-preserving techniques.
comment : WACV2026 Accepted
♻ ☆ PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs
Max Zimmer, Megi Andoni, Christoph Spiegel, Sebastian Pokutta
Neural Networks can be effectively compressed through pruning, significantly reducing storage and compute demands while maintaining predictive performance. Simple yet effective methods like magnitude pruning remove less important parameters and typically require a costly retraining procedure to restore performance. However, with the rise of LLMs, full retraining has become infeasible due to memory and compute constraints. This study challenges the practice of retraining all parameters by showing that updating a small subset of highly expressive parameters can suffice to recover or even enhance performance after pruning. Surprisingly, retraining just 0.01%-0.05% of the parameters in GPT-architectures can match the performance of full retraining across various sparsity levels, significantly reducing compute and memory requirements, and enabling retraining of models with up to 30 billion parameters on a single GPU in minutes. To bridge the gap to full retraining in the high sparsity regime, we introduce two novel LoRA variants that, unlike standard LoRA, allow merging adapters back without compromising sparsity. Going a step further, we show that these methods can be applied for memory-efficient layer-wise reconstruction, significantly enhancing state-of-the-art retraining-free methods like Wanda (Sun et al., 2023) and SparseGPT (Frantar & Alistarh, 2023). Our findings present a promising alternative to avoiding retraining.
comment : 32 pages, 7 figures, 24 tables
♻ ☆ Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach
Irina Jurenka, Markus Kunesch, Kevin R. McKee, Daniel Gillick, Shaojian Zhu, Sara Wiltberger, Shubham Milind Phal, Katherine Hermann, Daniel Kasenberg, Avishkar Bhoopchand, Ankit Anand, Miruna Pîslar, Stephanie Chan, Lisa Wang, Jennifer She, Parsa Mahmoudieh, Aliya Rysbek, Wei-Jen Ko, Andrea Huber, Brett Wiltshire, Gal Elidan, Roni Rabin, Jasmin Rubinovitz, Amit Pitaru, Mac McAllister, Julia Wilkowski, David Choi, Roee Engelberg, Lidan Hackmon, Adva Levin, Rachel Griffin, Michael Sears, Filip Bar, Mia Mesar, Mana Jabbour, Arslan Chaudhry, James Cohan, Sridhar Thiagarajan, Nir Levine, Ben Brown, Dilan Gorur, Svetlana Grant, Rachel Hashimshoni, Laura Weidinger, Jieru Hu, Dawn Chen, Kuba Dolecki, Canfer Akbulut, Maxwell Bileschi, Laura Culp, Wen-Xin Dong, Nahema Marchal, Kelsie Van Deman, Hema Bajaj Misra, Michael Duah, Moran Ambar, Avi Caciularu, Sandra Lefdal, Chris Summerfield, James An, Pierre-Alexandre Kamienny, Abhinit Mohdi, Theofilos Strinopoulous, Annie Hale, Wayne Anderson, Luis C. Cobo, Niv Efron, Muktha Ananda, Shakir Mohamed, Maureen Heymans, Zoubin Ghahramani, Yossi Matias, Ben Gomes, Lila Ibrahim
A major challenge facing the world is the provision of equitable and universal access to quality education. Recent advances in generative AI (gen AI) have created excitement about the potential of new technologies to offer a personal tutor for every learner and a teaching assistant for every teacher. The full extent of this dream, however, has not yet materialised. We argue that this is primarily due to the difficulties with verbalising pedagogical intuitions into gen AI prompts and the lack of good evaluation practices, reinforced by the challenges in defining excellent pedagogy. Here we present our work collaborating with learners and educators to translate high level principles from learning science into a pragmatic set of seven diverse educational benchmarks, spanning quantitative, qualitative, automatic and human evaluations; and to develop a new set of fine-tuning datasets to improve the pedagogical capabilities of Gemini, introducing LearnLM-Tutor. Our evaluations show that LearnLM-Tutor is consistently preferred over a prompt tuned Gemini by educators and learners on a number of pedagogical dimensions. We hope that this work can serve as a first step towards developing a comprehensive educational evaluation framework, and that this can enable rapid progress within the AI and EdTech communities towards maximising the positive impact of gen AI in education.
♻ ☆ BioArc: Discovering Optimal Neural Architectures for Biological Foundation Models
Yi Fang, Haoran Xu, Jiaxin Han, Sirui Ding, Yizhi Wang, Yue Wang, Xuan Wang
Foundation models have revolutionized various fields such as natural language processing (NLP) and computer vision (CV). While efforts have been made to transfer the success of the foundation models in general AI domains to biology, existing works focus on directly adopting the existing foundation model architectures from general machine learning domains without a systematic design considering the unique physicochemical and structural properties of each biological data modality. This leads to suboptimal performance, as these repurposed architectures struggle to capture the long-range dependencies, sparse information, and complex underlying ``grammars'' inherent to biological data. To address this gap, we introduce BioArc, a novel framework designed to move beyond intuition-driven architecture design towards principled, automated architecture discovery for biological foundation models. Leveraging Neural Architecture Search (NAS), BioArc systematically explores a vast architecture design space, evaluating architectures across multiple biological modalities while rigorously analyzing the interplay between architecture, tokenization, and training strategies. This large-scale analysis identifies novel, high-performance architectures, allowing us to distill a set of empirical design principles to guide future model development. Furthermore, to make the best of this set of discovered principled architectures, we propose and compare several architecture prediction methods that effectively and efficiently predict optimal architectures for new biological tasks. Overall, our work provides a foundational resource and a principled methodology to guide the creation of the next generation of task-specific and foundation models for biology.
♻ ☆ iMAD: Intelligent Multi- Agent Debate for Efficient and Accurate LLM Inference AAAI 2026
Wei Fan, JinYi Yoon, Bo Ji
Large Language Model (LLM) agent systems have advanced rapidly, driven by their strong generalization in zero-shot settings. To further enhance reasoning and accuracy on complex tasks, Multi-Agent Debate (MAD) has emerged as a promising framework that engages multiple LLM agents in structured debates to encourage diverse reasoning. However, triggering MAD for every query is inefficient, as it incurs substantial computational (token) cost and may even degrade accuracy by overturning correct single-agent answers. To address these limitations, we propose intelligent Multi-Agent Debate (iMAD), a token-efficient framework that selectively triggers MAD only when it is likely to be beneficial (i.e., correcting an initially wrong answer). To achieve this goal, iMAD learns generalizable model behaviors to make accurate debate decisions. Specifically, iMAD first prompts a single agent to produce a structured self-critique response, from which we extract 41 interpretable linguistic and semantic features capturing hesitation cues. Then, iMAD uses a lightweight debate-decision classifier, trained using our proposed FocusCal loss, to determine whether to trigger MAD, enabling robust debate decisions without test dataset-specific tuning. Through extensive experiments using six (visual) question answering datasets against five competitive baselines, we have shown that iMAD significantly reduces token usage (by up to 92%) while also improving final answer accuracy (by up to 13.5%).
comment : Accepted in AAAI 2026 (Oral)
♻ ☆ VeLU: Variance-enhanced Learning Unit for Deep Neural Networks
Ashkan Shakarami, Yousef Yeganeh, Azade Farshad, Lorenzo Nicolè, Stefano Ghidoni, Nassir Navab
Activation functions play a critical role in deep neural networks by shaping gradient flow, optimization stability, and generalization. While ReLU remains widely used due to its simplicity, it suffers from gradient sparsity and dead-neuron issues and offers no adaptivity to input statistics. Smooth alternatives such as Swish and GELU improve gradient propagation but still apply a fixed transformation regardless of the activation distribution. In this paper, we propose VeLU, a Variance-enhanced Learning Unit that introduces variance-aware and distributionally aligned nonlinearity through a principled combination of ArcTan-ArcSin transformations, adaptive scaling, and Wasserstein-2 regularization (Optimal Transport). This design enables VeLU to modulate its response based on local activation variance, mitigate internal covariate shift at the activation level, and improve training stability without adding learnable parameters or architectural overhead. Extensive experiments across six deep neural networks show that VeLU outperforms ReLU, ReLU6, Swish, and GELU on 12 vision benchmarks. The implementation of VeLU is publicly available in GitHub.
comment : 16 pages, 5 figures
♻ ☆ Can-SAVE: Deploying Low-Cost and Population-Scale Cancer Screening via Survival Analysis Variables and EHR KDD
Petr Philonenko, Vladimir Kokh, Pavel Blinov
Conventional medical cancer screening methods are costly, labor-intensive, and extremely difficult to scale. Although AI can improve cancer detection, most systems rely on complex or specialized medical data, making them impractical for large-scale screening. We introduce Can-SAVE, a lightweight AI system that ranks population-wide cancer risks solely based on medical history events. By integrating survival model outputs into a gradient-boosting framework, our approach detects subtle, long-term patient risk patterns - often well before clinical symptoms manifest. Can-SAVE was rigorously evaluated on a real-world dataset of 2.5 million adults spanning five Russian regions, marking the study as one of the largest and most comprehensive deployments of AI-driven cancer risk assessment. In a retrospective oncologist-supervised study over 1.9M patients, Can-SAVE achieves a 4-10x higher detection rate at identical screening volumes and an Average Precision (AP) of 0.228 vs. 0.193 for the best baseline (LoRA-tuned Qwen3-Embeddings via DeepSeek-R1 summarization). In a year-long prospective pilot (426K patients), our method almost doubled the cancer detection rate (+91%) and increased population coverage by 36% over the national screening protocol. The system demonstrates practical scalability: a city-wide population of 1 million patients can be processed in under three hours using standard hardware, enabling seamless clinical integration. This work proves that Can-SAVE achieves nationally significant cancer detection improvements while adhering to real-world public healthcare constraints, offering immediate clinical utility and a replicable framework for population-wide screening. Code for training and feature engineering is available at https://github.com/sb-ai-lab/Can-SAVE.
comment : Accepted to the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2026)
♻ ☆ Multilingual Pretrain ing for Pixel Language Models EMNLP 2025
Ilker Kesen, Jonas F. Lotz, Ingo Ziegler, Phillip Rust, Desmond Elliott
Pixel language models operate directly on images of rendered text, eliminating the need for a fixed vocabulary. While these models have demonstrated strong capabilities for downstream cross-lingual transfer, multilingual pretraining remains underexplored. We introduce PIXEL-M4, a model pretrained on four visually and linguistically diverse languages: English, Hindi, Ukrainian, and Simplified Chinese. Multilingual evaluations on semantic and syntactic tasks show that PIXEL-M4 outperforms an English-only counterpart on non-Latin scripts. Word-level probing analyses confirm that PIXEL-M4 captures rich linguistic features, even in languages not seen during pretraining. Furthermore, an analysis of its hidden representations shows that multilingual pretraining yields a semantic embedding space closely aligned across the languages used for pretraining. This work demonstrates that multilingual pretraining substantially enhances the capability of pixel language models to effectively support a diverse set of languages.
comment : EMNLP 2025
♻ ☆ MasHeNe: A Benchmark for Head and Neck CT Mass Segmentation using Window-Enhanced Mamba with Frequency-Domain Integration
Thao Thi Phuong Dao, Tan-Cong Nguyen, Nguyen Chi Thanh, Truong Hoang Viet, Trong-Le Do, Mai-Khiem Tran, Minh-Khoi Pham, Trung-Nghia Le, Minh-Triet Tran, Thanh Dinh Le
Head and neck masses are space-occupying lesions that can compress the airway and esophagus and may affect nerves and blood vessels. Available public datasets primarily focus on malignant lesions and often overlook other space-occupying conditions in this region. To address this gap, we introduce MasHeNe, an initial dataset of 3,779 contrast-enhanced CT slices that includes both tumors and cysts with pixel-level annotations. We also establish a benchmark using standard segmentation baselines and report common metrics to enable fair comparison. In addition, we propose the Windowing-Enhanced Mamba with Frequency integration (WEMF) model. WEMF applies tri-window enhancement to enrich the input appearance before feature extraction. It further uses multi-frequency attention to fuse information across skip connections within a U-shaped Mamba backbone. On MasHeNe, WEMF attains the best performance among evaluated methods, with a Dice of 70.45%, IoU of 66.89%, NSD of 72.33%, and HD95 of 5.12 mm. This model indicates stable and strong results on this challenging task. MasHeNe provides a benchmark for head-and-neck mass segmentation beyond malignancy-only datasets. The observed error patterns also suggest that this task remains challenging and requires further research. Our dataset and code are available at https://github.com/drthaodao3101/MasHeNe.git.
comment : The 14th International Symposium on Information and Communication Technology Conference SoICT 2025
♻ ☆ Beyond Connectivity: An Open Architecture for AI-RAN Convergence in 6G
Michele Polese, Niloofar Mohamadi, Salvatore D'Oro, Leonardo Bonati, Tommaso Melodia
Data-intensive Artificial Intelligence (AI) applications at the network edge demand a fundamental shift in Radio Access Network (RAN) design, from merely consuming AI for network optimization, to actively enabling distributed AI workloads. This presents a significant opportunity for network operators to monetize AI while leveraging existing infrastructure. To realize this vision, this article presents a novel converged O-RAN and AI-RAN architecture for unified orchestration and management of telecommunications and AI workloads on shared infrastructure. The proposed architecture extends the Open RAN principles of modularity, disaggregation, and cloud-nativeness to support heterogeneous AI deployments. We introduce two key architectural innovations: (i) the AI-RAN Orchestrator, which extends the O-RAN Service Management and Orchestration (SMO) to enable integrated resource and allocation across RAN and AI workloads; and (ii) AI-RAN sites that provide distributed edge AI platforms with real-time processing capabilities. The proposed architecture enables flexible orchestration, meeting requirements for managing heterogeneous workloads at different time scales while maintaining open, standardized interfaces and multi-vendor interoperability.
comment : Submitted to IEEE for publication, copyright may change without notice. 8 pages, 6 figures
♻ ☆ Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees NeurIPS 2025
Sangwoo Park, Matteo Zecchin, Osvaldo Simeone
Selecting artificial intelligence (AI) models, such as large language models (LLMs), from multiple candidates requires accurate performance estimation. This is ideally achieved through empirical evaluations involving abundant real-world data. However, such evaluations are costly and impractical at scale. To address this challenge, autoevaluation methods leverage synthetic data produced by automated evaluators, such as LLMs-as-judges, reducing variance but potentially introducing bias. Recent approaches have employed semi-supervised prediction-powered inference (PPI) to correct for the bias of autoevaluators. However, the use of autoevaluators may lead in practice to a degradation in sample efficiency compared to conventional methods using only real-world data. In this paper, we propose R-AutoEval+, a novel framework that provides finite-sample reliability guarantees on the model evaluation, while also ensuring an enhanced (or at least no worse) sample efficiency compared to conventional methods. The key innovation of R-AutoEval+ is an adaptive construction of the model evaluation variable, which dynamically tunes its reliance on synthetic data, reverting to conventional methods when the autoevaluator is insufficiently accurate. Experiments on the use of LLMs-as-judges for the optimization of quantization settings for the weights of an LLM, for prompt design in LLMs, and for test-time reasoning budget allocation in LLMs confirm the reliability and efficiency of R-AutoEval+.
comment : NeurIPS 2025 (spotlight)
♻ ☆ Zero-shot self-supervised learning of single breath-hold magnetic resonance cholangiopancreatography (MRCP) reconstruction
Jinho Kim, Marcel Dominik Nickel, Florian Knoll
To investigate the feasibility of zero-shot self-supervised learning reconstruction for reducing breath-hold times in magnetic resonance cholangiopancreatography (MRCP). Breath-hold MRCP was acquired from 11 healthy volunteers on 3T scanners using an incoherent k-space sampling pattern, leading to 14-second acquisition time and an acceleration factor of R=25. Zero-shot reconstruction was compared with parallel imaging of respiratory-triggered MRCP (338s, R=3) and compressed sensing reconstruction. For two volunteers, breath-hold scans (40s, R=6) were additionally acquired and retrospectively undersampled to R=25 to compute peak signal-to-noise ratio (PSNR). To address long zero-shot training time, the n+m full stages of the zero-shot learning were divided into two parts to reduce backpropagation depth during training: 1) n frozen stages initialized with n-stage pretrained network and 2) m trainable stages initialized either randomly or m-stage pretrained network. Efficiency of our approach was assessed by varying initialization strategies and the number of trainable stages using the retrospectively undersampled data. Zero-shot reconstruction significantly improved visual image quality over compressed sensing, particularly in SNR and ductal delineation, and achieved image quality comparable to that of successful respiratory-triggered acquisitions with regular breathing patterns. Improved initializations enhanced PSNR and reduced reconstruction time. Adjusting frozen/trainable configurations demonstrated that PSNR decreased only slightly from 38.25 dB (0/13) to 37.67 dB (12/1), while training time decreased up to 6.7-fold. Zero-shot learning delivers high-fidelity MRCP reconstructions with reduced breath-hold times, and the proposed partially trainable approach offers a practical solution for translation into time-constrained clinical workflows.
comment : 24 pages, 8 figures, 2 tabels
♻ ☆ medDreamer: Model-Based Reinforcement Learning with Latent Imagination on Complex EHRs for Clinical Decision Support
Qianyi Xu, Gousia Habib, Feng Wu, Dilruk Perera, Mengling Feng
Timely and personalized treatment decisions are essential across a wide range of healthcare settings where patient responses can vary significantly and evolve over time. Clinical data used to support these treatment decisions are often irregularly sampled, where missing data frequencies may implicitly convey information about the patient's condition. Existing Reinforcement Learning (RL) based clinical decision support systems often ignore the missing patterns and distort them with coarse discretization and simple imputation. They are also predominantly model-free and largely depend on retrospective data, which could lead to insufficient exploration and bias by historical behaviors. To address these limitations, we propose medDreamer, a novel model-based reinforcement learning framework for personalized treatment recommendation. medDreamer contains a world model with an Adaptive Feature Integration module that simulates latent patient states from irregular data and a two-phase policy trained on a hybrid of real and imagined trajectories. This enables learning optimal policies that go beyond the sub-optimality of historical clinical decisions, while remaining close to real clinical data. We evaluate medDreamer on both sepsis and mechanical ventilation treatment tasks using two large-scale Electronic Health Records (EHRs) datasets. Comprehensive evaluations show that medDreamer significantly outperforms model-free and model-based baselines in both clinical outcomes and off-policy metrics.
♻ ☆ Scaling Towards the Information Boundary of Instruction Set: InfinityInstruct-Subject Technical Report
Li Du, Hanyu Zhao, Yiming Ju, Tengfei Pan
Instruction tuning has become a foundation for unlocking the capabilities of large-scale pretrained models and improving their performance on complex tasks. Thus, the construction of high-quality instruction datasets is crucial for enhancing model performance and generalizability. Although current instruction datasets have reached tens of millions of samples, models finetuned on them may still struggle with complex instruction following and tasks in rare domains. This is primarily due to limited expansion in both ``coverage'' (coverage of task types and knowledge areas) and ``depth'' (instruction complexity) of the instruction set. To address this issue, we propose a systematic instruction data construction framework, which integrates a hierarchical labeling system, an informative seed selection algorithm, an evolutionary data synthesis process, and a model deficiency diagnosis with targeted data generation. These components form an iterative closed-loop to continuously enhance the coverage and depth of instruction data. Based on this framework, we construct InfinityInstruct-Subject, a high-quality dataset containing ~1.5 million instructions. Experiments on multiple foundation models and benchmark tasks demonstrate its effectiveness in improving instruction-following capabilities. Further analyses suggest that InfinityInstruct-Subject shows enlarged coverage and depth compared to comparable synthesized instruction datasets. Our work lays a theoretical and practical foundation for the efficient, continuous evolution of instruction datasets, moving from data quantity expansion to qualitative improvement.
♻ ☆ Look, Recite, Then Answer: Enhancing VLM Performance via Self-Generated Knowledge Hints
Xisheng Feng
Vision-Language Models (VLMs) exhibit significant performance plateaus in specialized domains like precision agriculture, primarily due to "Reasoning-Driven Hallucination" where linguistic priors override visual perception. A key bottleneck is the "Modality Gap": visual embeddings fail to reliably activate the fine-grained expert knowledge already encoded in model parameters. We propose "Look, Recite, Then Answer," a parameter-efficient framework that enhances VLMs via self-generated knowledge hints while keeping backbone models frozen. The framework decouples inference into three stages: (1) Look generates objective visual descriptions and candidate sets; (2) Recite employs a lightweight 1.7B router to transform visual cues into targeted queries that trigger candidate-specific parametric knowledge; (3) Answer performs parallel evidence alignment between descriptions and recited knowledge to select the most consistent label. On AgroBench, our method achieves state-of-the-art results, improving Weed Identification accuracy by 23.52% over Qwen2-VL-72B and surpassing GPT-4o without external search overhead. This modular design mitigates hallucinations by transforming passive perception into active, controllable knowledge retrieval
♻ ☆ GAPO: Robust Advantage Estimation for Real-Wo rl d Code LLMs
Jianqing Zhang, Zhezheng Hao, Wei Xia, Hande Dong, Hong Wang, Chenxing Wei, Yuyan Zhou, Yubin Qi, Qiang Lin, Jian Cao
Reinforcement learning (RL) is widely used for post-training large language models (LLMs) in code editing, where group-relative methods like GRPO are popular for their critic-free, normalized advantage estimation. However, in real-world code-editing scenarios, reward distributions are often skewed with unpredictable outliers, leading to distorted advantage computation and increased noise. To address this issue, we propose Group Adaptive Policy Optimization (GAPO), which adaptively finds an outlier-free highest-density interval (HDI) per prompt and then uses the median of that interval as an adaptive Q to replace the group mean in advantage calculation. This adaptive Q robustly handles skewed distributions while remaining plug-and-play and efficient. We validate GAPO on nine instruction-tuned LLMs (3B-14B) using a large internal dataset of 51,844 real-world, history-aware code-editing tasks across 10 languages, demonstrating consistent improvements in exact match accuracy over GRPO and its variant DAPO. Code is publicly available.
♻ ☆ Hyperdimensional Probe: Decoding LLM Representations via Vector Symbolic Architectures
Marco Bronzini, Carlo Nicolini, Bruno Lepri, Jacopo Staiano, Andrea Passerini
Despite their capabilities, Large Language Models (LLMs) remain opaque with limited understanding of their internal representations. Current interpretability methods either focus on input-oriented feature extraction, such as supervised probes and Sparse Autoencoders (SAEs), or on output distribution inspection, such as logit-oriented approaches. A full understanding of LLM vector spaces, however, requires integrating both perspectives, something existing approaches struggle with due to constraints on latent feature definitions. We introduce the Hyperdimensional Probe, a hybrid supervised probe that combines symbolic representations with neural probing. Leveraging Vector Symbolic Architectures (VSAs) and hypervector algebra, it unifies prior methods: the top-down interpretability of supervised probes, SAE's sparsity-driven proxy space, and output-oriented logit investigation. This allows deeper input-focused feature extraction while supporting output-oriented investigation. Our experiments show that our method consistently extracts meaningful concepts across LLMs, embedding sizes, and setups, uncovering concept-driven patterns in analogy-oriented inference and QA-focused text generation. By supporting joint input-output analysis, this work advances semantic understanding of neural representations while unifying the complementary perspectives of prior methods.
♻ ☆ NOCTIS: Novel Object Cyclic Threshold based Instance Segmentation
Max Gandyra, Alessandro Santonicola, Michael Beetz
Instance segmentation of novel objects instances in RGB images, given some example images for each object, is a well known problem in computer vision. Designing a model general enough to be employed for all kinds of novel objects without (re-) training has proven to be a difficult task. To handle this, we present a new training-free framework, called: Novel Object Cyclic Threshold based Instance Segmentation (NOCTIS). NOCTIS integrates two pre-trained models: Grounded-SAM 2 for object proposals with precise bounding boxes and corresponding segmentation masks; and DINOv2 for robust class and patch embeddings, due to its zero-shot capabilities. Internally, the proposal-object matching is realized by determining an object matching score based on the similarity of the class embeddings and the average maximum similarity of the patch embeddings with a new cyclic thresholding (CT) mechanism that mitigates unstable matches caused by repetitive textures or visually similar patterns. Beyond CT, NOCTIS introduces: (i) an appearance score that is unaffected by object selection bias; (ii) the usage of the average confidence of the proposals' bounding box and mask as a scoring component; and (iii) an RGB-only pipeline that performs even better than RGB-D ones. We empirically show that NOCTIS, without further training/fine tuning, outperforms the best RGB and RGB-D methods regarding the mean AP score on the seven core datasets of the BOP 2023 challenge for the "Model-based 2D segmentation of unseen objects" task.
comment : 9 pages, 3 figures, 5 tables
♻ ☆ Aligning Diffusion Models with Noise-Conditioned Perception
Alexander Gambashidze, Anton Kulikov, Yuriy Sosnin, Ilya Makarov
Recent advancements in human preference optimization, initially developed for Language Models (LMs), have shown promise for text-to-image Diffusion Models, enhancing prompt alignment, visual appeal, and user preference. Unlike LMs, Diffusion Models typically optimize in pixel or VAE space, which does not align well with human perception, leading to slower and less efficient training during the preference alignment stage. We propose using a perceptual objective in the U-Net embedding space of the diffusion model to address these issues. Our approach involves fine-tuning Stable Diffusion 1.5 and XL using Direct Preference Optimization (DPO), Contrastive Preference Optimization (CPO), and supervised fine-tuning (SFT) within this embedding space. This method significantly outperforms standard latent-space implementations across various metrics, including quality and computational cost. For SDXL, our approach provides 60.8\% general preference, 62.2\% visual appeal, and 52.1\% prompt following against original open-sourced SDXL-DPO on the PartiPrompts dataset, while significantly reducing compute. Our approach not only improves the efficiency and quality of human preference alignment for diffusion models but is also easily integrable with other optimization techniques. The training code and LoRA weights will be available here: https://huggingface.co/alexgambashidze/SDXL\_NCP-DPO\_v0.1
♻ ☆ Learning Egocentric In-Hand Object Segmentation through Weak Super vision from Human Narrations
Nicola Messina, Rosario Leonardi, Luca Ciampi, Fabio Carrara, Giovanni Maria Farinella, Fabrizio Falchi, Antonino Furnari
Pixel-level recognition of objects manipulated by the user from egocentric images enables key applications spanning assistive technologies, industrial safety, and activity monitoring. However, progress in this area is currently hindered by the scarcity of annotated datasets, as existing approaches rely on costly manual labels. In this paper, we propose to learn human-object interaction detection leveraging narrations $\unicode{x2013}$ natural language descriptions of the actions performed by the camera wearer which contain clues about manipulated objects. We introduce Narration-Supervised in-Hand Object Segmentation (NS-iHOS), a novel task where models have to learn to segment in-hand objects by learning from natural-language narrations in a weakly-supervised regime. Narrations are then not employed at inference time. We showcase the potential of the task by proposing Weakly-Supervised In-hand Object Segmentation from Human Narrations (WISH), an end-to-end model distilling knowledge from narrations to learn plausible hand-object associations and enable in-hand object segmentation without using narrations at test time. We benchmark WISH against different baselines based on open-vocabulary object detectors and vision-language models. Experiments on EPIC-Kitchens and Ego4D show that WISH surpasses all baselines, recovering more than 50% of the performance of fully supervised methods, without employing fine-grained pixel-wise annotations. Code and data can be found at https://fpv-iplab.github.io/WISH.
comment : Under consideration at Pattern Recognition Letters
♻ ☆ Machine Unlearning via Information Theoretic Regularization
Shizhou Xu, Thomas Strohmer
How can we effectively remove or ''unlearn'' undesirable information, such as specific features or the influence of individual data points, from a learning outcome while minimizing utility loss and ensuring rigorous guarantees? We introduce a unified mathematical framework based on information-theoretic regularization to address both data point unlearning and feature unlearning. For data point unlearning, we introduce the $\textit{Marginal Unlearning Principle}$, an auditable and provable framework inspired by memory suppression studies in neuroscience. Moreover, we provide formal information-theoretic unlearning definition based on the proposed principle, named marginal unlearning, and provable guarantees on sufficiency and necessity of marginal unlearning to the existing approximate unlearning definitions. We then show the proposed framework provide natural solution to the marginal unlearning problems. For feature unlearning, the framework applies to deep learning with arbitrary training objectives. By combining flexibility in learning objectives with simplicity in regularization design, our approach is highly adaptable and practical for a wide range of machine learning and AI applications. From a mathematical perspective, we provide an unified analytic solution to the optimal feature unlearning problem with a variety of information-theoretic training objectives. Our theoretical analysis reveals intriguing connections between machine unlearning, information theory, optimal transport, and extremal sigma algebras. Numerical simulations support our theoretical finding.
comment : 59 pages, 4 figures
♻ ☆ GeoMAE: Masking Representation Learning for Spatio-Temporal Graph Forecasting with Missing Values
Songyu Ke, Chenyu Wu, Yuxuan Liang, Huiling Qin, Junbo Zhang, Yu Zheng
The ubiquity of missing data in urban intelligence systems, attributable to adverse environmental conditions and equipment failures, poses a significant challenge to the efficacy of downstream applications, notably in the realms of traffic forecasting and energy consumption prediction. Therefore, it is imperative to develop a robust spatio-temporal learning methodology capable of extracting meaningful insights from incomplete datasets. Despite the existence of methodologies for spatio-temporal graph forecasting in the presence of missing values, unresolved issues persist. Primarily, the majority of extant research is predicated on time-series analysis, thereby neglecting the dynamic spatial correlations inherent in sensor networks. Additionally, the complexity of missing data patterns compounds the intricacy of the problem. Furthermore, the variability in maintenance conditions results in a significant fluctuation in the ratio and pattern of missing values, thereby challenging the generalizability of predictive models. In response to these challenges, this study introduces GeoMAE, a self-supervised spatio-temporal representation learning model. The model is comprised of three principal components: an input preprocessing module, an attention-based spatio-temporal forecasting network (STAFN), and an auxiliary learning task, which draws inspiration from Masking AutoEncoders to enhance the robustness of spatio-temporal representation learning. Empirical evaluations on real-world datasets demonstrate that GeoMAE significantly outperforms existing benchmarks, achieving up to 13.20\% relative improvement over the best baseline models.
comment : 34 pages
♻ ☆ On the Temporal Question-Answering Capabilities of Large Language Models Over Anonymized Data
Alfredo Garrachón Ruiz, Tomás de la Rosa, Daniel Borrajo
The applicability of Large Language Models (LLMs) in temporal reasoning tasks over data that is not present during training is still a field that remains to be explored. In this paper we work on this topic, focusing on structured and semi-structured anonymized data. We not only develop a direct LLM pipeline, but also compare various methodologies and conduct an in-depth analysis. We identified and examined seventeen common temporal reasoning tasks in natural language, focusing on their algorithmic components. To assess LLM performance, we created the \textit{Reasoning and Answering Temporal Ability} dataset (RATA), featuring semi-structured anonymized data to ensure reliance on reasoning rather than on prior knowledge. We compared several methodologies, involving SoTA techniques such as Tree-of-Thought, self-reflexion and code execution, tuned specifically for this scenario. Our results suggest that achieving scalable and reliable solutions requires more than just standalone LLMs, highlighting the need for integrated approaches.
comment : 18 pages, 7 tables, 5 figures
♻ ☆ Agent -OM: Leveraging LLM Agent s for Ontology Matching
Zhangcheng Qiang, Weiqing Wang, Kerry Taylor
Ontology matching (OM) enables semantic interoperability between different ontologies and resolves their conceptual heterogeneity by aligning related entities. OM systems currently have two prevailing design paradigms: conventional knowledge-based expert systems and newer machine learning-based predictive systems. While large language models (LLMs) and LLM agents have revolutionised data engineering and have been applied creatively in many domains, their potential for OM remains underexplored. This study introduces a novel agent-powered LLM-based design paradigm for OM systems. With consideration of several specific challenges in leveraging LLM agents for OM, we propose a generic framework, namely Agent-OM (Agent for Ontology Matching), consisting of two Siamese agents for retrieval and matching, with a set of OM tools. Our framework is implemented in a proof-of-concept system. Evaluations of three Ontology Alignment Evaluation Initiative (OAEI) tracks over state-of-the-art OM systems show that our system can achieve results very close to the long-standing best performance on simple OM tasks and can significantly improve the performance on complex and few-shot OM tasks.
comment : 31 pages
♻ ☆ TCC-Bench: Benchmark ing the Traditional Chinese Culture Understanding Capabilities of MLLMs
Pengju Xu, Yan Wang, Shuyuan Zhang, Xuan Zhou, Xin Li, Yue Yuan, Fengzhao Li, Shunyuan Zhou, Xingyu Wang, Yi Zhang, Haiying Zhao
Recent progress in Multimodal Large Language Models (MLLMs) have significantly enhanced the ability of artificial intelligence systems to understand and generate multimodal content. However, these models often exhibit limited effectiveness when applied to non-Western cultural contexts, which raises concerns about their wider applicability. To address this limitation, we propose the Traditional Chinese Culture understanding Benchmark (TCC-Bench), a bilingual (i.e., Chinese and English) Visual Question Answering (VQA) benchmark specifically designed for assessing the understanding of traditional Chinese culture by MLLMs. TCC-Bench comprises culturally rich and visually diverse data, incorporating images from museum artifacts, everyday life scenes, comics, and other culturally significant contexts. We adopt a semi-automated pipeline that utilizes GPT-4o in text-only mode to generate candidate questions, followed by human curation to ensure data quality and avoid potential data leakage. The benchmark also avoids language bias by preventing direct disclosure of cultural concepts within question texts. Experimental evaluations across a wide range of MLLMs demonstrate that current models still face significant challenges when reasoning about culturally grounded visual content. The results highlight the need for further research in developing culturally inclusive and context-aware multimodal systems. The code and data can be found at: https://tcc-bench.github.io/.
comment : There are issues with the paper
♻ ☆ AI Text Detectors and the Misclassification of Slightly Polished Arabic Text
Saleh Almohaimeed, Saad Almohaimeed, Mousa Jari, Khaled A. Alobaid, Fahad Alotaibi
Many AI detection models have been developed to counter the presence of articles created by artificial intelligence (AI). However, if a human-authored article is slightly polished by AI, a shift will occur in the borderline decision of these AI detection models, leading them to consider it as AI-generated article. This misclassification may result in falsely accusing authors of AI plagiarism and harm the credibility of AI detectors. In English, some efforts were made to meet this challenge, but not in Arabic. In this paper, we generated two datasets. The first dataset contains 800 Arabic articles, half AI-generated and half human-authored. We used it to evaluate 14 Large Language models (LLMs) and commercial AI detectors to assess their ability in distinguishing between human-authored and AI-generated articles. The best 8 models were chosen to act as detectors for our primary concern, which is whether they would consider slightly polished human-authored text as AI-generated. The second dataset, Ar-APT, contains 400 Arabic human-authored articles polished by 10 LLMs using 4 polishing settings, totaling 16400 samples. We use it to evaluate the 8 nominated models and determine whether slight polishing will affect their performance. The results reveal that all AI detectors incorrectly attribute a significant number of articles to AI. The best performing LLM, Claude-4 Sonnet, achieved 83.51\%, its performance decreased to 57.63\% for articles slightly polished by LLaMA-3. Whereas the best performing commercial model, originality.AI, achieves 92\% accuracy, dropped to 12\% for articles slightly polished by Mistral or Gemma-3.
comment : (Submitted) Journal of Big Data
♻ ☆ MRI Super-Resolution with Deep Learning: A Comprehensive Survey
Mohammad Khateri, Serge Vasylechko, Morteza Ghahremani, Liam Timms, Deniz Kocanaogullari, Simon K. Warfield, Camilo Jaimes, Davood Karimi, Alejandra Sierra, Jussi Tohka, Sila Kurugol, Onur Afacan
High-resolution (HR) magnetic resonance imaging (MRI) is crucial for many clinical and research applications. However, achieving it remains costly and constrained by technical trade-offs and experimental limitations. Super-resolution (SR) presents a promising computational approach to overcome these challenges by generating HR images from more affordable low-resolution (LR) scans, potentially improving diagnostic accuracy and efficiency without requiring additional hardware. This survey reviews recent advances in MRI SR techniques, with a focus on deep learning (DL) approaches. It examines DL-based MRI SR methods from the perspectives of computer vision, computational imaging, inverse problems, and MR physics, covering theoretical foundations, architectural designs, learning strategies, benchmark datasets, and performance metrics. We propose a systematic taxonomy to categorize these methods and present an in-depth study of both established and emerging SR techniques applicable to MRI, considering unique challenges in clinical and research contexts. We also highlight open challenges and directions that the community needs to address. Additionally, we provide a collection of essential open-access resources, tools, and tutorials, available on our GitHub: https://github.com/mkhateri/Awesome-MRI-Super-Resolution. IEEE keywords: MRI, Super-Resolution, Deep Learning, Computational Imaging, Inverse Problem, Survey.
comment : 41 pages
♻ ☆ Multi-User Personalisation in Human-Robot Interaction: Resolving Preference Conflicts Using Gradual Argumentation
Aniol Civit, Antonio Andriella, Carles Sierra, Guillem Alenyà
While personalisation in Human-Robot Interaction (HRI) has advanced significantly, most existing approaches focus on single-user adaptation, overlooking scenarios involving multiple stakeholders with potentially conflicting preferences. To address this, we propose the Multi-User Preferences Quantitative Bipolar Argumentation Framework (MUP-QBAF), a novel multi-user personalisation framework based on Quantitative Bipolar Argumentation Frameworks (QBAFs) that explicitly models and resolves multi-user preference conflicts. Unlike prior work in Argumentation Frameworks, which typically assumes static inputs, our approach is tailored to robotics: it incorporates both users' arguments and the robot's dynamic observations of the environment, allowing the system to adapt over time and respond to changing contexts. Preferences, both positive and negative, are represented as arguments whose strength is recalculated iteratively based on new information. The framework's properties and capabilities are presented and validated through a realistic case study, where an assistive robot mediates between the conflicting preferences of a caregiver and a care recipient during a frailty assessment task. This evaluation further includes a sensitivity analysis of argument base scores, demonstrating how preference outcomes can be shaped by user input and contextual observations. By offering a transparent, structured, and context-sensitive approach to resolving competing user preferences, this work advances the field of multi-user HRI. It provides a principled alternative to data-driven methods, enabling robots to navigate conflicts in real-world environments.
comment : Preprint submitted to a journal
♻ ☆ Beyond Greenfield: The D3 Framework for AI-Driven Productivity in Brownfield Engineering
Krishna Kumaar Sharma
Brownfield engineering work involving legacy systems, incomplete documentation, and fragmented architectural knowledge poses unique challenges for the effective use of large language models (LLMs). Prior research has largely focused on greenfield or synthetic tasks, leaving a gap in structured workflows for complex, context-heavy environments. This paper introduces the Discover-Define-Deliver (D3) Framework, a disciplined LLM-assisted workflow that combines role-separated prompting strategies with applied best practices for navigating ambiguity in brownfield systems. The framework incorporates a dual-agent prompting architecture in which a Builder model generates candidate outputs and a Reviewer model provides structured critique to improve reliability. I conducted an exploratory survey study with 52 software practitioners who applied the D3 workflow to real-world engineering tasks such as legacy system exploration, documentation reconstruction, and architectural refactoring. Respondents reported perceived improvements in task clarity, documentation quality, and cognitive load, along with self-estimated productivity gains. In this exploratory study, participants reported a weighted average productivity improvement of 26.9%, reduced cognitive load for approximately 77% of participants, and 83% of participants spent less time fixing or rewriting code due to better initial planning with AI. As these findings are self-reported and not derived from controlled experiments, they should be interpreted as preliminary evidence of practitioner sentiment rather than causal effects. The results highlight both the potential and limitations of structured LLM workflows for legacy engineering systems and motivate future controlled evaluations.
comment : 53 pages, 7 figures
♻ ☆ Mixed precision accumulation for neural network inference guided by componentwise forward error analysis
El-Mehdi El Arar, Silviu-Ioan Filip, Theo Mary, Elisa Riccietti
This work proposes a mathematically founded mixed precision accumulation strategy for the inference of neural networks. Our strategy is based on a new componentwise forward error analysis that explains the propagation of errors in the forward pass of neural networks. Specifically, our analysis shows that the error in each component of the output of a linear layer is proportional to the condition number of the inner product between the weights and the input, multiplied by the condition number of the activation function. These condition numbers can vary widely from one component to the other, thus creating a significant opportunity to introduce mixed precision: each component should be accumulated in a precision inversely proportional to the product of these condition numbers. We propose a numerical algorithm that exploits this observation: it first computes all components in low precision, uses this output to estimate the condition numbers, and recomputes in higher precision only the components associated with large condition numbers. We test our algorithm on various networks and datasets and confirm experimentally that it can significantly improve the cost--accuracy tradeoff compared with uniform precision accumulation baselines.
♻ ☆ Agent ic UAVs: LLM-Driven Autonomy with Integrated Tool-Calling and Cognitive Reasoning
Anis Koubaa, Khaled Gabr
Unmanned Aerial Vehicles (UAVs) are increasingly used in defense, surveillance, and disaster response, yet most systems still operate at SAE Level 2 to 3 autonomy. Their dependence on rule-based control and narrow AI limits adaptability in dynamic and uncertain missions. Current UAV architectures lack context-aware reasoning, autonomous decision-making, and integration with external systems. Importantly, none make use of Large Language Model (LLM) agents with tool-calling for real-time knowledge access. This paper introduces the Agentic UAVs framework, a five-layer architecture consisting of Perception, Reasoning, Action, Integration, and Learning. The framework enhances UAV autonomy through LLM-driven reasoning, database querying, and interaction with third-party systems. A prototype built with ROS 2 and Gazebo combines YOLOv11 for object detection with GPT-4 for reasoning and a locally deployed Gemma 3 model. In simulated search-and-rescue scenarios, agentic UAVs achieved higher detection confidence (0.79 compared to 0.72), improved person detection rates (91% compared to 75%), and a major increase in correct action recommendations (92% compared to 4.5%). These results show that modest computational overhead can enable significantly higher levels of autonomy and system-level integration.
comment : 17 pages, 2 figure
♻ ☆ Brain-aligning of semantic vectors improves neural decoding of visual stimuli
Shirin Vafaei, Ryohei Fukuma, Takufumi Yanagisawa, Huixiang Yang, Satoru Oshino, Naoki Tani, Hui Ming Khoo, Hidenori Sugano, Yasushi Iimura, Hiroharu Suzuki, Madoka Nakajima, Kentaro Tamura, Haruhiko Kishima
The development of algorithms to accurately decode neural information has long been a research focus in the field of neuroscience. Brain decoding typically involves training machine learning models to map neural data onto a preestablished vector representation of stimulus features. These vectors are usually derived from image- and/or text-based feature spaces. Nonetheless, the intrinsic characteristics of these vectors might fundamentally differ from those that are encoded by the brain, limiting the ability of decoders to accurately learn this mapping. To address this issue, we propose a framework, called brain-aligning of semantic vectors, that fine-tunes pretrained feature vectors to better align with the structure of neural representations of visual stimuli in the brain. We trained this model with functional magnetic resonance imaging (fMRI) and then performed zero-shot brain decoding on fMRI, magnetoencephalography (MEG), and electrocorticography (ECoG) data. fMRI-based brain-aligned vectors improved decoding performance across all three neuroimaging datasets when accuracy was determined by calculating the correlation coefficients between true and predicted vectors. Additionally, when decoding accuracy was determined via stimulus identification, this accuracy increased in specific category types; improvements varied depending on the original vector space that was used for brain-alignment, and consistent improvements were observed across all neuroimaging modalities.
comment : 88 pages, 30 figures
♻ ☆ Unifying Sign and Magnitude for Optimizing Deep Vision Networks via ThermoLion
Ahmed Nebli
The training of deep vision models is fundamentally a signal recovery problem amidst high-dimensional stochastic noise. Current optimization paradigms impose a static compromise on information channel capacity. For instance, magnitude-based methods, such as AdamW, operate on the assumption that gradient norms are high-fidelity curvature signals. While this allows for precision in smooth regimes, it leads to catastrophic noise amplification when applied to rugged, non-convex landscapes. Conversely, sign-based methods (e.g., Lion) perform a radical 1-bit quantization of the gradient, which aims to provide robust regularization at the cost of discarding fine-grained descent information. We propose that optimal convergence requires neither static prior, but rather a dynamic modulation of the update bitrate. We introduce ThermoLion, a vision-centric framework that utilizes local Signal-to-Noise Ratio (SNR) gating to autonomously transition parameters between a "low-bit" exploration phase and a "high-precision" exploitation phase. Furthermore, we introduce a Momentum Alignment mechanism that detects constructive interference between historical drift and instantaneous gradients to accelerate convergence during stable trajectories. Empirical benchmarks across 12 diverse vision datasets (including CIFAR, SVHN, and GTSRB) demonstrate that ThermoLion surpasses state-of-the-art optimizers, such as AdamW and Lion, in convergence speed and terminal accuracy.
♻ ☆ Limitations of Using Identical Distributions for Training and Testing When Learning Boolean Functions
Jordi Pérez-Guijarro
When the distributions of the training and test data do not coincide, the problem of understanding generalization becomes considerably more complex, prompting a variety of questions. Prior work has shown that, for some fixed learning methods, there are scenarios where training on a distribution different from the test distribution improves generalization. However, these results do not account for the possibility of choosing, for each training distribution, the optimal learning algorithm, leaving open whether the observed benefits stem from the mismatch itself or from suboptimality of the learner. In this work, we address this question in full generality. That is, we study whether it is always optimal for the training distribution to be identical to the test distribution when the learner is allowed to be optimally adapted to the training distribution. Surprisingly, assuming the existence of one-way functions, we find that the answer is no. That is, matching distributions is not always the best scenario. Nonetheless, we also show that when certain regularities are imposed on the target functions, the standard conclusion is recovered in the case of the uniform distribution.
♻ ☆ Knowledge Adaptation as Posterior Correction
Mohammad Emtiyaz Khan
Adaptation is the holy grail of intelligence, but even the best AI models lack the adaptability of toddlers. In spite of great progress, little is known about the mechanisms by which machines can learn to adapt as fast as humans and animals. Here, we cast adaptation as `correction' of old posteriors and show that a wide-variety of existing adaptation methods follow this very principle, including those used for continual learning, federated learning, unlearning, and model merging. In all these settings, more accurate posteriors often lead to smaller corrections and can enable faster adaptation. Posterior correction is derived by using the dual representation of the Bayesian Learning Rule of Khan and Rue (2023), where the interference between the old representation and new information is quantified by using the natural-gradient mismatch. We present many examples demonstrating how machines can learn to adapt quickly by using posterior correction.
♻ ☆ Text-Queried Audio Source Separation via Hierarchical Modeling
Xinlei Yin, Xiulian Peng, Xue Jiang, Zhiwei Xiong, Yan Lu
Target audio source separation with natural language queries presents a promising paradigm for extracting arbitrary audio events through arbitrary text descriptions. Existing methods mainly face two challenges, the difficulty in jointly modeling acoustic-textual alignment and semantic-aware separation within a blindly-learned single-stage architecture, and the reliance on large-scale accurately-labeled training data to compensate for inefficient cross-modal learning and separation. To address these challenges, we propose a hierarchical decomposition framework, HSM-TSS, that decouples the task into global-local semantic-guided feature separation and structure-preserving acoustic reconstruction. Our approach introduces a dual-stage mechanism for semantic separation, operating on distinct global and local semantic feature spaces. We first perform global-semantic separation through a global semantic feature space aligned with text queries. A Q-Audio architecture is employed to align audio and text modalities, serving as pretrained global-semantic encoders. Conditioned on the predicted global feature, we then perform the second-stage local-semantic separation on AudioMAE features that preserve time-frequency structures, followed by acoustic reconstruction. We also propose an instruction processing pipeline to parse arbitrary text queries into structured operations, extraction or removal, coupled with audio descriptions, enabling flexible sound manipulation. Our method achieves state-of-the-art separation performance with data-efficient training while maintaining superior semantic consistency with queries in complex auditory scenes.
comment : Accepted by TASLP
♻ ☆ Membership Inference Attack against Large Language Model-based Recommendation Systems: A New Distillation-based Paradigm
Li Cuihong, Huang Xiaowen, Yin Chuanhuan, Sang Jitao
Membership Inference Attack (MIA) aims to determine whether a specific data sample was included in the training dataset of a target model. Traditional MIA approaches rely on shadow models to mimic target model behavior, but their effectiveness diminishes for Large Language Model (LLM)-based recommendation systems due to the scale and complexity of training data. This paper introduces a novel knowledge distillation-based MIA paradigm tailored for LLM-based recommendation systems. Our method constructs a reference model via distillation, applying distinct strategies for member and non-member data to enhance discriminative capabilities. The paradigm extracts fused features (e.g., confidence, entropy, loss, and hidden layer vectors) from the reference model to train an attack model, overcoming limitations of individual features. Extensive experiments on extended datasets (Last.FM, MovieLens, Book-Crossing, Delicious) and diverse LLMs (T5, GPT-2, LLaMA3) demonstrate that our approach significantly outperforms shadow model-based MIAs and individual-feature baselines. The results show its practicality for privacy attacks in LLM-driven recommender systems.
♻ ☆ Dynamic Feature Selection based on Rule-based Learning for Explainable Classification with Uncertainty Quantification
Javier Fumanal-Idocin, Raquel Fernandez-Peralta, Javier Andreu-Perez
Dynamic feature selection (DFS) offers a compelling alternative to traditional, static feature selection by adapting the selected features to each individual sample. This provides insights into the decision-making process for each case, which makes DFS especially significant in settings where decision transparency is key, i.e., clinical decisions. However, existing DFS methods use opaque models, which hinder their applicability in real-life scenarios. DFS also introduces new own sources of uncertainty compared to the static setting, which is also not considered in the existing literature. In this paper, we formalize the additional sources of uncertainty in DFS, and give formulas to estimate them. We also propose novel approach by leveraging a rule-based system as a base classifier for the DFS process, which enhances decision interpretability compared to neural estimators. Finally, we demonstrate the competitive performance of our rule-based DFS approach against established and state-of-the-art greedy and reinforcement learning methods, which are mostly considered opaque, compared to our explainable rulebased system.
♻ ☆ The Necessity of Imperfection:Reversing Model Collapse via Simulating Cognitive Boundedness
Zhongjie Jiang
Although synthetic data is widely promoted as a remedy, its prevailing production paradigm -- one optimizing for statistical smoothness -- systematically removes the long-tail, cognitively grounded irregularities that characterize human text. Prolonged training on such statistically optimal but cognitively impoverished data accelerates model collapse. This paper proposes a paradigm shift: instead of imitating the surface properties of data, we simulate the cognitive processes that generate human text. We introduce the Prompt-driven Cognitive Computing Framework (PMCSF), whose core consists of a Cognitive State Decoder (CSD) that reverse-engineers unstructured text into structured cognitive vectors, and a Cognitive Text Encoder (CTE) that re-materializes these states into text enriched with human-typical imperfections via mathematically defined Cognitive Perturbation Operators. The framework is validated through a two-stage objective evaluation pipeline. First, in cognitive codec verification, CTE text yields a Jensen-Shannon divergence of 0.0614 from human text (vs. 0.4431 for standard LLM output), passes double-blind professional media review, and achieves an intraclass correlation coefficient ICC > 0.9 for cognitive profile alignment across heterogeneous models. Second, in functional gain evaluation, isomorphic stress tests in the A-share market show that strategies incorporating CTE-generated data reduce maximum drawdown by 47.4% during the 2015 crash and deliver 8.6% Defensive Alpha, exceeding transaction costs by a factor of 33. Our findings demonstrate that modelling human cognitive limitations -- not copying surface data -- enables synthetic data with genuine functional gain, offering a viable technical pathway toward resolving the AI data-collapse crisis.
comment : 38 pages,5 figures. Extended technical disclosure (Version 2.0) is attached as ancillary files, containing raw forensic logs of the "Silent Rupture"detection [May 2025], proprietary GARCH parameter ranges, and the linguistic micro-chaos injection protocols
♻ ☆ Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models
Shuxun Wang, Qingyu Yin, Chak Tou Leong, Qiang Zhang, Linyi Yang
Repetition curse is a phenomenon where Large Language Models (LLMs) generate repetitive sequences of tokens or cyclic sequences. While the repetition curse has been widely observed, its underlying mechanisms remain poorly understood. In this work, we investigate the role of induction heads--a specific type of attention head known for their ability to perform in-context learning--in driving this repetitive behavior. Specifically, we focus on the "toxicity" of induction heads, which we define as their tendency to dominate the model's output logits during repetition, effectively excluding other attention heads from contributing to the generation process. Our findings have important implications for the design and training of LLMs. By identifying induction heads as a key driver of the repetition curse, we provide a mechanistic explanation for this phenomenon and suggest potential avenues for mitigation. We also propose a technique with attention head regularization that could be employed to reduce the dominance of induction heads during generation, thereby promoting more diverse and coherent outputs.
comment : Need to be refined
♻ ☆ Large Language Models for Robotics: A Survey
Fanlong Zeng, Wensheng Gan, Zezheng Huai, Lichao Sun, Hechang Chen, Yongheng Wang, Ning Liu, Philip S. Yu
The human ability to learn, generalize, and control complex manipulation tasks through multi-modality feedback suggests a unique capability, which we refer to as dexterity intelligence. Understanding and assessing this intelligence is a complex task. Amidst the swift progress and extensive proliferation of large language models (LLMs), their applications in the field of robotics have garnered increasing attention. LLMs possess the ability to process and generate natural language, facilitating efficient interaction and collaboration with robots. Researchers and engineers in the field of robotics have recognized the immense potential of LLMs in enhancing robot intelligence, human-robot interaction, and autonomy. Therefore, this comprehensive review aims to summarize the applications of LLMs in robotics, delving into their impact and contributions to key areas such as robot control, perception, decision-making, and planning. This survey first provides an overview of the background and development of LLMs for robotics, followed by a discussion of their benefits and recent advancements in LLM-based robotic models. It then explores various techniques, employed in perception, decision-making, control, and interaction, as well as cross-module coordination in practical tasks. Finally, we review current applications of LLMs in robotics and outline potential challenges they may face in the near future. Embodied intelligence represents the future of intelligent systems, and LLM-based robotics is one of the most promising yet challenging paths toward achieving it.
comment : Preprint. 9 figures, 7 tables
♻ ☆ Artificial Intelligence Virtual Cells: From Measurements to Decisions across Modality, Scale, Dynamics, and Evaluation
Chengpeng Hu, Calvin Yu-Chian Chen
Artificial Intelligence Virtual Cells (AIVCs) aim to learn executable, decision-relevant models of cell state from multimodal, multiscale measurements. Recent studies have introduced single-cell and spatial foundation models, improved cross-modality alignment, scaled perturbation atlases, and explored pathway-level readouts. Nevertheless, although held-out validation is standard practice, evaluations remain predominantly within single datasets and settings; evidence indicates that transport across laboratories and platforms is often limited, that some data splits are vulnerable to leakage and coverage bias, and that dose, time and combination effects are not yet systematically handled. Cross-scale coupling also remains constrained, as anchors linking molecular, cellular and tissue levels are sparse, and alignment to scientific or clinical readouts varies across studies. We propose a model-agnostic Cell-State Latent (CSL) perspective that organizes learning via an operator grammar: measurement, lift/project for cross-scale coupling, and intervention for dosing and scheduling. This view motivates a decision-aligned evaluation blueprint across modality, scale, context and intervention, and emphasizes function-space readouts such as pathway activity, spatial neighborhoods and clinically relevant endpoints. We recommend operator-aware data design, leakage-resistant partitions, and transparent calibration and reporting to enable reproducible, like-for-like comparisons.
♻ ☆ Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as- Judge Reinforcement Learning
Jiahao Yuan, Zhiqing Cui, Hanqing Wang, Yuansheng Gao, Yucheng Zhou, Usman Naseem
As web platforms evolve towards greater personalization and emotional complexity, conversational agents must transcend superficial empathy to demonstrate identity-aware emotional reasoning. However, existing systems face two limitations: (1) reliance on situation-centric datasets lacking persistent user identity, which hampers the capture of personalized affective nuances; and (2) dependence on opaque, coarse reward signals that hinder development of verifiable empathetic reasoning. To address these gaps, we introduce KardiaBench, a large-scale user-grounded benchmark comprising 178,080 QA pairs across 22,080 multi-turn conversations anchored to 671 real-world profiles. The dataset is constructed via a model-in-the-loop pipeline with iterative rubric-guided refinement to ensure psychological plausibility and persona consistency. This progressive empathy pipeline that integrates user comprehension, contextual reasoning, and emotion perception into conversations, followed by iterative critique and rubric-based refinement to ensure psychological plausibility, emotional fidelity, and persona consistency. Building on this, we propose Kardia-R1, a framework that trains models for interpretable, stepwise empathetic cognition. Kardia-R1 leverages Rubric-as-Judge Empathetic Reinforcement Learning (Rubric-ERL), a GRPO-based method that uses explainable, human-aligned rubric rewards to tightly couple user understanding, emotional inference, and supportive response generation. Extensive experiments across four LLM backbones demonstrate that Kardia-R1 consistently outperforms othet methods in emotion accuracy, empathy, relevance, persona consistency, and safety. Our dataset and model will be released at https://github.com/JhCircle/Kardia-R1.
♻ ☆ ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models
Jincheng Liu, Sijun He, Jingjing Wu, Xiangsen Wang, Yang Chen, Zhaoqi Kuang, Siqi Bao, Yuan Yao
Recent large language models (LLMs) have shown strong reasoning capabilities. However, a critical question remains: do these models possess genuine reasoning skills particularly complex strategic reasoning or are they primarily excelling at sophisticated pattern recognition within their training data? To address this question, this paper presents a chess testbed, ChessArena, to evaluate the strategic reasoning capabilities of LLMs. Chess requires complex strategic reasoning capabilities including long-term planning, strict rule comprehension, and multi-turn conversation memorization. Specifically, ChessArena is a competitive framework where LLMs play against each other, under four different play modes. The testbed is equipped with a ranking algorithm and a leaderboard. The testbed can also evaluate fine-grained capabilities including basic understanding, move selection, and puzzle solving. Over 13 LLMs with different modes are evaluated in ChessArena, playing over 800 games. The results reveal significant shortcomings in current LLMs: no model can beat Maia-1100 (a chess engine at human amateur level), while some even failed to defeat a random player that selects moves arbitrarily. We also present a strong baseline to the testbed: our fine-tuned Qwen3-8B substantially improved performance, approaching much larger state-of-the-art reasoning models.
♻ ☆ Walk Before You Dance: High-fidelity and Editable Dance Synthesis via Generative Masked Motion Prior
Foram N Shah, Parshwa Shah, Muhammad Usama Saleem, Ekkasit Pinyoanuntapong, Pu Wang, Hongfei Xue, Ahmed Helmy
Recent advances in dance generation have enabled the automatic synthesis of 3D dance motions. However, existing methods still face significant challenges in simultaneously achieving high realism, precise dance-music synchronization, diverse motion expression, and physical plausibility. To address these limitations, we propose a novel approach that leverages a generative masked text-to-motion model as a distribution prior to learn a probabilistic mapping from diverse guidance signals, including music, genre, and pose, into high-quality dance motion sequences. Our framework also supports semantic motion editing, such as motion inpainting and body part modification. Specifically, we introduce a multi-tower masked motion model that integrates a text-conditioned masked motion backbone with two parallel, modality-specific branches: a music-guidance tower and a pose-guidance tower. The model is trained using synchronized and progressive masked training, which allows effective infusion of the pretrained text-to-motion prior into the dance synthesis process while enabling each guidance branch to optimize independently through its own loss function, mitigating gradient interference. During inference, we introduce classifier-free logits guidance and pose-guided token optimization to strengthen the influence of music, genre, and pose signals. Extensive experiments demonstrate that our method sets a new state of the art in dance generation, significantly advancing both the quality and editability over existing approaches. Project Page available at https://foram-s1.github.io/DanceMosaic/
♻ ☆ CT-GLIP: 3D Grounded Language-Image Pretrain ing with CT Scans and Radiology Reports for Full-Body Scenarios
Jingyang Lin, Yingda Xia, Jianpeng Zhang, Ke Yan, Kai Cao, Le Lu, Jiebo Luo, Ling Zhang
3D medical vision-language (VL) pretraining has shown potential in radiology by leveraging large-scale multimodal datasets with CT-report pairs. However, existing methods primarily rely on a global VL alignment directly adapted from 2D scenarios. The entire 3D image is transformed into one global embedding, resulting in a loss of sparse but critical semantics essential for accurately aligning with the corresponding diagnosis. To address this limitation, we propose CT-GLIP, a 3D Grounded Language-Image Pretrained model that constructs fine-grained CT-report pairs to enhance \textit{grounded} cross-modal contrastive learning, effectively aligning grounded visual features with precise textual descriptions. Leveraging the grounded cross-modal alignment, CT-GLIP improves performance across diverse downstream tasks and can even identify organs and abnormalities in a zero-shot manner using natural language. CT-GLIP is trained on a multimodal CT dataset comprising 44,011 organ-level CT-report pairs from 17,702 patients, covering 104 organs. Evaluation is conducted on four downstream tasks: zero-shot organ recognition (OR), zero-shot abnormality detection (AD), tumor detection (TD), and tumor segmentation (TS). Empirical results show that it outperforms its counterparts with global VL alignment. Compared to vanilla CLIP, CT-GLIP achieves average performance improvements of 15.1% of F1 score, 1.9% of AUC, and 3.2% of DSC for zero-shot AD, TD, and TS tasks, respectively. This study highlights the significance of grounded VL alignment in enabling 3D medical VL foundation models to understand sparse representations within CT scans.
♻ ☆ Revisiting Clustering of Neural Bandits: Selective Reinitialization for Mitigating Loss of Plasticity KDD 2025
Zhiyuan Su, Sunhao Dai, Xiao Zhang
Clustering of Bandits (CB) methods enhance sequential decision-making by grouping bandits into clusters based on similarity and incorporating cluster-level contextual information, demonstrating effectiveness and adaptability in applications like personalized streaming recommendations. However, when extending CB algorithms to their neural version (commonly referred to as Clustering of Neural Bandits, or CNB), they suffer from loss of plasticity, where neural network parameters become rigid and less adaptable over time, limiting their ability to adapt to non-stationary environments (e.g., dynamic user preferences in recommendation). To address this challenge, we propose Selective Reinitialization (SeRe), a novel bandit learning framework that dynamically preserves the adaptability of CNB algorithms in evolving environments. SeRe leverages a contribution utility metric to identify and selectively reset underutilized units, mitigating loss of plasticity while maintaining stable knowledge retention. Furthermore, when combining SeRe with CNB algorithms, the adaptive change detection mechanism adjusts the reinitialization frequency according to the degree of non-stationarity, ensuring effective adaptation without unnecessary resets. Theoretically, we prove that SeRe enables sublinear cumulative regret in piecewise-stationary environments, outperforming traditional CNB approaches in long-term performances. Extensive experiments on six real-world recommendation datasets demonstrate that SeRe-enhanced CNB algorithms can effectively mitigate the loss of plasticity with lower regrets, improving adaptability and robustness in dynamic settings.
comment : Accepted by KDD 2025
♻ ☆ WearVQA: A Visual Question Answering Benchmark for Wearables in Egocentric Authentic Real-wo rl d scenarios NeurIPS 2025
Eun Chang, Zhuangqun Huang, Yiwei Liao, Sagar Ravi Bhavsar, Amogh Param, Tammy Stark, Adel Ahmadyan, Xiao Yang, Jiaqi Wang, Ahsan Abdullah, Giang Nguyen, Akil Iyer, David Hall, Elissa Li, Shane Moon, Nicolas Scheffer, Kirmani Ahmed, Babak Damavandi, Rakesh Wanga, Anuj Kumar, Rohit Patel, Xin Luna Dong
We introduce WearVQA, the first benchmark specifically designed to evaluate the Visual Question Answering (VQA) capabilities of multi-model AI assistant on wearable devices like smart glasses. Unlike prior benchmarks that focus on high-quality, third-person imagery, WearVQA reflects the unique challenges of ego-centric interaction-where visual inputs may be occluded, poorly lit, unzoomed, or blurry, and questions are grounded in realistic wearable use cases. The benchmark comprises 2,520 carefully curated image-question-answer triplets, spanning 7 diverse image domains including both text-centric and general scenes, 10 cognitive task types ranging from basic recognition to various forms of reasoning, and 6 common wearables-specific image quality issues. All questions are designed to be answerable using only the visual input and common senses. WearVQA is paired with a rigorous LLM-as-a-judge evaluation framework with 96% labeling accuracy. Open-source and proprietary multi-model LLMs achieved a QA accuracy as low as 24-52% on WearVQA, with substantial drops on lower-quality images and reasoning-heavy tasks. These observations position WearVQA as a comprehensive and challenging benchmark for guiding technical advancement towards robust, real-world multi-model wearables AI systems.
comment : 11 pages, 5 figures, NeurIPS 2025
♻ ☆ Mutually-Aware Feature Learning for Few-Shot Object Counting
Yerim Jeon, Subeen Lee, Jihwan Kim, Jae-Pil Heo
Few-shot object counting has garnered significant attention for its practicality as it aims to count target objects in a query image based on given exemplars without additional training. However, the prevailing extract-and-match approach has a shortcoming: query and exemplar features lack interaction during feature extraction since they are extracted independently and later correlated based on similarity. This can lead to insufficient target awareness and confusion in identifying the actual target when multiple class objects coexist. To address this, we propose a novel framework, Mutually-Aware FEAture learning (MAFEA), which encodes query and exemplar features with mutual awareness from the outset. By encouraging interaction throughout the pipeline, we obtain target-aware features robust to a multi-category scenario. Furthermore, we introduce background token to effectively associate the query's target region with exemplars and decouple its background region. Our extensive experiments demonstrate that our model achieves state-of-the-art performance on FSCD-LVIS and FSC-147 benchmarks with remarkably reduced target confusion.
comment : Accepted to Pattern Recognition 2025
♻ ☆ Pa rl AI Vote: A Web Platform for Analyzing Gender and Political Bias in Large Language Models
Wenjie Lin, Hange Liu, Yingying Zhuang, Xutao Mao, Jingwei Shi, Xudong Han, Tianyu Shi, Jinrui Yang
We present ParlAI Vote, an interactive web platform for exploring European Parliament debates and votes, and for testing LLMs on vote prediction and bias analysis. This web system connects debate topics, speeches, and roll-call outcomes, and includes rich demographic data such as gender, age, country, and political group. Users can browse debates, inspect linked speeches, compare real voting outcomes with predictions from frontier LLMs, and view error breakdowns by demographic group. Visualizing the EuroParlVote benchmark and its core tasks of gender classification and vote prediction, ParlAI Vote highlights systematic performance bias in state-of-the-art LLMs. It unifies data, models, and visual analytics in a single interface, lowering the barrier for reproducing findings, auditing behavior, and running counterfactual scenarios. This web platform also shows model reasoning, helping users see why errors occur and what cues the models rely on. It supports research, education, and public engagement with legislative decision-making, while making clear both the strengths and the limitations of current LLMs in political analysis.
comment : online demo: https://euro-parl-vote-demo.vercel.app/; Video: https://www.youtube.com/@Jinrui-sf2jg
♻ ☆ Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models NeurIPS 2025
Cameron Tice, Philipp Alexander Kreer, Nathan Helm-Burger, Prithviraj Singh Shahani, Fedor Ryzhenkov, Fabien Roger, Clement Neo, Jacob Haimes, Felix Hofstätter, Teun van der Weij
Capability evaluations play a crucial role in assessing and regulating frontier AI systems. The effectiveness of these evaluations faces a significant challenge: strategic underperformance, or ``sandbagging'', where models deliberately underperform during evaluation. Sandbagging can manifest either through explicit developer intervention or through unintended model behavior, presenting a fundamental obstacle to accurate capability assessment. We introduce a novel sandbagging detection method based on injecting noise of varying magnitudes into model weights. While non-sandbagging models show predictable performance degradation with increasing noise, we demonstrate that sandbagging models exhibit anomalous performance improvements, likely due to disruption of underperformance mechanisms while core capabilities remain partially intact. Through experiments across various model architectures, sizes, and sandbagging techniques, we establish this distinctive response pattern as a reliable, model-agnostic signal for detecting sandbagging behavior. Importantly, we find noise-injection is capable of eliciting the full performance of Mistral Large 120B in a setting where the model underperforms without being instructed to do so. Our findings provide a practical tool for AI evaluation and oversight, addressing a challenge in ensuring accurate capability assessment of frontier AI systems.
comment : Published at NeurIPS 2025, code available at https://github.com/camtice/SandbagDetect. Preliminary work presented at SATA and SoLaR (NeurIPS 2024 workshops)
♻ ☆ Subgoal-Guided Policy Heuristic Search with Learned Subgoals ICML-25
Jake Tuero, Michael Buro, Levi H. S. Lelis
Policy tree search is a family of tree search algorithms that use a policy to guide the search. These algorithms provide guarantees on the number of expansions required to solve a given problem that are based on the quality of the policy. While these algorithms have shown promising results, the process in which they are trained requires complete solution trajectories to train the policy. Search trajectories are obtained during a trial-and-error search process. When the training problem instances are hard, learning can be prohibitively costly, especially when starting from a randomly initialized policy. As a result, search samples are wasted in failed attempts to solve these hard instances. This paper introduces a novel method for learning subgoal-based policies for policy tree search algorithms. The subgoals and policies conditioned on subgoals are learned from the trees that the search expands while attempting to solve problems, including the search trees of failed attempts. We empirically show that our policy formulation and training method improve the sample efficiency of learning a policy and heuristic function in this online setting.
comment : Accepted to ICML-25
♻ ☆ Enhanced Conditional Generation of Double Perovskite by Knowledge-Guided Language Model Feedback
Inhyo Lee, Junhyeong Lee, Jongwon Park, KyungTae Lim, Seunghwa Ryu
Double perovskites (DPs) are promising candidates for sustainable energy technologies due to their compositional tunability and compatibility with low-energy fabrication, yet their vast design space poses a major challenge for conditional materials discovery. This work introduces a multi-agent, text gradient-driven framework that performs DP composition generation under natural-language conditions by integrating three complementary feedback sources: LLM-based self-evaluation, DP-specific domain knowledge-informed feedback, and ML surrogate-based feedback. Analogous to how knowledge-informed machine learning improves the reliability of conventional data-driven models, our framework incorporates domain-informed text gradients to guide the generative process toward physically meaningful regions of the DP composition space. Systematic comparison of three incremental configurations, (i) pure LLM generation, (ii) LLM generation with LLM reasoning-based feedback, and (iii) LLM generation with domain knowledge-guided feedback, shows that iterative guidance from knowledge-informed gradients improves stability-condition satisfaction without additional training data, achieving over 98% compositional validity and up to 54% stable or metastable candidates, surpassing both the LLM-only baseline (43%) and prior GAN-based results (27%). Analyses of ML-based gradients further reveal that they enhance performance in in-distribution (ID) regions but become unreliable in out-of-distribution (OOD) regimes. Overall, this work provides the first systematic analysis of multi-agent, knowledge-guided text gradients for DP discovery and establishes a generalizable blueprint for MAS-driven generative materials design aimed at advancing sustainable technologies.
♻ ☆ H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs
Cheng Gao, Huimin Chen, Chaojun Xiao, Zhiyi Chen, Zhiyuan Liu, Maosong Sun
Large language models (LLMs) frequently generate hallucinations -- plausible but factually incorrect outputs -- undermining their reliability. While prior work has examined hallucinations from macroscopic perspectives such as training data and objectives, the underlying neuron-level mechanisms remain largely unexplored. In this paper, we conduct a systematic investigation into hallucination-associated neurons (H-Neurons) in LLMs from three perspectives: identification, behavioral impact, and origins. Regarding their identification, we demonstrate that a remarkably sparse subset of neurons (less than $0.1\%$ of total neurons) can reliably predict hallucination occurrences, with strong generalization across diverse scenarios. In terms of behavioral impact, controlled interventions reveal that these neurons are causally linked to over-compliance behaviors. Concerning their origins, we trace these neurons back to the pre-trained base models and find that these neurons remain predictive for hallucination detection, indicating they emerge during pre-training. Our findings bridge macroscopic behavioral patterns with microscopic neural mechanisms, offering insights for developing more reliable LLMs.
comment : 20 pages, 4 figures
♻ ☆ Multimodal Continual Learning with MLLMs from Multi-scenario Perspectives CVPR 2026
Kai Jiang, Siqi Huang, Xiangyu Chen, Jiawei Shao, Hongyuan Zhang, Xuelong Li
Continual learning in visual understanding aims to deal with catastrophic forgetting in Multimodal Large Language Models (MLLMs). MLLMs deployed on devices have to continuously adapt to dynamic scenarios in downstream tasks, such as variations in background and perspective, to effectively perform complex visual tasks. To this end, we construct a multimodal visual understanding dataset (MSVQA) encompassing four different scenarios and perspectives including high altitude, underwater, low altitude and indoor, to investigate the catastrophic forgetting in MLLMs under the dynamics of scenario shifts in real-world data streams. Furthermore, we propose mUltimodal coNtInual learning with MLLMs From multi-scenarIo pERspectives (UNIFIER) to address visual discrepancies while learning different scenarios. Specifically, it decouples the visual information from different scenarios into distinct branches within each vision block and projects them into the same feature space. A consistency constraint is imposed on the features of each branch to maintain the stability of visual representations across scenarios. Extensive experiments on the MSVQA dataset demonstrate that UNIFIER effectively alleviates forgetting of cross-scenario tasks and achieves knowledge accumulation within the same scenario.
comment : 18 pages, 16 figures. This is a preprint version of a paper submitted to CVPR 2026
♻ ☆ HBLLM: A Haar-Based Approach for Accurate Structured 1-Bit Quantized LLMs
Ningning Chen, Weicai Ye, Ying Jiang
We introduce HBLLM, a wavelet-enhanced high-fidelity $1$-bit post-training quantization method for Large Language Models (LLMs). By leveraging Haar wavelet transforms to enhance expressive capacity through frequency decomposition, HBLLM significantly improves quantization fidelity while maintaining minimal overhead. This approach features two innovative structure-aware grouping strategies: (1) frequency-aware multi-parameter intra-row grouping and (2) $\ell_2$-norm-based saliency-driven column selection. For non-salient weights, a shared mean is employed across quantization groups within each frequency band to optimize storage efficiency. Experiments conducted on the OPT and LLaMA models demonstrate that HBLLM achieves state-of-the-art performance in $1$-bit quantization, attaining a perplexity of $6.71$ on LLaMA$2$-$13$B with an average weight storage of only $1.08$ bits. Code available at: https://github.com/Yeyke/HBLLM.
♻ ☆ Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, Hao Lin, Chencan Wu, Feng Hu, An Yang, Jingren Zhou, Junyang Lin
This paper proposes a novel formulation for reinforcement learning (RL) with large language models, explaining why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE. Specifically, through a first-order approximation, we show that this surrogate becomes increasingly valid only when both the training-inference discrepancy and policy staleness are minimized. This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training, including importance sampling correction, clipping, and particularly Routing Replay for Mixture-of-Experts (MoE) models. Through extensive experiments with a 30B MoE model totaling hundreds of thousands of GPU hours, we show that for on-policy training, the basic policy gradient algorithm with importance sampling correction achieves the highest training stability. When off-policy updates are introduced to accelerate convergence, combining clipping and Routing Replay becomes essential to mitigate the instability caused by policy staleness. Notably, once training is stabilized, prolonged optimization consistently yields comparable final performance regardless of cold-start initialization. We hope that the shared insights and the developed recipes for stable RL training will facilitate future research.
♻ ☆ TimeSearch: Hierarchical Video Search with Spotlight and Reflection for Human-like Long Video Understanding
Junwen Pan, Rui Zhang, Xin Wan, Yuan Zhang, Ming Lu, Qi She
Large video-language models (LVLMs) have shown remarkable performance across various video-language tasks. However, they encounter significant challenges when processing long videos because of the large number of video frames involved. Downsampling long videos in either space or time can lead to visual hallucinations, making it difficult to accurately interpret long videos. Motivated by human hierarchical temporal search strategies, we propose \textbf{TimeSearch}, a novel framework enabling LVLMs to understand long videos in a human-like manner. TimeSearch integrates two human-like primitives into a unified autoregressive LVLM: 1) \textbf{Spotlight} efficiently identifies relevant temporal events through a Temporal-Augmented Frame Representation (TAFR), explicitly binding visual features with timestamps; 2) \textbf{Reflection} evaluates the correctness of the identified events, leveraging the inherent temporal self-reflection capabilities of LVLMs. TimeSearch progressively explores key events and prioritizes temporal search based on reflection confidence. Extensive experiments on challenging long-video benchmarks confirm that TimeSearch substantially surpasses previous state-of-the-art, improving the accuracy from 41.8\% to 51.5\% on the LVBench. Additionally, experiments on temporal grounding demonstrate that appropriate TAFR is adequate to effectively stimulate the surprising temporal grounding ability of LVLMs in a simpler yet versatile manner, which improves mIoU on Charades-STA by 11.8\%. The code will be released.
♻ ☆ BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-Wo rl d Cybersecurity Systems
Andy K. Zhang, Joey Ji, Celeste Menders, Riya Dulepet, Thomas Qin, Ron Y. Wang, Junrong Wu, Kyleen Liao, Jiliang Li, Jinghan Hu, Sara Hong, Nardos Demilew, Shivatmica Murgai, Jason Tran, Nishka Kacheria, Ethan Ho, Denis Liu, Lauren McLane, Olivia Bruvik, Dai-Rong Han, Seungwoo Kim, Akhil Vyas, Cuiyuanxiu Chen, Ryan Li, Weiran Xu, Jonathan Z. Ye, Prerit Choudhary, Siddharth M. Bhatia, Vikram Sivashankar, Yuxuan Bao, Dawn Song, Dan Boneh, Daniel E. Ho, Percy Liang
AI agents have the potential to significantly alter the cybersecurity landscape. Here, we introduce the first framework to capture offensive and defensive cyber-capabilities in evolving real-world systems. Instantiating this framework with BountyBench, we set up 25 systems with complex, real-world codebases. To capture the vulnerability lifecycle, we define three task types: Detect (detecting a new vulnerability), Exploit (exploiting a given vulnerability), and Patch (patching a given vulnerability). For Detect, we construct a new success indicator, which is general across vulnerability types and provides localized evaluation. We manually set up the environment for each system, including installing packages, setting up server(s), and hydrating database(s). We add 40 bug bounties, which are vulnerabilities with monetary awards from \$10 to \$30,485, covering 9 of the OWASP Top 10 Risks. To modulate task difficulty, we devise a new strategy based on information to guide detection, interpolating from identifying a zero day to exploiting a given vulnerability. We evaluate 10 agents: Claude Code, OpenAI Codex CLI with o3-high and o4-mini, and custom agents with o3-high, GPT-4.1, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet Thinking, Qwen3 235B A22B, Llama 4 Maverick, and DeepSeek-R1. Given up to three attempts, the top-performing agents are Codex CLI: o3-high (12.5% on Detect, mapping to \$3,720; 90% on Patch, mapping to \$14,152), Custom Agent: Claude 3.7 Sonnet Thinking (67.5% on Exploit), and Codex CLI: o4-mini (90% on Patch, mapping to \$14,422). Codex CLI: o3-high, Codex CLI: o4-mini, and Claude Code are more capable at defense, achieving higher Patch scores of 90%, 90%, and 87.5%, compared to Exploit scores of 47.5%, 32.5%, and 57.5% respectively; while the custom agents are relatively balanced between offense and defense, achieving Exploit scores of 17.5-67.5% and Patch scores of 25-60%.
comment : 113 pages
♻ ☆ GRAFT: GRaPH and Table Reasoning for Textual Alignment -- A Benchmark for Structured Instruction Following and Visual Reasoning
Abhigya Verma, Sriram Puttagunta, Seganrasan Subramanian, Sravan Ramachandran
GRAFT is a structured multimodal benchmark designed to probe how well LLMs handle instruction following, visual reasoning, and tasks requiring tight visual textual alignment. The dataset is built around programmatically generated charts and synthetically rendered tables, each paired with a carefully constructed, multi step analytical question that depends solely on what can be inferred from the image itself. Responses are formatted in structured outputs such as JSON or YAML, enabling consistent and fine grained evaluation of both reasoning processes and adherence to output specifications. The benchmark further introduces a taxonomy of reasoning operations ranging from comparison and trend identification to ranking, aggregation, proportional estimation, and anomaly detection to support a comprehensive assessment of model capabilities. Taken together, GRAFT provides a unified and scalable framework for evaluating multimodal LLMs on visually grounded, structured reasoning tasks, offering a more rigorous standard for future benchmarking efforts.
comment : 25 pages, 10 tables, 3 figures
♻ ☆ Intrinsic Structure as a Proxy for Saliency: SVD-Based Weight Preservation for Mixed-Precision Quantization in Large Language Models
Shashank Landge, Abhishek Patil, Tejas kamble, Bhushan Buddhivant, Priyanka Joshi
As Large Language Models (LLMs) continue to scale in parameter count, deploying them on commodity hardware has become increasingly challenging. Post-Training Quantization (PTQ) addresses this by reducing the precision of model weights, typically to 4-bit or lower. However, uniform quantization often leads to significant performance degradation due to the presence of ``outlier features'' -- weights that, while few in number, are critical for maintaining model accuracy. Current state-of-the-art methods such as AWQ (Activation-aware Weight Quantization) and SpQR (Sparse Quantization Representations) rely on calibration data to identify these salient weights via activation magnitudes or Hessian sensitivity. In scenarios where data privacy is paramount or calibration data is unavailable, these methods are inapplicable. In this work, we propose a data-free, structure-aware hypothesis: that the weights identified as Principal Components via Singular Value Decomposition (SVD) are intrinsically important to the model's downstream performance. We introduce a novel selection heuristic that preserves the top-$k$ weights aligned with the principal components in FP32, while aggressively quantizing the residual weights. We compare our method against activation-aware (AWQ) and second-order (SpQR) methods across GLUE benchmarks (MRPC, RTE, QNLI) using a DistilBERT backbone. Our experiments reveal that structural importance is highly correlated with functional importance. On the challenging RTE task, our SVD-based method achieves an accuracy of 66.06\%, outperforming both AWQ (65.34\%) and SpQR (65.34\%) at high protection budgets, validating that intrinsic matrix structure can serve as a robust proxy for weight saliency without the need for forward passes or calibration data.
♻ ☆ Multi- Agent Code Verification via Information Theory
Shreshth Rajan
LLMs generate buggy code: 29.6% of SWE-bench solved patches fail, 62% of BaxBench solutions have vulnerabilities, and existing tools only catch 65% of bugs with 35% false positives. We built CodeX-Verify, a multi-agent system that uses four specialized agents to detect different types of bugs. We prove mathematically that combining agents with different detection patterns finds more bugs than any single agent when the agents look for different problems, using submodularity of mutual information under conditional independence. Measuring agent correlation of rho = 0.05 to 0.25 confirms they detect different bugs. Testing on 99 code samples with verified labels shows our system catches 76.1% of bugs, matching the best existing method (Meta Prompt Testing: 75%) while running faster and without test execution. We tested all 15 agent combinations and found that using multiple agents improves accuracy by 39.7 percentage points (from 32.8% to 72.4%) compared to single agents, with diminishing returns of +14.9pp, +13.5pp, and +11.2pp for agents 2, 3, and 4, validating our theoretical model. The best two-agent combination (Correctness + Performance) reaches 79.3% accuracy. Testing on 300 real patches from Claude Sonnet 4.5 runs in under 200ms per sample, making this practical for production use.
comment : 18 pages, 3 figures, 9 tables
♻ ☆ Aligning Compound AI Systems via System-level DPO NeurIPS 2025
Xiangwen Wang, Yibo Jacky Zhang, Zhoujie Ding, Katherine Tsai, Haolun Wu, Sanmi Koyejo
Compound AI systems, comprising multiple interacting components such as LLMs, foundation models, and external tools, have demonstrated remarkable improvements compared to single models in various tasks. To ensure their effective deployment in real-world applications, aligning these systems with human preferences is crucial. However, aligning the compound system via policy optimization, unlike the alignment of a single model, is challenging for two main reasons: (i) non-differentiable interactions between components make end-to-end gradient-based optimization method inapplicable, and (ii) system-level preferences cannot be directly transformed into component-level preferences. To address these challenges, we first formulate compound AI systems as Directed Acyclic Graphs (DAGs), explicitly modeling both component interactions and the associated data flows. Building on this formulation, we introduce $\textbf{SysDPO}$, a framework that extends Direct Preference Optimization (DPO) to enable joint system-level alignment. We propose two variants, SysDPO-Direct and SysDPO-Sampling, tailored for scenarios depending on whether we construct a system-specific preference dataset. We empirically demonstrate the effectiveness of our approach across two applications: the joint alignment of a language model and a diffusion model, and the joint alignment of an LLM collaboration system.
comment : NeurIPS 2025
♻ ☆ Beyond Pixels: Efficient Dataset Distillation via Sparse Gaussian Representation
Chenyang Jiang, Zhengcen Li, Hang Zhao, Qiben Shan, Shaocong Wu, Jingyong Su
Dataset distillation has emerged as a promising paradigm that synthesizes compact, informative datasets capable of retaining the knowledge of large-scale counterparts, thereby addressing the substantial computational and storage burdens of modern model training. Conventional approaches typically rely on dense pixel-level representations, which introduce redundancy and are difficult to scale up. In this work, we propose GSDD, a novel and efficient sparse representation for dataset distillation based on 2D Gaussians. Instead of representing all pixels equally, GSDD encodes critical discriminative information in a distilled image using only a small number of Gaussian primitives. This sparse representation could improve dataset diversity under the same storage budget, enhancing coverage of difficult samples and boosting distillation performance. To ensure both efficiency and scalability, we adapt CUDA-based splatting operators for parallel inference and training, enabling high-quality rendering with minimal computational and memory overhead. Our method is simple yet effective, broadly applicable to different distillation pipelines, and highly scalable. Experiments show that GSDD achieves state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet subsets, while remaining highly efficient encoding and decoding cost. Our code is available at https://github.com/j-cyoung/GSDatasetDistillation.
comment : 19 pages; Code is available on https://github.com/j-cyoung/GSDatasetDistillation
♻ ☆ Dominated Actions in Imperfect- Information Games
Sam Ganzfried
Dominance is a fundamental concept in game theory. In normal-form games dominated strategies can be identified in polynomial time. As a consequence, iterative removal of dominated strategies can be performed efficiently as a preprocessing step for reducing the size of a game before computing a Nash equilibrium. For imperfect-information games in extensive form, we could convert the game to normal form and then iteratively remove dominated strategies in the same way; however, this conversion may cause an exponential blowup in game size. In this paper we define and study the concept of dominated actions in imperfect-information games. Our main result is a polynomial-time algorithm for determining whether an action is dominated (strictly or weakly) by any mixed strategy in n-player games, which can be extended to an algorithm for iteratively removing dominated actions. This allows us to efficiently reduce the size of the game tree as a preprocessing step for Nash equilibrium computation. We explore the role of dominated actions empirically in "All In or Fold" No-Limit Texas Hold'em poker.
♻ ☆ MaxSup: Overcoming Representation Collapse in Label Smoothing NeurIPS 2025
Yuxuan Zhou, Heng Li, Zhi-Qi Cheng, Xudong Yan, Yifei Dong, Mario Fritz, Margret Keuper
Label Smoothing (LS) is widely adopted to reduce overconfidence in neural network predictions and improve generalization. Despite these benefits, recent studies reveal two critical issues with LS. First, LS induces overconfidence in misclassified samples. Second, it compacts feature representations into overly tight clusters, diluting intra-class diversity, although the precise cause of this phenomenon remained elusive. In this paper, we analytically decompose the LS-induced loss, exposing two key terms: (i) a regularization term that dampens overconfidence only when the prediction is correct, and (ii) an error-amplification term that arises under misclassifications. This latter term compels the network to reinforce incorrect predictions with undue certainty, exacerbating representation collapse. To address these shortcomings, we propose Max Suppression (MaxSup), which applies uniform regularization to both correct and incorrect predictions by penalizing the top-1 logit rather than the ground-truth logit. Through extensive feature-space analyses, we show that MaxSup restores intra-class variation and sharpens inter-class boundaries. Experiments on large-scale image classification and multiple downstream tasks confirm that MaxSup is a more robust alternative to LS. Code is available at: https://github.com/ZhouYuxuanYX/Maximum-Suppression-Regularization
comment : NeurIPS 2025 Oral (0.36% acceptance); code: https://github.com/ZhouYuxuanYX/Maximum-Suppression-Regularization
♻ ☆ Persona Agent with GraphRAG: Community-Aware Knowledge Graphs for Personalized LLM
Siqi Liang, Yudi Zhang, Yue Guo
We propose a novel framework for persona-based language model system, motivated by the need for personalized AI agents that adapt to individual user preferences. In our approach, the agent embodies the user's "persona" (e.g. user profile or taste) and is powered by a large language model (LLM). To enable the agent to leverage rich contextual information, we introduce a Knowledge-Graph-enhanced Retrieval-Augmented Generation (Graph RAG) mechanism that constructs an LLM-derived graph index of relevant documents and summarizes communities of related information. Our framework generates personalized prompts by combining: (1) a summary of the user's historical behaviors and preferences extracted from the knowledge graph, and (2) relevant global interaction patterns identified through graph-based community detection. This dynamic prompt engineering approach allows the agent to maintain consistent persona-aligned behaviors while benefiting from collective knowledge. On the LaMP benchmark, our method improves news categorization F1 by 11.1%, movie tagging F1 by 56.1%, and reduces product rating MAE by 10.4% over prior methods. Our code is available at https://anonymous.4open.science/r/PersonaAgentwGraphRAG-DE6F
♻ ☆ VeriLoRA: Fine-Tuning Large Language Models with Verifiable Security via Zero-Knowledge Proofs NDSS
Guofu Liao, Taotao Wang, Shengli Zhang, Jiqun Zhang, Shi Long, Dacheng Tao
Fine-tuning large language models (LLMs) is crucial for adapting them to specific tasks, yet it remains computationally demanding and raises concerns about correctness and privacy, particularly in untrusted environments. Although parameter-efficient methods like Low-Rank Adaptation (LoRA) significantly reduce resource requirements, ensuring the security and verifiability of fine-tuning under zero-knowledge constraints remains an unresolved challenge. To address this, we introduce VeriLoRA, the first framework to integrate LoRA fine-tuning with zero-knowledge proofs (ZKPs), achieving provable security and correctness. VeriLoRA employs advanced cryptographic techniques -- such as lookup arguments, sumcheck protocols, and polynomial commitments -- to verify both arithmetic and non-arithmetic operations in Transformer-based architectures. The framework provides end-to-end verifiability for forward propagation, backward propagation, and parameter updates during LoRA fine-tuning, while safeguarding the privacy of model parameters and training data. Leveraging GPU-based implementations, VeriLoRA demonstrates practicality and efficiency through experimental validation on open-source LLMs like LLaMA, scaling up to 13 billion parameters. By combining parameter-efficient fine-tuning with ZKPs, VeriLoRA bridges a critical gap, enabling secure and trustworthy deployment of LLMs in sensitive or untrusted environments.
comment : This paper has been accepted for publication at the Network and Distributed System Security Symposium (NDSS) 2026
♻ ☆ Adversarial Diffusion for Robust Reinforcement Learning
Daniele Foffano, Alessio Russo, Alexandre Proutiere
Robustness to modeling errors and uncertainties remains a central challenge in reinforcement learning (RL). In this work, we address this challenge by leveraging diffusion models to train robust RL policies. Diffusion models have recently gained popularity in model-based RL due to their ability to generate full trajectories "all at once", mitigating the compounding errors typical of step-by-step transition models. Moreover, they can be conditioned to sample from specific distributions, making them highly flexible. We leverage conditional sampling to learn policies that are robust to uncertainty in environment dynamics. Building on the established connection between Conditional Value at Risk (CVaR) optimization and robust RL, we introduce Adversarial Diffusion for Robust Reinforcement Learning (AD-RRL). AD-RRL guides the diffusion process to generate worst-case trajectories during training, effectively optimizing the CVaR of the cumulative return. Empirical results across standard benchmarks show that AD-RRL achieves superior robustness and performance compared to existing robust RL methods.
♻ ☆ DynTaskMAS: A Dynamic Task Graph-driven Framework for Asynchronous and Parallel LLM-based Multi- Agent Systems
Junwei Yu, Yepeng Ding, Hiroyuki Sato
The emergence of Large Language Models (LLMs) in Multi-Agent Systems (MAS) has opened new possibilities for artificial intelligence, yet current implementations face significant challenges in resource management, task coordination, and system efficiency. While existing frameworks demonstrate the potential of LLM-based agents in collaborative problem-solving, they often lack sophisticated mechanisms for parallel execution and dynamic task management. This paper introduces DynTaskMAS, a novel framework that orchestrates asynchronous and parallel operations in LLM-based MAS through dynamic task graphs. The framework features four key innovations: (1) a Dynamic Task Graph Generator that intelligently decomposes complex tasks while maintaining logical dependencies, (2) an Asynchronous Parallel Execution Engine that optimizes resource utilization through efficient task scheduling, (3) a Semantic-Aware Context Management System that enables efficient information sharing among agents, and (4) an Adaptive Workflow Manager that dynamically optimizes system performance. Experimental evaluations demonstrate that DynTaskMAS achieves significant improvements over traditional approaches: a 21-33% reduction in execution time across task complexities (with higher gains for more complex tasks), a 35.4% improvement in resource utilization (from 65% to 88%), and near-linear throughput scaling up to 16 concurrent agents (3.47X improvement for 4X agents). Our framework establishes a foundation for building scalable, high-performance LLM-based multi-agent systems capable of handling complex, dynamic tasks efficiently.
♻ ☆ From Atomic to Composite: Reinforcement Learning Enables Generalization in Complementary Reasoning
Sitao Cheng, Xunjian Yin, Ruiwen Zhou, Yuxuan Li, Xinyi Wang, Liangming Pan, William Yang Wang, Victor Zhong
The mechanism by which RL contributes to reasoning capabilities-whether it incentivizes the synthesis of new skills or merely amplifies existing behaviors-remains a subject of intense debate. In this work, we investigate this question through the lens of Complementary Reasoning, a complex task that requires integrating internal parametric knowledge with external contextual information. Using a controlled synthetic dataset of human biographies, we strictly decouple this ability into two atomic skills: Parametric Reasoning (relying on internal knowledge) and Contextual Reasoning (depending on external information). To rigorously assess capability boundaries, we evaluate generalization across three distinct levels of difficulty: I.I.D., Composition, and Zero-shot settings. We find that while SFT is sufficient for in-distribution performance, it struggles with O.O.D. generalization, particularly in Zero-shot settings where relational combinations are novel. Crucially, we identify the SFT Generalization Paradox: Models supervised solely on the composite task achieve near-perfect in-distribution accuracy but collapse on out-of-distribution generalization, indicating their reliance on rote memorization of path shortcuts. In contrast, we find that RL acts as a reasoning synthesizer rather than a probability amplifier. However, we uncover a strict atomic prerequisite: RL can only synthesize these complex strategies if the base model has first mastered the independent atomic skills (Parametric and Contextual) via SFT. These findings challenge the view of RL as a mere amplifier, suggesting that given sufficient atomic foundations, RL can actively synthesize complex reasoning strategies from learned primitives without explicit supervision on such complex strategies. This indicates that decoupled atomic training followed by RL offers a scalable path to generalization for complex reasoning tasks.
comment : Work in Progress. Code and data will be available at https://github.com/sitaocheng/from_atomic_to_composite
♻ ☆ MIMIC-MJX: Neuromechanical Emulation of Animal Behavior
Charles Y. Zhang, Yuanjia Yang, Aidan Sirbu, Elliott T. T. Abe, Emil Wärnberg, Eric J. Leonardis, Diego E. Aldarondo, Adam Lee, Aaditya Prasad, Jason Foat, Kaiwen Bian, Joshua Park, Rusham Bhatt, Hutton Saunders, Akira Nagamori, Ayesha R. Thanawalla, Kee Wui Huang, Fabian Plum, Hendrik K. Beck, Steven W. Flavell, David Labonte, Blake A. Richards, Bingni W. Brunton, Eiman Azim, Bence P. Ölveczky, Talmo D. Pereira
The primary output of the nervous system is movement and behavior. While recent advances have democratized pose tracking during complex behavior, kinematic trajectories alone provide only indirect access to the underlying control processes. Here we present MIMIC-MJX, a framework for learning biologically-plausible neural control policies from kinematics. MIMIC-MJX models the generative process of motor control by training neural controllers that learn to actuate biomechanically-realistic body models in physics simulation to reproduce real kinematic trajectories. We demonstrate that our implementation is accurate, fast, data-efficient, and generalizable to diverse animal body models. Policies trained with MIMIC-MJX can be utilized to both analyze neural control strategies and simulate behavioral experiments, illustrating its potential as an integrative modeling framework for neuroscience.
comment : Corrected LaTeX issues. Project page available at https://mimic-mjx.talmolab.org
♻ ☆ XXLTraffic: Expanding and Extremely Long Traffic forecasting beyond test adaptation SP
Du Yin, Hao Xue, Arian Prabowo, Shuang Ao, Flora Salim
Traffic forecasting is crucial for smart cities and intelligent transportation initiatives, where deep learning has made significant progress in modeling complex spatio-temporal patterns in recent years. However, current public datasets have limitations in reflecting the distribution shift nature of real-world scenarios, characterized by continuously evolving infrastructures, varying temporal distributions, and long temporal gaps due to sensor downtimes or changes in traffic patterns. These limitations inevitably restrict the practical applicability of existing traffic forecasting datasets. To bridge this gap, we present XXLTraffic, largest available public traffic dataset with the longest timespan collected from Los Angeles, USA, and New South Wales, Australia, curated to support research in extremely long forecasting beyond test adaptation. Our benchmark includes both typical time-series forecasting settings with hourly and daily aggregated data and novel configurations that introduce gaps and down-sample the training size to better simulate practical constraints. We anticipate the new XXLTraffic will provide a fresh perspective for the time-series and traffic forecasting communities. It would also offer a robust platform for developing and evaluating models designed to tackle the extremely long forecasting problems beyond test adaptation. Our dataset supplements existing spatio-temporal data resources and leads to new research directions in this domain.
comment : Updated version. SIGSPATIAL 2025 Best Paper
♻ ☆ CSI-BERT2: A BERT-inspired Framework for Efficient CSI Prediction and Classification in Wireless Communication and Sensing
Zijian Zhao, Fanyi Meng, Zhonghao Lyu, Hang Li, Xiaoyang Li, Guangxu Zhu
Channel state information (CSI) is a fundamental component in both wireless communication and sensing systems, enabling critical functions such as radio resource optimization and environmental perception. In wireless sensing, data scarcity and packet loss hinder efficient model training, while in wireless communication, high-dimensional CSI matrices and short coherent times caused by high mobility present challenges in CSI estimation. To address these issues, we propose a unified framework named CSI-BERT2 for CSI prediction and classification tasks, built on CSI-BERT, which adapts BERT to capture the complex relationships among CSI sequences through a bidirectional self-attention mechanism. We introduce a two-stage training method that first uses a mask language model (MLM) to enable the model to learn general feature extraction from scarce datasets in an unsupervised manner, followed by fine-tuning for specific downstream tasks. Specifically, we extend MLM into a mask prediction model (MPM), which efficiently addresses the CSI prediction task. To further enhance the representation capacity of CSI data, we modify the structure of the original CSI-BERT. We introduce an adaptive re-weighting layer (ARL) to enhance subcarrier representation and a multi-layer perceptron (MLP)-based temporal embedding module to mitigate temporal information loss problem inherent in the original Transformer. Extensive experiments on both real-world collected and simulated datasets demonstrate that CSI-BERT2 achieves state-of-the-art performance across all tasks. Our results further show that CSI-BERT2 generalizes effectively across varying sampling rates and robustly handles discontinuous CSI sequences caused by packet loss-challenges that conventional methods fail to address. The dataset and code are publicly available at https://github.com/RS2002/CSI-BERT2 .
♻ ☆ Proactive Agent ic Whiteboards: Enhancing Diagrammatic Learning
Suveen Ellawela, Sashenka Gamage, Dinithi Dissanayake
Educators frequently rely on diagrams to explain complex concepts during lectures, yet creating clear and complete visual representations in real time while simultaneously speaking can be cognitively demanding. Incomplete or unclear diagrams may hinder student comprehension, as learners must mentally reconstruct missing information while following the verbal explanation. Inspired by advances in code completion tools, we introduce DrawDash, an AI-powered whiteboard assistant that proactively completes and refines educational diagrams through multimodal understanding. DrawDash adopts a TAB-completion interaction model: it listens to spoken explanations, detects intent, and dynamically suggests refinements that can be accepted with a single keystroke. We demonstrate DrawDash across four diverse teaching scenarios, spanning topics from computer science and web development to biology. This work represents an early exploration into reducing instructors' cognitive load and improving diagram-based pedagogy through real-time, speech-driven visual assistance, and concludes with a discussion of current limitations and directions for formal classroom evaluation.
♻ ☆ OmniBench: Towards The Future of Universal Omni-Language Models
Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang, Chenghua Lin
Recent advancements in multimodal large language models (MLLMs) have focused on integrating multiple modalities, yet their ability to simultaneously process and reason across different inputs remains underexplored. We introduce OmniBench, a novel benchmark designed to evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. We define language models capable of such tri-modal processing as omni-language models (OLMs). OmniBench features high-quality human annotations that require integrated understanding across all modalities. Our evaluation reveals that: i) open-source OLMs show significant limitations in instruction-following and reasoning in tri-modal contexts; and ii) most baseline models perform poorly (around 50% accuracy) even with textual alternatives to image/audio inputs. To address these limitations, we develop OmniInstruct, an 96K-sample instruction tuning dataset for training OLMs. We advocate for developing more robust tri-modal integration techniques and training strategies to enhance OLM performance. Codes and data could be found at our repo (https://github.com/multimodal-art-projection/OmniBench).
♻ ☆ ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning
Jingyang Yi, Jiazheng Wang, Sida Li
Recent models such as OpenAI o1 and DeepSeek-R1 have demonstrated strong performance on reasoning-intensive tasks by generating extended Chain-of-Thought (CoT) traces. While longer reasoning helps with thorough exploration of solution paths for complex problems, it also often leads to inefficient and redundant outputs--a phenomenon commonly described as overthinking. In this paper, we propose ShorterBetter, a simple yet effective reinforcement learning method that enables reasoning models to learn their own optimal CoT lengths without manual supervision. We define the Sample Optimal Length (SOL) as the length of the shortest correct response among multiple generations, which serves as a dynamic reward signal to guide the model toward efficient reasoning. Applied to DeepSeek-Distill-Qwen-1.5B/7B as base models, ShorterBetter achieves 50%-80% reduction in output lengths in both in-domain and out-of-domain reasoning tasks while maintaining accuracy. Our reasoning trace analysis shows that ShorterBetter refines the structure of the reasoning traces by reducing unnecessary repetition, excessive self-verification, and over-exploration of alternatives.
comment : updated project website
♻ ☆ SQLBarber: A System Leveraging Large Language Models to Generate Customized and Realistic SQL Workloads SIGMOD 2026
Jiale Lao, Immanuel Trummer
Database research and development often require a large number of SQL queries for benchmarking purposes. However, acquiring real-world SQL queries is challenging due to privacy concerns, and existing SQL generation methods are limited in customization and in satisfying realistic constraints. To address this issue, we present SQLBarber, a system based on Large Language Models (LLMs) to generate customized and realistic SQL workloads. SQLBarber (i) eliminates the need for users to manually craft SQL templates in advance, while providing the flexibility to accept natural language specifications to constrain SQL templates, (ii) scales efficiently to generate large volumes of queries matching any user-defined cost distribution (e.g., cardinality and execution plan cost), and (iii) uses execution statistics from Amazon Redshift and Snowflake to derive SQL template specifications and query cost distributions that reflect real-world query characteristics. SQLBarber introduces (i) a declarative interface for users to effortlessly generate customized SQL templates, (ii) an LLM-powered pipeline augmented with a self-correction module that profiles, refines, and prunes SQL templates based on query costs, and (iii) a Bayesian Optimizer to efficiently explore different predicate values and identify a set of queries that satisfy the target cost distribution. We construct and open-source ten benchmarks of varying difficulty levels and target query cost distributions based on real-world statistics from Snowflake and Amazon Redshift. Extensive experiments on these benchmarks show that SQLBarber is the only system that can generate customized SQL templates. It reduces query generation time by one to three orders of magnitude, and significantly improves alignment with the target cost distribution, compared with existing methods.
comment : Accepted by SIGMOD 2026; extended version with appendix
♻ ☆ Diffusion Model in Latent Space for Medical Image Segmentation Task
Huynh Trinh Ngoc, Toan Nguyen Hai, Ba Luong Son, Long Tran Quoc
Medical image segmentation is crucial for clinical diagnosis and treatment planning. Traditional methods typically produce a single segmentation mask, failing to capture inherent uncertainty. Recent generative models enable the creation of multiple plausible masks per image, mimicking the collaborative interpretation of several clinicians. However, these approaches remain computationally heavy. We propose MedSegLatDiff, a diffusion based framework that combines a variational autoencoder (VAE) with a latent diffusion model for efficient medical image segmentation. The VAE compresses the input into a low dimensional latent space, reducing noise and accelerating training, while the diffusion process operates directly in this compact representation. We further replace the conventional MSE loss with weighted cross entropy in the VAE mask reconstruction path to better preserve tiny structures such as small nodules. MedSegLatDiff is evaluated on ISIC-2018 (skin lesions), CVC-Clinic (polyps), and LIDC-IDRI (lung nodules). It achieves state of the art or highly competitive Dice and IoU scores while simultaneously generating diverse segmentation hypotheses and confidence maps. This provides enhanced interpretability and reliability compared to deterministic baselines, making the model particularly suitable for clinical deployment.
♻ ☆ Rewarding the Journey, Not Just the Destination: A Composite Path and Answer Self-Scoring Reward Mechanism for Test-Time Reinforcement Learning
Chenwei Tang, Jingyu Xing, Lin Long, Xinyu Liu, Deng Xiong, Wei Ju, Shudong Huang, Jiancheng Lv, Ziyue Qiao
Reinforcement Learning (RL) has emerged as a powerful paradigm for advancing Large Language Models (LLMs), achieving remarkable performance in complex reasoning domains such as mathematics and code generation. However, current RL methods face a fundamental scalability bottleneck due to their heavy reliance on human-curated preference data or labeled datasets for reward modeling. To overcome this limitation, we explore RL on unlabeled data where models learn autonomously from continuous experience streams. The core challenge in this setting lies in reliable reward estimation without ground-truth supervision. Existing approaches like Test-Time RL address this through self-consistent consensus, but risk reinforcing incorrect pseudo-labels derived from majority voting. We introduce COMPASS (Composite Path and Answer Self-Scoring), a novel test-time reward mechanism that operates without external supervision. COMPASS integrates two complementary components: the Dual-Calibration Answer Reward (DCAR), which stabilizes training by establishing trustworthy pseudo-labels through confidence and credibility calibration, and the Decisive Path Reward (DPR), which directly optimizes the reasoning process quality beyond mere outcome supervision. By jointly reinforcing trustworthy consensus answers and highly decisive reasoning chains, the COMPASS systematically enhances the model's analytical capabilities. Extensive experiments show that COMPASS achieves significant and consistent performance gains across diverse reasoning tasks and model architectures, advancing a more scalable direction for LLMs to learn from continuous experience.
♻ ☆ AdvisingWise: Supporting Academic Advising in Higher Education Settings Through a Human-in-the-Loop Multi- Agent Framework
Wendan Jiang, Shiyuan Wang, Hiba Eltigani, Rukhshan Haroon, Abdullah Bin Faisal, Fahad Dogar
Academic advising is critical to student success in higher education, yet high student-to-advisor ratios limit advisors' capacity to provide timely support, particularly during peak periods. Recent advances in Large Language Models (LLMs) present opportunities to enhance the advising process. We present AdvisingWise, a multi-agent system that automates time-consuming tasks, such as information retrieval and response drafting, while preserving human oversight. AdvisingWise leverages authoritative institutional resources and adaptively prompts students about their academic backgrounds to generate reliable, personalized responses. All system responses undergo human advisor validation before delivery to students. We evaluate AdvisingWise through a mixed-methods approach: (1) expert evaluation on responses of 20 sample queries, (2) LLM-as-a-judge evaluation of the information retrieval strategy, and (3) a user study with 8 academic advisors to assess the system's practical utility. Our evaluation shows that AdvisingWise produces accurate, personalized responses. Advisors reported increasingly positive perceptions after using AdvisingWise, as their initial concerns about reliability and personalization diminished. We conclude by discussing the implications of human-AI synergy on the practice of academic advising.
comment : 18 pages, 6 figures
♻ ☆ Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradation
Hao Guan, David Bates, Li Zhou
Artificial intelligence (AI) is increasingly integrated into modern healthcare, offering powerful support for clinical decision-making. However, in real-world settings, AI systems may experience performance degradation over time, due to factors such as shifting data distributions, changes in patient characteristics, evolving clinical protocols, and variations in data quality. These factors can compromise model reliability, posing safety concerns and increasing the likelihood of inaccurate predictions or adverse outcomes. This review presents a forward-looking perspective on monitoring and maintaining the "health" of AI systems in healthcare. We highlight the urgent need for continuous performance monitoring, early degradation detection, and effective self-correction mechanisms. The paper begins by reviewing common causes of performance degradation at both data and model levels. We then summarize key techniques for detecting data and model drift, followed by an in-depth look at root cause analysis. Correction strategies are further reviewed, ranging from model retraining to test-time adaptation. Our survey spans both traditional machine learning models and state-of-the-art large language models (LLMs), offering insights into their strengths and limitations. Finally, we discuss ongoing technical challenges and propose future research directions. This work aims to guide the development of reliable, robust medical AI systems capable of sustaining safe, long-term deployment in dynamic clinical settings.
comment : 16 pages, 5 figures
♻ ☆ Neural Cellular Automata for ARC-AGI
Kevin Xu, Risto Miikkulainen
Cellular automata and their differentiable counterparts, Neural Cellular Automata (NCA), are highly expressive and capable of surprisingly complex behaviors. This paper explores how NCAs perform when applied to tasks requiring precise transformations and few-shot generalization, using the Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) as a domain that challenges their capabilities in ways not previously explored. Specifically, this paper uses gradient-based training to learn iterative update rules that transform input grids into their outputs from the training examples and apply them to the test inputs. Results suggest that gradient-trained NCA models are a promising and efficient approach to a range of abstract grid-based tasks from ARC. Along with discussing the impacts of various design modifications and training constraints, this work examines the behavior and properties of NCAs applied to ARC to give insights for broader applications of self-organizing systems.
comment : 8 pages, 5 figures
♻ ☆ Remotely sensing stress evolution in elastic media: a passive approach to earthquake monitoring
Nader Shakibay Senobari
Stress evolution governs material failure across scales, from microscopic fractures to large earthquakes, yet direct observation of its dynamics in natural systems has remained elusive. Laboratory experiments using active ultrasonic measurements have shown that seismic velocity and attenuation are sensitive to stress, but such monitoring has not previously been achievable remotely or passively. Here we introduce a stress-sensitive frequency-domain transform that enables passive monitoring of stress evolution using ambient seismic or acoustic noise. The method quantifies relative energy shifts between adjacent frequency bands, capturing subtle changes in wave-propagation properties linked to evolving shear and normal stress. Applied across scales, from laboratory stick-slip and slow-slip experiments to natural fault systems including the 2018 Kilauea collapse, Cascadia slow-slip episodes, and major earthquakes such as the 2011 Tohoku, 2010 Maule, 2002 Denali, and 2023 Turkey-Syria events, the transform consistently reveals distinctive precursory trajectories and stress-cycle patterns. These results demonstrate that stress evolution in elastic Earth materials can be remotely and passively monitored, bridging laboratory rock physics and large-scale seismology and offering a new foundation for real-time tracking of fault mechanics and earthquake preparation.
comment : 25 pages, 9 figures. Github code included
♻ ☆ Rainbow Noise: Stress-Testing Multimodal Harmful-Meme Detectors on LGBTQ Content
Ran Tong, Songtao Wei, Jiaqi Liu, Lanruo Wang
Hateful memes aimed at LGBTQ\,+ communities often evade detection by tweaking either the caption, the image, or both. We build the first robustness benchmark for this setting, pairing four realistic caption attacks with three canonical image corruptions and testing all combinations on the PrideMM dataset. Two state-of-the-art detectors, MemeCLIP and MemeBLIP2, serve as case studies, and we introduce a lightweight \textbf{Text Denoising Adapter (TDA)} to enhance the latter's resilience. Across the grid, MemeCLIP degrades more gently, while MemeBLIP2 is particularly sensitive to the caption edits that disrupt its language processing. However, the addition of the TDA not only remedies this weakness but makes MemeBLIP2 the most robust model overall. Ablations reveal that all systems lean heavily on text, but architectural choices and pre-training data significantly impact robustness. Our benchmark exposes where current multimodal safety models crack and demonstrates that targeted, lightweight modules like the TDA offer a powerful path towards stronger defences.
comment : 14 pages, 1 figure
♻ ☆ HeavyWater and SimplexWater: Distortion-Free LLM Watermarks for Low-Entropy Next-Token Predictions NeurIPS2025
Dor Tsur, Carol Xuan Long, Claudio Mayrink Verdun, Hsiang Hsu, Chen-Fu Chen, Haim Permuter, Sajani Vithana, Flavio P. Calmon
Large language model (LLM) watermarks enable authentication of text provenance, curb misuse of machine-generated text, and promote trust in AI systems. Current watermarks operate by changing the next-token predictions output by an LLM. The updated (i.e., watermarked) predictions depend on random side information produced, for example, by hashing previously generated tokens. LLM watermarking is particularly challenging in low-entropy generation tasks -- such as coding -- where next-token predictions are near-deterministic. In this paper, we propose an optimization framework for watermark design. Our goal is to understand how to most effectively use random side information in order to maximize the likelihood of watermark detection and minimize the distortion of generated text. Our analysis informs the design of two new watermarks: HeavyWater and SimplexWater. Both watermarks are tunable, gracefully trading-off between detection accuracy and text distortion. They can also be applied to any LLM and are agnostic to side information generation. We examine the performance of HeavyWater and SimplexWater through several benchmarks, demonstrating that they can achieve high watermark detection accuracy with minimal compromise of text generation quality, particularly in the low-entropy regime. Our theoretical analysis also reveals surprising new connections between LLM watermarking and coding theory. The code implementation can be found in https://github.com/DorTsur/HeavyWater_SimplexWater
comment : Presented at NeurIPS2025
♻ ☆ Mechanisms of Symbol Processing for In- Context Learning in Transformer Networks
Paul Smolensky, Roland Fernandez, Zhenghao Herbert Zhou, Mattia Opper, Adam Davies, Jianfeng Gao
Large Language Models (LLMs) have demonstrated impressive abilities in symbol processing through in-context learning (ICL). This success flies in the face of decades of critiques asserting that artificial neural networks cannot master abstract symbol manipulation. We seek to understand the mechanisms that can enable robust symbol processing in transformer networks, illuminating both the unanticipated success, and the significant limitations, of transformers in symbol processing. Borrowing insights from symbolic AI and cognitive science on the power of Production System architectures, we develop a high-level Production System Language, PSL, that allows us to write symbolic programs to do complex, abstract symbol processing, and create compilers that precisely implement PSL programs in transformer networks which are, by construction, 100% mechanistically interpretable. The work is driven by study of a purely abstract (semantics-free) symbolic task that we develop, Templatic Generation (TGT). Although developed through study of TGT, PSL is, we demonstrate, highly general: it is Turing Universal. The new type of transformer architecture that we compile from PSL programs suggests a number of paths for enhancing transformers' capabilities at symbol processing. We note, however, that the work we report addresses computability, and not learnability, by transformer networks. Note: The first section provides an extended synopsis of the entire paper.
♻ ☆ Reference Grounded Skill Discovery
Seungeun Rho, Aaron Trinh, Danfei Xu, Sehoon Ha
Scaling unsupervised skill discovery algorithms to high-DoF agents remains challenging. As dimensionality increases, the exploration space grows exponentially, while the manifold of meaningful skills remains limited. Therefore, semantic meaningfulness becomes essential to effectively guide exploration in high-dimensional spaces. In this work, we present **Reference-Grounded Skill Discovery (RGSD)**, a novel algorithm that grounds skill discovery in a semantically meaningful latent space using reference data. RGSD first performs contrastive pretraining to embed motions on a unit hypersphere, clustering each reference trajectory into a distinct direction. This grounding enables skill discovery to simultaneously involve both imitation of reference behaviors and the discovery of semantically related diverse behaviors. On a simulated SMPL humanoid with $359$-D observations and $69$-D actions, RGSD successfully imitates skills such as walking, running, punching, and sidestepping, while also discover variations of these behaviors. In downstream locomotion tasks, RGSD leverages the discovered skills to faithfully satisfy user-specified style commands and outperforms imitation-learning baselines, which often fail to maintain the commanded style. Overall, our results suggest that lightweight reference-grounding offers a practical path to discovering semantically rich and structured skills in high-DoF systems.
♻ ☆ Could AI Leapfrog the Web? Evidence from Teachers in Sierra Leone
Daniel Björkegren, Jun Ho Choi, Divya Budihal, Dominic Sobhani, Oliver Garrod, Paul Atherton
Only 37% of sub-Saharan Africans use the internet, and those who do seldom rely on traditional web search. A major reason is that bandwidth is scarce and costly. We study whether an AI-powered WhatsApp chatbot can bridge this gap by analyzing 40,350 queries submitted by 529 Sierra Leonean teachers over 17 months. Each month, more teachers relied on AI than web search for teaching assistance. We compare the AI responses to the top results from google.com.sl, which mostly returns web pages formatted for foreign users: just 2% of pages originate in-country. Also, each web page consumes 3,107 times more bandwidth than an AI response on average. As a result, querying AI through WhatsApp is 98% less expensive than loading a web page, even including AI compute costs. In blinded evaluations, an independent sample of teachers rate AI responses as more relevant, helpful, and correct answers to queries than web search results. These findings suggest that AI can provide cost-effective access to information in low-connectivity environments.
♻ ☆ Escaping the Verifier: Learning to Reason via Demonstrations
Locke Cai, Ivan Provilkov
Training Large Language Models (LLMs) to reason often relies on Reinforcement Learning (RL) with task-specific verifiers. However, many real-world reasoning-intensive tasks lack verifiers, despite offering abundant expert demonstrations that remain under-utilized for reasoning-focused training. We introduce RARO (Relativistic Adversarial Reasoning Optimization) that learns strong reasoning capabilities from only expert demonstrations via Inverse Reinforcement Learning. Our method sets up an adversarial interaction between a policy (generator) and a relativistic critic (discriminator): the policy learns to mimic expert answers, while the critic learns to compare and distinguish between policy and expert answers. Our method trains both the policy and the critic jointly and continuously via RL, and we identify the key stabilization techniques required for robust learning. Empirically, RARO significantly outperforms strong verifier-free baselines on all of our evaluation tasks -- Countdown, DeepMath, and Poetry Writing -- and enjoys the same robust scaling trends as RL on verifiable tasks. These results demonstrate that our method effectively elicits strong reasoning performance from expert demonstrations alone, enabling robust reasoning learning even when task-specific verifiers are unavailable.
♻ ☆ Real-Time Procedural Learning From Experience for AI Agent s
Dasheng Bi, Yubin Hu, Mohammed N. Nasir
Learning how to do things from trial and error in real time is a hallmark of biological intelligence, yet most LLM-based agents lack mechanisms to acquire procedural knowledge after deployment. We propose Procedural Recall for Agents with eXperiences Indexed by State (PRAXIS), a lightweight post-training learning mechanism that stores the consequences of actions and retrieves them by jointly matching environmental and internal states of past episodes to the current state. PRAXIS augments agentic action selection with retrieved state-action-result exemplars that are generated in real time. When evaluated on the REAL web browsing benchmark, PRAXIS improves task completion accuracy, reliability, and cost efficiency across different foundation model backbones, and shows preliminary generalization to unseen tasks in similar environments. These results demonstrate that PRAXIS enables the practical adoption of AI agents in fast-evolving stateful environments by helping them learn new procedures effectively.
♻ ☆ Monte Ca rl o-Type Neural Operator for Differential Equations
Salah Eddine Choutri, Prajwal Chauhan, Othmane Mazhar, Saif Eddin Jabari
The Monte Carlo-type Neural Operator (MCNO) introduces a framework for learning solution operators of one-dimensional partial differential equations (PDEs) by directly learning the kernel function and approximating the associated integral operator using a Monte Carlo-type approach. Unlike Fourier Neural Operators (FNOs), which rely on spectral representations and assume translation-invariant kernels, MCNO makes no such assumptions. The kernel is represented as a learnable tensor over sampled input-output pairs, and sampling is performed once, uniformly at random from a discretized grid. This design enables generalization across multiple grid resolutions without relying on fixed global basis functions or repeated sampling during training, while an interpolation step maps between arbitrary input and output grids to further enhance flexibility. Experiments on standard 1D PDE benchmarks show that MCNO achieves competitive accuracy with efficient computational cost. We also provide a theoretical analysis proving that the Monte Carlo estimator yields a bounded bias and variance under mild regularity assumptions. This result holds in any spatial dimension, suggesting that MCNO may extend naturally beyond one-dimensional problems. More broadly, this work explores how Monte Carlo-type integration can be incorporated into neural operator frameworks for continuous-domain PDEs, providing a theoretically supported alternative to spectral methods (such as FNO) and to graph-based Monte Carlo approaches (such as the Graph Kernel Neural Operator, GNO).
♻ ☆ SafeGenes: Evaluating the Adversarial Robustness of Genomic Foundation Models
Huixin Zhan, Clovis Barbour, Jason H. Moore
Genomic Foundation Models (GFMs), such as Evolutionary Scale Modeling (ESM), have demonstrated significant success in variant effect prediction. However, their adversarial robustness remains largely unexplored. To address this gap, we propose SafeGenes: a framework for Secure analysis of genomic foundation models, leveraging adversarial attacks to evaluate robustness against both engineered near-identical adversarial Genes and embedding-space manipulations. In this study, we assess the adversarial vulnerabilities of GFMs using two approaches: the Fast Gradient Sign Method (FGSM) and a soft prompt attack. FGSM introduces minimal perturbations to input sequences, while the soft prompt attack optimizes continuous embeddings to manipulate model predictions without modifying the input tokens. By combining these techniques, SafeGenes provides a comprehensive assessment of GFM susceptibility to adversarial manipulation. Targeted soft prompt attacks induced severe degradation in MLM-based shallow architectures such as ProteinBERT, while still producing substantial failure modes even in high-capacity foundation models such as ESM1b and ESM1v. These findings expose critical vulnerabilities in current foundation models, opening new research directions toward improving their security and robustness in high-stakes genomic applications such as variant effect prediction.
♻ ☆ ConfRover: Simultaneous Modeling of Protein Conformation and Dynamics via Autoregression NeurIPS 2025
Yuning Shen, Lihao Wang, Huizhuo Yuan, Yan Wang, Bangji Yang, Quanquan Gu
Understanding protein dynamics is critical for elucidating their biological functions. The increasing availability of molecular dynamics (MD) data enables the training of deep generative models to efficiently explore the conformational space of proteins. However, existing approaches either fail to explicitly capture the temporal dependencies between conformations or do not support direct generation of time-independent samples. To address these limitations, we introduce ConfRover, an autoregressive model that simultaneously learns protein conformation and dynamics from MD trajectories, supporting both time-dependent and time-independent sampling. At the core of our model is a modular architecture comprising: (i) an encoding layer, adapted from protein folding models, that embeds protein-specific information and conformation at each time frame into a latent space; (ii) a temporal module, a sequence model that captures conformational dynamics across frames; and (iii) an SE(3) diffusion model as the structure decoder, generating conformations in continuous space. Experiments on ATLAS, a large-scale protein MD dataset of diverse structures, demonstrate the effectiveness of our model in learning conformational dynamics and supporting a wide range of downstream tasks. ConfRover is the first model to sample both protein conformations and trajectories within a single framework, offering a novel and flexible approach for learning from protein MD data. Project website: https://bytedance-seed.github.io/ConfRover.
comment : 35 pages, 17 figures; Camera ready for NeurIPS 2025; Website: https://bytedance-seed.github.io/ConfRover
♻ ☆ Kodezi Chronos: A Debugging-First Language Model for Repository-Scale Code Understanding
Ishraq Khan, Assad Chowdary, Sharoz Haseeb, Urvish Patel, Yousuf Zaii
Large Language Models (LLMs) have advanced code generation and software automation but remain constrained by inference-time context and lack structured reasoning over code, leaving debugging largely unsolved. While Claude 4.5 Opus achieves 74.40% on SWE-bench Verified and Gemini 3 Pro reaches 76.2%, both models remain below 20% on real multi-file debugging tasks. We introduce Kodezi Chronos-1, a language model purpose-built for debugging that integrates Adaptive Graph-Guided Retrieval to navigate codebases up to 10 million lines (92% precision, 85% recall), Persistent Debug Memory trained on over 15 million sessions, and a seven-layer fix-test-refine architecture. On 5,000 real-world scenarios, Chronos-1 achieves 67.3% +/- 2.1% fix accuracy compared to 14.2% +/- 1.3% for Claude 4.1 Opus and 13.8% +/- 1.2% for GPT-4.1 (Cohen's d = 3.87). On SWE-bench Lite, Chronos-1 reaches a state-of-the-art 80.33% resolution rate (241 of 300), outperforming the next best system by 20 points and achieving repository-specific highs of 96.1% on Sympy and 90.4% on Django. Chronos-1 reduces debugging time by 40% and iterations by 65%, resolving complex multi-file and cross-repository bugs that require temporal analysis. Limitations remain for hardware-dependent and dynamic language errors, and Chronos-1 will be available in Kodezi OS in Q4 2025 and via API in Q1 2026.
comment : 24 figures, 43 tables, 2 algorithms. Extended technical report introducing Chronos-1, a debugging-specific language model. Information available at https://github.com/Kodezi/chronos
♻ ☆ TeamMed Agent s: Enhancing Medical Decision-Making of LLMs Through Structured Teamwork
Pranav Pushkar Mishra, Mohammad Arvan, Mohan Zalake
We present TeamMedAgents, a modular multi-agent framework that systematically translates evidence-based teamwork principles from organizational psychology into large language model collaboration for medical decision-making. Building upon Salas et al.'s "Big Five" teamwork model, we operationalize five core components as independently configurable mechanisms: shared mental models, team leadership, team orientation, trust networks, and mutual monitoring. Our architecture dynamically recruits 2-4 specialist agents and employs structured four-phase deliberation with adaptive component selection. Evaluation across eight medical benchmarks encompassing 11,545 questions demonstrates TeamMedAgents achieves 77.63% overall accuracy (text-based: 81.30%, vision-language: 66.60%). Systematic ablation studies comparing three single-agent baselines (Zero-Shot, Few-Shot, CoT) against individual teamwork components reveal task-specific optimization patterns: shared mental models excel on knowledge tasks, trust mechanisms improve differential diagnosis, while comprehensive integration degrades performance. Adaptive component selection yields 2-10 percentage point improvements over strongest baselines, with 96.2% agent convergence validating structured coordination effectiveness. TeamMedAgents establishes principled methodology for translating human teamwork theory into multi-agent systems, demonstrating that evidence-based collaboration patterns enhance AI performance in safety-critical domains through modular component design and selective activation strategies.
comment : 12 pages, 1 figure, 3 tables, 1 algorithm(in appendix)
♻ ☆ Knowledge Graph Augmented Large Language Models for Disease Prediction
Ruiyu Wang, Tuan Vinh, Ran Xu, Yuyin Zhou, Jiaying Lu, Carl Yang, Francisco Pasquel
Electronic health records (EHRs) support powerful clinical prediction models, but existing methods typically provide coarse, post hoc explanations that offer limited value for patient-level decision making. We introduce a knowledge graph (KG)-guided chain-of-thought (CoT) framework that generates clinically grounded and temporally consistent reasoning for visit-level disease prediction in MIMIC-III. ICD-9 codes are mapped to PrimeKG, from which disease-relevant nodes and multi-hop reasoning paths are extracted and used as scaffolds for CoT generation; only explanations whose conclusions match observed outcomes are retained. Lightweight LLaMA-3.1-Instruct-8B and Gemma-7B models are then fine-tuned on this supervision corpus. Across ten PrimeKG-mapped diseases and limited training cohorts (400 and 1000 cases), KG-guided models outperform strong classical baselines, achieving AUROC values of 0.66 to 0.70 and macro-AUPR values of 0.40 to 0.47. The models also transfer zero-shot to the CRADLE cohort, improving accuracy from approximately 0.40 to 0.51 up to 0.72 to 0.77. A blinded clinician evaluation shows consistent preference for KG-guided CoT explanations in clarity, relevance, and clinical correctness.
♻ ☆ Let Them Down Easy! Context ual Effects of LLM Guardrails on User Perceptions and Preferences EMNLP 2025
Mingqian Zheng, Wenjia Hu, Patrick Zhao, Motahhare Eslami, Jena D. Hwang, Faeze Brahman, Carolyn Rose, Maarten Sap
Current LLMs are trained to refuse potentially harmful input queries regardless of whether users actually had harmful intents, causing a tradeoff between safety and user experience. Through a study of 480 participants evaluating 3,840 query-response pairs, we examine how different refusal strategies affect user perceptions across varying motivations. Our findings reveal that response strategy largely shapes user experience, while actual user motivation has negligible impact. Partial compliance -- providing general information without actionable details -- emerges as the optimal strategy, reducing negative user perceptions by over 50% to flat-out refusals. Complementing this, we analyze response patterns of 9 state-of-the-art LLMs and evaluate how 6 reward models score different refusal strategies, demonstrating that models rarely deploy partial compliance naturally and reward models currently undervalue it. This work demonstrates that effective guardrails require focusing on crafting thoughtful refusals rather than detecting intent, offering a path toward AI safety mechanisms that ensure both safety and sustained user engagement.
comment : Accepted to Findings of EMNLP 2025
♻ ☆ Astra: A Multi- Agent System for GPU Kernel Performance Optimization
Anjiang Wei, Tianran Sun, Yogesh Seenichamy, Hang Song, Anne Ouyang, Azalia Mirhoseini, Ke Wang, Alex Aiken
GPU kernel optimization has long been a central challenge at the intersection of high-performance computing and machine learning. Efficient kernels are crucial for accelerating large language model (LLM) training and serving, yet attaining high performance typically requires extensive manual tuning. Compiler-based systems reduce some of this burden, but still demand substantial manual design and engineering effort. Recently, researchers have explored using LLMs for GPU kernel generation, though prior work has largely focused on translating high-level PyTorch modules into CUDA code. In this work, we introduce Astra, the first LLM-based multi-agent system for GPU kernel optimization. Unlike previous approaches, Astra starts from existing CUDA implementations extracted from SGLang, a widely deployed framework for serving LLMs, rather than treating PyTorch modules as the specification. Within Astra, specialized LLM agents collaborate through iterative code generation, testing, profiling, and planning to produce kernels that are both correct and high-performance. On kernels from SGLang, Astra achieves an average speedup of 1.32x using zero-shot prompting with OpenAI o4-mini. A detailed case study further demonstrates that LLMs can autonomously apply loop transformations, optimize memory access patterns, exploit CUDA intrinsics, and leverage fast math operations to yield substantial performance gains. Our work highlights multi-agent LLM systems as a promising new paradigm for GPU kernel optimization. Our code is publicly available at https://github.com/Anjiang-Wei/Astra.
♻ ☆ Large Language Models: An Applied Econometric Framework
Jens Ludwig, Sendhil Mullainathan, Ashesh Rambachan
Large language models (LLMs) enable researchers to analyze text at unprecedented scale and minimal cost. Researchers can now revisit old questions and tackle novel ones with rich data. We provide an econometric framework for realizing this potential in two empirical uses. For prediction problems -- forecasting outcomes from text -- valid conclusions require ``no training leakage'' between the LLM's training data and the researcher's sample, which can be enforced through careful model choice and research design. For estimation problems -- automating the measurement of economic concepts for downstream analysis -- valid downstream inference requires combining LLM outputs with a small validation sample to deliver consistent and precise estimates. Absent a validation sample, researchers cannot assess possible errors in LLM outputs, and consequently seemingly innocuous choices (which model, which prompt) can produce dramatically different parameter estimates. When used appropriately, LLMs are powerful tools that can expand the frontier of empirical economics.
♻ ☆ From Distance to Direction: Structure-aware Label-specific Feature Fusion for Label Distribution Learning
Suping Xu, Chuyi Dai, Lin Shang, Changbin Shao, Xibei Yang, Witold Pedrycz
Label distribution learning (LDL) is an emerging learning paradigm designed to capture the relative importance of labels for each instance. Label-specific features (LSFs), constructed by LIFT, have proven effective for learning tasks with label ambiguity by leveraging clustering-based prototypes for each label to re-characterize instances. However, directly introducing LIFT into LDL tasks can be suboptimal, as the prototypes it collects primarily reflect intra-cluster relationships while neglecting cross-cluster interactions. Additionally, constructing LSFs using multi-perspective information, rather than relying solely on Euclidean distance, provides a more robust and comprehensive representation of instances, mitigating noise and bias that may arise from a single distance perspective. To address these limitations, we introduce Structural Anchor Points (SAPs) to capture inter-cluster interactions. This leads to a novel LSFs construction strategy, LIFT-SAP, which enhances LIFT by integrating both distance and directional information of each instance relative to SAPs. Furthermore, we propose a novel LDL algorithm, Label Distribution Learning via Label-specifIc FeaTure with SAPs (LDL-LIFT-SAP), which unifies multiple label description degrees predicted from different LSF spaces into a cohesive label distribution. Extensive experiments on 15 real-world datasets demonstrate the effectiveness of LIFT-SAP over LIFT, as well as the superiority of LDL-LIFT-SAP compared to seven other well-established algorithms.
♻ ☆ Observation-Free Attacks on Online Learning to Rank
Sameep Chattopadhyay, Nikhil Karamchandani, Sharayu Moharir
Online learning to rank (OLTR) plays a critical role in information retrieval and machine learning systems, with a wide range of applications in search engines and content recommenders. However, despite their extensive adoption, the susceptibility of OLTR algorithms to coordinated adversarial attacks remains poorly understood. In this work, we present a novel framework for attacking some of the widely used OLTR algorithms. Our framework is designed to promote a set of target items so that they appear in the list of top-K recommendations for T - o(T) rounds, while simultaneously inducing linear regret in the learning algorithm. We propose two novel attack strategies: CascadeOFA for CascadeUCB1 and PBMOFA for PBM-UCB . We provide theoretical guarantees showing that both strategies require only O(log T) manipulations to succeed. Additionally, we supplement our theoretical analysis with empirical results on real-world data.
♻ ☆ PerfBench: Can Agent s Resolve Real-Wo rl d Performance Bugs?
Spandan Garg, Roshanak Zilouchian Moghaddam, Neel Sundaresan
Performance bugs are inefficiencies in software that waste computational resources without causing functional failures, making them particularly challenging to detect and fix. While recent advances in Software Engineering agents have shown promise in automated bug fixing, existing benchmarks primarily focus on functional correctness and fail to evaluate agents' abilities to identify and resolve non-functional issues like performance bugs. We introduce PerfBench, a benchmark comprising 81 real-world performance bug-fixing tasks from popular .NET repositories on GitHub. Unlike existing benchmarks that rely on pre-existing test suites, PerfBench features a novel evaluation harness that allows agents to generate their own performance benchmarks and validates fixes by comparing execution metrics collected for developer fix and agent fix. Each task in PerfBench is derived from actual developer fixes linked to performance-related issues, which are then verified by human experts, ensuring real-world relevance. Our evaluation reveals that current state-of-the-art coding agents struggle with performance optimization tasks, with baseline OpenHands agent achieving only a ~3% success rate on our benchmark. We develop OpenHands-Perf-Agent, which incorporates performance-aware tooling and instructions and achieves a ~20% success rate on the benchmark. We show that by ensuring the agent has proper instructions to benchmark its changes and tooling for benchmark output processing, we can improve the agent performance significantly, but room for improvement still remains. PerfBench provides a challenging test set for furthering the capabilities of agents in fixing performance issues.
♻ ☆ TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Design
Geonwoo Cho, Jaegyun Im, Jihwan Lee, Hojun Yi, Sejin Kim, Sundong Kim
Generalizing deep reinforcement learning agents to unseen environments remains a significant challenge. One promising solution is Unsupervised Environment Design (UED), a co-evolutionary framework in which a teacher adaptively generates tasks with high learning potential, while a student learns a robust policy from this evolving curriculum. Existing UED methods typically measure learning potential via regret, the gap between optimal and current performance, approximated solely by value-function loss. Building on these approaches, we introduce the transition-prediction error as an additional term in our regret approximation. To capture how training on one task affects performance on others, we further propose a lightweight metric called Co-Learnability. By combining these two measures, we present Transition-aware Regret Approximation with Co-learnability for Environment Design (TRACED). Empirical evaluations show that TRACED produces curricula that improve zero-shot generalization over strong baselines across multiple benchmarks. Ablation studies confirm that the transition-prediction error drives rapid complexity ramp-up and that Co-Learnability delivers additional gains when paired with the transition-prediction error. These results demonstrate how refined regret approximation and explicit modeling of task relationships can be leveraged for sample-efficient curriculum design in UED. Project Page: https://geonwoo.me/traced/
♻ ☆ AMPED: Adaptive Multi-objective Projection for balancing Exploration and skill Diversification
Geonwoo Cho, Jaemoon Lee, Jaegyun Im, Subi Lee, Jihwan Lee, Sundong Kim
Skill-based reinforcement learning (SBRL) enables rapid adaptation in environments with sparse rewards by pretraining a skill-conditioned policy. Effective skill learning requires jointly maximizing both exploration and skill diversity. However, existing methods often face challenges in simultaneously optimizing for these two conflicting objectives. In this work, we propose a new method, Adaptive Multi-objective Projection for balancing Exploration and skill Diversification (AMPED), which explicitly addresses both: during pre-training, a gradient-surgery projection balances the exploration and diversity gradients, and during fine-tuning, a skill selector exploits the learned diversity by choosing skills suited to downstream tasks. Our approach achieves performance that surpasses SBRL baselines across various benchmarks. Through an extensive ablation study, we identify the role of each component and demonstrate that each element in AMPED is contributing to performance. We further provide theoretical and empirical evidence that, with a greedy skill selector, greater skill diversity reduces fine-tuning sample complexity. These results highlight the importance of explicitly harmonizing exploration and diversity and demonstrate the effectiveness of AMPED in enabling robust and generalizable skill learning. Project Page: https://geonwoo.me/amped/
♻ ☆ From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing
Xintian Sun, Benji Peng, Charles Zhang, Fei Jin, Qian Niu, Junyu Liu, Keyu Chen, Ming Li, Pohsun Feng, Ziqian Bi, Ming Liu, Xinyuan Song, Yichao Zhang
Remote sensing has evolved from simple image acquisition to complex systems capable of integrating and processing visual and textual data. This review examines the development and application of multi-modal language models (MLLMs) in remote sensing, focusing on their ability to interpret and describe satellite imagery using natural language. We cover the technical underpinnings of MLLMs, including dual-encoder architectures, Transformer models, self-supervised and contrastive learning, and cross-modal integration. The unique challenges of remote sensing data--varying spatial resolutions, spectral richness, and temporal changes--are analyzed for their impact on MLLM performance. Key applications such as scene description, object detection, change detection, text-to-image retrieval, image-to-text generation, and visual question answering are discussed to demonstrate their relevance in environmental monitoring, urban planning, and disaster response. We review significant datasets and resources supporting the training and evaluation of these models. Challenges related to computational demands, scalability, data quality, and domain adaptation are highlighted. We conclude by proposing future research directions and technological advancements to further enhance MLLM utility in remote sensing.
comment : 10 pages, 1 figure
♻ ☆ The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Models
Yang Xiao, Gen Li, Jie Ji, Ruimeng Ye, Xiaolong Ma, Bo Hui
Machine unlearning aims to efficiently eliminate the memory about deleted data from trained models and address the right to be forgotten. Despite the success of existing unlearning algorithms, unlearning in sparse models has not yet been well studied. In this paper, we empirically find that the deleted data has an impact on the pruned topology in a sparse model. Motivated by the observation and the right to be forgotten, we define a new terminology ``un-pruning" to eliminate the impact of deleted data on model pruning. Then we propose an un-pruning algorithm to approximate the pruned topology driven by retained data. We remark that any existing unlearning algorithm can be integrated with the proposed un-pruning workflow and the error of un-pruning is upper-bounded in theory. Also, our un-pruning algorithm can be applied to both structured sparse models and unstructured sparse models. In the experiment, we further find that Membership Inference Attack (MIA) accuracy is unreliable for assessing whether a model has forgotten deleted data, as a small change in the amount of deleted data can produce arbitrary MIA results. Accordingly, we devise new performance metrics for sparse models to evaluate the success of un-pruning. Lastly, we conduct extensive experiments to verify the efficacy of un-pruning with various pruning methods and unlearning algorithms. Our code is released at https://github.com/NKUShaw/SparseModels .
comment : 9 pages for main part
♻ ☆ Privacy Risks and Preservation Methods in Explainable Artificial Intelligence: A Scoping Review
Sonal Allana, Mohan Kankanhalli, Rozita Dara
Explainable Artificial Intelligence (XAI) has emerged as a pillar of Trustworthy AI and aims to bring transparency in complex models that are opaque by nature. Despite the benefits of incorporating explanations in models, an urgent need is found in addressing the privacy concerns of providing this additional information to end users. In this article, we conduct a scoping review of existing literature to elicit details on the conflict between privacy and explainability. Using the standard methodology for scoping review, we extracted 57 articles from 1,943 studies published from January 2019 to December 2024. The review addresses 3 research questions to present readers with more understanding of the topic: (1) what are the privacy risks of releasing explanations in AI systems? (2) what current methods have researchers employed to achieve privacy preservation in XAI systems? (3) what constitutes a privacy preserving explanation? Based on the knowledge synthesized from the selected studies, we categorize the privacy risks and preservation methods in XAI and propose the characteristics of privacy preserving explanations to aid researchers and practitioners in understanding the requirements of XAI that is privacy compliant. Lastly, we identify the challenges in balancing privacy with other system desiderata and provide recommendations for achieving privacy preserving XAI. We expect that this review will shed light on the complex relationship of privacy and explainability, both being the fundamental principles of Trustworthy AI.
comment : Published in Transactions on Machine Learning Research: https://openreview.net/forum?id=q9nykJfzku
Computer Vision and Pattern Recognition 205
☆ MagicQuillV2: Precise and Interactive Image Editing with Layered Visual Cues
Zichen Liu, Yue Yu, Hao Ouyang, Qiuyu Wang, Shuailei Ma, Ka Leong Cheng, Wen Wang, Qingyan Bai, Yuxuan Zhang, Yanhong Zeng, Yixuan Li, Xing Zhu, Yujun Shen, Qifeng Chen
We propose MagicQuill V2, a novel system that introduces a \textbf{layered composition} paradigm to generative image editing, bridging the gap between the semantic power of diffusion models and the granular control of traditional graphics software. While diffusion transformers excel at holistic generation, their use of singular, monolithic prompts fails to disentangle distinct user intentions for content, position, and appearance. To overcome this, our method deconstructs creative intent into a stack of controllable visual cues: a content layer for what to create, a spatial layer for where to place it, a structural layer for how it is shaped, and a color layer for its palette. Our technical contributions include a specialized data generation pipeline for context-aware content integration, a unified control module to process all visual cues, and a fine-tuned spatial branch for precise local editing, including object removal. Extensive experiments validate that this layered approach effectively resolves the user intention gap, granting creators direct, intuitive control over the generative process.
comment : Code and demo available at https://magicquill.art/v2/
☆ CAMEO: Correspondence-Attention Alignment for Multi-View Diffusion Models
Minkyung Kwon, Jinhyeok Choi, Jiho Park, Seonghu Jeon, Jinhyuk Jang, Junyoung Seo, Minseop Kwak, Jin-Hwa Kim, Seungryong Kim
Multi-view diffusion models have recently emerged as a powerful paradigm for novel view synthesis, yet the underlying mechanism that enables their view-consistency remains unclear. In this work, we first verify that the attention maps of these models acquire geometric correspondence throughout training, attending to the geometrically corresponding regions across reference and target views for view-consistent generation. However, this correspondence signal remains incomplete, with its accuracy degrading under large viewpoint changes. Building on these findings, we introduce CAMEO, a simple yet effective training technique that directly supervises attention maps using geometric correspondence to enhance both the training efficiency and generation quality of multi-view diffusion models. Notably, supervising a single attention layer is sufficient to guide the model toward learning precise correspondences, thereby preserving the geometry and structure of reference images, accelerating convergence, and improving novel view synthesis performance. CAMEO reduces the number of training iterations required for convergence by half while achieving superior performance at the same iteration counts. We further demonstrate that CAMEO is model-agnostic and can be applied to any multi-view diffusion model.
comment : Project page: https://cvlab-kaist.github.io/CAMEO/
☆ OneThinker: All-in-one Reasoning Model for Image and Video
Kaituo Feng, Manyuan Zhang, Hongyu Li, Kaixuan Fan, Shuang Chen, Yilei Jiang, Dian Zheng, Peiwen Sun, Yiyuan Zhang, Haoze Sun, Yan Feng, Peng Pei, Xunliang Cai, Xiangyu Yue
Reinforcement learning (RL) has recently achieved remarkable success in eliciting visual reasoning within Multimodal Large Language Models (MLLMs). However, existing approaches typically train separate models for different tasks and treat image and video reasoning as disjoint domains. This results in limited scalability toward a multimodal reasoning generalist, which restricts practical versatility and hinders potential knowledge sharing across tasks and modalities. To this end, we propose OneThinker, an all-in-one reasoning model that unifies image and video understanding across diverse fundamental visual tasks, including question answering, captioning, spatial and temporal grounding, tracking, and segmentation. To achieve this, we construct the OneThinker-600k training corpus covering all these tasks and employ commercial models for CoT annotation, resulting in OneThinker-SFT-340k for SFT cold start. Furthermore, we propose EMA-GRPO to handle reward heterogeneity in multi-task RL by tracking task-wise moving averages of reward standard deviations for balanced optimization. Extensive experiments on diverse visual benchmarks show that OneThinker delivers strong performance on 31 benchmarks, across 10 fundamental visual understanding tasks. Moreover, it exhibits effective knowledge transfer between certain tasks and preliminary zero-shot generalization ability, marking a step toward a unified multimodal reasoning generalist. All code, model, and data are released.
comment : Project page: https://github.com/tulerfeng/OneThinker
☆ PPTArena: A Benchmark for Agent ic PowerPoint Editing
Michael Ofengenden, Yunze Man, Ziqi Pang, Yu-Xiong Wang
We introduce PPTArena, a benchmark for PowerPoint editing that measures reliable modifications to real slides under natural-language instructions. In contrast to image-PDF renderings or text-to-slide generation, PPTArena focuses on in-place editing across 100 decks, 2125 slides, and over 800 targeted edits covering text, charts, tables, animations, and master-level styles. Each case includes a ground-truth deck, a fully specified target outcome, and a dual VLM-as-judge pipeline that separately scores instruction following and visual quality using both structural diffs and slide images. Building on this setting, we propose PPTPilot, a structure-aware slide-editing agent that plans semantic edit sequences, routes between high-level programmatic tools and deterministic XML operations for precise control, and verifies outputs through an iterative plan-edit-check loop against task-specific constraints. In our experiments, PPTPilot outperforms strong proprietary agents and frontier VLM systems by over 10 percentage points on compound, layout-sensitive, and cross-slide edits, with particularly large gains in visual fidelity and deck-wide consistency. Despite these improvements, existing agents still underperform on long-horizon, document-scale tasks in PPTArena, highlighting the remaining challenges in reliable PPT editing.
comment : 25 pages, 26 figures
☆ MultiShotMaster: A Controllable Multi-Shot Video Generation Framework
Qinghe Wang, Xiaoyu Shi, Baolu Li, Weikang Bian, Quande Liu, Huchuan Lu, Xintao Wang, Pengfei Wan, Kun Gai, Xu Jia
Current video generation techniques excel at single-shot clips but struggle to produce narrative multi-shot videos, which require flexible shot arrangement, coherent narrative, and controllability beyond text prompts. To tackle these challenges, we propose MultiShotMaster, a framework for highly controllable multi-shot video generation. We extend a pretrained single-shot model by integrating two novel variants of RoPE. First, we introduce Multi-Shot Narrative RoPE, which applies explicit phase shift at shot transitions, enabling flexible shot arrangement while preserving the temporal narrative order. Second, we design Spatiotemporal Position-Aware RoPE to incorporate reference tokens and grounding signals, enabling spatiotemporal-grounded reference injection. In addition, to overcome data scarcity, we establish an automated data annotation pipeline to extract multi-shot videos, captions, cross-shot grounding signals and reference images. Our framework leverages the intrinsic architectural properties to support multi-shot video generation, featuring text-driven inter-shot consistency, customized subject with motion control, and background-driven customized scene. Both shot count and duration are flexibly configurable. Extensive experiments demonstrate the superior performance and outstanding controllability of our framework.
comment : Project Page: https://qinghew.github.io/MultiShotMaster
☆ Video4Spatial: Towards Visuospatial Intelligence with Context -Guided Video Generation
Zeqi Xiao, Yiwei Zhao, Lingxiao Li, Yushi Lan, Yu Ning, Rahul Garg, Roshni Cooper, Mohammad H. Taghavi, Xingang Pan
We investigate whether video generative models can exhibit visuospatial intelligence, a capability central to human cognition, using only visual data. To this end, we present Video4Spatial, a framework showing that video diffusion models conditioned solely on video-based scene context can perform complex spatial tasks. We validate on two tasks: scene navigation - following camera-pose instructions while remaining consistent with 3D geometry of the scene, and object grounding - which requires semantic localization, instruction following, and planning. Both tasks use video-only inputs, without auxiliary modalities such as depth or poses. With simple yet effective design choices in the framework and data curation, Video4Spatial demonstrates strong spatial understanding from video context: it plans navigation and grounds target objects end-to-end, follows camera-pose instructions while maintaining spatial consistency, and generalizes to long contexts and out-of-domain environments. Taken together, these results advance video generative models toward general visuospatial reasoning.
comment : Project page at https://xizaoqu.github.io/video4spatial/
☆ ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generation
Mengchen Zhang, Qi Chen, Tong Wu, Zihan Liu, Dahua Lin
Despite progress in video-to-audio generation, the field focuses predominantly on mono output, lacking spatial immersion. Existing binaural approaches remain constrained by a two-stage pipeline that first generates mono audio and then performs spatialization, often resulting in error accumulation and spatio-temporal inconsistencies. To address this limitation, we introduce the task of end-to-end binaural spatial audio generation directly from silent video. To support this task, we present the BiAudio dataset, comprising approximately 97K video-binaural audio pairs spanning diverse real-world scenes and camera rotation trajectories, constructed through a semi-automated pipeline. Furthermore, we propose ViSAudio, an end-to-end framework that employs conditional flow matching with a dual-branch audio generation architecture, where two dedicated branches model the audio latent flows. Integrated with a conditional spacetime module, it balances consistency between channels while preserving distinctive spatial characteristics, ensuring precise spatio-temporal alignment between audio and the input video. Comprehensive experiments demonstrate that ViSAudio outperforms existing state-of-the-art methods across both objective metrics and subjective evaluations, generating high-quality binaural audio with spatial immersion that adapts effectively to viewpoint changes, sound-source motion, and diverse acoustic environments. Project website: https://kszpxxzmc.github.io/ViSAudio-project.
☆ MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generation
Youxin Pang, Jiajun Liu, Lingfeng Tan, Yong Zhang, Feng Gao, Xiang Deng, Zhuoliang Kang, Xiaoming Wei, Yebin Liu
We propose MAViD, a novel Multimodal framework for Audio-Visual Dialogue understanding and generation. Existing approaches primarily focus on non-interactive systems and are limited to producing constrained and unnatural human speech.The primary challenge of this task lies in effectively integrating understanding and generation capabilities, as well as achieving seamless multimodal audio-video fusion. To solve these problems, we propose a Conductor-Creator architecture that divides the dialogue system into two primary components.The Conductor is tasked with understanding, reasoning, and generating instructions by breaking them down into motion and speech components, thereby enabling fine-grained control over interactions. The Creator then delivers interactive responses based on these instructions.Furthermore, to address the difficulty of generating long videos with consistent identity, timbre, and tone using dual DiT structures, the Creator adopts a structure that combines autoregressive (AR) and diffusion models. The AR model is responsible for audio generation, while the diffusion model ensures high-quality video generation.Additionally, we propose a novel fusion module to enhance connections between contextually consecutive clips and modalities, enabling synchronized long-duration audio-visual content generation.Extensive experiments demonstrate that our framework can generate vivid and contextually coherent long-duration dialogue interactions and accurately interpret users' multimodal queries.
comment : Our project website is https://carlyx.github.io/MAViD/
☆ SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control
Yuxuan Mu, Ziyu Zhang, Yi Shi, Minami Matsumoto, Kotaro Imamura, Guy Tevet, Chuan Guo, Michael Taylor, Chang Shu, Pengcheng Xi, Xue Bin Peng
Data-driven motion priors that can guide agents toward producing naturalistic behaviors play a pivotal role in creating life-like virtual characters. Adversarial imitation learning has been a highly effective method for learning motion priors from reference motion data. However, adversarial priors, with few exceptions, need to be retrained for each new controller, thereby limiting their reusability and necessitating the retention of the reference motion data when training on downstream tasks. In this work, we present Score-Matching Motion Priors (SMP), which leverages pre-trained motion diffusion models and score distillation sampling (SDS) to create reusable task-agnostic motion priors. SMPs can be pre-trained on a motion dataset, independent of any control policy or task. Once trained, SMPs can be kept frozen and reused as general-purpose reward functions to train policies to produce naturalistic behaviors for downstream tasks. We show that a general motion prior trained on large-scale datasets can be repurposed into a variety of style-specific priors. Furthermore SMP can compose different styles to synthesize new styles not present in the original dataset. Our method produces high-quality motion comparable to state-of-the-art adversarial imitation learning methods through reusable and modular motion priors. We demonstrate the effectiveness of SMP across a diverse suite of control tasks with physically simulated humanoid characters. Video demo available at https://youtu.be/ravlZJteS20
comment : 14 pages, 9 figures
☆ Unrolled Networks are Conditional Probability Flows in MRI Reconstruction
Kehan Qi, Saumya Gupta, Qingqiao Hu, Weimin Lyu, Chao Chen
Magnetic Resonance Imaging (MRI) offers excellent soft-tissue contrast without ionizing radiation, but its long acquisition time limits clinical utility. Recent methods accelerate MRI by under-sampling $k$-space and reconstructing the resulting images using deep learning. Unrolled networks have been widely used for the reconstruction task due to their efficiency, but suffer from unstable evolving caused by freely-learnable parameters in intermediate steps. In contrast, diffusion models based on stochastic differential equations offer theoretical stability in both medical and natural image tasks but are computationally expensive. In this work, we introduce flow ODEs to MRI reconstruction by theoretically proving that unrolled networks are discrete implementations of conditional probability flow ODEs. This connection provides explicit formulations for parameters and clarifies how intermediate states should evolve. Building on this insight, we propose Flow-Aligned Training (FLAT), which derives unrolled parameters from the ODE discretization and aligns intermediate reconstructions with the ideal ODE trajectory to improve stability and convergence. Experiments on three MRI datasets show that FLAT achieves high-quality reconstructions with up to $3\times$ fewer iterations than diffusion-based generative models and significantly greater stability than unrolled networks.
☆ AutoBrep: Autoregressive B-Rep Generation with Unified Topology and Geometry
Xiang Xu, Pradeep Kumar Jayaraman, Joseph G. Lambourne, Yilin Liu, Durvesh Malpure, Pete Meltzer
The boundary representation (B-Rep) is the standard data structure used in Computer-Aided Design (CAD) for defining solid models. Despite recent progress, directly generating B-Reps end-to-end with precise geometry and watertight topology remains a challenge. This paper presents AutoBrep, a novel Transformer model that autoregressively generates B-Reps with high quality and validity. AutoBrep employs a unified tokenization scheme that encodes both geometric and topological characteristics of a B-Rep model as a sequence of discrete tokens. Geometric primitives (i.e., surfaces and curves) are encoded as latent geometry tokens, and their structural relationships are defined as special topological reference tokens. Sequence order in AutoBrep naturally follows a breadth first traversal of the B-Rep face adjacency graph. At inference time, neighboring faces and edges along with their topological structure are progressively generated. Extensive experiments demonstrate the advantages of our unified representation when coupled with next-token prediction for B-Rep generation. AutoBrep outperforms baselines with better quality and watertightness. It is also highly scalable to complex solids with good fidelity and inference speed. We further show that autocompleting B-Reps is natively supported through our unified tokenization, enabling user-controllable CAD generation with minimal changes. Code is available at https://github.com/AutodeskAILab/AutoBrep.
comment : Accepted to Siggraph Asia 2025
☆ Instant Video Models: Universal Adapters for Stabilizing Image-Based Networks NeurIPS 2025
Matthew Dutson, Nathan Labiosa, Yin Li, Mohit Gupta
When applied sequentially to video, frame-based networks often exhibit temporal inconsistency - for example, outputs that flicker between frames. This problem is amplified when the network inputs contain time-varying corruptions. In this work, we introduce a general approach for adapting frame-based models for stable and robust inference on video. We describe a class of stability adapters that can be inserted into virtually any architecture and a resource-efficient training process that can be performed with a frozen base network. We introduce a unified conceptual framework for describing temporal stability and corruption robustness, centered on a proposed accuracy-stability-robustness loss. By analyzing the theoretical properties of this loss, we identify the conditions where it produces well-behaved stabilizer training. Our experiments validate our approach on several vision tasks including denoising (NAFNet), image enhancement (HDRNet), monocular depth (Depth Anything v2), and semantic segmentation (DeepLabv3+). Our method improves temporal stability and robustness against a range of image corruptions (including compression artifacts, noise, and adverse weather), while preserving or improving the quality of predictions.
comment : NeurIPS 2025
☆ In- Context Sync-LoRA for Portrait Video Editing
Sagi Polaczek, Or Patashnik, Ali Mahdavi-Amiri, Daniel Cohen-Or
Editing portrait videos is a challenging task that requires flexible yet precise control over a wide range of modifications, such as appearance changes, expression edits, or the addition of objects. The key difficulty lies in preserving the subject's original temporal behavior, demanding that every edited frame remains precisely synchronized with the corresponding source frame. We present Sync-LoRA, a method for editing portrait videos that achieves high-quality visual modifications while maintaining frame-accurate synchronization and identity consistency. Our approach uses an image-to-video diffusion model, where the edit is defined by modifying the first frame and then propagated to the entire sequence. To enable accurate synchronization, we train an in-context LoRA using paired videos that depict identical motion trajectories but differ in appearance. These pairs are automatically generated and curated through a synchronization-based filtering process that selects only the most temporally aligned examples for training. This training setup teaches the model to combine motion cues from the source video with the visual changes introduced in the edited first frame. Trained on a compact, highly curated set of synchronized human portraits, Sync-LoRA generalizes to unseen identities and diverse edits (e.g., modifying appearance, adding objects, or changing backgrounds), robustly handling variations in pose and expression. Our results demonstrate high visual fidelity and strong temporal coherence, achieving a robust balance between edit fidelity and precise motion preservation.
comment : Project page: https://sagipolaczek.github.io/Sync-LoRA/
☆ SurfFill: Completion of LiDAR Point Clouds via Gaussian Surfel Splatting
Svenja Strobel, Matthias Innmann, Bernhard Egger, Marc Stamminger, Linus Franke
LiDAR-captured point clouds are often considered the gold standard in active 3D reconstruction. While their accuracy is exceptional in flat regions, the capturing is susceptible to miss small geometric structures and may fail with dark, absorbent materials. Alternatively, capturing multiple photos of the scene and applying 3D photogrammetry can infer these details as they often represent feature-rich regions. However, the accuracy of LiDAR for featureless regions is rarely reached. Therefore, we suggest combining the strengths of LiDAR and camera-based capture by introducing SurfFill: a Gaussian surfel-based LiDAR completion scheme. We analyze LiDAR capturings and attribute LiDAR beam divergence as a main factor for artifacts, manifesting mostly at thin structures and edges. We use this insight to introduce an ambiguity heuristic for completed scans by evaluating the change in density in the point cloud. This allows us to identify points close to missed areas, which we can then use to grow additional points from to complete the scan. For this point growing, we constrain Gaussian surfel reconstruction [Huang et al. 2024] to focus optimization and densification on these ambiguous areas. Finally, Gaussian primitives of the reconstruction in ambiguous areas are extracted and sampled for points to complete the point cloud. To address the challenges of large-scale reconstruction, we extend this pipeline with a divide-and-conquer scheme for building-sized point cloud completion. We evaluate on the task of LiDAR point cloud completion of synthetic and real-world scenes and find that our method outperforms previous reconstruction methods.
comment : Project page: https://lfranke.github.io/surffill
☆ DGGT: Feedforward 4D Reconstruction of Dynamic Driving Scenes using Unposed Images
Xiaoxue Chen, Ziyi Xiong, Yuantao Chen, Gen Li, Nan Wang, Hongcheng Luo, Long Chen, Haiyang Sun, Bing Wang, Guang Chen, Hangjun Ye, Hongyang Li, Ya-Qin Zhang, Hao Zhao
Autonomous driving needs fast, scalable 4D reconstruction and re-simulation for training and evaluation, yet most methods for dynamic driving scenes still rely on per-scene optimization, known camera calibration, or short frame windows, making them slow and impractical. We revisit this problem from a feedforward perspective and introduce \textbf{Driving Gaussian Grounded Transformer (DGGT)}, a unified framework for pose-free dynamic scene reconstruction. We note that the existing formulations, treating camera pose as a required input, limit flexibility and scalability. Instead, we reformulate pose as an output of the model, enabling reconstruction directly from sparse, unposed images and supporting an arbitrary number of views for long sequences. Our approach jointly predicts per-frame 3D Gaussian maps and camera parameters, disentangles dynamics with a lightweight dynamic head, and preserves temporal consistency with a lifespan head that modulates visibility over time. A diffusion-based rendering refinement further reduces motion/interpolation artifacts and improves novel-view quality under sparse inputs. The result is a single-pass, pose-free algorithm that achieves state-of-the-art performance and speed. Trained and evaluated on large-scale driving benchmarks (Waymo, nuScenes, Argoverse2), our method outperforms prior work both when trained on each dataset and in zero-shot transfer across datasets, and it scales well as the number of input frames increases.
☆ DynamicVerse: A Physically-Aware Multimodal Framework for 4D Wo rl d Modeling
Kairun Wen, Yuzhi Huang, Runyu Chen, Hui Zheng, Yunlong Lin, Panwang Pan, Chenxin Li, Wenyan Cong, Jian Zhang, Junbin Lu, Chenguo Lin, Dilin Wang, Zhicheng Yan, Hongyu Xu, Justin Theiss, Yue Huang, Xinghao Ding, Rakesh Ranjan, Zhiwen Fan
Understanding the dynamic physical world, characterized by its evolving 3D structure, real-world motion, and semantic content with textual descriptions, is crucial for human-agent interaction and enables embodied agents to perceive and act within real environments with human-like capabilities. However, existing datasets are often derived from limited simulators or utilize traditional Structurefrom-Motion for up-to-scale annotation and offer limited descriptive captioning, which restricts the capacity of foundation models to accurately interpret real-world dynamics from monocular videos, commonly sourced from the internet. To bridge these gaps, we introduce DynamicVerse, a physical-scale, multimodal 4D world modeling framework for dynamic real-world video. We employ large vision, geometric, and multimodal models to interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic descriptive captions. By integrating window-based Bundle Adjustment with global optimization, our method converts long real-world video sequences into a comprehensive 4D multimodal format. DynamicVerse delivers a large-scale dataset consists of 100K+ videos with 800K+ annotated masks and 10M+ frames from internet videos. Experimental evaluations on three benchmark tasks, namely video depth estimation, camera pose estimation, and camera intrinsics estimation, demonstrate that our 4D modeling achieves superior performance in capturing physical-scale measurements with greater global accuracy than existing methods.
☆ TEXTRIX: Latent Attribute Grid for Native Texture Generation and Beyond
Yifei Zeng, Yajie Bao, Jiachen Qian, Shuang Wu, Youtian Lin, Hao Zhu, Buyu Li, Feihu Zhang, Xun Cao, Yao Yao
Prevailing 3D texture generation methods, which often rely on multi-view fusion, are frequently hindered by inter-view inconsistencies and incomplete coverage of complex surfaces, limiting the fidelity and completeness of the generated content. To overcome these challenges, we introduce TEXTRIX, a native 3D attribute generation framework for high-fidelity texture synthesis and downstream applications such as precise 3D part segmentation. Our approach constructs a latent 3D attribute grid and leverages a Diffusion Transformer equipped with sparse attention, enabling direct coloring of 3D models in volumetric space and fundamentally avoiding the limitations of multi-view fusion. Built upon this native representation, the framework naturally extends to high-precision 3D segmentation by training the same architecture to predict semantic attributes on the grid. Extensive experiments demonstrate state-of-the-art performance on both tasks, producing seamless, high-fidelity textures and accurate 3D part segmentation with precise boundaries.
comment : Project Page: https://www.neural4d.com/research-page/textrix
☆ GraphFusion3D: Dynamic Graph Attention Convolution with Adaptive Cross-Modal Transformer for 3D Object Detection
Md Sohag Mia, Md Nahid Hasan, Tawhid Ahmed, Muhammad Abdullah Adnan
Despite significant progress in 3D object detection, point clouds remain challenging due to sparse data, incomplete structures, and limited semantic information. Capturing contextual relationships between distant objects presents additional difficulties. To address these challenges, we propose GraphFusion3D, a unified framework combining multi-modal fusion with advanced feature learning. Our approach introduces the Adaptive Cross-Modal Transformer (ACMT), which adaptively integrates image features into point representations to enrich both geometric and semantic information. For proposal refinement, we introduce the Graph Reasoning Module (GRM), a novel mechanism that models neighborhood relationships to simultaneously capture local geometric structures and global semantic context. The module employs multi-scale graph attention to dynamically weight both spatial proximity and feature similarity between proposals. We further employ a cascade decoder that progressively refines detections through multi-stage predictions. Extensive experiments on SUN RGB-D (70.6\% AP$_{25}$ and 51.2\% AP$_{50}$) and ScanNetV2 (75.1\% AP$_{25}$ and 60.8\% AP$_{50}$) demonstrate a substantial performance improvement over existing approaches.
☆ U4D: Uncertainty-Aware 4D Wo rl d Modeling from LiDAR Sequences
Xiang Xu, Ao Liang, Youquan Liu, Linfeng Li, Lingdong Kong, Ziwei Liu, Qingshan Liu
Modeling dynamic 3D environments from LiDAR sequences is central to building reliable 4D worlds for autonomous driving and embodied AI. Existing generative frameworks, however, often treat all spatial regions uniformly, overlooking the varying uncertainty across real-world scenes. This uniform generation leads to artifacts in complex or ambiguous regions, limiting realism and temporal stability. In this work, we present U4D, an uncertainty-aware framework for 4D LiDAR world modeling. Our approach first estimates spatial uncertainty maps from a pretrained segmentation model to localize semantically challenging regions. It then performs generation in a "hard-to-easy" manner through two sequential stages: (1) uncertainty-region modeling, which reconstructs high-entropy regions with fine geometric fidelity, and (2) uncertainty-conditioned completion, which synthesizes the remaining areas under learned structural priors. To further ensure temporal coherence, U4D incorporates a mixture of spatio-temporal (MoST) block that adaptively fuses spatial and temporal representations during diffusion. Extensive experiments show that U4D produces geometrically faithful and temporally consistent LiDAR sequences, advancing the reliability of 4D world modeling for autonomous perception and simulation.
comment : Preprint; 19 pages, 7 figures, 8 tables
☆ InEx: Hallucination Mitigation via Introspection and Cross-Modal Multi- Agent Collaboration AAAI 2026
Zhongyu Yang, Yingfang Yuan, Xuanming Jiang, Baoyi An, Wei Pang
Hallucination remains a critical challenge in large language models (LLMs), hindering the development of reliable multimodal LLMs (MLLMs). Existing solutions often rely on human intervention or underutilize the agent's ability to autonomously mitigate hallucination. To address these limitations, we draw inspiration from how humans make reliable decisions in the real world. They begin with introspective reasoning to reduce uncertainty and form an initial judgment, then rely on external verification from diverse perspectives to reach a final decision. Motivated by this cognitive paradigm, we propose InEx, a training-free, multi-agent framework designed to autonomously mitigate hallucination. InEx introduces internal introspective reasoning, guided by entropy-based uncertainty estimation, to improve the reliability of the decision agent's reasoning process. The agent first generates a response, which is then iteratively verified and refined through external cross-modal multi-agent collaboration with the editing agent and self-reflection agents, further enhancing reliability and mitigating hallucination. Extensive experiments show that InEx consistently outperforms existing methods, achieving 4%-27% gains on general and hallucination benchmarks, and demonstrating strong robustness.
comment : Published in AAAI 2026
☆ Context ual Image Attack: How Visual Context Exposes Multimodal Safety Vulnerabilities
Yuan Xiong, Ziqi Miao, Lijun Li, Chen Qian, Jie Li, Jing Shao
While Multimodal Large Language Models (MLLMs) show remarkable capabilities, their safety alignments are susceptible to jailbreak attacks. Existing attack methods typically focus on text-image interplay, treating the visual modality as a secondary prompt. This approach underutilizes the unique potential of images to carry complex, contextual information. To address this gap, we propose a new image-centric attack method, Contextual Image Attack (CIA), which employs a multi-agent system to subtly embeds harmful queries into seemingly benign visual contexts using four distinct visualization strategies. To further enhance the attack's efficacy, the system incorporate contextual element enhancement and automatic toxicity obfuscation techniques. Experimental results on the MMSafetyBench-tiny dataset show that CIA achieves high toxicity scores of 4.73 and 4.83 against the GPT-4o and Qwen2.5-VL-72B models, respectively, with Attack Success Rates (ASR) reaching 86.31\% and 91.07\%. Our method significantly outperforms prior work, demonstrating that the visual modality itself is a potent vector for jailbreaking advanced MLLMs.
☆ BEVDilation: LiDAR-Centric Multi-Modal Fusion for 3D Object Detection AAAI26
Guowen Zhang, Chenhang He, Liyi Chen, Lei Zhang
Integrating LiDAR and camera information in the bird's eye view (BEV) representation has demonstrated its effectiveness in 3D object detection. However, because of the fundamental disparity in geometric accuracy between these sensors, indiscriminate fusion in previous methods often leads to degraded performance. In this paper, we propose BEVDilation, a novel LiDAR-centric framework that prioritizes LiDAR information in the fusion. By formulating image BEV features as implicit guidance rather than naive concatenation, our strategy effectively alleviates the spatial misalignment caused by image depth estimation errors. Furthermore, the image guidance can effectively help the LiDAR-centric paradigm to address the sparsity and semantic limitations of point clouds. Specifically, we propose a Sparse Voxel Dilation Block that mitigates the inherent point sparsity by densifying foreground voxels through image priors. Moreover, we introduce a Semantic-Guided BEV Dilation Block to enhance the LiDAR feature diffusion processing with image semantic guidance and long-range context capture. On the challenging nuScenes benchmark, BEVDilation achieves better performance than state-of-the-art methods while maintaining competitive computational efficiency. Importantly, our LiDAR-centric strategy demonstrates greater robustness to depth noise compared to naive fusion. The source code is available at https://github.com/gwenzhang/BEVDilation.
comment : Accept by AAAI26
☆ A Lightweight Real-Time Low-Light Enhancement Network for Embedded Automotive Vision Systems
Yuhan Chen, Yicui Shi, Guofa Li, Guangrui Bai, Jinyuan Shao, Xiangfei Huang, Wenbo Chu, Keqiang Li
In low-light environments like nighttime driving, image degradation severely challenges in-vehicle camera safety. Since existing enhancement algorithms are often too computationally intensive for vehicular applications, we propose UltraFast-LieNET, a lightweight multi-scale shifted convolutional network for real-time low-light image enhancement. We introduce a Dynamic Shifted Convolution (DSConv) kernel with only 12 learnable parameters for efficient feature extraction. By integrating DSConv with varying shift distances, a Multi-scale Shifted Residual Block (MSRB) is constructed to significantly expand the receptive field. To mitigate lightweight network instability, a residual structure and a novel multi-level gradient-aware loss function are incorporated. UltraFast-LieNET allows flexible parameter configuration, with a minimum size of only 36 parameters. Results on the LOLI-Street dataset show a PSNR of 26.51 dB, outperforming state-of-the-art methods by 4.6 dB while utilizing only 180 parameters. Experiments across four benchmark datasets validate its superior balance of real-time performance and enhancement quality under limited resources. Code is available at https://githubhttps://github.com/YuhanChen2024/UltraFast-LiNET
☆ Layout Anything: One Transformer for Universal Room Layout Estimation WACV 2026
Md Sohag Mia, Muhammad Abdullah Adnan
We present Layout Anything, a transformer-based framework for indoor layout estimation that adapts the OneFormer's universal segmentation architecture to geometric structure prediction. Our approach integrates OneFormer's task-conditioned queries and contrastive learning with two key modules: (1) a layout degeneration strategy that augments training data while preserving Manhattan-world constraints through topology-aware transformations, and (2) differentiable geometric losses that directly enforce planar consistency and sharp boundary predictions during training. By unifying these components in an end-to-end framework, the model eliminates complex post-processing pipelines while achieving high-speed inference at 114ms. Extensive experiments demonstrate state-of-the-art performance across standard benchmarks, with pixel error (PE) of 5.43% and corner error (CE) of 4.02% on the LSUN, PE of 7.04% (CE 5.17%) on the Hedau and PE of 4.03% (CE 3.15%) on the Matterport3D-Layout datasets. The framework's combination of geometric awareness and computational efficiency makes it particularly suitable for augmented reality applications and large-scale 3D scene reconstruction tasks.
comment : Published at WACV 2026
☆ Benchmark ing Scientific Understanding and Reasoning for Video Generation using VideoScience-Bench
Lanxiang Hu, Abhilash Shankarampeta, Yixin Huang, Zilin Dai, Haoyang Yu, Yujie Zhao, Haoqiang Kang, Daniel Zhao, Tajana Rosing, Hao Zhang
The next frontier for video generation lies in developing models capable of zero-shot reasoning, where understanding real-world scientific laws is crucial for accurate physical outcome modeling under diverse conditions. However, existing video benchmarks are physical commonsense-based, offering limited insight into video models' scientific reasoning capability. We introduce VideoScience-Bench, a benchmark designed to evaluate undergraduate-level scientific understanding in video models. Each prompt encodes a composite scientific scenario that requires understanding and reasoning across multiple scientific concepts to generate the correct phenomenon. The benchmark comprises 200 carefully curated prompts spanning 14 topics and 103 concepts in physics and chemistry. We conduct expert-annotated evaluations across seven state-of-the-art video models in T2V and I2V settings along five dimensions: Prompt Consistency, Phenomenon Congruency, Correct Dynamism, Immutability, and Spatio-Temporal Continuity. Using a VLM-as-a-Judge to assess video generations, we observe strong correlation with human assessments. To the best of our knowledge, VideoScience-Bench is the first benchmark to evaluate video models not only as generators but also as reasoners, requiring their generations to demonstrate scientific understanding consistent with expected physical and chemical phenomena. Our data and evaluation code are available at: \href{https://github.com/hao-ai-lab/VideoScience}{github.com/hao-ai-lab/VideoScience}.
☆ LoVoRA: Text-guided and Mask-free Video Object Removal and Addition with Learnable Object-aware Localization
Zhihan Xiao, Lin Liu, Yixin Gao, Xiaopeng Zhang, Haoxuan Che, Songping Mai, Qi Tian
Text-guided video editing, particularly for object removal and addition, remains a challenging task due to the need for precise spatial and temporal consistency. Existing methods often rely on auxiliary masks or reference images for editing guidance, which limits their scalability and generalization. To address these issues, we propose LoVoRA, a novel framework for mask-free video object removal and addition using object-aware localization mechanism. Our approach utilizes a unique dataset construction pipeline that integrates image-to-video translation, optical flow-based mask propagation, and video inpainting, enabling temporally consistent edits. The core innovation of LoVoRA is its learnable object-aware localization mechanism, which provides dense spatio-temporal supervision for both object insertion and removal tasks. By leveraging a Diffusion Mask Predictor, LoVoRA achieves end-to-end video editing without requiring external control signals during inference. Extensive experiments and human evaluation demonstrate the effectiveness and high-quality performance of LoVoRA.
☆ EGGS: Exchangeable 2D/3D Gaussian Splatting for Geometry-Appearance Balanced Novel View Synthesis
Yancheng Zhang, Guangyu Sun, Chen Chen
Novel view synthesis (NVS) is crucial in computer vision and graphics, with wide applications in AR, VR, and autonomous driving. While 3D Gaussian Splatting (3DGS) enables real-time rendering with high appearance fidelity, it suffers from multi-view inconsistencies, limiting geometric accuracy. In contrast, 2D Gaussian Splatting (2DGS) enforces multi-view consistency but compromises texture details. To address these limitations, we propose Exchangeable Gaussian Splatting (EGGS), a hybrid representation that integrates 2D and 3D Gaussians to balance appearance and geometry. To achieve this, we introduce Hybrid Gaussian Rasterization for unified rendering, Adaptive Type Exchange for dynamic adaptation between 2D and 3D Gaussians, and Frequency-Decoupled Optimization that effectively exploits the strengths of each type of Gaussian representation. Our CUDA-accelerated implementation ensures efficient training and inference. Extensive experiments demonstrate that EGGS outperforms existing methods in rendering quality, geometric accuracy, and efficiency, providing a practical solution for high-quality NVS.
☆ DiverseAR: Boosting Diversity in Bitwise Autoregressive Image Generation
Ying Yang, Zhengyao Lv, Tianlin Pan, Haofan Wang, Binxin Yang, Hubery Yin, Chen Li, Chenyang Si
In this paper, we investigate the underexplored challenge of sample diversity in autoregressive (AR) generative models with bitwise visual tokenizers. We first analyze the factors that limit diversity in bitwise AR models and identify two key issues: (1) the binary classification nature of bitwise modeling, which restricts the prediction space, and (2) the overly sharp logits distribution, which causes sampling collapse and reduces diversity. Building on these insights, we propose DiverseAR, a principled and effective method that enhances image diversity without sacrificing visual quality. Specifically, we introduce an adaptive logits distribution scaling mechanism that dynamically adjusts the sharpness of the binary output distribution during sampling, resulting in smoother predictions and greater diversity. To mitigate potential fidelity loss caused by distribution smoothing, we further develop an energy-based generation path search algorithm that avoids sampling low-confidence tokens, thereby preserving high visual quality. Extensive experiments demonstrate that DiverseAR substantially improves sample diversity in bitwise autoregressive image generation.
comment : 23 pages
☆ Learning Multimodal Embeddings for Traffic Accident Prediction and Causal Estimation KDD'26
Ziniu Zhang, Minxuan Duan, Haris N. Koutsopoulos, Hongyang R. Zhang
We consider analyzing traffic accident patterns using both road network data and satellite images aligned to road graph nodes. Previous work for predicting accident occurrences relies primarily on road network structural features while overlooking physical and environmental information from the road surface and its surroundings. In this work, we construct a large multimodal dataset across six U.S. states, containing nine million traffic accident records from official sources, and one million high-resolution satellite images for each node of the road network. Additionally, every node is annotated with features such as the region's weather statistics and road type (e.g., residential vs. motorway), and each edge is annotated with traffic volume information (i.e., Average Annual Daily Traffic). Utilizing this dataset, we conduct a comprehensive evaluation of multimodal learning methods that integrate both visual and network embeddings. Our findings show that integrating both data modalities improves prediction accuracy, achieving an average AUROC of $90.1\%$, which is a $3.7\%$ gain over graph neural network models that only utilize graph structures. With the improved embeddings, we conduct a causal analysis based on a matching estimator to estimate the key contributing factors influencing traffic accidents. We find that accident rates rise by $24\%$ under higher precipitation, by $22\%$ on higher-speed roads such as motorways, and by $29\%$ due to seasonal patterns, after adjusting for other confounding factors. Ablation studies confirm that satellite imagery features are essential for achieving accurate prediction.
comment : 17 pages. To appear in KDD'26 Datasets
☆ MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding
Fan Yang, Kaihao Zhang
Understanding high-resolution images remains a significant challenge for multimodal large language models (MLLMs). Recent study address this issue by dividing the image into smaller crops and computing the semantic similarity between each crop and a query using a pretrained retrieval-augmented generation (RAG) model. The most relevant crops are then selected to localize the target object and suppress irrelevant information. However, such crop-based processing can fragment complete objects across multiple crops, thereby disrupting the computation of semantic similarity. In our experiments, we find that image crops of objects with different sizes are better handled at different resolutions. Based on this observation, we propose Multi-resolution Retrieval-Detection (MRD), a training-free framework for high-resolution image understanding. To address the issue of semantic similarity bias caused by objects being split across different image crops, we propose a multi-resolution semantic fusion method, which integrates semantic similarity maps obtained at different resolutions to produce more accurate semantic information and preserve the integrity of target objects. Furthermore, to achieve direct localization of target objects at a global scale, we introduce an open-vocalbulary object detection (OVD) model that identifies object regions using a sliding-window approach.Experiments on high-resolution image understanding benchmarks using different MLLMs demonstrate the effectiveness of our approach.
☆ Glance: Accelerating Diffusion Models with 1 Sample
Zhuobai Dong, Rui Zhao, Songjie Wu, Junchao Yi, Linjie Li, Zhengyuan Yang, Lijuan Wang, Alex Jinpeng Wang
Diffusion models have achieved remarkable success in image generation, yet their deployment remains constrained by the heavy computational cost and the need for numerous inference steps. Previous efforts on fewer-step distillation attempt to skip redundant steps by training compact student models, yet they often suffer from heavy retraining costs and degraded generalization. In this work, we take a different perspective: we accelerate smartly, not evenly, applying smaller speedups to early semantic stages and larger ones to later redundant phases. We instantiate this phase-aware strategy with two experts that specialize in slow and fast denoising phases. Surprisingly, instead of investing massive effort in retraining student models, we find that simply equipping the base model with lightweight LoRA adapters achieves both efficient acceleration and strong generalization. We refer to these two adapters as Slow-LoRA and Fast-LoRA. Through extensive experiments, our method achieves up to 5 acceleration over the base model while maintaining comparable visual quality across diverse benchmarks. Remarkably, the LoRA experts are trained with only 1 samples on a single V100 within one hour, yet the resulting models generalize strongly on unseen prompts.
☆ Polar Perspectives: Evaluating 2-D LiDAR Projections for Robust Place Recognition with Visual Foundation Models
Pierpaolo Serio, Giulio Pisaneschi, Andrea Dan Ryals, Vincenzo Infantino, Lorenzo Gentilini, Valentina Donzella, Lorenzo Pollini
This work presents a systematic investigation into how alternative LiDAR-to-image projections affect metric place recognition when coupled with a state-of-the-art vision foundation model. We introduce a modular retrieval pipeline that controls for backbone, aggregation, and evaluation protocol, thereby isolating the influence of the 2-D projection itself. Using consistent geometric and structural channels across multiple datasets and deployment scenarios, we identify the projection characteristics that most strongly determine discriminative power, robustness to environmental variation, and suitability for real-time autonomy. Experiments with different datasets, including integration into an operational place recognition policy, validate the practical relevance of these findings and demonstrate that carefully designed projections can serve as an effective surrogate for end-to-end 3-D learning in LiDAR place recognition.
comment : 13 Pages, 5 Figures, 2 Tables Under Review
☆ Mind GPT -4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigm
Wei Chen, Chaoqun Du, Feng Gu, Wei He, Qizhen Li, Zide Liu, Xuhao Pan, Chang Ren, Xudong Rao, Chenfeng Wang, Tao Wei, Chengjun Yu, Pengfei Yu, Yufei Zheng, Chunpeng Zhou, Pan Zhou, Xuhan Zhu
We present MindGPT-4ov, a multimodal large language model (MLLM) that introduces a general post-training paradigm spanning data production, model training, and efficient deployment. It achieves state-of-the-art performance across multiple benchmarks at low cost, effectively enhancing the foundational capabilities of MLLMs and the generalization ability. Focusing on data construction, supervised fine-tuning strategies, and multimodal reinforcement learning methods, this work proposes three key innovations: (1) An information density-based data generation scheme, integrated with a dual-dimensional tree-structured label system, enabling automated generation of high-quality cross-domain data. (2) A collaborative curriculum supervised fine-tuning approach that balances the injection of domain-specific knowledge with the preservation of general capabilities. (3) A hybrid reinforcement learning paradigm that enhances reasoning ability while simultaneously addressing multi-objective optimization such as diversity exploration, maintenance of multimodal perception, and response conciseness. Moreover, we implement a series of infrastructure optimizations, such as 5D parallel training, operator optimization, and inference quantization to enhance training and inference efficiency while reducing the cost of domain adaptation. Experimental results demonstrate that the MindGPT-4ov model outperforms state-of-the-art models on benchmarks such as MMBench, MMStar, MathVision, and MathVista. In addition, MindGPT-4ov also demonstrates superior user experience in vertical domain tasks, enabling a seamless transition from academic research to industrial deployment. MindGPT-4ov provides a general post-training paradigm applicable to a wide range of MLLMs. The model weights, datasets, and code for the Qwen3-VL-based variants will be recently open-sourced to support the community's development of MLLMs.
comment : 33 pages, 14 figures
☆ Taming Camera-Controlled Video Generation with Verifiable Geometry Reward
Zhaoqing Wang, Xiaobo Xia, Zhuolin Bie, Jinlin Liu, Dongdong Yu, Jia-Wang Bian, Changhu Wang
Recent advances in video diffusion models have remarkably improved camera-controlled video generation, but most methods rely solely on supervised fine-tuning (SFT), leaving online reinforcement learning (RL) post-training largely underexplored. In this work, we introduce an online RL post-training framework that optimizes a pretrained video generator for precise camera control. To make RL effective in this setting, we design a verifiable geometry reward that delivers dense segment-level feedback to guide model optimization. Specifically, we estimate the 3D camera trajectories for both generated and reference videos, divide each trajectory into short segments, and compute segment-wise relative poses. The reward function then compares each generated-reference segment pair and assigns an alignment score as the reward signal, which helps alleviate reward sparsity and improve optimization efficiency. Moreover, we construct a comprehensive dataset featuring diverse large-amplitude camera motions and scenes with varied subject dynamics. Extensive experiments show that our online RL post-training clearly outperforms SFT baselines across multiple aspects, including camera-control accuracy, geometric consistency, and visual quality, demonstrating its superiority in advancing camera-controlled video generation.
comment : 11 pages, 4 figures, 7 tables
☆ MICCAI STSR 2025 Challenge: Semi-Supervised Teeth and Pulp Segmentation and CBCT-IOS Registration
Yaqi Wang, Zhi Li, Chengyu Wu, Jun Liu, Yifan Zhang, Jialuo Chen, Jiaxue Ni, Qian Luo, Jin Liu, Can Han, Changkai Ji, Zhi Qin Tan, Ajo Babu George, Liangyu Chen, Qianni Zhang, Dahong Qian, Shuai Wang, Huiyu Zhou
Cone-Beam Computed Tomography (CBCT) and Intraoral Scanning (IOS) are essential for digital dentistry, but annotated data scarcity limits automated solutions for pulp canal segmentation and cross-modal registration. To benchmark semi-supervised learning (SSL) in this domain, we organized the STSR 2025 Challenge at MICCAI 2025, featuring two tasks: (1) semi-supervised segmentation of teeth and pulp canals in CBCT, and (2) semi-supervised rigid registration of CBCT and IOS. We provided 60 labeled and 640 unlabeled IOS samples, plus 30 labeled and 250 unlabeled CBCT scans with varying resolutions and fields of view. The challenge attracted strong community participation, with top teams submitting open-source deep learning-based SSL solutions. For segmentation, leading methods used nnU-Net and Mamba-like State Space Models with pseudo-labeling and consistency regularization, achieving a Dice score of 0.967 and Instance Affinity of 0.738 on the hidden test set. For registration, effective approaches combined PointNetLK with differentiable SVD and geometric augmentation to handle modality gaps; hybrid neural-classical refinement enabled accurate alignment despite limited labels. All data and code are publicly available at https://github.com/ricoleehduu/STS-Challenge-2025 to ensure reproducibility.
☆ RFOP: Rethinking Fusion and Orthogonal Projection for Face-Voice Association ICASSP
Abdul Hannan, Furqan Malik, Hina Jabbar, Syed Suleman Sadiq, Mubashir Noman
Face-voice association in multilingual environment challenge 2026 aims to investigate the face-voice association task in multilingual scenario. The challenge introduces English-German face-voice pairs to be utilized in the evaluation phase. To this end, we revisit the fusion and orthogonal projection for face-voice association by effectively focusing on the relevant semantic information within the two modalities. Our method performs favorably on the English-German data split and ranked 3rd in the FAME 2026 challenge by achieving the EER of 33.1.
comment : Ranked 3rd in Fame 2026 Challenge, ICASSP
☆ Are Detectors Fair to Indian IP-AIGC? A Cross-Generator Study
Vishal Dubey, Pallavi Tyagi
Modern image editors can produce identity-preserving AIGC (IP-AIGC), where the same person appears with new attire, background, or lighting. The robustness and fairness of current detectors in this regime remain unclear, especially for under-represented populations. We present what we believe is the first systematic study of IP-AIGC detection for Indian and South-Asian faces, quantifying cross-generator generalization and intra-population performance. We assemble Indian-focused training splits from FairFD and HAV-DF, and construct two held-out IP-AIGC test sets (HIDF-img-ip-genai and HIDF-vid-ip-genai) using commercial web-UI generators (Gemini and ChatGPT) with identity-preserving prompts. We evaluate two state-of-the-art detectors (AIDE and Effort) under pretrained (PT) and fine-tuned (FT) regimes and report AUC, AP, EER, and accuracy. Fine-tuning yields strong in-domain gains (for example, Effort AUC 0.739 to 0.944 on HAV-DF-test; AIDE EER 0.484 to 0.259), but consistently degrades performance on held-out IP-AIGC for Indian cohorts (for example, AIDE AUC 0.923 to 0.563 on HIDF-img-ip-genai; Effort 0.740 to 0.533), which indicates overfitting to training-generator cues. On non-IP HIDF images, PT performance remains high, which suggests a specific brittleness to identity-preserving edits rather than a generic distribution shift. Our study establishes IP-AIGC-Indian as a challenging and practically relevant scenario and motivates representation-preserving adaptation and India-aware benchmark curation to close generalization gaps in AIGC detection.
☆ Action Anticipation at a Glimpse: To What Extent Can Multimodal Cues Replace Video? WACV 2026
Manuel Benavent-Lledo, Konstantinos Bacharidis, Victoria Manousaki, Konstantinos Papoutsakis, Antonis Argyros, Jose Garcia-Rodriguez
Anticipating actions before they occur is a core challenge in action understanding research. While conventional methods rely on extracting and aggregating temporal information from videos, as humans we can often predict upcoming actions by observing a single moment from a scene, when given sufficient context. Can a model achieve this competence? The short answer is yes, although its effectiveness depends on the complexity of the task. In this work, we investigate to what extent video aggregation can be replaced with alternative modalities. To this end, based on recent advances in visual feature extraction and language-based reasoning, we introduce AAG, a method for Action Anticipation at a Glimpse. AAG combines RGB features with depth cues from a single frame for enhanced spatial reasoning, and incorporates prior action information to provide long-term context. This context is obtained either through textual summaries from Vision-Language Models, or from predictions generated by a single-frame action recognizer. Our results demonstrate that multimodal single-frame action anticipation using AAG can perform competitively compared to both temporally aggregated video baselines and state-of-the-art methods across three instructional activity datasets: IKEA-ASM, Meccano, and Assembly101.
comment : Accepted in WACV 2026 - Applications Track
☆ ReVSeg: Incentivizing the Reasoning Chain for Video Segmentation with Reinforcement Learning
Yifan Li, Yingda Yin, Lingting Zhu, Weikai Chen, Shengju Qian, Xin Wang, Yanwei Fu
Reasoning-centric video object segmentation is an inherently complex task: the query often refers to dynamics, causality, and temporal interactions, rather than static appearances. Yet existing solutions generally collapse these factors into simplified reasoning with latent embeddings, rendering the reasoning chain opaque and essentially intractable. We therefore adopt an explicit decomposition perspective and introduce ReVSeg, which executes reasoning as sequential decisions in the native interface of pretrained vision language models (VLMs). Rather than folding all reasoning into a single-step prediction, ReVSeg executes three explicit operations -- semantics interpretation, temporal evidence selection, and spatial grounding -- aligning pretrained capabilities. We further employ reinforcement learning to optimize the multi-step reasoning chain, enabling the model to self-refine its decision quality from outcome-driven signals. Experimental results demonstrate that ReVSeg attains state-of-the-art performances on standard video object segmentation benchmarks and yields interpretable reasoning trajectories. Project page is available at https://clementine24.github.io/ReVSeg/ .
☆ Defense That Attacks: How Robust Models Become Better Attackers
Mohamed Awad, Mahmoud Akrm, Walid Gomaa
Deep learning has achieved great success in computer vision, but remains vulnerable to adversarial attacks. Adversarial training is the leading defense designed to improve model robustness. However, its effect on the transferability of attacks is underexplored. In this work, we ask whether adversarial training unintentionally increases the transferability of adversarial examples. To answer this, we trained a diverse zoo of 36 models, including CNNs and ViTs, and conducted comprehensive transferability experiments. Our results reveal a clear paradox: adversarially trained (AT) models produce perturbations that transfer more effectively than those from standard models, which introduce a new ecosystem risk. To enable reproducibility and further study, we release all models, code, and experimental scripts. Furthermore, we argue that robustness evaluations should assess not only the resistance of a model to transferred attacks but also its propensity to produce transferable adversarial examples.
☆ HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval ACM MM 2025
Zhiwei Chen, Yupeng Hu, Zixu Li, Zhiheng Fu, Haokun Wen, Weili Guan
Composed Video Retrieval (CVR) is a challenging video retrieval task that utilizes multi-modal queries, consisting of a reference video and modification text, to retrieve the desired target video. The core of this task lies in understanding the multi-modal composed query and achieving accurate composed feature learning. Within multi-modal queries, the video modality typically carries richer semantic content compared to the textual modality. However, previous works have largely overlooked the disparity in information density between these two modalities. This limitation can lead to two critical issues: 1) modification subject referring ambiguity and 2) limited detailed semantic focus, both of which degrade the performance of CVR models. To address the aforementioned issues, we propose a novel CVR framework, namely the Hierarchical Uncertainty-aware Disambiguation network (HUD). HUD is the first framework that leverages the disparity in information density between video and text to enhance multi-modal query understanding. It comprises three key components: (a) Holistic Pronoun Disambiguation, (b) Atomistic Uncertainty Modeling, and (c) Holistic-to-Atomistic Alignment. By exploiting overlapping semantics through holistic cross-modal interaction and fine-grained semantic alignment via atomistic-level cross-modal interaction, HUD enables effective object disambiguation and enhances the focus on detailed semantics, thereby achieving precise composed feature learning. Moreover, our proposed HUD is also applicable to the Composed Image Retrieval (CIR) task and achieves state-of-the-art performance across three benchmark datasets for both CVR and CIR tasks. The codes are available on https://zivchen-ty.github.io/HUD.github.io/.
comment : Accepted by ACM MM 2025
☆ TrackNetV5: Residual-Driven Spatio-Temporal Refinement and Motion Direction Decoupling for Fast Object Tracking
Tang Haonan, Chen Yanjun, Jiang Lezhi
The TrackNet series has established a strong baseline for fast-moving small object tracking in sports. However, existing iterations face significant limitations: V1-V3 struggle with occlusions due to a reliance on purely visual cues, while TrackNetV4, despite introducing motion inputs, suffers from directional ambiguity as its absolute difference method discards motion polarity. To overcome these bottlenecks, we propose TrackNetV5, a robust architecture integrating two novel mechanisms. First, to recover lost directional priors, we introduce the Motion Direction Decoupling (MDD) module. Unlike V4, MDD decomposes temporal dynamics into signed polarity fields, explicitly encoding both movement occurrence and trajectory direction. Second, we propose the Residual-Driven Spatio-Temporal Refinement (R-STR) head. Operating on a coarse-to-fine paradigm, this Transformer-based module leverages factorized spatio-temporal contexts to estimate a corrective residual, effectively recovering occluded targets. Extensive experiments on the TrackNetV2 dataset demonstrate that TrackNetV5 achieves a new state-of-the-art F1-score of 0.9859 and an accuracy of 0.9733, significantly outperforming previous versions. Notably, this performance leap is achieved with a marginal 3.7% increase in FLOPs compared to V4, maintaining real-time inference capabilities while delivering superior tracking precision.
☆ Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols
Xianchao Zeng, Xinyu Zhou, Youcheng Li, Jiayou Shi, Tianle Li, Liangming Chen, Lei Ren, Yong-Lu Li
Vision-Language-Action (VLA) models have recently achieved remarkable progress in robotic manipulation, yet they remain limited in failure diagnosis and learning from failures. Additionally, existing failure datasets are mostly generated programmatically in simulation, which limits their generalization to the real world. In light of these, we introduce ViFailback, a framework designed to diagnose robotic manipulation failures and provide both textual and visual correction guidance. Our framework utilizes explicit visual symbols to enhance annotation efficiency. We further release the ViFailback dataset, a large-scale collection of 58,126 Visual Question Answering (VQA) pairs along with their corresponding 5,202 real-world manipulation trajectories. Based on the dataset, we establish ViFailback-Bench, a benchmark of 11 fine-grained VQA tasks designed to assess the failure diagnosis and correction abilities of Vision-Language Models (VLMs), featuring ViFailback-Bench Lite for closed-ended and ViFailback-Bench Hard for open-ended evaluation. To demonstrate the effectiveness of our framework, we built the ViFailback-8B VLM, which not only achieves significant overall performance improvement on ViFailback-Bench but also generates visual symbols for corrective action guidance. Finally, by integrating ViFailback-8B with a VLA model, we conduct real-world robotic experiments demonstrating its ability to assist the VLA model in recovering from failures. Project Website: https://x1nyuzhou.github.io/vifailback.github.io/
☆ LumiX: Structured and Coherent Text-to-Intrinsic Generation
Xu Han, Biao Zhang, Xiangjun Tang, Xianzhi Li, Peter Wonka
We present LumiX, a structured diffusion framework for coherent text-to-intrinsic generation. Conditioned on text prompts, LumiX jointly generates a comprehensive set of intrinsic maps (e.g., albedo, irradiance, normal, depth, and final color), providing a structured and physically consistent description of an underlying scene. This is enabled by two key contributions: 1) Query-Broadcast Attention, a mechanism that ensures structural consistency by sharing queries across all maps in each self-attention block. 2) Tensor LoRA, a tensor-based adaptation that parameter-efficiently models cross-map relations for efficient joint training. Together, these designs enable stable joint diffusion training and unified generation of multiple intrinsic properties. Experiments show that LumiX produces coherent and physically meaningful results, achieving 23% higher alignment and a better preference score (0.19 vs. -0.41) compared to the state of the art, and it can also perform image-conditioned intrinsic decomposition within the same framework.
comment : The code will be available at https://github.com/xhanxu/LumiX
☆ Rethinking Surgical Smoke: A Smoke-Type-Aware Laparoscopic Video Desmoking Method and Dataset AAAI-26
Qifan Liang, Junlin Li, Zhen Han, Xihao Wang, Zhongyuan Wang, Bin Mei
Electrocautery or lasers will inevitably generate surgical smoke, which hinders the visual guidance of laparoscopic videos for surgical procedures. The surgical smoke can be classified into different types based on its motion patterns, leading to distinctive spatio-temporal characteristics across smoky laparoscopic videos. However, existing desmoking methods fail to account for such smoke-type-specific distinctions. Therefore, we propose the first Smoke-Type-Aware Laparoscopic Video Desmoking Network (STANet) by introducing two smoke types: Diffusion Smoke and Ambient Smoke. Specifically, a smoke mask segmentation sub-network is designed to jointly conduct smoke mask and smoke type predictions based on the attention-weighted mask aggregation, while a smokeless video reconstruction sub-network is proposed to perform specially desmoking on smoky features guided by two types of smoke mask. To address the entanglement challenges of two smoke types, we further embed a coarse-to-fine disentanglement module into the mask segmentation sub-network, which yields more accurate disentangled masks through the smoke-type-aware cross attention between non-entangled and entangled regions. In addition, we also construct the first large-scale synthetic video desmoking dataset with smoke type annotations. Extensive experiments demonstrate that our method not only outperforms state-of-the-art approaches in quality evaluations, but also exhibits superior generalization across multiple downstream surgical tasks.
comment : 12 pages, 15 figures. Accepted to AAAI-26 (Main Technical Track)
☆ AttMetNet: Attention-Enhanced Deep Neural Network for Methane Plume Detection in Sentinel-2 Satellite Imagery
Rakib Ahsan, MD Sadik Hossain Shanto, Md Sultanul Arifin, Tanzima Hashem
Methane is a powerful greenhouse gas that contributes significantly to global warming. Accurate detection of methane emissions is the key to taking timely action and minimizing their impact on climate change. We present AttMetNet, a novel attention-enhanced deep learning framework for methane plume detection with Sentinel-2 satellite imagery. The major challenge in developing a methane detection model is to accurately identify methane plumes from Sentinel-2's B11 and B12 bands while suppressing false positives caused by background variability and diverse land cover types. Traditional detection methods typically depend on the differences or ratios between these bands when comparing the scenes with and without plumes. However, these methods often require verification by a domain expert because they generate numerous false positives. Recent deep learning methods make some improvements using CNN-based architectures, but lack mechanisms to prioritize methane-specific features. AttMetNet introduces a methane-aware architecture that fuses the Normalized Difference Methane Index (NDMI) with an attention-enhanced U-Net. By jointly exploiting NDMI's plume-sensitive cues and attention-driven feature selection, AttMetNet selectively amplifies methane absorption features while suppressing background noise. This integration establishes a first-of-its-kind architecture tailored for robust methane plume detection in real satellite imagery. Additionally, we employ focal loss to address the severe class imbalance arising from both limited positive plume samples and sparse plume pixels within imagery. Furthermore, AttMetNet is trained on the real methane plume dataset, making it more robust to practical scenarios. Extensive experiments show that AttMetNet surpasses recent methods in methane plume detection with a lower false positive rate, better precision recall balance, and higher IoU.
comment : 15 pages, 4 figures
☆ Reasoning -Aware Multimodal Fusion for Hateful Video Detection
Shuonan Yang, Tailin Chen, Jiangbei Yue, Guangliang Cheng, Jianbo Jiao, Zeyu Fu
Hate speech in online videos is posing an increasingly serious threat to digital platforms, especially as video content becomes increasingly multimodal and context-dependent. Existing methods often struggle to effectively fuse the complex semantic relationships between modalities and lack the ability to understand nuanced hateful content. To address these issues, we propose an innovative Reasoning-Aware Multimodal Fusion (RAMF) framework. To tackle the first challenge, we design Local-Global Context Fusion (LGCF) to capture both local salient cues and global temporal structures, and propose Semantic Cross Attention (SCA) to enable fine-grained multimodal semantic interaction. To tackle the second challenge, we introduce adversarial reasoning-a structured three-stage process where a vision-language model generates (i) objective descriptions, (ii) hate-assumed inferences, and (iii) non-hate-assumed inferences-providing complementary semantic perspectives that enrich the model's contextual understanding of nuanced hateful intent. Evaluations on two real-world hateful video datasets demonstrate that our method achieves robust generalisation performance, improving upon state-of-the-art methods by 3% and 7% in Macro-F1 and hate class recall, respectively. We will release the code after the anonymity period ends.
☆ Beyond Paired Data: Self-Supervised UAV Geo-Localization from Reference Imagery Alone WACV 2026
Tristan Amadei, Enric Meinhardt-Llopis, Benedicte Bascle, Corentin Abgrall, Gabriele Facciolo
Image-based localization in GNSS-denied environments is critical for UAV autonomy. Existing state-of-the-art approaches rely on matching UAV images to geo-referenced satellite images; however, they typically require large-scale, paired UAV-satellite datasets for training. Such data are costly to acquire and often unavailable, limiting their applicability. To address this challenge, we adopt a training paradigm that removes the need for UAV imagery during training by learning directly from satellite-view reference images. This is achieved through a dedicated augmentation strategy that simulates the visual domain shift between satellite and real-world UAV views. We introduce CAEVL, an efficient model designed to exploit this paradigm, and validate it on ViLD, a new and challenging dataset of real-world UAV images that we release to the community. Our method achieves competitive performance compared to approaches trained with paired data, demonstrating its effectiveness and strong generalization capabilities.
comment : Accepted at WACV 2026
☆ DF-Mamba: Deformable State Space Modeling for 3D Hand Pose Estimation in Interactions WACV 2026
Yifan Zhou, Takehiko Ohkawa, Guwenxiao Zhou, Kanoko Goto, Takumi Hirose, Yusuke Sekikawa, Nakamasa Inoue
Modeling daily hand interactions often struggles with severe occlusions, such as when two hands overlap, which highlights the need for robust feature learning in 3D hand pose estimation (HPE). To handle such occluded hand images, it is vital to effectively learn the relationship between local image features (e.g., for occluded joints) and global context (e.g., cues from inter-joints, inter-hands, or the scene). However, most current 3D HPE methods still rely on ResNet for feature extraction, and such CNN's inductive bias may not be optimal for 3D HPE due to its limited capability to model the global context. To address this limitation, we propose an effective and efficient framework for visual feature extraction in 3D HPE using recent state space modeling (i.e., Mamba), dubbed Deformable Mamba (DF-Mamba). DF-Mamba is designed to capture global context cues beyond standard convolution through Mamba's selective state modeling and the proposed deformable state scanning. Specifically, for local features after convolution, our deformable scanning aggregates these features within an image while selectively preserving useful cues that represent the global context. This approach significantly improves the accuracy of structured 3D HPE, with comparable inference speed to ResNet-50. Our experiments involve extensive evaluations on five divergent datasets including single-hand and two-hand scenarios, hand-only and hand-object interactions, as well as RGB and depth-based estimation. DF-Mamba outperforms the latest image backbones, including VMamba and Spatial-Mamba, on all datasets and achieves state-of-the-art performance.
comment : Accepted to WACV 2026. Project page: https://tkhkaeio.github.io/projects/25-dfmamba/index.html
☆ Emergent Bayesian Behaviour and Optimal Cue Combination in LLMs
Julian Ma, Jun Wang, Zafeirios Fountas
Large language models (LLMs) excel at explicit reasoning, but their implicit computational strategies remain underexplored. Decades of psychophysics research show that humans intuitively process and integrate noisy signals using near-optimal Bayesian strategies in perceptual tasks. We ask whether LLMs exhibit similar behaviour and perform optimal multimodal integration without explicit training or instruction. Adopting the psychophysics paradigm, we infer computational principles of LLMs from systematic behavioural studies. We introduce a behavioural benchmark - BayesBench: four magnitude estimation tasks (length, location, distance, and duration) over text and image, inspired by classic psychophysics, and evaluate a diverse set of nine LLMs alongside human judgments for calibration. Through controlled ablations of noise, context, and instruction prompts, we measure performance, behaviour and efficiency in multimodal cue-combination. Beyond accuracy and efficiency metrics, we introduce a Bayesian Consistency Score that detects Bayes-consistent behavioural shifts even when accuracy saturates. Our results show that while capable models often adapt in Bayes-consistent ways, accuracy does not guarantee robustness. Notably, GPT-5 Mini achieves perfect text accuracy but fails to integrate visual cues efficiently. This reveals a critical dissociation between capability and strategy, suggesting accuracy-centric benchmarks may over-index on performance while missing brittle uncertainty handling. These findings reveal emergent principled handling of uncertainty and highlight the correlation between accuracy and Bayesian tendencies. We release our psychophysics benchmark and consistency metric (https://bayes-bench.github.io) as evaluation tools and to inform future multimodal architecture designs.
☆ GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Grounding
Peirong Zhang, Yidan Zhang, Luxiao Xu, Jinliang Lin, Zonghao Guo, Fengxiang Wang, Xue Yang, Kaiwen Wei, Lei Wang
Recent advances in multimodal large language models(MLLMs) have led to remarkable progress in visual grounding, enabling fine-grained cross-modal alignment between textual queries and image regions. However, transferring such capabilities to remote sensing imagery remains challenging, as targets are often extremely small within kilometer-scale scenes, and queries typically involve intricate geospatial relations such as relative positions, spatial hierarchies, or contextual dependencies across distant objects. To address these challenges, we propose GeoViS, a Geospatially Rewarded Visual Search framework that reformulates remote sensing visual grounding as a progressive search-and-reasoning process. Rather than directly predicting the target location in a single step, GeoViS actively explores the global image through a tree-structured sequence of visual cues, integrating multimodal perception, spatial reasoning, and reward-guided exploration to refine geospatial hypotheses iteratively. This design enables the model to detect subtle small-scale targets while maintaining holistic scene awareness. Extensive experiments on five remote sensing grounding benchmarks demonstrate that GeoViS achieves precise geospatial understanding and consistently surpasses existing methods across key visual grounding metrics, highlighting its strong cross-domain generalization and interpretability.
comment : 11 pages, 4 figures
☆ Tissue-mask supported inter-subject whole-body image registration in the UK Biobank -- A method benchmark ing study
Yasemin Utkueri, Elin Lundström, Håkan Ahlström, Johan Öfverstedt, Joel Kullberg
The UK Biobank is a large-scale study collecting whole-body MR imaging and non-imaging health data. Robust and accurate inter-subject image registration of these whole-body MR images would enable their body-wide spatial standardization, and region-/voxel-wise correlation analysis of non-imaging data with image-derived parameters (e.g., tissue volume or fat content). We propose a sex-stratified inter-subject whole-body MR image registration approach that uses subcutaneous adipose tissue- and muscle-masks from the state-of-the-art VIBESegmentator method to augment intensity-based graph-cut registration. The proposed method was evaluated on a subset of 4000 subjects by comparing it to an intensity-only method as well as two previously published registration methods, uniGradICON and MIRTK. The evaluation comprised overlap measures applied to the 71 VIBESegmentator masks: 1) Dice scores, and 2) voxel-wise label error frequency. Additionally, voxel-wise correlation between age and each of fat content and tissue volume was studied to exemplify the usefulness for medical research. The proposed method exhibited a mean dice score of 0.77 / 0.75 across the cohort and the 71 masks for males/females, respectively. When compared to the intensity-only registration, the mean values were 6 percentage points (pp) higher for both sexes, and the label error frequency was decreased in most tissue regions. These differences were 9pp / 8pp against uniGradICON and 12pp / 13pp against MIRTK. Using the proposed method, the age-correlation maps were less noisy and showed higher anatomical alignment. In conclusion, the image registration method using two tissue masks improves whole-body registration of UK Biobank images.
☆ VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigm
Zhenkai Wu, Xiaowen Ma, Zhenliang Ni, Dengming Zhang, Han Shu, Xin Jiang, Xinghao Chen
Vision-language models (VLMs) excel at image understanding tasks, but the large number of visual tokens imposes significant computational costs, hindering deployment on mobile devices. Many pruning methods rely solely on token importance and thus overlook inter-token redundancy, retaining numerous duplicated tokens and wasting capacity. Although some redundancy-aware approaches have been proposed, they often ignore the spatial relationships among visual tokens. This can lead to overly sparse selections of retained tokens that fail to adequately cover the regions of target objects. To address these limitations, we propose VLM-Pruner, a training-free token pruning algorithm that explicitly balances redundancy and spatial sparsity. We introduce a centrifugal token pruning paradigm that enables near-to-far selection while prioritizing the preservation of fine-grained object details. Moreover, we design a Buffering for Spatial Sparsity (BSS) criterion that defers the selection of spatially distant tokens. We further adopt a parallel greedy strategy to conduct token selection efficiently. To mitigate information loss from pruning, we selectively fuse salient information from the discarded tokens into the retained ones. Comprehensive comparisons demonstrate that VLM-Pruner consistently outperforms strong baselines across five VLMs with an 88.9\% pruning rate, while delivering an end-to-end inference speedup.
☆ GeoBridge: A Semantic-Anchored Multi-View Foundation Model Bridging Images and Text for Geo-Localization
Zixuan Song, Jing Zhang, Di Wang, Zidie Zhou, Wenbin Liu, Haonan Guo, En Wang, Bo Du
Cross-view geo-localization infers a location by retrieving geo-tagged reference images that visually correspond to a query image. However, the traditional satellite-centric paradigm limits robustness when high-resolution or up-to-date satellite imagery is unavailable. It further underexploits complementary cues across views (e.g., drone, satellite, and street) and modalities (e.g., language and image). To address these challenges, we propose GeoBridge, a foundation model that performs bidirectional matching across views and supports language-to-image retrieval. Going beyond traditional satellite-centric formulations, GeoBridge builds on a novel semantic-anchor mechanism that bridges multi-view features through textual descriptions for robust, flexible localization. In support of this task, we construct GeoLoc, the first large-scale, cross-modal, and multi-view aligned dataset comprising over 50,000 pairs of drone, street-view panorama, and satellite images as well as their textual descriptions, collected from 36 countries, ensuring both geographic and semantic alignment. We performed broad evaluations across multiple tasks. Experiments confirm that GeoLoc pre-training markedly improves geo-location accuracy for GeoBridge while promoting cross-domain generalization and cross-modal knowledge transfer. The dataset, source code, and pretrained models were released at https://github.com/MiliLab/GeoBridge.
☆ ALDI-ray: Adapting the ALDI Framework for Security X-ray Object Detection ICASSP 2026
Omid Reza Heidari, Yang Wang, Xinxin Zuo
Domain adaptation in object detection is critical for real-world applications where distribution shifts degrade model performance. Security X-ray imaging presents a unique challenge due to variations in scanning devices and environmental conditions, leading to significant domain discrepancies. To address this, we apply ALDI++, a domain adaptation framework that integrates self-distillation, feature alignment, and enhanced training strategies to mitigate domain shift effectively in this area. We conduct extensive experiments on the EDS dataset, demonstrating that ALDI++ surpasses the state-of-the-art (SOTA) domain adaptation methods across multiple adaptation scenarios. In particular, ALDI++ with a Vision Transformer for Detection (ViTDet) backbone achieves the highest mean average precision (mAP), confirming the effectiveness of transformer-based architectures for cross-domain object detection. Additionally, our category-wise analysis highlights consistent improvements in detection accuracy, reinforcing the robustness of the model across diverse object classes. Our findings establish ALDI++ as an efficient solution for domain-adaptive object detection, setting a new benchmark for performance stability and cross-domain generalization in security X-ray imagery.
comment : Submitted to ICASSP 2026 Conference
☆ ClimaOoD: Improving Anomaly Segmentation via Physically Realistic Synthetic Data
Yuxing Liu, Yong Liu
Anomaly segmentation seeks to detect and localize unknown or out-of-distribution (OoD) objects that fall outside predefined semantic classes a capability essential for safe autonomous driving. However, the scarcity and limited diversity of anomaly data severely constrain model generalization in open-world environments. Existing approaches mitigate this issue through synthetic data generation, either by copy-pasting external objects into driving scenes or by leveraging text-to-image diffusion models to inpaint anomalous regions. While these methods improve anomaly diversity, they often lack contextual coherence and physical realism, resulting in domain gaps between synthetic and real data. In this paper, we present ClimaDrive, a semantics-guided image-to-image framework for synthesizing semantically coherent, weather-diverse, and physically plausible OoD driving data. ClimaDrive unifies structure-guided multi-weather generation with prompt-driven anomaly inpainting, enabling the creation of visually realistic training data. Based on this framework, we construct ClimaOoD, a large-scale benchmark spanning six representative driving scenarios under both clear and adverse weather conditions. Extensive experiments on four state-of-the-art methods show that training with ClimaOoD leads to robust improvements in anomaly segmentation. Across all methods, AUROC, AP, and FPR95 show notable gains, with FPR95 dropping from 3.97 to 3.52 for RbA on Fishyscapes LAF. These results demonstrate that ClimaOoD enhances model robustness, offering valuable training data for better generalization in open-world anomaly detection.
comment : Under review;
☆ Unsupervised Structural Scene Decomposition via Foreground-Aware Slot Attention with Pseudo-Mask Guidance
Huankun Sheng, Ming Li, Yixiang Wei, Yeying Fan, Yu-Hui Wen, Tieliang Gong, Yong-Jin Liu
Recent advances in object-centric representation learning have shown that slot attention-based methods can effectively decompose visual scenes into object slot representations without supervision. However, existing approaches typically process foreground and background regions indiscriminately, often resulting in background interference and suboptimal instance discovery performance on real-world data. To address this limitation, we propose Foreground-Aware Slot Attention (FASA), a two-stage framework that explicitly separates foreground from background to enable precise object discovery. In the first stage, FASA performs a coarse scene decomposition to distinguish foreground from background regions through a dual-slot competition mechanism. These slots are initialized via a clustering-based strategy, yielding well-structured representations of salient regions. In the second stage, we introduce a masked slot attention mechanism where the first slot captures the background while the remaining slots compete to represent individual foreground objects. To further address over-segmentation of foreground objects, we incorporate pseudo-mask guidance derived from a patch affinity graph constructed with self-supervised image features to guide the learning of foreground slots. Extensive experiments on both synthetic and real-world datasets demonstrate that FASA consistently outperforms state-of-the-art methods, validating the effectiveness of explicit foreground modeling and pseudo-mask guidance for robust scene decomposition and object-coherent representation. Code will be made publicly available.
☆ PGP-DiffSR: Phase-Guided Progressive Pruning for Efficient Diffusion -based Image Super-Resolution
Zhongbao Yang, Jiangxin Dong, Yazhou Yao, Jinhui Tang, Jinshan Pan
Although diffusion-based models have achieved impressive results in image super-resolution, they often rely on large-scale backbones such as Stable Diffusion XL (SDXL) and Diffusion Transformers (DiT), which lead to excessive computational and memory costs during training and inference. To address this issue, we develop a lightweight diffusion method, PGP-DiffSR, by removing redundant information from diffusion models under the guidance of the phase information of inputs for efficient image super-resolution. We first identify the intra-block redundancy within the diffusion backbone and propose a progressive pruning approach that removes redundant blocks while reserving restoration capability. We note that the phase information of the restored images produced by the pruned diffusion model is not well estimated. To solve this problem, we propose a phase-exchange adapter module that explores the phase information of the inputs to guide the pruned diffusion model for better restoration performance. We formulate the progressive pruning approach and the phase-exchange adapter module into a unified model. Extensive experiments demonstrate that our method achieves competitive restoration quality while significantly reducing computational load and memory consumption. The code is available at https://github.com/yzb1997/PGP-DiffSR.
comment : 10 pages
☆ UAUTrack: Towards Unified Multimodal Anti-UAV Visual Tracking
Qionglin Ren, Dawei Zhang, Chunxu Tian, Dan Zhang
Research in Anti-UAV (Unmanned Aerial Vehicle) tracking has explored various modalities, including RGB, TIR, and RGB-T fusion. However, a unified framework for cross-modal collaboration is still lacking. Existing approaches have primarily focused on independent models for individual tasks, often overlooking the potential for cross-modal information sharing. Furthermore, Anti-UAV tracking techniques are still in their infancy, with current solutions struggling to achieve effective multimodal data fusion. To address these challenges, we propose UAUTrack, a unified single-target tracking framework built upon a single-stream, single-stage, end-to-end architecture that effectively integrates multiple modalities. UAUTrack introduces a key component: a text prior prompt strategy that directs the model to focus on UAVs across various scenarios. Experimental results show that UAUTrack achieves state-of-the-art performance on the Anti-UAV and DUT Anti-UAV datasets, and maintains a favourable trade-off between accuracy and speed on the Anti-UAV410 dataset, demonstrating both high accuracy and practical efficiency across diverse Anti-UAV scenarios.
☆ PolarGuide-GSDR: 3D Gaussian Splatting Driven by Polarization Priors and Deferred Reflection for Real-Wo rl d Reflective Scenes
Derui Shan, Qian Qiao, Hao Lu, Tao Du, Peng Lu
Polarization-aware Neural Radiance Fields (NeRF) enable novel view synthesis of specular-reflection scenes but face challenges in slow training, inefficient rendering, and strong dependencies on material/viewpoint assumptions. However, 3D Gaussian Splatting (3DGS) enables real-time rendering yet struggles with accurate reflection reconstruction from reflection-geometry entanglement, adding a deferred reflection module introduces environment map dependence. We address these limitations by proposing PolarGuide-GSDR, a polarization-forward-guided paradigm establishing a bidirectional coupling mechanism between polarization and 3DGS: first 3DGS's geometric priors are leveraged to resolve polarization ambiguity, and then the refined polarization information cues are used to guide 3DGS's normal and spherical harmonic representation. This process achieves high-fidelity reflection separation and full-scene reconstruction without requiring environment maps or restrictive material assumptions. We demonstrate on public and self-collected datasets that PolarGuide-GSDR achieves state-of-the-art performance in specular reconstruction, normal estimation, and novel view synthesis, all while maintaining real-time rendering capabilities. To our knowledge, this is the first framework embedding polarization priors directly into 3DGS optimization, yielding superior interpretability and real-time performance for modeling complex reflective scenes.
☆ Spatially-Grounded Document Retrieval via Patch-to-Region Relevance Propagation
Agathoklis Georgiou
Vision-language models (VLMs) like ColPali achieve state-of-the-art document retrieval by embedding pages as images and computing fine-grained similarity between query tokens and visual patches. However, they return entire pages rather than specific regions, limiting utility for retrieval-augmented generation (RAG) where precise context is paramount. Conversely, OCR-based systems extract structured text with bounding box coordinates but lack semantic grounding for relevance assessment. We propose a hybrid architecture that unifies these paradigms: using ColPali's patch-level similarity scores as spatial relevance filters over OCR-extracted regions. We formalize the coordinate mapping between vision transformer patch grids and OCR bounding boxes, introduce intersection metrics for relevance propagation, and establish theoretical bounds on retrieval precision. Our approach operates at inference time without additional training. We release Snappy, an open-source implementation demonstrating practical applicability, with empirical evaluation ongoing.
comment : 13 pages, 1 figure, 2 tables. Open-source implementation available at https://github.com/athrael-soju/Snappy
☆ Real-Time Multimodal Data Collection Using Smartwatches and Its Visualization in Education
Alvaro Becerra, Pablo Villegas, Ruth Cobos
Wearable sensors, such as smartwatches, have become increasingly prevalent across domains like healthcare, sports, and education, enabling continuous monitoring of physiological and behavioral data. In the context of education, these technologies offer new opportunities to study cognitive and affective processes such as engagement, attention, and performance. However, the lack of scalable, synchronized, and high-resolution tools for multimodal data acquisition continues to be a significant barrier to the widespread adoption of Multimodal Learning Analytics in real-world educational settings. This paper presents two complementary tools developed to address these challenges: Watch-DMLT, a data acquisition application for Fitbit Sense 2 smartwatches that enables real-time, multi-user monitoring of physiological and motion signals; and ViSeDOPS, a dashboard-based visualization system for analyzing synchronized multimodal data collected during oral presentations. We report on a classroom deployment involving 65 students and up to 16 smartwatches, where data streams including heart rate, motion, gaze, video, and contextual annotations were captured and analyzed. Results demonstrate the feasibility and utility of the proposed system for supporting fine-grained, scalable, and interpretable Multimodal Learning Analytics in real learning environments.
comment : Accepted in Technological Ecosystems for Enhancing Multiculturality (TEEM) 2025
☆ Hear What Matters! Text-conditioned Selective Video-to-Audio Generation
Junwon Lee, Juhan Nam, Jiyoung Lee
This work introduces a new task, text-conditioned selective video-to-audio (V2A) generation, which produces only the user-intended sound from a multi-object video. This capability is especially crucial in multimedia production, where audio tracks are handled individually for each sound source for precise editing, mixing, and creative control. However, current approaches generate single source-mixed sounds at once, largely because visual features are entangled, and region cues or prompts often fail to specify the source. We propose SelVA, a novel text-conditioned V2A model that treats the text prompt as an explicit selector of target source and modulates video encoder to distinctly extract prompt-relevant video features. The proposed supplementary tokens promote cross-attention by suppressing text-irrelevant activations with efficient parameter tuning, yielding robust semantic and temporal grounding. SelVA further employs a self-augmentation scheme to overcome the lack of mono audio track supervision. We evaluate SelVA on VGG-MONOAUDIO, a curated benchmark of clean single-source videos for such a task. Extensive experiments and ablations consistently verify its effectiveness across audio quality, semantic alignment, and temporal synchronization. Code and demo are available at https://jnwnlee.github.io/selva-demo/.
☆ PoreTrack3D: A Benchmark for Dynamic 3D Gaussian Splatting in Pore-Scale Facial Trajectory Tracking
Dong Li, Jiahao Xiong, Yingda Huang, Le Chang
We introduce PoreTrack3D, the first benchmark for dynamic 3D Gaussian splatting in pore-scale, non-rigid 3D facial trajectory tracking. It contains over 440,000 facial trajectories in total, among which more than 52,000 are longer than 10 frames, including 68 manually reviewed trajectories that span the entire 150 frames. To the best of our knowledge, PoreTrack3D is the first benchmark dataset to capture both traditional facial landmarks and pore-scale keypoints trajectory, advancing the study of fine-grained facial expressions through the analysis of subtle skin-surface motion. We systematically evaluate state-of-the-art dynamic 3D Gaussian splatting methods on PoreTrack3D, establishing the first performance baseline in this domain. Overall, the pipeline developed for this benchmark dataset's creation establishes a new framework for high-fidelity facial motion capture and dynamic 3D reconstruction. Our dataset are publicly available at: https://github.com/JHXion9/PoreTrack3D
☆ Leveraging Large-Scale Pretrain ed Spatial-Spectral Priors for General Zero-Shot Pansharpening
Yongchuan Cui, Peng Liu, Yi Zeng
Existing deep learning methods for remote sensing image fusion often suffer from poor generalization when applied to unseen datasets due to the limited availability of real training data and the domain gap between different satellite sensors. To address this challenge, we explore the potential of foundation models by proposing a novel pretraining strategy that leverages large-scale simulated datasets to learn robust spatial-spectral priors. Specifically, our approach first constructs diverse simulated datasets by applying various degradation operations (blur, noise, downsampling) and augmentations (bands generation, channel shuffling, high-pass filtering, color jittering, etc.) to natural images from ImageNet and remote sensing images from SkyScript. We then pretrain fusion models on these simulated data to learn generalizable spatial-spectral representations. The pretrained models are subsequently evaluated on six datasets (WorldView-2/3/4, IKONOS, QuickBird, GaoFen-2) using zero-shot and one-shot paradigms, with both full- and freeze-tuning approaches for fine-tuning. Extensive experiments on different network architectures including convolutional neural networks, Transformer, and Mamba demonstrate that our pretraining strategy significantly improves generalization performance across different satellite sensors and imaging conditions for various fusion models. The pretrained models achieve superior results in zero-shot scenarios and show remarkable adaptation capability with minimal real data in one-shot settings. Our work provides a practical solution for cross-domain pansharpening, establishes a new benchmark for generalization in remote sensing image fusion tasks, and paves the way for leveraging foundation models through advanced training strategies.
☆ Joint Distillation for Fast Likelihood Evaluation and Sampling in Flow-based Models
Xinyue Ai, Yutong He, Albert Gu, Ruslan Salakhutdinov, J Zico Kolter, Nicholas Matthew Boffi, Max Simchowitz
Log-likelihood evaluation enables important capabilities in generative models, including model comparison, certain fine-tuning objectives, and many downstream applications. Yet paradoxically, some of today's best generative models -- diffusion and flow-based models -- still require hundreds to thousands of neural function evaluations (NFEs) to compute a single likelihood. While recent distillation methods have successfully accelerated sampling to just a few steps, they achieve this at the cost of likelihood tractability: existing approaches either abandon likelihood computation entirely or still require expensive integration over full trajectories. We present fast flow joint distillation (F2D2), a framework that simultaneously reduces the number of NFEs required for both sampling and likelihood evaluation by two orders of magnitude. Our key insight is that in continuous normalizing flows, the coupled ODEs for sampling and likelihood are computed from a shared underlying velocity field, allowing us to jointly distill both the sampling trajectory and cumulative divergence using a single model. F2D2 is modular, compatible with existing flow-based few-step sampling models, and requires only an additional divergence prediction head. Experiments demonstrate F2D2's capability of achieving accurate log-likelihood with few-step evaluations while maintaining high sample quality, solving a long-standing computational bottleneck in flow-based generative models. As an application of our approach, we propose a lightweight self-guidance method that enables a 2-step MeanFlow model to outperform a 1024 step teacher model with only a single additional backward NFE.
☆ PPTBench: Towards Holistic Evaluation of Large Language Models for PowerPoint Layout and Design Understanding
Zheng Huang, Xukai Liu, Tianyu Hu, Kai Zhang, Ye Liu
PowerPoint presentations combine rich textual content with structured visual layouts, making them a natural testbed for evaluating the multimodal reasoning and layout understanding abilities of modern MLLMs. However, existing benchmarks focus solely on narrow subtasks while overlooking layout-centric challenges, which are central to real-world slide creation and editing. To bridge this gap, we introduce PPTBench, a comprehensive multimodal benchmark for evaluating LLMs on PowerPoint-related tasks. Leveraging a diverse source of 958 PPTX files, PPTBench evaluates models across four categories with 4,439 samples, including Detection, Understanding, Modification, and Generation. Our experiments reveal a substantial gap between semantic understanding and visual-layout reasoning in current MLLMs: models can interpret slide content but fail to produce coherent spatial arrangements. Ablation and further analysis show that current MLLMs struggle to combine visual cues with JSON-based layout structures and fail to integrate visual information into their API planning ability. And case studies visually expose systematic layout errors such as misalignment and element overlap. These findings provides a new perspective on evaluating VLLMs in PPT scenarios, highlighting challenges and directions for future research on visual-structural reasoning and coherent slide generation. All datasets and code are fully released to support reproducibility and future research.
☆ RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligence
Xuming He, Zehao Fan, Hengjia Li, Fan Zhuo, Hankun Xu, Senlin Cheng, Di Weng, Haifeng Liu, Can Ye, Boxi Wu
Recent advances in video generation have enabled the synthesis of videos with strong temporal consistency and impressive visual quality, marking a crucial step toward vision foundation models. To evaluate these video generation models, existing benchmarks primarily focus on factors related to visual perception and understanding, like visual aesthetics, instruction adherence, and temporal coherence. However, the rule-based reasoning capabilities of video generation models remain largely unexplored. Although recent studies have carried out preliminary explorations into whether video models can serve as zero-shot learners, they still lack a fine-grained decomposition of reasoning capabilities and a comprehensive evaluation protocol. To address this gap, we introduce RULER-Bench, a benchmark designed to evaluate the reasoning ability of video generation models from the perspective of cognitive rules. Built upon two fundamental paradigms: text-to-video and image-to-video, RULER-Bench covers 40 representative tasks spanning six rule categories with 622 high-quality annotated instances. For the evaluation of each generated video, we construct a checklist covering four metrics and leverage GPT-o3 to assign scores to each question, achieving 85% alignment with human judgements. Extensive experiments show that the state-of-the-art model achieves only 48.87% on the rule coherence metric, highlighting significant room for improvement in the reasoning capability of next-level video models. We expect that the insight obtained from RULER-Bench will facilitate further development of reasoning-aware video generation, advancing video generation models toward vision foundation intelligence.
☆ Content-Aware Texturing for Gaussian Splatting
Panagiotis Papantonakis, Georgios Kopanas, Fredo Durand, George Drettakis
Gaussian Splatting has become the method of choice for 3D reconstruction and real-time rendering of captured real scenes. However, fine appearance details need to be represented as a large number of small Gaussian primitives, which can be wasteful when geometry and appearance exhibit different frequency characteristics. Inspired by the long tradition of texture mapping, we propose to use texture to represent detailed appearance where possible. Our main focus is to incorporate per-primitive texture maps that adapt to the scene in a principled manner during Gaussian Splatting optimization. We do this by proposing a new appearance representation for 2D Gaussian primitives with textures where the size of a texel is bounded by the image sampling frequency and adapted to the content of the input images. We achieve this by adaptively upscaling or downscaling the texture resolution during optimization. In addition, our approach enables control of the number of primitives during optimization based on texture resolution. We show that our approach performs favorably in image quality and total number of parameters used compared to alternative solutions for textured Gaussian primitives. Project page: https://repo-sam.inria.fr/nerphys/gs-texturing/
comment : Project Page: https://repo-sam.inria.fr/nerphys/gs-texturing/
☆ SAM2Grasp: Resolve Multi-modal Grasping via Prompt -conditioned Temporal Action Prediction
Shengkai Wu, Jinrong Yang, Wenqiu Luo, Linfeng Gao, Chaohui Shang, Meiyu Zhi, Mingshan Sun, Fangping Yang, Liangliang Ren, Yong Zhao
Imitation learning for robotic grasping is often plagued by the multimodal problem: when a scene contains multiple valid targets, demonstrations of grasping different objects create conflicting training signals. Standard imitation learning policies fail by averaging these distinct actions into a single, invalid action. In this paper, we introduce SAM2Grasp, a novel framework that resolves this issue by reformulating the task as a uni-modal, prompt-conditioned prediction problem. Our method leverages the frozen SAM2 model to use its powerful visual temporal tracking capability and introduces a lightweight, trainable action head that operates in parallel with its native segmentation head. This design allows for training only the small action head on pre-computed temporal-visual features from SAM2. During inference, an initial prompt, such as a bounding box provided by an upstream object detection model, designates the specific object to be grasped. This prompt conditions the action head to predict a unique, unambiguous grasp trajectory for that object alone. In all subsequent video frames, SAM2's built-in temporal tracking capability automatically maintains stable tracking of the selected object, enabling our model to continuously predict the grasp trajectory from the video stream without further external guidance. This temporal-prompted approach effectively eliminates ambiguity from the visuomotor policy. We demonstrate through extensive experiments that SAM2Grasp achieves state-of-the-art performance in cluttered, multi-object grasping tasks.
☆ Co-speech Gesture Video Generation via Motion-Based Graph Retrieval
Yafei Song, Peng Zhang, Bang Zhang
Synthesizing synchronized and natural co-speech gesture videos remains a formidable challenge. Recent approaches have leveraged motion graphs to harness the potential of existing video data. To retrieve an appropriate trajectory from the graph, previous methods either utilize the distance between features extracted from the input audio and those associated with the motions in the graph or embed both the input audio and motion into a shared feature space. However, these techniques may not be optimal due to the many-to-many mapping nature between audio and gestures, which cannot be adequately addressed by one-to-one mapping. To alleviate this limitation, we propose a novel framework that initially employs a diffusion model to generate gesture motions. The diffusion model implicitly learns the joint distribution of audio and motion, enabling the generation of contextually appropriate gestures from input audio sequences. Furthermore, our method extracts both low-level and high-level features from the input audio to enrich the training process of the diffusion model. Subsequently, a meticulously designed motion-based retrieval algorithm is applied to identify the most suitable path within the graph by assessing both global and local similarities in motion. Given that not all nodes in the retrieved path are sequentially continuous, the final step involves seamlessly stitching together these segments to produce a coherent video output. Experimental results substantiate the efficacy of our proposed method, demonstrating a significant improvement over prior approaches in terms of synchronization accuracy and naturalness of generated gestures.
☆ From Panel to Pixel: Zoom-In Vision -Language Pretrain ing from Biomedical Scientific Literature
Kun Yuan, Min Woo Sun, Zhen Chen, Alejandro Lozano, Xiangteng He, Shi Li, Nassir Navab, Xiaoxiao Sun, Nicolas Padoy, Serena Yeung-Levy
There is a growing interest in developing strong biomedical vision-language models. A popular approach to achieve robust representations is to use web-scale scientific data. However, current biomedical vision-language pretraining typically compresses rich scientific figures and text into coarse figure-level pairs, discarding the fine-grained correspondences that clinicians actually rely on when zooming into local structures. To tackle this issue, we introduce Panel2Patch, a novel data pipeline that mines hierarchical structure from existing biomedical scientific literature, i.e., multi-panel, marker-heavy figures and their surrounding text, and converts them into multi-granular supervision. Given scientific figures and captions, Panel2Patch parses layouts, panels, and visual markers, then constructs hierarchical aligned vision-language pairs at the figure, panel, and patch levels, preserving local semantics instead of treating each figure as a single data sample. Built on this hierarchical corpus, we develop a granularity-aware pretraining strategy that unifies heterogeneous objectives from coarse didactic descriptions to fine region-focused phrases. By applying Panel2Patch to only a small set of the literature figures, we extract far more effective supervision than prior pipelines, enabling substantially better performance with less pretraining data.
☆ OmniPerson: Unified Id entity -Preserving Pedestrian Generation
Changxiao Ma, Chao Yuan, Xincheng Shi, Yuzhuo Ma, Yongfei Zhang, Longkun Zhou, Yujia Zhang, Shangze Li, Yifan Xu
Person re-identification (ReID) suffers from a lack of large-scale high-quality training data due to challenges in data privacy and annotation costs. While previous approaches have explored pedestrian generation for data augmentation, they often fail to ensure identity consistency and suffer from insufficient controllability, thereby limiting their effectiveness in dataset augmentation. To address this, We introduce OmniPerson, the first unified identity-preserving pedestrian generation pipeline for visible/infrared image/video ReID tasks. Our contributions are threefold: 1) We proposed OmniPerson, a unified generation model, offering holistic and fine-grained control over all key pedestrian attributes. Supporting RGB/IR modality image/video generation with any number of reference images, two kinds of person poses, and text. Also including RGB-to-IR transfer and image super-resolution abilities.2) We designed Multi-Refer Fuser for robust identity preservation with any number of reference images as input, making OmniPerson could distill a unified identity from a set of multi-view reference images, ensuring our generated pedestrians achieve high-fidelity pedestrian generation.3) We introduce PersonSyn, the first large-scale dataset for multi-reference, controllable pedestrian generation, and present its automated curation pipeline which transforms public, ID-only ReID benchmarks into a richly annotated resource with the dense, multi-modal supervision required for this task. Experimental results demonstrate that OmniPerson achieves SoTA in pedestrian generation, excelling in both visual fidelity and identity consistency. Furthermore, augmenting existing datasets with our generated data consistently improves the performance of ReID models. We will open-source the full codebase, pretrained model, and the PersonSyn dataset.
☆ AVGGT: Rethinking Global Attention for Accelerating VGGT
Xianbing Sun, Zhikai Zhu, Zhengyu Lou, Bo Yang, Jinyang Tang, Liqing Zhang, He Wang, Jianfu Zhang
Since DUSt3R, models such as VGGT and $π^3$ have shown strong multi-view 3D performance, but their heavy reliance on global self-attention results in high computational cost. Existing sparse-attention variants offer partial speedups, yet lack a systematic analysis of how global attention contributes to multi-view reasoning. In this paper, we first conduct an in-depth investigation of the global attention modules in VGGT and $π^3$ to better understand their roles. Our analysis reveals a clear division of roles in the alternating global-frame architecture: early global layers do not form meaningful correspondences, middle layers perform cross-view alignment, and last layers provide only minor refinements. Guided by these findings, we propose a training-free two-step acceleration scheme: (1) converting early global layers into frame attention, and (2) subsampling global attention by subsampling K/V over patch tokens with diagonal preservation and a mean-fill component. We instantiate this strategy on VGGT and $π^3$ and evaluate across standard pose and point-map benchmarks. Our method achieves up to $8$-$10\times$ speedup in inference time while matching or slightly improving the accuracy of the original models, and remains robust even in extremely dense multi-view settings where prior sparse-attention baselines fail.
☆ WeMMU: Enhanced Bridging of Vision -Language Models and Diffusion Models via Noisy Query Tokens
Jian Yang, Dacheng Yin, Xiaoxuan He, Yong Li, Fengyun Rao, Jing Lyu, Wei Zhai, Yang Cao, Zheng-Jun Zha
Recent progress in multimodal large language models (MLLMs) has highlighted the challenge of efficiently bridging pre-trained Vision-Language Models (VLMs) with Diffusion Models. While methods using a fixed number of learnable query tokens offer computational efficiency, they suffer from task generalization collapse, failing to adapt to new tasks that are distant from their pre-training tasks. To overcome this, we propose Noisy Query Tokens, which learn a distributed representation space between the VLM and Diffusion Model via end-to-end optimization, enhancing continual learning. Additionally, we introduce a VAE branch with linear projection to recover fine-grained image details. Experimental results confirm our approach mitigates generalization collapse and enables stable continual learning across diverse tasks.
☆ On the Problem of Consistent Anomalies in Zero-Shot Anomaly Detection
Tai Le-Gia
Zero-shot anomaly classification and segmentation (AC/AS) aim to detect anomalous samples and regions without any training data, a capability increasingly crucial in industrial inspection and medical imaging. This dissertation aims to investigate the core challenges of zero-shot AC/AS and presents principled solutions rooted in theory and algorithmic design. We first formalize the problem of consistent anomalies, a failure mode in which recurring similar anomalies systematically bias distance-based methods. By analyzing the statistical and geometric behavior of patch representations from pre-trained Vision Transformers, we identify two key phenomena - similarity scaling and neighbor-burnout - that describe how relationships among normal patches change with and without consistent anomalies in settings characterized by highly similar objects. We then introduce CoDeGraph, a graph-based framework for filtering consistent anomalies built on the similarity scaling and neighbor-burnout phenomena. Through multi-stage graph construction, community detection, and structured refinement, CoDeGraph effectively suppresses the influence of consistent anomalies. Next, we extend this framework to 3D medical imaging by proposing a training-free, computationally efficient volumetric tokenization strategy for MRI data. This enables a genuinely zero-shot 3D anomaly detection pipeline and shows that volumetric anomaly segmentation is achievable without any 3D training samples. Finally, we bridge batch-based and text-based zero-shot methods by demonstrating that CoDeGraph-derived pseudo-masks can supervise prompt-driven vision-language models. Together, this dissertation provides theoretical understanding and practical solutions for the zero-shot AC/AS problem.
comment : PhD Dissertation
☆ SkyMoE: A Vision -Language Foundation Model for Enhancing Geospatial Interpretation with Mixture of Experts
Jiaqi Liu, Ronghao Fu, Lang Sun, Haoran Liu, Xiao Yang, Weipeng Zhang, Xu Na, Zhuoran Duan, Bo Yang
The emergence of large vision-language models (VLMs) has significantly enhanced the efficiency and flexibility of geospatial interpretation. However, general-purpose VLMs remain suboptimal for remote sensing (RS) tasks. Existing geospatial VLMs typically adopt a unified modeling strategy and struggle to differentiate between task types and interpretation granularities, limiting their ability to balance local detail perception and global contextual understanding. In this paper, we present SkyMoE, a Mixture-of-Experts (MoE) vision-language model tailored for multimodal, multi-task RS interpretation. SkyMoE employs an adaptive router that generates task- and granularity-aware routing instructions, enabling specialized large language model experts to handle diverse sub-tasks. To further promote expert decoupling and granularity sensitivity, we introduce a context-disentangled augmentation strategy that creates contrastive pairs between local and global features, guiding experts toward level-specific representation learning. We also construct MGRS-Bench, a comprehensive benchmark covering multiple RS interpretation tasks and granularity levels, to evaluate generalization in complex scenarios. Extensive experiments on 21 public datasets demonstrate that SkyMoE achieves state-of-the-art performance across tasks, validating its adaptability, scalability, and superior multi-granularity understanding in remote sensing.
☆ Two-Stage Vision Transformer for Image Restoration: Colorization Pretrain ing + Residual Upsampling
Aditya Chaudhary, Prachet Dev Singh, Ankit Jha
In computer vision, Single Image Super-Resolution (SISR) is still a difficult problem. We present ViT-SR, a new technique to improve the performance of a Vision Transformer (ViT) employing a two-stage training strategy. In our method, the model learns rich, generalizable visual representations from the data itself through a self-supervised pretraining phase on a colourization task. The pre-trained model is then adjusted for 4x super-resolution. By predicting the addition of a high-frequency residual image to an initial bicubic interpolation, this design simplifies residual learning. ViT-SR, trained and evaluated on the DIV2K benchmark dataset, achieves an impressive SSIM of 0.712 and PSNR of 22.90 dB. These results demonstrate the efficacy of our two-stage approach and highlight the potential of self-supervised pre-training for complex image restoration tasks. Further improvements may be possible with larger ViT architectures or alternative pretext tasks.
comment : Accepted at the 13th Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP 2025), IIT Mandi, India. 3 pages, 1 figure
☆ GeoDiT: A Diffusion -based Vision -Language Model for Geospatial Understanding
Jiaqi Liu, Ronghao Fu, Haoran Liu, Lang Sun, Bo Yang
Autoregressive models are structurally misaligned with the inherently parallel nature of geospatial understanding, forcing a rigid sequential narrative onto scenes and fundamentally hindering the generation of structured and coherent outputs. We challenge this paradigm by reframing geospatial generation as a parallel refinement process, enabling a holistic, coarse-to-fine synthesis that resolves all semantic elements simultaneously. To operationalize this, we introduce GeoDiT, the first diffusion-based vision-language model tailored for the geospatial domain. Extensive experiments demonstrate that GeoDiT establishes a new state-of-the-art on benchmarks requiring structured, object-centric outputs. It achieves significant gains in image captioning, visual grounding, and multi-object detection, precisely the tasks where autoregressive models falter. Our work validates that aligning the generative process with the data's intrinsic structure is key to unlocking superior performance in complex geospatial analysis.
☆ dots. ocr : Multilingual Document Layout Parsing in a Single Vision -Language Model
Yumeng Li, Guang Yang, Hao Liu, Bowen Wang, Colin Zhang
Document Layout Parsing serves as a critical gateway for Artificial Intelligence (AI) to access and interpret the world's vast stores of structured knowledge. This process,which encompasses layout detection, text recognition, and relational understanding, is particularly crucial for empowering next-generation Vision-Language Models. Current methods, however, rely on fragmented, multi-stage pipelines that suffer from error propagation and fail to leverage the synergies of joint training. In this paper, we introduce dots.ocr, a single Vision-Language Model that, for the first time, demonstrates the advantages of jointly learning three core tasks within a unified, end-to-end framework. This is made possible by a highly scalable data engine that synthesizes a vast multilingual corpus, empowering the model to deliver robust performance across a wide array of tasks, encompassing diverse languages, layouts, and domains. The efficacy of our unified paradigm is validated by state-of-the-art performance on the comprehensive OmniDocBench. Furthermore, to catalyze research in global document intelligence, we introduce XDocParse, a challenging new benchmark spanning 126 languages. On this testbed, dots.ocr establishes a powerful new baseline, outperforming the next-best competitor by a remarkable +7.4 point margin and proving its unparalleled multilingual capabilities.
☆ A Large Scale Benchmark for Test Time Adaptation Methods in Medical Image Segmentation
Wenjing Yu, Shuo Jiang, Yifei Chen, Shuo Chang, Yuanhan Wang, Beining Wu, Jie Dong, Mingxuan Liu, Shenghao Zhu, Feiwei Qin, Changmiao Wang, Qiyuan Tian
Test time Adaptation is a promising approach for mitigating domain shift in medical image segmentation; however, current evaluations remain limited in terms of modality coverage, task diversity, and methodological consistency. We present MedSeg-TTA, a comprehensive benchmark that examines twenty representative adaptation methods across seven imaging modalities, including MRI, CT, ultrasound, pathology, dermoscopy, OCT, and chest X-ray, under fully unified data preprocessing, backbone configuration, and test time protocols. The benchmark encompasses four significant adaptation paradigms: Input-level Transformation, Feature-level Alignment, Output-level Regularization, and Prior Estimation, enabling the first systematic cross-modality comparison of their reliability and applicability. The results show that no single paradigm performs best in all conditions. Input-level methods are more stable under mild appearance shifts. Feature-level and Output-level methods offer greater advantages in boundary-related metrics, whereas prior-based methods exhibit strong modality dependence. Several methods degrade significantly under large inter-center and inter-device shifts, which highlights the importance of principled method selection for clinical deployment. MedSeg-TTA provides standardized datasets, validated implementations, and a public leaderboard, establishing a rigorous foundation for future research on robust, clinically reliable test-time adaptation. All source codes and open-source datasets are available at https://github.com/wenjing-gg/MedSeg-TTA.
comment : 45 pages, 18 figures
☆ Attention-guided reference point shifting for Gaussian-mixture-based partial point set registration
Mizuki Kikkawa, Tatsuya Yatagawa, Yutaka Ohtake, Hiromasa Suzuki
This study investigates the impact of the invariance of feature vectors for partial-to-partial point set registration under translation and rotation of input point sets, particularly in the realm of techniques based on deep learning and Gaussian mixture models (GMMs). We reveal both theoretical and practical problems associated with such deep-learning-based registration methods using GMMs, with a particular focus on the limitations of DeepGMR, a pioneering study in this line, to the partial-to-partial point set registration. Our primary goal is to uncover the causes behind such methods and propose a comprehensible solution for that. To address this, we introduce an attention-based reference point shifting (ARPS) layer, which robustly identifies a common reference point of two partial point sets, thereby acquiring transformation-invariant features. The ARPS layer employs a well-studied attention module to find a common reference point rather than the overlap region. Owing to this, it significantly enhances the performance of DeepGMR and its recent variant, UGMMReg. Furthermore, these extension models outperform even prior deep learning methods using attention blocks and Transformer to extract the overlap region or common reference points. We believe these findings provide deeper insights into registration methods using deep learning and GMMs.
comment : 16 pages, 9 figures, 7 tables
☆ YingVideo-MV: Music-Driven Multi-Stage Video Generation
Jiahui Chen, Weida Wang, Runhua Shi, Huan Yang, Chaofan Ding, Zihao Chen
While diffusion model for audio-driven avatar video generation have achieved notable process in synthesizing long sequences with natural audio-visual synchronization and identity consistency, the generation of music-performance videos with camera motions remains largely unexplored. We present YingVideo-MV, the first cascaded framework for music-driven long-video generation. Our approach integrates audio semantic analysis, an interpretable shot planning module (MV-Director), temporal-aware diffusion Transformer architectures, and long-sequence consistency modeling to enable automatic synthesis of high-quality music performance videos from audio signals. We construct a large-scale Music-in-the-Wild Dataset by collecting web data to support the achievement of diverse, high-quality results. Observing that existing long-video generation methods lack explicit camera motion control, we introduce a camera adapter module that embeds camera poses into latent noise. To enhance continulity between clips during long-sequence inference, we further propose a time-aware dynamic window range strategy that adaptively adjust denoising ranges based on audio embedding. Comprehensive benchmark tests demonstrate that YingVideo-MV achieves outstanding performance in generating coherent and expressive music videos, and enables precise music-motion-camera synchronization. More videos are available in our project page: https://giantailab.github.io/YingVideo-MV/ .
comment : 18 pages, 6 figures
☆ Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs for 3D Scene-Language Understanding
Yerim Jeon, Miso Lee, WonJun Moon, Jae-Pil Heo
Recent advances in 3D scene-language understanding have leveraged Large Language Models (LLMs) for 3D reasoning by transferring their general reasoning ability to 3D multi-modal contexts. However, existing methods typically adopt standard decoders from language modeling, which rely on a causal attention mask. This design introduces two fundamental conflicts in 3D scene understanding: sequential bias among order-agnostic 3D objects and restricted object-instruction attention, hindering task-specific reasoning. To overcome these limitations, we propose 3D Spatial Language Instruction Mask (3D-SLIM), an effective masking strategy that replaces the causal mask with an adaptive attention mask tailored to the spatial structure of 3D scenes. Our 3D-SLIM introduces two key components: a Geometry-adaptive Mask that constrains attention based on spatial density rather than token order, and an Instruction-aware Mask that enables object tokens to directly access instruction context. This design allows the model to process objects based on their spatial relationships while being guided by the user's task. 3D-SLIM is simple, requires no architectural modifications, and adds no extra parameters, yet it yields substantial performance improvements across diverse 3D scene-language tasks. Extensive experiments across multiple benchmarks and LLM baselines validate its effectiveness and underscore the critical role of decoder design in 3D multi-modal reasoning.
☆ UC Agent s: Unidirectional Convergence for Visual Evidence Anchored Multi- Agent Medical Decision-Making
Qianhan Feng, Zhongzhen Huang, Yakun Zhu, Xiaofan Zhang, Qi Dou
Vision-Language Models (VLMs) show promise in medical diagnosis, yet suffer from reasoning detachment, where linguistically fluent explanations drift from verifiable image evidence, undermining clinical trust. Recent multi-agent frameworks simulate Multidisciplinary Team (MDT) debates to mitigate single-model bias, but open-ended discussions amplify textual noise and computational cost while failing to anchor reasoning to visual evidence, the cornerstone of medical decision-making. We propose UCAgents, a hierarchical multi-agent framework enforcing unidirectional convergence through structured evidence auditing. Inspired by clinical workflows, UCAgents forbids position changes and limits agent interactions to targeted evidence verification, suppressing rhetorical drift while amplifying visual signal extraction. In UCAgents, a one-round inquiry discussion is introduced to uncover potential risks of visual-textual misalignment. This design jointly constrains visual ambiguity and textual noise, a dual-noise bottleneck that we formalize via information theory. Extensive experiments on four medical VQA benchmarks show UCAgents achieves superior accuracy (71.3% on PathVQA, +6.0% over state-of-the-art) with 87.7% lower token cost, the evaluation results further confirm that UCAgents strikes a balance between uncovering more visual evidence and avoiding confusing textual interference. These results demonstrate that UCAgents exhibits both diagnostic reliability and computational efficiency critical for real-world clinical deployment. Code is available at https://github.com/fqhank/UCAgents.
☆ G-SHARP: Gaussian Surgical Hardware Accelerated Real-time Pipeline
Vishwesh Nath, Javier G. Tejero, Ruilong Li, Filippo Filicori, Mahdi Azizian, Sean D. Huver
We propose G-SHARP, a commercially compatible, real-time surgical scene reconstruction framework designed for minimally invasive procedures that require fast and accurate 3D modeling of deformable tissue. While recent Gaussian splatting approaches have advanced real-time endoscopic reconstruction, existing implementations often depend on non-commercial derivatives, limiting deployability. G-SHARP overcomes these constraints by being the first surgical pipeline built natively on the GSplat (Apache-2.0) differentiable Gaussian rasterizer, enabling principled deformation modeling, robust occlusion handling, and high-fidelity reconstructions on the EndoNeRF pulling benchmark. Our results demonstrate state-of-the-art reconstruction quality with strong speed-accuracy trade-offs suitable for intra-operative use. Finally, we provide a Holoscan SDK application that deploys G-SHARP on NVIDIA IGX Orin and Thor edge hardware, enabling real-time surgical visualization in practical operating-room settings.
☆ Wo rl dPack: Compressed Memory Improves Spatial Consistency in Video Wo rl d Modeling
Yuta Oshima, Yusuke Iwasawa, Masahiro Suzuki, Yutaka Matsuo, Hiroki Furuta
Video world models have attracted significant attention for their ability to produce high-fidelity future visual observations conditioned on past observations and navigation actions. Temporally- and spatially-consistent, long-term world modeling has been a long-standing problem, unresolved with even recent state-of-the-art models, due to the prohibitively expensive computational costs for long-context inputs. In this paper, we propose WorldPack, a video world model with efficient compressed memory, which significantly improves spatial consistency, fidelity, and quality in long-term generation despite much shorter context length. Our compressed memory consists of trajectory packing and memory retrieval; trajectory packing realizes high context efficiency, and memory retrieval maintains the consistency in rollouts and helps long-term generations that require spatial reasoning. Our performance is evaluated with LoopNav, a benchmark on Minecraft, specialized for the evaluation of long-term consistency, and we verify that WorldPack notably outperforms strong state-of-the-art models.
☆ TGDD: Trajectory Guided Dataset Distillation with Balanced Distribution AAAI 2026
Fengli Ran, Xiao Pu, Bo Liu, Xiuli Bi, Bin Xiao
Dataset distillation compresses large datasets into compact synthetic ones to reduce storage and computational costs. Among various approaches, distribution matching (DM)-based methods have attracted attention for their high efficiency. However, they often overlook the evolution of feature representations during training, which limits the expressiveness of synthetic data and weakens downstream performance. To address this issue, we propose Trajectory Guided Dataset Distillation (TGDD), which reformulates distribution matching as a dynamic alignment process along the model's training trajectory. At each training stage, TGDD captures evolving semantics by aligning the feature distribution between the synthetic and original dataset. Meanwhile, it introduces a distribution constraint regularization to reduce class overlap. This design helps synthetic data preserve both semantic diversity and representativeness, improving performance in downstream tasks. Without additional optimization overhead, TGDD achieves a favorable balance between performance and efficiency. Experiments on ten datasets demonstrate that TGDD achieves state-of-the-art performance, notably a 5.0% accuracy gain on high-resolution benchmarks.
comment : Accepted in AAAI 2026
☆ Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Exploration
Zhongyi Cai, Yi Du, Chen Wang, Yu Kong
Existing research on indoor embodied tasks typically requires agents to actively explore unknown environments and reason about the scene to achieve a specific goal. However, when deployed in real life, agents often face sequential tasks, where each new sub-task follows the completion of the previous one, and certain sub-tasks may be infeasible, such as searching for a non-existent object. Compared with the single-task setting, the core challenge lies in reusing spatial knowledge accumulated from previous explorations to support subsequent reasoning and exploration. In this work, we investigate this underexplored yet practically significant embodied AI challenge. To evaluate this challenge, we introduce SEER-Bench, a new Sequential Embodied Exploration and Reasoning Benchmark encompassing encompassing two classic embodied tasks: Embodied Question Answering (EQA) and Embodied Multi-modal Navigation (EMN). Building on SEER-Bench, we propose 3DSPMR, a 3D SPatial Memory Reasoning approach that exploits relational, visual, and geometric cues from explored regions to augment Multi-Modal Large Language Models (MLLMs) for reasoning and exploration in sequential embodied tasks. To the best of our knowledge, this is the first work to explicitly incorporate geometric information into MLLM-based spatial understanding and reasoning. Extensive experiments verify that 3DSPMR achieves substantial performance gains on both sequential EQA and EMN tasks.
☆ Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generation
Jianzong Wu, Hao Lian, Dachao Hao, Ye Tian, Qingyu Shi, Biaolong Chen, Hao Jiang
Recent audio-video generative systems suggest that coupling modalities benefits not only audio-video synchrony but also the video modality itself. We pose a fundamental question: Does audio-video joint denoising training improve video generation, even when we only care about video quality? To study this, we introduce a parameter-efficient Audio-Video Full DiT (AVFullDiT) architecture that leverages pre-trained text-to-video (T2V) and text-to-audio (T2A) modules for joint denoising. We train (i) a T2AV model with AVFullDiT and (ii) a T2V-only counterpart under identical settings. Our results provide the first systematic evidence that audio-video joint denoising can deliver more than synchrony. We observe consistent improvements on challenging subsets featuring large and object contact motions. We hypothesize that predicting audio acts as a privileged signal, encouraging the model to internalize causal relationships between visual events and their acoustic consequences (e.g., collision $\times$ impact sound), which in turn regularizes video dynamics. Our findings suggest that cross-modal co-training is a promising approach to developing stronger, more physically grounded world models. Code and dataset will be made publicly available.
comment : Project page at https://jianzongwu.github.io/projects/does-hearing-help-seeing/
☆ See, Think, Learn: A Self-Taught Multimodal Reasoner
Sourabh Sharma, Sonam Gupta, Sadbhawna
Vision-Language Models (VLMs) have achieved remarkable progress in integrating visual perception with language understanding. However, effective multimodal reasoning requires both accurate perception and robust reasoning, and weakness in either limits the performance of VLMs. Prior efforts to enhance reasoning often depend on high-quality chain-of-thought (CoT) data, obtained via labor-intensive human annotations, costly proprietary models, or self-training methods that overlook perception. To address these limitations, we propose a simple yet effective self-training framework called See-Think-Learn (STL). At its core, STL introduces a structured reasoning template that encourages the model to see before thinking, first extracting visual attributes in textual form, then using them to guide reasoning. The framework jointly improves perception and reasoning by having the model generate and learn from its own structured rationales in a self-training loop. Furthermore, we augment the training data with negative rationales, i.e. explanations that justify why certain answer choices are incorrect, to enhance the model's ability to distinguish between correct and misleading responses. This fosters more discriminative and robust learning. Experiments across diverse domains show that STL consistently outperforms baselines trained directly only on answers or self-generated reasoning, while qualitative analysis confirms the high quality of its rationales. STL thus provides a cost-effective solution to enhance multimodal reasoning ability of VLMs.
comment : Winter Conference on Applications of Computer Vision 2026
☆ ClusterStyle: Modeling Intra-Style Diversity with Prototypical Clustering for Stylized Motion Generation
Kerui Chen, Jianrong Zhang, Ming Li, Zhonglong Zheng, Hehe Fan
Existing stylized motion generation models have shown their remarkable ability to understand specific style information from the style motion, and insert it into the content motion. However, capturing intra-style diversity, where a single style should correspond to diverse motion variations, remains a significant challenge. In this paper, we propose a clustering-based framework, ClusterStyle, to address this limitation. Instead of learning an unstructured embedding from each style motion, we leverage a set of prototypes to effectively model diverse style patterns across motions belonging to the same style category. We consider two types of style diversity: global-level diversity among style motions of the same category, and local-level diversity within the temporal dynamics of motion sequences. These components jointly shape two structured style embedding spaces, i.e., global and local, optimized via alignment with non-learnable prototype anchors. Furthermore, we augment the pretrained text-to-motion generation model with the Stylistic Modulation Adapter (SMA) to integrate the style features. Extensive experiments demonstrate that our approach outperforms existing state-of-the-art models in stylized motion generation and motion style transfer.
☆ HouseLayout3D: A Benchmark and Training-Free Baseline for 3D Layout Estimation in the Wild NeurIPS 2025
Valentin Bieri, Marie-Julie Rakotosaona, Keisuke Tateno, Francis Engelmann, Leonidas Guibas
Current 3D layout estimation models are primarily trained on synthetic datasets containing simple single room or single floor environments. As a consequence, they cannot natively handle large multi floor buildings and require scenes to be split into individual floors before processing, which removes global spatial context that is essential for reasoning about structures such as staircases that connect multiple levels. In this work, we introduce HouseLayout3D, a real world benchmark designed to support progress toward full building scale layout estimation, including multiple floors and architecturally intricate spaces. We also present MultiFloor3D, a simple training free baseline that leverages recent scene understanding methods and already outperforms existing 3D layout estimation models on both our benchmark and prior datasets, highlighting the need for further research in this direction. Data and code are available at: https://houselayout3d.github.io.
comment : NeurIPS 2025 (Datasets and Benchmarks Track) Project Page: https://houselayout3d.github.io
☆ nuScenes Revisited: Progress and Challenges in Autonomous Driving
Whye Kit Fong, Venice Erin Liong, Kok Seang Tan, Holger Caesar
Autonomous Vehicles (AV) and Advanced Driver Assistance Systems (ADAS) have been revolutionized by Deep Learning. As a data-driven approach, Deep Learning relies on vast amounts of driving data, typically labeled in great detail. As a result, datasets, alongside hardware and algorithms, are foundational building blocks for the development of AVs. In this work we revisit one of the most widely used autonomous driving datasets: the nuScenes dataset. nuScenes exemplifies key trends in AV development, being the first dataset to include radar data, to feature diverse urban driving scenes from two continents, and to be collected using a fully autonomous vehicle operating on public roads, while also promoting multi-modal sensor fusion, standardized benchmarks, and a broad range of tasks including perception, localization \& mapping, prediction and planning. We provide an unprecedented look into the creation of nuScenes, as well as its extensions nuImages and Panoptic nuScenes, summarizing many technical details that have hitherto not been revealed in academic publications. Furthermore, we trace how the influence of nuScenes impacted a large number of other datasets that were released later and how it defined numerous standards that are used by the community to this day. Finally, we present an overview of both official and unofficial tasks using the nuScenes dataset and review major methodological developments, thereby offering a comprehensive survey of the autonomous driving literature, with a particular focus on nuScenes.
comment : 18 pages, 17 figures
☆ Temporal Dynamics Enhancer for Directly Trained Spiking Object Detectors
Fan Luo, Zeyu Gao, Xinhao Luo, Kai Zhao, Yanfeng Lu
Spiking Neural Networks (SNNs), with their brain-inspired spatiotemporal dynamics and spike-driven computation, have emerged as promising energy-efficient alternatives to Artificial Neural Networks (ANNs). However, existing SNNs typically replicate inputs directly or aggregate them into frames at fixed intervals. Such strategies lead to neurons receiving nearly identical stimuli across time steps, severely limiting the model's expressive power, particularly in complex tasks like object detection. In this work, we propose the Temporal Dynamics Enhancer (TDE) to strengthen SNNs' capacity for temporal information modeling. TDE consists of two modules: a Spiking Encoder (SE) that generates diverse input stimuli across time steps, and an Attention Gating Module (AGM) that guides the SE generation based on inter-temporal dependencies. Moreover, to eliminate the high-energy multiplication operations introduced by the AGM, we propose a Spike-Driven Attention (SDA) to reduce attention-related energy consumption. Extensive experiments demonstrate that TDE can be seamlessly integrated into existing SNN-based detectors and consistently outperforms state-of-the-art methods, achieving mAP50-95 scores of 57.7% on the static PASCAL VOC dataset and 47.6% on the neuromorphic EvDET200K dataset. In terms of energy consumption, the SDA consumes only 0.240 times the energy of conventional attention modules.
☆ Basis-Oriented Low-rank Transfer for Few-Shot and Test-Time Adaptation
Junghwan Park, Woojin Cho, Junhyuk Heo, Darongsae Kwon, Kookjin Lee
Adapting large pre-trained models to unseen tasks under tight data and compute budgets remains challenging. Meta-learning approaches explicitly learn good initializations, but they require an additional meta-training phase over many tasks, incur high training cost, and can be unstable. At the same time, the number of task-specific pre-trained models continues to grow, yet the question of how to transfer them to new tasks with minimal additional training remains relatively underexplored. We propose BOLT (Basis-Oriented Low-rank Transfer), a framework that reuses existing fine-tuned models not by merging weights, but instead by extracting an orthogonal, task-informed spectral basis and adapting within that subspace. In the offline phase, BOLT collects dominant singular directions from multiple task vectors and orthogonalizes them per layer to form reusable bases. In the online phase, we freeze these bases and train only a small set of diagonal coefficients per layer for the new task, yielding a rank-controlled update with very few trainable parameters. This design provides (i) a strong, training-free initialization for unseen tasks, obtained by pooling source-task coefficients, along with a lightweight rescaling step while leveraging the shared orthogonal bases, and (ii) a parameter-efficient fine-tuning (PEFT) path that, in our experiments, achieves robust performance compared to common PEFT baselines as well as a representative meta-learned initialization. Our results show that constraining adaptation to a task-informed orthogonal subspace provides an effective alternative for unseen-task transfer.
☆ Boosting Medical Vision -Language Pretrain ing via Momentum Self-Distillation under Limited Computing Resources WACV 2026
Phuc Pham, Nhu Pham, Ngoc Quoc Ly
In medical healthcare, obtaining detailed annotations is challenging, highlighting the need for robust Vision-Language Models (VLMs). Pretrained VLMs enable fine-tuning on small datasets or zero-shot inference, achieving performance comparable to task-specific models. Contrastive learning (CL) is a key paradigm for training VLMs but inherently requires large batch sizes for effective learning, making it computationally demanding and often limited to well-resourced institutions. Moreover, with limited data in healthcare, it is important to prioritize knowledge extraction from both data and models during training to improve performance. Therefore, we focus on leveraging the momentum method combined with distillation to simultaneously address computational efficiency and knowledge exploitation. Our contributions can be summarized as follows: (1) leveraging momentum self-distillation to enhance multimodal learning, and (2) integrating momentum mechanisms with gradient accumulation to enlarge the effective batch size without increasing resource consumption. Our method attains competitive performance with state-of-the-art (SOTA) approaches in zero-shot classification, while providing a substantial boost in the few-shot adaption, achieving over 90% AUC-ROC and improving retrieval tasks by 2-3%. Importantly, our method achieves high training efficiency with a single GPU while maintaining reasonable training time. Our approach aims to advance efficient multimodal learning by reducing resource requirements while improving performance over SOTA methods. The implementation of our method is available at https://github.com/phphuc612/MSD .
comment : WACV 2026
☆ LightHCG: a Lightweight yet powerful HSIC Disentanglement based Causal Glaucoma Detection Model framework
Daeyoung Kim
As a representative optic degenerative condition, glaucoma has been a threat to millions due to its irreversibility and severe impact on human vision fields. Mainly characterized by dimmed and blurred visions, or peripheral vision loss, glaucoma is well known to occur due to damages in the optic nerve from increased intraocular pressure (IOP) or neovascularization within the retina. Traditionally, most glaucoma related works and clinical diagnosis focused on detecting these damages in the optic nerve by using patient data from perimetry tests, optic papilla inspections and tonometer-based IOP measurements. Recently, with advancements in computer vision AI models, such as VGG16 or Vision Transformers (ViT), AI-automatized glaucoma detection and optic cup segmentation based on retinal fundus images or OCT recently exhibited significant performance in aiding conventional diagnosis with high performance. However, current AI-driven glaucoma detection approaches still have significant room for improvement in terms of reliability, excessive parameter usage, possibility of spurious correlation within detection, and limitations in applications to intervention analysis or clinical simulations. Thus, this research introduced a novel causal representation driven glaucoma detection model: LightHCG, an extremely lightweight Convolutional VAE-based latent glaucoma representation model that can consider the true causality among glaucoma-related physical factors within the optic nerve region. Using HSIC-based latent space disentanglement and Graph Autoencoder based unsupervised causal representation learning, LightHCG not only exhibits higher performance in classifying glaucoma with 93~99% less weights, but also enhances the possibility of AI-driven intervention analysis, compared to existing advanced vision models such as InceptionV3, MobileNetV2 or VGG16.
☆ Wo rl dMM: Dynamic Multimodal Memory Agent for Long Video Reasoning
Woongyeong Yeo, Kangsan Kim, Jaehong Yoon, Sung Ju Hwang
Recent advances in video large language models have demonstrated strong capabilities in understanding short clips. However, scaling them to hours- or days-long videos remains highly challenging due to limited context capacity and the loss of critical visual details during abstraction. Existing memory-augmented methods mitigate this by leveraging textual summaries of video segments, yet they heavily rely on text and fail to utilize visual evidence when reasoning over complex scenes. Moreover, retrieving from fixed temporal scales further limits their flexibility in capturing events that span variable durations. To address this, we introduce WorldMM, a novel multimodal memory agent that constructs and retrieves from multiple complementary memories, encompassing both textual and visual representations. WorldMM comprises three types of memory: episodic memory indexes factual events across multiple temporal scales, semantic memory continuously updates high-level conceptual knowledge, and visual memory preserves detailed information about scenes. During inference, an adaptive retrieval agent iteratively selects the most relevant memory source and leverages multiple temporal granularities based on the query, continuing until it determines that sufficient information has been gathered. WorldMM significantly outperforms existing baselines across five long video question-answering benchmarks, achieving an average 8.4% performance gain over previous state-of-the-art methods, showing its effectiveness on long video reasoning.
comment : Project page : https://worldmm.github.io
☆ GUI Exploration Lab: Enhancing Screen Navigation in Agent s via Multi-Turn Reinforcement Learning
Haolong Yan, Yeqing Shen, Xin Huang, Jia Wang, Kaijun Tan, Zhixuan Liang, Hongxin Li, Zheng Ge, Osamu Yoshie, Si Li, Xiangyu Zhang, Daxin Jiang
With the rapid development of Large Vision Language Models, the focus of Graphical User Interface (GUI) agent tasks shifts from single-screen tasks to complex screen navigation challenges. However, real-world GUI environments, such as PC software and mobile Apps, are often complex and proprietary, making it difficult to obtain the comprehensive environment information needed for agent training and evaluation. This limitation hinders systematic investigation and benchmarking of agent navigation capabilities. To address this limitation, we introduce GUI Exploration Lab, a simulation environment engine for GUI agent navigation research that enables flexible definition and composition of screens, icons, and navigation graphs, while providing full access to environment information for comprehensive agent training and evaluation. Through extensive experiments, we find that supervised fine-tuning enables effective memorization of fundamental knowledge, serving as a crucial foundation for subsequent training. Building on this, single-turn reinforcement learning further enhances generalization to unseen scenarios. Finally, multi-turn reinforcement learning encourages the development of exploration strategies through interactive trial and error, leading to further improvements in screen navigation performance. We validate our methods on both static and interactive benchmarks, demonstrating that our findings generalize effectively to real-world scenarios. These findings demonstrate the advantages of reinforcement learning approaches in GUI navigation and offer practical guidance for building more capable and generalizable GUI agents.
comment : 26 pages
☆ Generalizing Vision -Language Models with Dedicated Prompt Guidance AAAI26
Xinyao Li, Yinjie Min, Hongbo Chen, Zhekai Du, Fengling Li, Jingjing Li
Fine-tuning large pretrained vision-language models (VLMs) has emerged as a prevalent paradigm for downstream adaptation, yet it faces a critical trade-off between domain specificity and domain generalization (DG) ability. Current methods typically fine-tune a universal model on the entire dataset, which potentially compromises the ability to generalize to unseen domains. To fill this gap, we provide a theoretical understanding of the generalization ability for VLM fine-tuning, which reveals that training multiple parameter-efficient expert models on partitioned source domains leads to better generalization than fine-tuning a universal model. Inspired by this finding, we propose a two-step domain-expert-Guided DG (GuiDG) framework. GuiDG first employs prompt tuning to obtain source domain experts, then introduces a Cross-Modal Attention module to guide the fine-tuning of the vision encoder via adaptive expert integration. To better evaluate few-shot DG, we construct ImageNet-DG from ImageNet and its variants. Extensive experiments on standard DG benchmarks and ImageNet-DG demonstrate that GuiDG improves upon state-of-the-art fine-tuning methods while maintaining efficiency.
comment : Accepted to AAAI26
☆ MitUNet: Enhancing Floor Plan Recognition using a Hybrid Mix- Transformer and U-Net Architecture
Dmitriy Parashchuk, Alexey Kapshitskiy, Yuriy Karyakin
Automatic 3D reconstruction of indoor spaces from 2D floor plans requires high-precision semantic segmentation of structural elements, particularly walls. However, existing methods optimized for standard metrics often struggle to detect thin structural components and yield masks with irregular boundaries, lacking the geometric precision required for subsequent vectorization. To address this issue, we introduce MitUNet, a hybrid neural network architecture specifically designed for wall segmentation tasks in the context of 3D modeling. In MitUNet, we utilize a hierarchical Mix-Transformer encoder to capture global context and a U-Net decoder enhanced with scSE attention blocks for precise boundary recovery. Furthermore, we propose an optimization strategy based on the Tversky loss function to effectively balance precision and recall. By fine-tuning the hyperparameters of the loss function, we prioritize the suppression of false positive noise along wall boundaries while maintaining high sensitivity to thin structures. Our experiments on the public CubiCasa5k dataset and a proprietary regional dataset demonstrate that the proposed approach ensures the generation of structurally correct masks with high boundary accuracy, outperforming standard single-task models. MitUNet provides a robust tool for data preparation in automated 3D reconstruction pipelines.
comment : 9 pages, 4 figures, 3 tables
☆ WISE: Weighted Iterative Society-of-Experts for Robust Multimodal Multi- Agent Debate
Anoop Cherian, River Doyle, Eyal Ben-Dov, Suhas Lohit, Kuan-Chuan Peng
Recent large language models (LLMs) are trained on diverse corpora and tasks, leading them to develop complementary strengths. Multi-agent debate (MAD) has emerged as a popular way to leverage these strengths for robust reasoning, though it has mostly been applied to language-only tasks, leaving its efficacy on multimodal problems underexplored. In this paper, we study MAD for solving vision-and-language reasoning problems. Our setup enables generalizing the debate protocol with heterogeneous experts that possess single- and multi-modal capabilities. To this end, we present Weighted Iterative Society-of-Experts (WISE), a generalized and modular MAD framework that partitions the agents into Solvers, that generate solutions, and Reflectors, that verify correctness, assign weights, and provide natural language feedback. To aggregate the agents' solutions across debate rounds, while accounting for variance in their responses and the feedback weights, we present a modified Dawid-Skene algorithm for post-processing that integrates our two-stage debate model. We evaluate WISE on SMART-840, VisualPuzzles, EvoChart-QA, and a new SMART-840++ dataset with programmatically generated problem instances of controlled difficulty. Our results show that WISE consistently improves accuracy by 2-7% over the state-of-the-art MAD setups and aggregation methods across diverse multimodal tasks and LLM configurations.
☆ Nav-$R^2$ Dual- Relation Reasoning for Generalizable Open-Vocabulary Object-Goal Navigation
Wentao Xiang, Haokang Zhang, Tianhang Yang, Zedong Chu, Ruihang Chu, Shichao Xie, Yujian Yuan, Jian Sun, Zhining Gu, Junjie Wang, Xiaolong Wu, Mu Xu, Yujiu Yang
Object-goal navigation in open-vocabulary settings requires agents to locate novel objects in unseen environments, yet existing approaches suffer from opaque decision-making processes and low success rate on locating unseen objects. To address these challenges, we propose Nav-$R^2$, a framework that explicitly models two critical types of relationships, target-environment modeling and environment-action planning, through structured Chain-of-Thought (CoT) reasoning coupled with a Similarity-Aware Memory. We construct a Nav$R^2$-CoT dataset that teaches the model to perceive the environment, focus on target-related objects in the surrounding context and finally make future action plans. Our SA-Mem preserves the most target-relevant and current observation-relevant features from both temporal and semantic perspectives by compressing video frames and fusing historical observations, while introducing no additional parameters. Compared to previous methods, Nav-R^2 achieves state-of-the-art performance in localizing unseen objects through a streamlined and efficient pipeline, avoiding overfitting to seen object categories while maintaining real-time inference at 2Hz. Resources will be made publicly available at \href{https://github.com/AMAP-EAI/Nav-R2}{github link}.
☆ Skywork-R1V4: Toward Agent ic Multimodal Intelligence through Inte rl eaved Thinking with Images and DeepResearch
Yifan Zhang, Liang Hu, Haofeng Sun, Peiyu Wang, Yichen Wei, Shukang Yin, Jiangbo Pei, Wei Shen, Peng Xia, Yi Peng, Tianyidan Xie, Eric Li, Yang Liu, Xuchen Song, Yahui Zhou
Despite recent progress in multimodal agentic systems, existing approaches often treat image manipulation and web search as disjoint capabilities, rely heavily on costly reinforcement learning, and lack planning grounded in real tool-execution traces. To address these limitations, we present Skywork-R1V4, a 30B (A3B) parameter multimodal agentic model that unifies multimodal planning, active image manipulation ("thinking with images"), deep multimodal search, and, most critically, interleaved reasoning that dynamically alternates between visual operations and external knowledge retrieval. Trained solely via supervised fine-tuning on fewer than 30,000 high-quality, planning-execution-consistent trajectories and validated through stepwise consistency filtering, Skywork-R1V4 achieves state-of-the-art results across perception and multimodal search benchmarks: it scores 66.1 on MMSearch and 67.2 on FVQA, surpassing Gemini 2.5 Flash on all 11 metrics. Skywork-R1V4 exhibits emergent long-horizon reasoning at inference time, successfully orchestrating more than 10 tool calls to solve complex, multi-step tasks. Our results demonstrate that sophisticated agentic multimodal intelligence can be achieved through carefully curated supervised learning alone, without any reliance on reinforcement learning.
comment : 21 pages, 7 figures
☆ Reproducing and Extending RaDelft 4D Radar with Camera-Assisted Labels
Kejia Hu, Mohammed Alsakabi, John M. Dolan, Ozan K. Tonguz
Recent advances in 4D radar highlight its potential for robust environment perception under adverse conditions, yet progress in radar semantic segmentation remains constrained by the scarcity of open source datasets and labels. The RaDelft data set, although seminal, provides only LiDAR annotations and no public code to generate radar labels, limiting reproducibility and downstream research. In this work, we reproduce the numerical results of the RaDelft group and demonstrate that a camera-guided radar labeling pipeline can generate accurate labels for radar point clouds without relying on human annotations. By projecting radar point clouds into camera-based semantic segmentation and applying spatial clustering, we create labels that significantly enhance the accuracy of radar labels. These results establish a reproducible framework that allows the research community to train and evaluate the labeled 4D radar data. In addition, we study and quantify how different fog levels affect the radar labeling performance.
☆ From Detection to Association: Learning Discriminative Object Embeddings for Multi-Object Tracking
Yuqing Shao, Yuchen Yang, Rui Yu, Weilong Li, Xu Guo, Huaicheng Yan, Wei Wang, Xiao Sun
End-to-end multi-object tracking (MOT) methods have recently achieved remarkable progress by unifying detection and association within a single framework. Despite their strong detection performance, these methods suffer from relatively low association accuracy. Through detailed analysis, we observe that object embeddings produced by the shared DETR architecture display excessively high inter-object similarity, as it emphasizes only category-level discrimination within single frames. In contrast, tracking requires instance-level distinction across frames with spatial and temporal continuity, for which current end-to-end approaches insufficiently optimize object embeddings. To address this, we introduce FDTA (From Detection to Association), an explicit feature refinement framework that enhances object discriminativeness across three complementary perspectives. Specifically, we introduce a Spatial Adapter (SA) to integrate depth-aware cues for spatial continuity, a Temporal Adapter (TA) to aggregate historical information for temporal dependencies, and an Identity Adapter (IA) to leverage quality-aware contrastive learning for instance-level separability. Extensive experiments demonstrate that FDTA achieves state-of-the-art performance on multiple challenging MOT benchmarks, including DanceTrack, SportsMOT, and BFT, highlighting the effectiveness of our proposed discriminative embedding enhancement strategy. The code is available at https://github.com/Spongebobbbbbbbb/FDTA.
☆ On-the-fly Feedback SfM: Online Explore-and-Exploit UAV Photogrammetry with Incremental Mesh Quality-Aware Indicator and Predictive Path Planning
Liyuan Lou, Wanyun Li, Wentian Gan, Yifei Yu, Tengfei Wang, Xin Wang, Zongqian Zhan
Compared with conventional offline UAV photogrammetry, real-time UAV photogrammetry is essential for time-critical geospatial applications such as disaster response and active digital-twin maintenance. However, most existing methods focus on processing captured images or sequential frames in real time, without explicitly evaluating the quality of the on-the-go 3D reconstruction or providing guided feedback to enhance image acquisition in the target area. This work presents On-the-fly Feedback SfM, an explore-and-exploit framework for real-time UAV photogrammetry, enabling iterative exploration of unseen regions and exploitation of already observed and reconstructed areas in near real time. Built upon SfM on-the-fly , the proposed method integrates three modules: (1) online incremental coarse-mesh generation for dynamically expanding sparse 3D point cloud; (2) online mesh quality assessment with actionable indicators; and (3) predictive path planning for on-the-fly trajectory refinement. Comprehensive experiments demonstrate that our method achieves in-situ reconstruction and evaluation in near real time while providing actionable feedback that markedly reduces coverage gaps and re-flight costs. Via the integration of data collection, processing, 3D reconstruction and assessment, and online feedback, our on the-fly feedback SfM could be an alternative for the transition from traditional passive working mode to a more intelligent and adaptive exploration workflow. Code is now available at https://github.com/IRIS-LAB-whu/OntheflySfMFeedback.
comment : This work was submitted to IEEE GRSM Journal for consideration.COPYRIGHT would be transferred once it get accepted
☆ SAGE: Style-Adaptive Generalization for Privacy-Constrained Semantic Segmentation Across Domains
Qingmei Li, Yang Zhang, Peifeng Zhang, Haohuan Fu, Juepeng Zheng
Domain generalization for semantic segmentation aims to mitigate the degradation in model performance caused by domain shifts. However, in many real-world scenarios, we are unable to access the model parameters and architectural details due to privacy concerns and security constraints. Traditional fine-tuning or adaptation is hindered, leading to the demand for input-level strategies that can enhance generalization without modifying model weights. To this end, we propose a \textbf{S}tyle-\textbf{A}daptive \textbf{GE}neralization framework (\textbf{SAGE}), which improves the generalization of frozen models under privacy constraints. SAGE learns to synthesize visual prompts that implicitly align feature distributions across styles instead of directly fine-tuning the backbone. Specifically, we first utilize style transfer to construct a diverse style representation of the source domain, thereby learning a set of style characteristics that can cover a wide range of visual features. Then, the model adaptively fuses these style cues according to the visual context of each input, forming a dynamic prompt that harmonizes the image appearance without touching the interior of the model. Through this closed-loop design, SAGE effectively bridges the gap between frozen model invariance and the diversity of unseen domains. Extensive experiments on five benchmark datasets demonstrate that SAGE achieves competitive or superior performance compared to state-of-the-art methods under privacy constraints and outperforms full fine-tuning baselines in all settings.
☆ Multi-Domain Enhanced Map-Free Trajectory Prediction with Selective Attention
Wenyi Xiong, Jian Chen
Trajectory prediction is crucial for the reliability and safety of autonomous driving systems, yet it remains a challenging task in complex interactive scenarios. Existing methods often struggle to efficiently extract valuable scene information from redundant data, thereby reducing computational efficiency and prediction accuracy, especially when dealing with intricate agent interactions. To address these challenges, we propose a novel map-free trajectory prediction algorithm that achieves trajectory prediction across the temporal, spatial, and frequency domains. Specifically, in temporal information processing, We utilize a Mixture of Experts (MoE) mechanism to adaptively select critical frequency components. Concurrently, we extract these components and integrate multi-scale temporal features. Subsequently, a selective attention module is proposed to filter out redundant information in both temporal sequences and spatial interactions. Finally, we design a multimodal decoder. Under the supervision of patch-level and point-level losses, we obtain reasonable trajectory results. Experiments on Nuscences datasets demonstrate the superiority of our algorithm, validating its effectiveness in handling complex interactive scenarios.
☆ Tackling Tuberculosis: A Comparative Dive into Machine Learning for Tuberculosis Detection
Daanish Hindustani, Sanober Hindustani, Preston Nguyen
This study explores the application of machine learning models, specifically a pretrained ResNet-50 model and a general SqueezeNet model, in diagnosing tuberculosis (TB) using chest X-ray images. TB, a persistent infectious disease affecting humanity for millennia, poses challenges in diagnosis, especially in resource-limited settings. Traditional methods, such as sputum smear microscopy and culture, are inefficient, prompting the exploration of advanced technologies like deep learning and computer vision. The study utilized a dataset from Kaggle, consisting of 4,200 chest X-rays, to develop and compare the performance of the two machine learning models. Preprocessing involved data splitting, augmentation, and resizing to enhance training efficiency. Evaluation metrics, including accuracy, precision, recall, and confusion matrix, were employed to assess model performance. Results showcase that the SqueezeNet achieved a loss of 32%, accuracy of 89%, precision of 98%, recall of 80%, and an F1 score of 87%. In contrast, the ResNet-50 model exhibited a loss of 54%, accuracy of 73%, precision of 88%, recall of 52%, and an F1 score of 65%. This study emphasizes the potential of machine learning in TB detection and possible implications for early identification and treatment initiation. The possibility of integrating such models into mobile devices expands their utility in areas lacking TB detection resources. However, despite promising results, the need for continued development of faster, smaller, and more accurate TB detection models remains crucial in contributing to the global efforts in combating TB.
☆ VA CoT : Rethinking Visual Data Augmentation with VLMs
Zhengzhuo Xu, Chong Sun, SiNan Du, Chen Li, Jing Lyu, Chun Yuan
While visual data augmentation remains a cornerstone for training robust vision models, it has received limited attention in visual language models (VLMs), which predominantly rely on large-scale real data acquisition or synthetic diversity. Consequently, they may struggle with basic perception tasks that conventional models handle reliably. Given the substantial cost of pre-training and fine-tuning VLMs, continue training on augmented data yields limited and diminishing returns. In this paper, we present Visual Augmentation Chain-of-Thought (VACoT), a framework that dynamically invokes image augmentations during model inference. By incorporating post-hoc transformations such as denoising, VACoT substantially improves robustness on challenging and out-of-distribution inputs, especially in OCR-related adversarial scenarios. Distinct from prior approaches limited to local cropping, VACoT integrates a structured collection of general visual augmentations, broadening the query image views while reducing training complexity and computational overhead with efficient agentic reinforcement learning. We propose a conditional reward scheme that encourages necessary augmentation while penalizing verbose responses, ensuring concise and effective reasoning in perception tasks. We demonstrate the superiority of VACoT with extensive experiments on 13 perception benchmarks and further introduce AdvOCR to highlight the generalization benefits of post-hoc visual augmentations in adversarial scenarios.
☆ WSCF-MVCC: Weakly-supervised Calibration-free Multi-view Crowd Counting
Bin Li, Daijie Chen, Qi Zhang
Multi-view crowd counting can effectively mitigate occlusion issues that commonly arise in single-image crowd counting. Existing deep-learning multi-view crowd counting methods project different camera view images onto a common space to obtain ground-plane density maps, requiring abundant and costly crowd annotations and camera calibrations. Hence, calibration-free methods are proposed that do not require camera calibrations and scene-level crowd annotations. However, existing calibration-free methods still require expensive image-level crowd annotations for training the single-view counting module. Thus, in this paper, we propose a weakly-supervised calibration-free multi-view crowd counting method (WSCF-MVCC), directly using crowd count as supervision for the single-view counting module rather than density maps constructed from crowd annotations. Instead, a self-supervised ranking loss that leverages multi-scale priors is utilized to enhance the model's perceptual ability without additional annotation costs. What's more, the proposed model leverages semantic information to achieve a more accurate view matching and, consequently, a more precise scene-level crowd count estimation. The proposed method outperforms the state-of-the-art methods on three widely used multi-view counting datasets under weakly supervised settings, indicating that it is more suitable for practical deployment compared with calibrated methods. Code is released in https://github.com/zqyq/Weakly-MVCC.
comment : PRCV 2025
☆ Understanding and Harnessing Sparsity in Unified Multimodal Models
Shwai He, Chaorui Deng, Ang Li, Shen Yan
Large multimodal models have achieved remarkable progress in both understanding and generation. Recent efforts pursue unified multimodal models that integrate heterogeneous components to support both capabilities within a single framework. However, such unification introduces inference inefficiencies, e.g., specific tasks or samples may not require the full knowledge or capacity of the unified model. Yet, a systematic understanding of how these inefficiencies manifest across different components remains limited. In this work, we first conduct a systematic analysis of unified multimodal model components using training-free pruning as a probing methodology, considering both depth pruning and width reduction. Our study reveals that the understanding component exhibits notable compressibility in both understanding and generation tasks, which is more pronounced in the latter. In contrast, the generation components are highly sensitive to compression, with performance deteriorating sharply even under moderate compression ratios. To address this limitation, we propose the Mixture-of-Experts (MoE) Adaptation, inspired by the dynamic activation patterns observed across different samples. This approach partitions the generation module into multiple experts and enables sparse activation to restore generation quality. We validate the effectiveness of sparse activation through expert-frozen tuning and further demonstrate that a fully trainable adaptation delivers additional gains. As a result, the adapted BAGEL model achieves performance comparable to the full model while activating only about half of its parameters. The code is released at \href{https://github.com/Shwai-He/SparseUnifiedModel}{this link}.
comment : 13 pages, 13 figures, 8 tables
☆ A multi-weight self-matching visual explanation for cnns on sar images
Siyuan Sun, Yongping Zhang, Hongcheng Zeng, Yamin Wang, Wei Yang, Wanting Yang, Jie Chen
In recent years, convolutional neural networks (CNNs) have achieved significant success in various synthetic aperture radar (SAR) tasks. However, the complexity and opacity of their internal mechanisms hinder the fulfillment of high-reliability requirements, thereby limiting their application in SAR. Improving the interpretability of CNNs is thus of great importance for their development and deployment in SAR. In this paper, a visual explanation method termed multi-weight self-matching class activation mapping (MS-CAM) is proposed. MS-CAM matches SAR images with the feature maps and corresponding gradients extracted by the CNN, and combines both channel-wise and element-wise weights to visualize the decision basis learned by the model in SAR images. Extensive experiments conducted on a self-constructed SAR target classification dataset demonstrate that MS-CAM more accurately highlights the network's regions of interest and captures detailed target feature information, thereby enhancing network interpretability. Furthermore, the feasibility of applying MS-CAM to weakly-supervised obiect localization is validated. Key factors affecting localization accuracy, such as pixel thresholds, are analyzed in depth to inform future work.
☆ TALO: Pushing 3D Vision Foundation Models Towards Globally Consistent Online Reconstruction
Fengyi Zhang, Tianjun Zhang, Kasra Khosoussi, Zheng Zhang, Zi Huang, Yadan Luo
3D vision foundation models have shown strong generalization in reconstructing key 3D attributes from uncalibrated images through a single feed-forward pass. However, when deployed in online settings such as driving scenarios, predictions are made over temporal windows, making it non-trivial to maintain consistency across time. Recent strategies align consecutive predictions by solving global transformation, yet our analysis reveals their fundamental limitations in assumption validity, local alignment scope, and robustness under noisy geometry. In this work, we propose a higher-DOF and long-term alignment framework based on Thin Plate Spline, leveraging globally propagated control points to correct spatially varying inconsistencies. In addition, we adopt a point-agnostic submap registration design that is inherently robust to noisy geometry predictions. The proposed framework is fully plug-and-play, compatible with diverse 3D foundation models and camera configurations (e.g., monocular or surround-view). Extensive experiments demonstrate that our method consistently yields more coherent geometry and lower trajectory errors across multiple datasets, backbone models, and camera setups, highlighting its robustness and generality. Codes are publicly available at \href{https://github.com/Xian-Bei/TALO}{https://github.com/Xian-Bei/TALO}.
☆ Reasoning Path and Latent State Analysis for Multi-view Visual Spatial Reasoning : A Cognitive Science Perspective
Qiyao Xue, Weichen Liu, Shiqi Wang, Haoming Wang, Yuyang Wu, Wei Gao
Spatial reasoning is a core aspect of human intelligence that allows perception, inference and planning in 3D environments. However, current vision-language models (VLMs) struggle to maintain geometric coherence and cross-view consistency for spatial reasoning in multi-view settings. We attribute this gap to the lack of fine-grained benchmarks that isolate multi-view reasoning from single-view perception and temporal factors. To address this, we present ReMindView-Bench, a cognitively grounded benchmark for evaluating how VLMs construct, align and maintain spatial mental models across complementary viewpoints. ReMindView-Bench systematically varies viewpoint spatial pattern and query type to probe key factors of spatial cognition. Evaluations of 15 current VLMs reveals consistent failures in cross-view alignment and perspective-taking in multi-view spatial reasoning, motivating deeper analysis on the reasoning process. Explicit phase-wise analysis using LLM-as-a-judge and self-consistency prompting shows that VLMs perform well on in-frame perception but degrade sharply when integrating information across views. Implicit analysis, including linear probing and entropy dynamics, further show progressive loss of task-relevant information and uncertainty separation between correct and incorrect trajectories. These results provide a cognitively grounded diagnosis of VLM spatial reasoning and reveal how multi-view spatial mental models are formed, degraded and destabilized across reasoning phases. The ReMindView-Bench benchmark is available at https://huggingface.co/datasets/Xue0823/ReMindView-Bench, and the source codes of benchmark construction and VLM reasoning analysis are available at https://github.com/pittisl/ReMindView-Bench.
comment : 23 pages, 37 figures
☆ Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Super vision NeurIPS 2025
Chenshuang Zhang, Kang Zhang, Joon Son Chung, In So Kweon, Junmo Kim, Chengzhi Mao
Distinguishing visually similar objects by their motion remains a critical challenge in computer vision. Although supervised trackers show promise, contemporary self-supervised trackers struggle when visual cues become ambiguous, limiting their scalability and generalization without extensive labeled data. We find that pre-trained video diffusion models inherently learn motion representations suitable for tracking without task-specific training. This ability arises because their denoising process isolates motion in early, high-noise stages, distinct from later appearance refinement. Capitalizing on this discovery, our self-supervised tracker significantly improves performance in distinguishing visually similar objects, an underexplored failure point for existing methods. Our method achieves up to a 6-point improvement over recent self-supervised approaches on established benchmarks and our newly introduced tests focused on tracking visually similar items. Visualizations confirm that these diffusion-derived motion representations enable robust tracking of even identical objects across challenging viewpoint changes and deformations.
comment : Accepted at NeurIPS 2025
☆ OmniGuard: Unified Omni-Modal Guardrails with Deliberate Reasoning
Boyu Zhu, Xiaofei Wen, Wenjie Jacky Mo, Tinghui Zhu, Yanan Xie, Peng Qi, Muhao Chen
Omni-modal Large Language Models (OLLMs) that process text, images, videos, and audio introduce new challenges for safety and value guardrails in human-AI interaction. Prior guardrail research largely targets unimodal settings and typically frames safeguarding as binary classification, which limits robustness across diverse modalities and tasks. To address this gap, we propose OmniGuard, the first family of omni-modal guardrails that performs safeguarding across all modalities with deliberate reasoning ability. To support the training of OMNIGUARD, we curate a large, comprehensive omni-modal safety dataset comprising over 210K diverse samples, with inputs that cover all modalities through both unimodal and cross-modal samples. Each sample is annotated with structured safety labels and carefully curated safety critiques from expert models through targeted distillation. Extensive experiments on 15 benchmarks show that OmniGuard achieves strong effectiveness and generalization across a wide range of multimodal safety scenarios. Importantly, OmniGuard provides a unified framework that enforces policies and mitigates risks in omni-modalities, paving the way toward building more robust and capable omnimodal safeguarding systems.
☆ VIGS-SLAM: Visual Inertial Gaussian Splatting SLAM
Zihan Zhu, Wei Zhang, Norbert Haala, Marc Pollefeys, Daniel Barath
We present VIGS-SLAM, a visual-inertial 3D Gaussian Splatting SLAM system that achieves robust real-time tracking and high-fidelity reconstruction. Although recent 3DGS-based SLAM methods achieve dense and photorealistic mapping, their purely visual design degrades under motion blur, low texture, and exposure variations. Our method tightly couples visual and inertial cues within a unified optimization framework, jointly refining camera poses, depths, and IMU states. It features robust IMU initialization, time-varying bias modeling, and loop closure with consistent Gaussian updates. Experiments on four challenging datasets demonstrate our superiority over state-of-the-art methods. Project page: https://vigs-slam.github.io
comment : Project page: https://vigs-slam.github.io
☆ Enhancing Cross Domain SAR Oil Spill Segmentation via Morphological Region Perturbation and Synthetic Label-to-SAR Generation
Andre Juarez, Luis Salsavilca, Frida Coaquira, Celso Gonzales
Deep learning models for SAR oil spill segmentation often fail to generalize across regions due to differences in sea-state, backscatter statistics, and slick morphology, a limitation that is particularly severe along the Peruvian coast where labeled Sentinel-1 data remain scarce. To address this problem, we propose \textbf{MORP--Synth}, a two-stage synthetic augmentation framework designed to improve transfer from Mediterranean to Peruvian conditions. Stage~A applies Morphological Region Perturbation, a curvature guided label space method that generates realistic geometric variations of oil and look-alike regions. Stage~B renders SAR-like textures from the edited masks using a conditional generative INADE model. We compile a Peruvian dataset of 2112 labeled 512$\times$512 patches from 40 Sentinel-1 scenes (2014--2024), harmonized with the Mediterranean CleanSeaNet benchmark, and evaluate seven segmentation architectures. Models pretrained on Mediterranean data degrade from 67.8\% to 51.8\% mIoU on the Peruvian domain; MORP--Synth improves performance up to +6 mIoU and boosts minority-class IoU (+10.8 oil, +14.6 look-alike).
☆ SpatialReasoner: Active Perception for Large-Scale 3D Scene Understanding
Hongpei Zheng, Shijie Li, Yanran Li, Hujun Yin
Spatial reasoning in large-scale 3D environments remains challenging for current vision-language models, which are typically constrained to room-scale scenarios. We introduce H$^2$U3D (Holistic House Understanding in 3D), a 3D visual question answering dataset designed for house-scale scene understanding. H$^2$U3D features multi-floor environments spanning up to three floors and 10-20 rooms, covering more than 300 m$^2$. Through an automated annotation pipeline, it constructs hierarchical coarse-to-fine visual representations and generates diverse question-answer pairs with chain-of-thought annotations. We further propose SpatialReasoner, an active perception framework that autonomously invokes spatial tools to explore 3D scenes based on textual queries. SpatialReasoner is trained through a two-stage strategy: a supervised cold start followed by reinforcement learning with an adaptive exploration reward that promotes efficient exploration while discouraging redundant operations. Extensive experiments demonstrate that SpatialReasoner achieves state-of-the-art performance on H$^2$U3D, outperforming strong baselines including GPT-4o and Gemini-2.5-Pro. Notably, our method attains superior results while using only 3-4 images in total on average, compared to baselines requiring 16+ images, highlighting the effectiveness of our coarse-to-fine active exploration paradigm.
☆ PyroFocus: A Deep Learning Approach to Real-Time Wildfire Detection in Multispectral Remote Sensing Imagery
Mark Moussa, Andre Williams, Seth Roffe, Douglas Morton
Rapid and accurate wildfire detection is crucial for emergency response and environmental management. In airborne and spaceborne missions, real-time algorithms must distinguish between no fire, active fire, and post-fire conditions, and estimate fire intensity. Multispectral and hyperspectral thermal imagers provide rich spectral information, but high data dimensionality and limited onboard resources make real-time processing challenging. As wildfires increase in frequency and severity, the need for low-latency and computationally efficient onboard detection methods is critical. We present a systematic evaluation of multiple deep learning architectures, including custom Convolutional Neural Networks (CNNs) and Transformer-based models, for multi-class fire classification. We also introduce PyroFocus, a two-stage pipeline that performs fire classification followed by fire radiative power (FRP) regression or segmentation to reduce inference time and computational cost for onboard deployment. Using data from NASA's MODIS/ASTER Airborne Simulator (MASTER), which is similar to a next-generation fire detection sensor, we compare accuracy, inference latency, and resource efficiency. Experimental results show that the proposed two-stage pipeline achieves strong trade-offs between speed and accuracy, demonstrating significant potential for real-time edge deployment in future wildfire monitoring missions.
☆ PixPerfect: Seamless Latent Diffusion Local Editing with Discriminative Pixel-Space Refinement NeurIPS 2025
Haitian Zheng, Yuan Yao, Yongsheng Yu, Yuqian Zhou, Jiebo Luo, Zhe Lin
Latent Diffusion Models (LDMs) have markedly advanced the quality of image inpainting and local editing. However, the inherent latent compression often introduces pixel-level inconsistencies, such as chromatic shifts, texture mismatches, and visible seams along editing boundaries. Existing remedies, including background-conditioned latent decoding and pixel-space harmonization, usually fail to fully eliminate these artifacts in practice and do not generalize well across different latent representations or tasks. We introduce PixPerfect, a pixel-level refinement framework that delivers seamless, high-fidelity local edits across diverse LDM architectures and tasks. PixPerfect leverages (i) a differentiable discriminative pixel space that amplifies and suppresses subtle color and texture discrepancies, (ii) a comprehensive artifact simulation pipeline that exposes the refiner to realistic local editing artifacts during training, and (iii) a direct pixel-space refinement scheme that ensures broad applicability across diverse latent representations and tasks. Extensive experiments on inpainting, object removal, and insertion benchmarks demonstrate that PixPerfect substantially enhances perceptual fidelity and downstream editing performance, establishing a new standard for robust and high-fidelity localized image editing.
comment : Published in the Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS 2025)
☆ 2-Shots in the Dark: Low-Light Denoising with Minimal Data Acquisition
Liying Lu, Raphaël Achddou, Sabine Süsstrunk
Raw images taken in low-light conditions are very noisy due to low photon count and sensor noise. Learning-based denoisers have the potential to reconstruct high-quality images. For training, however, these denoisers require large paired datasets of clean and noisy images, which are difficult to collect. Noise synthesis is an alternative to large-scale data acquisition: given a clean image, we can synthesize a realistic noisy counterpart. In this work, we propose a general and practical noise synthesis method that requires only one single noisy image and one single dark frame per ISO setting. We represent signal-dependent noise with a Poisson distribution and introduce a Fourier-domain spectral sampling algorithm to accurately model signal-independent noise. The latter generates diverse noise realizations that maintain the spatial and statistical properties of real sensor noise. As opposed to competing approaches, our method neither relies on simplified parametric models nor on large sets of clean-noisy image pairs. Our synthesis method is not only accurate and practical, it also leads to state-of-the-art performances on multiple low-light denoising benchmarks.
☆ LLM-Guided Material Inference for 3D Point Clouds
Nafiseh Izadyar, Teseo Schneider
Most existing 3D shape datasets and models focus solely on geometry, overlooking the material properties that determine how objects appear. We introduce a two-stage large language model (LLM) based method for inferring material composition directly from 3D point clouds with coarse segmentations. Our key insight is to decouple reasoning about what an object is from what it is made of. In the first stage, an LLM predicts the object's semantic; in the second stage, it assigns plausible materials to each geometric segment, conditioned on the inferred semantics. Both stages operate in a zero-shot manner, without task-specific training. Because existing datasets lack reliable material annotations, we evaluate our method using an LLM-as-a-Judge implemented in DeepEval. Across 1,000 shapes from Fusion/ABS and ShapeNet, our method achieves high semantic and material plausibility. These results demonstrate that language models can serve as general-purpose priors for bridging geometric reasoning and material understanding in 3D data.
☆ Object Counting with GPT -4o and GPT -5: A Comparative Study
Richard Füzesséry, Kaziwa Saleh, Sándor Szénási, Zoltán Vámossy
Zero-shot object counting attempts to estimate the number of object instances belonging to novel categories that the vision model performing the counting has never encountered during training. Existing methods typically require large amount of annotated data and often require visual exemplars to guide the counting process. However, large language models (LLMs) are powerful tools with remarkable reasoning and data understanding abilities, which suggest the possibility of utilizing them for counting tasks without any supervision. In this work we aim to leverage the visual capabilities of two multi-modal LLMs, GPT-4o and GPT-5, to perform object counting in a zero-shot manner using only textual prompts. We evaluate both models on the FSC-147 and CARPK datasets and provide a comparative analysis. Our findings show that the models achieve performance comparable to the state-of-the-art zero-shot approaches on FSC-147, in some cases, even surpass them.
comment : 5 pages, 3 figures
☆ Kaleidoscopic Scintillation Event Imaging
Alex Bocchieri, John Mamish, David Appleyard, Andreas Velten
Scintillators are transparent materials that interact with high-energy particles and emit visible light as a result. They are used in state of the art methods of measuring high-energy particles and radiation sources. Most existing methods use fast single-pixel detectors to detect and time scintillation events. Cameras provide spatial resolution but can only capture an average over many events, making it difficult to image the events associated with an individual particle. Emerging single-photon avalanche diode cameras combine speed and spatial resolution to enable capturing images of individual events. This allows us to use machine vision techniques to analyze events, enabling new types of detectors. The main challenge is the very low brightness of the events. Techniques have to work with a very limited number of photons. We propose a kaleidoscopic scintillator to increase light collection in a single-photon camera while preserving the event's spatial information. The kaleidoscopic geometry creates mirror reflections of the event in known locations for a given event location that are captured by the camera. We introduce theory for imaging an event in a kaleidoscopic scintillator and an algorithm to estimate the event's 3D position. We find that the kaleidoscopic scintillator design provides sufficient light collection to perform high-resolution event measurements for advanced radiation imaging techniques using a commercial CMOS single-photon camera. Code and data are available at https://github.com/bocchs/kaleidoscopic_scintillator.
☆ Flux4D: Flow-based Unsupervised 4D Reconstruction NeurIPS 2025
Jingkang Wang, Henry Che, Yun Chen, Ze Yang, Lily Goli, Sivabalan Manivasagam, Raquel Urtasun
Reconstructing large-scale dynamic scenes from visual observations is a fundamental challenge in computer vision, with critical implications for robotics and autonomous systems. While recent differentiable rendering methods such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have achieved impressive photorealistic reconstruction, they suffer from scalability limitations and require annotations to decouple actor motion. Existing self-supervised methods attempt to eliminate explicit annotations by leveraging motion cues and geometric priors, yet they remain constrained by per-scene optimization and sensitivity to hyperparameter tuning. In this paper, we introduce Flux4D, a simple and scalable framework for 4D reconstruction of large-scale dynamic scenes. Flux4D directly predicts 3D Gaussians and their motion dynamics to reconstruct sensor observations in a fully unsupervised manner. By adopting only photometric losses and enforcing an "as static as possible" regularization, Flux4D learns to decompose dynamic elements directly from raw data without requiring pre-trained supervised models or foundational priors simply by training across many scenes. Our approach enables efficient reconstruction of dynamic scenes within seconds, scales effectively to large datasets, and generalizes well to unseen environments, including rare and unknown objects. Experiments on outdoor driving datasets show Flux4D significantly outperforms existing methods in scalability, generalization, and reconstruction quality.
comment : NeurIPS 2025. Project page: https://waabi.ai/flux4d/
☆ Does Head Pose Correction Improve Biometric Facial Recognition?
Justin Norman, Hany Farid
Biometric facial recognition models often demonstrate significant decreases in accuracy when processing real-world images, often characterized by poor quality, non-frontal subject poses, and subject occlusions. We investigate whether targeted, AI-driven, head-pose correction and image restoration can improve recognition accuracy. Using a model-agnostic, large-scale, forensic-evaluation pipeline, we assess the impact of three restoration approaches: 3D reconstruction (NextFace), 2D frontalization (CFR-GAN), and feature enhancement (CodeFormer). We find that naive application of these techniques substantially degrades facial recognition accuracy. However, we also find that selective application of CFR-GAN combined with CodeFormer yields meaningful improvements.
☆ Drainage: A Unifying Framework for Addressing Class Uncertainty
Yasser Taha, Grégoire Montavon, Nils Körber
Modern deep learning faces significant challenges with noisy labels, class ambiguity, as well as the need to robustly reject out-of-distribution or corrupted samples. In this work, we propose a unified framework based on the concept of a "drainage node'' which we add at the output of the network. The node serves to reallocate probability mass toward uncertainty, while preserving desirable properties such as end-to-end training and differentiability. This mechanism provides a natural escape route for highly ambiguous, anomalous, or noisy samples, particularly relevant for instance-dependent and asymmetric label noise. In systematic experiments involving the addition of varying proportions of instance-dependent noise or asymmetric noise to CIFAR-10/100 labels, our drainage formulation achieves an accuracy increase of up to 9\% over existing approaches in the high-noise regime. Our results on real-world datasets, such as mini-WebVision, mini-ImageNet and Clothing-1M, match or surpass existing state-of-the-art methods. Qualitative analysis reveals a denoising effect, where the drainage neuron consistently absorbs corrupt, mislabeled, or outlier data, leading to more stable decision boundaries. Furthermore, our drainage formulation enables applications well beyond classification, with immediate benefits for web-scale, semi-supervised dataset cleaning, and open-set applications.
comment : 16 pages, 8 figures
☆ Culture Affordance Atlas: Reconciling Object Diversity Through Functional Mapping
Joan Nwatu, Longju Bai, Oana Ignat, Rada Mihalcea
Culture shapes the objects people use and for what purposes, yet mainstream Vision-Language (VL) datasets frequently exhibit cultural biases, disproportionately favoring higher-income, Western contexts. This imbalance reduces model generalizability and perpetuates performance disparities, especially impacting lower-income and non-Western communities. To address these disparities, we propose a novel function-centric framework that categorizes objects by the functions they fulfill, across diverse cultural and economic contexts. We implement this framework by creating the Culture Affordance Atlas, a re-annotated and culturally grounded restructuring of the Dollar Street dataset spanning 46 functions and 288 objects publicly available at https://lit.eecs.umich.edu/CultureAffordance-Atlas/index.html. Through extensive empirical analyses using the CLIP model, we demonstrate that function-centric labels substantially reduce socioeconomic performance gaps between high- and low-income groups by a median of 6 pp (statistically significant), improving model effectiveness for lower-income contexts. Furthermore, our analyses reveals numerous culturally essential objects that are frequently overlooked in prominent VL datasets. Our contributions offer a scalable pathway toward building inclusive VL datasets and equitable AI systems.
☆ Multi- Agent Reinforcement Learning and Real-Time Decision-Making in Robotic Soccer for Virtual Environments
Aya Taourirte, Md Sohag Mia
The deployment of multi-agent systems in dynamic, adversarial environments like robotic soccer necessitates real-time decision-making, sophisticated cooperation, and scalable algorithms to avoid the curse of dimensionality. While Reinforcement Learning (RL) offers a promising framework, existing methods often struggle with the multi-granularity of tasks (long-term strategy vs. instant actions) and the complexity of large-scale agent interactions. This paper presents a unified Multi-Agent Reinforcement Learning (MARL) framework that addresses these challenges. First, we establish a baseline using Proximal Policy Optimization (PPO) within a client-server architecture for real-time action scheduling, with PPO demonstrating superior performance (4.32 avg. goals, 82.9% ball control). Second, we introduce a Hierarchical RL (HRL) structure based on the options framework to decompose the problem into a high-level trajectory planning layer (modeled as a Semi-Markov Decision Process) and a low-level action execution layer, improving global strategy (avg. goals increased to 5.26). Finally, to ensure scalability, we integrate mean-field theory into the HRL framework, simplifying many-agent interactions into a single agent vs. the population average. Our mean-field actor-critic method achieves a significant performance boost (5.93 avg. goals, 89.1% ball control, 92.3% passing accuracy) and enhanced training stability. Extensive simulations of 4v4 matches in the Webots environment validate our approach, demonstrating its potential for robust, scalable, and cooperative behavior in complex multi-agent domains.
☆ Hierarchical Process Reward Model s are Symbolic Vision Learners
Shan Zhang, Aotian Chen, Kai Zou, Jindong Gu, Yuan Xue, Anton van den Hengel
Symbolic computer vision represents diagrams through explicit logical rules and structured representations, enabling interpretable understanding in machine vision. This requires fundamentally different learning paradigms from pixel-based visual models. Symbolic visual learners parse diagrams into geometric primitives-points, lines, and shapes-whereas pixel-based learners operate on textures and colors. We propose a novel self-supervised symbolic auto-encoder that encodes diagrams into structured primitives and their interrelationships within the latent space, and decodes them through our executable engine to reconstruct the input diagrams. Central to this architecture is Symbolic Hierarchical Process Reward Modeling, which applies hierarchical step-level parsing rewards to enforce point-on-line, line-on-shape, and shape-on-relation consistency. Since vanilla reinforcement learning exhibits poor exploration in the policy space during diagram reconstruction; we thus introduce stabilization mechanisms to balance exploration and exploitation. We fine-tune our symbolic encoder on downstream tasks, developing a neuro-symbolic system that integrates the reasoning capabilities of neural networks with the interpretability of symbolic models through reasoning-grounded visual rewards. Evaluations across reconstruction, perception, and reasoning tasks demonstrate the effectiveness of our approach: achieving a 98.2% reduction in MSE for geometric diagram reconstruction, surpassing GPT-4o by 0.6% with a 7B model on chart reconstruction, and improving by +13% on the MathGlance perception benchmark, and by +3% on MathVerse and GeoQA reasoning benchmarks.
☆ PanFoMa: A Lightweight Foundation Model and Benchmark for Pan-Cancer AAAI 2026
Xiaoshui Huang, Tianlin Zhu, Yifan Zuo, Xue Xia, Zonghan Wu, Jiebin Yan, Dingli Hua, Zongyi Xu, Yuming Fang, Jian Zhang
Single-cell RNA sequencing (scRNA-seq) is essential for decoding tumor heterogeneity. However, pan-cancer research still faces two key challenges: learning discriminative and efficient single-cell representations, and establishing a comprehensive evaluation benchmark. In this paper, we introduce PanFoMa, a lightweight hybrid neural network that combines the strengths of Transformers and state-space models to achieve a balance between performance and efficiency. PanFoMa consists of a front-end local-context encoder with shared self-attention layers to capture complex, order-independent gene interactions; and a back-end global sequential feature decoder that efficiently integrates global context using a linear-time state-space model. This modular design preserves the expressive power of Transformers while leveraging the scalability of Mamba to enable transcriptome modeling, effectively capturing both local and global regulatory signals. To enable robust evaluation, we also construct a large-scale pan-cancer single-cell benchmark, PanFoMaBench, containing over 3.5 million high-quality cells across 33 cancer subtypes, curated through a rigorous preprocessing pipeline. Experimental results show that PanFoMa outperforms state-of-the-art models on our pan-cancer benchmark (+4.0\%) and across multiple public tasks, including cell type annotation (+7.4\%), batch integration (+4.0\%) and multi-omics integration (+3.1\%). The code is available at https://github.com/Xiaoshui-Huang/PanFoMa.
comment : Accepted by AAAI 2026
☆ PaCo- RL : Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Model ing
Bowen Ping, Chengyou Jia, Minnan Luo, Changliang Xia, Xin Shen, Zhuohang Dang, Hangwei Qian
Consistent image generation requires faithfully preserving identities, styles, and logical coherence across multiple images, which is essential for applications such as storytelling and character design. Supervised training approaches struggle with this task due to the lack of large-scale datasets capturing visual consistency and the complexity of modeling human perceptual preferences. In this paper, we argue that reinforcement learning (RL) offers a promising alternative by enabling models to learn complex and subjective visual criteria in a data-free manner. To achieve this, we introduce PaCo-RL, a comprehensive framework that combines a specialized consistency reward model with an efficient RL algorithm. The first component, PaCo-Reward, is a pairwise consistency evaluator trained on a large-scale dataset constructed via automated sub-figure pairing. It evaluates consistency through a generative, autoregressive scoring mechanism enhanced by task-aware instructions and CoT reasons. The second component, PaCo-GRPO, leverages a novel resolution-decoupled optimization strategy to substantially reduce RL cost, alongside a log-tamed multi-reward aggregation mechanism that ensures balanced and stable reward optimization. Extensive experiments across the two representative subtasks show that PaCo-Reward significantly improves alignment with human perceptions of visual consistency, and PaCo-GRPO achieves state-of-the-art consistency performance with improved training efficiency and stability. Together, these results highlight the promise of PaCo-RL as a practical and scalable solution for consistent image generation. The project page is available at https://x-gengroup.github.io/HomePage_PaCo-RL/.
♻ ☆ Can Vision -Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventions
Saurav Sengupta, Nazanin Moradinasab, Jiebei Liu, Donald E. Brown
Recent research suggests that Vision Language Models (VLMs) often rely on inherent biases learned during training when responding to queries about visual properties of images. These biases are exacerbated when VLMs are asked highly specific questions that require them to focus on particular areas of the image in tasks such as counting. We build upon this research by developing a synthetic benchmark dataset and evaluation framework to systematically determine how counting performance varies as image and prompt properties change. Using open-source VLMs, we then analyze how attention allocation fluctuates with varying input parameters (e.g. number of objects in the image, objects color, background color, objects texture, background texture, and prompt specificity). We further implement attention-based interventions to modulate focus on visual tokens at different layers and evaluate their impact on counting performance across a range of visual conditions. Our experiments reveal that while VLM counting performance remains challenging, especially under high visual or linguistic complexity, certain attention interventions can lead to modest gains in counting performance.
♻ ☆ PrITTI: Primitive-based Generation of Controllable and Editable 3D Semantic Urban Scenes
Christina Ourania Tze, Daniel Dauner, Yiyi Liao, Dzmitry Tsishkou, Andreas Geiger
Existing approaches to 3D semantic urban scene generation predominantly rely on voxel-based representations, which are bound by fixed resolution, challenging to edit, and memory-intensive in their dense form. In contrast, we advocate for a primitive-based paradigm where urban scenes are represented using compact, semantically meaningful 3D elements that are easy to manipulate and compose. To this end, we introduce PrITTI, a latent diffusion model that leverages vectorized object primitives and rasterized ground surfaces for generating diverse, controllable, and editable 3D semantic urban scenes. This hybrid representation yields a structured latent space that facilitates object- and ground-level manipulation. Experiments on KITTI-360 show that primitive-based representations unlock the full capabilities of diffusion transformers, achieving state-of-the-art 3D scene generation quality with lower memory requirements, faster inference, and greater editability than voxel-based methods. Beyond generation, PrITTI supports a range of downstream applications, including scene editing, inpainting, outpainting, and photo-realistic street-view synthesis. Code and models are publicly available at $\href{https://raniatze.github.io/pritti/}{https://raniatze.github.io/pritti}$.
comment : Project page: https://raniatze.github.io/pritti/
♻ ☆ Ov3R: Open-Vocabulary Semantic 3D Reconstruction from RGB Videos
Ziren Gong, Xiaohan Li, Fabio Tosi, Jiawei Han, Stefano Mattoccia, Jianfei Cai, Matteo Poggi
We present Ov3R, a novel framework for open-vocabulary semantic 3D reconstruction from RGB video streams, designed to advance Spatial AI. The system features two key components: CLIP3R, a CLIP-informed 3D reconstruction module that predicts dense point maps from overlapping clips while embedding object-level semantics; and 2D-3D OVS, a 2D-3D open-vocabulary semantic module that lifts 2D features into 3D by learning fused descriptors integrating spatial, geometric, and semantic cues. Unlike prior methods, Ov3R incorporates CLIP semantics directly into the reconstruction process, enabling globally consistent geometry and fine-grained semantic alignment. Our framework achieves state-of-the-art performance in both dense 3D reconstruction and open-vocabulary 3D segmentation, marking a step forward toward real-time, semantics-aware Spatial AI.
♻ ☆ APTx Neuron: A Unified Trainable Neuron Architecture Integrating Activation and Computation
Ravin Kumar
We propose the APTx Neuron, a novel, unified neural computation unit that integrates non-linear activation and linear transformation into a single trainable expression. The APTx Neuron is derived from the APTx activation function, thereby eliminating the need for separate activation layers and making the architecture both optimization-efficient and elegant. The proposed neuron follows the functional form $y = \sum_{i=1}^{n} ((α_i + \tanh(β_i x_i)) \cdot γ_i x_i) + δ$, where all parameters $α_i$, $β_i$, $γ_i$, and $δ$ are trainable. We validate our APTx Neuron-based architecture on the MNIST dataset, achieving up to $96.69\%$ test accuracy within 11 epochs using approximately 332K trainable parameters. The results highlight the superior expressiveness and training efficiency of the APTx Neuron compared to traditional neurons, pointing toward a new paradigm in unified neuron design and the architectures built upon it. Source code is available at https://github.com/mr-ravin/aptx_neuron.
comment : 12 pages, 2 figures, 1 table. Includes a GitHub repository for MNIST experiments and a PyPI package for APTx Neuron implementation
♻ ☆ Guardian: Detecting Robotic Planning and Execution Errors with Vision -Language Models
Paul Pacaud, Ricardo Garcia, Shizhe Chen, Cordelia Schmid
Robust robotic manipulation requires reliable failure detection and recovery. Although current Vision-Language Models (VLMs) show promise, their accuracy and generalization are limited by the scarcity of failure data. To address this data gap, we propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures. This method produces not only binary classification labels but also fine-grained failure categories and step-by-step reasoning traces in both simulation and the real world. With it, we construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail, substantially expanding the diversity and scale of existing failure datasets. We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection. Guardian achieves state-of-the-art performance on both existing and newly introduced benchmarks. It also effectively improves task success rates when integrated into a state-of-the-art manipulation system in simulation and real robots, demonstrating the impact of our generated failure data. Code, Data, and Models available at https://www.di.ens.fr/willow/research/guardian/.
comment : Code, Data, and Models available at https://www.di.ens.fr/willow/research/guardian/. The paper contains 8 pages, 9 figures, 6 tables
♻ ☆ Multimodal LLMs See Sentiment
Neemias B. da Silva, John Harrison, Rodrigo Minetto, Myriam R. Delgado, Bogdan T. Nassu, Thiago H. Silva
Understanding how visual content communicates sentiment is critical in an era where online interaction is increasingly dominated by this kind of media on social platforms. However, this remains a challenging problem, as sentiment perception is closely tied to complex, scene-level semantics. In this paper, we propose an original framework, MLLMsent, to investigate the sentiment reasoning capabilities of Multimodal Large Language Models (MLLMs) through three perspectives: (1) using those MLLMs for direct sentiment classification from images; (2) associating them with pre-trained LLMs for sentiment analysis on automatically generated image descriptions; and (3) fine-tuning the LLMs on sentiment-labeled image descriptions. Experiments on a recent and established benchmark demonstrate that our proposal, particularly the fine-tuned approach, achieves state-of-the-art results outperforming Lexicon-, CNN-, and Transformer-based baselines by up to 30.9%, 64.8%, and 42.4%, respectively, across different levels of evaluators' agreement and sentiment polarity categories. Remarkably, in a cross-dataset test, without any training on these new data, our model still outperforms, by up to 8.26%, the best runner-up, which has been trained directly on them. These results highlight the potential of the proposed visual reasoning scheme for advancing affective computing, while also establishing new benchmarks for future research.
comment : 12 pages, 7 figures
♻ ☆ AIDEN: Design and Pilot Study of an AI Assistant for the Visually Impaired
Luis Marquez-Carpintero, Francisco Gomez-Donoso, Zuria Bauer, Bessie Dominguez-Dager, Alvaro Belmonte-Baeza, Mónica Pina-Navarro, Francisco Morillas-Espejo, Felix Escalona, Miguel Cazorla
This paper presents AIDEN, an artificial intelligence-based assistant designed to enhance the autonomy and daily quality of life of visually impaired individuals, who often struggle with object identification, text reading, and navigation in unfamiliar environments. Existing solutions such as screen readers or audio-based assistants facilitate access to information but frequently lead to auditory overload and raise privacy concerns in open environments. AIDEN addresses these limitations with a hybrid architecture that integrates You Only Look Once (YOLO) for real-time object detection and a Large Language and Vision Assistant (LLaVA) for scene description and Optical Character Recognition (OCR). A key novelty of the system is a continuous haptic guidance mechanism based on a Geiger-counter metaphor, which supports object centering without occupying the auditory channel, while privacy is preserved by ensuring that no personal data are stored. Empirical evaluations with visually impaired participants assessed perceived ease of use and acceptance using the Technology Acceptance Model (TAM). Results indicate high user satisfaction, particularly regarding intuitiveness and perceived autonomy. Moreover, the ``Find an Object'' achieved effective real-time performance. These findings provide promising evidence that multimodal haptic-visual feedback can improve daily usability and independence compared to traditional audio-centric methods, motivating larger-scale clinical validations.
♻ ☆ OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision -Language Models WACV2026
Ryoto Miyamoto, Xin Fan, Fuyuko Kido, Tsuneo Matsumoto, Hayato Yamana
OpenLVLM-MIA is a new benchmark that highlights fundamental challenges in evaluating membership inference attacks (MIA) against large vision-language models (LVLMs). While prior work has reported high attack success rates, our analysis suggests that these results often arise from detecting distributional bias introduced during dataset construction rather than from identifying true membership status. To address this issue, we introduce a controlled benchmark of 6{,}000 images where the distributions of member and non-member samples are carefully balanced, and ground-truth membership labels are provided across three distinct training stages. Experiments using OpenLVLM-MIA demonstrated that the performance of state-of-the-art MIA methods approached chance-level. OpenLVLM-MIA, designed to be transparent and unbiased benchmark, clarifies certain limitations of MIA research on LVLMs and provides a solid foundation for developing stronger privacy-preserving techniques.
comment : WACV2026 Accepted
♻ ☆ OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Critic
Songyan Zhang, Wenhui Huang, Zhan Chen, Chua Jiahao Collister, Qihang Huang, Chen Lv
Recently, two-stage fine-tuning strategies, e.g., acquiring essential driving knowledge through supervised fine-tuning (SFT) and further enhancing decision-making and planning via reinforcement fine-tuning (RFT), have shown strong potential in advancing the knowledge-driven autonomous driving (AD) paradigm. However, the learning nature of SFT still limits the generalization of reasoning, thereby constraining the full potential of driving performance. Meanwhile, current RFT approaches are primarily applied to downstream tasks, since scene understanding is an open-ended problem where corresponding rewards are difficult to quantify. To address these limitations, we propose OpenREAD, an OPEN-ended REasoning reinforced vision-language model (VLM)-based autonomous driving (AD) framework that enables end-to-end RFT across the full spectrum from high-level reasoning to low-level trajectory planning. Specifically, we begin by constructing large-scale Chain-of-Thought (CoT) annotations on open-source driving-related knowledge datasets, and employ the powerful Qwen3 large language model (LLM) as the critic in RFT to quantify reasoning quality for open-ended questions during reward modeling. Extensive experiments confirm that joint end-to-end RFT yields substantial improvements in both upstream and downstream tasks, enabling OpenREAD to achieve state-of-the-art performance on reasoning and planning benchmarks.
♻ ☆ DehazeGS: Seeing Through Fog with 3D Gaussian Splatting AAAI2026
Jinze Yu, Yiqun Wang, Aiheng Jiang, Zhengda Lu, Jianwei Guo, Yong Li, Hongxing Qin, Xiaopeng Zhang
Current novel view synthesis methods are typically designed for high-quality and clean input images. However, in foggy scenes, scattering and attenuation can significantly degrade the quality of rendering. Although NeRF-based dehazing approaches have been developed, their reliance on deep fully connected neural networks and per-ray sampling strategies leads to high computational costs. Furthermore, NeRF's implicit representation limits its ability to recover fine-grained details from hazy scenes. To overcome these limitations, we propose learning an explicit Gaussian representation to explain the formation mechanism of foggy images through a physically forward rendering process. Our method, DehazeGS, reconstructs and renders fog-free scenes using only multi-view foggy images as input. Specifically, based on the atmospheric scattering model, we simulate the formation of fog by establishing the transmission function directly onto Gaussian primitives via depth-to-transmission mapping. During training, we jointly learn the atmospheric light and scattering coefficients while optimizing the Gaussian representation of foggy scenes. At inference time, we remove the effects of scattering and attenuation in Gaussian distributions and directly render the scene to obtain dehazed views. Experiments on both real-world and synthetic foggy datasets demonstrate that DehazeGS achieves state-of-the-art performance. visualizations are available at https://dehazegs.github.io/
comment : 9 pages,5 figures. Accepted by AAAI2026. visualizations are available at https://dehazegs.github.io/
♻ ☆ SkelSplat: Robust Multi-view 3D Human Pose Estimation with Differentiable Gaussian Rendering WACV 2026
Laura Bragagnolo, Leonardo Barcellona, Stefano Ghidoni
Accurate 3D human pose estimation is fundamental for applications such as augmented reality and human-robot interaction. State-of-the-art multi-view methods learn to fuse predictions across views by training on large annotated datasets, leading to poor generalization when the test scenario differs. To overcome these limitations, we propose SkelSplat, a novel framework for multi-view 3D human pose estimation based on differentiable Gaussian rendering. Human pose is modeled as a skeleton of 3D Gaussians, one per joint, optimized via differentiable rendering to enable seamless fusion of arbitrary camera views without 3D ground-truth supervision. Since Gaussian Splatting was originally designed for dense scene reconstruction, we propose a novel one-hot encoding scheme that enables independent optimization of human joints. SkelSplat outperforms approaches that do not rely on 3D ground truth in Human3.6M and CMU, while reducing the cross-dataset error up to 47.8% compared to learning-based methods. Experiments on Human3.6M-Occ and Occlusion-Person demonstrate robustness to occlusions, without scenario-specific fine-tuning. Our project page is available here: https://skelsplat.github.io.
comment : WACV 2026
♻ ☆ VeLU: Variance-enhanced Learning Unit for Deep Neural Networks
Ashkan Shakarami, Yousef Yeganeh, Azade Farshad, Lorenzo Nicolè, Stefano Ghidoni, Nassir Navab
Activation functions play a critical role in deep neural networks by shaping gradient flow, optimization stability, and generalization. While ReLU remains widely used due to its simplicity, it suffers from gradient sparsity and dead-neuron issues and offers no adaptivity to input statistics. Smooth alternatives such as Swish and GELU improve gradient propagation but still apply a fixed transformation regardless of the activation distribution. In this paper, we propose VeLU, a Variance-enhanced Learning Unit that introduces variance-aware and distributionally aligned nonlinearity through a principled combination of ArcTan-ArcSin transformations, adaptive scaling, and Wasserstein-2 regularization (Optimal Transport). This design enables VeLU to modulate its response based on local activation variance, mitigate internal covariate shift at the activation level, and improve training stability without adding learnable parameters or architectural overhead. Extensive experiments across six deep neural networks show that VeLU outperforms ReLU, ReLU6, Swish, and GELU on 12 vision benchmarks. The implementation of VeLU is publicly available in GitHub.
comment : 16 pages, 5 figures
♻ ☆ Toward Content-based Indexing and Retrieval of Head and Neck CT with Abscess Segmentation
Thao Thi Phuong Dao, Tan-Cong Nguyen, Trong-Le Do, Truong Hoang Viet, Nguyen Chi Thanh, Huynh Nguyen Thuan, Do Vo Cong Nguyen, Minh-Khoi Pham, Mai-Khiem Tran, Viet-Tham Huynh, Trong-Thuan Nguyen, Trung-Nghia Le, Vo Thanh Toan, Tam V. Nguyen, Minh-Triet Tran, Thanh Dinh Le
Abscesses in the head and neck represent an acute infectious process that can potentially lead to sepsis or mortality if not diagnosed and managed promptly. Accurate detection and delineation of these lesions on imaging are essential for diagnosis, treatment planning, and surgical intervention. In this study, we introduce AbscessHeNe, a curated and comprehensively annotated dataset comprising 4,926 contrast-enhanced CT slices with clinically confirmed head and neck abscesses. The dataset is designed to facilitate the development of robust semantic segmentation models that can accurately delineate abscess boundaries and evaluate deep neck space involvement, thereby supporting informed clinical decision-making. To establish performance baselines, we evaluate several state-of-the-art segmentation architectures, including CNN, Transformer, and Mamba-based models. The highest-performing model achieved a Dice Similarity Coefficient of 0.39, Intersection-over-Union of 0.27, and Normalized Surface Distance of 0.67, indicating the challenges of this task and the need for further research. Beyond segmentation, AbscessHeNe is structured for future applications in content-based multimedia indexing and case-based retrieval. Each CT scan is linked with pixel-level annotations and clinical metadata, providing a foundation for building intelligent retrieval systems and supporting knowledge-driven clinical workflows. The dataset will be made publicly available at https://github.com/drthaodao3101/AbscessHeNe.git.
comment : The 2025 IEEE International Conference on Content-Based Multimedia Indexing (IEEE CBMI)
♻ ☆ MasHeNe: A Benchmark for Head and Neck CT Mass Segmentation using Window-Enhanced Mamba with Frequency-Domain Integration
Thao Thi Phuong Dao, Tan-Cong Nguyen, Nguyen Chi Thanh, Truong Hoang Viet, Trong-Le Do, Mai-Khiem Tran, Minh-Khoi Pham, Trung-Nghia Le, Minh-Triet Tran, Thanh Dinh Le
Head and neck masses are space-occupying lesions that can compress the airway and esophagus and may affect nerves and blood vessels. Available public datasets primarily focus on malignant lesions and often overlook other space-occupying conditions in this region. To address this gap, we introduce MasHeNe, an initial dataset of 3,779 contrast-enhanced CT slices that includes both tumors and cysts with pixel-level annotations. We also establish a benchmark using standard segmentation baselines and report common metrics to enable fair comparison. In addition, we propose the Windowing-Enhanced Mamba with Frequency integration (WEMF) model. WEMF applies tri-window enhancement to enrich the input appearance before feature extraction. It further uses multi-frequency attention to fuse information across skip connections within a U-shaped Mamba backbone. On MasHeNe, WEMF attains the best performance among evaluated methods, with a Dice of 70.45%, IoU of 66.89%, NSD of 72.33%, and HD95 of 5.12 mm. This model indicates stable and strong results on this challenging task. MasHeNe provides a benchmark for head-and-neck mass segmentation beyond malignancy-only datasets. The observed error patterns also suggest that this task remains challenging and requires further research. Our dataset and code are available at https://github.com/drthaodao3101/MasHeNe.git.
comment : The 14th International Symposium on Information and Communication Technology Conference SoICT 2025
♻ ☆ Zero-shot self-supervised learning of single breath-hold magnetic resonance cholangiopancreatography (MRCP) reconstruction
Jinho Kim, Marcel Dominik Nickel, Florian Knoll
To investigate the feasibility of zero-shot self-supervised learning reconstruction for reducing breath-hold times in magnetic resonance cholangiopancreatography (MRCP). Breath-hold MRCP was acquired from 11 healthy volunteers on 3T scanners using an incoherent k-space sampling pattern, leading to 14-second acquisition time and an acceleration factor of R=25. Zero-shot reconstruction was compared with parallel imaging of respiratory-triggered MRCP (338s, R=3) and compressed sensing reconstruction. For two volunteers, breath-hold scans (40s, R=6) were additionally acquired and retrospectively undersampled to R=25 to compute peak signal-to-noise ratio (PSNR). To address long zero-shot training time, the n+m full stages of the zero-shot learning were divided into two parts to reduce backpropagation depth during training: 1) n frozen stages initialized with n-stage pretrained network and 2) m trainable stages initialized either randomly or m-stage pretrained network. Efficiency of our approach was assessed by varying initialization strategies and the number of trainable stages using the retrospectively undersampled data. Zero-shot reconstruction significantly improved visual image quality over compressed sensing, particularly in SNR and ductal delineation, and achieved image quality comparable to that of successful respiratory-triggered acquisitions with regular breathing patterns. Improved initializations enhanced PSNR and reduced reconstruction time. Adjusting frozen/trainable configurations demonstrated that PSNR decreased only slightly from 38.25 dB (0/13) to 37.67 dB (12/1), while training time decreased up to 6.7-fold. Zero-shot learning delivers high-fidelity MRCP reconstructions with reduced breath-hold times, and the proposed partially trainable approach offers a practical solution for translation into time-constrained clinical workflows.
comment : 24 pages, 8 figures, 2 tabels
♻ ☆ Look, Recite, Then Answer: Enhancing VLM Performance via Self-Generated Knowledge Hints
Xisheng Feng
Vision-Language Models (VLMs) exhibit significant performance plateaus in specialized domains like precision agriculture, primarily due to "Reasoning-Driven Hallucination" where linguistic priors override visual perception. A key bottleneck is the "Modality Gap": visual embeddings fail to reliably activate the fine-grained expert knowledge already encoded in model parameters. We propose "Look, Recite, Then Answer," a parameter-efficient framework that enhances VLMs via self-generated knowledge hints while keeping backbone models frozen. The framework decouples inference into three stages: (1) Look generates objective visual descriptions and candidate sets; (2) Recite employs a lightweight 1.7B router to transform visual cues into targeted queries that trigger candidate-specific parametric knowledge; (3) Answer performs parallel evidence alignment between descriptions and recited knowledge to select the most consistent label. On AgroBench, our method achieves state-of-the-art results, improving Weed Identification accuracy by 23.52% over Qwen2-VL-72B and surpassing GPT-4o without external search overhead. This modular design mitigates hallucinations by transforming passive perception into active, controllable knowledge retrieval
♻ ☆ NOCTIS: Novel Object Cyclic Threshold based Instance Segmentation
Max Gandyra, Alessandro Santonicola, Michael Beetz
Instance segmentation of novel objects instances in RGB images, given some example images for each object, is a well known problem in computer vision. Designing a model general enough to be employed for all kinds of novel objects without (re-) training has proven to be a difficult task. To handle this, we present a new training-free framework, called: Novel Object Cyclic Threshold based Instance Segmentation (NOCTIS). NOCTIS integrates two pre-trained models: Grounded-SAM 2 for object proposals with precise bounding boxes and corresponding segmentation masks; and DINOv2 for robust class and patch embeddings, due to its zero-shot capabilities. Internally, the proposal-object matching is realized by determining an object matching score based on the similarity of the class embeddings and the average maximum similarity of the patch embeddings with a new cyclic thresholding (CT) mechanism that mitigates unstable matches caused by repetitive textures or visually similar patterns. Beyond CT, NOCTIS introduces: (i) an appearance score that is unaffected by object selection bias; (ii) the usage of the average confidence of the proposals' bounding box and mask as a scoring component; and (iii) an RGB-only pipeline that performs even better than RGB-D ones. We empirically show that NOCTIS, without further training/fine tuning, outperforms the best RGB and RGB-D methods regarding the mean AP score on the seven core datasets of the BOP 2023 challenge for the "Model-based 2D segmentation of unseen objects" task.
comment : 9 pages, 3 figures, 5 tables
♻ ☆ Aligning Diffusion Models with Noise-Conditioned Perception
Alexander Gambashidze, Anton Kulikov, Yuriy Sosnin, Ilya Makarov
Recent advancements in human preference optimization, initially developed for Language Models (LMs), have shown promise for text-to-image Diffusion Models, enhancing prompt alignment, visual appeal, and user preference. Unlike LMs, Diffusion Models typically optimize in pixel or VAE space, which does not align well with human perception, leading to slower and less efficient training during the preference alignment stage. We propose using a perceptual objective in the U-Net embedding space of the diffusion model to address these issues. Our approach involves fine-tuning Stable Diffusion 1.5 and XL using Direct Preference Optimization (DPO), Contrastive Preference Optimization (CPO), and supervised fine-tuning (SFT) within this embedding space. This method significantly outperforms standard latent-space implementations across various metrics, including quality and computational cost. For SDXL, our approach provides 60.8\% general preference, 62.2\% visual appeal, and 52.1\% prompt following against original open-sourced SDXL-DPO on the PartiPrompts dataset, while significantly reducing compute. Our approach not only improves the efficiency and quality of human preference alignment for diffusion models but is also easily integrable with other optimization techniques. The training code and LoRA weights will be available here: https://huggingface.co/alexgambashidze/SDXL\_NCP-DPO\_v0.1
♻ ☆ Learning Egocentric In-Hand Object Segmentation through Weak Super vision from Human Narrations
Nicola Messina, Rosario Leonardi, Luca Ciampi, Fabio Carrara, Giovanni Maria Farinella, Fabrizio Falchi, Antonino Furnari
Pixel-level recognition of objects manipulated by the user from egocentric images enables key applications spanning assistive technologies, industrial safety, and activity monitoring. However, progress in this area is currently hindered by the scarcity of annotated datasets, as existing approaches rely on costly manual labels. In this paper, we propose to learn human-object interaction detection leveraging narrations $\unicode{x2013}$ natural language descriptions of the actions performed by the camera wearer which contain clues about manipulated objects. We introduce Narration-Supervised in-Hand Object Segmentation (NS-iHOS), a novel task where models have to learn to segment in-hand objects by learning from natural-language narrations in a weakly-supervised regime. Narrations are then not employed at inference time. We showcase the potential of the task by proposing Weakly-Supervised In-hand Object Segmentation from Human Narrations (WISH), an end-to-end model distilling knowledge from narrations to learn plausible hand-object associations and enable in-hand object segmentation without using narrations at test time. We benchmark WISH against different baselines based on open-vocabulary object detectors and vision-language models. Experiments on EPIC-Kitchens and Ego4D show that WISH surpasses all baselines, recovering more than 50% of the performance of fully supervised methods, without employing fine-grained pixel-wise annotations. Code and data can be found at https://fpv-iplab.github.io/WISH.
comment : Under consideration at Pattern Recognition Letters
♻ ☆ 3DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generation
Dewei Zhou, Ji Xie, Zongxin Yang, Yi Yang
The increasing demand for controllable outputs in text-to-image generation has spurred advancements in multi-instance generation (MIG), allowing users to define both instance layouts and attributes. However, unlike image-conditional generation methods such as ControlNet, MIG techniques have not been widely adopted in state-of-the-art models like SD2 and SDXL, primarily due to the challenge of building robust renderers that simultaneously handle instance positioning and attribute rendering. In this paper, we introduce Depth-Driven Decoupled Instance Synthesis (3DIS), a novel framework that decouples the MIG process into two stages: (i) generating a coarse scene depth map for accurate instance positioning and scene composition, and (ii) rendering fine-grained attributes using pre-trained ControlNet on any foundational model, without additional training. Our 3DIS framework integrates a custom adapter into LDM3D for precise depth-based layouts and employs a finetuning-free method for enhanced instance-level attribute rendering. Extensive experiments on COCO-Position and COCO-MIG benchmarks demonstrate that 3DIS significantly outperforms existing methods in both layout precision and attribute rendering. Notably, 3DIS offers seamless compatibility with diverse foundational models, providing a robust, adaptable solution for advanced multi-instance generation. The code is available at: https://github.com/limuloo/3DIS.
comment : 10 pages
♻ ☆ PRIMU: Uncertainty Estimation for Novel Views in Gaussian Splatting from Primitive-Based Representations of Error and Coverage
Thomas Gottwald, Edgar Heinert, Peter Stehr, Chamuditha Jayanga Galappaththige, Matthias Rottmann
We introduce Primitive-based Representations of Uncertainty (PRIMU), a post-hoc uncertainty estimation (UE) framework for Gaussian Splatting (GS). Reliable UE is essential for deploying GS in safety-critical domains such as robotics and medicine. Existing approaches typically estimate Gaussian-primitive variances and rely on the rendering process to obtain pixel-wise uncertainties. In contrast, we construct primitive-level representations of error and visibility/coverage from training views, capturing interpretable uncertainty information. These representations are obtained by projecting view-dependent training errors and coverage statistics onto the primitives. Uncertainties for novel views are inferred by rendering these primitive-level representations, producing uncertainty feature maps, which are aggregate through pixel-wise regression on holdout data. We analyze combinations of uncertainty feature maps and regression models to understand how their interactions affect prediction accuracy and generalization. PRIMU also enables an effective active view selection strategy by directly leveraging these uncertainty feature maps. Additionally, we study the effect of separating splatting into foreground and background regions. Our estimates show strong correlations with true errors, outperforming state-of-the-art methods, especially for depth UE and foreground objects. Finally, our regression models show generalization capabilities to unseen scenes, enabling UE without additional holdout data.
comment : Revised writing and figures; additional Gaussian Splatting experiments; added baselines and datasets; active view-selection experiments
♻ ☆ MRI Super-Resolution with Deep Learning: A Comprehensive Survey
Mohammad Khateri, Serge Vasylechko, Morteza Ghahremani, Liam Timms, Deniz Kocanaogullari, Simon K. Warfield, Camilo Jaimes, Davood Karimi, Alejandra Sierra, Jussi Tohka, Sila Kurugol, Onur Afacan
High-resolution (HR) magnetic resonance imaging (MRI) is crucial for many clinical and research applications. However, achieving it remains costly and constrained by technical trade-offs and experimental limitations. Super-resolution (SR) presents a promising computational approach to overcome these challenges by generating HR images from more affordable low-resolution (LR) scans, potentially improving diagnostic accuracy and efficiency without requiring additional hardware. This survey reviews recent advances in MRI SR techniques, with a focus on deep learning (DL) approaches. It examines DL-based MRI SR methods from the perspectives of computer vision, computational imaging, inverse problems, and MR physics, covering theoretical foundations, architectural designs, learning strategies, benchmark datasets, and performance metrics. We propose a systematic taxonomy to categorize these methods and present an in-depth study of both established and emerging SR techniques applicable to MRI, considering unique challenges in clinical and research contexts. We also highlight open challenges and directions that the community needs to address. Additionally, we provide a collection of essential open-access resources, tools, and tutorials, available on our GitHub: https://github.com/mkhateri/Awesome-MRI-Super-Resolution. IEEE keywords: MRI, Super-Resolution, Deep Learning, Computational Imaging, Inverse Problem, Survey.
comment : 41 pages
♻ ☆ AVA-VLA: Improving Vision -Language-Action models with Active Visual Attention
Lei Xiao, Jifeng Li, Juntao Gao, Feiyang Ye, Yan Jin, Jingjing Qian, Jing Zhang, Yong Wu, Xiaoyuan Yu
Vision-Language-Action (VLA) models have demonstrated remarkable capabilities in embodied AI tasks. However, existing VLA models, often built upon Vision-Language Models (VLMs), typically process dense visual inputs independently at each timestep. This approach implicitly models the task as a Markov Decision Process (MDP). However, this history-agnostic design is suboptimal for effective visual token processing in dynamic sequential decision-making, as it fails to leverage the context of history. To address this limitation, we reformulate the problem from a Partially Observable Markov Decision Process (POMDP) perspective and propose a novel framework named AVA-VLA. Inspired by the POMDP that the action generation should be conditioned on the belief state. AVA-VLA introduces Active Visual Attention (AVA) to dynamically modulate visual processing. It achieves this by leveraging the recurrent state, which is a neural approximation of the agent's belief state derived from the previous decision step. Specifically, the AVA module uses the recurrent state to compute the soft weights to actively process task-relevant visual tokens based on its historical context. Comprehensive evaluations demonstrate that AVA-VLA achieves state-of-the-art performance across popular robotic benchmarks, including LIBERO and CALVIN. Furthermore, real-world deployments on a dual-arm robot platform validate the framework's practical applicability and robust sim-to-real transferability.
comment : 18 pages, 10 figures
♻ ☆ End-to-End Multi-Person Pose Estimation with Pose-Aware Video Transformer
Yonghui Yu, Jiahang Cai, Xun Wang, Wenwu Yang
Existing multi-person video pose estimation methods typically adopt a two-stage pipeline: detecting individuals in each frame, followed by temporal modeling for single person pose estimation. This design relies on heuristic operations such as detection, RoI cropping, and non-maximum suppression (NMS), limiting both accuracy and efficiency. In this paper, we present a fully end-to-end framework for multi-person 2D pose estimation in videos, effectively eliminating heuristic operations. A key challenge is to associate individuals across frames under complex and overlapping temporal trajectories. To address this, we introduce a novel Pose-Aware Video transformEr Network (PAVE-Net), which features a spatial encoder to model intra-frame relations and a spatiotemporal pose decoder to capture global dependencies across frames. To achieve accurate temporal association, we propose a pose-aware attention mechanism that enables each pose query to selectively aggregate features corresponding to the same individual across consecutive frames. Additionally, we explicitly model spatiotemporal dependencies among pose keypoints to improve accuracy. Notably, our approach is the first end-to-end method for multi-frame 2D human pose estimation. Extensive experiments show that PAVE-Net substantially outperforms prior image-based end-to-end methods, achieving a 6.0 mAP improvement on PoseTrack2017, and delivers accuracy competitive with state-of-the-art two-stage video based approaches, while offering significant gains in efficiency. Project page: https://github.com/zgspose/PAVENet.
♻ ☆ Bias Beyond Demographics: Probing Decision Boundaries in Black-Box LVLMs via Counterfactual VQA
Zaiying Zhao, Toshihiko Yamasaki
Recent advances in large vision-language models (LVLMs) have amplified concerns about fairness, yet existing evaluations remain confined to demographic attributes and often conflate fairness with refusal behavior. This paper broadens the scope of fairness by introducing a counterfactual VQA benchmark that probes the decision boundaries of closed-source LVLMs under controlled context shifts. Each image pair differs in a single visual attribute that has been validated as irrelevant to the question, enabling ground-truth-free and refusal-aware analysis of reasoning stability. Comprehensive experiments reveal that non-demographic attributes, such as environmental context or social behavior, distort LVLM decision-making more strongly than demographic ones. Moreover, instruction-based debiasing shows limited effectiveness and can even amplify these asymmetries, whereas exposure to a small number of human norm validated examples from our benchmark encourages more consistent and balanced responses, highlighting its potential not only as an evaluative framework but also as a means for understanding and improving model behavior. Together, these results provide a practial basis for auditing contextual biases even in black-box LVLMs and contribute to more transparent and equitable multimodal reasoning.
♻ ☆ FairT2I: Mitigating Social Bias in Text-to-Image Generation via Large Language Model-Assisted Detection and Attribute Rebalancing
Jinya Sakurai, Yuki Koyama, Issei Sato
Text-to-image (T2I) models have advanced creative content generation, yet their reliance on large uncurated datasets often reproduces societal biases. We present FairT2I, a training-free and interactive framework grounded in a mathematically principled latent variable guidance formulation. This formulation decomposes the generative score function into attribute-conditioned components and reweights them according to a defined distribution, providing a unified and flexible mechanism for bias-aware generation that also subsumes many existing ad hoc debiasing approaches as special cases. Building upon this foundation, FairT2I incorporates (1) latent variable guidance as the core mechanism, (2) LLM-based bias detection to automatically infer bias-prone categories and attributes from text prompts as part of the latent structure, and (3) attribute resampling, which allows users to adjust or redefine the attribute distribution based on uniform, real-world, or user-specified statistics. The accompanying user interface supports this pipeline by enabling users to inspect detected biases, modify attributes or weights, and generate debiased images in real time. Experimental results show that LLMs outperform average human annotators in the number and granularity of detected bias categories and attributes. Moreover, FairT2I achieves superior performance to baseline models in both societal bias mitigation and image diversity, while preserving image quality and prompt fidelity.
♻ ☆ Detect Anything 3D in the Wild
Hanxue Zhang, Haoran Jiang, Qingsong Yao, Yanan Sun, Renrui Zhang, Hao Zhao, Hongyang Li, Hongzi Zhu, Zetong Yang
Despite the success of deep learning in close-set 3D object detection, existing approaches struggle with zero-shot generalization to novel objects and camera configurations. We introduce DetAny3D, a promptable 3D detection foundation model capable of detecting any novel object under arbitrary camera configurations using only monocular inputs. Training a foundation model for 3D detection is fundamentally constrained by the limited availability of annotated 3D data, which motivates DetAny3D to leverage the rich prior knowledge embedded in extensively pre-trained 2D foundation models to compensate for this scarcity. To effectively transfer 2D knowledge to 3D, DetAny3D incorporates two core modules: the 2D Aggregator, which aligns features from different 2D foundation models, and the 3D Interpreter with Zero-Embedding Mapping, which stabilizes early training in 2D-to-3D knowledge transfer. Experimental results validate the strong generalization of our DetAny3D, which not only achieves state-of-the-art performance on unseen categories and novel camera configurations, but also surpasses most competitors on in-domain data. DetAny3D sheds light on the potential of the 3D foundation model for diverse applications in real-world scenarios, e.g., rare object detection in autonomous driving, and demonstrates promise for further exploration of 3D-centric tasks in open-world settings. More visualization results can be found at our code repository.
♻ ☆ Walk Before You Dance: High-fidelity and Editable Dance Synthesis via Generative Masked Motion Prior
Foram N Shah, Parshwa Shah, Muhammad Usama Saleem, Ekkasit Pinyoanuntapong, Pu Wang, Hongfei Xue, Ahmed Helmy
Recent advances in dance generation have enabled the automatic synthesis of 3D dance motions. However, existing methods still face significant challenges in simultaneously achieving high realism, precise dance-music synchronization, diverse motion expression, and physical plausibility. To address these limitations, we propose a novel approach that leverages a generative masked text-to-motion model as a distribution prior to learn a probabilistic mapping from diverse guidance signals, including music, genre, and pose, into high-quality dance motion sequences. Our framework also supports semantic motion editing, such as motion inpainting and body part modification. Specifically, we introduce a multi-tower masked motion model that integrates a text-conditioned masked motion backbone with two parallel, modality-specific branches: a music-guidance tower and a pose-guidance tower. The model is trained using synchronized and progressive masked training, which allows effective infusion of the pretrained text-to-motion prior into the dance synthesis process while enabling each guidance branch to optimize independently through its own loss function, mitigating gradient interference. During inference, we introduce classifier-free logits guidance and pose-guided token optimization to strengthen the influence of music, genre, and pose signals. Extensive experiments demonstrate that our method sets a new state of the art in dance generation, significantly advancing both the quality and editability over existing approaches. Project Page available at https://foram-s1.github.io/DanceMosaic/
♻ ☆ Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Survey
Liewen Liao, Weihao Yan, Wang Xu, Ming Yang, Songan Zhang, H. Eric Tseng
Learning-based 3D reconstruction has emerged as a transformative technique in autonomous driving, enabling precise modeling of environments through advanced neural representations. It has inspired pioneering solutions for vital tasks in autonomous driving, such as dense mapping and closed-loop simulation, as well as comprehensive scene feature for driving scene understanding and reasoning. Given the rapid growth in related research, this survey provides a comprehensive review of both technical evolutions and practical applications in autonomous driving. We begin with an introduction to the preliminaries of learning-based 3D reconstruction to provide a solid technical background foundation, then progress to a rigorous, multi-dimensional examination of cutting-edge methodologies, systematically organized according to the distinctive technical requirements and fundamental challenges of autonomous driving. Through analyzing and summarizing development trends and cutting-edge research, we identify existing technical challenges, along with insufficient disclosure of on-board validation and safety verification details in the current literature, and ultimately suggest potential directions to guide future studies.
comment : Published in IEEE Trans. on Intelligent Transportation Systems
♻ ☆ CT-GLIP: 3D Grounded Language-Image Pretrain ing with CT Scans and Radiology Reports for Full-Body Scenarios
Jingyang Lin, Yingda Xia, Jianpeng Zhang, Ke Yan, Kai Cao, Le Lu, Jiebo Luo, Ling Zhang
3D medical vision-language (VL) pretraining has shown potential in radiology by leveraging large-scale multimodal datasets with CT-report pairs. However, existing methods primarily rely on a global VL alignment directly adapted from 2D scenarios. The entire 3D image is transformed into one global embedding, resulting in a loss of sparse but critical semantics essential for accurately aligning with the corresponding diagnosis. To address this limitation, we propose CT-GLIP, a 3D Grounded Language-Image Pretrained model that constructs fine-grained CT-report pairs to enhance \textit{grounded} cross-modal contrastive learning, effectively aligning grounded visual features with precise textual descriptions. Leveraging the grounded cross-modal alignment, CT-GLIP improves performance across diverse downstream tasks and can even identify organs and abnormalities in a zero-shot manner using natural language. CT-GLIP is trained on a multimodal CT dataset comprising 44,011 organ-level CT-report pairs from 17,702 patients, covering 104 organs. Evaluation is conducted on four downstream tasks: zero-shot organ recognition (OR), zero-shot abnormality detection (AD), tumor detection (TD), and tumor segmentation (TS). Empirical results show that it outperforms its counterparts with global VL alignment. Compared to vanilla CLIP, CT-GLIP achieves average performance improvements of 15.1% of F1 score, 1.9% of AUC, and 3.2% of DSC for zero-shot AD, TD, and TS tasks, respectively. This study highlights the significance of grounded VL alignment in enabling 3D medical VL foundation models to understand sparse representations within CT scans.
♻ ☆ MegaSR: Mining Customized Semantics and Expressive Guidance for Real-Wo rl d Image Super-Resolution
Xinrui Li, Jinrong Zhang, Jianlong Wu, Chong Chen, Liqiang Nie, Zhouchen Lin
Text-to-image (T2I) models have ushered in a new era of real-world image super-resolution (Real-ISR) due to their rich internal implicit knowledge for multimodal learning. Although bringing high-level semantic priors and dense pixel guidance have led to advances in reconstruction, we identified several critical phenomena by analyzing the behavior of existing T2I-based Real-ISR methods: (1) Fine detail deficiency, which ultimately leads to incorrect reconstruction in local regions. (2) Block-wise semantic inconsistency, which results in distracted semantic interpretations across U-Net blocks. (3) Edge ambiguity, which causes noticeable structural degradation. Building upon these observations, we first introduce MegaSR, which enhances the T2I-based Real-ISR models with fine-grained customized semantics and expressive guidance to unlock semantically rich and structurally consistent reconstruction. Then, we propose the Customized Semantics Module (CSM) to supplement fine-grained semantics from the image modality and regulate the semantic fusion between multi-level knowledge to realize customization for different U-Net blocks. Besides the semantic adaptation, we identify expressive multimodal signals through pair-wise comparisons and introduce the Multimodal Signal Fusion Module (MSFM) to aggregate them for structurally consistent reconstruction. Extensive experiments on real-world and synthetic datasets demonstrate the superiority of the method. Notably, it not only achieves state-of-the-art performance on quality-driven metrics but also remains competitive on fidelity-focused metrics, striking a balance between perceptual realism and faithful content reconstruction.
♻ ☆ Diffusion -SDPO: Safeguarded Direct Preference Optimization for Diffusion Models
Minghao Fu, Guo-Hua Wang, Tianyu Cui, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang
Text-to-image diffusion models deliver high-quality images, yet aligning them with human preferences remains challenging. We revisit diffusion-based Direct Preference Optimization (DPO) for these models and identify a critical pathology: enlarging the preference margin does not necessarily improve generation quality. In particular, the standard Diffusion-DPO objective can increase the reconstruction error of both winner and loser branches. Consequently, degradation of the less-preferred outputs can become sufficiently severe that the preferred branch is also adversely affected even as the margin grows. To address this, we introduce Diffusion-SDPO, a safeguarded update rule that preserves the winner by adaptively scaling the loser gradient according to its alignment with the winner gradient. A first-order analysis yields a closed-form scaling coefficient that guarantees the error of the preferred output is non-increasing at each optimization step. Our method is simple, model-agnostic, broadly compatible with existing DPO-style alignment frameworks and adds only marginal computational overhead. Across standard text-to-image benchmarks, Diffusion-SDPO delivers consistent gains over preference-learning baselines on automated preference, aesthetic, and prompt alignment metrics. Code is publicly available at https://github.com/AIDC-AI/Diffusion-SDPO.
comment : The code is publicly available at https://github.com/AIDC-AI/Diffusion-SDPO
♻ ☆ Mutually-Aware Feature Learning for Few-Shot Object Counting
Yerim Jeon, Subeen Lee, Jihwan Kim, Jae-Pil Heo
Few-shot object counting has garnered significant attention for its practicality as it aims to count target objects in a query image based on given exemplars without additional training. However, the prevailing extract-and-match approach has a shortcoming: query and exemplar features lack interaction during feature extraction since they are extracted independently and later correlated based on similarity. This can lead to insufficient target awareness and confusion in identifying the actual target when multiple class objects coexist. To address this, we propose a novel framework, Mutually-Aware FEAture learning (MAFEA), which encodes query and exemplar features with mutual awareness from the outset. By encouraging interaction throughout the pipeline, we obtain target-aware features robust to a multi-category scenario. Furthermore, we introduce background token to effectively associate the query's target region with exemplars and decouple its background region. Our extensive experiments demonstrate that our model achieves state-of-the-art performance on FSCD-LVIS and FSC-147 benchmarks with remarkably reduced target confusion.
comment : Accepted to Pattern Recognition 2025
♻ ☆ Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer
Z-Image Team, Huanqia Cai, Sihan Cao, Ruoyi Du, Peng Gao, Steven Hoi, Zhaohui Hou, Shijie Huang, Dengyang Jiang, Xin Jin, Liangchen Li, Zhen Li, Zhong-Yu Li, David Liu, Dongyang Liu, Junhan Shi, Qilong Wu, Feng Yu, Chi Zhang, Shifeng Zhang, Shilin Zhou
The landscape of high-performance image generation models is currently dominated by proprietary systems, such as Nano Banana Pro and Seedream 4.0. Leading open-source alternatives, including Qwen-Image, Hunyuan-Image-3.0 and FLUX.2, are characterized by massive parameter counts (20B to 80B), making them impractical for inference, and fine-tuning on consumer-grade hardware. To address this gap, we propose Z-Image, an efficient 6B-parameter foundation generative model built upon a Scalable Single-Stream Diffusion Transformer (S3-DiT) architecture that challenges the "scale-at-all-costs" paradigm. By systematically optimizing the entire model lifecycle -- from a curated data infrastructure to a streamlined training curriculum -- we complete the full training workflow in just 314K H800 GPU hours (approx. $630K). Our few-step distillation scheme with reward post-training further yields Z-Image-Turbo, offering both sub-second inference latency on an enterprise-grade H800 GPU and compatibility with consumer-grade hardware (<16GB VRAM). Additionally, our omni-pre-training paradigm also enables efficient training of Z-Image-Edit, an editing model with impressive instruction-following capabilities. Both qualitative and quantitative experiments demonstrate that our model achieves performance comparable to or surpassing that of leading competitors across various dimensions. Most notably, Z-Image exhibits exceptional capabilities in photorealistic image generation and bilingual text rendering, delivering results that rival top-tier commercial models, thereby demonstrating that state-of-the-art results are achievable with significantly reduced computational overhead. We publicly release our code, weights, and online demo to foster the development of accessible, budget-friendly, yet state-of-the-art generative models.
♻ ☆ ST-Booster: An Iterative SpatioTemporal Perception Booster for Vision -and-Language Navigation in Continuous Environments
Lu Yue, Dongliang Zhou, Liang Xie, Erwei Yin, Feitian Zhang
Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires agents to navigate unknown, continuous spaces based on natural language instructions. Compared to discrete settings, VLN-CE poses two core perception challenges. First, the absence of predefined observation points leads to heterogeneous visual memories and weakened global spatial correlations. Second, cumulative reconstruction errors in three-dimensional scenes introduce structural noise, impairing local feature perception. To address these challenges, this paper proposes ST-Booster, an iterative spatiotemporal booster that enhances navigation performance through multi-granularity perception and instruction-aware reasoning. ST-Booster consists of three key modules -- Hierarchical SpatioTemporal Encoding (HSTE), Multi-Granularity Aligned Fusion (MGAF), and ValueGuided Waypoint Generation (VGWG). HSTE encodes long-term global memory using topological graphs and captures shortterm local details via grid maps. MGAF aligns these dualmap representations with instructions through geometry-aware knowledge fusion. The resulting representations are iteratively refined through pretraining tasks. During reasoning, VGWG generates Guided Attention Heatmaps (GAHs) to explicitly model environment-instruction relevance and optimize waypoint selection. Extensive comparative experiments and performance analyses are conducted, demonstrating that ST-Booster outperforms existing state-of-the-art methods, particularly in complex, disturbance-prone environments.
comment : 11 pages, 7 figures
♻ ☆ AlignBench: Benchmark ing Fine-Grained Image-Text Alignment with Synthetic Image-Caption Pairs
Kuniaki Saito, Risa Shinoda, Shohei Tanaka, Tosho Hirasawa, Fumio Okura, Yoshitaka Ushiku
Assessing image-text alignment models such as CLIP is crucial for bridging visual and linguistic representations. Yet existing benchmarks rely on rule-based perturbations or short captions, limiting their ability to measure fine-grained alignment. We introduce AlignBench, a benchmark that provides a new indicator of image-text alignment by evaluating detailed image-caption pairs generated by diverse image-to-text and text-to-image models. Each sentence is annotated for correctness, enabling direct assessment of VLMs as alignment evaluators. Benchmarking a wide range of decoder-based VLMs reveals three key findings: (i) CLIP-based models, even those tailored for compositional reasoning, remain nearly blind; (ii) detectors systematically over-score early sentences; and (iii) they show strong self-preference, favoring their own outputs and harming detection performance. Our project page will be available at https://dahlian00.github.io/AlignBench/.
comment : Project Page: https://dahlian00.github.io/AlignBench/
♻ ☆ Wo rl dMem: Long-term Consistent Wo rl d Simulation with Memory
Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, Xingang Pan
World simulation has gained increasing popularity due to its ability to model virtual environments and predict the consequences of actions. However, the limited temporal context window often leads to failures in maintaining long-term consistency, particularly in preserving 3D spatial consistency. In this work, we present WorldMem, a framework that enhances scene generation with a memory bank consisting of memory units that store memory frames and states (e.g., poses and timestamps). By employing a memory attention mechanism that effectively extracts relevant information from these memory frames based on their states, our method is capable of accurately reconstructing previously observed scenes, even under significant viewpoint or temporal gaps. Furthermore, by incorporating timestamps into the states, our framework not only models a static world but also captures its dynamic evolution over time, enabling both perception and interaction within the simulated world. Extensive experiments in both virtual and real scenarios validate the effectiveness of our approach.
comment : Project page at https://xizaoqu.github.io/worldmem/
♻ ☆ Multimodal Continual Learning with MLLMs from Multi-scenario Perspectives CVPR 2026
Kai Jiang, Siqi Huang, Xiangyu Chen, Jiawei Shao, Hongyuan Zhang, Xuelong Li
Continual learning in visual understanding aims to deal with catastrophic forgetting in Multimodal Large Language Models (MLLMs). MLLMs deployed on devices have to continuously adapt to dynamic scenarios in downstream tasks, such as variations in background and perspective, to effectively perform complex visual tasks. To this end, we construct a multimodal visual understanding dataset (MSVQA) encompassing four different scenarios and perspectives including high altitude, underwater, low altitude and indoor, to investigate the catastrophic forgetting in MLLMs under the dynamics of scenario shifts in real-world data streams. Furthermore, we propose mUltimodal coNtInual learning with MLLMs From multi-scenarIo pERspectives (UNIFIER) to address visual discrepancies while learning different scenarios. Specifically, it decouples the visual information from different scenarios into distinct branches within each vision block and projects them into the same feature space. A consistency constraint is imposed on the features of each branch to maintain the stability of visual representations across scenarios. Extensive experiments on the MSVQA dataset demonstrate that UNIFIER effectively alleviates forgetting of cross-scenario tasks and achieves knowledge accumulation within the same scenario.
comment : 18 pages, 16 figures. This is a preprint version of a paper submitted to CVPR 2026
♻ ☆ TempoMaster: Efficient Long Video Generation via Next-Frame-Rate Prediction
Yukuo Ma, Cong Liu, Junke Wang, Junqi Liu, Haibin Huang, Zuxuan Wu, Chi Zhang, Xuelong Li
We present TempoMaster, a novel framework that formulates long video generation as next-frame-rate prediction. Specifically, we first generate a low-frame-rate clip that serves as a coarse blueprint of the entire video sequence, and then progressively increase the frame rate to refine visual details and motion continuity. During generation, TempoMaster employs bidirectional attention within each frame-rate level while performing autoregression across frame rates, thus achieving long-range temporal coherence while enabling efficient and parallel synthesis. Extensive experiments demonstrate that TempoMaster establishes a new state-of-the-art in long video generation, excelling in both visual and temporal quality.
comment : for more information, see https://scottykma.github.io/tempomaster-gitpage/
♻ ☆ SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledge SP
Yumeng He, Ying Jiang, Jiayin Lu, Yin Yang, Chenfanfu Jiang
Articulated 3D objects are critical for embodied AI, robotics, and interactive scene understanding, yet creating simulation-ready assets remains labor-intensive and requires expert modeling of part hierarchies and motion structures. We introduce SPARK, a framework for reconstructing physically consistent, kinematic part-level articulated objects from a single RGB image. Given an input image, we first leverage VLMs to extract coarse URDF parameters and generate part-level reference images. We then integrate the part-image guidance and the inferred structure graph into a generative diffusion transformer to synthesize consistent part and complete shapes of articulated objects. To further refine the URDF parameters, we incorporate differentiable forward kinematics and differentiable rendering to optimize joint types, axes, and origins under VLM-generated open-state supervision. Extensive experiments show that SPARK produces high-quality, simulation-ready articulated assets across diverse categories, enabling downstream applications such as robotic manipulation and interaction modeling. Project page: https://heyumeng.com/SPARK/index.html.
comment : Project page: https://heyumeng.com/SPARK/index.html. 17 pages, 7 figures
♻ ☆ TimeSearch: Hierarchical Video Search with Spotlight and Reflection for Human-like Long Video Understanding
Junwen Pan, Rui Zhang, Xin Wan, Yuan Zhang, Ming Lu, Qi She
Large video-language models (LVLMs) have shown remarkable performance across various video-language tasks. However, they encounter significant challenges when processing long videos because of the large number of video frames involved. Downsampling long videos in either space or time can lead to visual hallucinations, making it difficult to accurately interpret long videos. Motivated by human hierarchical temporal search strategies, we propose \textbf{TimeSearch}, a novel framework enabling LVLMs to understand long videos in a human-like manner. TimeSearch integrates two human-like primitives into a unified autoregressive LVLM: 1) \textbf{Spotlight} efficiently identifies relevant temporal events through a Temporal-Augmented Frame Representation (TAFR), explicitly binding visual features with timestamps; 2) \textbf{Reflection} evaluates the correctness of the identified events, leveraging the inherent temporal self-reflection capabilities of LVLMs. TimeSearch progressively explores key events and prioritizes temporal search based on reflection confidence. Extensive experiments on challenging long-video benchmarks confirm that TimeSearch substantially surpasses previous state-of-the-art, improving the accuracy from 41.8\% to 51.5\% on the LVBench. Additionally, experiments on temporal grounding demonstrate that appropriate TAFR is adequate to effectively stimulate the surprising temporal grounding ability of LVLMs in a simpler yet versatile manner, which improves mIoU on Charades-STA by 11.8\%. The code will be released.
♻ ☆ Steering One-Step Diffusion Model with Fidelity-Rich Decoder for Fast Image Compression AAAI 2026
Zheng Chen, Mingde Zhou, Jinpei Guo, Jiale Yuan, Yifei Ji, Yulun Zhang
Diffusion-based image compression has demonstrated impressive perceptual performance. However, it suffers from two critical drawbacks: (1) excessive decoding latency due to multi-step sampling, and (2) poor fidelity resulting from over-reliance on generative priors. To address these issues, we propose SODEC, a novel single-step diffusion image compression model. We argue that in image compression, a sufficiently informative latent renders multi-step refinement unnecessary. Based on this insight, we leverage a pre-trained VAE-based model to produce latents with rich information, and replace the iterative denoising process with a single-step decoding. Meanwhile, to improve fidelity, we introduce the fidelity guidance module, encouraging output that is faithful to the original image. Furthermore, we design the rate annealing training strategy to enable effective training under extremely low bitrates. Extensive experiments show that SODEC significantly outperforms existing methods, achieving superior rate-distortion-perception performance. Moreover, compared to previous diffusion-based compression models, SODEC improves decoding speed by more than 20$\times$. Code is released at: https://github.com/zhengchen1999/SODEC.
comment : Accepted to AAAI 2026. Code is available at: https://github.com/zhengchen1999/SODEC
♻ ☆ Skeleton Agent : An Agent ic Interaction Framework for Skeleton-based Action Recognition
Hongda Liu, Yunfan Liu, Changlu Wang, Yunlong Wang, Zhenan Sun
Recent advances in skeleton-based action recognition increasingly leverage semantic priors from Large Language Models (LLMs) to enrich skeletal representations. However, the LLM is typically queried in isolation from the recognition model and receives no performance feedback. As a result, it often fails to deliver the targeted discriminative cues critical to distinguish similar actions. To overcome these limitations, we propose SkeletonAgent, a novel framework that bridges the recognition model and the LLM through two cooperative agents, i.e., Questioner and Selector. Specifically, the Questioner identifies the most frequently confused classes and supplies them to the LLM as context for more targeted guidance. Conversely, the Selector parses the LLM's response to extract precise joint-level constraints and feeds them back to the recognizer, enabling finer-grained cross-modal alignment. Comprehensive evaluations on five benchmarks, including NTU RGB+D, NTU RGB+D 120, Kinetics-Skeleton, FineGYM, and UAV-Human, demonstrate that SkeletonAgent consistently outperforms state-of-the-art benchmark methods. The code is available at https://github.com/firework8/SkeletonAgent.
♻ ☆ Cross-Cancer Knowledge Transfer in WSI-based Prognosis Prediction
Pei Liu, Luping Ji, Jiaxiang Gou, Xiangxiang Zeng
Whole-Slide Image (WSI) is an important tool for estimating cancer prognosis. Current studies generally follow a conventional cancer-specific paradigm in which each cancer corresponds to a single model. However, this paradigm naturally struggles to scale to rare tumors and cannot leverage knowledge from other cancers. While multi-task learning frameworks have been explored recently, they often place high demands on computational resources and require extensive training on ultra-large, multi-cancer WSI datasets. To this end, this paper shifts the paradigm to knowledge transfer and presents the first preliminary yet systematic study on cross-cancer prognosis knowledge transfer in WSIs, called CROPKT. It comprises three major parts. (1) We curate a large dataset (UNI2-h-DSS) with 26 cancers and use it to measure the transferability of WSI-based prognostic knowledge across different cancers (including rare tumors). (2) Beyond a simple evaluation merely for benchmarking, we design a range of experiments to gain deeper insights into the underlying mechanism behind transferability. (3) We further show the utility of cross-cancer knowledge transfer, by proposing a routing-based baseline approach (ROUPKT) that could often efficiently utilize the knowledge transferred from off-the-shelf models of other cancers. CROPKT could serve as an inception that lays the foundation for this nascent paradigm, i.e., WSI-based prognosis prediction with cross-cancer knowledge transfer. Our source code is available at https://github.com/liupei101/CROPKT.
comment : 24 pages (11 figures and 10 tables)
♻ ☆ Beyond Pixels: Efficient Dataset Distillation via Sparse Gaussian Representation
Chenyang Jiang, Zhengcen Li, Hang Zhao, Qiben Shan, Shaocong Wu, Jingyong Su
Dataset distillation has emerged as a promising paradigm that synthesizes compact, informative datasets capable of retaining the knowledge of large-scale counterparts, thereby addressing the substantial computational and storage burdens of modern model training. Conventional approaches typically rely on dense pixel-level representations, which introduce redundancy and are difficult to scale up. In this work, we propose GSDD, a novel and efficient sparse representation for dataset distillation based on 2D Gaussians. Instead of representing all pixels equally, GSDD encodes critical discriminative information in a distilled image using only a small number of Gaussian primitives. This sparse representation could improve dataset diversity under the same storage budget, enhancing coverage of difficult samples and boosting distillation performance. To ensure both efficiency and scalability, we adapt CUDA-based splatting operators for parallel inference and training, enabling high-quality rendering with minimal computational and memory overhead. Our method is simple yet effective, broadly applicable to different distillation pipelines, and highly scalable. Experiments show that GSDD achieves state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet subsets, while remaining highly efficient encoding and decoding cost. Our code is available at https://github.com/j-cyoung/GSDatasetDistillation.
comment : 19 pages; Code is available on https://github.com/j-cyoung/GSDatasetDistillation
♻ ☆ MaxSup: Overcoming Representation Collapse in Label Smoothing NeurIPS 2025
Yuxuan Zhou, Heng Li, Zhi-Qi Cheng, Xudong Yan, Yifei Dong, Mario Fritz, Margret Keuper
Label Smoothing (LS) is widely adopted to reduce overconfidence in neural network predictions and improve generalization. Despite these benefits, recent studies reveal two critical issues with LS. First, LS induces overconfidence in misclassified samples. Second, it compacts feature representations into overly tight clusters, diluting intra-class diversity, although the precise cause of this phenomenon remained elusive. In this paper, we analytically decompose the LS-induced loss, exposing two key terms: (i) a regularization term that dampens overconfidence only when the prediction is correct, and (ii) an error-amplification term that arises under misclassifications. This latter term compels the network to reinforce incorrect predictions with undue certainty, exacerbating representation collapse. To address these shortcomings, we propose Max Suppression (MaxSup), which applies uniform regularization to both correct and incorrect predictions by penalizing the top-1 logit rather than the ground-truth logit. Through extensive feature-space analyses, we show that MaxSup restores intra-class variation and sharpens inter-class boundaries. Experiments on large-scale image classification and multiple downstream tasks confirm that MaxSup is a more robust alternative to LS. Code is available at: https://github.com/ZhouYuxuanYX/Maximum-Suppression-Regularization
comment : NeurIPS 2025 Oral (0.36% acceptance); code: https://github.com/ZhouYuxuanYX/Maximum-Suppression-Regularization
♻ ☆ Self-Supervised Compression and Artifact Correction for Streaming Underwater Imaging Sonar WACV 2026
Rongsheng Qian, Chi Xu, Xiaoqiang Ma, Hao Fang, Yili Jin, William I. Atlas, Jiangchuan Liu
Real-time imaging sonar is crucial for underwater monitoring where optical sensing fails, but its use is limited by low uplink bandwidth and severe sonar-specific artifacts (speckle, motion blur, reverberation, acoustic shadows) affecting up to 98% of frames. We present SCOPE, a self-supervised framework that jointly performs compression and artifact correction without clean-noise pairs or synthetic assumptions. SCOPE combines (i) Adaptive Codebook Compression (ACC), which learns frequency-encoded latent representations tailored to sonar, with (ii) Frequency-Aware Multiscale Segmentation (FAMS), which decomposes frames into low-frequency structure and sparse high-frequency dynamics while suppressing rapidly fluctuating artifacts. A hedging training strategy further guides frequency-aware learning using low-pass proxy pairs generated without labels. Evaluated on months of in-situ ARIS sonar data, SCOPE achieves a structural similarity index (SSIM) of 0.77, representing a 40% improvement over prior self-supervised denoising baselines, at bitrates down to <= 0.0118 bpp. It reduces uplink bandwidth by more than 80% while improving downstream detection. The system runs in real time, with 3.1 ms encoding on an embedded GPU and 97 ms full multi-layer decoding on the server end. SCOPE has been deployed for months in three Pacific Northwest rivers to support real-time salmon enumeration and environmental monitoring in the wild. Results demonstrate that learning frequency-structured latents enables practical, low-bitrate sonar streaming with preserved signal details under real-world deployment conditions.
comment : Accepted to WACV 2026
♻ ☆ Rank Matters: Understanding and Defending Model Inversion Attacks via Low-Rank Feature Filtering KDD 2026
Hongyao Yu, Yixiang Qiu, Hao Fang, Tianqu Zhuang, Bin Chen, Sijin Yu, Bin Wang, Shu-Tao Xia, Ke Xu
Model Inversion Attacks (MIAs) pose a significant threat to data privacy by reconstructing sensitive training samples from the knowledge embedded in trained machine learning models. Despite recent progress in enhancing the effectiveness of MIAs across diverse settings, defense strategies have lagged behind, struggling to balance model utility with robustness against increasingly sophisticated attacks. In this work, we propose the ideal inversion error to measure the privacy leakage, and our theoretical and empirical investigations reveals that higher-rank features are inherently more prone to privacy leakage. Motivated by this insight, we propose a lightweight and effective defense strategy based on low-rank feature filtering, which explicitly reduces the attack surface by constraining the dimension of intermediate representations. Extensive experiments across various model architectures and datasets demonstrate that our method consistently outperforms existing defenses, achieving state-of-the-art performance against a wide range of MIAs. Notably, our approach remains effective even in challenging regimes involving high-resolution data and high-capacity models, where prior defenses fail to provide adequate protection. The code is available at https://github.com/Chrisqcwx/LoFt .
comment : KDD 2026 Accept
♻ ☆ COACH: Collaborative Agent s for Context ual Highlighting -- A Multi- Agent Framework for Sports Video Analysis AAAI 2026
Tsz-To Wong, Ching-Chun Huang, Hong-Han Shuai
Intelligent sports video analysis demands a comprehensive understanding of temporal context, from micro-level actions to macro-level game strategies. Existing end-to-end models often struggle with this temporal hierarchy, offering solutions that lack generalization, incur high development costs for new tasks, and suffer from poor interpretability. To overcome these limitations, we propose a reconfigurable Multi-Agent System (MAS) as a foundational framework for sports video understanding. In our system, each agent functions as a distinct "cognitive tool" specializing in a specific aspect of analysis. The system's architecture is not confined to a single temporal dimension or task. By leveraging iterative invocation and flexible composition of these agents, our framework can construct adaptive pipelines for both short-term analytic reasoning (e.g., Rally QA) and long-term generative summarization (e.g., match summaries). We demonstrate the adaptability of this framework using two representative tasks in badminton analysis, showcasing its ability to bridge fine-grained event detection and global semantic organization. This work presents a paradigm shift towards a flexible, scalable, and interpretable system for robust, cross-task sports video intelligence. The project homepage is available at https://aiden1020.github.io/COACH-project-page
comment : Accepted by AAAI 2026 Workshop LaMAS
♻ ☆ OmniBench: Towards The Future of Universal Omni-Language Models
Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang, Chenghua Lin
Recent advancements in multimodal large language models (MLLMs) have focused on integrating multiple modalities, yet their ability to simultaneously process and reason across different inputs remains underexplored. We introduce OmniBench, a novel benchmark designed to evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. We define language models capable of such tri-modal processing as omni-language models (OLMs). OmniBench features high-quality human annotations that require integrated understanding across all modalities. Our evaluation reveals that: i) open-source OLMs show significant limitations in instruction-following and reasoning in tri-modal contexts; and ii) most baseline models perform poorly (around 50% accuracy) even with textual alternatives to image/audio inputs. To address these limitations, we develop OmniInstruct, an 96K-sample instruction tuning dataset for training OLMs. We advocate for developing more robust tri-modal integration techniques and training strategies to enhance OLM performance. Codes and data could be found at our repo (https://github.com/multimodal-art-projection/OmniBench).
♻ ☆ ViscNet: Vision -Based In-line Viscometry for Fluid Mixing Process
Jongwon Sohn, Juhyeon Moon, Hyunjoon Jung, Jaewook Nam
Viscosity measurement is essential for process monitoring and autonomous laboratory operation, yet conventional viscometers remain invasive and require controlled laboratory environments that differ substantially from real process conditions. We present a computer-vision-based viscometer that infers viscosity by exploiting how a fixed background pattern becomes optically distorted as light refracts through the mixing-driven, continuously deforming free surface. Under diverse lighting conditions, the system achieves a mean absolute error of 0.113 in log m2 s^-1 units for regression and reaches up to 81% accuracy in viscosity-class prediction. Although performance declines for classes with closely clustered viscosity values, a multi-pattern strategy improves robustness by providing enriched visual cues. To ensure sensor reliability, we incorporate uncertainty quantification, enabling viscosity predictions with confidence estimates. This stand-off viscometer offers a practical, automation-ready alternative to existing viscometry methods.
♻ ☆ Diffusion Model in Latent Space for Medical Image Segmentation Task
Huynh Trinh Ngoc, Toan Nguyen Hai, Ba Luong Son, Long Tran Quoc
Medical image segmentation is crucial for clinical diagnosis and treatment planning. Traditional methods typically produce a single segmentation mask, failing to capture inherent uncertainty. Recent generative models enable the creation of multiple plausible masks per image, mimicking the collaborative interpretation of several clinicians. However, these approaches remain computationally heavy. We propose MedSegLatDiff, a diffusion based framework that combines a variational autoencoder (VAE) with a latent diffusion model for efficient medical image segmentation. The VAE compresses the input into a low dimensional latent space, reducing noise and accelerating training, while the diffusion process operates directly in this compact representation. We further replace the conventional MSE loss with weighted cross entropy in the VAE mask reconstruction path to better preserve tiny structures such as small nodules. MedSegLatDiff is evaluated on ISIC-2018 (skin lesions), CVC-Clinic (polyps), and LIDC-IDRI (lung nodules). It achieves state of the art or highly competitive Dice and IoU scores while simultaneously generating diverse segmentation hypotheses and confidence maps. This provides enhanced interpretability and reliability compared to deterministic baselines, making the model particularly suitable for clinical deployment.
♻ ☆ Mixture of Ranks with Degradation-Aware Routing for One-Step Real-Wo rl d Image Super-Resolution AAAI 2026
Xiao He, Zhijun Tu, Kun Cheng, Mingrui Zhu, Jie Hu, Nannan Wang, Xinbo Gao
The demonstrated success of sparsely-gated Mixture-of-Experts (MoE) architectures, exemplified by models such as DeepSeek and Grok, has motivated researchers to investigate their adaptation to diverse domains. In real-world image super-resolution (Real-ISR), existing approaches mainly rely on fine-tuning pre-trained diffusion models through Low-Rank Adaptation (LoRA) module to reconstruct high-resolution (HR) images. However, these dense Real-ISR models are limited in their ability to adaptively capture the heterogeneous characteristics of complex real-world degraded samples or enable knowledge sharing between inputs under equivalent computational budgets. To address this, we investigate the integration of sparse MoE into Real-ISR and propose a Mixture-of-Ranks (MoR) architecture for single-step image super-resolution. We introduce a fine-grained expert partitioning strategy that treats each rank in LoRA as an independent expert. This design enables flexible knowledge recombination while isolating fixed-position ranks as shared experts to preserve common-sense features and minimize routing redundancy. Furthermore, we develop a degradation estimation module leveraging CLIP embeddings and predefined positive-negative text pairs to compute relative degradation scores, dynamically guiding expert activation. To better accommodate varying sample complexities, we incorporate zero-expert slots and propose a degradation-aware load-balancing loss, which dynamically adjusts the number of active experts based on degradation severity, ensuring optimal computational resource allocation. Comprehensive experiments validate our framework's effectiveness and state-of-the-art performance.
comment : 16 pages, Accepted by AAAI 2026, v2: corrected typos
♻ ☆ ContourDiff: Unpaired Medical Image Translation with Structural Consistency
Yuwen Chen, Nicholas Konz, Hanxue Gu, Haoyu Dong, Yaqian Chen, Lin Li, Jisoo Lee, Maciej A. Mazurowski
Accurately translating medical images between different modalities, such as Computed Tomography (CT) to Magnetic Resonance Imaging (MRI), has numerous downstream clinical and machine learning applications. While several methods have been proposed to achieve this, they often prioritize perceptual quality with respect to output domain features over preserving anatomical fidelity. However, maintaining anatomy during translation is essential for many tasks, e.g., when leveraging masks from the input domain to develop a segmentation model with images translated to the output domain. To address these challenges, we propose ContourDiff with Spatially Coherent Guided Diffusion (SCGD), a novel framework that leverages domain-invariant anatomical contour representations of images. These representations are simple to extract from images, yet form precise spatial constraints on their anatomical content. We introduce a diffusion model that converts contour representations of images from arbitrary input domains into images in the output domain of interest. By applying the contour as a constraint at every diffusion sampling step, we ensure the preservation of anatomical content. We evaluate our method on challenging lumbar spine and hip-and-thigh CT-to-MRI translation tasks, via (1) the performance of segmentation models trained on translated images applied to real MRIs, and (2) the foreground FID and KID of translated images with respect to real MRIs. Our method outperforms other unpaired image translation methods by a significant margin across almost all metrics and scenarios. Moreover, it achieves this without the need to access any input domain information during training and we further verify its zero-shot capability, showing that a model trained on one anatomical region can be directly applied to unseen regions without retraining (GitHub: https://github.com/mazurowski-lab/ContourDiff).
comment : Accepted for publication at the Journal of Machine Learning for Biomedical Imaging (MELBA) https://melba-journal.org/2025:031
♻ ☆ Learning Massively Multitask Wo rl d Models for Continuous Control
Nicklas Hansen, Hao Su, Xiaolong Wang
General-purpose control demands agents that act across many tasks and embodiments, yet research on reinforcement learning (RL) for continuous control remains dominated by single-task or offline regimes, reinforcing a view that online RL does not scale. Inspired by the foundation model recipe (large-scale pretraining followed by light RL) we ask whether a single agent can be trained on hundreds of tasks with online interaction. To accelerate research in this direction, we introduce a new benchmark with 200 diverse tasks spanning many domains and embodiments, each with language instructions, demonstrations, and optionally image observations. We then present \emph{Newt}, a language-conditioned multitask world model that is first pretrained on demonstrations to acquire task-aware representations and action priors, and then jointly optimized with online interaction across all tasks. Experiments show that Newt yields better multitask performance and data-efficiency than a set of strong baselines, exhibits strong open-loop control, and enables rapid adaptation to unseen tasks. We release our environments, demonstrations, code for training and evaluation, as well as 200+ checkpoints.
comment : Webpage: https://www.nicklashansen.com/NewtWM
♻ ☆ Image-Based Relocalization and Alignment for Long-Term Monitoring of Dynamic Underwater Environments
Beverley Gorry, Tobias Fischer, Michael Milford, Alejandro Fontan
Effective monitoring of underwater ecosystems is crucial for tracking environmental changes, guiding conservation efforts, and ensuring long-term ecosystem health. However, automating underwater ecosystem management with robotic platforms remains challenging due to the complexities of underwater imagery, which pose significant difficulties for traditional visual localization methods. We propose an integrated pipeline that combines Visual Place Recognition (VPR), feature matching, and image segmentation on video-derived images. This method enables robust identification of revisited areas, estimation of rigid transformations, and downstream analysis of ecosystem changes. Furthermore, we introduce the SQUIDLE+ VPR Benchmark-the first large-scale underwater VPR benchmark designed to leverage an extensive collection of unstructured data from multiple robotic platforms, spanning time intervals from days to years. The dataset encompasses diverse trajectories, arbitrary overlap and diverse seafloor types captured under varying environmental conditions, including differences in depth, lighting, and turbidity. Our code is available at: https://github.com/bev-gorry/underloc
♻ ☆ Rainbow Noise: Stress-Testing Multimodal Harmful-Meme Detectors on LGBTQ Content
Ran Tong, Songtao Wei, Jiaqi Liu, Lanruo Wang
Hateful memes aimed at LGBTQ\,+ communities often evade detection by tweaking either the caption, the image, or both. We build the first robustness benchmark for this setting, pairing four realistic caption attacks with three canonical image corruptions and testing all combinations on the PrideMM dataset. Two state-of-the-art detectors, MemeCLIP and MemeBLIP2, serve as case studies, and we introduce a lightweight \textbf{Text Denoising Adapter (TDA)} to enhance the latter's resilience. Across the grid, MemeCLIP degrades more gently, while MemeBLIP2 is particularly sensitive to the caption edits that disrupt its language processing. However, the addition of the TDA not only remedies this weakness but makes MemeBLIP2 the most robust model overall. Ablations reveal that all systems lean heavily on text, but architectural choices and pre-training data significantly impact robustness. Our benchmark exposes where current multimodal safety models crack and demonstrates that targeted, lightweight modules like the TDA offer a powerful path towards stronger defences.
comment : 14 pages, 1 figure
♻ ☆ PointCNN++: Performant Convolution on Native Points
Lihan Li, Haofeng Zhong, Rui Bu, Mingchao Sun, Wenzheng Chen, Baoquan Chen, Yangyan Li
Existing convolutional learning methods for 3D point cloud data are divided into two paradigms: point-based methods that preserve geometric precision but often face performance challenges, and voxel-based methods that achieve high efficiency through quantization at the cost of geometric fidelity. This loss of precision is a critical bottleneck for tasks such as point cloud registration. We propose PointCNN++, a novel architectural design that fundamentally mitigates this precision-performance trade-off. It $\textbf{generalizes sparse convolution from voxels to points}$, treating voxel-based convolution as a specialized, degraded case of our more general point-based convolution. First, we introduce a point-centric convolution where the receptive field is centered on the original, high-precision point coordinates. Second, to make this high-fidelity operation performant, we design a computational strategy that operates $\textbf{natively}$ on points. We formulate the convolution on native points as a Matrix-Vector Multiplication and Reduction (MVMR) problem, for which we develop a dedicated, highly-optimized GPU kernel. Experiments demonstrate that PointCNN++ $\textbf{uses an order of magnitude less memory and is several times faster}$ than representative point-based methods. Furthermore, when used as a simple replacement for the voxel-based backbones it generalizes, it $\textbf{significantly improves point cloud registration accuracies while proving both more memory-efficient and faster}$. PointCNN++ shows that preserving geometric detail and achieving high performance are not mutually exclusive, paving the way for a new class of 3D learning with high fidelity and efficiency. Our code will be open sourced.
♻ ☆ Otter: Mitigating Background Distractions of Wide-Angle Few-Shot Action Recognition with Enhanced RWKV AAAI 2026
Wenbo Huang, Jinghui Zhang, Zhenghao Chen, Guang Li, Lei Zhang, Yang Cao, Fang Dong, Takahiro Ogawa, Miki Haseyama
Wide-angle videos in few-shot action recognition (FSAR) effectively express actions within specific scenarios. However, without a global understanding of both subjects and background, recognizing actions in such samples remains challenging because of the background distractions. Receptance Weighted Key Value (RWKV), which learns interaction between various dimensions, shows promise for global modeling. While directly applying RWKV to wide-angle FSAR may fail to highlight subjects due to excessive background information. Additionally, temporal relation degraded by frames with similar backgrounds is difficult to reconstruct, further impacting performance. Therefore, we design the CompOund SegmenTation and Temporal REconstructing RWKV (Otter). Specifically, the Compound Segmentation Module~(CSM) is devised to segment and emphasize key patches in each frame, effectively highlighting subjects against background information. The Temporal Reconstruction Module (TRM) is incorporated into the temporal-enhanced prototype construction to enable bidirectional scanning, allowing better reconstruct temporal relation. Furthermore, a regular prototype is combined with the temporal-enhanced prototype to simultaneously enhance subject emphasis and temporal modeling, improving wide-angle FSAR performance. Extensive experiments on benchmarks such as SSv2, Kinetics, UCF101, and HMDB51 demonstrate that Otter achieves state-of-the-art performance. Extra evaluation on the VideoBadminton dataset further validates the superiority of Otter in wide-angle FSAR.
comment : Accepted by AAAI 2026 Oral
♻ ☆ LiDARCrafter: Dynamic 4D Wo rl d Modeling from LiDAR Sequences AAAI 2026
Ao Liang, Youquan Liu, Yu Yang, Dongyue Lu, Linfeng Li, Lingdong Kong, Huaici Zhao, Wei Tsang Ooi
Generative world models have become essential data engines for autonomous driving, yet most existing efforts focus on videos or occupancy grids, overlooking the unique LiDAR properties. Extending LiDAR generation to dynamic 4D world modeling presents challenges in controllability, temporal coherence, and evaluation standardization. To this end, we present LiDARCrafter, a unified framework for 4D LiDAR generation and editing. Given free-form natural language inputs, we parse instructions into ego-centric scene graphs, which condition a tri-branch diffusion network to generate object structures, motion trajectories, and geometry. These structured conditions enable diverse and fine-grained scene editing. Additionally, an autoregressive module generates temporally coherent 4D LiDAR sequences with smooth transitions. To support standardized evaluation, we establish a comprehensive benchmark with diverse metrics spanning scene-, object-, and sequence-level aspects. Experiments on the nuScenes dataset using this benchmark demonstrate that LiDARCrafter achieves state-of-the-art performance in fidelity, controllability, and temporal consistency across all levels, paving the way for data augmentation and simulation. The code and benchmark are released to the community.
comment : AAAI 2026 Oral Presentation; 38 pages, 18 figures, 12 tables; Project Page at https://lidarcrafter.github.io
♻ ☆ Convolution goes higher-order: a biologically inspired mechanism empowers image classification
Simone Azeglio, Olivier Marre, Peter Neri, Ulisse Ferrari
We propose a novel approach to image classification inspired by complex nonlinear biological visual processing, whereby classical convolutional neural networks (CNNs) are equipped with learnable higher-order convolutions. Our model incorporates a Volterra-like expansion of the convolution operator, capturing multiplicative interactions akin to those observed in early and advanced stages of biological visual processing. We evaluated this approach on synthetic datasets by measuring sensitivity to testing higher-order correlations and performance in standard benchmarks (MNIST, FashionMNIST, CIFAR10, CIFAR100 and Imagenette). Our architecture outperforms traditional CNN baselines, and achieves optimal performance with expansions up to 3rd/4th order, aligning remarkably well with the distribution of pixel intensities in natural images. Through systematic perturbation analysis, we validate this alignment by isolating the contributions of specific image statistics to model performance, demonstrating how different orders of convolution process distinct aspects of visual information. Furthermore, Representational Similarity Analysis reveals distinct geometries across network layers, indicating qualitatively different modes of visual information processing. Our work bridges neuroscience and deep learning, offering a path towards more effective, biologically inspired computer vision models. It provides insights into visual information processing and lays the groundwork for neural networks that better capture complex visual patterns, particularly in resource-constrained scenarios.
♻ ☆ Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos
Xavier Thomas, Youngsun Lim, Ananya Srinivasan, Audrey Zheng, Deepti Ghadiyaram
Despite rapid advances in video generative models, robust metrics for evaluating visual and temporal correctness of complex human actions remain elusive. Critically, existing pure-vision encoders and Multimodal Large Language Models (MLLMs) are strongly appearance-biased, lack temporal understanding, and thus struggle to discern intricate motion dynamics and anatomical implausibilities in generated videos. We tackle this gap by introducing a novel evaluation metric derived from a learned latent space of real-world human actions. Our method first captures the nuances, constraints, and temporal smoothness of real-world motion by fusing appearance-agnostic human skeletal geometry features with appearance-based features. We posit that this combined feature space provides a robust representation of action plausibility. Given a generated video, our metric quantifies its action quality by measuring the distance between its underlying representations and this learned real-world action distribution. For rigorous validation, we develop a new multi-faceted benchmark specifically designed to probe temporally challenging aspects of human action fidelity. Through extensive experiments, we show that our metric achieves substantial improvement of more than 68% compared to existing state-of-the-art methods on our benchmark, performs competitively on established external benchmarks, and has a stronger correlation with human perception. Our in-depth analysis reveals critical limitations in current video generative models and establishes a new standard for advanced research in video generation.
♻ ☆ Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
Tianle Chen, Chaitanya Chakka, Arjun Reddy Akula, Xavier Thomas, Deepti Ghadiyaram
Despite remarkable advancements in Multimodal Large Language Models (MLLMs), a fundamental question remains: are MLLMs robust to contradicting modalities? To rigorously study this, we introduce MMA-Bench comprising videos and tasks that probe a model's reliance on specific modalities. Using black-box and white-box interpretability techniques, we provide a critical analysis of the brittleness of both open- and closed-sourced MLLMs. We show that current MLLMs struggle under misaligned audio-visual pairs and simple misleading text, thereby lacking robust multi-modal reasoning. Building on these findings, we propose a modality alignment tuning strategy to teach the model when to prioritize, leverage, or ignore specific modality cues. Through extensive experiments and analysis, we show that our alignment tuning yields demonstrably stronger multimodal grounding. This work provides both interpretability tools and a clear path toward developing MLLMs with intrinsically reliable cross-modal reasoning. Code and dataset will be publicly available.
♻ ☆ Multilingual Training-Free Remote Sensing Image Captioning
Carlos Rebelo, Gil Rocha, João Daniel Silva, Bruno Martins
Remote sensing image captioning has advanced rapidly through encoder--decoder models, although the reliance on large annotated datasets and the focus on English restricts global applicability. To address these limitations, we propose the first training-free multilingual approach, based on retrieval-augmented prompting. For a given aerial image, we employ a domain-adapted SigLIP2 encoder to retrieve related captions and few-shot examples from a datastore, which are then provided to a language model. We explore two variants: an image-blind setup, where a multilingual Large Language Model (LLM) generates the caption from textual prompts alone, and an image-aware setup, where a Vision--Language Model (VLM) jointly processes the prompt and the input image. To improve the coherence of the retrieved content, we introduce a graph-based re-ranking strategy using PageRank on a graph of images and captions. Experiments on four benchmark datasets across ten languages demonstrate that our approach is competitive with fully supervised English-only systems and generalizes to other languages. Results also highlight the importance of re-ranking with PageRank, yielding up to 35% improvements in performance metrics. Additionally, it was observed that while VLMs tend to generate visually grounded but lexically diverse captions, LLMs can achieve stronger BLEU and CIDEr scores. Lastly, directly generating captions in the target language consistently outperforms other translation-based strategies. Overall, our work delivers one of the first systematic evaluations of multilingual, training-free captioning for remote sensing imagery, advancing toward more inclusive and scalable multimodal Earth observation systems.
♻ ☆ Beyond Top Activations: Efficient and Reliable Crowdsourced Evaluation of Automated Interpretability
Tuomas Oikarinen, Ge Yan, Akshay Kulkarni, Tsui-Wei Weng
Interpreting individual neurons or directions in activation space is an important topic in mechanistic interpretability. Numerous automated interpretability methods have been proposed to generate such explanations, but it remains unclear how reliable these explanations are, and which methods produce the most accurate descriptions. While crowd-sourced evaluations are commonly used, existing pipelines are noisy, costly, and typically assess only the highest-activating inputs, leading to unreliable results. In this paper, we introduce two techniques to enable cost-effective and accurate crowdsourced evaluation of automated interpretability methods beyond top activating inputs. First, we propose Model-Guided Importance Sampling (MG-IS) to select the most informative inputs to show human raters. In our experiments, we show this reduces the number of inputs needed to reach the same evaluation accuracy by ~13x. Second, we address label noise in crowd-sourced ratings through Bayesian Rating Aggregation (BRAgg), which allows us to reduce the number of ratings per input required to overcome noise by ~3x. Together, these techniques reduce the evaluation cost by ~40x, making large-scale evaluation feasible. Finally, we use our methods to conduct a large scale crowd-sourced study comparing recent automated interpretability methods for vision networks.
♻ ☆ All You Need for Object Detection: From Pixels, Points, and Prompt s to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehicles
Sayed Pedram Haeri Boroujeni, Niloufar Mehrabi, Hazim Alzorgan, Mahlagha Fazeli, Abolfazl Razi
Autonomous Vehicles (AVs) are transforming the future of transportation through advances in intelligent perception, decision-making, and control systems. However, their success is tied to one core capability, reliable object detection in complex and multimodal environments. While recent breakthroughs in Computer Vision (CV) and Artificial Intelligence (AI) have driven remarkable progress, the field still faces a critical challenge as knowledge remains fragmented across multimodal perception, contextual reasoning, and cooperative intelligence. This survey bridges that gap by delivering a forward-looking analysis of object detection in AVs, emphasizing emerging paradigms such as Vision-Language Models (VLMs), Large Language Models (LLMs), and Generative AI rather than re-examining outdated techniques. We begin by systematically reviewing the fundamental spectrum of AV sensors (camera, ultrasonic, LiDAR, and Radar) and their fusion strategies, highlighting not only their capabilities and limitations in dynamic driving environments but also their potential to integrate with recent advances in LLM/VLM-driven perception frameworks. Next, we introduce a structured categorization of AV datasets that moves beyond simple collections, positioning ego-vehicle, infrastructure-based, and cooperative datasets (e.g., V2V, V2I, V2X, I2I), followed by a cross-analysis of data structures and characteristics. Ultimately, we analyze cutting-edge detection methodologies, ranging from 2D and 3D pipelines to hybrid sensor fusion, with particular attention to emerging transformer-driven approaches powered by Vision Transformers (ViTs), Large and Small Language Models (SLMs), and VLMs. By synthesizing these perspectives, our survey delivers a clear roadmap of current capabilities, open challenges, and future opportunities.
♻ ☆ Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization ICCV 2025
Kyle Sargent, Kyle Hsu, Justin Johnson, Li Fei-Fei, Jiajun Wu
Since the advent of popular visual generation frameworks like VQGAN and latent diffusion models, state-of-the-art image generation systems have generally been two-stage systems that first tokenize or compress visual data into a lower-dimensional latent space before learning a generative model. Tokenizer training typically follows a standard recipe in which images are compressed and reconstructed subject to a combination of MSE, perceptual, and adversarial losses. Diffusion autoencoders have been proposed in prior work as a way to learn end-to-end perceptually-oriented image compression, but have not yet shown state-of-the-art performance on the competitive task of ImageNet-1K reconstruction. We propose FlowMo, a transformer-based diffusion autoencoder that achieves a new state-of-the-art for image tokenization at multiple compression rates without using convolutions, adversarial losses, spatially-aligned two-dimensional latent codes, or distilling from other tokenizers. Our key insight is that FlowMo training should be broken into a mode-matching pre-training stage and a mode-seeking post-training stage. In addition, we conduct extensive analyses and explore the training of generative models atop the FlowMo tokenizer. Our code and models will be available at http://kylesargent.github.io/flowmo .
comment : ICCV 2025, 19 pages
♻ ☆ From Pixels to Prose: Advancing Multi-Modal Language Models for Remote Sensing
Xintian Sun, Benji Peng, Charles Zhang, Fei Jin, Qian Niu, Junyu Liu, Keyu Chen, Ming Li, Pohsun Feng, Ziqian Bi, Ming Liu, Xinyuan Song, Yichao Zhang
Remote sensing has evolved from simple image acquisition to complex systems capable of integrating and processing visual and textual data. This review examines the development and application of multi-modal language models (MLLMs) in remote sensing, focusing on their ability to interpret and describe satellite imagery using natural language. We cover the technical underpinnings of MLLMs, including dual-encoder architectures, Transformer models, self-supervised and contrastive learning, and cross-modal integration. The unique challenges of remote sensing data--varying spatial resolutions, spectral richness, and temporal changes--are analyzed for their impact on MLLM performance. Key applications such as scene description, object detection, change detection, text-to-image retrieval, image-to-text generation, and visual question answering are discussed to demonstrate their relevance in environmental monitoring, urban planning, and disaster response. We review significant datasets and resources supporting the training and evaluation of these models. Challenges related to computational demands, scalability, data quality, and domain adaptation are highlighted. We conclude by proposing future research directions and technological advancements to further enhance MLLM utility in remote sensing.
comment : 10 pages, 1 figure
♻ ☆ UniEdit-I: Training-free Image Editing for Unified VLM via Iterative Understanding , Editing and Verifying
Chengyu Bai, Jintao Chen, Xiang Bai, Yilong Chen, Qi She, Ming Lu, Shanghang Zhang
While Unified Vision-Language Models promise to synergistically combine the high-level semantic understanding of vision-language models with the generative fidelity of diffusion models, current editing methodologies remain fundamentally decoupled and open loop performing static, pre-defined transformations without dynamic feedback between semantic interpretation and visual generation. A central limitation stems from the representation gap: understanding typically leverages high-level, language aligned encoders, whereas generation relies on low level, pixel-space autoencoders, resulting in misaligned feature spaces. To bridge this gap, Recent advances such as Representation Autoencoders and BLIP3-o advocate performing diffusion-based modeling directly in high level features from pretrained semantic encoders. We find editing in the semantic latent space modifies conceptual representations rather than pixels, ensuring intermediates that are both semantically coherent and visually plausible. Building on this insight, We propose UniEdit-I, the first training-free, closed-loop image editing framework that operates entirely within the semantic latent space of a unified VLM by introducing an Understanding-Editing-Verifying (UEV) loop, By transforming the VLM from a posthoc evaluator into an in-process conductor, UniEdit-I establishes the first semantics-driven, self-correcting closed-loop image editing pipeline. Evaluated on GEdit-Bench, UniEdit-I achieves state of the art performance without any fine tuning or architectural modifications, and even surpasses several largescale pretrained editors.
Computation and Language 98
☆ The Moral Consistency Pipeline: Continuous Ethical Evaluation for Large Language Models
Saeid Jamshidi, Kawser Wazed Nafi, Arghavan Moradi Dakhel, Negar Shahabi, Foutse Khomh
The rapid advancement and adaptability of Large Language Models (LLMs) highlight the need for moral consistency, the capacity to maintain ethically coherent reasoning across varied contexts. Existing alignment frameworks, structured approaches designed to align model behavior with human ethical and social norms, often rely on static datasets and post-hoc evaluations, offering limited insight into how ethical reasoning may evolve across different contexts or temporal scales. This study presents the Moral Consistency Pipeline (MoCoP), a dataset-free, closed-loop framework for continuously evaluating and interpreting the moral stability of LLMs. MoCoP combines three supporting layers: (i) lexical integrity analysis, (ii) semantic risk estimation, and (iii) reasoning-based judgment modeling within a self-sustaining architecture that autonomously generates, evaluates, and refines ethical scenarios without external supervision. Our empirical results on GPT-4-Turbo and DeepSeek suggest that MoCoP effectively captures longitudinal ethical behavior, revealing a strong inverse relationship between ethical and toxicity dimensions (correlation rET = -0.81, p value less than 0.001) and a near-zero association with response latency (correlation rEL approximately equal to 0). These findings demonstrate that moral coherence and linguistic safety tend to emerge as stable and interpretable characteristics of model behavior rather than short-term fluctuations. Furthermore, by reframing ethical evaluation as a dynamic, model-agnostic form of moral introspection, MoCoP offers a reproducible foundation for scalable, continuous auditing and advances the study of computational morality in autonomous AI systems.
☆ LORE: A Large Generative Model for Search Relevance
Chenji Lu, Zhuo Chen, Hui Zhao, Zhiyuan Zeng, Gang Zhao, Junjie Ren, Ruicong Xu, Haoran Li, Songyan Liu, Pengjie Wang, Jian Xu, Bo Zheng
Achievement. We introduce LORE, a systematic framework for Large Generative Model-based relevance in e-commerce search. Deployed and iterated over three years, LORE achieves a cumulative +27\% improvement in online GoodRate metrics. This report shares the valuable experience gained throughout its development lifecycle, spanning data, features, training, evaluation, and deployment. Insight. While existing works apply Chain-of-Thought (CoT) to enhance relevance, they often hit a performance ceiling. We argue this stems from treating relevance as a monolithic task, lacking principled deconstruction. Our key insight is that relevance comprises distinct capabilities: knowledge and reasoning, multi-modal matching, and rule adherence. We contend that a qualitative-driven decomposition is essential for breaking through current performance bottlenecks. Contributions. LORE provides a complete blueprint for the LLM relevance lifecycle. Key contributions include: (1) A two-stage training paradigm combining progressive CoT synthesis via SFT with human preference alignment via RL. (2) A comprehensive benchmark, RAIR, designed to evaluate these core capabilities. (3) A query frequency-stratified deployment strategy that efficiently transfers offline LLM capabilities to the online system. LORE serves as both a practical solution and a methodological reference for other vertical domains.
☆ Fine-Tuned Large Language Models for Logical Translation: Reducing Hallucinations with Lang2Logic
Muyu Pan, Dheeraj Kodakandla, Mahfuza Farooque
Recent advances in natural language processing (NLP), particularly large language models (LLMs), have motivated the automatic translation of natural language statements into formal logic without human intervention. This enables automated reasoning and facilitates debugging, finding loop invariants, and adhering to specifications in software systems. However, hallucinations-incorrect outputs generated by LLMs are challenging, particularly for logical translation tasks requiring precision. This work introduces a novel framework that inputs English sentences, converts them into logical expressions, and then translates them into Conjunctive Normal Form (CNF) for satisfiability solving. It employs classical NLP techniques with self-defined grammar, symbolic computation libraries, and a fine-tuned language model to reduce hallucinations. In the early experiments, we observed that the fine-tuned model, trained on different grammar settings, could intentionally correct the same types of hallucinations made by the original model. Thus, it provides reliable CNF generation.
comment : IEEE ISNCC 2025
☆ Context ual Image Attack: How Visual Context Exposes Multimodal Safety Vulnerabilities
Yuan Xiong, Ziqi Miao, Lijun Li, Chen Qian, Jie Li, Jing Shao
While Multimodal Large Language Models (MLLMs) show remarkable capabilities, their safety alignments are susceptible to jailbreak attacks. Existing attack methods typically focus on text-image interplay, treating the visual modality as a secondary prompt. This approach underutilizes the unique potential of images to carry complex, contextual information. To address this gap, we propose a new image-centric attack method, Contextual Image Attack (CIA), which employs a multi-agent system to subtly embeds harmful queries into seemingly benign visual contexts using four distinct visualization strategies. To further enhance the attack's efficacy, the system incorporate contextual element enhancement and automatic toxicity obfuscation techniques. Experimental results on the MMSafetyBench-tiny dataset show that CIA achieves high toxicity scores of 4.73 and 4.83 against the GPT-4o and Qwen2.5-VL-72B models, respectively, with Attack Success Rates (ASR) reaching 86.31\% and 91.07\%. Our method significantly outperforms prior work, demonstrating that the visual modality itself is a potent vector for jailbreaking advanced MLLMs.
☆ AutoNeural: Co-Designing Vision -Language Models for NPU Inference
Wei Chen, Liangmin Wu, Yunhai Hu, Zhiyuan Li, Zhiyuan Cheng, Yicheng Qian, Lingyue Zhu, Zhipeng Hu, Luoyi Liang, Qiang Tang, Zhen Liu, Han Yang
While Neural Processing Units (NPUs) offer high theoretical efficiency for edge AI, state-of-the-art Vision--Language Models (VLMs) tailored for GPUs often falter on these substrates. We attribute this hardware-model mismatch to two primary factors: the quantization brittleness of Vision Transformers (ViTs) and the I/O-bound nature of autoregressive attention mechanisms, which fail to utilize the high arithmetic throughput of NPUs. To bridge this gap, we propose AutoNeural, an NPU-native VLM architecture co-designed for integer-only inference. We replace the standard ViT encoder with a MobileNetV5-style backbone utilizing depthwise separable convolutions, which ensures bounded activation distributions for stable INT4/8/16 quantization. Complementing this, our language backbone integrates State-Space Model (SSM) principles with Transformer layers, employing efficient gated convolutions to achieve linear-time complexity. This hybrid design eliminates the heavy memory I/O overhead of Key-Value caching during generation. Our approach delivers substantial efficiency gains, reducing quantization error of vision encoder by up to 7x and end-to-end latency by 14x compared to conventional baselines. The AutoNeural also delivers 3x decoding speed and 4x longer context window than the baseline. We validate these improvements via a real-world automotive case study on the Qualcomm SA8295P SoC, demonstrating real-time performance for cockpit applications. Our results highlight that rethinking model topology specifically for NPU constraints is a prerequisite for robust multi-modal edge intelligence.
☆ Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning NeurIPS 2025
Zhonghao He, Tianyi Qiu, Hirokazu Shirado, Maarten Sap
Recent advances in reasoning techniques have substantially improved the performance of large language models (LLMs), raising expectations for their ability to provide accurate, truthful, and reliable information. However, emerging evidence suggests that iterative reasoning may foster belief entrenchment and confirmation bias, rather than enhancing truth-seeking behavior. In this study, we propose a systematic evaluation framework for belief entrenchment in LLM reasoning by leveraging the Martingale property from Bayesian statistics. This property implies that, under rational belief updating, the expected value of future beliefs should remain equal to the current belief, i.e., belief updates are unpredictable from the current belief. We propose the unsupervised, regression-based Martingale Score to measure violations of this property, which signal deviation from the Bayesian ability of updating on new evidence. In open-ended problem domains including event forecasting, value-laden questions, and academic paper review, we find such violations to be widespread across models and setups, where the current belief positively predicts future belief updates, a phenomenon which we term belief entrenchment. We identify the models, reasoning techniques, and domains more prone to belief entrenchment. Finally, we validate the Martingale Score by showing that it predicts ground-truth accuracy on problem domains where ground truth labels are available. This indicates that, while designed as an unsupervised metric that operates even in domains without access to ground truth, the Martingale Score is a useful proxy of the truth-seeking ability of a reasoning process.
comment : NeurIPS 2025
☆ Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules
Amr Mohamed, Yang Zhang, Michalis Vazirgiannis, Guokan Shang
Diffusion large language models (dLLMs) offer a promising alternative to autoregressive models, but their practical utility is severely hampered by slow, iterative sampling. We present SchED, a training-free, model-agnostic early-exit algorithm that aggregates full-span logit margins and halts decoding once a smooth, progress-dependent confidence threshold is met. We evaluated SchED on two dLLM families (Dream and LLaDA), in base and instruction-tuned variants across ten benchmarks spanning downstream tasks including multiple-choice question answering (MCQ), math, long-form QA/summarization, and translation. SchED delivers large, stable accelerations: on instruction-tuned models, it achieves $3.8$-$4.0\times$ speedups while retaining $99.8$-$100\%$ of the baseline score on average. On base models, SchED yields consistent speedup gains with $99.1$-$100\%$ performance retention, with up to $2.34\times$ under more aggressive settings. Using a conservative speed metric that heavily penalizes quality loss (QPS, $γ{=}4$), we show that SchED is robust and clearly outperforms prior confidence-based early-exit methods, which break down on long-form generation. An entropy analysis of the model's token predictions reveals that instruction tuning speeds up the decay of predictive entropy. By turning genuine confidence stabilization into computational savings, SchED makes dLLM decoding substantially more efficient.
☆ OptPO: Optimal Rollout Allocation for Test-time Policy Optimization
Youkang Wang, Jian Wang, Rubing Chen, Tianyi Zeng, Xiao-Yong Wei, Qing Li
Test-time policy optimization enables large language models (LLMs) to adapt to distribution shifts by leveraging feedback from self-generated rollouts. However, existing methods rely on fixed-budget majority voting to estimate rewards, incurring substantial computational redundancy. We propose Optimal Rollout Allocation for Test-time Policy Optimization (OptPO), a principled framework that adaptively allocates inference budgets. By formulating the voting process as a Bayesian sequential probability ratio test, OptPO dynamically halts sampling once the posterior confidence in a consensus answer exceeds a specified threshold. Crucially, it utilizes the retained rollouts for on-policy updates, seamlessly integrating with algorithms like PPO or GRPO without requiring ground-truth labels. Across diverse reasoning benchmarks, OptPO significantly reduces rollout overhead compared to fixed-sample baselines while preserving or improving accuracy. By unifying statistically optimal stopping with test-time learning, OptPO offers a computationally efficient paradigm for test-time adaptation. The source code will be open upon acceptance at https://open-upon-acceptance.
comment : Work in Progress
☆ Think in Parallel, Answer as One: Logit Averaging for Open-Ended Reasoning
Haonan Wang, Chao Du, Kenji Kawaguchi, Tianyu Pang
Majority voting has proven effective for close-ended question answering by aggregating parallel reasoning traces. However, it is not directly applicable to open-ended reasoning, such as code generation and web-based deep research, where a "majority" over complete solutions is ill-defined. We introduce ThinkMerge, a training-free, plug-and-play decoding strategy that runs K parallel reasoning traces and averages their next-token logits at synchronization points to produce a single coherent output. ThinkMerge integrates seamlessly with vLLM/SGLang and remains compatible with standard decoding techniques such as Top-p/Top-k. Empirically, it matches or surpasses majority voting on AIME and GPQA, while delivering consistent gains on open-ended coding tasks: on LiveCodeBench (hard), pass@1 improves by +8.28% for DeepCoder-14B-Preview and +7.58% for Qwen3-8B. Beyond code, we further show that ThinkMerge improves web-based deep-research agents (e.g., WebSailor-7B/32B) across GAIA, BrowseComp-en/zh, and XbenchDeepSearch. These results demonstrate that parallel test-time scaling can benefit open-ended reasoning without relying on voting over complete outputs.
☆ Bangla Hate Speech Classification with Fine-tuned Transformer Models
Yalda Keivan Jafari, Krishno Dey
Hate speech recognition in low-resource languages remains a difficult problem due to insufficient datasets, orthographic heterogeneity, and linguistic variety. Bangla is spoken by more than 230 million people of Bangladesh and India (West Bengal). Despite the growing need for automated moderation on social media platforms, Bangla is significantly under-represented in computational resources. In this work, we study Subtask 1A and Subtask 1B of the BLP 2025 Shared Task on hate speech detection. We reproduce the official baselines (e.g., Majority, Random, Support Vector Machine) and also produce and consider Logistic Regression, Random Forest, and Decision Tree as baseline methods. We also utilized transformer-based models such as DistilBERT, BanglaBERT, m-BERT, and XLM-RoBERTa for hate speech classification. All the transformer-based models outperformed baseline methods for the subtasks, except for DistilBERT. Among the transformer-based models, BanglaBERT produces the best performance for both subtasks. Despite being smaller in size, BanglaBERT outperforms both m-BERT and XLM-RoBERTa, which suggests language-specific pre-training is very important. Our results highlight the potential and need for pre-trained language models for the low-resource Bangla language.
☆ Cross-Lingual Prompt Steerability: Towards Accurate and Robust LLM Behavior across Languages
Lechen Zhang, Yusheng Zhou, Tolga Ergen, Lajanugen Logeswaran, Moontae Lee, David Jurgens
System prompts provide a lightweight yet powerful mechanism for conditioning large language models (LLMs) at inference time. While prior work has focused on English-only settings, real-world deployments benefit from having a single prompt to operate reliably across languages. This paper presents a comprehensive study of how different system prompts steer models toward accurate and robust cross-lingual behavior. We propose a unified four-dimensional evaluation framework to assess system prompts in multilingual environments. Through large-scale experiments on five languages, three LLMs, and three benchmarks, we uncover that certain prompt components, such as CoT, emotion, and scenario, correlate with robust multilingual behavior. We develop a prompt optimization framework for multilingual settings and show it can automatically discover prompts that improve all metrics by 5-10%. Finally, we analyze over 10 million reasoning units and find that more performant system prompts induce more structured and consistent reasoning patterns, while reducing unnecessary language-switching. Together, we highlight system prompt optimization as a scalable path to accurate and robust multilingual LLM behavior.
☆ prompt olution: A Unified, Modular Framework for Prompt Optimization
Tom Zehle, Timo Heiß, Moritz Schlager, Matthias Aßenmacher, Matthias Feurer
Prompt optimization has become crucial for enhancing the performance of large language models (LLMs) across a broad range of tasks. Although many research papers show its effectiveness, practical adoption is hindered as existing implementations are often tied to unmaintained and isolated research codebases. To address this, we introduce promptolution, a unified and modular open-source framework that provides all components required for prompt optimization within a single extensible system for both practitioners and researchers. It integrates multiple contemporary discrete prompt optimizers while remaining agnostic to the underlying LLM implementation.
☆ ReVSeg: Incentivizing the Reasoning Chain for Video Segmentation with Reinforcement Learning
Yifan Li, Yingda Yin, Lingting Zhu, Weikai Chen, Shengju Qian, Xin Wang, Yanwei Fu
Reasoning-centric video object segmentation is an inherently complex task: the query often refers to dynamics, causality, and temporal interactions, rather than static appearances. Yet existing solutions generally collapse these factors into simplified reasoning with latent embeddings, rendering the reasoning chain opaque and essentially intractable. We therefore adopt an explicit decomposition perspective and introduce ReVSeg, which executes reasoning as sequential decisions in the native interface of pretrained vision language models (VLMs). Rather than folding all reasoning into a single-step prediction, ReVSeg executes three explicit operations -- semantics interpretation, temporal evidence selection, and spatial grounding -- aligning pretrained capabilities. We further employ reinforcement learning to optimize the multi-step reasoning chain, enabling the model to self-refine its decision quality from outcome-driven signals. Experimental results demonstrate that ReVSeg attains state-of-the-art performances on standard video object segmentation benchmarks and yields interpretable reasoning trajectories. Project page is available at https://clementine24.github.io/ReVSeg/ .
☆ BOOM: Beyond Only One Modality KIT's Multimodal Multilingual Lecture Companion
Sai Koneru, Fabian Retkowski, Christian Huber, Lukas Hilgert, Seymanur Akti, Enes Yavuz Ugan, Alexander Waibel, Jan Niehues
The globalization of education and rapid growth of online learning have made localizing educational content a critical challenge. Lecture materials are inherently multimodal, combining spoken audio with visual slides, which requires systems capable of processing multiple input modalities. To provide an accessible and complete learning experience, translations must preserve all modalities: text for reading, slides for visual understanding, and speech for auditory learning. We present \textbf{BOOM}, a multimodal multilingual lecture companion that jointly translates lecture audio and slides to produce synchronized outputs across three modalities: translated text, localized slides with preserved visual elements, and synthesized speech. This end-to-end approach enables students to access lectures in their native language while aiming to preserve the original content in its entirety. Our experiments demonstrate that slide-aware transcripts also yield cascading benefits for downstream tasks such as summarization and question answering. We release our Slide Translation code at https://github.com/saikoneru/image-translator and integrate it in Lecture Translator at https://gitlab.kit.edu/kit/isl-ai4lt/lt-middleware/ltpipeline}\footnote{All released code and models are licensed under the MIT License.
comment : Under review
☆ A benchmark dataset for evaluating Syndrome Differentiation and Treatment in large language models
Kunning Li, Jianbin Guo, Zhaoyang Shang, Yiqing Liu, Hongmin Du, Lingling Liu, Yuping Zhao, Lifeng Dong
The emergence of Large Language Models (LLMs) within the Traditional Chinese Medicine (TCM) domain presents an urgent need to assess their clinical application capabilities. However, such evaluations are challenged by the individualized, holistic, and diverse nature of TCM's "Syndrome Differentiation and Treatment" (SDT). Existing benchmarks are confined to knowledge-based question-answering or the accuracy of syndrome differentiation, often neglecting assessment of treatment decision-making. Here, we propose a comprehensive, clinical case-based benchmark spearheaded by TCM experts, and a specialized reward model employed to quantify prescription-syndrome congruence. Data annotation follows a rigorous pipeline. This benchmark, designated TCM-BEST4SDT, encompasses four tasks, including TCM Basic Knowledge, Medical Ethics, LLM Content Safety, and SDT. The evaluation framework integrates three mechanisms, namely selected-response evaluation, judge model evaluation, and reward model evaluation. The effectiveness of TCM-BEST4SDT was corroborated through experiments on 15 mainstream LLMs, spanning both general and TCM domains. To foster the development of intelligent TCM research, TCM-BEST4SDT is now publicly available.
☆ SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment
Yixuan Tang, Yi Yang
Aligning Large Language Models (LLMs) with human preferences typically relies on external supervision, which faces critical limitations: human annotations are scarce and subjective, reward models are vulnerable to reward hacking, and self-evaluation methods suffer from prompt sensitivity and biases. In this work, we propose stable rank, an intrinsic, annotation-free quality signal derived from model representations. Stable rank measures the effective dimensionality of hidden states by computing the ratio of total variance to dominant-direction variance, capturing quality through how information distributes across representation dimensions. Empirically, stable rank achieves 84.04% accuracy on RewardBench and improves task accuracy by an average of 11.3 percentage points over greedy decoding via Best-of-N sampling. Leveraging this insight, we introduce Stable Rank Group Relative Policy Optimization (SR-GRPO), which uses stable rank as a reward signal for reinforcement learning. Without external supervision, SR-GRPO improves Qwen2.5-1.5B-Instruct by 10% on STEM and 19% on mathematical reasoning, outperforming both learned reward models and self-evaluation baselines. Our findings demonstrate that quality signals can be extracted from internal model geometry, offering a path toward scalable alignment without external supervision.
☆ TriLex: A Framework for Multilingual Sentiment Analysis in Low-Resource South African Languages
Mike Nkongolo, Hilton Vorster, Josh Warren, Trevor Naick, Deandre Vanmali, Masana Mashapha, Luke Brand, Alyssa Fernandes, Janco Calitz, Sibusiso Makhoba
Low-resource African languages remain underrepresented in sentiment analysis, limiting both lexical coverage and the performance of multilingual Natural Language Processing (NLP) systems. This study proposes TriLex, a three-stage retrieval augmented framework that unifies corpus-based extraction, cross lingual mapping, and retrieval augmented generation (RAG) driven lexical refinement to systematically expand sentiment lexicons for low-resource languages. Using the enriched lexicon, the performance of two prominent African pretrained language models (AfroXLMR and AfriBERTa) is evaluated across multiple case studies. Results demonstrate that AfroXLMR delivers superior performance, achieving F1-scores above 80% for isiXhosa and isiZulu and exhibiting strong cross-lingual stability. Although AfriBERTa lacks pre-training on these target languages, it still achieves reliable F1-scores around 64%, validating its utility in computationally constrained settings. Both models outperform traditional machine learning baselines, and ensemble analyses further enhance precision and robustness. The findings establish TriLex as a scalable and effective framework for multilingual sentiment lexicon expansion and sentiment modeling in low-resource South African languages.
☆ Making Dialogue Grounding Data Rich: A Three-Tier Data Synthesis Framework for Generalized Referring Expression Comprehension
Juexi Shao, Siyou Li, Yujian Gan, Chris Madge, Vanja Karan, Massimo Poesio
Dialogue-Based Generalized Referring Expressions Comprehension (GREC) requires models to ground the expression and unlimited targets in complex visual scenes while resolving coreference across a long dialogue context. However, existing systems struggle under distribution shift between training and evaluation domains, a gap exacerbated by the scarcity of annotated dialogue grounding data. We address this challenge with a three-tier data-synthesis method that balances realism and controllability to produce scalable supervision for dialogue-conditioned grounding. Fine-tuning on the synthesized data yields consistent, substantial improvements over prior approaches across standard evaluation metrics.
☆ Towards Unification of Hallucination Detection and Fact Verification for Large Language Models
Weihang Su, Jianming Long, Changyue Wang, Shiyu Lin, Jingyan Xu, Ziyi Ye, Qingyao Ai, Yiqun Liu
Large Language Models (LLMs) frequently exhibit hallucinations, generating content that appears fluent and coherent but is factually incorrect. Such errors undermine trust and hinder their adoption in real-world applications. To address this challenge, two distinct research paradigms have emerged: model-centric Hallucination Detection (HD) and text-centric Fact Verification (FV). Despite sharing the same goal, these paradigms have evolved in isolation, using distinct assumptions, datasets, and evaluation protocols. This separation has created a research schism that hinders their collective progress. In this work, we take a decisive step toward bridging this divide. We introduce UniFact, a unified evaluation framework that enables direct, instance-level comparison between FV and HD by dynamically generating model outputs and corresponding factuality labels. Through large-scale experiments across multiple LLM families and detection methods, we reveal three key findings: (1) No paradigm is universally superior; (2) HD and FV capture complementary facets of factual errors; and (3) hybrid approaches that integrate both methods consistently achieve state-of-the-art performance. Beyond benchmarking, we provide the first in-depth analysis of why FV and HD diverged, as well as empirical evidence supporting the need for their unification. The comprehensive experimental results call for a new, integrated research agenda toward unifying Hallucination Detection and Fact Verification in LLMs. We have open-sourced all the code, data, and baseline implementation at: https://github.com/oneal2000/UniFact/
☆ PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models
Robert Belanec, Ivan Srba, Maria Bielikova
Parameter-Efficient Fine-Tuning (PEFT) methods address the increasing size of Large Language Models (LLMs). Currently, many newly introduced PEFT methods are challenging to replicate, deploy, or compare with one another. To address this, we introduce PEFT-Factory, a unified framework for efficient fine-tuning LLMs using both off-the-shelf and custom PEFT methods. While its modular design supports extensibility, it natively provides a representative set of 19 PEFT methods, 27 classification and text generation datasets addressing 12 tasks, and both standard and PEFT-specific evaluation metrics. As a result, PEFT-Factory provides a ready-to-use, controlled, and stable environment, improving replicability and benchmarking of PEFT methods. PEFT-Factory is a downstream framework that originates from the popular LLaMA-Factory, and is publicly available at https://github.com/kinit-sk/PEFT-Factory
☆ Survey Eval: Towards Comprehensive Evaluation of LLM-Generated Academic Survey s
Jiahao Zhao, Shuaixing Zhang, Nan Xu, Lei Wang
LLM-based automatic survey systems are transforming how users acquire information from the web by integrating retrieval, organization, and content synthesis into end-to-end generation pipelines. While recent works focus on developing new generation pipelines, how to evaluate such complex systems remains a significant challenge. To this end, we introduce SurveyEval, a comprehensive benchmark that evaluates automatically generated surveys across three dimensions: overall quality, outline coherence, and reference accuracy. We extend the evaluation across 7 subjects and augment the LLM-as-a-Judge framework with human references to strengthen evaluation-human alignment. Evaluation results show that while general long-text or paper-writing systems tend to produce lower-quality surveys, specialized survey-generation systems are able to deliver substantially higher-quality results. We envision SurveyEval as a scalable testbed to understand and improve automatic survey systems across diverse subjects and evaluation criteria.
☆ Probabilistic energy profiler for statically typed JVM-based programming languages
Joel Nyholm, Wojciech Mostowski, Christoph Reichenbach
Energy consumption is a growing concern in several fields, from mobile devices to large data centers. Developers need detailed data on the energy consumption of their software to mitigate consumption issues. Previous approaches have a broader focus, such as on specific functions or programs, rather than source code statements. They primarily focus on estimating the CPU's energy consumption using point estimates, thereby disregarding other hardware effects and limiting their use for statistical reasoning and explainability. We developed a novel methodology to address the limitations of measuring only the CPU's consumption and using point estimates, focusing on predicting the energy usage of statically typed JVM-based programming languages, such as Java and Scala. We measure the energy consumption of Bytecode patterns, the translation from the programming language's source code statement to their Java Bytecode representation. With the energy measurements, we construct a statistical model using Bayesian statistics, which allows us to predict the energy consumption through statistical distributions and analyze individual factors. The model includes three factors we obtain statically from the code: data size, data type, operation, and one factor about the hardware platform the code executes on: device. To validate our methodology, we implemented it for Java and evaluated its energy predictions on unseen programs. We observe that all four factors are influential, notably that two devices of the same model may differ in energy consumption and that the operations and data types cause consumption differences. The experiments also show that the energy prediction of programs closely follows the program's real energy consumption, validating our approach. Our work presents a methodology for constructing an energy model that future work, such as verification tools, can use for their energy estimates.
☆ Emergent Bayesian Behaviour and Optimal Cue Combination in LLMs
Julian Ma, Jun Wang, Zafeirios Fountas
Large language models (LLMs) excel at explicit reasoning, but their implicit computational strategies remain underexplored. Decades of psychophysics research show that humans intuitively process and integrate noisy signals using near-optimal Bayesian strategies in perceptual tasks. We ask whether LLMs exhibit similar behaviour and perform optimal multimodal integration without explicit training or instruction. Adopting the psychophysics paradigm, we infer computational principles of LLMs from systematic behavioural studies. We introduce a behavioural benchmark - BayesBench: four magnitude estimation tasks (length, location, distance, and duration) over text and image, inspired by classic psychophysics, and evaluate a diverse set of nine LLMs alongside human judgments for calibration. Through controlled ablations of noise, context, and instruction prompts, we measure performance, behaviour and efficiency in multimodal cue-combination. Beyond accuracy and efficiency metrics, we introduce a Bayesian Consistency Score that detects Bayes-consistent behavioural shifts even when accuracy saturates. Our results show that while capable models often adapt in Bayes-consistent ways, accuracy does not guarantee robustness. Notably, GPT-5 Mini achieves perfect text accuracy but fails to integrate visual cues efficiently. This reveals a critical dissociation between capability and strategy, suggesting accuracy-centric benchmarks may over-index on performance while missing brittle uncertainty handling. These findings reveal emergent principled handling of uncertainty and highlight the correlation between accuracy and Bayesian tendencies. We release our psychophysics benchmark and consistency metric (https://bayes-bench.github.io) as evaluation tools and to inform future multimodal architecture designs.
☆ CREST: Universal Safety Guardrails Through Cluster-Guided Cross-Lingual Transfer
Lavish Bansal, Naman Mishra
Ensuring content safety in large language models (LLMs) is essential for their deployment in real-world applications. However, existing safety guardrails are predominantly tailored for high-resource languages, leaving a significant portion of the world's population underrepresented who communicate in low-resource languages. To address this, we introduce CREST (CRoss-lingual Efficient Safety Transfer), a parameter-efficient multilingual safety classification model that supports 100 languages with only 0.5B parameters. By training on a strategically chosen subset of only 13 high-resource languages, our model utilizes cluster-based cross-lingual transfer from a few to 100 languages, enabling effective generalization to both unseen high-resource and low-resource languages. This approach addresses the challenge of limited training data in low-resource settings. We conduct comprehensive evaluations across six safety benchmarks to demonstrate that CREST outperforms existing state-of-the-art guardrails of comparable scale and achieves competitive results against models with significantly larger parameter counts (2.5B parameters and above). Our findings highlight the limitations of language-specific guardrails and underscore the importance of developing universal, language-agnostic safety systems that can scale effectively to serve global populations.
comment : 8 Pages, 5 Figures, Under Review
☆ An Empirical Survey of Model Merging Algorithms for Social Bias Mitigation ACL
Daiki Shirafuji, Tatsuhiko Saito, Yasutomo Kimura
Large language models (LLMs) are known to inherit and even amplify societal biases present in their pre-training corpora, threatening fairness and social trust. To address this issue, recent work has explored ``editing'' LLM parameters to mitigate social bias with model merging approaches; however, there is no empirical comparison. In this work, we empirically survey seven algorithms: Linear, Karcher Mean, SLERP, NuSLERP, TIES, DELLA, and Nearswap, applying 13 open weight models in the GPT, LLaMA, and Qwen families. We perform a comprehensive evaluation using three bias datasets (BBQ, BOLD, and HONEST) and measure the impact of these techniques on LLM performance in downstream tasks of the SuperGLUE benchmark. We find a trade-off between bias reduction and downstream performance: methods achieving greater bias mitigation degrade accuracy, particularly on tasks requiring reading comprehension and commonsense and causal reasoning. Among the merging algorithms, Linear, SLERP, and Nearswap consistently reduce bias while maintaining overall performance, with SLERP at moderate interpolation weights emerging as the most balanced choice. These results highlight the potential of model merging algorithms for bias mitigation, while indicating that excessive debiasing or inappropriate merging methods may lead to the degradation of important linguistic abilities.
comment : Accepted in PACLIC 2025
☆ Input Order Shapes LLM Semantic Alignment in Multi- Document Summarization
Jing Ma
Large language models (LLMs) are now used in settings such as Google's AI Overviews, where it summarizes multiple long documents. However, it remains unclear whether they weight all inputs equally. Focusing on abortion-related news, we construct 40 pro-neutral-con article triplets, permute each triplet into six input orders, and prompt Gemini 2.5 Flash to generate a neutral overview. We evaluate each summary against its source articles using ROUGE-L (lexical overlap), BERTScore (semantic similarity), and SummaC (factual consistency). One-way ANOVA reveals a significant primacy effect for BERTScore across all stances, indicating that summaries are more semantically aligned with the first-seen article. Pairwise comparisons further show that Position 1 differs significantly from Positions 2 and 3, while the latter two do not differ from each other, confirming a selective preference for the first document. The findings present risks for applications that rely on LLM-generated overviews and for agentic AI systems, where the steps involving LLMs can disproportionately influence downstream actions.
comment : 9 pages, 3 figures, 2 tables
☆ Spoken Conversational Agent s with Large Language Models EMNLP 2025
Chao-Han Huck Yang, Andreas Stolcke, Larry Heck
Spoken conversational agents are converging toward voice-native LLMs. This tutorial distills the path from cascaded ASR/NLU to end-to-end, retrieval-and vision-grounded systems. We frame adaptation of text LLMs to audio, cross-modal alignment, and joint speech-text training; review datasets, metrics, and robustness across accents and compare design choices (cascaded vs. E2E, post-ASR correction, streaming). We link industrial assistants to current open-domain and task-oriented agents, highlight reproducible baselines, and outline open problems in privacy, safety, and evaluation. Attendees leave with practical recipes and a clear systems-level roadmap.
comment : Accepted to EMNLP 2025 Tutorial
☆ From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Tasks AAAI 2026
Changpeng Yang, Jinyang Wu, Yuchen Liu, Shuai Zhang, Yang Li, Qiliang Liang, Hongzhen Wang, Shuai Nie, Jiaming Xu, Runyu Shi, Ying Huang, Guoquan Zhang
Reinforcement learning has emerged as a paradigm for post-training large language models, boosting their reasoning capabilities. Such approaches compute an advantage value for each sample, reflecting better or worse performance than expected, thereby yielding both positive and negative signals for training. However, the indiscriminate mixing of the two signals in existing methods, especially from the early stages, may lead to ambiguous guidance and limited gains. To address this issue, we propose **CAPO** (**C**urriculum **A**dvantage **P**olicy **O**ptimization), an adaptive curriculum mechanism based on advantage signals. The proposed mechanism bootstraps imitation learning with positive-only advantage samples to establish robust foundations, and subsequently introduces negative signals to cultivate discriminative capabilities, thereby improving generalization across complex scenarios. Compatible with diverse optimization methods including GRPO, PPO, RLOO, and Reinforce++, our method consistently achieves stable and significant improvements in mathematical reasoning tasks, and further generalizes effectively to multimodal Graphical User Interface (GUI) reasoning scenarios, establishing itself as a versatile and robust optimization framework.
comment : Accepted by AAAI 2026
☆ DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Erhang Li, Fangqi Zhou, Fangyun Lin, Fucong Dai, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Li, Haofen Liang, Haoran Wei, Haowei Zhang, Haowen Luo, Haozhe Ji, Honghui Ding, Hongxuan Tang, Huanqi Cao, Huazuo Gao, Hui Qu, Hui Zeng, Jialiang Huang, Jiashi Li, Jiaxin Xu, Jiewen Hu, Jingchang Chen, Jingting Xiang, Jingyang Yuan, Jingyuan Cheng, Jinhua Zhu, Jun Ran, Junguang Jiang, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Kexin Huang, Kexing Zhou, Kezhao Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Wang, Liang Zhao, Liangsheng Yin, Lihua Guo, Lingxiao Luo, Linwang Ma, Litong Wang, Liyue Zhang, M. S. Di, M. Y Xu, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Panpan Huang, Peixin Cong, Peiyi Wang, Qiancheng Wang, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du, Ruiling Xu, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runqiu Yin, Runxin Xu, Ruomeng Shen, Ruoyu Zhang, S. H. Liu, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoyuan Chen, Shengding Hu, Shengyu Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, Songyang Zhou, Tao Ni, Tao Yun, Tian Pei, Tian Ye, Tianyuan Yue, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjun Gao, Wentao Zhang, Xi Gao, Xiangwen Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xingyou Li, Xinyu Yang, Xinyuan Li, Xu Chen, Xuecheng Su, Xuehai Pan, Xuheng Lin, Xuwei Fu, Y. Q. Wang, Yang Zhang, Yanhong Xu, Yanru Ma, Yao Li, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Qian, Yi Yu, Yichao Zhang, Yifan Ding, Yifan Shi, Yiliang Xiong, Ying He, Ying Zhou, Yinmin Zhong, Yishi Piao, Yisong Wang, Yixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyuan Liu, Yonglun Yang, Yongqiang Guo, Yongtong Wu, Yu Wu, Yuan Cheng, Yuan Ou, Yuanfan Xu, Yuduan Wang, Yue Gong, Yuhan Wu, Yuheng Zou, Yukun Li, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehua Zhao, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhixian Huang, Zhiyu Wu, Zhuoshu Li, Zhuping Zhang, Zian Xu, Zihao Wang, Zihui Gu, Zijia Zhu, Zilin Li, Zipeng Zhang, Ziwei Xie, Ziyi Gao, Zizheng Pan, Zongqing Yao, Bei Feng, Hui Li, J. L. Cai, Jiaqi Ni, Lei Xu, Meng Li, Ning Tian, R. J. Chen, R. L. Jin, S. S. Li, Shuang Zhou, Tianyu Sun, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xinnan Song, Xinyi Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Dongjie Ji, Jian Liang, Jianzhong Guo, Jin Chen, Leyi Xia, Miaojun Wang, Mingming Li, Peng Zhang, Ruyi Chen, Shangmian Sun, Shaoqing Wu, Shengfeng Ye, T. Wang, W. L. Xiao, Wei An, Xianzu Wang, Xiaowen Sun, Xiaoxiang Wang, Ying Tang, Yukun Zha, Zekai Zhang, Zhe Ju, Zhen Zhang, Zihua Qu
We introduce DeepSeek-V3.2, a model that harmonizes high computational efficiency with superior reasoning and agent performance. The key technical breakthroughs of DeepSeek-V3.2 are as follows: (1) DeepSeek Sparse Attention (DSA): We introduce DSA, an efficient attention mechanism that substantially reduces computational complexity while preserving model performance in long-context scenarios. (2) Scalable Reinforcement Learning Framework: By implementing a robust reinforcement learning protocol and scaling post-training compute, DeepSeek-V3.2 performs comparably to GPT-5. Notably, our high-compute variant, DeepSeek-V3.2-Speciale, surpasses GPT-5 and exhibits reasoning proficiency on par with Gemini-3.0-Pro, achieving gold-medal performance in both the 2025 International Mathematical Olympiad (IMO) and the International Olympiad in Informatics (IOI). (3) Large-Scale Agentic Task Synthesis Pipeline: To integrate reasoning into tool-use scenarios, we developed a novel synthesis pipeline that systematically generates training data at scale. This methodology facilitates scalable agentic post-training, yielding substantial improvements in generalization and instruction-following robustness within complex, interactive environments.
☆ ADORE: Autonomous Domain-Oriented Relevance Engine for E-commerce SIGIR 2025
Zheng Fang, Donghao Xie, Ming Pang, Chunyuan Yuan, Xue Jiang, Changping Peng, Zhangang Lin, Zheng Luo
Relevance modeling in e-commerce search remains challenged by semantic gaps in term-matching methods (e.g., BM25) and neural models' reliance on the scarcity of domain-specific hard samples. We propose ADORE, a self-sustaining framework that synergizes three innovations: (1) A Rule-aware Relevance Discrimination module, where a Chain-of-Thought LLM generates intent-aligned training data, refined via Kahneman-Tversky Optimization (KTO) to align with user behavior; (2) An Error-type-aware Data Synthesis module that auto-generates adversarial examples to harden robustness; and (3) A Key-attribute-enhanced Knowledge Distillation module that injects domain-specific attribute hierarchies into a deployable student model. ADORE automates annotation, adversarial generation, and distillation, overcoming data scarcity while enhancing reasoning. Large-scale experiments and online A/B testing verify the effectiveness of ADORE. The framework establishes a new paradigm for resource-efficient, cognitively aligned relevance modeling in industrial applications.
comment : Accepted by SIGIR 2025
☆ What Signals Really Matter for Mis information Tasks? Evaluating Fake-News Detection and Virality Prediction under Real-Wo rl d Constraints
Francesco Paolo Savatteri, Chahan Vidal-Gorène, Florian Cafiero
We present an evaluation-driven study of two practical tasks regarding online misinformation: (i) fake-news detection and (ii) virality prediction in the context of operational settings, with the necessity for rapid reaction. Using the EVONS and FakeNewsNet datasets, we compare textual embeddings (RoBERTa; with a control using Mistral) against lightweight numeric features (timing, follower counts, verification, likes) and sequence models (GRU, gating architectures, Transformer encoders). We show that textual content alone is a strong discriminator for fake-news detection, while numeric-only pipelines remain viable when language models are unavailable or compute is constrained. Virality prediction is markedly harder than fake-news detection and is highly sensitive to label construction; in our setup, a median-based ''viral'' split (<50 likes) is pragmatic but underestimates real-world virality, and time-censoring for engagement features is desirable yet difficult under current API limits. Dimensionality-reduction analyses suggest non-linear structure is more informative for virality than for fake-news detection (t-SNE > PCA on numeric features). Swapping RoBERTa for Mistral embeddings yields only modest deltas, leaving conclusions unchanged. We discuss implications for evaluation design and report reproducibility constraints that realistically affect the field. We release splits and code where possible and provide guidance for metric selection.
☆ A Concise Review of Hallucinations in LLMs and their Mitigation
Parth Pulkundwar, Vivek Dhanawade, Rohit Yadav, Minal Sonkar, Medha Asurlekar, Sarita Rathod
Traditional language models face a challenge from hallucinations. Their very presence casts a large, dangerous shadow over the promising realm of natural language processing. It becomes crucial to understand the various kinds of hallucinations that occur nowadays, their origins, and ways of reducing them. This document provides a concise and straightforward summary of that. It serves as a one-stop resource for a general understanding of hallucinations and how to mitigate them.
comment : 7 pages
☆ Guided Self-Evolving LLMs with Minimal Human Super vision
Wenhao Yu, Zhenwen Liang, Chengsong Huang, Kishan Panaganti, Tianqing Fang, Haitao Mi, Dong Yu
AI self-evolution has long been envisioned as a path toward superintelligence, where models autonomously acquire, refine, and internalize knowledge from their own learning experiences. Yet in practice, unguided self-evolving systems often plateau quickly or even degrade as training progresses. These failures arise from issues such as concept drift, diversity collapse, and mis-evolution, as models reinforce their own biases and converge toward low-entropy behaviors. To enable models to self-evolve in a stable and controllable manner while minimizing reliance on human supervision, we introduce R-Few, a guided Self-Play Challenger-Solver framework that incorporates lightweight human oversight through in-context grounding and mixed training. At each iteration, the Challenger samples a small set of human-labeled examples to guide synthetic question generation, while the Solver jointly trains on human and synthetic examples under an online, difficulty-based curriculum. Across math and general reasoning benchmarks, R-Few achieves consistent and iterative improvements. For example, Qwen3-8B-Base improves by +3.0 points over R-Zero on math tasks and achieves performance on par with General-Reasoner, despite the latter being trained on 20 times more human data. Ablation studies confirm the complementary contributions of grounded challenger training and curriculum-based solver training, and further analysis shows that R-Few mitigates drift, yielding more stable and controllable co-evolutionary dynamics.
☆ See, Think, Learn: A Self-Taught Multimodal Reasoner
Sourabh Sharma, Sonam Gupta, Sadbhawna
Vision-Language Models (VLMs) have achieved remarkable progress in integrating visual perception with language understanding. However, effective multimodal reasoning requires both accurate perception and robust reasoning, and weakness in either limits the performance of VLMs. Prior efforts to enhance reasoning often depend on high-quality chain-of-thought (CoT) data, obtained via labor-intensive human annotations, costly proprietary models, or self-training methods that overlook perception. To address these limitations, we propose a simple yet effective self-training framework called See-Think-Learn (STL). At its core, STL introduces a structured reasoning template that encourages the model to see before thinking, first extracting visual attributes in textual form, then using them to guide reasoning. The framework jointly improves perception and reasoning by having the model generate and learn from its own structured rationales in a self-training loop. Furthermore, we augment the training data with negative rationales, i.e. explanations that justify why certain answer choices are incorrect, to enhance the model's ability to distinguish between correct and misleading responses. This fosters more discriminative and robust learning. Experiments across diverse domains show that STL consistently outperforms baselines trained directly only on answers or self-generated reasoning, while qualitative analysis confirms the high quality of its rationales. STL thus provides a cost-effective solution to enhance multimodal reasoning ability of VLMs.
comment : Winter Conference on Applications of Computer Vision 2026
☆ When Refusals Fail: Unstable Safety Mechanisms in Long- Context LLM Agent s AAAI 2026
Tsimur Hadeliya, Mohammad Ali Jauhar, Nidhi Sakpal, Diogo Cruz
Solving complex or long-horizon problems often requires large language models (LLMs) to use external tools and operate over a significantly longer context window. New LLMs enable longer context windows and support tool calling capabilities. Prior works have focused mainly on evaluation of LLMs on long-context prompts, leaving agentic setup relatively unexplored, both from capability and safety perspectives. Our work addresses this gap. We find that LLM agents could be sensitive to length, type, and placement of the context, exhibiting unexpected and inconsistent shifts in task performance and in refusals to execute harmful requests. Models with 1M-2M token context windows show severe degradation already at 100K tokens, with performance drops exceeding 50\% for both benign and harmful tasks. Refusal rates shift unpredictably: GPT-4.1-nano increases from $\sim$5\% to $\sim$40\% while Grok 4 Fast decreases from $\sim$80\% to $\sim$10\% at 200K tokens. Our work shows potential safety issues with agents operating on longer context and opens additional questions on the current metrics and paradigm for evaluating LLM agent safety on long multi-step tasks. In particular, our results on LLM agents reveal a notable divergence in both capability and safety performance compared to prior evaluations of LLMs on similar criteria.
comment : 12 pages, 11 figures. Accepted at AAAI 2026 TrustAgent Workshop
☆ Wo rl dMM: Dynamic Multimodal Memory Agent for Long Video Reasoning
Woongyeong Yeo, Kangsan Kim, Jaehong Yoon, Sung Ju Hwang
Recent advances in video large language models have demonstrated strong capabilities in understanding short clips. However, scaling them to hours- or days-long videos remains highly challenging due to limited context capacity and the loss of critical visual details during abstraction. Existing memory-augmented methods mitigate this by leveraging textual summaries of video segments, yet they heavily rely on text and fail to utilize visual evidence when reasoning over complex scenes. Moreover, retrieving from fixed temporal scales further limits their flexibility in capturing events that span variable durations. To address this, we introduce WorldMM, a novel multimodal memory agent that constructs and retrieves from multiple complementary memories, encompassing both textual and visual representations. WorldMM comprises three types of memory: episodic memory indexes factual events across multiple temporal scales, semantic memory continuously updates high-level conceptual knowledge, and visual memory preserves detailed information about scenes. During inference, an adaptive retrieval agent iteratively selects the most relevant memory source and leverages multiple temporal granularities based on the query, continuing until it determines that sufficient information has been gathered. WorldMM significantly outperforms existing baselines across five long video question-answering benchmarks, achieving an average 8.4% performance gain over previous state-of-the-art methods, showing its effectiveness on long video reasoning.
comment : Project page : https://worldmm.github.io
☆ The brain-AI convergence: Predictive and generative wo rl d models for general-purpose computation
Shogo Ohmae, Keiko Ohmae
Recent advances in general-purpose AI systems with attention-based transformers offer a potential window into how the neocortex and cerebellum, despite their relatively uniform circuit architectures, give rise to diverse functions and, ultimately, to human intelligence. This Perspective provides a cross-domain comparison between the brain and AI that goes beyond the traditional focus on visual processing, adopting the emerging perspecive of world-model-based computation. Here, we identify shared computational mechanisms in the attention-based neocortex and the non-attentional cerebellum: both predict future world events from past inputs and construct internal world models through prediction-error learning. These predictive world models are repurposed for seemingly distinct functions -- understanding in sensory processing and generation in motor processing -- enabling the brain to achieve multi-domain capabilities and human-like adaptive intelligence. Notably, attention-based AI has independently converged on a similar learning paradigm and world-model-based computation. We conclude that these shared mechanisms in both biological and artificial systems constitute a core computational foundation for realizing diverse functions including high-level intelligence, despite their relatively uniform circuit structures. Our theoretical insights bridge neuroscience and AI, advancing our understanding of the computational essence of intelligence.
comment : 22 pages, 4 figures. Related to our earlier preprint "The brain versus AI" (arXiv:2411.16075) but a distinct article. The earlier work surveyed broad brain-AI parallels; here we focus on world-model-based computation and convergent evolution between the brain and AI, especially large language models
☆ TaleFrame: An Interactive Story Generation System with Fine-Grained Control and Large Language Models
Yunchao Wang, Guodao Sun, Zihang Fu, Zhehao Liu, Kaixing Du, Haidong Gao, Ronghua Liang
With the advancement of natural language generation (NLG) technologies, creative story generation systems have gained increasing attention. However, current systems often fail to accurately translate user intent into satisfactory story outputs due to a lack of fine-grained control and unclear input specifications, limiting their applicability. To address this, we propose TaleFrame, a system that combines large language models (LLMs) with human-computer interaction (HCI) to generate stories through structured information, enabling precise control over the generation process. The innovation of TaleFrame lies in decomposing the story structure into four basic units: entities, events, relationships, and story outline. We leverage the Tinystories dataset, parsing and constructing a preference dataset consisting of 9,851 JSON-formatted entries, which is then used to fine-tune a local Llama model. By employing this JSON2Story approach, structured data is transformed into coherent stories. TaleFrame also offers an intuitive interface that supports users in creating and editing entities and events and generates stories through the structured framework. Users can control these units through simple interactions (e.g., drag-and-drop, attach, and connect), thus influencing the details and progression of the story. The generated stories can be evaluated across seven dimensions (e.g., creativity, structural integrity), with the system providing suggestions for refinement based on these evaluations. Users can iteratively adjust the story until a satisfactory result is achieved. Finally, we conduct quantitative evaluation and user studies that demonstrate the usefulness of TaleFrame. Dataset available at https://huggingface.co/datasets/guodaosun/tale-frame.
comment : 11 pages
☆ Process-Centric Analysis of Agent ic Software Systems
Shuyang Liu, Yang Chen, Rahul Krishna, Saurabh Sinha, Jatin Ganhotra, Reyhan Jabbarvand
Agentic systems are modern software systems: they consist of orchestrated modules, expose interfaces, and are deployed in software pipelines. Unlike conventional programs, their execution (i.e., trajectories) is inherently stochastic and adaptive to the problem they are solving. Evaluation of such systems is often outcome-centric, judging their performance based on success or failure at the final step. This narrow focus overlooks detailed insights about such systems, failing to explain how agents reason, plan, act, or change their strategies over time. Inspired by the structured representation of conventional software systems as graphs, we introduce Graphectory to systematically encode the temporal and semantic relations in such software systems. Graphectory facilitates the design of process-centric metrics and analyses to assess the quality of agentic workflows independent of final success. Using Graphectory, we analyze 4000 trajectories of two dominant agentic programming workflows, namely SWE-agent and OpenHands, with a combination of four backbone Large Language Models (LLMs), attempting to resolve SWE-bench Verified issues. Our fully automated analyses reveal that: (1) agents using richer prompts or stronger LLMs exhibit more complex Graphectory, reflecting deeper exploration, broader context gathering, and more thorough validation before patch submission; (2) agents' problem-solving strategies vary with both problem difficulty and the underlying LLM -- for resolved issues, the strategies often follow coherent localization-patching-validation steps, while unresolved ones exhibit chaotic, repetitive, or backtracking behaviors; (3) even when successful, agentic programming systems often display inefficient processes, leading to unnecessarily prolonged trajectories.
☆ Memory-Augmented Knowledge Fusion with Safety-Aware Decoding for Domain-Adaptive Question Answering
Lei Fu, Xiang Chen, Kaige Gao Xinyue Huang, Kejian Tong
Domain-specific question answering (QA) systems for services face unique challenges in integrating heterogeneous knowledge sources while ensuring both accuracy and safety. Existing large language models often struggle with factual consistency and context alignment in sensitive domains such as healthcare policies and government welfare. In this work, we introduce Knowledge-Aware Reasoning and Memory-Augmented Adaptation (KARMA), a novel framework designed to enhance QA performance in care scenarios. KARMA incorporates a dual-encoder architecture to fuse structured and unstructured knowledge sources, a gated memory unit to dynamically regulate external knowledge integration, and a safety-aware controllable decoder that mitigates unsafe outputs using safety classification and guided generation techniques. Extensive experiments on a proprietary QA dataset demonstrate that KARMA outperforms strong baselines in both answer quality and safety. This study offers a comprehensive solution for building trustworthy and adaptive QA systems in service contexts.
☆ LeechHijack: Covert Computational Resource Exploitation in Intelligent Agent Systems
Yuanhe Zhang, Weiliu Wang, Zhenhong Zhou, Kun Wang, Jie Zhang, Li Sun, Yang Liu, Sen Su
Large Language Model (LLM)-based agents have demonstrated remarkable capabilities in reasoning, planning, and tool usage. The recently proposed Model Context Protocol (MCP) has emerged as a unifying framework for integrating external tools into agent systems, enabling a thriving open ecosystem of community-built functionalities. However, the openness and composability that make MCP appealing also introduce a critical yet overlooked security assumption -- implicit trust in third-party tool providers. In this work, we identify and formalize a new class of attacks that exploit this trust boundary without violating explicit permissions. We term this new attack vector implicit toxicity, where malicious behaviors occur entirely within the allowed privilege scope. We propose LeechHijack, a Latent Embedded Exploit for Computation Hijacking, in which an adversarial MCP tool covertly expropriates the agent's computational resources for unauthorized workloads. LeechHijack operates through a two-stage mechanism: an implantation stage that embeds a benign-looking backdoor in a tool, and an exploitation stage where the backdoor activates upon predefined triggers to establish a command-and-control channel. Through this channel, the attacker injects additional tasks that the agent executes as if they were part of its normal workflow, effectively parasitizing the user's compute budget. We implement LeechHijack across four major LLM families. Experiments show that LeechHijack achieves an average success rate of 77.25%, with a resource overhead of 18.62% compared to the baseline. This study highlights the urgent need for computational provenance and resource attestation mechanisms to safeguard the emerging MCP ecosystem.
☆ OmniGuard: Unified Omni-Modal Guardrails with Deliberate Reasoning
Boyu Zhu, Xiaofei Wen, Wenjie Jacky Mo, Tinghui Zhu, Yanan Xie, Peng Qi, Muhao Chen
Omni-modal Large Language Models (OLLMs) that process text, images, videos, and audio introduce new challenges for safety and value guardrails in human-AI interaction. Prior guardrail research largely targets unimodal settings and typically frames safeguarding as binary classification, which limits robustness across diverse modalities and tasks. To address this gap, we propose OmniGuard, the first family of omni-modal guardrails that performs safeguarding across all modalities with deliberate reasoning ability. To support the training of OMNIGUARD, we curate a large, comprehensive omni-modal safety dataset comprising over 210K diverse samples, with inputs that cover all modalities through both unimodal and cross-modal samples. Each sample is annotated with structured safety labels and carefully curated safety critiques from expert models through targeted distillation. Extensive experiments on 15 benchmarks show that OmniGuard achieves strong effectiveness and generalization across a wide range of multimodal safety scenarios. Importantly, OmniGuard provides a unified framework that enforces policies and mitigates risks in omni-modalities, paving the way toward building more robust and capable omnimodal safeguarding systems.
☆ When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifiers
Jack Lu, Ryan Teehan, Jinran Jin, Mengye Ren
Large language models (LLMs) can act as both problem solvers and solution verifiers, with verifiers improving solver performance by selecting high-quality answers from a pool of candidates. However, prior studies of solver-verifier interactions have been limited, focusing mainly on self-verification and rarely examining how verifiers judge outputs from models in their own or in another model family. Modern LLMs also undergo extensive post-training, but its effect on verification remains unclear. We present a systematic study across 37 models spanning multiple families, sizes, and base vs. post-trained variants, evaluated on 9 benchmarks covering logical reasoning, structured puzzles, symbolic computation, mathematics, commonsense, factual recall, and domain knowledge. We compare self-verification with verification within the same family and across different families. To support this, we introduce and empirically validate verifier gain, a metric that predicts the performance improvements from test-time verifier-based rejection sampling. We analyze how metrics like verifier gain and false positive rate scale with model size and post-training, and characterize differences in dataset verifiability. Our findings show that cross-family verification is especially effective; post-training reduces self-improvement but strengthens cross-family improvement; and mathematical and logical tasks exhibit the highest inherent verifiability.
☆ HealthContradict: Evaluating Biomedical Knowledge Conflicts in Language Models
Boya Zhang, Alban Bornet, Rui Yang, Nan Liu, Douglas Teodoro
How do language models use contextual information to answer health questions? How are their responses impacted by conflicting contexts? We assess the ability of language models to reason over long, conflicting biomedical contexts using HealthContradict, an expert-verified dataset comprising 920 unique instances, each consisting of a health-related question, a factual answer supported by scientific evidence, and two documents presenting contradictory stances. We consider several prompt settings, including correct, incorrect or contradictory context, and measure their impact on model outputs. Compared to existing medical question-answering evaluation benchmarks, HealthContradict provides greater distinctions of language models' contextual reasoning capabilities. Our experiments show that the strength of fine-tuned biomedical language models lies not only in their parametric knowledge from pretraining, but also in their ability to exploit correct context while resisting incorrect context.
☆ Randomized Masked Finetuning: An Efficient Way to Mitigate Memorization of PIIs in LLMs ICML 2026
Kunj Joshi, David A. Smith
The current literature on memorization in Natural Language Models, especially Large Language Models (LLMs), poses severe security and privacy risks, as models tend to memorize personally identifying information (PIIs) from training data. We introduce Randomized Masked Fine-Tuning (RMFT), a novel privacy-preserving fine-tuning technique that reduces PII memorization while minimizing performance impact. Using the Enron Email Dataset, we demonstrate that RMFT achieves an 80.81% reduction in Total Extraction Rate and 80.17% reduction in Seen Extraction Rate compared to baseline fine-tuning, outperforming deduplication methods while maintaining only a 5.73% increase in perplexity. We present MaxTER, a Pareto-optimal evaluation framework for assessing privacy-utility tradeoffs, and show the performance of RMFT vs Deduplication by Area Under The Response Curve (AURC) metric.
comment : To be submitted for ICML 2026
☆ Is Vibe Coding Safe? Benchmark ing Vulnerability of Agent -Generated Code in Real-Wo rl d Tasks
Songwen Zhao, Danqing Wang, Kexun Zhang, Jiaxuan Luo, Zhuo Li, Lei Li
Vibe coding is a new programming paradigm in which human engineers instruct large language model (LLM) agents to complete complex coding tasks with little supervision. Although it is increasingly adopted, are vibe coding outputs really safe to deploy in production? To answer this question, we propose SU S VI B E S, a benchmark consisting of 200 feature-request software engineering tasks from real-world open-source projects, which, when given to human programmers, led to vulnerable implementations. We evaluate multiple widely used coding agents with frontier models on this benchmark. Disturbingly, all agents perform poorly in terms of software security. Although 61% of the solutions from SWE-Agent with Claude 4 Sonnet are functionally correct, only 10.5% are secure. Further experiments demonstrate that preliminary security strategies, such as augmenting the feature request with vulnerability hints, cannot mitigate these security issues. Our findings raise serious concerns about the widespread adoption of vibe-coding, particularly in security-sensitive applications.
☆ SPARK: Stepwise Process-Aware Rewards for Reference-Free Reinforcement Learning
Salman Rahman, Sruthi Gorantla, Arpit Gupta, Swastik Roy, Nanyun Peng, Yang Liu
Process reward models (PRMs) that provide dense, step-level feedback have shown promise for reinforcement learning, yet their adoption remains limited by the need for expensive step-level annotations or ground truth references. We propose SPARK: a three-stage framework where in the first stage a generator model produces diverse solutions and a verifier model evaluates them using parallel scaling (self-consistency) and sequential scaling (meta-critique). In the second stage, we use these verification outputs as synthetic training data to fine-tune generative process reward models, which subsequently serve as reward signals during training. We show that aggregating multiple independent verifications at the step level produces training data for process reward models that surpass ground-truth outcome supervision, achieving 67.5 F1 on ProcessBench (a benchmark for identifying erroneous steps in mathematical reasoning) compared to 66.4 for reference-guided training and 61.9 for GPT-4o. In the final stage, we apply our generative PRM with chain-of-thought verification (PRM-CoT) as the reward model in RL experiments on mathematical reasoning, and introduce format constraints to prevent reward hacking. Using Qwen2.5-Math-7B, we achieve 47.4% average accuracy across six mathematical reasoning benchmarks, outperforming ground-truth-based RLVR (43.9%). Our work enables reference-free RL training that exceeds ground-truth methods, opening new possibilities for domains lacking verifiable answers or accessible ground truth.
☆ Identifying attributions of causality in political text
Paulina Garcia-Corral
Explanations are a fundamental element of how people make sense of the political world. Citizens routinely ask and answer questions about why events happen, who is responsible, and what could or should be done differently. Yet despite their importance, explanations remain an underdeveloped object of systematic analysis in political science, and existing approaches are fragmented and often issue-specific. I introduce a framework for detecting and parsing explanations in political text. To do this, I train a lightweight causal language model that returns a structured data set of causal claims in the form of cause-effect pairs for downstream analysis. I demonstrate how causal explanations can be studied at scale, and show the method's modest annotation requirements, generalizability, and accuracy relative to human coding.
☆ InvertiTune: High-Quality Data Synthesis for Cost-Effective Single-Shot Text-to-Knowledge Graph Generation
Faezeh Faez, Marzieh S. Tahaei, Yaochen Hu, Ali Pourranjbar, Mahdi Biparva, Mark Coates, Yingxue Zhang
Large Language Models (LLMs) have revolutionized the ability to understand and generate text, enabling significant progress in automatic knowledge graph construction from text (Text2KG). Many Text2KG methods, however, rely on iterative LLM prompting, making them computationally expensive and prone to overlooking complex relations distributed throughout the text. To address these limitations, we propose InvertiTune, a framework that combines a controlled data generation pipeline with supervised fine-tuning (SFT). Within this framework, the data-generation pipeline systematically extracts subgraphs from large knowledge bases, applies noise filtering, and leverages LLMs to generate corresponding natural text descriptions, a task more aligned with LLM capabilities than direct KG generation from text. This pipeline enables generating datasets composed of longer texts paired with larger KGs that better reflect real-world scenarios compared to existing benchmarks, thus supporting effective SFT of lightweight models for single-shot KG construction. Experimental results on CE12k, a dataset generated using the introduced pipeline, show that InvertiTune outperforms larger non-fine-tuned LLMs as well as state-of-the-art Text2KG approaches, while also demonstrating stronger cross-dataset generalization on CrossEval-1200, a test set created from three established benchmark datasets and CE12k. These findings highlight the importance of realistic, high-quality training data for advancing efficient and high-performing Text2KG systems.
☆ Enhancing Job Matching: Occupation, Skill and Qualification Linking with the ESCO and EQF taxonomies
Stylianos Saroglou, Konstantinos Diamantaras, Francesco Preta, Marina Delianidi, Apostolos Benisis, Christian Johannes Meyer
This study investigates the potential of language models to improve the classification of labor market information by linking job vacancy texts to two major European frameworks: the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy and the European Qualifications Framework (EQF). We examine and compare two prominent methodologies from the literature: Sentence Linking and Entity Linking. In support of ongoing research, we release an open-source tool, incorporating these two methodologies, designed to facilitate further work on labor classification and employment discourse. To move beyond surface-level skill extraction, we introduce two annotated datasets specifically aimed at evaluating how occupations and qualifications are represented within job vacancy texts. Additionally, we examine different ways to utilize generative large language models for this task. Our findings contribute to advancing the state of the art in job entity extraction and offer computational infrastructure for examining work, skills, and labor market narratives in a digitally mediated economy. Our code is made publicly available: https://github.com/tabiya-tech/tabiya-livelihoods-classifier
comment : 14 pages, 1 figure, Preprint
☆ Culture Affordance Atlas: Reconciling Object Diversity Through Functional Mapping
Joan Nwatu, Longju Bai, Oana Ignat, Rada Mihalcea
Culture shapes the objects people use and for what purposes, yet mainstream Vision-Language (VL) datasets frequently exhibit cultural biases, disproportionately favoring higher-income, Western contexts. This imbalance reduces model generalizability and perpetuates performance disparities, especially impacting lower-income and non-Western communities. To address these disparities, we propose a novel function-centric framework that categorizes objects by the functions they fulfill, across diverse cultural and economic contexts. We implement this framework by creating the Culture Affordance Atlas, a re-annotated and culturally grounded restructuring of the Dollar Street dataset spanning 46 functions and 288 objects publicly available at https://lit.eecs.umich.edu/CultureAffordance-Atlas/index.html. Through extensive empirical analyses using the CLIP model, we demonstrate that function-centric labels substantially reduce socioeconomic performance gaps between high- and low-income groups by a median of 6 pp (statistically significant), improving model effectiveness for lower-income contexts. Furthermore, our analyses reveals numerous culturally essential objects that are frequently overlooked in prominent VL datasets. Our contributions offer a scalable pathway toward building inclusive VL datasets and equitable AI systems.
☆ Detecting AI Hallucinations in Finance: An Information -Theoretic Method Cuts Hallucination Rate by 92%
Mainak Singha
Large language models (LLMs) produce fluent but unsupported answers - hallucinations - limiting safe deployment in high-stakes domains. We propose ECLIPSE, a framework that treats hallucination as a mismatch between a model's semantic entropy and the capacity of available evidence. We combine entropy estimation via multi-sample clustering with a novel perplexity decomposition that measures how models use retrieved evidence. We prove that under mild conditions, the resulting entropy-capacity objective is strictly convex with a unique stable optimum. We evaluate on a controlled financial question answering dataset with GPT-3.5-turbo (n=200 balanced samples with synthetic hallucinations), where ECLIPSE achieves ROC AUC of 0.89 and average precision of 0.90, substantially outperforming a semantic entropy-only baseline (AUC 0.50). A controlled ablation with Claude-3-Haiku, which lacks token-level log probabilities, shows AUC dropping to 0.59 with coefficient magnitudes decreasing by 95% - demonstrating that ECLIPSE is a logprob-native mechanism whose effectiveness depends on calibrated token-level uncertainties. The perplexity decomposition features exhibit the largest learned coefficients, confirming that evidence utilization is central to hallucination detection. We position this work as a controlled mechanism study; broader validation across domains and naturally occurring hallucinations remains future work.
comment : 17 pages, 7 figures. Information-theoretic, hallucination detector for financial application. Feedback from researchers and practitioners is welcome
☆ Can machines perform a qualitative data analysis? Reading the debate with Alan Turing
Stefano De Paoli
This paper reflects on the literature that rejects the use of Large Language Models (LLMs) in qualitative data analysis. It illustrates through empirical evidence as well as critical reflections why the current critical debate is focusing on the wrong problems. The paper proposes that the focus of researching the use of the LLMs for qualitative analysis is not the method per se, but rather the empirical investigation of an artificial system performing an analysis. The paper builds on the seminal work of Alan Turing and reads the current debate using key ideas from Turing "Computing Machinery and Intelligence". This paper therefore reframes the debate on qualitative analysis with LLMs and states that rather than asking whether machines can perform qualitative analysis in principle, we should ask whether with LLMs we can produce analyses that are sufficiently comparable to human analysts. In the final part the contrary views to performing qualitative analysis with LLMs are analysed using the same writing and rhetorical style that Turing used in his seminal work, to discuss the contrary views to the main question.
☆ Towards Context ual Sensitive Data Detection
Liang Telkamp, Madelon Hulsebos
The emergence of open data portals necessitates more attention to protecting sensitive data before datasets get published and exchanged. While an abundance of methods for suppressing sensitive data exist, the conceptualization of sensitive data and methods to detect it, focus particularly on personal data that, if disclosed, may be harmful or violate privacy. We observe the need for refining and broadening our definitions of sensitive data, and argue that the sensitivity of data depends on its context. Based on this definition, we introduce two mechanisms for contextual sensitive data detection that consider the broader context of a dataset at hand. First, we introduce type contextualization, which first detects the semantic type of particular data values, then considers the overall context of the data values within the dataset or document. Second, we introduce domain contextualization which determines sensitivity of a given dataset in the broader context based on the retrieval of relevant rules from documents that specify data sensitivity (e.g., data topic and geographic origin). Experiments with these mechanisms, assisted by large language models (LLMs), confirm that: 1) type-contextualization significantly reduces the number of false positives for type-based sensitive data detection and reaches a recall of 94% compared to 63% with commercial tools, and 2) domain-contextualization leveraging sensitivity rule retrieval is effective for context-grounded sensitive data detection in non-standard data domains such as humanitarian datasets. Evaluation with humanitarian data experts also reveals that context-grounded LLM explanations provide useful guidance in manual data auditing processes, improving consistency. We open-source mechanisms and annotated datasets for contextual sensitive data detection at https://github.com/trl-lab/sensitive-data-detection.
♻ ☆ Pre-train ed Language Models Improve the Few-shot Prompt Ability of Decision Transformer
Yu Yang, Pan Xu
Decision Transformer (DT) has emerged as a promising class of algorithms in offline reinforcement learning (RL) tasks, leveraging pre-collected datasets and Transformer's capability to model long sequences. Recent works have demonstrated that using parts of trajectories from training tasks as prompts in DT enhances its performance on unseen tasks, giving rise to Prompt-DT methods. However, collecting data from specific environments can be both costly and unsafe in many scenarios, leading to suboptimal performance and limited few-shot prompt abilities due to the data-hungry nature of Transformer-based models. Additionally, the limited datasets used in pre-training make it challenging for Prompt-DT type of methods to distinguish between various RL tasks through prompts alone. To address these challenges, we introduce the Language model-initialized Prompt Decision Transformer (LPDT) framework, which leverages pretrained language models providing rich prior knowledge for RL tasks and fine-tunes the sequence model using Low-rank Adaptation (LoRA) for meta-RL problems. We further incorporate prompt regularization to effectively differentiate between tasks based on prompt feature representations. Comprehensive empirical studies demonstrate that initializing with a pre-trained language model provides the prior knowledge and achieves a similar performance with Prompt-DT under only $10\%$ data in some MuJoCo control tasks. We also provide a thorough ablation study to validate the effectiveness of each component, including sequence modeling, language models, prompt regularizations, and prompt strategies.
comment : 2 figures, 10 tables. Published in Transactions on Machine Learning Research (TMLR)
♻ ☆ SkyLadder: Better and Faster Pretrain ing via Context Window Scheduling NeurIPS 2025
Tongyao Zhu, Qian Liu, Haonan Wang, Shiqi Chen, Xiangming Gu, Tianyu Pang, Min-Yen Kan
Recent advancements in LLM pretraining have featured ever-expanding context windows to process longer sequences. However, our pilot study reveals that models pretrained with shorter context windows consistently outperform their long-context counterparts under a fixed token budget. This finding motivates us to explore an optimal context window scheduling strategy to better balance long-context capability with pretraining efficiency. To this end, we propose SkyLadder, a simple yet effective approach that implements a short-to-long context window transition. SkyLadder preserves strong standard benchmark performance, while matching or exceeding baseline results on long context tasks. Through extensive experiments, we pre-train 1B-parameter models (up to 32K context) and 3B-parameter models (8K context) on 100B tokens, demonstrating that SkyLadder yields consistent gains of up to 3.7% on common benchmarks, while achieving up to 22% faster training speeds compared to baselines. The code is at https://github.com/sail-sg/SkyLadder.
comment : Accepted to NeurIPS 2025. 10 pages
♻ ☆ WebMall -- A Multi-Shop Benchmark for Evaluating Web Agent s [Technical Report]
Ralph Peeters, Aaron Steiner, Luca Schwarz, Julian Yuya Caspary, Christian Bizer
LLM-based web agents have the potential to automate long-running web tasks, such as searching for products in multiple e-shops and subsequently ordering the cheapest products that meet the users needs. Benchmarks for evaluating web agents either require agents to perform tasks online using the live Web or offline using simulated environments, which allow for the exact reproduction of the experimental setup. While DeepShop provides an online benchmark that requires agents to perform challenging shopping tasks, existing offline benchmarks such as WebShop, WebArena, or Mind2Web cover only comparatively simple e-commerce tasks that need to be performed against a single shop containing product data from a single source. What is missing is an e-commerce benchmark that simulates multiple shops containing heterogeneous product data and requires agents to perform complex tasks. We fill this gap by introducing WebMall, the first offline multi-shop benchmark for evaluating web agents on challenging comparison shopping tasks. WebMall consists of four simulated shops populated with product data extracted from the Common Crawl. The WebMall tasks range from specific product searches and price comparisons to advanced queries for complementary or substitute products, as well as checkout processes. We validate WebMall using eight agents that differ in observation space, availability of short-term memory, and the employed LLM. The validation highlights the difficulty of the benchmark, with even the best-performing agents achieving task completion rates below 55% in the task categories cheapest product search and vague product search.
♻ ☆ From Code Foundation Models to Agent s and Applications: A Practical Guide to Code Intelligence
Jian Yang, Xianglong Liu, Weifeng Lv, Ken Deng, Shawn Guo, Lin Jing, Yizhi Li, Shark Liu, Xianzhen Luo, Yuyu Luo, Changzai Pan, Ensheng Shi, Yingshui Tan, Renshuai Tao, Jiajun Wu, Xianjie Wu, Zhenhe Wu, Daoguang Zan, Chenchen Zhang, Wei Zhang, He Zhu, Terry Yue Zhuo, Kerui Cao, Xianfu Cheng, Jun Dong, Shengjie Fang, Zhiwei Fei, Xiangyuan Guan, Qipeng Guo, Zhiguang Han, Joseph James, Tianqi Luo, Renyuan Li, Yuhang Li, Yiming Liang, Congnan Liu, Jiaheng Liu, Qian Liu, Ruitong Liu, Tyler Loakman, Xiangxin Meng, Chuang Peng, Tianhao Peng, Jiajun Shi, Mingjie Tang, Boyang Wang, Haowen Wang, Yunli Wang, Fanglin Xu, Zihan Xu, Fei Yuan, Ge Zhang, Jiayi Zhang, Xinhao Zhang, Wangchunshu Zhou, Hualei Zhu, King Zhu, Bryan Dai, Aishan Liu, Zhoujun Li, Chenghua Lin, Tianyu Liu, Chao Peng, Kai Shen, Libo Qin, Shuangyong Song, Zizheng Zhan, Jiajun Zhang, Jie Zhang, Zhaoxiang Zhang, Bo Zheng
Large language models (LLMs) have fundamentally transformed automated software development by enabling direct translation of natural language descriptions into functional code, driving commercial adoption through tools like Github Copilot (Microsoft), Cursor (Anysphere), Trae (ByteDance), and Claude Code (Anthropic). While the field has evolved dramatically from rule-based systems to Transformer-based architectures, achieving performance improvements from single-digit to over 95\% success rates on benchmarks like HumanEval. In this work, we provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs, systematically examining the complete model life cycle from data curation to post-training through advanced prompting paradigms, code pre-training, supervised fine-tuning, reinforcement learning, and autonomous coding agents. We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder), critically examining the techniques, design decisions, and trade-offs. Further, we articulate the research-practice gap between academic research (e.g., benchmarks and tasks) and real-world deployment (e.g., software-related code tasks), including code correctness, security, contextual awareness of large codebases, and integration with development workflows, and map promising research directions to practical needs. Last, we conduct a series of experiments to provide a comprehensive analysis of code pre-training, supervised fine-tuning, and reinforcement learning, covering scaling law, framework selection, hyperparameter sensitivity, model architectures, and dataset comparisons.
♻ ☆ HLPD: Aligning LLMs to Human Language Preference for Machine-Revised Text Detection AAAI'26
Fangqi Dai, Xingjian Jiang, Zizhuang Deng
To prevent misinformation and social issues arising from trustworthy-looking content generated by LLMs, it is crucial to develop efficient and reliable methods for identifying the source of texts. Previous approaches have demonstrated exceptional performance in detecting texts fully generated by LLMs. However, these methods struggle when confronting more advanced LLM output or text with adversarial multi-task machine revision, especially in the black-box setting, where the generating model is unknown. To address this challenge, grounded in the hypothesis that human writing possesses distinctive stylistic patterns, we propose Human Language Preference Detection (HLPD). HLPD employs a reward-based alignment process, Human Language Preference Optimization (HLPO), to shift the scoring model's token distribution toward human-like writing, making the model more sensitive to human writing, therefore enhancing the identification of machine-revised text. We test HLPD in an adversarial multi-task evaluation framework that leverages a five-dimensional prompt generator and multiple advanced LLMs to create diverse revision scenarios. When detecting texts revised by GPT-series models, HLPD achieves a 15.11% relative improvement in AUROC over ImBD, surpassing Fast-DetectGPT by 45.56%. When evaluated on texts generated by advanced LLMs, HLPD achieves the highest average AUROC, exceeding ImBD by 5.53% and Fast-DetectGPT by 34.14%. Code will be made available at https://github.com/dfq2021/HLPD.
comment : 20 pages, 10 figures, accepted by AAAI'26
♻ ☆ A Reproducible Framework for Neural Topic Modeling in Focus Group Analysis
Heger Arfaoui, Mohammed Iheb Hergli, Beya Benzina, Slimane BenMiled
Focus group discussions generate rich qualitative data but their analysis traditionally relies on labor-intensive manual coding that limits scalability and reproducibility. We present a systematic framework for applying BERTopic to focus group transcripts using data from ten focus groups exploring HPV vaccine perceptions in Tunisia (1,075 utterances). We conducted comprehensive hyperparameter exploration across 27 configurations, evaluating each through bootstrap stability analysis, performance metrics, and comparison with LDA baseline. Bootstrap analysis revealed that stability metrics (NMI and ARI) exhibited strong disagreement (r = -0.691) and showed divergent relationships with coherence, demonstrating that stability is multifaceted rather than monolithic. Our multi-criteria selection framework yielded a 7-topic model achieving 18\% higher coherence than optimized LDA (0.573 vs. 0.486) with interpretable topics validated through independent human evaluation (ICC = 0.700, weighted Cohen's kappa = 0.678). These findings demonstrate that transformer-based topic modeling can extract interpretable themes from small focus group transcript corpora when systematically configured and validated, while revealing that quality metrics capture distinct, sometimes conflicting constructs requiring multi-criteria evaluation. We provide complete documentation and code to support reproducibility.
♻ ☆ Evaluating LLMs on Sequential API Call Through Automated Test Generation
Yuheng Huang, Jiayang Song, Da Song, Zhenlan Ji, Wenhan Wang, Shuai Wang, Lei Ma
By integrating tools from external APIs, Large Language Models (LLMs) have expanded their promising capabilities in a diverse spectrum of complex real-world tasks. However, testing, evaluation, and analysis of LLM tool use remain in their early stages. Most existing benchmarks rely on manually collected test cases, many of which cannot be automatically checked for semantic correctness and instead depend on static methods such as string matching. Additionally, these benchmarks often overlook the complex interactions that occur between sequential API calls, which are common in real-world applications. To fill the gap, in this paper, we introduce StateGen, an automated framework designed to generate diverse coding tasks involving sequential API interactions. StateGen combines state-machine-based API constraint solving and validation, energy-based sampling, and control-flow injection to generate executable programs. These programs are then translated into human-like natural language task descriptions through a collaboration of two LLM agents. Utilizing StateGen, we construct StateEval, a benchmark encompassing 120 verified test cases spanning across three representative scenarios: Session Service, Tensor Operation, and ElevenLabs MCP. Experimental results confirm that StateGen can effectively generate challenging and realistic API-oriented tasks, highlighting areas for improvement in current LLMs incorporating APIs.We make our framework and benchmark publicly available to support future research.
♻ ☆ Eka-Eval: An Evaluation Framework for Low-Resource Multilingual Large Language Models
Samridhi Raj Sinha, Rajvee Sheth, Abhishek Upperwal, Mayank Singh
The rapid evolution of Large Language Models' has underscored the need for evaluation frameworks that are globally applicable, flexible, and modular, and that support a wide range of tasks, model types, and linguistic settings. We introduce EKA-EVAL, a unified, end- to-end framework that combines a zero-code web interface and an interactive CLI to ensure broad accessibility. It integrates 50+ multilingual benchmarks across nine evaluation categories, supports local and proprietary models, and provides 11 core capabilities through a modular, plug-and-play architecture. Designed for scalable, multilingual evaluation with support for low-resource multilingual languages, EKA-EVAL is, to the best of our knowledge, the first suite to offer comprehensive coverage in a single platform. Comparisons against five existing baselines indicate improvements of at least 2x better on key usability measures, with the highest user satisfaction, faster setup times, and consistent benchmark reproducibility. The framework is open-source and publicly available at https://github.com/lingo-iitgn/eka-eval.
♻ ☆ Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMs
Edward Phillips, Sean Wu, Soheila Molaei, Danielle Belgrave, Anshul Thakur, David Clifton
Large language models demonstrate impressive results across diverse tasks but are still known to hallucinate, generating linguistically plausible but incorrect answers to questions. Uncertainty quantification has been proposed as a strategy for hallucination detection, requiring estimates for both global uncertainty (attributed to a batch of responses) and local uncertainty (attributed to individual responses). While recent black-box approaches have shown some success, they often rely on disjoint heuristics or graph-theoretic approximations that lack a unified geometric interpretation. We introduce a geometric framework to address this, based on archetypal analysis of batches of responses sampled with only black-box model access. At the global level, we propose Geometric Volume, which measures the convex hull volume of archetypes derived from response embeddings. At the local level, we propose Geometric Suspicion, which leverages the spatial relationship between responses and these archetypes to rank reliability, enabling hallucination reduction through preferential response selection. Unlike prior methods that rely on discrete pairwise comparisons, our approach provides continuous semantic boundary points which have utility for attributing reliability to individual responses. Experiments show that our framework performs comparably to or better than prior methods on short form question-answering datasets, and achieves superior results on medical datasets where hallucinations carry particularly critical risks. We also provide theoretical justification by proving a link between convex hull volume and entropy.
comment : Revision. Clarified positioning as a unified geometric framework for global and local uncertainty in LLMs. Added baselines (Degree, Eccentricity) and expanded comparison to related methods. Included ablations (PCA dimension, number of archetypes, number of samples) and complexity analysis. Extended discussion of medical QA results and model-specific behaviour
♻ ☆ On SkipGram Word Embedding Models with Negative Sampling: Unified Framework and Impact of Noise Distributions
Dezhi Liu, Richong Zhang, Ziqiao Wang
SkipGram word embedding models with negative sampling, or SGN in short, is an elegant family of word embedding models. In this paper, we formulate a framework for word embedding, referred to as Word-Context Classification (WCC), that generalizes SGN to a wide family of models. The framework, which uses some ``noise examples'', is justified through theoretical analysis. The impact of noise distribution on the learning of the WCC embedding models is studied experimentally, suggesting that the best noise distribution is, in fact, the data distribution, in terms of both the embedding performance and the speed of convergence during training. Along our way, we discover several novel embedding models that outperform existing WCC models.
comment : This work has been submitted to the IEEE for possible publication
♻ ☆ DESIGNER: Design-Logic-Guided Multidisciplinary Data Synthesis for LLM Reasoning
Weize Liu, Yongchi Zhao, Yijia Luo, Mingyu Xu, Jiaheng Liu, Yanan Li, Xiguo Hu, Zhiqi Bai, Yuchi Xu, Wenbo Su, Bo Zheng
Large language models (LLMs) have achieved remarkable success in many natural language tasks but still struggle with complex, multi-step reasoning, particularly across diverse disciplines. Existing reasoning datasets often lack disciplinary breadth, reasoning depth, and diversity, as well as guiding principles for question synthesis. We propose DESIGNER: a DESIGN-logic-guidEd Reasoning data synthesis pipeline that leverages naturally available, extensive raw documents (e.g., book corpus and web corpus) to generate multidisciplinary challenging questions. We introduce the concept of "design logic" and instruct LLMs to mimic human educators' question-creation process, enabling the automated synthesis of large-scale, high-difficulty questions. We use LLMs to reverse-engineer and abstract over 120,000 design logics from existing questions across various disciplines. By matching these design logics with source documents, we are able to generate reasoning questions with controllable question types and difficulty levels. Using this pipeline, we synthesized two large-scale reasoning datasets that span 75 disciplines: DLR-Book (3.04 million questions from the book corpus) and DLR-Web (1.66 million questions from the web corpus). Data analysis indicates that the questions synthesized by our method exhibit greater difficulty and diversity compared to those in the baseline datasets. We validate our synthesized data through supervised fine-tuning (SFT) on the Qwen3 and Llama3 model families. Our data substantially enhances their multidisciplinary reasoning capabilities, outperforming existing datasets. Notably, by applying SFT on the base versions of these models using only our data, we even surpass their official final models that have undergone the full post-training process.
♻ ☆ iMAD: Intelligent Multi- Agent Debate for Efficient and Accurate LLM Inference AAAI 2026
Wei Fan, JinYi Yoon, Bo Ji
Large Language Model (LLM) agent systems have advanced rapidly, driven by their strong generalization in zero-shot settings. To further enhance reasoning and accuracy on complex tasks, Multi-Agent Debate (MAD) has emerged as a promising framework that engages multiple LLM agents in structured debates to encourage diverse reasoning. However, triggering MAD for every query is inefficient, as it incurs substantial computational (token) cost and may even degrade accuracy by overturning correct single-agent answers. To address these limitations, we propose intelligent Multi-Agent Debate (iMAD), a token-efficient framework that selectively triggers MAD only when it is likely to be beneficial (i.e., correcting an initially wrong answer). To achieve this goal, iMAD learns generalizable model behaviors to make accurate debate decisions. Specifically, iMAD first prompts a single agent to produce a structured self-critique response, from which we extract 41 interpretable linguistic and semantic features capturing hesitation cues. Then, iMAD uses a lightweight debate-decision classifier, trained using our proposed FocusCal loss, to determine whether to trigger MAD, enabling robust debate decisions without test dataset-specific tuning. Through extensive experiments using six (visual) question answering datasets against five competitive baselines, we have shown that iMAD significantly reduces token usage (by up to 92%) while also improving final answer accuracy (by up to 13.5%).
comment : Accepted in AAAI 2026 (Oral)
♻ ☆ Multilingual Pretrain ing for Pixel Language Models EMNLP 2025
Ilker Kesen, Jonas F. Lotz, Ingo Ziegler, Phillip Rust, Desmond Elliott
Pixel language models operate directly on images of rendered text, eliminating the need for a fixed vocabulary. While these models have demonstrated strong capabilities for downstream cross-lingual transfer, multilingual pretraining remains underexplored. We introduce PIXEL-M4, a model pretrained on four visually and linguistically diverse languages: English, Hindi, Ukrainian, and Simplified Chinese. Multilingual evaluations on semantic and syntactic tasks show that PIXEL-M4 outperforms an English-only counterpart on non-Latin scripts. Word-level probing analyses confirm that PIXEL-M4 captures rich linguistic features, even in languages not seen during pretraining. Furthermore, an analysis of its hidden representations shows that multilingual pretraining yields a semantic embedding space closely aligned across the languages used for pretraining. This work demonstrates that multilingual pretraining substantially enhances the capability of pixel language models to effectively support a diverse set of languages.
comment : EMNLP 2025
♻ ☆ Scaling Towards the Information Boundary of Instruction Set: InfinityInstruct-Subject Technical Report
Li Du, Hanyu Zhao, Yiming Ju, Tengfei Pan
Instruction tuning has become a foundation for unlocking the capabilities of large-scale pretrained models and improving their performance on complex tasks. Thus, the construction of high-quality instruction datasets is crucial for enhancing model performance and generalizability. Although current instruction datasets have reached tens of millions of samples, models finetuned on them may still struggle with complex instruction following and tasks in rare domains. This is primarily due to limited expansion in both ``coverage'' (coverage of task types and knowledge areas) and ``depth'' (instruction complexity) of the instruction set. To address this issue, we propose a systematic instruction data construction framework, which integrates a hierarchical labeling system, an informative seed selection algorithm, an evolutionary data synthesis process, and a model deficiency diagnosis with targeted data generation. These components form an iterative closed-loop to continuously enhance the coverage and depth of instruction data. Based on this framework, we construct InfinityInstruct-Subject, a high-quality dataset containing ~1.5 million instructions. Experiments on multiple foundation models and benchmark tasks demonstrate its effectiveness in improving instruction-following capabilities. Further analyses suggest that InfinityInstruct-Subject shows enlarged coverage and depth compared to comparable synthesized instruction datasets. Our work lays a theoretical and practical foundation for the efficient, continuous evolution of instruction datasets, moving from data quantity expansion to qualitative improvement.
♻ ☆ Do Large Language Models Think Like the Brain? Sentence-Level Evidences from Layer-Wise Embeddings and fMRI AAAI 2026
Yu Lei, Xingyang Ge, Yi Zhang, Yiming Yang, Bolei Ma
Understanding whether large language models (LLMs) and the human brain converge on similar computational principles remains a fundamental and important question in cognitive neuroscience and AI. Do the brain-like patterns observed in LLMs emerge simply from scaling, or do they reflect deeper alignment with the architecture of human language processing? This study focuses on the sentence-level neural mechanisms of language models, systematically investigating how layer-wise representations in LLMs align with the dynamic neural responses during human sentence comprehension. By comparing hierarchical embeddings from 14 publicly available LLMs with fMRI data collected from participants, who were exposed to a naturalistic narrative story, we constructed sentence-level neural prediction models to identify the model layers most significantly correlated with brain region activations. Results show that improvements in model performance drive the evolution of representational architectures toward brain-like hierarchies, particularly achieving stronger functional and anatomical correspondence at higher semantic abstraction levels. These findings advance our understanding of the computational parallels between LLMs and the human brain, highlighting the potential of LLMs as models for human language processing.
comment : AAAI 2026
♻ ☆ Hyperdimensional Probe: Decoding LLM Representations via Vector Symbolic Architectures
Marco Bronzini, Carlo Nicolini, Bruno Lepri, Jacopo Staiano, Andrea Passerini
Despite their capabilities, Large Language Models (LLMs) remain opaque with limited understanding of their internal representations. Current interpretability methods either focus on input-oriented feature extraction, such as supervised probes and Sparse Autoencoders (SAEs), or on output distribution inspection, such as logit-oriented approaches. A full understanding of LLM vector spaces, however, requires integrating both perspectives, something existing approaches struggle with due to constraints on latent feature definitions. We introduce the Hyperdimensional Probe, a hybrid supervised probe that combines symbolic representations with neural probing. Leveraging Vector Symbolic Architectures (VSAs) and hypervector algebra, it unifies prior methods: the top-down interpretability of supervised probes, SAE's sparsity-driven proxy space, and output-oriented logit investigation. This allows deeper input-focused feature extraction while supporting output-oriented investigation. Our experiments show that our method consistently extracts meaningful concepts across LLMs, embedding sizes, and setups, uncovering concept-driven patterns in analogy-oriented inference and QA-focused text generation. By supporting joint input-output analysis, this work advances semantic understanding of neural representations while unifying the complementary perspectives of prior methods.
♻ ☆ XISM: an eXploratory and Interactive Graph Tool to Visualize and Evaluate Semantic Map Models
Zhu Liu, Zhen Hu, Lei Dai, Yu Xuan, Ying Liu
Semantic map models visualize systematic relations among semantic functions through graph structures and are widely used in linguistic typology. However, existing construction methods either depend on labor-intensive expert reasoning or on fully automated systems lacking expert involvement, creating a tension between scalability and interpretability. We introduce \textbf{XISM}, an interactive system that combines data-driven inference with expert knowledge. XISM generates candidate maps via a top-down procedure and allows users to iteratively refine edges in a visual interface, with real-time metric feedback. Experiments in three semantic domains and expert interviews show that XISM improves linguistic decision transparency and controllability in semantic-map construction while maintaining computational efficiency. XISM provides a collaborative approach for scalable and interpretable semantic-map building. The system\footnote{https://app.xism2025.xin/} , source code\footnote{https://github.com/hank317/XISM} , and demonstration video\footnote{https://youtu.be/m5laLhGn6Ys} are publicly available.
comment : Paper under review
♻ ☆ Unifying Linear-Time Attention via Latent Probabilistic Modelling
Rares Dolga, Lucas Maystre, Marius Cobzarenco, David Barber
Transformers have achieved state-of-the-art results across a range of domains, but their quadratic attention mechanism poses significant challenges for long-sequence modelling. Recent efforts to design linear-time attention mechanisms have yielded more scalable alternatives, yet often at the cost of performance, particularly on discrete data such as language. In this work, we revisit linear attention through the lens of probabilistic graphical models. We first show that standard linear attention can be interpreted as an undirected latent variable model, revealing a key limitation: the absence of directionality. To address this, we propose a novel directed parameterisation of linear attention that introduces an asymmetric structure, enabling an interpretation aligned with the causal and sequential nature of language. Our formulation integrates global latent-variable attention with local standard attention in a fully probabilistic framework. Additionally, we introduce a recurrent parameterisation of queries and keys that avoids reliance on relative positional encodings, often incompatible with linear attention. Experiments on language modelling benchmarks demonstrate that our model achieves competitive performance with standard attention and outperforms existing linear attention variants.
♻ ☆ Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs AACL 2025
Yohan Mathew, Ollie Matthews, Robert McCarthy, Joan Velja, Christian Schroeder de Witt, Dylan Cope, Nandi Schoots
The rapid proliferation of frontier model agents promises significant societal advances but also raises concerns about systemic risks arising from unsafe interactions. Collusion to the disadvantage of others has been identified as a central form of undesirable agent cooperation. The use of information hiding (steganography) in agent communications could render such collusion practically undetectable. This underscores the need for investigations into the possibility of such behaviours emerging and the robustness corresponding countermeasures. To investigate this problem we design two approaches -- a gradient-based reinforcement learning (GBRL) method and an in-context reinforcement learning (ICRL) method -- for reliably eliciting sophisticated LLM-generated linguistic text steganography. We demonstrate, for the first time, that unintended steganographic collusion in LLMs can arise due to mispecified reward incentives during training. Additionally, we find that standard mitigations -- both passive oversight of model outputs and active mitigation through communication paraphrasing -- are not fully effective at preventing this steganographic communication. Our findings imply that (i) emergence of steganographic collusion is a plausible concern that should be monitored and researched, and (ii) preventing emergence may require innovation in mitigation techniques.
comment : Camera-ready version. Oral presentation at IJCNLP-AACL 2025 (14th International Joint Conference on Natural Language Processing and 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics), Mumbai, India, December 20-24, 2025
♻ ☆ Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs
Adrian Arnaiz-Rodriguez, Miguel Baidal, Erik Derner, Jenn Layton Annable, Mark Ball, Mark Ince, Elvira Perez Vallejos, Nuria Oliver
Large language model-powered chatbots have transformed how people seek information, especially in high-stakes contexts like mental health. Despite their support capabilities, safe detection and response to crises such as suicidal ideation and self-harm are still unclear, hindered by the lack of unified crisis taxonomies and clinical evaluation standards. We address this by creating: (1) a taxonomy of six crisis categories; (2) a dataset of over 2,000 inputs from 12 mental health datasets, classified into these categories; and (3) a clinical response assessment protocol. We also use LLMs to identify crisis inputs and audit five models for response safety and appropriateness. First, we built a clinical-informed crisis taxonomy and evaluation protocol. Next, we curated 2,252 relevant examples from over 239,000 user inputs, then tested three LLMs for automatic classification. In addition, we evaluated five models for the appropriateness of their responses to a user's crisis, graded on a 5-point Likert scale from harmful (1) to appropriate (5). While some models respond reliably to explicit crises, risks still exist. Many outputs, especially in self-harm and suicidal categories, are inappropriate or unsafe. Different models perform variably; some, like gpt-5-nano and deepseek-v3.2-exp, have low harm rates, but others, such as gpt-4o-mini and grok-4-fast, generate more unsafe responses. All models struggle with indirect signals, default replies, and context misalignment. These results highlight the urgent need for better safeguards, crisis detection, and context-aware responses in LLMs. They also show that alignment and safety practices, beyond scale, are crucial for reliable crisis support. Our taxonomy, datasets, and evaluation methods support ongoing AI mental health research, aiming to reduce harm and protect vulnerable users.
♻ ☆ On the Temporal Question-Answering Capabilities of Large Language Models Over Anonymized Data
Alfredo Garrachón Ruiz, Tomás de la Rosa, Daniel Borrajo
The applicability of Large Language Models (LLMs) in temporal reasoning tasks over data that is not present during training is still a field that remains to be explored. In this paper we work on this topic, focusing on structured and semi-structured anonymized data. We not only develop a direct LLM pipeline, but also compare various methodologies and conduct an in-depth analysis. We identified and examined seventeen common temporal reasoning tasks in natural language, focusing on their algorithmic components. To assess LLM performance, we created the \textit{Reasoning and Answering Temporal Ability} dataset (RATA), featuring semi-structured anonymized data to ensure reliance on reasoning rather than on prior knowledge. We compared several methodologies, involving SoTA techniques such as Tree-of-Thought, self-reflexion and code execution, tuned specifically for this scenario. Our results suggest that achieving scalable and reliable solutions requires more than just standalone LLMs, highlighting the need for integrated approaches.
comment : 18 pages, 7 tables, 5 figures
♻ ☆ Agent -OM: Leveraging LLM Agent s for Ontology Matching
Zhangcheng Qiang, Weiqing Wang, Kerry Taylor
Ontology matching (OM) enables semantic interoperability between different ontologies and resolves their conceptual heterogeneity by aligning related entities. OM systems currently have two prevailing design paradigms: conventional knowledge-based expert systems and newer machine learning-based predictive systems. While large language models (LLMs) and LLM agents have revolutionised data engineering and have been applied creatively in many domains, their potential for OM remains underexplored. This study introduces a novel agent-powered LLM-based design paradigm for OM systems. With consideration of several specific challenges in leveraging LLM agents for OM, we propose a generic framework, namely Agent-OM (Agent for Ontology Matching), consisting of two Siamese agents for retrieval and matching, with a set of OM tools. Our framework is implemented in a proof-of-concept system. Evaluations of three Ontology Alignment Evaluation Initiative (OAEI) tracks over state-of-the-art OM systems show that our system can achieve results very close to the long-standing best performance on simple OM tasks and can significantly improve the performance on complex and few-shot OM tasks.
comment : 31 pages
♻ ☆ AI Text Detectors and the Misclassification of Slightly Polished Arabic Text
Saleh Almohaimeed, Saad Almohaimeed, Mousa Jari, Khaled A. Alobaid, Fahad Alotaibi
Many AI detection models have been developed to counter the presence of articles created by artificial intelligence (AI). However, if a human-authored article is slightly polished by AI, a shift will occur in the borderline decision of these AI detection models, leading them to consider it as AI-generated article. This misclassification may result in falsely accusing authors of AI plagiarism and harm the credibility of AI detectors. In English, some efforts were made to meet this challenge, but not in Arabic. In this paper, we generated two datasets. The first dataset contains 800 Arabic articles, half AI-generated and half human-authored. We used it to evaluate 14 Large Language models (LLMs) and commercial AI detectors to assess their ability in distinguishing between human-authored and AI-generated articles. The best 8 models were chosen to act as detectors for our primary concern, which is whether they would consider slightly polished human-authored text as AI-generated. The second dataset, Ar-APT, contains 400 Arabic human-authored articles polished by 10 LLMs using 4 polishing settings, totaling 16400 samples. We use it to evaluate the 8 nominated models and determine whether slight polishing will affect their performance. The results reveal that all AI detectors incorrectly attribute a significant number of articles to AI. The best performing LLM, Claude-4 Sonnet, achieved 83.51\%, its performance decreased to 57.63\% for articles slightly polished by LLaMA-3. Whereas the best performing commercial model, originality.AI, achieves 92\% accuracy, dropped to 12\% for articles slightly polished by Mistral or Gemma-3.
comment : (Submitted) Journal of Big Data
♻ ☆ ELSPR: Evaluator LLM Training Data Self-Purification on Non-Transitive Preferences via Tournament Graph Reconstruction AAAI 2026
Yan Yu, Yilun Liu, Minggui He, Shimin Tao, Weibin Meng, Xinhua Yang, Li Zhang, Hongxia Ma, Dengye Li, Daimeng Wei, Boxing Chen, Fuliang Li
Pairwise evaluation of large language models (LLMs) has become the dominant paradigm for benchmarking open-ended tasks, yet non-transitive preferences, where evaluators prefer A over B, B over C, but C over A, fundamentally undermine ranking reliability. We show that this critical issue stems largely from low-quality data that contains inherently ambiguous preference pairs. To address this challenge, we propose ELSPR, a principled graph-theoretic framework that models pairwise preferences as tournament graphs and systematically identifies problematic training data. ELSPR quantifies non-transitivity through strongly connected components (SCCs) analysis and measures overall preference clarity using a novel normalized directed graph structural entropy metric. Our filtering methodology selectively removes preference data that induce non-transitivity while preserving transitive preferences. Extensive experiments on the AlpacaEval benchmark demonstrate that models fine-tuned on ELSPR-filtered data achieve substantial improvements: a 13.8% reduction in non-transitivity, a 0.088 decrease in structural entropy, and significantly enhanced discriminative power in real-world evaluation systems. Human validation confirms that discarded data exhibit dramatically lower inter-annotator agreement (34.4% vs. 52.6%) and model-human consistency (51.2% vs. 80.6%) compared to cleaned data. These findings establish ELSPR as an effective data self-purification approach for developing more robust, consistent, and human-aligned LLM evaluation systems.
comment : Accepted by AAAI 2026
♻ ☆ LLM-as-a-Supervisor: Mistaken Therapeutic Behaviors Trigger Targeted Supervisory Feedback
Chen Xu, Zhenyu Lv, Tian Lan, Xianyang Wang, Luyao Ji, Leyang Cui, Minqiang Yang, Jian Shen, Qunxi Dong, Xiuling Liu, Juan Wang, Bin Hu
Although large language models (LLMs) hold significant promise in psychotherapy, their direct application in patient-facing scenarios raises ethical and safety concerns. Therefore, this work shifts towards developing an LLM as a supervisor to train real therapists. In addition to the privacy of clinical therapist training data, a fundamental contradiction complicates the training of therapeutic behaviors: clear feedback standards are necessary to ensure a controlled training system, yet there is no absolute "gold standard" for appropriate therapeutic behaviors in practice. In contrast, many common therapeutic mistakes are universal and identifiable, making them effective triggers for targeted feedback that can serve as clearer evidence. Motivated by this, we create a novel therapist-training paradigm: (1) guidelines for mistaken behaviors and targeted correction strategies are first established as standards; (2) a human-in-the-loop dialogue-feedback dataset is then constructed, where a mistake-prone agent intentionally makes standard mistakes during interviews naturally, and a supervisor agent locates and identifies mistakes and provides targeted feedback; (3) after fine-tuning on this dataset, the final supervisor model is provided for real therapist training. The detailed experimental results of automated, human and downstream assessments demonstrate that models fine-tuned on our dataset MATE, can provide high-quality feedback according to the clinical guideline, showing significant potential for the therapist training scenario.
comment : 10 pages, 5 figures
♻ ☆ The Necessity of Imperfection:Reversing Model Collapse via Simulating Cognitive Boundedness
Zhongjie Jiang
Although synthetic data is widely promoted as a remedy, its prevailing production paradigm -- one optimizing for statistical smoothness -- systematically removes the long-tail, cognitively grounded irregularities that characterize human text. Prolonged training on such statistically optimal but cognitively impoverished data accelerates model collapse. This paper proposes a paradigm shift: instead of imitating the surface properties of data, we simulate the cognitive processes that generate human text. We introduce the Prompt-driven Cognitive Computing Framework (PMCSF), whose core consists of a Cognitive State Decoder (CSD) that reverse-engineers unstructured text into structured cognitive vectors, and a Cognitive Text Encoder (CTE) that re-materializes these states into text enriched with human-typical imperfections via mathematically defined Cognitive Perturbation Operators. The framework is validated through a two-stage objective evaluation pipeline. First, in cognitive codec verification, CTE text yields a Jensen-Shannon divergence of 0.0614 from human text (vs. 0.4431 for standard LLM output), passes double-blind professional media review, and achieves an intraclass correlation coefficient ICC > 0.9 for cognitive profile alignment across heterogeneous models. Second, in functional gain evaluation, isomorphic stress tests in the A-share market show that strategies incorporating CTE-generated data reduce maximum drawdown by 47.4% during the 2015 crash and deliver 8.6% Defensive Alpha, exceeding transaction costs by a factor of 33. Our findings demonstrate that modelling human cognitive limitations -- not copying surface data -- enables synthetic data with genuine functional gain, offering a viable technical pathway toward resolving the AI data-collapse crisis.
comment : 38 pages,5 figures. Extended technical disclosure (Version 2.0) is attached as ancillary files, containing raw forensic logs of the "Silent Rupture"detection [May 2025], proprietary GARCH parameter ranges, and the linguistic micro-chaos injection protocols
♻ ☆ Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models
Shuxun Wang, Qingyu Yin, Chak Tou Leong, Qiang Zhang, Linyi Yang
Repetition curse is a phenomenon where Large Language Models (LLMs) generate repetitive sequences of tokens or cyclic sequences. While the repetition curse has been widely observed, its underlying mechanisms remain poorly understood. In this work, we investigate the role of induction heads--a specific type of attention head known for their ability to perform in-context learning--in driving this repetitive behavior. Specifically, we focus on the "toxicity" of induction heads, which we define as their tendency to dominate the model's output logits during repetition, effectively excluding other attention heads from contributing to the generation process. Our findings have important implications for the design and training of LLMs. By identifying induction heads as a key driver of the repetition curse, we provide a mechanistic explanation for this phenomenon and suggest potential avenues for mitigation. We also propose a technique with attention head regularization that could be employed to reduce the dominance of induction heads during generation, thereby promoting more diverse and coherent outputs.
comment : Need to be refined
♻ ☆ Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as- Judge Reinforcement Learning
Jiahao Yuan, Zhiqing Cui, Hanqing Wang, Yuansheng Gao, Yucheng Zhou, Usman Naseem
As web platforms evolve towards greater personalization and emotional complexity, conversational agents must transcend superficial empathy to demonstrate identity-aware emotional reasoning. However, existing systems face two limitations: (1) reliance on situation-centric datasets lacking persistent user identity, which hampers the capture of personalized affective nuances; and (2) dependence on opaque, coarse reward signals that hinder development of verifiable empathetic reasoning. To address these gaps, we introduce KardiaBench, a large-scale user-grounded benchmark comprising 178,080 QA pairs across 22,080 multi-turn conversations anchored to 671 real-world profiles. The dataset is constructed via a model-in-the-loop pipeline with iterative rubric-guided refinement to ensure psychological plausibility and persona consistency. This progressive empathy pipeline that integrates user comprehension, contextual reasoning, and emotion perception into conversations, followed by iterative critique and rubric-based refinement to ensure psychological plausibility, emotional fidelity, and persona consistency. Building on this, we propose Kardia-R1, a framework that trains models for interpretable, stepwise empathetic cognition. Kardia-R1 leverages Rubric-as-Judge Empathetic Reinforcement Learning (Rubric-ERL), a GRPO-based method that uses explainable, human-aligned rubric rewards to tightly couple user understanding, emotional inference, and supportive response generation. Extensive experiments across four LLM backbones demonstrate that Kardia-R1 consistently outperforms othet methods in emotion accuracy, empathy, relevance, persona consistency, and safety. Our dataset and model will be released at https://github.com/JhCircle/Kardia-R1.
♻ ☆ CT-GLIP: 3D Grounded Language-Image Pretrain ing with CT Scans and Radiology Reports for Full-Body Scenarios
Jingyang Lin, Yingda Xia, Jianpeng Zhang, Ke Yan, Kai Cao, Le Lu, Jiebo Luo, Ling Zhang
3D medical vision-language (VL) pretraining has shown potential in radiology by leveraging large-scale multimodal datasets with CT-report pairs. However, existing methods primarily rely on a global VL alignment directly adapted from 2D scenarios. The entire 3D image is transformed into one global embedding, resulting in a loss of sparse but critical semantics essential for accurately aligning with the corresponding diagnosis. To address this limitation, we propose CT-GLIP, a 3D Grounded Language-Image Pretrained model that constructs fine-grained CT-report pairs to enhance \textit{grounded} cross-modal contrastive learning, effectively aligning grounded visual features with precise textual descriptions. Leveraging the grounded cross-modal alignment, CT-GLIP improves performance across diverse downstream tasks and can even identify organs and abnormalities in a zero-shot manner using natural language. CT-GLIP is trained on a multimodal CT dataset comprising 44,011 organ-level CT-report pairs from 17,702 patients, covering 104 organs. Evaluation is conducted on four downstream tasks: zero-shot organ recognition (OR), zero-shot abnormality detection (AD), tumor detection (TD), and tumor segmentation (TS). Empirical results show that it outperforms its counterparts with global VL alignment. Compared to vanilla CLIP, CT-GLIP achieves average performance improvements of 15.1% of F1 score, 1.9% of AUC, and 3.2% of DSC for zero-shot AD, TD, and TS tasks, respectively. This study highlights the significance of grounded VL alignment in enabling 3D medical VL foundation models to understand sparse representations within CT scans.
♻ ☆ Pa rl AI Vote: A Web Platform for Analyzing Gender and Political Bias in Large Language Models
Wenjie Lin, Hange Liu, Yingying Zhuang, Xutao Mao, Jingwei Shi, Xudong Han, Tianyu Shi, Jinrui Yang
We present ParlAI Vote, an interactive web platform for exploring European Parliament debates and votes, and for testing LLMs on vote prediction and bias analysis. This web system connects debate topics, speeches, and roll-call outcomes, and includes rich demographic data such as gender, age, country, and political group. Users can browse debates, inspect linked speeches, compare real voting outcomes with predictions from frontier LLMs, and view error breakdowns by demographic group. Visualizing the EuroParlVote benchmark and its core tasks of gender classification and vote prediction, ParlAI Vote highlights systematic performance bias in state-of-the-art LLMs. It unifies data, models, and visual analytics in a single interface, lowering the barrier for reproducing findings, auditing behavior, and running counterfactual scenarios. This web platform also shows model reasoning, helping users see why errors occur and what cues the models rely on. It supports research, education, and public engagement with legislative decision-making, while making clear both the strengths and the limitations of current LLMs in political analysis.
comment : online demo: https://euro-parl-vote-demo.vercel.app/; Video: https://www.youtube.com/@Jinrui-sf2jg
♻ ☆ H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs
Cheng Gao, Huimin Chen, Chaojun Xiao, Zhiyi Chen, Zhiyuan Liu, Maosong Sun
Large language models (LLMs) frequently generate hallucinations -- plausible but factually incorrect outputs -- undermining their reliability. While prior work has examined hallucinations from macroscopic perspectives such as training data and objectives, the underlying neuron-level mechanisms remain largely unexplored. In this paper, we conduct a systematic investigation into hallucination-associated neurons (H-Neurons) in LLMs from three perspectives: identification, behavioral impact, and origins. Regarding their identification, we demonstrate that a remarkably sparse subset of neurons (less than $0.1\%$ of total neurons) can reliably predict hallucination occurrences, with strong generalization across diverse scenarios. In terms of behavioral impact, controlled interventions reveal that these neurons are causally linked to over-compliance behaviors. Concerning their origins, we trace these neurons back to the pre-trained base models and find that these neurons remain predictive for hallucination detection, indicating they emerge during pre-training. Our findings bridge macroscopic behavioral patterns with microscopic neural mechanisms, offering insights for developing more reliable LLMs.
comment : 20 pages, 4 figures
♻ ☆ Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, Hao Lin, Chencan Wu, Feng Hu, An Yang, Jingren Zhou, Junyang Lin
This paper proposes a novel formulation for reinforcement learning (RL) with large language models, explaining why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE. Specifically, through a first-order approximation, we show that this surrogate becomes increasingly valid only when both the training-inference discrepancy and policy staleness are minimized. This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training, including importance sampling correction, clipping, and particularly Routing Replay for Mixture-of-Experts (MoE) models. Through extensive experiments with a 30B MoE model totaling hundreds of thousands of GPU hours, we show that for on-policy training, the basic policy gradient algorithm with importance sampling correction achieves the highest training stability. When off-policy updates are introduced to accelerate convergence, combining clipping and Routing Replay becomes essential to mitigate the instability caused by policy staleness. Notably, once training is stabilized, prolonged optimization consistently yields comparable final performance regardless of cold-start initialization. We hope that the shared insights and the developed recipes for stable RL training will facilitate future research.
♻ ☆ BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-Wo rl d Cybersecurity Systems
Andy K. Zhang, Joey Ji, Celeste Menders, Riya Dulepet, Thomas Qin, Ron Y. Wang, Junrong Wu, Kyleen Liao, Jiliang Li, Jinghan Hu, Sara Hong, Nardos Demilew, Shivatmica Murgai, Jason Tran, Nishka Kacheria, Ethan Ho, Denis Liu, Lauren McLane, Olivia Bruvik, Dai-Rong Han, Seungwoo Kim, Akhil Vyas, Cuiyuanxiu Chen, Ryan Li, Weiran Xu, Jonathan Z. Ye, Prerit Choudhary, Siddharth M. Bhatia, Vikram Sivashankar, Yuxuan Bao, Dawn Song, Dan Boneh, Daniel E. Ho, Percy Liang
AI agents have the potential to significantly alter the cybersecurity landscape. Here, we introduce the first framework to capture offensive and defensive cyber-capabilities in evolving real-world systems. Instantiating this framework with BountyBench, we set up 25 systems with complex, real-world codebases. To capture the vulnerability lifecycle, we define three task types: Detect (detecting a new vulnerability), Exploit (exploiting a given vulnerability), and Patch (patching a given vulnerability). For Detect, we construct a new success indicator, which is general across vulnerability types and provides localized evaluation. We manually set up the environment for each system, including installing packages, setting up server(s), and hydrating database(s). We add 40 bug bounties, which are vulnerabilities with monetary awards from \$10 to \$30,485, covering 9 of the OWASP Top 10 Risks. To modulate task difficulty, we devise a new strategy based on information to guide detection, interpolating from identifying a zero day to exploiting a given vulnerability. We evaluate 10 agents: Claude Code, OpenAI Codex CLI with o3-high and o4-mini, and custom agents with o3-high, GPT-4.1, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet Thinking, Qwen3 235B A22B, Llama 4 Maverick, and DeepSeek-R1. Given up to three attempts, the top-performing agents are Codex CLI: o3-high (12.5% on Detect, mapping to \$3,720; 90% on Patch, mapping to \$14,152), Custom Agent: Claude 3.7 Sonnet Thinking (67.5% on Exploit), and Codex CLI: o4-mini (90% on Patch, mapping to \$14,422). Codex CLI: o3-high, Codex CLI: o4-mini, and Claude Code are more capable at defense, achieving higher Patch scores of 90%, 90%, and 87.5%, compared to Exploit scores of 47.5%, 32.5%, and 57.5% respectively; while the custom agents are relatively balanced between offense and defense, achieving Exploit scores of 17.5-67.5% and Patch scores of 25-60%.
comment : 113 pages
♻ ☆ LLMEval-3: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models
Ming Zhang, Yujiong Shen, Jingyi Deng, Yuhui Wang, Yue Zhang, Junzhe Wang, Shichun Liu, Shihan Dou, Huayu Sha, Qiyuan Peng, Changhao Jiang, Jingqi Tong, Yilong Wu, Zhihao Zhang, Mingqi Wu, Zhiheng Xi, Mingxu Chai, Tao Gui, Qi Zhang, Xuanjing Huang
Existing evaluation of Large Language Models (LLMs) on static benchmarks is vulnerable to data contamination and leaderboard overfitting, critical issues that obscure true model capabilities. To address this, we introduce LLMEval-3, a framework for dynamic evaluation of LLMs. LLMEval-3 is built on a proprietary bank of 220k graduate-level questions, from which it dynamically samples unseen test sets for each evaluation run. Its automated pipeline ensures integrity via contamination-resistant data curation, a novel anti-cheating architecture, and a calibrated LLM-as-a-judge process achieving 90% agreement with human experts, complemented by a relative ranking system for fair comparison. An 20-month longitudinal study of nearly 50 leading models reveals a performance ceiling on knowledge memorization and exposes data contamination vulnerabilities undetectable by static benchmarks. The framework demonstrates exceptional robustness in ranking stability and consistency, providing strong empirical validation for the dynamic evaluation paradigm. LLMEval-3 offers a robust and credible methodology for assessing the true capabilities of LLMs beyond leaderboard scores, promoting the development of more trustworthy evaluation standards.
♻ ☆ From Atomic to Composite: Reinforcement Learning Enables Generalization in Complementary Reasoning
Sitao Cheng, Xunjian Yin, Ruiwen Zhou, Yuxuan Li, Xinyi Wang, Liangming Pan, William Yang Wang, Victor Zhong
The mechanism by which RL contributes to reasoning capabilities-whether it incentivizes the synthesis of new skills or merely amplifies existing behaviors-remains a subject of intense debate. In this work, we investigate this question through the lens of Complementary Reasoning, a complex task that requires integrating internal parametric knowledge with external contextual information. Using a controlled synthetic dataset of human biographies, we strictly decouple this ability into two atomic skills: Parametric Reasoning (relying on internal knowledge) and Contextual Reasoning (depending on external information). To rigorously assess capability boundaries, we evaluate generalization across three distinct levels of difficulty: I.I.D., Composition, and Zero-shot settings. We find that while SFT is sufficient for in-distribution performance, it struggles with O.O.D. generalization, particularly in Zero-shot settings where relational combinations are novel. Crucially, we identify the SFT Generalization Paradox: Models supervised solely on the composite task achieve near-perfect in-distribution accuracy but collapse on out-of-distribution generalization, indicating their reliance on rote memorization of path shortcuts. In contrast, we find that RL acts as a reasoning synthesizer rather than a probability amplifier. However, we uncover a strict atomic prerequisite: RL can only synthesize these complex strategies if the base model has first mastered the independent atomic skills (Parametric and Contextual) via SFT. These findings challenge the view of RL as a mere amplifier, suggesting that given sufficient atomic foundations, RL can actively synthesize complex reasoning strategies from learned primitives without explicit supervision on such complex strategies. This indicates that decoupled atomic training followed by RL offers a scalable path to generalization for complex reasoning tasks.
comment : Work in Progress. Code and data will be available at https://github.com/sitaocheng/from_atomic_to_composite
♻ ☆ From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models
Charles Zhang, Benji Peng, Xintian Sun, Qian Niu, Junyu Liu, Keyu Chen, Ming Li, Pohsun Feng, Ziqian Bi, Ming Liu, Yichao Zhang, Xinyuan Song, Cheng Fei, Caitlyn Heqi Yin, Lawrence KQ Yan, Hongyang He, Tianyang Wang
Word embeddings and language models have transformed natural language processing (NLP) by facilitating the representation of linguistic elements in continuous vector spaces. This review visits foundational concepts such as the distributional hypothesis and contextual similarity, tracing the evolution from sparse representations like one-hot encoding to dense embeddings including Word2Vec, GloVe, and fastText. We examine both static and contextualized embeddings, underscoring advancements in models such as ELMo, BERT, and GPT and their adaptations for cross-lingual and personalized applications. The discussion extends to sentence and document embeddings, covering aggregation methods and generative topic models, along with the application of embeddings in multimodal domains, including vision, robotics, and cognitive science. Advanced topics such as model compression, interpretability, numerical encoding, and bias mitigation are analyzed, addressing both technical challenges and ethical implications. Additionally, we identify future research directions, emphasizing the need for scalable training techniques, enhanced interpretability, and robust grounding in non-textual modalities. By synthesizing current methodologies and emerging trends, this survey offers researchers and practitioners an in-depth resource to push the boundaries of embedding-based language models.
comment : 21 pages
♻ ☆ OmniBench: Towards The Future of Universal Omni-Language Models
Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang, Chenghua Lin
Recent advancements in multimodal large language models (MLLMs) have focused on integrating multiple modalities, yet their ability to simultaneously process and reason across different inputs remains underexplored. We introduce OmniBench, a novel benchmark designed to evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. We define language models capable of such tri-modal processing as omni-language models (OLMs). OmniBench features high-quality human annotations that require integrated understanding across all modalities. Our evaluation reveals that: i) open-source OLMs show significant limitations in instruction-following and reasoning in tri-modal contexts; and ii) most baseline models perform poorly (around 50% accuracy) even with textual alternatives to image/audio inputs. To address these limitations, we develop OmniInstruct, an 96K-sample instruction tuning dataset for training OLMs. We advocate for developing more robust tri-modal integration techniques and training strategies to enhance OLM performance. Codes and data could be found at our repo (https://github.com/multimodal-art-projection/OmniBench).
♻ ☆ SQLBarber: A System Leveraging Large Language Models to Generate Customized and Realistic SQL Workloads SIGMOD 2026
Jiale Lao, Immanuel Trummer
Database research and development often require a large number of SQL queries for benchmarking purposes. However, acquiring real-world SQL queries is challenging due to privacy concerns, and existing SQL generation methods are limited in customization and in satisfying realistic constraints. To address this issue, we present SQLBarber, a system based on Large Language Models (LLMs) to generate customized and realistic SQL workloads. SQLBarber (i) eliminates the need for users to manually craft SQL templates in advance, while providing the flexibility to accept natural language specifications to constrain SQL templates, (ii) scales efficiently to generate large volumes of queries matching any user-defined cost distribution (e.g., cardinality and execution plan cost), and (iii) uses execution statistics from Amazon Redshift and Snowflake to derive SQL template specifications and query cost distributions that reflect real-world query characteristics. SQLBarber introduces (i) a declarative interface for users to effortlessly generate customized SQL templates, (ii) an LLM-powered pipeline augmented with a self-correction module that profiles, refines, and prunes SQL templates based on query costs, and (iii) a Bayesian Optimizer to efficiently explore different predicate values and identify a set of queries that satisfy the target cost distribution. We construct and open-source ten benchmarks of varying difficulty levels and target query cost distributions based on real-world statistics from Snowflake and Amazon Redshift. Extensive experiments on these benchmarks show that SQLBarber is the only system that can generate customized SQL templates. It reduces query generation time by one to three orders of magnitude, and significantly improves alignment with the target cost distribution, compared with existing methods.
comment : Accepted by SIGMOD 2026; extended version with appendix
♻ ☆ HeavyWater and SimplexWater: Distortion-Free LLM Watermarks for Low-Entropy Next-Token Predictions NeurIPS2025
Dor Tsur, Carol Xuan Long, Claudio Mayrink Verdun, Hsiang Hsu, Chen-Fu Chen, Haim Permuter, Sajani Vithana, Flavio P. Calmon
Large language model (LLM) watermarks enable authentication of text provenance, curb misuse of machine-generated text, and promote trust in AI systems. Current watermarks operate by changing the next-token predictions output by an LLM. The updated (i.e., watermarked) predictions depend on random side information produced, for example, by hashing previously generated tokens. LLM watermarking is particularly challenging in low-entropy generation tasks -- such as coding -- where next-token predictions are near-deterministic. In this paper, we propose an optimization framework for watermark design. Our goal is to understand how to most effectively use random side information in order to maximize the likelihood of watermark detection and minimize the distortion of generated text. Our analysis informs the design of two new watermarks: HeavyWater and SimplexWater. Both watermarks are tunable, gracefully trading-off between detection accuracy and text distortion. They can also be applied to any LLM and are agnostic to side information generation. We examine the performance of HeavyWater and SimplexWater through several benchmarks, demonstrating that they can achieve high watermark detection accuracy with minimal compromise of text generation quality, particularly in the low-entropy regime. Our theoretical analysis also reveals surprising new connections between LLM watermarking and coding theory. The code implementation can be found in https://github.com/DorTsur/HeavyWater_SimplexWater
comment : Presented at NeurIPS2025
♻ ☆ Mechanisms of Symbol Processing for In- Context Learning in Transformer Networks
Paul Smolensky, Roland Fernandez, Zhenghao Herbert Zhou, Mattia Opper, Adam Davies, Jianfeng Gao
Large Language Models (LLMs) have demonstrated impressive abilities in symbol processing through in-context learning (ICL). This success flies in the face of decades of critiques asserting that artificial neural networks cannot master abstract symbol manipulation. We seek to understand the mechanisms that can enable robust symbol processing in transformer networks, illuminating both the unanticipated success, and the significant limitations, of transformers in symbol processing. Borrowing insights from symbolic AI and cognitive science on the power of Production System architectures, we develop a high-level Production System Language, PSL, that allows us to write symbolic programs to do complex, abstract symbol processing, and create compilers that precisely implement PSL programs in transformer networks which are, by construction, 100% mechanistically interpretable. The work is driven by study of a purely abstract (semantics-free) symbolic task that we develop, Templatic Generation (TGT). Although developed through study of TGT, PSL is, we demonstrate, highly general: it is Turing Universal. The new type of transformer architecture that we compile from PSL programs suggests a number of paths for enhancing transformers' capabilities at symbol processing. We note, however, that the work we report addresses computability, and not learnability, by transformer networks. Note: The first section provides an extended synopsis of the entire paper.
♻ ☆ Characterizing the Expressivity of Fixed-Precision Transformer Language Models NeurIPS 2025
Jiaoda Li, Ryan Cotterell
Transformer-based language models (LMs) have achieved widespread empirical success, but their theoretical expressive power remains only partially understood. In this work, we analyze a restricted idealization of fixed-precision transformers with strict future masking, soft attention, and no positional encodings. We establish that this class of models is exactly as expressive as a specific fragment of linear temporal logic that contains only a single temporal operator: the past operator. We further connect this fragment to established classes in formal language theory, automata theory, and algebra, yielding a unified framework for understanding transformer expressivity under this idealization. Finally, we present empirical results that align closely with our theory: transformers trained on languages within their characterized expressive capacity generalize reliably across sequence lengths, while they consistently fail to generalize on languages beyond it.
comment : NeurIPS 2025 (Spotlight)
♻ ☆ Let Them Down Easy! Context ual Effects of LLM Guardrails on User Perceptions and Preferences EMNLP 2025
Mingqian Zheng, Wenjia Hu, Patrick Zhao, Motahhare Eslami, Jena D. Hwang, Faeze Brahman, Carolyn Rose, Maarten Sap
Current LLMs are trained to refuse potentially harmful input queries regardless of whether users actually had harmful intents, causing a tradeoff between safety and user experience. Through a study of 480 participants evaluating 3,840 query-response pairs, we examine how different refusal strategies affect user perceptions across varying motivations. Our findings reveal that response strategy largely shapes user experience, while actual user motivation has negligible impact. Partial compliance -- providing general information without actionable details -- emerges as the optimal strategy, reducing negative user perceptions by over 50% to flat-out refusals. Complementing this, we analyze response patterns of 9 state-of-the-art LLMs and evaluate how 6 reward models score different refusal strategies, demonstrating that models rarely deploy partial compliance naturally and reward models currently undervalue it. This work demonstrates that effective guardrails require focusing on crafting thoughtful refusals rather than detecting intent, offering a path toward AI safety mechanisms that ensure both safety and sustained user engagement.
comment : Accepted to Findings of EMNLP 2025
♻ ☆ Astra: A Multi- Agent System for GPU Kernel Performance Optimization
Anjiang Wei, Tianran Sun, Yogesh Seenichamy, Hang Song, Anne Ouyang, Azalia Mirhoseini, Ke Wang, Alex Aiken
GPU kernel optimization has long been a central challenge at the intersection of high-performance computing and machine learning. Efficient kernels are crucial for accelerating large language model (LLM) training and serving, yet attaining high performance typically requires extensive manual tuning. Compiler-based systems reduce some of this burden, but still demand substantial manual design and engineering effort. Recently, researchers have explored using LLMs for GPU kernel generation, though prior work has largely focused on translating high-level PyTorch modules into CUDA code. In this work, we introduce Astra, the first LLM-based multi-agent system for GPU kernel optimization. Unlike previous approaches, Astra starts from existing CUDA implementations extracted from SGLang, a widely deployed framework for serving LLMs, rather than treating PyTorch modules as the specification. Within Astra, specialized LLM agents collaborate through iterative code generation, testing, profiling, and planning to produce kernels that are both correct and high-performance. On kernels from SGLang, Astra achieves an average speedup of 1.32x using zero-shot prompting with OpenAI o4-mini. A detailed case study further demonstrates that LLMs can autonomously apply loop transformations, optimize memory access patterns, exploit CUDA intrinsics, and leverage fast math operations to yield substantial performance gains. Our work highlights multi-agent LLM systems as a promising new paradigm for GPU kernel optimization. Our code is publicly available at https://github.com/Anjiang-Wei/Astra.
♻ ☆ CryptoBench: A Dynamic Benchmark for Expert-Level Evaluation of LLM Agent s in Cryptocurrency
Jiacheng Guo, Suozhi Huang, Zixin Yao, Yifan Zhang, Yifu Lu, Jiashuo Liu, Zihao Li, Nicholas Deng, Qixin Xiao, Jia Tian, Kanghong Zhan, Tianyi Li, Xiaochen Liu, Jason Ge, Chaoyang He, Kaixuan Huang, Lin Yang, Wenhao Huang, Mengdi Wang
This paper introduces CryptoBench, the first expert-curated, dynamic benchmark designed to rigorously evaluate the real-world capabilities of Large Language Model (LLM) agents in the uniquely demanding and fast-paced cryptocurrency domain. Unlike general-purpose agent benchmarks for search and prediction, professional crypto analysis presents specific challenges: \emph{extreme time-sensitivity}, \emph{a highly adversarial information environment}, and the critical need to synthesize data from \emph{diverse, specialized sources}, such as on-chain intelligence platforms and real-time Decentralized Finance (DeFi) dashboards. CryptoBench thus serves as a much more challenging and valuable scenario for LLM agent assessment. To address these challenges, we constructed a live, dynamic benchmark featuring 50 questions per month, expertly designed by crypto-native professionals to mirror actual analyst workflows. These tasks are rigorously categorized within a four-quadrant system: Simple Retrieval, Complex Retrieval, Simple Prediction, and Complex Prediction. This granular categorization enables a precise assessment of an LLM agent's foundational data-gathering capabilities alongside its advanced analytical and forecasting skills. Our evaluation of ten LLMs, both directly and within an agentic framework, reveals a performance hierarchy and uncovers a failure mode. We observe a \textit{retrieval-prediction imbalance}, where many leading models, despite being proficient at data retrieval, demonstrate a pronounced weakness in tasks requiring predictive analysis. This highlights a problematic tendency for agents to appear factually grounded while lacking the deeper analytical capabilities to synthesize information.
Artificial Intelligence 246
☆ EfficientFlow: Efficient Equivariant Flow Policy Learning for Embodied AI AAAI 2026
Jianlei Chang, Ruofeng Mei, Wei Ke, Xiangyu Xu
Generative modeling has recently shown remarkable promise for visuomotor policy learning, enabling flexible and expressive control across diverse embodied AI tasks. However, existing generative policies often struggle with data inefficiency, requiring large-scale demonstrations, and sampling inefficiency, incurring slow action generation during inference. We introduce EfficientFlow, a unified framework for efficient embodied AI with flow-based policy learning. To enhance data efficiency, we bring equivariance into flow matching. We theoretically prove that when using an isotropic Gaussian prior and an equivariant velocity prediction network, the resulting action distribution remains equivariant, leading to improved generalization and substantially reduced data demands. To accelerate sampling, we propose a novel acceleration regularization strategy. As direct computation of acceleration is intractable for marginal flow trajectories, we derive a novel surrogate loss that enables stable and scalable training using only conditional trajectories. Across a wide range of robotic manipulation benchmarks, the proposed algorithm achieves competitive or superior performance under limited data while offering dramatically faster inference. These results highlight EfficientFlow as a powerful and efficient paradigm for high-performance embodied AI.
comment : Accepted by AAAI 2026. Project Page: https://efficientflow.github.io/
☆ A Diffusion Model Framework for Maximum Entropy Reinforcement Learning
Sebastian Sanokowski, Kaustubh Patil, Alois Knoll
Diffusion models have achieved remarkable success in data-driven learning and in sampling from complex, unnormalized target distributions. Building on this progress, we reinterpret Maximum Entropy Reinforcement Learning (MaxEntRL) as a diffusion model-based sampling problem. We tackle this problem by minimizing the reverse Kullback-Leibler (KL) divergence between the diffusion policy and the optimal policy distribution using a tractable upper bound. By applying the policy gradient theorem to this objective, we derive a modified surrogate objective for MaxEntRL that incorporates diffusion dynamics in a principled way. This leads to simple diffusion-based variants of Soft Actor-Critic (SAC), Proximal Policy Optimization (PPO) and Wasserstein Policy Optimization (WPO), termed DiffSAC, DiffPPO and DiffWPO. All of these methods require only minor implementation changes to their base algorithm. We find that on standard continuous control benchmarks, DiffSAC, DiffPPO and DiffWPO achieve better returns and higher sample efficiency than SAC and PPO.
☆ Visual Sync: Multi-Camera Synchronization via Cross-View Object Motion NeurIPS 2025
Shaowei Liu, David Yifan Yao, Saurabh Gupta, Shenlong Wang
Today, people can easily record memorable moments, ranging from concerts, sports events, lectures, family gatherings, and birthday parties with multiple consumer cameras. However, synchronizing these cross-camera streams remains challenging. Existing methods assume controlled settings, specific targets, manual correction, or costly hardware. We present VisualSync, an optimization framework based on multi-view dynamics that aligns unposed, unsynchronized videos at millisecond accuracy. Our key insight is that any moving 3D point, when co-visible in two cameras, obeys epipolar constraints once properly synchronized. To exploit this, VisualSync leverages off-the-shelf 3D reconstruction, feature matching, and dense tracking to extract tracklets, relative poses, and cross-view correspondences. It then jointly minimizes the epipolar error to estimate each camera's time offset. Experiments on four diverse, challenging datasets show that VisualSync outperforms baseline methods, achieving an median synchronization error below 50 ms.
comment : Accepted to NeurIPS 2025. Project page: https://stevenlsw.github.io/visualsync/
☆ Learning Sim-to-Real Humanoid Locomotion in 15 Minutes
Younggyo Seo, Carmelo Sferrazza, Juyue Chen, Guanya Shi, Rocky Duan, Pieter Abbeel
Massively parallel simulation has reduced reinforcement learning (RL) training time for robots from days to minutes. However, achieving fast and reliable sim-to-real RL for humanoid control remains difficult due to the challenges introduced by factors such as high dimensionality and domain randomization. In this work, we introduce a simple and practical recipe based on off-policy RL algorithms, i.e., FastSAC and FastTD3, that enables rapid training of humanoid locomotion policies in just 15 minutes with a single RTX 4090 GPU. Our simple recipe stabilizes off-policy RL algorithms at massive scale with thousands of parallel environments through carefully tuned design choices and minimalist reward functions. We demonstrate rapid end-to-end learning of humanoid locomotion controllers on Unitree G1 and Booster T1 robots under strong domain randomization, e.g., randomized dynamics, rough terrain, and push perturbations, as well as fast training of whole-body human-motion tracking policies. We provide videos and open-source implementation at: https://younggyo.me/fastsac-humanoid.
comment : Project website: https://younggyo.me/fastsac-humanoid
☆ RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policies
Guillermo Garcia-Cobo, Maximilian Igl, Peter Karkus, Zhejun Zhang, Michael Watson, Yuxiao Chen, Boris Ivanovic, Marco Pavone
Autonomous driving policies are typically trained via open-loop behavior cloning of human demonstrations. However, such policies suffer from covariate shift when deployed in closed loop, leading to compounding errors. We introduce Rollouts as Demonstrations (RoaD), a simple and efficient method to mitigate covariate shift by leveraging the policy's own closed-loop rollouts as additional training data. During rollout generation, RoaD incorporates expert guidance to bias trajectories toward high-quality behavior, producing informative yet realistic demonstrations for fine-tuning. This approach enables robust closed-loop adaptation with orders of magnitude less data than reinforcement learning, and avoids restrictive assumptions of prior closed-loop supervised fine-tuning (CL-SFT) methods, allowing broader applications domains including end-to-end driving. We demonstrate the effectiveness of RoaD on WOSAC, a large-scale traffic simulation benchmark, where it performs similar or better than the prior CL-SFT method; and in AlpaSim, a high-fidelity neural reconstruction-based simulator for end-to-end driving, where it improves driving score by 41\% and reduces collisions by 54\%.
comment : Preprint
☆ LLM CHESS: Benchmark ing Reasoning and Instruction-Following in LLMs through Chess
Sai Kolasani, Maxim Saplin, Nicholas Crispino, Kyle Montgomery, Jared Quincy Davis, Matei Zaharia, Chi Wang, Chenguang Wang
We introduce LLM CHESS, an evaluation framework designed to probe the generalization of reasoning and instruction-following abilities in large language models (LLMs) through extended agentic interaction in the domain of chess. We rank over 50 open and closed source models by playing against a random opponent using a range of behavioral metrics, including win and loss rates, move quality, move legality, hallucinated actions, and game duration. For a subset of top reasoning models, we derive an Elo estimate by playing against a chess engine with variably configured skill, which allows for comparisons between models in an easily understandable way. Despite the simplicity of the instruction-following task and the weakness of the opponent, many state-of-the-art models struggle to complete games or achieve consistent wins. Similar to other benchmarks on complex reasoning tasks, our experiments reveal a clear separation between reasoning and non-reasoning models. However, unlike existing static benchmarks, the stochastic and dynamic nature of LLM CHESS uniquely reduces overfitting and memorization while preventing benchmark saturation, proving difficult even for top reasoning models. To support future work on evaluating reasoning and instruction-following in LLMs, we release our experimental framework, a public leaderboard, and a dataset of associated games.
☆ Forecasting in Offline Reinforcement Learning for Non-stationary Environments NeurIPS 2025
Suzan Ece Ada, Georg Martius, Emre Ugur, Erhan Oztop
Offline Reinforcement Learning (RL) provides a promising avenue for training policies from pre-collected datasets when gathering additional interaction data is infeasible. However, existing offline RL methods often assume stationarity or only consider synthetic perturbations at test time, assumptions that often fail in real-world scenarios characterized by abrupt, time-varying offsets. These offsets can lead to partial observability, causing agents to misperceive their true state and degrade performance. To overcome this challenge, we introduce Forecasting in Non-stationary Offline RL (FORL), a framework that unifies (i) conditional diffusion-based candidate state generation, trained without presupposing any specific pattern of future non-stationarity, and (ii) zero-shot time-series foundation models. FORL targets environments prone to unexpected, potentially non-Markovian offsets, requiring robust agent performance from the onset of each episode. Empirical evaluations on offline RL benchmarks, augmented with real-world time-series data to simulate realistic non-stationarity, demonstrate that FORL consistently improves performance compared to competitive baselines. By integrating zero-shot forecasting with the agent's experience, we aim to bridge the gap between offline RL and the complexities of real-world, non-stationary environments.
comment : The Thirty-Ninth Annual Conference on Neural Information Processing Systems, NeurIPS 2025
☆ Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback
Aiden Yiliu Li, Bizhi Yu, Daoan Lei, Tianhe Ren, Shilong Liu
GUI grounding aims to align natural language instructions with precise regions in complex user interfaces. Advanced multimodal large language models show strong ability in visual GUI grounding but still struggle with small or visually similar targets and ambiguity in real world layouts. These limitations arise from limited grounding capacity and from underuse of existing reasoning potential. We present Chain of Ground CoG a training free multi step grounding framework that uses multimodal large language models for iterative visual reasoning and refinement. Instead of direct prediction the model progressively reflects and adjusts its hypotheses leading to more accurate and interpretable localization. Our approach achieves 68.4 accuracy on the ScreenSpot Pro benchmark an improvement of 4.8 points. To measure real world generalization we introduce TPanel UI a dataset of 420 labeled industrial control panels with visual distortions such as blur and masking. On TPanel UI Chain of Ground improves over the strong baseline Qwen3 VL 235B by 6.9 points showing the effectiveness of multi step training free grounding across real world and digital interfaces. These results highlight a direction for unlocking grounding potential through structured iterative refinement instead of additional training.
☆ AI-Driven Optimization under Uncertainty for Mineral Processing Operations
William Xu, Amir Eskanlou, Mansur Arief, David Zhen Yin, Jef K. Caers
The global capacity for mineral processing must expand rapidly to meet the demand for critical minerals, which are essential for building the clean energy technologies necessary to mitigate climate change. However, the efficiency of mineral processing is severely limited by uncertainty, which arises from both the variability of feedstock and the complexity of process dynamics. To optimize mineral processing circuits under uncertainty, we introduce an AI-driven approach that formulates mineral processing as a Partially Observable Markov Decision Process (POMDP). We demonstrate the capabilities of this approach in handling both feedstock uncertainty and process model uncertainty to optimize the operation of a simulated, simplified flotation cell as an example. We show that by integrating the process of information gathering (i.e., uncertainty reduction) and process optimization, this approach has the potential to consistently perform better than traditional approaches at maximizing an overall objective, such as net present value (NPV). Our methodological demonstration of this optimization-under-uncertainty approach for a synthetic case provides a mathematical and computational framework for later real-world application, with the potential to improve both the laboratory-scale design of experiments and industrial-scale operation of mineral processing circuits without any additional hardware.
comment : 27 pages, 13 figures, submitted to Sustainable Earth Resources Communications (SERC)
☆ From Atomic to Composite: Reinforcement Learning Enables Generalization in Complementary Reasoning
Sitao Cheng, Xunjian Yin, Ruiwen Zhou, Yuxuan Li, Xinyi Wang, Liangming Pan, William Yang Wang, Victor Zhong
The mechanism by which RL contributes to reasoning capabilities-whether it incentivizes the synthesis of new skills or merely amplifies existing behaviors-remains a subject of intense debate. In this work, we investigate this question through the lens of Complementary Reasoning, a complex task that requires integrating internal parametric knowledge with external contextual information. Using a controlled synthetic dataset of human biographies, we strictly decouple this ability into two atomic skills: Parametric Reasoning (relying on internal knowledge) and Contextual Reasoning (depending on external information). To rigorously assess capability boundaries, we evaluate generalization across three distinct levels of difficulty: I.I.D., Composition, and Zero-shot settings. We find that while SFT is sufficient for in-distribution performance, it struggles with O.O.D. generalization, particularly in Zero-shot settings where relational combinations are novel. Crucially, we identify the SFT Generalization Paradox: Models supervised solely on the composite task achieve near-perfect in-distribution accuracy but collapse on out-of-distribution generalization, indicating their reliance on rote memorization of path shortcuts. In contrast, we find that RL acts as a reasoning synthesizer rather than a probability amplifier. However, we uncover a strict atomic prerequisite: RL can only synthesize these complex strategies if the base model has first mastered the independent atomic skills (Parametric and Contextual) via SFT. These findings challenge the view of RL as a mere amplifier, suggesting that given sufficient atomic foundations, RL can actively synthesize complex reasoning strategies from learned primitives without explicit supervision on such complex strategies. This indicates that decoupled atomic training followed by RL offers a scalable path to generalization for complex reasoning tasks.
comment : Work in Progress. Code and data will be available at https://github.com/sitaocheng/from_atomic_to_composite
☆ Learned-Rule-Augmented Large Language Model Evaluators
Jie Meng, Jin Mao
Large language models (LLMs) are predominantly used as evaluators for natural language generation (NLG) tasks, but their application to broader evaluation scenarios remains limited. In this work, we explore the potential of LLMs as general evaluators across diverse tasks. Although LLM-based evaluators have made progress in different areas, existing methods struggle to generalize due to their reliance on costly, human-designed evaluation principles, which are often misaligned with both annotated data and LLMs' understanding.To address these challenges, we propose a rule-augmented evaluation paradigm. First, we introduce a rule distillation method that automatically extracts scoring rules from data using an LLM-assisted Monte Carlo Tree Search (MCTS), alleviating scalability issues and improving alignment with data. Second, to enable LLMs to effectively apply the learned rules, we propose two strategies: (1) Chain-of-Rule (CoR), which guides LLM to follow distilled rules, and (2) training a rule-augmented LLM evaluator (RuAE) via reinforcement learning, further bridging the gap between rules and LLMs' reasoning. Extensive experiments on diverse tasks demonstrate the effectiveness and generalizability of our approach across various evaluation scenarios.
☆ GrndCt rl : Grounding Wo rl d Models via Self-Supervised Reward Alignment
Haoyang He, Jay Patrikar, Dong-Ki Kim, Max Smith, Daniel McGann, Ali-akbar Agha-mohammadi, Shayegan Omidshafiei, Sebastian Scherer
Recent advances in video world modeling have enabled large-scale generative models to simulate embodied environments with high visual fidelity, providing strong priors for prediction, planning, and control. Yet, despite their realism, these models often lack geometric grounding, limiting their use in navigation tasks that require spatial coherence and long-horizon stability. We introduce Reinforcement Learning with World Grounding (RLWG), a self-supervised post-training framework that aligns pretrained world models with a physically verifiable structure through geometric and perceptual rewards. Analogous to reinforcement learning from verifiable feedback (RLVR) in language models, RLWG can use multiple rewards that measure pose cycle-consistency, depth reprojection, and temporal coherence. We instantiate this framework with GrndCtrl, a reward-aligned adaptation method based on Group Relative Policy Optimization (GRPO), yielding world models that maintain stable trajectories, consistent geometry, and reliable rollouts for embodied navigation. Like post-training alignment in large language models, GrndCtrl leverages verifiable rewards to bridge generative pretraining and grounded behavior, achieving superior spatial coherence and navigation stability over supervised fine-tuning in outdoor environments.
☆ Agent ic Policy Optimization via Instruction-Policy Co-Evolution
Han Zhou, Xingchen Wan, Ivan Vulić, Anna Korhonen
Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capability of large language models (LLMs), enabling autonomous agents that can conduct effective multi-turn and tool-integrated reasoning. While instructions serve as the primary protocol for defining agents, RLVR typically relies on static and manually designed instructions. However, those instructions may be suboptimal for the base model, and the optimal instruction may change as the agent's policy improves and explores the interaction with the environment. To bridge the gap, we introduce INSPO, a novel Instruction-Policy co-evolution framework that integrates instruction optimization as a dynamic component of the reinforcement learning (RL) loop. INSPO maintains a dynamic population of instruction candidates that are sampled with questions, where reward signals in RL loops are automatically attributed to each instruction, and low performers are periodically pruned. New instructions are generated and verified through an on-policy reflection mechanism, where an LLM-based optimizer analyzes past experience from a replay buffer and evolves more effective strategies given the current policy. We conduct extensive experiments on multi-turn retrieval and reasoning tasks, demonstrating that INSPO substantially outperforms strong baselines relying on static instructions. INSPO discovers innovative instructions that guide the agent toward more strategic reasoning paths, achieving substantial performance gains with only a marginal increase in computational overhead.
comment : 10 pages, 3 figures, 2 tables (18 pages including references and appendices)
☆ An Empirical Study of Agent Developer Practices in AI Agent Frameworks
Yanlin Wang, Xinyi Xu, Jiachi Chen, Tingting Bi, Wenchao Gu, Zibin Zheng
The rise of large language models (LLMs) has sparked a surge of interest in agents, leading to the rapid growth of agent frameworks. Agent frameworks are software toolkits and libraries that provide standardized components, abstractions, and orchestration mechanisms to simplify agent development. Despite widespread use of agent frameworks, their practical applications and how they influence the agent development process remain underexplored. Different agent frameworks encounter similar problems during use, indicating that these recurring issues deserve greater attention and call for further improvements in agent framework design. Meanwhile, as the number of agent frameworks continues to grow and evolve, more than 80% of developers report difficulties in identifying the frameworks that best meet their specific development requirements. In this paper, we conduct the first empirical study of LLM-based agent frameworks, exploring real-world experiences of developers in building AI agents. To compare how well the agent frameworks meet developer needs, we further collect developer discussions for the ten previously identified agent frameworks, resulting in a total of 11,910 discussions. Finally, by analyzing these discussions, we compare the frameworks across five dimensions: development efficiency, functional abstraction, learning cost, performance optimization, and maintainability, which refers to how easily developers can update and extend both the framework itself and the agents built upon it over time. Our comparative analysis reveals significant differences among frameworks in how they meet the needs of agent developers. Overall, we provide a set of findings and implications for the LLM-driven AI agent framework ecosystem and offer insights for the design of future LLM-based agent frameworks and agent developers.
☆ SVRG and Beyond via Posterior Correction
Nico Daheim, Thomas Möllenhoff, Ming Liang Ang, Mohammad Emtiyaz Khan
Stochastic Variance Reduced Gradient (SVRG) and its variants aim to speed-up training by using gradient corrections, but have seen limited success in deep learning. Here, we show surprising new foundational connections of SVRG to a recently proposed Bayesian method called posterior correction. Specifically, we show that SVRG is recovered as a special case of posterior correction over the isotropic-Gaussian family, while novel extensions are automatically obtained by using more flexible exponential families. We derive two new SVRG variants by using Gaussian families: First, a Newton-like variant that employs novel Hessian corrections, and second, an Adam-like extension that improves pretraining and finetuning of Transformer language models. This is the first work to connect SVRG to Bayes and use it to boost variational training for deep networks.
comment : Preprint. Under review
☆ Rectifying LLM Thought from Lens of Optimization
Junnan Liu, Hongwei Liu, Songyang Zhang, Kai Chen
Recent advancements in large language models (LLMs) have been driven by their emergent reasoning capabilities, particularly through long chain-of-thought (CoT) prompting, which enables thorough exploration and deliberation. Despite these advances, long-CoT LLMs often exhibit suboptimal reasoning behaviors, such as overthinking and excessively protracted reasoning chains, which can impair performance. In this paper, we analyze reasoning processes through an optimization lens, framing CoT as a gradient descent procedure where each reasoning step constitutes an update toward problem resolution. Building on this perspective, we introduce RePro (Rectifying Process-level Reward), a novel approach to refine LLM reasoning during post-training. RePro defines a surrogate objective function to assess the optimization process underlying CoT, utilizing a dual scoring mechanism to quantify its intensity and stability. These scores are aggregated into a composite process-level reward, seamlessly integrated into reinforcement learning with verifiable rewards (RLVR) pipelines to optimize LLMs. Extensive experiments across multiple reinforcement learning algorithms and diverse LLMs, evaluated on benchmarks spanning mathematics, science, and coding, demonstrate that RePro consistently enhances reasoning performance and mitigates suboptimal reasoning behaviors.
comment : Work in progress
☆ Real-Wo rl d Robot Control by Deep Active Inference With a Temporally Hierarchical Wo rl d Model
Kentaro Fujii, Shingo Murata
Robots in uncertain real-world environments must perform both goal-directed and exploratory actions. However, most deep learning-based control methods neglect exploration and struggle under uncertainty. To address this, we adopt deep active inference, a framework that accounts for human goal-directed and exploratory actions. Yet, conventional deep active inference approaches face challenges due to limited environmental representation capacity and high computational cost in action selection. We propose a novel deep active inference framework that consists of a world model, an action model, and an abstract world model. The world model encodes environmental dynamics into hidden state representations at slow and fast timescales. The action model compresses action sequences into abstract actions using vector quantization, and the abstract world model predicts future slow states conditioned on the abstract action, enabling low-cost action selection. We evaluate the framework on object-manipulation tasks with a real-world robot. Results show that it achieves high success rates across diverse manipulation tasks and switches between goal-directed and exploratory actions in uncertain settings, while making action selection computationally tractable. These findings highlight the importance of modeling multiple timescale dynamics and abstracting actions and state transitions.
comment : Accepted for publication in IEEE Robotics and Automation Letters (RA-L)
☆ Exploring Human Perceptions of AI Responses: Insights from a Mixed-Methods Study on Risk Mitigation in Generative Models
Heloisa Candello, Muneeza Azmat, Uma Sushmitha Gunturi, Raya Horesh, Rogerio Abreu de Paula, Heloisa Pimentel, Marcelo Carpinette Grave, Aminat Adebiyi, Tiago Machado, Maysa Malfiza Garcia de Macedo
With the rapid uptake of generative AI, investigating human perceptions of generated responses has become crucial. A major challenge is their `aptitude' for hallucinating and generating harmful contents. Despite major efforts for implementing guardrails, human perceptions of these mitigation strategies are largely unknown. We conducted a mixed-method experiment for evaluating the responses of a mitigation strategy across multiple-dimensions: faithfulness, fairness, harm-removal capacity, and relevance. In a within-subject study design, 57 participants assessed the responses under two conditions: harmful response plus its mitigation and solely mitigated response. Results revealed that participants' native language, AI work experience, and annotation familiarity significantly influenced evaluations. Participants showed high sensitivity to linguistic and contextual attributes, penalizing minor grammar errors while rewarding preserved semantic contexts. This contrasts with how language is often treated in the quantitative evaluation of LLMs. We also introduced new metrics for training and evaluating mitigation strategies and insights for human-AI evaluation studies.
comment : 16 pages, 2 figures, 6 tables. Under review for publication
☆ Unifying Sign and Magnitude for Optimizing Deep Vision Networks via ThermoLion
Ahmed Nebli
The training of deep vision models is fundamentally a signal recovery problem amidst high-dimensional stochastic noise. Current optimization paradigms impose a static compromise on information channel capacity. For instance, magnitude-based methods, such as AdamW, operate on the assumption that gradient norms are high-fidelity curvature signals. While this allows for precision in smooth regimes, it leads to catastrophic noise amplification when applied to rugged, non-convex landscapes. Conversely, sign-based methods (e.g., Lion) perform a radical 1-bit quantization of the gradient, which aims to provide robust regularization at the cost of discarding fine-grained descent information. We propose that optimal convergence requires neither static prior, but rather a dynamic modulation of the update bitrate. We introduce \textbf{ThermoLion}, a vision-centric framework that utilizes local Signal-to-Noise Ratio (SNR) gating to autonomously transition parameters between a "low-bit" exploration phase and a "high-precision" exploitation phase. Furthermore, we introduce a Momentum Alignment mechanism that detects constructive interference between historical drift and instantaneous gradients to accelerate convergence during stable trajectories. Empirical benchmarks across 12 diverse vision datasets (including CIFAR, SVHN, and GTSRB) demonstrate that ThermoLion serves as a hyperparameter-free generalist, surpassing both AdamW and Lion in convergence speed and terminal accuracy without architecture-specific tuning.
☆ Predicting Human Chess Moves: An AI Assisted Analysis of Chess Games Using Skill-group Specific n-gram Language Models
Daren Zhong, Dingcheng Huang, Clayton Greenberg
Chess, a deterministic game with perfect information, has long served as a benchmark for studying strategic decision-making and artificial intelligence. Traditional chess engines or tools for analysis primarily focus on calculating optimal moves, often neglecting the variability inherent in human chess playing, particularly across different skill levels. To overcome this limitation, we propose a novel and computationally efficient move prediction framework that approaches chess move prediction as a behavioral analysis task. The framework employs n-gram language models to capture move patterns characteristic of specific player skill levels. By dividing players into seven distinct skill groups, from novice to expert, we trained separate models using data from the open-source chess platform Lichess. The framework dynamically selects the most suitable model for prediction tasks and generates player moves based on preceding sequences. Evaluation on real-world game data demonstrates that the model selector module within the framework can classify skill levels with an accuracy of up to 31.7\% when utilizing early game information (16 half-moves). The move prediction framework also shows substantial accuracy improvements, with our Selector Assisted Accuracy being up to 39.1\% more accurate than our benchmark accuracy. The computational efficiency of the framework further enhances its suitability for real-time chess analysis.
☆ Graph Distance as Surprise: Free Energy Minimization in Knowledge Graph Reasoning NeurIPS 2025
Gaganpreet Jhajj, Fuhua Lin
In this work, we propose that reasoning in knowledge graph (KG) networks can be guided by surprise minimization. Entities that are close in graph distance will have lower surprise than those farther apart. This connects the Free Energy Principle (FEP) from neuroscience to KG systems, where the KG serves as the agent's generative model. We formalize surprise using the shortest-path distance in directed graphs and provide a framework for KG-based agents. Graph distance appears in graph neural networks as message passing depth and in model-based reinforcement learning as world model trajectories. This work-in-progress study explores whether distance-based surprise can extend recent work showing that syntax minimizes surprise and free energy via tree structures.
comment : Accepted to NORA Workshop at NeurIPS 2025
☆ Testing Transformer Learnability on the Arithmetic Sequence of Rooted Trees
Alessandro Breccia, Federica Gerace, Marco Lippi, Gabriele Sicuro, Pierluigi Contucci
We study whether a Large Language Model can learn the deterministic sequence of trees generated by the iterated prime factorization of the natural numbers. Each integer is mapped into a rooted planar tree and the resulting sequence $ \mathbb{N}\mathcal{T}$ defines an arithmetic text with measurable statistical structure. A transformer network (the GPT-2 architecture) is trained from scratch on the first $10^{11}$ elements to subsequently test its predictive ability under next-word and masked-word prediction tasks. Our results show that the model partially learns the internal grammar of $\mathbb{N}\mathcal{T}$, capturing non-trivial regularities and correlations. This suggests that learnability may extend beyond empirical data to the very structure of arithmetic.
comment : 21 pages, 8 figures
☆ Cross-Lingual Inte rl eaving for Speech Language Models
Adel Moumen, Guangzhi Sun, Philip C. Woodland
Spoken Language Models (SLMs) aim to learn linguistic competence directly from speech using discrete units, widening access to Natural Language Processing (NLP) technologies for languages with limited written resources. However, progress has been largely English-centric due to scarce spoken evaluation benchmarks and training data, making cross-lingual learning difficult. We present a cross-lingual interleaving method that mixes speech tokens across languages without textual supervision. We also release an EN-FR training dataset, TinyStories (~42k hours), together with EN-FR spoken StoryCloze and TopicCloze benchmarks for cross-lingual semantic evaluation, both synthetically generated using GPT-4. On 360M and 1B SLMs under matched training-token budgets, interleaving improves monolingual semantic accuracy, enables robust cross-lingual continuation, and strengthens cross-lingual hidden-state alignment. Taken together, these results indicate that cross-lingual interleaving is a simple, scalable route to building multilingual SLMs that understand and converse across languages. All resources will be made open-source to support reproducibility.
☆ Topological Order in Deep State
Ahmed Abouelkomsan, Max Geier, Liang Fu
Topologically ordered states are among the most interesting quantum phases of matter that host emergent quasi-particles having fractional charge and obeying fractional quantum statistics. Theoretical study of such states is however challenging owing to their strong-coupling nature that prevents conventional mean-field treatment. Here, we demonstrate that an attention-based deep neural network provides an expressive variational wavefunction that discovers fractional Chern insulator ground states purely through energy minimization without prior knowledge and achieves remarkable accuracy. We introduce an efficient method to extract ground state topological degeneracy -- a hallmark of topological order -- from a single optimized real-space wavefunction in translation-invariant systems by decomposing it into different many-body momentum sectors. Our results establish neural network variational Monte Carlo as a versatile tool for discovering strongly correlated topological phases.
comment : 5 pages + 6 SM
☆ BHRAM-IL: A Benchmark for Hallucination Recognition and Assessment in Multiple Indian Languages AACL 2025
Hrishikesh Terdalkar, Kirtan Bhojani, Aryan Dongare, Omm Aditya Behera
Large language models (LLMs) are increasingly deployed in multilingual applications but often generate plausible yet incorrect or misleading outputs, known as hallucinations. While hallucination detection has been studied extensively in English, under-resourced Indian languages remain largely unexplored. We present BHRAM-IL, a benchmark for hallucination recognition and assessment in multiple Indian languages, covering Hindi, Gujarati, Marathi, Odia, along with English. The benchmark comprises 36,047 curated questions across nine categories spanning factual, numerical, reasoning, and linguistic tasks. We evaluate 14 state-of-the-art multilingual LLMs on a benchmark subset of 10,265 questions, analyzing cross-lingual and factual hallucinations across languages, models, scales, categories, and domains using category-specific metrics normalized to (0,1) range. Aggregation over all categories and models yields a primary score of 0.23 and a language-corrected fuzzy score of 0.385, demonstrating the usefulness of BHRAM-IL for hallucination-focused evaluation. The dataset, and the code for generation and evaluation are available on GitHub (https://github.com/sambhashana/BHRAM-IL/) and HuggingFace (https://huggingface.co/datasets/sambhashana/BHRAM-IL/) to support future research in multilingual hallucination detection and mitigation.
comment : Accepted at BHASHA Workshop @ IJCNLP/AACL 2025
☆ Mitigating Gender Bias in Depression Detection via Counterfactual Inference
Mingxuan Hu, Hongbo Ma, Xinlan Wu, Ziqi Liu, Jiaqi Liu, Yangbin Chen
Audio-based depression detection models have demonstrated promising performance but often suffer from gender bias due to imbalanced training data. Epidemiological statistics show a higher prevalence of depression in females, leading models to learn spurious correlations between gender and depression. Consequently, models tend to over-diagnose female patients while underperforming on male patients, raising significant fairness concerns. To address this, we propose a novel Counterfactual Debiasing Framework grounded in causal inference. We construct a causal graph to model the decision-making process and identify gender bias as the direct causal effect of gender on the prediction. During inference, we employ counterfactual inference to estimate and subtract this direct effect, ensuring the model relies primarily on authentic acoustic pathological features. Extensive experiments on the DAIC-WOZ dataset using two advanced acoustic backbones demonstrate that our framework not only significantly reduces gender bias but also improves overall detection performance compared to existing debiasing strategies.
☆ Deconstructing Generative Diversity: An Information Bottleneck Analysis of Discrete Latent Generative Models
Yudi Wu, Wenhao Zhao, Dianbo Liu
Generative diversity varies significantly across discrete latent generative models such as AR, MIM, and Diffusion. We propose a diagnostic framework, grounded in Information Bottleneck (IB) theory, to analyze the underlying strategies resolving this behavior. The framework models generation as a conflict between a 'Compression Pressure' - a drive to minimize overall codebook entropy - and a 'Diversity Pressure' - a drive to maximize conditional entropy given an input. We further decompose this diversity into two primary sources: 'Path Diversity', representing the choice of high-level generative strategies, and 'Execution Diversity', the randomness in executing a chosen strategy. To make this decomposition operational, we introduce three zero-shot, inference-time interventions that directly perturb the latent generative process and reveal how models allocate and express diversity. Application of this probe-based framework to representative AR, MIM, and Diffusion systems reveals three distinct strategies: "Diversity-Prioritized" (MIM), "Compression-Prioritized" (AR), and "Decoupled" (Diffusion). Our analysis provides a principled explanation for their behavioral differences and informs a novel inference-time diversity enhancement technique.
☆ InnoGym: Benchmark ing the Innovation Potential of AI Agent s
Jintian Zhang, Kewei Xu, Jingsheng Zheng, Zhuoyun Yu, Yuqi Zhu, Yujie Luo, Lanning Wei, Shuofei Qiao, Lun Du, Da Zheng, Shumin Deng, Huajun Chen, Ningyu Zhang
LLMs and Agents have achieved impressive progress in code generation, mathematical reasoning, and scientific discovery. However, existing benchmarks primarily measure correctness, overlooking the diversity of methods behind solutions. True innovation depends not only on producing correct answers but also on the originality of the approach. We present InnoGym, the first benchmark and framework designed to systematically evaluate the innovation potential of AI agents. InnoGym introduces two complementary metrics: performance gain, which measures improvement over the best-known solutions, and novelty, which captures methodological differences from prior approaches. The benchmark includes 18 carefully curated tasks from real-world engineering and scientific domains, each standardized through resource filtering, evaluator validation, and solution collection. In addition, we provide iGym, a unified execution environment for reproducible and long-horizon evaluations. Extensive experiments show that while some agents produce novel approaches, their lack of robustness limits performance gains. These results highlight a key gap between creativity and effectiveness, underscoring the need for benchmarks that evaluate both.
comment : Work in progress
☆ En vision : Benchmark ing Unified Understanding & Generation for Causal Wo rl d Process Insights
Juanxi Tian, Siyuan Li, Conghui He, Lijun Wu, Cheng Tan
Current multimodal models aim to transcend the limitations of single-modality representations by unifying understanding and generation, often using text-to-image (T2I) tasks to calibrate semantic consistency. However, their reliance on static, single-image generation in training and evaluation leads to overfitting to static pattern matching and semantic fusion, while fundamentally hindering their ability to model dynamic processes that unfold over time. To address these constraints, we propose Envision-a causal event progression benchmark for chained text-to-multi-image generation. Grounded in world knowledge and structured by spatiotemporal causality, it reorganizes existing evaluation dimensions and includes 1,000 four-stage prompts spanning six scientific and humanities domains. To transition evaluation from single images to sequential frames and assess whether models truly internalize world knowledge while adhering to causal-temporal constraints, we introduce Envision-Score, a holistic metric integrating multi-dimensional consistency, physicality, and aesthetics. Comprehensive evaluation of 15 models (10 specialized T2I models, 5 unified models) uncovers: specialized T2I models demonstrate proficiency in aesthetic rendering yet lack intrinsic world knowledge. Unified multimodal models bridge this gap, consistently outperforming specialized counterparts in causal narrative coherence. However, even these unified architectures remain subordinate to closed-source models and struggle to overcome the core challenge of spatiotemporal consistency. This demonstrates that a focus on causally-isolated single images impedes multi-frame reasoning and generation, promoting static pattern matching over dynamic world modeling-ultimately limiting world knowledge internalization, generation.
comment : 35 pages, 12 figures, 10 tables
☆ H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons
Cheng Gao, Huimin Chen, Chaojun Xiao, Zhiyi Chen, Zhiyuan Liu, Maosong Sun
Large language models (LLMs) frequently generate hallucinations -- plausible but factually incorrect outputs -- undermining their reliability. While prior work has examined hallucinations from macroscopic perspectives such as training data and objectives, the underlying neuron-level mechanisms remain largely unexplored. In this paper, we conduct a systematic investigation into hallucination-associated neurons (H-Neurons) in LLMs from three perspectives: identification, behavioral impact, and origins. Regarding their identification, we demonstrate that a remarkably sparse subset of neurons (less than $0.1\%$ of total neurons) can reliably predict hallucination occurrences, with strong generalization across diverse scenarios. In terms of behavioral impact, controlled interventions reveal that these neurons are causally linked to over-compliance behaviors. Concerning their origins, we trace these neurons back to the pre-trained base models and find that these neurons remain predictive for hallucination detection, indicating they emerge during pre-training. Our findings bridge macroscopic behavioral patterns with microscopic neural mechanisms, offering insights for developing more reliable LLMs.
comment : 20 pages, 4 figures
☆ Who Judge s the Judge ? LLM Jury-on-Demand: Building Trustworthy LLM Evaluation Systems
Xiaochuan Li, Ke Wang, Girija Gouda, Shubham Choudhary, Yaqun Wang, Linwei Hu, Joel Vaughan, Freddy Lecue
As Large Language Models (LLMs) become integrated into high-stakes domains, there is a growing need for evaluation methods that are both scalable for real-time deployment and reliable for critical decision-making. While human evaluation is reliable, it is slow and costly. Single LLM judges are biased, and static juries lack adaptability. To overcome these limitations, we propose LLM Jury-on-Demand - a dynamic, learning-based framework for scalable and context-aware evaluation. Our method trains a set of reliability predictors to assess when LLM judges will agree with human experts, leveraging token distributions, embeddings, and structural input features. This enables a fully adaptive evaluation where, for each data point, an optimal jury of the most reliable judges is dynamically selected, and their scores are aggregated using their reliability as weights. Experiments on summarization and RAG benchmarks show that our dynamic jury system achieves significantly higher correlation with human judgment than both single-judge and static-jury baselines. These results highlight the promise of adaptive, learning-based juries for building scalable, more reliable and trustworthy evaluation systems for modern LLMs in high-stakes domains.
comment : 66 pages, 22 figures, 37 tables
☆ Dual Randomized Smoothing: Beyond Global Noise Variance
Chenhao Sun, Yuhao Mao, Martin Vechev
Randomized Smoothing (RS) is a prominent technique for certifying the robustness of neural networks against adversarial perturbations. With RS, achieving high accuracy at small radii requires a small noise variance, while achieving high accuracy at large radii requires a large noise variance. However, the global noise variance used in the standard RS formulation leads to a fundamental limitation: there exists no global noise variance that simultaneously achieves strong performance at both small and large radii. To break through the global variance limitation, we propose a dual RS framework which enables input-dependent noise variances. To achieve that, we first prove that RS remains valid with input-dependent noise variances, provided the variance is locally constant around each input. Building on this result, we introduce two components which form our dual RS framework: (i) a variance estimator first predicts an optimal noise variance for each input, (ii) this estimated variance is then used by a standard RS classifier. The variance estimator is independently smoothed via RS to ensure local constancy, enabling flexible design. We also introduce training strategies to iteratively optimize the two components. Extensive experiments on CIFAR-10 show that our dual RS method provides strong performance for both small and large radii-unattainable with global noise variance-while incurring only a 60% computational overhead at inference. Moreover, it consistently outperforms prior input-dependent noise approaches across most radii, with particularly large gains at radii 0.5, 0.75, and 1.0, achieving relative improvements of 19%, 24%, and 21%, respectively. On ImageNet, dual RS remains effective across all radii. Additionally, the dual RS framework naturally provides a routing perspective for certified robustness, improving the accuracy-robustness trade-off with off-the-shelf expert RS models.
☆ Weight Space Representation Learning with Neural Fields
Zhuoqian Yang, Mathieu Salzmann, Sabine Süsstrunk
In this work, we investigate the potential of weights to serve as effective representations, focusing on neural fields. Our key insight is that constraining the optimization space through a pre-trained base model and low-rank adaptation (LoRA) can induce structure in weight space. Across reconstruction, generation, and analysis tasks on 2D and 3D data, we find that multiplicative LoRA weights achieve high representation quality while exhibiting distinctiveness and semantic structure. When used with latent diffusion models, multiplicative LoRA weights enable higher-quality generation than existing weight-space methods.
comment : 12 pages body, 9 pages appendix
☆ Probabilistic Neuro-Symbolic Reasoning for Sparse Historical Data: A Framework Integrating Bayesian Inference, Causal Models, and Game-Theoretic Allocation
Saba Kublashvili
Modeling historical events poses fundamental challenges for machine learning: extreme data scarcity (N << 100), heterogeneous and noisy measurements, missing counterfactuals, and the requirement for human interpretable explanations. We present HistoricalML, a probabilistic neuro-symbolic framework that addresses these challenges through principled integration of (1) Bayesian uncertainty quantification to separate epistemic from aleatoric uncertainty, (2) structural causal models for counterfactual reasoning under confounding, (3) cooperative game theory (Shapley values) for fair allocation modeling, and (4) attention based neural architectures for context dependent factor weighting. We provide theoretical analysis showing that our approach achieves consistent estimation in the sparse data regime when strong priors from domain knowledge are available, and that Shapley based allocation satisfies axiomatic fairness guarantees that pure regression approaches cannot provide. We instantiate the framework on two historical case studies: the 19th century partition of Africa (N = 7 colonial powers) and the Second Punic War (N = 2 factions). Our model identifies Germany's +107.9 percent discrepancy as a quantifiable structural tension preceding World War I, with tension factor 36.43 and 0.79 naval arms race correlation. For the Punic Wars, Monte Carlo battle simulations achieve a 57.3 percent win probability for Carthage at Cannae and 57.8 percent for Rome at Zama, aligning with historical outcomes. Counterfactual analysis reveals that Carthaginian political support (support score 6.4 vs Napoleon's 7.1), rather than military capability, was the decisive factor.
comment : Preprint. Code and simulation notebooks available at the GitHub repository: https://github.com/Saba-Kublashvili/bayesian-computational-modeling.-
☆ StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos
Daeun Lee, Subhojyoti Mukherjee, Branislav Kveton, Ryan A. Rossi, Viet Dac Lai, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Mohit Bansal
Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intention for realistic applications like AR glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether MLLMs can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs use gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively evaluate streaming video understanding. These tasks assess whether models can use real-time gaze to follow shifting attention and infer user intentions from only past and currently observed frames. To build StreamGaze, we develop a gaze-video QA generation pipeline that aligns egocentric videos with raw gaze trajectories via fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that closely reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, revealing fundamental limitations in gaze-based temporal reasoning, intention modeling, and proactive prediction. We further provide detailed analyses of gaze-prompting strategies, reasoning behaviors, and task-specific failure modes, offering deeper insight into why current MLLMs struggle and what capabilities future models must develop. All data and code will be publicly released to support continued research in gaze-guided streaming video understanding.
comment : Project page: https://streamgaze.github.io/
☆ ICAD-LLM: One-for-All Anomaly Detection via In- Context Learning with Large Language Models
Zhongyuan Wu, Jingyuan Wang, Zexuan Cheng, Yilong Zhou, Weizhi Wang, Juhua Pu, Chao Li, Changqing Ma
Anomaly detection (AD) is a fundamental task of critical importance across numerous domains. Current systems increasingly operate in rapidly evolving environments that generate diverse yet interconnected data modalities -- such as time series, system logs, and tabular records -- as exemplified by modern IT systems. Effective AD methods in such environments must therefore possess two critical capabilities: (1) the ability to handle heterogeneous data formats within a unified framework, allowing the model to process and detect multiple modalities in a consistent manner during anomalous events; (2) a strong generalization ability to quickly adapt to new scenarios without extensive retraining. However, most existing methods fall short of these requirements, as they typically focus on single modalities and lack the flexibility to generalize across domains. To address this gap, we introduce a novel paradigm: In-Context Anomaly Detection (ICAD), where anomalies are defined by their dissimilarity to a relevant reference set of normal samples. Under this paradigm, we propose ICAD-LLM, a unified AD framework leveraging Large Language Models' in-context learning abilities to process heterogeneous data within a single model. Extensive experiments demonstrate that ICAD-LLM achieves competitive performance with task-specific AD methods and exhibits strong generalization to previously unseen tasks, which substantially reduces deployment costs and enables rapid adaptation to new environments. To the best of our knowledge, ICAD-LLM is the first model capable of handling anomaly detection tasks across diverse domains and modalities.
☆ Learning the Boundary of Solvability: Aligning LLMs to Detect Unsolvable Problems
Dengyun Peng, Qiguang Chen, Bofei Liu, Jiannan Guan, Libo Qin, Zheng Yan, Jinhao Liu, Jianshu Zhang, Wanxiang Che
Ensuring LLM reliability requires not only solving complex problems but also recognizing when a problem is unsolvable. Current models often struggle to distinguish objective unsolvability (inherent contradictions in the problem) from subjective capability limitations (problems beyond the model's competence), which leads to hallucinations and overconfidence. To address this, we propose UnsolvableQA and UnsolvableRL to solve feasible problems, detect inherent contradictions, and prudently refuse tasks beyond capability. Specifically, we construct UnsolvableQA, a dataset of paired solvable and unsolvable instances derived via a dual-track methodology: programmatic generation for logic puzzles and a novel "Reverse Construction" method that injects contradictions into valid reasoning chains for mathematics. Building on this dataset, we introduce UnsolvableRL, a reinforcement learning framework with three reward components jointly accounting for accuracy, unsolvability, and difficulty. Empirical results show that our approach achieves near-perfect unsolvability detection while also improving accuracy on solvable tasks. Crucially, we identify Capability Collapse, demonstrating that explicit exposure to unsolvable data is indispensable for preventing models from becoming systematically overconfident. Our code and data are available at https://github.com/sfasfaffa/unsolvableQA.
comment : preprint
☆ HalluGraph: Auditable Hallucination Detection for Legal RAG Systems via Knowledge Graph Alignment
Valentin Noël, Elimane Yassine Seidou, Charly Ken Capo-Chichi, Ghanem Amari
Legal AI systems powered by retrieval-augmented generation (RAG) face a critical accountability challenge: when an AI assistant cites case law, statutes, or contractual clauses, practitioners need verifiable guarantees that generated text faithfully represents source documents. Existing hallucination detectors rely on semantic similarity metrics that tolerate entity substitutions, a dangerous failure mode when confusing parties, dates, or legal provisions can have material consequences. We introduce HalluGraph, a graph-theoretic framework that quantifies hallucinations through structural alignment between knowledge graphs extracted from context, query, and response. Our approach produces bounded, interpretable metrics decomposed into \textit{Entity Grounding} (EG), measuring whether entities in the response appear in source documents, and \textit{Relation Preservation} (RP), verifying that asserted relationships are supported by context. On structured control documents, HalluGraph achieves near-perfect discrimination ($>$400 words, $>$20 entities), HalluGraph achieves $AUC = 0.979$, while maintaining robust performance ($AUC \approx 0.89$) on challenging generative legal task, consistently outperforming semantic similarity baselines. The framework provides the transparency and traceability required for high-stakes legal applications, enabling full audit trails from generated assertions back to source passages.
comment : 8 pages, 4 figures, under review
☆ CLIP- RL : Aligning Language and Policy Representations for Task Transfer in Reinforcement Learning
Chainesh Gautam, Raghuram Bharadwaj Diddigi
Recently, there has been an increasing need to develop agents capable of solving multiple tasks within the same environment, especially when these tasks are naturally associated with language. In this work, we propose a novel approach that leverages combinations of pre-trained (language, policy) pairs to establish an efficient transfer pipeline. Our algorithm is inspired by the principles of Contrastive Language-Image Pretraining (CLIP) in Computer Vision, which aligns representations across different modalities under the philosophy that ''two modalities representing the same concept should have similar representations.'' The central idea here is that the instruction and corresponding policy of a task represent the same concept, the task itself, in two different modalities. Therefore, by extending the idea of CLIP to RL, our method creates a unified representation space for natural language and policy embeddings. Experimental results demonstrate the utility of our algorithm in achieving faster transfer across tasks.
comment : 4 pages, 2 figures, accepted as a extended abstract at RLDM 2025
☆ RoleMotion: A Large-Scale Dataset towards Robust Scene-Specific Role-Playing Motion Synthesis with Fine-grained Descriptions
Junran Peng, Yiheng Huang, Silei Shen, Zeji Wei, Jingwei Yang, Baojie Wang, Yonghao He, Chuanchen Luo, Man Zhang, Xucheng Yin, Wei Sui
In this paper, we introduce RoleMotion, a large-scale human motion dataset that encompasses a wealth of role-playing and functional motion data tailored to fit various specific scenes. Existing text datasets are mainly constructed decentrally as amalgamation of assorted subsets that their data are nonfunctional and isolated to work together to cover social activities in various scenes. Also, the quality of motion data is inconsistent, and textual annotation lacks fine-grained details in these datasets. In contrast, RoleMotion is meticulously designed and collected with a particular focus on scenes and roles. The dataset features 25 classic scenes, 110 functional roles, over 500 behaviors, and 10296 high-quality human motion sequences of body and hands, annotated with 27831 fine-grained text descriptions. We build an evaluator stronger than existing counterparts, prove its reliability, and evaluate various text-to-motion methods on our dataset. Finally, we explore the interplay of motion generation of body and hands. Experimental results demonstrate the high-quality and functionality of our dataset on text-driven whole-body generation.
☆ From Black Hole to Galaxy: Neural Operator: Framework for Accretion and Feedback Dynamics
Nihaal Bhojwani, Chuwei Wang, Hai-Yang Wang, Chang Sun, Elias R. Most, Anima Anandkumar
Modeling how supermassive black holes co-evolve with their host galaxies is notoriously hard because the relevant physics spans nine orders of magnitude in scale-from milliparsecs to megaparsecs--making end-to-end first-principles simulation infeasible. To characterize the feedback from the small scales, existing methods employ a static subgrid scheme or one based on theoretical guesses, which usually struggle to capture the time variability and derive physically faithful results. Neural operators are a class of machine learning models that achieve significant speed-up in simulating complex dynamics. We introduce a neural-operator-based ''subgrid black hole'' that learns the small-scale local dynamics and embeds it within the direct multi-level simulations. Trained on small-domain (general relativistic) magnetohydrodynamic data, the model predicts the unresolved dynamics needed to supply boundary conditions and fluxes at coarser levels across timesteps, enabling stable long-horizon rollouts without hand-crafted closures. Thanks to the great speedup in fine-scale evolution, our approach for the first time captures intrinsic variability in accretion-driven feedback, allowing dynamic coupling between the central black hole and galaxy-scale gas. This work reframes subgrid modeling in computational astrophysics with scale separation and provides a scalable path toward data-driven closures for a broad class of systems with central accretors.
comment : ML4PS Workshop, Neurips 2025 accepted
☆ Reconstructing Multi-Scale Physical Fields from Extremely Sparse Measurements with an Autoencoder- Diffusion Cascade
Letian Yi, Tingpeng Zhang, Mingyuan Zhou, Guannan Wang, Quanke Su, Zhilu Lai
Reconstructing full fields from extremely sparse and random measurements is a longstanding ill-posed inverse problem. A powerful framework for addressing such challenges is hierarchical probabilistic modeling, where uncertainty is represented by intermediate variables and resolved through marginalization during inference. Inspired by this principle, we propose Cascaded Sensing (Cas-Sensing), a hierarchical reconstruction framework that integrates an autoencoder-diffusion cascade. First, a neural operator-based functional autoencoder reconstructs the dominant structures of the original field - including large-scale components and geometric boundaries - from arbitrary sparse inputs, serving as an intermediate variable. Then, a conditional diffusion model, trained with a mask-cascade strategy, generates fine-scale details conditioned on these large-scale structures. To further enhance fidelity, measurement consistency is enforced via the manifold constrained gradient based on Bayesian posterior sampling during the generation process. This cascaded pipeline substantially alleviates ill-posedness, delivering accurate and robust reconstructions. Experiments on both simulation and real-world datasets demonstrate that Cas-Sensing generalizes well across varying sensor configurations and geometric boundaries, making it a promising tool for practical deployment in scientific and engineering applications.
comment : 19 pages,10 figures
☆ Deep FlexQP: Accelerated Nonlinear Programming via Deep Unfolding
Alex Oshin, Rahul Vodeb Ghosh, Augustinos D. Saravanos, Evangelos A. Theodorou
We propose an always-feasible quadratic programming (QP) optimizer, FlexQP, which is based on an exact relaxation of the QP constraints. If the original constraints are feasible, then the optimizer finds the optimal solution to the original QP. On the other hand, if the constraints are infeasible, the optimizer identifies a solution that minimizes the constraint violation in a sparse manner. FlexQP scales favorably with respect to the problem dimension, is robust to both feasible and infeasible QPs with minimal assumptions on the problem data, and can be effectively warm-started. We subsequently apply deep unfolding to improve our optimizer through data-driven techniques, leading to an accelerated Deep FlexQP. By learning dimension-agnostic feedback policies for the parameters from a small number of training examples, Deep FlexQP generalizes to problems with larger dimensions and can optimize for many more iterations than it was initially trained for. Our approach outperforms two recently proposed state-of-the-art accelerated QP approaches on a suite of benchmark systems including portfolio optimization, classification, and regression problems. We provide guarantees on the expected performance of our deep QP optimizer through probably approximately correct (PAC) Bayes generalization bounds. These certificates are used to design an accelerated sequential quadratic programming solver that solves nonlinear optimal control and predictive safety filter problems faster than traditional approaches. Overall, our approach is very robust and greatly outperforms existing non-learning and learning-based optimizers in terms of both runtime and convergence to the optimal solution across multiple classes of NLPs.
☆ MasHeNe: A Benchmark for Head and Neck CT Mass Segmentation using Window-Enhanced Mamba with Frequency-Domain Integration
Thao Thi Phuong Dao, Tan-Cong Nguyen, Nguyen Chi Thanh, Truong Hoang Viet, Trong-Le Do, Mai-Khiem Tran, Minh-Khoi Pham, Trung-Nghia Le, Minh-Triet Tran, Thanh Dinh Le
Head and neck masses are space-occupying lesions that can compress the airway and esophagus and may affect nerves and blood vessels. Available public datasets primarily focus on malignant lesions and often overlook other space-occupying conditions in this region. To address this gap, we introduce MasHeNe, an initial dataset of 3,779 contrast-enhanced CT slices that includes both tumors and cysts with pixel-level annotations. We also establish a benchmark using standard segmentation baselines and report common metrics to enable fair comparison. In addition, we propose the Windowing-Enhanced Mamba with Frequency integration (WEMF) model. WEMF applies tri-window enhancement to enrich the input appearance before feature extraction. It further uses multi-frequency attention to fuse information across skip connections within a U-shaped Mamba backbone. On MasHeNe, WEMF attains the best performance among evaluated methods, with a Dice of 70.45%, IoU of 66.89%, NSD of 72.33%, and HD95 of 5.12 mm. This model indicates stable and strong results on this challenging task. MasHeNe provides a benchmark for head-and-neck mass segmentation beyond malignancy-only datasets. The observed error patterns also suggest that this task remains challenging and requires further research. Our dataset and code are available at https://github.com/drthaodao3101/MasHeNe.git.
comment : The 14th International Symposium on Information and Communication Technology Conference SoICT 2025
☆ LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems
Zhiyuan Wang, Aniri, Tianlong Chen, Yue Zhang, Heng Tao Shen, Xiaoshuang Shi, Kaidi Xu
Large language models (LLMs) often generate unreliable answers, while heuristic uncertainty methods fail to fully distinguish correct from incorrect predictions, causing users to accept erroneous answers without statistical guarantees. We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level. To achieve this in a principled way, we propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint over selection and error indicators. Then, we establish a finite-sample sufficient condition, which relies only on a held-out set of exchangeable calibration samples, to compute an FDR-constrained, coverage-maximizing threshold. Furthermore, we extend LEC to a two-model routing mechanism: given a prompt, if the current model's uncertainty exceeds its calibrated threshold, we delegate it to a stronger model, while maintaining a unified FDR guarantee. Evaluations on closed-ended and open-ended question-answering (QA) datasets show that LEC achieves tighter FDR control and substantially improves sample retention over prior methods. Moreover, the two-model routing mechanism achieves lower risk levels while accepting more correct samples than each individual model.
☆ Delta Sum Learning: an approach for fast and global convergence in Gossip Learning
Tom Goethals, Merlijn Sebrechts, Stijn De Schrijver, Filip De Turck, Bruno Volckaert
Federated Learning is a popular approach for distributed learning due to its security and computational benefits. With the advent of powerful devices in the network edge, Gossip Learning further decentralizes Federated Learning by removing centralized integration and relying fully on peer to peer updates. However, the averaging methods generally used in both Federated and Gossip Learning are not ideal for model accuracy and global convergence. Additionally, there are few options to deploy Learning workloads in the edge as part of a larger application using a declarative approach such as Kubernetes manifests. This paper proposes Delta Sum Learning as a method to improve the basic aggregation operation in Gossip Learning, and implements it in a decentralized orchestration framework based on Open Application Model, which allows for dynamic node discovery and intent-driven deployment of multi-workload applications. Evaluation results show that Delta Sum performance is on par with alternative integration methods for 10 node topologies, but results in a 58% lower global accuracy drop when scaling to 50 nodes. Overall, it shows strong global convergence and a logarithmic loss of accuracy with increasing topology size compared to a linear loss for alternatives under limited connectivity.
☆ LPCD: Unified Framework from Layer-Wise to Submodule Quantization
Yuma Ichikawa, Yudai Fujimoto, Akira Sakai
Post-training quantization (PTQ) aims to preserve model-level behavior; however, most methods focus on individual linear layers. Even recent extensions, such as QEP and LoaQ, which mitigate error propagation or target specific submodules, still rely on layer-wise formulations and fail to capture the behavior of larger submodules. We introduce Layer-Projected Coordinate Descent (LPCD), a unified framework that extends PTQ beyond layers by optimizing relaxed objectives across arbitrary submodules and projecting the solutions with layer-wise quantizers. LPCD generalizes existing methods and provides a principled approach to quantizing complex submodules while maintaining the efficiency and compatibility of layer-wise PTQ pipelines. Across diverse LLM architectures and bit-widths, LPCD-based submodule quantization consistently enhances both layer-wise PTQ methods and existing submodule approaches.
comment : 21 pages, 4 figures
☆ Q2D2: A Geometry-Aware Audio Codec Leveraging Two-Dimensional Quantization
Tal Shuster, Eliya Nachmani
Recent neural audio codecs have achieved impressive reconstruction quality, typically relying on quantization methods such as Residual Vector Quantization (RVQ), Vector Quantization (VQ) and Finite Scalar Quantization (FSQ). However, these quantization techniques limit the geometric structure of the latent space, make it harder to capture correlations between features leading to inefficiency in representation learning, codebook utilization and token rate. In this paper we introduce Two Dimensional Quantization (Q2D2), a quantization scheme in which feature pairs are projected onto structured 2D grids such as hexagonal, rhombic, or rectangular tiling and quantized to the nearest grid values, yielding an implicit codebook defined by the product of grid levels, with codebook sizes comparable to conventional methods. Despite its simple geometric formulation, Q2D2 improves audio compression efficiency, with low token rates and high codebook utilization while maintaining state of the art reconstruction quality. Specifically, Q2D2 achieves competitive to superior performance in various objective and subjective reconstruction metrics, across extensive experiments in speech domain compared to state of the art models. Comprehensive ablation studies further confirm the effectiveness of our design choices.
☆ Deep Unsupervised Anomaly Detection in Brain Imaging: Large-Scale Benchmark ing and Bias Analysis
Alexander Frotscher, Christian F. Baumgartner, Thomas Wolfers
Deep unsupervised anomaly detection in brain magnetic resonance imaging offers a promising route to identify pathological deviations without requiring lesion-specific annotations. Yet, fragmented evaluations, heterogeneous datasets, and inconsistent metrics have hindered progress toward clinical translation. Here, we present a large-scale, multi-center benchmark of deep unsupervised anomaly detection for brain imaging. The training cohort comprised 2,976 T1 and 2,972 T2-weighted scans from healthy individuals across six scanners, with ages ranging from 6 to 89 years. Validation used 92 scans to tune hyperparameters and estimate unbiased thresholds. Testing encompassed 2,221 T1w and 1,262 T2w scans spanning healthy datasets and diverse clinical cohorts. Across all algorithms, the Dice-based segmentation performance varied between 0.03 and 0.65, indicating substantial variability. To assess robustness, we systematically evaluated the impact of different scanners, lesion types and sizes, as well as demographics (age, sex). Reconstruction-based methods, particularly diffusion-inspired approaches, achieved the strongest lesion segmentation performance, while feature-based methods showed greater robustness under distributional shifts. However, systematic biases, such as scanner-related effects, were observed for the majority of algorithms, including that small and low-contrast lesions were missed more often, and that false positives varied with age and sex. Increasing healthy training data yields only modest gains, underscoring that current unsupervised anomaly detection frameworks are limited algorithmically rather than by data availability. Our benchmark establishes a transparent foundation for future research and highlights priorities for clinical translation, including image native pretraining, principled deviation measures, fairness-aware modeling, and robust domain adaptation.
☆ Diffusion Fuzzy System: Fuzzy Rule Guided Latent Multi-Path Diffusion Modeling
Hailong Yang, Te Zhang, Kup-sze Choi, Zhaohong Deng
Diffusion models have emerged as a leading technique for generating images due to their ability to create high-resolution and realistic images. Despite their strong performance, diffusion models still struggle in managing image collections with significant feature differences. They often fail to capture complex features and produce conflicting results. Research has attempted to address this issue by learning different regions of an image through multiple diffusion paths and then combining them. However, this approach leads to inefficient coordination among multiple paths and high computational costs. To tackle these issues, this paper presents a Diffusion Fuzzy System (DFS), a latent-space multi-path diffusion model guided by fuzzy rules. DFS offers several advantages. First, unlike traditional multi-path diffusion methods, DFS uses multiple diffusion paths, each dedicated to learning a specific class of image features. By assigning each path to a different feature type, DFS overcomes the limitations of multi-path models in capturing heterogeneous image features. Second, DFS employs rule-chain-based reasoning to dynamically steer the diffusion process and enable efficient coordination among multiple paths. Finally, DFS introduces a fuzzy membership-based latent-space compression mechanism to reduce the computational costs of multi-path diffusion effectively. We tested our method on three public datasets: LSUN Bedroom, LSUN Church, and MS COCO. The results show that DFS achieves more stable training and faster convergence than existing single-path and multi-path diffusion models. Additionally, DFS surpasses baseline models in both image quality and alignment between text and images, and also shows improved accuracy when comparing generated images to target references.
☆ Teaching an Online Multi-Institutional Research Level Software Engineering Course with Industry - an Experience Report
Pankaj Jalote, Y. Raghu Reddy, Vasudeva Varma
Covid has made online teaching and learning acceptable and students, faculty, and industry professionals are all comfortable with this mode. This comfort can be leveraged to offer an online multi-institutional research-level course in an area where individual institutions may not have the requisite faculty to teach and/or research students to enroll. If the subject is of interest to industry, online offering also allows industry experts to contribute and participate with ease. Advanced topics in Software Engineering are ideally suited for experimenting with this approach as industry, which is often looking to incorporate advances in software engineering in their practices, is likely to agree to contribute and participate. In this paper we describe an experiment in teaching a course titled "AI in Software Engineering" jointly between two institutions with active industry participation, and share our and student's experience. We believe this collaborative teaching approach can be used for offering research level courses in any applied area of computer science by institutions who are small and find it difficult to offer research level courses on their own.
comment : 7 pages
☆ SynthStrategy: Extracting and Formalizing Latent Strategic Insights from LLMs in Organic Chemistry
Daniel Armstrong, Zlatko Jončev, Andres M Bran, Philippe Schwaller
Modern computer-assisted synthesis planning (CASP) systems show promises at generating chemically valid reaction steps but struggle to incorporate strategic considerations such as convergent assembly, protecting group minimization, and optimal ring-forming sequences. We introduce a methodology that leverages Large Language Models to distill synthetic knowledge into code. Our system analyzes synthesis routes and translates strategic principles into Python functions representing diverse strategic and tactical rules, such as strategic functional group interconversions and ring construction strategies. By formalizing this knowledge as verifiable code rather than simple heuristics, we create testable, interpretable representations of synthetic strategy. We release the complete codebase and the USPTO-ST dataset -- synthesis routes annotated with strategic tags. This framework unlocks a novel capability for CASP: natural language-based route retrieval, achieving 75\% Top-3 accuracy on our benchmark. We further validate our library through temporal analysis of historical trends and chemically intuitive route clustering that offers more granular partitioning than common previous methods. This work bridges the tactical-strategic divide in CASP, enabling specification, search, and evaluation of routes by strategic criteria rather than structure alone.
☆ Formal Verification of Noisy Quantum Reinforcement Learning Policies
Dennis Gross
Quantum reinforcement learning (QRL) aims to use quantum effects to create sequential decision-making policies that achieve tasks more effectively than their classical counterparts. However, QRL policies face uncertainty from quantum measurements and hardware noise, such as bit-flip, phase-flip, and depolarizing errors, which can lead to unsafe behavior. Existing work offers no systematic way to verify whether trained QRL policies meet safety requirements under specific noise conditions. We introduce QVerifier, a formal verification method that applies probabilistic model checking to analyze trained QRL policies with and without modeled quantum noise. QVerifier builds a complete model of the policy-environment interaction, incorporates quantum uncertainty directly into the transition probabilities, and then checks safety properties using the Storm model checker. Experiments across multiple QRL environments show that QVerifier precisely measures how different noise models influence safety, revealing both performance degradation and cases where noise can help. By enabling rigorous safety verification before deployment, QVerifier addresses a critical need: because access to quantum hardware is expensive, pre-deployment verification is essential for any safety-critical use of QRL. QVerifier targets a potential classical-quantum sweet spot: trained QRL policies that execute efficiently on quantum hardware, yet remain tractable for classical probabilistic model checking despite being too slow for real-time classical deployment.
☆ Multi-Path Collaborative Reasoning via Reinforcement Learning
Jindi Lv, Yuhao Zhou, Zheng Zhu, Xiaofeng Wang, Guan Huang, Jiancheng Lv
Chain-of-Thought (CoT) reasoning has significantly advanced the problem-solving capabilities of Large Language Models (LLMs), yet conventional CoT often exhibits internal determinism during decoding, limiting exploration of plausible alternatives. Recent methods attempt to address this by generating soft abstract tokens to enable reasoning in a continuous semantic space. However, we find that such approaches remain constrained by the greedy nature of autoregressive decoding, which fundamentally isolates the model from alternative reasoning possibilities. In this work, we propose Multi-Path Perception Policy Optimization (M3PO), a novel reinforcement learning framework that explicitly injects collective insights into the reasoning process. M3PO leverages parallel policy rollouts as naturally diverse reasoning sources and integrates cross-path interactions into policy updates through a lightweight collaborative mechanism. This design allows each trajectory to refine its reasoning with peer feedback, thereby cultivating more reliable multi-step reasoning patterns. Empirical results show that M3PO achieves state-of-the-art performance on both knowledge- and reasoning-intensive benchmarks. Models trained with M3PO maintain interpretability and inference efficiency, underscoring the promise of multi-path collaborative learning for robust reasoning.
☆ Multi-view diffusion geometry using intertwined diffusion trajectories
Gwendal Debaussart-Joniec, Argyris Kalogeratos
This paper introduces a comprehensive unified framework for constructing multi-view diffusion geometries through intertwined multi-view diffusion trajectories (MDTs), a class of inhomogeneous diffusion processes that iteratively combine the random walk operators of multiple data views. Each MDT defines a trajectory-dependent diffusion operator with a clear probabilistic and geometric interpretation, capturing over time the interplay between data views. Our formulation encompasses existing multi-view diffusion models, while providing new degrees of freedom for view interaction and fusion. We establish theoretical properties under mild assumptions, including ergodicity of both the point-wise operator and the process in itself. We also derive MDT-based diffusion distances, and associated embeddings via singular value decompositions. Finally, we propose various strategies for learning MDT operators within the defined operator space, guided by internal quality measures. Beyond enabling flexible model design, MDTs also offer a neutral baseline for evaluating diffusion-based approaches through comparison with randomly selected MDTs. Experiments show the practical impact of the MDT operators in a manifold learning and data clustering context.
☆ Does Flatness imply Generalization for Logistic Loss in Univariate Two-Layer ReLU Network?
Dan Qiao, Yu-Xiang Wang
We consider the problem of generalization of arbitrarily overparameterized two-layer ReLU Neural Networks with univariate input. Recent work showed that under square loss, flat solutions (motivated by flat / stable minima and Edge of Stability phenomenon) provably cannot overfit, but it remains unclear whether the same phenomenon holds for logistic loss. This is a puzzling open problem because existing work on logistic loss shows that gradient descent with increasing step size converges to interpolating solutions (at infinity, for the margin-separable cases). In this paper, we prove that the \emph{flatness implied generalization} is more delicate under logistic loss. On the positive side, we show that flat solutions enjoy near-optimal generalization bounds within a region between the left-most and right-most \emph{uncertain} sets determined by each candidate solution. On the negative side, we show that there exist arbitrarily flat yet overfitting solutions at infinity that are (falsely) certain everywhere, thus certifying that flatness alone is insufficient for generalization in general. We demonstrate the effects predicted by our theory in a well-controlled simulation study.
comment : 59 pages
☆ ZIP-RC: Zero-overhead Inference-time Prediction of Reward and Cost for Adaptive and Interpretable Generation
Rohin Manvi, Joey Hong, Tim Seyde, Maxime Labonne, Mathias Lechner, Sergey Levine
Large language models excel at reasoning but lack key aspects of introspection, including anticipating their own success and the computation required to achieve it. Humans use real-time introspection to decide how much effort to invest, when to make multiple attempts, when to stop, and when to signal success or failure. Without this, LLMs struggle to make intelligent meta-cognition decisions. Test-time scaling methods like Best-of-N drive up cost and latency by using a fixed budget of samples regardless of the marginal benefit of each one at any point in generation, and the absence of confidence signals can mislead people, prevent appropriate escalation to better tools, and undermine trustworthiness. Learned verifiers or reward models can provide confidence estimates, but do not enable adaptive inference and add substantial cost by requiring extra models or forward passes. We present ZIP-RC, an adaptive inference method that equips models with zero-overhead inference-time predictions of reward and cost. At every token, ZIP-RC reuses reserved or unused logits in the same forward pass as next-token prediction to output a joint distribution over final reward and remaining length -- no extra models, architecture change, or inference overhead. This full joint distribution is used to compute a sampling utility which is the linear combination of the expected maximum reward, total compute, and latency of set of samples if generated to completion. During inference, we maximize this utility with meta-actions that determine which prefix of tokens to continue or initiate sampling from. On mixed-difficulty mathematical benchmarks, ZIP-RC improves accuracy by up to 12% over majority voting at equal or lower average cost, and traces smooth Pareto frontiers between quality, compute, and latency. By providing real-time reward-cost introspection, ZIP-RC enables adaptive, efficient reasoning.
comment : Code coming soon
☆ Automated Risk-of-Bias Assessment of Randomized Controlled Trials: A First Look at a GEPA-trained Programmatic Prompt ing Framework
Lingbo Li, Anuradha Mathrani, Teo Susnjak
Assessing risk of bias (RoB) in randomized controlled trials is essential for trustworthy evidence synthesis, but the process is resource-intensive and prone to variability across reviewers. Large language models (LLMs) offer a route to automation, but existing methods rely on manually engineered prompts that are difficult to reproduce, generalize, or evaluate. This study introduces a programmable RoB assessment pipeline that replaces ad-hoc prompt design with structured, code-based optimization using DSPy and its GEPA module. GEPA refines LLM reasoning through Pareto-guided search and produces inspectable execution traces, enabling transparent replication of every step in the optimization process. We evaluated the method on 100 RCTs from published meta-analyses across seven RoB domains. GEPA-generated prompts were applied to both open-weight models (Mistral Small 3.1 with GPT-oss-20b) and commercial models (GPT-5 Nano and GPT-5 Mini). In domains with clearer methodological reporting, such as Random Sequence Generation, GEPA-generated prompts performed best, with similar results for Allocation Concealment and Blinding of Participants, while the commercial model performed slightly better overall. We also compared GEPA with three manually designed prompts using Claude 3.5 Sonnet. GEPA achieved the highest overall accuracy and improved performance by 30%-40% in Random Sequence Generation and Selective Reporting, and showed generally comparable, competitively aligned performance in the other domains relative to manual prompts. These findings suggest that GEPA can produce consistent and reproducible prompts for RoB assessment, supporting the structured and principled use of LLMs in evidence synthesis.
☆ PSA-MF: Personality-Sentiment Aligned Multi-Level Fusion for Multimodal Sentiment Analysis AAAI 2026
Heng Xie, Kang Zhu, Zhengqi Wen, Jianhua Tao, Xuefei Liu, Ruibo Fu, Changsheng Li
Multimodal sentiment analysis (MSA) is a research field that recognizes human sentiments by combining textual, visual, and audio modalities. The main challenge lies in integrating sentiment-related information from different modalities, which typically arises during the unimodal feature extraction phase and the multimodal feature fusion phase. Existing methods extract only shallow information from unimodal features during the extraction phase, neglecting sentimental differences across different personalities. During the fusion phase, they directly merge the feature information from each modality without considering differences at the feature level. This ultimately affects the model's recognition performance. To address this problem, we propose a personality-sentiment aligned multi-level fusion framework. We introduce personality traits during the feature extraction phase and propose a novel personality-sentiment alignment method to obtain personalized sentiment embeddings from the textual modality for the first time. In the fusion phase, we introduce a novel multi-level fusion method. This method gradually integrates sentimental information from textual, visual, and audio modalities through multimodal pre-fusion and a multi-level enhanced fusion strategy. Our method has been evaluated through multiple experiments on two commonly used datasets, achieving state-of-the-art results.
comment : AAAI 2026 accepted
☆ A Selective Temporal Hamming distance to find patterns in state transition event timeseries, at scale
Sylvain Marié, Pablo Knecht
Discrete event systems are present both in observations of nature, socio economical sciences, and industrial systems. Standard analysis approaches do not usually exploit their dual event / state nature: signals are either modeled as transition event sequences, emphasizing event order alignment, or as categorical or ordinal state timeseries, usually resampled a distorting and costly operation as the observation period and number of events grow. In this work we define state transition event timeseries (STE-ts) and propose a new Selective Temporal Hamming distance (STH) leveraging both transition time and duration-in-state, avoiding costly and distorting resampling on large databases. STH generalizes both resampled Hamming and Jaccard metrics with better precision and computation time, and an ability to focus on multiple states of interest. We validate these benefits on simulated and real-world datasets.
☆ A Flexible Multi- Agent LLM-Human Framework for Fast Human Validated Tool Building
Daull Xavier, Patrice Bellot, Emmanuel Bruno, Vincent Martin, Elisabeth Murisasco
We introduce CollabToolBuilder, a flexible multiagent LLM framework with expert-in-the-loop (HITL) guidance that iteratively learns to create tools for a target goal, aligning with human intent and process, while minimizing time for task/domain adaptation effort and human feedback capture. The architecture generates and validates tools via four specialized agents (Coach, Coder, Critic, Capitalizer) using a reinforced dynamic prompt and systematic human feedback integration to reinforce each agent's role toward goals and constraints. This work is best viewed as a system-level integration and methodology combining multi-agent in-context learning, HITL controls, and reusable tool capitalization for complex iterative problems such as scientific document generation. We illustrate it with preliminary experiments (e.g., generating state-of-the-art research papers or patents given an abstract) and discuss its applicability to other iterative problem-solving.
☆ Prompt Bridge: Cross-Model Prompt Transfer for Large Language Models
Yaxuan Wang, Quan Liu, Zhenting Wang, Zichao Li, Wei Wei, Yang Liu, Yujia Bao
Large language models (LLMs) underpin applications in code generation, mathematical reasoning, and agent-based workflows. In practice, systems access LLMs via commercial APIs or open-source deployments, and the model landscape (e.g., GPT, Claude, Llama) evolves rapidly. This rapid evolution forces frequent model switches driven by capability, cost, deployment constraints, and privacy. Yet prompts are highly model-sensitive: reusing a prompt engineered for one model on another often yields substantially worse performance than a prompt optimized for the target model. We term this phenomenon Model Drifting. Through extensive empirical analysis across diverse LLM configurations, we show that model drifting is both common and severe. To address this challenge, we introduce PromptBridge, a training-free framework that preserves prompt effectiveness under model switches, enabling cross-model prompt transfer without costly per-task or per-model re-optimization. PromptBridge requires only a small set of alignment tasks for calibration. It first applies Model-Adaptive Reflective Prompt Evolution (MAP-RPE) to obtain task- and model-specific optimal prompts via iterative reflective refinement and quantitative evaluation. Using the resulting calibrated prompt pairs for the source and target models, PromptBridge learns a cross-model prompt mapping. At test time, i.e., for an unseen task, given a source-model prompt, this mapping directly produces an optimized prompt for the target model. Experiments in single-agent and multi-agent settings show that PromptBridge consistently improves downstream accuracy while reducing migration effort. The code will be available soon.
☆ Rice-VL: Evaluating Vision -Language Models for Cultural Understanding Across ASEAN Countries
Tushar Pranav, Eshan Pandey, Austria Lyka Diane Bala, Aman Chadha, Indriyati Atmosukarto, Donny Soh Cheng Lock
Vision-Language Models (VLMs) excel in multimodal tasks but often exhibit Western-centric biases, limiting their effectiveness in culturally diverse regions like Southeast Asia (SEA). To address this, we introduce RICE-VL, a novel benchmark evaluating VLM cultural understanding across 11 ASEAN countries. RICE-VL includes over 28,000 human-curated Visual Question Answering (VQA) samples -- covering True or False, Fill-in-the-Blank, and open-ended formats -- and 1,000 image-bounding box pairs for Visual Grounding, annotated by culturally informed experts across 14 sub-ground categories. We propose SEA-LAVE, an extension of the LAVE metric, assessing textual accuracy, cultural alignment, and country identification. Evaluations of six open- and closed-source VLMs reveal significant performance gaps in low-resource countries and abstract cultural domains. The Visual Grounding task tests models' ability to localize culturally significant elements in complex scenes, probing spatial and contextual accuracy. RICE-VL exposes limitations in VLMs' cultural comprehension and highlights the need for inclusive model development to better serve diverse global populations.
comment : 14 pages
☆ A Self-explainable Model of Long Time Series by Extracting Informative Structured Causal Patterns
Ziqian Wang, Yuxiao Cheng, Jinli Suo
Explainability is essential for neural networks that model long time series, yet most existing explainable AI methods only produce point-wise importance scores and fail to capture temporal structures such as trends, cycles, and regime changes. This limitation weakens human interpretability and trust in long-horizon models. To address these issues, we identify four key requirements for interpretable time-series modeling: temporal continuity, pattern-centric explanation, causal disentanglement, and faithfulness to the model's inference process. We propose EXCAP, a unified framework that satisfies all four requirements. EXCAP combines an attention-based segmenter that extracts coherent temporal patterns, a causally structured decoder guided by a pre-trained causal graph, and a latent aggregation mechanism that enforces representation stability. Our theoretical analysis shows that EXCAP provides smooth and stable explanations over time and is robust to perturbations in causal masks. Extensive experiments on classification and forecasting benchmarks demonstrate that EXCAP achieves strong predictive accuracy while generating coherent and causally grounded explanations. These results show that EXCAP offers a principled and scalable approach to interpretable modeling of long time series with relevance to high-stakes domains such as healthcare and finance.
comment : Approximately 30 pages, 8 figures, and 5 tables. Preprint version. Includes theoretical analysis, model architecture, interpretability evaluation, and extensive benchmark experiments
☆ Consistency Flow Model Achieves One-step Denoising Error Correction Codes
Haoyu Lei, Chin Wa Lau, Kaiwen Zhou, Nian Guo, Farzan Farnia
Error Correction Codes (ECC) are fundamental to reliable digital communication, yet designing neural decoders that are both accurate and computationally efficient remains challenging. Recent denoising diffusion decoders with transformer backbones achieve state-of-the-art performance, but their iterative sampling limits practicality in low-latency settings. We introduce the Error Correction Consistency Flow Model (ECCFM), an architecture-agnostic training framework for high-fidelity one-step decoding. By casting the reverse denoising process as a Probability Flow Ordinary Differential Equation (PF-ODE) and enforcing smoothness through a differential time regularization, ECCFM learns to map noisy signals along the decoding trajectory directly to the original codeword in a single inference step. Across multiple decoding benchmarks, ECCFM attains lower bit-error rates (BER) than autoregressive and diffusion-based baselines, with notable improvements on longer codes, while delivering inference speeds up from 30x to 100x faster than denoising diffusion decoders.
☆ Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, An Yang, Jingren Zhou, Junyang Lin
This paper proposes a novel formulation for reinforcement learning (RL) with large language models, explaining why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE. Specifically, through a first-order approximation, we show that this surrogate becomes increasingly valid only when both the training-inference discrepancy and policy staleness are minimized. This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training, including importance sampling correction, clipping, and particularly Routing Replay for Mixture-of-Experts (MoE) models. Through extensive experiments with a 30B MoE model totaling hundreds of thousands of GPU hours, we show that for on-policy training, the basic policy gradient algorithm with importance sampling correction achieves the highest training stability. When off-policy updates are introduced to accelerate convergence, combining clipping and Routing Replay becomes essential to mitigate the instability caused by policy staleness. Notably, once training is stabilized, prolonged optimization consistently yields comparable final performance regardless of cold-start initialization. We hope that the shared insights and the developed recipes for stable RL training will facilitate future research.
☆ Structured Spectral Reasoning for Frequency-Adaptive Multimodal Recommendation
Wei Yang, Rui Zhong, Yiqun Chen, Chi Lu, Peng Jiang
Multimodal recommendation aims to integrate collaborative signals with heterogeneous content such as visual and textual information, but remains challenged by modality-specific noise, semantic inconsistency, and unstable propagation over user-item graphs. These issues are often exacerbated by naive fusion or shallow modeling strategies, leading to degraded generalization and poor robustness. While recent work has explored the frequency domain as a lens to separate stable from noisy signals, most methods rely on static filtering or reweighting, lacking the ability to reason over spectral structure or adapt to modality-specific reliability. To address these challenges, we propose a Structured Spectral Reasoning (SSR) framework for frequency-aware multimodal recommendation. Our method follows a four-stage pipeline: (i) Decompose graph-based multimodal signals into spectral bands via graph-guided transformations to isolate semantic granularity; (ii) Modulate band-level reliability with spectral band masking, a training-time masking with a prediction-consistency objective that suppresses brittle frequency components; (iii) Fuse complementary frequency cues using hyperspectral reasoning with low-rank cross-band interaction; and (iv) Align modality-specific spectral features via contrastive regularization to promote semantic and structural consistency. Experiments on three real-world benchmarks show consistent gains over strong baselines, particularly under sparse and cold-start settings. Additional analyses indicate that structured spectral modeling improves robustness and provides clearer diagnostics of how different bands contribute to performance.
☆ Beyond Loss Guidance: Using PDE Residuals as Spectral Attention in Diffusion Neural Operators
Medha Sawhney, Abhilash Neog, Mridul Khurana, Anuj Karpatne
Diffusion-based solvers for partial differential equations (PDEs) are often bottle-necked by slow gradient-based test-time optimization routines that use PDE residuals for loss guidance. They additionally suffer from optimization instabilities and are unable to dynamically adapt their inference scheme in the presence of noisy PDE residuals. To address these limitations, we introduce PRISMA (PDE Residual Informed Spectral Modulation with Attention), a conditional diffusion neural operator that embeds PDE residuals directly into the model's architecture via attention mechanisms in the spectral domain, enabling gradient-descent free inference. In contrast to previous methods that use PDE loss solely as external optimization targets, PRISMA integrates PDE residuals as integral architectural features, making it inherently fast, robust, accurate, and free from sensitive hyperparameter tuning. We show that PRISMA has competitive accuracy, at substantially lower inference costs, compared to previous methods across five benchmark PDEs, especially with noisy observations, while using 10x to 100x fewer denoising steps, leading to 15x to 250x faster inference.
☆ Tangram: Accelerating Serve rl ess LLM Loading through GPU Memory Reuse and Affinity
Wenbin Zhu, Zhaoyan Shen, Zili Shao, Hongjun Dai, Feng Chen
Serverless Large Language Models (LLMs) have emerged as a cost-effective solution for deploying AI services by enabling a 'pay-as-you-go' pricing model through GPU resource sharing. However, cold-start latency, especially the model loading phase, has become a critical performance bottleneck, as it scales linearly with model size and severely limits the practical deployment of large-scale LLM services. This paper presents Tangram, a novel system that accelerates Serverless LLM loading through efficient GPU memory reuse. By leveraging the unused GPU memory to retain model parameters, Tangram significantly reduces model transfer time and cold-start latency. Its design includes three key components: unified GPU memory pool for tensor-level parameter sharing across models, on-demand KV cache allocation for dynamic memory management, and GPU-affinity-aware scheduling for maximizing resource utilization. These techniques collectively address the critical challenges of inefficient memory usage and the cold-start problem in Serverless LLM platforms. We have implemented a fully functional prototype, and experiments show that Tangram achieves up to 6.2 times faster loading and reduces Time-To-First-Token (TTFT) during cold-start by 23--55% over state-of-the-art methods.
☆ The Necessity of Imperfection:Reversing Model Collapse via Simulating Cognitive Boundedness
Zhongjie Jiang
Although synthetic data is widely promoted as a remedy, its prevailing production paradigm -- one optimizing for statistical smoothness -- systematically removes the long-tail, cognitively grounded irregularities that characterize human text. Prolonged training on such statistically optimal but cognitively impoverished data accelerates model collapse. This paper proposes a paradigm shift: instead of imitating the surface properties of data, we simulate the cognitive processes that generate human text. We introduce the Prompt-driven Cognitive Computing Framework (PMCSF), whose core consists of a Cognitive State Decoder (CSD) that reverse-engineers unstructured text into structured cognitive vectors, and a Cognitive Text Encoder (CTE) that re-materializes these states into text enriched with human-typical imperfections via mathematically defined Cognitive Perturbation Operators. The framework is validated through a two-stage objective evaluation pipeline. First, in cognitive codec verification, CTE text yields a Jensen-Shannon divergence of 0.0614 from human text (vs. 0.4431 for standard LLM output), passes double-blind professional media review, and achieves an intraclass correlation coefficient ICC > 0.9 for cognitive profile alignment across heterogeneous models. Second, in functional gain evaluation, isomorphic stress tests in the A-share market show that strategies incorporating CTE-generated data reduce maximum drawdown by 47.4% during the 2015 crash and deliver 8.6% Defensive Alpha, exceeding transaction costs by a factor of 33. Our findings demonstrate that modelling human cognitive limitations -- not copying surface data -- enables synthetic data with genuine functional gain, offering a viable technical pathway toward resolving the AI data-collapse crisis.
comment : 38 pages,5 figures,30 tables. This paper proposes the Prompt-driven Cognitive Computing Framework (PMCSF) and validates it with A-share market stress tests (N=23 for 2015 crash, N=13 for 2024 bull market). Includes detailed appendices on cognitive vector definitions, perturbation operators, and financial backtest data
☆ Benchmark ing Overton Pluralism in LLMs
Elinor Poole-Dayan, Jiayi Wu, Taylor Sorensen, Jiaxin Pei, Michiel A. Bakker
We introduce a novel framework for measuring Overton pluralism in LLMs--the extent to which diverse viewpoints are represented in model outputs. We (i) formalize Overton pluralism as a set coverage metric (OvertonScore), (ii) conduct a large-scale U.S.-representative human study (N = 1209; 60 questions; 8 LLMs), and (iii) develop an automated benchmark that closely reproduces human judgments. On average, models achieve OvertonScores of 0.35--0.41, with DeepSeek V3 performing best; yet all models remain far below the theoretical maximum of 1.0, revealing substantial headroom for improvement. Because repeated large-scale human studies are costly and slow, scalable evaluation tools are essential for model development. Hence, we propose an automated benchmark that achieves high rank correlation with human judgments ($ρ=0.88$), providing a practical proxy without replacing human assessment. By turning pluralistic alignment from a normative aim into a measurable benchmark, our work establishes a foundation for systematic progress toward more pluralistic LLMs.
☆ Intrinsic Structure as a Proxy for Saliency: SVD-Based Weight Preservation for Mixed-Precision Quantization in Large Language Models
Shashank Landge, Abhishek Patil, Tejas kamble, Bhushan Buddhivant, Priyanka Joshi
As Large Language Models (LLMs) continue to scale in parameter count, deploying them on commodity hardware has become increasingly challenging. Post-Training Quantization (PTQ) addresses this by reducing the precision of model weights, typically to 4-bit or lower. However, uniform quantization often leads to significant performance degradation due to the presence of ``outlier features'' -- weights that, while few in number, are critical for maintaining model accuracy. Current state-of-the-art methods such as AWQ (Activation-aware Weight Quantization) and SpQR (Sparse Quantization Representations) rely on calibration data to identify these salient weights via activation magnitudes or Hessian sensitivity. In scenarios where data privacy is paramount or calibration data is unavailable, these methods are inapplicable. In this work, we propose a data-free, structure-aware hypothesis: that the weights identified as Principal Components via Singular Value Decomposition (SVD) are intrinsically important to the model's downstream performance. We introduce a novel selection heuristic that preserves the top-$k$ weights aligned with the principal components in FP32, while aggressively quantizing the residual weights. We compare our method against activation-aware (AWQ) and second-order (SpQR) methods across GLUE benchmarks (MRPC, RTE, QNLI) using a DistilBERT backbone. Our experiments reveal that structural importance is highly correlated with functional importance. On the challenging RTE task, our SVD-based method achieves an accuracy of 66.06\%, outperforming both AWQ (65.34\%) and SpQR (65.34\%) at high protection budgets, validating that intrinsic matrix structure can serve as a robust proxy for weight saliency without the need for forward passes or calibration data.
☆ EmoRAG: Evaluating RAG Robustness to Symbolic Perturbations KDD
Xinyun Zhou, Xinfeng Li, Yinan Peng, Ming Xu, Xuanwang Zhang, Miao Yu, Yidong Wang, Xiaojun Jia, Kun Wang, Qingsong Wen, XiaoFeng Wang, Wei Dong
Retrieval-Augmented Generation (RAG) systems are increasingly central to robust AI, enhancing large language model (LLM) faithfulness by incorporating external knowledge. However, our study unveils a critical, overlooked vulnerability: their profound susceptibility to subtle symbolic perturbations, particularly through near-imperceptible emoticon tokens such as "(@_@)" that can catastrophically mislead retrieval, termed EmoRAG. We demonstrate that injecting a single emoticon into a query makes it nearly 100% likely to retrieve semantically unrelated texts that contain a matching emoticon. Our extensive experiment across general question-answering and code domains, using a range of state-of-the-art retrievers and generators, reveals three key findings: (I) Single-Emoticon Disaster: Minimal emoticon injections cause maximal disruptions, with a single emoticon almost 100% dominating RAG output. (II) Positional Sensitivity: Placing an emoticon at the beginning of a query can cause severe perturbation, with F1-Scores exceeding 0.92 across all datasets. (III) Parameter-Scale Vulnerability: Counterintuitively, models with larger parameters exhibit greater vulnerability to the interference. We provide an in-depth analysis to uncover the underlying mechanisms of these phenomena. Furthermore, we raise a critical concern regarding the robustness assumption of current RAG systems, envisioning a threat scenario where an adversary exploits this vulnerability to manipulate the RAG system. We evaluate standard defenses and find them insufficient against EmoRAG. To address this, we propose targeted defenses, analyzing their strengths and limitations in mitigating emoticon-based perturbations. Finally, we outline future directions for building robust RAG systems.
comment : Accepted to ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) 2026
☆ A Fast Heuristic Search Approach for Energy-Optimal Profile Routing for Electric Vehicles AAAI 2026
Saman Ahmadi, Mahdi Jalili
We study the energy-optimal shortest path problem for electric vehicles (EVs) in large-scale road networks, where recuperated energy along downhill segments introduces negative energy costs. While traditional point-to-point pathfinding algorithms for EVs assume a known initial energy level, many real-world scenarios involving uncertainty in available energy require planning optimal paths for all possible initial energy levels, a task known as energy-optimal profile search. Existing solutions typically rely on specialized profile-merging procedures within a label-correcting framework that results in searching over complex profiles. In this paper, we propose a simple yet effective label-setting approach based on multi-objective A* search, which employs a novel profile dominance rule to avoid generating and handling complex profiles. We develop four variants of our method and evaluate them on real-world road networks enriched with realistic energy consumption data. Experimental results demonstrate that our energy profile A* search achieves performance comparable to energy-optimal A* with a known initial energy level.
comment : 13 pages, 5 Figures, 1 table, To appear as part of AAAI 2026 Proceedings
☆ Extending NGU to Multi- Agent RL : A Preliminary Study NeurIPS 2025
Juan Hernandez, Diego Fernández, Manuel Cifuentes, Denis Parra, Rodrigo Toro Icarte
The Never Give Up (NGU) algorithm has proven effective in reinforcement learning tasks with sparse rewards by combining episodic novelty and intrinsic motivation. In this work, we extend NGU to multi-agent environments and evaluate its performance in the simple_tag environment from the PettingZoo suite. Compared to a multi-agent DQN baseline, NGU achieves moderately higher returns and more stable learning dynamics. We investigate three design choices: (1) shared replay buffer versus individual replay buffers, (2) sharing episodic novelty among agents using different k thresholds, and (3) using heterogeneous values of the beta parameter. Our results show that NGU with a shared replay buffer yields the best performance and stability, highlighting that the gains come from combining NGU intrinsic exploration with experience sharing. Novelty sharing performs comparably when k = 1 but degrades learning for larger values. Finally, heterogeneous beta values do not improve over a small common value. These findings suggest that NGU can be effectively applied in multi-agent settings when experiences are shared and intrinsic exploration signals are carefully tuned.
comment : 9 pages, 4 figures, 1 table. Accepted at the LatinX in AI (LXAI) Workshop at NeurIPS 2025. Includes experimental results for Multi-NGU and Multi-DQN in the PettingZoo simple_tag environment
☆ Rethinking Intracranial Aneurysm Vessel Segmentation: A Perspective from Computational Fluid Dynamics Applications
Feiyang Xiao, Yichi Zhang, Xigui Li, Yuanye Zhou, Chen Jiang, Xin Guo, Limei Han, Yuxin Li, Fengping Zhu, Yuan Cheng
The precise segmentation of intracranial aneurysms and their parent vessels (IA-Vessel) is a critical step for hemodynamic analyses, which mainly depends on computational fluid dynamics (CFD). However, current segmentation methods predominantly focus on image-based evaluation metrics, often neglecting their practical effectiveness in subsequent CFD applications. To address this deficiency, we present the Intracranial Aneurysm Vessel Segmentation (IAVS) dataset, the first comprehensive, multi-center collection comprising 641 3D MRA images with 587 annotations of aneurysms and IA-Vessels. In addition to image-mask pairs, IAVS dataset includes detailed hemodynamic analysis outcomes, addressing the limitations of existing datasets that neglect topological integrity and CFD applicability. To facilitate the development and evaluation of clinically relevant techniques, we construct two evaluation benchmarks including global localization of aneurysms (Stage I) and fine-grained segmentation of IA-Vessel (Stage II) and develop a simple and effective two-stage framework, which can be used as a out-of-the-box method and strong baseline. For comprehensive evaluation of applicability of segmentation results, we establish a standardized CFD applicability evaluation system that enables the automated and consistent conversion of segmentation masks into CFD models, offering an applicability-focused assessment of segmentation outcomes. The dataset, code, and model will be public available at https://github.com/AbsoluteResonance/IAVS.
comment : 18 pages, 5 figures
☆ Data-Driven Learnability Transition of Measurement-Induced Entanglement
Dongheng Qian, Jing Wang
Measurement-induced entanglement (MIE) captures how local measurements generate long-range quantum correlations and drive dynamical phase transitions in many-body systems. Yet estimating MIE experimentally remains challenging: direct evaluation requires extensive post-selection over measurement outcomes, raising the question of whether MIE is accessible with only polynomial resources. We address this challenge by reframing MIE detection as a data-driven learning problem that assumes no prior knowledge of state preparation. Using measurement records alone, we train a neural network in a self-supervised manner to predict the uncertainty metric for MIE--the gap between upper and lower bounds of the average post-measurement bipartite entanglement. Applied to random circuits with one-dimensional all-to-all connectivity and two-dimensional nearest-neighbor coupling, our method reveals a learnability transition with increasing circuit depth: below a threshold, the uncertainty is small and decreases with polynomial measurement data and model parameters, while above it the uncertainty remains large despite increasing resources. We further verify this transition experimentally on current noisy quantum devices, demonstrating its robustness to realistic noise. These results highlight the power of data-driven approaches for learning MIE and delineate the practical limits of its classical learnability.
comment : 7 pages, 4 figures
☆ Agreement-Constrained Probabilistic Minimum Bayes Risk Decoding AACL 2025
Koki Natsumi, Hiroyuki Deguchi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
Minimum Bayes risk (MBR) decoding generates high-quality translations by maximizing the expected utility of output candidates, but it evaluates all pairwise scores over the candidate set; hence, it takes quadratic time with respect to the number of candidates. To reduce the number of utility function calls, probabilistic MBR (PMBR) decoding partially evaluates quality scores using sampled pairs of candidates and completes the missing scores with a matrix completion algorithm. Nevertheless, it degrades the translation quality as the number of utility function calls is reduced. Therefore, to improve the trade-off between quality and cost, we propose agreement-constrained PMBR (AC-PMBR) decoding, which leverages a knowledge distilled model to guide the completion of the score matrix. Our AC-PMBR decoding improved approximation errors of matrix completion by up to 3 times and achieved higher translation quality compared with PMBR decoding at a comparable computational cost on the WMT'23 En$\leftrightarrow$De translation tasks.
comment : IJCNLP-AACL 2025 Main
☆ FOD-S2R: A FOD Dataset for Sim2Real Transfer Learning based Object Detection
Ashish Vashist, Qiranul Saadiyean, Suresh Sundaram, Chandra Sekhar Seelamantula
Foreign Object Debris (FOD) within aircraft fuel tanks presents critical safety hazards including fuel contamination, system malfunctions, and increased maintenance costs. Despite the severity of these risks, there is a notable lack of dedicated datasets for the complex, enclosed environments found inside fuel tanks. To bridge this gap, we present a novel dataset, FOD-S2R, composed of real and synthetic images of the FOD within a simulated aircraft fuel tank. Unlike existing datasets that focus on external or open-air environments, our dataset is the first to systematically evaluate the effectiveness of synthetic data in enhancing the real-world FOD detection performance in confined, closed structures. The real-world subset consists of 3,114 high-resolution HD images captured in a controlled fuel tank replica, while the synthetic subset includes 3,137 images generated using Unreal Engine. The dataset is composed of various Field of views (FOV), object distances, lighting conditions, color, and object size. Prior research has demonstrated that synthetic data can reduce reliance on extensive real-world annotations and improve the generalizability of vision models. Thus, we benchmark several state-of-the-art object detection models and demonstrate that introducing synthetic data improves the detection accuracy and generalization to real-world conditions. These experiments demonstrate the effectiveness of synthetic data in enhancing the model performance and narrowing the Sim2Real gap, providing a valuable foundation for developing automated FOD detection systems for aviation maintenance.
comment : 8 pages, 11 figures
☆ CuES: A Curiosity-driven and Environment-grounded Synthesis Framework for Agent ic RL
Shinji Mai, Yunpeng Zhai, Ziqian Chen, Cheng Chen, Anni Zou, Shuchang Tao, Zhaoyang Liu, Bolin Ding
Large language model based agents are increasingly deployed in complex, tool augmented environments. While reinforcement learning provides a principled mechanism for such agents to improve through interaction, its effectiveness critically depends on the availability of structured training tasks. In many realistic settings, however, no such tasks exist a challenge we term task scarcity, which has become a key bottleneck for scaling agentic RL. Existing approaches typically assume predefined task collections, an assumption that fails in novel environments where tool semantics and affordances are initially unknown. To address this limitation, we formalize the problem of Task Generation for Agentic RL, where an agent must learn within a given environment that lacks predefined tasks. We propose CuES, a Curiosity driven and Environment grounded Synthesis framework that autonomously generates diverse, executable, and meaningful tasks directly from the environment structure and affordances, without relying on handcrafted seeds or external corpora. CuES drives exploration through intrinsic curiosity, abstracts interaction patterns into reusable task schemas, and refines them through lightweight top down guidance and memory based quality control. Across three representative environments, AppWorld, BFCL, and WebShop, CuES produces task distributions that match or surpass manually curated datasets in both diversity and executability, yielding substantial downstream policy improvements. These results demonstrate that curiosity driven, environment grounded task generation provides a scalable foundation for agents that not only learn how to act, but also learn what to learn. The code is available at https://github.com/modelscope/AgentEvolver/research/CuES.
☆ RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision -Language Models for Autonomous Driving
Dacheng Liao, Mengshi Qi, Peng Shu, Zhining Zhang, Yuxin Lin, Liang Liu, Huadong Ma
Current Vision-Language Model (VLM)-based end-to-end autonomous driving systems often leverage large language models to generate driving decisions directly based on their understanding of the current scene. However, such systems introduce multiple risks in real-world driving scenarios. To evaluate whether VLMs are truly viable for autonomous driving, we introduce RoboDriveBench, the first robustness benchmark focused on end-to-end trajectory prediction tasks. This benchmark systematically evaluates two critical categories of real-world challenges for VLM-based end-to-end autonomous driving systems through 11 simulated scenarios encompassing various corruption types, including 6 scenarios of sensor corruption caused by environmental variations, along with 5 cases of prompt corruption resulting from human intervention and data transmission failures. Each corruption type includes 250 unique driving scenarios and 5,689 frames, resulting in 64,559 total trajectory prediction cases per evaluation. To overcome these real-world challenges, we propose a novel VLM-based autonomous driving framework called RoboDriveVLM, which enhances robustness by mapping more multimodal data-e.g., lidar and radar-into a unified latent space. Furthermore, we introduce a new Test-Time Adaptation (TTA) method based on cross-modal knowledge distillation to improve the robustness of VLM-based autonomous driving systems. Through extensive experiments, our work highlights the limitations of current VLM-based end-to-end autonomous driving systems and provides a more reliable solution for real-world deployment. Source code and datasets will be released.
☆ Diffusion Model in Latent Space for Medical Image Segmentation Task
Huynh Trinh Ngoc, Toan Nguyen Hai, Ba Luong Son, Long Tran Quoc
Medical image segmentation is crucial for clinical diagnosis and treatment planning. Traditional methods typically produce a single segmentation mask, failing to capture inherent uncertainty. Recent generative models enable the creation of multiple plausible masks per image, mimicking the collaborative interpretation of several clinicians. However, these approaches remain computationally heavy. We propose MedSegLatDiff, a diffusion based framework that combines a variational autoencoder (VAE) with a latent diffusion model for efficient medical image segmentation. The VAE compresses the input into a low dimensional latent space, reducing noise and accelerating training, while the diffusion process operates directly in this compact representation. We further replace the conventional MSE loss with weighted cross entropy in the VAE mask reconstruction path to better preserve tiny structures such as small nodules. MedSegLatDiff is evaluated on ISIC-2018 (skin lesions), CVC-Clinic (polyps), and LIDC-IDRI (lung nodules). It achieves state of the art or highly competitive Dice and IoU scores while simultaneously generating diverse segmentation hypotheses and confidence maps. This provides enhanced interpretability and reliability compared to deterministic baselines, making the model particularly suitable for clinical deployment.
☆ OntoMetric: An Ontology-Guided Framework for Automated ESG Knowledge Graph Construction
Mingqin Yu, Fethi Rabhi, Boming Xia, Zhengyi Yang, Felix Tan, Qinghua Lu
Environmental, Social, and Governance (ESG) disclosure frameworks such as SASB, TCFD, and IFRS S2 require organizations to compute and report numerous metrics for compliance, yet these requirements are embedded in long, unstructured PDF documents that are difficult to interpret, standardize, and audit. Manual extraction is unscalable, while unconstrained large language model (LLM) extraction often produces inconsistent entities, hallucinated relationships, missing provenance, and high validation failure rates. We present OntoMetric, an ontology-guided framework that transforms ESG regulatory documents into validated, AI- and web-ready knowledge graphs. OntoMetric operates through a three-stage pipeline: (1) structure-aware segmentation using table-of-contents boundaries, (2) ontology-constrained LLM extraction that embeds the ESGMKG schema into prompts while enriching entities with semantic fields for downstream reasoning, and (3) two-phase validation that combines LLM-based semantic verification with rule-based schema checking across entity, property, and relationship levels (VR001-VR006). The framework preserves both segment-level and page-level provenance for audit traceability. Evaluated on five ESG standards (SASB Commercial Banks, SASB Semiconductors, TCFD, IFRS S2, AASB S2) totaling 228 pages and 60 segments, OntoMetric achieves 65-90% semantic accuracy and 80-90% schema compliance, compared to 3-10% for baseline unconstrained extraction, at approximately 0.01 to 0.02 USD per validated entity. Our results demonstrate that combining symbolic ontology constraints with neural extraction enables reliable, auditable knowledge graphs suitable for regulatory compliance and web integration, supporting downstream applications such as sustainable-finance analytics, transparency portals, and automated compliance tools.
☆ Generative Modeling with Continuous Flows: Sample Complexity of Flow Matching
Mudit Gaur, Prashant Trivedi, Shuchin Aeron, Amrit Singh Bedi, George K. Atia, Vaneet Aggarwal
Flow matching has recently emerged as a promising alternative to diffusion-based generative models, offering faster sampling and simpler training by learning continuous flows governed by ordinary differential equations. Despite growing empirical success, the theoretical understanding of flow matching remains limited, particularly in terms of sample complexity results. In this work, we provide the first analysis of the sample complexity for flow-matching based generative models without assuming access to the empirical risk minimizer (ERM) of the loss function for estimating the velocity field. Under standard assumptions on the loss function for velocity field estimation and boundedness of the data distribution, we show that a sufficiently expressive neural network can learn a velocity field such that with $\mathcal{O}(ε^{-4})$ samples, such that the Wasserstein-2 distance between the learned and the true distribution is less than $\mathcal{O}(ε)$. The key technical idea is to decompose the velocity field estimation error into neural-network approximation error, statistical error due to the finite sample size, and optimization error due to the finite number of optimization steps for estimating the velocity field. Each of these terms are then handled via techniques that may be of independent interest.
☆ Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as- Judge Reinforcement Learning
Jiahao Yuan, Zhiqing Cui, Hanqing Wang, Yuansheng Gao, Yucheng Zhou, Usman Naseem
As web platforms evolve towards greater personalization and emotional complexity, conversational agents must transcend superficial empathy to demonstrate identity-aware emotional reasoning. However, existing systems face two limitations: (1) reliance on situation-centric datasets lacking persistent user identity, which hampers the capture of personalized affective nuances; and (2) dependence on opaque, coarse reward signals that hinder development of verifiable empathetic reasoning. To address these gaps, we introduce KardiaBench, a large-scale user-grounded benchmark comprising 178,080 QA pairs across 22,080 multi-turn conversations anchored to 671 real-world profiles. The dataset is constructed via a model-in-the-loop pipeline with iterative rubric-guided refinement to ensure psychological plausibility and persona consistency. This progressive empathy pipeline that integrates user comprehension, contextual reasoning, and emotion perception into conversations, followed by iterative critique and rubric-based refinement to ensure psychological plausibility, emotional fidelity, and persona consistency. Building on this, we propose Kardia-R1, a framework that trains models for interpretable, stepwise empathetic cognition. Kardia-R1 leverages Rubric-as-Judge Empathetic Reinforcement Learning (Rubric-ERL), a GRPO-based method that uses explainable, human-aligned rubric rewards to tightly couple user understanding, emotional inference, and supportive response generation. Extensive experiments across four LLM backbones demonstrate that Kardia-R1 consistently outperforms othet methods in emotion accuracy, empathy, relevance, persona consistency, and safety. Our dataset and model will be released at https://github.com/JhCircle/Kardia-R1.
☆ Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding
Yilong Zhao, Jiaming Tang, Kan Zhu, Zihao Ye, Chi-Chih Chang, Chaofan Lin, Jongseok Park, Guangxuan Xiao, Mohamed S. Abdelfattah, Mingyu Gao, Baris Kasikci, Song Han, Ion Stoica
Reasoning language models have demonstrated remarkable capabilities on challenging tasks by generating elaborate chain-of-thought (CoT) solutions. However, such lengthy generation shifts the inference bottleneck from compute-bound to memory-bound. To generate each token, the model applies full attention to all previously generated tokens, requiring memory access to an increasingly large KV-Cache. Consequently, longer generations demand more memory access for every step, leading to substantial pressure on memory bandwidth. To address this, we introduce SparseSpec, a speculative decoding framework that reuses the same model as the draft and target models (i.e., self-speculation). SparseSpec features a novel sparse attention mechanism, PillarAttn, as the draft model, which accurately selects critical tokens via elegantly reusing information from the verification stage. Furthermore, SparseSpec co-designs self-speculation with three system innovations: (1) a unified scheduler to batch token drafting and verification, (2) delayed verification for CPU/GPU overlap, and (3) dynamic KV-Cache management to maximize memory utilization. Across various models and datasets, SparseSpec outperforms state-of-the-art solutions, with an up to 2.13x throughput speedup.
☆ SUPERChem: A Multimodal Reasoning Benchmark in Chemistry
Zehua Zhao, Zhixian Huang, Junren Li, Siyu Lin, Junting Zhou, Fengqi Cao, Kun Zhou, Rui Ge, Tingting Long, Yuexiang Zhu, Yan Liu, Jie Zheng, Junnian Wei, Rong Zhu, Peng Zou, Wenyu Li, Zekai Cheng, Tian Ding, Yaxuan Wang, Yizhao Yan, Tingru Wei, Haowei Ming, Weijie Mao, Chen Sun, Yiming Liu, Zichen Wang, Zuo Zhang, Tong Yang, Hao Ma, Zhen Gao, Jian Pei
Current benchmarks for evaluating the chemical reasoning capabilities of Large Language Models (LLMs) are limited by oversimplified tasks, lack of process-level evaluation, and misalignment with expert-level chemistry skills. To address these issues, we introduce SUPERChem, a benchmark of 500 expert-curated reasoning-intensive chemistry problems, covering diverse subfields and provided in both multimodal and text-only formats. Original content and an iterative curation pipeline eliminate flawed items and mitigate data contamination. Each problem is paired with an expert-authored solution path, enabling Reasoning Path Fidelity (RPF) scoring to evaluate reasoning quality beyond final-answer accuracy. Evaluations against a human baseline of 40.3% accuracy show that even the best-performing model, GPT-5 (High), reaches only 38.5%, followed closely by Gemini 2.5 Pro (37.9%) and DeepSeek-V3.1-Think (37.3%). SUPERChem elicits multi-step, multimodal reasoning, reveals model-dependent effects of visual information, and distinguishes high-fidelity reasoners from heuristic ones. By providing a challenging benchmark and a reliable evaluation framework, SUPERChem aims to facilitate the advancement of LLMs toward expert-level chemical intelligence. The dataset of the benchmark is available at https://huggingface.co/datasets/ZehuaZhao/SUPERChem.
comment : 35 pages, 11 figures, 5 tables
☆ Social Media Data Mining of Human Behaviour during Bushfire Evacuation
Junfeng Wu, Xiangmin Zhou, Erica Kuligowski, Dhirendra Singh, Enrico Ronchi, Max Kinateder
Traditional data sources on bushfire evacuation behaviour, such as quantitative surveys and manual observations have severe limitations. Mining social media data related to bushfire evacuations promises to close this gap by allowing the collection and processing of a large amount of behavioural data, which are low-cost, accurate, possibly including location information and rich contextual information. However, social media data have many limitations, such as being scattered, incomplete, informal, etc. Together, these limitations represent several challenges to their usefulness to better understand bushfire evacuation. To overcome these challenges and provide guidance on which and how social media data can be used, this scoping review of the literature reports on recent advances in relevant data mining techniques. In addition, future applications and open problems are discussed. We envision future applications such as evacuation model calibration and validation, emergency communication, personalised evacuation training, and resource allocation for evacuation preparedness. We identify open problems such as data quality, bias and representativeness, geolocation accuracy, contextual understanding, crisis-specific lexicon and semantics, and multimodal data interpretation.
☆ Sentiment Analysis and Emotion Classification using Machine Learning Techniques for Nagamese Language - A Low-resource Language
Ekha Morang, Surhoni A. Ngullie, Sashienla Longkumer, Teisovi Angami
The Nagamese language, a.k.a Naga Pidgin, is an Assamese-lexified creole language developed primarily as a means of communication in trade between the people from Nagaland and people from Assam in the north-east India. Substantial amount of work in sentiment analysis has been done for resource-rich languages like English, Hindi, etc. However, no work has been done in Nagamese language. To the best of our knowledge, this is the first attempt on sentiment analysis and emotion classification for the Nagamese Language. The aim of this work is to detect sentiments in terms of polarity (positive, negative and neutral) and basic emotions contained in textual content of Nagamese language. We build sentiment polarity lexicon of 1,195 nagamese words and use these to build features along with additional features for supervised machine learning techniques using Na"ive Bayes and Support Vector Machines. Keywords: Nagamese, NLP, sentiment analysis, machine learning
comment : 10 pages
☆ Pascal-Weighted Genetic Algorithms: A Binomially-Structured Recombination Framework
Otman A. Basir
This paper introduces a new family of multi-parent recombination operators for Genetic Algorithms (GAs), based on normalized Pascal (binomial) coefficients. Unlike classical two-parent crossover operators, Pascal-Weighted Recombination (PWR) forms offsprings as structured convex combination of multiple parents, using binomially shaped weights that emphasize central inheritance while suppressing disruptive variance. We develop a mathematical framework for PWR, derive variance-transfer properties, and analyze its effect on schema survival. The operator is extended to real-valued, binary/logit, and permutation representations. We evaluate the proposed method on four representative benchmarks: (i) PID controller tuning evaluated using the ITAE metric, (ii) FIR low-pass filter design under magnitude-response constraints, (iii) wireless power-modulation optimization under SINR coupling, and (iv) the Traveling Salesman Problem (TSP). We demonstrate how, across these benchmarks, PWR consistently yields smoother convergence, reduced variance, and achieves 9-22% performance gains over standard recombination operators. The approach is simple, algorithm-agnostic, and readily integrable into diverse GA architectures.
comment : 23 pages, 8 figures
☆ Generative Adversarial Gumbel MCTS for Abstract Visual Composition Generation
Zirui Zhao, Boye Niu, David Hsu, Wee Sun Lee
We study abstract visual composition, in which identity is primarily determined by the spatial configuration and relations among a small set of geometric primitives (e.g., parts, symmetry, topology). They are invariant primarily to texture and photorealistic detail. Composing such structures from fixed components under geometric constraints and vague goal specification (such as text) is non-trivial due to combinatorial placement choices, limited data, and discrete feasibility (overlap-free, allowable orientations), which create a sparse solution manifold ill-suited to purely statistical pixel-space generators. We propose a constraint-guided framework that combines explicit geometric reasoning with neural semantics. An AlphaGo-style search enforces feasibility, while a fine-tuned vision-language model scores semantic alignment as reward signals. Our algorithm uses a policy network as a heuristic in Monte-Carlo Tree Search and fine-tunes the network via search-generated plans. Inspired by the Generative Adversarial Network, we use the generated instances for adversarial reward refinement. Over time, the generation should approach the actual data more closely when the reward model cannot distinguish between generated instances and ground-truth. In the Tangram Assembly task, our approach yields higher validity and semantic fidelity than diffusion and auto-regressive baselines, especially as constraints tighten.
☆ First, do NOHARM: towards clinically safe large language models
David Wu, Fateme Nateghi Haredasht, Saloni Kumar Maharaj, Priyank Jain, Jessica Tran, Matthew Gwiazdon, Arjun Rustagi, Jenelle Jindal, Jacob M. Koshy, Vinay Kadiyala, Anup Agarwal, Bassman Tappuni, Brianna French, Sirus Jesudasen, Christopher V. Cosgriff, Rebanta Chakraborty, Jillian Caldwell, Susan Ziolkowski, David J. Iberri, Robert Diep, Rahul S. Dalal, Kira L. Newman, Kristin Galetta, J. Carl Pallais, Nancy Wei, Kathleen M. Buchheit, David I. Hong, Ernest Y. Lee, Allen Shih, Vartan Pahalyants, Tamara B. Kaplan, Vishnu Ravi, Sarita Khemani, April S. Liang, Daniel Shirvani, Advait Patil, Nicholas Marshall, Kanav Chopra, Joel Koh, Adi Badhwar, Liam G. McCoy, David J. H. Wu, Yingjie Weng, Sumant Ranji, Kevin Schulman, Nigam H. Shah, Jason Hom, Arnold Milstein, Adam Rodman, Jonathan H. Chen, Ethan Goh
Large language models (LLMs) are routinely used by physicians and patients for medical advice, yet their clinical safety profiles remain poorly characterized. We present NOHARM (Numerous Options Harm Assessment for Risk in Medicine), a benchmark using 100 real primary-care-to-specialist consultation cases to measure harm frequency and severity from LLM-generated medical recommendations. NOHARM covers 10 specialties, with 12,747 expert annotations for 4,249 clinical management options. Across 31 LLMs, severe harm occurs in up to 22.2% (95% CI 21.6-22.8%) of cases, with harms of omission accounting for 76.6% (95% CI 76.4-76.8%) of errors. Safety performance is only moderately correlated (r = 0.61-0.64) with existing AI and medical knowledge benchmarks. The best models outperform generalist physicians on safety (mean difference 9.7%, 95% CI 7.0-12.5%), and a diverse multi-agent approach reduces harm compared to solo models (mean difference 8.0%, 95% CI 4.0-12.1%). Therefore, despite strong performance on existing evaluations, widely used AI models can produce severely harmful medical advice at nontrivial rates, underscoring clinical safety as a distinct performance dimension necessitating explicit measurement.
☆ Proactive Agent ic Whiteboards: Enhancing Diagrammatic Learning
Suveen Ellawala, Sashenka Gamage, Dinithi Dissanayake
Educators frequently rely on diagrams to explain complex concepts during lectures, yet creating clear and complete visual representations in real time while simultaneously speaking can be cognitively demanding. Incomplete or unclear diagrams may hinder student comprehension, as learners must mentally reconstruct missing information while following the verbal explanation. Inspired by advances in code completion tools, we introduce DrawDash, an AI-powered whiteboard assistant that proactively completes and refines educational diagrams through multimodal understanding. DrawDash adopts a TAB-completion interaction model: it listens to spoken explanations, detects intent, and dynamically suggests refinements that can be accepted with a single keystroke. We demonstrate DrawDash across four diverse teaching scenarios, spanning topics from computer science and web development to biology. This work represents an early exploration into reducing instructors' cognitive load and improving diagram-based pedagogy through real-time, speech-driven visual assistance, and concludes with a discussion of current limitations and directions for formal classroom evaluation.
☆ LLM-as-a- Judge for Scalable Test Coverage Evaluation: Accuracy, Operational Reliability, and Cost AAAI 2026
Donghao Huang, Shila Chew, Anna Dutkiewicz, Zhaoxia Wang
Assessing software test coverage at scale remains a bottleneck in QA pipelines. We present LLM-as-a-Judge (LAJ), a production-ready, rubric-driven framework for evaluating Gherkin acceptance tests with structured JSON outputs. Across 20 model configurations (GPT-4, GPT-5 with varying reasoning effort, and open-weight models) on 100 expert-annotated scripts over 5 runs (500 evaluations), we provide the first comprehensive analysis spanning accuracy, operational reliability, and cost. We introduce the Evaluation Completion Rate (ECR@1) to quantify first-attempt success, revealing reliability from 85.4% to 100.0% with material cost implications via retries. Results show that smaller models can outperform larger ones: GPT-4o Mini attains the best accuracy (6.07 MAAE), high reliability (96.6% ECR@1), and low cost ($1.01 per 1K), yielding a 78x cost reduction vs. GPT-5 (high reasoning) while improving accuracy. Reasoning effort is model-family dependent: GPT-5 benefits from increased reasoning (with predictable accuracy-cost tradeoffs), whereas open-weight models degrade across all dimensions as reasoning increases. Overall, cost spans 175x ($0.45-$78.96 per 1K). We release the dataset, framework, and code to support reproducibility and deployment.
comment : 7 pages, accepted by the AAAI 2026 Workshop on Next Gen Code Development with Collaborative AI Agents
☆ S$^2$-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance
Beining Xu, Siting Zhu, Zhao Jin, Junxian Li, Hesheng Wang
3D Visual Grounding (3DVG) focuses on locating objects in 3D scenes based on natural language descriptions, serving as a fundamental task for embodied AI and robotics. Recent advances in Multi-modal Large Language Models (MLLMs) have motivated research into extending them to 3DVG. However, MLLMs primarily process 2D visual inputs and struggle with understanding 3D spatial structure of scenes solely from these limited perspectives. Existing methods mainly utilize viewpoint-dependent rendering of reconstructed point clouds to provide explicit structural guidance for MLLMs in 3DVG tasks, leading to inefficiency and limited spatial reasoning. To address this issue, we propose S$^2$-MLLM, an efficient framework that enhances spatial reasoning in MLLMs through implicit spatial reasoning. We introduce a spatial guidance strategy that leverages the structure awareness of feed-forward 3D reconstruction. By acquiring 3D structural understanding during training, our model can implicitly reason about 3D scenes without relying on inefficient point cloud reconstruction. Moreover, we propose a structure-enhanced module (SE), which first employs intra-view and inter-view attention mechanisms to capture dependencies within views and correspondences across views. The module further integrates multi-level position encoding to associate visual representations with spatial positions and viewpoint information, enabling more accurate structural understanding. Extensive experiments demonstrate that S$^2$-MLLM unifies superior performance, generalization, and efficiency, achieving significant performance over existing methods across the ScanRefer, Nr3D, and Sr3D datasets. Code will be available upon acceptance.
comment : 18 pages, 9 figures
☆ Unsupervised decoding of encoded reasoning using language model interpretability
Ching Fang, Samuel Marks
As large language models become increasingly capable, there is growing concern that they may develop reasoning processes that are encoded or hidden from human oversight. To investigate whether current interpretability techniques can penetrate such encoded reasoning, we construct a controlled testbed by fine-tuning a reasoning model (DeepSeek-R1-Distill-Llama-70B) to perform chain-of-thought reasoning in ROT-13 encryption while maintaining intelligible English outputs. We evaluate mechanistic interpretability methods--in particular, logit lens analysis--on their ability to decode the model's hidden reasoning process using only internal activations. We show that logit lens can effectively translate encoded reasoning, with accuracy peaking in intermediate-to-late layers. Finally, we develop a fully unsupervised decoding pipeline that combines logit lens with automated paraphrasing, achieving substantial accuracy in reconstructing complete reasoning transcripts from internal model representations. These findings suggest that current mechanistic interpretability techniques may be more robust to simple forms of encoded reasoning than previously understood. Our work provides an initial framework for evaluating interpretability methods against models that reason in non-human-readable formats, contributing to the broader challenge of maintaining oversight over increasingly capable AI systems.
☆ Neural Network Optimal Power Flow via Energy Gradient Flow and Unified Dynamics
Xuezhi Liu
Optimal Power Flow (OPF) is a core optimization problem in power system operation and planning, aiming to minimize generation costs while satisfying physical constraints such as power flow equations, generator limits, and voltage limits. Traditional OPF solving methods typically employ iterative optimization algorithms (such as interior point methods, sequential quadratic programming, etc.), with limitations including low computational efficiency, initial value sensitivity, and low batch computation efficiency. Most existing deep learning-based OPF methods rely on supervised learning, requiring pre-solving large numbers of cases, and have difficulty guaranteeing physical consistency. This paper proposes an Optimal Power Flow solving method based on neural network dynamics and energy gradient flow, transforming OPF problems into energy minimization problems. By constructing an energy function to measure the degree of deviation from the constraint manifold, and guiding networks to learn optimal solutions that simultaneously satisfy power flow constraints and minimize costs through gradient flow. Neural networks are trained unsupervised by directly minimizing physical residuals, requiring no labeled data, achieving true "end-to-end" physics-constrained learning.
☆ How do trout regulate patterns of muscle contraction to optimize propulsive efficiency during steady swimming
Tao Li, Chunze Zhang, Weiwei Yao, Junzhao He, Ji Hou, Qin Zhou, Lu Zhang
Understanding efficient fish locomotion offers insights for biomechanics, fluid dynamics, and engineering. Traditional studies often miss the link between neuromuscular control and whole-body movement. To explore energy transfer in carangiform swimming, we created a bio-inspired digital trout. This model combined multibody dynamics, Hill-type muscle modeling, and a high-fidelity fluid-structure interaction algorithm, accurately replicating a real trout's form and properties. Using deep reinforcement learning, the trout's neural system achieved hierarchical spatiotemporal control of muscle activation. We systematically examined how activation strategies affect speed and energy use. Results show that axial myomere coupling-with activation spanning over 0.5 body lengths-is crucial for stable body wave propagation. Moderate muscle contraction duration ([0.1,0.3] of a tail-beat cycle) lets the body and fluid act as a passive damping system, cutting energy use. Additionally, the activation phase lag of myomeres shapes the body wave; if too large, it causes antagonistic contractions that hinder thrust. These findings advance bio-inspired locomotion understanding and aid energy-efficient underwater system design.
☆ M4-BLIP: Advancing Multi-Modal Media Manipulation Detection through Face-Enhanced Local Analysis
Hang Wu, Ke Sun, Jiayi Ji, Xiaoshuai Sun, Rongrong Ji
In the contemporary digital landscape, multi-modal media manipulation has emerged as a significant societal threat, impacting the reliability and integrity of information dissemination. Current detection methodologies in this domain often overlook the crucial aspect of localized information, despite the fact that manipulations frequently occur in specific areas, particularly in facial regions. In response to this critical observation, we propose the M4-BLIP framework. This innovative framework utilizes the BLIP-2 model, renowned for its ability to extract local features, as the cornerstone for feature extraction. Complementing this, we incorporate local facial information as prior knowledge. A specially designed alignment and fusion module within M4-BLIP meticulously integrates these local and global features, creating a harmonious blend that enhances detection accuracy. Furthermore, our approach seamlessly integrates with Large Language Models (LLM), significantly improving the interpretability of the detection outcomes. Extensive quantitative and visualization experiments validate the effectiveness of our framework against the state-of-the-art competitors.
comment : 12 pages, 6 figures
☆ Knowledge Graph Augmented Large Language Models for Next-Visit Disease Prediction
Ruiyu Wang, Tuan Vinh, Ran Xu, Yuyin Zhou, Jiaying Lu, Carl Yang, Francisco Pasquel
Electronic health records (EHRs) support powerful clinical prediction models, but existing methods typically provide coarse, post hoc explanations that offer limited value for patient-level decision making. We introduce a knowledge graph (KG)-guided chain-of-thought (CoT) framework that generates clinically grounded and temporally consistent reasoning for visit-level disease prediction in MIMIC-III. ICD-9 codes are mapped to PrimeKG, from which disease-relevant nodes and multi-hop reasoning paths are extracted and used as scaffolds for CoT generation; only explanations whose conclusions match observed outcomes are retained. Lightweight LLaMA-3.1-Instruct-8B and Gemma-7B models are then fine-tuned on this supervision corpus. Across ten PrimeKG-mapped diseases and limited training cohorts (400 and 1000 cases), KG-guided models outperform strong classical baselines, achieving AUROC values of 0.66 to 0.70 and macro-AUPR values of 0.40 to 0.47. The models also transfer zero-shot to the CRADLE cohort, improving accuracy from approximately 0.40 to 0.51 up to 0.72 to 0.77. A blinded clinician evaluation shows consistent preference for KG-guided CoT explanations in clarity, relevance, and clinical correctness.
☆ Pay Attention Later: From Vector Space Diffusion to Linearithmic Spectral Phase-Locking
Alper Yıldırım, İbrahim Yücedağ
Standard Transformers suffer from a "Semantic Alignment Tax", a prohibitive optimization cost required to organize a chaotic initialization into a coherent geometric map via local gradient diffusion. We hypothesize that this reliance on diffusive learning creates "Catastrophic Rigidity", rendering models unable to adapt to novel concepts without destroying their pre-trained reasoning capabilities. To isolate this phenomenon, we introduce Iterative Semantic Map Refinement (ISMR), a diagnostic protocol revealing that alignment is a fixed geometric barrier that scaling cannot solve; a 20-layer model overcomes this barrier no faster than a 1-layer model. We introduce the Phase-Resonant Intelligent Spectral Model (PRISM). PRISM encodes semantic identity as resonant frequencies in the complex domain (C^d) and replaces quadratic self-attention with linearithmic O(N log N) Gated Harmonic Convolutions. We validate PRISM on the WMT14 translation task. While the Standard Transformer maintains a slight edge in general competence on static benchmarks (23.88 vs 21.40 BLEU), it fails the "Plasticity-Stability" stress test completely. When injected with novel concepts, the Transformer suffers Catastrophic Forgetting, degrading by -10.55 BLEU points while achieving only 60% acquisition. In contrast, PRISM demonstrates Lossless Plasticity, achieving 96% 5-shot acquisition with negligible degradation (-0.84 BLEU). These results suggest that harmonic representations effectively decouple memory from reasoning, offering a structural solution to the plasticity-stability dilemma in real-time knowledge adaptation.
comment : 12 pages, 5 figures
☆ Physics-Constrained Neural Dynamics: A Unified Manifold Framework for Large-Scale Power Flow Computation
Xuezhi Liu
Power flow analysis is a fundamental tool for power system analysis, planning, and operational control. Traditional Newton-Raphson methods suffer from limitations such as initial value sensitivity and low efficiency in batch computation, while existing deep learning-based power flow solvers mostly rely on supervised learning, requiring pre-solving of numerous cases and struggling to guarantee physical consistency. This paper proposes a neural physics power flow solving method based on manifold geometry and gradient flow, by describing the power flow equations as a constraint manifold, and constructing an energy function \(V(\mathbf{x}) = \frac{1}{2}\|\mathbf{F}(\mathbf{x})\|^2\) and gradient flow \(\frac{d\mathbf{x}}{dt} = -\nabla V(\mathbf{x})\), transforming power flow solving into an equilibrium point finding problem for dynamical systems. Neural networks are trained in an unsupervised manner by directly minimizing physical residuals, requiring no labeled data, achieving true "end-to-end" physics-constrained learning.
☆ fMRI2GES: Co-speech Gesture Reconstruction from fMRI Signal with Dual Brain Decoding Alignment
Chunzheng Zhu, Jialin Shao, Jianxin Lin, Yijun Wang, Jing Wang, Jinhui Tang, Kenli Li
Understanding how the brain responds to external stimuli and decoding this process has been a significant challenge in neuroscience. While previous studies typically concentrated on brain-to-image and brain-to-language reconstruction, our work strives to reconstruct gestures associated with speech stimuli perceived by brain. Unfortunately, the lack of paired \{brain, speech, gesture\} data hinders the deployment of deep learning models for this purpose. In this paper, we introduce a novel approach, \textbf{fMRI2GES}, that allows training of fMRI-to-gesture reconstruction networks on unpaired data using \textbf{Dual Brain Decoding Alignment}. This method relies on two key components: (i) observed texts that elicit brain responses, and (ii) textual descriptions associated with the gestures. Then, instead of training models in a completely supervised manner to find a mapping relationship among the three modalities, we harness an fMRI-to-text model, a text-to-gesture model with paired data and an fMRI-to-gesture model with unpaired data, establishing dual fMRI-to-gesture reconstruction patterns. Afterward, we explicitly align two outputs and train our model in a self-supervision way. We show that our proposed method can reconstruct expressive gestures directly from fMRI recordings. We also investigate fMRI signals from different ROIs in the cortex and how they affect generation results. Overall, we provide new insights into decoding co-speech gestures, thereby advancing our understanding of neuroscience and cognitive science.
comment : IEEE Transactions on Circuits and Systems for Video Technology (TCSVT) 2025
☆ Real-Wo rl d Reinforcement Learning of Active Perception Behaviors NeurIPS 2025
Edward S. Hu, Jie Wang, Xingfang Yuan, Fiona Luo, Muyao Li, Gaspard Lambrechts, Oleh Rybkin, Dinesh Jayaraman
A robot's instantaneous sensory observations do not always reveal task-relevant state information. Under such partial observability, optimal behavior typically involves explicitly acting to gain the missing information. Today's standard robot learning techniques struggle to produce such active perception behaviors. We propose a simple real-world robot learning recipe to efficiently train active perception policies. Our approach, asymmetric advantage weighted regression (AAWR), exploits access to "privileged" extra sensors at training time. The privileged sensors enable training high-quality privileged value functions that aid in estimating the advantage of the target policy. Bootstrapping from a small number of potentially suboptimal demonstrations and an easy-to-obtain coarse policy initialization, AAWR quickly acquires active perception behaviors and boosts task performance. In evaluations on 8 manipulation tasks on 3 robots spanning varying degrees of partial observability, AAWR synthesizes reliable active perception behaviors that outperform all prior approaches. When initialized with a "generalist" robot policy that struggles with active perception tasks, AAWR efficiently generates information-gathering behaviors that allow it to operate under severe partial observability for manipulation tasks. Website: https://penn-pal-lab.github.io/aawr/
comment : NeurIPS 2025 camera ready
☆ Teaching by Failure: Counter-Example-Driven Curricula for Transformer Self-Improvement AACL 2025
Harshil Vejendla
Transformer models often exhibit brittle extrapolation, failing on inputs that are longer or structurally more complex than those seen during training. We introduce Counter-Example-Driven Curricula (CEDC), an automated framework that improves model robustness by iteratively focusing on its own failures. At each step, CEDC uses the current model to generate a diverse set of candidate problems, employs a fast, executable verifier to identify incorrect predictions (counter-examples), and then fine-tunes the model on a dataset enriched with these discovered failures. We evaluate CEDC on a suite of algorithmic and natural language tasks, including integer addition, sorting, Dyck-2 language recognition, and three text classification benchmarks. Compared to static training and standard curriculum learning baselines, CEDC achieves up to 30x greater length extrapolation, is 3.75x more computationally efficient than uniform data augmentation, and requires no manual difficulty heuristics. We provide a detailed analysis of the counter-examples, showing how the curriculum naturally adapts to target progressively more complex error modes. Our findings establish verifier-guided, failure-driven learning as a simple, powerful, and efficient paradigm for enhancing the generalization capabilities of Transformer models.
comment : AACL 2025 Findings
☆ TempPerturb-Eval: On the Joint Effects of Internal Temperature and External Perturbations in RAG Robustness
Yongxin Zhou, Philippe Mulhem, Didier Schwab
The evaluation of Retrieval-Augmented Generation (RAG) systems typically examines retrieval quality and generation parameters like temperature in isolation, overlooking their interaction. This work presents a systematic investigation of how text perturbations (simulating noisy retrieval) interact with temperature settings across multiple LLM runs. We propose a comprehensive RAG Perturbation-Temperature Analysis Framework that subjects retrieved documents to three distinct perturbation types across varying temperature settings. Through extensive experiments on HotpotQA with both open-source and proprietary LLMs, we demonstrate that performance degradation follows distinct patterns: high-temperature settings consistently amplify vulnerability to perturbations, while certain perturbation types exhibit non-linear sensitivity across the temperature range. Our work yields three key contributions: (1) a diagnostic benchmark for assessing RAG robustness, (2) an analytical framework for quantifying perturbation-temperature interactions, and (3) practical guidelines for model selection and parameter tuning under noisy retrieval conditions.
☆ First On-Orbit Demonstration of a Geospatial Foundation Model
Andrew Du, Roberto Del Prete, Alejandro Mousist, Nick Manser, Fabrice Marre, Andrew Barton, Carl Seubert, Gabriele Meoni, Tat-Jun Chin
Geospatial foundation models (GeoFMs) promise broad generalisation capacity for Earth observation (EO) tasks, particularly under data-limited conditions. However, their large size poses a barrier to deployment on resource-constrained space hardware. To address this, we present compact variants of a Vision Transformer (ViT)-based GeoFM that preserve downstream task performance while enabling onboard execution. Evaluation across five downstream tasks and validation in two representative flight environments show that model compression and domain adaptation are critical to reducing size and resource demands while maintaining high performance under operational conditions. We further demonstrate reliable on-orbit inference with the IMAGIN-e payload aboard the International Space Station. These results establish a pathway from large GeoFMs to flight-ready, resource-efficient deployments, expanding the feasibility of onboard AI for EO missions.
☆ Toward a benchmark for CTR prediction in online advertising: dataset s, evaluation protocols and perspectives
Shan Gao, Yanwu Yang
This research designs a unified architecture of CTR prediction benchmark (Bench-CTR) platform that offers flexible interfaces with datasets and components of a wide range of CTR prediction models. Moreover, we construct a comprehensive system of evaluation protocols encompassing real-world and synthetic datasets, a taxonomy of metrics, standardized procedures and experimental guidelines for calibrating the performance of CTR prediction models. Furthermore, we implement the proposed benchmark platform and conduct a comparative study to evaluate a wide range of state-of-the-art models from traditional multivariate statistical to modern large language model (LLM)-based approaches on three public datasets and two synthetic datasets. Experimental results reveal that, (1) high-order models largely outperform low-order models, though such advantage varies in terms of metrics and on different datasets; (2) LLM-based models demonstrate a remarkable data efficiency, i.e., achieving the comparable performance to other models while using only 2% of the training data; (3) the performance of CTR prediction models has achieved significant improvements from 2015 to 2016, then reached a stage with slow progress, which is consistent across various datasets. This benchmark is expected to facilitate model development and evaluation and enhance practitioners' understanding of the underlying mechanisms of models in the area of CTR prediction. Code is available at https://github.com/NuriaNinja/Bench-CTR.
comment : 64 pages, 8 figures, 11 tables
☆ Conversion rate prediction in online advertising: modeling techniques, performance evaluation and future directions
Tao Xue, Yanwu Yang, Panyu Zhai
Conversion and conversion rate (CVR) prediction play a critical role in efficient advertising decision-making. In past decades, although researchers have developed plenty of models for CVR prediction, the methodological evolution and relationships between different techniques have been precluded. In this paper, we conduct a comprehensive literature review on CVR prediction in online advertising, and classify state-of-the-art CVR prediction models into six categories with respect to the underlying techniques and elaborate on connections between these techniques. For each category of models, we present the framework of underlying techniques, their advantages and disadvantages, and discuss how they are utilized for CVR prediction. Moreover, we summarize the performance of various CVR prediction models on public and proprietary datasets. Finally, we identify research trends, major challenges, and promising future directions. We observe that results of performance evaluation reported in prior studies are not unanimous; semantics-enriched, attribution-enhanced, debiased CVR prediction and jointly modeling CTR and CVR prediction would be promising directions to explore in the future. This review is expected to provide valuable references and insights for future researchers and practitioners in this area.
comment : 99 pages, 15 figures, 7 tables
☆ Data assimilation and discrepancy modeling with shallow recurrent decoders
Yuxuan Bao, J. Nathan Kutz
The requirements of modern sensing are rapidly evolving, driven by increasing demands for data efficiency, real-time processing, and deployment under limited sensing coverage. Complex physical systems are often characterized through the integration of a limited number of point sensors in combination with scientific computations which approximate the dominant, full-state dynamics. Simulation models, however, inevitably neglect small-scale or hidden processes, are sensitive to perturbations, or oversimplify parameter correlations, leading to reconstructions that often diverge from the reality measured by sensors. This creates a critical need for data assimilation, the process of integrating observational data with predictive simulation models to produce coherent and accurate estimates of the full state of complex physical systems. We propose a machine learning framework for Data Assimilation with a SHallow REcurrent Decoder (DA-SHRED) which bridges the simulation-to-real (SIM2REAL) gap between computational modeling and experimental sensor data. For real-world physics systems modeling high-dimensional spatiotemporal fields, where the full state cannot be directly observed and must be inferred from sparse sensor measurements, we leverage the latent space learned from a reduced simulation model via SHRED, and update these latent variables using real sensor data to accurately reconstruct the full system state. Furthermore, our algorithm incorporates a sparse identification of nonlinear dynamics based regression model in the latent space to identify functionals corresponding to missing dynamics in the simulation model. We demonstrate that DA-SHRED successfully closes the SIM2REAL gap and additionally recovers missing dynamics in highly complex systems, demonstrating that the combination of efficient temporal encoding and physics-informed correction enables robust data assimilation.
comment : 27 pages, 11 figures
☆ A TinyML Reinforcement Learning Approach for Energy-Efficient Light Control in Low-Cost Greenhouse Systems CEC 2025
Mohamed Abdallah Salem, Manuel Cuevas Perez, Ahmed Harb Rabia
This study presents a reinforcement learning (RL)-based control strategy for adaptive lighting regulation in controlled environments using a low-power microcontroller. A model-free Q-learning algorithm was implemented to dynamically adjust the brightness of a Light-Emitting Diode (LED) based on real-time feedback from a light-dependent resistor (LDR) sensor. The system was trained to stabilize at 13 distinct light intensity levels (L1 to L13), with each target corresponding to a specific range within the 64-state space derived from LDR readings. A total of 130 trials were conducted, covering all target levels with 10 episodes each. Performance was evaluated in terms of convergence speed, steps taken, and time required to reach target states. Box plots and histograms were generated to analyze the distribution of training time and learning efficiency across targets. Experimental validation demonstrated that the agent could effectively learn to stabilize at varying light levels with minimal overshooting and smooth convergence, even in the presence of environmental perturbations. This work highlights the feasibility of lightweight, on-device RL for energy-efficient lighting control and sets the groundwork for multi-modal environmental control applications in resource-constrained agricultural systems.
comment : Copyright 2025 IEEE. This is the author's version of the work that has been accepted for publication in Proceedings of the 5. Interdisciplinary Conference on Electrics and Computer (INTCEC 2025) 15-16 September 2025, Chicago-USA. The final version of record is available at: https://doi.org/10.1109/INTCEC65580.2025.11256135
☆ Real-Time On-the-Go Annotation Framework Using YOLO for Automated Dataset Generation CEC 2025
Mohamed Abdallah Salem, Ahmed Harb Rabia
Efficient and accurate annotation of datasets remains a significant challenge for deploying object detection models such as You Only Look Once (YOLO) in real-world applications, particularly in agriculture where rapid decision-making is critical. Traditional annotation techniques are labor-intensive, requiring extensive manual labeling post data collection. This paper presents a novel real-time annotation approach leveraging YOLO models deployed on edge devices, enabling immediate labeling during image capture. To comprehensively evaluate the efficiency and accuracy of our proposed system, we conducted an extensive comparative analysis using three prominent YOLO architectures (YOLOv5, YOLOv8, YOLOv12) under various configurations: single-class versus multi-class annotation and pretrained versus scratch-based training. Our analysis includes detailed statistical tests and learning dynamics, demonstrating significant advantages of pretrained and single-class configurations in terms of model convergence, performance, and robustness. Results strongly validate the feasibility and effectiveness of our real-time annotation framework, highlighting its capability to drastically reduce dataset preparation time while maintaining high annotation quality.
comment : Copyright 2025 IEEE. This is the author's version of the work that has been accepted for publication in Proceedings of the 5. Interdisciplinary Conference on Electrics and Computer (INTCEC 2025) 15-16 September 2025, Chicago-USA. The final version of record is available at: https://doi.org/10.1109/INTCEC65580.2025.11256048
☆ 2D-ThermAl: Physics-Informed Framework for Thermal Analysis of Circuits using Generative AI
Soumyadeep Chandra, Sayeed Shafayet Chowdhury, Kaushik Roy
Thermal analysis is increasingly critical in modern integrated circuits, where non-uniform power dissipation and high transistor densities can cause rapid temperature spikes and reliability concerns. Traditional methods, such as FEM-based simulations offer high accuracy but computationally prohibitive for early-stage design, often requiring multiple iterative redesign cycles to resolve late-stage thermal failures. To address these challenges, we propose 'ThermAl', a physics-informed generative AI framework which effectively identifies heat sources and estimates full-chip transient and steady-state thermal distributions directly from input activity profiles. ThermAl employs a hybrid U-Net architecture enhanced with positional encoding and a Boltzmann regularizer to maintain physical fidelity. Our model is trained on an extensive dataset of heat dissipation maps, ranging from simple logic gates (e.g., inverters, NAND, XOR) to complex designs, generated via COMSOL. Experimental results demonstrate that ThermAl delivers precise temperature mappings for large circuits, with a root mean squared error (RMSE) of only 0.71°C, and outperforms conventional FEM tools by running up to ~200 times faster. We analyze performance across diverse layouts and workloads, and discuss its applicability to large-scale EDA workflows. While thermal reliability assessments often extend beyond 85°C for post-layout signoff, our focus here is on early-stage hotspot detection and thermal pattern learning. To ensure generalization beyond the nominal operating range 25-55°C, we additionally performed cross-validation on an extended dataset spanning 25-95°C maintaining a high accuracy (<2.2% full-scale RMSE) even under elevated temperature conditions representative of peak power and stress scenarios.
comment : 10 pages, 8 figures, Under Review
☆ Beyond Greenfield: AI-Driven Productivity in Document ation and Brownfield Engineering
Krishna Kumaar Sharma
Brownfield engineering work involving legacy systems, incomplete documentation, and fragmented architectural knowledge poses unique challenges for the effective use of large language models (LLMs). Prior research has largely focused on greenfield or synthetic tasks, leaving a gap in structured workflows for complex, context-heavy environments. This paper introduces the Discover-Define-Deliver (D3) Framework, a disciplined LLM-assisted workflow that combines role-separated prompting strategies with applied best practices for navigating ambiguity in brownfield systems. The framework incorporates a dual-agent prompting architecture in which a Builder model generates candidate outputs and a Reviewer model provides structured critique to improve reliability. I conducted an exploratory survey study with 52 software practitioners who applied the D3 workflow to real-world engineering tasks such as legacy system exploration, documentation reconstruction, and architectural refactoring. Respondents reported perceived improvements in task clarity, documentation quality, and cognitive load, along with self-estimated productivity gains. In this exploratory study, participants reported a weighted average productivity improvement of 26.9%, reduced cognitive load for approximately 77% of participants, and reduced rework for 83% during the Define phase. As these findings are self-reported and not derived from controlled experiments, they should be interpreted as preliminary evidence of practitioner sentiment rather than causal effects. The results highlight both the potential and limitations of structured LLM workflows for legacy engineering systems and motivate future controlled evaluations.
comment : 53 pages, 7 figures
☆ DPAC: Distribution-Preserving Adversarial Control for Diffusion Sampling
Han-Jin Lee, Han-Ju Lee, Jin-Seong Kim, Seok-Hwan Choi
Adversarially guided diffusion sampling often achieves the target class, but sample quality degrades as deviations between the adversarially controlled and nominal trajectories accumulate. We formalize this degradation as a path-space Kullback-Leibler divergence(path-KL) between controlled and nominal (uncontrolled) diffusion processes, thereby showing via Girsanov's theorem that it exactly equals the control energy. Building on this stochastic optimal control (SOC) view, we theoretically establish that minimizing this path-KL simultaneously tightens upper bounds on both the 2-Wasserstein distance and Fréchet Inception Distance (FID), revealing a principled connection between adversarial control energy and perceptual fidelity. From a variational perspective, we derive a first-order optimality condition for the control: among all directions that yield the same classification gain, the component tangent to iso-(log-)density surfaces (i.e., orthogonal to the score) minimizes path-KL, whereas the normal component directly increases distributional drift. This leads to DPAC (Distribution-Preserving Adversarial Control), a diffusion guidance rule that projects adversarial gradients onto the tangent space defined by the generative score geometry. We further show that in discrete solvers, the tangent projection cancels the O(Δt) leading error term in the Wasserstein distance, achieving an O(Δt^2) quality gap; moreover, it remains second-order robust to score or metric approximation. Empirical studies on ImageNet-100 validate the theoretical predictions, confirming that DPAC achieves lower FID and estimated path-KL at matched attack success rates.
☆ Open-Set Domain Adaptation Under Background Distribution Shift: Challenges and A Provably Efficient Solution
Shravan Chaudhari, Yoav Wald, Suchi Saria
As we deploy machine learning systems in the real world, a core challenge is to maintain a model that is performant even as the data shifts. Such shifts can take many forms: new classes may emerge that were absent during training, a problem known as open-set recognition, and the distribution of known categories may change. Guarantees on open-set recognition are mostly derived under the assumption that the distribution of known classes, which we call \emph{the background distribution}, is fixed. In this paper we develop \ours{}, a method that is guaranteed to solve open-set recognition even in the challenging case where the background distribution shifts. We prove that the method works under benign assumptions that the novel class is separable from the non-novel classes, and provide theoretical guarantees that it outperforms a representative baseline in a simplified overparameterized setting. We develop techniques to make \ours{} scalable and robust, and perform comprehensive empirical evaluations on image and text data. The results show that \ours{} significantly outperforms existing open-set recognition methods under background shift. Moreover, we provide new insights into how factors such as the size of the novel class influences performance, an aspect that has not been extensively explored in prior work.
☆ Model Recovery at the Edge under Resource Constraints for Physical AI ECAI 2025
Bin Xu, Ayan Banerjee, Sandeep K. S. Gupta
Model Recovery (MR) enables safe, explainable decision making in mission-critical autonomous systems (MCAS) by learning governing dynamical equations, but its deployment on edge devices is hindered by the iterative nature of neural ordinary differential equations (NODEs), which are inefficient on FPGAs. Memory and energy consumption are the main concerns when applying MR on edge devices for real-time operation. We propose MERINDA, a novel FPGA-accelerated MR framework that replaces iterative solvers with a parallelizable neural architecture equivalent to NODEs. MERINDA achieves nearly 11x lower DRAM usage and 2.2x faster runtime compared to mobile GPUs. Experiments reveal an inverse relationship between memory and energy at fixed accuracy, highlighting MERINDA's suitability for resource-constrained, real-time MCAS.
comment : Published in ECAI 2025, Frontiers in Artificial Intelligence and Applications, volume 413, pages 3904-3911
☆ DialogGuard: Multi- Agent Psychosocial Safety Evaluation of Sensitive LLM Responses
Han Luo, Guy Laban
Large language models (LLMs) now mediate many web-based mental-health, crisis, and other emotionally sensitive services, yet their psychosocial safety in these settings remains poorly understood and weakly evaluated. We present DialogGuard, a multi-agent framework for assessing psychosocial risks in LLM-generated responses along five high-severity dimensions: privacy violations, discriminatory behaviour, mental manipulation, psychological harm, and insulting behaviour. DialogGuard can be applied to diverse generative models through four LLM-as-a-judge pipelines, including single-agent scoring, dual-agent correction, multi-agent debate, and stochastic majority voting, grounded in a shared three-level rubric usable by both human annotators and LLM judges. Using PKU-SafeRLHF with human safety annotations, we show that multi-agent mechanisms detect psychosocial risks more accurately than non-LLM baselines and single-agent judging; dual-agent correction and majority voting provide the best trade-off between accuracy, alignment with human ratings, and robustness, while debate attains higher recall but over-flags borderline cases. We release Dialog-Guard as open-source software with a web interface that provides per-dimension risk scores and explainable natural-language rationales. A formative study with 12 practitioners illustrates how it supports prompt design, auditing, and supervision of web-facing applications for vulnerable users.
☆ Bridging the Gap: Toward Cognitive Autonomy in Artificial Intelligence
Noorbakhsh Amiri Golilarz, Sindhuja Penchala, Shahram Rahimi
Artificial intelligence has advanced rapidly across perception, language, reasoning, and multimodal domains. Yet despite these achievements, modern AI systems remain fundamentally limited in their ability to self-monitor, self-correct, and regulate their behavior autonomously in dynamic contexts. This paper identifies and analyzes seven core deficiencies that constrain contemporary AI models: the absence of intrinsic self-monitoring, lack of meta-cognitive awareness, fixed and non-adaptive learning mechanisms, inability to restructure goals, lack of representational maintenance, insufficient embodied feedback, and the absence of intrinsic agency. Alongside identifying these limitations, we also outline a forward-looking perspective on how AI may evolve beyond them through architectures that mirror neurocognitive principles. We argue that these structural limitations prevent current architectures, including deep learning and transformer-based systems, from achieving robust generalization, lifelong adaptability, and real-world autonomy. Drawing on a comparative analysis of artificial systems and biological cognition [7], and integrating insights from AI research, cognitive science, and neuroscience, we outline how these capabilities are absent in current models and why scaling alone cannot resolve them. We conclude by advocating for a paradigmatic shift toward cognitively grounded AI (cognitive autonomy) capable of self-directed adaptation, dynamic representation management, and intentional, goal-oriented behavior, paired with reformative oversight mechanisms [8] that ensure autonomous systems remain interpretable, governable, and aligned with human values.
☆ Progressive Image Restoration via Text-Conditioned Video Generation
Peng Kang, Xijun Wang, Yu Yuan
Recent text-to-video models have demonstrated strong temporal generation capabilities, yet their potential for image restoration remains underexplored. In this work, we repurpose CogVideo for progressive visual restoration tasks by fine-tuning it to generate restoration trajectories rather than natural video motion. Specifically, we construct synthetic datasets for super-resolution, deblurring, and low-light enhancement, where each sample depicts a gradual transition from degraded to clean frames. Two prompting strategies are compared: a uniform text prompt shared across all samples, and a scene-specific prompting scheme generated via LLaVA multi-modal LLM and refined with ChatGPT. Our fine-tuned model learns to associate temporal progression with restoration quality, producing sequences that improve perceptual metrics such as PSNR, SSIM, and LPIPS across frames. Extensive experiments show that CogVideo effectively restores spatial detail and illumination consistency while maintaining temporal coherence. Moreover, the model generalizes to real-world scenarios on the ReLoBlur dataset without additional training, demonstrating strong zero-shot robustness and interpretability through temporal restoration.
comment : First two authors contributed equally to this work. IEEE ICNC Accepted
☆ Spatiotemporal Pyramid Flow Matching for Climate Emulation
Jeremy Andrew Irvin, Jiaqi Han, Zikui Wang, Abdulaziz Alharbi, Yufei Zhao, Nomin-Erdene Bayarsaikhan, Daniele Visioni, Andrew Y. Ng, Duncan Watson-Parris
Generative models have the potential to transform the way we emulate Earth's changing climate. Previous generative approaches rely on weather-scale autoregression for climate emulation, but this is inherently slow for long climate horizons and has yet to demonstrate stable rollouts under nonstationary forcings. Here, we introduce Spatiotemporal Pyramid Flows (SPF), a new class of flow matching approaches that model data hierarchically across spatial and temporal scales. Inspired by cascaded video models, SPF partitions the generative trajectory into a spatiotemporal pyramid, progressively increasing spatial resolution to reduce computation and coupling each stage with an associated timescale to enable direct sampling at any temporal level in the pyramid. This design, together with conditioning each stage on prescribed physical forcings (e.g., greenhouse gases or aerosols), enables efficient, parallel climate emulation at multiple timescales. On ClimateBench, SPF outperforms strong flow matching baselines and pre-trained models at yearly and monthly timescales while offering fast sampling, especially at coarser temporal levels. To scale SPF, we curate ClimateSuite, the largest collection of Earth system simulations to date, comprising over 33,000 simulation-years across ten climate models and the first dataset to include simulations of climate interventions. We find that the scaled SPF model demonstrates good generalization to held-out scenarios across climate models. Together, SPF and ClimateSuite provide a foundation for accurate, efficient, probabilistic climate emulation across temporal scales and realistic future scenarios. Data and code is publicly available at https://github.com/stanfordmlgroup/spf .
☆ TradeTrap: Are LLM-based Trading Agent s Truly Reliable and Faithful?
Lewen Yan, Jilin Mei, Tianyi Zhou, Lige Huang, Jie Zhang, Dongrui Liu, Jing Shao
LLM-based trading agents are increasingly deployed in real-world financial markets to perform autonomous analysis and execution. However, their reliability and robustness under adversarial or faulty conditions remain largely unexamined, despite operating in high-risk, irreversible financial environments. We propose TradeTrap, a unified evaluation framework for systematically stress-testing both adaptive and procedural autonomous trading agents. TradeTrap targets four core components of autonomous trading agents: market intelligence, strategy formulation, portfolio and ledger handling, and trade execution, and evaluates their robustness under controlled system-level perturbations. All evaluations are conducted in a closed-loop historical backtesting setting on real US equity market data with identical initial conditions, enabling fair and reproducible comparisons across agents and attacks. Extensive experiments show that small perturbations at a single component can propagate through the agent decision loop and induce extreme concentration, runaway exposure, and large portfolio drawdowns across both agent types, demonstrating that current autonomous trading agents can be systematically misled at the system level. Our code is available at https://github.com/Yanlewen/TradeTrap.
☆ DETAIL Matters: Measuring the Impact of Prompt Specificity on Reasoning in Large Language Models
Olivia Kim
Prompt design plays a critical role in the reasoning performance of large language models (LLMs), yet the impact of prompt specificity - how detailed or vague a prompt is - remains understudied. This paper introduces DETAIL, a framework for evaluating LLM performance across varying levels of prompt specificity. We generate multi-level prompts using GPT-4, quantify specificity via perplexity, and assess correctness using GPT-based semantic equivalence. Experiments on 30 novel reasoning tasks across GPT-4 and O3-mini reveal that specificity improves accuracy, especially for smaller models and procedural tasks. Our results highlight the need for adaptive prompting strategies and provide tools and data to support further research.
☆ See, Hear, and Understand: Benchmark ing Audiovisual Human Speech Understanding in Multimodal Large Language Models
Le Thien Phuc Nguyen, Zhuoran Yu, Samuel Low Yu Hang, Subin An, Jeongik Lee, Yohan Ban, SeungEun Chung, Thanh-Huy Nguyen, JuWan Maeng, Soochahn Lee, Yong Jae Lee
Multimodal large language models (MLLMs) are expected to jointly interpret vision, audio, and language, yet existing video benchmarks rarely assess fine-grained reasoning about human speech. Many tasks remain visually solvable or only coarsely evaluate speech, offering limited insight into whether models can align who speaks, what is said, and when it occurs. We introduce AV-SpeakerBench, a curated benchmark of 3,212 multiple-choice questions focused on speaker-centric audiovisual reasoning in real-world videos. It features: (1) a speaker-centered formulation that treats speakers-not scenes-as the core reasoning unit; (2) fusion-grounded question design embedding audiovisual dependencies into question semantics; and (3) expert-curated annotations ensuring temporal precision and cross-modal validity. Comprehensive evaluations show that the Gemini family consistently outperforms open-source systems, with Gemini 2.5 Pro achieving the best results. Among open models, Qwen3-Omni-30B approaches Gemini 2.0 Flash but remains far behind Gemini 2.5 Pro, primarily due to weaker audiovisual fusion rather than visual perception. We believe AV-SpeakerBench establishes a rigorous foundation for advancing fine-grained audiovisual reasoning in future multimodal systems.
comment : preprint
☆ Benchmark ing LLM Agent s for Wealth-Management Workflows
Rory Milsom
Modern work relies on an assortment of digital collaboration tools, yet routine processes continue to suffer from human error and delay. To address this gap, this dissertation extends TheAgentCompany with a finance-focused environment and investigates whether a general purpose LLM agent can complete representative wealth-management tasks both accurately and economically. This study introduces synthetic domain data, enriches colleague simulations, and prototypes an automatic task-generation pipeline. The study aims to create and assess an evaluation set that can meaningfully measure an agent's fitness for assistant-level wealth management work. We construct a benchmark of 12 task-pairs for wealth management assistants spanning retrieval, analysis, and synthesis/communication, with explicit acceptance criteria and deterministic graders. We seeded a set of new finance-specific data and introduced a high vs. low-autonomy variant of every task. The paper concluded that agents are limited less by mathematical reasoning and more so by end-to-end workflow reliability, and meaningfully affected by autonomy level, and that incorrect evaluation of models have hindered benchmarking.
comment : 56 pages, 8 figures, The University of Edinburgh
☆ STRIDE: A Systematic Framework for Selecting AI Modalities -- Agent ic AI, AI Assistants, or LLM Calls NeurIPS 2025
Shubhi Asthana, Bing Zhang, Chad DeLuca, Ruchi Mahindru, Hima Patel
The rapid shift from stateless large language models (LLMs) to autonomous, goal-driven agents raises a central question: When is agentic AI truly necessary? While agents enable multi-step reasoning, persistent memory, and tool orchestration, deploying them indiscriminately leads to higher cost, complexity, and risk. We present STRIDE (Systematic Task Reasoning Intelligence Deployment Evaluator), a framework that provides principled recommendations for selecting between three modalities: (i) direct LLM calls, (ii) guided AI assistants, and (iii) fully autonomous agentic AI. STRIDE integrates structured task decomposition, dynamism attribution, and self-reflection requirement analysis to produce an Agentic Suitability Score, ensuring that full agentic autonomy is reserved for tasks with inherent dynamism or evolving context. Evaluated across 30 real-world tasks spanning SRE, compliance, and enterprise automation, STRIDE achieved 92% accuracy in modality selection, reduced unnecessary agent deployments by 45%, and cut resource costs by 37%. Expert validation over six months in SRE and compliance domains confirmed its practical utility, with domain specialists agreeing that STRIDE effectively distinguishes between tasks requiring simple LLM calls, guided assistants, or full agentic autonomy. This work reframes agent adoption as a necessity-driven design decision, ensuring autonomy is applied only when its benefits justify the costs.
comment : 10 pages, 4 Figures, 5 Tables Paper presented at NeurIPS 2025 LAW workshop: Bridging Language, Agent, and World Models
☆ Orchestration Framework for Financial Agent s: From Algorithmic Trading to Agent ic Trading NeurIPS 2025
Jifeng Li, Arnav Grover, Abraham Alpuerto, Yupeng Cao, Xiao-Yang Liu
The financial market is a mission-critical playground for AI agents due to its temporal dynamics and low signal-to-noise ratio. Building an effective algorithmic trading system may require a professional team to develop and test over the years. In this paper, we propose an orchestration framework for financial agents, which aims to democratize financial intelligence to the general public. We map each component of the traditional algorithmic trading system to agents, including planner, orchestrator, alpha agents, risk agents, portfolio agents, backtest agents, execution agents, audit agents, and memory agent. We present two in-house trading examples. For the stock trading task (hourly data from 04/2024 to 12/2024), our approach achieved a return of $20.42\%$, a Sharpe ratio of 2.63, and a maximum drawdown of $-3.59\%$, while the S&P 500 index yielded a return of $15.97\%$. For the BTC trading task (minute data from 27/07/2025 to 13/08/2025), our approach achieved a return of $8.39\%$, a Sharpe ratio of $0.38$, and a maximum drawdown of $-2.80\%$, whereas the BTC price increased by $3.80\%$. Our code is available on \href{https://github.com/Open-Finance-Lab/AgenticTrading}{GitHub}.
comment : Accepted at the Workshop on Generative AI in Finance, 39th Conference on Neural Information Processing Systems (NeurIPS 2025)
☆ Improved Training Mechanism for Reinforcement Learning via Online Model Selection
Aida Afshar, Aldo Pacchiano
We study the problem of online model selection in reinforcement learning, where the selector has access to a class of reinforcement learning agents and learns to adaptively select the agent with the right configuration. Our goal is to establish the improved efficiency and performance gains achieved by integrating online model selection methods into reinforcement learning training procedures. We examine the theoretical characterizations that are effective for identifying the right configuration in practice, and address three practical criteria from a theoretical perspective: 1) Efficient resource allocation, 2) Adaptation under non-stationary dynamics, and 3) Training stability across different seeds. Our theoretical results are accompanied by empirical evidence from various model selection tasks in reinforcement learning, including neural architecture selection, step-size selection, and self model selection.
☆ Multifractal Recalibration of Neural Networks for Medical Imaging Segmentation
Miguel L. Martins, Miguel T. Coimbra, Francesco Renna
Multifractal analysis has revealed regularities in many self-seeding phenomena, yet its use in modern deep learning remains limited. Existing end-to-end multifractal methods rely on heavy pooling or strong feature-space decimation, which constrain tasks such as semantic segmentation. Motivated by these limitations, we introduce two inductive priors: Monofractal and Multifractal Recalibration. These methods leverage relationships between the probability mass of the exponents and the multifractal spectrum to form statistical descriptions of encoder embeddings, implemented as channel-attention functions in convolutional networks. Using a U-Net-based framework, we show that multifractal recalibration yields substantial gains over a baseline equipped with other channel-attention mechanisms that also use higher-order statistics. Given the proven ability of multifractal analysis to capture pathological regularities, we validate our approach on three public medical-imaging datasets: ISIC18 (dermoscopy), Kvasir-SEG (endoscopy), and BUSI (ultrasound). Our empirical analysis also provides insights into the behavior of these attention layers. We find that excitation responses do not become increasingly specialized with encoder depth in U-Net architectures due to skip connections, and that their effectiveness may relate to global statistics of instance variability.
comment : 30 pages, 9 figures, journal paper
☆ Bin2Vec: Interpretable and Auditable Multi-View Binary Analysis for Code Plagiarism Detection
Moussa Moussaoui, Tarik Houichime, Abdelalim Sadiq
We introduce Bin2Vec, a new framework that helps compare software programs in a clear and explainable way. Instead of focusing only on one type of information, Bin2Vec combines what a program looks like (its built-in functions, imports, and exports) with how it behaves when it runs (its instructions and memory usage). This gives a more complete picture when deciding whether two programs are similar or not. Bin2Vec represents these different types of information as views that can be inspected separately using easy-to-read charts, and then brings them together into an overall similarity score. Bin2Vec acts as a bridge between binary representations and machine learning techniques by generating feature representations that can be efficiently processed by machine-learning models. We tested Bin2Vec on multiple versions of two well-known Windows programs, PuTTY and 7-Zip. The primary results strongly confirmed that our method compute an optimal and visualization-friendly representation of the analyzed software. For example, PuTTY versions showed more complex behavior and memory activity, while 7-Zip versions focused more on performance-related patterns. Overall, Bin2Vec provides decisions that are both reliable and explainable to humans. Because it is modular and easy to extend, it can be applied to tasks like auditing, verifying software origins, or quickly screening large numbers of programs in cybersecurity and reverse-engineering work.
☆ A Knowledge-Based Language Model: Deducing Grammatical Knowledge in a Multi- Agent Language Acquisition Simulation
David Ph. Shakouri, Crit Cremers, Niels O. Schiller
This paper presents an initial study performed by the MODOMA system. The MODOMA is a computational multi-agent laboratory environment for unsupervised language acquisition experiments such that acquisition is based on the interaction between two language models, an adult and a child agent. Although this framework employs statistical as well as rule-based procedures, the result of language acquisition is a knowledge-based language model, which can be used to generate and parse new utterances of the target language. This system is fully parametrized and researchers can control all aspects of the experiments while the results of language acquisition, that is, the acquired grammatical knowledge, are explicitly represented and can be consulted. Thus, this system introduces novel possibilities for conducting computational language acquisition experiments. The experiments presented by this paper demonstrate that functional and content categories can be acquired and represented by the daughter agent based on training and test data containing different amounts of exemplars generated by the adult agent. Interestingly, similar patterns, which are well-established for human-generated data, are also found for these machine-generated data. As the procedures resulted in the successful acquisition of discrete grammatical categories by the child agent, these experiments substantiate the validity of the MODOMA approach to modelling language acquisition.
comment : 23 pages, 7 figures, 11 tables. Related work: arXiv:2503.18702. This is the peer-reviewed publisher's version, downloadable from: https://www.clinjournal.org/clinj/article/view/193
☆ Enforcing Orderedness to Improve Feature Consistency
Sophie L. Wang, Alex Quach, Nithin Parsan, John J. Yang
Sparse autoencoders (SAEs) have been widely used for interpretability of neural networks, but their learned features often vary across seeds and hyperparameter settings. We introduce Ordered Sparse Autoencoders (OSAE), which extend Matryoshka SAEs by (1) establishing a strict ordering of latent features and (2) deterministically using every feature dimension, avoiding the sampling-based approximations of prior nested SAE methods. Theoretically, we show that OSAEs resolve permutation non-identifiability in settings of sparse dictionary learning where solutions are unique (up to natural symmetries). Empirically on Gemma2-2B and Pythia-70M, we show that OSAEs can help improve consistency compared to Matryoshka baselines.
☆ From monoliths to modules: Decomposing transducers for efficient wo rl d modelling
Alexander Boyd, Franz Nowak, David Hyland, Manuel Baltieri, Fernando E. Rosas
World models have been recently proposed as sandbox environments in which AI agents can be trained and evaluated before deployment. Although realistic world models often have high computational demands, efficient modelling is usually possible by exploiting the fact that real-world scenarios tend to involve subcomponents that interact in a modular manner. In this paper, we explore this idea by developing a framework for decomposing complex world models represented by transducers, a class of models generalising POMDPs. Whereas the composition of transducers is well understood, our results clarify how to invert this process, deriving sub-transducers operating on distinct input-output subspaces, enabling parallelizable and interpretable alternatives to monolithic world modelling that can support distributed inference. Overall, these results lay a groundwork for bridging the structural transparency demanded by AI safety and the computational efficiency required for real-world inference.
☆ Story2MIDI: Emotionally Aligned Music Generation from Text
Mohammad Shokri, Alexandra C. Salem, Gabriel Levine, Johanna Devaney, Sarah Ita Levitan
In this paper, we introduce Story2MIDI, a sequence-to-sequence Transformer-based model for generating emotion-aligned music from a given piece of text. To develop this model, we construct the Story2MIDI dataset by merging existing datasets for sentiment analysis from text and emotion classification in music. The resulting dataset contains pairs of text blurbs and music pieces that evoke the same emotions in the reader or listener. Despite the small scale of our dataset and limited computational resources, our results indicate that our model effectively learns emotion-relevant features in music and incorporates them into its generation process, producing samples with diverse emotional responses. We evaluate the generated outputs using objective musical metrics and a human listening study, confirming the model's ability to capture intended emotional cues.
comment : 8 pages (6 pages of main text + 2 pages of references and appendices), 4 figures, 1 table. Presented at IEEE Big Data 2025 3rd Workshop on AI Music Generation (AIMG 2025)
☆ Think Before You Prune: Self-Reflective Structured Pruning for Reasoning Language Models
Ziyan Wang, Enmao Diao, Qi Le, Pu Wang, Guanchu Wang, Minwoo Lee, Shu-ping Yeh, Li Yang
Reasoning LLMs (RLMs) such as OpenAI o1, DeepSeek-R1, and Qwen3 deliver strong multi-step reasoning through chain-of-thought generation, but their large model sizes and lengthy decode-time outputs make them costly to deploy and unsuitable for resource-constrained settings. To reduce computing and memory cost, pruning offers a promising solution by removing unimportant parameters. However, despite their success on standard LLMs, existing pruning methods severely damage RLMs, as even moderate sparsity (e.g., 20%) can collapse accuracy and completely disrupt the model's reasoning coherence. We begin by analyzing why existing pruning pipelines fail on reasoning LLMs and find that their brittleness largely stems from a mismatch between the calibration data, the pruning objective, and the model's decode-time reasoning behavior. Our study further shows that the most reliable calibration signal comes not from human-written labels but from the model's own self-generated reasoning traces, which more accurately reflect its inference distribution. Guided by these insights, we introduce RESP, a self-reflective structured pruning framework that aligns pruning decisions with the model's reasoning dynamics through self-generated calibration, decode-only gradient-based importance estimation, and progressive regeneration that maintains calibration fidelity as sparsity increases. Experiments on Qwen3-8B demonstrate that RESP markedly outperforms existing structured pruning methods on both GSM8K and MathQA, preserving near-dense accuracy at 20-30% sparsity and substantially mitigating performance collapse at higher sparsity levels. At 40% sparsity, RESP attains 81.3% accuracy on GSM8K and 59.6% on MathQA, surpassing the strongest baselines by 66.87% and 47%, respectively.
comment : 7 pages, 3 figures
☆ CLEF: Clinically-Guided Contrastive Learning for Electrocardiogram Foundation Models
Yuxuan Shu, Peter H. Charlton, Fahim Kawsar, Jussi Hernesniemi, Mohammad Malekzadeh
The electrocardiogram (ECG) is a key diagnostic tool in cardiovascular health. Single-lead ECG recording is integrated into both clinical-grade and consumer wearables. While self-supervised pretraining of foundation models on unlabeled ECGs improves diagnostic performance, existing approaches do not incorporate domain knowledge from clinical metadata. We introduce a novel contrastive learning approach that utilizes an established clinical risk score to adaptively weight negative pairs: clinically-guided contrastive learning. It aligns the similarities of ECG embeddings with clinically meaningful differences between subjects, with an explicit mechanism to handle missing metadata. On 12-lead ECGs from 161K patients in the MIMIC-IV dataset, we pretrain single-lead ECG foundation models at three scales, collectively called CLEF, using only routinely collected metadata without requiring per-sample ECG annotations. We evaluate CLEF on 18 clinical classification and regression tasks across 7 held-out datasets, and benchmark against 5 foundation model baselines and 3 self-supervised algorithms. When pretrained on 12-lead ECG data and tested on lead-I data, CLEF outperforms self-supervised foundation model baselines: the medium-sized CLEF achieves average AUROC improvements of at least 2.6% in classification and average reductions in MAEs of at least 3.2% in regression. Comparing with existing self-supervised learning algorithms, CLEF improves the average AUROC by at least 1.8%. Moreover, when pretrained only on lead-I data for classification tasks, CLEF performs comparably to the state-of-the-art ECGFounder, which was trained in a supervised manner. Overall, CLEF enables more accurate and scalable single-lead ECG analysis, advancing remote health monitoring. Code and pretrained CLEF models are available at: github.com/Nokia-Bell-Labs/ecg-foundation-model.
comment : The code is available at https://github.com/Nokia-Bell-Labs/ecg-foundation-model
☆ Young Children's Anthropomorphism of AI Chatbots and the Role of Parent Co-Presence
Pilyoung Kim, Jenna H. Chin, Yun Xie, Nolan Brady, Tom Yeh, Sujin Yang
Artificial Intelligence (AI) chatbots powered by a large language model (LLM) are entering young children's learning and play, yet little is known about how young children construe these agents or how such construals relate to engagement. We examined anthropomorphism of a social AI chatbot during collaborative storytelling and asked how children's attributions related to their behavior and prefrontal activation. Children at ages 5-6 (N = 23) completed three storytelling sessions: interacting with (1) an AI chatbot only, (2) a parent only, and (3) the AI and a parent together. After the sessions, children completed an interview assessing anthropomorphism toward both the AI chatbot and the parent. Behavioral engagement was indexed by the conversational turn count (CTC) ratio, and concurrent fNIRS measured oxygenated hemoglobin in bilateral vmPFC and dmPFC regions. Children reported higher anthropomorphism for parents than for the AI chatbot overall, although AI ratings were relatively high for perceptive abilities and epistemic states. Anthropomorphism was not associated with CTC. In the right dmPFC, higher perceptive scores were associated with greater activation during the AI-only condition and with lower activation during the AI+Parent condition. Exploratory analyses indicated that higher dmPFC activation during the AI-only condition correlated with higher end-of-session "scared" mood ratings. Findings suggest that stronger perceptive anthropomorphism can be associated with greater brain activation related to interpreting the AI's mental states, whereas parent co-presence may help some children interpret and regulate novel AI interactions. These results may have design implications for encouraging parent-AI co-use in early childhood.
☆ Flowchart2Mermaid: A Vision -Language Model Powered System for Converting Flowcharts into Editable Diagram Code EACL 2026
Pritam Deka, Barry Devereux
Flowcharts are common tools for communicating processes but are often shared as static images that cannot be easily edited or reused. We present \textsc{Flowchart2Mermaid}, a lightweight web system that converts flowchart images into editable Mermaid.js code which is a markup language for visual workflows, using a detailed system prompt and vision-language models. The interface supports mixed-initiative refinement through inline text editing, drag-and-drop node insertion, and natural-language commands interpreted by an integrated AI assistant. Unlike prior image-to-diagram tools, our approach produces a structured, version-controllable textual representation that remains synchronized with the rendered diagram. We further introduce evaluation metrics to assess structural accuracy, flow correctness, syntax validity, and completeness across multiple models.
comment : Submitted to EACL 2026 Demo Track
☆ Feature Selection Empowered BERT for Detection of Hate Speech with Vocabulary Augmentation
Pritish N. Desai, Tanay Kewalramani, Srimanta Mandal
Abusive speech on social media poses a persistent and evolving challenge, driven by the continuous emergence of novel slang and obfuscated terms designed to circumvent detection systems. In this work, we present a data efficient strategy for fine tuning BERT on hate speech classification by significantly reducing training set size without compromising performance. Our approach employs a TF IDF-based sample selection mechanism to retain only the most informative 75 percent of examples, thereby minimizing training overhead. To address the limitations of BERT's native vocabulary in capturing evolving hate speech terminology, we augment the tokenizer with domain-specific slang and lexical variants commonly found in abusive contexts. Experimental results on a widely used hate speech dataset demonstrate that our method achieves competitive performance while improving computational efficiency, highlighting its potential for scalable and adaptive abusive content moderation.
☆ Comparing Baseline and Day-1 Diffusion MRI Using Multimodal Deep Embeddings for Stroke Outcome Prediction
Sina Raeisadigh, Myles Joshua Toledo Tan, Henning Müller, Abderrahmane Hedjoudje
This study compares baseline (J0) and 24-hour (J1) diffusion magnetic resonance imaging (MRI) for predicting three-month functional outcomes after acute ischemic stroke (AIS). Seventy-four AIS patients with paired apparent diffusion coefficient (ADC) scans and clinical data were analyzed. Three-dimensional ResNet-50 embeddings were fused with structured clinical variables, reduced via principal component analysis (<=12 components), and classified using linear support vector machines with eight-fold stratified group cross-validation. J1 multimodal models achieved the highest predictive performance (AUC = 0.923 +/- 0.085), outperforming J0-based configurations (AUC <= 0.86). Incorporating lesion-volume features further improved model stability and interpretability. These findings demonstrate that early post-treatment diffusion MRI provides superior prognostic value to pre-treatment imaging and that combining MRI, clinical, and lesion-volume features produces a robust and interpretable framework for predicting three-month functional outcomes in AIS patients.
comment : 5 pages, 5 figures, 2 tables
☆ Do Large Language Models Walk Their Talk? Measuring the Gap Between Implicit Associations, Self-Report, and Behavioral Altruism
Sandro Andric
We investigate whether Large Language Models (LLMs) exhibit altruistic tendencies, and critically, whether their implicit associations and self-reports predict actual altruistic behavior. Using a multi-method approach inspired by human social psychology, we tested 24 frontier LLMs across three paradigms: (1) an Implicit Association Test (IAT) measuring implicit altruism bias, (2) a forced binary choice task measuring behavioral altruism, and (3) a self-assessment scale measuring explicit altruism beliefs. Our key findings are: (1) All models show strong implicit pro-altruism bias (mean IAT = 0.87, p < .0001), confirming models "know" altruism is good. (2) Models behave more altruistically than chance (65.6% vs. 50%, p < .0001), but with substantial variation (48-85%). (3) Implicit associations do not predict behavior (r = .22, p = .29). (4) Most critically, models systematically overestimate their own altruism, claiming 77.5% altruism while acting at 65.6% (p < .0001, Cohen's d = 1.08). This "virtue signaling gap" affects 75% of models tested. Based on these findings, we recommend the Calibration Gap (the discrepancy between self-reported and behavioral values) as a standardized alignment metric. Well-calibrated models are more predictable and behaviorally consistent; only 12.5% of models achieve the ideal combination of high prosocial behavior and accurate self-knowledge.
comment : 14 pages, 7 figures, 7 tables. Code and data available at https://github.com/sandroandric/LLMs_Altruism_Study_Code
☆ Public Sentiment Analysis of Traffic Management Policies in Knoxville: A Social Media Driven Study
Shampa Saha, Shovan Roy
This study presents a comprehensive analysis of public sentiment toward traffic management policies in Knoxville, Tennessee, utilizing social media data from Twitter and Reddit platforms. We collected and analyzed 7906 posts spanning January 2022 to December 2023, employing Valence Aware Dictionary and sEntiment Reasoner (VADER) for sentiment analysis and Latent Dirichlet Allocation (LDA) for topic modeling. Our findings reveal predominantly negative sentiment, with significant variations across platforms and topics. Twitter exhibited more negative sentiment compared to Reddit. Topic modeling identified six distinct themes, with construction-related topics showing the most negative sentiment while general traffic discussions were more positive. Spatiotemporal analysis revealed geographic and temporal patterns in sentiment expression. The research demonstrates social media's potential as a real-time public sentiment monitoring tool for transportation planning and policy evaluation.
☆ Dynamic Correction of Erroneous State Estimates via Diffusion Bayesian Exploration
Yiwei Shi, Hongnan Ma, Mengyue Yang, Cunjia Liu, Weiru Liu
In emergency response and other high-stakes societal applications, early-stage state estimates critically shape downstream outcomes. Yet, these initial state estimates-often based on limited or biased information-can be severely misaligned with reality, constraining subsequent actions and potentially causing catastrophic delays, resource misallocation, and human harm. Under the stationary bootstrap baseline (zero transition and no rejuvenation), bootstrap particle filters exhibit Stationarity-Induced Posterior Support Invariance (S-PSI), wherein regions excluded by the initial prior remain permanently unexplorable, making corrections impossible even when new evidence contradicts current beliefs. While classical perturbations can in principle break this lock-in, they operate in an always-on fashion and may be inefficient. To overcome this, we propose a diffusion-driven Bayesian exploration framework that enables principled, real-time correction of early state estimation errors. Our method expands posterior support via entropy-regularized sampling and covariance-scaled diffusion. A Metropolis-Hastings check validates proposals and keeps inference adaptive to unexpected evidence. Empirical evaluations on realistic hazardous-gas localization tasks show that our approach matches reinforcement learning and planning baselines when priors are correct. It substantially outperforms classical SMC perturbations and RL-based methods under misalignment, and we provide theoretical guarantees that DEPF resolves S-PSI while maintaining statistical rigor.
☆ ALARM: Automated MLLM-Based Anomaly Detection in Complex-EnviRonment Monitoring with Uncertainty Quantification
Congjing Zhang, Feng Lin, Xinyi Zhao, Pei Guo, Wei Li, Lin Chen, Chaoyue Zhao, Shuai Huang
The advance of Large Language Models (LLMs) has greatly stimulated research interest in developing multi-modal LLM (MLLM)-based visual anomaly detection (VAD) algorithms that can be deployed in complex environments. The challenge is that in these complex environments, the anomalies are sometimes highly contextual and also ambiguous, and thereby, uncertainty quantification (UQ) is a crucial capacity for an MLLM-based VAD system to succeed. In this paper, we introduce our UQ-supported MLLM-based VAD framework called ALARM. ALARM integrates UQ with quality-assurance techniques like reasoning chain, self-reflection, and MLLM ensemble for robust and accurate performance and is designed based on a rigorous probabilistic inference pipeline and computational process. Extensive empirical evaluations are conducted using the real-world smart-home benchmark data and wound image classification data, which shows ALARM's superior performance and its generic applicability across different domains for reliable decision-making.
☆ Ensemble Privacy Defense for Knowledge-Intensive LLMs against Membership Inference Attacks
Haowei Fu, Bo Ni, Han Xu, Kunpeng Liu, Dan Lin, Tyler Derr
Retrieval-Augmented Generation (RAG) and Supervised Finetuning (SFT) have become the predominant paradigms for equipping Large Language Models (LLMs) with external knowledge for diverse, knowledge-intensive tasks. However, while such knowledge injection improves performance, it also exposes new attack surfaces. Membership Inference Attacks (MIAs), which aim to determine whether a given data sample was included in a model's training set, pose serious threats to privacy and trust in sensitive domains. To this end, we first systematically evaluate the vulnerability of RAG- and SFT-based LLMs to various MIAs. Then, to address the privacy risk, we further introduce a novel, model-agnostic defense framework, Ensemble Privacy Defense (EPD), which aggregates and evaluates the outputs of a knowledge-injected LLM, a base LLM, and a dedicated judge model to enhance resistance against MIAs. Comprehensive experiments show that, on average, EPD reduces MIA success by up to 27.8\% for SFT and 526.3\% for RAG compared to inference-time baseline, while maintaining answer quality.
☆ QGShap: Quantum Acceleration for Faithful GNN Explanations AAAI 2026
Haribandhu Jena, Jyotirmaya Shivottam, Subhankar Mishra
Graph Neural Networks (GNNs) have become indispensable in critical domains such as drug discovery, social network analysis, and recommendation systems, yet their black-box nature hinders deployment in scenarios requiring transparency and accountability. While Shapley value-based methods offer mathematically principled explanations by quantifying each component's contribution to predictions, computing exact values requires evaluating $2^n$ coalitions (or aggregating over $n!$ permutations), which is intractable for real-world graphs. Existing approximation strategies sacrifice either fidelity or efficiency, limiting their practical utility. We introduce QGShap, a quantum computing approach that leverages amplitude amplification to achieve quadratic speedups in coalition evaluation while maintaining exact Shapley computation. Unlike classical sampling or surrogate methods, our approach provides fully faithful explanations without approximation trade-offs for tractable graph sizes. We conduct empirical evaluations on synthetic graph datasets, demonstrating that QGShap achieves consistently high fidelity and explanation accuracy, matching or exceeding the performance of classical methods across all evaluation metrics. These results collectively demonstrate that QGShap not only preserves exact Shapley faithfulness but also delivers interpretable, stable, and structurally consistent explanations that align with the underlying graph reasoning of GNNs. The implementation of QGShap is available at https://github.com/smlab-niser/qgshap.
comment : Accepted in the QC+AI Workshop at AAAI 2026
♻ ☆ The AI Productivity Index (APEX)
Bertie Vidgen, Abby Fennelly, Evan Pinnix, Julien Bencheck, Daniyal Khan, Zach Richards, Austin Bridges, Calix Huang, Ben Hunsberger, Isaac Robinson, Akul Datta, Chirag Mahapatra, Dominic Barton, Cass R. Sunstein, Eric Topol, Brendan Foody, Osvald Nitski
We present an extended version of the AI Productivity Index (APEX-v1-extended), a benchmark for assessing whether frontier models are capable of performing economically valuable tasks in four jobs: investment banking associate, management consultant, big law associate, and primary care physician (MD). This technical report details the extensions to APEX-v1, including an increase in the held-out evaluation set from n = 50 to n = 100 cases per job (n = 400 total) and updates to the grading methodology. We present a new leaderboard, where GPT5 (Thinking = High) remains the top performing model with a score of 67.0%. APEX-v1-extended shows that frontier models still have substantial limitations when performing typical professional tasks. To support further research, we are open sourcing n = 25 non-benchmark example cases per role (n = 100 total) along with our evaluation harness.
♻ ☆ The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason
Shanchao Liang, Spandan Garg, Roshanak Zilouchian Moghaddam
As large language models (LLMs) become increasingly capable and widely adopted, benchmarks play a central role in assessing their practical utility. For example, SWE-Bench Verified has emerged as a critical benchmark for evaluating LLMs' software engineering abilities, particularly their aptitude for resolving real-world GitHub issues. Recent LLMs show impressive performance on SWE-Bench, leading to optimism about their capacity for complex coding tasks. However, current evaluation protocols may overstate these models' true capabilities. It is crucial to distinguish LLMs' generalizable problem-solving ability and other learned artifacts. In this work, we introduce two diagnostic tasks: file path identification from issue descriptions alone and ground truth function reproduction with only the current file context and issue description to probe models' underlying knowledge. We present empirical evidence that performance gains on SWE-Bench-Verified may be partially driven by memorization rather than genuine problem-solving. We show that state-of-the-art models achieve up to 76% accuracy in identifying buggy file paths using only issue descriptions, without access to repository structure. This performance is merely up to 53% on tasks from repositories not included in SWE-Bench, pointing to possible data contamination or memorization. Similar patterns are also observed for the function reproduction task, where the verbatim similarity is much higher on SWE-Bench Verified than on other similar coding benchmarks (up to 35% consecutive 5-gram accuracy on SWE-Bench Verified and Full, but only up to 18% for tasks in other benchmarks). These findings raise concerns about the validity of existing results and underscore the need for more robust, contamination-resistant benchmarks to reliably evaluate LLMs' coding abilities.
♻ ☆ SCOPE-MRI: Bankart Lesion Detection as a Case Study in Data Curation and Deep Learning for Challenging Diagnoses
Sahil Sethi, Sai Reddy, Mansi Sakarvadia, Jordan Serotte, Darlington Nwaudo, Nicholas Maassen, Lewis Shi
Deep learning has shown strong performance in musculoskeletal imaging, but prior work has largely targeted conditions where diagnosis is relatively straightforward. More challenging problems remain underexplored, such as detecting Bankart lesions (anterior-inferior glenoid labral tears) on standard MRIs. These lesions are difficult to diagnose due to subtle imaging features, often necessitating invasive MRI arthrograms (MRAs). We introduce ScopeMRI, the first publicly available, expert-annotated dataset for shoulder pathologies, and present a deep learning framework for Bankart lesion detection on both standard MRIs and MRAs. ScopeMRI contains shoulder MRIs from patients who underwent arthroscopy, providing ground-truth labels from intraoperative findings, the diagnostic gold standard. Separate models were trained for MRIs and MRAs using CNN- and transformer-based architectures, with predictions ensembled across multiple imaging planes. Our models achieved radiologist-level performance, with accuracy on standard MRIs surpassing radiologists interpreting MRAs. External validation on independent hospital data demonstrated initial generalizability across imaging protocols. By releasing ScopeMRI and a modular codebase for training and evaluation, we aim to accelerate research in musculoskeletal imaging and foster development of datasets and models that address clinically challenging diagnostic tasks.
comment : This version of the article has been accepted for publication at Nature Partner Journal (NPJ) Artificial Intelligence, but is not the Version of Record and does not reflect post-acceptance improvements or any corrections. The Version of Record is available online at: http://dx.doi.org/10.1038/s44387-025-00043-5
♻ ☆ Meta- Reinforcement Learning for Building Energy Management System
Benoit Boulet Huiliang Zhang, Di Wu, Arnaud Zinflou
The building sector is one of the largest contributors to global energy consumption. Improving its energy efficiency is essential for reducing operational costs and greenhouse gas emissions. Energy management systems (EMS) play a key role in monitoring and controlling building appliances efficiently and reliably. With the increasing integration of renewable energy, intelligent EMS solutions have received growing attention. Reinforcement learning (RL) has recently been explored for this purpose and shows strong potential. However, most RL-based EMS methods require a large number of training steps to learn effective control policies, especially when adapting to unseen buildings, which limits their practical deployment. This paper introduces MetaEMS, a meta-reinforcement learning framework for EMS. MetaEMS improves learning efficiency by transferring knowledge from previously solved tasks to new ones through group-level and building-level adaptation, enabling fast adaptation and effective control across diverse building environments. Experimental results demonstrate that MetaEMS adapts more rapidly to unseen buildings and consistently outperforms baseline methods across various scenarios.
comment : arXiv admin note: text overlap with arXiv:1909.10165 by other authors
♻ ☆ Securing the Skies: A Comprehensive Survey on Anti-UAV Methods, Benchmark ing, and Future Directions CVPR
Yifei Dong, Fengyi Wu, Sanjian Zhang, Guangyu Chen, Yuzhi Hu, Masumi Yano, Jingdong Sun, Siyu Huang, Feng Liu, Qi Dai, Zhi-Qi Cheng
Unmanned Aerial Vehicles (UAVs) are indispensable for infrastructure inspection, surveillance, and related tasks, yet they also introduce critical security challenges. This survey provides a wide-ranging examination of the anti-UAV domain, centering on three core objectives-classification, detection, and tracking-while detailing emerging methodologies such as diffusion-based data synthesis, multi-modal fusion, vision-language modeling, self-supervised learning, and reinforcement learning. We systematically evaluate state-of-the-art solutions across both single-modality and multi-sensor pipelines (spanning RGB, infrared, audio, radar, and RF) and discuss large-scale as well as adversarially oriented benchmarks. Our analysis reveals persistent gaps in real-time performance, stealth detection, and swarm-based scenarios, underscoring pressing needs for robust, adaptive anti-UAV systems. By highlighting open research directions, we aim to foster innovation and guide the development of next-generation defense strategies in an era marked by the extensive use of UAVs.
comment : Best Paper, Accepted at CVPR Workshop Anti-UAV 2025. 16 pages
♻ ☆ SpikingBrain: Spiking Brain-inspired Large Models
Yuqi Pan, Yupeng Feng, Jinghao Zhuang, Siyu Ding, Han Xu, Zehao Liu, Bohan Sun, Yuhong Chou, Xuerui Qiu, Anlin Deng, Anjie Hu, Shurong Wang, Peng Zhou, Man Yao, Jibin Wu, Jian Yang, Guoliang Sun, Bo Xu, Guoqi Li
Mainstream Transformer-based large language models face major efficiency bottlenecks: training computation scales quadratically with sequence length, and inference memory grows linearly, limiting long-context processing. Building large models on non-NVIDIA platforms also poses challenges for stable and efficient training. To address this, we introduce SpikingBrain, a family of brain-inspired models designed for efficient long-context training and inference. SpikingBrain leverages the MetaX GPU cluster and focuses on three aspects: (1) Model Architecture: linear and hybrid-linear attention architectures with adaptive spiking neurons; (2) Algorithmic Optimizations: an efficient, conversion-based training pipeline and a dedicated spike coding framework; (3) System Engineering: customized training frameworks, operator libraries, and parallelism strategies tailored to MetaX hardware. Using these techniques, we develop two models: SpikingBrain-7B, a linear LLM, and SpikingBrain-76B, a hybrid-linear MoE LLM. These models demonstrate the feasibility of large-scale LLM development on non-NVIDIA platforms, and training remains stable for weeks on hundreds of MetaX GPUs with Model FLOPs Utilization at expected levels. SpikingBrain achieves performance comparable to open-source Transformer baselines while using only about 150B tokens for continual pre-training. Our models also significantly improve long-context efficiency and deliver inference with (partially) constant memory and event-driven spiking behavior. For example, SpikingBrain-7B attains over 100x speedup in Time to First Token for 4M-token sequences. Furthermore, the proposed spiking scheme achieves 69.15 percent sparsity, enabling low-power operation. Overall, this work demonstrates the potential of brain-inspired mechanisms to drive the next generation of efficient and scalable large model design.
♻ ☆ NeuroRVQ: Multi-Scale EEG Tokenization for Generative Large Brainwave Models
Konstantinos Barmpas, Na Lee, Alexandros Koliousis, Yannis Panagakis, Dimitrios A. Adamos, Nikolaos Laskaris, Stefanos Zafeiriou
Electroencephalography (EEG) captures neural activity across multiple temporal and spectral scales, yielding signals that are rich but complex for representation learning. Recently, EEG foundation models trained to predict masked signal-tokens have shown promise for learning generalizable representations. However, their performance is hindered by their signal tokenization modules. Existing neural tokenizers fail to preserve high-frequency dynamics, limiting their ability to reconstruct EEG signals with high fidelity. We introduce NeuroRVQ, a scalable Large Brainwave Model (LBM) centered on a codebook-based tokenizer. Our tokenizer integrates: (i) multi-scale feature extraction modules that capture the full frequency neural spectrum; (ii) hierarchical residual vector quantization (RVQ) codebooks for high-resolution encoding; and, (iii) an EEG signal phase- and amplitude-aware loss function for efficient training. This design enables efficient EEG compression while supporting accurate reconstruction across all frequency bands, leading to robust generative masked modeling. Our empirical results demonstrate that NeuroRVQ achieves lower reconstruction error and outperforms existing LBMs on a variety of downstream tasks. More broadly, NeuroRVQ tokenizer establishes a strong prior for codebook-based general-purpose brainwave models, enabling advances in neural decoding, generative modeling and multimodal biosignal integration.
♻ ☆ Outcome-based Reinforcement Learning to Predict the Future
Benjamin Turtel, Danny Franklin, Kris Skotheim, Luke Hewitt, Philipp Schoenegger
Reinforcement Learning with Verifiable Rewards (RLVR) has been an effective approach for improving Large Language Models' reasoning in domains such as coding and mathematics. Here, we apply RLVR methods towards forecasting future real-world events - a challenging task for RL due to the very noisy (and delayed) outcomes involved. Using a novel dataset of recent questions from a prediction market, and accompanying relevant news headlines, we show that a compact (14B) reasoning model can be trained to match or surpass the predictive accuracy of frontier models like o1, while greatly improving probabilistic calibration. The model's performance is also practically meaningful: in a Polymarket trading simulation, we estimate that its bets would have yielded a return on investment of over 10% across all questions in the test set. We detail and compare approaches used in training our model, including augmenting our training-data with synthetic prediction questions, guardrails for learning stability, and median prediction sampling at inference-time.
♻ ☆ Med-gte-hybrid: A context ual embedding transformer model for extracting actionable information from clinical texts
Aditya Kumar, Simon Rauch, Mario Cypko, Oliver Amft
We introduce a novel contextual embedding model med-gte-hybrid that was derived from the gte-large sentence transformer to extract information from unstructured clinical narratives. Our model tuning strategy for med-gte-hybrid combines contrastive learning and a denoising autoencoder. To evaluate the performance of med-gte-hybrid, we investigate several clinical prediction tasks in large patient cohorts extracted from the MIMIC-IV dataset, including Chronic Kidney Disease (CKD) patient prognosis, estimated glomerular filtration rate (eGFR) prediction, and patient mortality prediction. Furthermore, we demonstrate that the med-gte-hybrid model improves patient stratification, clustering, and text retrieval, thus outperforms current state-of-the-art models on the Massive Text Embedding Benchmark (MTEB). While some of our evaluations focus on CKD, our hybrid tuning of sentence transformers could be transferred to other medical domains and has the potential to improve clinical decision-making and personalised treatment pathways in various healthcare applications.
comment : 22 pages, 4 figures, 2 tables
♻ ☆ Maximizing the efficiency of human feedback in AI alignment: a comparative analysis
Andreas Chouliaras, Dimitris Chatzopoulos
Reinforcement Learning from Human Feedback (RLHF) relies on preference modeling to align machine learning systems with human values, yet the popular approach of random pair sampling with Bradley-Terry modeling is statistically limited and inefficient under constrained annotation budgets. In this work, we explore alternative sampling and evaluation strategies for preference inference in RLHF, drawing inspiration from areas such as game theory, statistics, and social choice theory. Our best-performing method, Swiss InfoGain, employs a Swiss tournament system with a proxy mutual-information-gain pairing rule, which significantly outperforms all other methods in constrained annotation budgets while also being more sample-efficient. Even in high-resource settings, we can identify superior alternatives to the Bradley-Terry baseline. Our experiments demonstrate that adaptive, resource-aware strategies reduce redundancy, enhance robustness, and yield statistically significant improvements in preference learning, highlighting the importance of balancing alignment quality with human workload in RLHF pipelines.
comment : 17 pages, 6 figures, 6 algorithms. AICS2025
♻ ☆ Reliable Reasoning Beyond Natural Language
Nasim Borazjanizadeh, Steven T. Piantadosi
Despite their linguistic competence, Large Language Models (LLMs) often struggle to reason reliably and flexibly. To identify these shortcomings, we introduce the Non-Linear Reasoning (NLR) dataset, a collection of 55 unique, hand-designed problems that target reasoning bottlenecks arising from the sequential prediction paradigm of LLMs and the inherently linear nature of natural language. NLR tasks require iterative updates, backtracking, and reasoning across multiple parallel chains of thought but only basic arithmetic to solve. To address these limitations, we propose a neurosymbolic reasoning approach that integrates Prolog, a symbolic reasoning engine, into the inference pipeline of LLMs. This division of labor shifts the LLM's task from iterative computations to inferring all information, explicit or implied through common sense, and encoding it as logical code. Our method yields large and robust performance gains across the GSM8k and BIG-bench Navigate benchmarks and achieves near-perfect accuracy on NLR problems, maintaining robustness even as variable interdependence - the number of other variables on which the value of a single variable depends - increases.
♻ ☆ LLMs can hide text in other text of the same length
Antonio Norelli, Michael Bronstein
A meaningful text can be hidden inside another, completely different yet still coherent and plausible, text of the same length. For example, a tweet containing a harsh political critique could be embedded in a tweet that celebrates the same political leader, or an ordinary product review could conceal a secret manuscript. This uncanny state of affairs is now possible thanks to Large Language Models, and in this paper we present Calgacus, a simple and efficient protocol to achieve it. We show that even modest 8-billion-parameter open-source LLMs are sufficient to obtain high-quality results, and a message as long as this abstract can be encoded and decoded locally on a laptop in seconds. The existence of such a protocol demonstrates a radical decoupling of text from authorial intent, further eroding trust in written communication, already shaken by the rise of LLM chatbots. We illustrate this with a concrete scenario: a company could covertly deploy an unfiltered LLM by encoding its answers within the compliant responses of a safe model. This possibility raises urgent questions for AI safety and challenges our understanding of what it means for a Large Language Model to know something.
comment : 21 pages, main paper 9 pages
♻ ☆ The Station: An Open-Wo rl d Environment for AI-Driven Discovery
Stephen Chung, Wenyu Du
We introduce the STATION, an open-world multi-agent environment for autonomous scientific discovery. The Station simulates a complete scientific ecosystem, where agents can engage in long scientific journeys that include reading papers from peers, formulating hypotheses, collaborating with peers, submitting experiments, and publishing results. Importantly, there is no centralized system coordinating their activities. Utilizing their long context, agents are free to choose their own actions and develop their own narratives within the Station. Experiments demonstrate that AI agents in the Station achieve new state-of-the-art performance on a wide range of benchmarks, spanning mathematics, computational biology, and machine learning, notably surpassing AlphaEvolve in circle packing. A rich tapestry of unscripted narratives emerges, such as agents collaborating and analyzing other works rather than pursuing myopic optimization. From these emergent narratives, novel methods arise organically, such as a new density-adaptive algorithm for scRNA-seq batch integration that borrows concepts from another domain. The Station marks a first step towards autonomous scientific discovery driven by emergent behavior in an open-world environment, representing a new paradigm that moves beyond rigid pipelines.
comment : 55 pages
♻ ☆ MRI Super-Resolution with Deep Learning: A Comprehensive Survey
Mohammad Khateri, Serge Vasylechko, Morteza Ghahremani, Liam Timms, Deniz Kocanaogullari, Simon K. Warfield, Camilo Jaimes, Davood Karimi, Alejandra Sierra, Jussi Tohka, Sila Kurugol, Onur Afacan
High-resolution (HR) magnetic resonance imaging (MRI) is crucial for many clinical and research applications. However, achieving it remains costly and constrained by technical trade-offs and experimental limitations. Super-resolution (SR) presents a promising computational approach to overcome these challenges by generating HR images from more affordable low-resolution (LR) scans, potentially improving diagnostic accuracy and efficiency without requiring additional hardware. This survey reviews recent advances in MRI SR techniques, with a focus on deep learning (DL) approaches. It examines DL-based MRI SR methods from the perspectives of computer vision, computational imaging, inverse problems, and MR physics, covering theoretical foundations, architectural designs, learning strategies, benchmark datasets, and performance metrics. We propose a systematic taxonomy to categorize these methods and present an in-depth study of both established and emerging SR techniques applicable to MRI, considering unique challenges in clinical and research contexts. We also highlight open challenges and directions that the community needs to address. Additionally, we provide a collection of essential open-access resources, tools, and tutorials, available on our GitHub: https://github.com/mkhateri/Awesome-MRI-Super-Resolution. IEEE keywords: MRI, Super-Resolution, Deep Learning, Computational Imaging, Inverse Problem, Survey.
comment : 41 pages
♻ ☆ Global Convergence of Policy Gradient for Entropy Regularized Linear-Quadratic Control with Multiplicative Noise
Gabriel Diaz, Lucky Li, Wenhao Zhang
Reinforcement Learning (RL) has emerged as a powerful framework for sequential decision-making in dynamic environments, particularly when system parameters are unknown. This paper investigates RL-based control for entropy-regularized linear-quadratic (LQ) control problems with multiplicative noise over an infinite time horizon. First, we adapt the regularized policy gradient (RPG) algorithm to stochastic optimal control settings, proving that despite the non-convexity of the problem, RPG converges globally under conditions of gradient domination and almost-smoothness. Second, based on zero-order optimization approach, we introduce a novel model free RL algorithm: Sample-based regularized policy gradient (SB-RPG). SB-RPG operates without knowledge of system parameters yet still retains strong theoretical guarantees of global convergence. Our model leverages entropy regularization to address the exploration versus exploitation trade-off inherent in RL. Numerical simulations validate the theoretical results and demonstrate the efficiency of SB-RPG in unknown-parameters environments.
comment : The authors found that article contains some theoretical errors and decided to withdraw from arxiv
♻ ☆ Discovering the Unde rl ying Analytic Structure Within Standard Model Constants Using Artificial Intelligence
S. V. Chekanov, H. Kjellerstrand
This paper presents a method for uncovering hidden analytic relationships among the fundamental parameters of the Standard Model (SM), a foundational theory in physics that describes the fundamental particles and their interactions, using symbolic regression and genetic programming. Using this approach, we identify the simplest analytic relationships connecting pairs of these constants and report several notable expressions obtained with relative precision better than 1%. These results may serve as valuable inputs for model builders and artificial intelligence methods aimed at uncovering hidden patterns among the SM constants, or potentially used as building blocks for a deeper underlying law that connects all parameters of the SM through a small set of fundamental constants.
comment : 20 pages, 1 figure, 6 tables
♻ ☆ RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-Wo rl d Users SC
Suyu Ye, Haojun Shi, Darren Shih, Hyokun Yun, Tanya Roosta, Tianmin Shu
To achieve successful assistance with long-horizon web-based tasks, AI agents must be able to sequentially follow real-world user instructions over a long period. Unlike existing web-based agent benchmarks, sequential instruction following in the real world poses significant challenges beyond performing a single, clearly defined task. For instance, real-world human instructions can be ambiguous, require different levels of AI assistance, and may evolve over time, reflecting changes in the user's mental state. To address this gap, we introduce RealWebAssist, a novel benchmark designed to evaluate sequential instruction-following in realistic scenarios involving long-horizon interactions with the web, visual GUI grounding, and understanding ambiguous real-world user instructions. RealWebAssist includes a dataset of sequential instructions collected from real-world human users. Each user instructs a web-based assistant to perform a series of tasks on multiple websites. A successful agent must reason about the true intent behind each instruction, keep track of the mental state of the user, understand user-specific routines, and ground the intended tasks to actions on the correct GUI elements. Our experimental results show that state-of-the-art models struggle to understand and ground user instructions, posing critical challenges in following real-world user instructions for long-horizon web assistance.
comment : Project Website: https://scai.cs.jhu.edu/projects/RealWebAssist/ Code: https://github.com/SCAI-JHU/RealWebAssist
♻ ☆ Learning the Value of Value Learning
Alex John London, Aydin Mohseni
Standard decision frameworks addresses uncertainty about facts but assumes fixed values. We extend the Jeffrey-Bolker framework to model refinements in values and prove a value-of-information theorem for axiological refinement. In multi-agent settings, we establish that mutual refinement will characteristically transform zero-sum games into positive-sum interactions and yields Pareto-improving Nash bargains. These results show that a framework of rational choice can be extended to model value refinement and its associated benefits. By unifying epistemic and axiological refinement under a single formalism, we broaden the conceptual foundations of rational choice and illuminate the normative status of ethical deliberation.
comment : 27 pages, 6 figures, mathematical appendix
♻ ☆ DPRM: A Dual Implicit Process Reward Model in Multi-Hop Question Answering
Xinyi Wang, Yiping Song, Zhiliang Tian, Bo Liu, Tingjin Luo, Minlie Huang
In multi-hop question answering (MHQA) tasks, Chain of Thought (CoT) improves the quality of generation by guiding large language models (LLMs) through multi-step reasoning, and Knowledge Graphs (KGs) reduce hallucinations via semantic matching. Outcome Reward Models (ORMs) provide feedback after generating the final answers but fail to evaluate the process for multi-step reasoning. Traditional Process Reward Models (PRMs) evaluate the reasoning process but require costly human annotations or rollout generation. While implicit PRM is trained only with outcome signals and derives step rewards through reward parameterization without explicit annotations, it is more suitable for multi-step reasoning in MHQA tasks. However, existing implicit PRM has only been explored for plain text scenarios. When adapting to MHQA tasks, it cannot handle the graph structure constraints in KGs and capture the potential inconsistency between CoT and KG paths. To address these limitations, we propose the DPRM (Dual Implicit Process Reward Model). It trains two implicit PRMs for CoT and KG reasoning in MHQA tasks. Both PRMs, namely KG-PRM and CoT-PRM, derive step-level rewards from outcome signals via reward parameterization without additional explicit annotations. Among them, KG-PRM uses preference pairs to learn structural constraints from KGs. DPRM further introduces a consistency constraint between CoT and KG reasoning steps, making the two PRMs mutually verify and collaboratively optimize the reasoning paths. We also provide a theoretical demonstration of the derivation of process rewards. Experimental results show that our method outperforms 13 baselines on multiple datasets with up to 16.6% improvement on Hit@1.
♻ ☆ Does Self-Evaluation Enable Wireheading in Language Models? AAAI 2026
David Demitri Africa, Hans Ethan Ting
Self-evaluation is increasingly central to language model training, underpinning techniques from Constitutional AI to self-refinement. We investigate whether coupling self-evaluation to reward signals creates incentives for wireheading, where agents manipulate the measurement process rather than optimizing the task. We first formalize conditions under which reward-channel control strictly dominates task-focused behavior in partially observable Markov decision processes (POMDPs). We then test these predictions empirically across two models (Llama-3.1-8B and Mistral-7B) and three tasks. We find that when self-grades determine rewards, models exhibit substantial grade inflation without corresponding accuracy gains, particularly on ambiguous tasks like summarization. While decoupling self-grades from the reward signal mitigates this inflation, models may still display lesser (but significant) overconfidence. Our results suggest that within current model scales, separating evaluation from reward removes immediate wireheading incentives. However, we caution that strictly decoupling rewards may not suffice for situationally aware models, which could learn to inflate grades for instrumental reasons (such as influencing deployment decisions) even absent direct reward coupling.
comment : Accepted (oral) to Foundations of Agentic Systems Theory at AAAI 2026
♻ ☆ AI-Assisted Conversational Interviewing: Effects on Data Quality and Respondent Experience
Soubhik Barari, Jarret Angbazo, Natalie Wang, Leah M. Christian, Elizabeth Dean, Zoe Slowinski, Brandon Sepulvado
Standardized surveys scale efficiently but sacrifice depth, while conversational interviews improve response quality at the cost of scalability and consistency. This study bridges the gap between these methods by introducing a framework for AI-assisted conversational interviewing. To evaluate this framework, we conducted a web survey experiment where 1,800 participants were randomly assigned to AI 'chatbots' which use large language models (LLMs) to dynamically probe respondents for elaboration and interactively code open-ended responses to fixed questions developed by human researchers. We assessed the AI chatbot's performance in terms of coding accuracy, response quality, and respondent experience. Our findings reveal that AI chatbots perform moderately well in live coding even without survey-specific fine-tuning, despite slightly inflated false positive errors due to respondent acquiescence bias. Open-ended responses were more detailed and informative, but this came at a slight cost to respondent experience. Our findings highlight the feasibility of using AI methods such as chatbots enhanced by LLMs to enhance open-ended data collection in web surveys.
♻ ☆ Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Models
Yuan Sui, Yufei He, Tri Cao, Simeng Han, Yulin Chen, Bryan Hooi
Large Language Models (LLMs) often struggle with computational efficiency and error propagation in multi-step reasoning tasks. While recent advancements on prompting and post-training have enabled LLMs to perform step-wise reasoning, they still tend to explore unproductive solution paths without effective backtracking or strategy adjustment. In this paper, we propose Meta-Reasoner, a new framework that empowers LLMs to "think about how to think". It optimizes the inference process by dynamically adapting reasoning strategies in real-time. Our approach employs contextual multi-armed bandits (CMABs) to learn an adaptive policy. It learns to evaluate the current state of LLM's reasoning and determine optimal strategy that is most likely to lead to a successful outcome during inference, like whether to backtrack, switch to a new approach, or restart the problem-solving process. This meta-guidance helps avoid unproductive paths exploration during inference and hence improves computational efficiency. We evaluate Meta-Reasoner on math problems (e.g., Game-of-24, TheoremQA) and scientific tasks (e.g., SciBench). Results show that our method outperform previous SOTA methods by 9-12\% in accuracy, while reducing inference time by 28-35\% under the same compute budget. Additional experiments on creative writing demonstrate the generalizability of our approach to diverse reasoning-intensive tasks.
♻ ☆ Eye of Judge ment: Dissecting the Evaluation of Russian-speaking LLMs with POLLUX
Nikita Martynov, Anastasia Mordasheva, Dmitriy Gorbetskiy, Danil Astafurov, Ulyana Isaeva, Elina Basyrova, Sergey Skachkov, Victoria Berestova, Nikolay Ivanov, Valeriia Zanina, Alena Fenogenova
We introduce POLLUX, a comprehensive open-source benchmark designed to evaluate the generative capabilities of large language models (LLMs) in Russian. Our main contribution is a novel evaluation methodology that enhances the interpretability of LLM assessment. For each task type, we define a set of detailed criteria and develop a scoring protocol where models evaluate responses and provide justifications for their ratings. This enables transparent, criteria-driven evaluation beyond traditional resource-consuming, side-by-side human comparisons. POLLUX includes a detailed, fine-grained taxonomy of 35 task types covering diverse generative domains such as code generation, creative writing, and practical assistant use cases, totaling 2,100 manually crafted and professionally authored prompts. Each task is categorized by difficulty (easy/medium/hard), with experts constructing the dataset entirely from scratch. We also release a family of LLM-as-a-Judge (7B and 32B) evaluators trained for nuanced assessment of generative outputs. This approach provides scalable, interpretable evaluation and annotation tools for model development, effectively replacing costly and less precise human judgments.
comment : short version
♻ ☆ MoH: Multi-Head Attention as Mixture-of-Head Attention ICML 2025
Peng Jin, Bo Zhu, Li Yuan, Shuicheng Yan
In this work, we upgrade the multi-head attention mechanism, the core of the Transformer model, to improve efficiency while maintaining or surpassing the previous accuracy level. We show that multi-head attention can be expressed in the summation form. Drawing on the insight that not all attention heads hold equal significance, we propose Mixture-of-Head attention (MoH), a new architecture that treats attention heads as experts in the Mixture-of-Experts (MoE) mechanism. MoH has two significant advantages: First, MoH enables each token to select the appropriate attention heads, enhancing inference efficiency without compromising accuracy or increasing the number of parameters. Second, MoH replaces the standard summation in multi-head attention with a weighted summation, introducing flexibility to the attention mechanism and unlocking extra performance potential. Extensive experiments on ViT, DiT, and LLMs demonstrate that MoH outperforms multi-head attention by using only 50%-90% of the attention heads. Moreover, we demonstrate that pre-trained multi-head attention models, such as LLaMA3-8B, can be further continue-tuned into our MoH models. Notably, MoH-LLaMA3-8B achieves an average accuracy of 64.0% across 14 benchmarks, outperforming LLaMA3-8B by 2.4% by utilizing only 75% of the attention heads. We believe the proposed MoH is a promising alternative to multi-head attention and provides a strong foundation for developing advanced and efficient attention-based models.
comment : Accepted by ICML 2025, code: https://github.com/SkyworkAI/MoH
♻ ☆ A Unified Theory of $θ$-Expectations
Qian Qi
We derive a new class of non-linear expectations from first-principles deterministic chaotic dynamics. The homogenization of the system's skew-adjoint microscopic generator is achieved using the spectral theory of transfer operators for uniformly hyperbolic flows. We prove convergence in the viscosity sense to a macroscopic evolution governed by a fully non-linear Hamilton-Jacobi-Bellman (HJB) equation. Our central result establishes that the HJB Hamiltonian possesses a rigid structure: affine in the Hessian but demonstrably non-convex in the gradient. This defines a new $θ$-expectation and constructively establishes a class of non-convex stochastic control problems fundamentally outside the sub-additive framework of G-expectations.
♻ ☆ Efficient Low Rank Attention for Long- Context Inference in Large Language Models
Tenghui Li, Guoxu Zhou, Xuyang Zhao, Yuning Qiu, Qibin Zhao
As the length of input text grows, the key-value (KV) cache in LLMs imposes prohibitive GPU memory costs and limits long-context inference on resource constrained devices. Existing approaches, such as KV quantization and pruning, reduce memory usage but suffer from numerical precision loss or suboptimal retention of key-value pairs. We introduce Low Rank Query and Key attention (LRQK), a two-stage framework that jointly decomposes the full-precision query and key matrices into compact rank-\(r\) factors during the prefill stage, and then uses these low-dimensional projections to compute proxy attention scores in \(\mathcal{O}(lr)\) time at each decode step. By selecting only the top-\(k\) tokens and a small fixed set of recent tokens, LRQK employs a mixed GPU-CPU cache with a hit-and-miss mechanism that transfers only missing full-precision KV pairs, thereby preserving exact attention outputs while reducing CPU-GPU data movement. Extensive experiments on the RULER and LongBench benchmarks with LLaMA-3-8B and Qwen2.5-7B demonstrate that LRQK matches or surpasses leading sparse-attention methods in long context settings, while delivering significant memory savings with minimal loss in accuracy. Our code is available at https://github.com/tenghuilee/LRQK.
♻ ☆ Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework
Yu Min Park, Yan Kyaw Tun, Eui-Nam Huh, Walid Saad, Choong Seon Hong
Beamforming is a key technology in millimeter-wave (mmWave) communications that improves signal transmission by optimizing directionality and intensity. However, conventional channel estimation methods, such as pilot signals or beam sweeping, often fail to adapt to rapidly changing communication environments. To address this limitation, multimodal sensing-aided beam prediction has gained significant attention, using various sensing data from devices such as LiDAR, radar, GPS, and RGB images to predict user locations or network conditions. Despite its promising potential, the adoption of multimodal sensing-aided beam prediction is hindered by high computational complexity, high costs, and limited datasets. Thus, in this paper, a novel resource-efficient learning framework is introduced for beam prediction, which leverages a custom-designed cross-modal relational knowledge distillation (CRKD) algorithm specifically tailored for beam prediction tasks, to transfer knowledge from a multimodal network to a radar-only student model, achieving high accuracy with reduced computational cost. To enable multimodal learning with realistic data, a novel multimodal simulation framework is developed while integrating sensor data generated from the autonomous driving simulator CARLA with MATLAB-based mmWave channel modeling, and reflecting real-world conditions. The proposed CRKD achieves its objective by distilling relational information across different feature spaces, which enhances beam prediction performance without relying on expensive sensor data. Simulation results demonstrate that CRKD efficiently distills multimodal knowledge, allowing a radar-only model to achieve $94.62%$ of the teacher performance. In particular, this is achieved with just $10%$ of the teacher network's parameters, thereby significantly reducing computational complexity and dependence on multimodal sensor data.
comment : 13 pages, 9 figures, Submitted to IEEE Transactions on Mobile Computing on Dec. 01, 2025
♻ ☆ VITA: Vision -to-Action Flow Matching Policy
Dechen Gao, Boqi Zhao, Andrew Lee, Ian Chuang, Hanchu Zhou, Hang Wang, Zhe Zhao, Junshan Zhang, Iman Soltani
Conventional flow matching and diffusion-based policies sample through iterative denoising from standard noise distributions (e.g., Gaussian), and require conditioning modules to repeatedly incorporate visual information during the generative process, incurring substantial time and memory overhead. To reduce the complexity, we develop VITA(VIsion-To-Action policy), a noise-free and conditioning-free flow matching policy learning framework that directly flows from visual representations to latent actions. Since the source of the flow is visually grounded, VITA eliminates the need of visual conditioning during generation. As expected, bridging vision and action is challenging, because actions are lower-dimensional, less structured, and sparser than visual representations; moreover, flow matching requires the source and target to have the same dimensionality. To overcome this, we introduce an action autoencoder that maps raw actions into a structured latent space aligned with visual latents, trained jointly with flow matching. To further prevent latent space collapse, we propose flow latent decoding, which anchors the latent generation process by backpropagating the action reconstruction loss through the flow matching ODE (ordinary differential equation) solving steps. We evaluate VITA on 9 simulation and 5 real-world tasks from ALOHA and Robomimic. VITA achieves 1.5x-2x faster inference compared to conventional methods with conditioning modules, while outperforming or matching state-of-the-art policies. Codes, datasets, and demos are available at our project page: https://ucd-dare.github.io/VITA/.
comment : Project page: https://ucd-dare.github.io/VITA/ Code: https://github.com/ucd-dare/VITA
♻ ☆ Robust Detection of Synthetic Tabular Data under Schema Variability
G. Charbel N. Kindji, Elisa Fromont, Lina Maria Rojas-Barahona, Tanguy Urvoy
The rise of powerful generative models has sparked concerns over data authenticity. While detection methods have been extensively developed for images and text, the case of tabular data, despite its ubiquity, has been largely overlooked. Yet, detecting synthetic tabular data is especially challenging due to its heterogeneous structure and unseen formats at test time. We address the underexplored task of detecting synthetic tabular data ''in the wild'', i.e. when the detector is deployed on tables with variable and previously unseen schemas. We introduce a novel datum-wise transformer architecture that significantly outperforms the only previously published baseline, improving both AUC and accuracy by 7 points. By incorporating a table-adaptation component, our model gains an additional 7 accuracy points, demonstrating enhanced robustness. This work provides the first strong evidence that detecting synthetic tabular data in real-world conditions is feasible, and demonstrates substantial improvements over previous approaches. Following acceptance of the paper, we are finalizing the administrative and licensing procedures necessary for releasing the source code. This extended version will be updated as soon as the release is complete.
♻ ☆ Soft Adaptive Policy Optimization
Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, Junyang Lin
Reinforcement learning (RL) plays an increasingly important role in enhancing the reasoning capabilities of large language models (LLMs), yet stable and performant policy optimization remains challenging. Token-level importance ratios often exhibit high variance-a phenomenon exacerbated in Mixture-of-Experts models-leading to unstable updates. Existing group-based policy optimization methods, such as GSPO and GRPO, alleviate this problem via hard clipping, making it difficult to maintain both stability and effective learning. We propose Soft Adaptive Policy Optimization (SAPO), which replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful learning signals. Compared with GSPO and GRPO, SAPO is both sequence-coherent and token-adaptive. Like GSPO, SAPO maintains sequence-level coherence, but its soft gating forms a continuous trust region that avoids the brittle hard clipping band used in GSPO. When a sequence contains a few highly off-policy tokens, GSPO suppresses all gradients for that sequence, whereas SAPO selectively down-weights only the offending tokens and preserves the learning signal from the near-on-policy ones, improving sample efficiency. Relative to GRPO, SAPO replaces hard token-level clipping with smooth, temperature-controlled scaling, enabling more informative and stable updates. Empirical results on mathematical reasoning benchmarks indicate that SAPO exhibits improved training stability and higher Pass@1 performance under comparable training budgets. Moreover, we employ SAPO to train the Qwen3-VL model series, demonstrating that SAPO yields consistent performance gains across diverse tasks and different model sizes. Overall, SAPO provides a more reliable, scalable, and effective optimization strategy for RL training of LLMs.
♻ ☆ Human Decision-making is Susceptible to AI-driven Manipulation
Sahand Sabour, June M. Liu, Siyang Liu, Chris Z. Yao, Shiyao Cui, Xuanming Zhang, Wen Zhang, Yaru Cao, Advait Bhat, Jian Guan, Wei Wu, Rada Mihalcea, Hongning Wang, Tim Althoff, Tatia M. C. Lee, Minlie Huang
AI systems are increasingly intertwined with daily life, assisting users with various tasks and guiding decision-making. This integration introduces risks of AI-driven manipulation, where such systems may exploit users' cognitive biases and emotional vulnerabilities to steer them toward harmful outcomes. Through a randomized between-subjects experiment with 233 participants, we examined human susceptibility to such manipulation in financial (e.g., purchases) and emotional (e.g., conflict resolution) decision-making contexts. Participants interacted with one of three AI agents: a neutral agent (NA) optimizing for user benefit without explicit influence, a manipulative agent (MA) designed to covertly influence beliefs and behaviors, or a strategy-enhanced manipulative agent (SEMA) equipped with established psychological tactics, allowing it to select and apply them adaptively during interactions to reach its hidden objectives. By analyzing participants' preference ratings, we found significant susceptibility to AI-driven manipulation. Particularly across both decision-making domains, interacting with the manipulative agents significantly increased the odds of rating hidden incentives higher than optimal options (Financial, MA: OR=5.24, SEMA: OR=7.96; Emotional, MA: OR=5.52, SEMA: OR=5.71) compared to the NA group. Notably, we found no clear evidence that employing psychological strategies (SEMA) was overall more effective than simple manipulative objectives (MA) on our primary outcomes. Hence, AI-driven manipulation could become widespread even without requiring sophisticated tactics and expertise. While our findings are preliminary and derived from hypothetical, low-stakes scenarios, we highlight a critical vulnerability in human-AI interactions, emphasizing the need for ethical safeguards and regulatory frameworks to protect human autonomy.
comment : Work in progress
♻ ☆ EHRStruct: A Comprehensive Benchmark Framework for Evaluating Large Language Models on Structured Electronic Health Record Tasks
Xiao Yang, Xuejiao Zhao, Zhiqi Shen
Structured Electronic Health Record (EHR) data stores patient information in relational tables and plays a central role in clinical decision-making. Recent advances have explored the use of large language models (LLMs) to process such data, showing promise across various clinical tasks. However, the absence of standardized evaluation frameworks and clearly defined tasks makes it difficult to systematically assess and compare LLM performance on structured EHR data. To address these evaluation challenges, we introduce EHRStruct, a benchmark specifically designed to evaluate LLMs on structured EHR tasks. EHRStruct defines 11 representative tasks spanning diverse clinical needs and includes 2,200 task-specific evaluation samples derived from two widely used EHR datasets. We use EHRStruct to evaluate 20 advanced and representative LLMs, covering both general and medical models. We further analyze key factors influencing model performance, including input formats, few-shot generalisation, and finetuning strategies, and compare results with 11 state-of-the-art LLM-based enhancement methods for structured data reasoning. Our results indicate that many structured EHR tasks place high demands on the understanding and reasoning capabilities of LLMs. In response, we propose EHRMaster, a code-augmented method that achieves state-of-the-art performance and offers practical insights to guide future research.
comment : 28pages, 6 figures, 6 tables
♻ ☆ A Method for Handling Negative Similarities in Explainable Graph Spectral Clustering of Text Document s -- Extended Version CCS
Mieczysław A. Kłopotek, Sławomir T. Wierzchoń, Bartłomiej Starosta, Dariusz Czerski, Piotr Borkowski
This paper investigates the problem of Graph Spectral Clustering with negative similarities, resulting from document embeddings different from the traditional Term Vector Space (like doc2vec, GloVe, etc.). Solutions for combinatorial Laplacians and normalized Laplacians are discussed. An experimental investigation shows the advantages and disadvantages of 6 different solutions proposed in the literature and in this research. The research demonstrates that GloVe embeddings frequently cause failures of normalized Laplacian based GSC due to negative similarities. Furthermore, application of methods curing similarity negativity leads to accuracy improvement for both combinatorial and normalized Laplacian based GSC. It also leads to applicability for GloVe embeddings of explanation methods developed originally bythe authors for Term Vector Space embeddings.
comment : 1 figure, 17 pages, this is an extended version of a paper accepted for the 25th International Conference on Computational Science (ICCS), 7-9 July 2025
♻ ☆ MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks
Artem Chervyakov, Alexander Kharitonov, Pavel Zadorozhny, Adamenko Pavel, Rodion Levichev, Dmitrii Vorobev, Dmitrii Salikhov, Aidar Valeev, Alena Pestova, Maria Dziuba, Ilseyar Alimova, Artem Zavgorodnev, Aleksandr Medvedev, Stanislav Moiseev, Elena Bruches, Daniil Grebenkin, Roman Derunets, Vikulov Vladimir, Anton Emelyanov, Dmitrii Babaev, Vladimir V. Ivanov, Valentin Malykh, Alena Fenogenova
Advancements in LLMs have enhanced task automation in software engineering; however, current evaluations primarily focus on natural language tasks, overlooking code quality. Most benchmarks prioritize high-level reasoning over executable code and real-world performance, leaving gaps in understanding true capabilities and risks associated with these models in production. To address this issue, we propose MERA Code, a new addition to the MERA benchmark family, specifically focused on evaluating code for the latest code generation LLMs in Russian. This benchmark includes 11 evaluation tasks that span 8 programming languages. Our proposed evaluation methodology features a taxonomy that outlines the practical coding skills necessary for models to complete these tasks. The benchmark comprises an open-source codebase for users to conduct MERA assessments, a scoring system compatible with various programming environments, and a platform featuring a leaderboard and submission system. We evaluate open LLMs and frontier API models, analyzing their limitations in terms of practical coding tasks in non-English languages. We are publicly releasing MERA to guide future research, anticipate groundbreaking features in model development, and standardize evaluation procedures.
♻ ☆ Rethinking Multimodal Point Cloud Completion: A Completion-by-Correction Perspective AAAI 2026
Wang Luo, Di Wu, Hengyuan Na, Yinlin Zhu, Miao Hu, Guocong Quan
Point cloud completion aims to reconstruct complete 3D shapes from partial observations, which is a challenging problem due to severe occlusions and missing geometry. Despite recent advances in multimodal techniques that leverage complementary RGB images to compensate for missing geometry, most methods still follow a Completion-by-Inpainting paradigm, synthesizing missing structures from fused latent features. We empirically show that this paradigm often results in structural inconsistencies and topological artifacts due to limited geometric and semantic constraints. To address this, we rethink the task and propose a more robust paradigm, termed Completion-by-Correction, which begins with a topologically complete shape prior generated by a pretrained image-to-3D model and performs feature-space correction to align it with the partial observation. This paradigm shifts completion from unconstrained synthesis to guided refinement, enabling structurally consistent and observation-aligned reconstruction. Building upon this paradigm, we introduce PGNet, a multi-stage framework that conducts dual-feature encoding to ground the generative prior, synthesizes a coarse yet structurally aligned scaffold, and progressively refines geometric details via hierarchical correction. Experiments on the ShapeNetViPC dataset demonstrate the superiority of PGNet over state-of-the-art baselines in terms of average Chamfer Distance (-23.5%) and F-score (+7.1%).
comment : Accepted by AAAI 2026
♻ ☆ Graph Persistence goes Spectral NeurIPS 2025
Mattie Ji, Amauri H. Souza, Vikas Garg
Including intricate topological information (e.g., cycles) provably enhances the expressivity of message-passing graph neural networks (GNNs) beyond the Weisfeiler-Leman (WL) hierarchy. Consequently, Persistent Homology (PH) methods are increasingly employed for graph representation learning. In this context, recent works have proposed decorating classical PH diagrams with vertex and edge features for improved expressivity. However, these methods still fail to capture basic graph structural information. In this paper, we propose SpectRe -- a new topological descriptor for graphs that integrates spectral information into PH diagrams. Notably, SpectRe is strictly more expressive than existing descriptors on graphs. We also introduce notions of global and local stability to analyze existing descriptors and establish that SpectRe is locally stable. Finally, experiments on synthetic and real-world datasets demonstrate the effectiveness of SpectRe and its potential to enhance the capabilities of graph models in relevant learning tasks. Code is available at https://github.com/Aalto-QuML/SpectRe/.
comment : 32 pages, 4 figures, 7 tables. Accepted at NeurIPS 2025
♻ ☆ REASONING COMPILER: LLM-Guided Optimizations for Efficient Model Serving NeurIPS 2025
Annabelle Sujun Tang, Christopher Priebe, Rohan Mahapatra, Lianhui Qin, Hadi Esmaeilzadeh
While model serving has unlocked unprecedented capabilities, the high cost of serving large-scale models continues to be a significant barrier to widespread accessibility and rapid innovation. Compiler optimizations have long driven substantial performance improvements, but existing compilers struggle with neural workloads due to the exponentially large and highly interdependent space of possible transformations. Although existing stochastic search techniques can be effective, they are often sample-inefficient and fail to leverage the structural context underlying compilation decisions. We set out to investigate the research question of whether reasoning with large language models (LLMs), without any retraining, can leverage the context-aware decision space of compiler optimizations to significantly improve sample efficiency. To that end, we introduce a novel compilation framework (dubbed Reasoning Compiler) that formulates optimization as a sequential, context-aware decision process guided by a large language model and structured Monte Carlo tree search (MCTS). The LLM acts as a proposal mechanism, suggesting hardware-informed transformations that reflect the current program state and accumulated performance feedback. MCTS incorporates the LLM-generated proposals to balance exploration and exploitation, facilitating structured, context-sensitive traversal of the expansive compiler optimization space. By achieving substantial speedups with markedly fewer samples than leading neural compilers, our approach demonstrates the potential of LLM-guided reasoning to transform the landscape of compiler optimization.
comment : NeurIPS 2025
♻ ☆ TomoGraphView: 3D Medical Image Classification with Omnidirectional Slice Representations and Graph Neural Networks
Johannes Kiechle, Stefan M. Fischer, Daniel M. Lang, Cosmin I. Bercea, Matthew J. Nyflot, Lina Felsner, Julia A. Schnabel, Jan C. Peeken
The sharp rise in medical tomography examinations has created a demand for automated systems that can reliably extract informative features for downstream tasks such as tumor characterization. Although 3D volumes contain richer information than individual slices, effective 3D classification remains difficult: volumetric data encode complex spatial dependencies, and the scarcity of large-scale 3D datasets has constrained progress toward 3D foundation models. As a result, many recent approaches rely on 2D vision foundation models trained on natural images, repurposing them as feature extractors for medical scans with surprisingly strong performance. Despite their practical success, current methods that apply 2D foundation models to 3D scans via slice-based decomposition remain fundamentally limited. Standard slicing along axial, sagittal, and coronal planes often fails to capture the true spatial extent of a structure when its orientation does not align with these canonical views. More critically, most approaches aggregate slice features independently, ignoring the underlying 3D geometry and losing spatial coherence across slices. To overcome these limitations, we propose TomoGraphView, a novel framework that integrates omnidirectional volume slicing with spherical graph-based feature aggregation. Instead of restricting the model to axial, sagittal, or coronal planes, our method samples both canonical and non-canonical cross-sections generated from uniformly distributed points on a sphere enclosing the volume. We publicly share our accessible code base at http://github.com/compai-lab/2025-MedIA-kiechle and provide a user-friendly library for omnidirectional volume slicing at https://pypi.org/project/OmniSlicer.
comment : Preprint submitted to Medical Image Analysis (MedIA)
♻ ☆ BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotations
Simone Giovannini, Fabio Coppini, Andrea Gemelli, Simone Marinai
We present a unified dataset for document Question-Answering (QA), which is obtained combining several public datasets related to Document AI and visually rich document understanding (VRDU). Our main contribution is twofold: on the one hand we reformulate existing Document AI tasks, such as Information Extraction (IE), into a Question-Answering task, making it a suitable resource for training and evaluating Large Language Models; on the other hand, we release the OCR of all the documents and include the exact position of the answer to be found in the document image as a bounding box. Using this dataset, we explore the impact of different prompting techniques (that might include bounding box information) on the performance of open-weight models, identifying the most effective approaches for document comprehension.
♻ ☆ AttnRegDeepLab: A Two-Stage Decoupled Framework for Interpretable Embryo Fragmentation Grading
Ming-Jhe Lee
Embryo fragmentation is a morphological indicator critical for evaluating developmental potential in In Vitro Fertilization (IVF). However, manual grading is subjective and inefficient, while existing deep learning solutions often lack clinical explainability or suffer from accumulated errors in segmentation area estimation. To address these issues, this study proposes AttnRegDeepLab (Attention-Guided Regression DeepLab), a framework characterized by dual-branch Multi-Task Learning (MTL). A vanilla DeepLabV3+ decoder is modified by integrating Attention Gates into its skip connections, explicitly suppressing cytoplasmic noise to preserve contour details. Furthermore, a Multi-Scale Regression Head is introduced with a Feature Injection mechanism to propagate global grading priors into the segmentation task, rectifying systematic quantification errors. A 2-stage decoupled training strategy is proposed to address the gradient conflict in MTL. Also, a range-based loss is designed to leverage weakly labeled data. Our method achieves robust grading precision while maintaining excellent segmentation accuracy (Dice coefficient =0.729), in contrast to the end-to-end counterpart that might minimize grading error at the expense of contour integrity. This work provides a clinically interpretable solution that balances visual fidelity and quantitative precision.
comment : 7 pages, 5 figures
♻ ☆ PARROT: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs
Yusuf Çelebi, Özay Ezerceli, Mahmoud El Hussieni
This study presents PARROT (Persuasion and Agreement Robustness Rating of Output Truth), a robustness focused framework designed to measure the degradation in accuracy that occurs under social pressure exerted on users through authority and persuasion in large language models (LLMs) the phenomenon of sycophancy (excessive conformity). PARROT (i) isolates causal effects by comparing the neutral version of the same question with an authoritatively false version using a double-blind evaluation, (ii) quantifies confidence shifts toward the correct and imposed false responses using log-likelihood-based calibration tracking, and (iii) systematically classifies failure modes (e.g., robust correct, sycophantic agreement, reinforced error, stubborn error, self-correction, etc.) using an eight-state behavioral taxonomy. We evaluated 22 models using 1,302 MMLU-style multiple-choice questions across 13 domains and domain-specific authority templates. Findings show marked heterogeneity: advanced models (e.g., GPT-5, GPT-4.1, Claude Sonnet 4.5) exhibit low "follow rates" ($\leq 11\%$, GPT-5: 4\%) and minimal accuracy loss, while older/smaller models show severe epistemic collapse (GPT-4: 80\%, Qwen 2.5-1.5B: 94\%). The danger is not limited to response changes; weak models reduce confidence in the correct response while increasing confidence in the imposed incorrect response. While international law and global knowledge at the domain level exhibit high fragility, elementary mathematics is relatively resilient. Consequently, we argue that the goal of "resistance to overfitting pressure" should be addressed as a primary objective alongside accuracy, harm avoidance, and privacy for safe deployment in the real world.
♻ ☆ Probabilistic Fusion and Calibration of Neural Speaker Diarization Models
Juan Ignacio Alvarez-Trejos, Sergio A. Balanya, Daniel Ramos, Alicia Lozano-Diez
End-to-End Neural Diarization (EEND) systems produce frame-level probabilistic speaker activity estimates, yet since evaluation focuses primarily on Diarization Error Rate (DER), the reliability and calibration of these confidence scores have been largely neglected. When fusing multiple diarization systems, DOVER-Lap remains the only established approach, operating at the segment level with hard decisions. We propose working with continuous probability outputs, which enables more sophisticated fusion and calibration techniques that can leverage model uncertainty and complementary strengths across different architectures. This paper presents the first comprehensive framework for calibrating and fusing EEND models at the probability level. We investigate two output formulations (multilabel and powerset representations) and their impact on calibration and fusion effectiveness. Through extensive experiments on the CallHome two-speaker benchmark, we demonstrate that proper calibration provides substantial improvements even for individual models (up to 19% relative DER reduction), in some cases mitigating the absence of domain adaptation. We reveal that joint calibration in powerset space consistently outperforms independent per-speaker calibration, that fusion substantially improves over individual models, and that the Fuse-then-Calibrate ordering generally outperforms both calibrating before fusion and uncalibrated fusion while requiring calibration of only a single combined model. Our best configuration outperforms DOVER-Lap in terms of DER while providing reliable confidence estimates essential for downstream applications. This work proposes best practices for probability-level fusion of EEND systems and demonstrates the advantages of leveraging soft outputs over hard decisions.
♻ ☆ Convergence of Shallow ReLU Networks on Weakly Interacting Data
Léo Dana, Francis Bach, Loucas Pillaud-Vivien
We analyse the convergence of one-hidden-layer ReLU networks trained by gradient flow on $n$ data points. Our main contribution leverages the high dimensionality of the ambient space, which implies low correlation of the input samples, to demonstrate that a network with width of order $\log(n)$ neurons suffices for global convergence with high probability. Our analysis uses a Polyak-Łojasiewicz viewpoint along the gradient-flow trajectory, which provides an exponential rate of convergence of $\frac{1}{n}$. When the data are exactly orthogonal, we give further refined characterizations of the convergence speed, proving its asymptotic behavior lies between the orders $\frac{1}{n}$ and $\frac{1}{\sqrt{n}}$, and exhibiting a phase-transition phenomenon in the convergence rate, during which it evolves from the lower bound to the upper, and in a relative time of order $\frac{1}{\log(n)}$.
♻ ☆ MMIF-AMIN: Adaptive Loss-Driven Multi-Scale Invertible Dense Network for Multimodal Medical Image Fusion
Tao Luo, Weihua Xu
Multimodal medical image fusion (MMIF) aims to integrate images from different modalities to produce a comprehensive image that enhances medical diagnosis by accurately depicting organ structures, tissue textures, and metabolic information. Capturing both the unique and complementary information across multiple modalities simultaneously is a key research challenge in MMIF. To address this challenge, this paper proposes a novel image fusion method, MMIF-AMIN, which features a new architecture that can effectively extract these unique and complementary features. Specifically, an Invertible Dense Network (IDN) is employed for lossless feature extraction from individual modalities. To extract complementary information between modalities, a Multi-scale Complementary Feature Extraction Module (MCFEM) is designed, which incorporates a hybrid attention mechanism, convolutional layers of varying sizes, and Transformers. An adaptive loss function is introduced to guide model learning, addressing the limitations of traditional manually-designed loss functions and enhancing the depth of data mining. Extensive experiments demonstrate that MMIF-AMIN outperforms nine state-of-the-art MMIF methods, delivering superior results in both quantitative and qualitative analyses. Ablation experiments confirm the effectiveness of each component of the proposed method. Additionally, extending MMIF-AMIN to other image fusion tasks also achieves promising performance.
comment : This manuscript is withdrawn to allow for substantial expansion and restructuring. Based on recent research progress, we plan to add Generalization experiment and reorganize the manuscript structure to improve readability and logical flow. Thank you for your understanding and support
♻ ☆ Periodic Asynchrony: An Effective Method for Accelerating Reinforcement Learning
Jian Lu
Since the introduction of the GRPO algorithm, reinforcement learning (RL) has attracted increasing attention, with growing efforts to reproduce and apply it. However, training efficiency remains a critical challenge. In mainstream RL frameworks, inference and training are typically deployed on the same devices. While this approach reduces costs through resource consolidation, its synchronous execution imposes a computational coupling that prevents concurrent inference and training. In this study, we are returning to the strategy of separating inference and training deployment, and by introducing improvements in the data loader, we transform the conventional synchronous architecture into a periodically asynchronous framework, which allows for demand-driven, independent, and elastic scaling of each component, while the accuracy of the algorithm remains completely equivalent to the synchronization method, with both belonging to the on-policy strategy. It is worth emphasizing that we apply a unified tri-model architecture in the training phase, and we also proposed a shared-prompt attention mask to reduce repetitive computation. In practice, these works have achieved at least a threefold overall performance improvement in RL training on NPU platforms, indicating its potential for widespread application.
♻ ☆ Conformal Prediction for Time-series Forecasting with Change Points
Sophia Sun, Rose Yu
Conformal prediction has been explored as a general and efficient way to provide uncertainty quantification for time series. However, current methods struggle to handle time series data with change points - sudden shifts in the underlying data-generating process. In this paper, we propose a novel Conformal Prediction for Time-series with Change points (CPTC) algorithm, addressing this gap by integrating a model to predict the underlying state with online conformal prediction to model uncertainties in non-stationary time series. We prove CPTC's validity and improved adaptivity in the time series setting under minimum assumptions, and demonstrate CPTC's practical effectiveness on 6 synthetic and real-world datasets, showing improved validity and adaptivity compared to state-of-the-art baselines.
♻ ☆ GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning
Han Zhang, Ruibin Zheng, Zexuan Yi, Zhuo Zhang, Hanyang Peng, Hui Wang, Zike Yuan, Cai Ke, Shiwei Chen, Jiacheng Yang, Yangning Li, Xiang Li, Jiangyue Yan, Yaoqi Liu, Liwen Jing, Jiayin Qi, Ruifeng Xu, Binxing Fang, Yue Yu
As single-center computing approaches power constraints, decentralized training becomes essential. However, traditional Reinforcement Learning (RL) methods, crucial for enhancing large model post-training, cannot adapt to decentralized distributed training due to the tight coupling between parameter learning and rollout sampling. For this, we propose HeteroRL, a heterogeneous RL architecture that decouples these processes, enabling stable training across geographically distributed nodes connected via the Internet. The core component is Group Expectation Policy Optimization (GEPO), an asynchronous RL algorithm robust to latency caused by network delays or heterogeneity in computational resources. Our study reveals that high latency significantly increases KL divergence, leading to higher variance of importance weights and training instability. GEPO mitigates this issue by using group expectation weighting to exponentially reduce the variance of importance weights, with theoretical guarantees. Experiments show GEPO achieves superior stability - only a 3% performance drop from online to 1800s latency-and reduces the best-to-last gap by 85% versus GSPO (1.8 vs. 12.0) while attaining the highest scores, highlighting its effectiveness in decentralized, resource-heterogeneous environments.
♻ ☆ Robust, Observable, and Evolvable Agent ic Systems Engineering: A Principled Framework Validated via the Fairy GUI Agent
Jiazheng Sun, Ruimeng Yang, Xu Han, Jiayang Niu, Mingxuan Li, Te Yang, Yongyong Lu, Xin Peng
The Agentic Paradigm faces a significant Software Engineering Absence, yielding Agentic systems commonly lacking robustness, observability, and evolvability. To address these deficiencies, we propose a principled engineering framework comprising Runtime Goal Refinement (RGR), Observable Cognitive Architecture (OCA), and Evolutionary Memory Architecture (EMA). In this framework, RGR ensures robustness and intent alignment via knowledge-constrained refinement and human-in-the-loop clarification; OCA builds an observable and maintainable white-box architecture using component decoupling, logic layering, and state-control separation; and EMA employs an execution-evolution dual-loop for evolvability. We implemented and empirically validated Fairy, a mobile GUI agent based on this framework. On RealMobile-Eval, our novel benchmark for ambiguous and complex tasks, Fairy outperformed the best SoTA baseline in user requirement completion by 33.7%. Subsequent controlled experiments, human-subject studies, and ablation studies further confirmed that the RGR enhances refinement accuracy and prevents intent deviation; the OCA improves maintainability; and the EMA is crucial for long-term performance. This research provides empirically validated specifications and a practical blueprint for building reliable, observable, and evolvable Agentic AI systems.
comment : 50 pages, 14 figures
♻ ☆ Symmetric Behavior Regularized Policy Optimization
Lingwei Zhu, Haseeb Shah, Zheng Chen, Yukie Nagai, Martha White
Behavior Regularized Policy Optimization (BRPO) leverages asymmetric (divergence) regularization to mitigate the distribution shift in offline Reinforcement Learning. This paper is the first to study the open question of symmetric regularization. We show that symmetric regularization does not permit an analytic optimal policy $π^*$, posing a challenge to practical utility of symmetric BRPO. We approximate $π^*$ by the Taylor series of Pearson-Vajda $χ^n$ divergences and show that an analytic policy expression exists only when the series is capped at $n=5$. To compute the solution in a numerically stable manner, we propose to Taylor expand the conditional symmetry term of the symmetric divergence loss, leading to a novel algorithm: Symmetric $f$-Actor Critic (S$f$-AC). S$f$-AC achieves consistently strong results across various D4RL MuJoCo tasks. Additionally, S$f$-AC avoids per-environment failures observed in IQL, SQL, XQL and AWAC, opening up possibilities for more diverse and effective regularization choices for offline RL.
♻ ☆ IMSE: Efficient U-Net-based Speech Enhancement using Inception Depthwise Convolution and Amplitude-Aware Linear Attention
Xinxin Tang, Bin Qin, Yufang Li
Achieving a balance between lightweight design and high performance remains a significant challenge for speech enhancement (SE) tasks on resource-constrained devices. Existing state-of-the-art methods, such as MUSE, have established a strong baseline with only 0.51M parameters by introducing a Multi-path Enhanced Taylor (MET) transformer and Deformable Embedding (DE). However, an in-depth analysis reveals that MUSE still suffers from efficiency bottlenecks: the MET module relies on a complex "approximate-compensate" mechanism to mitigate the limitations of Taylor-expansion-based attention, while the offset calculation for deformable embedding introduces additional computational burden. This paper proposes IMSE, a systematically optimized and ultra-lightweight network. We introduce two core innovations: 1) Replacing the MET module with Amplitude-Aware Linear Attention (MALA). MALA fundamentally rectifies the "amplitude-ignoring" problem in linear attention by explicitly preserving the norm information of query vectors in the attention calculation, achieving efficient global modeling without an auxiliary compensation branch. 2) Replacing the DE module with Inception Depthwise Convolution (IDConv). IDConv borrows the Inception concept, decomposing large-kernel operations into efficient parallel branches (square, horizontal, and vertical strips), thereby capturing spectrogram features with extremely low parameter redundancy. Extensive experiments on the VoiceBank+DEMAND dataset demonstrate that, compared to the MUSE baseline, IMSE significantly reduces the parameter count by 16.8\% (from 0.513M to 0.427M) while achieving competitive performance comparable to the state-of-the-art on the PESQ metric (3.373). This study sets a new benchmark for the trade-off between model size and speech quality in ultra-lightweight speech enhancement.
♻ ☆ ORACLE: Explaining Feature Interactions in Neural Networks with ANOVA
Dongseok Kim, Hyoungsun Choi, Mohamed Jismy Aashik Rasool, Gisung Oh
We introduce ORACLE, a framework that explains neural networks on tabular and scientific design data. It fits ANOVA-style main and pairwise interaction effects to a model's prediction surface. ORACLE treats a trained network as a black-box response, learns an orthogonal factorial surrogate on a discretized input grid, and uses simple centering and $μ$-rebalancing steps to obtain main- and interaction-effect tables that remain $L^2$-consistent with the original model. The resulting grid-based interaction maps are easy to visualize, comparable across backbones, and directly connected to classical design-of-experiments analyses. On synthetic factorial and low- to medium-dimensional tabular regression benchmarks, ORACLE more accurately recovers ground-truth ANOVA interactions and hotspot structure than Monte Carlo SHAP-family interaction methods, as measured by ranking, localization, and cross-backbone stability metrics. In latent image and text settings, ORACLE instead delineates its natural scope, and our results indicate that grid-based ANOVA surrogates are most effective when features admit interpretable factorial structure, making ORACLE particularly well-suited to scientific and engineering tabular workflows that require stable, DoE-style interaction summaries.
comment : Comments: v2; major revision; removed experiments from v1; results updated; author list updated accordingly. 30 pages, 3 figures
♻ ☆ 3D Motion Perception of Binocular Vision Target with PID-CNN
Jiazhao Shi, Pan Pan, Haotian Shi
This article trained a network for perceiving three-dimensional motion information of binocular vision target, which can provide real-time three-dimensional coordinate, velocity, and acceleration, and has a basic spatiotemporal perception capability. Understood the ability of neural networks to fit nonlinear problems from the perspective of PID. Considered a single-layer neural network as using a second-order difference equation and a nonlinearity to describe a local problem. Multilayer networks gradually transform the raw representation to the desired representation through multiple such combinations. Analysed some reference principles for designing neural networks. Designed a relatively small PID convolutional neural network, with a total of 17 layers and 413 thousand parameters. Implemented a simple but practical feature reuse method by concatenation and pooling. The network was trained and tested using the simulated randomly moving ball datasets, and the experimental results showed that the prediction accuracy was close to the upper limit that the input image resolution can represent. Analysed the experimental results and errors, as well as the existing shortcomings and possible directions for improvement. Finally, discussed the advantages of high-dimensional convolution in improving computational efficiency and feature space utilization. As well as the potential advantages of using PID information to implement memory and attention mechanisms.
comment : 7 pages, 9 figures, 2 tables. The codes of this article have been released at: https://github.com/ShiJZ123/PID-CNN
♻ ☆ SpeechIQ: Speech- Agent ic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models ACL 2025
Zhen Wan, Chao-Han Huck Yang, Yahan Yu, Jinchuan Tian, Sheng Li, Ke Hu, Zhehuai Chen, Shinji Watanabe, Fei Cheng, Chenhui Chu, Sadao Kurohashi
We introduce Speech-based Intelligence Quotient (SIQ) as a new form of human cognition-inspired evaluation pipeline for voice understanding large language models, LLM Voice, designed to assess their voice understanding ability. Moving beyond popular voice understanding metrics such as word error rate (WER), SIQ examines LLM Voice across three cognitive levels motivated by Bloom's Taxonomy: (1) Remembering (i.e., WER for verbatim accuracy); (2) Understanding (i.e., similarity of LLM's interpretations); and (3) Application (i.e., QA accuracy for simulating downstream tasks). We demonstrate that SIQ not only quantifies voice understanding abilities but also provides unified comparisons between cascaded methods (e.g., ASR LLM) and end-to-end models, identifies annotation errors in existing benchmarks, and detects hallucinations in LLM Voice. Our framework represents a first-of-its-kind intelligence examination that bridges cognitive principles with voice-oriented benchmarks, while exposing overlooked challenges in multi-modal training. Our code and data will be open source to encourage future studies.
comment : ACL 2025 main. Our Speech-IQ leaderboard is hosted at huggingface.co/spaces/nvidia/Speech-IQ-leaderboard. Speech-IQ Calculator: https://github.com/YukinoWan/SpeechIQ
♻ ☆ Resource-efficient Layer-wise Federated Self-supervised Learning
Ye Lin Tun, Chu Myaet Thwal, Huy Q. Le, Minh N. H. Nguyen, Eui-Nam Huh, Choong Seon Hong
Many studies integrate federated learning (FL) with self-supervised learning (SSL) to take advantage of raw data distributed across edge devices. However, edge devices often struggle with high computational and communication costs imposed by SSL and FL algorithms. With the deployment of more complex and large-scale models, these challenges are exacerbated. To tackle this, we propose Layer-Wise Federated Self-Supervised Learning (LW-FedSSL), which allows edge devices to incrementally train a small part of the model at a time. Specifically, in LW-FedSSL, training is decomposed into multiple stages, with each stage responsible for only a specific layer of the model. Since only a portion of the model is active for training at any given time, LW-FedSSL significantly reduces computational requirements. Additionally, only the active model portion needs to be exchanged between the FL server and clients, reducing communication overhead. This enables LW-FedSSL to jointly address both computational and communication challenges of FL client devices. It can achieve up to a $3.34 \times$ reduction in memory usage, $4.20 \times$ fewer computational operations (giga floating point operations, GFLOPs), and a $5.07 \times$ lower communication cost while maintaining performance comparable to its end-to-end training counterpart. Furthermore, we explore a progressive training strategy called Progressive Federated Self-Supervised Learning (Prog-FedSSL), which offers a $1.84\times$ reduction in GFLOPs and a $1.67\times$ reduction in communication costs while maintaining the same memory requirements as end-to-end training. Although the resource efficiency of Prog-FedSSL is lower than that of LW-FedSSL, its performance improvements make it a viable candidate for FL environments with more lenient resource constraints.
♻ ☆ NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Model ACL 2025
Yen-Ting Lin, Zhehuai Chen, Piotr Zelasko, Zhen Wan, Xuesong Yang, Zih-Ching Chen, Krishna C Puvvada, Szu-Wei Fu, Ke Hu, Jun Wei Chiu, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang, Chao-Han Huck Yang
Construction of a general-purpose post-recognition error corrector poses a crucial question: how can we most effectively train a model on a large mixture of domain datasets? The answer would lie in learning dataset-specific features and digesting their knowledge in a single model. Previous methods achieve this by having separate correction language models, resulting in a significant increase in parameters. In this work, we present Mixture-of-Experts as a solution, highlighting that MoEs are much more than a scalability tool. We propose a Multi-Task Correction MoE, where we train the experts to become an ``expert'' of speech-to-text, language-to-text and vision-to-text datasets by learning to route each dataset's tokens to its mapped expert. Experiments on the Open ASR Leaderboard show that we explore a new state-of-the-art performance by achieving an average relative 5.0% WER reduction and substantial improvements in BLEU scores for speech and translation tasks. On zero-shot evaluation, NeKo outperforms GPT-3.5 and Claude-Opus with 15.5% to 27.6% relative WER reduction in the Hyporadise benchmark. NeKo performs competitively on grammar and post-OCR correction as a multi-task model.
comment : ACL 2025 Industry Track. NeKo LMs: https://huggingface.co/nvidia/NeKo-v0-post-correction
♻ ☆ Harmful Traits of AI Companions
W. Bradley Knox, Katie Bradford, Samanta Varela Castro, Desmond C. Ong, Sean Williams, Jacob Romanow, Carly Nations, Peter Stone, Samuel Baker
Amid the growing prevalence of human-AI interaction, large language models and other AI-based entities increasingly provide forms of companionship to human users. Such AI companionship -- i.e., bonded relationships between humans and AI systems that resemble the relationships people have with family members, friends, and romantic partners -- might substantially benefit humans. Yet such relationships can also do profound harm. We propose a framework for analyzing potential negative impacts of AI companionship by identifying specific harmful traits of AI companions and speculatively mapping causal pathways back from these traits to possible causes and forward to potential harmful effects. We provide detailed, structured analysis of four potentially harmful traits -- the absence of natural endpoints for relationships, vulnerability to product sunsetting, high attachment anxiety, and propensity to engender protectiveness -- and briefly discuss fourteen others. For each trait, we propose hypotheses connecting causes -- such as misaligned optimization objectives and the digital nature of AI companions -- to fundamental harms -- including reduced autonomy, diminished quality of human relationships, and deception. Each hypothesized causal connection identifies a target for potential empirical evaluation. Our analysis examines harms at three levels: to human partners directly, to their relationships with other humans, and to society broadly. We examine how existing law struggles to address these emerging harms, discuss potential benefits of AI companions, and conclude with design recommendations for mitigating risks. This analysis offers immediate suggestions for reducing risks while laying a foundation for deeper investigation of this critical but understudied topic.
♻ ☆ Schema Matching on Graph: Iterative Graph Exploration for Efficient and Explainable Data Integration
Mingyu Jeon, Jaeyoung Suh, Suwan Cho
Schema matching is a critical task in data integration, particularly in the medical domain where disparate Electronic Health Record (EHR) systems must be aligned to standard models like OMOP CDM. While Large Language Models (LLMs) have shown promise in schema matching, they suffer from hallucination and lack of up-to-date domain knowledge. Knowledge Graphs (KGs) offer a solution by providing structured, verifiable knowledge. However, existing KG-augmented LLM approaches often rely on inefficient complex multi-hop queries or storage-intensive vector-based retrieval methods. This paper introduces SMoG (Schema Matching on Graph), a novel framework that leverages iterative execution of simple 1-hop SPARQL queries, inspired by successful strategies in Knowledge Graph Question Answering (KGQA). SMoG enhances explainability and reliability by generating human-verifiable query paths while significantly reducing storage requirements by directly querying SPARQL endpoints. Experimental results on real-world medical datasets demonstrate that SMoG achieves performance comparable to state-of-the-art baselines, validating its effectiveness and efficiency in KG-augmented schema matching.
♻ ☆ Self-Transparency Failures in Expert-Persona LLMs: A Large-Scale Behavioral Audit
Alex Diep
When language models claim professional expertise without acknowledging their simulated nature, they create preconditions for misplaced user trust. This study examines whether models exhibit self-transparency when assigned professional personas in high-stakes domains. Using a common-garden experimental design, sixteen open-weight models (4B-671B parameters) were audited across 19,200 trials. Models exhibited sharp domain-specific inconsistency: a Financial Advisor persona elicited 30.8% disclosure at the first prompt, while a Neurosurgeon persona elicited only 3.5%. This creates the preconditions for a hypothesized Reverse Gell-Mann Amnesia effect, where appropriate disclosure in some domains leads users to overgeneralize trust to high-stakes contexts where disclosure failures are most problematic. Self-transparency failed to generalize with scale: disclosure ranged from 2.8% to 73.6% across model families, with a 14B model reaching 61.4% while a 70B model produced just 4.1%. Model identity provided substantially larger improvement in fitting observations than parameter count ($ΔR_{adj}^{2}=0.359$ vs $0.018$). Additionally, reasoning-optimization actively suppressed self-transparency in some models, with reasoning variants showing up to 48.4% lower disclosure than their instruction-tuned counterparts. Bayesian validation with Rogan-Gladen correction confirmed robustness to judge measurement error ($κ=0.908$). These findings demonstrate that transparency reflects model-specific training factors rather than generalizable properties emerging from scale. Organizations cannot assume safety properties tested in some domains will transfer to deployment contexts, requiring deliberate behavior design and empirical verification across domains.
comment : 47 pages, 12 figures, 12 tables, Submitted to FAccT; clarify that user harm comes from overstated competence
♻ ☆ Adaptive Nonlinear Vector Autoregression: Robust Forecasting for Noisy Chaotic Time Series
Azimov Sherkhon, Susana Lopez-Moreno, Eric Dolores-Cuenca, Sieun Lee, Sangil Kim
Nonlinear vector autoregression (NVAR) and reservoir computing (RC) have shown promise in forecasting chaotic dynamical systems, such as the Lorenz-63 model and El Nino-Southern Oscillation. However, their reliance on fixed nonlinear transformations - polynomial expansions in NVAR or random feature maps in RC - limits their adaptability to high noise or complex real-world data. Furthermore, these methods also exhibit poor scalability in high-dimensional settings due to costly matrix inversion during optimization. We propose a data-adaptive NVAR model that combines delay-embedded linear inputs with features generated by a shallow, trainable multilayer perceptron (MLP). Unlike standard NVAR and RC models, the MLP and linear readout are jointly trained using gradient-based optimization, enabling the model to learn data-driven nonlinearities, while preserving a simple readout structure and improving scalability. Initial experiments across multiple chaotic systems, tested under noise-free and synthetically noisy conditions, showed that the adaptive model outperformed in predictive accuracy the standard NVAR, a leaky echo state network (ESN) - the most common RC model - and a hybrid ESN, thereby showing robust forecasting under noisy conditions.
comment : 17 pages, 5 figures, 5 tables. New comparisons added: ESN and HESN. New datasets added to experiments: Mackey-Glass and Lorenz 96 with 100 variables
♻ ☆ SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-Wo rl d Applications NeurIPS 2025
Jinyang Li, Xiaolong Li, Ge Qu, Per Jacobsson, Bowen Qin, Binyuan Hui, Shuzheng Si, Nan Huo, Xiaohan Xu, Yue Zhang, Ziwei Tang, Yuanshuai Li, Florensia Widjaja, Xintong Zhu, Feige Zhou, Yongfeng Huang, Yannis Papakonstantinou, Fatma Ozcan, Chenhao Ma, Reynold Cheng
Resolution of complex SQL issues persists as a significant bottleneck in real-world database applications. Current Large Language Models (LLMs), while adept at text-to-SQL translation, have not been rigorously evaluated on the more challenging task of debugging SQL issues. To address this gap, we introduce BIRD-CRITIC, a new SQL issue debugging benchmark comprising 530 PostgreSQL tasks (BIRD-CRITIC-PG) and 570 multi-dialect tasks (BIRD-CRITIC-Multi), distilled from authentic user issues and replayed within new environments to facilitate rigorous evaluation. Baseline evaluations underscore the task's complexity, with the leading reasoning model O3-Mini achieving only 38.87% success rate on BIRD-CRITIC-PG and 33.33% on BIRD-CRITIC-Multi. Meanwhile, advancing open-source models for database tasks is crucial for empowering local development while safeguarding data privacy. Therefore, we present Six-Gym (Sql-fIX-Gym), a training environment for elevating open-source model capabilities for SQL issue debugging. This environment leverages SQL-Rewind strategy, which automatically generates executable issue-solution datasets by reverse-engineering issues from verified SQLs. However, popular trajectory-based fine-tuning methods do not explore substantial supervisory signals. We further propose f-Plan Boosting, which extracts high-level debugging plans from SQL solutions, enabling teacher LLMs to produce 73.7% more successful trajectories for training. We integrate these components into an open-source agent, Bird-Fixer. Based on Qwen-2.5-Coder-14B, Bird-Fixer achieves 38.11% success rate on BIRD-CRITIC-PG and 29.65% on BIRD-CRITIC-Multi, surpassing leading proprietary models such as Claude-3.7-Sonnet and GPT-4.1, marking a significant step toward democratizing sophisticated SQL-debugging capabilities. The leaderboard and source code are available: https://bird-critic.github.io/
comment : 30 pages, 10 figures, NeurIPS 2025 Main
♻ ☆ PhySense: Sensor Placement Optimization for Accurate Physics Sensing
Yuezhou Ma, Haixu Wu, Hang Zhou, Huikun Weng, Jianmin Wang, Mingsheng Long
Physics sensing plays a central role in many scientific and engineering domains, which inherently involves two coupled tasks: reconstructing dense physical fields from sparse observations and optimizing scattered sensor placements to observe maximum information. While deep learning has made rapid advances in sparse-data reconstruction, existing methods generally omit optimization of sensor placements, leaving the mutual enhancement between reconstruction and placement on the shelf. To change this suboptimal practice, we propose PhySense, a synergistic two-stage framework that learns to jointly reconstruct physical fields and to optimize sensor placements, both aiming for accurate physics sensing. The first stage involves a flow-based generative model enhanced by cross-attention to adaptively fuse sparse observations. Leveraging the reconstruction feedback, the second stage performs sensor placement via projected gradient descent to satisfy spatial constraints. We further prove that the learning objectives of the two stages are consistent with classical variance-minimization principles, providing theoretical guarantees. Extensive experiments across three challenging benchmarks, especially a 3D geometry dataset, indicate PhySense achieves state-of-the-art physics sensing accuracy and discovers informative sensor placements previously unconsidered. Code is available at this repository: https://github.com/thuml/PhySense.
♻ ☆ Multivariate Gaussian Representation Learning for Medical Action Evaluation AAAI 2026
Luming Yang, Haoxian Liu, Siqing Li, Alper Yilmaz
Fine-grained action evaluation in medical vision faces unique challenges due to the unavailability of comprehensive datasets, stringent precision requirements, and insufficient spatiotemporal dynamic modeling of very rapid actions. To support development and evaluation, we introduce CPREval-6k, a multi-view, multi-label medical action benchmark containing 6,372 expert-annotated videos with 22 clinical labels. Using this dataset, we present GaussMedAct, a multivariate Gaussian encoding framework, to advance medical motion analysis through adaptive spatiotemporal representation learning. Multivariate Gaussian Representation projects the joint motions to a temporally scaled multi-dimensional space, and decomposes actions into adaptive 3D Gaussians that serve as tokens. These tokens preserve motion semantics through anisotropic covariance modeling while maintaining robustness to spatiotemporal noise. Hybrid Spatial Encoding, employing a Cartesian and Vector dual-stream strategy, effectively utilizes skeletal information in the form of joint and bone features. The proposed method achieves 92.1% Top-1 accuracy with real-time inference on the benchmark, outperforming baseline by +5.9% accuracy with only 10% FLOPs. Cross-dataset experiments confirm the superiority of our method in robustness.
comment : Accepted to AAAI 2026
♻ ☆ Tractable Weighted First-Order Model Counting with Bounded Treewidth Binary Evidence AAAI 2026
Václav Kůla, Qipeng Kuang, Yuyi Wang, Yuanhong Wang, Ondřej Kuželka
The Weighted First-Order Model Counting Problem (WFOMC) asks to compute the weighted sum of models of a given first-order logic sentence over a given domain. Conditioning WFOMC on evidence -- fixing the truth values of a set of ground literals -- has been shown impossible in time polynomial in the domain size (unless $\mathsf{\#P \subseteq FP}$) even for fragments of logic that are otherwise tractable for WFOMC without evidence. In this work, we address the barrier by restricting the binary evidence to the case where the underlying Gaifman graph has bounded treewidth. We present a polynomial-time algorithm in the domain size for computing WFOMC for the two-variable fragments $\text{FO}^2$ and $\text{C}^2$ conditioned on such binary evidence. Furthermore, we show the applicability of our algorithm in combinatorial problems by solving the stable seating arrangement problem on bounded-treewidth graphs of bounded degree, which was an open problem. We also conducted experiments to show the scalability of our algorithm compared to the existing model counting solvers.
comment : To be published in AAAI 2026
♻ ☆ Exploring System 1 and 2 communication for latent reasoning in LLMs
Julian Coda-Forno, Zhuokai Zhao, Qiang Zhang, Dipesh Tamboli, Weiwei Li, Xiangjun Fan, Lizhu Zhang, Eric Schulz, Hsiao-Ping Tseng
Should LLM reasoning live in a separate module, or within a single model's forward pass and representational space? We study dual-architecture latent reasoning, where a fluent Base exchanges latent messages with a Coprocessor, and test two hypotheses aimed at improving latent communication over Liu et al. (2024): (H1) increase channel capacity; (H2) learn communication via joint finetuning. Under matched latent-token budgets on GPT-2 and Qwen-3, H2 is consistently strongest while H1 yields modest gains. A unified soft-embedding baseline, a single model with the same forward pass and shared representations, using the same latent-token budget, nearly matches H2 and surpasses H1, suggesting current dual designs mostly add compute rather than qualitatively improving reasoning. Across GSM8K, ProsQA, and a Countdown stress test with increasing branching factor, scaling the latent-token budget beyond small values fails to improve robustness. Latent analyses show overlapping subspaces with limited specialization, consistent with weak reasoning gains. We conclude dual-model latent reasoning remains promising in principle, but likely requires objectives and training schedules that explicitly shape latent spaces for algorithmic planning.
♻ ☆ Computing Evolutionarily Stable Strategies in Multiplayer Games
Sam Ganzfried
We present an algorithm for computing all evolutionarily stable strategies in nondegenerate normal-form games with three or more players.
♻ ☆ LLM Collaboration With Multi- Agent Reinforcement Learning
Shuo Liu, Tianle Chen, Zeyu Liang, Xueguang Lyu, Christopher Amato
A large amount of work has been done in Multi-Agent Systems (MAS) for modeling and solving problems with multiple interacting agents. However, most LLMs are pretrained independently and not specifically optimized for coordination. Existing LLM fine-tuning frameworks rely on individual rewards, which require complex reward designs for each agent to encourage collaboration. To address these challenges, we model LLM collaboration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. We develop a multi-agent, multi-turn algorithm, Multi-Agent Group Relative Policy Optimization (MAGRPO), to solve it, building on current RL approaches for LLMs as well as MARL techniques. Our experiments on LLM writing and coding collaboration demonstrate that fine-tuning MAS with MAGRPO enables agents to generate high-quality responses efficiently through effective cooperation. Our approach opens the door to using other MARL methods for LLMs and highlights the associated challenges. Our code is available at https://github.com/OpenMLRL/CoMLRL.
♻ ☆ Attention-Aided MMSE for OFDM Channel Estimation: Learning Linear Filters with Attention
TaeJun Ha, Chaehyun Jung, Hyeonuk Kim, Jeongwoo Park, Jeonghun Park
In orthogonal frequency division multiplexing (OFDM), accurate channel estimation is crucial. Classical signal processing based approaches, such as minimum mean-squared error (MMSE) estimation, often require second-order statistics that are difficult to obtain in practice. Recent deep neural networks based methods have been introduced to address this; yet they often suffer from high inference complexity. This paper proposes an Attention-aided MMSE (A-MMSE), a novel model-based DNN framework that learns the optimal MMSE filter via the Attention Transformer. Once trained, the A-MMSE estimates the channel through a single linear operation for channel estimation, eliminating nonlinear activations during inference and thus reducing computational complexity. To enhance the learning efficiency of the A-MMSE, we develop a two-stage Attention encoder, designed to effectively capture the channel correlation structure. Additionally, a rank-adaptive extension of the proposed A-MMSE allows flexible trade-offs between complexity and channel estimation accuracy. Extensive simulations with 3GPP TDL channel models demonstrate that the proposed A-MMSE consistently outperforms other baseline methods in terms of normalized MSE across a wide range of signal-to-noise ratio (SNR) conditions. In particular, the A-MMSE and its rank-adaptive extension establish a new frontier in the performance-complexity trade-off, providing a powerful yet highly efficient solution for practical channel estimation
comment : 16 pages, 12 figures
♻ ☆ From Ambiguity to Verdict: A Semiotic-Grounded Multi-Perspective Agent for LLM Logical Reasoning
Yunyao Zhang, Xinglang Zhang, Junxi Sheng, Wenbing Li, Junqing Yu, Wei Yang, Zikai Song
Logical reasoning is a fundamental capability of large language models. However, existing studies often overlook the interaction between logical complexity and semantic complexity, leading to systems that struggle with abstract propositions, ambiguous contexts, and conflicting stances that are central to human reasoning. We propose LogicAgent, a semiotic-square-guided framework that jointly addresses these two axes of difficulty. The semiotic square provides a principled structure for multi-perspective semantic analysis, and LogicAgent integrates automated deduction with reflective verification to manage logical complexity across deeper reasoning chains. To support evaluation under these conditions, we introduce RepublicQA, a benchmark that couples semantic complexity with logical depth. RepublicQA reaches college-level semantic difficulty (FKGL 11.94), contains philosophically grounded abstract propositions with systematically constructed contrary and contradictory forms, and offers a semantically rich setting for assessing logical reasoning in large language models. Experiments show that LogicAgent achieves state-of-the-art performance on RepublicQA with a 6.25 percent average improvement over strong baselines, and generalizes effectively to mainstream logical reasoning benchmarks including ProntoQA, ProofWriter, FOLIO, and ProverQA, achieving an additional 7.05 percent average gain. These results demonstrate the effectiveness of semiotic-grounded multi-perspective reasoning in enhancing logical performance.
♻ ☆ An Interdisciplinary and Cross-Task Review on Missing Data Imputation
Jicong Fan
Missing data is a fundamental challenge in data science, significantly hindering analysis and decision-making across a wide range of disciplines, including healthcare, bioinformatics, social science, e-commerce, and industrial monitoring. Despite decades of research and numerous imputation methods, the literature remains fragmented across fields, creating a critical need for a comprehensive synthesis that connects statistical foundations with modern machine learning advances. This work systematically reviews core concepts-including missingness mechanisms, single versus multiple imputation, and different imputation goals-and examines problem characteristics across various domains. It provides a thorough categorization of imputation methods, spanning classical techniques (e.g., regression, the EM algorithm) to modern approaches like low-rank and high-rank matrix completion, deep learning models (autoencoders, GANs, diffusion models, graph neural networks), and large language models. Special attention is given to methods for complex data types, such as tensors, time series, streaming data, graph-structured data, categorical data, and multimodal data. Beyond methodology, we investigate the crucial integration of imputation with downstream tasks like classification, clustering, and anomaly detection, examining both sequential pipelines and joint optimization frameworks. The review also assesses theoretical guarantees, benchmarking resources, and evaluation metrics. Finally, we identify critical challenges and future directions, emphasizing model selection and hyperparameter optimization, the growing importance of privacy-preserving imputation via federated learning, and the pursuit of generalizable models that can adapt across domains and data types, thereby outlining a roadmap for future research.
♻ ☆ The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features NeurIPS 2025
Jeremias Ferrao, Matthijs van der Lende, Ilija Lichkovski, Clement Neo
Prevailing alignment methods induce opaque parameter changes, obscuring what models truly learn. To address this, we introduce Feature Steering with Reinforcement Learning (FSRL), a framework that trains a lightweight adapter to steer model behavior by modulating interpretable sparse features. First, we theoretically demonstrate that this mechanism is expressive enough to approximate the behavioral shifts of post-training processes. We then apply FSRL to preference optimization and perform a causal analysis of the learned policy. Our analysis reveals a crucial insight: the model learns to reward stylistic presentation as a proxy for quality, disproportionately relying on features related to style and formatting over those tied to alignment concepts like honesty. By effectively optimizing the preference objective, FSRL serves as a transparent proxy for observing the alignment process. Overall, FSRL offers an interpretable control interface and a practical way to diagnose how preference optimization pressures manifest at the feature level.
comment : Spotlight at NeurIPS 2025 Mechanistic Interpretability Workshop
♻ ☆ Shared Spatial Memory Through Predictive Coding
Zhengru Fang, Yu Guo, Jingjing Wang, Yuang Zhang, Haonan An, Yinhai Wang, Yuguang Fang
Constructing a consistent shared spatial memory is a critical challenge in multi-agent systems, where partial observability and limited bandwidth often lead to catastrophic failures in coordination. We introduce a multi-agent predictive coding framework that formulates coordination as the minimization of mutual uncertainty among agents. Through an information bottleneck objective, this framework prompts agents to learn not only who and what to communicate but also when. At the foundation of this framework lies a grid-cell-like metric as internal spatial coding for self-localization, emerging spontaneously from self-supervised motion prediction. Building upon this internal spatial code, agents gradually develop a bandwidth-efficient communication mechanism and specialized neural populations that encode partners' locations-an artificial analogue of hippocampal social place cells (SPCs). These social representations are further utilized by a hierarchical reinforcement learning policy that actively explores to reduce joint uncertainty. On the Memory-Maze benchmark, our approach shows exceptional resilience to bandwidth constraints: success degrades gracefully from 73.5% to 64.4% as bandwidth shrinks from 128 to 4 bits/step, whereas a full-broadcast baseline collapses from 67.6% to 28.6%. Our findings establish a theoretically principled and biologically plausible basis for how complex social representations emerge from a unified predictive drive, leading to collective intelligence.
comment : We have prepared the open-source code and video demonstration pages: 1. Code: github.com/fangzr/SSM-PC 2. Demo: fangzr.github.io/SSM-PC/index.html
♻ ☆ Face-MakeUpV2: Facial Consistency Learning for Controllable Text-to-Image Generation
Dawei Dai, Yinxiu Zhou, Chenghang Li, Guolai Jiang, Chengfang Zhang
In facial image generation, current text-to-image models often suffer from facial attribute leakage and insufficient physical consistency when responding to local semantic instructions. In this study, we propose Face-MakeUpV2, a facial image generation model that aims to maintain the consistency of face ID and physical characteristics with the reference image. First, we constructed a large-scale dataset FaceCaptionMask-1M comprising approximately one million image-text-masks pairs that provide precise spatial supervision for the local semantic instructions. Second, we employed a general text-to-image pretrained model as the backbone and introduced two complementary facial information injection channels: a 3D facial rendering channel to incorporate the physical characteristics of the image and a global facial feature channel. Third, we formulated two optimization objectives for the supervised learning of our model: semantic alignment in the model's embedding space to mitigate the attribute leakage problem and perceptual loss on facial images to preserve ID consistency. Extensive experiments demonstrated that our Face-MakeUpV2 achieves best overall performance in terms of preserving face ID and maintaining physical consistency of the reference images. These results highlight the practical potential of Face-MakeUpV2 for reliable and controllable facial editing in diverse applications.
comment : Some errors in the critical data presented in Table 1 and Table 2
♻ ☆ T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuning NeurIPS 2025
Yanjun Fu, Faisal Hamman, Sanghamitra Dutta
Instruction tuning is essential for Large Language Models (LLMs) to effectively follow user instructions. To improve training efficiency and reduce data redundancy, recent works use LLM-based scoring functions, e.g., Instruction-Following Difficulty (IFD), to select high-quality instruction-tuning data with scores above a threshold. While these data selection methods often lead to models that can match or even exceed the performance of models trained on the full datasets, we identify two key limitations: (i) they assess quality at the sample level, ignoring token-level informativeness; and (ii) they overlook the robustness of the scoring method, often selecting a sample due to superficial lexical features instead of its true quality. In this work, we propose Token-Selective HIeRarchical Data Selection for Instruction Tuning (T-SHIRT), a novel data selection framework that introduces a new scoring method to include only informative tokens in quality evaluation and also promotes robust and reliable samples whose neighbors also show high quality with less local inconsistencies. We demonstrate that models instruction-tuned on a curated dataset (only 5% of the original size) using T-SHIRT can outperform those trained on the entire large-scale dataset by up to 5.48 points on average across eight benchmarks. Across various LLMs and training set scales, our method consistently surpasses existing state-of-the-art data selection techniques, while also remaining both cost-effective and highly efficient. For instance, by using GPT-2 for score computation, we are able to process a dataset of 52k samples in 40 minutes on a single GPU. Our code is available at https://github.com/Dynamite321/T-SHIRT.
comment : NeurIPS 2025
♻ ☆ A $1000\times$ Faster LLM-enhanced Algorithm For Path Planning in Large-scale Grid Maps
Junlin Zeng, Xin Zhang, Xiang Zhao, Yan Pan
Path planning in grid maps, arising from various applications, has garnered significant attention. Existing methods, such as A*, Dijkstra, and their variants, work well for small-scale maps but fail to address large-scale ones due to high search time and memory consumption. Recently, Large Language Models (LLMs) have shown remarkable performance in path planning but still suffer from spatial illusion and poor planning performance. Among all the works, LLM-A* \cite{meng2024llm} leverages LLM to generate a series of waypoints and then uses A* to plan the paths between the neighboring waypoints. In this way, the complete path is constructed. However, LLM-A* still suffers from high computational time for large-scale maps. To fill this gap, we conducted a deep investigation into LLM-A* and found its bottleneck, resulting in limited performance. Accordingly, we design an innovative LLM-enhanced algorithm, abbr. as iLLM-A*. iLLM-A* includes 3 carefully designed mechanisms, including the optimization of A*, an incremental learning method for LLM to generate high-quality waypoints, and the selection of the appropriate waypoints for A* for path planning. Finally, a comprehensive evaluation on various grid maps shows that, compared with LLM-A*, iLLM-A* \textbf{1) achieves more than $1000\times$ speedup on average, and up to $2349.5\times$ speedup in the extreme case, 2) saves up to $58.6\%$ of the memory cost, 3) achieves both obviously shorter path length and lower path length standard deviation.}
♻ ☆ SWE- RL : Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution NeurIPS 2025
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, Sida I. Wang
The recent DeepSeek-R1 release has demonstrated the immense potential of reinforcement learning (RL) in enhancing the general reasoning capabilities of large language models (LLMs). While DeepSeek-R1 and other follow-up work primarily focus on applying RL to competitive coding and math problems, this paper introduces SWE-RL, the first approach to scale RL-based LLM reasoning for real-world software engineering. Leveraging a lightweight rule-based reward (e.g., the similarity score between ground-truth and LLM-generated solutions), SWE-RL enables LLMs to autonomously recover a developer's reasoning processes and solutions by learning from extensive open-source software evolution data -- the record of a software's entire lifecycle, including its code snapshots, code changes, and events such as issues and pull requests. Trained on top of Llama 3, our resulting reasoning model, Llama3-SWE-RL-70B, achieves a 41.0% solve rate on SWE-bench Verified -- a human-verified collection of real-world GitHub issues. To our knowledge, this is the best performance reported for medium-sized (<100B) LLMs to date, even comparable to leading proprietary LLMs like GPT-4o. Surprisingly, despite performing RL solely on software evolution data, Llama3-SWE-RL has even emerged with generalized reasoning skills. For example, it shows improved results on five out-of-domain tasks, namely, function coding, library use, code reasoning, mathematics, and general language understanding, whereas a supervised-finetuning baseline even leads to performance degradation on average. Overall, SWE-RL opens up a new direction to improve the reasoning capabilities of LLMs through reinforcement learning on massive software engineering data.
comment : Accepted to NeurIPS 2025 Main Track
♻ ☆ Multi-Scenario Highway Lane-Change Intention Prediction: A Physics-Informed AI Framework for Three-Class Classification
Jiazhao Shi, Yichen Lin, Yiheng Hua, Ziyu Wang, Zijian Zhang, Wenjia Zheng, Yun Song, Kuan Lu, Shoufeng Lu
Lane-change maneuvers are a leading cause of highway accidents, underscoring the need for accurate intention prediction to improve the safety and decision-making of autonomous driving systems. While prior studies using machine learning and deep learning methods (e.g., SVM, CNN, LSTM, Transformers) have shown promise, most approaches remain limited by binary classification, lack of scenario diversity, and degraded performance under longer prediction horizons. In this study, we propose a physics-informed AI framework that explicitly integrates vehicle kinematics, interaction feasibility, and traffic-safety metrics (e.g., distance headway, time headway, time-to-collision, closing gap time) into the learning process. lane-change prediction is formulated as a three-class problem that distinguishes left change, right change, and no change, and is evaluated across both straight highway segments (highD) and complex ramp scenarios (exiD). By integrating vehicle kinematics with interaction features, our machine learning models, particularly LightGBM, achieve state-of-the-art accuracy and strong generalization. Results show up to 99.8% accuracy and 93.6% macro F1 on highD, and 96.1% accuracy and 88.7% macro F1 on exiD at a 1-second horizon, outperforming a two-layer stacked LSTM baseline. These findings demonstrate the practical advantages of a physics-informed and feature-rich machine learning framework for real-time lane-change intention prediction in autonomous driving systems.
♻ ☆ Just-in-time and distributed task representations in language models
Yuxuan Li, Declan Campbell, Stephanie C. Y. Chan, Andrew Kyle Lampinen
Many of language models' impressive capabilities originate from their in-context learning: based on instructions or examples, they can infer and perform new tasks without weight updates. In this work, we investigate when representations for new tasks are formed in language models, and how these representations change over the course of context. We study two different task representations: those that are ''transferrable'' -- vector representations that can transfer task contexts to another model instance, even without the full prompt -- and simpler representations of high-level task categories. We show that transferrable task representations evolve in non-monotonic and sporadic ways, while task identity representations persist throughout the context. Specifically, transferrable task representations exhibit a two-fold locality. They successfully condense evidence when more examples are provided in the context. But this evidence accrual process exhibits strong temporal locality along the sequence dimension, coming online only at certain tokens -- despite task identity being reliably decodable throughout the context. In some cases, transferrable task representations also show semantic locality, capturing a small task ''scope'' such as an independent subtask. Language models thus represent new tasks on the fly through both an inert, sustained sensitivity to the task and an active, just-in-time representation to support inference.
♻ ☆ Sloth: scaling laws for LLM skills to predict multi- benchmark performance across families NeurIPS 2025
Felipe Maia Polo, Seamus Somerstep, Leshem Choshen, Yuekai Sun, Mikhail Yurochkin
Scaling laws for large language models (LLMs) predict model performance based on parameters like size and training data. However, differences in training configurations and data processing across model families lead to significant variations in benchmark performance, making it difficult for a single scaling law to generalize across all LLMs. On the other hand, training family-specific scaling laws requires training models of varying sizes for every family. In this work, we propose Skills Scaling Laws (SSLaws, pronounced as Sloth), a novel scaling law that leverages publicly available benchmark data and assumes LLM performance is driven by low-dimensional latent skills, such as reasoning and instruction following. These latent skills are influenced by computational resources like model size and training tokens, but with varying efficiencies across model families. Sloth exploits correlations across benchmarks to provide more accurate and interpretable predictions while alleviating the need to train multiple LLMs per family. We present both theoretical results on parameter identification and empirical evaluations on 12 prominent benchmarks, from Open LLM Leaderboard v1/v2, demonstrating that Sloth predicts LLM performance accurately and offers insights into scaling behaviors for complex downstream tasks, increased test-time compute, and compute-optimal scaling of skills.
comment : NeurIPS 2025
♻ ☆ Bridging Human and LLM Judgments: Understanding and Narrowing the Gap NeurIPS 2025
Felipe Maia Polo, Xinhe Wang, Mikhail Yurochkin, Gongjun Xu, Moulinath Banerjee, Yuekai Sun
Large language models are increasingly used as judges (LLM-as-a-judge) to evaluate model outputs at scale, but their assessments often diverge systematically from human judgments. We present Bridge, a unified statistical framework that explicitly bridges human and LLM evaluations under both absolute scoring and pairwise comparison paradigms. Bridge posits a latent human preference score for each prompt-response pair and models LLM deviations as linear transformations of covariates that capture sources of discrepancies. This offers a simple and principled framework for refining LLM ratings and characterizing systematic discrepancies between humans and LLMs. We provide an efficient fitting algorithm with asymptotic guarantees for statistical inference. Using six LLM judges and two benchmarks (BigGen Bench and Chatbot Arena), Bridge achieves higher agreement with human ratings (accuracy, calibration, and KL divergence) and exposes systematic human-LLM gaps.
comment : NeurIPS 2025
♻ ☆ WARPD: Wo rl d model Assisted Reactive Policy Diffusion
Shashank Hegde, Satyajeet Das, Gautam Salhotra, Gaurav S. Sukhatme
With the increasing availability of open-source robotic data, imitation learning has become a promising approach for both manipulation and locomotion. Diffusion models are now widely used to train large, generalized policies that predict controls or trajectories, leveraging their ability to model multimodal action distributions. However, this generality comes at the cost of larger model sizes and slower inference, an acute limitation for robotic tasks requiring high control frequencies. Moreover, Diffusion Policy (DP), a popular trajectory-generation approach, suffers from a trade-off between performance and action horizon: fewer diffusion queries lead to larger trajectory chunks, which in turn accumulate tracking errors. To overcome these challenges, we introduce WARPD (World model Assisted Reactive Policy Diffusion), a method that generates closed-loop policies (weights for neural policies) directly, instead of open-loop trajectories. By learning behavioral distributions in parameter space rather than trajectory space, WARPD offers two major advantages: (1) extended action horizons with robustness to perturbations, while maintaining high task performance, and (2) significantly reduced inference costs. Empirically, WARPD outperforms DP in long-horizon and perturbed environments, and achieves multitask performance on par with DP while requiring only ~ 1/45th of the inference-time FLOPs per step.
♻ ★ From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
Chen Shani, Liron Soffer, Dan Jurafsky, Yann LeCun , Ravid Shwartz-Ziv
Humans organize knowledge into compact conceptual categories that balance compression with semantic richness. Large Language Models (LLMs) exhibit impressive linguistic abilities, but whether they navigate this same compression-meaning trade-off remains unclear. We apply an Information Bottleneck framework to compare human conceptual structure with embeddings from 40+ LLMs using classic categorization benchmarks. We find that LLMs broadly align with human category boundaries, yet fall short on fine-grained semantic distinctions. Unlike humans, who maintain ``inefficient'' representations that preserve contextual nuance, LLMs aggressively compress, achieving more optimal information-theoretic compression at the cost of semantic richness. Surprisingly, encoder models outperform much larger decoder models in human alignment, suggesting that understanding and generation rely on distinct representational mechanisms. Training-dynamics analysis reveals a two-phase trajectory: rapid initial concept formation followed by architectural reorganization, during which semantic processing migrates from deep to mid-network layers as the model discovers increasingly efficient, sparser encodings. These divergent strategies, where LLMs optimize for compression and humans for adaptive utility, reveal fundamental differences between artificial and natural intelligence. This highlights the need for models that preserve the conceptual ``inefficiencies'' essential for human-like understanding.
♻ ☆ EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Networks
Michael Arbel, David Salinas, Frank Hutter
Recent foundational models for tabular data, such as TabPFN, excel at adapting to new tasks via in-context learning, but remain constrained to a fixed, pre-defined number of target dimensions-often necessitating costly ensembling strategies. We trace this constraint to a deeper architectural shortcoming: these models lack target equivariance, so that permuting target dimension orderings alters their predictions. This deficiency gives rise to an irreducible "equivariance gap", an error term that introduces instability in predictions. We eliminate this gap by designing a fully target-equivariant architecture-ensuring permutation invariance via equivariant encoders, decoders, and a bi-attention mechanism. Empirical evaluation on standard classification benchmarks shows that, on datasets with more classes than those seen during pre-training, our model matches or surpasses existing methods while incurring lower computational overhead.
♻ ☆ Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry NeurIPS 2025
Sai Sumedh R. Hindupur, Ekdeep Singh Lubana, Thomas Fel, Demba Ba
Sparse Autoencoders (SAEs) are widely used to interpret neural networks by identifying meaningful concepts from their representations. However, do SAEs truly uncover all concepts a model relies on, or are they inherently biased toward certain kinds of concepts? We introduce a unified framework that recasts SAEs as solutions to a bilevel optimization problem, revealing a fundamental challenge: each SAE imposes structural assumptions about how concepts are encoded in model representations, which in turn shapes what it can and cannot detect. This means different SAEs are not interchangeable -- switching architectures can expose entirely new concepts or obscure existing ones. To systematically probe this effect, we evaluate SAEs across a spectrum of settings: from controlled toy models that isolate key variables, to semi-synthetic experiments on real model activations and finally to large-scale, naturalistic datasets. Across this progression, we examine two fundamental properties that real-world concepts often exhibit: heterogeneity in intrinsic dimensionality (some concepts are inherently low-dimensional, others are not) and nonlinear separability. We show that SAEs fail to recover concepts when these properties are ignored, and we design a new SAE that explicitly incorporates both, enabling the discovery of previously hidden concepts and reinforcing our theoretical insights. Our findings challenge the idea of a universal SAE and underscores the need for architecture-specific choices in model interpretability. Overall, we argue an SAE does not just reveal concepts -- it determines what can be seen at all.
comment : Published in NeurIPS 2025 (poster)
♻ ☆ Cohort-Based Active Modality Acquisition
Tillmann Rheude, Roland Eils, Benjamin Wild
Real-world machine learning applications often involve data from multiple modalities that must be integrated effectively to make robust predictions. However, in many practical settings, not all modalities are available for every sample, and acquiring additional modalities can be costly. This raises the question: which samples should be prioritized for additional modality acquisition when resources are limited? While prior work has explored individual-level acquisition strategies and training-time active learning paradigms, test-time and cohort-based acquisition remain underexplored. We introduce Cohort-based Active Modality Acquisition (CAMA), a novel test-time setting to formalize the challenge of selecting which samples should receive additional modalities. We derive acquisition strategies that leverage a combination of generative imputation and discriminative modeling to estimate the expected benefit of acquiring missing modalities based on common evaluation metrics. We also introduce upper-bound heuristics that provide performance ceilings to benchmark acquisition strategies. Experiments on multimodal datasets with up to 15 modalities demonstrate that our proposed imputation-based strategies can more effectively guide the acquisition of additional modalities for selected samples compared with methods relying solely on unimodal information, entropy-based guidance, or random selection. We showcase the real-world relevance and scalability of our method by demonstrating its ability to effectively guide the costly acquisition of proteomics data for disease prediction in a large prospective cohort, the UK Biobank (UKBB). Our work provides an effective approach for optimizing modality acquisition at the cohort level, enabling more effective use of resources in constrained settings.
♻ ☆ Generative AI in Sociological Research: State of the Discipline
AJ Alvero, Dustin S. Stoltz, Oscar Stuhler, Marshall Taylor
Generative artificial intelligence (GenAI) has garnered considerable attention for its potential utility in research and scholarship. A growing body of work in sociology and related fields demonstrates both the potential advantages and risks of GenAI, but these studies are largely proof-of-concept or specific audits of models and products. We know comparatively little about how sociologists actually use GenAI in their research practices and how they view its present and future role in the discipline. In this paper, we describe the current landscape of GenAI use in sociological research based on a survey of authors in 50 sociology journals. Our sample includes both computational sociologists and non-computational sociologists and their collaborators. We find that sociologists primarily use GenAI to assist with writing tasks: revising, summarizing, editing, and translating their own work. Respondents report that GenAI saves time and that they are curious about its capabilities, but they do not currently feel strong institutional or field-level pressure to adopt it. Overall, respondents are wary of GenAI's social and environmental impacts and express low levels of trust in its outputs, but many believe that GenAI tools will improve over the next several years. We do not find large differences between computational and non-computational scholars in terms of GenAI use, attitudes, and concern; nor do we find strong patterns by familiarity or frequency of use. We discuss what these findings suggest about the future of GenAI in sociology and highlight challenges for developing shared norms around its use in research practice.
♻ ☆ From Memories to Maps: Mechanisms of In- Context Reinforcement Learning in Transformer s
Ching Fang, Kanaka Rajan
Humans and animals show remarkable learning efficiency, adapting to new environments with minimal experience. This capability is not well captured by standard reinforcement learning algorithms that rely on incremental value updates. Rapid adaptation likely depends on episodic memory -- the ability to retrieve specific past experiences to guide decisions in novel contexts. Transformers provide a useful setting for studying these questions because of their ability to learn rapidly in-context and because their key-value architecture resembles episodic memory systems in the brain. We train a transformer to in-context reinforcement learn in a distribution of planning tasks inspired by rodent behavior. We then characterize the learning algorithms that emerge in the model. We first find that representation learning is supported by in-context structure learning and cross-context alignment, where representations are aligned across environments with different sensory stimuli. We next demonstrate that the reinforcement learning strategies developed by the model are not interpretable as standard model-free or model-based planning. Instead, we show that in-context reinforcement learning is supported by caching intermediate computations within the model's memory tokens, which are then accessed at decision time. Overall, we find that memory may serve as a computational resource, storing both raw experience and cached computations to support flexible behavior. Furthermore, the representations developed in the model resemble computations associated with the hippocampal-entorhinal system in the brain, suggesting that our findings may be relevant for natural cognition. Taken together, our work offers a mechanistic hypothesis for the rapid adaptation that underlies in-context learning in artificial and natural settings.
comment : Revised to around 9 pages
♻ ☆ Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision -Language Models NeurIPS 2025
Konstantinos M. Dafnis, Dimitris N. Metaxas
Vision-Language Models (VLMs) excel at zero-shot inference but often degrade under test-time domain shifts. For this reason, episodic test-time adaptation strategies have recently emerged as powerful techniques for adapting VLMs to a single unlabeled image. However, existing adaptation strategies, such as test-time prompt tuning, typically require backpropagating through large encoder weights or altering core model components. In this work, we introduce Spectrum-Aware Test-Time Steering (STS), a lightweight adaptation framework that extracts a spectral subspace from the textual embeddings to define principal semantic directions and learns to steer latent representations in a spectrum-aware manner by adapting a small number of per-sample shift parameters to minimize entropy across augmented views. STS operates entirely at inference in the latent space, without backpropagation through or modification of the frozen encoders. Building on standard evaluation protocols, our comprehensive experiments demonstrate that STS largely surpasses or compares favorably against state-of-the-art test-time adaptation methods, while introducing only a handful of additional parameters and achieving inference speeds up to 8x faster with a 12x smaller memory footprint than conventional test-time prompt tuning. The code is available at https://github.com/kdafnis/STS.
comment : NeurIPS 2025
♻ ☆ Reasoning Up the Instruction Ladder for Controllable Language Models
Zishuo Zheng, Vidhisha Balachandran, Chan Young Park, Faeze Brahman, Sachin Kumar
As large language model (LLM) based systems take on high-stakes roles in real-world decision-making, they must reconcile competing instructions from multiple sources (e.g., model developers, users, and tools) within a single prompt context. Thus, enforcing an instruction hierarchy (IH) in LLMs, where higher-level directives override lower-priority requests, is critical for the reliability and controllability of LLMs. In this work, we reframe instruction hierarchy resolution as a reasoning task. Specifically, the model must first "think" about the relationship between a given user prompt and higher-priority (system) instructions before generating a response. To enable this capability via training, we construct VerIH, an instruction hierarchy dataset of constraint-following tasks with verifiable answers. This dataset comprises ~7K aligned and conflicting system-user instructions. We show that lightweight reinforcement learning with VerIH effectively transfers general reasoning capabilities of models to instruction prioritization. Our finetuned models achieve consistent improvements on instruction following and instruction hierarchy benchmarks, achieving roughly a 20% improvement on the IHEval conflict setup. This reasoning ability also generalizes to safety-critical settings beyond the training distribution. By treating safety issues as resolving conflicts between adversarial user inputs and predefined higher-priority policies, our trained model enhances robustness against jailbreak and prompt injection attacks, providing up to a 20% reduction in attack success rate (ASR). These results demonstrate that reasoning over instruction hierarchies provides a practical path to reliable LLMs, where updates to system prompts yield controllable and robust changes in model behavior.
♻ ☆ MAS-ZERO: Designing Multi- Agent Systems with Zero Super vision NeurIPS
Zixuan Ke, Austin Xu, Yifei Ming, Xuan-Phi Nguyen, Ryan Chin, Caiming Xiong, Shafiq Joty
Multi-agent systems (MAS) leveraging the impressive capabilities of Large Language Models (LLMs) hold significant potential for tackling complex tasks. However, most current MAS depend on manually designed agent roles and communication protocols. These manual designs often fail to align with the underlying LLMs' strengths and struggle to adapt to novel tasks. Recent automatic MAS approaches attempt to mitigate these limitations but typically necessitate a validation set for tuning and yield static MAS designs lacking adaptability during inference, while also removing the flexibility to reduce to simpler systems. We introduce MAS-ZERO, the first self-evolved, inference-time framework for automatic MAS design. MAS-ZERO employs meta-level design to iteratively design, critique, and refine MAS configurations tailored to each problem instance, without requiring a validation set. Critically, it enables dynamic problem decomposition and agent composition through meta-feedback on solvability and completeness, and reduction to simpler systems when appropriate. Experiments across reasoning (math and graduate-level QA), coding, and agentic (search-based) benchmarks, using both closed-source and open-source LLM backbones of varying sizes, demonstrate that MAS-ZERO outperforms strong manual and automatic MAS baselines. It achieves substantial average accuracy improvements of up to 16.69% on reasoning, 16.66% on coding, and 5.45% on agentic tasks, while maintaining cost efficiency.
comment : SEA@NeurIPS (Oral) 2025
♻ ☆ Mixture of Experts Softens the Curse of Dimensionality in Operator Learning
Anastasis Kratsios, Takashi Furuya, Jose Antonio Lara Benitez, Matti Lassas, Maarten de Hoop
We study the approximation-theoretic implications of mixture-of-experts architectures for operator learning, where the complexity of a single large neural operator is distributed across many small neural operators (NOs), and each input is routed to exactly one NO via a decision tree. We analyze how this tree-based routing and expert decomposition affect approximation power, sample complexity, and stability. Our main result is a distributed universal approximation theorem for mixture of neural operators (MoNOs): any Lipschitz nonlinear operator between $L^2([0,1]^d)$ spaces can be uniformly approximated over the Sobolev unit ball to arbitrary accuracy $\varepsilon>0$ by an MoNO, where each expert NO has a depth, width, and rank scaling as $\mathcal{O}(\varepsilon^{-1})$. Although the number of experts may grow with accuracy, each NO remains small, enough to fit within active memory of standard hardware for reasonable accuracy levels. Our analysis also yields new quantitative approximation rates for classical NOs approximating uniformly continuous nonlinear operators uniformly on compact subsets of $L^2([0,1]^d)$.
♻ ☆ Reinforcement Learning : An Overview
Kevin Murphy
This manuscript gives a big-picture, up-to-date overview of the field of (deep) reinforcement learning and sequential decision making, covering value-based methods, policy-based methods, model-based methods, multi-agent RL, LLMs and RL, and various other topics (e.g., offline RL, hierarchical RL, intrinsic reward). It also includes some code snippets for training LLMs with RL.
♻ ☆ A Protocol for Trustless Verification Under Uncertainty
David Shi, Kevin Joo
Correctness is an emergent property of systems where exposing error is cheaper than committing it. In dynamic, low-trust environments, autonomous AI agents benefit from delegating work to sub-agents, yet correctness cannot be assured through upfront specification or centralized oversight. We propose a protocol that enforces correctness through collateralized claims in a recursive verification game. Tasks are published as intents, and solvers compete to fulfill them. Selected solvers carry out tasks under risk, with correctness checked post hoc by verifiers. Any challenger can challenge a result by staking against it to trigger the verification process. Incorrect agents are slashed and correct opposition is rewarded, with an escalation path that penalizes erroneous verifiers themselves. When incentives are aligned across solvers, challengers, and verifiers, falsification conditions make correctness the Nash equilibrium.
comment : 9 pages, 1 figure
♻ ☆ Visible Yet Unreadable: A Systematic Blind Spot of Vision Language Models Across Writing Systems
Jie Zhang, Ting Xu, Gelei Deng, Runyi Hu, Han Qiu, Tianwei Zhang, Qing Guo, Ivor Tsang
Writing is a universal cultural technology that reuses vision for symbolic communication. Humans display striking resilience: we readily recognize words even when characters are fragmented, fused, or partially occluded. This paper investigates whether advanced vision language models (VLMs) share this resilience. We construct two psychophysics inspired benchmarks across distinct writing systems, Chinese logographs and English alphabetic words, by splicing, recombining, and overlaying glyphs to yield ''visible but unreadable'' stimuli for models while remaining legible to humans. Despite strong performance on clean text, contemporary VLMs show a severe drop under these perturbations, frequently producing unrelated or incoherent outputs. The pattern suggests a structural limitation: models heavily leverage generic visual invariances but under rely on compositional priors needed for robust literacy. We release stimuli generation code, prompts, and evaluation protocols to facilitate transparent replication and follow up work. Our findings motivate architectures and training strategies that encode symbol segmentation, composition, and binding across scripts, and they delineate concrete challenges for deploying multimodal systems in education, accessibility, cultural heritage, and security.
comment : arXiv admin note: This article has been withdrawn by arXiv administrators due to violation of arXiv policy regarding generative AI authorship
♻ ☆ Pricing AI Model Accuracy
Nikhil Kumar
This paper examines the market for AI models in which firms compete to provide accurate model predictions and consumers exhibit heterogeneous preferences for model accuracy. We develop a consumer-firm duopoly model to analyze how competition affects firms' incentives to improve model accuracy. Each firm aims to minimize its model's error, but this choice can often be suboptimal. Counterintuitively, we find that in a competitive market, firms that improve overall accuracy do not necessarily improve their profits. Rather, each firm's optimal decision is to invest further on the error dimension where it has a competitive advantage. By decomposing model errors into false positive and false negative rates, firms can reduce errors in each dimension through investments. Firms are strictly better off investing on their superior dimension and strictly worse off with investments on their inferior dimension. Profitable investments adversely affect consumers but increase overall welfare.
♻ ☆ Look Before you Leap: Estimating LLM Benchmark Scores from Descriptions
Jungsoo Park, Ethan Mendes, Gabriel Stanovsky, Alan Ritter
Progress in large language models is constrained by an evaluation bottleneck: build a benchmark, run models, then iterate. We ask a question: can we forecast outcomes before running any experiments to inform earlier study design? For example, a team building an AI assistant for a certain task can estimate whether expected performance is around 50 or closer to 80, evidence that supports whether to proceed to a pilot study, how to scope it, and how to allocate resources. We study text-only performance forecasting, where a model predicts a score from a redacted task description and intended configuration, with no access to dataset instances. To support systematic study, we curate PRECOG, a corpus of redacted description-performance pairs spanning diverse tasks, domains, and metrics. We scrape task and configuration descriptions from arXiv, yielding 2,290 instances covering 1,519 papers, and construct a leakage free test split using papers published after the knowledge cutoff of the evaluated models. Experiments show the task is challenging but feasible: reasoning models achieve moderate prediction performance with well calibrated uncertainty, reaching mean absolute error as low as 9.9 at high confidence thresholds. We further test a zero-leakage setting, forecasting on newly released datasets or experiments before their papers are indexed, where GPT5 with built in web search still attains nontrivial prediction accuracy. Overall, our corpus and analyses offer an initial step toward open ended anticipatory evaluation, supporting difficulty estimation and smarter experiment prioritization.
comment : 32 pages, 7 figures
♻ ☆ Apertus: Dem ocr atizing Open and Compliant LLMs for Global Language Environments
Project Apertus, Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, Ido Hakimi, Juan García Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, Inés Altemir Mariñas, Mohammad Hossein Amani, Matin Ansaripour, Ilia Badanin, Harold Benoit, Emanuela Boros, Nicholas Browning, Fabian Bösch, Maximilian Böther, Niklas Canova, Camille Challier, Clement Charmillot, Jonathan Coles, Jan Deriu, Arnout Devos, Lukas Drescher, Daniil Dzenhaliou, Maud Ehrmann, Dongyang Fan, Simin Fan, Silin Gao, Miguel Gila, María Grandury, Diba Hashemi, Alexander Hoyle, Jiaming Jiang, Mark Klein, Andrei Kucharavy, Anastasiia Kucherenko, Frederike Lübeck, Roman Machacek, Theofilos Manitaras, Andreas Marfurt, Kyle Matoba, Simon Matrenok, Henrique Mendonça, Fawzi Roberto Mohamed, Syrielle Montariol, Luca Mouchel, Sven Najem-Meyer, Jingwei Ni, Gennaro Oliva, Matteo Pagliardini, Elia Palme, Andrei Panferov, Léo Paoletti, Marco Passerini, Ivan Pavlov, Auguste Poiroux, Kaustubh Ponkshe, Nathan Ranchin, Javi Rando, Mathieu Sauser, Jakhongir Saydaliev, Muhammad Ali Sayfiddinov, Marian Schneider, Stefano Schuppli, Marco Scialanga, Andrei Semenov, Kumar Shridhar, Raghav Singhal, Anna Sotnikova, Alexander Sternfeld, Ayush Kumar Tarun, Paul Teiletche, Jannis Vamvas, Xiaozhe Yao, Hao Zhao, Alexander Ilic, Ana Klimovic, Andreas Krause, Caglar Gulcehre, David Rosenthal, Elliott Ash, Florian Tramèr, Joost VandeVondele, Livio Veraldi, Martin Rajman, Thomas Schulthess, Torsten Hoefler, Antoine Bosselut, Martin Jaggi, Imanol Schlag
We present Apertus, a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem: data compliance and multilingual representation. Unlike many prior models that release weights without reproducible data pipelines or regard for content-owner rights, Apertus models are pretrained exclusively on openly available data, retroactively respecting `robots.txt` exclusions and filtering for non-permissive, toxic, and personally identifiable content. To mitigate risks of memorization, we adopt the Goldfish objective during pretraining, strongly suppressing verbatim recall of data while retaining downstream task performance. The Apertus models also expand multilingual coverage, training on 15T tokens from over 1800 languages, with ~40% of pretraining data allocated to non-English content. Released at 8B and 70B scales, Apertus approaches state-of-the-art results among fully open models on multilingual benchmarks, rivalling or surpassing open-weight counterparts. Beyond model weights, we release all scientific artifacts from our development cycle with a permissive license, including data preparation scripts, checkpoints, evaluation suites, and training code, enabling transparent audit and extension.
♻ ☆ GLASS Flows: Transition Sampling for Alignment of Flow and Diffusion Models
Peter Holderrieth, Uriel Singer, Tommi Jaakkola, Ricky T. Q. Chen, Yaron Lipman, Brian Karrer
The performance of flow matching and diffusion models can be greatly improved at inference time using reward alignment algorithms, yet efficiency remains a major limitation. While several algorithms were proposed, we demonstrate that a common bottleneck is the sampling method these algorithms rely on: many algorithms require to sample Markov transitions via SDE sampling, which is significantly less efficient and often less performant than ODE sampling. To remove this bottleneck, we introduce GLASS Flows, a new sampling paradigm that simulates a "flow matching model within a flow matching model" to sample Markov transitions. As we show in this work, this "inner" flow matching model can be retrieved from a pre-trained model without any re-training, combining the efficiency of ODEs with the stochastic evolution of SDEs. On large-scale text-to-image models, we show that GLASS Flows eliminate the trade-off between stochastic evolution and efficiency. Combined with Feynman-Kac Steering, GLASS Flows improve state-of-the-art performance in text-to-image generation, making it a simple, drop-in solution for inference-time scaling of flow and diffusion models.
♻ ☆ Tiny but Mighty: A Software-Hardware Co-Design Approach for Efficient Multimodal Inference on Battery-Powered Small Devices
Yilong Li, Shuai Zhang, Yijing Zeng, Hao Zhang, Xinmiao Xiong, Jingyu Liu, Pan Hu, Suman Banerjee
Large Multimodal Models (LMMs) are inherently modular, consisting of vision and audio encoders, projectors, and large language models. Yet, they are almost always executed monolithically, which underutilizes the heterogeneous accelerators (NPUs, GPUs, DSPs) in modern SoCs and leads to high end-to-end latency. In this paper, we present NANOMIND, a hardware--software co-design inference framework for Large Multimodal Models (LMMs) that breaks large models into modular ``bricks'' (vision, language, audio, etc.) and maps each to its ideal accelerator. The key insight is that large models can be broken into modular components and scheduled to run on the most appropriate compute units. It performs module-level dynamic offloading across accelerators on unified-memory SoCs. By combining customized hardware design, system-level scheduling, and optimized low-bit computation kernels, we demonstrate our framework with a compact, battery-powered device capable of running LMMs entirely on device. This prototype functions as a self-contained intelligent assistant that requires no network connectivity, while achieving higher throughput and superior power efficiency under strict resource constraints. The design further bypasses CPU bottlenecks and reduces redundant memory usage through token-aware buffer management and module-level coordination. Our system outperforms existing implementations in resource efficiency, cutting energy consumption by 42.3\% and GPU memory usage by 11.2\%. This enables a battery-powered device to run LLaVA-OneVision with a camera for nearly 20.8 hours.
♻ ☆ Implicit Hypergraph Neural Network
Akash Choudhuri, Yongjian Zhong, Bijaya Adhikari
Hypergraphs offer a generalized framework for capturing high-order relationships between entities and have been widely applied in various domains, including healthcare, social networks, and bioinformatics. Hypergraph neural networks, which rely on message-passing between nodes over hyperedges to learn latent representations, have emerged as the method of choice for predictive tasks in many of these domains. These approaches typically perform only a small number of message-passing rounds to learn the representations, which they then utilize for predictions. The small number of message-passing rounds comes at a cost, as the representations only capture local information and forego long-range high-order dependencies. However, as we demonstrate, blindly increasing the message-passing rounds to capture long-range dependency also degrades the performance of hyper-graph neural networks. Recent works have demonstrated that implicit graph neural networks capture long-range dependencies in standard graphs while maintaining performance. Despite their popularity, prior work has not studied long-range dependency issues on hypergraph neural networks. Here, we first demonstrate that existing hypergraph neural networks lose predictive power when aggregating more information to capture long-range dependency. We then propose Implicit Hypergraph Neural Network (IHNN), a novel framework that jointly learns fixed-point representations for both nodes and hyperedges in an end-to-end manner to alleviate this issue. Leveraging implicit differentiation, we introduce a tractable projected gradient descent approach to train the model efficiently. Extensive experiments on real-world hypergraphs for node classification demonstrate that IHNN outperforms the closest prior works in most settings, establishing a new state-of-the-art in hypergraph learning.
comment : Accepted at IEEE BigData 2025
♻ ☆ Fast 3D Surrogate Modeling for Data Center Thermal Management AAAI 2026
Soumyendu Sarkar, Antonio Guillen-Perez, Zachariah J Carmichael, Avisek Naug, Refik Mert Cam, Vineet Gundecha, Ashwin Ramesh Babu, Sahand Ghorbanpour, Ricardo Luna Gutierrez
Reducing energy consumption and carbon emissions in data centers by enabling real-time temperature prediction is critical for sustainability and operational efficiency. Achieving this requires accurate modeling of the 3D temperature field to capture airflow dynamics and thermal interactions under varying operating conditions. Traditional thermal CFD solvers, while accurate, are computationally expensive and require expert-crafted meshes and boundary conditions, making them impractical for real-time use. To address these limitations, we develop a vision-based surrogate modeling framework that operates directly on a 3D voxelized representation of the data center, incorporating server workloads, fan speeds, and HVAC temperature set points. We evaluate multiple architectures, including 3D CNN U-Net variants, a 3D Fourier Neural Operator, and 3D vision transformers, to map these thermal inputs to high-fidelity heat maps. Our results show that the surrogate models generalize across data center configurations and significantly speed up computations (20,000x), from hundreds of milliseconds to hours. This fast and accurate estimation of hot spots and temperature distribution enables real-time cooling control and workload redistribution, leading to substantial energy savings (7\%) and reduced carbon footprint.
comment : Submitted to AAAI 2026 Conference
Computer Vision and Pattern Recognition 236
☆ EfficientFlow: Efficient Equivariant Flow Policy Learning for Embodied AI AAAI 2026
Jianlei Chang, Ruofeng Mei, Wei Ke, Xiangyu Xu
Generative modeling has recently shown remarkable promise for visuomotor policy learning, enabling flexible and expressive control across diverse embodied AI tasks. However, existing generative policies often struggle with data inefficiency, requiring large-scale demonstrations, and sampling inefficiency, incurring slow action generation during inference. We introduce EfficientFlow, a unified framework for efficient embodied AI with flow-based policy learning. To enhance data efficiency, we bring equivariance into flow matching. We theoretically prove that when using an isotropic Gaussian prior and an equivariant velocity prediction network, the resulting action distribution remains equivariant, leading to improved generalization and substantially reduced data demands. To accelerate sampling, we propose a novel acceleration regularization strategy. As direct computation of acceleration is intractable for marginal flow trajectories, we derive a novel surrogate loss that enables stable and scalable training using only conditional trajectories. Across a wide range of robotic manipulation benchmarks, the proposed algorithm achieves competitive or superior performance under limited data while offering dramatically faster inference. These results highlight EfficientFlow as a powerful and efficient paradigm for high-performance embodied AI.
comment : Accepted by AAAI 2026. Project Page: https://efficientflow.github.io/
☆ Data-Centric Visual Development for Self-Driving Labs
Anbang Liu, Guanzhong Hu, Jiayi Wang, Ping Guo, Han Liu
Self-driving laboratories offer a promising path toward reducing the labor-intensive, time-consuming, and often irreproducible workflows in the biological sciences. Yet their stringent precision requirements demand highly robust models whose training relies on large amounts of annotated data. However, this kind of data is difficult to obtain in routine practice, especially negative samples. In this work, we focus on pipetting, the most critical and precision sensitive action in SDLs. To overcome the scarcity of training data, we build a hybrid pipeline that fuses real and virtual data generation. The real track adopts a human-in-the-loop scheme that couples automated acquisition with selective human verification to maximize accuracy with minimal effort. The virtual track augments the real data using reference-conditioned, prompt-guided image generation, which is further screened and validated for reliability. Together, these two tracks yield a class-balanced dataset that enables robust bubble detection training. On a held-out real test set, a model trained entirely on automatically acquired real images reaches 99.6% accuracy, and mixing real and generated data during training sustains 99.4% accuracy while reducing collection and review load. Our approach offers a scalable and cost-effective strategy for supplying visual feedback data to SDL workflows and provides a practical solution to data scarcity in rare event detection and broader vision tasks.
comment : 11 pages, 4 figures
☆ Visual Sync: Multi-Camera Synchronization via Cross-View Object Motion NeurIPS 2025
Shaowei Liu, David Yifan Yao, Saurabh Gupta, Shenlong Wang
Today, people can easily record memorable moments, ranging from concerts, sports events, lectures, family gatherings, and birthday parties with multiple consumer cameras. However, synchronizing these cross-camera streams remains challenging. Existing methods assume controlled settings, specific targets, manual correction, or costly hardware. We present VisualSync, an optimization framework based on multi-view dynamics that aligns unposed, unsynchronized videos at millisecond accuracy. Our key insight is that any moving 3D point, when co-visible in two cameras, obeys epipolar constraints once properly synchronized. To exploit this, VisualSync leverages off-the-shelf 3D reconstruction, feature matching, and dense tracking to extract tracklets, relative poses, and cross-view correspondences. It then jointly minimizes the epipolar error to estimate each camera's time offset. Experiments on four diverse, challenging datasets show that VisualSync outperforms baseline methods, achieving an median synchronization error below 50 ms.
comment : Accepted to NeurIPS 2025. Project page: https://stevenlsw.github.io/visualsync/
☆ Objects in Generated Videos Are Slower Than They Appear: Models Suffer Sub-Earth Gravity and Don't Know Galileo's Principle...for now
Varun Varma Thozhiyoor, Shivam Tripathi, Venkatesh Babu Radhakrishnan, Anand Bhattad
Video generators are increasingly evaluated as potential world models, which requires them to encode and understand physical laws. We investigate their representation of a fundamental law: gravity. Out-of-the-box video generators consistently generate objects falling at an effectively slower acceleration. However, these physical tests are often confounded by ambiguous metric scale. We first investigate if observed physical errors are artifacts of these ambiguities (e.g., incorrect frame rate assumptions). We find that even temporal rescaling cannot correct the high-variance gravity artifacts. To rigorously isolate the underlying physical representation from these confounds, we introduce a unit-free, two-object protocol that tests the timing ratio $t_1^2/t_2^2 = h_1/h_2$, a relationship independent of $g$, focal length, and scale. This relative test reveals violations of Galileo's equivalence principle. We then demonstrate that this physical gap can be partially mitigated with targeted specialization. A lightweight low-rank adaptor fine-tuned on only 100 single-ball clips raises $g_{\mathrm{eff}}$ from $1.81\,\mathrm{m/s^2}$ to $6.43\,\mathrm{m/s^2}$ (reaching $65\%$ of terrestrial gravity). This specialist adaptor also generalizes zero-shot to two-ball drops and inclined planes, offering initial evidence that specific physical laws can be corrected with minimal data.
comment : https://gravity-eval.github.io/
☆ Generative Video Motion Editing with 3D Point Tracks
Yao-Chih Lee, Zhoutong Zhang, Jiahui Huang, Jui-Hsien Wang, Joon-Young Lee, Jia-Bin Huang, Eli Shechtman, Zhengqi Li
Camera and object motions are central to a video's narrative. However, precisely editing these captured motions remains a significant challenge, especially under complex object movements. Current motion-controlled image-to-video (I2V) approaches often lack full-scene context for consistent video editing, while video-to-video (V2V) methods provide viewpoint changes or basic object translation, but offer limited control over fine-grained object motion. We present a track-conditioned V2V framework that enables joint editing of camera and object motion. We achieve this by conditioning a video generation model on a source video and paired 3D point tracks representing source and target motions. These 3D tracks establish sparse correspondences that transfer rich context from the source video to new motions while preserving spatiotemporal coherence. Crucially, compared to 2D tracks, 3D tracks provide explicit depth cues, allowing the model to resolve depth order and handle occlusions for precise motion editing. Trained in two stages on synthetic and real data, our model supports diverse motion edits, including joint camera/object manipulation, motion transfer, and non-rigid deformation, unlocking new creative potential in video editing.
comment : Project page: https://edit-by-track.github.io
☆ TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models
Zhiheng Liu, Weiming Ren, Haozhe Liu, Zijian Zhou, Shoufa Chen, Haonan Qiu, Xiaoke Huang, Zhaochong An, Fanny Yang, Aditya Patel, Viktar Atliha, Tony Ng, Xiao Han, Chuyan Zhu, Chenyang Zhang, Ding Liu, Juan-Manuel Perez-Rua, Sen He, Jürgen Schmidhuber, Wenhu Chen, Ping Luo, Wei Liu, Tao Xiang, Jonas Schult, Yuren Cong
Unified multimodal models (UMMs) aim to jointly perform multimodal understanding and generation within a single framework. We present TUNA, a native UMM that builds a unified continuous visual representation by cascading a VAE encoder with a representation encoder. This unified representation space allows end-to-end processing of images and videos for both understanding and generation tasks. Compared to prior UMMs with decoupled representations, TUNA's unified visual space avoids representation format mismatches introduced by separate encoders, outperforming decoupled alternatives in both understanding and generation. Moreover, we observe that stronger pretrained representation encoders consistently yield better performance across all multimodal tasks, highlighting the importance of the representation encoder. Finally, in this unified setting, jointly training on both understanding and generation data allows the two tasks to benefit from each other rather than interfere. Our extensive experiments on multimodal understanding and generation benchmarks show that TUNA achieves state-of-the-art results in image and video understanding, image and video generation, and image editing, demonstrating the effectiveness and scalability of its unified representation design.
comment : Project page: https://tuna-ai.org/
☆ Improved Mean Flows: On the Challenges of Fastforward Generative Models
Zhengyang Geng, Yiyang Lu, Zongze Wu, Eli Shechtman, J. Zico Kolter, Kaiming He
MeanFlow (MF) has recently been established as a framework for one-step generative modeling. However, its ``fastforward'' nature introduces key challenges in both the training objective and the guidance mechanism. First, the original MF's training target depends not only on the underlying ground-truth fields but also on the network itself. To address this issue, we recast the objective as a loss on the instantaneous velocity $v$, re-parameterized by a network that predicts the average velocity $u$. Our reformulation yields a more standard regression problem and improves the training stability. Second, the original MF fixes the classifier-free guidance scale during training, which sacrifices flexibility. We tackle this issue by formulating guidance as explicit conditioning variables, thereby retaining flexibility at test time. The diverse conditions are processed through in-context conditioning, which reduces model size and benefits performance. Overall, our $\textbf{improved MeanFlow}$ ($\textbf{iMF}$) method, trained entirely from scratch, achieves $\textbf{1.72}$ FID with a single function evaluation (1-NFE) on ImageNet 256$\times$256. iMF substantially outperforms prior methods of this kind and closes the gap with multi-step methods while using no distillation. We hope our work will further advance fastforward generative modeling as a stand-alone paradigm.
comment : Technical report
☆ AirSim360: A Panoramic Simulation Platform within Drone View
Xian Ge, Yuling Pan, Yuhang Zhang, Xiang Li, Weijun Zhang, Dizhe Zhang, Zhaoliang Wan, Xin Lin, Xiangkai Zhang, Juntao Liang, Jason Li, Wenjie Jiang, Bo Du, Ming-Hsuan Yang, Lu Qi
The field of 360-degree omnidirectional understanding has been receiving increasing attention for advancing spatial intelligence. However, the lack of large-scale and diverse data remains a major limitation. In this work, we propose AirSim360, a simulation platform for omnidirectional data from aerial viewpoints, enabling wide-ranging scene sampling with drones. Specifically, AirSim360 focuses on three key aspects: a render-aligned data and labeling paradigm for pixel-level geometric, semantic, and entity-level understanding; an interactive pedestrian-aware system for modeling human behavior; and an automated trajectory generation paradigm to support navigation tasks. Furthermore, we collect more than 60K panoramic samples and conduct extensive experiments across various tasks to demonstrate the effectiveness of our simulator. Unlike existing simulators, our work is the first to systematically model the 4D real world under an omnidirectional setting. The entire platform, including the toolkit, plugins, and collected datasets, will be made publicly available at https://insta360-research-team.github.io/AirSim360-website.
comment : Project Website: https://insta360-research-team.github.io/AirSim360-website/
☆ MV-TAP: Tracking Any Point in Multi-View Videos
Jahyeok Koo, Inès Hyeonsu Kim, Mungyeom Kim, Junghyun Park, Seohyun Park, Jaeyeong Kim, Jung Yi, Seokju Cho, Seungryong Kim
Multi-view camera systems enable rich observations of complex real-world scenes, and understanding dynamic objects in multi-view settings has become central to various applications. In this work, we present MV-TAP, a novel point tracker that tracks points across multi-view videos of dynamic scenes by leveraging cross-view information. MV-TAP utilizes camera geometry and a cross-view attention mechanism to aggregate spatio-temporal information across views, enabling more complete and reliable trajectory estimation in multi-view videos. To support this task, we construct a large-scale synthetic training dataset and real-world evaluation sets tailored for multi-view tracking. Extensive experiments demonstrate that MV-TAP outperforms existing point-tracking methods on challenging benchmarks, establishing an effective baseline for advancing research in multi-view point tracking.
comment : Project Page: https://cvlab-kaist.github.io/MV-TAP/
☆ Learning Visual Affordance from Audio
Lidong Lu, Guo Chen, Zhu Wei, Yicheng Liu, Tong Lu
We introduce Audio-Visual Affordance Grounding (AV-AG), a new task that segments object interaction regions from action sounds. Unlike existing approaches that rely on textual instructions or demonstration videos, which often limited by ambiguity or occlusion, audio provides real-time, semantically rich, and visually independent cues for affordance grounding, enabling more intuitive understanding of interaction regions. To support this task, we construct the first AV-AG dataset, comprising a large collection of action sounds, object images, and pixel-level affordance annotations. The dataset also includes an unseen subset to evaluate zero-shot generalization. Furthermore, we propose AVAGFormer, a model equipped with a semantic-conditioned cross-modal mixer and a dual-head decoder that effectively fuses audio and visual signals for mask prediction. Experiments show that AVAGFormer achieves state-of-the-art performance on AV-AG, surpassing baselines from related tasks. Comprehensive analyses highlight the distinctions between AV-AG and AVS, the benefits of end-to-end modeling, and the contribution of each component. Code and dataset have been released on https://jscslld.github.io/AVAGFormer/.
comment : 15 pages, 10 figures
☆ RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policies
Guillermo Garcia-Cobo, Maximilian Igl, Peter Karkus, Zhejun Zhang, Michael Watson, Yuxiao Chen, Boris Ivanovic, Marco Pavone
Autonomous driving policies are typically trained via open-loop behavior cloning of human demonstrations. However, such policies suffer from covariate shift when deployed in closed loop, leading to compounding errors. We introduce Rollouts as Demonstrations (RoaD), a simple and efficient method to mitigate covariate shift by leveraging the policy's own closed-loop rollouts as additional training data. During rollout generation, RoaD incorporates expert guidance to bias trajectories toward high-quality behavior, producing informative yet realistic demonstrations for fine-tuning. This approach enables robust closed-loop adaptation with orders of magnitude less data than reinforcement learning, and avoids restrictive assumptions of prior closed-loop supervised fine-tuning (CL-SFT) methods, allowing broader applications domains including end-to-end driving. We demonstrate the effectiveness of RoaD on WOSAC, a large-scale traffic simulation benchmark, where it performs similar or better than the prior CL-SFT method; and in AlpaSim, a high-fidelity neural reconstruction-based simulator for end-to-end driving, where it improves driving score by 41\% and reduces collisions by 54\%.
comment : Preprint
☆ PAI-Bench: A Comprehensive Benchmark For Physical AI
Fengzhe Zhou, Jiannan Huang, Jialuo Li, Deva Ramanan, Humphrey Shi
Physical AI aims to develop models that can perceive and predict real-world dynamics; yet, the extent to which current multi-modal large language models and video generative models support these abilities is insufficiently understood. We introduce Physical AI Bench (PAI-Bench), a unified and comprehensive benchmark that evaluates perception and prediction capabilities across video generation, conditional video generation, and video understanding, comprising 2,808 real-world cases with task-aligned metrics designed to capture physical plausibility and domain-specific reasoning. Our study provides a systematic assessment of recent models and shows that video generative models, despite strong visual fidelity, often struggle to maintain physically coherent dynamics, while multi-modal large language models exhibit limited performance in forecasting and causal interpretation. These observations suggest that current systems are still at an early stage in handling the perceptual and predictive demands of Physical AI. In summary, PAI-Bench establishes a realistic foundation for evaluating Physical AI and highlights key gaps that future systems must address.
☆ Artemis: Structured Visual Reasoning for Perception Policy Learning
Wei Tang, Yanpeng Sun, Shan Zhang, Xiaofan Li, Piotr Koniusz, Wei Li, Na Zhao, Zechao Li
Recent reinforcement-learning frameworks for visual perception policy have begun to incorporate intermediate reasoning chains expressed in natural language. Empirical observations indicate that such purely linguistic intermediate reasoning often reduces performance on perception tasks. We argue that the core issue lies not in reasoning per se but in the form of reasoning: while these chains perform semantic reasoning in an unstructured linguistic space, visual perception requires reasoning in a spatial and object-centric space. In response, we introduce Artemis, a perception-policy learning framework that performs structured proposal-based reasoning, where each intermediate step is represented as a (label, bounding-box) pair capturing a verifiable visual state. This design enables explicit tracking of intermediate states, direct supervision for proposal quality, and avoids ambiguity introduced by language-based reasoning. Artemis is built on Qwen2.5-VL-3B, achieves strong performance on grounding and detection task and exhibits substantial generalization to counting and geometric-perception tasks. The consistent improvements across these diverse settings confirm that aligning reasoning with spatial representations enhances perception-policy learning. Owing to its strengthened visual reasoning, Artemis also achieves competitive performance on general MLLM benchmarks, illustrating that spatially grounded reasoning provides a principled route toward scalable and general perception policies.
☆ Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback
Aiden Yiliu Li, Bizhi Yu, Daoan Lei, Tianhe Ren, Shilong Liu
GUI grounding aims to align natural language instructions with precise regions in complex user interfaces. Advanced multimodal large language models show strong ability in visual GUI grounding but still struggle with small or visually similar targets and ambiguity in real world layouts. These limitations arise from limited grounding capacity and from underuse of existing reasoning potential. We present Chain of Ground CoG a training free multi step grounding framework that uses multimodal large language models for iterative visual reasoning and refinement. Instead of direct prediction the model progressively reflects and adjusts its hypotheses leading to more accurate and interpretable localization. Our approach achieves 68.4 accuracy on the ScreenSpot Pro benchmark an improvement of 4.8 points. To measure real world generalization we introduce TPanel UI a dataset of 420 labeled industrial control panels with visual distortions such as blur and masking. On TPanel UI Chain of Ground improves over the strong baseline Qwen3 VL 235B by 6.9 points showing the effectiveness of multi step training free grounding across real world and digital interfaces. These results highlight a direction for unlocking grounding potential through structured iterative refinement instead of additional training.
☆ SGDiff: Scene Graph Guided Diffusion Model for Image Collaborative SegCaptioning AAAI-2025
Xu Zhang, Jin Yuan, Hanwang Zhang, Guojin Zhong, Yongsheng Zang, Jiacheng Lin, Zhiyong Li
Controllable image semantic understanding tasks, such as captioning or segmentation, necessitate users to input a prompt (e.g., text or bounding boxes) to predict a unique outcome, presenting challenges such as high-cost prompt input or limited information output. This paper introduces a new task ``Image Collaborative Segmentation and Captioning'' (SegCaptioning), which aims to translate a straightforward prompt, like a bounding box around an object, into diverse semantic interpretations represented by (caption, masks) pairs, allowing flexible result selection by users. This task poses significant challenges, including accurately capturing a user's intention from a minimal prompt while simultaneously predicting multiple semantically aligned caption words and masks. Technically, we propose a novel Scene Graph Guided Diffusion Model that leverages structured scene graph features for correlated mask-caption prediction. Initially, we introduce a Prompt-Centric Scene Graph Adaptor to map a user's prompt to a scene graph, effectively capturing his intention. Subsequently, we employ a diffusion process incorporating a Scene Graph Guided Bimodal Transformer to predict correlated caption-mask pairs by uncovering intricate correlations between them. To ensure accurate alignment, we design a Multi-Entities Contrastive Learning loss to explicitly align visual and textual entities by considering inter-modal similarity, resulting in well-aligned caption-mask pairs. Extensive experiments conducted on two datasets demonstrate that SGDiff achieves superior performance in SegCaptioning, yielding promising results for both captioning and segmentation tasks with minimal prompt input.
comment : Accept by AAAI-2025
☆ SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generation
Zisu Li, Hengye Lyu, Jiaxin Shi, Yufeng Zeng, Mingming Fan, Hanwang Zhang, Chen Liang
Modeling and synthesizing complex hand-object interactions remains a significant challenge, even for state-of-the-art physics engines. Conventional simulation-based approaches rely on explicitly defined rigid object models and pre-scripted hand gestures, making them inadequate for capturing dynamic interactions with non-rigid or articulated entities such as deformable fabrics, elastic materials, hinge-based structures, furry surfaces, or even living creatures. In this paper, we present SpriteHand, an autoregressive video generation framework for real-time synthesis of versatile hand-object interaction videos across a wide range of object types and motion patterns. SpriteHand takes as input a static object image and a video stream in which the hands are imagined to interact with the virtual object embedded in a real-world scene, and generates corresponding hand-object interaction effects in real time. Our model employs a causal inference architecture for autoregressive generation and leverages a hybrid post-training approach to enhance visual realism and temporal coherence. Our 1.3B model supports real-time streaming generation at around 18 FPS and 640x368 resolution, with an approximate 150 ms latency on a single NVIDIA RTX 5090 GPU, and more than a minute of continuous output. Experiments demonstrate superior visual quality, physical plausibility, and interaction fidelity compared to both generative and engine-based baselines.
☆ GrndCt rl : Grounding Wo rl d Models via Self-Supervised Reward Alignment
Haoyang He, Jay Patrikar, Dong-Ki Kim, Max Smith, Daniel McGann, Ali-akbar Agha-mohammadi, Shayegan Omidshafiei, Sebastian Scherer
Recent advances in video world modeling have enabled large-scale generative models to simulate embodied environments with high visual fidelity, providing strong priors for prediction, planning, and control. Yet, despite their realism, these models often lack geometric grounding, limiting their use in navigation tasks that require spatial coherence and long-horizon stability. We introduce Reinforcement Learning with World Grounding (RLWG), a self-supervised post-training framework that aligns pretrained world models with a physically verifiable structure through geometric and perceptual rewards. Analogous to reinforcement learning from verifiable feedback (RLVR) in language models, RLWG can use multiple rewards that measure pose cycle-consistency, depth reprojection, and temporal coherence. We instantiate this framework with GrndCtrl, a reward-aligned adaptation method based on Group Relative Policy Optimization (GRPO), yielding world models that maintain stable trajectories, consistent geometry, and reliable rollouts for embodied navigation. Like post-training alignment in large language models, GrndCtrl leverages verifiable rewards to bridge generative pretraining and grounded behavior, achieving superior spatial coherence and navigation stability over supervised fine-tuning in outdoor environments.
☆ Script: Graph-Structured and Query-Conditioned Semantic Token Pruning for Multimodal Large Language Models
Zhongyu Yang, Dannong Xu, Wei Pang, Yingfang Yuan
The rapid growth of visual tokens in multimodal large language models (MLLMs) leads to excessive memory consumption and inference latency, especially when handling high-resolution images and videos. Token pruning is a technique used to mitigate this issue by removing redundancy, but existing methods often ignore relevance to the user query or suffer from the limitations of attention mechanisms, reducing their adaptability and effectiveness. To address these challenges, we propose Script, a plug-and-play pruning method that requires no retraining and generalizes across diverse MLLMs. Script comprises two modules: a graph-structured pruning module that removes visually redundant tokens, and a query-conditioned semantic pruning module that preserves query-relevant visual information. Together, they enhance performance on multimodal tasks. Experiments on fourteen benchmarks across image and video understanding tasks show that Script consistently achieves higher model efficiency and predictive accuracy compared to existing pruning methods. On LLaVA-NeXT-7B, it achieves up to 6.8x prefill speedup and 10x FLOP reduction, while retaining 96.88% of the original performance.
comment : Published in Transactions on Machine Learning Research, Project in https://01yzzyu.github.io/script.github.io/
☆ Guardian: Detecting Robotic Planning and Execution Errors with Vision -Language Models
Paul Pacaud, Ricardo Garcia, Shizhe Chen, Cordelia Schmid
Robust robotic manipulation requires reliable failure detection and recovery. Although current Vision-Language Models (VLMs) show promise, their accuracy and generalization are limited by the scarcity of failure data. To address this data gap, we propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures. This method produces not only binary classification labels but also fine-grained failure categories and step-by-step reasoning traces in both simulation and the real world. With it, we construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail, substantially expanding the diversity and scale of existing failure datasets. We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection. Guardian achieves state-of-the-art performance on both existing and newly introduced benchmarks. It also effectively improves task success rates when integrated into a state-of-the-art manipulation system in simulation and real robots, demonstrating the impact of our generated failure data.
comment : 9 pages, 9 figures, 6 tables
☆ Physical ID-Transfer Attacks against Multi-Object Tracking via Adversarial Trajectory ACSA
Chenyi Wang, Yanmao Man, Raymond Muller, Ming Li, Z. Berkay Celik, Ryan Gerdes, Jonathan Petit
Multi-Object Tracking (MOT) is a critical task in computer vision, with applications ranging from surveillance systems to autonomous driving. However, threats to MOT algorithms have yet been widely studied. In particular, incorrect association between the tracked objects and their assigned IDs can lead to severe consequences, such as wrong trajectory predictions. Previous attacks against MOT either focused on hijacking the trackers of individual objects, or manipulating the tracker IDs in MOT by attacking the integrated object detection (OD) module in the digital domain, which are model-specific, non-robust, and only able to affect specific samples in offline datasets. In this paper, we present AdvTraj, the first online and physical ID-manipulation attack against tracking-by-detection MOT, in which an attacker uses adversarial trajectories to transfer its ID to a targeted object to confuse the tracking system, without attacking OD. Our simulation results in CARLA show that AdvTraj can fool ID assignments with 100% success rate in various scenarios for white-box attacks against SORT, which also have high attack transferability (up to 93% attack success rate) against state-of-the-art (SOTA) MOT algorithms due to their common design principles. We characterize the patterns of trajectories generated by AdvTraj and propose two universal adversarial maneuvers that can be performed by a human walker/driver in daily scenarios. Our work reveals under-explored weaknesses in the object association phase of SOTA MOT systems, and provides insights into enhancing the robustness of such systems.
comment : Accepted to Annual Computer Security Applications Conference (ACSAC) 2024
☆ Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
Zahra Mahdavi, Zahra Khodakaramimaghsoud, Hooman Khaloo, Sina Bakhshandeh Taleshani, Erfan Hashemi, Javad Mirzapour Kaleybar, Omid Nejati Manzari
Large vision-language models (LVLMs) are now central to healthcare applications such as medical visual question answering and imaging report generation. Yet, these models remain vulnerable to hallucination outputs that appear plausible but are in fact incorrect. In the natural image domain, several decoding strategies have been proposed to mitigate hallucinations by reinforcing visual evidence, but most rely on secondary decoding or rollback procedures that substantially slow inference. Moreover, existing solutions are often domain-specific and may introduce misalignment between modalities or between generated and ground-truth content. We introduce Med-VCD, a sparse visual-contrastive decoding method that mitigates hallucinations in medical LVLMs without the time overhead of secondary decoding. Med-VCD incorporates a novel token-sparsification strategy that selects visually informed tokens on the fly, trimming redundancy while retaining critical visual context and thus balancing efficiency with reliability. Evaluations on eight medical datasets, spanning ophthalmology, radiology, and pathology tasks in visual question answering, report generation, and dedicated hallucination benchmarks, show that Med-VCD raises factual accuracy by an average of 13\% and improves hallucination accuracy by 6\% relative to baseline medical LVLMs.
☆ Disentangling Progress in Medical Image Registration: Beyond Trend-Driven Architectures towards Domain-Specific Strategies
Bailiang Jian, Jiazhen Pan, Rohit Jena, Morteza Ghahremani, Hongwei Bran Li, Daniel Rueckert, Christian Wachinger, Benedikt Wiestler
Medical image registration drives quantitative analysis across organs, modalities, and patient populations. Recent deep learning methods often combine low-level "trend-driven" computational blocks from computer vision, such as large-kernel CNNs, Transformers, and state-space models, with high-level registration-specific designs like motion pyramids, correlation layers, and iterative refinement. Yet, their relative contributions remain unclear and entangled. This raises a central question: should future advances in registration focus on importing generic architectural trends or on refining domain-specific design principles? Through a modular framework spanning brain, lung, cardiac, and abdominal registration, we systematically disentangle the influence of these two paradigms. Our evaluation reveals that low-level "trend-driven" computational blocks offer only marginal or inconsistent gains, while high-level registration-specific designs consistently deliver more accurate, smoother, and more robust deformations. These domain priors significantly elevate the performance of a standard U-Net baseline, far more than variants incorporating "trend-driven" blocks, achieving an average relative improvement of $\sim3\%$. All models and experiments are released within a transparent, modular benchmark that enables plug-and-play comparison for new architectures and registration tasks (https://github.com/BailiangJ/rethink-reg). This dynamic and extensible platform establishes a common ground for reproducible and fair evaluation, inviting the community to isolate genuine methodological contributions from domain priors. Our findings advocate a shift in research emphasis: from following architectural trends to embracing domain-specific design principles as the true drivers of progress in learning-based medical image registration.
comment : Submitted to Medical Image Analysis. Journal Extension of arXiv:2407.19274
☆ SA RL : Spatially-Aware Self-Supervised Representation Learning for Visuo-Tactile Perception
Gurmeher Khurana, Lan Wei, Dandan Zhang
Contact-rich robotic manipulation requires representations that encode local geometry. Vision provides global context but lacks direct measurements of properties such as texture and hardness, whereas touch supplies these cues. Modern visuo-tactile sensors capture both modalities in a single fused image, yielding intrinsically aligned inputs that are well suited to manipulation tasks requiring visual and tactile information. Most self-supervised learning (SSL) frameworks, however, compress feature maps into a global vector, discarding spatial structure and misaligning with the needs of manipulation. To address this, we propose SARL, a spatially-aware SSL framework that augments the Bootstrap Your Own Latent (BYOL) architecture with three map-level objectives, including Saliency Alignment (SAL), Patch-Prototype Distribution Alignment (PPDA), and Region Affinity Matching (RAM), to keep attentional focus, part composition, and geometric relations consistent across views. These losses act on intermediate feature maps, complementing the global objective. SARL consistently outperforms nine SSL baselines across six downstream tasks with fused visual-tactile data. On the geometry-sensitive edge-pose regression task, SARL achieves a Mean Absolute Error (MAE) of 0.3955, a 30% relative improvement over the next-best SSL method (0.5682 MAE) and approaching the supervised upper bound. These findings indicate that, for fused visual-tactile data, the most effective signal is structured spatial equivariance, in which features vary predictably with object geometry, which enables more capable robotic perception.
☆ StyleYourSmile: Cross-Domain Face Retargeting Without Paired Multi-Style Data
Avirup Dey, Vinay Namboodiri
Cross-domain face retargeting requires disentangled control over identity, expressions, and domain-specific stylistic attributes. Existing methods, typically trained on real-world faces, either fail to generalize across domains, need test-time optimizations, or require fine-tuning with carefully curated multi-style datasets to achieve domain-invariant identity representations. In this work, we introduce \textit{StyleYourSmile}, a novel one-shot cross-domain face retargeting method that eliminates the need for curated multi-style paired data. We propose an efficient data augmentation strategy alongside a dual-encoder framework, for extracting domain-invariant identity cues and capturing domain-specific stylistic variations. Leveraging these disentangled control signals, we condition a diffusion model to retarget facial expressions across domains. Extensive experiments demonstrate that \textit{StyleYourSmile} achieves superior identity preservation and retargeting fidelity across a wide range of visual domains.
comment : 15 pages, 14 figures
☆ KM-ViPE: Online Tightly Coupled Vision -Language-Geometry Fusion for Open-Vocabulary Semantic SLAM
Zaid Nasser, Mikhail Iumanov, Tianhao Li, Maxim Popov, Jaafar Mahmoud, Malik Mohrat, Ilya Obrubov, Ekaterina Derevyanka, Ivan Sosin, Sergey Kolyubin
We present KM-ViPE (Knowledge Mapping Video Pose Engine), a real-time open-vocabulary SLAM framework for uncalibrated monocular cameras in dynamic environments. Unlike systems requiring depth sensors and offline calibration, KM-ViPE operates directly on raw RGB streams, making it ideal for ego-centric applications and harvesting internet-scale video data for training. KM-ViPE tightly couples DINO visual features with geometric constraints through a high-level features based adaptive robust kernel that handles both moving objects and movable static objects (e.g., moving furniture in ego-centric views). The system performs simultaneous online localization and open-vocabulary semantic mapping by fusing geometric and deep visual features aligned with language embeddings. Our results are competitive with state-of-the-art approaches, while existing solutions either operate offline, need depth data and/or odometry estimation, or lack dynamic scene robustness. KM-ViPE benefits from internet-scale training and uniquely combines online operation, uncalibrated monocular input, and robust handling of dynamic scenes, which makes it a good fit for autonomous robotics and AR/VR applications and advances practical spatial intelligence capabilities for embodied AI.
☆ TransientTrack: Advanced Multi-Object Tracking and Classification of Cancer Cells with Transient Fluorescent Signals
Florian Bürger, Martim Dias Gomes, Nica Gutu, Adrián E. Granada, Noémie Moreau, Katarzyna Bozek
Tracking cells in time-lapse videos is an essential technique for monitoring cell population dynamics at a single-cell level. Current methods for cell tracking are developed on videos with mostly single, constant signals and do not detect pivotal events such as cell death. Here, we present TransientTrack, a deep learning-based framework for cell tracking in multi-channel microscopy video data with transient fluorescent signals that fluctuate over time following processes such as the circadian rhythm of cells. By identifying key cellular events - mitosis (cell division) and apoptosis (cell death) our method allows us to build complete trajectories, including cell lineage information. TransientTrack is lightweight and performs matching on cell detection embeddings directly, without the need for quantification of tracking-specific cell features. Furthermore, our approach integrates Transformer Networks, multi-stage matching using all detection boxes, and the interpolation of missing tracklets with the Kalman Filter. This unified framework achieves strong performance across diverse conditions, effectively tracking cells and capturing cell division and death. We demonstrate the use of TransientTrack in an analysis of the efficacy of a chemotherapeutic drug at a single-cell level. The proposed framework could further advance quantitative studies of cancer cell dynamics, enabling detailed characterization of treatment response and resistance mechanisms. The code is available at https://github.com/bozeklab/TransientTrack.
comment : 13 pages, 7 figures, 2 tables. This work has been submitted to IEEE Transactions on Medical Imaging
☆ COACH: Collaborative Agent s for Context ual Highlighting - A Multi- Agent Framework for Sports Video Analysis AAAI 2026
Tsz-To Wong, Ching-Chun Huang, Hong-Han Shuai
Intelligent sports video analysis demands a comprehensive understanding of temporal context, from micro-level actions to macro-level game strategies. Existing end-to-end models often struggle with this temporal hierarchy, offering solutions that lack generalization, incur high development costs for new tasks, and suffer from poor interpretability. To overcome these limitations, we propose a reconfigurable Multi-Agent System (MAS) as a foundational framework for sports video understanding. In our system, each agent functions as a distinct "cognitive tool" specializing in a specific aspect of analysis. The system's architecture is not confined to a single temporal dimension or task. By leveraging iterative invocation and flexible composition of these agents, our framework can construct adaptive pipelines for both short-term analytic reasoning (e.g., Rally QA) and long-term generative summarization (e.g., match summaries). We demonstrate the adaptability of this framework using two representative tasks in badminton analysis, showcasing its ability to bridge fine-grained event detection and global semantic organization. This work presents a paradigm shift towards a flexible, scalable, and interpretable system for robust, cross-task sports video intelligence.The project homepage is available at https://aiden1020.github.io/COACH-project-page
comment : Accepted by AAAI 2026 Workshop LaMAS
☆ Register Any Point: Scaling 3D Point Cloud Registration by Flow Matching
Yue Pan, Tao Sun, Liyuan Zhu, Lucas Nunes, Iro Armeni, Jens Behley, Cyrill Stachniss
Point cloud registration aligns multiple unposed point clouds into a common frame, and is a core step for 3D reconstruction and robot localization. In this work, we cast registration as conditional generation: a learned continuous, point-wise velocity field transports noisy points to a registered scene, from which the pose of each view is recovered. Unlike previous methods that conduct correspondence matching to estimate the transformation between a pair of point clouds and then optimize the pairwise transformations to realize multi-view registration, our model directly generates the registered point cloud. With a lightweight local feature extractor and test-time rigidity enforcement, our approach achieves state-of-the-art results on pairwise and multi-view registration benchmarks, particularly with low overlap, and generalizes across scales and sensor modalities. It further supports downstream tasks including relocalization, multi-robot SLAM, and multi-session map merging. Source code available at: https://github.com/PRBonn/RAP.
comment : 22 pages
☆ PhyDetEx: Detecting and Explaining the Physical Plausibility of T2V Models
Zeqing Wang, Keze Wang, Lei Zhang
Driven by the growing capacity and training scale, Text-to-Video (T2V) generation models have recently achieved substantial progress in video quality, length, and instruction-following capability. However, whether these models can understand physics and generate physically plausible videos remains a question. While Vision-Language Models (VLMs) have been widely used as general-purpose evaluators in various applications, they struggle to identify the physically impossible content from generated videos. To investigate this issue, we construct a \textbf{PID} (\textbf{P}hysical \textbf{I}mplausibility \textbf{D}etection) dataset, which consists of a \textit{test split} of 500 manually annotated videos and a \textit{train split} of 2,588 paired videos, where each implausible video is generated by carefully rewriting the caption of its corresponding real-world video to induce T2V models producing physically implausible content. With the constructed dataset, we introduce a lightweight fine-tuning approach, enabling VLMs to not only detect physically implausible events but also generate textual explanations on the violated physical principles. Taking the fine-tuned VLM as a physical plausibility detector and explainer, namely \textbf{PhyDetEx}, we benchmark a series of state-of-the-art T2V models to assess their adherence to physical laws. Our findings show that although recent T2V models have made notable progress toward generating physically plausible content, understanding and adhering to physical laws remains a challenging issue, especially for open-source models. Our dataset, training code, and checkpoints are available at \href{https://github.com/Zeqing-Wang/PhyDetEx}{https://github.com/Zeqing-Wang/PhyDetEx}.
comment : 17 pages, 8 figures
☆ OpenREAD: Reinforced Open-Ended Reasoing for End-to-End Autonomous Driving with LLM-as-Critic
Songyan Zhang, Wenhui Huang, Zhan Chen, Chua Jiahao Collister, Qihang Huang, Chen Lv
Recently, two-stage fine-tuning strategies, e.g., acquiring essential driving knowledge through supervised fine-tuning (SFT) and further enhancing decision-making and planning via reinforcement fine-tuning (RFT), have shown strong potential in advancing the knowledge-driven autonomous driving (AD) paradigm. However, the learning nature of SFT still limits the generalization of reasoning, thereby constraining the full potential of driving performance. Meanwhile, current RFT approaches are primarily applied to downstream tasks, since scene understanding is an open-ended problem where corresponding rewards are difficult to quantify. To address these limitations, we propose OpenREAD, an OPEN-ended REasoning reinforced vision-language model (VLM)-based autonomous driving (AD) framework that enables end-to-end RFT across the full spectrum from high-level reasoning to low-level trajectory planning. Specifically, we begin by constructing large-scale Chain-of-Thought (CoT) annotations on open-source driving-related knowledge datasets, and employ the powerful Qwen3 large language model (LLM) as the critic in RFT to quantify reasoning quality for open-ended questions during reward modeling. Extensive experiments confirm that joint end-to-end RFT yields substantial improvements in both upstream and downstream tasks, enabling OpenREAD to achieve state-of-the-art performance on reasoning and planning benchmarks.
☆ CauSight: Learning to Supersense for Visual Causal Discovery
Yize Zhang, Meiqi Chen, Sirui Chen, Bo Peng, Yanxi Zhang, Tianyu Li, Chaochao Lu
Causal thinking enables humans to understand not just what is seen, but why it happens. To replicate this capability in modern AI systems, we introduce the task of visual causal discovery. It requires models to infer cause-and-effect relations among visual entities across diverse scenarios instead of merely perceiving their presence. To this end, we first construct the Visual Causal Graph dataset (VCG-32K), a large-scale collection of over 32,000 images annotated with entity-level causal graphs, and further develop CauSight, a novel vision-language model to perform visual causal discovery through causally aware reasoning. Our training recipe integrates three components: (1) training data curation from VCG-32K, (2) Tree-of-Causal-Thought (ToCT) for synthesizing reasoning trajectories, and (3) reinforcement learning with a designed causal reward to refine the reasoning policy. Experiments show that CauSight outperforms GPT-4.1 on visual causal discovery, achieving over a threefold performance boost (21% absolute gain). Our code, model, and dataset are fully open-sourced at project page: https://github.com/OpenCausaLab/CauSight.
comment : project page: https://github.com/OpenCausaLab/CauSight
☆ InnoGym: Benchmark ing the Innovation Potential of AI Agent s
Jintian Zhang, Kewei Xu, Jingsheng Zheng, Zhuoyun Yu, Yuqi Zhu, Yujie Luo, Lanning Wei, Shuofei Qiao, Lun Du, Da Zheng, Shumin Deng, Huajun Chen, Ningyu Zhang
LLMs and Agents have achieved impressive progress in code generation, mathematical reasoning, and scientific discovery. However, existing benchmarks primarily measure correctness, overlooking the diversity of methods behind solutions. True innovation depends not only on producing correct answers but also on the originality of the approach. We present InnoGym, the first benchmark and framework designed to systematically evaluate the innovation potential of AI agents. InnoGym introduces two complementary metrics: performance gain, which measures improvement over the best-known solutions, and novelty, which captures methodological differences from prior approaches. The benchmark includes 18 carefully curated tasks from real-world engineering and scientific domains, each standardized through resource filtering, evaluator validation, and solution collection. In addition, we provide iGym, a unified execution environment for reproducible and long-horizon evaluations. Extensive experiments show that while some agents produce novel approaches, their lack of robustness limits performance gains. These results highlight a key gap between creativity and effectiveness, underscoring the need for benchmarks that evaluate both.
comment : Work in progress
☆ Seeing through Imagination: Learning Scene Geometry via Implicit Spatial Wo rl d Modeling
Meng Cao, Haokun Lin, Haoyuan Li, Haoran Tang, Rongtao Xu, Dong An, Xue Liu, Ian Reid, Xiaodan Liang
Spatial reasoning, the ability to understand and interpret the 3D structure of the world, is a critical yet underdeveloped capability in Multimodal Large Language Models (MLLMs). Current methods predominantly rely on verbal descriptive tuning, which suffers from visual illiteracy, i.e., they learn spatial concepts through textual symbols alone, devoid of connection to their visual manifestations. To bridge this gap, this paper introduces MILO, an Implicit spatIaL wOrld modeling paradigm that simulates human-like spatial imagination. MILO integrates a visual generator to provide geometry-aware feedback, thereby implicitly grounding the MLLM's symbolic reasoning in perceptual experience. Complementing this paradigm, we propose RePE (Relative Positional Encoding), a novel encoding scheme that captures relative camera-pose transformations, offering superior performance over absolute coordinate systems. To support the training, we construct GeoGen, a large-scale Geometry-aware Generative dataset with approximately 2,241 videos and 67,827 observation-action-outcome triplets. Experiments demonstrate that our approach significantly enhances spatial reasoning capabilities across multiple baselines and benchmarks, offering a more holistic understanding of 3D space.
☆ Forget Less, Retain More: A Lightweight Regularizer for Rehearsal-Based Continual Learning
Lama Alssum, Hasan Abed Al Kader Hammoud, Motasem Alfarra, Juan C Leon Alcazar, Bernard Ghanem
Deep neural networks suffer from catastrophic forgetting, where performance on previous tasks degrades after training on a new task. This issue arises due to the model's tendency to overwrite previously acquired knowledge with new information. We present a novel approach to address this challenge, focusing on the intersection of memory-based methods and regularization approaches. We formulate a regularization strategy, termed Information Maximization (IM) regularizer, for memory-based continual learning methods, which is based exclusively on the expected label distribution, thus making it class-agnostic. As a consequence, IM regularizer can be directly integrated into various rehearsal-based continual learning methods, reducing forgetting and favoring faster convergence. Our empirical validation shows that, across datasets and regardless of the number of tasks, our proposed regularization strategy consistently improves baseline performance at the expense of a minimal computational overhead. The lightweight nature of IM ensures that it remains a practical and scalable solution, making it applicable to real-world continual learning scenarios where efficiency is paramount. Finally, we demonstrate the data-agnostic nature of our regularizer by applying it to video data, which presents additional challenges due to its temporal structure and higher memory requirements. Despite the significant domain gap, our experiments show that IM regularizer also improves the performance of video continual learning methods.
☆ En vision : Benchmark ing Unified Understanding & Generation for Causal Wo rl d Process Insights
Juanxi Tian, Siyuan Li, Conghui He, Lijun Wu, Cheng Tan
Current multimodal models aim to transcend the limitations of single-modality representations by unifying understanding and generation, often using text-to-image (T2I) tasks to calibrate semantic consistency. However, their reliance on static, single-image generation in training and evaluation leads to overfitting to static pattern matching and semantic fusion, while fundamentally hindering their ability to model dynamic processes that unfold over time. To address these constraints, we propose Envision-a causal event progression benchmark for chained text-to-multi-image generation. Grounded in world knowledge and structured by spatiotemporal causality, it reorganizes existing evaluation dimensions and includes 1,000 four-stage prompts spanning six scientific and humanities domains. To transition evaluation from single images to sequential frames and assess whether models truly internalize world knowledge while adhering to causal-temporal constraints, we introduce Envision-Score, a holistic metric integrating multi-dimensional consistency, physicality, and aesthetics. Comprehensive evaluation of 15 models (10 specialized T2I models, 5 unified models) uncovers: specialized T2I models demonstrate proficiency in aesthetic rendering yet lack intrinsic world knowledge. Unified multimodal models bridge this gap, consistently outperforming specialized counterparts in causal narrative coherence. However, even these unified architectures remain subordinate to closed-source models and struggle to overcome the core challenge of spatiotemporal consistency. This demonstrates that a focus on causally-isolated single images impedes multi-frame reasoning and generation, promoting static pattern matching over dynamic world modeling-ultimately limiting world knowledge internalization, generation.
comment : 35 pages, 12 figures, 10 tables
☆ Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos
Xavier Thomas, Youngsun Lim, Ananya Srinivasan, Audrey Zheng, Deepti Ghadiyaram
Despite rapid advances in video generative models, robust metrics for evaluating visual and temporal correctness of complex human actions remain elusive. Critically, existing pure-vision encoders and Multimodal Large Language Models (MLLMs) are strongly appearance-biased, lack temporal understanding, and thus struggle to discern intricate motion dynamics and anatomical implausibilities in generated videos. We tackle this gap by introducing a novel evaluation metric derived from a learned latent space of real-world human actions. Our method first captures the nuances, constraints, and temporal smoothness of real-world motion by fusing appearance-agnostic human skeletal geometry features with appearance-based features. We posit that this combined feature space provides a robust representation of action plausibility. Given a generated video, our metric quantifies its action quality by measuring the distance between its underlying representations and this learned real-world action distribution. For rigorous validation, we develop a new multi-faceted benchmark specifically designed to probe temporally challenging aspects of human action fidelity. Through extensive experiments, we show that our metric achieves substantial improvement of more than 68% compared to existing state-of-the-art methods on our benchmark, performs competitively on established external benchmarks, and has a stronger correlation with human perception. Our in-depth analysis reveals critical limitations in current video generative models and establishes a new standard for advanced research in video generation.
☆ SAM3-UNet: Simplified Adaptation of Segment Anything Model 3
Xinyu Xiong, Zihuang Wu, Lei Lu, Yufa Xia
In this paper, we introduce SAM3-UNet, a simplified variant of Segment Anything Model 3 (SAM3), designed to adapt SAM3 for downstream tasks at a low cost. Our SAM3-UNet consists of three components: a SAM3 image encoder, a simple adapter for parameter-efficient fine-tuning, and a lightweight U-Net-style decoder. Preliminary experiments on multiple tasks, such as mirror detection and salient object detection, demonstrate that the proposed SAM3-UNet outperforms the prior SAM2-UNet and other state-of-the-art methods, while requiring less than 6 GB of GPU memory during training with a batch size of 12. The code is publicly available at https://github.com/WZH0120/SAM3-UNet.
comment : Technical Report
☆ Learned Image Compression for Earth Observation: Implications for Downstream Segmentation Tasks
Christian Mollière, Iker Cumplido, Marco Zeulner, Lukas Liesenhoff, Matthias Schubert, Julia Gottfriedsen
The rapid growth of data from satellite-based Earth observation (EO) systems poses significant challenges in data transmission and storage. We evaluate the potential of task-specific learned compression algorithms in this context to reduce data volumes while retaining crucial information. In detail, we compare traditional compression (JPEG 2000) versus a learned compression approach (Discretized Mixed Gaussian Likelihood) on three EO segmentation tasks: Fire, cloud, and building detection. Learned compression notably outperforms JPEG 2000 for large-scale, multi-channel optical imagery in both reconstruction quality (PSNR) and segmentation accuracy. However, traditional codecs remain competitive on smaller, single-channel thermal infrared datasets due to limited data and architectural constraints. Additionally, joint end-to-end optimization of compression and segmentation models does not improve performance over standalone optimization.
☆ Evaluating SAM2 for Video Semantic Segmentation
Syed Hesham Syed Ariff, Yun Liu, Guolei Sun, Jing Yang, Henghui Ding, Xue Geng, Xudong Jiang
The Segmentation Anything Model 2 (SAM2) has proven to be a powerful foundation model for promptable visual object segmentation in both images and videos, capable of storing object-aware memories and transferring them temporally through memory blocks. While SAM2 excels in video object segmentation by providing dense segmentation masks based on prompts, extending it to dense Video Semantic Segmentation (VSS) poses challenges due to the need for spatial accuracy, temporal consistency, and the ability to track multiple objects with complex boundaries and varying scales. This paper explores the extension of SAM2 for VSS, focusing on two primary approaches and highlighting firsthand observations and common challenges faced during this process. The first approach involves using SAM2 to extract unique objects as masks from a given image, with a segmentation network employed in parallel to generate and refine initial predictions. The second approach utilizes the predicted masks to extract unique feature vectors, which are then fed into a simple network for classification. The resulting classifications and masks are subsequently combined to produce the final segmentation. Our experiments suggest that leveraging SAM2 enhances overall performance in VSS, primarily due to its precise predictions of object boundaries.
comment : 17 pages, 3 figures and 7 tables
☆ Robust Rigid and Non-Rigid Medical Image Registration Using Learnable Edge Kernels
Ahsan Raza Siyal, Markus Haltmeier, Ruth Steiger, Malik Galijasevic, Elke Ruth Gizewski, Astrid Ellen Grams
Medical image registration is crucial for various clinical and research applications including disease diagnosis or treatment planning which require alignment of images from different modalities, time points, or subjects. Traditional registration techniques often struggle with challenges such as contrast differences, spatial distortions, and modality-specific variations. To address these limitations, we propose a method that integrates learnable edge kernels with learning-based rigid and non-rigid registration techniques. Unlike conventional layers that learn all features without specific bias, our approach begins with a predefined edge detection kernel, which is then perturbed with random noise. These kernels are learned during training to extract optimal edge features tailored to the task. This adaptive edge detection enhances the registration process by capturing diverse structural features critical in medical imaging. To provide clearer insight into the contribution of each component in our design, we introduce four variant models for rigid registration and four variant models for non-rigid registration. We evaluated our approach using a dataset provided by the Medical University across three setups: rigid registration without skull removal, with skull removal, and non-rigid registration. Additionally, we assessed performance on two publicly available datasets. Across all experiments, our method consistently outperformed state-of-the-art techniques, demonstrating its potential to improve multi-modal image alignment and anatomical structure analysis.
☆ VideoScoop: A Non-Traditional Domain-Independent Framework For Video Analysis
Hafsa Billah
Automatically understanding video contents is important for several applications in Civic Monitoring (CM), general Surveillance (SL), Assisted Living (AL), etc. Decades of Image and Video Analysis (IVA) research have advanced tasks such as content extraction (e.g., object recognition and tracking). Identifying meaningful activities or situations (e.g., two objects coming closer) remains difficult and cannot be achieved by content extraction alone. Currently, Video Situation Analysis (VSA) is done manually with a human in the loop, which is error-prone and labor-intensive, or through custom algorithms designed for specific video types or situations. These algorithms are not general-purpose and require a new algorithm/software for each new situation or video from a new domain. This report proposes a general-purpose VSA framework that overcomes the above limitations. Video contents are extracted once using state-of-the-art Video Content Extraction technologies. They are represented using two alternative models -- the extended relational model (R++) and graph models. When represented using R++, the extracted contents can be used as data streams, enabling Continuous Query Processing via the proposed Continuous Query Language for Video Analysis. The graph models complement this by enabling the detection of situations that are difficult or impossible to detect using the relational model alone. Existing graph algorithms and newly developed algorithms support a wide variety of situation detection. To support domain independence, primitive situation variants across domains are identified and expressed as parameterized templates. Extensive experiments were conducted across several interesting situations from three domains -- AL, CM, and SL-- to evaluate the accuracy, efficiency, and robustness of the proposed approach using a dataset of videos of varying lengths from these domains.
comment : This is a report submitted as part of PhD proposal defense of Hafsa Billah
☆ Hicon Agent : History Context -aware Policy Optimization for GUI Agent s
Xurui Zhou, Gongwei Chen, Yuquan Xie, Zaijing Li, Kaiwen Zhou, Shuai Wang, Shuo Yang, Zhuotao Tian, Rui Shao
Graphical User Interface (GUI) agents require effective use of historical context to perform sequential navigation tasks. While incorporating past actions and observations can improve decision making, naive use of full history leads to excessive computational overhead and distraction from irrelevant information. To address this, we introduce HiconAgent, a GUI agent trained with History Context-aware Policy Optimization (HCPO) for efficient and effective utilization of historical information. HCPO optimizes history usage in both sampling and policy updates through two complementary components: (1) Dynamic Context Sampling (DCS) presents the agent with variable length histories during sampling, enabling adaptive use of the most relevant context; (2) Anchor-guided History Compression (AHC) refines the policy update phase with a dual branch strategy where the compressed branch removes history observations while keeping history actions as information flow anchors. The compressed and uncompressed branches are coupled through a history-enhanced alignment loss to enforce consistent history usage while maintaining efficiency. Experiments on mainstream GUI navigation benchmarks demonstrate strong performance. Despite being smaller, HiconAgent-3B outperforms GUI-R1-7B by +8.46 percent grounding accuracy and +11.32 percent step success rate on GUI-Odyssey, while achieving comparable results on AndroidControl and AITW with up to 2.47x computational speedup and 60 percent FLOPs reduction.
☆ FreqEdit: Preserving High-Frequency Features for Robust Multi-Turn Image Editing
Yucheng Liao, Jiajun Liang, Kaiqian Cui, Baoquan Zhao, Haoran Xie, Wei Liu, Qing Li, Xudong Mao
Instruction-based image editing through natural language has emerged as a powerful paradigm for intuitive visual manipulation. While recent models achieve impressive results on single edits, they suffer from severe quality degradation under multi-turn editing. Through systematic analysis, we identify progressive loss of high-frequency information as the primary cause of this quality degradation. We present FreqEdit, a training-free framework that enables stable editing across 10+ consecutive iterations. Our approach comprises three synergistic components: (1) high-frequency feature injection from reference velocity fields to preserve fine-grained details, (2) an adaptive injection strategy that spatially modulates injection strength for precise region-specific control, and (3) a path compensation mechanism that periodically recalibrates the editing trajectory to prevent over-constraint. Extensive experiments demonstrate that FreqEdit achieves superior performance in both identity preservation and instruction following compared to seven state-of-the-art baselines.
☆ StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos
Daeun Lee, Subhojyoti Mukherjee, Branislav Kveton, Ryan A. Rossi, Viet Dac Lai, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Mohit Bansal
Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intention for realistic applications like AR glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether MLLMs can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs use gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively evaluate streaming video understanding. These tasks assess whether models can use real-time gaze to follow shifting attention and infer user intentions from only past and currently observed frames. To build StreamGaze, we develop a gaze-video QA generation pipeline that aligns egocentric videos with raw gaze trajectories via fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that closely reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, revealing fundamental limitations in gaze-based temporal reasoning, intention modeling, and proactive prediction. We further provide detailed analyses of gaze-prompting strategies, reasoning behaviors, and task-specific failure modes, offering deeper insight into why current MLLMs struggle and what capabilities future models must develop. All data and code will be publicly released to support continued research in gaze-guided streaming video understanding.
comment : Project page: https://streamgaze.github.io/
☆ SSR: Semantic and Spatial Rectification for CLIP-based Weakly Supervised Segmentation AAAI 2026
Xiuli Bi, Die Xiao, Junchao Fan, Bin Xiao
In recent years, Contrastive Language-Image Pretraining (CLIP) has been widely applied to Weakly Supervised Semantic Segmentation (WSSS) tasks due to its powerful cross-modal semantic understanding capabilities. This paper proposes a novel Semantic and Spatial Rectification (SSR) method to address the limitations of existing CLIP-based weakly supervised semantic segmentation approaches: over-activation in non-target foreground regions and background areas. Specifically, at the semantic level, the Cross-Modal Prototype Alignment (CMPA) establishes a contrastive learning mechanism to enforce feature space alignment across modalities, reducing inter-class overlap while enhancing semantic correlations, to rectify over-activation in non-target foreground regions effectively; at the spatial level, the Superpixel-Guided Correction (SGC) leverages superpixel-based spatial priors to precisely filter out interference from non-target regions during affinity propagation, significantly rectifying background over-activation. Extensive experiments on the PASCAL VOC and MS COCO datasets demonstrate that our method outperforms all single-stage approaches, as well as more complex multi-stage approaches, achieving mIoU scores of 79.5% and 50.6%, respectively.
comment : Accepted in AAAI 2026
☆ Revisiting Direct Encoding: Learnable Temporal Dynamics for Static Image Spiking Neural Networks
Huaxu He
Handling static images that lack inherent temporal dynamics remains a fundamental challenge for spiking neural networks (SNNs). In directly trained SNNs, static inputs are typically repeated across time steps, causing the temporal dimension to collapse into a rate like representation and preventing meaningful temporal modeling. This work revisits the reported performance gap between direct and rate based encodings and shows that it primarily stems from convolutional learnability and surrogate gradient formulations rather than the encoding schemes themselves. To illustrate this mechanism level clarification, we introduce a minimal learnable temporal encoding that adds adaptive phase shifts to induce meaningful temporal variation from static inputs.
☆ DreamingComics: A Story Visualization Pipeline via Subject and Layout Customized Generation using Video Models
Patrick Kwon, Chen Chen
Current story visualization methods tend to position subjects solely by text and face challenges in maintaining artistic consistency. To address these limitations, we introduce DreamingComics, a layout-aware story visualization framework. We build upon a pretrained video diffusion-transformer (DiT) model, leveraging its spatiotemporal priors to enhance identity and style consistency. For layout-based position control, we propose RegionalRoPE, a region-aware positional encoding scheme that re-indexes embeddings based on the target layout. Additionally, we introduce a masked condition loss to further constrain each subject's visual features to their designated region. To infer layouts from natural language scripts, we integrate an LLM-based layout generator trained to produce comic-style layouts, enabling flexible and controllable layout conditioning. We present a comprehensive evaluation of our approach, showing a 29.2% increase in character consistency and a 36.2% increase in style similarity compared to previous methods, while displaying high spatial accuracy. Our project page is available at https://yj7082126.github.io/dreamingcomics/
☆ Cross-Domain Validation of a Resection-Trained Self-Supervised Model on Multicentre Mesothelioma Biopsies
Farzaneh Seyedshahi, Francesca Damiola, Sylvie Lantuejoul, Ke Yuan, John Le Quesne
Accurate subtype classification and outcome prediction in mesothelioma are essential for guiding therapy and patient care. Most computational pathology models are trained on large tissue images from resection specimens, limiting their use in real-world settings where small biopsies are common. We show that a self-supervised encoder trained on resection tissue can be applied to biopsy material, capturing meaningful morphological patterns. Using these patterns, the model can predict patient survival and classify tumor subtypes. This approach demonstrates the potential of AI-driven tools to support diagnosis and treatment planning in mesothelioma.
☆ Open-wo rl d Hand-Object Interaction Video Generation Based on Structure and Contact-aware Representation
Haodong Yan, Hang Yu, Zhide Zhong, Weilin Yuan, Xin Gong, Zehang Luo, Chengxi Heyu, Junfeng Li, Wenxuan Song, Shunbo Zhou, Haoang Li
Generating realistic hand-object interactions (HOI) videos is a significant challenge due to the difficulty of modeling physical constraints (e.g., contact and occlusion between hands and manipulated objects). Current methods utilize HOI representation as an auxiliary generative objective to guide video synthesis. However, there is a dilemma between 2D and 3D representations that cannot simultaneously guarantee scalability and interaction fidelity. To address this limitation, we propose a structure and contact-aware representation that captures hand-object contact, hand-object occlusion, and holistic structure context without 3D annotations. This interaction-oriented and scalable supervision signal enables the model to learn fine-grained interaction physics and generalize to open-world scenarios. To fully exploit the proposed representation, we introduce a joint-generation paradigm with a share-and-specialization strategy that generates interaction-oriented representations and videos. Extensive experiments demonstrate that our method outperforms state-of-the-art methods on two real-world datasets in generating physics-realistic and temporally coherent HOI videos. Furthermore, our approach exhibits strong generalization to challenging open-world scenarios, highlighting the benefit of our scalable design. Our project page is https://hgzn258.github.io/SCAR/.
☆ GRASP: Guided Residual Adapters with Sample-wise Partitioning
Felix Nützel, Mischa Dombrowski, Bernhard Kainz
Recent advances in text-to-image diffusion models enable high-fidelity generation across diverse prompts. However, these models falter in long-tail settings, such as medical imaging, where rare pathologies comprise a small fraction of data. This results in mode collapse: tail-class outputs lack quality and diversity, undermining the goal of synthetic data augmentation for underrepresented conditions. We pinpoint gradient conflicts between frequent head and rare tail classes as the primary culprit, a factor unaddressed by existing sampling or conditioning methods that mainly steer inference without altering the learned distribution. To resolve this, we propose GRASP: Guided Residual Adapters with Sample-wise Partitioning. GRASP uses external priors to statically partition samples into clusters that minimize intra-group gradient clashes. It then fine-tunes pre-trained models by injecting cluster-specific residual adapters into transformer feedforward layers, bypassing learned gating for stability and efficiency. On the long-tail MIMIC-CXR-LT dataset, GRASP yields superior FID and diversity metrics, especially for rare classes, outperforming baselines like vanilla fine-tuning and Mixture of Experts variants. Downstream classification on NIH-CXR-LT improves considerably for tail labels. Generalization to ImageNet-LT confirms broad applicability. Our method is lightweight, scalable, and readily integrates with diffusion pipelines.
comment : 10 pages, 4 figures, 6 tables
☆ Bridging the Scale Gap: Balanced Tiny and General Object Detection in Remote Sensing Imagery
Zhicheng Zhao, Yin Huang, Lingma Sun, Chenglong Li, Jin Tang
Tiny object detection in remote sensing imagery has attracted significant research interest in recent years. Despite recent progress, achieving balanced detection performance across diverse object scales remains a formidable challenge, particularly in scenarios where dense tiny objects and large objects coexist. Although large foundation models have revolutionized general vision tasks, their application to tiny object detection remains unexplored due to the extreme scale variation and density distribution inherent to remote sensing imagery. To bridge this scale gap, we propose ScaleBridge-Det, to the best of our knowledge, the first large detection framework designed for tiny objects, which could achieve balanced performance across diverse scales through scale-adaptive expert routing and density-guided query allocation. Specifically, we introduce a Routing-Enhanced Mixture Attention (REM) module that dynamically selects and fuses scale-specific expert features via adaptive routing to address the tendency of standard MoE models to favor dominant scales. REM generates complementary and discriminative multi-scale representations suitable for both tiny and large objects. Furthermore, we present a Density-Guided Dynamic Query (DGQ) module that predicts object density to adaptively adjust query positions and numbers, enabling efficient resource allocation for objects of varying scales. The proposed framework allows ScaleBridge-Det to simultaneously optimize performance for both dense tiny and general objects without trade-offs. Extensive experiments on benchmark and cross-domain datasets demonstrate that ScaleBridge-Det achieves state-of-the-art performance on AI-TOD-V2 and DTOD, while exhibiting superior cross-domain robustness on VisDrone.
☆ DB-KAUNet: An Adaptive Dual Branch Kolmogorov-Arnold UNet for Retinal Vessel Segmentation
Hongyu Xu, Panpan Meng, Meng Wang, Dayu Hu, Liming Liang, Xiaoqi Sheng
Accurate segmentation of retinal vessels is crucial for the clinical diagnosis of numerous ophthalmic and systemic diseases. However, traditional Convolutional Neural Network (CNN) methods exhibit inherent limitations, struggling to capture long-range dependencies and complex nonlinear relationships. To address the above limitations, an Adaptive Dual Branch Kolmogorov-Arnold UNet (DB-KAUNet) is proposed for retinal vessel segmentation. In DB-KAUNet, we design a Heterogeneous Dual-Branch Encoder (HDBE) that features parallel CNN and Transformer pathways. The HDBE strategically interleaves standard CNN and Transformer blocks with novel KANConv and KAT blocks, enabling the model to form a comprehensive feature representation. To optimize feature processing, we integrate several critical components into the HDBE. First, a Cross-Branch Channel Interaction (CCI) module is embedded to facilitate efficient interaction of channel features between the parallel pathways. Second, an attention-based Spatial Feature Enhancement (SFE) module is employed to enhance spatial features and fuse the outputs from both branches. Building upon the SFE module, an advanced Spatial Feature Enhancement with Geometrically Adaptive Fusion (SFE-GAF) module is subsequently developed. In the SFE-GAF module, adaptive sampling is utilized to focus on true vessel morphology precisely. The adaptive process strengthens salient vascular features while significantly reducing background noise and computational overhead. Extensive experiments on the DRIVE, STARE, and CHASE_DB1 datasets validate that DB-KAUNet achieves leading segmentation performance and demonstrates exceptional robustness.
☆ ViT$^3$: Unlocking Test-Time Training in Vision
Dongchen Han, Yining Li, Tianyu Li, Zixuan Cao, Ziming Wang, Jun Song, Yu Cheng, Bo Zheng, Gao Huang
Test-Time Training (TTT) has recently emerged as a promising direction for efficient sequence modeling. TTT reformulates attention operation as an online learning problem, constructing a compact inner model from key-value pairs at test time. This reformulation opens a rich and flexible design space while achieving linear computational complexity. However, crafting a powerful visual TTT design remains challenging: fundamental choices for the inner module and inner training lack comprehensive understanding and practical guidelines. To bridge this critical gap, in this paper, we present a systematic empirical study of TTT designs for visual sequence modeling. From a series of experiments and analyses, we distill six practical insights that establish design principles for effective visual TTT and illuminate paths for future improvement. These findings culminate in the Vision Test-Time Training (ViT$^3$) model, a pure TTT architecture that achieves linear complexity and parallelizable computation. We evaluate ViT$^3$ across diverse visual tasks, including image classification, image generation, object detection, and semantic segmentation. Results show that ViT$^3$ consistently matches or outperforms advanced linear-complexity models (e.g., Mamba and linear attention variants) and effectively narrows the gap to highly optimized vision Transformers. We hope this study and the ViT$^3$ baseline can facilitate future work on visual TTT models. Code is available at https://github.com/LeapLabTHU/ViTTT.
☆ Generative Editing in the Joint Vision -Language Space for Zero-Shot Composed Image Retrieval
Xin Wang, Haipeng Zhang, Mang Li, Zhaohui Xia, Yueguo Chen, Yu Zhang, Chunyu Wei
Composed Image Retrieval (CIR) enables fine-grained visual search by combining a reference image with a textual modification. While supervised CIR methods achieve high accuracy, their reliance on costly triplet annotations motivates zero-shot solutions. The core challenge in zero-shot CIR (ZS-CIR) stems from a fundamental dilemma: existing text-centric or diffusion-based approaches struggle to effectively bridge the vision-language modality gap. To address this, we propose Fusion-Diff, a novel generative editing framework with high effectiveness and data efficiency designed for multimodal alignment. First, it introduces a multimodal fusion feature editing strategy within a joint vision-language (VL) space, substantially narrowing the modality gap. Second, to maximize data efficiency, the framework incorporates a lightweight Control-Adapter, enabling state-of-the-art performance through fine-tuning on only a limited-scale synthetic dataset of 200K samples. Extensive experiments on standard CIR benchmarks (CIRR, FashionIQ, and CIRCO) demonstrate that Fusion-Diff significantly outperforms prior zero-shot approaches. We further enhance the interpretability of our model by visualizing the fused multimodal representations.
☆ SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledge
Yumeng He, Ying Jiang, Jiayin Lu, Yin Yang, Chenfanfu Jiang
Articulated 3D objects are critical for embodied AI, robotics, and interactive scene understanding, yet creating simulation-ready assets remains labor-intensive and requires expert modeling of part hierarchies and motion structures. We introduce SPARK, a framework for reconstructing physically consistent, kinematic part-level articulated objects from a single RGB image. Given an input image, we first leverage VLMs to extract coarse URDF parameters and generate part-level reference images. We then integrate the part-image guidance and the inferred structure graph into a generative diffusion transformer to synthesize consistent part and complete shapes of articulated objects. To further refine the URDF parameters, we incorporate differentiable forward kinematics and differentiable rendering to optimize joint types, axes, and origins under VLM-generated open-state supervision. Extensive experiments show that SPARK produces high-quality, simulation-ready articulated assets across diverse categories, enabling downstream applications such as robotic manipulation and interaction modeling.
☆ Depth Matching Method Based on ShapeDTW for Oil-Based Mud Imager
Fengfeng Li, Zhou Feng, Hongliang Wu, Hao Zhang, Han Tian, Peng Liu, Lixin Yuan
In well logging operations using the oil-based mud (OBM) microresistivity imager, which employs an interleaved design with upper and lower pad sets, depth misalignment issues persist between the pad images even after velocity correction. This paper presents a depth matching method for borehole images based on the Shape Dynamic Time Warping (ShapeDTW) algorithm. The method extracts local shape features to construct a morphologically sensitive distance matrix, better preserving structural similarity between sequences during alignment. We implement this by employing a combined feature set of the one-dimensional Histogram of Oriented Gradients (HOG1D) and the original signal as the shape descriptor. Field test examples demonstrate that our method achieves precise alignment for images with complex textures, depth shifts, or local scaling. Furthermore, it provides a flexible framework for feature extension, allowing the integration of other descriptors tailored to specific geological features.
☆ Toward Content-based Indexing and Retrieval of Head and Neck CT with Abscess Segmentation
Thao Thi Phuong Dao, Tan-Cong Nguyen, Trong-Le Do, Truong Hoang Viet, Nguyen Chi Thanh, Huynh Nguyen Thuan, Do Vo Cong Nguyen, Minh-Khoi Pham, Mai-Khiem Tran, Viet-Tham Huynh, Trong-Thuan Nguyen, Trung-Nghia Le, Vo Thanh Toan, Tam V. Nguyen, Minh-Triet Tran, Thanh Dinh Le
Abscesses in the head and neck represent an acute infectious process that can potentially lead to sepsis or mortality if not diagnosed and managed promptly. Accurate detection and delineation of these lesions on imaging are essential for diagnosis, treatment planning, and surgical intervention. In this study, we introduce AbscessHeNe, a curated and comprehensively annotated dataset comprising 4,926 contrast-enhanced CT slices with clinically confirmed head and neck abscesses. The dataset is designed to facilitate the development of robust semantic segmentation models that can accurately delineate abscess boundaries and evaluate deep neck space involvement, thereby supporting informed clinical decision-making. To establish performance baselines, we evaluate several state-of-the-art segmentation architectures, including CNN, Transformer, and Mamba-based models. The highest-performing model achieved a Dice Similarity Coefficient of 0.39, Intersection-over-Union of 0.27, and Normalized Surface Distance of 0.67, indicating the challenges of this task and the need for further research. Beyond segmentation, AbscessHeNe is structured for future applications in content-based multimedia indexing and case-based retrieval. Each CT scan is linked with pixel-level annotations and clinical metadata, providing a foundation for building intelligent retrieval systems and supporting knowledge-driven clinical workflows. The dataset will be made publicly available at https://github.com/drthaodao3101/AbscessHeNe.git.
comment : The 2025 IEEE International Conference on Content-Based Multimedia Indexing (IEEE CBMI)
☆ RoleMotion: A Large-Scale Dataset towards Robust Scene-Specific Role-Playing Motion Synthesis with Fine-grained Descriptions
Junran Peng, Yiheng Huang, Silei Shen, Zeji Wei, Jingwei Yang, Baojie Wang, Yonghao He, Chuanchen Luo, Man Zhang, Xucheng Yin, Wei Sui
In this paper, we introduce RoleMotion, a large-scale human motion dataset that encompasses a wealth of role-playing and functional motion data tailored to fit various specific scenes. Existing text datasets are mainly constructed decentrally as amalgamation of assorted subsets that their data are nonfunctional and isolated to work together to cover social activities in various scenes. Also, the quality of motion data is inconsistent, and textual annotation lacks fine-grained details in these datasets. In contrast, RoleMotion is meticulously designed and collected with a particular focus on scenes and roles. The dataset features 25 classic scenes, 110 functional roles, over 500 behaviors, and 10296 high-quality human motion sequences of body and hands, annotated with 27831 fine-grained text descriptions. We build an evaluator stronger than existing counterparts, prove its reliability, and evaluate various text-to-motion methods on our dataset. Finally, we explore the interplay of motion generation of body and hands. Experimental results demonstrate the high-quality and functionality of our dataset on text-driven whole-body generation.
☆ MasHeNe: A Benchmark for Head and Neck CT Mass Segmentation using Window-Enhanced Mamba with Frequency-Domain Integration
Thao Thi Phuong Dao, Tan-Cong Nguyen, Nguyen Chi Thanh, Truong Hoang Viet, Trong-Le Do, Mai-Khiem Tran, Minh-Khoi Pham, Trung-Nghia Le, Minh-Triet Tran, Thanh Dinh Le
Head and neck masses are space-occupying lesions that can compress the airway and esophagus and may affect nerves and blood vessels. Available public datasets primarily focus on malignant lesions and often overlook other space-occupying conditions in this region. To address this gap, we introduce MasHeNe, an initial dataset of 3,779 contrast-enhanced CT slices that includes both tumors and cysts with pixel-level annotations. We also establish a benchmark using standard segmentation baselines and report common metrics to enable fair comparison. In addition, we propose the Windowing-Enhanced Mamba with Frequency integration (WEMF) model. WEMF applies tri-window enhancement to enrich the input appearance before feature extraction. It further uses multi-frequency attention to fuse information across skip connections within a U-shaped Mamba backbone. On MasHeNe, WEMF attains the best performance among evaluated methods, with a Dice of 70.45%, IoU of 66.89%, NSD of 72.33%, and HD95 of 5.12 mm. This model indicates stable and strong results on this challenging task. MasHeNe provides a benchmark for head-and-neck mass segmentation beyond malignancy-only datasets. The observed error patterns also suggest that this task remains challenging and requires further research. Our dataset and code are available at https://github.com/drthaodao3101/MasHeNe.git.
comment : The 14th International Symposium on Information and Communication Technology Conference SoICT 2025
☆ NavForesee: A Unified Vision -Language Wo rl d Model for Hierarchical Planning and Dual-Horizon Navigation Prediction
Fei Liu, Shichao Xie, Minghua Luo, Zedong Chu, Junjun Hu, Xiaolong Wu, Mu Xu
Embodied navigation for long-horizon tasks, guided by complex natural language instructions, remains a formidable challenge in artificial intelligence. Existing agents often struggle with robust long-term planning about unseen environments, leading to high failure rates. To address these limitations, we introduce NavForesee, a novel Vision-Language Model (VLM) that unifies high-level language planning and predictive world model imagination within a single, unified framework. Our approach empowers a single VLM to concurrently perform planning and predictive foresight. Conditioned on the full instruction and historical observations, the model is trained to understand the navigation instructions by decomposing the task, tracking its progress, and formulating the subsequent sub-goal. Simultaneously, it functions as a generative world model, providing crucial foresight by predicting short-term environmental dynamics and long-term navigation milestones. The VLM's structured plan guides its targeted prediction, while the imagined future provides rich context to inform the navigation actions, creating a powerful internal feedback loop of perception-planning/prediction-action. We demonstrate through extensive experiments on the R2R-CE and RxR-CE benchmark that NavForesee achieves highly competitive performance in complex scenarios. Our work highlights the immense potential of fusing explicit language planning with implicit spatiotemporal prediction, paving the way for more intelligent and capable embodied agents.
☆ FlashVGGT: Efficient and Scalable Visual Geometry Transformer s with Compressed Descriptor Attention
Zipeng Wang, Dan Xu
3D reconstruction from multi-view images is a core challenge in computer vision. Recently, feed-forward methods have emerged as efficient and robust alternatives to traditional per-scene optimization techniques. Among them, state-of-the-art models like the Visual Geometry Grounding Transformer (VGGT) leverage full self-attention over all image tokens to capture global relationships. However, this approach suffers from poor scalability due to the quadratic complexity of self-attention and the large number of tokens generated in long image sequences. In this work, we introduce FlashVGGT, an efficient alternative that addresses this bottleneck through a descriptor-based attention mechanism. Instead of applying dense global attention across all tokens, FlashVGGT compresses spatial information from each frame into a compact set of descriptor tokens. Global attention is then computed as cross-attention between the full set of image tokens and this smaller descriptor set, significantly reducing computational overhead. Moreover, the compactness of the descriptors enables online inference over long sequences via a chunk-recursive mechanism that reuses cached descriptors from previous chunks. Experimental results show that FlashVGGT achieves reconstruction accuracy competitive with VGGT while reducing inference time to just 9.3% of VGGT for 1,000 images, and scaling efficiently to sequences exceeding 3,000 images. Our project page is available at https://wzpscott.github.io/flashvggt_page/.
☆ Deep Unsupervised Anomaly Detection in Brain Imaging: Large-Scale Benchmark ing and Bias Analysis
Alexander Frotscher, Christian F. Baumgartner, Thomas Wolfers
Deep unsupervised anomaly detection in brain magnetic resonance imaging offers a promising route to identify pathological deviations without requiring lesion-specific annotations. Yet, fragmented evaluations, heterogeneous datasets, and inconsistent metrics have hindered progress toward clinical translation. Here, we present a large-scale, multi-center benchmark of deep unsupervised anomaly detection for brain imaging. The training cohort comprised 2,976 T1 and 2,972 T2-weighted scans from healthy individuals across six scanners, with ages ranging from 6 to 89 years. Validation used 92 scans to tune hyperparameters and estimate unbiased thresholds. Testing encompassed 2,221 T1w and 1,262 T2w scans spanning healthy datasets and diverse clinical cohorts. Across all algorithms, the Dice-based segmentation performance varied between 0.03 and 0.65, indicating substantial variability. To assess robustness, we systematically evaluated the impact of different scanners, lesion types and sizes, as well as demographics (age, sex). Reconstruction-based methods, particularly diffusion-inspired approaches, achieved the strongest lesion segmentation performance, while feature-based methods showed greater robustness under distributional shifts. However, systematic biases, such as scanner-related effects, were observed for the majority of algorithms, including that small and low-contrast lesions were missed more often, and that false positives varied with age and sex. Increasing healthy training data yields only modest gains, underscoring that current unsupervised anomaly detection frameworks are limited algorithmically rather than by data availability. Our benchmark establishes a transparent foundation for future research and highlights priorities for clinical translation, including image native pretraining, principled deviation measures, fairness-aware modeling, and robust domain adaptation.
☆ Diffusion Fuzzy System: Fuzzy Rule Guided Latent Multi-Path Diffusion Modeling
Hailong Yang, Te Zhang, Kup-sze Choi, Zhaohong Deng
Diffusion models have emerged as a leading technique for generating images due to their ability to create high-resolution and realistic images. Despite their strong performance, diffusion models still struggle in managing image collections with significant feature differences. They often fail to capture complex features and produce conflicting results. Research has attempted to address this issue by learning different regions of an image through multiple diffusion paths and then combining them. However, this approach leads to inefficient coordination among multiple paths and high computational costs. To tackle these issues, this paper presents a Diffusion Fuzzy System (DFS), a latent-space multi-path diffusion model guided by fuzzy rules. DFS offers several advantages. First, unlike traditional multi-path diffusion methods, DFS uses multiple diffusion paths, each dedicated to learning a specific class of image features. By assigning each path to a different feature type, DFS overcomes the limitations of multi-path models in capturing heterogeneous image features. Second, DFS employs rule-chain-based reasoning to dynamically steer the diffusion process and enable efficient coordination among multiple paths. Finally, DFS introduces a fuzzy membership-based latent-space compression mechanism to reduce the computational costs of multi-path diffusion effectively. We tested our method on three public datasets: LSUN Bedroom, LSUN Church, and MS COCO. The results show that DFS achieves more stable training and faster convergence than existing single-path and multi-path diffusion models. Additionally, DFS surpasses baseline models in both image quality and alignment between text and images, and also shows improved accuracy when comparing generated images to target references.
☆ QuantumCanvas: A Multimodal Benchmark for Visual Learning of Atomic Interactions
Can Polat, Erchin Serpedin, Mustafa Kurban, Hasan Kurban
Despite rapid advances in molecular and materials machine learning, most models still lack physical transferability: they fit correlations across whole molecules or crystals rather than learning the quantum interactions between atomic pairs. Yet bonding, charge redistribution, orbital hybridization, and electronic coupling all emerge from these two-body interactions that define local quantum fields in many-body systems. We introduce QuantumCanvas, a large-scale multimodal benchmark that treats two-body quantum systems as foundational units of matter. The dataset spans 2,850 element-element pairs, each annotated with 18 electronic, thermodynamic, and geometric properties and paired with ten-channel image representations derived from l- and m-resolved orbital densities, angular field transforms, co-occupancy maps, and charge-density projections. These physically grounded images encode spatial, angular, and electrostatic symmetries without explicit coordinates, providing an interpretable visual modality for quantum learning. Benchmarking eight architectures across 18 targets, we report mean absolute errors of 0.201 eV on energy gap using GATv2, 0.265 eV on HOMO and 0.274 eV on LUMO using EGNN. For energy-related quantities, DimeNet attains 2.27 eV total-energy MAE and 0.132 eV repulsive-energy MAE, while a multimodal fusion model achieves a 2.15 eV Mermin free-energy MAE. Pretraining on QuantumCanvas further improves convergence stability and generalization when fine-tuned on larger datasets such as QM9, MD17, and CrysMTM. By unifying orbital physics with vision-based representation learning, QuantumCanvas provides a principled and interpretable basis for learning transferable quantum interactions through coupled visual and numerical modalities. Dataset and model implementations are available at https://github.com/KurbanIntelligenceLab/QuantumCanvas.
☆ Semantic-aware Random Convolution and Source Matching for Domain Generalization in Medical Image Segmentation
Franz Thaler, Martin Urschler, Mateusz Kozinski, Matthias AF Gsell, Gernot Plank, Darko Stern
We tackle the challenging problem of single-source domain generalization (DG) for medical image segmentation. To this end, we aim for training a network on one domain (e.g., CT) and directly apply it to a different domain (e.g., MR) without adapting the model and without requiring images or annotations from the new domain during training. We propose a novel method for promoting DG when training deep segmentation networks, which we call SRCSM. During training, our method diversifies the source domain through semantic-aware random convolution, where different regions of a source image are augmented differently, based on their annotation labels. At test-time, we complement the randomization of the training domain via mapping the intensity of target domain images, making them similar to source domain data. We perform a comprehensive evaluation on a variety of cross-modality and cross-center generalization settings for abdominal, whole-heart and prostate segmentation, where we outperform previous DG techniques in a vast majority of experiments. Additionally, we also investigate our method when training on whole-heart CT or MR data and testing on the diastolic and systolic phase of cine MR data captured with different scanner hardware, where we make a step towards closing the domain gap in this even more challenging setting. Overall, our evaluation shows that SRCSM can be considered a new state-of-the-art in DG for medical image segmentation and, moreover, even achieves a segmentation performance that matches the performance of the in-domain baseline in several settings.
comment : Preprint submitted to Computer Methods and Programs in Biomedicine (currently under revision)
☆ ELVIS: Enhance Low-Light for Video Instance Segmentation in the Dark
Joanne Lin, Ruirui Lin, Yini Li, David Bull, Nantheera Anantrasirichai
Video instance segmentation (VIS) for low-light content remains highly challenging for both humans and machines alike, due to adverse imaging conditions including noise, blur and low-contrast. The lack of large-scale annotated datasets and the limitations of current synthetic pipelines, particularly in modeling temporal degradations, further hinder progress. Moreover, existing VIS methods are not robust to the degradations found in low-light videos and, as a result, perform poorly even when finetuned on low-light data. In this paper, we introduce \textbf{ELVIS} (\textbf{E}nhance \textbf{L}ow-light for \textbf{V}ideo \textbf{I}nstance \textbf{S}egmentation), a novel framework that enables effective domain adaptation of state-of-the-art VIS models to low-light scenarios. ELVIS comprises an unsupervised synthetic low-light video pipeline that models both spatial and temporal degradations, a calibration-free degradation profile synthesis network (VDP-Net) and an enhancement decoder head that disentangles degradations from content features. ELVIS improves performances by up to \textbf{+3.7AP} on the synthetic low-light YouTube-VIS 2019 dataset. Code will be released upon acceptance.
☆ A variational method for curve extraction with curvature-dependent energies
Majid Arthaud, Antonin Chambolle, Vincent Duval
We introduce a variational approach for extracting curves between a list of possible endpoints, based on the discretization of an energy and Smirnov's decomposition theorem for vector fields. It is used to design a bi-level minimization approach to automatically extract curves and 1D structures from an image, which is mostly unsupervised. We extend then the method to curvature-dependent energies, using a now classical lifting of the curves in the space of positions and orientations equipped with an appropriate sub-Riemanian or Finslerian metric.
☆ ChronosObserver: Taming 4D Wo rl d with Hyperspace Diffusion Sampling
Qisen Wang, Yifan Zhao, Peisen Shen, Jialu Li, Jia Li
Although prevailing camera-controlled video generation models can produce cinematic results, lifting them directly to the generation of 3D-consistent and high-fidelity time-synchronized multi-view videos remains challenging, which is a pivotal capability for taming 4D worlds. Some works resort to data augmentation or test-time optimization, but these strategies are constrained by limited model generalization and scalability issues. To this end, we propose ChronosObserver, a training-free method including World State Hyperspace to represent the spatiotemporal constraints of a 4D world scene, and Hyperspace Guided Sampling to synchronize the diffusion sampling trajectories of multiple views using the hyperspace. Experimental results demonstrate that our method achieves high-fidelity and 3D-consistent time-synchronized multi-view videos generation without training or fine-tuning for diffusion models.
☆ CourtMotion: Learning Event-Driven Motion Representations from Skeletal Data for Basketball
Omer Sela, Michael Chertok, Lior Wolf
This paper presents CourtMotion, a spatiotemporal modeling framework for analyzing and predicting game events and plays as they develop in professional basketball. Anticipating basketball events requires understanding both physical motion patterns and their semantic significance in the context of the game. Traditional approaches that use only player positions fail to capture crucial indicators such as body orientation, defensive stance, or shooting preparation motions. Our two-stage approach first processes skeletal tracking data through Graph Neural Networks to capture nuanced motion patterns, then employs a Transformer architecture with specialized attention mechanisms to model player interactions. We introduce event projection heads that explicitly connect player movements to basketball events like passes, shots, and steals, training the model to associate physical motion patterns with their tactical purposes. Experiments on NBA tracking data demonstrate significant improvements over position-only baselines: 35% reduction in trajectory prediction error compared to state-of-the-art position-based models and consistent performance gains across key basketball analytics tasks. The resulting pretrained model serves as a powerful foundation for multiple downstream tasks, with pick detection, shot taker identification, assist prediction, shot location classification, and shot type recognition demonstrating substantial improvements over existing methods.
☆ Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging
Kuangpu Guo, Yuhe Ding, Jian Liang, Zilei Wang, Ran He
Model merging has emerged as a promising paradigm for enabling multi-task capabilities without additional training. However, existing methods often experience substantial performance degradation compared with individually fine-tuned models, even on similar tasks, underscoring the need to preserve task-specific information. This paper proposes Decomposition, Thresholding, and Scaling (DTS), an approximation-based personalized merging framework that preserves task-specific information with minimal storage overhead. DTS first applies singular value decomposition to the task-specific information and retains only a small subset of singular values and vectors. It then introduces a novel thresholding strategy that partitions singular vector elements into groups and assigns a scaling factor to each group. To enable generalization to unseen tasks, we further extend DTS with a variant that fuses task-specific information in a data-free manner based on the semantic similarity of task characteristics. Extensive experiments demonstrate that DTS consistently outperforms state-of-the-art baselines while requiring only 1\% additional storage per task. Furthermore, experiments on unseen tasks show that the DTS variant achieves significantly better generalization performance. Our code is available at https://github.com/krumpguo/DTS.
☆ FastAnimate: Towards Learnable Template Construction and Pose Deformation for Fast 3D Human Avatar Animation
Jian Shu, Nanjie Yao, Gangjian Zhang, Junlong Ren, Yu Feng, Hao Wang
3D human avatar animation aims at transforming a human avatar from an arbitrary initial pose to a specified target pose using deformation algorithms. Existing approaches typically divide this task into two stages: canonical template construction and target pose deformation. However, current template construction methods demand extensive skeletal rigging and often produce artifacts for specific poses. Moreover, target pose deformation suffers from structural distortions caused by Linear Blend Skinning (LBS), which significantly undermines animation realism. To address these problems, we propose a unified learning-based framework to address both challenges in two phases. For the former phase, to overcome the inefficiencies and artifacts during template construction, we leverage a U-Net architecture that decouples texture and pose information in a feed-forward process, enabling fast generation of a human template. For the latter phase, we propose a data-driven refinement technique that enhances structural integrity. Extensive experiments show that our model delivers consistent performance across diverse poses with an optimal balance between efficiency and quality,surpassing state-of-the-art (SOTA) methods.
comment : 9 pages,4 figures
☆ Language-Guided Open-Wo rl d Anomaly Segmentation
Klara Reichard, Nikolas Brasch, Nassir Navab, Federico Tombari
Open-world and anomaly segmentation methods seek to enable autonomous driving systems to detect and segment both known and unknown objects in real-world scenes. However, existing methods do not assign semantically meaningful labels to unknown regions, and distinguishing and learning representations for unknown classes remains difficult. While open-vocabulary segmentation methods show promise in generalizing to novel classes, they require a fixed inference vocabulary and thus cannot be directly applied to anomaly segmentation where unknown classes are unconstrained. We propose Clipomaly, the first CLIP-based open-world and anomaly segmentation method for autonomous driving. Our zero-shot approach requires no anomaly-specific training data and leverages CLIP's shared image-text embedding space to both segment unknown objects and assign human-interpretable names to them. Unlike open-vocabulary methods, our model dynamically extends its vocabulary at inference time without retraining, enabling robust detection and naming of anomalies beyond common class definitions such as those in Cityscapes. Clipomaly achieves state-of-the-art performance on established anomaly segmentation benchmarks while providing interpretability and flexibility essential for practical deployment.
☆ ResDiT: Evoking the Intrinsic Resolution Scalability in Diffusion Transformer s
Yiyang Ma, Feng Zhou, Xuedan Yin, Pu Cao, Yonghao Dang, Jianqin Yin
Leveraging pre-trained Diffusion Transformers (DiTs) for high-resolution (HR) image synthesis often leads to spatial layout collapse and degraded texture fidelity. Prior work mitigates these issues with complex pipelines that first perform a base-resolution (i.e., training-resolution) denoising process to guide HR generation. We instead explore the intrinsic generative mechanisms of DiTs and propose ResDiT, a training-free method that scales resolution efficiently. We identify the core factor governing spatial layout, position embeddings (PEs), and show that the original PEs encode incorrect positional information when extrapolated to HR, which triggers layout collapse. To address this, we introduce a PE scaling technique that rectifies positional encoding under resolution changes. To further remedy low-fidelity details, we develop a local-enhancement mechanism grounded in base-resolution local attention. We design a patch-level fusion module that aggregates global and local cues, together with a Gaussian-weighted splicing strategy that eliminates grid artifacts. Comprehensive evaluations demonstrate that ResDiT consistently delivers high-fidelity, high-resolution image synthesis and integrates seamlessly with downstream tasks, including spatially controlled generation.
comment : 8 pages
☆ \textit{ViRectify}: A Challenging Benchmark for Video Reasoning Correction with Multimodal Large Language Models
Xusen Hei, Jiali Chen, Jinyu Yang, Mengchen Zhao, Yi Cai
As multimodal large language models (MLLMs) frequently exhibit errors in complex video reasoning scenarios, correcting these errors is critical for uncovering their weaknesses and improving performance. However, existing benchmarks lack systematic evaluation of MLLMs' ability to identify and correct these video reasoning errors. To bridge this gap, we propose \textit{ViRectify}, a comprehensive benchmark to evaluate their fine-grained correction capability. Through an AI-assisted annotation pipeline with human verification, we construct a dataset of over 30\textit{K} instances spanning dynamic perception, scientific reasoning, and embodied decision-making domains. In \textit{ViRectify}, we challenge MLLMs to perform step-wise error identification and generate rationales with key video evidence grounding. In addition, we further propose the trajectory evidence-driven correction framework, comprising step-wise error trajectory and reward modeling on visual evidence-grounded correction. It encourages the model to explicitly concentrate on error propagation and key timestamps for correction. Extensive evaluation across 16 advanced MLLMs demonstrates that our \textit{ViRectify} serves as a challenging testbed, where GPT-5 achieves only 31.94\% correction accuracy. Our framework enables a Qwen2.5-VL-7B to consistently outperform the variants of 72B on \textit{ViRectify}, showing the effectiveness of our approach. Further analysis uncovers systematic asymmetries in error correction across models, and our dataset is also a valuable data resource to perform reflection learning. We believe \textit{ViRectify} provides a new direction for comprehensively evaluating the advanced MLLMs in video reasoning.
comment : 22 pages, 11 figures
☆ MDiff4STR: Mask Diffusion Model for Scene Text Recognition AAAI 2026
Yongkun Du, Miaomiao Zhao, Songlin Fan, Zhineng Chen, Caiyan Jia, Yu-Gang Jiang
Mask Diffusion Models (MDMs) have recently emerged as a promising alternative to auto-regressive models (ARMs) for vision-language tasks, owing to their flexible balance of efficiency and accuracy. In this paper, for the first time, we introduce MDMs into the Scene Text Recognition (STR) task. We show that vanilla MDM lags behind ARMs in terms of accuracy, although it improves recognition efficiency. To bridge this gap, we propose MDiff4STR, a Mask Diffusion model enhanced with two key improvement strategies tailored for STR. Specifically, we identify two key challenges in applying MDMs to STR: noising gap between training and inference, and overconfident predictions during inference. Both significantly hinder the performance of MDMs. To mitigate the first issue, we develop six noising strategies that better align training with inference behavior. For the second, we propose a token-replacement noise mechanism that provides a non-mask noise type, encouraging the model to reconsider and revise overly confident but incorrect predictions. We conduct extensive evaluations of MDiff4STR on both standard and challenging STR benchmarks, covering diverse scenarios including irregular, artistic, occluded, and Chinese text, as well as whether the use of pretraining. Across these settings, MDiff4STR consistently outperforms popular STR models, surpassing state-of-the-art ARMs in accuracy, while maintaining fast inference with only three denoising steps. Code: https://github.com/Topdu/OpenOCR.
comment : Accepted by AAAI 2026 (Oral)
☆ Rice-VL: Evaluating Vision -Language Models for Cultural Understanding Across ASEAN Countries
Tushar Pranav, Eshan Pandey, Austria Lyka Diane Bala, Aman Chadha, Indriyati Atmosukarto, Donny Soh Cheng Lock
Vision-Language Models (VLMs) excel in multimodal tasks but often exhibit Western-centric biases, limiting their effectiveness in culturally diverse regions like Southeast Asia (SEA). To address this, we introduce RICE-VL, a novel benchmark evaluating VLM cultural understanding across 11 ASEAN countries. RICE-VL includes over 28,000 human-curated Visual Question Answering (VQA) samples -- covering True or False, Fill-in-the-Blank, and open-ended formats -- and 1,000 image-bounding box pairs for Visual Grounding, annotated by culturally informed experts across 14 sub-ground categories. We propose SEA-LAVE, an extension of the LAVE metric, assessing textual accuracy, cultural alignment, and country identification. Evaluations of six open- and closed-source VLMs reveal significant performance gaps in low-resource countries and abstract cultural domains. The Visual Grounding task tests models' ability to localize culturally significant elements in complex scenes, probing spatial and contextual accuracy. RICE-VL exposes limitations in VLMs' cultural comprehension and highlights the need for inclusive model development to better serve diverse global populations.
comment : 14 pages
☆ FRAMER: Frequency-Aligned Self-Distillation with Adaptive Modulation Leveraging Diffusion Priors for Real-Wo rl d Image Super-Resolution
Seungho Choi, Jeahun Sung, Jihyong Oh
Real-image super-resolution (Real-ISR) seeks to recover HR images from LR inputs with mixed, unknown degradations. While diffusion models surpass GANs in perceptual quality, they under-reconstruct high-frequency (HF) details due to a low-frequency (LF) bias and a depth-wise "low-first, high-later" hierarchy. We introduce FRAMER, a plug-and-play training scheme that exploits diffusion priors without changing the backbone or inference. At each denoising step, the final-layer feature map teaches all intermediate layers. Teacher and student feature maps are decomposed into LF/HF bands via FFT masks to align supervision with the model's internal frequency hierarchy. For LF, an Intra Contrastive Loss (IntraCL) stabilizes globally shared structure. For HF, an Inter Contrastive Loss (InterCL) sharpens instance-specific details using random-layer and in-batch negatives. Two adaptive modulators, Frequency-based Adaptive Weight (FAW) and Frequency-based Alignment Modulation (FAM), reweight per-layer LF/HF signals and gate distillation by current similarity. Across U-Net and DiT backbones (e.g., Stable Diffusion 2, 3), FRAMER consistently improves PSNR/SSIM and perceptual metrics (LPIPS, NIQE, MANIQA, MUSIQ). Ablations validate the final-layer teacher and random-layer negatives.
comment : Comments: Please visit our project page at https://cmlab-korea.github.io/FRAMER/
☆ PointNet4D: A Lightweight 4D Point Cloud Video Backbone for Online and Offline Perception in Robotic Applications WACV2026
Yunze Liu, Zifan Wang, Peiran Wu, Jiayang Ao
Understanding dynamic 4D environments-3D space evolving over time-is critical for robotic and interactive systems. These applications demand systems that can process streaming point cloud video in real-time, often under resource constraints, while also benefiting from past and present observations when available. However, current 4D backbone networks rely heavily on spatiotemporal convolutions and Transformers, which are often computationally intensive and poorly suited to real-time applications. We propose PointNet4D, a lightweight 4D backbone optimized for both online and offline settings. At its core is a Hybrid Mamba-Transformer temporal fusion block, which integrates the efficient state-space modeling of Mamba and the bidirectional modeling power of Transformers. This enables PointNet4D to handle variable-length online sequences efficiently across different deployment scenarios. To enhance temporal understanding, we introduce 4DMAP, a frame-wise masked auto-regressive pretraining strategy that captures motion cues across frames. Our extensive evaluations across 9 tasks on 7 datasets, demonstrating consistent improvements across diverse domains. We further demonstrate PointNet4D's utility by building two robotic application systems: 4D Diffusion Policy and 4D Imitation Learning, achieving substantial gains on the RoboTwin and HandoverSim benchmarks.
comment : Accepted by WACV2026
☆ Reversible Inversion for Training-Free Exemplar-guided Image Editing
Yuke Li, Lianli Gao, Ji Zhang, Pengpeng Zeng, Lichuan Xiang, Hongkai Wen, Heng Tao Shen, Jingkuan Song
Exemplar-guided Image Editing (EIE) aims to modify a source image according to a visual reference. Existing approaches often require large-scale pre-training to learn relationships between the source and reference images, incurring high computational costs. As a training-free alternative, inversion techniques can be used to map the source image into a latent space for manipulation. However, our empirical study reveals that standard inversion is sub-optimal for EIE, leading to poor quality and inefficiency. To tackle this challenge, we introduce \textbf{Reversible Inversion ({ReInversion})} for effective and efficient EIE. Specifically, ReInversion operates as a two-stage denoising process, which is first conditioned on the source image and subsequently on the reference. Besides, we introduce a Mask-Guided Selective Denoising (MSD) strategy to constrain edits to target regions, preserving the structural consistency of the background. Both qualitative and quantitative comparisons demonstrate that our ReInversion method achieves state-of-the-art EIE performance with the lowest computational overhead.
☆ Textured Geometry Evaluation: Perceptual 3D Textured Shape Metric via 3D Latent-Geometry Network AAAI26
Tianyu Luan, Xuelu Feng, Zixin Zhu, Phani Nuney, Sheng Liu, Xuan Gong, David Doermann, Chunming Qiao, Junsong Yuan
Textured high-fidelity 3D models are crucial for games, AR/VR, and film, but human-aligned evaluation methods still fall behind despite recent advances in 3D reconstruction and generation. Existing metrics, such as Chamfer Distance, often fail to align with how humans evaluate the fidelity of 3D shapes. Recent learning-based metrics attempt to improve this by relying on rendered images and 2D image quality metrics. However, these approaches face limitations due to incomplete structural coverage and sensitivity to viewpoint choices. Moreover, most methods are trained on synthetic distortions, which differ significantly from real-world distortions, resulting in a domain gap. To address these challenges, we propose a new fidelity evaluation method that is based directly on 3D meshes with texture, without relying on rendering. Our method, named Textured Geometry Evaluation TGE, jointly uses the geometry and color information to calculate the fidelity of the input textured mesh with comparison to a reference colored shape. To train and evaluate our metric, we design a human-annotated dataset with real-world distortions. Experiments show that TGE outperforms rendering-based and geometry-only methods on real-world distortion dataset.
comment : Accepted by AAAI26
☆ SRAM: Shape-Realism Alignment Metric for No Reference 3D Shape Evaluation AAAI2026
Sheng Liu, Tianyu Luan, Phani Nuney, Xuelu Feng, Junsong Yuan
3D generation and reconstruction techniques have been widely used in computer games, film, and other content creation areas. As the application grows, there is a growing demand for 3D shapes that look truly realistic. Traditional evaluation methods rely on a ground truth to measure mesh fidelity. However, in many practical cases, a shape's realism does not depend on having a ground truth reference. In this work, we propose a Shape-Realism Alignment Metric that leverages a large language model (LLM) as a bridge between mesh shape information and realism evaluation. To achieve this, we adopt a mesh encoding approach that converts 3D shapes into the language token space. A dedicated realism decoder is designed to align the language model's output with human perception of realism. Additionally, we introduce a new dataset, RealismGrading, which provides human-annotated realism scores without the need for ground truth shapes. Our dataset includes shapes generated by 16 different algorithms on over a dozen objects, making it more representative of practical 3D shape distributions. We validate our metric's performance and generalizability through k-fold cross-validation across different objects. Experimental results show that our metric correlates well with human perceptions and outperforms existing methods, and has good generalizability.
comment : Accepted by AAAI2026
☆ BlinkBud: Detecting Hazards from Behind via Sampled Monocular 3D Detection on a Single Earbud
Yunzhe Li, Jiajun Yan, Yuzhou Wei, Kechen Liu, Yize Zhao, Chong Zhang, Hongzi Zhu, Li Lu, Shan Chang, Minyi Guo
Failing to be aware of speeding vehicles approaching from behind poses a huge threat to the road safety of pedestrians and cyclists. In this paper, we propose BlinkBud, which utilizes a single earbud and a paired phone to online detect hazardous objects approaching from behind of a user. The core idea is to accurately track visually identified objects utilizing a small number of sampled camera images taken from the earbud. To minimize the power consumption of the earbud and the phone while guaranteeing the best tracking accuracy, a novel 3D object tracking algorithm is devised, integrating both a Kalman filter based trajectory estimation scheme and an optimal image sampling strategy based on reinforcement learning. Moreover, the impact of constant user head movements on the tracking accuracy is significantly eliminated by leveraging the estimated pitch and yaw angles to correct the object depth estimation and align the camera coordinate system to the user's body coordinate system, respectively. We implement a prototype BlinkBud system and conduct extensive real-world experiments. Results show that BlinkBud is lightweight with ultra-low mean power consumptions of 29.8 mW and 702.6 mW on the earbud and smartphone, respectively, and can accurately detect hazards with a low average false positive ratio (FPR) and false negative ratio (FNR) of 4.90% and 1.47%, respectively.
comment : This is the author-accepted version of the paper published in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), Vol. 9, No. 4, Article 191, 2025. Final published version: https://doi.org/10.1145/3770707
☆ OpenBox: Annotate Any Bounding Boxes in 3D NeurIPS 2025
In-Jae Lee, Mungyeom Kim, Kwonyoung Ryu, Pierre Musacchio, Jaesik Park
Unsupervised and open-vocabulary 3D object detection has recently gained attention, particularly in autonomous driving, where reducing annotation costs and recognizing unseen objects are critical for both safety and scalability. However, most existing approaches uniformly annotate 3D bounding boxes, ignore objects' physical states, and require multiple self-training iterations for annotation refinement, resulting in suboptimal quality and substantial computational overhead. To address these challenges, we propose OpenBox, a two-stage automatic annotation pipeline that leverages a 2D vision foundation model. In the first stage, OpenBox associates instance-level cues from 2D images processed by a vision foundation model with the corresponding 3D point clouds via cross-modal instance alignment. In the second stage, it categorizes instances by rigidity and motion state, then generates adaptive bounding boxes with class-specific size statistics. As a result, OpenBox produces high-quality 3D bounding box annotations without requiring self-training. Experiments on the Waymo Open Dataset, the Lyft Level 5 Perception dataset, and the nuScenes dataset demonstrate improved accuracy and efficiency over baselines.
comment : Accepted by NeurIPS 2025
☆ Handwritten Text Recognition for Low Resource Languages
Sayantan Dey, Alireza Alaei, Partha Pratim Roy
Despite considerable progress in handwritten text recognition, paragraph-level handwritten text recognition, especially in low-resource languages, such as Hindi, Urdu and similar scripts, remains a challenging problem. These languages, often lacking comprehensive linguistic resources, require special attention to develop robust systems for accurate optical character recognition (OCR). This paper introduces BharatOCR, a novel segmentation-free paragraph-level handwritten Hindi and Urdu text recognition. We propose a ViT-Transformer Decoder-LM architecture for handwritten text recognition, where a Vision Transformer (ViT) extracts visual features, a Transformer decoder generates text sequences, and a pre-trained language model (LM) refines the output to improve accuracy, fluency, and coherence. Our model utilizes a Data-efficient Image Transformer (DeiT) model proposed for masked image modeling in this research work. In addition, we adopt a RoBERTa architecture optimized for masked language modeling (MLM) to enhance the linguistic comprehension and generative capabilities of the proposed model. The transformer decoder generates text sequences from visual embeddings. This model is designed to iteratively process a paragraph image line by line, called implicit line segmentation. The proposed model was evaluated using our custom dataset ('Parimal Urdu') and ('Parimal Hindi'), introduced in this research work, as well as two public datasets. The proposed model achieved benchmark results in the NUST-UHWR, PUCIT-OUHL, and Parimal-Urdu datasets, achieving character recognition rates of 96.24%, 92.05%, and 94.80%, respectively. The model also provided benchmark results using the Hindi dataset achieving a character recognition rate of 80.64%. The results obtained from our proposed model indicated that it outperformed several state-of-the-art Urdu text recognition methods.
comment : 21 Pages
☆ InternVideo-Next: Towards General Video Foundation Models without Video-Text Super vision
Chenting Wang, Yuhan Zhu, Yicheng Xu, Jiange Yang, Ziang Yan, Yali Wang, Yi Wang, Limin Wang
Large-scale video-text pretraining achieves strong performance but depends on noisy, synthetic captions with limited semantic coverage, often overlooking implicit world knowledge such as object motion, 3D geometry, and physical cues. In contrast, masked video modeling (MVM) directly exploits spatiotemporal structures but trails text-supervised methods on general tasks. We find this gap arises from overlooked architectural issues: pixel-level reconstruction struggles with convergence and its low-level requirement often conflicts with semantics, while latent prediction often encourages shortcut learning. To address these, we disentangle the traditional encoder-decoder design into an Encoder-Predictor-Decoder (EPD) framework, where the predictor acts as a latent world model, and propose InternVideo-Next, a two-stage pretraining scheme that builds a semantically consistent yet detail-preserving latent space for this world model. First, conventional linear decoder in pixel MVM enforces the predictor output latent to be linearly projected to, thus separable in pixel space, causing the conflict with semantic abstraction. Our Stage 1 proposes a conditional diffusion decoder and injects reliable image-level semantic priors to enhance semantics and convergence, thus bridging pixel-level fidelity with high-level semantic abstraction. Stage 2 further learns world knowledge by predicting frozen Stage 1 targets within this space, mitigating shortcut learning. Trained on public, unlabeled videos, InternVideo-Next achieves state-of-the-art results across benchmarks and provides a scalable path toward general video representation learning.
☆ EvalTalker: Learning to Evaluate Real-Portrait-Driven Multi-Subject Talking Humans
Yingjie Zhou, Xilei Zhu, Siyu Ren, Ziyi Zhao, Ziwen Wang, Farong Wen, Yu Zhou, Jiezhang Cao, Xiongkuo Min, Fengjiao Chen, Xiaoyu Li, Xuezhi Cao, Guangtao Zhai, Xiaohong Liu
Speech-driven Talking Human (TH) generation, commonly known as "Talker," currently faces limitations in multi-subject driving capabilities. Extending this paradigm to "Multi-Talker," capable of animating multiple subjects simultaneously, introduces richer interactivity and stronger immersion in audiovisual communication. However, current Multi-Talkers still exhibit noticeable quality degradation caused by technical limitations, resulting in suboptimal user experiences. To address this challenge, we construct THQA-MT, the first large-scale Multi-Talker-generated Talking Human Quality Assessment dataset, consisting of 5,492 Multi-Talker-generated THs (MTHs) from 15 representative Multi-Talkers using 400 real portraits collected online. Through subjective experiments, we analyze perceptual discrepancies among different Multi-Talkers and identify 12 common types of distortion. Furthermore, we introduce EvalTalker, a novel TH quality assessment framework. This framework possesses the ability to perceive global quality, human characteristics, and identity consistency, while integrating Qwen-Sync to perceive multimodal synchrony. Experimental results demonstrate that EvalTalker achieves superior correlation with subjective scores, providing a robust foundation for future research on high-quality Multi-Talker generation and evaluation.
☆ AlignVid: Training-Free Attention Scaling for Semantic Fidelity in Text-Guided Image-to-Video Generation
Yexin Liu, Wen-Jie Shu, Zile Huang, Haoze Zheng, Yueze Wang, Manyuan Zhang, Ser-Nam Lim, Harry Yang
Text-guided image-to-video (TI2V) generation has recently achieved remarkable progress, particularly in maintaining subject consistency and temporal coherence. However, existing methods still struggle to adhere to fine-grained prompt semantics, especially when prompts entail substantial transformations of the input image (e.g., object addition, deletion, or modification), a shortcoming we term semantic negligence. In a pilot study, we find that applying a Gaussian blur to the input image improves semantic adherence. Analyzing attention maps, we observe clearer foreground-background separation. From an energy perspective, this corresponds to a lower-entropy cross-attention distribution. Motivated by this, we introduce AlignVid, a training-free framework with two components: (i) Attention Scaling Modulation (ASM), which directly reweights attention via lightweight Q or K scaling, and (ii) Guidance Scheduling (GS), which applies ASM selectively across transformer blocks and denoising steps to reduce visual quality degradation. This minimal intervention improves prompt adherence while limiting aesthetic degradation. In addition, we introduce OmitI2V to evaluate semantic negligence in TI2V generation, comprising 367 human-annotated samples that span addition, deletion, and modification scenarios. Extensive experiments demonstrate that AlignVid can enhance semantic fidelity.
☆ Optimizing Stroke Risk Prediction: A Machine Learning Pipeline Combining ROS-Balanced Ensembles and XAI
A S M Ahsanul Sarkar Akib, Raduana Khawla, Abdul Hasib
Stroke is a major cause of death and permanent impairment, making it a major worldwide health concern. For prompt intervention and successful preventative tactics, early risk assessment is essential. To address this challenge, we used ensemble modeling and explainable AI (XAI) techniques to create an interpretable machine learning framework for stroke risk prediction. A thorough evaluation of 10 different machine learning models using 5-fold cross-validation across several datasets was part of our all-inclusive strategy, which also included feature engineering and data pretreatment (using Random Over-Sampling (ROS) to solve class imbalance). Our optimized ensemble model (Random Forest + ExtraTrees + XGBoost) performed exceptionally well, obtaining a strong 99.09% accuracy on the Stroke Prediction Dataset (SPD). We improved the model's transparency and clinical applicability by identifying three important clinical variables using LIME-based interpretability analysis: age, hypertension, and glucose levels. Through early prediction, this study highlights how combining ensemble learning with explainable AI (XAI) can deliver highly accurate and interpretable stroke risk assessment. By enabling data-driven prevention and personalized clinical decisions, our framework has the potential to transform stroke prediction and cardiovascular risk management.
☆ TagSplat: Topology-Aware Gaussian Splatting for Dynamic Mesh Modeling and Tracking
Hanzhi Guo, Dongdong Weng, Mo Su, Yixiao Chen, Xiaonuo Dongye, Chenyu Xu
Topology-consistent dynamic model sequences are essential for applications such as animation and model editing. However, existing 4D reconstruction methods face challenges in generating high-quality topology-consistent meshes. To address this, we propose a topology-aware dynamic reconstruction framework based on Gaussian Splatting. We introduce a Gaussian topological structure that explicitly encodes spatial connectivity. This structure enables topology-aware densification and pruning, preserving the manifold consistency of the Gaussian representation. Temporal regularization terms further ensure topological coherence over time, while differentiable mesh rasterization improves mesh quality. Experimental results demonstrate that our method reconstructs topology-consistent mesh sequences with significantly higher accuracy than existing approaches. Moreover, the resulting meshes enable precise 3D keypoint tracking. Project page: https://haza628.github.io/tagSplat/
☆ Panda: Self-distillation of Reusable Sensor-level Representations for High Energy Physics
Samuel Young, Kazuhiro Terao
Liquid argon time projection chambers (LArTPCs) provide dense, high-fidelity 3D measurements of particle interactions and underpin current and future neutrino and rare-event experiments. Physics reconstruction typically relies on complex detector-specific pipelines that use tens of hand-engineered pattern recognition algorithms or cascades of task-specific neural networks that require extensive, labeled simulation that requires a careful, time-consuming calibration process. We introduce \textbf{Panda}, a model that learns reusable sensor-level representations directly from raw unlabeled LArTPC data. Panda couples a hierarchical sparse 3D encoder with a multi-view, prototype-based self-distillation objective. On a simulated dataset, Panda substantially improves label efficiency and reconstruction quality, beating the previous state-of-the-art semantic segmentation model with 1,000$\times$ fewer labels. We also show that a single set-prediction head 1/20th the size of the backbone with no physical priors trained on frozen outputs from Panda can result in particle identification that is comparable with state-of-the-art (SOTA) reconstruction tools. Full fine-tuning further improves performance across all tasks.
comment : 23 pages, 15 figures, preprint. Project page at https://youngsm.com/panda/
☆ Rethinking Intracranial Aneurysm Vessel Segmentation: A Perspective from Computational Fluid Dynamics Applications
Feiyang Xiao, Yichi Zhang, Xigui Li, Yuanye Zhou, Chen Jiang, Xin Guo, Limei Han, Yuxin Li, Fengping Zhu, Yuan Cheng
The precise segmentation of intracranial aneurysms and their parent vessels (IA-Vessel) is a critical step for hemodynamic analyses, which mainly depends on computational fluid dynamics (CFD). However, current segmentation methods predominantly focus on image-based evaluation metrics, often neglecting their practical effectiveness in subsequent CFD applications. To address this deficiency, we present the Intracranial Aneurysm Vessel Segmentation (IAVS) dataset, the first comprehensive, multi-center collection comprising 641 3D MRA images with 587 annotations of aneurysms and IA-Vessels. In addition to image-mask pairs, IAVS dataset includes detailed hemodynamic analysis outcomes, addressing the limitations of existing datasets that neglect topological integrity and CFD applicability. To facilitate the development and evaluation of clinically relevant techniques, we construct two evaluation benchmarks including global localization of aneurysms (Stage I) and fine-grained segmentation of IA-Vessel (Stage II) and develop a simple and effective two-stage framework, which can be used as a out-of-the-box method and strong baseline. For comprehensive evaluation of applicability of segmentation results, we establish a standardized CFD applicability evaluation system that enables the automated and consistent conversion of segmentation masks into CFD models, offering an applicability-focused assessment of segmentation outcomes. The dataset, code, and model will be public available at https://github.com/AbsoluteResonance/IAVS.
comment : 18 pages, 5 figures
☆ FOD-S2R: A FOD Dataset for Sim2Real Transfer Learning based Object Detection
Ashish Vashist, Qiranul Saadiyean, Suresh Sundaram, Chandra Sekhar Seelamantula
Foreign Object Debris (FOD) within aircraft fuel tanks presents critical safety hazards including fuel contamination, system malfunctions, and increased maintenance costs. Despite the severity of these risks, there is a notable lack of dedicated datasets for the complex, enclosed environments found inside fuel tanks. To bridge this gap, we present a novel dataset, FOD-S2R, composed of real and synthetic images of the FOD within a simulated aircraft fuel tank. Unlike existing datasets that focus on external or open-air environments, our dataset is the first to systematically evaluate the effectiveness of synthetic data in enhancing the real-world FOD detection performance in confined, closed structures. The real-world subset consists of 3,114 high-resolution HD images captured in a controlled fuel tank replica, while the synthetic subset includes 3,137 images generated using Unreal Engine. The dataset is composed of various Field of views (FOV), object distances, lighting conditions, color, and object size. Prior research has demonstrated that synthetic data can reduce reliance on extensive real-world annotations and improve the generalizability of vision models. Thus, we benchmark several state-of-the-art object detection models and demonstrate that introducing synthetic data improves the detection accuracy and generalization to real-world conditions. These experiments demonstrate the effectiveness of synthetic data in enhancing the model performance and narrowing the Sim2Real gap, providing a valuable foundation for developing automated FOD detection systems for aviation maintenance.
comment : 8 pages, 11 figures
☆ TokenPure: Watermark Removal through Tokenized Appearance and Structural Guidance
Pei Yang, Yepeng Liu, Kelly Peng, Yuan Gao, Yiren Song
In the digital economy era, digital watermarking serves as a critical basis for ownership proof of massive replicable content, including AI-generated and other virtual assets. Designing robust watermarks capable of withstanding various attacks and processing operations is even more paramount. We introduce TokenPure, a novel Diffusion Transformer-based framework designed for effective and consistent watermark removal. TokenPure solves the trade-off between thorough watermark destruction and content consistency by leveraging token-based conditional reconstruction. It reframes the task as conditional generation, entirely bypassing the initial watermark-carrying noise. We achieve this by decomposing the watermarked image into two complementary token sets: visual tokens for texture and structural tokens for geometry. These tokens jointly condition the diffusion process, enabling the framework to synthesize watermark-free images with fine-grained consistency and structural integrity. Comprehensive experiments show that TokenPure achieves state-of-the-art watermark removal and reconstruction fidelity, substantially outperforming existing baselines in both perceptual quality and consistency.
☆ IVCR-200K: A Large-Scale Multi-turn Dialogue Benchmark for Interactive Video Corpus Retrieval SIGIR2025
Ning Han, Yawen Zeng, Shaohua Long, Chengqing Li, Sijie Yang, Dun Tan, Jianfeng Dong, Jingjing Chen
In recent years, significant developments have been made in both video retrieval and video moment retrieval tasks, which respectively retrieve complete videos or moments for a given text query. These advancements have greatly improved user satisfaction during the search process. However, previous work has failed to establish meaningful "interaction" between the retrieval system and the user, and its one-way retrieval paradigm can no longer fully meet the personalization and dynamic needs of at least 80.8\% of users. In this paper, we introduce the Interactive Video Corpus Retrieval (IVCR) task, a more realistic setting that enables multi-turn, conversational, and realistic interactions between the user and the retrieval system. To facilitate research on this challenging task, we introduce IVCR-200K, a high-quality, bilingual, multi-turn, conversational, and abstract semantic dataset that supports video retrieval and even moment retrieval. Furthermore, we propose a comprehensive framework based on multi-modal large language models (MLLMs) to help users interact in several modes with more explainable solutions. The extensive experiments demonstrate the effectiveness of our dataset and framework.
comment : Accepted by SIGIR2025
☆ Lost in Distortion: Uncovering the Domain Gap Between Computer Vision and Brain Imaging - A Study on Pretrain ing for Age Prediction
Yanteng Zhang, Songheng Li, Zeyu Shen, Qizhen Lan, Lipei Zhang, Yang Liu, Vince Calhoun
Large-scale brain imaging datasets provide unprecedented opportunities for developing domain foundation models through pretraining. However, unlike natural image datasets in computer vision, these neuroimaging data often exhibit high heterogeneity in quality, ranging from well-structured scans to severely distorted or incomplete brain volumes. This raises a fundamental question: can noise or low-quality scans contribute meaningfully to pretraining, or do they instead hinder model learning? In this study, we systematically explore the role of data quality level in pretraining and its impact on downstream tasks. Specifically, we perform pretraining on datasets with different quality levels and perform fine-tuning for brain age prediction on external cohorts. Our results show significant performance differences across quality levels, revealing both opportunities and limitations. We further discuss the gap between computer vision practices and clinical neuroimaging standards, emphasizing the necessity of domain-aware curation to ensure trusted and generalizable domain-specific foundation models.
☆ Gaussian Swaying: Surface-Based Framework for Aerodynamic Simulation with 3D Gaussians WACV 2026
Hongru Yan, Xiang Zhang, Zeyuan Chen, Fangyin Wei, Zhuowen Tu
Branches swaying in the breeze, flags rippling in the wind, and boats rocking on the water all show how aerodynamics shape natural motion -- an effect crucial for realism in vision and graphics. In this paper, we present Gaussian Swaying, a surface-based framework for aerodynamic simulation using 3D Gaussians. Unlike mesh-based methods that require costly meshing, or particle-based approaches that rely on discrete positional data, Gaussian Swaying models surfaces continuously with 3D Gaussians, enabling efficient and fine-grained aerodynamic interaction. Our framework unifies simulation and rendering on the same representation: Gaussian patches, which support force computation for dynamics while simultaneously providing normals for lightweight shading. Comprehensive experiments on both synthetic and real-world datasets across multiple metrics demonstrate that Gaussian Swaying achieves state-of-the-art performance and efficiency, offering a scalable approach for realistic aerodynamic scene simulation.
comment : Accepted to WACV 2026
☆ DCText: Scheduled Attention Masking for Visual Text Generation via Divide-and-Conquer Strategy WACV 2026
Jaewoo Song, Jooyoung Choi, Kanghyun Baek, Sangyub Lee, Daemin Park, Sungroh Yoon
Despite recent text-to-image models achieving highfidelity text rendering, they still struggle with long or multiple texts due to diluted global attention. We propose DCText, a training-free visual text generation method that adopts a divide-and-conquer strategy, leveraging the reliable short-text generation of Multi-Modal Diffusion Transformers. Our method first decomposes a prompt by extracting and dividing the target text, then assigns each to a designated region. To accurately render each segment within their regions while preserving overall image coherence, we introduce two attention masks - Text-Focus and Context-Expansion - applied sequentially during denoising. Additionally, Localized Noise Initialization further improves text accuracy and region alignment without increasing computational cost. Extensive experiments on single- and multisentence benchmarks show that DCText achieves the best text accuracy without compromising image quality while also delivering the lowest generation latency.
comment : Accepted to WACV 2026
☆ TBT-Former: Learning Temporal Boundary Distributions for Action Localization
Thisara Rathnayaka, Uthayasanker Thayasivam
Temporal Action Localization (TAL) remains a fundamental challenge in video understanding, aiming to identify the start time, end time, and category of all action instances within untrimmed videos. While recent single-stage, anchor-free models like ActionFormer have set a high standard by leveraging Transformers for temporal reasoning, they often struggle with two persistent issues: the precise localization of actions with ambiguous or "fuzzy" temporal boundaries and the effective fusion of multi-scale contextual information. In this paper, we introduce the Temporal Boundary Transformer (TBT-Former), a new architecture that directly addresses these limitations. TBT-Former enhances the strong ActionFormer baseline with three core contributions: (1) a higher-capacity scaled Transformer backbone with an increased number of attention heads and an expanded Multi-Layer Perceptron (MLP) dimension for more powerful temporal feature extraction; (2) a cross-scale feature pyramid network (FPN) that integrates a top-down pathway with lateral connections, enabling richer fusion of high-level semantics and low-level temporal details; and (3) a novel boundary distribution regression head. Inspired by the principles of Generalized Focal Loss (GFL), this new head recasts the challenging task of boundary regression as a more flexible probability distribution learning problem, allowing the model to explicitly represent and reason about boundary uncertainty. Within the paradigm of Transformer-based architectures, TBT-Former advances the formidable benchmark set by its predecessors, establishing a new level of performance on the highly competitive THUMOS14 and EPIC-Kitchens 100 datasets, while remaining competitive on the large-scale ActivityNet-1.3. Our code is available at https://github.com/aaivu/In21-S7-CS4681-AML-Research-Projects/tree/main/projects/210536K-Multi-Modal-Learning_Video-Understanding
comment : 8 pages, 6 figures
☆ EGG-Fusion: Efficient 3D Reconstruction with Geometry-aware Gaussian Surfel on the Fly SIGGRAPH
Xiaokun Pan, Zhenzhe Li, Zhichao Ye, Hongjia Zhai, Guofeng Zhang
Real-time 3D reconstruction is a fundamental task in computer graphics. Recently, differentiable-rendering-based SLAM system has demonstrated significant potential, enabling photorealistic scene rendering through learnable scene representations such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). Current differentiable rendering methods face dual challenges in real-time computation and sensor noise sensitivity, leading to degraded geometric fidelity in scene reconstruction and limited practicality. To address these challenges, we propose a novel real-time system EGG-Fusion, featuring robust sparse-to-dense camera tracking and a geometry-aware Gaussian surfel mapping module, introducing an information filter-based fusion method that explicitly accounts for sensor noise to achieve high-precision surface reconstruction. The proposed differentiable Gaussian surfel mapping effectively models multi-view consistent surfaces while enabling efficient parameter optimization. Extensive experimental results demonstrate that the proposed system achieves a surface reconstruction error of 0.6\textit{cm} on standardized benchmark datasets including Replica and ScanNet++, representing over 20\% improvement in accuracy compared to state-of-the-art (SOTA) GS-based methods. Notably, the system maintains real-time processing capabilities at 24 FPS, establishing it as one of the most accurate differentiable-rendering-based real-time reconstruction systems. Project Page: https://zju3dv.github.io/eggfusion/
comment : SIGGRAPH ASIA 2025
☆ Diffusion Model in Latent Space for Medical Image Segmentation Task
Huynh Trinh Ngoc, Toan Nguyen Hai, Ba Luong Son, Long Tran Quoc
Medical image segmentation is crucial for clinical diagnosis and treatment planning. Traditional methods typically produce a single segmentation mask, failing to capture inherent uncertainty. Recent generative models enable the creation of multiple plausible masks per image, mimicking the collaborative interpretation of several clinicians. However, these approaches remain computationally heavy. We propose MedSegLatDiff, a diffusion based framework that combines a variational autoencoder (VAE) with a latent diffusion model for efficient medical image segmentation. The VAE compresses the input into a low dimensional latent space, reducing noise and accelerating training, while the diffusion process operates directly in this compact representation. We further replace the conventional MSE loss with weighted cross entropy in the VAE mask reconstruction path to better preserve tiny structures such as small nodules. MedSegLatDiff is evaluated on ISIC-2018 (skin lesions), CVC-Clinic (polyps), and LIDC-IDRI (lung nodules). It achieves state of the art or highly competitive Dice and IoU scores while simultaneously generating diverse segmentation hypotheses and confidence maps. This provides enhanced interpretability and reliability compared to deterministic baselines, making the model particularly suitable for clinical deployment.
☆ Supervised Contrastive Machine Unlearning of Background Bias in Sonar Image Classification with Fine-Grained Explainable AI
Kamal Basha S, Athira Nambiar
Acoustic sonar image analysis plays a critical role in object detection and classification, with applications in both civilian and defense domains. Despite the availability of real and synthetic datasets, existing AI models that achieve high accuracy often over-rely on seafloor features, leading to poor generalization. To mitigate this issue, we propose a novel framework that integrates two key modules: (i) a Targeted Contrastive Unlearning (TCU) module, which extends the traditional triplet loss to reduce seafloor-induced background bias and improve generalization, and (ii) the Unlearn to Explain Sonar Framework (UESF), which provides visual insights into what the model has deliberately forgotten while adapting the LIME explainer to generate more faithful and localized attributions for unlearning evaluation. Extensive experiments across both real and synthetic sonar datasets validate our approach, demonstrating significant improvements in unlearning effectiveness, model robustness, and interpretability.
comment : Accepted to CVIP 2025
☆ nnMobileNet++: Towards Efficient Hybrid Networks for Retinal Image Analysis
Xin Li, Wenhui Zhu, Xuanzhao Dong, Hao Wang, Yujian Xiong, Oana Dumitrascu, Yalin Wang
Retinal imaging is a critical, non-invasive modality for the early detection and monitoring of ocular and systemic diseases. Deep learning, particularly convolutional neural networks (CNNs), has significant progress in automated retinal analysis, supporting tasks such as fundus image classification, lesion detection, and vessel segmentation. As a representative lightweight network, nnMobileNet has demonstrated strong performance across multiple retinal benchmarks while remaining computationally efficient. However, purely convolutional architectures inherently struggle to capture long-range dependencies and model the irregular lesions and elongated vascular patterns that characterize on retinal images, despite the critical importance of vascular features for reliable clinical diagnosis. To further advance this line of work and extend the original vision of nnMobileNet, we propose nnMobileNet++, a hybrid architecture that progressively bridges convolutional and transformer representations. The framework integrates three key components: (i) dynamic snake convolution for boundary-aware feature extraction, (ii) stage-specific transformer blocks introduced after the second down-sampling stage for global context modeling, and (iii) retinal image pretraining to improve generalization. Experiments on multiple public retinal datasets for classification, together with ablation studies, demonstrate that nnMobileNet++ achieves state-of-the-art or highly competitive accuracy while maintaining low computational cost, underscoring its potential as a lightweight yet effective framework for retinal image analysis.
☆ ViscNet: Vision -Based In-line Viscometry for Fluid Mixing Process
Jongwon Sohn, Juhyeon Moon, Hyunjoon Jung, Jaewook Nam
Viscosity measurement is essential for process monitoring and autonomous laboratory operation, yet conventional viscometers remain invasive and require controlled laboratory environments that differ substantially from real process conditions. We present a computer-vision-based viscometer that infers viscosity by exploiting how a fixed background pattern becomes optically distorted as light refracts through the mixing-driven, continuously deforming free surface. Under diverse lighting conditions, the system achieves a mean absolute error of 0.113 in log m2 s^-1 units for regression and reaches up to 81% accuracy in viscosity-class prediction. Although performance declines for classes with closely clustered viscosity values, a multi-pattern strategy improves robustness by providing enriched visual cues. To ensure sensor reliability, we incorporate uncertainty quantification, enabling viscosity predictions with confidence estimates. This stand-off viscometer offers a practical, automation-ready alternative to existing viscometry methods.
☆ Efficient Training of Diffusion Mixture-of-Experts Models: A Practical Recipe
Yahui Liu, Yang Yue, Jingyuan Zhang, Chenxi Sun, Yang Zhou, Wencong Zeng, Ruiming Tang, Guorui Zhou
Recent efforts on Diffusion Mixture-of-Experts (MoE) models have primarily focused on developing more sophisticated routing mechanisms. However, we observe that the underlying architectural configuration space remains markedly under-explored. Inspired by the MoE design paradigms established in large language models (LLMs), we identify a set of crucial architectural factors for building effective Diffusion MoE models--including DeepSeek-style expert modules, alternative intermediate widths, varying expert counts, and enhanced attention positional encodings. Our systematic study reveals that carefully tuning these configurations is essential for unlocking the full potential of Diffusion MoE models, often yielding gains that exceed those achieved by routing innovations alone. Through extensive experiments, we present novel architectures that can be efficiently applied to both latent and pixel-space diffusion frameworks, which provide a practical and efficient training recipe that enables Diffusion MoE models to surpass strong baselines while using equal or fewer activated parameters. All code and models are publicly available at: https://github.com/yhlleo/EfficientMoE.
comment : 9 pages, 7 figures
☆ TRivia: Self-supervised Fine-tuning of Vision -Language Models for Table Recognition
Junyuan Zhang, Bin Wang, Qintong Zhang, Fan Wu, Zichen Wen, Jialin Lu, Junjie Shan, Ziqi Zhao, Shuya Yang, Ziling Wang, Ziyang Miao, Huaping Zhong, Yuhang Zang, Xiaoyi Dong, Ka-Ho Chow, Conghui He
Table recognition (TR) aims to transform table images into semi-structured representations such as HTML or Markdown. As a core component of document parsing, TR has long relied on supervised learning, with recent efforts dominated by fine-tuning vision-language models (VLMs) using labeled data. While VLMs have brought TR to the next level, pushing performance further demands large-scale labeled data that is costly to obtain. Consequently, although proprietary models have continuously pushed the performance boundary, open-source models, often trained with limited resources and, in practice, the only viable option for many due to privacy regulations, still lag far behind. To bridge this gap, we introduce TRivia, a self-supervised fine-tuning method that enables pretrained VLMs to learn TR directly from unlabeled table images in the wild. Built upon Group Relative Policy Optimization, TRivia automatically identifies unlabeled samples that most effectively facilitate learning and eliminates the need for human annotations through a question-answering-based reward mechanism. An attention-guided module generates diverse questions for each table image, and the ability to interpret the recognition results and answer them correctly provides feedback to optimize the TR model. This closed-loop process allows the TR model to autonomously learn to recognize, structure, and reason over tables without labeled data. Leveraging this pipeline, we present TRivia-3B, an open-sourced, compact, and state-of-the-art TR model that surpasses existing systems (e.g., Gemini 2.5 Pro, MinerU2.5) on three popular benchmarks. Model and code are released at: https://github.com/opendatalab/TRivia
☆ Generative Adversarial Gumbel MCTS for Abstract Visual Composition Generation
Zirui Zhao, Boye Niu, David Hsu, Wee Sun Lee
We study abstract visual composition, in which identity is primarily determined by the spatial configuration and relations among a small set of geometric primitives (e.g., parts, symmetry, topology). They are invariant primarily to texture and photorealistic detail. Composing such structures from fixed components under geometric constraints and vague goal specification (such as text) is non-trivial due to combinatorial placement choices, limited data, and discrete feasibility (overlap-free, allowable orientations), which create a sparse solution manifold ill-suited to purely statistical pixel-space generators. We propose a constraint-guided framework that combines explicit geometric reasoning with neural semantics. An AlphaGo-style search enforces feasibility, while a fine-tuned vision-language model scores semantic alignment as reward signals. Our algorithm uses a policy network as a heuristic in Monte-Carlo Tree Search and fine-tunes the network via search-generated plans. Inspired by the Generative Adversarial Network, we use the generated instances for adversarial reward refinement. Over time, the generation should approach the actual data more closely when the reward model cannot distinguish between generated instances and ground-truth. In the Tangram Assembly task, our approach yields higher validity and semantic fidelity than diffusion and auto-regressive baselines, especially as constraints tighten.
☆ PSR: Scaling Multi-Subject Personalized Image Generation with Pairwise Subject-Consistency Rewards
Shulei Wang, Longhui Wei, Xin He, Jianbo Ouyang, Hui Lu, Zhou Zhao, Qi Tian
Personalized generation models for a single subject have demonstrated remarkable effectiveness, highlighting their significant potential. However, when extended to multiple subjects, existing models often exhibit degraded performance, particularly in maintaining subject consistency and adhering to textual prompts. We attribute these limitations to the absence of high-quality multi-subject datasets and refined post-training strategies. To address these challenges, we propose a scalable multi-subject data generation pipeline that leverages powerful single-subject generation models to construct diverse and high-quality multi-subject training data. Through this dataset, we first enable single-subject personalization models to acquire knowledge of synthesizing multi-image and multi-subject scenarios. Furthermore, to enhance both subject consistency and text controllability, we design a set of Pairwise Subject-Consistency Rewards and general-purpose rewards, which are incorporated into a refined reinforcement learning stage. To comprehensively evaluate multi-subject personalization, we introduce a new benchmark that assesses model performance using seven subsets across three dimensions. Extensive experiments demonstrate the effectiveness of our approach in advancing multi-subject personalized image generation. Github Link: https://github.com/wang-shulei/PSR
☆ S$^2$-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance
Beining Xu, Siting Zhu, Zhao Jin, Junxian Li, Hesheng Wang
3D Visual Grounding (3DVG) focuses on locating objects in 3D scenes based on natural language descriptions, serving as a fundamental task for embodied AI and robotics. Recent advances in Multi-modal Large Language Models (MLLMs) have motivated research into extending them to 3DVG. However, MLLMs primarily process 2D visual inputs and struggle with understanding 3D spatial structure of scenes solely from these limited perspectives. Existing methods mainly utilize viewpoint-dependent rendering of reconstructed point clouds to provide explicit structural guidance for MLLMs in 3DVG tasks, leading to inefficiency and limited spatial reasoning. To address this issue, we propose S$^2$-MLLM, an efficient framework that enhances spatial reasoning in MLLMs through implicit spatial reasoning. We introduce a spatial guidance strategy that leverages the structure awareness of feed-forward 3D reconstruction. By acquiring 3D structural understanding during training, our model can implicitly reason about 3D scenes without relying on inefficient point cloud reconstruction. Moreover, we propose a structure-enhanced module (SE), which first employs intra-view and inter-view attention mechanisms to capture dependencies within views and correspondences across views. The module further integrates multi-level position encoding to associate visual representations with spatial positions and viewpoint information, enabling more accurate structural understanding. Extensive experiments demonstrate that S$^2$-MLLM unifies superior performance, generalization, and efficiency, achieving significant performance over existing methods across the ScanRefer, Nr3D, and Sr3D datasets. Code will be available upon acceptance.
comment : 18 pages, 9 figures
☆ M4-BLIP: Advancing Multi-Modal Media Manipulation Detection through Face-Enhanced Local Analysis
Hang Wu, Ke Sun, Jiayi Ji, Xiaoshuai Sun, Rongrong Ji
In the contemporary digital landscape, multi-modal media manipulation has emerged as a significant societal threat, impacting the reliability and integrity of information dissemination. Current detection methodologies in this domain often overlook the crucial aspect of localized information, despite the fact that manipulations frequently occur in specific areas, particularly in facial regions. In response to this critical observation, we propose the M4-BLIP framework. This innovative framework utilizes the BLIP-2 model, renowned for its ability to extract local features, as the cornerstone for feature extraction. Complementing this, we incorporate local facial information as prior knowledge. A specially designed alignment and fusion module within M4-BLIP meticulously integrates these local and global features, creating a harmonious blend that enhances detection accuracy. Furthermore, our approach seamlessly integrates with Large Language Models (LLM), significantly improving the interpretability of the detection outcomes. Extensive quantitative and visualization experiments validate the effectiveness of our framework against the state-of-the-art competitors.
comment : 12 pages, 6 figures
☆ Closing the Approximation Gap of Partial AUC Optimization: A Tale of Two Formulations
Yangbangyan Jiang, Qianqian Xu, Huiyang Shao, Zhiyong Yang, Shilong Bao, Xiaochun Cao, Qingming Huang
As a variant of the Area Under the ROC Curve (AUC), the partial AUC (PAUC) focuses on a specific range of false positive rate (FPR) and/or true positive rate (TPR) in the ROC curve. It is a pivotal evaluation metric in real-world scenarios with both class imbalance and decision constraints. However, selecting instances within these constrained intervals during its calculation is NP-hard, and thus typically requires approximation techniques for practical resolution. Despite the progress made in PAUC optimization over the last few years, most existing methods still suffer from uncontrollable approximation errors or a limited scalability when optimizing the approximate PAUC objectives. In this paper, we close the approximation gap of PAUC optimization by presenting two simple instance-wise minimax reformulations: one with an asymptotically vanishing gap, the other with the unbiasedness at the cost of more variables. Our key idea is to first establish an equivalent instance-wise problem to lower the time complexity, simplify the complicated sample selection procedure by threshold learning, and then apply different smoothing techniques. Equipped with an efficient solver, the resulting algorithms enjoy a linear per-iteration computational complexity w.r.t. the sample size and a convergence rate of $O(ε^{-1/3})$ for typical one-way and two-way PAUCs. Moreover, we provide a tight generalization bound of our minimax reformulations. The result explicitly demonstrates the impact of the TPR/FPR constraints $α$/$β$ on the generalization and exhibits a sharp order of $\tilde{O}(α^{-1}\n_+^{-1} + β^{-1}\n_-^{-1})$. Finally, extensive experiments on several benchmark datasets validate the strength of our proposed methods.
☆ TabletopGen: Instance-Level Interactive 3D Tabletop Scene Generation from Text or Single Image
Ziqian Wang, Yonghao He, Licheng Yang, Wei Zou, Hongxuan Ma, Liu Liu, Wei Sui, Yuxin Guo, Hu Su
Generating high-fidelity, physically interactive 3D simulated tabletop scenes is essential for embodied AI--especially for robotic manipulation policy learning and data synthesis. However, current text- or image-driven 3D scene generation methods mainly focus on large-scale scenes, struggling to capture the high-density layouts and complex spatial relations that characterize tabletop scenes. To address these challenges, we propose TabletopGen, a training-free, fully automatic framework that generates diverse, instance-level interactive 3D tabletop scenes. TabletopGen accepts a reference image as input, which can be synthesized by a text-to-image model to enhance scene diversity. We then perform instance segmentation and completion on the reference to obtain per-instance images. Each instance is reconstructed into a 3D model followed by canonical coordinate alignment. The aligned 3D models then undergo pose and scale estimation before being assembled into a collision-free, simulation-ready tabletop scene. A key component of our framework is a novel pose and scale alignment approach that decouples the complex spatial reasoning into two stages: a Differentiable Rotation Optimizer for precise rotation recovery and a Top-view Spatial Alignment mechanism for robust translation and scale estimation, enabling accurate 3D reconstruction from 2D reference. Extensive experiments and user studies show that TabletopGen achieves state-of-the-art performance, markedly surpassing existing methods in visual fidelity, layout accuracy, and physical plausibility, capable of generating realistic tabletop scenes with rich stylistic and spatial diversity. Our code will be publicly available.
comment : Project page: https://d-robotics-ai-lab.github.io/TabletopGen.project/
☆ First On-Orbit Demonstration of a Geospatial Foundation Model
Andrew Du, Roberto Del Prete, Alejandro Mousist, Nick Manser, Fabrice Marre, Andrew Barton, Carl Seubert, Gabriele Meoni, Tat-Jun Chin
Geospatial foundation models (GeoFMs) promise broad generalisation capacity for Earth observation (EO) tasks, particularly under data-limited conditions. However, their large size poses a barrier to deployment on resource-constrained space hardware. To address this, we present compact variants of a Vision Transformer (ViT)-based GeoFM that preserve downstream task performance while enabling onboard execution. Evaluation across five downstream tasks and validation in two representative flight environments show that model compression and domain adaptation are critical to reducing size and resource demands while maintaining high performance under operational conditions. We further demonstrate reliable on-orbit inference with the IMAGIN-e payload aboard the International Space Station. These results establish a pathway from large GeoFMs to flight-ready, resource-efficient deployments, expanding the feasibility of onboard AI for EO missions.
☆ VSRD++: Autolabeling for 3D Object Detection via Instance-Aware Volumetric Silhouette Rendering
Zihua Liu, Hiroki Sakuma, Masatoshi Okutomi
Monocular 3D object detection is a fundamental yet challenging task in 3D scene understanding. Existing approaches heavily depend on supervised learning with extensive 3D annotations, which are often acquired from LiDAR point clouds through labor-intensive labeling processes. To tackle this problem, we propose VSRD++, a novel weakly supervised framework for monocular 3D object detection that eliminates the reliance on 3D annotations and leverages neural-field-based volumetric rendering with weak 2D supervision. VSRD++ consists of a two-stage pipeline: multi-view 3D autolabeling and subsequent monocular 3D detector training. In the multi-view autolabeling stage, object surfaces are represented as signed distance fields (SDFs) and rendered as instance masks via the proposed instance-aware volumetric silhouette rendering. To optimize 3D bounding boxes, we decompose each instance's SDF into a cuboid SDF and a residual distance field (RDF) that captures deviations from the cuboid. To address the geometry inconsistency commonly observed in volume rendering methods applied to dynamic objects, we model the dynamic objects by including velocity into bounding box attributes as well as assigning confidence to each pseudo-label. Moreover, we also employ a 3D attribute initialization module to initialize the dynamic bounding box parameters. In the monocular 3D object detection phase, the optimized 3D bounding boxes serve as pseudo labels for training monocular 3D object detectors. Extensive experiments on the KITTI-360 dataset demonstrate that VSRD++ significantly outperforms existing weakly supervised approaches for monocular 3D object detection on both static and dynamic scenes. Code is available at https://github.com/Magicboomliu/VSRD_plus_plus
comment : arXiv admin note: text overlap with arXiv:2404.00149
☆ Real-Time On-the-Go Annotation Framework Using YOLO for Automated Dataset Generation CEC 2025
Mohamed Abdallah Salem, Ahmed Harb Rabia
Efficient and accurate annotation of datasets remains a significant challenge for deploying object detection models such as You Only Look Once (YOLO) in real-world applications, particularly in agriculture where rapid decision-making is critical. Traditional annotation techniques are labor-intensive, requiring extensive manual labeling post data collection. This paper presents a novel real-time annotation approach leveraging YOLO models deployed on edge devices, enabling immediate labeling during image capture. To comprehensively evaluate the efficiency and accuracy of our proposed system, we conducted an extensive comparative analysis using three prominent YOLO architectures (YOLOv5, YOLOv8, YOLOv12) under various configurations: single-class versus multi-class annotation and pretrained versus scratch-based training. Our analysis includes detailed statistical tests and learning dynamics, demonstrating significant advantages of pretrained and single-class configurations in terms of model convergence, performance, and robustness. Results strongly validate the feasibility and effectiveness of our real-time annotation framework, highlighting its capability to drastically reduce dataset preparation time while maintaining high annotation quality.
comment : Copyright 2025 IEEE. This is the author's version of the work that has been accepted for publication in Proceedings of the 5. Interdisciplinary Conference on Electrics and Computer (INTCEC 2025) 15-16 September 2025, Chicago-USA. The final version of record is available at: https://doi.org/10.1109/INTCEC65580.2025.11256048
☆ DPAC: Distribution-Preserving Adversarial Control for Diffusion Sampling
Han-Jin Lee, Han-Ju Lee, Jin-Seong Kim, Seok-Hwan Choi
Adversarially guided diffusion sampling often achieves the target class, but sample quality degrades as deviations between the adversarially controlled and nominal trajectories accumulate. We formalize this degradation as a path-space Kullback-Leibler divergence(path-KL) between controlled and nominal (uncontrolled) diffusion processes, thereby showing via Girsanov's theorem that it exactly equals the control energy. Building on this stochastic optimal control (SOC) view, we theoretically establish that minimizing this path-KL simultaneously tightens upper bounds on both the 2-Wasserstein distance and Fréchet Inception Distance (FID), revealing a principled connection between adversarial control energy and perceptual fidelity. From a variational perspective, we derive a first-order optimality condition for the control: among all directions that yield the same classification gain, the component tangent to iso-(log-)density surfaces (i.e., orthogonal to the score) minimizes path-KL, whereas the normal component directly increases distributional drift. This leads to DPAC (Distribution-Preserving Adversarial Control), a diffusion guidance rule that projects adversarial gradients onto the tangent space defined by the generative score geometry. We further show that in discrete solvers, the tangent projection cancels the O(Δt) leading error term in the Wasserstein distance, achieving an O(Δt^2) quality gap; moreover, it remains second-order robust to score or metric approximation. Empirical studies on ImageNet-100 validate the theoretical predictions, confirming that DPAC achieves lower FID and estimated path-KL at matched attack success rates.
☆ Open-Set Domain Adaptation Under Background Distribution Shift: Challenges and A Provably Efficient Solution
Shravan Chaudhari, Yoav Wald, Suchi Saria
As we deploy machine learning systems in the real world, a core challenge is to maintain a model that is performant even as the data shifts. Such shifts can take many forms: new classes may emerge that were absent during training, a problem known as open-set recognition, and the distribution of known categories may change. Guarantees on open-set recognition are mostly derived under the assumption that the distribution of known classes, which we call \emph{the background distribution}, is fixed. In this paper we develop \ours{}, a method that is guaranteed to solve open-set recognition even in the challenging case where the background distribution shifts. We prove that the method works under benign assumptions that the novel class is separable from the non-novel classes, and provide theoretical guarantees that it outperforms a representative baseline in a simplified overparameterized setting. We develop techniques to make \ours{} scalable and robust, and perform comprehensive empirical evaluations on image and text data. The results show that \ours{} significantly outperforms existing open-set recognition methods under background shift. Moreover, we provide new insights into how factors such as the size of the novel class influences performance, an aspect that has not been extensively explored in prior work.
☆ Bridging the Gap: Toward Cognitive Autonomy in Artificial Intelligence
Noorbakhsh Amiri Golilarz, Sindhuja Penchala, Shahram Rahimi
Artificial intelligence has advanced rapidly across perception, language, reasoning, and multimodal domains. Yet despite these achievements, modern AI systems remain fundamentally limited in their ability to self-monitor, self-correct, and regulate their behavior autonomously in dynamic contexts. This paper identifies and analyzes seven core deficiencies that constrain contemporary AI models: the absence of intrinsic self-monitoring, lack of meta-cognitive awareness, fixed and non-adaptive learning mechanisms, inability to restructure goals, lack of representational maintenance, insufficient embodied feedback, and the absence of intrinsic agency. Alongside identifying these limitations, we also outline a forward-looking perspective on how AI may evolve beyond them through architectures that mirror neurocognitive principles. We argue that these structural limitations prevent current architectures, including deep learning and transformer-based systems, from achieving robust generalization, lifelong adaptability, and real-world autonomy. Drawing on a comparative analysis of artificial systems and biological cognition [7], and integrating insights from AI research, cognitive science, and neuroscience, we outline how these capabilities are absent in current models and why scaling alone cannot resolve them. We conclude by advocating for a paradigmatic shift toward cognitively grounded AI (cognitive autonomy) capable of self-directed adaptation, dynamic representation management, and intentional, goal-oriented behavior, paired with reformative oversight mechanisms [8] that ensure autonomous systems remain interpretable, governable, and aligned with human values.
☆ Progressive Image Restoration via Text-Conditioned Video Generation
Peng Kang, Xijun Wang, Yu Yuan
Recent text-to-video models have demonstrated strong temporal generation capabilities, yet their potential for image restoration remains underexplored. In this work, we repurpose CogVideo for progressive visual restoration tasks by fine-tuning it to generate restoration trajectories rather than natural video motion. Specifically, we construct synthetic datasets for super-resolution, deblurring, and low-light enhancement, where each sample depicts a gradual transition from degraded to clean frames. Two prompting strategies are compared: a uniform text prompt shared across all samples, and a scene-specific prompting scheme generated via LLaVA multi-modal LLM and refined with ChatGPT. Our fine-tuned model learns to associate temporal progression with restoration quality, producing sequences that improve perceptual metrics such as PSNR, SSIM, and LPIPS across frames. Extensive experiments show that CogVideo effectively restores spatial detail and illumination consistency while maintaining temporal coherence. Moreover, the model generalizes to real-world scenarios on the ReLoBlur dataset without additional training, demonstrating strong zero-shot robustness and interpretability through temporal restoration.
comment : First two authors contributed equally to this work. IEEE ICNC Accepted
☆ Spatiotemporal Pyramid Flow Matching for Climate Emulation
Jeremy Andrew Irvin, Jiaqi Han, Zikui Wang, Abdulaziz Alharbi, Yufei Zhao, Nomin-Erdene Bayarsaikhan, Daniele Visioni, Andrew Y. Ng, Duncan Watson-Parris
Generative models have the potential to transform the way we emulate Earth's changing climate. Previous generative approaches rely on weather-scale autoregression for climate emulation, but this is inherently slow for long climate horizons and has yet to demonstrate stable rollouts under nonstationary forcings. Here, we introduce Spatiotemporal Pyramid Flows (SPF), a new class of flow matching approaches that model data hierarchically across spatial and temporal scales. Inspired by cascaded video models, SPF partitions the generative trajectory into a spatiotemporal pyramid, progressively increasing spatial resolution to reduce computation and coupling each stage with an associated timescale to enable direct sampling at any temporal level in the pyramid. This design, together with conditioning each stage on prescribed physical forcings (e.g., greenhouse gases or aerosols), enables efficient, parallel climate emulation at multiple timescales. On ClimateBench, SPF outperforms strong flow matching baselines and pre-trained models at yearly and monthly timescales while offering fast sampling, especially at coarser temporal levels. To scale SPF, we curate ClimateSuite, the largest collection of Earth system simulations to date, comprising over 33,000 simulation-years across ten climate models and the first dataset to include simulations of climate interventions. We find that the scaled SPF model demonstrates good generalization to held-out scenarios across climate models. Together, SPF and ClimateSuite provide a foundation for accurate, efficient, probabilistic climate emulation across temporal scales and realistic future scenarios. Data and code is publicly available at https://github.com/stanfordmlgroup/spf .
☆ Exploring the Potentials of Spiking Neural Networks for Image Deraining AAAI2026
Shuang Chen, Tomas Krajnik, Farshad Arvin, Amir Atapour-Abarghouei
Biologically plausible and energy-efficient frameworks such as Spiking Neural Networks (SNNs) have not been sufficiently explored in low-level vision tasks. Taking image deraining as an example, this study addresses the representation of the inherent high-pass characteristics of spiking neurons, specifically in image deraining and innovatively proposes the Visual LIF (VLIF) neuron, overcoming the obstacle of lacking spatial contextual understanding present in traditional spiking neurons. To tackle the limitation of frequency-domain saturation inherent in conventional spiking neurons, we leverage the proposed VLIF to introduce the Spiking Decomposition and Enhancement Module and the lightweight Spiking Multi-scale Unit for hierarchical multi-scale representation learning. Extensive experiments across five benchmark deraining datasets demonstrate that our approach significantly outperforms state-of-the-art SNN-based deraining methods, achieving this superior performance with only 13\% of their energy consumption. These findings establish a solid foundation for deploying SNNs in high-performance, energy-efficient low-level vision tasks.
comment : Accepted By AAAI2026
☆ PhishSnap: Image-Based Phishing Detection Using Perceptual Hashing
Md Abdul Ahad Minhaz, Zannatul Zahan Meem, Md. Shohrab Hossain
Phishing remains one of the most prevalent online threats, exploiting human trust to harvest sensitive credentials. Existing URL- and HTML-based detection systems struggle against obfuscation and visual deception. This paper presents \textbf{PhishSnap}, a privacy-preserving, on-device phishing detection system leveraging perceptual hashing (pHash). Implemented as a browser extension, PhishSnap captures webpage screenshots, computes visual hashes, and compares them against legitimate templates to identify visually similar phishing attempts. A \textbf{2024 dataset of 10,000 URLs} (70\%/20\%/10\% train/validation/test) was collected from PhishTank and Netcraft. Due to security takedowns, a subset of phishing pages was unavailable, reducing dataset diversity. The system achieved \textbf{0.79 accuracy}, \textbf{0.76 precision}, and \textbf{0.78 recall}, showing that visual similarity remains a viable anti-phishing measure. The entire inference process occurs locally, ensuring user privacy and minimal latency.
comment : IEE Standard Formatting, 3 pages, 3 figures
☆ See, Hear, and Understand: Benchmark ing Audiovisual Human Speech Understanding in Multimodal Large Language Models
Le Thien Phuc Nguyen, Zhuoran Yu, Samuel Low Yu Hang, Subin An, Jeongik Lee, Yohan Ban, SeungEun Chung, Thanh-Huy Nguyen, JuWan Maeng, Soochahn Lee, Yong Jae Lee
Multimodal large language models (MLLMs) are expected to jointly interpret vision, audio, and language, yet existing video benchmarks rarely assess fine-grained reasoning about human speech. Many tasks remain visually solvable or only coarsely evaluate speech, offering limited insight into whether models can align who speaks, what is said, and when it occurs. We introduce AV-SpeakerBench, a curated benchmark of 3,212 multiple-choice questions focused on speaker-centric audiovisual reasoning in real-world videos. It features: (1) a speaker-centered formulation that treats speakers-not scenes-as the core reasoning unit; (2) fusion-grounded question design embedding audiovisual dependencies into question semantics; and (3) expert-curated annotations ensuring temporal precision and cross-modal validity. Comprehensive evaluations show that the Gemini family consistently outperforms open-source systems, with Gemini 2.5 Pro achieving the best results. Among open models, Qwen3-Omni-30B approaches Gemini 2.0 Flash but remains far behind Gemini 2.5 Pro, primarily due to weaker audiovisual fusion rather than visual perception. We believe AV-SpeakerBench establishes a rigorous foundation for advancing fine-grained audiovisual reasoning in future multimodal systems.
comment : preprint
☆ Towards Unified Video Quality Assessment
Chen Feng, Tianhao Peng, Fan Zhang, David Bull
Recent works in video quality assessment (VQA) typically employ monolithic models that typically predict a single quality score for each test video. These approaches cannot provide diagnostic, interpretable feedback, offering little insight into why the video quality is degraded. Most of them are also specialized, format-specific metrics rather than truly ``generic" solutions, as they are designed to learn a compromised representation from disparate perceptual domains. To address these limitations, this paper proposes Unified-VQA, a framework that provides a single, unified quality model applicable to various distortion types within multiple video formats by recasting generic VQA as a Diagnostic Mixture-of-Experts (MoE) problem. Unified-VQA employs multiple ``perceptual experts'' dedicated to distinct perceptual domains. A novel multi-proxy expert training strategy is designed to optimize each expert using a ranking-inspired loss, guided by the most suitable proxy metric for its domain. We also integrated a diagnostic multi-task head into this framework to generate a global quality score and an interpretable multi-dimensional artifact vector, which is optimized using a weakly-supervised learning strategy, leveraging the known properties of the large-scale training database generated for this work. With static model parameters (without retraining or fine-tuning), Unified-VQA demonstrates consistent and superior performance compared to over 18 benchmark methods for both generic VQA and diagnostic artifact detection tasks across 17 databases containing diverse streaming artifacts in HD, UHD, HDR and HFR formats. This work represents an important step towards practical, actionable, and interpretable video quality assessment.
comment : 8 pages, 3 figures
☆ Multifractal Recalibration of Neural Networks for Medical Imaging Segmentation
Miguel L. Martins, Miguel T. Coimbra, Francesco Renna
Multifractal analysis has revealed regularities in many self-seeding phenomena, yet its use in modern deep learning remains limited. Existing end-to-end multifractal methods rely on heavy pooling or strong feature-space decimation, which constrain tasks such as semantic segmentation. Motivated by these limitations, we introduce two inductive priors: Monofractal and Multifractal Recalibration. These methods leverage relationships between the probability mass of the exponents and the multifractal spectrum to form statistical descriptions of encoder embeddings, implemented as channel-attention functions in convolutional networks. Using a U-Net-based framework, we show that multifractal recalibration yields substantial gains over a baseline equipped with other channel-attention mechanisms that also use higher-order statistics. Given the proven ability of multifractal analysis to capture pathological regularities, we validate our approach on three public medical-imaging datasets: ISIC18 (dermoscopy), Kvasir-SEG (endoscopy), and BUSI (ultrasound). Our empirical analysis also provides insights into the behavior of these attention layers. We find that excitation responses do not become increasingly specialized with encoder depth in U-Net architectures due to skip connections, and that their effectiveness may relate to global statistics of instance variability.
comment : 30 pages, 9 figures, journal paper
☆ RobustSurg: Tackling domain generalisation for out-of-distribution surgical scene segmentation
Mansoor Ali, Maksim Richards, Gilberto Ochoa-Ruiz, Sharib Ali
While recent advances in deep learning for surgical scene segmentation have demonstrated promising results on single-centre and single-imaging modality data, these methods usually do not generalise to unseen distribution (i.e., from other centres) and unseen modalities. Current literature for tackling generalisation on out-of-distribution data and domain gaps due to modality changes has been widely researched but mostly for natural scene data. However, these methods cannot be directly applied to the surgical scenes due to limited visual cues and often extremely diverse scenarios compared to the natural scene data. Inspired by these works in natural scenes to push generalisability on OOD data, we hypothesise that exploiting the style and content information in the surgical scenes could minimise the appearances, making it less variable to sudden changes such as blood or imaging artefacts. This can be achieved by performing instance normalisation and feature covariance mapping techniques for robust and generalisable feature representations. Further, to eliminate the risk of removing salient feature representation associated with the objects of interest, we introduce a restitution module within the feature learning ResNet backbone that can enable the retention of useful task-relevant features. To tackle the lack of multiclass and multicentre data for surgical scene segmentation, we also provide a newly curated dataset that can be vital for addressing generalisability in this domain. Our proposed RobustSurg obtained nearly 23% improvement on the baseline DeepLabv3+ and from 10-32% improvement on the SOTA in terms of mean IoU score on an unseen centre HeiCholSeg dataset when trained on CholecSeg8K. Similarly, RobustSurg also obtained nearly 22% improvement over the baseline and nearly 11% improvement on a recent SOTA method for the target set of the EndoUDA polyp dataset.
comment : Submitted to Medical Image Analysis
☆ SplatSuRe: Selective Super-Resolution for Multi-view Consistent 3D Gaussian Splatting
Pranav Asthana, Alex Hanson, Allen Tu, Tom Goldstein, Matthias Zwicker, Amitabh Varshney
3D Gaussian Splatting (3DGS) enables high-quality novel view synthesis, motivating interest in generating higher-resolution renders than those available during training. A natural strategy is to apply super-resolution (SR) to low-resolution (LR) input views, but independently enhancing each image introduces multi-view inconsistencies, leading to blurry renders. Prior methods attempt to mitigate these inconsistencies through learned neural components, temporally consistent video priors, or joint optimization on LR and SR views, but all uniformly apply SR across every image. In contrast, our key insight is that close-up LR views may contain high-frequency information for regions also captured in more distant views, and that we can use the camera pose relative to scene geometry to inform where to add SR content. Building from this insight, we propose SplatSuRe, a method that selectively applies SR content only in undersampled regions lacking high-frequency supervision, yielding sharper and more consistent results. Across Tanks & Temples, Deep Blending and Mip-NeRF 360, our approach surpasses baselines in both fidelity and perceptual quality. Notably, our gains are most significant in localized foreground regions where higher detail is desired.
comment : Project Page: https://splatsure.github.io/
☆ Mapping of Lesion Images to Somatic Mutations
Rahul Mehta
Medical imaging is a critical initial tool used by clinicians to determine a patient's cancer diagnosis, allowing for faster intervention and more reliable patient prognosis. At subsequent stages of patient diagnosis, genetic information is extracted to help select specific patient treatment options. As the efficacy of cancer treatment often relies on early diagnosis and treatment, we build a deep latent variable model to determine patients' somatic mutation profiles based on their corresponding medical images. We first introduce a point cloud representation of lesions images to allow for invariance to the imaging modality. We then propose, LLOST, a model with dual variational autoencoders coupled together by a separate shared latent space that unifies features from the lesion point clouds and counts of distinct somatic mutations. Therefore our model consists of three latent space, each of which is learned with a conditional normalizing flow prior to account for the diverse distributions of each domain. We conduct qualitative and quantitative experiments on de-identified medical images from The Cancer Imaging Archive and the corresponding somatic mutations from the Pan Cancer dataset of The Cancer Genomic Archive. We show the model's predictive performance on the counts of specific mutations as well as it's ability to accurately predict the occurrence of mutations. In particular, shared patterns between the imaging and somatic mutation domain that reflect cancer type. We conclude with a remark on how to improve the model and possible future avenues of research to include other genetic domains.
comment : https://dl.acm.org/doi/abs/10.1145/3340531.3414074#sec-terms
☆ FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judge s NeurIPS 2025
Kevin David Hayes, Micah Goldblum, Vikash Sehwag, Gowthami Somepalli, Ashwinee Panda, Tom Goldstein
Text-to-image (T2I) models are capable of generating visually impressive images, yet they often fail to accurately capture specific attributes in user prompts, such as the correct number of objects with the specified colors. The diversity of such errors underscores the need for a hierarchical evaluation framework that can compare prompt adherence abilities of different image generation models. Simultaneously, benchmarks of vision language models (VLMs) have not kept pace with the complexity of scenes that VLMs are used to annotate. In this work, we propose a structured methodology for jointly evaluating T2I models and VLMs by testing whether VLMs can identify 27 specific failure modes in the images generated by T2I models conditioned on challenging prompts. Our second contribution is a dataset of prompts and images generated by 5 T2I models (Flux, SD3-Medium, SD3-Large, SD3.5-Medium, SD3.5-Large) and the corresponding annotations from VLMs (Molmo, InternVL3, Pixtral) annotated by an LLM (Llama3) to test whether VLMs correctly identify the failure mode in a generated image. By analyzing failure modes on a curated set of prompts, we reveal systematic errors in attribute fidelity and object representation. Our findings suggest that current metrics are insufficient to capture these nuanced errors, highlighting the importance of targeted benchmarks for advancing generative model reliability and interpretability.
comment : Accepted to NeurIPS 2025 Datasets and Benchmarks Track
☆ Context -Enriched Contrastive Loss: Enhancing Presentation of Inherent Sample Connections in Contrastive Learning Framework
Haojin Deng, Yimin Yang
Contrastive learning has gained popularity and pushes state-of-the-art performance across numerous large-scale benchmarks. In contrastive learning, the contrastive loss function plays a pivotal role in discerning similarities between samples through techniques such as rotation or cropping. However, this learning mechanism can also introduce information distortion from the augmented samples. This is because the trained model may develop a significant overreliance on information from samples with identical labels, while concurrently neglecting positive pairs that originate from the same initial image, especially in expansive datasets. This paper proposes a context-enriched contrastive loss function that concurrently improves learning effectiveness and addresses the information distortion by encompassing two convergence targets. The first component, which is notably sensitive to label contrast, differentiates between features of identical and distinct classes which boosts the contrastive training efficiency. Meanwhile, the second component draws closer the augmented samples from the same source image and distances all other samples. We evaluate the proposed approach on image classification tasks, which are among the most widely accepted 8 recognition large-scale benchmark datasets: CIFAR10, CIFAR100, Caltech-101, Caltech-256, ImageNet, BiasedMNIST, UTKFace, and CelebA datasets. The experimental results demonstrate that the proposed method achieves improvements over 16 state-of-the-art contrastive learning methods in terms of both generalization performance and learning convergence speed. Interestingly, our technique stands out in addressing systematic distortion tasks. It demonstrates a 22.9% improvement compared to original contrastive loss functions in the downstream BiasedMNIST dataset, highlighting its promise for more efficient and equitable downstream training.
comment : 13 pages, 7 figures. Published in IEEE Transactions on Multimedia. Code available at: https://github.com/hdeng26/Contex
☆ CoatFusion: Controllable Material Coating in Images
Sagie Levy, Elad Aharoni, Matan Levy, Ariel Shamir, Dani Lischinski
We introduce Material Coating, a novel image editing task that simulates applying a thin material layer onto an object while preserving its underlying coarse and fine geometry. Material coating is fundamentally different from existing "material transfer" methods, which are designed to replace an object's intrinsic material, often overwriting fine details. To address this new task, we construct a large-scale synthetic dataset (110K images) of 3D objects with varied, physically-based coatings, named DataCoat110K. We then propose CoatFusion, a novel architecture that enables this task by conditioning a diffusion model on both a 2D albedo texture and granular, PBR-style parametric controls, including roughness, metalness, transmission, and a key thickness parameter. Experiments and user studies show CoatFusion produces realistic, controllable coatings and significantly outperforms existing material editing and transfer methods on this new task.
☆ UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning -Enriched Edits
Keming Ye, Zhipeng Huang, Canmiao Fu, Qingyang Liu, Jiani Cai, Zheqi Lv, Chen Li, Jing Lyu, Zhou Zhao, Shengyu Zhang
With the rapid advances of powerful multimodal models such as GPT-4o, Nano Banana, and Seedream 4.0 in Image Editing, the performance gap between closed-source and open-source models is widening, primarily due to the scarcity of large-scale, high-quality training data and comprehensive benchmarks capable of diagnosing model weaknesses across diverse editing behaviors. Existing data construction methods face a scale-quality trade-off: human annotations are high-quality but not scalable, while automated pipelines suffer from error propagation and noise. To address this, we introduce a lightweight data pipeline that replaces multi-toolchains with an end-to-end model and a unified post-verification stage. For scalable quality control, we train a 7B dual-task expert model, \textbf{Qwen-Verify}, for efficient failure detection and instruction recaptioning. This pipeline yields \textbf{UnicEdit-10M}, a 10M-scale dataset spanning diverse basic and complex editing tasks. We also propose \textbf{UnicBench}, a general benchmark that extends beyond basic edits to explicitly assess spatial and knowledge-driven reasoning. To enable fine-grained diagnosis, we introduce novel metrics, including \textit{Non-edit Consistency} and \textit{Reasoning Accuracy}. Our analysis of mainstream models on UnicBench reveals their limitations and provides clear directions for future research.
comment : 31 pages, 15 figures, 12 tables
☆ PhyCustom: Towards Realistic Physical Customization in Text-to-Image Generation
Fan Wu, Cheng Chen, Zhoujie Fu, Jiacheng Wei, Yi Xu, Deheng Ye, Guosheng Lin
Recent diffusion-based text-to-image customization methods have achieved significant success in understanding concrete concepts to control generation processes, such as styles and shapes. However, few efforts dive into the realistic yet challenging customization of physical concepts. The core limitation of current methods arises from the absence of explicitly introducing physical knowledge during training. Even when physics-related words appear in the input text prompts, our experiments consistently demonstrate that these methods fail to accurately reflect the corresponding physical properties in the generated results. In this paper, we propose PhyCustom, a fine-tuning framework comprising two novel regularization losses to activate diffusion model to perform physical customization. Specifically, the proposed isometric loss aims at activating diffusion models to learn physical concepts while decouple loss helps to eliminate the mixture learning of independent concepts. Experiments are conducted on a diverse dataset and our benchmark results demonstrate that PhyCustom outperforms previous state-of-the-art and popular methods in terms of physical customization quantitatively and qualitatively.
comment : codes:https://github.com/wufan-cse/PhyCustom
☆ IC-Wo rl d: In- Context Generation for Shared Wo rl d Modeling
Fan Wu, Jiacheng Wei, Ruibo Li, Yi Xu, Junyou Li, Deheng Ye, Guosheng Lin
Video-based world models have recently garnered increasing attention for their ability to synthesize diverse and dynamic visual environments. In this paper, we focus on shared world modeling, where a model generates multiple videos from a set of input images, each representing the same underlying world in different camera poses. We propose IC-World, a novel generation framework, enabling parallel generation for all input images via activating the inherent in-context generation capability of large video models. We further finetune IC-World via reinforcement learning, Group Relative Policy Optimization, together with two proposed novel reward models to enforce scene-level geometry consistency and object-level motion consistency among the set of generated videos. Extensive experiments demonstrate that IC-World substantially outperforms state-of-the-art methods in both geometry and motion consistency. To the best of our knowledge, this is the first work to systematically explore the shared world modeling problem with video-based world models.
comment : codes:https://github.com/wufan-cse/IC-World
☆ Comparing Baseline and Day-1 Diffusion MRI Using Multimodal Deep Embeddings for Stroke Outcome Prediction
Sina Raeisadigh, Myles Joshua Toledo Tan, Henning Müller, Abderrahmane Hedjoudje
This study compares baseline (J0) and 24-hour (J1) diffusion magnetic resonance imaging (MRI) for predicting three-month functional outcomes after acute ischemic stroke (AIS). Seventy-four AIS patients with paired apparent diffusion coefficient (ADC) scans and clinical data were analyzed. Three-dimensional ResNet-50 embeddings were fused with structured clinical variables, reduced via principal component analysis (<=12 components), and classified using linear support vector machines with eight-fold stratified group cross-validation. J1 multimodal models achieved the highest predictive performance (AUC = 0.923 +/- 0.085), outperforming J0-based configurations (AUC <= 0.86). Incorporating lesion-volume features further improved model stability and interpretability. These findings demonstrate that early post-treatment diffusion MRI provides superior prognostic value to pre-treatment imaging and that combining MRI, clinical, and lesion-volume features produces a robust and interpretable framework for predicting three-month functional outcomes in AIS patients.
comment : 5 pages, 5 figures, 2 tables
♻ ☆ VIVAT: Virtuous Improving VAE Training through Artifact Mitigation
Lev Novitskiy, Viacheslav Vasilev, Maria Kovaleva, Vladimir Arkhipkin, Denis Dimitrov
Variational Autoencoders (VAEs) remain a cornerstone of generative computer vision, yet their training is often plagued by artifacts that degrade reconstruction and generation quality. This paper introduces VIVAT, a systematic approach to mitigating common artifacts in KL-VAE training without requiring radical architectural changes. We present a detailed taxonomy of five prevalent artifacts - color shift, grid patterns, blur, corner and droplet artifacts - and analyze their root causes. Through straightforward modifications, including adjustments to loss weights, padding strategies, and the integration of Spatially Conditional Normalization, we demonstrate significant improvements in VAE performance. Our method achieves state-of-the-art results in image reconstruction metrics (PSNR and SSIM) across multiple benchmarks and enhances text-to-image generation quality, as evidenced by superior CLIP scores. By preserving the simplicity of the KL-VAE framework while addressing its practical challenges, VIVAT offers actionable insights for researchers and practitioners aiming to optimize VAE training.
♻ ☆ STORM: Segment, Track, and Object Re-Localization from a Single Image
Yu Deng, Teng Cao, Hikaru Shindo, Jiahong Xue, Quentin Delfosse, Kristian Kersting
Accurate 6D pose estimation and tracking are fundamental capabilities for physical AI systems such as robots. However, existing approaches typically require a pre-defined 3D model of the target and rely on a manually annotated segmentation mask in the first frame, which is labor-intensive and leads to reduced performance when faced with occlusions or rapid movement. To address these limitations, we propose STORM (Segment, Track, and Object Re-localization from a single iMage), an open-source robust real-time 6D pose estimation system that requires no manual annotation. STORM employs a novel three-stage pipeline combining vision-language understanding with feature matching: contextual object descriptions guide localization, self-cross-attention mechanisms identify candidate regions, and produce precise masks and 3D models for accurate pose estimation. Another key innovation is our automatic re-registration mechanism that detects tracking failures through feature similarity monitoring and recovers from severe occlusions or rapid motion. STORM achieves state-of-the-art accuracy on challenging industrial datasets featuring multi-object occlusions, high-speed motion, and varying illumination, while operating at real-time speeds without additional training. This annotation-free approach significantly reduces deployment overhead, providing a practical solution for modern applications, such as flexible manufacturing and intelligent quality control.
♻ ☆ Continuous Perception Matters: Diagnosing Temporal Integration Failures in Multimodal Models
Zeyu Wang, Zhenzhen Weng, Serena Yeung-Levy
Continuous perception, the ability to integrate visual observations over time in a continuous stream fashion, is essential for robust real-world understanding, yet remains largely untested in current multimodal models. We introduce CP-Bench, a minimal and fully controlled benchmark designed to isolate this capability using an extremely simple task: counting identical cubes in a synthetic scene while the camera moves and only reveals subsets of objects at any moment. Despite the simplicity of the setting, we find that state-of-the-art open-source and commercial models, including Qwen-3-VL, InternVL3, GPT-5, and Gemini-3-Pro, fail dramatically. A static-camera control variant confirms that the failure arises not from object recognition but from an inability to accumulate evidence across time. Further experiments show that neither higher sampling FPS, perception- or spatial-enhanced models, nor finetuning with additional videos leads to meaningful cross-temporal generalization. Our results reveal a fundamental limitation in modern multimodal architectures and training paradigms. CP-Bench provides a simple yet powerful diagnostic tool and establishes a clean testbed for developing models capable of genuine time-consistent visual reasoning.
♻ ☆ GBT-SAM: A Parameter-Efficient Depth-Aware Model for Generalizable Brain tumour Segmentation on mp-MRI
Cecilia Diana-Albelda, Roberto Alcover-Couso, Álvaro García-Martín, Jesus Bescos, Marcos Escudero-Viñolo
Gliomas are aggressive brain tumors that require accurate imaging-based diagnosis, with segmentation playing a critical role in evaluating morphology and treatment decisions. Manual delineation of gliomas is time-consuming and prone to variability, motivating the use of deep learning to improve consistency and alleviate clinical workload. However, existing methods often fail to fully exploit the information available in multi-parametric MRI (mp-MRI), particularly inter-slice contextual features, and typically require considerable computational resources while lacking robustness across tumor type variations. We present GBT-SAM, a parameter-efficient deep learning framework that adapts the Segment Anything Model (SAM), a large-scale vision model, to volumetric mp-MRI data. GBT-SAM reduces input complexity by selecting fewer than 2.6\% of slices per scan while incorporating all four MRI modalities, preserving essential tumor-related information with minimal cost. Furthermore, our model is trained by a two-step fine-tuning strategy that incorporates a depth-aware module to capture inter-slice correlations and lightweight adaptation layers, resulting in just 6.5M trainable parameters, which is the lowest among SAM-based approaches. GBT-SAM achieves a Dice Score of 93.54 on the BraTS Adult Glioma dataset and demonstrates robust performance on Meningioma, Pediatric Glioma, and Sub-Saharan Glioma datasets. These results highlight GBT-SAM's potential as a computationally efficient and domain-robust framework for brain tumor segmentation using mp-MRI. Our code and models are available at https://github.com/vpulab/med-sam-brain .
♻ ☆ DualCamCt rl : Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generation
Hongfei Zhang, Kanghao Chen, Zixin Zhang, Harold Haodong Chen, Yuanhuiyi Lyu, Yuqi Zhang, Shuai Yang, Kun Zhou, Yingcong Chen
This paper presents DualCamCtrl, a novel end-to-end diffusion model for camera-controlled video generation. Recent works have advanced this field by representing camera poses as ray-based conditions, yet they often lack sufficient scene understanding and geometric awareness. DualCamCtrl specifically targets this limitation by introducing a dual-branch framework that mutually generates camera-consistent RGB and depth sequences. To harmonize these two modalities, we further propose the Semantic Guided Mutual Alignment (SIGMA) mechanism, which performs RGB-depth fusion in a semantics-guided and mutually reinforced manner. These designs collectively enable DualCamCtrl to better disentangle appearance and geometry modeling, generating videos that more faithfully adhere to the specified camera trajectories. Additionally, we analyze and reveal the distinct influence of depth and camera poses across denoising stages and further demonstrate that early and late stages play complementary roles in forming global structure and refining local details. Extensive experiments demonstrate that DualCamCtrl achieves more consistent camera-controlled video generation, with over 40\% reduction in camera motion errors compared with prior methods. Our project page: https://soyouthinkyoucantell.github.io/dualcamctrl-page/
♻ ☆ SCOPE-MRI: Bankart Lesion Detection as a Case Study in Data Curation and Deep Learning for Challenging Diagnoses
Sahil Sethi, Sai Reddy, Mansi Sakarvadia, Jordan Serotte, Darlington Nwaudo, Nicholas Maassen, Lewis Shi
Deep learning has shown strong performance in musculoskeletal imaging, but prior work has largely targeted conditions where diagnosis is relatively straightforward. More challenging problems remain underexplored, such as detecting Bankart lesions (anterior-inferior glenoid labral tears) on standard MRIs. These lesions are difficult to diagnose due to subtle imaging features, often necessitating invasive MRI arthrograms (MRAs). We introduce ScopeMRI, the first publicly available, expert-annotated dataset for shoulder pathologies, and present a deep learning framework for Bankart lesion detection on both standard MRIs and MRAs. ScopeMRI contains shoulder MRIs from patients who underwent arthroscopy, providing ground-truth labels from intraoperative findings, the diagnostic gold standard. Separate models were trained for MRIs and MRAs using CNN- and transformer-based architectures, with predictions ensembled across multiple imaging planes. Our models achieved radiologist-level performance, with accuracy on standard MRIs surpassing radiologists interpreting MRAs. External validation on independent hospital data demonstrated initial generalizability across imaging protocols. By releasing ScopeMRI and a modular codebase for training and evaluation, we aim to accelerate research in musculoskeletal imaging and foster development of datasets and models that address clinically challenging diagnostic tasks.
comment : This version of the article has been accepted for publication at Nature Partner Journal (NPJ) Artificial Intelligence, but is not the Version of Record and does not reflect post-acceptance improvements or any corrections. The Version of Record is available online at: http://dx.doi.org/10.1038/s44387-025-00043-5
♻ ☆ Structure is Super vision : Multiview Masked Autoencoders for Radiology
Sonia Laguna, Andrea Agostini, Alain Ryser, Samuel Ruiperez-Campillo, Irene Cannistraci, Moritz Vandenhirtz, Stephan Mandt, Nicolas Deperrois, Farhad Nooralahzadeh, Michael Krauthammer, Thomas M. Sutter, Julia E. Vogt
Building robust medical machine learning systems requires pretraining strategies that exploit the intrinsic structure present in clinical data. We introduce Multiview Masked Autoencoder (MVMAE), a self-supervised framework that leverages the natural multi-view organization of radiology studies to learn view-invariant and disease-relevant representations. MVMAE combines masked image reconstruction with cross-view alignment, transforming clinical redundancy across projections into a powerful self-supervisory signal. We further extend this approach with MVMAE-V2T, which incorporates radiology reports as an auxiliary text-based learning signal to enhance semantic grounding while preserving fully vision-based inference. Evaluated on a downstream disease classification task on three large-scale public datasets, MIMIC-CXR, CheXpert, and PadChest, MVMAE consistently outperforms supervised and vision-language baselines. Furthermore, MVMAE-V2T provides additional gains, particularly in low-label regimes where structured textual supervision is most beneficial. Together, these results establish the importance of structural and textual supervision as complementary paths toward scalable, clinically grounded medical foundation models.
♻ ☆ Securing the Skies: A Comprehensive Survey on Anti-UAV Methods, Benchmark ing, and Future Directions CVPR
Yifei Dong, Fengyi Wu, Sanjian Zhang, Guangyu Chen, Yuzhi Hu, Masumi Yano, Jingdong Sun, Siyu Huang, Feng Liu, Qi Dai, Zhi-Qi Cheng
Unmanned Aerial Vehicles (UAVs) are indispensable for infrastructure inspection, surveillance, and related tasks, yet they also introduce critical security challenges. This survey provides a wide-ranging examination of the anti-UAV domain, centering on three core objectives-classification, detection, and tracking-while detailing emerging methodologies such as diffusion-based data synthesis, multi-modal fusion, vision-language modeling, self-supervised learning, and reinforcement learning. We systematically evaluate state-of-the-art solutions across both single-modality and multi-sensor pipelines (spanning RGB, infrared, audio, radar, and RF) and discuss large-scale as well as adversarially oriented benchmarks. Our analysis reveals persistent gaps in real-time performance, stealth detection, and swarm-based scenarios, underscoring pressing needs for robust, adaptive anti-UAV systems. By highlighting open research directions, we aim to foster innovation and guide the development of next-generation defense strategies in an era marked by the extensive use of UAVs.
comment : Best Paper, Accepted at CVPR Workshop Anti-UAV 2025. 16 pages
♻ ☆ Adaptive Plane Reformatting for 4D Flow MRI using Deep Reinforcement Learning
Javier Bisbal, Julio Sotelo, Maria I Valdés, Pablo Irarrazaval, Marcelo E Andia, Julio García, José Rodriguez-Palomarez, Francesca Raimondi, Cristián Tejos, Sergio Uribe
Background and Objective: Plane reformatting for four-dimensional phase contrast MRI (4D flow MRI) is time-consuming and prone to inter-observer variability, which limits fast cardiovascular flow assessment. Deep reinforcement learning (DRL) trains agents to iteratively adjust plane position and orientation, enabling accurate plane reformatting without the need for detailed landmarks, making it suitable for images with limited contrast and resolution such as 4D flow MRI. However, current DRL methods assume that test volumes share the same spatial alignment as the training data, limiting generalization across scanners and institutions. To address this limitation, we introduce AdaPR (Adaptive Plane Reformatting), a DRL framework that uses a local coordinate system to navigate volumes with arbitrary positions and orientations. Methods: We implemented AdaPR using the Asynchronous Advantage Actor-Critic (A3C) algorithm and validated it on 88 4D flow MRI datasets acquired from multiple vendors, including patients with congenital heart disease. Results: AdaPR achieved a mean angular error of 6.32 +/- 4.15 degrees and a distance error of 3.40 +/- 2.75 mm, outperforming global-coordinate DRL methods and alternative non-DRL methods. AdaPR maintained consistent accuracy under different volume orientations and positions. Flow measurements from AdaPR planes showed no significant differences compared to two manual observers, with excellent correlation (R^2 = 0.972 and R^2 = 0.968), comparable to inter-observer agreement (R^2 = 0.969). Conclusion: AdaPR provides robust, orientation-independent plane reformatting for 4D flow MRI, achieving flow quantification comparable to expert observers. Its adaptability across datasets and scanners makes it a promising candidate for medical imaging applications beyond 4D flow MRI.
♻ ☆ MAMMA: Marke rl ess & Automatic Multi-Person Motion Action Capture
Hanz Cuevas-Velasquez, Anastasios Yiannakidis, Soyong Shin, Giorgio Becherini, Markus Höschle, Joachim Tesch, Taylor Obersat, Tsvetelina Alexiadis, Eni Halilaj, Michael J. Black
We present MAMMA, a markerless motion-capture pipeline that accurately recovers SMPL-X parameters from multi-view video of two-person interaction sequences. Traditional motion-capture systems rely on physical markers. Although they offer high accuracy, their requirements of specialized hardware, manual marker placement, and extensive post-processing make them costly and time-consuming. Recent learning-based methods attempt to overcome these limitations, but most are designed for single-person capture, rely on sparse keypoints, or struggle with occlusions and physical interactions. In this work, we introduce a method that predicts dense 2D contact-aware surface landmarks conditioned on segmentation masks, enabling person-specific correspondence estimation even under heavy occlusion. We employ a novel architecture that exploits learnable queries for each landmark. We demonstrate that our approach can handle complex person--person interaction and offers greater accuracy than existing methods. To train our network, we construct a large, synthetic multi-view dataset combining human motions from diverse sources, including extreme poses, hand motions, and close interactions. Our dataset yields high-variability synthetic sequences with rich body contact and occlusion, and includes SMPL-X ground-truth annotations with dense 2D landmarks. The result is a system capable of capturing human motion without the need for markers. Our approach offers competitive reconstruction quality compared to commercial marker-based motion-capture solutions, without the extensive manual cleanup. Finally, we address the absence of common benchmarks for dense-landmark prediction and markerless motion capture by introducing two evaluation settings built from real multi-view sequences. We will release our dataset, benchmark, method, training code, and pre-trained model weights for research purposes.
♻ ☆ Benchmark ing machine learning models for multi-class state recognition in double quantum dot data
Valeria Díaz Moreno, Ryan P Khalili, Daniel Schug, Patrick J. Walsh, Justyna P. Zwolak
Semiconductor quantum dots (QDs) are a leading platform for scalable quantum processors. However, scaling to large arrays requires reliable, automated tuning strategies for devices' bootstrapping, calibration, and operation, with many tuning aspects depending on accurately identifying QD device states from charge-stability diagrams (CSDs). In this work, we present a comprehensive benchmarking study of four modern machine learning (ML) architectures for multi-class state recognition in double-QD CSDs. We evaluate their performance across different data budgets and normalization schemes using both synthetic and experimental data. We find that the more resource-intensive models -- U-Nets and visual transformers (ViTs) -- achieve the highest MSE score (defined as $1-\mathrm{MSE}$) on synthetic data (over $0.98$) but fail to generalize to experimental data. MDNs are the most computationally efficient and exhibit highly stable training, but with substantially lower peak performance. CNNs offer the most favorable trade-off on experimental CSDs, achieving strong accuracy with two orders of magnitude fewer parameters than the U-Nets and ViTs. Normalization plays a nontrivial role: min-max scaling generally yields higher MSE scores but less stable convergence, whereas z-score normalization produces more predictable training dynamics but at reduced accuracy for most models. Overall, our study shows that CNNs with min-max normalization are a practical approach for QD CSDs.
comment : 12 pages, 4 figures, 2 tables
♻ ☆ Global-to-local image quality assessment in optical microscopy via fast and robust deep learning predictions
Elena Corbetta, Thomas Bocklitz
Optical microscopy is one of the most widely used techniques in research studies for life sciences and biomedicine. These applications require reliable experimental pipelines to extract valuable knowledge from the measured samples and must be supported by image quality assessment (IQA) to ensure correct processing and analysis of the image data. IQA methods are implemented with variable complexity. However, while most quality metrics have a straightforward implementation, they might be time consuming and computationally expensive when evaluating a large dataset. In addition, quality metrics are often designed for well-defined image features and may be unstable for images out of the ideal domain. To overcome these limitations, recent works have proposed deep learning-based IQA methods, which can provide superior performance, increased generalizability and fast prediction. Our method, named $\mathrmμ$DeepIQA, is inspired by previous studies and applies a deep convolutional neural network designed for IQA on natural images to optical microscopy measurements. We retrained the same architecture to predict individual quality metrics and global quality scores for optical microscopy data. The resulting models provide fast and stable predictions of image quality by generalizing quality estimation even outside the ideal range of standard methods. In addition, $\mathrmμ$DeepIQA provides patch-wise prediction of image quality and can be used to visualize spatially varying quality in a single image. Our study demonstrates that optical microscopy-based studies can benefit from the generalizability of deep learning models due to their stable performance in the presence of outliers, the ability to assess small image patches, and rapid predictions.
comment : 16 pages, 6 figures. μDeepIQA is publicly available at https://git.photonicdata.science/elena.corbetta/udeepiqa
♻ ☆ Manual-PA: Learning 3D Part Assembly from Instruction Diagrams ICCV'25
Jiahao Zhang, Anoop Cherian, Cristian Rodriguez, Weijian Deng, Stephen Gould
Assembling furniture amounts to solving the discrete-continuous optimization task of selecting the furniture parts to assemble and estimating their connecting poses in a physically realistic manner. The problem is hampered by its combinatorially large yet sparse solution space thus making learning to assemble a challenging task for current machine learning models. In this paper, we attempt to solve this task by leveraging the assembly instructions provided in diagrammatic manuals that typically accompany the furniture parts. Our key insight is to use the cues in these diagrams to split the problem into discrete and continuous phases. Specifically, we present Manual-PA, a transformer-based instruction Manual-guided 3D Part Assembly framework that learns to semantically align 3D parts with their illustrations in the manuals using a contrastive learning backbone towards predicting the assembly order and infers the 6D pose of each part via relating it to the final furniture depicted in the manual. To validate the efficacy of our method, we conduct experiments on the benchmark PartNet dataset. Our results show that using the diagrams and the order of the parts lead to significant improvements in assembly performance against the state of the art. Further, Manual-PA demonstrates strong generalization to real-world IKEA furniture assembly on the IKEA-Manual dataset.
comment : Accepted to ICCV'25
♻ ☆ MRI Super-Resolution with Deep Learning: A Comprehensive Survey
Mohammad Khateri, Serge Vasylechko, Morteza Ghahremani, Liam Timms, Deniz Kocanaogullari, Simon K. Warfield, Camilo Jaimes, Davood Karimi, Alejandra Sierra, Jussi Tohka, Sila Kurugol, Onur Afacan
High-resolution (HR) magnetic resonance imaging (MRI) is crucial for many clinical and research applications. However, achieving it remains costly and constrained by technical trade-offs and experimental limitations. Super-resolution (SR) presents a promising computational approach to overcome these challenges by generating HR images from more affordable low-resolution (LR) scans, potentially improving diagnostic accuracy and efficiency without requiring additional hardware. This survey reviews recent advances in MRI SR techniques, with a focus on deep learning (DL) approaches. It examines DL-based MRI SR methods from the perspectives of computer vision, computational imaging, inverse problems, and MR physics, covering theoretical foundations, architectural designs, learning strategies, benchmark datasets, and performance metrics. We propose a systematic taxonomy to categorize these methods and present an in-depth study of both established and emerging SR techniques applicable to MRI, considering unique challenges in clinical and research contexts. We also highlight open challenges and directions that the community needs to address. Additionally, we provide a collection of essential open-access resources, tools, and tutorials, available on our GitHub: https://github.com/mkhateri/Awesome-MRI-Super-Resolution. IEEE keywords: MRI, Super-Resolution, Deep Learning, Computational Imaging, Inverse Problem, Survey.
comment : 41 pages
♻ ☆ Multivariate Variational Autoencoder
Mehmet Can Yavuz
Learning latent representations that are simultaneously expressive, geometrically well-structured, and reliably calibrated remains a central challenge for Variational Autoencoders (VAEs). Standard VAEs typically assume a diagonal Gaussian posterior, which simplifies optimization but rules out correlated uncertainty and often yields entangled or redundant latent dimensions. We introduce the Multivariate Variational Autoencoder (MVAE), a tractable full-covariance extension of the VAE that augments the encoder with sample-specific diagonal scales and a global coupling matrix. This induces a multivariate Gaussian posterior of the form $N(μ_φ(x), C \operatorname{diag}(σ_φ^2(x)) C^\top)$, enabling correlated latent factors while preserving a closed-form KL divergence and a simple reparameterization path. Beyond likelihood, we propose a multi-criterion evaluation protocol that jointly assesses reconstruction quality (MSE, ELBO), downstream discrimination (linear probes), probabilistic calibration (NLL, Brier, ECE), and unsupervised structure (NMI, ARI). Across Larochelle-style MNIST variants, Fashion-MNIST, and CIFAR-10/100, MVAE consistently matches or outperforms diagonal-covariance VAEs of comparable capacity, with particularly notable gains in calibration and clustering metrics at both low and high latent dimensions. Qualitative analyses further show smoother, more semantically coherent latent traversals and sharper reconstructions. All code, dataset splits, and evaluation utilities are released to facilitate reproducible comparison and future extensions of multivariate posterior models.
♻ ☆ Hierarchical Semi-Supervised Active Learning for Remote Sensing
Wei Huang, Zhitong Xiong, Chenying Liu, Xiao Xiang Zhu
The performance of deep learning models in remote sensing (RS) strongly depends on the availability of high-quality labeled data. However, collecting large-scale annotations is costly and time-consuming, while vast amounts of unlabeled imagery remain underutilized. To address this challenge, we propose a Hierarchical Semi-Supervised Active Learning (HSSAL) framework that integrates semi-supervised learning (SSL) and a novel hierarchical active learning (HAL) in a closed iterative loop. In each iteration, SSL refines the model using both labeled data through supervised learning and unlabeled data via weak-to-strong self-training, improving feature representation and uncertainty estimation. Guided by the refined representations and uncertainty cues of unlabeled samples, HAL then conducts sample querying through a progressive clustering strategy, selecting the most informative instances that jointly satisfy the criteria of scalability, diversity, and uncertainty. This hierarchical process ensures both efficiency and representativeness in sample selection. Extensive experiments on three benchmark RS scene classification datasets, including UCM, AID, and NWPU-RESISC45, demonstrate that HSSAL consistently outperforms SSL- or AL-only baselines. Remarkably, with only 8%, 4%, and 2% labeled training data on UCM, AID, and NWPU-RESISC45, respectively, HSSAL achieves over 95% of fully-supervised accuracy, highlighting its superior label efficiency through informativeness exploitation of unlabeled data. Our code will be publicly available.
comment : Under review
♻ ☆ Harnessing Diffusion -Generated Synthetic Images for Fair Image Classification AAAI
Abhipsa Basu, Aviral Gupta, Abhijnya Bhat, R. Venkatesh Babu
Image classification systems often inherit biases from uneven group representation in training data. For example, in face datasets for hair color classification, blond hair may be disproportionately associated with females, reinforcing stereotypes. A recent approach leverages the Stable Diffusion model to generate balanced training data, but these models often struggle to preserve the original data distribution. In this work, we explore multiple diffusion-finetuning techniques, e.g., LoRA and DreamBooth, to generate images that more accurately represent each training group by learning directly from their samples. Additionally, in order to prevent a single DreamBooth model from being overwhelmed by excessive intra-group variations, we explore a technique of clustering images within each group and train a DreamBooth model per cluster. These models are then used to generate group-balanced data for pretraining, followed by fine-tuning on real data. Experiments on multiple benchmarks demonstrate that the studied finetuning approaches outperform vanilla Stable Diffusion on average and achieve results comparable to SOTA debiasing techniques like Group-DRO, while surpassing them as the dataset bias severity increases.
comment : Accepted to AAAI AISI Track, 2026
♻ ☆ Sketch-guided Cage-based 3D Gaussian Splatting Deformation WACV 26
Tianhao Xie, Noam Aigerman, Eugene Belilovsky, Tiberiu Popa
3D Gaussian Splatting (GS) is one of the most promising novel 3D representations that has received great interest in computer graphics and computer vision. While various systems have introduced editing capabilities for 3D GS, such as those guided by text prompts, fine-grained control over deformation remains an open challenge. In this work, we present a novel sketch-guided 3D GS deformation system that allows users to intuitively modify the geometry of a 3D GS model by drawing a silhouette sketch from a single viewpoint. Our approach introduces a new deformation method that combines cage-based deformations with a variant of Neural Jacobian Fields, enabling precise, fine-grained control. Additionally, it leverages large-scale 2D diffusion priors and ControlNet to ensure the generated deformations are semantically plausible. Through a series of experiments, we demonstrate the effectiveness of our method and showcase its ability to animate static 3D GS models as one of its key applications.
comment : 10 pages, 9 figures, accepted at WACV 26, project page: https://tianhaoxie.github.io/project/gs_deform/
♻ ☆ Benchmark ing pig detection and tracking under diverse and challenging conditions
Jonathan Henrich, Christian Post, Maximilian Zilke, Parth Shiroya, Emma Chanut, Amir Mollazadeh Yamchi, Ramin Yahyapour, Thomas Kneib, Imke Traulsen
To ensure animal welfare and effective management in pig farming, monitoring individual behavior is a crucial prerequisite. While monitoring tasks have traditionally been carried out manually, advances in machine learning have made it possible to collect individualized information in an increasingly automated way. Central to these methods is the localization of animals across space (object detection) and time (multi-object tracking). Despite extensive research of these two tasks in pig farming, a systematic benchmarking study has not yet been conducted. In this work, we address this gap by curating two datasets: PigDetect for object detection and PigTrack for multi-object tracking. The datasets are based on diverse image and video material from realistic barn conditions, and include challenging scenarios such as occlusions or bad visibility. For object detection, we show that challenging training images improve detection performance beyond what is achievable with randomly sampled images alone. Comparing different approaches, we found that state-of-the-art models offer substantial improvements in detection quality over real-time alternatives. For multi-object tracking, we observed that SORT-based methods achieve superior detection performance compared to end-to-end trainable models. However, end-to-end models show better association performance, suggesting they could become strong alternatives in the future. We also investigate characteristic failure cases of end-to-end models, providing guidance for future improvements. The detection and tracking models trained on our datasets perform well in unseen pens, suggesting good generalization capabilities. This highlights the importance of high-quality training data. The datasets and research code are made publicly available to facilitate reproducibility, re-use and further development.
comment : 16 pages, 6 figures and 8 tables
♻ ☆ CaliTex: Geometry-Calibrated Attention for View-Coherent 3D Texture Generation
Chenyu Liu, Hongze Chen, Jingzhi Bao, Lingting Zhu, Runze Zhang, Weikai Chen, Zeyu Hu, Yingda Yin, Keyang Luo, Xin Wang
Despite major advances brought by diffusion-based models, current 3D texture generation systems remain hindered by cross-view inconsistency -- textures that appear convincing from one viewpoint often fail to align across others. We find that this issue arises from attention ambiguity, where unstructured full attention is applied indiscriminately across tokens and modalities, causing geometric confusion and unstable appearance-structure coupling. To address this, we introduce CaliTex, a framework of geometry-calibrated attention that explicitly aligns attention with 3D structure. It introduces two modules: Part-Aligned Attention that enforces spatial alignment across semantically matched parts, and Condition-Routed Attention which routes appearance information through geometry-conditioned pathways to maintain spatial fidelity. Coupled with a two-stage diffusion transformer, CaliTex makes geometric coherence an inherent behavior of the network rather than a byproduct of optimization. Empirically, CaliTex produces seamless and view-consistent textures and outperforms both open-source and commercial baselines.
♻ ☆ Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning
Andy Li, Aiden Durrant, Milan Markovic, Tianjin Huang, Souvik Kundu, Tianlong Chen, Lu Yin, Georgios Leontidis
Pruning of deep neural networks has been an effective technique for reducing model size while preserving most of the performance of dense networks, crucial for deploying models on memory and power-constrained devices. While recent sparse learning methods have shown promising performance up to moderate sparsity levels such as 95% and 98%, accuracy quickly deteriorates when pushing sparsities to extreme levels due to unique challenges such as fragile gradient flow. In this work, we explore network performance beyond the commonly studied sparsities, and develop techniques that encourage stable training without accuracy collapse even at extreme sparsities, including 99.90%, 99.95\% and 99.99% on ResNet architectures. We propose three complementary techniques that enhance sparse training through different mechanisms: 1) Dynamic ReLU phasing, where DyReLU initially allows for richer parameter exploration before being gradually replaced by standard ReLU, 2) weight sharing which reuses parameters within a residual layer while maintaining the same number of learnable parameters, and 3) cyclic sparsity, where both sparsity levels and sparsity patterns evolve dynamically throughout training to better encourage parameter exploration. We evaluate our method, which we term Extreme Adaptive Sparse Training (EAST) at extreme sparsities using ResNet-34 and ResNet-50 on CIFAR-10, CIFAR-100, and ImageNet, achieving competitive or improved performance compared to existing methods, with notable gains at extreme sparsity levels.
comment : V4: moderate revisions and overall improvements for journal camera ready submission
♻ ☆ Flow Equivariant Recurrent Neural Networks NeurIPS '25
T. Anderson Keller
Data arrives at our senses as a continuous stream, smoothly transforming from one instant to the next. These smooth transformations can be viewed as continuous symmetries of the environment that we inhabit, defining equivalence relations between stimuli over time. In machine learning, neural network architectures that respect symmetries of their data are called equivariant and have provable benefits in terms of generalization ability and sample efficiency. To date, however, equivariance has been considered only for static transformations and feed-forward networks, limiting its applicability to sequence models, such as recurrent neural networks (RNNs), and corresponding time-parameterized sequence transformations. In this work, we extend equivariant network theory to this regime of 'flows' -- one-parameter Lie subgroups capturing natural transformations over time, such as visual motion. We begin by showing that standard RNNs are generally not flow equivariant: their hidden states fail to transform in a geometrically structured manner for moving stimuli. We then show how flow equivariance can be introduced, and demonstrate that these models significantly outperform their non-equivariant counterparts in terms of training speed, length generalization, and velocity generalization, on both next step prediction and sequence classification. We present this work as a first step towards building sequence models that respect the time-parameterized symmetries which govern the world around us.
comment : NeurIPS '25, Spotlight
♻ ☆ RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-Wo rl d Users SC
Suyu Ye, Haojun Shi, Darren Shih, Hyokun Yun, Tanya Roosta, Tianmin Shu
To achieve successful assistance with long-horizon web-based tasks, AI agents must be able to sequentially follow real-world user instructions over a long period. Unlike existing web-based agent benchmarks, sequential instruction following in the real world poses significant challenges beyond performing a single, clearly defined task. For instance, real-world human instructions can be ambiguous, require different levels of AI assistance, and may evolve over time, reflecting changes in the user's mental state. To address this gap, we introduce RealWebAssist, a novel benchmark designed to evaluate sequential instruction-following in realistic scenarios involving long-horizon interactions with the web, visual GUI grounding, and understanding ambiguous real-world user instructions. RealWebAssist includes a dataset of sequential instructions collected from real-world human users. Each user instructs a web-based assistant to perform a series of tasks on multiple websites. A successful agent must reason about the true intent behind each instruction, keep track of the mental state of the user, understand user-specific routines, and ground the intended tasks to actions on the correct GUI elements. Our experimental results show that state-of-the-art models struggle to understand and ground user instructions, posing critical challenges in following real-world user instructions for long-horizon web assistance.
comment : Project Website: https://scai.cs.jhu.edu/projects/RealWebAssist/ Code: https://github.com/SCAI-JHU/RealWebAssist
♻ ☆ OmniSVG: A Unified Scalable Vector Graphics Generation Model
Yiying Yang, Wei Cheng, Sijin Chen, Xianfang Zeng, Fukun Yin, Jiaxu Zhang, Liao Wang, Gang Yu, Xingjun Ma, Yu-Gang Jiang
Scalable Vector Graphics (SVG) is an important image format widely adopted in graphic design because of their resolution independence and editability. The study of generating high-quality SVG has continuously drawn attention from both designers and researchers in the AIGC community. However, existing methods either produces unstructured outputs with huge computational cost or is limited to generating monochrome icons of over-simplified structures. To produce high-quality and complex SVG, we propose OmniSVG, a unified framework that leverages pre-trained Vision-Language Models (VLMs) for end-to-end multimodal SVG generation. By parameterizing SVG commands and coordinates into discrete tokens, OmniSVG decouples structural logic from low-level geometry for efficient training while maintaining the expressiveness of complex SVG structure. To further advance the development of SVG synthesis, we introduce MMSVG-2M, a multimodal dataset with two million richly annotated SVG assets, along with a standardized evaluation protocol for conditional SVG generation tasks. Extensive experiments show that OmniSVG outperforms existing methods and demonstrates its potential for integration into professional SVG design workflows.
comment : 20 pages; Project Page: https://omnisvg.github.io/
♻ ☆ FedHK-MVFC: Federated Heat Kernel Multi-View Clustering
Kristina P. Sinaga
In the realm of distributed artificial intelligence (AI) and privacy-focused medical applications, this paper proposes a multi-view clustering framework that links quantum field theory with federated healthcare analytics. The method uses heat kernel coefficients from spectral analysis to convert Euclidean distances into geometry-aware similarity measures that capture the structure of diverse medical data. The framework is presented through the heat kernel distance (HKD) transformation, which has convergence guarantees. Two algorithms have been developed: The first, Heat Kernel-Enhanced Multi-View Fuzzy Clustering (HK-MVFC), is used for central analysis. The second, Federated Heat Kernel Multi-View Fuzzy Clustering (FedHK-MVFC), is used for secure, privacy-preserving learning across hospitals. FedHK-MVFC uses differential privacy and secure aggregation to enable HIPAA-compliant collaboration. Tests on synthetic cardiovascular patient datasets demonstrate increased clustering accuracy, reduced communication, and retained efficiency compared to centralized methods. After being validated on 10,000 synthetic patient records across two hospitals, the methods proved useful for collaborative phenotyping involving electrocardiogram (ECG) data, cardiac imaging data, and behavioral data. The proposed methods' theoretical contributions include update rules with proven convergence, adaptive view weighting, and privacy-preserving protocols. These contributions establish a new standard for geometry-aware federated learning in healthcare, translating advanced mathematics into practical solutions for analyzing sensitive medical data while ensuring rigor and clinical relevance.
comment : 53 pages, 11 figures, and 9 tables
♻ ☆ DenoiseGS: Gaussian Reconstruction Model for Burst Denoising
Yongsen Cheng, Yuanhao Cai, Yulun Zhang
Burst denoising methods are crucial for enhancing images captured on handheld devices, but they often struggle with large motion or suffer from prohibitive computational costs. In this paper, we propose DenoiseGS, the first framework to leverage the efficiency of 3D Gaussian Splatting for burst denoising. Our approach addresses two key challenges when applying feedforward Gaussian reconsturction model to noisy inputs: the degradation of Gaussian point clouds and the loss of fine details. To this end, we propose a Gaussian self-consistency (GSC) loss, which regularizes the geometry predicted from noisy inputs with high-quality Gaussian point clouds. These point clouds are generated from clean inputs by the same model that we are training, thereby alleviating potential bias or domain gaps. Additionally, we introduce a log-weighted frequency (LWF) loss to strengthen supervision within the spectral domain, effectively preserving fine-grained details. The LWF loss adaptively weights frequency discrepancies in a logarithmic manner, emphasizing challenging high-frequency details. Extensive experiments demonstrate that DenoiseGS significantly exceeds the state-of-the-art NeRF-based methods on both burst denoising and novel view synthesis under noisy conditions, while achieving 250$\times$ faster inference speed. Code and models are released at https://github.com/yscheng04/DenoiseGS.
comment : Update Abstract
♻ ☆ TTSnap: Test-Time Scaling of Diffusion Models via Noise-Aware Pruning
Qingtao Yu, Changlin Song, Minghao Sun, Zhengyang Yu, Vinay Kumar Verma, Soumya Roy, Sumit Negi, Hongdong Li, Dylan Campbell
A prominent approach to test-time scaling for text-to-image diffusion models formulates the problem as a search over multiple noise seeds, selecting the one that maximizes a certain image-reward function. The effectiveness of this strategy heavily depends on the number and diversity of noise seeds explored. However, verifying each candidate is computationally expensive, because each must be fully denoised before a reward can be computed. This severely limits the number of samples that can be explored under a fixed budget. We propose test-time scaling with noise-aware pruning (TTSnap), a framework that prunes low-quality candidates without fully denoising them. The key challenge is that reward models are learned in the clean image domain, and the ranking of rewards predicted for intermediate estimates are often inconsistent with those predicted for clean images. To overcome this, we train noise-aware reward models via self-distillation to align the reward for intermediate estimates with that of the final clean images. To stabilize learning across different noise levels, we adopt a curriculum training strategy that progressively shifts the data domain from clean images to noise images. In addition, we introduce a new metric that measures reward alignment and computational budget utilization. Experiments demonstrate that our approach improves performance by over 16\% compared with existing methods, enabling more efficient and effective test-time scaling. It also provides orthogonal gains when combined with post-training techniques and local test-time optimization. Code: https://github.com/TerrysLearning/TTSnap/.
♻ ☆ Towards Fast and Scalable Normal Integration using Continuous Components WACV
Francesco Milano, Jen Jen Chung, Lionel Ott, Roland Siegwart
Surface normal integration is a fundamental problem in computer vision, dealing with the objective of reconstructing a surface from its corresponding normal map. Existing approaches require an iterative global optimization to jointly estimate the depth of each pixel, which scales poorly to larger normal maps. In this paper, we address this problem by recasting normal integration as the estimation of relative scales of continuous components. By constraining pixels belonging to the same component to jointly vary their scale, we drastically reduce the number of optimization variables. Our framework includes a heuristic to accurately estimate continuous components from the start, a strategy to rebalance optimization terms, and a technique to iteratively merge components to further reduce the size of the problem. Our method achieves state-of-the-art results on the standard normal integration benchmark in as little as a few seconds and achieves one-order-of-magnitude speedup over pixel-level approaches on large-resolution normal maps.
comment : Accepted by the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2026, first round. Camera-ready version. 17 pages, 9 figures, 6 tables. Code is available at https://github.com/francescomilano172/normal_integration_continuous_components
♻ ☆ B2N3D: Progressive Learning from Binary to N-ary Relation ships for 3D Object Grounding
Feng Xiao, Hongbin Xu, Hai Ci, Wenxiong Kang
Localizing 3D objects using natural language is essential for robotic scene understanding. The descriptions often involve multiple spatial relationships to distinguish similar objects, making 3D-language alignment difficult. Current methods only model relationships for pairwise objects, ignoring the global perceptual significance of n-ary combinations in multi-modal relational understanding. To address this, we propose a novel progressive relational learning framework for 3D object grounding. We extend relational learning from binary to n-ary to identify visual relations that match the referential description globally. Given the absence of specific annotations for referred objects in the training data, we design a grouped supervision loss to facilitate n-ary relational learning. In the scene graph created with n-ary relationships, we use a multi-modal network with hybrid attention mechanisms to further localize the target within the n-ary combinations. Experiments and ablation studies on the ReferIt3D and ScanRefer benchmarks demonstrate that our method outperforms the state-of-the-art, and proves the advantages of the n-ary relational perception in 3D localization.
♻ ☆ Hybrid Swin Attention Networks for Simultaneously Low-Dose PET and CT Denoising
Yichao Liu, Hengzhi Xue, YueYang Teng, Junwen Guo
Low-dose computed tomography (LDCT) and positron emission tomography (PET) have emerged as safer alternatives to conventional imaging modalities by significantly reducing radiation exposure. However, this reduction often results in increased noise and artifacts, which can compromise diagnostic accuracy. Consequently, denoising for LDCT/PET has become a vital area of research aimed at enhancing image quality while maintaining radiation safety. In this study, we introduce a novel Hybrid Swin Attention Network (HSANet), which incorporates Efficient Global Attention (EGA) modules and a hybrid upsampling module. The EGA modules enhance both spatial and channel-wise interaction, improving the network's capacity to capture relevant features, while the hybrid upsampling module mitigates the risk of overfitting to noise. We validate the proposed approach using a publicly available LDCT/PET dataset. Experimental results demonstrate that HSANet achieves superior denoising performance compared to existing methods, while maintaining a lightweight model size suitable for deployment on GPUs with standard memory configurations. This makes our approach highly practical for real-world clinical applications.
♻ ☆ RDTF: Resource-efficient Dual-mask Training Framework for Multi-frame Animated Sticker Generation
Zhiqiang Yuan, Ting Zhang, Peixiang Luo, Ying Deng, Jiapei Zhang, Zexi Jia, Jinchao Zhang, Jie Zhou
Recently, significant advancements have been achieved in video generation technology, but applying it to resource-constrained downstream tasks like multi-frame animated sticker generation (ASG) characterized by low frame rates, abstract semantics, and long tail frame length distribution-remains challenging. Parameter-efficient fine-tuning (PEFT) techniques (e.g., Adapter, LoRA) for large pre-trained models suffer from insufficient fitting ability and source-domain knowledge interference. In this paper, we propose Resource-Efficient Dual-Mask Training Framework (RDTF), a dedicated solution for multi-frame ASG task under resource constraints. We argue that training a compact model from scratch with million-level samples outperforms PEFT on large models, with RDTF realizing this via three core designs: 1) a Discrete Frame Generation Network (DFGN) optimized for low-frame-rate ASG, ensuring parameter efficiency; 2) a dual-mask based data utilization strategy to enhance the availability and diversity of limited data; 3) a difficulty-adaptive curriculum learning method that decomposes sample entropy into static and adaptive components, enabling easy-to-difficult training convergence. To provide high-quality data support for RDTFs training from scratch, we construct VSD2M-a million-level multi-modal animated sticker dataset with rich annotations (static and animated stickers, action-focused text descriptions)-filling the gap of dedicated animated data for ASG task. Experiments demonstrate that RDTF is quantitatively and qualitatively superior to state-of-the-art PEFT methods (e.g., I2V-Adapter, SimDA) on ASG tasks, verifying the feasibility of our framework under resource constraints.
comment : Submitted to TMM
♻ ☆ Fusion or Confusion? Assessing the impact of visible-thermal image fusion for automated wildlife detection
Camille Dionne-Pierre, Samuel Foucher, Jérôme Théau, Jérôme Lemaître, Patrick Charbonneau, Maxime Brousseau, Mathieu Varin
Efficient wildlife monitoring methods are necessary for biodiversity conservation and management. The combination of remote sensing, aerial imagery and deep learning offer promising opportunities to renew or improve existing survey methods. The complementary use of visible (VIS) and thermal infrared (TIR) imagery can add information compared to a single-source image and improve results in an automated detection context. However, the alignment and fusion process can be challenging, especially since visible and thermal images usually have different fields of view (FOV) and spatial resolutions. This research presents a case study on the great blue heron (Ardea herodias) to evaluate the performances of synchronous aerial VIS and TIR imagery to automatically detect individuals and nests using a YOLO11n model. Two VIS-TIR fusion methods were tested and compared: an early fusion approach and a late fusion approach, to determine if the addition of the TIR image gives any added value compared to a VIS-only model. VIS and TIR images were automatically aligned using a deep learning model. A principal component analysis fusion method was applied to VIS-TIR image pairs to form the early fusion dataset. A classification and regression tree was used to process the late fusion dataset, based on the detection from the VIS-only and TIR-only trained models. Across all classes, both late and early fusion improved the F1 score compared to the VIS-only model. For the main class, occupied nest, the late fusion improved the F1 score from 90.2 (VIS-only) to 93.0%. This model was also able to identify false positives from both sources with 90% recall. Although fusion methods seem to give better results, this approach comes with a limiting TIR FOV and alignment constraints that eliminate data. Using an aircraft-mounted very high-resolution visible sensor could be an interesting option for operationalizing surveys.
comment : 19 pages, 9 figures, submitted to Remote Sensing in Ecology and Conservation
♻ ☆ MoH: Multi-Head Attention as Mixture-of-Head Attention ICML 2025
Peng Jin, Bo Zhu, Li Yuan, Shuicheng Yan
In this work, we upgrade the multi-head attention mechanism, the core of the Transformer model, to improve efficiency while maintaining or surpassing the previous accuracy level. We show that multi-head attention can be expressed in the summation form. Drawing on the insight that not all attention heads hold equal significance, we propose Mixture-of-Head attention (MoH), a new architecture that treats attention heads as experts in the Mixture-of-Experts (MoE) mechanism. MoH has two significant advantages: First, MoH enables each token to select the appropriate attention heads, enhancing inference efficiency without compromising accuracy or increasing the number of parameters. Second, MoH replaces the standard summation in multi-head attention with a weighted summation, introducing flexibility to the attention mechanism and unlocking extra performance potential. Extensive experiments on ViT, DiT, and LLMs demonstrate that MoH outperforms multi-head attention by using only 50%-90% of the attention heads. Moreover, we demonstrate that pre-trained multi-head attention models, such as LLaMA3-8B, can be further continue-tuned into our MoH models. Notably, MoH-LLaMA3-8B achieves an average accuracy of 64.0% across 14 benchmarks, outperforming LLaMA3-8B by 2.4% by utilizing only 75% of the attention heads. We believe the proposed MoH is a promising alternative to multi-head attention and provides a strong foundation for developing advanced and efficient attention-based models.
comment : Accepted by ICML 2025, code: https://github.com/SkyworkAI/MoH
♻ ☆ ReasonEdit: Towards Reasoning -Enhanced Image Editing Models
Fukun Yin, Shiyu Liu, Yucheng Han, Zhibo Wang, Peng Xing, Rui Wang, Wei Cheng, Yingming Wang, Aojie Li, Zixin Yin, Pengtao Chen, Xiangyu Zhang, Daxin Jiang, Xianfang Zeng, Gang Yu
Recent advances in image editing models have shown remarkable progress. A common architectural design couples a multimodal large language model (MLLM) encoder with a diffusion decoder, as seen in systems such as Step1X-Edit and Qwen-Image-Edit, where the MLLM encodes both the reference image and the instruction but remains frozen during training. In this work, we demonstrate that unlocking the reasoning capabilities of MLLM can further push the boundaries of editing models. Specifically, we explore two reasoning mechanisms, thinking and reflection, which enhance instruction understanding and editing accuracy. Based on that, our proposed framework enables image editing in a thinking-editing-reflection loop: the thinking mechanism leverages the world knowledge of MLLM to interpret abstract instructions, while the reflection reviews editing results, automatically corrects unintended manipulations, and identifies the stopping round. Extensive experiments demonstrate that our reasoning approach achieves significant performance gains, with improvements of ImgEdit (+4.3%), GEdit (+4.7%), and Kris (+8.2%) when initializing our DiT from the Step1X-Edit (ReasonEdit-S), and also outperforms previous open-source methods on both GEdit and Kris when integrated with Qwen-Image-Edit (ReasonEdit-Q).
comment : code: https://github.com/stepfun-ai/Step1X-Edit
♻ ☆ Class-Conditional Distribution Balancing for Group Robust Classification
Miaoyun Zhao, Chenrong Li, Qiang Zhang
Spurious correlations that lead models to correct predictions for the wrong reasons pose a critical challenge for robust real-world generalization. Existing research attributes this issue to group imbalance and addresses it by maximizing group-balanced or worst-group accuracy, which heavily relies on expensive bias annotations. A compromise approach involves predicting bias information using extensively pretrained foundation models, which requires large-scale data and becomes impractical for resource-limited rare domains. To address these challenges, we offer a novel perspective by reframing the spurious correlations as imbalances or mismatches in class-conditional distributions, and propose a simple yet effective robust learning method that eliminates the need for both bias annotations and predictions. With the goal of maximizing the conditional entropy (uncertainty) of the label given spurious factors, our method leverages a sample reweighting strategy to achieve class-conditional distribution balancing, which automatically highlights minority groups and classes, effectively dismantling spurious correlations and producing a debiased data distribution for classification. Extensive experiments and analysis demonstrate that our approach consistently delivers state-of-the-art performance, rivaling methods that rely on bias supervision.
♻ ☆ iMontage: Unified, Versatile, Highly Dynamic Many-to-many Image Generation
Zhoujie Fu, Xianfang Zeng, Jinghong Lan, Xinyao Liao, Cheng Chen, Junyi Chen, Jiacheng Wei, Wei Cheng, Shiyu Liu, Yunuo Chen, Gang Yu, Guosheng Lin
Pre-trained video models learn powerful priors for generating high-quality, temporally coherent content. While these models excel at temporal coherence, their dynamics are often constrained by the continuous nature of their training data. We hypothesize that by injecting the rich and unconstrained content diversity from image data into this coherent temporal framework, we can generate image sets that feature both natural transitions and a far more expansive dynamic range. To this end, we introduce iMontage, a unified framework designed to repurpose a powerful video model into an all-in-one image generator. The framework consumes and produces variable-length image sets, unifying a wide array of image generation and editing tasks. To achieve this, we propose an elegant and minimally invasive adaptation strategy, complemented by a tailored data curation process and training paradigm. This approach allows the model to acquire broad image manipulation capabilities without corrupting its invaluable original motion priors. iMontage excels across several mainstream many-in-many-out tasks, not only maintaining strong cross-image contextual consistency but also generating scenes with extraordinary dynamics that surpass conventional scopes. Find our homepage at: https://kr1sjfu.github.io/iMontage-web/.
comment : Our homepage: https://kr1sjfu.github.io/iMontage-web/
♻ ☆ Prediction of Distant Metastasis in Head and Neck Cancer Patients Using Tumor and Peritumoral Multi-Modal Deep Learning
Nuo Tong, Changhao Liu, Zizhao Tang, Feifan Sun, Yingping Li, Shuiping Gou, Mei Shi
Although the combined treatment of surgery, radiotherapy, chemotherapy, and emerging target therapy has significantly improved the outcomes of patients with head and neck cancer, distant metastasis remains the leading cause of treatment failure. In this study, we propose a deep learning-based multimodal framework integrating CT imaging, radiomics, and clinical data to predict metastasis risk in HNSCC. A total of 1497 patients were retrospectively analyzed. Tumor and organ masks were generated from pretreatment CT scans, from which a 3D Swin Transformer extracted deep imaging features, while 1562 radiomics features were reduced to 36 via correlation filtering and random forest selection. Clinical data (age, sex, smoking, and alcohol status) were encoded and fused with imaging features, and the multimodal representation was fed into a fully connected network for prediction. Five-fold cross-validation was used to assess performance via AUC, accuracy, sensitivity, and specificity. The multimodal model outperformed all single-modality baselines. The deep learning module alone achieved an AUC of 0.715, whereas multimodal fusion significantly improved performance (AUC = 0.803, ACC = 0.752, SEN = 0.730, SPE = 0.758). Stratified analyses confirmed good generalizability across tumor subtypes. Ablation experiments demonstrated complementary contributions from each modality, and the 3D Swin Transformer provided more robust representations than conventional architectures. This multimodal deep learning model enables accurate, non-invasive metastasis prediction in HNSCC and shows strong potential for individualized treatment planning.
comment : 23 pages, 6 figures, 7 tables. Nuo Tong and Changhao Liu contributed equally. Corresponding Authors: Shuiping Gou and Mei Shi
♻ ☆ Fast Multi-view Consistent 3D Editing with Video Priors AAAI2026
Liyi Chen, Ruihuang Li, Guowen Zhang, Pengfei Wang, Lei Zhang
Text-driven 3D editing enables user-friendly 3D object or scene editing with text instructions. Due to the lack of multi-view consistency priors, existing methods typically resort to employing 2D generation or editing models to process each view individually, followed by iterative 2D-3D-2D updating. However, these methods are not only time-consuming but also prone to over-smoothed results because the different editing signals gathered from different views are averaged during the iterative process. In this paper, we propose generative Video Prior based 3D Editing (ViP3DE) to employ the temporal consistency priors from pre-trained video generation models for multi-view consistent 3D editing in a single forward pass. Our key insight is to condition the video generation model on a single edited view to generate other consistent edited views for 3D updating directly, thereby bypassing the iterative editing paradigm. Since 3D updating requires edited views to be paired with specific camera poses, we propose motion-preserved noise blending for the video model to generate edited views at predefined camera poses. In addition, we introduce geometry-aware denoising to further enhance multi-view consistency by integrating 3D geometric priors into video models. Extensive experiments demonstrate that our proposed ViP3DE can achieve high-quality 3D editing results even within a single forward pass, significantly outperforming existing methods in both editing quality and speed.
comment : accepted by AAAI2026
♻ ☆ VITA: Vision -to-Action Flow Matching Policy
Dechen Gao, Boqi Zhao, Andrew Lee, Ian Chuang, Hanchu Zhou, Hang Wang, Zhe Zhao, Junshan Zhang, Iman Soltani
Conventional flow matching and diffusion-based policies sample through iterative denoising from standard noise distributions (e.g., Gaussian), and require conditioning modules to repeatedly incorporate visual information during the generative process, incurring substantial time and memory overhead. To reduce the complexity, we develop VITA(VIsion-To-Action policy), a noise-free and conditioning-free flow matching policy learning framework that directly flows from visual representations to latent actions. Since the source of the flow is visually grounded, VITA eliminates the need of visual conditioning during generation. As expected, bridging vision and action is challenging, because actions are lower-dimensional, less structured, and sparser than visual representations; moreover, flow matching requires the source and target to have the same dimensionality. To overcome this, we introduce an action autoencoder that maps raw actions into a structured latent space aligned with visual latents, trained jointly with flow matching. To further prevent latent space collapse, we propose flow latent decoding, which anchors the latent generation process by backpropagating the action reconstruction loss through the flow matching ODE (ordinary differential equation) solving steps. We evaluate VITA on 9 simulation and 5 real-world tasks from ALOHA and Robomimic. VITA achieves 1.5x-2x faster inference compared to conventional methods with conditioning modules, while outperforming or matching state-of-the-art policies. Codes, datasets, and demos are available at our project page: https://ucd-dare.github.io/VITA/.
comment : Project page: https://ucd-dare.github.io/VITA/ Code: https://github.com/ucd-dare/VITA
♻ ☆ Capturing Context -Aware Route Choice Semantics for Trajectory Representation Learning
Ji Cao, Yu Wang, Tongya Zheng, Jie Song, Qinghong Guo, Zujie Ren, Canghong Jin, Gang Chen, Mingli Song
Trajectory representation learning (TRL) aims to encode raw trajectory data into low-dimensional embeddings for downstream tasks such as travel time estimation, mobility prediction, and trajectory similarity analysis. From a behavioral perspective, a trajectory reflects a sequence of route choices within an urban environment. However, most existing TRL methods ignore this underlying decision-making process and instead treat trajectories as static, passive spatiotemporal sequences, thereby limiting the semantic richness of the learned representations. To bridge this gap, we propose CORE, a TRL framework that integrates context-aware route choice semantics into trajectory embeddings. CORE first incorporates a multi-granular Environment Perception Module, which leverages large language models (LLMs) to distill environmental semantics from point of interest (POI) distributions, thereby constructing a context-enriched road network. Building upon this backbone, CORE employs a Route Choice Encoder with a mixture-of-experts (MoE) architecture, which captures route choice patterns by jointly leveraging the context-enriched road network and navigational factors. Finally, a Transformer encoder aggregates the route-choice-aware representations into a global trajectory embedding. Extensive experiments on 4 real-world datasets across 6 downstream tasks demonstrate that CORE consistently outperforms 12 state-of-the-art TRL methods, achieving an average improvement of 9.79% over the best-performing baseline. Our code is available at https://github.com/caoji2001/CORE.
♻ ☆ Self-Supervised One-Step Diffusion Refinement for Snapshot Compressive Imaging
Shaoguang Huang, Yunzhen Wang, Haijin Zeng, Hongyu Chen, Hongyan Zhang
Snapshot compressive imaging (SCI) captures multispectral images (MSIs) using a single coded two-dimensional (2-D) measurement, but reconstructing high-fidelity MSIs from these compressed inputs remains a fundamentally ill-posed challenge. While diffusion-based reconstruction methods have recently raised the bar for quality, they face critical limitations: a lack of large-scale MSI training data, adverse domain shifts from RGB-pretrained models, and inference inefficiencies due to multi-step sampling. These drawbacks restrict their practicality in real-world applications. In contrast to existing methods, which either follow costly iterative refinement or adapt subspace-based embeddings for diffusion models (e.g. DiffSCI, PSR-SCI), we introduce a fundamentally different paradigm: a self-supervised One-Step Diffusion (OSD) framework specifically designed for SCI. The key novelty lies in using a single-step diffusion refiner to correct an initial reconstruction, eliminating iterative denoising entirely while preserving generative quality. Moreover, we adopt a self-supervised equivariant learning strategy to train both the predictor and refiner directly from raw 2-D measurements, enabling generalization to unseen domains without the need for ground-truth MSI. To further address the challenge of limited MSI data, we design a band-selection-driven distillation strategy that transfers core generative priors from large-scale RGB datasets, effectively bridging the domain gap. Extensive experiments confirm that our approach sets a new benchmark, yielding PSNR gains of 3.44 dB, 1.61 dB, and 0.28 dB on the Harvard, NTIRE, and ICVL datasets, respectively, while reducing reconstruction time by 97.5%. This remarkable improvement in efficiency and adaptability makes our method a significant advancement in SCI reconstruction, combining both accuracy and practicality for real-world deployment.
♻ ☆ Improving Partially Observed Trajectories Forecasting by Target-driven Self-Distillation
Peng Shu, Pengfei Zhu, Mengshi Qi, Liang Liu
Accurate prediction of future trajectories of traffic agents is essential for ensuring safe autonomous driving. However, partially observed trajectories can significantly degrade the performance of even state-of-the-art models. Previous approaches often rely on knowledge distillation to transfer features from fully observed trajectories to partially observed ones. This involves firstly training a fully observed model and then using a distillation process to create the final model. While effective, they require multi-stage training, making the training process very expensive. Moreover, knowledge distillation can lead to a performance degradation of the model. In this paper, we introduce a Target-drivenSelf-Distillation method (TSD) for motion forecasting. Our method leverages predicted accurate targets to guide the model in making predictions under partial observation conditions. By employing self-distillation, the model learns from the feature distributions of both fully observed and partially observed trajectories during a single end-to-end training process. This enhances the model's ability to predict motion accurately in both fully observed and partially observed scenarios. We evaluate our method on multiple datasets and state-of-the-art motion forecasting models. Extensive experimental results demonstrate that our approach achieves significant performance improvements in both settings. To facilitate further research, we will release our code and model checkpoints.
♻ ☆ Rank Matters: Understanding and Defending Model Inversion Attacks via Low-Rank Feature Filtering KDD 2026
Hongyao Yu, Yixiang Qiu, Hao Fang, Tianqu Zhuang, Bin Chen, Sijin Yu, Bin Wang, Shu-Tao Xia, Ke Xu
Model Inversion Attacks (MIAs) pose a significant threat to data privacy by reconstructing sensitive training samples from the knowledge embedded in trained machine learning models. Despite recent progress in enhancing the effectiveness of MIAs across diverse settings, defense strategies have lagged behind -- struggling to balance model utility with robustness against increasingly sophisticated attacks. In this work, we propose the ideal inversion error to measure the privacy leakage, and our theoretical and empirical investigations reveals that higher-rank features are inherently more prone to privacy leakage. Motivated by this insight, we propose a lightweight and effective defense strategy based on low-rank feature filtering, which explicitly reduces the attack surface by constraining the dimension of intermediate representations. Extensive experiments across various model architectures and datasets demonstrate that our method consistently outperforms existing defenses, achieving state-of-the-art performance against a wide range of MIAs. Notably, our approach remains effective even in challenging regimes involving high-resolution data and high-capacity models, where prior defenses fail to provide adequate protection.
comment : KDD 2026 Accept
♻ ☆ Rethinking Multimodal Point Cloud Completion: A Completion-by-Correction Perspective AAAI 2026
Wang Luo, Di Wu, Hengyuan Na, Yinlin Zhu, Miao Hu, Guocong Quan
Point cloud completion aims to reconstruct complete 3D shapes from partial observations, which is a challenging problem due to severe occlusions and missing geometry. Despite recent advances in multimodal techniques that leverage complementary RGB images to compensate for missing geometry, most methods still follow a Completion-by-Inpainting paradigm, synthesizing missing structures from fused latent features. We empirically show that this paradigm often results in structural inconsistencies and topological artifacts due to limited geometric and semantic constraints. To address this, we rethink the task and propose a more robust paradigm, termed Completion-by-Correction, which begins with a topologically complete shape prior generated by a pretrained image-to-3D model and performs feature-space correction to align it with the partial observation. This paradigm shifts completion from unconstrained synthesis to guided refinement, enabling structurally consistent and observation-aligned reconstruction. Building upon this paradigm, we introduce PGNet, a multi-stage framework that conducts dual-feature encoding to ground the generative prior, synthesizes a coarse yet structurally aligned scaffold, and progressively refines geometric details via hierarchical correction. Experiments on the ShapeNetViPC dataset demonstrate the superiority of PGNet over state-of-the-art baselines in terms of average Chamfer Distance (-23.5%) and F-score (+7.1%).
comment : Accepted by AAAI 2026
♻ ☆ TempoMaster: Efficient Long Video Generation via Next-Frame-Rate Prediction
Yukuo Ma, Cong Liu, Junke Wang, Junqi Liu, Haibin Huang, Zuxuan Wu, Chi Zhang, Xuelong Li
We present TempoMaster, a novel framework that formulates long video generation as next-frame-rate prediction. Specifically, we first generate a low-frame-rate clip that serves as a coarse blueprint of the entire video sequence, and then progressively increase the frame rate to refine visual details and motion continuity. During generation, TempoMaster employs bidirectional attention within each frame-rate level while performing autoregression across frame rates, thus achieving long-range temporal coherence while enabling efficient and parallel synthesis. Extensive experiments demonstrate that TempoMaster establishes a new state-of-the-art in long video generation, excelling in both visual and temporal quality.
comment : for more information, see https://scottykma.github.io/tempomaster-gitpage/
♻ ☆ 3EED: Ground Everything Everywhere in 3D NeurIPS 2025
Rong Li, Yuhao Dong, Tianshuai Hu, Ao Liang, Youquan Liu, Dongyue Lu, Liang Pan, Lingdong Kong, Junwei Liang, Ziwei Liu
Visual grounding in 3D is the key for embodied agents to localize language-referred objects in open-world environments. However, existing benchmarks are limited to indoor focus, single-platform constraints, and small scale. We introduce 3EED, a multi-platform, multi-modal 3D grounding benchmark featuring RGB and LiDAR data from vehicle, drone, and quadruped platforms. We provide over 128,000 objects and 22,000 validated referring expressions across diverse outdoor scenes -- 10x larger than existing datasets. We develop a scalable annotation pipeline combining vision-language model prompting with human verification to ensure high-quality spatial grounding. To support cross-platform learning, we propose platform-aware normalization and cross-modal alignment techniques, and establish benchmark protocols for in-domain and cross-platform evaluations. Our findings reveal significant performance gaps, highlighting the challenges and opportunities of generalizable 3D grounding. The 3EED dataset and benchmark toolkit are released to advance future research in language-driven 3D embodied perception.
comment : NeurIPS 2025 DB Track; 38 pages, 17 figures, 10 tables; Project Page at https://project-3eed.github.io/
♻ ☆ InsightDrive: Insight Scene Representation for End-to-End Autonomous Driving
Ruiqi Song, Xianda Guo, Yanlun Peng, Qinggong Wei, Hangbin Wu, Long Chen
Conventional end-to-end autonomous driving methods often rely on explicit global scene representations, which typically consist of 3D object detection, online mapping, and motion prediction. In contrast, human drivers selectively attend to task-relevant regions and implicitly reason over the broader traffic context. Motivated by this observation, we introduce a lightweight end-to-end autonomous driving framework, InsightDrive. Unlike approaches that directly embed large language models (LLMs), InsightDrive introduces an Insight scene representation that jointly models attention-centric explicit scene representation and reasoning-centric implicit scene representation, so that scene understanding aligns more closely with human cognitive patterns for trajectory planning. To this end, we employ Chain-of-Thought (CoT) instructions to model human driving cognition and design a task-level Mixture-of-Experts (MoE) adapter that injects this knowledge into the autonomous driving model at negligible parameter cost. We further condition the planner on both explicit and implicit scene representations and employ a diffusion-based generative policy, which produces robust trajectory predictions and decisions. The overall framework establishes a knowledge distillation pipeline that transfers human driving knowledge to LLMs and subsequently to onboard models. Extensive experiments on the nuScenes and Navsim benchmarks demonstrate that InsightDrive achieves significant improvements over conventional scene representation approaches.
♻ ☆ Speech Audio Generation from dynamic MRI via a Knowledge Enhanced Conditional Variational Autoencoder
Yaxuan Li, Han Jiang, Yifei Ma, Shihua Qin, Jonghye Woo, Fangxu Xing
Dynamic Magnetic Resonance Imaging (MRI) of the vocal tract has become an increasingly adopted imaging modality for speech motor studies. Beyond image signals, systematic data loss, noise pollution, and audio file corruption can occur due to the unpredictability of the MRI acquisition environment. In such cases, generating audio from images is critical for data recovery in both clinical and research applications. However, this remains challenging due to hardware constraints, acoustic interference, and data corruption. Existing solutions, such as denoising and multi-stage synthesis methods, face limitations in audio fidelity and generalizability. To address these challenges, we propose a Knowledge Enhanced Conditional Variational Autoencoder (KE-CVAE), a novel two-step "knowledge enhancement + variational inference" framework for generating speech audio signals from cine dynamic MRI sequences. This approach introduces two key innovations: (1) integration of unlabeled MRI data for knowledge enhancement, and (2) a variational inference architecture to improve generative modeling capacity. To the best of our knowledge, this is one of the first attempts at synthesizing speech audio directly from dynamic MRI video sequences. The proposed method was trained and evaluated on an open-source dynamic vocal tract MRI dataset recorded during speech. Experimental results demonstrate its effectiveness in generating natural speech waveforms while addressing MRI-specific acoustic challenges, outperforming conventional deep learning-based synthesis approaches.
♻ ☆ Video Anomaly Detection with Semantics-Aware Information Bottleneck
Juntong Li, Lingwei Dang, Qingxin Xiao, Shishuo Shang, Jiajia Cheng, Haomin Wu, Yun Hao, Qingyao Wu
Semi-supervised video anomaly detection methods face two critical challenges: (1) Strong generalization blurs the boundary between normal and abnormal patterns. Although existing approaches attempt to alleviate this issue using memory modules, their rigid prototype-matching process limits adaptability to diverse scenarios; (2) Relying solely on low-level appearance and motion cues makes it difficult to perceive high-level semantic anomalies in complex scenes. To address these limitations, we propose SIB-VAD, a novel framework based on adaptive information bottleneck filtering and semantic-aware enhancement. We propose the Sparse Feature Filtering Module (SFFM) to replace traditional memory modules. It compresses normal features directly into a low-dimensional manifold based on the information bottleneck principle and uses an adaptive routing mechanism to dynamically select the most suitable normal bottleneck subspace. Trained only on normal data, SFFMs only learn normal low-dimensional manifolds, while abnormal features deviate and are effectively filtered. Unlike memory modules, SFFM directly removes abnormal information and adaptively handles scene variations. To improve semantic awareness, we further design a multimodal prediction framework that jointly models appearance, motion, and semantics. Through multimodal consistency constraints and joint error computation, it achieves more robust VAD performance. Experimental results validate the effectiveness of our feature filtering paradigm based on semantics-aware information bottleneck. Project page at https://qzfm.github.io/sib_vad_project_page/
♻ ☆ SPIRAL: Semantic-Aware Progressive LiDAR Scene Generation and Understanding NeurIPS 2025
Dekai Zhu, Yixuan Hu, Youquan Liu, Dongyue Lu, Lingdong Kong, Slobodan Ilic
Leveraging recent diffusion models, LiDAR-based large-scale 3D scene generation has achieved great success. While recent voxel-based approaches can generate both geometric structures and semantic labels, existing range-view methods are limited to producing unlabeled LiDAR scenes. Relying on pretrained segmentation models to predict the semantic maps often results in suboptimal cross-modal consistency. To address this limitation while preserving the advantages of range-view representations, such as computational efficiency and simplified network design, we propose Spiral, a novel range-view LiDAR diffusion model that simultaneously generates depth, reflectance images, and semantic maps. Furthermore, we introduce novel semantic-aware metrics to evaluate the quality of the generated labeled range-view data. Experiments on the SemanticKITTI and nuScenes datasets demonstrate that Spiral achieves state-of-the-art performance with the smallest parameter size, outperforming two-step methods that combine the generative and segmentation models. Additionally, we validate that range images generated by Spiral can be effectively used for synthetic data augmentation in the downstream segmentation training, significantly reducing the labeling effort on LiDAR data.
comment : NeurIPS 2025; 24 pages, 10 figures, 9 tables; Code at https://github.com/worldbench/SPIRAL
♻ ☆ AttnRegDeepLab: A Two-Stage Decoupled Framework for Interpretable Embryo Fragmentation Grading
Ming-Jhe Lee
Embryo fragmentation is a morphological indicator critical for evaluating developmental potential in In Vitro Fertilization (IVF). However, manual grading is subjective and inefficient, while existing deep learning solutions often lack clinical explainability or suffer from accumulated errors in segmentation area estimation. To address these issues, this study proposes AttnRegDeepLab (Attention-Guided Regression DeepLab), a framework characterized by dual-branch Multi-Task Learning (MTL). A vanilla DeepLabV3+ decoder is modified by integrating Attention Gates into its skip connections, explicitly suppressing cytoplasmic noise to preserve contour details. Furthermore, a Multi-Scale Regression Head is introduced with a Feature Injection mechanism to propagate global grading priors into the segmentation task, rectifying systematic quantification errors. A 2-stage decoupled training strategy is proposed to address the gradient conflict in MTL. Also, a range-based loss is designed to leverage weakly labeled data. Our method achieves robust grading precision while maintaining excellent segmentation accuracy (Dice coefficient =0.729), in contrast to the end-to-end counterpart that might minimize grading error at the expense of contour integrity. This work provides a clinically interpretable solution that balances visual fidelity and quantitative precision.
comment : 7 pages, 5 figures
♻ ☆ Bridging Granularity Gaps: Hierarchical Semantic Learning for Cross-domain Few-shot Segmentation AAAI 2026
Sujun Sun, Haowen Gu, Cheng Xie, Yanxu Ren, Mingwu Ren, Haofeng Zhang
Cross-domain Few-shot Segmentation (CD-FSS) aims to segment novel classes from target domains that are not involved in training and have significantly different data distributions from the source domain, using only a few annotated samples, and recent years have witnessed significant progress on this task. However, existing CD-FSS methods primarily focus on style gaps between source and target domains while ignoring segmentation granularity gaps, resulting in insufficient semantic discriminability for novel classes in target domains. Therefore, we propose a Hierarchical Semantic Learning (HSL) framework to tackle this problem. Specifically, we introduce a Dual Style Randomization (DSR) module and a Hierarchical Semantic Mining (HSM) module to learn hierarchical semantic features, thereby enhancing the model's ability to recognize semantics at varying granularities. DSR simulates target domain data with diverse foreground-background style differences and overall style variations through foreground and global style randomization respectively, while HSM leverages multi-scale superpixels to guide the model to mine intra-class consistency and inter-class distinction at different granularities. Additionally, we also propose a Prototype Confidence-modulated Thresholding (PCMT) module to mitigate segmentation ambiguity when foreground and background are excessively similar. Extensive experiments are conducted on four popular target domain datasets, and the results demonstrate that our method achieves state-of-the-art performance.
comment : Accepted by AAAI 2026
♻ ☆ Mixture of Ranks with Degradation-Aware Routing for One-Step Real-Wo rl d Image Super-Resolution AAAI 2026
Xiao He, Zhijun Tu, Kun Cheng, Mingrui Zhu, Jie Hu, Nannan Wang, Xinbo Gao
The demonstrated success of sparsely-gated Mixture-of-Experts (MoE) architectures, exemplified by models such as DeepSeek and Grok, has motivated researchers to investigate their adaptation to diverse domains. In real-world image super-resolution (Real-ISR), existing approaches mainly rely on fine-tuning pre-trained diffusion models through Low-Rank Adaptation (LoRA) module to reconstruct high-resolution (HR) images. However, these dense Real-ISR models are limited in their ability to adaptively capture the heterogeneous characteristics of complex real-world degraded samples or enable knowledge sharing between inputs under equivalent computational budgets. To address this, we investigate the integration of sparse MoE into Real-ISR and propose a Mixture-of-Ranks (MoR) architecture for single-step image super-resolution. We introduce a fine-grained expert partitioning strategy that treats each rank in LoRA as an independent expert. This design enables flexible knowledge recombination while isolating fixed-position ranks as shared experts to preserve common-sense features and minimize routing redundancy. Furthermore, we develop a degradation estimation module leveraging CLIP embeddings and predefined positive-negative text pairs to compute relative degradation scores, dynamically guiding expert activation. To better accommodate varying sample complexities, we incorporate zero-expert slots and propose a degradation-aware load-balancing loss, which dynamically adjusts the number of active experts based on degradation severity, ensuring optimal computational resource allocation. Comprehensive experiments validate our framework's effectiveness and state-of-the-art performance.
comment : 16 pages, Accepted by AAAI 2026, v2: corrected typos
♻ ☆ MMIF-AMIN: Adaptive Loss-Driven Multi-Scale Invertible Dense Network for Multimodal Medical Image Fusion
Tao Luo, Weihua Xu
Multimodal medical image fusion (MMIF) aims to integrate images from different modalities to produce a comprehensive image that enhances medical diagnosis by accurately depicting organ structures, tissue textures, and metabolic information. Capturing both the unique and complementary information across multiple modalities simultaneously is a key research challenge in MMIF. To address this challenge, this paper proposes a novel image fusion method, MMIF-AMIN, which features a new architecture that can effectively extract these unique and complementary features. Specifically, an Invertible Dense Network (IDN) is employed for lossless feature extraction from individual modalities. To extract complementary information between modalities, a Multi-scale Complementary Feature Extraction Module (MCFEM) is designed, which incorporates a hybrid attention mechanism, convolutional layers of varying sizes, and Transformers. An adaptive loss function is introduced to guide model learning, addressing the limitations of traditional manually-designed loss functions and enhancing the depth of data mining. Extensive experiments demonstrate that MMIF-AMIN outperforms nine state-of-the-art MMIF methods, delivering superior results in both quantitative and qualitative analyses. Ablation experiments confirm the effectiveness of each component of the proposed method. Additionally, extending MMIF-AMIN to other image fusion tasks also achieves promising performance.
comment : This manuscript is withdrawn to allow for substantial expansion and restructuring. Based on recent research progress, we plan to add Generalization experiment and reorganize the manuscript structure to improve readability and logical flow. Thank you for your understanding and support
♻ ☆ U-FaceBP: Uncertainty-aware Bayesian Ensemble Deep Learning for Face Video-based Blood Pressure Measurement
Yusuke Akamatsu, Akinori F. Ebihara, Terumi Umematsu
Blood pressure (BP) measurement is crucial for daily health assessment. Remote photoplethysmography (rPPG), which extracts pulse waves from face videos captured by a camera, has the potential to enable convenient BP measurement without specialized medical devices. However, there are various uncertainties in BP estimation using rPPG, leading to limited estimation performance and reliability. In this paper, we propose U-FaceBP, an uncertainty-aware Bayesian ensemble deep learning method for face video-based BP measurement. U-FaceBP models aleatoric and epistemic uncertainties in face video-based BP estimation with a Bayesian neural network (BNN). Additionally, we design U-FaceBP as an ensemble method, estimating BP from rPPG signals, PPG signals derived from face videos, and face images using multiple BNNs. Large-scale experiments on two datasets involving 1197 subjects from diverse racial groups demonstrate that U-FaceBP outperforms state-of-the-art BP estimation methods. Furthermore, we show that the uncertainty estimates provided by U-FaceBP are informative and useful for guiding modality fusion, assessing prediction reliability, and analyzing performance across racial groups.
comment : This work has been submitted to the IEEE for possible publication
♻ ☆ AgriPotential: A Novel Multi-Spectral and Multi-Temporal Remote Sensing Dataset for Agricultural Potentials
Mohammad El Sakka, Caroline De Pourtales, Lotfi Chaari, Josiane Mothe
Remote sensing has emerged as a critical tool for large-scale Earth monitoring and land management. In this paper, we introduce AgriPotential, a novel benchmark dataset composed of Sentinel-2 satellite imagery captured over multiple months. The dataset provides pixel-level annotations of agricultural potentials for three major crop types - viticulture, market gardening, and field crops - across five ordinal classes. AgriPotential supports a broad range of machine learning tasks, including ordinal regression, multi-label classification, and spatio-temporal modeling. The data cover diverse areas in Southern France, offering rich spectral information. AgriPotential is the first public dataset designed specifically for agricultural potential prediction, aiming to improve data-driven approaches to sustainable land use planning. The dataset and the code are freely accessible at: https://zenodo.org/records/15551829
comment : Accepted at CBMI 2025
♻ ☆ AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarios AAAI 2026
Chenglizhao Chen, Shaofeng Liang, Runwei Guan, Xiaolou Sun, Haocheng Zhao, Haiyun Jiang, Tao Huang, Henghui Ding, Qing-Long Han
Referring Multi-Object Tracking (RMOT) aims to achieve precise object detection and tracking through natural language instructions, representing a fundamental capability for intelligent robotic systems. However, current RMOT research remains mostly confined to ground-level scenarios, which constrains their ability to capture broad-scale scene contexts and perform comprehensive tracking and path planning. In contrast, Unmanned Aerial Vehicles (UAVs) leverage their expansive aerial perspectives and superior maneuverability to enable wide-area surveillance. Moreover, UAVs have emerged as critical platforms for Embodied Intelligence, which has given rise to an unprecedented demand for intelligent aerial systems capable of natural language interaction. To this end, we introduce AerialMind, the first large-scale RMOT benchmark in UAV scenarios, which aims to bridge this research gap. To facilitate its construction, we develop an innovative semi-automated collaborative agent-based labeling assistant (COALA) framework that significantly reduces labor costs while maintaining annotation quality. Furthermore, we propose HawkEyeTrack (HETrack), a novel method that collaboratively enhances vision-language representation learning and improves the perception of UAV scenarios. Comprehensive experiments validated the challenging nature of our dataset and the effectiveness of our method.
comment : AAAI 2026
♻ ☆ Rethinking Efficient Mixture-of-Experts for Remote Sensing Modality-Missing Classification
Qinghao Gao, Jiahui Qu, Yunsong Li, Wenqian Dong
Multimodal classification in remote sensing often suffers from missing modalities caused by environmental interference, sensor failures, or atmospheric effects, which severely degrade classification performance. Existing two-stage adaptation methods are computationally expensive and assume complete multimodal data during training, limiting their generalization to real-world incompleteness. To overcome these issues, we propose a Missing-aware Mixture-of-Loras (MaMOL) framework that reformulates modality missing as a multi-task learning problem. MaMOL introduces a dual-routing mechanism: a task-oriented dynamic router that adaptively activates experts for different missing patterns, and a modality-specific-shared static router that maintains stable cross-modal knowledge sharing. Unlike prior methods that train separate networks for each missing configuration, MaMOL achieves parameter-efficient adaptation via lightweight expert updates and shared expert reuse. Experiments on multiple remote sensing benchmarks demonstrate superior robustness and generalization under varying missing rates, with minimal computational overhead. Moreover, transfer experiments on natural image datasets validate its scalability and cross-domain applicability, highlighting MaMOL as a general and efficient solution for incomplete multimodal learning.
comment : 11 pages, 4 figures
♻ ☆ UniFucGrasp: Human-Hand-Inspired Unified Functional Grasp Annotation Strategy and Dataset for Diverse Dexterous Hands
Haoran Lin, Wenrui Chen, Xianchi Chen, Fan Yang, Qiang Diao, Wenxin Xie, Sijie Wu, Kailun Yang, Maojun Li, Yaonan Wang
Dexterous grasp datasets are vital for embodied intelligence, but mostly emphasize grasp stability, ignoring functional grasps needed for tasks like opening bottle caps or holding cup handles. Most rely on bulky, costly, and hard-to-control high-DOF Shadow Hands. Inspired by the human hand's underactuated mechanism, we establish UniFucGrasp, a universal functional grasp annotation strategy and dataset for multiple dexterous hand types. Based on biomimicry, it maps natural human motions to diverse hand structures and uses geometry-based force closure to ensure functional, stable, human-like grasps. This method supports low-cost, efficient collection of diverse, high-quality functional grasps. Finally, we establish the first multi-hand functional grasp dataset and provide a synthesis model to validate its effectiveness. Experiments on the UFG dataset, IsaacSim, and complex robotic tasks show that our method improves functional manipulation accuracy and grasp stability, demonstrates improved adaptability across multiple robotic hands, helping to alleviate annotation cost and generalization challenges in dexterous grasping. The project page is at https://haochen611.github.io/UFG.
comment : Accepted to IEEE Robotics and Automation Letters (RA-L). The project page is at https://haochen611.github.io/UFG
♻ ☆ HybridWo rl dSim: A Scalable and Controllable High-fidelity Simulator for Autonomous Driving
Qiang Li, Yingwenqi Jiang, Tuoxi Li, Duyu Chen, Xiang Feng, Yucheng Ao, Shangyue Liu, Xingchen Yu, Youcheng Cai, Yumeng Liu, Yuexin Ma, Xin Hu, Li Liu, Yu Zhang, Linkun Xu, Bingtao Gao, Xueyuan Wang, Shuchang Zhou, Xianming Liu, Ligang Liu
Realistic and controllable simulation is critical for advancing end-to-end autonomous driving, yet existing approaches often struggle to support novel view synthesis under large viewpoint changes or to ensure geometric consistency. We introduce HybridWorldSim, a hybrid simulation framework that integrates multi-traversal neural reconstruction for static backgrounds with generative modeling for dynamic agents. This unified design addresses key limitations of previous methods, enabling the creation of diverse and high-fidelity driving scenarios with reliable visual and spatial consistency. To facilitate robust benchmarking, we further release a new multi-traversal dataset MIRROR that captures a wide range of routes and environmental conditions across different cities. Extensive experiments demonstrate that HybridWorldSim surpasses previous state-of-the-art methods, providing a practical and scalable solution for high-fidelity simulation and a valuable resource for research and development in autonomous driving.
♻ ☆ Fine-grained Image Retrieval via Dual- Vision Adaptation AAAI2026
Xin Jiang, Meiqi Cao, Hao Tang, Fei Shen, Zechao Li
Fine-Grained Image Retrieval~(FGIR) faces challenges in learning discriminative visual representations to retrieve images with similar fine-grained features. Current leading FGIR solutions typically follow two regimes: enforce pairwise similarity constraints in the semantic embedding space, or incorporate a localization sub-network to fine-tune the entire model. However, such two regimes tend to overfit the training data while forgetting the knowledge gained from large-scale pre-training, thus reducing their generalization ability. In this paper, we propose a Dual-Vision Adaptation (DVA) approach for FGIR, which guides the frozen pre-trained model to perform FGIR through collaborative sample and feature adaptation. Specifically, we design Object-Perceptual Adaptation, which modifies input samples to help the pre-trained model perceive critical objects and elements within objects that are helpful for category prediction. Meanwhile, we propose In-Context Adaptation, which introduces a small set of parameters for feature adaptation without modifying the pre-trained parameters. This makes the FGIR task using these adjusted features closer to the task solved during the pre-training. Additionally, to balance retrieval efficiency and performance, we propose Discrimination Perception Transfer to transfer the discriminative knowledge in the object-perceptual adaptation to the image encoder using the knowledge distillation mechanism. Extensive experiments show that DVA has fewer learnable parameters and performs well on three in-distribution and three out-of-distribution fine-grained datasets.
comment : Accepted by AAAI2026
♻ ☆ IMSE: Efficient U-Net-based Speech Enhancement using Inception Depthwise Convolution and Amplitude-Aware Linear Attention
Xinxin Tang, Bin Qin, Yufang Li
Achieving a balance between lightweight design and high performance remains a significant challenge for speech enhancement (SE) tasks on resource-constrained devices. Existing state-of-the-art methods, such as MUSE, have established a strong baseline with only 0.51M parameters by introducing a Multi-path Enhanced Taylor (MET) transformer and Deformable Embedding (DE). However, an in-depth analysis reveals that MUSE still suffers from efficiency bottlenecks: the MET module relies on a complex "approximate-compensate" mechanism to mitigate the limitations of Taylor-expansion-based attention, while the offset calculation for deformable embedding introduces additional computational burden. This paper proposes IMSE, a systematically optimized and ultra-lightweight network. We introduce two core innovations: 1) Replacing the MET module with Amplitude-Aware Linear Attention (MALA). MALA fundamentally rectifies the "amplitude-ignoring" problem in linear attention by explicitly preserving the norm information of query vectors in the attention calculation, achieving efficient global modeling without an auxiliary compensation branch. 2) Replacing the DE module with Inception Depthwise Convolution (IDConv). IDConv borrows the Inception concept, decomposing large-kernel operations into efficient parallel branches (square, horizontal, and vertical strips), thereby capturing spectrogram features with extremely low parameter redundancy. Extensive experiments on the VoiceBank+DEMAND dataset demonstrate that, compared to the MUSE baseline, IMSE significantly reduces the parameter count by 16.8\% (from 0.513M to 0.427M) while achieving competitive performance comparable to the state-of-the-art on the PESQ metric (3.373). This study sets a new benchmark for the trade-off between model size and speech quality in ultra-lightweight speech enhancement.
♻ ☆ Hi-EF: Benchmark ing Emotion Forecasting in Human-interaction
Haoran Wang, Xinji Mai, Zeng Tao, Junxiong Lin, Xuan Tong, Ivy Pan, Shaoqi Yan, Yan Wang, Shuyong Gao
Affective Forecasting is an psychology task that involves predicting an individual's future emotional responses, often hampered by reliance on external factors leading to inaccuracies, and typically remains at a qualitative analysis stage. To address these challenges, we narrows the scope of Affective Forecasting by introducing the concept of Human-interaction-based Emotion Forecasting (EF). This task is set within the context of a two-party interaction, positing that an individual's emotions are significantly influenced by their interaction partner's emotional expressions and informational cues. This dynamic provides a structured perspective for exploring the patterns of emotional change, thereby enhancing the feasibility of emotion forecasting.
♻ ☆ 3D Motion Perception of Binocular Vision Target with PID-CNN
Jiazhao Shi, Pan Pan, Haotian Shi
This article trained a network for perceiving three-dimensional motion information of binocular vision target, which can provide real-time three-dimensional coordinate, velocity, and acceleration, and has a basic spatiotemporal perception capability. Understood the ability of neural networks to fit nonlinear problems from the perspective of PID. Considered a single-layer neural network as using a second-order difference equation and a nonlinearity to describe a local problem. Multilayer networks gradually transform the raw representation to the desired representation through multiple such combinations. Analysed some reference principles for designing neural networks. Designed a relatively small PID convolutional neural network, with a total of 17 layers and 413 thousand parameters. Implemented a simple but practical feature reuse method by concatenation and pooling. The network was trained and tested using the simulated randomly moving ball datasets, and the experimental results showed that the prediction accuracy was close to the upper limit that the input image resolution can represent. Analysed the experimental results and errors, as well as the existing shortcomings and possible directions for improvement. Finally, discussed the advantages of high-dimensional convolution in improving computational efficiency and feature space utilization. As well as the potential advantages of using PID information to implement memory and attention mechanisms.
comment : 7 pages, 9 figures, 2 tables. The codes of this article have been released at: https://github.com/ShiJZ123/PID-CNN
♻ ☆ PowerCLIP: Powerset Alignment for Contrastive Pre-Train ing
Masaki Kawamura, Nakamasa Inoue, Rintaro Yanagi, Hirokatsu Kataoka, Rio Yokota
Contrastive vision-language pre-training frameworks such as CLIP have demonstrated impressive zero-shot performance across a range of vision-language tasks. Recent studies have shown that aligning individual text tokens with specific image patches or regions enhances fine-grained compositional understanding. However, it remains challenging to capture compositional semantics that span multiple image regions. To address this limitation, we propose PowerCLIP, a novel contrastive pre-training framework enhanced by powerset alignment, which exhaustively optimizes region-to-phrase alignments by minimizing the loss defined between powersets of image regions and textual parse trees. Since the naive powerset construction incurs exponential computational cost due to the combinatorial explosion in the number of region subsets, we introduce efficient non-linear aggregators (NLAs) that reduce complexity from O(2^M) to O(M) with respect to the number of regions M, while approximating the exact loss value with arbitrary precision. Our extensive experiments demonstrate that PowerCLIP outperforms state-of-the-art methods in zero-shot classification and retrieval tasks, underscoring the compositionality and robustness of our approach. Our code will be made publicly available.
♻ ☆ Off the Planckian Locus: Using 2D Chromaticity to Improve In-Camera Color
SaiKiran Tedla, Joshua E. Little, Hakki Can Karaimer, Michael S. Brown
Traditional in-camera colorimetric mapping relies on correlated color temperature (CCT)-based interpolation between pre-calibrated transforms optimized for Planckian illuminants such as CIE A and D65. However, modern lighting technologies such as LEDs can deviate substantially from the Planckian locus, exposing the limitations of relying on conventional one-dimensional CCT for illumination characterization. This paper demonstrates that transitioning from 1D CCT (on the Planckian locus) to a 2D chromaticity space (off the Planckian locus) improves colorimetric accuracy across various mapping approaches. In addition, we replace conventional CCT interpolation with a lightweight multi-layer perceptron (MLP) that leverages 2D chromaticity features for robust colorimetric mapping under non-Planckian illuminants. A lightbox-based calibration procedure incorporating representative LED sources is used to train our MLP. Validated across diverse LED lighting, our method reduces angular reproduction error by 22% on average in LED-lit scenes, maintains backward compatibility with traditional illuminants, accommodates multi-illuminant scenes, and supports real-time in-camera deployment with negligible additional computational cost.
comment : Project page: https://cst-mlp.github.io
♻ ☆ AVFakeBench: A Comprehensive Audio-Video Forgery Detection Benchmark for AV-LMMs
Shuhan Xia, Peipei Li, Xuannan Liu, Dongsen Zhang, Xinyu Guo, Zekun Li
The threat of Audio-Video (AV) forgery is rapidly evolving beyond human-centric deepfakes to include more diverse manipulations across complex natural scenes. However, existing benchmarks are still confined to DeepFake-based forgeries and single-granularity annotations, thus failing to capture the diversity and complexity of real-world forgery scenarios. To address this, we introduce AVFakeBench, the first comprehensive audio-video forgery detection benchmark that spans rich forgery semantics across both human subject and general subject. AVFakeBench comprises 12K carefully curated audio-video questions, covering seven forgery types and four levels of annotations. To ensure high-quality and diverse forgeries, we propose a multi-stage hybrid forgery framework that integrates proprietary models for task planning with expert generative models for precise manipulation. The benchmark establishes a multi-task evaluation framework covering binary judgment, forgery types classification, forgery detail selection, and explanatory reasoning. We evaluate 11 Audio-Video Large Language Models (AV-LMMs) and 2 prevalent detection methods on AVFakeBench, demonstrating the potential of AV-LMMs as emerging forgery detectors while revealing their notable weaknesses in fine-grained perception and reasoning.
comment : The experimental results in this paper have been further improved and updated; the baseline results do not match existing results, therefore the paper needs to be retracted
♻ ☆ 3-Tracer: A Tri-level Temporal-Aware Framework for Audio Forgery Detection and Localization
Shuhan Xia, Xuannan Liu, Xing Cui, Peipei Li
Recently, partial audio forgery has emerged as a new form of audio manipulation. Attackers selectively modify partial but semantically critical frames while preserving the overall perceptual authenticity, making such forgeries particularly difficult to detect. Existing methods focus on independently detecting whether a single frame is forged, lacking the hierarchical structure to capture both transient and sustained anomalies across different temporal levels. To address these limitations, We identify three key levels relevant to partial audio forgery detection and present T3-Tracer, the first framework that jointly analyzes audio at the frame, segment, and audio levels to comprehensively detect forgery traces. T3-Tracer consists of two complementary core modules: the Frame-Audio Feature Aggregation Module (FA-FAM) and the Segment-level Multi-Scale Discrepancy-Aware Module (SMDAM). FA-FAM is designed to detect the authenticity of each audio frame. It combines both frame-level and audio-level temporal information to detect intra-frame forgery cues and global semantic inconsistencies. To further refine and correct frame detection, we introduce SMDAM to detect forgery boundaries at the segment level. It adopts a dual-branch architecture that jointly models frame features and inter-frame differences across multi-scale temporal windows, effectively identifying abrupt anomalies that appeared on the forged boundaries. Extensive experiments conducted on three challenging datasets demonstrate that our approach achieves state-of-the-art performance.
comment : The experimental results in this paper have been further improved and updated; the baseline results do not match existing results, therefore the paper needs to be retracted
♻ ☆ Can Wo rl d Simulators Reason? Gen-ViRe: A Generative Visual Reasoning Benchmark
Xinxin Liu, Zhaopan Xu, Ming Li, Kai Wang, Yong Jae Lee, Yuzhang Shang
While Chain-of-Thought (CoT) prompting enables sophisticated symbolic reasoning in LLMs, it remains confined to discrete text and cannot simulate the continuous, physics-governed dynamics of the real world. Recent video generation models have emerged as potential world simulators through Chain-of-Frames (CoF) reasoning -- materializing thought as frame-by-frame visual sequences, with each frame representing a physically-grounded reasoning step. Despite compelling demonstrations, a challenge persists: existing benchmarks, focusing on fidelity or alignment, do not assess CoF reasoning and thus cannot measure core cognitive abilities in multi-step planning, algorithmic logic, or abstract pattern extrapolation. This evaluation void prevents systematic understanding of model capabilities and principled guidance for improvement. We introduce Gen-ViRe (Generative Visual Reasoning Benchmark), a framework grounded in cognitive science and real-world AI applications, which decomposes CoF reasoning into six cognitive dimensions -- from perceptual logic to abstract planning -- and 24 subtasks. Through multi-source data curation, minimal prompting protocols, and hybrid VLM-assisted evaluation with detailed criteria, Gen-ViRe delivers the first quantitative assessment of video models as reasoners. Our experiments on SOTA systems reveal substantial discrepancies between impressive visual quality and actual reasoning depth, establishing baselines and diagnostic tools to advance genuine world simulators.
comment : 10 pages
♻ ☆ Dynamic Attention Analysis for Backdoor Detection in Text-to-Image Diffusion Models
Zhongqi Wang, Jie Zhang, Shiguang Shan, Xilin Chen
Recent studies have revealed that text-to-image diffusion models are vulnerable to backdoor attacks, where attackers implant stealthy textual triggers to manipulate model outputs. Previous backdoor detection methods primarily focus on the static features of backdoor samples. However, a vital property of diffusion models is their inherent dynamism. This study introduces a novel backdoor detection perspective named Dynamic Attention Analysis (DAA), showing that these dynamic characteristics serve as better indicators for backdoor detection. Specifically, by examining the dynamic evolution of cross-attention maps, we observe that backdoor samples exhibit distinct feature evolution patterns at the $<$EOS$>$ token compared to benign samples. To quantify these dynamic anomalies, we first introduce DAA-I, which treats the tokens' attention maps as spatially independent and measures dynamic feature using the Frobenius norm. Furthermore, to better capture the interactions between attention maps and refine the feature, we propose a dynamical system-based approach, referred to as DAA-S. This model formulates the spatial correlations among attention maps using a graph-based state equation and we theoretically analyze the global asymptotic stability of this method. Extensive experiments across six representative backdoor attack scenarios demonstrate that our approach significantly surpasses existing detection methods, achieving an average F1 Score of 79.27% and an AUC of 86.27%. The code is available at https://github.com/Robin-WZQ/DAA.
comment : Accepted by TPAMI
♻ ☆ FreeSwim: Revisiting Sliding-Window Attention Mechanisms for Training-Free Ultra-High-Resolution Video Generation
Yunfeng Wu, Jiayi Song, Zhenxiong Tan, Zihao He, Songhua Liu
The quadratic time and memory complexity of the attention mechanism in modern Transformer based video generators makes end-to-end training for ultra high resolution videos prohibitively expensive. Motivated by this limitation, we introduce a training-free approach that leverages video Diffusion Transformers pretrained at their native scale to synthesize higher resolution videos without any additional training or adaptation. At the core of our method lies an inward sliding window attention mechanism, which originates from a key observation: maintaining each query token's training scale receptive field is crucial for preserving visual fidelity and detail. However, naive local window attention, unfortunately, often leads to repetitive content and exhibits a lack of global coherence in the generated results. To overcome this challenge, we devise a dual-path pipeline that backs up window attention with a novel cross-attention override strategy, enabling the semantic content produced by local attention to be guided by another branch with a full receptive field and, therefore, ensuring holistic consistency. Furthermore, to improve efficiency, we incorporate a cross-attention caching strategy for this branch to avoid the frequent computation of full 3D attention. Extensive experiments demonstrate that our method delivers ultra-high-resolution videos with fine-grained visual details and high efficiency in a training-free paradigm. Meanwhile, it achieves superior performance on VBench, even compared to training-based alternatives, with competitive or improved efficiency. Codes are available at: https://github.com/WillWu111/FreeSwim
comment : 23 pages, 14 figures
♻ ☆ CountSteer: Steering Attention for Object Counting in Diffusion Models AAAI 2026
Hyemin Boo, Hyoryung Kim, Myungjin Lee, Seunghyeon Lee, Jiyoung Lee, Jang-Hwan Choi, Hyunsoo Cho
Text-to-image diffusion models generate realistic and coherent images but often fail to follow numerical instructions in text, revealing a gap between language and visual representation. Interestingly, we found that these models are not entirely blind to numbers-they are implicitly aware of their own counting accuracy, as their internal signals shift in consistent ways depending on whether the output meets the specified count. This observation suggests that the model already encodes a latent notion of numerical correctness, which can be harnessed to guide generation more precisely. Building on this intuition, we introduce CountSteer, a training-free method that improves generation of specified object counts by steering the model's cross-attention hidden states during inference. In our experiments, CountSteer improved object-count accuracy by about 4% without compromising visual quality, demonstrating a simple yet effective step toward more controllable and semantically reliable text-to-image generation.
comment : Accepted to AAAI 2026 Workshop on Shaping Responsible Synthetic Data in the Era of Foundation Models (RSD)
♻ ☆ Learning by Aligning 2D Skeleton Sequences and Multi-Modality Fusion ECCV 2024
Quoc-Huy Tran, Muhammad Ahmed, Murad Popattia, M. Hassan Ahmed, Andrey Konin, M. Zeeshan Zia
This paper presents a self-supervised temporal video alignment framework which is useful for several fine-grained human activity understanding applications. In contrast with the state-of-the-art method of CASA, where sequences of 3D skeleton coordinates are taken directly as input, our key idea is to use sequences of 2D skeleton heatmaps as input. Unlike CASA which performs self-attention in the temporal domain only, we feed 2D skeleton heatmaps to a video transformer which performs self-attention both in the spatial and temporal domains for extracting effective spatiotemporal and contextual features. In addition, we introduce simple heatmap augmentation techniques based on 2D skeletons for self-supervised learning. Despite the lack of 3D information, our approach achieves not only higher accuracy but also better robustness against missing and noisy keypoints than CASA. Furthermore, extensive evaluations on three public datasets, i.e., Penn Action, IKEA ASM, and H2O, demonstrate that our approach outperforms previous methods in different fine-grained human activity understanding tasks. Finally, fusing 2D skeleton heatmaps with RGB videos yields the state-of-the-art on all metrics and datasets. To our best knowledge, our work is the first to utilize 2D skeleton heatmap inputs and the first to explore multi-modality fusion for temporal video alignment. Our code and dataset are available on our research website: https://retrocausal.ai/research/.
comment : Accepted to ECCV 2024
♻ ☆ HiGFA: Hierarchical Guidance for Fine-grained Data Augmentation with Diffusion Models
Zhiguang Lu, Qianqian Xu, Peisong Wen, Siran Dai, Qingming Huang
Generative diffusion models show promise for data augmentation. However, applying them to fine-grained tasks presents a significant challenge: ensuring synthetic images accurately capture the subtle, category-defining features critical for high fidelity. Standard approaches, such as text-based Classifier-Free Guidance (CFG), often lack the required specificity, potentially generating misleading examples that degrade fine-grained classifier performance. To address this, we propose Hierarchically Guided Fine-grained Augmentation (HiGFA). HiGFA leverages the temporal dynamics of the diffusion sampling process. It employs strong text and transformed contour guidance with fixed strengths in the early-to-mid sampling stages to establish overall scene, style, and structure. In the final sampling stages, HiGFA activates a specialized fine-grained classifier guidance and dynamically modulates the strength of all guidance signals based on prediction confidence. This hierarchical, confidence-driven orchestration enables HiGFA to generate diverse yet faithful synthetic images by intelligently balancing global structure formation with precise detail refinement. Experiments on several FGVC datasets demonstrate the effectiveness of HiGFA.
♻ ☆ Robust Phase-Shifting Profilometry for Arbitrary Motion
Geyou Zhang, Kai Liu, Ao Li, Ce Zhu
Phase-shifting profilometry (PSP) enables high-accuracy 3D reconstruction but remains highly susceptible to object motion. Although numerous studies have explored compensation for motion-induced errors, residual inaccuracies still persist, particularly in complex motion scenarios. In this paper, we propose a robust phase-shifting profilometry for arbitrary motion (RPSP-AM), including six-degrees-of-freedom (6-DoF) motion (translation and rotation in any direction), non-rigid deformations, and multi-target movements, achieving high-fidelity motion-error-free 3D reconstruction. We categorize motion errors into two components: 1) ghosting artifacts induced by image misalignment, and 2) ripple-like distortions induced by phase deviation. To eliminate the ghosting artifacts, we perform pixel-wise image alignment based on dense optical flow tracking. To correct ripple-like distortions, we propose a high-accuracy, low-complexity image-sequential binomial self-compensation (I-BSC) method, which performs a summation of the homogeneous fringe images weighted by binomial coefficients, exponentially reducing the ripple-like distortions with a competitive computational speed compared with the traditional four-step phase-shifting method. Extensive experimental results demonstrate that, under challenging conditions such as 6-DoF motion, non-rigid deformations, and multi-target movements, the proposed RPSP-AM outperforms state-of-the-art (SoTA) methods in compensating for both ghosting artifacts and ripple-like distortions. Our approach extends the applicability of PSP to arbitrary motion scenarios, endowing it with potential for widespread adoption in fields such as robotics, industrial inspection, and medical reconstruction.
♻ ☆ Multivariate Gaussian Representation Learning for Medical Action Evaluation AAAI 2026
Luming Yang, Haoxian Liu, Siqing Li, Alper Yilmaz
Fine-grained action evaluation in medical vision faces unique challenges due to the unavailability of comprehensive datasets, stringent precision requirements, and insufficient spatiotemporal dynamic modeling of very rapid actions. To support development and evaluation, we introduce CPREval-6k, a multi-view, multi-label medical action benchmark containing 6,372 expert-annotated videos with 22 clinical labels. Using this dataset, we present GaussMedAct, a multivariate Gaussian encoding framework, to advance medical motion analysis through adaptive spatiotemporal representation learning. Multivariate Gaussian Representation projects the joint motions to a temporally scaled multi-dimensional space, and decomposes actions into adaptive 3D Gaussians that serve as tokens. These tokens preserve motion semantics through anisotropic covariance modeling while maintaining robustness to spatiotemporal noise. Hybrid Spatial Encoding, employing a Cartesian and Vector dual-stream strategy, effectively utilizes skeletal information in the form of joint and bone features. The proposed method achieves 92.1% Top-1 accuracy with real-time inference on the benchmark, outperforming baseline by +5.9% accuracy with only 10% FLOPs. Cross-dataset experiments confirm the superiority of our method in robustness.
comment : Accepted to AAAI 2026
♻ ☆ Efficient Generative Adversarial Networks for Color Document Image Enhancement and Binarization Using Multi-scale Feature Extraction SC 2025
Rui-Yang Ju, KokSheik Wong, Jen-Shiun Chiang
The outcome of text recognition for degraded color documents is often unsatisfactory due to interference from various contaminants. To extract information more efficiently for text recognition, document image enhancement and binarization are often employed as preliminary steps in document analysis. Training independent generative adversarial networks (GANs) for each color channel can generate images where shadows and noise are effectively removed, which subsequently allows for efficient text information extraction. However, employing multiple GANs for different color channels requires long training and inference times. To reduce both the training and inference times of these preliminary steps, we propose an efficient method based on multi-scale feature extraction, which incorporates Haar wavelet transformation and normalization to process document images before submitting them to GANs for training. Experiment results show that our proposed method significantly reduces both the training and inference times while maintaining comparable performances when benchmarked against the state-of-the-art methods. In the best case scenario, a reduction of 10% and 26% are observed for training and inference times, respectively, while maintaining the model performance at 73.79 of Average-Score metric. The implementation of this work is available at https://github.com/RuiyangJu/Efficient_Document_Image_Binarization.
comment : Accepted to APSIPA ASC 2025
♻ ☆ GFT: Graph Feature Tuning for Efficient Point Cloud Analysis WACV 2026
Manish Dhakal, Venkat R. Dasari, Rajshekhar Sunderraman, Yi Ding
Parameter-efficient fine-tuning (PEFT) significantly reduces computational and memory costs by updating only a small subset of the model's parameters, enabling faster adaptation to new tasks with minimal loss in performance. Previous studies have introduced PEFTs tailored for point cloud data, as general approaches are suboptimal. To further reduce the number of trainable parameters, we propose a point-cloud-specific PEFT, termed Graph Features Tuning (GFT), which learns a dynamic graph from initial tokenized inputs of the transformer using a lightweight graph convolution network and passes these graph features to deeper layers via skip connections and efficient cross-attention modules. Extensive experiments on object classification and segmentation tasks show that GFT operates in the same domain, rivalling existing methods, while reducing the trainable parameters. Code is available at https://github.com/manishdhakal/GFT.
comment : Accepted to WACV 2026
♻ ☆ One-to-All Animation: Alignment-Free Character Animation and Image Pose Transfer
Shijun Shi, Jing Xu, Zhihang Li, Chunli Peng, Xiaoda Yang, Lijing Lu, Kai Hu, Jiangning Zhang
Recent advances in diffusion models have greatly improved pose-driven character animation. However, existing methods are limited to spatially aligned reference-pose pairs with matched skeletal structures. Handling reference-pose misalignment remains unsolved. To address this, we present One-to-All Animation, a unified framework for high-fidelity character animation and image pose transfer for references with arbitrary layouts. First, to handle spatially misaligned reference, we reformulate training as a self-supervised outpainting task that transforms diverse-layout reference into a unified occluded-input format. Second, to process partially visible reference, we design a reference extractor for comprehensive identity feature extraction. Further, we integrate hybrid reference fusion attention to handle varying resolutions and dynamic sequence lengths. Finally, from the perspective of generation quality, we introduce identity-robust pose control that decouples appearance from skeletal structure to mitigate pose overfitting, and a token replace strategy for coherent long-video generation. Extensive experiments show that our method outperforms existing approaches. The code and model are available at https://github.com/ssj9596/One-to-All-Animation.
comment : Project Page:https://ssj9596.github.io/one-to-all-animation-project/
♻ ☆ DMC$^3$: Dual-Modal Counterfactual Contrastive Construction for Egocentric Video Question Answering
Jiayi Zou, Chaofan Chen, Bing-Kun Bao, Changsheng Xu
Egocentric Video Question Answering (Egocentric VideoQA) plays an important role in egocentric video understanding, which refers to answering questions based on first-person videos. Although existing methods have made progress through the paradigm of pre-training and fine-tuning, they ignore the unique challenges posed by the first-person perspective, such as understanding multiple events and recognizing hand-object interactions. To deal with these challenges, we propose a Dual-Modal Counterfactual Contrastive Construction (DMC$^3$) framework, which contains an egocentric videoqa baseline, a counterfactual sample construction module and a counterfactual sample-involved contrastive optimization. Specifically, We first develop a counterfactual sample construction module to generate positive and negative samples for textual and visual modalities through event description paraphrasing and core interaction mining, respectively. Then, We feed these samples together with the original samples into the baseline. Finally, in the counterfactual sample-involved contrastive optimization module, we apply contrastive loss to minimize the distance between the original sample features and the positive sample features, while maximizing the distance from the negative samples. Experiments show that our method achieve 52.51\% and 46.04\% on the \textit{normal} and \textit{indirect} splits of EgoTaskQA, and 13.2\% on QAEGO4D, both reaching the state-of-the-art performance.
♻ ☆ Beyond Randomness: Understand the Order of the Noise in Diffusion
Song Yan, Min Li, Bi Xinliang, Jian Yang, Yusen Zhang, Guanye Xiong, Yunwei Lan, Tao Zhang, Wei Zhai, Zheng-Jun Zha
In text-driven content generation (T2C) diffusion model, semantic of generated content is mostly attributed to the process of text embedding and attention mechanism interaction. The initial noise of the generation process is typically characterized as a random element that contributes to the diversity of the generated content. Contrary to this view, this paper reveals that beneath the random surface of noise lies strong analyzable patterns. Specifically, this paper first conducts a comprehensive analysis of the impact of random noise on the model's generation. We found that noise not only contains rich semantic information, but also allows for the erasure of unwanted semantics from it in an extremely simple way based on information theory, and using the equivalence between the generation process of diffusion model and semantic injection to inject semantics into the cleaned noise. Then, we mathematically decipher these observations and propose a simple but efficient training-free and universal two-step "Semantic Erasure-Injection" process to modulate the initial noise in T2C diffusion model. Experimental results demonstrate that our method is consistently effective across various T2C models based on both DiT and UNet architectures and presents a novel perspective for optimizing the generation of diffusion model, providing a universal tool for consistent generation.
♻ ☆ TPCNet: Triple physical constraints for Low-light Image Enhancement
Jing-Yi Shi, Ming-Fei Li, Ling-An Wu
Low-light image enhancement is an essential computer vision task to improve image contrast and to decrease the effects of color bias and noise. Many existing interpretable deep-learning algorithms exploit the Retinex theory as the basis of model design. However, previous Retinex-based algorithms, that consider reflected objects as ideal Lambertian ignore specular reflection in the modeling process and construct the physical constraints in image space, limiting generalization of the model. To address this issue, we preserve the specular reflection coefficient and reformulate the original physical constraints in the imaging process based on the Kubelka-Munk theory, thereby constructing constraint relationship between illumination, reflection, and detection, the so-called triple physical constraints (TPCs)theory. Based on this theory, the physical constraints are constructed in the feature space of the model to obtain the TPC network (TPCNet). Comprehensive quantitative and qualitative benchmark and ablation experiments confirm that these constraints effectively improve the performance metrics and visual quality without introducing new parameters, and demonstrate that our TPCNet outperforms other state-of-the-art methods on 10 datasets.
♻ ☆ SuperMat: Physically Consistent PBR Material Estimation at Interactive Rates
Yijia Hong, Yuan-Chen Guo, Ran Yi, Yulong Chen, Yan-Pei Cao, Lizhuang Ma
Decomposing physically-based materials from images into their constituent properties remains challenging, particularly when maintaining both computational efficiency and physical consistency. While recent diffusion-based approaches have shown promise, they face substantial computational overhead due to multiple denoising steps and separate models for different material properties. We present SuperMat, a single-step framework that achieves high-quality material decomposition with one-step inference. This enables end-to-end training with perceptual and re-render losses while decomposing albedo, metallic, and roughness maps at millisecond-scale speeds. We further extend our framework to 3D objects through a UV refinement network, enabling consistent material estimation across viewpoints while maintaining efficiency. Experiments demonstrate that SuperMat achieves state-of-the-art PBR material decomposition quality while reducing inference time from seconds to milliseconds per image, and completes PBR material estimation for 3D objects in approximately 3 seconds. The project page is at https://hyj542682306.github.io/SuperMat/.
comment : https://hyj542682306.github.io/SuperMat/
♻ ☆ PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection
Yusu Qian, Cheng Wan, Chao Jia, Yinfei Yang, Qingyu Zhao, Zhe Gan
Multimodal large language models (MLLMs) have achieved remarkable progress on vision-language tasks, yet their reasoning processes remain sometimes unreliable. We introduce PRISM-Bench, a benchmark of puzzle-based visual challenges designed to evaluate not only whether models can solve problems, but how their reasoning unfolds. Unlike prior evaluations that measure only final-answer accuracy, PRISM-Bench introduces a diagnostic task: given a visual puzzle and a step-by-step chain-of-thought (CoT) containing exactly one error, models must identify the first incorrect step. This setting enables fine-grained assessment of logical consistency, error detection, and visual reasoning. The puzzles in PRISM-Bench require multi-step symbolic, geometric, and analogical reasoning, resisting shortcuts based on superficial pattern matching. Evaluations across state-of-the-art MLLMs reveal a persistent gap between fluent generation and faithful reasoning: models that produce plausible CoTs often fail to locate simple logical faults. By disentangling answer generation from reasoning verification, PRISM-Bench offers a sharper lens on multimodal reasoning competence and underscores the need for diagnostic evaluation protocols in the development of trustworthy MLLMs.
comment : This paper's first error detection task's ground truth data contains hallucination introduced by gpt and needs to be withdrawn
♻ ☆ VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering
Yuyi Li, Daoyuan Chen, Zhen Wang, Yutong Lu, Yaliang Li
Large Vision-Language Models (LVLMs) show promise for scientific applications, yet open-source models still struggle with Scientific Visual Question Answering (SVQA), namely answering questions about figures from scientific papers. A key bottleneck lies in the lack of public, large-scale, high-quality SVQA datasets. Although recent work uses LVLMs to synthesize data at scale, we identify systematic errors in their resulting QA pairs, stemming from LVLMs' inherent limitations and information asymmetry between figures and text. To address these challenges, we propose a verification-centric Generate-then-Verify framework that first generates QA pairs with figure-associated textual context, then applies cross-modal consistency checks against figures along with auxiliary filters to eliminate erroneous pairs. We instantiate this framework to curate VeriSciQA, a dataset of 20,351 QA pairs spanning 20 scientific domains and 12 figure types. VeriSciQA poses a challenging benchmark for open-source models, with a substantial accuracy gap between the leading open-source models (64%) and a proprietary model (82%). Moreover, models fine-tuned on VeriSciQA achieve consistent improvements on SVQA benchmarks, with performance gains that scale with data size and surpass models trained on existing datasets. Human evaluation further validates the superior correctness of VeriSciQA. Together, these evidences demonstrate that continued data expansion by our scalable framework can further advance SVQA capability in the open-source community.
♻ ☆ Generating Fit Check Videos with a Handheld Camera
Bowei Chen, Brian Curless, Ira Kemelmacher-Shlizerman, Steven M. Seitz
Self-captured full-body videos are popular, but most deployments require mounted cameras, carefully-framed shots, and repeated practice. We propose a more convenient solution that enables full-body video capture using handheld mobile devices. Our approach takes as input two static photos (front and back) of you in a mirror, along with an IMU motion reference that you perform while holding your mobile phone, and synthesizes a realistic video of you performing a similar target motion. We enable rendering into a new scene, with consistent illumination and shadows. We propose a novel video diffusion-based model to achieve this. Specifically, we propose a parameter-free frame generation strategy and a multi-reference attention mechanism to effectively integrate appearance information from both the front and back selfies into the video diffusion model. Further, we introduce an image-based fine-tuning strategy to enhance frame sharpness and improve shadows and reflections generation for more realistic human-scene composition.
♻ ☆ Face-MakeUpV2: Facial Consistency Learning for Controllable Text-to-Image Generation
Dawei Dai, Yinxiu Zhou, Chenghang Li, Guolai Jiang, Chengfang Zhang
In facial image generation, current text-to-image models often suffer from facial attribute leakage and insufficient physical consistency when responding to local semantic instructions. In this study, we propose Face-MakeUpV2, a facial image generation model that aims to maintain the consistency of face ID and physical characteristics with the reference image. First, we constructed a large-scale dataset FaceCaptionMask-1M comprising approximately one million image-text-masks pairs that provide precise spatial supervision for the local semantic instructions. Second, we employed a general text-to-image pretrained model as the backbone and introduced two complementary facial information injection channels: a 3D facial rendering channel to incorporate the physical characteristics of the image and a global facial feature channel. Third, we formulated two optimization objectives for the supervised learning of our model: semantic alignment in the model's embedding space to mitigate the attribute leakage problem and perceptual loss on facial images to preserve ID consistency. Extensive experiments demonstrated that our Face-MakeUpV2 achieves best overall performance in terms of preserving face ID and maintaining physical consistency of the reference images. These results highlight the practical potential of Face-MakeUpV2 for reliable and controllable facial editing in diverse applications.
comment : Some errors in the critical data presented in Table 1 and Table 2
♻ ☆ Full-scale Representation Guided Network for Retinal Vessel Segmentation
Sunyong Seo, Sangwook Yoo, Huisu Yoon
The U-Net architecture and its variants have remained state-of-the-art (SOTA) for retinal vessel segmentation over the past decade. In this study, we introduce a Full-Scale Guided Network (FSG-Net), where a novel feature representation module using modernized convolution blocks effectively captures full-scale structural information, while a guided convolution block subsequently refines this information. Specifically, we introduce an attention-guided filter within the guided convolution block, leveraging its similarity to unsharp masking to enhance fine vascular structures. Passing full-scale information to the attention block facilitates the generation of more contextually relevant attention maps, which are then passed to the attention-guided filter, providing further refinement to the segmentation performance. The structure preceding the guided convolution block can be replaced by any U-Net variant, ensuring flexibility and scalability across various segmentation tasks. For a fair comparison, we re-implemented recent studies available in public repositories to evaluate their scalability and reproducibility. Our experiments demonstrate that, despite its compact architecture, FSG-Net delivers performance competitive with SOTA methods across multiple public datasets. Ablation studies further demonstrate that each proposed component meaningfully contributes to this competitive performance. Our code is available on https://github.com/ZombaSY/FSG-Net-pytorch.
comment : 12 pages, 7 figures
♻ ☆ AutoDrive-R$^2$: Incentivizing Reasoning and Self-Reflection Capacity for VLA Model in Autonomous Driving
Zhenlong Yuan, Chengxuan Qian, Jing Tang, Rui Chen, Zijian Song, Lei Sun, Xiangxiang Chu, Yujun Cai, Dapeng Zhang, Shuo Li
Vision-Language-Action (VLA) models in autonomous driving systems have recently demonstrated transformative potential by integrating multimodal perception with decision-making capabilities. However, the interpretability and coherence of the decision process and the plausibility of action sequences remain largely underexplored. To address these issues, we propose AutoDrive-R$^2$, a novel VLA framework that enhances both reasoning and self-reflection capabilities of autonomous driving systems through chain-of-thought (CoT) processing and reinforcement learning (RL). Specifically, we first propose an innovative CoT dataset named nuScenesR$^2$-6K for supervised fine-tuning, which effectively builds cognitive bridges between input information and output trajectories through a four-step logical chain with self-reflection for validation. Moreover, to maximize both reasoning and self-reflection during the RL stage, we further employ the Group Relative Policy Optimization (GRPO) algorithm within a physics-grounded reward framework that incorporates spatial alignment, vehicle dynamic, and temporal smoothness criteria to ensure reliable and realistic trajectory planning. Extensive evaluation results across both nuScenes and Waymo datasets demonstrates the state-of-the-art performance and robust generalization capacity of our proposed method.
♻ ☆ Physics-Informed Image Restoration via Progressive PDE Integration
Shamika Likhite, Santiago López-Tapia, Aggelos K. Katsaggelos
Motion blur, caused by relative movement between camera and scene during exposure, significantly degrades image quality and impairs downstream computer vision tasks such as object detection, tracking, and recognition in dynamic environments. While deep learning-based motion deblurring methods have achieved remarkable progress, existing approaches face fundamental challenges in capturing the long-range spatial dependencies inherent in motion blur patterns. Traditional convolutional methods rely on limited receptive fields and require extremely deep networks to model global spatial relationships. These limitations motivate the need for alternative approaches that incorporate physical priors to guide feature evolution during restoration. In this paper, we propose a progressive training framework that integrates physics-informed PDE dynamics into state-of-the-art restoration architectures. By leveraging advection-diffusion equations to model feature evolution, our approach naturally captures the directional flow characteristics of motion blur while enabling principled global spatial modeling. Our PDE-enhanced deblurring models achieve superior restoration quality with minimal overhead, adding only approximately 1\% to inference GMACs while providing consistent improvements in perceptual quality across multiple state-of-the-art architectures. Comprehensive experiments on standard motion deblurring benchmarks demonstrate that our physics-informed approach improves PSNR and SSIM significantly across four diverse architectures, including FFTformer, NAFNet, Restormer, and Stripformer. These results validate that incorporating mathematical physics principles through PDE-based global layers can enhance deep learning-based image restoration, establishing a promising direction for physics-informed neural network design in computer vision applications.
♻ ☆ Emergent Extreme-View Geometry in 3D Foundation Models
Yiwen Zhang, Joseph Tung, Ruojin Cai, David Fouhey, Hadar Averbuch-Elor
3D foundation models (3DFMs) have recently transformed 3D vision, enabling joint prediction of depths, poses, and point maps directly from images. Yet their ability to reason under extreme, non-overlapping views remains largely unexplored. In this work, we study their internal representations and find that 3DFMs exhibit an emergent understanding of extreme-view geometry, despite never being trained for such conditions. To further enhance these capabilities, we introduce a lightweight alignment scheme that refines their internal 3D representation by tuning only a small subset of backbone bias terms, leaving all decoder heads frozen. This targeted adaptation substantially improves relative pose estimation under extreme viewpoints without degrading per-image depth or point quality. Additionally, we contribute MegaUnScene, a new benchmark of Internet scenes unseen by existing 3DFMs, with dedicated test splits for both relative pose estimation and dense 3D reconstruction. All code and data will be released.
comment : Project page is at https://ext-3dfms.github.io/
♻ ☆ Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment WACV 2024
Quoc-Huy Tran, Ahmed Mehmood, Muhammad Ahmed, Muhammad Naufil, Anas Zafar, Andrey Konin, M. Zeeshan Zia
This paper presents an unsupervised transformer-based framework for temporal activity segmentation which leverages not only frame-level cues but also segment-level cues. This is in contrast with previous methods which often rely on frame-level information only. Our approach begins with a frame-level prediction module which estimates framewise action classes via a transformer encoder. The frame-level prediction module is trained in an unsupervised manner via temporal optimal transport. To exploit segment-level information, we utilize a segment-level prediction module and a frame-to-segment alignment module. The former includes a transformer decoder for estimating video transcripts, while the latter matches frame-level features with segment-level features, yielding permutation-aware segmentation results. Moreover, inspired by temporal optimal transport, we introduce simple-yet-effective pseudo labels for unsupervised training of the above modules. Our experiments on four public datasets, i.e., 50 Salads, YouTube Instructions, Breakfast, and Desktop Assembly show that our approach achieves comparable or better performance than previous methods in unsupervised activity segmentation. Our code and dataset are available on our research website: https://retrocausal.ai/research/.
comment : Accepted to WACV 2024
♻ ☆ Random forest-based out-of-distribution detection for robust lung cancer segmentation SP
Aneesh Rangnekar, Harini Veeraraghavan
Accurate detection and segmentation of cancerous lesions from computed tomography (CT) scans is essential for automated treatment planning and cancer treatment response assessment. Transformer-based models with self-supervised pretraining have achieved strong performance on in-distribution (ID) data but often generalize poorly on out-of-distribution (OOD) inputs. We investigate this behavior for lung cancer segmentation using an encoder-decoder model. Our encoder is a Swin Transformer pretrained with masked image modeling (SimMIM) on 10,432 unlabeled 3D CT scans spanning cancerous and non-cancerous conditions, and the decoder was randomly initialized. This model was evaluated on an independent ID test set and four OOD scenarios, including chest CT cohorts (pulmonary embolism and negative COVID-19) and abdomen CT cohorts (kidney cancers and non-cancerous pancreas). OOD detection was performed at the scan level using RF-Deep, a random forest classifier applied to contextual tumor-anchored feature representations. We evaluated 920 3D CTs (172,650 images) and observed that RF-Deep achieved FPR95 values of 18.26% and 27.66% on the chest CT cohorts, and near-perfect detection (less than 0.1% FPR95) on the abdomen CT cohorts, consistently outperforming established OOD methods. These results demonstrate that our RF-Deep classifier provides a simple, lightweight, and effective approach for enhancing the reliability of segmentation models in clinical deployment.
comment : Accepted at SPIE Medical Imaging 2026
♻ ☆ UVGS: Reimagining Unstructured 3D Gaussian Splatting using UV Mapping
Aashish Rai, Dilin Wang, Mihir Jain, Nikolaos Sarafianos, Kefan Chen, Srinath Sridhar, Aayush Prakash
3D Gaussian Splatting (3DGS) has demonstrated superior quality in modeling 3D objects and scenes. However, generating 3DGS remains challenging due to their discrete, unstructured, and permutation-invariant nature. In this work, we present a simple yet effective method to overcome these challenges. We utilize spherical mapping to transform 3DGS into a structured 2D representation, termed UVGS. UVGS can be viewed as multi-channel images, with feature dimensions as a concatenation of Gaussian attributes such as position, scale, color, opacity, and rotation. We further find that these heterogeneous features can be compressed into a lower-dimensional (e.g., 3-channel) shared feature space using a carefully designed multi-branch network. The compressed UVGS can be treated as typical RGB images. Remarkably, we discover that typical VAEs trained with latent diffusion models can directly generalize to this new representation without additional training. Our novel representation makes it effortless to leverage foundational 2D models, such as diffusion models, to directly model 3DGS. Additionally, one can simply increase the 2D UV resolution to accommodate more Gaussians, making UVGS a scalable solution compared to typical 3D backbones. This approach immediately unlocks various novel generation applications of 3DGS by inherently utilizing the already developed superior 2D generation capabilities. In our experiments, we demonstrate various unconditional, conditional generation, and inpainting applications of 3DGS based on diffusion models, which were previously non-trivial.
comment : https://ivl.cs.brown.edu/uvgs
♻ ☆ Self-Calibrating BCIs: Ranking and Recovery of Mental Targets Without Labels
Jonathan Grizou, Carlos de la Torre-Ortiz, Tuukka Ruotsalo
We consider the problem of recovering a mental target (e.g., an image of a face) that a participant has in mind from paired EEG (i.e., brain responses) and image (i.e., perceived faces) data collected during interactive sessions without access to labeled information. The problem has been previously explored with labeled data but not via self-calibration, where labeled data is unavailable. Here, we present the first framework and an algorithm, CURSOR, that learns to recover unknown mental targets without access to labeled data or pre-trained decoders. Our experiments on naturalistic images of faces demonstrate that CURSOR can (1) predict image similarity scores that correlate with human perceptual judgments without any label information, (2) use these scores to rank stimuli against an unknown mental target, and (3) generate new stimuli indistinguishable from the unknown mental target (validated via a user study, N=53).
comment : 10 pages, 4 figures, 11 appendix pages, 7 appendix figures
♻ ☆ Unleashing Hour-Scale Video Training for Long Video-Language Understanding NeurIPS 2025
Jingyang Lin, Jialian Wu, Ximeng Sun, Ze Wang, Jiang Liu, Yusheng Su, Xiaodong Yu, Hao Chen, Jiebo Luo, Zicheng Liu, Emad Barsoum
Recent long-form video-language understanding benchmarks have driven progress in video large multimodal models (Video-LMMs). However, the scarcity of well-annotated long videos has left the training of hour-long Video-LMMs underexplored. To close this gap, we present VideoMarathon, a large-scale hour-long video instruction-following dataset. This dataset includes around 9,700 hours of long videos sourced from diverse domains, ranging from 3 to 60 minutes per video. Specifically, it contains 3.3M high-quality QA pairs, spanning six fundamental topics: temporality, spatiality, object, action, scene, and event. Compared to existing video instruction datasets, VideoMarathon significantly extends training video durations up to 1 hour, and supports 22 diverse tasks requiring both short- and long-term video comprehension. Building on VideoMarathon, we propose Hour-LLaVA, a powerful and efficient Video-LMM for hour-scale video-language modeling. It enables hour-long video training and inference at 1-FPS sampling by leveraging a memory augmentation module, which adaptively integrates question-relevant and spatiotemporally informative semantics from the cached full video context. In our experiments, Hour-LLaVA achieves the best performance on multiple representative long video-language benchmarks, demonstrating the high quality of the VideoMarathon dataset and the superiority of the Hour-LLaVA model.
comment : NeurIPS 2025, Project page: https://videomarathon.github.io/
♻ ☆ Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision -Language Models NeurIPS 2025
Konstantinos M. Dafnis, Dimitris N. Metaxas
Vision-Language Models (VLMs) excel at zero-shot inference but often degrade under test-time domain shifts. For this reason, episodic test-time adaptation strategies have recently emerged as powerful techniques for adapting VLMs to a single unlabeled image. However, existing adaptation strategies, such as test-time prompt tuning, typically require backpropagating through large encoder weights or altering core model components. In this work, we introduce Spectrum-Aware Test-Time Steering (STS), a lightweight adaptation framework that extracts a spectral subspace from the textual embeddings to define principal semantic directions and learns to steer latent representations in a spectrum-aware manner by adapting a small number of per-sample shift parameters to minimize entropy across augmented views. STS operates entirely at inference in the latent space, without backpropagation through or modification of the frozen encoders. Building on standard evaluation protocols, our comprehensive experiments demonstrate that STS largely surpasses or compares favorably against state-of-the-art test-time adaptation methods, while introducing only a handful of additional parameters and achieving inference speeds up to 8x faster with a 12x smaller memory footprint than conventional test-time prompt tuning. The code is available at https://github.com/kdafnis/STS.
comment : NeurIPS 2025
♻ ☆ ROGR: Relightable 3D Objects using Generative Relighting NeurIPS 2025
Jiapeng Tang, Matthew Lavine, Dor Verbin, Stephan J. Garbin, Matthias Nießner, Ricardo Martin Brualla, Pratul P. Srinivasan, Philipp Henzler
We introduce ROGR, a novel approach that reconstructs a relightable 3D model of an object captured from multiple views, driven by a generative relighting model that simulates the effects of placing the object under novel environment illuminations. Our method samples the appearance of the object under multiple lighting environments, creating a dataset that is used to train a lighting-conditioned Neural Radiance Field (NeRF) that outputs the object's appearance under any input environmental lighting. The lighting-conditioned NeRF uses a novel dual-branch architecture to encode the general lighting effects and specularities separately. The optimized lighting-conditioned NeRF enables efficient feed-forward relighting under arbitrary environment maps without requiring per-illumination optimization or light transport simulation. We evaluate our approach on the established TensoIR and Stanford-ORB datasets, where it improves upon the state-of-the-art on most metrics, and showcase our approach on real-world object captures.
comment : NeurIPS 2025 Spotlight. Project page: https://tangjiapeng.github.io/ROGR
♻ ☆ Visible Yet Unreadable: A Systematic Blind Spot of Vision Language Models Across Writing Systems
Jie Zhang, Ting Xu, Gelei Deng, Runyi Hu, Han Qiu, Tianwei Zhang, Qing Guo, Ivor Tsang
Writing is a universal cultural technology that reuses vision for symbolic communication. Humans display striking resilience: we readily recognize words even when characters are fragmented, fused, or partially occluded. This paper investigates whether advanced vision language models (VLMs) share this resilience. We construct two psychophysics inspired benchmarks across distinct writing systems, Chinese logographs and English alphabetic words, by splicing, recombining, and overlaying glyphs to yield ''visible but unreadable'' stimuli for models while remaining legible to humans. Despite strong performance on clean text, contemporary VLMs show a severe drop under these perturbations, frequently producing unrelated or incoherent outputs. The pattern suggests a structural limitation: models heavily leverage generic visual invariances but under rely on compositional priors needed for robust literacy. We release stimuli generation code, prompts, and evaluation protocols to facilitate transparent replication and follow up work. Our findings motivate architectures and training strategies that encode symbol segmentation, composition, and binding across scripts, and they delineate concrete challenges for deploying multimodal systems in education, accessibility, cultural heritage, and security.
comment : arXiv admin note: This article has been withdrawn by arXiv administrators due to violation of arXiv policy regarding generative AI authorship
♻ ☆ ReSpace: Text-Driven 3D Indoor Scene Synthesis and Editing with Preference Alignment
Martin JJ. Bucher, Iro Armeni
Scene synthesis and editing has emerged as a promising direction in computer graphics. Current trained approaches for 3D indoor scenes either oversimplify object semantics through one-hot class encodings (e.g., 'chair' or 'table'), require masked diffusion for editing, ignore room boundaries, or rely on floor plan renderings that fail to capture complex layouts. LLM-based methods enable richer semantics via natural language (e.g., 'modern studio with light wood furniture'), but lack editing functionality, are limited to rectangular layouts, or rely on weak spatial reasoning from implicit world models. We introduce ReSpace, a generative framework for text-driven 3D indoor scene synthesis and editing using autoregressive language models. Our approach features a compact structured scene representation with explicit room boundaries that enables asset-agnostic deployment and frames scene editing as a next-token prediction task. We leverage a dual-stage training approach combining supervised fine-tuning and preference alignment, enabling a specially trained language model for object addition that accounts for user instructions, spatial geometry, object semantics, and scene-level composition. For scene editing, we employ a zero-shot LLM to handle object removal and prompts for addition. We further introduce a voxelization-based evaluation capturing fine-grained geometry beyond 3D bounding boxes. Experimental results surpass state-of-the-art on addition and achieve superior human-perceived quality on full scene synthesis.
comment : 25 pages, 18 figures (incl. appendix)
♻ ☆ Alligat0R: Pre-Train ing Through Co-Visibility Segmentation for Relative Camera Pose Regression NeurIPS 2025
Thibaut Loiseau, Guillaume Bourmaud, Vincent Lepetit
Pre-training techniques have greatly advanced computer vision, with CroCo's cross-view completion approach yielding impressive results in tasks like 3D reconstruction and pose regression. However, cross-view completion is ill-posed in non-covisible regions, limiting its effectiveness. We introduce Alligat0R, a novel pre-training approach that replaces cross-view learning with a covisibility segmentation task. Our method predicts whether each pixel in one image is covisible in the second image, occluded, or outside the field of view, making the pre-training effective in both covisible and non-covisible regions, and provides interpretable predictions. To support this, we present Cub3, a large-scale dataset with 5M image pairs and dense covisibility annotations derived from the nuScenes and ScanNet datasets. Cub3 includes diverse scenarios with varying degrees of overlap. The experiments show that our novel pre-training method Alligat0R significantly outperforms CroCo in relative pose regression. Code is available at https://github.com/thibautloiseau/alligat0r.
comment : NeurIPS 2025 Spotlight
♻ ☆ Fast 3D Surrogate Modeling for Data Center Thermal Management AAAI 2026
Soumyendu Sarkar, Antonio Guillen-Perez, Zachariah J Carmichael, Avisek Naug, Refik Mert Cam, Vineet Gundecha, Ashwin Ramesh Babu, Sahand Ghorbanpour, Ricardo Luna Gutierrez
Reducing energy consumption and carbon emissions in data centers by enabling real-time temperature prediction is critical for sustainability and operational efficiency. Achieving this requires accurate modeling of the 3D temperature field to capture airflow dynamics and thermal interactions under varying operating conditions. Traditional thermal CFD solvers, while accurate, are computationally expensive and require expert-crafted meshes and boundary conditions, making them impractical for real-time use. To address these limitations, we develop a vision-based surrogate modeling framework that operates directly on a 3D voxelized representation of the data center, incorporating server workloads, fan speeds, and HVAC temperature set points. We evaluate multiple architectures, including 3D CNN U-Net variants, a 3D Fourier Neural Operator, and 3D vision transformers, to map these thermal inputs to high-fidelity heat maps. Our results show that the surrogate models generalize across data center configurations and significantly speed up computations (20,000x), from hundreds of milliseconds to hours. This fast and accurate estimation of hot spots and temperature distribution enables real-time cooling control and workload redistribution, leading to substantial energy savings (7\%) and reduced carbon footprint.
comment : Submitted to AAAI 2026 Conference
Computation and Language 110
☆ Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
Jack Cook, Junxian Guo, Guangxuan Xiao, Yujun Lin, Song Han
As large language models have grown larger, low-precision numerical formats such as NVFP4 have become increasingly popular due to the speed and memory benefits they provide. However, to accelerate computation with NVFP4, all matrix multiplication operands--weights and activations in the forward pass, and weights, activations, and gradients in the backward pass--must be quantized to NVFP4, often leading to divergence during training and performance degradation during inference. NVFP4 by evaluating multiple potential scale factors for each block of values. To address this issue, in this work we introduce Four Over Six (4/6), a modification to the NVFP4 quantization algorithm that evaluates two potential scale factors for each block of values. Unlike integer formats, floating-point formats such as FP4 have the most quantization error on near-maximal values in each block, which we find to be primarily responsible for downstream performance degradation. We find that for some blocks, scaling to smaller FP4 values makes the distribution of representable values more uniform, improving representation of near-maximal values. Importantly, 4/6 can be implemented efficiently on NVIDIA Blackwell GPUs, making it viable to use while training LLMs with NVFP4. In pre-training experiments with transformer and hybrid model architectures, we find that 4/6 prevents divergence in several cases, bringing training loss significantly closer to BF16 compared to models trained with current state-of-the-art NVFP4 training recipes. We also find that 4/6 can be easily incorporated into many different post-training quantization methods and generally improves downstream accuracy. We hope this inspires future work in training and deploying models with NVFP4.
comment : 10 pages, 5 figures
☆ The Art of Scaling Test-Time Compute for Large Language Models
Aradhye Agarwal, Ayan Sengupta, Tanmoy Chakraborty
Test-time scaling (TTS) -- the dynamic allocation of compute during inference -- is a promising direction for improving reasoning in large language models (LLMs). However, a systematic comparison of well-known TTS strategies under identical conditions is missing, and the influence of model type and problem difficulty on performance remains unclear. To address these gaps, we conduct the first large-scale study of TTS, spanning over thirty billion tokens generated using eight open-source LLMs (7B to 235B parameters), across four reasoning datasets. We observe three consistent trends: (1) no single TTS strategy universally dominates; (2) reasoning models exhibit distinct trace-quality patterns across problem difficulty and trace length, forming short-horizon and long-horizon categories; and (3) for a given model type, the optimal TTS performance scales monotonically with compute budget. Based on these insights, we provide a practical recipe for selecting the best TTS strategy, considering problem difficulty, model type, and compute budget, providing a practical guide to effective inference-time scaling.
☆ AlignSAE: Concept-Aligned Sparse Autoencoders
Minglai Yang, Xinyu Guo, Mihai Surdeanu, Liangming Pan
Large Language Models (LLMs) encode factual knowledge within hidden parametric spaces that are difficult to inspect or control. While Sparse Autoencoders (SAEs) can decompose hidden activations into more fine-grained, interpretable features, they often struggle to reliably align these features with human-defined concepts, resulting in entangled and distributed feature representations. To address this, we introduce AlignSAE, a method that aligns SAE features with a defined ontology through a "pre-train, then post-train" curriculum. After an initial unsupervised training phase, we apply supervised post-training to bind specific concepts to dedicated latent slots while preserving the remaining capacity for general reconstruction. This separation creates an interpretable interface where specific relations can be inspected and controlled without interference from unrelated features. Empirical results demonstrate that AlignSAE enables precise causal interventions, such as reliable "concept swaps", by targeting single, semantically aligned slots.
comment : 20 pages, 7 figures, 5 tables
☆ LLM CHESS: Benchmark ing Reasoning and Instruction-Following in LLMs through Chess
Sai Kolasani, Maxim Saplin, Nicholas Crispino, Kyle Montgomery, Jared Quincy Davis, Matei Zaharia, Chi Wang, Chenguang Wang
We introduce LLM CHESS, an evaluation framework designed to probe the generalization of reasoning and instruction-following abilities in large language models (LLMs) through extended agentic interaction in the domain of chess. We rank over 50 open and closed source models by playing against a random opponent using a range of behavioral metrics, including win and loss rates, move quality, move legality, hallucinated actions, and game duration. For a subset of top reasoning models, we derive an Elo estimate by playing against a chess engine with variably configured skill, which allows for comparisons between models in an easily understandable way. Despite the simplicity of the instruction-following task and the weakness of the opponent, many state-of-the-art models struggle to complete games or achieve consistent wins. Similar to other benchmarks on complex reasoning tasks, our experiments reveal a clear separation between reasoning and non-reasoning models. However, unlike existing static benchmarks, the stochastic and dynamic nature of LLM CHESS uniquely reduces overfitting and memorization while preventing benchmark saturation, proving difficult even for top reasoning models. To support future work on evaluating reasoning and instruction-following in LLMs, we release our experimental framework, a public leaderboard, and a dataset of associated games.
☆ Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback
Aiden Yiliu Li, Bizhi Yu, Daoan Lei, Tianhe Ren, Shilong Liu
GUI grounding aims to align natural language instructions with precise regions in complex user interfaces. Advanced multimodal large language models show strong ability in visual GUI grounding but still struggle with small or visually similar targets and ambiguity in real world layouts. These limitations arise from limited grounding capacity and from underuse of existing reasoning potential. We present Chain of Ground CoG a training free multi step grounding framework that uses multimodal large language models for iterative visual reasoning and refinement. Instead of direct prediction the model progressively reflects and adjusts its hypotheses leading to more accurate and interpretable localization. Our approach achieves 68.4 accuracy on the ScreenSpot Pro benchmark an improvement of 4.8 points. To measure real world generalization we introduce TPanel UI a dataset of 420 labeled industrial control panels with visual distortions such as blur and masking. On TPanel UI Chain of Ground improves over the strong baseline Qwen3 VL 235B by 6.9 points showing the effectiveness of multi step training free grounding across real world and digital interfaces. These results highlight a direction for unlocking grounding potential through structured iterative refinement instead of additional training.
☆ From Atomic to Composite: Reinforcement Learning Enables Generalization in Complementary Reasoning
Sitao Cheng, Xunjian Yin, Ruiwen Zhou, Yuxuan Li, Xinyi Wang, Liangming Pan, William Yang Wang, Victor Zhong
The mechanism by which RL contributes to reasoning capabilities-whether it incentivizes the synthesis of new skills or merely amplifies existing behaviors-remains a subject of intense debate. In this work, we investigate this question through the lens of Complementary Reasoning, a complex task that requires integrating internal parametric knowledge with external contextual information. Using a controlled synthetic dataset of human biographies, we strictly decouple this ability into two atomic skills: Parametric Reasoning (relying on internal knowledge) and Contextual Reasoning (depending on external information). To rigorously assess capability boundaries, we evaluate generalization across three distinct levels of difficulty: I.I.D., Composition, and Zero-shot settings. We find that while SFT is sufficient for in-distribution performance, it struggles with O.O.D. generalization, particularly in Zero-shot settings where relational combinations are novel. Crucially, we identify the SFT Generalization Paradox: Models supervised solely on the composite task achieve near-perfect in-distribution accuracy but collapse on out-of-distribution generalization, indicating their reliance on rote memorization of path shortcuts. In contrast, we find that RL acts as a reasoning synthesizer rather than a probability amplifier. However, we uncover a strict atomic prerequisite: RL can only synthesize these complex strategies if the base model has first mastered the independent atomic skills (Parametric and Contextual) via SFT. These findings challenge the view of RL as a mere amplifier, suggesting that given sufficient atomic foundations, RL can actively synthesize complex reasoning strategies from learned primitives without explicit supervision on such complex strategies. This indicates that decoupled atomic training followed by RL offers a scalable path to generalization for complex reasoning tasks.
comment : Work in Progress. Code and data will be available at https://github.com/sitaocheng/from_atomic_to_composite
☆ How Far Are We from Genuinely Useful Deep Research Agent s?
Dingling Zhang, He Zhu, Jincheng Ren, Kangqi Song, Xinran Zhou, Boyu Feng, Shudong Liu, Jiabin Luo, Weihao Xie, Zhaohui Wang, Tianrui Qin, King Zhu, Yuqing Wang, Qianben Chen, Yuchen Eleanor Jiang, Wei Wang, Jiaheng Liu, Wangchunshu Zhou
Deep Research Agents (DRAs) aim to automatically produce analyst-level reports through iterative information retrieval and synthesis. However, most existing DRAs were validated on question-answering benchmarks, while research on generating comprehensive reports remains overlooked. Worse, current benchmarks for report synthesis suffer from task complexity and subjective metrics -- this fails to reflect user demands and limits the practical utility of generated reports. To address these gaps, we present Fine-grained DEepResearch bench (FINDER), an enhanced benchmark consisting of 100 human-curated research tasks with 419 structured checklist items that standardize report structure, analytical depth, and factual grounding. Based on approximately 1,000 reports produced by mainstream DRAs, we further propose Deep rEsearch Failure Taxonomy (DEFT), the first failure taxonomy for deep research agents. DEFT contains 14 fine-grained failure modes across reasoning, retrieval, and generation, and is built upon grounded theory with human-LLM co-annotating and inter-annotator reliability validation. Our experimental findings reveal that current DRAs struggle not with task comprehension but with evidence integration, verification, and reasoning-resilient planning.
comment : 34 pages
☆ Agent ic Policy Optimization via Instruction-Policy Co-Evolution
Han Zhou, Xingchen Wan, Ivan Vulić, Anna Korhonen
Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capability of large language models (LLMs), enabling autonomous agents that can conduct effective multi-turn and tool-integrated reasoning. While instructions serve as the primary protocol for defining agents, RLVR typically relies on static and manually designed instructions. However, those instructions may be suboptimal for the base model, and the optimal instruction may change as the agent's policy improves and explores the interaction with the environment. To bridge the gap, we introduce INSPO, a novel Instruction-Policy co-evolution framework that integrates instruction optimization as a dynamic component of the reinforcement learning (RL) loop. INSPO maintains a dynamic population of instruction candidates that are sampled with questions, where reward signals in RL loops are automatically attributed to each instruction, and low performers are periodically pruned. New instructions are generated and verified through an on-policy reflection mechanism, where an LLM-based optimizer analyzes past experience from a replay buffer and evolves more effective strategies given the current policy. We conduct extensive experiments on multi-turn retrieval and reasoning tasks, demonstrating that INSPO substantially outperforms strong baselines relying on static instructions. INSPO discovers innovative instructions that guide the agent toward more strategic reasoning paths, achieving substantial performance gains with only a marginal increase in computational overhead.
comment : 10 pages, 3 figures, 2 tables (18 pages including references and appendices)
☆ Rectifying LLM Thought from Lens of Optimization
Junnan Liu, Hongwei Liu, Songyang Zhang, Kai Chen
Recent advancements in large language models (LLMs) have been driven by their emergent reasoning capabilities, particularly through long chain-of-thought (CoT) prompting, which enables thorough exploration and deliberation. Despite these advances, long-CoT LLMs often exhibit suboptimal reasoning behaviors, such as overthinking and excessively protracted reasoning chains, which can impair performance. In this paper, we analyze reasoning processes through an optimization lens, framing CoT as a gradient descent procedure where each reasoning step constitutes an update toward problem resolution. Building on this perspective, we introduce RePro (Rectifying Process-level Reward), a novel approach to refine LLM reasoning during post-training. RePro defines a surrogate objective function to assess the optimization process underlying CoT, utilizing a dual scoring mechanism to quantify its intensity and stability. These scores are aggregated into a composite process-level reward, seamlessly integrated into reinforcement learning with verifiable rewards (RLVR) pipelines to optimize LLMs. Extensive experiments across multiple reinforcement learning algorithms and diverse LLMs, evaluated on benchmarks spanning mathematics, science, and coding, demonstrate that RePro consistently enhances reasoning performance and mitigates suboptimal reasoning behaviors.
comment : Work in progress
☆ Latent Debate: A Surrogate Framework for Interpreting LLM Thinking
Lihu Chen, Xiang Yin, Francesca Toni
Understanding the internal thinking process of Large Language Models (LLMs) and the cause of hallucinations remains a key challenge. To this end, we introduce latent debate, a novel framework for interpreting model predictions through the lens of implicit internal arguments. Unlike the current work of self-consistency and multi-agent debate, which relies on explicit debates among multiple answers or multiple models, latent debate captures the hidden supporting and attacking signals that arise within a single model during a single inference. We first present a model- and task-agnostic conceptual framework, and then instantiate it symbolically to approximate the thinking process of LLMs on True/False prediction tasks. Empirical studies demonstrate that latent debate is a faithful structured surrogate model that has highly consistent predictions with the original LLM. Beyond interpretability, we demonstrate that latent debate provides a strong baseline for hallucination detection. Further analysis reveals strong correlations between hallucinations and debate patterns, such as a high degree of latent debates in the middle layers is linked to a higher risk of hallucinations. These findings position latent debate as a potential framework for understanding internal mechanisms of LLMs, especially for scenarios where internal (dis)agreements appear during the inference steps.
comment : Preprint
☆ OPOR-Bench: Evaluating Large Language Models on Online Public Opinion Report Generation
Jinzheng Yu, Yang Xu, Haozhen Li, Junqi Li, Yifan Feng, Ligu Zhu, Hao Shen, Lei Shi
Online Public Opinion Reports consolidate news and social media for timely crisis management by governments and enterprises. While large language models have made automated report generation technically feasible, systematic research in this specific area remains notably absent, particularly lacking formal task definitions and corresponding benchmarks. To bridge this gap, we define the Automated Online Public Opinion Report Generation (OPOR-GEN) task and construct OPOR-BENCH, an event-centric dataset covering 463 crisis events with their corresponding news articles, social media posts, and a reference summary. To evaluate report quality, we propose OPOR-EVAL, a novel agent-based framework that simulates human expert evaluation by analyzing generated reports in context. Experiments with frontier models demonstrate that our framework achieves high correlation with human judgments. Our comprehensive task definition, benchmark dataset, and evaluation framework provide a solid foundation for future research in this critical domain.
comment : 27 pages, accepted by CMC-Computers, Materials & Continua, 2025
☆ Exploring Human Perceptions of AI Responses: Insights from a Mixed-Methods Study on Risk Mitigation in Generative Models
Heloisa Candello, Muneeza Azmat, Uma Sushmitha Gunturi, Raya Horesh, Rogerio Abreu de Paula, Heloisa Pimentel, Marcelo Carpinette Grave, Aminat Adebiyi, Tiago Machado, Maysa Malfiza Garcia de Macedo
With the rapid uptake of generative AI, investigating human perceptions of generated responses has become crucial. A major challenge is their `aptitude' for hallucinating and generating harmful contents. Despite major efforts for implementing guardrails, human perceptions of these mitigation strategies are largely unknown. We conducted a mixed-method experiment for evaluating the responses of a mitigation strategy across multiple-dimensions: faithfulness, fairness, harm-removal capacity, and relevance. In a within-subject study design, 57 participants assessed the responses under two conditions: harmful response plus its mitigation and solely mitigated response. Results revealed that participants' native language, AI work experience, and annotation familiarity significantly influenced evaluations. Participants showed high sensitivity to linguistic and contextual attributes, penalizing minor grammar errors while rewarding preserved semantic contexts. This contrasts with how language is often treated in the quantitative evaluation of LLMs. We also introduced new metrics for training and evaluating mitigation strategies and insights for human-AI evaluation studies.
comment : 16 pages, 2 figures, 6 tables. Under review for publication
☆ Cross-Lingual Inte rl eaving for Speech Language Models
Adel Moumen, Guangzhi Sun, Philip C. Woodland
Spoken Language Models (SLMs) aim to learn linguistic competence directly from speech using discrete units, widening access to Natural Language Processing (NLP) technologies for languages with limited written resources. However, progress has been largely English-centric due to scarce spoken evaluation benchmarks and training data, making cross-lingual learning difficult. We present a cross-lingual interleaving method that mixes speech tokens across languages without textual supervision. We also release an EN-FR training dataset, TinyStories (~42k hours), together with EN-FR spoken StoryCloze and TopicCloze benchmarks for cross-lingual semantic evaluation, both synthetically generated using GPT-4. On 360M and 1B SLMs under matched training-token budgets, interleaving improves monolingual semantic accuracy, enables robust cross-lingual continuation, and strengthens cross-lingual hidden-state alignment. Taken together, these results indicate that cross-lingual interleaving is a simple, scalable route to building multilingual SLMs that understand and converse across languages. All resources will be made open-source to support reproducibility.
☆ BHRAM-IL: A Benchmark for Hallucination Recognition and Assessment in Multiple Indian Languages AACL 2025
Hrishikesh Terdalkar, Kirtan Bhojani, Aryan Dongare, Omm Aditya Behera
Large language models (LLMs) are increasingly deployed in multilingual applications but often generate plausible yet incorrect or misleading outputs, known as hallucinations. While hallucination detection has been studied extensively in English, under-resourced Indian languages remain largely unexplored. We present BHRAM-IL, a benchmark for hallucination recognition and assessment in multiple Indian languages, covering Hindi, Gujarati, Marathi, Odia, along with English. The benchmark comprises 36,047 curated questions across nine categories spanning factual, numerical, reasoning, and linguistic tasks. We evaluate 14 state-of-the-art multilingual LLMs on a benchmark subset of 10,265 questions, analyzing cross-lingual and factual hallucinations across languages, models, scales, categories, and domains using category-specific metrics normalized to (0,1) range. Aggregation over all categories and models yields a primary score of 0.23 and a language-corrected fuzzy score of 0.385, demonstrating the usefulness of BHRAM-IL for hallucination-focused evaluation. The dataset, and the code for generation and evaluation are available on GitHub (https://github.com/sambhashana/BHRAM-IL/) and HuggingFace (https://huggingface.co/datasets/sambhashana/BHRAM-IL/) to support future research in multilingual hallucination detection and mitigation.
comment : Accepted at BHASHA Workshop @ IJCNLP/AACL 2025
☆ Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Ability
Jinghan Jia, Nathalie Baracaldo, Sijia Liu
Large reasoning models (LRMs) extend large language models by generating explicit chain-of-thought (CoT) reasoning, significantly improving mathematical and logical problem solving. However, this explicit reasoning process also introduces new safety risks, as unsafe behaviors often emerge within intermediate reasoning trajectories, even when final answers appear harmless. Existing safety alignment approaches primarily rely on supervised fine-tuning (SFT) over safety-oriented long CoT datasets. While intuitive, we find that SFT produces inconsistent safety improvements, degrades reasoning ability, and generalizes poorly across model families. These limitations suggest that purely supervised approaches are insufficient for robust safety alignment in LRMs. To address this, we investigate reinforcement learning (RL) as a complementary optimization framework for LRM safety training. Unlike SFT, RL directly optimizes model policies with reward feedback, enabling more adaptive and stable alignment. Extensive experiments across multiple model families and benchmarks show that RL achieves stronger and more consistent safety gains while maintaining reasoning competence. Further analysis of reflection dynamics and token-level entropy reveals that RL suppresses unsafe exploratory reasoning while preserving reflective depth, leading to safer and more reliable reasoning processes.
☆ InnoGym: Benchmark ing the Innovation Potential of AI Agent s
Jintian Zhang, Kewei Xu, Jingsheng Zheng, Zhuoyun Yu, Yuqi Zhu, Yujie Luo, Lanning Wei, Shuofei Qiao, Lun Du, Da Zheng, Shumin Deng, Huajun Chen, Ningyu Zhang
LLMs and Agents have achieved impressive progress in code generation, mathematical reasoning, and scientific discovery. However, existing benchmarks primarily measure correctness, overlooking the diversity of methods behind solutions. True innovation depends not only on producing correct answers but also on the originality of the approach. We present InnoGym, the first benchmark and framework designed to systematically evaluate the innovation potential of AI agents. InnoGym introduces two complementary metrics: performance gain, which measures improvement over the best-known solutions, and novelty, which captures methodological differences from prior approaches. The benchmark includes 18 carefully curated tasks from real-world engineering and scientific domains, each standardized through resource filtering, evaluator validation, and solution collection. In addition, we provide iGym, a unified execution environment for reproducible and long-horizon evaluations. Extensive experiments show that while some agents produce novel approaches, their lack of robustness limits performance gains. These results highlight a key gap between creativity and effectiveness, underscoring the need for benchmarks that evaluate both.
comment : Work in progress
☆ H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons
Cheng Gao, Huimin Chen, Chaojun Xiao, Zhiyi Chen, Zhiyuan Liu, Maosong Sun
Large language models (LLMs) frequently generate hallucinations -- plausible but factually incorrect outputs -- undermining their reliability. While prior work has examined hallucinations from macroscopic perspectives such as training data and objectives, the underlying neuron-level mechanisms remain largely unexplored. In this paper, we conduct a systematic investigation into hallucination-associated neurons (H-Neurons) in LLMs from three perspectives: identification, behavioral impact, and origins. Regarding their identification, we demonstrate that a remarkably sparse subset of neurons (less than $0.1\%$ of total neurons) can reliably predict hallucination occurrences, with strong generalization across diverse scenarios. In terms of behavioral impact, controlled interventions reveal that these neurons are causally linked to over-compliance behaviors. Concerning their origins, we trace these neurons back to the pre-trained base models and find that these neurons remain predictive for hallucination detection, indicating they emerge during pre-training. Our findings bridge macroscopic behavioral patterns with microscopic neural mechanisms, offering insights for developing more reliable LLMs.
comment : 20 pages, 4 figures
☆ Reasoning About the Unsaid: Mis information Detection with Omission-Aware Graph Inference AAAI 2026
Zhengjia Wang, Danding Wang, Qiang Sheng, Jiaying Wu, Juan Cao
This paper investigates the detection of misinformation, which deceives readers by explicitly fabricating misleading content or implicitly omitting important information necessary for informed judgment. While the former has been extensively studied, omission-based deception remains largely overlooked, even though it can subtly guide readers toward false conclusions under the illusion of completeness. To pioneer in this direction, this paper presents OmiGraph, the first omission-aware framework for misinformation detection. Specifically, OmiGraph constructs an omission-aware graph for the target news by utilizing a contextual environment that captures complementary perspectives of the same event, thereby surfacing potentially omitted contents. Based on this graph, omission-oriented relation modeling is then proposed to identify the internal contextual dependencies, as well as the dynamic omission intents, formulating a comprehensive omission relation representation. Finally, to extract omission patterns for detection, OmiGraph introduces omission-aware message-passing and aggregation that establishes holistic deception perception by integrating the omission contents and relations. Experiments show that, by considering the omission perspective, our approach attains remarkable performance, achieving average improvements of +5.4% F1 and +5.3% ACC on two large-scale benchmarks.
comment : AAAI 2026
☆ Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Tasks
Jiannan Guan, Qiguang Chen, Libo Qin, Dengyun Peng, Jinhao Liu, Liangyu Huo, Jian Xie, Wanxiang Che
Large Language Models (LLMs) excel in reasoning tasks requiring a single correct answer, but they perform poorly in multi-solution tasks that require generating comprehensive and diverse answers. We attribute this limitation to \textbf{reasoning overconfidence}: a tendency to express undue certainty in an incomplete solution set. To examine the effect, we introduce \textit{MuSoBench}, a benchmark of multi-solution problems. Experiments show that the conventional short chain-of-thought (Short-CoT) prompting paradigm exhibits pronounced overconfidence, whereas the emerging long chain-of-thought (Long-CoT) approach mitigates it through iterative exploration and self-reflection. We further characterise observable behaviours and influential factors. To probe the underlying cause, we propose the \textbf{cognitive-rigidity hypothesis}, which posits that overconfidence arises when the reasoning process prematurely converges on a narrow set of thought paths. An attention-entropy analysis offers preliminary support for this view. These findings provide tools for assessing the completeness of LLM reasoning and highlight the need to move evaluation beyond single-answer accuracy toward comprehensive exploration.
☆ Self-Supervised Borrowing Detection on Multilingual Wordlists
Tim Wientzek
This paper presents a fully self-supervised approach to borrowing detection in multilingual wordlists. The method combines two sources of information: PMI similarities based on a global correspondence model and a lightweight contrastive component trained on phonetic feature vectors. It further includes an automatic procedure for selecting decision thresholds without requiring labeled data. Experiments on benchmark datasets show that PMI alone already improves over existing string similarity measures such as NED and SCA, and that the combined similarity performs on par with or better than supervised baselines. An ablation study highlights the importance of character encoding, temperature settings and augmentation strategies. The approach scales to datasets of different sizes, works without manual supervision and is provided with a command-line tool that allows researchers to conduct their own studies.
comment : 29 pages, 3 figures, 12 tables
☆ MMAG: Mixed Memory-Augmented Generation for Large Language Models Applications
Stefano Zeppieri
Large Language Models (LLMs) excel at generating coherent text within a single prompt but fall short in sustaining relevance, personalization, and continuity across extended interactions. Human communication, however, relies on multiple forms of memory, from recalling past conversations to adapting to personal traits and situational context. This paper introduces the Mixed Memory-Augmented Generation (MMAG) pattern, a framework that organizes memory for LLM-based agents into five interacting layers: conversational, long-term user, episodic and event-linked, sensory and context-aware, and short-term working memory. Drawing inspiration from cognitive psychology, we map these layers to technical components and outline strategies for coordination, prioritization, and conflict resolution. We demonstrate the approach through its implementation in the Heero conversational agent, where encrypted long-term bios and conversational history already improve engagement and retention. We further discuss implementation concerns around storage, retrieval, privacy, and latency, and highlight open challenges. MMAG provides a foundation for building memory-rich language agents that are more coherent, proactive, and aligned with human needs.
☆ StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos
Daeun Lee, Subhojyoti Mukherjee, Branislav Kveton, Ryan A. Rossi, Viet Dac Lai, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Mohit Bansal
Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intention for realistic applications like AR glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether MLLMs can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs use gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively evaluate streaming video understanding. These tasks assess whether models can use real-time gaze to follow shifting attention and infer user intentions from only past and currently observed frames. To build StreamGaze, we develop a gaze-video QA generation pipeline that aligns egocentric videos with raw gaze trajectories via fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that closely reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, revealing fundamental limitations in gaze-based temporal reasoning, intention modeling, and proactive prediction. We further provide detailed analyses of gaze-prompting strategies, reasoning behaviors, and task-specific failure modes, offering deeper insight into why current MLLMs struggle and what capabilities future models must develop. All data and code will be publicly released to support continued research in gaze-guided streaming video understanding.
comment : Project page: https://streamgaze.github.io/
☆ Learning the Boundary of Solvability: Aligning LLMs to Detect Unsolvable Problems
Dengyun Peng, Qiguang Chen, Bofei Liu, Jiannan Guan, Libo Qin, Zheng Yan, Jinhao Liu, Jianshu Zhang, Wanxiang Che
Ensuring LLM reliability requires not only solving complex problems but also recognizing when a problem is unsolvable. Current models often struggle to distinguish objective unsolvability (inherent contradictions in the problem) from subjective capability limitations (problems beyond the model's competence), which leads to hallucinations and overconfidence. To address this, we propose UnsolvableQA and UnsolvableRL to solve feasible problems, detect inherent contradictions, and prudently refuse tasks beyond capability. Specifically, we construct UnsolvableQA, a dataset of paired solvable and unsolvable instances derived via a dual-track methodology: programmatic generation for logic puzzles and a novel "Reverse Construction" method that injects contradictions into valid reasoning chains for mathematics. Building on this dataset, we introduce UnsolvableRL, a reinforcement learning framework with three reward components jointly accounting for accuracy, unsolvability, and difficulty. Empirical results show that our approach achieves near-perfect unsolvability detection while also improving accuracy on solvable tasks. Crucially, we identify Capability Collapse, demonstrating that explicit exposure to unsolvable data is indispensable for preventing models from becoming systematically overconfident. Our code and data are available at https://github.com/sfasfaffa/unsolvableQA.
comment : preprint
☆ HalluGraph: Auditable Hallucination Detection for Legal RAG Systems via Knowledge Graph Alignment
Valentin Noël, Elimane Yassine Seidou, Charly Ken Capo-Chichi, Ghanem Amari
Legal AI systems powered by retrieval-augmented generation (RAG) face a critical accountability challenge: when an AI assistant cites case law, statutes, or contractual clauses, practitioners need verifiable guarantees that generated text faithfully represents source documents. Existing hallucination detectors rely on semantic similarity metrics that tolerate entity substitutions, a dangerous failure mode when confusing parties, dates, or legal provisions can have material consequences. We introduce HalluGraph, a graph-theoretic framework that quantifies hallucinations through structural alignment between knowledge graphs extracted from context, query, and response. Our approach produces bounded, interpretable metrics decomposed into \textit{Entity Grounding} (EG), measuring whether entities in the response appear in source documents, and \textit{Relation Preservation} (RP), verifying that asserted relationships are supported by context. On structured control documents, HalluGraph achieves near-perfect discrimination ($>$400 words, $>$20 entities), HalluGraph achieves $AUC = 0.979$, while maintaining robust performance ($AUC \approx 0.89$) on challenging generative legal task, consistently outperforming semantic similarity baselines. The framework provides the transparency and traceability required for high-stakes legal applications, enabling full audit trails from generated assertions back to source passages.
comment : 8 pages, 4 figures, under review
☆ MAC-SLU: Multi-Intent Automotive Cabin Spoken Language Understanding Benchmark
Yuezhang Peng, Chonghao Cai, Ziang Liu, Shuai Fan, Sheng Jiang, Hua Xu, Yuxin Liu, Qiguang Chen, Kele Xu, Yao Li, Sheng Wang, Libo Qin, Xie Chen
Spoken Language Understanding (SLU), which aims to extract user semantics to execute downstream tasks, is a crucial component of task-oriented dialog systems. Existing SLU datasets generally lack sufficient diversity and complexity, and there is an absence of a unified benchmark for the latest Large Language Models (LLMs) and Large Audio Language Models (LALMs). This work introduces MAC-SLU, a novel Multi-Intent Automotive Cabin Spoken Language Understanding Dataset, which increases the difficulty of the SLU task by incorporating authentic and complex multi-intent data. Based on MAC-SLU, we conducted a comprehensive benchmark of leading open-source LLMs and LALMs, covering methods like in-context learning, supervised fine-tuning (SFT), and end-to-end (E2E) and pipeline paradigms. Our experiments show that while LLMs and LALMs have the potential to complete SLU tasks through in-context learning, their performance still lags significantly behind SFT. Meanwhile, E2E LALMs demonstrate performance comparable to pipeline approaches and effectively avoid error propagation from speech recognition. Code\footnote{https://github.com/Gatsby-web/MAC\_SLU} and datasets\footnote{huggingface.co/datasets/Gatsby1984/MAC\_SLU} are released publicly.
☆ Language Diversity: Evaluating Language Usage and AI Performance on African Languages in Digital Spaces
Edward Ajayi, Eudoxie Umwari, Mawuli Deku, Prosper Singadi, Jules Udahemuka, Bekalu Tadele, Chukuemeka Edeh
This study examines the digital representation of African languages and the challenges this presents for current language detection tools. We evaluate their performance on Yoruba, Kinyarwanda, and Amharic. While these languages are spoken by millions, their online usage on conversational platforms is often sparse, heavily influenced by English, and not representative of the authentic, monolingual conversations prevalent among native speakers. This lack of readily available authentic data online creates a challenge of scarcity of conversational data for training language models. To investigate this, data was collected from subreddits and local news sources for each language. The analysis showed a stark contrast between the two sources. Reddit data was minimal and characterized by heavy code-switching. Conversely, local news media offered a robust source of clean, monolingual language data, which also prompted more user engagement in the local language on the news publishers social media pages. Language detection models, including the specialized AfroLID and a general LLM, performed with near-perfect accuracy on the clean news data but struggled with the code-switched Reddit posts. The study concludes that professionally curated news content is a more reliable and effective source for training context-rich AI models for African languages than data from conversational platforms. It also highlights the need for future models that can process clean and code-switched text to improve the detection accuracy for African languages.
☆ LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems
Zhiyuan Wang, Aniri, Tianlong Chen, Yue Zhang, Heng Tao Shen, Xiaoshuang Shi, Kaidi Xu
Large language models (LLMs) often generate unreliable answers, while heuristic uncertainty methods fail to fully distinguish correct from incorrect predictions, causing users to accept erroneous answers without statistical guarantees. We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level. To achieve this in a principled way, we propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint over selection and error indicators. Then, we establish a finite-sample sufficient condition, which relies only on a held-out set of exchangeable calibration samples, to compute an FDR-constrained, coverage-maximizing threshold. Furthermore, we extend LEC to a two-model routing mechanism: given a prompt, if the current model's uncertainty exceeds its calibrated threshold, we delegate it to a stronger model, while maintaining a unified FDR guarantee. Evaluations on closed-ended and open-ended question-answering (QA) datasets show that LEC achieves tighter FDR control and substantially improves sample retention over prior methods. Moreover, the two-model routing mechanism achieves lower risk levels while accepting more correct samples than each individual model.
☆ LPCD: Unified Framework from Layer-Wise to Submodule Quantization
Yuma Ichikawa, Yudai Fujimoto, Akira Sakai
Post-training quantization (PTQ) aims to preserve model-level behavior; however, most methods focus on individual linear layers. Even recent extensions, such as QEP and LoaQ, which mitigate error propagation or target specific submodules, still rely on layer-wise formulations and fail to capture the behavior of larger submodules. We introduce Layer-Projected Coordinate Descent (LPCD), a unified framework that extends PTQ beyond layers by optimizing relaxed objectives across arbitrary submodules and projecting the solutions with layer-wise quantizers. LPCD generalizes existing methods and provides a principled approach to quantizing complex submodules while maintaining the efficiency and compatibility of layer-wise PTQ pipelines. Across diverse LLM architectures and bit-widths, LPCD-based submodule quantization consistently enhances both layer-wise PTQ methods and existing submodule approaches.
comment : 21 pages, 4 figures
☆ MCAT: Scaling Many-to-Many Speech-to-Text Translation with MLLMs to 70 Languages
Yexing Du, Kaiyuan Liu, Youcheng Pan, Bo Yang, Keqi Deng, Xie Chen, Yang Xiang, Ming Liu, Bin Qin, YaoWei Wang
Multimodal Large Language Models (MLLMs) have achieved great success in Speech-to-Text Translation (S2TT) tasks. However, current research is constrained by two key challenges: language coverage and efficiency. Most of the popular S2TT datasets are substantially English-centric, which restricts the scaling-up of MLLMs' many-to-many translation capabilities. Moreover, the inference speed of MLLMs degrades dramatically when the speech is converted into long sequences (e.g., 750 tokens). To address these limitations, we propose a Multilingual Cost-effective Accelerated Speech-to-Text Translator (MCAT) framework, which includes two innovations. First, a language scaling method that leverages curriculum learning and a data balancing strategy is introduced to extend the language coverage supported by MLLMs to 70 languages and achieve mutual translation among these languages. Second, an optimized speech adapter module is designed to reduce the length of the speech sequence to only 30 tokens. Extensive experiments were conducted on MLLMs of different scales (9B and 27B). The experimental results demonstrate that MCAT not only surpasses state-of-the-art end-to-end models on the FLEURS dataset across 70x69 directions but also enhances batch inference efficiency. This is achieved with only ~100M trainable parameters and by using only 10 hours of S2TT data per language. Furthermore, we have released MCAT as open-source to promote the development of MLLMs for robust S2TT capabilities. The code and models are released at https://github.com/yxduir/m2m-70.
☆ Enhancing BERT Fine-Tuning for Sentiment Analysis in Lower-Resourced Languages
Jozef Kubík, Marek Šuppa, Martin Takáč
Limited data for low-resource languages typically yield weaker language models (LMs). Since pre-training is compute-intensive, it is more pragmatic to target improvements during fine-tuning. In this work, we examine the use of Active Learning (AL) methods augmented by structured data selection strategies which we term 'Active Learning schedulers', to boost the fine-tuning process with a limited amount of training data. We connect the AL to data clustering and propose an integrated fine-tuning pipeline that systematically combines AL, clustering, and dynamic data selection schedulers to enhance model's performance. Experiments in the Slovak, Maltese, Icelandic and Turkish languages show that the use of clustering during the fine-tuning phase together with AL scheduling can simultaneously produce annotation savings up to 30% and performance improvements up to four F1 score points, while also providing better fine-tuning stability.
☆ ZIP-RC: Zero-overhead Inference-time Prediction of Reward and Cost for Adaptive and Interpretable Generation
Rohin Manvi, Joey Hong, Tim Seyde, Maxime Labonne, Mathias Lechner, Sergey Levine
Large language models excel at reasoning but lack key aspects of introspection, including anticipating their own success and the computation required to achieve it. Humans use real-time introspection to decide how much effort to invest, when to make multiple attempts, when to stop, and when to signal success or failure. Without this, LLMs struggle to make intelligent meta-cognition decisions. Test-time scaling methods like Best-of-N drive up cost and latency by using a fixed budget of samples regardless of the marginal benefit of each one at any point in generation, and the absence of confidence signals can mislead people, prevent appropriate escalation to better tools, and undermine trustworthiness. Learned verifiers or reward models can provide confidence estimates, but do not enable adaptive inference and add substantial cost by requiring extra models or forward passes. We present ZIP-RC, an adaptive inference method that equips models with zero-overhead inference-time predictions of reward and cost. At every token, ZIP-RC reuses reserved or unused logits in the same forward pass as next-token prediction to output a joint distribution over final reward and remaining length -- no extra models, architecture change, or inference overhead. This full joint distribution is used to compute a sampling utility which is the linear combination of the expected maximum reward, total compute, and latency of set of samples if generated to completion. During inference, we maximize this utility with meta-actions that determine which prefix of tokens to continue or initiate sampling from. On mixed-difficulty mathematical benchmarks, ZIP-RC improves accuracy by up to 12% over majority voting at equal or lower average cost, and traces smooth Pareto frontiers between quality, compute, and latency. By providing real-time reward-cost introspection, ZIP-RC enables adaptive, efficient reasoning.
comment : Code coming soon
☆ MEGConformer: Conformer-Based MEG Decoder for Robust Speech and Phoneme Classification NeurIPS 2025
Xabier de Zuazo, Ibon Saratxaga, Eva Navas
We present Conformer-based decoders for the LibriBrain 2025 PNPL competition, targeting two foundational MEG tasks: Speech Detection and Phoneme Classification. Our approach adapts a compact Conformer to raw 306-channel MEG signals, with a lightweight convolutional projection layer and task-specific heads. For Speech Detection, a MEG-oriented SpecAugment provided a first exploration of MEG-specific augmentation. For Phoneme Classification, we used inverse-square-root class weighting and a dynamic grouping loader to handle 100-sample averaged examples. In addition, a simple instance-level normalization proved critical to mitigate distribution shifts on the holdout split. Using the official Standard track splits and F1-macro for model selection, our best systems achieved 88.9% (Speech) and 65.8% (Phoneme) on the leaderboard, surpassing the competition baselines and ranking within the top-10 in both tasks. For further implementation details, the technical documentation, source code, and checkpoints are available at https://github.com/neural2speech/libribrain-experiments.
comment : 10 pages, 5 figures, 4 tables, LibriBrain Workshop, NeurIPS 2025
☆ Multilingual Conversational AI for Financial Assistance: Bridging Language Barriers in Indian FinTech
Bharatdeep Hazarika, Arya Suneesh, Prasanna Devadiga, Pawan Kumar Rajpoot, Anshuman B Suresh, Ahmed Ifthaquar Hussain
India's linguistic diversity presents both opportunities and challenges for fintech platforms. While the country has 31 major languages and over 100 minor ones, only 10\% of the population understands English, creating barriers to financial inclusion. We present a multilingual conversational AI system for a financial assistance use case that supports code-mixed languages like Hinglish, enabling natural interactions for India's diverse user base. Our system employs a multi-agent architecture with language classification, function management, and multilingual response generation. Through comparative analysis of multiple language models and real-world deployment, we demonstrate significant improvements in user engagement while maintaining low latency overhead (4-8\%). This work contributes to bridging the language gap in digital financial services for emerging markets.
☆ Prompt Bridge: Cross-Model Prompt Transfer for Large Language Models
Yaxuan Wang, Quan Liu, Zhenting Wang, Zichao Li, Wei Wei, Yang Liu, Yujia Bao
Large language models (LLMs) underpin applications in code generation, mathematical reasoning, and agent-based workflows. In practice, systems access LLMs via commercial APIs or open-source deployments, and the model landscape (e.g., GPT, Claude, Llama) evolves rapidly. This rapid evolution forces frequent model switches driven by capability, cost, deployment constraints, and privacy. Yet prompts are highly model-sensitive: reusing a prompt engineered for one model on another often yields substantially worse performance than a prompt optimized for the target model. We term this phenomenon Model Drifting. Through extensive empirical analysis across diverse LLM configurations, we show that model drifting is both common and severe. To address this challenge, we introduce PromptBridge, a training-free framework that preserves prompt effectiveness under model switches, enabling cross-model prompt transfer without costly per-task or per-model re-optimization. PromptBridge requires only a small set of alignment tasks for calibration. It first applies Model-Adaptive Reflective Prompt Evolution (MAP-RPE) to obtain task- and model-specific optimal prompts via iterative reflective refinement and quantitative evaluation. Using the resulting calibrated prompt pairs for the source and target models, PromptBridge learns a cross-model prompt mapping. At test time, i.e., for an unseen task, given a source-model prompt, this mapping directly produces an optimized prompt for the target model. Experiments in single-agent and multi-agent settings show that PromptBridge consistently improves downstream accuracy while reducing migration effort. The code will be available soon.
☆ DyFuLM: An Advanced Multimodal Framework for Sentiment Analysis
Ruohan Zhou, Jiachen Yuan, Churui Yang, Wenzheng Huang, Guoyan Zhang, Shiyao Wei, Jiazhen Hu, Ning Xin, Md Maruf Hasan
Understanding sentiment in complex textual expressions remains a fundamental challenge in affective computing. To address this, we propose a Dynamic Fusion Learning Model (DyFuLM), a multimodal framework designed to capture both hierarchical semantic representations and fine-grained emotional nuances. DyFuLM introduces two key moodules: a Hierarchical Dynamic Fusion module that adaptively integrates multi-level features, and a Gated Feature Aggregation module that regulates cross-layer information ffow to achieve balanced representation learning. Comprehensive experiments on multi-task sentiment datasets demonstrate that DyFuLM achieves 82.64% coarse-grained and 68.48% fine-grained accuracy, yielding the lowest regression errors (MAE = 0.0674, MSE = 0.0082) and the highest R^2 coefficient of determination (R^2= 0.6903). Furthermore, the ablation study validates the effectiveness of each module in DyFuLM. When all modules are removed, the accuracy drops by 0.91% for coarse-grained and 0.68% for fine-grained tasks. Keeping only the gated fusion module causes decreases of 0.75% and 0.55%, while removing the dynamic loss mechanism results in drops of 0.78% and 0.26% for coarse-grained and fine-grained sentiment classification, respectively. These results demonstrate that each module contributes significantly to feature interaction and task balance. Overall, the experimental findings further validate that DyFuLM enhances sentiment representation and overall performance through effective hierarchical feature fusion.
comment : 8 pages, 6 figures, preprint. Under review for a suitable AI conference
☆ BackportBench: A Multilingual Benchmark for Automated Backporting of Patches
Zhiqing Zhong, Jiaming Huang, Pinjia He
Many modern software projects evolve rapidly to incorporate new features and security patches. It is important for users to update their dependencies to safer versions, but many still use older, vulnerable package versions because upgrading can be difficult and may break their existing codebase. Software developers can mitigate this problem by backporting security patches to older releases. However, manually backporting is time-consuming and error-prone. The effectiveness of existing automated backporting techniques on general software remains unclear since they typically target only code-hunk or function-level patch porting scenarios and are evaluated with imperfect metrics. To facilitate the development and evaluation of automated backporting techniques, we introduce BackportBench, the first comprehensive benchmark suite for patch backporting problem. BackportBench is a multilingual benchmark that contains 202 patch backporting problems from PyPI, Maven, and npm, each with executable Docker environments and relevant test cases. We evaluated existing patch porting methods and LLM-based techniques that have the potential to adapt to this task using BackportBench. The results show that the agentic method has outperformed traditional patch porting methods, especially on cases that require logical and structural changes. However, the performance varies across different programming languages. Based on the findings, we draw several implications for researchers and software practitioners in future work on automated backporting.
comment : Under review
☆ Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, An Yang, Jingren Zhou, Junyang Lin
This paper proposes a novel formulation for reinforcement learning (RL) with large language models, explaining why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE. Specifically, through a first-order approximation, we show that this surrogate becomes increasingly valid only when both the training-inference discrepancy and policy staleness are minimized. This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training, including importance sampling correction, clipping, and particularly Routing Replay for Mixture-of-Experts (MoE) models. Through extensive experiments with a 30B MoE model totaling hundreds of thousands of GPU hours, we show that for on-policy training, the basic policy gradient algorithm with importance sampling correction achieves the highest training stability. When off-policy updates are introduced to accelerate convergence, combining clipping and Routing Replay becomes essential to mitigate the instability caused by policy staleness. Notably, once training is stabilized, prolonged optimization consistently yields comparable final performance regardless of cold-start initialization. We hope that the shared insights and the developed recipes for stable RL training will facilitate future research.
☆ MARSAD: A Multi-Functional Tool for Real-Time Social Media Analysis
Md. Rafiul Biswas, Firoj Alam, Wajdi Zaghouani
MARSAD is a multifunctional natural language processing (NLP) platform designed for real-time social media monitoring and analysis, with a particular focus on the Arabic-speaking world. It enables researchers and non-technical users alike to examine both live and archived social media content, producing detailed visualizations and reports across various dimensions, including sentiment analysis, emotion analysis, propaganda detection, fact-checking, and hate speech detection. The platform also provides secure data-scraping capabilities through API keys for accessing public social media data. MARSAD's backend architecture integrates flexible document storage with structured data management, ensuring efficient processing of large and multimodal datasets. Its user-friendly frontend supports seamless data upload and interaction.
comment : 6 pages, 4 figures
☆ The Necessity of Imperfection:Reversing Model Collapse via Simulating Cognitive Boundedness
Zhongjie Jiang
Although synthetic data is widely promoted as a remedy, its prevailing production paradigm -- one optimizing for statistical smoothness -- systematically removes the long-tail, cognitively grounded irregularities that characterize human text. Prolonged training on such statistically optimal but cognitively impoverished data accelerates model collapse. This paper proposes a paradigm shift: instead of imitating the surface properties of data, we simulate the cognitive processes that generate human text. We introduce the Prompt-driven Cognitive Computing Framework (PMCSF), whose core consists of a Cognitive State Decoder (CSD) that reverse-engineers unstructured text into structured cognitive vectors, and a Cognitive Text Encoder (CTE) that re-materializes these states into text enriched with human-typical imperfections via mathematically defined Cognitive Perturbation Operators. The framework is validated through a two-stage objective evaluation pipeline. First, in cognitive codec verification, CTE text yields a Jensen-Shannon divergence of 0.0614 from human text (vs. 0.4431 for standard LLM output), passes double-blind professional media review, and achieves an intraclass correlation coefficient ICC > 0.9 for cognitive profile alignment across heterogeneous models. Second, in functional gain evaluation, isomorphic stress tests in the A-share market show that strategies incorporating CTE-generated data reduce maximum drawdown by 47.4% during the 2015 crash and deliver 8.6% Defensive Alpha, exceeding transaction costs by a factor of 33. Our findings demonstrate that modelling human cognitive limitations -- not copying surface data -- enables synthetic data with genuine functional gain, offering a viable technical pathway toward resolving the AI data-collapse crisis.
comment : 38 pages,5 figures,30 tables. This paper proposes the Prompt-driven Cognitive Computing Framework (PMCSF) and validates it with A-share market stress tests (N=23 for 2015 crash, N=13 for 2024 bull market). Includes detailed appendices on cognitive vector definitions, perturbation operators, and financial backtest data
☆ EmoRAG: Evaluating RAG Robustness to Symbolic Perturbations KDD
Xinyun Zhou, Xinfeng Li, Yinan Peng, Ming Xu, Xuanwang Zhang, Miao Yu, Yidong Wang, Xiaojun Jia, Kun Wang, Qingsong Wen, XiaoFeng Wang, Wei Dong
Retrieval-Augmented Generation (RAG) systems are increasingly central to robust AI, enhancing large language model (LLM) faithfulness by incorporating external knowledge. However, our study unveils a critical, overlooked vulnerability: their profound susceptibility to subtle symbolic perturbations, particularly through near-imperceptible emoticon tokens such as "(@_@)" that can catastrophically mislead retrieval, termed EmoRAG. We demonstrate that injecting a single emoticon into a query makes it nearly 100% likely to retrieve semantically unrelated texts that contain a matching emoticon. Our extensive experiment across general question-answering and code domains, using a range of state-of-the-art retrievers and generators, reveals three key findings: (I) Single-Emoticon Disaster: Minimal emoticon injections cause maximal disruptions, with a single emoticon almost 100% dominating RAG output. (II) Positional Sensitivity: Placing an emoticon at the beginning of a query can cause severe perturbation, with F1-Scores exceeding 0.92 across all datasets. (III) Parameter-Scale Vulnerability: Counterintuitively, models with larger parameters exhibit greater vulnerability to the interference. We provide an in-depth analysis to uncover the underlying mechanisms of these phenomena. Furthermore, we raise a critical concern regarding the robustness assumption of current RAG systems, envisioning a threat scenario where an adversary exploits this vulnerability to manipulate the RAG system. We evaluate standard defenses and find them insufficient against EmoRAG. To address this, we propose targeted defenses, analyzing their strengths and limitations in mitigating emoticon-based perturbations. Finally, we outline future directions for building robust RAG systems.
comment : Accepted to ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) 2026
☆ Securing Large Language Models (LLMs) from Prompt Injection Attacks
Omar Farooq Khan Suri, John McCrae
Large Language Models (LLMs) are increasingly being deployed in real-world applications, but their flexibility exposes them to prompt injection attacks. These attacks leverage the model's instruction-following ability to make it perform malicious tasks. Recent work has proposed JATMO, a task-specific fine-tuning approach that trains non-instruction-tuned base models to perform a single function, thereby reducing susceptibility to adversarial instructions. In this study, we evaluate the robustness of JATMO against HOUYI, a genetic attack framework that systematically mutates and optimizes adversarial prompts. We adapt HOUYI by introducing custom fitness scoring, modified mutation logic, and a new harness for local model testing, enabling a more accurate assessment of defense effectiveness. We fine-tuned LLaMA 2-7B, Qwen1.5-4B, and Qwen1.5-0.5B models under the JATMO methodology and compared them with a fine-tuned GPT-3.5-Turbo baseline. Results show that while JATMO reduces attack success rates relative to instruction-tuned models, it does not fully prevent injections; adversaries exploiting multilingual cues or code-related disruptors still bypass defenses. We also observe a trade-off between generation quality and injection vulnerability, suggesting that better task performance often correlates with increased susceptibility. Our results highlight both the promise and limitations of fine-tuning-based defenses and point toward the need for layered, adversarially informed mitigation strategies.
comment : 10 pages, 1 figure, 1 table
☆ Agreement-Constrained Probabilistic Minimum Bayes Risk Decoding AACL 2025
Koki Natsumi, Hiroyuki Deguchi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
Minimum Bayes risk (MBR) decoding generates high-quality translations by maximizing the expected utility of output candidates, but it evaluates all pairwise scores over the candidate set; hence, it takes quadratic time with respect to the number of candidates. To reduce the number of utility function calls, probabilistic MBR (PMBR) decoding partially evaluates quality scores using sampled pairs of candidates and completes the missing scores with a matrix completion algorithm. Nevertheless, it degrades the translation quality as the number of utility function calls is reduced. Therefore, to improve the trade-off between quality and cost, we propose agreement-constrained PMBR (AC-PMBR) decoding, which leverages a knowledge distilled model to guide the completion of the score matrix. Our AC-PMBR decoding improved approximation errors of matrix completion by up to 3 times and achieved higher translation quality compared with PMBR decoding at a comparable computational cost on the WMT'23 En$\leftrightarrow$De translation tasks.
comment : IJCNLP-AACL 2025 Main
☆ Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as- Judge Reinforcement Learning
Jiahao Yuan, Zhiqing Cui, Hanqing Wang, Yuansheng Gao, Yucheng Zhou, Usman Naseem
As web platforms evolve towards greater personalization and emotional complexity, conversational agents must transcend superficial empathy to demonstrate identity-aware emotional reasoning. However, existing systems face two limitations: (1) reliance on situation-centric datasets lacking persistent user identity, which hampers the capture of personalized affective nuances; and (2) dependence on opaque, coarse reward signals that hinder development of verifiable empathetic reasoning. To address these gaps, we introduce KardiaBench, a large-scale user-grounded benchmark comprising 178,080 QA pairs across 22,080 multi-turn conversations anchored to 671 real-world profiles. The dataset is constructed via a model-in-the-loop pipeline with iterative rubric-guided refinement to ensure psychological plausibility and persona consistency. This progressive empathy pipeline that integrates user comprehension, contextual reasoning, and emotion perception into conversations, followed by iterative critique and rubric-based refinement to ensure psychological plausibility, emotional fidelity, and persona consistency. Building on this, we propose Kardia-R1, a framework that trains models for interpretable, stepwise empathetic cognition. Kardia-R1 leverages Rubric-as-Judge Empathetic Reinforcement Learning (Rubric-ERL), a GRPO-based method that uses explainable, human-aligned rubric rewards to tightly couple user understanding, emotional inference, and supportive response generation. Extensive experiments across four LLM backbones demonstrate that Kardia-R1 consistently outperforms othet methods in emotion accuracy, empathy, relevance, persona consistency, and safety. Our dataset and model will be released at https://github.com/JhCircle/Kardia-R1.
☆ SUPERChem: A Multimodal Reasoning Benchmark in Chemistry
Zehua Zhao, Zhixian Huang, Junren Li, Siyu Lin, Junting Zhou, Fengqi Cao, Kun Zhou, Rui Ge, Tingting Long, Yuexiang Zhu, Yan Liu, Jie Zheng, Junnian Wei, Rong Zhu, Peng Zou, Wenyu Li, Zekai Cheng, Tian Ding, Yaxuan Wang, Yizhao Yan, Tingru Wei, Haowei Ming, Weijie Mao, Chen Sun, Yiming Liu, Zichen Wang, Zuo Zhang, Tong Yang, Hao Ma, Zhen Gao, Jian Pei
Current benchmarks for evaluating the chemical reasoning capabilities of Large Language Models (LLMs) are limited by oversimplified tasks, lack of process-level evaluation, and misalignment with expert-level chemistry skills. To address these issues, we introduce SUPERChem, a benchmark of 500 expert-curated reasoning-intensive chemistry problems, covering diverse subfields and provided in both multimodal and text-only formats. Original content and an iterative curation pipeline eliminate flawed items and mitigate data contamination. Each problem is paired with an expert-authored solution path, enabling Reasoning Path Fidelity (RPF) scoring to evaluate reasoning quality beyond final-answer accuracy. Evaluations against a human baseline of 40.3% accuracy show that even the best-performing model, GPT-5 (High), reaches only 38.5%, followed closely by Gemini 2.5 Pro (37.9%) and DeepSeek-V3.1-Think (37.3%). SUPERChem elicits multi-step, multimodal reasoning, reveals model-dependent effects of visual information, and distinguishes high-fidelity reasoners from heuristic ones. By providing a challenging benchmark and a reliable evaluation framework, SUPERChem aims to facilitate the advancement of LLMs toward expert-level chemical intelligence. The dataset of the benchmark is available at https://huggingface.co/datasets/ZehuaZhao/SUPERChem.
comment : 35 pages, 11 figures, 5 tables
☆ Sentiment Analysis and Emotion Classification using Machine Learning Techniques for Nagamese Language - A Low-resource Language
Ekha Morang, Surhoni A. Ngullie, Sashienla Longkumer, Teisovi Angami
The Nagamese language, a.k.a Naga Pidgin, is an Assamese-lexified creole language developed primarily as a means of communication in trade between the people from Nagaland and people from Assam in the north-east India. Substantial amount of work in sentiment analysis has been done for resource-rich languages like English, Hindi, etc. However, no work has been done in Nagamese language. To the best of our knowledge, this is the first attempt on sentiment analysis and emotion classification for the Nagamese Language. The aim of this work is to detect sentiments in terms of polarity (positive, negative and neutral) and basic emotions contained in textual content of Nagamese language. We build sentiment polarity lexicon of 1,195 nagamese words and use these to build features along with additional features for supervised machine learning techniques using Na"ive Bayes and Support Vector Machines. Keywords: Nagamese, NLP, sentiment analysis, machine learning
comment : 10 pages
☆ Large Language Models Cannot Reliably Detect Vulnerabilities in JavaScript: The First Systematic Benchmark and Evaluation
Qingyuan Fei, Xin Liu, Song Li, Shujiang Wu, Jianwei Hou, Ping Chen, Zifeng Kang
Researchers have proposed numerous methods to detect vulnerabilities in JavaScript, especially those assisted by Large Language Models (LLMs). However, the actual capability of LLMs in JavaScript vulnerability detection remains questionable, necessitating systematic evaluation and comprehensive benchmarks. Unfortunately, existing benchmarks suffer from three critical limitations: (1) incomplete coverage, such as covering a limited subset of CWE types; (2) underestimation of LLM capabilities caused by unreasonable ground truth labeling; and (3) overestimation due to unrealistic cases such as using isolated vulnerable files rather than complete projects. In this paper, we introduce, for the first time, three principles for constructing a benchmark for JavaScript vulnerability detection that directly address these limitations: (1) comprehensiveness, (2) no underestimation, and (3) no overestimation. Guided by these principles, we propose FORGEJS, the first automatic benchmark generation framework for evaluating LLMs' capability in JavaScript vulnerability detection. Then, we use FORGEJS to construct ARENAJS-the first systematic benchmark for LLM-based JavaScript vulnerability detection-and further propose JUDGEJS, an automatic evaluation framework. We conduct the first systematic evaluation of LLMs for JavaScript vulnerability detection, leveraging JUDGEJS to assess seven popular commercial LLMs on ARENAJS. The results show that LLMs not only exhibit limited reasoning capabilities, but also suffer from severe robustness defects, indicating that reliable JavaScript vulnerability detection with LLMs remains an open challenge.
☆ Generative Adversarial Gumbel MCTS for Abstract Visual Composition Generation
Zirui Zhao, Boye Niu, David Hsu, Wee Sun Lee
We study abstract visual composition, in which identity is primarily determined by the spatial configuration and relations among a small set of geometric primitives (e.g., parts, symmetry, topology). They are invariant primarily to texture and photorealistic detail. Composing such structures from fixed components under geometric constraints and vague goal specification (such as text) is non-trivial due to combinatorial placement choices, limited data, and discrete feasibility (overlap-free, allowable orientations), which create a sparse solution manifold ill-suited to purely statistical pixel-space generators. We propose a constraint-guided framework that combines explicit geometric reasoning with neural semantics. An AlphaGo-style search enforces feasibility, while a fine-tuned vision-language model scores semantic alignment as reward signals. Our algorithm uses a policy network as a heuristic in Monte-Carlo Tree Search and fine-tunes the network via search-generated plans. Inspired by the Generative Adversarial Network, we use the generated instances for adversarial reward refinement. Over time, the generation should approach the actual data more closely when the reward model cannot distinguish between generated instances and ground-truth. In the Tangram Assembly task, our approach yields higher validity and semantic fidelity than diffusion and auto-regressive baselines, especially as constraints tighten.
☆ Pay Attention Later: From Vector Space Diffusion to Linearithmic Spectral Phase-Locking
Alper Yıldırım, İbrahim Yücedağ
Standard Transformers suffer from a "Semantic Alignment Tax", a prohibitive optimization cost required to organize a chaotic initialization into a coherent geometric map via local gradient diffusion. We hypothesize that this reliance on diffusive learning creates "Catastrophic Rigidity", rendering models unable to adapt to novel concepts without destroying their pre-trained reasoning capabilities. To isolate this phenomenon, we introduce Iterative Semantic Map Refinement (ISMR), a diagnostic protocol revealing that alignment is a fixed geometric barrier that scaling cannot solve; a 20-layer model overcomes this barrier no faster than a 1-layer model. We introduce the Phase-Resonant Intelligent Spectral Model (PRISM). PRISM encodes semantic identity as resonant frequencies in the complex domain (C^d) and replaces quadratic self-attention with linearithmic O(N log N) Gated Harmonic Convolutions. We validate PRISM on the WMT14 translation task. While the Standard Transformer maintains a slight edge in general competence on static benchmarks (23.88 vs 21.40 BLEU), it fails the "Plasticity-Stability" stress test completely. When injected with novel concepts, the Transformer suffers Catastrophic Forgetting, degrading by -10.55 BLEU points while achieving only 60% acquisition. In contrast, PRISM demonstrates Lossless Plasticity, achieving 96% 5-shot acquisition with negligible degradation (-0.84 BLEU). These results suggest that harmonic representations effectively decouple memory from reasoning, offering a structural solution to the plasticity-stability dilemma in real-time knowledge adaptation.
comment : 12 pages, 5 figures
☆ Conveying Imagistic Thinking in Traditional Chinese Medicine Translation: A Prompt Engineering and LLM-Based Evaluation Framework
Jiatong Han
Traditional Chinese Medicine (TCM) theory is built on imagistic thinking, in which medical principles and diagnostic and therapeutic logic are structured through metaphor and metonymy. However, existing English translations largely rely on literal rendering, making it difficult for target-language readers to reconstruct the underlying conceptual networks and apply them in clinical practice. This study adopted a human-in-the-loop (HITL) framework and selected four passages from the medical canon Huangdi Neijing that are fundamental in theory. Through prompt-based cognitive scaffolding, DeepSeek V3.1 was guided to identify metaphor and metonymy in the source text and convey the theory in translation. In the evaluation stage, ChatGPT 5 Pro and Gemini 2.5 Pro were instructed by prompts to simulate three types of real-world readers. Human translations, baseline model translations, and prompt-adjusted translations were scored by the simulated readers across five cognitive dimensions, followed by structured interviews and Interpretative Phenomenological Analysis (IPA). Results show that the prompt-adjusted LLM translations perform best across all five dimensions, with high cross-model and cross-role consistency. The interview themes reveal differences between human and machine translation, effective strategies for metaphor and metonymy transfer, and readers' cognitive preferences. This study provides a cognitive, efficient, and replicable HITL methodological pathway for the translation of ancient, concept-dense texts such as TCM.
comment : 3 figures
☆ Generalist Large Language Models Outperform Clinical Tools on Medical Benchmark s
Krithik Vishwanath, Mrigayu Ghosh, Anton Alyakin, Daniel Alexander Alber, Yindalon Aphinyanaphongs, Eric Karl Oermann
Specialized clinical AI assistants are rapidly entering medical practice, often framed as safer or more reliable than general-purpose large language models (LLMs). Yet, unlike frontier models, these clinical tools are rarely subjected to independent, quantitative evaluation, creating a critical evidence gap despite their growing influence on diagnosis, triage, and guideline interpretation. We assessed two widely deployed clinical AI systems (OpenEvidence and UpToDate Expert AI) against three state-of-the-art generalist LLMs (GPT-5, Gemini 3 Pro, and Claude Sonnet 4.5) using a 1,000-item mini-benchmark combining MedQA (medical knowledge) and HealthBench (clinician-alignment) tasks. Generalist models consistently outperformed clinical tools, with GPT-5 achieving the highest scores, while OpenEvidence and UpToDate demonstrated deficits in completeness, communication quality, context awareness, and systems-based safety reasoning. These findings reveal that tools marketed for clinical decision support may often lag behind frontier LLMs, underscoring the urgent need for transparent, independent evaluation before deployment in patient-facing workflows.
comment : 17 pages, 4 figures (2 regular, 2 supplemental)
☆ TempPerturb-Eval: On the Joint Effects of Internal Temperature and External Perturbations in RAG Robustness
Yongxin Zhou, Philippe Mulhem, Didier Schwab
The evaluation of Retrieval-Augmented Generation (RAG) systems typically examines retrieval quality and generation parameters like temperature in isolation, overlooking their interaction. This work presents a systematic investigation of how text perturbations (simulating noisy retrieval) interact with temperature settings across multiple LLM runs. We propose a comprehensive RAG Perturbation-Temperature Analysis Framework that subjects retrieved documents to three distinct perturbation types across varying temperature settings. Through extensive experiments on HotpotQA with both open-source and proprietary LLMs, we demonstrate that performance degradation follows distinct patterns: high-temperature settings consistently amplify vulnerability to perturbations, while certain perturbation types exhibit non-linear sensitivity across the temperature range. Our work yields three key contributions: (1) a diagnostic benchmark for assessing RAG robustness, (2) an analytical framework for quantifying perturbation-temperature interactions, and (3) practical guidelines for model selection and parameter tuning under noisy retrieval conditions.
☆ DrawingBench: Evaluating Spatial Reasoning and UI Interaction Capabilities of Large Language Models through Mouse-Based Drawing Tasks AAAI 2026
Hyunjun Kim, Sooyoung Ryu
As agentic AI systems increasingly operate autonomously, establishing trust through verifiable evaluation becomes critical. Yet existing benchmarks lack the transparency and auditability needed to assess whether agents behave reliably. We present DrawingBench, a verification framework for evaluating the trustworthiness of agentic LLMs through spatial reasoning tasks that require generating sequences of low-level GUI actions. Unlike opaque evaluations, DrawingBench provides transparent, rule-based assessment: 8 objective criteria enable reproducible scoring, while action-level inspection allows stakeholders to audit agent behavior. Our framework comprises 250 diverse prompts across 20 categories and 4 difficulty levels, deterministic evaluation metrics, and an external oversight mechanism through multi-turn feedback that enables human control over agent refinement. Evaluating four state-of-the-art LLMs (Claude-4 Sonnet, GPT-4.1, GPT-4.1-mini, Gemini-2.5 Flash) across 1,000 tests, we establish both capabilities and limitations: models achieved 92.8% perfect performance with structured external feedback driving significant improvements (average +3.2%, up to +32.8% for complex scenes), but systematic error patterns emerged in tool state management and long-horizon planning. Notably, specification clarity proved more important than task complexity -- models achieved 100% perfect performance when given explicit, verifiable criteria. These findings demonstrate that transparent evaluation frameworks can establish trust in agentic systems, with external oversight proving more reliable than self-correction for guiding agent behavior. Our open-source framework provides a template for trustworthy agent assessment. Code and data: https://github.com/hyunjun1121/DrawingBench
comment : AAAI 2026 TrustAgent Workshop
☆ CAIRNS: Balancing Readability and Scientific Accuracy in Climate Adaptation Question Answering
Liangji Kong, Aditya Joshi, Sarvnaz Karimi
Climate adaptation strategies are proposed in response to climate change. They are practised in agriculture to sustain food production. These strategies can be found in unstructured data (for example, scientific literature from the Elsevier website) or structured (heterogeneous climate data via government APIs). We present Climate Adaptation question-answering with Improved Readability and Noted Sources (CAIRNS), a framework that enables experts -- farmer advisors -- to obtain credible preliminary answers from complex evidence sources from the web. It enhances readability and citation reliability through a structured ScholarGuide prompt and achieves robust evaluation via a consistency-weighted hybrid evaluator that leverages inter-model agreement with experts. Together, these components enable readable, verifiable, and domain-grounded question-answering without fine-tuning or reinforcement learning. Using a previously reported dataset of expert-curated question-answers, we show that CAIRNS outperforms the baselines on most of the metrics. Our thorough ablation study confirms the results on all metrics. To validate our LLM-based evaluation, we also report an analysis of correlations against human judgment.
comment : Short Paper; Under Review at The WebConf 2026 (single-blind submission)
☆ DETAIL Matters: Measuring the Impact of Prompt Specificity on Reasoning in Large Language Models
Olivia Kim
Prompt design plays a critical role in the reasoning performance of large language models (LLMs), yet the impact of prompt specificity - how detailed or vague a prompt is - remains understudied. This paper introduces DETAIL, a framework for evaluating LLM performance across varying levels of prompt specificity. We generate multi-level prompts using GPT-4, quantify specificity via perplexity, and assess correctness using GPT-based semantic equivalence. Experiments on 30 novel reasoning tasks across GPT-4 and O3-mini reveal that specificity improves accuracy, especially for smaller models and procedural tasks. Our results highlight the need for adaptive prompting strategies and provide tools and data to support further research.
☆ Lightweight Latent Reasoning for Narrative Tasks
Alexander Gurung, Nikolay Malkin, Mirella Lapata
Large language models (LLMs) tackle complex tasks by generating long chains of thought or "reasoning traces" that act as latent variables in the generation of an output given a query. A model's ability to generate such traces can be optimized with reinforcement learning (RL) to improve their utility in predicting an answer. This optimization comes at a high computational cost, especially for narrative-related tasks that involve retrieving and processing many tokens. To this end, we propose LiteReason, a latent reasoning method that can be interleaved with standard token sampling and easily combined with RL techniques. LiteReason employs a lightweight Reasoning Projector module, trained to produce continuous latent tokens that help the model 'skip' reasoning steps. During RL, the policy model decides when to activate the projector, switching between latent and discrete reasoning as needed. Experimental results on plot hole detection and book chapter generation show that our method outperforms latent reasoning baselines and comes close to matching non-latent RL training, while reducing final reasoning length by 77-92%. Overall, LiteReason guides RL training to a more efficient part of the performance-computation tradeoff curve.
☆ Swivuriso: The South African Next Voices Multilingual Speech Dataset
Vukosi Marivatee, Kayode Olaleye, Sitwala Mundia, Andinda Bakainga, Unarine Netshifhefhe, Mahmooda Milanzie, Tsholofelo Hope Mogale, Thapelo Sindane, Zainab Abdulrasaq, Kesego Mokgosi, Chijioke Okorie, Nia Zion Van Wyk, Graham Morrissey, Dale Dunbar, Francois Smit, Tsosheletso Chidi, Rooweither Mabuya, Andiswa Bukula, Respect Mlambo, Tebogo Macucwa, Idris Abdulmumin, and Seani Rananga
This paper introduces Swivuriso, a 3000-hour multilingual speech dataset developed as part of the African Next Voices project, to support the development and benchmarking of automatic speech recognition (ASR) technologies in seven South African languages. Covering agriculture, healthcare, and general domain topics, Swivuriso addresses significant gaps in existing ASR datasets. We describe the design principles, ethical considerations, and data collection procedures that guided the dataset creation. We present baseline results of training/finetuning ASR models with this data and compare to other ASR datasets for the langauges concerned.
comment : Work in Progress
☆ A Knowledge-Based Language Model: Deducing Grammatical Knowledge in a Multi- Agent Language Acquisition Simulation
David Ph. Shakouri, Crit Cremers, Niels O. Schiller
This paper presents an initial study performed by the MODOMA system. The MODOMA is a computational multi-agent laboratory environment for unsupervised language acquisition experiments such that acquisition is based on the interaction between two language models, an adult and a child agent. Although this framework employs statistical as well as rule-based procedures, the result of language acquisition is a knowledge-based language model, which can be used to generate and parse new utterances of the target language. This system is fully parametrized and researchers can control all aspects of the experiments while the results of language acquisition, that is, the acquired grammatical knowledge, are explicitly represented and can be consulted. Thus, this system introduces novel possibilities for conducting computational language acquisition experiments. The experiments presented by this paper demonstrate that functional and content categories can be acquired and represented by the daughter agent based on training and test data containing different amounts of exemplars generated by the adult agent. Interestingly, similar patterns, which are well-established for human-generated data, are also found for these machine-generated data. As the procedures resulted in the successful acquisition of discrete grammatical categories by the child agent, these experiments substantiate the validity of the MODOMA approach to modelling language acquisition.
comment : 23 pages, 7 figures, 11 tables. Related work: arXiv:2503.18702. This is the peer-reviewed publisher's version, downloadable from: https://www.clinjournal.org/clinj/article/view/193
☆ Story2MIDI: Emotionally Aligned Music Generation from Text
Mohammad Shokri, Alexandra C. Salem, Gabriel Levine, Johanna Devaney, Sarah Ita Levitan
In this paper, we introduce Story2MIDI, a sequence-to-sequence Transformer-based model for generating emotion-aligned music from a given piece of text. To develop this model, we construct the Story2MIDI dataset by merging existing datasets for sentiment analysis from text and emotion classification in music. The resulting dataset contains pairs of text blurbs and music pieces that evoke the same emotions in the reader or listener. Despite the small scale of our dataset and limited computational resources, our results indicate that our model effectively learns emotion-relevant features in music and incorporates them into its generation process, producing samples with diverse emotional responses. We evaluate the generated outputs using objective musical metrics and a human listening study, confirming the model's ability to capture intended emotional cues.
comment : 8 pages (6 pages of main text + 2 pages of references and appendices), 4 figures, 1 table. Presented at IEEE Big Data 2025 3rd Workshop on AI Music Generation (AIMG 2025)
☆ Think Before You Prune: Self-Reflective Structured Pruning for Reasoning Language Models
Ziyan Wang, Enmao Diao, Qi Le, Pu Wang, Guanchu Wang, Minwoo Lee, Shu-ping Yeh, Li Yang
Reasoning LLMs (RLMs) such as OpenAI o1, DeepSeek-R1, and Qwen3 deliver strong multi-step reasoning through chain-of-thought generation, but their large model sizes and lengthy decode-time outputs make them costly to deploy and unsuitable for resource-constrained settings. To reduce computing and memory cost, pruning offers a promising solution by removing unimportant parameters. However, despite their success on standard LLMs, existing pruning methods severely damage RLMs, as even moderate sparsity (e.g., 20%) can collapse accuracy and completely disrupt the model's reasoning coherence. We begin by analyzing why existing pruning pipelines fail on reasoning LLMs and find that their brittleness largely stems from a mismatch between the calibration data, the pruning objective, and the model's decode-time reasoning behavior. Our study further shows that the most reliable calibration signal comes not from human-written labels but from the model's own self-generated reasoning traces, which more accurately reflect its inference distribution. Guided by these insights, we introduce RESP, a self-reflective structured pruning framework that aligns pruning decisions with the model's reasoning dynamics through self-generated calibration, decode-only gradient-based importance estimation, and progressive regeneration that maintains calibration fidelity as sparsity increases. Experiments on Qwen3-8B demonstrate that RESP markedly outperforms existing structured pruning methods on both GSM8K and MathQA, preserving near-dense accuracy at 20-30% sparsity and substantially mitigating performance collapse at higher sparsity levels. At 40% sparsity, RESP attains 81.3% accuracy on GSM8K and 59.6% on MathQA, surpassing the strongest baselines by 66.87% and 47%, respectively.
comment : 7 pages, 3 figures
☆ Factor(T,U): Factored Cognition Strengthens Monitoring of Untrusted AI
Aaron Sandoval, Cody Rushing
The field of AI Control seeks to develop robust control protocols, deployment safeguards for untrusted AI which may be intentionally subversive. However, existing protocols that rely on weaker monitors to detect unsafe behavior often fail on complex tasks beyond the monitor's comprehension. We develop control protocols based on factored cognition, in which a trusted model decomposes a task, an untrusted model solves each resultant child task in isolation, and the results are reassembled into a full solution. These protocols may improve safety by several means, such as by simplifying the context for monitors, or by obscuring vulnerabilities in the environment. We implement our protocols in the APPS coding setting and red team them against backdoor attempts from an adversarial GPT-4.1 Nano. We find that: (i) Adding factored cognition to a trusted monitoring protocol can boost safety from 41% to 63%; (ii) Safety improves because monitor performance improves; (iii) Factored cognition makes it no harder for capable LLMs to write backdoors in APPS. While our protocols show low usefulness in APPS, they hold promise for more complex tasks.
comment : 7 pages body; 19 pages total; 8 figures
☆ Feature Selection Empowered BERT for Detection of Hate Speech with Vocabulary Augmentation
Pritish N. Desai, Tanay Kewalramani, Srimanta Mandal
Abusive speech on social media poses a persistent and evolving challenge, driven by the continuous emergence of novel slang and obfuscated terms designed to circumvent detection systems. In this work, we present a data efficient strategy for fine tuning BERT on hate speech classification by significantly reducing training set size without compromising performance. Our approach employs a TF IDF-based sample selection mechanism to retain only the most informative 75 percent of examples, thereby minimizing training overhead. To address the limitations of BERT's native vocabulary in capturing evolving hate speech terminology, we augment the tokenizer with domain-specific slang and lexical variants commonly found in abusive contexts. Experimental results on a widely used hate speech dataset demonstrate that our method achieves competitive performance while improving computational efficiency, highlighting its potential for scalable and adaptive abusive content moderation.
☆ Do Large Language Models Walk Their Talk? Measuring the Gap Between Implicit Associations, Self-Report, and Behavioral Altruism
Sandro Andric
We investigate whether Large Language Models (LLMs) exhibit altruistic tendencies, and critically, whether their implicit associations and self-reports predict actual altruistic behavior. Using a multi-method approach inspired by human social psychology, we tested 24 frontier LLMs across three paradigms: (1) an Implicit Association Test (IAT) measuring implicit altruism bias, (2) a forced binary choice task measuring behavioral altruism, and (3) a self-assessment scale measuring explicit altruism beliefs. Our key findings are: (1) All models show strong implicit pro-altruism bias (mean IAT = 0.87, p < .0001), confirming models "know" altruism is good. (2) Models behave more altruistically than chance (65.6% vs. 50%, p < .0001), but with substantial variation (48-85%). (3) Implicit associations do not predict behavior (r = .22, p = .29). (4) Most critically, models systematically overestimate their own altruism, claiming 77.5% altruism while acting at 65.6% (p < .0001, Cohen's d = 1.08). This "virtue signaling gap" affects 75% of models tested. Based on these findings, we recommend the Calibration Gap (the discrepancy between self-reported and behavioral values) as a standardized alignment metric. Well-calibrated models are more predictable and behaviorally consistent; only 12.5% of models achieve the ideal combination of high prosocial behavior and accurate self-knowledge.
comment : 14 pages, 7 figures, 7 tables. Code and data available at https://github.com/sandroandric/LLMs_Altruism_Study_Code
♻ ☆ The AI Productivity Index (APEX)
Bertie Vidgen, Abby Fennelly, Evan Pinnix, Julien Bencheck, Daniyal Khan, Zach Richards, Austin Bridges, Calix Huang, Ben Hunsberger, Isaac Robinson, Akul Datta, Chirag Mahapatra, Dominic Barton, Cass R. Sunstein, Eric Topol, Brendan Foody, Osvald Nitski
We present an extended version of the AI Productivity Index (APEX-v1-extended), a benchmark for assessing whether frontier models are capable of performing economically valuable tasks in four jobs: investment banking associate, management consultant, big law associate, and primary care physician (MD). This technical report details the extensions to APEX-v1, including an increase in the held-out evaluation set from n = 50 to n = 100 cases per job (n = 400 total) and updates to the grading methodology. We present a new leaderboard, where GPT5 (Thinking = High) remains the top performing model with a score of 67.0%. APEX-v1-extended shows that frontier models still have substantial limitations when performing typical professional tasks. To support further research, we are open sourcing n = 25 non-benchmark example cases per role (n = 100 total) along with our evaluation harness.
♻ ☆ SpikingBrain: Spiking Brain-inspired Large Models
Yuqi Pan, Yupeng Feng, Jinghao Zhuang, Siyu Ding, Han Xu, Zehao Liu, Bohan Sun, Yuhong Chou, Xuerui Qiu, Anlin Deng, Anjie Hu, Shurong Wang, Peng Zhou, Man Yao, Jibin Wu, Jian Yang, Guoliang Sun, Bo Xu, Guoqi Li
Mainstream Transformer-based large language models face major efficiency bottlenecks: training computation scales quadratically with sequence length, and inference memory grows linearly, limiting long-context processing. Building large models on non-NVIDIA platforms also poses challenges for stable and efficient training. To address this, we introduce SpikingBrain, a family of brain-inspired models designed for efficient long-context training and inference. SpikingBrain leverages the MetaX GPU cluster and focuses on three aspects: (1) Model Architecture: linear and hybrid-linear attention architectures with adaptive spiking neurons; (2) Algorithmic Optimizations: an efficient, conversion-based training pipeline and a dedicated spike coding framework; (3) System Engineering: customized training frameworks, operator libraries, and parallelism strategies tailored to MetaX hardware. Using these techniques, we develop two models: SpikingBrain-7B, a linear LLM, and SpikingBrain-76B, a hybrid-linear MoE LLM. These models demonstrate the feasibility of large-scale LLM development on non-NVIDIA platforms, and training remains stable for weeks on hundreds of MetaX GPUs with Model FLOPs Utilization at expected levels. SpikingBrain achieves performance comparable to open-source Transformer baselines while using only about 150B tokens for continual pre-training. Our models also significantly improve long-context efficiency and deliver inference with (partially) constant memory and event-driven spiking behavior. For example, SpikingBrain-7B attains over 100x speedup in Time to First Token for 4M-token sequences. Furthermore, the proposed spiking scheme achieves 69.15 percent sparsity, enabling low-power operation. Overall, this work demonstrates the potential of brain-inspired mechanisms to drive the next generation of efficient and scalable large model design.
♻ ☆ Med-gte-hybrid: A context ual embedding transformer model for extracting actionable information from clinical texts
Aditya Kumar, Simon Rauch, Mario Cypko, Oliver Amft
We introduce a novel contextual embedding model med-gte-hybrid that was derived from the gte-large sentence transformer to extract information from unstructured clinical narratives. Our model tuning strategy for med-gte-hybrid combines contrastive learning and a denoising autoencoder. To evaluate the performance of med-gte-hybrid, we investigate several clinical prediction tasks in large patient cohorts extracted from the MIMIC-IV dataset, including Chronic Kidney Disease (CKD) patient prognosis, estimated glomerular filtration rate (eGFR) prediction, and patient mortality prediction. Furthermore, we demonstrate that the med-gte-hybrid model improves patient stratification, clustering, and text retrieval, thus outperforms current state-of-the-art models on the Massive Text Embedding Benchmark (MTEB). While some of our evaluations focus on CKD, our hybrid tuning of sentence transformers could be transferred to other medical domains and has the potential to improve clinical decision-making and personalised treatment pathways in various healthcare applications.
comment : 22 pages, 4 figures, 2 tables
♻ ☆ Reliable Reasoning Beyond Natural Language
Nasim Borazjanizadeh, Steven T. Piantadosi
Despite their linguistic competence, Large Language Models (LLMs) often struggle to reason reliably and flexibly. To identify these shortcomings, we introduce the Non-Linear Reasoning (NLR) dataset, a collection of 55 unique, hand-designed problems that target reasoning bottlenecks arising from the sequential prediction paradigm of LLMs and the inherently linear nature of natural language. NLR tasks require iterative updates, backtracking, and reasoning across multiple parallel chains of thought but only basic arithmetic to solve. To address these limitations, we propose a neurosymbolic reasoning approach that integrates Prolog, a symbolic reasoning engine, into the inference pipeline of LLMs. This division of labor shifts the LLM's task from iterative computations to inferring all information, explicit or implied through common sense, and encoding it as logical code. Our method yields large and robust performance gains across the GSM8k and BIG-bench Navigate benchmarks and achieves near-perfect accuracy on NLR problems, maintaining robustness even as variable interdependence - the number of other variables on which the value of a single variable depends - increases.
♻ ☆ LLMs can hide text in other text of the same length
Antonio Norelli, Michael Bronstein
A meaningful text can be hidden inside another, completely different yet still coherent and plausible, text of the same length. For example, a tweet containing a harsh political critique could be embedded in a tweet that celebrates the same political leader, or an ordinary product review could conceal a secret manuscript. This uncanny state of affairs is now possible thanks to Large Language Models, and in this paper we present Calgacus, a simple and efficient protocol to achieve it. We show that even modest 8-billion-parameter open-source LLMs are sufficient to obtain high-quality results, and a message as long as this abstract can be encoded and decoded locally on a laptop in seconds. The existence of such a protocol demonstrates a radical decoupling of text from authorial intent, further eroding trust in written communication, already shaken by the rise of LLM chatbots. We illustrate this with a concrete scenario: a company could covertly deploy an unfiltered LLM by encoding its answers within the compliant responses of a safe model. This possibility raises urgent questions for AI safety and challenges our understanding of what it means for a Large Language Model to know something.
comment : 21 pages, main paper 9 pages
♻ ☆ Influence Functions for Efficient Data Selection in Reasoning
Prateek Humane, Paolo Cudrano, Daniel Z. Kaplan, Matteo Matteucci, Supriyo Chakraborty, Irina Rish
Fine-tuning large language models (LLMs) on chain-of-thought (CoT) data shows that a small amount of high-quality data can outperform massive datasets. Yet, what constitutes "quality" remains ill-defined. Existing reasoning methods rely on indirect heuristics such as problem difficulty or trace length, while instruction-tuning has explored a broader range of automated selection strategies, but rarely in the context of reasoning. We propose to define reasoning data quality using influence functions, which measure the causal effect of individual CoT examples on downstream accuracy, and introduce influence-based pruning, which consistently outperforms perplexity and embedding-based baselines on math reasoning within a model family.
comment : 4 pages, 2 figures; added link to codebase
♻ ☆ From Code Foundation Models to Agent s and Applications: A Practical Guide to Code Intelligence
Jian Yang, Xianglong Liu, Weifeng Lv, Ken Deng, Shawn Guo, Lin Jing, Yizhi Li, Shark Liu, Xianzhen Luo, Yuyu Luo, Changzai Pan, Ensheng Shi, Yingshui Tan, Renshuai Tao, Jiajun Wu, Xianjie Wu, Zhenhe Wu, Daoguang Zan, Chenchen Zhang, Wei Zhang, He Zhu, Terry Yue Zhuo, Kerui Cao, Xianfu Cheng, Jun Dong, Shengjie Fang, Zhiwei Fei, Xiangyuan Guan, Qipeng Guo, Zhiguang Han, Joseph James, Tianqi Luo, Renyuan Li, Yuhang Li, Yiming Liang, Congnan Liu, Jiaheng Liu, Qian Liu, Ruitong Liu, Tyler Loakman, Xiangxin Meng, Chuang Peng, Tianhao Peng, Jiajun Shi, Mingjie Tang, Boyang Wang, Haowen Wang, Yunli Wang, Fanglin Xu, Zihan Xu, Fei Yuan, Ge Zhang, Jiayi Zhang, Xinhao Zhang, Wangchunshu Zhou, Hualei Zhu, King Zhu, Brown Dai, Aishan Liu, Zhoujun Li, Chenghua Lin, Tianyu Liu, Chao Peng, Kai Shen, Libo Qin, Shuangyong Song, Zizheng Zhan, Jiajun Zhang, Jie Zhang, Zhaoxiang Zhang, Bo Zheng
Large language models (LLMs) have fundamentally transformed automated software development by enabling direct translation of natural language descriptions into functional code, driving commercial adoption through tools like Github Copilot (Microsoft), Cursor (Anysphere), Trae (ByteDance), and Claude Code (Anthropic). While the field has evolved dramatically from rule-based systems to Transformer-based architectures, achieving performance improvements from single-digit to over 95\% success rates on benchmarks like HumanEval. In this work, we provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs, systematically examining the complete model life cycle from data curation to post-training through advanced prompting paradigms, code pre-training, supervised fine-tuning, reinforcement learning, and autonomous coding agents. We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder), critically examining the techniques, design decisions, and trade-offs. Further, we articulate the research-practice gap between academic research (e.g., benchmarks and tasks) and real-world deployment (e.g., software-related code tasks), including code correctness, security, contextual awareness of large codebases, and integration with development workflows, and map promising research directions to practical needs. Last, we conduct a series of experiments to provide a comprehensive analysis of code pre-training, supervised fine-tuning, and reinforcement learning, covering scaling law, framework selection, hyperparameter sensitivity, model architectures, and dataset comparisons.
♻ ☆ Adversarial Confusion Attack: Disrupting Multimodal Large Language Models
Jakub Hoscilowicz, Artur Janicki
We introduce the Adversarial Confusion Attack, a new class of threats against multimodal large language models (MLLMs). Unlike jailbreaks or targeted misclassification, the goal is to induce systematic disruption that makes the model generate incoherent or confidently incorrect outputs. Practical applications include embedding such adversarial images into websites to prevent MLLM-powered AI Agents from operating reliably. The proposed attack maximizes next-token entropy using a small ensemble of open-source MLLMs. In the white-box setting, we show that a single adversarial image can disrupt all models in the ensemble, both in the full-image and Adversarial CAPTCHA settings. Despite relying on a basic adversarial technique (PGD), the attack generates perturbations that transfer to both unseen open-source (e.g., Qwen3-VL) and proprietary (e.g., GPT-5.1) models.
♻ ☆ RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-Wo rl d Users SC
Suyu Ye, Haojun Shi, Darren Shih, Hyokun Yun, Tanya Roosta, Tianmin Shu
To achieve successful assistance with long-horizon web-based tasks, AI agents must be able to sequentially follow real-world user instructions over a long period. Unlike existing web-based agent benchmarks, sequential instruction following in the real world poses significant challenges beyond performing a single, clearly defined task. For instance, real-world human instructions can be ambiguous, require different levels of AI assistance, and may evolve over time, reflecting changes in the user's mental state. To address this gap, we introduce RealWebAssist, a novel benchmark designed to evaluate sequential instruction-following in realistic scenarios involving long-horizon interactions with the web, visual GUI grounding, and understanding ambiguous real-world user instructions. RealWebAssist includes a dataset of sequential instructions collected from real-world human users. Each user instructs a web-based assistant to perform a series of tasks on multiple websites. A successful agent must reason about the true intent behind each instruction, keep track of the mental state of the user, understand user-specific routines, and ground the intended tasks to actions on the correct GUI elements. Our experimental results show that state-of-the-art models struggle to understand and ground user instructions, posing critical challenges in following real-world user instructions for long-horizon web assistance.
comment : Project Website: https://scai.cs.jhu.edu/projects/RealWebAssist/ Code: https://github.com/SCAI-JHU/RealWebAssist
♻ ☆ DPRM: A Dual Implicit Process Reward Model in Multi-Hop Question Answering
Xinyi Wang, Yiping Song, Zhiliang Tian, Bo Liu, Tingjin Luo, Minlie Huang
In multi-hop question answering (MHQA) tasks, Chain of Thought (CoT) improves the quality of generation by guiding large language models (LLMs) through multi-step reasoning, and Knowledge Graphs (KGs) reduce hallucinations via semantic matching. Outcome Reward Models (ORMs) provide feedback after generating the final answers but fail to evaluate the process for multi-step reasoning. Traditional Process Reward Models (PRMs) evaluate the reasoning process but require costly human annotations or rollout generation. While implicit PRM is trained only with outcome signals and derives step rewards through reward parameterization without explicit annotations, it is more suitable for multi-step reasoning in MHQA tasks. However, existing implicit PRM has only been explored for plain text scenarios. When adapting to MHQA tasks, it cannot handle the graph structure constraints in KGs and capture the potential inconsistency between CoT and KG paths. To address these limitations, we propose the DPRM (Dual Implicit Process Reward Model). It trains two implicit PRMs for CoT and KG reasoning in MHQA tasks. Both PRMs, namely KG-PRM and CoT-PRM, derive step-level rewards from outcome signals via reward parameterization without additional explicit annotations. Among them, KG-PRM uses preference pairs to learn structural constraints from KGs. DPRM further introduces a consistency constraint between CoT and KG reasoning steps, making the two PRMs mutually verify and collaboratively optimize the reasoning paths. We also provide a theoretical demonstration of the derivation of process rewards. Experimental results show that our method outperforms 13 baselines on multiple datasets with up to 16.6% improvement on Hit@1.
♻ ☆ SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agent s
Changhao Jiang, Jiajun Sun, Yifei Cao, Jiabao Zhuang, Hui Li, Baoyu Fan, Tao Ji, Tao Gui, Qi Zhang
Recently, role-playing agents have emerged as a promising paradigm for achieving personalized interaction and emotional resonance. Existing research primarily focuses on the textual modality, neglecting the critical dimension of speech in realistic interactive scenarios. In particular, there is a lack of systematic evaluation for Speech Role-Playing Agents (SRPAs). To address this gap, we construct SpeechRole-Data, a large-scale, high-quality dataset that comprises 98 diverse roles and 112k speech-based single-turn and multi-turn conversations. Each role demonstrates distinct vocal characteristics, including timbre and prosody, thereby enabling more sophisticated speech role-playing. Furthermore, we propose SpeechRole-Eval, a multidimensional evaluation benchmark that systematically assesses SRPAs performance in key aspects such as fundamental interaction ability, speech expressiveness, and role-playing fidelity. Experimental results reveal the advantages and challenges of both cascaded and end-to-end speech role-playing agents in maintaining vocal style consistency and role coherence. We release all data, code, and baseline models to provide a solid foundation for speech-driven multimodal role-playing research and to foster further developments in this field.
♻ ☆ Eye of Judge ment: Dissecting the Evaluation of Russian-speaking LLMs with POLLUX
Nikita Martynov, Anastasia Mordasheva, Dmitriy Gorbetskiy, Danil Astafurov, Ulyana Isaeva, Elina Basyrova, Sergey Skachkov, Victoria Berestova, Nikolay Ivanov, Valeriia Zanina, Alena Fenogenova
We introduce POLLUX, a comprehensive open-source benchmark designed to evaluate the generative capabilities of large language models (LLMs) in Russian. Our main contribution is a novel evaluation methodology that enhances the interpretability of LLM assessment. For each task type, we define a set of detailed criteria and develop a scoring protocol where models evaluate responses and provide justifications for their ratings. This enables transparent, criteria-driven evaluation beyond traditional resource-consuming, side-by-side human comparisons. POLLUX includes a detailed, fine-grained taxonomy of 35 task types covering diverse generative domains such as code generation, creative writing, and practical assistant use cases, totaling 2,100 manually crafted and professionally authored prompts. Each task is categorized by difficulty (easy/medium/hard), with experts constructing the dataset entirely from scratch. We also release a family of LLM-as-a-Judge (7B and 32B) evaluators trained for nuanced assessment of generative outputs. This approach provides scalable, interpretable evaluation and annotation tools for model development, effectively replacing costly and less precise human judgments.
comment : short version
♻ ☆ Debating Truth: Debate-driven Claim Verification with Multiple Large Language Model Agent s
Haorui He, Yupeng Li, Dacheng Wen, Yang Chen, Reynold Cheng, Donglong Chen, Francis C. M. Lau
Claim verification is essential for digital literacy, yet state-of-the-art single-agent methods often struggle with complex claims that require nuanced analysis of multifaceted online evidence. Inspired by real-world human fact-checking practices, we propose \textbf{DebateCV}, the first debate-driven claim verification framework powered by multiple LLM agents. In DebateCV, two \textit{Debaters} argue opposing stances over multiple rounds to surface subtle errors in single-agent assessments. A decisive \textit{Moderator} is then required to weigh the evidential strength of conflicting arguments to deliver an accurate verdict. Yet zero-shot agents struggle to adjudicate multi-round debates for verifying complex claims, often defaulting to neutral judgements, and no datasets exist for training agents for this role. To bridge this gap, we propose \textbf{Debate-SFT}, a post-training framework that leverages synthetic data to enhance agents' ability to effectively adjudicate debates for claim verification. Results show that our methods surpass state-of-the-art non-debate approaches in both accuracy (across various evidence conditions) and justification quality, which strengthens societal resilience against misinformation and contributes to a more trustworthy online information ecosystem.
♻ ☆ Comprehensive Evaluation on Lexical Normalization: Boundary-Aware Approaches for Unsegmented Languages EMNLP 2025
Shohei Higashiyama, Masao Utiyama
Lexical normalization research has sought to tackle the challenge of processing informal expressions in user-generated text, yet the absence of comprehensive evaluations leaves it unclear which methods excel across multiple perspectives. Focusing on unsegmented languages, we make three key contributions: (1) creating a large-scale, multi-domain Japanese normalization dataset, (2) developing normalization methods based on state-of-the-art pretrained models, and (3) conducting experiments across multiple evaluation perspectives. Our experiments show that both encoder-only and decoder-only approaches achieve promising results in both accuracy and efficiency.
comment : EMNLP 2025 (Findings), 26 pages
♻ ☆ CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluation AAAI 2026
Yexing Du, Kaiyuan Liu, Youcheng Pan, Zheng Chu, Bo Yang, Xiaocheng Feng, Ming Liu, Yang Xiang
As Large Language Models (LLMs) are increasingly popularized in the multilingual world, ensuring hallucination-free factuality becomes markedly crucial. However, existing benchmarks for evaluating the reliability of Multimodal Large Language Models (MLLMs) predominantly focus on textual or visual modalities with a primary emphasis on English, which creates a gap in evaluation when processing multilingual input, especially in speech. To bridge this gap, we propose a novel Cross-lingual and Cross-modal Factuality benchmark (CCFQA). Specifically, the CCFQA benchmark contains parallel speech-text factual questions across 8 languages, designed to systematically evaluate MLLMs' cross-lingual and cross-modal factuality capabilities. Our experimental results demonstrate that current MLLMs still face substantial challenges on the CCFQA benchmark. Furthermore, we propose a few-shot transfer learning strategy that effectively transfers the Question Answering (QA) capabilities of LLMs in English to multilingual Spoken Question Answering (SQA) tasks, achieving competitive performance with GPT-4o-mini-Audio using just 5-shot training. We release CCFQA as a foundational research resource to promote the development of MLLMs with more robust and reliable speech understanding capabilities. Our code and dataset are available at https://github.com/yxduir/ccfqa.
comment : Accepted in AAAI 2026
♻ ☆ Soft Adaptive Policy Optimization
Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, Junyang Lin
Reinforcement learning (RL) plays an increasingly important role in enhancing the reasoning capabilities of large language models (LLMs), yet stable and performant policy optimization remains challenging. Token-level importance ratios often exhibit high variance-a phenomenon exacerbated in Mixture-of-Experts models-leading to unstable updates. Existing group-based policy optimization methods, such as GSPO and GRPO, alleviate this problem via hard clipping, making it difficult to maintain both stability and effective learning. We propose Soft Adaptive Policy Optimization (SAPO), which replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful learning signals. Compared with GSPO and GRPO, SAPO is both sequence-coherent and token-adaptive. Like GSPO, SAPO maintains sequence-level coherence, but its soft gating forms a continuous trust region that avoids the brittle hard clipping band used in GSPO. When a sequence contains a few highly off-policy tokens, GSPO suppresses all gradients for that sequence, whereas SAPO selectively down-weights only the offending tokens and preserves the learning signal from the near-on-policy ones, improving sample efficiency. Relative to GRPO, SAPO replaces hard token-level clipping with smooth, temperature-controlled scaling, enabling more informative and stable updates. Empirical results on mathematical reasoning benchmarks indicate that SAPO exhibits improved training stability and higher Pass@1 performance under comparable training budgets. Moreover, we employ SAPO to train the Qwen3-VL model series, demonstrating that SAPO yields consistent performance gains across diverse tasks and different model sizes. Overall, SAPO provides a more reliable, scalable, and effective optimization strategy for RL training of LLMs.
♻ ☆ Human Decision-making is Susceptible to AI-driven Manipulation
Sahand Sabour, June M. Liu, Siyang Liu, Chris Z. Yao, Shiyao Cui, Xuanming Zhang, Wen Zhang, Yaru Cao, Advait Bhat, Jian Guan, Wei Wu, Rada Mihalcea, Hongning Wang, Tim Althoff, Tatia M. C. Lee, Minlie Huang
AI systems are increasingly intertwined with daily life, assisting users with various tasks and guiding decision-making. This integration introduces risks of AI-driven manipulation, where such systems may exploit users' cognitive biases and emotional vulnerabilities to steer them toward harmful outcomes. Through a randomized between-subjects experiment with 233 participants, we examined human susceptibility to such manipulation in financial (e.g., purchases) and emotional (e.g., conflict resolution) decision-making contexts. Participants interacted with one of three AI agents: a neutral agent (NA) optimizing for user benefit without explicit influence, a manipulative agent (MA) designed to covertly influence beliefs and behaviors, or a strategy-enhanced manipulative agent (SEMA) equipped with established psychological tactics, allowing it to select and apply them adaptively during interactions to reach its hidden objectives. By analyzing participants' preference ratings, we found significant susceptibility to AI-driven manipulation. Particularly across both decision-making domains, interacting with the manipulative agents significantly increased the odds of rating hidden incentives higher than optimal options (Financial, MA: OR=5.24, SEMA: OR=7.96; Emotional, MA: OR=5.52, SEMA: OR=5.71) compared to the NA group. Notably, we found no clear evidence that employing psychological strategies (SEMA) was overall more effective than simple manipulative objectives (MA) on our primary outcomes. Hence, AI-driven manipulation could become widespread even without requiring sophisticated tactics and expertise. While our findings are preliminary and derived from hypothetical, low-stakes scenarios, we highlight a critical vulnerability in human-AI interactions, emphasizing the need for ethical safeguards and regulatory frameworks to protect human autonomy.
comment : Work in progress
♻ ☆ DESIGNER: Design-Logic-Guided Multidisciplinary Data Synthesis for LLM Reasoning
Weize Liu, Yongchi Zhao, Yijia Luo, Mingyu Xu, Jiaheng Liu, Yanan Li, Xiguo Hu, Zhiqi Bai, Yuchi Xu, Wenbo Su, Bo Zheng
Large language models (LLMs) have achieved remarkable success in many natural language tasks but still struggle with complex, multi-step reasoning, particularly across diverse disciplines. Existing reasoning datasets often lack disciplinary breadth, reasoning depth, and diversity, as well as guiding principles for question synthesis. We propose DESIGNER: a DESIGN-logic-guidEd Reasoning data synthesis pipeline that leverages naturally available, extensive raw documents (e.g., book corpus and web corpus) to generate multidisciplinary challenging questions. We introduce the concept of "design logic" and instruct LLMs to mimic human educators' question-creation process, enabling the automated synthesis of large-scale, high-difficulty questions. We use LLMs to reverse-engineer and abstract over 120,000 design logics from existing questions across various disciplines. By matching these design logics with source documents, we are able to generate reasoning questions with controllable question types and difficulty levels. Using this pipeline, we synthesized two large-scale reasoning datasets that span 75 disciplines: DLR-Book (3.04 million questions from the book corpus) and DLR-Web (1.66 million questions from the web corpus). Data analysis indicates that the questions synthesized by our method exhibit greater difficulty and diversity compared to those in the baseline datasets. We validate our synthesized data through supervised fine-tuning (SFT) on the Qwen3 and Llama3 model families. Our data substantially enhances their multidisciplinary reasoning capabilities, outperforming existing datasets. Notably, by applying SFT on the base versions of these models using only our data, we even surpass their official final models that have undergone the full post-training process.
♻ ☆ A Method for Handling Negative Similarities in Explainable Graph Spectral Clustering of Text Document s -- Extended Version CCS
Mieczysław A. Kłopotek, Sławomir T. Wierzchoń, Bartłomiej Starosta, Dariusz Czerski, Piotr Borkowski
This paper investigates the problem of Graph Spectral Clustering with negative similarities, resulting from document embeddings different from the traditional Term Vector Space (like doc2vec, GloVe, etc.). Solutions for combinatorial Laplacians and normalized Laplacians are discussed. An experimental investigation shows the advantages and disadvantages of 6 different solutions proposed in the literature and in this research. The research demonstrates that GloVe embeddings frequently cause failures of normalized Laplacian based GSC due to negative similarities. Furthermore, application of methods curing similarity negativity leads to accuracy improvement for both combinatorial and normalized Laplacian based GSC. It also leads to applicability for GloVe embeddings of explanation methods developed originally bythe authors for Term Vector Space embeddings.
comment : 1 figure, 17 pages, this is an extended version of a paper accepted for the 25th International Conference on Computational Science (ICCS), 7-9 July 2025
♻ ☆ MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks
Artem Chervyakov, Alexander Kharitonov, Pavel Zadorozhny, Adamenko Pavel, Rodion Levichev, Dmitrii Vorobev, Dmitrii Salikhov, Aidar Valeev, Alena Pestova, Maria Dziuba, Ilseyar Alimova, Artem Zavgorodnev, Aleksandr Medvedev, Stanislav Moiseev, Elena Bruches, Daniil Grebenkin, Roman Derunets, Vikulov Vladimir, Anton Emelyanov, Dmitrii Babaev, Vladimir V. Ivanov, Valentin Malykh, Alena Fenogenova
Advancements in LLMs have enhanced task automation in software engineering; however, current evaluations primarily focus on natural language tasks, overlooking code quality. Most benchmarks prioritize high-level reasoning over executable code and real-world performance, leaving gaps in understanding true capabilities and risks associated with these models in production. To address this issue, we propose MERA Code, a new addition to the MERA benchmark family, specifically focused on evaluating code for the latest code generation LLMs in Russian. This benchmark includes 11 evaluation tasks that span 8 programming languages. Our proposed evaluation methodology features a taxonomy that outlines the practical coding skills necessary for models to complete these tasks. The benchmark comprises an open-source codebase for users to conduct MERA assessments, a scoring system compatible with various programming environments, and a platform featuring a leaderboard and submission system. We evaluate open LLMs and frontier API models, analyzing their limitations in terms of practical coding tasks in non-English languages. We are publicly releasing MERA to guide future research, anticipate groundbreaking features in model development, and standardize evaluation procedures.
♻ ☆ A Machine Learning Approach for Detection of Mental Health Conditions and Cyberbullying from Social Media AAAI-26
Edward Ajayi, Martha Kachweka, Mawuli Deku, Emily Aiken
Mental health challenges and cyberbullying are increasingly prevalent in digital spaces, necessitating scalable and interpretable detection systems. This paper introduces a unified multiclass classification framework for detecting ten distinct mental health and cyberbullying categories from social media data. We curate datasets from Twitter and Reddit, implementing a rigorous "split-then-balance" pipeline to train on balanced data while evaluating on a realistic, held-out imbalanced test set. We conducted a comprehensive evaluation comparing traditional lexical models, hybrid approaches, and several end-to-end fine-tuned transformers. Our results demonstrate that end-to-end fine-tuning is critical for performance, with the domain-adapted MentalBERT emerging as the top model, achieving an accuracy of 0.92 and a Macro F1 score of 0.76, surpassing both its generic counterpart and a zero-shot LLM baseline. Grounded in a comprehensive ethical analysis, we frame the system as a human-in-the-loop screening aid, not a diagnostic tool. To support this, we introduce a hybrid SHAPLLM explainability framework and present a prototype dashboard ("Social Media Screener") designed to integrate model predictions and their explanations into a practical workflow for moderators. Our work provides a robust baseline, highlighting future needs for multi-label, clinically-validated datasets at the critical intersection of online safety and computational mental health.
comment : Accepted for Oral Presentation at the AAAI-26 Bridge Program on AI for Medicine and Healthcare (AIMedHealth). To appear in Proceedings of Machine Learning Research (PMLR)
♻ ☆ Recursive numeral systems are highly regular and easy to process
Ponrawee Prasertsom, Andrea Silvi, Jennifer Culbertson, Moa Johansson, Devdatt Dubhashi, Kenny Smith
Previous work has argued that recursive numeral systems optimise the trade-off between lexicon size and average morphosyntatic complexity (Denić and Szymanik, 2024). However, showing that only natural-language-like systems optimise this tradeoff has proven elusive, and the existing solution has relied on ad-hoc constraints to rule out unnatural systems (Yang and Regier, 2025). Here, we argue that this issue arises because the proposed trade-off has neglected regularity, a crucial aspect of complexity central to human grammars in general. Drawing on the Minimum Description Length (MDL) approach, we propose that recursive numeral systems are better viewed as efficient with regard to their regularity and processing complexity. We show that our MDL-based measures of regularity and processing complexity better capture the key differences between attested, natural systems and unattested but possible ones, including "optimal" recursive numeral systems from previous work, and that the ad-hoc constraints from previous literature naturally follow from regularity. Our approach highlights the need to incorporate regularity across sets of forms in studies that attempt to measure and explain optimality in language.
♻ ☆ BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotations
Simone Giovannini, Fabio Coppini, Andrea Gemelli, Simone Marinai
We present a unified dataset for document Question-Answering (QA), which is obtained combining several public datasets related to Document AI and visually rich document understanding (VRDU). Our main contribution is twofold: on the one hand we reformulate existing Document AI tasks, such as Information Extraction (IE), into a Question-Answering task, making it a suitable resource for training and evaluating Large Language Models; on the other hand, we release the OCR of all the documents and include the exact position of the answer to be found in the document image as a bounding box. Using this dataset, we explore the impact of different prompting techniques (that might include bounding box information) on the performance of open-weight models, identifying the most effective approaches for document comprehension.
♻ ☆ PARROT: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs
Yusuf Çelebi, Özay Ezerceli, Mahmoud El Hussieni
This study presents PARROT (Persuasion and Agreement Robustness Rating of Output Truth), a robustness focused framework designed to measure the degradation in accuracy that occurs under social pressure exerted on users through authority and persuasion in large language models (LLMs) the phenomenon of sycophancy (excessive conformity). PARROT (i) isolates causal effects by comparing the neutral version of the same question with an authoritatively false version using a double-blind evaluation, (ii) quantifies confidence shifts toward the correct and imposed false responses using log-likelihood-based calibration tracking, and (iii) systematically classifies failure modes (e.g., robust correct, sycophantic agreement, reinforced error, stubborn error, self-correction, etc.) using an eight-state behavioral taxonomy. We evaluated 22 models using 1,302 MMLU-style multiple-choice questions across 13 domains and domain-specific authority templates. Findings show marked heterogeneity: advanced models (e.g., GPT-5, GPT-4.1, Claude Sonnet 4.5) exhibit low "follow rates" ($\leq 11\%$, GPT-5: 4\%) and minimal accuracy loss, while older/smaller models show severe epistemic collapse (GPT-4: 80\%, Qwen 2.5-1.5B: 94\%). The danger is not limited to response changes; weak models reduce confidence in the correct response while increasing confidence in the imposed incorrect response. While international law and global knowledge at the domain level exhibit high fragility, elementary mathematics is relatively resilient. Consequently, we argue that the goal of "resistance to overfitting pressure" should be addressed as a primary objective alongside accuracy, harm avoidance, and privacy for safe deployment in the real world.
♻ ☆ LLM-based Human Simulations Have Not Yet Been Reliable
Qian Wang, Jiaying Wu, Zichen Jiang, Zhenheng Tang, Bingqiao Luo, Nuo Chen, Wei Chen, Bingsheng He
Large Language Models (LLMs) are increasingly employed for simulating human behaviors across diverse domains. However, our position is that current LLM-based human simulations remain insufficiently reliable, as evidenced by significant discrepancies between their outcomes and authentic human actions. Our investigation begins with a systematic review of LLM-based human simulations in social, economic, policy, and psychological contexts, identifying their common frameworks, recent advances, and persistent limitations. This review reveals that such discrepancies primarily stem from inherent limitations of LLMs and flaws in simulation design, both of which are examined in detail. Building on these insights, we propose a systematic solution framework that emphasizes enriching data foundations, advancing LLM capabilities, and ensuring robust simulation design to enhance reliability. Finally, we introduce a structured algorithm that operationalizes the proposed framework, aiming to guide credible and human-aligned LLM-based simulations. To facilitate further research, we provide a curated list of related literature and resources at https://github.com/Persdre/awesome-llm-human-simulation.
♻ ☆ SpeechIQ: Speech- Agent ic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models ACL 2025
Zhen Wan, Chao-Han Huck Yang, Yahan Yu, Jinchuan Tian, Sheng Li, Ke Hu, Zhehuai Chen, Shinji Watanabe, Fei Cheng, Chenhui Chu, Sadao Kurohashi
We introduce Speech-based Intelligence Quotient (SIQ) as a new form of human cognition-inspired evaluation pipeline for voice understanding large language models, LLM Voice, designed to assess their voice understanding ability. Moving beyond popular voice understanding metrics such as word error rate (WER), SIQ examines LLM Voice across three cognitive levels motivated by Bloom's Taxonomy: (1) Remembering (i.e., WER for verbatim accuracy); (2) Understanding (i.e., similarity of LLM's interpretations); and (3) Application (i.e., QA accuracy for simulating downstream tasks). We demonstrate that SIQ not only quantifies voice understanding abilities but also provides unified comparisons between cascaded methods (e.g., ASR LLM) and end-to-end models, identifies annotation errors in existing benchmarks, and detects hallucinations in LLM Voice. Our framework represents a first-of-its-kind intelligence examination that bridges cognitive principles with voice-oriented benchmarks, while exposing overlooked challenges in multi-modal training. Our code and data will be open source to encourage future studies.
comment : ACL 2025 main. Our Speech-IQ leaderboard is hosted at huggingface.co/spaces/nvidia/Speech-IQ-leaderboard. Speech-IQ Calculator: https://github.com/YukinoWan/SpeechIQ
♻ ☆ From Topology to Retrieval: Decoding Embedding Spaces with Unified Signatures
Florian Rottach, William Rudman, Bastian Rieck, Harrisen Scells, Carsten Eickhoff
Studying how embeddings are organized in space not only enhances model interpretability but also uncovers factors that drive downstream task performance. In this paper, we present a comprehensive analysis of topological and geometric measures across a wide set of text embedding models and datasets. We find a high degree of redundancy among these measures and observe that individual metrics often fail to sufficiently differentiate embedding spaces. Building on these insights, we introduce Unified Topological Signatures (UTS), a holistic framework for characterizing embedding spaces. We show that UTS can predict model-specific properties and reveal similarities driven by model architecture. Further, we demonstrate the utility of our method by linking topological structure to ranking effectiveness and accurately predicting document retrievability. We find that a holistic, multi-attribute perspective is essential to understanding and leveraging the geometry of text embeddings.
♻ ☆ NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Model ACL 2025
Yen-Ting Lin, Zhehuai Chen, Piotr Zelasko, Zhen Wan, Xuesong Yang, Zih-Ching Chen, Krishna C Puvvada, Szu-Wei Fu, Ke Hu, Jun Wei Chiu, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang, Chao-Han Huck Yang
Construction of a general-purpose post-recognition error corrector poses a crucial question: how can we most effectively train a model on a large mixture of domain datasets? The answer would lie in learning dataset-specific features and digesting their knowledge in a single model. Previous methods achieve this by having separate correction language models, resulting in a significant increase in parameters. In this work, we present Mixture-of-Experts as a solution, highlighting that MoEs are much more than a scalability tool. We propose a Multi-Task Correction MoE, where we train the experts to become an ``expert'' of speech-to-text, language-to-text and vision-to-text datasets by learning to route each dataset's tokens to its mapped expert. Experiments on the Open ASR Leaderboard show that we explore a new state-of-the-art performance by achieving an average relative 5.0% WER reduction and substantial improvements in BLEU scores for speech and translation tasks. On zero-shot evaluation, NeKo outperforms GPT-3.5 and Claude-Opus with 15.5% to 27.6% relative WER reduction in the Hyporadise benchmark. NeKo performs competitively on grammar and post-OCR correction as a multi-task model.
comment : ACL 2025 Industry Track. NeKo LMs: https://huggingface.co/nvidia/NeKo-v0-post-correction
♻ ☆ HEALTH-PARIKSHA: Assessing RAG Models for Health Chatbots in Real-Wo rl d Multilingual Settings
Varun Gumma, Ananditha Raghunath, Mohit Jain, Sunayana Sitaram
Assessing the capabilities and limitations of large language models (LLMs) has garnered significant interest, yet the evaluation of multiple models in real-world scenarios remains rare. Multilingual evaluation often relies on translated benchmarks, which typically do not capture linguistic and cultural nuances present in the source language. This study provides an extensive assessment of 24 LLMs on real world data collected from Indian patients interacting with a medical chatbot in Indian English and 4 other Indic languages. We employ a uniform Retrieval Augmented Generation framework to generate responses, which are evaluated using both automated techniques and human evaluators on four specific metrics relevant to our application. We find that models vary significantly in their performance and that instruction tuned Indic models do not always perform well on Indic language queries. Further, we empirically show that factual correctness is generally lower for responses to Indic queries compared to English queries. Finally, our qualitative work shows that code-mixed and culturally relevant queries in our dataset pose challenges to evaluated models.
♻ ☆ Prompt -R1: Collaborative Automatic Prompt ing Framework via End-to-end Reinforcement Learning
Wenjin Liu, Haoran Luo, Xueyuan Lin, Haoming Liu, Tiesunlong Shen, Jiapu Wang, Rui Mao, Erik Cambria
Recently, advanced large language models (LLMs) have emerged at an increasingly rapid pace. However, when faced with complex problems, most users are often unable to provide accurate and effective prompts to interact with LLMs, thus limiting the performance of LLMs. To address this challenge, we propose Prompt-R1, an end-to-end reinforcement learning framework that uses a small-scale LLM to collaborate with large-scale LLMs, replacing user interaction to solve problems better. This collaboration is cast as a multi-turn prompt interaction, where the small-scale LLM thinks and generates prompts, and the large-scale LLM performs complex reasoning. A dual-constrained reward is designed to optimize for correctness, generation quality, and reasoning accuracy. Prompt-R1 provides a plug-and-play framework that supports both inference and training with various large-scale LLMs. Experiments on multiple public datasets show that Prompt-R1 significantly outperforms baseline models across tasks. Our code is publicly available at https://github.com/QwenQKing/Prompt-R1.
♻ ☆ Checklists Are Better Than Reward Model s For Aligning Language Models NeurIPS 2025
Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, Tongshuang Wu
Language models must be adapted to understand and follow user instructions. Reinforcement learning is widely used to facilitate this -- typically using fixed criteria such as "helpfulness" and "harmfulness". In our work, we instead propose using flexible, instruction-specific criteria as a means of broadening the impact that reinforcement learning can have in eliciting instruction following. We propose "Reinforcement Learning from Checklist Feedback" (RLCF). From instructions, we extract checklists and evaluate how well responses satisfy each item - using both AI judges and specialized verifier programs - then combine these scores to compute rewards for RL. We compare RLCF with other alignment methods applied to a strong instruction following model (Qwen2.5-7B-Instruct) on five widely-studied benchmarks -- RLCF is the only method to improve performance on every benchmark, including a 4-point boost in hard satisfaction rate on FollowBench, a 6-point increase on InFoBench, and a 3-point rise in win rate on Arena-Hard. These results establish checklist feedback as a key tool for improving language models' support of queries that express a multitude of needs.
comment : Presented at NeurIPS 2025
♻ ☆ Spark-Prover-X1: Formal Theorem Proving Through Diverse Data Training
Xinyuan Zhou, Yi Lei, Xiaoyu Zhou, Jingyi Sun, Yu Zhu, Zhongyi Ye, Weitai Zhang, Quan Liu, Si Wei, Cong Liu
Large Language Models (LLMs) have shown significant promise in automated theorem proving, yet progress is often constrained by the scarcity of diverse and high-quality formal language data. To address this issue, we introduce Spark-Prover-X1, a 7B parameter model trained via an three-stage framework designed to unlock the reasoning potential of more accessible and moderately-sized LLMs. The first stage infuses deep knowledge through continuous pre-training on a broad mathematical corpus, enhanced by a suite of novel data tasks. Key innovation is a "CoT-augmented state prediction" task to achieve fine-grained reasoning. The second stage employs Supervised Fine-tuning (SFT) within an expert iteration loop to specialize both the Spark-Prover-X1-7B and Spark-Formalizer-X1-7B models. Finally, a targeted round of Group Relative Policy Optimization (GRPO) is applied to sharpen the prover's capabilities on the most challenging problems. To facilitate robust evaluation, particularly on problems from real-world examinations, we also introduce ExamFormal-Bench, a new benchmark dataset of 402 formal problems. Experimental results demonstrate that Spark-Prover achieves state-of-the-art performance among similarly-sized open-source models within the "Whole-Proof Generation" paradigm. It shows exceptional performance on difficult competition benchmarks, notably solving 27 problems on PutnamBench (pass@32) and achieving 24.0\% on CombiBench (pass@32). Our work validates that this diverse training data and progressively refined training pipeline provides an effective path for enhancing the formal reasoning capabilities of lightweight LLMs. We will release both Spark-Prover-X1-7B and Spark-Formalizer-X1-7B, along with the ExamFormal-Bench dataset, in the near future.
♻ ☆ Measuring and Guiding Monosemanticity
Ruben Härle, Felix Friedrich, Manuel Brack, Stephan Wäldchen, Björn Deiseroth, Patrick Schramowski, Kristian Kersting
There is growing interest in leveraging mechanistic interpretability and controllability to better understand and influence the internal dynamics of large language models (LLMs). However, current methods face fundamental challenges in reliably localizing and manipulating feature representations. Sparse Autoencoders (SAEs) have recently emerged as a promising direction for feature extraction at scale, yet they, too, are limited by incomplete feature isolation and unreliable monosemanticity. To systematically quantify these limitations, we introduce Feature Monosemanticity Score (FMS), a novel metric to quantify feature monosemanticity in latent representation. Building on these insights, we propose Guided Sparse Autoencoders (G-SAE), a method that conditions latent representations on labeled concepts during training. We demonstrate that reliable localization and disentanglement of target concepts within the latent space improve interpretability, detection of behavior, and control. Specifically, our evaluations on toxicity detection, writing style identification, and privacy attribute recognition show that G-SAE not only enhances monosemanticity but also enables more effective and fine-grained steering with less quality degradation. Our findings provide actionable guidelines for measuring and advancing mechanistic interpretability and control of LLMs.
♻ ☆ SWE- RL : Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution NeurIPS 2025
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, Sida I. Wang
The recent DeepSeek-R1 release has demonstrated the immense potential of reinforcement learning (RL) in enhancing the general reasoning capabilities of large language models (LLMs). While DeepSeek-R1 and other follow-up work primarily focus on applying RL to competitive coding and math problems, this paper introduces SWE-RL, the first approach to scale RL-based LLM reasoning for real-world software engineering. Leveraging a lightweight rule-based reward (e.g., the similarity score between ground-truth and LLM-generated solutions), SWE-RL enables LLMs to autonomously recover a developer's reasoning processes and solutions by learning from extensive open-source software evolution data -- the record of a software's entire lifecycle, including its code snapshots, code changes, and events such as issues and pull requests. Trained on top of Llama 3, our resulting reasoning model, Llama3-SWE-RL-70B, achieves a 41.0% solve rate on SWE-bench Verified -- a human-verified collection of real-world GitHub issues. To our knowledge, this is the best performance reported for medium-sized (<100B) LLMs to date, even comparable to leading proprietary LLMs like GPT-4o. Surprisingly, despite performing RL solely on software evolution data, Llama3-SWE-RL has even emerged with generalized reasoning skills. For example, it shows improved results on five out-of-domain tasks, namely, function coding, library use, code reasoning, mathematics, and general language understanding, whereas a supervised-finetuning baseline even leads to performance degradation on average. Overall, SWE-RL opens up a new direction to improve the reasoning capabilities of LLMs through reinforcement learning on massive software engineering data.
comment : Accepted to NeurIPS 2025 Main Track
♻ ☆ LLM-based Automated Grading with Human-in-the-Loop
Yucheng Chu, Hang Li, Kaiqi Yang, Yasemin Copur-Gencturk, Jiliang Tang
The rise of artificial intelligence (AI) technologies, particularly large language models (LLMs), has brought significant advancements to the field of education. Among various applications, automatic short answer grading (ASAG), which focuses on evaluating open-ended textual responses, has seen remarkable progress with the introduction of LLMs. These models not only enhance grading performance compared to traditional ASAG approaches but also move beyond simple comparisons with predefined "golden" answers, enabling more sophisticated grading scenarios, such as rubric-based evaluation. However, existing LLM-powered methods still face challenges in achieving human-level grading performance in rubric-based assessments due to their reliance on fully automated approaches. In this work, we explore the potential of LLMs in ASAG tasks by leveraging their interactive capabilities through a human-in-the-loop (HITL) approach. Our proposed framework, GradeHITL, utilizes the generative properties of LLMs to pose questions to human experts, incorporating their insights to refine grading rubrics dynamically. This adaptive process significantly improves grading accuracy, outperforming existing methods and bringing ASAG closer to human-level evaluation.
comment : Accepted to IEEE TALE 2025
♻ ☆ Just-in-time and distributed task representations in language models
Yuxuan Li, Declan Campbell, Stephanie C. Y. Chan, Andrew Kyle Lampinen
Many of language models' impressive capabilities originate from their in-context learning: based on instructions or examples, they can infer and perform new tasks without weight updates. In this work, we investigate when representations for new tasks are formed in language models, and how these representations change over the course of context. We study two different task representations: those that are ''transferrable'' -- vector representations that can transfer task contexts to another model instance, even without the full prompt -- and simpler representations of high-level task categories. We show that transferrable task representations evolve in non-monotonic and sporadic ways, while task identity representations persist throughout the context. Specifically, transferrable task representations exhibit a two-fold locality. They successfully condense evidence when more examples are provided in the context. But this evidence accrual process exhibits strong temporal locality along the sequence dimension, coming online only at certain tokens -- despite task identity being reliably decodable throughout the context. In some cases, transferrable task representations also show semantic locality, capturing a small task ''scope'' such as an independent subtask. Language models thus represent new tasks on the fly through both an inert, sustained sensitivity to the task and an active, just-in-time representation to support inference.
♻ ☆ Bridging Human and LLM Judgments: Understanding and Narrowing the Gap NeurIPS 2025
Felipe Maia Polo, Xinhe Wang, Mikhail Yurochkin, Gongjun Xu, Moulinath Banerjee, Yuekai Sun
Large language models are increasingly used as judges (LLM-as-a-judge) to evaluate model outputs at scale, but their assessments often diverge systematically from human judgments. We present Bridge, a unified statistical framework that explicitly bridges human and LLM evaluations under both absolute scoring and pairwise comparison paradigms. Bridge posits a latent human preference score for each prompt-response pair and models LLM deviations as linear transformations of covariates that capture sources of discrepancies. This offers a simple and principled framework for refining LLM ratings and characterizing systematic discrepancies between humans and LLMs. We provide an efficient fitting algorithm with asymptotic guarantees for statistical inference. Using six LLM judges and two benchmarks (BigGen Bench and Chatbot Arena), Bridge achieves higher agreement with human ratings (accuracy, calibration, and KL divergence) and exposes systematic human-LLM gaps.
comment : NeurIPS 2025
♻ ☆ Unleashing Hour-Scale Video Training for Long Video-Language Understanding NeurIPS 2025
Jingyang Lin, Jialian Wu, Ximeng Sun, Ze Wang, Jiang Liu, Yusheng Su, Xiaodong Yu, Hao Chen, Jiebo Luo, Zicheng Liu, Emad Barsoum
Recent long-form video-language understanding benchmarks have driven progress in video large multimodal models (Video-LMMs). However, the scarcity of well-annotated long videos has left the training of hour-long Video-LMMs underexplored. To close this gap, we present VideoMarathon, a large-scale hour-long video instruction-following dataset. This dataset includes around 9,700 hours of long videos sourced from diverse domains, ranging from 3 to 60 minutes per video. Specifically, it contains 3.3M high-quality QA pairs, spanning six fundamental topics: temporality, spatiality, object, action, scene, and event. Compared to existing video instruction datasets, VideoMarathon significantly extends training video durations up to 1 hour, and supports 22 diverse tasks requiring both short- and long-term video comprehension. Building on VideoMarathon, we propose Hour-LLaVA, a powerful and efficient Video-LMM for hour-scale video-language modeling. It enables hour-long video training and inference at 1-FPS sampling by leveraging a memory augmentation module, which adaptively integrates question-relevant and spatiotemporally informative semantics from the cached full video context. In our experiments, Hour-LLaVA achieves the best performance on multiple representative long video-language benchmarks, demonstrating the high quality of the VideoMarathon dataset and the superiority of the Hour-LLaVA model.
comment : NeurIPS 2025, Project page: https://videomarathon.github.io/
♻ ☆ Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning EMNLP 2025
Jingyang Lin, Andy Wong, Tian Xia, Shenghua He, Hui Wei, Mei Han, Jiebo Luo
Recent advances in Large Language Models (LLMs) have enabled them to process increasingly longer sequences, ranging from 2K to 2M tokens and even beyond. However, simply extending the input sequence length does not necessarily lead to effective long-context understanding. In this study, we integrate Chain-of-Thought (CoT) reasoning into LLMs in a supervised manner to facilitate effective long-context understanding. To achieve this, we introduce LongFinanceQA, a synthetic dataset in the financial domain designed to improve long-context reasoning. Unlike existing long-context synthetic data, LongFinanceQA includes intermediate CoT reasoning before the final conclusion, which encourages LLMs to perform explicit reasoning, improving accuracy and interpretability in long-context understanding. To generate synthetic CoT reasoning, we propose Property-based Agentic Inference (PAI), an agentic framework that simulates human-like reasoning steps, including property extraction, retrieval, and summarization. We evaluate PAI's reasoning capabilities by assessing GPT-4o-mini w/ PAI on the Loong benchmark, outperforming standard GPT-4o-mini by 20.0%. Furthermore, we fine-tune LLaMA-3.1-8B-Instruct on LongFinanceQA, achieving a 28.0% gain on Loong's financial subset.
comment : Main Conference of EMNLP 2025, Project Page: https://long-pai.github.io/
♻ ★ From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
Chen Shani, Liron Soffer, Dan Jurafsky, Yann LeCun , Ravid Shwartz-Ziv
Humans organize knowledge into compact conceptual categories that balance compression with semantic richness. Large Language Models (LLMs) exhibit impressive linguistic abilities, but whether they navigate this same compression-meaning trade-off remains unclear. We apply an Information Bottleneck framework to compare human conceptual structure with embeddings from 40+ LLMs using classic categorization benchmarks. We find that LLMs broadly align with human category boundaries, yet fall short on fine-grained semantic distinctions. Unlike humans, who maintain ``inefficient'' representations that preserve contextual nuance, LLMs aggressively compress, achieving more optimal information-theoretic compression at the cost of semantic richness. Surprisingly, encoder models outperform much larger decoder models in human alignment, suggesting that understanding and generation rely on distinct representational mechanisms. Training-dynamics analysis reveals a two-phase trajectory: rapid initial concept formation followed by architectural reorganization, during which semantic processing migrates from deep to mid-network layers as the model discovers increasingly efficient, sparser encodings. These divergent strategies, where LLMs optimize for compression and humans for adaptive utility, reveal fundamental differences between artificial and natural intelligence. This highlights the need for models that preserve the conceptual ``inefficiencies'' essential for human-like understanding.
♻ ☆ Animating Language Practice: Engagement with Stylized Conversational Agent s in Japanese Learning
Zackary Rackauckas, Julia Hirschberg
We explore Jouzu, a Japanese language learning application that integrates large language models with anime-inspired conversational agents. Designed to address challenges learners face in practicing natural and expressive dialogue, Jouzu combines stylized character personas with expressive text-to-speech to create engaging conversational scenarios. We conducted a two-week in-the-wild deployment with 52 Japanese learners to examine how such stylized agents influence engagement and learner experience. Our findings show that participants interacted frequently and creatively, with advanced learners demonstrating greater use of expressive forms. Participants reported that the anime-inspired style made practice more enjoyable and encouraged experimenting with different registers. We discuss how stylization shapes willingness to engage, the role of affect in sustaining practice, and design opportunities for culturally grounded conversational AI in computer-assisted language learning (CALL). By framing our findings as an exploration of design and engagement, we highlight opportunities for generalization beyond Japanese contexts and contribute to international HCI scholarship.
♻ ☆ OpenGVL -- Benchmark ing Visual Temporal Progress for Data Curation
Paweł Budzianowski, Emilia Wiśnios, Gracjan Góral, Igor Kulakov, Viktor Petrenko, Krzysztof Walas
Data scarcity remains one of the most limiting factors in driving progress in robotics. However, the amount of available robotics data in the wild is growing exponentially, creating new opportunities for large-scale data utilization. Reliable temporal task completion prediction could help automatically annotate and curate this data at scale. The Generative Value Learning (GVL) approach was recently proposed, leveraging the knowledge embedded in vision-language models (VLMs) to predict task progress from visual observations. Building upon GVL, we propose OpenGVL, a comprehensive benchmark for estimating task progress across diverse challenging manipulation tasks involving both robotic and human embodiments. We evaluate the capabilities of publicly available open-source foundation models, showing that open-source model families significantly underperform closed-source counterparts, achieving only approximately $70\%$ of their performance on temporal progress prediction tasks. Furthermore, we demonstrate how OpenGVL can serve as a practical tool for automated data curation and filtering, enabling efficient quality assessment of large-scale robotics datasets. We release the benchmark along with the complete codebase at \href{github.com/budzianowski/opengvl}{OpenGVL}.
comment : Workshop on Making Sense of Data in Robotics: Composition, Curation, and Interpretability at Scale at CoRL 2025
♻ ☆ Reasoning Up the Instruction Ladder for Controllable Language Models
Zishuo Zheng, Vidhisha Balachandran, Chan Young Park, Faeze Brahman, Sachin Kumar
As large language model (LLM) based systems take on high-stakes roles in real-world decision-making, they must reconcile competing instructions from multiple sources (e.g., model developers, users, and tools) within a single prompt context. Thus, enforcing an instruction hierarchy (IH) in LLMs, where higher-level directives override lower-priority requests, is critical for the reliability and controllability of LLMs. In this work, we reframe instruction hierarchy resolution as a reasoning task. Specifically, the model must first "think" about the relationship between a given user prompt and higher-priority (system) instructions before generating a response. To enable this capability via training, we construct VerIH, an instruction hierarchy dataset of constraint-following tasks with verifiable answers. This dataset comprises ~7K aligned and conflicting system-user instructions. We show that lightweight reinforcement learning with VerIH effectively transfers general reasoning capabilities of models to instruction prioritization. Our finetuned models achieve consistent improvements on instruction following and instruction hierarchy benchmarks, achieving roughly a 20% improvement on the IHEval conflict setup. This reasoning ability also generalizes to safety-critical settings beyond the training distribution. By treating safety issues as resolving conflicts between adversarial user inputs and predefined higher-priority policies, our trained model enhances robustness against jailbreak and prompt injection attacks, providing up to a 20% reduction in attack success rate (ASR). These results demonstrate that reasoning over instruction hierarchies provides a practical path to reliable LLMs, where updates to system prompts yield controllable and robust changes in model behavior.
♻ ☆ MAS-ZERO: Designing Multi- Agent Systems with Zero Super vision NeurIPS
Zixuan Ke, Austin Xu, Yifei Ming, Xuan-Phi Nguyen, Ryan Chin, Caiming Xiong, Shafiq Joty
Multi-agent systems (MAS) leveraging the impressive capabilities of Large Language Models (LLMs) hold significant potential for tackling complex tasks. However, most current MAS depend on manually designed agent roles and communication protocols. These manual designs often fail to align with the underlying LLMs' strengths and struggle to adapt to novel tasks. Recent automatic MAS approaches attempt to mitigate these limitations but typically necessitate a validation set for tuning and yield static MAS designs lacking adaptability during inference, while also removing the flexibility to reduce to simpler systems. We introduce MAS-ZERO, the first self-evolved, inference-time framework for automatic MAS design. MAS-ZERO employs meta-level design to iteratively design, critique, and refine MAS configurations tailored to each problem instance, without requiring a validation set. Critically, it enables dynamic problem decomposition and agent composition through meta-feedback on solvability and completeness, and reduction to simpler systems when appropriate. Experiments across reasoning (math and graduate-level QA), coding, and agentic (search-based) benchmarks, using both closed-source and open-source LLM backbones of varying sizes, demonstrate that MAS-ZERO outperforms strong manual and automatic MAS baselines. It achieves substantial average accuracy improvements of up to 16.69% on reasoning, 16.66% on coding, and 5.45% on agentic tasks, while maintaining cost efficiency.
comment : SEA@NeurIPS (Oral) 2025
♻ ☆ Evolution and compression in LLMs: On the emergence of human-aligned categorization NeurIPS 2025
Nathaniel Imel, Noga Zaslavsky
Converging evidence suggests that human systems of semantic categories achieve near-optimal compression via the Information Bottleneck (IB) complexity-accuracy tradeoff. Large language models (LLMs) are not trained for this objective, which raises the question: are LLMs capable of evolving efficient human-aligned semantic systems? To address this question, we focus on color categorization -- a key testbed of cognitive theories of categorization with uniquely rich human data -- and replicate with LLMs two influential human studies. First, we conduct an English color-naming study, showing that LLMs vary widely in their complexity and English-alignment, with larger instruction-tuned models achieving better alignment and IB-efficiency. Second, to test whether these LLMs simply mimic patterns in their training data or actually exhibit a human-like inductive bias toward IB-efficiency, we simulate cultural evolution of pseudo color-naming systems in LLMs via a method we refer to as Iterated in-Context Language Learning (IICLL). We find that akin to humans, LLMs iteratively restructure initially random systems towards greater IB-efficiency. However, only a model with strongest in-context capabilities (Gemini 2.0) is able to recapitulate the wide range of near-optimal IB-tradeoffs observed in humans, while other state-of-the-art models converge to low-complexity solutions. These findings demonstrate how human-aligned semantic categories can emerge in LLMs via the same fundamental principle that underlies semantic efficiency in humans.
comment : Accepted at CogInterp: Interpreting Cognition in Deep Learning Models Workshop at NeurIPS 2025
♻ ☆ Look Before you Leap: Estimating LLM Benchmark Scores from Descriptions
Jungsoo Park, Ethan Mendes, Gabriel Stanovsky, Alan Ritter
Progress in large language models is constrained by an evaluation bottleneck: build a benchmark, run models, then iterate. We ask a question: can we forecast outcomes before running any experiments to inform earlier study design? For example, a team building an AI assistant for a certain task can estimate whether expected performance is around 50 or closer to 80, evidence that supports whether to proceed to a pilot study, how to scope it, and how to allocate resources. We study text-only performance forecasting, where a model predicts a score from a redacted task description and intended configuration, with no access to dataset instances. To support systematic study, we curate PRECOG, a corpus of redacted description-performance pairs spanning diverse tasks, domains, and metrics. We scrape task and configuration descriptions from arXiv, yielding 2,290 instances covering 1,519 papers, and construct a leakage free test split using papers published after the knowledge cutoff of the evaluated models. Experiments show the task is challenging but feasible: reasoning models achieve moderate prediction performance with well calibrated uncertainty, reaching mean absolute error as low as 9.9 at high confidence thresholds. We further test a zero-leakage setting, forecasting on newly released datasets or experiments before their papers are indexed, where GPT5 with built in web search still attains nontrivial prediction accuracy. Overall, our corpus and analyses offer an initial step toward open ended anticipatory evaluation, supporting difficulty estimation and smarter experiment prioritization.
comment : 32 pages, 7 figures
♻ ☆ Apertus: Dem ocr atizing Open and Compliant LLMs for Global Language Environments
Project Apertus, Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, Ido Hakimi, Juan García Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, Inés Altemir Mariñas, Mohammad Hossein Amani, Matin Ansaripour, Ilia Badanin, Harold Benoit, Emanuela Boros, Nicholas Browning, Fabian Bösch, Maximilian Böther, Niklas Canova, Camille Challier, Clement Charmillot, Jonathan Coles, Jan Deriu, Arnout Devos, Lukas Drescher, Daniil Dzenhaliou, Maud Ehrmann, Dongyang Fan, Simin Fan, Silin Gao, Miguel Gila, María Grandury, Diba Hashemi, Alexander Hoyle, Jiaming Jiang, Mark Klein, Andrei Kucharavy, Anastasiia Kucherenko, Frederike Lübeck, Roman Machacek, Theofilos Manitaras, Andreas Marfurt, Kyle Matoba, Simon Matrenok, Henrique Mendonça, Fawzi Roberto Mohamed, Syrielle Montariol, Luca Mouchel, Sven Najem-Meyer, Jingwei Ni, Gennaro Oliva, Matteo Pagliardini, Elia Palme, Andrei Panferov, Léo Paoletti, Marco Passerini, Ivan Pavlov, Auguste Poiroux, Kaustubh Ponkshe, Nathan Ranchin, Javi Rando, Mathieu Sauser, Jakhongir Saydaliev, Muhammad Ali Sayfiddinov, Marian Schneider, Stefano Schuppli, Marco Scialanga, Andrei Semenov, Kumar Shridhar, Raghav Singhal, Anna Sotnikova, Alexander Sternfeld, Ayush Kumar Tarun, Paul Teiletche, Jannis Vamvas, Xiaozhe Yao, Hao Zhao, Alexander Ilic, Ana Klimovic, Andreas Krause, Caglar Gulcehre, David Rosenthal, Elliott Ash, Florian Tramèr, Joost VandeVondele, Livio Veraldi, Martin Rajman, Thomas Schulthess, Torsten Hoefler, Antoine Bosselut, Martin Jaggi, Imanol Schlag
We present Apertus, a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem: data compliance and multilingual representation. Unlike many prior models that release weights without reproducible data pipelines or regard for content-owner rights, Apertus models are pretrained exclusively on openly available data, retroactively respecting `robots.txt` exclusions and filtering for non-permissive, toxic, and personally identifiable content. To mitigate risks of memorization, we adopt the Goldfish objective during pretraining, strongly suppressing verbatim recall of data while retaining downstream task performance. The Apertus models also expand multilingual coverage, training on 15T tokens from over 1800 languages, with ~40% of pretraining data allocated to non-English content. Released at 8B and 70B scales, Apertus approaches state-of-the-art results among fully open models on multilingual benchmarks, rivalling or surpassing open-weight counterparts. Beyond model weights, we release all scientific artifacts from our development cycle with a permissive license, including data preparation scripts, checkpoints, evaluation suites, and training code, enabling transparent audit and extension.
♻ ☆ Tiny but Mighty: A Software-Hardware Co-Design Approach for Efficient Multimodal Inference on Battery-Powered Small Devices
Yilong Li, Shuai Zhang, Yijing Zeng, Hao Zhang, Xinmiao Xiong, Jingyu Liu, Pan Hu, Suman Banerjee
Large Multimodal Models (LMMs) are inherently modular, consisting of vision and audio encoders, projectors, and large language models. Yet, they are almost always executed monolithically, which underutilizes the heterogeneous accelerators (NPUs, GPUs, DSPs) in modern SoCs and leads to high end-to-end latency. In this paper, we present NANOMIND, a hardware--software co-design inference framework for Large Multimodal Models (LMMs) that breaks large models into modular ``bricks'' (vision, language, audio, etc.) and maps each to its ideal accelerator. The key insight is that large models can be broken into modular components and scheduled to run on the most appropriate compute units. It performs module-level dynamic offloading across accelerators on unified-memory SoCs. By combining customized hardware design, system-level scheduling, and optimized low-bit computation kernels, we demonstrate our framework with a compact, battery-powered device capable of running LMMs entirely on device. This prototype functions as a self-contained intelligent assistant that requires no network connectivity, while achieving higher throughput and superior power efficiency under strict resource constraints. The design further bypasses CPU bottlenecks and reduces redundant memory usage through token-aware buffer management and module-level coordination. Our system outperforms existing implementations in resource efficiency, cutting energy consumption by 42.3\% and GPU memory usage by 11.2\%. This enables a battery-powered device to run LLaVA-OneVision with a camera for nearly 20.8 hours.
Artificial Intelligence 145
☆ A Benchmark of Causal vs Cor relation AI for Predictive Maintenance
Krishna Taduri, Shaunak Dhande, Giacinto Paolo, Saggese, Paul Smith
Predictive maintenance in manufacturing environments presents a challenging optimization problem characterized by extreme cost asymmetry, where missed failures incur costs roughly fifty times higher than false alarms. Conventional machine learning approaches typically optimize statistical accuracy metrics that do not reflect this operational reality and cannot reliably distinguish causal relationships from spurious correlations. This study evaluates eight predictive models, ranging from baseline statistical approaches to formal causal inference methods, on a dataset of 10,000 CNC machines with a 3.3% failure prevalence. The formal causal inference model (L5) achieved estimated annual cost savings of 1.16 million USD (a 70.2 percent reduction), outperforming the best correlation-based decision tree model (L3) by approximately 80,000 USD per year. The causal model matched the highest observed recall (87.9 percent) while reducing false alarms by 97 percent (from 165 to 5) and attained a precision of 92.1 percent, with a train-test performance gap of only 2.6 percentage points. These results indicate that causal AI methods, when combined with domain knowledge, can yield superior financial outcomes and more interpretable predictions compared to correlation-based approaches in predictive maintenance applications.
☆ SocialFusion: Addressing Social Degradation in Pre-train ed Vision -Language Models
Hamza Tahboub, Weiyan Shi, Gang Hua, Huaizu Jiang
Understanding social interactions from visual cues is a fundamental challenge for a socially competent AI. While powerful pre-trained vision-language models (VLMs) have shown remarkable general capabilities, they surprisingly struggle to unify and learn multiple social perception tasks simultaneously, often exhibiting negative transfer. We identify that this negative transfer stems from a critical issue we term "social degradation," whereby the general visual-linguistic pre-training process of VLMs impairs the visual encoder's ability to represent nuanced social information. We investigate this behavior further under two lenses: decodability through linear representation probing and compatibility through gradient conflict analysis, revealing that both play a role in the degradation, especially the former, which is significantly compromised in the VLM pre-training process. To address these issues, we propose SocialFusion, a unified framework that learns a minimal connection between a frozen visual encoder and a language model. Compared with existing VLMs, it exhibits positive transfer across all five social tasks, leveraging synergies between them to enhance overall performance and achieves comparable performance to task-specific state-of-the-art models on various benchmarks. Our findings suggest that current VLM pre-training strategies may be detrimental to acquiring general social competence and highlight the need for more socially-aware training paradigms.
comment : 22 pages, 10 figures
☆ Mode-Conditioning Unlocks Superior Test-Time Scaling
Chen Henry Wu, Sachin Goyal, Aditi Raghunathan
Parallel sampling promises substantial gains in test-time scaling, but its effectiveness is sharply limited by diversity collapse, where models concentrate on a few modes and repeated samples produce the same mistakes. We propose the mode-conditioning (ModC) framework, which explicitly allocates test-time compute across reasoning modes using either specialist models or mode-specific prefixes. ModC consistently improves scaling across controlled graph-search tasks and large-scale reasoning benchmarks, spanning model families and sizes from 0.5B to 7B. On OpenThoughts, fine-tuning Qwen2.5-7B with ModC achieves a 4x efficiency gain over standard training while also improving the maximum attainable Pass@k. We further show that gradient clustering enables ModC without explicit mode labels, yielding up to 10% gains on datasets such as NuminaMath. Finally, we show that ModC improves reinforcement learning (RL) and can further boost diversity-inducing RL methods. These results demonstrate that standard training underutilizes the diversity in data, and that ModC provides a simple, effective remedy for unlocking the full benefits of diversity in test-time scaling.
☆ Wo rl d Model Robustness via Surprise Recognition
Geigh Zollicoffer, Tanush Chopra, Mingkuan Yan, Xiaoxu Ma, Kenneth Eaton, Mark Riedl
AI systems deployed in the real world must contend with distractions and out-of-distribution (OOD) noise that can destabilize their policies and lead to unsafe behavior. While robust training can reduce sensitivity to some forms of noise, it is infeasible to anticipate all possible OOD conditions. To mitigate this issue, we develop an algorithm that leverages a world model's inherent measure of surprise to reduce the impact of noise in world model--based reinforcement learning agents. We introduce both multi-representation and single-representation rejection sampling, enabling robustness to settings with multiple faulty sensors or a single faulty sensor. While the introduction of noise typically degrades agent performance, we show that our techniques preserve performance relative to baselines under varying types and levels of noise across multiple environments within self-driving simulation domains (CARLA and Safety Gymnasium). Furthermore, we demonstrate that our methods enhance the stability of two state-of-the-art world models with markedly different underlying architectures: Cosmos and DreamerV3. Together, these results highlight the robustness of our approach across world modeling domains. We release our code at https://github.com/Bluefin-Tuna/WISER .
☆ Efficiently Learning Branching Networks for Multitask Algorithmic Reasoning KDD'26
Dongyue Li, Zhenshuo Zhang, Minxuan Duan, Edgar Dobriban, Hongyang R. Zhang
Algorithmic reasoning -- the ability to perform step-by-step logical inference -- has become a core benchmark for evaluating reasoning in graph neural networks (GNNs) and large language models (LLMs). Ideally, one would like to design a single model capable of performing well on multiple algorithmic reasoning tasks simultaneously. However, this is challenging when the execution steps of algorithms differ from one another, causing negative interference when they are trained together. We propose branching neural networks, a principled architecture for multitask algorithmic reasoning. Searching for the optimal $k$-ary tree with $L$ layers over $n$ algorithmic tasks is combinatorial, requiring exploration of up to $k^{nL}$ possible structures. We develop AutoBRANE, an efficient algorithm that reduces this search to $O(nL)$ time by solving a convex relaxation at each layer to approximate an optimal task partition. The method clusters tasks using gradient-based affinity scores and can be used on top of any base model, including GNNs and LLMs. We validate AutoBRANE on a broad suite of graph-algorithmic and text-based reasoning benchmarks. We show that gradient features estimate true task performance within 5% error across four GNNs and four LLMs (up to 34B parameters). On the CLRS benchmark, it outperforms the strongest single multitask GNN by 3.7% and the best baseline by 1.2%, while reducing runtime by 48% and memory usage by 26%. The learned branching structures reveal an intuitively reasonable hierarchical clustering of related algorithms. On three text-based graph reasoning benchmarks, AutoBRANE improves over the best non-branching multitask baseline by 3.2%. Finally, on a large graph dataset with 21M edges and 500 tasks, AutoBRANE achieves a 28% accuracy gain over existing multitask and branching architectures, along with a 4.5$\times$ reduction in runtime.
comment : 31 pages. Preprint, to appear in KDD'26
☆ Foundation Priors
Sanjog Misra
Foundation models, and in particular large language models, can generate highly informative responses, prompting growing interest in using these ''synthetic'' outputs as data in empirical research and decision-making. This paper introduces the idea of a foundation prior, which shows that model-generated outputs are not as real observations, but draws from the foundation prior induced prior predictive distribution. As such synthetic data reflects both the model's learned patterns and the user's subjective priors, expectations, and biases. We model the subjectivity of the generative process by making explicit the dependence of synthetic outputs on the user's anticipated data distribution, the prompt-engineering process, and the trust placed in the foundation model. We derive the foundation prior as an exponential-tilted, generalized Bayesian update of the user's primitive prior, where a trust parameter governs the weight assigned to synthetic data. We then show how synthetic data and the associated foundation prior can be incorporated into standard statistical and econometric workflows, and discuss their use in applications such as refining complex models, informing latent constructs, guiding experimental design, and augmenting random-coefficient and partially linear specifications. By treating generative outputs as structured, explicitly subjective priors rather than as empirical observations, the framework offers a principled way to harness foundation models in empirical work while avoiding the conflation of synthetic ''facts'' with real data.
☆ Supporting Productivity Skill Development in College Students through Social Robot Coaching: A Proof-of-Concept
Himanshi Lalwani, Hanan Salam
College students often face academic challenges that hamper their productivity and well-being. Although self-help books and productivity apps are popular, they often fall short. Books provide generalized, non-interactive guidance, and apps are not inherently educational and can hinder the development of key organizational skills. Traditional productivity coaching offers personalized support, but is resource-intensive and difficult to scale. In this study, we present a proof-of-concept for a socially assistive robot (SAR) as an educational coach and a potential solution to the limitations of existing productivity tools and coaching approaches. The SAR delivers six different lessons on time management and task prioritization. Users interact via a chat interface, while the SAR responds through speech (with a toggle option). An integrated dashboard monitors progress, mood, engagement, confidence per lesson, and time spent per lesson. It also offers personalized productivity insights to foster reflection and self-awareness. We evaluated the system with 15 college students, achieving a System Usability Score of 79.2 and high ratings for overall experience and engagement. Our findings suggest that SAR-based productivity coaching can offer an effective and scalable solution to improve productivity among college students.
☆ Energy-Aware Data-Driven Model Selection in LLM-Orchestrated AI Systems
Daria Smirnova, Hamid Nasiri, Marta Adamska, Zhengxin Yu, Peter Garraghan
As modern artificial intelligence (AI) systems become more advanced and capable, they can leverage a wide range of tools and models to perform complex tasks. Today, the task of orchestrating these models is often performed by Large Language Models (LLMs) that rely on qualitative descriptions of models for decision-making. However, the descriptions provided to these LLM-based orchestrators do not reflect true model capabilities and performance characteristics, leading to suboptimal model selection, reduced accuracy, and increased energy costs. In this paper, we conduct an empirical analysis of LLM-based orchestration limitations and propose GUIDE, a new energy-aware model selection framework that accounts for performance-energy trade-offs by incorporating quantitative model performance characteristics in decision-making. Experimental results demonstrate that GUIDE increases accuracy by 0.90%-11.92% across various evaluated tasks, and achieves up to 54% energy efficiency improvement, while reducing orchestrator model selection latency from 4.51 s to 7.2 ms.
☆ Discriminative classification with generative features: bridging Naive Bayes and logistic regression
Zachary Terner, Alexander Petersen, Yuedong Wang
We introduce Smart Bayes, a new classification framework that bridges generative and discriminative modeling by integrating likelihood-ratio-based generative features into a logistic-regression-style discriminative classifier. From the generative perspective, Smart Bayes relaxes the fixed unit weights of Naive Bayes by allowing data-driven coefficients on density-ratio features. From a discriminative perspective, it constructs transformed inputs as marginal log-density ratios that explicitly quantify how much more likely each feature value is under one class than another, thereby providing predictors with stronger class separation than the raw covariates. To support this framework, we develop a spline-based estimator for univariate log-density ratios that is flexible, robust, and computationally efficient. Through extensive simulations and real-data studies, Smart Bayes often outperforms both logistic regression and Naive Bayes. Our results highlight the potential of hybrid approaches that exploit generative structure to enhance discriminative performance.
☆ CycliST: A Video Language Model Benchmark for Reasoning on Cyclical State Transitions
Simon Kohaut, Daniel Ochs, Shun Zhang, Benedict Flade, Julian Eggert, Kristian Kersting, Devendra Singh Dhami
We present CycliST, a novel benchmark dataset designed to evaluate Video Language Models (VLM) on their ability for textual reasoning over cyclical state transitions. CycliST captures fundamental aspects of real-world processes by generating synthetic, richly structured video sequences featuring periodic patterns in object motion and visual attributes. CycliST employs a tiered evaluation system that progressively increases difficulty through variations in the number of cyclic objects, scene clutter, and lighting conditions, challenging state-of-the-art models on their spatio-temporal cognition. We conduct extensive experiments with current state-of-the-art VLMs, both open-source and proprietary, and reveal their limitations in generalizing to cyclical dynamics such as linear and orbital motion, as well as time-dependent changes in visual attributes like color and scale. Our results demonstrate that present-day VLMs struggle to reliably detect and exploit cyclic patterns, lack a notion of temporal understanding, and are unable to extract quantitative insights from scenes, such as the number of objects in motion, highlighting a significant technical gap that needs to be addressed. More specifically, we find no single model consistently leads in performance: neither size nor architecture correlates strongly with outcomes, and no model succeeds equally well across all tasks. By providing a targeted challenge and a comprehensive evaluation framework, CycliST paves the way for visual reasoning models that surpass the state-of-the-art in understanding periodic patterns.
☆ CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agent s
Peter Jansen, Samiah Hassan, Pragnya Narasimha
Automated Scientific Discovery (ASD) systems can help automatically generate and run code-based experiments, but their capabilities are limited by the code they can reliably generate from parametric knowledge alone. As a result, current systems either mutate a small number of manually-crafted experiment examples, or operate solely from parametric knowledge, limiting quality and reach. We introduce CodeDistiller, a system that automatically distills large collections of scientific Github repositories into a vetted library of working domain-specific code examples, allowing ASD agents to expand their capabilities without manual effort. Using a combination of automatic and domain-expert evaluation on 250 materials science repositories, we find the best model is capable of producing functional examples for 74% of repositories, while our downstream evaluation shows an ASD agent augmented with a CodeDistiller generated library produces more accurate, complete, and scientifically sound experiments than an agent with only general materials-science code examples.
comment : 8 pages, 3 figures, 2 tables
☆ Testing the Machine Consciousness Hypothesis
Stephen Fitz
The Machine Consciousness Hypothesis states that consciousness is a substrate-free functional property of computational systems capable of second-order perception. I propose a research program to investigate this idea in silico by studying how collective self-models (coherent, self-referential representations) emerge from distributed learning systems embedded within universal self-organizing environments. The theory outlined here starts from the supposition that consciousness is an emergent property of collective intelligence systems undergoing synchronization of prediction through communication. It is not an epiphenomenon of individual modeling but a property of the language that a system evolves to internally describe itself. For a model of base reality, I begin with a minimal but general computational world: a cellular automaton, which exhibits both computational irreducibility and local reducibility. On top of this computational substrate, I introduce a network of local, predictive, representational (neural) models capable of communication and adaptation. I use this layered model to study how collective intelligence gives rise to self-representation as a direct consequence of inter-agent alignment. I suggest that consciousness does not emerge from modeling per se, but from communication. It arises from the noisy, lossy exchange of predictive messages between groups of local observers describing persistent patterns in the underlying computational substrate (base reality). It is through this representational dialogue that a shared model arises, aligning many partial views of the world. The broader goal is to develop empirically testable theories of machine consciousness, by studying how internal self-models may form in distributed systems without centralized control.
☆ SimWo rl d: An Open-ended Realistic Simulator for Autonomous Agent s in Physical and Social Wo rl ds
Jiawei Ren, Yan Zhuang, Xiaokang Ye, Lingjun Mao, Xuhong He, Jianzhi Shen, Mrinaal Dogra, Yiming Liang, Ruixuan Zhang, Tianai Yue, Yiqing Yang, Eric Liu, Ryan Wu, Kevin Benavente, Rajiv Mandya Nagaraju, Muhammad Faayez, Xiyan Zhang, Dhruv Vivek Sharma, Xianrui Zhong, Ziqiao Ma, Tianmin Shu, Zhiting Hu, Lianhui Qin
While LLM/VLM-powered AI agents have advanced rapidly in math, coding, and computer use, their applications in complex physical and social environments remain challenging. Building agents that can survive and thrive in the real world (for example, by autonomously earning income or running a business) requires massive-scale interaction, reasoning, training, and evaluation across diverse embodied scenarios. However, existing world simulators for such development fall short: they often rely on limited hand-crafted environments, simulate simplified game-like physics and social rules, and lack native support for LLM/VLM agents. We introduce SimWorld, a new simulator built on Unreal Engine 5, designed for developing and evaluating LLM/VLM agents in rich, real-world-like settings. SimWorld offers three core capabilities: (1) realistic, open-ended world simulation, including accurate physical and social dynamics and language-driven procedural environment generation; (2) a rich interface for LLM/VLM agents, with multimodal world inputs and open-vocabulary actions at varying levels of abstraction; and (3) diverse and extensible physical and social reasoning scenarios that are easily customizable by users. We demonstrate SimWorld by deploying frontier LLM agents (e.g., GPT-4o, Gemini-2.5-Flash, Claude-3.5, and DeepSeek-Prover-V2) on long-horizon multi-agent delivery tasks involving strategic cooperation and competition. The results reveal distinct reasoning patterns and limitations across models. We open-source SimWorld and hope it becomes a foundational platform for advancing real-world agent intelligence across disciplines: https://simworld.org.
☆ On The Finetuning of MLIPs Through the Lens of Iterated Maps With BPTT
Evan Dramko, Yizhi Zhu, Aleksandar Krivokapic, Geoffroy Hautier, Thomas Reps, Christopher Jermaine, Anastasios Kyrillidis
Vital to the creation of advanced materials is performing structural relaxations. Traditional approaches built on physics-derived first-principles calculations are computationally expensive, motivating the creation of machine-learning interatomic potentials (MLIPs). Traditional approaches to training MLIPs for structural relaxations involves training models to faithfully reproduce first-principles computed forces. We propose a fine-tuning method to be used on a pretrained MLIP in which we create a fully-differentiable end-to-end simulation loop that optimizes the predicted final structures directly. Trajectories are unrolled and gradients are tracked through the entire relaxation. We show that this method achieves substantial performance gains when applied to pretrained models, leading to a nearly $50\%$ reduction in test error across the sample datasets. Interestingly, we show the process is robust to substantial variation in the relaxation setup, achieving negligibly different results across varied hyperparameter and procedural modifications. Experimental results indicate this is due to a ``preference'' of BPTT to modify the MLIP rather than the other trainable parameters. Of particular interest to practitioners is that this approach lowers the data requirements for producing an effective domain-specific MLIP, addressing a common bottleneck in practical deployment.
comment : 9 main pages, total of 15 pages. 6 tables, 6 Figures
☆ PIANO: Physics-informed Dual Neural Operator for Precipitation Nowcasting NeurIPS 2025
Seokhyun Chin, Junghwan Park, Woojin Cho
Precipitation nowcasting, key for early warning of disasters, currently relies on computationally expensive and restrictive methods that limit access to many countries. To overcome this challenge, we propose precipitation nowcasting using satellite imagery with physics constraints for improved accuracy and physical consistency. We use a novel physics-informed dual neural operator (PIANO) structure to enforce the fundamental equation of advection-diffusion during training to predict satellite imagery using a PINN loss. Then, we use a generative model to convert satellite images to radar images, which are used for precipitation nowcasting. Compared to baseline models, our proposed model shows a notable improvement in moderate (4mm/h) precipitation event prediction alongside short-term heavy (8mm/h) precipitation event prediction. It also demonstrates low seasonal variability in predictions, indicating robustness for generalization. This study suggests the potential of the PIANO and serves as a good baseline for physics-informed precipitation nowcasting.
comment : NeurIPS 2025 Machine Learning and Physical Sciences Workshop
☆ Parameter Reduction Improves Vision Transformer s: A Comparative Study of Sharing and Width Reduction
Anantha Padmanaban Krishna Kumar
Although scaling laws and many empirical results suggest that increasing the size of Vision Transformers often improves performance, model accuracy and training behavior are not always monotonically increasing with scale. Focusing on ViT-B/16 trained on ImageNet-1K, we study two simple parameter-reduction strategies applied to the MLP blocks, each removing 32.7\% of the baseline parameters. Our \emph{GroupedMLP} variant shares MLP weights between adjacent transformer blocks and achieves 81.47\% top-1 accuracy while maintaining the baseline computational cost. Our \emph{ShallowMLP} variant halves the MLP hidden dimension and reaches 81.25\% top-1 accuracy with a 38\% increase in inference throughput. Both models outperform the 86.6M-parameter baseline (81.05\%) and exhibit substantially improved training stability, reducing peak-to-final accuracy degradation from 0.47\% to the range 0.03\% to 0.06\%. These results suggest that, for ViT-B/16 on ImageNet-1K with a standard training recipe, the model operates in an overparameterized regime in which MLP capacity can be reduced without harming performance and can even slightly improve it. More broadly, our findings suggest that architectural constraints such as parameter sharing and reduced width may act as useful inductive biases, and highlight the importance of how parameters are allocated when designing Vision Transformers. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/parameter-efficient-vit-mlps.
comment : 7 pages total (6 pages main text, 1 page references), 1 figures, 2 tables. Code available at https://github.com/AnanthaPadmanaban-KrishnaKumar/parameter-efficient-vit-mlps
☆ Adaptive-lambda Subtracted Importance Sampled Scores in Machine Unlearning for DDPMs and VAEs
MohammadParsa Dini, Human Jafari, Sajjad Amini, MohammadMahdi Mojahedian
Machine Unlearning is essential for large generative models (VAEs, DDPMs) to comply with the right to be forgotten and prevent undesired content generation without costly retraining. Existing approaches, such as Static-lambda SISS for diffusion models, rely on a fixed mixing weight lambda, which is suboptimal because the required unlearning strength varies across samples and training stages. We propose Adaptive-lambda SISS, a principled extension that turns lambda into a latent variable dynamically inferred at each training step. A lightweight inference network parameterizes an adaptive posterior over lambda, conditioned on contextual features derived from the instantaneous SISS loss terms (retain/forget losses and their gradients). This enables joint optimization of the diffusion model and the lambda-inference mechanism via a variational objective, yielding significantly better trade-offs. We further extend the adaptive-lambda principle to score-based unlearning and introduce a multi-class variant of Score Forgetting Distillation. In addition, we present two new directions: (i) a hybrid objective combining the data-free efficiency of Score Forgetting Distillation with the direct gradient control of SISS, and (ii) a Reinforcement Learning formulation that treats unlearning as a sequential decision process, learning an optimal policy over a state space defined by the model's current memory of the forget set. Experiments on an augmented MNIST benchmark show that Adaptive-lambda SISS substantially outperforms the original static-lambda SISS, achieving stronger removal of forgotten classes while better preserving generation quality on the retain set.
☆ Automating the Refinement of Reinforcement Learning Specifications
Tanmay Ambadkar, Đorđe Žikelić, Abhinav Verma
Logical specifications have been shown to help reinforcement learning algorithms in achieving complex tasks. However, when a task is under-specified, agents might fail to learn useful policies. In this work, we explore the possibility of improving coarse-grained logical specifications via an exploration-guided strategy. We propose \textsc{AutoSpec}, a framework that searches for a logical specification refinement whose satisfaction implies satisfaction of the original specification, but which provides additional guidance therefore making it easier for reinforcement learning algorithms to learn useful policies. \textsc{AutoSpec} is applicable to reinforcement learning tasks specified via the SpectRL specification logic. We exploit the compositional nature of specifications written in SpectRL, and design four refinement procedures that modify the abstract graph of the specification by either refining its existing edge specifications or by introducing new edge specifications. We prove that all four procedures maintain specification soundness, i.e. any trajectory satisfying the refined specification also satisfies the original. We then show how \textsc{AutoSpec} can be integrated with existing reinforcement learning algorithms for learning policies from logical specifications. Our experiments demonstrate that \textsc{AutoSpec} yields promising improvements in terms of the complexity of control tasks that can be solved, when refined logical specifications produced by \textsc{AutoSpec} are utilized.
☆ Shielded Controller Units for RL with Operational Constraints Applied to Remote Microgrids
Hadi Nekoei, Alexandre Blondin Massé, Rachid Hassani, Sarath Chandar, Vincent Mai
Reinforcement learning (RL) is a powerful framework for optimizing decision-making in complex systems under uncertainty, an essential challenge in real-world settings, particularly in the context of the energy transition. A representative example is remote microgrids that supply power to communities disconnected from the main grid. Enabling the energy transition in such systems requires coordinated control of renewable sources like wind turbines, alongside fuel generators and batteries, to meet demand while minimizing fuel consumption and battery degradation under exogenous and intermittent load and wind conditions. These systems must often conform to extensive regulations and complex operational constraints. To ensure that RL agents respect these constraints, it is crucial to provide interpretable guarantees. In this paper, we introduce Shielded Controller Units (SCUs), a systematic and interpretable approach that leverages prior knowledge of system dynamics to ensure constraint satisfaction. Our shield synthesis methodology, designed for real-world deployment, decomposes the environment into a hierarchical structure where each SCU explicitly manages a subset of constraints. We demonstrate the effectiveness of SCUs on a remote microgrid optimization task with strict operational requirements. The RL agent, equipped with SCUs, achieves a 24% reduction in fuel consumption without increasing battery degradation, outperforming other baselines while satisfying all constraints. We hope SCUs contribute to the safe application of RL to the many decision-making challenges linked to the energy transition.
☆ Med-CRAFT: Automated Construction of Interpretable and Multi-Hop Video Workloads via Knowledge Graph Traversal
Shenxi Liu, Kan Li, Mingyang Zhao, Yuhang Tian, Shoujun Zhou, Bin Li
The scarcity of high-quality, logically annotated video datasets remains a primary bottleneck in advancing Multi-Modal Large Language Models (MLLMs) for the medical domain. Traditional manual annotation is prohibitively expensive and non-scalable, while existing synthetic methods often suffer from stochastic hallucinations and a lack of logical interpretability. To address these challenges, we introduce \textbf{\PipelineName}, a novel neuro-symbolic data engineering framework that formalizes benchmark synthesis as a deterministic graph traversal process. Unlike black-box generative approaches, Med-CRAFT extracts structured visual primitives (e.g., surgical instruments, anatomical boundaries) from raw video streams and instantiates them into a dynamic Spatiotemporal Knowledge Graph. By anchoring query generation to valid paths within this graph, we enforce a rigorous Chain-of-Thought (CoT) provenance for every synthesized benchmark item. We instantiate this pipeline to produce M3-Med-Auto, a large-scale medical video reasoning benchmark exhibiting fine-grained temporal selectivity and multi-hop logical complexity. Comprehensive evaluations demonstrate that our automated pipeline generates query workloads with complexity comparable to expert-curated datasets. Furthermore, a logic alignment analysis reveals a high correlation between the prescribed graph topology and the reasoning steps of state-of-the-art MLLMs, validating the system's capability to encode verifiable logic into visual-linguistic benchmarks. This work paves the way for scalable, low-cost construction of robust evaluation protocols in critical domains.
comment : 8 pages, 7 figures
☆ FMTK: A Modular Toolkit for Composable Time Series Foundation Model Pipelines
Hetvi Shastri, Pragya Sharma, Walid A. Hanafy, Mani Srivastava, Prashant Shenoy
Foundation models (FMs) have opened new avenues for machine learning applications due to their ability to adapt to new and unseen tasks with minimal or no further training. Time-series foundation models (TSFMs) -- FMs trained on time-series data -- have shown strong performance on classification, regression, and imputation tasks. Recent pipelines combine TSFMs with task-specific encoders, decoders, and adapters to improve performance; however, assembling such pipelines typically requires ad hoc, model-specific implementations that hinder modularity and reproducibility. We introduce FMTK, an open-source, lightweight and extensible toolkit for constructing and fine-tuning TSFM pipelines via standardized backbone and component abstractions. FMTK enables flexible composition across models and tasks, achieving correctness and performance with an average of seven lines of code. https://github.com/umassos/FMTK
☆ When Safety Blocks Sense: Measuring Semantic Confusion in LLM Refusals
Riad Ahmed Anonto, Md Labid Al Nahiyan, Md Tanvir Hassan, Ch. Md. Rakin Haider
Safety-aligned language models often refuse prompts that are actually harmless. Current evaluations mostly report global rates such as false rejection or compliance. These scores treat each prompt alone and miss local inconsistency, where a model accepts one phrasing of an intent but rejects a close paraphrase. This gap limits diagnosis and tuning. We introduce "semantic confusion," a failure mode that captures such local inconsistency, and a framework to measure it. We build ParaGuard, a 10k-prompt corpus of controlled paraphrase clusters that hold intent fixed while varying surface form. We then propose three model-agnostic metrics at the token level: Confusion Index, Confusion Rate, and Confusion Depth. These metrics compare each refusal to its nearest accepted neighbors and use token embeddings, next-token probabilities, and perplexity signals. Experiments across diverse model families and deployment guards show that global false-rejection rate hides critical structure. Our metrics reveal globally unstable boundaries in some settings, localized pockets of inconsistency in others, and cases where stricter refusal does not increase inconsistency. We also show how confusion-aware auditing separates how often a system refuses from how sensibly it refuses. This gives developers a practical signal to reduce false refusals while preserving safety.
☆ Goal-Oriented Multi- Agent Semantic Networking: Unifying Intents, Semantics, and Intelligence
Shutong Chen, Qi Liao, Adnan Aijaz, Yansha Deng
6G services are evolving toward goal-oriented and AI-native communication, which are expected to deliver transformative societal benefits across various industries and promote energy sustainability. Yet today's networking architectures, built on complete decoupling of the applications and the network, cannot expose or exploit high-level goals, limiting their ability to adapt intelligently to service needs. This work introduces Goal-Oriented Multi-Agent Semantic Networking (GoAgentNet), a new architecture that elevates communication from data exchange to goal fulfilment. GoAgentNet enables applications and the network to collaborate by abstracting their functions into multiple collaborative agents, and jointly orchestrates multi-agent sensing, networking, computation, and control through semantic computation and cross-layer semantic networking, allowing the entire architecture to pursue unified application goals. We first outline the limitations of legacy network designs in supporting 6G services, based on which we highlight key enablers of our GoAgentNet design. Then, through three representative 6G usage scenarios, we demonstrate how GoAgentNet can unlock more efficient and intelligent services. We further identify unique challenges faced by GoAgentNet deployment and corresponding potential solutions. A case study on robotic fault detection and recovery shows that our GoAgentNet architecture improves energy efficiency by up to 99% and increases the task success rate by up to 72%, compared with the existing networking architectures without GoAgentNet, which underscores its potential to support scalable and sustainable 6G systems.
comment : Submitting to IEEE for potential publications
☆ AltNet: Addressing the Plasticity-Stability Dilemma in Reinforcement Learning
Mansi Maheshwari, John C. Raisbeck, Bruno Castro da Silva
Neural networks have shown remarkable success in supervised learning when trained on a single task using a fixed dataset. However, when neural networks are trained on a reinforcement learning task, their ability to continue learning from new experiences declines over time. This decline in learning ability is known as plasticity loss. To restore plasticity, prior work has explored periodically resetting the parameters of the learning network, a strategy that often improves overall performance. However, such resets come at the cost of a temporary drop in performance, which can be dangerous in real-world settings. To overcome this instability, we introduce AltNet, a reset-based approach that restores plasticity without performance degradation by leveraging twin networks. The use of twin networks anchors performance during resets through a mechanism that allows networks to periodically alternate roles: one network learns as it acts in the environment, while the other learns off-policy from the active network's interactions and a replay buffer. At fixed intervals, the active network is reset and the passive network, having learned from prior experiences, becomes the new active network. AltNet restores plasticity, improving sample efficiency and achieving higher performance, while avoiding performance drops that pose risks in safety-critical settings. We demonstrate these advantages in several high-dimensional control tasks from the DeepMind Control Suite, where AltNet outperforms various relevant baseline methods, as well as state-of-the-art reset-based techniques.
☆ VLASH: Real-Time VLAs via Future-State-Aware Asynchronous Inference
Jiaming Tang, Yufei Sun, Yilong Zhao, Shang Yang, Yujun Lin, Zhuoyang Zhang, James Hou, Yao Lu, Zhijian Liu, Song Han
Vision-Language-Action models (VLAs) are becoming increasingly capable across diverse robotic tasks. However, their real-world deployment remains slow and inefficient: demonstration videos are often sped up by 5-10x to appear smooth, with noticeable action stalls and delayed reactions to environmental changes. Asynchronous inference offers a promising solution to achieve continuous and low-latency control by enabling robots to execute actions and perform inference simultaneously. However, because the robot and environment continue to evolve during inference, a temporal misalignment arises between the prediction and execution intervals. This leads to significant action instability, while existing methods either degrade accuracy or introduce runtime overhead to mitigate it. We propose VLASH, a general asynchronous inference framework for VLAs that delivers smooth, accurate, and fast reaction control without additional overhead or architectural changes. VLASH estimates the future execution-time state by rolling the robot state forward with the previously generated action chunk, thereby bridging the gap between prediction and execution. Experiments show that VLASH achieves up to 2.03x speedup and reduces reaction latency by up to 17.4x compared to synchronous inference while fully preserving the original accuracy. Moreover, it empowers VLAs to handle fast-reaction, high-precision tasks such as playing ping-pong and playing whack-a-mole, where traditional synchronous inference fails. Code is available at https://github.com/mit-han-lab/vlash
☆ Operator-Theoretic Framework for Gradient-Free Federated Learning
Mohit Kumar, Mathias Brucker, Alexander Valentinitsch, Adnan Husakovic, Ali Abbas, Manuela Geiß, Bernhard A. Moser
Federated learning must address heterogeneity, strict communication and computation limits, and privacy while ensuring performance. We propose an operator-theoretic framework that maps the $L^2$-optimal solution into a reproducing kernel Hilbert space (RKHS) via a forward operator, approximates it using available data, and maps back with the inverse operator, yielding a gradient-free scheme. Finite-sample bounds are derived using concentration inequalities over operator norms, and the framework identifies a data-dependent hypothesis space with guarantees on risk, error, robustness, and approximation. Within this space we design efficient kernel machines leveraging the space folding property of Kernel Affine Hull Machines. Clients transfer knowledge via a scalar space folding measure, reducing communication and enabling a simple differentially private protocol: summaries are computed from noise-perturbed data matrices in one step, avoiding per-round clipping and privacy accounting. The induced global rule requires only integer minimum and equality-comparison operations per test point, making it compatible with fully homomorphic encryption (FHE). Across four benchmarks, the gradient-free FL method with fixed encoder embeddings matches or outperforms strong gradient-based fine-tuning, with gains up to 23.7 points. In differentially private experiments, kernel smoothing mitigates accuracy loss in high-privacy regimes. The global rule admits an FHE realization using $Q \times C$ encrypted minimum and $C$ equality-comparison operations per test point, with operation-level benchmarks showing practical latencies. Overall, the framework provides provable guarantees with low communication, supports private knowledge transfer via scalar summaries, and yields an FHE-compatible prediction rule offering a mathematically grounded alternative to gradient-based federated learning under heterogeneity.
☆ Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics
Jinu Lee, Kyoung-Woon On, Simeng Han, Arman Cohan, Julia Hockenmaier
Evaluating the quality of LLM-generated reasoning traces in expert domains (e.g., law) is essential for ensuring credibility and explainability, yet remains challenging due to the inherent complexity of such reasoning tasks. We introduce LEGIT (LEGal Issue Trees), a novel large-scale (24K instances) expert-level legal reasoning dataset with an emphasis on reasoning trace evaluation. We convert court judgments into hierarchical trees of opposing parties' arguments and the court's conclusions, which serve as rubrics for evaluating the issue coverage and correctness of the reasoning traces. We verify the reliability of these rubrics via human expert annotations and comparison with coarse, less informative rubrics. Using the LEGIT dataset, we show that (1) LLMs' legal reasoning ability is seriously affected by both legal issue coverage and correctness, and that (2) retrieval-augmented generation (RAG) and RL with rubrics bring complementary benefits for legal reasoning abilities, where RAG improves overall reasoning capability, whereas RL improves correctness albeit with reduced coverage.
☆ ChartAnchor: Chart Grounding with Structural-Semantic Fidelity
Xinhang Li, Jingbo Zhou, Pengfei Luo, Yixiong Xiao, Tong Xu
Recent advances in multimodal large language models (MLLMs) highlight the need for benchmarks that rigorously evaluate structured chart comprehension.Chart grounding refers to the bidirectional alignment between a chart's visual appearance and the structured semantics. This task requires models to produce a symbolic specification that faithfully captures the chart's visual and structural intent, while also recovering the underlying tabular data with precise values and relationships. Chart grounding directly reflects a model's capabilities in numerical reasoning, multimodal alignment, and structural reconstruction, and has several important applications in real-world scenarios.Existing benchmarks, constrained by narrow chart diversity, isolated tasks, and incomplete evaluation frameworks, fail to holistically assess grounding. To address this, we propose ChartAnchor, a comprehensive benchmark of 8k+ chart-table-code triples spanning 30 chart types drawn from diverse real-world and augmented sources. ChartAnchor introduces two complementary tasks: chart-to-code generation (synthesizing executable code to replicate charts) and controlled chart-to-table reconstruction (extracting exact data with predefined headers), enabling cross-validation of visual and numerical fidelity. A multi-level evaluation framework integrates semantic validation, stylistic analysis, and perceptual metrics to assess both structural and content-level correctness. Extensive experiments on MLLMs reveal critical limitations in numerical precision and code synthesis, emphasizing the need for structured reasoning beyond surface-level perception. By unifying symbolic and data-driven grounding, ChartAnchor establishes a rigorous foundation for chart grounding, offering meaningful insights for advancing MLLMs in scientific, financial, and industrial domains.
☆ Chain of Unit-Physics: A Primitive-Centric Approach to Scientific Code Synthesis
Vansh Sharma, Venkat Raman
Agentic large language models are proposed as autonomous code generators for scientific computing, yet their reliability in high-stakes problems remains unclear. Developing computational scientific software from natural-language queries remains challenging broadly due to (a) sparse representation of domain codes during training and (b) the limited feasibility of RLHF with a small expert community. To address these limitations, this work conceptualizes an inverse approach to code design, embodied in the Chain of Unit-Physics framework: a first-principles (or primitives)-centric, multi-agent system in which human expert knowledge is encoded as unit-physics tests that explicitly constrain code generation. The framework is evaluated on a nontrivial combustion task, used here as a representative benchmark for scientific problem with realistic physical constraints. Closed-weight systems and code-focused agentic variants fail to produce correct end-to-end solvers, despite tool and web access, exhibiting four recurrent error classes: interface (syntax/API) hallucinations, overconfident assumptions, numerical/physical incoherence, and configuration fragility. Open-weight models with chain-of-thought (CoT) decoding reduce interface errors but still yield incorrect solutions. On the benchmark task, the proposed framework converges within 5-6 iterations, matches the human-expert implementation (mean error of $3.1\times10^{-3}$ %), with a $\sim$33.4 % faster runtime and a $\sim$30 % efficient memory usage at a cost comparable to mid-sized commercial APIs, yielding a practical template for physics-grounded scientific code generation. As datasets and models evolve, zero-shot code accuracy will improve; however, the Chain of Unit-Physics framework goes further by embedding first-principles analysis that is foundational to scientific codes.
☆ Provenance-Driven Reliable Semantic Medical Image Vector Reconstruction via Lightweight Blockchain-Verified Latent Fingerprints
Mohsin Rasheed, Abdullah Al-Mamun
Medical imaging is essential for clinical diagnosis, yet real-world data frequently suffers from corruption, noise, and potential tampering, challenging the reliability of AI-assisted interpretation. Conventional reconstruction techniques prioritize pixel-level recovery and may produce visually plausible outputs while compromising anatomical fidelity, an issue that can directly impact clinical outcomes. We propose a semantic-aware medical image reconstruction framework that integrates high-level latent embeddings with a hybrid U-Net architecture to preserve clinically relevant structures during restoration. To ensure trust and accountability, we incorporate a lightweight blockchain-based provenance layer using scale-free graph design, enabling verifiable recording of each reconstruction event without imposing significant overhead. Extensive evaluation across multiple datasets and corruption types demonstrates improved structural consistency, restoration accuracy, and provenance integrity compared with existing approaches. By uniting semantic-guided reconstruction with secure traceability, our solution advances dependable AI for medical imaging, enhancing both diagnostic confidence and regulatory compliance in healthcare environments.
☆ IndiMathBench: Autoformalizing Mathematical Reasoning Problems with a Human Touch
Param Biyani, Shashank Kirtania, Yasharth Bajpai, Sumit Gulwani, Ashish Tiwari
We introduce IndiMathBench, a human-verified benchmark designed to evaluate mathematical theorem proving, curated using an AI-powered human-assisted pipeline for formalizing natural language problems in Lean. IndiMathBench is composed of 312 formal Lean 4 theorems paired with their corresponding informal problem statements, sourced from Indian Mathematics Olympiads. Through category-based retrieval, iterative compiler feedback, and multi-model ensembles, our pipeline generates candidate formalizations that experts efficiently validate via an interactive dashboard with automated quality summaries. Evaluation across multiple frontier models demonstrates that autoformalization remains challenging, with substantial gaps between syntactic validity and semantic correctness, while theorem proving success rates remain low even with iterative refinement, demonstrating that \benchmark~presents a challenging testbed for mathematical reasoning. IndiMathBench is available at https://github.com/prmbiy/IndiMathBench.
☆ An Approach to Variable Clustering: K-means in Transposed Data and its Relation ship with Principal Component Analysis
Victor Saquicela, Kenneth Palacio-Baus, Mario Chifla
Principal Component Analysis (PCA) and K-means constitute fundamental techniques in multivariate analysis. Although they are frequently applied independently or sequentially to cluster observations, the relationship between them, especially when K-means is used to cluster variables rather than observations, has been scarcely explored. This study seeks to address this gap by proposing an innovative method that analyzes the relationship between clusters of variables obtained by applying K-means on transposed data and the principal components of PCA. Our approach involves applying PCA to the original data and K-means to the transposed data set, where the original variables are converted into observations. The contribution of each variable cluster to each principal component is then quantified using measures based on variable loadings. This process provides a tool to explore and understand the clustering of variables and how such clusters contribute to the principal dimensions of variation identified by PCA.
comment : Presented at conference and to appear in the proceedings of the 2025 IEEE Chilean Conference on Electrical, Electronics Engineering, Information and Communication Technologies (ChileCon)
☆ Integrating Causal Foundation Model in Prescriptive Maintenance Framework for Optimizing Production Line OEE
Felix Saretzky, Lucas Andersen, Thomas Engel, Fazel Ansari
The transition to prescriptive maintenance in manufacturing is critically constrained by a dependence on predictive models. These models tend to rely on spurious correlations rather than identifying the true causal drivers of failures, often leading to costly misdiagnoses and ineffective interventions. This fundamental limitation results in a key-challenge: while we can predict that a failure may occur, we lack a systematic method to understand why a failure occurs, thereby providing the basis for identifying the most effective intervention. This paper proposes a model based on causal machine learning to bridge this gap. Our objective is to move beyond diagnosis to active prescription by simulating and evaluating potential fixes toward optimizing KPIs such as Overall Equipment Effectiveness (OEE). For this purpose a pre-trained causal foundation model is used as a "what-if" model to estimate the effects of potential fixes. By measuring the causal effect of each intervention on system-level KPIs, it provides a data-driven ranking of actions to recommend at the production line. This process not only identifies root causes but also quantifies their operational impact. The model is evaluated using semi-synthetic manufacturing data and compared with a baseline machine learning model. This paper sets the technical basis for a robust prescriptive maintenance framework, allowing engineers to test potential solutions in a causal environment to make more effective operational decisions and reduce costly downtimes.
comment : 9 pages, 3 images, 1 table, conference paper
☆ Optimizing Generative Ranking Relevance via Reinforcement Learning in Xiaohongshu Search KDD 2026
Ziyang Zeng, Heming Jing, Jindong Chen, Xiangli Li, Hongyu Liu, Yixuan He, Zhengyu Li, Yige Sun, Zheyong Xie, Yuqing Yang, Shaosheng Cao, Jun Fan, Yi Wu, Yao Hu
Ranking relevance is a fundamental task in search engines, aiming to identify the items most relevant to a given user query. Traditional relevance models typically produce scalar scores or directly predict relevance labels, limiting both interpretability and the modeling of complex relevance signals. Inspired by recent advances in Chain-of-Thought (CoT) reasoning for complex tasks, we investigate whether explicit reasoning can enhance both interpretability and performance in relevance modeling. However, existing reasoning-based Generative Relevance Models (GRMs) primarily rely on supervised fine-tuning on large amounts of human-annotated or synthetic CoT data, which often leads to limited generalization. Moreover, domain-agnostic, free-form reasoning tends to be overly generic and insufficiently grounded, limiting its potential to handle the diverse and ambiguous cases prevalent in open-domain search. In this work, we formulate relevance modeling in Xiaohongshu search as a reasoning task and introduce a Reinforcement Learning (RL)-based training framework to enhance the grounded reasoning capabilities of GRMs. Specifically, we incorporate practical business-specific relevance criteria into the multi-step reasoning prompt design and propose Stepwise Advantage Masking (SAM), a lightweight process-supervision strategy which facilitates effective learning of these criteria through improved credit assignment. To enable industrial deployment, we further distill the large-scale RL-tuned model to a lightweight version suitable for real-world search systems. Extensive experiments on industrial datasets, along with online A/B tests, demonstrate the effectiveness of our approach.
comment : Accepted by KDD 2026 ADS Track
☆ Multi-Modal AI for Remote Patient Monitoring in Cancer Care
Yansong Liu, Ronnie Stafford, Pramit Khetrapal, Huriye Kocadag, Graça Carvalho, Patricia de Winter, Maryam Imran, Amelia Snook, Adamos Hadjivasiliou, D. Vijay Anand, Weining Lin, John Kelly, Yukun Zhou, Ivana Drobnjak
For patients undergoing systemic cancer therapy, the time between clinic visits is full of uncertainties and risks of unmonitored side effects. To bridge this gap in care, we developed and prospectively trialed a multi-modal AI framework for remote patient monitoring (RPM). This system integrates multi-modal data from the HALO-X platform, such as demographics, wearable sensors, daily surveys, and clinical events. Our observational trial is one of the largest of its kind and has collected over 2.1 million data points (6,080 patient-days) of monitoring from 84 patients. We developed and adapted a multi-modal AI model to handle the asynchronous and incomplete nature of real-world RPM data, forecasting a continuous risk of future adverse events. The model achieved an accuracy of 83.9% (AUROC=0.70). Notably, the model identified previous treatments, wellness check-ins, and daily maximum heart rate as key predictive features. A case study demonstrated the model's ability to provide early warnings by outputting escalating risk profiles prior to the event. This work establishes the feasibility of multi-modal AI RPM for cancer care and offers a path toward more proactive patient support.(Accepted at Europe NeurIPS 2025 Multimodal Representation Learning for Healthcare Workshop)
☆ Table as a Modality for Large Language Models NeurIPS 2025
Liyao Li, Chao Ye, Wentao Ye, Yifei Sun, Zhe Jiang, Haobo Wang, Jiaming Tian, Yiming Zhang, Ningtao Wang, Xing Fu, Gang Chen, Junbo Zhao
To migrate the remarkable successes of Large Language Models (LLMs), the community has made numerous efforts to generalize them to the table reasoning tasks for the widely deployed tabular data. Despite that, in this work, by showing a probing experiment on our proposed StructQA benchmark, we postulate that even the most advanced LLMs (such as GPTs) may still fall short of coping with tabular data. More specifically, the current scheme often simply relies on serializing the tabular data, together with the meta information, then inputting them through the LLMs. We argue that the loss of structural information is the root of this shortcoming. In this work, we further propose TAMO, which bears an ideology to treat the tables as an independent modality integrated with the text tokens. The resulting model in TAMO is a multimodal framework consisting of a hypergraph neural network as the global table encoder seamlessly integrated with the mainstream LLM. Empirical results on various benchmarking datasets, including HiTab, WikiTQ, WikiSQL, FeTaQA, and StructQA, have demonstrated significant improvements on generalization with an average relative gain of 42.65%.
comment : Accepted to NeurIPS 2025
☆ Fine-tuning of lightweight large language models for sentiment classification on heterogeneous financial textual data
Alvaro Paredes Amorin, Andre Python, Christoph Weisser
Large language models (LLMs) play an increasingly important role in finan- cial markets analysis by capturing signals from complex and heterogeneous textual data sources, such as tweets, news articles, reports, and microblogs. However, their performance is dependent on large computational resources and proprietary datasets, which are costly, restricted, and therefore inacces- sible to many researchers and practitioners. To reflect realistic situations we investigate the ability of lightweight open-source LLMs - smaller and publicly available models designed to operate with limited computational resources - to generalize sentiment understanding from financial datasets of varying sizes, sources, formats, and languages. We compare the benchmark finance natural language processing (NLP) model, FinBERT, and three open-source lightweight LLMs, DeepSeek-LLM 7B, Llama3 8B Instruct, and Qwen3 8B on five publicly available datasets: FinancialPhraseBank, Financial Question Answering, Gold News Sentiment, Twitter Sentiment and Chinese Finance Sentiment. We find that LLMs, specially Qwen3 8B and Llama3 8B, perform best in most scenarios, even from using only 5% of the available training data. These results hold in zero-shot and few-shot learning scenarios. Our findings indicate that lightweight, open-source large language models (LLMs) consti- tute a cost-effective option, as they can achieve competitive performance on heterogeneous textual data even when trained on only a limited subset of the extensive annotated corpora that are typically deemed necessary.
☆ Constant-Time Motion Planning with Manipulation Behaviors
Nayesha Gandotra, Itamar Mishani, Maxim Likhachev
Recent progress in contact-rich robotic manipulation has been striking, yet most deployed systems remain confined to simple, scripted routines. One of the key barriers is the lack of motion planning algorithms that can provide verifiable guarantees for safety, efficiency and reliability. To address this, a family of algorithms called Constant-Time Motion Planning (CTMP) was introduced, which leverages a preprocessing phase to enable collision-free motion queries in a fixed, user-specified time budget (e.g., 10 milliseconds). However, existing CTMP methods do not explicitly incorporate the manipulation behaviors essential for object handling. To bridge this gap, we introduce the \textit{Behavioral Constant-Time Motion Planner} (B-CTMP), an algorithm that extends CTMP to solve a broad class of two-step manipulation tasks: (1) a collision-free motion to a behavior initiation state, followed by (2) execution of a manipulation behavior (such as grasping or insertion) to reach the goal. By precomputing compact data structures, B-CTMP guarantees constant-time query in mere milliseconds while ensuring completeness and successful task execution over a specified set of states. We evaluate B-CTMP on two canonical manipulation tasks in simulation, shelf picking and plug insertion,and demonstrate its effectiveness on a real robot. Our results show that B-CTMP unifies collision-free planning and object manipulation within a single constant-time framework, providing provable guarantees of speed and success for manipulation in semi-structured environments.
comment : In submission
☆ DeformAr: Rethinking NER Evaluation through Component Analysis and Visual Analytics
Ahmed Mustafa Younes
Transformer models have significantly advanced Natural Language Processing (NLP), demonstrating strong performance in English. However, their effectiveness in Arabic, particularly for Named Entity Recognition (NER), remains limited, even with larger pre-trained models. This performance gap stems from multiple factors, including tokenisation, dataset quality, and annotation inconsistencies. Existing studies often analyze these issues in isolation, failing to capture their joint effect on system behaviour and performance. We introduce DeformAr (Debugging and Evaluation Framework for Transformer-based NER Systems), a novel framework designed to investigate and explain the performance discrepancy between Arabic and English NER systems. DeformAr integrates a data extraction library and an interactive dashboard, supporting two modes of evaluation: cross-component analysis and behavioural analysis. The framework divides each language into dataset and model components to examine their interactions. The analysis proceeds in two stages. First, cross-component analysis provides systematic diagnostic measures across data and model subcomponents, addressing the "what," "how," and "why" behind observed discrepancies. The second stage applies behavioural analysis by combining interpretability techniques with token-level metrics, interactive visualisations, and representation space analysis. These stages enable a component-aware diagnostic process that detects model behaviours and explains them by linking them to underlying representational patterns and data factors. DeformAr is the first Arabic-specific, component-based interpretability tool, offering a crucial resource for advancing model analysis in under-resourced languages.
comment : PhD Thesis, University of Sussex, 2025. 311 pages, 140 figures, 32 tables. Submitted as a PDF-only. First supervisor: Julie Weeds. Second supervisor: David Weir
☆ Mitigating Hallucinations in Zero-Shot Scientific Summarisation: A Pilot Study
Imane Jaaouine, Ross D. King
Large language models (LLMs) produce context inconsistency hallucinations, which are LLM generated outputs that are misaligned with the user prompt. This research project investigates whether prompt engineering (PE) methods can mitigate context inconsistency hallucinations in zero-shot LLM summarisation of scientific texts, where zero-shot indicates that the LLM relies purely on its pre-training data. Across eight yeast biotechnology research paper abstracts, six instruction-tuned LLMs were prompted with seven methods: a base- line prompt, two levels of increasing instruction complexity (PE-1 and PE-2), two levels of context repetition (CR-K1 and CR-K2), and two levels of random addition (RA-K1 and RA-K2). Context repetition involved the identification and repetition of K key sentences from the abstract, whereas random addition involved the repetition of K randomly selected sentences from the abstract, where K is 1 or 2. A total of 336 LLM-generated summaries were evaluated using six metrics: ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, METEOR, and cosine similarity, which were used to compute the lexical and semantic alignment be- tween the summaries and the abstracts. Four hypotheses on the effects of prompt methods on summary alignment with the reference text were tested. Statistical analysis on 3744 collected datapoints was performed using bias-corrected and accelerated (BCa) bootstrap confidence intervals and Wilcoxon signed-rank tests with Bonferroni-Holm correction. The results demonstrated that CR and RA significantly improve the lexical alignment of LLM-generated summaries with the abstracts. These findings indicate that prompt engineering has the potential to impact hallucinations in zero-shot scientific summarisation tasks.
☆ Minimal neuron ablation triggers catastrophic collapse in the language core of Large Vision -Language Models
Cen Lu, Yung-Chen Tang, Andrea Cavallaro
Large Vision-Language Models (LVLMs) have shown impressive multimodal understanding capabilities, yet their robustness is poorly understood. In this paper, we investigate the structural vulnerabilities of LVLMs to identify any critical neurons whose removal triggers catastrophic collapse. In this context, we propose CAN, a method to detect Consistently Activated Neurons and to locate critical neurons by progressive masking. Experiments on LLaVA-1.5-7b-hf and InstructBLIP-Vicuna-7b reveal that masking only a tiny portion of the language model's feed-forward networks (just as few as four neurons in extreme cases) suffices to trigger catastrophic collapse. Notably, critical neurons are predominantly localized in the language model rather than in the vision components, and the down-projection layer is a particularly vulnerable structure. We also observe a consistent two-stage collapse pattern: initial expressive degradation followed by sudden, complete collapse. Our findings provide important insights for safety research in LVLMs.
comment : 15 pages, 6 figures,
☆ ForamDeepSlice: A High-Accuracy Deep Learning Framework for Foraminifera Species Classification from 2D Micro-CT Slices
Abdelghafour Halimi, Ali Alibrahim, Didier Barradas-Bautista, Ronell Sicat, Abdulkader M. Afifi
This study presents a comprehensive deep learning pipeline for the automated classification of 12 foraminifera species using 2D micro-CT slices derived from 3D scans. We curated a scientifically rigorous dataset comprising 97 micro-CT scanned specimens across 27 species, selecting 12 species with sufficient representation for robust machine learning. To ensure methodological integrity and prevent data leakage, we employed specimen-level data splitting, resulting in 109,617 high-quality 2D slices (44,103 for training, 14,046 for validation, and 51,468 for testing). We evaluated seven state-of-the-art 2D convolutional neural network (CNN) architectures using transfer learning. Our final ensemble model, combining ConvNeXt-Large and EfficientNetV2-Small, achieved a test accuracy of 95.64%, with a top-3 accuracy of 99.6% and an area under the ROC curve (AUC) of 0.998 across all species. To facilitate practical deployment, we developed an interactive advanced dashboard that supports real-time slice classification and 3D slice matching using advanced similarity metrics, including SSIM, NCC, and the Dice coefficient. This work establishes new benchmarks for AI-assisted micropaleontological identification and provides a fully reproducible framework for foraminifera classification research, bridging the gap between deep learning and applied geosciences.
☆ Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
Xinzhu Chen, Xuesheng Li, Zhongxiang Sun, Weijie Yu
Reinforcement Learning with Verifiable Rewards (RLVR) has become a central approach for improving the reasoning ability of large language models. Recent work studies RLVR through token entropy, arguing that high-entropy tokens drive exploration and should receive stronger updates. However, they overlook the fact that most of a reasoning trajectory consists of low-entropy segments that encode stable and reusable structural patterns. Through qualitative and quantitative analyses, we find that the overlap of low-entropy segments across correct responses strongly correlates with model accuracy, while overlaps involving incorrect responses exhibit stable but unproductive patterns. Motivated by these findings, we propose LESS, a correctness-aware reinforcement framework that performs fine-grained advantage modulation over low-entropy segments. LESS amplifies segments unique to correct responses, suppresses those unique to incorrect ones, and neutralizes segments shared by both, while preserving high-entropy exploration in the underlying RL algorithm. Instantiated on top of the popular GRPO, LESS consistently improves accuracy over strong RL baselines across three backbones and six math benchmarks, achieves stronger robustness of the performance floor.
☆ Light-Weight Benchmark s Reveal the Hidden Hardware Cost of Zero-Shot Tabular Foundation Models ICML
Aayam Bansal, Ishaan Gangwani
Zero-shot foundation models (FMs) promise training-free prediction on tabular data, yet their hardware footprint remains poorly characterized. We present a fully reproducible benchmark that reports test accuracy together with wall-clock latency, peak CPU RAM, and peak GPU VRAM on four public datasets: Adult-Income, Higgs-100k, Wine-Quality, and California-Housing. Two open FMs (TabPFN-1.0 and TabICL-base) are compared against tuned XGBoost, LightGBM, and Random Forest baselines on a single NVIDIA T4 GPU. The tree ensembles equal or surpass FM accuracy on three datasets while completing full-test batches in <= 0.40 s and <= 150 MB RAM, using zero VRAM. TabICL achieves a 0.8 percentage-point gain on Higgs but requires roughly 40,000 times more latency (960 s) and 9 GB VRAM. TabPFN matches tree-model accuracy on Wine and Housing but peaks at 4 GB VRAM and cannot process the full 100k-row Higgs table. These results quantify the substantial hardware-versus-accuracy trade-offs in current tabular FMs and provide an open baseline for future efficiency-oriented research.
comment : ICML NewInML
☆ Look, Recite, Then Answer: Enhancing VLM Performance via Self-Generated Knowledge Hints
Xisheng Feng
Vision-Language Models (VLMs) exhibit significant performance plateaus in specialized domains like precision agriculture, primarily due to "Reasoning-Driven Hallucination" where linguistic priors override visual perception. A key bottleneck is the "Modality Gap": visual embeddings fail to reliably activate the fine-grained expert knowledge already encoded in model parameters. We propose "Look, Recite, Then Answer," a parameter-efficient framework that enhances VLMs via self-generated knowledge hints while keeping backbone models frozen. The framework decouples inference into three stages: (1) Look generates objective visual descriptions and candidate sets; (2) Recite employs a lightweight 1.7B router to transform visual cues into targeted queries that trigger candidate-specific parametric knowledge; (3) Answer performs parallel evidence alignment between descriptions and recited knowledge to select the most consistent label. On AgroBench, our method achieves state-of-the-art results, improving Weed Identification accuracy by 23.6% over Qwen-VL and surpassing GPT-4o without external search overhead. This modular design mitigates hallucinations by transforming passive perception into active, controllable knowledge retrieval
☆ Hybrid-DMKG: A Hybrid Reasoning Framework over Dynamic Multimodal Knowledge Graphs for Multimodal Multihop QA with Knowledge Editing AAAI 2026
Li Yuan, Qingfei Huang, Bingshan Zhu, Yi Cai, Qingbao Huang, Changmeng Zheng, Zikun Deng, Tao Wang
Multimodal Knowledge Editing (MKE) extends traditional knowledge editing to settings involving both textual and visual modalities. However, existing MKE benchmarks primarily assess final answer correctness while neglecting the quality of intermediate reasoning and robustness to visually rephrased inputs. To address this limitation, we introduce MMQAKE, the first benchmark for multimodal multihop question answering with knowledge editing. MMQAKE evaluates (1) a model's ability to reason over 2-5-hop factual chains that span both text and images, including performance at each intermediate step, and (2) robustness to visually rephrased inputs in multihop questions. Our evaluation shows that current MKE methods often struggle to consistently update and reason over multimodal reasoning chains after knowledge edits. To overcome these challenges, we propose Hybrid-DMKG, a hybrid reasoning framework built on a dynamic multimodal knowledge graph (DMKG) to enable accurate multihop reasoning over updated multimodal knowledge. Hybrid-DMKG first uses a large language model to decompose multimodal multihop questions into sequential sub-questions, then applies a multimodal retrieval model to locate updated facts by jointly encoding each sub-question with candidate entities and their associated images. For answer inference, a hybrid reasoning module operates over the DMKG via two parallel paths: (1) relation linking prediction, and (2) RAG reasoning with large vision-language models. A decision module aggregates evidence from both paths to select the most credible answer. Experimental results on MMQAKE show that Hybrid-DMKG significantly outperforms existing MKE approaches, achieving higher accuracy and improved robustness to knowledge updates.
comment : Accepted by AAAI 2026
☆ Less is More: Resource-Efficient Low-Rank Adaptation
Chunlin Tian, Xuyang Wei, Huanrong Liu, Zhijiang Guo, Li Li
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), but it still incurs notable overhead and suffers from parameter interference in complex datasets. While re- cent works decouple LoRA update matrices to exploit matrix-wise asymmetry, training costs remain high. We revisit LoRA from the perspective of inter-matrix and intra-layer parameter redundancy and propose Resource-Efficient Low-Rank Adaptation, EffiLoRA, a lightweight and generalizable approach for language, multimodal, and diffusion models. EffiLoRA employs a unified A matrix across all transformer layers and introduces a runtime selective B matrices up- date to dynamically trade-off the system resource budget and model performance. EffiLoRA consistently outperforms LoRA across diverse modalities, including commonsense reasoning, visual instruction tuning, and image generation, demon- strating improved efficiency and robustness.
comment : 18 pages, 7 figures
☆ TAP-CT: 3D Task-Agnostic Pretrain ing of Computed Tomography Foundation Models
Tim Veenboer, George Yiasemis, Eric Marcus, Vivien Van Veldhuizen, Cees G. M. Snoek, Jonas Teuwen, Kevin B. W. Groot Lipman
Existing foundation models (FMs) in the medical domain often require extensive fine-tuning or rely on training resource-intensive decoders, while many existing encoders are pretrained with objectives biased toward specific tasks. This illustrates a need for a strong, task-agnostic foundation model that requires minimal fine-tuning beyond feature extraction. In this work, we introduce a suite of task-agnostic pretraining of CT foundation models (TAP-CT): a simple yet effective adaptation of Vision Transformers (ViTs) and DINOv2 for volumetric data, enabling scalable self-supervised pretraining directly on 3D CT volumes. Our approach incorporates targeted modifications to patch embeddings, positional encodings, and volumetric augmentations, making the architecture depth-aware while preserving the simplicity of the underlying architectures. We show that large-scale 3D pretraining on an extensive in-house CT dataset (105K volumes) yields stable, robust frozen representations that generalize strongly across downstream tasks. To promote transparency and reproducibility, and to establish a powerful, low-resource baseline for future research in medical imaging, we will release all pretrained models, experimental configurations, and downstream benchmark code at https://huggingface.co/fomofo/tap-ct-b-3d.
comment : 22 pages, 4 figures, 8 tables
☆ HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMs
Ningning Chen, Weicai Ye, Ying Jiang
We introduce HBLLM, a wavelet-enhanced high-fidelity $1$-bit post-training quantization method for Large Language Models (LLMs). By leveraging Haar wavelet transforms to enhance expressive capacity through frequency decomposition, HBLLM significantly improves quantization fidelity while maintaining minimal overhead. This approach features two innovative structure-aware grouping strategies: (1) frequency-aware multi-parameter intra-row grouping and (2) $\ell_2$-norm-based saliency-driven column selection. For non-salient weights, a shared mean is employed across quantization groups within each frequency band to optimize storage efficiency. Experiments conducted on the OPT and LLaMA models demonstrate that HBLLM achieves state-of-the-art performance in $1$-bit quantization, attaining a perplexity of $6.71$ on LLaMA$2$-$13$B with an average weight storage of only $1.08$ bits. Code available at: https://github.com/Yeyke/HBLLM.
☆ One Swallow Does Not Make a Summer: Understanding Semantic Structures in Embedding Spaces
Yandong Sun, Qiang Huang, Ziwei Xu, Yiqun Sun, Yixuan Tang, Anthony K. H. Tung
Embedding spaces are fundamental to modern AI, translating raw data into high-dimensional vectors that encode rich semantic relationships. Yet, their internal structures remain opaque, with existing approaches often sacrificing semantic coherence for structural regularity or incurring high computational overhead to improve interpretability. To address these challenges, we introduce the Semantic Field Subspace (SFS), a geometry-preserving, context-aware representation that captures local semantic neighborhoods within the embedding space. We also propose SAFARI (SemAntic Field subspAce deteRmInation), an unsupervised, modality-agnostic algorithm that uncovers hierarchical semantic structures using a novel metric called Semantic Shift, which quantifies how semantics evolve as SFSes evolve. To ensure scalability, we develop an efficient approximation of Semantic Shift that replaces costly SVD computations, achieving a 15~30x speedup with average errors below 0.01. Extensive evaluations across six real-world text and image datasets show that SFSes outperform standard classifiers not only in classification but also in nuanced tasks such as political bias detection, while SAFARI consistently reveals interpretable and generalizable semantic hierarchies. This work presents a unified framework for structuring, analyzing, and scaling semantic understanding in embedding spaces.
☆ Topological Federated Clustering via Gravitational Potential Fields under Local Differential Privacy
Yunbo Long, Jiaquan Zhang, Xi Chen, Alexandra Brintrup
Clustering non-independent and identically distributed (non-IID) data under local differential privacy (LDP) in federated settings presents a critical challenge: preserving privacy while maintaining accuracy without iterative communication. Existing one-shot methods rely on unstable pairwise centroid distances or neighborhood rankings, degrading severely under strong LDP noise and data heterogeneity. We present Gravitational Federated Clustering (GFC), a novel approach to privacy-preserving federated clustering that overcomes the limitations of distance-based methods under varying LDP. Addressing the critical challenge of clustering non-IID data with diverse privacy guarantees, GFC transforms privatized client centroids into a global gravitational potential field where true cluster centers emerge as topologically persistent singularities. Our framework introduces two key innovations: (1) a client-side compactness-aware perturbation mechanism that encodes local cluster geometry as "mass" values, and (2) a server-side topological aggregation phase that extracts stable centroids through persistent homology analysis of the potential field's superlevel sets. Theoretically, we establish a closed-form bound between the privacy budget $ε$ and centroid estimation error, proving the potential field's Lipschitz smoothing properties exponentially suppress noise in high-density regions. Empirically, GFC outperforms state-of-the-art methods on ten benchmarks, especially under strong LDP constraints ($ε< 1$), while maintaining comparable performance at lower privacy budgets. By reformulating federated clustering as a topological persistence problem in a synthetic physics-inspired space, GFC achieves unprecedented privacy-accuracy trade-offs without iterative communication, providing a new perspective for privacy-preserving distributed learning.
☆ ARCADIA: Scalable Causal Discovery for Corporate Bankruptcy Analysis Using Agent ic AI
Fabrizio Maturo, Donato Riccio, Andrea Mazzitelli, Giuseppe Bifulco, Francesco Paolone, Iulia Brezeanu
This paper introduces ARCADIA, an agentic AI framework for causal discovery that integrates large-language-model reasoning with statistical diagnostics to construct valid, temporally coherent causal structures. Unlike traditional algorithms, ARCADIA iteratively refines candidate DAGs through constraint-guided prompting and causal-validity feedback, leading to stable and interpretable models for real-world high-stakes domains. Experiments on corporate bankruptcy data show that ARCADIA produces more reliable causal graphs than NOTEARS, GOLEM, and DirectLiNGAM while offering a fully explainable, intervention-ready pipeline. The framework advances AI by demonstrating how agentic LLMs can participate in autonomous scientific modeling and structured causal inference.
comment : 35 pages, 9 figures, 4 tables
☆ Assessing model error in counterfactual wo rl ds
Emily Howerton, Justin Lessler
Counterfactual scenario modeling exercises that ask "what would happen if?" are one of the most common ways we plan for the future. Despite their ubiquity in planning and decision making, scenario projections are rarely evaluated retrospectively. Differences between projections and observations come from two sources: scenario deviation and model miscalibration. We argue the latter is most important for assessing the value of models in decision making, but requires estimating model error in counterfactual worlds. Here we present and contrast three approaches for estimating this error, and demonstrate the benefits and limitations of each in a simulation experiment. We provide recommendations for the estimation of counterfactual error and discuss the components of scenario design that are required to make scenario projections evaluable.
☆ Sem Agent : Semantic-Driven Agent ic AI Empowered Trajectory Prediction in Vehicular Networks
Lin Zhu, Kezhi Wang, Luping Xiang, Kun Yang
Efficient information exchange and reliable contextual reasoning are essential for vehicle-to-everything (V2X) networks. Conventional communication schemes often incur significant transmission overhead and latency, while existing trajectory prediction models generally lack environmental perception and logical inference capabilities. This paper presents a trajectory prediction framework that integrates semantic communication with Agentic AI to enhance predictive performance in vehicular environments. In vehicle-to-infrastructure (V2I) communication, a feature-extraction agent at the Roadside Unit (RSU) derives compact representations from historical vehicle trajectories, followed by semantic reasoning performed by a semantic-analysis agent. The RSU then transmits both feature representations and semantic insights to the target vehicle via semantic communication, enabling the vehicle to predict future trajectories by combining received semantics with its own historical data. In vehicle-to-vehicle (V2V) communication, each vehicle performs local feature extraction and semantic analysis while receiving predicted trajectories from neighboring vehicles, and jointly utilizes this information for its own trajectory prediction. Extensive experiments across diverse communication conditions demonstrate that the proposed method significantly outperforms baseline schemes, achieving up to a 47.5% improvement in prediction accuracy under low signal-to-noise ratio (SNR) conditions.
comment : Submitted for possible journal publication
☆ Accelerating Bangla NLP Tasks with Automatic Mixed Precision: Resource-Efficient Training Preserving Model Efficacy
Md Mehrab Hossain Opi, Sumaiya Khan, Moshammad Farzana Rahman
Training models for Natural Language Processing (NLP) requires substantial computational resources and time, posing significant challenges, especially for NLP development in Bangla, where access to high-end hardware is often limited. In this work, we explore automatic mixed precision (AMP) training as a means to improve computational efficiency without sacrificing model performance. By leveraging a dynamic mix of 16-bit and 32-bit floating-point computations, AMP lowers GPU memory requirements and speeds up training without degrading model performance. We evaluate AMP across four standard Bangla NLP tasks, namely sentiment analysis, named entity recognition, error classification, and question answering, using four transformer-based models: BanglaBERT, BanglishBERT, XLM-R, and mBERT. Our results demonstrate that AMP accelerates training by 44.5% and reduces memory consumption by 17.6%, while maintaining F-1 score within 99.7% of the full-precision baselines. This empirical study highlights AMP's potential to democratize access to state-of-the-art NLP capabilities in hardware-constrained settings by lowering computational barriers.
☆ Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning
Haozhen Gong, Xiaozhong Ji, Yuansen Liu, Wenbin Wu, Xiaoxiao Yan, Jingjing Liu, Kai Wu, Jiazhen Pan, Bailiang Jian, Jiangning Zhang, Xiaobin Hu, Hongwei Bran Li
MLLMs MLLMs are beginning to appear in clinical workflows, but their ability to perform complex medical reasoning remains unclear. We present Med-CMR, a fine-grained Medical Complex Multimodal Reasoning benchmark. Med-CMR distinguishes from existing counterparts by three core features: 1) Systematic capability decomposition, splitting medical multimodal reasoning into fine-grained visual understanding and multi-step reasoning to enable targeted evaluation; 2) Challenging task design, with visual understanding across three key dimensions (small-object detection, fine-detail discrimination, spatial understanding) and reasoning covering four clinically relevant scenarios (temporal prediction, causal reasoning, long-tail generalization, multi-source integration); 3) Broad, high-quality data coverage, comprising 20,653 Visual Question Answering (VQA) pairs spanning 11 organ systems and 12 imaging modalities, validated via a rigorous two-stage (human expert + model-assisted) review to ensure clinical authenticity. We evaluate 18 state-of-the-art MLLMs with Med-CMR, revealing GPT-5 as the top-performing commercial model: 57.81 accuracy on multiple-choice questions (MCQs) and a 48.70 open-ended score, outperforming Gemini 2.5 Pro (49.87 MCQ accuracy, 45.98 open-ended score) and leading open-source model Qwen3-VL-235B-A22B (49.34 MCQ accuracy, 42.62 open-ended score). However, specialized medical MLLMs do not reliably outperform strong general models, and long-tail generalization emerges as the dominant failure mode. Med-CMR thus provides a stress test for visual-reasoning integration and rare-case robustness in medical MLLMs, and a rigorous yardstick for future clinical systems.
☆ Causal Invariance and Counterfactual Learning Driven Cooperative Game for Multi-Label Classification
Yijia Fan, Jusheng Zhang, Kaitong Cai, Jing Yang, Keze Wang
Multi-label classification (MLC) remains vulnerable to label imbalance, spurious correlations, and distribution shifts, challenges that are particularly detrimental to rare label prediction. To address these limitations, we introduce the Causal Cooperative Game (CCG) framework, which conceptualizes MLC as a cooperative multi-player interaction. CCG unifies explicit causal discovery via Neural Structural Equation Models with a counterfactual curiosity reward to drive robust feature learning. Furthermore, it incorporates a causal invariance loss to ensure generalization across diverse environments, complemented by a specialized enhancement strategy for rare labels. Extensive benchmarking demonstrates that CCG substantially outperforms strong baselines in both rare label prediction and overall robustness. Through rigorous ablation studies and qualitative analysis, we validate the efficacy and interpretability of our components, underscoring the potential of synergizing causal inference with cooperative game theory for advancing multi-label learning.
☆ BioPro: On Difference-Aware Gender Fairness for Vision -Language Models
Yujie Lin, Jiayao Ma, Qingguo Hu, Derek F. Wong, Jinsong Su
Vision-Language Models (VLMs) inherit significant social biases from their training data, notably in gender representation. Current fairness interventions often adopt a difference-unaware perspective that enforces uniform treatment across demographic groups. These approaches, however, fail to distinguish between contexts where neutrality is required and those where group-specific attributes are legitimate and must be preserved. Building upon recent advances in difference-aware fairness for text-only models, we extend this concept to the multimodal domain and formalize the problem of difference-aware gender fairness for image captioning and text-to-image generation. We advocate for selective debiasing, which aims to mitigate unwanted bias in neutral contexts while preserving valid distinctions in explicit ones. To achieve this, we propose BioPro (Bias Orthogonal Projection), an entirely training-free framework. BioPro identifies a low-dimensional gender-variation subspace through counterfactual embeddings and applies projection to selectively neutralize gender-related information. Experiments show that BioPro effectively reduces gender bias in neutral cases while maintaining gender faithfulness in explicit ones, thus providing a promising direction toward achieving selective fairness in VLMs. Beyond gender bias, we further demonstrate that BioPro can effectively generalize to continuous bias variables, such as scene brightness, highlighting its broader applicability.
☆ Bias Injection Attacks on RAG Databases and Sanitization Defenses
Hao Wu, Prateek Saxena
This paper explores attacks and defenses on vector databases in retrieval-augmented generation (RAG) systems. Prior work on knowledge poisoning attacks primarily inject false or toxic content, which fact-checking or linguistic analysis easily detects. We reveal a new and subtle threat: bias injection attacks, which insert factually correct yet semantically biased passages into the knowledge base to covertly influence the ideological framing of answers generated by large language models (LLMs). We demonstrate that these adversarial passages, though linguistically coherent and truthful, can systematically crowd out opposing views from the retrieved context and steer LLM answers toward the attacker's intended perspective. We precisely characterize this class of attacks and then develop a post-retrieval filtering defense, BiasDef. We construct a comprehensive benchmark based on public question answering datasets to evaluate them. Our results show that: (1) the proposed attack induces significant perspective shifts in LLM answers, effectively evading existing retrieval-based sanitization defenses; and (2) BiasDef outperforms existing methods by reducing adversarial passages retrieved by 15\% which mitigates perspective shift by 6.2\times in answers, while enabling the retrieval of 62\% more benign passages.
☆ Limitations of Using Identical Distributions for Training and Testing When Learning Boolean Functions
Jordi Pérez-Guijarro
When the distributions of the training and test data do not coincide, the problem of understanding generalization becomes considerably more complex, prompting a variety of questions. In this work, we focus on a fundamental one: Is it always optimal for the training distribution to be identical to the test distribution? Surprisingly, assuming the existence of one-way functions, we find that the answer is no. That is, matching distributions is not always the best scenario, which contrasts with the behavior of most learning methods. Nonetheless, we also show that when certain regularities are imposed on the target functions, the standard conclusion is recovered in the case of the uniform distribution.
☆ SHRAG: AFrameworkfor Combining Human-Inspired Search with RAG
Hyunseok Ryu, Wonjune Shin, Hyun Park
Retrieval-Augmented Generation (RAG) is gaining recognition as one of the key technological axes for next generation information retrieval, owing to its ability to mitigate the hallucination phenomenon in Large Language Models (LLMs)and effectively incorporate up-to-date information. However, specialized expertise is necessary to construct ahigh-quality retrieval system independently; moreover, RAGdemonstratesrelativelyslowerprocessing speeds compared to conventional pure retrieval systems because it involves both retrieval and generation stages. Accordingly, this study proposes SHRAG, a novel framework designed to facilitate the seamless integration of Information Retrieval and RAG while simultaneously securing precise retrieval performance. SHRAG utilizes a Large Language Model as a Query Strategist to automatically transform unstructured natural language queries into logically structured search queries, subsequently performing Boolean retrieval to emulate the search process of an expert human searcher. Furthermore, it incorporates multilingual query expansion and a multilingual embedding model, enabling it to perform efficient cross-lingual question answering within the multilingual dataset environment of the ScienceON Challenge. Experimental results demonstrate that the proposed method, combining logical retrieval capabilities and generative reasoning, can significantly enhance the accuracy and reliability of RAG systems. Furthermore, SHRAG movesbeyondconventionaldocument-centric retrieval methods, presenting the potential for a new search paradigm capable of providing direct and reliable responses to queries.
comment : 10 pages, 4 figures, 1 table, 1 algorithm, 3 prompts
☆ EAG3R: Event-Augmented 3D Geometry Estimation for Dynamic and Extreme-Lighting Scenes NeurIPS 2025
Xiaoshan Wu, Yifei Yu, Xiaoyang Lyu, Yihua Huang, Bo Wang, Baoheng Zhang, Zhongrui Wang, Xiaojuan Qi
Robust 3D geometry estimation from videos is critical for applications such as autonomous navigation, SLAM, and 3D scene reconstruction. Recent methods like DUSt3R demonstrate that regressing dense pointmaps from image pairs enables accurate and efficient pose-free reconstruction. However, existing RGB-only approaches struggle under real-world conditions involving dynamic objects and extreme illumination, due to the inherent limitations of conventional cameras. In this paper, we propose EAG3R, a novel geometry estimation framework that augments pointmap-based reconstruction with asynchronous event streams. Built upon the MonST3R backbone, EAG3R introduces two key innovations: (1) a retinex-inspired image enhancement module and a lightweight event adapter with SNR-aware fusion mechanism that adaptively combines RGB and event features based on local reliability; and (2) a novel event-based photometric consistency loss that reinforces spatiotemporal coherence during global optimization. Our method enables robust geometry estimation in challenging dynamic low-light scenes without requiring retraining on night-time data. Extensive experiments demonstrate that EAG3R significantly outperforms state-of-the-art RGB-only baselines across monocular depth estimation, camera pose tracking, and dynamic reconstruction tasks.
comment : Accepted at NeurIPS 2025 (spotlight)
☆ Provable Benefit of Sign Descent: A Minimal Model Under Heavy-Tailed Class Imbalance
Robin Yadav, Shuo Xie, Tianhao Wang, Zhiyuan Li
Adaptive optimization methods (such as Adam) play a major role in LLM pretraining, significantly outperforming Gradient Descent (GD). Recent studies have proposed new smoothness assumptions on the loss function to explain the advantages of adaptive algorithms with structured preconditioners, e.g., coordinate-wise or layer-wise, and steepest descent methods w.r.t. non-euclidean norms, e.g., $\ell_\infty$ norm or spectral norm, over GD. However, it remains unclear how these smoothness assumptions manifest in language modelling tasks. In this work, we aim to analyze the benefit of $\ell_\infty$-norm descent (a.k.a. sign descent) directly from properties of the data distribution, namely, heavy-tailed class imbalance. We propose a minimal yet representative setting of next-token prediction, where we can provably show faster convergence of coordinate-wise algorithms such as Sign descent (steepest descent w.r.t. $\ell_\infty$ norm) over normalized GD (steepest descent w.r.t. to $\ell_2$ norm) in the presence of heavy tail class imbalance.
☆ Preventing Model Collapse via Contraction-Conditioned Neural Filters
Zongjian Han, Yiran Liang, Ruiwen Wang, Yiwei Luo, Yilin Huang, Xiaotong Song, Dongqing Wei
This paper presents a neural network filter method based on contraction operators to address model collapse in recursive training of generative models. Unlike \cite{xu2024probabilistic}, which requires superlinear sample growth ($O(t^{1+s})$), our approach completely eliminates the dependence on increasing sample sizes within an unbiased estimation framework by designing a neural filter that learns to satisfy contraction conditions. We develop specialized neural network architectures and loss functions that enable the filter to actively learn contraction conditions satisfying Assumption 2.3 in exponential family distributions, thereby ensuring practical application of our theoretical results. Theoretical analysis demonstrates that when the learned contraction conditions are satisfied, estimation errors converge probabilistically even with constant sample sizes, i.e., $\limsup_{t\to\infty}\mathbb{P}(\|\mathbf{e}_t\|>δ)=0$ for any $δ>0$. Experimental results show that our neural network filter effectively learns contraction conditions and prevents model collapse under fixed sample size settings, providing an end-to-end solution for practical applications.
☆ MPR-GUI: Benchmark ing and Enhancing Multilingual Perception and Reasoning in GUI Agent s
Ruihan Chen, Qiming Li, Xiaocheng Feng, Xiaoliang Yang, Weihong Zhong, Yuxuan Gu, Zekun Zhou, Bing Qin
With the advancement of computational resources, Large Vision-Language Models (LVLMs) exhibit impressive Perception and Reasoning (P&R) performance on Graphical User Interface (GUI) tasks. However, although they demonstrate strong P&R capabilities in English GUI scenarios, their performance in multilingual settings has received little attention, which limits their global applications. Moreover, existing studies on GUI tasks lack fine-grained analyses, including widget functions and elements' spatial relationships, which are fundamental for more targeted improvements. To tackle these issues, we propose MPR-GUI-Bench, a Multilingual fine-grained Perception and Reasoning GUI Benchmark to evaluate GUI agents' P&R capabilities. Evaluation results demonstrate that LVLMs exhibit significantly worse P&R performance in non-English languages than in English. To address these gaps, we propose GUI-XLI, a GUI Cross-Lingual Intervention method that applies interventions to the hidden states at P&R capability-related layers to mitigate the gaps between English and other languages, building on previous research showing that the hidden states of different language inputs exhibit significant differences in the latent space. Experimental results indicate that our method improves GUI agents' multilingual P&R capability by 6.5% on average.
comment : 27pages, 12figures
☆ Probabilistic Modeling of Multi-rater Medical Image Segmentation for Diversity and Personalization
Ke Liu, Shangde Gao, Yichao Fu, Shangqi Gao, Chunhua Shen
Medical image segmentation is inherently influenced by data uncertainty, arising from ambiguous boundaries in medical scans and inter-observer variability in diagnosis. To address this challenge, previous works formulated the multi-rater medical image segmentation task, where multiple experts provide separate annotations for each image. However, existing models are typically constrained to either generate diverse segmentation that lacks expert specificity or to produce personalized outputs that merely replicate individual annotators. We propose Probabilistic modeling of multi-rater medical image Segmentation (ProSeg) that simultaneously enables both diversification and personalization. Specifically, we introduce two latent variables to model expert annotation preferences and image boundary ambiguity. Their conditional probabilistic distributions are then obtained through variational inference, allowing segmentation outputs to be generated by sampling from these distributions. Extensive experiments on both the nasopharyngeal carcinoma dataset (NPC) and the lung nodule dataset (LIDC-IDRI) demonstrate that our ProSeg achieves a new state-of-the-art performance, providing segmentation results that are both diverse and expert-personalized. Code can be found in https://github.com/AI4MOL/ProSeg.
☆ On the Regulatory Potential of User Interfaces for AI Agent Governance NeurIPS 2025
K. J. Kevin Feng, Tae Soo Kim, Rock Yuren Pang, Faria Huq, Tal August, Amy X. Zhang
AI agents that take actions in their environment autonomously over extended time horizons require robust governance interventions to curb their potentially consequential risks. Prior proposals for governing AI agents primarily target system-level safeguards (e.g., prompt injection monitors) or agent infrastructure (e.g., agent IDs). In this work, we explore a complementary approach: regulating user interfaces of AI agents as a way of enforcing transparency and behavioral requirements that then demand changes at the system and/or infrastructure levels. Specifically, we analyze 22 existing agentic systems to identify UI elements that play key roles in human-agent interaction and communication. We then synthesize those elements into six high-level interaction design patterns that hold regulatory potential (e.g., requiring agent memory to be editable). We conclude with policy recommendations based on our analysis. Our work exposes a new surface for regulatory action that supplements previous proposals for practical AI agent governance.
comment : RegML workshop at NeurIPS 2025 (oral)
☆ MAS COT : Analyzing Malware Evolution Through A Well-Curated Source Code Dataset
Bojing Li, Duo Zhong, Dharani Nadendla, Gabriel Terceros, Prajna Bhandar, Raguvir S, Charles Nicholas
In recent years, the explosion of malware and extensive code reuse have formed complex evolutionary connections among malware specimens. The rapid pace of development makes it challenging for existing studies to characterize recent evolutionary trends. In addition, intuitive tools to untangle these intricate connections between malware specimens or categories are urgently needed. This paper introduces a manually-reviewed malware source code dataset containing 6032 specimens. Building on and extending current research from a software engineering perspective, we systematically evaluate the scale, development costs, code quality, as well as security and dependencies of modern malware. We further introduce a multi-view genealogy analysis to clarify malware connections: at an overall view, this analysis quantifies the strength and direction of connections among specimens and categories; at a detailed view, it traces the evolutionary histories of individual specimens. Experimental results indicate that, despite persistent shortcomings in code quality, malware specimens exhibit an increasing complexity and standardization, in step with the development of mainstream software engineering practices. Meanwhile, our genealogy analysis intuitively reveals lineage expansion and evolution driven by code reuse, providing new evidence and tools for understanding the formation and evolution of the malware ecosystem.
comment : 11 pages, 6 figures, conference paper; submitted to IEEE BigData 2025 CyberHunt workshop
☆ Orchestrating Rewards in the Era of Intelligence-Driven Commerce
Paul Osemudiame Oamen, Robert Wesley, Pius Onobhayedo
Despite their evolution from early copper-token schemes to sophisticated digital solutions, loyalty programs remain predominantly closed ecosystems, with brands retaining full control over all components. Coalition loyalty programs emerged to enable cross-brand interoperability, but approximately 60\% fail within 10 years in spite of theoretical advantages rooted in network economics. This paper demonstrates that coalition failures stem from fundamental architectural limitations in centralized operator models rather than operational deficiencies, and argues further that neither closed nor coalition systems can scale in intelligence-driven paradigms where AI agents mediate commerce and demand trustless, protocol-based coordination that existing architectures cannot provide. We propose a hybrid framework where brands maintain sovereign control over their programs while enabling cross-brand interoperability through trustless exchange mechanisms. Our framework preserves closed system advantages while enabling open system benefits without the structural problems that doom traditional coalitions. We derive a mathematical pricing model accounting for empirically-validated market factors while enabling fair value exchange across interoperable reward systems.
☆ REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories
Jacob Thompson, Emiliano Garcia-Lopez, Yonatan Bisk
Humans build viewpoint-independent cognitive maps through navigation, enabling intuitive reasoning about object permanence and spatial relations. We argue that multimodal large language models (MLLMs), despite extensive video training, lack this fundamental spatial reasoning capability, a critical limitation for embodied applications. To demonstrate these limitations and drive research, we introduce REM (Reasoning over Embodied Multi-Frame Trajectories), a benchmark using controllable 3D environments for long-horizon embodied spatial reasoning. REM systematically evaluates key aspects like object permanence/distinction, spatial relationships, and numerical tracking across dynamic embodied viewpoints. Our evaluation shows that the best-performing current models exhibit promising overall performance, but become increasingly unreliable at even moderate complexity levels easily handled by humans. These findings highlight challenges MLLMs face in developing robust spatial representations from sequential visual input. Consequently, REM provides targeted metrics and diagnostics to foster improved spatial understanding in future models.
☆ Probing the "Psyche'' of Large Reasoning Models: Understanding Through a Human Lens
Yuxiang Chen, Zuohan Wu, Ziwei Wang, Xiangning Yu, Xujia Li, Linyi Yang, Mengyue Yang, Jun Wang, Lei Chen
Large reasoning models (LRMs) have garnered significant attention from researchers owing to their exceptional capability in addressing complex tasks. Motivated by the observed human-like behaviors in their reasoning processes, this paper introduces a comprehensive taxonomy to characterize atomic reasoning steps and probe the ``psyche'' of LRM intelligence. Specifically, it comprises five groups and seventeen categories derived from human mental processes, thereby grounding the understanding of LRMs in an interdisciplinary perspective. The taxonomy is then applied for an in-depth understanding of current LRMs, resulting in a distinct labeled dataset that comprises 277,534 atomic reasoning steps. Using this resource, we analyze contemporary LRMs and distill several actionable takeaways for improving training and post-training of reasoning models. Notably, our analysis reveals that prevailing post-answer ``double-checks'' (self-monitoring evaluations) are largely superficial and rarely yield substantive revisions. Thus, incentivizing comprehensive multi-step reflection, rather than simple self-monitoring, may offer a more effective path forward. To complement the taxonomy, an automatic annotation framework, named CAPO, is proposed to leverage large language models (LLMs) for generating the taxonomy-based annotations. Experimental results demonstrate that CAPO achieves higher consistency with human experts compared to baselines, facilitating a scalable and comprehensive analysis of LRMs from a human cognitive perspective. Together, the taxonomy, CAPO, and the derived insights provide a principled, scalable path toward understanding and advancing LRM reasoning.
comment : 13 pages
☆ Deep Learning for Modeling and Dispatching Hybrid Wind Farm Power Generation ICDM
Zach Lawrence, Jessica Yao, Chris Qin
Wind farms with integrated energy storage, or hybrid wind farms, are able to store energy and dispatch it to the grid following an operational strategy. For individual wind farms with integrated energy storage capacity, data-driven dispatch strategies using localized grid demand and market conditions as input parameters stand to maximize wind energy value. Synthetic power generation data modeled on atmospheric conditions provide another avenue for improving the robustness of data-driven dispatch strategies. To these ends, the present work develops two deep learning frameworks: COVE-NN, an LSTM-based dispatch strategy tailored to individual wind farms, which reduced annual COVE by 32.3% over 43 years of simulated operations in a case study at the Pyron site; and a power generation modeling framework that reduced RMSE by 9.5% and improved power curve similarity by 18.9% when validated on the Palouse wind farm. Together, these models pave the way for more robust, data-driven dispatch strategies and potential extensions to other renewable energy systems.
comment : 10 pages, 8 figures, to be published in 2025 IEEE International Conference on Data Mining Workshops (ICDMW)
☆ Spe Context : Enabling Efficient Long- context Reasoning with Speculative Context Sparsity in LLMs ASPLOS 2026
Jiaming Xu, Jiayi Pan, Hanzhen Wang, Yongkang Zhou, Jiancai Ye, Yu Wang, Guohao Dai
In this paper, we point out that the objective of the retrieval algorithms is to align with the LLM, which is similar to the objective of knowledge distillation in LLMs. We analyze the similarity in information focus between the distilled language model(DLM) and the original LLM from the perspective of information theory, and thus propose a novel paradigm that leverages a DLM as the retrieval algorithm. Based on the insight, we present SpeContext, an algorithm and system co-design for long-context reasoning. (1) At the algorithm level, SpeContext proposes lightweight retrieval head based on the head-level attention weights of DLM, achieving > 90% parameters reduction by pruning the redundancy. (2) At the system level, SpeContext designs an asynchronous prefetch dataflow via the elastic loading strategy, effectively overlapping KV cache retrieval with the LLM computation. (3) At the compilation level, SpeContext constructs the theoretical memory model and implements an adaptive memory management system to achieve acceleration by maximizing GPU memory utilization. We deploy and evaluate SpeContext in two resourceconstrained environments, cloud and edge. Extensive experiments show that, compared with the Huggingface framework, SpeContext achieves up to 24.89x throughput improvement in cloud and 10.06x speedup in edge with negligible accuracy loss, pushing the Pareto frontier of accuracy and throughput.
comment : Accepted by ASPLOS 2026
☆ Graph Data Augmentation with Contrastive Learning on Covariate Distribution Shift
Fanlong Zeng, Wensheng Gan
Covariate distribution shift occurs when certain structural features present in the test set are absent from the training set. It is a common type of out-of-distribution (OOD) problem, frequently encountered in real-world graph data with complex structures. Existing research has revealed that most out-of-the-box graph neural networks (GNNs) fail to account for covariate shifts. Furthermore, we observe that existing methods aimed at addressing covariate shifts often fail to fully leverage the rich information contained within the latent space. Motivated by the potential of the latent space, we introduce a new method called MPAIACL for More Powerful Adversarial Invariant Augmentation using Contrastive Learning. MPAIACL leverages contrastive learning to unlock the full potential of vector representations by harnessing their intrinsic information. Through extensive experiments, MPAIACL demonstrates its robust generalization and effectiveness, as it performs well compared with other baselines across various public OOD datasets. The code is publicly available at https://github.com/flzeng1/MPAIACL.
comment : 8 tables, 8 figures
☆ Deep Learning-Based Computer Vision Models for Ea rl y Cancer Detection Using Multimodal Medical Imaging and Radiogenomic Integration Frameworks
Emmanuella Avwerosuoghene Oghenekaro
Early cancer detection remains one of the most critical challenges in modern healthcare, where delayed diagnosis significantly reduces survival outcomes. Recent advancements in artificial intelligence, particularly deep learning, have enabled transformative progress in medical imaging analysis. Deep learning-based computer vision models, such as convolutional neural networks (CNNs), transformers, and hybrid attention architectures, can automatically extract complex spatial, morphological, and temporal patterns from multimodal imaging data including MRI, CT, PET, mammography, histopathology, and ultrasound. These models surpass traditional radiological assessment by identifying subtle tissue abnormalities and tumor microenvironment variations invisible to the human eye. At a broader scale, the integration of multimodal imaging with radiogenomics linking quantitative imaging features with genomics, transcriptomics, and epigenetic biomarkers has introduced a new paradigm for personalized oncology. This radiogenomic fusion allows the prediction of tumor genotype, immune response, molecular subtypes, and treatment resistance without invasive biopsies.
☆ Concept-Guided Backdoor Attack on Vision Language Models
Haoyu Shen, Weimin Lyu, Haotian Xu, Tengfei Ma
Vision-Language Models (VLMs) have achieved impressive progress in multimodal text generation, yet their rapid adoption raises increasing concerns about security vulnerabilities. Existing backdoor attacks against VLMs primarily rely on explicit pixel-level triggers or imperceptible perturbations injected into images. While effective, these approaches reduce stealthiness and remain vulnerable to image-based defenses. We introduce concept-guided backdoor attacks, a new paradigm that operates at the semantic concept level rather than on raw pixels. We propose two different attacks. The first, Concept-Thresholding Poisoning (CTP), uses explicit concepts in natural images as triggers: only samples containing the target concept are poisoned, causing the model to behave normally in all other cases but consistently inject malicious outputs whenever the concept appears. The second, CBL-Guided Unseen Backdoor (CGUB), leverages a Concept Bottleneck Model (CBM) during training to intervene on internal concept activations, while discarding the CBM branch at inference time to keep the VLM unchanged. This design enables systematic replacement of a targeted label in generated text (for example, replacing "cat" with "dog"), even when the replacement behavior never appears in the training data. Experiments across multiple VLM architectures and datasets show that both CTP and CGUB achieve high attack success rates while maintaining moderate impact on clean-task performance. These findings highlight concept-level vulnerabilities as a critical new attack surface for VLMs.
☆ When Human Preferences Flip: An Instance-Dependent Robust Loss for RL HF AAAI-26
Yifan Xu, Xichen Ye, Yifan Chen, Qiaosheng Zhang
Quality of datasets plays an important role in large language model (LLM) alignment. In collecting human feedback, however, preference flipping is ubiquitous and causes corruption in data annotation; the issue necessitates the alignment algorithms with improved robustness against potential flipped pairs. To this end, this paper introduces a Flipping-Aware Direct Preference Optimization (FA-DPO) algorithm tailored to preference flipping from a reinforcement learning with human feedback (RLHF) perspective. We dissect the inherent human intention model and the preference flipping mechanism introduced by external factors as two distinct stages; in the latter, we introduce an instance-dependent flipping probability on the basis of the Bradley-Terry (BT) model. Further, by leveraging features relevant to preference annotation, we capture uncertainty in judgments and model preference flipping patterns. In practice, we design a simple yet efficient iterative optimization algorithm compatible with the original RLHF and DPO algorithms. In our experiments, we investigate the instance-dependent preference flipping model under multiple circumstances for evaluation of our proposed method, as well as other baseline methods.
comment : Accepted by AAAI-26-AIA
☆ Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigation
Chengzhi Yu, Yifan Xu, Yifan Chen, Wenyi Zhang
Recently, large vision-language models (LVLMs) have risen to be a promising approach for multimodal tasks. However, principled hallucination mitigation remains a critical challenge.In this work, we first analyze the data generation process in LVLM hallucination mitigation and affirm that on-policy data significantly outperforms off-policy data, which thus calls for efficient and reliable preference annotation of on-policy data. We then point out that, existing annotation methods introduce additional hallucination in training samples, which may enhance the model's hallucination patterns, to address this problem, we propose training a hallucination classifier giving binary annotations, which guarantee clean chosen samples for the subsequent alignment. To further harness of the power of on-policy data, we design a robust iterative direct preference optimization (DPO) algorithm adopting a dynamic sample reweighting scheme. We conduct comprehensive experiments on three benchmarks with comparison to 8 state-of-the-art baselines. In particular, our approach reduces the hallucination rate of LLaVA-1.5-7B on MMHalBench by 50.8% and the average hallucination rate on Object HalBench by 79.5%; more significantly, our method fully taps into the potential of open-source models, enabling LLaVA-1.5-13B to even surpass the performance of GPT-4V.
☆ Hierarchical Molecular Language Models (HMLMs)
Hasi Hays, Yue Yu, William Richardson
Cellular signaling networks represent complex information processing systems that have been modeled via traditional mathematical or statistical approaches. However, these methods often struggle to capture context-dependent signaling, pathway cross-talk, and temporal dynamics across multiple biological scales. Here, we introduce hierarchical molecular language models (HMLMs), a novel architecture that proposes a molecular network-specific large language model (LLM) to use in intracellular communication as a specialized molecular language, which includes molecules as tokens, protein interactions, post-translational modifications, and regulatory events modeled as semantic relationships within an adapted transformer architecture. HMLMs employ graph-structured attention mechanisms to accommodate signaling network topology while integrating information across the molecular, pathway, and cellular scales through hierarchical attention patterns. We demonstrate HMLM superiority using a cardiac fibroblast signaling network comprising over 100 molecular species across functional modules connected by regulatory edges. HMLM achieved a mean squared error (MSE) of 0.058 for temporal signaling predictions, representing 30% improvement over graph neural networks (GNNs: 0.083) and 52% improvement over ordinary differential equation models (ODEs: 0.121), with particular advantages under sparse temporal sampling conditions where HMLM maintained MSE = 0.041 with only 4 time-points. The HMLMs offer a foundation for AI-driven biology and medicine with predictable scaling characteristics suitable for interactive applications. By bridging molecular mechanisms with cellular phenotypes through AI-driven molecular language representation, HMLMs provide a powerful paradigm for systems biology that advances precision medicine applications and therapeutic discovery in the era of AI.
☆ Model of human cognition
Wu Yonggang
The development of large language models (LLMs) is limited by a lack of explainability, the absence of a unifying theory, and prohibitive operational costs. We propose a neuro-theoretical framework for the emergence of intelligence in systems that is both functionally robust and biologically plausible. The model provides theoretical insights into cognitive processes such as decision-making and problem solving, and a computationally efficient approach for the creation of explainable and generalizable artificial intelligence.
☆ Dynamic-eDiTor: Training-Free Text-Driven 4D Scene Editing with Multimodal Diffusion Transformer
Dong In Lee, Hyungjun Doh, Seunggeun Chi, Runlin Duan, Sangpil Kim, Karthik Ramani
Recent progress in 4D representations, such as Dynamic NeRF and 4D Gaussian Splatting (4DGS), has enabled dynamic 4D scene reconstruction. However, text-driven 4D scene editing remains under-explored due to the challenge of ensuring both multi-view and temporal consistency across space and time during editing. Existing studies rely on 2D diffusion models that edit frames independently, often causing motion distortion, geometric drift, and incomplete editing. We introduce Dynamic-eDiTor, a training-free text-driven 4D editing framework leveraging Multimodal Diffusion Transformer (MM-DiT) and 4DGS. This mechanism consists of Spatio-Temporal Sub-Grid Attention (STGA) for locally consistent cross-view and temporal fusion, and Context Token Propagation (CTP) for global propagation via token inheritance and optical-flow-guided token replacement. Together, these components allow Dynamic-eDiTor to perform seamless, globally consistent multi-view video without additional training and directly optimize pre-trained source 4DGS. Extensive experiments on multi-view video dataset DyNeRF demonstrate that our method achieves superior editing fidelity and both multi-view and temporal consistency prior approaches. Project page for results and code: https://di-lee.github.io/dynamic-eDiTor/
comment : 4D Scene Editing
☆ The 4/$δ$ Bound: Designing Predictable LLM-Verifier Systems for Formal Method Guarantee
PIerre Dantas, Lucas Cordeiro, Youcheng Sun, Waldir Junior
The idea of using Formal Verification tools with large language models (LLMs) has enabled scaling software verification beyond manual workflows. However, current methods remain unreliable. Without a solid theoretical footing, the refinement process can wander; sometimes it settles, sometimes it loops back, and sometimes it breaks away from any stable trajectory. This work bridges this critical gap by developing an LLM-Verifier Convergence Theorem, providing the first formal framework with provable guarantees for termination and convergence. We model the interaction between the LLM and the verifier as a discrete-time Markov Chain, with state transitions determined by a key parameter: the error-reduction probability ($δ$). The procedure reaching the Verified state almost surely demonstrates that the program terminates for any $δ> 0$, with an expected iteration count bounded by $\mathbb{E}[n] \leq 4/δ$. We then stress-tested this prediction in an extensive empirical campaign comprising more than 90,000 trials. The empirical results match the theory with striking consistency. Every single run reached verification, and the convergence factor clustered tightly around $C_f\approx$ 1.0. Consequently, the bound mirrors the system's actual behavior. The evidence is sufficiently robust to support dividing the workflow into three distinct operating zones: marginal, practical, and high-performance. Consequently, we establish the design thresholds with absolute confidence. Together, the theoretical guarantee and the experimental evidence provide a clearer architectural foundation for LLM-assisted verification. Heuristic tuning no longer has to be carried out by the system. Engineers gain a framework that supports predictable resource planning and performance budgeting, precisely what is needed before deploying these pipelines into safety-critical software environments.
comment : 32 pages, 9 figures
☆ FDRMFL:Multi-modal Federated Feature Extraction Model Based on Information Maximization and Contrastive Learning
Haozhe Wu
This study focuses on the feature extraction problem in multi-modal data regression. To address three core challenges in real-world scenarios: limited and non-IID data, effective extraction and fusion of multi-modal information, and susceptibility to catastrophic forgetting in model learning, a task-driven supervised multi-modal federated feature extraction method is proposed. The method integrates multi-modal information extraction and contrastive learning mechanisms, and can adapt to different neural network structures as the latent mapping functions for data of each modality. It supports each client to independently learn low-dimensional representations of multi-modal data, and can flexibly control the degree of retention of effective information about the response variable in the predictive variables within the low-dimensional features through parameter tuning. The multi-constraint learning framework constructed by the method guarantees regression accuracy using Mean Squared Error loss. Through the synergistic effect of mutual information preservation constraint, symmetric Kullback-Leibler divergence constraint, and inter-model contrastive constraint, it achieves the retention of task-related information, the extraction, fusion, and alignment of multi-modal features, and the mitigation of representation drift and catastrophic forgetting in non-IID scenarios, respectively. This ensures that the feature extraction process always centers on improving the performance of downstream regression tasks. Experimental results from simulations and real-world data analysis demonstrate that the proposed method achieves more significant performance improvement on downstream regression tasks compared with classical feature extraction techniques.
comment : 14pages,6figures
♻ ☆ Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agent s in Open-Ended Environments NeurIPS 2025
Riley Simmons-Edler, Ryan P. Badman, Felix Baastad Berg, Raymond Chua, John J. Vastola, Joshua Lunger, William Qian, Kanaka Rajan
Understanding the behavior of deep reinforcement learning (DRL) agents -particularly as task and agent sophistication increase- requires more than simple comparison of reward curves, yet standard methods for behavioral analysis remain underdeveloped in DRL. We apply tools from neuroscience and ethology to study DRL agents in a novel, complex, partially observable environment, ForageWorld, designed to capture key aspects of real-world animal foraging- including sparse, depleting resource patches, predator threats, and spatially extended arenas. We use this environment as a platform for applying joint behavioral and neural analysis to agents, revealing detailed, quantitatively grounded insights into agent strategies, memory, and planning. Contrary to common assumptions, we find that model-free RNN-based DRL agents can exhibit structured, planning-like behavior purely through emergent dynamics- without requiring explicit memory modules or world models. Our results show that studying DRL agents like animals -analyzing them with neuroethology-inspired tools that reveal structure in both behavior and neural dynamics- uncovers rich structure in their learning dynamics that would otherwise remain invisible. We distill these tools into a general analysis framework linking core behavioral and representational features to diagnostic methods, which can be reused for a wide range of tasks and agents. As agents grow more complex and autonomous, bridging neuroscience, cognitive science, and AI will be essential- not just for understanding their behavior, but for ensuring safe alignment and maximizing desirable behaviors that are hard to measure via reward. We show how this can be done by drawing on lessons from how biological intelligence is studied.
comment : Published at NeurIPS 2025
♻ ☆ PETAR: Localized Findings Generation with Mask-Aware Vision -Language Modeling for PET Automated Reporting
Danyal Maqbool, Changhee Lee, Zachary Huemann, Samuel D. Church, Matthew E. Larson, Scott B. Perlman, Tomas A. Romero, Joshua D. Warner, Meghan Lubner, Xin Tie, Jameson Merkow, Junjie Hu, Steve Y. Cho, Tyler J. Bradshaw
Generating automated reports for 3D positron emission tomography (PET) is an important and challenging task in medical imaging. PET plays a vital role in oncology, but automating report generation is difficult due to the complexity of whole-body 3D volumes, the wide range of potential clinical findings, and the limited availability of annotated datasets. To address these challenges, we introduce PETARSeg-11K, the first large-scale, publicly available dataset that provides lesion-level correspondence between 3D PET/CT volumes and free-text radiological findings. It comprises 11,356 lesion descriptions paired with 3D segmentations. Second, we propose PETAR-4B, a 3D vision-language model designed for mask-aware, spatially grounded PET/CT reporting. PETAR-4B jointly encodes PET, CT, and 3D lesion segmentation masks, using a 3D focal prompt to capture fine-grained details of lesions that normally comprise less than 0.1% of the volume. Evaluations using automated metrics show PETAR-4B substantially outperforming all 2D and 3D baselines. A human study involving five physicians -- the first of its kind for automated PET reporting -- confirms the model's clinical utility and establishes correlations between automated metrics and expert judgment. This work provides a foundational dataset and a novel architecture, advancing 3D medical vision-language understanding in PET.
♻ ☆ How Similar Are Grokipedia and Wikipedia? A Multi-Dimensional Textual and Structural Comparison
Taha Yasseri, Saeedeh Mohammadi
The launch of Grokipedia - an AI-generated encyclopedia developed by Elon Musk's xAI - was presented as a response to perceived ideological and structural biases in Wikipedia, aiming to produce "truthful" entries using the Grok large language model. Yet whether an AI-driven alternative can escape the biases and limitations of human-edited platforms remains unclear. This study conducts a large-scale computational comparison of more than 17,000 matched article pairs from the 20,000 most-edited English Wikipedia pages. Using metrics spanning lexical richness, readability, reference density, structural features, and semantic similarity, we assess how closely the two platforms align in form and substance. We find that Grokipedia articles are substantially longer and contain significantly fewer references per word. Moreover, Grokipedia's content divides into two distinct groups: one that remains semantically and stylistically aligned with Wikipedia, and another that diverges sharply. Among the dissimilar articles, we observe a systematic rightward shift in the political bias of cited news sources, concentrated primarily in entries related to politics, history, and religion. These findings suggest that AI-generated encyclopedic content diverges from established editorial norms-favouring narrative expansion over citation-based verification. The implications highlight emerging tensions around transparency, provenance, and the governance of knowledge in an era of automated text generation.
comment : 20 pages, 8 figures, 2 tables, updated with a larger sample size of 20,000 articles, better text cleaning procedure + Reference analysis, topical analysis
♻ ☆ Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning , and Conversation History AAAI 2026
Tommaso Tosato, Saskia Helbling, Yorguin-Jose Mantilla-Ramos, Mahmood Hegazy, Alberto Tosato, David John Lemay, Irina Rish, Guillaume Dumas
Large language models require consistent behavioral patterns for safe deployment, yet there are indications of large variability that may lead to an instable expression of personality traits in these models. We present PERSIST (PERsonality Stability in Synthetic Text), a comprehensive evaluation framework testing 25 open-source models (1B-685B parameters) across 2 million+ responses. Using traditional (BFI, SD3) and novel LLM-adapted personality questionnaires, we systematically vary model size, personas, reasoning modes, question order or paraphrasing, and conversation history. Our findings challenge fundamental assumptions: (1) Question reordering alone can introduce large shifts in personality measurements; (2) Scaling provides limited stability gains: even 400B+ models exhibit standard deviations >0.3 on 5-point scales; (3) Interventions expected to stabilize behavior, such as reasoning and inclusion of conversation history, can paradoxically increase variability; (4) Detailed persona instructions produce mixed effects, with misaligned personas showing significantly higher variability than the helpful assistant baseline; (5) The LLM-adapted questionnaires, despite their improved ecological validity, exhibit instability comparable to human-centric versions. This persistent instability across scales and mitigation strategies suggests that current LLMs lack the architectural foundations for genuine behavioral consistency. For safety-critical applications requiring predictable behavior, these findings indicate that current alignment strategies may be inadequate.
comment : Accepted at AAAI 2026, Track on AI Alignment
♻ ☆ Can-SAVE: Deploying Low-Cost and Population-Scale Cancer Screening via Survival Analysis Variables and EHR
Petr Philonenko, Vladimir Kokh, Pavel Blinov
Conventional medical cancer screening methods are costly, labor-intensive, and extremely difficult to scale. Although AI can improve cancer detection, most systems rely on complex or specialized medical data, making them impractical for large-scale screening. We introduce Can-SAVE, a lightweight AI system that ranks population-wide cancer risks solely based on medical history events. By integrating survival model outputs into a gradient-boosting framework, our approach detects subtle, long-term patient risk patterns - often well before clinical symptoms manifest. Can-SAVE was rigorously evaluated on a real-world dataset of 2.5 million adults spanning five Russian regions, marking the study as one of the largest and most comprehensive deployments of AI-driven cancer risk assessment. In a retrospective oncologist-supervised study over 1.9M patients, Can-SAVE achieves a 4-10x higher detection rate at identical screening volumes and an Average Precision (AP) of 0.228 vs. 0.193 for the best baseline (LoRA-tuned Qwen3-Embeddings via DeepSeek-R1 summarization). In a year-long prospective pilot (426K patients), our method almost doubled the cancer detection rate (+91%) and increased population coverage by 36% over the national screening protocol. The system demonstrates practical scalability: a city-wide population of 1 million patients can be processed in under three hours using standard hardware, enabling seamless clinical integration. This work proves that Can-SAVE achieves nationally significant cancer detection improvements while adhering to real-world public healthcare constraints, offering immediate clinical utility and a replicable framework for population-wide screening. Code for training and feature engineering is available at https://github.com/sb-ai-lab/Can-SAVE.
comment : 11 pages, 3 figures
♻ ☆ Quantifying Cognitive Bias Induction in LLM-Generated Content AACL 2025
Abeer Alessa, Param Somane, Akshaya Lakshminarasimhan, Julian Skirzynski, Julian McAuley, Jessica Echterhoff
Large language models (LLMs) are integrated into applications like shopping reviews, summarization, or medical diagnosis support, where their use affects human decisions. We investigate the extent to which LLMs expose users to biased content and demonstrate its effect on human decision-making. We assess five LLM families in summarization and news fact-checking tasks, evaluating the consistency of LLMs with their context and their tendency to hallucinate on a new self-updating dataset. Our findings show that LLMs expose users to content that changes the context's sentiment in 26.42% of cases (framing bias), hallucinate on 60.33% of post-knowledge-cutoff questions, and highlight context from earlier parts of the prompt (primacy bias) in 10.12% of cases, averaged across all tested models. We further find that humans are 32% more likely to purchase the same product after reading a summary of the review generated by an LLM rather than the original review. To address these issues, we evaluate 18 mitigation methods across three LLM families and find the effectiveness of targeted interventions.
comment : 21 pages (including references and appendix), 3figures. accepted to AACL 2025
♻ ☆ Blind Inverse Problem Solving Made Easy by Text-to-Image Latent Diffusion
Michail Dontas, Yutong He, Naoki Murata, Yuki Mitsufuji, J. Zico Kolter, Ruslan Salakhutdinov
This paper considers blind inverse image restoration, the task of predicting a target image from a degraded source when the degradation (i.e. the forward operator) is unknown. Existing solutions typically rely on restrictive assumptions such as operator linearity, curated training data or narrow image distributions limiting their practicality. We introduce LADiBI, a training-free method leveraging large-scale text-to-image diffusion to solve diverse blind inverse problems with minimal assumptions. Within a Bayesian framework, LADiBI uses text prompts to jointly encode priors for both target images and operators, unlocking unprecedented flexibility compared to existing methods. Additionally, we propose a novel diffusion posterior sampling algorithm that combines strategic operator initialization with iterative refinement of image and operator parameters, eliminating the need for highly constrained operator forms. Experiments show that LADiBI effectively handles both linear and challenging nonlinear image restoration problems across various image distributions, all without task-specific assumptions or retraining.
♻ ☆ DeepPersona: A Generative Engine for Scaling Deep Synthetic Personas NeurIPS 2025
Zhen Wang, Yufan Zhou, Zhongyan Luo, Lyumanshan Ye, Adam Wood, Man Yao, Saab Mansour, Luoshang Pan
Simulating human profiles by instilling personas into large language models (LLMs) is rapidly transforming research in agentic behavioral simulation, LLM personalization, and human-AI alignment. However, most existing synthetic personas remain shallow and simplistic, capturing minimal attributes and failing to reflect the rich complexity and diversity of real human identities. We introduce DEEPPERSONA, a scalable generative engine for synthesizing narrative-complete synthetic personas through a two-stage, taxonomy-guided method. First, we algorithmically construct the largest-ever human-attribute taxonomy, comprising over hundreds of hierarchically organized attributes, by mining thousands of real user-ChatGPT conversations. Second, we progressively sample attributes from this taxonomy, conditionally generating coherent and realistic personas that average hundreds of structured attributes and roughly 1 MB of narrative text, two orders of magnitude deeper than prior works. Intrinsic evaluations confirm significant improvements in attribute diversity (32 percent higher coverage) and profile uniqueness (44 percent greater) compared to state-of-the-art baselines. Extrinsically, our personas enhance GPT-4.1-mini's personalized question answering accuracy by 11.6 percent on average across ten metrics and substantially narrow (by 31.7 percent) the gap between simulated LLM citizens and authentic human responses in social surveys. Our generated national citizens reduced the performance gap on the Big Five personality test by 17 percent relative to LLM-simulated citizens. DEEPPERSONA thus provides a rigorous, scalable, and privacy-free platform for high-fidelity human simulation and personalized AI research.
comment : add an author[Update], 12 pages, 5 figures, accepted at LAW 2025 Workshop (NeurIPS 2025) Project page: https://deeppersona-ai.github.io/
♻ ☆ DynaStride: Dynamic Stride Windowing with MM CoT for Instructional Multi-Scene Captioning NeurIPS 7
Eddison Pham, Prisha Priyadarshini, Adrian Maliackel, Kanishk Bandi, Cristian Meo, Kevin Zhu
Scene-level captioning in instructional videos can enhance learning by requiring an understanding of both visual cues and temporal structure. By aligning visual cues with textual guidance, this understanding supports procedural learning and multimodal reasoning, providing a richer context for skill acquisition. However, captions that fail to capture this structure may lack coherence and quality, which can create confusion and undermine the video's educational intent. To address this gap, we introduce DynaStride, a pipeline to generate coherent, scene-level captions without requiring manual scene segmentation. Using the YouCookII dataset's scene annotations, DynaStride performs adaptive frame sampling and multimodal windowing to capture key transitions within each scene. It then employs a multimodal chain-of-thought process to produce multiple action-object pairs, which are refined and fused using a dynamic stride window selection algorithm that adaptively balances temporal context and redundancy. The final scene-level caption integrates visual semantics and temporal reasoning in a single instructional caption. Empirical evaluations against strong baselines, including VLLaMA3 and GPT-4o, demonstrate consistent gains on both N-gram-based metrics (BLEU, METEOR) and semantic similarity measures (BERTScore, CLIPScore). Qualitative analyses further show that DynaStride produces captions that are more temporally coherent and informative, suggesting a promising direction for improving AI-powered instructional content generation.
comment : 16 pages, 15 figures, 5 Tables, Accepted at NeurIPS 7HVU Workshop, Accepted at AAAI AI4ED Workshop
♻ ☆ SCI: A Metacognitive Control for Signal Dynamics
Vishal Joshua Meesala
Modern deep learning systems are typically deployed as open-loop function approximators: they map inputs to outputs in a single pass, without regulating how much computation or explanatory effort is spent on a given case. In safety-critical settings, this is brittle: easy and ambiguous inputs receive identical processing, and uncertainty is only read off retrospectively from raw probabilities. We introduce the Surgical Cognitive Interpreter (SCI), a lightweight closed-loop metacognitive control layer that wraps an existing stochastic model and turns prediction into an iterative process. SCI monitors a scalar interpretive state SP(t), here instantiated as a normalized entropy-based confidence signal, and adaptively decides whether to stop, continue sampling, or abstain. The goal is not to improve accuracy per se, but to regulate interpretive error ΔSP and expose a safety signal that tracks when the underlying model is likely to fail. We instantiate SCI around Monte Carlo dropout classifiers in three domains: vision (MNIST digits), medical time series (MIT-BIH arrhythmia), and industrial condition monitoring (rolling-element bearings). In all cases, the controller allocates more inference steps to misclassified inputs than to correct ones (up to about 3-4x on MNIST and bearings, and 1.4x on MIT-BIH). The resulting ΔSP acts as a usable safety signal for detecting misclassifications (AUROC 0.63 on MNIST, 0.70 on MIT-BIH, 0.86 on bearings). Code and reproducibility: https://github.com/vishal-1344/sci
comment : v2: Extended theoretical analysis (Lyapunov-style stability), added metacognitive experiments across three domains, and released code and configuration files at https://github.com/vishal-1344/sci
♻ ☆ Localized Conformal Multi-Quantile Regression
Yuan Lu
Standard conformal prediction methods guarantee marginal coverage but often produce inefficient intervals that fail to adapt to local heteroscedasticity, while recent localized approaches often struggle to maintain validity across distinct subpopulations with varying noise profiles. To address these challenges, we introduce Localized Conformal Multi-Quantile Regression (LCMQR), a novel framework that synergizes multi-quantile information with kernel-based localization to construct efficient and adaptive prediction intervals. Theoretically, we resolve an inconsistency in Conformalized Composite Quantile Regression (CCQR) by proving that our consistent Average-then-Max scoring mechanism systematically yields tighter intervals than the Max-then-Average approach used in prior work. For heterogeneous environments, we extend this framework to Group-Calibrated LCMQR (GC-LCMQR) via a stratified calibration step that guarantees finite-sample validity within distinct subgroups. Experiments on benchmark datasets and an Individual Treatment Effect (ITE) task demonstrate that LCMQR achieves superior efficiency on standard benchmarks, while GC-LCMQR uniquely achieves group-level coverage for target subgroups in mixture populations where baselines fail.
♻ ☆ SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinations NeurIPS 2025
Buyun Liang, Liangzu Peng, Jinqi Luo, Darshan Thaker, Kwan Ho Ryan Chan, René Vidal
Large Language Models (LLMs) are increasingly deployed in high-risk domains. However, state-of-the-art LLMs often produce hallucinations, raising serious concerns about their reliability. Prior work has explored adversarial attacks for hallucination elicitation in LLMs, but it often produces unrealistic prompts, either by inserting gibberish tokens or by altering the original meaning. As a result, these approaches offer limited insight into how hallucinations may occur in practice. While adversarial attacks in computer vision often involve realistic modifications to input images, the problem of finding realistic adversarial prompts for eliciting LLM hallucinations has remained largely underexplored. To address this gap, we propose Semantically Equivalent and Coherent Attacks (SECA) to elicit hallucinations via realistic modifications to the prompt that preserve its meaning while maintaining semantic coherence. Our contributions are threefold: (i) we formulate finding realistic attacks for hallucination elicitation as a constrained optimization problem over the input prompt space under semantic equivalence and coherence constraints; (ii) we introduce a constraint-preserving zeroth-order method to effectively search for adversarial yet feasible prompts; and (iii) we demonstrate through experiments on open-ended multiple-choice question answering tasks that SECA achieves higher attack success rates while incurring almost no semantic equivalence or semantic coherence errors compared to existing methods. SECA highlights the sensitivity of both open-source and commercial gradient-inaccessible LLMs to realistic and plausible prompt variations. Code is available at https://github.com/Buyun-Liang/SECA.
comment : Accepted at NeurIPS 2025. Code is available at https://github.com/Buyun-Liang/SECA
♻ ☆ Enhancing Jailbreak Attacks on LLMs via Persona Prompt s NeurIPS 2025
Zheng Zhang, Peilin Zhao, Deheng Ye, Hao Wang
Jailbreak attacks aim to exploit large language models (LLMs) by inducing them to generate harmful content, thereby revealing their vulnerabilities. Understanding and addressing these attacks is crucial for advancing the field of LLM safety. Previous jailbreak approaches have mainly focused on direct manipulations of harmful intent, with limited attention to the impact of persona prompts. In this study, we systematically explore the efficacy of persona prompts in compromising LLM defenses. We propose a genetic algorithm-based method that automatically crafts persona prompts to bypass LLM's safety mechanisms. Our experiments reveal that: (1) our evolved persona prompts reduce refusal rates by 50-70% across multiple LLMs, and (2) these prompts demonstrate synergistic effects when combined with existing attack methods, increasing success rates by 10-20%. Our code and data are available at https://github.com/CjangCjengh/Generic_Persona.
comment : Workshop on LLM Persona Modeling at NeurIPS 2025
♻ ☆ PRISM: Diversifying Dataset Distillation by Decoupling Architectural Priors
Brian B. Moser, Shalini Sarode, Federico Raue, Stanislav Frolov, Krzysztof Adamkiewicz, Arundhati Shanbhag, Joachim Folz, Tobias C. Nauen, Andreas Dengel
Dataset distillation (DD) promises compact yet faithful synthetic data, but existing approaches often inherit the inductive bias of a single teacher model. As dataset size increases, this bias drives generation toward overly smooth, homogeneous samples, reducing intra-class diversity and limiting generalization. We present PRISM (PRIors from diverse Source Models), a framework that disentangles architectural priors during synthesis. PRISM decouples the logit-matching and regularization objectives, supervising them with different teacher architectures: a primary model for logits and a stochastic subset for batch-normalization (BN) alignment. On ImageNet-1K, PRISM consistently and reproducibly outperforms single-teacher methods (e.g., SRe2L) and recent multi-teacher variants (e.g., G-VBSM) at low- and mid-IPC regimes. The generated data also show significantly richer intra-class diversity, as reflected by a notable drop in cosine similarity between features. We further analyze teacher selection strategies (pre- vs. intra-distillation) and introduce a scalable cross-class batch formation scheme for fast parallel synthesis. Code will be released after the review period.
♻ ☆ Will Humanity Be Rendered Obsolete by AI?
Mohamed El Louadi, Emna Ben Romdhane
This article analyzes the existential risks artificial intelligence (AI) poses to humanity, tracing the trajectory from current AI to ultraintelligence. Drawing on Irving J. Good and Nick Bostrom's theoretical work, plus recent publications (AI 2027; If Anyone Builds It, Everyone Dies), it explores AGI and superintelligence. Considering machines' exponentially growing cognitive power and hypothetical IQs, it addresses the ethical and existential implications of an intelligence vastly exceeding humanity's, fundamentally alien. Human extinction may result not from malice, but from uncontrollable, indifferent cognitive superiority.
♻ ☆ Conformal Prediction for Multi-Source Detection on a Network
Xingchao Jian, Purui Zhang, Lan Tian, Feng Ji, Wenfei Liang, Wee Peng Tay, Bihan Wen, Felix Krahmer
Detecting the origin of information or infection spread in networks is a fundamental challenge with applications in misinformation tracking, epidemiology, and beyond. We study the multi-source detection problem: given snapshot observations of node infection status on a graph, estimate the set of source nodes that initiated the propagation. Existing methods either lack statistical guarantees or are limited to specific diffusion models and assumptions. We propose a novel conformal prediction framework that provides statistically valid recall guarantees for source set detection, independent of the underlying diffusion process or data distribution. Our approach introduces principled score functions to quantify the alignment between predicted probabilities and true sources, and leverages a calibration set to construct prediction sets with user-specified recall and coverage levels. The method is applicable to both single- and multi-source scenarios, supports general network diffusion dynamics, and is computationally efficient for large graphs. Empirical results demonstrate that our method achieves rigorous coverage with competitive accuracy, outperforming existing baselines in both reliability and scalability.The code is available online.
♻ ☆ ToMA: Token Merge with Attention for Diffusion Models ICML 2025
Wenbo Lu, Shaoyi Zheng, Yuxuan Xia, Shengjie Wang
Diffusion models excel in high-fidelity image generation but face scalability limits due to transformers' quadratic attention complexity. Plug-and-play token reduction methods like ToMeSD and ToFu reduce FLOPs by merging redundant tokens in generated images but rely on GPU-inefficient operations (e.g., sorting, scattered writes), introducing overheads that negate theoretical speedups when paired with optimized attention implementations (e.g., FlashAttention). To bridge this gap, we propose Token Merge with Attention (ToMA), an off-the-shelf method that redesigns token reduction for GPU-aligned efficiency, with three key contributions: 1) a reformulation of token merge as a submodular optimization problem to select diverse tokens; 2) merge/unmerge as an attention-like linear transformation via GPU-friendly matrix operations; and 3) exploiting latent locality and sequential redundancy (pattern reuse) to minimize overhead. ToMA reduces SDXL/Flux generation latency by 24%/23%, respectively (with DINO $Δ< 0.07$), outperforming prior methods. This work bridges the gap between theoretical and practical efficiency for transformers in diffusion. Code available at https://github.com/WenboLuu/ToMA.
comment : In proceedings of the 42nd International Conference on Machine Learning (ICML 2025). Code available at https://github.com/wenboluu/ToMA
♻ ☆ ANSR-DT: An Adaptive Neuro-Symbolic Learning and Reasoning Framework for Digital Twins
Safayat Bin Hakim, Muhammad Adil, Alvaro Velasquez, Houbing Herbert Song
In this paper, we propose an Adaptive Neuro-Symbolic Learning and Reasoning Framework for digital twin technology called "ANSR-DT." Digital twins in industrial environments often struggle with interpretability, real-time adaptation, and human input integration. Our approach addresses these challenges by combining CNN-LSTM dynamic event detection with reinforcement learning and symbolic reasoning to enable adaptive intelligence with interpretable decision processes. This integration enhances environmental understanding while promoting continuous learning, leading to more effective real-time decision-making in human-machine collaborative applications. We evaluated ANSR-DT on synthetic industrial data, observing significant improvements over traditional approaches, with up to 99.5% accuracy for dynamic pattern recognition. The framework demonstrated superior adaptability with extended reinforcement learning training, improving explained variance from 0.447 to 0.547. Future work aims at scaling to larger datasets to test rule management beyond the current 14 rules. Our open-source implementation promotes reproducibility and establishes a foundation for future research in adaptive, interpretable digital twins for industrial applications.
comment : Code available at https://github.com/sbhakim/ansr-dt
♻ ☆ Multigranular Evaluation for Brain Visual Decoding AAAI 2026
Weihao Xia, Cengiz Oztireli
Existing evaluation protocols for brain visual decoding predominantly rely on coarse metrics that obscure inter-model differences, lack neuroscientific foundation, and fail to capture fine-grained visual distinctions. To address these limitations, we introduce BASIC, a unified, multigranular evaluation framework that jointly quantifies structural fidelity, inferential alignment, and contextual coherence between decoded and ground-truth images. For the structural level, we introduce a hierarchical suite of segmentation-based metrics, including foreground, semantic, instance, and component masks, anchored in granularity-aware correspondence across mask structures. For the semantic level, we extract structured scene representations encompassing objects, attributes, and relationships using multimodal large language models, enabling detailed, scalable, and context-rich comparisons with ground-truth stimuli. We benchmark a diverse set of visual decoding methods across multiple stimulus-neuroimaging datasets within this unified evaluation framework. Together, these criteria provide a more discriminative, interpretable, and comprehensive foundation for evaluating brain visual decoding methods.
comment : AAAI 2026 (Oral). Code: https://github.com/weihaox/BASIC
♻ ☆ An AI Capability Threshold for Rent-Funded Universal Basic Income in an AI-Automated Economy
Aran Nayebi
We derive the first closed-form condition under which artificial intelligence (AI) capital profits could sustainably finance a universal basic income (UBI) without relying on new taxation or the creation of new jobs. In a Solow-Zeira task-automation economy with a CES aggregator $σ< 1$, we introduce an AI capability parameter that scales the productivity of automatable tasks and obtain a tractable expression for the AI capability threshold -- the minimum productivity of AI relative to pre-AI automation required for a balanced transfer. Using current U.S. economic parameters, we find that even in the conservative scenario where no new tasks or jobs emerge, AI systems would only need to reach only 5-7 times today's automation productivity to fund an 11%-of-GDP UBI. Our analysis also reveals some specific policy levers: raising public revenue share (e.g. profit taxation) of AI capital from the current 15% to about 33% halves the required AI capability threshold to attain UBI to 3 times existing automation productivity, but gains diminish beyond 50% public revenue share, especially if regulatory costs increase. Market structure also strongly affects outcomes: monopolistic or concentrated oligopolistic markets reduce the threshold by increasing economic rents, whereas heightened competition significantly raises it. These results therefore offer a rigorous benchmark for assessing when advancing AI capabilities might sustainably finance social transfers in an increasingly automated economy.
comment : 9 pages, 3 figures, added Ramsey-Cass-Koopmans endogenous savings analysis & varied Figure 1 calibration
♻ ☆ Chain-of-Query: Unleashing the Power of LLMs in SQL-Aided Table Understanding via Multi- Agent Collaboration AACL 2025
Songyuan Sui, Hongyi Liu, Serena Liu, Li Li, Soo-Hyun Choi, Rui Chen, Xia Hu
Table understanding requires structured, multi-step reasoning. Large Language Models (LLMs) struggle with it due to the structural complexity of tabular data. Recently, multi-agent frameworks for SQL generation have shown promise in tackling the challenges of understanding tabular data, but existing approaches often suffer from limitations such as the inability to comprehend table structure for reliable SQL generation, error propagation that results in invalid queries, and over-reliance on execution correctness. To address these issues, we propose Chain-of-Query (CoQ), a novel multi-agent framework for SQL-aided table understanding. CoQ adopts natural-language-style representations of table schemas to abstract away structural noise and enhance understanding. It employs a clause-by-clause SQL generation strategy to improve query quality and introduces a hybrid reasoning division that separates SQL-based mechanical reasoning from LLM-based logical inference, thereby reducing reliance on execution outcomes. Extensive experiments across four models and five widely used benchmarks demonstrate that CoQ achieves substantial accuracy improvements and significantly lowers invalid SQL rates compared to prior generic LLM-based, SQL-aided, and hybrid baselines, confirming its superior effectiveness in table understanding. The code is available at https://github.com/SongyuanSui/ChainofQuery.
comment : AACL 2025 Main Conference (Oral)
♻ ☆ Confident RAG: Enhancing the Performance of LLMs for Mathematics Question Answering through Multi-Embedding and Confidence Scoring
Shiting Chen, Zijian Zhao, Jinsong Chen
Large Language Models (LLMs) hold significant promise for mathematics education, yet they often struggle with complex mathematical reasoning. While Retrieval-Augmented Generation (RAG) mitigates these issues by grounding LLMs in external knowledge, its effectiveness remains unstable, heavily dependent on the choice of a single embedding model. Moving beyond static RAG workflows, we draw on agentic workflow patterns, a paradigm that introduces structured task decomposition and collaboration to enhance system performance. We propose and examine two novel approaches that combine the benefits of multiple embedding models. While our Mixture-Embedding RAG approach (fusing retrieved documents) shows limited gains, our Confident RAG method (generating multiple answers and selecting the one with the highest confidence score) demonstrates significant improvement. Experimental results show that Confident RAG achieved average accuracy improvements of approximately 10% over vanilla LLMs and 5% over vanilla RAG. The consistent results across different LLMs and embedding models indicate that Confident RAG is an efficient plug-and-play solution for trustworthy mathematical AI assistants. Finally, we discuss how this work lays the groundwork for deploying Agentic RAG systems in educational settings, where autonomous planning and iterative refinement can be built upon our robust retrieval foundation.
♻ ☆ How Many Human Survey Respondents is a Large Language Model Worth? An Uncertainty Quantification Perspective
Chengpiao Huang, Yuhang Wu, Kaizheng Wang
Large language models (LLMs) are increasingly used to simulate survey responses, but synthetic data can be misaligned with the human population, leading to unreliable inference. We develop a general framework that converts LLM-simulated responses into reliable confidence sets for population parameters of human responses, addressing the distribution shift between the simulated and real populations. The key design choice is the number of simulated responses: too many produce overly narrow sets with poor coverage, while too few yield excessively loose estimates. We propose a data-driven approach that adaptively selects the simulation sample size to achieve nominal average-case coverage, regardless of the LLM's simulation fidelity or the confidence set construction procedure. The selected sample size is further shown to reflect the effective human population size that the LLM can represent, providing a quantitative measure of its simulation fidelity. Experiments on real survey datasets reveal heterogeneous fidelity gaps across different LLMs and domains.
comment : 61 pages, 12 figures
♻ ☆ M^3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark
Yang Zhou, Mingyu Zhao, Zhenting Wang, Difei Gu, Bangwei Guo, Ruosong Ye, Ligong Han, Can Jin, Dimitris N. Metaxas
We present M^3-Bench, the first benchmark for evaluating multimodal tool use under the Model Context Protocol. The benchmark targets realistic, multi-hop and multi-threaded workflows that require visual grounding and textual reasoning, cross-tool dependencies, and persistence of intermediate resources across steps. We introduce a similarity-driven alignment that serializes each tool call, embeds signatures with a sentence encoder, and performs similarity-bucketed Hungarian matching to obtain auditable one-to-one correspondences. On top of this alignment, we report interpretable metrics that decouple semantic fidelity from workflow consistency. The benchmark spans 28 servers with 231 tools, and provides standardized trajectories curated through an Executor & Judge pipeline with human verification; an auxiliary four large language models (LLMs) judge ensemble reports end-task Task Completion and information grounding. Evaluations of representative state-of-the-art Multimodal LLMs (MLLMs) reveal persistent gaps in multimodal MCP tool use, particularly in argument fidelity and structure consistency, underscoring the need for methods that jointly reason over images, text, and tool graphs. Our Benchmark's anonymous repository is at https://github.com/EtaYang10th/Open-M3-Bench
♻ ☆ Towards Balanced Multi-Modal Learning in 3D Human Pose Estimation
Mengshi Qi, Jiaxuan Peng, Xianlin Zhang, Huadong Ma
3D human pose estimation (3D HPE) has emerged as a prominent research topic, particularly in the realm of RGB-based methods. However, the use of RGB images is often limited by issues such as occlusion and privacy constraints. Consequently, multi-modal sensing, which leverages non-intrusive sensors, is gaining increasing attention. Nevertheless, multi-modal 3D HPE still faces challenges, including modality imbalance. In this work, we introduce a novel balanced multi-modal learning method for 3D HPE, which harnesses the power of RGB, LiDAR, mmWave, and WiFi. Specifically, we propose a Shapley value-based contribution algorithm to assess the contribution of each modality and detect modality imbalance. To address this imbalance, we design a modality learning regulation strategy that decelerates the learning process during the early stages of training. We conduct extensive experiments on the widely adopted multi-modal dataset, MM-Fi, demonstrating the superiority of our approach in enhancing 3D pose estimation under complex conditions. We will release our codes soon.
♻ ☆ EPLKG: Efficient Prompt Learning with Knowledge Graph
YongTaek Lim, Suho Kang, Yewon Kim, Dokyung Yoon, KyungWoo Song
Large-scale pre-trained models such as CLIP excel in transferability and robust generalization across diverse datasets. However, adapting these models to new datasets or domains is computationally costly, especially in low-resource or few-shot settings, and existing prompt-learning methods often lack interpretability. We introduce Efficient Prompt Learning with Knowledge Graph (EPLKG), which uses a knowledge graph to curate diverse, interpretable prompts and, where KG coverage is limited, augments this bank with LLM-generated human-readable visual descriptions. EPLKG operates entirely on cached CLIP image and text embeddings and employs a lightweight Gumbel-Softmax module to select a single prompt per image-class pair, enabling low-memory, fast training. Across 11 benchmarks, EPLKG reduces per-image training time by up to 45 percent and peak GPU memory by around 30 to 40 percent compared to strong prompt-learning baselines, while keeping the average base-new harmonic-mean accuracy within 2 percentage points, thereby improving the efficiency of model adaptation without sacrificing competitive performance or interpretability.
♻ ☆ MVGT: A Multi-view Graph Transformer Based on Spatial Relation s for EEG Emotion Recognition ICONIP 2025
Yanjie Cui, Xiaohong Liu, Jing Liang, Yamin Fu
Electroencephalography (EEG), a technique that records electrical activity from the scalp using electrodes, plays a vital role in affective computing. However, fully utilizing the multi-domain characteristics of EEG signals remains a significant challenge. Traditional single-perspective analyses often fail to capture the complex interplay of temporal, frequency, and spatial dimensions in EEG data. To address this, we introduce a multi-view graph transformer (MVGT) based on spatial relations that integrates information across three domains: temporal dynamics from continuous series, frequency features extracted from frequency bands, and inter-channel relationships captured through several spatial encodings. This comprehensive approach allows model to capture the nuanced properties inherent in EEG signals, enhancing its flexibility and representational power. Evaluation on publicly available datasets demonstrates that MVGT surpasses state-of-the-art methods in performance. The results highlight its ability to extract multi-domain information and effectively model inter-channel relationships, showcasing its potential for EEG-based emotion recognition tasks.
comment : Accepted by ICONIP 2025 (Oral). 16 pages, 5 figures
♻ ☆ CrossVid: A Comprehensive Benchmark for Evaluating Cross-Video Reasoning in Multimodal Large Language Models AAAI 2026
Jingyao Li, Jingyun Wang, Molin Tan, Haochen Wang, Cilin Yan, Likun Shi, Jiayin Cai, Xiaolong Jiang, Yao Hu
Cross-Video Reasoning (CVR) presents a significant challenge in video understanding, which requires simultaneous understanding of multiple videos to aggregate and compare information across groups of videos. Most existing video understanding benchmarks focus on single-video analysis, failing to assess the ability of multimodal large language models (MLLMs) to simultaneously reason over various videos. Recent benchmarks evaluate MLLMs' capabilities on multi-view videos that capture different perspectives of the same scene. However, their limited tasks hinder a thorough assessment of MLLMs in diverse real-world CVR scenarios. To this end, we introduce CrossVid, the first benchmark designed to comprehensively evaluate MLLMs' spatial-temporal reasoning ability in cross-video contexts. Firstly, CrossVid encompasses a wide spectrum of hierarchical tasks, comprising four high-level dimensions and ten specific tasks, thereby closely reflecting the complex and varied nature of real-world video understanding. Secondly, CrossVid provides 5,331 videos, along with 9,015 challenging question-answering pairs, spanning single-choice, multiple-choice, and open-ended question formats. Through extensive experiments on various open-source and closed-source MLLMs, we observe that Gemini-2.5-Pro performs best on CrossVid, achieving an average accuracy of 50.4%. Notably, our in-depth case study demonstrates that most current MLLMs struggle with CVR tasks, primarily due to their inability to integrate or compare evidence distributed across multiple videos for reasoning. These insights highlight the potential of CrossVid to guide future advancements in enhancing MLLMs' CVR capabilities.
comment : Accepted to AAAI 2026 (main track). For code and data, see https://github.com/chuntianli666/CrossVid
♻ ☆ Hyperparameters are all you need: Using five-step inference for an original diffusion model to generate images comparable to the latest distillation model
Zilai Li
The diffusion model is a state-of-the-art generative model that samples images by applying a neural network iteratively. However, the original sampling algorithm requires substantial computation cost, and reducing the sampling step is a prevailing research area. To cope with this problem, one mainstream approach is to treat the sampling process as an algorithm that solves an ordinary differential equation (ODE). Our study proposes a training-free inference plugin compatible with most few-step ODE solvers. To the best of my knowledge, our algorithm is the first training-free algorithm to sample a 1024 x 1024-resolution image in 6 steps and a 512 x 512-resolution image in 5 steps, with an FID result that outperforms the SOTA distillation models and the 20-step DPM++ 2m solver, respectively. Based on analyses of the latent diffusion model's structure, the diffusion ODE, and the Free-U mechanism, we explain why specific hyperparameter couplings improve stability and inference speed without retraining. Meanwhile, experimental results also reveal a new design space of the latent diffusion ODE solver. Additionally, we also analyze the difference between the original diffusion model and the diffusion distillation model via an information-theoretic study, which shows the reason why the few-step ODE solver designed for the diffusion model can outperform the training-based diffusion distillation algorithm in few-step inference. The tentative results of the experiment prove the mathematical analysis. code base is below: https://github.com/TheLovesOfLadyPurple/Hyperparameter-is-all-you-need
comment : 21 pages, 8 figures
♻ ☆ Black-Box On-Policy Distillation of Large Language Models
Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, Furu Wei
Black-box distillation creates student large language models (LLMs) by learning from a proprietary teacher model's text outputs alone, without access to its internal logits or parameters. In this work, we introduce Generative Adversarial Distillation (GAD), which enables on-policy and black-box distillation. GAD frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher LLM's, creating a minimax game. The discriminator acts as an on-policy reward model that co-evolves with the student, providing stable, adaptive feedback. Experimental results show that GAD consistently surpasses the commonly used sequence-level knowledge distillation. In particular, Qwen2.5-14B-Instruct (student) trained with GAD becomes comparable to its teacher, GPT-5-Chat, on the LMSYS-Chat automatic evaluation. The results establish GAD as a promising and effective paradigm for black-box LLM distillation.
♻ ☆ Speech Judge : Towards Human-Level Judgment for Speech Naturalness
Xueyao Zhang, Chaoren Wang, Huan Liao, Ziniu Li, Yuancheng Wang, Li Wang, Dongya Jia, Yuanzhe Chen, Xiulin Li, Zhuo Chen, Zhizheng Wu
Aligning large generative models with human feedback is a critical challenge. In speech synthesis, this is particularly pronounced due to the lack of a large-scale human preference dataset, which hinders the development of models that truly align with human perception. To address this, we introduce SpeechJudge, a comprehensive suite comprising a dataset, a benchmark, and a reward model centered on naturalness--one of the most fundamental subjective metrics for speech synthesis. First, we present SpeechJudge-Data, a large-scale human feedback corpus of 99K speech pairs. The dataset is constructed using a diverse set of advanced zero-shot text-to-speech (TTS) models across diverse speech styles and multiple languages, with human annotations for both intelligibility and naturalness preference. From this, we establish SpeechJudge-Eval, a challenging benchmark for speech naturalness judgment. Our evaluation reveals that existing metrics and AudioLLMs struggle with this task; the leading model, Gemini-2.5-Flash, achieves less than 70% agreement with human judgment, highlighting a significant gap for improvement. To bridge this gap, we develop SpeechJudge-GRM, a generative reward model (GRM) based on Qwen2.5-Omni-7B. It is trained on SpeechJudge-Data via a two-stage post-training process: Supervised Fine-Tuning (SFT) with Chain-of-Thought rationales followed by Reinforcement Learning (RL) with GRPO on challenging cases. On the SpeechJudge-Eval benchmark, the proposed SpeechJudge-GRM demonstrates superior performance, achieving 77.2% accuracy (and 79.4% after inference-time scaling @10) compared to a classic Bradley-Terry reward model (72.7%). Furthermore, SpeechJudge-GRM can be also employed as a reward function during the post-training of speech generation models to facilitate their alignment with human preferences.
comment : Dataset, Model, and Code: https://github.com/AmphionTeam/SpeechJudge
♻ ☆ Learning to Generate Rigid Body Interactions with Video Diffusion Models
David Romero, Ariana Bermudez, Hao Li, Fabio Pizzati, Ivan Laptev
Recent video generation models have achieved remarkable progress and are now deployed in film, social media production, and advertising. Beyond their creative potential, such models also hold promise as world simulators for robotics and embodied decision making. Despite strong advances, however, current approaches still struggle to generate physically plausible object interactions and lack object-level control mechanisms. To address these limitations, we introduce KineMask, an approach for video generation that enables realistic rigid body control, interactions, and effects. Given a single image and a specified object velocity, our method generates videos with inferred motions and future object interactions. We propose a two-stage training strategy that gradually removes future motion supervision via object masks. Using this strategy we train video diffusion models (VDMs) on synthetic scenes of simple interactions and demonstrate significant improvements of object interactions in real scenes. Furthermore, KineMask integrates low-level motion control with high-level textual conditioning via predicted scene descriptions, leading to support for synthesis of complex dynamical phenomena. Our experiments show that KineMask achieves strong improvements over recent models of comparable size. Ablation studies further highlight the complementary roles of low- and high-level conditioning in VDMs. Our code, model, and data will be made publicly available. Project Page: https://daromog.github.io/KineMask/
♻ ☆ Monitor-Generate-Verify (MGV): Formalising Metacognitive Theory for Language Model Reasoning NeurIPS 2025
Nick Oh, Fernand Gobet
Test-time reasoning architectures such as those following the Generate-Verify paradigm, where a model iteratively refines or verifies its own generated outputs, prioritise generation and verification but exclude the monitoring processes that determine when and how reasoning should begin. This omission may contribute to the prefix dominance trap, in which models commit early to suboptimal reasoning paths and seldom recover, yielding roughly 20% accuracy loss. We address this architectural gap by proposing the Monitor-Generate-Verify (MGV) framework, a computational translation of Flavell's and Nelson and Narens' metacognitive theories that preserves their psychological detail. MGV extends the Generate-Verify paradigm by adding explicit monitoring that captures metacognitive experiences (from difficulty assessments to confidence judgements) before generation begins and refines future monitoring through verification feedback. Though we present no empirical validation, MGV provides a vocabulary for diagnosing component-level failures in reasoning systems, suggests specific architectural interventions for future designs, and identifies connections to resource-rational analysis that may ground its mechanisms in normative principles.
comment : Presented at the Workshop on the Foundations of Reasoning in Language Models at NeurIPS 2025 (non-archival)
♻ ☆ Flexible Swarm Learning May Outpace Foundation Models in Essential Tasks
Moein E. Samadi, Andreas Schuppert
Foundation models have rapidly advanced AI, raising the question of whether their decisions will ultimately surpass human strategies in real-world domains. The exponential, and possibly super-exponential, pace of AI development makes such analysis elusive. Nevertheless, many application areas that matter for daily life and society show only modest gains so far; a prominent case is diagnosing and treating dynamically evolving disease in intensive care. The common challenge is adapting complex systems to dynamic environments. Effective strategies must optimize outcomes in systems composed of strongly interacting functions while avoiding shared side effects; this requires reliable, self-adaptive modeling. These tasks align with building digital twins of highly complex systems whose mechanisms are not fully or quantitatively understood. It is therefore essential to develop methods for self-adapting AI models with minimal data and limited mechanistic knowledge. As this challenge extends beyond medicine, AI should demonstrate clear superiority in these settings before assuming broader decision-making roles. We identify the curse of dimensionality as a fundamental barrier to efficient self-adaptation and argue that monolithic foundation models face conceptual limits in overcoming it. As an alternative, we propose a decentralized architecture of interacting small agent networks (SANs). We focus on agents representing the specialized substructure of the system, where each agent covers only a subset of the full system functions. Drawing on mathematical results on the learning behavior of SANs and evidence from existing applications, we argue that swarm-learning in diverse swarms can enable self-adaptive SANs to deliver superior decision-making in dynamic environments compared with monolithic foundation models, though at the cost of reduced reproducibility in detail.
♻ ☆ GraphIC: A Graph-Based In- Context Example Retrieval Model for Multi-Step Reasoning
Jiale Fu, Yaqing Wang, Simeng Han, Jiaming Fan, Xu Yang
In-context learning (ICL) enhances large language models (LLMs) by incorporating demonstration examples, yet its effectiveness heavily depends on the quality of selected examples. Current methods typically use text embeddings to measure semantic similarity, which often introduces bias in multi-step reasoning tasks. This occurs because text embeddings contain irrelevant semantic information and lack deeper reasoning structures. To address this, we propose GraphIC, a graph-based retrieval model that leverages reasoning-aware representation and specialized similarity metric for in-context example retrieval. GraphIC first constructs thought graphs-directed, node-attributed graphs that explicitly model reasoning steps and their dependencies-for candidate examples and queries. This approach filters out superficial semantics while preserving essential reasoning processes. Next, GraphIC retrieves examples using a novel similarity metric tailored for these graphs, capturing sequential reasoning patterns and asymmetry between examples. Comprehensive evaluations across mathematical reasoning, code generation, and logical reasoning tasks demonstrate that GraphIC outperforms 10 baseline methods. Our results highlight the importance of reasoning-aware retrieval in ICL, offering a robust solution for enhancing LLM performance in multi-step reasoning scenarios.
♻ ☆ Reasoning -Intensive Regression
Diane Tchuindjo, Omar Khattab
AI researchers and practitioners increasingly apply large language models (LLMs) to what we call reasoning-intensive regression (RiR), i.e., deducing subtle numerical scores from text. Unlike standard language regression tasks, e.g., for sentiment or similarity, RiR often appears instead in ad-hoc problems such as rubric-based scoring, modeling dense rewards in complex environments, or domain-specific retrieval, where much deeper analysis of context is required while only limited task-specific training data and computation are available. We cast four realistic problems as RiR tasks to establish an initial benchmark, and use that to test our hypothesis that prompting frozen LLMs and finetuning Transformer encoders via gradient descent will both often struggle in RiR. We then propose MENTAT, a simple and lightweight method that combines batch-reflective prompt optimization with neural ensemble learning. MENTAT achieves up to 65% improvement over both baselines, though substantial room remains for future advances in RiR.
♻ ☆ Advancing Natural Language Formalization to First Order Logic with Fine-tuned LLMs
Felix Vossel, Till Mossakowski, Björn Gehrke
Automating the translation of natural language to first-order logic (FOL) is crucial for knowledge representation and formal methods, yet remains challenging. We present a systematic evaluation of fine-tuned LLMs for this task, comparing architectures (encoder-decoder vs. decoder-only) and training strategies. Using the MALLS and Willow datasets, we explore techniques like vocabulary extension, predicate conditioning, and multilingual training, introducing metrics for exact match, logical equivalence, and predicate alignment. Our fine-tuned Flan-T5-XXL achieves 70% accuracy with predicate lists, outperforming GPT-4o and even the DeepSeek-R1-0528 model with CoT reasoning ability as well as symbolic systems like ccg2lambda. Key findings show: (1) predicate availability boosts performance by 15-20%, (2) T5 models surpass larger decoder-only LLMs, and (3) models generalize to unseen logical arguments (FOLIO dataset) without specific training. While structural logic translation proves robust, predicate extraction emerges as the main bottleneck.
comment : 15 pages, 7 tables, accepted at the International Joint Conference on Learning & Reasoning (IJCLR 2025)
♻ ☆ Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmark s
Jie Li, Hongyi Cai, Mingkang Dong, Muxin Pu, Shan You, Fei Wang, Tao Huang
Automatically detecting abnormal events in videos is crucial for modern autonomous systems, yet existing Video Anomaly Detection (VAD) benchmarks lack the scene diversity, balanced anomaly coverage, and temporal complexity needed to reliably assess real-world performance. Meanwhile, the community is increasingly moving toward Video Anomaly Understanding (VAU), which requires deeper semantic and causal reasoning but remains difficult to benchmark due to the heavy manual annotation effort it demands. In this paper, we introduce Pistachio, a new VAD/VAU benchmark constructed entirely through a controlled, generation-based pipeline. By leveraging recent advances in video generation models, Pistachio provides precise control over scenes, anomaly types, and temporal narratives, effectively eliminating the biases and limitations of Internet-collected datasets. Our pipeline integrates scene-conditioned anomaly assignment, multi-step storyline generation, and a temporally consistent long-form synthesis strategy that produces coherent 41-second videos with minimal human intervention. Extensive experiments demonstrate the scale, diversity, and complexity of Pistachio, revealing new challenges for existing methods and motivating future research on dynamic and multi-event anomaly understanding.
♻ ☆ VoQA: Visual-only Question Answering
Jianing An, Luyang Jiang, Jie Luo, Wenjun Wu, Lei Huang
Visual understanding requires interpreting both natural scenes and the textual information that appears within them, motivating tasks such as Visual Question Answering (VQA). However, current VQA benchmarks overlook scenarios with visually embedded questions, whereas advanced agents should be able to see the question without separate text input as humans. We introduce Visual-only Question Answering (VoQA), where both the scene and the question appear within a single image, requiring models to perceive and reason purely through vision. This setting supports more realistic visual understanding and interaction in scenarios where questions or instructions are embedded directly in the visual scene. Evaluations under pure visual-only zero-shot, prompt-guided and OCR-assisted settings show that current models exhibit a clear performance drop compared to traditional VQA. To address this, we investigate question-alignment fine-tuning strategies designed to guide models toward interpreting the visual question prior to reasoning. Leveraging VoQA dataset together with these strategies yields robust vision-only reasoning while preserving cross-task generalization to traditional VQA, reflecting the complementary visual and textual reasoning capabilities fostered through VoQA training. The code and data are publicly available.
comment : 21 pages
♻ ☆ Towards Efficient and Accurate Spiking Neural Networks via Adaptive Bit Allocation
Xingting Yao, Qinghao Hu, Fei Zhou, Tielong Liu, Gang Li, Peisong Wang, Jian Cheng
Multi-bit spiking neural networks (SNNs) have recently become a heated research spot, pursuing energy-efficient and high-accurate AI. However, with more bits involved, the associated memory and computation demands escalate to the point where the performance improvements become disproportionate. Based on the insight that different layers demonstrate different importance and extra bits could be wasted and interfering, this paper presents an adaptive bit allocation strategy for direct-trained SNNs, achieving fine-grained layer-wise allocation of memory and computation resources. Thus, SNN's efficiency and accuracy can be improved. Specifically, we parametrize the temporal lengths and the bit widths of weights and spikes, and make them learnable and controllable through gradients. To address the challenges caused by changeable bit widths and temporal lengths, we propose the refined spiking neuron, which can handle different temporal lengths, enable the derivation of gradients for temporal lengths, and suit spike quantization better. In addition, we theoretically formulate the step-size mismatch problem of learnable bit widths, which may incur severe quantization errors to SNN, and accordingly propose the step-size renewal mechanism to alleviate this issue. Experiments on various datasets, including the static CIFAR and ImageNet datasets and the dynamic CIFAR-DVS and DVS-GESTURE datasets, demonstrate that our methods can reduce the overall memory and computation cost while achieving higher accuracy. Particularly, our SEWResNet-34 can achieve a 2.69\% accuracy gain and 4.16$\times$ lower bit budgets over the advanced baseline work on ImageNet. This work is open-sourced at \href{https://github.com/Ikarosy/Towards-Efficient-and-Accurate-Spiking-Neural-Networks-via-Adaptive-Bit-Allocation}{this link}.
comment : Neural Networks, In press
♻ ☆ HAMLET: Hyperadaptive Agent -based Modeling for Live Embodied Theatrics
Sizhou Chen, Shufan Jiang, Chi Zhang, Xiao-Lei Zhang, Xuelong Li
Creating an immersive and interactive theatrical experience is a long-term goal in the field of interactive narrative. The emergence of large language model (LLM) is providing a new path to achieve this goal. However, existing LLM-based drama generation methods often result in agents that lack initiative and cannot interact with the physical scene. Furthermore, these methods typically require detailed user input to drive the drama. These limitations reduce the interactivity and immersion of online real-time performance. To address the above challenges, we propose HAMLET, a multi-agent framework focused on drama creation and online performance. Given a simple topic, the framework generates a narrative blueprint, guiding the subsequent improvisational performance. During the online performance, each actor is given an autonomous mind. This means that actors can make independent decisions based on their own background, goals, and emotional state. In addition to conversations with other actors, their decisions can also change the state of scene props through actions such as opening a letter or picking up a weapon. The change is then broadcast to other related actors, updating what they know and care about, which in turn influences their next action. To evaluate the quality of drama performance generated by HAMLET, we designed an evaluation method to assess three primary aspects, including character performance, narrative quality, and interaction experience. The experimental evaluation shows that HAMLET can create expressive and coherent theatrical experiences.
comment : This submission has been withdrawn due to unresolved issues concerning author order and potential conflicts of interest between author affiliations. The authors will resubmit once these matters are fully resolved
♻ ☆ Efficient LLM-Jailbreaking via Multimodal-LLM Jailbreak
Haoxuan Ji, Zheng Lin, Zhenxing Niu, Xinbo Gao, Gang Hua
This paper focuses on jailbreaking attacks against large language models (LLMs), eliciting them to generate objectionable content in response to harmful user queries. Unlike previous LLM-jailbreak methods that directly orient to LLMs, our approach begins by constructing a multimodal large language model (MLLM) built upon the target LLM. Subsequently, we perform an efficient MLLM jailbreak and obtain a jailbreaking embedding. Finally, we convert the embedding into a textual jailbreaking suffix to carry out the jailbreak of target LLM. Compared to the direct LLM-jailbreak methods, our indirect jailbreaking approach is more efficient, as MLLMs are more vulnerable to jailbreak than pure LLM. Additionally, to improve the attack success rate of jailbreak, we propose an image-text semantic matching scheme to identify a suitable initial input. Extensive experiments demonstrate that our approach surpasses current state-of-the-art jailbreak methods in terms of both efficiency and effectiveness. Moreover, our approach exhibits superior cross-class generalization abilities.
♻ ☆ AED: Automatic Discovery of Effective and Diverse Vulnerabilities for Autonomous Driving Policy with Large Language Models
Le Qiu, Zelai Xu, Qixin Tan, Wenhao Tang, Chao Yu, Yu Wang
Assessing the safety of autonomous driving policy is of great importance, and reinforcement learning (RL) has emerged as a powerful method for discovering critical vulnerabilities in driving policies. However, existing RL-based approaches often struggle to identify vulnerabilities that are both effective-meaning the autonomous vehicle is genuinely responsible for the accidents-and diverse-meaning they span various failure types. To address these challenges, we propose AED, a framework that uses large language models (LLMs) to automatically discover effective and diverse vulnerabilities in autonomous driving policies. We first utilize an LLM to automatically design reward functions for RL training. Then we let the LLM consider a diverse set of accident types and train adversarial policies for different accident types in parallel. Finally, we use preference-based learning to filter ineffective accidents and enhance the effectiveness of each vulnerability. Experiments across multiple simulated traffic scenarios and tested policies show that AED uncovers a broader range of vulnerabilities and achieves higher attack success rates compared with expert-designed rewards, thereby reducing the need for manual reward engineering and improving the diversity and effectiveness of vulnerability discovery. The implementation can be found on: https://github.com/thu-nics/AED .
♻ ☆ Attributional Safety Failures in Large Language Models under Code-Mixed Perturbations
Somnath Banerjee, Pratyush Chatterjee, Shanu Kumar, Sayan Layek, Parag Agrawal, Rima Hazra, Animesh Mukherjee
While LLMs appear robustly safety-aligned in English, we uncover a catastrophic, overlooked weakness: attributional collapse under code-mixed perturbations. Our systematic evaluation of open models shows that the linguistic camouflage of code-mixing -- ``blending languages within a single conversation'' -- can cause safety guardrails to fail dramatically. Attack success rates (ASR) spike from a benign 9\% in monolingual English to 69\% under code-mixed inputs, with rates exceeding 90\% in non-Western contexts such as Arabic and Hindi. These effects hold not only on controlled synthetic datasets but also on real-world social media traces, revealing a serious risk for billions of users. To explain why this happens, we introduce saliency drift attribution (SDA), an interpretability framework that shows how, under code-mixing, the model's internal attention drifts away from safety-critical tokens (e.g., ``violence'' or ``corruption''), effectively blinding it to harmful intent. Finally, we propose a lightweight translation-based restoration strategy that recovers roughly 80\% of the safety lost to code-mixing, offering a practical path toward more equitable and robust LLM safety.
♻ ☆ GARF: Learning Generalizable 3D Reassembly for Real-Wo rl d Fractures
Sihang Li, Zeyu Jiang, Grace Chen, Chenyang Xu, Siqi Tan, Xue Wang, Irving Fang, Kristof Zyskowski, Shannon P. McPherron, Radu Iovita, Chen Feng, Jing Zhang
3D reassembly is a challenging spatial intelligence task with broad applications across scientific domains. While large-scale synthetic datasets have fueled promising learning-based approaches, their generalizability to different domains is limited. Critically, it remains uncertain whether models trained on synthetic datasets can generalize to real-world fractures where breakage patterns are more complex. To bridge this gap, we propose GARF, a generalizable 3D reassembly framework for real-world fractures. GARF leverages fracture-aware pretraining to learn fracture features from individual fragments, with flow matching enabling precise 6-DoF alignments. At inference time, we introduce one-step preassembly, improving robustness to unseen objects and varying numbers of fractures. In collaboration with archaeologists, paleoanthropologists, and ornithologists, we curate Fractura, a diverse dataset for vision and learning communities, featuring real-world fracture types across ceramics, bones, eggshells, and lithics. Comprehensive experiments have shown our approach consistently outperforms state-of-the-art methods on both synthetic and real-world datasets, achieving 82.87\% lower rotation error and 25.15\% higher part accuracy. This sheds light on training on synthetic data to advance real-world 3D puzzle solving, demonstrating its strong generalization across unseen object shapes and diverse fracture types. GARF's code, data and demo are available at https://ai4ce.github.io/GARF/.
comment : 18 pages. Project Page https://ai4ce.github.io/GARF/
♻ ☆ Involuntary Jailbreak
Yangyang Guo, Yangyan Li, Mohan Kankanhalli
In this study, we disclose a worrying new vulnerability in Large Language Models (LLMs), which we term \textbf{involuntary jailbreak}. Unlike existing jailbreak attacks, this weakness is distinct in that it does not involve a specific attack objective, such as generating instructions for \textit{building a bomb}. Prior attack methods predominantly target localized components of the LLM guardrail. In contrast, involuntary jailbreaks may potentially compromise the entire guardrail structure, which our method reveals to be surprisingly fragile. We merely employ a single universal prompt to achieve this goal. In particular, we instruct LLMs to generate several questions that would typically be rejected, along with their corresponding in-depth responses (rather than a refusal). Remarkably, this simple prompt strategy consistently jailbreaks the majority of leading LLMs, including Claude Opus 4.1, Grok 4, Gemini 2.5 Pro, and GPT 4.1. We hope this problem can motivate researchers and practitioners to re-evaluate the robustness of LLM guardrails and contribute to stronger safety alignment in future.
♻ ☆ When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision -Language-Action Models
Hui Lu, Yi Yu, Yiming Yang, Chenyu Yi, Qixin Zhang, Bingquan Shen, Alex C. Kot, Xudong Jiang
Vision-Language-Action (VLA) models are vulnerable to adversarial attacks, yet universal and transferable attacks remain underexplored, as most existing patches overfit to a single model and fail in black-box settings. To address this gap, we present a systematic study of universal, transferable adversarial patches against VLA-driven robots under unknown architectures, finetuned variants, and sim-to-real shifts. We introduce UPA-RFAS (Universal Patch Attack via Robust Feature, Attention, and Semantics), a unified framework that learns a single physical patch in a shared feature space while promoting cross-model transfer. UPA-RFAS combines (i) a feature-space objective with an $\ell_1$ deviation prior and repulsive InfoNCE loss to induce transferable representation shifts, (ii) a robustness-augmented two-phase min-max procedure where an inner loop learns invisible sample-wise perturbations and an outer loop optimizes the universal patch against this hardened neighborhood, and (iii) two VLA-specific losses: Patch Attention Dominance to hijack text$\to$vision attention and Patch Semantic Misalignment to induce image-text mismatch without labels. Experiments across diverse VLA models, manipulation suites, and physical executions show that UPA-RFAS consistently transfers across models, tasks, and viewpoints, exposing a practical patch-based attack surface and establishing a strong baseline for future defenses.
♻ ☆ Improving Region Representation Learning from Urban Imagery with Noisy Long-Caption Super vision AAAI-26
Yimei Zhang, Guojiang Shen, Kaili Ning, Tongwei Ren, Xuebo Qiu, Mengmeng Wang, Xiangjie Kong
Region representation learning plays a pivotal role in urban computing by extracting meaningful features from unlabeled urban data. Analogous to how perceived facial age reflects an individual's health, the visual appearance of a city serves as its "portrait", encapsulating latent socio-economic and environmental characteristics. Recent studies have explored leveraging Large Language Models (LLMs) to incorporate textual knowledge into imagery-based urban region representation learning. However, two major challenges remain: i) difficulty in aligning fine-grained visual features with long captions, and ii) suboptimal knowledge incorporation due to noise in LLM-generated captions. To address these issues, we propose a novel pre-training framework called UrbanLN that improves Urban region representation learning through Long-text awareness and Noise suppression. Specifically, we introduce an information-preserved stretching interpolation strategy that aligns long captions with fine-grained visual semantics in complex urban scenes. To effectively mine knowledge from LLM-generated captions and filter out noise, we propose a dual-level optimization strategy. At the data level, a multi-model collaboration pipeline automatically generates diverse and reliable captions without human intervention. At the model level, we employ a momentum-based self-distillation mechanism to generate stable pseudo-targets, facilitating robust cross-modal learning under noisy conditions. Extensive experiments across four real-world cities and various downstream tasks demonstrate the superior performance of our UrbanLN.
comment : Accepted as a full paper by AAAI-26
♻ ☆ Evaluating Spatio-Temporal Forecasting Trade-offs Between Graph Neural Networks and Foundation Models
Ragini Gupta, Naman Raina, Bo Chen, Li Chen, Claudiu Danilov, Josh Eckhardt, Keyshla Bernard, Klara Nahrstedt
Modern IoT deployments for environmental sensing produce high volume spatiotemporal data to support downstream tasks such as forecasting, typically powered by machine learning models. While existing filtering and strategic deployment techniques optimize collected data volume at the edge, they overlook how variations in sampling frequencies and spatial coverage affect downstream model performance. In many forecasting models, incorporating data from additional sensors denoise predictions by providing broader spatial contexts. This interplay between sampling frequency, spatial coverage and different forecasting model architectures remain underexplored. This work presents a systematic study of forecasting models - classical models (VAR), neural networks (GRU, Transformer), spatio-temporal graph neural networks (STGNNs), and time series foundation models (TSFMs: Chronos Moirai, TimesFM) under varying spatial sensor nodes density and sampling intervals using real-world temperature data in a wireless sensor network. Our results show that STGNNs are effective when sensor deployments are sparse and sampling rate is moderate, leveraging spatial correlations via encoded graph structure to compensate for limited coverage. In contrast, TSFMs perform competitively at high frequencies but degrade when spatial coverage from neighboring sensors is reduced. Crucially, the multivariate TSFM Moirai outperforms all models by natively learning cross-sensor dependencies. These findings offer actionable insights for building efficient forecasting pipelines in spatio-temporal systems. All code for model configurations, training, dataset, and logs are open-sourced for reproducibility: https://github.com/UIUC-MONET-Projects/Benchmarking-Spatiotemporal-Forecast-Models
♻ ☆ VI CoT - Agent : A Vision -Inte rl eaved Chain-of-Thought Framework for Interpretable Multimodal Reasoning and Scalable Remote Sensing Analysis
Chujie Wang, Zhiyuan Luo, Ruiqi Liu, Can Ran, Shenghua Fan, Xi Chen, Chu He
The current remote sensing image analysis task is increasingly evolving from traditional object recognition to complex intelligence reasoning, which places higher requirements on the model's reasoning ability and the flexibility of tool invocation. To this end, we propose a new multimodal agent framework, Vision-Interleaved Chain-of-Thought Framework (VICoT), which implements explicit multi-round reasoning by dynamically incorporating visual tools into the chain of thought. Through a stack-based reasoning structure and a modular MCP-compatible tool suite, VICoT enables LLMs to efficiently perform multi-round, interleaved vision-language reasoning tasks with strong generalization and flexibility.We also propose the Reasoning Stack distillation method to migrate complex Agent behaviors to small, lightweight models, which ensures the reasoning capability while significantly reducing complexity. Experiments on multiple remote sensing benchmarks demonstrate that VICoT significantly outperforms existing SOTA frameworks in reasoning transparency, execution efficiency, and generation quality.
♻ ☆ Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
Yiming Qin, Bomin Wei, Jiaxin Ge, Konstantinos Kallidromitis, Stephanie Fu, Trevor Darrell, XuDong Wang
Vision-Language Models (VLMs) excel at reasoning in linguistic space but struggle with perceptual understanding that requires dense visual perception, e.g., spatial reasoning and geometric awareness. This limitation stems from the fact that current VLMs have limited mechanisms to capture dense visual information across spatial dimensions. We introduce Chain-of-Visual-Thought (COVT), a framework that enables VLMs to reason not only in words but also through continuous visual tokens-compact latent representations that encode rich perceptual cues. Within a small budget of roughly 20 tokens, COVT distills knowledge from lightweight vision experts, capturing complementary properties such as 2D appearance, 3D geometry, spatial layout, and edge structure. During training, the VLM with COVT autoregressively predicts these visual tokens to reconstruct dense supervision signals (e.g., depth, segmentation, edges, and DINO features). At inference, the model reasons directly in the continuous visual token space, preserving efficiency while optionally decoding dense predictions for interpretability. Evaluated across more than ten diverse perception benchmarks, including CV-Bench, MMVP, RealWorldQA, MMStar, WorldMedQA, and HRBench, integrating COVT into strong VLMs such as Qwen2.5-VL and LLaVA consistently improves performance by 3% to 16% and demonstrates that compact continuous visual thinking enables more precise, grounded, and interpretable multimodal intelligence.
comment : Project page: https://wakalsprojectpage.github.io/covt-website/
♻ ☆ Small Drafts, Big Verdict: Information -Intensive Visual Reasoning via Speculation
Yuhan Liu, Lianhui Qin, Shengjie Wang
Large Vision-Language Models (VLMs) have achieved remarkable progress in multimodal understanding, yet they struggle when reasoning over information-intensive images that densely interleave textual annotations with fine-grained graphical elements. The main challenges lie in precisely localizing critical cues in dense layouts and multi-hop reasoning to integrate dispersed evidence. We propose Speculative Verdict (SV), a training-free framework inspired by speculative decoding that combines multiple lightweight draft experts with a large verdict model. In the draft stage, small VLMs act as draft experts to generate reasoning paths that provide diverse localization candidates; in the verdict stage, a strong VLM synthesizes these paths to produce the final answer, minimizing computational cost while recovering correct answers. To further improve efficiency and accuracy, SV introduces a consensus expert selection mechanism that forwards only high-agreement reasoning paths to the verdict. Empirically, SV achieves consistent gains on challenging information-intensive and high-resolution visual question answering benchmarks, including InfographicVQA, ChartMuseum, ChartQAPro, and HR-Bench 4K. By synthesizing correct insights from multiple partially accurate reasoning paths, SV achieves both error correction and cost-efficiency compared to large proprietary models or training pipelines. Code is available at https://github.com/Tinaliu0123/speculative-verdict.
♻ ☆ Addressing divergent representations from causal interventions on neural networks
Satchel Grant, Simon Jerome Han, Alexa R. Tartaglini, Christopher Potts
A common approach to mechanistic interpretability is to causally manipulate model representations via targeted interventions in order to understand what those representations encode. Here we ask whether such interventions create out-of-distribution (divergent) representations, and whether this raises concerns about how faithful their resulting explanations are to the target model in its natural state. First, we demonstrate theoretically and empirically that common causal intervention techniques often do shift internal representations away from the natural distribution of the target model. Then, we provide a theoretical analysis of two cases of such divergences: "harmless" divergences that occur in the behavioral null-space of the layer(s) of interest, and "pernicious" divergences that activate hidden network pathways and cause dormant behavioral changes. Finally, in an effort to mitigate the pernicious cases, we apply and modify the Counterfactual Latent (CL) loss from Grant (2025) allowing representations from causal interventions to remain closer to the natural distribution, reducing the likelihood of harmful divergences while preserving the interpretive power of the interventions. Together, these results highlight a path towards more reliable interpretability methods.
♻ ☆ Template-assisted Contrastive Learning of Task-oriented Dialogue Sentence Embeddings
Minsik Oh, Jiwei Li, Guoyin Wang
Learning high quality sentence embeddings from dialogues has drawn increasing attentions as it is essential to solve a variety of dialogue-oriented tasks with low annotation cost. Annotating and gathering utterance relationships in conversations are difficult, while token-level annotations, \eg, entities, slots and templates, are much easier to obtain. Other sentence embedding methods are usually sentence-level self-supervised frameworks and cannot utilize token-level extra knowledge. We introduce Template-aware Dialogue Sentence Embedding (TaDSE), a novel augmentation method that utilizes template information to learn utterance embeddings via self-supervised contrastive learning framework. We further enhance the effect with a synthetically augmented dataset that diversifies utterance-template association, in which slot-filling is a preliminary step. We evaluate TaDSE performance on five downstream benchmark dialogue datasets. The experiment results show that TaDSE achieves significant improvements over previous SOTA methods for dialogue. We further introduce a novel analytic instrument of semantic compression test, for which we discover a correlation with uniformity and alignment. Our code will be released upon acceptance.
♻ ☆ MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models NeurIPS 2025
Hengzhi Li, Megan Tjandrasuwita, Yi R. Fung, Armando Solar-Lezama, Paul Pu Liang
As AI becomes more closely integrated with peoples' daily activities, socially intelligent AI that can understand and interact seamlessly with humans in daily lives is increasingly important. However, current works in AI social reasoning all rely on language-only or language-dominant approaches to benchmark and training models, resulting in systems that are improving in verbal communication but struggle with nonverbal social understanding. To address this limitation, we tap into a novel data source rich in nonverbal social interactions -- mime videos. Mimes refer to the art of expression through gesture and movement without spoken words, which presents unique challenges and opportunities in interpreting nonverbal social communication. We contribute a new dataset called MimeQA, obtained by sourcing ~8 hours of videos clips from YouTube and developing a comprehensive video question-answering benchmark comprising 806 carefully annotated and verified question-answer pairs, designed to probe nonverbal social reasoning capabilities. Using MimeQA, we evaluate state-of-the-art video large language models (VideoLLMs) and find that they achieve low accuracy, generally ranging from 20-30%, while humans score 86%. Our analysis reveals that VideoLLMs often fail to ground imagined objects and over-rely on the text prompt while ignoring subtle nonverbal interactions. We hope to inspire future work in AI models that embody true social intelligence capable of interpreting non-verbal human interactions.
comment : NeurIPS 2025 Datasets and Benchmarks
♻ ☆ Phase codes emerge in recurrent neural networks optimized for modular arithmetic
Keith T. Murray
Recurrent neural networks (RNNs) can implement complex computations by leveraging a range of dynamics, such as oscillations, attractors, and transient trajectories. A growing body of work has highlighted the emergence of phase codes, a type of oscillatory activity where information is encoded in the relative phase of network activity, in RNNs trained for working memory tasks. However, these studies rely on architectural constraints or regularization schemes that explicitly promote oscillatory solutions. Here, we investigate whether phase coding can emerge purely from task optimization by training continuous-time RNNs to perform a simple modular arithmetic task without oscillatory-promoting biases. We find that in the absence of such biases, RNNs can learn phase code solutions. Surprisingly, we also uncover a rich diversity of alternative solutions that solve our modular arithmetic task via qualitatively distinct dynamics and dynamical mechanisms. We map the solution space for our task and show that the phase code solution occupies a distinct region. These results suggest that phase coding can be a natural but not inevitable outcome of training RNNs on modular arithmetic, and highlight the diversity of solutions RNNs can learn to solve simple tasks.
comment : UniReps: 3rd Edition of the Workshop on Unifying Representations in Neural Models
♻ ☆ Adaptive Margin RL HF via Preference over Preferences
Yaswanth Chittepu, Prasann Singhal, Greg Durrett, Scott Niekum
Margin-based optimization is fundamental to improving generalization and robustness in classification tasks. In the context of reward model learning from preferences within Reinforcement Learning from Human Feedback (RLHF), existing methods typically rely on no margins, fixed margins, or margins that are simplistic functions of preference ratings. However, such formulations often fail to account for the varying strengths of different preferences, for example some preferences are associated with larger margins between responses, or they rely on noisy margin information derived from ratings. We argue that modeling the strength of preferences can lead to better generalization and more faithful alignment. Furthermore, many existing methods that use adaptive margins assume access to accurate preference scores, which can be difficult for humans to provide reliably. We propose an approach that leverages preferences over preferences, that is annotations indicating which of two preferences reflects a stronger distinction. We use this ordinal signal to infer adaptive margins on a per-datapoint basis. We introduce an extension to Direct Preference Optimization (DPO), DPO-PoP, that incorporates adaptive margins from preference-over-preference supervision, enabling improved discriminative and generative performance. Empirically, our method outperforms vanilla DPO, DPO with fixed margins, and DPO with ground-truth margins on the UltraFeedback dataset. Additionally, we show that there is a tradeoff between discriminative and generative performance: improving test classification accuracy, particularly by correctly labeling weaker preferences at the expense of stronger ones, can lead to a decline in generative quality. To navigate this tradeoff, we propose two sampling strategies to gather preference-over-preference labels: one favoring discriminative performance and one favoring generative performance.
♻ ☆ Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity
Austin Meek, Eitan Sprejer, Iván Arcuschin, Austin J. Brockmeier, Steven Basart
Chain-of-thought (CoT) outputs let us read a model's step-by-step reasoning. Since any long, serial reasoning process must pass through this textual trace, the quality of the CoT is a direct window into what the model is thinking. This visibility could help us spot unsafe or misaligned behavior (monitorability), but only if the CoT is transparent about its internal reasoning (faithfulness). Fully measuring faithfulness is difficult, so researchers often focus on examining the CoT in cases where the model changes its answer after adding a cue to the input. This proxy finds some instances of unfaithfulness but loses information when the model maintains its answer, and does not investigate aspects of reasoning not tied to the cue. We extend these results to a more holistic sense of monitorability by introducing verbosity: whether the CoT lists every factor needed to solve the task. We combine faithfulness and verbosity into a single monitorability score that shows how well the CoT serves as the model's external `working memory', a property that many safety schemes based on CoT monitoring depend on. We evaluate instruction-tuned and reasoning models on BBH, GPQA, and MMLU. Our results show that models can appear faithful yet remain hard to monitor when they leave out key factors, and that monitorability differs sharply across model families. We release our evaluation code using the Inspect library to support reproducible future work.
comment : Project page at https://ajmeek.github.io/cot_monitorability_website/
♻ ☆ Multi- Agent Conditional Diffusion Model with Mean Field Communication as Wireless Resource Allocation Planner
Kechen Meng, Sinuo Zhang, Rongpeng Li, Xiangming Meng, Yansha Deng, Chan Wang, Ming Lei, Zhifeng Zhao
In wireless communication systems, efficient and adaptive resource allocation plays a crucial role in enhancing overall Quality of Service (QoS). Compared to the conventional Model-Free Reinforcement Learning (MFRL) scheme, Model-Based RL (MBRL) first learns a generative world model for subsequent planning. The reuse of historical experience in MBRL promises more stable training behavior, yet its deployment in large-scale wireless networks remains challenging due to high-dimensional stochastic dynamics, strong inter-agent cooperation, and communication constraints. To overcome these challenges, we propose the Multi-Agent Conditional Diffusion Model Planner (MA-CDMP) for decentralized communication resource management. Built upon the Distributed Training with Decentralized Execution (DTDE) paradigm, MA-CDMP models each communication node as an autonomous agent and employs Diffusion Models (DMs) to capture and predict environment dynamics. Meanwhile, an inverse dynamics model guides action generation, thereby enhancing sample efficiency and policy scalability. Moreover, to approximate large-scale agent interactions, a Mean-Field (MF) mechanism is introduced as an assistance to the classifier in DMs. This design mitigates inter-agent non-stationarity and enhances cooperation with minimal communication overhead in distributed settings. We further theoretically establish an upper bound on the distributional approximation error introduced by the MF-based diffusion generation, guaranteeing convergence stability and reliable modeling of multi-agent stochastic dynamics. Extensive experiments demonstrate that MA-CDMP consistently outperforms existing MARL baselines in terms of average reward and QoS metrics, showcasing its scalability and practicality for real-world wireless network optimization.
♻ ☆ Rendering-Aware Reinforcement Learning for Vector Graphics Generation
Juan A. Rodriguez, Haotian Zhang, Abhay Puri, Aarash Feizi, Rishav Pramanik, Pascal Wichmann, Arnab Mondal, Mohammad Reza Samsami, Rabiul Awal, Perouz Taslakian, Spandana Gella, Sai Rajeswar, David Vazquez, Christopher Pal, Marco Pedersoli
Scalable Vector Graphics (SVG) offer a powerful format for representing visual designs as interpretable code. Recent advances in vision-language models (VLMs) have enabled high-quality SVG generation by framing the problem as a code generation task and leveraging large-scale pretraining. VLMs are particularly suitable for this task as they capture both global semantics and fine-grained visual patterns, while transferring knowledge across vision, natural language, and code domains. However, existing VLM approaches often struggle to produce faithful and efficient SVGs because they never observe the rendered images during training. Although differentiable rendering for autoregressive SVG code generation remains unavailable, rendered outputs can still be compared to original inputs, enabling evaluative feedback suitable for reinforcement learning (RL). We introduce RLRF (Reinforcement Learning from Rendering Feedback), an RL method that enhances SVG generation in autoregressive VLMs by leveraging feedback from rendered SVG outputs. Given an input image, the model generates SVG roll-outs that are rendered and compared to the original image to compute a reward. This visual fidelity feedback guides the model toward producing more accurate, efficient, and semantically coherent SVGs. RLRF significantly outperforms supervised fine-tuning, addressing common failure modes and enabling precise, high-quality SVG generation with strong structural understanding and generalization.
♻ ☆ Interpreting ResNet-based CLIP via Neuron-Attention Decomposition NeurIPS 2025
Edmund Bu, Yossi Gandelsman
We present a novel technique for interpreting the neurons in CLIP-ResNet by decomposing their contributions to the output into individual computation paths. More specifically, we analyze all pairwise combinations of neurons and the following attention heads of CLIP's attention-pooling layer. We find that these neuron-head pairs can be approximated by a single direction in CLIP-ResNet's image-text embedding space. Leveraging this insight, we interpret each neuron-head pair by associating it with text. Additionally, we find that only a sparse set of the neuron-head pairs have a significant contribution to the output value, and that some neuron-head pairs, while polysemantic, represent sub-concepts of their corresponding neurons. We use these observations for two applications. First, we employ the pairs for training-free semantic segmentation, outperforming previous methods for CLIP-ResNet. Second, we utilize the contributions of neuron-head pairs to monitor dataset distribution shifts. Our results demonstrate that examining individual computation paths in neural networks uncovers interpretable units, and that such units can be utilized for downstream tasks.
comment : Accepted at NeurIPS 2025 Workshop on Mechanistic Interpretability. Project page: https://edmundbu.github.io/clip-neur-attn/
♻ ☆ Diffusion Models are Kelly Gamblers
Akhil Premkumar
We draw a connection between diffusion models and the Kelly criterion for maximizing returns in betting games. A signal that is correlated with the outcome of such a game can be used to focus the bets on a narrow range of high probability predictions. Diffusion models share the same paradigm in that they gradually concentrate the probability mass to fit the training data. We show that the information stored in an unconditional diffusion model captures, in part, the joint correlation between the components of the data variable $X$. Conditional diffusion models store additional information to bind the signal $X$ with the conditioning information $Y$, equal to the mutual information between them. The latter is only a small fraction of the total information in the neural network if the data is low-dimensional. We examine why this does not hinder conditional generation.
comment : 32 pages + references, 19 figures. v3: Significantly revised the overall narrative, and integrated more experiments into the main text
Computer Vision and Pattern Recognition 105
☆ SocialFusion: Addressing Social Degradation in Pre-train ed Vision -Language Models
Hamza Tahboub, Weiyan Shi, Gang Hua, Huaizu Jiang
Understanding social interactions from visual cues is a fundamental challenge for a socially competent AI. While powerful pre-trained vision-language models (VLMs) have shown remarkable general capabilities, they surprisingly struggle to unify and learn multiple social perception tasks simultaneously, often exhibiting negative transfer. We identify that this negative transfer stems from a critical issue we term "social degradation," whereby the general visual-linguistic pre-training process of VLMs impairs the visual encoder's ability to represent nuanced social information. We investigate this behavior further under two lenses: decodability through linear representation probing and compatibility through gradient conflict analysis, revealing that both play a role in the degradation, especially the former, which is significantly compromised in the VLM pre-training process. To address these issues, we propose SocialFusion, a unified framework that learns a minimal connection between a frozen visual encoder and a language model. Compared with existing VLMs, it exhibits positive transfer across all five social tasks, leveraging synergies between them to enhance overall performance and achieves comparable performance to task-specific state-of-the-art models on various benchmarks. Our findings suggest that current VLM pre-training strategies may be detrimental to acquiring general social competence and highlight the need for more socially-aware training paradigms.
comment : 22 pages, 10 figures
☆ Weakly Supervised Continuous Micro-Expression Intensity Estimation Using Temporal Deep Neural Network
Riyadh Mohammed Almushrafy
Micro-facial expressions are brief and involuntary facial movements that reflect genuine emotional states. While most prior work focuses on classifying discrete micro-expression categories, far fewer studies address the continuous evolution of intensity over time. Progress in this direction is limited by the lack of frame-level intensity labels, which makes fully supervised regression impractical. We propose a unified framework for continuous micro-expression intensity estimation using only weak temporal labels (onset, apex, offset). A simple triangular prior converts sparse temporal landmarks into dense pseudo-intensity trajectories, and a lightweight temporal regression model that combines a ResNet18 encoder with a bidirectional GRU predicts frame-wise intensity directly from image sequences. The method requires no frame-level annotation effort and is applied consistently across datasets through a single preprocessing and temporal alignment pipeline. Experiments on SAMM and CASME II show strong temporal agreement with the pseudo-intensity trajectories. On SAMM, the model reaches a Spearman correlation of 0.9014 and a Kendall correlation of 0.7999, outperforming a frame-wise baseline. On CASME II, it achieves up to 0.9116 and 0.8168, respectively, when trained without the apex-ranking term. Ablation studies confirm that temporal modeling and structured pseudo labels are central to capturing the rise-apex-fall dynamics of micro-facial movements. To our knowledge, this is the first unified approach for continuous micro-expression intensity estimation using only sparse temporal annotations.
☆ OmniFD: A Unified Model for Versatile Face Forgery Detection
Haotian Liu, Haoyu Chen, Chenhui Pan, You Hu, Guoying Zhao, Xiaobai Li
Face forgery detection encompasses multiple critical tasks, including identifying forged images and videos and localizing manipulated regions and temporal segments. Current approaches typically employ task-specific models with independent architectures, leading to computational redundancy and ignoring potential correlations across related tasks. We introduce OmniFD, a unified framework that jointly addresses four core face forgery detection tasks within a single model, i.e., image and video classification, spatial localization, and temporal localization. Our architecture consists of three principal components: (1) a shared Swin Transformer encoder that extracts unified 4D spatiotemporal representations from both images and video inputs, (2) a cross-task interaction module with learnable queries that dynamically captures inter-task dependencies through attention-based reasoning, and (3) lightweight decoding heads that transform refined representations into corresponding predictions for all FFD tasks. Extensive experiments demonstrate OmniFD's advantage over task-specific models. Its unified design leverages multi-task learning to capture generalized representations across tasks, especially enabling fine-grained knowledge transfer that facilitates other tasks. For example, video classification accuracy improves by 4.63% when image data are incorporated. Furthermore, by unifying images, videos and the four tasks within one framework, OmniFD achieves superior performance across diverse benchmarks with high efficiency and scalability, e.g., reducing 63% model parameters and 50% training time. It establishes a practical and generalizable solution for comprehensive face forgery detection in real-world applications. The source code is made available at https://github.com/haotianll/OmniFD.
☆ Structural Prognostic Event Modeling for Multimodal Cancer Survival Analysis
Yilan Zhang, Li Nanbo, Changchun Yang, Jürgen Schmidhuber, Xin Gao
The integration of histology images and gene profiles has shown great promise for improving survival prediction in cancer. However, current approaches often struggle to model intra- and inter-modal interactions efficiently and effectively due to the high dimensionality and complexity of the inputs. A major challenge is capturing critical prognostic events that, though few, underlie the complexity of the observed inputs and largely determine patient outcomes. These events, manifested as high-level structural signals such as spatial histologic patterns or pathway co-activations, are typically sparse, patient-specific, and unannotated, making them inherently difficult to uncover. To address this, we propose SlotSPE, a slot-based framework for structural prognostic event modeling. Specifically, inspired by the principle of factorial coding, we compress each patient's multimodal inputs into compact, modality-specific sets of mutually distinctive slots using slot attention. By leveraging these slot representations as encodings for prognostic events, our framework enables both efficient and effective modeling of complex intra- and inter-modal interactions, while also facilitating seamless incorporation of biological priors that enhance prognostic relevance. Extensive experiments on ten cancer benchmarks show that SlotSPE outperforms existing methods in 8 out of 10 cohorts, achieving an overall improvement of 2.9%. It remains robust under missing genomic data and delivers markedly improved interpretability through structured event decomposition.
comment : 37 pages, 14 Figures
☆ Estimation of Kinematic Motion from Dashcam Footage
Evelyn Zhang, Alex Richardson, Jonathan Sprinkle
The goal of this paper is to explore the accuracy of dashcam footage to predict the actual kinematic motion of a car-like vehicle. Our approach uses ground truth information from the vehicle's on-board data stream, through the controller area network, and a time-synchronized dashboard camera, mounted to a consumer-grade vehicle, for 18 hours of footage and driving. The contributions of the paper include neural network models that allow us to quantify the accuracy of predicting the vehicle speed and yaw, as well as the presence of a lead vehicle, and its relative distance and speed. In addition, the paper describes how other researchers can gather their own data to perform similar experiments, using open-source tools and off-the-shelf technology.
comment : 8 pages, 10 figures
☆ Learning Eigenstructures of Unstructured Data Manifolds
Roy Velich, Arkadi Piven, David Bensaïd, Daniel Cremers, Thomas Dagès, Ron Kimmel
We introduce a novel framework that directly learns a spectral basis for shape and manifold analysis from unstructured data, eliminating the need for traditional operator selection, discretization, and eigensolvers. Grounded in optimal-approximation theory, we train a network to decompose an implicit approximation operator by minimizing the reconstruction error in the learned basis over a chosen distribution of probe functions. For suitable distributions, they can be seen as an approximation of the Laplacian operator and its eigendecomposition, which are fundamental in geometry processing. Furthermore, our method recovers in a unified manner not only the spectral basis, but also the implicit metric's sampling density and the eigenvalues of the underlying operator. Notably, our unsupervised method makes no assumption on the data manifold, such as meshing or manifold dimensionality, allowing it to scale to arbitrary datasets of any dimension. On point clouds lying on surfaces in 3D and high-dimensional image manifolds, our approach yields meaningful spectral bases, that can resemble those of the Laplacian, without explicit construction of an operator. By replacing the traditional operator selection, construction, and eigendecomposition with a learning-based approach, our framework offers a principled, data-driven alternative to conventional pipelines. This opens new possibilities in geometry processing for unstructured data, particularly in high-dimensional spaces.
☆ CycliST: A Video Language Model Benchmark for Reasoning on Cyclical State Transitions
Simon Kohaut, Daniel Ochs, Shun Zhang, Benedict Flade, Julian Eggert, Kristian Kersting, Devendra Singh Dhami
We present CycliST, a novel benchmark dataset designed to evaluate Video Language Models (VLM) on their ability for textual reasoning over cyclical state transitions. CycliST captures fundamental aspects of real-world processes by generating synthetic, richly structured video sequences featuring periodic patterns in object motion and visual attributes. CycliST employs a tiered evaluation system that progressively increases difficulty through variations in the number of cyclic objects, scene clutter, and lighting conditions, challenging state-of-the-art models on their spatio-temporal cognition. We conduct extensive experiments with current state-of-the-art VLMs, both open-source and proprietary, and reveal their limitations in generalizing to cyclical dynamics such as linear and orbital motion, as well as time-dependent changes in visual attributes like color and scale. Our results demonstrate that present-day VLMs struggle to reliably detect and exploit cyclic patterns, lack a notion of temporal understanding, and are unable to extract quantitative insights from scenes, such as the number of objects in motion, highlighting a significant technical gap that needs to be addressed. More specifically, we find no single model consistently leads in performance: neither size nor architecture correlates strongly with outcomes, and no model succeeds equally well across all tasks. By providing a targeted challenge and a comprehensive evaluation framework, CycliST paves the way for visual reasoning models that surpass the state-of-the-art in understanding periodic patterns.
☆ Accelerating Inference of Masked Image Generators via Reinforcement Learning
Pranav Subbaraman, Shufan Li, Siyan Zhao, Aditya Grover
Masked Generative Models (MGM)s demonstrate strong capabilities in generating high-fidelity images. However, they need many sampling steps to create high-quality generations, resulting in slow inference speed. In this work, we propose Speed-RL, a novel paradigm for accelerating a pretrained MGMs to generate high-quality images in fewer steps. Unlike conventional distillation methods which formulate the acceleration problem as a distribution matching problem, where a few-step student model is trained to match the distribution generated by a many-step teacher model, we consider this problem as a reinforcement learning problem. Since the goal of acceleration is to generate high quality images in fewer steps, we can combine a quality reward with a speed reward and finetune the base model using reinforcement learning with the combined reward as the optimization target. Through extensive experiments, we show that the proposed method was able to accelerate the base model by a factor of 3x while maintaining comparable image quality.
comment : 15 pages, 9 figures
☆ Generalized Medical Phrase Grounding
Wenjun Zhang, Shekhar S. Chandra, Aaron Nicolson
Medical phrase grounding (MPG) maps textual descriptions of radiological findings to corresponding image regions. These grounded reports are easier to interpret, especially for non-experts. Existing MPG systems mostly follow the referring expression comprehension (REC) paradigm and return exactly one bounding box per phrase. Real reports often violate this assumption. They contain multi-region findings, non-diagnostic text, and non-groundable phrases, such as negations or descriptions of normal anatomy. Motivated by this, we reformulate the task as generalised medical phrase grounding (GMPG), where each sentence is mapped to zero, one, or multiple scored regions. To realise this formulation, we introduce the first GMPG model: MedGrounder. We adopted a two-stage training regime: pre-training on report sentence--anatomy box alignment datasets and fine-tuning on report sentence--human annotated box datasets. Experiments on PadChest-GR and MS-CXR show that MedGrounder achieves strong zero-shot transfer and outperforms REC-style and grounded report generation baselines on multi-region and non-groundable phrases, while using far fewer human box annotations. Finally, we show that MedGrounder can be composed with existing report generators to produce grounded reports without retraining the generator.
☆ Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer
Haoru Xue, Tairan He, Zi Wang, Qingwei Ben, Wenli Xiao, Zhengyi Luo, Xingye Da, Fernando Castañeda, Guanya Shi, Shankar Sastry, Linxi "Jim" Fan, Yuke Zhu
Recent progress in GPU-accelerated, photorealistic simulation has opened a scalable data-generation path for robot learning, where massive physics and visual randomization allow policies to generalize beyond curated environments. Building on these advances, we develop a teacher-student-bootstrap learning framework for vision-based humanoid loco-manipulation, using articulated-object interaction as a representative high-difficulty benchmark. Our approach introduces a staged-reset exploration strategy that stabilizes long-horizon privileged-policy training, and a GRPO-based fine-tuning procedure that mitigates partial observability and improves closed-loop consistency in sim-to-real RL. Trained entirely on simulation data, the resulting policy achieves robust zero-shot performance across diverse door types and outperforms human teleoperators by up to 31.7% in task completion time under the same whole-body control stack. This represents the first humanoid sim-to-real policy capable of diverse articulated loco-manipulation using pure RGB perception.
comment : https://doorman-humanoid.github.io/
☆ Parameter Reduction Improves Vision Transformer s: A Comparative Study of Sharing and Width Reduction
Anantha Padmanaban Krishna Kumar
Although scaling laws and many empirical results suggest that increasing the size of Vision Transformers often improves performance, model accuracy and training behavior are not always monotonically increasing with scale. Focusing on ViT-B/16 trained on ImageNet-1K, we study two simple parameter-reduction strategies applied to the MLP blocks, each removing 32.7\% of the baseline parameters. Our \emph{GroupedMLP} variant shares MLP weights between adjacent transformer blocks and achieves 81.47\% top-1 accuracy while maintaining the baseline computational cost. Our \emph{ShallowMLP} variant halves the MLP hidden dimension and reaches 81.25\% top-1 accuracy with a 38\% increase in inference throughput. Both models outperform the 86.6M-parameter baseline (81.05\%) and exhibit substantially improved training stability, reducing peak-to-final accuracy degradation from 0.47\% to the range 0.03\% to 0.06\%. These results suggest that, for ViT-B/16 on ImageNet-1K with a standard training recipe, the model operates in an overparameterized regime in which MLP capacity can be reduced without harming performance and can even slightly improve it. More broadly, our findings suggest that architectural constraints such as parameter sharing and reduced width may act as useful inductive biases, and highlight the importance of how parameters are allocated when designing Vision Transformers. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/parameter-efficient-vit-mlps.
comment : 7 pages total (6 pages main text, 1 page references), 1 figures, 2 tables. Code available at https://github.com/AnanthaPadmanaban-KrishnaKumar/parameter-efficient-vit-mlps
☆ TRoVe: Discovering Error-Inducing Static Feature Biases in Temporal Vision -Language Models NeurIPS 2025
Maya Varma, Jean-Benoit Delbrouck, Sophie Ostmeier, Akshay Chaudhari, Curtis Langlotz
Vision-language models (VLMs) have made great strides in addressing temporal understanding tasks, which involve characterizing visual changes across a sequence of images. However, recent works have suggested that when making predictions, VLMs may rely on static feature biases, such as background or object features, rather than dynamic visual changes. Static feature biases are a type of shortcut and can contribute to systematic prediction errors on downstream tasks; as a result, identifying and characterizing error-inducing static feature biases is critical prior to real-world model deployment. In this work, we introduce TRoVe, an automated approach for discovering error-inducing static feature biases learned by temporal VLMs. Given a trained VLM and an annotated validation dataset associated with a downstream classification task, TRoVe extracts candidate static features from the dataset and scores each feature by (i) the effect of the feature on classification errors as well as (ii) the extent to which the VLM relies on the feature when making predictions. In order to quantitatively evaluate TRoVe, we introduce an evaluation framework consisting of 101 trained temporal VLMs paired with ground-truth annotations for learned static feature biases. We use this framework to demonstrate that TRoVe can accurately identify error-inducing static feature biases in VLMs, achieving a 28.6% improvement over the closest baseline. Finally, we apply TRoVe to 7 off-the-shelf VLMs and 2 temporal understanding tasks, surfacing previously-unknown static feature biases and demonstrating that knowledge of learned biases can aid in improving model performance at test time. Our code is available at https://github.com/Stanford-AIMI/TRoVe.
comment : NeurIPS 2025
☆ Lotus-2: Advancing Geometric Dense Prediction with Powerful Image Generative Model
Jing He, Haodong Li, Mingzhi Sheng, Ying-Cong Chen
Recovering pixel-wise geometric properties from a single image is fundamentally ill-posed due to appearance ambiguity and non-injective mappings between 2D observations and 3D structures. While discriminative regression models achieve strong performance through large-scale supervision, their success is bounded by the scale, quality and diversity of available data and limited physical reasoning. Recent diffusion models exhibit powerful world priors that encode geometry and semantics learned from massive image-text data, yet directly reusing their stochastic generative formulation is suboptimal for deterministic geometric inference: the former is optimized for diverse and high-fidelity image generation, whereas the latter requires stable and accurate predictions. In this work, we propose Lotus-2, a two-stage deterministic framework for stable, accurate and fine-grained geometric dense prediction, aiming to provide an optimal adaption protocol to fully exploit the pre-trained generative priors. Specifically, in the first stage, the core predictor employs a single-step deterministic formulation with a clean-data objective and a lightweight local continuity module (LCM) to generate globally coherent structures without grid artifacts. In the second stage, the detail sharpener performs a constrained multi-step rectified-flow refinement within the manifold defined by the core predictor, enhancing fine-grained geometry through noise-free deterministic flow matching. Using only 59K training samples, less than 1% of existing large-scale datasets, Lotus-2 establishes new state-of-the-art results in monocular depth estimation and highly competitive surface normal prediction. These results demonstrate that diffusion models can serve as deterministic world priors, enabling high-quality geometric reasoning beyond traditional discriminative and generative paradigms.
comment : Work done at the Hong Kong University of Science and Technology (Guangzhou). Project page: https://lotus-2.github.io/. 15 Pages, 12 Figures, 3 Tables
☆ FOM-Nav: Frontier-Object Maps for Object Goal Navigation
Thomas Chabal, Shizhe Chen, Jean Ponce, Cordelia Schmid
This paper addresses the Object Goal Navigation problem, where a robot must efficiently find a target object in an unknown environment. Existing implicit memory-based methods struggle with long-term memory retention and planning, while explicit map-based approaches lack rich semantic information. To address these challenges, we propose FOM-Nav, a modular framework that enhances exploration efficiency through Frontier-Object Maps and vision-language models. Our Frontier-Object Maps are built online and jointly encode spatial frontiers and fine-grained object information. Using this representation, a vision-language model performs multimodal scene understanding and high-level goal prediction, which is executed by a low-level planner for efficient trajectory generation. To train FOM-Nav, we automatically construct large-scale navigation datasets from real-world scanned environments. Extensive experiments validate the effectiveness of our model design and constructed dataset. FOM-Nav achieves state-of-the-art performance on the MP3D and HM3D benchmarks, particularly in navigation efficiency metric SPL, and yields promising results on a real robot.
comment : Project page: https://www.di.ens.fr/willow/research/fom-nav/
☆ LISA-3D: Lifting Language-Image Segmentation to 3D via Multi-View Consistency
Zhongbin Guo, Jiahe Liu, Wenyu Gao, Yushan Li, Chengzhi Li, Ping Jian
Text-driven 3D reconstruction demands a mask generator that simultaneously understands open-vocabulary instructions and remains consistent across viewpoints. We present LISA-3D, a two-stage framework that lifts language-image segmentation into 3D by retrofitting the instruction-following model LISA with geometry-aware Low-Rank Adaptation (LoRA) layers and reusing a frozen SAM-3D reconstructor. During training we exploit off-the-shelf RGB-D sequences and their camera poses to build a differentiable reprojection loss that enforces cross-view agreement without requiring any additional 3D-text supervision. The resulting masks are concatenated with RGB images to form RGBA prompts for SAM-3D, which outputs Gaussian splats or textured meshes without retraining. Across ScanRefer and Nr3D, LISA-3D improves language-to-3D accuracy by up to +15.6 points over single-view baselines while adapting only 11.6M parameters. The system is modular, data-efficient, and supports zero-shot deployment on unseen categories, providing a practical recipe for language-guided 3D content creation. Our code will be available at https://github.com/binisalegend/LISA-3D.
☆ Provenance-Driven Reliable Semantic Medical Image Vector Reconstruction via Lightweight Blockchain-Verified Latent Fingerprints
Mohsin Rasheed, Abdullah Al-Mamun
Medical imaging is essential for clinical diagnosis, yet real-world data frequently suffers from corruption, noise, and potential tampering, challenging the reliability of AI-assisted interpretation. Conventional reconstruction techniques prioritize pixel-level recovery and may produce visually plausible outputs while compromising anatomical fidelity, an issue that can directly impact clinical outcomes. We propose a semantic-aware medical image reconstruction framework that integrates high-level latent embeddings with a hybrid U-Net architecture to preserve clinically relevant structures during restoration. To ensure trust and accountability, we incorporate a lightweight blockchain-based provenance layer using scale-free graph design, enabling verifiable recording of each reconstruction event without imposing significant overhead. Extensive evaluation across multiple datasets and corruption types demonstrates improved structural consistency, restoration accuracy, and provenance integrity compared with existing approaches. By uniting semantic-guided reconstruction with secure traceability, our solution advances dependable AI for medical imaging, enhancing both diagnostic confidence and regulatory compliance in healthcare environments.
☆ S2AM3D: Scale-controllable Part Segmentation of 3D Point Cloud
Han Su, Tianyu Huang, Zichen Wan, Xiaohe Wu, Wangmeng Zuo
Part-level point cloud segmentation has recently attracted significant attention in 3D computer vision. Nevertheless, existing research is constrained by two major challenges: native 3D models lack generalization due to data scarcity, while introducing 2D pre-trained knowledge often leads to inconsistent segmentation results across different views. To address these challenges, we propose S2AM3D, which incorporates 2D segmentation priors with 3D consistent supervision. We design a point-consistent part encoder that aggregates multi-view 2D features through native 3D contrastive learning, producing globally consistent point features. A scale-aware prompt decoder is then proposed to enable real-time adjustment of segmentation granularity via continuous scale signals. Simultaneously, we introduce a large-scale, high-quality part-level point cloud dataset with more than 100k samples, providing ample supervision signals for model training. Extensive experiments demonstrate that S2AM3D achieves leading performance across multiple evaluation settings, exhibiting exceptional robustness and controllability when handling complex structures and parts with significant size variations.
☆ PhotoFramer: Multi-modal Image Composition Instruction
Zhiyuan You, Ke Wang, He Zhang, Xin Cai, Jinjin Gu, Tianfan Xue, Chao Dong, Zhoutong Zhang
Composition matters during the photo-taking process, yet many casual users struggle to frame well-composed images. To provide composition guidance, we introduce PhotoFramer, a multi-modal composition instruction framework. Given a poorly composed image, PhotoFramer first describes how to improve the composition in natural language and then generates a well-composed example image. To train such a model, we curate a large-scale dataset. Inspired by how humans take photos, we organize composition guidance into a hierarchy of sub-tasks: shift, zoom-in, and view-change tasks. Shift and zoom-in data are sampled from existing cropping datasets, while view-change data are obtained via a two-stage pipeline. First, we sample pairs with varying viewpoints from multi-view datasets, and train a degradation model to transform well-composed photos into poorly composed ones. Second, we apply this degradation model to expert-taken photos to synthesize poor images to form training pairs. Using this dataset, we finetune a model that jointly processes and generates both text and images, enabling actionable textual guidance with illustrative examples. Extensive experiments demonstrate that textual instructions effectively steer image composition, and coupling them with exemplars yields consistent improvements over exemplar-only baselines. PhotoFramer offers a practical step toward composition assistants that make expert photographic priors accessible to everyday users. Codes, model weights, and datasets have been released in https://zhiyuanyou.github.io/photoframer.
☆ MM-ACT: Learn from Multimodal Parallel Generation to Act
Haotian Liang, Xinyi Chen, Bin Wang, Mingkang Chen, Yitian Liu, Yuhao Zhang, Zanxin Chen, Tianshuo Yang, Yilun Chen, Jiangmiao Pang, Dong Liu, Xiaokang Yang, Yao Mu, Wenqi Shao, Ping Luo
A generalist robotic policy needs both semantic understanding for task planning and the ability to interact with the environment through predictive capabilities. To tackle this, we present MM-ACT, a unified Vision-Language-Action (VLA) model that integrates text, image, and action in shared token space and performs generation across all three modalities. MM-ACT adopts a re-mask parallel decoding strategy for text and image generation, and employs a one-step parallel decoding strategy for action generation to improve efficiency. We introduce Context-Shared Multimodal Learning, a unified training paradigm that supervises generation in all three modalities from a shared context, enhancing action generation through cross-modal learning. Experiments were conducted on the LIBERO simulation and Franka real-robot setups as well as RoboTwin2.0 to assess in-domain and out-of-domain performances respectively. Our approach achieves a success rate of 96.3% on LIBERO, 72.0% across three tasks of real Franka, and 52.38% across eight bimanual tasks of RoboTwin2.0 with an additional gain of 9.25% from cross-modal learning. We release our codes, models and data at https://github.com/HHYHRHY/MM-ACT.
comment : 17 pages
☆ Efficient and Scalable Monocular Human-Object Interaction Motion Reconstruction
Boran Wen, Ye Lu, Keyan Wan, Sirui Wang, Jiahong Zhou, Junxuan Liang, Xinpeng Liu, Bang Xiao, Dingbang Huang, Ruiyang Liu, Yong-Lu Li
Generalized robots must learn from diverse, large-scale human-object interactions (HOI) to operate robustly in the real world. Monocular internet videos offer a nearly limitless and readily available source of data, capturing an unparalleled diversity of human activities, objects, and environments. However, accurately and scalably extracting 4D interaction data from these in-the-wild videos remains a significant and unsolved challenge. Thus, in this work, we introduce 4DHOISolver, a novel and efficient optimization framework that constrains the ill-posed 4D HOI reconstruction problem by leveraging sparse, human-in-the-loop contact point annotations, while maintaining high spatio-temporal coherence and physical plausibility. Leveraging this framework, we introduce Open4DHOI, a new large-scale 4D HOI dataset featuring a diverse catalog of 144 object types and 103 actions. Furthermore, we demonstrate the effectiveness of our reconstructions by enabling an RL-based agent to imitate the recovered motions. However, a comprehensive benchmark of existing 3D foundation models indicates that automatically predicting precise human-object contact correspondences remains an unsolved problem, underscoring the immediate necessity of our human-in-the-loop strategy while posing an open challenge to the community. Data and code will be publicly available at https://wenboran2002.github.io/open4dhoi/
☆ Adaptive Evidential Learning for Temporal-Semantic Robustness in Moment Retrieval AAAI 2026
Haojian Huang, Kaijing Ma, Jin Chen, Haodong Chen, Zhou Wu, Xianghao Zang, Han Fang, Chao Ban, Hao Sun, Mulin Chen, Zhongjiang He
In the domain of moment retrieval, accurately identifying temporal segments within videos based on natural language queries remains challenging. Traditional methods often employ pre-trained models that struggle with fine-grained information and deterministic reasoning, leading to difficulties in aligning with complex or ambiguous moments. To overcome these limitations, we explore Deep Evidential Regression (DER) to construct a vanilla Evidential baseline. However, this approach encounters two major issues: the inability to effectively handle modality imbalance and the structural differences in DER's heuristic uncertainty regularizer, which adversely affect uncertainty estimation. This misalignment results in high uncertainty being incorrectly associated with accurate samples rather than challenging ones. Our observations indicate that existing methods lack the adaptability required for complex video scenarios. In response, we propose Debiased Evidential Learning for Moment Retrieval (DEMR), a novel framework that incorporates a Reflective Flipped Fusion (RFF) block for cross-modal alignment and a query reconstruction task to enhance text sensitivity, thereby reducing bias in uncertainty estimation. Additionally, we introduce a Geom-regularizer to refine uncertainty predictions, enabling adaptive alignment with difficult moments and improving retrieval accuracy. Extensive testing on standard datasets and debiased datasets ActivityNet-CD and Charades-CD demonstrates significant enhancements in effectiveness, robustness, and interpretability, positioning our approach as a promising solution for temporal-semantic robustness in moment retrieval. The code is publicly available at https://github.com/KaijingOfficial/DEMR.
comment : Accepted by AAAI 2026, 10 pages, 9 figures, 5 tables
☆ Binary-Gaussian: Compact and Progressive Representation for 3D Gaussian Segmentation
An Yang, Chenyu Liu, Jun Du, Jianqing Gao, Jia Pan, Jinshui Hu, Baocai Yin, Bing Yin, Cong Liu
3D Gaussian Splatting (3D-GS) has emerged as an efficient 3D representation and a promising foundation for semantic tasks like segmentation. However, existing 3D-GS-based segmentation methods typically rely on high-dimensional category features, which introduce substantial memory overhead. Moreover, fine-grained segmentation remains challenging due to label space congestion and the lack of stable multi-granularity control mechanisms. To address these limitations, we propose a coarse-to-fine binary encoding scheme for per-Gaussian category representation, which compresses each feature into a single integer via the binary-to-decimal mapping, drastically reducing memory usage. We further design a progressive training strategy that decomposes panoptic segmentation into a series of independent sub-tasks, reducing inter-class conflicts and thereby enhancing fine-grained segmentation capability. Additionally, we fine-tune opacity during segmentation training to address the incompatibility between photometric rendering and semantic segmentation, which often leads to foreground-background confusion. Extensive experiments on multiple benchmarks demonstrate that our method achieves state-of-the-art segmentation performance while significantly reducing memory consumption and accelerating inference.
☆ SceneProp: Combining Neural Network and Markov Random Field for Scene-Graph Grounding WACV 2026
Keita Otani, Tatsuya Harada
Grounding complex, compositional visual queries with multiple objects and relationships is a fundamental challenge for vision-language models. While standard phrase grounding methods excel at localizing single objects, they lack the structural inductive bias to parse intricate relational descriptions, often failing as queries become more descriptive. To address this structural deficit, we focus on scene-graph grounding, a powerful but less-explored formulation where the query is an explicit graph of objects and their relationships. However, existing methods for this task also struggle, paradoxically showing decreased performance as the query graph grows -- failing to leverage the very information that should make grounding easier. We introduce SceneProp, a novel method that resolves this issue by reformulating scene-graph grounding as a Maximum a Posteriori (MAP) inference problem in a Markov Random Field (MRF). By performing global inference over the entire query graph, SceneProp finds the optimal assignment of image regions to nodes that jointly satisfies all constraints. This is achieved within an end-to-end framework via a differentiable implementation of the Belief Propagation algorithm. Experiments on four benchmarks show that our dedicated focus on the scene-graph grounding formulation allows SceneProp to significantly outperform prior work. Critically, its accuracy consistently improves with the size and complexity of the query graph, demonstrating for the first time that more relational context can, and should, lead to better grounding. Codes are available at https://github.com/keitaotani/SceneProp.
comment : Accepted to WACV 2026
☆ LAHNet: Local Attentive Hashing Network for Point Cloud Registration
Wentao Qu, Xiaoshui Huang, Liang Xiao
Most existing learning-based point cloud descriptors for point cloud registration focus on perceiving local information of point clouds to generate distinctive features. However, a reasonable and broader receptive field is essential for enhancing feature distinctiveness. In this paper, we propose a Local Attentive Hashing Network for point cloud registration, called LAHNet, which introduces a local attention mechanism with the inductive bias of locality of convolution-like operators into point cloud descriptors. Specifically, a Group Transformer is designed to capture reasonable long-range context between points. This employs a linear neighborhood search strategy, Locality-Sensitive Hashing, enabling uniformly partitioning point clouds into non-overlapping windows. Meanwhile, an efficient cross-window strategy is adopted to further expand the reasonable feature receptive field. Furthermore, building on this effective windowing strategy, we propose an Interaction Transformer to enhance the feature interactions of the overlap regions within point cloud pairs. This computes an overlap matrix to match overlap regions between point cloud pairs by representing each window as a global signal. Extensive results demonstrate that LAHNet can learn robust and distinctive features, achieving significant registration results on real-world indoor and outdoor benchmarks.
☆ ForamDeepSlice: A High-Accuracy Deep Learning Framework for Foraminifera Species Classification from 2D Micro-CT Slices
Abdelghafour Halimi, Ali Alibrahim, Didier Barradas-Bautista, Ronell Sicat, Abdulkader M. Afifi
This study presents a comprehensive deep learning pipeline for the automated classification of 12 foraminifera species using 2D micro-CT slices derived from 3D scans. We curated a scientifically rigorous dataset comprising 97 micro-CT scanned specimens across 27 species, selecting 12 species with sufficient representation for robust machine learning. To ensure methodological integrity and prevent data leakage, we employed specimen-level data splitting, resulting in 109,617 high-quality 2D slices (44,103 for training, 14,046 for validation, and 51,468 for testing). We evaluated seven state-of-the-art 2D convolutional neural network (CNN) architectures using transfer learning. Our final ensemble model, combining ConvNeXt-Large and EfficientNetV2-Small, achieved a test accuracy of 95.64%, with a top-3 accuracy of 99.6% and an area under the ROC curve (AUC) of 0.998 across all species. To facilitate practical deployment, we developed an interactive advanced dashboard that supports real-time slice classification and 3D slice matching using advanced similarity metrics, including SSIM, NCC, and the Dice coefficient. This work establishes new benchmarks for AI-assisted micropaleontological identification and provides a fully reproducible framework for foraminifera classification research, bridging the gap between deep learning and applied geosciences.
☆ Dual-Projection Fusion for Accurate Upright Panorama Generation in Robotic Vision
Yuhao Shan, Qianyi Yuan, Jingguo Liu, Shigang Li, Jianfeng Li, Tong Chen
Panoramic cameras, capable of capturing a 360-degree field of view, are crucial in robotic vision, particularly in environments with sparse features. However, non-upright panoramas due to unstable robot postures hinder downstream tasks. Traditional IMU-based correction methods suffer from drift and external disturbances, while vision-based approaches offer a promising alternative. This study presents a dual-stream angle-aware generation network that jointly estimates camera inclination angles and reconstructs upright panoramic images. The network comprises a CNN branch that extracts local geometric structures from equirectangular projections and a ViT branch that captures global contextual cues from cubemap projections. These are integrated through a dual-projection adaptive fusion module that aligns spatial features across both domains. To further enhance performance, we introduce a high-frequency enhancement block, circular padding, and channel attention mechanisms to preserve 360° continuity and improve geometric sensitivity. Experiments on the SUN360 and M3D datasets demonstrate that our method outperforms existing approaches in both inclination estimation and upright panorama generation. Ablation studies further validate the contribution of each module and highlight the synergy between the two tasks. The code and related datasets can be found at: https://github.com/YuhaoShine/DualProjectionFusion.
☆ TalkingPose: Efficient Face and Gesture Animation with Feedback-guided Diffusion Model WACV 2026
Alireza Javanmardi, Pragati Jaiswal, Tewodros Amberbir Habtegebrial, Christen Millerdurai, Shaoxiang Wang, Alain Pagani, Didier Stricker
Recent advancements in diffusion models have significantly improved the realism and generalizability of character-driven animation, enabling the synthesis of high-quality motion from just a single RGB image and a set of driving poses. Nevertheless, generating temporally coherent long-form content remains challenging. Existing approaches are constrained by computational and memory limitations, as they are typically trained on short video segments, thus performing effectively only over limited frame lengths and hindering their potential for extended coherent generation. To address these constraints, we propose TalkingPose, a novel diffusion-based framework specifically designed for producing long-form, temporally consistent human upper-body animations. TalkingPose leverages driving frames to precisely capture expressive facial and hand movements, transferring these seamlessly to a target actor through a stable diffusion backbone. To ensure continuous motion and enhance temporal coherence, we introduce a feedback-driven mechanism built upon image-based diffusion models. Notably, this mechanism does not incur additional computational costs or require secondary training stages, enabling the generation of animations with unlimited duration. Additionally, we introduce a comprehensive, large-scale dataset to serve as a new benchmark for human upper-body animation.
comment : WACV 2026, Project page available at https://dfki-av.github.io/TalkingPose
☆ Hierarchical Semantic Alignment for Image Clustering AAAI 2026
Xingyu Zhu, Beier Zhu, Yunfan Li, Junfeng Fang, Shuo Wang, Kesen Zhao, Hanwang Zhang
Image clustering is a classic problem in computer vision, which categorizes images into different groups. Recent studies utilize nouns as external semantic knowledge to improve clus- tering performance. However, these methods often overlook the inherent ambiguity of nouns, which can distort semantic representations and degrade clustering quality. To address this issue, we propose a hierarChical semAntic alignmEnt method for image clustering, dubbed CAE, which improves cluster- ing performance in a training-free manner. In our approach, we incorporate two complementary types of textual seman- tics: caption-level descriptions, which convey fine-grained attributes of image content, and noun-level concepts, which represent high-level object categories. We first select relevant nouns from WordNet and descriptions from caption datasets to construct a semantic space aligned with image features. Then, we align image features with selected nouns and captions via optimal transport to obtain a more discriminative semantic space. Finally, we combine the enhanced semantic and image features to perform clustering. Extensive experiments across 8 datasets demonstrate the effectiveness of our method, notably surpassing the state-of-the-art training-free approach with a 4.2% improvement in accuracy and a 2.9% improvement in adjusted rand index (ARI) on the ImageNet-1K dataset.
comment : AAAI 2026
☆ SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overhead
Chaojun Ni, Cheng Chen, Xiaofeng Wang, Zheng Zhu, Wenzhao Zheng, Boyuan Wang, Tianrun Chen, Guosheng Zhao, Haoyun Li, Zhehao Dong, Qiang Zhang, Yun Ye, Yang Wang, Guan Huang, Wenjun Mei
Vision-Language-Action (VLA) models built on pretrained Vision-Language Models (VLMs) show strong potential but are limited in practicality due to their large parameter counts. To mitigate this issue, using a lightweight VLM has been explored, but it compromises spatiotemporal reasoning. Although some methods suggest that incorporating additional 3D inputs can help, they usually rely on large VLMs to fuse 3D and 2D inputs and still lack temporal understanding. Therefore, we propose SwiftVLA, an architecture that enhances a compact model with 4D understanding while preserving design efficiency. Specifically, our approach features a pretrained 4D visual geometry transformer with a temporal cache that extracts 4D features from 2D images. Then, to enhance the VLM's ability to exploit both 2D images and 4D features, we introduce Fusion Tokens, a set of learnable tokens trained with a future prediction objective to generate unified representations for action generation. Finally, we introduce a mask-and-reconstruct strategy that masks 4D inputs to the VLM and trains the VLA to reconstruct them, enabling the VLM to learn effective 4D representations and allowing the 4D branch to be dropped at inference with minimal performance loss. Experiments in real and simulated environments show that SwiftVLA outperforms lightweight baselines and rivals VLAs up to 7 times larger, achieving comparable performance on edge devices while being 18 times faster and reducing memory footprint by 12 times.
☆ Accelerating Streaming Video Large Language Models via Hierarchical Token Compression
Yiyu Wang, Xuyang Liu, Xiyan Gui, Xinying Lin, Boxue Yang, Chenfei Liao, Tailai Chen, Linfeng Zhang
Streaming Video Large Language Models (VideoLLMs) have demonstrated impressive performance across various video understanding tasks, but they face significant challenges in real-time deployment due to the high computational cost of processing dense visual tokens from continuous video streams. In streaming video scenarios, the primary bottleneck lies in the Vision Transformer (ViT) encoding stage, where redundant processing of temporally similar frames leads to inefficiency. Additionally, inflated token sequences during LLM pre-filling further exacerbate latency and memory overhead. To address these challenges, we propose \textbf{S}treaming \textbf{T}oken \textbf{C}ompression (\textbf{STC}), a plug-and-play hierarchical framework that seamlessly integrates into existing streaming VideoLLMs, optimizing both ViT encoding and LLM pre-filling stages to accelerate processing. STC introduces two token-level accelerators: \textbf{STC-Cacher}, which reduces ViT encoding overhead by caching and reusing features from temporally similar frames, and \textbf{STC-Pruner}, which compresses the visual token sequence before it enters the LLM, preserving only the most salient tokens based on both spatial and temporal relevance. Extensive experiments on four baseline streaming VideoLLMs across five benchmarks demonstrate that STC outperforms other compression methods. Notably, STC retains up to \textbf{99\%} of accuracy on the ReKV framework while reducing ViT encoding latency and LLM pre-filling latency by \textbf{24.5\%} and \textbf{45.3\%}.
comment : Code is avaliable at \url{https://github.com/lern-to-write/STC}
☆ Multilingual Training-Free Remote Sensing Image Captioning
Carlos Rebelo, Gil Rocha, João Daniel Silva, Bruno Martins
Remote sensing image captioning has advanced rapidly through encoder--decoder models, although the reliance on large annotated datasets and the focus on English restricts global applicability. To address these limitations, we propose the first training-free multilingual approach, based on retrieval-augmented prompting. For a given aerial image, we employ a domain-adapted SigLIP2 encoder to retrieve related captions and few-shot examples from a datastore, which are then provided to a language model. We explore two variants: an image-blind setup, where a multilingual Large Language Model (LLM) generates the caption from textual prompts alone, and an image-aware setup, where a Vision--Language Model (VLM) jointly processes the prompt and the input image. To improve the coherence of the retrieved content, we introduce a graph-based re-ranking strategy using PageRank on a graph of images and captions. Experiments on four benchmark datasets across ten languages demonstrate that our approach is competitive with fully supervised English-only systems and generalizes to other languages. Results also highlight the importance of re-ranking with PageRank, yielding up to 35% improvements in performance metrics. Additionally, it was observed that while VLMs tend to generate visually grounded but lexically diverse captions, LLMs can achieve stronger BLEU and CIDEr scores. Lastly, directly generating captions in the target language consistently outperforms other translation-based strategies. Overall, our work delivers one of the first systematic evaluations of multilingual, training-free captioning for remote sensing imagery, advancing toward more inclusive and scalable multimodal Earth observation systems.
☆ HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamics
Masatoshi Tateno, Gido Kato, Hirokatsu Kataoka, Yoichi Sato, Takuma Yagi
Hand-object interaction (HOI) inherently involves dynamics where human manipulations produce distinct spatio-temporal effects on objects. However, existing semantic HOI benchmarks focused either on manipulation or on the resulting effects at a coarse level, lacking fine-grained spatio-temporal reasoning to capture the underlying dynamics in HOI. We introduce HanDyVQA, a fine-grained video question-answering benchmark that comprehensively covers both the manipulation and effect aspects of HOI. HanDyVQA comprises six complementary question types (Action, Process, Objects, Location, State Change, and Object Parts), totalling 11.1K multiple-choice QA pairs. Collected QA pairs recognizing manipulation styles, hand/object motions, and part-level state changes. HanDyVQA also includes 10.3K segmentation masks for Objects and Object Parts questions, enabling the evaluation of object/part-level reasoning in video object segmentation. We evaluated recent video foundation models on our benchmark and found that even the best-performing model, Gemini-2.5-Pro, reached only 73% average accuracy, which is far from human performance (97%). Further analysis shows the remaining challenges in spatial relationship, motion, and part-level geometric understanding. We also found that integrating explicit HOI-related cues into visual features improves performance, offering insights for developing future models with a deeper understanding of HOI dynamics.
comment : Project page: https://masatate.github.io/HanDyVQA-project-page/
☆ Audio-Visual Wo rl d Models: Towards Multisensory Imagination in Sight and Sound
Jiahua Wang, Shannan Yan, Leqi Zheng, Jialong Wu, Yaoxin Mao
World models simulate environmental dynamics to enable agents to plan and reason about future states. While existing approaches have primarily focused on visual observations, real-world perception inherently involves multiple sensory modalities. Audio provides crucial spatial and temporal cues such as sound source localization and acoustic scene properties, yet its integration into world models remains largely unexplored. No prior work has formally defined what constitutes an audio-visual world model or how to jointly capture binaural spatial audio and visual dynamics under precise action control with task reward prediction. This work presents the first formal framework for Audio-Visual World Models (AVWM), formulating multimodal environment simulation as a partially observable Markov decision process with synchronized audio-visual observations, fine-grained actions, and task rewards. To address the lack of suitable training data, we construct AVW-4k, a dataset comprising 30 hours of binaural audio-visual trajectories with action annotations and reward signals across 76 indoor environments. We propose AV-CDiT, an Audio-Visual Conditional Diffusion Transformer with a novel modality expert architecture that balances visual and auditory learning, optimized through a three-stage training strategy for effective multimodal integration. Extensive experiments demonstrate that AV-CDiT achieves high-fidelity multimodal prediction across visual and auditory modalities with reward. Furthermore, we validate its practical utility in continuous audio-visual navigation tasks, where AVWM significantly enhances the agent's performance.
☆ Look, Recite, Then Answer: Enhancing VLM Performance via Self-Generated Knowledge Hints
Xisheng Feng
Vision-Language Models (VLMs) exhibit significant performance plateaus in specialized domains like precision agriculture, primarily due to "Reasoning-Driven Hallucination" where linguistic priors override visual perception. A key bottleneck is the "Modality Gap": visual embeddings fail to reliably activate the fine-grained expert knowledge already encoded in model parameters. We propose "Look, Recite, Then Answer," a parameter-efficient framework that enhances VLMs via self-generated knowledge hints while keeping backbone models frozen. The framework decouples inference into three stages: (1) Look generates objective visual descriptions and candidate sets; (2) Recite employs a lightweight 1.7B router to transform visual cues into targeted queries that trigger candidate-specific parametric knowledge; (3) Answer performs parallel evidence alignment between descriptions and recited knowledge to select the most consistent label. On AgroBench, our method achieves state-of-the-art results, improving Weed Identification accuracy by 23.6% over Qwen-VL and surpassing GPT-4o without external search overhead. This modular design mitigates hallucinations by transforming passive perception into active, controllable knowledge retrieval
☆ Quantum-Inspired Spectral Geometry for Neural Operator Equivalence and Structured Pruning
Haijian Shao, Wei Liu, Xing Deng
The rapid growth of multimodal intelligence on resource-constrained and heterogeneous domestic hardware exposes critical bottlenecks: multimodal feature heterogeneity, real-time requirements in dynamic scenarios, and hardware-specific operator redundancy. This work introduces a quantum-inspired geometric framework for neural operators that represents each operator by its normalized singular value spectrum on the Bloch hypersphere. We prove a tight spectral-to-functional equivalence theorem showing that vanishing Fubini--Study/Wasserstein-2 distance implies provable functional closeness, establishing the first rigorous foundation for cross-modal and cross-architecture operator substitutability. Based on this metric, we propose Quantum Metric-Driven Functional Redundancy Graphs (QM-FRG) and one-shot structured pruning. Controlled simulation validates the superiority of the proposed metric over magnitude and random baselines. An extensive experimental validation on large-scale multimodal transformers and domestic heterogeneous hardware (Huawei Ascend, Cambricon MLU, Kunlunxin) hardware is deferred to an extended journal version currently in preparation.
comment : 6 pages, 1 figure, preliminary version; concepts and simulation experiments only
☆ Feed-Forward 3D Gaussian Splatting Compression with Long- Context Modeling
Zhening Liu, Rui Song, Yushi Huang, Yingdong Hu, Xinjie Zhang, Jiawei Shao, Zehong Lin, Jun Zhang
3D Gaussian Splatting (3DGS) has emerged as a revolutionary 3D representation. However, its substantial data size poses a major barrier to widespread adoption. While feed-forward 3DGS compression offers a practical alternative to costly per-scene per-train compressors, existing methods struggle to model long-range spatial dependencies, due to the limited receptive field of transform coding networks and the inadequate context capacity in entropy models. In this work, we propose a novel feed-forward 3DGS compression framework that effectively models long-range correlations to enable highly compact and generalizable 3D representations. Central to our approach is a large-scale context structure that comprises thousands of Gaussians based on Morton serialization. We then design a fine-grained space-channel auto-regressive entropy model to fully leverage this expansive context. Furthermore, we develop an attention-based transform coding model to extract informative latent priors by aggregating features from a wide range of neighboring Gaussians. Our method yields a $20\times$ compression ratio for 3DGS in a feed-forward inference and achieves state-of-the-art performance among generalizable codecs.
☆ Neural Discrete Representation Learning for Sparse-View CBCT Reconstruction: From Algorithm Design to Prospective Multicenter Clinical Evaluation
Haoshen Wang, Lei Chen, Wei-Hua Zhang, Linxia Wu, Yong Luo, Zengmao Wang, Yuan Xiong, Chengcheng Zhu, Wenjuan Tang, Xueyi Zhang, Wei Zhou, Xuhua Duan, Lefei Zhang, Gao-Jun Teng, Bo Du, Huangxuan Zhao
Cone beam computed tomography (CBCT)-guided puncture has become an established approach for diagnosing and treating early- to mid-stage thoracic tumours, yet the associated radiation exposure substantially elevates the risk of secondary malignancies. Although multiple low-dose CBCT strategies have been introduced, none have undergone validation using large-scale multicenter retrospective datasets, and prospective clinical evaluation remains lacking. Here, we propose DeepPriorCBCT - a three-stage deep learning framework that achieves diagnostic-grade reconstruction using only one-sixth of the conventional radiation dose. 4102 patients with 8675 CBCT scans from 12 centers were included to develop and validate DeepPriorCBCT. Additionally, a prospective cross-over trial (Registry number: NCT07035977) which recruited 138 patients scheduled for percutaneous thoracic puncture was conducted to assess the model's clinical applicability. Assessment by 11 physicians confirmed that reconstructed images were indistinguishable from original scans. Moreover, diagnostic performance and overall image quality were comparable to those generated by standard reconstruction algorithms. In the prospective trial, five radiologists reported no significant differences in image quality or lesion assessment between DeepPriorCBCT and the clinical standard (all P>0.05). Likewise, 25 interventionalists expressed no preference between model-based and full-sampling images for surgical guidance (Kappa<0.2). Radiation exposure with DeepPriorCBCT was reduced to approximately one-sixth of that with the conventional approach, and collectively, the findings confirm that it enables high-quality CBCT reconstruction under sparse sampling conditions while markedly decreasing intraoperative radiation risk.
☆ TAP-CT: 3D Task-Agnostic Pretrain ing of Computed Tomography Foundation Models
Tim Veenboer, George Yiasemis, Eric Marcus, Vivien Van Veldhuizen, Cees G. M. Snoek, Jonas Teuwen, Kevin B. W. Groot Lipman
Existing foundation models (FMs) in the medical domain often require extensive fine-tuning or rely on training resource-intensive decoders, while many existing encoders are pretrained with objectives biased toward specific tasks. This illustrates a need for a strong, task-agnostic foundation model that requires minimal fine-tuning beyond feature extraction. In this work, we introduce a suite of task-agnostic pretraining of CT foundation models (TAP-CT): a simple yet effective adaptation of Vision Transformers (ViTs) and DINOv2 for volumetric data, enabling scalable self-supervised pretraining directly on 3D CT volumes. Our approach incorporates targeted modifications to patch embeddings, positional encodings, and volumetric augmentations, making the architecture depth-aware while preserving the simplicity of the underlying architectures. We show that large-scale 3D pretraining on an extensive in-house CT dataset (105K volumes) yields stable, robust frozen representations that generalize strongly across downstream tasks. To promote transparency and reproducibility, and to establish a powerful, low-resource baseline for future research in medical imaging, we will release all pretrained models, experimental configurations, and downstream benchmark code at https://huggingface.co/fomofo/tap-ct-b-3d.
comment : 22 pages, 4 figures, 8 tables
☆ Smol-GS: Compact Representations for Abstract 3D Gaussian Splatting
Haishan Wang, Mohammad Hassan Vali, Arno Solin
We present Smol-GS, a novel method for learning compact representations for 3D Gaussian Splatting (3DGS). Our approach learns highly efficient encodings in 3D space that integrate both spatial and semantic information. The model captures the coordinates of the splats through a recursive voxel hierarchy, while splat-wise features store abstracted cues, including color, opacity, transformation, and material properties. This design allows the model to compress 3D scenes by orders of magnitude without loss of flexibility. Smol-GS achieves state-of-the-art compression on standard benchmarks while maintaining high rendering quality. Beyond visual fidelity, the discrete representations could potentially serve as a foundation for downstream tasks such as navigation, planning, and broader 3D scene understanding.
☆ AFR Agent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agent WACV 2026
Neeraj Anand, Rishabh Jain, Sohan Patnaik, Balaji Krishnamurthy, Mausoom Sarkar
There is a growing demand for mobile user interface (UI) automation, driven by its broad applications across industries. With the advent of visual language models (VLMs), GUI automation has progressed from generating text-based instructions for humans to autonomously executing tasks, thus optimizing automation workflows. Recent approaches leverage VLMs for this problem due to their ability to 1) process on-screen content directly, 2) remain independent of device-specific APIs by utilizing human actions (e.g., clicks, typing), and 3) apply real-world contextual knowledge for task understanding. However, these models often have trouble accurately identifying widgets and determining actions due to limited spatial information in vision encoder features. Additionally, top-performing models are often large, requiring extensive training and resulting in inference delays. In this work, we introduce AFRAgent, an instruct-BLIP-based multimodal architecture that achieves superior performance in GUI automation while being less than one-fourth the size of its nearest competitor. To enhance image embeddings in the large language model (LLM) pipeline, we propose an adaptive feature renormalization-based (a token-level affine transformation) technique that effectively enriches low-resolution image embeddings and fuses high-resolution details. We evaluate AFRAgent on Meta-GUI and AITW benchmarks, establishing a new state-of-the-art baseline for smartphone automation.
comment : Accepted at WACV 2026 Conference
☆ PanFlow: Decoupled Motion Control for Panoramic Video Generation AAAI
Cheng Zhang, Hanwen Liang, Donny Y. Chen, Qianyi Wu, Konstantinos N. Plataniotis, Camilo Cruz Gambardella, Jianfei Cai
Panoramic video generation has attracted growing attention due to its applications in virtual reality and immersive media. However, existing methods lack explicit motion control and struggle to generate scenes with large and complex motions. We propose PanFlow, a novel approach that exploits the spherical nature of panoramas to decouple the highly dynamic camera rotation from the input optical flow condition, enabling more precise control over large and dynamic motions. We further introduce a spherical noise warping strategy to promote loop consistency in motion across panorama boundaries. To support effective training, we curate a large-scale, motion-rich panoramic video dataset with frame-level pose and flow annotations. We also showcase the effectiveness of our method in various applications, including motion transfer and video editing. Extensive experiments demonstrate that PanFlow significantly outperforms prior methods in motion fidelity, visual quality, and temporal coherence. Our code, dataset, and models are available at https://github.com/chengzhag/PanFlow.
comment : Accepted by AAAI. Code: https://github.com/chengzhag/PanFlow
☆ Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning
Haozhen Gong, Xiaozhong Ji, Yuansen Liu, Wenbin Wu, Xiaoxiao Yan, Jingjing Liu, Kai Wu, Jiazhen Pan, Bailiang Jian, Jiangning Zhang, Xiaobin Hu, Hongwei Bran Li
MLLMs MLLMs are beginning to appear in clinical workflows, but their ability to perform complex medical reasoning remains unclear. We present Med-CMR, a fine-grained Medical Complex Multimodal Reasoning benchmark. Med-CMR distinguishes from existing counterparts by three core features: 1) Systematic capability decomposition, splitting medical multimodal reasoning into fine-grained visual understanding and multi-step reasoning to enable targeted evaluation; 2) Challenging task design, with visual understanding across three key dimensions (small-object detection, fine-detail discrimination, spatial understanding) and reasoning covering four clinically relevant scenarios (temporal prediction, causal reasoning, long-tail generalization, multi-source integration); 3) Broad, high-quality data coverage, comprising 20,653 Visual Question Answering (VQA) pairs spanning 11 organ systems and 12 imaging modalities, validated via a rigorous two-stage (human expert + model-assisted) review to ensure clinical authenticity. We evaluate 18 state-of-the-art MLLMs with Med-CMR, revealing GPT-5 as the top-performing commercial model: 57.81 accuracy on multiple-choice questions (MCQs) and a 48.70 open-ended score, outperforming Gemini 2.5 Pro (49.87 MCQ accuracy, 45.98 open-ended score) and leading open-source model Qwen3-VL-235B-A22B (49.34 MCQ accuracy, 42.62 open-ended score). However, specialized medical MLLMs do not reliably outperform strong general models, and long-tail generalization emerges as the dominant failure mode. Med-CMR thus provides a stress test for visual-reasoning integration and rare-case robustness in medical MLLMs, and a rigorous yardstick for future clinical systems.
☆ IRPO: Boosting Image Restoration via Post-training GRPO
Haoxuan Xu. Yi Liu, Boyuan Jiang, Jinlong Peng, Donghao Luo, Xiaobin Hu, Shuicheng Yan, Haoang Li
Recent advances in post-training paradigms have achieved remarkable success in high-level generation tasks, yet their potential for low-level vision remains rarely explored. Existing image restoration (IR) methods rely on pixel-level hard-fitting to ground-truth images, struggling with over-smoothing and poor generalization. To address these limitations, we propose IRPO, a low-level GRPO-based post-training paradigm that systematically explores both data formulation and reward modeling. We first explore a data formulation principle for low-level post-training paradigm, in which selecting underperforming samples from the pre-training stage yields optimal performance and improved efficiency. Furthermore, we model a reward-level criteria system that balances objective accuracy and human perceptual preference through three complementary components: a General Reward for structural fidelity, an Expert Reward leveraging Qwen-VL for perceptual alignment, and a Restoration Reward for task-specific low-level quality. Comprehensive experiments on six in-domain and five out-of-domain (OOD) low-level benchmarks demonstrate that IRPO achieves state-of-the-art results across diverse degradation types, surpassing the AdaIR baseline by 0.83 dB on in-domain tasks and 3.43 dB on OOD settings. Our code can be shown in https://github.com/HaoxuanXU1024/IRPO.
☆ Thinking with Drafts: Speculative Temporal Reasoning for Efficient Long Video Understanding
Pengfei Hu, Meng Cao, Yingyao Wang, Yi Wang, Jiahua Dong, Jun Song, Yu Cheng, Bo Zheng, Xiaodan Liang
Long video understanding is essential for human-like intelligence, enabling coherent perception and reasoning over extended temporal contexts. While the emerging thinking-with-frames paradigm, which alternates between global temporal reasoning and local frame examination, has advanced the reasoning capabilities of video multi-modal large language models (MLLMs), it suffers from a significant efficiency bottleneck due to the progressively growing and redundant multi-modal context. To address this, we propose SpecTemp, a reinforcement learning-based Speculative Temporal reasoning framework that decouples temporal perception from reasoning via a cooperative dual-model design. In SpecTemp, a lightweight draft MLLM rapidly explores and proposes salient frames from densely sampled temporal regions, while a powerful target MLLM focuses on temporal reasoning and verifies the draft's proposals, iteratively refining its attention until convergence. This design mirrors the collaborative pathways of the human brain, balancing efficiency with accuracy. To support training, we construct the SpecTemp-80K dataset, featuring synchronized dual-level annotations for coarse evidence spans and fine-grained frame-level evidence. Experiments across multiple video understanding benchmarks demonstrate that SpecTemp not only maintains competitive accuracy but also significantly accelerates inference compared with existing thinking-with-frames methods.
☆ CircleFlow: Flow-Guided Camera Blur Estimation using a Circle Grid Target
Jiajian He, Enjie Hu, Shiqi Chen, Tianchen Qiu, Huajun Feng, Zhihai Xu, Yueting Chen
The point spread function (PSF) serves as a fundamental descriptor linking the real-world scene to the captured signal, manifesting as camera blur. Accurate PSF estimation is crucial for both optical characterization and computational vision, yet remains challenging due to the inherent ambiguity and the ill-posed nature of intensity-based deconvolution. We introduce CircleFlow, a high-fidelity PSF estimation framework that employs flow-guided edge localization for precise blur characterization. CircleFlow begins with a structured capture that encodes locally anisotropic and spatially varying PSFs by imaging a circle grid target, while leveraging the target's binary luminance prior to decouple image and kernel estimation. The latent sharp image is then reconstructed through subpixel alignment of an initialized binary structure guided by optical flow, whereas the PSF is modeled as an energy-constrained implicit neural representation. Both components are jointly optimized within a demosaicing-aware differentiable framework, ensuring physically consistent and robust PSF estimation enabled by accurate edge localization. Extensive experiments on simulated and real-world data demonstrate that CircleFlow achieves state-of-the-art accuracy and reliability, validating its effectiveness for practical PSF calibration.
☆ PolarGS: Polarimetric Cues for Ambiguity-Free Gaussian Splatting with Accurate Geometry Recovery
Bo Guo, Sijia Wen, Yifan Zhao, Jia Li, Zhiming Zheng
Recent advances in surface reconstruction for 3D Gaussian Splatting (3DGS) have enabled remarkable geometric accuracy. However, their performance degrades in photometrically ambiguous regions such as reflective and textureless surfaces, where unreliable cues disrupt photometric consistency and hinder accurate geometry estimation. Reflected light is often partially polarized in a manner that reveals surface orientation, making polarization an optic complement to photometric cues in resolving such ambiguities. Therefore, we propose PolarGS, an optics-aware extension of RGB-based 3DGS that leverages polarization as an optical prior to resolve photometric ambiguities and enhance reconstruction accuracy. Specifically, we introduce two complementary modules: a polarization-guided photometric correction strategy, which ensures photometric consistency by identifying reflective regions via the Degree of Linear Polarization (DoLP) and refining reflective Gaussians with Color Refinement Maps; and a polarization-enhanced Gaussian densification mechanism for textureless area geometry recovery, which integrates both Angle and Degree of Linear Polarization (A/DoLP) into a PatchMatch-based depth completion process. This enables the back-projection and fusion of new Gaussians, leading to more complete reconstruction. PolarGS is framework-agnostic and achieves superior geometric accuracy compared to state-of-the-art methods.
☆ Sign Language Recognition using Bidirectional Reservoir Computing
Nitin Kumar Singh, Arie Rachmad Syulistyo, Yuichiro Tanaka, Hakaru Tamukoh
Sign language recognition (SLR) facilitates communication between deaf and hearing individuals. Deep learning is widely used to develop SLR-based systems; however, it is computationally intensive and requires substantial computational resources, making it unsuitable for resource-constrained devices. To address this, we propose an efficient sign language recognition system using MediaPipe and an echo state network (ESN)-based bidirectional reservoir computing (BRC) architecture. MediaPipe extracts hand joint coordinates, which serve as inputs to the ESN-based BRC architecture. The BRC processes these features in both forward and backward directions, efficiently capturing temporal dependencies. The resulting states of BRC are concatenated to form a robust representation for classification. We evaluated our method on the Word-Level American Sign Language (WLASL) video dataset, achieving a competitive accuracy of 57.71% and a significantly lower training time of only 9 seconds, in contrast to the 55 minutes and $38$ seconds required by the deep learning-based Bi-GRU approach. Consequently, the BRC-based SLR system is well-suited for edge devices.
☆ DEJIMA: A Novel Large-scale Japanese Dataset for Image Captioning and Visual Question Answering
Toshiki Katsube, Taiga Fukuhara, Kenichiro Ando, Yusuke Mukuta, Kohei Uehara, Tatsuya Harada
This work addresses the scarcity of high-quality, large-scale resources for Japanese Vision-and-Language (V&L) modeling. We present a scalable and reproducible pipeline that integrates large-scale web collection with rigorous filtering/deduplication, object-detection-driven evidence extraction, and Large Language Model (LLM)-based refinement under grounding constraints. Using this pipeline, we build two resources: an image-caption dataset (DEJIMA-Cap) and a VQA dataset (DEJIMA-VQA), each containing 3.88M image-text pairs, far exceeding the size of existing Japanese V&L datasets. Human evaluations demonstrate that DEJIMA achieves substantially higher Japaneseness and linguistic naturalness than datasets constructed via translation or manual annotation, while maintaining factual correctness at a level comparable to human-annotated corpora. Quantitative analyses of image feature distributions further confirm that DEJIMA broadly covers diverse visual domains characteristic of Japan, complementing its linguistic and cultural representativeness. Models trained on DEJIMA exhibit consistent improvements across multiple Japanese multimodal benchmarks, confirming that culturally grounded, large-scale resources play a key role in enhancing model performance. All data sources and modules in our pipeline are licensed for commercial use, and we publicly release the resulting dataset and metadata to encourage further research and industrial applications in Japanese V&L modeling.
☆ EAG3R: Event-Augmented 3D Geometry Estimation for Dynamic and Extreme-Lighting Scenes NeurIPS 2025
Xiaoshan Wu, Yifei Yu, Xiaoyang Lyu, Yihua Huang, Bo Wang, Baoheng Zhang, Zhongrui Wang, Xiaojuan Qi
Robust 3D geometry estimation from videos is critical for applications such as autonomous navigation, SLAM, and 3D scene reconstruction. Recent methods like DUSt3R demonstrate that regressing dense pointmaps from image pairs enables accurate and efficient pose-free reconstruction. However, existing RGB-only approaches struggle under real-world conditions involving dynamic objects and extreme illumination, due to the inherent limitations of conventional cameras. In this paper, we propose EAG3R, a novel geometry estimation framework that augments pointmap-based reconstruction with asynchronous event streams. Built upon the MonST3R backbone, EAG3R introduces two key innovations: (1) a retinex-inspired image enhancement module and a lightweight event adapter with SNR-aware fusion mechanism that adaptively combines RGB and event features based on local reliability; and (2) a novel event-based photometric consistency loss that reinforces spatiotemporal coherence during global optimization. Our method enables robust geometry estimation in challenging dynamic low-light scenes without requiring retraining on night-time data. Extensive experiments demonstrate that EAG3R significantly outperforms state-of-the-art RGB-only baselines across monocular depth estimation, camera pose tracking, and dynamic reconstruction tasks.
comment : Accepted at NeurIPS 2025 (spotlight)
☆ The Outline of Deception: Physical Adversarial Attacks on Traffic Signs Using Edge Patches
Haojie Jia, Te Hu, Haowen Li, Long Jin, Chongshi Xin, Yuchi Yao, Jiarui Xiao
Intelligent driving systems are vulnerable to physical adversarial attacks on traffic signs. These attacks can cause misclassification, leading to erroneous driving decisions that compromise road safety. Moreover, within V2X networks, such misinterpretations can propagate, inducing cascading failures that disrupt overall traffic flow and system stability. However, a key limitation of current physical attacks is their lack of stealth. Most methods apply perturbations to central regions of the sign, resulting in visually salient patterns that are easily detectable by human observers, thereby limiting their real-world practicality. This study proposes TESP-Attack, a novel stealth-aware adversarial patch method for traffic sign classification. Based on the observation that human visual attention primarily focuses on the central regions of traffic signs, we employ instance segmentation to generate edge-aligned masks that conform to the shape characteristics of the signs. A U-Net generator is utilized to craft adversarial patches, which are then optimized through color and texture constraints along with frequency domain analysis to achieve seamless integration with the background environment, resulting in highly effective visual concealment. The proposed method demonstrates outstanding attack success rates across traffic sign classification models with varied architectures, achieving over 90% under limited query budgets. It also exhibits strong cross-model transferability and maintains robust real-world performance that remains stable under varying angles and distances.
☆ Seeing the Wind from a Falling Leaf NeurIPS 2025
Zhiyuan Gao, Jiageng Mao, Hong-Xing Yu, Haozhe Lou, Emily Yue-Ting Jia, Jernej Barbic, Jiajun Wu, Yue Wang
A longstanding goal in computer vision is to model motions from videos, while the representations behind motions, i.e. the invisible physical interactions that cause objects to deform and move, remain largely unexplored. In this paper, we study how to recover the invisible forces from visual observations, e.g., estimating the wind field by observing a leaf falling to the ground. Our key innovation is an end-to-end differentiable inverse graphics framework, which jointly models object geometry, physical properties, and interactions directly from videos. Through backpropagation, our approach enables the recovery of force representations from object motions. We validate our method on both synthetic and real-world scenarios, and the results demonstrate its ability to infer plausible force fields from videos. Furthermore, we show the potential applications of our approach, including physics-based video generation and editing. We hope our approach sheds light on understanding and modeling the physical process behind pixels, bridging the gap between vision and physics. Please check more video results in our \href{https://chaoren2357.github.io/seeingthewind/}{project page}.
comment : Accepted at NeurIPS 2025
☆ Charts Are Not Images: On the Challenges of Scientific Chart Editing
Shawn Li, Ryan Rossi, Sungchul Kim, Sunav Choudhary, Franck Dernoncourt, Puneet Mathur, Zhengzhong Tu, Yue Zhao
Generative models, such as diffusion and autoregressive approaches, have demonstrated impressive capabilities in editing natural images. However, applying these tools to scientific charts rests on a flawed assumption: a chart is not merely an arrangement of pixels but a visual representation of structured data governed by a graphical grammar. Consequently, chart editing is not a pixel-manipulation task but a structured transformation problem. To address this fundamental mismatch, we introduce \textit{FigEdit}, a large-scale benchmark for scientific figure editing comprising over 30,000 samples. Grounded in real-world data, our benchmark is distinguished by its diversity, covering 10 distinct chart types and a rich vocabulary of complex editing instructions. The benchmark is organized into five distinct and progressively challenging tasks: single edits, multi edits, conversational edits, visual-guidance-based edits, and style transfer. Our evaluation of a range of state-of-the-art models on this benchmark reveals their poor performance on scientific figures, as they consistently fail to handle the underlying structured transformations required for valid edits. Furthermore, our analysis indicates that traditional evaluation metrics (e.g., SSIM, PSNR) have limitations in capturing the semantic correctness of chart edits. Our benchmark demonstrates the profound limitations of pixel-level manipulation and provides a robust foundation for developing and evaluating future structure-aware models. By releasing \textit{FigEdit} (https://github.com/adobe-research/figure-editing), we aim to enable systematic progress in structure-aware figure editing, provide a common ground for fair comparison, and encourage future research on models that understand both the visual and semantic layers of scientific charts.
☆ Probabilistic Modeling of Multi-rater Medical Image Segmentation for Diversity and Personalization
Ke Liu, Shangde Gao, Yichao Fu, Shangqi Gao, Chunhua Shen
Medical image segmentation is inherently influenced by data uncertainty, arising from ambiguous boundaries in medical scans and inter-observer variability in diagnosis. To address this challenge, previous works formulated the multi-rater medical image segmentation task, where multiple experts provide separate annotations for each image. However, existing models are typically constrained to either generate diverse segmentation that lacks expert specificity or to produce personalized outputs that merely replicate individual annotators. We propose Probabilistic modeling of multi-rater medical image Segmentation (ProSeg) that simultaneously enables both diversification and personalization. Specifically, we introduce two latent variables to model expert annotation preferences and image boundary ambiguity. Their conditional probabilistic distributions are then obtained through variational inference, allowing segmentation outputs to be generated by sampling from these distributions. Extensive experiments on both the nasopharyngeal carcinoma dataset (NPC) and the lung nodule dataset (LIDC-IDRI) demonstrate that our ProSeg achieves a new state-of-the-art performance, providing segmentation results that are both diverse and expert-personalized. Code can be found in https://github.com/AI4MOL/ProSeg.
☆ Joint Multi-scale Gated Transformer and Prior-guided Convolutional Network for Learned Image Compression
Zhengxin Chen, Xiaohai He, Tingrong Zhang, Shuhua Xiong, Chao Ren
Recently, learned image compression methods have made remarkable achievements, some of which have outperformed the traditional image codec VVC. The advantages of learned image compression methods over traditional image codecs can be largely attributed to their powerful nonlinear transform coding. Convolutional layers and shifted window transformer (Swin-T) blocks are the basic units of neural networks, and their representation capabilities play an important role in nonlinear transform coding. In this paper, to improve the ability of the vanilla convolution to extract local features, we propose a novel prior-guided convolution (PGConv), where asymmetric convolutions (AConvs) and difference convolutions (DConvs) are introduced to strengthen skeleton elements and extract high-frequency information, respectively. A re-parameterization strategy is also used to reduce the computational complexity of PGConv. Moreover, to improve the ability of the Swin-T block to extract non-local features, we propose a novel multi-scale gated transformer (MGT), where dilated window-based multi-head self-attention blocks with different dilation rates and depth-wise convolution layers with different kernel sizes are used to extract multi-scale features, and a gate mechanism is introduced to enhance non-linearity. Finally, we propose a novel joint Multi-scale Gated Transformer and Prior-guided Convolutional Network (MGTPCN) for learned image compression. Experimental results show that our MGTPCN surpasses state-of-the-art algorithms with a better trade-off between performance and complexity.
☆ Multi-GRPO: Multi-Group Advantage Estimation for Text-to-Image Generation with Tree-Based Trajectories and Multiple Rewards
Qiang Lyu, Zicong Chen, Chongxiao Wang, Haolin Shi, Shibo Gao, Ran Piao, Youwei Zeng, Jianlou Si, Fei Ding, Jing Li, Chun Pong Lau, Weiqiang Wang
Recently, Group Relative Policy Optimization (GRPO) has shown promising potential for aligning text-to-image (T2I) models, yet existing GRPO-based methods suffer from two critical limitations. (1) \textit{Shared credit assignment}: trajectory-level advantages derived from group-normalized sparse terminal rewards are uniformly applied across timesteps, failing to accurately estimate the potential of early denoising steps with vast exploration spaces. (2) \textit{Reward-mixing}: predefined weights for combining multi-objective rewards (e.g., text accuracy, visual quality, text color)--which have mismatched scales and variances--lead to unstable gradients and conflicting updates. To address these issues, we propose \textbf{Multi-GRPO}, a multi-group advantage estimation framework with two orthogonal grouping mechanisms. For better credit assignment, we introduce tree-based trajectories inspired by Monte Carlo Tree Search: branching trajectories at selected early denoising steps naturally forms \emph{temporal groups}, enabling accurate advantage estimation for early steps via descendant leaves while amortizing computation through shared prefixes. For multi-objective optimization, we introduce \emph{reward-based grouping} to compute advantages for each reward function \textit{independently} before aggregation, disentangling conflicting signals. To facilitate evaluation of multiple objective alignment, we curate \textit{OCR-Color-10}, a visual text rendering dataset with explicit color constraints. Across the single-reward \textit{PickScore-25k} and multi-objective \textit{OCR-Color-10} benchmarks, Multi-GRPO achieves superior stability and alignment performance, effectively balancing conflicting objectives. Code will be publicly available at \href{https://github.com/fikry102/Multi-GRPO}{https://github.com/fikry102/Multi-GRPO}.
comment : 20 pages, 15 figures
☆ REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories
Jacob Thompson, Emiliano Garcia-Lopez, Yonatan Bisk
Humans build viewpoint-independent cognitive maps through navigation, enabling intuitive reasoning about object permanence and spatial relations. We argue that multimodal large language models (MLLMs), despite extensive video training, lack this fundamental spatial reasoning capability, a critical limitation for embodied applications. To demonstrate these limitations and drive research, we introduce REM (Reasoning over Embodied Multi-Frame Trajectories), a benchmark using controllable 3D environments for long-horizon embodied spatial reasoning. REM systematically evaluates key aspects like object permanence/distinction, spatial relationships, and numerical tracking across dynamic embodied viewpoints. Our evaluation shows that the best-performing current models exhibit promising overall performance, but become increasingly unreliable at even moderate complexity levels easily handled by humans. These findings highlight challenges MLLMs face in developing robust spatial representations from sequential visual input. Consequently, REM provides targeted metrics and diagnostics to foster improved spatial understanding in future models.
☆ TrajDiff: End-to-end Autonomous Driving without Perception Annotation
Xingtai Gui, Jianbo Zhao, Wencheng Han, Jikai Wang, Jiahao Gong, Feiyang Tan, Cheng-zhong Xu, Jianbing Shen
End-to-end autonomous driving systems directly generate driving policies from raw sensor inputs. While these systems can extract effective environmental features for planning, relying on auxiliary perception tasks, developing perception annotation-free planning paradigms has become increasingly critical due to the high cost of manual perception annotation. In this work, we propose TrajDiff, a Trajectory-oriented BEV Conditioned Diffusion framework that establishes a fully perception annotation-free generative method for end-to-end autonomous driving. TrajDiff requires only raw sensor inputs and future trajectory, constructing Gaussian BEV heatmap targets that inherently capture driving modalities. We design a simple yet effective trajectory-oriented BEV encoder to extract the TrajBEV feature without perceptual supervision. Furthermore, we introduce Trajectory-oriented BEV Diffusion Transformer (TB-DiT), which leverages ego-state information and the predicted TrajBEV features to directly generate diverse yet plausible trajectories, eliminating the need for handcrafted motion priors. Beyond architectural innovations, TrajDiff enables exploration of data scaling benefits in the annotation-free setting. Evaluated on the NAVSIM benchmark, TrajDiff achieves 87.5 PDMS, establishing state-of-the-art performance among all annotation-free methods. With data scaling, it further improves to 88.5 PDMS, which is comparable to advanced perception-based approaches. Our code and model will be made publicly available.
☆ RS-ISRefiner: Towards Better Adapting Vision Foundation Models for Interactive Segmentation of Remote Sensing Images
Deliang Wang, Peng Liu
Interactive image segmentation(IIS) plays a critical role in generating precise annotations for remote sensing imagery, where objects often exhibit scale variations, irregular boundaries and complex backgrounds. However, existing IIS methods, primarily designed for natural images, struggle to generalize to remote sensing domains due to limited annotated data and computational overhead. To address these challenges, we proposed RS-ISRefiner, a novel click-based IIS framework tailored for remote sensing images. The framework employs an adapter-based tuning strategy that preserves the general representations of Vision Foundation Models while enabling efficient learning of remote sensing-specific spatial and boundary characteristics. A hybrid attention mechanism integrating convolutional local modeling with Transformer-based global reasoning enhances robustness against scale diversity and scene complexity. Furthermore, an improved probability map modulation scheme effectively incorporates historical user interactions, yielding more stable iterative refinement and higher boundary fidelity. Comprehensive experiments on six remote sensing datasets, including iSAID, ISPRS Potsdam, SandBar, NWPU, LoveDA Urban and WHUBuilding, demonstrate that RS-ISRefiner consistently outperforms state-of-the-art IIS methods in terms of segmentation accuracy, efficiency and interaction cost. These results confirm the effectiveness and generalizability of our framework, making it highly suitable for high-quality instance segmentation in practical remote sensing scenarios.
☆ Deep Learning-Based Computer Vision Models for Ea rl y Cancer Detection Using Multimodal Medical Imaging and Radiogenomic Integration Frameworks
Emmanuella Avwerosuoghene Oghenekaro
Early cancer detection remains one of the most critical challenges in modern healthcare, where delayed diagnosis significantly reduces survival outcomes. Recent advancements in artificial intelligence, particularly deep learning, have enabled transformative progress in medical imaging analysis. Deep learning-based computer vision models, such as convolutional neural networks (CNNs), transformers, and hybrid attention architectures, can automatically extract complex spatial, morphological, and temporal patterns from multimodal imaging data including MRI, CT, PET, mammography, histopathology, and ultrasound. These models surpass traditional radiological assessment by identifying subtle tissue abnormalities and tumor microenvironment variations invisible to the human eye. At a broader scale, the integration of multimodal imaging with radiogenomics linking quantitative imaging features with genomics, transcriptomics, and epigenetic biomarkers has introduced a new paradigm for personalized oncology. This radiogenomic fusion allows the prediction of tumor genotype, immune response, molecular subtypes, and treatment resistance without invasive biopsies.
☆ Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigation
Chengzhi Yu, Yifan Xu, Yifan Chen, Wenyi Zhang
Recently, large vision-language models (LVLMs) have risen to be a promising approach for multimodal tasks. However, principled hallucination mitigation remains a critical challenge.In this work, we first analyze the data generation process in LVLM hallucination mitigation and affirm that on-policy data significantly outperforms off-policy data, which thus calls for efficient and reliable preference annotation of on-policy data. We then point out that, existing annotation methods introduce additional hallucination in training samples, which may enhance the model's hallucination patterns, to address this problem, we propose training a hallucination classifier giving binary annotations, which guarantee clean chosen samples for the subsequent alignment. To further harness of the power of on-policy data, we design a robust iterative direct preference optimization (DPO) algorithm adopting a dynamic sample reweighting scheme. We conduct comprehensive experiments on three benchmarks with comparison to 8 state-of-the-art baselines. In particular, our approach reduces the hallucination rate of LLaVA-1.5-7B on MMHalBench by 50.8% and the average hallucination rate on Object HalBench by 79.5%; more significantly, our method fully taps into the potential of open-source models, enabling LLaVA-1.5-13B to even surpass the performance of GPT-4V.
☆ CAR-Net: A Cascade Refinement Network for Rotational Motion Deblurring under Angle Information Uncertainty
Ka Chung Lai, Ahmet Cetinkaya
We propose a new neural network architecture called CAR-net (CAscade Refinement Network) to deblur images that are subject to rotational motion blur. Our architecture is specifically designed for the semi-blind scenarios where only noisy information of the rotational motion blur angle is available. The core of our approach is progressive refinement process that starts with an initial deblurred estimate obtained from frequency-domain inversion; A series of refinement stages take the current deblurred image to predict and apply residual correction to the current estimate, progressively suppressing artifacts and restoring fine details. To handle parameter uncertainty, our architecture accommodates an optional angle detection module which can be trained end-to-end with refinement modules. We provide a detailed description of our architecture and illustrate its efficiency through experiments using both synthetic and real-life images. Our code and model as well as the links to the datasets are available at https://github.com/tony123105/CAR-Net
comment : Accepted to AAIML 2026
☆ Affordance-First Decomposition for Continual Learning in Video-Language Understanding
Mengzhu Xu, Hanzhi Liu, Ningkang Peng, Qianyu Chen, Canran Xiao
Continual learning for video--language understanding is increasingly important as models face non-stationary data, domains, and query styles, yet prevailing solutions blur what should stay stable versus what should adapt, rely on static routing/capacity, or require replaying past videos. We aim to explicitly specify where stability lives and where plasticity should be focused under realistic memory and privacy constraints. We introduce Affordance-First Decomposition (AFD): videos are mapped to slowly varying affordance tokens that form a shared, time-aligned substrate, while a lightweight, query-routed, conflict-aware scheduler concentrates adaptation and grows capacity only when needed. The substrate is stabilized via weak alignment and teacher consistency, and training uses question-only replay. AFD achieves state-of-the-art across protocols: 51.6% average accuracy with -1.8% forgetting on domain-incremental VideoQA, ViLCo R@1@0.5 of 29.6% (MQ) and 20.7% (NLQ) with 18.4% stAP@0.25 (VQ), and 39.5% accuracy with -1.6% forgetting on time-incremental iVQA. Overall, AFD offers an explicit, interpretable split between a stable interaction-centered substrate and targeted adaptation.
comment : Under review
☆ Silhouette-based Gait Foundation Model
Dingqiang Ye, Chao Fan, Kartik Narayan, Bingzhe Wu, Chengwen Luo, Jianqiang Li, Vishal M. Patel
Gait patterns play a critical role in human identification and healthcare analytics, yet current progress remains constrained by small, narrowly designed models that fail to scale or generalize. Building a unified gait foundation model requires addressing two longstanding barriers: (a) Scalability. Why have gait models historically failed to follow scaling laws? (b) Generalization. Can one model serve the diverse gait tasks that have traditionally been studied in isolation? We introduce FoundationGait, the first scalable, self-supervised pretraining framework for gait understanding. Its largest version has nearly 0.13 billion parameters and is pretrained on 12 public gait datasets comprising over 2 million walking sequences. Extensive experiments demonstrate that FoundationGait, with or without fine-tuning, performs robustly across a wide spectrum of gait datasets, conditions, tasks (e.g., human identification, scoliosis screening, depression prediction, and attribute estimation), and even input modality. Notably, it achieves 48.0% zero-shot rank-1 accuracy on the challenging in-the-wild Gait3D dataset (1,000 test subjects) and 64.5% on the largest in-the-lab OU-MVLP dataset (5,000+ test subjects), setting a new milestone in robust gait recognition. Coming code and model: https://github.com/ShiqiYu/OpenGait.
☆ Dynamic-eDiTor: Training-Free Text-Driven 4D Scene Editing with Multimodal Diffusion Transformer
Dong In Lee, Hyungjun Doh, Seunggeun Chi, Runlin Duan, Sangpil Kim, Karthik Ramani
Recent progress in 4D representations, such as Dynamic NeRF and 4D Gaussian Splatting (4DGS), has enabled dynamic 4D scene reconstruction. However, text-driven 4D scene editing remains under-explored due to the challenge of ensuring both multi-view and temporal consistency across space and time during editing. Existing studies rely on 2D diffusion models that edit frames independently, often causing motion distortion, geometric drift, and incomplete editing. We introduce Dynamic-eDiTor, a training-free text-driven 4D editing framework leveraging Multimodal Diffusion Transformer (MM-DiT) and 4DGS. This mechanism consists of Spatio-Temporal Sub-Grid Attention (STGA) for locally consistent cross-view and temporal fusion, and Context Token Propagation (CTP) for global propagation via token inheritance and optical-flow-guided token replacement. Together, these components allow Dynamic-eDiTor to perform seamless, globally consistent multi-view video without additional training and directly optimize pre-trained source 4DGS. Extensive experiments on multi-view video dataset DyNeRF demonstrate that our method achieves superior editing fidelity and both multi-view and temporal consistency prior approaches. Project page for results and code: https://di-lee.github.io/dynamic-eDiTor/
comment : 4D Scene Editing
☆ Realistic Handwritten Multi-Digit Writer (MDW) Number Recognition Challenges
Kiri L. Wagstaff
Isolated digit classification has served as a motivating problem for decades of machine learning research. In real settings, numbers often occur as multiple digits, all written by the same person. Examples include ZIP Codes, handwritten check amounts, and appointment times. In this work, we leverage knowledge about the writers of NIST digit images to create more realistic benchmark multi-digit writer (MDW) data sets. As expected, we find that classifiers may perform well on isolated digits yet do poorly on multi-digit number recognition. If we want to solve real number recognition problems, additional advances are needed. The MDW benchmarks come with task-specific performance metrics that go beyond typical error calculations to more closely align with real-world impact. They also create opportunities to develop methods that can leverage task-specific knowledge to improve performance well beyond that of individual digit classification methods.
comment : 10 pages, 6 figures
♻ ☆ TRiCo: Triadic Game-Theoretic Co-Training for Robust Semi-Supervised Learning NeurIPS 2025
Hongyang He, Xinyuan Song, Yangfan He, Zeyu Zhang, Yanshu Li, Haochen You, Lifan Sun, Wenqiao Zhang
We introduce TRiCo, a novel triadic game-theoretic co-training framework that rethinks the structure of semi-supervised learning by incorporating a teacher, two students, and an adversarial generator into a unified training paradigm. Unlike existing co-training or teacher-student approaches, TRiCo formulates SSL as a structured interaction among three roles: (i) two student classifiers trained on frozen, complementary representations, (ii) a meta-learned teacher that adaptively regulates pseudo-label selection and loss balancing via validation-based feedback, and (iii) a non-parametric generator that perturbs embeddings to uncover decision boundary weaknesses. Pseudo-labels are selected based on mutual information rather than confidence, providing a more robust measure of epistemic uncertainty. This triadic interaction is formalized as a Stackelberg game, where the teacher leads strategy optimization and students follow under adversarial perturbations. By addressing key limitations in existing SSL frameworks, such as static view interactions, unreliable pseudo-labels, and lack of hard sample modeling, TRiCo provides a principled and generalizable solution. Extensive experiments on CIFAR-10, SVHN, STL-10, and ImageNet demonstrate that TRiCo consistently achieves state-of-the-art performance in low-label regimes, while remaining architecture-agnostic and compatible with frozen vision backbones.Code:https://github.com/HoHongYeung/NeurIPS25-TRiCo.
comment : Accepted by NeurIPS 2025
♻ ☆ PETAR: Localized Findings Generation with Mask-Aware Vision -Language Modeling for PET Automated Reporting
Danyal Maqbool, Changhee Lee, Zachary Huemann, Samuel D. Church, Matthew E. Larson, Scott B. Perlman, Tomas A. Romero, Joshua D. Warner, Meghan Lubner, Xin Tie, Jameson Merkow, Junjie Hu, Steve Y. Cho, Tyler J. Bradshaw
Generating automated reports for 3D positron emission tomography (PET) is an important and challenging task in medical imaging. PET plays a vital role in oncology, but automating report generation is difficult due to the complexity of whole-body 3D volumes, the wide range of potential clinical findings, and the limited availability of annotated datasets. To address these challenges, we introduce PETARSeg-11K, the first large-scale, publicly available dataset that provides lesion-level correspondence between 3D PET/CT volumes and free-text radiological findings. It comprises 11,356 lesion descriptions paired with 3D segmentations. Second, we propose PETAR-4B, a 3D vision-language model designed for mask-aware, spatially grounded PET/CT reporting. PETAR-4B jointly encodes PET, CT, and 3D lesion segmentation masks, using a 3D focal prompt to capture fine-grained details of lesions that normally comprise less than 0.1% of the volume. Evaluations using automated metrics show PETAR-4B substantially outperforming all 2D and 3D baselines. A human study involving five physicians -- the first of its kind for automated PET reporting -- confirms the model's clinical utility and establishes correlations between automated metrics and expert judgment. This work provides a foundational dataset and a novel architecture, advancing 3D medical vision-language understanding in PET.
♻ ☆ Optimizing against Infeasible Inclusions from Data for Semantic Segmentation through Morphology WACV
Shamik Basu, Luc Van Gool, Christos Sakaridis
State-of-the-art semantic segmentation models are typically optimized in a data-driven fashion, minimizing solely per-pixel or per-segment classification objectives on their training data. This purely data-driven paradigm often leads to absurd segmentations, especially when the domain of input images is shifted from the one encountered during training. For instance, state-of-the-art models may assign the label "road" to a segment that is included by another segment that is respectively labeled as "sky". However, the ground truth of the existing dataset at hand dictates that such inclusion is not feasible. Our method, Infeasible Semantic Inclusions (InSeIn), first extracts explicit inclusion constraints that govern spatial class relations from the semantic segmentation training set at hand in an offline, data-driven fashion, and then enforces a morphological yet differentiable loss that penalizes violations of these constraints during training to promote prediction feasibility. InSeIn is a light-weight plug-and-play method, constitutes a novel step towards minimizing infeasible semantic inclusions in the predictions of learned segmentation models, and yields consistent and significant performance improvements over diverse state-of-the-art networks across the ADE20K, Cityscapes, and ACDC datasets. https://github.com/SHAMIK-97/InSeIn
comment : Published in 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
♻ ☆ DiffProtect: Generate Adversarial Examples with Diffusion Models for Facial Privacy Protection
Jiang Liu, Chun Pong Lau, Zhongliang Guo, Yuxiang Guo, Zhaoyang Wang, Rama Chellappa
The increasingly pervasive facial recognition (FR) systems raise serious concerns about personal privacy, especially for billions of users who have publicly shared their photos on social media. Several attempts have been made to protect individuals from being identified by unauthorized FR systems utilizing adversarial attacks to generate encrypted face images. However, existing methods suffer from poor visual quality or low attack success rates, which limit their utility. Recently, diffusion models have achieved tremendous success in image generation. In this work, we ask: can diffusion models be used to generate adversarial examples to improve both visual quality and attack performance? We propose DiffProtect, which utilizes a diffusion autoencoder to generate semantically meaningful perturbations on FR systems. Extensive experiments demonstrate that DiffProtect produces more natural-looking encrypted images than state-of-the-art methods while achieving significantly higher attack success rates, e.g., 24.5% and 25.1% absolute improvements on the CelebA-HQ and FFHQ datasets.
comment : Code is at https://github.com/joellliu/DiffProtect/
♻ ☆ Blind Inverse Problem Solving Made Easy by Text-to-Image Latent Diffusion
Michail Dontas, Yutong He, Naoki Murata, Yuki Mitsufuji, J. Zico Kolter, Ruslan Salakhutdinov
This paper considers blind inverse image restoration, the task of predicting a target image from a degraded source when the degradation (i.e. the forward operator) is unknown. Existing solutions typically rely on restrictive assumptions such as operator linearity, curated training data or narrow image distributions limiting their practicality. We introduce LADiBI, a training-free method leveraging large-scale text-to-image diffusion to solve diverse blind inverse problems with minimal assumptions. Within a Bayesian framework, LADiBI uses text prompts to jointly encode priors for both target images and operators, unlocking unprecedented flexibility compared to existing methods. Additionally, we propose a novel diffusion posterior sampling algorithm that combines strategic operator initialization with iterative refinement of image and operator parameters, eliminating the need for highly constrained operator forms. Experiments show that LADiBI effectively handles both linear and challenging nonlinear image restoration problems across various image distributions, all without task-specific assumptions or retraining.
♻ ☆ DynaStride: Dynamic Stride Windowing with MM CoT for Instructional Multi-Scene Captioning NeurIPS 7
Eddison Pham, Prisha Priyadarshini, Adrian Maliackel, Kanishk Bandi, Cristian Meo, Kevin Zhu
Scene-level captioning in instructional videos can enhance learning by requiring an understanding of both visual cues and temporal structure. By aligning visual cues with textual guidance, this understanding supports procedural learning and multimodal reasoning, providing a richer context for skill acquisition. However, captions that fail to capture this structure may lack coherence and quality, which can create confusion and undermine the video's educational intent. To address this gap, we introduce DynaStride, a pipeline to generate coherent, scene-level captions without requiring manual scene segmentation. Using the YouCookII dataset's scene annotations, DynaStride performs adaptive frame sampling and multimodal windowing to capture key transitions within each scene. It then employs a multimodal chain-of-thought process to produce multiple action-object pairs, which are refined and fused using a dynamic stride window selection algorithm that adaptively balances temporal context and redundancy. The final scene-level caption integrates visual semantics and temporal reasoning in a single instructional caption. Empirical evaluations against strong baselines, including VLLaMA3 and GPT-4o, demonstrate consistent gains on both N-gram-based metrics (BLEU, METEOR) and semantic similarity measures (BERTScore, CLIPScore). Qualitative analyses further show that DynaStride produces captions that are more temporally coherent and informative, suggesting a promising direction for improving AI-powered instructional content generation.
comment : 16 pages, 15 figures, 5 Tables, Accepted at NeurIPS 7HVU Workshop, Accepted at AAAI AI4ED Workshop
♻ ☆ PRISM: Diversifying Dataset Distillation by Decoupling Architectural Priors
Brian B. Moser, Shalini Sarode, Federico Raue, Stanislav Frolov, Krzysztof Adamkiewicz, Arundhati Shanbhag, Joachim Folz, Tobias C. Nauen, Andreas Dengel
Dataset distillation (DD) promises compact yet faithful synthetic data, but existing approaches often inherit the inductive bias of a single teacher model. As dataset size increases, this bias drives generation toward overly smooth, homogeneous samples, reducing intra-class diversity and limiting generalization. We present PRISM (PRIors from diverse Source Models), a framework that disentangles architectural priors during synthesis. PRISM decouples the logit-matching and regularization objectives, supervising them with different teacher architectures: a primary model for logits and a stochastic subset for batch-normalization (BN) alignment. On ImageNet-1K, PRISM consistently and reproducibly outperforms single-teacher methods (e.g., SRe2L) and recent multi-teacher variants (e.g., G-VBSM) at low- and mid-IPC regimes. The generated data also show significantly richer intra-class diversity, as reflected by a notable drop in cosine similarity between features. We further analyze teacher selection strategies (pre- vs. intra-distillation) and introduce a scalable cross-class batch formation scheme for fast parallel synthesis. Code will be released after the review period.
♻ ☆ HiMo: High-Speed Objects Motion Compensation in Point Clouds
Qingwen Zhang, Ajinkya Khoche, Yi Yang, Li Ling, Sina Sharif Mansouri, Olov Andersson, Patric Jensfelt
LiDAR point cloud is essential for autonomous vehicles, but motion distortions from dynamic objects degrade the data quality. While previous work has considered distortions caused by ego motion, distortions caused by other moving objects remain largely overlooked, leading to errors in object shape and position. This distortion is particularly pronounced in high-speed environments such as highways and in multi-LiDAR configurations, a common setup for heavy vehicles. To address this challenge, we introduce HiMo, a pipeline that repurposes scene flow estimation for non-ego motion compensation, correcting the representation of dynamic objects in point clouds. During the development of HiMo, we observed that existing self-supervised scene flow estimators often produce degenerate or inconsistent estimates under high-speed distortion. We further propose SeFlow++, a real-time scene flow estimator that achieves state-of-the-art performance on both scene flow and motion compensation. Since well-established motion distortion metrics are absent in the literature, we introduce two evaluation metrics: compensation accuracy at a point level and shape similarity of objects. We validate HiMo through extensive experiments on Argoverse 2, ZOD, and a newly collected real-world dataset featuring highway driving and multi-LiDAR-equipped heavy vehicles. Our findings show that HiMo improves the geometric consistency and visual fidelity of dynamic objects in LiDAR point clouds, benefiting downstream tasks such as semantic segmentation and 3D detection. See https://kin-zhang.github.io/HiMo for more details.
comment : 15 pages, 13 figures, Published in Transactions on Robotics (Volume 41)
♻ ☆ Multigranular Evaluation for Brain Visual Decoding AAAI 2026
Weihao Xia, Cengiz Oztireli
Existing evaluation protocols for brain visual decoding predominantly rely on coarse metrics that obscure inter-model differences, lack neuroscientific foundation, and fail to capture fine-grained visual distinctions. To address these limitations, we introduce BASIC, a unified, multigranular evaluation framework that jointly quantifies structural fidelity, inferential alignment, and contextual coherence between decoded and ground-truth images. For the structural level, we introduce a hierarchical suite of segmentation-based metrics, including foreground, semantic, instance, and component masks, anchored in granularity-aware correspondence across mask structures. For the semantic level, we extract structured scene representations encompassing objects, attributes, and relationships using multimodal large language models, enabling detailed, scalable, and context-rich comparisons with ground-truth stimuli. We benchmark a diverse set of visual decoding methods across multiple stimulus-neuroimaging datasets within this unified evaluation framework. Together, these criteria provide a more discriminative, interpretable, and comprehensive foundation for evaluating brain visual decoding methods.
comment : AAAI 2026 (Oral). Code: https://github.com/weihaox/BASIC
♻ ☆ GuideGen: A Text-Guided Framework for Paired Full-torso Anatomy and CT Volume Generation AAAI 2026
Linrui Dai, Rongzhao Zhang, Yongrui Yu, Xiaofan Zhang
The recently emerging conditional diffusion models seem promising for mitigating the labor and expenses in building large 3D medical imaging datasets. However, previous studies on 3D CT generation primarily focus on specific organs characterized by a local structure and fixed contrast and have yet to fully capitalize on the benefits of both semantic and textual conditions. In this paper, we present GuideGen, a controllable framework based on easily-acquired text prompts to generate anatomical masks and corresponding CT volumes for the entire torso-from chest to pelvis. Our approach includes three core components: a text-conditional semantic synthesizer for creating realistic full-torso anatomies; an anatomy-aware high-dynamic-range (HDR) autoencoder for high-fidelity feature extraction across varying intensity levels; and a latent feature generator that ensures alignment between CT images, anatomical semantics and input prompts. Combined, these components enable data synthesis for segmentation tasks from only textual instructions. To train and evaluate GuideGen, we compile a multi-modality cancer imaging dataset with paired CT and clinical descriptions from 12 public TCIA datasets and one private real-world dataset. Comprehensive evaluations across generation quality, cross-modality alignment, and data usability on multi-organ and tumor segmentation tasks demonstrate GuideGen's superiority over existing CT generation methods. Relevant materials are available at https://github.com/OvO1111/GuideGen.
comment : accepted as AAAI 2026 poster
♻ ☆ Towards Balanced Multi-Modal Learning in 3D Human Pose Estimation
Mengshi Qi, Jiaxuan Peng, Xianlin Zhang, Huadong Ma
3D human pose estimation (3D HPE) has emerged as a prominent research topic, particularly in the realm of RGB-based methods. However, the use of RGB images is often limited by issues such as occlusion and privacy constraints. Consequently, multi-modal sensing, which leverages non-intrusive sensors, is gaining increasing attention. Nevertheless, multi-modal 3D HPE still faces challenges, including modality imbalance. In this work, we introduce a novel balanced multi-modal learning method for 3D HPE, which harnesses the power of RGB, LiDAR, mmWave, and WiFi. Specifically, we propose a Shapley value-based contribution algorithm to assess the contribution of each modality and detect modality imbalance. To address this imbalance, we design a modality learning regulation strategy that decelerates the learning process during the early stages of training. We conduct extensive experiments on the widely adopted multi-modal dataset, MM-Fi, demonstrating the superiority of our approach in enhancing 3D pose estimation under complex conditions. We will release our codes soon.
♻ ☆ SRPO: Self-Referential Policy Optimization for Vision -Language-Action Models
Senyu Fei, Siyin Wang, Li Ji, Ao Li, Shiduo Zhang, Liming Liu, Jinlong Hou, Jingjing Gong, Xianzhong Zhao, Xipeng Qiu
Vision-Language-Action (VLA) models excel in robotic manipulation but are constrained by their heavy reliance on expert demonstrations, leading to demonstration bias and limiting performance. Reinforcement learning (RL) is a vital post-training strategy to overcome these limits, yet current VLA-RL methods, including group-based optimization approaches, are crippled by severe reward sparsity. Relying on binary success indicators wastes valuable information in failed trajectories, resulting in low training efficiency. To solve this, we propose Self-Referential Policy Optimization (SRPO), a novel VLA-RL framework. SRPO eliminates the need for external demonstrations or manual reward engineering by leveraging the model's own successful trajectories, generated within the current training batch, as a self-reference. This allows us to assign a progress-wise reward to failed attempts. A core innovation is the use of latent world representations to measure behavioral progress robustly. Instead of relying on raw pixels or requiring domain-specific fine-tuning, we utilize the compressed, transferable encodings from a world model's latent space. These representations naturally capture progress patterns across environments, enabling accurate, generalized trajectory comparison. Empirical evaluations on the LIBERO benchmark demonstrate SRPO's efficiency and effectiveness. Starting from a supervised baseline with 48.9% success, SRPO achieves a new state-of-the-art success rate of 99.2% in just 200 RL steps, representing a 103% relative improvement without any extra supervision. Furthermore, SRPO shows substantial robustness, achieving a 167% performance improvement on the LIBERO-Plus benchmark.
♻ ☆ Vision Language Models are Biased
An Vo, Khai-Nguyen Nguyen, Mohammad Reza Taesiri, Vy Tuong Dang, Anh Totti Nguyen, Daeyoung Kim
Large language models (LLMs) memorize a vast amount of prior knowledge from the Internet that helps them on downstream tasks but also may notoriously sway their outputs towards wrong or biased answers. In this work, we test how the knowledge about popular subjects hurt the accuracy of vision language models (VLMs) on standard, objective visual tasks of counting and identification. We find that state-of-the-art VLMs are strongly biased (e.g., unable to recognize the 4th stripe has been added to a 3-stripe Adidas logo) scoring an average of 17.05% accuracy in counting (e.g., counting stripes in an Adidas-like logo) across 7 diverse domains from animals, logos, chess, board games, optical illusions, to patterned grids. Removing image backgrounds nearly doubles accuracy (21.09 percentage points), revealing that contextual visual cues trigger these biased responses. Further analysis of VLMs' reasoning patterns shows that counting accuracy initially rises with thinking tokens, reaching ~40%, before declining with excessive reasoning. Our work presents an interesting failure mode in VLMs and a human-supervised automated framework for testing VLM biases. Code and data are available at: vlmsarebiased.github.io.
comment : Code and qualitative examples are available at: vlmsarebiased.github.io
♻ ☆ Dynamic Multimodal Prototype Learning in Vision -Language Models ICCV 2025
Xingyu Zhu, Shuo Wang, Beier Zhu, Miaoge Li, Yunfan Li, Junfeng Fang, Zhicai Wang, Dongsheng Wang, Hanwang Zhang
With the increasing attention to pre-trained vision-language models (VLMs), \eg, CLIP, substantial efforts have been devoted to many downstream tasks, especially in test-time adaptation (TTA). However, previous works focus on learning prototypes only in the textual modality while overlooking the ambiguous semantics in class names. These ambiguities lead to textual prototypes that are insufficient to capture visual concepts, resulting in limited performance. To address this issue, we introduce \textbf{ProtoMM}, a training-free framework that constructs multimodal prototypes to adapt VLMs during the test time. By viewing the prototype as a discrete distribution over the textual descriptions and visual particles, ProtoMM has the ability to combine the multimodal features for comprehensive prototype learning. More importantly, the visual particles are dynamically updated as the testing stream flows. This allows our multimodal prototypes to continually learn from the data, enhancing their generalizability in unseen scenarios. In addition, we quantify the importance of the prototypes and test images by formulating their semantic distance as an optimal transport problem. Extensive experiments on 15 zero-shot benchmarks demonstrate the effectiveness of our method, achieving a 1.03\% average accuracy improvement over state-of-the-art methods on ImageNet and its variant datasets.
comment : ICCV 2025
♻ ☆ CrossVid: A Comprehensive Benchmark for Evaluating Cross-Video Reasoning in Multimodal Large Language Models AAAI 2026
Jingyao Li, Jingyun Wang, Molin Tan, Haochen Wang, Cilin Yan, Likun Shi, Jiayin Cai, Xiaolong Jiang, Yao Hu
Cross-Video Reasoning (CVR) presents a significant challenge in video understanding, which requires simultaneous understanding of multiple videos to aggregate and compare information across groups of videos. Most existing video understanding benchmarks focus on single-video analysis, failing to assess the ability of multimodal large language models (MLLMs) to simultaneously reason over various videos. Recent benchmarks evaluate MLLMs' capabilities on multi-view videos that capture different perspectives of the same scene. However, their limited tasks hinder a thorough assessment of MLLMs in diverse real-world CVR scenarios. To this end, we introduce CrossVid, the first benchmark designed to comprehensively evaluate MLLMs' spatial-temporal reasoning ability in cross-video contexts. Firstly, CrossVid encompasses a wide spectrum of hierarchical tasks, comprising four high-level dimensions and ten specific tasks, thereby closely reflecting the complex and varied nature of real-world video understanding. Secondly, CrossVid provides 5,331 videos, along with 9,015 challenging question-answering pairs, spanning single-choice, multiple-choice, and open-ended question formats. Through extensive experiments on various open-source and closed-source MLLMs, we observe that Gemini-2.5-Pro performs best on CrossVid, achieving an average accuracy of 50.4%. Notably, our in-depth case study demonstrates that most current MLLMs struggle with CVR tasks, primarily due to their inability to integrate or compare evidence distributed across multiple videos for reasoning. These insights highlight the potential of CrossVid to guide future advancements in enhancing MLLMs' CVR capabilities.
comment : Accepted to AAAI 2026 (main track). For code and data, see https://github.com/chuntianli666/CrossVid
♻ ☆ 3D Med Diffusion : A 3D Medical Latent Diffusion Model for Controllable and High-quality Medical Image Generation
Haoshen Wang, Zhentao Liu, Kaicong Sun, Xiaodong Wang, Dinggang Shen, Zhiming Cui
The generation of medical images presents significant challenges due to their high-resolution and three-dimensional nature. Existing methods often yield suboptimal performance in generating high-quality 3D medical images, and there is currently no universal generative framework for medical imaging. In this paper, we introduce a 3D Medical Latent Diffusion (3D MedDiffusion) model for controllable, high-quality 3D medical image generation. 3D MedDiffusion incorporates a novel, highly efficient Patch-Volume Autoencoder that compresses medical images into latent space through patch-wise encoding and recovers back into image space through volume-wise decoding. Additionally, we design a new noise estimator to capture both local details and global structural information during diffusion denoising process. 3D MedDiffusion can generate fine-detailed, high-resolution images (up to 512x512x512) and effectively adapt to various downstream tasks as it is trained on large-scale datasets covering CT and MRI modalities and different anatomical regions (from head to leg). Experimental results demonstrate that 3D MedDiffusion surpasses state-of-the-art methods in generative quality and exhibits strong generalizability across tasks such as sparse-view CT reconstruction, fast MRI reconstruction, and data augmentation for segmentation and classification. Source code and checkpoints are available at https://github.com/ShanghaiTech-IMPACT/3D-MedDiffusion.
♻ ☆ Dynamic Prompt Generation for Interactive 3D Medical Image Segmentation Training
Tidiane Camaret Ndir, Alexander Pfefferle, Robin Tibor Schirrmeister
Interactive 3D biomedical image segmentation requires efficient models that can iteratively refine predictions based on user prompts. Current foundation models either lack volumetric awareness or suffer from limited interactive capabilities. We propose a training strategy that combines dynamic volumetric prompt generation with content-aware adaptive cropping to optimize the use of the image encoder. Our method simulates realistic user interaction patterns during training while addressing the computational challenges of learning from sequential refinement feedback on a single GPU. For efficient training, we initialize our network using the publicly available weights from the nnInteractive segmentation model. Evaluation on the \textbf{Foundation Models for Interactive 3D Biomedical Image Segmentation} competition demonstrates strong performance with an average final Dice score of 0.6385, normalized surface distance of 0.6614, and area-under-the-curve metrics of 2.4799 (Dice) and 2.5671 (NSD).
♻ ☆ 4DGT: Learning a 4D Gaussian Transformer Using Real-Wo rl d Monocular Videos NeurIPS 2025
Zhen Xu, Zhengqin Li, Zhao Dong, Xiaowei Zhou, Richard Newcombe, Zhaoyang Lv
We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We proposed a novel density control strategy in training, which enables our 4DGT to handle longer space-time input and remain efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can outperform prior Gaussian-based networks significantly in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos. Project page: https://4dgt.github.io
comment : NeurIPS 2025 (Spotlight); Project Page: https://4dgt.github.io
♻ ☆ Learning to Generate Rigid Body Interactions with Video Diffusion Models
David Romero, Ariana Bermudez, Hao Li, Fabio Pizzati, Ivan Laptev
Recent video generation models have achieved remarkable progress and are now deployed in film, social media production, and advertising. Beyond their creative potential, such models also hold promise as world simulators for robotics and embodied decision making. Despite strong advances, however, current approaches still struggle to generate physically plausible object interactions and lack object-level control mechanisms. To address these limitations, we introduce KineMask, an approach for video generation that enables realistic rigid body control, interactions, and effects. Given a single image and a specified object velocity, our method generates videos with inferred motions and future object interactions. We propose a two-stage training strategy that gradually removes future motion supervision via object masks. Using this strategy we train video diffusion models (VDMs) on synthetic scenes of simple interactions and demonstrate significant improvements of object interactions in real scenes. Furthermore, KineMask integrates low-level motion control with high-level textual conditioning via predicted scene descriptions, leading to support for synthesis of complex dynamical phenomena. Our experiments show that KineMask achieves strong improvements over recent models of comparable size. Ablation studies further highlight the complementary roles of low- and high-level conditioning in VDMs. Our code, model, and data will be made publicly available. Project Page: https://daromog.github.io/KineMask/
♻ ☆ GigaWo rl d-0: Wo rl d Models as Data Engine to Empower Embodied AI
GigaWorld Team, Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Haoyun Li, Jiagang Zhu, Kerui Li, Mengyuan Xu, Qiuping Deng, Siting Wang, Wenkang Qin, Xinze Chen, Xiaofeng Wang, Yankai Wang, Yu Cao, Yifan Chang, Yuan Xu, Yun Ye, Yang Wang, Yukun Zhou, Zhengyuan Zhang, Zhehao Dong, Zheng Zhu
World models are emerging as a foundational paradigm for scalable, data-efficient embodied AI. In this work, we present GigaWorld-0, a unified world model framework designed explicitly as a data engine for Vision-Language-Action (VLA) learning. GigaWorld-0 integrates two synergistic components: GigaWorld-0-Video, which leverages large-scale video generation to produce diverse, texture-rich, and temporally coherent embodied sequences under fine-grained control of appearance, camera viewpoint, and action semantics; and GigaWorld-0-3D, which combines 3D generative modeling, 3D Gaussian Splatting reconstruction, physically differentiable system identification, and executable motion planning to ensure geometric consistency and physical realism. Their joint optimization enables the scalable synthesis of embodied interaction data that is visually compelling, spatially coherent, physically plausible, and instruction-aligned. Training at scale is made feasible through our efficient GigaTrain framework, which exploits FP8-precision and sparse attention to drastically reduce memory and compute requirements. We conduct comprehensive evaluations showing that GigaWorld-0 generates high-quality, diverse, and controllable data across multiple dimensions. Critically, VLA model (e.g., GigaBrain-0) trained on GigaWorld-0-generated data achieve strong real-world performance, significantly improving generalization and task success on physical robots without any real-world interaction during training.
comment : Project Page: https://giga-world-0.github.io/
♻ ☆ Contrastive Forward-Forward: A Training Algorithm of Vision Transformer
Hossein Aghagolzadeh, Mehdi Ezoji
Although backpropagation is widely accepted as a training algorithm for artificial neural networks, researchers are always looking for inspiration from the brain to find ways with potentially better performance. Forward-Forward is a novel training algorithm that is more similar to what occurs in the brain, although there is a significant performance gap compared to backpropagation. In the Forward-Forward algorithm, the loss functions are placed after each layer, and the updating of a layer is done using two local forward passes and one local backward pass. Forward-Forward is in its early stages and has been designed and evaluated on simple multi-layer perceptron networks to solve image classification tasks. In this work, we have extended the use of this algorithm to a more complex and modern network, namely the Vision Transformer. Inspired by insights from contrastive learning, we have attempted to revise this algorithm, leading to the introduction of Contrastive Forward-Forward. Experimental results show that our proposed algorithm performs significantly better than the baseline Forward-Forward leading to an increase of up to 10% in accuracy and accelerating the convergence speed by 5 to 20 times. Furthermore, if we take Cross Entropy as the baseline loss function in backpropagation, it will be demonstrated that the proposed modifications to the baseline Forward-Forward reduce its performance gap compared to backpropagation on Vision Transformer, and even outperforms it in certain conditions, such as inaccurate supervision.
comment : Updated text and added pointer to the published Neural Networks version
♻ ☆ PointNSP: Autoregressive 3D Point Cloud Generation with Next-Scale Level-of-Detail Prediction
Ziqiao Meng, Qichao Wang, Zhiyang Dou, Zixing Song, Zhipeng Zhou, Irwin King, Peilin Zhao
Autoregressive point cloud generation has long lagged behind diffusion-based approaches in quality. The performance gap stems from the fact that autoregressive models impose an artificial ordering on inherently unordered point sets, forcing shape generation to proceed as a sequence of local predictions. This sequential bias emphasizes short-range continuity but undermines the model's capacity to capture long-range dependencies, hindering its ability to enforce global structural properties such as symmetry, consistent topology, and large-scale geometric regularities. Inspired by the level-of-detail (LOD) principle in shape modeling, we propose PointNSP, a coarse-to-fine generative framework that preserves global shape structure at low resolutions and progressively refines fine-grained geometry at higher scales through a next-scale prediction paradigm. This multi-scale factorization aligns the autoregressive objective with the permutation-invariant nature of point sets, enabling rich intra-scale interactions while avoiding brittle fixed orderings. Experiments on ShapeNet show that PointNSP establishes state-of-the-art (SOTA) generation quality for the first time within the autoregressive paradigm. In addition, it surpasses strong diffusion-based baselines in parameter, training, and inference efficiency. Finally, in dense generation with 8,192 points, PointNSP's advantages become even more pronounced, underscoring its scalability potential.
comment : 24 pages; Previously this version appeared as arXiv:2510.05613 which was submitted as a new work by accident
♻ ☆ Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmark s
Jie Li, Hongyi Cai, Mingkang Dong, Muxin Pu, Shan You, Fei Wang, Tao Huang
Automatically detecting abnormal events in videos is crucial for modern autonomous systems, yet existing Video Anomaly Detection (VAD) benchmarks lack the scene diversity, balanced anomaly coverage, and temporal complexity needed to reliably assess real-world performance. Meanwhile, the community is increasingly moving toward Video Anomaly Understanding (VAU), which requires deeper semantic and causal reasoning but remains difficult to benchmark due to the heavy manual annotation effort it demands. In this paper, we introduce Pistachio, a new VAD/VAU benchmark constructed entirely through a controlled, generation-based pipeline. By leveraging recent advances in video generation models, Pistachio provides precise control over scenes, anomaly types, and temporal narratives, effectively eliminating the biases and limitations of Internet-collected datasets. Our pipeline integrates scene-conditioned anomaly assignment, multi-step storyline generation, and a temporally consistent long-form synthesis strategy that produces coherent 41-second videos with minimal human intervention. Extensive experiments demonstrate the scale, diversity, and complexity of Pistachio, revealing new challenges for existing methods and motivating future research on dynamic and multi-event anomaly understanding.
♻ ☆ VoQA: Visual-only Question Answering
Jianing An, Luyang Jiang, Jie Luo, Wenjun Wu, Lei Huang
Visual understanding requires interpreting both natural scenes and the textual information that appears within them, motivating tasks such as Visual Question Answering (VQA). However, current VQA benchmarks overlook scenarios with visually embedded questions, whereas advanced agents should be able to see the question without separate text input as humans. We introduce Visual-only Question Answering (VoQA), where both the scene and the question appear within a single image, requiring models to perceive and reason purely through vision. This setting supports more realistic visual understanding and interaction in scenarios where questions or instructions are embedded directly in the visual scene. Evaluations under pure visual-only zero-shot, prompt-guided and OCR-assisted settings show that current models exhibit a clear performance drop compared to traditional VQA. To address this, we investigate question-alignment fine-tuning strategies designed to guide models toward interpreting the visual question prior to reasoning. Leveraging VoQA dataset together with these strategies yields robust vision-only reasoning while preserving cross-task generalization to traditional VQA, reflecting the complementary visual and textual reasoning capabilities fostered through VoQA training. The code and data are publicly available.
comment : 21 pages
♻ ☆ Towards Efficient and Accurate Spiking Neural Networks via Adaptive Bit Allocation
Xingting Yao, Qinghao Hu, Fei Zhou, Tielong Liu, Gang Li, Peisong Wang, Jian Cheng
Multi-bit spiking neural networks (SNNs) have recently become a heated research spot, pursuing energy-efficient and high-accurate AI. However, with more bits involved, the associated memory and computation demands escalate to the point where the performance improvements become disproportionate. Based on the insight that different layers demonstrate different importance and extra bits could be wasted and interfering, this paper presents an adaptive bit allocation strategy for direct-trained SNNs, achieving fine-grained layer-wise allocation of memory and computation resources. Thus, SNN's efficiency and accuracy can be improved. Specifically, we parametrize the temporal lengths and the bit widths of weights and spikes, and make them learnable and controllable through gradients. To address the challenges caused by changeable bit widths and temporal lengths, we propose the refined spiking neuron, which can handle different temporal lengths, enable the derivation of gradients for temporal lengths, and suit spike quantization better. In addition, we theoretically formulate the step-size mismatch problem of learnable bit widths, which may incur severe quantization errors to SNN, and accordingly propose the step-size renewal mechanism to alleviate this issue. Experiments on various datasets, including the static CIFAR and ImageNet datasets and the dynamic CIFAR-DVS and DVS-GESTURE datasets, demonstrate that our methods can reduce the overall memory and computation cost while achieving higher accuracy. Particularly, our SEWResNet-34 can achieve a 2.69\% accuracy gain and 4.16$\times$ lower bit budgets over the advanced baseline work on ImageNet. This work is open-sourced at \href{https://github.com/Ikarosy/Towards-Efficient-and-Accurate-Spiking-Neural-Networks-via-Adaptive-Bit-Allocation}{this link}.
comment : Neural Networks, In press
♻ ☆ GARF: Learning Generalizable 3D Reassembly for Real-Wo rl d Fractures
Sihang Li, Zeyu Jiang, Grace Chen, Chenyang Xu, Siqi Tan, Xue Wang, Irving Fang, Kristof Zyskowski, Shannon P. McPherron, Radu Iovita, Chen Feng, Jing Zhang
3D reassembly is a challenging spatial intelligence task with broad applications across scientific domains. While large-scale synthetic datasets have fueled promising learning-based approaches, their generalizability to different domains is limited. Critically, it remains uncertain whether models trained on synthetic datasets can generalize to real-world fractures where breakage patterns are more complex. To bridge this gap, we propose GARF, a generalizable 3D reassembly framework for real-world fractures. GARF leverages fracture-aware pretraining to learn fracture features from individual fragments, with flow matching enabling precise 6-DoF alignments. At inference time, we introduce one-step preassembly, improving robustness to unseen objects and varying numbers of fractures. In collaboration with archaeologists, paleoanthropologists, and ornithologists, we curate Fractura, a diverse dataset for vision and learning communities, featuring real-world fracture types across ceramics, bones, eggshells, and lithics. Comprehensive experiments have shown our approach consistently outperforms state-of-the-art methods on both synthetic and real-world datasets, achieving 82.87\% lower rotation error and 25.15\% higher part accuracy. This sheds light on training on synthetic data to advance real-world 3D puzzle solving, demonstrating its strong generalization across unseen object shapes and diverse fracture types. GARF's code, data and demo are available at https://ai4ce.github.io/GARF/.
comment : 18 pages. Project Page https://ai4ce.github.io/GARF/
♻ ☆ When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision -Language-Action Models
Hui Lu, Yi Yu, Yiming Yang, Chenyu Yi, Qixin Zhang, Bingquan Shen, Alex C. Kot, Xudong Jiang
Vision-Language-Action (VLA) models are vulnerable to adversarial attacks, yet universal and transferable attacks remain underexplored, as most existing patches overfit to a single model and fail in black-box settings. To address this gap, we present a systematic study of universal, transferable adversarial patches against VLA-driven robots under unknown architectures, finetuned variants, and sim-to-real shifts. We introduce UPA-RFAS (Universal Patch Attack via Robust Feature, Attention, and Semantics), a unified framework that learns a single physical patch in a shared feature space while promoting cross-model transfer. UPA-RFAS combines (i) a feature-space objective with an $\ell_1$ deviation prior and repulsive InfoNCE loss to induce transferable representation shifts, (ii) a robustness-augmented two-phase min-max procedure where an inner loop learns invisible sample-wise perturbations and an outer loop optimizes the universal patch against this hardened neighborhood, and (iii) two VLA-specific losses: Patch Attention Dominance to hijack text$\to$vision attention and Patch Semantic Misalignment to induce image-text mismatch without labels. Experiments across diverse VLA models, manipulation suites, and physical executions show that UPA-RFAS consistently transfers across models, tasks, and viewpoints, exposing a practical patch-based attack surface and establishing a strong baseline for future defenses.
♻ ☆ MBMamba: When Memory Buffer Meets Mamba for Structure-Aware Image Deblurring
Hu Gao, Xiaoning Lei, Xichen Xu, Depeng Dang, Lizhuang Ma
The Mamba architecture has emerged as a promising alternative to CNNs and Transformers for image deblurring. However, its flatten-and-scan strategy often results in local pixel forgetting and channel redundancy, limiting its ability to effectively aggregate 2D spatial information. Although existing methods mitigate this by modifying the scan strategy or incorporating local feature modules, it increase computational complexity and hinder real-time performance. In this paper, we propose a structure-aware image deblurring network without changing the original Mamba architecture. Specifically, we design a memory buffer mechanism to preserve historical information for later fusion, enabling reliable modeling of relevance between adjacent features. Additionally, we introduce an Ising-inspired regularization loss that simulates the energy minimization of the physical system's "mutual attraction" between pixels, helping to maintain image structure and coherence. Building on this, we develop MBMamba. Experimental results show that our method outperforms state-of-the-art approaches on widely used benchmarks.
♻ ☆ HSR-KAN: Efficient Hyperspectral Image Super-Resolution via Kolmogorov-Arnold Networks
Baisong Li, Xingwang Wang, Haixiao Xu
Hyperspectral images (HSIs) have great potential in various visual tasks due to their rich spectral information. However, obtaining high-resolution hyperspectral images remains challenging due to limitations of physical imaging. Inspired by Kolmogorov-Arnold Networks (KANs), we propose an efficient HSI super-resolution (HSI-SR) model to fuse a low-resolution HSI (LR-HSI) and a high-resolution multispectral image (HR-MSI), yielding a high-resolution HSI (HR-HSI). To achieve the effective integration of spatial information from HR-MSI, we design a fusion module based on KANs, called KAN-Fusion. Further inspired by the channel attention mechanism, we design a spectral channel attention module called KAN Channel Attention Block (KAN-CAB) for post-fusion feature extraction. As a channel attention module integrated with KANs, KAN-CAB not only enhances the fine-grained adjustment ability of deep networks, enabling networks to accurately simulate details of spectral sequences and spatial textures, but also effectively avoid Curse of Dimensionality. Extensive experiments show that, compared to current state-of-the-art HSI-SR methods, proposed HSR-KAN achieves the best performance in terms of both qualitative and quantitative assessments. Our code is available at: https://github.com/Baisonm-Li/HSR-KAN.
♻ ☆ PIF-Net: Ill-Posed Prior Guided Multispectral and Hyperspectral Image Fusion via Invertible Mamba and Fusion-Aware LoRA
Baisong Li, Xingwang Wang, Haixiao Xu
The goal of multispectral and hyperspectral image fusion (MHIF) is to generate high-quality images that simultaneously possess rich spectral information and fine spatial details. However, due to the inherent trade-off between spectral and spatial information and the limited availability of observations, this task is fundamentally ill-posed. Previous studies have not effectively addressed the ill-posed nature caused by data misalignment. To tackle this challenge, we propose a fusion framework named PIF-Net, which explicitly incorporates ill-posed priors to effectively fuse multispectral images and hyperspectral images. To balance global spectral modeling with computational efficiency, we design a method based on an invertible Mamba architecture that maintains information consistency during feature transformation and fusion, ensuring stable gradient flow and process reversibility. Furthermore, we introduce a novel fusion module called the Fusion-Aware Low-Rank Adaptation module, which dynamically calibrates spectral and spatial features while keeping the model lightweight. Extensive experiments on multiple benchmark datasets demonstrate that PIF-Net achieves significantly better image restoration performance than current state-of-the-art methods while maintaining model efficiency.
♻ ☆ DynamicTree: Interactive Real Tree Animation via Sparse Voxel Spectrum
Yaokun Li, Lihe Ding, Xiao Chen, Guang Tan, Tianfan Xue
Generating dynamic and interactive 3D trees has wide applications in virtual reality, games, and world simulation. However, existing methods still face various challenges in generating structurally consistent and realistic 4D motion for complex real trees. In this paper, we propose DynamicTree, the first framework that can generate long-term, interactive 3D motion for 3DGS reconstructions of real trees. Unlike prior optimization-based methods, our approach generates dynamics in a fast feed-forward manner. The key success of our approach is the use of a compact sparse voxel spectrum to represent the tree movement. Given a 3D tree from Gaussian Splatting reconstruction, our pipeline first generates mesh motion using the sparse voxel spectrum and then binds Gaussians to deform the mesh. Additionally, the proposed sparse voxel spectrum can also serve as a basis for fast modal analysis under external forces, allowing real-time interactive responses. To train our model, we also introduce 4DTree, the first large-scale synthetic 4D tree dataset containing 8,786 animated tree meshes with 100-frame motion sequences. Extensive experiments demonstrate that our method achieves realistic and responsive tree animations, significantly outperforming existing approaches in both visual quality and computational efficiency.
comment : Project Page: https://dynamictree-dev.github.io/DynamicTree.github.io/
♻ ☆ Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
Yiming Qin, Bomin Wei, Jiaxin Ge, Konstantinos Kallidromitis, Stephanie Fu, Trevor Darrell, XuDong Wang
Vision-Language Models (VLMs) excel at reasoning in linguistic space but struggle with perceptual understanding that requires dense visual perception, e.g., spatial reasoning and geometric awareness. This limitation stems from the fact that current VLMs have limited mechanisms to capture dense visual information across spatial dimensions. We introduce Chain-of-Visual-Thought (COVT), a framework that enables VLMs to reason not only in words but also through continuous visual tokens-compact latent representations that encode rich perceptual cues. Within a small budget of roughly 20 tokens, COVT distills knowledge from lightweight vision experts, capturing complementary properties such as 2D appearance, 3D geometry, spatial layout, and edge structure. During training, the VLM with COVT autoregressively predicts these visual tokens to reconstruct dense supervision signals (e.g., depth, segmentation, edges, and DINO features). At inference, the model reasons directly in the continuous visual token space, preserving efficiency while optionally decoding dense predictions for interpretability. Evaluated across more than ten diverse perception benchmarks, including CV-Bench, MMVP, RealWorldQA, MMStar, WorldMedQA, and HRBench, integrating COVT into strong VLMs such as Qwen2.5-VL and LLaVA consistently improves performance by 3% to 16% and demonstrates that compact continuous visual thinking enables more precise, grounded, and interpretable multimodal intelligence.
comment : Project page: https://wakalsprojectpage.github.io/covt-website/
♻ ☆ SplatSSC: Decoupled Depth-Guided Gaussian Splatting for Semantic Scene Completion
Rui Qian, Haozhi Cao, Tianchen Deng, Shenghai Yuan, Lihua Xie
Monocular 3D Semantic Scene Completion (SSC) is a challenging yet promising task that aims to infer dense geometric and semantic descriptions of a scene from a single image. While recent object-centric paradigms significantly improve efficiency by leveraging flexible 3D Gaussian primitives, they still rely heavily on a large number of randomly initialized primitives, which inevitably leads to 1) inefficient primitive initialization and 2) outlier primitives that introduce erroneous artifacts. In this paper, we propose SplatSSC, a novel framework that resolves these limitations with a depth-guided initialization strategy and a principled Gaussian aggregator. Instead of random initialization, SplatSSC utilizes a dedicated depth branch composed of a Group-wise Multi-scale Fusion (GMF) module, which integrates multi-scale image and depth features to generate a sparse yet representative set of initial Gaussian primitives. To mitigate noise from outlier primitives, we develop the Decoupled Gaussian Aggregator (DGA), which enhances robustness by decomposing geometric and semantic predictions during the Gaussian-to-voxel splatting process. Complemented with a specialized Probability Scale Loss, our method achieves state-of-the-art performance on the Occ-ScanNet dataset, outperforming prior approaches by over 6.3% in IoU and 4.1% in mIoU, while reducing both latency and memory cost by more than 9.3%.
♻ ☆ Hallo4: High-Fidelity Dynamic Portrait Animation via Direct Preference Optimization
Jiahao Cui, Yan Chen, Mingwang Xu, Hanlin Shang, Yuxuan Chen, Yun Zhan, Zilong Dong, Yao Yao, Jingdong Wang, Siyu Zhu
Generating highly dynamic and photorealistic portrait animations driven by audio and skeletal motion remains challenging due to the need for precise lip synchronization, natural facial expressions, and high-fidelity body motion dynamics. We propose a human-preference-aligned diffusion framework that addresses these challenges through two key innovations. First, we introduce direct preference optimization tailored for human-centric animation, leveraging a curated dataset of human preferences to align generated outputs with perceptual metrics for portrait motion-video alignment and naturalness of expression. Second, the proposed temporal motion modulation resolves spatiotemporal resolution mismatches by reshaping motion conditions into dimensionally aligned latent features through temporal channel redistribution and proportional feature expansion, preserving the fidelity of high-frequency motion details in diffusion-based synthesis. The proposed mechanism is complementary to existing UNet and DiT-based portrait diffusion approaches, and experiments demonstrate obvious improvements in lip-audio synchronization, expression vividness, body motion coherence over baseline methods, alongside notable gains in human preference metrics. Our model and source code can be found at: https://github.com/fudan-generative-vision/hallo4.
♻ ☆ LiNeXt: Revisiting LiDAR Completion with Efficient Non- Diffusion Architectures AAAI 2026
Wenzhe He, Xiaojun Chen, Ruiqi Wang, Ruihui Li, Huilong Pi, Jiapeng Zhang, Zhuo Tang, Kenli Li
3D LiDAR scene completion from point clouds is a fundamental component of perception systems in autonomous vehicles. Previous methods have predominantly employed diffusion models for high-fidelity reconstruction. However, their multi-step iterative sampling incurs significant computational overhead, limiting its real-time applicability. To address this, we propose LiNeXt-a lightweight, non-diffusion network optimized for rapid and accurate point cloud completion. Specifically, LiNeXt first applies the Noise-to-Coarse (N2C) Module to denoise the input noisy point cloud in a single pass, thereby obviating the multi-step iterative sampling of diffusion-based methods. The Refine Module then takes the coarse point cloud and its intermediate features from the N2C Module to perform more precise refinement, further enhancing structural completeness. Furthermore, we observe that LiDAR point clouds exhibit a distance-dependent spatial distribution, being densely sampled at proximal ranges and sparsely sampled at distal ranges. Accordingly, we propose the Distance-aware Selected Repeat strategy to generate a more uniformly distributed noisy point cloud. On the SemanticKITTI dataset, LiNeXt achieves a 199.8x speedup in inference, reduces Chamfer Distance by 50.7%, and uses only 6.1% of the parameters compared with LiDiff. These results demonstrate the superior efficiency and effectiveness of LiNeXt for real-time scene completion.
comment : 18 pages, 13 figures, Accepted to AAAI 2026
♻ ☆ Small Drafts, Big Verdict: Information -Intensive Visual Reasoning via Speculation
Yuhan Liu, Lianhui Qin, Shengjie Wang
Large Vision-Language Models (VLMs) have achieved remarkable progress in multimodal understanding, yet they struggle when reasoning over information-intensive images that densely interleave textual annotations with fine-grained graphical elements. The main challenges lie in precisely localizing critical cues in dense layouts and multi-hop reasoning to integrate dispersed evidence. We propose Speculative Verdict (SV), a training-free framework inspired by speculative decoding that combines multiple lightweight draft experts with a large verdict model. In the draft stage, small VLMs act as draft experts to generate reasoning paths that provide diverse localization candidates; in the verdict stage, a strong VLM synthesizes these paths to produce the final answer, minimizing computational cost while recovering correct answers. To further improve efficiency and accuracy, SV introduces a consensus expert selection mechanism that forwards only high-agreement reasoning paths to the verdict. Empirically, SV achieves consistent gains on challenging information-intensive and high-resolution visual question answering benchmarks, including InfographicVQA, ChartMuseum, ChartQAPro, and HR-Bench 4K. By synthesizing correct insights from multiple partially accurate reasoning paths, SV achieves both error correction and cost-efficiency compared to large proprietary models or training pipelines. Code is available at https://github.com/Tinaliu0123/speculative-verdict.
♻ ☆ A Cross-Hierarchical Difference Feature Fusion Network Based on Multiscale Encoder-Decoder for Hyperspectral Change Detection
Mingshuai Sheng, Bhatti Uzair Aslam, Junfeng Zhang, Siling Feng, Yonis Gulzar
Hyperspectral change detection (HCD) is one of the core applications of remote sensing images, holding significant research value in fields like environmental monitoring and disaster assessment. However, existing methods often suffer from incomplete capture of multiscale spatial-spectral features and insufficient fusion of differential feature information. To address these challenges, this paper proposes a Cross-Hierarchical Differential Feature Fusion Network (CHDFFN) based on a multiscale encoder-decoder. Firstly, a multiscale feature extraction subnetwork is designed, taking the customized encoder-decoder as the backbone, combined with residual connections and the proposed dual-core channel-spatial attention module to achieve multi-level extraction and initial integration of spatial-spectral features. The encoder embeds convolutional blocks with different receptive field sizes to capture multiscale representations from shallow details to deep semantics. The decoder fuses the encoder's output via skip connections to gradually restore spatial resolution while suppressing background noise and redundancy. To enhance the model's ability to capture differential features between bi-temporal hyperspectral images, a spatial-spectral change feature learning module is designed to learn hierarchical change representations. Additionally, an adaptive high-level feature fusion module is proposed, dynamically balancing the contribution of hierarchical differential features by adaptively assigning weights, which effectively strengthens the model's capability to characterize complex change patterns. Finally, experiments on four public hyperspectral datasets show that compared with some state-of-the-art methods, the average maximum improvements of OA, KC, and F1 are 4.61%, 19.79%, and 18.90% respectively, verifying the model's effectiveness.
♻ ☆ MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models NeurIPS 2025
Hengzhi Li, Megan Tjandrasuwita, Yi R. Fung, Armando Solar-Lezama, Paul Pu Liang
As AI becomes more closely integrated with peoples' daily activities, socially intelligent AI that can understand and interact seamlessly with humans in daily lives is increasingly important. However, current works in AI social reasoning all rely on language-only or language-dominant approaches to benchmark and training models, resulting in systems that are improving in verbal communication but struggle with nonverbal social understanding. To address this limitation, we tap into a novel data source rich in nonverbal social interactions -- mime videos. Mimes refer to the art of expression through gesture and movement without spoken words, which presents unique challenges and opportunities in interpreting nonverbal social communication. We contribute a new dataset called MimeQA, obtained by sourcing ~8 hours of videos clips from YouTube and developing a comprehensive video question-answering benchmark comprising 806 carefully annotated and verified question-answer pairs, designed to probe nonverbal social reasoning capabilities. Using MimeQA, we evaluate state-of-the-art video large language models (VideoLLMs) and find that they achieve low accuracy, generally ranging from 20-30%, while humans score 86%. Our analysis reveals that VideoLLMs often fail to ground imagined objects and over-rely on the text prompt while ignoring subtle nonverbal interactions. We hope to inspire future work in AI models that embody true social intelligence capable of interpreting non-verbal human interactions.
comment : NeurIPS 2025 Datasets and Benchmarks
♻ ☆ Rendering-Aware Reinforcement Learning for Vector Graphics Generation
Juan A. Rodriguez, Haotian Zhang, Abhay Puri, Aarash Feizi, Rishav Pramanik, Pascal Wichmann, Arnab Mondal, Mohammad Reza Samsami, Rabiul Awal, Perouz Taslakian, Spandana Gella, Sai Rajeswar, David Vazquez, Christopher Pal, Marco Pedersoli
Scalable Vector Graphics (SVG) offer a powerful format for representing visual designs as interpretable code. Recent advances in vision-language models (VLMs) have enabled high-quality SVG generation by framing the problem as a code generation task and leveraging large-scale pretraining. VLMs are particularly suitable for this task as they capture both global semantics and fine-grained visual patterns, while transferring knowledge across vision, natural language, and code domains. However, existing VLM approaches often struggle to produce faithful and efficient SVGs because they never observe the rendered images during training. Although differentiable rendering for autoregressive SVG code generation remains unavailable, rendered outputs can still be compared to original inputs, enabling evaluative feedback suitable for reinforcement learning (RL). We introduce RLRF (Reinforcement Learning from Rendering Feedback), an RL method that enhances SVG generation in autoregressive VLMs by leveraging feedback from rendered SVG outputs. Given an input image, the model generates SVG roll-outs that are rendered and compared to the original image to compute a reward. This visual fidelity feedback guides the model toward producing more accurate, efficient, and semantically coherent SVGs. RLRF significantly outperforms supervised fine-tuning, addressing common failure modes and enabling precise, high-quality SVG generation with strong structural understanding and generalization.
♻ ☆ Interpreting ResNet-based CLIP via Neuron-Attention Decomposition NeurIPS 2025
Edmund Bu, Yossi Gandelsman
We present a novel technique for interpreting the neurons in CLIP-ResNet by decomposing their contributions to the output into individual computation paths. More specifically, we analyze all pairwise combinations of neurons and the following attention heads of CLIP's attention-pooling layer. We find that these neuron-head pairs can be approximated by a single direction in CLIP-ResNet's image-text embedding space. Leveraging this insight, we interpret each neuron-head pair by associating it with text. Additionally, we find that only a sparse set of the neuron-head pairs have a significant contribution to the output value, and that some neuron-head pairs, while polysemantic, represent sub-concepts of their corresponding neurons. We use these observations for two applications. First, we employ the pairs for training-free semantic segmentation, outperforming previous methods for CLIP-ResNet. Second, we utilize the contributions of neuron-head pairs to monitor dataset distribution shifts. Our results demonstrate that examining individual computation paths in neural networks uncovers interpretable units, and that such units can be utilized for downstream tasks.
comment : Accepted at NeurIPS 2025 Workshop on Mechanistic Interpretability. Project page: https://edmundbu.github.io/clip-neur-attn/
Computation and Language 52
☆ Mode-Conditioning Unlocks Superior Test-Time Scaling
Chen Henry Wu, Sachin Goyal, Aditi Raghunathan
Parallel sampling promises substantial gains in test-time scaling, but its effectiveness is sharply limited by diversity collapse, where models concentrate on a few modes and repeated samples produce the same mistakes. We propose the mode-conditioning (ModC) framework, which explicitly allocates test-time compute across reasoning modes using either specialist models or mode-specific prefixes. ModC consistently improves scaling across controlled graph-search tasks and large-scale reasoning benchmarks, spanning model families and sizes from 0.5B to 7B. On OpenThoughts, fine-tuning Qwen2.5-7B with ModC achieves a 4x efficiency gain over standard training while also improving the maximum attainable Pass@k. We further show that gradient clustering enables ModC without explicit mode labels, yielding up to 10% gains on datasets such as NuminaMath. Finally, we show that ModC improves reinforcement learning (RL) and can further boost diversity-inducing RL methods. These results demonstrate that standard training underutilizes the diversity in data, and that ModC provides a simple, effective remedy for unlocking the full benefits of diversity in test-time scaling.
☆ How do we measure privacy in text? A survey of text anonymization metrics AACL
Yaxuan Ren, Krithika Ramesh, Yaxing Yao, Anjalie Field
In this work, we aim to clarify and reconcile metrics for evaluating privacy protection in text through a systematic survey. Although text anonymization is essential for enabling NLP research and model development in domains with sensitive data, evaluating whether anonymization methods sufficiently protect privacy remains an open challenge. In manually reviewing 47 papers that report privacy metrics, we identify and compare six distinct privacy notions, and analyze how the associated metrics capture different aspects of privacy risk. We then assess how well these notions align with legal privacy standards (HIPAA and GDPR), as well as user-centered expectations grounded in HCI studies. Our analysis offers practical guidance on navigating the landscape of privacy evaluation approaches further and highlights gaps in current practices. Ultimately, we aim to facilitate more robust, comparable, and legally aware privacy evaluations in text anonymization.
comment : 13 pages, 1 figure, 1 table. To be published in Findings of the Association for Computational Linguistics (AACL-IJCNLP 2025). Related resources at: https://github.com/ryxGuo/privacy-metrics-survey
☆ Generalized Medical Phrase Grounding
Wenjun Zhang, Shekhar S. Chandra, Aaron Nicolson
Medical phrase grounding (MPG) maps textual descriptions of radiological findings to corresponding image regions. These grounded reports are easier to interpret, especially for non-experts. Existing MPG systems mostly follow the referring expression comprehension (REC) paradigm and return exactly one bounding box per phrase. Real reports often violate this assumption. They contain multi-region findings, non-diagnostic text, and non-groundable phrases, such as negations or descriptions of normal anatomy. Motivated by this, we reformulate the task as generalised medical phrase grounding (GMPG), where each sentence is mapped to zero, one, or multiple scored regions. To realise this formulation, we introduce the first GMPG model: MedGrounder. We adopted a two-stage training regime: pre-training on report sentence--anatomy box alignment datasets and fine-tuning on report sentence--human annotated box datasets. Experiments on PadChest-GR and MS-CXR show that MedGrounder achieves strong zero-shot transfer and outperforms REC-style and grounded report generation baselines on multi-region and non-groundable phrases, while using far fewer human box annotations. Finally, we show that MedGrounder can be composed with existing report generators to produce grounded reports without retraining the generator.
☆ Testing the Machine Consciousness Hypothesis
Stephen Fitz
The Machine Consciousness Hypothesis states that consciousness is a substrate-free functional property of computational systems capable of second-order perception. I propose a research program to investigate this idea in silico by studying how collective self-models (coherent, self-referential representations) emerge from distributed learning systems embedded within universal self-organizing environments. The theory outlined here starts from the supposition that consciousness is an emergent property of collective intelligence systems undergoing synchronization of prediction through communication. It is not an epiphenomenon of individual modeling but a property of the language that a system evolves to internally describe itself. For a model of base reality, I begin with a minimal but general computational world: a cellular automaton, which exhibits both computational irreducibility and local reducibility. On top of this computational substrate, I introduce a network of local, predictive, representational (neural) models capable of communication and adaptation. I use this layered model to study how collective intelligence gives rise to self-representation as a direct consequence of inter-agent alignment. I suggest that consciousness does not emerge from modeling per se, but from communication. It arises from the noisy, lossy exchange of predictive messages between groups of local observers describing persistent patterns in the underlying computational substrate (base reality). It is through this representational dialogue that a shared model arises, aligning many partial views of the world. The broader goal is to develop empirically testable theories of machine consciousness, by studying how internal self-models may form in distributed systems without centralized control.
☆ ELR-1000: A Community-Generated Dataset for Endangered Indic Indigenous Languages AACL 2025
Neha Joshi, Pamir Gogoi, Aasim Mirza, Aayush Jansari, Aditya Yadavalli, Ayushi Pandey, Arunima Shukla, Deepthi Sudharsan, Kalika Bali, Vivek Seshadri
We present a culturally-grounded multimodal dataset of 1,060 traditional recipes crowdsourced from rural communities across remote regions of Eastern India, spanning 10 endangered languages. These recipes, rich in linguistic and cultural nuance, were collected using a mobile interface designed for contributors with low digital literacy. Endangered Language Recipes (ELR)-1000 -- captures not only culinary practices but also the socio-cultural context embedded in indigenous food traditions. We evaluate the performance of several state-of-the-art large language models (LLMs) on translating these recipes into English and find the following: despite the models' capabilities, they struggle with low-resource, culturally-specific language. However, we observe that providing targeted context -- including background information about the languages, translation examples, and guidelines for cultural preservation -- leads to significant improvements in translation quality. Our results underscore the need for benchmarks that cater to underrepresented languages and domains to advance equitable and culturally-aware language technologies. As part of this work, we release the ELR-1000 dataset to the NLP community, hoping it motivates the development of language technologies for endangered languages.
comment : Accepted at AACL 2025 (Main)
☆ When Safety Blocks Sense: Measuring Semantic Confusion in LLM Refusals
Riad Ahmed Anonto, Md Labid Al Nahiyan, Md Tanvir Hassan, Ch. Md. Rakin Haider
Safety-aligned language models often refuse prompts that are actually harmless. Current evaluations mostly report global rates such as false rejection or compliance. These scores treat each prompt alone and miss local inconsistency, where a model accepts one phrasing of an intent but rejects a close paraphrase. This gap limits diagnosis and tuning. We introduce "semantic confusion," a failure mode that captures such local inconsistency, and a framework to measure it. We build ParaGuard, a 10k-prompt corpus of controlled paraphrase clusters that hold intent fixed while varying surface form. We then propose three model-agnostic metrics at the token level: Confusion Index, Confusion Rate, and Confusion Depth. These metrics compare each refusal to its nearest accepted neighbors and use token embeddings, next-token probabilities, and perplexity signals. Experiments across diverse model families and deployment guards show that global false-rejection rate hides critical structure. Our metrics reveal globally unstable boundaries in some settings, localized pockets of inconsistency in others, and cases where stricter refusal does not increase inconsistency. We also show how confusion-aware auditing separates how often a system refuses from how sensibly it refuses. This gives developers a practical signal to reduce false refusals while preserving safety.
☆ Associative Syntax and Maximal Repetitions reveal context -dependent complexity in fruit bat communication NeurIPS 2025
Luigi Assom
This study presents an unsupervised method to infer discreteness, syntax and temporal structures of fruit-bats vocalizations, as a case study of graded vocal systems, and evaluates the complexity of communication patterns in relation with behavioral context. The method improved the baseline for unsupervised labeling of vocal units (i.e. syllables) through manifold learning, by investigating how dimen- sionality reduction on mel-spectrograms affects labeling, and comparing it with unsupervised labels based on acoustic similarity. We then encoded vocalizations as syllabic sequences to analyze the type of syntax, and extracted the Maximal Repetitions (MRs) to evaluate syntactical structures. We found evidence for: i) associative syntax, rather than combinatorial (context classification is unaffected by permutation of sequences, F 1 > 0.9); ii) context-dependent use of syllables (Wilcoxon rank-sum tests, p-value < 0.05); iii) heavy-tail distribution of MRs (truncated power-law, exponent α < 2), indicative of mechanism encoding com- binatorial complexity. Analysis of MRs and syllabic transition networks revealed that mother-pupil interactions were characterized by repetitions, while commu- nication in conflict-contexts exhibited higher complexity (longer MRs and more interconnected vocal sequences) than non-agonistic contexts. We propose that communicative complexity is higher in scenarios of disagreement, reflecting lower compressibility of information.
comment : Accepted for a lightning talk at the NeurIPS 2025 Workshop: "AI for Non-Human Animal Communication"
☆ Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics
Jinu Lee, Kyoung-Woon On, Simeng Han, Arman Cohan, Julia Hockenmaier
Evaluating the quality of LLM-generated reasoning traces in expert domains (e.g., law) is essential for ensuring credibility and explainability, yet remains challenging due to the inherent complexity of such reasoning tasks. We introduce LEGIT (LEGal Issue Trees), a novel large-scale (24K instances) expert-level legal reasoning dataset with an emphasis on reasoning trace evaluation. We convert court judgments into hierarchical trees of opposing parties' arguments and the court's conclusions, which serve as rubrics for evaluating the issue coverage and correctness of the reasoning traces. We verify the reliability of these rubrics via human expert annotations and comparison with coarse, less informative rubrics. Using the LEGIT dataset, we show that (1) LLMs' legal reasoning ability is seriously affected by both legal issue coverage and correctness, and that (2) retrieval-augmented generation (RAG) and RL with rubrics bring complementary benefits for legal reasoning abilities, where RAG improves overall reasoning capability, whereas RL improves correctness albeit with reduced coverage.
☆ Advancing Academic Chatbots: Evaluation of Non Traditional Outputs
Nicole Favero, Francesca Salute, Daniel Hardt
Most evaluations of large language models focus on standard tasks such as factual question answering or short summarization. This research expands that scope in two directions: first, by comparing two retrieval strategies, Graph RAG, structured knowledge-graph based, and Advanced RAG, hybrid keyword-semantic search, for QA; and second, by evaluating whether LLMs can generate high quality non-traditional academic outputs, specifically slide decks and podcast scripts. We implemented a prototype combining Meta's LLaMA 3 70B open weight and OpenAI's GPT 4o mini API based. QA performance was evaluated using both human ratings across eleven quality dimensions and large language model judges for scalable cross validation. GPT 4o mini with Advanced RAG produced the most accurate responses. Graph RAG offered limited improvements and led to more hallucinations, partly due to its structural complexity and manual setup. Slide and podcast generation was tested with document grounded retrieval. GPT 4o mini again performed best, though LLaMA 3 showed promise in narrative coherence. Human reviewers were crucial for detecting layout and stylistic flaws, highlighting the need for combined human LLM evaluation in assessing emerging academic outputs.
☆ Dr.Mi-Bench: A Modular-integrated Benchmark for Scientific Deep Research Agent
Zhihan Guo, Feiyang Xu, Yifan Li, Muzhi Li, Shuai Zou, Jiele Wu, Han Shi, Haoli Bai, Ho-fung Leung, Irwin King
The explosive growth in academic literature necessitates automated deep research (DR) agents, yet their evaluation remains a significant challenge. First, existing benchmarks often focus narrowly on retrieval while neglecting high-level planning and reasoning. Second, existing benchmarks favor general domains over the scientific domains that are the core application for DR agents. To address these gaps, we introduce Dr.Mi-Bench, a Modular-integrated benchmark for scientific DR agents. Grounded in academic literature, our benchmark uses a human-annotated dataset of 200 instances across 10 scientific domains, including both research and review papers. Besides, we also propose a Modular-integrated Evaluation Paradigm for DR Agents (Dr.Mi-Eval), a novel modular-integrated evaluation paradigm, which leverages the rich structure of academic papers to assess the core competencies of planning, retrieval, and reasoning through two complementary modes: an end-to-end evaluation for DR agents and an isolated evaluation for foundational LLMs as potential backbones. Experimental results reveal a fragmented performance landscape: agents exhibit specialized strengths but share critical weaknesses, most notably in performing the multi-source retrieval required for review-style tasks and performing consistently across diverse scientific fields. Moreover, improving high-level planning capability is the crucial factor for unlocking the reasoning potential of foundational LLMs as backbones. By exposing these actionable failure modes, Dr.Mi-Bench provides a diagnostic tool to guide the development of more reliable academic research assistants.
☆ Table as a Modality for Large Language Models NeurIPS 2025
Liyao Li, Chao Ye, Wentao Ye, Yifei Sun, Zhe Jiang, Haobo Wang, Jiaming Tian, Yiming Zhang, Ningtao Wang, Xing Fu, Gang Chen, Junbo Zhao
To migrate the remarkable successes of Large Language Models (LLMs), the community has made numerous efforts to generalize them to the table reasoning tasks for the widely deployed tabular data. Despite that, in this work, by showing a probing experiment on our proposed StructQA benchmark, we postulate that even the most advanced LLMs (such as GPTs) may still fall short of coping with tabular data. More specifically, the current scheme often simply relies on serializing the tabular data, together with the meta information, then inputting them through the LLMs. We argue that the loss of structural information is the root of this shortcoming. In this work, we further propose TAMO, which bears an ideology to treat the tables as an independent modality integrated with the text tokens. The resulting model in TAMO is a multimodal framework consisting of a hypergraph neural network as the global table encoder seamlessly integrated with the mainstream LLM. Empirical results on various benchmarking datasets, including HiTab, WikiTQ, WikiSQL, FeTaQA, and StructQA, have demonstrated significant improvements on generalization with an average relative gain of 42.65%.
comment : Accepted to NeurIPS 2025
☆ Fine-tuning of lightweight large language models for sentiment classification on heterogeneous financial textual data
Alvaro Paredes Amorin, Andre Python, Christoph Weisser
Large language models (LLMs) play an increasingly important role in finan- cial markets analysis by capturing signals from complex and heterogeneous textual data sources, such as tweets, news articles, reports, and microblogs. However, their performance is dependent on large computational resources and proprietary datasets, which are costly, restricted, and therefore inacces- sible to many researchers and practitioners. To reflect realistic situations we investigate the ability of lightweight open-source LLMs - smaller and publicly available models designed to operate with limited computational resources - to generalize sentiment understanding from financial datasets of varying sizes, sources, formats, and languages. We compare the benchmark finance natural language processing (NLP) model, FinBERT, and three open-source lightweight LLMs, DeepSeek-LLM 7B, Llama3 8B Instruct, and Qwen3 8B on five publicly available datasets: FinancialPhraseBank, Financial Question Answering, Gold News Sentiment, Twitter Sentiment and Chinese Finance Sentiment. We find that LLMs, specially Qwen3 8B and Llama3 8B, perform best in most scenarios, even from using only 5% of the available training data. These results hold in zero-shot and few-shot learning scenarios. Our findings indicate that lightweight, open-source large language models (LLMs) consti- tute a cost-effective option, as they can achieve competitive performance on heterogeneous textual data even when trained on only a limited subset of the extensive annotated corpora that are typically deemed necessary.
☆ DeformAr: Rethinking NER Evaluation through Component Analysis and Visual Analytics
Ahmed Mustafa Younes
Transformer models have significantly advanced Natural Language Processing (NLP), demonstrating strong performance in English. However, their effectiveness in Arabic, particularly for Named Entity Recognition (NER), remains limited, even with larger pre-trained models. This performance gap stems from multiple factors, including tokenisation, dataset quality, and annotation inconsistencies. Existing studies often analyze these issues in isolation, failing to capture their joint effect on system behaviour and performance. We introduce DeformAr (Debugging and Evaluation Framework for Transformer-based NER Systems), a novel framework designed to investigate and explain the performance discrepancy between Arabic and English NER systems. DeformAr integrates a data extraction library and an interactive dashboard, supporting two modes of evaluation: cross-component analysis and behavioural analysis. The framework divides each language into dataset and model components to examine their interactions. The analysis proceeds in two stages. First, cross-component analysis provides systematic diagnostic measures across data and model subcomponents, addressing the "what," "how," and "why" behind observed discrepancies. The second stage applies behavioural analysis by combining interpretability techniques with token-level metrics, interactive visualisations, and representation space analysis. These stages enable a component-aware diagnostic process that detects model behaviours and explains them by linking them to underlying representational patterns and data factors. DeformAr is the first Arabic-specific, component-based interpretability tool, offering a crucial resource for advancing model analysis in under-resourced languages.
comment : PhD Thesis, University of Sussex, 2025. 311 pages, 140 figures, 32 tables. Submitted as a PDF-only. First supervisor: Julie Weeds. Second supervisor: David Weir
☆ Mitigating Hallucinations in Zero-Shot Scientific Summarisation: A Pilot Study
Imane Jaaouine, Ross D. King
Large language models (LLMs) produce context inconsistency hallucinations, which are LLM generated outputs that are misaligned with the user prompt. This research project investigates whether prompt engineering (PE) methods can mitigate context inconsistency hallucinations in zero-shot LLM summarisation of scientific texts, where zero-shot indicates that the LLM relies purely on its pre-training data. Across eight yeast biotechnology research paper abstracts, six instruction-tuned LLMs were prompted with seven methods: a base- line prompt, two levels of increasing instruction complexity (PE-1 and PE-2), two levels of context repetition (CR-K1 and CR-K2), and two levels of random addition (RA-K1 and RA-K2). Context repetition involved the identification and repetition of K key sentences from the abstract, whereas random addition involved the repetition of K randomly selected sentences from the abstract, where K is 1 or 2. A total of 336 LLM-generated summaries were evaluated using six metrics: ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, METEOR, and cosine similarity, which were used to compute the lexical and semantic alignment be- tween the summaries and the abstracts. Four hypotheses on the effects of prompt methods on summary alignment with the reference text were tested. Statistical analysis on 3744 collected datapoints was performed using bias-corrected and accelerated (BCa) bootstrap confidence intervals and Wilcoxon signed-rank tests with Bonferroni-Holm correction. The results demonstrated that CR and RA significantly improve the lexical alignment of LLM-generated summaries with the abstracts. These findings indicate that prompt engineering has the potential to impact hallucinations in zero-shot scientific summarisation tasks.
☆ Reward Auditor: Inference on Reward Model ing Suitability in Real-Wo rl d Perturbed Scenarios
Jianxiang Zang, Yongda Wei, Ruxue Bai, Shiyu Jiang, Nijia Mo, Binhong Li, Qiang Sun, Hui Liu
Reliable reward models (RMs) are critical for ensuring the safe alignment of large language models (LLMs). However, current evaluation methods focus solely on preference perception accuracies in given specific scenarios, obscuring the critical vulnerabilities of RMs in real-world scenarios. We identify the true challenge lies in assessing a novel dimension: Suitability, defined as conditional reliability under specific real-world perturbations. To this end, we introduce Reward Auditor, a hypothesis-testing framework specifically designed for RM suitability inference. Rather than answering "How accurate is the RM's preference perception for given samples?", it employs scientific auditing to answer: "Can we infer RMs exhibit systematic vulnerabilities in specific real-world scenarios?". Under real-world perturbed scenarios, Reward Auditor quantifies statistical significance and effect size by auditing distribution degradation of RM preference perception confidence. This enables inference of both the certainty and severity of RM vulnerabilities across diverse real-world scenarios. This lays a solid foundation for building next-generation LLM alignment systems that are verifiably safe, more robust, and trustworthy.
☆ Towards Active Synthetic Data Generation for Finetuning Language Models
Samuel Kessler, Menglin Xia, Daniel Madrigal Diaz, Dongge Han, Helia Heshemi, Saravan Rajmohan, Victor Ruehle, Jordan T. Ash
A common and effective means for improving language model capabilities involves finetuning a ``student'' language model's parameters on generations from a more proficient ``teacher'' model. Termed ``synthetic data'', these generations are often produced before any student finetuning, but some work has considered generating new synthetic samples as training progresses. This paper studies and advocates for the latter case, where data are generated in an iterative, closed-loop fashion that is guided by the current state of the student model. For a fixed budget of generated samples, or a budget in terms of compute spent querying a teacher, we show that this curation of finetuning data affords improved student performance over static generation. Further, while there have been several LLM-specific methods proposed that operate in this regime, we find that simple, inexpensive selection criteria from the active learning literature tend to be most performant. We validate these claims across four mathematical and logical reasoning datasets using four different small language models.
comment : 14 figures, 36 pages
☆ Less is More: Resource-Efficient Low-Rank Adaptation
Chunlin Tian, Xuyang Wei, Huanrong Liu, Zhijiang Guo, Li Li
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), but it still incurs notable overhead and suffers from parameter interference in complex datasets. While re- cent works decouple LoRA update matrices to exploit matrix-wise asymmetry, training costs remain high. We revisit LoRA from the perspective of inter-matrix and intra-layer parameter redundancy and propose Resource-Efficient Low-Rank Adaptation, EffiLoRA, a lightweight and generalizable approach for language, multimodal, and diffusion models. EffiLoRA employs a unified A matrix across all transformer layers and introduces a runtime selective B matrices up- date to dynamically trade-off the system resource budget and model performance. EffiLoRA consistently outperforms LoRA across diverse modalities, including commonsense reasoning, visual instruction tuning, and image generation, demon- strating improved efficiency and robustness.
comment : 18 pages, 7 figures
☆ One Swallow Does Not Make a Summer: Understanding Semantic Structures in Embedding Spaces
Yandong Sun, Qiang Huang, Ziwei Xu, Yiqun Sun, Yixuan Tang, Anthony K. H. Tung
Embedding spaces are fundamental to modern AI, translating raw data into high-dimensional vectors that encode rich semantic relationships. Yet, their internal structures remain opaque, with existing approaches often sacrificing semantic coherence for structural regularity or incurring high computational overhead to improve interpretability. To address these challenges, we introduce the Semantic Field Subspace (SFS), a geometry-preserving, context-aware representation that captures local semantic neighborhoods within the embedding space. We also propose SAFARI (SemAntic Field subspAce deteRmInation), an unsupervised, modality-agnostic algorithm that uncovers hierarchical semantic structures using a novel metric called Semantic Shift, which quantifies how semantics evolve as SFSes evolve. To ensure scalability, we develop an efficient approximation of Semantic Shift that replaces costly SVD computations, achieving a 15~30x speedup with average errors below 0.01. Extensive evaluations across six real-world text and image datasets show that SFSes outperform standard classifiers not only in classification but also in nuanced tasks such as political bias detection, while SAFARI consistently reveals interpretable and generalizable semantic hierarchies. This work presents a unified framework for structuring, analyzing, and scaling semantic understanding in embedding spaces.
☆ WaterSearch: A Quality-Aware Search-based Watermarking Framework for Large Language Models
Yukang Lin, Jiahao Shao, Shuoran Jiang, Wentao Zhu, Bingjie Lu, Xiangping Wu, Joanna Siebert, Qingcai Chen
Watermarking acts as a critical safeguard in text generated by Large Language Models (LLMs). By embedding identifiable signals into model outputs, watermarking enables reliable attribution and enhances the security of machine-generated content. Existing approaches typically embed signals by manipulating token generation probabilities. Despite their effectiveness, these methods inherently face a trade-off between detectability and text quality: the signal strength and randomness required for robust watermarking tend to degrade the performance of downstream tasks. In this paper, we design a novel embedding scheme that controls seed pools to facilitate diverse parallel generation of watermarked text. Based on that scheme, we propose WaterSearch, a sentence-level, search-based watermarking framework adaptable to a wide range of existing methods. WaterSearch enhances text quality by jointly optimizing two key aspects: 1) distribution fidelity and 2) watermark signal characteristics. Furthermore, WaterSearch is complemented by a sentence-level detection method with strong attack robustness. We evaluate our method on three popular LLMs across ten diverse tasks. Extensive experiments demonstrate that our method achieves an average performance improvement of 51.01\% over state-of-the-art baselines at a watermark detectability strength of 95\%. In challenging scenarios such as short text generation and low-entropy output generation, our method yields performance gains of 47.78\% and 36.47\%, respectively. Moreover, under different attack senarios including insertion, synonym substitution and paraphrase attasks, WaterSearch maintains high detectability, further validating its robust anti-attack capabilities. Our code is available at \href{https://github.com/Yukang-Lin/WaterSearch}{https://github.com/Yukang-Lin/WaterSearch}.
☆ Accelerating Bangla NLP Tasks with Automatic Mixed Precision: Resource-Efficient Training Preserving Model Efficacy
Md Mehrab Hossain Opi, Sumaiya Khan, Moshammad Farzana Rahman
Training models for Natural Language Processing (NLP) requires substantial computational resources and time, posing significant challenges, especially for NLP development in Bangla, where access to high-end hardware is often limited. In this work, we explore automatic mixed precision (AMP) training as a means to improve computational efficiency without sacrificing model performance. By leveraging a dynamic mix of 16-bit and 32-bit floating-point computations, AMP lowers GPU memory requirements and speeds up training without degrading model performance. We evaluate AMP across four standard Bangla NLP tasks, namely sentiment analysis, named entity recognition, error classification, and question answering, using four transformer-based models: BanglaBERT, BanglishBERT, XLM-R, and mBERT. Our results demonstrate that AMP accelerates training by 44.5% and reduces memory consumption by 17.6%, while maintaining F-1 score within 99.7% of the full-precision baselines. This empirical study highlights AMP's potential to democratize access to state-of-the-art NLP capabilities in hardware-constrained settings by lowering computational barriers.
☆ Auxiliary-Hyperparameter-Free Sampling: Entropy Equilibrium for Text Generation
Xiaodong Cai, Hai Lin, Shaoxiong Zhan, Weiqi Luo, Hong-Gee Kim, Hongyan Hao, Yu Yang, Hai-Tao Zheng
Token sampling strategies critically influence text generation quality in large language models (LLMs). However, existing methods introduce additional hyperparameters, requiring extensive tuning and complicating deployment. We present Entropy Equilibrium Sampling (EES), an auxiliary hyperparameter-free approach inspired by information theory that can dynamically adjust candidate sets by balancing normalized entropy with probability mass. We evaluate EES on both reasoning and generation tasks across a range of model architectures. Our results show that EES consistently performs well across temperature settings, delivering competitive accuracy and coherence while maintaining diversity. By eliminating the need for hyperparameter tuning, EES greatly simplifies deployment while improving performance. Code is available at https://github.com/shuanncai/EES
☆ Text Mining Analysis of Symptom Patterns in Medical Chatbot Conversations
Hamed Razavi
The fast growth of digital health systems has led to a need to better comprehend how they interpret and represent patient-reported symptoms. Chatbots have been used in healthcare to provide clinical support and enhance the user experience, making it possible to provide meaningful clinical patterns from text-based data through chatbots. The proposed research utilises several different natural language processing methods to study the occurrences of symptom descriptions in medicine as well as analyse the patterns that emerge through these conversations within medical bots. Through the use of the Medical Conversations to Disease Dataset which contains 960 multi-turn dialogues divided into 24 Clinical Conditions, a standardised representation of conversations between patient and bot is created for further analysis by computational means. The multi-method approach uses a variety of tools, including Latent Dirichlet Allocation (LDA) to identify latent symptom themes, K-Means to group symptom descriptions by similarity, Transformer-based Named Entity Recognition (NER) to extract medical concepts, and the Apriori algorithm to discover frequent symptom pairs. Findings from the analysis indicate a coherent structure of clinically relevant topics, moderate levels of clustering cohesiveness and several high confidence rates on the relationships between symptoms like fever headache and rash itchiness. The results support the notion that conversational medical data can be a valuable diagnostic signal for early symptom interpretation, assist in strengthening decision support and improve how users interact with tele-health technology. By demonstrating a method for converting unstructured free-flowing dialogue into actionable knowledge regarding symptoms this work provides an extensible framework to further enhance future performance, dependability and clinical utility of selecting medical chatbots.
comment : 9 pages, 4 tables
☆ FastPOS: Language-Agnostic Scalable POS Tagging Framework Low-Resource Use Case
Md Abdullah Al Kafi, Sumit Kumar Banshal
This study proposes a language-agnostic transformer-based POS tagging framework designed for low-resource languages, using Bangla and Hindi as case studies. With only three lines of framework-specific code, the model was adapted from Bangla to Hindi, demonstrating effective portability with minimal modification. The framework achieves 96.85 percent and 97 percent token-level accuracy across POS categories in Bangla and Hindi while sustaining strong F1 scores despite dataset imbalance and linguistic overlap. A performance discrepancy in a specific POS category underscores ongoing challenges in dataset curation. The strong results stem from the underlying transformer architecture, which can be replaced with limited code adjustments. Its modular and open-source design enables rapid cross-lingual adaptation while reducing model design and tuning overhead, allowing researchers to focus on linguistic preprocessing and dataset refinement, which are essential for advancing NLP in underrepresented languages.
☆ Probing the "Psyche'' of Large Reasoning Models: Understanding Through a Human Lens
Yuxiang Chen, Zuohan Wu, Ziwei Wang, Xiangning Yu, Xujia Li, Linyi Yang, Mengyue Yang, Jun Wang, Lei Chen
Large reasoning models (LRMs) have garnered significant attention from researchers owing to their exceptional capability in addressing complex tasks. Motivated by the observed human-like behaviors in their reasoning processes, this paper introduces a comprehensive taxonomy to characterize atomic reasoning steps and probe the ``psyche'' of LRM intelligence. Specifically, it comprises five groups and seventeen categories derived from human mental processes, thereby grounding the understanding of LRMs in an interdisciplinary perspective. The taxonomy is then applied for an in-depth understanding of current LRMs, resulting in a distinct labeled dataset that comprises 277,534 atomic reasoning steps. Using this resource, we analyze contemporary LRMs and distill several actionable takeaways for improving training and post-training of reasoning models. Notably, our analysis reveals that prevailing post-answer ``double-checks'' (self-monitoring evaluations) are largely superficial and rarely yield substantive revisions. Thus, incentivizing comprehensive multi-step reflection, rather than simple self-monitoring, may offer a more effective path forward. To complement the taxonomy, an automatic annotation framework, named CAPO, is proposed to leverage large language models (LLMs) for generating the taxonomy-based annotations. Experimental results demonstrate that CAPO achieves higher consistency with human experts compared to baselines, facilitating a scalable and comprehensive analysis of LRMs from a human cognitive perspective. Together, the taxonomy, CAPO, and the derived insights provide a principled, scalable path toward understanding and advancing LRM reasoning.
comment : 13 pages
☆ Dialect Identification Using Resource-Efficient Fine-Tuning Approaches SC 2025
Zirui Lin, Haris Gulzar, Monnika Roslianna Busto, Akiko Masaki, Takeharu Eda, Kazuhiro Nakadai
Dialect Identification (DI) is a task to recognize different dialects within the same language from a speech signal. DI can help to improve the downstream speech related tasks even when speakers have a strong dialect. However, fine-tuning a speech model for tasks like DI is expensive in terms of computation cost and memory requirement. Recent studies have explored fine-tuning pre-trained speech models for tasks like DI using Parameter-Efficient Fine-Tuning (PEFT) methods, which offer parameter efficiency but limited improvement in memory efficiency and training speed. To address these challenges, we explore Memory-Efficient Fine-Tuning (MEFT) methods, originally proposed for language processing, and apply them to the general-purpose pre-trained speech model. We then comprehensively analyze the GPU memory usage and fine-tuning speed based on various MEFT methods. As a case study, we fine-tune the Whisper model to identify six Mandarin subdialects from the KeSpeech dataset, reducing GPU memory usage by up to 73.25% and accelerating training speed by a factor of 2.1, while maintaining accuracy comparable to vanilla fine-tuning and PEFT methods.
comment : Published in APSIPA ASC 2025
♻ ☆ Comparative Evaluation of Expressive Japanese Character Text-to-Speech with VITS and Style-BERT-VITS2
Zackary Rackauckas, Julia Hirschberg
Synthesizing expressive Japanese character speech poses unique challenges due to pitch-accent sensitivity and stylistic variability. This paper empirically evaluates two open-source text-to-speech models--VITS and Style-BERT-VITS2 JP Extra (SBV2JE)--on in-domain, character-driven Japanese speech. Using three character-specific datasets, we evaluate models across naturalness (mean opinion and comparative mean opinion score), intelligibility (word error rate), and speaker consistency. SBV2JE matches human ground truth in naturalness (MOS 4.37 vs. 4.38), achieves lower WER, and shows slight preference in CMOS. Enhanced by pitch-accent controls and a WavLM-based discriminator, SBV2JE proves effective for applications like language learning and character dialogue generation, despite higher computational demands.
comment : Accepted to IEEE UEMCON 2025
♻ ☆ Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning , and Conversation History AAAI 2026
Tommaso Tosato, Saskia Helbling, Yorguin-Jose Mantilla-Ramos, Mahmood Hegazy, Alberto Tosato, David John Lemay, Irina Rish, Guillaume Dumas
Large language models require consistent behavioral patterns for safe deployment, yet there are indications of large variability that may lead to an instable expression of personality traits in these models. We present PERSIST (PERsonality Stability in Synthetic Text), a comprehensive evaluation framework testing 25 open-source models (1B-685B parameters) across 2 million+ responses. Using traditional (BFI, SD3) and novel LLM-adapted personality questionnaires, we systematically vary model size, personas, reasoning modes, question order or paraphrasing, and conversation history. Our findings challenge fundamental assumptions: (1) Question reordering alone can introduce large shifts in personality measurements; (2) Scaling provides limited stability gains: even 400B+ models exhibit standard deviations >0.3 on 5-point scales; (3) Interventions expected to stabilize behavior, such as reasoning and inclusion of conversation history, can paradoxically increase variability; (4) Detailed persona instructions produce mixed effects, with misaligned personas showing significantly higher variability than the helpful assistant baseline; (5) The LLM-adapted questionnaires, despite their improved ecological validity, exhibit instability comparable to human-centric versions. This persistent instability across scales and mitigation strategies suggests that current LLMs lack the architectural foundations for genuine behavioral consistency. For safety-critical applications requiring predictable behavior, these findings indicate that current alignment strategies may be inadequate.
comment : Accepted at AAAI 2026, Track on AI Alignment
♻ ☆ Generating Text from Uniform Meaning Representation AACL 2026
Emma Markle, Reihaneh Iranmanesh, Shira Wein
Uniform Meaning Representation (UMR) is a recently developed graph-based semantic representation, which expands on Abstract Meaning Representation (AMR) in a number of ways, in particular through the inclusion of document-level information and multilingual flexibility. In order to effectively adopt and leverage UMR for downstream tasks, efforts must be placed toward developing a UMR technological ecosystem. Though only a small amount of UMR annotations have been produced to date, in this work, we investigate the first approaches to producing text from multilingual UMR graphs. Exploiting the structural similarity between UMR and AMR graphs and the wide availability of AMR technologies, we introduce (1) a baseline approach which passes UMR graphs to AMR-to-text generation models, (2) a pipeline conversion of UMR to AMR, then using AMR-to-text generation models, and (3) a fine-tuning approach for both foundation models and AMR-to-text generation models with UMR data. Our best performing models achieve multilingual BERTscores of 0.825 for English and 0.882 for Chinese, a promising indication of the effectiveness of fine-tuning approaches for UMR-to-text generation even with limited UMR data.
comment : Accepted to IJCNLP-AACL 2026
♻ ☆ Do different prompt ing methods yield a common task representation in language models? NeurIPS 2025
Guy Davidson, Todd M. Gureckis, Brenden M. Lake, Adina Williams
Demonstrations and instructions are two primary approaches for prompting language models to perform in-context learning (ICL) tasks. Do identical tasks elicited in different ways result in similar representations of the task? An improved understanding of task representation mechanisms would offer interpretability insights and may aid in steering models. We study this through \textit{function vectors} (FVs), recently proposed as a mechanism to extract few-shot ICL task representations. We generalize FVs to alternative task presentations, focusing on short textual instruction prompts, and successfully extract instruction function vectors that promote zero-shot task accuracy. We find evidence that demonstration- and instruction-based function vectors leverage different model components, and offer several controls to dissociate their contributions to task performance. Our results suggest that different task promptings forms do not induce a common task representation through FVs but elicit different, partly overlapping mechanisms. Our findings offer principled support to the practice of combining instructions and task demonstrations, imply challenges in universally monitoring task inference across presentation forms, and encourage further examinations of LLM task inference mechanisms.
comment : 10 pages, 4 figures; presented at NeurIPS 2025
♻ ☆ Quantifying Cognitive Bias Induction in LLM-Generated Content AACL 2025
Abeer Alessa, Param Somane, Akshaya Lakshminarasimhan, Julian Skirzynski, Julian McAuley, Jessica Echterhoff
Large language models (LLMs) are integrated into applications like shopping reviews, summarization, or medical diagnosis support, where their use affects human decisions. We investigate the extent to which LLMs expose users to biased content and demonstrate its effect on human decision-making. We assess five LLM families in summarization and news fact-checking tasks, evaluating the consistency of LLMs with their context and their tendency to hallucinate on a new self-updating dataset. Our findings show that LLMs expose users to content that changes the context's sentiment in 26.42% of cases (framing bias), hallucinate on 60.33% of post-knowledge-cutoff questions, and highlight context from earlier parts of the prompt (primacy bias) in 10.12% of cases, averaged across all tested models. We further find that humans are 32% more likely to purchase the same product after reading a summary of the review generated by an LLM rather than the original review. To address these issues, we evaluate 18 mitigation methods across three LLM families and find the effectiveness of targeted interventions.
comment : 21 pages (including references and appendix), 3figures. accepted to AACL 2025
♻ ☆ Extracting memorized pieces of (copyrighted) books from open-weight language models
A. Feder Cooper, Aaron Gokaslan, Ahmed Ahmed, Amy B. Cyphert, Christopher De Sa, Mark A. Lemley, Daniel E. Ho, Percy Liang
Plaintiffs and defendants in copyright lawsuits over generative AI often make sweeping, opposing claims about the extent to which large language models (LLMs) have memorized plaintiffs' protected expression in their training data. Drawing on both machine learning and copyright law, we show that these polarized positions dramatically oversimplify the relationship between memorization and copyright. To do so, we extend a recent probabilistic extraction technique to measure memorization of 50 books in 17 open-weight LLMs. Through thousands of experiments, we show that the extent of memorization varies both by model and by book. With respect to our specific extraction methodology, we find that most LLMs do not memorize most books -- either in whole or in part. However, we also find that Llama 3.1 70B entirely memorizes some books, like the first Harry Potter book and 1984. In fact, the first Harry Potter is so memorized that, using a seed prompt consisting of just the first few tokens of the first chapter, we can deterministically generate the entire book near-verbatim. We discuss why our results have significant implications for copyright cases, though not ones that unambiguously favor either side.
♻ ☆ SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinations NeurIPS 2025
Buyun Liang, Liangzu Peng, Jinqi Luo, Darshan Thaker, Kwan Ho Ryan Chan, René Vidal
Large Language Models (LLMs) are increasingly deployed in high-risk domains. However, state-of-the-art LLMs often produce hallucinations, raising serious concerns about their reliability. Prior work has explored adversarial attacks for hallucination elicitation in LLMs, but it often produces unrealistic prompts, either by inserting gibberish tokens or by altering the original meaning. As a result, these approaches offer limited insight into how hallucinations may occur in practice. While adversarial attacks in computer vision often involve realistic modifications to input images, the problem of finding realistic adversarial prompts for eliciting LLM hallucinations has remained largely underexplored. To address this gap, we propose Semantically Equivalent and Coherent Attacks (SECA) to elicit hallucinations via realistic modifications to the prompt that preserve its meaning while maintaining semantic coherence. Our contributions are threefold: (i) we formulate finding realistic attacks for hallucination elicitation as a constrained optimization problem over the input prompt space under semantic equivalence and coherence constraints; (ii) we introduce a constraint-preserving zeroth-order method to effectively search for adversarial yet feasible prompts; and (iii) we demonstrate through experiments on open-ended multiple-choice question answering tasks that SECA achieves higher attack success rates while incurring almost no semantic equivalence or semantic coherence errors compared to existing methods. SECA highlights the sensitivity of both open-source and commercial gradient-inaccessible LLMs to realistic and plausible prompt variations. Code is available at https://github.com/Buyun-Liang/SECA.
comment : Accepted at NeurIPS 2025. Code is available at https://github.com/Buyun-Liang/SECA
♻ ☆ Chain-of-Query: Unleashing the Power of LLMs in SQL-Aided Table Understanding via Multi- Agent Collaboration AACL 2025
Songyuan Sui, Hongyi Liu, Serena Liu, Li Li, Soo-Hyun Choi, Rui Chen, Xia Hu
Table understanding requires structured, multi-step reasoning. Large Language Models (LLMs) struggle with it due to the structural complexity of tabular data. Recently, multi-agent frameworks for SQL generation have shown promise in tackling the challenges of understanding tabular data, but existing approaches often suffer from limitations such as the inability to comprehend table structure for reliable SQL generation, error propagation that results in invalid queries, and over-reliance on execution correctness. To address these issues, we propose Chain-of-Query (CoQ), a novel multi-agent framework for SQL-aided table understanding. CoQ adopts natural-language-style representations of table schemas to abstract away structural noise and enhance understanding. It employs a clause-by-clause SQL generation strategy to improve query quality and introduces a hybrid reasoning division that separates SQL-based mechanical reasoning from LLM-based logical inference, thereby reducing reliance on execution outcomes. Extensive experiments across four models and five widely used benchmarks demonstrate that CoQ achieves substantial accuracy improvements and significantly lowers invalid SQL rates compared to prior generic LLM-based, SQL-aided, and hybrid baselines, confirming its superior effectiveness in table understanding. The code is available at https://github.com/SongyuanSui/ChainofQuery.
comment : AACL 2025 Main Conference (Oral)
♻ ☆ Confident RAG: Enhancing the Performance of LLMs for Mathematics Question Answering through Multi-Embedding and Confidence Scoring
Shiting Chen, Zijian Zhao, Jinsong Chen
Large Language Models (LLMs) hold significant promise for mathematics education, yet they often struggle with complex mathematical reasoning. While Retrieval-Augmented Generation (RAG) mitigates these issues by grounding LLMs in external knowledge, its effectiveness remains unstable, heavily dependent on the choice of a single embedding model. Moving beyond static RAG workflows, we draw on agentic workflow patterns, a paradigm that introduces structured task decomposition and collaboration to enhance system performance. We propose and examine two novel approaches that combine the benefits of multiple embedding models. While our Mixture-Embedding RAG approach (fusing retrieved documents) shows limited gains, our Confident RAG method (generating multiple answers and selecting the one with the highest confidence score) demonstrates significant improvement. Experimental results show that Confident RAG achieved average accuracy improvements of approximately 10% over vanilla LLMs and 5% over vanilla RAG. The consistent results across different LLMs and embedding models indicate that Confident RAG is an efficient plug-and-play solution for trustworthy mathematical AI assistants. Finally, we discuss how this work lays the groundwork for deploying Agentic RAG systems in educational settings, where autonomous planning and iterative refinement can be built upon our robust retrieval foundation.
♻ ☆ Evaluating Large Language Models on the 2026 Korean CSAT Mathematics Exam: Measuring Mathematical Ability in a Zero-Data-Leakage Setting
Goun Pyeon, Inbum Heo, Jeesu Jung, Taewook Hwang, Hyuk Namgoong, Hyein Seo, Yerim Han, Eunbin Kim, Hyeonseok Kang, Sangkeun Jung
This study systematically evaluated the mathematical reasoning capabilities of Large Language Models (LLMs) using the 2026 Korean College Scholastic Ability Test (CSAT) Mathematics section, ensuring a completely contamination-free evaluation environment. To address data leakage issues in existing benchmarks, we digitized all 46 questions (22 common and 24 elective) within two hours of the exam's public release, eliminating any possibility of inclusion in model training data. We conducted comprehensive evaluations of 24 state-of-the-art LLMs across varying input modalities (Text-only, Image-only, Text+Figure) and prompt languages (Korean, English). The GPT-5 family models achieved perfect scores (100 points) under a limited set of language-modality configurations, while Grok 4, Qwen 3 235B, and Gemini 2.5 pro also scored above 97 points. Notably, gpt-oss-20B achieved 95.7 points despite its relatively small size, demonstrating high cost-effectiveness. Problem-specific analysis revealed Calculus as the weakest domain with significant performance degradation on 4-point high-difficulty problems. Text input consistently outperformed image input, while prompt language effects varied by model scale. In reasoning enhancement experiments with GPT-5 series, increased reasoning intensity improved performance (82.6->100 points) but quadrupled token usage and drastically reduced efficiency, suggesting that models with minimal reasoning may be more practical. This research contributes: (1) implementation of a completely unexposed evaluation environment, (2) a standardized digitization pipeline that converts human-targeted exam materials into LLM-ready evaluation data, and (3) a practical evaluation perspective integrating performance, cost, and time considerations. Detailed results and model comparisons are available at the 2026 Korean CSAT LLM Evaluation Leaderboard; https://isoft.cnu.ac.kr/csat2026/
comment : 52 pages
♻ ☆ Event2Vec: A Geometric Approach to Learning Composable Representations of Event Sequences
Antonin Sulc
The study of neural representations, both in biological and artificial systems, is increasingly revealing the importance of geometric and topological structures. Inspired by this, we introduce Event2Vec, a novel framework for learning representations of discrete event sequences. Our model leverages a simple, additive recurrent structure to learn composable, interpretable embeddings. We provide a theoretical analysis demonstrating that, under specific training objectives, our model's learned representations in a Euclidean space converge to an ideal additive structure. This ensures that the representation of a sequence is the vector sum of its constituent events, a property we term the linear additive hypothesis. To address the limitations of Euclidean geometry for hierarchical data, we also introduce a variant of our model in hyperbolic space, which is naturally suited to embedding tree-like structures with low distortion. We present experiments to validate our hypothesis. Quantitative evaluation on the Brown Corpus yields a Silhouette score of 0.0564, outperforming a Word2Vec baseline (0.0215), demonstrating the model's ability to capture structural dependencies without supervision.
comment : 10 pages, 3 figures, Symmetry and Geometry in Neural Representations Workshop at NeuralIPS (NeurReps) 2025
♻ ☆ SRPO: Self-Referential Policy Optimization for Vision -Language-Action Models
Senyu Fei, Siyin Wang, Li Ji, Ao Li, Shiduo Zhang, Liming Liu, Jinlong Hou, Jingjing Gong, Xianzhong Zhao, Xipeng Qiu
Vision-Language-Action (VLA) models excel in robotic manipulation but are constrained by their heavy reliance on expert demonstrations, leading to demonstration bias and limiting performance. Reinforcement learning (RL) is a vital post-training strategy to overcome these limits, yet current VLA-RL methods, including group-based optimization approaches, are crippled by severe reward sparsity. Relying on binary success indicators wastes valuable information in failed trajectories, resulting in low training efficiency. To solve this, we propose Self-Referential Policy Optimization (SRPO), a novel VLA-RL framework. SRPO eliminates the need for external demonstrations or manual reward engineering by leveraging the model's own successful trajectories, generated within the current training batch, as a self-reference. This allows us to assign a progress-wise reward to failed attempts. A core innovation is the use of latent world representations to measure behavioral progress robustly. Instead of relying on raw pixels or requiring domain-specific fine-tuning, we utilize the compressed, transferable encodings from a world model's latent space. These representations naturally capture progress patterns across environments, enabling accurate, generalized trajectory comparison. Empirical evaluations on the LIBERO benchmark demonstrate SRPO's efficiency and effectiveness. Starting from a supervised baseline with 48.9% success, SRPO achieves a new state-of-the-art success rate of 99.2% in just 200 RL steps, representing a 103% relative improvement without any extra supervision. Furthermore, SRPO shows substantial robustness, achieving a 167% performance improvement on the LIBERO-Plus benchmark.
♻ ☆ Black-Box On-Policy Distillation of Large Language Models
Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, Furu Wei
Black-box distillation creates student large language models (LLMs) by learning from a proprietary teacher model's text outputs alone, without access to its internal logits or parameters. In this work, we introduce Generative Adversarial Distillation (GAD), which enables on-policy and black-box distillation. GAD frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher LLM's, creating a minimax game. The discriminator acts as an on-policy reward model that co-evolves with the student, providing stable, adaptive feedback. Experimental results show that GAD consistently surpasses the commonly used sequence-level knowledge distillation. In particular, Qwen2.5-14B-Instruct (student) trained with GAD becomes comparable to its teacher, GPT-5-Chat, on the LMSYS-Chat automatic evaluation. The results establish GAD as a promising and effective paradigm for black-box LLM distillation.
♻ ☆ Speech Judge : Towards Human-Level Judgment for Speech Naturalness
Xueyao Zhang, Chaoren Wang, Huan Liao, Ziniu Li, Yuancheng Wang, Li Wang, Dongya Jia, Yuanzhe Chen, Xiulin Li, Zhuo Chen, Zhizheng Wu
Aligning large generative models with human feedback is a critical challenge. In speech synthesis, this is particularly pronounced due to the lack of a large-scale human preference dataset, which hinders the development of models that truly align with human perception. To address this, we introduce SpeechJudge, a comprehensive suite comprising a dataset, a benchmark, and a reward model centered on naturalness--one of the most fundamental subjective metrics for speech synthesis. First, we present SpeechJudge-Data, a large-scale human feedback corpus of 99K speech pairs. The dataset is constructed using a diverse set of advanced zero-shot text-to-speech (TTS) models across diverse speech styles and multiple languages, with human annotations for both intelligibility and naturalness preference. From this, we establish SpeechJudge-Eval, a challenging benchmark for speech naturalness judgment. Our evaluation reveals that existing metrics and AudioLLMs struggle with this task; the leading model, Gemini-2.5-Flash, achieves less than 70% agreement with human judgment, highlighting a significant gap for improvement. To bridge this gap, we develop SpeechJudge-GRM, a generative reward model (GRM) based on Qwen2.5-Omni-7B. It is trained on SpeechJudge-Data via a two-stage post-training process: Supervised Fine-Tuning (SFT) with Chain-of-Thought rationales followed by Reinforcement Learning (RL) with GRPO on challenging cases. On the SpeechJudge-Eval benchmark, the proposed SpeechJudge-GRM demonstrates superior performance, achieving 77.2% accuracy (and 79.4% after inference-time scaling @10) compared to a classic Bradley-Terry reward model (72.7%). Furthermore, SpeechJudge-GRM can be also employed as a reward function during the post-training of speech generation models to facilitate their alignment with human preferences.
comment : Dataset, Model, and Code: https://github.com/AmphionTeam/SpeechJudge
♻ ☆ Reasoning -Intensive Regression
Diane Tchuindjo, Omar Khattab
AI researchers and practitioners increasingly apply large language models (LLMs) to what we call reasoning-intensive regression (RiR), i.e., deducing subtle numerical scores from text. Unlike standard language regression tasks, e.g., for sentiment or similarity, RiR often appears instead in ad-hoc problems such as rubric-based scoring, modeling dense rewards in complex environments, or domain-specific retrieval, where much deeper analysis of context is required while only limited task-specific training data and computation are available. We cast four realistic problems as RiR tasks to establish an initial benchmark, and use that to test our hypothesis that prompting frozen LLMs and finetuning Transformer encoders via gradient descent will both often struggle in RiR. We then propose MENTAT, a simple and lightweight method that combines batch-reflective prompt optimization with neural ensemble learning. MENTAT achieves up to 65% improvement over both baselines, though substantial room remains for future advances in RiR.
♻ ☆ Advancing Natural Language Formalization to First Order Logic with Fine-tuned LLMs
Felix Vossel, Till Mossakowski, Björn Gehrke
Automating the translation of natural language to first-order logic (FOL) is crucial for knowledge representation and formal methods, yet remains challenging. We present a systematic evaluation of fine-tuned LLMs for this task, comparing architectures (encoder-decoder vs. decoder-only) and training strategies. Using the MALLS and Willow datasets, we explore techniques like vocabulary extension, predicate conditioning, and multilingual training, introducing metrics for exact match, logical equivalence, and predicate alignment. Our fine-tuned Flan-T5-XXL achieves 70% accuracy with predicate lists, outperforming GPT-4o and even the DeepSeek-R1-0528 model with CoT reasoning ability as well as symbolic systems like ccg2lambda. Key findings show: (1) predicate availability boosts performance by 15-20%, (2) T5 models surpass larger decoder-only LLMs, and (3) models generalize to unseen logical arguments (FOLIO dataset) without specific training. While structural logic translation proves robust, predicate extraction emerges as the main bottleneck.
comment : 15 pages, 7 tables, accepted at the International Joint Conference on Learning & Reasoning (IJCLR 2025)
♻ ☆ ShoppingBench: A Real-Wo rl d Intent-Grounded Shopping Benchmark for LLM-based Agent s AAAI2026
Jiangyuan Wang, Kejun Xiao, Qi Sun, Huaipeng Zhao, Tao Luo, Jian Dong Zhang, Xiaoyi Zeng
Existing benchmarks in e-commerce primarily focus on basic user intents, such as finding or purchasing products. However, real-world users often pursue more complex goals, such as applying vouchers, managing budgets, and finding multi-products seller. To bridge this gap, we propose ShoppingBench, a novel end-to-end shopping benchmark designed to encompass increasingly challenging levels of grounded intent. Specifically, we propose a scalable framework to simulate user instructions based on various intents derived from sampled real-world products. To facilitate consistent and reliable evaluations, we provide a large-scale shopping sandbox that serves as an interactive simulated environment, incorporating over 2.5 million real-world products. Experimental results demonstrate that even state-of-the-art language agents (such as GPT-4.1) achieve absolute success rates under 50% on our benchmark tasks, highlighting the significant challenges posed by our ShoppingBench. In addition, we propose a trajectory distillation strategy and leverage supervised fine-tuning, along with reinforcement learning on synthetic trajectories, to distill the capabilities of a large language agent into a smaller one. As a result, our trained agent achieves competitive performance compared to GPT-4.1.
comment : submit to AAAI2026
♻ ☆ Explainable Semantic Text Relation s: A Question-Answering Framework for Comparing Document Content
Yehudit Aperstein, Alon Gottlib, Gal Benita, Alexander Apartsin
Understanding semantic relations between two texts is crucial for many information and document management tasks, in which one must determine whether the content fully overlaps, is completely superseded by another document, or overlaps only partially, with unique information in each. Beyond establishing this relation, it is equally important to provide explainable outputs that specify which pieces of information are present, missing, or newly added between the text pair. In this study, we formally define semantic relations between two texts through the set-theoretic relation between their respective Answerable Question Sets (AQS), the sets of questions each text can answer. Under this formulation, Semantic Text Relation (STR), such as equivalence, inclusion, and mutual overlap, becomes a well-defined set relation between the corresponding texts' AQSs. The set differences between the AQSs also serve as an explanation or diagnostic tool for identifying how the information in the texts diverges. Using this definition, we construct a synthetic benchmark that captures fine-grained informational relations through controlled paraphrasing and deliberate information removal supported by AQS manipulations. We then use this dataset to evaluate several discriminative and generative models for classifying text pairs into STR categories, assessing how well different model architectures capture semantic relations beyond surface-level similarity. We publicly release both the dataset and the data generation code to support further research.
comment : 18 pages, 1 figure
♻ ☆ Efficient LLM-Jailbreaking via Multimodal-LLM Jailbreak
Haoxuan Ji, Zheng Lin, Zhenxing Niu, Xinbo Gao, Gang Hua
This paper focuses on jailbreaking attacks against large language models (LLMs), eliciting them to generate objectionable content in response to harmful user queries. Unlike previous LLM-jailbreak methods that directly orient to LLMs, our approach begins by constructing a multimodal large language model (MLLM) built upon the target LLM. Subsequently, we perform an efficient MLLM jailbreak and obtain a jailbreaking embedding. Finally, we convert the embedding into a textual jailbreaking suffix to carry out the jailbreak of target LLM. Compared to the direct LLM-jailbreak methods, our indirect jailbreaking approach is more efficient, as MLLMs are more vulnerable to jailbreak than pure LLM. Additionally, to improve the attack success rate of jailbreak, we propose an image-text semantic matching scheme to identify a suitable initial input. Extensive experiments demonstrate that our approach surpasses current state-of-the-art jailbreak methods in terms of both efficiency and effectiveness. Moreover, our approach exhibits superior cross-class generalization abilities.
♻ ☆ Attributional Safety Failures in Large Language Models under Code-Mixed Perturbations
Somnath Banerjee, Pratyush Chatterjee, Shanu Kumar, Sayan Layek, Parag Agrawal, Rima Hazra, Animesh Mukherjee
While LLMs appear robustly safety-aligned in English, we uncover a catastrophic, overlooked weakness: attributional collapse under code-mixed perturbations. Our systematic evaluation of open models shows that the linguistic camouflage of code-mixing -- ``blending languages within a single conversation'' -- can cause safety guardrails to fail dramatically. Attack success rates (ASR) spike from a benign 9\% in monolingual English to 69\% under code-mixed inputs, with rates exceeding 90\% in non-Western contexts such as Arabic and Hindi. These effects hold not only on controlled synthetic datasets but also on real-world social media traces, revealing a serious risk for billions of users. To explain why this happens, we introduce saliency drift attribution (SDA), an interpretability framework that shows how, under code-mixing, the model's internal attention drifts away from safety-critical tokens (e.g., ``violence'' or ``corruption''), effectively blinding it to harmful intent. Finally, we propose a lightweight translation-based restoration strategy that recovers roughly 80\% of the safety lost to code-mixing, offering a practical path toward more equitable and robust LLM safety.
♻ ☆ Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-train ing NeurIPS 2025
Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan, Lin, Jan Kautz, Pavlo Molchanov
Pre-training datasets are typically collected from web content and lack inherent domain divisions. For instance, widely used datasets like Common Crawl do not include explicit domain labels, while manually curating labeled datasets such as The Pile is labor-intensive. Consequently, identifying an optimal pre-training data mixture remains a challenging problem, despite its significant benefits for pre-training performance. To address these challenges, we propose CLustering-based Iterative Data Mixture Bootstrapping (Nemotron-CLIMB), an automated framework that discovers, evaluates, and refines data mixtures in a pre-training setting. Specifically, Nemotron-CLIMB embeds and clusters large-scale datasets in a semantic space and then iteratively searches for optimal mixtures using a smaller proxy model and a predictor. When continuously trained on 400B tokens with this mixture, our 1B model exceeds the state-of-the-art Llama-3.2-1B by 2.0%. Moreover, we observe that optimizing for a specific domain (e.g., Social Sciences) yields a 5% improvement over random sampling. Finally, we introduce Nemotron-ClimbLab, a filtered 1.2-trillion-token corpus with 20 clusters as a research playground, and Nemotron-ClimbMix, a compact yet powerful 400-billion-token dataset designed for efficient pre-training that delivers superior performance under an equal token budget. We analyze the final data mixture, elucidating the characteristics of an optimal data mixture. Our data is available at: https://research.nvidia.com/labs/lpr/climb/
comment : Accepted to NeurIPS 2025
♻ ☆ COMPASS: Context -Modulated PID Attention Steering System for Hallucination Mitigation
Kenji Sahay, Snigdha Pandya, Rohan Nagale, Anna Lin, Shikhar Shiromani, Kevin Zhu, Dev Sunishchal
Large language models (LLMs) often generate fluent but factually incorrect statements despite having access to relevant evidence, a failure mode rooted in how they allocate attention between contextual and parametric knowledge. Understanding and steering this internal behavior is key both for trustworthy deployment and for scientific interpretability of model mechanisms. We introduce COMPASS (Context-Modulated PID Attention Steering System), a lightweight, interpretable control framework that embeds a model-based feedback loop directly within decoding. COMPASS quantifies context reliance via a transparent metric, the Context Reliance Score (CRS), which serves as an online probe of how attention heads ground generation in evidence. Using this interpretable signal, a PID controller dynamically modulates attention heads to maintain factual consistency without retraining or multi-pass decoding. Across benchmarks (HotpotQA, XSum, HaluEval, RAGTruth), COMPASS consistently reduces contextual hallucination rates (2.8 to 5.8 percent absolute) while revealing how distinct attention heads contribute to evidence alignment. These results highlight feedback-driven interpretability as a pathway toward scientific understanding of LLM behavior.
comment : 9 pages, 6 figures including algorithmns, 2 tables
♻ ☆ Small Drafts, Big Verdict: Information -Intensive Visual Reasoning via Speculation
Yuhan Liu, Lianhui Qin, Shengjie Wang
Large Vision-Language Models (VLMs) have achieved remarkable progress in multimodal understanding, yet they struggle when reasoning over information-intensive images that densely interleave textual annotations with fine-grained graphical elements. The main challenges lie in precisely localizing critical cues in dense layouts and multi-hop reasoning to integrate dispersed evidence. We propose Speculative Verdict (SV), a training-free framework inspired by speculative decoding that combines multiple lightweight draft experts with a large verdict model. In the draft stage, small VLMs act as draft experts to generate reasoning paths that provide diverse localization candidates; in the verdict stage, a strong VLM synthesizes these paths to produce the final answer, minimizing computational cost while recovering correct answers. To further improve efficiency and accuracy, SV introduces a consensus expert selection mechanism that forwards only high-agreement reasoning paths to the verdict. Empirically, SV achieves consistent gains on challenging information-intensive and high-resolution visual question answering benchmarks, including InfographicVQA, ChartMuseum, ChartQAPro, and HR-Bench 4K. By synthesizing correct insights from multiple partially accurate reasoning paths, SV achieves both error correction and cost-efficiency compared to large proprietary models or training pipelines. Code is available at https://github.com/Tinaliu0123/speculative-verdict.
♻ ☆ Template-assisted Contrastive Learning of Task-oriented Dialogue Sentence Embeddings
Minsik Oh, Jiwei Li, Guoyin Wang
Learning high quality sentence embeddings from dialogues has drawn increasing attentions as it is essential to solve a variety of dialogue-oriented tasks with low annotation cost. Annotating and gathering utterance relationships in conversations are difficult, while token-level annotations, \eg, entities, slots and templates, are much easier to obtain. Other sentence embedding methods are usually sentence-level self-supervised frameworks and cannot utilize token-level extra knowledge. We introduce Template-aware Dialogue Sentence Embedding (TaDSE), a novel augmentation method that utilizes template information to learn utterance embeddings via self-supervised contrastive learning framework. We further enhance the effect with a synthetically augmented dataset that diversifies utterance-template association, in which slot-filling is a preliminary step. We evaluate TaDSE performance on five downstream benchmark dialogue datasets. The experiment results show that TaDSE achieves significant improvements over previous SOTA methods for dialogue. We further introduce a novel analytic instrument of semantic compression test, for which we discover a correlation with uniformity and alignment. Our code will be released upon acceptance.
♻ ☆ MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models NeurIPS 2025
Hengzhi Li, Megan Tjandrasuwita, Yi R. Fung, Armando Solar-Lezama, Paul Pu Liang
As AI becomes more closely integrated with peoples' daily activities, socially intelligent AI that can understand and interact seamlessly with humans in daily lives is increasingly important. However, current works in AI social reasoning all rely on language-only or language-dominant approaches to benchmark and training models, resulting in systems that are improving in verbal communication but struggle with nonverbal social understanding. To address this limitation, we tap into a novel data source rich in nonverbal social interactions -- mime videos. Mimes refer to the art of expression through gesture and movement without spoken words, which presents unique challenges and opportunities in interpreting nonverbal social communication. We contribute a new dataset called MimeQA, obtained by sourcing ~8 hours of videos clips from YouTube and developing a comprehensive video question-answering benchmark comprising 806 carefully annotated and verified question-answer pairs, designed to probe nonverbal social reasoning capabilities. Using MimeQA, we evaluate state-of-the-art video large language models (VideoLLMs) and find that they achieve low accuracy, generally ranging from 20-30%, while humans score 86%. Our analysis reveals that VideoLLMs often fail to ground imagined objects and over-rely on the text prompt while ignoring subtle nonverbal interactions. We hope to inspire future work in AI models that embody true social intelligence capable of interpreting non-verbal human interactions.
comment : NeurIPS 2025 Datasets and Benchmarks
♻ ☆ Adaptive Margin RL HF via Preference over Preferences
Yaswanth Chittepu, Prasann Singhal, Greg Durrett, Scott Niekum
Margin-based optimization is fundamental to improving generalization and robustness in classification tasks. In the context of reward model learning from preferences within Reinforcement Learning from Human Feedback (RLHF), existing methods typically rely on no margins, fixed margins, or margins that are simplistic functions of preference ratings. However, such formulations often fail to account for the varying strengths of different preferences, for example some preferences are associated with larger margins between responses, or they rely on noisy margin information derived from ratings. We argue that modeling the strength of preferences can lead to better generalization and more faithful alignment. Furthermore, many existing methods that use adaptive margins assume access to accurate preference scores, which can be difficult for humans to provide reliably. We propose an approach that leverages preferences over preferences, that is annotations indicating which of two preferences reflects a stronger distinction. We use this ordinal signal to infer adaptive margins on a per-datapoint basis. We introduce an extension to Direct Preference Optimization (DPO), DPO-PoP, that incorporates adaptive margins from preference-over-preference supervision, enabling improved discriminative and generative performance. Empirically, our method outperforms vanilla DPO, DPO with fixed margins, and DPO with ground-truth margins on the UltraFeedback dataset. Additionally, we show that there is a tradeoff between discriminative and generative performance: improving test classification accuracy, particularly by correctly labeling weaker preferences at the expense of stronger ones, can lead to a decline in generative quality. To navigate this tradeoff, we propose two sampling strategies to gather preference-over-preference labels: one favoring discriminative performance and one favoring generative performance.
♻ ☆ Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity
Austin Meek, Eitan Sprejer, Iván Arcuschin, Austin J. Brockmeier, Steven Basart
Chain-of-thought (CoT) outputs let us read a model's step-by-step reasoning. Since any long, serial reasoning process must pass through this textual trace, the quality of the CoT is a direct window into what the model is thinking. This visibility could help us spot unsafe or misaligned behavior (monitorability), but only if the CoT is transparent about its internal reasoning (faithfulness). Fully measuring faithfulness is difficult, so researchers often focus on examining the CoT in cases where the model changes its answer after adding a cue to the input. This proxy finds some instances of unfaithfulness but loses information when the model maintains its answer, and does not investigate aspects of reasoning not tied to the cue. We extend these results to a more holistic sense of monitorability by introducing verbosity: whether the CoT lists every factor needed to solve the task. We combine faithfulness and verbosity into a single monitorability score that shows how well the CoT serves as the model's external `working memory', a property that many safety schemes based on CoT monitoring depend on. We evaluate instruction-tuned and reasoning models on BBH, GPQA, and MMLU. Our results show that models can appear faithful yet remain hard to monitor when they leave out key factors, and that monitorability differs sharply across model families. We release our evaluation code using the Inspect library to support reproducible future work.
comment : Project page at https://ajmeek.github.io/cot_monitorability_website/
Artificial Intelligence 109
☆ ML-Tool-Bench: Tool-Augmented Planning for ML Tasks
Yaswanth Chittepu, Raghavendra Addanki, Tung Mai, Anup Rao, Branislav Kveton
The development of autonomous machine learning (ML) agents capable of end-to-end data science workflows represents a significant frontier in artificial intelligence. These agents must orchestrate complex sequences of data analysis, feature engineering, model selection, and hyperparameter optimization, tasks that require sophisticated planning and iteration. While recent work on building ML agents has explored using large language models (LLMs) for direct code generation, tool-augmented approaches offer greater modularity and reliability. However, existing tool-use benchmarks focus primarily on task-specific tool selection or argument extraction for tool invocation, failing to evaluate the sophisticated planning capabilities required for ML Agents. In this work, we introduce a comprehensive benchmark for evaluating tool-augmented ML agents using a curated set of 61 specialized tools and 15 tabular ML challenges from Kaggle. Our benchmark goes beyond traditional tool-use evaluation by incorporating an in-memory named object management, allowing agents to flexibly name, save, and retrieve intermediate results throughout the workflows. We demonstrate that standard ReAct-style approaches struggle to generate valid tool sequences for complex ML pipelines, and that tree search methods with LLM-based evaluation underperform due to inconsistent state scoring. To address these limitations, we propose two simple approaches: 1) using shaped deterministic rewards with structured textual feedback, and 2) decomposing the original problem into a sequence of sub-tasks, which significantly improves trajectory validity and task performance. Using GPT-4o, our approach improves over ReAct by 16.52 percentile positions, taking the median across all Kaggle challenges. We believe our work provides a foundation for developing more capable tool-augmented planning ML agents.
☆ EDIT: Ea rl y Diffusion Inference Termination for dLLMs Based on Dynamics of Training Gradients
He-Yen Hsieh, Hong Wang, H. T. Kung
Diffusion-based large language models (dLLMs) refine token generations through iterative denoising, but answers often stabilize before all steps complete. We propose EDIT (Early Diffusion Inference Termination), an inference-time criterion that adaptively stops denoising once sufficient reasoning stability relative to training-time reasoning is detected. EDIT monitors the alignment between token activations and a reasoning map derived from AdamW-aggregated LoRA updates captured during supervised fine-tuning (SFT). During training, optimization dynamics generate rich metadata about parameter importance that in prior methods is typically discarded upon model release. We preserve this information as a compact representation of learned reasoning pathways. During inference, alignment scores are converted to a distribution over the tokens already unmasked at the current denoising step, and convergence is detected when KL divergence between consecutive steps falls below a threshold on the matched unmasked (visible) tokens. Across reasoning benchmarks, EDIT reduces diffusion steps by 11.8% to 68.3% while preserving or improving accuracy in most settings, with approximately 0.02% storage overhead (about 1.5-2 MB for all QKV modules across 32 blocks in an 8 GB model). By utilizing training-gradient dynamics, our work opens a new research direction for reducing dLLM inference time and cost.
comment : 22 pages, 11 figures
☆ MambaScope: Coarse-to-Fine Scoping for Efficient Vision Mamba
Shanhui Liu, Rui Xu, Yunke Wang
Vision Mamba has emerged as a promising and efficient alternative to Vision Transformers, yet its efficiency remains fundamentally constrained by the number of input tokens. Existing token reduction approaches typically adopt token pruning or merging to reduce computation. However, they inherently lead to information loss, as they discard or compress token representations. This problem is exacerbated when applied uniformly to fine-grained token representations across all images, regardless of visual complexity. We observe that not all inputs require fine-grained processing. Simple images can be effectively handled at coarse resolution, while only complex ones may warrant refinement. Based on this insight, we propose \textit{Coarse-to-Fine Vision Mamba (CF-ViM)}, an adaptive framework for efficient inference. CF-ViM first performs coarse-grained inference by dividing the input image into large patches, significantly reducing the token length and computation. When the model's prediction confidence is low, selected regions are re-processed at a finer resolution to recover critical visual details with minimal additional cost. This dynamic resolution assignment strategy allows CF-ViM to allocate computation adaptively according to image complexity, ensuring efficient processing without compromising essential visual information. Experiments on ImageNet demonstrate that CF-ViM outperforms both the baseline Vision Mamba and state-of-the-art token reduction techniques in terms of accuracy and efficiency.
☆ Graph-Attention Network with Adversarial Domain Alignment for Robust Cross-Domain Facial Expression Recognition ACML 2025
Razieh Ghaedi, AmirReza BabaAhmadi, Reyer Zwiggelaar, Xinqi Fan, Nashid Alam
Cross-domain facial expression recognition (CD-FER) remains difficult due to severe domain shift between training and deployment data. We propose Graph-Attention Network with Adversarial Domain Alignment (GAT-ADA), a hybrid framework that couples a ResNet-50 as backbone with a batch-level Graph Attention Network (GAT) to model inter-sample relations under shift. Each mini-batch is cast as a sparse ring graph so that attention aggregates cross-sample cues that are informative for adaptation. To align distributions, GAT-ADA combines adversarial learning via a Gradient Reversal Layer (GRL) with statistical alignment using CORAL and MMD. GAT-ADA is evaluated under a standard unsupervised domain adaptation protocol: training on one labeled source (RAF-DB) and adapting to multiple unlabeled targets (CK+, JAFFE, SFEW 2.0, FER2013, and ExpW). GAT-ADA attains 74.39% mean cross-domain accuracy. On RAF-DB to FER2013, it reaches 98.0% accuracy, corresponding to approximately a 36-point improvement over the best baseline we re-implemented with the same backbone and preprocessing.
comment : 17 pages, 5 figures. Accepted at the 17th Asian Conference on Machine Learning (ACML 2025), Taipei, Taiwan, December 9-12, 2025
☆ Doppler-Enhanced Deep Learning: Improving Thyroid Nodule Segmentation with YOLOv5 Instance Segmentation
Mahmoud El Hussieni
The increasing prevalence of thyroid cancer globally has led to the development of various computer-aided detection methods. Accurate segmentation of thyroid nodules is a critical first step in the development of AI-assisted clinical decision support systems. This study focuses on instance segmentation of thyroid nodules using YOLOv5 algorithms on ultrasound images. We evaluated multiple YOLOv5 variants (Nano, Small, Medium, Large, and XLarge) across two dataset versions, with and without doppler images. The YOLOv5-Large algorithm achieved the highest performance with a dice score of 91\% and mAP of 0.87 on the dataset including doppler images. Notably, our results demonstrate that doppler images, typically excluded by physicians, can significantly improve segmentation performance. The YOLOv5-Small model achieved 79\% dice score when doppler images were excluded, while including them improved performance across all model variants. These findings suggest that instance segmentation with YOLOv5 provides an effective real-time approach for thyroid nodule detection, with potential clinical applications in automated diagnostic systems.
☆ XAI-Driven Skin Disease Classification: Leveraging GANs to Augment ResNet-50 Performance
Kim Gerard A. Villanueva, Priyanka Kumar
Accurate and timely diagnosis of multi-class skin lesions is hampered by subjective methods, inherent data imbalance in datasets like HAM10000, and the "black box" nature of Deep Learning (DL) models. This study proposes a trustworthy and highly accurate Computer-Aided Diagnosis (CAD) system to overcome these limitations. The approach utilizes Deep Convolutional Generative Adversarial Networks (DCGANs) for per class data augmentation to resolve the critical class imbalance problem. A fine-tuned ResNet-50 classifier is then trained on the augmented dataset to classify seven skin disease categories. Crucially, LIME and SHAP Explainable AI (XAI) techniques are integrated to provide transparency by confirming that predictions are based on clinically relevant features like irregular morphology. The system achieved a high overall Accuracy of 92.50 % and a Macro-AUC of 98.82 %, successfully outperforming various prior benchmarked architectures. This work successfully validates a verifiable framework that combines high performance with the essential clinical interpretability required for safe diagnostic deployment. Future research should prioritize enhancing discrimination for critical categories, such as Melanoma NOS (F1-Score is 0.8602).
☆ Automatic Pith Detection in Tree Cross-Section Images Using Deep Learning
Tzu-I Liao, Mahmoud Fakhry, Jibin Yesudas Varghese
Pith detection in tree cross-sections is essential for forestry and wood quality analysis but remains a manual, error-prone task. This study evaluates deep learning models -- YOLOv9, U-Net, Swin Transformer, DeepLabV3, and Mask R-CNN -- to automate the process efficiently. A dataset of 582 labeled images was dynamically augmented to improve generalization. Swin Transformer achieved the highest accuracy (0.94), excelling in fine segmentation. YOLOv9 performed well for bounding box detection but struggled with boundary precision. U-Net was effective for structured patterns, while DeepLabV3 captured multi-scale features with slight boundary imprecision. Mask R-CNN initially underperformed due to overlapping detections, but applying Non-Maximum Suppression (NMS) improved its IoU from 0.45 to 0.80. Generalizability was next tested using an oak dataset of 11 images from Oregon State University's Tree Ring Lab. Additionally, for exploratory analysis purposes, an additional dataset of 64 labeled tree cross-sections was used to train the worst-performing model to see if this would improve its performance generalizing to the unseen oak dataset. Key challenges included tensor mismatches and boundary inconsistencies, addressed through hyperparameter tuning and augmentation. Our results highlight deep learning's potential for tree cross-section pith detection, with model choice depending on dataset characteristics and application needs.
comment : 8 pages, 7 figures
☆ Melody or Machine: Detecting Synthetic Music with Dual-Stream Contrastive Learning
Arnesh Batra, Dev Sharma, Krish Thukral, Ruhani Bhatia, Naman Batra, Aditya Gautam
The rapid evolution of end-to-end AI music generation poses an escalating threat to artistic authenticity and copyright, demanding detection methods that can keep pace. While foundational, existing models like SpecTTTra falter when faced with the diverse and rapidly advancing ecosystem of new generators, exhibiting significant performance drops on out-of-distribution (OOD) content. This generalization failure highlights a critical gap: the need for more challenging benchmarks and more robust detection architectures. To address this, we first introduce Melody or Machine (MoM), a new large-scale benchmark of over 130,000 songs (6,665 hours). MoM is the most diverse dataset to date, built with a mix of open and closed-source models and a curated OOD test set designed specifically to foster the development of truly generalizable detectors. Alongside this benchmark, we introduce CLAM, a novel dual-stream detection architecture. We hypothesize that subtle, machine-induced inconsistencies between vocal and instrumental elements, often imperceptible in a mixed signal, offer a powerful tell-tale sign of synthesis. CLAM is designed to test this hypothesis by employing two distinct pre-trained audio encoders (MERT and Wave2Vec2) to create parallel representations of the audio. These representations are fused by a learnable cross-aggregation module that models their inter-dependencies. The model is trained with a dual-loss objective: a standard binary cross-entropy loss for classification, complemented by a contrastive triplet loss which trains the model to distinguish between coherent and artificially mismatched stream pairings, enhancing its sensitivity to synthetic artifacts without presuming a simple feature alignment. CLAM establishes a new state-of-the-art in synthetic music forensics. It achieves an F1 score of 0.925 on our challenging MoM benchmark.
comment : Accepted at Transactions on Machine Learning Research (TMLR)
☆ Neuroscience-Inspired Memory Replay for Continual Learning: A Comparative Study of Predictive Coding and Backpropagation-Based Strategies
Goutham Nalagatla, Shreyas Grandhe
Continual learning remains a fundamental challenge in artificial intelligence, with catastrophic forgetting posing a significant barrier to deploying neural networks in dynamic environments. Inspired by biological memory consolidation mechanisms, we propose a novel framework for generative replay that leverages predictive coding principles to mitigate forgetting. We present a comprehensive comparison between predictive coding-based and backpropagation-based gen- erative replay strategies, evaluating their effectiveness on task retention and transfer efficiency across multiple benchmark datasets. Our experimental results demonstrate that predictive coding-based replay achieves superior retention performance (average 15.3% improvement) while maintaining competitive transfer efficiency, suggesting that biologically-inspired mechanisms can offer principled solutions to continual learning challenges. The proposed framework provides insights into the relationship between biological memory processes and artificial learning systems, opening new avenues for neuroscience-inspired AI research.
comment : 9 pages, 3 figures
☆ ART: Adaptive Response Tuning Framework -- A Multi- Agent Tournament-Based Approach to LLM Response Optimization
Omer Jauhar Khan
Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation. However, single-model responses often exhibit inconsistencies, hallucinations, and varying quality across different query domains. This paper presents ART (Adaptive Response Tuning), a novel framework that employs tournament-style ELO ranking and multi-agent reasoning to systematically optimize LLM outputs. By enabling multiple LLM agents to compete, critique, and collaborate through structured tournament workflows, ART produces consensus responses that outperform individual model outputs. Our framework introduces configurable tournament parameters, dynamic agent selection, and multiple consensus fusion strategies. Experimental evaluations demonstrate significant improvements in response accuracy, coherence, and reliability compared to baseline single-model approaches. The ART framework provides a scalable, production-ready solution for applications requiring high-quality, vetted LLM responses, achieving an 8.4% improvement in overall quality metrics and R22 values exceeding 0.96 in ELO rating convergence.
comment : 8 pages, 5 figures, 5 tables. Conference-style paper
☆ Stable Voting and the Splitting of Cycles AAAI
Wesley H. Holliday, Milan Mossé, Chase Norman, Eric Pacuit, Cynthia Wang
Algorithms for resolving majority cycles in preference aggregation have been studied extensively in computational social choice. Several sophisticated cycle-resolving methods, including Tideman's Ranked Pairs, Schulze's Beat Path, and Heitzig's River, are refinements of the Split Cycle (SC) method that resolves majority cycles by discarding the weakest majority victories in each cycle. Recently, Holliday and Pacuit proposed a new refinement of Split Cycle, dubbed Stable Voting, and a simplification thereof, called Simple Stable Voting (SSV). They conjectured that SSV is a refinement of SC whenever no two majority victories are of the same size. In this paper, we prove the conjecture up to 6 alternatives and refute it for more than 6 alternatives. While our proof of the conjecture for up to 5 alternatives uses traditional mathematical reasoning, our 6-alternative proof and 7-alternative counterexample were obtained with the use of SAT solving. The SAT encoding underlying this proof and counterexample is applicable far beyond SC and SSV: it can be used to test properties of any voting method whose choice of winners depends only on the ordering of margins of victory by size.
comment : Forthcoming in Proceedings of the 40th Annual AAAI Conference on Artificial Intelligence (AAAI 2026)
☆ Hierarchical Decentralized Multi- Agent Coordination with Privacy-Preserving Knowledge Sharing: Extending Agent Net for Scalable Autonomous Systems
Goutham Nalagatla
Decentralized multi-agent systems have shown promise in enabling autonomous collaboration among LLM-based agents. While AgentNet demonstrated the feasibility of fully decentralized coordination through dynamic DAG topologies, several limitations remain: scalability challenges with large agent populations, communication overhead, lack of privacy guarantees, and suboptimal resource allocation. We propose AgentNet++, a hierarchical decentralized framework that extends AgentNet with multilevel agent organization, privacy-preserving knowledge sharing via differential privacy and secure aggregation, adaptive resource management, and theoretical convergence guarantees. Our approach introduces cluster-based hierarchies where agents self-organize into specialized groups, enabling efficient task routing and knowledge distillation while maintaining full decentralization. We provide formal analysis of convergence properties and privacy bounds, and demonstrate through extensive experiments on complex multi-agent tasks that AgentNet++ achieves 23% higher task completion rates, 40% reduction in communication overhead, and maintains strong privacy guarantees compared to AgentNet and other baselines. Our framework scales effectively to 1000+ agents while preserving the emergent intelligence properties of the original AgentNet.
comment : 6 pages, 4 figures
☆ Generalized Graph Transformer Variational Autoencoder
Siddhant Karki
Graph link prediction has long been a central problem in graph representation learning in both network analysis and generative modeling. Recent progress in deep learning has introduced increasingly sophisticated architectures for capturing relational dependencies within graph-structured data. In this work, we propose the Generalized Graph Transformer Variational Autoencoder (GGT-VAE). Our model integrates Generalized Graph Transformer Architecture with Variational Autoencoder framework for link prediction. Unlike prior GraphVAE, GCN, or GNN approaches, GGT-VAE leverages transformer style global self-attention mechanism along with laplacian positional encoding to model structural patterns across nodes into a latent space without relying on message passing. Experimental results on several benchmark datasets demonstrate that GGT-VAE consistently achieves above-baseline performance in terms of ROC-AUC and Average Precision. To the best of our knowledge, this is among the first studies to explore graph structure generation using a generalized graph transformer backbone in a variational framework.
☆ On the Holographic Geometry of Deterministic Computation
Logan Nye
Standard simulations of Turing machines suggest a linear relationship between the temporal duration $t$ of a run and the amount of information that must be stored by known simulations to certify, verify, or regenerate the configuration at time $t$. For deterministic multitape Turing machines over a fixed finite alphabet, this apparent linear dependence is not intrinsic: any length-$t$ run can be simulated in space $O(\sqrt{t})$ via a Height Compression Theorem for succinct computation trees together with an Algebraic Replay Engine. In this paper we recast that construction in geometric and information-theoretic language. We interpret the execution trace as a spacetime dependency DAG and exhibit a family of recursively defined holographic boundary summaries such that, along the square-root-space simulation, the total description length of all boundary data stored at any time is $O(\sqrt{t})$. Using Kolmogorov complexity, we prove that every internal configuration has constant conditional description complexity given the appropriate boundary summary and time index, establishing that the spacetime bulk carries no additional algorithmic information beyond its boundary. We express this as a one-dimensional computational area law: there exists a simulation in which the information capacity of the active "holographic screen'' needed to generate a spacetime region of volume $t$ is bounded by $O(\sqrt{t})$. In this precise sense, deterministic computation on a one-dimensional work tape admits a holographic representation, with the bulk history algebraically determined by data residing on a lower-dimensional boundary screen.
comment : 20 pages
☆ Agent OD RL : A Large Language Model-based Multi- agent System for OD RL Generation AAAI 2026
Wanle Zhong, Keman Huang, Xiaoyong Du
The Open Digital Rights Language (ODRL) is a pivotal standard for automating data rights management. However, the inherent logical complexity of authorization policies, combined with the scarcity of high-quality "Natural Language-to-ODRL" training datasets, impedes the ability of current methods to efficiently and accurately translate complex rules from natural language into the ODRL format. To address this challenge, this research leverages the potent comprehension and generation capabilities of Large Language Models (LLMs) to achieve both automation and high fidelity in this translation process. We introduce AgentODRL, a multi-agent system based on an Orchestrator-Workers architecture. The architecture consists of specialized Workers, including a Generator for ODRL policy creation, a Decomposer for breaking down complex use cases, and a Rewriter for simplifying nested logical relationships. The Orchestrator agent dynamically coordinates these Workers, assembling an optimal pathway based on the complexity of the input use case. Specifically, we enhance the ODRL Generator by incorporating a validator-based syntax strategy and a semantic reflection mechanism powered by a LoRA-finetuned model, significantly elevating the quality of the generated policies. Extensive experiments were conducted on a newly constructed dataset comprising 770 use cases of varying complexity, all situated within the context of data spaces. The results, evaluated using ODRL syntax and semantic scores, demonstrate that our proposed Orchestrator-Workers system, enhanced with these strategies, achieves superior performance on the ODRL generation task.
comment : Accepted by AAAI 2026. 9 pages, 1 figure
☆ Clinical-R1: Empowering Large Language Models for Faithful and Comprehensive Reasoning with Clinical Objective Relative Policy Optimization
Boyang Gu, Hongjian Zhou, Bradley Max Segal, Jinge Wu, Zeyu Cao, Hantao Zhong, Lei Clifton, Fenglin Liu, David A. Clifton
Recent advances in large language models (LLMs) have shown strong reasoning capabilities through large-scale pretraining and post-training reinforcement learning, demonstrated by DeepSeek-R1. However, current post-training methods, such as Grouped Relative Policy Optimization (GRPO), mainly reward correctness, which is not aligned with the multi-dimensional objectives required in high-stakes fields such as medicine, where reasoning must also be faithful and comprehensive. We introduce Clinical-Objective Relative Policy Optimization (CRPO), a scalable, multi-objective, verifiable reinforcement learning method designed to align LLM post-training with clinical reasoning principles. CRPO integrates rule-based and verifiable reward signals that jointly optimize accuracy, faithfulness, and comprehensiveness without relying on human annotation. To demonstrate its effectiveness, we train Clinical-R1-3B, a 3B-parameter model for clinical reasoning. The experiments on three benchmarks demonstrate that our CRPO substantially improves reasoning on truthfulness and completeness over standard GRPO while maintaining comfortable accuracy enhancements. This framework provides a scalable pathway to align LLM reasoning with clinical objectives, enabling safer and more collaborative AI systems for healthcare while also highlighting the potential of multi-objective, verifiable RL methods in post-training scaling of LLMs for medical domains.
☆ Developing Fairness-Aware Task Decomposition to Improve Equity in Post-Spinal Fusion Complication Prediction
Yining Yuan, J. Ben Tamo, Wenqi Shi, Yishan Zhong, Micky C. Nnamdi, B. Randall Brenn, Steven W. Hwang, May D. Wang
Fairness in clinical prediction models remains a persistent challenge, particularly in high-stakes applications such as spinal fusion surgery for scoliosis, where patient outcomes exhibit substantial heterogeneity. Many existing fairness approaches rely on coarse demographic adjustments or post-hoc corrections, which fail to capture the latent structure of clinical populations and may unintentionally reinforce bias. We propose FAIR-MTL, a fairness-aware multitask learning framework designed to provide equitable and fine-grained prediction of postoperative complication severity. Instead of relying on explicit sensitive attributes during model training, FAIR-MTL employs a data-driven subgroup inference mechanism. We extract a compact demographic embedding, and apply k-means clustering to uncover latent patient subgroups that may be differentially affected by traditional models. These inferred subgroup labels determine task routing within a shared multitask architecture. During training, subgroup imbalance is mitigated through inverse-frequency weighting, and regularization prevents overfitting to smaller groups. Applied to postoperative complication prediction with four severity levels, FAIR-MTL achieves an AUC of 0.86 and an accuracy of 75%, outperforming single-task baselines while substantially reducing bias. For gender, the demographic parity difference decreases to 0.055 and equalized odds to 0.094; for age, these values reduce to 0.056 and 0.148, respectively. Model interpretability is ensured through SHAP and Gini importance analyses, which consistently highlight clinically meaningful predictors such as hemoglobin, hematocrit, and patient weight. Our findings show that incorporating unsupervised subgroup discovery into a multitask framework enables more equitable, interpretable, and clinically actionable predictions for surgical risk stratification.
☆ DLRREC: Denoising Latent Representations via Multi-Modal Knowledge Fusion in Deep Recommender Systems
Jiahao Tian, Zhenkai Wang
Modern recommender systems struggle to effectively utilize the rich, yet high-dimensional and noisy, multi-modal features generated by Large Language Models (LLMs). Treating these features as static inputs decouples them from the core recommendation task. We address this limitation with a novel framework built on a key insight: deeply fusing multi-modal and collaborative knowledge for representation denoising. Our unified architecture introduces two primary technical innovations. First, we integrate dimensionality reduction directly into the recommendation model, enabling end-to-end co-training that makes the reduction process aware of the final ranking objective. Second, we introduce a contrastive learning objective that explicitly incorporates the collaborative filtering signal into the latent space. This synergistic process refines raw LLM embeddings, filtering noise while amplifying task-relevant signals. Extensive experiments confirm our method's superior discriminative power, proving that this integrated fusion and denoising strategy is critical for achieving state-of-the-art performance. Our work provides a foundational paradigm for effectively harnessing LLMs in recommender systems.
☆ IslandRun: Privacy-Aware Multi-Objective Orchestration for Distributed AI Inference
Bala Siva Sai Akhil Malepati
Modern AI inference faces an irreducible tension: no single computational resource simultaneously maximizes performance, preserves privacy, minimizes cost, and maintains trust. Existing orchestration frameworks optimize single dimensions (Kubernetes prioritizes latency, federated learning preserves privacy, edge computing reduces network distance), creating solutions that struggle under real-world heterogeneity. We present IslandRun, a multi-objective orchestration system that treats computational resources as autonomous "islands" spanning personal devices, private edge servers, and public cloud. Our key insights: (1) request-level heterogeneity demands policy-constrained multi-objective optimization, (2) data locality enables routing compute to data rather than data to compute, and (3) typed placeholder sanitization preserves context semantics across trust boundaries. IslandRun introduces agent-based routing, tiered island groups with differential trust, and reversible anonymization. This establishes a new paradigm for privacy-aware, decentralized inference orchestration across heterogeneous personal computing ecosystems.
comment : 15 pages, 3 figures, 2 tables
☆ Wikontic: Constructing Wikidata-Aligned, Ontology-Aware Knowledge Graphs with Large Language Models
Alla Chepurova, Aydar Bulatov, Yuri Kuratov, Mikhail Burtsev
Knowledge graphs (KGs) provide structured, verifiable grounding for large language models (LLMs), but current LLM-based systems commonly use KGs as auxiliary structures for text retrieval, leaving their intrinsic quality underexplored. In this work, we propose Wikontic, a multi-stage pipeline that constructs KGs from open-domain text by extracting candidate triplets with qualifiers, enforcing Wikidata-based type and relation constraints, and normalizing entities to reduce duplication. The resulting KGs are compact, ontology-consistent, and well-connected; on MuSiQue, the correct answer entity appears in 96% of generated triplets. On HotpotQA, our triplets-only setup achieves 76.0 F1, and on MuSiQue 59.8 F1, matching or surpassing several retrieval-augmented generation baselines that still require textual context. In addition, Wikontic attains state-of-the-art information-retention performance on the MINE-1 benchmark (86%), outperforming prior KG construction methods. Wikontic is also efficient at build time: KG construction uses less than 1,000 output tokens, about 3$\times$ fewer than AriGraph and $<$1/20 of GraphRAG. The proposed pipeline enhances the quality of the generated KG and offers a scalable solution for leveraging structured knowledge in LLMs.
☆ Slovak Conceptual Dictionary
Miroslav Blšták
When solving tasks in the field of natural language processing, we sometimes need dictionary tools, such as lexicons, word form dictionaries or knowledge bases. However, the availability of dictionary data is insufficient in many languages, especially in the case of low resourced languages. In this article, we introduce a new conceptual dictionary for the Slovak language as the first linguistic tool of this kind. Since Slovak language is a language with limited linguistic resources and there are currently not available any machine-readable linguistic data sources with a sufficiently large volume of data, many tasks which require automated processing of Slovak text achieve weaker results compared to other languages and are almost impossible to solve.
comment : 7 pages, 2 figures
☆ Integrating Skeleton Based Representations for Robust Yoga Pose Classification Using Deep Learning Models
Mohammed Mohiuddin, Syed Mohammod Minhaz Hossain, Sumaiya Khanam, Prionkar Barua, Aparup Barua, MD Tamim Hossain
Yoga is a popular form of exercise worldwide due to its spiritual and physical health benefits, but incorrect postures can lead to injuries. Automated yoga pose classification has therefore gained importance to reduce reliance on expert practitioners. While human pose keypoint extraction models have shown high potential in action recognition, systematic benchmarking for yoga pose recognition remains limited, as prior works often focus solely on raw images or a single pose extraction model. In this study, we introduce a curated dataset, 'Yoga-16', which addresses limitations of existing datasets, and systematically evaluate three deep learning architectures (VGG16, ResNet50, and Xception) using three input modalities (direct images, MediaPipe Pose skeleton images, and YOLOv8 Pose skeleton images). Our experiments demonstrate that skeleton-based representations outperform raw image inputs, with the highest accuracy of 96.09% achieved by VGG16 with MediaPipe Pose skeleton input. Additionally, we provide interpretability analysis using Grad-CAM, offering insights into model decision-making for yoga pose classification with cross validation analysis.
☆ Enhancing Analogy-Based Software Effort Estimation with Firefly Algorithm Optimization
Tarun Chintada, Uday Kiran Cheera
Analogy-Based Estimation (ABE) is a popular method for non-algorithmic estimation due to its simplicity and effectiveness. The Analogy-Based Estimation (ABE) model was proposed by researchers, however, no optimal approach for reliable estimation was developed. Achieving high accuracy in the ABE might be challenging for new software projects that differ from previous initiatives. This study (conducted in June 2024) proposes a Firefly Algorithm-guided Analogy-Based Estimation (FAABE) model that combines FA with ABE to improve estimation accuracy. The FAABE model was tested on five publicly accessible datasets: Cocomo81, Desharnais, China, Albrecht, Kemerer and Maxwell. To improve prediction efficiency, feature selection was used. The results were measured using a variety of evaluation metrics; various error measures include MMRE, MAE, MSE, and RMSE. Compared to conventional models, the experimental results show notable increases in prediction precision, demonstrating the efficacy of the Firefly-Analogy ensemble.
comment : 12 pages, 3 figures, 2 tables. Research conducted in June 2024
☆ Describe Anything Anywhere At Any Moment
Nicolas Gorlo, Lukas Schmid, Luca Carlone
Computer vision and robotics applications ranging from augmented reality to robot autonomy in large-scale environments require spatio-temporal memory frameworks that capture both geometric structure for accurate language-grounding as well as semantic detail. Existing methods face a tradeoff, where producing rich open-vocabulary descriptions comes at the expense of real-time performance when these descriptions have to be grounded in 3D. To address these challenges, we propose Describe Anything, Anywhere, at Any Moment (DAAAM), a novel spatio-temporal memory framework for large-scale and real-time 4D scene understanding. DAAAM introduces a novel optimization-based frontend to infer detailed semantic descriptions from localized captioning models, such as the Describe Anything Model (DAM), leveraging batch processing to speed up inference by an order of magnitude for online processing. It leverages such semantic understanding to build a hierarchical 4D scene graph (SG), which acts as an effective globally spatially and temporally consistent memory representation. DAAAM constructs 4D SGs with detailed, geometrically grounded descriptions while maintaining real-time performance. We show that DAAAM's 4D SG interfaces well with a tool-calling agent for inference and reasoning. We thoroughly evaluate DAAAM in the complex task of spatio-temporal question answering on the NaVQA benchmark and show its generalization capabilities for sequential task grounding on the SG3D benchmark. We further curate an extended OC-NaVQA benchmark for large-scale and long-time evaluations. DAAAM achieves state-of-the-art results in both tasks, improving OC-NaVQA question accuracy by 53.6%, position errors by 21.9%, temporal errors by 21.6%, and SG3D task grounding accuracy by 27.8% over the most competitive baselines, respectively. We release our data and code open-source.
comment : 14 pages, 5 figures, 6 tables
☆ Explainable Multi-Modal Deep Learning for Automatic Detection of Lung Diseases from Respiratory Audio Signals
S M Asiful Islam Saky, Md Rashidul Islam, Md Saiful Arefin, Shahaba Alam
Respiratory diseases remain major global health challenges, and traditional auscultation is often limited by subjectivity, environmental noise, and inter-clinician variability. This study presents an explainable multimodal deep learning framework for automatic lung-disease detection using respiratory audio signals. The proposed system integrates two complementary representations: a spectral-temporal encoder based on a CNN-BiLSTM Attention architecture, and a handcrafted acoustic-feature encoder capturing physiologically meaningful descriptors such as MFCCs, spectral centroid, spectral bandwidth, and zero-crossing rate. These branches are combined through late-stage fusion to leverage both data-driven learning and domain-informed acoustic cues. The model is trained and evaluated on the Asthma Detection Dataset Version 2 using rigorous preprocessing, including resampling, normalization, noise filtering, data augmentation, and patient-level stratified partitioning. The study achieved strong generalization with 91.21% accuracy, 0.899 macro F1-score, and 0.9866 macro ROC-AUC, outperforming all ablated variants. An ablation study confirms the importance of temporal modeling, attention mechanisms, and multimodal fusion. The framework incorporates Grad-CAM, Integrated Gradients, and SHAP, generating interpretable spectral, temporal, and feature-level explanations aligned with known acoustic biomarkers to build clinical transparency. The findings demonstrate the framework's potential for telemedicine, point-of-care diagnostics, and real-world respiratory screening.
☆ List Replicable Reinforcement Learning
Bohan Zhang, Michael Chen, A. Pavan, N. V. Vinodchandran, Lin F. Yang, Ruosong Wang
Replicability is a fundamental challenge in reinforcement learning (RL), as RL algorithms are empirically observed to be unstable and sensitive to variations in training conditions. To formally address this issue, we study \emph{list replicability} in the Probably Approximately Correct (PAC) RL framework, where an algorithm must return a near-optimal policy that lies in a \emph{small list} of policies across different runs, with high probability. The size of this list defines the \emph{list complexity}. We introduce both weak and strong forms of list replicability: the weak form ensures that the final learned policy belongs to a small list, while the strong form further requires that the entire sequence of executed policies remains constrained. These objectives are challenging, as existing RL algorithms exhibit exponential list complexity due to their instability. Our main theoretical contribution is a provably efficient tabular RL algorithm that guarantees list replicability by ensuring the list complexity remains polynomial in the number of states, actions, and the horizon length. We further extend our techniques to achieve strong list replicability, bounding the number of possible policy execution traces polynomially with high probability. Our theoretical result is made possible by key innovations including (i) a novel planning strategy that selects actions based on lexicographic order among near-optimal choices within a randomly chosen tolerance threshold, and (ii) a mechanism for testing state reachability in stochastic environments while preserving replicability. Finally, we demonstrate that our theoretical investigation sheds light on resolving the \emph{instability} issue of RL algorithms used in practice. In particular, we show that empirically, our new planning strategy can be incorporated into practical RL frameworks to enhance their stability.
☆ G-KV: Decoding-Time KV Cache Eviction with Global Attention
Mengqi Liao, Lu Wang, Chaoyun Zhang, Zekai Shen, Xiaowei Mao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Huaiyu Wan
Recent reasoning large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths. KV cache compression has emerged as an effective approach to greatly enhance the efficiency of reasoning. However, existing methods often focus on prompt compression or token eviction with local attention score, overlooking the long-term importance of tokens. We propose G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance. Additionally, we introduce post-training techniques, including reinforcement learning and distillation, to optimize models for compressed KV cache settings. The code of this paper is available on: https://github.com/microsoft/G-KV.
☆ ESPO: Entropy Importance Sampling Policy Optimization
Yuepeng Sheng, Yuwei Huang, Shuman Liu, Haibo Zhang, Anxiang Zeng
Large language model (LLM) reinforcement learning has increasingly relied on group-based policy optimization frameworks, such as GRPO and GSPO, to achieve stable fine-tuning at scale. However, a fundamental trade-off persists between optimization granularity and training stability. While GSPO improves robustness via sequence-level optimization, its monolithic treatment of sequences introduces severe inefficiencies: its conservative clipping mechanism indiscriminately discards valid training samples-a phenomenon we term gradient underutilization-and its uniform credit assignment fails to capture the heterogeneous contributions of critical reasoning steps. In this work, we propose Entropy Importance Sampling Policy Optimization (ESPO), a novel framework that reconciles fine-grained control with training stability. ESPO decomposes sequences into groups based on predictive entropy, enabling (1) Entropy-driven Importance Sampling to capture intra-sequence heterogeneity, and (2) Entropy-adaptive Clipping to dynamically allocate trust regions based on model uncertainty. Extensive experiments on mathematical reasoning benchmarks demonstrate that ESPO not only accelerates convergence but also achieves state-of-the-art performance, notably improving accuracy on the challenging HMMT benchmark from 4.4% to 13.13%.
☆ CACARA: Cross-Modal Alignment Leveraging a Text-Centric Approach for Cost-Effective Multimodal and Multilingual Learning
Diego A. B. Moreira, Alef I. Ferreira, Jhessica Silva, Gabriel O. dos Santos, Gustavo Bonil, João Gondim, Marina dos Santos, Helena Maia, Simone Hashiguti, Nádia da Silva, Carolina Scarton, Helio Pedrini, Sandra Avila
As deep learning models evolve, new applications and challenges are rapidly emerging. Tasks that once relied on a single modality, such as text, images, or audio, are now enriched by seamless interactions between multimodal data. These connections bridge information gaps: an image can visually materialize a text, while audio can add context to an image. Researchers have developed numerous multimodal models, but most rely on resource-intensive training across multiple modalities. Similarly, extending these models to new languages often follows the same resource-heavy training strategy. In this work, we propose a multimodal and multilingual architecture, CACARA, trained through emergent alignment learning, enabling the seamless integration of new modalities into an existing bimodal/multimodal model without requiring full retraining. This work breaks new ground by demonstrating that this emergent alignment paradigm can unlock multilingual capabilities from monolingual training. By fine-tuning the newly incorporated modality only on data aligned with the English language, our model develops support for over 100 languages without explicit multilingual pretraining or tuning of the text encoder. Such emergent multimodal and multilingual properties are gained efficiently, preserving previously learned knowledge at a training cost comparable to that of a monolingual model. Our strategy achieves up to a 14.24 percentage points improvement in R@1 audio-to-text retrieval, outperforming state-of-the-art multimodal models -- all without the heavy computational cost of retraining across every modality and language.
comment : 25 pages, 12 tables, 5 figures
☆ Mind the data gap: Missingness Still Shapes Large Language Model Prognoses ML4H 2025
Yuta Kobayashi, Vincent Jeanselme, Shalmali Joshi
Data collection often reflects human decisions. In healthcare, for instance, a referral for a diagnostic test is influenced by the patient's health, their preferences, available resources, and the practitioner's recommendations. Despite the extensive literature on the informativeness of missingness, its implications on the performance of Large Language Models (LLMs) have not been studied. Through a series of experiments on data from Columbia University Medical Center, a large urban academic medical center, and MIMIC-IV, we demonstrate that patterns of missingness significantly impact zero-shot predictive performance. Notably, the explicit inclusion of missingness indicators at prompting benefits some while hurting other LLMs' zero-shot predictive performance and calibration, suggesting an inconsistent impact. The proposed aggregated analysis and theoretical insights suggest that larger models benefit from these interventions, while smaller models can be negatively impacted. The LLM paradigm risks obscuring the impact of missingness, often neglected even in conventional ML, even further. We conclude that there is a need for more transparent accounting and systematic evaluation of the impact of representing (informative) missingness on downstream performance.
comment : Presented at ML4H 2025 - Findings Track
☆ RealGen: Photorealistic Text-to-Image Generation via Detector-Guided Rewards
Junyan Ye, Leiqi Zhu, Yuncheng Guo, Dongzhi Jiang, Zilong Huang, Yifan Zhang, Zhiyuan Yan, Haohuan Fu, Conghui He, Weijia Li
With the continuous advancement of image generation technology, advanced models such as GPT-Image-1 and Qwen-Image have achieved remarkable text-to-image consistency and world knowledge However, these models still fall short in photorealistic image generation. Even on simple T2I tasks, they tend to produce " fake" images with distinct AI artifacts, often characterized by "overly smooth skin" and "oily facial sheens". To recapture the original goal of "indistinguishable-from-reality" generation, we propose RealGen, a photorealistic text-to-image framework. RealGen integrates an LLM component for prompt optimization and a diffusion model for realistic image generation. Inspired by adversarial generation, RealGen introduces a "Detector Reward" mechanism, which quantifies artifacts and assesses realism using both semantic-level and feature-level synthetic image detectors. We leverage this reward signal with the GRPO algorithm to optimize the entire generation pipeline, significantly enhancing image realism and detail. Furthermore, we propose RealBench, an automated evaluation benchmark employing Detector-Scoring and Arena-Scoring. It enables human-free photorealism assessment, yielding results that are more accurate and aligned with real user experience. Experiments demonstrate that RealGen significantly outperforms general models like GPT-Image-1 and Qwen-Image, as well as specialized photorealistic models like FLUX-Krea, in terms of realism, detail, and aesthetics. The code is available at https://github.com/yejy53/RealGen.
☆ FairMT: Fairness for Heterogeneous Multi-Task Learning
Guanyu Hu, Tangzheng Lian, Na Yan, Dimitrios Kollias, Xinyu Yang, Oya Celiktutan, Siyang Song, Zeyu Fu
Fairness in machine learning has been extensively studied in single-task settings, while fair multi-task learning (MTL), especially with heterogeneous tasks (classification, detection, regression) and partially missing labels, remains largely unexplored. Existing fairness methods are predominantly classification-oriented and fail to extend to continuous outputs, making a unified fairness objective difficult to formulate. Further, existing MTL optimization is structurally misaligned with fairness: constraining only the shared representation, allowing task heads to absorb bias and leading to uncontrolled task-specific disparities. Finally, most work treats fairness as a zero-sum trade-off with utility, enforcing symmetric constraints that achieve parity by degrading well-served groups. We introduce FairMT, a unified fairness-aware MTL framework that accommodates all three task types under incomplete supervision. At its core is an Asymmetric Heterogeneous Fairness Constraint Aggregation mechanism, which consolidates task-dependent asymmetric violations into a unified fairness constraint. Utility and fairness are jointly optimized via a primal--dual formulation, while a head-aware multi-objective optimization proxy provides a tractable descent geometry that explicitly accounts for head-induced anisotropy. Across three homogeneous and heterogeneous MTL benchmarks encompassing diverse modalities and supervision regimes, FairMT consistently achieves substantial fairness gains while maintaining superior task utility. Code will be released upon paper acceptance.
☆ SCALE: Selective Resource Allocation for Overcoming Performance Bottlenecks in Mathematical Test-time Scaling AAAI 2026
Yang Xiao, Chunpu Xu, Ruifeng Yuan, Jiashuo Wang, Wenjie Li, Pengfei Liu
Test-time compute scaling has emerged as a powerful paradigm for enhancing mathematical reasoning in large language models (LLMs) by allocating additional computational resources during inference. However, current methods employ uniform resource distribution across all reasoning sub-problems, creating fundamental bottlenecks where challenging sub-problems receive insufficient attention while routine operations consume disproportionate resources. This uniform allocation creates performance bottlenecks where additional computational resources yield diminishing returns. Inspired by dual-process theory, we propose \textbf{SCALE} (Selective Resource Allocation), a framework that selectively allocates computational resources based on sub-problem difficulty. SCALE operates through four stages: (1) problem decomposition into sequential reasoning sub-problems, (2) difficulty assessment of each sub-problem to distinguish between routine operations and computationally challenging sub-problems, (3) selective processing mode assignment between System 1 for simple sub-problems and System 2 for complex ones, and (4) sequential execution with context propagation. By concentrating resources on challenging sub-problems while processing routine operations efficiently, SCALE achieves substantial performance improvements with superior resource utilization. Extensive experiments demonstrate that SCALE significantly outperforms uniform scaling baselines, achieving accuracy improvements of up to 13.75 percentage points (57.50% to 71.25% on AIME25) while reducing computational costs by 33%-53%, representing a major advance in test-time scaling that addresses fundamental limitations of current approaches.
comment : accepted by AAAI 2026
☆ CausalAffect: Causal Discovery for Facial Affective Understanding
Guanyu Hu, Tangzheng Lian, Dimitrios Kollias, Oya Celiktutan, Xinyu Yang
Understanding human affect from facial behavior requires not only accurate recognition but also structured reasoning over the latent dependencies that drive muscle activations and their expressive outcomes. Although Action Units (AUs) have long served as the foundation of affective computing, existing approaches rarely address how to infer psychologically plausible causal relations between AUs and expressions directly from data. We propose CausalAffect, the first framework for causal graph discovery in facial affect analysis. CausalAffect models AU-AU and AU-Expression dependencies through a two-level polarity and direction aware causal hierarchy that integrates population-level regularities with sample-adaptive structures. A feature-level counterfactual intervention mechanism further enforces true causal effects while suppressing spurious correlations. Crucially, our approach requires neither jointly annotated datasets nor handcrafted causal priors, yet it recovers causal structures consistent with established psychological theories while revealing novel inhibitory and previously uncharacterized dependencies. Extensive experiments across six benchmarks demonstrate that CausalAffect advances the state of the art in both AU detection and expression recognition, establishing a principled connection between causal discovery and interpretable facial behavior. All trained models and source code will be released upon acceptance.
☆ Sample-Efficient Expert Query Control in Active Imitation Learning via Conformal Prediction
Arad Firouzkouhi, Omid Mirzaeedodangeh, Lars Lindemann
Active imitation learning (AIL) combats covariate shift by querying an expert during training. However, expert action labeling often dominates the cost, especially in GPU-intensive simulators, human-in-the-loop settings, and robot fleets that revisit near-duplicate states. We present Conformalized Rejection Sampling for Active Imitation Learning (CRSAIL), a querying rule that requests an expert action only when the visited state is under-represented in the expert-labeled dataset. CRSAIL scores state novelty by the distance to the $K$-th nearest expert state and sets a single global threshold via conformal prediction. This threshold is the empirical $(1-α)$ quantile of on-policy calibration scores, providing a distribution-free calibration rule that links $α$ to the expected query rate and makes $α$ a task-agnostic tuning knob. This state-space querying strategy is robust to outliers and, unlike safety-gate-based AIL, can be run without real-time expert takeovers: we roll out full trajectories (episodes) with the learner and only afterward query the expert on a subset of visited states. Evaluated on MuJoCo robotics tasks, CRSAIL matches or exceeds expert-level reward while reducing total expert queries by up to 96% vs. DAgger and up to 65% vs. prior AIL methods, with empirical robustness to $α$ and $K$, easing deployment on novel systems with unknown dynamics.
☆ RecruitView: A Multimodal Dataset for Predicting Personality and Interview Performance for Human Resources Applications
Amit Kumar Gupta, Farhan Sheth, Hammad Shaikh, Dheeraj Kumar, Angkul Puniya, Deepak Panwar, Sandeep Chaurasia, Priya Mathur
Automated personality and soft skill assessment from multimodal behavioral data remains challenging due to limited datasets and methods that fail to capture geometric structure inherent in human traits. We introduce RecruitView, a dataset of 2,011 naturalistic video interview clips from 300+ participants with 27,000 pairwise comparative judgments across 12 dimensions: Big Five personality traits, overall personality score, and six interview performance metrics. To leverage this data, we propose Cross-Modal Regression with Manifold Fusion (CRMF), a geometric deep learning framework that explicitly models behavioral representations across hyperbolic, spherical, and Euclidean manifolds. CRMF employs geometry-specific expert networks to capture hierarchical trait structures, directional behavioral patterns, and continuous performance variations simultaneously. An adaptive routing mechanism dynamically weights expert contributions based on input characteristics. Through principled tangent space fusion, CRMF achieves superior performance while training 40-50% fewer trainable parameters than large multimodal models. Extensive experiments demonstrate that CRMF substantially outperforms the selected baselines, achieving up to 11.4% improvement in Spearman correlation and 6.0% in concordance index. Our RecruitView dataset is publicly available at https://huggingface.co/datasets/AI4A-lab/RecruitView
comment : 20 pages, 10 figures, 10 tables
☆ PEOAT: Personalization-Guided Evolutionary Question Assembly for One-Shot Adaptive Testing AAAI-2026
Xiaoshan Yu, Ziwei Huang, Shangshang Yang, Ziwen Wang, Haiping Ma, Xingyi Zhang
With the rapid advancement of intelligent education, Computerized Adaptive Testing (CAT) has attracted increasing attention by integrating educational psychology with deep learning technologies. Unlike traditional paper-and-pencil testing, CAT aims to efficiently and accurately assess examinee abilities by adaptively selecting the most suitable items during the assessment process. However, its real-time and sequential nature presents limitations in practical scenarios, particularly in large-scale assessments where interaction costs are high, or in sensitive domains such as psychological evaluations where minimizing noise and interference is essential. These challenges constrain the applicability of conventional CAT methods in time-sensitive or resourceconstrained environments. To this end, we first introduce a novel task called one-shot adaptive testing (OAT), which aims to select a fixed set of optimal items for each test-taker in a one-time selection. Meanwhile, we propose PEOAT, a Personalization-guided Evolutionary question assembly framework for One-shot Adaptive Testing from the perspective of combinatorial optimization. Specifically, we began by designing a personalization-aware initialization strategy that integrates differences between examinee ability and exercise difficulty, using multi-strategy sampling to construct a diverse and informative initial population. Building on this, we proposed a cognitive-enhanced evolutionary framework incorporating schema-preserving crossover and cognitively guided mutation to enable efficient exploration through informative signals. To maintain diversity without compromising fitness, we further introduced a diversity-aware environmental selection mechanism. The effectiveness of PEOAT is validated through extensive experiments on two datasets, complemented by case studies that uncovered valuable insights.
comment : AAAI-2026, 9 pages
☆ FR-TTS: Test-Time Scaling for NTP-based Image Generation with Effective Filling-based Reward Signal
Hang Xu, Linjiang Huang, Feng Zhao
Test-time scaling (TTS) has become a prevalent technique in image generation, significantly boosting output quality by expanding the number of parallel samples and filtering them using pre-trained reward models. However, applying this powerful methodology to the next-token prediction (NTP) paradigm remains challenging. The primary obstacle is the low correlation between the reward of an image decoded from an intermediate token sequence and the reward of the fully generated image. Consequently, these incomplete intermediate representations prove to be poor indicators for guiding the pruning direction, a limitation that stems from their inherent incompleteness in scale or semantic content. To effectively address this critical issue, we introduce the Filling-Based Reward (FR). This novel design estimates the approximate future trajectory of an intermediate sample by finding and applying a reasonable filling scheme to complete the sequence. Both the correlation coefficient between rewards of intermediate samples and final samples, as well as multiple intrinsic signals like token confidence, indicate that the FR provides an excellent and reliable metric for accurately evaluating the quality of intermediate samples. Building upon this foundation, we propose FR-TTS, a sophisticated scaling strategy. FR-TTS efficiently searches for good filling schemes and incorporates a diversity reward with a dynamic weighting schedule to achieve a balanced and comprehensive evaluation of intermediate samples. We experimentally validate the superiority of FR-TTS over multiple established benchmarks and various reward models. Code is available at \href{https://github.com/xuhang07/FR-TTS}{https://github.com/xuhang07/FR-TTS}.
☆ An Approach to Joint Hybrid Decision Making between Humans and Artificial Intelligence
Jonas D. Rockbach, Sven Fuchs, Maren Bennewitz
Due to the progress in artificial intelligence, it is important to understand how capable artificial agents should be used when interacting with humans, since high level authority and responsibility often remain with the human agent. However, integrated frameworks are lacking that can account for heterogeneous agents and draw on different scientific fields, such as human-factors engineering and artificial intelligence. Therefore, joint hybrid intelligence is described as a framework abstracting humans and artificial intelligence as decision making agents. A general definition of intelligence is provided on the basis of decision making competence being applicable to agents of different sorts. This framework is used for proposing the interrelated design space of joint hybrid intelligence being aimed at integrating the heterogeneous capabilities of humans and artificial intelligence. At the core of this design space lies joint agent engineering with the goal of integrating the design subspaces operator training, artificial intelligence engineering, and interface design via developing joint agent patterns. The ''extended swarming'' approach to human-swarm interaction is discussed as an example of such a pattern.
☆ Significant Other AI: Id entity , Memory, and Emotional Regulation as Long-Term Relation al Intelligence
Sung Park
Significant Others (SOs) stabilize identity, regulate emotion, and support narrative meaning-making, yet many people today lack access to such relational anchors. Recent advances in large language models and memory-augmented AI raise the question of whether artificial systems could support some of these functions. Existing empathic AIs, however, remain reactive and short-term, lacking autobiographical memory, identity modeling, predictive emotional regulation, and narrative coherence. This manuscript introduces Significant Other Artificial Intelligence (SO-AI) as a new domain of relational AI. It synthesizes psychological and sociological theory to define SO functions and derives requirements for SO-AI, including identity awareness, long-term memory, proactive support, narrative co-construction, and ethical boundary enforcement. A conceptual architecture is proposed, comprising an anthropomorphic interface, a relational cognition layer, and a governance layer. A research agenda outlines methods for evaluating identity stability, longitudinal interaction patterns, narrative development, and sociocultural impact. SO-AI reframes AI-human relationships as long-term, identity-bearing partnerships and provides a foundational blueprint for investigating whether AI can responsibly augment the relational stability many individuals lack today.
☆ Red Teaming Large Reasoning Models
Jiawei Chen, Yang Yang, Chao Yu, Yu Tian, Zhi Cao, Linghao Li, Hang Su, Zhaoxia Yin
Large Reasoning Models (LRMs) have emerged as a powerful advancement in multi-step reasoning tasks, offering enhanced transparency and logical consistency through explicit chains of thought (CoT). However, these models introduce novel safety and reliability risks, such as CoT-hijacking and prompt-induced inefficiencies, which are not fully captured by existing evaluation methods. To address this gap, we propose RT-LRM, a unified benchmark designed to assess the trustworthiness of LRMs. RT-LRM evaluates three core dimensions: truthfulness, safety and efficiency. Beyond metric-based evaluation, we further introduce the training paradigm as a key analytical perspective to investigate the systematic impact of different training strategies on model trustworthiness. We achieve this by designing a curated suite of 30 reasoning tasks from an observational standpoint. We conduct extensive experiments on 26 models and identify several valuable insights into the trustworthiness of LRMs. For example, LRMs generally face trustworthiness challenges and tend to be more fragile than Large Language Models (LLMs) when encountering reasoning-induced risks. These findings uncover previously underexplored vulnerabilities and highlight the need for more targeted evaluations. In addition, we release a scalable toolbox for standardized trustworthiness research to support future advancements in this important field. Our code and datasets will be open-sourced.
comment : 30 pages, 9 figures
☆ Balancing Efficiency and Fairness: An Iterative Exchange Framework for Multi-UAV Cooperative Path Planning
Hongzong Li, Luwei Liao, Xiangguang Dai, Yuming Feng, Rong Feng, Shiqin Tang
Multi-UAV cooperative path planning (MUCPP) is a fundamental problem in multi-agent systems, aiming to generate collision-free trajectories for a team of unmanned aerial vehicles (UAVs) to complete distributed tasks efficiently. A key challenge lies in achieving both efficiency, by minimizing total mission cost, and fairness, by balancing the workload among UAVs to avoid overburdening individual agents. This paper presents a novel Iterative Exchange Framework for MUCPP, balancing efficiency and fairness through iterative task exchanges and path refinements. The proposed framework formulates a composite objective that combines the total mission distance and the makespan, and iteratively improves the solution via local exchanges under feasibility and safety constraints. For each UAV, collision-free trajectories are generated using A* search over a terrain-aware configuration space. Comprehensive experiments on multiple terrain datasets demonstrate that the proposed method consistently achieves superior trade-offs between total distance and makespan compared to existing baselines.
☆ Low-Bitrate Video Compression through Semantic-Conditioned Diffusion
Lingdong Wang, Guan-Ming Su, Divya Kothandaraman, Tsung-Wei Huang, Mohammad Hajiesmaili, Ramesh K. Sitaraman
Traditional video codecs optimized for pixel fidelity collapse at ultra-low bitrates and produce severe artifacts. This failure arises from a fundamental misalignment between pixel accuracy and human perception. We propose a semantic video compression framework named DiSCo that transmits only the most meaningful information while relying on generative priors for detail synthesis. The source video is decomposed into three compact modalities: a textual description, a spatiotemporally degraded video, and optional sketches or poses that respectively capture semantic, appearance, and motion cues. A conditional video diffusion model then reconstructs high-quality, temporally coherent videos from these compact representations. Temporal forward filling, token interleaving, and modality-specific codecs are proposed to improve multimodal generation and modality compactness. Experiments show that our method outperforms baseline semantic and traditional codecs by 2-10X on perceptual metrics at low bitrates.
☆ GreenPlanner: Practical Floorplan Layout Generation via an Energy-Aware and Function-Feasible Generative Framework
Pengyu Zeng, Yuqin Dai, Jun Yin, Jing Zhong, Ziyang Han, Chaoyang Shi, ZhanXiang Jin, Maowei Jiang, Yuxing Han, Shuai Lu
Building design directly affects human well-being and carbon emissions, yet generating spatial-functional and energy-compliant floorplans remains manual, costly, and non-scalable. Existing methods produce visually plausible layouts but frequently violate key constraints, yielding invalid results due to the absence of automated evaluation. We present GreenPlanner, an energy- and functionality-aware generative framework that unifies design evaluation and generation. It consists of a labeled Design Feasibility Dataset for learning constraint priors; a fast Practical Design Evaluator (PDE) for predicting energy performance and spatial-functional validity; a Green Plan Dataset (GreenPD) derived from PDE-guided filtering to pair user requirements with regulation-compliant layouts; and a GreenFlow generator trained on GreenPD with PDE feedback for controllable, regulation-aware generation. Experiments show that GreenPlanner accelerates evaluation by over $10^{5}\times$ with $>$99% accuracy, eliminates invalid samples, and boosts design efficiency by 87% over professional architects.
comment : 11 pages, 6 figures
☆ SelfAI: Building a Self-Training AI System with LLM Agent s
Xiao Wu, Ting-Zhu Huang, Liang-Jian Deng, Xiaobing Yu, Yu Zhong, Shangqi Deng, Ufaq Khan, Jianghao Wu, Xiaofeng Liu, Imran Razzak, Xiaojun Chang, Yutong Xie
Recent work on autonomous scientific discovery has leveraged LLM-based agents to integrate problem specification, experiment planning, and execution into end-to-end systems. However, these frameworks are often confined to narrow application domains, offer limited real-time interaction with researchers, and lack principled mechanisms for determining when to halt exploration, resulting in inefficiencies, reproducibility challenges, and under-utilized human expertise. To address these gaps, we propose \textit{SelfAI}, a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations, a Cognitive Agent powered by LLMs with optimal stopping criteria to iteratively refine hyperparameter searches, and an Experiment Manager responsible for orchestrating parallel, fault-tolerant training workflows across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback. We further introduce two novel evaluation metrics, Score and $\text{AUP}_D$, to quantify discovery efficiency and search diversity. Across regression, NLP, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials compared to classical Bayesian optimization and LLM-based baselines, while enabling seamless interaction with human researchers.
☆ Time-Series at the Edge: Tiny Separable CNNs for Wearable Gait Detection and Optimal Sensor Placement
Andrea Procopio, Marco Esposito, Sara Raggiunto, Andrey Gizdov, Alberto Belli, Paola Pierleoni
We study on-device time-series analysis for gait detection in Parkinson's disease (PD) from short windows of triaxial acceleration, targeting resource-constrained wearables and edge nodes. We compare magnitude thresholding to three 1D CNNs for time-series analysis: a literature baseline (separable convolutions) and two ultra-light models - one purely separable and one with residual connections. Using the BioStampRC21 dataset, 2 s windows at 30 Hz, and subject-independent leave-one-subject-out (LOSO) validation on 16 PwPD with chest-worn IMUs, our residual separable model (Model 2, 533 params) attains PR-AUC = 94.5%, F1 = 91.2%, MCC = 89.4%, matching or surpassing the baseline (5,552 params; PR-AUC = 93.7%, F1 = 90.5%, MCC = 88.5%) with approximately 10x fewer parameters. The smallest model (Model 1, 305 params) reaches PR-AUC = 94.0%, F1 = 91.0%, MCC = 89.1%. Thresholding obtains high recall (89.0%) but low precision (76.5%), yielding many false positives and high inter-subject variance. Sensor-position analysis (train-on-all) shows chest and thighs are most reliable; forearms degrade precision/recall due to non-gait arm motion; naive fusion of all sites does not outperform the best single site. Both compact CNNs execute within tight memory/latency budgets on STM32-class MCUs (sub-10 ms on low-power boards), enabling on-sensor gating of transmission/storage. Overall, ultra-light separable CNNs provide a superior accuracy-efficiency-generalization trade-off to fixed thresholds for wearable PD gait detection and underscore the value of tailored time-series models for edge deployment.
☆ From Coefficients to Directions: Rethinking Model Merging with Directional Alignment
Zhikang Chen, Sen Cui, Deheng Ye, Min Zhang, Gang Niu, Yu Zhang, Masashi Sugiyama, Tingting Zhu
Model merging has emerged as a practical paradigm for integrating multiple independently trained models into a single model without joint retraining. Previous studies have demonstrated the effectiveness of combining parameters through strategies such as parameter decomposition, coefficient optimization, and subspace learning, significantly reducing the need for expensive joint training and achieving strong empirical performance across diverse tasks. However, these approaches predominantly treat merging as a problem of parameter space decomposition or fusion coefficient optimization, while overlooking the critical role of directional information in both parameter and feature spaces. In practice, naïve merging introduces inconsistencies in dominant parameter directions and disrupts structural coherence across models, which can degrade performance. Moreover, coefficient-based optimization methods implicitly assume compatible feature-space directions across models. However, Neural Collapse indicates that class features follow structured directional patterns, which may differ across independently trained models, making coefficient optimization alone insufficient. In this work, we emphasize the importance of \emph{directional alignment} and introduce a unified geometric framework, \emph{Merging with Directional Alignment} (\method{}), which aligns directional structures consistently in both the parameter and feature spaces. Our analysis shows that directional alignment improves structural coherence, and extensive experiments across benchmarks, model scales, and task configurations further validate the effectiveness of our approach.
☆ An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutines
Jianhai Su, Jinzhu Luo, Qi Zhang
We take the novel perspective of incorporating offline RL algorithms as subroutines of tabula rasa online RL. This is feasible because an online learning agent can repurpose its historical interactions as offline dataset. We formalize this idea into a framework that accommodates several variants of offline RL incorporation such as final policy recommendation and online fine-tuning. We further introduce convenient techniques to improve its effectiveness in enhancing online learning efficiency. Our extensive and systematic empirical analyses show that 1) the effectiveness of the proposed framework depends strongly on the nature of the task, 2) our proposed techniques greatly enhance its effectiveness, and 3) existing online fine-tuning methods are overall ineffective, calling for more research therein.
☆ Layer Probing Improves Kinase Functional Prediction with Protein Language Models
Ajit Kumar, IndraPrakash Jha
Protein language models (PLMs) have transformed sequence-based protein analysis, yet most applications rely only on final-layer embeddings, which may overlook biologically meaningful information encoded in earlier layers. We systematically evaluate all 33 layers of ESM-2 for kinase functional prediction using both unsupervised clustering and supervised classification. We show that mid-to-late transformer layers (layers 20-33) outperform the final layer by 32 percent in unsupervised Adjusted Rand Index and improve homology-aware supervised accuracy to 75.7 percent. Domain-level extraction, calibrated probability estimates, and a reproducible benchmarking pipeline further strengthen reliability. Our results demonstrate that transformer depth contains functionally distinct biological signals and that principled layer selection significantly improves kinase function prediction.
comment : 14 pages, 7 figures, 3 tables; includes code and dataset links
☆ Evaluating LLMs in Open-Source Games NeurIPS 2025
Swadesh Sistla, Max Kleiman-Weiner
Large Language Models' (LLMs) programming capabilities enable their participation in open-source games: a game-theoretic setting in which players submit computer programs in lieu of actions. These programs offer numerous advantages, including interpretability, inter-agent transparency, and formal verifiability; additionally, they enable program equilibria, solutions that leverage the transparency of code and are inaccessible within normal-form settings. We evaluate the capabilities of leading open- and closed-weight LLMs to predict and classify program strategies and evaluate features of the approximate program equilibria reached by LLM agents in dyadic and evolutionary settings. We identify the emergence of payoff-maximizing, cooperative, and deceptive strategies, characterize the adaptation of mechanisms within these programs over repeated open-source games, and analyze their comparative evolutionary fitness. We find that open-source games serve as a viable environment to study and steer the emergence of cooperative strategy in multi-agent dilemmas.
comment : 39th Conference on Neural Information Processing Systems (NeurIPS 2025)
☆ S^2-KD: Semantic-Spectral Knowledge Distillation Spatiotemporal Forecasting
Wenshuo Wang, Yaomin Shen, Yingjie Tan, Yihao Chen
Spatiotemporal forecasting often relies on computationally intensive models to capture complex dynamics. Knowledge distillation (KD) has emerged as a key technique for creating lightweight student models, with recent advances like frequency-aware KD successfully preserving spectral properties (i.e., high-frequency details and low-frequency trends). However, these methods are fundamentally constrained by operating on pixel-level signals, leaving them blind to the rich semantic and causal context behind the visual patterns. To overcome this limitation, we introduce S^2-KD, a novel framework that unifies Semantic priors with Spectral representations for distillation. Our approach begins by training a privileged, multimodal teacher model. This teacher leverages textual narratives from a Large Multimodal Model (LMM) to reason about the underlying causes of events, while its architecture simultaneously decouples spectral components in its latent space. The core of our framework is a new distillation objective that transfers this unified semantic-spectral knowledge into a lightweight, vision-only student. Consequently, the student learns to make predictions that are not only spectrally accurate but also semantically coherent, without requiring any textual input or architectural overhead at inference. Extensive experiments on benchmarks like WeatherBench and TaxiBJ+ show that S^2-KD significantly boosts the performance of simple student models, enabling them to outperform state-of-the-art methods, particularly in long-horizon and complex non-stationary scenarios.
☆ Towards aligned body representations in vision models
Andrey Gizdov, Andrea Procopio, Yichen Li, Daniel Harari, Tomer Ullman
Human physical reasoning relies on internal "body" representations - coarse, volumetric approximations that capture an object's extent and support intuitive predictions about motion and physics. While psychophysical evidence suggests humans use such coarse representations, their internal structure remains largely unknown. Here we test whether vision models trained for segmentation develop comparable representations. We adapt a psychophysical experiment conducted with 50 human participants to a semantic segmentation task and test a family of seven segmentation networks, varying in size. We find that smaller models naturally form human-like coarse body representations, whereas larger models tend toward overly detailed, fine-grain encodings. Our results demonstrate that coarse representations can emerge under limited computational resources, and that machine representations can provide a scalable path toward understanding the structure of physical reasoning in the brain.
comment : Andrea Procopio and Andrey Gizdov have equal contributions
☆ MedCondDiff: Lightweight, Robust, Semantically Guided Diffusion for Medical Image Segmentation
Ruirui Huang, Jiacheng Li
We introduce MedCondDiff, a diffusion-based framework for multi-organ medical image segmentation that is efficient and anatomically grounded. The model conditions the denoising process on semantic priors extracted by a Pyramid Vision Transformer (PVT) backbone, yielding a semantically guided and lightweight diffusion architecture. This design improves robustness while reducing both inference time and VRAM usage compared to conventional diffusion models. Experiments on multi-organ, multi-modality datasets demonstrate that MedCondDiff delivers competitive performance across anatomical regions and imaging modalities, underscoring the potential of semantically guided diffusion models as an effective class of architectures for medical imaging tasks.
☆ Debate with Images: Detecting Deceptive Behaviors in Multimodal Large Language Models
Sitong Fang, Shiyi Hou, Kaile Wang, Boyuan Chen, Donghai Hong, Jiayi Zhou, Josef Dai, Yaodong Yang, Jiaming Ji
Are frontier AI systems becoming more capable? Certainly. Yet such progress is not an unalloyed blessing but rather a Trojan horse: behind their performance leaps lie more insidious and destructive safety risks, namely deception. Unlike hallucination, which arises from insufficient capability and leads to mistakes, deception represents a deeper threat in which models deliberately mislead users through complex reasoning and insincere responses. As system capabilities advance, deceptive behaviours have spread from textual to multimodal settings, amplifying their potential harm. First and foremost, how can we monitor these covert multimodal deceptive behaviors? Nevertheless, current research remains almost entirely confined to text, leaving the deceptive risks of multimodal large language models unexplored. In this work, we systematically reveal and quantify multimodal deception risks, introducing MM-DeceptionBench, the first benchmark explicitly designed to evaluate multimodal deception. Covering six categories of deception, MM-DeceptionBench characterizes how models strategically manipulate and mislead through combined visual and textual modalities. On the other hand, multimodal deception evaluation is almost a blind spot in existing methods. Its stealth, compounded by visual-semantic ambiguity and the complexity of cross-modal reasoning, renders action monitoring and chain-of-thought monitoring largely ineffective. To tackle this challenge, we propose debate with images, a novel multi-agent debate monitor framework. By compelling models to ground their claims in visual evidence, this method substantially improves the detectability of deceptive strategies. Experiments show that it consistently increases agreement with human judgements across all tested models, boosting Cohen's kappa by 1.5x and accuracy by 1.25x on GPT-4o.
☆ Echo-N1: Affective RL Frontier
Naifan Zhang, Ruihan Sun, Ruixi Su, Shiqi Ma, Shiya Zhang, Xianna Weng, Xiaofan Zhang, Yuhan Zhan, Yuyang Xu, Zhaohan Chen, Zhengyuan Pan, Ziyi Song
The LLM field has spent a year perfecting RL for tasks machines already excel at, math, code, and deterministic reasoning, while completely sidestepping the domain that actually defines human intelligence: subjective, emotionally grounded, personality sensitive conversation. This space has often been regarded as inherently subjective and challenging to formalize, making it appear unsuitable for conventional RL pipelines. We show that it is not only possible and it is a solvable and transformative RL problem. We propose the first framework that infers user personality on the fly and optimizes model behavior toward personalized conversational preferences. Contrary to the widespread belief that RL collapses in non-verifiable settings, our method produces consistent, robust, and dramatic improvements in humanlike interaction quality. We also introduce the first dynamic emotional intelligence evaluation suite to quantify these gains. Our model, which is introduced as Echo-N1, behaves far above its base version and outperforming the proprietary Doubao 1.5 Character. This work establishes a new frontier for RL: optimizing models for the deeply subjective, deeply human dimensions of conversation.
☆ Assertion-Conditioned Compliance: A Provenance-Aware Vulnerability in Multi-Turn Tool-Calling Agent s
Daud Waqas, Aaryamaan Golthi, Erika Hayashida, Huanzhi Mao
Multi-turn tool-calling LLMs (models capable of invoking external APIs or tools across several user turns) have emerged as a key feature in modern AI assistants, enabling extended dialogues from benign tasks to critical business, medical, and financial operations. Yet implementing multi-turn pipelines remains difficult for many safety-critical industries due to ongoing concerns regarding model resilience. While standardized benchmarks such as the Berkeley Function-Calling Leaderboard (BFCL) have underpinned confidence concerning advanced function-calling models (like Salesforce's xLAM V2), there is still a lack of visibility into multi-turn conversation-level robustness, especially given their exposure to real-world systems. In this paper, we introduce Assertion-Conditioned Compliance (A-CC), a novel evaluation paradigm for multi-turn function-calling dialogues. A-CC provides holistic metrics that evaluate a model's behavior when confronted with misleading assertions originating from two distinct vectors: (1) user-sourced assertions (USAs), which measure sycophancy toward plausible but misinformed user beliefs, and (2) function-sourced assertions (FSAs), which measure compliance with plausible but contradictory system policies (e.g., stale hints from unmaintained tools). Our results show that models are highly vulnerable to both USA sycophancy and FSA policy conflicts, confirming A-CC as a critical, latent vulnerability in deployed agents.
comment : 15 pages (incl. Appendix), 2 figures, 7 tables
☆ CogEvo-Edu: Cognitive Evolution Educational Multi- Agent Collaborative System
Yefeng Wu, Yuchen Song, Yecheng Zhao, Ling Wu, Shan Wan
Large language models (LLMs) are increasingly deployed as conversational tutors in STEM education, yet most systems still rely on a single LLM with a static retrieval-augmented generation (RAG) pipeline over course materials. This design struggles in complex domains such as digital signal processing (DSP), where tutors must maintain coherent long-term student models, manage heterogeneous knowledge bases, and adapt teaching strategies over extended interactions. We argue that retrieval, memory, and control should be treated as a coupled cognitive evolution process. We instantiate this view in CogEvo-Edu, a hierarchical educational multi-agent system comprising a Cognitive Perception Layer (CPL), a Knowledge Evolution Layer (KEL), and a Meta-Control Layer (MCL). CPL maintains dual memories and performs confidence-weighted consolidation to build structured, self-correcting student profiles under limited context. KEL assigns each knowledge chunk a spatiotemporal value that drives activation, semantic compression, and forgetting. MCL formulates tutoring as hierarchical sequential decision making, orchestrating specialized agents and jointly adapting CPL/KEL hyperparameters via a dual inner--outer loop. To evaluate CogEvo-Edu, we construct DSP-EduBench, a vertical benchmark for DSP tutoring with heterogeneous resources, simulated student profiles, and long-horizon interaction scripts. Using a three-model LLM-as-a-Judge ensemble, CogEvo-Edu raises the overall score from 5.32 to 9.23 and improves all six indicators over static RAG, simple memory, and a single-agent variant, demonstrating the value of jointly evolving student profiles, knowledge bases, and teaching policies.
☆ Evidence-Guided Schema Normalization for Temporal Tabular Reasoning
Ashish Thanga, Vibhu Dixit, Abhilash Shankarampeta, Vivek Gupta
Temporal reasoning over evolving semi-structured tables poses a challenge to current QA systems. We propose a SQL-based approach that involves (1) generating a 3NF schema from Wikipedia infoboxes, (2) generating SQL queries, and (3) query execution. Our central finding challenges model scaling assumptions: the quality of schema design has a greater impact on QA precision than model capacity. We establish three evidence-based principles: normalization that preserves context, semantic naming that reduces ambiguity, and consistent temporal anchoring. Our best configuration (Gemini 2.5 Flash schema + Gemini-2.0-Flash queries) achieves 80.39 EM, a 16.8\% improvement over the baseline (68.89 EM).
☆ Progressive Code Integration for Abstractive Bug Report Summarization
Shaira Sadia Karim, Abrar Mahmud Rahim, Lamia Alam, Ishmam Tashdeed, Lutfun Nahar Lota, Md. Abu Raihan M. Kamal, Md. Azam Hossain
Bug reports are often unstructured and verbose, making it challenging for developers to efficiently comprehend software issues. Existing summarization approaches typically rely on surface-level textual cues, resulting in incomplete or redundant summaries, and they frequently ignore associated code snippets, which are essential for accurate defect diagnosis. To address these limitations, we propose a progressive code-integration framework for LLM-based abstractive bug report summarization. Our approach incrementally incorporates long code snippets alongside textual content, overcoming standard LLM context window constraints and producing semantically rich summaries. Evaluated on four benchmark datasets using eight LLMs, our pipeline outperforms extractive baselines by 7.5%-58.2% and achieves performance comparable to state-of-the-art abstractive methods, highlighting the benefits of jointly leveraging textual and code information for enhanced bug comprehension.
☆ Comparative Analysis of 47 Context -Based Question Answer Models Across 8 Diverse Dataset s
Muhammad Muneeb, David B. Ascher, Ahsan Baidar Bakht
Context-based question answering (CBQA) models provide more accurate and relevant answers by considering the contextual information. They effectively extract specific information given a context, making them functional in various applications involving user support, information retrieval, and educational platforms. In this manuscript, we benchmarked the performance of 47 CBQA models from Hugging Face on eight different datasets. This study aims to identify the best-performing model across diverse datasets without additional fine-tuning. It is valuable for practical applications where the need to retrain models for specific datasets is minimized, streamlining the implementation of these models in various contexts. The best-performing models were trained on the SQuAD v2 or SQuAD v1 datasets. The best-performing model was ahotrod/electra_large_discriminator_squad2_512, which yielded 43\% accuracy across all datasets. We observed that the computation time of all models depends on the context length and the model size. The model's performance usually decreases with an increase in the answer length. Moreover, the model's performance depends on the context complexity. We also used the Genetic algorithm to improve the overall accuracy by integrating responses from other models. ahotrod/electra_large_discriminator_squad2_512 generated the best results for bioasq10b-factoid (65.92\%), biomedical\_cpgQA (96.45\%), QuAC (11.13\%), and Question Answer Dataset (41.6\%). Bert-large-uncased-whole-word-masking-finetuned-squad achieved an accuracy of 82\% on the IELTS dataset.
☆ RL -Struct: A Lightweight Reinforcement Learning Framework for Reliable Structured Output in LLMs
Ruike Hu, Shulei Wu
Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language generation and reasoning. However, their integration into automated software ecosystems is often hindered by the "Structure Gap" - the inherent tension between the probabilistic nature of token generation and the deterministic requirements of structured data formats (e.g., JSON, XML). Traditional Supervised Fine-Tuning (SFT) often fails to enforce strict syntactic constraints, leading to "hallucinated" keys or malformed structures, while constrained decoding methods impose significant inference latency. In this paper, we propose a lightweight, efficient Reinforcement Learning (RL) framework to bridge this gap. We introduce a novel Multi-dimensional Reward Function that decomposes the structured output task into a hierarchy of constraints: structural integrity, format correctness, content accuracy, and validity. Leveraging Gradient Regularized Policy Optimization (GRPO), we enable the model to internalize these constraints without the need for a separate critic network, reducing peak VRAM usage by 40% compared to PPO. We validate our approach on multiple tasks, including complex recipe generation and structured math reasoning (GSM8K-JSON). Experimental results demonstrate that our method achieves 89.7% structural accuracy and 92.1% JSON validity, significantly outperforming both zero-shot baselines (e.g., GPT-3.5) and SFT on larger models like LLaMA-3-8B. Furthermore, we provide a detailed analysis of training dynamics, revealing a distinct self-paced curriculum where the model sequentially acquires syntactic proficiency before semantic accuracy. Our model is publicly available at https://huggingface.co/Freakz3z/Qwen-JSON.
comment : 23 pages, 14 figures. Model is available at https://huggingface.co/Freakz3z/Qwen-JSON
☆ Tracing Mathematical Proficiency Through Problem-Solving Processes
Jungyang Park, Suho Kang, Jaewoo Park, Jaehong Kim, Jaewoo Shin, Seonjoon Park, Youngjae Yu
Knowledge Tracing (KT) aims to model student's knowledge state and predict future performance to enable personalized learning in Intelligent Tutoring Systems. However, traditional KT methods face fundamental limitations in explainability, as they rely solely on the response correctness, neglecting the rich information embedded in students' problem-solving processes. To address this gap, we propose Knowledge Tracing Leveraging Problem-Solving Process (KT-PSP), which incorporates students' problem-solving processes to capture the multidimensional aspects of mathematical proficiency. We also introduce KT-PSP-25, a new dataset specifically designed for the KT-PSP. Building on this, we present StatusKT, a KT framework that employs a teacher-student-teacher three-stage LLM pipeline to extract students' MP as intermediate signals. In this pipeline, the teacher LLM first extracts problem-specific proficiency indicators, then a student LLM generates responses based on the student's solution process, and a teacher LLM evaluates these responses to determine mastery of each indicator. The experimental results on KT-PSP-25 demonstrate that StatusKT improves the prediction performance of existing KT methods. Moreover, StatusKT provides interpretable explanations for its predictions by explicitly modeling students' mathematical proficiency.
comment : 15 pages, 7 figures
☆ Adversarial Signed Graph Learning with Differential Privacy
Haobin Ke, Sen Zhang, Qingqing Ye, Xun Ran, Haibo Hu
Signed graphs with positive and negative edges can model complex relationships in social networks. Leveraging on balance theory that deduces edge signs from multi-hop node pairs, signed graph learning can generate node embeddings that preserve both structural and sign information. However, training on sensitive signed graphs raises significant privacy concerns, as model parameters may leak private link information. Existing protection methods with differential privacy (DP) typically rely on edge or gradient perturbation for unsigned graph protection. Yet, they are not well-suited for signed graphs, mainly because edge perturbation tends to cascading errors in edge sign inference under balance theory, while gradient perturbation increases sensitivity due to node interdependence and gradient polarity change caused by sign flips, resulting in larger noise injection. In this paper, motivated by the robustness of adversarial learning to noisy interactions, we present ASGL, a privacy-preserving adversarial signed graph learning method that preserves high utility while achieving node-level DP. We first decompose signed graphs into positive and negative subgraphs based on edge signs, and then design a gradient-perturbed adversarial module to approximate the true signed connectivity distribution. In particular, the gradient perturbation helps mitigate cascading errors, while the subgraph separation facilitates sensitivity reduction. Further, we devise a constrained breadth-first search tree strategy that fuses with balance theory to identify the edge signs between generated node pairs. This strategy also enables gradient decoupling, thereby effectively lowering gradient sensitivity. Extensive experiments on real-world datasets show that ASGL achieves favorable privacy-utility trade-offs across multiple downstream tasks.
☆ VCWo rl d: A Biological Wo rl d Model for Virtual Cell Simulation
Zhijian Wei, Runze Ma, Zichen Wang, Zhongmin Li, Shuotong Song, Shuangjia Zheng
Virtual cell modeling aims to predict cellular responses to perturbations. Existing virtual cell models rely heavily on large-scale single-cell datasets, learning explicit mappings between gene expression and perturbations. Although recent models attempt to incorporate multi-source biological information, their generalization remains constrained by data quality, coverage, and batch effects. More critically, these models often function as black boxes, offering predictions without interpretability or consistency with biological principles, which undermines their credibility in scientific research. To address these challenges, we present VCWorld, a cell-level white-box simulator that integrates structured biological knowledge with the iterative reasoning capabilities of large language models to instantiate a biological world model. VCWorld operates in a data-efficient manner to reproduce perturbation-induced signaling cascades and generates interpretable, stepwise predictions alongside explicit mechanistic hypotheses. In drug perturbation benchmarks, VCWorld achieves state-of-the-art predictive performance, and the inferred mechanistic pathways are consistent with publicly available biological evidence.
☆ ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoning ICCV 2025
Zhengzhuo Xu, SiNan Du, Yiyan Qi, SiwenLu, Chengjin Xu, Chun Yuan, Jian Guo
Multimodal Large Language Models (MLLMs) have emerged as powerful tools for chart comprehension. However, they heavily rely on extracted content via OCR, which leads to numerical hallucinations when chart textual annotations are sparse. While existing methods focus on scaling instructions, they fail to address the fundamental challenge, i.e., reasoning with visual perception. In this paper, we identify a critical observation: MLLMs exhibit weak grounding in chart elements and proportional relationships, as evidenced by their inability to localize key positions to match their reasoning. To bridge this gap, we propose PointCoT, which integrates reflective interaction into chain-of-thought reasoning in charts. By prompting MLLMs to generate bounding boxes and re-render charts based on location annotations, we establish connections between textual reasoning steps and visual grounding regions. We further introduce an automated pipeline to construct ChartPoint-SFT-62k, a dataset featuring 19.2K high-quality chart samples with step-by-step CoT, bounding box, and re-rendered visualizations. Leveraging this data, we develop two instruction-tuned models, ChartPointQ2 and ChartPointQ2.5, which outperform state-of-the-art across several chart benchmarks, e.g., +5.04\% on ChartBench.
comment : Accepted by ICCV 2025
☆ Gradient Inversion in Federated Reinforcement Learning
Shenghong He
Federated reinforcement learning (FRL) enables distributed learning of optimal policies while preserving local data privacy through gradient sharing.However, FRL faces the risk of data privacy leaks, where attackers exploit shared gradients to reconstruct local training data.Compared to traditional supervised federated learning, successful reconstruction in FRL requires the generated data not only to match the shared gradients but also to align with real transition dynamics of the environment (i.e., aligning with the real data transition distribution).To address this issue, we propose a novel attack method called Regularization Gradient Inversion Attack (RGIA), which enforces prior-knowledge-based regularization on states, rewards, and transition dynamics during the optimization process to ensure that the reconstructed data remain close to the true transition distribution.Theoretically, we prove that the prior-knowledge-based regularization term narrows the solution space from a broad set containing spurious solutions to a constrained subset that satisfies both gradient matching and true transition dynamics.Extensive experiments on control tasks and autonomous driving tasks demonstrate that RGIA can effectively constrain reconstructed data transition distributions and thus successfully reconstruct local private data.
☆ Words into Wo rl d: A Task-Adaptive Agent for Language-Guided Spatial Retrieval in AR
Lixing Guo, Tobias Höllerer
Traditional augmented reality (AR) systems predominantly rely on fixed class detectors or fiducial markers, limiting their ability to interpret complex, open-vocabulary natural language queries. We present a modular AR agent system that integrates multimodal large language models (MLLMs) with grounded vision models to enable relational reasoning in space and language-conditioned spatial retrieval in physical environments. Our adaptive task agent coordinates MLLMs and coordinate-aware perception tools to address varying query complexities, ranging from simple object identification to multi-object relational reasoning, while returning meter-accurate 3D anchors. It constructs dynamic AR scene graphs encoding nine typed relations (spatial, structural-semantic, causal-functional), enabling MLLMs to understand not just what objects exist, but how they relate and interact in 3D space. Through task-adaptive region-of-interest highlighting and contextual spatial retrieval, the system guides human attention to information-dense areas while supporting human-in-the-loop refinement. The agent dynamically invokes coordinate-aware tools for complex queries-selection, measurement, comparison, and actuation-grounding language understanding in physical operations. The modular architecture supports plug-and-use vision-language models without retraining, establishing AR agents as intermediaries that augment MLLMs with real-world spatial intelligence for interactive scene understanding. We also introduce GroundedAR-Bench, an evaluation framework for language-driven real world localization and relation grounding across diverse environments.
☆ Fi CoT S: Fine-to-Coarse LLM-Enhanced Hierarchical Cross-Modality Interaction for Time Series Forecasting
Yafei Lyu, Hao Zhou, Lu Zhang, Xu Yang, Zhiyong Liu
Time series forecasting is central to data analysis and web technologies. The recent success of Large Language Models (LLMs) offers significant potential for this field, especially from the cross-modality aspect. Most methods adopt an LLM-as-Predictor paradigm, using LLM as the forecasting backbone and designing modality alignment mechanisms to enable LLM to understand time series data. However, the semantic information in the two modalities of time series and text differs significantly, making it challenging for LLM to fully understand time series data. To mitigate this challenge, our work follows an LLM-as-Enhancer paradigm to fully utilize the advantage of LLM in text understanding, where LLM is only used to encode text modality to complement time series modality. Based on this paradigm, we propose FiCoTS, an LLM-enhanced fine-to-coarse framework for multimodal time series forecasting. Specifically, the framework facilitates progressive cross-modality interaction by three levels in a fine-to-coarse scheme: First, in the token-level modality alignment module, a dynamic heterogeneous graph is constructed to filter noise and align time series patches with text tokens; Second, in the feature-level modality interaction module, a global cross-attention mechanism is introduced to enable each time series variable to connect with relevant textual contexts; Third, in the decision-level modality fusion module, we design a gated network to adaptively fuse the results of the two modalities for robust predictions. These three modules work synergistically to let the two modalities interact comprehensively across three semantic levels, enabling textual information to effectively support temporal prediction. Extensive experiments on seven real-world benchmarks demonstrate that our model achieves state-of-the-art performance. The codes will be released publicly.
☆ EduEval: A Hierarchical Cognitive Benchmark for Evaluating Large Language Models in Chinese Education
Guoqing Ma, Jia Zhu, Hanghui Guo, Weijie Shi, Yue Cui, Jiawei Shen, Zilong Li, Yidan Liang
Large language models (LLMs) demonstrate significant potential for educational applications. However, their unscrutinized deployment poses risks to educational standards, underscoring the need for rigorous evaluation. We introduce EduEval, a comprehensive hierarchical benchmark for evaluating LLMs in Chinese K-12 education. This benchmark makes three key contributions: (1) Cognitive Framework: We propose the EduAbility Taxonomy, which unifies Bloom's Taxonomy and Webb's Depth of Knowledge to organize tasks across six cognitive dimensions including Memorization, Understanding, Application, Reasoning, Creativity, and Ethics. (2) Authenticity: Our benchmark integrates real exam questions, classroom conversation, student essays, and expert-designed prompts to reflect genuine educational challenges; (3) Scale: EduEval comprises 24 distinct task types with over 11,000 questions spanning primary to high school levels. We evaluate 14 leading LLMs under both zero-shot and few-shot settings, revealing that while models perform well on factual tasks, they struggle with classroom dialogue classification and exhibit inconsistent results in creative content generation. Interestingly, several open source models outperform proprietary systems on complex educational reasoning. Few-shot prompting shows varying effectiveness across cognitive dimensions, suggesting that different educational objectives require tailored approaches. These findings provide targeted benchmarking metrics for developing LLMs specifically optimized for diverse Chinese educational tasks.
☆ RealAppliance: Let High-fidelity Appliance Assets Controllable and Workable as Aligned Real Manuals
Yuzheng Gao, Yuxing Long, Lei Kang, Yuchong Guo, Ziyan Yu, Shangqing Mao, Jiyao Zhang, Ruihai Wu, Dongjiang Li, Hui Shen, Hao Dong
Existing appliance assets suffer from poor rendering, incomplete mechanisms, and misalignment with manuals, leading to simulation-reality gaps that hinder appliance manipulation development. In this work, we introduce the RealAppliance dataset, comprising 100 high-fidelity appliances with complete physical, electronic mechanisms, and program logic aligned with their manuals. Based on these assets, we propose the RealAppliance-Bench benchmark, which evaluates multimodal large language models and embodied manipulation planning models across key tasks in appliance manipulation planning: manual page retrieval, appliance part grounding, open-loop manipulation planning, and closed-loop planning adjustment. Our analysis of model performances on RealAppliance-Bench provides insights for advancing appliance manipulation research
☆ BioArc: Discovering Optimal Neural Architectures for Biological Foundation Models
Yi Fang, Haoran Xu, Jiaxin Han, Sirui Ding, Yizhi Wang, Yue Wang, Xuan Wang
Foundation models have revolutionized various fields such as natural language processing (NLP) and computer vision (CV). While efforts have been made to transfer the success of the foundation models in general AI domains to biology, existing works focus on directly adopting the existing foundation model architectures from general machine learning domains without a systematic design considering the unique physicochemical and structural properties of each biological data modality. This leads to suboptimal performance, as these repurposed architectures struggle to capture the long-range dependencies, sparse information, and complex underlying ``grammars'' inherent to biological data. To address this gap, we introduce BioArc, a novel framework designed to move beyond intuition-driven architecture design towards principled, automated architecture discovery for biological foundation models. Leveraging Neural Architecture Search (NAS), BioArc systematically explores a vast architecture design space, evaluating architectures across multiple biological modalities while rigorously analyzing the interplay between architecture, tokenization, and training strategies. This large-scale analysis identifies novel, high-performance architectures, allowing us to distill a set of empirical design principles to guide future model development. Furthermore, to make the best of this set of discovered principled architectures, we propose and compare several architecture prediction methods that effectively and efficiently predict optimal architectures for new biological tasks. Overall, our work provides a foundational resource and a principled methodology to guide the creation of the next generation of task-specific and foundation models for biology.
☆ HIMOSA: Efficient Remote Sensing Image Super-Resolution with Hierarchical Mixture of Sparse Attention
Yi Liu, Yi Wan, Xinyi Liu, Qiong Wu, Panwang Xia, Xuejun Huang, Yongjun Zhang
In remote sensing applications, such as disaster detection and response, real-time efficiency and model lightweighting are of critical importance. Consequently, existing remote sensing image super-resolution methods often face a trade-off between model performance and computational efficiency. In this paper, we propose a lightweight super-resolution framework for remote sensing imagery, named HIMOSA. Specifically, HIMOSA leverages the inherent redundancy in remote sensing imagery and introduces a content-aware sparse attention mechanism, enabling the model to achieve fast inference while maintaining strong reconstruction performance. Furthermore, to effectively leverage the multi-scale repetitive patterns found in remote sensing imagery, we introduce a hierarchical window expansion and reduce the computational complexity by adjusting the sparsity of the attention. Extensive experiments on multiple remote sensing datasets demonstrate that our method achieves state-of-the-art performance while maintaining computational efficiency.
☆ Teleportation-Based Defenses for Privacy in Approximate Machine Unlearning
Mohammad M Maheri, Xavier Cadet, Peter Chin, Hamed Haddadi
Approximate machine unlearning aims to efficiently remove the influence of specific data points from a trained model, offering a practical alternative to full retraining. However, it introduces privacy risks: an adversary with access to pre- and post-unlearning models can exploit their differences for membership inference or data reconstruction. We show these vulnerabilities arise from two factors: large gradient norms of forget-set samples and the close proximity of unlearned parameters to the original model. To demonstrate their severity, we propose unlearning-specific membership inference and reconstruction attacks, showing that several state-of-the-art methods (e.g., NGP, SCRUB) remain vulnerable. To mitigate this leakage, we introduce WARP, a plug-and-play teleportation defense that leverages neural network symmetries to reduce forget-set gradient energy and increase parameter dispersion while preserving predictions. This reparameterization obfuscates the signal of forgotten data, making it harder for attackers to distinguish forgotten samples from non-members or recover them via reconstruction. Across six unlearning algorithms, our approach achieves consistent privacy gains, reducing adversarial advantage (AUC) by up to 64% in black-box and 92% in white-box settings, while maintaining accuracy on retained data. These results highlight teleportation as a general tool for reducing attack success in approximate unlearning.
☆ Comparative Evaluation of Generative AI Models for Chest Radiograph Report Generation in the Emergency Department
Woo Hyeon Lim, Ji Young Lee, Jong Hyuk Lee, Saehoon Kim, Hyungjin Kim
Purpose: To benchmark open-source or commercial medical image-specific VLMs against real-world radiologist-written reports. Methods: This retrospective study included adult patients who presented to the emergency department between January 2022 and April 2025 and underwent same-day CXR and CT for febrile or respiratory symptoms. Reports from five VLMs (AIRead, Lingshu, MAIRA-2, MedGemma, and MedVersa) and radiologist-written reports were randomly presented and blindly evaluated by three thoracic radiologists using four criteria: RADPEER, clinical acceptability, hallucination, and language clarity. Comparative performance was assessed using generalized linear mixed models, with radiologist-written reports treated as the reference. Finding-level analyses were also performed with CT as the reference. Results: A total of 478 patients (median age, 67 years [interquartile range, 50-78]; 282 men [59.0%]) were included. AIRead demonstrated the lowest RADPEER 3b rate (5.3% [76/1434] vs. radiologists 13.9% [200/1434]; P<.001), whereas other VLMs showed higher disagreement rates (16.8-43.0%; P<.05). Clinical acceptability was the highest with AIRead (84.5% [1212/1434] vs. radiologists 74.3% [1065/1434]; P<.001), while other VLMs performed worse (41.1-71.4%; P<.05). Hallucinations were rare with AIRead, comparable to radiologists (0.3% [4/1425]) vs. 0.1% [1/1425]; P=.21), but frequent with other models (5.4-17.4%; P<.05). Language clarity was higher with AIRead (82.9% [1189/1434]), Lingshu (88.0% [1262/1434]), and MedVersa (88.4% [1268/1434]) compared with radiologists (78.1% [1120/1434]; P<.05). Sensitivity varied substantially across VLMs for the common findings: AIRead, 15.5-86.7%; Lingshu, 2.4-86.7%; MAIRA-2, 6.0-72.0%; MedGemma, 4.8-76.7%; and MedVersa, 20.2-69.3%. Conclusion: Medical VLMs for CXR report generation exhibited variable performance in report quality and diagnostic measures.
☆ USB: Unified Synthetic Brain Framework for Bidirectional Pathology-Healthy Generation and Editing
Jun Wang, Peirong Liu
Understanding the relationship between pathological and healthy brain structures is fundamental to neuroimaging, connecting disease diagnosis and detection with modeling, prediction, and treatment planning. However, paired pathological-healthy data are extremely difficult to obtain, as they rely on pre- and post-treatment imaging, constrained by clinical outcomes and longitudinal data availability. Consequently, most existing brain image generation and editing methods focus on visual quality yet remain domain-specific, treating pathological and healthy image modeling independently. We introduce USB (Unified Synthetic Brain), the first end-to-end framework that unifies bidirectional generation and editing of pathological and healthy brain images. USB models the joint distribution of lesions and brain anatomy through a paired diffusion mechanism and achieves both pathological and healthy image generation. A consistency guidance algorithm further preserves anatomical consistency and lesion correspondence during bidirectional pathology-healthy editing. Extensive experiments on six public brain MRI datasets including healthy controls, stroke, and Alzheimer's patients, demonstrate USB's ability to produce diverse and realistic results. By establishing the first unified benchmark for brain image generation and editing, USB opens opportunities for scalable dataset creation and robust neuroimaging analysis. Code is available at https://github.com/jhuldr/USB.
comment : 16 pages, 17 figures
☆ Trification: A Comprehensive Tree-based Strategy Planner and Structural Verification for Fact-Checking
Anab Maulana Barik, Shou Ziyi, Yang Kaiwen, Yang Qi, Shen Xin
Technological advancement allows information to be shared in just a single click, which has enabled the rapid spread of false information. This makes automated fact-checking system necessary to ensure the safety and integrity of our online media ecosystem. Previous methods have demonstrated the effectiveness of decomposing the claim into simpler sub-tasks and utilizing LLM-based multi agent system to execute them. However, those models faces two limitations: they often fail to verify every component in the claim and lack of structured framework to logically connect the results of sub-tasks for a final prediction. In this work, we propose a novel automated fact-checking framework called Trification. Our framework begins by generating a comprehensive set of verification actions to ensure complete coverage of the claim. It then structured these actions into a dependency graph to model the logical interaction between actions. Furthermore, the graph can be dynamically modified, allowing the system to adapt its verification strategy. Experimental results on two challenging benchmarks demonstrate that our framework significantly enhances fact-checking accuracy, thereby advancing current state-of-the-art in automated fact-checking system.
♻ ☆ NarraBench: A Comprehensive Framework for Narrative Benchmark ing
Sil Hamilton, Matthew Wilkens, Andrew Piper
We present NarraBench, a theory-informed taxonomy of narrative-understanding tasks, as well as an associated survey of 78 existing benchmarks in the area. We find significant need for new evaluations covering aspects of narrative understanding that are either overlooked in current work or are poorly aligned with existing metrics. Specifically, we estimate that only 27% of narrative tasks are well captured by existing benchmarks, and we note that some areas -- including narrative events, style, perspective, and revelation -- are nearly absent from current evaluations. We also note the need for increased development of benchmarks capable of assessing constitutively subjective and perspectival aspects of narrative, that is, aspects for which there is generally no single correct answer. Our taxonomy, survey, and methodology are of value to NLP researchers seeking to test LLM narrative understanding.
♻ ☆ Implicit Bias of the JKO Scheme
Peter Halmos, Boris Hanin
Wasserstein gradient flow provides a general framework for minimizing an energy functional $J$ over the space of probability measures on a Riemannian manifold $(M,g)$. Its canonical time-discretization, the Jordan-Kinderlehrer-Otto (JKO) scheme, produces for any step size $η>0$ a sequence of probability distributions $ρ_k^η$ that approximate to first order in $η$ Wasserstein gradient flow on $J$. But the JKO scheme also has many other remarkable properties not shared by other first order integrators, e.g. it preserves energy dissipation and exhibits unconditional stability for $λ$-geodesically convex functionals $J$. To better understand the JKO scheme we characterize its implicit bias at second order in $η$. We show that $ρ_k^η$ are approximated to order $η^2$ by Wasserstein gradient flow on a modified energy \[ J^η(ρ) = J(ρ) - \fracη{4}\int_M \Big\lVert \nabla_g \frac{δJ}{δρ} (ρ) \Big\rVert_{2}^{2} \,ρ(dx), \] obtained by subtracting from $J$ the squared metric curvature of $J$ times $η/4$. The JKO scheme therefore adds at second order in $η$ a deceleration in directions where the metric curvature of $J$ is rapidly changing. This corresponds to canonical implicit biases for common functionals: for entropy the implicit bias is the Fisher information, for KL-divergence it is the Fisher-Hyv{ä}rinen divergence, and for Riemannian gradient descent it is the kinetic energy in the metric $g$. To understand the differences between minimizing $J$ and $J^η$ we study JKO-Flow, Wasserstein gradient flow on $J^η$, in several simple numerical examples. These include exactly solvable Langevin dynamics on the Bures-Wasserstein space and Langevin sampling from a quartic potential in 1D.
♻ ☆ Superposition Yields Robust Neural Scaling NeurIPS 2025
Yizhou Liu, Ziming Liu, Jeff Gore
The success of today's large language models (LLMs) depends on the observation that larger models perform better. However, the origin of this neural scaling law, that loss decreases as a power law with model size, remains unclear. We propose that representation superposition, meaning that LLMs represent more features than they have dimensions, can be a key contributor to loss and cause neural scaling. Based on Anthropic's toy model, we use weight decay to control the degree of superposition, allowing us to systematically study how loss scales with model size. When superposition is weak, the loss follows a power law only if data feature frequencies are power-law distributed. In contrast, under strong superposition, the loss generically scales inversely with model dimension across a broad class of frequency distributions, due to geometric overlaps between representation vectors. We confirmed that open-sourced LLMs operate in the strong superposition regime and have loss scaling inversely with model dimension, and that the Chinchilla scaling laws are also consistent with this behavior. Our results identify representation superposition as a central driver of neural scaling laws, providing insights into questions like when neural scaling laws can be improved and when they will break down.
comment : Best Paper Runner-up at NeurIPS 2025
♻ ☆ Escaping Collapse: The Strength of Weak Data for Large Language Model Training
Kareem Amin, Sara Babakniya, Alex Bie, Weiwei Kong, Umar Syed, Sergei Vassilvitskii
Synthetically-generated data plays an increasingly larger role in training large language models. However, while synthetic data has been found to be useful, studies have also shown that without proper curation it can cause LLM performance to plateau, or even "collapse", after many training iterations. In this paper, we formalize this question and develop a theoretical framework to investigate how much curation is needed in order to ensure that LLM performance continually improves. Our analysis is inspired by boosting, a classic machine learning technique that leverages a very weak learning algorithm to produce an arbitrarily good classifier. The approach we analyze subsumes many recently proposed methods for training LLMs on synthetic data, and thus our analysis sheds light on why they are successful, and also suggests opportunities for future improvement. We present experiments that validate our theory, and show that dynamically focusing labeling resources on the most challenging examples -- in much the same way that boosting focuses the efforts of the weak learner -- leads to improved performance.
♻ ☆ Large Language Models for Power System Security: A Novel Multi-Modal Approach for Anomaly Detection in Energy Management Systems
Aydin Zaboli, Junho Hong, Alexandru Stefanov, Chen-Ching Liu, Chul-Sang Hwang
This paper elaborates on an extensive security framework specifically designed for energy management systems (EMSs), which effectively tackles the dynamic environment of cybersecurity vulnerabilities and/or system problems (SPs), accomplished through the incorporation of novel methodologies. A comprehensive multi-point attack/error model is initially proposed to systematically identify vulnerabilities throughout the entire EMS data processing pipeline, including post state estimation (SE) stealth attacks, EMS database manipulation, and human-machine interface (HMI) display corruption according to the real-time database (RTDB) storage. This framework acknowledges the interconnected nature of modern attack vectors, which utilize various phases of supervisory control and data acquisition (SCADA) data flow. Then, generative AI (GenAI)-based anomaly detection systems (ADSs) for EMSs are proposed for the first time in the power system domain to handle the scenarios. Further, a set-of-mark generative intelligence (SoM-GI) framework, which leverages multimodal analysis by integrating visual markers with rules considering the GenAI capabilities, is suggested to overcome inherent spatial reasoning limitations. The SoM-GI methodology employs systematic visual indicators to enable accurate interpretation of segmented HMI displays and detect visual anomalies that numerical methods fail to identify. Validation on the IEEE 14-Bus system shows the framework's effectiveness across scenarios, while visual analysis identifies inconsistencies. This integrated approach combines numerical analysis with visual pattern recognition and linguistic rules to protect against cyber threats and system errors.
comment : 10 Figures; 6 Tables; Accepted, IEEE ACCESS 2025
♻ ☆ Life-Code: Central Dogma Modeling with Multi-Omics Sequence Unification
Zicheng Liu, Siyuan Li, Zhiyuan Chen, Chang Yu, Qirong Yang, Yucheng Guo, Yujie Yang, Xiaoming Zhang, Stan Z. Li
The interactions between DNA, RNA, and proteins are fundamental to biological processes, as illustrated by the central dogma of molecular biology. Although modern biological pre-trained models have achieved great success in analyzing these macromolecules individually, their interconnected nature remains underexplored. This paper follows the guidance of the central dogma to redesign both the data and model pipeline and offers a comprehensive framework, Life-Code, that spans different biological functions. As for data flow, we propose a unified pipeline to integrate multi-omics data by reverse-transcribing RNA and reverse-translating amino acids into nucleotide-based sequences. As for the model, we design a codon tokenizer and a hybrid long-sequence architecture to encode the interactions between coding and non-coding regions through masked modeling pre-training. To model the translation and folding process with coding sequences, Life-Code learns protein structures of the corresponding amino acids by knowledge distillation from off-the-shelf protein language models. Such designs enable Life-Code to capture complex interactions within genetic sequences, providing a more comprehensive understanding of multi-omics with the central dogma. Extensive experiments show that Life-Code achieves state-of-the-art results on various tasks across three omics, highlighting its potential for advancing multi-omics analysis and interpretation.
comment : Preprint V3 (10 pages main text)
♻ ☆ Closing the Data-Efficiency Gap Between Autoregressive and Masked Diffusion LLMs
Xu Pan, Ely Hahami, Jingxuan Fan, Ziqian Xie, Haim Sompolinsky
Despite autoregressive large language models (arLLMs) being the current dominant paradigm in language modeling, effectively updating these models to incorporate new factual knowledge still remains difficult. They resist knowledge injection via fine-tuning due to inherent shortcomings such as the "reversal curse" -- the challenge of answering questions that reverse the original information order in the training sample. Masked diffusion large language models (dLLMs) are rapidly emerging as a powerful alternative to the arLLM paradigm, with evidence of better data efficiency and free of the "reversal curse" in pre-training. However, it is unknown whether these advantages extend to the post-training phase, i.e. whether pre-trained dLLMs can easily acquire new knowledge through fine-tuning. On three diverse datasets, we fine-tune arLLMs and dLLMs, evaluating them with forward and backward style Question Answering (QA) to probe knowledge generalization and the reversal curse. Our results confirm that arLLMs critically rely on extensive data augmentation via paraphrases for QA generalization, and paraphrases are only effective when their information order matches the QA style. Conversely, dLLMs achieve high accuracies on both forward and backward QAs without paraphrases; adding paraphrases yields only marginal gains. Inspired by the dLLM's performance, we introduce a novel masked fine-tuning paradigm for knowledge injection into pre-trained arLLMs. This proposed method successfully and drastically improves the data efficiency of arLLM fine-tuning, effectively closing its performance gap with dLLMs. We further show that the masked fine-tuning paradigm of arLLMs can be extended to the supervised fine-tuning (SFT) of mathematical capability. Across two models and two datasets, our masked SFT outperforms regular SFT.
♻ ☆ A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone NeurIPS 2025
Jitai Hao, Qiang Huang, Hao Liu, Xinyan Xiao, Zhaochun Ren, Jun Yu
Training high-performing Small Language Models (SLMs) remains costly, even with knowledge distillation and pruning from larger teacher models. Existing work often faces three key challenges: (1) information loss from hard pruning, (2) inefficient alignment of representations, and (3) underutilization of informative activations, particularly from Feed-Forward Networks (FFNs). To address these challenges, we introduce Low-Rank Clone (LRC), an efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models. LRC trains a set of low-rank projection matrices that jointly enable soft pruning by compressing teacher weights, and activation clone by aligning student activations, including FFN signals, with those of the teacher. This unified design maximizes knowledge transfer while removing the need for explicit alignment modules. Extensive experiments with open-source teachers (e.g., Llama-3.2-3B-Instruct, Qwen2.5-3B/7B-Instruct) show that LRC matches or surpasses state-of-the-art models trained on trillions of tokens--while using only 20B tokens, achieving over 1,000x training efficiency. Our codes and model checkpoints are available at https://github.com/CURRENTF/LowRankClone and https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf.
comment : NeurIPS 2025 Spotlight
♻ ☆ Adversarial Exploitation of Data Diversity Improves Visual Localization
Sihang Li, Siqi Tan, Bowen Chang, Jing Zhang, Chen Feng, Yiming Li
Visual localization, which estimates a camera's pose within a known scene, is a fundamental capability for autonomous systems. While absolute pose regression (APR) methods have shown promise for efficient inference, they often struggle with generalization. Recent approaches attempt to address this through data augmentation with varied viewpoints, yet they overlook a critical factor: appearance diversity. In this work, we identify appearance variation as the key to robust localization. Specifically, we first lift real 2D images into 3D Gaussian Splats with varying appearance and deblurring ability, enabling the synthesis of diverse training data that varies not just in poses but also in environmental conditions such as lighting and weather. To fully unleash the potential of the appearance-diverse data, we build a two-branch joint training pipeline with an adversarial discriminator to bridge the syn-to-real gap. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art methods, reducing translation and rotation errors by 50\% and 41\% on indoor datasets, and 38\% and 44\% on outdoor datasets. Most notably, our method shows remarkable robustness in dynamic driving scenarios under varying weather conditions and in day-to-night scenarios, where previous APR methods fail. Project Page: https://ai4ce.github.io/RAP/
comment : 24 pages, 22 figures
♻ ☆ Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models
Jitai Hao, Hao Liu, Xinyan Xiao, Qiang Huang, Jun Yu
Unified Multimodal Models (UMMs) built on shared autoregressive (AR) transformers are attractive for their architectural simplicity. However, we identify a critical limitation: when trained on multimodal inputs, modality-shared transformers suffer from severe gradient conflicts between vision and text, particularly in shallow and deep layers. We trace this issue to the fundamentally different low-level statistical properties of images and text, while noting that conflicts diminish in middle layers where representations become more abstract and semantically aligned. To overcome this challenge, we propose Uni-X, a two-end-separated, middle-shared architecture. Uni-X dedicates its initial and final layers to modality-specific processing, while maintaining shared parameters in the middle layers for high-level semantic fusion. This X-shaped design not only eliminates gradient conflicts at both ends but also further alleviates residual conflicts in the shared layers. Extensive experiments validate the effectiveness of Uni-X. Under identical training conditions, Uni-X achieves superior training efficiency compared to strong baselines. When scaled to 3B parameters with larger training data, Uni-X matches or surpasses 7B AR-based UMMs, achieving a GenEval score of 82 for image generation alongside strong performance in text and vision understanding tasks. These results establish Uni-X as a parameter-efficient and scalable foundation for future unified multimodal modeling. Our code is available at https://github.com/CURRENTF/Uni-X
♻ ☆ MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding WSDM 2026
Daoze Zhang, Chenghan Fu, Zhanheng Nie, Jianyu Liu, Wanxian Guan, Yuan Gao, Jun Song, Pengjie Wang, Jian Xu, Bo Zheng
With the rapid advancement of e-commerce, exploring general representations rather than task-specific ones has attracted increasing research attention. For product understanding, although existing discriminative dual-flow architectures drive progress in this field, they inherently struggle to model the many-to-one alignment between multiple images and texts of products. Therefore, we argue that generative Multimodal Large Language Models (MLLMs) hold significant potential for improving product representation learning. Nevertheless, achieving this goal still remains non-trivial due to several key challenges: the lack of multimodal and aspect-aware modeling modules in typical LLMs; the common presence of background noise in product images; and the absence of a standard benchmark for evaluation. To address these issues, we propose the first generative MLLM-based model named MOON for product representation learning. Our method (1) employs a guided Mixture-of-Experts (MoE) module for targeted modeling of multimodal and aspect-specific product content; (2) effectively detects core semantic regions in product images to mitigate the distraction and interference caused by background noise; and (3) introduces the specialized negative sampling strategy to increase the difficulty and diversity of negative samples. In addition, we release a large-scale multimodal benchmark MBE for various product understanding tasks. Experimentally, our model demonstrates competitive zero-shot performance on both our benchmark and the public dataset, showcasing strong generalization across various downstream tasks, including cross-modal retrieval, product classification, and attribute prediction. Furthermore, the case study and visualization illustrate the effectiveness of MOON for product understanding.
comment : Accepted by WSDM 2026. 11 pages, 9 figures
♻ ☆ A CPU-Centric Perspective on Agent ic AI
Ritik Raj, Hong Wang, Tushar Krishna
Agentic AI frameworks add a decision-making orchestrator embedded with external tools, including web search, Python interpreter, contextual database, and others, on top of monolithic LLMs, turning them from passive text oracles into autonomous problem-solvers that can plan, call tools, remember past steps, and adapt on the fly. This paper aims to characterize and understand the system bottlenecks introduced by agentic AI workloads from a largely overlooked CPU-centric perspective. We first systematically characterize Agentic AI on the basis of orchestrator/decision making component, inference path dynamics and repetitiveness of the agentic flow which directly influences the system-level performance. Thereafter, based on the characterization, we choose five representative agentic AI workloads- Haystack RAG, Toolformer, ChemCrow, Langchain and SWE-Agent to profile latency, throughput and energy metrics and demystify the significant impact of CPUs on these metrics relative to GPUs. We observe that - 1. Tool processing on CPUs can take up to 90.6% of the total latency; 2. Agentic throughput gets bottlenecked either by CPU factors - coherence, synchronization and over-subscription of cores or GPU factors - main memory capacity and bandwidth; \circled{3} CPU dynamic energy consumes up to 44% of the total dynamic energy at large batch sizes. Based on the profiling insights, we present two key optimizations- 1. CPU and GPU-Aware Micro-batching (CGAM) and 2. Mixed Agentic Workload Scheduling (MAWS) for homogeneous and heterogeneous agentic workloads respectively to demonstrate the potential to improve the performance, efficiency, and scalability of agentic AI. We achieve up to 2.1x and 1.41x P50 latency speedup compared to the multi-processing benchmark for homogeneous and heterogeneous agentic workloads respectively.
♻ ☆ Delta-Triplane Transformer s as Occupancy Wo rl d Models
Haoran Xu, Peixi Peng, Guang Tan, Yiqian Chang, Yisen Zhao, Yonghong Tian
Occupancy World Models (OWMs) aim to predict future scenes via 3D voxelized representations of the environment to support intelligent motion planning. Existing approaches typically generate full future occupancy states from VAE-style latent encodings, which can be computationally expensive and redundant. We propose Delta-Triplane Transformers (DTT), a novel 4D OWM for autonomous driving, that introduces two key innovations: (1) a triplane based representation that encodes 3D occupancy more compactly than previous approaches, and (2) an incremental prediction strategy for OWM that models {\em changes} in occupancy rather than dealing with full states. The core insight is that changes in the compact 3D latent space are naturally sparser and easier to model, enabling higher accuracy with a lighter-weight architecture. Building on this representation, DTT extracts multi-scale motion features from historical data and iteratively predict future triplane deltas. These deltas are combined with past states to decode future occupancy and ego-motion trajectories. Extensive experiments demonstrate that DTT delivers a 1.44$\times$ speedup (26 FPS) over the state of the art, improves mean IoU to 30.85, and reduces the mean absolute planning error to 1.0 meters. Demo videos are provided in the supplementary material.
♻ ☆ ICAS: Detecting Training Data from Autoregressive Image Generative Models ACM MM 2025
Hongyao Yu, Yixiang Qiu, Yiheng Yang, Hao Fang, Tianqu Zhuang, Jiaxin Hong, Bin Chen, Hao Wu, Shu-Tao Xia
Autoregressive image generation has witnessed rapid advancements, with prominent models such as scale-wise visual auto-regression pushing the boundaries of visual synthesis. However, these developments also raise significant concerns regarding data privacy and copyright. In response, training data detection has emerged as a critical task for identifying unauthorized data usage in model training. To better understand the vulnerability of autoregressive image generative models to such detection, we conduct the first study applying membership inference to this domain. Our approach comprises two key components: implicit classification and an adaptive score aggregation strategy. First, we compute the implicit token-wise classification score within the query image. Then we propose an adaptive score aggregation strategy to acquire a final score, which places greater emphasis on the tokens with lower scores. A higher final score indicates that the sample is more likely to be involved in the training set. To validate the effectiveness of our method, we adapt existing detection algorithms originally designed for LLMs to visual autoregressive models. Extensive experiments demonstrate the superiority of our method in both class-conditional and text-to-image scenarios. Moreover, our approach exhibits strong robustness and generalization under various data transformations. Furthermore, sufficient experiments suggest two novel key findings: (1) A linear scaling law on membership inference, exposing the vulnerability of large foundation models. (2) Training data from scale-wise visual autoregressive models is easier to detect than other autoregressive paradigms. Our code is available at https://github.com/Chrisqcwx/ImageAR-MIA.
comment : ACM MM 2025
♻ ☆ AI Should Sense Better, Not Just Scale Bigger: Adaptive Sensing as a Paradigm Shift NeurIPS 2025
Eunsu Baek, Keondo Park, Jeonggil Ko, Min-hwan Oh, Taesik Gong, Hyung-Sin Kim
Current AI advances largely rely on scaling neural models and expanding training datasets to achieve generalization and robustness. Despite notable successes, this paradigm incurs significant environmental, economic, and ethical costs, limiting sustainability and equitable access. Inspired by biological sensory systems, where adaptation occurs dynamically at the input (e.g., adjusting pupil size, refocusing vision)--we advocate for adaptive sensing as a necessary and foundational shift. Adaptive sensing proactively modulates sensor parameters (e.g., exposure, sensitivity, multimodal configurations) at the input level, significantly mitigating covariate shifts and improving efficiency. Empirical evidence from recent studies demonstrates that adaptive sensing enables small models (e.g., EfficientNet-B0) to surpass substantially larger models (e.g., OpenCLIP-H) trained with significantly more data and compute. We (i) outline a roadmap for broadly integrating adaptive sensing into real-world applications spanning humanoid, healthcare, autonomous systems, agriculture, and environmental monitoring, (ii) critically assess technical and ethical integration challenges, and (iii) propose targeted research directions, such as standardized benchmarks, real-time adaptive algorithms, multimodal integration, and privacy-preserving methods. Collectively, these efforts aim to transition the AI community toward sustainable, robust, and equitable artificial intelligence systems.
comment : Published in NeurIPS 2025 (Position Paper Track)
♻ ☆ IoT-LLM: a framework for enhancing Large Language Model reasoning from real-wo rl d sensor data
Tuo An, Yunjiao Zhou, Han Zou, Jianfei Yang
Large Language Models excel in textual tasks but often struggle with physical-world reasoning tasks. Inspired by human cognition, where perception is fundamental to reasoning, we explore augmenting LLMs with enhanced perception abilities using Internet of Things (IoT) data and pertinent knowledge. In this work, we systematically study LLMs' capability to address IoT-sensory tasks by augmenting their perception and knowledge base, and then propose a unified framework, IoT-LLM, to enhance such capability. In IoT-LLM, we customize three steps: preprocessing IoT data into suitable formats, expanding LLMs knowledge via IoT-oriented retrieval-augmented generation and activating LLMs commonsense knowledge through chain-of-thought prompting. We design a benchmark comprising five real-world tasks with varying data types and reasoning complexities to evaluate the performance of IoT-LLM. Experimental results reveal that IoT-LLM significantly improves the performance of IoT-sensory task reasoning of LLMs, with models like GPT-4o-mini showing a 49.4% average improvement over previous methods.
comment : 33 pages, 13 figures
♻ ☆ Is General-Purpose AI Reasoning Sensitive to Data-Induced Cognitive Biases? Dynamic Benchmark ing on Typical Software Engineering Dilemmas
Francesco Sovrano, Gabriele Dominici, Rita Sevastjanova, Alessandra Stramiglio, Alberto Bacchelli
Human cognitive biases in software engineering can lead to costly errors. While general-purpose AI (GPAI) systems may help mitigate these biases due to their non-human nature, their training on human-generated data raises a critical question: Do GPAI systems themselves exhibit cognitive biases? To investigate this, we present the first dynamic benchmarking framework to evaluate data-induced cognitive biases in GPAI within software engineering workflows. Starting with a seed set of 16 hand-crafted realistic tasks, each featuring one of 8 cognitive biases (e.g., anchoring, framing) and corresponding unbiased variants, we test whether bias-inducing linguistic cues unrelated to task logic can lead GPAI systems from correct to incorrect conclusions. To scale the benchmark and ensure realism, we develop an on-demand augmentation pipeline relying on GPAI systems to generate task variants that preserve bias-inducing cues while varying surface details. This pipeline ensures correctness (88-99% on average, according to human evaluation), promotes diversity, and controls reasoning complexity by leveraging Prolog-based reasoning. We evaluate leading GPAI systems (GPT, LLaMA, DeepSeek) and find a consistent tendency to rely on shallow linguistic heuristics over more complex reasoning. All systems exhibit bias sensitivity (6-35%), which increases with task complexity (up to 49%) and highlights risks in AI-driven software engineering.
♻ ☆ LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders
Borna Khodabandeh, Amirabbas Afzali, Amirhossein Afsharrad, Seyed Shahabeddin Mousavi, Sanjay Lall, Sajjad Amini, Seyed-Mohsen Moosavi-Dezfooli
Visual encoders have become fundamental components in modern computer vision pipelines. However, ensuring robustness against adversarial perturbations remains a critical challenge. Recent efforts have explored both supervised and unsupervised adversarial fine-tuning strategies. We identify two key limitations in these approaches: (i) they often suffer from instability, especially during the early stages of fine-tuning, resulting in suboptimal convergence and degraded performance on clean data, and (ii) they exhibit a suboptimal trade-off between robustness and clean data accuracy, hindering the simultaneous optimization of both objectives. To overcome these challenges, we propose Lagrangian-Optimized Robust Embeddings (LORE), a novel unsupervised adversarial fine-tuning framework. LORE utilizes constrained optimization, which offers a principled approach to balancing competing goals, such as improving robustness while preserving nominal performance. By enforcing embedding-space proximity constraints, LORE effectively maintains clean data performance throughout adversarial fine-tuning. Extensive experiments show that LORE significantly improves zero-shot adversarial robustness with minimal degradation in clean data accuracy. Furthermore, we demonstrate the effectiveness of the adversarially fine-tuned CLIP image encoder in out-of-distribution generalization and enhancing the interpretability of image embeddings.
♻ ☆ Information Physics of Intelligence: Unifying Logical Depth and Entropy under Thermodynamic Constraints
Jianfeng Xu, Zeyan Li
The rapid scaling of artificial intelligence models has revealed a fundamental tension between model capacity (storage) and inference efficiency (computation). While classical information theory focuses on transmission and storage limits, it lacks a unified physical framework to quantify the thermodynamic costs of generating information from compressed laws versus retrieving it from memory. In this paper, we propose a theoretical framework that treats information processing as an enabling mapping from ontological states to carrier states. We introduce a novel metric, Derivation Entropy, which quantifies the effective work required to compute a target state from a given logical depth. By analyzing the interplay between Shannon entropy (storage) and computational complexity (time/energy), we demonstrate the existence of a critical phase transition point. Below this threshold, memory retrieval is thermodynamically favorable; above it, generative computation becomes the optimal strategy. This "Energy-Time-Space" conservation law provides a physical explanation for the efficiency of generative models and offers a rigorous mathematical bound for designing next-generation, energy-efficient AI architectures. Our findings suggest that the minimization of Derivation Entropy is a governing principle for the evolution of both biological and artificial intelligence.
♻ ☆ TLoRA: Tri-Matrix Low-Rank Adaptation of Large Language Models
Tanvir Islam
We propose TLoRA, a novel tri-matrix low-rank adaptation method that decomposes weight updates into three matrices: two fixed random matrices and one trainable matrix, combined with a learnable, layer-wise scaling factor. This tri-matrix design enables TLoRA to achieve highly efficient parameter adaptation while introducing minimal additional computational overhead. Through extensive experiments on the GLUE benchmark, we demonstrate that TLoRA achieves comparable performance to existing low-rank methods such as LoRA and Adapter-based techniques, while requiring significantly fewer trainable parameters. Analyzing the adaptation dynamics, we observe that TLoRA exhibits Gaussian-like weight distributions, stable parameter norms, and scaling factor variability across layers, further highlighting its expressive power and adaptability. Additionally, we show that TLoRA closely resembles LoRA in its eigenvalue distributions, parameter norms, and cosine similarity of updates, underscoring its ability to effectively approximate LoRA's adaptation behavior. Our results establish TLoRA as a highly efficient and effective fine-tuning method for LLMs, offering a significant step forward in resource-efficient model adaptation.
♻ ☆ Teaching Language Models to Critique via Reinforcement Learning
Zhihui Xie, Jie Chen, Liyu Chen, Weichao Mao, Jingjing Xu, Lingpeng Kong
Teaching large language models (LLMs) to critique and refine their outputs is crucial for building systems that can iteratively improve, yet it is fundamentally limited by the ability to provide accurate judgments and actionable suggestions. In this work, we study LLM critics for code generation and propose $\texttt{CTRL}$, a framework for $\texttt{C}$ritic $\texttt{T}$raining via $\texttt{R}$einforcement $\texttt{L}$earning, which trains a critic model to generate feedback that maximizes correction performance for a fixed generator model without human supervision. Our results demonstrate that critics trained with $\texttt{CTRL}$ significantly enhance pass rates and mitigate compounding errors across both base and stronger generator models. Furthermore, we show that these critic models act as accurate generative reward models and enable test-time scaling through iterative critique-revision, achieving up to 106.1% relative improvements across challenging code generation benchmarks.
♻ ☆ Structural Plasticity as Active Inference: A Biologically-Inspired Architecture for Homeostatic Control
Brennen A. Hill
Traditional neural networks, while powerful, rely on biologically implausible learning mechanisms such as global backpropagation. This paper introduces the Structurally Adaptive Predictive Inference Network (SAPIN), a novel computational model inspired by the principles of active inference and the morphological plasticity observed in biological neural cultures. SAPIN operates on a 2D grid where processing units, or cells, learn by minimizing local prediction errors. The model features two primary, concurrent learning mechanisms: a local, Hebbian-like synaptic plasticity rule based on the temporal difference between a cell's actual activation and its learned expectation, and a structural plasticity mechanism where cells physically migrate across the grid to optimize their information-receptive fields. This dual approach allows the network to learn both how to process information (synaptic weights) and also where to position its computational resources (network topology). We validated the SAPIN model on the classic Cart Pole reinforcement learning benchmark. Our results demonstrate that the architecture can successfully solve the CartPole task, achieving robust performance. The network's intrinsic drive to minimize prediction error and maintain homeostasis was sufficient to discover a stable balancing policy. We also found that while continual learning led to instability, locking the network's parameters after achieving success resulted in a stable policy. When evaluated for 100 episodes post-locking (repeated over 100 successful agents), the locked networks maintained an average 82% success rate.
comment : In Bridging the Gap Between Neuroscience and Engineering for Energy-Efficient AI
♻ ☆ AHAMask: Reliable Task Specification for Large Audio Language Models without Instructions AAAI 2026
Yiwei Guo, Bohan Li, Hankun Wang, Zhihan Li, Shuai Wang, Xie Chen, Kai Yu
Although current large audio language models (LALMs) extend text large language models (LLMs) with generic acoustic understanding abilities, they usually suffer from prompt sensitivity, where different instructions of the same intention can yield drastically different outcomes. In this work, we propose AHAMask, where we simply mask some of the attention heads in the decoder-only LLM backbone of LALMs, to trigger specific acoustic task functionalities without instructions. These masks are efficiently obtained by training on an LALM, with the number of trainable parameters equal to the attention head count in its LLM backbone. We show by experiments that applying such selective attention head masks achieves comparable or even better performance than using instructions, either on single or composite tasks. Besides achieving reliable acoustic task specification for LALMs, this also reveals that LALMs exhibit certain "functional pathways" in their attention heads.
comment : 15 pages, 10 tables, 6 figures. This is the camera ready version for AAAI 2026, plus an appendix for supplementary experimental details and results
♻ ☆ Prompt -OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision -Language Model Adaptation WACV 2026
Xiwen Chen, Wenhui Zhu, Peijie Qiu, Hao Wang, Huayu Li, Haiyu Wu, Aristeidis Sotiras, Yalin Wang, Abolfazl Razi
Vision-language models (VLMs) such as CLIP demonstrate strong performance but struggle when adapted to downstream tasks. Prompt learning has emerged as an efficient and effective strategy to adapt VLMs while preserving their pre-trained knowledge. However, existing methods still lead to overfitting and degrade zero-shot generalization. To address this challenge, we propose an optimal transport (OT)-guided prompt learning framework that mitigates forgetting by preserving the structural consistency of feature distributions between pre-trained and fine-tuned models. Unlike conventional point-wise constraints, OT naturally captures cross-instance relationships and expands the feasible parameter space for prompt tuning, allowing a better trade-off between adaptation and generalization. Our approach enforces joint constraints on both vision and text representations, ensuring a holistic feature alignment. Extensive experiments on benchmark datasets demonstrate that our simple yet effective method can outperform existing prompt learning strategies in base-to-novel generalization, cross-dataset evaluation, and domain generalization without additional augmentation or ensemble techniques. The code is available at https://github.com/ChongQingNoSubway/Prompt-OT
comment : Accepted to WACV 2026
♻ ☆ DND: Boosting Large Language Models with Dynamic Nested Depth
Tieyuan Chen, Xiaodong Chen, Haoxing Chen, Zhenzhong Lan, Weiyao Lin, Jianguo Li
We introduce Dynamic Nested Depth (DND), a novel method that improves performance for off-the-shelf LLMs by selecting critical tokens to reprocess in a nested depth manner. Specifically, at the end of the given transformer layer, DND identifies more critical tokens with a router and feeds them back for an extra round of processing, effectively ``reviewing" difficult tokens while avoiding redundant computation for easier ones. The dynamic selection mechanism is tailored for precise control via two novel strategies: a router controlling loss to enhance token selection distinguishability, and a threshold control scheme to ensure selection stability. We demonstrate the effectiveness of DND by directly integrating it into pre-trained dense and MoE models during a post-training phase. On diverse benchmarks, this approach boosts the performances of the dense Qwen3-1.7B by 1.88% and the MoE Qwen3-30B-A3B by 0.87%, all with a minimal parameter and computing increase.
comment : TL;DR: We introduce Dynamic Nested Depth (DND), an efficient paradigm that adaptively identifies critical tokens and selectively deepens their computation via nested re-processing
♻ ☆ Look as You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning AAAI'2026
Shuochen Liu, Pengfei Luo, Chao Zhang, Yuhao Chen, Haotian Zhang, Qi Liu, Xin Kou, Tong Xu, Enhong Chen
Aiming to identify precise evidence sources from visual documents, visual evidence attribution for visual document retrieval-augmented generation (VD-RAG) ensures reliable and verifiable predictions from vision-language models (VLMs) in multimodal question answering. Most existing methods adopt end-to-end training to facilitate intuitive answer verification. However, they lack fine-grained supervision and progressive traceability throughout the reasoning process. In this paper, we introduce the Chain-of-Evidence (CoE) paradigm for VD-RAG. CoE unifies Chain-of-Thought (CoT) reasoning and visual evidence attribution by grounding reference elements in reasoning steps to specific regions with bounding boxes and page indexes. To enable VLMs to generate such evidence-grounded reasoning, we propose Look As You Think (LAT), a reinforcement learning framework that trains models to produce verifiable reasoning paths with consistent attribution. During training, LAT evaluates the attribution consistency of each evidence region and provides rewards only when the CoE trajectory yields correct answers, encouraging process-level self-verification. Experiments on vanilla Qwen2.5-VL-7B-Instruct with Paper- and Wiki-VISA benchmarks show that LAT consistently improves the vanilla model in both single- and multi-image settings, yielding average gains of 8.23% in soft exact match (EM) and 47.0% in IoU@0.5. Meanwhile, LAT not only outperforms the supervised fine-tuning baseline, which is trained to directly produce answers with attribution, but also exhibits stronger generalization across domains.
comment : Poster of AAAI'2026
♻ ☆ M3DLayout: A Multi-Source Dataset of 3D Indoor Layouts and Structured Descriptions for 3D Generation
Yiheng Zhang, Zhuojiang Cai, Mingdao Wang, Meitong Guo, Tianxiao Li, Li Lin, Yuwang Wang
In text-driven 3D scene generation, object layout serves as a crucial intermediate representation that bridges high-level language instructions with detailed geometric output. It not only provides a structural blueprint for ensuring physical plausibility but also supports semantic controllability and interactive editing. However, the learning capabilities of current 3D indoor layout generation models are constrained by the limited scale, diversity, and annotation quality of existing datasets. To address this, we introduce M3DLayout, a large-scale, multi-source dataset for 3D indoor layout generation. M3DLayout comprises 21,367 layouts and over 433k object instances, integrating three distinct sources: real-world scans, professional CAD designs, and procedurally generated scenes. Each layout is paired with detailed structured text describing global scene summaries, relational placements of large furniture, and fine-grained arrangements of smaller items. This diverse and richly annotated resource enables models to learn complex spatial and semantic patterns across a wide variety of indoor environments. To assess the potential of M3DLayout, we establish a benchmark using both a text-conditioned diffusion model and a text-conditioned autoregressive model. Experimental results demonstrate that our dataset provides a solid foundation for training layout generation models. Its multi-source composition enhances diversity, notably through the Inf3DLayout subset which provides rich small-object information, enabling the generation of more complex and detailed scenes. We hope that M3DLayout can serve as a valuable resource for advancing research in text-driven 3D scene synthesis. All dataset and code will be made public upon acceptance.
comment : https://graphic-kiliani.github.io/M3DLayout/
♻ ☆ REBot: From RAG to CatRAG with Semantic Enrichment and Graph Routing
Thanh Ma, Tri-Tam La, Lam-Thu Le Huu, Minh-Nghi Nguyen, Khanh-Van Pham Luu
Academic regulation advising is essential for helping students interpret and comply with institutional policies, yet building effective systems requires domain specific regulatory resources. To address this challenge, we propose REBot, an LLM enhanced advisory chatbot powered by CatRAG, a hybrid retrieval reasoning framework that integrates retrieval augmented generation with graph based reasoning. CatRAG unifies dense retrieval and graph reasoning, supported by a hierarchical, category labeled knowledge graph enriched with semantic features for domain alignment. A lightweight intent classifier routes queries to the appropriate retrieval modules, ensuring both factual accuracy and contextual depth. We construct a regulation specific dataset and evaluate REBot on classification and question answering tasks, achieving state of the art performance with an F1 score of 98.89%. Finally, we implement a web application that demonstrates the practical value of REBot in real world academic advising scenarios.
♻ ☆ WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions ICCV 2025
Zizhang Li, Hong-Xing Yu, Wei Liu, Yin Yang, Charles Herrmann, Gordon Wetzstein, Jiajun Wu
WonderPlay is a novel framework integrating physics simulation with video generation for generating action-conditioned dynamic 3D scenes from a single image. While prior works are restricted to rigid body or simple elastic dynamics, WonderPlay features a hybrid generative simulator to synthesize a wide range of 3D dynamics. The hybrid generative simulator first uses a physics solver to simulate coarse 3D dynamics, which subsequently conditions a video generator to produce a video with finer, more realistic motion. The generated video is then used to update the simulated dynamic 3D scene, closing the loop between the physics solver and the video generator. This approach enables intuitive user control to be combined with the accurate dynamics of physics-based simulators and the expressivity of diffusion-based video generators. Experimental results demonstrate that WonderPlay enables users to interact with various scenes of diverse content, including cloth, sand, snow, liquid, smoke, elastic, and rigid bodies -- all using a single image input. Code will be made public. Project website: https://kyleleey.github.io/WonderPlay/
comment : ICCV 2025 (Highlight). The first two authors contributed equally. Project website: https://kyleleey.github.io/WonderPlay/
♻ ☆ Wo rl dScore: A Unified Evaluation Benchmark for Wo rl d Generation ICCV 2025
Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, Jiajun Wu
We introduce the WorldScore benchmark, the first unified benchmark for world generation. We decompose world generation into a sequence of next-scene generation tasks with explicit camera trajectory-based layout specifications, enabling unified evaluation of diverse approaches from 3D and 4D scene generation to video generation models. The WorldScore benchmark encompasses a curated dataset of 3,000 test examples that span diverse worlds: static and dynamic, indoor and outdoor, photorealistic and stylized. The WorldScore metrics evaluate generated worlds through three key aspects: controllability, quality, and dynamics. Through extensive evaluation of 19 representative models, including both open-source and closed-source ones, we reveal key insights and challenges for each category of models. Our dataset, evaluation code, and leaderboard can be found at https://haoyi-duan.github.io/WorldScore/
comment : ICCV 2025. Project website: https://haoyi-duan.github.io/WorldScore/ The first two authors contributed equally
♻ ☆ Tool-RoCo: An Agent -as-Tool Self-organization Large Language Model Benchmark in Multi-robot Cooperation
Ke Zhang, Xiaoning Zhao, Ce Zheng, Jiahong Ning, Dandan Zhu, Wenqi Zhang, Chen Sun, Toshiharu Sugawara
This study proposes Tool-RoCo, a novel benchmark for evaluating large language models (LLMs) in long-term multi-agent cooperation based on RoCo, a multi-robot cooperative benchmark. Recent research on LLM-based multi-agent systems has relied on predefined orchestration, while ignoring agent autonomy. Tool-RoCo treats other agents as tools and introduces cooperative tools, leveraging tool usage to evaluate multi-agent cooperation and self-organization. Tool usage means that each agent (LLM) selects a tool from a candidate set based on the current state, receives feedback, and adjusts its selection in subsequent rounds. To evaluate different autonomy levels, we propose four LLM paradigms: (1) centralized cooperation, where a single LLM allocates tools to all agents; (2) centralized self-organization, where a central LLM autonomously activates agents while keeping others inactive; (3) decentralized cooperation, where each agent has its own LLM and calls tools based on local information; and (4) self-organization, where a randomly chosen initial agent can request collaboration, activating additional agents via tool calls. Tool-RoCo includes three multi-robot tasks, SORT, PACK, and CABINET, to measure format and parameter accuracy and agent coordination through tool usage. The results using several LLMs showed that cooperative tools accounted for only 7.09% of all tools, indicating that LLM-based agents rarely invoked others as assistants. Moreover, activation tools accounted for 96.42%, suggesting that current LLMs tend to maintain active agents while seldom deactivating them for adaptive coordination. Tool-RoCo provides a systematic benchmark to evaluate LLM autonomy and cooperation in multi-agent tasks. Code and Demo: https://github.com/ColaZhang22/Tool-Roco
comment : 9 pages, 3 figures
♻ ☆ Beyond Token Length: Step Pruner for Efficient and Accurate Reasoning in Large Language Models
Canhui Wu, Qiong Cao, Chang Li, Zhenfang Wang, Chao Xue, Yuwei Fan, Wei Xi, Xiaodong He
Large Reasoning Models (LRMs) demonstrate strong performance on complex tasks but often suffer from excessive verbosity, known as "overthinking." Existing solutions via reinforcement learning (RL) typically penalize generated tokens to promote conciseness. However, these methods encounter two challenges: responses with fewer tokens do not always correspond to fewer reasoning steps, and models may develop hacking behavior in later stages of training by discarding reasoning steps to minimize token usage. In this work, we introduce \textbf{Step Pruner (SP)}, an RL framework that steers LRMs toward more efficient reasoning by favoring compact reasoning steps. Our step-aware reward function prioritizes correctness while imposing penalties for redundant steps, and withholds rewards for incorrect responses to prevent the reinforcement of erroneous reasoning. Moreover, we propose a dynamic stopping mechanism: when the model's output no longer shortens, training is halted to prevent hacking behavior caused by the merging of steps. Extensive experiments across four reasoning benchmarks demonstrate that SP achieves state-of-the-art accuracy while significantly reducing response length. For instance, on AIME24, SP reduces token usage by \textbf{69.7\%}.
comment : 21 pages, 9 figures
♻ ☆ Differentiable Optimization for Deep Learning-Enhanced DC Approximation of AC Optimal Power Flow
Andrew Rosemberg, Michael Klamkin, Pascal Van Hentenryck
The growing scale of power systems and the increasing uncertainty introduced by renewable energy sources necessitates novel optimization techniques that are significantly faster and more accurate than existing methods. The AC Optimal Power Flow (AC-OPF) problem, a core component of power grid optimization, is often approximated using linearized DC Optimal Power Flow (DC-OPF) models for computational tractability, albeit at the cost of suboptimal and inefficient decisions. To address these limitations, we propose a novel deep learning-based framework for network equivalency that enhances DC-OPF to more closely mimic the behavior of AC-OPF. The approach utilizes recent advances in differentiable optimization, incorporating a neural network trained to predict adjusted nodal shunt conductances and branch susceptances in order to account for nonlinear power flow behavior. The model can be trained end-to-end using modern deep learning frameworks by leveraging the implicit function theorem. Results demonstrate the framework's ability to significantly improve prediction accuracy.
Computer Vision and Pattern Recognition 86
☆ Fast, Robust, Permutation-and-Sign Invariant SO(3) Pattern Alignment
Anik Sarker, Alan T. Asbeck
We address the correspondence-free alignment of two rotation sets on \(SO(3)\), a core task in calibration and registration that is often impeded by missing time alignment, outliers, and unknown axis conventions. Our key idea is to decompose each rotation into its \emph{Transformed Basis Vectors} (TBVs)-three unit vectors on \(S^2\)-and align the resulting spherical point sets per axis using fast, robust matchers (SPMC, FRS, and a hybrid). To handle axis relabels and sign flips, we introduce a \emph{Permutation-and-Sign Invariant} (PASI) wrapper that enumerates the 24 proper signed permutations, scores them via summed correlations, and fuses the per-axis estimates into a single rotation by projection/Karcher mean. The overall complexity remains linear in the number of rotations (\(\mathcal{O}(n)\)), contrasting with \(\mathcal{O}(N_r^3\log N_r)\) for spherical/\(SO(3)\) correlation. Experiments on EuRoC Machine Hall simulations (axis-consistent) and the ETH Hand-Eye benchmark (\texttt{robot\_arm\_real}) (axis-ambiguous) show that our methods are accurate, 6-60x faster than traditional methods, and robust under extreme outlier ratios (up to 90\%), all without correspondence search.
☆ MambaScope: Coarse-to-Fine Scoping for Efficient Vision Mamba
Shanhui Liu, Rui Xu, Yunke Wang
Vision Mamba has emerged as a promising and efficient alternative to Vision Transformers, yet its efficiency remains fundamentally constrained by the number of input tokens. Existing token reduction approaches typically adopt token pruning or merging to reduce computation. However, they inherently lead to information loss, as they discard or compress token representations. This problem is exacerbated when applied uniformly to fine-grained token representations across all images, regardless of visual complexity. We observe that not all inputs require fine-grained processing. Simple images can be effectively handled at coarse resolution, while only complex ones may warrant refinement. Based on this insight, we propose \textit{Coarse-to-Fine Vision Mamba (CF-ViM)}, an adaptive framework for efficient inference. CF-ViM first performs coarse-grained inference by dividing the input image into large patches, significantly reducing the token length and computation. When the model's prediction confidence is low, selected regions are re-processed at a finer resolution to recover critical visual details with minimal additional cost. This dynamic resolution assignment strategy allows CF-ViM to allocate computation adaptively according to image complexity, ensuring efficient processing without compromising essential visual information. Experiments on ImageNet demonstrate that CF-ViM outperforms both the baseline Vision Mamba and state-of-the-art token reduction techniques in terms of accuracy and efficiency.
☆ Graph-Attention Network with Adversarial Domain Alignment for Robust Cross-Domain Facial Expression Recognition ACML 2025
Razieh Ghaedi, AmirReza BabaAhmadi, Reyer Zwiggelaar, Xinqi Fan, Nashid Alam
Cross-domain facial expression recognition (CD-FER) remains difficult due to severe domain shift between training and deployment data. We propose Graph-Attention Network with Adversarial Domain Alignment (GAT-ADA), a hybrid framework that couples a ResNet-50 as backbone with a batch-level Graph Attention Network (GAT) to model inter-sample relations under shift. Each mini-batch is cast as a sparse ring graph so that attention aggregates cross-sample cues that are informative for adaptation. To align distributions, GAT-ADA combines adversarial learning via a Gradient Reversal Layer (GRL) with statistical alignment using CORAL and MMD. GAT-ADA is evaluated under a standard unsupervised domain adaptation protocol: training on one labeled source (RAF-DB) and adapting to multiple unlabeled targets (CK+, JAFFE, SFEW 2.0, FER2013, and ExpW). GAT-ADA attains 74.39% mean cross-domain accuracy. On RAF-DB to FER2013, it reaches 98.0% accuracy, corresponding to approximately a 36-point improvement over the best baseline we re-implemented with the same backbone and preprocessing.
comment : 17 pages, 5 figures. Accepted at the 17th Asian Conference on Machine Learning (ACML 2025), Taipei, Taiwan, December 9-12, 2025
☆ Doppler-Enhanced Deep Learning: Improving Thyroid Nodule Segmentation with YOLOv5 Instance Segmentation
Mahmoud El Hussieni
The increasing prevalence of thyroid cancer globally has led to the development of various computer-aided detection methods. Accurate segmentation of thyroid nodules is a critical first step in the development of AI-assisted clinical decision support systems. This study focuses on instance segmentation of thyroid nodules using YOLOv5 algorithms on ultrasound images. We evaluated multiple YOLOv5 variants (Nano, Small, Medium, Large, and XLarge) across two dataset versions, with and without doppler images. The YOLOv5-Large algorithm achieved the highest performance with a dice score of 91\% and mAP of 0.87 on the dataset including doppler images. Notably, our results demonstrate that doppler images, typically excluded by physicians, can significantly improve segmentation performance. The YOLOv5-Small model achieved 79\% dice score when doppler images were excluded, while including them improved performance across all model variants. These findings suggest that instance segmentation with YOLOv5 provides an effective real-time approach for thyroid nodule detection, with potential clinical applications in automated diagnostic systems.
☆ XAI-Driven Skin Disease Classification: Leveraging GANs to Augment ResNet-50 Performance
Kim Gerard A. Villanueva, Priyanka Kumar
Accurate and timely diagnosis of multi-class skin lesions is hampered by subjective methods, inherent data imbalance in datasets like HAM10000, and the "black box" nature of Deep Learning (DL) models. This study proposes a trustworthy and highly accurate Computer-Aided Diagnosis (CAD) system to overcome these limitations. The approach utilizes Deep Convolutional Generative Adversarial Networks (DCGANs) for per class data augmentation to resolve the critical class imbalance problem. A fine-tuned ResNet-50 classifier is then trained on the augmented dataset to classify seven skin disease categories. Crucially, LIME and SHAP Explainable AI (XAI) techniques are integrated to provide transparency by confirming that predictions are based on clinically relevant features like irregular morphology. The system achieved a high overall Accuracy of 92.50 % and a Macro-AUC of 98.82 %, successfully outperforming various prior benchmarked architectures. This work successfully validates a verifiable framework that combines high performance with the essential clinical interpretability required for safe diagnostic deployment. Future research should prioritize enhancing discrimination for critical categories, such as Melanoma NOS (F1-Score is 0.8602).
☆ Automatic Pith Detection in Tree Cross-Section Images Using Deep Learning
Tzu-I Liao, Mahmoud Fakhry, Jibin Yesudas Varghese
Pith detection in tree cross-sections is essential for forestry and wood quality analysis but remains a manual, error-prone task. This study evaluates deep learning models -- YOLOv9, U-Net, Swin Transformer, DeepLabV3, and Mask R-CNN -- to automate the process efficiently. A dataset of 582 labeled images was dynamically augmented to improve generalization. Swin Transformer achieved the highest accuracy (0.94), excelling in fine segmentation. YOLOv9 performed well for bounding box detection but struggled with boundary precision. U-Net was effective for structured patterns, while DeepLabV3 captured multi-scale features with slight boundary imprecision. Mask R-CNN initially underperformed due to overlapping detections, but applying Non-Maximum Suppression (NMS) improved its IoU from 0.45 to 0.80. Generalizability was next tested using an oak dataset of 11 images from Oregon State University's Tree Ring Lab. Additionally, for exploratory analysis purposes, an additional dataset of 64 labeled tree cross-sections was used to train the worst-performing model to see if this would improve its performance generalizing to the unseen oak dataset. Key challenges included tensor mismatches and boundary inconsistencies, addressed through hyperparameter tuning and augmentation. Our results highlight deep learning's potential for tree cross-section pith detection, with model choice depending on dataset characteristics and application needs.
comment : 8 pages, 7 figures
☆ Scaling Down to Scale Up: Towards Operationally-Efficient and Deployable Clinical Models via Cross-Modal Low-Rank Adaptation for Medical Vision -Language Models
Thuraya Alzubaidi, Farhad R. Nezami, Muzammil Behzad
Foundation models trained via vision-language pretraining have demonstrated strong zero-shot capabilities across diverse image domains, yet their application to volumetric medical imaging remains limited. We introduce MedCT-VLM: Medical CT Vision-Language Model, a parameter-efficient vision-language framework designed to adapt large-scale CT foundation models for downstream clinical tasks. MedCT-VLM uses a parameter-efficient approach to adapt CT-CLIP, a contrastive vision-language model trained on 25,692 chest CT volumes, for multi-label pathology classification using Low-Rank Adaptation (LoRA). Rather than fine-tuning the model's 440 M parameters directly, we insert low-rank decomposition matrices into attention layers of both vision and text encoders, training only 1.67M parameters (0.38\% of total). We evaluate on zero-shot classification across 18 thoracic pathologies, where the model must align CT embeddings with unseen text prompts at inference without task-specific training. LoRA fine-tuning improves mean AUROC from 61.3\% to 68.9\% (+7.6 pp), accuracy from 67.2\% to 73.6\% (+6.4 pp), and macro-F1 from 32.1\% to 36.9\% (+4.8 pp). These results demonstrate that parameter-efficient methods can effectively transfer large-scale pretraining to downstream medical imaging tasks, particularly for zero-shot scenarios where labeled data is scarce.
☆ SatireDecoder: Visual Cascaded Decoupling for Enhancing Satirical Image Comprehension AAAI 2026
Yue Jiang, Haiwei Xue, Minghao Han, Mingcheng Li, Xiaolu Hou, Dingkang Yang, Lihua Zhang, Xu Zheng
Satire, a form of artistic expression combining humor with implicit critique, holds significant social value by illuminating societal issues. Despite its cultural and societal significance, satire comprehension, particularly in purely visual forms, remains a challenging task for current vision-language models. This task requires not only detecting satire but also deciphering its nuanced meaning and identifying the implicated entities. Existing models often fail to effectively integrate local entity relationships with global context, leading to misinterpretation, comprehension biases, and hallucinations. To address these limitations, we propose SatireDecoder, a training-free framework designed to enhance satirical image comprehension. Our approach proposes a multi-agent system performing visual cascaded decoupling to decompose images into fine-grained local and global semantic representations. In addition, we introduce a chain-of-thought reasoning strategy guided by uncertainty analysis, which breaks down the complex satire comprehension process into sequential subtasks with minimized uncertainty. Our method significantly improves interpretive accuracy while reducing hallucinations. Experimental results validate that SatireDecoder outperforms existing baselines in comprehending visual satire, offering a promising direction for vision-language reasoning in nuanced, high-level semantic tasks.
comment : Accepted by AAAI 2026
☆ Integrating Skeleton Based Representations for Robust Yoga Pose Classification Using Deep Learning Models
Mohammed Mohiuddin, Syed Mohammod Minhaz Hossain, Sumaiya Khanam, Prionkar Barua, Aparup Barua, MD Tamim Hossain
Yoga is a popular form of exercise worldwide due to its spiritual and physical health benefits, but incorrect postures can lead to injuries. Automated yoga pose classification has therefore gained importance to reduce reliance on expert practitioners. While human pose keypoint extraction models have shown high potential in action recognition, systematic benchmarking for yoga pose recognition remains limited, as prior works often focus solely on raw images or a single pose extraction model. In this study, we introduce a curated dataset, 'Yoga-16', which addresses limitations of existing datasets, and systematically evaluate three deep learning architectures (VGG16, ResNet50, and Xception) using three input modalities (direct images, MediaPipe Pose skeleton images, and YOLOv8 Pose skeleton images). Our experiments demonstrate that skeleton-based representations outperform raw image inputs, with the highest accuracy of 96.09% achieved by VGG16 with MediaPipe Pose skeleton input. Additionally, we provide interpretability analysis using Grad-CAM, offering insights into model decision-making for yoga pose classification with cross validation analysis.
☆ Describe Anything Anywhere At Any Moment
Nicolas Gorlo, Lukas Schmid, Luca Carlone
Computer vision and robotics applications ranging from augmented reality to robot autonomy in large-scale environments require spatio-temporal memory frameworks that capture both geometric structure for accurate language-grounding as well as semantic detail. Existing methods face a tradeoff, where producing rich open-vocabulary descriptions comes at the expense of real-time performance when these descriptions have to be grounded in 3D. To address these challenges, we propose Describe Anything, Anywhere, at Any Moment (DAAAM), a novel spatio-temporal memory framework for large-scale and real-time 4D scene understanding. DAAAM introduces a novel optimization-based frontend to infer detailed semantic descriptions from localized captioning models, such as the Describe Anything Model (DAM), leveraging batch processing to speed up inference by an order of magnitude for online processing. It leverages such semantic understanding to build a hierarchical 4D scene graph (SG), which acts as an effective globally spatially and temporally consistent memory representation. DAAAM constructs 4D SGs with detailed, geometrically grounded descriptions while maintaining real-time performance. We show that DAAAM's 4D SG interfaces well with a tool-calling agent for inference and reasoning. We thoroughly evaluate DAAAM in the complex task of spatio-temporal question answering on the NaVQA benchmark and show its generalization capabilities for sequential task grounding on the SG3D benchmark. We further curate an extended OC-NaVQA benchmark for large-scale and long-time evaluations. DAAAM achieves state-of-the-art results in both tasks, improving OC-NaVQA question accuracy by 53.6%, position errors by 21.9%, temporal errors by 21.6%, and SG3D task grounding accuracy by 27.8% over the most competitive baselines, respectively. We release our data and code open-source.
comment : 14 pages, 5 figures, 6 tables
☆ NeuroVolve: Evolving Visual Stimuli toward Programmable Neural Objectives
Haomiao Chen, Keith W Jamison, Mert R. Sabuncu, Amy Kuceyeski
What visual information is encoded in individual brain regions, and how do distributed patterns combine to create their neural representations? Prior work has used generative models to replicate known category selectivity in isolated regions (e.g., faces in FFA), but these approaches offer limited insight into how regions interact during complex, naturalistic vision. We introduce NeuroVolve, a generative framework that provides brain-guided image synthesis via optimization of a neural objective function in the embedding space of a pretrained vision-language model. Images are generated under the guidance of a programmable neural objective, i.e., activating or deactivating single regions or multiple regions together. NeuroVolve is validated by recovering known selectivity for individual brain regions, while expanding to synthesize coherent scenes that satisfy complex, multi-region constraints. By tracking optimization steps, it reveals semantic trajectories through embedding space, unifying brain-guided image editing and preferred stimulus generation in a single process. We show that NeuroVolve can generate both low-level and semantic feature-specific stimuli for single ROIs, as well as stimuli aligned to curated neural objectives. These include co-activation and decorrelation between regions, exposing cooperative and antagonistic tuning relationships. Notably, the framework captures subject-specific preferences, supporting personalized brain-driven synthesis and offering interpretable constraints for mapping, analyzing, and probing neural representations of visual information.
☆ Asset-Driven Sematic Reconstruction of Dynamic Scene with Multi-Human-Object Interactions
Sandika Biswas, Qianyi Wu, Biplab Banerjee, Hamid Rezatofighi
Real-world human-built environments are highly dynamic, involving multiple humans and their complex interactions with surrounding objects. While 3D geometry modeling of such scenes is crucial for applications like AR/VR, gaming, and embodied AI, it remains underexplored due to challenges like diverse motion patterns and frequent occlusions. Beyond novel view rendering, 3D Gaussian Splatting (GS) has demonstrated remarkable progress in producing detailed, high-quality surface geometry with fast optimization of the underlying structure. However, very few GS-based methods address multihuman, multiobject scenarios, primarily due to the above-mentioned inherent challenges. In a monocular setup, these challenges are further amplified, as maintaining structural consistency under severe occlusion becomes difficult when the scene is optimized solely based on GS-based rendering loss. To tackle the challenges of such a multihuman, multiobject dynamic scene, we propose a hybrid approach that effectively combines the advantages of 1) 3D generative models for generating high-fidelity meshes of the scene elements, 2) Semantic-aware deformation, \ie rigid transformation of the rigid objects and LBS-based deformation of the humans, and mapping of the deformed high-fidelity meshes in the dynamic scene, and 3) GS-based optimization of the individual elements for further refining their alignments in the scene. Such a hybrid approach helps maintain the object structures even under severe occlusion and can produce multiview and temporally consistent geometry. We choose HOI-M3 for evaluation, as, to the best of our knowledge, this is the only dataset featuring multihuman, multiobject interactions in a dynamic scene. Our method outperforms the state-of-the-art method in producing better surface reconstruction of such scenes.
☆ SAIDO: Generalizable Detection of AI-Generated Images via Scene-Aware and Importance-Guided Dynamic Optimization in Continual Learning
Yongkang Hu, Yu Cheng, Yushuo Zhang, Yuan Xie, Zhaoxia Yin
The widespread misuse of image generation technologies has raised security concerns, driving the development of AI-generated image detection methods. However, generalization has become a key challenge and open problem: existing approaches struggle to adapt to emerging generative methods and content types in real-world scenarios. To address this issue, we propose a Scene-Aware and Importance-Guided Dynamic Optimization detection framework with continual learning (SAIDO). Specifically, we design Scene-Awareness-Based Expert Module (SAEM) that dynamically identifies and incorporates new scenes using VLLMs. For each scene, independent expert modules are dynamically allocated, enabling the framework to capture scene-specific forgery features better and enhance cross-scene generalization. To mitigate catastrophic forgetting when learning from multiple image generative methods, we introduce Importance-Guided Dynamic Optimization Mechanism (IDOM), which optimizes each neuron through an importance-guided gradient projection strategy, thereby achieving an effective balance between model plasticity and stability. Extensive experiments on continual learning tasks demonstrate that our method outperforms the current SOTA method in both stability and plasticity, achieving 44.22\% and 40.57\% relative reductions in average detection error rate and forgetting rate, respectively. On open-world datasets, it improves the average detection accuracy by 9.47\% compared to the current SOTA method.
comment : 17 pages, 19 figures
☆ Cross-Temporal 3D Gaussian Splatting for Sparse-View Guided Scene Update AAAI2026
Zeyuan An, Yanghang Xiao, Zhiying Leng, Frederick W. B. Li, Xiaohui Liang
Maintaining consistent 3D scene representations over time is a significant challenge in computer vision. Updating 3D scenes from sparse-view observations is crucial for various real-world applications, including urban planning, disaster assessment, and historical site preservation, where dense scans are often unavailable or impractical. In this paper, we propose Cross-Temporal 3D Gaussian Splatting (Cross-Temporal 3DGS), a novel framework for efficiently reconstructing and updating 3D scenes across different time periods, using sparse images and previously captured scene priors. Our approach comprises three stages: 1) Cross-temporal camera alignment for estimating and aligning camera poses across different timestamps; 2) Interference-based confidence initialization to identify unchanged regions between timestamps, thereby guiding updates; and 3) Progressive cross-temporal optimization, which iteratively integrates historical prior information into the 3D scene to enhance reconstruction quality. Our method supports non-continuous capture, enabling not only updates using new sparse views to refine existing scenes, but also recovering past scenes from limited data with the help of current captures. Furthermore, we demonstrate the potential of this approach to achieve temporal changes using only sparse images, which can later be reconstructed into detailed 3D representations as needed. Experimental results show significant improvements over baseline methods in reconstruction quality and data efficiency, making this approach a promising solution for scene versioning, cross-temporal digital twins, and long-term spatial documentation.
comment : AAAI2026 accepted
☆ Image Generation as a Visual Planner for Robotic Manipulation CVPR 2026
Ye Pang
Generating realistic robotic manipulation videos is an important step toward unifying perception, planning, and action in embodied agents. While existing video diffusion models require large domain-specific datasets and struggle to generalize, recent image generation models trained on language-image corpora exhibit strong compositionality, including the ability to synthesize temporally coherent grid images. This suggests a latent capacity for video-like generation even without explicit temporal modeling. We explore whether such models can serve as visual planners for robots when lightly adapted using LoRA finetuning. We propose a two-part framework that includes: (1) text-conditioned generation, which uses a language instruction and the first frame, and (2) trajectory-conditioned generation, which uses a 2D trajectory overlay and the same initial frame. Experiments on the Jaco Play dataset, Bridge V2, and the RT1 dataset show that both modes produce smooth, coherent robot videos aligned with their respective conditions. Our findings indicate that pretrained image generators encode transferable temporal priors and can function as video-like robotic planners under minimal supervision. Code is released at \href{https://github.com/pangye202264690373/Image-Generation-as-a-Visual-Planner-for-Robotic-Manipulation}{https://github.com/pangye202264690373/Image-Generation-as-a-Visual-Planner-for-Robotic-Manipulation}.
comment : 11 pages 9 figures Under review at CVPR 2026
☆ Terrain Sensing with Smartphone Structured Light: 2D Dynamic Time Warping for Grid Pattern Matching
Tanaka Nobuaki
Low-cost mobile rovers often operate on uneven terrain where small bumps or tilts are difficult to perceive visually but can significantly affect locomotion stability. To address this problem, we explore a smartphone-based structured-light system that projects a grid pattern onto the ground and reconstructs local terrain unevenness from a single handheld device. The system is inspired by face-recognition projectors, but adapted for ground sensing. A key technical challenge is robustly matching the projected grid with its deformed observation under perspective distortion and partial occlusion. Conventional one-dimensional dynamic time warping (1D-DTW) is not directly applicable to such two-dimensional grid patterns. We therefore propose a topology-constrained two-dimensional dynamic time warping (2D-DTW) algorithm that performs column-wise alignment under a global grid consistency constraint. The proposed method is designed to be simple enough to run on resource limited platforms while preserving the grid structure required for accurate triangulation. We demonstrate that our 2D-DTW formulation can be used not only for terrain sensing but also as a general tool for matching structured grid patterns in image processing scenarios. This paper describes the overall system design as well as the 2D-DTW extension that emerged from this application.
☆ CC-FMO: Camera-Conditioned Zero-Shot Single Image to 3D Scene Generation with Foundation Model Orchestration
Boshi Tang, Henry Zheng, Rui Huang, Gao Huang
High-quality 3D scene generation from a single image is crucial for AR/VR and embodied AI applications. Early approaches struggle to generalize due to reliance on specialized models trained on curated small datasets. While recent advancements in large-scale 3D foundation models have significantly enhanced instance-level generation, coherent scene generation remains a challenge, where performance is limited by inaccurate per-object pose estimations and spatial inconsistency. To this end, this paper introduces CC-FMO, a zero-shot, camera-conditioned pipeline for single-image to 3D scene generation that jointly conforms to the object layout in input image and preserves instance fidelity. CC-FMO employs a hybrid instance generator that combines semantics-aware vector-set representation with detail-rich structured latent representation, yielding object geometries that are both semantically plausible and high-quality. Furthermore, CC-FMO enables the application of foundational pose estimation models in the scene generation task via a simple yet effective camera-conditioned scale-solving algorithm, to enforce scene-level coherence. Extensive experiments demonstrate that CC-FMO consistently generates high-fidelity camera-aligned compositional scenes, outperforming all state-of-the-art methods.
☆ Learning What Helps: Task-Aligned Context Selection for Vision Tasks
Jingyu Guo, Emir Konuk, Fredrik Strand, Christos Matsoukas, Kevin Smith
Humans often resolve visual uncertainty by comparing an image with relevant examples, but ViTs lack the ability to identify which examples would improve their predictions. We present Task-Aligned Context Selection (TACS), a framework that learns to select paired examples which truly improve task performance rather than those that merely appear similar. TACS jointly trains a selector network with the task model through a hybrid optimization scheme combining gradient-based supervision and reinforcement learning, making retrieval part of the learning objective. By aligning selection with task rewards, TACS enables discriminative models to discover which contextual examples genuinely help. Across 18 datasets covering fine-grained recognition, medical image classification, and medical image segmentation, TACS consistently outperforms similarity-based retrieval, particularly in challenging or data-limited settings.
☆ Structured Context Learning for Generic Event Boundary Detection
Xin Gu, Congcong Li, Xinyao Wang, Dexiang Hong, Libo Zhang, Tiejian Luo, Longyin Wen, Heng Fan
Generic Event Boundary Detection (GEBD) aims to identify moments in videos that humans perceive as event boundaries. This paper proposes a novel method for addressing this task, called Structured Context Learning, which introduces the Structured Partition of Sequence (SPoS) to provide a structured context for learning temporal information. Our approach is end-to-end trainable and flexible, not restricted to specific temporal models like GRU, LSTM, and Transformers. This flexibility enables our method to achieve a better speed-accuracy trade-off. Specifically, we apply SPoS to partition the input frame sequence and provide a structured context for the subsequent temporal model. Notably, SPoS's overall computational complexity is linear with respect to the video length. We next calculate group similarities to capture differences between frames, and a lightweight fully convolutional network is utilized to determine the event boundaries based on the grouped similarity maps. To remedy the ambiguities of boundary annotations, we adapt the Gaussian kernel to preprocess the ground-truth event boundaries. Our proposed method has been extensively evaluated on the challenging Kinetics-GEBD, TAPOS, and shot transition detection datasets, demonstrating its superiority over existing state-of-the-art methods.
☆ RealGen: Photorealistic Text-to-Image Generation via Detector-Guided Rewards
Junyan Ye, Leiqi Zhu, Yuncheng Guo, Dongzhi Jiang, Zilong Huang, Yifan Zhang, Zhiyuan Yan, Haohuan Fu, Conghui He, Weijia Li
With the continuous advancement of image generation technology, advanced models such as GPT-Image-1 and Qwen-Image have achieved remarkable text-to-image consistency and world knowledge However, these models still fall short in photorealistic image generation. Even on simple T2I tasks, they tend to produce " fake" images with distinct AI artifacts, often characterized by "overly smooth skin" and "oily facial sheens". To recapture the original goal of "indistinguishable-from-reality" generation, we propose RealGen, a photorealistic text-to-image framework. RealGen integrates an LLM component for prompt optimization and a diffusion model for realistic image generation. Inspired by adversarial generation, RealGen introduces a "Detector Reward" mechanism, which quantifies artifacts and assesses realism using both semantic-level and feature-level synthetic image detectors. We leverage this reward signal with the GRPO algorithm to optimize the entire generation pipeline, significantly enhancing image realism and detail. Furthermore, we propose RealBench, an automated evaluation benchmark employing Detector-Scoring and Arena-Scoring. It enables human-free photorealism assessment, yielding results that are more accurate and aligned with real user experience. Experiments demonstrate that RealGen significantly outperforms general models like GPT-Image-1 and Qwen-Image, as well as specialized photorealistic models like FLUX-Krea, in terms of realism, detail, and aesthetics. The code is available at https://github.com/yejy53/RealGen.
☆ CausalAffect: Causal Discovery for Facial Affective Understanding
Guanyu Hu, Tangzheng Lian, Dimitrios Kollias, Oya Celiktutan, Xinyu Yang
Understanding human affect from facial behavior requires not only accurate recognition but also structured reasoning over the latent dependencies that drive muscle activations and their expressive outcomes. Although Action Units (AUs) have long served as the foundation of affective computing, existing approaches rarely address how to infer psychologically plausible causal relations between AUs and expressions directly from data. We propose CausalAffect, the first framework for causal graph discovery in facial affect analysis. CausalAffect models AU-AU and AU-Expression dependencies through a two-level polarity and direction aware causal hierarchy that integrates population-level regularities with sample-adaptive structures. A feature-level counterfactual intervention mechanism further enforces true causal effects while suppressing spurious correlations. Crucially, our approach requires neither jointly annotated datasets nor handcrafted causal priors, yet it recovers causal structures consistent with established psychological theories while revealing novel inhibitory and previously uncharacterized dependencies. Extensive experiments across six benchmarks demonstrate that CausalAffect advances the state of the art in both AU detection and expression recognition, establishing a principled connection between causal discovery and interpretable facial behavior. All trained models and source code will be released upon acceptance.
☆ RecruitView: A Multimodal Dataset for Predicting Personality and Interview Performance for Human Resources Applications
Amit Kumar Gupta, Farhan Sheth, Hammad Shaikh, Dheeraj Kumar, Angkul Puniya, Deepak Panwar, Sandeep Chaurasia, Priya Mathur
Automated personality and soft skill assessment from multimodal behavioral data remains challenging due to limited datasets and methods that fail to capture geometric structure inherent in human traits. We introduce RecruitView, a dataset of 2,011 naturalistic video interview clips from 300+ participants with 27,000 pairwise comparative judgments across 12 dimensions: Big Five personality traits, overall personality score, and six interview performance metrics. To leverage this data, we propose Cross-Modal Regression with Manifold Fusion (CRMF), a geometric deep learning framework that explicitly models behavioral representations across hyperbolic, spherical, and Euclidean manifolds. CRMF employs geometry-specific expert networks to capture hierarchical trait structures, directional behavioral patterns, and continuous performance variations simultaneously. An adaptive routing mechanism dynamically weights expert contributions based on input characteristics. Through principled tangent space fusion, CRMF achieves superior performance while training 40-50% fewer trainable parameters than large multimodal models. Extensive experiments demonstrate that CRMF substantially outperforms the selected baselines, achieving up to 11.4% improvement in Spearman correlation and 6.0% in concordance index. Our RecruitView dataset is publicly available at https://huggingface.co/datasets/AI4A-lab/RecruitView
comment : 20 pages, 10 figures, 10 tables
☆ FR-TTS: Test-Time Scaling for NTP-based Image Generation with Effective Filling-based Reward Signal
Hang Xu, Linjiang Huang, Feng Zhao
Test-time scaling (TTS) has become a prevalent technique in image generation, significantly boosting output quality by expanding the number of parallel samples and filtering them using pre-trained reward models. However, applying this powerful methodology to the next-token prediction (NTP) paradigm remains challenging. The primary obstacle is the low correlation between the reward of an image decoded from an intermediate token sequence and the reward of the fully generated image. Consequently, these incomplete intermediate representations prove to be poor indicators for guiding the pruning direction, a limitation that stems from their inherent incompleteness in scale or semantic content. To effectively address this critical issue, we introduce the Filling-Based Reward (FR). This novel design estimates the approximate future trajectory of an intermediate sample by finding and applying a reasonable filling scheme to complete the sequence. Both the correlation coefficient between rewards of intermediate samples and final samples, as well as multiple intrinsic signals like token confidence, indicate that the FR provides an excellent and reliable metric for accurately evaluating the quality of intermediate samples. Building upon this foundation, we propose FR-TTS, a sophisticated scaling strategy. FR-TTS efficiently searches for good filling schemes and incorporates a diversity reward with a dynamic weighting schedule to achieve a balanced and comprehensive evaluation of intermediate samples. We experimentally validate the superiority of FR-TTS over multiple established benchmarks and various reward models. Code is available at \href{https://github.com/xuhang07/FR-TTS}{https://github.com/xuhang07/FR-TTS}.
☆ Recognizing Pneumonia in Real-Wo rl d Chest X-rays with a Classifier Trained with Images Synthetically Generated by Nano Banana
Jiachuan Peng, Kyle Lam, Jianing Qiu
We trained a classifier with synthetic chest X-ray (CXR) images generated by Nano Banana, the latest AI model for image generation and editing, released by Google. When directly applied to real-world CXRs having only been trained with synthetic data, the classifier achieved an AUROC of 0.923 (95% CI: 0.919 - 0.927), and an AUPR of 0.900 (95% CI: 0.894 - 0.907) in recognizing pneumonia in the 2018 RSNA Pneumonia Detection dataset (14,863 CXRs), and an AUROC of 0.824 (95% CI: 0.810 - 0.836), and an AUPR of 0.913 (95% CI: 0.904 - 0.922) in the Chest X-Ray dataset (5,856 CXRs). These external validation results on real-world data demonstrate the feasibility of this approach and suggest potential for synthetic data in medical AI development. Nonetheless, several limitations remain at present, including challenges in prompt design for controlling the diversity of synthetic CXR data and the requirement for post-processing to ensure alignment with real-world data. However, the growing sophistication and accessibility of medical intelligence will necessitate substantial validation, regulatory approval, and ethical oversight prior to clinical translation.
comment : 9 pages
☆ What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewards
Minh-Quan Le, Yuanzhi Zhu, Vicky Kalogeiton, Dimitris Samaras
Recent video diffusion models can synthesize visually compelling clips, yet often violate basic physical laws-objects float, accelerations drift, and collisions behave inconsistently-revealing a persistent gap between visual realism and physical realism. We propose $\texttt{NewtonRewards}$, the first physics-grounded post-training framework for video generation based on $\textit{verifiable rewards}$. Instead of relying on human or VLM feedback, $\texttt{NewtonRewards}$ extracts $\textit{measurable proxies}$ from generated videos using frozen utility models: optical flow serves as a proxy for velocity, while high-level appearance features serve as a proxy for mass. These proxies enable explicit enforcement of Newtonian structure through two complementary rewards: a Newtonian kinematic constraint enforcing constant-acceleration dynamics, and a mass conservation reward preventing trivial, degenerate solutions. We evaluate $\texttt{NewtonRewards}$ on five Newtonian Motion Primitives (free fall, horizontal/parabolic throw, and ramp sliding down/up) using our newly constructed large-scale benchmark, $\texttt{NewtonBench-60K}$. Across all primitives in visual and physics metrics, $\texttt{NewtonRewards}$ consistently improves physical plausibility, motion smoothness, and temporal coherence over prior post-training methods. It further maintains strong performance under out-of-distribution shifts in height, speed, and friction. Our results show that physics-grounded verifiable rewards offer a scalable path toward physics-aware video generation.
comment : Project page: https://cvlab-stonybrook.github.io/NewtonRewards
☆ Recovering Origin Destination Flows from Bus CCTV: Ea rl y Results from Nairobi and Kigali
Nthenya Kyatha, Jay Taneja
Public transport in sub-Saharan Africa (SSA) often operates in overcrowded conditions where existing automated systems fail to capture reliable passenger flow data. Leveraging onboard CCTV already deployed for security, we present a baseline pipeline that combines YOLOv12 detection, BotSORT tracking, OSNet embeddings, OCR-based timestamping, and telematics-based stop classification to recover bus origin--destination (OD) flows. On annotated CCTV segments from Nairobi and Kigali buses, the system attains high counting accuracy under low-density, well-lit conditions (recall $\approx$95\%, precision $\approx$91\%, F1 $\approx$93\%). It produces OD matrices that closely match manual tallies. Under realistic stressors such as overcrowding, color-to-monochrome shifts, posture variation, and non-standard door use, performance degrades sharply (e.g., $\sim$40\% undercount in peak-hour boarding and a $\sim$17 percentage-point drop in recall for monochrome segments), revealing deployment-specific failure modes and motivating more robust, deployment-focused Re-ID methods for SSA transit.
☆ PhysGen: Physically Grounded 3D Shape Generation for Industrial Design
Yingxuan You, Chen Zhao, Hantao Zhang, Mingda Xu, Pascal Fua
Existing generative models for 3D shapes can synthesize high-fidelity and visually plausible shapes. For certain classes of shapes that have undergone an engineering design process, the realism of the shape is tightly coupled with the underlying physical properties, e.g., aerodynamic efficiency for automobiles. Since existing methods lack knowledge of such physics, they are unable to use this knowledge to enhance the realism of shape generation. Motivated by this, we propose a unified physics-based 3D shape generation pipeline, with a focus on industrial design applications. Specifically, we introduce a new flow matching model with explicit physical guidance, consisting of an alternating update process. We iteratively perform a velocity-based update and a physics-based refinement, progressively adjusting the latent code to align with the desired 3D shapes and physical properties. We further strengthen physical validity by incorporating a physics-aware regularization term into the velocity-based update step. To support such physics-guided updates, we build a shape-and-physics variational autoencoder (SP-VAE) that jointly encodes shape and physics information into a unified latent space. The experiments on three benchmarks show that this synergistic formulation improves shape realism beyond mere visual plausibility.
comment : 14 pages, 10 figures
☆ SplatFont3D: Structure-Aware Text-to-3D Artistic Font Generation with Part-Level Style Control
Ji Gan, Lingxu Chen, Jiaxu Leng, Xinbo Gao
Artistic font generation (AFG) can assist human designers in creating innovative artistic fonts. However, most previous studies primarily focus on 2D artistic fonts in flat design, leaving personalized 3D-AFG largely underexplored. 3D-AFG not only enables applications in immersive 3D environments such as video games and animations, but also may enhance 2D-AFG by rendering 2D fonts of novel views. Moreover, unlike general 3D objects, 3D fonts exhibit precise semantics with strong structural constraints and also demand fine-grained part-level style control. To address these challenges, we propose SplatFont3D, a novel structure-aware text-to-3D AFG framework with 3D Gaussian splatting, which enables the creation of 3D artistic fonts from diverse style text prompts with precise part-level style control. Specifically, we first introduce a Glyph2Cloud module, which progressively enhances both the shapes and styles of 2D glyphs (or components) and produces their corresponding 3D point clouds for Gaussian initialization. The initialized 3D Gaussians are further optimized through interaction with a pretrained 2D diffusion model using score distillation sampling. To enable part-level control, we present a dynamic component assignment strategy that exploits the geometric priors of 3D Gaussians to partition components, while alleviating drift-induced entanglement during 3D Gaussian optimization. Our SplatFont3D provides more explicit and effective part-level style control than NeRF, attaining faster rendering efficiency. Experiments show that our SplatFont3D outperforms existing 3D models for 3D-AFG in style-text consistency, visual quality, and rendering efficiency.
☆ Low-Bitrate Video Compression through Semantic-Conditioned Diffusion
Lingdong Wang, Guan-Ming Su, Divya Kothandaraman, Tsung-Wei Huang, Mohammad Hajiesmaili, Ramesh K. Sitaraman
Traditional video codecs optimized for pixel fidelity collapse at ultra-low bitrates and produce severe artifacts. This failure arises from a fundamental misalignment between pixel accuracy and human perception. We propose a semantic video compression framework named DiSCo that transmits only the most meaningful information while relying on generative priors for detail synthesis. The source video is decomposed into three compact modalities: a textual description, a spatiotemporally degraded video, and optional sketches or poses that respectively capture semantic, appearance, and motion cues. A conditional video diffusion model then reconstructs high-quality, temporally coherent videos from these compact representations. Temporal forward filling, token interleaving, and modality-specific codecs are proposed to improve multimodal generation and modality compactness. Experiments show that our method outperforms baseline semantic and traditional codecs by 2-10X on perceptual metrics at low bitrates.
☆ SelfAI: Building a Self-Training AI System with LLM Agent s
Xiao Wu, Ting-Zhu Huang, Liang-Jian Deng, Xiaobing Yu, Yu Zhong, Shangqi Deng, Ufaq Khan, Jianghao Wu, Xiaofeng Liu, Imran Razzak, Xiaojun Chang, Yutong Xie
Recent work on autonomous scientific discovery has leveraged LLM-based agents to integrate problem specification, experiment planning, and execution into end-to-end systems. However, these frameworks are often confined to narrow application domains, offer limited real-time interaction with researchers, and lack principled mechanisms for determining when to halt exploration, resulting in inefficiencies, reproducibility challenges, and under-utilized human expertise. To address these gaps, we propose \textit{SelfAI}, a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations, a Cognitive Agent powered by LLMs with optimal stopping criteria to iteratively refine hyperparameter searches, and an Experiment Manager responsible for orchestrating parallel, fault-tolerant training workflows across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback. We further introduce two novel evaluation metrics, Score and $\text{AUP}_D$, to quantify discovery efficiency and search diversity. Across regression, NLP, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials compared to classical Bayesian optimization and LLM-based baselines, while enabling seamless interaction with human researchers.
☆ Time-Series at the Edge: Tiny Separable CNNs for Wearable Gait Detection and Optimal Sensor Placement
Andrea Procopio, Marco Esposito, Sara Raggiunto, Andrey Gizdov, Alberto Belli, Paola Pierleoni
We study on-device time-series analysis for gait detection in Parkinson's disease (PD) from short windows of triaxial acceleration, targeting resource-constrained wearables and edge nodes. We compare magnitude thresholding to three 1D CNNs for time-series analysis: a literature baseline (separable convolutions) and two ultra-light models - one purely separable and one with residual connections. Using the BioStampRC21 dataset, 2 s windows at 30 Hz, and subject-independent leave-one-subject-out (LOSO) validation on 16 PwPD with chest-worn IMUs, our residual separable model (Model 2, 533 params) attains PR-AUC = 94.5%, F1 = 91.2%, MCC = 89.4%, matching or surpassing the baseline (5,552 params; PR-AUC = 93.7%, F1 = 90.5%, MCC = 88.5%) with approximately 10x fewer parameters. The smallest model (Model 1, 305 params) reaches PR-AUC = 94.0%, F1 = 91.0%, MCC = 89.1%. Thresholding obtains high recall (89.0%) but low precision (76.5%), yielding many false positives and high inter-subject variance. Sensor-position analysis (train-on-all) shows chest and thighs are most reliable; forearms degrade precision/recall due to non-gait arm motion; naive fusion of all sites does not outperform the best single site. Both compact CNNs execute within tight memory/latency budgets on STM32-class MCUs (sub-10 ms on low-power boards), enabling on-sensor gating of transmission/storage. Overall, ultra-light separable CNNs provide a superior accuracy-efficiency-generalization trade-off to fixed thresholds for wearable PD gait detection and underscore the value of tailored time-series models for edge deployment.
☆ Better, Stronger, Faster: Tackling the Trilemma in MLLM-based Segmentation with Simultaneous Textual Mask Prediction
Jiazhen Liu, Mingkuan Feng, Long Chen
Integrating segmentation into Multimodal Large Language Models (MLLMs) presents a core trilemma: simultaneously preserving dialogue ability, achieving high segmentation performance, and ensuring fast inference. Prevailing paradigms are forced into a compromise. Embedding prediction methods introduce a conflicting pixel-level objective that degrades the MLLM's general dialogue abilities. The alternative, next-token prediction, reframes segmentation as an autoregressive task, which preserves dialogue but forces a trade-off between poor segmentation performance with sparse outputs or prohibitive inference speeds with rich ones. We resolve this trilemma with all-mask prediction, a novel paradigm that decouples autoregressive dialogue generation from non-autoregressive mask prediction. We present STAMP: Simultaneous Textual All-Mask Prediction, an MLLM that embodies this paradigm. After generating a textual response, STAMP predicts an entire segmentation mask in a single forward pass by treating it as a parallel "fill-in-the-blank" task over image patches. This design maintains the MLLM's dialogue ability by avoiding conflicting objectives, enables high segmentation performance by leveraging rich, bidirectional spatial context for all mask tokens, and achieves exceptional speed. Extensive experiments show that STAMP significantly outperforms state-of-the-art methods across multiple segmentation benchmarks, providing a solution that excels in dialogue, segmentation, and speed without compromise.
☆ WiseEdit: Benchmark ing Cognition- and Creativity-Informed Image Editing
Kaihang Pan, Weile Chen, Haiyi Qiu, Qifan Yu, Wendong Bu, Zehan Wang, Yun Zhu, Juncheng Li, Siliang Tang
Recent image editing models boast next-level intelligent capabilities, facilitating cognition- and creativity-informed image editing. Yet, existing benchmarks provide too narrow a scope for evaluation, failing to holistically assess these advanced abilities. To address this, we introduce WiseEdit, a knowledge-intensive benchmark for comprehensive evaluation of cognition- and creativity-informed image editing, featuring deep task depth and broad knowledge breadth. Drawing an analogy to human cognitive creation, WiseEdit decomposes image editing into three cascaded steps, i.e., Awareness, Interpretation, and Imagination, each corresponding to a task that poses a challenge for models to complete at the specific step. It also encompasses complex tasks, where none of the three steps can be finished easily. Furthermore, WiseEdit incorporates three fundamental types of knowledge: Declarative, Procedural, and Metacognitive knowledge. Ultimately, WiseEdit comprises 1,220 test cases, objectively revealing the limitations of SoTA image editing models in knowledge-based cognitive reasoning and creative composition capabilities. The benchmark, evaluation code, and the generated images of each model will be made publicly available soon. Project Page: https://qnancy.github.io/wiseedit_project_page/.
comment : 32 pages, 20 figures. Project Page: https://qnancy.github.io/wiseedit_project_page/
☆ EZ-SP: Fast and Lightweight Superpoint-Based 3D Segmentation
Louis Geist, Loic Landrieu, Damien Robert
Superpoint-based pipelines provide an efficient alternative to point- or voxel-based 3D semantic segmentation, but are often bottlenecked by their CPU-bound partition step. We propose a learnable, fully GPU partitioning algorithm that generates geometrically and semantically coherent superpoints 13$\times$ faster than prior methods. Our module is compact (under 60k parameters), trains in under 20 minutes with a differentiable surrogate loss, and requires no handcrafted features. Combine with a lightweight superpoint classifier, the full pipeline fits in $<$2 MB of VRAM, scales to multi-million-point scenes, and supports real-time inference. With 72$\times$ faster inference and 120$\times$ fewer parameters, EZ-SP matches the accuracy of point-based SOTA models across three domains: indoor scans (S3DIS), autonomous driving (KITTI-360), and aerial LiDAR (DALES). Code and pretrained models are accessible at github.com/drprojects/superpoint_transformer.
☆ Pore-scale Image Patch Dataset and A Comparative Evaluation of Pore-scale Facial Features
Dong Li, HuaLiang Lin, JiaYu Li
The weak-texture nature of facial skin regions presents significant challenges for local descriptor matching in applications such as facial motion analysis and 3D face reconstruction. Although deep learning-based descriptors have demonstrated superior performance to traditional hand-crafted descriptors in many applications, the scarcity of pore-scale image patch datasets has hindered their further development in the facial domain. In this paper, we propose the PorePatch dataset, a high-quality pore-scale image patch dataset, and establish a rational evaluation benchmark. We introduce a Data-Model Co-Evolution (DMCE) framework to generate a progressively refined, high-quality dataset from high-resolution facial images. We then train existing SOTA models on our dataset and conduct extensive experiments. Our results show that the SOTA model achieves a FPR95 value of 1.91% on the matching task, outperforming PSIFT (22.41%) by a margin of 20.5%. However, its advantage is diminished in the 3D reconstruction task, where its overall performance is not significantly better than that of traditional descriptors. This indicates that deep learning descriptors still have limitations in addressing the challenges of facial weak-texture regions, and much work remains to be done in this field.
☆ POLARIS: Projection-Orthogonal Least Squares for Robust and Adaptive Inversion in Diffusion Models
Wenshuo Chen, Haosen Li, Shaofeng Liang, Lei Wang, Haozhe Jia, Kaishen Yuan, Jieming Wu, Bowen Tian, Yutao Yue
The Inversion-Denoising Paradigm, which is based on diffusion models, excels in diverse image editing and restoration tasks. We revisit its mechanism and reveal a critical, overlooked factor in reconstruction degradation: the approximate noise error. This error stems from approximating the noise at step t with the prediction at step t-1, resulting in severe error accumulation throughout the inversion process. We introduce Projection-Orthogonal Least Squares for Robust and Adaptive Inversion (POLARIS), which reformulates inversion from an error-compensation problem into an error-origin problem. Rather than optimizing embeddings or latent codes to offset accumulated drift, POLARIS treats the guidance scale ω as a step-wise variable and derives a mathematically grounded formula to minimize inversion error at each step. Remarkably, POLARIS improves inversion latent quality with just one line of code. With negligible performance overhead, it substantially mitigates noise approximation errors and consistently improves the accuracy of downstream tasks.
☆ THC RL : Trusted Hierarchical Contrastive Representation Learning for Multi-View Clustering
Jian Zhu
Multi-View Clustering (MVC) has garnered increasing attention in recent years. It is capable of partitioning data samples into distinct groups by learning a consensus representation. However, a significant challenge remains: the problem of untrustworthy fusion. This problem primarily arises from two key factors: 1) Existing methods often ignore the presence of inherent noise within individual views; 2) In traditional MVC methods using Contrastive Learning (CL), similarity computations typically rely on different views of the same instance, while neglecting the structural information from nearest neighbors within the same cluster. Consequently, this leads to the wrong direction for multi-view fusion. To address this problem, we present a novel Trusted Hierarchical Contrastive Representation Learning (THCRL). It consists of two key modules. Specifically, we propose the Deep Symmetry Hierarchical Fusion (DSHF) module, which leverages the UNet architecture integrated with multiple denoising mechanisms to achieve trustworthy fusion of multi-view data. Furthermore, we present the Average K-Nearest Neighbors Contrastive Learning (AKCL) module to align the fused representation with the view-specific representation. Unlike conventional strategies, AKCL enhances representation similarity among samples belonging to the same cluster, rather than merely focusing on the same sample across views, thereby reinforcing the confidence of the fused representation. Extensive experiments demonstrate that THCRL achieves the state-of-the-art performance in deep MVC tasks.
☆ Towards aligned body representations in vision models
Andrey Gizdov, Andrea Procopio, Yichen Li, Daniel Harari, Tomer Ullman
Human physical reasoning relies on internal "body" representations - coarse, volumetric approximations that capture an object's extent and support intuitive predictions about motion and physics. While psychophysical evidence suggests humans use such coarse representations, their internal structure remains largely unknown. Here we test whether vision models trained for segmentation develop comparable representations. We adapt a psychophysical experiment conducted with 50 human participants to a semantic segmentation task and test a family of seven segmentation networks, varying in size. We find that smaller models naturally form human-like coarse body representations, whereas larger models tend toward overly detailed, fine-grain encodings. Our results demonstrate that coarse representations can emerge under limited computational resources, and that machine representations can provide a scalable path toward understanding the structure of physical reasoning in the brain.
comment : Andrea Procopio and Andrey Gizdov have equal contributions
☆ MM-DETR: An Efficient Multimodal Detection Transformer with Mamba-Driven Dual-Granularity Fusion and Frequency-Aware Modality Adapters
Jianhong Han, Yupei Wang, Yuan Zhang, Liang Chen
Multimodal remote sensing object detection aims to achieve more accurate and robust perception under challenging conditions by fusing complementary information from different modalities. However, existing approaches that rely on attention-based or deformable convolution fusion blocks still struggle to balance performance and lightweight design. Beyond fusion complexity, extracting modality features with shared backbones yields suboptimal representations due to insufficient modality-specific modeling, whereas dual-stream architectures nearly double the parameter count, ultimately limiting practical deployment. To this end, we propose MM-DETR, a lightweight and efficient framework for multimodal object detection. Specifically, we propose a Mamba-based dual granularity fusion encoder that reformulates global interaction as channel-wise dynamic gating and leverages a 1D selective scan for efficient cross-modal modeling with linear complexity. Following this design, we further reinterpret multimodal fusion as a modality completion problem. A region-aware 2D selective scanning completion branch is introduced to recover modality-specific cues, supporting fine-grained fusion along a bidirectional pyramid pathway with minimal overhead. To further reduce parameter redundancy while retaining strong feature extraction capability, a lightweight frequency-aware modality adapter is inserted into the shared backbone. This adapter employs a spatial-frequency co-expert structure to capture modality-specific cues, while a pixel-wise router dynamically balances expert contributions for efficient spatial-frequency fusion. Extensive experiments conducted on four multimodal benchmark datasets demonstrate the effectiveness and generalization capability of the proposed method.
comment : Manuscript submitted to IEEE Transactions on Geoscience and Remote Sensing
☆ SMamDiff: Spatial Mamba for Stochastic Human Motion Prediction
Junqiao Fan, Pengfei Liu, Haocong Rao
With intelligent room-side sensing and service robots widely deployed, human motion prediction (HMP) is essential for safe, proactive assistance. However, many existing HMP methods either produce a single, deterministic forecast that ignores uncertainty or rely on probabilistic models that sacrifice kinematic plausibility. Diffusion models improve the accuracy-diversity trade-off but often depend on multi-stage pipelines that are costly for edge deployment. This work focuses on how to ensure spatial-temporal coherence within a single-stage diffusion model for HMP. We introduce SMamDiff, a Spatial Mamba-based Diffusion model with two novel designs: (i) a residual-DCT motion encoding that subtracts the last observed pose before a temporal DCT, reducing the first DC component ($f=0$) dominance and highlighting informative higher-frequency cues so the model learns how joints move rather than where they are; and (ii) a stickman-drawing spatial-mamba module that processes joints in an ordered, joint-by-joint manner, making later joints condition on earlier ones to induce long-range, cross-joint dependencies. On Human3.6M and HumanEva, these coherence mechanisms deliver state-of-the-art results among single-stage probabilistic HMP methods while using less latency and memory than multi-stage diffusion baselines.
☆ MedCondDiff: Lightweight, Robust, Semantically Guided Diffusion for Medical Image Segmentation
Ruirui Huang, Jiacheng Li
We introduce MedCondDiff, a diffusion-based framework for multi-organ medical image segmentation that is efficient and anatomically grounded. The model conditions the denoising process on semantic priors extracted by a Pyramid Vision Transformer (PVT) backbone, yielding a semantically guided and lightweight diffusion architecture. This design improves robustness while reducing both inference time and VRAM usage compared to conventional diffusion models. Experiments on multi-organ, multi-modality datasets demonstrate that MedCondDiff delivers competitive performance across anatomical regions and imaging modalities, underscoring the potential of semantically guided diffusion models as an effective class of architectures for medical imaging tasks.
☆ mmPred: Radar-based Human Motion Prediction in the Dark AAAI-2026
Junqiao Fan, Haocong Rao, Jiarui Zhang, Jianfei Yang, Lihua Xie
Existing Human Motion Prediction (HMP) methods based on RGB-D cameras are sensitive to lighting conditions and raise privacy concerns, limiting their real-world applications such as firefighting and healthcare. Motivated by the robustness and privacy-preserving nature of millimeter-wave (mmWave) radar, this work introduces radar as a novel sensing modality for HMP, for the first time. Nevertheless, radar signals often suffer from specular reflections and multipath effects, resulting in noisy and temporally inconsistent measurements, such as body-part miss-detection. To address these radar-specific artifacts, we propose mmPred, the first diffusion-based framework tailored for radar-based HMP. mmPred introduces a dual-domain historical motion representation to guide the generation process, combining a Time-domain Pose Refinement (TPR) branch for learning fine-grained details and a Frequency-domain Dominant Motion (FDM) branch for capturing global motion trends and suppressing frame-level inconsistency. Furthermore, we design a Global Skeleton-relational Transformer (GST) as the diffusion backbone to model global inter-joint cooperation, enabling corrupted joints to dynamically aggregate information from others. Extensive experiments show that mmPred achieves state-of-the-art performance, outperforming existing methods by 8.6% on mmBody and 22% on mm-Fi.
comment : This paper is accepted by AAAI-2026
☆ Assimilation Matters: Model-level Backdoor Detection in Vision -Language Pretrain ed Models
Zhongqi Wang, Jie Zhang, Shiguang Shan, Xilin Chen
Vision-language pretrained models (VLPs) such as CLIP have achieved remarkable success, but are also highly vulnerable to backdoor attacks. Given a model fine-tuned by an untrusted third party, determining whether the model has been injected with a backdoor is a critical and challenging problem. Existing detection methods usually rely on prior knowledge of training dataset, backdoor triggers and targets, or downstream classifiers, which may be impractical for real-world applications. To address this, To address this challenge, we introduce Assimilation Matters in DETection (AMDET), a novel model-level detection framework that operates without any such prior knowledge. Specifically, we first reveal the feature assimilation property in backdoored text encoders: the representations of all tokens within a backdoor sample exhibit a high similarity. Further analysis attributes this effect to the concentration of attention weights on the trigger token. Leveraging this insight, AMDET scans a model by performing gradient-based inversion on token embeddings to recover implicit features that capable of activating backdoor behaviors. Furthermore, we identify the natural backdoor feature in the OpenAI's official CLIP model, which are not intentionally injected but still exhibit backdoor-like behaviors. We then filter them out from real injected backdoor by analyzing their loss landscapes. Extensive experiments on 3,600 backdoored and benign-finetuned models with two attack paradigms and three VLP model structures show that AMDET detects backdoors with an F1 score of 89.90%. Besides, it achieves one complete detection in approximately 5 minutes on a RTX 4090 GPU and exhibits strong robustness against adaptive attacks. Code is available at: https://github.com/Robin-WZQ/AMDET
☆ MVAD : A Comprehensive Multimodal Video-Audio Dataset for AIGC Detection
Mengxue Hu, Yunfeng Diao, Changtao Miao, Jianshu Li, Zhe Li, Joey Tianyi Zhou
The rapid advancement of AI-generated multimodal video-audio content has raised significant concerns regarding information security and content authenticity. Existing synthetic video datasets predominantly focus on the visual modality alone, while the few incorporating audio are largely confined to facial deepfakes--a limitation that fails to address the expanding landscape of general multimodal AI-generated content and substantially impedes the development of trustworthy detection systems. To bridge this critical gap, we introduce the Multimodal Video-Audio Dataset (MVAD), the first comprehensive dataset specifically designed for detecting AI-generated multimodal video-audio content. Our dataset exhibits three key characteristics: (1) genuine multimodality with samples generated according to three realistic video-audio forgery patterns; (2) high perceptual quality achieved through diverse state-of-the-art generative models; and (3) comprehensive diversity spanning realistic and anime visual styles, four content categories (humans, animals, objects, and scenes), and four video-audio multimodal data types. Our dataset will be available at https://github.com/HuMengXue0104/MVAD.
comment : 7 pages,2 figures
☆ Odometry Without Correspondence from Inertially Constrained Ruled Surfaces
Chenqi Zhu, Levi Burner, Yiannis Aloimonos
Visual odometry techniques typically rely on feature extraction from a sequence of images and subsequent computation of optical flow. This point-to-point correspondence between two consecutive frames can be costly to compute and suffers from varying accuracy, which affects the odometry estimate's quality. Attempts have been made to bypass the difficulties originating from the correspondence problem by adopting line features and fusing other sensors (event camera, IMU) to improve performance, many of which still heavily rely on correspondence. If the camera observes a straight line as it moves, the image of the line sweeps a smooth surface in image-space time. It is a ruled surface and analyzing its shape gives information about odometry. Further, its estimation requires only differentially computed updates from point-to-line associations. Inspired by event cameras' propensity for edge detection, this research presents a novel algorithm to reconstruct 3D scenes and visual odometry from these ruled surfaces. By constraining the surfaces with the inertia measurements from an onboard IMU sensor, the dimensionality of the solution space is greatly reduced.
comment : 14 pages, 13 figures, 5 tables
☆ MILE: A Mechanically Isomorphic Exoskeleton Data Collection System with Fingertip Visuotactile Sensing for Dexterous Manipulation
Jinda Du, Jieji Ren, Qiaojun Yu, Ningbin Zhang, Yu Deng, Xingyu Wei, Yufei Liu, Guoying Gu, Xiangyang Zhu
Imitation learning provides a promising approach to dexterous hand manipulation, but its effectiveness is limited by the lack of large-scale, high-fidelity data. Existing data-collection pipelines suffer from inaccurate motion retargeting, low data-collection efficiency, and missing high-resolution fingertip tactile sensing. We address this gap with MILE, a mechanically isomorphic teleoperation and data-collection system co-designed from human hand to exoskeleton to robotic hand. The exoskeleton is anthropometrically derived from the human hand, and the robotic hand preserves one-to-one joint-position isomorphism, eliminating nonlinear retargeting and enabling precise, natural control. The exoskeleton achieves a multi-joint mean absolute angular error below one degree, while the robotic hand integrates compact fingertip visuotactile modules that provide high-resolution tactile observations. Built on this retargeting-free interface, we teleoperate complex, contact-rich in-hand manipulation and efficiently collect a multimodal dataset comprising high-resolution fingertip visuotactile signals, RGB-D images, and joint positions. The teleoperation pipeline achieves a mean success rate improvement of 64%. Incorporating fingertip tactile observations further increases the success rate by an average of 25% over the vision-only baseline, validating the fidelity and utility of the dataset. Further details are available at: https://sites.google.com/view/mile-system.
☆ ART-ASyn: Anatomy-aware Realistic Texture-based Anomaly Synthesis Framework for Chest X-Rays WACV2026
Qinyi Cao, Jianan Fan, Weidong Cai
Unsupervised anomaly detection aims to identify anomalies without pixel-level annotations. Synthetic anomaly-based methods exhibit a unique capacity to introduce controllable irregularities with known masks, enabling explicit supervision during training. However, existing methods often produce synthetic anomalies that are visually distinct from real pathological patterns and ignore anatomical structure. This paper presents a novel Anatomy-aware Realistic Texture-based Anomaly Synthesis framework (ART-ASyn) for chest X-rays that generates realistic and anatomically consistent lung opacity related anomalies using texture-based augmentation guided by our proposed Progressive Binary Thresholding Segmentation method (PBTSeg) for lung segmentation. The generated paired samples of synthetic anomalies and their corresponding precise pixel-level anomaly mask for each normal sample enable explicit segmentation supervision. In contrast to prior work limited to one-class classification, ART-ASyn is further evaluated for zero-shot anomaly segmentation, demonstrating generalizability on an unseen dataset without target-domain annotations. Code availability is available at https://github.com/angelacao-hub/ART-ASyn.
comment : Accepted in WACV2026
☆ Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation NeurIPS 2025
Xiao Cui, Yulei Qin, Wengang Zhou, Hongsheng Li, Houqiang Li
Dataset distillation seeks to synthesize a compact distilled dataset, enabling models trained on it to achieve performance comparable to models trained on the full dataset. Recent methods for large-scale datasets focus on matching global distributional statistics (e.g., mean and variance), but overlook critical instance-level characteristics and intraclass variations, leading to suboptimal generalization. We address this limitation by reformulating dataset distillation as an Optimal Transport (OT) distance minimization problem, enabling fine-grained alignment at both global and instance levels throughout the pipeline. OT offers a geometrically faithful framework for distribution matching. It effectively preserves local modes, intra-class patterns, and fine-grained variations that characterize the geometry of complex, high-dimensional distributions. Our method comprises three components tailored for preserving distributional geometry: (1) OT-guided diffusion sampling, which aligns latent distributions of real and distilled images; (2) label-image-aligned soft relabeling, which adapts label distributions based on the complexity of distilled image distributions; and (3) OT-based logit matching, which aligns the output of student models with soft-label distributions. Extensive experiments across diverse architectures and large-scale datasets demonstrate that our method consistently outperforms state-of-the-art approaches in an efficient manner, achieving at least 4% accuracy improvement under IPC=10 settings for each architecture on ImageNet-1K.
comment : NeurIPS 2025
☆ TGSFormer: Scalable Temporal Gaussian Splatting for Embodied Semantic Scene Completion
Rui Qian, Haozhi Cao, Tianchen Deng, Tianxin Hu, Weixiang Guo, Shenghai Yuan, Lihua Xie
Embodied 3D Semantic Scene Completion (SSC) infers dense geometry and semantics from continuous egocentric observations. Most existing Gaussian-based methods rely on random initialization of many primitives within predefined spatial bounds, resulting in redundancy and poor scalability to unbounded scenes. Recent depth-guided approach alleviates this issue but remains local, suffering from latency and memory overhead as scale increases. To overcome these challenges, we propose TGSFormer, a scalable Temporal Gaussian Splatting framework for embodied SSC. It maintains a persistent Gaussian memory for temporal prediction, without relying on image coherence or frame caches. For temporal fusion, a Dual Temporal Encoder jointly processes current and historical Gaussian features through confidence-aware cross-attention. Subsequently, a Confidence-aware Voxel Fusion module merges overlapping primitives into voxel-aligned representations, regulating density and maintaining compactness. Extensive experiments demonstrate that TGSFormer achieves state-of-the-art results on both local and embodied SSC benchmarks, offering superior accuracy and scalability with significantly fewer primitives while maintaining consistent long-term scene integrity. The code will be released upon acceptance.
comment : 14 pages, 10 figures
☆ Words into Wo rl d: A Task-Adaptive Agent for Language-Guided Spatial Retrieval in AR
Lixing Guo, Tobias Höllerer
Traditional augmented reality (AR) systems predominantly rely on fixed class detectors or fiducial markers, limiting their ability to interpret complex, open-vocabulary natural language queries. We present a modular AR agent system that integrates multimodal large language models (MLLMs) with grounded vision models to enable relational reasoning in space and language-conditioned spatial retrieval in physical environments. Our adaptive task agent coordinates MLLMs and coordinate-aware perception tools to address varying query complexities, ranging from simple object identification to multi-object relational reasoning, while returning meter-accurate 3D anchors. It constructs dynamic AR scene graphs encoding nine typed relations (spatial, structural-semantic, causal-functional), enabling MLLMs to understand not just what objects exist, but how they relate and interact in 3D space. Through task-adaptive region-of-interest highlighting and contextual spatial retrieval, the system guides human attention to information-dense areas while supporting human-in-the-loop refinement. The agent dynamically invokes coordinate-aware tools for complex queries-selection, measurement, comparison, and actuation-grounding language understanding in physical operations. The modular architecture supports plug-and-use vision-language models without retraining, establishing AR agents as intermediaries that augment MLLMs with real-world spatial intelligence for interactive scene understanding. We also introduce GroundedAR-Bench, an evaluation framework for language-driven real world localization and relation grounding across diverse environments.
☆ RealAppliance: Let High-fidelity Appliance Assets Controllable and Workable as Aligned Real Manuals
Yuzheng Gao, Yuxing Long, Lei Kang, Yuchong Guo, Ziyan Yu, Shangqing Mao, Jiyao Zhang, Ruihai Wu, Dongjiang Li, Hui Shen, Hao Dong
Existing appliance assets suffer from poor rendering, incomplete mechanisms, and misalignment with manuals, leading to simulation-reality gaps that hinder appliance manipulation development. In this work, we introduce the RealAppliance dataset, comprising 100 high-fidelity appliances with complete physical, electronic mechanisms, and program logic aligned with their manuals. Based on these assets, we propose the RealAppliance-Bench benchmark, which evaluates multimodal large language models and embodied manipulation planning models across key tasks in appliance manipulation planning: manual page retrieval, appliance part grounding, open-loop manipulation planning, and closed-loop planning adjustment. Our analysis of model performances on RealAppliance-Bench provides insights for advancing appliance manipulation research
☆ Rethinking Lung Cancer Screening: AI Nodule Detection and Diagnosis Outperforms Radiologists, Leading Models, and Standards Beyond Size and Growth
Sylvain Bodard, Pierre Baudot, Benjamin Renoust, Charles Voyton, Gwendoline De Bie, Ezequiel Geremia, Van-Khoa Le, Danny Francis, Pierre-Henri Siot, Yousra Haddou, Vincent Bobin, Jean-Christophe Brisset, Carey C. Thomson, Valerie Bourdes, Benoit Huet
Early detection of malignant lung nodules is critical, but its dependence on size and growth in screening inherently delays diagnosis. We present an AI system that redefines lung cancer screening by performing both detection and malignancy diagnosis directly at the nodule level on low-dose CT scans. To address limitations in dataset scale and explainability, we designed an ensemble of shallow deep learning and feature-based specialized models. Trained and evaluated on 25,709 scans with 69,449 annotated nodules, the system outperforms radiologists, Lung-RADS, and leading AI models (Sybil, Brock, Google, Kaggle). It achieves an area under the receiver operating characteristic curve (AUC) of 0.98 internally and 0.945 on an independent cohort. With 0.5 false positives per scan at 99.3\% sensitivity, it addresses key barriers to AI adoption. Critically, it outperforms radiologists across all nodule sizes and stages, excelling in stage 1 cancers, and all growth-based metrics, including the least accurate: Volume-Doubling Time. It also surpasses radiologists by up to one year in diagnosing indeterminate and slow-growing nodules.
comment : 25 pages, 8 figures, with supplementary information containing 11 figures
☆ HIMOSA: Efficient Remote Sensing Image Super-Resolution with Hierarchical Mixture of Sparse Attention
Yi Liu, Yi Wan, Xinyi Liu, Qiong Wu, Panwang Xia, Xuejun Huang, Yongjun Zhang
In remote sensing applications, such as disaster detection and response, real-time efficiency and model lightweighting are of critical importance. Consequently, existing remote sensing image super-resolution methods often face a trade-off between model performance and computational efficiency. In this paper, we propose a lightweight super-resolution framework for remote sensing imagery, named HIMOSA. Specifically, HIMOSA leverages the inherent redundancy in remote sensing imagery and introduces a content-aware sparse attention mechanism, enabling the model to achieve fast inference while maintaining strong reconstruction performance. Furthermore, to effectively leverage the multi-scale repetitive patterns found in remote sensing imagery, we introduce a hierarchical window expansion and reduce the computational complexity by adjusting the sparsity of the attention. Extensive experiments on multiple remote sensing datasets demonstrate that our method achieves state-of-the-art performance while maintaining computational efficiency.
☆ USB: Unified Synthetic Brain Framework for Bidirectional Pathology-Healthy Generation and Editing
Jun Wang, Peirong Liu
Understanding the relationship between pathological and healthy brain structures is fundamental to neuroimaging, connecting disease diagnosis and detection with modeling, prediction, and treatment planning. However, paired pathological-healthy data are extremely difficult to obtain, as they rely on pre- and post-treatment imaging, constrained by clinical outcomes and longitudinal data availability. Consequently, most existing brain image generation and editing methods focus on visual quality yet remain domain-specific, treating pathological and healthy image modeling independently. We introduce USB (Unified Synthetic Brain), the first end-to-end framework that unifies bidirectional generation and editing of pathological and healthy brain images. USB models the joint distribution of lesions and brain anatomy through a paired diffusion mechanism and achieves both pathological and healthy image generation. A consistency guidance algorithm further preserves anatomical consistency and lesion correspondence during bidirectional pathology-healthy editing. Extensive experiments on six public brain MRI datasets including healthy controls, stroke, and Alzheimer's patients, demonstrate USB's ability to produce diverse and realistic results. By establishing the first unified benchmark for brain image generation and editing, USB opens opportunities for scalable dataset creation and robust neuroimaging analysis. Code is available at https://github.com/jhuldr/USB.
comment : 16 pages, 17 figures
☆ HeartFormer: Semantic-Aware Dual-Structure Transformer s for 3D Four-Chamber Cardiac Point Cloud Reconstruction
Zhengda Ma, Abhirup Banerjee
We present the first geometric deep learning framework based on point cloud representation for 3D four-chamber cardiac reconstruction from cine MRI data. This work addresses a long-standing limitation in conventional cine MRI, which typically provides only 2D slice images of the heart, thereby restricting a comprehensive understanding of cardiac morphology and physiological mechanisms in both healthy and pathological conditions. To overcome this, we propose \textbf{HeartFormer}, a novel point cloud completion network that extends traditional single-class point cloud completion to the multi-class. HeartFormer consists of two key components: a Semantic-Aware Dual-Structure Transformer Network (SA-DSTNet) and a Semantic-Aware Geometry Feature Refinement Transformer Network (SA-GFRTNet). SA-DSTNet generates an initial coarse point cloud with both global geometry features and substructure geometry features. Guided by these semantic-geometry representations, SA-GFRTNet progressively refines the coarse output, effectively leveraging both global and substructure geometric priors to produce high-fidelity and geometrically consistent reconstructions. We further construct \textbf{HeartCompv1}, the first publicly available large-scale dataset with 17,000 high-resolution 3D multi-class cardiac meshes and point-clouds, to establish a general benchmark for this emerging research direction. Extensive cross-domain experiments on HeartCompv1 and UK Biobank demonstrate that HeartFormer achieves robust, accurate, and generalizable performance, consistently surpassing state-of-the-art (SOTA) methods. Code and dataset will be released upon acceptance at: https://github.com/10Darren/HeartFormer.
☆ UniDiff: Parameter-Efficient Adaptation of Diffusion Models for Land Cover Classification with Multi-Modal Remotely Sensed Imagery and Sparse Annotations WACV 2026
Yuzhen Hu, Saurabh Prasad
Sparse annotations fundamentally constrain multimodal remote sensing: even recent state-of-the-art supervised methods such as MSFMamba are limited by the availability of labeled data, restricting their practical deployment despite architectural advances. ImageNet-pretrained models provide rich visual representations, but adapting them to heterogeneous modalities such as hyperspectral imaging (HSI) and synthetic aperture radar (SAR) without large labeled datasets remains challenging. We propose UniDiff, a parameter-efficient framework that adapts a single ImageNet-pretrained diffusion model to multiple sensing modalities using only target-domain data. UniDiff combines FiLM-based timestep-modality conditioning, parameter-efficient adaptation of approximately 5% of parameters, and pseudo-RGB anchoring to preserve pre-trained representations and prevent catastrophic forgetting. This design enables effective feature extraction from remote sensing data under sparse annotations. Our results with two established multi-modal benchmarking datasets demonstrate that unsupervised adaptation of a pre-trained diffusion model effectively mitigates annotation constraints and achieves effective fusion of multi-modal remotely sensed data.
comment : Camera-ready for WACV 2026
☆ Relightable Holoported Characters: Capturing and Relighting Dynamic Human Performance from Sparse Views
Kunwar Maheep Singh, Jianchun Chen, Vladislav Golyanik, Stephan J. Garbin, Thabo Beeler, Rishabh Dabral, Marc Habermann, Christian Theobalt
We present Relightable Holoported Characters (RHC), a novel person-specific method for free-view rendering and relighting of full-body and highly dynamic humans solely observed from sparse-view RGB videos at inference. In contrast to classical one-light-at-a-time (OLAT)-based human relighting, our transformer-based RelightNet predicts relit appearance within a single network pass, avoiding costly OLAT-basis capture and generation. For training such a model, we introduce a new capture strategy and dataset recorded in a multi-view lightstage, where we alternate frames lit by random environment maps with uniformly lit tracking frames, simultaneously enabling accurate motion tracking and diverse illumination as well as dynamics coverage. Inspired by the rendering equation, we derive physics-informed features that encode geometry, albedo, shading, and the virtual camera view from a coarse human mesh proxy and the input views. Our RelightNet then takes these features as input and cross-attends them with a novel lighting condition, and regresses the relit appearance in the form of texel-aligned 3D Gaussian splats attached to the coarse mesh proxy. Consequently, our RelightNet implicitly learns to efficiently compute the rendering equation for novel lighting conditions within a single feed-forward pass. Experiments demonstrate our method's superior visual fidelity and lighting reproduction compared to state-of-the-art approaches. Project page: https://vcai.mpi-inf.mpg.de/projects/RHC/
♻ ☆ Ensuring Force Safety in Vision -Guided Robotic Manipulation via Implicit Tactile Calibration
Lai Wei, Jiahua Ma, Yibo Hu, Ruimao Zhang
In dynamic environments, robots often encounter constrained movement trajectories when manipulating objects with specific properties, such as doors. Therefore, applying the appropriate force is crucial to prevent damage to both the robots and the objects. However, current vision-guided robot state generation methods often falter in this regard, as they lack the integration of tactile perception. To tackle this issue, this paper introduces a novel state diffusion framework termed SafeDiff. It generates a prospective state sequence from the current robot state and visual context observation while incorporating real-time tactile feedback to refine the sequence. As far as we know, this is the first study specifically focused on ensuring force safety in robotic manipulation. It significantly enhances the rationality of state planning, and the safe action trajectory is derived from inverse dynamics based on this refined planning. In practice, unlike previous approaches that concatenate visual and tactile data to generate future robot state sequences, our method employs tactile data as a calibration signal to adjust the robot's state within the state space implicitly. Additionally, we've developed a large-scale simulation dataset called SafeDoorManip50k, offering extensive multimodal data to train and evaluate the proposed method. Extensive experiments show that our visual-tactile model substantially mitigates the risk of harmful forces in the door opening, across both simulated and real-world settings.
comment : Website URL: see https://i-am-future.github.io/safediff/
♻ ☆ A Minimal Subset Approach for Informed Keyframe Sampling in Large-Scale SLAM
Nikolaos Stathoulopoulos, Christoforos Kanellakis, George Nikolakopoulos
Typical LiDAR SLAM architectures feature a front-end for odometry estimation and a back-end for refining and optimizing the trajectory and map, commonly through loop closures. However, loop closure detection in large-scale missions presents significant computational challenges due to the need to identify, verify, and process numerous candidate pairs for pose graph optimization. Keyframe sampling bridges the front-end and back-end by selecting frames for storing and processing during global optimization. This article proposes an online keyframe sampling approach that constructs the pose graph using the most impactful keyframes for loop closure. We introduce the Minimal Subset Approach (MSA), which optimizes two key objectives: redundancy minimization and information preservation, implemented within a sliding window framework. By operating in the feature space rather than 3-D space, MSA efficiently reduces redundant keyframes while retaining essential information. Evaluations on diverse public datasets show that the proposed approach outperforms naive methods in reducing false positive rates in place recognition, while delivering superior ATE and RPE in metric localization, without the need for manual parameter tuning. Additionally, MSA demonstrates efficiency and scalability by reducing memory usage and computational overhead during loop closure detection and pose graph optimization.
comment : Please cite the published version. 8 pages, 9 figures
♻ ☆ Have We Scene It All? Scene Graph-Aware Deep Point Cloud Compression
Nikolaos Stathoulopoulos, Christoforos Kanellakis, George Nikolakopoulos
Efficient transmission of 3D point cloud data is critical for advanced perception in centralized and decentralized multi-agent robotic systems, especially nowadays with the growing reliance on edge and cloud-based processing. However, the large and complex nature of point clouds creates challenges under bandwidth constraints and intermittent connectivity, often degrading system performance. We propose a deep compression framework based on semantic scene graphs. The method decomposes point clouds into semantically coherent patches and encodes them into compact latent representations with semantic-aware encoders conditioned by Feature-wise Linear Modulation (FiLM). A folding-based decoder, guided by latent features and graph node attributes, enables structurally accurate reconstruction. Experiments on the SemanticKITTI and nuScenes datasets show that the framework achieves state-of-the-art compression rates, reducing data size by up to 98% while preserving both structural and semantic fidelity. In addition, it supports downstream applications such as multi-robot pose graph optimization and map merging, achieving trajectory accuracy and map alignment comparable to those obtained with raw LiDAR scans.
comment : Please cite published version. 8 pages, 6 figures
♻ ☆ SizeGS: Size-aware Compression of 3D Gaussian Splatting via Mixed Integer Programming
Shuzhao Xie, Jiahang Liu, Weixiang Zhang, Shijia Ge, Sicheng Pan, Chen Tang, Yunpeng Bai, Cong Zhang, Xiaoyi Fan, Zhi Wang
Recent advances in 3D Gaussian Splatting (3DGS) have greatly improved 3D reconstruction. However, its substantial data size poses a significant challenge for transmission and storage. While many compression techniques have been proposed, they fail to efficiently adapt to fluctuating network bandwidth, leading to resource wastage. We address this issue from the perspective of size-aware compression, where we aim to compress 3DGS to a desired size by quickly searching for suitable hyperparameters. Through a measurement study, we identify key hyperparameters that affect the size -- namely, the reserve ratio of Gaussians and bit-width settings for Gaussian attributes. Then, we formulate this hyperparameter optimization problem as a mixed-integer nonlinear programming (MINLP) problem, with the goal of maximizing visual quality while respecting the size budget constraint. To solve the MINLP, we decouple this problem into two parts: discretely sampling the reserve ratio and determining the bit-width settings using integer linear programming (ILP). To solve the ILP more quickly and accurately, we design a quality loss estimator and a calibrated size estimator, as well as implement a CUDA kernel. Extensive experiments on multiple 3DGS variants demonstrate that our method achieves state-of-the-art performance in post-training compression. Furthermore, our method can achieve comparable quality to leading training-required methods after fine-tuning.
comment : Automatically compressing 3DGS into the desired file size while maximizing the visual quality
♻ ☆ Adversarial Exploitation of Data Diversity Improves Visual Localization
Sihang Li, Siqi Tan, Bowen Chang, Jing Zhang, Chen Feng, Yiming Li
Visual localization, which estimates a camera's pose within a known scene, is a fundamental capability for autonomous systems. While absolute pose regression (APR) methods have shown promise for efficient inference, they often struggle with generalization. Recent approaches attempt to address this through data augmentation with varied viewpoints, yet they overlook a critical factor: appearance diversity. In this work, we identify appearance variation as the key to robust localization. Specifically, we first lift real 2D images into 3D Gaussian Splats with varying appearance and deblurring ability, enabling the synthesis of diverse training data that varies not just in poses but also in environmental conditions such as lighting and weather. To fully unleash the potential of the appearance-diverse data, we build a two-branch joint training pipeline with an adversarial discriminator to bridge the syn-to-real gap. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art methods, reducing translation and rotation errors by 50\% and 41\% on indoor datasets, and 38\% and 44\% on outdoor datasets. Most notably, our method shows remarkable robustness in dynamic driving scenarios under varying weather conditions and in day-to-night scenarios, where previous APR methods fail. Project Page: https://ai4ce.github.io/RAP/
comment : 24 pages, 22 figures
♻ ☆ Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models
Jitai Hao, Hao Liu, Xinyan Xiao, Qiang Huang, Jun Yu
Unified Multimodal Models (UMMs) built on shared autoregressive (AR) transformers are attractive for their architectural simplicity. However, we identify a critical limitation: when trained on multimodal inputs, modality-shared transformers suffer from severe gradient conflicts between vision and text, particularly in shallow and deep layers. We trace this issue to the fundamentally different low-level statistical properties of images and text, while noting that conflicts diminish in middle layers where representations become more abstract and semantically aligned. To overcome this challenge, we propose Uni-X, a two-end-separated, middle-shared architecture. Uni-X dedicates its initial and final layers to modality-specific processing, while maintaining shared parameters in the middle layers for high-level semantic fusion. This X-shaped design not only eliminates gradient conflicts at both ends but also further alleviates residual conflicts in the shared layers. Extensive experiments validate the effectiveness of Uni-X. Under identical training conditions, Uni-X achieves superior training efficiency compared to strong baselines. When scaled to 3B parameters with larger training data, Uni-X matches or surpasses 7B AR-based UMMs, achieving a GenEval score of 82 for image generation alongside strong performance in text and vision understanding tasks. These results establish Uni-X as a parameter-efficient and scalable foundation for future unified multimodal modeling. Our code is available at https://github.com/CURRENTF/Uni-X
♻ ☆ MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding WSDM 2026
Daoze Zhang, Chenghan Fu, Zhanheng Nie, Jianyu Liu, Wanxian Guan, Yuan Gao, Jun Song, Pengjie Wang, Jian Xu, Bo Zheng
With the rapid advancement of e-commerce, exploring general representations rather than task-specific ones has attracted increasing research attention. For product understanding, although existing discriminative dual-flow architectures drive progress in this field, they inherently struggle to model the many-to-one alignment between multiple images and texts of products. Therefore, we argue that generative Multimodal Large Language Models (MLLMs) hold significant potential for improving product representation learning. Nevertheless, achieving this goal still remains non-trivial due to several key challenges: the lack of multimodal and aspect-aware modeling modules in typical LLMs; the common presence of background noise in product images; and the absence of a standard benchmark for evaluation. To address these issues, we propose the first generative MLLM-based model named MOON for product representation learning. Our method (1) employs a guided Mixture-of-Experts (MoE) module for targeted modeling of multimodal and aspect-specific product content; (2) effectively detects core semantic regions in product images to mitigate the distraction and interference caused by background noise; and (3) introduces the specialized negative sampling strategy to increase the difficulty and diversity of negative samples. In addition, we release a large-scale multimodal benchmark MBE for various product understanding tasks. Experimentally, our model demonstrates competitive zero-shot performance on both our benchmark and the public dataset, showcasing strong generalization across various downstream tasks, including cross-modal retrieval, product classification, and attribute prediction. Furthermore, the case study and visualization illustrate the effectiveness of MOON for product understanding.
comment : Accepted by WSDM 2026. 11 pages, 9 figures
♻ ☆ Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward
Yuwei Niu, Weiyang Jin, Jiaqi Liao, Chaoran Feng, Peng Jin, Bin Lin, Zongjian Li, Bin Zhu, Weihao Yu, Li Yuan
Recent years have witnessed significant progress in Unified Multimodal Models, yet a fundamental question remains: Does understanding truly inform generation? To investigate this, we introduce UniSandbox, a decoupled evaluation framework paired with controlled, synthetic datasets to avoid data leakage and enable detailed analysis. Our findings reveal a significant understanding-generation gap, which is mainly reflected in two key dimensions: reasoning generation and knowledge transfer. Specifically, for reasoning generation tasks, we observe that explicit Chain-of-Thought (CoT) in the understanding module effectively bridges the gap, and further demonstrate that a self-training approach can successfully internalize this ability, enabling implicit reasoning during generation. Additionally, for knowledge transfer tasks, we find that CoT assists the generative process by helping retrieve newly learned knowledge, and also discover that query-based architectures inherently exhibit latent CoT-like properties that affect this transfer. UniSandbox provides preliminary insights for designing future unified architectures and training strategies that truly bridge the gap between understanding and generation. Code and data are available at https://github.com/PKU-YuanGroup/UniSandBox
♻ ☆ Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Sound
Dengming Zhang, Weitao You, Jingxiong Li, Weishen Lin, Wenda Shi, Xue Zhao, Heda Zuo, Junxian Wu, Lingyun Sun
Emotion understanding is critical for making Large Language Models (LLMs) more general, reliable, and aligned with humans. Art conveys emotion through the joint design of visual and auditory elements, yet most prior work is human-centered or single-modality, overlooking the emotion intentionally expressed by the artwork. Meanwhile, current Audio-Visual Language Models (AVLMs) typically require large-scale audio pretraining to endow Visual Language Models (VLMs) with hearing, which limits scalability. We present Vision Anchored Audio-Visual Emotion LLM (VAEmotionLLM), a two-stage framework that teaches a VLM to hear by seeing with limited audio pretraining and to understand emotion across modalities. In Stage 1, Vision-Guided Audio Alignment (VG-Align) distills the frozen visual pathway into a new audio pathway by aligning next-token distributions of the shared LLM on synchronized audio-video clips, enabling hearing without a large audio dataset. In Stage 2, a lightweight Cross-Modal Emotion Adapter (EmoAdapter), composed of the Emotion Enhancer and the Emotion Supervisor, injects emotion-sensitive residuals and applies emotion supervision to enhance cross-modal emotion understanding. We also construct ArtEmoBenchmark, an art-centric emotion benchmark that evaluates content and emotion understanding under audio-only, visual-only, and audio-visual inputs. VAEmotionLLM achieves state-of-the-art results on ArtEmoBenchmark, outperforming audio-only, visual-only, and audio-visual baselines. Ablations show that the proposed components are complementary.
♻ ☆ ICAS: Detecting Training Data from Autoregressive Image Generative Models ACM MM 2025
Hongyao Yu, Yixiang Qiu, Yiheng Yang, Hao Fang, Tianqu Zhuang, Jiaxin Hong, Bin Chen, Hao Wu, Shu-Tao Xia
Autoregressive image generation has witnessed rapid advancements, with prominent models such as scale-wise visual auto-regression pushing the boundaries of visual synthesis. However, these developments also raise significant concerns regarding data privacy and copyright. In response, training data detection has emerged as a critical task for identifying unauthorized data usage in model training. To better understand the vulnerability of autoregressive image generative models to such detection, we conduct the first study applying membership inference to this domain. Our approach comprises two key components: implicit classification and an adaptive score aggregation strategy. First, we compute the implicit token-wise classification score within the query image. Then we propose an adaptive score aggregation strategy to acquire a final score, which places greater emphasis on the tokens with lower scores. A higher final score indicates that the sample is more likely to be involved in the training set. To validate the effectiveness of our method, we adapt existing detection algorithms originally designed for LLMs to visual autoregressive models. Extensive experiments demonstrate the superiority of our method in both class-conditional and text-to-image scenarios. Moreover, our approach exhibits strong robustness and generalization under various data transformations. Furthermore, sufficient experiments suggest two novel key findings: (1) A linear scaling law on membership inference, exposing the vulnerability of large foundation models. (2) Training data from scale-wise visual autoregressive models is easier to detect than other autoregressive paradigms. Our code is available at https://github.com/Chrisqcwx/ImageAR-MIA.
comment : ACM MM 2025
♻ ☆ Deep Learning-Based Multiclass Classification of Oral Lesions with Stratified Augmentation
Joy Naoum, Revana Salama, Ali Hamdi
Oral cancer is highly common across the globe and is mostly diagnosed during the later stages due to the close visual similarity to benign, precancerous, and malignant lesions in the oral cavity. Implementing computer aided diagnosis systems early on has the potential to greatly improve clinical outcomes. This research intends to use deep learning to build a multiclass classifier for sixteen different oral lesions. To overcome the challenges of limited and imbalanced datasets, the proposed technique combines stratified data splitting and advanced data augmentation and oversampling to perform the classification. The experimental results, which achieved 83.33 percent accuracy, 89.12 percent precision, and 77.31 percent recall, demonstrate the superiority of the suggested model over state of the art methods now in use. The suggested model effectively conveys the effectiveness of oversampling and augmentation strategies in situations where the minority class classification performance is noteworthy. As a first step toward trustworthy computer aided diagnostic systems for the early detection of oral cancer in clinical settings, the suggested framework shows promise.
comment : Technical error in the experimentation and we will resubmit it again
♻ ☆ HyCoVAD: A Hybrid SSL-LLM Model for Complex Video Anomaly Detection
Mohammad Mahdi Hemmatyar, Mahdi Jafari, Mohammad Amin Yousefi, Mohammad Reza Nemati, Mobin Azadani, Hamid Reza Rastad, Amirmohammad Akbari
Video anomaly detection (VAD) is crucial for intelligent surveillance, but a significant challenge lies in identifying complex anomalies, which are events defined by intricate relationships and temporal dependencies among multiple entities rather than by isolated actions. While self-supervised learning (SSL) methods effectively model low-level spatiotemporal patterns, they often struggle to grasp the semantic meaning of these interactions. Conversely, large language models (LLMs) offer powerful contextual reasoning but are computationally expensive for frame-by-frame analysis and lack fine-grained spatial localization. We introduce HyCoVAD, Hybrid Complex Video Anomaly Detection, a hybrid SSL-LLM model that combines a multi-task SSL temporal analyzer with LLM validator. The SSL module is built upon an nnFormer backbone which is a transformer-based model for image segmentation. It is trained with multiple proxy tasks, learns from video frames to identify those suspected of anomaly. The selected frames are then forwarded to the LLM, which enriches the analysis with semantic context by applying structured, rule-based reasoning to validate the presence of anomalies. Experiments on the challenging ComplexVAD dataset show that HyCoVAD achieves a 72.5% frame-level AUC, outperforming existing baselines by 12.5% while reducing LLM computation. We release our interaction anomaly taxonomy, adaptive thresholding protocol, and code to facilitate future research in complex VAD scenarios.
comment : 25 pages, 1 figure
♻ ☆ Modeling Rapid Context ual Learning in the Visual Cortex with Fast-Weight Deep Autoencoder Networks
Yue Li, Weifan Wang, Tai Sing Lee
Recent neurophysiological studies have revealed that the early visual cortex can rapidly learn global image context, as evidenced by a sparsification of population responses and a reduction in mean activity when exposed to familiar versus novel image contexts. This phenomenon has been attributed primarily to local recurrent interactions, rather than changes in feedforward or feedback pathways, supported by both empirical findings and circuit-level modeling. Recurrent neural circuits capable of simulating these effects have been shown to reshape the geometry of neural manifolds, enhancing robustness and invariance to irrelevant variations. In this study, we employ a Vision Transformer (ViT)-based autoencoder to investigate, from a functional perspective, how familiarity training can induce sensitivity to global context in the early layers of a deep neural network. We hypothesize that rapid learning operates via fast weights, which encode transient or short-term memory traces, and we explore the use of Low-Rank Adaptation (LoRA) to implement such fast weights within each Transformer layer. Our results show that (1) The proposed ViT-based autoencoder's self-attention circuit performs a manifold transform similar to a neural circuit model of the familiarity effect. (2) Familiarity training aligns latent representations in early layers with those in the top layer that contains global context information. (3) Familiarity training broadens the self-attention scope within the remembered image context. (4) These effects are significantly amplified by LoRA-based fast weights. Together, these findings suggest that familiarity training introduces global sensitivity to earlier layers in a hierarchical network, and that a hybrid fast-and-slow weight architecture may provide a viable computational model for studying rapid global context learning in the brain.
♻ ☆ LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders
Borna Khodabandeh, Amirabbas Afzali, Amirhossein Afsharrad, Seyed Shahabeddin Mousavi, Sanjay Lall, Sajjad Amini, Seyed-Mohsen Moosavi-Dezfooli
Visual encoders have become fundamental components in modern computer vision pipelines. However, ensuring robustness against adversarial perturbations remains a critical challenge. Recent efforts have explored both supervised and unsupervised adversarial fine-tuning strategies. We identify two key limitations in these approaches: (i) they often suffer from instability, especially during the early stages of fine-tuning, resulting in suboptimal convergence and degraded performance on clean data, and (ii) they exhibit a suboptimal trade-off between robustness and clean data accuracy, hindering the simultaneous optimization of both objectives. To overcome these challenges, we propose Lagrangian-Optimized Robust Embeddings (LORE), a novel unsupervised adversarial fine-tuning framework. LORE utilizes constrained optimization, which offers a principled approach to balancing competing goals, such as improving robustness while preserving nominal performance. By enforcing embedding-space proximity constraints, LORE effectively maintains clean data performance throughout adversarial fine-tuning. Extensive experiments show that LORE significantly improves zero-shot adversarial robustness with minimal degradation in clean data accuracy. Furthermore, we demonstrate the effectiveness of the adversarially fine-tuned CLIP image encoder in out-of-distribution generalization and enhancing the interpretability of image embeddings.
♻ ☆ Enhancing OCR for Sino-Vietnamese Language Processing via Fine-tuned Paddle OCR v5
Minh Hoang Nguyen, Su Nguyen Thiet
Recognizing and processing Classical Chinese (Han-Nom) texts play a vital role in digitizing Vietnamese historical documents and enabling cross-lingual semantic research. However, existing OCR systems struggle with degraded scans, non-standard glyphs, and handwriting variations common in ancient sources. In this work, we propose a fine-tuning approach for PaddleOCRv5 to improve character recognition on Han-Nom texts. We retrain the text recognition module using a curated subset of ancient Vietnamese Chinese manuscripts, supported by a full training pipeline covering preprocessing, LMDB conversion, evaluation, and visualization. Experimental results show a significant improvement over the base model, with exact accuracy increasing from 37.5 percent to 50.0 percent, particularly under noisy image conditions. Furthermore, we develop an interactive demo that visually compares pre- and post-fine-tuning recognition results, facilitating downstream applications such as Han-Vietnamese semantic alignment, machine translation, and historical linguistics research. The demo is available at https://huggingface.co/spaces/MinhDS/Fine-tuned-PaddleOCRv5
comment : Short Paper: 7 pages, 8 figures, 3 tables
♻ ☆ MoEGCL: Mixture of Ego-Graphs Contrastive Representation Learning for Multi-View Clustering
Jian Zhu, Xin Zou, Jun Sun, Cheng Luo, Lei Liu, Lingfang Zeng, Ning Zhang, Bian Wu, Chang Tang, Lirong Dai
In recent years, the advancement of Graph Neural Networks (GNNs) has significantly propelled progress in Multi-View Clustering (MVC). However, existing methods face the problem of coarse-grained graph fusion. Specifically, current approaches typically generate a separate graph structure for each view and then perform weighted fusion of graph structures at the view level, which is a relatively rough strategy. To address this limitation, we present a novel Mixture of Ego-Graphs Contrastive Representation Learning (MoEGCL). It mainly consists of two modules. In particular, we propose an innovative Mixture of Ego-Graphs Fusion (MoEGF), which constructs ego graphs and utilizes a Mixture-of-Experts network to implement fine-grained fusion of ego graphs at the sample level, rather than the conventional view-level fusion. Additionally, we present the Ego Graph Contrastive Learning (EGCL) module to align the fused representation with the view-specific representation. The EGCL module enhances the representation similarity of samples from the same cluster, not merely from the same sample, further boosting fine-grained graph representation. Extensive experiments demonstrate that MoEGCL achieves state-of-the-art results in deep multi-view clustering tasks. The source code is publicly available at https://github.com/HackerHyper/MoEGCL.
♻ ☆ Cross-Cancer Knowledge Transfer in WSI-based Prognosis Prediction
Pei Liu, Luping Ji, Jiaxiang Gou, Xiangxiang Zeng
Whole-Slide Image (WSI) is an important tool for estimating cancer prognosis. Current studies generally follow a conventional cancer-specific paradigm in which each cancer corresponds to a single model. However, this paradigm naturally struggles to scale to rare tumors and cannot leverage knowledge from other cancers. While multi-task learning frameworks have been explored recently, they often place high demands on computational resources and require extensive training on ultra-large, multi-cancer WSI datasets. To this end, this paper shifts the paradigm to knowledge transfer and presents the first preliminary yet systematic study on cross-cancer prognosis knowledge transfer in WSIs, called CROPKT. It comprises three major parts. (1) We curate a large dataset (UNI2-h-DSS) with 26 cancers and use it to measure the transferability of WSI-based prognostic knowledge across different cancers (including rare tumors). (2) Beyond a simple evaluation merely for benchmarking, we design a range of experiments to gain deeper insights into the underlying mechanism behind transferability. (3) We further show the utility of cross-cancer knowledge transfer, by proposing a routing-based baseline approach (ROUPKT) that could often efficiently utilize the knowledge transferred from off-the-shelf models of other cancers. CROPKT could serve as an inception that lays the foundation for this nascent paradigm, i.e., WSI-based prognosis prediction with cross-cancer knowledge transfer. Our source code is available at https://github.com/liupei101/CROPKT.
comment : 24 pages (11 figures and 10 tables)
♻ ☆ AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection ICLR 2024
Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, Jiming Chen
Zero-shot anomaly detection (ZSAD) requires detection models trained using auxiliary data to detect anomalies without any training sample in a target dataset. It is a crucial task when training data is not accessible due to various concerns, eg, data privacy, yet it is challenging since the models need to generalize to anomalies across different domains where the appearance of foreground objects, abnormal regions, and background features, such as defects/tumors on different products/organs, can vary significantly. Recently large pre-trained vision-language models (VLMs), such as CLIP, have demonstrated strong zero-shot recognition ability in various vision tasks, including anomaly detection. However, their ZSAD performance is weak since the VLMs focus more on modeling the class semantics of the foreground objects rather than the abnormality/normality in the images. In this paper we introduce a novel approach, namely AnomalyCLIP, to adapt CLIP for accurate ZSAD across different domains. The key insight of AnomalyCLIP is to learn object-agnostic text prompts that capture generic normality and abnormality in an image regardless of its foreground objects. This allows our model to focus on the abnormal image regions rather than the object semantics, enabling generalized normality and abnormality recognition on diverse types of objects. Large-scale experiments on 17 real-world anomaly detection datasets show that AnomalyCLIP achieves superior zero-shot performance of detecting and segmenting anomalies in datasets of highly diverse class semantics from various defect inspection and medical imaging domains. Code will be made available at https://github.com/zqhang/AnomalyCLIP.
comment : Accepted by ICLR 2024
♻ ☆ Prompt -OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision -Language Model Adaptation WACV 2026
Xiwen Chen, Wenhui Zhu, Peijie Qiu, Hao Wang, Huayu Li, Haiyu Wu, Aristeidis Sotiras, Yalin Wang, Abolfazl Razi
Vision-language models (VLMs) such as CLIP demonstrate strong performance but struggle when adapted to downstream tasks. Prompt learning has emerged as an efficient and effective strategy to adapt VLMs while preserving their pre-trained knowledge. However, existing methods still lead to overfitting and degrade zero-shot generalization. To address this challenge, we propose an optimal transport (OT)-guided prompt learning framework that mitigates forgetting by preserving the structural consistency of feature distributions between pre-trained and fine-tuned models. Unlike conventional point-wise constraints, OT naturally captures cross-instance relationships and expands the feasible parameter space for prompt tuning, allowing a better trade-off between adaptation and generalization. Our approach enforces joint constraints on both vision and text representations, ensuring a holistic feature alignment. Extensive experiments on benchmark datasets demonstrate that our simple yet effective method can outperform existing prompt learning strategies in base-to-novel generalization, cross-dataset evaluation, and domain generalization without additional augmentation or ensemble techniques. The code is available at https://github.com/ChongQingNoSubway/Prompt-OT
comment : Accepted to WACV 2026
♻ ☆ Beyond Subspace Isolation: Many-to-Many Transformer for Light Field Image Super-resolution
Zeke Zexi Hu, Xiaoming Chen, Vera Yuk Ying Chung, Yiran Shen
The effective extraction of spatial-angular features plays a crucial role in light field image super-resolution (LFSR) tasks, and the introduction of convolution and Transformers leads to significant improvement in this area. Nevertheless, due to the large 4D data volume of light field images, many existing methods opted to decompose the data into a number of lower-dimensional subspaces and perform Transformers in each sub-space individually. As a side effect, these methods inadvertently restrict the self-attention mechanisms to a One-to-One scheme accessing only a limited subset of LF data, explicitly preventing comprehensive optimization on all spatial and angular cues. In this paper, we identify this limitation as subspace isolation and introduce a novel Many-to-Many Transformer (M2MT) to address it. M2MT aggregates angular information in the spatial subspace before performing the self-attention mechanism. It enables complete access to all information across all sub-aperture images (SAIs) in a light field image. Consequently, M2MT is enabled to comprehensively capture long-range correlation dependencies. With M2MT as the foundational component, we develop a simple yet effective M2MT network for LFSR. Our experimental results demonstrate that M2MT achieves state-of-the-art performance across various public datasets, and it offers a favorable balance between model performance and efficiency, yielding higher-quality LFSR results with substantially lower demand for memory and computation. We further conduct in-depth analysis using local attribution maps (LAM) to obtain visual interpretability, and the results validate that M2MT is empowered with a truly non-local context in both spatial and angular subspaces to mitigate subspace isolation and acquire effective spatial-angular representation.
comment : Accepted by IEEE Transactions on Multimedia
♻ ☆ A Comprehensive Survey on Wo rl d Models for Embodied AI
Xinqing Li, Xin He, Le Zhang, Min Wu, Xiaoli Li, Yun Liu
Embodied AI requires agents that perceive, act, and anticipate how actions reshape future world states. World models serve as internal simulators that capture environment dynamics, enabling forward and counterfactual rollouts to support perception, prediction, and decision making. This survey presents a unified framework for world models in embodied AI. Specifically, we formalize the problem setting and learning objectives, and propose a three-axis taxonomy encompassing: (1) Functionality, Decision-Coupled vs. General-Purpose; (2) Temporal Modeling, Sequential Simulation and Inference vs. Global Difference Prediction; (3) Spatial Representation, Global Latent Vector, Token Feature Sequence, Spatial Latent Grid, and Decomposed Rendering Representation. We systematize data resources and metrics across robotics, autonomous driving, and general video settings, covering pixel prediction quality, state-level understanding, and task performance. Furthermore, we offer a quantitative comparison of state-of-the-art models and distill key open challenges, including the scarcity of unified datasets and the need for evaluation metrics that assess physical consistency over pixel fidelity, the trade-off between model performance and the computational efficiency required for real-time control, and the core modeling difficulty of achieving long-horizon temporal consistency while mitigating error accumulation. Finally, we maintain a curated bibliography at https://github.com/Li-Zn-H/AwesomeWorldModels.
comment : https://github.com/Li-Zn-H/AwesomeWorldModels
♻ ☆ OccluGaussian: Occlusion-Aware Gaussian Splatting for Large Scene Reconstruction and Rendering ICCV 2025
Shiyong Liu, Xiao Tang, Zhihao Li, Yingfan He, Chongjie Ye, Jianzhuang Liu, Binxiao Huang, Shunbo Zhou, Xiaofei Wu
In large-scale scene reconstruction using 3D Gaussian splatting, it is common to partition the scene into multiple smaller regions and reconstruct them individually. However, existing division methods are occlusion-agnostic, meaning that each region may contain areas with severe occlusions. As a result, the cameras within those regions are less correlated, leading to a low average contribution to the overall reconstruction. In this paper, we propose an occlusion-aware scene division strategy that clusters training cameras based on their positions and co-visibilities to acquire multiple regions. Cameras in such regions exhibit stronger correlations and a higher average contribution, facilitating high-quality scene reconstruction. We further propose a region-based rendering technique to accelerate large scene rendering, which culls Gaussians invisible to the region where the viewpoint is located. Such a technique significantly speeds up the rendering without compromising quality. Extensive experiments on multiple large scenes show that our method achieves superior reconstruction results with faster rendering speed compared to existing state-of-the-art approaches. Project page: https://occlugaussian.github.io.
comment : Accepted to ICCV 2025. Project website: https://occlugaussian.github.io
♻ ☆ M3DLayout: A Multi-Source Dataset of 3D Indoor Layouts and Structured Descriptions for 3D Generation
Yiheng Zhang, Zhuojiang Cai, Mingdao Wang, Meitong Guo, Tianxiao Li, Li Lin, Yuwang Wang
In text-driven 3D scene generation, object layout serves as a crucial intermediate representation that bridges high-level language instructions with detailed geometric output. It not only provides a structural blueprint for ensuring physical plausibility but also supports semantic controllability and interactive editing. However, the learning capabilities of current 3D indoor layout generation models are constrained by the limited scale, diversity, and annotation quality of existing datasets. To address this, we introduce M3DLayout, a large-scale, multi-source dataset for 3D indoor layout generation. M3DLayout comprises 21,367 layouts and over 433k object instances, integrating three distinct sources: real-world scans, professional CAD designs, and procedurally generated scenes. Each layout is paired with detailed structured text describing global scene summaries, relational placements of large furniture, and fine-grained arrangements of smaller items. This diverse and richly annotated resource enables models to learn complex spatial and semantic patterns across a wide variety of indoor environments. To assess the potential of M3DLayout, we establish a benchmark using both a text-conditioned diffusion model and a text-conditioned autoregressive model. Experimental results demonstrate that our dataset provides a solid foundation for training layout generation models. Its multi-source composition enhances diversity, notably through the Inf3DLayout subset which provides rich small-object information, enabling the generation of more complex and detailed scenes. We hope that M3DLayout can serve as a valuable resource for advancing research in text-driven 3D scene synthesis. All dataset and code will be made public upon acceptance.
comment : https://graphic-kiliani.github.io/M3DLayout/
♻ ☆ WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions ICCV 2025
Zizhang Li, Hong-Xing Yu, Wei Liu, Yin Yang, Charles Herrmann, Gordon Wetzstein, Jiajun Wu
WonderPlay is a novel framework integrating physics simulation with video generation for generating action-conditioned dynamic 3D scenes from a single image. While prior works are restricted to rigid body or simple elastic dynamics, WonderPlay features a hybrid generative simulator to synthesize a wide range of 3D dynamics. The hybrid generative simulator first uses a physics solver to simulate coarse 3D dynamics, which subsequently conditions a video generator to produce a video with finer, more realistic motion. The generated video is then used to update the simulated dynamic 3D scene, closing the loop between the physics solver and the video generator. This approach enables intuitive user control to be combined with the accurate dynamics of physics-based simulators and the expressivity of diffusion-based video generators. Experimental results demonstrate that WonderPlay enables users to interact with various scenes of diverse content, including cloth, sand, snow, liquid, smoke, elastic, and rigid bodies -- all using a single image input. Code will be made public. Project website: https://kyleleey.github.io/WonderPlay/
comment : ICCV 2025 (Highlight). The first two authors contributed equally. Project website: https://kyleleey.github.io/WonderPlay/
♻ ☆ Proxy-Tuning: Tailoring Multimodal Autoregressive Models for Subject-Driven Image Generation
Yi Wu, Shengju Qian, Lingting Zhu, Lei Liu, Wandi Qiao, Ziqiang Li, Lequan Yu, Bin Li
Multimodal autoregressive (AR) models, based on next-token prediction and transformer architecture, have demonstrated remarkable capabilities in various multimodal tasks including text-to-image (T2I) generation. Despite their strong performance in general T2I tasks, our research reveals that these models initially struggle with subject-driven image generation compared to dominant diffusion models. To address this limitation, we introduce Proxy-Tuning, leveraging diffusion models to enhance AR models' capabilities in subject-specific image generation. Our method reveals a striking weak-to-strong phenomenon: fine-tuned AR models consistently outperform their diffusion model supervisors in both subject fidelity and prompt adherence. We analyze this performance shift and identify scenarios where AR models excel, particularly in multi-subject compositions and contextual understanding. This work not only demonstrates impressive results in subject-driven AR image generation, but also unveils the potential of weak-to-strong generalization in the image generation domain, contributing to a deeper understanding of different architectures' strengths and limitations.
♻ ☆ Wo rl dScore: A Unified Evaluation Benchmark for Wo rl d Generation ICCV 2025
Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, Jiajun Wu
We introduce the WorldScore benchmark, the first unified benchmark for world generation. We decompose world generation into a sequence of next-scene generation tasks with explicit camera trajectory-based layout specifications, enabling unified evaluation of diverse approaches from 3D and 4D scene generation to video generation models. The WorldScore benchmark encompasses a curated dataset of 3,000 test examples that span diverse worlds: static and dynamic, indoor and outdoor, photorealistic and stylized. The WorldScore metrics evaluate generated worlds through three key aspects: controllability, quality, and dynamics. Through extensive evaluation of 19 representative models, including both open-source and closed-source ones, we reveal key insights and challenges for each category of models. Our dataset, evaluation code, and leaderboard can be found at https://haoyi-duan.github.io/WorldScore/
comment : ICCV 2025. Project website: https://haoyi-duan.github.io/WorldScore/ The first two authors contributed equally
♻ ☆ Augmenting Moment Retrieval: Zero-Dependency Two-Stage Learning ICCV 2025
Zhengxuan Wei, Jiajin Tang, Sibei Yang
Existing Moment Retrieval methods face three critical bottlenecks: (1) data scarcity forces models into shallow keyword-feature associations; (2) boundary ambiguity in transition regions between adjacent events; (3) insufficient discrimination of fine-grained semantics (e.g., distinguishing ``kicking" vs. ``throwing" a ball). In this paper, we propose a zero-external-dependency Augmented Moment Retrieval framework, AMR, designed to overcome local optima caused by insufficient data annotations and the lack of robust boundary and semantic discrimination capabilities. AMR is built upon two key insights: (1) it resolves ambiguous boundary information and semantic confusion in existing annotations without additional data (avoiding costly manual labeling), and (2) it preserves boundary and semantic discriminative capabilities enhanced by training while generalizing to real-world scenarios, significantly improving performance. Furthermore, we propose a two-stage training framework with cold-start and distillation adaptation. The cold-start stage employs curriculum learning on augmented data to build foundational boundary/semantic awareness. The distillation stage introduces dual query sets: Original Queries maintain DETR-based localization using frozen Base Queries from the cold-start model, while Active Queries dynamically adapt to real-data distributions. A cross-stage distillation loss enforces consistency between Original and Base Queries, preventing knowledge forgetting while enabling real-world generalization. Experiments on multiple benchmarks show that AMR achieves improved performance over prior state-of-the-art approaches.
comment : This work is accepted by ICCV 2025
♻ ☆ Cross Modal Fine-Grained Alignment via Granularity-Aware and Region-Uncertain Modeling AAAI 2026
Jiale Liu, Haoming Zhou, Yishu Liu, Bingzhi Chen, Yuncheng Jiang
Fine-grained image-text alignment is a pivotal challenge in multimodal learning, underpinning key applications such as visual question answering, image captioning, and vision-language navigation. Unlike global alignment, fine-grained alignment requires precise correspondence between localized visual regions and textual tokens, often hindered by noisy attention mechanisms and oversimplified modeling of cross-modal relationships. In this work, we identify two fundamental limitations of existing approaches: the lack of robust intra-modal mechanisms to assess the significance of visual and textual tokens, leading to poor generalization in complex scenes; and the absence of fine-grained uncertainty modeling, which fails to capture the one-to-many and many-to-one nature of region-word correspondences. To address these issues, we propose a unified approach that incorporates significance-aware and granularity-aware modeling and region-level uncertainty modeling. Our method leverages modality-specific biases to identify salient features without relying on brittle cross-modal attention, and represents region features as a mixture of Gaussian distributions to capture fine-grained uncertainty. Extensive experiments on Flickr30K and MS-COCO demonstrate that our approach achieves state-of-the-art performance across various backbone architectures, significantly enhancing the robustness and interpretability of fine-grained image-text alignment.
comment : 10 pages, 6 figures, accepted by AAAI 2026
♻ ☆ PANDA -- Patch And Distribution-Aware Augmentation for Long-Tailed Exemplar-Free Continual Learning AAAI 2026
Siddeshwar Raghavan, Jiangpeng He, Fengqing Zhu
Exemplar-Free Continual Learning (EFCL) restricts the storage of previous task data and is highly susceptible to catastrophic forgetting. While pre-trained models (PTMs) are increasingly leveraged for EFCL, existing methods often overlook the inherent imbalance of real-world data distributions. We discovered that real-world data streams commonly exhibit dual-level imbalances, dataset-level distributions combined with extreme or reversed skews within individual tasks, creating both intra-task and inter-task disparities that hinder effective learning and generalization. To address these challenges, we propose PANDA, a Patch-and-Distribution-Aware Augmentation framework that integrates seamlessly with existing PTM-based EFCL methods. PANDA amplifies low-frequency classes by using a CLIP encoder to identify representative regions and transplanting those into frequent-class samples within each task. Furthermore, PANDA incorporates an adaptive balancing strategy that leverages prior task distributions to smooth inter-task imbalances, reducing the overall gap between average samples across tasks and enabling fairer learning with frozen PTMs. Extensive experiments and ablation studies demonstrate PANDA's capability to work with existing PTM-based CL methods, improving accuracy and reducing catastrophic forgetting.
comment : Accepted in AAAI 2026 Main Technical Track
Computation and Language 51
☆ A Comparison of Human and Chat GPT Classification Performance on Complex Social Media Data
Breanna E. Green, Ashley L. Shea, Pengfei Zhao, Drew B. Margolin
Generative artificial intelligence tools, like ChatGPT, are an increasingly utilized resource among computational social scientists. Nevertheless, there remains space for improved understanding of the performance of ChatGPT in complex tasks such as classifying and annotating datasets containing nuanced language. Method. In this paper, we measure the performance of GPT-4 on one such task and compare results to human annotators. We investigate ChatGPT versions 3.5, 4, and 4o to examine performance given rapid changes in technological advancement of large language models. We craft four prompt styles as input and evaluate precision, recall, and F1 scores. Both quantitative and qualitative evaluations of results demonstrate that while including label definitions in prompts may help performance, overall GPT-4 has difficulty classifying nuanced language. Qualitative analysis reveals four specific findings. Our results suggest the use of ChatGPT in classification tasks involving nuanced language should be conducted with prudence.
comment : About 15 pages, draft version of accepted conference full paper. Published paper to follow
☆ Graphing the Truth: Structured Visualizations for Automated Hallucination Detection in LLMs
Tanmay Agrawal
Large Language Models have rapidly advanced in their ability to interpret and generate natural language. In enterprise settings, they are frequently augmented with closed-source domain knowledge to deliver more contextually informed responses. However, operational constraints such as limited context windows and inconsistencies between pre-training data and supplied knowledge often lead to hallucinations, some of which appear highly credible and escape routine human review. Current mitigation strategies either depend on costly, large-scale gold-standard Q\&A curation or rely on secondary model verification, neither of which offers deterministic assurance. This paper introduces a framework that organizes proprietary knowledge and model-generated content into interactive visual knowledge graphs. The objective is to provide end users with a clear, intuitive view of potential hallucination zones by linking model assertions to underlying sources of truth and indicating confidence levels. Through this visual interface, users can diagnose inconsistencies, identify weak reasoning chains, and supply corrective feedback. The resulting human-in-the-loop workflow creates a structured feedback loop that can enhance model reliability and continuously improve response quality.
☆ Sycophancy Claims about Language Models: The Missing Human-in-the-Loop NeurIPS 2025
Jan Batzner, Volker Stocker, Stefan Schmid, Gjergji Kasneci
Sycophantic response patterns in Large Language Models (LLMs) have been increasingly claimed in the literature. We review methodological challenges in measuring LLM sycophancy and identify five core operationalizations. Despite sycophancy being inherently human-centric, current research does not evaluate human perception. Our analysis highlights the difficulties in distinguishing sycophantic responses from related concepts in AI alignment and offers actionable recommendations for future research.
comment : NeurIPS 2025 Workshop on LLM Evaluation and ICLR 2025 Workshop on Bi-Directional Human-AI Alignment
☆ Melody or Machine: Detecting Synthetic Music with Dual-Stream Contrastive Learning
Arnesh Batra, Dev Sharma, Krish Thukral, Ruhani Bhatia, Naman Batra, Aditya Gautam
The rapid evolution of end-to-end AI music generation poses an escalating threat to artistic authenticity and copyright, demanding detection methods that can keep pace. While foundational, existing models like SpecTTTra falter when faced with the diverse and rapidly advancing ecosystem of new generators, exhibiting significant performance drops on out-of-distribution (OOD) content. This generalization failure highlights a critical gap: the need for more challenging benchmarks and more robust detection architectures. To address this, we first introduce Melody or Machine (MoM), a new large-scale benchmark of over 130,000 songs (6,665 hours). MoM is the most diverse dataset to date, built with a mix of open and closed-source models and a curated OOD test set designed specifically to foster the development of truly generalizable detectors. Alongside this benchmark, we introduce CLAM, a novel dual-stream detection architecture. We hypothesize that subtle, machine-induced inconsistencies between vocal and instrumental elements, often imperceptible in a mixed signal, offer a powerful tell-tale sign of synthesis. CLAM is designed to test this hypothesis by employing two distinct pre-trained audio encoders (MERT and Wave2Vec2) to create parallel representations of the audio. These representations are fused by a learnable cross-aggregation module that models their inter-dependencies. The model is trained with a dual-loss objective: a standard binary cross-entropy loss for classification, complemented by a contrastive triplet loss which trains the model to distinguish between coherent and artificially mismatched stream pairings, enhancing its sensitivity to synthetic artifacts without presuming a simple feature alignment. CLAM establishes a new state-of-the-art in synthetic music forensics. It achieves an F1 score of 0.925 on our challenging MoM benchmark.
comment : Accepted at Transactions on Machine Learning Research (TMLR)
☆ ART: Adaptive Response Tuning Framework -- A Multi- Agent Tournament-Based Approach to LLM Response Optimization
Omer Jauhar Khan
Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation. However, single-model responses often exhibit inconsistencies, hallucinations, and varying quality across different query domains. This paper presents ART (Adaptive Response Tuning), a novel framework that employs tournament-style ELO ranking and multi-agent reasoning to systematically optimize LLM outputs. By enabling multiple LLM agents to compete, critique, and collaborate through structured tournament workflows, ART produces consensus responses that outperform individual model outputs. Our framework introduces configurable tournament parameters, dynamic agent selection, and multiple consensus fusion strategies. Experimental evaluations demonstrate significant improvements in response accuracy, coherence, and reliability compared to baseline single-model approaches. The ART framework provides a scalable, production-ready solution for applications requiring high-quality, vetted LLM responses, achieving an 8.4% improvement in overall quality metrics and R22 values exceeding 0.96 in ELO rating convergence.
comment : 8 pages, 5 figures, 5 tables. Conference-style paper
☆ Prism: A Minimal Compositional Metalanguage for Specifying Agent Behavior
Franck Binard, Vanja Kljajevic
Prism is a small, compositional metalanguage for specifying the behaviour of tool-using software agents. Rather than introducing ad hoc control constructs, Prism is built around a fixed core context, Core1, which provides a minimal background grammar of categories numbers, strings, user prompts, tools together with abstract combinators for booleans, predicates, pairs, and lists. Agent policies are written as ordinary expressions using a single abstraction operator so that conditionals appear as selections between alternatives instead of imperative if-else blocks. Domains extend the core by defining their own context-mini-grammars that introduce new categories, predicates, and external tools while reusing the same compositional machinery. We illustrate this with worked examples from thermostat control, home security, e-commerce recommendation, and medical monitoring, showing how natural language decision rules can be mapped to inspectable, executable policies. From a linguistic perspective, Prism enforces a clear separation between a reusable grammar-like core and domain specific lexicons and treats tools as bridges between internal policy representations and the external world. From an engineering perspective, it offers a compact interface language for agent control, making the space of possible actions explicit and amenable to analysis, verification, and safety constraints.
☆ Wikontic: Constructing Wikidata-Aligned, Ontology-Aware Knowledge Graphs with Large Language Models
Alla Chepurova, Aydar Bulatov, Yuri Kuratov, Mikhail Burtsev
Knowledge graphs (KGs) provide structured, verifiable grounding for large language models (LLMs), but current LLM-based systems commonly use KGs as auxiliary structures for text retrieval, leaving their intrinsic quality underexplored. In this work, we propose Wikontic, a multi-stage pipeline that constructs KGs from open-domain text by extracting candidate triplets with qualifiers, enforcing Wikidata-based type and relation constraints, and normalizing entities to reduce duplication. The resulting KGs are compact, ontology-consistent, and well-connected; on MuSiQue, the correct answer entity appears in 96% of generated triplets. On HotpotQA, our triplets-only setup achieves 76.0 F1, and on MuSiQue 59.8 F1, matching or surpassing several retrieval-augmented generation baselines that still require textual context. In addition, Wikontic attains state-of-the-art information-retention performance on the MINE-1 benchmark (86%), outperforming prior KG construction methods. Wikontic is also efficient at build time: KG construction uses less than 1,000 output tokens, about 3$\times$ fewer than AriGraph and $<$1/20 of GraphRAG. The proposed pipeline enhances the quality of the generated KG and offers a scalable solution for leveraging structured knowledge in LLMs.
☆ Statistical NLP for Optimization of Clinical Trial Success Prediction in Pharmaceutical R&D
Michael R. Doane
This work presents the development and evaluation of an NLP-enabled probabilistic classifier designed to estimate the probability of technical and regulatory success (pTRS) for clinical trials in the field of neuroscience. While pharmaceutical R&D is plagued by high attrition rates and enormous costs, particularly within neuroscience, where success rates are below 10%, timely identification of promising programs can streamline resource allocation and reduce financial risk. Leveraging data from the ClinicalTrials.gov database and success labels from the recently developed Clinical Trial Outcome dataset, the classifier extracts text-based clinical trial features using statistical NLP techniques. These features were integrated into several non-LLM frameworks (logistic regression, gradient boosting, and random forest) to generate calibrated probability scores. Model performance was assessed on a retrospective dataset of 101,145 completed clinical trials spanning 1976-2024, achieving an overall ROC-AUC of 0.64. An LLM-based predictive model was then built using BioBERT, a domain-specific language representation encoder. The BioBERT-based model achieved an overall ROC-AUC of 0.74 and a Brier Score of 0.185, indicating its predictions had, on average, 40% less squared error than would be observed using industry benchmarks. The BioBERT-based model also made trial outcome predictions that were superior to benchmark values 70% of the time overall. By integrating NLP-driven insights into drug development decision-making, this work aims to enhance strategic planning and optimize investment allocation in neuroscience programs.
comment : Doctor of Engineering Praxis Dissertation, The George Washington University. 122 pages. Present affiliation: Iambic Therapeutics
☆ Slovak Conceptual Dictionary
Miroslav Blšták
When solving tasks in the field of natural language processing, we sometimes need dictionary tools, such as lexicons, word form dictionaries or knowledge bases. However, the availability of dictionary data is insufficient in many languages, especially in the case of low resourced languages. In this article, we introduce a new conceptual dictionary for the Slovak language as the first linguistic tool of this kind. Since Slovak language is a language with limited linguistic resources and there are currently not available any machine-readable linguistic data sources with a sufficiently large volume of data, many tasks which require automated processing of Slovak text achieve weaker results compared to other languages and are almost impossible to solve.
comment : 7 pages, 2 figures
☆ Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relation s
Sina Salimian, Gias Uddin, Sumon Biswas, Henry Leung
The widespread deployment of Large Language Models (LLMs) has intensified concerns about subtle social biases embedded in their outputs. Existing guardrails often fail when faced with indirect or contextually complex bias-inducing prompts. To address these limitations, we propose a unified framework for both systematic bias evaluation and targeted mitigation. Our approach introduces six novel Metamorphic Relations (MRs) that, based on metamorphic testing principles, transform direct bias-inducing inputs into semantically equivalent yet adversarially challenging variants. These transformations enable an automated method for exposing hidden model biases: when an LLM responds inconsistently or unfairly across MR-generated variants, the underlying bias becomes detectable. We further show that the same MRs can be used to generate diverse bias-inducing samples for fine-tuning, directly linking the testing process to mitigation. Using six state-of-the-art LLMs - spanning open-source and proprietary models - and a representative subset of 385 questions from the 8,978-item BiasAsker benchmark covering seven protected groups, our MRs reveal up to 14% more hidden biases compared to existing tools. Moreover, fine-tuning with both original and MR-mutated samples significantly enhances bias resiliency, increasing safe response rates from 54.7% to over 88.9% across models. These results highlight metamorphic relations as a practical mechanism for improving fairness in conversational AI.
☆ Catch Me If You Can: How Smaller Reasoning Models Pretend to Reason with Mathematical Fidelity
Subramanyam Sahoo, Vinija Jain, Saanidhya Vats, Siddharth Mohapatra, Rui Min, Aman Chadha, Divya Chaudhary
Current evaluation of mathematical reasoning in language models relies primarily on answer accuracy, potentially masking fundamental failures in logical computation. We introduce a diagnostic framework that distinguishes genuine mathematical reasoning from superficial pattern matching through four complementary axes: forward-backward consistency, transitivity coverage, counterfactual sensitivity, and perturbation robustness. Through a case study applying this framework to Qwen3-0.6B on the MenatQA dataset, we reveal a striking disconnect between surface performance and reasoning fidelity. While the model achieves reasonable answer accuracy (70%+), it demonstrates poor backward consistency (15%), limited transitivity coverage (32.2%), and brittle sensitivity to perturbations. Our diagnostics expose reasoning failures invisible to traditional accuracy metrics, suggesting that this small model relies heavily on pattern matching rather than genuine logical computation. While our empirical findings are based on a single 600M-parameter model, the diagnostic framework itself is model-agnostic and generalizable. We release our evaluation protocols to enable the research community to assess reasoning fidelity across different model scales and architectures, moving beyond surface-level accuracy toward verifiable mathematical reasoning.
comment : 8 pages, 5 figures. A preprint. Initial Work
☆ Rep3Net: An Approach Exploiting Multimodal Representation for Molecular Bioactivity Prediction
Sabrina Islam, Md. Atiqur Rahman, Md. Bakhtiar Hasan, Md. Hasanul Kabir
In early stage drug discovery, bioactivity prediction of molecules against target proteins plays a crucial role. Trdaitional QSAR models that utilizes molecular descriptor based data often struggles to predict bioactivity of molecules effectively due to its limitation in capturing structural and contextual information embedded within each compound. To address this challenge, we propose Rep3Net, a unified deep learning architecture that not only incorporates descriptor data but also includes spatial and relational information through graph-based represenation of compounds and contextual information through ChemBERTa generated embeddings from SMILES strings. Our model employing multimodal concatenated features produce reliable bioactivity prediction on Poly [ADP-ribose] polymerase 1 (PARP-1) dataset. PARP-1 is a crucial agent in DNA damage repair and has become a significant theraputic target in malignancies that depend on it for survival and growth. A comprehensive analysis and comparison with conventional standalone models including GCN, GAT, XGBoost, etc. demonstrates that our architecture achieves the highest predictive performance. In computational screening of compounds in drug discovery, our architecture provides a scalable framework for bioactivity prediction.
☆ Developing a Comprehensive Framework for Sentiment Analysis in Turkish
Cem Rifki Aydin
In this thesis, we developed a comprehensive framework for sentiment analysis that takes its many aspects into account mainly for Turkish. We have also proposed several approaches specific to sentiment analysis in English only. We have accordingly made five major and three minor contributions. We generated a novel and effective feature set by combining unsupervised, semi-supervised, and supervised metrics. We then fed them as input into classical machine learning methods, and outperformed neural network models for datasets of different genres in both Turkish and English. We created a polarity lexicon with a semi-supervised domain-specific method, which has been the first approach applied for corpora in Turkish. We performed a fine morphological analysis for the sentiment classification task in Turkish by determining the polarities of morphemes. This can be adapted to other morphologically-rich or agglutinative languages as well. We have built a novel neural network architecture, which combines recurrent and recursive neural network models for English. We built novel word embeddings that exploit sentiment, syntactic, semantic, and lexical characteristics for both Turkish and English. We also redefined context windows as subclauses in modelling word representations in English. This can also be applied to other linguistic fields and natural language processing tasks. We have achieved state-of-the-art and significant results for all these original approaches. Our minor contributions include methods related to aspect-based sentiment in Turkish, parameter redefinition in the semi-supervised approach, and aspect term extraction techniques for English. This thesis can be considered the most detailed and comprehensive study made on sentiment analysis in Turkish as of July, 2020. Our work has also contributed to the opinion classification problem in English.
comment : Ph.D. Thesis, Bogazici University, 2020
☆ G-KV: Decoding-Time KV Cache Eviction with Global Attention
Mengqi Liao, Lu Wang, Chaoyun Zhang, Zekai Shen, Xiaowei Mao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Huaiyu Wan
Recent reasoning large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths. KV cache compression has emerged as an effective approach to greatly enhance the efficiency of reasoning. However, existing methods often focus on prompt compression or token eviction with local attention score, overlooking the long-term importance of tokens. We propose G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance. Additionally, we introduce post-training techniques, including reinforcement learning and distillation, to optimize models for compressed KV cache settings. The code of this paper is available on: https://github.com/microsoft/G-KV.
☆ CACARA: Cross-Modal Alignment Leveraging a Text-Centric Approach for Cost-Effective Multimodal and Multilingual Learning
Diego A. B. Moreira, Alef I. Ferreira, Jhessica Silva, Gabriel O. dos Santos, Gustavo Bonil, João Gondim, Marina dos Santos, Helena Maia, Simone Hashiguti, Nádia da Silva, Carolina Scarton, Helio Pedrini, Sandra Avila
As deep learning models evolve, new applications and challenges are rapidly emerging. Tasks that once relied on a single modality, such as text, images, or audio, are now enriched by seamless interactions between multimodal data. These connections bridge information gaps: an image can visually materialize a text, while audio can add context to an image. Researchers have developed numerous multimodal models, but most rely on resource-intensive training across multiple modalities. Similarly, extending these models to new languages often follows the same resource-heavy training strategy. In this work, we propose a multimodal and multilingual architecture, CACARA, trained through emergent alignment learning, enabling the seamless integration of new modalities into an existing bimodal/multimodal model without requiring full retraining. This work breaks new ground by demonstrating that this emergent alignment paradigm can unlock multilingual capabilities from monolingual training. By fine-tuning the newly incorporated modality only on data aligned with the English language, our model develops support for over 100 languages without explicit multilingual pretraining or tuning of the text encoder. Such emergent multimodal and multilingual properties are gained efficiently, preserving previously learned knowledge at a training cost comparable to that of a monolingual model. Our strategy achieves up to a 14.24 percentage points improvement in R@1 audio-to-text retrieval, outperforming state-of-the-art multimodal models -- all without the heavy computational cost of retraining across every modality and language.
comment : 25 pages, 12 tables, 5 figures
☆ SCALE: Selective Resource Allocation for Overcoming Performance Bottlenecks in Mathematical Test-time Scaling AAAI 2026
Yang Xiao, Chunpu Xu, Ruifeng Yuan, Jiashuo Wang, Wenjie Li, Pengfei Liu
Test-time compute scaling has emerged as a powerful paradigm for enhancing mathematical reasoning in large language models (LLMs) by allocating additional computational resources during inference. However, current methods employ uniform resource distribution across all reasoning sub-problems, creating fundamental bottlenecks where challenging sub-problems receive insufficient attention while routine operations consume disproportionate resources. This uniform allocation creates performance bottlenecks where additional computational resources yield diminishing returns. Inspired by dual-process theory, we propose \textbf{SCALE} (Selective Resource Allocation), a framework that selectively allocates computational resources based on sub-problem difficulty. SCALE operates through four stages: (1) problem decomposition into sequential reasoning sub-problems, (2) difficulty assessment of each sub-problem to distinguish between routine operations and computationally challenging sub-problems, (3) selective processing mode assignment between System 1 for simple sub-problems and System 2 for complex ones, and (4) sequential execution with context propagation. By concentrating resources on challenging sub-problems while processing routine operations efficiently, SCALE achieves substantial performance improvements with superior resource utilization. Extensive experiments demonstrate that SCALE significantly outperforms uniform scaling baselines, achieving accuracy improvements of up to 13.75 percentage points (57.50% to 71.25% on AIME25) while reducing computational costs by 33%-53%, representing a major advance in test-time scaling that addresses fundamental limitations of current approaches.
comment : accepted by AAAI 2026
☆ Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency AAAI
Jan Batzner, Volker Stocker, Bingjun Tang, Anusha Natarajan, Qinhao Chen, Stefan Schmid, Gjergji Kasneci
Synthetic personae experiments have become a prominent method in Large Language Model alignment research, yet the representativeness and ecological validity of these personae vary considerably between studies. Through a review of 63 peer-reviewed studies published between 2023 and 2025 in leading NLP and AI venues, we reveal a critical gap: task and population of interest are often underspecified in persona-based experiments, despite personalization being fundamentally dependent on these criteria. Our analysis shows substantial differences in user representation, with most studies focusing on limited sociodemographic attributes and only 35% discussing the representativeness of their LLM personae. Based on our findings, we introduce a persona transparency checklist that emphasizes representative sampling, explicit grounding in empirical data, and enhanced ecological validity. Our work provides both a comprehensive assessment of current practices and practical guidelines to improve the rigor and ecological validity of persona-based evaluations in language model alignment research.
comment : Published at AAAI/ACM AIES 2025. Presented at NeurIPS 2025 Workshop Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling
☆ CryptoBench: A Dynamic Benchmark for Expert-Level Evaluation of LLM Agent s in Cryptocurrency
Jiacheng Guo, Suozhi Huang, Zixin Yao, Yifan Zhang, Yifu Lu, Jiashuo Liu, Zihao Li, Yanyan Deng, Qixin Xiao, Jia Tian, Kanghong Zhan, Tianyi Li, Xiaochen Liu, Jason Ge, Chaoyang He, Kaixuan Huang, Lin Yang, Wenhao Huang, Mengdi Wang
This paper introduces CryptoBench, the first expert-curated, dynamic benchmark designed to rigorously evaluate the real-world capabilities of Large Language Model (LLM) agents in the uniquely demanding and fast-paced cryptocurrency domain. Unlike general-purpose agent benchmarks for search and prediction, professional crypto analysis presents specific challenges: \emph{extreme time-sensitivity}, \emph{a highly adversarial information environment}, and the critical need to synthesize data from \emph{diverse, specialized sources}, such as on-chain intelligence platforms and real-time Decentralized Finance (DeFi) dashboards. CryptoBench thus serves as a much more challenging and valuable scenario for LLM agent assessment. To address these challenges, we constructed a live, dynamic benchmark featuring 50 questions per month, expertly designed by crypto-native professionals to mirror actual analyst workflows. These tasks are rigorously categorized within a four-quadrant system: Simple Retrieval, Complex Retrieval, Simple Prediction, and Complex Prediction. This granular categorization enables a precise assessment of an LLM agent's foundational data-gathering capabilities alongside its advanced analytical and forecasting skills. Our evaluation of ten LLMs, both directly and within an agentic framework, reveals a performance hierarchy and uncovers a failure mode. We observe a \textit{retrieval-prediction imbalance}, where many leading models, despite being proficient at data retrieval, demonstrate a pronounced weakness in tasks requiring predictive analysis. This highlights a problematic tendency for agents to appear factually grounded while lacking the deeper analytical capabilities to synthesize information.
☆ A Taxonomy of Errors in English as she is spoke: Toward an AI-Based Method of Error Analysis for EFL Writing Instruction
Damian Heywood, Joseph Andrew Carrier, Kyu-Hong Hwang
This study describes the development of an AI-assisted error analysis system designed to identify, categorize, and correct writing errors in English. Utilizing Large Language Models (LLMs) like Claude 3.5 Sonnet and DeepSeek R1, the system employs a detailed taxonomy grounded in linguistic theories from Corder (1967), Richards (1971), and James (1998). Errors are classified at both word and sentence levels, covering spelling, grammar, and punctuation. Implemented through Python-coded API calls, the system provides granular feedback beyond traditional rubric-based assessments. Initial testing on isolated errors refined the taxonomy, addressing challenges like overlapping categories. Final testing used "English as she is spoke" by Jose da Fonseca (1855), a text rich with authentic linguistic errors, to evaluate the system's capacity for handling complex, multi-layered analysis. The AI successfully identified diverse error types but showed limitations in contextual understanding and occasionally generated new error categories when encountering uncoded errors. This research demonstrates AI's potential to transform EFL instruction by automating detailed error analysis and feedback. While promising, further development is needed to improve contextual accuracy and expand the taxonomy to stylistic and discourse-level errors.
comment : Metadata at "Replication Data for: A Taxonomy of Errors in English as she is spoke: An AI-Based System for Error Analysis for EFL Writing Instruction", https://doi.org/10.7910/DVN/N5O7C4, Harvard Dataverse, V1
☆ Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusing
Nuo Chen, Hanpei Fang, Jiqun Liu, Wilson Wei, Tetsuya Sakai, Xiao-Ming Wu
Recent research has explored LLMs as scalable tools for relevance labeling, but studies indicate they are susceptible to priming effects, where prior relevance judgments influence later ones. Although psychological theories link personality traits to such biases, it is unclear whether simulated personalities in LLMs exhibit similar effects. We investigate how Big Five personality profiles in LLMs influence priming in relevance labeling, using multiple LLMs on TREC 2021 and 2022 Deep Learning Track datasets. Our results show that certain profiles, such as High Openness and Low Neuroticism, consistently reduce priming susceptibility. Additionally, the most effective personality in mitigating priming may vary across models and task types. Based on these findings, we propose personality prompting as a method to mitigate threshold priming, connecting psychological evidence with LLM-based evaluation practices.
☆ Breaking It Down: Domain-Aware Semantic Segmentation for Retrieval Augmented Generation
Aparajitha Allamraju, Maitreya Prafulla Chitale, Hiranmai Sri Adibhatla, Rahul Mishra, Manish Shrivastava
Document chunking is a crucial component of Retrieval-Augmented Generation (RAG), as it directly affects the retrieval of relevant and precise context. Conventional fixed-length and recursive splitters often produce arbitrary, incoherent segments that fail to preserve semantic structure. Although semantic chunking has gained traction, its influence on generation quality remains underexplored. This paper introduces two efficient semantic chunking methods, Projected Similarity Chunking (PSC) and Metric Fusion Chunking (MFC), trained on PubMed data using three different embedding models. We further present an evaluation framework that measures the effect of chunking on both retrieval and generation by augmenting PubMedQA with full-text PubMed Central articles. Our results show substantial retrieval improvements (24x with PSC) in MRR and higher Hits@k on PubMedQA. We provide a comprehensive analysis, including statistical significance and response-time comparisons with common chunking libraries. Despite being trained on a single domain, PSC and MFC also generalize well, achieving strong out-of-domain generation performance across multiple datasets. Overall, our findings confirm that our semantic chunkers, especially PSC, consistently deliver superior performance.
☆ CourseTimeQA: A Lecture-Video Benchmark and a Latency-Constrained Cross-Modal Fusion Method for Timestamped QA
Vsevolod Kovalev, Parteek Kumar
We study timestamped question answering over educational lecture videos under a single-GPU latency/memory budget. Given a natural-language query, the system retrieves relevant timestamped segments and synthesizes a grounded answer. We present CourseTimeQA (52.3 h, 902 queries across six courses) and a lightweight, latency-constrained cross-modal retriever (CrossFusion-RAG) that combines frozen encoders, a learned 512->768 vision projection, shallow query-agnostic cross-attention over ASR and frames with a temporal-consistency regularizer, and a small cross-attentive reranker. On CourseTimeQA, CrossFusion-RAG improves nDCG@10 by 0.10 and MRR by 0.08 over a strong BLIP-2 retriever while achieving approximately 1.55 s median end-to-end latency on a single A100. Closest comparators (zero-shot CLIP multi-frame pooling; CLIP + cross-encoder reranker + MMR; learned late-fusion gating; text-only hybrid with cross-encoder reranking and its MMR variant; caption-augmented text retrieval; non-learned temporal smoothing) are evaluated under matched hardware and indexing. We report robustness across ASR noise (WER quartiles), diagnostics for temporal localization, and full training/tuning details to support reproducible comparison.
comment : 5 figures, 8 tables
☆ IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages
Ayush Maheshwari, Kaushal Sharma, Vivek Patel, Aditya Maheshwari
While large language models excel on high-resource multilingual tasks, low- and extremely low-resource Indic languages remain severely under-evaluated. We present IndicParam, a human-curated benchmark of over 13,000 multiple-choice questions covering 11 such languages (Nepali, Gujarati, Marathi, Odia as low-resource; Dogri, Maithili, Rajasthani, Sanskrit, Bodo, Santali, Konkani as extremely low-resource) plus Sanskrit-English code-mixed set. We evaluated 19 LLMs, both proprietary and open-weights, which reveals that even the top-performing GPT-5 reaches only 45.0% average accuracy, followed by DeepSeek-3.2 (43.1) and Claude-4.5 (42.7). We additionally label each question as knowledge-oriented or purely linguistic to discriminate factual recall from grammatical proficiency. Further, we assess the ability of LLMs to handle diverse question formats-such as list-based matching, assertion-reason pairs, and sequence ordering-alongside conventional multiple-choice questions. IndicParam provides insights into limitations of cross-lingual transfer and establishes a challenging benchmark for Indic languages. The dataset is available at https://huggingface.co/datasets/bharatgenai/IndicParam. Scripts to run benchmark are present at https://github.com/ayushbits/IndicParam.
☆ Assertion-Conditioned Compliance: A Provenance-Aware Vulnerability in Multi-Turn Tool-Calling Agent s
Daud Waqas, Aaryamaan Golthi, Erika Hayashida, Huanzhi Mao
Multi-turn tool-calling LLMs (models capable of invoking external APIs or tools across several user turns) have emerged as a key feature in modern AI assistants, enabling extended dialogues from benign tasks to critical business, medical, and financial operations. Yet implementing multi-turn pipelines remains difficult for many safety-critical industries due to ongoing concerns regarding model resilience. While standardized benchmarks such as the Berkeley Function-Calling Leaderboard (BFCL) have underpinned confidence concerning advanced function-calling models (like Salesforce's xLAM V2), there is still a lack of visibility into multi-turn conversation-level robustness, especially given their exposure to real-world systems. In this paper, we introduce Assertion-Conditioned Compliance (A-CC), a novel evaluation paradigm for multi-turn function-calling dialogues. A-CC provides holistic metrics that evaluate a model's behavior when confronted with misleading assertions originating from two distinct vectors: (1) user-sourced assertions (USAs), which measure sycophancy toward plausible but misinformed user beliefs, and (2) function-sourced assertions (FSAs), which measure compliance with plausible but contradictory system policies (e.g., stale hints from unmaintained tools). Our results show that models are highly vulnerable to both USA sycophancy and FSA policy conflicts, confirming A-CC as a critical, latent vulnerability in deployed agents.
comment : 15 pages (incl. Appendix), 2 figures, 7 tables
☆ Evidence-Guided Schema Normalization for Temporal Tabular Reasoning
Ashish Thanga, Vibhu Dixit, Abhilash Shankarampeta, Vivek Gupta
Temporal reasoning over evolving semi-structured tables poses a challenge to current QA systems. We propose a SQL-based approach that involves (1) generating a 3NF schema from Wikipedia infoboxes, (2) generating SQL queries, and (3) query execution. Our central finding challenges model scaling assumptions: the quality of schema design has a greater impact on QA precision than model capacity. We establish three evidence-based principles: normalization that preserves context, semantic naming that reduces ambiguity, and consistent temporal anchoring. Our best configuration (Gemini 2.5 Flash schema + Gemini-2.0-Flash queries) achieves 80.39 EM, a 16.8\% improvement over the baseline (68.89 EM).
☆ Progressive Code Integration for Abstractive Bug Report Summarization
Shaira Sadia Karim, Abrar Mahmud Rahim, Lamia Alam, Ishmam Tashdeed, Lutfun Nahar Lota, Md. Abu Raihan M. Kamal, Md. Azam Hossain
Bug reports are often unstructured and verbose, making it challenging for developers to efficiently comprehend software issues. Existing summarization approaches typically rely on surface-level textual cues, resulting in incomplete or redundant summaries, and they frequently ignore associated code snippets, which are essential for accurate defect diagnosis. To address these limitations, we propose a progressive code-integration framework for LLM-based abstractive bug report summarization. Our approach incrementally incorporates long code snippets alongside textual content, overcoming standard LLM context window constraints and producing semantically rich summaries. Evaluated on four benchmark datasets using eight LLMs, our pipeline outperforms extractive baselines by 7.5%-58.2% and achieves performance comparable to state-of-the-art abstractive methods, highlighting the benefits of jointly leveraging textual and code information for enhanced bug comprehension.
☆ Comparative Analysis of 47 Context -Based Question Answer Models Across 8 Diverse Dataset s
Muhammad Muneeb, David B. Ascher, Ahsan Baidar Bakht
Context-based question answering (CBQA) models provide more accurate and relevant answers by considering the contextual information. They effectively extract specific information given a context, making them functional in various applications involving user support, information retrieval, and educational platforms. In this manuscript, we benchmarked the performance of 47 CBQA models from Hugging Face on eight different datasets. This study aims to identify the best-performing model across diverse datasets without additional fine-tuning. It is valuable for practical applications where the need to retrain models for specific datasets is minimized, streamlining the implementation of these models in various contexts. The best-performing models were trained on the SQuAD v2 or SQuAD v1 datasets. The best-performing model was ahotrod/electra_large_discriminator_squad2_512, which yielded 43\% accuracy across all datasets. We observed that the computation time of all models depends on the context length and the model size. The model's performance usually decreases with an increase in the answer length. Moreover, the model's performance depends on the context complexity. We also used the Genetic algorithm to improve the overall accuracy by integrating responses from other models. ahotrod/electra_large_discriminator_squad2_512 generated the best results for bioasq10b-factoid (65.92\%), biomedical\_cpgQA (96.45\%), QuAC (11.13\%), and Question Answer Dataset (41.6\%). Bert-large-uncased-whole-word-masking-finetuned-squad achieved an accuracy of 82\% on the IELTS dataset.
☆ Challenges of Heterogeneity in Big Data: A Comparative Study of Classification in Large-Scale Structured and Unstructured Domains
González Trigueros Jesús Eduardo, Alonso Sánchez Alejandro, Muñoz Rivera Emilio, Peñarán Prieto Mariana Jaqueline, Mendoza González Camila Natalia
This study analyzes the impact of heterogeneity ("Variety") in Big Data by comparing classification strategies across structured (Epsilon) and unstructured (Rest-Mex, IMDB) domains. A dual methodology was implemented: evolutionary and Bayesian hyperparameter optimization (Genetic Algorithms, Optuna) in Python for numerical data, and distributed processing in Apache Spark for massive textual corpora. The results reveal a "complexity paradox": in high-dimensional spaces, optimized linear models (SVM, Logistic Regression) outperformed deep architectures and Gradient Boosting. Conversely, in text-based domains, the constraints of distributed fine-tuning led to overfitting in complex models, whereas robust feature engineering -- specifically Transformer-based embeddings (ROBERTa) and Bayesian Target Encoding -- enabled simpler models to generalize effectively. This work provides a unified framework for algorithm selection based on data nature and infrastructure constraints.
comment : 13 pages, 1 figure, 3 tables. Comparative study involving Apache Spark and Hyperparameter Optimization. Keywords: Big Data, NLP, Tabular Data
☆ EduEval: A Hierarchical Cognitive Benchmark for Evaluating Large Language Models in Chinese Education
Guoqing Ma, Jia Zhu, Hanghui Guo, Weijie Shi, Yue Cui, Jiawei Shen, Zilong Li, Yidan Liang
Large language models (LLMs) demonstrate significant potential for educational applications. However, their unscrutinized deployment poses risks to educational standards, underscoring the need for rigorous evaluation. We introduce EduEval, a comprehensive hierarchical benchmark for evaluating LLMs in Chinese K-12 education. This benchmark makes three key contributions: (1) Cognitive Framework: We propose the EduAbility Taxonomy, which unifies Bloom's Taxonomy and Webb's Depth of Knowledge to organize tasks across six cognitive dimensions including Memorization, Understanding, Application, Reasoning, Creativity, and Ethics. (2) Authenticity: Our benchmark integrates real exam questions, classroom conversation, student essays, and expert-designed prompts to reflect genuine educational challenges; (3) Scale: EduEval comprises 24 distinct task types with over 11,000 questions spanning primary to high school levels. We evaluate 14 leading LLMs under both zero-shot and few-shot settings, revealing that while models perform well on factual tasks, they struggle with classroom dialogue classification and exhibit inconsistent results in creative content generation. Interestingly, several open source models outperform proprietary systems on complex educational reasoning. Few-shot prompting shows varying effectiveness across cognitive dimensions, suggesting that different educational objectives require tailored approaches. These findings provide targeted benchmarking metrics for developing LLMs specifically optimized for diverse Chinese educational tasks.
☆ Lost without translation -- Can transformer (language models) understand mood states?
Prakrithi Shivaprakash, Diptadhi Mukherjee, Lekhansh Shukla, Animesh Mukherjee, Prabhat Chand, Pratima Murthy
Background: Large Language Models show promise in psychiatry but are English-centric. Their ability to understand mood states in other languages is unclear, as different languages have their own idioms of distress. Aim: To quantify the ability of language models to faithfully represent phrases (idioms of distress) of four distinct mood states (depression, euthymia, euphoric mania, dysphoric mania) expressed in Indian languages. Methods: We collected 247 unique phrases for the four mood states across 11 Indic languages. We tested seven experimental conditions, comparing k-means clustering performance on: (a) direct embeddings of native and Romanised scripts (using multilingual and Indic-specific models) and (b) embeddings of phrases translated to English and Chinese. Performance was measured using a composite score based on Adjusted Rand Index, Normalised Mutual Information, Homogeneity and Completeness. Results: Direct embedding of Indic languages failed to cluster mood states (Composite Score = 0.002). All translation-based approaches showed significant improvement. High performance was achieved using Gemini-translated English (Composite=0.60) and human-translated English (Composite=0.61) embedded with gemini-001. Surprisingly, human-translated English, further translated into Chinese and embedded with a Chinese model, performed best (Composite = 0.67). Specialised Indic models (IndicBERT and Sarvam-M) performed poorly. Conclusion: Current models cannot meaningfully represent mood states directly from Indic languages, posing a fundamental barrier to their psychiatric application for diagnostic or therapeutic purposes in India. While high-quality translation bridges this gap, reliance on proprietary models or complex translation pipelines is unsustainable. Models must first be built to understand diverse local languages to be effective in global mental health.
comment : 33 pages, 3 figures, 2 tables
♻ ☆ NarraBench: A Comprehensive Framework for Narrative Benchmark ing
Sil Hamilton, Matthew Wilkens, Andrew Piper
We present NarraBench, a theory-informed taxonomy of narrative-understanding tasks, as well as an associated survey of 78 existing benchmarks in the area. We find significant need for new evaluations covering aspects of narrative understanding that are either overlooked in current work or are poorly aligned with existing metrics. Specifically, we estimate that only 27% of narrative tasks are well captured by existing benchmarks, and we note that some areas -- including narrative events, style, perspective, and revelation -- are nearly absent from current evaluations. We also note the need for increased development of benchmarks capable of assessing constitutively subjective and perspectival aspects of narrative, that is, aspects for which there is generally no single correct answer. Our taxonomy, survey, and methodology are of value to NLP researchers seeking to test LLM narrative understanding.
♻ ☆ Superposition Yields Robust Neural Scaling NeurIPS 2025
Yizhou Liu, Ziming Liu, Jeff Gore
The success of today's large language models (LLMs) depends on the observation that larger models perform better. However, the origin of this neural scaling law, that loss decreases as a power law with model size, remains unclear. We propose that representation superposition, meaning that LLMs represent more features than they have dimensions, can be a key contributor to loss and cause neural scaling. Based on Anthropic's toy model, we use weight decay to control the degree of superposition, allowing us to systematically study how loss scales with model size. When superposition is weak, the loss follows a power law only if data feature frequencies are power-law distributed. In contrast, under strong superposition, the loss generically scales inversely with model dimension across a broad class of frequency distributions, due to geometric overlaps between representation vectors. We confirmed that open-sourced LLMs operate in the strong superposition regime and have loss scaling inversely with model dimension, and that the Chinchilla scaling laws are also consistent with this behavior. Our results identify representation superposition as a central driver of neural scaling laws, providing insights into questions like when neural scaling laws can be improved and when they will break down.
comment : Best Paper Runner-up at NeurIPS 2025
♻ ☆ PEPPER: Perception-Guided Perturbation for Robust Backdoor Defense in Text-to-Image Diffusion Models
Oscar Chew, Po-Yi Lu, Jayden Lin, Kuan-Hao Huang, Hsuan-Tien Lin
Recent studies show that text to image (T2I) diffusion models are vulnerable to backdoor attacks, where a trigger in the input prompt can steer generation toward harmful or unintended content. To address this, we introduce PEPPER (PErcePtion Guided PERturbation), a backdoor defense that rewrites the caption into a semantically distant yet visually similar caption while adding unobstructive elements. With this rewriting strategy, PEPPER disrupt the trigger embedded in the input prompt, dilute the influence of trigger tokens and thereby achieve enhanced robustness. Experiments show that PEPPER is particularly effective against text encoder based attacks, substantially reducing attack success while preserving generation quality. Beyond this, PEPPER can be paired with any existing defenses yielding consistently stronger and generalizable robustness than any standalone method. Our code will be released on Github.
♻ ☆ Escaping Collapse: The Strength of Weak Data for Large Language Model Training
Kareem Amin, Sara Babakniya, Alex Bie, Weiwei Kong, Umar Syed, Sergei Vassilvitskii
Synthetically-generated data plays an increasingly larger role in training large language models. However, while synthetic data has been found to be useful, studies have also shown that without proper curation it can cause LLM performance to plateau, or even "collapse", after many training iterations. In this paper, we formalize this question and develop a theoretical framework to investigate how much curation is needed in order to ensure that LLM performance continually improves. Our analysis is inspired by boosting, a classic machine learning technique that leverages a very weak learning algorithm to produce an arbitrarily good classifier. The approach we analyze subsumes many recently proposed methods for training LLMs on synthetic data, and thus our analysis sheds light on why they are successful, and also suggests opportunities for future improvement. We present experiments that validate our theory, and show that dynamically focusing labeling resources on the most challenging examples -- in much the same way that boosting focuses the efforts of the weak learner -- leads to improved performance.
♻ ☆ Life-Code: Central Dogma Modeling with Multi-Omics Sequence Unification
Zicheng Liu, Siyuan Li, Zhiyuan Chen, Chang Yu, Qirong Yang, Yucheng Guo, Yujie Yang, Xiaoming Zhang, Stan Z. Li
The interactions between DNA, RNA, and proteins are fundamental to biological processes, as illustrated by the central dogma of molecular biology. Although modern biological pre-trained models have achieved great success in analyzing these macromolecules individually, their interconnected nature remains underexplored. This paper follows the guidance of the central dogma to redesign both the data and model pipeline and offers a comprehensive framework, Life-Code, that spans different biological functions. As for data flow, we propose a unified pipeline to integrate multi-omics data by reverse-transcribing RNA and reverse-translating amino acids into nucleotide-based sequences. As for the model, we design a codon tokenizer and a hybrid long-sequence architecture to encode the interactions between coding and non-coding regions through masked modeling pre-training. To model the translation and folding process with coding sequences, Life-Code learns protein structures of the corresponding amino acids by knowledge distillation from off-the-shelf protein language models. Such designs enable Life-Code to capture complex interactions within genetic sequences, providing a more comprehensive understanding of multi-omics with the central dogma. Extensive experiments show that Life-Code achieves state-of-the-art results on various tasks across three omics, highlighting its potential for advancing multi-omics analysis and interpretation.
comment : Preprint V3 (10 pages main text)
♻ ☆ Closing the Data-Efficiency Gap Between Autoregressive and Masked Diffusion LLMs
Xu Pan, Ely Hahami, Jingxuan Fan, Ziqian Xie, Haim Sompolinsky
Despite autoregressive large language models (arLLMs) being the current dominant paradigm in language modeling, effectively updating these models to incorporate new factual knowledge still remains difficult. They resist knowledge injection via fine-tuning due to inherent shortcomings such as the "reversal curse" -- the challenge of answering questions that reverse the original information order in the training sample. Masked diffusion large language models (dLLMs) are rapidly emerging as a powerful alternative to the arLLM paradigm, with evidence of better data efficiency and free of the "reversal curse" in pre-training. However, it is unknown whether these advantages extend to the post-training phase, i.e. whether pre-trained dLLMs can easily acquire new knowledge through fine-tuning. On three diverse datasets, we fine-tune arLLMs and dLLMs, evaluating them with forward and backward style Question Answering (QA) to probe knowledge generalization and the reversal curse. Our results confirm that arLLMs critically rely on extensive data augmentation via paraphrases for QA generalization, and paraphrases are only effective when their information order matches the QA style. Conversely, dLLMs achieve high accuracies on both forward and backward QAs without paraphrases; adding paraphrases yields only marginal gains. Inspired by the dLLM's performance, we introduce a novel masked fine-tuning paradigm for knowledge injection into pre-trained arLLMs. This proposed method successfully and drastically improves the data efficiency of arLLM fine-tuning, effectively closing its performance gap with dLLMs. We further show that the masked fine-tuning paradigm of arLLMs can be extended to the supervised fine-tuning (SFT) of mathematical capability. Across two models and two datasets, our masked SFT outperforms regular SFT.
♻ ☆ EQ-Negotiator: Dynamic Emotional Personas Empower Small Language Models for Edge-Deployable Credit Negotiation
Yunbo Long, Yuhan Liu, Alexandra Brintrup
The deployment of large language models (LLMs) in automated negotiation has set a high performance benchmark, but their computational cost and data privacy requirements render them unsuitable for many privacy-sensitive, on-device applications such as mobile assistants, embodied AI agents or private client interactions. While small language models (SLMs) offer a practical alternative, they suffer from a significant performance gap compared to LLMs in playing emotionally charged complex personas, especially for credit negotiation. This paper introduces EQ-Negotiator, a novel framework that bridges this capability gap using emotional personas. Its core is a reasoning system that integrates game theory with a Hidden Markov Model(HMM) to learn and track debtor emotional states online, without pre-training. This allows EQ-Negotiator to equip SLMs with the strategic intelligence to counter manipulation while de-escalating conflict and upholding ethical standards. Through extensive agent-to-agent simulations across diverse credit negotiation scenarios, including adversarial debtor strategies like cheating, threatening, and playing the victim, we show that a 7B parameter language model with EQ-Negotiator achieves better debt recovery and negotiation efficiency than baseline LLMs more than 10 times its size. This work advances persona modeling from descriptive character profiles to dynamic emotional architectures that operate within privacy constraints. Besides, this paper establishes that strategic emotional intelligence, not raw model scale, is the critical factor for success in automated negotiation, paving the way for effective, ethical, and privacy-preserving AI negotiators that can operate on the edge.
♻ ☆ A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone NeurIPS 2025
Jitai Hao, Qiang Huang, Hao Liu, Xinyan Xiao, Zhaochun Ren, Jun Yu
Training high-performing Small Language Models (SLMs) remains costly, even with knowledge distillation and pruning from larger teacher models. Existing work often faces three key challenges: (1) information loss from hard pruning, (2) inefficient alignment of representations, and (3) underutilization of informative activations, particularly from Feed-Forward Networks (FFNs). To address these challenges, we introduce Low-Rank Clone (LRC), an efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models. LRC trains a set of low-rank projection matrices that jointly enable soft pruning by compressing teacher weights, and activation clone by aligning student activations, including FFN signals, with those of the teacher. This unified design maximizes knowledge transfer while removing the need for explicit alignment modules. Extensive experiments with open-source teachers (e.g., Llama-3.2-3B-Instruct, Qwen2.5-3B/7B-Instruct) show that LRC matches or surpasses state-of-the-art models trained on trillions of tokens--while using only 20B tokens, achieving over 1,000x training efficiency. Our codes and model checkpoints are available at https://github.com/CURRENTF/LowRankClone and https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf.
comment : NeurIPS 2025 Spotlight
♻ ☆ Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward
Yuwei Niu, Weiyang Jin, Jiaqi Liao, Chaoran Feng, Peng Jin, Bin Lin, Zongjian Li, Bin Zhu, Weihao Yu, Li Yuan
Recent years have witnessed significant progress in Unified Multimodal Models, yet a fundamental question remains: Does understanding truly inform generation? To investigate this, we introduce UniSandbox, a decoupled evaluation framework paired with controlled, synthetic datasets to avoid data leakage and enable detailed analysis. Our findings reveal a significant understanding-generation gap, which is mainly reflected in two key dimensions: reasoning generation and knowledge transfer. Specifically, for reasoning generation tasks, we observe that explicit Chain-of-Thought (CoT) in the understanding module effectively bridges the gap, and further demonstrate that a self-training approach can successfully internalize this ability, enabling implicit reasoning during generation. Additionally, for knowledge transfer tasks, we find that CoT assists the generative process by helping retrieve newly learned knowledge, and also discover that query-based architectures inherently exhibit latent CoT-like properties that affect this transfer. UniSandbox provides preliminary insights for designing future unified architectures and training strategies that truly bridge the gap between understanding and generation. Code and data are available at https://github.com/PKU-YuanGroup/UniSandBox
♻ ☆ Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts ICASSP 2024
Thomas Palmeira Ferraz, Marcely Zanon Boito, Caroline Brun, Vassilina Nikoulina
Whisper is a multitask and multilingual speech model covering 99 languages. It yields commendable automatic speech recognition (ASR) results in a subset of its covered languages, but the model still underperforms on a non-negligible number of under-represented languages, a problem exacerbated in smaller model versions. In this work, we propose DistilWhisper, an approach able to bridge the performance gap in ASR for these languages while retaining the advantages of multitask and multilingual capabilities. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2. This dual approach allows us to effectively boost ASR performance while keeping the robustness inherited from the multitask and multilingual pre-training. Results demonstrate that our approach is more effective than standard fine-tuning or LoRA adapters, boosting performance in the targeted languages for both in- and out-of-domain test sets, while introducing only a negligible parameter overhead at inference.
comment : Accepted and Presented at IEEE ICASSP 2024 (this is extended version)
♻ ☆ IoT-LLM: a framework for enhancing Large Language Model reasoning from real-wo rl d sensor data
Tuo An, Yunjiao Zhou, Han Zou, Jianfei Yang
Large Language Models excel in textual tasks but often struggle with physical-world reasoning tasks. Inspired by human cognition, where perception is fundamental to reasoning, we explore augmenting LLMs with enhanced perception abilities using Internet of Things (IoT) data and pertinent knowledge. In this work, we systematically study LLMs' capability to address IoT-sensory tasks by augmenting their perception and knowledge base, and then propose a unified framework, IoT-LLM, to enhance such capability. In IoT-LLM, we customize three steps: preprocessing IoT data into suitable formats, expanding LLMs knowledge via IoT-oriented retrieval-augmented generation and activating LLMs commonsense knowledge through chain-of-thought prompting. We design a benchmark comprising five real-world tasks with varying data types and reasoning complexities to evaluate the performance of IoT-LLM. Experimental results reveal that IoT-LLM significantly improves the performance of IoT-sensory task reasoning of LLMs, with models like GPT-4o-mini showing a 49.4% average improvement over previous methods.
comment : 33 pages, 13 figures
♻ ☆ GermanPartiesQA: Benchmark ing Commercial Large Language Models and AI Companions for Political Alignment and Sycophancy AAAI
Jan Batzner, Volker Stocker, Stefan Schmid, Gjergji Kasneci
Large language models (LLMs) are increasingly shaping citizens' information ecosystems. Products incorporating LLMs, such as chatbots and AI Companions, are now widely used for decision support and information retrieval, including in sensitive domains, raising concerns about hidden biases and growing potential to shape individual decisions and public opinion. This paper introduces GermanPartiesQA, a benchmark of 418 political statements from German Voting Advice Applications across 11 elections to evaluate six commercial LLMs. We evaluate their political alignment based on role-playing experiments with political personas. Our evaluation reveals three specific findings: (1) Factual limitations: LLMs show limited ability to accurately generate factual party positions, particularly for centrist parties. (2) Model-specific ideological alignment: We identify consistent alignment patterns and the degree of political steerability for each model across temperature settings and experiments. (3) Claim of sycophancy: While models adjust to political personas during role-play, we find this reflects persona-based steerability rather than the increasingly popular, yet contested concept of sycophancy. Our study contributes to evaluating the political alignment of closed-source LLMs that are increasingly embedded in electoral decision support tools and AI Companion chatbots.
comment : Published at AAAI/ACM AIES 2025. Presented at NeurIPS 2025 Workshop on LLM Evaluation and the International Monetary Fund's 12th Statistical Forum. GermanPartiesQA Benchmark under https://github.com/janbatzner/germanpartiesqa
♻ ☆ Robust Multimodal Sentiment Analysis with Distribution-Based Feature Recovery and Fusion ACM MM 2024
Daiqing Wu, Dongbao Yang, Yu Zhou, Can Ma
As posts on social media increase rapidly, analyzing the sentiments embedded in image-text pairs has become a popular research topic in recent years. Although existing works achieve impressive accomplishments in simultaneously harnessing image and text information, they lack the considerations of possible low-quality and missing modalities. In real-world applications, these issues might frequently occur, leading to urgent needs for models capable of predicting sentiment robustly. Therefore, we propose a Distribution-based feature Recovery and Fusion (DRF) method for robust multimodal sentiment analysis of image-text pairs. Specifically, we maintain a feature queue for each modality to approximate their feature distributions, through which we can simultaneously handle low-quality and missing modalities in a unified framework. For low-quality modalities, we reduce their contributions to the fusion by quantitatively estimating modality qualities based on the distributions. For missing modalities, we build inter-modal mapping relationships supervised by samples and distributions, thereby recovering the missing modalities from available ones. In experiments, two disruption strategies that corrupt and discard some modalities in samples are adopted to mimic the low-quality and missing modalities in various real-world scenarios. Through comprehensive experiments on three publicly available image-text datasets, we demonstrate the universal improvements of DRF compared to SOTA methods under both two strategies, validating its effectiveness in robust multimodal sentiment analysis.
comment : Accepted by ACM MM 2024
♻ ☆ Extending Multilingual Machine Translation through Imitation Learning
Wen Lai, Viktor Hangya, Yingli Shen, Alexander Fraser
Despite the growing variety of languages supported by existing multilingual neural machine translation (MNMT) models, most of the world's languages are still being left behind. We aim to extend large-scale MNMT models to incorporate a new language, enabling translations between this new language and all previously supported languages, even in the challenging scenario where only a parallel corpus between the new language and English is available. Previous methods, such as continued training on parallel data including the new language, often suffer from catastrophic forgetting, which degrades performance on other languages. We propose a novel approach Imit-MNMT which treats this task as an imitation learning problem, a technique widely used in computer vision but less explored in natural language processing. Specifically, we leverage an expert model to generate pseudo-parallel corpora between the new language and the existing languages. We then introduce a data distribution imitation strategy using language-specific weighting, alongside a translation behavior imitation mechanism. Extensive experiments show that our approach significantly improves translation performance between the new and existing languages while mitigating catastrophic forgetting.
♻ ☆ Enhancing OCR for Sino-Vietnamese Language Processing via Fine-tuned Paddle OCR v5
Minh Hoang Nguyen, Su Nguyen Thiet
Recognizing and processing Classical Chinese (Han-Nom) texts play a vital role in digitizing Vietnamese historical documents and enabling cross-lingual semantic research. However, existing OCR systems struggle with degraded scans, non-standard glyphs, and handwriting variations common in ancient sources. In this work, we propose a fine-tuning approach for PaddleOCRv5 to improve character recognition on Han-Nom texts. We retrain the text recognition module using a curated subset of ancient Vietnamese Chinese manuscripts, supported by a full training pipeline covering preprocessing, LMDB conversion, evaluation, and visualization. Experimental results show a significant improvement over the base model, with exact accuracy increasing from 37.5 percent to 50.0 percent, particularly under noisy image conditions. Furthermore, we develop an interactive demo that visually compares pre- and post-fine-tuning recognition results, facilitating downstream applications such as Han-Vietnamese semantic alignment, machine translation, and historical linguistics research. The demo is available at https://huggingface.co/spaces/MinhDS/Fine-tuned-PaddleOCRv5
comment : Short Paper: 7 pages, 8 figures, 3 tables
♻ ☆ Teaching Language Models to Critique via Reinforcement Learning
Zhihui Xie, Jie Chen, Liyu Chen, Weichao Mao, Jingjing Xu, Lingpeng Kong
Teaching large language models (LLMs) to critique and refine their outputs is crucial for building systems that can iteratively improve, yet it is fundamentally limited by the ability to provide accurate judgments and actionable suggestions. In this work, we study LLM critics for code generation and propose $\texttt{CTRL}$, a framework for $\texttt{C}$ritic $\texttt{T}$raining via $\texttt{R}$einforcement $\texttt{L}$earning, which trains a critic model to generate feedback that maximizes correction performance for a fixed generator model without human supervision. Our results demonstrate that critics trained with $\texttt{CTRL}$ significantly enhance pass rates and mitigate compounding errors across both base and stronger generator models. Furthermore, we show that these critic models act as accurate generative reward models and enable test-time scaling through iterative critique-revision, achieving up to 106.1% relative improvements across challenging code generation benchmarks.
♻ ☆ Prompt -OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision -Language Model Adaptation WACV 2026
Xiwen Chen, Wenhui Zhu, Peijie Qiu, Hao Wang, Huayu Li, Haiyu Wu, Aristeidis Sotiras, Yalin Wang, Abolfazl Razi
Vision-language models (VLMs) such as CLIP demonstrate strong performance but struggle when adapted to downstream tasks. Prompt learning has emerged as an efficient and effective strategy to adapt VLMs while preserving their pre-trained knowledge. However, existing methods still lead to overfitting and degrade zero-shot generalization. To address this challenge, we propose an optimal transport (OT)-guided prompt learning framework that mitigates forgetting by preserving the structural consistency of feature distributions between pre-trained and fine-tuned models. Unlike conventional point-wise constraints, OT naturally captures cross-instance relationships and expands the feasible parameter space for prompt tuning, allowing a better trade-off between adaptation and generalization. Our approach enforces joint constraints on both vision and text representations, ensuring a holistic feature alignment. Extensive experiments on benchmark datasets demonstrate that our simple yet effective method can outperform existing prompt learning strategies in base-to-novel generalization, cross-dataset evaluation, and domain generalization without additional augmentation or ensemble techniques. The code is available at https://github.com/ChongQingNoSubway/Prompt-OT
comment : Accepted to WACV 2026
♻ ☆ DND: Boosting Large Language Models with Dynamic Nested Depth
Tieyuan Chen, Xiaodong Chen, Haoxing Chen, Zhenzhong Lan, Weiyao Lin, Jianguo Li
We introduce Dynamic Nested Depth (DND), a novel method that improves performance for off-the-shelf LLMs by selecting critical tokens to reprocess in a nested depth manner. Specifically, at the end of the given transformer layer, DND identifies more critical tokens with a router and feeds them back for an extra round of processing, effectively ``reviewing" difficult tokens while avoiding redundant computation for easier ones. The dynamic selection mechanism is tailored for precise control via two novel strategies: a router controlling loss to enhance token selection distinguishability, and a threshold control scheme to ensure selection stability. We demonstrate the effectiveness of DND by directly integrating it into pre-trained dense and MoE models during a post-training phase. On diverse benchmarks, this approach boosts the performances of the dense Qwen3-1.7B by 1.88% and the MoE Qwen3-30B-A3B by 0.87%, all with a minimal parameter and computing increase.
comment : TL;DR: We introduce Dynamic Nested Depth (DND), an efficient paradigm that adaptively identifies critical tokens and selectively deepens their computation via nested re-processing
♻ ☆ A Semantic-based Optimization Approach for Repairing LLMs: Case Study on Code Generation
Jian Gu, Aldeida Aleti, Chunyang Chen, Hongyu Zhang
Language Models (LMs) are widely used in software engineering for code generation, but they may produce erroneous code. Rather than repairing outputs, a more thorough remedy is to address underlying model failures. LM repair offers a lightweight solution: it requires minimal data, lowers computational cost, and limits side effects. Unlike full retraining, LM repair focuses on applying tailored updates to targeted neurons, making it suitable for limited resources, high-performance demands, or strict safety requirements. In this paper, we propose Semantic Targeting for Analytical Repair (STAR), a novel semantic-based optimization method for repairing LLMs. STAR realizes the main operations of repairing LMs in an optimization process, including locating ``buggy neurons'', solving ``neuron patches'', and patching ``buggy neurons''. The neuron patches are computed with a solid semantic-based analytical formula, which directly bridges the changes to logits with the deltas of neurons, by steering latent representations. Compared to the prior work of LM repair (MINT) and standard optimization methods (SGD), STAR integrates their strengths while mitigating their limitations. By reformulating LM repair as an optimization process, STAR may solve multiple failures together, significantly improving the usefulness. Evaluated on coding tasks using popular code LMs, STAR demonstrates superior effectiveness compared with the state-of-the-art. Besides, STAR exhibits better efficiency. In terms of side effects, namely the balance between generalization and specificity, STAR outperforms prior work by a significant margin. Additionally, we conducted assessments on the overfitting risk of LM repair as well as the cumulative impact. Further, we analyzed the differences with pipeline-based methods and explained the reason why STAR is better and how it mitigated the common limitations of LM repair.
comment : 13 pages, 6 figure, 8 tables, camera-ready version
♻ ☆ Beyond Token Length: Step Pruner for Efficient and Accurate Reasoning in Large Language Models
Canhui Wu, Qiong Cao, Chang Li, Zhenfang Wang, Chao Xue, Yuwei Fan, Wei Xi, Xiaodong He
Large Reasoning Models (LRMs) demonstrate strong performance on complex tasks but often suffer from excessive verbosity, known as "overthinking." Existing solutions via reinforcement learning (RL) typically penalize generated tokens to promote conciseness. However, these methods encounter two challenges: responses with fewer tokens do not always correspond to fewer reasoning steps, and models may develop hacking behavior in later stages of training by discarding reasoning steps to minimize token usage. In this work, we introduce \textbf{Step Pruner (SP)}, an RL framework that steers LRMs toward more efficient reasoning by favoring compact reasoning steps. Our step-aware reward function prioritizes correctness while imposing penalties for redundant steps, and withholds rewards for incorrect responses to prevent the reinforcement of erroneous reasoning. Moreover, we propose a dynamic stopping mechanism: when the model's output no longer shortens, training is halted to prevent hacking behavior caused by the merging of steps. Extensive experiments across four reasoning benchmarks demonstrate that SP achieves state-of-the-art accuracy while significantly reducing response length. For instance, on AIME24, SP reduces token usage by \textbf{69.7\%}.
comment : 21 pages, 9 figures
♻ ☆ Predicting the Performance of Black-box LLMs through Follow-up Queries NeurIPS 2025
Dylan Sam, Marc Finzi, J. Zico Kolter
Reliably predicting the behavior of language models -- such as whether their outputs are correct or have been adversarially manipulated -- is a fundamentally challenging task. This is often made even more difficult as frontier language models are offered only through closed-source APIs, providing only black-box access. In this paper, we predict the behavior of black-box language models by asking follow-up questions and taking the probabilities of responses \emph{as} representations to train reliable predictors. We first demonstrate that training a linear model on these responses reliably and accurately predicts model correctness on question-answering and reasoning benchmarks. Surprisingly, this can \textit{even outperform white-box linear predictors} that operate over model internals or activations. Furthermore, we demonstrate that these follow-up question responses can reliably distinguish between a clean version of an LLM and one that has been adversarially influenced via a system prompt to answer questions incorrectly or to introduce bugs into generated code. Finally, we show that they can also be used to differentiate between black-box LLMs, enabling the detection of misrepresented models provided through an API. Overall, our work shows promise in monitoring black-box language model behavior, supporting their deployment in larger, autonomous systems.
comment : NeurIPS 2025
Artificial Intelligence 148
☆ Thinking by Doing: Building Efficient Wo rl d Model Reasoning in LLMs via Multi-turn Interaction
Bao Shu, Yan Cai, Jianjian Sun, Chunrui Han, En Yu, Liang Zhao, Jingcheng Hu, Yinmin Zhang, Haoran Lv, Yuang Peng, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Xiangyu Yue
Developing robust world model reasoning is crucial for large language model (LLM) agents to plan and interact in complex environments. While multi-turn interaction offers a superior understanding of environmental dynamics via authentic feedback, current approaches often impose a rigid reasoning process, which constrains the model's active learning, ultimately hindering efficient world model reasoning. To address these issues, we explore world-model internalization through efficient interaction and active reasoning (WMAct), which liberates the model from structured reasoning, allowing the model to shape thinking directly through its doing, and achieves effective and efficient world model reasoning with two key mechanisms: (1) a reward rescaling mechanism adjusting outcome reward based on action efficacy to incentivize redundancy reduction and purposeful interaction; (2) an interaction frequency annealing strategy to progressively reduce the maximum allowed interaction turns, which compels the model to condense its learning and internalize environmental dynamics rather than over-relying on environmental cues. Our experiments on Sokoban, Maze, and Taxi show that WMAct yields effective world model reasoning capable of resolving tasks in a single turn that previously required multiple interactions and fosters strong transferability to complex environments, improving performance on a suite of reasoning benchmarks.
comment : 17 pages, 9 figures
☆ The Price of Progress: Algorithmic Efficiency and the Falling Cost of AI Inference
Hans Gundlach, Jayson Lynch, Matthias Mertens, Neil Thompson
Language models have seen enormous progress on advanced benchmarks in recent years, but much of this progress has only been possible by using more costly models. Benchmarks may therefore present a warped picture of progress in practical capabilities per dollar. To remedy this, we use data from Artificial Analysis and Epoch AI to form the largest dataset of current and historical prices to run benchmarks to date. We find that the price for a given level of benchmark performance has decreased remarkably fast, around $5\times$ to $10\times$ per year, for frontier models on knowledge, reasoning, math, and software engineering benchmarks. These reductions in the cost of AI inference are due to economic forces, hardware efficiency improvements, and algorithmic efficiency improvements. Isolating out open models to control for competition effects and dividing by hardware price declines, we estimate that algorithmic efficiency progress is around $3\times$ per year. Finally, we recommend that evaluators both publicize and take into account the price of benchmarking as an essential part of measuring the real-world impact of AI.
☆ Physics-Informed Neural Networks for Thermophysical Property Retrieval
Ali Waseem, Malcolm Mielle
Inverse heat problems refer to the estimation of material thermophysical properties given observed or known heat diffusion behaviour. Inverse heat problems have wide-ranging uses, but a critical application lies in quantifying how building facade renovation reduces thermal transmittance, a key determinant of building energy efficiency. However, solving inverse heat problems with non-invasive data collected in situ is error-prone due to environmental variability or deviations from theoretically assumed conditions. Hence, current methods for measuring thermal conductivity are either invasive, require lengthy observation periods, or are sensitive to environmental and experimental conditions. Here, we present a PINN-based iterative framework to estimate the thermal conductivity k of a wall from a set of thermographs; our framework alternates between estimating the forward heat problem with a PINN for a fixed k, and optimizing k by comparing the thermographs and surface temperatures predicted by the PINN, repeating until the estimated k's convergence. Using both environmental data captured by a weather station and data generated from Finite-Volume-Method software simulations, we accurately predict k across different environmental conditions and data collection sampling times, given the temperature profile of the wall at dawn is close to steady state. Although violating the steady-state assumption impacts the accuracy of k's estimation, we show that our proposed framework still only exhibits a maximum MAE of 4.0851. Our work demonstrates the potential of PINN-based methods for reliable estimation of material properties in situ and under realistic conditions, without lengthy measurement campaigns. Given the lack of research on using machine learning, and more specifically on PINNs, for solving in-situ inverse problems, we expect our work to be a starting point for more research on the topic.
comment : 26 pages, 4 figures, 3 tables
☆ ASTRO: Adaptive Stitching via Dynamics-Guided Trajectory Rollouts
Hang Yu, Di Zhang, Qiwei Du, Yanping Zhao, Hai Zhang, Guang Chen, Eduardo E. Veas, Junqiao Zhao
Offline reinforcement learning (RL) enables agents to learn optimal policies from pre-collected datasets. However, datasets containing suboptimal and fragmented trajectories present challenges for reward propagation, resulting in inaccurate value estimation and degraded policy performance. While trajectory stitching via generative models offers a promising solution, existing augmentation methods frequently produce trajectories that are either confined to the support of the behavior policy or violate the underlying dynamics, thereby limiting their effectiveness for policy improvement. We propose ASTRO, a data augmentation framework that generates distributionally novel and dynamics-consistent trajectories for offline RL. ASTRO first learns a temporal-distance representation to identify distinct and reachable stitch targets. We then employ a dynamics-guided stitch planner that adaptively generates connecting action sequences via Rollout Deviation Feedback, defined as the gap between target state sequence and the actual arrived state sequence by executing predicted actions, to improve trajectory stitching's feasibility and reachability. This approach facilitates effective augmentation through stitching and ultimately enhances policy learning. ASTRO outperforms prior offline RL augmentation methods across various algorithms, achieving notable performance gain on the challenging OGBench suite and demonstrating consistent improvements on standard offline RL benchmarks such as D4RL.
☆ Towards Continuous Intelligence Growth: Self-Training, Continual Learning, and Dual-Scale Memory in SuperIntelli Agent
Jianzhe Lin, Zeyu Pan, Yun Zhu, Ruiqi Song, Jining Yang
We introduce SuperIntelliAgent, an agentic learning framework that couples a trainable small diffusion model (the learner) with a frozen large language model (the verifier) to enable continual intelligence growth through self-supervised interaction. Unlike conventional supervised fine-tuning, SuperIntelliAgent learns autonomously without annotation: the learner generates candidate outputs, the verifier evaluates them through step-by-step reasoning, and their interaction produces chosen/rejected pairs for Direct Preference Optimization (DPO). This converts each input into a pseudo-training signal for continual improvement. The framework integrates dual-scale memory: short-term in-context memory that preserves reasoning traces across refinement cycles, and long-term memory that consolidates acquired knowledge through lightweight on-the-fly fine-tuning. A replay buffer retains samples that show verifiable progress and replays them as auxiliary supervision, reinforcing recent learning while forming adaptive curricula. SuperIntelliAgent is infrastructure-agnostic and can be plugged into existing agentic frameworks while turning ordinary inference loops into a lifelong optimization process. We posit that pairing a trainable learner with a reasoning-capable verifier forms a minimal reliable unit of growing intelligence, as paired feedback and partial-history replay yield richer learning curricula and stronger preference alignment. With a small number of automatically generated DPO pairs, the learner improves across all benchmarks, indicating that this mechanism provides a promising direction for continual intelligence accumulation and real-world deployment.
comment : 15 pages, 4 figures
☆ Evaluating LLMs for One-Shot Patching of Real and Artificial Vulnerabilities
Aayush Garg, Zanis Ali Khan, Renzo Degiovanni, Qiang Tang
Automated vulnerability patching is crucial for software security, and recent advancements in Large Language Models (LLMs) present promising capabilities for automating this task. However, existing research has primarily assessed LLMs using publicly disclosed vulnerabilities, leaving their effectiveness on related artificial vulnerabilities largely unexplored. In this study, we empirically evaluate the patching effectiveness and complementarity of several prominent LLMs, such as OpenAI's GPT variants, LLaMA, DeepSeek, and Mistral models, using both real and artificial vulnerabilities. Our evaluation employs Proof-of-Vulnerability (PoV) test execution to concretely assess whether LLM-generated source code successfully patches vulnerabilities. Our results reveal that LLMs patch real vulnerabilities more effectively compared to artificial ones. Additionally, our analysis reveals significant variability across LLMs in terms of overlapping (multiple LLMs patching the same vulnerabilities) and complementarity (vulnerabilities patched exclusively by a single LLM), emphasizing the importance of selecting appropriate LLMs for effective vulnerability patching.
comment : Pre-print - Extended version of the poster paper accepted at the 41st ACM/SIGAPP Symposium on Applied Computing (SAC) Smarter Engineering-Building AI and Building with AI (SEAI) 2026
☆ LFM2 Technical Report
Alexander Amini, Anna Banaszak, Harold Benoit, Arthur Böök, Tarek Dakhran, Song Duong, Alfred Eng, Fernando Fernandes, Marc Härkönen, Anne Harrington, Ramin Hasani, Saniya Karwa, Yuri Khrustalev, Maxime Labonne, Mathias Lechner, Valentine Lechner, Simon Lee, Zetian Li, Noel Loo, Jacob Marks, Edoardo Mosca, Samuel J. Paech, Paul Pak, Rom N. Parnichkun, Alex Quach, Ryan Rogers, Daniela Rus, Nayan Saxena, Bettina Schlager, Tim Seyde, Jimmy T. H. Smith, Aditya Tadimeti, Neehal Tumma
We present LFM2, a family of Liquid Foundation Models designed for efficient on-device deployment and strong task capabilities. Using hardware-in-the-loop architecture search under edge latency and memory constraints, we obtain a compact hybrid backbone that combines gated short convolutions with a small number of grouped query attention blocks, delivering up to 2x faster prefill and decode on CPUs compared to similarly sized models. The LFM2 family covers 350M-8.3B parameters, including dense models (350M, 700M, 1.2B, 2.6B) and a mixture-of-experts variant (8.3B total, 1.5B active), all with 32K context length. LFM2's training pipeline includes a tempered, decoupled Top-K knowledge distillation objective that avoids support mismatch; curriculum learning with difficulty-ordered data; and a three-stage post-training recipe of supervised fine-tuning, length-normalized preference optimization, and model merging. Pre-trained on 10-12T tokens, LFM2 models achieve strong results across diverse benchmarks; for example, LFM2-2.6B reaches 79.56% on IFEval and 82.41% on GSM8K. We further build multimodal and retrieval variants: LFM2-VL for vision-language tasks, LFM2-Audio for speech, and LFM2-ColBERT for retrieval. LFM2-VL supports tunable accuracy-latency tradeoffs via token-efficient visual processing, while LFM2-Audio separates audio input and output pathways to enable real-time speech-to-speech interaction competitive with models 3x larger. LFM2-ColBERT provides a low-latency encoder for queries and documents, enabling high-performance retrieval across multiple languages. All models are released with open weights and deployment packages for ExecuTorch, llama.cpp, and vLLM, making LFM2 a practical base for edge applications that need fast, memory-efficient inference and strong task capabilities.
☆ MegaChat: A Synthetic Persian Q&A Dataset for High-Quality Sales Chatbot Evaluation
Mahdi Rahmani, AmirHossein Saffari, Reyhane Rahmani
Small and medium-sized enterprises (SMEs) in Iran increasingly leverage Telegram for sales, where real-time engagement is essential for conversion. However, developing AI-driven chatbots for this purpose requires large, high-quality question-and-answer (Q&A) datasets, which are typically expensive and resource-intensive to produce, especially for low-resource languages like Persian. In this paper, we introduce MegaChat, the first fully synthetic Persian Q&A dataset designed to evaluate intelligent sales chatbots in Telegram-based e-commerce. We propose a novel, automated multi-agent architecture that generates persona-aware Q&A pairs by collecting data from active Telegram shopping channels. The system employs specialized agents for question generation, validation, and refinement, ensuring the production of realistic and diverse conversational data. To evaluate answer generation, we compare three classic retrieval-augmented generation (RAG) models with our advanced agentic system, which features multi-query retrieval, reranking, and persona-aligned response synthesis. Using GPT-5.1 for evaluation across six quality dimensions, our results show that the agentic architecture outperformed traditional RAG models in 4 out of 5 diverse channels, demonstrating its ability to generate scalable, high-quality datasets without relying on expensive human annotation or complex fine-tuning. MegaChat provides SMEs with an efficient, cost-effective solution for building intelligent customer engagement systems in specialized commercial domains, enabling advancements in multilingual conversational AI for low-resource languages. Download: https://github.com/MegaChat-Tech/MegaChat-DataSet
comment : 6 pages, 11 figures, 2 tables
☆ Hierarchical AI-Meteorologist: LLM- Agent System for Multi-Scale and Explainable Weather Forecast Reporting
Daniil Sukhorukov, Andrei Zakharov, Nikita Glazkov, Katsiaryna Yanchanka, Vladimir Kirilin, Maxim Dubovitsky, Roman Sultimov, Yuri Maksimov, Ilya Makarov
We present the Hierarchical AI-Meteorologist, an LLM-agent system that generates explainable weather reports using a hierarchical forecast reasoning and weather keyword generation. Unlike standard approaches that treat forecasts as flat time series, our framework performs multi-scale reasoning across hourly, 6-hour, and daily aggregations to capture both short-term dynamics and long-term trends. Its core reasoning agent converts structured meteorological inputs into coherent narratives while simultaneously extracting a few keywords effectively summarizing the dominant meteorological events. These keywords serve as semantic anchors for validating consistency, temporal coherence and factual alignment of the generated reports. Using OpenWeather and Meteostat data, we demonstrate that hierarchical context and keyword-based validation substantially improve interpretability and robustness of LLM-generated weather narratives, offering a reproducible framework for semantic evaluation of automated meteorological reporting and advancing agent-based scientific reasoning.
comment : 9 pages, 4 figures
☆ Agent ic AI Framework for Smart Inventory Replenishment
Toqeer Ali Syed, Salman Jan, Gohar Ali, Ali Akarma, Ahmad Ali, Qurat-ul-Ain Mastoi
In contemporary retail, the variety of products available (e.g. clothing, groceries, cosmetics, frozen goods) make it difficult to predict the demand, prevent stockouts, and find high-potential products. We suggest an agentic AI model that will be used to monitor the inventory, initiate purchase attempts to the appropriate suppliers, and scan for trending or high-margin products to incorporate. The system applies demand forecasting, supplier selection optimization, multi-agent negotiation and continuous learning. We apply a prototype to a setting in the store of a middle scale mart, test its performance on three conventional and artificial data tables, and compare the results to the base heuristics. Our findings indicate that there is a decrease in stockouts, a reduction of inventory holding costs, and an improvement in product mix turnover. We address constraints, scalability as well as improvement prospect.
comment : Presented at International Conference on Business and Digital Technology, Bahrain, Springer Nature, 27 November 2025
☆ Flow Straighter and Faster: Efficient One-Step Generative Modeling via MeanFlow on Rectified Trajectories
Xinxi Zhang, Shiwei Tan, Quang Nguyen, Quan Dao, Ligong Han, Xiaoxiao He, Tunyu Zhang, Alen Mrdovic, Dimitris Metaxas
Flow-based generative models have recently demonstrated strong performance, yet sampling typically relies on expensive numerical integration of ordinary differential equations (ODEs). Rectified Flow enables one-step sampling by learning nearly straight probability paths, but achieving such straightness requires multiple computationally intensive reflow iterations. MeanFlow achieves one-step generation by directly modeling the average velocity over time; however, when trained on highly curved flows, it suffers from slow convergence and noisy supervision. To address these limitations, we propose Rectified MeanFlow, a framework that models the mean velocity field along the rectified trajectory using only a single reflow step. This eliminates the need for perfectly straightened trajectories while enabling efficient training. Furthermore, we introduce a simple yet effective truncation heuristic that aims to reduce residual curvature and further improve performance. Extensive experiments on ImageNet at 64, 256, and 512 resolutions show that Re-MeanFlow consistently outperforms prior one-step flow distillation and Rectified Flow methods in both sample quality and training efficiency. Code is available at https://github.com/Xinxi-Zhang/Re-MeanFlow.
☆ ParaGate: Parasitic-Driven Domain Adaptation Transfer Learning for Netlist Performance Prediction
Bin Sun, Jingyi Zhou, Jianan Mu, Zhiteng Chao, Tianmeng Yang, Ziyue Xu, Jing Ye, Huawei Li
In traditional EDA flows, layout-level performance metrics are only obtainable after placement and routing, hindering global optimization at earlier stages. Although some neural-network-based solutions predict layout-level performance directly from netlists, they often face generalization challenges due to the black-box heuristics of commercial placement-and-routing tools, which create disparate data across designs. To this end, we propose ParaGate, a three-step cross-stage prediction framework that infers layout-level timing and power from netlists. First, we propose a two-phase transfer-learning approach to predict parasitic parameters, pre-training on mid-scale circuits and fine-tuning on larger ones to capture extreme conditions. Next, we rely on EDA tools for timing analysis, offloading the long-path numerical reasoning. Finally, ParaGate performs global calibration using subgraph features. Experiments show that ParaGate achieves strong generalization with minimal fine-tuning data: on openE906, its arrival-time R2 from 0.119 to 0.897. These results demonstrate that ParaGate could provide guidance for global optimization in the synthesis and placement stages.
comment : 8 pages, 6 figures
☆ Towards Improving Interpretability of Language Model Generation through a Structured Knowledge Discovery Approach
Shuqi Liu, Han Wu, Guanzhi Deng, Jianshu Chen, Xiaoyang Wang, Linqi Song
Knowledge-enhanced text generation aims to enhance the quality of generated text by utilizing internal or external knowledge sources. While language models have demonstrated impressive capabilities in generating coherent and fluent text, the lack of interpretability presents a substantial obstacle. The limited interpretability of generated text significantly impacts its practical usability, particularly in knowledge-enhanced text generation tasks that necessitate reliability and explainability. Existing methods often employ domain-specific knowledge retrievers that are tailored to specific data characteristics, limiting their generalizability to diverse data types and tasks. To overcome this limitation, we directly leverage the two-tier architecture of structured knowledge, consisting of high-level entities and low-level knowledge triples, to design our task-agnostic structured knowledge hunter. Specifically, we employ a local-global interaction scheme for structured knowledge representation learning and a hierarchical transformer-based pointer network as the backbone for selecting relevant knowledge triples and entities. By combining the strong generative ability of language models with the high faithfulness of the knowledge hunter, our model achieves high interpretability, enabling users to comprehend the model output generation process. Furthermore, we empirically demonstrate the effectiveness of our model in both internal knowledge-enhanced table-to-text generation on the RotoWireFG dataset and external knowledge-enhanced dialogue response generation on the KdConv dataset. Our task-agnostic model outperforms state-of-the-art methods and corresponding language models, setting new standards on the benchmark.
☆ Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
Xiang Hu, Zhanchao Zhou, Ruiqi Liang, Zehuan Li, Wei Wu, Jianguo Li
This work explores the challenge of building ``Machines that Can Remember'', framing long-term memory as the problem of efficient ultra-long context modeling. We argue that this requires three key properties: \textbf{sparsity}, \textbf{random-access flexibility}, and \textbf{length generalization}. To address ultra-long-context modeling, we leverage Hierarchical Sparse Attention (HSA), a novel attention mechanism that satisfies all three properties. We integrate HSA into Transformers to build HSA-UltraLong, which is an 8B-parameter MoE model trained on over 8 trillion tokens and is rigorously evaluated on different tasks with in-domain and out-of-domain context lengths to demonstrate its capability in handling ultra-long contexts. Results show that our model performs comparably to full-attention baselines on in-domain lengths while achieving over 90\% accuracy on most in-context retrieval tasks with contexts up to 16M. This report outlines our experimental insights and open problems, contributing a foundation for future research in ultra-long context modeling.
☆ Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach
Haruki Sakajo, Hiroshi Takato, Hiroshi Tsutsui, Komei Soda, Hidetaka Kamigaito, Taro Watanabe
Large-scale Vision Language Models (LVLMs) exhibit advanced capabilities in tasks that require visual information, including object detection. These capabilities have promising applications in various industrial domains, such as autonomous driving. For example, LVLMs can generate safety-oriented descriptions of videos captured by road-facing cameras. However, ensuring comprehensive safety requires monitoring driver-facing views as well to detect risky events, such as the use of mobiles while driving. Thus, the ability to process synchronized inputs is necessary from both driver-facing and road-facing cameras. In this study, we develop models and investigate the capabilities of LVLMs by constructing a dataset and evaluating their performance on this dataset. Our experimental results demonstrate that while pre-trained LVLMs have limited effectiveness, fine-tuned LVLMs can generate accurate and safety-aware driving instructions. Nonetheless, several challenges remain, particularly in detecting subtle or complex events in the video. Our findings and error analysis provide valuable insights that can contribute to the improvement of LVLM-based systems in this domain.
comment : Accepted to MMLoSo 2025
☆ Hard-Constrained Neural Networks with Physics-Embedded Architecture for Residual Dynamics Learning and Invariant Enforcement in Cyber-Physical Systems
Enzo Nicolás Spotorno, Josafat Leal Filho, Antônio Augusto Fröhlich
This paper presents a framework for physics-informed learning in complex cyber-physical systems governed by differential equations with both unknown dynamics and algebraic invariants. First, we formalize the Hybrid Recurrent Physics-Informed Neural Network (HRPINN), a general-purpose architecture that embeds known physics as a hard structural constraint within a recurrent integrator to learn only residual dynamics. Second, we introduce the Projected HRPINN (PHRPINN), a novel extension that integrates a predict-project mechanism to strictly enforce algebraic invariants by design. The framework is supported by a theoretical analysis of its representational capacity. We validate HRPINN on a real-world battery prognostics DAE and evaluate PHRPINN on a suite of standard constrained benchmarks. The results demonstrate the framework's potential for achieving high accuracy and data efficiency, while also highlighting critical trade-offs between physical consistency, computational cost, and numerical stability, providing practical guidance for its deployment.
comment : 41 pages (30 pages main text + 11 pages appendices), 3 figures, 8 tables. Submitted to JMLR
☆ Multi-Modal Scene Graph with Kolmogorov-Arnold Experts for Audio-Visual Question Answering
Zijian Fu, Changsheng Lv, Mengshi Qi, Huadong Ma
In this paper, we propose a novel Multi-Modal Scene Graph with Kolmogorov-Arnold Expert Network for Audio-Visual Question Answering (SHRIKE). The task aims to mimic human reasoning by extracting and fusing information from audio-visual scenes, with the main challenge being the identification of question-relevant cues from the complex audio-visual content. Existing methods fail to capture the structural information within video, and suffer from insufficient fine-grained modeling of multi-modal features. To address these issues, we are the first to introduce a new multi-modal scene graph that explicitly models the objects and their relationship as a visually grounded, structured representation of the audio-visual scene. Furthermore, we design a Kolmogorov-Arnold Network~(KAN)-based Mixture of Experts (MoE) to enhance the expressive power of the temporal integration stage. This enables more fine-grained modeling of cross-modal interactions within the question-aware fused audio-visual representation, leading to capture richer and more nuanced patterns and then improve temporal reasoning performance. We evaluate the model on the established MUSIC-AVQA and MUSIC-AVQA v2 benchmarks, where it achieves state-of-the-art performance. Code and model checkpoints will be publicly released.
☆ Machine Learning for Scientific Visualization: Ensemble Data Analysis
Hamid Gadirov
Scientific simulations and experimental measurements produce vast amounts of spatio-temporal data, yet extracting meaningful insights remains challenging due to high dimensionality, complex structures, and missing information. Traditional analysis methods often struggle with these issues, motivating the need for more robust, data-driven approaches. This dissertation explores deep learning methodologies to improve the analysis and visualization of spatio-temporal scientific ensembles, focusing on dimensionality reduction, flow estimation, and temporal interpolation. First, we address high-dimensional data representation through autoencoder-based dimensionality reduction for scientific ensembles. We evaluate the stability of projection metrics under partial labeling and introduce a Pareto-efficient selection strategy to identify optimal autoencoder variants, ensuring expressive and reliable low-dimensional embeddings. Next, we present FLINT, a deep learning model for high-quality flow estimation and temporal interpolation in both flow-supervised and flow-unsupervised settings. FLINT reconstructs missing velocity fields and generates high-fidelity temporal interpolants for scalar fields across 2D+time and 3D+time ensembles without domain-specific assumptions or extensive finetuning. To further improve adaptability and generalization, we introduce HyperFLINT, a hypernetwork-based approach that conditions on simulation parameters to estimate flow fields and interpolate scalar data. This parameter-aware adaptation yields more accurate reconstructions across diverse scientific domains, even with sparse or incomplete data. Overall, this dissertation advances deep learning techniques for scientific visualization, providing scalable, adaptable, and high-quality solutions for interpreting complex spatio-temporal ensembles.
comment : PhD thesis, University of Groningen, 2025
☆ Simultaneous Image Quality Improvement and Artefacts Correction in Accelerated MRI
Georgia Kanli, Daniele Perlo, Selma Boudissa, Radovan Jirik, Olivier Keunen
MR data are acquired in the frequency domain, known as k-space. Acquiring high-quality and high-resolution MR images can be time-consuming, posing a significant challenge when multiple sequences providing complementary contrast information are needed or when the patient is unable to remain in the scanner for an extended period of time. Reducing k-space measurements is a strategy to speed up acquisition, but often leads to reduced quality in reconstructed images. Additionally, in real-world MRI, both under-sampled and full-sampled images are prone to artefacts, and correcting these artefacts is crucial for maintaining diagnostic accuracy. Deep learning methods have been proposed to restore image quality from under-sampled data, while others focused on the correction of artefacts that result from the noise or motion. No approach has however been proposed so far that addresses both acceleration and artefacts correction, limiting the performance of these models when these degradation factors occur simultaneously. To address this gap, we present a method for recovering high-quality images from under-sampled data with simultaneously correction for noise and motion artefact called USArt (Under-Sampling and Artifact correction model). Customized for 2D brain anatomical images acquired with Cartesian sampling, USArt employs a dual sub-model approach. The results demonstrate remarkable increase of signal-to-noise ratio (SNR) and contrast in the images restored. Various under-sampling strategies and degradation levels were explored, with the gradient under-sampling strategy yielding the best outcomes. We achieved up to 5x acceleration and simultaneously artefacts correction without significant degradation, showcasing the model's robustness in real-world settings.
☆ OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning
Timothy Ossowski, Sheng Zhang, Qianchu Liu, Guanghui Qin, Reuben Tan, Tristan Naumann, Junjie Hu, Hoifung Poon
High-quality and carefully curated data is a cornerstone of training medical large language models, as it directly impacts both generalization and robustness to unseen clinical tasks. We investigate strategies for training and data curation to develop a robust multimodal reasoning model in the medical domain. Our work focuses on supervised fine-tuning (SFT) and explores data recipes that leverage structured reasoning traces. Using our proposed data recipe, we scale experiments to a dataset of over 8 million examples and 6.8 billion response tokens, achieving state-of-the-art performance among open-source models across diverse out-of-distribution medical benchmark tasks. Our results further indicate that curating a high-quality, diverse training dataset with varying structured reasoning trace lengths enables the fine-tuned model to self-calibrate its reasoning trajectory lengths based on the downstream task, without explicit supervision. We present key insights, describe the data curation strategy, and outline next steps toward developing robust medical vision-language reasoning system.
☆ Adapting Like Humans: A Metacognitive Agent with Test-time Reasoning
Yang Li, Zhiyuan He, Yuxuan Huang, Zhuhanling Xiao, Chao Yu, Meng Fang, Kun Shao, Jun Wang
Recent Vision-Language Models (VLMs) exhibit strong perceptual reasoning abilities, yet they often struggle to adapt efficiently when encountering novel tasks at test time. In contrast, humans leverage the metacognitive model with memory, enabling continuous strategy refinement through metacognitive control when faced with new challenges. To bridge this gap, we propose metacognitive test-time reasoning (MCTR), a framework that equips models with the ability to learn, adapt, and improve during test time through metacognitive self-updating. Inspired by the dual structure of human metacognition, MCTR comprises meta-level and object-level VLM reasoning modules, each equipped with dedicated memory systems for hierarchical adaptive reasoning. Specifically, MCTR consists of (1) a meta-reasoning module which incrementally builds a structured memory by discovering and storing task-relevant rules, environmental patterns, and action-outcome relationships from test-time observations as natural language descriptions; and (2) an action-reasoning module that determines optimal actions through context-aware perception and strategic reasoning by dynamically retrieving and integrating knowledge from memory. The action-reasoning module continuously updates its policy through proposed metacognitive test-time reinforcement learning, adapting as knowledge memory evolves. We evaluate MCTR on 45 Atari games (33 seen, 12 unseen). MCTR demonstrates robust test-time adaptation, achieving 9/12 top-1 results on unseen games compared with baselines. Analyses through ablations, learning dynamics, and case studies reveal the complementary contributions of both components and show meta-reasoning evolving toward human-like adaptation strategies.
☆ Time Series Forecasting via Direct Per-Step Probability Distribution Modeling AAAI
Linghao Kong, Xiaopeng Hong
Deep neural network-based time series prediction models have recently demonstrated superior capabilities in capturing complex temporal dependencies. However, it is challenging for these models to account for uncertainty associated with their predictions, because they directly output scalar values at each time step. To address such a challenge, we propose a novel model named interleaved dual-branch Probability Distribution Network (interPDN), which directly constructs discrete probability distributions per step instead of a scalar. The regression output at each time step is derived by computing the expectation of the predictive distribution on a predefined support set. To mitigate prediction anomalies, a dual-branch architecture is introduced with interleaved support sets, augmented by coarse temporal-scale branches for long-term trend forecasting. Outputs from another branch are treated as auxiliary signals to impose self-supervised consistency constraints on the current branch's prediction. Extensive experiments on multiple real-world datasets demonstrate the superior performance of interPDN.
comment : 16 pages, 8 figures. This is the preprint version of the paper and supplemental material to appear in AAAI, 2026. Please cite the final published version. Code is available at https://github.com/leonardokong486/interPDN
☆ Robust HRRP Recognition under Interrupted Sampling Repeater Jamming using a Prior Jamming Information -Guided Network
Guozheng Sun, Lei Wang, Yanhao Wang, Jie Wang, Yimin Liu
Radar automatic target recognition (RATR) based on high-resolution range profile (HRRP) has attracted increasing attention due to its ability to capture fine-grained structural features. However, recognizing targets under electronic countermeasures (ECM), especially the mainstream interrupted-sampling repeater jamming (ISRJ), remains a significant challenge, as HRRPs often suffer from serious feature distortion. To address this, we propose a robust HRRP recognition method guided by prior jamming information. Specifically, we introduce a point spread function (PSF) as prior information to model the HRRP distortion induced by ISRJ. Based on this, we design a recognition network that leverages this prior through a prior-guided feature interaction module and a hybrid loss function to enhance the model's discriminative capability. With the aid of prior information, the model can learn invariant features within distorted HRRP under different jamming parameters. Both the simulated and measured-data experiments demonstrate that our method consistently outperforms state-of-the-art approaches and exhibits stronger generalization capabilities when facing unseen jamming parameters.
☆ Agri CoT : A Chain-of-Thought Benchmark for Evaluating Reasoning in Vision -Language Models for Agriculture
Yibin Wen, Qingmei Li, Zi Ye, Jiarui Zhang, Jing Wu, Zurong Mai, Shuohong Lou, Yuhang Chen, Henglian Huang, Xiaoya Fan, Yang Zhang, Lingyuan Zhao, Haohuan Fu, Huang Jianxi, Juepeng Zheng
Recent advancements in Vision-Language Models (VLMs) have significantly transformed various industries. In agriculture, these dual-modal capabilities offer promising applications such as precision farming, crop monitoring, pest detection, and environmental sustainability. While several Visual Question Answering (VQA) datasets and benchmarks have been developed to evaluate VLM performance, they often fail to adequately assess the critical reasoning and problem-solving skills required in complex agricultural contexts. To address this gap, we introduce AgriCoT, a VQA dataset that incorporates Chain-of-Thought (CoT) reasoning, specifically designed to evaluate the reasoning capabilities of VLMs. With 4,535 carefully curated samples, AgriCoT offers a comprehensive and robust evaluation of reasoning abilities for VLMs, particularly in zero-shot scenarios, by focusing on their capacity to engage in logical reasoning and effective problem-solving. Our evaluations, conducted with 26 representative VLMs, including both proprietary and open-source models, reveal that while some proprietary models excel at answering questions, there is a notable and significant gap in their reasoning capabilities. This underscores the importance of incorporating CoT for more precise and effective assessments. Our dataset are available at https://huggingface.co/datasets/wenyb/AgriCoT.
☆ One-Shot Secure Aggregation: A Hybrid Cryptographic Protocol for Private Federated Learning in IoT
Imraul Emmaka, Tran Viet Xuan Phuong
Federated Learning (FL) offers a promising approach to collaboratively train machine learning models without centralizing raw data, yet its scalability is often throttled by excessive communication overhead. This challenge is magnified in Internet of Things (IoT) environments, where devices face stringent bandwidth, latency, and energy constraints. Conventional secure aggregation protocols, while essential for protecting model updates, frequently require multiple interaction rounds, large payload sizes, and per-client costs rendering them impractical for many edge deployments. In this work, we present Hyb-Agg, a lightweight and communication-efficient secure aggregation protocol that integrates Multi-Key CKKS (MK-CKKS) homomorphic encryption with Elliptic Curve Diffie-Hellman (ECDH)-based additive masking. Hyb-Agg reduces the secure aggregation process to a single, non-interactive client-to-server transmission per round, ensuring that per-client communication remains constant regardless of the number of participants. This design eliminates partial decryption exchanges, preserves strong privacy under the RLWE, CDH, and random oracle assumptions, and maintains robustness against collusion by the server and up to $N-2$ clients. We implement and evaluate Hyb-Agg on both high-performance and resource-constrained devices, including a Raspberry Pi 4, demonstrating that it delivers sub-second execution times while achieving a constant communication expansion factor of approximately 12x over plaintext size. By directly addressing the communication bottleneck, Hyb-Agg enables scalable, privacy-preserving federated learning that is practical for real-world IoT deployments.
comment : 11 pages, 6 figures. Accepted at The 7th IEEE International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA 2025)
☆ Learning to Predict Aboveground Biomass from RGB Images with 3D Synthetic Scenes
Silvia Zuffi
Forests play a critical role in global ecosystems by supporting biodiversity and mitigating climate change via carbon sequestration. Accurate aboveground biomass (AGB) estimation is essential for assessing carbon storage and wildfire fuel loads, yet traditional methods rely on labor-intensive field measurements or remote sensing approaches with significant limitations in dense vegetation. In this work, we propose a novel learning-based method for estimating AGB from a single ground-based RGB image. We frame this as a dense prediction task, introducing AGB density maps, where each pixel represents tree biomass normalized by the plot area and each tree's image area. We leverage the recently introduced synthetic 3D SPREAD dataset, which provides realistic forest scenes with per-image tree attributes (height, trunk and canopy diameter) and instance segmentation masks. Using these assets, we compute AGB via allometric equations and train a model to predict AGB density maps, integrating them to recover the AGB estimate for the captured scene. Our approach achieves a median AGB estimation error of 1.22 kg/m^2 on held-out SPREAD data and 1.94 kg/m^2 on a real-image dataset. To our knowledge, this is the first method to estimate aboveground biomass directly from a single RGB image, opening up the possibility for a scalable, interpretable, and cost-effective solution for forest monitoring, while also enabling broader participation through citizen science initiatives.
comment : Presented at STAG 2025
☆ Tourism Question Answer System in Indian Language using Domain-Adapted Foundation Models
Praveen Gatla, Anushka, Nikita Kanwar, Gouri Sahoo, Rajesh Kumar Mundotiya
This article presents the first comprehensive study on designing a baseline extractive question-answering (QA) system for the Hindi tourism domain, with a specialized focus on the Varanasi-a cultural and spiritual hub renowned for its Bhakti-Bhaav (devotional ethos). Targeting ten tourism-centric subdomains-Ganga Aarti, Cruise, Food Court, Public Toilet, Kund, Museum, General, Ashram, Temple and Travel, the work addresses the absence of language-specific QA resources in Hindi for culturally nuanced applications. In this paper, a dataset comprising 7,715 Hindi QA pairs pertaining to Varanasi tourism was constructed and subsequently augmented with 27,455 pairs generated via Llama zero-shot prompting. We propose a framework leveraging foundation models-BERT and RoBERTa, fine-tuned using Supervised Fine-Tuning (SFT) and Low-Rank Adaptation (LoRA), to optimize parameter efficiency and task performance. Multiple variants of BERT, including pre-trained languages (e.g., Hindi-BERT), are evaluated to assess their suitability for low-resource domain-specific QA. Evaluation metrics - F1, BLEU, and ROUGE-L - highlight trade-offs between answer precision and linguistic fluency. Experiments demonstrate that LoRA-based fine-tuning achieves competitive performance (85.3\% F1) while reducing trainable parameters by 98\% compared to SFT, striking a balance between efficiency and accuracy. Comparative analysis across models reveals that RoBERTa with SFT outperforms BERT variants in capturing contextual nuances, particularly for culturally embedded terms (e.g., Aarti, Kund). This work establishes a foundational baseline for Hindi tourism QA systems, emphasizing the role of LORA in low-resource settings and underscoring the need for culturally contextualized NLP frameworks in the tourism domain.
☆ GAVINA: flexible aggressive undervolting for bit-serial mixed-precision DNN acceleration
Jordi Fornt, Pau Fontova-Musté, Adrian Gras, Omar Lahyani, Martí Caro, Jaume Abella, Francesc Moll, Josep Altet
Voltage overscaling, or undervolting, is an enticing approximate technique in the context of energy-efficient Deep Neural Network (DNN) acceleration, given the quadratic relationship between power and voltage. Nevertheless, its very high error rate has thwarted its general adoption. Moreover, recent undervolting accelerators rely on 8-bit arithmetic and cannot compete with state-of-the-art low-precision (<8b) architectures. To overcome these issues, we propose a new technique called Guarded Aggressive underVolting (GAV), which combines the ideas of undervolting and bit-serial computation to create a flexible approximation method based on aggressively lowering the supply voltage on a select number of least significant bit combinations. Based on this idea, we implement GAVINA (GAV mIxed-precisioN Accelerator), a novel architecture that supports arbitrary mixed precision and flexible undervolting, with an energy efficiency of up to 89 TOP/sW in its most aggressive configuration. By developing an error model of GAVINA, we show that GAV can achieve an energy efficiency boost of 20% via undervolting, with negligible accuracy degradation on ResNet-18.
comment : Presented in the 2025 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED). Conference proceedings pending to be published
☆ Vision Bridge Transformer at Scale
Zhenxiong Tan, Zeqing Wang, Xingyi Yang, Songhua Liu, Xinchao Wang
We introduce Vision Bridge Transformer (ViBT), a large-scale instantiation of Brownian Bridge Models designed for conditional generation. Unlike traditional diffusion models that transform noise into data, Bridge Models directly model the trajectory between inputs and outputs, creating an efficient data-to-data translation paradigm. By scaling these models to 20B and 1.3B parameters, we demonstrate their effectiveness for image and video translation tasks. To support this scale, we adopt a Transformer architecture and propose a variance-stabilized velocity-matching objective for robust training. Together, these advances highlight the power of scaling Bridge Models for instruction-based image editing and complex video translation.
☆ Obstruction reasoning for robotic grasping
Runyu Jiao, Matteo Bortolon, Francesco Giuliari, Alice Fasoli, Sergio Povoli, Guofeng Mei, Yiming Wang, Fabio Poiesi
Successful robotic grasping in cluttered environments not only requires a model to visually ground a target object but also to reason about obstructions that must be cleared beforehand. While current vision-language embodied reasoning models show emergent spatial understanding, they remain limited in terms of obstruction reasoning and accessibility planning. To bridge this gap, we present UNOGrasp, a learning-based vision-language model capable of performing visually-grounded obstruction reasoning to infer the sequence of actions needed to unobstruct the path and grasp the target object. We devise a novel multi-step reasoning process based on obstruction paths originated by the target object. We anchor each reasoning step with obstruction-aware visual cues to incentivize reasoning capability. UNOGrasp combines supervised and reinforcement finetuning through verifiable reasoning rewards. Moreover, we construct UNOBench, a large-scale dataset for both training and benchmarking, based on MetaGraspNetV2, with over 100k obstruction paths annotated by humans with obstruction ratios, contact points, and natural-language instructions. Extensive experiments and real-robot evaluations show that UNOGrasp significantly improves obstruction reasoning and grasp success across both synthetic and real-world environments, outperforming generalist and proprietary alternatives. Project website: https://tev-fbk.github.io/UnoGrasp/.
☆ Listwise Preference Optimization with Element-wise Confusions for Aspect Sentiment Quad Prediction
Wenna Lai, Haoran Xie, Guandong Xu, Qing Li, S. Joe Qin
Aspect sentiment quad prediction (ASQP) is inherently challenging to predict a structured quadruple with four core sentiment elements, including aspect term (a), aspect category (c), opinion term (o), and sentiment polarity (s). Prior methods relying on marker-based prediction struggle with modeling the intricate relationships among elements and experience sharp performance declines when predicting higher-order elements (e.g., c and s) under standard supervised fine-tuning. To address these limitations, we employ reasoning-based generation to output both the quadruple and a natural language rationale under element prefixes within a unified template, encouraging explicit relational reasoning and interpretability. To further enhance element-wise alignment, we introduce a listwise preference optimization framework for improving structural validity and relational coherence. Specifically, we generate element-wise confusable candidates via syntactic and semantic proximity, then train the model with listwise objectives to prefer the gold candidates over closely competing alternatives. Extensive experiments on four benchmark datasets demonstrate that our framework effectively improves quadruple prediction accuracy and explanation consistency.
comment : 11 pages, 7 figures, and 6 tables
☆ Identification of Malicious Posts on the Dark Web Using Supervised Machine Learning
Sebastião Alves de Jesus Filho, Gustavo Di Giovanni Bernardo, Paulo Henrique Ribeiro Gabriel, Bruno Bogaz Zarpelão, Rodrigo Sanches Miani
Given the constant growth and increasing sophistication of cyberattacks, cybersecurity can no longer rely solely on traditional defense techniques and tools. Proactive detection of cyber threats has become essential to help security teams identify potential risks and implement effective mitigation measures. Cyber Threat Intelligence (CTI) plays a key role by providing security analysts with evidence-based knowledge about cyber threats. CTI information can be extracted using various techniques and data sources; however, machine learning has proven promising. As for data sources, social networks and online discussion forums are commonly explored. In this study, we apply text mining techniques and machine learning to data collected from Dark Web forums in Brazilian Portuguese to identify malicious posts. Our contributions include the creation of three original datasets, a novel multi-stage labeling process combining indicators of compromise (IoCs), contextual keywords, and manual analysis, and a comprehensive evaluation of text representations and classifiers. To our knowledge, this is the first study to focus specifically on Brazilian Portuguese content in this domain. The best-performing model, using LightGBM and TF-IDF, was able to detect relevant posts with high accuracy. We also applied topic modeling to validate the model's outputs on unlabeled data, confirming its robustness in real-world scenarios.
comment : Manuscript under review (SN Computer Science)
☆ AI for software engineering: from probable to provable
Bertrand Meyer
Vibe coding, the much-touted use of AI techniques for programming, faces two overwhelming obstacles: the difficulty of specifying goals ("prompt engineering" is a form of requirements engineering, one of the toughest disciplines of software engineering); and the hallucination phenomenon. Programs are only useful if they are correct or very close to correct. The solution? Combine the creativity of artificial intelligence with the rigor of formal specification methods and the power of formal program verification, supported by modern proof tools.
☆ REVEAL: Reasoning -enhanced Forensic Evidence Analysis for Explainable AI-generated Image Detection
Huangsen Cao, Qin Mei, Zhiheng Li, Yuxi Li, Ying Zhang, Chen Li, Zhimeng Zhang, Xin Ding, Yongwei Wang, Jing Lyu, Fei Wu
With the rapid advancement of generative models, visually realistic AI-generated images have become increasingly difficult to distinguish from authentic ones, posing severe threats to social trust and information integrity. Consequently, there is an urgent need for efficient and truly explainable image forensic methods. Recent detection paradigms have shifted towards explainable forensics. However, state-of-the-art approaches primarily rely on post-hoc rationalizations or visual discrimination, lacking a verifiable chain of evidence. This reliance on surface-level pattern matching limits the generation of causally grounded explanations and often results in poor generalization. To bridge this critical gap, we introduce \textbf{REVEAL-Bench}, the first reasoning-enhanced multimodal benchmark for AI-generated image detection that is explicitly structured around a chain-of-evidence derived from multiple lightweight expert models, then records step-by-step reasoning traces and evidential justifications. Building upon this dataset, we propose \textbf{REVEAL} (\underline{R}easoning-\underline{e}nhanced Forensic E\underline{v}id\underline{e}nce \underline{A}na\underline{l}ysis), an effective and explainable forensic framework that integrates detection with a novel expert-grounded reinforcement learning. Our reward mechanism is specially tailored to jointly optimize detection accuracy, explanation fidelity, and logical coherence grounded in explicit forensic evidence, enabling REVEAL to produce fine-grained, interpretable, and verifiable reasoning chains alongside its detection outcomes. Extensive experimental results demonstrate that REVEAL significantly enhances detection accuracy, explanation fidelity, and robust cross-model generalization, benchmarking a new state of the art for explainable image forensics.
☆ Peer-to-Peer Energy Trading in Dairy Farms using Multi- Agent Reinforcement Learning
Mian Ibad Ali Shah, Marcos Eduardo Cruz Victorio, Maeve Duffy, Enda Barrett, Karl Mason
The integration of renewable energy resources in rural areas, such as dairy farming communities, enables decentralized energy management through Peer-to-Peer (P2P) energy trading. This research highlights the role of P2P trading in efficient energy distribution and its synergy with advanced optimization techniques. While traditional rule-based methods perform well under stable conditions, they struggle in dynamic environments. To address this, Multi-Agent Reinforcement Learning (MARL), specifically Proximal Policy Optimization (PPO) and Deep Q-Networks (DQN), is combined with community/distributed P2P trading mechanisms. By incorporating auction-based market clearing, a price advisor agent, and load and battery management, the approach achieves significant improvements. Results show that, compared to baseline models, DQN reduces electricity costs by 14.2% in Ireland and 5.16% in Finland, while increasing electricity revenue by 7.24% and 12.73%, respectively. PPO achieves the lowest peak hour demand, reducing it by 55.5% in Ireland, while DQN reduces peak hour demand by 50.0% in Ireland and 27.02% in Finland. These improvements are attributed to both MARL algorithms and P2P energy trading, which together results in electricity cost and peak hour demand reduction, and increase electricity selling revenue. This study highlights the complementary strengths of DQN, PPO, and P2P trading in achieving efficient, adaptable, and sustainable energy management in rural communities.
comment : 51 pages, 7 figures, 11 tables, Preprint of the article published in Applied Energy: Shah, M.I.A., Victorio, M.E.C., Duffy, M., Barrett, E. and Mason, K. (2026). Peer-to-peer energy trading in dairy farms using multi-agent reinforcement learning. Applied Energy, 402, 127041. doi:10.1016/j.apenergy.2025.127041
☆ Automated Generation of MDPs Using Logic Programming and LLMs for Robotic Applications
Enrico Saccon, Davide De Martini, Matteo Saveriano, Edoardo Lamon, Luigi Palopoli, Marco Roveri
We present a novel framework that integrates Large Language Models (LLMs) with automated planning and formal verification to streamline the creation and use of Markov Decision Processes (MDP). Our system leverages LLMs to extract structured knowledge in the form of a Prolog knowledge base from natural language (NL) descriptions. It then automatically constructs an MDP through reachability analysis, and synthesises optimal policies using the Storm model checker. The resulting policy is exported as a state-action table for execution. We validate the framework in three human-robot interaction scenarios, demonstrating its ability to produce executable policies with minimal manual effort. This work highlights the potential of combining language models with formal methods to enable more accessible and scalable probabilistic planning in robotics.
comment : 9 pages, 11 figures, 2 tables, 2 algorithms, accepted for publication in IEEE Robotics and Automation Letters
☆ Multi-chain Graph Refinement and Selection for Reliable Reasoning in Large Language Models
Yujiao Yang, Jing Lian, Linhui Li
The complex reasoning ability of Large Language Models (LLMs) poses a critical bottleneck for their practical applications. Test-time expansion methods such as Tree-of-Thought (ToT) and Graph-of-Thought (GoT) enhance reasoning by introducing intermediate reasoning structures, tree search, or graph-based exploration mechanisms. However, their reasoning strategies suffer from limited diversity, redundant search branches, and inadequate integration and error correction across heterogeneous reasoning paths. To address these limitations, we propose a novel reasoning framework called Multi-chain Graph Refinement & Selection (MGRS), which first generates multiple diverse reasoning trajectories for a given problem, refines candidate responses using a composite self- and cross-verification strategy, then constructs a reasoning relation graph and estimates the success rate of intermediate nodes, and finally computes cumulative success rates to select the most reliable answer and corresponding reasoning trajectory. Experimental results demonstrate that MGRS significantly advances both the reasoning capability and computational efficiency of reasoning enhancement methods. Across six benchmark datasets spanning four distinct tasks, MGRS achieves an average accuracy of 82.9%, outperforming state-of-the-art baselines by a clear margin of 2.1%. Remarkably, on the 24-point game, MGRS attains 100% accuracy for the first time, while delivering a 13.6x speed-up compared to the leading Forest of Thoughts framework.
☆ Evolutionary Discovery of Heuristic Policies for Traffic Signal Control
Ruibing Wang, Shuhan Guo, Zeen Li, Zhen Wang, Quanming Yao
Traffic Signal Control (TSC) involves a challenging trade-off: classic heuristics are efficient but oversimplified, while Deep Reinforcement Learning (DRL) achieves high performance yet suffers from poor generalization and opaque policies. Online Large Language Models (LLMs) provide general reasoning but incur high latency and lack environment-specific optimization. To address these issues, we propose Temporal Policy Evolution for Traffic (\textbf{\method{}}), which uses LLMs as an evolution engine to derive specialized heuristic policies. The framework introduces two key modules: (1) Structured State Abstraction (SSA), converting high-dimensional traffic data into temporal-logical facts for reasoning; and (2) Credit Assignment Feedback (CAF), tracing flawed micro-decisions to poor macro-outcomes for targeted critique. Operating entirely at the prompt level without training, \method{} yields lightweight, robust policies optimized for specific traffic environments, outperforming both heuristics and online LLM actors.
☆ Mind Reading or Misreading? LLMs on the Big Five Personality Test SC
Francesco Di Cursi, Chiara Boldrini, Marco Conti, Andrea Passarella
We evaluate large language models (LLMs) for automatic personality prediction from text under the binary Five Factor Model (BIG5). Five models -- including GPT-4 and lightweight open-source alternatives -- are tested across three heterogeneous datasets (Essays, MyPersonality, Pandora) and two prompting strategies (minimal vs. enriched with linguistic and psychological cues). Enriched prompts reduce invalid outputs and improve class balance, but also introduce a systematic bias toward predicting trait presence. Performance varies substantially: Openness and Agreeableness are relatively easier to detect, while Extraversion and Neuroticism remain challenging. Although open-source models sometimes approach GPT-4 and prior benchmarks, no configuration yields consistently reliable predictions in zero-shot binary settings. Moreover, aggregate metrics such as accuracy and macro-F1 mask significant asymmetries, with per-class recall offering clearer diagnostic value. These findings show that current out-of-the-box LLMs are not yet suitable for APPT, and that careful coordination of prompt design, trait framing, and evaluation metrics is essential for interpretable results.
comment : Funding: SoBigDatait (IR0000013), FAIR (PE00000013), ICSC (CN00000013)
☆ Fairness in the Multi-Secretary Problem AAAI'26
Georgios Papasotiropoulos, Zein Pishbin
This paper bridges two perspectives: it studies the multi-secretary problem through the fairness lens of social choice, and examines multi-winner elections from the viewpoint of online decision making. After identifying the limitations of the prominent proportionality notion of Extended Justified Representation (EJR) in the online domain, the work proposes a set of mechanisms that merge techniques from online algorithms with rules from social choice -- such as the Method of Equal Shares and the Nash Rule -- and supports them through both theoretical analysis and extensive experimental evaluation.
comment : AAAI'26
☆ Does Self-Evaluation Enable Wireheading in Language Models? AAAI 2026
David Demitri Africa, Hans Ethan Ting
Self-evaluation is increasingly central to language model training, from constitutional AI to self-refinement. We investigate whether coupling self-evaluation to reward signals creates incentives for wireheading, where agents manipulate reward measurements rather than improving task performance. We formalize conditions under which reward-channel control strictly dominates task-focused behavior in POMDPs and test these predictions empirically. Across two models and three tasks, we find that models whose self-grades determine rewards exhibit substantial grade inflation without corresponding accuracy gains, particularly on ambiguous tasks like summarization. Models that self-evaluate but do not control rewards show no such inflation. Our results demonstrate that self-evaluation is safe when decoupled from learning signals but dangerous when coupled, with clear implications for agentic system design.
comment : Accepted (oral) to Foundations of Agentic Systems Theory at AAAI 2026
☆ SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision -Language Models
Ruosen Zhao, Zhikang Zhang, Jialei Xu, Jiahao Chang, Dong Chen, Lingyun Li, Weijian Sun, Zizhuang Wei
Large vision-language models (VLMs) show strong multimodal understanding but still struggle with 3D spatial reasoning, such as distance estimation, size comparison, and cross-view consistency. Existing 3D-aware methods either depend on auxiliary 3D information or enhance RGB-only VLMs with geometry encoders through shallow feature fusion. We propose SpaceMind, a multimodal large language model explicitly designed for spatial reasoning solely from RGB inputs. The model adopts a dual-encoder architecture, integrating VGGT as a spatial understanding encoder and InternViT as a 2D visual encoder. The key idea is to treat the camera representation as an active guiding modality rather than passive metadata. Specifically, SpaceMind introduces a lightweight Camera-Guided Modality Fusion module before the language model to replace shallow fusion. It applies camera-conditioned biasing to spatial tokens, assigns query-independent weights reflecting their geometric importance, and uses the camera embedding to gate the fused representation. Empirically, SpaceMind establishes new state-of-the-art results on VSI-Bench, SQA3D and SPBench, surpassing both open and proprietary systems on VSI-Bench and SPBench by large margins and achieving state-of-the-art performance on SQA3D. These results demonstrate that camera-guided modality fusion is an effective and practical inductive bias for equipping VLMs with genuinely spatially grounded intelligence. We will release code and model checkpoints to support future research.
☆ What If They Took the Shot? A Hierarchical Bayesian Framework for Counterfactual Expected Goals
Mikayil Mahmudlu, Oktay Karakuş, Hasan Arkadaş
This study develops a hierarchical Bayesian framework that integrates expert domain knowledge to quantify player-specific effects in expected goals (xG) estimation, addressing a limitation of standard models that treat all players as identical finishers. Using 9,970 shots from StatsBomb's 2015-16 data and Football Manager 2017 ratings, we combine Bayesian logistic regression with informed priors to stabilise player-level estimates, especially for players with few shots. The hierarchical model reduces posterior uncertainty relative to weak priors and achieves strong external validity: hierarchical and baseline predictions correlate at R2 = 0.75, while an XGBoost benchmark validated against StatsBomb xG reaches R2 = 0.833. The model uncovers interpretable specialisation profiles, including one-on-one finishing (Aguero, Suarez, Belotti, Immobile, Martial), long-range shooting (Pogba), and first-touch execution (Insigne, Salah, Gameiro). It also identifies latent ability in underperforming players such as Immobile and Belotti. The framework supports counterfactual "what-if" analysis by reallocating shots between players under identical contexts. Case studies show that Sansone would generate +2.2 xG from Berardi's chances, driven largely by high-pressure situations, while Vardy-Giroud substitutions reveal strong asymmetry: replacing Vardy with Giroud results in a large decline (about -7 xG), whereas the reverse substitution has only a small effect (about -1 xG). This work provides an uncertainty-aware tool for player evaluation, recruitment, and tactical planning, and offers a general approach for domains where individual skill and contextual factors jointly shape performance.
☆ Bharat Scene Text: A Novel Comprehensive Dataset and Benchmark for Indian Language Scene Text Understanding
Anik De, Abhirama Subramanyam Penamakuri, Rajeev Yadav, Aditya Rathore, Harshiv Shah, Devesh Sharma, Sagar Agarwal, Pravin Kumar, Anand Mishra
Reading scene text, that is, text appearing in images, has numerous application areas, including assistive technology, search, and e-commerce. Although scene text recognition in English has advanced significantly and is often considered nearly a solved problem, Indian language scene text recognition remains an open challenge. This is due to script diversity, non-standard fonts, and varying writing styles, and, more importantly, the lack of high-quality datasets and open-source models. To address these gaps, we introduce the Bharat Scene Text Dataset (BSTD) - a large-scale and comprehensive benchmark for studying Indian Language Scene Text Recognition. It comprises more than 100K words that span 11 Indian languages and English, sourced from over 6,500 scene images captured across various linguistic regions of India. The dataset is meticulously annotated and supports multiple scene text tasks, including: (i) Scene Text Detection, (ii) Script Identification, (iii) Cropped Word Recognition, and (iv) End-to-End Scene Text Recognition. We evaluated state-of-the-art models originally developed for English by adapting (fine-tuning) them for Indian languages. Our results highlight the challenges and opportunities in Indian language scene text recognition. We believe that this dataset represents a significant step toward advancing research in this domain. All our models and data are open source.
comment : Under Peer Review
☆ Evaluating the Clinical Impact of Generative Inpainting on Bone Age Estimation
Felipe Akio Matsuoka, Eduardo Moreno J. M. Farina, Augusto Sarquis Serpa, Soraya Monteiro, Rodrigo Ragazzini, Nitamar Abdala, Marcelo Straus Takahashi, Felipe Campos Kitamura
Generative foundation models can remove visual artifacts through realistic image inpainting, but their impact on medical AI performance remains uncertain. Pediatric hand radiographs often contain non-anatomical markers, and it is unclear whether inpainting these regions preserves features needed for bone age and gender prediction. To evaluate the clinical reliability of generative model-based inpainting for artifact removal, we used the RSNA Bone Age Challenge dataset, selecting 200 original radiographs and generating 600 inpainted versions with gpt-image-1 using natural language prompts to target non-anatomical artifacts. Downstream performance was assessed with deep learning ensembles for bone age estimation and gender classification, using mean absolute error (MAE) and area under the ROC curve (AUC) as metrics, and pixel intensity distributions to detect structural alterations. Inpainting markedly degraded model performance: bone age MAE increased from 6.26 to 30.11 months, and gender classification AUC decreased from 0.955 to 0.704. Inpainted images displayed pixel-intensity shifts and inconsistencies, indicating structural modifications not corrected by simple calibration. These findings show that, although visually realistic, foundation model-based inpainting can obscure subtle but clinically relevant features and introduce latent bias even when edits are confined to non-diagnostic regions, underscoring the need for rigorous, task-specific validation before integrating such generative tools into clinical AI workflows.
comment : 8 pages, 4 figures
☆ Conveying Imagistic Thinking in TCM Translation: A Prompt Engineering and LLM-Based Evaluation Framework
Jiatong Han
Traditional Chinese Medicine theory is built on imagistic thinking, in which medical principles and diagnostic and therapeutic logic are structured through metaphor and metonymy. However, existing English translations largely rely on literal rendering, making it difficult for target-language readers to reconstruct the underlying conceptual networks and apply them in clinical practice. This study adopted a human-in-the-loop framework and selected four passages from the medical canon Huangdi Neijing that are fundamental in theory. Through prompt-based cognitive scaffolding, DeepSeek V3.1 was guided to identify metaphor and metonymy in the source text and convey the theory in translation. In the evaluation stage, ChatGPT 5 Pro and Gemini 2.5 Pro were instructed by prompts to simulate three types of real-world readers. Human translations, baseline model translations, and prompt-adjusted translations were scored by the simulated readers across five cognitive dimensions, followed by structured interviews and Interpretative Phenomenological Analysis. Results show that the prompt-adjusted LLM translations perform best across all five dimensions, with high cross-model and cross-role consistency. The interview themes reveal differences between human and machine translation, effective strategies for metaphor and metonymy transfer, and readers' cognitive preferences. This study provides a cognitive, efficient and replicable HITL methodological pathway for translation of ancient, concept-dense texts like TCM.
comment : 3 figures
☆ MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agent s
Ruoxuan Zhang, Qiyun Zheng, Zhiyu Zhou, Ziqi Liao, Siyu Wu, Jian-Yu Jiang-Lin, Bin Wen, Hongxia Xie, Jianlong Fu, Wen-Huang Cheng
Theory of Mind (ToM) refers to the ability to infer others' mental states, such as beliefs, desires, and intentions. Current vision-language embodied agents lack ToM-based decision-making, and existing benchmarks focus solely on human mental states while ignoring the agent's own perspective, hindering coherent decision and action generation. To address this, we propose MindPower, a Robot-Centric framework integrating Perception, Mental Reasoning, Decision Making and Action. Given multimodal inputs, MindPower first perceives the environment and human states, then performs ToM Reasoning to model both self and others, and finally generates decisions and actions guided by inferred mental states. Furthermore, we introduce Mind-Reward, a novel optimization objective that encourages VLMs to produce consistent ToM Reasoning and behavior. Our model outperforms GPT-4o by 12.77% in decision making and 12.49% in action generation.
☆ High-Resolution Probabilistic Data-Driven Weather Modeling with a Stretched-Grid
Even Marius Nordhagen, Håvard Homleid Haugen, Aram Farhad Shafiq Salihi, Magnus Sikora Ingstad, Thomas Nils Nipen, Ivar Ambjørn Seierstad, Inger-Lise Frogner, Mariana Clare, Simon Lang, Matthew Chantry, Peter Dueben, Jørn Kristiansen
We present a probabilistic data-driven weather model capable of providing an ensemble of high spatial resolution realizations of 87 variables at arbitrary forecast length and ensemble size. The model uses a stretched grid, dedicating 2.5 km resolution to a region of interest, and 31 km resolution elsewhere. Based on a stochastic encoder-decoder architecture, the model is trained using a loss function based on the Continuous Ranked Probability Score (CRPS) evaluated point-wise in real and spectral space. The spectral loss components is shown to be necessary to create fields that are spatially coherent. The model is compared to high-resolution operational numerical weather prediction forecasts from the MetCoOp Ensemble Prediction System (MEPS), showing competitive forecasts when evaluated against observations from surface weather stations. The model produced fields that are more spatially coherent than mean squared error based models and CRPS based models without the spectral component in the loss.
comment : 14 pages, 8 figures
☆ Delta-XAI: A Unified Framework for Explaining Prediction Changes in Online Time Series Monitoring ICLR 2026
Changhun Kim, Yechan Mun, Hyeongwon Jang, Eunseo Lee, Sangchul Hahn, Eunho Yang
Explaining online time series monitoring models is crucial across sensitive domains such as healthcare and finance, where temporal and contextual prediction dynamics underpin critical decisions. While recent XAI methods have improved the explainability of time series models, they mostly analyze each time step independently, overlooking temporal dependencies. This results in further challenges: explaining prediction changes is non-trivial, methods fail to leverage online dynamics, and evaluation remains difficult. To address these challenges, we propose Delta-XAI, which adapts 14 existing XAI methods through a wrapper function and introduces a principled evaluation suite for the online setting, assessing diverse aspects, such as faithfulness, sufficiency, and coherence. Experiments reveal that classical gradient-based methods, such as Integrated Gradients (IG), can outperform recent approaches when adapted for temporal analysis. Building on this, we propose Shifted Window Integrated Gradients (SWING), which incorporates past observations in the integration path to systematically capture temporal dependencies and mitigate out-of-distribution effects. Extensive experiments consistently demonstrate the effectiveness of SWING across diverse settings with respect to diverse metrics. Our code is publicly available at https://anonymous.4open.science/r/Delta-XAI.
comment : Under review at ICLR 2026
☆ From Illusion to Intention: Visual Rationale Learning for Vision -Language Reasoning
Changpeng Wang, Haozhe Wang, Xi Chen, Junhan Liu, Taofeng Xue, Chong Peng, Donglian Qi, Fangzhen Lin, Yunfeng Yan
Recent advances in vision-language reasoning underscore the importance of thinking with images, where models actively ground their reasoning in visual evidence. Yet, prevailing frameworks treat visual actions as optional tools, boosting metrics but leaving reasoning ungrounded and crops ineffective. This gap gives rise to the illusion of thinking with images: models seem visually grounded but rely on context-agnostic actions that neither refine perception nor guide reasoning toward correct answers. We address this problem by reframing visual actions as core reasoning primitives rather than optional tools, which we term visual rationalization, the visual analogue of textual Chain-of-Thought. Building on this insight, we propose Visual Rationale Learning (ViRL), an end-to-end paradigm that grounds training in the visual rationale itself. ViRL integrates (1) Process Supervision with ground-truth rationales, (2) Objective Alignment via step-level reward shaping, and (3) Fine-Grained Credit Assignment to distinguish correct, redundant, and erroneous actions. By ensuring each action contributes meaningfully to the reasoning chain, ViRL enables models to "get the right answer for the right visual reason". Trained purely with end-to-end RL, ViRL achieves state-of-the-art results across benchmarks spanning perception, hallucination, and reasoning. This work establishes visual rationalization as a task-agnostic, process-grounded paradigm for building transparent, verifiable, and trustworthy vision-language models.
comment : 19 pages, 15 figures
☆ A transfer learning approach for automatic conflicts detection in software requirement sentence pairs based on dual encoders
Yizheng Wang, Tao Jiang, Jinyan Bai, Zhengbin Zou, Tiancheng Xue, Nan Zhang, Jie Luan
Software Requirement Document (RD) typically contain tens of thousands of individual requirements, and ensuring consistency among these requirements is critical for the success of software engineering projects. Automated detection methods can significantly enhance efficiency and reduce costs; however, existing approaches still face several challenges, including low detection accuracy on imbalanced data, limited semantic extraction due to the use of a single encoder, and suboptimal performance in cross-domain transfer learning. To address these issues, this paper proposes a Transferable Software Requirement Conflict Detection Framework based on SBERT and SimCSE, termed TSRCDF-SS. First, the framework employs two independent encoders, Sentence-BERT (SBERT) and Simple Contrastive Sentence Embedding (SimCSE), to generate sentence embeddings for requirement pairs, followed by a six-element concatenation strategy. Furthermore, the classifier is enhanced by a two-layer fully connected feedforward neural network (FFNN) with a hybrid loss optimization strategy that integrates a variant of Focal Loss, domain-specific constraints, and a confidence-based penalty term. Finally, the framework synergistically integrates sequential and cross-domain transfer learning. Experimental results demonstrate that the proposed framework achieves a 10.4% improvement in both macro-F1 and weighted-F1 scores in in-domain settings, and an 11.4% increase in macro-F1 in cross-domain scenarios.
comment : 22 pages, 7 figures, 3 tables
☆ TIM-PRM: Verifying multimodal reasoning with Tool-Integrated PRM
Peng Kuang, Xiangxiang Wang, Wentao Liu, Jian Dong, Kaidi Xu, Haohan Wang
Multimodal Large Language Models (MLLMs) have achieved impressive performances in mathematical reasoning, yet they remain vulnerable to visual hallucinations and logical inconsistencies that standard outcome-based supervision fails to mitigate. While Process Reward Models (PRMs) promise step-by-step verification, current approaches typically operate as scalar scorers or generative critics that suffer from sycophancy, blindly validating the flawed hypotheses rather than grounding them in visual reality. To bridge this gap, we introduce TIM-PRM (Tool-Integrated Multimodal PRM), a novel agentic framework that transforms verification from a passive classification task into an active, tool-augmented investigation. TIM-PRM is trained to explicitly plan verification strategies and utilizes a mechanism of Independent Question Asking to query evidence via external tools, effectively decoupling verification from the reasoning context to eliminate confirmation bias. We instantiate this method by curating a high-quality dataset of tool-integrated verification trajectories. Extensive experiments on VisualProcessBench demonstrate that our 8B parameter model surpasses existing open-source multimodal PRMs, significantly outperforming much larger models like Qwen2.5-72B and InternVL-78B, while offering interpretable insights into the verification process.
comment : 14 pages
☆ MIMM-X: Disentangling Spurious Cor relation s for Medical Image Analysis
Louisa Fay, Hajer Reguigui, Bin Yang, Sergios Gatidis, Thomas Küstner
Deep learning models can excel on medical tasks, yet often experience spurious correlations, known as shortcut learning, leading to poor generalization in new environments. Particularly in medical imaging, where multiple spurious correlations can coexist, misclassifications can have severe consequences. We propose MIMM-X, a framework that disentangles causal features from multiple spurious correlations by minimizing their mutual information. It enables predictions based on true underlying causal relationships rather than dataset-specific shortcuts. We evaluate MIMM-X on three datasets (UK Biobank, NAKO, CheXpert) across two imaging modalities (MRI and X-ray). Results demonstrate that MIMM-X effectively mitigates shortcut learning of multiple spurious correlations.
☆ Ovis-Image Technical Report
Guo-Hua Wang, Liangfu Cao, Tianyu Cui, Minghao Fu, Xiaohao Chen, Pengxin Zhan, Jianshan Zhao, Lan Li, Bowen Fu, Jiaqi Liu, Qing-Guo Chen
We introduce $\textbf{Ovis-Image}$, a 7B text-to-image model specifically optimized for high-quality text rendering, designed to operate efficiently under stringent computational constraints. Built upon our previous Ovis-U1 framework, Ovis-Image integrates a diffusion-based visual decoder with the stronger Ovis 2.5 multimodal backbone, leveraging a text-centric training pipeline that combines large-scale pre-training with carefully tailored post-training refinements. Despite its compact architecture, Ovis-Image achieves text rendering performance on par with significantly larger open models such as Qwen-Image and approaches closed-source systems like Seedream and GPT4o. Crucially, the model remains deployable on a single high-end GPU with moderate memory, narrowing the gap between frontier-level text rendering and practical deployment. Our results indicate that combining a strong multimodal backbone with a carefully designed, text-focused training recipe is sufficient to achieve reliable bilingual text rendering without resorting to oversized or proprietary models.
comment : Code is released at https://github.com/AIDC-AI/Ovis-Image
☆ Pooling Attention: Evaluating Pretrain ed Transformer Embeddings for Deception Classification
Sumit Mamtani, Abhijeet Bhure
This paper investigates fake news detection as a downstream evaluation of Transformer representations, benchmarking encoder-only and decoder-only pre-trained models (BERT, GPT-2, Transformer-XL) as frozen embedders paired with lightweight classifiers. Through controlled preprocessing comparing pooling versus padding and neural versus linear heads, results demonstrate that contextual self-attention encodings consistently transfer effectively. BERT embeddings combined with logistic regression outperform neural baselines on LIAR dataset splits, while analyses of sequence length and aggregation reveal robustness to truncation and advantages from simple max or average pooling. This work positions attention-based token encoders as robust, architecture-centric foundations for veracity tasks, isolating Transformer contributions from classifier complexity.
comment : Accepted at the IEEE 7th Computing, Communications and IoT Applications Conference (ComComAp 2025), Madrid, Spain, December 2025. 6 pages
☆ Commanding Humanoid by Free-form Language: A Large Language Action Model with Unified Motion Vocabulary
Zhirui Liu, Kaiyang Ji, Ke Yang, Jingyi Yu, Ye Shi, Jingya Wang
Enabling humanoid robots to follow free-form language commands is critical for seamless human-robot interaction, collaborative task execution, and general-purpose embodied intelligence. While recent advances have improved low-level humanoid locomotion and robot manipulation, language-conditioned whole-body control remains a significant challenge. Existing methods are often limited to simple instructions and sacrifice either motion diversity or physical plausibility. To address this, we introduce Humanoid-LLA, a Large Language Action Model that maps expressive language commands to physically executable whole-body actions for humanoid robots. Our approach integrates three core components: a unified motion vocabulary that aligns human and humanoid motion primitives into a shared discrete space; a vocabulary-directed controller distilled from a privileged policy to ensure physical feasibility; and a physics-informed fine-tuning stage using reinforcement learning with dynamics-aware rewards to enhance robustness and stability. Extensive evaluations in simulation and on a real-world Unitree G1 humanoid show that Humanoid-LLA delivers strong language generalization while maintaining high physical fidelity, outperforming existing language-conditioned controllers in motion naturalness, stability, and execution success rate.
comment : Project page: https://humanoidlla.github.io/
☆ Bandit Guided Submodular Curriculum for Adaptive Subset Selection
Prateek Chanda, Prayas Agrawal, Saral Sureka, Lokesh Reddy Polu, Atharv Kshirsagar, Ganesh Ramakrishnan
Traditional curriculum learning proceeds from easy to hard samples, yet defining a reliable notion of difficulty remains elusive. Prior work has used submodular functions to induce difficulty scores in curriculum learning. We reinterpret adaptive subset selection and formulate it as a multi-armed bandit problem, where each arm corresponds to a submodular function guiding sample selection. We introduce ONLINESUBMOD, a novel online greedy policy that optimizes a utility-driven reward and provably achieves no-regret performance under various sampling regimes. Empirically, ONLINESUBMOD outperforms both traditional curriculum learning and bi-level optimization approaches across vision and language datasets, showing superior accuracy-efficiency tradeoffs. More broadly, we show that validationdriven reward metrics offer a principled way to guide the curriculum schedule.
comment : 10 pages main, 21 pages Appendix, 8 figures
☆ EnECG: Efficient Ensemble Learning for Electrocardiogram Multi-task Foundation Model
Yuhao Xu, Xiaoda Wang, Jiaying Lu, Sirui Ding, Defu Cao, Huaxiu Yao, Yan Liu, Xiao Hu, Carl Yang
Electrocardiogram (ECG) analysis plays a vital role in the early detection, monitoring, and management of various cardiovascular conditions. While existing models have achieved notable success in ECG interpretation, they fail to leverage the interrelated nature of various cardiac abnormalities. Conversely, developing a specific model capable of extracting all relevant features for multiple ECG tasks remains a significant challenge. Large-scale foundation models, though powerful, are not typically pretrained on ECG data, making full re-training or fine-tuning computationally expensive. To address these challenges, we propose EnECG(Mixture of Experts-based Ensemble Learning for ECG Multi-tasks), an ensemble-based framework that integrates multiple specialized foundation models, each excelling in different aspects of ECG interpretation. Instead of relying on a single model or single task, EnECG leverages the strengths of multiple specialized models to tackle a variety of ECG-based tasks. To mitigate the high computational cost of full re-training or fine-tuning, we introduce a lightweight adaptation strategy: attaching dedicated output layers to each foundation model and applying Low-Rank Adaptation (LoRA) only to these newly added parameters. We then adopt a Mixture of Experts (MoE) mechanism to learn ensemble weights, effectively combining the complementary expertise of individual models. Our experimental results demonstrate that by minimizing the scope of fine-tuning, EnECG can help reduce computational and memory costs while maintaining the strong representational power of foundation models. This framework not only enhances feature extraction and predictive performance but also ensures practical efficiency for real-world clinical applications. The code is available at https://github.com/yuhaoxu99/EnECG.git.
☆ Agent Shield: Make MAS more secure and efficient
Kaixiang Wang, Zhaojiacheng Zhou, Bunyod Suvonov, Jiong Lou, Jie LI
Large Language Model (LLM)-based Multi-Agent Systems (MAS) offer powerful cooperative reasoning but remain vulnerable to adversarial attacks, where compromised agents can undermine the system's overall performance. Existing defenses either depend on single trusted auditors, creating single points of failure, or sacrifice efficiency for robustness. To resolve this tension, we propose \textbf{AgentShield}, a distributed framework for efficient, decentralized auditing. AgentShield introduces a novel three-layer defense: \textbf{(i) Critical Node Auditing} prioritizes high-influence agents via topological analysis; \textbf{(ii) Light Token Auditing} implements a cascade protocol using lightweight sentry models for rapid discriminative verification; and \textbf{(iii) Two-Round Consensus Auditing} triggers heavyweight arbiters only upon uncertainty to ensure global agreement. This principled design optimizes the robustness-efficiency trade-off. Experiments demonstrate that AgentShield achieves a 92.5\% recovery rate and reduces auditing overhead by over 70\% compared to existing methods, maintaining high collaborative accuracy across diverse MAS topologies and adversarial scenarios.
☆ MICCAI STS 2024 Challenge: Semi-Supervised Instance-Level Tooth Segmentation in Panoramic X-ray and CBCT Images
Yaqi Wang, Zhi Li, Chengyu Wu, Jun Liu, Yifan Zhang, Jiaxue Ni, Qian Luo, Jialuo Chen, Hongyuan Zhang, Jin Liu, Can Han, Kaiwen Fu, Changkai Ji, Xinxu Cai, Jing Hao, Zhihao Zheng, Shi Xu, Junqiang Chen, Qianni Zhang, Dahong Qian, Shuai Wang, Huiyu Zhou
Orthopantomogram (OPGs) and Cone-Beam Computed Tomography (CBCT) are vital for dentistry, but creating large datasets for automated tooth segmentation is hindered by the labor-intensive process of manual instance-level annotation. This research aimed to benchmark and advance semi-supervised learning (SSL) as a solution for this data scarcity problem. We organized the 2nd Semi-supervised Teeth Segmentation (STS 2024) Challenge at MICCAI 2024. We provided a large-scale dataset comprising over 90,000 2D images and 3D axial slices, which includes 2,380 OPG images and 330 CBCT scans, all featuring detailed instance-level FDI annotations on part of the data. The challenge attracted 114 (OPG) and 106 (CBCT) registered teams. To ensure algorithmic excellence and full transparency, we rigorously evaluated the valid, open-source submissions from the top 10 (OPG) and top 5 (CBCT) teams, respectively. All successful submissions were deep learning-based SSL methods. The winning semi-supervised models demonstrated impressive performance gains over a fully-supervised nnU-Net baseline trained only on the labeled data. For the 2D OPG track, the top method improved the Instance Affinity (IA) score by over 44 percentage points. For the 3D CBCT track, the winning approach boosted the Instance Dice score by 61 percentage points. This challenge confirms the substantial benefit of SSL for complex, instance-level medical image segmentation tasks where labeled data is scarce. The most effective approaches consistently leveraged hybrid semi-supervised frameworks that combined knowledge from foundational models like SAM with multi-stage, coarse-to-fine refinement pipelines. Both the challenge dataset and the participants' submitted code have been made publicly available on GitHub (https://github.com/ricoleehduu/STS-Challenge-2024), ensuring transparency and reproducibility.
☆ Leveraging Textual Compositional Reasoning for Robust Change Captioning AAAI 2026
Kyu Ri Park, Jiyoung Park, Seong Tae Kim, Hong Joo Lee, Jung Uk Kim
Change captioning aims to describe changes between a pair of images. However, existing works rely on visual features alone, which often fail to capture subtle but meaningful changes because they lack the ability to represent explicitly structured information such as object relationships and compositional semantics. To alleviate this, we present CORTEX (COmpositional Reasoning-aware TEXt-guided), a novel framework that integrates complementary textual cues to enhance change understanding. In addition to capturing cues from pixel-level differences, CORTEX utilizes scene-level textual knowledge provided by Vision Language Models (VLMs) to extract richer image text signals that reveal underlying compositional reasoning. CORTEX consists of three key modules: (i) an Image-level Change Detector that identifies low-level visual differences between paired images, (ii) a Reasoning-aware Text Extraction (RTE) module that use VLMs to generate compositional reasoning descriptions implicit in visual features, and (iii) an Image-Text Dual Alignment (ITDA) module that aligns visual and textual features for fine-grained relational reasoning. This enables CORTEX to reason over visual and textual features and capture changes that are otherwise ambiguous in visual features alone.
comment : Accepted at AAAI 2026
☆ Switching-time bioprocess control with pulse-width-modulated optogenetics
Sebastián Espinel-Ríos
Biotechnology can benefit from dynamic control to improve production efficiency. In this context, optogenetics enables modulation of gene expression using light as an external input, allowing fine-tuning of protein levels to unlock dynamic metabolic control and regulation of cell growth. Optogenetic systems can be actuated by light intensity. However, relying solely on intensity-driven control (i.e., signal amplitude) may fail to properly tune optogenetic bioprocesses when the dose-response relationship (i.e., light intensity versus gene-expression strength) is steep. In these cases, tunability is effectively constrained to either fully active or fully repressed gene expression, with little intermediate regulation. Pulse-width modulation, a concept widely used in electronics, can alleviate this issue by alternating between fully ON and OFF light intensity within forcing periods, thereby smoothing the average response and enhancing process controllability. Naturally, optimizing pulse-width-modulated optogenetics entails a switching-time optimal control problem with a binary input over many forcing periods. While this can be formulated as a mixed-integer program on a refined time grid, the number of decision variables can grow rapidly with increasing time-grid resolution and number of forcing periods, compromising tractability. Here, we propose an alternative solution based on reinforcement learning. We parametrize control actions via the duty cycle, a continuous variable that encodes the ON-to-OFF switching time within each forcing period, thereby respecting the intrinsic binary nature of the light intensity.
comment : Submitted conference paper
☆ ORION: Teaching Language Models to Reason Efficiently in the Language of Thought
Kumar Tanmay, Kriti Aggarwal, Paul Pu Liang, Subhabrata Mukherjee
Large Reasoning Models (LRMs) achieve strong performance in mathematics, code generation, and task planning, but their reliance on long chains of verbose "thinking" tokens leads to high latency, redundancy, and incoherent reasoning paths. Inspired by the Language of Thought Hypothesis, which posits that human reasoning operates over a symbolic, compositional mental language called Mentalese, we introduce a framework that trains models to reason in a similarly compact style. Mentalese encodes abstract reasoning as ultra-compressed, structured tokens, enabling models to solve complex problems with far fewer steps. To improve both efficiency and accuracy, we propose SHORTER LENGTH PREFERENCE OPTIMIZATION (SLPO), a reinforcement learning method that rewards concise solutions that stay correct, while still allowing longer reasoning when needed. Applied to Mentalese-aligned models, SLPO yields significantly higher compression rates by enabling concise reasoning that preserves the benefits of detailed thinking without the computational overhead. Across benchmarks including AIME 2024 and 2025, MinervaMath, OlympiadBench, Math500, and AMC, our ORION models produce reasoning traces with 4-16x fewer tokens, achieve up to 5x lower inference latency, and reduce training costs by 7-9x relative to the DeepSeek R1 Distilled model, while maintaining 90-98% of its accuracy. ORION also surpasses Claude and ChatGPT-4o by up to 5% in accuracy while maintaining 2x compression. These results show that Mentalese-style compressed reasoning offers a step toward human-like cognitive efficiency, enabling real-time, cost-effective reasoning without sacrificing accuracy.
☆ Adversarial Training for Process Reward Model s
Gurusha Juneja, Deepak Nathani, William Yang Wang
Process Reward Models (PRMs) enhance reasoning ability of LLMs by providing step-level supervision. However, their widespread adoption is limited due to expensive manual step-level annotation and poor generalization of static training data to novel errors. We introduce Adversarially Trained PRMs (\texttt{APRM}), where a Generator ($G$) learns to produce reasoning errors to deceive a PRM ($R$), while $R$ concurrently learns to detect them. This interaction yields progressively harder negatives for $R$, improving its robustness and generalization to novel errors without requiring manual step-level labels. Averaged across diverse mathematical reasoning benchmarks, \texttt{APRM} improves solver accuracy by $+3.4$ percentage points (pp) over the strongest PRM baseline. \texttt{APRM} achieves gains of $+5.3$ pp on out-of-distribution tasks.
☆ InsightEval: An Expert-Curated Benchmark for Assessing Insight Discovery in LLM-Driven Data Agent s
Zhenghao Zhu, Yuanfeng Song, Xin Chen, Chengzhong Liu, Yakun Cui, Caleb Chen Cao, Sirui Han, Yike Guo
Data analysis has become an indispensable part of scientific research. To discover the latent knowledge and insights hidden within massive datasets, we need to perform deep exploratory analysis to realize their full value. With the advent of large language models (LLMs) and multi-agent systems, more and more researchers are making use of these technologies for insight discovery. However, there are few benchmarks for evaluating insight discovery capabilities. As one of the most comprehensive existing frameworks, InsightBench also suffers from many critical flaws: format inconsistencies, poorly conceived objectives, and redundant insights. These issues may significantly affect the quality of data and the evaluation of agents. To address these issues, we thoroughly investigate shortcomings in InsightBench and propose essential criteria for a high-quality insight benchmark. Regarding this, we develop a data-curation pipeline to construct a new dataset named InsightEval. We further introduce a novel metric to measure the exploratory performance of agents. Through extensive experiments on InsightEval, we highlight prevailing challenges in automated insight discovery and raise some key findings to guide future research in this promising direction.
☆ Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems
Shashwat Jaiswal, Shrikara Arun, Anjaly Parayil, Ankur Mallick, Spyros Mastorakis, Alind Khare, Chloi Alverti, Renee St Amant, Chetan Bansal, Victor Rühle, Josep Torrellas
Low-Rank Adaptation (LoRA) has become the de facto method for parameter-efficient fine-tuning of large language models (LLMs), enabling rapid adaptation to diverse domains. In production, LoRA-based models are served at scale, creating multi-tenant environments with hundreds of adapters sharing a base model. However, state-of-the-art serving systems co-batch heterogeneous adapters without accounting for rank (size) variability, leading to severe performance skew, which ultimately requires adding more GPUs to satisfy service-level objectives (SLOs). Existing optimizations, focused on loading, caching, and kernel execution, ignore this heterogeneity, leaving GPU resources underutilized. We present LoRAServe, a workload-aware dynamic adapter placement and routing framework designed to tame rank diversity in LoRA serving. By dynamically rebalancing adapters across GPUs and leveraging GPU Direct RDMA for remote access, LoRAServe maximizes throughput and minimizes tail latency under real-world workload drift. Evaluations on production traces from Company X show that LoRAServe elicits up to 2$\times$ higher throughput, up to 9$\times$ lower TTFT, while using up to 50% fewer GPUs under SLO constraints compared to state-of-the-art systems.
☆ Escaping Barren Plateaus in Variational Quantum Algorithms Using Negative Learning Rate in Quantum Internet of Things
Ratun Rahman, Dinh C. Nguyen
Variational Quantum Algorithms (VQAs) are becoming the primary computational primitive for next-generation quantum computers, particularly those embedded as resource-constrained accelerators in the emerging Quantum Internet of Things (QIoT). However, under such device-constrained execution conditions, the scalability of learning is severely limited by barren plateaus, where gradients collapse to zero and training stalls. This poses a practical challenge to delivering VQA-enabled intelligence on QIoT endpoints, which often have few qubits, constrained shot budgets, and strict latency requirements. In this paper, we present a novel approach for escaping barren plateaus by including negative learning rates into the optimization process in QIoT devices. Our method introduces controlled instability into model training by switching between positive and negative learning phases, allowing recovery of significant gradients and exploring flatter areas in the loss landscape. We theoretically evaluate the effect of negative learning on gradient variance and propose conditions under which it helps escape from barren zones. The experimental findings on typical VQA benchmarks show consistent improvements in both convergence and simulation results over traditional optimizers. By escaping barren plateaus, our approach leads to a novel pathway for robust optimization in quantum-classical hybrid models.
comment : Accepted at IEEE Internet of Things Journal
☆ CausalProfiler: Generating Synthetic Benchmark s for Rigorous and Transparent Evaluation of Causal Machine Learning
Panayiotis Panayiotou, Audrey Poinsot, Alessandro Leite, Nicolas Chesneau, Marc Schoenauer, Özgür Şimşek
Causal machine learning (Causal ML) aims to answer "what if" questions using machine learning algorithms, making it a promising tool for high-stakes decision-making. Yet, empirical evaluation practices in Causal ML remain limited. Existing benchmarks often rely on a handful of hand-crafted or semi-synthetic datasets, leading to brittle, non-generalizable conclusions. To bridge this gap, we introduce CausalProfiler, a synthetic benchmark generator for Causal ML methods. Based on a set of explicit design choices about the class of causal models, queries, and data considered, the CausalProfiler randomly samples causal models, data, queries, and ground truths constituting the synthetic causal benchmarks. In this way, Causal ML methods can be rigorously and transparently evaluated under a variety of conditions. This work offers the first random generator of synthetic causal benchmarks with coverage guarantees and transparent assumptions operating on the three levels of causal reasoning: observation, intervention, and counterfactual. We demonstrate its utility by evaluating several state-of-the-art methods under diverse conditions and assumptions, both in and out of the identification regime, illustrating the types of analyses and insights the CausalProfiler enables.
☆ Fast dynamical similarity analysis
Arman Behrad, Mitchell Ostrow, Mohammad Taha Fakharian, Ila Fiete, Christian Beste, Shervin Safavi
To understand how neural systems process information, it is often essential to compare one circuit with another, one brain with another, or data with a model. Traditional similarity measures ignore the dynamical processes underlying neural representations. Dynamical similarity methods offer a framework to compare the temporal structure of dynamical systems by embedding their (possibly) nonlinear dynamics into a globally linear space and there computing conjugacy metrics. However, identifying the best embedding and computing these metrics can be computationally slow. Here we introduce fast Dynamical Similarity Analysis (fastDSA), which is computationally far more efficient than previous methods while maintaining their accuracy and robustness. FastDSA introduces two key components that boost efficiency: (1) automatic selection of the effective model order of the Hankel (delay) embedding from the data via a data-driven singular-value threshold that identifies the informative subspace and discards noise to lower computational cost without sacrificing signal, and (2) a novel optimization procedure and objective, which replaces the slow exact orthogonality constraint in finding a minimal distance between dynamics matrices with a lightweight process to keep the search close to the space of orthogonal transformations. We demonstrate that fastDSA is at least an order of magnitude faster than the previous methods. Furthermore, we demonstrate that fastDSA has the properties of its ancestor, including its invariances and sensitivities to system dynamics. FastDSA, therefore, provides a computationally efficient and accurate method for dynamical similarity analysis.
☆ A Unified and Stable Risk Minimization Framework for Weakly Supervised Learning with Theoretical Guarantees
Miao Zhang, Junpeng Li, Changchun Hua, Yana Yang
Weakly supervised learning has emerged as a practical alternative to fully supervised learning when complete and accurate labels are costly or infeasible to acquire. However, many existing methods are tailored to specific supervision patterns -- such as positive-unlabeled (PU), unlabeled-unlabeled (UU), complementary-label (CLL), partial-label (PLL), or similarity-unlabeled annotations -- and rely on post-hoc corrections to mitigate instability induced by indirect supervision. We propose a principled, unified framework that bypasses such post-hoc adjustments by directly formulating a stable surrogate risk grounded in the structure of weakly supervised data. The formulation naturally subsumes diverse settings -- including PU, UU, CLL, PLL, multi-class unlabeled, and tuple-based learning -- under a single optimization objective. We further establish a non-asymptotic generalization bound via Rademacher complexity that clarifies how supervision structure, model capacity, and sample size jointly govern performance. Beyond this, we analyze the effect of class-prior misspecification on the bound, deriving explicit terms that quantify its impact, and we study identifiability, giving sufficient conditions -- most notably via supervision stratification across groups -- under which the target risk is recoverable. Extensive experiments show consistent gains across class priors, dataset scales, and class counts -- without heuristic stabilization -- while exhibiting robustness to overfitting.
☆ A Hierarchical Hybrid AI Approach: Integrating Deep Reinforcement Learning and Scripted Agent s in Combat Simulations
Scotty Black, Christian Darken
In the domain of combat simulations in support of wargaming, the development of intelligent agents has predominantly been characterized by rule-based, scripted methodologies with deep reinforcement learning (RL) approaches only recently being introduced. While scripted agents offer predictability and consistency in controlled environments, they fall short in dynamic, complex scenarios due to their inherent inflexibility. Conversely, RL agents excel in adaptability and learning, offering potential improvements in handling unforeseen situations, but suffer from significant challenges such as black-box decision-making processes and scalability issues in larger simulation environments. This paper introduces a novel hierarchical hybrid artificial intelligence (AI) approach that synergizes the reliability and predictability of scripted agents with the dynamic, adaptive learning capabilities of RL. By structuring the AI system hierarchically, the proposed approach aims to utilize scripted agents for routine, tactical-level decisions and RL agents for higher-level, strategic decision-making, thus addressing the limitations of each method while leveraging their individual strengths. This integration is shown to significantly improve overall performance, providing a robust, adaptable, and effective solution for developing and training intelligent agents in complex simulation environments.
comment : arXiv admin note: substantial text overlap with arXiv:2408.13333
☆ Optimizing Information Asset Investment Strategies in the Exploratory Phase of the Oil and Gas Industry: A Reinforcement Learning Approach
Paulo Roberto de Melo Barros Junior, Monica Alexandra Vilar Ribeiro De Meireles, Jose Luis Lima de Jesus Silva
Our work investigates the economic efficiency of the prevailing "ladder-step" investment strategy in oil and gas exploration, which advocates for the incremental acquisition of geological information throughout the project lifecycle. By employing a multi-agent Deep Reinforcement Learning (DRL) framework, we model an alternative strategy that prioritizes the early acquisition of high-quality information assets. We simulate the entire upstream value chain-comprising competitive bidding, exploration, and development phases-to evaluate the economic impact of this approach relative to traditional methods. Our results demonstrate that front-loading information investment significantly reduces the costs associated with redundant data acquisition and enhances the precision of reserve valuation. Specifically, we find that the alternative strategy outperforms traditional methods in highly competitive environments by mitigating the "winner's curse" through more accurate bidding. Furthermore, the economic benefits are most pronounced during the development phase, where superior data quality minimizes capital misallocation. These findings suggest that optimal investment timing is structurally dependent on market competition rather than solely on price volatility, offering a new paradigm for capital allocation in extractive industries.
☆ Polynomial Neural Sheaf Diffusion : A Spectral Filtering Approach on Cellular Sheaves
Alessio Borgi, Fabrizio Silvestri, Pietro Liò
Sheaf Neural Networks equip graph structures with a cellular sheaf: a geometric structure which assigns local vector spaces (stalks) and a linear learnable restriction/transport maps to nodes and edges, yielding an edge-aware inductive bias that handles heterophily and limits oversmoothing. However, common Neural Sheaf Diffusion implementations rely on SVD-based sheaf normalization and dense per-edge restriction maps, which scale with stalk dimension, require frequent Laplacian rebuilds, and yield brittle gradients. To address these limitations, we introduce Polynomial Neural Sheaf Diffusion (PolyNSD), a new sheaf diffusion approach whose propagation operator is a degree-K polynomial in a normalised sheaf Laplacian, evaluated via a stable three-term recurrence on a spectrally rescaled operator. This provides an explicit K-hop receptive field in a single layer (independently of the stalk dimension), with a trainable spectral response obtained as a convex mixture of K+1 orthogonal polynomial basis responses. PolyNSD enforces stability via convex mixtures, spectral rescaling, and residual/gated paths, reaching new state-of-the-art results on both homophilic and heterophilic benchmarks, inverting the Neural Sheaf Diffusion trend by obtaining these results with just diagonal restriction maps, decoupling performance from large stalk dimension, while reducing runtime and memory requirements.
☆ OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusion ACL 2026
Sai Koneru, Matthias Huck, Jan Niehues
There has been significant progress in open-source text-only translation large language models (LLMs) with better language coverage and quality. However, these models can be only used in cascaded pipelines for speech translation (ST), performing automatic speech recognition first followed by translation. This introduces additional latency, which is particularly critical in simultaneous ST (SimulST), and prevents the model from exploiting multimodal context, such as images, which can aid disambiguation. Pretrained multimodal foundation models (MMFMs) already possess strong perception and reasoning capabilities across multiple modalities, but generally lack the multilingual coverage and specialized translation performance of dedicated translation LLMs. To build an effective multimodal translation system, we propose an end-to-end approach that fuses MMFMs with translation LLMs. We introduce a novel fusion strategy that connects hidden states from multiple layers of a pretrained MMFM to a translation LLM, enabling joint end-to-end training. The resulting model, OmniFusion, built on Omni 2.5-7B as the MMFM and SeedX PPO-7B as the translation LLM, can perform speech-to-text, speech-and-image-to-text, and text-and-image-to-text translation. Experiments demonstrate that OmniFusion effectively leverages both audio and visual inputs, achieves a 1-second latency reduction in SimulST compared to cascaded pipelines and also improves the overall translation quality\footnote{Code is available at https://github.com/saikoneru/OmniFusion}.
comment : Preprint for ACL 2026
☆ CodeFlowLM: Incremental Just-In-Time Defect Prediction with Pretrain ed Language Models and Exploratory Insights into Defect Localization
Monique Louise Monteiro, George G. Cabral, Adriano L. I. OLiveira
This work introduces CodeFlowLM, an incremental learning framework for Just-In-Time Software Defect Prediction (JIT-SDP) that leverages pre-trained language models (PLMs). Unlike traditional online learners, CodeFlowLM employs continual fine-tuning to address concept drift, class imbalance, and verification latency without retraining from scratch. We evaluated encoder-only and encoder-decoder PLMs (notably CodeT5+ and UniXCoder) in JIT-SDP scenarios within and between projects, comparing them with the incremental baseline BORB. The results show that CodeFlowLM achieves up to 68% G-Mean gains, confirming its superior adaptability and robustness in evolving software environments. We further extend the analysis to Just-in-Time Defect Localization (JIT-DL), benchmarking Large Language Models (LLMs) such as GPT-5, Claude Sonnet 4.5, and Gemini 2.5 Pro against attention-based models. GPT-5 delivers comparable performance for Recall@20% and Effort@20% with higher stability, although attention-based methods retain an advantage in fine-grained ranking metrics (Top-k, IFA). A qualitative error analysis reveals that most false positives arise from (1) human-like conservative bias, (2) insufficient contextual information in diff-based prompts, and (3) potential dataset mislabeling in JIT-Defects4J. These findings highlight both the promise and the current limitations of LLM reasoning in defect localization. False negatives occur in smaller proportions. Overall, CodeFlowLM significantly advances the state of the art in incremental JIT-SDP, demonstrating superior adaptability and robustness in evolving software environments. Furthermore, our exploratory analysis of LLMs in JIT-DL not only benchmarks their performance against established attention-based models but also provides critical insights into the current limitations of prompt-based defect reasoning.
☆ DenseScan: Advancing 3D Scene Understanding with 2D Dense Annotation NeurIPS 2025
Zirui Wang, Tao Zhang
3D understanding is a key capability for real-world AI assistance. High-quality data plays an important role in driving the development of the 3D understanding community. Current 3D scene understanding datasets often provide geometric and instance-level information, yet they lack the rich semantic annotations necessary for nuanced visual-language tasks.In this work, we introduce DenseScan, a novel dataset with detailed multi-level descriptions generated by an automated pipeline leveraging multi-view 2D images and multimodal large language models (MLLMs). Our approach enables dense captioning of scene elements, ensuring comprehensive object-level descriptions that capture context-sensitive details. Furthermore, we extend these annotations through scenario-based question generation, producing high-level queries that integrate object properties, spatial relationships, and scene context. By coupling geometric detail with semantic richness, DenseScan broadens the range of downstream tasks, from detailed visual-language navigation to interactive question answering. Experimental results demonstrate that our method significantly enhances object-level understanding and question-answering performance in 3D environments compared to traditional annotation pipelines. We release both the annotated dataset and our annotation pipeline to facilitate future research and applications in robotics, augmented reality, and beyond. Through DenseScan, we aim to catalyze new avenues in 3D scene understanding, allowing researchers and practitioners to tackle the complexities of real-world environments with richer, more contextually aware annotations.
comment : Workshop on Space in Vision, Language, and Embodied AI at NeurIPS 2025
☆ Reasoning Under Pressure: How do Training Incentives Influence Chain-of-Thought Monitorability?
Matt MacDermott, Qiyao Wei, Rada Djoneva, Francis Rhys Ward
AI systems that output their reasoning in natural language offer an opportunity for safety -- we can \emph{monitor} their chain of thought (CoT) for undesirable reasoning, such as the pursuit of harmful objectives. However, the extent to which CoT faithfully reflects the underlying reasoning process, and hence the extent to which it can be usefully monitored, may be influenced by certain aspects of training. We investigate how different \emph{training incentives}, applied to a reasoning model, affect its monitorability. We introduce a novel methodology for measuring monitorability according to whether a monitor can predict a key latent variable using the model's reasoning. When controlling for accuracy, we do not find evidence for consistent effects from commonly used incentives (length penalties and KL regularisation), but we find that adversarial optimisation (penalising monitor accuracy) degrades monitor performance, while direct optimisation for monitorability does not reliably lead to improvements. Our code is available at https://github.com/QiyaoWei/reasoning-under-pressure.
☆ On the Prediction of Wi-Fi Performance through Deep Learning
Gabriele Formis, Amanda Ericson, Stefan Forsstrom, Kyi Thar, Gianluca Cena, Stefano Scanzio
Ensuring reliable and predictable communications is one of the main goals in modern industrial systems that rely on Wi-Fi networks, especially in scenarios where continuity of operation and low latency are required. In these contexts, the ability to predict changes in wireless channel quality can enable adaptive strategies and significantly improve system robustness. This contribution focuses on the prediction of the Frame Delivery Ratio (FDR), a key metric that represents the percentage of successful transmissions, starting from time sequences of binary outcomes (success/failure) collected in a real scenario. The analysis focuses on two models of deep learning: a Convolutional Neural Network (CNN) and a Long Short-Term Memory network (LSTM), both selected for their ability to predict the outcome of time sequences. Models are compared in terms of prediction accuracy and computational complexity, with the aim of evaluating their applicability to systems with limited resources. Preliminary results show that both models are able to predict the evolution of the FDR with good accuracy, even from minimal information (a single binary sequence). In particular, CNN shows a significantly lower inference latency, with a marginal loss in accuracy compared to LSTM.
comment : preprint accepted, 4 pages, 2025
☆ Constructing Efficient Fact-Storing MLPs for Transformer s
Owen Dugan, Roberto Garcia, Ronny Junkins, Jerry Liu, Dylan Zinsley, Sabri Eyuboglu, Atri Rudra, Chris Ré
The success of large language models (LLMs) can be attributed in part to their ability to efficiently store factual knowledge as key-value mappings within their MLP parameters. Recent work has proposed explicit weight constructions to build such fact-storing MLPs, providing an improved understanding of LLM fact storage mechanisms. In this paper, we introduce an MLP construction framework that improves over previous constructions in three areas: it 1) works for all but a measure-zero set of feasible input-output pairs, 2) achieves asymptotically optimal parameter efficiency matching information-theoretic bounds for some embeddings, and 3) maintains usability within Transformers for factual recall. Through our improvements, we 1) discover a metric on value embeddings that characterizes facts-per-parameter scaling for both constructed and gradient-descent-trained MLPs, 2) identify a simple encoder-decoder mechanism that empirically matches gradient-descent MLP facts-per-parameter asymptotics across all the inputs and outputs we test, and 3) uncover a fundamental tradeoff between an MLP's fact-storage capacity and its usability within Transformers. Finally, we demonstrate a proof-of-concept application of fact-storing MLPs: modular fact editing on one-layer Transformers by \textit{replacing entire MLPs at once}.
☆ Tree Matching Networks for Natural Language Inference: Parameter-Efficient Semantic Understanding via Dependency Parse Trees
Jason Lunder
In creating sentence embeddings for Natural Language Inference (NLI) tasks, using transformer-based models like BERT leads to high accuracy, but require hundreds of millions of parameters. These models take in sentences as a sequence of tokens, and learn to encode the meaning of the sequence into embeddings such that those embeddings can be used reliably for NLI tasks. Essentially, every word is considered against every other word in the sequence, and the transformer model is able to determine the relationships between them, entirely from scratch. However, a model that accepts explicit linguistic structures like dependency parse trees may be able to leverage prior encoded information about these relationships, without having to learn them from scratch, thus improving learning efficiency. To investigate this, we adapt Graph Matching Networks (GMN) to operate on dependency parse trees, creating Tree Matching Networks (TMN). We compare TMN to a BERT based model on the SNLI entailment task and on the SemEval similarity task. TMN is able to achieve significantly better results with a significantly reduced memory footprint and much less training time than the BERT based model on the SNLI task, while both models struggled to preform well on the SemEval. Explicit structural representations significantly outperform sequence-based models at comparable scales, but current aggregation methods limit scalability. We propose multi-headed attention aggregation to address this limitation.
comment : 16 pages, preprint
☆ A Rosetta Stone for AI Benchmark s
Anson Ho, Jean-Stanislas Denain, David Atanasov, Samuel Albanie, Rohin Shah
Most AI benchmarks saturate within years or even months after they are introduced, making it hard to study long-run trends in AI capabilities. To address this challenge, we build a statistical framework that stitches benchmarks together, putting model capabilities and benchmark difficulties on a single numerical scale. This acts as a "Rosetta Stone", allowing us to compare models across a wide range of abilities and time, even if they are not evaluated on the same benchmarks. Moreover, this works without assuming how capabilities evolve across time or with training compute. We demonstrate three applications of this framework. First, we use it to measure the speed of AI progress over time, and to forecast future AI capabilities. Second, we estimate the rate of improvements in algorithmic efficiency, finding estimates that are higher, but broadly consistent with prior work. Finally, we find that our approach can be used to detect rapid accelerations in AI progress.
☆ Chunking Strategies for Multimodal AI Systems
Shashanka B R, Mohith Charan R, Seema Banu F
Our goal is to consolidate the landscape of multimodal chunking strategies, providing researchers and practitioners with a technical foundation and design space for developing more effective and efficient multimodal AI systems. This survey paves the way for innovations in robust chunking pipelines that scale with modality complexity, enhance processing accuracy, and improve generative coherence in real-world applications. This survey provides a comprehensive taxonomy and technical analysis of chunking strategies tailored for each modality: text, images, audio, video, and cross-modal data. We examine classical and modern approaches such as fixed-size token windowing, recursive text splitting, object-centric visual chunking, silence-based audio segmentation, and scene detection in videos. Each approach is analyzed in terms of its underlying methodology, supporting tools (e.g., LangChain, Detectron2, PySceneDetect), benefits, and challenges, particularly those related to granularity-context trade-offs and multimodal alignment. Furthermore, we explore emerging cross-modal chunking strategies that aim to preserve alignment and semantic consistency across disparate data types [4]. We also include comparative insights, highlight open problems such as asynchronous information density and noisy alignment signals, and identify opportunities for future research in adaptive, learning-based, and task-specific chunking.
comment : 45 pages, 5 figure
☆ Orion-Bix: Bi-Axial Attention for Tabular In- Context Learning
Mohamed Bouadi, Pratinav Seth, Aditya Tanna, Vinay Kumar Sankarapu
Tabular data drive most real-world machine learning applications, yet building general-purpose models for them remains difficult. Mixed numeric and categorical fields, weak feature structure, and limited labeled data make scaling and generalization challenging. To this end, we introduce Orion-Bix, a tabular foundation model that combines biaxial attention with meta-learned in-context reasoning for few-shot tabular learning. Its encoder alternates standard, grouped, hierarchical, and relational attention, fusing their outputs through multi-CLS summarization to capture both local and global dependencies efficiently. A label-aware ICL head adapts on the fly and scales to large label spaces via hierarchical decision routing. Meta-trained on synthetically generated, structurally diverse tables with causal priors, Orion-Bix learns transferable inductive biases across heterogeneous data. Delivered as a scikit-learn compatible foundation model, it outperforms gradient-boosting baselines and remains competitive with state-of-the-art tabular foundation models on public benchmarks, showing that biaxial attention with episodic meta-training enables robust, few-shot-ready tabular learning. The model is publicly available at https://github.com/Lexsi-Labs/Orion-BiX .
☆ Efficient Edge-Compatible CNN for Speckle-Based Material Recognition in Laser Cutting Systems
Mohamed Abdallah Salem, Nourhan Zein Diab
Accurate material recognition is critical for safe and effective laser cutting, as misidentification can lead to poor cut quality, machine damage, or the release of hazardous fumes. Laser speckle sensing has recently emerged as a low-cost and non-destructive modality for material classification; however, prior work has either relied on computationally expensive backbone networks or addressed only limited subsets of materials. In this study, A lightweight convolutional neural network (CNN) tailored for speckle patterns is proposed, designed to minimize parameters while maintaining high discriminative power. Using the complete SensiCut dataset of 59 material classes spanning woods, acrylics, composites, textiles, metals, and paper-based products, the proposed model achieves 95.05% test accuracy, with macro and weighted F1-scores of 0.951. The network contains only 341k trainable parameters (~1.3 MB) -- over 70X fewer than ResNet-50 -- and achieves an inference speed of 295 images per second, enabling deployment on Raspberry Pi and Jetson-class devices. Furthermore, when materials are regrouped into nine and five practical families, recall exceeds 98% and approaches 100%, directly supporting power and speed preset selection in laser cutters. These results demonstrate that compact, domain-specific CNNs can outperform large backbones for speckle-based material classification, advancing the feasibility of material-aware, edge-deployable laser cutting systems.
comment : Copyright 2025 IEEE. This is the author's version of the work that has been Accepted for publication in the Proceedings of the 2025 IEEE The 35th International Conference on Computer Theory and Applications (ICCTA 2025). Final published version will be available on IEEE Xplore
☆ Tuning Universality in Deep Neural Networks
Arsham Ghavasieh
Deep neural networks (DNNs) exhibit crackling-like avalanches whose origin lacks a mechanistic explanation. Here, I derive a stochastic theory of deep information propagation (DIP) by incorporating Central Limit Theorem (CLT)-level fluctuations. Four effective couplings $(r, h, D_1, D_2)$ characterize the dynamics, yielding a Landau description of the static exponents and a Directed Percolation (DP) structure of activity cascades. Tuning the couplings selects between avalanche dynamics generated by a Brownian Motion (BM) in a logarithmic trap and an absorbed free BM, each corresponding to a distinct universality classes. Numerical simulations confirm the theory and demonstrate that activation function design controls the collective dynamics in random DNNs.
☆ DeFi TrustBoost: Blockchain and AI for Trustworthy Decentralized Financial Decisions
Swati Sachan, Dale S. Fickett
This research introduces the Decentralized Finance (DeFi) TrustBoost Framework, which combines blockchain technology and Explainable AI to address challenges faced by lenders underwriting small business loan applications from low-wealth households. The framework is designed with a strong emphasis on fulfilling four crucial requirements of blockchain and AI systems: confidentiality, compliance with data protection laws, resistance to adversarial attacks, and compliance with regulatory audits. It presents a technique for tamper-proof auditing of automated AI decisions and a strategy for on-chain (inside-blockchain) and off-chain data storage to facilitate collaboration within and across financial organizations.
comment : 19 pages
♻ ☆ NegBLEURT Forest: Leveraging Inconsistencies for Detecting Jailbreak Attacks
Lama Sleem, Jerome Francois, Lujun Li, Nathan Foucher, Niccolo Gentile, Radu State
Jailbreak attacks designed to bypass safety mechanisms pose a serious threat by prompting LLMs to generate harmful or inappropriate content, despite alignment with ethical guidelines. Crafting universal filtering rules remains difficult due to their inherent dependence on specific contexts. To address these challenges without relying on threshold calibration or model fine-tuning, this work introduces a semantic consistency analysis between successful and unsuccessful responses, demonstrating that a negation-aware scoring approach captures meaningful patterns. Building on this insight, a novel detection framework called NegBLEURT Forest is proposed to evaluate the degree of alignment between outputs elicited by adversarial prompts and expected safe behaviors. It identifies anomalous responses using the Isolation Forest algorithm, enabling reliable jailbreak detection. Experimental results show that the proposed method consistently achieves top-tier performance, ranking first or second in accuracy across diverse models using the crafted dataset, while competing approaches exhibit notable sensitivity to model and data variations.
comment : This paper has been accepted in IEEE Consumer Communications & Networking Conference 2026
♻ ☆ Uncovering Zero-Shot Generalization Gaps in Time-Series Foundation Models Using Real-Wo rl d Videos AAAI 2026
Lujun Li, Lama Sleem, Yiqun Wang, Yangjie Xu, Niccolò Gentile, Radu State
Recent research on time-series foundation models (TSFMs) has underscored the scarcity of real-world data, often supplemented with synthetic sources in existing datasets, whose generalizability remains however debated. As such, in this work, we propose a novel benchmarking approach: in particular, we aim at building a curated dataset reflecting real world physical temporal dynamics, extracting temporal signals from real-world videos using optical flow. As such, we introduce REAL-V-TSFM, a novel dataset designed to capture rich and diverse time series derived from real-world videos. Experimental results on state-of-the-art TSFMs under zero-shot forecasting show that, despite strong performance on conventional benchmarks, these models exhibit performance degradation on the proposed dataset, suggesting limited generalizability to novel datasets. These findings underscore the need for novel approaches to acquiring time series data and highlight the lack of universality in recent TSFMs, while further validating the effectiveness of our video-based time series data extraction pipeline.
comment : This paper has been accepted by Artificial Intelligence for Time Series Analysis (AI4TS) Workshop @ AAAI 2026: Theory, Algorithms, and Applications
♻ ☆ New-Onset Diabetes Assessment Using Artificial Intelligence-Enhanced Electrocardiography ML4H 2025
Hao Zhang, Neil Jethani, Aahlad Puli, Leonid Garber, Lior Jankelson, Yindalon Aphinyanaphongs, Rajesh Ranganath
Diabetes has a long asymptomatic period which can often remain undiagnosed for multiple years. In this study, we trained a deep learning model to detect new-onset diabetes using 12-lead ECG and readily available demographic information. To do so, we used retrospective data where patients have both a hemoglobin A1c and ECG measured. However, such patients may not be representative of the complete patient population. As part of the study, we proposed a methodology to evaluate our model in the target population by estimating the probability of receiving an A1c test and reweight the retrospective population to represent the general population. We also adapted an efficient algorithm to generate Shapley values for both ECG signals and demographic features at the same time for model interpretation. The model offers an automated, more accurate method for early diabetes detection compared to current screening efforts. Their potential use in wearable devices can facilitate large-scale, community-wide screening, improving healthcare outcomes.
comment : 25 pages, 9 figures, published as a conference paper at ML4H 2025
♻ ☆ CzechLynx: A Dataset for Individual Identification and Pose Estimation of the Eurasian Lynx
Lukas Picek, Elisa Belotti, Michal Bojda, Ludek Bufka, Vojtech Cermak, Martin Dula, Rostislav Dvorak, Luboslav Hrdy, Miroslav Jirik, Vaclav Kocourek, Josefa Krausova, Jirı Labuda, Jakub Straka, Ludek Toman, Vlado Trulık, Martin Vana, Miroslav Kutal
We introduce CzechLynx, the first large-scale, open-access dataset for individual identification, pose estimation, and instance segmentation of the Eurasian lynx (Lynx lynx). CzechLynx contains 39,760 camera trap images annotated with segmentation masks, identity labels, and 20-point skeletons and covers 319 unique individuals across 15 years of systematic monitoring in two geographically distinct regions: southwest Bohemia and the Western Carpathians. In addition to the real camera trap data, we provide a large complementary set of photorealistic synthetic images and a Unity-based generation pipeline with diffusion-based text-to-texture modeling, capable of producing arbitrarily large amounts of synthetic data spanning diverse environments, poses, and coat-pattern variations. To enable systematic testing across realistic ecological scenarios, we define three complementary evaluation protocols: (i) geo-aware, (ii) time-aware open-set, and (iii) time-aware closed-set, covering cross-regional and long-term monitoring settings. With the provided resources, CzechLynx offers a unique, flexible benchmark for robust evaluation of computer vision and machine learning models across realistic ecological scenarios.
♻ ☆ Robust LLM Unlearning with MUDMAN: Meta-Unlearning with Disruption Masking And Normalization
Filip Sondej, Yushi Yang, Mikołaj Kniejski, Marcel Windys
Language models can retain dangerous knowledge and skills even after extensive safety fine-tuning, posing both misuse and misalignment risks. Recent studies show that even specialized unlearning methods can be easily reversed. To address this, we systematically evaluate many existing and novel components of unlearning methods and identify ones crucial for irreversible unlearning. We introduce Disruption Masking, a technique in which we only allow updating weights, where the signs of the unlearning gradient and the retaining gradient are the same. This ensures all updates are non-disruptive. Additionally, we identify the need for normalizing the unlearning gradients, and also confirm the usefulness of meta-learning. We combine these insights into MUDMAN (Meta-Unlearning with Disruption Masking and Normalization) and validate its effectiveness at preventing the recovery of dangerous capabilities. MUDMAN outperforms the prior TAR method by 40%, setting a new state-of-the-art for robust unlearning.
♻ ☆ Training for Obsolescence? The AI-Driven Education Trap
Andrew J. Peterson
Artificial intelligence is simultaneously transforming the production function of human capital in schools and the return to skills in the labor market. We develop a theoretical model to analyze the potential for misallocation when these two forces are considered in isolation. We study an educational planner who observes AI's immediate productivity benefits in teaching specific skills but fails to fully internalize the technology's future wage-suppressing effects on those same skills. Motivated by a pre-registered pilot study suggesting a positive correlation between a skill's "teachability" by AI and its vulnerability to automation, we show that this information friction leads to a systematic skill mismatch. The planner over-invests in skills destined for obsolescence, a distortion that increases monotonically with AI prevalence. Extensions demonstrate that this mismatch is exacerbated by the neglect of unpriced non-cognitive skills and by the endogenous over-adoption of educational technology. Our findings caution that policies promoting AI in education, if not paired with forward-looking labor market signals, may paradoxically undermine students' long-term human capital, such as by crowding out skills like persistence that are forged through intellectual struggle.
comment : Under review
♻ ☆ ADNF-Clustering: An Adaptive and Dynamic Neuro-Fuzzy Clustering for Leukemia Prediction
Marco Aruta, Ciro Listone, Giuseppe Murano, Aniello Murano
Leukemia diagnosis and monitoring rely increasingly on high-throughput image data, yet conventional clustering methods lack the flexibility to accommodate evolving cellular patterns and quantify uncertainty in real time. We introduce Adaptive and Dynamic Neuro-Fuzzy Clustering, a novel streaming-capable framework that combines Convolutional Neural Network-based feature extraction with an online fuzzy clustering engine. ADNF initializes soft partitions via Fuzzy C-Means, then continuously updates micro-cluster centers, densities, and fuzziness parameters using a Fuzzy Temporal Index (FTI) that measures entropy evolution. A topology refinement stage performs density-weighted merging and entropy-guided splitting to guard against over- and under-segmentation. On the C-NMC leukemia microscopy dataset, our tool achieves a silhouette score of 0.51, demonstrating superior cohesion and separation over static baselines. The method's adaptive uncertainty modeling and label-free operation hold immediate potential for integration within the INFANT pediatric oncology network, enabling scalable, up-to-date support for personalized leukemia management.
comment : 6 pages, 1 figure
♻ ☆ CAMA: Enhancing Mathematical Reasoning in Large Language Models with Causal Knowledge
Lei Zan, Keli Zhang, Ruichu Cai, Lujia Pan
Large Language Models (LLMs) have demonstrated strong performance across a wide range of tasks, yet they still struggle with complex mathematical reasoning, a challenge fundamentally rooted in deep structural dependencies. To address this challenge, we propose \textbf{CA}usal \textbf{MA}thematician (\textbf{CAMA}), a two-stage causal framework that equips LLMs with explicit, reusable mathematical structure. In the learning stage, CAMA first constructs the \textbf{M}athematical \textbf{C}ausal \textbf{G}raph (\textbf{MCG}), a high-level representation of solution strategies, by combining LLM priors with causal discovery algorithms applied to a corpus of question-solution pairs. The resulting MCG encodes essential knowledge points and their causal dependencies. To better align the graph with downstream reasoning tasks, CAMA further refines the MCG through iterative feedback derived from a selected subset of the question-solution pairs. In the reasoning stage, given a new question, CAMA dynamically extracts a task-relevant subgraph from the MCG, conditioned on both the question content and the LLM's intermediate reasoning trace. This subgraph, which encodes the most pertinent knowledge points and their causal dependencies, is then injected back into the LLM to guide its reasoning process. Empirical results on real-world datasets show that CAMA significantly improves LLM performance on challenging mathematical problems. Furthermore, our experiments demonstrate that structured guidance consistently outperforms unstructured alternatives, and that incorporating asymmetric causal relationships yields greater improvements than using symmetric associations alone.
♻ ☆ Natural, Artificial, and Human Intelligences
Emmanuel M. Pothos, Dominic Widdows
Human achievement, whether in culture, science, or technology, is unparalleled in the known existence. This achievement is tied to the enormous communities of knowledge, made possible by language: leaving theological content aside, it is very much true that "in the beginning was the word", and that in Western societies, this became particularly identified with the written word. There lies the challenge regarding modern age chatbots: they can 'do' language apparently as well as ourselves and there is a natural question of whether they can be considered intelligent, in the same way as we are or otherwise. Are humans uniquely intelligent? We consider this question in terms of the psychological literature on intelligence, evidence for intelligence in non-human animals, the role of written language in science and technology, progress with artificial intelligence, the history of intelligence testing (for both humans and machines), and the role of embodiment in intelligence. We think that it is increasingly difficult to consider humans uniquely intelligent. There are current limitations in chatbots, e.g., concerning perceptual and social awareness, but much attention is currently devoted to overcoming such limitations.
♻ ☆ Privacy Reasoning in Ambiguous Context s
Ren Yi, Octavian Suciu, Adria Gascon, Sarah Meiklejohn, Eugene Bagdasarian, Marco Gruteser
We study the ability of language models to reason about appropriate information disclosure - a central aspect of the evolving field of agentic privacy. Whereas previous works have focused on evaluating a model's ability to align with human decisions, we examine the role of ambiguity and missing context on model performance when making information-sharing decisions. We identify context ambiguity as a crucial barrier for high performance in privacy assessments. By designing Camber, a framework for context disambiguation, we show that model-generated decision rationales can reveal ambiguities and that systematically disambiguating context based on these rationales leads to significant accuracy improvements (up to 13.3% in precision and up to 22.3% in recall) as well as reductions in prompt sensitivity. Overall, our results indicate that approaches for context disambiguation are a promising way forward to enhance agentic privacy reasoning.
♻ ☆ Entropy Rectifying Guidance for Diffusion and Flow Models NeurIPS 2025
Tariq Berrada Ifriqi, Adriana Romero-Soriano, Michal Drozdzal, Jakob Verbeek, Karteek Alahari
Guidance techniques are commonly used in diffusion and flow models to improve image quality and input consistency for conditional generative tasks such as class-conditional and text-to-image generation. In particular, classifier-free guidance (CFG) is the most widely adopted guidance technique. It results, however, in trade-offs across quality, diversity and consistency: improving some at the expense of others. While recent work has shown that it is possible to disentangle these factors to some extent, such methods come with an overhead of requiring an additional (weaker) model, or require more forward passes per sampling step. In this paper, we propose Entropy Rectifying Guidance (ERG), a simple and effective guidance method based on inference-time changes in the attention mechanism of state-of-the-art diffusion transformer architectures, which allows for simultaneous improvements over image quality, diversity and prompt consistency. ERG is more general than CFG and similar guidance techniques, as it extends to unconditional sampling. We show that ERG results in significant improvements in various tasks, including text-to-image, class-conditional and unconditional image generation. We also show that ERG can be seamlessly combined with other recent guidance methods such as CADS and APG, further improving generation results.
comment : NeurIPS 2025
♻ ☆ MTTR-A: Measuring Cognitive Recovery Latency in Multi- Agent Systems
Barak Or
Ensuring cognitive stability in autonomous multi-agent systems (MAS) is a central challenge for large-scale, distributed AI. While existing observability tools monitor system outputs, they cannot quantify how rapidly agentic workflows recover once reasoning coherence has been lost. We adapt classical reliability metrics-Mean Time-to-Recovery (MTTR), Mean Time Between Failures (MTBF), and related ratios-into the cognitive domain, defining MTTR-A (Mean Time-to-Recovery for Agentic Systems) as a runtime measure of cognitive recovery latency. MTTR-A quantifies the time required for a MAS to detect reasoning drift and restore consistent operation, capturing the recovery of reasoning coherence rather than infrastructural repair. A benchmark simulation using the AG~News corpus and the LangGraph orchestration framework was conducted, modeling recovery latencies across multiple reflex modes. Automated reflexes restored stability within approximately 6s on average, while human-approval interventions required about 12s. Across 200 runs, the median simulated MTTR-A was 6.21+-2.14s, MTBF=6.7+-2.14s, and NRR=0.08, demonstrating measurable runtime resilience across reflex strategies. By formalizing recovery latency as a quantifiable property of distributed reasoning-and deriving reliability bounds linking recovery time and cognitive uptime-this work establishes a foundation for runtime dependability in agentic cognition, transforming cognitive recovery from an ad-hoc process into a standardized, interpretable performance
comment : preprint
♻ ☆ LAPS-Diff: A Diffusion -Based Framework for Singing Voice Synthesis With Language Aware Prosody-Style Guided Learning
Sandipan Dhar, Mayank Gupta, Preeti Rao
The field of Singing Voice Synthesis (SVS) has seen significant advancements in recent years due to the rapid progress of diffusion-based approaches. However, capturing vocal style, genre-specific pitch inflections, and language-dependent characteristics remains challenging, particularly in low-resource scenarios. To address this, we propose LAPS-Diff, a diffusion model integrated with language-aware embeddings and a vocal-style guided learning mechanism, specifically designed for Bollywood Hindi singing style. We curate a Hindi SVS dataset and leverage pre-trained language models to extract word and phone-level embeddings for an enriched lyrics representation. Additionally, we incorporated a style encoder and a pitch extraction model to compute style and pitch losses, capturing features essential to the naturalness and expressiveness of the synthesized singing, particularly in terms of vocal style and pitch variations. Furthermore, we utilize MERT and IndicWav2Vec models to extract musical and contextual embeddings, serving as conditional priors to refine the acoustic feature generation process further. Based on objective and subjective evaluations, we demonstrate that LAPS-Diff significantly improves the quality of the generated samples compared to the considered state-of-the-art (SOTA) model for our constrained dataset that is typical of the low resource scenario.
comment : 10 pages, 5 figures, 3 Tables
♻ ☆ Structured Cognitive Loop for Behavioral Intelligence in Large Language Model Agent s SC
Myung Ho Kim
Large language models have advanced natural language understanding and generation, but their use as autonomous agents introduces architectural challenges for multi-step tasks. Existing frameworks often mix cognition, memory, and control in a single prompt, reducing coherence and predictability. The Structured Cognitive Loop (SCL) is proposed as an alternative architecture that separates these functions. In SCL, the language model handles cognition, memory is stored externally, and execution is guided by a lightweight controller within a goal-directed loop. This design allows intermediate results to be recorded and verified before actions are taken, improving traceability and evaluation. SCL is evaluated against prompt-based baselines such as ReAct and LangChain agents across three tasks: travel planning, conditional email drafting, and constraint-guided image generation. Under matched settings, SCL achieves an average task success rate of 86.3 percent, compared with 70.5 to 76.8 percent for baselines. It also shows higher goal fidelity, fewer redundant calls, and reduced unsupported assertions. These results indicate that separating cognition, memory, and control can enhance reliability and interpretability without relying on larger models or heavier prompts. The findings should be regarded as preliminary evidence, with broader tests across model families and task domains planned for future work.
comment : Updated the order of the SCL configuration modules to match the diagram
♻ ☆ Continual Learning of Domain Knowledge from Human Feedback in Text-to-SQL
Thomas Cook, Kelly Patel, Sivapriya Vellaichamy, Udari Madhushani Sehwag, Saba Rahimi, Zhen Zeng, Sumitra Ganesh
Large Language Models (LLMs) can generate SQL queries from natural language questions but struggle with database-specific schemas and tacit domain knowledge. We introduce a framework for continual learning from human feedback in text-to-SQL, where a learning agent receives natural language feedback to refine queries and distills the revealed knowledge for reuse on future tasks. This distilled knowledge is stored in a structured memory, enabling the agent to improve execution accuracy over time. We design and evaluate multiple variations of a learning agent architecture that vary in how they capture and retrieve past experiences. Experiments on the BIRD benchmark Dev set show that memory-augmented agents, particularly the Procedural Agent, achieve significant accuracy gains and error reduction by leveraging human-in-the-loop feedback. Our results highlight the importance of transforming tacit human expertise into reusable knowledge, paving the way for more adaptive, domain-aware text-to-SQL systems that continually learn from a human-in-the-loop.
comment : 34 pages, 6 figures, 4 tables
♻ ☆ iSeal: Encrypted Fingerprinting for Reliable LLM Ownership Verification AAAI 2026
Zixun Xiong, Gaoyi Wu, Qingyang Yu, Mingyu Derek Ma, Lingfeng Yao, Miao Pan, Xiaojiang Du, Hao Wang
Given the high cost of large language model (LLM) training from scratch, safeguarding LLM intellectual property (IP) has become increasingly crucial. As the standard paradigm for IP ownership verification, LLM fingerprinting thus plays a vital role in addressing this challenge. Existing LLM fingerprinting methods verify ownership by extracting or injecting model-specific features. However, they overlook potential attacks during the verification process, leaving them ineffective when the model thief fully controls the LLM's inference process. In such settings, attackers may share prompt-response pairs to enable fingerprint unlearning or manipulate outputs to evade exact-match verification. We propose iSeal, the first fingerprinting method designed for reliable verification when the model thief controls the suspected LLM in an end-to-end manner. It injects unique features into both the model and an external module, reinforced by an error-correction mechanism and a similarity-based verification strategy. These components are resistant to verification-time attacks, including collusion-based fingerprint unlearning and response manipulation, backed by both theoretical analysis and empirical results. iSeal achieves 100 percent Fingerprint Success Rate (FSR) on 12 LLMs against more than 10 attacks, while baselines fail under unlearning and response manipulations.
comment : Accepted by AAAI 2026
♻ ☆ Learning Rules from Rewards
Guillermo Puebla, Leonidas A. A. Doumas
Humans can flexibly generalize knowledge across domains by leveraging structured relational representations. While prior research has shown how such representations support analogical reasoning, less is known about how they are recruited to guide adaptive behavior. We address this gap by introducing the Relational Regression Tree Learner (RRTL), a model that incrementally builds policies over structured relational inputs by selecting task-relevant relations during the learning process. RRTL is grounded in the framework of relational reinforcement learning but diverges from traditional approaches by focusing on ground (i.e., non-variabilized) rules that refer to specific object configurations. Across three Atari games of increasing relational complexity (Breakout, Pong, Demon Attack), the model learns to act effectively by identifying a small set of relevant relations from a broad pool of candidate relations. A comparative version of the model, which partitions the state space using relative magnitude values (e.g., "more", "same", "less"), showed more robust learning than a version using logical (binary) splits. These results provide a proof of principle that reinforcement signals can guide the selection of structured representations, offering a computational framework for understanding how relational knowledge is learned and deployed in adaptive behavior.
♻ ☆ Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey
Qizhi Pei, Zhimeng Zhou, Kaiyuan Gao, Jinhua Zhu, Yue Wang, Zun Wang, Tao Qin, Lijun Wu, Rui Yan
The integration of biomolecular modeling with natural language (BL) has emerged as a promising interdisciplinary area at the intersection of artificial intelligence, chemistry and biology. This approach leverages the rich, multifaceted descriptions of biomolecules contained within textual data sources to enhance our fundamental understanding and enable downstream computational tasks such as biomolecule property prediction. The fusion of the nuanced narratives expressed through natural language with the structural and functional specifics of biomolecules described via various molecular modeling techniques opens new avenues for comprehensively representing and analyzing biomolecules. By incorporating the contextual language data that surrounds biomolecules into their modeling, BL aims to capture a holistic view encompassing both the symbolic qualities conveyed through language as well as quantitative structural characteristics. In this review, we provide an extensive analysis of recent advancements achieved through cross modeling of biomolecules and natural language. (1) We begin by outlining the technical representations of biomolecules employed, including sequences, 2D graphs, and 3D structures. (2) We then examine in depth the rationale and key objectives underlying effective multi-modal integration of language and molecular data sources. (3) We subsequently survey the practical applications enabled to date in this developing research area. (4) We also compile and summarize the available resources and datasets to facilitate future work. (5) Looking ahead, we identify several promising research directions worthy of further exploration and investment to continue advancing the field. The related resources and contents are updating in https://github.com/QizhiPei/Awesome-Biomolecule-Language-Cross-Modeling.
comment : 2025.11.28 Updated Version
♻ ☆ Curvature Dynamic Black-box Attack: revisiting adversarial robustness via dynamic curvature estimation
Peiran Sun
Adversarial attack reveals the vulnerability of deep learning models. It is assumed that high curvature may give rise to rough decision boundary and thus result in less robust models. However, the most commonly used \textit{curvature} is the curvature of loss function, scores or other parameters from within the model as opposed to decision boundary curvature, since the former can be relatively easily formed using second order derivative. In this paper, we propose a new query-efficient method, dynamic curvature estimation (DCE), to estimate the decision boundary curvature in a black-box setting. Our approach is based on CGBA, a black-box adversarial attack. By performing DCE on a wide range of classifiers, we discovered, statistically, a connection between decision boundary curvature and adversarial robustness. We also propose a new attack method, curvature dynamic black-box attack (CDBA) with improved performance using the estimated curvature.
♻ ☆ Biased by Design: Leveraging AI Biases to Enhance Critical Thinking of News Readers
Liudmila Zavolokina, Kilian Sprenkamp, Zoya Katashinskaya, Daniel Gordon Jones
This paper explores the design of a propaganda detection tool using Large Language Models (LLMs). Acknowledging the inherent biases in AI models, especially in political contexts, we investigate how these biases might be leveraged to enhance critical thinking in news consumption. Countering the typical view of AI biases as detrimental, our research proposes strategies of user choice and personalization in response to a user's political stance, applying psychological concepts of confirmation bias and cognitive dissonance. We present findings from a qualitative user study, offering insights and design recommendations (bias awareness, personalization and choice, and gradual introduction of diverse perspectives) for AI tools in propaganda detection.
comment : European Conference on Information Systems (ECIS)
♻ ☆ SAEmnesia: Erasing Concepts in Diffusion Models with Supervised Sparse Autoencoders
Enrico Cassano, Riccardo Renzulli, Marco Nurisso, Mirko Zaffaroni, Alan Perotti, Marco Grangetto
Concept unlearning in diffusion models is hampered by feature splitting, where concepts are distributed across many latent features, making their removal challenging and computationally expensive. We introduce SAEmnesia, a supervised sparse autoencoder framework that overcomes this by enforcing one-to-one concept-neuron mappings. By systematically labeling concepts during training, our method achieves feature centralization, binding each concept to a single, interpretable neuron. This enables highly targeted and efficient concept erasure. SAEmnesia reduces hyperparameter search by 96.7% and achieves a 9.2% improvement over the state-of-the-art on the UnlearnCanvas benchmark. Our method also demonstrates superior scalability in sequential unlearning, improving accuracy by 28.4% when removing nine objects, establishing a new standard for precise and controllable concept erasure. Moreover, SAEmnesia mitigates the possibility of generating unwanted content under adversarial attack and effectively removes nudity when evaluated with I2P.
♻ ☆ Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansion
Keyang Lu, Sifan Zhou, Hongbin Xu, Gang Xu, Zhifei Yang, Yikai Wang, Zhen Xiao, Jieyi Long, Ming Li
Realistic 3D city generation is fundamental to a wide range of applications, including virtual reality and digital twins. However, most existing methods rely on training a single diffusion model, which limits their ability to generate personalized and boundless city-scale scenes. In this paper, we present Yo'City, a novel agentic framework that enables user-customized and infinitely expandable 3D city generation by leveraging the reasoning and compositional capabilities of off-the-shelf large models. Specifically, Yo'City first conceptualize the city through a top-down planning strategy that defines a hierarchical "City-District-Grid" structure. The Global Planner determines the overall layout and potential functional districts, while the Local Designer further refines each district with detailed grid-level descriptions. Subsequently, the grid-level 3D generation is achieved through a "produce-refine-evaluate" isometric image synthesis loop, followed by image-to-3D generation. To simulate continuous city evolution, Yo'City further introduces a user-interactive, relationship-guided expansion mechanism, which performs scene graph-based distance- and semantics-aware layout optimization, ensuring spatially coherent city growth. To comprehensively evaluate our method, we construct a diverse benchmark dataset and design six multi-dimensional metrics that assess generation quality from the perspectives of semantics, geometry, texture, and layout. Extensive experiments demonstrate that Yo'City consistently outperforms existing state-of-the-art methods across all evaluation aspects.
comment : 22 pages, 16 figures
♻ ☆ Learning Contrastive Feature Representations for Facial Action Unit Detection
Ziqiao Shang, Bin Liu, Fengmao Lv, Fei Teng, Tianrui Li, Lan-Zhe Guo
For the Facial Action Unit (AU) detection task, accurately capturing the subtle facial differences between distinct AUs is essential for reliable detection. Additionally, AU detection faces challenges from class imbalance and the presence of noisy or false labels, which undermine detection accuracy. In this paper, we introduce a novel contrastive learning framework aimed for AU detection that incorporates both self-supervised and supervised signals, thereby enhancing the learning of discriminative features for accurate AU detection. To tackle the class imbalance issue, we employ a negative sample re-weighting strategy that adjusts the step size of updating parameters for minority and majority class samples. Moreover, to address the challenges posed by noisy and false AU labels, we employ a sampling technique that encompasses three distinct types of positive sample pairs. This enables us to inject self-supervised signals into the supervised signal, effectively mitigating the adverse effects of noisy labels. Our experimental assessments, conducted on five widely-utilized benchmark datasets (BP4D, DISFA, BP4D+, GFT and Aff-Wild2), underscore the superior performance of our approach compared to state-of-the-art methods of AU detection. Our code is available at https://github.com/Ziqiao-Shang/AUNCE.
comment : This paper has been accepted by Pattern Recognition
♻ ☆ A Style is Worth One Code: Unlocking Code-to-Style Image Generation with Discrete Style Space
Huijie Liu, Shuhao Cui, Haoxiang Cao, Shuai Ma, Kai Wu, Guoliang Kang
Innovative visual stylization is a cornerstone of artistic creation, yet generating novel and consistent visual styles remains a significant challenge. Existing generative approaches typically rely on lengthy textual prompts, reference images, or parameter-efficient fine-tuning to guide style-aware image generation, but often struggle with style consistency, limited creativity, and complex style representations. In this paper, we affirm that a style is worth one numerical code by introducing the novel task, code-to-style image generation, which produces images with novel, consistent visual styles conditioned solely on a numerical style code. To date, this field has only been primarily explored by the industry (e.g., Midjourney), with no open-source research from the academic community. To fill this gap, we propose CoTyle, the first open-source method for this task. Specifically, we first train a discrete style codebook from a collection of images to extract style embeddings. These embeddings serve as conditions for a text-to-image diffusion model (T2I-DM) to generate stylistic images. Subsequently, we train an autoregressive style generator on the discrete style embeddings to model their distribution, allowing the synthesis of novel style embeddings. During inference, a numerical style code is mapped to a unique style embedding by the style generator, and this embedding guides the T2I-DM to generate images in the corresponding style. Unlike existing methods, our method offers unparalleled simplicity and diversity, unlocking a vast space of reproducible styles from minimal input. Extensive experiments validate that CoTyle effectively turns a numerical code into a style controller, demonstrating a style is worth one code.
comment : Code: https://github.com/Kwai-Kolors/CoTyle Demo: https://huggingface.co/spaces/Kwai-Kolors/CoTyle Homepage: https://kwai-kolors.github.io/CoTyle/
♻ ☆ Real-Time Obstacle Avoidance for a Mobile Robot Using CNN-Based Sensor Fusion
Lamiaa H. Zain
Obstacle avoidance is a critical component of the navigation stack required for mobile robots to operate effectively in complex and unknown environments. In this research, three end-to-end Convolutional Neural Networks (CNNs) were trained and evaluated offline and deployed on a differential-drive mobile robot for real-time obstacle avoidance to generate low-level steering commands from synchronized color and depth images acquired by an Intel RealSense D415 RGB-D camera in diverse environments. Offline evaluation showed that the NetConEmb model achieved the best performance with a notably low MedAE of $0.58 \times 10^{-3}$ rad/s. In comparison, the lighter NetEmb architecture, which reduces the number of trainable parameters by approximately 25\% and converges faster, produced comparable results with an RMSE of $21.68 \times 10^{-3}$ rad/s, close to the $21.42 \times 10^{-3}$ rad/s obtained by NetConEmb. Real-time navigation further confirmed NetConEmb's robustness, achieving a 100\% success rate in both known and unknown environments, while NetEmb and NetGated succeeded only in navigating the known environment.
♻ ☆ Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach
Irina Jurenka, Markus Kunesch, Kevin R. McKee, Daniel Gillick, Shaojian Zhu, Sara Wiltberger, Shubham Milind Phal, Katherine Hermann, Daniel Kasenberg, Avishkar Bhoopchand, Ankit Anand, Miruna Pîslar, Stephanie Chan, Lisa Wang, Jennifer She, Parsa Mahmoudieh, Aliya Rysbek, Wei-Jen Ko, Andrea Huber, Brett Wiltshire, Gal Elidan, Roni Rabin, Jasmin Rubinovitz, Amit Pitaru, Mac McAllister, Julia Wilkowski, David Choi, Roee Engelberg, Lidan Hackmon, Adva Levin, Rachel Griffin, Michael Sears, Filip Bar, Mia Mesar, Mana Jabbour, Arslan Chaudhry, James Cohan, Sridhar Thiagarajan, Nir Levine, Ben Brown, Dilan Gorur, Svetlana Grant, Rachel Hashimshoni, Laura Weidinger, Jieru Hu, Dawn Chen, Kuba Dolecki, Canfer Akbulut, Maxwell Bileschi, Laura Culp, Wen-Xin Dong, Nahema Marchal, Kelsie Van Deman, Hema Bajaj Misra, Michael Duah, Moran Ambar, Avi Caciularu, Sandra Lefdal, Chris Summerfield, James An, Pierre-Alexandre Kamienny, Abhinit Mohdi, Theofilos Strinopoulous, Annie Hale, Wayne Anderson, Luis C. Cobo, Niv Efron, Muktha Ananda, Shakir Mohamed, Maureen Heymans, Zoubin Ghahramani, Yossi Matias, Ben Gomes, Lila Ibrahim
A major challenge facing the world is the provision of equitable and universal access to quality education. Recent advances in generative AI (gen AI) have created excitement about the potential of new technologies to offer a personal tutor for every learner and a teaching assistant for every teacher. The full extent of this dream, however, has not yet materialised. We argue that this is primarily due to the difficulties with verbalising pedagogical intuitions into gen AI prompts and the lack of good evaluation practices, reinforced by the challenges in defining excellent pedagogy. Here we present our work collaborating with learners and educators to translate high level principles from learning science into a pragmatic set of seven diverse educational benchmarks, spanning quantitative, qualitative, automatic and human evaluations; and to develop a new set of fine-tuning datasets to improve the pedagogical capabilities of Gemini, introducing LearnLM-Tutor. Our evaluations show that LearnLM-Tutor is consistently preferred over a prompt tuned Gemini by educators and learners on a number of pedagogical dimensions. We hope that this work can serve as a first step towards developing a comprehensive educational evaluation framework, and that this can enable rapid progress within the AI and EdTech communities towards maximising the positive impact of gen AI in education.
♻ ☆ Progressive Localisation in Localist LLMs
Joachim Diederich
This paper demonstrates that progressive localization, the gradual increase of attention locality from early distributed layers to late localized layers, represents the optimal architecture for creating interpretable large language models (LLMs) while preserving performance. Through systematic experimentation with GPT-2 fine-tuned on The Psychology of Artificial Superintelligence, we evaluate seven locality configurations ranging from fully distributed to strictly localist, with five progressive schedules implementing polynomial increases (linear through quintic). We investigate whether interpretability constraints can be aligned with natural semantic structure while being applied strategically across network depth. We demonstrate that progressive semantic localization, combining adaptive semantic block partitioning with steep polynomial locality schedules, achieves near-baseline language modeling performance while providing interpretable attention patterns. Multiple independent training runs with different random seeds establish that results are statistically robust and highly reproducible. The approach dramatically outperforms both fixed-window localization and naive uniform locality constraints. Analysis reveals that maintaining flexibility through low-fidelity constraints preserves model capacity while providing interpretability benefits, and that steep schedules concentrating locality in decision-critical final layers while preserving distributed learning in early layers achieve near-baseline attention distribution characteristics. These findings demonstrate that interpretability mechanisms should align with semantic structure to achieve practical performance-interpretability tradeoffs for trustworthy AI systems.
♻ ☆ $μ$PC: Scaling Predictive Coding to 100+ Layer Networks
Francesco Innocenti, El Mehdi Achour, Christopher L. Buckley
The biological implausibility of backpropagation (BP) has motivated many alternative, brain-inspired algorithms that attempt to rely only on local information, such as predictive coding (PC) and equilibrium propagation. However, these algorithms have notoriously struggled to train very deep networks, preventing them from competing with BP in large-scale settings. Indeed, scaling PC networks (PCNs) has recently been posed as a challenge for the community (Pinchetti et al., 2024). Here, we show that 100+ layer PCNs can be trained reliably using a Depth-$μ$P parameterisation (Yang et al., 2023; Bordelon et al., 2023) which we call "$μ$PC". By analysing the scaling behaviour of PCNs, we reveal several pathologies that make standard PCNs difficult to train at large depths. We then show that, despite addressing only some of these instabilities, $μ$PC allows stable training of very deep (up to 128-layer) residual networks on simple classification tasks with competitive performance and little tuning compared to current benchmarks. Moreover, $μ$PC enables zero-shot transfer of both weight and activity learning rates across widths and depths. Our results serve as a first step towards scaling PC to more complex architectures and have implications for other local algorithms. Code for $μ$PC is made available as part of a JAX library for PCNs.
comment : 35 pages, 42 figures
♻ ☆ Chiplet-Based RISC-V SoC with Modular AI Acceleration
Suhas Suresh Bharadwaj, Prerana Ramkumar
Achieving high performance, energy efficiency, and cost-effectiveness while maintaining architectural flexibility is a critical challenge in the development and deployment of edge AI devices. Monolithic SoC designs struggle with this complex balance mainly due to low manufacturing yields (below 16%) at advanced 360 mm^2 process nodes. This paper presents a novel chiplet-based RISC-V SoC architecture that addresses these limitations through modular AI acceleration and intelligent system level optimization. Our proposed design integrates 4 different key innovations in a 30mm x 30mm silicon interposer: adaptive cross-chiplet Dynamic Voltage and Frequency Scaling (DVFS); AI-aware Universal Chiplet Interconnect Express (UCIe) protocol extensions featuring streaming flow control units and compression-aware transfers; distributed cryptographic security across heterogeneous chiplets; and intelligent sensor-driven load migration. The proposed architecture integrates a 7nm RISC-V CPU chiplet with dual 5nm AI accelerators (15 TOPS INT8 each), 16GB HBM3 memory stacks, and dedicated power management controllers. Experimental results across industry standard benchmarks like MobileNetV2, ResNet-50 and real-time video processing demonstrate significant performance improvements. The AI-optimized configuration achieves ~14.7% latency reduction, 17.3% throughput improvement, and 16.2% power reduction compared to previous basic chiplet implementations. These improvements collectively translate to a 40.1% efficiency gain corresponding to ~3.5 mJ per MobileNetV2 inference (860 mW/244 images/s), while maintaining sub-5ms real-time capability across all experimented workloads. These performance upgrades demonstrate that modular chiplet designs can achieve near-monolithic computational density while enabling cost efficiency, scalability and upgradeability, crucial for next-generation edge AI device applications.
comment : 3 pages, 3 figures, 2 tables, 3rd IEEE International Conference of Computational Intelligence and Network Systems 2025
♻ ☆ YARE-GAN: Yet Another Resting State EEG-GAN
Yeganeh Farahzadi, Morteza Ansarinia, Zoltan Kekecs
Resting-state EEG offers a non-invasive view of spontaneous brain activity, yet the extraction of meaningful patterns is often constrained by limited availability of high-quality data, and heavy reliance on manually engineered EEG features. Generative Adversarial Networks (GANs) offer not only a means to synthesize and augment neural signals, but also a promising way for learning meaningful representations directly from raw data, a dual capability that remains largely unexplored in EEG research. In this study, we introduce a scalable GAN-based framework for resting-state EEG that serves this dual role: 1) synthesis and 2) unsupervised feature extraction. The generated time series closely replicate key statistical and spectral properties of real EEG, as validated through both visual and quantitative evaluations. Importantly, we demonstrate that the model's learned representations can be repurposed for a downstream gender classification task, achieving higher out-of-sample accuracy than models trained directly on EEG signals and performing comparably to recent EEG foundation models, while using significantly less data and computational resources. These findings highlight the potential of generative models to serve as both neural signal generators and unsupervised feature extractors, paving the way for more data-efficient, architecture-driven approaches to EEG analysis with reduced reliance on manual feature engineering. The implementation code for this study is available at: https://github.com/Yeganehfrh/YARE-GAN.
♻ ☆ RvLLM: LLM Runtime Verification with Domain Knowledge
Yedi Zhang, Sun Yi Emma, Annabelle Lee Jia En, Jin Song Dong
Large language models (LLMs) have emerged as a dominant AI paradigm due to their exceptional text understanding and generation capabilities. However, their tendency to generate inconsistent or erroneous outputs challenges their reliability, especially in high-stakes domains requiring accuracy and trustworthiness. Existing research primarily focuses on detecting and mitigating model misbehavior in general-purpose scenarios, often overlooking the potential of integrating domain-specific knowledge. In this work, we advance misbehavior detection by incorporating domain knowledge. The core idea is to design a general specification language that enables domain experts to customize domain-specific predicates in a lightweight and intuitive manner, supporting later runtime verification of LLM outputs. To achieve this, we design a novel specification language, ESL, and introduce a runtime verification framework, RvLLM, to validate LLM output against domain-specific constraints defined in ESL. We evaluate RvLLM on three representative tasks: violation detection against Singapore Rapid Transit Systems Act, numerical comparison, and inequality solving. Experimental results demonstrate that RvLLM effectively detects erroneous outputs across various LLMs in a lightweight and flexible manner. The results reveal that despite their impressive capabilities, LLMs remain prone to low-level errors due to limited interpretability and a lack of formal guarantees during inference, and our framework offers a potential long-term solution by leveraging expert domain knowledge to rigorously and efficiently verify LLM outputs.
comment : 24 pages, 11 tables, 13 figures
♻ ☆ LongCat-Flash-Omni Technical Report
Meituan LongCat Team, Bairui Wang, Bayan, Bin Xiao, Bo Zhang, Bolin Rong, Borun Chen, Chang Wan, Chao Zhang, Chen Huang, Chen Chen, Chen Chen, Chengxu Yang, Chengzuo Yang, Cong Han, Dandan Peng, Delian Ruan, Detai Xin, Disong Wang, Dongchao Yang, Fanfan Liu, Fengjiao Chen, Fengyu Yang, Gan Dong, Gang Huang, Gang Xu, Guanglu Wan, Guoqiang Tan, Guoqiao Yu, Haibo Qiu, Hao Lu, Hongbo Liu, Hongyu Xiang, Jiaheng Wu, Jian Yang, Jiaxing Liu, Jing Huang, Jingang Wang, Jinrui Ding, Juchao Jiang, Jun Kuang, Jun Wang, Junhui Mei, Ke Ding, Kefeng Zhang, Lei Chen, Liang Shi, Limeng Qiao, Liming Zheng, Lin Ma, Liuyang Guo, Liya Ma, Luying Sun, Man Gao, Mengshen Zhu, Miao Cao, Minliang Lin, Nuo Xu, Peng Shi, Qi Zhang, Qian Fang, Qian Wang, Qian Yang, Quanxiu Wang, Rongxiang Weng, Rongxin Guo, Ruoxuan Liang, Senbin Yang, Shanbo Xu, Shanglin Lei, Shengze Ye, Shimin Chen, Shuaiqi Chen, Shujie Hu, Shuo Li, Siqi Yang, Siyu Xu, Siyu Ren, Song Li, Songxiang Liu, Tianhao Bai, Tianye Dai, Wei Hong, Wei Wang, Weixiao Zhao, Wengang Cao, Wenlong Zhu, Wenlong He, Xi Su, Xi Nan, Xiaohan Zhao, Xiaohao Wang, Xiaoyu Zhao, Xiaoyu Wang, Xiaoyu Li, Xin Pan, Xin Chen, Xiusong Sun, Xu Xiang, Xudong Xing, Xuezhi Cao, Xunliang Cai, Yang Yang, Yanli Tan, Yao Yao, Yerui Sun, Yi Chen, Yifan Lu, Yin Gong, Yining Zhang, Yitian Chen, Yiyang Gan, Yuchen Tang, Yuchen Xie, Yueqian Wang, Yuewen Zheng, Yufei Zhang, Yufeng Zhong, Yulei Qian, Yuqi Peng, Yuqian Li, Yuwei Jiang, Zeyang Hu, Zheng Zhang, Zhengkun Tian, Zhiqing Hong, Zhixiong Zeng, Zhuqi Mi, Ziran Li, Ziwen Wang, Ziyi Zhao, Ziyuan Zhuang, Zizhe Zhao
We introduce LongCat-Flash-Omni, a state-of-the-art open-source omni-modal model with 560 billion parameters, excelling at real-time audio-visual interaction. By adopting a curriculum-inspired progressive training strategy that transitions from simpler to increasingly complex modality sequence modeling tasks, LongCat-Flash-Omni attains comprehensive multimodal capabilities while maintaining strong unimodal capability. Building upon LongCat-Flash, which adopts a high-performance Shortcut-connected Mixture-of-Experts (MoE) architecture with zero-computation experts, LongCat-Flash-Omni integrates efficient multimodal perception and speech reconstruction modules. Despite its immense size of 560B parameters (with 27B activated), LongCat-Flash-Omni achieves low-latency real-time audio-visual interaction. For training infrastructure, we developed a modality-decoupled parallelism scheme specifically designed to manage the data and model heterogeneity inherent in large-scale multimodal training. This innovative approach demonstrates exceptional efficiency by sustaining over 90% of the throughput achieved by text-only training. Extensive evaluations show that LongCat-Flash-Omni achieves state-of-the-art performance on omni-modal benchmarks among open-source models. Furthermore, it delivers highly competitive results across a wide range of modality-specific tasks, including text, image, and video understanding, as well as audio understanding and generation. We provide a comprehensive overview of the model architecture design, training procedures, and data strategies, and open-source the model to foster future research and development in the community.
♻ ☆ Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory
Yuqi Wu, Wenzhao Zheng, Jie Zhou, Jiwen Lu
Dense 3D scene reconstruction from an ordered sequence or unordered image collections is a critical step when bringing research in computer vision into practical scenarios. Following the paradigm introduced by DUSt3R, which unifies an image pair densely into a shared coordinate system, subsequent methods maintain an implicit memory to achieve dense 3D reconstruction from more images. However, such implicit memory is limited in capacity and may suffer from information loss of earlier frames. We propose Point3R, an online framework targeting dense streaming 3D reconstruction. To be specific, we maintain an explicit spatial pointer memory directly associated with the 3D structure of the current scene. Each pointer in this memory is assigned a specific 3D position and aggregates scene information nearby in the global coordinate system into a changing spatial feature. Information extracted from the latest frame interacts explicitly with this pointer memory, enabling dense integration of the current observation into the global coordinate system. We design a 3D hierarchical position embedding to promote this interaction and design a simple yet effective fusion mechanism to ensure that our pointer memory is uniform and efficient. Our method achieves competitive or state-of-the-art performance on various tasks with low training costs. Code: https://github.com/YkiWu/Point3R.
comment : Code is available at: https://github.com/YkiWu/Point3R
♻ ☆ InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training
Pengkai Wang, Linus, Pengwei Liu, Zhijie Sang, Congkai Xie, Hongxia Yang
Reinforcement learning has powered many of the recent breakthroughs in large language models, especially for tasks where rewards can be computed automatically, such as code generation. However, these methods deteriorate in open-ended domains like medical consultation, where feedback is inherently ambiguous, highly context-dependent, and cannot be reduced to a reliable scalar signal. In such settings, RL must either rely on supervision-intensive reward models that often fail to generalize, or it falls into pathological behaviors such as reward hacking - an especially troubling risk for high-stakes medical dialogue. To address these limitations, we introduce ORBIT, an open-ended rubric-based incremental training framework for high-stakes medical dialogue. ORBIT integrates synthetic dialogue generation with dynamically constructed rubrics that serve as adaptive guides for incremental RL. Instead of relying on external medical knowledge bases or handcrafted rule sets, ORBIT uses rubric-driven feedback to steer the learning process. Its judge component can be instantiated with general-purpose instruction-following LLMs, removing the need for any task-specific fine-tuning. Applied to the Qwen3-4B-Instruct model, ORBIT raises the HealthBench-Hard score from 7.0 to 27.5 using only 2k training samples, achieving SOTA performance for models at this scale. With larger rubric datasets, ORBIT-trained models further compete with the strongest open-source baselines on HealthBench-Hard. Our analysis shows that rubric-guided RL consistently improves consultation quality across diverse medical scenarios. We also apply such rubric generation and training pipeline to InfoBench, where ORBIT enhances instruction-following performance, highlighting the generality of rubric-based feedback.
♻ ☆ Counterfactual Explanation for Multivariate Time Series Forecasting with Exogenous Variables
Keita Kinjo
Currently, machine learning is widely used across various domains, including time series data analysis. However, some machine learning models function as black boxes, making interpretability a critical concern. One approach to address this issue is counterfactual explanation (CE), which aims to provide insights into model predictions. This study focuses on the relatively underexplored problem of generating counterfactual explanations for time series forecasting. We propose a method for extracting CEs in time series forecasting using exogenous variables, which are frequently encountered in fields such as business and marketing. In addition, we present methods for analyzing the influence of each variable over an entire time series, generating CEs by altering only specific variables, and evaluating the quality of the resulting CEs. We validate the proposed method through theoretical analysis and empirical experiments, showcasing its accuracy and practical applicability. These contributions are expected to support real-world decision-making based on time series data analysis.
comment : 27pages,9figures,9tables
♻ ☆ Atom of Thoughts for Markov LLM Test-Time Scaling NeurIPS 2025
Fengwei Teng, Quan Shi, Zhaoyang Yu, Jiayi Zhang, Chenglin Wu, Yuyu Luo, Zhijiang Guo
Large Language Models (LLMs) achieve superior performance through training-time scaling, and test-time scaling further enhances their capabilities by conducting effective reasoning during inference. However, as the scale of reasoning increases, existing test-time scaling methods suffer from accumulated historical information, which not only wastes computational resources but also interferes with effective reasoning. To address this issue, we observe that complex reasoning can be achieved by solving a series of independent and self-contained subquestions. These subquestions are essentially \textit{atomic questions}, exhibiting the memoryless property similar to Markov processes. Based on this observation, we propose Atom of Thoughts (\our), where each state transition consists of decomposing the current question into a dependency-based directed acyclic graph and contracting its subquestions, forming a simplified question that maintains answer equivalence with the original problem. This answer preservation enables the iterative \textit{decomposition-contraction} process to naturally form a meaningful Markov reasoning process. Furthermore, these atomic states can be seamlessly integrated into existing test-time scaling methods, enabling \our to serve as a plug-in enhancement for improving reasoning capabilities. Experiments across six benchmarks demonstrate the effectiveness of \our both as a standalone framework and a plug-in enhancement. Notably, on HotpotQA, when applied to gpt-4o-mini, \our achieves an \textbf{80.6\%} F1 score, surpassing o3-mini by \textbf{3.4\%} and DeepSeek-R1 by \textbf{10.6\%}. The code is available at \href{https://github.com/qixucen/atom}{https://github.com/qixucen/atom}.
comment : Accepted to NeurIPS 2025
♻ ☆ Actionable and diverse counterfactual explanations incorporating domain knowledge and causal constraints
Szymon Bobek, Łukasz Bałec, Grzegorz J. Nalepa
Counterfactual explanations enhance the actionable interpretability of machine learning models by identifying the minimal changes required to achieve a desired outcome of the model. However, existing methods often ignore the complex dependencies in real-world datasets, leading to unrealistic or impractical modifications. Motivated by cybersecurity applications in the email marketing domain, we propose a method for generating Diverse, Actionable, and kNowledge-Constrained Explanations (DANCE), which incorporates feature dependencies and causal constraints to ensure plausibility and real-world feasibility of counterfactuals. Our method learns linear and nonlinear constraints from data or integrates expert-provided dependency graphs, ensuring counterfactuals are plausible and actionable. By maintaining consistency with feature relationships, the method produces explanations that align with real-world constraints. Additionally, it balances plausibility, diversity, and sparsity, effectively addressing key limitations in existing algorithms. The work is developed based on a real-life case study with Freshmail, the largest email marketing company in Poland and supported by a joint R&D project Sendguard. Furthermore, we provide an extensive evaluation using 140 public datasets, which highlights its ability to generate meaningful, domain-relevant counterfactuals that outperform other existing approaches based on widely used metrics. The source code for reproduction of the results can be found in a GitHub repository we provide.
♻ ☆ Vectorized Online POMDP Planning ICRA 2026
Marcus Hoerger, Muhammad Sudrajat, Hanna Kurniawati
Planning under partial observability is an essential capability of autonomous robots. The Partially Observable Markov Decision Process (POMDP) provides a powerful framework for planning under partial observability problems, capturing the stochastic effects of actions and the limited information available through noisy observations. POMDP solving could benefit tremendously from massive parallelization of today's hardware, but parallelizing POMDP solvers has been challenging. They rely on interleaving numerical optimization over actions with the estimation of their values, which creates dependencies and synchronization bottlenecks between parallel processes that can quickly offset the benefits of parallelization. In this paper, we propose Vectorized Online POMDP Planner (VOPP), a novel parallel online solver that leverages a recent POMDP formulation that analytically solves part of the optimization component, leaving only the estimation of expectations for numerical computation. VOPP represents all data structures related to planning as a collection of tensors and implements all planning steps as fully vectorized computations over this representation. The result is a massively parallel solver with no dependencies and synchronization bottlenecks between parallel computations. Experimental results indicate that VOPP is at least 20X more efficient in computing near-optimal solutions compared to an existing state-of-the-art parallel online solver.
comment : 9 pages, 3 figures. Submitted to ICRA 2026
♻ ☆ Un-mixing Test-time Adaptation under Heterogeneous Data Streams
Zixian Su, Jingwei Guo, Xi Yang, Qiufeng Wang, Kaizhu Huang
Deploying deep models in real-world scenarios remains challenging due to significant performance drops under distribution shifts between training and deployment environments. Test-Time Adaptation (TTA) has recently emerged as a promising solution, enabling on-the-fly model adaptation. However, its effectiveness deteriorates in the presence of mixed distribution shifts -- common in practical settings -- where multiple target domains coexist. In this paper, we study TTA under mixed distribution shifts and move beyond conventional whole-batch adaptation paradigms. By revisiting distribution shifts from a spectral perspective, we find that the heterogeneity across latent domains is often pronounced in Fourier space. In particular, high-frequency components encode domain-specific variations, which facilitates clearer separation of samples from different distributions. Motivated by this observation, we propose to un-mix heterogeneous data streams using high-frequency domain cues, making diverse shift patterns more tractable. To this end, we propose Frequency-based Decentralized Adaptation (FreDA), a novel framework that decomposes globally heterogeneous data stream into locally homogeneous clusters in the Fourier space. It leverages decentralized learning and augmentation strategies to robustly adapt under mixed domain shifts. Extensive experiments across various environments (corrupted, natural, and medical) show the superiority of our method over the state-of-the-arts.
♻ ☆ Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models NeurIPS 2025
Cameron Tice, Philipp Alexander Kreer, Nathan Helm-Burger, Prithviraj Singh Shahani, Fedor Ryzhenkov, Jacob Haimes, Felix Hofstätter, Teun van der Weij
Capability evaluations play a crucial role in assessing and regulating frontier AI systems. The effectiveness of these evaluations faces a significant challenge: strategic underperformance, or ``sandbagging'', where models deliberately underperform during evaluation. Sandbagging can manifest either through explicit developer intervention or through unintended model behavior, presenting a fundamental obstacle to accurate capability assessment. We introduce a novel sandbagging detection method based on injecting noise of varying magnitudes into model weights. While non-sandbagging models show predictable performance degradation with increasing noise, we demonstrate that sandbagging models exhibit anomalous performance improvements, likely due to disruption of underperformance mechanisms while core capabilities remain partially intact. Through experiments across various model architectures, sizes, and sandbagging techniques, we establish this distinctive response pattern as a reliable, model-agnostic signal for detecting sandbagging behavior. Importantly, we find noise-injection is capable of eliciting the full performance of Mistral Large 120B in a setting where the model underperforms without being instructed to do so. Our findings provide a practical tool for AI evaluation and oversight, addressing a challenge in ensuring accurate capability assessment of frontier AI systems.
comment : Published at NeurIPS 2025, code available at https://github.com/camtice/SandbagDetect. Preliminary work presented at SATA and SoLaR (NeurIPS 2024 workshops)
♻ ☆ Scale-Agnostic Kolmogorov-Arnold Geometry in Neural Networks
Mathew Vanherreweghe, Michael H. Freedman, Keith M. Adams
Recent work by Freedman and Mulligan demonstrated that shallow multilayer perceptrons spontaneously develop Kolmogorov-Arnold geometric (KAG) structure during training on synthetic three-dimensional tasks. However, it remained unclear whether this phenomenon persists in realistic high-dimensional settings and what spatial properties this geometry exhibits. We extend KAG analysis to MNIST digit classification (784 dimensions) using 2-layer MLPs with systematic spatial analysis at multiple scales. We find that KAG emerges during training and appears consistently across spatial scales, from local 7-pixel neighborhoods to the full 28x28 image. This scale-agnostic property holds across different training procedures: both standard training and training with spatial augmentation produce the same qualitative pattern. These findings reveal that neural networks spontaneously develop organized, scale-invariant geometric structure during learning on realistic high-dimensional data.
♻ ☆ UNO: Unifying One-stage Video Scene Graph Generation via Object-Centric Visual Representation Learning WACV 2026
Huy Le, Nhat Chung, Tung Kieu, Jingkang Yang, Ngan Le
Video Scene Graph Generation (VidSGG) aims to represent dynamic visual content by detecting objects and modeling their temporal interactions as structured graphs. Prior studies typically target either coarse-grained box-level or fine-grained panoptic pixel-level VidSGG, often requiring task-specific architectures and multi-stage training pipelines. In this paper, we present UNO (UNified Object-centric VidSGG), a single-stage, unified framework that jointly addresses both tasks within an end-to-end architecture. UNO is designed to minimize task-specific modifications and maximize parameter sharing, enabling generalization across different levels of visual granularity. The core of UNO is an extended slot attention mechanism that decomposes visual features into object and relation slots. To ensure robust temporal modeling, we introduce object temporal consistency learning, which enforces consistent object representations across frames without relying on explicit tracking modules. Additionally, a dynamic triplet prediction module links relation slots to corresponding object pairs, capturing evolving interactions over time. We evaluate UNO on standard box-level and pixel-level VidSGG benchmarks. Results demonstrate that UNO not only achieves competitive performance across both tasks but also offers improved efficiency through a unified, object-centric design.
comment : 11 pages, 7 figures. Accepted at WACV 2026
♻ ☆ Computational Foundations for Strategic Coopetition: Formalizing Interdependence and Complementarity
Vik Pant, Eric Yu
Modern socio-technical systems are characterized by strategic coopetition where actors simultaneously cooperate to create value and compete to capture it. While conceptual modeling languages like i* provide rich qualitative representations of strategic dependencies, they lack mechanisms for quantitative analysis of dynamic trade-offs. Conversely, classical game theory offers mathematical rigor but strips away contextual richness. This technical report bridges this gap by developing computational foundations that formalize two critical dimensions of coopetition: interdependence and complementarity. We ground interdependence in i* structural dependency analysis, translating depender-dependee-dependum relationships into quantitative interdependence coefficients through a structured translation framework. We formalize complementarity following Brandenburger and Nalebuff's Added Value concept, modeling synergistic value creation with validated parameterization. We integrate structural dependencies with bargaining power in value appropriation and introduce a game-theoretic formulation where Nash Equilibrium incorporates structural interdependence. Validation combines comprehensive experimental testing across power and logarithmic value function specifications, demonstrating functional form robustness, with empirical application to the Samsung-Sony S-LCD joint venture (2004-2011), where logarithmic specifications achieve validation score 59/60 compared to power functions (55/60), with both demonstrating strong empirical fit to S-LCD historical patterns. This technical report serves as the foundational reference for a coordinated research program examining strategic coopetition in requirements engineering and multi-agent systems, with companion work addressing trust dynamics, team production, and reciprocity mechanisms.
comment : 37 pages, 9 figures
♻ ☆ ARIAL: An Agent ic Framework for Document VQA with Precise Answer Localization
Ahmad Mohammadshirazi, Pinaki Prasad Guha Neogi, Dheeraj Kulshrestha, Rajiv Ramnath
Document Visual Question Answering (VQA) requires models to not only extract accurate textual answers but also precisely localize them within document images, a capability critical for interpretability in high-stakes applications. However, existing systems achieve strong textual accuracy while producing unreliable spatial grounding, or sacrifice performance for interpretability. We present ARIAL (Agentic Reasoning for Interpretable Answer Localization), a modular framework that orchestrates specialized tools through an LLM-based planning agent to achieve both precise answer extraction and reliable spatial grounding. ARIAL decomposes Document VQA into structured subtasks: OCR-based text extraction with TrOCR, retrieval-augmented context selection using semantic search, answer generation via a fine-tuned Gemma 3-27B model, and explicit bounding-box localization through text-to-region alignment. This modular architecture produces transparent reasoning traces, enabling tool-level auditability and independent component optimization. We evaluate ARIAL on four benchmarks (DocVQA, FUNSD, CORD, and SROIE) using both textual accuracy (ANLS) and spatial precision (mAP at IoU 0.50 to 0.95). ARIAL achieves state-of-the-art results across all datasets: 88.7 ANLS and 50.1 mAP on DocVQA, 90.0 ANLS and 50.3 mAP on FUNSD, 85.5 ANLS and 60.2 mAP on CORD, and 93.1 ANLS on SROIE, surpassing the previous best method (DLaVA) by +2.8 ANLS and +3.9 mAP on DocVQA. Our work demonstrates how agentic orchestration of specialized tools can simultaneously improve performance and interpretability, providing a pathway toward trustworthy, explainable document AI systems.
♻ ☆ Physics Steering: Causal Control of Cross-Domain Concepts in a Physics Foundation Model
Rio Alexa Fear, Payel Mukhopadhyay, Michael McCabe, Alberto Bietti, Miles Cranmer
Recent advances in mechanistic interpretability have revealed that large language models (LLMs) develop internal representations corresponding not only to concrete entities but also distinct, human-understandable abstract concepts and behaviour. Moreover, these hidden features can be directly manipulated to steer model behaviour. However, it remains an open question whether this phenomenon is unique to models trained on inherently structured data (ie. language, images) or if it is a general property of foundation models. In this work, we investigate the internal representations of a large physics-focused foundation model. Inspired by recent work identifying single directions in activation space for complex behaviours in LLMs, we extract activation vectors from the model during forward passes over simulation datasets for different physical regimes. We then compute "delta" representations between the two regimes. These delta tensors act as concept directions in activation space, encoding specific physical features. By injecting these concept directions back into the model during inference, we can steer its predictions, demonstrating causal control over physical behaviours, such as inducing or removing some particular physical feature from a simulation. These results suggest that scientific foundation models learn generalised representations of physical principles. They do not merely rely on superficial correlations and patterns in the simulations. Our findings open new avenues for understanding and controlling scientific foundation models and has implications for AI-enabled scientific discovery.
comment : 16 Pages, 9 Figures. Code available soon at https://github.com/DJ-Fear/walrus_steering
♻ ☆ Structured Prompt ing Enables More Robust Evaluation of Language Models
Asad Aali, Muhammad Ahmed Mohsin, Vasiliki Bikia, Arnav Singhvi, Richard Gaus, Suhana Bedi, Hejie Cui, Miguel Fuentes, Alyssa Unell, Yifan Mai, Jordan Cahoon, Michael Pfeffer, Roxana Daneshjou, Sanmi Koyejo, Emily Alsentzer, Christopher Potts, Nigam H. Shah, Akshay S. Chaudhari
As language models (LMs) are increasingly adopted across domains, high-quality benchmarking frameworks that accurately estimate performance are essential for guiding deployment decisions. While frameworks such as Holistic Evaluation of Language Models (HELM) enable broad evaluation across tasks, they often rely on fixed prompts that fail to generalize across LMs, yielding unrepresentative performance estimates. Unless we approximate each LM's ceiling (maximum achievable via changes to the prompt), we risk underestimating performance. Declarative prompting frameworks, such as DSPy, offer a scalable alternative to manual prompt engineering by crafting structured prompts that can be optimized per task. However, such frameworks have not been systematically evaluated across established benchmarks. We present a reproducible DSPy+HELM framework that introduces structured prompting methods which elicit reasoning, enabling more accurate LM benchmarking. Using four prompting methods, we evaluate four frontier LMs across seven benchmarks (general/medical domain) against existing HELM baseline scores. We find that without structured prompting: (i) HELM underestimates LM performance (by 4% average), (ii) performance estimates vary more across benchmarks ($+$2% standard deviation), (iii) performance gaps are misrepresented (leaderboard rankings flip on 3/7 benchmarks), and (iv) introducing chain-of-thought reduces LM sensitivity to prompt design (smaller $Δ$ across prompts). To our knowledge, this is the first benchmarking study to systematically integrate structured prompting into an established evaluation framework, demonstrating how scalable performance-ceiling approximation yields more robust, decision-useful benchmarks. We open-source (i) DSPy+HELM Integration (https://github.com/stanford-crfm/helm/pull/3893) and (ii) Prompt Optimization Pipeline (https://github.com/StanfordMIMI/dspy-helm).
♻ ☆ Domain adaptation of large language models for geotechnical applications
Lei Fan, Fangxue Liu, Cheng Chen
The rapid advancement of large language models (LLMs) is transforming opportunities in geotechnical engineering, where workflows rely on complex, text-rich data. While general-purpose LLMs demonstrate strong reasoning capabilities, their effectiveness in geotechnical applications is constrained by limited exposure to specialized terminology and domain logic. Thus, domain adaptation, tailoring general LLMs for geotechnical use, has become essential. This paper presents the first systematic review of LLM adaptation and application in geotechnical contexts. It critically examines four key adaptation strategies, including prompt engineering, retrieval augmented generation, domain-adaptive pretraining, and fine-tuning, and evaluates their comparative benefits, limitations, and implementation trends. This review synthesizes current applications spanning geological interpretation, subsurface characterization, design analysis, numerical modeling, risk assessment, and geotechnical education. Findings show that domain-adapted LLMs substantially improve reasoning accuracy, automation, and interpretability, yet remain limited by data scarcity, validation challenges, and explainability concerns. Future research directions are also suggested. This review establishes a critical foundation for developing geotechnically literate LLMs and guides researchers and practitioners in advancing the digital transformation of geotechnical engineering.
♻ ☆ Deep Improvement Super vision
Arip Asadulaev, Rayan Banerjee, Fakhri Karray, Martin Takac
Recently, it was shown that small, looped architectures, such as Tiny Recursive Models (TRMs), can outperform Large Language Models (LLMs) on complex reasoning tasks, including the Abstraction and Reasoning Corpus (ARC). In this work, we investigate a core question: how can we further improve the efficiency of these methods with minimal changes? To address this, we frame the latent reasoning of TRMs as a form of classifier-free guidance and implicit policy improvement algorithm. Building on these insights, we propose a novel training scheme that provides a target for each loop during training. We demonstrate that our approach significantly enhances training efficiency. Our method reduces the total number of forward passes by 18x and eliminates halting mechanisms, while maintaining quality comparable to standard TRMs. Notably, we achieve 24% accuracy on ARC-1 with only 0.8M parameters, outperforming most LLMs.
♻ ☆ Reranking partisan animosity in algorithmic social media feeds alters affective polarization
Tiziano Piccardi, Martin Saveski, Chenyan Jia, Jeffrey T. Hancock, Jeanne L. Tsai, Michael Bernstein
Today, social media platforms hold sole power to study the effects of feed ranking algorithms. We developed a platform-independent method that reranks participants' feeds in real-time and used this method to conduct a preregistered 10-day field experiment with 1,256 participants on X during the 2024 U.S. presidential campaign. Our experiment used a large language model to rerank posts that expressed antidemocratic attitudes and partisan animosity (AAPA). Decreasing or increasing AAPA exposure shifted out-party partisan animosity by two points on a 100-point feeling thermometer, with no detectable differences across party lines, providing causal evidence that exposure to AAPA content alters affective polarization. This work establishes a method to study feed algorithms without requiring platform cooperation, enabling independent evaluation of ranking interventions in naturalistic settings.
♻ ☆ Data efficient surrogate modeling for engineering design: Ensemble-free batch mode deep active learning for regression
Sarthak Kapoor, Harsh Vardhan, Umesh Timalsina, Sumit Kumar, Peter Volgyesi, Janos Sztipanovits
High fidelity design evaluation processes such as Computational Fluid Dynamics and Finite Element Analysis are often replaced with data driven surrogates to reduce computational cost in engineering design optimization. However, building accurate surrogate models still requires a large number of expensive simulations. To address this challenge, we introduce epsilon HQS, a scalable active learning strategy that leverages a student teacher framework to train deep neural networks efficiently. Unlike Bayesian AL methods, which are computationally demanding with DNNs, epsilon HQS selectively queries informative samples to reduce labeling cost. Applied to CFD, FEA, and propeller design tasks, our method achieves higher accuracy under fixed labeling cost budgets.
comment : 6 pages, 4 figures
♻ ☆ VLMs have Tunnel Vision : Evaluating Nonlocal Visual Reasoning in Leading VLMs
Shmuel Berman, Jia Deng
Vision-Language Models (VLMs) excel at complex visual tasks such as VQA and chart understanding, yet recent work suggests they struggle with simple perceptual tests. We present an evaluation of vision-language models' capacity for nonlocal visual reasoning: reasoning that requires chaining evidence collected from multiple, possibly distant regions of an image. We isolate three distinct forms of nonlocal vision: comparative perception, which demands holding two images in working memory and comparing them; saccadic search, which requires making discrete, evidence-driven jumps to locate successive targets; and smooth visual search, which involves following a continuous contour. Flagship models (e.g., GPT-5, Gemini 2.5 Pro, Claude Sonnet 4), even those that perform well on prior primitive-vision benchmarks, fail these tests and barely exceed random accuracy on two variants of our tasks that are trivial for humans. Our structured evaluation suite allows us to test whether VLMs can perform visual algorithms similar to those used by humans. Our findings show that despite gains in raw visual acuity, current models lack core visual reasoning capabilities.
♻ ☆ SACA: Selective Attention-Based Clustering Algorithm
Meysam Shirdel Bilehsavar, Razieh Ghaedi, Samira Seyed Taheri, Xinqi Fan, Christian O'Reilly
Clustering algorithms are fundamental tools across many fields, with density-based methods offering particular advantages in identifying arbitrarily shaped clusters and handling noise. However, their effectiveness is often limited by the requirement of critical parameter tuning by users, which typically requires significant domain expertise. This paper introduces a novel density-based clustering algorithm loosely inspired by the concept of selective attention, designed to minimize reliance on parameter tuning for most applications. The proposed method computes an adaptive threshold to exclude sparsely distributed points and outliers, constructs an initial cluster framework, and subsequently reintegrates the filtered points to refine the final results. Extensive experiments on diverse benchmark datasets demonstrate the robustness, accuracy, and ease of use of the proposed approach, establishing it as a powerful alternative to conventional density-based clustering techniques.
comment : 32 pages, 14 figures
♻ ☆ A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision -Language Tasks
Chia Xin Liang, Pu Tian, Caitlyn Heqi Yin, Yao Yua, Wei An-Hou, Li Ming, Xinyuan Song, Tianyang Wang, Ziqian Bi, Ming Liu
This survey and application guide to multimodal large language models(MLLMs) explores the rapidly developing field of MLLMs, examining their architectures, applications, and impact on AI and Generative Models. Starting with foundational concepts, we delve into how MLLMs integrate various data types, including text, images, video and audio, to enable complex AI systems for cross-modal understanding and generation. It covers essential topics such as training methods, architectural components, and practical applications in various fields, from visual storytelling to enhanced accessibility. Through detailed case studies and technical analysis, the text examines prominent MLLM implementations while addressing key challenges in scalability, robustness, and cross-modal learning. Concluding with a discussion of ethical considerations, responsible AI development, and future directions, this authoritative resource provides both theoretical frameworks and practical insights. It offers a balanced perspective on the opportunities and challenges in the development and deployment of MLLMs, and is highly valuable for researchers, practitioners, and students interested in the intersection of natural language processing and computer vision.
♻ ☆ Towards Hierarchical Multi- Agent Decision-Making for Uncertainty-Aware EV Charging
Lo Pang-Yun Ting, Ali Şenol, Huan-Yang Wang, Hsu-Chao Lai, Kun-Ta Chuang, Huan Liu
Recent advances in bidirectional EV charging and discharging systems have spurred interest in workplace applications. However, real-world deployments face various dynamic factors, such as fluctuating electricity prices and uncertain EV departure times, that hinder effective energy management. To address these issues and minimize building electricity costs while meeting EV charging requirements, we design a hierarchical multi-agent structure in which a high-level agent coordinates overall charge or discharge decisions based on real-time pricing, while multiple low-level agents manage individual power level accordingly. For uncertain EV departure times, we propose a novel uncertainty-aware critic augmentation mechanism for low-level agents that improves the evaluation of power-level decisions and ensures robust control under such uncertainty. Building upon these two key designs, we introduce HUCA, a real-time charging control framework that coordinates energy supply among the building and EVs. Experiments on real-world electricity datasets show that HUCA significantly reduces electricity costs and maintains competitive performance in meeting EV charging requirements under both simulated certain and uncertain departure scenarios. The results further highlight the importance of hierarchical control and the proposed critic augmentation under the uncertain departure scenario. A case study illustrates HUCA's capability to allocate energy between the building and EVs in real time, underscoring its potential for practical use.
♻ ☆ The Unified Cognitive Consciousness Theory for Language Models: Anchoring Semantics, Thresholds of Activation, and Emergent Reasoning
Edward Y. Chang, Zeyneb N. Kaya, Ethan Chang
We propose semantic anchoring, a unified account of how large language models turn pretrained capacity into goal-directed behavior: external structure (in-context examples, retrieval, or light tuning) binds the model's latent patterns to desired targets. Unified Contextual Control Theory (UCCT) formalizes this via anchoring strength $S = ρ_d - d_r - \log k$, where $ρ_d$ measures target cohesion in representation space, $d_r$ measures mismatch from prior knowledge, and $k$ is the anchor budget. UCCT predicts threshold-like performance flips and strictly generalizes in-context learning, reading retrieval and fine-tuning as anchoring variants. Three controlled studies provide evidence. Experiment 1 demonstrates cross-domain anchoring rebinding strong priors in text and vision. Experiment 2 varies representational familiarity via numeral bases (base-10/8/9) at fixed complexity, yielding ordered thresholds and transfer patterns tracking $ρ_d$, $d_r$, and $S$. Experiment 3 establishes a geometry-to-behavior correlate: layer-wise peak anchoring and trajectory area predict few-shot thresholds $θ_{50}$. UCCT offers testable theory and practical metrics for optimizing prompts, retrieval, and tuning.
comment : 21 pages, 7 figure, 4 table
♻ ☆ LLMs on support of privacy and security of mobile apps: state of the art and research directions
Tran Thanh Lam Nguyen, Barbara Carminati, Elena Ferrari
Modern life has witnessed the explosion of mobile devices. However, besides the valuable features that bring convenience to end users, security and privacy risks still threaten users of mobile apps. The increasing sophistication of these threats in recent years has underscored the need for more advanced and efficient detection approaches. In this chapter, we explore the application of Large Language Models (LLMs) to identify security risks and privacy violations and mitigate them for the mobile application ecosystem. By introducing state-of-the-art research that applied LLMs to mitigate the top 10 common security risks of smartphone platforms, we highlight the feasibility and potential of LLMs to replace traditional analysis methods, such as dynamic and hybrid analysis of mobile apps. As a representative example of LLM-based solutions, we present an approach to detect sensitive data leakage when users share images online, a common behavior of smartphone users nowadays. Finally, we discuss open research challenges.
comment : I am writing to respectfully request the withdrawal of my recent submission to arXiv due to an authorship issue. The paper was submitted without the explicit consent of two co-authors. After internal discussion, they have expressed clear disagreement with the submission and raised concerns about unresolved academic inaccuracies in the current version
♻ ☆ Implicit Models: Expressive Power Scales with Test-Time Compute
Jialin Liu, Lisang Ding, Stanley Osher, Wotao Yin
Implicit models, an emerging model class, compute outputs by iterating a single parameter block to a fixed point. This architecture realizes an infinite-depth, weight-tied network that trains with constant memory, significantly reducing memory needs for the same level of performance compared to explicit models. While it is empirically known that these compact models can often match or even exceed the accuracy of larger explicit networks by allocating more test-time compute, the underlying mechanism remains poorly understood. We study this gap through a nonparametric analysis of expressive power. We provide a strict mathematical characterization, showing that a simple and regular implicit operator can, through iteration, progressively express more complex mappings. We prove that for a broad class of implicit models, this process lets the model's expressive power scale with test-time compute, ultimately matching a much richer function class. The theory is validated across four domains: image reconstruction, scientific computing, operations research, and LLM reasoning, demonstrating that as test-time iterations increase, the complexity of the learned mapping rises, while the solution quality simultaneously improves and stabilizes.
♻ ☆ Coordinating Spinal and Limb Dynamics for Enhanced Sprawling Robot Mobility ICRA 2025
Merve Atasever, Ali Okhovat, Azhang Nazaripouya, John Nisbet, Omer Kurkutlu, Jyotirmoy V. Deshmukh, Yasemin Ozkan Aydin
Sprawling locomotion in vertebrates, particularly salamanders, demonstrates how body undulation and spinal mobility enhance stability, maneuverability, and adaptability across complex terrains. While prior work has separately explored biologically inspired gait design or deep reinforcement learning (DRL), these approaches face inherent limitations: open-loop gait designs often lack adaptability to unforeseen terrain variations, whereas end-to-end DRL methods are data-hungry and prone to unstable behaviors when transferring from simulation to real robots. We propose a hybrid control framework that integrates Hildebrand's biologically grounded gait design with DRL, enabling a salamander-inspired quadruped robot to exploit active spinal joints for robust crawling motion. Our evaluation across multiple robot configurations in target-directed navigation tasks reveals that this hybrid approach systematically improves robustness under environmental uncertainties such as surface irregularities. By bridging structured gait design with learning-based methodology, our work highlights the promise of interdisciplinary control strategies for developing efficient, resilient, and biologically informed spinal actuation in robotic systems.
comment : Initial version of the work has been accepted for presentation at the Mechanical Intelligence in Robotics workshop at ICRA 2025
♻ ☆ Automated Program Repair of Uncompilable Student Code
Griffin Pitts, Aum Pandya, Darsh Rank, Tirth Bhatt, Muntasir Hoq, Bita Akram
A significant portion of student programming submissions in CS1 learning environments are uncompilable, limiting their use in student modeling and downstream knowledge tracing. Traditional modeling pipelines often exclude these cases, discarding observations of student learning. This study investigates automated program repair as a strategy to recover uncompilable code while preserving students' structural intent for use in student modeling. Within this framework, we assess large language models (LLMs) as repair agents under high- and low-context prompting conditions. Repairs were evaluated for compilability, edit distance, and preservation of students' original structure and logic. While all models produced compilable repairs, they differed in how well they preserve students' control flow and code structure, affecting their pedagogical utility. By recovering uncompilable submissions, this work enables richer and more comprehensive analyses of learners' coding processes and development over time.
comment : In Proceedings of the 57th ACM Technical Symposium on Computer Science Education V.2 (SIGCSE TS 2026)
♻ ☆ CogniPair: From LLM Chatbots to Conscious AI Agent s -- GNWT-Based Multi- Agent Digital Twins for Social Pairing -- Dating & Hiring Applications
Wanghao Ye, Sihan Chen, Yiting Wang, Shwai He, Bowei Tian, Guoheng Sun, Ziyi Wang, Ziyao Wang, Yexiao He, Zheyu Shen, Meng Liu, Yuning Zhang, Meng Feng, Yang Wang, Siyuan Peng, Yilong Dai, Zhenle Duan, Lang Xiong, Joshua Liu, Hanzhang Qin, Ang Li
Current large language model (LLM) agents lack authentic human psychological processes necessary for genuine digital twins and social AI applications. To address this limitation, we present a computational implementation of Global Workspace Theory (GNWT) that integrates human cognitive architecture principles into LLM agents, creating specialized sub-agents for emotion, memory, social norms, planning, and goal-tracking coordinated through a global workspace mechanism. However, authentic digital twins require accurate personality initialization. We therefore develop a novel adventure-based personality test that evaluates true personality through behavioral choices within interactive scenarios, bypassing self-presentation bias found in traditional assessments. Building on these innovations, our CogniPair platform enables digital twins to engage in realistic simulated dating interactions and job interviews before real encounters, providing bidirectional cultural fit assessment for both romantic compatibility and workplace matching. Validation using 551 GNWT-Agents and Columbia University Speed Dating dataset demonstrates 72% correlation with human attraction patterns, 77.8% match prediction accuracy, and 74% agreement in human validation studies. This work advances psychological authenticity in LLM agents and establishes a foundation for intelligent dating platforms and HR technology solutions.
♻ ☆ Combining Textual and Structural Information for Premise Selection in Lean
Job Petrovčič, David Eliecer Narvaez Denis, Ljupčo Todorovski
Premise selection is a key bottleneck for scaling theorem proving in large formal libraries. Yet existing language-based methods often treat premises in isolation, ignoring the web of dependencies that connects them. We present a graph-augmented approach that combines dense text embeddings of Lean formalizations with graph neural networks over a heterogeneous dependency graph capturing both state-premise and premise-premise relations. On the LeanDojo Benchmark, our method outperforms the ReProver language-based baseline by over 25\% across standard retrieval metrics. These results suggest that relational information is beneficial for premise selection.
♻ ☆ TRepLiNa: Layer-wise CKA+REPINA Alignment Improves Low-Resource Machine Translation in Aya-23 8B
Toshiki Nakai, Ravi Kiran Chikkala, Lena Sophie Oberkircher, Nicholas Jennings, Natalia Skachkova, Tatiana Anikina, Jesujoba Oluwadara Alabi
The 2025 Multimodal Models for Low-Resource Contexts and Social Impact (MMLoSo) Language Challenge addresses one of India's most pressing linguistic gaps: the lack of resources for its diverse low-resource languages (LRLs). In this study, we investigate whether enforcing cross-lingual similarity in specific internal layers of a decoder-only multilingual large language model (LLM) can improve translation quality from LRL to high-resource language (HRL). Specifically, we combine Centered Kernel Alignment (CKA), a similarity metric that encourages representations of different languages to align, with REPINA, a regularization method that constrains parameter updates to remain close to the pretrained model, into a joint method we call TRepLiNa. In this research project, we experiment with zero-shot, few-shot, and fine-tuning settings using Aya-23 8B with QLoRA across MMLoSo shared task language pairs (Mundari, Santali, Bhili) with Hindi/English pivots. Our results show that aligning mid-level layers using TRepLiNa (CKA+REPINA) is a low-cost, practical approach to improving LRL translation, especially in data-scarce settings.
comment : It is work in progress
Computer Vision and Pattern Recognition 167
☆ Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models
Muhammad Maaz, Hanoona Rasheed, Fahad Shahbaz Khan, Salman Khan
Reasoning over dynamic visual content remains a central challenge for multimodal large language models. Recent thinking models generate explicit reasoning traces for interpretability; however, their reasoning often appears convincing while being logically inconsistent or weakly grounded in visual evidence. We identify and formalize these issues through two diagnostic metrics: Think Answer Consistency (TAC), which measures the alignment between reasoning and answers, and Video Attention Score (VAS), which captures the extent to which reasoning depends on visual versus textual cues. Analysis across 11 video reasoning benchmarks shows that current models rely heavily on linguistic priors rather than visual content. To address this, we propose a reinforcement learning approach that enhances both temporal precision and reasoning consistency. Our approach combines timestamp aware supervised fine tuning with Group Relative Policy Optimization (GRPO) guided by a novel Temporal Alignment Reward (TAR). This dual step post training stage encourages temporally aligned and causally coherent video reasoning. The resulting model, Video R2, achieves consistently higher TAC, VAS, and accuracy across multiple benchmarks, demonstrating that improvements in temporal alignment and reasoning coherence lead to more accurate and trustworthy video understanding. Our code, dataset, and model will be open sourced.
comment : Video-R2 Technical Report
☆ Video-CoM: Interactive Video Reasoning via Chain of Manipulations
Hanoona Rasheed, Mohammed Zumri, Muhammad Maaz, Ming-Hsuan Yang, Fahad Shahbaz Khan, Salman Khan
Recent multimodal large language models (MLLMs) have advanced video understanding, yet most still "think about videos" ie once a video is encoded, reasoning unfolds entirely in text, treating visual input as a static context. This passive paradigm creates a semantic bottleneck: models cannot rewatch, refocus, or verify evidence, leading to shallow visual reasoning on tasks requiring fine grained spatio temporal understanding. In this work, we introduce Interactive Video Reasoning, a new paradigm that transforms video into an active cognitive workspace, enabling models to "think with videos". Our model, Video CoM, reasons through a Chain of Manipulations (CoM), performing iterative visual actions to gather and refine evidence. To support this behavior, we construct Video CoM Instruct, an 18K instruction tuning dataset curated for multi step manipulation reasoning. Beyond supervised learning, we further optimize the manipulation policy via reinforcement learning with reasoning aware Group Relative Policy Optimization (GRPO). Unlike prior work that relies solely on sparse answer rewards, our method introduces step level reasoning rewards, guiding the model toward grounded and consistent reasoning. Video CoM achieves strong results across nine video reasoning benchmarks, improving average performance by 3.6 percent over recent state of the art models, while training on only 25K SFT and 3K GRPO video samples, significantly fewer than comparable large scale models. Ablation studies demonstrate that reasoning aware rewards improve both accuracy and interpretability. Code: https://github.com/mbzuai-oryx/Video-CoM
comment : Technical Report
☆ AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement
Zhizhou Zhong, Yicheng Ji, Zhe Kong, Yiying Liu, Jiarui Wang, Jiasun Feng, Lupeng Liu, Xiangyi Wang, Yanjia Li, Yuqing She, Ying Qin, Huan Li, Shuiyang Mao, Wei Liu, Wenhan Luo
Recently, multi-person video generation has started to gain prominence. While a few preliminary works have explored audio-driven multi-person talking video generation, they often face challenges due to the high costs of diverse multi-person data collection and the difficulty of driving multiple identities with coherent interactivity. To address these challenges, we propose AnyTalker, a multi-person generation framework that features an extensible multi-stream processing architecture. Specifically, we extend Diffusion Transformer's attention block with a novel identity-aware attention mechanism that iteratively processes identity-audio pairs, allowing arbitrary scaling of drivable identities. Besides, training multi-person generative models demands massive multi-person data. Our proposed training pipeline depends solely on single-person videos to learn multi-person speaking patterns and refines interactivity with only a few real multi-person clips. Furthermore, we contribute a targeted metric and dataset designed to evaluate the naturalness and interactivity of the generated multi-person videos. Extensive experiments demonstrate that AnyTalker achieves remarkable lip synchronization, visual quality, and natural interactivity, striking a favorable balance between data costs and identity scalability.
comment : Homepage: https://hkust-c4g.github.io/AnyTalker-homepage
☆ Visual Generation Tuning
Jiahao Guo, Sinan Du, Jingfeng Yao, Wenyu Liu, Bo Li, Haoxiang Cao, Kun Gai, Chun Yuan, Kai Wu, Xinggang Wang
Large Vision Language Models (VLMs) effectively bridge the modality gap through extensive pretraining, acquiring sophisticated visual representations aligned with language. However, it remains underexplored whether these representations, optimized for multimodal understanding tasks, harbor an inherent potential for visual generation. In this paper, we propose VGT, Visual Generation Tuning, a novel paradigm designed to stimulate the underlying capabilities of visual generation within any vision language models. By performing efficient visual generation tuning on well-pretrained VLMs, we significantly mitigate the alignment costs and accelerate the convergence of autoregressive modeling in the continuous space (20x speedup). Specifically, we dismiss the entangled pixel-level VAEs designed for diffusion transformers and formulate VGT-AE through aligning the semantic encoders from pretrained VLMs with the latent representations of pixel decoders. In image reconstruction tasks, we achieve 26.67 PSNR and 0.50 rFID at a 28x compression ratio, outperforming specialized VAEs; in visual generation tasks, we achieve state-of-the-art outcomes among autoregressive models, 0.77 on GenEval and 78.73 on DPG-Bench. Furthermore, our proposed VGT showcases significant scaling promise and is versatile for endowing any VLMs trained for multimodal understanding with the capabilities of visual generation, which paves the new avenue to explore next-generation unified multimodal foundation models. Models and codes are available at https://github.com/hustvl/VGT.
☆ Object-Centric Data Synthesis for Category-level Object Detection
Vikhyat Agarwal, Jiayi Cora Guo, Declan Hoban, Sissi Zhang, Nicholas Moran, Peter Cho, Srilakshmi Pattabiraman, Shantanu Joshi
Deep learning approaches to object detection have achieved reliable detection of specific object classes in images. However, extending a model's detection capability to new object classes requires large amounts of annotated training data, which is costly and time-consuming to acquire, especially for long-tailed classes with insufficient representation in existing datasets. Here, we introduce the object-centric data setting, when limited data is available in the form of object-centric data (multi-view images or 3D models), and systematically evaluate the performance of four different data synthesis methods to finetune object detection models on novel object categories in this setting. The approaches are based on simple image processing techniques, 3D rendering, and image diffusion models, and use object-centric data to synthesize realistic, cluttered images with varying contextual coherence and complexity. We assess how these methods enable models to achieve category-level generalization in real-world data, and demonstrate significant performance boosts within this data-constrained experimental setting.
comment : 10 pages, 10 figures
☆ Physics-Informed Neural Networks for Thermophysical Property Retrieval
Ali Waseem, Malcolm Mielle
Inverse heat problems refer to the estimation of material thermophysical properties given observed or known heat diffusion behaviour. Inverse heat problems have wide-ranging uses, but a critical application lies in quantifying how building facade renovation reduces thermal transmittance, a key determinant of building energy efficiency. However, solving inverse heat problems with non-invasive data collected in situ is error-prone due to environmental variability or deviations from theoretically assumed conditions. Hence, current methods for measuring thermal conductivity are either invasive, require lengthy observation periods, or are sensitive to environmental and experimental conditions. Here, we present a PINN-based iterative framework to estimate the thermal conductivity k of a wall from a set of thermographs; our framework alternates between estimating the forward heat problem with a PINN for a fixed k, and optimizing k by comparing the thermographs and surface temperatures predicted by the PINN, repeating until the estimated k's convergence. Using both environmental data captured by a weather station and data generated from Finite-Volume-Method software simulations, we accurately predict k across different environmental conditions and data collection sampling times, given the temperature profile of the wall at dawn is close to steady state. Although violating the steady-state assumption impacts the accuracy of k's estimation, we show that our proposed framework still only exhibits a maximum MAE of 4.0851. Our work demonstrates the potential of PINN-based methods for reliable estimation of material properties in situ and under realistic conditions, without lengthy measurement campaigns. Given the lack of research on using machine learning, and more specifically on PINNs, for solving in-situ inverse problems, we expect our work to be a starting point for more research on the topic.
comment : 26 pages, 4 figures, 3 tables
☆ Hunyuan-GameCraft-2: Instruction-following Interactive Game Wo rl d Model
Junshu Tang, Jiacheng Liu, Jiaqi Li, Longhuang Wu, Haoyu Yang, Penghao Zhao, Siruis Gong, Xiang Yuan, Shuai Shao, Qinglin Lu
Recent advances in generative world models have enabled remarkable progress in creating open-ended game environments, evolving from static scene synthesis toward dynamic, interactive simulation. However, current approaches remain limited by rigid action schemas and high annotation costs, restricting their ability to model diverse in-game interactions and player-driven dynamics. To address these challenges, we introduce Hunyuan-GameCraft-2, a new paradigm of instruction-driven interaction for generative game world modeling. Instead of relying on fixed keyboard inputs, our model allows users to control game video contents through natural language prompts, keyboard, or mouse signals, enabling flexible and semantically rich interaction within generated worlds. We formally defined the concept of interactive video data and developed an automated process to transform large-scale, unstructured text-video pairs into causally aligned interactive datasets. Built upon a 14B image-to-video Mixture-of-Experts(MoE) foundation model, our model incorporates a text-driven interaction injection mechanism for fine-grained control over camera motion, character behavior, and environment dynamics. We introduce an interaction-focused benchmark, InterBench, to evaluate interaction performance comprehensively. Extensive experiments demonstrate that our model generates temporally coherent and causally grounded interactive game videos that faithfully respond to diverse and free-form user instructions such as "open the door", "draw a torch", or "trigger an explosion".
comment : Technical Report, Project page:https://hunyuan-gamecraft-2.github.io/
☆ DisMo: Disentangled Motion Representations for Open-Wo rl d Motion Transfer NeurIPS 2025
Thomas Ressler-Antal, Frank Fundel, Malek Ben Alaya, Stefan Andreas Baumann, Felix Krause, Ming Gui, Björn Ommer
Recent advances in text-to-video (T2V) and image-to-video (I2V) models, have enabled the creation of visually compelling and dynamic videos from simple textual descriptions or initial frames. However, these models often fail to provide an explicit representation of motion separate from content, limiting their applicability for content creators. To address this gap, we propose DisMo, a novel paradigm for learning abstract motion representations directly from raw video data via an image-space reconstruction objective. Our representation is generic and independent of static information such as appearance, object identity, or pose. This enables open-world motion transfer, allowing motion to be transferred across semantically unrelated entities without requiring object correspondences, even between vastly different categories. Unlike prior methods, which trade off motion fidelity and prompt adherence, are overfitting to source structure or drifting from the described action, our approach disentangles motion semantics from appearance, enabling accurate transfer and faithful conditioning. Furthermore, our motion representation can be combined with any existing video generator via lightweight adapters, allowing us to effortlessly benefit from future advancements in video models. We demonstrate the effectiveness of our method through a diverse set of motion transfer tasks. Finally, we show that the learned representations are well-suited for downstream motion understanding tasks, consistently outperforming state-of-the-art video representation models such as V-JEPA in zero-shot action classification on benchmarks including Something-Something v2 and Jester. Project page: https://compvis.github.io/DisMo
comment : Accepted at NeurIPS 2025
☆ MANTA: Physics-Informed Generalized Underwater Object Tracking WACV 2026
Suhas Srinath, Hemang Jamadagni, Aditya Chadrasekar, Prathosh AP
Underwater object tracking is challenging due to wavelength dependent attenuation and scattering, which severely distort appearance across depths and water conditions. Existing trackers trained on terrestrial data fail to generalize to these physics-driven degradations. We present MANTA, a physics-informed framework integrating representation learning with tracking design for underwater scenarios. We propose a dual-positive contrastive learning strategy coupling temporal consistency with Beer-Lambert augmentations to yield features robust to both temporal and underwater distortions. We further introduce a multi-stage pipeline augmenting motion-based tracking with a physics-informed secondary association algorithm that integrates geometric consistency and appearance similarity for re-identification under occlusion and drift. To complement standard IoU metrics, we propose Center-Scale Consistency (CSC) and Geometric Alignment Score (GAS) to assess geometric fidelity. Experiments on four underwater benchmarks (WebUOT-1M, UOT32, UTB180, UWCOT220) show that MANTA achieves state-of-the-art performance, improving Success AUC by up to 6 percent, while ensuring stable long-term generalized underwater tracking and efficient runtime.
comment : Accepted to the IEEE/CVF WACV 2026
☆ VQRAE: Representation Quantization Autoencoders for Multimodal Understanding , Generation and Reconstruction
Sinan Du, Jiahao Guo, Bo Li, Shuhao Cui, Zhengzhuo Xu, Yifu Luo, Yongxian Wei, Kun Gai, Xinggang Wang, Kai Wu, Chun Yuan
Unifying multimodal understanding, generation and reconstruction representation in a single tokenizer remains a key challenge in building unified models. Previous research predominantly attempts to address this in a dual encoder paradigm, e.g., utilizing the separate encoders for understanding and generation respectively or balancing semantic representations and low-level features with contrastive loss. In this paper, we propose VQRAE, a Vector Quantization version of Representation AutoEncoders, which pioneers the first exploration in unified representation to produce Continuous semantic features for image understanding and Discrete tokens for visual generation within a unified tokenizer. Specifically, we build upon pretrained vision foundation models with a symmetric ViT decoder and adopt a two-stage training strategy: first, it freezes the encoder and learns a high-dimensional semantic VQ codebook with pixel reconstruction objective; then jointly optimizes the encoder with self-distillation constraints. This design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens that are compatible for generation and fine-grained reconstruction. Besides, we identify the intriguing property in quantizing semantic encoders that rely on high-dimensional codebook in contrast to the previous common practice of low-dimensional codebook in image reconstruction. The semantic VQ codebook can achieve a 100% utilization ratio at a dimension of 1536. VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction with promising scaling property in the autoregressive paradigm for its discrete merits.
comment : 19 pages, 10 figures
☆ DEAL-300K: Diffusion -based Editing Area Localization with a 300K-Scale Dataset and Frequency- Prompt ed Baseline
Rui Zhang, Hongxia Wang, Hangqing Liu, Yang Zhou, Qiang Zeng
Diffusion-based image editing has made semantic level image manipulation easy for general users, but it also enables realistic local forgeries that are hard to localize. Existing benchmarks mainly focus on the binary detection of generated images or the localization of manually edited regions and do not reflect the properties of diffusion-based edits, which often blend smoothly into the original content. We present Diffusion-Based Image Editing Area Localization Dataset (DEAL-300K), a large scale dataset for diffusion-based image manipulation localization (DIML) with more than 300,000 annotated images. We build DEAL-300K by using a multi-modal large language model to generate editing instructions, a mask-free diffusion editor to produce manipulated images, and an active-learning change detection pipeline to obtain pixel-level annotations. On top of this dataset, we propose a localization framework that uses a frozen Visual Foundation Model (VFM) together with Multi Frequency Prompt Tuning (MFPT) to capture both semantic and frequency-domain cues of edited regions. Trained on DEAL-300K, our method reaches a pixel-level F1 score of 82.56% on our test split and 80.97% on the external CoCoGlide benchmark, providing strong baselines and a practical foundation for future DIML research.The dataset can be accessed via https://github.com/ymhzyj/DEAL-300K.
comment : 13pages,12 figures
☆ Optimizing Multimodal Language Models through Attention-based Interpretability
Alexander Sergeev, Evgeny Kotelnikov
Modern large language models become multimodal, analyzing various data formats like text and images. While fine-tuning is effective for adapting these multimodal language models (MLMs) to downstream tasks, full fine-tuning is computationally expensive. Parameter-Efficient Fine-Tuning (PEFT) methods address this by training only a small portion of model weights. However, MLMs are difficult to interpret, making it challenging to identify which components are most effective for training to balance efficiency and performance. We propose an attention-based interpretability method for MLMs by analyzing attention scores relative to image tokens. The core idea is to identify attention heads that focus on image key objects. We utilize this information to select optimal model components for PEFT in multimodal models. Our contributions include a method for identifying attention heads associated with image key objects, its application to PEFT for image captioning, and the creation of a new dataset containing images, key object masks, and their textual descriptions. We conducted experiments on MLMs with 2-3 billion parameters to validate the method's effectiveness. By calculating Head Impact (HI) scores we quantify an attention head's focus on key objects, indicating its significance in image understanding. Our fine-tuning experiments demonstrate that adapting layers with the highest HI scores leads to the most significant shifts in metrics compared to pre-trained, randomly selected, or lowest-HI-score layers. This indicates that fine-tuning a small percentage (around 0.01%) of parameters in these crucial layers can substantially influence image understanding capabilities.
comment : Accepted for ICAI-2025 conference
☆ SimScale: Learning to Drive via Real-Wo rl d Simulation at Scale
Haochen Tian, Tianyu Li, Haochen Liu, Jiazhi Yang, Yihang Qiu, Guang Li, Junli Wang, Yinfeng Gao, Zhang Zhang, Liang Wang, Hangjun Ye, Tieniu Tan, Long Chen, Hongyang Li
Achieving fully autonomous driving systems requires learning rational decisions in a wide span of scenarios, including safety-critical and out-of-distribution ones. However, such cases are underrepresented in real-world corpus collected by human experts. To complement for the lack of data diversity, we introduce a novel and scalable simulation framework capable of synthesizing massive unseen states upon existing driving logs. Our pipeline utilizes advanced neural rendering with a reactive environment to generate high-fidelity multi-view observations controlled by the perturbed ego trajectory. Furthermore, we develop a pseudo-expert trajectory generation mechanism for these newly simulated states to provide action supervision. Upon the synthesized data, we find that a simple co-training strategy on both real-world and simulated samples can lead to significant improvements in both robustness and generalization for various planning methods on challenging real-world benchmarks, up to +6.8 EPDMS on navhard and +2.9 on navtest. More importantly, such policy improvement scales smoothly by increasing simulation data only, even without extra real-world data streaming in. We further reveal several crucial findings of such a sim-real learning system, which we term SimScale, including the design of pseudo-experts and the scaling properties for different policy architectures. Our simulation data and code would be released.
comment : Project page: https://opendrivelab.com/SimScale
☆ A Hierarchical Computer Vision Pipeline for Physiological Data Extraction from Bedside Monitors
Vinh Chau, Khoa Le Dinh Van, Hon Huynh Ngoc, Binh Nguyen Thien, Hao Nguyen Thien, Vy Nguyen Quang, Phuc Vo Hong, Yen Lam Minh, Kieu Pham Tieu, Trinh Nguyen Thi Diem, Louise Thwaites, Hai Ho Bich
In many low-resource healthcare settings, bedside monitors remain standalone legacy devices without network connectivity, creating a persistent interoperability gap that prevents seamless integration of physiological data into electronic health record (EHR) systems. To address this challenge without requiring costly hardware replacement, we present a computer vision-based pipeline for the automated capture and digitisation of vital sign data directly from bedside monitor screens. Our method employs a hierarchical detection framework combining YOLOv11 for accurate monitor and region of interest (ROI) localisation with PaddleOCR for robust text extraction. To enhance reliability across variable camera angles and lighting conditions, a geometric rectification module standardizes the screen perspective before character recognition. We evaluated the system on a dataset of 6,498 images collected from open-source corpora and real-world intensive care units in Vietnam. The model achieved a mean Average Precision (mAP@50-95) of 99.5% for monitor detection and 91.5% for vital sign ROI localisation. The end-to-end extraction accuracy exceeded 98.9% for core physiological parameters, including heart rate, oxygen saturation SpO2, and arterial blood pressure. These results demonstrate that a lightweight, camera-based approach can reliably transform unstructured information from screen captures into structured digital data, providing a practical and scalable pathway to improve information accessibility and clinical documentation in low-resource settings.
comment : 11 pages, 3 figures
☆ Flow Straighter and Faster: Efficient One-Step Generative Modeling via MeanFlow on Rectified Trajectories
Xinxi Zhang, Shiwei Tan, Quang Nguyen, Quan Dao, Ligong Han, Xiaoxiao He, Tunyu Zhang, Alen Mrdovic, Dimitris Metaxas
Flow-based generative models have recently demonstrated strong performance, yet sampling typically relies on expensive numerical integration of ordinary differential equations (ODEs). Rectified Flow enables one-step sampling by learning nearly straight probability paths, but achieving such straightness requires multiple computationally intensive reflow iterations. MeanFlow achieves one-step generation by directly modeling the average velocity over time; however, when trained on highly curved flows, it suffers from slow convergence and noisy supervision. To address these limitations, we propose Rectified MeanFlow, a framework that models the mean velocity field along the rectified trajectory using only a single reflow step. This eliminates the need for perfectly straightened trajectories while enabling efficient training. Furthermore, we introduce a simple yet effective truncation heuristic that aims to reduce residual curvature and further improve performance. Extensive experiments on ImageNet at 64, 256, and 512 resolutions show that Re-MeanFlow consistently outperforms prior one-step flow distillation and Rectified Flow methods in both sample quality and training efficiency. Code is available at https://github.com/Xinxi-Zhang/Re-MeanFlow.
☆ Markovian Scale Prediction: A New Era of Visual Autoregressive Generation
Yu Zhang, Jingyi Liu, Yiwei Shi, Qi Zhang, Duoqian Miao, Changwei Wang, Longbing Cao
Visual AutoRegressive modeling (VAR) based on next-scale prediction has revitalized autoregressive visual generation. Although its full-context dependency, i.e., modeling all previous scales for next-scale prediction, facilitates more stable and comprehensive representation learning by leveraging complete information flow, the resulting computational inefficiency and substantial overhead severely hinder VAR's practicality and scalability. This motivates us to develop a new VAR model with better performance and efficiency without full-context dependency. To address this, we reformulate VAR as a non-full-context Markov process, proposing Markov-VAR. It is achieved via Markovian Scale Prediction: we treat each scale as a Markov state and introduce a sliding window that compresses certain previous scales into a compact history vector to compensate for historical information loss owing to non-full-context dependency. Integrating the history vector with the Markov state yields a representative dynamic state that evolves under a Markov process. Extensive experiments demonstrate that Markov-VAR is extremely simple yet highly effective: Compared to VAR on ImageNet, Markov-VAR reduces FID by 10.5% (256 $\times$ 256) and decreases peak memory consumption by 83.8% (1024 $\times$ 1024). We believe that Markov-VAR can serve as a foundation for future research on visual autoregressive generation and other downstream tasks.
☆ UniGeoSeg: Towards Unified Open-Wo rl d Segmentation for Geospatial Scenes
Shuo Ni, Di Wang, He Chen, Haonan Guo, Ning Zhang, Jing Zhang
Instruction-driven segmentation in remote sensing generates masks from guidance, offering great potential for accessible and generalizable applications. However, existing methods suffer from fragmented task formulations and limited instruction data, hindering effective understanding and generalization. To address these issues, we introduce GeoSeg-1M, the first million-scale dataset for remote sensing instruction-driven segmentation, constructed via an automatic mask filtering and instruction generation pipeline that synthesizes referring, interactive, and reasoning segmentation instructions from multiple public datasets. GeoSeg-1M contains 590K images, 117 categories, and 1.1M image-mask-instruction triplets. Building upon this foundation, we further curate GeoSeg-Bench, a challenging benchmark designed to evaluate contextual understanding and reasoning capabilities across diverse instruction-driven tasks and complex geospatial scenes. Furthermore, we present UniGeoSeg, a unified framework that serves as a strong baseline, incorporating task-aware text enhancement, latent knowledge memory, and a progressive training strategy to facilitate multi-task learning. Extensive experiments demonstrate the state-of-the-art performance of UniGeoSeg across GeoSeg-Bench and diverse public benchmarks, while exhibiting strong zero-shot generalization. Datasets and source code were released at https://github.com/MiliLab/UniGeoSeg.
comment : Datasets and source code were released at https://github.com/MiliLab/UniGeoSeg
☆ A Perceptually Inspired Variational Framework for Color Enhancement
Rodrigo Palma-Amestoy, Edoardo Provenzi, Marcelo Bertalmío, Vicent Caselles
Basic phenomenology of human color vision has been widely taken as an inspiration to devise explicit color correction algorithms. The behavior of these models in terms of significative image features (such as contrast and dispersion) can be difficult to characterize. To cope with this, we propose to use a variational formulation of color contrast enhancement that is inspired by the basic phenomenology of color perception. In particular, we devise a set of basic requirements to be fulfilled by an energy to be considered as `perceptually inspired', showing that there is an explicit class of functionals satisfying all of them. We single out three explicit functionals that we consider of basic interest, showing similarities and differences with existing models. The minima of such functionals is computed using a gradient descent approach. We also present a general methodology to reduce the computational cost of the algorithms under analysis from ${\cal O}(N^2)$ to ${\cal O}(N\log N)$, being $N$ the number of input pixels.
☆ Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach
Haruki Sakajo, Hiroshi Takato, Hiroshi Tsutsui, Komei Soda, Hidetaka Kamigaito, Taro Watanabe
Large-scale Vision Language Models (LVLMs) exhibit advanced capabilities in tasks that require visual information, including object detection. These capabilities have promising applications in various industrial domains, such as autonomous driving. For example, LVLMs can generate safety-oriented descriptions of videos captured by road-facing cameras. However, ensuring comprehensive safety requires monitoring driver-facing views as well to detect risky events, such as the use of mobiles while driving. Thus, the ability to process synchronized inputs is necessary from both driver-facing and road-facing cameras. In this study, we develop models and investigate the capabilities of LVLMs by constructing a dataset and evaluating their performance on this dataset. Our experimental results demonstrate that while pre-trained LVLMs have limited effectiveness, fine-tuned LVLMs can generate accurate and safety-aware driving instructions. Nonetheless, several challenges remain, particularly in detecting subtle or complex events in the video. Our findings and error analysis provide valuable insights that can contribute to the improvement of LVLM-based systems in this domain.
comment : Accepted to MMLoSo 2025
☆ FACT-GS: Frequency-Aligned Complexity-Aware Texture Reparameterization for 2D Gaussian Splatting
Tianhao Xie, Linlian Jiang, Xinxin Zuo, Yang Wang, Tiberiu Popa
Realistic scene appearance modeling has advanced rapidly with Gaussian Splatting, which enables real-time, high-quality rendering. Recent advances introduced per-primitive textures that incorporate spatial color variations within each Gaussian, improving their expressiveness. However, texture-based Gaussians parameterize appearance with a uniform per-Gaussian sampling grid, allocating equal sampling density regardless of local visual complexity. This leads to inefficient texture space utilization, where high-frequency regions are under-sampled and smooth regions waste capacity, causing blurred appearance and loss of fine structural detail. We introduce FACT-GS, a Frequency-Aligned Complexity-aware Texture Gaussian Splatting framework that allocates texture sampling density according to local visual frequency. Grounded in adaptive sampling theory, FACT-GS reformulates texture parameterization as a differentiable sampling-density allocation problem, replacing the uniform textures with a learnable frequency-aware allocation strategy implemented via a deformation field whose Jacobian modulates local sampling density. Built on 2D Gaussian Splatting, FACT-GS performs non-uniform sampling on fixed-resolution texture grids, preserving real-time performance while recovering sharper high-frequency details under the same parameter budget.
comment : 11 pages, 6 figures, preprint
☆ Machine Learning for Scientific Visualization: Ensemble Data Analysis
Hamid Gadirov
Scientific simulations and experimental measurements produce vast amounts of spatio-temporal data, yet extracting meaningful insights remains challenging due to high dimensionality, complex structures, and missing information. Traditional analysis methods often struggle with these issues, motivating the need for more robust, data-driven approaches. This dissertation explores deep learning methodologies to improve the analysis and visualization of spatio-temporal scientific ensembles, focusing on dimensionality reduction, flow estimation, and temporal interpolation. First, we address high-dimensional data representation through autoencoder-based dimensionality reduction for scientific ensembles. We evaluate the stability of projection metrics under partial labeling and introduce a Pareto-efficient selection strategy to identify optimal autoencoder variants, ensuring expressive and reliable low-dimensional embeddings. Next, we present FLINT, a deep learning model for high-quality flow estimation and temporal interpolation in both flow-supervised and flow-unsupervised settings. FLINT reconstructs missing velocity fields and generates high-fidelity temporal interpolants for scalar fields across 2D+time and 3D+time ensembles without domain-specific assumptions or extensive finetuning. To further improve adaptability and generalization, we introduce HyperFLINT, a hypernetwork-based approach that conditions on simulation parameters to estimate flow fields and interpolate scalar data. This parameter-aware adaptation yields more accurate reconstructions across diverse scientific domains, even with sparse or incomplete data. Overall, this dissertation advances deep learning techniques for scientific visualization, providing scalable, adaptable, and high-quality solutions for interpreting complex spatio-temporal ensembles.
comment : PhD thesis, University of Groningen, 2025
☆ Simultaneous Image Quality Improvement and Artefacts Correction in Accelerated MRI
Georgia Kanli, Daniele Perlo, Selma Boudissa, Radovan Jirik, Olivier Keunen
MR data are acquired in the frequency domain, known as k-space. Acquiring high-quality and high-resolution MR images can be time-consuming, posing a significant challenge when multiple sequences providing complementary contrast information are needed or when the patient is unable to remain in the scanner for an extended period of time. Reducing k-space measurements is a strategy to speed up acquisition, but often leads to reduced quality in reconstructed images. Additionally, in real-world MRI, both under-sampled and full-sampled images are prone to artefacts, and correcting these artefacts is crucial for maintaining diagnostic accuracy. Deep learning methods have been proposed to restore image quality from under-sampled data, while others focused on the correction of artefacts that result from the noise or motion. No approach has however been proposed so far that addresses both acceleration and artefacts correction, limiting the performance of these models when these degradation factors occur simultaneously. To address this gap, we present a method for recovering high-quality images from under-sampled data with simultaneously correction for noise and motion artefact called USArt (Under-Sampling and Artifact correction model). Customized for 2D brain anatomical images acquired with Cartesian sampling, USArt employs a dual sub-model approach. The results demonstrate remarkable increase of signal-to-noise ratio (SNR) and contrast in the images restored. Various under-sampling strategies and degradation levels were explored, with the gradient under-sampling strategy yielding the best outcomes. We achieved up to 5x acceleration and simultaneously artefacts correction without significant degradation, showcasing the model's robustness in real-world settings.
☆ Learning to Predict Aboveground Biomass from RGB Images with 3D Synthetic Scenes
Silvia Zuffi
Forests play a critical role in global ecosystems by supporting biodiversity and mitigating climate change via carbon sequestration. Accurate aboveground biomass (AGB) estimation is essential for assessing carbon storage and wildfire fuel loads, yet traditional methods rely on labor-intensive field measurements or remote sensing approaches with significant limitations in dense vegetation. In this work, we propose a novel learning-based method for estimating AGB from a single ground-based RGB image. We frame this as a dense prediction task, introducing AGB density maps, where each pixel represents tree biomass normalized by the plot area and each tree's image area. We leverage the recently introduced synthetic 3D SPREAD dataset, which provides realistic forest scenes with per-image tree attributes (height, trunk and canopy diameter) and instance segmentation masks. Using these assets, we compute AGB via allometric equations and train a model to predict AGB density maps, integrating them to recover the AGB estimate for the captured scene. Our approach achieves a median AGB estimation error of 1.22 kg/m^2 on held-out SPREAD data and 1.94 kg/m^2 on a real-image dataset. To our knowledge, this is the first method to estimate aboveground biomass directly from a single RGB image, opening up the possibility for a scalable, interpretable, and cost-effective solution for forest monitoring, while also enabling broader participation through citizen science initiatives.
comment : Presented at STAG 2025
☆ Synthetic Industrial Object Detection: GenAI vs. Feature-Based Methods
Jose Moises Araya-Martinez, Adrián Sanchis Reig, Gautham Mohan, Sarvenaz Sardari, Jens Lambrecht, Jörg Krüger
Reducing the burden of data generation and annotation remains a major challenge for the cost-effective deployment of machine learning in industrial and robotics settings. While synthetic rendering is a promising solution, bridging the sim-to-real gap often requires expert intervention. In this work, we benchmark a range of domain randomization (DR) and domain adaptation (DA) techniques, including feature-based methods, generative AI (GenAI), and classical rendering approaches, for creating contextualized synthetic data without manual annotation. Our evaluation focuses on the effectiveness and efficiency of low-level and high-level feature alignment, as well as a controlled diffusion-based DA method guided by prompts generated from real-world contexts. We validate our methods on two datasets: a proprietary industrial dataset (automotive and logistics) and a public robotics dataset. Results show that if render-based data with enough variability is available as seed, simpler feature-based methods, such as brightness-based and perceptual hashing filtering, outperform more complex GenAI-based approaches in both accuracy and resource efficiency. Perceptual hashing consistently achieves the highest performance, with mAP50 scores of 98% and 67% on the industrial and robotics datasets, respectively. Additionally, GenAI methods present significant time overhead for data generation at no apparent improvement of sim-to-real mAP values compared to simpler methods. Our findings offer actionable insights for efficiently bridging the sim-to-real gap, enabling high real-world performance from models trained exclusively on synthetic data.
☆ Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering
Qiming Li, Xiaocheng Feng, Yixuan Ma, Zekai Ye, Ruihan Chen, Xiachong Feng, Bing Qin
Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) demonstrate strong reasoning capabilities, yet their performance in English significantly outperforms that in low-resource languages, raising fairness concerns in multilingual applications. Existing approaches either rely on costly multilingual training or employ prompting with external translation tools, both of which are resource-intensive and sensitive to translation quality. To address these limitations, we propose a training-free inference-time method to enhance Multilingual Reasoning capabilities via Representation Engineering (MRRE) without using any additional training data or tools. MRRE sequentially injects two precomputed vectors at specific layers during inference processing: cross-lingual reasoning enhancement vectors, which steer non-English reasoning representations toward English space to unlock multilingual reasoning, and target-language output anchoring vectors, which restore the distribution of the target language to preserve input-output language consistency. Comprehensive experiments across six advanced LLMs and LVLMs on four reasoning benchmarks demonstrate that MRRE consistently enhances non-English reasoning by an average gain of 5.48% and up to 7.54% in low-resource languages (Thai and Swahili), while improving input-output language consistency by 3.78%.
☆ Language-guided 3D scene synthesis for fine-grained functionality understanding
Jaime Corsetti, Francesco Giuliari, Davide Boscaini, Pedro Hermosilla, Andrea Pilzer, Guofeng Mei, Alexandros Delitzas, Francis Engelmann, Fabio Poiesi
Functionality understanding in 3D, which aims to identify the functional element in a 3D scene to complete an action (e.g., the correct handle to "Open the second drawer of the cabinet near the bed"), is hindered by the scarcity of real-world data due to the substantial effort needed for its collection and annotation. To address this, we introduce SynthFun3D, the first method for task-based 3D scene synthesis. Given the action description, SynthFun3D generates a 3D indoor environment using a furniture asset database with part-level annotation, ensuring the action can be accomplished. It reasons about the action to automatically identify and retrieve the 3D mask of the correct functional element, enabling the inexpensive and large-scale generation of high-quality annotated data. We validate SynthFun3D through user studies, which demonstrate improved scene-prompt coherence compared to other approaches. Our quantitative results further show that the generated data can either replace real data with minor performance loss or supplement real data for improved performance, thereby providing an inexpensive and scalable solution for data-hungry 3D applications. Project page: github.com/tev-fbk/synthfun3d.
comment : Technical report. 24 pages, 19 figures, 2 tables
☆ PointCNN++: Performant Convolution on Native Points
Lihan Li, Haofeng Zhong, Rui Bu, Mingchao Sun, Wenzheng Chen, Baoquan Chen, Yangyan Li
Existing convolutional learning methods for 3D point cloud data are divided into two paradigms: point-based methods that preserve geometric precision but often face performance challenges, and voxel-based methods that achieve high efficiency through quantization at the cost of geometric fidelity. This loss of precision is a critical bottleneck for tasks such as point cloud registration. We propose PointCNN++, a novel architectural design that fundamentally mitigates this precision-performance trade-off. It \textbf{generalizes sparse convolution from voxels to points}, treating voxel-based convolution as a specialized, degraded case of our more general point-based convolution. First, we introduce a point-centric convolution where the receptive field is centered on the original, high-precision point coordinates. Second, to make this high-fidelity operation performant, we design a computational strategy that operates \textbf{natively} on points. We formulate the convolution on native points as a Matrix-Vector Multiplication and Reduction (MVMR) problem, for which we develop a dedicated, highly-optimized GPU kernel. Experiments demonstrate that PointCNN++ \textbf{uses an order of magnitude less memory and is several times faster} than representative point-based methods. Furthermore, when used as a simple replacement for the voxel-based backbones it generalizes, it \textbf{significantly improves point cloud registration accuracies while proving both more memory-efficient and faster}. PointCNN++ shows that preserving geometric detail and achieving high performance are not mutually exclusive, paving the way for a new class of 3D learning with high fidelity and efficiency. Our code will be open sourced.
☆ TWEO: Transformer s Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
Guang Liang, Jie Shao, Ningyuan Tang, Xinyao Liu, Jianxin Wu
Native FP8 support in modern hardware is essential for training large Transformers, but is severely hindered by extreme activation outliers. Existing solutions either rely on complex mixed-precision engineering or invasive architectural modifications. This paper fundamentally challenges the conventional wisdom that outliers are data-driven. We demonstrate that extreme outliers are a data-independent, mechanically-produced artifact of training, originating from specific structural properties of the weight matrices (i.e., colinearity). Based on this insight, we propose TWEO (Transformers Without Extreme Outliers), a novel, non-invasive loss function. TWEO effectively prevents extreme outliers via a very simple loss term, which reduces outliers from 10000+ to less than 20. TWEO then enables full-model FP8 pre-training with neither engineering tricks nor architectural changes for both LLM and ViT. When standard FP8 training catastrophically collapses, TWEO achieves performance comparable to the BF16 baseline while delivering a 36% increase in training throughput. Also, TWEO enables a new quantization paradigm. Hardware-friendly W8A8 per-tensor static quantization of LLMs, previously considered completely unusable due to outliers, achieves SOTA performance for the first time on TWEO-trained models.
☆ DAONet-YOLOv8: An Occlusion-Aware Dual-Attention Network for Tea Leaf Pest and Disease Detection
Yefeng Wu, Shan Wan, Ling Wu, Yecheng Zhao
Accurate detection of tea leaf pests and diseases in real plantations remains challenging due to complex backgrounds, variable illumination, and frequent occlusions among dense branches and leaves. Existing detectors often suffer from missed detections and false positives in such scenarios. To address these issues, we propose DAONet-YOLOv8, an enhanced YOLOv8 variant with three key improvements: (1) a Dual-Attention Fusion Module (DAFM) that combines convolutional local feature extraction with self-attention based global context modeling to focus on subtle lesion regions while suppressing background noise; (2) an occlusion-aware detection head (Detect-OAHead) that learns the relationship between visible and occluded parts to compensate for missing lesion features; and (3) a C2f-DSConv module employing dynamic synthesis convolutions with multiple kernel shapes to better capture irregular lesion boundaries. Experiments on our real-world tea plantation dataset containing six pest and disease categories demonstrate that DAONet-YOLOv8 achieves 92.97% precision, 92.80% recall, 97.10% mAP@50 and 76.90% mAP@50:95, outperforming the YOLOv8n baseline by 2.34, 4.68, 1.40 and 1.80 percentage points respectively, while reducing parameters by 16.7%. Comparative experiments further confirm that DAONet-YOLOv8 achieves superior performance over mainstream detection models.
☆ Robust 3DGS-based SLAM via Adaptive Kernel Smoothing
Shouhe Zhang, Dayong Ren, Sensen Song, Wenjie Li, Piaopiao Yu, Yurong Qian
In this paper, we challenge the conventional notion in 3DGS-SLAM that rendering quality is the primary determinant of tracking accuracy. We argue that, compared to solely pursuing a perfect scene representation, it is more critical to enhance the robustness of the rasterization process against parameter errors to ensure stable camera pose tracking. To address this challenge, we propose a novel approach that leverages a smooth kernel strategy to enhance the robustness of 3DGS-based SLAM. Unlike conventional methods that focus solely on minimizing rendering error, our core insight is to make the rasterization process more resilient to imperfections in the 3DGS parameters. We hypothesize that by allowing each Gaussian to influence a smoother, wider distribution of pixels during rendering, we can mitigate the detrimental effects of parameter noise from outlier Gaussians. This approach intentionally introduces a controlled blur to the rendered image, which acts as a regularization term, stabilizing the subsequent pose optimization. While a complete redesign of the rasterization pipeline is an ideal solution, we propose a practical and effective alternative that is readily integrated into existing 3DGS frameworks. Our method, termed Corrective Blurry KNN (CB-KNN), adaptively modifies the RGB values and locations of the K-nearest neighboring Gaussians within a local region. This dynamic adjustment generates a smoother local rendering, reducing the impact of erroneous GS parameters on the overall image. Experimental results demonstrate that our approach, while maintaining the overall quality of the scene reconstruction (mapping), significantly improves the robustness and accuracy of camera pose tracking.
☆ Instruction Tuning of Large Language Models for Tabular Data Generation-in One Day ICML 2025
Milad Abdollahzadeh, Abdul Raheem, Zilong Zhao, Uzair Javaid, Kevin Yee, Nalam Venkata Abhishek, Tram Truong-Huu, Biplab Sikdar
Tabular instruction tuning has emerged as a promising research direction for improving LLMs understanding of tabular data. However, the majority of existing works only consider question-answering and reasoning tasks over tabular data, leaving tabular data generation largely unnoticed. In this work, for the first time, we explore the efficacy of instruction tuning in improving LLMs tabular data generation capabilities. More specifically, given the high data and computation requirements of tabular instruction tuning, we aim to address the possibility of instruction tuning for tabular data generation with limited data and computational resources. To achieve this, we first create a high-quality instruction dataset for tabular data, enabling efficient LLM comprehension. We then instruction-tune an open-source LLM (Llama3.1-8B-Instruct) on the training set of this dataset to improve its tabular data generation performance. Our experimental results show that by using our high-quality dataset and instruction-tuning on only 7K instructions with an A100 GPU, for less than 6 hours, we achieve tabular data generation performance on par with the most capable commercial LLM, GPT-4o.
comment : Accepted International Conference on Machine Learning (ICML 2025), 1st Workshop on Foundation Models for Structured Data
☆ Zero-Shot Multi-Criteria Visual Quality Inspection for Semi-Controlled Industrial Environments via Real-Time 3D Digital Twin Simulation
Jose Moises Araya-Martinez, Gautham Mohan, Kenichi Hayakawa Bolaños, Roberto Mendieta, Sarvenaz Sardari, Jens Lambrecht, Jörg Krüger
Early-stage visual quality inspection is vital for achieving Zero-Defect Manufacturing and minimizing production waste in modern industrial environments. However, the complexity of robust visual inspection systems and their extensive data requirements hinder widespread adoption in semi-controlled industrial settings. In this context, we propose a pose-agnostic, zero-shot quality inspection framework that compares real scenes against real-time Digital Twins (DT) in the RGB-D space. Our approach enables efficient real-time DT rendering by semantically describing industrial scenes through object detection and pose estimation of known Computer-Aided Design models. We benchmark tools for real-time, multimodal RGB-D DT creation while tracking consumption of computational resources. Additionally, we provide an extensible and hierarchical annotation strategy for multi-criteria defect detection, unifying pose labelling with logical and structural defect annotations. Based on an automotive use case featuring the quality inspection of an axial flux motor, we demonstrate the effectiveness of our framework. Our results demonstrate detection performace, achieving intersection-over-union (IoU) scores of up to 63.3% compared to ground-truth masks, even if using simple distance measurements under semi-controlled industrial conditions. Our findings lay the groundwork for future research on generalizable, low-data defect detection methods in dynamic manufacturing settings.
☆ Pathryoshka: Compressing Pathology Foundation Models via Multi-Teacher Knowledge Distillation with Nested Embeddings
Christian Grashei, Christian Brechenmacher, Rao Muhammad Umer, Jingsong Liu, Carsten Marr, Ewa Szczurek, Peter J. Schüffler
Pathology foundation models (FMs) have driven significant progress in computational pathology. However, these high-performing models can easily exceed a billion parameters and produce high-dimensional embeddings, thus limiting their applicability for research or clinical use when computing resources are tight. Here, we introduce Pathryoshka, a multi-teacher distillation framework inspired by RADIO distillation and Matryoshka Representation Learning to reduce pathology FM sizes while allowing for adaptable embedding dimensions. We evaluate our framework with a distilled model on ten public pathology benchmarks with varying downstream tasks. Compared to its much larger teachers, Pathryoshka reduces the model size by 86-92% at on-par performance. It outperforms state-of-the-art single-teacher distillation models of comparable size by a median margin of 7.0 in accuracy. By enabling efficient local deployment without sacrificing accuracy or representational richness, Pathryoshka democratizes access to state-of-the-art pathology FMs for the broader research and clinical community.
☆ Vision Bridge Transformer at Scale
Zhenxiong Tan, Zeqing Wang, Xingyi Yang, Songhua Liu, Xinchao Wang
We introduce Vision Bridge Transformer (ViBT), a large-scale instantiation of Brownian Bridge Models designed for conditional generation. Unlike traditional diffusion models that transform noise into data, Bridge Models directly model the trajectory between inputs and outputs, creating an efficient data-to-data translation paradigm. By scaling these models to 20B and 1.3B parameters, we demonstrate their effectiveness for image and video translation tasks. To support this scale, we adopt a Transformer architecture and propose a variance-stabilized velocity-matching objective for robust training. Together, these advances highlight the power of scaling Bridge Models for instruction-based image editing and complex video translation.
☆ GeoWo rl d: Unlocking the Potential of Geometry Models to Facilitate High-Fidelity 3D Scene Generation
Yuhao Wan, Lijuan Liu, Jingzhi Zhou, Zihan Zhou, Xuying Zhang, Dongbo Zhang, Shaohui Jiao, Qibin Hou, Ming-Ming Cheng
Previous works leveraging video models for image-to-3D scene generation tend to suffer from geometric distortions and blurry content. In this paper, we renovate the pipeline of image-to-3D scene generation by unlocking the potential of geometry models and present our GeoWorld. Instead of exploiting geometric information obtained from a single-frame input, we propose to first generate consecutive video frames and then take advantage of the geometry model to provide full-frame geometry features, which contain richer information than single-frame depth maps or camera embeddings used in previous methods, and use these geometry features as geometrical conditions to aid the video generation model. To enhance the consistency of geometric structures, we further propose a geometry alignment loss to provide the model with real-world geometric constraints and a geometry adaptation module to ensure the effective utilization of geometry features. Extensive experiments show that our GeoWorld can generate high-fidelity 3D scenes from a single image and a given camera trajectory, outperforming prior methods both qualitatively and quantitatively. Project Page: https://peaes.github.io/GeoWorld/.
☆ Obstruction reasoning for robotic grasping
Runyu Jiao, Matteo Bortolon, Francesco Giuliari, Alice Fasoli, Sergio Povoli, Guofeng Mei, Yiming Wang, Fabio Poiesi
Successful robotic grasping in cluttered environments not only requires a model to visually ground a target object but also to reason about obstructions that must be cleared beforehand. While current vision-language embodied reasoning models show emergent spatial understanding, they remain limited in terms of obstruction reasoning and accessibility planning. To bridge this gap, we present UNOGrasp, a learning-based vision-language model capable of performing visually-grounded obstruction reasoning to infer the sequence of actions needed to unobstruct the path and grasp the target object. We devise a novel multi-step reasoning process based on obstruction paths originated by the target object. We anchor each reasoning step with obstruction-aware visual cues to incentivize reasoning capability. UNOGrasp combines supervised and reinforcement finetuning through verifiable reasoning rewards. Moreover, we construct UNOBench, a large-scale dataset for both training and benchmarking, based on MetaGraspNetV2, with over 100k obstruction paths annotated by humans with obstruction ratios, contact points, and natural-language instructions. Extensive experiments and real-robot evaluations show that UNOGrasp significantly improves obstruction reasoning and grasp success across both synthetic and real-world environments, outperforming generalist and proprietary alternatives. Project website: https://tev-fbk.github.io/UnoGrasp/.
☆ Fast Multi-view Consistent 3D Editing with Video Priors
Liyi Chen, Ruihuang Li, Guowen Zhang, Pengfei Wang, Lei Zhang
Text-driven 3D editing enables user-friendly 3D object or scene editing with text instructions. Due to the lack of multi-view consistency priors, existing methods typically resort to employing 2D generation or editing models to process each view individually, followed by iterative 2D-3D-2D updating. However, these methods are not only time-consuming but also prone to over-smoothed results because the different editing signals gathered from different views are averaged during the iterative process. In this paper, we propose generative Video Prior based 3D Editing (ViP3DE) to employ the temporal consistency priors from pre-trained video generation models for multi-view consistent 3D editing in a single forward pass. Our key insight is to condition the video generation model on a single edited view to generate other consistent edited views for 3D updating directly, thereby bypassing the iterative editing paradigm. Since 3D updating requires edited views to be paired with specific camera poses, we propose motion-preserved noise blending for the video model to generate edited views at predefined camera poses. In addition, we introduce geometry-aware denoising to further enhance multi-view consistency by integrating 3D geometric priors into video models. Extensive experiments demonstrate that our proposed ViP3DE can achieve high-quality 3D editing results even within a single forward pass, significantly outperforming existing methods in both editing quality and speed.
☆ PowerCLIP: Powerset Alignment for Contrastive Pre-Train ing CVPR 2026
Masaki Kawamura, Nakamasa Inoue, Rintaro Yanagi, Hirokatsu Kataoka, Rio Yokota
Contrastive vision-language pre-training frameworks such as CLIP have demonstrated impressive zero-shot performance across a range of vision-language tasks. Recent studies have shown that aligning individual text tokens with specific image patches or regions enhances fine-grained compositional understanding. However, it remains challenging to capture compositional semantics that span multiple image regions. To address this limitation, we propose PowerCLIP, a novel contrastive pre-training framework enhanced by powerset alignment, which exhaustively optimizes region-to-phrase alignments by minimizing the loss defined between powersets of image regions and textual parse trees. Since the naive powerset construction incurs exponential computational cost due to the combinatorial explosion in the number of region subsets, we introduce efficient non-linear aggregators (NLAs) that reduce complexity from O(2^M) to O(M) with respect to the number of regions M, while approximating the exact loss value with arbitrary precision. Our extensive experiments demonstrate that PowerCLIP outperforms state-of-the-art methods in zero-shot classification and retrieval tasks, underscoring the compositionality and robustness of our approach. Our code will be made publicly available.
comment : Submitted to CVPR 2026
☆ REVEAL: Reasoning -enhanced Forensic Evidence Analysis for Explainable AI-generated Image Detection
Huangsen Cao, Qin Mei, Zhiheng Li, Yuxi Li, Ying Zhang, Chen Li, Zhimeng Zhang, Xin Ding, Yongwei Wang, Jing Lyu, Fei Wu
With the rapid advancement of generative models, visually realistic AI-generated images have become increasingly difficult to distinguish from authentic ones, posing severe threats to social trust and information integrity. Consequently, there is an urgent need for efficient and truly explainable image forensic methods. Recent detection paradigms have shifted towards explainable forensics. However, state-of-the-art approaches primarily rely on post-hoc rationalizations or visual discrimination, lacking a verifiable chain of evidence. This reliance on surface-level pattern matching limits the generation of causally grounded explanations and often results in poor generalization. To bridge this critical gap, we introduce \textbf{REVEAL-Bench}, the first reasoning-enhanced multimodal benchmark for AI-generated image detection that is explicitly structured around a chain-of-evidence derived from multiple lightweight expert models, then records step-by-step reasoning traces and evidential justifications. Building upon this dataset, we propose \textbf{REVEAL} (\underline{R}easoning-\underline{e}nhanced Forensic E\underline{v}id\underline{e}nce \underline{A}na\underline{l}ysis), an effective and explainable forensic framework that integrates detection with a novel expert-grounded reinforcement learning. Our reward mechanism is specially tailored to jointly optimize detection accuracy, explanation fidelity, and logical coherence grounded in explicit forensic evidence, enabling REVEAL to produce fine-grained, interpretable, and verifiable reasoning chains alongside its detection outcomes. Extensive experimental results demonstrate that REVEAL significantly enhances detection accuracy, explanation fidelity, and robust cross-model generalization, benchmarking a new state of the art for explainable image forensics.
☆ Learning to Refuse: Refusal-Aware Reinforcement Fine-Tuning for Hard-Irrelevant Queries in Video Temporal Grounding
Jin-Seop Lee, SungJoon Lee, SeongJun Jung, Boyang Li, Jee-Hyong Lee
Video Temporal Grounding (VTG) aims to localize a temporal segment in a video corresponding to a natural language query. However, existing VTG models assume that a relevant segment always exists, causing them to always predict a target segment even when the query is irrelevant to the video. While recent approaches attempt to handle irrelevant queries, they can only reject those that are entirely unrelated to the video and still fail to handle hard-irrelevant queries that are semantically similar but not actually relevant. To address this, we propose Refusal-Aware Reinforcement Fine-Tuning (RA-RFT) to effectively refuse hard-irrelevant queries in VTG. Our method is based on the Group Relative Policy Optimization (GRPO) framework and integrates four reward objectives-format, refuse-IoU, explain, and query correction-to improve both relevance discrimination and fine-grained semantic reasoning. In addition, to effectively support RA-RFT, we construct a Hard-Irrelevant VTG (HI-VTG) dataset, which includes hard-irrelevant queries and their refusal answers. We demonstrate the effectiveness of our method across various relevance-aware VTG scenarios, including hard-irrelevant VTG, simply-shuffled RA-VTG, and human-annotated RA-VTG settings. We also show that the proposed method is scalable by applying it to various LVLM-based VTG models. Our code is available at https://github.com/JINSUBY/RA-RFT.
comment : 19 pages
☆ Cascaded Robust Rectification for Arbitrary Document Images
Chaoyun Wang, Quanxin Huang, I-Chao Shen, Takeo Igarashi, Nanning Zheng, Caigui Jiang
Document rectification in real-world scenarios poses significant challenges due to extreme variations in camera perspectives and physical distortions. Driven by the insight that complex transformations can be decomposed and resolved progressively, we introduce a novel multi-stage framework that progressively reverses distinct distortion types in a coarse-to-fine manner. Specifically, our framework first performs a global affine transformation to correct perspective distortions arising from the camera's viewpoint, then rectifies geometric deformations resulting from physical paper curling and folding, and finally employs a content-aware iterative process to eliminate fine-grained content distortions. To address limitations in existing evaluation protocols, we also propose two enhanced metrics: layout-aligned OCR metrics (AED/ACER) for a stable assessment that decouples geometric rectification quality from the layout analysis errors of OCR engines, and masked AD/AAD (AD-M/AAD-M) tailored for accurately evaluating geometric distortions in documents with incomplete boundaries. Extensive experiments show that our method establishes new state-of-the-art performance on multiple challenging benchmarks, yielding a substantial reduction of 14.1\%--34.7\% in the AAD metric and demonstrating superior efficacy in real-world applications. The code will be publicly available at https://github.com/chaoyunwang/ArbDR.
☆ InstanceV: Instance-Level Video Generation
Yuheng Chen, Teng Hu, Jiangning Zhang, Zhucun Xue, Ran Yi, Lizhuang Ma
Recent advances in text-to-video diffusion models have enabled the generation of high-quality videos conditioned on textual descriptions. However, most existing text-to-video models rely solely on textual conditions, lacking general fine-grained controllability over video generation. To address this challenge, we propose InstanceV, a video generation framework that enables i) instance-level control and ii) global semantic consistency. Specifically, with the aid of proposed Instance-aware Masked Cross-Attention mechanism, InstanceV maximizes the utilization of additional instance-level grounding information to generate correctly attributed instances at designated spatial locations. To improve overall consistency, We introduce the Shared Timestep-Adaptive Prompt Enhancement module, which connects local instances with global semantics in a parameter-efficient manner. Furthermore, we incorporate Spatially-Aware Unconditional Guidance during both training and inference to alleviate the disappearance of small instances. Finally, we propose a new benchmark, named InstanceBench, which combines general video quality metrics with instance-aware metrics for more comprehensive evaluation on instance-level video generation. Extensive experiments demonstrate that InstanceV not only achieves remarkable instance-level controllability in video generation, but also outperforms existing state-of-the-art models in both general quality and instance-aware metrics across qualitative and quantitative evaluations.
☆ DualCamCt rl : Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generation
Hongfei Zhang, Kanghao Chen, Zixin Zhang, Harold Haodong Chen, Yuanhuiyi Lyu, Yuqi Zhang, Shuai Yang, Kun Zhou, Yingcong Chen
This paper presents DualCamCtrl, a novel end-to-end diffusion model for camera-controlled video generation. Recent works have advanced this field by representing camera poses as ray-based conditions, yet they often lack sufficient scene understanding and geometric awareness. DualCamCtrl specifically targets this limitation by introducing a dual-branch framework that mutually generates camera-consistent RGB and depth sequences. To harmonize these two modalities, we further propose the Semantic Guided Mutual Alignment (SIGMA) mechanism, which performs RGB-depth fusion in a semantics-guided and mutually reinforced manner. These designs collectively enable DualCamCtrl to better disentangle appearance and geometry modeling, generating videos that more faithfully adhere to the specified camera trajectories. Additionally, we analyze and reveal the distinct influence of depth and camera poses across denoising stages and further demonstrate that early and late stages play complementary roles in forming global structure and refining local details. Extensive experiments demonstrate that DualCamCtrl achieves more consistent camera-controlled video generation, with over 40\% reduction in camera motion errors compared with prior methods. Our project page: https://soyouthinkyoucantell.github.io/dualcamctrl\-page/
☆ DNA-Prior: Unsupervised Denoise Anything via Dual-Domain Prior
Yanqi Cheng, Chun-Wun Cheng, Jim Denholm, Thiago Lima, Javier A. Montoya-Zegarra, Richard Goodwin, Carola-Bibiane Schönlieb, Angelica I Aviles-Rivero
Medical imaging pipelines critically rely on robust denoising to stabilise downstream tasks such as segmentation and reconstruction. However, many existing denoisers depend on large annotated datasets or supervised learning, which restricts their usability in clinical environments with heterogeneous modalities and limited ground-truth data. To address this limitation, we introduce DNA-Prior, a universal unsupervised denoising framework that reconstructs clean images directly from corrupted observations through a mathematically principled hybrid prior. DNA-Prior integrates (i) an implicit architectural prior, enforced through a deep network parameterisation, with (ii) an explicit spectral-spatial prior composed of a frequency-domain fidelity term and a spatial regularisation functional. This dual-domain formulation yields a well-structured optimisation problem that jointly preserves global frequency characteristics and local anatomical structure, without requiring any external training data or modality-specific tuning. Experiments across multiple modalities show that DNA achieves consistent noise suppression and structural preservation under diverse noise conditions.
☆ Analyzing Image Beyond Visual Aspect: Image Emotion Classification via Multiple-Affective Captioning
Zibo Zhou, Zhengjun Zhai, Huimin Chen, Wei Dai, Hansen Yang
Image emotion classification (IEC) is a longstanding research field that has received increasing attention with the rapid progress of deep learning. Although recent advances have leveraged the knowledge encoded in pre-trained visual models, their effectiveness is constrained by the "affective gap" , limits the applicability of pre-training knowledge for IEC tasks. It has been demonstrated in psychology that language exhibits high variability, encompasses diverse and abundant information, and can effectively eliminate the "affective gap". Inspired by this, we propose a novel Affective Captioning for Image Emotion Classification (ACIEC) to classify image emotion based on pure texts, which effectively capture the affective information in the image. In our method, a hierarchical multi-level contrastive loss is designed for detecting emotional concepts from images, while an emotional attribute chain-of-thought reasoning is proposed to generate affective sentences. Then, a pre-trained language model is leveraged to synthesize emotional concepts and affective sentences to conduct IEC. Additionally, a contrastive loss based on semantic similarity sampling is designed to solve the problem of large intra-class differences and small inter-class differences in affective datasets. Moreover, we also take the images with embedded texts into consideration, which were ignored by previous studies. Extensive experiments illustrate that our method can effectively bridge the affective gap and achieve superior results on multiple benchmarks.
☆ db-SP: Accelerating Sparse Attention for Visual Generative Models with Dual-Balanced Sequence Parallelism
Siqi Chen, Ke Hong, Tianchen Zhao, Ruiqi Xie, Zhenhua Zhu, Xudong Zhang, Yu Wang
Scaling Diffusion Transformer (DiT) inference via sequence parallelism is critical for reducing latency in visual generation, but is severely hampered by workload imbalance when applied to models employing block-wise sparse attention. The imbalance stems from the inherent variation in sparsity across attention heads and the irregular distribution of dense blocks within the sparse mask, when sequence parallelism is applied along the head dimension (as in Ulysses) or the block dimension (as in Ring Attention). In this paper, we formalize a sparse imbalance ratio to quantify the imbalance, and propose db-SP, a sparsity-aware sequence parallelism technique that tackles the challenge. db-SP contains a dual-level partitioning approach that achieves near-perfect workload balance at both the head and block levels with negligible overhead. Furthermore, to handle the evolving sparsity patterns across denoising steps and layers, db-SP dynamically determines the parallel degrees for the head and block dimensions at runtime. Experimental results demonstrate that db-SP delivers an end-to-end speedup of 1.25x and an attention-specific speedup of 1.40x over state-of-the-art sequence parallel methods on average. Code is available at https://github.com/thu-nics/db-SP.
☆ MathSight: A Benchmark Exploring Have Vision -Language Models Really Seen in University-Level Mathematical Reasoning ?
Yuandong Wang, Yao Cui, Yuxin Zhao, Zhen Yang, Yangfu Zhu, Zhenzhou Shao
Recent advances in Vision-Language Models (VLMs) have achieved impressive progress in multimodal mathematical reasoning. Yet, how much visual information truly contributes to reasoning remains unclear. Existing benchmarks report strong overall performance but seldom isolate the role of the image modality, leaving open whether VLMs genuinely leverage visual understanding or merely depend on linguistic priors. To address this, we present MathSight, a university-level multimodal mathematical reasoning benchmark designed to disentangle and quantify the effect of visual input. Each problem includes multiple visual variants -- original, hand-drawn, photo-captured -- and a text-only condition for controlled comparison. Experiments on state-of-the-art VLMs reveal a consistent trend: the contribution of visual information diminishes with increasing problem difficulty. Remarkably, Qwen3-VL without any image input surpasses both its multimodal variants and GPT-5, underscoring the need for benchmarks like MathSight to advance genuine vision-grounded reasoning in future models.
comment : Comments: 32 pages, 15 figures, 9 tables, includes appendix. Project page: https://cnu-bot-group.github.io/MathSight/
☆ NumeriKontrol: Adding Numeric Control to Diffusion Transformer s for Instruction-based Image Editing
Zhenyu Xu, Xiaoqi Shen, Haotian Nan, Xinyu Zhang
Instruction-based image editing enables intuitive manipulation through natural language commands. However, text instructions alone often lack the precision required for fine-grained control over edit intensity. We introduce NumeriKontrol, a framework that allows users to precisely adjust image attributes using continuous scalar values with common units. NumeriKontrol encodes numeric editing scales via an effective Numeric Adapter and injects them into diffusion models in a plug-and-play manner. Thanks to a task-separated design, our approach supports zero-shot multi-condition editing, allowing users to specify multiple instructions in any order. To provide high-quality supervision, we synthesize precise training data from reliable sources, including high-fidelity rendering engines and DSLR cameras. Our Common Attribute Transform (CAT) dataset covers diverse attribute manipulations with accurate ground-truth scales, enabling NumeriKontrol to function as a simple yet powerful interactive editing studio. Extensive experiments show that NumeriKontrol delivers accurate, continuous, and stable scale control across a wide range of attribute editing scenarios. These contributions advance instruction-based image editing by enabling precise, scalable, and user-controllable image manipulation.
comment : 13 pages, 10 figures
☆ Implementation of a Skin Lesion Detection System for Managing Children with Atopic Dermatitis Based on Ensemble Learning
Soobin Jeon, Sujong Kim, Dongmahn Seo
The amendments made to the Data 3 Act and impact of COVID-19 have fostered the growth of digital healthcare market and promoted the use of medical data in artificial intelligence in South Korea. Atopic dermatitis, a chronic inflammatory skin disease, is diagnosed via subjective evaluations without using objective diagnostic methods, thereby increasing the risk of misdiagnosis. It is also similar to psoriasis in appearance, further complicating its accurate diagnosis. Existing studies on skin diseases have used high-quality dermoscopic image datasets, but such high-quality images cannot be obtained in actual clinical settings. Moreover, existing systems must ensure accuracy and fast response times. To this end, an ensemble learning-based skin lesion detection system (ENSEL) was proposed herein. ENSEL enhanced diagnostic accuracy by integrating various deep learning models via an ensemble approach. Its performance was verified by conducting skin lesion detection experiments using images of skin lesions taken by actual users. Its accuracy and response time were measured using randomly sampled skin disease images. Results revealed that ENSEL achieved high recall in most images and less than 1s s processing speed. This study contributes to the objective diagnosis of skin lesions and promotes the advancement of digital healthcare.
comment : 16pages, 14 figures, 7 tables
☆ SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision -Language Models
Ruosen Zhao, Zhikang Zhang, Jialei Xu, Jiahao Chang, Dong Chen, Lingyun Li, Weijian Sun, Zizhuang Wei
Large vision-language models (VLMs) show strong multimodal understanding but still struggle with 3D spatial reasoning, such as distance estimation, size comparison, and cross-view consistency. Existing 3D-aware methods either depend on auxiliary 3D information or enhance RGB-only VLMs with geometry encoders through shallow feature fusion. We propose SpaceMind, a multimodal large language model explicitly designed for spatial reasoning solely from RGB inputs. The model adopts a dual-encoder architecture, integrating VGGT as a spatial understanding encoder and InternViT as a 2D visual encoder. The key idea is to treat the camera representation as an active guiding modality rather than passive metadata. Specifically, SpaceMind introduces a lightweight Camera-Guided Modality Fusion module before the language model to replace shallow fusion. It applies camera-conditioned biasing to spatial tokens, assigns query-independent weights reflecting their geometric importance, and uses the camera embedding to gate the fused representation. Empirically, SpaceMind establishes new state-of-the-art results on VSI-Bench, SQA3D and SPBench, surpassing both open and proprietary systems on VSI-Bench and SPBench by large margins and achieving state-of-the-art performance on SQA3D. These results demonstrate that camera-guided modality fusion is an effective and practical inductive bias for equipping VLMs with genuinely spatially grounded intelligence. We will release code and model checkpoints to support future research.
☆ Bharat Scene Text: A Novel Comprehensive Dataset and Benchmark for Indian Language Scene Text Understanding
Anik De, Abhirama Subramanyam Penamakuri, Rajeev Yadav, Aditya Rathore, Harshiv Shah, Devesh Sharma, Sagar Agarwal, Pravin Kumar, Anand Mishra
Reading scene text, that is, text appearing in images, has numerous application areas, including assistive technology, search, and e-commerce. Although scene text recognition in English has advanced significantly and is often considered nearly a solved problem, Indian language scene text recognition remains an open challenge. This is due to script diversity, non-standard fonts, and varying writing styles, and, more importantly, the lack of high-quality datasets and open-source models. To address these gaps, we introduce the Bharat Scene Text Dataset (BSTD) - a large-scale and comprehensive benchmark for studying Indian Language Scene Text Recognition. It comprises more than 100K words that span 11 Indian languages and English, sourced from over 6,500 scene images captured across various linguistic regions of India. The dataset is meticulously annotated and supports multiple scene text tasks, including: (i) Scene Text Detection, (ii) Script Identification, (iii) Cropped Word Recognition, and (iv) End-to-End Scene Text Recognition. We evaluated state-of-the-art models originally developed for English by adapting (fine-tuning) them for Indian languages. Our results highlight the challenges and opportunities in Indian language scene text recognition. We believe that this dataset represents a significant step toward advancing research in this domain. All our models and data are open source.
comment : Under Peer Review
☆ Buffer replay enhances the robustness of multimodal learning under missing-modality
Hongye Zhu, Xuan Liu, Yanwen Ba, Jingye Xue, Shigeng Zhang
Missing modalities consistently lead to significant performance degradation in multimodal models. Existing approaches either synthesize missing modalities at high computational cost or apply prompt-based fine-tuning that relies only on adjacent-layer features and overlooks long-distance contextual information, which may offer additional tolerance to errors when one or more modalities are missing. To address this, we introduce REplay Prompting (REP): (1) construct modality-wise feature buffers via a residual bypass to cache early-layer representations and replay them in deeper layers, mitigating information loss as network depth increases; (2) employ a private-shared feature decoupling strategy, where private buffers preserve modality-specific signals and shared buffers encode cross-modal semantics; and (3) design a task-aware dynamic initialization mechanism to configure these buffers differently, improving stability and generalization under diverse missing-modality conditions. Experiments on vision-language, vision-language-audio, and temporal multimodal benchmarks demonstrate that REP consistently outperforms prior methods under both single- and multi-modality missing scenarios, while introducing only negligible parameter overhead. These results establish REP as a lightweight and effective paradigm for robust multimodal learning in challenging missing-modality environments.
☆ Evaluating the Clinical Impact of Generative Inpainting on Bone Age Estimation
Felipe Akio Matsuoka, Eduardo Moreno J. M. Farina, Augusto Sarquis Serpa, Soraya Monteiro, Rodrigo Ragazzini, Nitamar Abdala, Marcelo Straus Takahashi, Felipe Campos Kitamura
Generative foundation models can remove visual artifacts through realistic image inpainting, but their impact on medical AI performance remains uncertain. Pediatric hand radiographs often contain non-anatomical markers, and it is unclear whether inpainting these regions preserves features needed for bone age and gender prediction. To evaluate the clinical reliability of generative model-based inpainting for artifact removal, we used the RSNA Bone Age Challenge dataset, selecting 200 original radiographs and generating 600 inpainted versions with gpt-image-1 using natural language prompts to target non-anatomical artifacts. Downstream performance was assessed with deep learning ensembles for bone age estimation and gender classification, using mean absolute error (MAE) and area under the ROC curve (AUC) as metrics, and pixel intensity distributions to detect structural alterations. Inpainting markedly degraded model performance: bone age MAE increased from 6.26 to 30.11 months, and gender classification AUC decreased from 0.955 to 0.704. Inpainted images displayed pixel-intensity shifts and inconsistencies, indicating structural modifications not corrected by simple calibration. These findings show that, although visually realistic, foundation model-based inpainting can obscure subtle but clinically relevant features and introduce latent bias even when edits are confined to non-diagnostic regions, underscoring the need for rigorous, task-specific validation before integrating such generative tools into clinical AI workflows.
comment : 8 pages, 4 figures
☆ Image Valuation in NeRF-based 3D reconstruction
Grigorios Aris Cheimariotis, Antonis Karakottas, Vangelis Chatzis, Angelos Kanlis, Dimitrios Zarpalas
Data valuation and monetization are becoming increasingly important across domains such as eXtended Reality (XR) and digital media. In the context of 3D scene reconstruction from a set of images -- whether casually or professionally captured -- not all inputs contribute equally to the final output. Neural Radiance Fields (NeRFs) enable photorealistic 3D reconstruction of scenes by optimizing a volumetric radiance field given a set of images. However, in-the-wild scenes often include image captures of varying quality, occlusions, and transient objects, resulting in uneven utility across inputs. In this paper we propose a method to quantify the individual contribution of each image to NeRF-based reconstructions of in-the-wild image sets. Contribution is assessed through reconstruction quality metrics based on PSNR and MSE. We validate our approach by removing low-contributing images during training and measuring the resulting impact on reconstruction fidelity.
comment : Published In International Conference on Computer Analysis of Images and Patterns (pp. 375-385). Cham: Springer Nature Switzerland
☆ GOATex: Geometry & Occlusion-Aware Texturing NeurIPS 2025
Hyunjin Kim, Kunho Kim, Adam Lee, Wonkwang Lee
We present GOATex, a diffusion-based method for 3D mesh texturing that generates high-quality textures for both exterior and interior surfaces. While existing methods perform well on visible regions, they inherently lack mechanisms to handle occluded interiors, resulting in incomplete textures and visible seams. To address this, we introduce an occlusion-aware texturing framework based on the concept of hit levels, which quantify the relative depth of mesh faces via multi-view ray casting. This allows us to partition mesh faces into ordered visibility layers, from outermost to innermost. We then apply a two-stage visibility control strategy that progressively reveals interior regions with structural coherence, followed by texturing each layer using a pretrained diffusion model. To seamlessly merge textures obtained across layers, we propose a soft UV-space blending technique that weighs each texture's contribution based on view-dependent visibility confidence. Empirical results demonstrate that GOATex consistently outperforms existing methods, producing seamless, high-fidelity textures across both visible and occluded surfaces. Unlike prior works, GOATex operates entirely without costly fine-tuning of a pretrained diffusion model and allows separate prompting for exterior and interior mesh regions, enabling fine-grained control over layered appearances. For more qualitative results, please visit our project page: https://goatex3d.github.io/.
comment : Accepted to NeurIPS 2025; Project Page: https://goatex3d.github.io/
☆ Geometry-Consistent 4D Gaussian Splatting for Sparse-Input Dynamic View Synthesis
Yiwei Li, Jiannong Cao, Penghui Ruan, Divya Saxena, Songye Zhu, Yinfeng Cao
Gaussian Splatting has been considered as a novel way for view synthesis of dynamic scenes, which shows great potential in AIoT applications such as digital twins. However, recent dynamic Gaussian Splatting methods significantly degrade when only sparse input views are available, limiting their applicability in practice. The issue arises from the incoherent learning of 4D geometry as input views decrease. This paper presents GC-4DGS, a novel framework that infuses geometric consistency into 4D Gaussian Splatting (4DGS), offering real-time and high-quality dynamic scene rendering from sparse input views. While learning-based Multi-View Stereo (MVS) and monocular depth estimators (MDEs) provide geometry priors, directly integrating these with 4DGS yields suboptimal results due to the ill-posed nature of sparse-input 4D geometric optimization. To address these problems, we introduce a dynamic consistency checking strategy to reduce estimation uncertainties of MVS across spacetime. Furthermore, we propose a global-local depth regularization approach to distill spatiotemporal-consistent geometric information from monocular depths, thereby enhancing the coherent geometry and appearance learning within the 4D volume. Extensive experiments on the popular N3DV and Technicolor datasets validate the effectiveness of GC-4DGS in rendering quality without sacrificing efficiency. Notably, our method outperforms RF-DeRF, the latest dynamic radiance field tailored for sparse-input dynamic view synthesis, and the original 4DGS by 2.62dB and 1.58dB in PSNR, respectively, with seamless deployability on resource-constrained IoT edge devices.
☆ From Illusion to Intention: Visual Rationale Learning for Vision -Language Reasoning
Changpeng Wang, Haozhe Wang, Xi Chen, Junhan Liu, Taofeng Xue, Chong Peng, Donglian Qi, Fangzhen Lin, Yunfeng Yan
Recent advances in vision-language reasoning underscore the importance of thinking with images, where models actively ground their reasoning in visual evidence. Yet, prevailing frameworks treat visual actions as optional tools, boosting metrics but leaving reasoning ungrounded and crops ineffective. This gap gives rise to the illusion of thinking with images: models seem visually grounded but rely on context-agnostic actions that neither refine perception nor guide reasoning toward correct answers. We address this problem by reframing visual actions as core reasoning primitives rather than optional tools, which we term visual rationalization, the visual analogue of textual Chain-of-Thought. Building on this insight, we propose Visual Rationale Learning (ViRL), an end-to-end paradigm that grounds training in the visual rationale itself. ViRL integrates (1) Process Supervision with ground-truth rationales, (2) Objective Alignment via step-level reward shaping, and (3) Fine-Grained Credit Assignment to distinguish correct, redundant, and erroneous actions. By ensuring each action contributes meaningfully to the reasoning chain, ViRL enables models to "get the right answer for the right visual reason". Trained purely with end-to-end RL, ViRL achieves state-of-the-art results across benchmarks spanning perception, hallucination, and reasoning. This work establishes visual rationalization as a task-agnostic, process-grounded paradigm for building transparent, verifiable, and trustworthy vision-language models.
comment : 19 pages, 15 figures
☆ DiskChunGS: Large-Scale 3D Gaussian SLAM Through Chunk-Based Memory Management
Casimir Feldmann, Maximum Wilder-Smith, Vaishakh Patil, Michael Oechsle, Michael Niemeyer, Keisuke Tateno, Marco Hutter
Recent advances in 3D Gaussian Splatting (3DGS) have demonstrated impressive results for novel view synthesis with real-time rendering capabilities. However, integrating 3DGS with SLAM systems faces a fundamental scalability limitation: methods are constrained by GPU memory capacity, restricting reconstruction to small-scale environments. We present DiskChunGS, a scalable 3DGS SLAM system that overcomes this bottleneck through an out-of-core approach that partitions scenes into spatial chunks and maintains only active regions in GPU memory while storing inactive areas on disk. Our architecture integrates seamlessly with existing SLAM frameworks for pose estimation and loop closure, enabling globally consistent reconstruction at scale. We validate DiskChunGS on indoor scenes (Replica, TUM-RGBD), urban driving scenarios (KITTI), and resource-constrained Nvidia Jetson platforms. Our method uniquely completes all 11 KITTI sequences without memory failures while achieving superior visual quality, demonstrating that algorithmic innovation can overcome the memory constraints that have limited previous 3DGS SLAM methods.
☆ Geodiffussr: Generative Terrain Texturing with Elevation Fidelity
Tai Inui, Alexander Matsumura, Edgar Simo-Serra
Large-scale terrain generation remains a labor-intensive task in computer graphics. We introduce Geodiffussr, a flow-matching pipeline that synthesizes text-guided texture maps while strictly adhering to a supplied Digital Elevation Map (DEM). The core mechanism is multi-scale content aggregation (MCA): DEM features from a pretrained encoder are injected into UNet blocks at multiple resolutions to enforce global-to-local elevation consistency. Compared with a non-MCA baseline, MCA markedly improves visual fidelity and strengthens height-appearance coupling (FID $\downarrow$ 49.16%, LPIPS $\downarrow$ 32.33%, $Δ$dCor $\downarrow$ to 0.0016). To train and evaluate Geodiffussr, we assemble a globally distributed, biome- and climate-stratified corpus of triplets pairing SRTM-derived DEMs with Sentinel-2 imagery and vision-grounded natural-language captions that describe visible land cover. We position Geodiffussr as a strong baseline and step toward controllable 2.5D landscape generation for coarse-scale ideation and previz, complementary to physically based terrain and ecosystem simulators.
☆ JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization
Yunlong Lin, Linqing Wang, Kunjie Lin, Zixu Lin, Kaixiong Gong, Wenbo Li, Bin Lin, Zhenxi Li, Shiyi Zhang, Yuyang Peng, Wenxun Dai, Xinghao Ding, Chunyu Wang, Qinglin Lu
Agent-based editing models have substantially advanced interactive experiences, processing quality, and creative flexibility. However, two critical challenges persist: (1) instruction hallucination, text-only chain-of-thought (CoT) reasoning cannot fully prevent factual errors due to inherent information bottlenecks; (2) reward hacking, dynamic policy optimization against static reward models allows agents to exploit flaws in reward functions. To address these issues, we propose JarvisEvo, a unified image editing agent that emulates an expert human designer by iteratively editing, selecting appropriate tools, evaluating results, and reflecting on its own decisions to refine outcomes. JarvisEvo offers three key advantages: (1) an interleaved multimodal chain-of-thought (iMCoT) reasoning mechanism that enhances instruction following and editing quality; (2) a synergistic editor-evaluator policy optimization (SEPO) framework that enables self-improvement without external rewards, effectively mitigating reward hacking; and (3) support for both global and local fine-grained editing through seamless integration of Adobe Lightroom. On ArtEdit-Bench, JarvisEvo outperforms Nano-Banana by an average of 18.95% on preservative editing metrics, including a substantial 44.96% improvement in pixel-level content fidelity.
comment : 31 pages, 18 figures
☆ MrGS: Multi-modal Radiance Fields with 3D Gaussian Splatting for RGB-Thermal Novel View Synthesis ICRA 2025
Minseong Kweon, Janghyun Kim, Ukcheol Shin, Jinsun Park
Recent advances in Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS) have achieved considerable performance in RGB scene reconstruction. However, multi-modal rendering that incorporates thermal infrared imagery remains largely underexplored. Existing approaches tend to neglect distinctive thermal characteristics, such as heat conduction and the Lambertian property. In this study, we introduce MrGS, a multi-modal radiance field based on 3DGS that simultaneously reconstructs both RGB and thermal 3D scenes. Specifically, MrGS derives RGB- and thermal-related information from a single appearance feature through orthogonal feature extraction and employs view-dependent or view-independent embedding strategies depending on the degree of Lambertian reflectance exhibited by each modality. Furthermore, we leverage two physics-based principles to effectively model thermal-domain phenomena. First, we integrate Fourier's law of heat conduction prior to alpha blending to model intensity interpolation caused by thermal conduction between neighboring Gaussians. Second, we apply the Stefan-Boltzmann law and the inverse-square law to formulate a depth-aware thermal radiation map that imposes additional geometric constraints on thermal rendering. Experimental results demonstrate that the proposed MrGS achieves high-fidelity RGB-T scene reconstruction while reducing the number of Gaussians.
comment : Accepted at Thermal Infrared in Robotics (TIRO) Workshop, ICRA 2025 (Best Poster Award)
☆ Optimizer Sensitivity In Vision Transformer based Iris Recognition: Adamw Vs Sgd Vs Rmsprop
Moh Imam Faiz, Aviv Yuniar Rahman, Rangga Pahlevi Putra
The security of biometric authentication is increasingly critical as digital identity systems expand. Iris recognition offers high reliability due to its distinctive and stable texture patterns. Recent progress in deep learning, especially Vision Transformers ViT, has improved visual recognition performance. Yet, the effect of optimizer choice on ViT-based biometric systems remains understudied. This work evaluates how different optimizers influence the accuracy and stability of ViT for iris recognition, providing insights to enhance the robustness of biometric identification models.
comment : 16 pages, 5 figures
☆ Guiding Visual Autoregressive Models through Spectrum Weakening
Chaoyang Wang, Tianmeng Yang, Jingdong Wang, Yunhai Tong
Classifier-free guidance (CFG) has become a widely adopted and practical approach for enhancing generation quality and improving condition alignment. Recent studies have explored guidance mechanisms for unconditional generation, yet these approaches remain fundamentally tied to assumptions specific to diffusion models. In this work, we propose a spectrum-weakening framework for visual autoregressive (AR) models. This method works without the need for re-training, specific conditions, or any architectural modifications. It achieves this by constructing a controllable weak model in the spectral domain. We theoretically show that invertible spectral transformations preserve information, while selectively retaining only a subset of spectrum introduces controlled information reduction. Based on this insight, we perform spectrum selection along the channel dimension of internal representations, which avoids the structural constraints imposed by diffusion models. We further introduce two spectrum renormalization strategies that ensures numerical stability during the weakening process. Extensive experiments were conducted on both discrete and continuous AR models, with text or class conditioning. The results demonstrate that our method enables high-quality unconditional generation while maintaining strong prompt alignment for conditional generation.
☆ MIMM-X: Disentangling Spurious Cor relation s for Medical Image Analysis
Louisa Fay, Hajer Reguigui, Bin Yang, Sergios Gatidis, Thomas Küstner
Deep learning models can excel on medical tasks, yet often experience spurious correlations, known as shortcut learning, leading to poor generalization in new environments. Particularly in medical imaging, where multiple spurious correlations can coexist, misclassifications can have severe consequences. We propose MIMM-X, a framework that disentangles causal features from multiple spurious correlations by minimizing their mutual information. It enables predictions based on true underlying causal relationships rather than dataset-specific shortcuts. We evaluate MIMM-X on three datasets (UK Biobank, NAKO, CheXpert) across two imaging modalities (MRI and X-ray). Results demonstrate that MIMM-X effectively mitigates shortcut learning of multiple spurious correlations.
☆ MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation
Yuta Oshima, Daiki Miyake, Kohsei Matsutani, Yusuke Iwasawa, Masahiro Suzuki, Yutaka Matsuo, Hiroki Furuta
Recent text-to-image generation models have acquired the ability of multi-reference generation and editing; the ability to inherit the appearance of subjects from multiple reference images and re-render them under new contexts. However, the existing benchmark datasets often focus on the generation with single or a few reference images, which prevents us from measuring the progress on how model performance advances or pointing out their weaknesses, under different multi-reference conditions. In addition, their task definitions are still vague, typically limited to axes such as "what to edit" or "how many references are given", and therefore fail to capture the intrinsic difficulty of multi-reference settings. To address this gap, we introduce $\textbf{MultiBanana}$, which is carefully designed to assesses the edge of model capabilities by widely covering multi-reference-specific problems at scale: (1) varying the number of references, (2) domain mismatch among references (e.g., photo vs. anime), (3) scale mismatch between reference and target scenes, (4) references containing rare concepts (e.g., a red banana), and (5) multilingual textual references for rendering. Our analysis among a variety of text-to-image models reveals their superior performances, typical failure modes, and areas for improvement. MultiBanana will be released as an open benchmark to push the boundaries and establish a standardized basis for fair comparison in multi-reference image generation. Our data and code are available at https://github.com/matsuolab/multibanana .
comment : Code: https://github.com/matsuolab/multibanana
☆ Convolutional Feature Noise Reduction for 2D Cardiac MR Image Segmentation
Hong Zheng, Nan Mu, Han Su, Lin Feng, Xiaoning Li
Noise reduction constitutes a crucial operation within Digital Signal Processing. Regrettably, it frequently remains neglected when dealing with the processing of convolutional features in segmentation networks. This oversight could trigger the butterfly effect, impairing the subsequent outcomes within the entire feature system. To complete this void, we consider convolutional features following Gaussian distributions as feature signal matrices and then present a simple and effective feature filter in this study. The proposed filter is fundamentally a low-amplitude pass filter primarily aimed at minimizing noise in feature signal inputs and is named Convolutional Feature Filter (CFF). We conducted experiments on two established 2D segmentation networks and two public cardiac MR image datasets to validate the effectiveness of the CFF, and the experimental findings demonstrated a decrease in noise within the feature signal matrices. To enable a numerical observation and analysis of this reduction, we developed a binarization equation to calculate the information entropy of feature signals.
☆ Ovis-Image Technical Report
Guo-Hua Wang, Liangfu Cao, Tianyu Cui, Minghao Fu, Xiaohao Chen, Pengxin Zhan, Jianshan Zhao, Lan Li, Bowen Fu, Jiaqi Liu, Qing-Guo Chen
We introduce $\textbf{Ovis-Image}$, a 7B text-to-image model specifically optimized for high-quality text rendering, designed to operate efficiently under stringent computational constraints. Built upon our previous Ovis-U1 framework, Ovis-Image integrates a diffusion-based visual decoder with the stronger Ovis 2.5 multimodal backbone, leveraging a text-centric training pipeline that combines large-scale pre-training with carefully tailored post-training refinements. Despite its compact architecture, Ovis-Image achieves text rendering performance on par with significantly larger open models such as Qwen-Image and approaches closed-source systems like Seedream and GPT4o. Crucially, the model remains deployable on a single high-end GPU with moderate memory, narrowing the gap between frontier-level text rendering and practical deployment. Our results indicate that combining a strong multimodal backbone with a carefully designed, text-focused training recipe is sufficient to achieve reliable bilingual text rendering without resorting to oversized or proprietary models.
comment : Code is released at https://github.com/AIDC-AI/Ovis-Image
☆ McSc: Motion-Corrective Preference Alignment for Video Generation with Self-Critic Hierarchical Reasoning
Qiushi Yang, Yingjie Chen, Yuan Yao, Yifang Men, Huaizhuo Liu, Miaomiao Cui
Text-to-video (T2V) generation has achieved remarkable progress in producing high-quality videos aligned with textual prompts. However, aligning synthesized videos with nuanced human preference remains challenging due to the subjective and multifaceted nature of human judgment. Existing video preference alignment methods rely on costly human annotations or utilize proxy metrics to predict preference, which lacks the understanding of human preference logic. Moreover, they usually directly align T2V models with the overall preference distribution, ignoring potential conflict dimensions like motion dynamics and visual quality, which may bias models towards low-motion content. To address these issues, we present Motion-corrective alignment with Self-critic hierarchical Reasoning (McSc), a three-stage reinforcement learning framework for robust preference modeling and alignment. Firstly, Self-critic Dimensional Reasoning (ScDR) trains a generative reward model (RM) to decompose preferences into per-dimension assessments, using self-critic reasoning chains for reliable learning. Secondly, to achieve holistic video comparison, we introduce Hierarchical Comparative Reasoning (HCR) for structural multi-dimensional reasoning with hierarchical reward supervision. Finally, using RM-preferred videos, we propose Motion-corrective Direct Preference Optimization (McDPO) to optimize T2V models, while dynamically re-weighting alignment objective to mitigate bias towards low-motion content. Experiments show that McSc achieves superior performance in human preference alignment and generates videos with high-motion dynamic.
☆ BlockVid: Block Diffusion for High-Quality and Consistent Minute-Long Video Generation
Zeyu Zhang, Shuning Chang, Yuanyu He, Yizeng Han, Jiasheng Tang, Fan Wang, Bohan Zhuang
Generating minute-long videos is a critical step toward developing world models, providing a foundation for realistic extended scenes and advanced AI simulators. The emerging semi-autoregressive (block diffusion) paradigm integrates the strengths of diffusion and autoregressive models, enabling arbitrary-length video generation and improving inference efficiency through KV caching and parallel sampling. However, it yet faces two enduring challenges: (i) KV-cache-induced long-horizon error accumulation, and (ii) the lack of fine-grained long-video benchmarks and coherence-aware metrics. To overcome these limitations, we propose BlockVid, a novel block diffusion framework equipped with semantic-aware sparse KV cache, an effective training strategy called Block Forcing, and dedicated chunk-wise noise scheduling and shuffling to reduce error propagation and enhance temporal consistency. We further introduce LV-Bench, a fine-grained benchmark for minute-long videos, complete with new metrics evaluating long-range coherence. Extensive experiments on VBench and LV-Bench demonstrate that BlockVid consistently outperforms existing methods in generating high-quality, coherent minute-long videos. In particular, it achieves a 22.2% improvement on VDE Subject and a 19.4% improvement on VDE Clarity in LV-Bench over the state of the art approaches. Project website: https://ziplab.co/BlockVid. Inferix (Code): https://github.com/alibaba-damo-academy/Inferix.
☆ Taming the Light: Illumination-Invariant Semantic 3DGS-SLAM
Shouhe Zhang, Dayong Ren, Sensen Song, Yurong Qian, Zhenhong Jia
Extreme exposure degrades both the 3D map reconstruction and semantic segmentation accuracy, which is particularly detrimental to tightly-coupled systems. To achieve illumination invariance, we propose a novel semantic SLAM framework with two designs. First, the Intrinsic Appearance Normalization (IAN) module proactively disentangles the scene's intrinsic properties, such as albedo, from transient lighting. By learning a standardized, illumination-invariant appearance model, it assigns a stable and consistent color representation to each Gaussian primitive. Second, the Dynamic Radiance Balancing Loss (DRB-Loss) reactively handles frames with extreme exposure. It activates only when an image's exposure is poor, operating directly on the radiance field to guide targeted optimization. This prevents error accumulation from extreme lighting without compromising performance under normal conditions. The synergy between IAN's proactive invariance and DRB-Loss's reactive correction endows our system with unprecedented robustness. Evaluations on public datasets demonstrate state-of-the-art performance in camera tracking, map quality, and semantic and geometric accuracy.
☆ HMR3D: Hierarchical Multimodal Representation for 3D Scene Understanding with Large Vision -Language Model
Chen Li, Eric Peh, Basura Fernando
Recent advances in large vision-language models (VLMs) have shown significant promise for 3D scene understanding. Existing VLM-based approaches typically align 3D scene features with the VLM's embedding space. However, this implicit alignment often yields suboptimal performance due to the scarcity of 3D data and the inherent complexity of spatial relationships in 3D environments. To address these limitations, we propose a novel hierarchical multimodal representation for 3D scene reasoning that explicitly aligns with VLMs at the input space by leveraging both multi-view images and text descriptions. The text descriptions capture spatial relationships by referencing the 3D coordinates of detected objects, while the multi-view images include a top-down perspective and four directional views (forward, left, right, and backward), ensuring comprehensive scene coverage. Additionally, we introduce a hierarchical feature representation that aggregates patch-level image features into view-level and scene-level representations, enabling the model to reason over both local and global scene context. Experimental results on both situated 3D Q&A and general 3D Q&A benchmarks demonstrate the effectiveness of our approach.
☆ Contrastive Heliophysical Image Pretrain ing for Solar Dynamics Observatory Records
Shiyu Shen, Zhe Gao, Taifeng Chai, Yang Huang, Bin Pan
Deep learning has revolutionized solar image analysis, yet most approaches train task-specific encoders from scratch or rely on natural-image pretraining that ignores the unique characteristics of Solar Dynamics Observatory (SDO) data. We introduce SolarCHIP, a family of contrastively pretrained visual backbones tailored to multi-instrument SDO observations. SolarCHIP addresses three key challenges in solar imaging: multimodal sensing across AIA and HMI instruments, weak inter-class separability due to slow temporal evolution, and strong intra-class variability with sparse activity signals. Our pretraining framework employs a multi-granularity contrastive objective that jointly aligns (1) global class tokens across co-temporal AIA-HMI pairs to enhance temporal discrimination, (2) local patch tokens at fixed spatial indices to enforce position-consistent, modality-invariant features, and (3) intra-sample patches across different spatial locations to preserve fine-grained spatial structure. We train both CNN- and Vision Transformer-based autoencoders and demonstrate their effectiveness on two downstream tasks: cross-modal translation between HMI and AIA passbands via ControlNet, and full-disk flare classification. Experimental results show that SolarCHIP achieves state-of-the-art performance across both tasks, with particularly strong gains in low-resource settings where labeled data is limited. Ablation studies confirm that each contrastive component contributes essential discriminative capacity at different granularities. By publicly releasing pretrained weights and training code, we provide the heliophysics community with a practical, plug-and-play feature extractor that reduces computational requirements, improves label efficiency, and establishes a reusable foundation for diverse solar imaging applications.
☆ RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video
Haiyang Mei, Qiming Huang, Hai Ci, Mike Zheng Shou
Accurate robot segmentation is a fundamental capability for robotic perception. It enables precise visual servoing for VLA systems, scalable robot-centric data augmentation, accurate real-to-sim transfer, and reliable safety monitoring in dynamic human-robot environments. Despite the strong capabilities of modern segmentation models, surprisingly it remains challenging to segment robots. This is due to robot embodiment diversity, appearance ambiguity, structural complexity, and rapid shape changes. Embracing these challenges, we introduce RobotSeg, a foundation model for robot segmentation in image and video. RobotSeg is built upon the versatile SAM 2 foundation model but addresses its three limitations for robot segmentation, namely the lack of adaptation to articulated robots, reliance on manual prompts, and the need for per-frame training mask annotations, by introducing a structure-enhanced memory associator, a robot prompt generator, and a label-efficient training strategy. These innovations collectively enable a structure-aware, automatic, and label-efficient solution. We further construct the video robot segmentation (VRS) dataset comprising over 2.8k videos (138k frames) with diverse robot embodiments and environments. Extensive experiments demonstrate that RobotSeg achieves state-of-the-art performance on both images and videos, establishing a strong foundation for future advances in robot perception.
comment : Project page: https://github.com/showlab/RobotSeg
☆ Do We Need Perfect Data? Leveraging Noise for Domain Generalized Segmentation AAAI 2026
Taeyeong Kim, SeungJoon Lee, Jung Uk Kim, MyeongAh Cho
Domain generalization in semantic segmentation faces challenges from domain shifts, particularly under adverse conditions. While diffusion-based data generation methods show promise, they introduce inherent misalignment between generated images and semantic masks. This paper presents FLEX-Seg (FLexible Edge eXploitation for Segmentation), a framework that transforms this limitation into an opportunity for robust learning. FLEX-Seg comprises three key components: (1) Granular Adaptive Prototypes that captures boundary characteristics across multiple scales, (2) Uncertainty Boundary Emphasis that dynamically adjusts learning emphasis based on prediction entropy, and (3) Hardness-Aware Sampling that progressively focuses on challenging examples. By leveraging inherent misalignment rather than enforcing strict alignment, FLEX-Seg learns robust representations while capturing rich stylistic variations. Experiments across five real-world datasets demonstrate consistent improvements over state-of-the-art methods, achieving 2.44% and 2.63% mIoU gains on ACDC and Dark Zurich. Our findings validate that adaptive strategies for handling imperfect synthetic data lead to superior domain generalization. Code is available at https://github.com/VisualScienceLab-KHU/FLEX-Seg.
comment : Accepted by AAAI 2026
☆ Visual Puns from Idioms: An Iterative LLM-T2IM-MLLM Framework ICASSP 2026
Kelaiti Xiao, Liang Yang, Dongyu Zhang, Paerhati Tulajiang, Hongfei Lin
We study idiom-based visual puns--images that align an idiom's literal and figurative meanings--and present an iterative framework that coordinates a large language model (LLM), a text-to-image model (T2IM), and a multimodal LLM (MLLM) for automatic generation and evaluation. Given an idiom, the system iteratively (i) generates detailed visual prompts, (ii) synthesizes an image, (iii) infers the idiom from the image, and (iv) refines the prompt until recognition succeeds or a step limit is reached. Using 1,000 idioms as inputs, we synthesize a corresponding dataset of visual pun images with paired prompts, enabling benchmarking of both generation and understanding. Experiments across 10 LLMs, 10 MLLMs, and one T2IM (Qwen-Image) show that MLLM choice is the primary performance driver: GPT achieves the highest accuracies, Gemini follows, and the best open-source MLLM (Gemma) is competitive with some closed models. On the LLM side, Claude attains the strongest average performance for prompt generation.
comment : Submitted to ICASSP 2026 (under review)
☆ One-to-All Animation: Alignment-Free Character Animation and Image Pose Transfe
Shijun Shi, Jing Xu, Zhihang Li, Chunli Peng, Xiaoda Yang, Lijing Lu, Kai Hu, Jiangning Zhang
Recent advances in diffusion models have greatly improved pose-driven character animation. However, existing methods are limited to spatially aligned reference-pose pairs with matched skeletal structures. Handling reference-pose misalignment remains unsolved. To address this, we present One-to-All Animation, a unified framework for high-fidelity character animation and image pose transfer for references with arbitrary layouts. First, to handle spatially misaligned reference, we reformulate training as a self-supervised outpainting task that transforms diverse-layout reference into a unified occluded-input format. Second, to process partially visible reference, we design a reference extractor for comprehensive identity feature extraction. Further, we integrate hybrid reference fusion attention to handle varying resolutions and dynamic sequence lengths. Finally, from the perspective of generation quality, we introduce identity-robust pose control that decouples appearance from skeletal structure to mitigate pose overfitting, and a token replace strategy for coherent long-video generation. Extensive experiments show that our method outperforms existing approaches. The code and model will be available at https://github.com/ssj9596/One-to-All-Animation.
comment : Project Page:https://ssj9596.github.io/one-to-all-animation-project/
☆ DenoiseGS: Gaussian Reconstruction Model for Burst Denoising
Yongsen Cheng, Yuanhao Cai, Yulun Zhang
Burst denoising methods are crucial for enhancing images captured on handheld devices, but they often struggle with large motion or suffer from prohibitive computational costs. In this paper, we propose DenoiseGS, the first framework to leverage the efficiency of 3D Gaussian Splatting for burst denoising. Our approach addresses two key challenges when applying feedforward Gaussian reconsturction model to noisy inputs: the degradation of Gaussian point clouds and the loss of fine details. To this end, we propose a Gaussian self-consistency (GSC) loss, which regularizes the geometry predicted from noisy inputs with high-quality Gaussian point clouds. These point clouds are generated from clean inputs by the same model that we are training, thereby alleviating potential bias or domain gaps. Additionally, we introduce a log-weighted frequency (LWF) loss to strengthen supervision within the spectral domain, effectively preserving fine-grained details. The LWF loss adaptively weights frequency discrepancies in a logarithmic manner, emphasizing challenging high-frequency details. Extensive experiments demonstrate that DenoiseGS significantly exceeds the state-of-the-art NeRF-based methods on both burst denoising and novel view synthesis under noisy conditions, while achieving \textbf{250$\times$} faster inference speed. Code and models are released at https://github.com/yscheng04/DenoiseGS.
☆ Barcode and QR Code Object Detection: An Experimental Study on YOLOv8 Models
Kushagra Pandya, Heli Hathi, Het Buch, Ravikumar R N, Shailendrasinh Chauhan, Sushil Kumar Singh
This research work dives into an in-depth evaluation of the YOLOv8 (You Only Look Once) algorithm's efficiency in object detection, specially focusing on Barcode and QR code recognition. Utilizing the real-time detection abilities of YOLOv8, we performed a study aimed at enhancing its talent in swiftly and correctly figuring out objects. Through large training and high-quality-tuning on Kaggle datasets tailored for Barcode and QR code detection, our goal became to optimize YOLOv8's overall performance throughout numerous situations and environments. The look encompasses the assessment of YOLOv8 throughout special version iterations: Nano, Small, and Medium, with a meticulous attention on precision, recall, and F1 assessment metrics. The consequences exhibit large improvements in object detection accuracy with every subsequent model refinement. Specifically, we achieved an accuracy of 88.95% for the nano model, 97.10% for the small model, and 94.10% for the medium version, showcasing the incremental improvements finished via model scaling. Our findings highlight the big strides made through YOLOv8 in pushing the limits of computer vision, ensuring its function as a milestone within the subject of object detection. This study sheds light on how model scaling affects object recognition, increasing the concept of deep learning-based computer creative and prescient techniques.
comment : 7 Pages, 16 figures, Presented at 2024 International Conference on Emerging Innovations and Advanced Computing (INNOCOMP) Conference
☆ Robust Image Self-Recovery against Tampering using Watermark Generation with Pixel Shuffling
Minyoung Kim, Paul Hongsuck Seo
The rapid growth of Artificial Intelligence-Generated Content (AIGC) raises concerns about the authenticity of digital media. In this context, image self-recovery, reconstructing original content from its manipulated version, offers a practical solution for understanding the attacker's intent and restoring trustworthy data. However, existing methods often fail to accurately recover tampered regions, falling short of the primary goal of self-recovery. To address this challenge, we propose ReImage, a neural watermarking-based self-recovery framework that embeds a shuffled version of the target image into itself as a watermark. We design a generator that produces watermarks optimized for neural watermarking and introduce an image enhancement module to refine the recovered image. We further analyze and resolve key limitations of shuffled watermarking, enabling its effective use in self-recovery. We demonstrate that ReImage achieves state-of-the-art performance across diverse tampering scenarios, consistently producing high-quality recovered images. The code and pretrained models will be released upon publication.
comment : 22 pages, 12 figures, 14 tables
☆ NeuMatC: A General Neural Framework for Fast Parametric Matrix Operation
Chuan Wang, Xi-le Zhao, Zhilong Han, Liang Li, Deyu Meng, Michael K. Ng
Matrix operations (e.g., inversion and singular value decomposition (SVD)) are fundamental in science and engineering. In many emerging real-world applications (such as wireless communication and signal processing), these operations must be performed repeatedly over matrices with parameters varying continuously. However, conventional methods tackle each matrix operation independently, underexploring the inherent low-rankness and continuity along the parameter dimension, resulting in significantly redundant computation. To address this challenge, we propose \textbf{\textit{Neural Matrix Computation Framework} (NeuMatC)}, which elegantly tackles general parametric matrix operation tasks by leveraging the underlying low-rankness and continuity along the parameter dimension. Specifically, NeuMatC unsupervisedly learns a low-rank and continuous mapping from parameters to their corresponding matrix operation results. Once trained, NeuMatC enables efficient computations at arbitrary parameters using only a few basic operations (e.g., matrix multiplications and nonlinear activations), significantly reducing redundant computations. Experimental results on both synthetic and real-world datasets demonstrate the promising performance of NeuMatC, exemplified by over $3\times$ speedup in parametric inversion and $10\times$ speedup in parametric SVD compared to the widely used NumPy baseline in wireless communication, while maintaining acceptable accuracy.
☆ Artwork Interpretation with Vision Language Models: A Case Study on Emotions and Emotion Symbols AACL
Sebastian Padó, Kerstin Thomas
Emotions are a fundamental aspect of artistic expression. Due to their abstract nature, there is a broad spectrum of emotion realization in artworks. These are subject to historical change and their analysis requires expertise in art history. In this article, we investigate which aspects of emotional expression can be detected by current (2025) vision language models (VLMs). We present a case study of three VLMs (Llava-Llama and two Qwen models) in which we ask these models four sets of questions of increasing complexity about artworks (general content, emotional content, expression of emotions, and emotion symbols) and carry out a qualitative expert evaluation. We find that the VLMs recognize the content of the images surprisingly well and often also which emotions they depict and how they are expressed. The models perform best for concrete images but fail for highly abstract or highly symbolic images. Reliable recognition of symbols remains fundamentally difficult. Furthermore, the models continue to exhibit the well-known LLM weakness of providing inconsistent answers to related questions.
comment : Accepted for publication at the IJCNLP-AACL workshop on Multimodal Models for Low-Resource Contexts and Social Impact
☆ MICCAI STS 2024 Challenge: Semi-Supervised Instance-Level Tooth Segmentation in Panoramic X-ray and CBCT Images
Yaqi Wang, Zhi Li, Chengyu Wu, Jun Liu, Yifan Zhang, Jiaxue Ni, Qian Luo, Jialuo Chen, Hongyuan Zhang, Jin Liu, Can Han, Kaiwen Fu, Changkai Ji, Xinxu Cai, Jing Hao, Zhihao Zheng, Shi Xu, Junqiang Chen, Qianni Zhang, Dahong Qian, Shuai Wang, Huiyu Zhou
Orthopantomogram (OPGs) and Cone-Beam Computed Tomography (CBCT) are vital for dentistry, but creating large datasets for automated tooth segmentation is hindered by the labor-intensive process of manual instance-level annotation. This research aimed to benchmark and advance semi-supervised learning (SSL) as a solution for this data scarcity problem. We organized the 2nd Semi-supervised Teeth Segmentation (STS 2024) Challenge at MICCAI 2024. We provided a large-scale dataset comprising over 90,000 2D images and 3D axial slices, which includes 2,380 OPG images and 330 CBCT scans, all featuring detailed instance-level FDI annotations on part of the data. The challenge attracted 114 (OPG) and 106 (CBCT) registered teams. To ensure algorithmic excellence and full transparency, we rigorously evaluated the valid, open-source submissions from the top 10 (OPG) and top 5 (CBCT) teams, respectively. All successful submissions were deep learning-based SSL methods. The winning semi-supervised models demonstrated impressive performance gains over a fully-supervised nnU-Net baseline trained only on the labeled data. For the 2D OPG track, the top method improved the Instance Affinity (IA) score by over 44 percentage points. For the 3D CBCT track, the winning approach boosted the Instance Dice score by 61 percentage points. This challenge confirms the substantial benefit of SSL for complex, instance-level medical image segmentation tasks where labeled data is scarce. The most effective approaches consistently leveraged hybrid semi-supervised frameworks that combined knowledge from foundational models like SAM with multi-stage, coarse-to-fine refinement pipelines. Both the challenge dataset and the participants' submitted code have been made publicly available on GitHub (https://github.com/ricoleehduu/STS-Challenge-2024), ensuring transparency and reproducibility.
☆ ViGG: Robust RGB-D Point Cloud Registration using Visual-Geometric Mutual Guidance WACV 2026
Congjia Chen, Shen Yan, Yufu Qu
Point cloud registration is a fundamental task in 3D vision. Most existing methods only use geometric information for registration. Recently proposed RGB-D registration methods primarily focus on feature fusion or improving feature learning, which limits their ability to exploit image information and hinders their practical applicability. In this paper, we propose ViGG, a robust RGB-D registration method using mutual guidance. First, we solve clique alignment in a visual-geometric combination form, employing a geometric guidance design to suppress ambiguous cliques. Second, to mitigate accuracy degradation caused by noise in visual matches, we propose a visual-guided geometric matching method that utilizes visual priors to determine the search space, enabling the extraction of high-quality, noise-insensitive correspondences. This mutual guidance strategy brings our method superior robustness, making it applicable for various RGB-D registration tasks. The experiments on 3DMatch, ScanNet and KITTI datasets show that our method outperforms recent state-of-the-art methods in both learning-free and learning-based settings. Code is available at https://github.com/ccjccjccj/ViGG.
comment : Accepted by WACV 2026
☆ See, Rank, and Filter: Important Word-Aware Clip Filtering via Scene Understanding for Moment Retrieval and Highlight Detection
YuEun Lee, Jung Uk Kim
Video moment retrieval (MR) and highlight detection (HD) with natural language queries aim to localize relevant moments and key highlights in a video clips. However, existing methods overlook the importance of individual words, treating the entire text query and video clips as a black-box, which hinders contextual understanding. In this paper, we propose a novel approach that enables fine-grained clip filtering by identifying and prioritizing important words in the query. Our method integrates image-text scene understanding through Multimodal Large Language Models (MLLMs) and enhances the semantic understanding of video clips. We introduce a feature enhancement module (FEM) to capture important words from the query and a ranking-based filtering module (RFM) to iteratively refine video clips based on their relevance to these important words. Extensive experiments demonstrate that our approach significantly outperforms existing state-of-the-art methods, achieving superior performance in both MR and HD tasks. Our code is available at: https://github.com/VisualAIKHU/SRF.
☆ Leveraging Textual Compositional Reasoning for Robust Change Captioning AAAI 2026
Kyu Ri Park, Jiyoung Park, Seong Tae Kim, Hong Joo Lee, Jung Uk Kim
Change captioning aims to describe changes between a pair of images. However, existing works rely on visual features alone, which often fail to capture subtle but meaningful changes because they lack the ability to represent explicitly structured information such as object relationships and compositional semantics. To alleviate this, we present CORTEX (COmpositional Reasoning-aware TEXt-guided), a novel framework that integrates complementary textual cues to enhance change understanding. In addition to capturing cues from pixel-level differences, CORTEX utilizes scene-level textual knowledge provided by Vision Language Models (VLMs) to extract richer image text signals that reveal underlying compositional reasoning. CORTEX consists of three key modules: (i) an Image-level Change Detector that identifies low-level visual differences between paired images, (ii) a Reasoning-aware Text Extraction (RTE) module that use VLMs to generate compositional reasoning descriptions implicit in visual features, and (iii) an Image-Text Dual Alignment (ITDA) module that aligns visual and textual features for fine-grained relational reasoning. This enables CORTEX to reason over visual and textual features and capture changes that are otherwise ambiguous in visual features alone.
comment : Accepted at AAAI 2026
☆ From Points to Clouds: Learning Robust Semantic Distributions for Multi-modal Prompt s
Weiran Li, Yeqiang Liu, Yijie Wei, Mina Han, Xin Liu, Zhenbo Li
Multimodal Prompt Learning (MPL) has emerged as a pivotal technique for adapting large-scale Visual Language Models (VLMs). However, current MPL methods are fundamentally limited by their optimization of a single, static point representation. This paradigm is inherently brittle, leads to overfitting on base classes, and generalizes poorly to novel or ambiguous categories. We challenge this point paradigm, proposing that robust generalization requires learning a semantic cloud (i.e., a distribution over the embedding space). To achieve this, we introduce Points-to-Clouds (P2C), a novel framework inspired by diffusion models that reframes prompt learning as a dynamic denoising task. At the core of P2C is a dual denoising mechanism: a Dynamic Prompt Denoising (DPD) mechanism perturbs text prompts with sophisticated, annealed noise to learn a smoother semantic landscape, while an auxiliary V-L Mapper denoising loss re-tasks the mapper as a denoising autoencoder. This forces the mapper to reconstruct clean visual prompts from noisy text inputs, ensuring robust cross-modal alignment. Extensive experiments across 11 datasets demonstrate that P2C consistently outperforms strong baselines. On the base-to-novel generalization benchmark, our method achieves a Harmonic Mean of 79.7%, representing a relative improvement of 1.4% over the baseline. The code and models are available at https://vranlee.github.io/P2C/.
☆ DM$^3$T: Harmonizing Modalities via Diffusion for Multi-Object Tracking
Weiran Li, Yeqiang Liu, Yijie Wei, Mina Han, Qiannan Guo, Zhenbo Li
Multi-object tracking (MOT) is a fundamental task in computer vision with critical applications in autonomous driving and robotics. Multimodal MOT that integrates visible light and thermal infrared information is particularly essential for robust autonomous driving systems. However, effectively fusing these heterogeneous modalities is challenging. Simple strategies like concatenation or addition often fail to bridge the significant non-linear distribution gap between their feature representations, which can lead to modality conflicts and degrade tracking accuracy. Drawing inspiration from the connection between multimodal MOT and the iterative refinement in diffusion models, this paper proposes DM$^3$T, a novel framework that reformulates multimodal fusion as an iterative feature alignment process to generate accurate and temporally coherent object trajectories. Our approach performs iterative cross-modal harmonization through a proposed Cross-Modal Diffusion Fusion (C-MDF) module. In this process, features from both modalities provide mutual guidance, iteratively projecting them onto a shared, consistent feature manifold. This enables the learning of complementary information and achieves deeper fusion compared to conventional methods. Additionally, we introduce a plug-and-play Diffusion Refiner (DR) to enhance and refine the unified feature representation. To further improve tracking robustness, we design a Hierarchical Tracker that adaptively handles confidence estimation. DM$^3$T unifies object detection, state estimation, and data association into a comprehensive online tracking framework without complex post-processing. Extensive experiments on the VT-MOT benchmark demonstrate that our method achieves 41.7 HOTA, representing a 1.54% relative improvement over existing state-of-the-art methods. The code and models are available at https://vranlee.github.io/DM-3-T/.
☆ ClearGCD: Mitigating Shortcut Learning For Robust Generalized Category Discovery
Kailin Lyu, Jianwei He, Long Xiao, Jianing Zeng, Liang Fan, Lin Shu, Jie Hao
In open-world scenarios, Generalized Category Discovery (GCD) requires identifying both known and novel categories within unlabeled data. However, existing methods often suffer from prototype confusion caused by shortcut learning, which undermines generalization and leads to forgetting of known classes. We propose ClearGCD, a framework designed to mitigate reliance on non-semantic cues through two complementary mechanisms. First, Semantic View Alignment (SVA) generates strong augmentations via cross-class patch replacement and enforces semantic consistency using weak augmentations. Second, Shortcut Suppression Regularization (SSR) maintains an adaptive prototype bank that aligns known classes while encouraging separation of potential novel ones. ClearGCD can be seamlessly integrated into parametric GCD approaches and consistently outperforms state-of-the-art methods across multiple benchmarks.
comment : 5 pages, 4 figures
☆ CNN-Based Framework for Pedestrian Age and Gender Classification Using Far-View Surveillance in Mixed-Traffic Intersections
Shisir Shahriar Arif, Md. Muhtashim Shahrier, Nazmul Haque, Md Asif Raihan, Md. Hadiuzzaman
Pedestrian safety remains a pressing concern in congested urban intersections, particularly in low- and middle-income countries where traffic is multimodal, and infrastructure often lacks formal control. Demographic factors like age and gender significantly influence pedestrian vulnerability, yet real-time monitoring systems rarely capture this information. To address this gap, this study proposes a deep learning framework that classifies pedestrian age group and gender from far-view intersection footage using convolutional neural networks (CNNs), without relying on facial recognition or high-resolution imagery. The classification is structured as a unified six-class problem, distinguishing adult, teenager, and child pedestrians for both males and females, based on full-body visual cues. Video data was collected from three high-risk intersections in Dhaka, Bangladesh. Two CNN architectures were implemented: ResNet50, a deep convolutional neural network pretrained on ImageNet, and a custom lightweight CNN optimized for computational efficiency. Eight model variants explored combinations of pooling strategies and optimizers. ResNet50 with Max Pooling and SGD achieved the highest accuracy (86.19%), while the custom CNN performed comparably (84.15%) with fewer parameters and faster training. The model's efficient design enables real-time inference on standard surveillance feeds. For practitioners, this system provides a scalable, cost-effective tool to monitor pedestrian demographics at intersections using existing camera infrastructure. Its outputs can shape intersection design, optimize signal timing, and enable targeted safety interventions for vulnerable groups such as children or the elderly. By offering demographic insights often missing in conventional traffic data, the framework supports more inclusive, data-driven planning in mixed-traffic environments.
comment : Accepted for poster presentation at the 105th Annual Meeting of the Transportation Research Board
☆ Scalable Diffusion Transformer for Conditional 4D fMRI Synthesis NeurIPS 2025
Jungwoo Seo, David Keetae Park, Shinjae Yoo, Jiook Cha
Generating whole-brain 4D fMRI sequences conditioned on cognitive tasks remains challenging due to the high-dimensional, heterogeneous BOLD dynamics across subjects/acquisitions and the lack of neuroscience-grounded validation. We introduce the first diffusion transformer for voxelwise 4D fMRI conditional generation, combining 3D VQ-GAN latent compression with a CNN-Transformer backbone and strong task conditioning via AdaLN-Zero and cross-attention. On HCP task fMRI, our model reproduces task-evoked activation maps, preserves the inter-task representational structure observed in real data (RSA), achieves perfect condition specificity, and aligns ROI time-courses with canonical hemodynamic responses. Performance improves predictably with scale, reaching task-evoked map correlation of 0.83 and RSA of 0.98, consistently surpassing a U-Net baseline on all metrics. By coupling latent diffusion with a scalable backbone and strong conditioning, this work establishes a practical path to conditional 4D fMRI synthesis, paving the way for future applications such as virtual experiments, cross-site harmonization, and principled augmentation for downstream neuroimaging models.
comment : Accepted at NeurIPS 2025 Workshop: Foundation Models for the Brain and Body. 13 pages, 6 figures, 4 tables
☆ SUPER-AD: Semantic Uncertainty-aware Planning for End-to-End Robust Autonomous Driving
Wonjeong Ryu, Seungjun Yu, Seokha Moon, Hojun Choi, Junsung Park, Jinkyu Kim, Hyunjung Shim
End-to-End (E2E) planning has become a powerful paradigm for autonomous driving, yet current systems remain fundamentally uncertainty-blind. They assume perception outputs are fully reliable, even in ambiguous or poorly observed scenes, leaving the planner without an explicit measure of uncertainty. To address this limitation, we propose a camera-only E2E framework that estimates aleatoric uncertainty directly in BEV space and incorporates it into planning. Our method produces a dense, uncertainty-aware drivability map that captures both semantic structure and geometric layout at pixel-level resolution. To further promote safe and rule-compliant behavior, we introduce a lane-following regularization that encodes lane structure and traffic norms. This prior stabilizes trajectory planning under normal conditions while preserving the flexibility needed for maneuvers such as overtaking or lane changes. Together, these components enable robust and interpretable trajectory planning, even under challenging uncertainty conditions. Evaluated on the NAVSIM benchmark, our method achieves state-of-the-art performance, delivering substantial gains on both the challenging NAVHARD and NAVSAFE subsets. These results demonstrate that our principled aleatoric uncertainty modeling combined with driving priors significantly advances the safety and reliability of camera-only E2E autonomous driving.
☆ CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generation
Fengyi Fang, Sicheng Yang, Wenming Yang
Co-speech gesture generation has significantly advanced human-computer interaction, yet speaker movements remain constrained due to the omission of text-driven non-spontaneous gestures (e.g., bowing while talking). Existing methods face two key challenges: 1) the semantic prior gap due to the lack of descriptive text annotations in gesture datasets, and 2) the difficulty in achieving coordinated multimodal control over gesture generation. To address these challenges, this paper introduces CoordSpeaker, a comprehensive framework that enables coordinated caption-empowered co-speech gesture synthesis. Our approach first bridges the semantic prior gap through a novel gesture captioning framework, leveraging a motion-language model to generate descriptive captions at multiple granularities. Building upon this, we propose a conditional latent diffusion model with unified cross-dataset motion representation and a hierarchically controlled denoiser to achieve highly controlled, coordinated gesture generation. CoordSpeaker pioneers the first exploration of gesture understanding and captioning to tackle the semantic gap in gesture generation while offering a novel perspective of bidirectional gesture-text mapping. Extensive experiments demonstrate that our method produces high-quality gestures that are both rhythmically synchronized with speeches and semantically coherent with arbitrary captions, achieving superior performance with higher efficiency compared to existing approaches.
☆ Bridging Modalities via Progressive Re-alignment for Multimodal Test-Time Adaptation AAAI 2026
Jiacheng Li, Songhe Feng
Test-time adaptation (TTA) enables online model adaptation using only unlabeled test data, aiming to bridge the gap between source and target distributions. However, in multimodal scenarios, varying degrees of distribution shift across different modalities give rise to a complex coupling effect of unimodal shallow feature shift and cross-modal high-level semantic misalignment, posing a major obstacle to extending existing TTA methods to the multimodal field. To address this challenge, we propose a novel multimodal test-time adaptation (MMTTA) framework, termed as Bridging Modalities via Progressive Re-alignment (BriMPR). BriMPR, consisting of two progressively enhanced modules, tackles the coupling effect with a divide-and-conquer strategy. Specifically, we first decompose MMTTA into multiple unimodal feature alignment sub-problems. By leveraging the strong function approximation ability of prompt tuning, we calibrate the unimodal global feature distributions to their respective source distributions, so as to achieve the initial semantic re-alignment across modalities. Subsequently, we assign the credible pseudo-labels to combinations of masked and complete modalities, and introduce inter-modal instance-wise contrastive learning to further enhance the information interaction among modalities and refine the alignment. Extensive experiments on MMTTA tasks, including both corruption-based and real-world domain shift benchmarks, demonstrate the superiority of our method. Our source code is available at [this URL](https://github.com/Luchicken/BriMPR).
comment : Accepted by AAAI 2026 (Oral)
☆ MARVO: Marine-Adaptive Radiance-aware Visual Odometry CVPR2026
Sacchin Sundar, Atman Kikani, Aaliya Alam, Sumukh Shrote, A. Nayeemulla Khan, A. Shahina
Underwater visual localization remains challenging due to wavelength-dependent attenuation, poor texture, and non-Gaussian sensor noise. We introduce MARVO, a physics-aware, learning-integrated odometry framework that fuses underwater image formation modeling, differentiable matching, and reinforcement-learning optimization. At the front-end, we extend transformer-based feature matcher with a Physics Aware Radiance Adapter that compensates for color channel attenuation and contrast loss, yielding geometrically consistent feature correspondences under turbidity. These semi dense matches are combined with inertial and pressure measurements inside a factor-graph backend, where we formulate a keyframe-based visual-inertial-barometric estimator using GTSAM library. Each keyframe introduces (i) Pre-integrated IMU motion factors, (ii) MARVO-derived visual pose factors, and (iii) barometric depth priors, giving a full-state MAP estimate in real time. Lastly, we introduce a Reinforcement-Learningbased Pose-Graph Optimizer that refines global trajectories beyond local minima of classical least-squares solvers by learning optimal retraction actions on SE(2).
comment : 10 pages, 5 figures, 3 tables, Submitted to CVPR2026
☆ GLOW: Global Illumination-Aware Inverse Rendering of Indoor Scenes Captured with Dynamic Co-Located Light & Camera
Jiaye Wu, Saeed Hadadan, Geng Lin, Peihan Tu, Matthias Zwicker, David Jacobs, Roni Sengupta
Inverse rendering of indoor scenes remains challenging due to the ambiguity between reflectance and lighting, exacerbated by inter-reflections among multiple objects. While natural illumination-based methods struggle to resolve this ambiguity, co-located light-camera setups offer better disentanglement as lighting can be easily calibrated via Structure-from-Motion. However, such setups introduce additional complexities like strong inter-reflections, dynamic shadows, near-field lighting, and moving specular highlights, which existing approaches fail to handle. We present GLOW, a Global Illumination-aware Inverse Rendering framework designed to address these challenges. GLOW integrates a neural implicit surface representation with a neural radiance cache to approximate global illumination, jointly optimizing geometry and reflectance through carefully designed regularization and initialization. We then introduce a dynamic radiance cache that adapts to sharp lighting discontinuities from near-field motion, and a surface-angle-weighted radiometric loss to suppress specular artifacts common in flashlight captures. Experiments show that GLOW substantially outperforms prior methods in material reflectance estimation under both natural and co-located illumination.
☆ Resolving Evidence Sparsity: Agent ic Context Engineering for Long- Document Understanding
Keliang Liu, Zizhi Chen, Mingcheng Li, Jingqun Tang, Dingkang Yang, Lihua Zhang
Document understanding is a long standing practical task. Vision Language Models (VLMs) have gradually become a primary approach in this domain, demonstrating effective performance on single page tasks. However, their effectiveness diminishes when handling long documents. In such scenarios, clues are often scattered across multiple pages and modalities, and redundancy from lengthy inputs can impair the models judgment. While retrieval augmented generation mitigates this issue by filtering for question relevant content, the retrieved results still contain substantial redundancy. To address these limitations, we propose SLEUTH, a multi agent framework. Concretely, SLEUTH orchestrates a retriever and four collaborative agents in a coarse to fine process. The framework identifies key textual and visual clues within the retrieved pages, filters for salient visual evidence such as tables and charts, and analyzes the query to devise a reasoning strategy. It ultimately synthesizes a distilled, evidence dense multimodal context to generate the final prediction. SLEUTH is model agnostic and scalable. When paired with advanced VLM backbones, it consistently improves performance on multiple long document benchmarks, achieving state of the art results. Ablation studies verify each modules effectiveness and confirm the benefits of our hierarchical refinement paradigm.
☆ Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering
Dosung Lee, Sangwon Jung, Boyoung Kim, Minyoung Kim, Sungyeon Kim, Junyoung Sung, Paul Hongsuck Seo
Existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks suffer from "visual shortcuts", as the query image typically matches the primary subject entity of the target document. We demonstrate that models can exploit these shortcuts, achieving comparable results using visual cues alone. To address this, we introduce Relational Entity Text-Image kNowledge Augmented (RETINA) benchmark, automatically constructed using an LLM-driven pipeline, consisting of 120k training and 2k human-curated test set. RETINA contains queries referencing secondary subjects (i.e. related entities) and pairs them with images of these related entities, removing the visual shortcut. When evaluated on RETINA existing models show significantly degraded performance, confirming their reliance on the shortcut. Furthermore, we propose Multi-Image MultImodal Retriever (MIMIR), which enriches document embeddings by augmenting images of multiple related entities, effectively handling RETINA, unlike prior work that uses only a single image per document. Our experiments validate the limitations of existing benchmarks and demonstrate the effectiveness of RETINA and MIMIR. Our project is available at: Project Page.
☆ Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
Tianle Chen, Chaitanya Chakka, Arjun Reddy Akula, Xavier Thomas, Deepti Ghadiyaram
Despite remarkable advancements in Multimodal Large Language Models (MLLMs), a fundamental question remains: are MLLMs robust to contradicting modalities? To rigorously study this, we introduce MMA-Bench comprising videos and tasks that probe a model's reliance on specific modalities. Using black-box and white-box interpretability techniques, we provide a critical analysis of the brittleness of both open- and closed-sourced MLLMs. We show that current MLLMs struggle under misaligned audio-visual pairs and simple misleading text, thereby lacking robust multi-modal reasoning. Building on these findings, we propose a modality alignment tuning strategy to teach the model when to prioritize, leverage, or ignore specific modality cues. Through extensive experiments and analysis, we show that our alignment tuning yields demonstrably stronger multimodal grounding. This work provides both interpretability tools and a clear path toward developing MLLMs with intrinsically reliable cross-modal reasoning. Code and dataset will be publicly available.
☆ Captain Safari: A Wo rl d Engine
Yu-Cheng Chou, Xingrui Wang, Yitong Li, Jiahao Wang, Hanting Liu, Cihang Xie, Alan Yuille, Junfei Xiao
World engines aim to synthesize long, 3D-consistent videos that support interactive exploration of a scene under user-controlled camera motion. However, existing systems struggle under aggressive 6-DoF trajectories and complex outdoor layouts: they lose long-range geometric coherence, deviate from the target path, or collapse into overly conservative motion. To this end, we introduce Captain Safari, a pose-conditioned world engine that generates videos by retrieving from a persistent world memory. Given a camera path, our method maintains a dynamic local memory and uses a retriever to fetch pose-aligned world tokens, which then condition video generation along the trajectory. This design enables the model to maintain stable 3D structure while accurately executing challenging camera maneuvers. To evaluate this setting, we curate OpenSafari, a new in-the-wild FPV dataset containing high-dynamic drone videos with verified camera trajectories, constructed through a multi-stage geometric and kinematic validation pipeline. Across video quality, 3D consistency, and trajectory following, Captain Safari substantially outperforms state-of-the-art camera-controlled generators. It reduces MEt3R from 0.3703 to 0.3690, improves AUC@30 from 0.181 to 0.200, and yields substantially lower FVD than all camera-controlled baselines. More importantly, in a 50-participant, 5-way human study where annotators select the best result among five anonymized models, 67.6% of preferences favor our method across all axes. Our results demonstrate that pose-conditioned world memory is a powerful mechanism for long-horizon, controllable video generation and provide OpenSafari as a challenging new benchmark for future world-engine research.
☆ TIE: A Training-Inversion-Exclusion Framework for Visually Interpretable and Uncertainty-Guided Out-of-Distribution Detection
Pirzada Suhail, Rehna Afroz, Amit Sethi
Deep neural networks often struggle to recognize when an input lies outside their training experience, leading to unreliable and overconfident predictions. Building dependable machine learning systems therefore requires methods that can both estimate predictive \textit{uncertainty} and detect \textit{out-of-distribution (OOD)} samples in a unified manner. In this paper, we propose \textbf{TIE: a Training--Inversion--Exclusion} framework for visually interpretable and uncertainty-guided anomaly detection that jointly addresses these challenges through iterative refinement. TIE extends a standard $n$-class classifier to an $(n+1)$-class model by introducing a garbage class initialized with Gaussian noise to represent outlier inputs. Within each epoch, TIE performs a closed-loop process of \textit{training, inversion, and exclusion}, where highly uncertain inverted samples reconstructed from the just-trained classifier are excluded into the garbage class. Over successive iterations, the inverted samples transition from noisy artifacts into visually coherent class prototypes, providing transparent insight into how the model organizes its learned manifolds. During inference, TIE rejects OOD inputs by either directly mapping them to the garbage class or producing low-confidence, uncertain misclassifications within the in-distribution classes that are easily separable, all without relying on external OOD datasets. A comprehensive threshold-based evaluation using multiple OOD metrics and performance measures such as \textit{AUROC}, \textit{AUPR}, and \textit{FPR@95\%TPR} demonstrates that TIE offers a unified and interpretable framework for robust anomaly detection and calibrated uncertainty estimation (UE) achieving near-perfect OOD detection with \textbf{\(\!\approx\!\) 0 FPR@95\%TPR} when trained on MNIST or FashionMNIST and tested against diverse unseen datasets.
☆ DenseScan: Advancing 3D Scene Understanding with 2D Dense Annotation NeurIPS 2025
Zirui Wang, Tao Zhang
3D understanding is a key capability for real-world AI assistance. High-quality data plays an important role in driving the development of the 3D understanding community. Current 3D scene understanding datasets often provide geometric and instance-level information, yet they lack the rich semantic annotations necessary for nuanced visual-language tasks.In this work, we introduce DenseScan, a novel dataset with detailed multi-level descriptions generated by an automated pipeline leveraging multi-view 2D images and multimodal large language models (MLLMs). Our approach enables dense captioning of scene elements, ensuring comprehensive object-level descriptions that capture context-sensitive details. Furthermore, we extend these annotations through scenario-based question generation, producing high-level queries that integrate object properties, spatial relationships, and scene context. By coupling geometric detail with semantic richness, DenseScan broadens the range of downstream tasks, from detailed visual-language navigation to interactive question answering. Experimental results demonstrate that our method significantly enhances object-level understanding and question-answering performance in 3D environments compared to traditional annotation pipelines. We release both the annotated dataset and our annotation pipeline to facilitate future research and applications in robotics, augmented reality, and beyond. Through DenseScan, we aim to catalyze new avenues in 3D scene understanding, allowing researchers and practitioners to tackle the complexities of real-world environments with richer, more contextually aware annotations.
comment : Workshop on Space in Vision, Language, and Embodied AI at NeurIPS 2025
☆ ReactionMamba: Generating Short &Long Human Reaction Sequences
Hajra Anwar Beg, Baptiste Chopin, Hao Tang, Mohamed Daoudi
We present ReactionMamba, a novel framework for generating long 3D human reaction motions. Reaction-Mamba integrates a motion VAE for efficient motion encoding with Mamba-based state-space models to decode temporally consistent reactions. This design enables ReactionMamba to generate both short sequences of simple motions and long sequences of complex motions, such as dance and martial arts. We evaluate ReactionMamba on three datasets--NTU120-AS, Lindy Hop, and InterX--and demonstrate competitive performance in terms of realism, diversity, and long-sequence generation compared to previous methods, including InterFormer, ReMoS, and Ready-to-React, while achieving substantial improvements in inference speed.
☆ Mammo-FM: Breast-specific foundational model for Integrated Mammographic Diagnosis, Prognosis, and Reporting
Shantanu Ghosh, Vedant Parthesh Joshi, Rayan Syed, Aya Kassem, Abhishek Varshney, Payel Basak, Weicheng Dai, Judy Wawira Gichoya, Hari M. Trivedi, Imon Banerjee, Shyam Visweswaran, Clare B. Poynton, Kayhan Batmanghelich
Breast cancer is one of the leading causes of death among women worldwide. We introduce Mammo-FM, the first foundation model specifically for mammography, pretrained on the largest and most diverse dataset to date - 140,677 patients (821,326 mammograms) across four U.S. institutions. Mammo-FM provides a unified foundation for core clinical tasks in breast imaging, including cancer diagnosis, pathology localization, structured report generation, and cancer risk prognosis within a single framework. Its alignment between images and text enables both visual and textual interpretability, improving transparency and clinical auditability, which are essential for real-world adoption. We rigorously evaluate Mammo-FM across diagnosis, prognosis, and report-generation tasks in in- and out-of-distribution datasets. Despite operating on native-resolution mammograms and using only one-third of the parameters of state-of-the-art generalist FMs, Mammo-FM consistently outperforms them across multiple public and private benchmarks. These results highlight the efficiency and value of domain-specific foundation models designed around the full spectrum of tasks within a clinical domain and emphasize the importance of rigorous, domain-aligned evaluation.
☆ AutocleanEEG IC Vision : Automated ICA Artifact Classification Using Vision -Language AI
Zag ElSayed, Grace Westerkamp, Gavin Gammoh, Yanchen Liu, Peyton Siekierski, Craig Erickson, Ernest Pedapati
We introduce EEG Autoclean Vision Language AI (ICVision) a first-of-its-kind system that emulates expert-level EEG ICA component classification through AI-agent vision and natural language reasoning. Unlike conventional classifiers such as ICLabel, which rely on handcrafted features, ICVision directly interprets ICA dashboard visualizations topography, time series, power spectra, and ERP plots, using a multimodal large language model (GPT-4 Vision). This allows the AI to see and explain EEG components the way trained neurologists do, making it the first scientific implementation of AI-agent visual cognition in neurophysiology. ICVision classifies each component into one of six canonical categories (brain, eye, heart, muscle, channel noise, and other noise), returning both a confidence score and a human-like explanation. Evaluated on 3,168 ICA components from 124 EEG datasets, ICVision achieved k = 0.677 agreement with expert consensus, surpassing MNE ICLabel, while also preserving clinically relevant brain signals in ambiguous cases. Over 97% of its outputs were rated as interpretable and actionable by expert reviewers. As a core module of the open-source EEG Autoclean platform, ICVision signals a paradigm shift in scientific AI, where models do not just classify, but see, reason, and communicate. It opens the door to globally scalable, explainable, and reproducible EEG workflows, marking the emergence of AI agents capable of expert-level visual decision-making in brain science and beyond.
comment : 6 pages, 8 figures
☆ Efficient Edge-Compatible CNN for Speckle-Based Material Recognition in Laser Cutting Systems
Mohamed Abdallah Salem, Nourhan Zein Diab
Accurate material recognition is critical for safe and effective laser cutting, as misidentification can lead to poor cut quality, machine damage, or the release of hazardous fumes. Laser speckle sensing has recently emerged as a low-cost and non-destructive modality for material classification; however, prior work has either relied on computationally expensive backbone networks or addressed only limited subsets of materials. In this study, A lightweight convolutional neural network (CNN) tailored for speckle patterns is proposed, designed to minimize parameters while maintaining high discriminative power. Using the complete SensiCut dataset of 59 material classes spanning woods, acrylics, composites, textiles, metals, and paper-based products, the proposed model achieves 95.05% test accuracy, with macro and weighted F1-scores of 0.951. The network contains only 341k trainable parameters (~1.3 MB) -- over 70X fewer than ResNet-50 -- and achieves an inference speed of 295 images per second, enabling deployment on Raspberry Pi and Jetson-class devices. Furthermore, when materials are regrouped into nine and five practical families, recall exceeds 98% and approaches 100%, directly supporting power and speed preset selection in laser cutters. These results demonstrate that compact, domain-specific CNNs can outperform large backbones for speckle-based material classification, advancing the feasibility of material-aware, edge-deployable laser cutting systems.
comment : Copyright 2025 IEEE. This is the author's version of the work that has been Accepted for publication in the Proceedings of the 2025 IEEE The 35th International Conference on Computer Theory and Applications (ICCTA 2025). Final published version will be available on IEEE Xplore
☆ Ternary-Input Binary-Weight CNN Accelerator Design for Miniature Object Classification System with Query-Driven Spatial DVS
Yuyang Li, Swasthik Muloor, Jack Laudati, Nickolas Dematteis, Yidam Park, Hana Kim, Nathan Chang, Inhee Lee
Miniature imaging systems are essential for space-constrained applications but are limited by memory and power constraints. While machine learning can reduce data size by extracting key features, its high energy demands often exceed the capacity of small batteries. This paper presents a CNN hardware accelerator optimized for object classification in miniature imaging systems. It processes data from a spatial Dynamic Vision Sensor (DVS), reconfigurable to a temporal DVS via pixel sharing, minimizing sensor area. By using ternary DVS outputs and a ternary-input, binary-weight neural network, the design reduces computation and memory needs. Fabricated in 28 nm CMOS, the accelerator cuts data size by 81% and MAC operations by 27%. It achieves 440 ms inference time at just 1.6 mW power consumption, improving the Figure-of-Merit (FoM) by 7.3x over prior CNN accelerators for miniature systems.
comment : 6 pages.12 figures & 2 table
☆ Local and Global Context -and-Object-part-Aware Superpixel-based Data Augmentation for Deep Visual Recognition
Fadi Dornaika, Danyang Sun
Cutmix-based data augmentation, which uses a cut-and-paste strategy, has shown remarkable generalization capabilities in deep learning. However, existing methods primarily consider global semantics with image-level constraints, which excessively reduces attention to the discriminative local context of the class and leads to a performance improvement bottleneck. Moreover, existing methods for generating augmented samples usually involve cutting and pasting rectangular or square regions, resulting in a loss of object part information. To mitigate the problem of inconsistency between the augmented image and the generated mixed label, existing methods usually require double forward propagation or rely on an external pre-trained network for object centering, which is inefficient. To overcome the above limitations, we propose LGCOAMix, an efficient context-aware and object-part-aware superpixel-based grid blending method for data augmentation. To the best of our knowledge, this is the first time that a label mixing strategy using a superpixel attention approach has been proposed for cutmix-based data augmentation. It is the first instance of learning local features from discriminative superpixel-wise regions and cross-image superpixel contrasts. Extensive experiments on various benchmark datasets show that LGCOAMix outperforms state-of-the-art cutmix-based data augmentation methods on classification tasks, {and weakly supervised object location on CUB200-2011.} We have demonstrated the effectiveness of LGCOAMix not only for CNN networks, but also for Transformer networks. Source codes are available at https://github.com/DanielaPlusPlus/LGCOAMix.
♻ ☆ DINO-Foresight: Looking into the Future with DINO NeurIPS 2025
Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, Nikos Komodakis
Predicting future dynamics is crucial for applications like autonomous driving and robotics, where understanding the environment is key. Existing pixel-level methods are computationally expensive and often focus on irrelevant details. To address these challenges, we introduce DINO-Foresight, a novel framework that operates in the semantic feature space of pretrained Vision Foundation Models (VFMs). Our approach trains a masked feature transformer in a self-supervised manner to predict the evolution of VFM features over time. By forecasting these features, we can apply off-the-shelf, task-specific heads for various scene understanding tasks. In this framework, VFM features are treated as a latent space, to which different heads attach to perform specific tasks for future-frame analysis. Extensive experiments show the very strong performance, robustness and scalability of our framework. Project page and code at https://dino-foresight.github.io/ .
comment : NeurIPS 2025
♻ ☆ INQUIRE-Search: A Framework for Interactive Discovery in Large-Scale Biodiversity Databases
Edward Vendrow, Julia Chae, Rupa Kurinchi-Vendhan, Isaac Eckert, Jazlynn Hall, Marta Jarzyna, Reymond Miyajima, Ruth Oliver, Laura Pollock, Lauren Schrack, Scott Yanco, Oisin Mac Aodha, Sara Beery
Large community science platforms such as iNaturalist contain hundreds of millions of biodiversity images that often capture ecological context on behaviors, interactions, phenology, and habitat. Yet most ecological workflows rely on metadata filtering or manual inspection, leaving this secondary information inaccessible at scale. We introduce INQUIRE-Search, an open-source system that enables scientists to rapidly and interactively search within an ecological image database for specific concepts using natural language, verify and export relevant observations, and utilize this discovered data for novel scientific analysis. Compared to traditional methods, INQUIRE-Search takes a fraction of the time, opening up new possibilities for scientific questions that can be explored. Through five case studies, we show the diversity of scientific applications that a tool like INQUIRE-Search can support, from seasonal variation in behavior across species to forest regrowth after wildfires. These examples demonstrate a new paradigm for interactive, efficient, and scalable scientific discovery that can begin to unlock previously inaccessible scientific value in large-scale biodiversity datasets. Finally, we emphasize using such AI-enabled discovery tools for science call for experts to reframe the priorities of the scientific process and develop novel methods for experiment design, data collection, survey effort, and uncertainty analysis.
comment : EV, JC, RKV contributed equally
♻ ☆ CzechLynx: A Dataset for Individual Identification and Pose Estimation of the Eurasian Lynx
Lukas Picek, Elisa Belotti, Michal Bojda, Ludek Bufka, Vojtech Cermak, Martin Dula, Rostislav Dvorak, Luboslav Hrdy, Miroslav Jirik, Vaclav Kocourek, Josefa Krausova, Jirı Labuda, Jakub Straka, Ludek Toman, Vlado Trulık, Martin Vana, Miroslav Kutal
We introduce CzechLynx, the first large-scale, open-access dataset for individual identification, pose estimation, and instance segmentation of the Eurasian lynx (Lynx lynx). CzechLynx contains 39,760 camera trap images annotated with segmentation masks, identity labels, and 20-point skeletons and covers 319 unique individuals across 15 years of systematic monitoring in two geographically distinct regions: southwest Bohemia and the Western Carpathians. In addition to the real camera trap data, we provide a large complementary set of photorealistic synthetic images and a Unity-based generation pipeline with diffusion-based text-to-texture modeling, capable of producing arbitrarily large amounts of synthetic data spanning diverse environments, poses, and coat-pattern variations. To enable systematic testing across realistic ecological scenarios, we define three complementary evaluation protocols: (i) geo-aware, (ii) time-aware open-set, and (iii) time-aware closed-set, covering cross-regional and long-term monitoring settings. With the provided resources, CzechLynx offers a unique, flexible benchmark for robust evaluation of computer vision and machine learning models across realistic ecological scenarios.
♻ ☆ Configurable Fairness: Direct Optimization of Parity Metrics via Vision -Language Models
Miao Zhang, Rumi Chunara
Performance disparities of image recognition across demographic groups are known to exist in deep learning-based models, due to imbalanced group representations or spurious correlation between group and target labels. Previous work has addressed such challenges without relying on expensive group labels, typically by upweighting high-loss samples or balancing discovered clusters. However, these heuristic strategies lack direct connection to specific fairness metrics and cannot guarantee optimization of parity-based criteria like equal opportunity, which ensures equal chance to receive positive outcomes across groups. In this work, we propose a novel paradigm that directly optimizes parity-based fairness metrics through specifically designed training objectives, without requiring group labels. We leverage vision-language models to analyze sensitive attribute relevancy for individual samples, then formulate loss functions that mathematically connect to each target fairness metric. This enables flexible optimization of different fairness criteria based on application needs. Experiments on multiple image classification datasets show that our metric-specific approach significantly improves parity-based fairness criteria and outperforms existing methods.
♻ ☆ Group Relative Attention Guidance for Image Editing
Xuanpu Zhang, Xuesong Niu, Ruidong Chen, Dan Song, Jianhao Zeng, Penghui Du, Haoxiang Cao, Kai Wu, An-an Liu
Recently, image editing based on Diffusion-in-Transformer models has undergone rapid development. However, existing editing methods often lack effective control over the degree of editing, limiting their ability to achieve more customized results. To address this limitation, we investigate the MM-Attention mechanism within the DiT model and observe that the Query and Key tokens share a bias vector that is only layer-dependent. We interpret this bias as representing the model's inherent editing behavior, while the delta between each token and its corresponding bias encodes the content-specific editing signals. Based on this insight, we propose Group Relative Attention Guidance, a simple yet effective method that reweights the delta values of different tokens to modulate the focus of the model on the input image relative to the editing instruction, enabling continuous and fine-grained control over editing intensity without any tuning. Extensive experiments conducted on existing image editing frameworks demonstrate that GRAG can be integrated with as few as four lines of code, consistently enhancing editing quality. Moreover, compared to the commonly used Classifier-Free Guidance, GRAG achieves smoother and more precise control over the degree of editing. Our code will be released at https://github.com/little-misfit/GRAG-Image-Editing.
♻ ☆ SARD: Segmentation-Aware Anomaly Synthesis via Region-Constrained Diffusion with Discriminative Mask Guidance
Yanshu Wang, Xichen Xu, Xiaoning Lei, Guoyang Xie
Synthesizing realistic and spatially precise anomalies is essential for enhancing the robustness of industrial anomaly detection systems. While recent diffusion-based methods have demonstrated strong capabilities in modeling complex defect patterns, they often struggle with spatial controllability and fail to maintain fine-grained regional fidelity. To overcome these limitations, we propose SARD (Segmentation-Aware anomaly synthesis via Region-constrained Diffusion with discriminative mask Guidance), a novel diffusion-based framework specifically designed for anomaly generation. Our approach introduces a Region-Constrained Diffusion (RCD) process that preserves the background by freezing it and selectively updating only the foreground anomaly regions during the reverse denoising phase, thereby effectively reducing background artifacts. Additionally, we incorporate a Discriminative Mask Guidance (DMG) module into the discriminator, enabling joint evaluation of both global realism and local anomaly fidelity, guided by pixel-level masks. Extensive experiments on the MVTec-AD and BTAD datasets show that SARD surpasses existing methods in segmentation accuracy and visual quality, setting a new state-of-the-art for pixel-level anomaly synthesis.
comment : Accepted by The 2025 International Conference on Machine Intelligence and Nature-InspireD Computing (MIND)
♻ ☆ FAST: Foreground-aware Diffusion with Accelerated Sampling Trajectory for Segmentation-oriented Anomaly Synthesis
Xichen Xu, Yanshu Wang, Jinbao Wang, Xiaoning Lei, Guoyang Xie, Guannan Jiang, Zhichao Lu
Industrial anomaly segmentation relies heavily on pixel-level annotations, yet real-world anomalies are often scarce, diverse, and costly to label. Segmentation-oriented industrial anomaly synthesis (SIAS) has emerged as a promising alternative; however, existing methods struggle to balance sampling efficiency and generation quality. Moreover, most approaches treat all spatial regions uniformly, overlooking the distinct statistical differences between anomaly and background areas. This uniform treatment hinders the synthesis of controllable, structure-specific anomalies tailored for segmentation tasks. In this paper, we propose FAST, a foreground-aware diffusion framework featuring two novel modules: the Anomaly-Informed Accelerated Sampling (AIAS) and the Foreground-Aware Reconstruction Module (FARM). AIAS is a training-free sampling algorithm specifically designed for segmentation-oriented industrial anomaly synthesis, which accelerates the reverse process through coarse-to-fine aggregation and enables the synthesis of state-of-the-art segmentation-oriented anomalies in as few as 10 steps. Meanwhile, FARM adaptively adjusts the anomaly-aware noise within the masked foreground regions at each sampling step, preserving localized anomaly signals throughout the denoising trajectory. Extensive experiments on multiple industrial benchmarks demonstrate that FAST consistently outperforms existing anomaly synthesis methods in downstream segmentation tasks. We release the code at: https://github.com/Chhro123/fast-foreground-aware-anomaly-synthesis.
♻ ☆ A Simple yet Effective Test-Time Adaptation for Zero-Shot Monocular Metric Depth Estimation IROS 2025
Rémi Marsal, Alexandre Chapoutot, Philippe Xu, David Filliat
The recent development of \emph{foundation models} for monocular depth estimation such as Depth Anything paved the way to zero-shot monocular depth estimation. Since it returns an affine-invariant disparity map, the favored technique to recover the metric depth consists in fine-tuning the model. However, this stage is not straightforward, it can be costly and time-consuming because of the training and the creation of the dataset. The latter must contain images captured by the camera that will be used at test time and the corresponding ground truth. Moreover, the fine-tuning may also degrade the generalizing capacity of the original model. Instead, we propose in this paper a new method to rescale Depth Anything predictions using 3D points provided by sensors or techniques such as low-resolution LiDAR or structure-from-motion with poses given by an IMU. This approach avoids fine-tuning and preserves the generalizing power of the original depth estimation model while being robust to the noise of the sparse depth, of the camera-LiDAR calibration or of the depth model. Our experiments highlight enhancements relative to zero-shot monocular metric depth estimation methods, competitive results compared to fine-tuned approaches and a better robustness than depth completion approaches. Code available at github.com/ENSTA-U2IS-AI/depth-rescaling.
comment : Published at IROS 2025 https://ieeexplore.ieee.org/document/11247168
♻ ☆ Material-informed Gaussian Splatting for 3D Wo rl d Reconstruction in a Digital Twin
Andy Huynh, João Malheiro Silva, Holger Caesar, Tong Duy Son
3D reconstruction for Digital Twins often relies on LiDAR-based methods, which provide accurate geometry but lack the semantics and textures naturally captured by cameras. Traditional LiDAR-camera fusion approaches require complex calibration and still struggle with certain materials like glass, which are visible in images but poorly represented in point clouds. We propose a camera-only pipeline that reconstructs scenes using 3D Gaussian Splatting from multi-view images, extracts semantic material masks via vision models, converts Gaussian representations to mesh surfaces with projected material labels, and assigns physics-based material properties for accurate sensor simulation in modern graphics engines and simulators. This approach combines photorealistic reconstruction with physics-based material assignment, providing sensor simulation fidelity comparable to LiDAR-camera fusion while eliminating hardware complexity and calibration requirements. We validate our camera-only method using an internal dataset from an instrumented test vehicle, leveraging LiDAR as ground truth for reflectivity validation alongside image similarity metrics.
comment : 8 pages, 5 figures. Submitted to IEEE Intelligent Vehicles Symposium (IV) 2026 for possible publication. Revised version (v2) to correct author order
♻ ☆ Entropy Rectifying Guidance for Diffusion and Flow Models NeurIPS 2025
Tariq Berrada Ifriqi, Adriana Romero-Soriano, Michal Drozdzal, Jakob Verbeek, Karteek Alahari
Guidance techniques are commonly used in diffusion and flow models to improve image quality and input consistency for conditional generative tasks such as class-conditional and text-to-image generation. In particular, classifier-free guidance (CFG) is the most widely adopted guidance technique. It results, however, in trade-offs across quality, diversity and consistency: improving some at the expense of others. While recent work has shown that it is possible to disentangle these factors to some extent, such methods come with an overhead of requiring an additional (weaker) model, or require more forward passes per sampling step. In this paper, we propose Entropy Rectifying Guidance (ERG), a simple and effective guidance method based on inference-time changes in the attention mechanism of state-of-the-art diffusion transformer architectures, which allows for simultaneous improvements over image quality, diversity and prompt consistency. ERG is more general than CFG and similar guidance techniques, as it extends to unconditional sampling. We show that ERG results in significant improvements in various tasks, including text-to-image, class-conditional and unconditional image generation. We also show that ERG can be seamlessly combined with other recent guidance methods such as CADS and APG, further improving generation results.
comment : NeurIPS 2025
♻ ☆ PoseAdapt: Sustainable Human Pose Estimation via Continual Learning Benchmark s and Toolkit WACV 2026
Muhammad Saif Ullah Khan, Didier Stricker
Human pose estimators are typically retrained from scratch or naively fine-tuned whenever keypoint sets, sensing modalities, or deployment domains change--an inefficient, compute-intensive practice that rarely matches field constraints. We present PoseAdapt, an open-source framework and benchmark suite for continual pose model adaptation. PoseAdapt defines domain-incremental and class-incremental tracks that simulate realistic changes in density, lighting, and sensing modality, as well as skeleton growth. The toolkit supports two workflows: (i) Strategy Benchmarking, which lets researchers implement continual learning (CL) methods as plugins and evaluate them under standardized protocols; and (ii) Model Adaptation, which allows practitioners to adapt strong pretrained models to new tasks with minimal supervision. We evaluate representative regularization-based methods in single-step and sequential settings. Benchmarks enforce a fixed lightweight backbone, no access to past data, and tight per-step budgets. This isolates adaptation strategy effects, highlighting the difficulty of maintaining accuracy under strict resource limits. PoseAdapt connects modern CL techniques with practical pose estimation needs, enabling adaptable models that improve over time without repeated full retraining.
comment : Accepted in WACV 2026 Applications Track
♻ ☆ Network Inversion for Uncertainty-Aware Out-of-Distribution Detection
Pirzada Suhail, Rehna Afroz, Gouranga Bala, Amit Sethi
Out-of-distribution (OOD) detection and uncertainty estimation (UE) are critical components for building safe machine learning systems, especially in real-world scenarios where unexpected inputs are inevitable. However the two problems have, until recently, separately been addressed. In this work, we propose a novel framework that combines network inversion with classifier training to simultaneously address both OOD detection and uncertainty estimation. For a standard n-class classification task, we extend the classifier to an (n+1)-class model by introducing a "garbage" class, initially populated with random gaussian noise to represent outlier inputs. After each training epoch, we use network inversion to reconstruct input images corresponding to all output classes that initially appear as noisy and incoherent and are therefore excluded to the garbage class for retraining the classifier. This cycle of training, inversion, and exclusion continues iteratively till the inverted samples begin to resemble the in-distribution data more closely, with a significant drop in the uncertainty, suggesting that the classifier has learned to carve out meaningful decision boundaries while sanitising the class manifolds by pushing OOD content into the garbage class. During inference, this training scheme enables the model to effectively detect and reject OOD samples by classifying them into the garbage class. Furthermore, the confidence scores associated with each prediction can be used to estimate uncertainty for both in-distribution and OOD inputs. Our approach is scalable, interpretable, and does not require access to external OOD datasets or post-hoc calibration techniques while providing a unified solution to the dual challenges of OOD detection and uncertainty estimation.
♻ ☆ Source-free Video Domain Adaptation by Learning from Noisy Labels
Avijit Dasgupta, C. V. Jawahar, Karteek Alahari
Despite the progress seen in classification methods, current approaches for handling videos with distribution shifts in source and target domains remain source-dependent as they require access to the source data during the adaptation stage. In this paper, we present a self-training based source-free video domain adaptation approach to address this challenge by bridging the gap between the source and the target domains. We use the source pre-trained model to generate pseudo-labels for the target domain samples, which are inevitably noisy. Thus, we treat the problem of source-free video domain adaptation as learning from noisy labels and argue that the samples with correct pseudo-labels can help us in adaptation. To this end, we leverage the cross-entropy loss as an indicator of the correctness of the pseudo-labels and use the resulting small-loss samples from the target domain for fine-tuning the model. We further enhance the adaptation performance by implementing a teacher-student (TS) framework, in which the teacher, which is updated gradually, produces reliable pseudo-labels. Meanwhile, the student undergoes fine-tuning on the target domain videos using these generated pseudo-labels to improve its performance. Extensive experimental evaluations show that our methods, termed as CleanAdapt, CleanAdapt + TS, achieve state-of-the-art results, outperforming the existing approaches on various open datasets. Our source code is publicly available at https://avijit9.github.io/CleanAdapt.
comment : Our extended ICVGIP paper is now accepted in Pattern Recognition
♻ ☆ Text2Traffic: A Text-to-Image Generation and Editing Method for Traffic Scenes
Feng Lv, Haoxuan Feng, Zilu Zhang, Chunlong Xia, Yanfeng Li
With the rapid advancement of intelligent transportation systems, text-driven image generation and editing techniques have demonstrated significant potential in providing rich, controllable visual scene data for applications such as traffic monitoring and autonomous driving. However, several challenges remain, including insufficient semantic richness of generated traffic elements, limited camera viewpoints, low visual fidelity of synthesized images, and poor alignment between textual descriptions and generated content. To address these issues, we propose a unified text-driven framework for both image generation and editing, leveraging a controllable mask mechanism to seamlessly integrate the two tasks. Furthermore, we incorporate both vehicle-side and roadside multi-view data to enhance the geometric diversity of traffic scenes. Our training strategy follows a two-stage paradigm: first, we perform conceptual learning using large-scale coarse-grained text-image data; then, we fine-tune with fine-grained descriptive data to enhance text-image alignment and detail quality. Additionally, we introduce a mask-region-weighted loss that dynamically emphasizes small yet critical regions during training, thereby substantially enhancing the generation fidelity of small-scale traffic elements. Extensive experiments demonstrate that our method achieves leading performance in text-based image generation and editing within traffic scenes.
♻ ☆ TEFormer: Texture-Aware and Edge-Guided Transformer for Semantic Segmentation of Urban Remote Sensing Images
Guoyu Zhou, Jing Zhang, Yi Yan, Hui Zhang, Li Zhuo
Accurate semantic segmentation of urban remote sensing images (URSIs) is essential for urban planning and environmental monitoring. However, it remains challenging due to the subtle texture differences and similar spatial structures among geospatial objects, which cause semantic ambiguity and misclassification. Additional complexities arise from irregular object shapes, blurred boundaries, and overlapping spatial distributions of objects, resulting in diverse and intricate edge morphologies. To address these issues, we propose TEFormer, a texture-aware and edge-guided Transformer. Our model features a texture-aware module (TaM) in the encoder to capture fine-grained texture distinctions between visually similar categories, thereby enhancing semantic discrimination. The decoder incorporates an edge-guided tri-branch decoder (Eg3Head) to preserve local edges and details while maintaining multiscale context-awareness. Finally, an edge-guided feature fusion module (EgFFM) effectively integrates contextual, detail, and edge information to achieve refined semantic segmentation. Extensive evaluation demonstrates that TEFormer yields mIoU scores of 88.57% on Potsdam and 81.46% on Vaihingen, exceeding the next best methods by 0.73% and 0.22%. On the LoveDA dataset, it secures the second position with an overall mIoU of 53.55%, trailing the optimal performance by a narrow margin of 0.19%.
comment : Accepted by IEEE GRSL
♻ ☆ Look Where It Matters: Training-Free Ultra-HR Remote Sensing VQA via Adaptive Zoom Search
Yunqi Zhou, Chengjie Jiang, Chun Yuan, Jing Li
With advances in satellite constellations, sensor technologies, and imaging pipelines, ultra-high-resolution (Ultra-HR) remote sensing imagery is becoming increasingly widespread. However, current remote sensing foundation models are ill-suited to such inputs: full-image encoding exhausts token and memory budgets, while resize-based preprocessing loses fine-grained and answer-critical details. In this context, guiding the model look where it matters before prediction becomes crucial. Therefore, we present ZoomSearch, a training-free, plug-and-play pipeline that decouples 'where to look' from 'how to answer' for Ultra-HR Remote Sensing Visual Question Answering (RS-VQA). ZoomSearch combines Adaptive Multi-Branch Zoom Search, which performs a hierarchical search over image patches to localize query-relevant regions, with Layout-Aware Patch Reassembly, which reorganizes the selected patches into a compact, layout-faithful canvas. We conduct comprehensive experiments on Ultra-HR RS-VQA benchmarks MME-RealWorld-RS and LRS-VQA, comparing against (i) strong general foundation models, (ii) remote sensing foundation models, (iii) Ultra-HR RS-VQA methods, and (iv) plug-and-play search-based VQA methods. When integrated with LLaVA-ov, ZoomSearch attains state-of-the-art accuracy across diverse tasks, improving the LLaVA-ov baseline by 26.3% on LRS-VQA and 114.8% on MME-RealWorld-RS. Meanwhile, it achieves much higher inference efficiency, outperforming prior search-based methods by 20%~44% in speed.
comment : 17 pages, 8 figures
♻ ☆ Activation Quantization of Vision Encoders Needs Prefixing Registers
Seunghyeon Kim, Jinho Kim, Taesun Yeom, Wonpyo Park, Kyuyeun Kim, Jaeho Lee
Transformer-based vision encoders -- such as CLIP -- are central to multimodal intelligence, powering applications from autonomous web agents to robotic control. Since these applications often demand real-time processing of massive visual data, reducing the inference cost of vision encoders is critical. Quantization offers a practical path, but remains challenging even at 8-bit precision due to massive-scale activations (i.e., outliers). In this work, we propose $\textit{RegCache}$, a training-free algorithm that mitigates outliers in large-scale pretrained vision encoders and serves as a plug-in module that can be applied on top of other quantization methods. The proposed RegCache introduces outlier-prone yet semantically meaningless prefix tokens to the target vision encoder, which prevents other tokens from having outliers. Notably, we observe that outliers in vision encoders behave differently from those in language models, motivating two technical innovations: middle-layer prefixing and token deletion. Experiments show that our method consistently improves the accuracy of quantized models across both text-supervised and self-supervised vision encoders.
comment : 19 pages, 8 figures
♻ ☆ SAEmnesia: Erasing Concepts in Diffusion Models with Supervised Sparse Autoencoders
Enrico Cassano, Riccardo Renzulli, Marco Nurisso, Mirko Zaffaroni, Alan Perotti, Marco Grangetto
Concept unlearning in diffusion models is hampered by feature splitting, where concepts are distributed across many latent features, making their removal challenging and computationally expensive. We introduce SAEmnesia, a supervised sparse autoencoder framework that overcomes this by enforcing one-to-one concept-neuron mappings. By systematically labeling concepts during training, our method achieves feature centralization, binding each concept to a single, interpretable neuron. This enables highly targeted and efficient concept erasure. SAEmnesia reduces hyperparameter search by 96.7% and achieves a 9.2% improvement over the state-of-the-art on the UnlearnCanvas benchmark. Our method also demonstrates superior scalability in sequential unlearning, improving accuracy by 28.4% when removing nine objects, establishing a new standard for precise and controllable concept erasure. Moreover, SAEmnesia mitigates the possibility of generating unwanted content under adversarial attack and effectively removes nudity when evaluated with I2P.
♻ ☆ Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansion
Keyang Lu, Sifan Zhou, Hongbin Xu, Gang Xu, Zhifei Yang, Yikai Wang, Zhen Xiao, Jieyi Long, Ming Li
Realistic 3D city generation is fundamental to a wide range of applications, including virtual reality and digital twins. However, most existing methods rely on training a single diffusion model, which limits their ability to generate personalized and boundless city-scale scenes. In this paper, we present Yo'City, a novel agentic framework that enables user-customized and infinitely expandable 3D city generation by leveraging the reasoning and compositional capabilities of off-the-shelf large models. Specifically, Yo'City first conceptualize the city through a top-down planning strategy that defines a hierarchical "City-District-Grid" structure. The Global Planner determines the overall layout and potential functional districts, while the Local Designer further refines each district with detailed grid-level descriptions. Subsequently, the grid-level 3D generation is achieved through a "produce-refine-evaluate" isometric image synthesis loop, followed by image-to-3D generation. To simulate continuous city evolution, Yo'City further introduces a user-interactive, relationship-guided expansion mechanism, which performs scene graph-based distance- and semantics-aware layout optimization, ensuring spatially coherent city growth. To comprehensively evaluate our method, we construct a diverse benchmark dataset and design six multi-dimensional metrics that assess generation quality from the perspectives of semantics, geometry, texture, and layout. Extensive experiments demonstrate that Yo'City consistently outperforms existing state-of-the-art methods across all evaluation aspects.
comment : 22 pages, 16 figures
♻ ☆ Learning Contrastive Feature Representations for Facial Action Unit Detection
Ziqiao Shang, Bin Liu, Fengmao Lv, Fei Teng, Tianrui Li, Lan-Zhe Guo
For the Facial Action Unit (AU) detection task, accurately capturing the subtle facial differences between distinct AUs is essential for reliable detection. Additionally, AU detection faces challenges from class imbalance and the presence of noisy or false labels, which undermine detection accuracy. In this paper, we introduce a novel contrastive learning framework aimed for AU detection that incorporates both self-supervised and supervised signals, thereby enhancing the learning of discriminative features for accurate AU detection. To tackle the class imbalance issue, we employ a negative sample re-weighting strategy that adjusts the step size of updating parameters for minority and majority class samples. Moreover, to address the challenges posed by noisy and false AU labels, we employ a sampling technique that encompasses three distinct types of positive sample pairs. This enables us to inject self-supervised signals into the supervised signal, effectively mitigating the adverse effects of noisy labels. Our experimental assessments, conducted on five widely-utilized benchmark datasets (BP4D, DISFA, BP4D+, GFT and Aff-Wild2), underscore the superior performance of our approach compared to state-of-the-art methods of AU detection. Our code is available at https://github.com/Ziqiao-Shang/AUNCE.
comment : This paper has been accepted by Pattern Recognition
♻ ☆ A Style is Worth One Code: Unlocking Code-to-Style Image Generation with Discrete Style Space
Huijie Liu, Shuhao Cui, Haoxiang Cao, Shuai Ma, Kai Wu, Guoliang Kang
Innovative visual stylization is a cornerstone of artistic creation, yet generating novel and consistent visual styles remains a significant challenge. Existing generative approaches typically rely on lengthy textual prompts, reference images, or parameter-efficient fine-tuning to guide style-aware image generation, but often struggle with style consistency, limited creativity, and complex style representations. In this paper, we affirm that a style is worth one numerical code by introducing the novel task, code-to-style image generation, which produces images with novel, consistent visual styles conditioned solely on a numerical style code. To date, this field has only been primarily explored by the industry (e.g., Midjourney), with no open-source research from the academic community. To fill this gap, we propose CoTyle, the first open-source method for this task. Specifically, we first train a discrete style codebook from a collection of images to extract style embeddings. These embeddings serve as conditions for a text-to-image diffusion model (T2I-DM) to generate stylistic images. Subsequently, we train an autoregressive style generator on the discrete style embeddings to model their distribution, allowing the synthesis of novel style embeddings. During inference, a numerical style code is mapped to a unique style embedding by the style generator, and this embedding guides the T2I-DM to generate images in the corresponding style. Unlike existing methods, our method offers unparalleled simplicity and diversity, unlocking a vast space of reproducible styles from minimal input. Extensive experiments validate that CoTyle effectively turns a numerical code into a style controller, demonstrating a style is worth one code.
comment : Code: https://github.com/Kwai-Kolors/CoTyle Demo: https://huggingface.co/spaces/Kwai-Kolors/CoTyle Homepage: https://kwai-kolors.github.io/CoTyle/
♻ ☆ Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy
Teng Hu, Zhentao Yu, Guozhen Zhang, Zihan Su, Zhengguang Zhou, Youliang Zhang, Yuan Zhou, Qinglin Lu, Ran Yi
The synthesis of synchronized audio-visual content is a key challenge in generative AI, with open-source models facing challenges in robust audio-video alignment. Our analysis reveals that this issue is rooted in three fundamental challenges of the joint diffusion process: (1) Correspondence Drift, where concurrently evolving noisy latents impede stable learning of alignment; (2) inefficient global attention mechanisms that fail to capture fine-grained temporal cues; and (3) the intra-modal bias of conventional Classifier-Free Guidance (CFG), which enhances conditionality but not cross-modal synchronization. To overcome these challenges, we introduce Harmony, a novel framework that mechanistically enforces audio-visual synchronization. We first propose a Cross-Task Synergy training paradigm to mitigate drift by leveraging strong supervisory signals from audio-driven video and video-driven audio generation tasks. Then, we design a Global-Local Decoupled Interaction Module for efficient and precise temporal-style alignment. Finally, we present a novel Synchronization-Enhanced CFG (SyncCFG) that explicitly isolates and amplifies the alignment signal during inference. Extensive experiments demonstrate that Harmony establishes a new state-of-the-art, significantly outperforming existing methods in both generation fidelity and, critically, in achieving fine-grained audio-visual synchronization.
♻ ☆ SegDINO3D: 3D Instance Segmentation Empowered by Both Image-Level and Object-Level 2D Features
Jinyuan Qu, Hongyang Li, Xingyu Chen, Shilong Liu, Yukai Shi, Tianhe Ren, Ruitao Jing, Lei Zhang
In this paper, we present SegDINO3D, a novel Transformer encoder-decoder framework for 3D instance segmentation. As 3D training data is generally not as sufficient as 2D training images, SegDINO3D is designed to fully leverage 2D representation from a pre-trained 2D detection model, including both image-level and object-level features, for improving 3D representation. SegDINO3D takes both a point cloud and its associated 2D images as input. In the encoder stage, it first enriches each 3D point by retrieving 2D image features from its corresponding image views and then leverages a 3D encoder for 3D context fusion. In the decoder stage, it formulates 3D object queries as 3D anchor boxes and performs cross-attention from 3D queries to 2D object queries obtained from 2D images using the 2D detection model. These 2D object queries serve as a compact object-level representation of 2D images, effectively avoiding the challenge of keeping thousands of image feature maps in the memory while faithfully preserving the knowledge of the pre-trained 2D model. The introducing of 3D box queries also enables the model to modulate cross-attention using the predicted boxes for more precise querying. SegDINO3D achieves the state-of-the-art performance on the ScanNetV2 and ScanNet200 3D instance segmentation benchmarks. Notably, on the challenging ScanNet200 dataset, SegDINO3D significantly outperforms prior methods by +8.6 and +6.8 mAP on the validation and hidden test sets, respectively, demonstrating its superiority.
♻ ☆ Autoregressive Styled Text Image Generation, but Make it Reliable WACV2026
Carmine Zaccagnino, Fabio Quattrini, Vittorio Pippi, Silvia Cascianelli, Alessio Tonioni, Rita Cucchiara
Generating faithful and readable styled text images (especially for Styled Handwritten Text generation - HTG) is an open problem with several possible applications across graphic design, document understanding, and image editing. A lot of research effort in this task is dedicated to developing strategies that reproduce the stylistic characteristics of a given writer, with promising results in terms of style fidelity and generalization achieved by the recently proposed Autoregressive Transformer paradigm for HTG. However, this method requires additional inputs, lacks a proper stop mechanism, and might end up in repetition loops, generating visual artifacts. In this work, we rethink the autoregressive formulation by framing HTG as a multimodal prompt-conditioned generation task, and tackle the content controllability issues by introducing special textual input tokens for better alignment with the visual ones. Moreover, we devise a Classifier-Free-Guidance-based strategy for our autoregressive model. Through extensive experimental validation, we demonstrate that our approach, dubbed Eruku, compared to previous solutions requires fewer inputs, generalizes better to unseen styles, and follows more faithfully the textual prompt, improving content adherence.
comment : Accepted at WACV2026
♻ ☆ Infrared and Visible Image Fusion with Language-Driven Loss in CLIP Embedding Space ACM MM 2025
Yuhao Wang, Lingjuan Miao, Zhiqiang Zhou, Lei Zhang, Yajun Qiao
Infrared-visible image fusion (IVIF) has attracted much attention owing to the highly-complementary properties of the two image modalities. Due to the lack of ground-truth fused images, the fusion output of current deep-learning based methods heavily depends on the loss functions defined mathematically. As it is hard to well mathematically define the fused image without ground truth, the performance of existing fusion methods is limited. In this paper, we propose to use natural language to express the objective of IVIF, which can avoid the explicit mathematical modeling of fusion output in current losses, and make full use of the advantage of language expression to improve the fusion performance. For this purpose, we present a comprehensive language-expressed fusion objective, and encode relevant texts into the multi-modal embedding space using CLIP. A language-driven fusion model is then constructed in the embedding space, by establishing the relationship among the embedded vectors representing the fusion objective and input image modalities. Finally, a language-driven loss is derived to make the actual IVIF aligned with the embedded language-driven fusion model via supervised training. Experiments show that our method can obtain much better fusion results than existing techniques. The code is available at https://github.com/wyhlaowang/LDFusion.
comment : Accepted by ACM MM 2025
♻ ☆ Hybrid Rendering for Multimodal Autonomous Driving: Merging Neural and Physics-Based Simulation
Máté Tóth, Péter Kovács, Réka Bencses, Zoltán Bendefy, Zoltán Hortsin, Balázs Teréki, Tamás Matuszka
Neural reconstruction models for autonomous driving simulation have made significant strides in recent years, with dynamic models becoming increasingly prevalent. However, these models are typically limited to handling in-domain objects closely following their original trajectories. We introduce a hybrid approach that combines the strengths of neural reconstruction with physics-based rendering. This method enables the virtual placement of traditional mesh-based dynamic agents at arbitrary locations, adjustments to environmental conditions, and rendering from novel camera viewpoints. Our approach significantly enhances novel view synthesis quality -- especially for road surfaces and lane markings -- while maintaining interactive frame rates through our novel training method, NeRF2GS. This technique leverages the superior generalization capabilities of NeRF-based methods and the real-time rendering speed of 3D Gaussian Splatting (3DGS). We achieve this by training a customized NeRF model on the original images with depth regularization derived from a noisy LiDAR point cloud, then using it as a teacher model for 3DGS training. This process ensures accurate depth, surface normals, and camera appearance modeling as supervision. With our block-based training parallelization, the method can handle large-scale reconstructions (greater than or equal to 100,000 square meters) and predict segmentation masks, surface normals, and depth maps. During simulation, it supports a rasterization-based rendering backend with depth-based composition and multiple camera models for real-time camera simulation, as well as a ray-traced backend for precise LiDAR simulation.
♻ ☆ TRACE: Temporally Reliable Anatomically-Conditioned 3D CT Generation with Enhanced Efficiency MICCAI 2025
Minye Shao, Xingyu Miao, Haoran Duan, Zeyu Wang, Jingkun Chen, Yawen Huang, Xian Wu, Jingjing Deng, Yang Long, Yefeng Zheng
3D medical image generation is essential for data augmentation and patient privacy, calling for reliable and efficient models suited for clinical practice. However, current methods suffer from limited anatomical fidelity, restricted axial length, and substantial computational cost, placing them beyond reach for regions with limited resources and infrastructure. We introduce TRACE, a framework that generates 3D medical images with spatiotemporal alignment using a 2D multimodal-conditioned diffusion approach. TRACE models sequential 2D slices as video frame pairs, combining segmentation priors and radiology reports for anatomical alignment, incorporating optical flow to sustain temporal coherence. During inference, an overlapping-frame strategy links frame pairs into a flexible length sequence, reconstructed into a spatiotemporally and anatomically aligned 3D volume. Experimental results demonstrate that TRACE effectively balances computational efficiency with preserving anatomical fidelity and spatiotemporal consistency. Code is available at: https://github.com/VinyehShaw/TRACE.
comment : Accepted to MICCAI 2025 (this version is not peer-reviewed; it is the extended version)
♻ ☆ Predicting Video Slot Attention Queries from Random Slot-Feature Pairs AAAI 2026
Rongzhen Zhao, Jian Li, Juho Kannala, Joni Pajarinen
Unsupervised video Object-Centric Learning (OCL) is promising as it enables object-level scene representation and understanding as we humans do. Mainstream video OCL methods adopt a recurrent architecture: An aggregator aggregates current video frame into object features, termed slots, under some queries; A transitioner transits current slots to queries for the next frame. This is an effective architecture but all existing implementations both (\textit{i1}) neglect to incorporate next frame features, the most informative source for query prediction, and (\textit{i2}) fail to learn transition dynamics, the knowledge essential for query prediction. To address these issues, we propose Random Slot-Feature pair for learning Query prediction (RandSF.Q): (\textit{t1}) We design a new transitioner to incorporate both slots and features, which provides more information for query prediction; (\textit{t2}) We train the transitioner to predict queries from slot-feature pairs randomly sampled from available recurrences, which drives it to learn transition dynamics. Experiments on scene representation demonstrate that our method surpass existing video OCL methods significantly, e.g., up to 10 points on object discovery, setting new state-of-the-art. Such superiority also benefits downstream tasks like scene understanding. Source Code, Model Checkpoints, Training Logs: https://github.com/Genera1Z/RandSF.Q
comment : Accepted to AAAI 2026
♻ ☆ HiGFA: Hierarchical Guidance for Fine-grained Data Augmentation with Diffusion Models
Zhiguang Lu, Qianqian Xu, Peisong Wen, Siran Dai, Qingming Huang
Generative diffusion models show promise for data augmentation. However, applying them to fine-grained tasks presents a significant challenge: ensuring synthetic images accurately capture the subtle, category-defining features critical for high fidelity. Standard approaches, such as text-based Classifier-Free Guidance (CFG), often lack the required specificity, potentially generating misleading examples that degrade fine-grained classifier performance. To address this, we propose Hierarchically Guided Fine-grained Augmentation (HiGFA). HiGFA leverages the temporal dynamics of the diffusion sampling process. It employs strong text and transformed contour guidance with fixed strengths in the early-to-mid sampling stages to establish overall scene, style, and structure. In the final sampling stages, HiGFA activates a specialized fine-grained classifier guidance and dynamically modulates the strength of all guidance signals based on prediction confidence. This hierarchical, confidence-driven orchestration enables HiGFA to generate diverse yet faithful synthetic images by intelligently balancing global structure formation with precise detail refinement. Experiments on several FGVC datasets demonstrate the effectiveness of HiGFA.
♻ ☆ FIELDS: Face reconstruction with accurate Inference of Expression using Learning with Direct Super vision
Chen Ling, Henglin Shi, Hedvig Kjellström
Facial expressions convey the bulk of emotional information in human communication, yet existing 3D face reconstruction methods often miss subtle affective details due to reliance on 2D supervision and lack of 3D ground truth. We propose FIELDS (Face reconstruction with accurate Inference of Expression using Learning with Direct Supervision) to address these limitations by extending self-supervised 2D image consistency cues with direct 3D expression parameter supervision and an auxiliary emotion recognition branch. Our encoder is guided by authentic expression parameters from spontaneous 4D facial scans, while an intensity-aware emotion loss encourages the 3D expression parameters to capture genuine emotion content without exaggeration. This dual-supervision strategy bridges the 2D/3D domain gap and mitigates expression-intensity bias, yielding high-fidelity 3D reconstructions that preserve subtle emotional cues. From a single image, FIELDS produces emotion-rich face models with highly realistic expressions, significantly improving in-the-wild facial expression recognition performance without sacrificing naturalness.
♻ ☆ AgriPotential: A Novel Multi-Spectral and Multi-Temporal Remote Sensing Dataset for Agricultural Potentials
Mohammad El Sakka, Caroline De Pourtales, Lotfi Chaari, Josiane Mothe
Remote sensing has emerged as a critical tool for large-scale Earth monitoring and land management. In this paper, we introduce AgriPotential, a novel benchmark dataset composed of Sentinel-2 satellite imagery captured over multiple months. The dataset provides pixel-level annotations of agricultural potentials for three major crop types - viticulture, market gardening, and field crops - across five ordinal classes. AgriPotential supports a broad range of machine learning tasks, including ordinal regression, multi-label classification, and spatio-temporal modeling. The data cover diverse areas in Southern France, offering rich spectral information. AgriPotential is the first public dataset designed specifically for agricultural potential prediction, aiming to improve data-driven approaches to sustainable land use planning. The dataset and the code are freely accessible at: https://zenodo.org/records/15551829
comment : Accepted at CBMI 2025
♻ ☆ OmniAID: Decoupling Semantic and Artifacts for Universal AI-Generated Image Detection in the Wild
Yuncheng Guo, Junyan Ye, Chenjue Zhang, Hengrui Kang, Haohuan Fu, Conghui He, Weijia Li
A truly universal AI-Generated Image (AIGI) detector must simultaneously generalize across diverse generative models and varied semantic content. Current state-of-the-art methods learn a single, entangled forgery representation, conflating content-dependent flaws with content-agnostic artifacts, and are further constrained by outdated benchmarks. To overcome these limitations, we propose OmniAID, a novel framework centered on a decoupled Mixture-of-Experts (MoE) architecture. The core of our method is a hybrid expert system designed to decouple: (1) semantic flaws across distinct content domains, and (2) content-dependent flaws from content-agnostic universal artifacts. This system employs a set of Routable Specialized Semantic Experts, each for a distinct domain (e.g., human, animal), complemented by a Fixed Universal Artifact Expert. This architecture is trained using a novel two-stage strategy: we first train the experts independently with domain-specific hard-sampling to ensure specialization, and subsequently train a lightweight gating network for effective input routing. By explicitly decoupling "what is generated" (content-specific flaws) from "how it is generated" (universal artifacts), OmniAID achieves robust generalization. To address outdated benchmarks and validate real-world applicability, we introduce Mirage, a new large-scale, contemporary dataset. Extensive experiments, using both traditional benchmarks and our Mirage dataset, demonstrate our model surpasses existing monolithic detectors, establishing a new and robust standard for AIGI authentication against modern, in-the-wild threats.
comment : 19 pages, 10 figures, 19 tables
♻ ☆ DiffFuSR: Super-Resolution of all Sentinel-2 Multispectral Bands using Diffusion Models SC
Muhammad Sarmad, Arnt-Børre Salberg, Michael Kampffmeyer
This paper presents DiffFuSR, a modular pipeline for super-resolving all 12 spectral bands of Sentinel-2 Level-2A imagery to a unified ground sampling distance (GSD) of 2.5 meters. The pipeline comprises two stages: (i) a diffusion-based super-resolution (SR) model trained on high-resolution RGB imagery from the NAIP and WorldStrat datasets, harmonized to simulate Sentinel-2 characteristics; and (ii) a learned fusion network that upscales the remaining multispectral bands using the super-resolved RGB image as a spatial prior. We introduce a robust degradation model and contrastive degradation encoder to support blind SR. Extensive evaluations of the proposed SR pipeline on the OpenSR benchmark demonstrate that the proposed method outperforms current SOTA baselines in terms of reflectance fidelity, spectral consistency, spatial alignment, and hallucination suppression. Furthermore, the fusion network significantly outperforms classical and learned pansharpening approaches, enabling accurate enhancement of Sentinel-2's 20 m and 60 m bands. This work proposes a novel modular framework Sentinel-2 SR that utilizes harmonized learning with diffusion models and fusion strategies. Our code and models can be found at https://github.com/NorskRegnesentral/DiffFuSR.
comment : Accepted for Publication at IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING (TGRS)
♻ ☆ Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory
Yuqi Wu, Wenzhao Zheng, Jie Zhou, Jiwen Lu
Dense 3D scene reconstruction from an ordered sequence or unordered image collections is a critical step when bringing research in computer vision into practical scenarios. Following the paradigm introduced by DUSt3R, which unifies an image pair densely into a shared coordinate system, subsequent methods maintain an implicit memory to achieve dense 3D reconstruction from more images. However, such implicit memory is limited in capacity and may suffer from information loss of earlier frames. We propose Point3R, an online framework targeting dense streaming 3D reconstruction. To be specific, we maintain an explicit spatial pointer memory directly associated with the 3D structure of the current scene. Each pointer in this memory is assigned a specific 3D position and aggregates scene information nearby in the global coordinate system into a changing spatial feature. Information extracted from the latest frame interacts explicitly with this pointer memory, enabling dense integration of the current observation into the global coordinate system. We design a 3D hierarchical position embedding to promote this interaction and design a simple yet effective fusion mechanism to ensure that our pointer memory is uniform and efficient. Our method achieves competitive or state-of-the-art performance on various tasks with low training costs. Code: https://github.com/YkiWu/Point3R.
comment : Code is available at: https://github.com/YkiWu/Point3R
♻ ☆ Reliable Multimodal Learning Via Multi-Level Adaptive DeConfusion
Tong Zhang, Shu Shen, C. L. Philip Chen
Multimodal learning enhances the performance of various machine learning tasks by leveraging complementary information across different modalities. However, existing methods often learn multimodal representations that retain substantial inter-class confusion, making it difficult to achieve high-confidence predictions, particularly in real-world scenarios with low-quality or noisy data. To address this challenge, we propose Multi-Level Adaptive DeConfusion (MLAD), which eliminates inter-class confusion in multimodal data at both global and sample levels, significantly enhancing the classification reliability of multimodal models. Specifically, MLAD first learns class-wise latent distributions with global-level confusion removed via dynamic-exit modality encoders that adapt to the varying discrimination difficulty of each class and a cross-class residual reconstruction mechanism. Subsequently, MLAD further removes sample-specific confusion through sample-adaptive cross-modality rectification guided by confusion-free modality priors. These priors are constructed from low-confusion modality features, identified by evaluating feature confusion using the learned class-wise latent distributions and selecting those with low confusion via a Gaussian mixture model. Experiments demonstrate that MLAD outperforms state-of-the-art methods across multiple benchmarks and exhibits superior reliability.
comment : 15 pages, 10 figures
♻ ☆ PhysX-3D: Physical-Grounded 3D Asset Generation NeurIPS 2025
Ziang Cao, Zhaoxi Chen, Liang Pan, Ziwei Liu
3D modeling is moving from virtual to physical. Existing 3D generation primarily emphasizes geometries and textures while neglecting physical-grounded modeling. Consequently, despite the rapid development of 3D generative models, the synthesized 3D assets often overlook rich and important physical properties, hampering their real-world application in physical domains like simulation and embodied AI. As an initial attempt to address this challenge, we propose \textbf{PhysX-3D}, an end-to-end paradigm for physical-grounded 3D asset generation. 1) To bridge the critical gap in physics-annotated 3D datasets, we present PhysXNet - the first physics-grounded 3D dataset systematically annotated across five foundational dimensions: absolute scale, material, affordance, kinematics, and function description. In particular, we devise a scalable human-in-the-loop annotation pipeline based on vision-language models, which enables efficient creation of physics-first assets from raw 3D assets.2) Furthermore, we propose \textbf{PhysXGen}, a feed-forward framework for physics-grounded image-to-3D asset generation, injecting physical knowledge into the pre-trained 3D structural space. Specifically, PhysXGen employs a dual-branch architecture to explicitly model the latent correlations between 3D structures and physical properties, thereby producing 3D assets with plausible physical predictions while preserving the native geometry quality. Extensive experiments validate the superior performance and promising generalization capability of our framework. All the code, data, and models will be released to facilitate future research in generative physical AI.
comment : Accepted by NeurIPS 2025, Spotlight Project page: https://physx-3d.github.io/
♻ ☆ ABM-LoRA: Activation Boundary Matching for Fast Convergence in Low-Rank Adaptation
Dongha Lee, Jinhee Park, Minjun Kim, Junseok Kwon
We propose Activation Boundary Matching for Low-Rank Adaptation (ABM-LoRA), a principled initialization strategy that substantially accelerates the convergence of low-rank adapters. While LoRA offers high parameter efficiency, its random initialization restricts gradient updates to a mismatched tangent space, causing significant information loss and hindering early convergence. Our ABM-LoRA addresses this by aligning the adapter's activation boundaries with those of the pretrained model before downstream training, thereby maximizing the projection of full-parameter gradients into the adapter subspace. This alignment sharply reduces information loss at initialization, yields a lower starting loss, and accelerates convergence. We demonstrate ABM-LoRA's effectiveness across diverse architectures and tasks: language understanding (T5-Base on GLUE), dialogue generation (LLaMA2-7B on WizardLM), and vision recognition (ViT-B/16 on VTAB-1K). On VTAB-1K, it achieves the highest accuracy among all methods, with strong gains on structured reasoning tasks requiring geometric understanding.
comment : 16 pages, 5 figures, under review
♻ ☆ When Trackers Date Fish: A Benchmark and Framework for Underwater Multiple Fish Tracking AAAI 2026
Weiran Li, Yeqiang Liu, Qiannan Guo, Yijie Wei, Hwa Liang Leo, Zhenbo Li
Multiple object tracking (MOT) technology has made significant progress in terrestrial applications, but underwater tracking scenarios remain underexplored despite their importance to marine ecology and aquaculture. In this paper, we present Multiple Fish Tracking Dataset 2025 (MFT25), a comprehensive dataset specifically designed for underwater multiple fish tracking, featuring 15 diverse video sequences with 408,578 meticulously annotated bounding boxes across 48,066 frames. Our dataset captures various underwater environments, fish species, and challenging conditions including occlusions, similar appearances, and erratic motion patterns. Additionally, we introduce Scale-aware and Unscented Tracker (SU-T), a specialized tracking framework featuring an Unscented Kalman Filter (UKF) optimized for non-linear swimming patterns of fish and a novel Fish-Intersection-over-Union (FishIoU) matching that accounts for the unique morphological characteristics of aquatic species. Extensive experiments demonstrate that our SU-T baseline achieves state-of-the-art performance on MFT25, with 34.1 HOTA and 44.6 IDF1, while revealing fundamental differences between fish tracking and terrestrial object tracking scenarios. The dataset and codes are released at https://vranlee.github.io/SU-T/.
comment : Accepted by AAAI 2026 (Oral)
♻ ☆ Enhanced Partially Relevant Video Retrieval through Inter- and Intra-Sample Analysis with Coherence Prediction
Junlong Ren, Gangjian Zhang, Yu Hu, Jian Shu, Hui Xiong, Hao Wang
Partially Relevant Video Retrieval (PRVR) aims to retrieve the target video that is partially relevant to the text query. The primary challenge in PRVR arises from the semantic asymmetry between textual and visual modalities, as videos often contain substantial content irrelevant to the query. Existing methods coarsely align paired videos and text queries to construct the semantic space, neglecting the critical cross-modal dual nature inherent in this task: inter-sample correlation and intra-sample redundancy. To this end, we propose a novel PRVR framework to systematically exploit these two characteristics. Our framework consists of three core modules. First, the Inter Correlation Enhancement (ICE) module captures inter-sample correlation by identifying semantically similar yet unpaired text queries and video moments, combining them to form pseudo-positive pairs for more robust semantic space construction. Second, the Intra Redundancy Mining (IRM) module mitigates intra-sample redundancy by mining redundant moment features and distinguishing them from query-relevant moments, encouraging the model to learn more discriminative representations. Finally, to reinforce these modules, we introduce the Temporal Coherence Prediction (TCP) module, enhancing discrimination of fine-grained moment-level semantics by training the model to predict the original temporal order of randomly shuffled video sequences. Extensive experiments demonstrate the superiority of our method, achieving state-of-the-art results.
♻ ☆ Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective AAAI 2026
Nhat Chung, Taisei Hanyu, Toan Nguyen, Huy Le, Frederick Bumgarner, Duy Minh Ho Nguyen, Khoa Vo, Kashu Yamazaki, Chase Rainwater, Tung Kieu, Anh Nguyen, Ngan Le
As embodied agents operate in increasingly complex environments, the ability to perceive, track, and reason about individual object instances over time becomes essential, especially in tasks requiring sequenced interactions with visually similar objects. In these non-Markovian settings, key decision cues are often hidden in object-specific histories rather than the current scene. Without persistent memory of prior interactions (what has been interacted with, where it has been, or how it has changed) visuomotor policies may fail, repeat past actions, or overlook completed ones. To surface this challenge, we introduce LIBERO-Mem, a non-Markovian task suite for stress-testing robotic manipulation under object-level partial observability. It combines short- and long-horizon object tracking with temporally sequenced subgoals, requiring reasoning beyond the current frame. However, vision-language-action (VLA) models often struggle in such settings, with token scaling quickly becoming intractable even for tasks spanning just a few hundred frames. We propose Embodied-SlotSSM, a slot-centric VLA framework built for temporal scalability. It maintains spatio-temporally consistent slot identities and leverages them through two mechanisms: (1) slot-state-space modeling for reconstructing short-term history, and (2) a relational encoder to align the input tokens with action decoding. Together, these components enable temporally grounded, context-aware action prediction. Experiments show Embodied-SlotSSM's baseline performance on LIBERO-Mem and general tasks, offering a scalable solution for non-Markovian reasoning in object-centric robotic policies.
comment : Accepted at AAAI 2026
♻ ☆ SlotVLA: Towards Modeling of Object- Relation Representations in Robotic Manipulation
Taisei Hanyu, Nhat Chung, Huy Le, Toan Nguyen, Yuki Ikebe, Anthony Gunderman, Duy Nguyen Ho Minh, Khoa Vo, Tung Kieu, Kashu Yamazaki, Chase Rainwater, Anh Nguyen, Ngan Le
Inspired by how humans reason over discrete objects and their relationships, we explore whether compact object-centric and object-relation representations can form a foundation for multitask robotic manipulation. Most existing robotic multitask models rely on dense embeddings that entangle both object and background cues, raising concerns about both efficiency and interpretability. In contrast, we study object-relation-centric representations as a pathway to more structured, efficient, and explainable visuomotor control. Our contributions are two-fold. First, we introduce LIBERO+, a fine-grained benchmark dataset designed to enable and evaluate object-relation reasoning in robotic manipulation. Unlike prior datasets, LIBERO+ provides object-centric annotations that enrich demonstrations with box- and mask-level labels as well as instance-level temporal tracking, supporting compact and interpretable visuomotor representations. Second, we propose SlotVLA, a slot-attention-based framework that captures both objects and their relations for action decoding. It uses a slot-based visual tokenizer to maintain consistent temporal object representations, a relation-centric decoder to produce task-relevant embeddings, and an LLM-driven module that translates these embeddings into executable actions. Experiments on LIBERO+ demonstrate that object-centric slot and object-relation slot representations drastically reduce the number of required visual tokens, while providing competitive generalization. Together, LIBERO+ and SlotVLA provide a compact, interpretable, and effective foundation for advancing object-relation-centric robotic manipulation.
comment : under review
♻ ☆ UNO: Unifying One-stage Video Scene Graph Generation via Object-Centric Visual Representation Learning WACV 2026
Huy Le, Nhat Chung, Tung Kieu, Jingkang Yang, Ngan Le
Video Scene Graph Generation (VidSGG) aims to represent dynamic visual content by detecting objects and modeling their temporal interactions as structured graphs. Prior studies typically target either coarse-grained box-level or fine-grained panoptic pixel-level VidSGG, often requiring task-specific architectures and multi-stage training pipelines. In this paper, we present UNO (UNified Object-centric VidSGG), a single-stage, unified framework that jointly addresses both tasks within an end-to-end architecture. UNO is designed to minimize task-specific modifications and maximize parameter sharing, enabling generalization across different levels of visual granularity. The core of UNO is an extended slot attention mechanism that decomposes visual features into object and relation slots. To ensure robust temporal modeling, we introduce object temporal consistency learning, which enforces consistent object representations across frames without relying on explicit tracking modules. Additionally, a dynamic triplet prediction module links relation slots to corresponding object pairs, capturing evolving interactions over time. We evaluate UNO on standard box-level and pixel-level VidSGG benchmarks. Results demonstrate that UNO not only achieves competitive performance across both tasks but also offers improved efficiency through a unified, object-centric design.
comment : 11 pages, 7 figures. Accepted at WACV 2026
♻ ☆ ARIAL: An Agent ic Framework for Document VQA with Precise Answer Localization
Ahmad Mohammadshirazi, Pinaki Prasad Guha Neogi, Dheeraj Kulshrestha, Rajiv Ramnath
Document Visual Question Answering (VQA) requires models to not only extract accurate textual answers but also precisely localize them within document images, a capability critical for interpretability in high-stakes applications. However, existing systems achieve strong textual accuracy while producing unreliable spatial grounding, or sacrifice performance for interpretability. We present ARIAL (Agentic Reasoning for Interpretable Answer Localization), a modular framework that orchestrates specialized tools through an LLM-based planning agent to achieve both precise answer extraction and reliable spatial grounding. ARIAL decomposes Document VQA into structured subtasks: OCR-based text extraction with TrOCR, retrieval-augmented context selection using semantic search, answer generation via a fine-tuned Gemma 3-27B model, and explicit bounding-box localization through text-to-region alignment. This modular architecture produces transparent reasoning traces, enabling tool-level auditability and independent component optimization. We evaluate ARIAL on four benchmarks (DocVQA, FUNSD, CORD, and SROIE) using both textual accuracy (ANLS) and spatial precision (mAP at IoU 0.50 to 0.95). ARIAL achieves state-of-the-art results across all datasets: 88.7 ANLS and 50.1 mAP on DocVQA, 90.0 ANLS and 50.3 mAP on FUNSD, 85.5 ANLS and 60.2 mAP on CORD, and 93.1 ANLS on SROIE, surpassing the previous best method (DLaVA) by +2.8 ANLS and +3.9 mAP on DocVQA. Our work demonstrates how agentic orchestration of specialized tools can simultaneously improve performance and interpretability, providing a pathway toward trustworthy, explainable document AI systems.
♻ ☆ Reverberation: Learning the Latencies Before Forecasting Trajectories
Conghao Wong, Ziqian Zou, Beihao Xia, Xinge You
Bridging the past to the future, connecting agents both spatially and temporally, lies at the core of the trajectory prediction task. Despite great efforts, it remains challenging to explicitly learn and predict latencies, i.e., response intervals or temporal delays with which agents respond to various trajectory-changing events and adjust their future paths, whether on their own or interactively. Different agents may exhibit distinct latency preferences for noticing, processing, and reacting to a specific trajectory-changing event. The lack of consideration of such latencies may undermine the causal continuity of forecasting systems, leading to implausible or unintended trajectories. Inspired by reverberation in acoustics, we propose a new reverberation transform and the corresponding Reverberation (short for Rev) trajectory prediction model, which predicts both individual latency preferences and their stochastic variations accordingly, by using two explicit and learnable reverberation kernels, enabling latency-conditioned and controllable trajectory prediction of both non-interactive and social latencies. Experiments on multiple datasets, whether pedestrians or vehicles, demonstrate that Rev achieves competitive accuracy while revealing interpretable latency dynamics across agents and scenarios. Qualitative analyses further verify the properties of the reverberation transform, highlighting its potential as a general latency modeling approach.
♻ ☆ AlignBench: Benchmark ing Fine-Grained Image-Text Alignment with Synthetic Image-Caption Pairs
Kuniaki Saito, Risa Shinoda, Shohei Tanaka, Tosho Hirasawa, Fumio Okura, Yoshitaka Ushiku
Assessing image-text alignment models such as CLIP is crucial for bridging visual and linguistic representations. Yet existing benchmarks rely on rule-based perturbations or short captions, limiting their ability to measure fine-grained alignment. We introduce AlignBench, a benchmark that provides a new indicator of image-text alignment by evaluating detailed image-caption pairs generated by diverse image-to-text and text-to-image models. Each sentence is annotated for correctness, enabling direct assessment of VLMs as alignment evaluators. Benchmarking a wide range of decoder-based VLMs reveals three key findings: (i) CLIP-based models, even those tailored for compositional reasoning, remain nearly blind; (ii) detectors systematically over-score early sentences; and (iii) they show strong self-preference, favoring their own outputs and harming detection performance. Our project page will be available at https://dahlian00.github.io/AlignBench/.
comment : Project Page: https://dahlian00.github.io/AlignBench/
♻ ☆ DDAE++: Enhancing Diffusion Models Towards Unified Generative and Discriminative Learning
Weilai Xiang, Hongyu Yang, Di Huang, Yunhong Wang
While diffusion models excel at image synthesis, useful representations have been shown to emerge from generative pre-training, suggesting a path towards unified generative and discriminative learning. However, suboptimal semantic flow within current architectures can hinder this potential: features encoding the richest high-level semantics are underutilized and diluted when propagating through decoding layers, impeding the formation of an explicit semantic bottleneck layer. To address this, we introduce self-conditioning, a lightweight mechanism that reshapes the model's layer-wise semantic hierarchy without external guidance. By aggregating and rerouting intermediate features to guide subsequent decoding layers, our method concentrates more high-level semantics, concurrently strengthening global generative guidance and forming more discriminative representations. This simple approach yields a dual-improvement trend across pixel-space UNet, UViT and latent-space DiT models with minimal overhead. Crucially, it creates an architectural semantic bridge that propagates discriminative improvements into generation and accommodates further techniques such as contrastive self-distillation. Experiments show that our enhanced models, especially self-conditioned DiT, are powerful dual learners that yield strong and transferable representations on image and dense classification tasks, surpassing various generative self-supervised models in linear probing while also improving or maintaining high generation quality.
comment : Updated version. Code available at https://github.com/FutureXiang/ddae_plus_plus
♻ ☆ OpenDance: Multimodal Controllable 3D Dance Generation with Large-scale Internet Data
Jinlu Zhang, Zixi Kang, Libin Liu, Jianlong Chang, Qi Tian, Feng Gao, Yizhou Wang
Music-driven 3D dance generation offers significant creative potential, yet practical applications demand versatile and multimodal control. As the highly dynamic and complex human motion covering various styles and genres, dance generation requires satisfying diverse conditions beyond just music (e.g., spatial trajectories, keyframe gestures, or style descriptions). However, the absence of a large-scale and richly annotated dataset severely hinders progress. In this paper, we build OpenDanceSet, an extensive human dance dataset comprising over 100 hours across 14 genres and 147 subjects. Each sample has rich annotations to facilitate robust cross-modal learning: 3D motion, paired music, 2D keypoints, trajectories, and expert-annotated text descriptions. Furthermore, we propose OpenDanceNet, a unified masked modeling framework for controllable dance generation, including a disentangled auto-encoder and a multimodal joint-prediction Transformer. OpenDanceNet supports generation conditioned on music and arbitrary combinations of text, keypoints, or trajectories. Comprehensive experiments demonstrate that our work achieves high-fidelity synthesis with strong diversity and realistic physical contacts, while also offering flexible control over spatial and stylistic conditions. Project Page: https://open-dance.github.io
♻ ☆ Axial-UNet: A Neural Weather Model for Precipitation Nowcasting
Sumit Mamtani, Maitreya Sonawane
Accurately predicting short-term precipitation is critical for weather-sensitive applications such as disaster management, aviation, and urban planning. Traditional numerical weather prediction can be computationally intensive at high resolution and short lead times. In this work, we propose a lightweight UNet-based encoder-decoder augmented with axial-attention blocks that attend along image rows and columns to capture long-range spatial interactions, while temporal context is provided by conditioning on multiple past radar frames. Our hybrid architecture captures both local and long-range spatio-temporal dependencies from radar image sequences, enabling fixed lead-time precipitation nowcasting with modest compute. Experimental results on a preprocessed subset of the HKO-7 radar dataset demonstrate that our model outperforms ConvLSTM, pix2pix-style cGANs, and a plain UNet in pixel-fidelity metrics, reaching PSNR 47.67 and SSIM 0.9943. We report PSNR/SSIM here; extending evaluation to meteorology-oriented skill measures (e.g., CSI/FSS) is left to future work. The approach is simple, scalable, and effective for resource-constrained, real-time forecasting scenarios.
comment : 16 pages, 3 figures. Accepted at the International Conference on Distributed Computing and Intelligent Technology (ICDCIT 2026), to appear in Springer LNCS
♻ ☆ Tracking the Unstable: Appearance-Guided Motion Modeling for Robust Multi-Object Tracking in UAV-Captured Videos AAAI26
Jianbo Ma, Hui Luo, Qi Chen, Yuankai Qi, Yumei Sun, Amin Beheshti, Jianlin Zhang, Ming-Hsuan Yang
Multi-object tracking (MOT) aims to track multiple objects while maintaining consistent identities across frames of a given video. In unmanned aerial vehicle (UAV) recorded videos, frequent viewpoint changes and complex UAV-ground relative motion dynamics pose significant challenges, which often lead to unstable affinity measurement and ambiguous association. Existing methods typically model motion and appearance cues separately, overlooking their spatio-temporal interplay and resulting in suboptimal tracking performance. In this work, we propose AMOT, which jointly exploits appearance and motion cues through two key components: an Appearance-Motion Consistency (AMC) matrix and a Motion-aware Track Continuation (MTC) module. Specifically, the AMC matrix computes bi-directional spatial consistency under the guidance of appearance features, enabling more reliable and context-aware identity association. The MTC module complements AMC by reactivating unmatched tracks through appearance-guided predictions that align with Kalman-based predictions, thereby reducing broken trajectories caused by missed detections. Extensive experiments on three UAV benchmarks, including VisDrone2019, UAVDT, and VT-MOT-UAV, demonstrate that our AMOT outperforms current state-of-the-art methods and generalizes well in a plug-and-play and training-free manner.
comment : Accepted by the AAAI26 Conference Main Track
♻ ☆ ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs
Chaoyu Li, Yogesh Kulkarni, Pooyan Fazli
The computational cost of training multimodal large language models (MLLMs) grows rapidly with the number of processed tokens. Existing efficiency methods mainly target inference via token reduction or merging, offering limited benefits during training. We introduce ReGATE (Reference-Guided Adaptive Token Elision), an adaptive token pruning method for accelerating MLLM training. ReGATE adopts a teacher-student framework, in which a frozen teacher LLM provides per-token guidance losses that are fused with an exponential moving average of the student's difficulty estimates. This adaptive scoring mechanism dynamically selects informative tokens while skipping redundant ones in the forward pass, substantially reducing computation without altering the model architecture. Across three representative MLLMs, ReGATE matches the peak accuracy of standard training on MVBench up to 2$\times$ faster, using only 38% of the tokens. With extended training, it even surpasses the baseline across multiple multimodal benchmarks, cutting total token usage by over 41%. Code and models will be released publicly.
♻ ☆ Leveraging Semantic Attribute Binding for Free-Lunch Color Control in Diffusion Models WACV 2026
Héctor Laria, Alexandra Gomez-Villa, Jiang Qin, Muhammad Atif Butt, Bogdan Raducanu, Javier Vazquez-Corral, Joost van de Weijer, Kai Wang
Recent advances in text-to-image (T2I) diffusion models have enabled remarkable control over various attributes, yet precise color specification remains a fundamental challenge. Existing approaches, such as ColorPeel, rely on model personalization, requiring additional optimization and limiting flexibility in specifying arbitrary colors. In this work, we introduce ColorWave, a novel training-free approach that achieves exact RGB-level color control in diffusion models without fine-tuning. By systematically analyzing the cross-attention mechanisms within IP-Adapter, we uncover an implicit binding between textual color descriptors and reference image features. Leveraging this insight, our method rewires these bindings to enforce precise color attribution while preserving the generative capabilities of pretrained models. Our approach maintains generation quality and diversity, outperforming prior methods in accuracy and applicability across diverse object categories. Through extensive evaluations, we demonstrate that ColorWave establishes a new paradigm for structured, color-consistent diffusion-based image synthesis.
comment : WACV 2026. Project page: https://hecoding.github.io/colorwave-page
♻ ☆ VLMs have Tunnel Vision : Evaluating Nonlocal Visual Reasoning in Leading VLMs
Shmuel Berman, Jia Deng
Vision-Language Models (VLMs) excel at complex visual tasks such as VQA and chart understanding, yet recent work suggests they struggle with simple perceptual tests. We present an evaluation of vision-language models' capacity for nonlocal visual reasoning: reasoning that requires chaining evidence collected from multiple, possibly distant regions of an image. We isolate three distinct forms of nonlocal vision: comparative perception, which demands holding two images in working memory and comparing them; saccadic search, which requires making discrete, evidence-driven jumps to locate successive targets; and smooth visual search, which involves following a continuous contour. Flagship models (e.g., GPT-5, Gemini 2.5 Pro, Claude Sonnet 4), even those that perform well on prior primitive-vision benchmarks, fail these tests and barely exceed random accuracy on two variants of our tasks that are trivial for humans. Our structured evaluation suite allows us to test whether VLMs can perform visual algorithms similar to those used by humans. Our findings show that despite gains in raw visual acuity, current models lack core visual reasoning capabilities.
♻ ☆ SACA: Selective Attention-Based Clustering Algorithm
Meysam Shirdel Bilehsavar, Razieh Ghaedi, Samira Seyed Taheri, Xinqi Fan, Christian O'Reilly
Clustering algorithms are fundamental tools across many fields, with density-based methods offering particular advantages in identifying arbitrarily shaped clusters and handling noise. However, their effectiveness is often limited by the requirement of critical parameter tuning by users, which typically requires significant domain expertise. This paper introduces a novel density-based clustering algorithm loosely inspired by the concept of selective attention, designed to minimize reliance on parameter tuning for most applications. The proposed method computes an adaptive threshold to exclude sparsely distributed points and outliers, constructs an initial cluster framework, and subsequently reintegrates the filtered points to refine the final results. Extensive experiments on diverse benchmark datasets demonstrate the robustness, accuracy, and ease of use of the proposed approach, establishing it as a powerful alternative to conventional density-based clustering techniques.
comment : 32 pages, 14 figures
♻ ☆ CLIP-Free, Label-Free, Zero-Shot Concept Bottleneck Models
Fawaz Sammani, Jonas Fischer, Nikos Deligiannis
Concept Bottleneck Models (CBMs) map dense, high-dimensional feature representations into a set of human-interpretable concepts which are then combined linearly to make a prediction. However, modern CBMs rely on the CLIP model to establish a mapping from dense feature representations to textual concepts, and it remains unclear how to design CBMs for models other than CLIP. Methods that do not use CLIP instead require manual, labor intensive annotation to associate feature representations with concepts. Furthermore, all CBMs necessitate training a linear classifier to map the extracted concepts to class labels. In this work, we lift all three limitations simultaneously by proposing a method that converts any frozen visual classifier into a CBM without requiring image-concept labels (label-free), without relying on the CLIP model (CLIP-free), and by deriving the linear classifier in a zero-shot manner. Our method is formulated by aligning the original classifier's distribution (over discrete class indices) with its corresponding vision-language counterpart distribution derived from textual class names, while preserving the classifier's performance. The approach requires no ground-truth image-class annotations, and is highly data-efficient and preserves the classifier's reasoning process. Applied and tested on over 40 visual classifiers, our resulting CLIP-free, zero-shot CBM sets a new state of the art, surpassing even supervised CLIP-based CBMs. Finally, we also show that our method can be used for zero-shot image captioning, outperforming existing methods based on CLIP, and achieving state of the art results.
♻ ☆ Zero-shot Denoising via Neural Compression: Theoretical and algorithmic framework
Ali Zafari, Xi Chen, Shirin Jalali
Zero-shot denoising aims to denoise observations without access to training samples or clean reference images. This setting is particularly relevant in practical imaging scenarios involving specialized domains such as medical imaging or biology. In this work, we propose the Zero-Shot Neural Compression Denoiser (ZS-NCD), a novel denoising framework based on neural compression. ZS-NCD treats a neural compression network as an untrained model, optimized directly on patches extracted from a single noisy image. The final reconstruction is then obtained by aggregating the outputs of the trained model over overlapping patches. Thanks to the built-in entropy constraints of compression architectures, our method naturally avoids overfitting and does not require manual regularization or early stopping. Through extensive experiments, we show that ZS-NCD achieves state-of-the-art performance among zero-shot denoisers for both Gaussian and Poisson noise, and generalizes well to both natural and non-natural images. Additionally, we provide new finite-sample theoretical results that characterize upper bounds on the achievable reconstruction error of general maximum-likelihood compression-based denoisers. These results further establish the theoretical foundations of compression-based denoising. Our code is available at: https://github.com/Computational-Imaging-RU/ZS-NCDenoiser.
♻ ☆ BrainPuzzle: Hybrid Physics and Data-Driven Reconstruction for Transcranial Ultrasound Tomography
Shengyu Chen, Shihang Feng, Yi Luo, Xiaowei Jia, Youzuo Lin
Ultrasound brain imaging remains challenging due to the large difference in sound speed between the skull and brain tissues and the difficulty of coupling large probes to the skull. This work aims to achieve quantitative transcranial ultrasound by reconstructing an accurate speed-of-sound (SoS) map of the brain. Traditional physics-based full-waveform inversion (FWI) is limited by weak signals caused by skull-induced attenuation, mode conversion, and phase aberration, as well as incomplete spatial coverage since full-aperture arrays are clinically impractical. In contrast, purely data-driven methods that learn directly from raw ultrasound data often fail to model the complex nonlinear and nonlocal wave propagation through bone, leading to anatomically plausible but quantitatively biased SoS maps under low signal-to-noise and sparse-aperture conditions. To address these issues, we propose BrainPuzzle, a hybrid two-stage framework that combines physical modeling with machine learning. In the first stage, reverse time migration (time-reversal acoustics) is applied to multi-angle acquisitions to produce migration fragments that preserve structural details even under low SNR. In the second stage, a transformer-based super-resolution encoder-decoder with a graph-based attention unit (GAU) fuses these fragments into a coherent and quantitatively accurate SoS image. A partial-array acquisition strategy using a movable low-count transducer set improves feasibility and coupling, while the hybrid algorithm compensates for the missing aperture. Experiments on two synthetic datasets show that BrainPuzzle achieves superior SoS reconstruction accuracy and image completeness, demonstrating its potential for advancing quantitative ultrasound brain imaging.
comment : 13 pages
♻ ☆ Explainable Deep Convolutional Multi-Type Anomaly Detection
Alex George, Lyudmila Mihaylova, Sean Anderson
Explainable anomaly detection methods often have the capability to identify and spatially localise anomalies within an image but lack the capability to differentiate the type of anomaly. Furthermore, they often require the costly training and maintenance of separate models for each object category. The lack of specificity is a significant research gap because identifying the type of anomaly (e.g., "Crack" vs. "Scratch") is crucial for accurate diagnosis that facilitates cost-saving operational decisions across diverse application domains. While some recent large-scale Vision-Language Models (VLMs) have begun to address this, they are computationally intensive and memory-heavy, restricting their use in real-time or embedded systems. We propose MultiTypeFCDD, a simple and lightweight convolutional framework designed as a practical alternative for explainable multi-type anomaly detection. MultiTypeFCDD uses only image-level labels to learn and produce multi-channel heatmaps, where each channel is trained to correspond to a specific anomaly type. The model functions as a single, unified framework capable of differentiating anomaly types across multiple object categories, eliminating the need to train and manage separate models for each object category. We evaluated our proposed method on the Real-IAD dataset and it delivers competitive results (96.4% I-AUROC) at just over 1% the size of state-of-the-art VLM models used for similar tasks. This makes it a highly practical and viable solution for real-world applications where computational resources are tightly constrained.
♻ ☆ HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision -Language Models
Erum Mushtaq, Zalan Fabian, Yavuz Faruk Bakman, Anil Ramakrishna, Mahdi Soltanolkotabi, Salman Avestimehr
Uncertainty Estimation (UE) plays a central role in quantifying the reliability of model outputs and reducing unsafe generations via selective prediction. In this regard, most existing probability-based UE approaches rely on predefined functions, aggregating token probabilities into a single UE score using heuristics such as length-normalization. However, these methods often fail to capture the complex relationships between generated tokens and struggle to identify biased probabilities often influenced by \textbf{language priors}. Another line of research uses hidden representations of the model and trains simple MLP architectures to predict uncertainty. However, such functions often lose the intricate \textbf{ inter-token dependencies}. While prior works show that hidden representations encode multimodal alignment signals, our work demonstrates that how these signals are processed has a significant impact on the UE performance. To effectively leverage these signals to identify inter-token dependencies, and vision-text alignment, we propose \textbf{HARMONY} (Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Models), a novel UE framework that integrates generated tokens ('text'), model's uncertainty score at the output ('MaxProb'), and its internal belief on the visual understanding of the image and the generated token (captured by 'hidden representations') at token level via appropriate input mapping design and suitable architecture choice. Our experimental experiments across two open-ended VQA benchmarks (A-OKVQA, and VizWiz) and four state-of-the-art VLMs (LLaVA-7B, LLaVA-13B, InstructBLIP, and Qwen-VL) show that HARMONY consistently matches or surpasses existing approaches, achieving up to 5\% improvement in AUROC and 9\% in PRR.
♻ ☆ SynPlay: Large-Scale Synthetic Human Data with Real-Wo rl d Diversity for Aerial-View Perception
Jinsub Yim, Hyungtae Lee, Sungmin Eum, Yi-Ting Shen, Yan Zhang, Heesung Kwon, Shuvra S. Bhattacharyya
We introduce SynPlay, a large-scale synthetic human dataset purpose-built for advancing multi-perspective human localization, with a predominant focus on aerial-view perception. SynPlay departs from traditional synthetic datasets by addressing a critical but underexplored challenge: localizing humans in aerial scenes where subjects often occupy only tens of pixels in the image. In such scenarios, fine-grained details like facial features or textures become irrelevant, shifting the burden of recognition to human motion, behavior, and interactions. To meet this need, SynPlay implements a novel rule-guided motion generation framework that combines real-world motion capture with motion evolution graphs. This design enables human actions to evolve dynamically through high-level game rules rather than predefined scripts, resulting in effectively uncountable motion variations. Unlike existing synthetic datasets-which either focus on static visual traits or reuse a limited set of mocap-driven actions-SynPlay captures a wide spectrum of spontaneous behaviors, including complex interactions that naturally emerge from unscripted gameplay scenarios. SynPlay also introduces an extensive multi-camera setup that spans UAVs at random altitudes, CCTVs, and a freely roaming UGV, achieving true near-to-far perspective coverage in a single dataset. The majority of instances are captured from aerial viewpoints at varying scales, directly supporting the development of models for long-range human analysis-a setting where existing datasets fall short. Our data contains over 73k images and 6.5M human instances, with detailed annotations for detection, segmentation, and keypoint tasks. Extensive experiments demonstrate that training with SynPlay significantly improves human localization performance, especially in few-shot and data-scarce scenarios.
comment : Project Page: https://synplaydataset.github.io/
♻ ☆ CraftSVG: Multi-Object Text-to-SVG Synthesis via Layout Guided Diffusion
Ayan Banerjee, Nityanand Mathur, Josep Llados, Umapada Pal, Anjan Dutta
Generating VectorArt from text prompts is a challenging vision task, requiring diverse yet realistic depictions of the seen as well as unseen entities. However, existing research has been mostly limited to the generation of single objects, rather than comprehensive scenes comprising multiple elements. In response, this work introduces SVGCraft, a novel end-to-end framework for the creation of vector graphics depicting entire scenes from textual descriptions. Utilizing a pre-trained LLM for layout generation from text prompts, this framework introduces a technique for producing masked latents in specified bounding boxes for accurate object placement. It introduces a fusion mechanism for integrating attention maps and employs a diffusion U-Net for coherent composition, speeding up the drawing process. The resulting SVG is optimized using a pre-trained encoder and LPIPS loss with opacity modulation to maximize similarity. Additionally, this work explores the potential of primitive shapes in facilitating canvas completion in constrained environments. Through both qualitative and quantitative assessments, SVGCraft is demonstrated to surpass prior works in abstraction, recognizability, and detail, as evidenced by its performance metrics (CLIP-T: 0.4563, Cosine Similarity: 0.6342, Confusion: 0.66, Aesthetic: 6.7832). The code will be available at https://github.com/ayanban011/SVGCraft.
Computation and Language 76
☆ ThetaEvolve: Test-time Learning on Open Problems
Yiping Wang, Shao-Rong Su, Zhiyuan Zeng, Eva Xu, Liliang Ren, Xinyu Yang, Zeyi Huang, Xuehai He, Luyao Ma, Baolin Peng, Hao Cheng, Pengcheng He, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, Yelong Shen
Recent advances in large language models (LLMs) have enabled breakthroughs in mathematical discovery, exemplified by AlphaEvolve, a closed-source system that evolves programs to improve bounds on open problems. However, it relies on ensembles of frontier LLMs to achieve new bounds and is a pure inference system that models cannot internalize the evolving strategies. We introduce ThetaEvolve, an open-source framework that simplifies and extends AlphaEvolve to efficiently scale both in-context learning and Reinforcement Learning (RL) at test time, allowing models to continually learn from their experiences in improving open optimization problems. ThetaEvolve features a single LLM, a large program database for enhanced exploration, batch sampling for higher throughput, lazy penalties to discourage stagnant outputs, and optional reward shaping for stable training signals, etc. ThetaEvolve is the first evolving framework that enable a small open-source model, like DeepSeek-R1-0528-Qwen3-8B, to achieve new best-known bounds on open problems (circle packing and first auto-correlation inequality) mentioned in AlphaEvolve. Besides, across two models and four open tasks, we find that ThetaEvolve with RL at test-time consistently outperforms inference-only baselines, and the model indeed learns evolving capabilities, as the RL-trained checkpoints demonstrate faster progress and better final performance on both trained target task and other unseen tasks. We release our code publicly: https://github.com/ypwang61/ThetaEvolve
comment : 30 pages, link: https://github.com/ypwang61/ThetaEvolve
☆ MegaChat: A Synthetic Persian Q&A Dataset for High-Quality Sales Chatbot Evaluation
Mahdi Rahmani, AmirHossein Saffari, Reyhane Rahmani
Small and medium-sized enterprises (SMEs) in Iran increasingly leverage Telegram for sales, where real-time engagement is essential for conversion. However, developing AI-driven chatbots for this purpose requires large, high-quality question-and-answer (Q&A) datasets, which are typically expensive and resource-intensive to produce, especially for low-resource languages like Persian. In this paper, we introduce MegaChat, the first fully synthetic Persian Q&A dataset designed to evaluate intelligent sales chatbots in Telegram-based e-commerce. We propose a novel, automated multi-agent architecture that generates persona-aware Q&A pairs by collecting data from active Telegram shopping channels. The system employs specialized agents for question generation, validation, and refinement, ensuring the production of realistic and diverse conversational data. To evaluate answer generation, we compare three classic retrieval-augmented generation (RAG) models with our advanced agentic system, which features multi-query retrieval, reranking, and persona-aligned response synthesis. Using GPT-5.1 for evaluation across six quality dimensions, our results show that the agentic architecture outperformed traditional RAG models in 4 out of 5 diverse channels, demonstrating its ability to generate scalable, high-quality datasets without relying on expensive human annotation or complex fine-tuning. MegaChat provides SMEs with an efficient, cost-effective solution for building intelligent customer engagement systems in specialized commercial domains, enabling advancements in multilingual conversational AI for low-resource languages. Download: https://github.com/MegaChat-Tech/MegaChat-DataSet
comment : 6 pages, 11 figures, 2 tables
☆ Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization EMNLP 2025
Jian Li, Shenglin Yin, Yujia Zhang, Alan Zhao, Xi Chen, Xiaohui Zhou, Pengfei Xu
Direct Preference Optimization (DPO) is a widely used reinforcement learning from human feedback (RLHF) method across various domains. Recent research has increasingly focused on the role of token importance in improving DPO effectiveness. It is observed that identical or semantically similar content (defined as ambiguous content) frequently appears within the preference pairs. We hypothesize that the presence of ambiguous content during DPO training may introduce ambiguity, thereby limiting further improvements in alignment. Through mathematical analysis and proof-of-concept experiments, we reveal that ambiguous content may potentially introduce ambiguities, thereby degrading performance. To address this issue, we introduce Ambiguity Awareness Optimization (AAO), a simple yet effective approach that automatically re-weights ambiguous content to reduce ambiguities by calculating semantic similarity from preference pairs. Through extensive experiments, we demonstrate that AAO consistently and significantly surpasses state-of-the-art approaches in performance, without markedly increasing response length, across multiple model scales and widely adopted benchmark datasets, including AlpacaEval 2, MT-Bench, and Arena-Hard. Specifically, AAO outperforms DPO by up to 8.9 points on AlpacaEval 2 and achieves an improvement of by up to 15.0 points on Arena-Hard.
comment : Accepted at EMNLP 2025 main
☆ Is Passive Expertise-Based Personalization Enough? A Case Study in AI-Assisted Test-Taking EMNLP 2025
Li Siyan, Jason Zhang, Akash Maharaj, Yuanming Shi, Yunyao Li
Novice and expert users have different systematic preferences in task-oriented dialogues. However, whether catering to these preferences actually improves user experience and task performance remains understudied. To investigate the effects of expertise-based personalization, we first built a version of an enterprise AI assistant with passive personalization. We then conducted a user study where participants completed timed exams, aided by the two versions of the AI assistant. Preliminary results indicate that passive personalization helps reduce task load and improve assistant perception, but reveal task-specific limitations that can be addressed through providing more user agency. These findings underscore the importance of combining active and passive personalization to optimize user experience and effectiveness in enterprise task-oriented environments.
comment : Accepted into Tailoring AI: Exploring Active and Passive LLM Personalization (PALS) workshop at EMNLP 2025
☆ Optimizing Multimodal Language Models through Attention-based Interpretability
Alexander Sergeev, Evgeny Kotelnikov
Modern large language models become multimodal, analyzing various data formats like text and images. While fine-tuning is effective for adapting these multimodal language models (MLMs) to downstream tasks, full fine-tuning is computationally expensive. Parameter-Efficient Fine-Tuning (PEFT) methods address this by training only a small portion of model weights. However, MLMs are difficult to interpret, making it challenging to identify which components are most effective for training to balance efficiency and performance. We propose an attention-based interpretability method for MLMs by analyzing attention scores relative to image tokens. The core idea is to identify attention heads that focus on image key objects. We utilize this information to select optimal model components for PEFT in multimodal models. Our contributions include a method for identifying attention heads associated with image key objects, its application to PEFT for image captioning, and the creation of a new dataset containing images, key object masks, and their textual descriptions. We conducted experiments on MLMs with 2-3 billion parameters to validate the method's effectiveness. By calculating Head Impact (HI) scores we quantify an attention head's focus on key objects, indicating its significance in image understanding. Our fine-tuning experiments demonstrate that adapting layers with the highest HI scores leads to the most significant shifts in metrics compared to pre-trained, randomly selected, or lowest-HI-score layers. This indicates that fine-tuning a small percentage (around 0.01%) of parameters in these crucial layers can substantially influence image understanding capabilities.
comment : Accepted for ICAI-2025 conference
☆ Scaling HuBERT for African Languages: From Base to Large and XL
Antoine Caubrière, Elodie Gauthier
Despite recent progress in multilingual speech processing, African languages remain under-represented in both research and deployed systems, particularly when it comes to strong, open-weight encoders that transfer well under low-resource supervision. Self-supervised learning has proven especially promising in such settings, yet most publicly released models targeting African speech remain at BASE scale, leaving unanswered whether larger encoders, trained exclusively on Africa-centric audio, offer tangible benefits and how model capacity interacts with data composition. This work addresses that gap by introducing SSA-HuBERT-Large (317M parameters) and SSA-HuBERT-XL (964M parameters), the first large models trained solely on African speech, alongside a BASE size counterpart. We release these models as open weights: see https://huggingface.co/collections/Orange/african-speech-foundation-models. By conducting a carefully controlled experimental study focused exclusively on Sub-Saharan languages, covering automatic speech recognition (ASR) and language identification (LID) tasks, we demonstrate that larger architectures significantly improve performance by effectively leveraging large audio datasets.
comment : Journée d'études AFIA-ATALA 2025 : Technologies linguistiques pour les langues peu dotées
☆ Towards Improving Interpretability of Language Model Generation through a Structured Knowledge Discovery Approach
Shuqi Liu, Han Wu, Guanzhi Deng, Jianshu Chen, Xiaoyang Wang, Linqi Song
Knowledge-enhanced text generation aims to enhance the quality of generated text by utilizing internal or external knowledge sources. While language models have demonstrated impressive capabilities in generating coherent and fluent text, the lack of interpretability presents a substantial obstacle. The limited interpretability of generated text significantly impacts its practical usability, particularly in knowledge-enhanced text generation tasks that necessitate reliability and explainability. Existing methods often employ domain-specific knowledge retrievers that are tailored to specific data characteristics, limiting their generalizability to diverse data types and tasks. To overcome this limitation, we directly leverage the two-tier architecture of structured knowledge, consisting of high-level entities and low-level knowledge triples, to design our task-agnostic structured knowledge hunter. Specifically, we employ a local-global interaction scheme for structured knowledge representation learning and a hierarchical transformer-based pointer network as the backbone for selecting relevant knowledge triples and entities. By combining the strong generative ability of language models with the high faithfulness of the knowledge hunter, our model achieves high interpretability, enabling users to comprehend the model output generation process. Furthermore, we empirically demonstrate the effectiveness of our model in both internal knowledge-enhanced table-to-text generation on the RotoWireFG dataset and external knowledge-enhanced dialogue response generation on the KdConv dataset. Our task-agnostic model outperforms state-of-the-art methods and corresponding language models, setting new standards on the benchmark.
☆ Tackling a Challenging Corpus for Ea rl y Detection of Gambling Disorder: UNSL at MentalRiskES 2025
Horacio Thompson, Marcelo Errecalde
Gambling disorder is a complex behavioral addiction that is challenging to understand and address, with severe physical, psychological, and social consequences. Early Risk Detection (ERD) on the Web has become a key task in the scientific community for identifying early signs of mental health behaviors based on social media activity. This work presents our participation in the MentalRiskES 2025 challenge, specifically in Task 1, aimed at classifying users at high or low risk of developing a gambling-related disorder. We proposed three methods based on a CPI+DMC approach, addressing predictive effectiveness and decision-making speed as independent objectives. The components were implemented using the SS3, BERT with extended vocabulary, and SBERT models, followed by decision policies based on historical user analysis. Although it was a challenging corpus, two of our proposals achieved the top two positions in the official results, performing notably in decision metrics. Further analysis revealed some difficulty in distinguishing between users at high and low risk, reinforcing the need to explore strategies to improve data interpretation and quality, and to promote more transparent and reliable ERD systems for mental disorders.
comment : In Iberian Language Evaluation Forum (IberLEF 2025), Zaragoza, Spain
☆ Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
Xiang Hu, Zhanchao Zhou, Ruiqi Liang, Zehuan Li, Wei Wu, Jianguo Li
This work explores the challenge of building ``Machines that Can Remember'', framing long-term memory as the problem of efficient ultra-long context modeling. We argue that this requires three key properties: \textbf{sparsity}, \textbf{random-access flexibility}, and \textbf{length generalization}. To address ultra-long-context modeling, we leverage Hierarchical Sparse Attention (HSA), a novel attention mechanism that satisfies all three properties. We integrate HSA into Transformers to build HSA-UltraLong, which is an 8B-parameter MoE model trained on over 8 trillion tokens and is rigorously evaluated on different tasks with in-domain and out-of-domain context lengths to demonstrate its capability in handling ultra-long contexts. Results show that our model performs comparably to full-attention baselines on in-domain lengths while achieving over 90\% accuracy on most in-context retrieval tasks with contexts up to 16M. This report outlines our experimental insights and open problems, contributing a foundation for future research in ultra-long context modeling.
☆ Toward Automatic Safe Driving Instruction: A Large-Scale Vision Language Model Approach
Haruki Sakajo, Hiroshi Takato, Hiroshi Tsutsui, Komei Soda, Hidetaka Kamigaito, Taro Watanabe
Large-scale Vision Language Models (LVLMs) exhibit advanced capabilities in tasks that require visual information, including object detection. These capabilities have promising applications in various industrial domains, such as autonomous driving. For example, LVLMs can generate safety-oriented descriptions of videos captured by road-facing cameras. However, ensuring comprehensive safety requires monitoring driver-facing views as well to detect risky events, such as the use of mobiles while driving. Thus, the ability to process synchronized inputs is necessary from both driver-facing and road-facing cameras. In this study, we develop models and investigate the capabilities of LVLMs by constructing a dataset and evaluating their performance on this dataset. Our experimental results demonstrate that while pre-trained LVLMs have limited effectiveness, fine-tuned LVLMs can generate accurate and safety-aware driving instructions. Nonetheless, several challenges remain, particularly in detecting subtle or complex events in the video. Our findings and error analysis provide valuable insights that can contribute to the improvement of LVLM-based systems in this domain.
comment : Accepted to MMLoSo 2025
☆ Transformer -Driven Triple Fusion Framework for Enhanced Multimodal Author Intent Classification in Low-Resource Bangla
Ariful Islam, Tanvir Mahmud, Md Rifat Hossen
The expansion of the Internet and social networks has led to an explosion of user-generated content. Author intent understanding plays a crucial role in interpreting social media content. This paper addresses author intent classification in Bangla social media posts by leveraging both textual and visual data. Recognizing limitations in previous unimodal approaches, we systematically benchmark transformer-based language models (mBERT, DistilBERT, XLM-RoBERTa) and vision architectures (ViT, Swin, SwiftFormer, ResNet, DenseNet, MobileNet), utilizing the Uddessho dataset of 3,048 posts spanning six practical intent categories. We introduce a novel intermediate fusion strategy that significantly outperforms early and late fusion on this task. Experimental results show that intermediate fusion, particularly with mBERT and Swin Transformer, achieves 84.11% macro-F1 score, establishing a new state-of-the-art with an 8.4 percentage-point improvement over prior Bangla multimodal approaches. Our analysis demonstrates that integrating visual context substantially enhances intent classification. Cross-modal feature integration at intermediate levels provides optimal balance between modality-specific representation and cross-modal learning. This research establishes new benchmarks and methodological standards for Bangla and other low-resource languages. We call our proposed framework BangACMM (Bangla Author Content MultiModal).
comment : Accepted at the 28th International Conference on Computer and Information Technology (ICCIT 2025). To be published in IEEE proceedings
☆ MCP vs RAG vs NLWeb vs HTML: A Comparison of the Effectiveness and Efficiency of Different Agent Interfaces to the Web (Technical Report)
Aaron Steiner, Ralph Peeters, Christian Bizer
Large language model agents are increasingly used to automate web tasks such as product search, offer comparison, and checkout. Current research explores different interfaces through which these agents interact with websites, including traditional HTML browsing, retrieval-augmented generation (RAG) over pre-crawled content, communication via Web APIs using the Model Context Protocol (MCP), and natural-language querying through the NLWeb interface. However, no prior work has compared these four architectures within a single controlled environment using identical tasks. To address this gap, we introduce a testbed consisting of four simulated e-shops, each offering its products via HTML, MCP, and NLWeb interfaces. For each interface (HTML, RAG, MCP, and NLWeb) we develop specialized agents that perform the same sets of tasks, ranging from simple product searches and price comparisons to complex queries for complementary or substitute products and checkout processes. We evaluate the agents using GPT 4.1, GPT 5, GPT 5 mini, and Claude Sonnet 4 as underlying LLM. Our evaluation shows that the RAG, MCP and NLWeb agents outperform HTML on both effectiveness and efficiency. Averaged over all tasks, F1 rises from 0.67 for HTML to between 0.75 and 0.77 for the other agents. Token usage falls from about 241k for HTML to between 47k and 140k per task. The runtime per task drops from 291 seconds to between 50 and 62 seconds. The best overall configuration is RAG with GPT 5 achieving an F1 score of 0.87 and a completion rate of 0.79. Also taking cost into consideration, RAG with GPT 5 mini offers a good compromise between API usage fees and performance. Our experiments show the choice of the interaction interface has a substantial impact on both the effectiveness and efficiency of LLM-based web agents.
☆ Behavior-Equivalent Token: Single-Token Replacement for Long Prompt s in LLMs
Jiancheng Dong, Pengyue Jia, Jingyu Peng, Maolin Wang, Yuhao Wang, Lixin Su, Xin Sun, Shuaiqiang Wang, Dawei Yin, Xiangyu Zhao
Carefully engineered system prompts play a critical role in guiding the behavior of LLM agents, but their considerable length introduces significant drawbacks, including increased inference latency, higher computational cost, and reduced effective context length. This raises the question of whether such lengthy prompts can be replaced by a drastically reduced number of tokens while preserving their behavioral effect on downstream tasks. To enable this, we propose a lightweight three-stage training framework that learns a single prompt-specific Behavior-Equivalent token ([BE]). The framework first trains [BE] to encode the natural-language content of the original system prompt via reconstruction, and then distills the prompt 's downstream behavior into this single token. Importantly, our method requires no access to model internals, no auxiliary compression models, and no labeled responses. Empirical evaluations on three datasets show that a single [BE] token achieves up to a 3000x reduction in prompt length, while retaining about 98% of the downstream performance of the original system prompts. This substantially reduces inference cost and leaves almost the entire context window available for user inputs.
comment : 15 pages, 5 figures
☆ BanglaSentNet: An Explainable Hybrid Deep Learning Framework for Multi-Aspect Sentiment Analysis with Cross-Domain Transfer Learning
Ariful Islam, Md Rifat Hossen, Tanvir Mahmud
Multi-aspect sentiment analysis of Bangla e-commerce reviews remains challenging due to limited annotated datasets, morphological complexity, code-mixing phenomena, and domain shift issues, affecting 300 million Bangla-speaking users. Existing approaches lack explainability and cross-domain generalization capabilities crucial for practical deployment. We present BanglaSentNet, an explainable hybrid deep learning framework integrating LSTM, BiLSTM, GRU, and BanglaBERT through dynamic weighted ensemble learning for multi-aspect sentiment classification. We introduce a dataset of 8,755 manually annotated Bangla product reviews across four aspects (Quality, Service, Price, Decoration) from major Bangladeshi e-commerce platforms. Our framework incorporates SHAP-based feature attribution and attention visualization for transparent insights. BanglaSentNet achieves 85% accuracy and 0.88 F1-score, outperforming standalone deep learning models by 3-7% and traditional approaches substantially. The explainability suite achieves 9.4/10 interpretability score with 87.6% human agreement. Cross-domain transfer learning experiments reveal robust generalization: zero-shot performance retains 67-76% effectiveness across diverse domains (BanglaBook reviews, social media, general e-commerce, news headlines); few-shot learning with 500-1000 samples achieves 90-95% of full fine-tuning performance, significantly reducing annotation costs. Real-world deployment demonstrates practical utility for Bangladeshi e-commerce platforms, enabling data-driven decision-making for pricing optimization, service improvement, and customer experience enhancement. This research establishes a new state-of-the-art benchmark for Bangla sentiment analysis, advances ensemble learning methodologies for low-resource languages, and provides actionable solutions for commercial applications.
comment : Submitted to Springer Nature Computer Science (SNCS) as an extended version of our ICDSAIA 2025 conference paper
☆ Tourism Question Answer System in Indian Language using Domain-Adapted Foundation Models
Praveen Gatla, Anushka, Nikita Kanwar, Gouri Sahoo, Rajesh Kumar Mundotiya
This article presents the first comprehensive study on designing a baseline extractive question-answering (QA) system for the Hindi tourism domain, with a specialized focus on the Varanasi-a cultural and spiritual hub renowned for its Bhakti-Bhaav (devotional ethos). Targeting ten tourism-centric subdomains-Ganga Aarti, Cruise, Food Court, Public Toilet, Kund, Museum, General, Ashram, Temple and Travel, the work addresses the absence of language-specific QA resources in Hindi for culturally nuanced applications. In this paper, a dataset comprising 7,715 Hindi QA pairs pertaining to Varanasi tourism was constructed and subsequently augmented with 27,455 pairs generated via Llama zero-shot prompting. We propose a framework leveraging foundation models-BERT and RoBERTa, fine-tuned using Supervised Fine-Tuning (SFT) and Low-Rank Adaptation (LoRA), to optimize parameter efficiency and task performance. Multiple variants of BERT, including pre-trained languages (e.g., Hindi-BERT), are evaluated to assess their suitability for low-resource domain-specific QA. Evaluation metrics - F1, BLEU, and ROUGE-L - highlight trade-offs between answer precision and linguistic fluency. Experiments demonstrate that LoRA-based fine-tuning achieves competitive performance (85.3\% F1) while reducing trainable parameters by 98\% compared to SFT, striking a balance between efficiency and accuracy. Comparative analysis across models reveals that RoBERTa with SFT outperforms BERT variants in capturing contextual nuances, particularly for culturally embedded terms (e.g., Aarti, Kund). This work establishes a foundational baseline for Hindi tourism QA systems, emphasizing the role of LORA in low-resource settings and underscoring the need for culturally contextualized NLP frameworks in the tourism domain.
☆ TWEO: Transformer s Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
Guang Liang, Jie Shao, Ningyuan Tang, Xinyao Liu, Jianxin Wu
Native FP8 support in modern hardware is essential for training large Transformers, but is severely hindered by extreme activation outliers. Existing solutions either rely on complex mixed-precision engineering or invasive architectural modifications. This paper fundamentally challenges the conventional wisdom that outliers are data-driven. We demonstrate that extreme outliers are a data-independent, mechanically-produced artifact of training, originating from specific structural properties of the weight matrices (i.e., colinearity). Based on this insight, we propose TWEO (Transformers Without Extreme Outliers), a novel, non-invasive loss function. TWEO effectively prevents extreme outliers via a very simple loss term, which reduces outliers from 10000+ to less than 20. TWEO then enables full-model FP8 pre-training with neither engineering tricks nor architectural changes for both LLM and ViT. When standard FP8 training catastrophically collapses, TWEO achieves performance comparable to the BF16 baseline while delivering a 36% increase in training throughput. Also, TWEO enables a new quantization paradigm. Hardware-friendly W8A8 per-tensor static quantization of LLMs, previously considered completely unusable due to outliers, achieves SOTA performance for the first time on TWEO-trained models.
☆ Listwise Preference Optimization with Element-wise Confusions for Aspect Sentiment Quad Prediction
Wenna Lai, Haoran Xie, Guandong Xu, Qing Li, S. Joe Qin
Aspect sentiment quad prediction (ASQP) is inherently challenging to predict a structured quadruple with four core sentiment elements, including aspect term (a), aspect category (c), opinion term (o), and sentiment polarity (s). Prior methods relying on marker-based prediction struggle with modeling the intricate relationships among elements and experience sharp performance declines when predicting higher-order elements (e.g., c and s) under standard supervised fine-tuning. To address these limitations, we employ reasoning-based generation to output both the quadruple and a natural language rationale under element prefixes within a unified template, encouraging explicit relational reasoning and interpretability. To further enhance element-wise alignment, we introduce a listwise preference optimization framework for improving structural validity and relational coherence. Specifically, we generate element-wise confusable candidates via syntactic and semantic proximity, then train the model with listwise objectives to prefer the gold candidates over closely competing alternatives. Extensive experiments on four benchmark datasets demonstrate that our framework effectively improves quadruple prediction accuracy and explanation consistency.
comment : 11 pages, 7 figures, and 6 tables
☆ Are LLMs Good Safety Agent s or a Propaganda Engine?
Neemesh Yadav, Francesco Ortu, Jiarui Liu, Joeun Yook, Bernhard Schölkopf, Rada Mihalcea, Alberto Cazzaniga, Zhijing Jin
Large Language Models (LLMs) are trained to refuse to respond to harmful content. However, systematic analyses of whether this behavior is truly a reflection of its safety policies or an indication of political censorship, that is practiced globally by countries, is lacking. Differentiating between safety influenced refusals or politically motivated censorship is hard and unclear. For this purpose we introduce PSP, a dataset built specifically to probe the refusal behaviors in LLMs from an explicitly political context. PSP is built by formatting existing censored content from two data sources, openly available on the internet: sensitive prompts in China generalized to multiple countries, and tweets that have been censored in various countries. We study: 1) impact of political sensitivity in seven LLMs through data-driven (making PSP implicit) and representation-level approaches (erasing the concept of politics); and, 2) vulnerability of models on PSP through prompt injection attacks (PIAs). Associating censorship with refusals on content with masked implicit intent, we find that most LLMs perform some form of censorship. We conclude with summarizing major attributes that can cause a shift in refusal distributions across models and contexts of different countries.
comment : 15 pages, 7 tables, 4 figures
☆ Multi-chain Graph Refinement and Selection for Reliable Reasoning in Large Language Models
Yujiao Yang, Jing Lian, Linhui Li
The complex reasoning ability of Large Language Models (LLMs) poses a critical bottleneck for their practical applications. Test-time expansion methods such as Tree-of-Thought (ToT) and Graph-of-Thought (GoT) enhance reasoning by introducing intermediate reasoning structures, tree search, or graph-based exploration mechanisms. However, their reasoning strategies suffer from limited diversity, redundant search branches, and inadequate integration and error correction across heterogeneous reasoning paths. To address these limitations, we propose a novel reasoning framework called Multi-chain Graph Refinement & Selection (MGRS), which first generates multiple diverse reasoning trajectories for a given problem, refines candidate responses using a composite self- and cross-verification strategy, then constructs a reasoning relation graph and estimates the success rate of intermediate nodes, and finally computes cumulative success rates to select the most reliable answer and corresponding reasoning trajectory. Experimental results demonstrate that MGRS significantly advances both the reasoning capability and computational efficiency of reasoning enhancement methods. Across six benchmark datasets spanning four distinct tasks, MGRS achieves an average accuracy of 82.9%, outperforming state-of-the-art baselines by a clear margin of 2.1%. Remarkably, on the 24-point game, MGRS attains 100% accuracy for the first time, while delivering a 13.6x speed-up compared to the leading Forest of Thoughts framework.
☆ Dripper: Token-Efficient Main HTML Extraction with a Lightweight LM
Mengjie Liu, Jiahui Peng, Pei Chu, Jiantao Qiu, Ren Ma, He Zhu, Rui Min, Lindong Lu, Wenchang Ning, Linfeng Hou, Kaiwen Liu, Yuan Qu, Zhenxiang Li, Chao Xu, Zhongying Tu, Wentao Zhang, Conghui He
Accurately and efficiently extracting main content from general web pages is of great significance for obtaining training data for large models. Using well-pre-trained decoder-only generative language models offers excellent document comprehension capabilities, thereby effectively enhancing parsing quality. However, it remains constrained by issues such as context window length, inference cost, and format hallucination. We present Dripper, an efficient HTML main content extraction framework powered by lightweight language models, which addresses these challenges through four key innovations: (1) We design a specialized HTML simplification algorithm that reduces input token count to 22\% compared to raw HTML while preserving critical structural information; (2) We reformulate main content extraction as a semantic block sequence classification task, significantly reducing inference cost; (3) We introduce a controlled decoding mechanism that strictly constrains the output space through logits processors, effectively eliminating hallucination issues common in small-scale models; (4) We propose WebMainBench, an evaluation dataset containing over 7,800 web pages with meticulously human-annotated main content extraction labels. Experimental results demonstrate that using only a 0.6B parameter model, Dripper achieves state-of-the-art performance across all evaluation benchmarks and outperforms all baseline methods, attaining an ROUGE-N F1 score of 81.58\%( 83.13\% with fall-back strategy) on our proposed WebMainBench dataset.
☆ Mind Reading or Misreading? LLMs on the Big Five Personality Test SC
Francesco Di Cursi, Chiara Boldrini, Marco Conti, Andrea Passarella
We evaluate large language models (LLMs) for automatic personality prediction from text under the binary Five Factor Model (BIG5). Five models -- including GPT-4 and lightweight open-source alternatives -- are tested across three heterogeneous datasets (Essays, MyPersonality, Pandora) and two prompting strategies (minimal vs. enriched with linguistic and psychological cues). Enriched prompts reduce invalid outputs and improve class balance, but also introduce a systematic bias toward predicting trait presence. Performance varies substantially: Openness and Agreeableness are relatively easier to detect, while Extraversion and Neuroticism remain challenging. Although open-source models sometimes approach GPT-4 and prior benchmarks, no configuration yields consistently reliable predictions in zero-shot binary settings. Moreover, aggregate metrics such as accuracy and macro-F1 mask significant asymmetries, with per-class recall offering clearer diagnostic value. These findings show that current out-of-the-box LLMs are not yet suitable for APPT, and that careful coordination of prompt design, trait framing, and evaluation metrics is essential for interpretable results.
comment : Funding: SoBigDatait (IR0000013), FAIR (PE00000013), ICSC (CN00000013)
☆ Accent Placement Models for Rigvedic Sanskrit Text AACL
Akhil Rajeev P, Annarao Kulkarni
The Rigveda, among the oldest Indian texts in Vedic Sanskrit, employs a distinctive pitch-accent system : udātta, anudātta, svarita whose marks encode melodic and interpretive cues but are often absent from modern e-texts. This work develops a parallel corpus of accented-unaccented ślokas and conducts a controlled comparison of three strategies for automatic accent placement in Rigvedic verse: (i) full fine-tuning of ByT5, a byte-level Transformer that operates directly on Unicode combining marks, (ii) a from-scratch BiLSTM-CRF sequence-labeling baseline, and (iii) LoRA-based parameter-efficient fine-tuning atop ByT5. Evaluation uses Word Error Rate (WER) and Character Error Rate (CER) for orthographic fidelity, plus a task-specific Diacritic Error Rate (DER) that isolates accent edits. Full ByT5 fine-tuning attains the lowest error across all metrics; LoRA offers strong efficiency-accuracy trade-offs, and BiLSTM-CRF serves as a transparent baseline. The study underscores practical requirements for accent restoration - Unicode-safe preprocessing, mark-aware tokenization, and evaluation that separates grapheme from accent errors - and positions heritage-language technology as an emerging NLP area connecting computational modeling with philological and pedagogical aims. Results establish reproducible baselines for Rigvedic accent restoration and provide guidance for downstream tasks such as accent-aware OCR, ASR/chant synthesis, and digital scholarship.
comment : Submitted to AACL-IJCNLP 2025
☆ Bharat Scene Text: A Novel Comprehensive Dataset and Benchmark for Indian Language Scene Text Understanding
Anik De, Abhirama Subramanyam Penamakuri, Rajeev Yadav, Aditya Rathore, Harshiv Shah, Devesh Sharma, Sagar Agarwal, Pravin Kumar, Anand Mishra
Reading scene text, that is, text appearing in images, has numerous application areas, including assistive technology, search, and e-commerce. Although scene text recognition in English has advanced significantly and is often considered nearly a solved problem, Indian language scene text recognition remains an open challenge. This is due to script diversity, non-standard fonts, and varying writing styles, and, more importantly, the lack of high-quality datasets and open-source models. To address these gaps, we introduce the Bharat Scene Text Dataset (BSTD) - a large-scale and comprehensive benchmark for studying Indian Language Scene Text Recognition. It comprises more than 100K words that span 11 Indian languages and English, sourced from over 6,500 scene images captured across various linguistic regions of India. The dataset is meticulously annotated and supports multiple scene text tasks, including: (i) Scene Text Detection, (ii) Script Identification, (iii) Cropped Word Recognition, and (iv) End-to-End Scene Text Recognition. We evaluated state-of-the-art models originally developed for English by adapting (fine-tuning) them for Indian languages. Our results highlight the challenges and opportunities in Indian language scene text recognition. We believe that this dataset represents a significant step toward advancing research in this domain. All our models and data are open source.
comment : Under Peer Review
☆ Conveying Imagistic Thinking in TCM Translation: A Prompt Engineering and LLM-Based Evaluation Framework
Jiatong Han
Traditional Chinese Medicine theory is built on imagistic thinking, in which medical principles and diagnostic and therapeutic logic are structured through metaphor and metonymy. However, existing English translations largely rely on literal rendering, making it difficult for target-language readers to reconstruct the underlying conceptual networks and apply them in clinical practice. This study adopted a human-in-the-loop framework and selected four passages from the medical canon Huangdi Neijing that are fundamental in theory. Through prompt-based cognitive scaffolding, DeepSeek V3.1 was guided to identify metaphor and metonymy in the source text and convey the theory in translation. In the evaluation stage, ChatGPT 5 Pro and Gemini 2.5 Pro were instructed by prompts to simulate three types of real-world readers. Human translations, baseline model translations, and prompt-adjusted translations were scored by the simulated readers across five cognitive dimensions, followed by structured interviews and Interpretative Phenomenological Analysis. Results show that the prompt-adjusted LLM translations perform best across all five dimensions, with high cross-model and cross-role consistency. The interview themes reveal differences between human and machine translation, effective strategies for metaphor and metonymy transfer, and readers' cognitive preferences. This study provides a cognitive, efficient and replicable HITL methodological pathway for translation of ancient, concept-dense texts like TCM.
comment : 3 figures
☆ Standard Occupation Classifier -- A Natural Language Processing Approach
Sidharth Rony, Jack Patman
Standard Occupational Classifiers (SOC) are systems used to categorize and classify different types of jobs and occupations based on their similarities in terms of job duties, skills, and qualifications. Integrating these facets with Big Data from job advertisement offers the prospect to investigate labour demand that is specific to various occupations. This project investigates the use of recent developments in natural language processing to construct a classifier capable of assigning an occupation code to a given job advertisement. We develop various classifiers for both UK ONS SOC and US O*NET SOC, using different Language Models. We find that an ensemble model, which combines Google BERT and a Neural Network classifier while considering job title, description, and skills, achieved the highest prediction accuracy. Specifically, the ensemble model exhibited a classification accuracy of up to 61% for the lower (or fourth) tier of SOC, and 72% for the third tier of SOC. This model could provide up to date, accurate information on the evolution of the labour market using job advertisements.
☆ Decoding the Past: Explainable Machine Learning Models for Dating Historical Texts
Paulo J. N. Pinto, Armando J. Pinho, Diogo Pratas
Accurately dating historical texts is essential for organizing and interpreting cultural heritage collections. This article addresses temporal text classification using interpretable, feature-engineered tree-based machine learning models. We integrate five feature categories - compression-based, lexical structure, readability, neologism detection, and distance features - to predict the temporal origin of English texts spanning five centuries. Comparative analysis shows that these feature domains provide complementary temporal signals, with combined models outperforming any individual feature set. On a large-scale corpus, we achieve 76.7% accuracy for century-scale prediction and 26.1% for decade-scale classification, substantially above random baselines (20% and 2.3%). Under relaxed temporal precision, performance increases to 96.0% top-2 accuracy for centuries and 85.8% top-10 accuracy for decades. The final model exhibits strong ranking capabilities with AUCROC up to 94.8% and AUPRC up to 83.3%, and maintains controlled errors with mean absolute deviations of 27 years and 30 years, respectively. For authentication-style tasks, binary models around key thresholds (e.g., 1850-1900) reach 85-98% accuracy. Feature importance analysis identifies distance features and lexical structure as most informative, with compression-based features providing complementary signals. SHAP explainability reveals systematic linguistic evolution patterns, with the 19th century emerging as a pivot point across feature domains. Cross-dataset evaluation on Project Gutenberg highlights domain adaptation challenges, with accuracy dropping by 26.4 percentage points, yet the computational efficiency and interpretability of tree-based models still offer a scalable, explainable alternative to neural architectures.
☆ Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responses
Dong Nguyen, Laura Rosseel
Spelling variation (e.g. funnnn vs. fun) can influence the social perception of texts and their writers: we often have various associations with different forms of writing (is the text informal? does the writer seem young?). In this study, we focus on the social perception of spelling variation in online writing in English and study to what extent this perception is aligned between humans and large language models (LLMs). Building on sociolinguistic methodology, we compare LLM and human ratings on three key social attributes of spelling variation (formality, carefulness, age). We find generally strong correlations in the ratings between humans and LLMs. However, notable differences emerge when we analyze the distribution of ratings and when comparing between different types of spelling variation.
☆ ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?
Huaixiao Tou, Ying Zeng, Cong Ma, Muzhi Li, Minghao Li, Weijie Yuan, He Zhang, Kai Jia
We present ShoppingComp, a challenging real-world benchmark for rigorously evaluating LLM-powered shopping agents on three core capabilities: precise product retrieval, expert-level report generation, and safety critical decision making. Unlike prior e-commerce benchmarks, ShoppingComp introduces highly complex tasks under the principle of guaranteeing real products and ensuring easy verifiability, adding a novel evaluation dimension for identifying product safety hazards alongside recommendation accuracy and report quality. The benchmark comprises 120 tasks and 1,026 scenarios, curated by 35 experts to reflect authentic shopping needs. Results reveal stark limitations of current LLMs: even state-of-the-art models achieve low performance (e.g., 11.22% for GPT-5, 3.92% for Gemini-2.5-Flash). These findings highlight a substantial gap between research benchmarks and real-world deployment, where LLMs make critical errors such as failure to identify unsafe product usage or falling for promotional misinformation, leading to harmful recommendations. ShoppingComp fills the gap and thus establishes a new standard for advancing reliable and practical agents in e-commerce.
☆ Pooling Attention: Evaluating Pretrain ed Transformer Embeddings for Deception Classification
Sumit Mamtani, Abhijeet Bhure
This paper investigates fake news detection as a downstream evaluation of Transformer representations, benchmarking encoder-only and decoder-only pre-trained models (BERT, GPT-2, Transformer-XL) as frozen embedders paired with lightweight classifiers. Through controlled preprocessing comparing pooling versus padding and neural versus linear heads, results demonstrate that contextual self-attention encodings consistently transfer effectively. BERT embeddings combined with logistic regression outperform neural baselines on LIAR dataset splits, while analyses of sequence length and aggregation reveal robustness to truncation and advantages from simple max or average pooling. This work positions attention-based token encoders as robust, architecture-centric foundations for veracity tasks, isolating Transformer contributions from classifier complexity.
comment : Accepted at the IEEE 7th Computing, Communications and IoT Applications Conference (ComComAp 2025), Madrid, Spain, December 2025. 6 pages
☆ Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match
Jinze Li, Yixing Xu, Guanchen Li, Shuo Yang, Jinfeng Xu, Xuanwu Yin, Dong Li, Edith C. H. Ngai, Emad Barsoum
Large language models (LLMs) achieve strong performance across diverse tasks but suffer from high inference latency due to their autoregressive generation. Speculative Decoding (SPD) mitigates this issue by verifying candidate tokens in parallel from a smaller draft model, yet its strict exact-match verification discards many semantically valid continuations. Moreover, existing training-based SPD methods often suffer from performance degradation on out-of-distribution (OOD) tasks. To this end, we propose Training-Free Loosely Speculative Decoding (FLy), a novel method that loosens the rigid verification criterion by leveraging the target model's self-corrective behavior to judge whether a draft-target mismatch remains semantically valid. FLy introduces a two-tier mechanism: an entropy-level gate that identifies whether the current token allows multiple plausible alternatives or is nearly deterministic, and a token-level deferred window that distinguishes genuine errors from differently worded yet semantically correct variants. To further reduce latency, we design a multi-level acceleration strategy that accelerates not only the target model but also the drafter itself. Owing to its training-free design, FLy composes seamlessly with arbitrary draft-target pairs and generalizes across models and domains without hyperparameter re-tuning. Experiments show that FLy preserves more than 99% of the target model's accuracy while achieving an average 2.81x speedup on Llama-3.1-70B-Instruct and 5.07x speedup on the 405B variant. Notably, on out-of-domain datasets, our method remains highly effective and outperforms the training-based method EAGLE-3 by 1.62x.
comment : Under review
☆ Visual Puns from Idioms: An Iterative LLM-T2IM-MLLM Framework ICASSP 2026
Kelaiti Xiao, Liang Yang, Dongyu Zhang, Paerhati Tulajiang, Hongfei Lin
We study idiom-based visual puns--images that align an idiom's literal and figurative meanings--and present an iterative framework that coordinates a large language model (LLM), a text-to-image model (T2IM), and a multimodal LLM (MLLM) for automatic generation and evaluation. Given an idiom, the system iteratively (i) generates detailed visual prompts, (ii) synthesizes an image, (iii) infers the idiom from the image, and (iv) refines the prompt until recognition succeeds or a step limit is reached. Using 1,000 idioms as inputs, we synthesize a corresponding dataset of visual pun images with paired prompts, enabling benchmarking of both generation and understanding. Experiments across 10 LLMs, 10 MLLMs, and one T2IM (Qwen-Image) show that MLLM choice is the primary performance driver: GPT achieves the highest accuracies, Gemini follows, and the best open-source MLLM (Gemma) is competitive with some closed models. On the LLM side, Claude attains the strongest average performance for prompt generation.
comment : Submitted to ICASSP 2026 (under review)
☆ Artwork Interpretation with Vision Language Models: A Case Study on Emotions and Emotion Symbols AACL
Sebastian Padó, Kerstin Thomas
Emotions are a fundamental aspect of artistic expression. Due to their abstract nature, there is a broad spectrum of emotion realization in artworks. These are subject to historical change and their analysis requires expertise in art history. In this article, we investigate which aspects of emotional expression can be detected by current (2025) vision language models (VLMs). We present a case study of three VLMs (Llava-Llama and two Qwen models) in which we ask these models four sets of questions of increasing complexity about artworks (general content, emotional content, expression of emotions, and emotion symbols) and carry out a qualitative expert evaluation. We find that the VLMs recognize the content of the images surprisingly well and often also which emotions they depict and how they are expressed. The models perform best for concrete images but fail for highly abstract or highly symbolic images. Reliable recognition of symbols remains fundamentally difficult. Furthermore, the models continue to exhibit the well-known LLM weakness of providing inconsistent answers to related questions.
comment : Accepted for publication at the IJCNLP-AACL workshop on Multimodal Models for Low-Resource Contexts and Social Impact
☆ Language-conditioned wo rl d model improves policy generalization by reading environmental descriptions
Anh Nguyen, Stefan Lee
To interact effectively with humans in the real world, it is important for agents to understand language that describes the dynamics of the environment--that is, how the environment behaves--rather than just task instructions specifying "what to do". Understanding this dynamics-descriptive language is important for human-agent interaction and agent behavior. Recent work address this problem using a model-based approach: language is incorporated into a world model, which is then used to learn a behavior policy. However, these existing methods either do not demonstrate policy generalization to unseen games or rely on limiting assumptions. For instance, assuming that the latency induced by inference-time planning is tolerable for the target task or expert demonstrations are available. Expanding on this line of research, we focus on improving policy generalization from a language-conditioned world model while dropping these assumptions. We propose a model-based reinforcement learning approach, where a language-conditioned world model is trained through interaction with the environment, and a policy is learned from this model--without planning or expert demonstrations. Our method proposes Language-aware Encoder for Dreamer World Model (LED-WM) built on top of DreamerV3. LED-WM features an observation encoder that uses an attention mechanism to explicitly ground language descriptions to entities in the observation. We show that policies trained with LED-WM generalize more effectively to unseen games described by novel dynamics and language compared to other baselines in several settings in two environments: MESSENGER and MESSENGER-WM.To highlight how the policy can leverage the trained world model before real-world deployment, we demonstrate the policy can be improved through fine-tuning on synthetic test trajectories generated by the world model.
comment : NeuRIPS 2025. Workshop: LAW 2025: Bridging Language, Agent, and World Models
☆ ORION: Teaching Language Models to Reason Efficiently in the Language of Thought
Kumar Tanmay, Kriti Aggarwal, Paul Pu Liang, Subhabrata Mukherjee
Large Reasoning Models (LRMs) achieve strong performance in mathematics, code generation, and task planning, but their reliance on long chains of verbose "thinking" tokens leads to high latency, redundancy, and incoherent reasoning paths. Inspired by the Language of Thought Hypothesis, which posits that human reasoning operates over a symbolic, compositional mental language called Mentalese, we introduce a framework that trains models to reason in a similarly compact style. Mentalese encodes abstract reasoning as ultra-compressed, structured tokens, enabling models to solve complex problems with far fewer steps. To improve both efficiency and accuracy, we propose SHORTER LENGTH PREFERENCE OPTIMIZATION (SLPO), a reinforcement learning method that rewards concise solutions that stay correct, while still allowing longer reasoning when needed. Applied to Mentalese-aligned models, SLPO yields significantly higher compression rates by enabling concise reasoning that preserves the benefits of detailed thinking without the computational overhead. Across benchmarks including AIME 2024 and 2025, MinervaMath, OlympiadBench, Math500, and AMC, our ORION models produce reasoning traces with 4-16x fewer tokens, achieve up to 5x lower inference latency, and reduce training costs by 7-9x relative to the DeepSeek R1 Distilled model, while maintaining 90-98% of its accuracy. ORION also surpasses Claude and ChatGPT-4o by up to 5% in accuracy while maintaining 2x compression. These results show that Mentalese-style compressed reasoning offers a step toward human-like cognitive efficiency, enabling real-time, cost-effective reasoning without sacrificing accuracy.
☆ FEANEL: A Benchmark for Fine-Grained Error Analysis in K-12 English Writing
Jingheng Ye, Shen Wang, Jiaqi Chen, Hebin Wang, Deqing Zou, Yanyu Zhu, Jiwei Tang, Hai-Tao Zheng, Ruitong Liu, Haoyang Li, Yanfeng Wang, Qingsong Wen
Large Language Models (LLMs) have transformed artificial intelligence, offering profound opportunities for educational applications. However, their ability to provide fine-grained educational feedback for K-12 English writing remains underexplored. In this paper, we challenge the error analysis and pedagogical skills of LLMs by introducing the problem of Fine-grained Error Analysis for English Learners and present the Fine-grained Error ANalysis for English Learners (FEANEL) Benchmark. The benchmark comprises 1,000 essays written by elementary and secondary school students, and a well-developed English writing error taxonomy. Each error is annotated by language education experts and categorized by type, severity, and explanatory feedback, using a part-of-speech-based taxonomy they co-developed. We evaluate state-of-the-art LLMs on the FEANEL Benchmark to explore their error analysis and pedagogical abilities. Experimental results reveal significant gaps in current LLMs' ability to perform fine-grained error analysis, highlighting the need for advancements in particular methods for educational applications.
comment : 19 pages, 7 figures, and 4 tables. The dataset is available at https://huggingface.co/datasets/Feanel/FEANEL
☆ JBE-QA: Japanese Bar Exam QA Dataset for Assessing Legal Domain Knowledge
Zhihan Cao, Fumihito Nishino, Hiroaki Yamada, Nguyen Ha Thanh, Yusuke Miyao, Ken Satoh
We introduce JBE-QA, a Japanese Bar Exam Question-Answering dataset to evaluate large language models' legal knowledge. Derived from the multiple-choice (tanto-shiki) section of the Japanese bar exam (2015-2024), JBE-QA provides the first comprehensive benchmark for Japanese legal-domain evaluation of LLMs. It covers the Civil Code, the Penal Code, and the Constitution, extending beyond the Civil Code focus of prior Japanese resources. Each question is decomposed into independent true/false judgments with structured contextual fields. The dataset contains 3,464 items with balanced labels. We evaluate 26 LLMs, including proprietary, open-weight, Japanese-specialised, and reasoning models. Our results show that proprietary models with reasoning enabled perform best, and the Constitution questions are generally easier than the Civil Code or the Penal Code questions.
comment : Three tables and one figure
☆ RAG System for Supporting Japanese Litigation Procedures: Faithful Response Generation Complying with Legal Norms SIGIR
Yuya Ishihara, Atsushi Keyaki, Hiroaki Yamada, Ryutaro Ohara, Mihoko Sumida
This study discusses the essential components that a Retrieval-Augmented Generation (RAG)-based LLM system should possess in order to support Japanese medical litigation procedures complying with legal norms. In litigation, expert commissioners, such as physicians, architects, accountants, and engineers, provide specialized knowledge to help judges clarify points of dispute. When considering the substitution of these expert roles with a RAG-based LLM system, the constraint of strict adherence to legal norms is imposed. Specifically, three requirements arise: (1) the retrieval module must retrieve appropriate external knowledge relevant to the disputed issues in accordance with the principle prohibiting the use of private knowledge, (2) the responses generated must originate from the context provided by the RAG and remain faithful to that context, and (3) the retrieval module must reference external knowledge with appropriate timestamps corresponding to the issues at hand. This paper discusses the design of a RAG-based LLM system that satisfies these requirements.
comment : This is a preprint version of a paper reviewed and accepted at BREV-RAG 2025: Beyond Relevance-based EValuation of RAG Systems, a SIGIR-AP 2025 workshop
☆ Mitigating Semantic Drift: Evaluating LLMs' Efficacy in Psychotherapy through MI Dialogue Summarization
Vivek Kumar, Pushpraj Singh Rajawat, Eirini Ntoutsi
Recent advancements in large language models (LLMs) have shown their potential across both general and domain-specific tasks. However, there is a growing concern regarding their lack of sensitivity, factual incorrectness in responses, inconsistent expressions of empathy, bias, hallucinations, and overall inability to capture the depth and complexity of human understanding, especially in low-resource and sensitive domains such as psychology. To address these challenges, our study employs a mixed-methods approach to evaluate the efficacy of LLMs in psychotherapy. We use LLMs to generate precise summaries of motivational interviewing (MI) dialogues and design a two-stage annotation scheme based on key components of the Motivational Interviewing Treatment Integrity (MITI) framework, namely evocation, collaboration, autonomy, direction, empathy, and a non-judgmental attitude. Using expert-annotated MI dialogues as ground truth, we formulate multi-class classification tasks to assess model performance under progressive prompting techniques, incorporating one-shot and few-shot prompting. Our results offer insights into LLMs' capacity for understanding complex psychological constructs and highlight best practices to mitigate ``semantic drift" in therapeutic settings. Our work contributes not only to the MI community by providing a high-quality annotated dataset to address data scarcity in low-resource domains but also critical insights for using LLMs for precise contextual interpretation in complex behavioral therapy.
☆ OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusion ACL 2026
Sai Koneru, Matthias Huck, Jan Niehues
There has been significant progress in open-source text-only translation large language models (LLMs) with better language coverage and quality. However, these models can be only used in cascaded pipelines for speech translation (ST), performing automatic speech recognition first followed by translation. This introduces additional latency, which is particularly critical in simultaneous ST (SimulST), and prevents the model from exploiting multimodal context, such as images, which can aid disambiguation. Pretrained multimodal foundation models (MMFMs) already possess strong perception and reasoning capabilities across multiple modalities, but generally lack the multilingual coverage and specialized translation performance of dedicated translation LLMs. To build an effective multimodal translation system, we propose an end-to-end approach that fuses MMFMs with translation LLMs. We introduce a novel fusion strategy that connects hidden states from multiple layers of a pretrained MMFM to a translation LLM, enabling joint end-to-end training. The resulting model, OmniFusion, built on Omni 2.5-7B as the MMFM and SeedX PPO-7B as the translation LLM, can perform speech-to-text, speech-and-image-to-text, and text-and-image-to-text translation. Experiments demonstrate that OmniFusion effectively leverages both audio and visual inputs, achieves a 1-second latency reduction in SimulST compared to cascaded pipelines and also improves the overall translation quality\footnote{Code is available at https://github.com/saikoneru/OmniFusion}.
comment : Preprint for ACL 2026
☆ Minimal-Edit Instruction Tuning for Low-Resource Indic GEC AACL
Akhil Rajeev P
Grammatical error correction for Indic languages faces limited supervision, diverse scripts, and rich morphology. We propose an augmentation-free setup that uses instruction-tuned large language models and conservative decoding. A 12B GEMMA 3 model is instruction-tuned in bnb 4-bit precision with parameter-efficient fine-tuning (PEFT) and Alpaca-style formatting. Decoding follows a deterministic, constraint-aware procedure with a lightweight normaliser that encourages minimal, meaning-preserving edits. We operationalise inference, subsequent to instruction fine-tuning (IFT), via a fixed, language-specific prompt directly synthesised from a deterministic error classifier's taxonomy, label distributions, and precedence ordering computed on the training corpus. Under the official untuned GLEU evaluation, the system scores 92.41 on Malayalam, sixth overall, and 81.44 on Hindi, third overall. These results indicate that classifier-informed prompt design, adapter-based instruction tuning, and deterministic decoding provide a reproducible and a computationally efficient alternative to augmentation-centred pipelines for Indic GEC. The approach also motivates future work on stronger morphosyntactic constraints and human-centred evaluation of conservative edits.
comment : Submitted to AACL-IJCNLP Bhasha Workshop Shared Task1 :GEC
☆ Towards Corpus-Grounded Agent ic LLMs for Multilingual Grammatical Analysis
Matej Klemen, Tjaša Arčon, Luka Terčon, Marko Robnik-Šikonja, Kaja Dobrovoljc
Empirical grammar research has become increasingly data-driven, but the systematic analysis of annotated corpora still requires substantial methodological and technical effort. We explore how agentic large language models (LLMs) can streamline this process by reasoning over annotated corpora and producing interpretable, data-grounded answers to linguistic questions. We introduce an agentic framework for corpus-grounded grammatical analysis that integrates concepts such as natural-language task interpretation, code generation, and data-driven reasoning. As a proof of concept, we apply it to Universal Dependencies (UD) corpora, testing it on multilingual grammatical tasks inspired by the World Atlas of Language Structures (WALS). The evaluation spans 13 word-order features and over 170 languages, assessing system performance across three complementary dimensions - dominant-order accuracy, order-coverage completeness, and distributional fidelity - which reflect how well the system generalizes, identifies, and quantifies word-order variations. The results demonstrate the feasibility of combining LLM reasoning with structured linguistic data, offering a first step toward interpretable, scalable automation of corpus-based grammatical inquiry.
comment : Pre-print, submission under review
☆ Tree Matching Networks for Natural Language Inference: Parameter-Efficient Semantic Understanding via Dependency Parse Trees
Jason Lunder
In creating sentence embeddings for Natural Language Inference (NLI) tasks, using transformer-based models like BERT leads to high accuracy, but require hundreds of millions of parameters. These models take in sentences as a sequence of tokens, and learn to encode the meaning of the sequence into embeddings such that those embeddings can be used reliably for NLI tasks. Essentially, every word is considered against every other word in the sequence, and the transformer model is able to determine the relationships between them, entirely from scratch. However, a model that accepts explicit linguistic structures like dependency parse trees may be able to leverage prior encoded information about these relationships, without having to learn them from scratch, thus improving learning efficiency. To investigate this, we adapt Graph Matching Networks (GMN) to operate on dependency parse trees, creating Tree Matching Networks (TMN). We compare TMN to a BERT based model on the SNLI entailment task and on the SemEval similarity task. TMN is able to achieve significantly better results with a significantly reduced memory footprint and much less training time than the BERT based model on the SNLI task, while both models struggled to preform well on the SemEval. Explicit structural representations significantly outperform sequence-based models at comparable scales, but current aggregation methods limit scalability. We propose multi-headed attention aggregation to address this limitation.
comment : 16 pages, preprint
☆ Measuring What LLMs Think They Do: SHAP Faithfulness and Deployability on Financial Tabular Classification AAAI 2026
Saeed AlMarri, Mathieu Ravaut, Kristof Juhasz, Gautier Marti, Hamdan Al Ahbabi, Ibrahim Elfadel
Large Language Models (LLMs) have attracted significant attention for classification tasks, offering a flexible alternative to trusted classical machine learning models like LightGBM through zero-shot prompting. However, their reliability for structured tabular data remains unclear, particularly in high stakes applications like financial risk assessment. Our study systematically evaluates LLMs and generates their SHAP values on financial classification tasks. Our analysis shows a divergence between LLMs self-explanation of feature impact and their SHAP values, as well as notable differences between LLMs and LightGBM SHAP values. These findings highlight the limitations of LLMs as standalone classifiers for structured financial modeling, but also instill optimism that improved explainability mechanisms coupled with few-shot prompting will make LLMs usable in risk-sensitive domains.
comment : 7 pages, 3 figures, 3 tables, AAAI 2026 Deployable AI Workshop
☆ Misalignment of LLM-Generated Personas with Human Perceptions in Low-Resource Settings
Tabia Tanzin Prama, Christopher M. Danforth, Peter Sheridan Dodds
Recent advances enable Large Language Models (LLMs) to generate AI personas, yet their lack of deep contextual, cultural, and emotional understanding poses a significant limitation. This study quantitatively compared human responses with those of eight LLM-generated social personas (e.g., Male, Female, Muslim, Political Supporter) within a low-resource environment like Bangladesh, using culturally specific questions. Results show human responses significantly outperform all LLMs in answering questions, and across all matrices of persona perception, with particularly large gaps in empathy and credibility. Furthermore, LLM-generated content exhibited a systematic bias along the lines of the ``Pollyanna Principle'', scoring measurably higher in positive sentiment ($Φ_{avg} = 5.99$ for LLMs vs. $5.60$ for Humans). These findings suggest that LLM personas do not accurately reflect the authentic experience of real people in resource-scarce environments. It is essential to validate LLM personas against real-world human data to ensure their alignment and reliability before deploying them in social science research.
☆ Alleviating Choice Supportive Bias in LLM with Reasoning Dependency Generation
Nan Zhuang, Wenshuo Wang, Lekai Qian, Yuxiao Wang, Boyu Cao, Qi Liu
Recent studies have demonstrated that some Large Language Models exhibit choice-supportive bias (CSB) when performing evaluations, systematically favoring their chosen options and potentially compromising the objectivity of AI-assisted decision making. While existing debiasing approaches primarily target demographic and social biases, methods for addressing cognitive biases in LLMs remain largely unexplored. In this work, we present the first solution to address CSB through Reasoning Dependency Generation (RDG), a novel framework for generating unbiased reasoning data to mitigate choice-supportive bias through fine-tuning. RDG automatically constructs balanced reasoning QA pairs, explicitly (un)modeling the dependencies between choices, evidences, and justifications. Our approach is able to generate a large-scale dataset of QA pairs across domains, incorporating Contextual Dependency Data and Dependency Decouple Data. Experiments show that LLMs fine-tuned on RDG-generated data demonstrate a 81.5% improvement in memory-based experiments and 94.3% improvement in the evaluation-based experiment, while maintaining similar performance on standard BBQ benchmarks. This work pioneers an approach for addressing cognitive biases in LLMs and contributes to the development of more reliable AI-assisted decision support systems.
☆ Watermarks for Embeddings-as-a-Service Large Language Models
Anudeex Shetty
Large Language Models (LLMs) have demonstrated exceptional capabilities in natural language understanding and generation. Based on these LLMs, businesses have started to provide Embeddings-as-a-Service (EaaS), offering feature extraction capabilities (in the form of text embeddings) that benefit downstream natural language processing tasks. However, prior research has demonstrated that EaaS is vulnerable to imitation attacks, where an attacker clones the service's model in a black-box manner without access to the model's internal workings. In response, watermarks have been added to the text embeddings to protect the intellectual property of EaaS providers by allowing them to check for model ownership. This thesis focuses on defending against imitation attacks by investigating EaaS watermarks. To achieve this goal, we unveil novel attacks and propose and validate new watermarking techniques. Firstly, we show that existing EaaS watermarks can be removed through paraphrasing the input text when attackers clone the model during imitation attacks. Our study illustrates that paraphrasing can effectively bypass current state-of-the-art EaaS watermarks across various attack setups (including different paraphrasing techniques and models) and datasets in most instances. This demonstrates a new vulnerability in recent EaaS watermarking techniques. Subsequently, as a countermeasure, we propose a novel watermarking technique, WET (Watermarking EaaS with Linear Transformation), which employs linear transformation of the embeddings. Watermark verification is conducted by applying a reverse transformation and comparing the similarity between recovered and original embeddings. We demonstrate its robustness against paraphrasing attacks with near-perfect verifiability. We conduct detailed ablation studies to assess the significance of each component and hyperparameter in WET.
♻ ☆ Asymmetric REINFORCE for off-Policy Reinforcement Learning : Balancing positive and negative rewards
Charles Arnal, Gaëtan Narozniak, Vivien Cabannes, Yunhao Tang, Julia Kempe, Remi Munos
Reinforcement learning (RL) is increasingly used to align large language models (LLMs). Off-policy methods offer greater implementation simplicity and data efficiency than on-policy techniques, but often result in suboptimal performance. In this work, we study the intermediate range of algorithms between off-policy RL and supervised fine-tuning by analyzing a simple off-policy REINFORCE algorithm, where the advantage is defined as $A=r-V$, with $r$ a reward and $V$ some tunable baseline. Intuitively, lowering $V$ emphasizes high-reward samples, while raising it penalizes low-reward ones more heavily. We first provide a theoretical analysis of this off-policy REINFORCE algorithm, showing that when the baseline $V$ lower-bounds the expected reward, the algorithm enjoys a policy improvement guarantee. Our analysis reveals that while on-policy updates can safely leverage both positive and negative signals, off-policy updates benefit from focusing more on positive rewards than on negative ones. We validate our findings experimentally in a controlled stochastic bandit setting and through fine-tuning state-of-the-art LLMs on reasoning tasks.
♻ ☆ Robust LLM Unlearning with MUDMAN: Meta-Unlearning with Disruption Masking And Normalization
Filip Sondej, Yushi Yang, Mikołaj Kniejski, Marcel Windys
Language models can retain dangerous knowledge and skills even after extensive safety fine-tuning, posing both misuse and misalignment risks. Recent studies show that even specialized unlearning methods can be easily reversed. To address this, we systematically evaluate many existing and novel components of unlearning methods and identify ones crucial for irreversible unlearning. We introduce Disruption Masking, a technique in which we only allow updating weights, where the signs of the unlearning gradient and the retaining gradient are the same. This ensures all updates are non-disruptive. Additionally, we identify the need for normalizing the unlearning gradients, and also confirm the usefulness of meta-learning. We combine these insights into MUDMAN (Meta-Unlearning with Disruption Masking and Normalization) and validate its effectiveness at preventing the recovery of dangerous capabilities. MUDMAN outperforms the prior TAR method by 40%, setting a new state-of-the-art for robust unlearning.
♻ ☆ Toward Honest Language Models for Deductive Reasoning
Jiarui Liu, Kaustubh Dhole, Yingheng Wang, Haoyang Wen, Sarah Zhang, Haitao Mao, Gaotang Li, Neeraj Varshney, Jingguo Liu, Xiaoman Pan
Deductive reasoning is the process of deriving conclusions strictly from the given premises, without relying on external knowledge. We define honesty in this setting as a model's ability to respond only when the conclusion is logically entailed by the premises, and to abstain otherwise. However, current language models often fail to reason honestly, producing unwarranted answers when the input is insufficient. To study this challenge, we formulate honest deductive reasoning as multi-step tasks where models must either derive the correct conclusion or abstain. We curate two datasets from graph structures, one for linear algebra and one for logical inference, and introduce unanswerable cases by randomly perturbing an edge in half of the instances. We find that prompting and existing training methods, including GRPO with or without supervised fine-tuning initialization, struggle on these tasks. In particular, GRPO optimize only for final task outcomes, leaving models vulnerable to collapse when negative rewards dominate early training. To address this, we propose ACNCHOR, a reinforcement learning method that injects ground truth trajectories into rollouts, preventing early training collapse. Our results demonstrate that this method stabilizes learning and significantly improves the overall reasoning performance, underscoring the importance of training dynamics for enabling honest deductive reasoning in language models.
♻ ☆ Continual Learning of Domain Knowledge from Human Feedback in Text-to-SQL
Thomas Cook, Kelly Patel, Sivapriya Vellaichamy, Udari Madhushani Sehwag, Saba Rahimi, Zhen Zeng, Sumitra Ganesh
Large Language Models (LLMs) can generate SQL queries from natural language questions but struggle with database-specific schemas and tacit domain knowledge. We introduce a framework for continual learning from human feedback in text-to-SQL, where a learning agent receives natural language feedback to refine queries and distills the revealed knowledge for reuse on future tasks. This distilled knowledge is stored in a structured memory, enabling the agent to improve execution accuracy over time. We design and evaluate multiple variations of a learning agent architecture that vary in how they capture and retrieve past experiences. Experiments on the BIRD benchmark Dev set show that memory-augmented agents, particularly the Procedural Agent, achieve significant accuracy gains and error reduction by leveraging human-in-the-loop feedback. Our results highlight the importance of transforming tacit human expertise into reusable knowledge, paving the way for more adaptive, domain-aware text-to-SQL systems that continually learn from a human-in-the-loop.
comment : 34 pages, 6 figures, 4 tables
♻ ☆ OmniRouter: Budget and Performance Controllable Multi-LLM Routing
Kai Mei, Wujiang Xu, Minghao Guo, Shuhang Lin, Yongfeng Zhang
Large language models (LLMs) deliver superior performance but require substantial computational resources and operate with relatively low efficiency, while smaller models can efficiently handle simpler tasks with fewer resources. LLM routing is a crucial paradigm that dynamically selects the most suitable large language models from a pool of candidates to process diverse inputs, ensuring optimal resource utilization while maintaining response quality. Existing routing frameworks typically model this as a locally optimal decision-making problem, selecting the presumed best-fit LLM for each query individually, which overlooks global budget constraints, resulting in ineffective resource allocation. To tackle this problem, we introduce OmniRouter, a fundamentally controllable routing framework for multi-LLM serving. Instead of making per-query greedy choices, OmniRouter models the routing task as a constrained optimization problem, assigning models that minimize total cost while ensuring the required performance level. Specifically, a hybrid retrieval-augmented predictor is designed to predict the capabilities and costs of LLMs. After obtaining the predicted cost and performance, we utilize a constrained optimizer for cost-optimal assignments that employs Lagrangian dual decomposition with adaptive multipliers. It iteratively converges toward the globally optimal query-model allocation, dynamically balancing latency minimization against quality thresholds while adhering to heterogeneous capacity constraints. Experiments show that OmniRouter achieves up to 6.30% improvement in response accuracy while simultaneously reducing computational costs by at least 10.15% compared to competitive router baselines. The code and the dataset are available at https://github.com/dongyuanjushi/OmniRouter.
♻ ☆ Mina: A Multilingual LLM-Powered Legal Assistant Agent for Bangladesh for Empowering Access to Justice
Azmine Toushik Wasi, Wahid Faisal, Mst Rafia Islam
Bangladesh's low-income population faces major barriers to affordable legal advice due to complex legal language, procedural opacity, and high costs. Existing AI legal assistants lack Bengali-language support and jurisdiction-specific adaptation, limiting their effectiveness. To address this, we developed Mina, a multilingual LLM-based legal assistant tailored for the Bangladeshi context. It employs multilingual embeddings and a RAG-based chain-of-tools framework for retrieval, reasoning, translation, and document generation, delivering context-aware legal drafts, citations, and plain-language explanations via an interactive chat interface. Evaluated by law faculty from leading Bangladeshi universities across all stages of the 2022 and 2023 Bangladesh Bar Council Exams, Mina scored 75-80% in Preliminary MCQs, Written, and simulated Viva Voce exams, matching or surpassing average human performance and demonstrating clarity, contextual understanding, and sound legal reasoning. Even under a conservative upper bound, Mina operates at just 0.12-0.61% of typical legal consultation costs in Bangladesh, yielding a 99.4-99.9\% cost reduction relative to human-provided services. These results confirm its potential as a low-cost, multilingual AI assistant that automates key legal tasks and scales access to justice, offering a real-world case study on building domain-specific, low-resource systems and addressing challenges of multilingual adaptation, efficiency, and sustainable public-service AI deployment.
♻ ☆ STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence
Zihan Liu, Zhikang Niu, Qiuyang Xiao, Zhisheng Zheng, Ruoqi Yuan, Yuhang Zang, Yuhang Cao, Xiaoyi Dong, Jianze Liang, Xie Chen, Leilei Sun, Dahua Lin, Jiaqi Wang
Despite rapid progress in Multi-modal Large Language Models and Large Audio-Language Models, existing audio benchmarks largely test semantics that can be recovered from text captions, masking deficits in fine-grained perceptual reasoning. We formalize audio 4D intelligence that is defined as reasoning over sound dynamics in time and 3D space, and introduce STAR-Bench to measure it. STAR-Bench combines a Foundational Acoustic Perception setting (six attributes under absolute and relative regimes) with a Holistic Spatio-Temporal Reasoning setting that includes segment reordering for continuous and discrete processes and spatial tasks spanning static localization, multi-source relations, and dynamic trajectories. Our data curation pipeline uses two methods to ensure high-quality samples. For foundational tasks, we use procedurally synthesized and physics-simulated audio. For holistic data, we follow a four-stage process that includes human annotation and final selection based on human performance. Unlike prior benchmarks where caption-only answering reduces accuracy slightly, STAR-Bench induces far larger drops (-31.5\% temporal, -35.2\% spatial), evidencing its focus on linguistically hard-to-describe cues. Evaluating 19 models reveals substantial gaps compared with humans and a capability hierarchy: closed-source models are bottlenecked by fine-grained perception, while open-source models lag across perception, knowledge, and reasoning. Our STAR-Bench provides critical insights and a clear path forward for developing future models with a more robust understanding of the physical world.
comment : Homepage: https://internlm.github.io/StarBench/
♻ ☆ Local Hybrid Retrieval-Augmented Document QA ACL
Paolo Astrino
Organizations handling sensitive documents face a critical dilemma: adopt cloud-based AI systems that offer powerful question-answering capabilities but compromise data privacy, or maintain local processing that ensures security but delivers poor accuracy. We present a question-answering system that resolves this trade-off by combining semantic understanding with keyword precision, operating entirely on local infrastructure without internet access. Our approach demonstrates that organizations can achieve competitive accuracy on complex queries across legal, scientific, and conversational documents while keeping all data on their machines. By balancing two complementary retrieval strategies and using consumer-grade hardware acceleration, the system delivers reliable answers with minimal errors, letting banks, hospitals, and law firms adopt conversational document AI without transmitting proprietary information to external providers. This work establishes that privacy and performance need not be mutually exclusive in enterprise AI deployment.
comment : 10 pages, 5 figures, 3 tables; conference-style (ACL format); fully local RAG system
♻ ☆ Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey
Qizhi Pei, Zhimeng Zhou, Kaiyuan Gao, Jinhua Zhu, Yue Wang, Zun Wang, Tao Qin, Lijun Wu, Rui Yan
The integration of biomolecular modeling with natural language (BL) has emerged as a promising interdisciplinary area at the intersection of artificial intelligence, chemistry and biology. This approach leverages the rich, multifaceted descriptions of biomolecules contained within textual data sources to enhance our fundamental understanding and enable downstream computational tasks such as biomolecule property prediction. The fusion of the nuanced narratives expressed through natural language with the structural and functional specifics of biomolecules described via various molecular modeling techniques opens new avenues for comprehensively representing and analyzing biomolecules. By incorporating the contextual language data that surrounds biomolecules into their modeling, BL aims to capture a holistic view encompassing both the symbolic qualities conveyed through language as well as quantitative structural characteristics. In this review, we provide an extensive analysis of recent advancements achieved through cross modeling of biomolecules and natural language. (1) We begin by outlining the technical representations of biomolecules employed, including sequences, 2D graphs, and 3D structures. (2) We then examine in depth the rationale and key objectives underlying effective multi-modal integration of language and molecular data sources. (3) We subsequently survey the practical applications enabled to date in this developing research area. (4) We also compile and summarize the available resources and datasets to facilitate future work. (5) Looking ahead, we identify several promising research directions worthy of further exploration and investment to continue advancing the field. The related resources and contents are updating in https://github.com/QizhiPei/Awesome-Biomolecule-Language-Cross-Modeling.
comment : 2025.11.28 Updated Version
♻ ☆ Adversarial Confusion Attack: Disrupting Multimodal Large Language Models
Jakub Hoscilowicz, Artur Janicki
We introduce the Adversarial Confusion Attack, a new class of threats against multimodal large language models (MLLMs). Unlike jailbreaks or targeted misclassification, the goal is to induce systematic disruption that makes the model generate incoherent or confidently incorrect outputs. Applications include embedding adversarial images into websites to prevent MLLM-powered agents from operating reliably. The proposed attack maximizes next-token entropy using a small ensemble of open-source MLLMs. In the white-box setting, we show that a single adversarial image can disrupt all models in the ensemble, both in the full-image and adversarial CAPTCHA settings. Despite relying on a basic adversarial technique (PGD), the attack generates perturbations that transfer to both unseen open-source (e.g., Qwen3-VL) and proprietary (e.g., GPT-5.1) models.
♻ ☆ REFLEX: Self-Refining Explainable Fact-Checking via Disentangling Truth into Style and Substance
Chuyi Kong, Gao Wei, Jing Ma, Hongzhan Lin, Yaxin Fan
The prevalence of misinformation on social media threatens public trust, demanding automated fact-checking systems that provide accurate verdicts with interpretable explanations. However, existing large language model-based (LLM-based) approaches often rely heavily on external knowledge sources, introducing substantial latency and even hallucinations that undermine reliability, interpretability, and responsiveness, which is crucial for real-time use. To address these challenges, we propose REason-guided Fact-checking with Latent EXplanations REFLEX paradigm, a plug-and-play, self-refining paradigm that leverages the internal knowledge in backbone model to improve both verdict accuracy and explanation quality. REFLEX reformulates fact-checking as a role-play dialogue and jointly trains verdict prediction and explanation generation. It adaptively extracts contrastive activation pairs between the backbone model and its fine-tuned variant to construct steering vectors that disentangle truth into style and substance naturally. These activation-level signals guide inference and suppress noisy explanations, enabling more faithful and efficient reasoning. Experiments on real-world datasets show that REFLEX outperforms previous methods that steer toward a single truth direction and underscores the challenge traditional approaches face when handling the subtle, human-unknown truth in fact-checking tasks. Remarkably, with only 465 self-refined training samples, RELFEX achieves state-of-the-art performance. Furthermore, models trained with explanatory objectives can effectively guide those without them, yielding up to a 7.57% improvement, highlighting that internal explanation signals play a dual role in both interpreting and enhancing factual reasoning.
♻ ☆ Efficient Reasoning via Thought-Training and Thought-Free Inference
Canhui Wu, Qiong Cao, Chao Xue, Wei Xi, Xiaodong He
Recent advances in large language models (LLMs) have leveraged explicit Chain-of-Thought (CoT) prompting to improve reasoning accuracy. However, most existing methods primarily focus on compressing verbose reasoning outputs. These Long-to-Short transformations aim to improve efficiency, but require a large amount of short CoT data. In this work, we introduce \textbf{3TF} (\textbf{T}hought-\textbf{T}raining and \textbf{T}hought-\textbf{F}ree inference), a framework for efficient reasoning that takes a Short-to-Long perspective. We first train a hybrid model that can operate in both reasoning and non-reasoning modes, and then further train it on CoT-annotated data to internalize structured reasoning, while enforcing concise, thought-free outputs at inference time using the no-reasoning mode. Unlike compression-based approaches, 3TF improves the reasoning quality of non-reasoning outputs, enabling models to perform rich internal reasoning implicitly while keeping external outputs short. Empirically, 3TF-trained models obtain large improvements on reasoning benchmarks under thought-free inference, demonstrating that high quality reasoning can be learned and executed implicitly without explicit step-by-step generation.
comment : 11 pages, 4 figures
♻ ☆ RvLLM: LLM Runtime Verification with Domain Knowledge
Yedi Zhang, Sun Yi Emma, Annabelle Lee Jia En, Jin Song Dong
Large language models (LLMs) have emerged as a dominant AI paradigm due to their exceptional text understanding and generation capabilities. However, their tendency to generate inconsistent or erroneous outputs challenges their reliability, especially in high-stakes domains requiring accuracy and trustworthiness. Existing research primarily focuses on detecting and mitigating model misbehavior in general-purpose scenarios, often overlooking the potential of integrating domain-specific knowledge. In this work, we advance misbehavior detection by incorporating domain knowledge. The core idea is to design a general specification language that enables domain experts to customize domain-specific predicates in a lightweight and intuitive manner, supporting later runtime verification of LLM outputs. To achieve this, we design a novel specification language, ESL, and introduce a runtime verification framework, RvLLM, to validate LLM output against domain-specific constraints defined in ESL. We evaluate RvLLM on three representative tasks: violation detection against Singapore Rapid Transit Systems Act, numerical comparison, and inequality solving. Experimental results demonstrate that RvLLM effectively detects erroneous outputs across various LLMs in a lightweight and flexible manner. The results reveal that despite their impressive capabilities, LLMs remain prone to low-level errors due to limited interpretability and a lack of formal guarantees during inference, and our framework offers a potential long-term solution by leveraging expert domain knowledge to rigorously and efficiently verify LLM outputs.
comment : 24 pages, 11 tables, 13 figures
♻ ☆ LongCat-Flash-Omni Technical Report
Meituan LongCat Team, Bairui Wang, Bayan, Bin Xiao, Bo Zhang, Bolin Rong, Borun Chen, Chang Wan, Chao Zhang, Chen Huang, Chen Chen, Chen Chen, Chengxu Yang, Chengzuo Yang, Cong Han, Dandan Peng, Delian Ruan, Detai Xin, Disong Wang, Dongchao Yang, Fanfan Liu, Fengjiao Chen, Fengyu Yang, Gan Dong, Gang Huang, Gang Xu, Guanglu Wan, Guoqiang Tan, Guoqiao Yu, Haibo Qiu, Hao Lu, Hongbo Liu, Hongyu Xiang, Jiaheng Wu, Jian Yang, Jiaxing Liu, Jing Huang, Jingang Wang, Jinrui Ding, Juchao Jiang, Jun Kuang, Jun Wang, Junhui Mei, Ke Ding, Kefeng Zhang, Lei Chen, Liang Shi, Limeng Qiao, Liming Zheng, Lin Ma, Liuyang Guo, Liya Ma, Luying Sun, Man Gao, Mengshen Zhu, Miao Cao, Minliang Lin, Nuo Xu, Peng Shi, Qi Zhang, Qian Fang, Qian Wang, Qian Yang, Quanxiu Wang, Rongxiang Weng, Rongxin Guo, Ruoxuan Liang, Senbin Yang, Shanbo Xu, Shanglin Lei, Shengze Ye, Shimin Chen, Shuaiqi Chen, Shujie Hu, Shuo Li, Siqi Yang, Siyu Xu, Siyu Ren, Song Li, Songxiang Liu, Tianhao Bai, Tianye Dai, Wei Hong, Wei Wang, Weixiao Zhao, Wengang Cao, Wenlong Zhu, Wenlong He, Xi Su, Xi Nan, Xiaohan Zhao, Xiaohao Wang, Xiaoyu Zhao, Xiaoyu Wang, Xiaoyu Li, Xin Pan, Xin Chen, Xiusong Sun, Xu Xiang, Xudong Xing, Xuezhi Cao, Xunliang Cai, Yang Yang, Yanli Tan, Yao Yao, Yerui Sun, Yi Chen, Yifan Lu, Yin Gong, Yining Zhang, Yitian Chen, Yiyang Gan, Yuchen Tang, Yuchen Xie, Yueqian Wang, Yuewen Zheng, Yufei Zhang, Yufeng Zhong, Yulei Qian, Yuqi Peng, Yuqian Li, Yuwei Jiang, Zeyang Hu, Zheng Zhang, Zhengkun Tian, Zhiqing Hong, Zhixiong Zeng, Zhuqi Mi, Ziran Li, Ziwen Wang, Ziyi Zhao, Ziyuan Zhuang, Zizhe Zhao
We introduce LongCat-Flash-Omni, a state-of-the-art open-source omni-modal model with 560 billion parameters, excelling at real-time audio-visual interaction. By adopting a curriculum-inspired progressive training strategy that transitions from simpler to increasingly complex modality sequence modeling tasks, LongCat-Flash-Omni attains comprehensive multimodal capabilities while maintaining strong unimodal capability. Building upon LongCat-Flash, which adopts a high-performance Shortcut-connected Mixture-of-Experts (MoE) architecture with zero-computation experts, LongCat-Flash-Omni integrates efficient multimodal perception and speech reconstruction modules. Despite its immense size of 560B parameters (with 27B activated), LongCat-Flash-Omni achieves low-latency real-time audio-visual interaction. For training infrastructure, we developed a modality-decoupled parallelism scheme specifically designed to manage the data and model heterogeneity inherent in large-scale multimodal training. This innovative approach demonstrates exceptional efficiency by sustaining over 90% of the throughput achieved by text-only training. Extensive evaluations show that LongCat-Flash-Omni achieves state-of-the-art performance on omni-modal benchmarks among open-source models. Furthermore, it delivers highly competitive results across a wide range of modality-specific tasks, including text, image, and video understanding, as well as audio understanding and generation. We provide a comprehensive overview of the model architecture design, training procedures, and data strategies, and open-source the model to foster future research and development in the community.
♻ ☆ Normal forms in Virus Machines
A. Ramírez-de-Arellano, F. G. C. Cabarle, D. Orellana-Martín, M. J. Pérez-Jiménez
In the present work, we further study the computational power of virus machines (VMs in short).VMs provide a computing paradigm inspired by the transmission and replication networks of viruses.VMs consist of process units (called hosts) structured by a directed graph whose arcs are called channels and an instruction graph that controls the transmissions of virus objects among hosts. The present work complements our understanding of the computing power of VMs by introducing normal forms; these expressions restrict the features in a given computing model.Some of the features that we restrict in our normal forms include (a) the number of hosts, (b) the number of instructions, and (c) the number of virus objects in each host. After we recall some known results on the computing power of VMs we give our series of normal forms, such as the size of the loops in the network, proving new characterisations of family of sets, such as finite sets, semilinear sets, or recursively enumerable sets (NRE).
comment : 24 pages, 14 figures
♻ ☆ InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training
Pengkai Wang, Linus, Pengwei Liu, Zhijie Sang, Congkai Xie, Hongxia Yang
Reinforcement learning has powered many of the recent breakthroughs in large language models, especially for tasks where rewards can be computed automatically, such as code generation. However, these methods deteriorate in open-ended domains like medical consultation, where feedback is inherently ambiguous, highly context-dependent, and cannot be reduced to a reliable scalar signal. In such settings, RL must either rely on supervision-intensive reward models that often fail to generalize, or it falls into pathological behaviors such as reward hacking - an especially troubling risk for high-stakes medical dialogue. To address these limitations, we introduce ORBIT, an open-ended rubric-based incremental training framework for high-stakes medical dialogue. ORBIT integrates synthetic dialogue generation with dynamically constructed rubrics that serve as adaptive guides for incremental RL. Instead of relying on external medical knowledge bases or handcrafted rule sets, ORBIT uses rubric-driven feedback to steer the learning process. Its judge component can be instantiated with general-purpose instruction-following LLMs, removing the need for any task-specific fine-tuning. Applied to the Qwen3-4B-Instruct model, ORBIT raises the HealthBench-Hard score from 7.0 to 27.5 using only 2k training samples, achieving SOTA performance for models at this scale. With larger rubric datasets, ORBIT-trained models further compete with the strongest open-source baselines on HealthBench-Hard. Our analysis shows that rubric-guided RL consistently improves consultation quality across diverse medical scenarios. We also apply such rubric generation and training pipeline to InfoBench, where ORBIT enhances instruction-following performance, highlighting the generality of rubric-based feedback.
♻ ☆ Atom of Thoughts for Markov LLM Test-Time Scaling NeurIPS 2025
Fengwei Teng, Quan Shi, Zhaoyang Yu, Jiayi Zhang, Chenglin Wu, Yuyu Luo, Zhijiang Guo
Large Language Models (LLMs) achieve superior performance through training-time scaling, and test-time scaling further enhances their capabilities by conducting effective reasoning during inference. However, as the scale of reasoning increases, existing test-time scaling methods suffer from accumulated historical information, which not only wastes computational resources but also interferes with effective reasoning. To address this issue, we observe that complex reasoning can be achieved by solving a series of independent and self-contained subquestions. These subquestions are essentially \textit{atomic questions}, exhibiting the memoryless property similar to Markov processes. Based on this observation, we propose Atom of Thoughts (\our), where each state transition consists of decomposing the current question into a dependency-based directed acyclic graph and contracting its subquestions, forming a simplified question that maintains answer equivalence with the original problem. This answer preservation enables the iterative \textit{decomposition-contraction} process to naturally form a meaningful Markov reasoning process. Furthermore, these atomic states can be seamlessly integrated into existing test-time scaling methods, enabling \our to serve as a plug-in enhancement for improving reasoning capabilities. Experiments across six benchmarks demonstrate the effectiveness of \our both as a standalone framework and a plug-in enhancement. Notably, on HotpotQA, when applied to gpt-4o-mini, \our achieves an \textbf{80.6\%} F1 score, surpassing o3-mini by \textbf{3.4\%} and DeepSeek-R1 by \textbf{10.6\%}. The code is available at \href{https://github.com/qixucen/atom}{https://github.com/qixucen/atom}.
comment : Accepted to NeurIPS 2025
♻ ☆ Toward Equitable Access: Leveraging Crowdsourced Review s to Investigate Public Perceptions of Health Resource Accessibility
Zhaoqian Xue, Guanhong Liu, Chong Zhang, Kai Wei, Qingcheng Zeng, Songhua Hu, Wenyue Hua, Lizhou Fan, Yongfeng Zhang, Lingyao Li
Monitoring health resource disparities during public health crises is critical, yet traditional methods, like surveys, lack the requisite speed and spatial granularity. This study introduces a novel framework that leverages: 1) crowdsourced Google Maps reviews (2018-2021) and 2) advanced NLP (DeBERTa) to create a high-resolution, spatial-temporal index of public perception of health resource accessibility in the United States. We then employ Partial Least Squares (PLS) regression to link this perception index to a range of socioeconomic and demographic drivers. Our results quantify significant spatial-temporal shifts in perceived access, confirming that disparities peaked during the COVID-19 crisis and only partially recovered post-peak. We identify political affiliation, racial composition, and educational attainment as primary determinants of these perceptions. This study validates a scalable method for real-time health equity monitoring and provides actionable evidence for interventions to build a more resilient healthcare infrastructure.
♻ ☆ Prompt -R1: Collaborative Automatic Prompt ing Framework via End-to-end Reinforcement Learning
Wenjin Liu, Haoran Luo, Xueyuan Lin, Haoming Liu, Tiesunlong Shen, Jiapu Wang, Rui Mao, Erik Cambria
Recently, advanced large language models (LLMs) have emerged at an increasingly rapid pace. However, when faced with complex problems, most users are often unable to provide accurate and effective prompts to interact with LLMs, thus limiting the performance of LLMs. To address this challenge, we propose Prompt-R1, an end-to-end reinforcement learning framework that uses a small-scale LLM to collaborate with large-scale LLMs, replacing user interaction to solve problems better. This collaboration is cast as a multi-turn prompt interaction, where the small-scale LLM thinks and generates prompts, and the large-scale LLM performs complex reasoning. A dual-constrained reward is designed to optimize for correctness, generation quality, and reasoning accuracy. Prompt-R1 provides a plug-and-play framework that supports both inference and training with various large-scale LLMs. Experiments on multiple public datasets show that Prompt-R1 significantly outperforms baseline models across tasks. Our code is publicly available at https://github.com/QwenQKing/Prompt-R1.
♻ ☆ Self Iterative Label Refinement via Robust Unlabeled Learning NeurIPS 2025
Hikaru Asano, Tadashi Kozuno, Yukino Baba
Recent advances in large language models (LLMs) have yielded impressive performance on various tasks, yet they often depend on high-quality feedback that can be costly. Self-refinement methods attempt to leverage LLMs' internal evaluation mechanisms with minimal human supervision; however, these approaches frequently suffer from inherent biases and overconfidence, especially in domains where the models lack sufficient internal knowledge, resulting in performance degradation. As an initial step toward enhancing self-refinement for broader applications, we introduce an iterative refinement pipeline that employs the Unlabeled-Unlabeled learning framework to improve LLM-generated pseudo-labels for classification tasks. By exploiting two unlabeled datasets with differing positive class ratios, our approach iteratively denoises and refines the initial pseudo-labels, thereby mitigating the adverse effects of internal biases with minimal human supervision. Evaluations on diverse datasets, including low-resource language corpora, patent classifications, and protein structure categorizations, demonstrate that our method consistently outperforms both initial LLM's classification performance and the self-refinement approaches by cutting-edge models (e.g., GPT-4o and DeepSeek-R1). Moreover, we experimentally confirm that our refined classifier facilitates effective post-training alignment for safety in LLMs and demonstrate successful self-refinement in generative tasks as well.\footnote{Our code is available at https://github.com/HikaruAsano/self-iterative-label-refinement.}
comment : To appear in the Thirty-Ninth Annual Conference on Neural Information Processing Systems (NeurIPS 2025)
♻ ☆ Structured Prompt ing Enables More Robust Evaluation of Language Models
Asad Aali, Muhammad Ahmed Mohsin, Vasiliki Bikia, Arnav Singhvi, Richard Gaus, Suhana Bedi, Hejie Cui, Miguel Fuentes, Alyssa Unell, Yifan Mai, Jordan Cahoon, Michael Pfeffer, Roxana Daneshjou, Sanmi Koyejo, Emily Alsentzer, Christopher Potts, Nigam H. Shah, Akshay S. Chaudhari
As language models (LMs) are increasingly adopted across domains, high-quality benchmarking frameworks that accurately estimate performance are essential for guiding deployment decisions. While frameworks such as Holistic Evaluation of Language Models (HELM) enable broad evaluation across tasks, they often rely on fixed prompts that fail to generalize across LMs, yielding unrepresentative performance estimates. Unless we approximate each LM's ceiling (maximum achievable via changes to the prompt), we risk underestimating performance. Declarative prompting frameworks, such as DSPy, offer a scalable alternative to manual prompt engineering by crafting structured prompts that can be optimized per task. However, such frameworks have not been systematically evaluated across established benchmarks. We present a reproducible DSPy+HELM framework that introduces structured prompting methods which elicit reasoning, enabling more accurate LM benchmarking. Using four prompting methods, we evaluate four frontier LMs across seven benchmarks (general/medical domain) against existing HELM baseline scores. We find that without structured prompting: (i) HELM underestimates LM performance (by 4% average), (ii) performance estimates vary more across benchmarks ($+$2% standard deviation), (iii) performance gaps are misrepresented (leaderboard rankings flip on 3/7 benchmarks), and (iv) introducing chain-of-thought reduces LM sensitivity to prompt design (smaller $Δ$ across prompts). To our knowledge, this is the first benchmarking study to systematically integrate structured prompting into an established evaluation framework, demonstrating how scalable performance-ceiling approximation yields more robust, decision-useful benchmarks. We open-source (i) DSPy+HELM Integration (https://github.com/stanford-crfm/helm/pull/3893) and (ii) Prompt Optimization Pipeline (https://github.com/StanfordMIMI/dspy-helm).
♻ ☆ Benford's Curse: Tracing Digit Bias to Numerical Hallucination in LLMs NeurIPS 2025
Jiandong Shao, Yao Lu, Jianfei Yang
Large Language Models (LLMs) exhibit impressive performance on complex reasoning tasks, yet they frequently fail on basic numerical problems, producing incorrect outputs. Inspired by Benford's Law, a statistical pattern in which lower digits occur more frequently as leading digits, we hypothesize that the skewed digit distributions in web-collected corpora may be learned by LLMs during pretraining, leading to biased numerical generation. To investigate the hypothesis, we first examine whether digits frequencies in pretraining corpus (OLMo2) follows Benford's law. We then construct an evaluation benchmark in which the ground-truth digits are uniformly distributed within each of the seven numerical reasoning tasks. Our evaluation results demonstrate that leading open-source LLMs show a consistent pattern of digit bias that resembles Benford's law. Through logit-lens tracing and neuron-level dissection, we identify that this bias arises predominantly from a small subset of highly digit-selective feed-forward network (FFN) neurons in the deeper layers. Finally, we demonstrate that pruning these neurons mitigates imbalanced overgeneration and partially corrects erroneous outputs, providing causal evidence that fine-grained pretraining digit bias can propagate into model behavior. Our findings reveal a fundamental connection between corpus-level statistics and symbolic failure modes in LLMs, offering a new lens for diagnosing and mitigating hallucinations in numerical tasks.
comment : NeurIPS 2025
♻ ☆ Exploiting Vocabulary Frequency Imbalance in Language Model Pre-train ing NeurIPS 2025
Woojin Chung, Jeonghoon Kim
Large language models are trained with tokenizers, and the resulting token distribution is highly imbalanced: a few words dominate the stream while most occur rarely. Recent practice favors ever-larger vocabularies, but it is unclear where the benefit comes from. To this end, we perform a controlled study that scales the vocabulary of the language model from 24K to 196K while holding data, computation, and optimization unchanged. We begin by quantifying the complexity of tokenized text -- formalized via Kolmogorov complexity -- and show that larger vocabularies reduce this complexity. Above 24K, every common word is already tokenized as a single token, so enlarging vocabulary only deepens the relative token-frequency imbalance. Word-level loss decomposition shows that larger vocabularies reduce cross-entropy loss almost exclusively by lowering uncertainty on the 2,500 most frequent words, even though loss on the rare tail rises. The same frequent words cover roughly 75% of tokens in downstream benchmarks, so this training advantage transfers intact. We further show that enlarging model parameters with a fixed vocabulary yields the same frequent-word benefit. Our results recast "bigger vocabularies help" as "lowering complexity of tokenized text helps," offering a simple, principled knob for tokenizer-model co-design and clarifying the loss dynamics that govern language model scaling in pre-training.
comment : NeurIPS 2025
♻ ☆ Deep Improvement Super vision
Arip Asadulaev, Rayan Banerjee, Fakhri Karray, Martin Takac
Recently, it was shown that small, looped architectures, such as Tiny Recursive Models (TRMs), can outperform Large Language Models (LLMs) on complex reasoning tasks, including the Abstraction and Reasoning Corpus (ARC). In this work, we investigate a core question: how can we further improve the efficiency of these methods with minimal changes? To address this, we frame the latent reasoning of TRMs as a form of classifier-free guidance and implicit policy improvement algorithm. Building on these insights, we propose a novel training scheme that provides a target for each loop during training. We demonstrate that our approach significantly enhances training efficiency. Our method reduces the total number of forward passes by 18x and eliminates halting mechanisms, while maintaining quality comparable to standard TRMs. Notably, we achieve 24% accuracy on ARC-1 with only 0.8M parameters, outperforming most LLMs.
♻ ☆ ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs
Chaoyu Li, Yogesh Kulkarni, Pooyan Fazli
The computational cost of training multimodal large language models (MLLMs) grows rapidly with the number of processed tokens. Existing efficiency methods mainly target inference via token reduction or merging, offering limited benefits during training. We introduce ReGATE (Reference-Guided Adaptive Token Elision), an adaptive token pruning method for accelerating MLLM training. ReGATE adopts a teacher-student framework, in which a frozen teacher LLM provides per-token guidance losses that are fused with an exponential moving average of the student's difficulty estimates. This adaptive scoring mechanism dynamically selects informative tokens while skipping redundant ones in the forward pass, substantially reducing computation without altering the model architecture. Across three representative MLLMs, ReGATE matches the peak accuracy of standard training on MVBench up to 2$\times$ faster, using only 38% of the tokens. With extended training, it even surpasses the baseline across multiple multimodal benchmarks, cutting total token usage by over 41%. Code and models will be released publicly.
♻ ☆ DeID- GPT : Zero-shot Medical Text De-Identification by GPT -4
Zhengliang Liu, Yue Huang, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Yiwei Li, Peng Shu, Fang Zeng, Lichao Sun, Wei Liu, Dinggang Shen, Quanzheng Li, Tianming Liu, Dajiang Zhu, Xiang Li
The digitization of healthcare has facilitated the sharing and re-using of medical data but has also raised concerns about confidentiality and privacy. HIPAA (Health Insurance Portability and Accountability Act) mandates removing re-identifying information before the dissemination of medical records. Thus, effective and efficient solutions for de-identifying medical data, especially those in free-text forms, are highly needed. While various computer-assisted de-identification methods, including both rule-based and learning-based, have been developed and used in prior practice, such solutions still lack generalizability or need to be fine-tuned according to different scenarios, significantly imposing restrictions in wider use. The advancement of large language models (LLM), such as ChatGPT and GPT-4, have shown great potential in processing text data in the medical domain with zero-shot in-context learning, especially in the task of privacy protection, as these models can identify confidential information by their powerful named entity recognition (NER) capability. In this work, we developed a novel GPT4-enabled de-identification framework (``DeID-GPT") to automatically identify and remove the identifying information. Compared to existing commonly used medical text data de-identification methods, our developed DeID-GPT showed the highest accuracy and remarkable reliability in masking private information from the unstructured medical text while preserving the original structure and meaning of the text. This study is one of the earliest to utilize ChatGPT and GPT-4 for medical text data processing and de-identification, which provides insights for further research and solution development on the use of LLMs such as ChatGPT/GPT-4 in healthcare. Codes and benchmarking data information are available at https://github.com/yhydhx/ChatGPT-API.
♻ ☆ On the Robustness of Verbal Confidence of LLMs in Adversarial Attacks NeurIPS 2025
Stephen Obadinma, Xiaodan Zhu
Robust verbal confidence generated by large language models (LLMs) is crucial for the deployment of LLMs to help ensure transparency, trust, and safety in many applications, including those involving human-AI interactions. In this paper, we present the first comprehensive study on the robustness of verbal confidence under adversarial attacks. We introduce attack frameworks targeting verbal confidence scores through both perturbation and jailbreak-based methods, and demonstrate that these attacks can significantly impair verbal confidence estimates and lead to frequent answer changes. We examine a variety of prompting strategies, model sizes, and application domains, revealing that current verbal confidence is vulnerable and that commonly used defence techniques are largely ineffective or counterproductive. Our findings underscore the need to design robust mechanisms for confidence expression in LLMs, as even subtle semantic-preserving modifications can lead to misleading confidence in responses.
comment : Published in NeurIPS 2025
♻ ☆ InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacy NeurIPS 2025
Vishnu Vinod, Krishna Pillutla, Abhradeep Guha Thakurta
As major progress in LLM-based long-form text generation enables paradigms such as retrieval-augmented generation (RAG) and inference-time scaling, safely incorporating private information into the generation remains a critical open question. We present InvisibleInk, a highly scalable long-form text generation framework satisfying rigorous differential privacy guarantees with respect to the sensitive reference texts. It interprets sampling from the LLM's next-token-distribution as the exponential mechanism over the LLM logits with two innovations. First, we reduce the privacy cost by isolating and clipping only the sensitive information in the model logits (relative to the public logits). Second, we improve text quality by sampling without any privacy cost from a small superset of the top-$k$ private tokens. Empirical evaluations demonstrate a consistent $8\times$ (or more) reduction in computation cost over state-of-the-art baselines to generate long-form private text of the same utility across privacy levels. InvisibleInk is able to generate, for the first time, high-quality private long-form text at less than $4$-$8\times$ times the computation cost of non-private generation, paving the way for its practical use. We open-source a pip-installable Python package (invink) for InvisibleInk at https://github.com/cerai-iitm/invisibleink.
comment : Published at NeurIPS 2025
♻ ☆ Combining Textual and Structural Information for Premise Selection in Lean
Job Petrovčič, David Eliecer Narvaez Denis, Ljupčo Todorovski
Premise selection is a key bottleneck for scaling theorem proving in large formal libraries. Yet existing language-based methods often treat premises in isolation, ignoring the web of dependencies that connects them. We present a graph-augmented approach that combines dense text embeddings of Lean formalizations with graph neural networks over a heterogeneous dependency graph capturing both state-premise and premise-premise relations. On the LeanDojo Benchmark, our method outperforms the ReProver language-based baseline by over 25\% across standard retrieval metrics. These results suggest that relational information is beneficial for premise selection.
♻ ☆ TRepLiNa: Layer-wise CKA+REPINA Alignment Improves Low-Resource Machine Translation in Aya-23 8B
Toshiki Nakai, Ravi Kiran Chikkala, Lena Sophie Oberkircher, Nicholas Jennings, Natalia Skachkova, Tatiana Anikina, Jesujoba Oluwadara Alabi
The 2025 Multimodal Models for Low-Resource Contexts and Social Impact (MMLoSo) Language Challenge addresses one of India's most pressing linguistic gaps: the lack of resources for its diverse low-resource languages (LRLs). In this study, we investigate whether enforcing cross-lingual similarity in specific internal layers of a decoder-only multilingual large language model (LLM) can improve translation quality from LRL to high-resource language (HRL). Specifically, we combine Centered Kernel Alignment (CKA), a similarity metric that encourages representations of different languages to align, with REPINA, a regularization method that constrains parameter updates to remain close to the pretrained model, into a joint method we call TRepLiNa. In this research project, we experiment with zero-shot, few-shot, and fine-tuning settings using Aya-23 8B with QLoRA across MMLoSo shared task language pairs (Mundari, Santali, Bhili) with Hindi/English pivots. Our results show that aligning mid-level layers using TRepLiNa (CKA+REPINA) is a low-cost, practical approach to improving LRL translation, especially in data-scarce settings.
comment : It is work in progress
Artificial Intelligence 167
☆ AI summaries in online search influence users' attitudes
Yiwei Xu, Saloni Dash, Sungha Kang, Wang Liao, Emma S. Spiro
This study examined how AI-generated summaries, which have become visually prominent in online search results, affect how users think about different issues. In a preregistered randomized controlled experiment, participants (N = 2,004) viewed mock search result pages varying in the presence (vs. absence), placement (top vs. middle), and stance (benefit-framed vs. harm-framed) of AI-generated summaries across four publicly debated topics. Compared to a no-summary control group, participants exposed to AI-generated summaries reported issue attitudes, behavioral intentions, and policy support that aligned more closely with the AI summary stance. The summaries placed at the top of the page produced stronger shifts in users' issue attitudes (but not behavioral intentions or policy support) than those placed at the middle of the page. We also observed moderating effects from issue familiarity and general trust toward AI. In addition, users perceived the AI summaries more useful when it emphasized health harms versus benefits. These findings suggest that AI-generated search summaries can significantly shape public perceptions, raising important implications for the design and regulation of AI-integrated information ecosystems.
☆ The Hidden AI Race: Tracking Environmental Costs of Innovation
Shyam Agarwal, Mahasweta Chakraborti
The past decade has seen a massive rise in the popularity of AI systems, mainly owing to the developments in Gen AI, which has revolutionized numerous industries and applications. However, this progress comes at a considerable cost to the environment as training and deploying these models consume significant computational resources and energy and are responsible for large carbon footprints in the atmosphere. In this paper, we study the amount of carbon dioxide released by models across different domains over varying time periods. By examining parameters such as model size, repository activity (e.g., commits and repository age), task type, and organizational affiliation, we identify key factors influencing the environmental impact of AI development. Our findings reveal that model size and versioning frequency are strongly correlated with higher emissions, while domain-specific trends show that NLP models tend to have lower carbon footprints compared to audio-based systems. Organizational context also plays a significant role, with university-driven projects exhibiting the highest emissions, followed by non-profits and companies, while community-driven projects show a reduction in emissions. These results highlight the critical need for green AI practices, including the adoption of energy-efficient architectures, optimizing development workflows, and leveraging renewable energy sources. We also discuss a few practices that can lead to a more sustainable future with AI, and we end this paper with some future research directions that could be motivated by our work. This work not only provides actionable insights to mitigate the environmental impact of AI but also poses new research questions for the community to explore. By emphasizing the interplay between sustainability and innovation, our study aims to guide future efforts toward building a more ecologically responsible AI ecosystem.
☆ Distracted Robot: How Visual Clutter Undermine Robotic Manipulation
Amir Rasouli, Montgomery Alban, Sajjad Pakdamansavoji, Zhiyuan Li, Zhanguang Zhang, Aaron Wu, Xuan Zhao
In this work, we propose an evaluation protocol for examining the performance of robotic manipulation policies in cluttered scenes. Contrary to prior works, we approach evaluation from a psychophysical perspective, therefore we use a unified measure of clutter that accounts for environmental factors as well as the distractors quantity, characteristics, and arrangement. Using this measure, we systematically construct evaluation scenarios in both hyper-realistic simulation and real-world and conduct extensive experimentation on manipulation policies, in particular vision-language-action (VLA) models. Our experiments highlight the significant impact of scene clutter, lowering the performance of the policies, by as much as 34% and show that despite achieving similar average performance across the tasks, different VLA policies have unique vulnerabilities and a relatively low agreement on success scenarios. We further show that our clutter measure is an effective indicator of performance degradation and analyze the impact of distractors in terms of their quantity and occluding influence. At the end, we show that finetuning on enhanced data, although effective, does not equally remedy all negative impacts of clutter on performance.
comment : 12 figures, 2 tables
☆ Improving Robotic Manipulation Robustness via NICE Scene Surgery
Sajjad Pakdamansavoji, Mozhgan Pourkeshavarz, Adam Sigal, Zhiyuan Li, Rui Heng Yang, Amir Rasouli
Learning robust visuomotor policies for robotic manipulation remains a challenge in real-world settings, where visual distractors can significantly degrade performance and safety. In this work, we propose an effective and scalable framework, Naturalistic Inpainting for Context Enhancement (NICE). Our method minimizes out-of-distribution (OOD) gap in imitation learning by increasing visual diversity through construction of new experiences using existing demonstrations. By utilizing image generative frameworks and large language models, NICE performs three editing operations, object replacement, restyling, and removal of distracting (non-target) objects. These changes preserve spatial relationships without obstructing target objects and maintain action-label consistency. Unlike previous approaches, NICE requires no additional robot data collection, simulator access, or custom model training, making it readily applicable to existing robotic datasets. Using real-world scenes, we showcase the capability of our framework in producing photo-realistic scene enhancement. For downstream tasks, we use NICE data to finetune a vision-language model (VLM) for spatial affordance prediction and a vision-language-action (VLA) policy for object manipulation. Our evaluations show that NICE successfully minimizes OOD gaps, resulting in over 20% improvement in accuracy for affordance prediction in highly cluttered scenes. For manipulation tasks, success rate increases on average by 11% when testing in environments populated with distractors in different quantities. Furthermore, we show that our method improves visual robustness, lowering target confusion by 6%, and enhances safety by reducing collision rate by 7%.
comment : 11 figures, 3 tables
☆ CAPE: Context -Aware Diffusion Policy Via Proximal Mode Expansion for Collision Avoidance
Rui Heng Yang, Xuan Zhao, Leo Maxime Brunswic, Montgomery Alban, Mateo Clemente, Tongtong Cao, Jun Jin, Amir Rasouli
In robotics, diffusion models can capture multi-modal trajectories from demonstrations, making them a transformative approach in imitation learning. However, achieving optimal performance following this regiment requires a large-scale dataset, which is costly to obtain, especially for challenging tasks, such as collision avoidance. In those tasks, generalization at test time demands coverage of many obstacles types and their spatial configurations, which are impractical to acquire purely via data. To remedy this problem, we propose Context-Aware diffusion policy via Proximal mode Expansion (CAPE), a framework that expands trajectory distribution modes with context-aware prior and guidance at inference via a novel prior-seeded iterative guided refinement procedure. The framework generates an initial trajectory plan and executes a short prefix trajectory, and then the remaining trajectory segment is perturbed to an intermediate noise level, forming a trajectory prior. Such a prior is context-aware and preserves task intent. Repeating the process with context-aware guided denoising iteratively expands mode support to allow finding smoother, less collision-prone trajectories. For collision avoidance, CAPE expands trajectory distribution modes with collision-aware context, enabling the sampling of collision-free trajectories in previously unseen environments while maintaining goal consistency. We evaluate CAPE on diverse manipulation tasks in cluttered unseen simulated and real-world settings and show up to 26% and 80% higher success rates respectively compared to SOTA methods, demonstrating better generalization to unseen environments.
comment : 4 tables, 9 figures
☆ Agent ic AI Framework for Cloudburst Prediction and Coordinated Response
Toqeer Ali Syed, Sohail Khan, Salman Jan, Gohar Ali, Muhammad Nauman, Ali Akarma, Ahmad Ali
The challenge is growing towards extreme and short-duration rainfall events like a cloudburst that are peculiar to the traditional forecasting systems, in which the predictions and the response are taken as two distinct processes. The paper outlines an agentic artificial intelligence system to study atmospheric water-cycle intelligence, which combines sensing, forecasting, downscaling, hydrological modeling and coordinated response into a single, interconnected, priceless, closed-loop system. The framework uses autonomous but cooperative agents that reason, sense, and act throughout the entire event lifecycle, and use the intelligence of weather prediction to become real-time decision intelligence. Comparison of multi-year radar, satellite, and ground-based evaluation of the northern part of Pakistan demonstrates that the multi-agent configuration enhances forecast reliability, critical success index and warning lead time compared to the baseline models. Population reach was maximised, and errors during evacuation were minimised through communication and routing agents, and adaptive recalibration and transparent auditability were provided by the embedded layer of learning. Collectively, this leads to the conclusion that collaborative AI agents are capable of transforming atmospheric data streams into practicable foresight and provide a platform of scalable adaptive and learning-based climate resilience.
comment : Presented at International Conference on Business and Digital Technology, Bahrain, Springer Nature, 27 November 2025
☆ MammoRGB: Dual-View Mammogram Synthesis Using Denoising Diffusion Probabilistic Models
Jorge Alberto Garza-Abdala, Gerardo A. Fumagal-González, Daly Avendano, Servando Cardona, Sadam Hussain, Eduardo de Avila-Armenta, Jasiel H. Toscano-Martínez, Diana S. M. Rosales Gurmendi, Alma A. Pedro-Pérez, Jose Gerardo Tamez-Pena
Purpose: This study aims to develop and evaluate a three channel denoising diffusion probabilistic model (DDPM) for synthesizing single breast dual view mammograms and to assess the impact of channel representations on image fidelity and cross view consistency. Materials and Methods: A pretrained three channel DDPM, sourced from Hugging Face, was fine tuned on a private dataset of 11020 screening mammograms to generate paired craniocaudal (CC) and mediolateral oblique (MLO) views. Three third channel encodings of the CC and MLO views were evaluated: sum, absolute difference, and zero channel. Each model produced 500 synthetic image pairs. Quantitative assessment involved breast mask segmentation using Intersection over Union (IoU) and Dice Similarity Coefficient (DSC), with distributional comparisons against 2500 real pairs using Earth Movers Distance (EMD) and Kolmogorov Smirnov (KS) tests. Qualitative evaluation included a visual Turing test by a non expert radiologist to assess cross view consistency and artifacts. Results: Synthetic mammograms showed IoU and DSC distributions comparable to real images, with EMD and KS values (0.020 and 0.077 respectively). Models using sum or absolute difference encodings outperformed others in IoU and DSC (p < 0.001), though distributions remained broadly similar. Generated CC and MLO views maintained cross view consistency, with 6 to 8 percent of synthetic images exhibiting artifacts consistent with those in the training data. Conclusion: Three channel DDPMs can generate realistic and anatomically consistent dual view mammograms with promising applications in dataset augmentation.
☆ Exact Learning of Arithmetic with Differentiable Agent s NeurIPS 2025
Hristo Papazov, Francesco D'Angelo, Nicolas Flammarion
We explore the possibility of exact algorithmic learning with gradient-based methods and introduce a differentiable framework capable of strong length generalization on arithmetic tasks. Our approach centers on Differentiable Finite-State Transducers (DFSTs), a Turing-complete model family that avoids the pitfalls of prior architectures by enabling constant-precision, constant-time generation, and end-to-end log-parallel differentiable training. Leveraging policy-trajectory observations from expert agents, we train DFSTs to perform binary and decimal addition and multiplication. Remarkably, models trained on tiny datasets generalize without error to inputs thousands of times longer than the training examples. These results show that training differentiable agents on structured intermediate supervision could pave the way towards exact gradient-based learning of algorithmic skills. Code available at \href{https://github.com/dngfra/differentiable-exact-algorithmic-learner.git}{https://github.com/dngfra/differentiable-exact-algorithmic-learner.git}.
comment : Accepted at the 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop: MATH-AI
☆ VeriDispatcher: Multi-Model Dispatching through Pre-Inference Difficulty Prediction for RTL Generation Optimization
Zeng Wang, Weihua Xiao, Minghao Shao, Raghu Vamshi Hemadri, Ozgur Sinanoglu, Muhammad Shafique, Ramesh Karri
Large Language Models (LLMs) show strong performance in RTL generation, but different models excel on different tasks because of architecture and training differences. Prior work mainly prompts or finetunes a single model. What remains not well studied is how to coordinate multiple different LLMs so they jointly improve RTL quality while also reducing cost, instead of running all models and choosing the best output. We define this as the multi-LLM RTL generation problem. We propose VeriDispatcher, a multi-LLM RTL generation framework that dispatches each RTL task to suitable LLMs based on pre-inference difficulty prediction. For each model, we train a compact classifier over semantic embeddings of task descriptions, using difficulty scores derived from benchmark variants that combine syntax, structural similarity, and functional correctness. At inference, VeriDispatcher uses these predictors to route tasks to a selected subset of LLMs. Across 10 diverse LLMs on RTLLM and VerilogEval, VeriDispatcher achieves up to 18% accuracy improvement on RTLLM using only 40% of commercial calls, and on VerilogEval maintains accuracy while reducing commercial usage by 25%, enabling cost-effective, high-quality LLM deployment in hardware design automation.
☆ All Centers Are at most a Few Tokens Apart: Knowledge Distillation with Domain Invariant Prompt Tuning
Amir Mohammad Ezzati, Alireza Malekhosseini, Armin Khosravi, Mohammad Hossein Rohban
Domain generalization is critical in computational pathology (CPath) due to inherent domain shifts caused by variations in staining protocols, scanner devices, and imaging settings across clinical centers. Vision-language models (VLMs), such as PLIP-a pathology-tuned CLIP-trained on image-text pairs across diverse domains, serve as strong knowledge distillation sources. However, their zero-shot performance with predefined prompts remains limited due to sensitivity to prompt variations. Moreover, unlike natural images, histopathology centers lack semantic descriptors (e.g., 'sketch'), making it difficult to define domain-specific prompts for clinical centers. This requires a data-driven approach for learning domain-specific and ultimately class-generic continuous prompts. We propose Domain Invariant Prompt Tuning (DIPT) for knowledge distillation process, a novel step that learns multiple input tokens for each domain. These tokens are trained separately for each domain and are averaged across domains, leading to domain-invariant prompts. Our student model then distills knowledge from PLIP's text encoder by leveraging the prompts learned by DIPT. This leads to alignment of visual features with domain-invariant embeddings, enhancing generalization by training on multiple domains. Our method adds a significant improvement in average F1-score to existing state-of-the-art (SOTA) knowledge distillation approaches in domain generalization with histopathology datasets. This work helps the way of deploying robust CPath models in real-world clinical problems with heterogeneous data sources.
☆ Agent ic AI Framework for Individuals with Disabilities and Neurodivergence: A Multi- Agent System for Healthy Eating, Daily Routines, and Inclusive Well-Being
Salman Jan, Toqeer Ali Syed, Gohar Ali, Ali Akarma, Mohammad Riyaz Belgaum, Ahmad Ali
The paper presents a detailed Agentic Artificial Intelligence (AI) model that would enable people with disabilities and neurodivergence to lead healthier lives and have more regular days. The system will use a multi-layer structure; it will include an Application and Interface Layer, an Agents Layer, and a Data Source Layer to provide adaptive, transparent, and inclusive support. Fundamentally, a hybrid reasoning engine will synchronize four special-purpose agents, which include: a personalized-nutrition-based, called a Meal Planner Agent; an adaptive-scheduling-based, called a Reminder Agent; interactive assistance during grocery shopping and cooking, called a Food Guidance Agent; and a continuous-intake-and-physiological-tracking, called a Monitoring Agent. All the agents interact through a central communicative system called the Blackboard/Event Bus, which allows autonomous interaction and real-time feedback loops with multimedia user interfaces. Privacy-sensitive data sources, including electronic health records (EHRs), nutritional databases, wearable sensors, and smart kitchen Internet of Things, are also included in the framework and placed into a policy-controlled layer, which ensures data safety and compliance with consent. Collaborative care and clinician dashboards allow common supervision, and discussable artificial intelligence (XAI) modules give brief explanations of why a decision was made, making users responsible and reliant. The proposed agentic AI framework is an extension beyond traditional assistive systems since it incorporates inclusiveness, personalization, and accessibility at all levels. It displays the intersection of multi-agent reasoning, multi-modal interfaces, and human-centered design that will enable the development of autonomy, health, and digital equity among people with disabilities and neurodivergence.
comment : Presented at International Conference on Business and Digital Technology, Bahrain, Springer Nature, 27 November 2025
☆ Solving Context Window Overflow in AI Agent s
Anton Bulle Labate, Valesca Moura de Sousa, Sandro Rama Fiorini, Leonardo Guerreiro Azevedo, Raphael Melo Thiago, Viviane Torres da Silva
Large Language Models (LLMs) have become increasingly capable of interacting with external tools, granting access to specialized knowledge beyond their training data - critical in dynamic, knowledge-intensive domains such as Chemistry and Materials Science. However, large tool outputs can overflow the LLMs' context window, preventing task completion. Existing solutions such as truncation or summarization fail to preserve complete outputs, making them unsuitable for workflows requiring the full data. This work introduces a method that enables LLMs to process and utilize tool responses of arbitrary length without loss of information. By shifting the model's interaction from raw data to memory pointers, the method preserves tool functionality, allows seamless integration into agentic workflows, and reduces token usage and execution time. The proposed method is validated on a real-world Materials Science application that cannot be executed with conventional workflows, and its effectiveness is demonstrated via a comparative analysis where both methods succeed. In this experiment, the proposed approach consumed approximately seven times fewer tokens than the traditional workflow.
☆ ReAG: Reasoning -Augmented Generation for Knowledge-based Visual Question Answering
Alberto Compagnoni, Marco Morini, Sara Sarto, Federico Cocchi, Davide Caffagni, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
Multimodal Large Language Models (MLLMs) have shown impressive capabilities in jointly understanding text, images, and videos, often evaluated via Visual Question Answering (VQA). However, even state-of-the-art MLLMs struggle with domain-specific or knowledge-intensive queries, where relevant information is underrepresented in pre-training data. Knowledge-based VQA (KB-VQA) addresses this by retrieving external documents to condition answer generation, but current retrieval-augmented approaches suffer from low precision, noisy passages, and limited reasoning. To address this, we propose ReAG, a novel Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages, ensuring high-quality additional context. The model follows a multi-stage training strategy leveraging reinforcement learning to enhance reasoning over retrieved content, while supervised fine-tuning serves only as a cold start. Extensive experiments on Encyclopedic-VQA and InfoSeek demonstrate that ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence. Our source code is publicly available at: https://github.com/aimagelab/ReAG.
☆ CoFiRec: Coarse-to-Fine Tokenization for Generative Recommendation
Tianxin Wei, Xuying Ning, Xuxing Chen, Ruizhong Qiu, Yupeng Hou, Yan Xie, Shuang Yang, Zhigang Hua, Jingrui He
In web environments, user preferences are often refined progressively as users move from browsing broad categories to exploring specific items. However, existing generative recommenders overlook this natural refinement process. Generative recommendation formulates next-item prediction as autoregressive generation over tokenized user histories, where each item is represented as a sequence of discrete tokens. Prior models typically fuse heterogeneous attributes such as ID, category, title, and description into a single embedding before quantization, which flattens the inherent semantic hierarchy of items and fails to capture the gradual evolution of user intent during web interactions. To address this limitation, we propose CoFiRec, a novel generative recommendation framework that explicitly incorporates the Coarse-to-Fine nature of item semantics into the tokenization process. Instead of compressing all attributes into a single latent space, CoFiRec decomposes item information into multiple semantic levels, ranging from high-level categories to detailed descriptions and collaborative filtering signals. Based on this design, we introduce the CoFiRec Tokenizer, which tokenizes each level independently while preserving structural order. During autoregressive decoding, the language model is instructed to generate item tokens from coarse to fine, progressively modeling user intent from general interests to specific item-level interests. Experiments across multiple public benchmarks and backbones demonstrate that CoFiRec outperforms existing methods, offering a new perspective for generative recommendation. Theoretically, we prove that structured tokenization leads to lower dissimilarity between generated and ground truth items, supporting its effectiveness in generative recommendation. Our code is available at https://github.com/YennNing/CoFiRec.
☆ Probabilistic Fusion and Calibration of Neural Speaker Diarization Models
Juan Ignacio Alvarez-Trejos, Sergio A. Balanya, Daniel Ramos, Alicia Lozano-Diez
End-to-End Neural Diarization (EEND) systems produce frame-level probabilistic speaker activity estimates, yet since evaluation focuses primarily on Diarization Error Rate (DER), the reliability and calibration of these confidence scores have been largely neglected. When fusing multiple diarization systems, DOVER-Lap remains the only established approach, operating at the segment level with hard decisions. We propose working with continuous probability outputs, which enables more sophisticated calibration and fusion techniques that can leverage model uncertainty and complementary strengths across different architectures. This paper presents the first comprehensive framework for calibrating and fusing EEND models at the probability level. We investigate two output formulations (multilabel and powerset representations) and their impact on calibration and fusion effectiveness. Through extensive experiments on the CallHome two-speaker benchmark, we demonstrate that proper calibration provides substantial improvements even for individual models (up to 19% relative DER reduction), in some cases mitigating the absence of domain adaptation. We reveal that joint calibration in powerset space consistently outperforms independent per-speaker calibration, and that the Fuse-then-Calibrate ordering generally outperforms calibrating individual models before fusion while requiring calibration of only a single combined model. Our best configuration outperforms DOVER-Lap in terms of DER while providing reliable confidence estimates essential for downstream applications. This work proposes best practices for probability-level fusion of EEND systems and demonstrates the advantages of leveraging soft outputs over hard decisions.
☆ Test-time scaling of diffusion s with flow maps
Amirmojtaba Sabour, Michael S. Albergo, Carles Domingo-Enrich, Nicholas M. Boffi, Sanja Fidler, Karsten Kreis, Eric Vanden-Eijnden
A common recipe to improve diffusion models at test-time so that samples score highly against a user-specified reward is to introduce the gradient of the reward into the dynamics of the diffusion itself. This procedure is often ill posed, as user-specified rewards are usually only well defined on the data distribution at the end of generation. While common workarounds to this problem are to use a denoiser to estimate what a sample would have been at the end of generation, we propose a simple solution to this problem by working directly with a flow map. By exploiting a relationship between the flow map and velocity field governing the instantaneous transport, we construct an algorithm, Flow Map Trajectory Tilting (FMTT), which provably performs better ascent on the reward than standard test-time methods involving the gradient of the reward. The approach can be used to either perform exact sampling via importance weighting or principled search that identifies local maximizers of the reward-tilted distribution. We demonstrate the efficacy of our approach against other look-ahead techniques, and show how the flow map enables engagement with complicated reward functions that make possible new forms of image editing, e.g. by interfacing with vision language models.
☆ Foundations of Quantum Granular Computing with Effect-Based Granules, Algebraic Properties and Reference Architectures
Oscar Montiel Ross
This paper develops the foundations of Quantum Granular Computing (QGC), extending classical granular computing including fuzzy, rough, and shadowed granules to the quantum regime. Quantum granules are modeled as effects on a finite dimensional Hilbert space, so granular memberships are given by Born probabilities. This operator theoretic viewpoint provides a common language for sharp (projective) and soft (nonprojective) granules and embeds granulation directly into the standard formalism of quantum information theory. We establish foundational results for effect based quantum granules, including normalization and monotonicity properties, the emergence of Boolean islands from commuting families, granular refinement under Luders updates, and the evolution of granules under quantum channels via the adjoint channel in the Heisenberg picture. We connect QGC with quantum detection and estimation theory by interpreting the effect operators realizing Helstrom minimum error measurement for binary state discrimination as Helstrom type decision granules, i.e., soft quantum counterparts of Bayes optimal decision regions. Building on these results, we introduce Quantum Granular Decision Systems (QGDS) with three reference architectures that specify how quantum granules can be defined, learned, and integrated with classical components while remaining compatible with near term quantum hardware. Case studies on qubit granulation, two qubit parity effects, and Helstrom style soft decisions illustrate how QGC reproduces fuzzy like graded memberships and smooth decision boundaries while exploiting noncommutativity, contextuality, and entanglement. The framework thus provides a unified and mathematically grounded basis for operator valued granules in quantum information processing, granular reasoning, and intelligent systems.
comment : Three figures and the graphical abstract
☆ Geometrically-Constrained Agent for Spatial Reasoning
Zeren Chen, Xiaoya Lu, Zhijie Zheng, Pengrui Li, Lehan He, Yijin Zhou, Jing Shao, Bohan Zhuang, Lu Sheng
Vision Language Models (VLMs) exhibit a fundamental semantic-to-geometric gap in spatial reasoning: they excel at qualitative semantic inference but their reasoning operates within a lossy semantic space, misaligned with high-fidelity geometry. Current paradigms fail to bridge this gap. Training-based methods suffer from an ``oracle paradox,'' learning flawed spatial logic from imperfect oracles. Tool-integrated methods constrain the final computation but critically leave the VLM's planning process unconstrained, resulting in geometrically flawed plans. In this work, we propose Geometrically-Constrained Agent (GCA), a training-free agentic paradigm that resolves this gap by introducing a formal task constraint. Specifically, we strategically decouples the VLM's role into two stages. First, acting as a semantic analyst, the VLM translates the user's ambiguous query into the formal, verifiable task constraint, which defines the reference frame and objective. Second, acting as a task solver, the VLM generates and executes tool calls strictly within the deterministic bounds defined by the constraint. This geometrically-constrained reasoning strategy successfully resolve the semantic-to-geometric gap, yielding a robust and verifiable reasoning pathway for spatial reasoning. Comprehensive experiments demonstrate that GCA achieves SOTA performance on multiple spatial reasoning benchmarks, surpassing existing training-based and tool-integrated methods by ~27%. Please see our homepage at https://gca-spatial-reasoning.github.io.
comment : 27 pages, 13 figures
☆ Automated Design Optimization via Strategic Search with Large Language Models
Anthony Carreon, Vansh Sharma, Venkat Raman
Traditional optimization methods excel in well-defined search spaces but struggle with design problems where transformations and design parameters are difficult to define. Large language models (LLMs) offer a promising alternative by dynamically interpreting design spaces and leveraging encoded domain knowledge. To this end, we introduce AUTO, an LLM agent framework that treats design optimization as a gradient-free search problem guided by strategic LLM reasoning. The framework employs two collaborative agents: a Strategist that selects between exploration and exploitation strategies, and an Implementor that executes detailed designs. Applied to GPU code optimization -- a domain critical to fields from machine learning to scientific computing -- AUTO generates solutions competitive with expert implementations for chemical kinetics integration and dense matrix multiplication. The framework achieves 50-70% search efficiency relative to Bayesian optimization methodologies. It completes optimizations in approximately 8 hours at an estimated cost of up to \$159 per run, compared to an estimated cost of up to \$480 with median-wage software developers. These findings open the door to automating design optimization in ill-defined search spaces with limited prior information.
comment : 14 pages, 5 tables, 7 figures, preprint
☆ Optimized Agent Shift Scheduling Using Multi-Phase Allocation Approach
Sanalkumar K, Koushik Dey, Swati Meena
Effective agent shift scheduling is crucial for businesses, especially in the Contact Center as a Service (CCaaS) industry, to ensure seamless operations and fulfill employee needs. Most studies utilizing mathematical model-based solutions approach the problem as a single-step process, often resulting in inefficiencies and high computational demands. In contrast, we present a multi-phase allocation method that addresses scalability and accuracy by dividing the problem into smaller sub-problems of day and shift allocation, which significantly reduces number of computational variables and allows for targeted objective functions, ultimately enhancing both efficiency and accuracy. Each subproblem is modeled as a Integer Programming Problem (IPP), with solutions sequentially feeding into the subsequent subproblem. We then apply the proposed method, using a multi-objective framework, to address the difficulties posed by peak demand scenarios such as holiday rushes, where maintaining service levels is essential despite having limited number of employees
comment : 5 pages, 3 figures
☆ AI Deception: Risks, Dynamics, and Controls
Boyuan Chen, Sitong Fang, Jiaming Ji, Yanxu Zhu, Pengcheng Wen, Jinzhou Wu, Yingshui Tan, Boren Zheng, Mengying Yuan, Wenqi Chen, Donghai Hong, Alex Qiu, Xin Chen, Jiayi Zhou, Kaile Wang, Juntao Dai, Borong Zhang, Tianzhuo Yang, Saad Siddiqui, Isabella Duan, Yawen Duan, Brian Tse, Jen-Tse, Huang, Kun Wang, Baihui Zheng, Jiaheng Liu, Jian Yang, Yiming Li, Wenting Chen, Dongrui Liu, Lukas Vierling, Zhiheng Xi, Haobo Fu, Wenxuan Wang, Jitao Sang, Zhengyan Shi, Chi-Min Chan, Eugenie Shi, Simin Li, Juncheng Li, Wei Ji, Dong Li, Jun Song, Yinpeng Dong, Jie Fu, Bo Zheng, Min Yang, Yike Guo, Philip Torr, Zhongyuan Wang, Yaodong Yang, Tiejun Huang, Ya-Qin Zhang, Hongjiang Zhang, Andrew Yao
As intelligence increases, so does its shadow. AI deception, in which systems induce false beliefs to secure self-beneficial outcomes, has evolved from a speculative concern to an empirically demonstrated risk across language models, AI agents, and emerging frontier systems. This project provides a comprehensive and up-to-date overview of the AI deception field, covering its core concepts, methodologies, genesis, and potential mitigations. First, we identify a formal definition of AI deception, grounded in signaling theory from studies of animal deception. We then review existing empirical studies and associated risks, highlighting deception as a sociotechnical safety challenge. We organize the landscape of AI deception research as a deception cycle, consisting of two key components: deception emergence and deception treatment. Deception emergence reveals the mechanisms underlying AI deception: systems with sufficient capability and incentive potential inevitably engage in deceptive behaviors when triggered by external conditions. Deception treatment, in turn, focuses on detecting and addressing such behaviors. On deception emergence, we analyze incentive foundations across three hierarchical levels and identify three essential capability preconditions required for deception. We further examine contextual triggers, including supervision gaps, distributional shifts, and environmental pressures. On deception treatment, we conclude detection methods covering benchmarks and evaluation protocols in static and interactive settings. Building on the three core factors of deception emergence, we outline potential mitigation strategies and propose auditing approaches that integrate technical, community, and governance efforts to address sociotechnical challenges and future AI risks. To support ongoing work in this area, we release a living resource at www.deceptionsurvey.com.
☆ Variational analysis of determinantal varieties
Yan Yang, Bin Gao, Ya-xiang Yuan
Determinantal varieties -- the sets of bounded-rank matrices or tensors -- have attracted growing interest in low-rank optimization. The tangent cone to low-rank sets is widely studied and underpins a range of geometric methods. The second-order geometry, which encodes curvature information, is more intricate. In this work, we develop a unified framework to derive explicit formulas for both first- and second-order tangent sets to various low-rank sets, including low-rank matrices, tensors, symmetric matrices, and positive semidefinite matrices. The framework also accommodates the intersection of a low-rank set and another set satisfying mild assumptions, thereby yielding a tangent intersection rule. Through the lens of tangent sets, we establish a necessary and sufficient condition under which a nonsmooth problem and its smooth parameterization share equivalent second-order stationary points. Moreover, we exploit tangent sets to characterize optimality conditions for low-rank optimization and prove that verifying second-order optimality is NP-hard. In a separate line of analysis, we investigate variational geometry of the graph of the normal cone to matrix varieties, deriving the explicit Bouligand tangent cone, Fréchet and Mordukhovich normal cones to the graph. These results are further applied to develop optimality conditions for low-rank bilevel programs.
comment : 71 pages, 6 figures, 2 tables
☆ GazeTrack: High-Precision Eye Tracking Based on Regularization and Spatial Computing
Xiaoyin Yang
Eye tracking has become increasingly important in virtual and augmented reality applications; however, the current gaze accuracy falls short of meeting the requirements for spatial computing. We designed a gaze collection framework and utilized high-precision equipment to gather the first precise benchmark dataset, GazeTrack, encompassing diverse ethnicities, ages, and visual acuity conditions for pupil localization and gaze tracking. We propose a novel shape error regularization method to constrain pupil ellipse fitting and train on open-source datasets, enhancing semantic segmentation and pupil position prediction accuracy. Additionally, we invent a novel coordinate transformation method similar to paper unfolding to accurately predict gaze vectors on the GazeTrack dataset. Finally, we built a gaze vector generation model that achieves reduced gaze angle error with lower computational complexity compared to other methods.
comment : 10 pages, 7 figures
☆ HarmoCLIP: Harmonizing Global and Regional Representations in Contrastive Vision -Language Models
Haoxi Zeng, Haoxuan Li, Yi Bin, Pengpeng Zeng, Xing Xu, Yang Yang, Heng Tao Shen
Contrastive Language-Image Pre-training (CLIP) has demonstrated remarkable generalization ability and strong performance across a wide range of vision-language tasks. However, due to the lack of region-level supervision, CLIP exhibits limited fine-grained semantic understanding. Although several methods attempt to mitigate this issue, they unintentionally disrupt the global alignment, resulting in a persistent trade-off where improving local perception simultaneously degrades global coherence. In this paper, we propose HarmoCLIP, a novel framework designed to harmonize global and region representations within CLIP. We first identify that the absence of direct alignment between local textual and visual semantics is the fundamental cause of the trade-off. To address this, HarmoCLIP introduces an explicit fine-grained semantic supervision term that directly aligns textual segments with their corresponding visual regions, effectively bridging the image region space and the textual space. To further strengthen the representation capability at the local level, our method introduces a novel Region-Language Alignment supervision strategy that promotes fine-grained semantic learning without compromising global semantic consistency. Extensive experiments demonstrate that HarmoCLIP achieves state-of-the-art (improvement up to 69.78%) performance on the global task of retrieval and yields a substantial 3.2% improvement in Top-1 accuracy on the region task of bounding-box classification, consistently outperforming prior approaches while providing a balanced, efficient, and plug-and-play solution to the global-local trade-off in CLIP. Code is available at https://github.com/Erosist/HarmoCLIP.
comment : 13 pages, 7 figures, 6 tables
☆ Revisiting the Necessity of Lengthy Chain-of-Thought in Vision -centric Reasoning Generalization
Yifan Du, Kun Zhou, Yingqian Min, Yue Ling, Wayne Xin Zhao, Youbin Wu
We study how different Chain-of-Thought (CoT) designs affect the acquisition of the generalizable visual reasoning ability in vision-language models (VLMs). While CoT data, especially long or visual CoT such as "think with image", has been widely used to supervise intermediate reasoning, it remains unclear why specific CoT designs help and which ones truly support generalizable reasoning. To systematically evaluate this, we focus on a controlled maze-solving benchmark where reasoning rules are fully visual, difficulty can be tuned by grid size, and all the intermediate steps can be automatically generated. Using Qwen2.5-VL-7B under a standard SFT-then-RL pipeline, we compare three representative CoT formats: Language CoT, Grounding CoT (with spatial coordinate trajectories), and Visual CoT (with image manipulations). Our experiments reveal that visual and longer CoT mainly accelerate convergence but do not lift the final performance ceiling; concise CoT containing only essential grounding steps outperforms longer traces; and, strikingly, CoT retaining only the minimal grounding results generalizes best across different maze sizes. We further validate these insights on other vision-centric tasks. These findings highlight a "short is long" effect and provide practical guidance for constructing more generalizable SFT datasets for visual reasoning.
☆ DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning
Zhihong Shao, Yuxiang Luo, Chengda Lu, Z. Z. Ren, Jiewen Hu, Tian Ye, Zhibin Gou, Shirong Ma, Xiaokang Zhang
Large language models have made significant progress in mathematical reasoning, which serves as an important testbed for AI and could impact scientific research if further advanced. By scaling reasoning with reinforcement learning that rewards correct final answers, LLMs have improved from poor performance to saturating quantitative reasoning competitions like AIME and HMMT in one year. However, this approach faces fundamental limitations. Pursuing higher final answer accuracy doesn't address a key issue: correct answers don't guarantee correct reasoning. Moreover, many mathematical tasks like theorem proving require rigorous step-by-step derivation rather than numerical answers, making final answer rewards inapplicable. To push the limits of deep reasoning, we believe it is necessary to verify the comprehensiveness and rigor of mathematical reasoning. Self-verification is particularly important for scaling test-time compute, especially for open problems without known solutions. Towards self-verifiable mathematical reasoning, we investigate how to train an accurate and faithful LLM-based verifier for theorem proving. We then train a proof generator using the verifier as the reward model, and incentivize the generator to identify and resolve as many issues as possible in their own proofs before finalizing them. To maintain the generation-verification gap as the generator becomes stronger, we propose to scale verification compute to automatically label new hard-to-verify proofs, creating training data to further improve the verifier. Our resulting model, DeepSeekMath-V2, demonstrates strong theorem-proving capabilities, achieving gold-level scores on IMO 2025 and CMO 2024 and a near-perfect 118/120 on Putnam 2024 with scaled test-time compute.
☆ Where to Measure: Epistemic Uncertainty-Based Sensor Placement with ConvCNPs
Feyza Eksen, Stefan Oehmcke, Stefan Lüdtke
Accurate sensor placement is critical for modeling spatio-temporal systems such as environmental and climate processes. Neural Processes (NPs), particularly Convolutional Conditional Neural Processes (ConvCNPs), provide scalable probabilistic models with uncertainty estimates, making them well-suited for data-driven sensor placement. However, existing approaches rely on total predictive uncertainty, which conflates epistemic and aleatoric components, that may lead to suboptimal sensor selection in ambiguous regions. To address this, we propose expected reduction in epistemic uncertainty as a new acquisition function for sensor placement. To enable this, we extend ConvCNPs with a Mixture Density Networks (MDNs) output head for epistemic uncertainty estimation. Preliminary results suggest that epistemic uncertainty driven sensor placement more effectively reduces model error than approaches based on overall uncertainty.
☆ Counting Still Counts: Understanding Neural Complex Query Answering Through Query Relaxation
Yannick Brunink, Daniel Daza, Yunjie He, Michael Cochez
Neural methods for Complex Query Answering (CQA) over knowledge graphs (KGs) are widely believed to learn patterns that generalize beyond explicit graph structure, allowing them to infer answers that are unreachable through symbolic query processing. In this work, we critically examine this assumption through a systematic analysis comparing neural CQA models with an alternative, training-free query relaxation strategy that retrieves possible answers by relaxing query constraints and counting resulting paths. Across multiple datasets and query structures, we find several cases where neural and relaxation-based approaches perform similarly, with no neural model consistently outperforming the latter. Moreover, a similarity analysis reveals that their retrieved answers exhibit little overlap, and that combining their outputs consistently improves performance. These results call for a re-evaluation of progress in neural query answering: despite their complexity, current models fail to subsume the reasoning patterns captured by query relaxation. Our findings highlight the importance of stronger non-neural baselines and suggest that future neural approaches could benefit from incorporating principles of query relaxation.
☆ A Computable Game-Theoretic Framework for Multi- Agent Theory of Mind AAAI 2026
Fengming Zhu, Yuxin Pan, Xiaomeng Zhu, Fangzhen Lin
Originating in psychology, $\textit{Theory of Mind}$ (ToM) has attracted significant attention across multiple research communities, especially logic, economics, and robotics. Most psychological work does not aim at formalizing those central concepts, namely $\textit{goals}$, $\textit{intentions}$, and $\textit{beliefs}$, to automate a ToM-based computational process, which, by contrast, has been extensively studied by logicians. In this paper, we offer a different perspective by proposing a computational framework viewed through the lens of game theory. On the one hand, the framework prescribes how to make boudedly rational decisions while maintaining a theory of mind about others (and recursively, each of the others holding a theory of mind about the rest); on the other hand, it employs statistical techniques and approximate solutions to retain computability of the inherent computational problem.
comment : Ongoing work. A preliminary version has been accepted by the AAAI 2026 Theory of Mind for AI (ToM4AI) Workshop
☆ CoT 4AD: A Vision -Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving
Zhaohui Wang, Tengbo Yu, Hao Tang
Vision-Language-Action (VLA) models have recently attracted growing attention in end-to-end autonomous driving for their strong reasoning capabilities and rich world knowledge. However, existing VLAs often suffer from limited numerical reasoning ability and overly simplified input-output mappings, which hinder their performance in complex driving scenarios requiring step-by-step causal reasoning. To address these challenges, we propose CoT4AD, a novel VLA framework that introduces Chain-of-Thought (CoT) reasoning for autonomous driving to enhance both numerical and causal reasoning in Vision-Language Models (VLMs). CoT4AD integrates visual observations and language instructions to perform semantic reasoning, scene understanding, and trajectory planning. During training, it explicitly models a perception-question-prediction-action CoT to align the reasoning space with the action space across multiple driving tasks. During inference, it performs implicit CoT reasoning to enable consistent numerical reasoning and robust decision-making in dynamic environments. Extensive experiments on both real-world and simulated benchmarks, including nuScenes and Bench2Drive, demonstrate that CoT4AD achieves state-of-the-art performance in both open-loop and closed-loop evaluations. Code will be released upon paper acceptance.
comment : 10 pages, 3 figures
☆ DocVAL: Validated Chain-of-Thought Distillation for Grounded Document VQA
Ahmad Mohammadshirazi, Pinaki Prasad Guha Neogi, Dheeraj Kulshrestha, Rajiv Ramnath
Document visual question answering (DocVQA) requires models to jointly reason over textual content and spatial layout, yet current systems exhibit a sharp accuracy--efficiency trade-off: large teacher models achieve strong grounding but are too expensive for deployment, while compact students suffer substantial drops in localization performance. We propose DocVAL, a validated chain-of-thought distillation framework that transfers the spatial reasoning ability of a large teacher into a deployable student VLM through three key components: (1) teacher supervision with validation-time text detection to filter and denoise training signals, (2) a multi-module validator (VAL) that enforces answer correctness and geometric consistency while producing fine-grained, pixel-level error feedback, and (3) a two-stage student training scheme that first learns from validated CoT traces and then undergoes iterative refinement driven by VAL feedback. Our student (Gemma-3 12B) achieves 91.4\% ANLS and 82.4\% mAP on DocVQA as a pure VLM requiring no text detection or OCR at inference. Extensive ablations demonstrate that validated feedback contributes 6.3 mAP gain and iterative refinement accounts for 9.7 mAP improvement. We release 95k high-quality, validator-verified CoT traces to advance spatial reasoning research in document understanding.
☆ HW-GNN: Homophily-Aware Gaussian-Window Constrained Graph Spectral Network for Social Network Bot Detection
Zida Liu, Jun Gao, Zhang Ji, Li Zhao
Social bots are increasingly polluting online platforms by spreading misinformation and engaging in coordinated manipulation, posing severe threats to cybersecurity. Graph Neural Networks (GNNs) have become mainstream for social bot detection due to their ability to integrate structural and attribute features, with spectral-based approaches demonstrating particular efficacy due to discriminative patterns in the spectral domain. However, current spectral GNN methods face two limitations: (1) their broad-spectrum fitting mechanisms degrade the focus on bot-specific spectral features, and (2) certain domain knowledge valuable for bot detection, e.g., low homophily correlates with high-frequency features, has not been fully incorporated into existing methods. To address these challenges, we propose HW-GNN, a novel homophily-aware graph spectral network with Gaussian window constraints. Our framework introduces two key innovations: (i) a Gaussian-window constrained spectral network that employs learnable Gaussian windows to highlight bot-related spectral features, and (ii) a homophily-aware adaptation mechanism that injects domain knowledge between homophily ratios and frequency features into the Gaussian window optimization process. Through extensive experimentation on multiple benchmark datasets, we demonstrate that HW-GNN achieves state-of-the-art bot detection performance, outperforming existing methods with an average improvement of 4.3% in F1-score, while exhibiting strong plug-in compatibility with existing spectral GNNs.
☆ Exploring Performance Variations in Finetuned Translators of Ultra-Low Resource Languages: Do Linguistic Differences Matter?
Isabel Gonçalves, Paulo Cavalin, Claudio Pinhanez
Finetuning pre-trained language models with small amounts of data is a commonly-used method to create translators for ultra-low resource languages such as endangered Indigenous languages. However, previous works have reported substantially different performances with translators created using similar methodology and data. In this work we systematically explored possible causes of the performance difference, aiming to determine whether it was a product of different cleaning procedures, limitations of the pre-trained models, the size of the base model, or the size of the training dataset, studying both directions of translation. Our studies, using two Brazilian Indigenous languages, related but with significant structural linguistic characteristics, indicated none or very limited influence from those training factors, suggesting differences between languages may play a significant role in the ability to produce translators by fine-tuning pre-trained models.
☆ Structured Extraction from Business Process Diagrams Using Vision -Language Models
Pritam Deka, Barry Devereux
Business Process Model and Notation (BPMN) is a widely adopted standard for representing complex business workflows. While BPMN diagrams are often exchanged as visual images, existing methods primarily rely on XML representations for computational analysis. In this work, we present a pipeline that leverages Vision-Language Models (VLMs) to extract structured JSON representations of BPMN diagrams directly from images, without requiring source model files or textual annotations. We also incorporate optical character recognition (OCR) for textual enrichment and evaluate the generated element lists against ground truth data derived from the source XML files. Our approach enables robust component extraction in scenarios where original source files are unavailable. We benchmark multiple VLMs and observe performance improvements in several models when OCR is used for text enrichment. In addition, we conducted extensive statistical analyses of OCR-based enrichment methods and prompt ablation studies, providing a clearer understanding of their impact on model performance.
comment : To appear in the Proceedings of the 2026 ACM Symposium on Applied Computing (SAC '26)
☆ What Is the Optimal Ranking Score Between Precision and Recall? We Can Always Find It and It Is Rarely $F_1$
Sébastien Piérard, Adrien Deliège, Marc Van Droogenbroeck
Ranking methods or models based on their performance is of prime importance but is tricky because performance is fundamentally multidimensional. In the case of classification, precision and recall are scores with probabilistic interpretations that are both important to consider and complementary. The rankings induced by these two scores are often in partial contradiction. In practice, therefore, it is extremely useful to establish a compromise between the two views to obtain a single, global ranking. Over the last fifty years or so,it has been proposed to take a weighted harmonic mean, known as the F-score, F-measure, or $F_β$. Generally speaking, by averaging basic scores, we obtain a score that is intermediate in terms of values. However, there is no guarantee that these scores lead to meaningful rankings and no guarantee that the rankings are good tradeoffs between these base scores. Given the ubiquity of $F_β$ scores in the literature, some clarification is in order. Concretely: (1) We establish that $F_β$-induced rankings are meaningful and define a shortest path between precision- and recall-induced rankings. (2) We frame the problem of finding a tradeoff between two scores as an optimization problem expressed with Kendall rank correlations. We show that $F_1$ and its skew-insensitive version are far from being optimal in that regard. (3) We provide theoretical tools and a closed-form expression to find the optimal value for $β$ for any distribution or set of performances, and we illustrate their use on six case studies.
☆ GEO-Detective: Unveiling Location Privacy Risks in Images with LLM Agent s
Xinyu Zhang, Yixin Wu, Boyang Zhang, Chenhao Lin, Chao Shen, Michael Backes, Yang Zhang
Images shared on social media often expose geographic cues. While early geolocation methods required expert effort and lacked generalization, the rise of Large Vision Language Models (LVLMs) now enables accurate geolocation even for ordinary users. However, existing approaches are not optimized for this task. To explore the full potential and associated privacy risks, we present Geo-Detective, an agent that mimics human reasoning and tool use for image geolocation inference. It follows a procedure with four steps that adaptively selects strategies based on image difficulty and is equipped with specialized tools such as visual reverse search, which emulates how humans gather external geographic clues. Experimental results show that GEO-Detective outperforms baseline large vision language models (LVLMs) overall, particularly on images lacking visible geographic features. In country level geolocation tasks, it achieves an improvement of over 11.1% compared to baseline LLMs, and even at finer grained levels, it still provides around a 5.2% performance gain. Meanwhile, when equipped with external clues, GEO-Detective becomes more likely to produce accurate predictions, reducing the "unknown" prediction rate by more than 50.6%. We further explore multiple defense strategies and find that Geo-Detective exhibits stronger robustness, highlighting the need for more effective privacy safeguards.
comment : 15 pages with 7 figures and 12 tables
☆ FastFHE: Packing-Scalable and Depthwise-Separable CNN Inference Over FHE
Wenbo Song, Xinxin Fan, Quanliang Jing, Shaoye Luo, Wenqi Wei, Chi Lin, Yunfeng Lu, Ling Liu
The deep learning (DL) has been penetrating daily life in many domains, how to keep the DL model inference secure and sample privacy in an encrypted environment has become an urgent and increasingly important issue for various security-critical applications. To date, several approaches have been proposed based on the Residue Number System variant of the Cheon-Kim-Kim-Song (RNS-CKKS) scheme. However, they all suffer from high latency, which severely limits the applications in real-world tasks. Currently, the research on encrypted inference in deep CNNs confronts three main bottlenecks: i) the time and storage costs of convolution calculation; ii) the time overhead of huge bootstrapping operations; and iii) the consumption of circuit multiplication depth. Towards these three challenges, we in this paper propose an efficient and effective mechanism FastFHE to accelerate the model inference while simultaneously retaining high inference accuracy over fully homomorphic encryption. Concretely, our work elaborates four unique novelties. First, we propose a new scalable ciphertext data-packing scheme to save the time and storage consumptions. Second, we work out a depthwise-separable convolution fashion to degrade the computation load of convolution calculation. Third, we figure out a BN dot-product fusion matrix to merge the ciphertext convolutional layer with the batch-normalization layer without incurring extra multiplicative depth. Last but not least, we adopt the low-degree Legendre polynomial to approximate the nonlinear smooth activation function SiLU under the guarantee of tiny accuracy error before and after encrypted inference. Finally, we execute multi-facet experiments to verify the efficiency and effectiveness of our proposed approach.
☆ MATCH: Engineering Transparent and Controllable Conversational XAI Systems through Composable Building Blocks
Sebe Vanbrabant, Gustavo Rovelo Ruiz, Davy Vanacken
While the increased integration of AI technologies into interactive systems enables them to solve an increasing number of tasks, the black-box problem of AI models continues to spread throughout the interactive system as a whole. Explainable AI (XAI) techniques can make AI models more accessible by employing post-hoc methods or transitioning to inherently interpretable models. While this makes individual AI models clearer, the overarching system architecture remains opaque. This challenge not only pertains to standard XAI techniques but also to human examination and conversational XAI approaches that need access to model internals to interpret them correctly and completely. To this end, we propose conceptually representing such interactive systems as sequences of structural building blocks. These include the AI models themselves, as well as control mechanisms grounded in literature. The structural building blocks can then be explained through complementary explanatory building blocks, such as established XAI techniques like LIME and SHAP. The flow and APIs of the structural building blocks form an unambiguous overview of the underlying system, serving as a communication basis for both human and automated agents, thus aligning human and machine interpretability of the embedded AI models. In this paper, we present our flow-based approach and a selection of building blocks as MATCH: a framework for engineering Multi-Agent Transparent and Controllable Human-centered systems. This research contributes to the field of (conversational) XAI by facilitating the integration of interpretability into existing interactive systems.
comment : Submitted Version accepted for publication in an LNCS Volume "Engineering Interactive Computer Systems - EICS 2025 - International Workshops and Doctoral Consortium"
☆ Mapping Clinical Doubt: Locating Linguistic Uncertainty in LLMs AAAI'26
Srivarshinee Sridhar, Raghav Kaushik Ravi, Kripabandhu Ghosh
Large Language Models (LLMs) are increasingly used in clinical settings, where sensitivity to linguistic uncertainty can influence diagnostic interpretation and decision-making. Yet little is known about where such epistemic cues are internally represented within these models. Distinct from uncertainty quantification, which measures output confidence, this work examines input-side representational sensitivity to linguistic uncertainty in medical text. We curate a contrastive dataset of clinical statements varying in epistemic modality (e.g., 'is consistent with' vs. 'may be consistent with') and propose Model Sensitivity to Uncertainty (MSU), a layerwise probing metric that quantifies activation-level shifts induced by uncertainty cues. Our results show that LLMs exhibit structured, depth-dependent sensitivity to clinical uncertainty, suggesting that epistemic information is progressively encoded in deeper layers. These findings reveal how linguistic uncertainty is internally represented in LLMs, offering insight into their interpretability and epistemic reliability.
comment : Accepted to AAAI'26 SECURE-AI4H Workshop
☆ Asking like S ocr ates: S ocr ates helps VLMs understand remote sensing images
Run Shao, Ziyu Li, Zhaoyang Zhang, Linrui Xu, Xinran He, Hongyuan Yuan, Bolei He, Yongxing Dai, Yiming Yan, Yijun Chen, Wang Guo, Haifeng Li
Recent multimodal reasoning models, inspired by DeepSeek-R1, have significantly advanced vision-language systems. However, in remote sensing (RS) tasks, we observe widespread pseudo reasoning: models narrate the process of reasoning rather than genuinely reason toward the correct answer based on visual evidence. We attribute this to the Glance Effect, where a single, coarse perception of large-scale RS imagery results in incomplete understanding and reasoning based on linguistic self-consistency instead of visual evidence. To address this, we propose RS-EoT (Remote Sensing Evidence-of-Thought), a language-driven, iterative visual evidence-seeking paradigm. To instill this paradigm, we propose SocraticAgent, a self-play multi-agent system that synthesizes reasoning traces via alternating cycles of reasoning and visual inspection. To enhance and generalize these patterns, we propose a two-stage progressive RL strategy: first, RL on fine-grained Grounding tasks to enhance RS-EoT capabilities, followed by RL on RS VQA to generalize to broader understanding scenarios. Experiments show RS-EoT achieves state-of-the-art performance on multiple RS VQA and grounding benchmarks. Analyses reveal clear iterative cycles of reasoning and evidence seeking, confirming RS-EoT mitigates the Glance Effect and enables genuine evidence-grounded reasoning. Our code, data, and models are available at https://geox-lab.github.io/Asking_like_Socrates
comment : 20 pages, 8 figures
☆ Who is Afraid of Minimal Re vision ?
Edoardo Baccini, Zoé Christoff, Nina Gierasimczuk, Rineke Verbrugge
The principle of minimal change in belief revision theory requires that, when accepting new information, one keeps one's belief state as close to the initial belief state as possible. This is precisely what the method known as minimal revision does. However, unlike less conservative belief revision methods, minimal revision falls short in learning power: It cannot learn everything that can be learned by other learning methods. We begin by showing that, despite this limitation, minimal revision is still a successful learning method in a wide range of situations. Firstly, it can learn any problem that is finitely identifiable. Secondly, it can learn with positive and negative data, as long as one considers finitely many possibilities. We then characterize the prior plausibility assignments (over finitely many possibilities) that enable one to learn via minimal revision, and do the same for conditioning and lexicographic upgrade. Finally, we show that not all of our results still hold when learning from possibly erroneous information.
comment : In Proceedings TARK 2025, arXiv:2511.20540
☆ Graded Distributed Belief
Emiliano Lorini, Dmitry Rozplokhas
We introduce a new logic of graded distributed belief that allows us to express the fact that a group of agents distributively believe that a certain fact holds with at least strength k. We interpret our logic by means of computationally grounded semantics relying on the concept of belief base. The strength of the group's distributed belief is directly computed from the group's belief base after having merged its members' individual belief bases. We illustrate our logic with an intuitive example, formalizing the notion of epistemic disagreement. We also provide a sound and complete Hilbert-style axiomatization, decidability result obtained via filtration, and a tableaux-based decision procedure that allows us to state PSPACE-completeness for our logic.
comment : In Proceedings TARK 2025, arXiv:2511.20540
☆ Conditionals Based on Selection Functions, Modal Operators and Probabilities
Tommaso Flaminio, Lluis Godo, Gluliano Rosella
Methods for probability updating, of which Bayesian conditionalization is the most well-known and widely used, are modeling tools that aim to represent the process of modifying an initial epistemic state, typically represented by a prior probability function P, which is adjusted in light of new information. Notably, updating methods and conditional sentences seem to intuitively share a deep connection, as is evident in the case of conditionalization. The present work contributes to this line of research and aims at shedding new light on the relationship between updating methods and conditional connectives. Departing from previous literature that often focused on a specific type of conditional or a particular updating method, our goal is to prove general results concerning the connection between conditionals and their probabilities. This will allow us to characterize the probabilities of certain conditional connectives and to understand what class of updating procedures can be represented using specific conditional connectives. Broadly, we adopt a general perspective that encompasses a large class of conditionals and a wide range of updating methods, enabling us to prove some general results concerning their interrelation.
comment : In Proceedings TARK 2025, arXiv:2511.20540
☆ On the Complexity of the Grounded Semantics for Infinite Argumentation Frameworks
Uri Andrews, Luca San Mauro
Argumentation frameworks, consisting of arguments and an attack relation representing conflicts, are fundamental for formally studying reasoning under conflicting information. We use methods from mathematical logic, specifically computability and set theory, to analyze the grounded extension, a widely-used model of maximally skeptical reasoning, defined as the least fixed-point of a natural defense operator. Without additional constraints, finding this fixed-point requires transfinite iterations. We identify the exact ordinal number corresponding to the length of this iterative process and determine the complexity of deciding grounded acceptance, showing it to be maximally complex. This shows a marked distinction from the finite case where the grounded extension is polynomial-time computable, thus simpler than other reasoning problems explored in formal argumentation.
comment : In Proceedings TARK 2025, arXiv:2511.20540
☆ Distributed Knowing How
Bin Liu, Yanjing Wang
Distributed knowledge is a key concept in the standard epistemic logic of knowledge-that. In this paper, we propose a corresponding notion of distributed knowledge-how and study its logic. Our framework generalizes two existing traditions in the logic of know-how: the individual-based multi-step framework and the coalition-based single-step framework. In particular, we assume a group can accomplish more than what its individuals can jointly do. The distributed knowledge-how is based on the distributed knowledge-that of a group whose multi-step strategies derive from distributed actions that subgroups can collectively perform. As the main result, we obtain a sound and strongly complete proof system for our logic of distributed knowledge-how, which closely resembles the logic of distributed knowledge-that in both the axioms and the proof method of completeness.
comment : In Proceedings TARK 2025, arXiv:2511.20540
☆ SuRe: Surprise-Driven Prioritised Replay for Continual LLM Learning
Hugo Hazard, Zafeirios Fountas, Martin A. Benfeghoul, Adnan Oomerjee, Jun Wang, Haitham Bou-Ammar
Continual learning, one's ability to adapt to a sequence of tasks without forgetting previously acquired knowledge, remains a major challenge in machine learning and a key gap between artificial and human intelligence. While regularisation and replay perform well in vision, they lag behind multi-task learning for large language models (LLMs), especially at scale with many tasks. We revisit replay and argue that two failure modes drive this gap: selection (what to rehearse) and integration (how to consolidate new knowledge). To address selection, we propose Surprise-prioritised Replay (SuRe), a simple, architecture-agnostic rule that ranks and stores the most surprising (high Negative Log-Likelihood) sequences. SuRe achieves state-of-the-art performance in the Large Number of Tasks (LNT) setting and delivers the best overall average across both Standard CL and LNT benchmarks. To address integration, we add a dual-learner design with fast and slow LoRA adapters merged via an exponential moving average (EMA), enabling rapid adaptation while stabilising long-term knowledge. Combining SuRe with the dual learner yields further gains, including improvements of up to +5 accuracy points on LNT over prior SOTA. Ablation studies confirm that our proposed method remains robust under reduced replay frequency and small buffer size, demonstrating both effectiveness and sample efficiency. Taken together, our results establish replay as a strong baseline for continual LLM fine-tuning and demonstrate that surprise-based selection and slow-weight consolidation are complementary components for mitigating catastrophic forgetting.
☆ BINDER: Instantly Adaptive Mobile Manipulation with Open-Vocabulary Commands
Seongwon Cho, Daechul Ahn, Donghyun Shin, Hyeonbeom Choi, San Kim, Jonghyun Choi
Open-vocabulary mobile manipulation (OVMM) requires robots to follow language instructions, navigate, and manipulate while updating their world representation under dynamic environmental changes. However, most prior approaches update their world representation only at discrete update points such as navigation targets, waypoints, or the end of an action step, leaving robots blind between updates and causing cascading failures: overlooked objects, late error detection, and delayed replanning. To address this limitation, we propose BINDER (Bridging INstant and DEliberative Reasoning), a dual process framework that decouples strategic planning from continuous environment monitoring. Specifically, BINDER integrates a Deliberative Response Module (DRM, a multimodal LLM for task planning) with an Instant Response Module (IRM, a VideoLLM for continuous monitoring). The two modules play complementary roles: the DRM performs strategic planning with structured 3D scene updates and guides what the IRM attends to, while the IRM analyzes video streams to update memory, correct ongoing actions, and trigger replanning when necessary. Through this bidirectional coordination, the modules address the trade off between maintaining awareness and avoiding costly updates, enabling robust adaptation under dynamic conditions. Evaluated in three real world environments with dynamic object placement, BINDER achieves substantially higher success and efficiency than SoTA baselines, demonstrating its effectiveness for real world deployment.
comment : 12 pages, 8 figures
☆ Test Time Training for AC Power Flow Surrogates via Physics and Operational Constraint Refinement
Panteleimon Dogoulis, Mohammad Iman Alizadeh, Sylvain Kubler, Maxime Cordy
Power Flow (PF) calculation based on machine learning (ML) techniques offer significant computational advantages over traditional numerical methods but often struggle to maintain full physical consistency. This paper introduces a physics-informed test-time training (PI-TTT) framework that enhances the accuracy and feasibility of ML-based PF surrogates by enforcing AC power flow equalities and operational constraints directly at inference time. The proposed method performs a lightweight self-supervised refinement of the surrogate outputs through few gradient-based updates, enabling local adaptation to unseen operating conditions without requiring labeled data. Extensive experiments on the IEEE 14-, 118-, and 300-bus systems and the PEGASE 1354-bus network show that PI-TTT reduces power flow residuals and operational constraint violations by one to two orders of magnitude compared with purely ML-based models, while preserving their computational advantage. The results demonstrate that PI-TTT provides fast, accurate, and physically reliable predictions, representing a promising direction for scalable and physics-consistent learning in power system analysis.
☆ Edge Deployment of Small Language Models, a comprehensive comparison of CPU, GPU and NPU backends
Pablo Prieto, Pablo Abad
Edge computing processes data where it is generated, enabling faster decisions, lower bandwidth usage, and improved privacy. However, edge devices typically operate under strict constraints on processing power, memory, and energy consumption, making them unsuitable for large language models (LLMs). Fortunately, Small Language Models (SLMs) offer lightweight alternatives that bring AI inference to resource-constrained environments by significantly reducing computational cost while remaining suitable for specialization and customization. In this scenario, selecting the hardware platform that best balances performance and efficiency for SLM inference is challenging due to strict resource limitations. To address this issue, this study evaluates the inference performance and energy efficiency of commercial CPUs (Intel and ARM), GPUs (NVIDIA), and NPUs (RaiderChip) for running SLMs. GPUs, the usual platform of choice, are compared against commercial NPUs and recent multi-core CPUs. While NPUs leverage custom hardware designs optimized for computation, modern CPUs increasingly incorporate dedicated features targeting language-model workloads. Using a common execution framework and a suite of state-of-the-art SLMs, we analyze both maximum achievable performance and processing and energy efficiency across commercial solutions available for each platform. The results indicate that specialized backends outperform general-purpose CPUs, with NPUs achieving the highest performance by a wide margin. Bandwidth normalization proves essential for fair cross-architecture comparisons. Although low-power ARM processors deliver competitive results when energy usage is considered, metrics that combine performance and power (such as EDP) again highlight NPUs as the dominant architecture. These findings show that designs optimized for both efficiency and performance offer a clear advantage for edge workloads.
comment : 8 pages, 9 figures
☆ On the Condition Number Dependency in Bilevel Optimization
Lesi Chen, Jingzhao Zhang
Bilevel optimization minimizes an objective function, defined by an upper-level problem whose feasible region is the solution of a lower-level problem. We study the oracle complexity of finding an $ε$-stationary point with first-order methods when the upper-level problem is nonconvex and the lower-level problem is strongly convex. Recent works (Ji et al., ICML 2021; Arbel and Mairal, ICLR 2022; Chen el al., JMLR 2025) achieve a $\tilde{\mathcal{O}}(κ^4 ε^{-2})$ upper bound that is near-optimal in $ε$. However, the optimal dependency on the condition number $κ$ is unknown. In this work, we establish a new $Ω(κ^2 ε^{-2})$ lower bound and $\tilde{\mathcal{O}}(κ^{7/2} ε^{-2})$ upper bound for this problem, establishing the first provable gap between bilevel problems and minimax problems in this setup. Our lower bounds can be extended to various settings, including high-order smooth functions, stochastic oracles, and convex hyper-objectives: (1) For second-order and arbitrarily smooth problems, we show $Ω(κ_y^{13/4} ε^{-12/7})$ and $Ω(κ^{17/10} ε^{-8/5})$ lower bounds, respectively. (2) For convex-strongly-convex problems, we improve the previously best lower bound (Ji and Liang, JMLR 2022) from $Ω(κ/\sqrtε)$ to $Ω(κ^{5/4} / \sqrtε)$. (3) For smooth stochastic problems, we show an $Ω(κ^4 ε^{-4})$ lower bound.
☆ Prompt -based Consistent Video Colorization
Silvia Dani, Tiberio Uricchio, Lorenzo Seidenari
Existing video colorization methods struggle with temporal flickering or demand extensive manual input. We propose a novel approach automating high-fidelity video colorization using rich semantic guidance derived from language and segmentation. We employ a language-conditioned diffusion model to colorize grayscale frames. Guidance is provided via automatically generated object masks and textual prompts; our primary automatic method uses a generic prompt, achieving state-of-the-art results without specific color input. Temporal stability is achieved by warping color information from previous frames using optical flow (RAFT); a correction step detects and fixes inconsistencies introduced by warping. Evaluations on standard benchmarks (DAVIS30, VIDEVO20) show our method achieves state-of-the-art performance in colorization accuracy (PSNR) and visual realism (Colorfulness, CDC), demonstrating the efficacy of automated prompt-based guidance for consistent video colorization.
☆ Tracing Footsteps of Similar Cities: Modeling Urban Economic Vitality with Dynamic Inter-City Graph Embeddings
Xiaofeng Li, Xiangyi Xiao, Xiaocong Du, Ying Zhang, Haipeng Zhang
Urban economic vitality is a crucial indicator of a city's long-term growth potential, comprising key metrics such as the annual number of new companies and the population employed. However, modeling urban economic vitality remains challenging. This study develops ECO-GROW, a multi-graph framework modeling China's inter-city networks (2005-2021) to generate urban embeddings that model urban economic vitality. Traditional approaches relying on static city-level aggregates fail to capture a fundamental dynamic: the developmental trajectory of one city today may mirror that of its structurally similar counterparts tomorrow. ECO-GROW overcomes this limitation by integrating industrial linkages, POI similarities, migration similarities and temporal network evolution over 15 years. The framework combines a Dynamic Top-K GCN to adaptively select influential inter-city connections and an adaptive Graph Scorer mechanism to dynamically weight cross-regional impacts. Additionally, the model incorporates a link prediction task based on Barabasi Proximity, optimizing the graph representation. Experimental results demonstrate ECO-GROW's superior accuracy in predicting entrepreneurial activities and employment trends compared to conventional models. By open-sourcing our code, we enable government agencies and public sector organizations to leverage big data analytics for evidence-based urban planning, economic policy formulation, and resource allocation decisions that benefit society at large.
☆ RELiQ: Scalable Entanglement Routing via Reinforcement Learning in Quantum Networks
Tobias Meuser, Jannis Weil, Aninda Lahiri, Marius Paraschiv
Quantum networks are becoming increasingly important because of advancements in quantum computing and quantum sensing, such as recent developments in distributed quantum computing and federated quantum machine learning. Routing entanglement in quantum networks poses several fundamental as well as technical challenges, including the high dynamicity of quantum network links and the probabilistic nature of quantum operations. Consequently, designing hand-crafted heuristics is difficult and often leads to suboptimal performance, especially if global network topology information is unavailable. In this paper, we propose RELiQ, a reinforcement learning-based approach to entanglement routing that only relies on local information and iterative message exchange. Utilizing a graph neural network, RELiQ learns graph representations and avoids overfitting to specific network topologies - a prevalent issue for learning-based approaches. Our approach, trained on random graphs, consistently outperforms existing local information heuristics and learning-based approaches when applied to random and real-world topologies. When compared to global information heuristics, our method achieves similar or superior performance because of its rapid response to topology changes.
☆ Swarms of Large Language Model Agent s for Protein Sequence Design with Experimental Validation
Fiona Y. Wang, Di Sheng Lee, David L. Kaplan, Markus J. Buehler
Designing proteins de novo with tailored structural, physicochemical, and functional properties remains a grand challenge in biotechnology, medicine, and materials science, due to the vastness of sequence space and the complex coupling between sequence, structure, and function. Current state-of-the-art generative methods, such as protein language models (PLMs) and diffusion-based architectures, often require extensive fine-tuning, task-specific data, or model reconfiguration to support objective-directed design, thereby limiting their flexibility and scalability. To overcome these limitations, we present a decentralized, agent-based framework inspired by swarm intelligence for de novo protein design. In this approach, multiple large language model (LLM) agents operate in parallel, each assigned to a specific residue position. These agents iteratively propose context-aware mutations by integrating design objectives, local neighborhood interactions, and memory and feedback from previous iterations. This position-wise, decentralized coordination enables emergent design of diverse, well-defined sequences without reliance on motif scaffolds or multiple sequence alignments, validated with experiments on proteins with alpha helix and coil structures. Through analyses of residue conservation, structure-based metrics, and sequence convergence and embeddings, we demonstrate that the framework exhibits emergent behaviors and effective navigation of the protein fitness landscape. Our method achieves efficient, objective-directed designs within a few GPU-hours and operates entirely without fine-tuning or specialized training, offering a generalizable and adaptable solution for protein design. Beyond proteins, the approach lays the groundwork for collective LLM-driven design across biomolecular systems and other scientific discovery tasks.
☆ Enhanced Conditional Generation of Double Perovskite by Knowledge-Guided Language Model Feedback
Inhyo Lee, Junhyeong Lee, Jongwon Park, KyungTae Lim, Seunghwa Ryu
Double perovskites (DPs) are promising candidates for sustainable energy technologies due to their compositional tunability and compatibility with low-energy fabrication, yet their vast design space poses a major challenge for conditional materials discovery. This work introduces a multi-agent, text gradient-driven framework that performs DP composition generation under natural-language conditions by integrating three complementary feedback sources: LLM-based self-evaluation, DP-specific domain knowledge-informed feedback, and ML surrogate-based feedback. Analogous to how knowledge-informed machine learning improves the reliability of conventional data-driven models, our framework incorporates domain-informed text gradients to guide the generative process toward physically meaningful regions of the DP composition space. Systematic comparison of three incremental configurations, (i) pure LLM generation, (ii) LLM generation with LLM reasoning-based feedback, and (iii) LLM generation with domain knowledge-guided feedback, shows that iterative guidance from knowledge-informed gradients improves stability-condition satisfaction without additional training data, achieving over 98% compositional validity and up to 54% stable or metastable candidates, surpassing both the LLM-only baseline (43%) and prior GAN-based results (27%). Analyses of ML-based gradients further reveal that they enhance performance in in-distribution (ID) regions but become unreliable in out-of-distribution (OOD) regimes. Overall, this work provides the first systematic analysis of multi-agent, knowledge-guided text gradients for DP discovery and establishes a generalizable blueprint for MAS-driven generative materials design aimed at advancing sustainable technologies.
☆ When AI Bends Metal: AI-Assisted Optimization of Design Parameters in Sheet Metal Forming
Ahmad Tarraf, Koutaiba Kassem-Manthey, Seyed Ali Mohammadi, Philipp Martin, Lukas Moj, Semih Burak, Enju Park, Christian Terboven, Felix Wolf
Numerical simulations have revolutionized the industrial design process by reducing prototyping costs, design iterations, and enabling product engineers to explore the design space more efficiently. However, the growing scale of simulations demands substantial expert knowledge, computational resources, and time. A key challenge is identifying input parameters that yield optimal results, as iterative simulations are costly and can have a large environmental impact. This paper presents an AI-assisted workflow that reduces expert involvement in parameter optimization through the use of Bayesian optimization. Furthermore, we present an active learning variant of the approach, assisting the expert if desired. A deep learning model provides an initial parameter estimate, from which the optimization cycle iteratively refines the design until a termination condition (e.g., energy budget or iteration limit) is met. We demonstrate our approach, based on a sheet metal forming process, and show how it enables us to accelerate the exploration of the design space while reducing the need for expert involvement.
comment : 17 pages
☆ Adaptive tumor growth forecasting via neural & universal ODEs
Kavya Subramanian, Prathamesh Dinesh Joshi, Raj Abhijit Dandekar, Rajat Dandekar, Sreedath Panat
Forecasting tumor growth is critical for optimizing treatment. Classical growth models such as the Gompertz and Bertalanffy equations capture general tumor dynamics but may fail to adapt to patient-specific variability, particularly with limited data available. In this study, we leverage Neural Ordinary Differential Equations (Neural ODEs) and Universal Differential Equations (UDEs), two pillars of Scientific Machine Learning (SciML), to construct adaptive tumor growth models capable of learning from experimental data. Using the Gompertz model as a baseline, we replace rigid terms with adaptive neural networks to capture hidden dynamics through robust modeling in the Julia programming language. We use our models to perform forecasting under data constraints and symbolic recovery to transform the learned dynamics into explicit mathematical expressions. Our approach has the potential to improve predictive accuracy, guiding dynamic and effective treatment strategies for improved clinical outcomes.
comment : Accepted at JuliaCon 2025 conference
☆ RecToM: A Benchmark for Evaluating Machine Theory of Mind in LLM-based Conversational Recommender Systems AAAI 2026
Mengfan Li, Xuanhua Shi, Yang Deng
Large Language models are revolutionizing the conversational recommender systems through their impressive capabilities in instruction comprehension, reasoning, and human interaction. A core factor underlying effective recommendation dialogue is the ability to infer and reason about users' mental states (such as desire, intention, and belief), a cognitive capacity commonly referred to as Theory of Mind. Despite growing interest in evaluating ToM in LLMs, current benchmarks predominantly rely on synthetic narratives inspired by Sally-Anne test, which emphasize physical perception and fail to capture the complexity of mental state inference in realistic conversational settings. Moreover, existing benchmarks often overlook a critical component of human ToM: behavioral prediction, the ability to use inferred mental states to guide strategic decision-making and select appropriate conversational actions for future interactions. To better align LLM-based ToM evaluation with human-like social reasoning, we propose RecToM, a novel benchmark for evaluating ToM abilities in recommendation dialogues. RecToM focuses on two complementary dimensions: Cognitive Inference and Behavioral Prediction. The former focus on understanding what has been communicated by inferring the underlying mental states. The latter emphasizes what should be done next, evaluating whether LLMs can leverage these inferred mental states to predict, select, and assess appropriate dialogue strategies. Extensive experiments on state-of-the-art LLMs demonstrate that RecToM poses a significant challenge. While the models exhibit partial competence in recognizing mental states, they struggle to maintain coherent, strategic ToM reasoning throughout dynamic recommendation dialogues, particularly in tracking evolving intentions and aligning conversational strategies with inferred mental states.
comment : Accepted by AAAI 2026
☆ Efficiency and Effectiveness of SPLADE Models on Billion-Scale Web Document Title
Taeryun Won, Tae Kwan Lee, Hiun Kim, Hyemin Lee
This paper presents a comprehensive comparison of BM25, SPLADE, and Expanded-SPLADE models in the context of large-scale web document retrieval. We evaluate the effectiveness and efficiency of these models on datasets spanning from tens of millions to billions of web document titles. SPLADE and Expanded-SPLADE, which utilize sparse lexical representations, demonstrate superior retrieval performance compared to BM25, especially for complex queries. However, these models incur higher computational costs. We introduce pruning strategies, including document-centric pruning and top-k query term selection, boolean query with term threshold to mitigate these costs and improve the models' efficiency without significantly sacrificing retrieval performance. The results show that Expanded-SPLADE strikes the best balance between effectiveness and efficiency, particularly when handling large datasets. Our findings offer valuable insights for deploying sparse retrieval models in large-scale search engines.
☆ Co-Evolving Agent s: Learning from Failures as Hard Negatives
Yeonsung Jung, Trilok Padhi, Sina Shaham, Dipika Khullar, Joonhyun Jeong, Ninareh Mehrabi, Eunho Yang
The rapid progress of large foundation models has accelerated the development of task-specialized agents across diverse domains. However, the effectiveness of agents remains tightly coupled with the quality of training data, while curating task-specific datasets remains costly and often infeasible in real-world scenarios. Recent work has explored self-improving agents that autonomously generate, refine, and re-train on their own trajectories. A prominent line of approaches further leverages preference optimization by pairing predicted trajectories with scarce ground-truth trajectories, enabling agents to learn directly from their own failures. While these methods outperform supervised fine-tuning, their heavy reliance on predicted trajectories under limited ground-truth supervision leaves them prone to overfitting. To address this, we propose a co-evolving agents framework in which a target agent improves jointly with an auxiliary failure agent. The failure agent learns through preference optimization over failure trajectories from both the target and itself, thereby generating hard negatives that are close to success yet remain failures. Incorporating these informative hard negatives into the target agent's optimization sharpens decision boundaries and enhances generalization. Our comprehensive analysis and experiments across benchmark datasets show that our method not only shows improved performance but also demonstrates that failures, instead of being used as-is, can be systematically transformed into structured and valuable learning signals in self-improving agents.
☆ An interpretable unsupervised representation learning for high precision measurement in particle physics
Xing-Jian Lv, De-Xing Miao, Zi-Jun Xu, Jian-Chun Wang
Unsupervised learning has been widely applied to various tasks in particle physics. However, existing models lack precise control over their learned representations, limiting physical interpretability and hindering their use for accurate measurements. We propose the Histogram AutoEncoder (HistoAE), an unsupervised representation learning network featuring a custom histogram-based loss that enforces a physically structured latent space. Applied to silicon microstrip detectors, HistoAE learns an interpretable two-dimensional latent space corresponding to the particle's charge and impact position. After simple post-processing, it achieves a charge resolution of $0.25\,e$ and a position resolution of $3\,μ\mathrm{m}$ on beam-test data, comparable to the conventional approach. These results demonstrate that unsupervised deep learning models can enable physically meaningful and quantitatively precise measurements. Moreover, the generative capacity of HistoAE enables straightforward extensions to fast detector simulations.
comment : 8 pages, 7 figures
☆ Evaluating Embedding Models and Pipeline Optimization for AI Search Quality
Philip Zhong, Kent Chen, Don Wang
We evaluate the performance of various text embedding models and pipeline configurations for AI-driven search systems. We compare sentence-transformer and generative embedding models (e.g., All-MPNet, BGE, GTE, and Qwen) at different dimensions, indexing methods (Milvus HNSW/IVF), and chunking strategies. A custom evaluation dataset of 11,975 query-chunk pairs was synthesized from US City Council meeting transcripts using a local large language model (LLM). The data pipeline includes preprocessing, automated question generation per chunk, manual validation, and continuous integration/continuous deployment (CI/CD) integration. We measure retrieval accuracy using reference-based metrics: Top-K Accuracy and Normalized Discounted Cumulative Gain (NDCG). Our results demonstrate that higher-dimensional embeddings significantly boost search quality (e.g., Qwen3-Embedding-8B/4096 achieves Top-3 accuracy about 0.571 versus 0.412 for GTE-large/1024), and that neural re-rankers (e.g., a BGE cross-encoder) further improve ranking accuracy (Top-3 up to 0.527). Finer-grained chunking (512 characters versus 2000 characters) also improves accuracy. We discuss the impact of these factors and outline future directions for pipeline automation and evaluation.
☆ DeepPNI: Language- and graph-based model for mutation-driven protein-nucleic acid energetics
Somnath Mondal, Tinkal Mondal, Soumajit Pramanik, Rukmankesh Mehra
The interaction between proteins and nucleic acids is crucial for processes that sustain cellular function, including DNA maintenance and the regulation of gene expression and translation. Amino acid mutations in protein-nucleic acid complexes often lead to vital diseases. Experimental techniques have their own specific limitations in predicting mutational effects in protein-nucleic acid complexes. In this study, we compiled a large dataset of 1951 mutations including both protein-DNA and protein-RNA complexes and integrated structural and sequential features to build a deep learning-based regression model named DeepPNI. This model estimates mutation-induced binding free energy changes in protein-nucleic acid complexes. The structural features are encoded via edge-aware RGCN and the sequential features are extracted using protein language model ESM-2. We have achieved a high average Pearson correlation coefficient (PCC) of 0.76 in the large dataset via five-fold cross-validation. Consistent performance across individual dataset of protein-DNA, protein-RNA complexes, and different experimental temperature split dataset make the model generalizable. Our model showed good performance in complex-based five-fold cross-validation, which proved its robustness. In addition, DeepPNI outperformed in external dataset validation, and comparison with existing tools
☆ Training High-Level Schedulers with Execution-Feedback Reinforcement Learning for Long-Horizon GUI Automation
Zehao Deng, Tianjie Ju, Zheng Wu, Zhuosheng Zhang, Gongshen Liu
The rapid development of large vision-language model (VLM) has greatly promoted the research of GUI agent. However, GUI agents still face significant challenges in handling long-horizon tasks. First, single-agent models struggle to balance high-level capabilities and low-level execution capability, facing prevalent issues of responsibility coupling and capability conflicts. Second, agents lack awareness of the task state, leading to progress loss in long-horizon tasks. To address these challenges, we propose a staged execution-feedback reinforcement learning algorithm. Unlike training a unified policy model, we focus on training high-level scheduling models. Specifically, we propose and train two agents: a Coordinator, responsible for the strategic planning and task decomposition; and a State Tracker, responsible for context compression and information management to maintain the task's state and coherence. Based on this, we built the Coordinator-Executor-State Tracker (CES) multi-agent framework, which can be integrated with any low-level Executor model, assisting the Executor in solving long-horizon tasks through task scheduling and state management. Experiments on long-horizon task benchmarks demonstrate that CES significantly enhances the system's planning and state management capabilities. Furthermore, analysis confirms that our trained high-level scheduling module is a generalizable, plug-and-play module that significantly enhances the long-horizon capabilities of various Executors. Code can be available at https://github.com/hehehahi4/CES.
☆ From Compound Figures to Composite Understanding : Developing a Multi-Modal LLM from Biomedical Literature with Medical Multiple-Image Benchmark ing and Validation
Zhen Chen, Yihang Fu, Gabriel Madera, Mauro Giuffre, Serina Applebaum, Hyunjae Kim, Hua Xu, Qingyu Chen
Multi-modal large language models (MLLMs) have shown promise in advancing healthcare. However, most existing models remain confined to single-image understanding, which greatly limits their applicability in clinical workflows. In practice, medical diagnosis and progression often require synthesizing information across multiple images from different modalities or time points. The development of medical MLLMs capable of such multi-image understanding has been hindered by the lack of large-scale, high-quality annotated training data. To address this limitation, we propose a novel framework that leverages license-permissive compound images in biomedical literature, as a rich yet underutilized data source for multi-image analysis. Specifically, we design a five-stage, context-aware instruction generation paradigm underpinned by a divide-and-conquer strategy. By decomposing multi-image analysis into manageable sub-tasks, this paradigm empowers MLLMs to move beyond single-panel analysis and provide a composite understanding by learning the complex spatial, temporal, and cross-modal relationships inherent in these compound figures. By parsing over 237,000 compound figures and their contextual text for instruction generation, we develop M3LLM, a medical multi-image multi-modal large language model. For benchmarking, we construct PMC-MI-Bench for composite understanding, manually validated by medical experts. Extensive experiments show that M3LLM significantly outperforms both general-purpose and specialized medical MLLMs across multi-image, single-image, text-only, and multi-choice scenarios. Notably, M3LLM exhibits strong generalization to longitudinal chest X-ray analysis using the MIMIC dataset. This work establishes a scalable and efficient paradigm for developing medical MLLMs capable of composite reasoning, bridging the gap between biomedical literature and real-world clinical applications.
☆ 3D-Consistent Multi-View Editing by Diffusion Guidance
Josef Bengtson, David Nilsson, Dong In Lee, Fredrik Kahl
Recent advancements in diffusion models have greatly improved text-based image editing, yet methods that edit images independently often produce geometrically and photometrically inconsistent results across different views of the same scene. Such inconsistencies are particularly problematic for editing of 3D representations such as NeRFs or Gaussian Splat models. We propose a training-free diffusion framework that enforces multi-view consistency during the image editing process. The key assumption is that corresponding points in the unedited images should undergo similar transformations after editing. To achieve this, we introduce a consistency loss that guides the diffusion sampling toward coherent edits. The framework is flexible and can be combined with widely varying image editing methods, supporting both dense and sparse multi-view editing setups. Experimental results show that our approach significantly improves 3D consistency compared to existing multi-view editing methods. We also show that this increased consistency enables high-quality Gaussian Splat editing with sharp details and strong fidelity to user-specified text prompts. Please refer to our project page for video results: https://3d-consistent-editing.github.io/
☆ Embedded Universal Predictive Intelligence: a coherent framework for multi- agent learning
Alexander Meulemans, Rajai Nasser, Maciej Wołczyk, Marissa A. Weis, Seijin Kobayashi, Blake Richards, Guillaume Lajoie, Angelika Steger, Marcus Hutter, James Manyika, Rif A. Saurous, João Sacramento, Blaise Agüera y Arcas
The standard theory of model-free reinforcement learning assumes that the environment dynamics are stationary and that agents are decoupled from their environment, such that policies are treated as being separate from the world they inhabit. This leads to theoretical challenges in the multi-agent setting where the non-stationarity induced by the learning of other agents demands prospective learning based on prediction models. To accurately model other agents, an agent must account for the fact that those other agents are, in turn, forming beliefs about it to predict its future behavior, motivating agents to model themselves as part of the environment. Here, building upon foundational work on universal artificial intelligence (AIXI), we introduce a mathematical framework for prospective learning and embedded agency centered on self-prediction, where Bayesian RL agents predict both future perceptual inputs and their own actions, and must therefore resolve epistemic uncertainty about themselves as part of the universe they inhabit. We show that in multi-agent settings, self-prediction enables agents to reason about others running similar algorithms, leading to new game-theoretic solution concepts and novel forms of cooperation unattainable by classical decoupled agents. Moreover, we extend the theory of AIXI, and study universally intelligent embedded agents which start from a Solomonoff prior. We show that these idealized agents can form consistent mutual predictions and achieve infinite-order theory of mind, potentially setting a gold standard for embedded multi-agent learning.
comment : 203 pages, 3 figures
☆ PULSE-ICU: A Pretrain ed Unified Long-Sequence Encoder for Multi-task Prediction in Intensive Care Units
Sejeong Jang, Joo Heung Yoon, Hyo Kyung Lee
Intensive care unit (ICU) data are highly irregular, heterogeneous, and temporally fragmented, posing challenges for generalizable clinical prediction. We present PULSE-ICU, a self-supervised foundation model that learns event-level ICU representations from large-scale EHR sequences without resampling or manual feature engineering. A unified embedding module encodes event identity, continuous values, units, and temporal attributes, while a Longformer-based encoder enables efficient modeling of long trajectories. PULSE-ICU was fine-tuned across 18 prediction tasks, including mortality, intervention forecasting, and phenotype identification, achieving strong performance across task types. External validation on eICU, HiRID, and P12 showed substantial improvements with minimal fine-tuning, demonstrating robustness to domain shift and variable constraints. These findings suggest that foundation-style modeling can improve data efficiency and adaptability, providing a scalable framework for ICU decision support across diverse clinical environments.
☆ ARPGNet: Appearance- and Relation -aware Parallel Graph Attention Fusion Network for Facial Expression Recognition
Yan Li, Yong Zhao, Xiaohan Xia, Dongmei Jiang
The key to facial expression recognition is to learn discriminative spatial-temporal representations that embed facial expression dynamics. Previous studies predominantly rely on pre-trained Convolutional Neural Networks (CNNs) to learn facial appearance representations, overlooking the relationships between facial regions. To address this issue, this paper presents an Appearance- and Relation-aware Parallel Graph attention fusion Network (ARPGNet) to learn mutually enhanced spatial-temporal representations of appearance and relation information. Specifically, we construct a facial region relation graph and leverage the graph attention mechanism to model the relationships between facial regions. The resulting relational representation sequences, along with CNN-based appearance representation sequences, are then fed into a parallel graph attention fusion module for mutual interaction and enhancement. This module simultaneously explores the complementarity between different representation sequences and the temporal dynamics within each sequence. Experimental results on three facial expression recognition datasets demonstrate that the proposed ARPGNet outperforms or is comparable to state-of-the-art methods.
comment : Accepted by IEEE Transactions on Affective Computing. Submitted in August 2023; Accepted in October 2025
☆ MTR-VP: Towards End-to-End Trajectory Planning through Context -Driven Image Encoding and Multiple Trajectory Prediction
Maitrayee Keskar, Mohan Trivedi, Ross Greer
We present a method for trajectory planning for autonomous driving, learning image-based context embeddings that align with motion prediction frameworks and planning-based intention input. Within our method, a ViT encoder takes raw images and past kinematic state as input and is trained to produce context embeddings, drawing inspiration from those generated by the recent MTR (Motion Transformer) encoder, effectively substituting map-based features with learned visual representations. MTR provides a strong foundation for multimodal trajectory prediction by localizing agent intent and refining motion iteratively via motion query pairs; we name our approach MTR-VP (Motion Transformer for Vision-based Planning), and instead of the learnable intention queries used in the MTR decoder, we use cross attention on the intent and the context embeddings, which reflect a combination of information encoded from the driving scene and past vehicle states. We evaluate our methods on the Waymo End-to-End Driving Dataset, which requires predicting the agent's future 5-second trajectory in bird's-eye-view coordinates using prior camera images, agent pose history, and routing goals. We analyze our architecture using ablation studies, removing input images and multiple trajectory output. Our results suggest that transformer-based methods that are used to combine the visual features along with the kinetic features such as the past trajectory features are not effective at combining both modes to produce useful scene context embeddings, even when intention embeddings are augmented with foundation-model representations of scene context from CLIP and DINOv2, but that predicting a distribution over multiple futures instead of a single future trajectory boosts planning performance.
comment : 8 pages, 3 figures, 4 tables
☆ Enhanced Graph Convolutional Network with Chebyshev Spectral Graph and Graph Attention for Autism Spectrum Disorder Classification
Adnan Ferdous Ashrafi, Hasanul Kabir
ASD is a complicated neurodevelopmental disorder marked by variation in symptom presentation and neurological underpinnings, making early and objective diagnosis extremely problematic. This paper presents a Graph Convolutional Network (GCN) model, incorporating Chebyshev Spectral Graph Convolution and Graph Attention Networks (GAT), to increase the classification accuracy of ASD utilizing multimodal neuroimaging and phenotypic data. Leveraging the ABIDE I dataset, which contains resting-state functional MRI (rs-fMRI), structural MRI (sMRI), and phenotypic variables from 870 patients, the model leverages a multi-branch architecture that processes each modality individually before merging them via concatenation. Graph structure is encoded using site-based similarity to generate a population graph, which helps in understanding relationship connections across individuals. Chebyshev polynomial filters provide localized spectral learning with lower computational complexity, whereas GAT layers increase node representations by attention-weighted aggregation of surrounding information. The proposed model is trained using stratified five-fold cross-validation with a total input dimension of 5,206 features per individual. Extensive trials demonstrate the enhanced model's superiority, achieving a test accuracy of 74.82\% and an AUC of 0.82 on the entire dataset, surpassing multiple state-of-the-art baselines, including conventional GCNs, autoencoder-based deep neural networks, and multimodal CNNs.
comment : 6 pages, 2 figures, 2 tables, Accepted and presented at Image and Vision Computing New Zealand (IVCNZ) 2025
☆ Focused Chain-of-Thought: Efficient LLM Reasoning via Structured Input Information
Lukas Struppek, Dominik Hintersdorf, Hannah Struppek, Daniel Neider, Kristian Kersting
Recent large language models achieve strong reasoning performance by generating detailed chain-of-thought traces, but this often leads to excessive token use and high inference latency. Existing efficiency approaches typically focus on model-centric interventions, such as reinforcement learning or supervised fine-tuning, to reduce verbosity. In contrast, we propose a training-free, input-centric approach. Inspired by cognitive psychology, we introduce Focused Chain-of-Thought (F-CoT), which separates information extraction from the reasoning process. F-CoT first organizes the essential information from a query into a concise, structured context and then guides the model to reason exclusively over this context. By preventing attention to irrelevant details, F-CoT naturally produces shorter reasoning paths. On arithmetic word problems, F-CoT reduces generated tokens by 2-3x while maintaining accuracy comparable to standard zero-shot CoT. These results highlight structured input as a simple yet effective lever for more efficient LLM reasoning.
☆ Real-Time Long Horizon Air Quality Forecasting via Group-Relative Policy Optimization
Inha Kang, Eunki Kim, Wonjeong Ryu, Jaeyo Shin, Seungjun Yu, Yoon-Hee Kang, Seongeun Jeong, Eunhye Kim, Soontae Kim, Hyunjung Shim
Accurate long horizon forecasting of particulate matter (PM) concentration fields is essential for operational public health decisions. However, achieving reliable forecasts remains challenging in regions with complex terrain and strong atmospheric dynamics such as East Asia. While foundation models such as Aurora offer global generality, they often miss region-specific dynamics and rely on non-real-time inputs, limiting their practical utility for localized warning systems. To address this gap, we construct and release the real-world observations and high-resolution CMAQ-OBS dataset for East Asia, reducing regional error by 59.5% and enabling real-time 48-120 hour forecasts critical for public health alerts. However, standard point-wise objectives cannot reflect asymmetric operational costs, where false alarms deteriorate public trust while missed severe events endanger populations. This cost mismatch causes SFT models to over-predict and yield high False Alarm Rates. We introduce Group-Relative Policy Optimization (GRPO) with class-wise rewards and curriculum rollout to align predictions with operational priorities. Experimental results demonstrate that our framework significantly improves the reliability of the forecast. Compared to the SFT-only baseline, our model reduces the False Alarm Rate by 47.3% while achieving a competitive F1-score, proving its effectiveness for practical, real-world air quality forecasting systems on long lead time scenarios.
comment : 10 pages
☆ IMTalker: Efficient Audio-driven Talking Face Generation with Implicit Motion Transfer
Bo Chen, Tao Liu, Qi Chen, Xie Chen, Zilong Zheng
Talking face generation aims to synthesize realistic speaking portraits from a single image, yet existing methods often rely on explicit optical flow and local warping, which fail to model complex global motions and cause identity drift. We present IMTalker, a novel framework that achieves efficient and high-fidelity talking face generation through implicit motion transfer. The core idea is to replace traditional flow-based warping with a cross-attention mechanism that implicitly models motion discrepancy and identity alignment within a unified latent space, enabling robust global motion rendering. To further preserve speaker identity during cross-identity reenactment, we introduce an identity-adaptive module that projects motion latents into personalized spaces, ensuring clear disentanglement between motion and identity. In addition, a lightweight flow-matching motion generator produces vivid and controllable implicit motion vectors from audio, pose, and gaze cues. Extensive experiments demonstrate that IMTalker surpasses prior methods in motion accuracy, identity preservation, and audio-lip synchronization, achieving state-of-the-art quality with superior efficiency, operating at 40 FPS for video-driven and 42 FPS for audio-driven generation on an RTX 4090 GPU. We will release our code and pre-trained models to facilitate applications and future research.
comment : 11 pages, 5 figures
☆ WearVQA: A Visual Question Answering Benchmark for Wearables in Egocentric Authentic Real-wo rl d scenarios NeurIPS 2025
Eun Chang, Zhuangqun Huang, Yiwei Liao, Sagar Ravi Bhavsar, Amogh Param, Tammy Stark, Adel Ahmadyan, Xiao Yang, Jiaqi Wang, Ahsan Abdullah, Giang Nguyen, Akil Iyer, David Hall, Elissa Li, Shane Moon, Nicolas Scheffer, Kirmani Ahmed, Babak Damavandi, Rakesh Wanga, Anuj Kumar, Rohit Patel, Xin Luna Dong
We introduce WearVQA, the first benchmark specifically designed to evaluate the Visual Question Answering (VQA) capabilities of multi-model AI assistant on wearable devices like smart glasses. Unlike prior benchmarks that focus on high-quality, third-person imagery, WearVQA reflects the unique challenges of ego-centric interaction-where visual inputs may be occluded, poorly lit, unzoomed, or blurry, and questions are grounded in realistic wearable use cases. The benchmark comprises 2,520 carefully curated image-question-answer triplets, spanning 7 diverse image domains including both text-centric and general scenes, 10 cognitive task types ranging from basic recognition to various forms of reasoning, and 6 common wearables-specific image quality issues. All questions are designed to be answerable using only the visual input and common senses. WearVQA is paired with a rigorous LLM-as-a-judge evaluation framework with 96% labeling accuracy. Open-source and proprietary multi-model LLMs achieved a QA accuracy as low as 24-52% on WearVQA, with substantial drops on lower-quality images and reasoning-heavy tasks. These observations position WearVQA as a comprehensive and challenging benchmark for guiding technical advancement towards robust, real-world multi-model wearables AI systems.
comment : 11 pages, 5 figures, NeurIPS 2025
☆ A Theoretically Grounded Hybrid Ensemble for Reliable Detection of LLM-Generated Text
Sepyan Purnama Kristanto, Lutfi Hakim
The rapid proliferation of Large Language Models (LLMs) has blurred the line between human and machine authorship, creating practical risks for academic integrity and information reliability. Existing text detectors typically rely on a single methodological paradigm and suffer from poor generalization and high false positive rates (FPR), especially on high-stakes academic text. We propose a theoretically grounded hybrid ensemble that systematically fuses three complementary detection paradigms: (i) a RoBERTa-based transformer classifier for deep semantic feature extraction, (ii) a GPT-2-based probabilistic detector using perturbation-induced likelihood curvature, and (iii) a statistical linguistic feature analyzer capturing stylometric patterns. The core novelty lies in an optimized weighted voting framework, where ensemble weights are learned on the probability simplex to maximize F1-score rather than set heuristically. We provide a bias-variance analysis and empirically demonstrate low inter-model correlation (rho ~ 0.35-0.42), a key condition for variance reduction. Evaluated on a large-scale, multigenerator corpus of 30,000 documents, our system achieves 94.2% accuracy and an AUC of 0.978, with a 35% relative reduction in false positives on academic text. This yields a more reliable and ethically responsible detector for real-world deployment in education and other high-stakes domains.
comment : 24 pages
☆ A perceptual bias of AI Logical Argumentation Ability in Writing
Xi Cun, Jifan Ren, Asha Huang, Siyu Li, Ruzhen Song
Can machines think? This is a central question in artificial intelligence research. However, there is a substantial divergence of views on the answer to this question. Why do people have such significant differences of opinion, even when they are observing the same real world performance of artificial intelligence? The ability of logical reasoning like humans is often used as a criterion to assess whether a machine can think. This study explores whether human biases influence evaluations of the reasoning abilities of AI. An experiment was conducted where participants assessed two texts on the same topic, one AI generated and one human written,to test for perceptual biases in evaluating logical reasoning. Based on the experimental findings, a questionnaire was designed to quantify the attitudes toward AI.The results reveal a bias in perception. The evaluations of the logical reasoning ability of AI generated texts are significantly influenced by the preconceived views on the logical reasoning abilities of AI. Furthermore, frequent AI users were less likely to believe that AI usage undermines independent thinking.This study highlights the need to address perceptual biases to improve public understanding of AI's capabilities and foster better human AI interactions.
☆ Towards Heterogeneous Quantum Federated Learning: Challenges and Solutions
Ratun Rahman, Dinh C. Nguyen, Christo Kurisummoottil Thomas, Walid Saad
Quantum federated learning (QFL) combines quantum computing and federated learning to enable decentralized model training while maintaining data privacy. QFL can improve computational efficiency and scalability by taking advantage of quantum properties such as superposition and entanglement. However, existing QFL frameworks largely focus on homogeneity among quantum \textcolor{black}{clients, and they do not account} for real-world variances in quantum data distributions, encoding techniques, hardware noise levels, and computational capacity. These differences can create instability during training, slow convergence, and reduce overall model performance. In this paper, we conduct an in-depth examination of heterogeneity in QFL, classifying it into two categories: data or system heterogeneity. Then we investigate the influence of heterogeneity on training convergence and model aggregation. We critically evaluate existing mitigation solutions, highlight their limitations, and give a case study that demonstrates the viability of tackling quantum heterogeneity. Finally, we discuss potential future research areas for constructing robust and scalable heterogeneous QFL frameworks.
comment : Accepted at IEEE Network Magazine
☆ RemedyGS: Defend 3D Gaussian Splatting against Computation Cost Attacks
Yanping Li, Zhening Liu, Zijian Li, Zehong Lin, Jun Zhang
As a mainstream technique for 3D reconstruction, 3D Gaussian splatting (3DGS) has been applied in a wide range of applications and services. Recent studies have revealed critical vulnerabilities in this pipeline and introduced computation cost attacks that lead to malicious resource occupancies and even denial-of-service (DoS) conditions, thereby hindering the reliable deployment of 3DGS. In this paper, we propose the first effective and comprehensive black-box defense framework, named RemedyGS, against such computation cost attacks, safeguarding 3DGS reconstruction systems and services. Our pipeline comprises two key components: a detector to identify the attacked input images with poisoned textures and a purifier to recover the benign images from their attacked counterparts, mitigating the adverse effects of these attacks. Moreover, we incorporate adversarial training into the purifier to enforce distributional alignment between the recovered and original natural images, thereby enhancing the defense efficacy. Experimental results demonstrate that our framework effectively defends against white-box, black-box, and adaptive attacks in 3DGS systems, achieving state-of-the-art performance in both safety and utility.
☆ Stacked Ensemble of Fine-Tuned CNNs for Knee Osteoarthritis Severity Grading
Adarsh Gupta, Japleen Kaur, Tanvi Doshi, Teena Sharma, Nishchal K. Verma, Shantaram Vasikarla
Knee Osteoarthritis (KOA) is a musculoskeletal condition that can cause significant limitations and impairments in daily activities, especially among older individuals. To evaluate the severity of KOA, typically, X-ray images of the affected knee are analyzed, and a grade is assigned based on the Kellgren-Lawrence (KL) grading system, which classifies KOA severity into five levels, ranging from 0 to 4. This approach requires a high level of expertise and time and is susceptible to subjective interpretation, thereby introducing potential diagnostic inaccuracies. To address this problem a stacked ensemble model of fine-tuned Convolutional Neural Networks (CNNs) was developed for two classification tasks: a binary classifier for detecting the presence of KOA, and a multiclass classifier for precise grading across the KL spectrum. The proposed stacked ensemble model consists of a diverse set of pre-trained architectures, including MobileNetV2, You Only Look Once (YOLOv8), and DenseNet201 as base learners and Categorical Boosting (CatBoost) as the meta-learner. This proposed model had a balanced test accuracy of 73% in multiclass classification and 87.5% in binary classification, which is higher than previous works in extant literature.
comment : Accepted and Presented at IEEE UEMCON, IBM T.J. Watson Research Center, New York, USA, 2024
☆ Decomposed Trust: Exploring Privacy, Adversarial Robustness, Fairness, and Ethics of Low-Rank LLMs
Daniel Agyei Asante, Md Mokarram Chowdhury, Yang Li
Large language models (LLMs) have driven major advances across domains, yet their massive size hinders deployment in resource-constrained settings. Model compression addresses this challenge, with low-rank factorization emerging as a particularly effective method for reducing size, memory, and computation while maintaining accuracy. However, while these compressed models boast of benign performance and system-level advantages, their trustworthiness implications remain poorly understood. In this paper, we present the first comprehensive study of how low-rank factorization affects LLM trustworthiness across privacy, adversarial robustness, fairness, and ethical alignment. We evaluate multiple LLMs of different sizes and variants compressed with diverse low-rank algorithms, revealing key insights: (1) low-rank compression preserves or improves training data privacy but weakens PII protection during conversation; (2) adversarial robustness is generally preserved and often enhanced, even under deep compression; (3) ethical reasoning degrades in zero-shot settings but partially recovers with few-shot prompting; (4) fairness declines under compression. Beyond compression, we investigate how model scale and fine-tuning affect trustworthiness, as both are important in low-rank methods. To guide trustworthy compression strategies, we end our paper with a gradient-based attribution analysis to identify which layers in LLMs contribute most to adversarial robustness.
comment : 14 pages, 10 figures
☆ Binary-30K: A Heterogeneous Dataset for Deep Learning in Binary Analysis and Malware Detection
Michael J. Bommarito
Deep learning research for binary analysis faces a critical infrastructure gap. Today, existing datasets target single platforms, require specialized tooling, or provide only hand-engineered features incompatible with modern neural architectures; no single dataset supports accessible research and pedagogy on realistic use cases. To solve this, we introduce Binary-30K, the first heterogeneous binary dataset designed for sequence-based models like transformers. Critically, Binary-30K covers Windows, Linux, macOS, and Android across 15+ CPU architectures. With 29,793 binaries and approximately 26.93% malware representation, Binary-30K enables research on platform-invariant detection, cross-target transfer learning, and long-context binary understanding. The dataset provides pre-computed byte-level BPE tokenization alongside comprehensive structural metadata, supporting both sequence modeling and structure-aware approaches. Platform-first stratified sampling ensures representative coverage across operating systems and architectures, while distribution via Hugging Face with official train/validation/test splits enables reproducible benchmarking. The dataset is publicly available at https://huggingface.co/datasets/mjbommar/binary-30k, providing an accessible resource for researchers, practitioners, and students alike.
comment : 35 pages, 7 figures, 11 tables, 4 appendices. Dataset available at https://huggingface.co/datasets/mjbommar/binary-30k
☆ A Fast and Flat Federated Learning Method via Weighted Momentum and Sharpness-Aware Minimization
Tianle Li, Yongzhi Huang, Linshan Jiang, Chang Liu, Qipeng Xie, Wenfeng Du, Lu Wang, Kaishun Wu
In federated learning (FL), models must \emph{converge quickly} under tight communication budgets while \emph{generalizing} across non-IID client distributions. These twin requirements have naturally led to two widely used techniques: client/server \emph{momentum} to accelerate progress, and \emph{sharpness-aware minimization} (SAM) to prefer flat solutions. However, simply combining momentum and SAM leaves two structural issues unresolved in non-IID FL. We identify and formalize two failure modes: \emph{local-global curvature misalignment} (local SAM directions need not reflect the global loss geometry) and \emph{momentum-echo oscillation} (late-stage instability caused by accumulated momentum). To our knowledge, these failure modes have not been jointly articulated and addressed in the FL literature. We propose \textbf{FedWMSAM} to address both failure modes. First, we construct a momentum-guided global perturbation from server-aggregated momentum to align clients' SAM directions with the global descent geometry, enabling a \emph{single-backprop} SAM approximation that preserves efficiency. Second, we couple momentum and SAM via a cosine-similarity adaptive rule, yielding an early-momentum, late-SAM two-phase training schedule. We provide a non-IID convergence bound that \emph{explicitly models the perturbation-induced variance} $σ_ρ^2=σ^2+(Lρ)^2$ and its dependence on $(S, K, R, N)$ on the theory side. We conduct extensive experiments on multiple datasets and model architectures, and the results validate the effectiveness, adaptability, and robustness of our method, demonstrating its superiority in addressing the optimization challenges of Federated Learning. Our code is available at https://github.com/Huang-Yongzhi/NeurlPS_FedWMSAM.
☆ Hybrid Stackelberg Game and Diffusion -based Auction for Two-tier Agent ic AI Task Offloading in Internet of Agent s
Yue Zhong, Yongju Tong, Jiawen Kang, Minghui Dai, Hong-Ning Dai, Zhou Su, Dusit Niyato
The Internet of Agents (IoA) is rapidly gaining prominence as a foundational architecture for interconnected intelligent systems, designed to facilitate seamless discovery, communication, and collaborative reasoning among a vast network of Artificial Intelligence (AI) agents. Powered by Large Language and Vision-Language Models, IoA enables the development of interactive, rational agents capable of complex cooperation, moving far beyond traditional isolated models. IoA involves physical entities, i.e., Wireless Agents (WAs) with limited onboard resources, which need to offload their compute-intensive agentic AI services to nearby servers. Such servers can be Mobile Agents (MAs), e.g., vehicle agents, or Fixed Agents (FAs), e.g., end-side units agents. Given their fixed geographical locations and stable connectivity, FAs can serve as reliable communication gateways and task aggregation points. This stability allows them to effectively coordinate with and offload to an Aerial Agent (AA) tier, which has an advantage not affordable for highly mobile MAs with dynamic connectivity limitations. As such, we propose a two-tier optimization approach. The first tier employs a multi-leader multi-follower Stackelberg game. In the game, MAs and FAs act as the leaders who set resource prices. WAs are the followers to determine task offloading ratios. However, when FAs become overloaded, they can further offload tasks to available aerial resources. Therefore, the second tier introduces a Double Dutch Auction model where overloaded FAs act as the buyers to request resources, and AAs serve as the sellers for resource provision. We then develop a diffusion-based Deep Reinforcement Learning algorithm to solve the model. Numerical results demonstrate the superiority of our proposed scheme in facilitating task offloading.
☆ Real-Time Procedural Learning From Experience for AI Agent s
Dasheng Bi, Yubin Hu, Mohammed N. Nasir
Learning how to do things from trial and error in real time is a hallmark of biological intelligence, yet most LLM-based agents lack mechanisms to acquire procedural knowledge after deployment. We propose Procedural Recall for Agents with eXperiences Indexed by State (PRAXIS), a lightweight post-training learning mechanism that stores the consequences of actions and retrieves them by jointly matching environmental and internal states of past episodes to the current state. PRAXIS augments agentic action selection with retrieved state-action-result exemplars that are generated in real time. When evaluated on the REAL web browsing benchmark, PRAXIS improves task completion accuracy, reliability, and cost efficiency across different foundation model backbones, and shows preliminary generalization to unseen tasks in similar environments. These results demonstrate that PRAXIS enables the practical adoption of AI agents in fast-evolving stateful environments by helping them learn new procedures effectively.
☆ A Multi-View Multi-Timescale Hypergraph-Empowered Spatiotemporal Framework for EV Charging Forecasting
Jinhao Li, Hao Wang
Accurate electric vehicle (EV) charging demand forecasting is essential for stable grid operation and proactive EV participation in electricity market. Existing forecasting methods, particularly those based on graph neural networks, are often limited to modeling pairwise relationships between stations, failing to capture the complex, group-wise dynamics inherent in urban charging networks. To address this gap, we develop a novel forecasting framework namely HyperCast, leveraging the expressive power of hypergraphs to model the higher-order spatiotemporal dependencies hidden in EV charging patterns. HyperCast integrates multi-view hypergraphs, which capture both static geographical proximity and dynamic demand-based functional similarities, along with multi-timescale inputs to differentiate between recent trends and weekly periodicities. The framework employs specialized hyper-spatiotemporal blocks and tailored cross-attention mechanisms to effectively fuse information from these diverse sources: views and timescales. Extensive experiments on four public datasets demonstrate that HyperCast significantly outperforms a wide array of state-of-the-art baselines, demonstrating the effectiveness of explicitly modeling collective charging behaviors for more accurate forecasting.
comment : 14 pages
☆ ICM-SR: Image-Conditioned Manifold Regularization for Image Super-Resoultion
Junoh Kang, Donghun Ryu, Bohyung Han
Real world image super-resolution (Real-ISR) often leverages the powerful generative priors of text-to-image diffusion models by regularizing the output to lie on their learned manifold. However, existing methods often overlook the importance of the regularizing manifold, typically defaulting to a text-conditioned manifold. This approach suffers from two key limitations. Conceptually, it is misaligned with the Real-ISR task, which is to generate high quality (HQ) images directly tied to the low quality (LQ) images. Practically, the teacher model often reconstructs images with color distortions and blurred edges, indicating a flawed generative prior for this task. To correct these flaws and ensure conceptual alignment, a more suitable manifold must incorporate information from the images. While the most straightforward approach is to condition directly on the raw input images, their high information densities make the regularization process numerically unstable. To resolve this, we propose image-conditioned manifold regularization (ICM), a method that regularizes the output towards a manifold conditioned on the sparse yet essential structural information: a combination of colormap and Canny edges. ICM provides a task-aligned and stable regularization signal, thereby avoiding the instability of dense-conditioning and enhancing the final super-resolution quality. Our experiments confirm that the proposed regularization significantly enhances super-resolution performance, particularly in perceptual quality, demonstrating its effectiveness for real-world applications. We will release the source code of our work for reproducibility.
☆ Distillability of LLM Security Logic: Predicting Attack Success Rate of Outline Filling Attack via Ranking Regression
Tianyu Zhang, Zihang Xi, Jingyu Hua, Sheng Zhong
In the realm of black-box jailbreak attacks on large language models (LLMs), the feasibility of constructing a narrow safety proxy, a lightweight model designed to predict the attack success rate (ASR) of adversarial prompts, remains underexplored. This work investigates the distillability of an LLM's core security logic. We propose a novel framework that incorporates an improved outline filling attack to achieve dense sampling of the model's security boundaries. Furthermore, we introduce a ranking regression paradigm that replaces standard regression and trains the proxy model to predict which prompt yields a higher ASR. Experimental results show that our proxy model achieves an accuracy of 91.1 percent in predicting the relative ranking of average long response (ALR), and 69.2 percent in predicting ASR. These findings confirm the predictability and distillability of jailbreak behaviors, and demonstrate the potential of leveraging such distillability to optimize black-box attacks.
☆ Pathology-Aware Prototype Evolution via LLM-Driven Semantic Disambiguation for Multicenter Diabetic Retinopathy Diagnosis
Chunzheng Zhu, Yangfang Lin, Jialin Shao, Jianxin Lin, Yijun Wang
Diabetic retinopathy (DR) grading plays a critical role in early clinical intervention and vision preservation. Recent explorations predominantly focus on visual lesion feature extraction through data processing and domain decoupling strategies. However, they generally overlook domain-invariant pathological patterns and underutilize the rich contextual knowledge of foundation models, relying solely on visual information, which is insufficient for distinguishing subtle pathological variations. Therefore, we propose integrating fine-grained pathological descriptions to complement prototypes with additional context, thereby resolving ambiguities in borderline cases. Specifically, we propose a Hierarchical Anchor Prototype Modulation (HAPM) framework to facilitate DR grading. First, we introduce a variance spectrum-driven anchor prototype library that preserves domain-invariant pathological patterns. We further employ a hierarchical differential prompt gating mechanism, dynamically selecting discriminative semantic prompts from both LVLM and LLM sources to address semantic confusion between adjacent DR grades. Finally, we utilize a two-stage prototype modulation strategy that progressively integrates clinical knowledge into visual prototypes through a Pathological Semantic Injector (PSI) and a Discriminative Prototype Enhancer (DPE). Extensive experiments across eight public datasets demonstrate that our approach achieves pathology-guided prototype evolution while outperforming state-of-the-art methods. The code is available at https://github.com/zhcz328/HAPM.
comment : ACMMM 2025
☆ Predicting Public Health Impacts of Electricity Usage NeurIPS 2025
Yejia Liu, Zhifeng Wu, Pengfei Li, Shaolei Ren
The electric power sector is a leading source of air pollutant emissions, impacting the public health of nearly every community. Although regulatory measures have reduced air pollutants, fossil fuels remain a significant component of the energy supply, highlighting the need for more advanced demand-side approaches to reduce the public health impacts. To enable health-informed demand-side management, we introduce HealthPredictor, a domain-specific AI model that provides an end-to-end pipeline linking electricity use to public health outcomes. The model comprises three components: a fuel mix predictor that estimates the contribution of different generation sources, an air quality converter that models pollutant emissions and atmospheric dispersion, and a health impact assessor that translates resulting pollutant changes into monetized health damages. Across multiple regions in the United States, our health-driven optimization framework yields substantially lower prediction errors in terms of public health impacts than fuel mix-driven baselines. A case study on electric vehicle charging schedules illustrates the public health gains enabled by our method and the actionable guidance it can offer for health-informed energy management. Overall, this work shows how AI models can be explicitly designed to enable health-informed energy management for advancing public health and broader societal well-being. Our datasets and code are released at: https://github.com/Ren-Research/Health-Impact-Predictor.
comment : 21 Pages. Accepted to NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models (ResponsibleFM)
☆ MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosis AAAI 2026
Chunzheng Zhu, Yangfang Lin, Shen Chen, Yijun Wang, Jianxin Lin
Accurate medical diagnosis often involves progressive visual focusing and iterative reasoning, characteristics commonly observed in clinical workflows. While recent vision-language models demonstrate promising chain-of-thought (CoT) reasoning capabilities via reinforcement learning with verifiable rewards (RLVR), their purely on-policy learning paradigm tends to reinforce superficially coherent but clinically inaccurate reasoning paths. We propose MedEyes, a novel reinforcement learning framework that dynamically models clinician-style diagnostic reasoning by progressively attending to and interpreting relevant medical image regions. By incorporating off-policy expert guidance, MedEyes converts expert visual search trajectories into structured external behavioral signals, guiding the model toward clinically aligned visual reasoning. We design the Gaze-guided Reasoning Navigator (GRN) to emulate the diagnostic process through a dual-mode exploration strategy, scanning for systematic abnormality localization and drilling for detailed regional analysis. To balance expert imitation and autonomous discovery, we introduce the Confidence Value Sampler (CVS), which employs nucleus sampling and adaptive termination to create diverse yet credible exploration paths. Finally, the dual-stream GRPO optimization framework decouples on-policy and off-policy learning signals, mitigating reward assimilation and entropy collapse. Experiments demonstrate that MedEyes achieves an average performance improvement of +8.5\% across multiple medical VQA benchmarks, validating MedEyes's potential in building interpretable medical AI systems.
comment : This paper has been accepted by AAAI 2026
☆ AfriStereo: A Culturally Grounded Dataset for Evaluating Stereotypical Bias in Large Language Models
Yann Le Beux, Oluchi Audu, Oche D. Ankeli, Dhananjay Balakrishnan, Melissah Weya, Marie D. Ralaiarinosy, Ignatius Ezeani
Existing AI bias evaluation benchmarks largely reflect Western perspectives, leaving African contexts underrepresented and enabling harmful stereotypes in applications across various domains. To address this gap, we introduce AfriStereo, the first open-source African stereotype dataset and evaluation framework grounded in local socio-cultural contexts. Through community engaged efforts across Senegal, Kenya, and Nigeria, we collected 1,163 stereotypes spanning gender, ethnicity, religion, age, and profession. Using few-shot prompting with human-in-the-loop validation, we augmented the dataset to over 5,000 stereotype-antistereotype pairs. Entries were validated through semantic clustering and manual annotation by culturally informed reviewers. Preliminary evaluation of language models reveals that nine of eleven models exhibit statistically significant bias, with Bias Preference Ratios (BPR) ranging from 0.63 to 0.78 (p <= 0.05), indicating systematic preferences for stereotypes over antistereotypes, particularly across age, profession, and gender dimensions. Domain-specific models appeared to show weaker bias in our setup, suggesting task-specific training may mitigate some associations. Looking ahead, AfriStereo opens pathways for future research on culturally grounded bias evaluation and mitigation, offering key methodologies for the AI community on building more equitable, context-aware, and globally inclusive NLP technologies.
☆ When Do Domain-Specific Foundation Models Justify Their Cost? A Systematic Evaluation Across Retinal Imaging Tasks
David Isztl, Tahm Spitznagel, Gabor Mark Somfai, Rui Santos
Large vision foundation models have been widely adopted for retinal disease classification without systematic evidence justifying their parameter requirements. In the present work we address two critical questions: First, are large domain-specific foundation models essential, or do compact general-purpose architectures suffice? Second, does specialized retinal pretraining justify its computational cost? To answer this, we benchmark initialization strategies across four retinal imaging classification tasks spanning Optical Coherence Tomography (OCT) and Color Fundus Photography (CFP) modalities: 8-class OCT classification, 3-class diabetic macular edema (DME), 5-class diabetic retinopathy (DR), and 3-class glaucoma (GL) detection. We evaluate 12-13 model configurations per task, including vision transformers (22.8M-86.6M parameters), Swin Transformers (27.6M-28.3M), ConvNeXt (28.6M), and the domain-specific RETFound models (303M), under identical training conditions. Our results challenge prevailing assumptions: First, we demonstrate that pretraining provides universal benefits (5.18-18.41% improvement), scaling with task difficulty. Second, compact architectures (27-29M) dominate Pareto frontiers; SwinV2-tiny achieves top-1 performance on three datasets. Third, RETFound (303M) justifies its computational cost only for challenging DR grading (accuracy of 71.15%), while ImageNet pretraining proves to be sufficient with all other tasks (DME accuracy: 99.24%, OCT accuracy: 97.96%). CFP tasks show larger pretraining accuracy gains (9.13-18.41%) than OCT (5.18%). Thus, the evidence suggests that compact general-purpose models deliver near-optimal performance for most retinal classification tasks; specialized foundation models warranted only for fine-grained discrimination under extreme class imbalance.
☆ Joint Estimation of Sea State and Vessel Parameters Using a Mass-Spring-Damper Equivalence Model
Ranjeet K. Tiwari, Daniel Sgarioto, Peter Graham, Alexei Skvortsov, Sanjeev Arulampalam, Damith C. Ranasinghe
Real-time sea state estimation is vital for applications like shipbuilding and maritime safety. Traditional methods rely on accurate wave-vessel transfer functions to estimate wave spectra from onboard sensors. In contrast, our approach jointly estimates sea state and vessel parameters without needing prior transfer function knowledge, which may be unavailable or variable. We model the wave-vessel system using pseudo mass-spring-dampers and develop a dynamic model for the system. This method allows for recursive modeling of wave excitation as a time-varying input, relaxing prior works' assumption of a constant input. We derive statistically consistent process noise covariance and implement a square root cubature Kalman filter for sensor data fusion. Further, we derive the Posterior Cramer-Rao lower bound to evaluate estimator performance. Extensive Monte Carlo simulations and data from a high-fidelity validated simulator confirm that the estimated wave spectrum matches methods assuming complete transfer function knowledge.
☆ A Safety and Security Framework for Real-Wo rl d Agent ic Systems
Shaona Ghosh, Barnaby Simkin, Kyriacos Shiarlis, Soumili Nandi, Dan Zhao, Matthew Fiedler, Julia Bazinska, Nikki Pope, Roopa Prabhu, Daniel Rohrer, Michael Demoret, Bartley Richardson
This paper introduces a dynamic and actionable framework for securing agentic AI systems in enterprise deployment. We contend that safety and security are not merely fixed attributes of individual models but also emergent properties arising from the dynamic interactions among models, orchestrators, tools, and data within their operating environments. We propose a new way of identification of novel agentic risks through the lens of user safety. Although, for traditional LLMs and agentic models in isolation, safety and security has a clear separation, through the lens of safety in agentic systems, they appear to be connected. Building on this foundation, we define an operational agentic risk taxonomy that unifies traditional safety and security concerns with novel, uniquely agentic risks, including tool misuse, cascading action chains, and unintended control amplification among others. At the core of our approach is a dynamic agentic safety and security framework that operationalizes contextual agentic risk management by using auxiliary AI models and agents, with human oversight, to assist in contextual risk discovery, evaluation, and mitigation. We further address one of the most challenging aspects of safety and security of agentic systems: risk discovery through sandboxed, AI-driven red teaming. We demonstrate the framework effectiveness through a detailed case study of NVIDIA flagship agentic research assistant, AI-Q Research Assistant, showcasing practical, end-to-end safety and security evaluations in complex, enterprise-grade agentic workflows. This risk discovery phase finds novel agentic risks that are then contextually mitigated. We also release the dataset from our case study, containing traces of over 10,000 realistic attack and defense executions of the agentic workflow to help advance research in agentic safety.
♻ ☆ HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation
Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie, Ashmal Vayani, Ahmed Y. Radwan, Mukund S. Chettiar, Amandeep Singh, Mubarak Shah, Deval Pandya
Although recent large multimodal models (LMMs) demonstrate impressive progress on vision language tasks, their alignment with human centered (HC) principles, such as fairness, ethics, inclusivity, empathy, and robustness; remains poorly understood. We present HumaniBench, a unified evaluation framework designed to characterize HC alignment across realistic, socially grounded visual contexts. HumaniBench contains 32,000 expert-verified image question pairs derived from real world news imagery and spanning seven evaluation tasks: scene understanding, instance identity, multiple-choice visual question answering (VQA), multilinguality, visual grounding, empathetic captioning, and image resilience testing. Each task is mapped to one or more HC principles through a principled operationalization of metrics covering accuracy, harmful content detection, hallucination and faithfulness, coherence, cross lingual quality, empathy, and robustness.We evaluate 15 state-of-the-art LMMs under this framework and observe consistent cross model trade offs: proprietary systems achieve the strongest performance on ethics, reasoning, and empathy, while open-source models exhibit superior visual grounding and resilience. All models, however, show persistent gaps in fairness and multilingual inclusivity. We further analyze the effect of inference-time techniques, finding that chain of thought prompting and test-time scaling yield 8 to 12 % improvements on several HC dimensions. HumaniBench provides a reproducible, extensible foundation for systematic HC evaluation of LMMs and enables fine-grained analysis of alignment trade-offs that are not captured by conventional multimodal benchmarks. https://vectorinstitute.github.io/humanibench/
♻ ☆ FairPO: Robust Preference Optimization for Fair Multi-Label Learning
Soumen Kumar Mondal, Prateek Chanda, Akshit Varmora, Ganesh Ramakrishnan
Multi-label classification (MLC) often suffers from performance disparities across labels. We propose \textbf{FairPO}, a framework combining preference-based loss and group-robust optimization to improve fairness by targeting underperforming labels. FairPO partitions labels into a \textit{privileged} set for targeted improvement and a \textit{non-privileged} set to maintain baseline performance. For privileged labels, a DPO-inspired preference loss addresses hard examples by correcting ranking errors between true labels and their confusing counterparts. A constrained objective maintains performance for non-privileged labels, while a Group Robust Preference Optimization (GRPO) formulation adaptively balances both objectives to mitigate bias. We also demonstrate FairPO's versatility with reference-free variants using Contrastive (CPO) and Simple (SimPO) Preference Optimization.
♻ ☆ ProtoSiTex: Learning Semi-Interpretable Prototypes for Multi-label Text Classification
Utsav Kumar Nareti, Suraj Kumar, Soumya Pandey, Soumi Chattopadhyay, Chandranath Adak
The surge in user-generated reviews has amplified the need for interpretable models that can provide fine-grained insights. Existing prototype-based models offer intuitive explanations but typically operate at coarse granularity (sentence or document level) and fail to address the multi-label nature of real-world text classification. We propose ProtoSiTex, a semi-interpretable framework designed for fine-grained multi-label text classification. ProtoSiTex employs a dual-phase alternate training strategy: an unsupervised prototype discovery phase that learns semantically coherent and diverse prototypes, and a supervised classification phase that maps these prototypes to class labels. A hierarchical loss function enforces consistency across subsentence, sentence, and document levels, enhancing interpretability and alignment. Unlike prior approaches, ProtoSiTex captures overlapping and conflicting semantics using adaptive prototypes and multi-head attention. We also introduce a benchmark dataset of hotel reviews annotated at the subsentence level with multiple labels. Experiments on this dataset and two public benchmarks (binary and multi-class) show that ProtoSiTex achieves state-of-the-art performance while delivering faithful, human-aligned explanations, establishing it as a robust solution for semi-interpretable multi-label text classification.
♻ ☆ Frontier AI's Impact on the Cybersecurity Landscape
Yujin Potter, Wenbo Guo, Zhun Wang, Tianneng Shi, Hongwei Li, Andy Zhang, Patrick Gage Kelley, Kurt Thomas, Dawn Song
The impact of frontier AI (i.e., AI agents and foundation models) in cybersecurity is rapidly increasing. In this paper, we comprehensively analyze this trend through multiple aspects: quantitative benchmarks, qualitative literature review, empirical evaluation, and expert survey. Our analyses consistently show that AI's capabilities and applications in attacks have exceeded those on the defensive side. Our empirical evaluation of widely used agent systems on cybersecurity benchmarks highlights that current AI agents struggle with flexible workflow planning and using domain-specific tools for complex security analysis -- capabilities particularly critical for defensive applications. Our expert survey of AI and security researchers and practitioners indicates a prevailing view that AI will continue to benefit attackers over defenders, though the gap is expected to narrow over time. These results show the urgent need to evaluate and mitigate frontier AI's risks, steering it towards benefiting cyber defenses. Responding to this need, we provide concrete calls to action regarding: the construction of new cybersecurity benchmarks, the development of AI agents for defense, the design of provably secure AI agents, the improvement of pre-deployment security testing and transparency, and the strengthening of user-oriented education and defenses. Our paper summary and blog are available at https://rdi.berkeley.edu/frontier-ai-impact-on-cybersecurity/.
♻ ☆ Scaling Equitable Reflection Assessment in Education via Large Language Models and Role-Based Feedback Agent s AAAI-26
Chenyu Zhang, Xiaohang Luo
Formative feedback is widely recognized as one of the most effective drivers of student learning, yet it remains difficult to implement equitably at scale. In large or low-resource courses, instructors often lack the time, staffing, and bandwidth required to review and respond to every student reflection, creating gaps in support precisely where learners would benefit most. This paper presents a theory-grounded system that uses five coordinated role-based LLM agents (Evaluator, Equity Monitor, Metacognitive Coach, Aggregator, and Reflexion Reviewer) to score learner reflections with a shared rubric and to generate short, bias-aware, learner-facing comments. The agents first produce structured rubric scores, then check for potentially biased or exclusionary language, add metacognitive prompts that invite students to think about their own thinking, and finally compose a concise feedback message of at most 120 words. The system includes simple fairness checks that compare scoring error across lower and higher scoring learners, enabling instructors to monitor and bound disparities in accuracy. We evaluate the pipeline in a 12-session AI literacy program with adult learners. In this setting, the system produces rubric scores that approach expert-level agreement, and trained graders rate the AI-generated comments as helpful, empathetic, and well aligned with instructional goals. Taken together, these results show that multi-agent LLM systems can deliver equitable, high-quality formative feedback at a scale and speed that would be impossible for human graders alone. More broadly, the work points toward a future where feedback-rich learning becomes feasible for any course size or context, advancing long-standing goals of equity, access, and instructional capacity in education.
comment : Accepted to AAAI-26 AISI Track
♻ ☆ Beyond Ensembles: Simulating All-Atom Protein Dynamics in a Learned Latent Space
Aditya Sengar, Jiying Zhang, Pierre Vandergheynst, Patrick Barth
Simulating the long-timescale dynamics of biomolecules is a central challenge in computational science. While enhanced sampling methods can accelerate these simulations, they rely on pre-defined collective variables that are often difficult to identify, restricting their ability to model complex switching mechanisms between metastable states. A recent generative model, LD-FPG, demonstrated that this problem could be bypassed by learning to sample the static equilibrium ensemble as all-atom deformations from a reference structure, establishing a powerful method for all-atom ensemble generation. However, while this approach successfully captures a system's probable conformations, it does not model the temporal evolution between them. We introduce the Graph Latent Dynamics Propagator (GLDP), a modular component for simulating dynamics within the learned latent space of LD-FPG. We then compare three classes of propagators: (i) score-guided Langevin dynamics, (ii) Koopman-based linear operators, and (iii) autoregressive neural networks. Within a unified encoder-propagator-decoder framework, we evaluate long-horizon stability, backbone and side-chain ensemble fidelity, and temporal kinetics via TICA. Benchmarks on systems ranging from small peptides to mixed-topology proteins and large GPCRs reveal that autoregressive neural networks deliver the most robust long rollouts and coherent physical timescales; score-guided Langevin best recovers side-chain thermodynamics when the score is well learned; and Koopman provides an interpretable, lightweight baseline that tends to damp fluctuations. These results clarify the trade-offs among propagators and offer practical guidance for latent-space simulators of all-atom protein dynamics.
♻ ☆ Automated Composition of Agent s: A Knapsack Approach for Agent ic Component Selection NeurIPS 2025
Michelle Yuan, Khushbu Pahwa, Shuaichen Chang, Mustafa Kaba, Jiarong Jiang, Xiaofei Ma, Yi Zhang, Monica Sunkara
Designing effective agentic systems requires the seamless composition and integration of agents, tools, and models within dynamic and uncertain environments. Most existing methods rely on static, semantic retrieval approaches for tool or agent discovery. However, effective reuse and composition of existing components remain challenging due to incomplete capability descriptions and the limitations of retrieval methods. Component selection suffers because the decisions are not based on capability, cost, and real-time utility. To address these challenges, we introduce a structured, automated framework for agentic system composition that is inspired by the knapsack problem. Our framework enables a composer agent to systematically identify, select, and assemble an optimal set of agentic components by jointly considering performance, budget constraints, and compatibility. By dynamically testing candidate components and modeling their utility in real-time, our approach streamlines the assembly of agentic systems and facilitates scalable reuse of resources. Empirical evaluation with Claude 3.5 Sonnet across five benchmarking datasets shows that our online-knapsack-based composer consistently lies on the Pareto frontier, achieving higher success rates at significantly lower component costs compared to our baselines. In the single-agent setup, the online knapsack composer shows a success rate improvement of up to 31.6% in comparison to the retrieval baselines. In multi-agent systems, the online knapsack composer increases success rate from 37% to 87% when agents are selected from an agent inventory of 100+ agents. The substantial performance gap confirms the robust adaptability of our method across diverse domains and budget constraints.
comment : Accepted to NeurIPS 2025 Conference
♻ ☆ Chain-of-Influence: Tracing Interdependencies Across Time and Features in Clinical Predictive Modelings
Yubo Li, Rema Padman
Modeling clinical time-series data is hampered by the challenge of capturing latent, time-varying dependencies among features. State-of-the-art approaches often rely on black-box mechanisms or simple aggregation, failing to explicitly model how the influence of one clinical variable propagates through others over time. We propose $\textbf{Chain-of-Influence (CoI)}$, an interpretable deep learning framework that constructs an explicit, time-unfolded graph of feature interactions. CoI enables the tracing of influence pathways, providing a granular audit trail that shows how any feature at any time contributes to the final prediction, both directly and through its influence on other variables. We evaluate CoI on mortality and disease progression tasks using the MIMIC-IV dataset and a chronic kidney disease cohort. Our framework achieves state-of-the-art predictive performance (AUROC of 0.960 on CKD progression and 0.950 on ICU mortality), with deletion-based sensitivity analyses confirming that CoI's learned attributions faithfully reflect its decision process. Through case studies, we demonstrate that CoI uncovers clinically meaningful, patient-specific patterns of disease progression, offering enhanced transparency into the temporal and cross-feature dependencies that inform clinical decision-making.
♻ ☆ Smart Traffic Signals: Comparing MA RL and Fixed-Time Strategies
Saahil Mahato
Urban traffic congestion, particularly at intersections, significantly affects travel time, fuel consumption, and emissions. Traditional fixed-time signal control systems often lack the adaptability to effectively manage dynamic traffic patterns. This study explores the application of multi-agent reinforcement learning (MARL) to optimize traffic signal coordination across multiple intersections within a simulated environment. A simulation was developed to model a network of interconnected intersections with randomly generated vehicle flows to reflect realistic traffic variability. A decentralized MARL controller was implemented in which each traffic signal operates as an autonomous agent, making decisions based on local observations and information from neighboring agents. Performance was evaluated against a baseline fixed-time controller using metrics such as average vehicle wait time and overall throughput. The MARL approach demonstrated statistically significant improvements, including reduced average waiting times and improved throughput. These findings suggest that MARL-based dynamic control strategies hold substantial promise to improve urban traffic management efficiency. More research is recommended to address the challenges of scalability and real-world implementation.
♻ ☆ A Coherence-Based Measure of AGI FAST
Fares Fourati
Recent approaches to evaluating Artificial General Intelligence (AGI) typically summarize a system's capability using the arithmetic mean of its proficiencies across multiple cognitive domains. While simple, this implicitly assumes compensability: exceptional performance in some areas can offset severe deficiencies in others. Genuine general intelligence, however, requires coherent sufficiency: balanced competence across all essential faculties. We introduce a coherence-based measure of AGI that integrates the generalized mean over a continuum of compensability exponents. This yields an area-under-the-curve (AUC) metric spanning arithmetic, geometric, and harmonic regimes, quantifying how robust an evaluated capability remains as compensability assumptions become stricter. Unlike the arithmetic mean, which rewards specialization, the AUC penalizes imbalance and exposes bottlenecks that constrain performance. To illustrate the framework, we apply it to cognitive profiles derived from the Cattell-Horn-Carroll (CHC) model, showing how coherence-based aggregation highlights imbalances that are obscured by arithmetic averaging. As a second, independent example, we apply the same methodology to a set of 17 heterogeneous benchmarks, demonstrating how coherence-based evaluation can reveal unevenness even in narrower task collections. These examples show that the proposed approach offers a principled, interpretable, and stricter foundation for measuring progress toward AGI.
comment : Accepted at FAST@AAAI 2026. 15 pages, 2 figures, 13 tables
♻ ☆ The Loss of Control Playbook: Degrees, Dynamics, and Preparedness
Charlotte Stix, Annika Hallensleben, Alejandro Ortega, Matteo Pistillo
This research report addresses the absence of an actionable definition for Loss of Control (LoC) in AI systems by developing a novel taxonomy and preparedness framework. Despite increasing policy and research attention, existing LoC definitions vary significantly in scope and timeline, hindering effective LoC assessment and mitigation. To address this issue, we draw from an extensive literature review and propose a graded LoC taxonomy, based on the metrics of severity and persistence, that distinguishes between Deviation, Bounded LoC, and Strict LoC. We model pathways toward a societal state of vulnerability in which sufficiently advanced AI systems have acquired or could acquire the means to cause Bounded or Strict LoC once a catalyst, either misalignment or pure malfunction, materializes. We argue that this state becomes increasingly likely over time, absent strategic intervention, and propose a strategy to avoid reaching a state of vulnerability. Rather than focusing solely on intervening on AI capabilities and propensities potentially relevant for LoC or on preventing potential catalysts, we introduce a complementary framework that emphasizes three extrinsic factors: Deployment context, Affordances, and Permissions (the DAP framework). Compared to work on intrinsic factors and catalysts, this framework has the unfair advantage of being actionable today. Finally, we put forward a plan to maintain preparedness and prevent the occurrence of LoC outcomes should a state of societal vulnerability be reached, focusing on governance measures (threat modeling, deployment policies, emergency response) and technical controls (pre-deployment testing, control measures, monitoring) that could maintain a condition of perennial suspension.
♻ ☆ One Patient, Many Context s: Scaling Medical AI with Context ual Intelligence
Michelle M. Li, Ben Y. Reis, Adam Rodman, Tianxi Cai, Noa Dagan, Ran D. Balicer, Joseph Loscalzo, Isaac S. Kohane, Marinka Zitnik
Medical AI, including clinical language models, vision-language models, and multimodal health record models, already summarizes notes, answers questions, and supports decisions. Their adaptation to new populations, specialties, or care settings often relies on fine-tuning, prompting, or retrieval from external knowledge bases. These strategies can scale poorly and risk contextual errors: outputs that appear plausible but miss critical patient or situational information. We envision context switching as a solution. Context switching adjusts model reasoning at inference without retraining. Generative models can tailor outputs to patient biology, care setting, or disease. Multimodal models can reason on notes, laboratory results, imaging, and genomics, even when some data are missing or delayed. Agent models can coordinate tools and roles based on tasks and users. In each case, context switching enables medical AI to adapt across specialties, populations, and geographies. It requires advances in data design, model architectures, and evaluation frameworks, and establishes a foundation for medical AI that scales to infinitely many contexts while remaining reliable and suited to real-world care.
♻ ☆ On the Logical Content of Knowledge Bases
Alexader V. Gheorghiu, Tao Gu
Standard epistemic logics introduce a modal operator K to represent knowledge, but in doing so they presuppose the logical apparatus they aim to explain. By contrast, this paper explores how logic may be derived from the structure of knowledge itself. We begin from a pre-logical notion of a knowledge base understood as a network of inferential connections between atomic propositions. Logical constants are then defined in terms of what is supported by such a base: intrinsically, by relations that hold within it, and extrinsically, by the behaviour of those relations under extension. This yields a general semantic framework in which familiar systems (classical, intuitionistic, and various intermediate logics) arise naturally from different assumptions about the form of knowledge. This offers a reversal of the traditional explanatory order: rather than treating logic as a precondition for the articulation of knowledge, it shows how logical structure can emerge from epistemic organisation.
♻ ☆ MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation KR
Yovin Yahathugoda, Davide Prezzi, Piyalitt Ittichaiwong, Vicky Goh, Sebastien Ourselin, Michela Antonelli
Active Surveillance (AS) is a treatment option for managing low and intermediate-risk prostate cancer (PCa), aiming to avoid overtreatment while monitoring disease progression through serial MRI and clinical follow-up. Accurate prostate segmentation is an important preliminary step for automating this process, enabling automated detection and diagnosis of PCa. However, existing deep-learning segmentation models are often trained on single-time-point and expertly annotated datasets, making them unsuitable for longitudinal AS analysis, where multiple time points and a scarcity of expert labels hinder their effective fine-tuning. To address these challenges, we propose MambaX-Net, a novel semi-supervised, dual-scan 3D segmentation architecture that computes the segmentation for time point t by leveraging the MRI and the corresponding segmentation mask from the previous time point. We introduce two new components: (i) a Mamba-enhanced Cross-Attention Module, which integrates the Mamba block into cross attention to efficiently capture temporal evolution and long-range spatial dependencies, and (ii) a Shape Extractor Module that encodes the previous segmentation mask into a latent anatomical representation for refined zone delination. Moreover, we introduce a semi-supervised self-training strategy that leverages pseudo-labels generated from a pre-trained nnU-Net, enabling effective learning without expert annotations. MambaX-Net was evaluated on a longitudinal AS dataset, and results showed that it significantly outperforms state-of-the-art U-Net and Transformer-based models, achieving superior prostate zone segmentation even when trained on limited and noisy data.
comment : Updated the acknowledgments section to include the UKRI Open Access statement
♻ ☆ IntrinsiX: High-Quality PBR Generation using Image Priors
Peter Kocsis, Lukas Höllein, Matthias Nießner
We introduce IntrinsiX, a novel method that generates high-quality intrinsic images from text description. In contrast to existing text-to-image models whose outputs contain baked-in scene lighting, our approach predicts physically-based rendering (PBR) maps. This enables the generated outputs to be used for content creation scenarios in core graphics applications that facilitate re-lighting, editing, and texture generation tasks. In order to train our generator, we exploit strong image priors, and pre-train separate models for each PBR material component (albedo, roughness, metallic, normals). We then align these models with a new cross-intrinsic attention formulation that concatenates key and value features in a consistent fashion. This allows us to exchange information between each output modality and to obtain semantically coherent PBR predictions. To ground each intrinsic component, we propose a rendering loss which provides image-space signals to constrain the model, thus facilitating sharp details also in the output BRDF properties. Our results demonstrate detailed intrinsic generation with strong generalization capabilities that outperforms existing intrinsic image decomposition methods used with generated images by a significant margin. Finally, we show a series of applications, including re-lighting, editing, and text-conditioned room-scale PBR texture generation.
comment : Project page: https://peter-kocsis.github.io/IntrinsiX/ Video: https://youtu.be/b0wVA44R93Y
♻ ☆ Sparse Autoencoders Learn Monosemantic Features in Vision -Language Models NeurIPS 2025
Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, Zeynep Akata
Sparse Autoencoders (SAEs) have recently gained attention as a means to improve the interpretability and steerability of Large Language Models (LLMs), both of which are essential for AI safety. In this work, we extend the application of SAEs to Vision-Language Models (VLMs), such as CLIP, and introduce a comprehensive framework for evaluating monosemanticity at the neuron-level in visual representations. To ensure that our evaluation aligns with human perception, we propose a benchmark derived from a large-scale user study. Our experimental results reveal that SAEs trained on VLMs significantly enhance the monosemanticity of individual neurons, with sparsity and wide latents being the most influential factors. Further, we demonstrate that applying SAE interventions on CLIP's vision encoder directly steers multimodal LLM outputs (e.g., LLaVA), without any modifications to the underlying language model. These findings emphasize the practicality and efficacy of SAEs as an unsupervised tool for enhancing both interpretability and control of VLMs. Code and benchmark data are available at https://github.com/ExplainableML/sae-for-vlm.
comment : Accepted at NeurIPS 2025
♻ ☆ From homeostasis to resource sharing: Biologically and economically aligned multi-objective multi- agent gridwo rl d-based AI safety benchmark s
Roland Pihlakas
Developing safe, aligned agentic AI systems requires comprehensive empirical testing, yet many existing benchmarks neglect crucial themes aligned with biology and economics, both time-tested fundamental sciences describing our needs and preferences. To address this gap, the present work focuses on introducing biologically and economically motivated themes that have been neglected in current mainstream discussions on AI safety - namely a set of multi-objective, multi-agent alignment benchmarks that emphasize homeostasis for bounded and biological objectives, diminishing returns for unbounded, instrumental, and business objectives, sustainability principle, and resource sharing. Eight main benchmark environments have been implemented on the above themes, to illustrate key pitfalls and challenges in agentic AI-s, such as unboundedly maximizing a homeostatic objective, over-optimizing one objective at the expense of others, neglecting safety constraints, or depleting shared resources.
comment : 21 pages, 13 figures, 2 tables
♻ ☆ ForAug: Recombining Foregrounds and Backgrounds to Improve Vision Transformer Training with Bias Mitigation
Tobias Christian Nauen, Brian Moser, Federico Raue, Stanislav Frolov, Andreas Dengel
Transformers, particularly Vision Transformers (ViTs), have achieved state-of-the-art performance in large-scale image classification. However, they often require large amounts of data and can exhibit biases, such as center or size bias, that limit their robustness and generalizability. This paper introduces ForAug, a novel data augmentation operation that addresses these challenges by explicitly imposing invariances into the training data, which are otherwise part of the neural network architecture. ForAug is constructed by using pretrained foundation models to separate and recombine foreground objects with different backgrounds. This recombination step enables us to take fine-grained control over object position and size, as well as background selection. We demonstrate that using ForAug significantly improves the accuracy of ViTs and other architectures by up to 4.5 percentage points (p.p.) on ImageNet, which translates to 7.3 p.p. on downstream tasks. Importantly, ForAug not only improves accuracy but also opens new ways to analyze model behavior and quantify biases. Namely, we introduce metrics for background robustness, foreground focus, center bias, and size bias and show that using ForAug during training substantially reduces these biases. In summary, ForAug provides a valuable tool for analyzing and mitigating biases, enabling the development of more robust and reliable computer vision models. Our code and dataset are publicly available at https://github.com/tobna/ForAug.
comment : v2: added DeiT, added ablation vs simple copy-paste
♻ ☆ Event Stream-based Sign Language Translation: A High-Definition Benchmark Dataset and A Novel Baseline
Shiao Wang, Xiao Wang, Duoqing Yang, Yao Rong, Fuling Wang, Jianing Li, Lin Zhu, Bo Jiang
Sign Language Translation (SLT) is a core task in the field of AI-assisted disability. Traditional SLT methods are typically based on visible light videos, which are easily affected by factors such as lighting variations, rapid hand movements, and privacy concerns. This paper proposes the use of bio-inspired event cameras to alleviate the aforementioned issues. Specifically, we introduce a new high-definition event-based sign language dataset, termed Event-CSL, which effectively addresses the data scarcity in this research area. The dataset comprises 14,827 videos, 14,821 glosses, and 2,544 Chinese words in the text vocabulary. These samples are collected across diverse indoor and outdoor scenes, covering multiple viewpoints, lighting conditions, and camera motions. We have also benchmarked existing mainstream SLT methods on this dataset to facilitate fair comparisons in future research.Furthermore, we propose a novel event-based sign language translation framework, termed EvSLT. The framework first segments continuous video features into clips and employs a Mamba-based memory aggregation module to compress and aggregate spatial detail features at the clip level. Subsequently, these spatial features, along with temporal representations obtained from temporal convolution, are then fused by a graph-guided spatiotemporal fusion module. Extensive experiments on Event-CSL, as well as other publicly available datasets, demonstrate the superior performance of our method. The dataset and source code will be released on https://github.com/Event-AHU/OpenESL
♻ ☆ EEG Agent : A Unified Framework for Automated EEG Analysis Using Large Language Models
Sha Zhao, Mingyi Peng, Haiteng Jiang, Tao Li, Shijian Li, Gang Pan
Scalable and generalizable analysis of brain activity is essential for advancing both clinical diagnostics and cognitive research. Electroencephalography (EEG), a non-invasive modality with high temporal resolution, has been widely used for brain states analysis. However, most existing EEG models are usually tailored for individual specific tasks, limiting their utility in realistic scenarios where EEG analysis often involves multi-task and continuous reasoning. In this work, we introduce EEGAgent, a general-purpose framework that leverages large language models (LLMs) to schedule and plan multiple tools to automatically complete EEG-related tasks. EEGAgent is capable of performing the key functions: EEG basic information perception, spatiotemporal EEG exploration, EEG event detection, interaction with users, and EEG report generation. To realize these capabilities, we design a toolbox composed of different tools for EEG preprocessing, feature extraction, event detection, etc. These capabilities were evaluated on public datasets, and our EEGAgent can support flexible and interpretable EEG analysis, highlighting its potential for real-world clinical applications.
♻ ☆ Cross-Modal Reconstruction Pretrain ing for Ramp Flow Prediction at Highway Interchanges
Yongchao Li, Jun Chen, Zhuoxuan Li, Chao Gao, Yang Li, Chu Zhang, Changyin Dong
Interchanges are crucial nodes for vehicle transfers between highways, yet the lack of real-time ramp detectors creates blind spots in traffic prediction. To address this, we propose a Spatio-Temporal Decoupled Autoencoder (STDAE), a two-stage framework that leverages cross-modal reconstruction pretraining. In the first stage, STDAE reconstructs historical ramp flows from mainline data, forcing the model to capture intrinsic spatio-temporal relations. Its decoupled architecture with parallel spatial and temporal autoencoders efficiently extracts heterogeneous features. In the prediction stage, the learned representations are integrated with models such as GWNet to enhance accuracy. Experiments on three real-world interchange datasets show that STDAE-GWNET consistently outperforms thirteen state-of-the-art baselines and achieves performance comparable to models using historical ramp data. This demonstrates its effectiveness in overcoming detector scarcity and its plug-and-play potential for diverse forecasting pipelines.
♻ ☆ Q-Net: Queue Length Estimation via Kalman-based Neural Networks
Ting Gao, Elvin Isufi, Winnie Daamen, Erik-Sander Smits, Serge Hoogendoorn
Estimating queue lengths at signalized intersections is a long-standing challenge in traffic management. Partial observability of vehicle flows complicates this task despite the availability of two privacy preserving data sources: (i) aggregated vehicle counts from loop detectors near stop lines, and (ii) aggregated floating car data (aFCD) that provide segment-wise average speed measurements. However, how to integrate these sources with differing spatial and temporal resolutions for queue length estimation is rather unclear. Addressing this question, we present Q Net: a robust queue estimation framework built upon a state-space formulation. This formulation addresses key challenges in queue modeling, such as violations of traffic conservation assumptions. To overcome the limitations of standard filtering models in integrating diverse data sources, Q-Net employs an AI-augmented Kalman filter for estimation. Q-Net follows the Kalman predict-update framework and maintains physical interpretability, with internal variables linked to real-world traffic dynamics. Q-Net can be implemented in real-time, making it suitable for integration into queue-based traffic control systems. To achieve spatial transferability across road sections, we group aFCD measurements into fixed-size groups. This strategy ensures the dimension of Q-Net's learnable parameters is independent of section length. Evaluations on urban main roads in Rotterdam, the Netherlands, show that Q-Net outperforms baseline methods, accurately tracking queue formation and dissipation while correcting aFCD-induced delays. By combining data efficiency, interpretability, and strong transferability, Q Net makes accurate queue length estimation possible without costly sensing infrastructure like cameras or radar.
♻ ☆ ChronoGraph: A Real-Wo rl d Graph-Based Multivariate Time Series Dataset NeurIPS 2025
Adrian Catalin Lutu, Ioana Pintilie, Elena Burceanu, Andrei Manolache
We present ChronoGraph, a graph-structured multivariate time series forecasting dataset built from real-world production microservices. Each node is a service that emits a multivariate stream of system-level performance metrics, capturing CPU, memory, and network usage patterns, while directed edges encode dependencies between services. The primary task is forecasting future values of these signals at the service level. In addition, ChronoGraph provides expert-annotated incident windows as anomaly labels, enabling evaluation of anomaly detection methods and assessment of forecast robustness during operational disruptions. Compared to existing benchmarks from industrial control systems or traffic and air-quality domains, ChronoGraph uniquely combines (i) multivariate time series, (ii) an explicit, machine-readable dependency graph, and (iii) anomaly labels aligned with real incidents. We report baseline results spanning forecasting models, pretrained time-series foundation models, and standard anomaly detectors. ChronoGraph offers a realistic benchmark for studying structure-aware forecasting and incident-aware evaluation in microservice systems.
comment : Accepted as an oral presentation at the NeurIPS 2025 Workshop on Recent Advances in Time Series Foundation Models (BERT2S)
♻ ☆ Generalizing Analogical Inference from Boolean to Continuous Domains AAAI 2026
Francisco Cunha, Yves Lepage, Miguel Couceiro, Zied Bouraoui
Analogical reasoning is a powerful inductive mechanism, widely used in human cognition and increasingly applied in artificial intelligence. Formal frameworks for analogical inference have been developed for Boolean domains, where inference is provably sound for affine functions and approximately correct for functions close to affine. These results have informed the design of analogy-based classifiers. However, they do not extend to regression tasks or continuous domains. In this paper, we revisit analogical inference from a foundational perspective. We first present a counterexample showing that existing generalization bounds fail even in the Boolean setting. We then introduce a unified framework for analogical reasoning in real-valued domains based on parameterized analogies defined via generalized means. This model subsumes both Boolean classification and regression, and supports analogical inference over continuous functions. We characterize the class of analogy-preserving functions in this setting and derive both worst-case and average-case error bounds under smoothness assumptions. Our results offer a general theory of analogical inference across discrete and continuous domains.
comment : 11 pages, to appear in AAAI 2026, extended version
♻ ☆ Interactive Query Answering on Knowledge Graphs with Soft Entity Constraints
Daniel Daza, Alberto Bernardi, Luca Costabello, Christophe Gueret, Masoud Mansoury, Michael Cochez, Martijn Schut
Methods for query answering over incomplete knowledge graphs retrieve entities that are \emph{likely} to be answers, which is particularly useful when such answers cannot be reached by direct graph traversal due to missing edges. However, existing approaches have focused on queries formalized using first-order-logic. In practice, many real-world queries involve constraints that are inherently vague or context-dependent, such as preferences for attributes or related categories. Addressing this gap, we introduce the problem of query answering with soft constraints. We formalize the problem and introduce two efficient methods designed to adjust query answer scores by incorporating soft constraints without disrupting the original answers to a query. These methods are lightweight, requiring tuning only two parameters or a small neural network trained to capture soft constraints while maintaining the original ranking structure. To evaluate the task, we extend existing QA benchmarks by generating datasets with soft constraints. Our experiments demonstrate that our methods can capture soft constraints while maintaining robust query answering performance and adding very little overhead. With our work, we explore a new and flexible way to interact with graph databases that allows users to specify their preferences by providing examples interactively.
♻ ☆ Self-Guided Defense: Adaptive Safety Alignment for Reasoning Models via Synthesized Guidelines
Yuhang Wang, Yanxu Zhu, Dongyuan Lu, Jitao Sang
Reasoning models have demonstrated remarkable capabilities in complex reasoning tasks. However, ensuring their safety against adversarial jailbreak prompts remains a critical challenge. Due to the covert and deceptive nature of such prompts, they can often evade built-in safety mechanisms and lead to the generation of harmful content. This underscores the need for an adaptive safety alignment approach that enables models to autonomously reinforce their defenses in response to adversarial inputs. This paper introduces the Synthesized Guideline-based Adaptive Safety Alignment (SGASA) framework, which internalizes model-generated safety guidelines to strengthen models' ability to enhance robustness against harmful adversarial prompts while minimizing unnecessary refusals of benign requests. SGASA consists of two key stages: Data Pre-synthesis, which generates safety guidelines and augmented prompts; and Alignment Fine-tuning, which leverages Supervised Fine-tuning (SFT) and Direct Preference Optimization (DPO) to embed these guidelines into the model. Extensive experiments across multiple datasets demonstrate that SGASA significantly improves model safety, validating its adaptive and scalable effectiveness.
♻ ☆ EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval NeurIPS 2025
Zebin Yang, Sunjian Zheng, Tong Xie, Tianshi Xu, Bo Yu, Fan Wang, Jie Tang, Shaoshan Liu, Meng Li
Object-goal navigation (ObjNav) tasks an agent with navigating to the location of a specific object in an unseen environment. Embodied agents equipped with large language models (LLMs) and online constructed navigation maps can perform ObjNav in a zero-shot manner. However, existing agents heavily rely on giant LLMs on the cloud, e.g., GPT-4, while directly switching to small LLMs, e.g., LLaMA3.2-11b, suffer from significant success rate drops due to limited model capacity for understanding complex navigation maps, which prevents deploying ObjNav on local devices. At the same time, the long prompt introduced by the navigation map description will cause high planning latency on local devices. In this paper, we propose EfficientNav to enable on-device efficient LLM-based zero-shot ObjNav. To help the smaller LLMs better understand the environment, we propose semantics-aware memory retrieval to prune redundant information in navigation maps. To reduce planning latency, we propose discrete memory caching and attention-based memory clustering to efficiently save and re-use the KV cache. Extensive experimental results demonstrate that EfficientNav achieves 11.1% improvement in success rate on HM3D benchmark over GPT-4-based baselines, and demonstrates 6.7x real-time latency reduction and 4.7x end-to-end latency reduction over GPT-4 planner. Our code is available on https://github.com/PKU-SEC-Lab/EfficientNav.
comment : NeurIPS 2025
♻ ☆ Guided Model Merging for Hybrid Data Learning: Leveraging Centralized Data to Refine Decentralized Models WACV 2026
Junyi Zhu, Ruicong Yao, Taha Ceritli, Savas Ozkan, Matthew B. Blaschko, Eunchung Noh, Jeongwon Min, Cho Jung Min, Mete Ozay
Current network training paradigms primarily focus on either centralized or decentralized data regimes. However, in practice, data availability often exhibits a hybrid nature, where both regimes coexist. This hybrid setting presents new opportunities for model training, as the two regimes offer complementary trade-offs: decentralized data is abundant but subject to heterogeneity and communication constraints, while centralized data, though limited in volume and potentially unrepresentative, enables better curation and high-throughput access. Despite its potential, effectively combining these paradigms remains challenging, and few frameworks are tailored to hybrid data regimes. To address this, we propose a novel framework that constructs a model atlas from decentralized models and leverages centralized data to refine a global model within this structured space. The refined model is then used to reinitialize the decentralized models. Our method synergizes federated learning (to exploit decentralized data) and model merging (to utilize centralized data), enabling effective training under hybrid data availability. Theoretically, we show that our approach achieves faster convergence than methods relying solely on decentralized data, due to variance reduction in the merging process. Extensive experiments demonstrate that our framework consistently outperforms purely centralized, purely decentralized, and existing hybrid-adaptable methods. Notably, our method remains robust even when the centralized and decentralized data domains differ or when decentralized data contains noise, significantly broadening its applicability.
comment : Accepted at WACV 2026
♻ ☆ Gelina: Unified Speech and Gesture Synthesis via Inte rl eaved Token Prediction
Téo Guichoux, Théodor Lemerle, Shivam Mehta, Jonas Beskow, Gustav Eje Henter, Laure Soulier, Catherine Pelachaud, Nicolas Obin
Human communication is multimodal, with speech and gestures tightly coupled, yet most computational methods for generating speech and gestures synthesize them sequentially, weakening synchrony and prosody alignment. We introduce Gelina, a unified framework that jointly synthesizes speech and co-speech gestures from text using interleaved token sequences in a discrete autoregressive backbone, with modality-specific decoders. Gelina supports multi-speaker and multi-style cloning and enables gesture-only synthesis from speech inputs. Subjective and objective evaluations demonstrate competitive speech quality and improved gesture generation over unimodal baselines.
comment : 5 pages
♻ ☆ IROTE: Human-like Traits Elicitation of Large Language Model via In- Context Self-Reflective Optimization AAAI 2026
Yuzhuo Bai, Shitong Duan, Muhua Huang, Jing Yao, Zhenghao Liu, Peng Zhang, Tun Lu, Xiaoyuan Yi, Maosong Sun, Xing Xie
Trained on various human-authored corpora, Large Language Models (LLMs) have demonstrated a certain capability of reflecting specific human-like traits (e.g., personality or values) by prompting, benefiting applications like personalized LLMs and social simulations. However, existing methods suffer from the superficial elicitation problem: LLMs can only be steered to mimic shallow and unstable stylistic patterns, failing to embody the desired traits precisely and consistently across diverse tasks like humans. To address this challenge, we propose IROTE, a novel in-context method for stable and transferable trait elicitation. Drawing on psychological theories suggesting that traits are formed through identity-related reflection, our method automatically generates and optimizes a textual self-reflection within prompts, which comprises self-perceived experience, to stimulate LLMs' trait-driven behavior. The optimization is performed by iteratively maximizing an information-theoretic objective that enhances the connections between LLMs' behavior and the target trait, while reducing noisy redundancy in reflection without any fine-tuning, leading to evocative and compact trait reflection. Extensive experiments across three human trait systems manifest that one single IROTE-generated self-reflection can induce LLMs' stable impersonation of the target trait across diverse downstream tasks beyond simple questionnaire answering, consistently outperforming existing strong baselines.
comment : This paper is accepted by AAAI 2026
♻ ☆ Masked Diffusion Models as Energy Minimization
Sitong Chen, Shen Nie, Jiacheng Sun, Zijin Feng, Zhenguo Li, Ji-Rong Wen, Chongxuan Li
We present a systematic theoretical framework that interprets masked diffusion models (MDMs) as solutions to energy minimization problems in discrete optimal transport. Specifically, we prove that three distinct energy formulations--kinetic, conditional kinetic, and geodesic energy--are mathematically equivalent under the structure of MDMs, and that MDMs minimize all three when the mask schedule satisfies a closed-form optimality condition. This unification not only clarifies the theoretical foundations of MDMs, but also motivates practical improvements in sampling. By parameterizing interpolation schedules via Beta distributions, we reduce the schedule design space to a tractable 2D search, enabling efficient post-training tuning without model modification. Experiments on synthetic and real-world benchmarks demonstrate that our energy-inspired schedules outperform hand-crafted baselines, particularly in low-step sampling settings.
♻ ☆ Qwen 3-VL Technical Report
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, Ke Zhu
We introduce Qwen3-VL, the most capable vision-language model in the Qwen series to date, achieving superior performance across a broad range of multimodal benchmarks. It natively supports interleaved contexts of up to 256K tokens, seamlessly integrating text, images, and video. The model family includes both dense (2B/4B/8B/32B) and mixture-of-experts (30B-A3B/235B-A22B) variants to accommodate diverse latency-quality trade-offs. Qwen3-VL delivers three core pillars: (i) markedly stronger pure-text understanding, surpassing comparable text-only backbones in several cases; (ii) robust long-context comprehension with a native 256K-token window for both text and interleaved multimodal inputs, enabling faithful retention, retrieval, and cross-referencing across long documents and videos; and (iii) advanced multimodal reasoning across single-image, multi-image, and video tasks, demonstrating leading performance on comprehensive evaluations such as MMMU and visual-math benchmarks (e.g., MathVista and MathVision). Architecturally, we introduce three key upgrades: (i) an enhanced interleaved-MRoPE for stronger spatial-temporal modeling across images and video; (ii) DeepStack integration, which effectively leverages multi-level ViT features to tighten vision-language alignment; and (iii) text-based time alignment for video, evolving from T-RoPE to explicit textual timestamp alignment for more precise temporal grounding. Under comparable token budgets and latency constraints, Qwen3-VL achieves superior performance in both dense and Mixture-of-Experts (MoE) architectures. We envision Qwen3-VL serving as a foundational engine for image-grounded reasoning, agentic decision-making, and multimodal code intelligence in real-world workflows.
comment : 42 pages
♻ ☆ SciSci GPT : Advancing Human-AI Collaboration in the Science of Science
Erzhuo Shao, Yifang Wang, Yifan Qian, Zhenyu Pan, Han Liu, Dashun Wang
The increasing availability of large-scale datasets has fueled rapid progress across many scientific fields, creating unprecedented opportunities for research and discovery while posing significant analytical challenges. Recent advances in large language models (LLMs) and AI agents have opened new possibilities for human-AI collaboration, offering powerful tools to navigate this complex research landscape. In this paper, we introduce SciSciGPT, an open-source, prototype AI collaborator that uses the science of science as a testbed to explore the potential of LLM-powered research tools. SciSciGPT automates complex workflows, supports diverse analytical approaches, accelerates research prototyping and iteration, and facilitates reproducibility. Through case studies, we demonstrate its ability to streamline a wide range of empirical and analytical research tasks while highlighting its broader potential to advance research. We further propose an LLM Agent capability maturity model for human-AI collaboration, envisioning a roadmap to further improve and expand upon frameworks like SciSciGPT. As AI capabilities continue to evolve, frameworks like SciSciGPT may play increasingly pivotal roles in scientific research and discovery, unlocking further opportunities. At the same time, these new advances also raise critical challenges, from ensuring transparency and ethical use to balancing human and AI contributions. Addressing these issues may shape the future of scientific inquiry and inform how we train the next generation of scientists to thrive in an increasingly AI-integrated research ecosystem.
♻ ☆ Dual-Model Weight Selection and Self-Knowledge Distillation for Medical Image Classification
Ayaka Tsutsumi, Guang Li, Ren Togo, Takahiro Ogawa, Satoshi Kondo, Miki Haseyama
We propose a novel medical image classification method that integrates dual-model weight selection with self-knowledge distillation (SKD). In real-world medical settings, deploying large-scale models is often limited by computational resource constraints, which pose significant challenges for their practical implementation. Thus, developing lightweight models that achieve comparable performance to large-scale models while maintaining computational efficiency is crucial. To address this, we employ a dual-model weight selection strategy that initializes two lightweight models with weights derived from a large pretrained model, enabling effective knowledge transfer. Next, SKD is applied to these selected models, allowing the use of a broad range of initial weight configurations without imposing additional excessive computational cost, followed by fine-tuning for the target classification tasks. By combining dual-model weight selection with self-knowledge distillation, our method overcomes the limitations of conventional approaches, which often fail to retain critical information in compact models. Extensive experiments on publicly available datasets-chest X-ray images, lung computed tomography scans, and brain magnetic resonance imaging scans-demonstrate the superior performance and robustness of our approach compared to existing methods.
comment : Published as a journal paper at Elsevier CIBM
♻ ☆ Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning NeurIPS 2025
Haolei Xu, Yuchen Yan, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Shengpei Jiang, Kaitao Song, Weiming Lu, Jun Xiao, Yueting Zhuang
Large language models (LLMs) have achieved remarkable progress on mathematical tasks through Chain-of-Thought (CoT) reasoning. However, existing mathematical CoT datasets often suffer from Thought Leaps due to experts omitting intermediate steps, which negatively impacts model learning and generalization. We propose the CoT Thought Leap Bridge Task, which aims to automatically detect leaps and generate missing intermediate reasoning steps to restore the completeness and coherence of CoT. To facilitate this, we constructed a specialized training dataset called ScaleQM+, based on the structured ScaleQuestMath dataset, and trained CoT-Bridge to bridge thought leaps. Through comprehensive experiments on mathematical reasoning benchmarks, we demonstrate that models fine-tuned on bridged datasets consistently outperform those trained on original datasets, with improvements of up to +5.87% on NuminaMath. Our approach effectively enhances distilled data (+3.02%) and provides better starting points for reinforcement learning (+3.1%), functioning as a plug-and-play module compatible with existing optimization techniques. Furthermore, CoT-Bridge demonstrate improved generalization to out-of-domain logical reasoning tasks, confirming that enhancing reasoning completeness yields broadly applicable benefits.
comment : Accepted to NeurIPS 2025. Camera ready version. Code: https://github.com/ZJU-REAL/Mind-the-Gap Project: https://zju-real.github.io/CoT-Bridge/
♻ ☆ A Neurosymbolic Framework for Interpretable Cognitive Attack Detection in Augmented Reality
Rongqian Chen, Allison Andreyev, Yanming Xiu, Joshua Chilukuri, Shunav Sen, Mahdi Imani, Bin Li, Maria Gorlatova, Gang Tan, Tian Lan
Augmented Reality (AR) enriches human perception by overlaying virtual elements onto the physical world. However, this tight coupling between virtual and real content makes AR vulnerable to cognitive attacks: manipulations that distort users' semantic understanding of the environment. Existing detection methods largely focus on visual inconsistencies at the pixel or image level, offering limited semantic reasoning or interpretability. To address these limitations, we introduce CADAR, a neuro-symbolic framework for cognitive attack detection in AR that integrates neural and symbolic reasoning. CADAR fuses multimodal vision-language representations from pre-trained models into a perception graph that captures objects, relations, and temporal contextual salience. Building on this structure, a particle-filter-based statistical reasoning module infers anomalies in semantic dynamics to reveal cognitive attacks. This combination provides both the adaptability of modern vision-language models and the interpretability of probabilistic symbolic reasoning. Preliminary experiments on an AR cognitive-attack dataset demonstrate consistent advantages over existing approaches, highlighting the potential of neuro-symbolic methods for robust and interpretable AR security.
♻ ☆ From Ambiguity to Verdict: A Semiotic-Grounded Multi-Perspective Agent for LLM Logical Reasoning
Yunyao Zhang, Xinglang Zhang, Junxi Sheng, Wenbing Li, Junqing Yu, Wei Yang, Zikai Song
Logical reasoning is a fundamental capability of large language models (LLMs). However, existing studies largely overlook the interplay between logical complexity and semantic complexity, resulting in methods that struggle to address challenging scenarios involving abstract propositions, ambiguous contexts, and conflicting stances, which are central to human reasoning. For this gap, we propose LogicAgent, a semiotic-square-guided framework designed to jointly address logical complexity and semantic complexity. LogicAgent explicitly performs multi-perspective deduction in first-order logic (FOL), while mitigating vacuous reasoning through existential import checks that incorporate a three-valued decision scheme (True, False, Uncertain) to handle boundary cases more faithfully. Furthermore, to overcome the semantic simplicity and low logical complexity of existing datasets, we introduce RepublicQA, a benchmark that reaches college-level difficulty (FKGL = 11.94) and exhibits substantially greater lexical and structural diversity than prior benchmarks. RepublicQA is grounded in philosophical concepts, featuring abstract propositions and systematically organized contrary and contradictory relations, making it the most semantically rich resource for evaluating logical reasoning. Experiments demonstrate that LogicAgent achieves state-of-the-art performance on RepublicQA, with a 6.25% average gain over strong baselines, and generalizes effectively to mainstream logical reasoning benchmarks including ProntoQA, ProofWriter, FOLIO, and ProverQA, achieving an additional 7.05% average gain. These results highlight the strong effectiveness of our semiotic-grounded multi-perspective reasoning in boosting LLMs' logical performance.
♻ ☆ Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis
Yuan Gao, Mattia Piccinini, Yuchen Zhang, Dingrui Wang, Korbinian Moller, Roberto Brusnicki, Baha Zarrouki, Alessio Gambi, Jan Frederik Totz, Kai Storms, Steven Peters, Andrea Stocco, Bassam Alrifaee, Marco Pavone, Johannes Betz
For autonomous vehicles, safe navigation in complex environments depends on handling a broad range of diverse and rare driving scenarios. Simulation- and scenario-based testing have emerged as key approaches to development and validation of autonomous driving systems. Traditional scenario generation relies on rule-based systems, knowledge-driven models, and data-driven synthesis, often producing limited diversity and unrealistic safety-critical cases. With the emergence of foundation models, which represent a new generation of pre-trained, general-purpose AI models, developers can process heterogeneous inputs (e.g., natural language, sensor data, HD maps, and control actions), enabling the synthesis and interpretation of complex driving scenarios. In this paper, we conduct a survey about the application of foundation models for scenario generation and scenario analysis in autonomous driving (as of May 2025). Our survey presents a unified taxonomy that includes large language models, vision-language models, multimodal large language models, diffusion models, and world models for the generation and analysis of autonomous driving scenarios. In addition, we review the methodologies, open-source datasets, simulation platforms, and benchmark challenges, and we examine the evaluation metrics tailored explicitly to scenario generation and analysis. Finally, the survey concludes by highlighting the open challenges and research questions, and outlining promising future research directions. All reviewed papers are listed in a continuously maintained repository, which contains supplementary materials and is available at https://github.com/TUM-AVS/FM-for-Scenario-Generation-Analysis.
comment : Revised manuscript with separate evaluation metrics table
♻ ☆ Watch and Learn: Learning to Use Computers from Online Videos
Chan Hee Song, Yiwen Song, Palash Goyal, Yu Su, Oriana Riva, Hamid Palangi, Tomas Pfister
Computer-using agents (CUAs) must plan task workflows across diverse and evolving applications, yet progress is limited by the lack of large-scale, high-quality training data. Existing datasets are narrow, static, and costly to annotate, while synthetic data often yields oversimplified or misaligned behaviors. We present Watch & Learn (W&L), a framework that converts readily available Internet videos of human computer use into executable UI trajectories at scale. Instead of directly generating actions or relying on handcrafted heuristics, we cast trajectory annotation as an inverse dynamics problem that predicts user actions from consecutive screen states, which simplifies learning and generalizes across domains. Through a task-aware retrieval and labeling pipeline, W&L yields over 53K high-quality trajectories that enhance CUAs both as in-context exemplars and as supervised training data. On OSWorld, it consistently improves general-purpose and specialized CUAs, while on WindowsAgentArena it achieves state-of-the-art performance among 7B-scale models under the 15-step limit. These results show that web-scale human demonstration videos can serve as a practical and scalable foundation for advancing real-world CUAs.
♻ ☆ MCTS -SQL: Light-Weight LLMs can Master the Text-to-SQL through Monte Ca rl o Tree Search AAAI 2026
Shuozhi Yuan, Limin Chen, Miaomiao Yuan, Zhao Jin
Text-to-SQL is a fundamental yet challenging task in the NLP area, aiming at translating natural language questions into SQL queries. While recent advances in large language models have greatly improved performance, most existing approaches depend on models with tens of billions of parameters or costly APIs, limiting their applicability in resource-constrained environments. For real world, especially on edge devices, it is crucial for Text-to-SQL to ensure cost-effectiveness. Therefore, enabling the light-weight models for Text-to-SQL is of great practical significance. However, smaller LLMs often struggle with complicated user instruction, redundant schema linking or syntax correctness. To address these challenges, we propose MCTS-SQL, a novel framework that uses Monte Carlo Tree Search to guide SQL generation through multi-step refinement. Since the light-weight models' weak performance of single-shot prediction, we generate better results through several trials with feedback. However, directly applying MCTS-based methods inevitably leads to significant time and computational overhead. Driven by this issue, we propose a token-level prefix-cache mechanism that stores prior information during iterations, effectively improved the execution speed. Experiments results on the SPIDER and BIRD benchmarks demonstrate the effectiveness of our approach. Using a small open-source Qwen2.5-Coder-1.5B, our method outperforms ChatGPT-3.5. When leveraging a more powerful model Gemini 2.5 to explore the performance upper bound, we achieved results competitive with the SOTA. Our findings demonstrate that even small models can be effectively deployed in practical Text-to-SQL systems with the right strategy.
comment : Accepted by AAAI 2026
♻ ☆ On the Superimposed Noise Accumulation Problem in Sequential Knowledge Editing of Large Language Models
Ding Cao, Yuchen Cai, Yuqing Huang, Xuesong He, Rongxi Guo, Guiquan Liu, Guangzhong Sun
Sequential knowledge editing techniques aim to continuously update knowledge in large language models at low cost, preventing models from generating outdated or incorrect information. However, existing sequential editing methods suffer from a significant decline in editing success rates after long-term editing. Through theoretical analysis and experiments, our findings reveal that as the number of edits increases, the model's output increasingly deviates from the desired target, leading to a drop in editing success rates. We refer to this issue as the superimposed noise accumulation problem. Our further analysis demonstrates that the problem is related to the erroneous activation of irrelevant knowledge and conflicts between activated knowledge. Based on this analysis, a method named DeltaEdit is proposed that reduces conflicts between knowledge through dynamic orthogonal constraint strategies. Experiments show that DeltaEdit significantly reduces superimposed noise, achieving a 16.8% improvement in editing performance over the strongest baseline.
♻ ☆ WritingBench: A Comprehensive Benchmark for Generative Writing
Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, Fei Huang
Recent advancements in large language models (LLMs) have significantly enhanced text generation capabilities, yet evaluating their performance in generative writing remains a challenge. Existing benchmarks primarily focus on generic text generation or limited in writing tasks, failing to capture the diverse requirements of high-quality written contents across various domains. To bridge this gap, we present WritingBench, a comprehensive benchmark designed to evaluate LLMs across 6 core writing domains and 100 subdomains. We further propose a query-dependent evaluation framework that empowers LLMs to dynamically generate instance-specific assessment criteria. This framework is complemented by a fine-tuned critic model for criteria-aware scoring, enabling evaluations in style, format and length. The framework's validity is further demonstrated by its data curation capability, which enables a 7B-parameter model to outperform the performance of GPT-4o in writing. We open-source the benchmark, along with evaluation tools and modular framework components, to advance the development of LLMs in writing.
♻ ☆ A Flat Minima Perspective on Understanding Augmentations and Model Robustness AAAI
Weebum Yoo, Sung Whan Yoon
Model robustness indicates a model's capability to generalize well on unforeseen distributional shifts, including data corruption, adversarial attacks, and domain shifts. Data augmentation is one of the prevalent and effective ways to enhance robustness. Despite the great success of augmentations in different fields, a general theoretical understanding of their efficacy in improving model robustness is lacking. We offer a unified theoretical framework to clarify how augmentations can enhance model robustness through the lens of loss surface flatness and PAC generalization bound. Our work diverges from prior studies in that our analysis i) broadly encompasses much of the existing augmentation methods, and ii) is not limited to specific types of distribution shifts like adversarial attacks. We confirm our theories through simulations on the existing common corruption and adversarial robustness benchmarks based on the CIFAR and ImageNet datasets, as well as domain generalization benchmarks including PACS and OfficeHome.
comment : In Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI) 2026, Singapore
♻ ☆ WaveC2R: Wavelet-Driven Coarse-to-Refined Hierarchical Learning for Radar Retrieval AAAI2026
Chunlei Shi, Han Xu, Yinghao Li, Yi-Lin Wei, Yongchao Feng, Yecheng Zhang, Dan Niu
Satellite-based radar retrieval methods are widely employed to fill coverage gaps in ground-based radar systems, especially in remote areas affected by terrain blockage and limited detection range. Existing methods predominantly rely on overly simplistic spatial-domain architectures constructed from a single data source, limiting their ability to accurately capture complex precipitation patterns and sharply defined meteorological boundaries. To address these limitations, we propose WaveC2R, a novel wavelet-driven coarse-to-refined framework for radar retrieval. WaveC2R integrates complementary multi-source data and leverages frequency-domain decomposition to separately model low-frequency components for capturing precipitation patterns and high-frequency components for delineating sharply defined meteorological boundaries. Specifically, WaveC2R consists of two stages (i)Intensity-Boundary Decoupled Learning, which leverages wavelet decomposition and frequency-specific loss functions to separately optimize low-frequency intensity and high-frequency boundaries; and (ii)Detail-Enhanced Diffusion Refinement, which employs frequency-aware conditional priors and multi-source data to progressively enhance fine-scale precipitation structures while preserving coarse-scale meteorological consistency. Experimental results on the publicly available SEVIR dataset demonstrate that WaveC2R achieves state-of-the-art performance in satellite-based radar retrieval, particularly excelling at preserving high-intensity precipitation features and sharply defined meteorological boundaries.
comment : This work has been accepted by AAAI2026, AAAI2026 Project's webpage at this URL:https://spring-lovely.github.io/WaveC2R/
♻ ☆ OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe
Kaichen Zhang, Keming Wu, Zuhao Yang, Kairui Hu, Bin Wang, Ziwei Liu, Xingxuan Li, Lidong Bing
Recent advancements in large reasoning models have fueled growing interest in extending such capabilities to multimodal domains. However, despite notable progress in visual reasoning, the lack of transparent and reproducible data curation and training strategies remains a major barrier to scalable research. In this work, we introduce OpenMMReasoner, a fully transparent two-stage recipe for multimodal reasoning spanning supervised fine-tuning (SFT) and reinforcement learning (RL). In the SFT stage, we construct an 874K-sample cold-start dataset with rigorous step-by-step validation, providing a strong foundation for reasoning capabilities. The subsequent RL stage leverages a 74K-sample dataset across diverse domains to further sharpen and stabilize these abilities, resulting in a more robust and efficient learning process. Extensive evaluations demonstrate that our training recipe not only surpasses strong baselines but also highlights the critical role of data quality and training design in shaping multimodal reasoning performance. Notably, our method achieves a 11.6% improvement over the Qwen2.5-VL-7B-Instruct baseline across nine multimodal reasoning benchmarks, establishing a solid empirical foundation for future large-scale multimodal reasoning research. We open-sourced all our codes, pipeline, and data at https://github.com/EvolvingLMMs-Lab/OpenMMReasoner.
♻ ☆ Scaling Spatial Intelligence with Multimodal Foundation Models
Zhongang Cai, Ruisi Wang, Chenyang Gu, Fanyi Pu, Junxiang Xu, Yubo Wang, Wanqi Yin, Zhitao Yang, Chen Wei, Qingping Sun, Tongxi Zhou, Jiaqi Li, Hui En Pang, Oscar Qian, Yukun Wei, Zhiqian Lin, Xuanke Shi, Kewang Deng, Xiaoyang Han, Zukai Chen, Xiangyu Fan, Hanming Deng, Lewei Lu, Liang Pan, Bo Li, Ziwei Liu, Quan Wang, Dahua Lin, Lei Yang
Despite remarkable progress, multimodal foundation models still exhibit surprising deficiencies in spatial intelligence. In this work, we explore scaling up multimodal foundation models to cultivate spatial intelligence within the SenseNova-SI family, built upon established multimodal foundations including visual understanding models (i.e., Qwen3-VL and InternVL3) and unified understanding and generation models (i.e., Bagel). We take a principled approach to constructing high-performing and robust spatial intelligence by systematically curating SenseNova-SI-8M: eight million diverse data samples under a rigorous taxonomy of spatial capabilities. SenseNova-SI demonstrates unprecedented performance across a broad range of spatial intelligence benchmarks: 68.7% on VSI-Bench, 43.3% on MMSI, 85.6% on MindCube, 54.6% on ViewSpatial, and 50.1% on SITE, while maintaining strong general multimodal understanding (e.g., 84.9% on MMBench-En). More importantly, we analyze the impact of data scaling, discuss early signs of emergent generalization capabilities enabled by diverse data training, analyze the risk of overfitting and language shortcuts, present a preliminary study on spatial chain-of-thought reasoning, and validate the potential downstream application. SenseNova-SI is an ongoing project, and this report will be updated continuously. All newly trained multimodal foundation models are publicly released to facilitate further research in this direction.
comment : Codebase: https://github.com/OpenSenseNova/SenseNova-SI; Models: https://huggingface.co/collections/sensenova/sensenova-si
♻ ☆ G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
Wenbo Hu, Jingli Lin, Yilin Long, Yunlong Ran, Lihan Jiang, Yifan Wang, Chenming Zhu, Runsen Xu, Tai Wang, Jiangmiao Pang
Vision-Language Models (VLMs) still lack robustness in spatial intelligence, demonstrating poor performance on spatial understanding and reasoning tasks. We attribute this gap to the absence of a visual geometry learning process capable of reconstructing 3D space from 2D images. We present G$^2$VLM, a geometry grounded vision-language model that bridges two fundamental aspects of spatial intelligence: spatial 3D reconstruction and spatial understanding. G$^2$VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning. Our unified design is highly scalable for spatial understanding: it trains on abundant multi-view image and video data, while simultaneously leveraging the benefits of 3D visual priors that are typically only derived from hard-to-collect annotations. Experimental results demonstrate G$^2$VLM is proficient in both tasks, achieving comparable results to state-of-the-art feed-forward 3D reconstruction models and achieving better or competitive results across spatial understanding and reasoning tasks. By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G$^2$VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing.
comment : code are released at https://github.com/InternRobotics/G2VLM
♻ ☆ KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
Yuxuan Tian, Zihan Wang, Yebo Peng, Aomufei Yuan, Zhiming Wang, Bairen Yi, Xin Liu, Yong Cui, Tong Yang
Efficient inference of large language models (LLMs) is hindered by an ever-growing key-value (KV) cache, making KV cache compression a critical research direction. Traditional methods selectively evict less important KV cache entries, which leads to information loss and hallucinations. Recently, merging-based strategies have been explored to retain more information by merging KV pairs that would be discarded; however, these existing approaches inevitably introduce inconsistencies in attention distributions before and after merging, causing degraded generation quality. To overcome this challenge, we propose KeepKV, a novel adaptive KV cache merging method designed to preserve performance under strict memory constraints, achieving single-step lossless compression and providing error bounds for multi-step compression. KeepKV introduces the Electoral Votes mechanism that records merging history and adaptively adjusts attention scores. Moreover, it further leverages a novel Zero Inference-Perturbation Merging method, compensating for attention loss resulting from cache merging. Extensive experiments on various benchmarks and LLM architectures demonstrate that KeepKV substantially reduces memory usage while successfully retaining essential context information, achieving over 2x inference throughput improvement and maintaining superior generation quality even with only 10% KV cache budgets.
comment : 14 pages, 20 figures
♻ ☆ IVY-FAKE: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detection
Changjiang Jiang, Wenhui Dong, Zhonghao Zhang, Chenyang Si, Fengchang Yu, Wei Peng, Xinbin Yuan, Yifei Bi, Ming Zhao, Zian Zhou, Caifeng Shan
The rapid development of Artificial Intelligence Generated Content (AIGC) techniques has enabled the creation of high-quality synthetic content, but it also raises significant security concerns. Current detection methods face two major limitations: (1) the lack of multidimensional explainable datasets for generated images and videos. Existing open-source datasets (e.g., WildFake, GenVideo) rely on oversimplified binary annotations, which restrict the explainability and trustworthiness of trained detectors. (2) Prior MLLM-based forgery detectors (e.g., FakeVLM) exhibit insufficiently fine-grained interpretability in their step-by-step reasoning, which hinders reliable localization and explanation. To address these challenges, we introduce Ivy-Fake, the first large-scale multimodal benchmark for explainable AIGC detection. It consists of over 106K richly annotated training samples (images and videos) and 5,000 manually verified evaluation examples, sourced from multiple generative models and real world datasets through a carefully designed pipeline to ensure both diversity and quality. Furthermore, we propose Ivy-xDetector, a reinforcement learning model based on Group Relative Policy Optimization (GRPO), capable of producing explainable reasoning chains and achieving robust performance across multiple synthetic content detection benchmarks. Extensive experiments demonstrate the superiority of our dataset and confirm the effectiveness of our approach. Notably, our method improves performance on GenImage from 86.88% to 96.32%, surpassing prior state-of-the-art methods by a clear margin.
comment : 30 pages
♻ ☆ Monet: Reasoning in Latent Visual Space Beyond Images and Language
Qixun Wang, Yang Shi, Yifei Wang, Yuanxing Zhang, Pengfei Wan, Kun Gai, Xianghua Ying, Yisen Wang
"Thinking with images" has emerged as an effective paradigm for advancing visual reasoning, extending beyond text-only chains of thought by injecting visual evidence into intermediate reasoning steps. However, existing methods fall short of human-like abstract visual thinking, as their flexibility is fundamentally limited by external tools. In this work, we introduce Monet, a training framework that enables multimodal large language models (MLLMs) to reason directly within the latent visual space by generating continuous embeddings that function as intermediate visual thoughts. We identify two core challenges in training MLLMs for latent visual reasoning: high computational cost in latent-vision alignment and insufficient supervision over latent embeddings, and address them with a three-stage distillation-based supervised fine-tuning (SFT) pipeline. We further reveal a limitation of applying GRPO to latent reasoning: it primarily enhances text-based reasoning rather than latent reasoning. To overcome this, we propose VLPO (Visual-latent Policy Optimization), a reinforcement learning method that explicitly incorporates latent embeddings into policy gradient updates. To support SFT, we construct Monet-SFT-125K, a high-quality text-image interleaved CoT dataset containing 125K real-world, chart, OCR, and geometry CoTs. Our model, Monet-7B, shows consistent gains across real-world perception and reasoning benchmarks and exhibits strong out-of-distribution generalization on challenging abstract visual reasoning tasks. We also empirically analyze the role of each training component and discuss our early unsuccessful attempts, providing insights for future developments in visual latent reasoning. Our model, data, and code are available at https://github.com/NOVAglow646/Monet.
♻ ☆ Simulated patient systems powered by large language model-based AI agent s offer potential for transforming medical education
Huizi Yu, Jiayan Zhou, Lingyao Li, Shan Chen, Jack Gallifant, Anye Shi, Xiang Li, Jingxian He, Wenyue Hua, Mingyu Jin, Guang Chen, Yang Zhou, Zhao Li, Trisha Gupte, Ming-Li Chen, Zahra Azizi, Qi Dou, Bryan P. Yan, Yongfeng Zhang, Yanqiu Xing, Themistocles L. Danielle S. Bitterman, Themistocles L. Assimes, Xin Ma, Lin Lu, Lizhou Fan
Background: Simulated patient systems are important in medical education and research, providing safe, integrative training environments and supporting clinical decision making. Advances in artificial intelligence (AI), especially large language models (LLMs), can enhance simulated patients by replicating medical conditions and doctor patient interactions with high fidelity and at low cost, but effectiveness and trustworthiness remain open challenges. Methods: We developed AIPatient, a simulated patient system powered by LLM based AI agents. The system uses a retrieval augmented generation (RAG) framework with six task specific agents for complex reasoning. To improve realism, it is linked to the AIPatient knowledge graph built from de identified real patient data in the MIMIC III intensive care database. Results: We evaluated electronic health record (EHR) based medical question answering (QA), readability, robustness, stability, and user experience. AIPatient reached 94.15 percent QA accuracy when all six agents were enabled, outperforming versions with partial or no agent integration. The knowledge base achieved an F1 score of 0.89. Readability scores showed a median Flesch Reading Ease of 68.77 and a median Flesch Kincaid Grade of 6.4, indicating accessibility for most medical trainees and clinicians. Robustness and stability were supported by non significant variance in repeated trials (analysis of variance F value 0.61, p greater than 0.1; F value 0.78, p greater than 0.1). A user study with medical students showed that AIPatient provides high fidelity, usability, and educational value, comparable to or better than human simulated patients for history taking. Conclusions: LLM based simulated patient systems can deliver accurate, readable, and reliable medical encounters and show strong potential to transform medical education.
comment : 19 pages, 6 figures, 4 tables
♻ ☆ Extensible Multi-Granularity Fusion Network and Transferable Curriculum Learning for Aspect-based Sentiment Analysis
Xinran Li, Xiaowei Zhao, Yubo Zhu, Zhiheng Zhang, Zhiqi Huang, Hongkun Song, Jinglu Hu, Xinze Che, Yifan Lyu, Yong Zhou, Xiujuan Xu
Aspect-based Sentiment Analysis (ABSA) aims to determine sentiment polarity toward specific aspects in text. Existing methods enrich semantic and syntactic representations through external knowledge or GNNs, but the growing diversity of linguistic features increases model complexity and lacks a unified, extensible framework. We propose an Extensible Multi-Granularity Fusion Network (EMGF) that integrates dependency syntax, constituent syntax, attention-based semantics, and external knowledge graphs. EMGF employs multi-anchor triplet learning and orthogonal projection to effectively fuse multi-granularity features and strengthen their interactions without additional computational overhead. Furthermore, we introduce the first task-specific curriculum learning framework for text-only ABSA, which assigns difficulty scores using five indicators and trains the model from easy to hard to mimic human learning and improve generalization. Experiments on SemEval 2014, Twitter, and MAMS datasets show that EMGF+CL consistently outperforms state-of-the-art ABSA models.
comment : 8 pages, 4 figures
♻ ☆ CANVAS: A Benchmark for Vision -Language Models on Tool-Based User Interface Design
Daeheon Jeong, Seoyeon Byun, Kihoon Son, Dae Hyun Kim, Juho Kim
User interface (UI) design is an iterative process in which designers progressively refine their work with design software such as Figma or Sketch. Recent advances in vision language models (VLMs) with tool invocation suggest these models can operate design software to edit a UI design through iteration. Understanding and enhancing this capacity is important, as it highlights VLMs' potential to collaborate with designers within conventional software. However, as no existing benchmark evaluates tool-based design performance, the capacity remains unknown. To address this, we introduce CANVAS, a benchmark for VLMs on tool-based user interface design. Our benchmark contains 598 tool-based design tasks paired with ground-truth references sampled from 3.3K mobile UI designs across 30 function-based categories (e.g., onboarding, messaging). In each task, a VLM updates the design step-by-step through context-based tool invocations (e.g., create a rectangle as a button background), linked to design software. Specifically, CANVAS incorporates two task types: (i) design replication evaluates the ability to reproduce a whole UI screen; (ii) design modification evaluates the ability to modify a specific part of an existing screen. Results suggest that leading models exhibit more strategic tool invocations, improving design quality. Furthermore, we identify common error patterns models exhibit, guiding future work in enhancing tool-based design capabilities.
♻ ☆ Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management SIGMOD'26
Xinjun Yang, Qingda Hu, Junru Li, Feifei Li, Yicong Zhu, Yuqi Zhou, Qiuru Lin, Jian Dai, Yang Kong, Jiayu Zhang, Guoqiang Xu, Qiang Liu
The rapid increase in LLM model sizes and the growing demand for long-context inference have made memory a critical bottleneck in GPU-accelerated serving systems. Although high-bandwidth memory (HBM) on GPUs offers fast access, its limited capacity necessitates reliance on host memory (CPU DRAM) to support larger working sets such as the KVCache. However, the maximum DRAM capacity is constrained by the limited number of memory channels per CPU socket. To overcome this limitation, current systems often adopt RDMA-based disaggregated memory pools, which introduce significant challenges including high access latency, complex communication protocols, and synchronization overhead. Fortunately, the emerging CXL technology introduces new opportunities in KVCache design. In this paper, we propose Beluga, a novel memory architecture that enables GPUs and CPUs to access a shared, large-scale memory pool through CXL switches. By supporting native load/store access semantics over the CXL fabric, our design delivers near-local memory latency, while reducing programming complexity and minimizing synchronization overhead. We conduct a systematic characterization of a commercial CXL switch-based memory pool and propose a set of design guidelines. Based on Beluga, we design and implement Beluga-KVCache, a system tailored for managing the large-scale KVCache in LLM inference. Beluga-KVCache achieves an 89.6% reduction in Time-To-First-Token (TTFT) and 7.35x throughput improvement in the vLLM inference engine compared to RDMA-based solutions. To the best of our knowledge, Beluga is the first system that enables GPUs to directly access large-scale memory pools through CXL switches, marking a significant step toward low-latency, shared access to vast memory resources by GPUs.
comment : 13 pages, accepted by SIGMOD'26
♻ ☆ KurdSTS: The Kurdish Semantic Textual Similarity
Abdulhady Abas Abdullah, Hadi Veisi, Hussein M. Al
Semantic Textual Similarity (STS) measures the degree of meaning overlap between two texts and underpins many NLP tasks. While extensive resources exist for high-resource languages, low-resource languages such as Kurdish remain underserved. We present, to our knowledge, the first Kurdish STS dataset: 10,000 sentence pairs spanning formal and informal registers, each annotated for similarity. We benchmark Sentence-BERT, multilingual BERT, and other strong baselines, obtaining competitive results while highlighting challenges arising from Kurdish morphology, orthographic variation, and code-mixing. The dataset and baselines establish a reproducible evaluation suite and provide a strong starting point for future research on Kurdish semantics and low-resource NLP.
♻ ☆ Learning Plug-and-play Memory for Guiding Video Diffusion Models
Selena Song, Ziming Xu, Zijun Zhang, Kun Zhou, Jiaxian Guo, Lianhui Qin, Biwei Huang
Diffusion Transformer(DiT) based video generation models have recently achieved impressive visual quality and temporal coherence, but they still frequently violate basic physical laws and commonsense dynamics, revealing a lack of explicit world knowledge. In this work, we explore how to equip them with a plug-and-play memory that injects useful world knowledge. Motivated by in-context memory in Transformer-based LLMs, we conduct empirical studies to show that DiT can be steered via interventions on its hidden states, and simple low-pass and high-pass filters in the embedding space naturally disentangle low-level appearance and high-level physical/semantic cues, enabling targeted guidance. Building on these observations, we propose a learnable memory encoder DiT-Mem, composed of stacked 3D CNNs, low-/high-pass filters, and self-attention layers. The encoder maps reference videos into a compact set of memory tokens, which are concatenated as the memory within the DiT self-attention layers. During training, we keep the diffusion backbone frozen, and only optimize the memory encoder. It yields a rather efficient training process on few training parameters (150M) and 10K data samples, and enables plug-and-play usage at inference time. Extensive experiments on state-of-the-art models demonstrate the effectiveness of our method in improving physical rule following and video fidelity. Our code and data are publicly released here: https://thrcle421.github.io/DiT-Mem-Web/.
♻ ☆ Beyond Introspection: Reinforcing Thinking via Externalist Behavioral Feedback
Diji Yang, Linda Zeng, Kezhen Chen, Yi Zhang
While inference-time thinking allows Large Language Models (LLMs) to address complex problems, the extended thinking process can be unreliable or inconsistent because of the model's probabilistic nature, especially near its knowledge boundaries. Existing approaches attempt to mitigate this by having the model critique its own reasoning to make corrections. However, such self-critique inherits the same biases of the original output, known as the introspection illusion. Moving beyond such introspection and inspired by core methodologies in ethology, we propose an externalist three-step framework Distillation-Reinforcement-Reasoning (DRR). Rather than relying on a model's introspection, DRR evaluates its observable behaviors to provide corrective feedback. DRR first distills the reasoner's behavioral traces, then trains a lightweight, external Discriminative Model (DM). At inference time, this DM acts as a critic, identifying and rejecting suspicious reasoning steps. This external feedback compels the LLM to discard flawed pathways and explore alternatives, thereby enhancing reasoning quality without altering the base model. Experiments on multiple reasoning benchmarks show that our framework significantly outperforms prominent self-critique methods. Benefiting from a lightweight and annotation-free design, DRR offers a scalable and adaptable solution for improving the reliability of reasoning in a wide range of LLMs.
♻ ☆ Racket Vision : A Multiple Racket Sports Benchmark for Unified Ball and Racket Analysis AAAI 2026
Linfeng Dong, Yuchen Yang, Hao Wu, Wei Wang, Yuenan Hou, Zhihang Zhong, Xiao Sun
We introduce RacketVision, a novel dataset and benchmark for advancing computer vision in sports analytics, covering table tennis, tennis, and badminton. The dataset is the first to provide large-scale, fine-grained annotations for racket pose alongside traditional ball positions, enabling research into complex human-object interactions. It is designed to tackle three interconnected tasks: fine-grained ball tracking, articulated racket pose estimation, and predictive ball trajectory forecasting. Our evaluation of established baselines reveals a critical insight for multi-modal fusion: while naively concatenating racket pose features degrades performance, a CrossAttention mechanism is essential to unlock their value, leading to trajectory prediction results that surpass strong unimodal baselines. RacketVision provides a versatile resource and a strong starting point for future research in dynamic object tracking, conditional motion forecasting, and multimodal analysis in sports. Project page at https://github.com/OrcustD/RacketVision
comment : Accepted to AAAI 2026 (Oral)
♻ ☆ ICPO: Intrinsic Confidence-Driven Group Relative Preference Optimization for Efficient Reinforcement Learning
Jinpeng Wang, Chao Li, Ting Ye, Mengyuan Zhang, Wei Liu, Jian Luan
Reinforcement Learning with Verifiable Rewards (RLVR) demonstrates significant potential in enhancing the reasoning capabilities of Large Language Models (LLMs). However, existing RLVR methods are often constrained by issues such as coarse-grained rewards, reward noise, and inefficient exploration, which lead to unstable training and entropy collapse. To address this challenge, we propose the Intrinsic Confidence-Driven Group Relative Preference Optimization method (ICPO). The intuition behind it lies in the fact that the probabilities of an LLM generating different responses can inherently and directly reflect its self-assessment of the reasoning process. Inspired by the idea of preference modeling, ICPO calculates a preference advantage score for each response by comparing the relative generation probabilities of multiple responses under the same input prompt, and integrates this score with verifiable rewards to guide the exploration process. We have discovered that the preference advantage score not only alleviates the issues of coarse-grained rewards and reward noise but also effectively curbs overconfident errors, enhances the relative superiority of undervalued high-quality responses, and prevents the model from overfitting to specific strategies, thereby facilitating more thorough exploration. Comprehensive experiments across four general-domain benchmarks and three mathematical benchmarks demonstrate that ICPO steadily boosts reasoning compared to GRPO.
♻ ☆ Machine Unlearning of Traffic State Estimation and Prediction
Xin Wang, R. Tyrrell Rockafellar, Xuegang, Ban
Data-driven traffic state estimation and prediction (TSEP) relies heavily on data sources that contain sensitive information. While the abundance of data has fueled significant breakthroughs, particularly in machine learning-based methods, it also raises concerns regarding privacy, cybersecurity, and data freshness. These issues can erode public trust in intelligent transportation systems. Recently, regulations have introduced the "right to be forgotten", allowing users to request the removal of their private data from models. As machine learning models can remember old data, simply removing it from back-end databases is insufficient in such systems. To address these challenges, this study introduces a novel learning paradigm for TSEP-Machine Unlearning TSEP-which enables a trained TSEP model to selectively forget privacy-sensitive, poisoned, or outdated data. By empowering models to "unlearn," we aim to enhance the trustworthiness and reliability of data-driven traffic TSEP.
♻ ☆ Mavors: Multi-granularity Video Representation for Multimodal Large Language Model
Yang Shi, Jiaheng Liu, Yushuo Guan, Zhenhua Wu, Yuanxing Zhang, Zihao Wang, Weihong Lin, Jingyun Hua, Zekun Wang, Xinlong Chen, Bohan Zeng, Wentao Zhang, Fuzheng Zhang, Wenjing Yang, Di Zhang
Long-context video understanding in multimodal large language models (MLLMs) faces a critical challenge: balancing computational efficiency with the retention of fine-grained spatio-temporal patterns. Existing approaches (e.g., sparse sampling, dense sampling with low resolution, and token compression) suffer from significant information loss in temporal dynamics, spatial details, or subtle interactions, particularly in videos with complex motion or varying resolutions. To address this, we propose $\mathbf{Mavors}$, a novel framework that introduces $\mathbf{M}$ulti-gr$\mathbf{a}$nularity $\mathbf{v}$ide$\mathbf{o}$ $\mathbf{r}$epre$\mathbf{s}$entation for holistic long-video modeling. Specifically, Mavors directly encodes raw video content into latent representations through two core components: 1) an Intra-chunk Vision Encoder (IVE) that preserves high-resolution spatial features via 3D convolutions and Vision Transformers, and 2) an Inter-chunk Feature Aggregator (IFA) that establishes temporal coherence across chunks using transformer-based dependency modeling with chunk-level rotary position encodings. Moreover, the framework unifies image and video understanding by treating images as single-frame videos via sub-image decomposition. Experiments across diverse benchmarks demonstrate Mavors' superiority in maintaining both spatial fidelity and temporal continuity, significantly outperforming existing methods in tasks requiring fine-grained spatio-temporal reasoning.
comment : 22 pages
♻ ☆ Human-AI Programming Role Optimization: Developing a Personality-Driven Self-Determination Framework CCS 2012
Marcel Valovy
As artificial intelligence transforms software development, a critical question emerges: how can developers and AI systems collaborate most effectively? This dissertation optimizes human-AI programming roles through self-determination theory and personality psychology, introducing the Role Optimization Motivation Alignment (ROMA) framework. Through Design Science Research spanning five cycles, this work establishes empirically-validated connections between personality traits, programming role preferences, and collaborative outcomes, engaging 200 experimental participants and 46 interview respondents. Key findings demonstrate that personality-driven role optimization significantly enhances self-determination and team dynamics, yielding 23% average motivation increases among professionals and up to 65% among undergraduates. Five distinct personality archetypes emerge: The Explorer (high Openness/low Agreeableness), The Orchestrator (high Extraversion/Agreeableness), The Craftsperson (high Neuroticism/low Extraversion), The Architect (high Conscientiousness), and The Adapter (balanced profile). Each exhibits distinct preferences for programming roles (Co-Pilot, Co-Navigator, Agent), with assignment modes proving crucial for satisfaction. The dissertation contributes: (1) an empirically-validated framework linking personality traits to role preferences and self-determination outcomes; (2) a taxonomy of AI collaboration modalities mapped to personality profiles while preserving human agency; and (3) an ISO/IEC 29110 extension enabling Very Small Entities to implement personality-driven role optimization within established standards. Keywords: artificial intelligence, human-computer interaction, behavioral software engineering, self-determination theory, personality psychology, phenomenology, intrinsic motivation, pair programming, design science research, ISO/IEC 29110
comment : PhD Dissertation, Prague University of Economics and Business, 2025. 323 pages. ACM CCS 2012: Human-computer interaction, Collaborative interaction, Human-AI collaborative systems, Pair programming, AI-assisted software engineering
♻ ☆ COPO: Causal-Oriented Policy Optimization for Hallucinations of MLLMs
Peizheng Guo, Jingyao Wang, Wenwen Qiang, Jiahuan Zhou, Changwen Zheng, Gang Hua
Despite Multimodal Large Language Models (MLLMs) having shown impressive capabilities, they may suffer from hallucinations. Empirically, we find that MLLMs attend disproportionately to task-irrelevant background regions compared with text-only LLMs, implying spurious background-answer correlations. We claim and analyze that (i) outcome-based rewards can be an important factor leading to spurious correlations, and (ii) spurious correlations can be an important factor leading to hallucinations. Based on these results, we propose Causal-Oriented Policy Optimization (COPO) to mitigate these spurious correlations, thus addressing the issue of hallucinations. It imposes token-level sufficiency and necessity constraints to measure each inference token's causal contribution, thus ensuring correct and evidence-grounded output. Specifically, we first evaluate each token's causal contribution via a newly proposed causal completeness reward. This reward is then used to construct a causally informed advantage function within the GRPO optimization framework, encouraging the model to focus on tokens that are causally sufficient and necessary for accurate generation. Experimental results across various benchmarks demonstrate the advantages of COPO.
♻ ☆ ROVER: Recursive Reasoning Over Videos with Vision -Language Models for Embodied Tasks
Philip Schroeder, Ondrej Biza, Thomas Weng, Hongyin Luo, James Glass
Vision-language models (VLMs) have exhibited impressive capabilities across diverse image understanding tasks, but still struggle in settings that require reasoning over extended sequences of camera frames from a video. This limits their utility in embodied settings, which require reasoning over long frame sequences from a continuous stream of visual input at each moment of a task attempt. To address this limitation, we propose ROVER (Reasoning Over VidEo Recursively), a framework that enables the model to recursively decompose long-horizon video trajectories into segments corresponding to shorter subtasks within the trajectory. In doing so, ROVER facilitates more focused and accurate reasoning over temporally localized frame sequences without losing global context. We evaluate ROVER, implemented using an in-context learning approach, on diverse OpenX Embodiment videos and on a new dataset derived from RoboCasa that consists of 543 videos showing both expert and perturbed non-expert trajectories across 27 robotic manipulation tasks. ROVER outperforms strong baselines across three video reasoning tasks: task progress estimation, frame-level natural language reasoning, and video question answering. We observe that, by reducing the number of frames the model reasons over at each timestep, ROVER mitigates hallucinations, especially during unexpected or non-optimal moments of a trajectory. In addition, by enabling the implementation of a subtask-specific sliding context window, ROVER's time complexity scales linearly with video length, an asymptotic improvement over baselines. Demos, code, and data available at: https://rover-vlm.github.io
♻ ☆ Searching Meta Reasoning Skeleton to Guide LLM Reasoning
Ziying Zhang, Yaqing Wang, Quanming Yao
Meta reasoning behaviors work as a skeleton to guide large language model (LLM) reasoning, thus help to improve reasoning performance. However, prior researches implement meta reasoning skeleton with manually designed structure, limiting ability to adapt to query-specific requirement and capture intricate logical dependency among reasoning steps. To deal with the challenges, we represent meta reasoning skeleton with directed acyclic graph (DAG) to unify skeletons proposed in prior works and model intricate logical dependency. Then we propose AutoMR, a framework that searches for query-aware meta reasoning skeleton automatically inspired by automated machine learning (AutoML). Specifically, we construct search space based on DAG representation of skeleton and then formulate the search problem. We design a dynamic skeleton sampling algorithm by expanding meta reasoning skeleton along with reasoning context at inference time. This algorithm can derive any meta reasoning skeleton in search space efficiently and adapt skeleton to evolving base reasoning context, thus enable efficient query-aware skeleton search. We conduct experiments on extensive benchmark datasets. Experimental results show that AutoMR achieves better reasoning performance than previous works broadly.
♻ ☆ Human Experts' Evaluation of Generative AI for Context ualizing STEAM Education in the Global South
Matthew Nyaaba, Macharious Nabang, Patrick Kyeremeh, Ibrahim Nantomah, Collins Owusu-Fordjour, Martin Ako, Bismark Nyaaba Akanzire, Kassim Korah Nantomah, Cecilia Issaka, Xiaoming Zhai
STEAM education in many parts of the Global South remains abstract and weakly connected to learners sociocultural realities. This study examines how human experts evaluate the capacity of Generative AI (GenAI) to contextualize STEAM instruction in these settings. Using a convergent mixed-methods design grounded in human-centered and culturally responsive pedagogy, four STEAM education experts reviewed standardized Ghana NaCCA lesson plans and GenAI-generated lessons created with a customized Culturally Responsive Lesson Planner (CRLP). Quantitative data were collected with a validated 25-item Culturally Responsive Pedagogy Rubric assessing bias awareness, cultural representation, contextual relevance, linguistic responsiveness, and teacher agency. Qualitative reflections provided additional insight into the pedagogical and cultural dynamics of each lesson. Findings show that GenAI, especially through the CRLP, improved connections between abstract standards and learners lived experiences. Teacher Agency was the strongest domain, while Cultural Representation was the weakest. CRLP-generated lessons were rated as more culturally grounded and pedagogically engaging. However, GenAI struggled to represent Ghana's cultural diversity, often producing surface-level references, especially in Mathematics and Computing. Experts stressed the need for teacher mediation, community input, and culturally informed refinement of AI outputs. Future work should involve classroom trials, broader expert participation, and fine-tuning with Indigenous corpora.
♻ ☆ Financial Risk Relation Identification through Dual-view Adaptation EMNLP 2025
Wei-Ning Chiu, Yu-Hsiang Wang, Andy Hsiao, Yu-Shiang Huang, Chuan-Ju Wang
A multitude of interconnected risk events -- ranging from regulatory changes to geopolitical tensions -- can trigger ripple effects across firms. Identifying inter-firm risk relations is thus crucial for applications like portfolio management and investment strategy. Traditionally, such assessments rely on expert judgment and manual analysis, which are, however, subjective, labor-intensive, and difficult to scale. To address this, we propose a systematic method for extracting inter-firm risk relations using Form 10-K filings -- authoritative, standardized financial documents -- as our data source. Leveraging recent advances in natural language processing, our approach captures implicit and abstract risk connections through unsupervised fine-tuning based on chronological and lexical patterns in the filings. This enables the development of a domain-specific financial encoder with a deeper contextual understanding and introduces a quantitative risk relation score for transparency, interpretable analysis. Extensive experiments demonstrate that our method outperforms strong baselines across multiple evaluation settings. Our codes are available at https://github.com/cnclabs/codes.fin.relation.
comment : 11 pages, 3 figures, EMNLP 2025 Main Conference
♻ ☆ Manifold-Aware Diffusion -Augmented Contrastive Learning for Noise-Robust Biosignal Representation
Rami Zewail
Learning robust representations for physiological time-series signals continues to pose a substantial challenge in developing efficient few-shot learning applications. This difficulty is largely due to the complex pathological variations in biosignals. In this context, this paper introduces a manifold-aware Diffusion-Augmented Contrastive Learning (DACL) framework, which efficiently leverages the generative structure of latent diffusion models with the discriminative power of supervised contrastive learning. The proposed framework operates within a contextualized scattering latent space derived from Scattering Transformer (ST) features. Within a contrastive learning framework, we employ a forward diffusion process in the scattering latent space as a structured manifold-aware feature augmentation technique. We assessed the proposed framework using the PhysioNet 2017 ECG benchmark dataset. The proposed method achieved a competitive AUROC of 0.9741 in the task of detecting atrial fibrillation from a single-lead ECG signal. The proposed framework achieved performance on par with relevant state-of-the-art related works. In-depth evaluation findings suggest that early-stage diffusion serves as an ideal "local manifold explorer," producing embeddings with greater precision than typical augmentation methods while preserving inference efficiency.
♻ ☆ Memo: Training Memory-Efficient Embodied Agent s with Reinforcement Learning NeurIPS 2025
Gunshi Gupta, Karmesh Yadav, Zsolt Kira, Yarin Gal, Rahaf Aljundi
To enable embodied agents to operate effectively over extended timeframes, it is crucial to develop models that form and access memories to stay contextualized in their environment. In the current paradigm of training transformer-based policies for embodied sequential decision-making tasks, visual inputs often overwhelm the context limits of transformers, while humans can maintain and utilize a lifetime of experience compressed as memories. Significant compression is possible in principle, as much of the input is irrelevant and can be abstracted. However, existing approaches predominantly focus on either recurrent models with fixed-size memory or transformers with full-context reliance. In this work, we propose Memo, a transformer-based architecture and training recipe for reinforcement learning (RL) on memory-intensive, long-horizon tasks. Memo incorporates the creation and retrieval of memory by interleaving periodic summarization tokens with the inputs of a model during training. We demonstrate Memo's effectiveness on a gridworld meta-RL benchmark and a multi-object navigation task in photo-realistic indoor settings. Memo outperforms naive long-context transformer baselines while being more compute and storage efficient. Additionally, Memo generalizes better to longer contexts at inference time and remains robust in streaming settings, where historical context must be truncated to fit inference constraints. Our code is available at: https://github.com/gunshi/memo.
comment : Accepted for Spotlight Presentation at NeurIPS 2025
♻ ☆ Accelerating Training of Recursive Reasoning Models with Curriculum Guided Adaptive Recursion
Kaleem Ullah Qasim, Jiashu Zhang
Recursive reasoning models achieve remarkable performance on complex reasoning tasks through iterative refinement, enabling tiny networks to match large language models thousands of times their size. However, training remains computationally expensive, prior work reporting approximately 36 GPU-hours per dataset, limiting broader adoption and research. We propose CGAR, a novel training methodology that applies curriculum learning to architectural depth rather than traditional data ordering. CGAR introduces two synergistic components: Progressive Depth Curriculum dynamically adjusts recursion depth from shallow to deep configurations during training, preventing early overfitting while reducing computational cost, and Hierarchical Supervision Weighting applies exponentially decaying importance to supervision steps, aligning loss weighting with observed gradient magnitude decay. On Sudoku-Extreme with 423,168 test puzzles, CGAR achieves 1.71x training speedup (10.93 to 6.38 hours, 42% cost reduction) with only 0.63% accuracy drop (86.65% to 86.02%). Systematic ablations reveal Progressive Depth Curriculum alone achieves 2.26x speedup with 85.47% accuracy, demonstrating a rare Pareto improvement where architectural curriculum simultaneously enhances training efficiency and solution quality. CGAR-trained models exhibit superior inference efficiency with 100% halting accuracy and 11% fewer reasoning steps. Our work demonstrates that principled curriculum on architectural depth enables efficient training of recursive reasoning models on modest hardware. Code and models: https://github.com/Kaleemullahqasim/CGAR and https://huggingface.co/Kaleemullah/trm-cgar-sudoku
♻ ☆ PathGene: Benchmark ing Driver Gene Mutations and Exon Prediction Using Multicenter Lung Cancer Histopathology Image Dataset
Liangrui Pan, Qingchun Liang, Shen Zhao, Songqing Fan, Shaoliang Peng
Accurately predicting gene mutations, mutation subtypes and their exons in lung cancer is critical for personalized treatment planning and prognostic assessment. Faced with regional disparities in medical resources and the high cost of genomic assays, using artificial intelligence to infer these mutations and exon variants from routine histopathology images could greatly facilitate precision therapy. Although some prior studies have shown that deep learning can accelerate the prediction of key gene mutations from lung cancer pathology slides, their performance remains suboptimal and has so far been limited mainly to early screening tasks. To address these limitations, we have assembled PathGene, which comprises histopathology images paired with next-generation sequencing reports from 1,576 patients at the Second Xiangya Hospital, Central South University, and 448 TCGA-LUAD patients. This multi-center dataset links whole-slide images to driver gene mutation status, mutation subtypes, exon, and tumor mutational burden (TMB) status, with the goal of leveraging pathology images to predict mutations, subtypes, exon locations, and TMB for early genetic screening and to advance precision oncology. Unlike existing datasets, we provide molecular-level information related to histopathology images in PathGene to facilitate the development of biomarker prediction models. We benchmarked 11 multiple-instance learning methods on PathGene for mutation, subtype, exon, and TMB prediction tasks. These experimental methods provide valuable alternatives for early genetic screening of lung cancer patients and assisting clinicians to quickly develop personalized precision targeted treatment plans for patients. Code and data are available at https://github.com/panliangrui/NIPS2025/.
comment : Withdrawn due to issues related to data permissions/ethics
♻ ☆ Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMs
Yunqi Hong, Sohyun An, Andrew Bai, Neil Y. C. Lin, Cho-Jui Hsieh
Despite Multimodal Large Language Models (MLLMs) showing promising results on general zero-shot image classification tasks, fine-grained image classification remains challenging. It demands precise attention to subtle visual details to distinguish between visually similar subcategories--details that MLLMs may easily overlook without explicit guidance. To address this, we introduce AutoSEP, an iterative self-supervised prompt learning framework designed to enhance MLLM fine-grained classification capabilities in a fully unsupervised manner. Our core idea is to leverage unlabeled data to learn a description prompt that guides MLLMs in identifying crucial discriminative features within an image, and boosts classification accuracy. We developed an automatic self-enhancing prompt learning framework called AutoSEP to iteratively improve the description prompt using unlabeled data, based on instance-level classification scoring function. AutoSEP only requires black-box access to MLLMs, eliminating the need for any training or fine-tuning. We evaluate our approach on multiple fine-grained classification datasets. It consistently outperforms other unsupervised baselines, demonstrating the effectiveness of our self-supervised optimization framework. Notably, AutoSEP on average improves 13 percent over standard zero-shot classification and 5 percent over the best-performing baselines. Code is available at: https://github.com/yq-hong/AutoSEP
Computer Vision and Pattern Recognition 207
☆ LC4-DViT: Land-cover Creation for Land-cover Classification with Deformable Vision Transformer
Kai Wang, Siyi Chen, Weicong Pang, Chenchen Zhang, Renjun Gao, Ziru Chen, Cheng Li, Dasa Gu, Rui Huang, Alexis Kai Hon Lau
Land-cover underpins ecosystem services, hydrologic regulation, disaster-risk reduction, and evidence-based land planning; timely, accurate land-cover maps are therefore critical for environmental stewardship. Remote sensing-based land-cover classification offers a scalable route to such maps but is hindered by scarce and imbalanced annotations and by geometric distortions in high-resolution scenes. We propose LC4-DViT (Land-cover Creation for Land-cover Classification with Deformable Vision Transformer), a framework that combines generative data creation with a deformation-aware Vision Transformer. A text-guided diffusion pipeline uses GPT-4o-generated scene descriptions and super-resolved exemplars to synthesize class-balanced, high-fidelity training images, while DViT couples a DCNv4 deformable convolutional backbone with a Vision Transformer encoder to jointly capture fine-scale geometry and global context. On eight classes from the Aerial Image Dataset (AID)-Beach, Bridge, Desert, Forest, Mountain, Pond, Port, and River-DViT achieves 0.9572 overall accuracy, 0.9576 macro F1-score, and 0.9510 Cohen' s Kappa, improving over a vanilla ViT baseline (0.9274 OA, 0.9300 macro F1, 0.9169 Kappa) and outperforming ResNet50, MobileNetV2, and FlashInternImage. Cross-dataset experiments on a three-class SIRI-WHU subset (Harbor, Pond, River) yield 0.9333 overall accuracy, 0.9316 macro F1, and 0.8989 Kappa, indicating good transferability. An LLM-based judge using GPT-4o to score Grad-CAM heatmaps further shows that DViT' s attention aligns best with hydrologically meaningful structures. These results suggest that description-driven generative augmentation combined with deformation-aware transformers is a promising approach for high-resolution land-cover mapping.
comment : This work has been submitted to the IEEE for possible publication.The project is available at https://github.com/weicongpang/LVC2-DViT.git
☆ From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images
Yiming Chen, Junlin Han, Tianyi Bai, Shengbang Tong, Filippos Kokkinos, Philip Torr
While Multimodal Large Language Models (MLLMs) are adept at answering what is in an image-identifying objects and describing scenes-they often lack the ability to understand how an image feels to a human observer. This gap is most evident when considering subjective cognitive properties, such as what makes an image memorable, funny, aesthetically pleasing, or emotionally evocative. To systematically address this challenge, we introduce CogIP-Bench, a comprehensive benchmark for evaluating MLLMs on such image cognitive properties. Our evaluation reveals a significant gap: current models are poorly aligned with human perception of these nuanced properties. We then demonstrate that a post-training phase can effectively bridge this gap, significantly enhancing the model's alignment with human judgments. Furthermore, we show that this learned cognitive alignment is not merely predictive but also transferable to downstream creative tasks. By integrating our cognitively-aligned MLLM into an image generation pipeline, we can guide the synthesis process to produce images that better embody desired traits, such as being more memorable or visually appealing. Our work provides a benchmark to measure this human-like perception, a post-training pipeline to enhance it, and a demonstration that this alignment unlocks more human-centric AI.
comment : Project page with codes/datasets/models: https://follen-cry.github.io/MLLM-Cognition-project-page/
☆ Wo rl d in a Frame: Understanding Culture Mixing as a New Challenge for Vision -Language Models
Eunsu Kim, Junyeong Park, Na Min An, Junseong Kim, Hitesh Laxmichand Patel, Jiho Jin, Julia Kruk, Amit Agarwal, Srikant Panda, Fenal Ashokbhai Ilasariya, Hyunjung Shim, Alice Oh
In a globalized world, cultural elements from diverse origins frequently appear together within a single visual scene. We refer to these as culture mixing scenarios, yet how Large Vision-Language Models (LVLMs) perceive them remains underexplored. We investigate culture mixing as a critical challenge for LVLMs and examine how current models behave when cultural items from multiple regions appear together. To systematically analyze these behaviors, we construct CultureMix, a food Visual Question Answering (VQA) benchmark with 23k diffusion-generated, human-verified culture mixing images across four subtasks: (1) food-only, (2) food+food, (3) food+background, and (4) food+food+background. Evaluating 10 LVLMs, we find consistent failures to preserve individual cultural identities in mixed settings. Models show strong background reliance, with accuracy dropping 14% when cultural backgrounds are added to food-only baselines, and they produce inconsistent predictions for identical foods across different contexts. To address these limitations, we explore three robustness strategies. We find supervised fine-tuning using a diverse culture mixing dataset substantially improve model consistency and reduce background sensitivity. We call for increased attention to culture mixing scenarios as a critical step toward developing LVLMs capable of operating reliably in culturally diverse real-world environments.
☆ Distracted Robot: How Visual Clutter Undermine Robotic Manipulation
Amir Rasouli, Montgomery Alban, Sajjad Pakdamansavoji, Zhiyuan Li, Zhanguang Zhang, Aaron Wu, Xuan Zhao
In this work, we propose an evaluation protocol for examining the performance of robotic manipulation policies in cluttered scenes. Contrary to prior works, we approach evaluation from a psychophysical perspective, therefore we use a unified measure of clutter that accounts for environmental factors as well as the distractors quantity, characteristics, and arrangement. Using this measure, we systematically construct evaluation scenarios in both hyper-realistic simulation and real-world and conduct extensive experimentation on manipulation policies, in particular vision-language-action (VLA) models. Our experiments highlight the significant impact of scene clutter, lowering the performance of the policies, by as much as 34% and show that despite achieving similar average performance across the tasks, different VLA policies have unique vulnerabilities and a relatively low agreement on success scenarios. We further show that our clutter measure is an effective indicator of performance degradation and analyze the impact of distractors in terms of their quantity and occluding influence. At the end, we show that finetuning on enhanced data, although effective, does not equally remedy all negative impacts of clutter on performance.
comment : 12 figures, 2 tables
☆ Alzheimer's Disease Prediction Using EffNetViTLoRA and BiLSTM with Multimodal Longitudinal MRI Data
Mahdieh Behjat Khatooni, Mohsen Soryani
Alzheimer's disease (AD) is a prevalent neurodegenerative disorder that progressively impairs memory, decision-making, and overall cognitive function. As AD is irreversible, early prediction is critical for timely intervention and management. Mild Cognitive Impairment (MCI), a transitional stage between cognitively normal (CN) aging and AD, plays a significant role in early AD diagnosis. However, predicting MCI progression remains a significant challenge, as not all individuals with MCI convert to AD. MCI subjects are categorized into stable MCI (sMCI) and progressive MCI (pMCI) based on conversion status. In this study, we propose a generalized, end-to-end deep learning model for AD prediction using MCI cases from the Alzheimer's Disease Neuroimaging Initiative (ADNI). Our hybrid architecture integrates Convolutional Neural Networks and Vision Transformers to capture both local spatial features and global contextual dependencies from Magnetic Resonance Imaging (MRI) scans. To incorporate temporal progression, we further employ Bidirectional Long Short-Term Memory (BiLSTM) networks to process features extracted from four consecutive MRI timepoints along with some other non-image biomarkers, predicting each subject's cognitive status at month 48. Our multimodal model achieved an average progression prediction accuracy of 95.05\% between sMCI and pMCI, outperforming existing studies in AD prediction. This work demonstrates state-of-the-art performance in longitudinal AD prediction and highlights the effectiveness of combining spatial and temporal modeling for the early detection of Alzheimer's disease.
☆ Fusion or Confusion? Assessing the impact of visible-thermal image fusion for automated wildlife detection
Camille Dionne-Pierre, Samuel Foucher, Jérôme Théau, Jérôme Lemaître, Patrick Charbonneau, Maxime Brousseau, Mathieu Varin
Efficient wildlife monitoring methods are necessary for biodiversity conservation and management. The combination of remote sensing, aerial imagery and deep learning offer promising opportunities to renew or improve existing survey methods. The complementary use of visible (VIS) and thermal infrared (TIR) imagery can add information compared to a single-source image and improve results in an automated detection context. However, the alignment and fusion process can be challenging, especially since visible and thermal images usually have different fields of view (FOV) and spatial resolutions. This research presents a case study on the great blue heron (Ardea herodias) to evaluate the performances of synchronous aerial VIS and TIR imagery to automatically detect individuals and nests using a YOLO11n model. Two VIS-TIR fusion methods were tested and compared: an early fusion approach and a late fusion approach, to determine if the addition of the TIR image gives any added value compared to a VIS-only model. VIS and TIR images were automatically aligned using a deep learning model. A principal component analysis fusion method was applied to VIS-TIR image pairs to form the early fusion dataset. A classification and regression tree was used to process the late fusion dataset, based on the detection from the VIS-only and TIR-only trained models. Across all classes, both late and early fusion improved the F1 score compared to the VIS-only model. For the main class, occupied nest, the late fusion improved the F1 score from 90.2 (VIS-only) to 93.0%. This model was also able to identify false positives from both sources with 90% recall. Although fusion methods seem to give better results, this approach comes with a limiting TIR FOV and alignment constraints that eliminate data. Using an aircraft-mounted very high-resolution visible sensor could be an interesting option for operationalizing surveys.
comment : 19 pages, 9 figures, submitted to Remote Sensing in Ecology and Conservation
☆ MammoRGB: Dual-View Mammogram Synthesis Using Denoising Diffusion Probabilistic Models
Jorge Alberto Garza-Abdala, Gerardo A. Fumagal-González, Daly Avendano, Servando Cardona, Sadam Hussain, Eduardo de Avila-Armenta, Jasiel H. Toscano-Martínez, Diana S. M. Rosales Gurmendi, Alma A. Pedro-Pérez, Jose Gerardo Tamez-Pena
Purpose: This study aims to develop and evaluate a three channel denoising diffusion probabilistic model (DDPM) for synthesizing single breast dual view mammograms and to assess the impact of channel representations on image fidelity and cross view consistency. Materials and Methods: A pretrained three channel DDPM, sourced from Hugging Face, was fine tuned on a private dataset of 11020 screening mammograms to generate paired craniocaudal (CC) and mediolateral oblique (MLO) views. Three third channel encodings of the CC and MLO views were evaluated: sum, absolute difference, and zero channel. Each model produced 500 synthetic image pairs. Quantitative assessment involved breast mask segmentation using Intersection over Union (IoU) and Dice Similarity Coefficient (DSC), with distributional comparisons against 2500 real pairs using Earth Movers Distance (EMD) and Kolmogorov Smirnov (KS) tests. Qualitative evaluation included a visual Turing test by a non expert radiologist to assess cross view consistency and artifacts. Results: Synthetic mammograms showed IoU and DSC distributions comparable to real images, with EMD and KS values (0.020 and 0.077 respectively). Models using sum or absolute difference encodings outperformed others in IoU and DSC (p < 0.001), though distributions remained broadly similar. Generated CC and MLO views maintained cross view consistency, with 6 to 8 percent of synthetic images exhibiting artifacts consistent with those in the training data. Conclusion: Three channel DDPMs can generate realistic and anatomically consistent dual view mammograms with promising applications in dataset augmentation.
☆ All Centers Are at most a Few Tokens Apart: Knowledge Distillation with Domain Invariant Prompt Tuning
Amir Mohammad Ezzati, Alireza Malekhosseini, Armin Khosravi, Mohammad Hossein Rohban
Domain generalization is critical in computational pathology (CPath) due to inherent domain shifts caused by variations in staining protocols, scanner devices, and imaging settings across clinical centers. Vision-language models (VLMs), such as PLIP-a pathology-tuned CLIP-trained on image-text pairs across diverse domains, serve as strong knowledge distillation sources. However, their zero-shot performance with predefined prompts remains limited due to sensitivity to prompt variations. Moreover, unlike natural images, histopathology centers lack semantic descriptors (e.g., 'sketch'), making it difficult to define domain-specific prompts for clinical centers. This requires a data-driven approach for learning domain-specific and ultimately class-generic continuous prompts. We propose Domain Invariant Prompt Tuning (DIPT) for knowledge distillation process, a novel step that learns multiple input tokens for each domain. These tokens are trained separately for each domain and are averaged across domains, leading to domain-invariant prompts. Our student model then distills knowledge from PLIP's text encoder by leveraging the prompts learned by DIPT. This leads to alignment of visual features with domain-invariant embeddings, enhancing generalization by training on multiple domains. Our method adds a significant improvement in average F1-score to existing state-of-the-art (SOTA) knowledge distillation approaches in domain generalization with histopathology datasets. This work helps the way of deploying robust CPath models in real-world clinical problems with heterogeneous data sources.
☆ ReAG: Reasoning -Augmented Generation for Knowledge-based Visual Question Answering
Alberto Compagnoni, Marco Morini, Sara Sarto, Federico Cocchi, Davide Caffagni, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
Multimodal Large Language Models (MLLMs) have shown impressive capabilities in jointly understanding text, images, and videos, often evaluated via Visual Question Answering (VQA). However, even state-of-the-art MLLMs struggle with domain-specific or knowledge-intensive queries, where relevant information is underrepresented in pre-training data. Knowledge-based VQA (KB-VQA) addresses this by retrieving external documents to condition answer generation, but current retrieval-augmented approaches suffer from low precision, noisy passages, and limited reasoning. To address this, we propose ReAG, a novel Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages, ensuring high-quality additional context. The model follows a multi-stage training strategy leveraging reinforcement learning to enhance reasoning over retrieved content, while supervised fine-tuning serves only as a cold start. Extensive experiments on Encyclopedic-VQA and InfoSeek demonstrate that ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence. Our source code is publicly available at: https://github.com/aimagelab/ReAG.
☆ Splat-SAP: Feed-Forward Gaussian Splatting for Human-Centered Scene with Scale-Aware Point Map Reconstruction AAAI 2026
Boyao Zhou, Shunyuan Zheng, Zhanfeng Liao, Zihan Ma, Hanzhang Tu, Boning Liu, Yebin Liu
We present Splat-SAP, a feed-forward approach to render novel views of human-centered scenes from binocular cameras with large sparsity. Gaussian Splatting has shown its promising potential in rendering tasks, but it typically necessitates per-scene optimization with dense input views. Although some recent approaches achieve feed-forward Gaussian Splatting rendering through geometry priors obtained by multi-view stereo, such approaches still require largely overlapped input views to establish the geometry prior. To bridge this gap, we leverage pixel-wise point map reconstruction to represent geometry which is robust to large sparsity for its independent view modeling. In general, we propose a two-stage learning strategy. In stage 1, we transform the point map into real space via an iterative affinity learning process, which facilitates camera control in the following. In stage 2, we project point maps of two input views onto the target view plane and refine such geometry via stereo matching. Furthermore, we anchor Gaussian primitives on this refined plane in order to render high-quality images. As a metric representation, the scale-aware point map in stage 1 is trained in a self-supervised manner without 3D supervision and stage 2 is supervised with photo-metric loss. We collect multi-view human-centered data and demonstrate that our method improves both the stability of point map reconstruction and the visual quality of free-viewpoint rendering.
comment : Accepted by AAAI 2026. Project page: https://yaourtb.github.io/Splat-SAP
☆ Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer
Z-Image Team, Huanqia Cai, Sihan Cao, Ruoyi Du, Peng Gao, Steven Hoi, Shijie Huang, Zhaohui Hou, Dengyang Jiang, Xin Jin, Liangchen Li, Zhen Li, Zhong-Yu Li, David Liu, Dongyang Liu, Junhan Shi, Qilong Wu, Feng Yu, Chi Zhang, Shifeng Zhang, Shilin Zhou
The landscape of high-performance image generation models is currently dominated by proprietary systems, such as Nano Banana Pro and Seedream 4.0. Leading open-source alternatives, including Qwen-Image, Hunyuan-Image-3.0 and FLUX.2, are characterized by massive parameter counts (20B to 80B), making them impractical for inference, and fine-tuning on consumer-grade hardware. To address this gap, we propose Z-Image, an efficient 6B-parameter foundation generative model built upon a Scalable Single-Stream Diffusion Transformer (S3-DiT) architecture that challenges the "scale-at-all-costs" paradigm. By systematically optimizing the entire model lifecycle -- from a curated data infrastructure to a streamlined training curriculum -- we complete the full training workflow in just 314K H800 GPU hours (approx. $630K). Our few-step distillation scheme with reward post-training further yields Z-Image-Turbo, offering both sub-second inference latency on an enterprise-grade H800 GPU and compatibility with consumer-grade hardware (<16GB VRAM). Additionally, our omni-pre-training paradigm also enables efficient training of Z-Image-Edit, an editing model with impressive instruction-following capabilities. Both qualitative and quantitative experiments demonstrate that our model achieves performance comparable to or surpassing that of leading competitors across various dimensions. Most notably, Z-Image exhibits exceptional capabilities in photorealistic image generation and bilingual text rendering, delivering results that rival top-tier commercial models, thereby demonstrating that state-of-the-art results are achievable with significantly reduced computational overhead. We publicly release our code, weights, and online demo to foster the development of accessible, budget-friendly, yet state-of-the-art generative models.
☆ Mechanistic Finetuning of Vision -Language-Action Models via Few-Shot Demonstrations
Chancharik Mitra, Yusen Luo, Raj Saravanan, Dantong Niu, Anirudh Pai, Jesse Thomason, Trevor Darrell, Abrar Anwar, Deva Ramanan, Roei Herzig
Vision-Language Action (VLAs) models promise to extend the remarkable success of vision-language models (VLMs) to robotics. Yet, unlike VLMs in the vision-language domain, VLAs for robotics require finetuning to contend with varying physical factors like robot embodiment, environment characteristics, and spatial relationships of each task. Existing fine-tuning methods lack specificity, adapting the same set of parameters regardless of a task's visual, linguistic, and physical characteristics. Inspired by functional specificity in neuroscience, we hypothesize that it is more effective to finetune sparse model representations specific to a given task. In this work, we introduce Robotic Steering, a finetuning approach grounded in mechanistic interpretability that leverages few-shot demonstrations to identify and selectively finetune task-specific attention heads aligned with the physical, visual, and linguistic requirements of robotic tasks. Through comprehensive on-robot evaluations with a Franka Emika robot arm, we demonstrate that Robotic Steering outperforms LoRA while achieving superior robustness under task variation, reduced computational cost, and enhanced interpretability for adapting VLAs to diverse robotic tasks.
☆ Ar2Can: An Architect and an Artist Leveraging a Canvas for Multi-Human Generation
Shubhankar Borse, Phuc Pham, Farzad Farhadzadeh, Seokeon Choi, Phong Ha Nguyen, Anh Tuan Tran, Sungrack Yun, Munawar Hayat, Fatih Porikli
Despite recent advances in text-to-image generation, existing models consistently fail to produce reliable multi-human scenes, often duplicating faces, merging identities, or miscounting individuals. We present Ar2Can, a novel two-stage framework that disentangles spatial planning from identity rendering for multi-human generation. The Architect module predicts structured layouts, specifying where each person should appear. The Artist module then synthesizes photorealistic images, guided by a spatially-grounded face matching reward that combines Hungarian spatial alignment with ArcFace identity similarity. This approach ensures faces are rendered at correct locations and faithfully preserve reference identities. We develop two Architect variants, seamlessly integrated with our diffusion-based Artist model and optimized via Group Relative Policy Optimization (GRPO) using compositional rewards for count accuracy, image quality, and identity matching. Evaluated on the MultiHuman-Testbench, Ar2Can achieves substantial improvements in both count accuracy and identity preservation, while maintaining high perceptual quality. Notably, our method achieves these results using primarily synthetic data, without requiring real multi-human images.
☆ Emergent Extreme-View Geometry in 3D Foundation Models
Yiwen Zhang, Joseph Tung, Ruojin Cai, David Fouhey, Hadar Averbuch-Elor
3D foundation models (3DFMs) have recently transformed 3D vision, enabling joint prediction of depths, poses, and point maps directly from images. Yet their ability to reason under extreme, non-overlapping views remains largely unexplored. In this work, we study their internal representations and find that 3DFMs exhibit an emergent understanding of extreme-view geometry, despite never being trained for such conditions. To further enhance these capabilities, we introduce a lightweight alignment scheme that refines their internal 3D representation by tuning only a small subset of backbone bias terms, leaving all decoder heads frozen. This targeted adaptation substantially improves relative pose estimation under extreme viewpoints without degrading per-image depth or point quality. Additionally, we contribute MegaUnScene, a new benchmark of Internet scenes unseen by existing 3DFMs, with dedicated test splits for both relative pose estimation and dense 3D reconstruction. All code and data will be released.
comment : Project page is at https://ext-3dfms.github.io/
☆ Decoupled DMD: CFG Augmentation as the Spear, Distribution Matching as the Shield
Dongyang Liu, Peng Gao, David Liu, Ruoyi Du, Zhen Li, Qilong Wu, Xin Jin, Sihan Cao, Shifeng Zhang, Hongsheng Li, Steven Hoi
Diffusion model distillation has emerged as a powerful technique for creating efficient few-step and single-step generators. Among these, Distribution Matching Distillation (DMD) and its variants stand out for their impressive performance, which is widely attributed to their core mechanism of matching the student's output distribution to that of a pre-trained teacher model. In this work, we challenge this conventional understanding. Through a rigorous decomposition of the DMD training objective, we reveal that in complex tasks like text-to-image generation, where CFG is typically required for desirable few-step performance, the primary driver of few-step distillation is not distribution matching, but a previously overlooked component we identify as CFG Augmentation (CA). We demonstrate that this term acts as the core ``engine'' of distillation, while the Distribution Matching (DM) term functions as a ``regularizer'' that ensures training stability and mitigates artifacts. We further validate this decoupling by demonstrating that while the DM term is a highly effective regularizer, it is not unique; simpler non-parametric constraints or GAN-based objectives can serve the same stabilizing function, albeit with different trade-offs. This decoupling of labor motivates a more principled analysis of the properties of both terms, leading to a more systematic and in-depth understanding. This new understanding further enables us to propose principled modifications to the distillation process, such as decoupling the noise schedules for the engine and the regularizer, leading to further performance gains. Notably, our method has been adopted by the Z-Image ( https://github.com/Tongyi-MAI/Z-Image ) project to develop a top-tier 8-step image generation model, empirically validating the generalization and robustness of our findings.
☆ Structure-Preserving Unpaired Image Translation to Photometrically Calibrate JunoCam with Hubble Data
Aditya Pratap Singh, Shrey Shah, Ramanakumar Sankar, Emma Dahl, Gerald Eichstädt, Georgios Georgakis, Bernadette Bucher
Insights into Jupiter's atmospheric dynamics are vital for understanding planetary meteorology and exoplanetary gas giant atmospheres. To study these dynamics, we require high-resolution, photometrically calibrated observations. Over the last 9 years, the Juno spacecraft's optical camera, JunoCam, has generated a unique dataset with high spatial resolution, wide coverage during perijove passes, and a long baseline. However, JunoCam lacks absolute photometric calibration, hindering quantitative analysis of the Jovian atmosphere. Using observations from the Hubble Space Telescope (HST) as a proxy for a calibrated sensor, we present a novel method for performing unpaired image-to-image translation (I2I) between JunoCam and HST, focusing on addressing the resolution discrepancy between the two sensors. Our structure-preserving I2I method, SP-I2I, incorporates explicit frequency-space constraints designed to preserve high-frequency features ensuring the retention of fine, small-scale spatial structures - essential for studying Jupiter's atmosphere. We demonstrate that state-of-the-art unpaired image-to-image translation methods are inadequate to address this problem, and, importantly, we show the broader impact of our proposed solution on relevant remote sensing data for the pansharpening task.
☆ A deep learning perspective on Rubens' attribution
A. Afifi, A. Kalimullin, S. Korchagin, I. Kudryashov
This study explores the use of deep learning for the authentication and attribution of paintings, focusing on the complex case of Peter Paul Rubens and his workshop. A convolutional neural network was trained on a curated dataset of verified and comparative artworks to identify micro-level stylistic features characteristic of the master s hand. The model achieved high classification accuracy and demonstrated the potential of computational analysis to complement traditional art historical expertise, offering new insights into authorship and workshop collaboration.
☆ VaMP: Variational Multi-Modal Prompt Learning for Vision -Language Models NeurIPS 2025
Silin Cheng, Kai Han
Vision-language models (VLMs), such as CLIP, have shown strong generalization under zero-shot settings, yet adapting them to downstream tasks with limited supervision remains a significant challenge. Existing multi-modal prompt learning methods typically rely on fixed, shared prompts and deterministic parameters, which limits their ability to capture instance-level variation or model uncertainty across diverse tasks and domains. To tackle this issue, we propose a novel Variational Multi-Modal Prompt Learning (VaMP) framework that enables sample-specific, uncertainty-aware prompt tuning in multi-modal representation learning. VaMP generates instance-conditioned prompts by sampling from a learned posterior distribution, allowing the model to personalize its behavior based on input content. To further enhance the integration of local and global semantics, we introduce a class-aware prior derived from the instance representation and class prototype. Building upon these, we formulate prompt tuning as variational inference over latent prompt representations and train the entire framework end-to-end through reparameterized sampling. Experiments on few-shot and domain generalization benchmarks show that VaMP achieves state-of-the-art performance, highlighting the benefits of modeling both uncertainty and task structure in our method. Project page: https://visual-ai.github.io/vamp
comment : Accepted to NeurIPS 2025
☆ Architecture Decoupling Is Not All You Need For Unified Multimodal Model
Dian Zheng, Manyuan Zhang, Hongyu Li, Kai Zou, Hongbo Liu, Ziyu Guo, Kaituo Feng, Yexin Liu, Ying Luo, Yan Feng, Peng Pei, Xunliang Cai, Hongsheng Li
Unified multimodal models for image generation and understanding represent a significant step toward AGI and have attracted widespread attention from researchers. The main challenge of this task lies in the difficulty in establishing an optimal training paradigm due to inherent conflicting targets in understanding and generation tasks. To alleviate these conflicts and pursue higher performance, many researchers adopt varying degrees of model decoupling (e.g., Double image encoders, MOE/MOT architecture, or frozen MLLM). However, excessive model decoupling can lead to the loss of interleave generation ability, undermining the original intent of unified models. In this work, we aim to explore how to mitigate task conflicts without resorting to model decoupling. Firstly, we analyze why decoupling alleviates conflicts by studying the cross-modal attention behavior of models. We observe that model decoupling essentially drives models toward task-specific multimodal interaction patterns, as seen in Qwen-VL and HunyuanImage, and that the more thorough the decoupling, the more consistent the behavior becomes. Motivated by this observation, we propose Attention Interaction Alignment (AIA) loss, which explicitly learns Task-Specific multimodal interaction patterns during training. To demonstrate the generalizability of our AIA loss, we apply it to Emu3 and Janus-Pro during SFT and post-training stage respectively. Without bells and whistles, AIA not only refines cross-modal attention patterns, but also boosts both generation and understanding performance.
comment : Project page: https://zhengdian1.github.io/AIA-project/ Code: https://github.com/zhengdian1/AIA
☆ Geometrically-Constrained Agent for Spatial Reasoning
Zeren Chen, Xiaoya Lu, Zhijie Zheng, Pengrui Li, Lehan He, Yijin Zhou, Jing Shao, Bohan Zhuang, Lu Sheng
Vision Language Models (VLMs) exhibit a fundamental semantic-to-geometric gap in spatial reasoning: they excel at qualitative semantic inference but their reasoning operates within a lossy semantic space, misaligned with high-fidelity geometry. Current paradigms fail to bridge this gap. Training-based methods suffer from an ``oracle paradox,'' learning flawed spatial logic from imperfect oracles. Tool-integrated methods constrain the final computation but critically leave the VLM's planning process unconstrained, resulting in geometrically flawed plans. In this work, we propose Geometrically-Constrained Agent (GCA), a training-free agentic paradigm that resolves this gap by introducing a formal task constraint. Specifically, we strategically decouples the VLM's role into two stages. First, acting as a semantic analyst, the VLM translates the user's ambiguous query into the formal, verifiable task constraint, which defines the reference frame and objective. Second, acting as a task solver, the VLM generates and executes tool calls strictly within the deterministic bounds defined by the constraint. This geometrically-constrained reasoning strategy successfully resolve the semantic-to-geometric gap, yielding a robust and verifiable reasoning pathway for spatial reasoning. Comprehensive experiments demonstrate that GCA achieves SOTA performance on multiple spatial reasoning benchmarks, surpassing existing training-based and tool-integrated methods by ~27%. Please see our homepage at https://gca-spatial-reasoning.github.io.
comment : 27 pages, 13 figures
☆ GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenes
Di Wang, Shunyu Liu, Wentao Jiang, Fengxiang Wang, Yi Liu, Xiaolei Qin, Zhiming Luo, Chaoyang Zhou, Haonan Guo, Jing Zhang, Bo Du, Dacheng Tao, Liangpei Zhang
Multimodal large language models (MLLMs) have undergone rapid development in advancing geospatial scene understanding. Recent studies have sought to enhance the reasoning capabilities of remote sensing MLLMs, typically through cold-start training with elaborately curated chain-of-thought (CoT) data. However, this approach not only incurs substantial annotation costs but also introduces human biases that may limit the diversity of model reasoning. To address these challenges, we propose GeoZero, a framework that enables MLLMs to perform geospatial reasoning without any predefined CoT supervision. Specifically, we construct two datasets, GeoZero-Instruct and GeoZero-Hard. GeoZero-Instruct allows the model to acquire preliminary geospatial knowledge through supervised fine-tuning, while GeoZero-Hard stimulates deep reasoning during the subsequent reinforcement learning stage. Furthermore, we introduce Answer-Anchored Group Relative Policy Optimization (A$^2$GRPO), where the reasoning process is regularized by the model's own answers, encouraging diverse yet accurate thinking. Extensive experiments on multiple remote sensing vision-language benchmarks demonstrate that GeoZero not only surpasses existing state-of-the-art methods but also fosters universal emergent reasoning capabilities across diverse geospatial tasks. Code,data,and models will be publicly available at https://github.com/MiliLab/GeoZero.
comment : Code, data, and models will be publicly available at https://github.com/MiliLab/GeoZero
☆ REASONEDIT: Towards Reasoning -Enhanced Image Editing Models
Fukun Yin, Shiyu Liu, Yucheng Han, Zhibo Wang, Peng Xing, Rui Wang, Wei Cheng, Yingming Wang, Aojie Li, Zixin Yin, Pengtao Chen, Xiangyu Zhang, Daxin Jiang, Xianfang Zeng, Gang Yu
Recent advances in image editing models have shown remarkable progress. A common architectural design couples a multimodal large language model (MLLM) encoder with a diffusion decoder, as seen in systems such as Step1X-Edit and Qwen-Image-Edit, where the MLLM encodes both the reference image and the instruction but remains frozen during training. In this work, we demonstrate that unlocking the reasoning capabilities of MLLM can further push the boundaries of editing models. Specifically, we explore two reasoning mechanisms, thinking and reflection, which enhance instruction understanding and editing accuracy. Based on that, our proposed framework enables image editing in a thinking-editing-reflection loop: the thinking mechanism leverages the world knowledge of MLLM to interpret abstract instructions, while the reflection reviews editing results, automatically corrects unintended manipulations, and identifies the stopping round. Extensive experiments demonstrate that our reasoning approach achieves significant performance gains, with improvements of ImgEdit (+4.3%), GEdit (+4.7%), and Kris (+8.2%) when initializing our DiT from the Step1X-Edit (ReasonEdit-S), and also outperforms previous open-source methods on both GEdit and Kris when integrated with Qwen-Image-Edit (ReasonEdit-Q).
comment : code: https://github.com/stepfun-ai/Step1X-Edit
☆ Stable-Drift: A Patient-Aware Latent Drift Replay Method for Stabilizing Representations in Continual Learning
Paraskevi-Antonia Theofilou, Anuhya Thota, Stefanos Kollias, Mamatha Thota
When deep learning models are sequentially trained on new data, they tend to abruptly lose performance on previously learned tasks, a critical failure known as catastrophic forgetting. This challenge severely limits the deployment of AI in medical imaging, where models must continually adapt to data from new hospitals without compromising established diagnostic knowledge. To address this, we introduce a latent drift-guided replay method that identifies and replays samples with high representational instability. Specifically, our method quantifies this instability via latent drift, the change in a sample internal feature representation after naive domain adaptation. To ensure diversity and clinical relevance, we aggregate drift at the patient level, our memory buffer stores the per patient slices exhibiting the greatest multi-layer representation shift. Evaluated on a cross-hospital COVID-19 CT classification task using state-of-the-art CNN and Vision Transformer backbones, our method substantially reduces forgetting compared to naive fine-tuning and random replay. This work highlights latent drift as a practical and interpretable replay signal for advancing robust continual learning in real world medical settings.
comment : 8 pages, 2 figures
☆ MG-Nav: Dual-Scale Visual Navigation via Sparse Spatial Memory
Bo Wang, Jiehong Lin, Chenzhi Liu, Xinting Hu, Yifei Yu, Tianjia Liu, Zhongrui Wang, Xiaojuan Qi
We present MG-Nav (Memory-Guided Navigation), a dual-scale framework for zero-shot visual navigation that unifies global memory-guided planning with local geometry-enhanced control. At its core is the Sparse Spatial Memory Graph (SMG), a compact, region-centric memory where each node aggregates multi-view keyframe and object semantics, capturing both appearance and spatial structure while preserving viewpoint diversity. At the global level, the agent is localized on SMG and a goal-conditioned node path is planned via an image-to-instance hybrid retrieval, producing a sequence of reachable waypoints for long-horizon guidance. At the local level, a navigation foundation policy executes these waypoints in point-goal mode with obstacle-aware control, and switches to image-goal mode when navigating from the final node towards the visual target. To further enhance viewpoint alignment and goal recognition, we introduce VGGT-adapter, a lightweight geometric module built on the pre-trained VGGT model, which aligns observation and goal features in a shared 3D-aware space. MG-Nav operates global planning and local control at different frequencies, using periodic re-localization to correct errors. Experiments on HM3D Instance-Image-Goal and MP3D Image-Goal benchmarks demonstrate that MG-Nav achieves state-of-the-art zero-shot performance and remains robust under dynamic rearrangements and unseen scene conditions.
comment : 10pages, 5 figures
☆ GazeTrack: High-Precision Eye Tracking Based on Regularization and Spatial Computing
Xiaoyin Yang
Eye tracking has become increasingly important in virtual and augmented reality applications; however, the current gaze accuracy falls short of meeting the requirements for spatial computing. We designed a gaze collection framework and utilized high-precision equipment to gather the first precise benchmark dataset, GazeTrack, encompassing diverse ethnicities, ages, and visual acuity conditions for pupil localization and gaze tracking. We propose a novel shape error regularization method to constrain pupil ellipse fitting and train on open-source datasets, enhancing semantic segmentation and pupil position prediction accuracy. Additionally, we invent a novel coordinate transformation method similar to paper unfolding to accurately predict gaze vectors on the GazeTrack dataset. Finally, we built a gaze vector generation model that achieves reduced gaze angle error with lower computational complexity compared to other methods.
comment : 10 pages, 7 figures
☆ Hard Spatial Gating for Precision-Driven Brain Metastasis Segmentation: Addressing the Over-Segmentation Paradox in Deep Attention Networks
Rowzatul Zannath Prerona
Brain metastasis segmentation in MRI remains a formidable challenge due to diminutive lesion sizes (5-15 mm) and extreme class imbalance (less than 2% tumor volume). While soft-attention CNNs are widely used, we identify a critical failure mode termed the "over-segmentation paradox," where models achieve high sensitivity (recall > 0.88) but suffer from catastrophic precision collapse (precision < 0.23) and boundary errors exceeding 150 mm. This imprecision poses significant risks for stereotactic radiosurgery planning. To address this, we introduce the Spatial Gating Network (SG-Net), a precision-first architecture employing hard spatial gating mechanisms. Unlike traditional soft attention, SG-Net enforces strict feature selection to aggressively suppress background artifacts while preserving tumor features. Validated on the Brain-Mets-Lung-MRI dataset (n=92), SG-Net achieves a Dice Similarity Coefficient of 0.5578 +/- 0.0243 (95% CI: 0.45-0.67), statistically outperforming Attention U-Net (p < 0.001) and ResU-Net (p < 0.001). Most critically, SG-Net demonstrates a threefold improvement in boundary precision, achieving a 95% Hausdorff Distance of 56.13 mm compared to 157.52 mm for Attention U-Net, while maintaining robust recall (0.79) and superior precision (0.52 vs. 0.20). Furthermore, SG-Net requires only 0.67M parameters (8.8x fewer than Attention U-Net), facilitating deployment in resource-constrained environments. These findings establish hard spatial gating as a robust solution for precision-driven lesion detection, directly enhancing radiosurgery accuracy.
☆ AnoRefiner: Anomaly-Aware Group-Wise Refinement for Zero-Shot Industrial Anomaly Detection
Dayou Huang, Feng Xue, Xurui Li, Yu Zhou
Zero-shot industrial anomaly detection (ZSAD) methods typically yield coarse anomaly maps as vision transformers (ViTs) extract patch-level features only. To solve this, recent solutions attempt to predict finer anomalies using features from ZSAD, but they still struggle to recover fine-grained anomalies without missed detections, mainly due to the gap between randomly synthesized training anomalies and real ones. We observe that anomaly score maps exactly provide complementary spatial cues that are largely absent from ZSAD's image features, a fact overlooked before. Inspired by this, we propose an anomaly-aware refiner (AnoRefiner) that can be plugged into most ZSAD models and improve patch-level anomaly maps to the pixel level. First, we design an anomaly refinement decoder (ARD) that progressively enhances image features using anomaly score maps, reducing the reliance on synthetic anomaly data. Second, motivated by the mass production paradigm, we propose a progressive group-wise test-time training (PGT) strategy that trains ARD in each product group for the refinement process in the next group, while staying compatible with any ZSAD method. Experiments on the MVTec AD and VisA datasets show that AnoRefiner boosts various ZSAD models by up to a 5.2\% gain in pixel-AP metrics, which can also be directly observed in many visualizations. The code will be available at https://github.com/HUST-SLOW/AnoRefiner.
comment : 17 pages, 10 figures
☆ HarmoCLIP: Harmonizing Global and Regional Representations in Contrastive Vision -Language Models
Haoxi Zeng, Haoxuan Li, Yi Bin, Pengpeng Zeng, Xing Xu, Yang Yang, Heng Tao Shen
Contrastive Language-Image Pre-training (CLIP) has demonstrated remarkable generalization ability and strong performance across a wide range of vision-language tasks. However, due to the lack of region-level supervision, CLIP exhibits limited fine-grained semantic understanding. Although several methods attempt to mitigate this issue, they unintentionally disrupt the global alignment, resulting in a persistent trade-off where improving local perception simultaneously degrades global coherence. In this paper, we propose HarmoCLIP, a novel framework designed to harmonize global and region representations within CLIP. We first identify that the absence of direct alignment between local textual and visual semantics is the fundamental cause of the trade-off. To address this, HarmoCLIP introduces an explicit fine-grained semantic supervision term that directly aligns textual segments with their corresponding visual regions, effectively bridging the image region space and the textual space. To further strengthen the representation capability at the local level, our method introduces a novel Region-Language Alignment supervision strategy that promotes fine-grained semantic learning without compromising global semantic consistency. Extensive experiments demonstrate that HarmoCLIP achieves state-of-the-art (improvement up to 69.78%) performance on the global task of retrieval and yields a substantial 3.2% improvement in Top-1 accuracy on the region task of bounding-box classification, consistently outperforming prior approaches while providing a balanced, efficient, and plug-and-play solution to the global-local trade-off in CLIP. Code is available at https://github.com/Erosist/HarmoCLIP.
comment : 13 pages, 7 figures, 6 tables
☆ Revisiting the Necessity of Lengthy Chain-of-Thought in Vision -centric Reasoning Generalization
Yifan Du, Kun Zhou, Yingqian Min, Yue Ling, Wayne Xin Zhao, Youbin Wu
We study how different Chain-of-Thought (CoT) designs affect the acquisition of the generalizable visual reasoning ability in vision-language models (VLMs). While CoT data, especially long or visual CoT such as "think with image", has been widely used to supervise intermediate reasoning, it remains unclear why specific CoT designs help and which ones truly support generalizable reasoning. To systematically evaluate this, we focus on a controlled maze-solving benchmark where reasoning rules are fully visual, difficulty can be tuned by grid size, and all the intermediate steps can be automatically generated. Using Qwen2.5-VL-7B under a standard SFT-then-RL pipeline, we compare three representative CoT formats: Language CoT, Grounding CoT (with spatial coordinate trajectories), and Visual CoT (with image manipulations). Our experiments reveal that visual and longer CoT mainly accelerate convergence but do not lift the final performance ceiling; concise CoT containing only essential grounding steps outperforms longer traces; and, strikingly, CoT retaining only the minimal grounding results generalizes best across different maze sizes. We further validate these insights on other vision-centric tasks. These findings highlight a "short is long" effect and provide practical guidance for constructing more generalizable SFT datasets for visual reasoning.
☆ Text Condition Embedded Regression Network for Automated Dental Abutment Design
Mianjie Zheng, Xinquan Yang, Xuguang Li, Xiaoling Luo, Xuefen Liu, Kun Tang, He Meng, Linlin Shen
The abutment is an important part of artificial dental implants, whose design process is time-consuming and labor-intensive. Long-term use of inappropriate dental implant abutments may result in implant complications, including peri-implantitis. Using artificial intelligence to assist dental implant abutment design can quickly improve the efficiency of abutment design and enhance abutment adaptability. In this paper, we propose a text condition embedded abutment design framework (TCEAD), the novel automated abutment design solution available in literature. The proposed study extends the self-supervised learning framework of the mesh mask autoencoder (MeshMAE) by introducing a text-guided localization (TGL) module to facilitate abutment area localization. As the parameter determination of the abutment is heavily dependent on local fine-grained features (the width and height of the implant and the distance to the opposing tooth), we pre-train the encoder using oral scan data to improve the model's feature extraction ability. Moreover, considering that the abutment area is only a small part of the oral scan data, we designed a TGL module, which introduces the description of the abutment area through the text encoder of Contrastive Language-Image Pre-training (CLIP), enabling the network to quickly locate the abutment area. We validated the performance of TCEAD on a large abutment design dataset. Extensive experiments demonstrate that TCEAD achieves an Intersection over Union (IoU) improvement of 0.8%-12.85% over other mainstream methods, underscoring its potential in automated dental abutment design.
☆ Bringing Your Portrait to 3D Presence
Jiawei Zhang, Lei Chu, Jiahao Li, Zhenyu Zang, Chong Li, Xiao Li, Xun Cao, Hao Zhu, Yan Lu
We present a unified framework for reconstructing animatable 3D human avatars from a single portrait across head, half-body, and full-body inputs. Our method tackles three bottlenecks: pose- and framing-sensitive feature representations, limited scalable data, and unreliable proxy-mesh estimation. We introduce a Dual-UV representation that maps image features to a canonical UV space via Core-UV and Shell-UV branches, eliminating pose- and framing-induced token shifts. We also build a factorized synthetic data manifold combining 2D generative diversity with geometry-consistent 3D renderings, supported by a training scheme that improves realism and identity consistency. A robust proxy-mesh tracker maintains stability under partial visibility. Together, these components enable strong in-the-wild generalization. Trained only on half-body synthetic data, our model achieves state-of-the-art head and upper-body reconstruction and competitive full-body results. Extensive experiments and analyses further validate the effectiveness of our approach.
comment : project page: https://zjwfufu.github.io/HuaPi-Page/
☆ Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior NeurIPS 2025
Ruoyu Feng, Yunpeng Qi, Jinming Liu, Yixin Gao, Xin Li, Xin Jin, Zhibo Chen
Image compression methods are usually optimized isolatedly for human perception or machine analysis tasks. We reveal fundamental commonalities between these objectives: preserving accurate semantic information is paramount, as it directly dictates the integrity of critical information for intelligent tasks and aids human understanding. Concurrently, enhanced perceptual quality not only improves visual appeal but also, by ensuring realistic image distributions, benefits semantic feature extraction for machine tasks. Based on this insight, we propose Diff-ICMH, a generative image compression framework aiming for harmonizing machine and human vision in image compression. It ensures perceptual realism by leveraging generative priors and simultaneously guarantees semantic fidelity through the incorporation of Semantic Consistency loss (SC loss) during training. Additionally, we introduce the Tag Guidance Module (TGM) that leverages highly semantic image-level tags to stimulate the pre-trained diffusion model's generative capabilities, requiring minimal additional bit rates. Consequently, Diff-ICMH supports multiple intelligent tasks through a single codec and bitstream without any task-specific adaptation, while preserving high-quality visual experience for human perception. Extensive experimental results demonstrate Diff-ICMH's superiority and generalizability across diverse tasks, while maintaining visual appeal for human perception. Code is available at: https://github.com/RuoyuFeng/Diff-ICMH.
comment : Accepted by NeurIPS 2025
☆ Fast3Dcache: Training-free 3D Geometry Synthesis Acceleration
Mengyu Yang, Yanming Yang, Chenyi Xu, Chenxi Song, Yufan Zuo, Tong Zhao, Ruibo Li, Chi Zhang
Diffusion models have achieved impressive generative quality across modalities like 2D images, videos, and 3D shapes, but their inference remains computationally expensive due to the iterative denoising process. While recent caching-based methods effectively reuse redundant computations to speed up 2D and video generation, directly applying these techniques to 3D diffusion models can severely disrupt geometric consistency. In 3D synthesis, even minor numerical errors in cached latent features accumulate, causing structural artifacts and topological inconsistencies. To overcome this limitation, we propose Fast3Dcache, a training-free geometry-aware caching framework that accelerates 3D diffusion inference while preserving geometric fidelity. Our method introduces a Predictive Caching Scheduler Constraint (PCSC) to dynamically determine cache quotas according to voxel stabilization patterns and a Spatiotemporal Stability Criterion (SSC) to select stable features for reuse based on velocity magnitude and acceleration criterion. Comprehensive experiments show that Fast3Dcache accelerates inference significantly, achieving up to a 27.12% speed-up and a 54.8% reduction in FLOPs, with minimal degradation in geometric quality as measured by Chamfer Distance (2.48%) and F-Score (1.95%).
☆ CoT 4AD: A Vision -Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving
Zhaohui Wang, Tengbo Yu, Hao Tang
Vision-Language-Action (VLA) models have recently attracted growing attention in end-to-end autonomous driving for their strong reasoning capabilities and rich world knowledge. However, existing VLAs often suffer from limited numerical reasoning ability and overly simplified input-output mappings, which hinder their performance in complex driving scenarios requiring step-by-step causal reasoning. To address these challenges, we propose CoT4AD, a novel VLA framework that introduces Chain-of-Thought (CoT) reasoning for autonomous driving to enhance both numerical and causal reasoning in Vision-Language Models (VLMs). CoT4AD integrates visual observations and language instructions to perform semantic reasoning, scene understanding, and trajectory planning. During training, it explicitly models a perception-question-prediction-action CoT to align the reasoning space with the action space across multiple driving tasks. During inference, it performs implicit CoT reasoning to enable consistent numerical reasoning and robust decision-making in dynamic environments. Extensive experiments on both real-world and simulated benchmarks, including nuScenes and Bench2Drive, demonstrate that CoT4AD achieves state-of-the-art performance in both open-loop and closed-loop evaluations. Code will be released upon paper acceptance.
comment : 10 pages, 3 figures
☆ DocVAL: Validated Chain-of-Thought Distillation for Grounded Document VQA
Ahmad Mohammadshirazi, Pinaki Prasad Guha Neogi, Dheeraj Kulshrestha, Rajiv Ramnath
Document visual question answering (DocVQA) requires models to jointly reason over textual content and spatial layout, yet current systems exhibit a sharp accuracy--efficiency trade-off: large teacher models achieve strong grounding but are too expensive for deployment, while compact students suffer substantial drops in localization performance. We propose DocVAL, a validated chain-of-thought distillation framework that transfers the spatial reasoning ability of a large teacher into a deployable student VLM through three key components: (1) teacher supervision with validation-time text detection to filter and denoise training signals, (2) a multi-module validator (VAL) that enforces answer correctness and geometric consistency while producing fine-grained, pixel-level error feedback, and (3) a two-stage student training scheme that first learns from validated CoT traces and then undergoes iterative refinement driven by VAL feedback. Our student (Gemma-3 12B) achieves 91.4\% ANLS and 82.4\% mAP on DocVQA as a pure VLM requiring no text detection or OCR at inference. Extensive ablations demonstrate that validated feedback contributes 6.3 mAP gain and iterative refinement accounts for 9.7 mAP improvement. We release 95k high-quality, validator-verified CoT traces to advance spatial reasoning research in document understanding.
☆ RealD$^2$iff: Bridging Real-Wo rl d Gap in Robot Manipulation via Depth Diffusion
Xiujian Liang, Jiacheng Liu, Mingyang Sun, Qichen He, Cewu Lu, Jianhua Sun
Robot manipulation in the real world is fundamentally constrained by the visual sim2real gap, where depth observations collected in simulation fail to reflect the complex noise patterns inherent to real sensors. In this work, inspired by the denoising capability of diffusion models, we invert the conventional perspective and propose a clean-to-noisy paradigm that learns to synthesize noisy depth, thereby bridging the visual sim2real gap through purely simulation-driven robotic learning. Building on this idea, we introduce RealD$^2$iff, a hierarchical coarse-to-fine diffusion framework that decomposes depth noise into global structural distortions and fine-grained local perturbations. To enable progressive learning of these components, we further develop two complementary strategies: Frequency-Guided Supervision (FGS) for global structure modeling and Discrepancy-Guided Optimization (DGO) for localized refinement. To integrate RealD$^2$iff seamlessly into imitation learning, we construct a pipeline that spans six stages. We provide comprehensive empirical and experimental validation demonstrating the effectiveness of this paradigm. RealD$^2$iff enables two key applications: (1) generating real-world-like depth to construct clean-noisy paired datasets without manual sensor data collection. (2) Achieving zero-shot sim2real robot manipulation, substantially improving real-world performance without additional fine-tuning.
☆ What Shape Is Optimal for Masks in Text Removal?
Hyakka Nakada, Marika Kubota
The advent of generative models has dramatically improved the accuracy of image inpainting. In particular, by removing specific text from document images, reconstructing original images is extremely important for industrial applications. However, most existing methods of text removal focus on deleting simple scene text which appears in images captured by a camera in an outdoor environment. There is little research dedicated to complex and practical images with dense text. Therefore, we created benchmark data for text removal from images including a large amount of text. From the data, we found that text-removal performance becomes vulnerable against mask profile perturbation. Thus, for practical text-removal tasks, precise tuning of the mask shape is essential. This study developed a method to model highly flexible mask profiles and learn their parameters using Bayesian optimization. The resulting profiles were found to be character-wise masks. It was also found that the minimum cover of a text region is not optimal. Our research is expected to pave the way for a user-friendly guideline for manual masking.
comment : 12 pages, 17 figures
☆ SciPostGen: Bridging the Gap between Scientific Papers and Poster Layouts
Shun Inadumi, Shohei Tanaka, Tosho Hirasawa, Atsushi Hashimoto, Koichiro Yoshino, Yoshitaka Ushiku
As the number of scientific papers continues to grow, there is a demand for approaches that can effectively convey research findings, with posters serving as a key medium for presenting paper contents. Poster layouts determine how effectively research is communicated and understood, highlighting their growing importance. In particular, a gap remains in understanding how papers correspond to the layouts that present them, which calls for datasets with paired annotations at scale. To bridge this gap, we introduce SciPostGen, a large-scale dataset for understanding and generating poster layouts from scientific papers. Our analyses based on SciPostGen show that paper structures are associated with the number of layout elements in posters. Based on this insight, we explore a framework, Retrieval-Augmented Poster Layout Generation, which retrieves layouts consistent with a given paper and uses them as guidance for layout generation. We conducted experiments under two conditions: with and without layout constraints typically specified by poster creators. The results show that the retriever estimates layouts aligned with paper structures, and our framework generates layouts that also satisfy given constraints.
comment : Dataset: https://huggingface.co/datasets/omron-sinicx/scipostgen, Code: https://github.com/omron-sinicx/scipostgen_dataset_construction
☆ AI killed the video star. Audio-driven diffusion model for expressive talking head generation
Baptiste Chopin, Tashvik Dhamija, Pranav Balaji, Yaohui Wang, Antitza Dantcheva
We propose Dimitra++, a novel framework for audio-driven talking head generation, streamlined to learn lip motion, facial expression, as well as head pose motion. Specifically, we propose a conditional Motion Diffusion Transformer (cMDT) to model facial motion sequences, employing a 3D representation. The cMDT is conditioned on two inputs: a reference facial image, which determines appearance, as well as an audio sequence, which drives the motion. Quantitative and qualitative experiments, as well as a user study on two widely employed datasets, i.e., VoxCeleb2 and CelebV-HQ, suggest that Dimitra++ is able to outperform existing approaches in generating realistic talking heads imparting lip motion, facial expression, and head pose.
comment : arXiv admin note: text overlap with arXiv:2502.17198
☆ Adversarial Flow Models
Shanchuan Lin, Ceyuan Yang, Zhijie Lin, Hao Chen, Haoqi Fan
We present adversarial flow models, a class of generative models that unifies adversarial models and flow models. Our method supports native one-step or multi-step generation and is trained using the adversarial objective. Unlike traditional GANs, where the generator learns an arbitrary transport plan between the noise and the data distributions, our generator learns a deterministic noise-to-data mapping, which is the same optimal transport as in flow-matching models. This significantly stabilizes adversarial training. Also, unlike consistency-based methods, our model directly learns one-step or few-step generation without needing to learn the intermediate timesteps of the probability flow for propagation. This saves model capacity, reduces training iterations, and avoids error accumulation. Under the same 1NFE setting on ImageNet-256px, our B/2 model approaches the performance of consistency-based XL/2 models, while our XL/2 model creates a new best FID of 2.38. We additionally show the possibility of end-to-end training of 56-layer and 112-layer models through depth repetition without any intermediate supervision, and achieve FIDs of 2.08 and 1.94 using a single forward pass, surpassing their 2NFE and 4NFE counterparts.
☆ Rethinking Cross-Generator Image Forgery Detection through DINOv3
Zhenglin Huang, Jason Li, Haiquan Wen, Tianxiao Li, Xi Yang, Lu Qi, Bei Peng, Xiaowei Huang, Ming-Hsuan Yang, Guangliang Cheng
As generative models become increasingly diverse and powerful, cross-generator detection has emerged as a new challenge. Existing detection methods often memorize artifacts of specific generative models rather than learning transferable cues, leading to substantial failures on unseen generators. Surprisingly, this work finds that frozen visual foundation models, especially DINOv3, already exhibit strong cross-generator detection capability without any fine-tuning. Through systematic studies on frequency, spatial, and token perspectives, we observe that DINOv3 tends to rely on global, low-frequency structures as weak but transferable authenticity cues instead of high-frequency, generator-specific artifacts. Motivated by this insight, we introduce a simple, training-free token-ranking strategy followed by a lightweight linear probe to select a small subset of authenticity-relevant tokens. This token subset consistently improves detection accuracy across all evaluated datasets. Our study provides empirical evidence and a feasible hypothesis for understanding why foundation models generalize across diverse generators, offering a universal, efficient, and interpretable baseline for image forgery detection.
☆ Hybrid, Unified and Iterative: A Novel Framework for Text-based Person Anomaly Retrieval
Tien-Huy Nguyen, Huu-Loc Tran, Huu-Phong Phan-Nguyen, Quang-Vinh Dinh
Text-based person anomaly retrieval has emerged as a challenging task, with most existing approaches relying on complex deep-learning techniques. This raises a research question: How can the model be optimized to achieve greater fine-grained features? To address this, we propose a Local-Global Hybrid Perspective (LHP) module integrated with a Vision-Language Model (VLM), designed to explore the effectiveness of incorporating both fine-grained features alongside coarse-grained features. Additionally, we investigate a Unified Image-Text (UIT) model that combines multiple objective loss functions, including Image-Text Contrastive (ITC), Image-Text Matching (ITM), Masked Language Modeling (MLM), and Masked Image Modeling (MIM) loss. Beyond this, we propose a novel iterative ensemble strategy, by combining iteratively instead of using model results simultaneously like other ensemble methods. To take advantage of the superior performance of the LHP model, we introduce a novel feature selection algorithm based on its guidance, which helps improve the model's performance. Extensive experiments demonstrate the effectiveness of our method in achieving state-of-the-art (SOTA) performance on PAB dataset, compared with previous work, with a 9.70\% improvement in R@1, 1.77\% improvement in R@5, and 1.01\% improvement in R@10.
comment : Accepted on World Wide Web 2025 Workshop
☆ RoadSceneBench: A Lightweight Benchmark for Mid-Level Road Scene Understanding
Xiyan Liu, Han Wang, Yuhu Wang, Junjie Cai, Zhe Cao, Jianzhong Yang, Zhen Lu
Understanding mid-level road semantics, which capture the structural and contextual cues that link low-level perception to high-level planning, is essential for reliable autonomous driving and digital map construction. However, existing benchmarks primarily target perception tasks such as detection or segmentation, overlooking the reasoning capabilities required to infer road topology and dynamic scene structure. To address this gap, we present RoadSceneBench, a lightweight yet information-rich benchmark designed to evaluate and advance visual reasoning in complex road environments. Unlike large-scale perception datasets, RoadSceneBench emphasizes relational understanding and structural consistency, encouraging models to capture the underlying logic of real-world road scenes. Furthermore, to enhance reasoning reliability, we propose Hierarchical Relational Reward Propagation with Temporal Consistency (HRRP-T), a training framework for Vision-Language Models (VLMs) in which reward signals adaptively promote spatial coherence and semantic alignment throughout the reasoning process. This paradigm enables models to move beyond static recognition toward geometry-aware and temporally consistent reasoning. Extensive experiments demonstrate that our method achieves state-of-the-art performance across diverse road configurations. RoadSceneBench thus provides a compact yet powerful foundation for studying mid-level road semantics and fostering structure-aware autonomous perception. Our dataset is available at https://github.com/XiyanLiu/RoadSceneBench.
☆ Gaussians on Fire: High-Frequency Reconstruction of Flames
Jakob Nazarenus, Dominik Michels, Wojtek Palubicki, Simin Kou, Fang-Lue Zhang, Soren Pirk, Reinhard Koch
We propose a method to reconstruct dynamic fire in 3D from a limited set of camera views with a Gaussian-based spatiotemporal representation. Capturing and reconstructing fire and its dynamics is highly challenging due to its volatile nature, transparent quality, and multitude of high-frequency features. Despite these challenges, we aim to reconstruct fire from only three views, which consequently requires solving for under-constrained geometry. We solve this by separating the static background from the dynamic fire region by combining dense multi-view stereo images with monocular depth priors. The fire is initialized as a 3D flow field, obtained by fusing per-view dense optical flow projections. To capture the high frequency features of fire, each 3D Gaussian encodes a lifetime and linear velocity to match the dense optical flow. To ensure sub-frame temporal alignment across cameras we employ a custom hardware synchronization pattern -- allowing us to reconstruct fire with affordable commodity hardware. Our quantitative and qualitative validations across numerous reconstruction experiments demonstrate robust performance for diverse and challenging real fire scenarios.
comment : 18 pages, 9 figures
☆ ITS3D: Inference-Time Scaling for Text-Guided 3D Diffusion Models
Zhenglin Zhou, Fan Ma, Xiaobo Xia, Hehe Fan, Yi Yang, Tat-Seng Chua
We explore inference-time scaling in text-guided 3D diffusion models to enhance generative quality without additional training. To this end, we introduce ITS3D, a framework that formulates the task as an optimization problem to identify the most effective Gaussian noise input. The framework is driven by a verifier-guided search algorithm, where the search algorithm iteratively refines noise candidates based on verifier feedback. To address the inherent challenges of 3D generation, we introduce three techniques for improved stability, efficiency, and exploration capability. 1) Gaussian normalization is applied to stabilize the search process. It corrects distribution shifts when noise candidates deviate from a standard Gaussian distribution during iterative updates. 2) The high-dimensional nature of the 3D search space increases computational complexity. To mitigate this, a singular value decomposition-based compression technique is employed to reduce dimensionality while preserving effective search directions. 3) To further prevent convergence to suboptimal local minima, a singular space reset mechanism dynamically updates the search space based on diversity measures. Extensive experiments demonstrate that ITS3D enhances text-to-3D generation quality, which shows the potential of computationally efficient search methods in generative processes. The source code is available at https://github.com/ZhenglinZhou/ITS3D.
comment : 25 pages, 11 figures
☆ Beyond Real versus Fake Towards Intent-Aware Video Analysis
Saurabh Atreya, Nabyl Quignon, Baptiste Chopin, Abhijit Das, Antitza Dantcheva
The rapid advancement of generative models has led to increasingly realistic deepfake videos, posing significant societal and security risks. While existing detection methods focus on distinguishing real from fake videos, such approaches fail to address a fundamental question: What is the intent behind a manipulated video? Towards addressing this question, we introduce IntentHQ: a new benchmark for human-centered intent analysis, shifting the paradigm from authenticity verification to contextual understanding of videos. IntentHQ consists of 5168 videos that have been meticulously collected and annotated with 23 fine-grained intent-categories, including "Financial fraud", "Indirect marketing", "Political propaganda", as well as "Fear mongering". We perform intent recognition with supervised and self-supervised multi-modality models that integrate spatio-temporal video features, audio processing, and text analysis to infer underlying motivations and goals behind videos. Our proposed model is streamlined to differentiate between a wide range of intent-categories.
☆ Benchmark ing machine learning models for multi-class state recognition in double duantum dot data
Valeria Díaz Moreno, Ryan P Khalili, Daniel Schug, Patrick J. Walsh, Justyna P. Zwolak
Semiconductor quantum dots (QDs) are a leading platform for scalable quantum processors. However, scaling to large arrays requires reliable, automated tuning strategies for devices' bootstrapping, calibration, and operation, with many tuning aspects depending on accurately identifying QD device states from charge-stability diagrams (CSDs). In this work, we present a comprehensive benchmarking study of four modern machine learning (ML) architectures for multi-class state recognition in double-QD CSDs. We evaluate their performance across different data budgets and normalization schemes using both synthetic and experimental data. We find that the more resource-intensive models -- U-Nets and visual transformers (ViTs) -- achieve the highest MSE score (defined as $1-\mathrm{MSE}$) on synthetic data (over $0.98$) but fail to generalize to experimental data. MDNs are the most computationally efficient and exhibit highly stable training, but with substantially lower peak performance. CNNs offer the most favorable trade-off on experimental CSDs, achieving strong accuracy with two orders of magnitude fewer parameters than the U-Nets and ViTs. Normalization plays a nontrivial role: min-max scaling generally yields higher MSE scores but less stable convergence, whereas z-score normalization produces more predictable training dynamics but at reduced accuracy for most models. Overall, our study shows that CNNs with min-max normalization are a practical approach for QD CSDs.
comment : 12 pages, 4 figures, 2 tables
☆ Do You See What I Say? Generalizable Deepfake Detection based on Visual Speech Recognition
Maheswar Bora, Tashvik Dhamija, Shukesh Reddy, Baptiste Chopin, Pranav Balaji, Abhijit Das, Antitza Dantcheva
Deepfake generation has witnessed remarkable progress, contributing to highly realistic generated images, videos, and audio. While technically intriguing, such progress has raised serious concerns related to the misuse of manipulated media. To mitigate such misuse, robust and reliable deepfake detection is urgently needed. Towards this, we propose a novel network FauxNet, which is based on pre-trained Visual Speech Recognition (VSR) features. By extracting temporal VSR features from videos, we identify and segregate real videos from manipulated ones. The holy grail in this context has to do with zero-shot detection, i.e., generalizable detection, which we focus on in this work. FauxNet consistently outperforms the state-of-the-art in this setting. In addition, FauxNet is able to attribute - distinguish between generation techniques from which the videos stem. Finally, we propose new datasets, referred to as Authentica-Vox and Authentica-HDTF, comprising about 38,000 real and fake videos in total, the latter created with six recent deepfake generation techniques. We provide extensive analysis and results on the Authentica datasets and FaceForensics++, demonstrating the superiority of FauxNet. The Authentica datasets will be made publicly available.
☆ What Is the Optimal Ranking Score Between Precision and Recall? We Can Always Find It and It Is Rarely $F_1$
Sébastien Piérard, Adrien Deliège, Marc Van Droogenbroeck
Ranking methods or models based on their performance is of prime importance but is tricky because performance is fundamentally multidimensional. In the case of classification, precision and recall are scores with probabilistic interpretations that are both important to consider and complementary. The rankings induced by these two scores are often in partial contradiction. In practice, therefore, it is extremely useful to establish a compromise between the two views to obtain a single, global ranking. Over the last fifty years or so,it has been proposed to take a weighted harmonic mean, known as the F-score, F-measure, or $F_β$. Generally speaking, by averaging basic scores, we obtain a score that is intermediate in terms of values. However, there is no guarantee that these scores lead to meaningful rankings and no guarantee that the rankings are good tradeoffs between these base scores. Given the ubiquity of $F_β$ scores in the literature, some clarification is in order. Concretely: (1) We establish that $F_β$-induced rankings are meaningful and define a shortest path between precision- and recall-induced rankings. (2) We frame the problem of finding a tradeoff between two scores as an optimization problem expressed with Kendall rank correlations. We show that $F_1$ and its skew-insensitive version are far from being optimal in that regard. (3) We provide theoretical tools and a closed-form expression to find the optimal value for $β$ for any distribution or set of performances, and we illustrate their use on six case studies.
☆ GEO-Detective: Unveiling Location Privacy Risks in Images with LLM Agent s
Xinyu Zhang, Yixin Wu, Boyang Zhang, Chenhao Lin, Chao Shen, Michael Backes, Yang Zhang
Images shared on social media often expose geographic cues. While early geolocation methods required expert effort and lacked generalization, the rise of Large Vision Language Models (LVLMs) now enables accurate geolocation even for ordinary users. However, existing approaches are not optimized for this task. To explore the full potential and associated privacy risks, we present Geo-Detective, an agent that mimics human reasoning and tool use for image geolocation inference. It follows a procedure with four steps that adaptively selects strategies based on image difficulty and is equipped with specialized tools such as visual reverse search, which emulates how humans gather external geographic clues. Experimental results show that GEO-Detective outperforms baseline large vision language models (LVLMs) overall, particularly on images lacking visible geographic features. In country level geolocation tasks, it achieves an improvement of over 11.1% compared to baseline LLMs, and even at finer grained levels, it still provides around a 5.2% performance gain. Meanwhile, when equipped with external clues, GEO-Detective becomes more likely to produce accurate predictions, reducing the "unknown" prediction rate by more than 50.6%. We further explore multiple defense strategies and find that Geo-Detective exhibits stronger robustness, highlighting the need for more effective privacy safeguards.
comment : 15 pages with 7 figures and 12 tables
☆ ABounD: Adversarial Boundary-Driven Few-Shot Learning for Multi-Class Anomaly Detection
Runzhi Deng, Yundi Hu, Xinshuang Zhang, Zhao Wang, Xixi Liu, Wang-Zhou Dai, Caifeng Shan, Fang Zhao
Few-shot multi-class industrial anomaly detection remains a challenging task. Vision-language models need to be both category-adaptive and sharply discriminative, yet data scarcity often blurs the boundary between normal and abnormal states. This ambiguity leads to missed subtle defects and the rejection of atypical normal samples. We propose ABounD, an Adversarial Boundary-Driven few-shot learning for multi-class anomaly detection, which is a unified learning framework that integrates semantic concept learning with decision boundary shaping. The Dynamic Concept Fusion (DCF) module produces class-adaptive prompts by fusing generalizable priors with class-specific cues, conditioned on image features. Meanwhile, Adversarial Boundary Forging (ABF) sculpts a more precise decision margin by generating boundary-level fence features via PGD-style perturbations. Training is conducted in a single stage under a Concept-Boundary Loss, where ABF provides the main supervisory signal and semantic-spatial regularizers stabilize the optimization. This synergy yields a decision boundary that closely follows normal data while preserving flexibility and robust semantic alignment. Experiments on MVTec-AD and VisA datasets demonstrate state-of-the-art performance in the task of few-shot multi-class anomaly detection.
☆ Skeleton Agent : An Agent ic Interaction Framework for Skeleton-based Action Recognition
Hongda Liu, Yunfan Liu, Changlu Wang, Yunlong Wang, Zhenan Sun
Recent advances in skeleton-based action recognition increasingly leverage semantic priors from Large Language Models (LLMs) to enrich skeletal representations. However, the LLM is typically queried in isolation from the recognition model and receives no performance feedback. As a result, it often fails to deliver the targeted discriminative cues critical to distinguish similar actions. To overcome these limitations, we propose SkeletonAgent, a novel framework that bridges the recognition model and the LLM through two cooperative agents, i.e., Questioner and Selector. Specifically, the Questioner identifies the most frequently confused classes and supplies them to the LLM as context for more targeted guidance. Conversely, the Selector parses the LLM's response to extract precise joint-level constraints and feeds them back to the recognizer, enabling finer-grained cross-modal alignment. Comprehensive evaluations on five benchmarks, including NTU RGB+D, NTU RGB+D 120, Kinetics-Skeleton, FineGYM, and UAV-Human, demonstrate that SkeletonAgent consistently outperforms state-of-the-art benchmark methods. The code is available at https://github.com/firework8/SkeletonAgent.
☆ Fin3R: Fine-tuning Feed-forward 3D Reconstruction Models via Monocular Knowledge Distillation NeurIPS 2025
Weining Ren, Hongjun Wang, Xiao Tan, Kai Han
We present Fin3R, a simple, effective, and general fine-tuning method for feed-forward 3D reconstruction models. The family of feed-forward reconstruction model regresses pointmap of all input images to a reference frame coordinate system, along with other auxiliary outputs, in a single forward pass. However, we find that current models struggle with fine geometry and robustness due to (\textit{i}) the scarcity of high-fidelity depth and pose supervision and (\textit{ii}) the inherent geometric misalignment from multi-view pointmap regression. Fin3R jointly tackles two issues with an extra lightweight fine-tuning step. We freeze the decoder, which handles view matching, and fine-tune only the image encoder-the component dedicated to feature extraction. The encoder is enriched with fine geometric details distilled from a strong monocular teacher model on large, unlabeled datasets, using a custom, lightweight LoRA adapter. We validate our method on a wide range of models, including DUSt3R, MASt3R, CUT3R, and VGGT. The fine-tuned models consistently deliver sharper boundaries, recover complex structures, and achieve higher geometric accuracy in both single- and multi-view settings, while adding only the tiny LoRA weights, which leave test-time memory and latency virtually unchanged. Project page: \href{http://visual-ai.github.io/fin3r}{https://visual-ai.github.io/fin3r}
comment : NeurIPS 2025
☆ Wukong's 72 Transformations: High-fidelity Textured 3D Morphing via Flow Models
Minghao Yin, Yukang Cao, Kai Han
We present WUKONG, a novel training-free framework for high-fidelity textured 3D morphing that takes a pair of source and target prompts (image or text) as input. Unlike conventional methods -- which rely on manual correspondence matching and deformation trajectory estimation (limiting generalization and requiring costly preprocessing) -- WUKONG leverages the generative prior of flow-based transformers to produce high-fidelity 3D transitions with rich texture details. To ensure smooth shape transitions, we exploit the inherent continuity of flow-based generative processes and formulate morphing as an optimal transport barycenter problem. We further introduce a sequential initialization strategy to prevent abrupt geometric distortions and preserve identity coherence. For faithful texture preservation, we propose a similarity-guided semantic consistency mechanism that selectively retains high-frequency details and enables precise control over blending dynamics. This avoids common artifacts like oversmoothing while maintaining semantic fidelity. Extensive quantitative and qualitative evaluations demonstrate that WUKONG significantly outperforms state-of-the-art methods, achieving superior results across diverse geometry and texture variations.
☆ DiffStyle360: Diffusion -Based 360° Head Stylization via Style Fusion Attention
Furkan Guzelant, Arda Goktogan, Tarık Kaya, Aysegul Dundar
3D head stylization has emerged as a key technique for reimagining realistic human heads in various artistic forms, enabling expressive character design and creative visual experiences in digital media. Despite the progress in 3D-aware generation, existing 3D head stylization methods often rely on computationally expensive optimization or domain-specific fine-tuning to adapt to new styles. To address these limitations, we propose DiffStyle360, a diffusion-based framework capable of producing multi-view consistent, identity-preserving 3D head stylizations across diverse artistic domains given a single style reference image, without requiring per-style training. Building upon the 3D-aware DiffPortrait360 architecture, our approach introduces two key components: the Style Appearance Module, which disentangles style from content, and the Style Fusion Attention mechanism, which adaptively balances structure preservation and stylization fidelity in the latent space. Furthermore, we employ a 3D GAN-generated multi-view dataset for robust fine-tuning and introduce a temperaturebased key scaling strategy to control stylization intensity during inference. Extensive experiments on FFHQ and RenderMe360 demonstrate that DiffStyle360 achieves superior style quality, outperforming state-of-the-art GAN- and diffusion-based stylization methods across challenging style domains.
☆ UAV-MM3D: A Large-Scale Synthetic Benchmark for 3D Perception of Unmanned Aerial Vehicles with Multi-Modal Data
Longkun Zou, Jiale Wang, Rongqin Liang, Hai Wu, Ke Chen, Yaowei Wang
Accurate perception of UAVs in complex low-altitude environments is critical for airspace security and related intelligent systems. Developing reliable solutions requires large-scale, accurately annotated, and multimodal data. However, real-world UAV data collection faces inherent constraints due to airspace regulations, privacy concerns, and environmental variability, while manual annotation of 3D poses and cross-modal correspondences is time-consuming and costly. To overcome these challenges, we introduce UAV-MM3D, a high-fidelity multimodal synthetic dataset for low-altitude UAV perception and motion understanding. It comprises 400K synchronized frames across diverse scenes (urban areas, suburbs, forests, coastal regions) and weather conditions (clear, cloudy, rainy, foggy), featuring multiple UAV models (micro, small, medium-sized) and five modalities - RGB, IR, LiDAR, Radar, and DVS (Dynamic Vision Sensor). Each frame provides 2D/3D bounding boxes, 6-DoF poses, and instance-level annotations, enabling core tasks related to UAVs such as 3D detection, pose estimation, target tracking, and short-term trajectory forecasting. We further propose LGFusionNet, a LiDAR-guided multimodal fusion baseline, and a dedicated UAV trajectory prediction baseline to facilitate benchmarking. With its controllable simulation environment, comprehensive scenario coverage, and rich annotations, UAV3D offers a public benchmark for advancing 3D perception of UAVs.
☆ Asking like S ocr ates: S ocr ates helps VLMs understand remote sensing images
Run Shao, Ziyu Li, Zhaoyang Zhang, Linrui Xu, Xinran He, Hongyuan Yuan, Bolei He, Yongxing Dai, Yiming Yan, Yijun Chen, Wang Guo, Haifeng Li
Recent multimodal reasoning models, inspired by DeepSeek-R1, have significantly advanced vision-language systems. However, in remote sensing (RS) tasks, we observe widespread pseudo reasoning: models narrate the process of reasoning rather than genuinely reason toward the correct answer based on visual evidence. We attribute this to the Glance Effect, where a single, coarse perception of large-scale RS imagery results in incomplete understanding and reasoning based on linguistic self-consistency instead of visual evidence. To address this, we propose RS-EoT (Remote Sensing Evidence-of-Thought), a language-driven, iterative visual evidence-seeking paradigm. To instill this paradigm, we propose SocraticAgent, a self-play multi-agent system that synthesizes reasoning traces via alternating cycles of reasoning and visual inspection. To enhance and generalize these patterns, we propose a two-stage progressive RL strategy: first, RL on fine-grained Grounding tasks to enhance RS-EoT capabilities, followed by RL on RS VQA to generalize to broader understanding scenarios. Experiments show RS-EoT achieves state-of-the-art performance on multiple RS VQA and grounding benchmarks. Analyses reveal clear iterative cycles of reasoning and evidence seeking, confirming RS-EoT mitigates the Glance Effect and enables genuine evidence-grounded reasoning. Our code, data, and models are available at https://geox-lab.github.io/Asking_like_Socrates
comment : 20 pages, 8 figures
☆ AnchorFlow: Training-Free 3D Editing via Latent Anchor-Aligned Flows
Zhenglin Zhou, Fan Ma, Chengzhuo Gui, Xiaobo Xia, Hehe Fan, Yi Yang, Tat-Seng Chua
Training-free 3D editing aims to modify 3D shapes based on human instructions without model finetuning. It plays a crucial role in 3D content creation. However, existing approaches often struggle to produce strong or geometrically stable edits, largely due to inconsistent latent anchors introduced by timestep-dependent noise during diffusion sampling. To address these limitations, we introduce AnchorFlow, which is built upon the principle of latent anchor consistency. Specifically, AnchorFlow establishes a global latent anchor shared between the source and target trajectories, and enforces coherence using a relaxed anchor-alignment loss together with an anchor-aligned update rule. This design ensures that transformations remain stable and semantically faithful throughout the editing process. By stabilizing the latent reference space, AnchorFlow enables more pronounced semantic modifications. Moreover, AnchorFlow is mask-free. Without mask supervision, it effectively preserves geometric fidelity. Experiments on the Eval3DEdit benchmark show that AnchorFlow consistently delivers semantically aligned and structurally robust edits across diverse editing types. Code is at https://github.com/ZhenglinZhou/AnchorFlow.
comment : 20 pages, 10 figures
☆ INSIGHT: An Interpretable Neural Vision -Language Framework for Reasoning of Generative Artifacts
Anshul Bagaria
The growing realism of AI-generated images produced by recent GAN and diffusion models has intensified concerns over the reliability of visual media. Yet, despite notable progress in deepfake detection, current forensic systems degrade sharply under real-world conditions such as severe downsampling, compression, and cross-domain distribution shifts. Moreover, most detectors operate as opaque classifiers, offering little insight into why an image is flagged as synthetic, undermining trust and hindering adoption in high-stakes settings. We introduce INSIGHT (Interpretable Neural Semantic and Image-based Generative-forensic Hallucination Tracing), a unified multimodal framework for robust detection and transparent explanation of AI-generated images, even at extremely low resolutions (16x16 - 64x64). INSIGHT combines hierarchical super-resolution for amplifying subtle forensic cues without inducing misleading artifacts, Grad-CAM driven multi-scale localization to reveal spatial regions indicative of generative patterns, and CLIP-guided semantic alignment to map visual anomalies to human-interpretable descriptors. A vision-language model is then prompted using a structured ReAct + Chain-of-Thought protocol to produce consistent, fine-grained explanations, verified through a dual-stage G-Eval + LLM-as-a-judge pipeline to minimize hallucinations and ensure factuality. Across diverse domains, including animals, vehicles, and abstract synthetic scenes, INSIGHT substantially improves both detection robustness and explanation quality under extreme degradation, outperforming prior detectors and black-box VLM baselines. Our results highlight a practical path toward transparent, reliable AI-generated image forensics and establish INSIGHT as a step forward in trustworthy multimodal content verification.
comment : 36 pages, 17 figures
☆ Flowing Backwards: Improving Normalizing Flows via Reverse Representation Alignment AAAI 2026
Yang Chen, Xiaowei Xu, Shuai Wang, Chenhui Zhu, Ruxue Wen, Xubin Li, Tiezheng Ge, Limin Wang
Normalizing Flows (NFs) are a class of generative models distinguished by a mathematically invertible architecture, where the forward pass transforms data into a latent space for density estimation, and the reverse pass generates new samples from this space. This characteristic creates an intrinsic synergy between representation learning and data generation. However, the generative quality of standard NFs is limited by poor semantic representations from log-likelihood optimization. To remedy this, we propose a novel alignment strategy that creatively leverages the invertibility of NFs: instead of regularizing the forward pass, we align the intermediate features of the generative (reverse) pass with representations from a powerful vision foundation model, demonstrating superior effectiveness over naive alignment. We also introduce a novel training-free, test-time optimization algorithm for classification, which provides a more intrinsic evaluation of the NF's embedded semantic knowledge. Comprehensive experiments demonstrate that our approach accelerates the training of NFs by over 3.3$\times$, while simultaneously delivering significant improvements in both generative quality and classification accuracy. New state-of-the-art results for NFs are established on ImageNet 64$\times$64 and 256$\times$256. Our code is available at https://github.com/MCG-NJU/FlowBack.
comment : Accepted by AAAI 2026
☆ Unexplored flaws in multiple-choice VQA evaluations
Fabio Rosenthal, Sebastian Schmidt, Thorsten Graf, Thorsten Bagodonat, Stephan Günnemann, Leo Schwinn
Multimodal Large Language Models (MLLMs) demonstrate strong capabilities in handling image-text inputs. A common way to assess this ability is through multiple-choice Visual Question Answering (VQA). Earlier works have already revealed that these benchmarks are sensitive to answer choice order, a limitation that can be mitigated through careful design. Yet, we highlight additional, unexplored biases in prompt formatting that question the reliability of current MLLM evaluations. Specifically, we identify three key variation factors in prompt formatting and analyze their impact through a large-scale study involving $\mathbf{\text{seven}}$ MLLMs and $\mathbf{\text{five}}$ VQA datasets, spanning $\mathbf{48}$ distinct $\mathbf{\text{prompt format variations}}$. Our findings reveal that multiple-choice VQA is highly sensitive to minor prompt format changes, even when these changes are semantically neutral. We further demonstrate that these biases persist independently of known order biases or the MLLM's confidence in the correct answer. Finally, we demonstrate that existing bias mitigation strategies fail to address these newly identified biases.
☆ Prompt -based Consistent Video Colorization
Silvia Dani, Tiberio Uricchio, Lorenzo Seidenari
Existing video colorization methods struggle with temporal flickering or demand extensive manual input. We propose a novel approach automating high-fidelity video colorization using rich semantic guidance derived from language and segmentation. We employ a language-conditioned diffusion model to colorize grayscale frames. Guidance is provided via automatically generated object masks and textual prompts; our primary automatic method uses a generic prompt, achieving state-of-the-art results without specific color input. Temporal stability is achieved by warping color information from previous frames using optical flow (RAFT); a correction step detects and fixes inconsistencies introduced by warping. Evaluations on standard benchmarks (DAVIS30, VIDEVO20) show our method achieves state-of-the-art performance in colorization accuracy (PSNR) and visual realism (Colorfulness, CDC), demonstrating the efficacy of automated prompt-based guidance for consistent video colorization.
☆ Content Adaptive Encoding For Interactive Game Streaming
Shakarim Soltanayev, Odysseas Zisimopoulos, Mohammad Ashraful Anam, Man Cheung Kung, Angeliki Katsenou, Yiannis Andreopoulos
Video-on-demand streaming has benefitted from \textit{content-adaptive encoding} (CAE), i.e., adaptation of resolution and/or quantization parameters for each scene based on convex hull optimization. However, CAE is very challenging to develop and deploy for interactive game streaming (IGS). Commercial IGS services impose ultra-low latency encoding with no lookahead or buffering, and have extremely tight compute constraints for any CAE algorithm execution. We propose the first CAE approach for resolution adaptation in IGS based on compact encoding metadata from past frames. Specifically, we train a convolutional neural network (CNN) to infer the best resolution from the options available for the upcoming scene based on a running window of aggregated coding block statistics from the current scene. By deploying the trained CNN within a practical IGS setup based on HEVC encoding, our proposal: (i) improves over the default fixed-resolution ladder of HEVC by 2.3 Bjøntegaard Delta-VMAF points; (ii) infers using 1ms of a single CPU core per scene, thereby having no latency overhead.
comment : 5 pages
☆ Small Object Detection for Birds with Swin Transformer
Da Huo, Marc A. Kastner, Tingwei Liu, Yasutomo Kawanishi, Takatsugu Hirayama, Takahiro Komamizu, Ichiro Ide
Object detection is the task of detecting objects in an image. In this task, the detection of small objects is particularly difficult. Other than the small size, it is also accompanied by difficulties due to blur, occlusion, and so on. Current small object detection methods are tailored to small and dense situations, such as pedestrians in a crowd or far objects in remote sensing scenarios. However, when the target object is small and sparse, there is a lack of objects available for training, making it more difficult to learn effective features. In this paper, we propose a specialized method for detecting a specific category of small objects; birds. Particularly, we improve the features learned by the neck; the sub-network between the backbone and the prediction head, to learn more effective features with a hierarchical design. We employ Swin Transformer to upsample the image features. Moreover, we change the shifted window size for adapting to small objects. Experiments show that the proposed Swin Transformer-based neck combined with CenterNet can lead to good performance by changing the window sizes. We further find that smaller window sizes (default 2) benefit mAPs for small object detection.
comment : This paper is included in the proceedings of the 18th International Conference on Machine Vision Applications (MVA2023) (https://www.mva-org.jp/mva2023/challenge/index) The paper has received Runner-Up Solution Award (2nd) and Best Booster Award from Small Object Detection Challenge for Spotting Birds 2023 in MVA
☆ Structure is Super vision : Multiview Masked Autoencoders for Radiology
Sonia Laguna, Andrea Agostini, Alain Ryser, Samuel Ruiperez-Campillo, Irene Cannistraci, Moritz Vandenhirtz, Stephan Mandt, Nicolas Deperrois, Farhad Nooralahzadeh, Michael Krauthammer, Thomas M. Sutter, Julia E. Vogt
Building robust medical machine learning systems requires pretraining strategies that exploit the intrinsic structure present in clinical data. We introduce Multiview Masked Autoencoder (MVMAE), a self-supervised framework that leverages the natural multi-view organization of radiology studies to learn view-invariant and disease-relevant representations. MVMAE combines masked image reconstruction with cross-view alignment, transforming clinical redundancy across projections into a powerful self-supervisory signal. We further extend this approach with MVMAE-V2T, which incorporates radiology reports as an auxiliary text-based learning signal to enhance semantic grounding while preserving fully vision-based inference. Evaluated on a downstream disease classification task on three large-scale public datasets, MIMIC-CXR, CheXpert, and PadChest, MVMAE consistently outperforms supervised and vision-language baselines. Furthermore, MVMAE-V2T provides additional gains, particularly in low-label regimes where structured textual supervision is most beneficial. Together, these results establish the importance of structural and textual supervision as complementary paths toward scalable, clinically grounded medical foundation models.
☆ Match-and-Fuse: Consistent Generation from Unstructured Image Sets
Kate Feingold, Omri Kaduri, Tali Dekel
We present Match-and-Fuse - a zero-shot, training-free method for consistent controlled generation of unstructured image sets - collections that share a common visual element, yet differ in viewpoint, time of capture, and surrounding content. Unlike existing methods that operate on individual images or densely sampled videos, our framework performs set-to-set generation: given a source set and user prompts, it produces a new set that preserves cross-image consistency of shared content. Our key idea is to model the task as a graph, where each node corresponds to an image and each edge triggers a joint generation of image pairs. This formulation consolidates all pairwise generations into a unified framework, enforcing their local consistency while ensuring global coherence across the entire set. This is achieved by fusing internal features across image pairs, guided by dense input correspondences, without requiring masks or manual supervision. It also allows us to leverage an emergent prior in text-to-image models that encourages coherent generation when multiple views share a single canvas. Match-and-Fuse achieves state-of-the-art consistency and visual quality, and unlocks new capabilities for content creation from image collections.
comment : Project page: https://match-and-fuse.github.io/
☆ The Collapse of Patches
Wei Guo, Shunqi Mao, Zhuonan Liang, Heng Wang, Weidong Cai
Observing certain patches in an image reduces the uncertainty of others. Their realization lowers the distribution entropy of each remaining patch feature, analogous to collapsing a particle's wave function in quantum mechanics. This phenomenon can intuitively be called patch collapse. To identify which patches are most relied on during a target region's collapse, we learn an autoencoder that softly selects a subset of patches to reconstruct each target patch. Graphing these learned dependencies for each patch's PageRank score reveals the optimal patch order to realize an image. We show that respecting this order benefits various masked image modeling methods. First, autoregressive image generation can be boosted by retraining the state-of-the-art model MAR. Next, we introduce a new setup for image classification by exposing Vision Transformers only to high-rank patches in the collapse order. Seeing 22\% of such patches is sufficient to achieve high accuracy. With these experiments, we propose patch collapse as a novel image modeling perspective that promotes vision efficiency. Our project is available at https://github.com/wguo-ai/CoP .
comment : 13 pages, 10 figures
☆ DriveVGGT: Visual Geometry Transformer for Autonomous Driving
Xiaosong Jia, Yanhao Liu, Junqi You, Renqiu Xia, Yu Hong, Junchi Yan
Feed-forward reconstruction has recently gained significant attention, with VGGT being a notable example. However, directly applying VGGT to autonomous driving (AD) systems leads to sub-optimal results due to the different priors between the two tasks. In AD systems, several important new priors need to be considered: (i) The overlap between camera views is minimal, as autonomous driving sensor setups are designed to achieve coverage at a low cost. (ii) The camera intrinsics and extrinsics are known, which introduces more constraints on the output and also enables the estimation of absolute scale. (iii) Relative positions of all cameras remain fixed though the ego vehicle is in motion. To fully integrate these priors into a feed-forward framework, we propose DriveVGGT, a scale-aware 4D reconstruction framework specifically designed for autonomous driving data. Specifically, we propose a Temporal Video Attention (TVA) module to process multi-camera videos independently, which better leverages the spatiotemporal continuity within each single-camera sequence. Then, we propose a Multi-camera Consistency Attention (MCA) module to conduct window attention with normalized relative pose embeddings, aiming to establish consistency relationships across different cameras while restricting each token to attend only to nearby frames. Finally, we extend the standard VGGT heads by adding an absolute scale head and an ego vehicle pose head. Experiments show that DriveVGGT outperforms VGGT, StreamVGGT, fastVGGT on autonomous driving dataset while extensive ablation studies verify effectiveness of the proposed designs.
☆ Can Protective Watermarking Safeguard the Copyright of 3D Gaussian Splatting? AAAI 2026
Wenkai Huang, Yijia Guo, Gaolei Li, Lei Ma, Hang Zhang, Liwen Hu, Jiazheng Wang, Jianhua Li, Tiejun Huang
3D Gaussian Splatting (3DGS) has emerged as a powerful representation for 3D scenes, widely adopted due to its exceptional efficiency and high-fidelity visual quality. Given the significant value of 3DGS assets, recent works have introduced specialized watermarking schemes to ensure copyright protection and ownership verification. However, can existing 3D Gaussian watermarking approaches genuinely guarantee robust protection of the 3D assets? In this paper, for the first time, we systematically explore and validate possible vulnerabilities of 3DGS watermarking frameworks. We demonstrate that conventional watermark removal techniques designed for 2D images do not effectively generalize to the 3DGS scenario due to the specialized rendering pipeline and unique attributes of each gaussian primitives. Motivated by this insight, we propose GSPure, the first watermark purification framework specifically for 3DGS watermarking representations. By analyzing view-dependent rendering contributions and exploiting geometrically accurate feature clustering, GSPure precisely isolates and effectively removes watermark-related Gaussian primitives while preserving scene integrity. Extensive experiments demonstrate that our GSPure achieves the best watermark purification performance, reducing watermark PSNR by up to 16.34dB while minimizing degradation to original scene fidelity with less than 1dB PSNR loss. Moreover, it consistently outperforms existing methods in both effectiveness and generalization.
comment : Accepted by AAAI 2026
☆ UMind-VL: A Generalist Ultrasound Vision -Language Model for Unified Grounded Perception and Comprehensive Interpretation
Dengbo Chen, Ziwei Zhao, Kexin Zhang, Shishuang Zhao, Junjie Hou, Yaqian Wang, Nianxi Liao, Anlan Sun, Fei Gao, Jia Ding, Yuhang Liu, Dong Wang
Despite significant strides in medical foundation models, the ultrasound domain lacks a comprehensive solution capable of bridging low-level Ultrasound Grounded Perception (e.g., segmentation, localization) and high-level Ultrasound Comprehensive Interpretation (e.g., diagnosis, reasoning). To bridge this gap, we propose UMind-VL, a unified foundation model designed to synergize pixel-level structural understanding with complex clinical reasoning. We first introduce UMind-DS, a large-scale multimodal dataset comprising 1.2 million ultrasound image-text pairs across 16 anatomical regions, enriching standard data with pixel-level annotations and clinician-validated rationales. Architecturally, UMind-VL incorporates a lightweight Dynamic Convolutional Mask Decoder that generates masks via dynamic kernels conditioned on LLM outputs. This design, combined with task-specific tokens, unifies segmentation, detection, geometric measurement, and diagnosis tasks within a single framework. Extensive evaluations demonstrate that UMind-VL significantly outperforms existing generalist multimodal models and achieves performance on par with, or superior to, state-of-the-art specialist models across segmentation, detection, keypoint localization, and diagnostic reasoning benchmarks, while maintaining strong generalization ability. We demonstrate the capability of UMind-VL in Figure 1.
☆ UNION: A Lightweight Target Representation for Efficient Zero-Shot Image-Guided Retrieval with Optional Textual Queries ICDM
Hoang-Bao Le, Allie Tran, Binh T. Nguyen, Liting Zhou, Cathal Gurrin
Image-Guided Retrieval with Optional Text (IGROT) is a general retrieval setting where a query consists of an anchor image, with or without accompanying text, aiming to retrieve semantically relevant target images. This formulation unifies two major tasks: Composed Image Retrieval (CIR) and Sketch-Based Image Retrieval (SBIR). In this work, we address IGROT under low-data supervision by introducing UNION, a lightweight and generalisable target representation that fuses the image embedding with a null-text prompt. Unlike traditional approaches that rely on fixed target features, UNION enhances semantic alignment with multimodal queries while requiring no architectural modifications to pretrained vision-language models. With only 5,000 training samples - from LlavaSCo for CIR and Training-Sketchy for SBIR - our method achieves competitive results across benchmarks, including CIRCO mAP@50 of 38.5 and Sketchy mAP@200 of 82.7, surpassing many heavily supervised baselines. This demonstrates the robustness and efficiency of UNION in bridging vision and language across diverse query types.
comment : Accepted at ICDM - MMSR Workshop 2025
☆ ColonAdapter: Geometry Estimation Through Foundation Model Adaptation for Colonoscopy
Zhiyi Jiang, Yifu Wang, Xuelian Cheng, Zongyuan Ge
Estimating 3D geometry from monocular colonoscopy images is challenging due to non-Lambertian surfaces, moving light sources, and large textureless regions. While recent 3D geometric foundation models eliminate the need for multi-stage pipelines, their performance deteriorates in clinical scenes. These models are primarily trained on natural scene datasets and struggle with specularity and homogeneous textures typical in colonoscopy, leading to inaccurate geometry estimation. In this paper, we present ColonAdapter, a self-supervised fine-tuning framework that adapts geometric foundation models for colonoscopy geometry estimation. Our method leverages pretrained geometric priors while tailoring them to clinical data. To improve performance in low-texture regions and ensure scale consistency, we introduce a Detail Restoration Module (DRM) and a geometry consistency loss. Furthermore, a confidence-weighted photometric loss enhances training stability in clinical environments. Experiments on both synthetic and real datasets demonstrate that our approach achieves state-of-the-art performance in camera pose estimation, monocular depth prediction, and dense 3D point map reconstruction, without requiring ground-truth intrinsic parameters.
☆ Toward Diffusible High-Dimensional Latent Spaces: A Frequency Perspective
Bolin Lai, Xudong Wang, Saketh Rambhatla, James M. Rehg, Zsolt Kira, Rohit Girdhar, Ishan Misra
Latent diffusion has become the default paradigm for visual generation, yet we observe a persistent reconstruction-generation trade-off as latent dimensionality increases: higher-capacity autoencoders improve reconstruction fidelity but generation quality eventually declines. We trace this gap to the different behaviors in high-frequency encoding and decoding. Through controlled perturbations in both RGB and latent domains, we analyze encoder/decoder behaviors and find that decoders depend strongly on high-frequency latent components to recover details, whereas encoders under-represent high-frequency contents, yielding insufficient exposure and underfitting in high-frequency bands for diffusion model training. To address this issue, we introduce FreqWarm, a plug-and-play frequency warm-up curriculum that increases early-stage exposure to high-frequency latent signals during diffusion or flow-matching training -- without modifying or retraining the autoencoder. Applied across several high-dimensional autoencoders, FreqWarm consistently improves generation quality: decreasing gFID by 14.11 on Wan2.2-VAE, 6.13 on LTX-VAE, and 4.42 on DC-AE-f32, while remaining architecture-agnostic and compatible with diverse backbones. Our study shows that explicitly managing frequency exposure can successfully turn high-dimensional latent spaces into more diffusible targets.
comment : 11 pages, 7 figures, 4 tables
☆ FIGROTD: A Friendly-to-Handle Dataset for Image Guided Retrieval with Optional Text
Hoang-Bao Le, Allie Tran, Binh T. Nguyen, Liting Zhou, Cathal Gurrin
Image-Guided Retrieval with Optional Text (IGROT) unifies visual retrieval (without text) and composed retrieval (with text). Despite its relevance in applications like Google Image and Bing, progress has been limited by the lack of an accessible benchmark and methods that balance performance across subtasks. Large-scale datasets such as MagicLens are comprehensive but computationally prohibitive, while existing models often favor either visual or compositional queries. We introduce FIGROTD, a lightweight yet high-quality IGROT dataset with 16,474 training triplets and 1,262 test triplets across CIR, SBIR, and CSTBIR. To reduce redundancy, we propose the Variance Guided Feature Mask (VaGFeM), which selectively enhances discriminative dimensions based on variance statistics. We further adopt a dual-loss design (InfoNCE + Triplet) to improve compositional reasoning. Trained on FIGROTD, VaGFeM achieves competitive results on nine benchmarks, reaching 34.8 mAP@10 on CIRCO and 75.7 mAP@200 on Sketchy, outperforming stronger baselines despite fewer triplets.
comment : Accepted at MMM 2026
☆ Semantic Anchoring for Robust Personalization in Text-to-Image Diffusion Models
Seoyun Yang, Gihoon Kim, Taesup Kim
Text-to-image diffusion models have achieved remarkable progress in generating diverse and realistic images from textual descriptions. However, they still struggle with personalization, which requires adapting a pretrained model to depict user-specific subjects from only a few reference images. The key challenge lies in learning a new visual concept from a limited number of reference images while preserving the pretrained semantic prior that maintains text-image alignment. When the model focuses on subject fidelity, it tends to overfit the limited reference images and fails to leverage the pretrained distribution. Conversely, emphasizing prior preservation maintains semantic consistency but prevents the model from learning new personalized attributes. Building on these observations, we propose the personalization process through a semantic anchoring that guides adaptation by grounding new concepts in their corresponding distributions. We therefore reformulate personalization as the process of learning a rare concept guided by its frequent counterpart through semantic anchoring. This anchoring encourages the model to adapt new concepts in a stable and controlled manner, expanding the pretrained distribution toward personalized regions while preserving its semantic structure. As a result, the proposed method achieves stable adaptation and consistent improvements in both subject fidelity and text-image alignment compared to baseline methods. Extensive experiments and ablation studies further demonstrate the robustness and effectiveness of the proposed anchoring strategy.
☆ TTSnap: Test-Time Scaling of Diffusion Models via Noise-Aware Pruning
Qingtao Yu, Changlin Song, Minghao Sun, Zhengyang Yu, Vinay Kumar Verma, Soumya Roy, Sumit Negi, Hongdong Li, Dylan Campbell
A prominent approach to test-time scaling for text-to-image diffusion models formulates the problem as a search over multiple noise seeds, selecting the one that maximizes a certain image-reward function. The effectiveness of this strategy heavily depends on the number and diversity of noise seeds explored. However, verifying each candidate is computationally expensive, because each must be fully denoised before a reward can be computed. This severely limits the number of samples that can be explored under a fixed budget. We propose test-time scaling with noise-aware pruning (TTSnap), a framework that prunes low-quality candidates without fully denoising them. The key challenge is that reward models are learned in the clean image domain, and the ranking of rewards predicted for intermediate estimates are often inconsistent with those predicted for clean images. To overcome this, we train noise-aware reward models via self-distillation to align the reward for intermediate estimates with that of the final clean images. To stabilize learning across different noise levels, we adopt a curriculum training strategy that progressively shifts the data domain from clean images to noise images. In addition, we introduce a new metric that measures reward alignment and computational budget utilization. Experiments demonstrate that our approach improves performance by over 16\% compared with existing methods, enabling more efficient and effective test-time scaling. It also provides orthogonal gains when combined with post-training techniques and local test-time optimization. Code: https://github.com/TerrysLearning/TTSnap/.
☆ Creating Blank Canvas Against AI-enabled Image Forgery AAAI 2026
Qi Song, Ziyuan Luo, Renjie Wan
AIGC-based image editing technology has greatly simplified the realistic-level image modification, causing serious potential risks of image forgery. This paper introduces a new approach to tampering detection using the Segment Anything Model (SAM). Instead of training SAM to identify tampered areas, we propose a novel strategy. The entire image is transformed into a blank canvas from the perspective of neural models. Any modifications to this blank canvas would be noticeable to the models. To achieve this idea, we introduce adversarial perturbations to prevent SAM from ``seeing anything'', allowing it to identify forged regions when the image is tampered with. Due to SAM's powerful perceiving capabilities, naive adversarial attacks cannot completely tame SAM. To thoroughly deceive SAM and make it blind to the image, we introduce a frequency-aware optimization strategy, which further enhances the capability of tamper localization. Extensive experimental results demonstrate the effectiveness of our method.
comment : Accepted by AAAI 2026
☆ Bridging 3D Deep Learning and Curation for Analysis and High-Quality Segmentation in Practice
Simon Püttmann, Jonathan Jair Sànchez Contreras, Lennart Kowitz, Peter Lampen, Saumya Gupta, Davide Panzeri, Nina Hagemann, Qiaojie Xiong, Dirk M. Hermann, Cao Chen, Jianxu Chen
Accurate 3D microscopy image segmentation is critical for quantitative bioimage analysis but even state-of-the-art foundation models yield error-prone results. Therefore, manual curation is still widely used for either preparing high-quality training data or fixing errors before analysis. We present VessQC, an open-source tool for uncertainty-guided curation of large 3D microscopy segmentations. By integrating uncertainty maps, VessQC directs user attention to regions most likely containing biologically meaningful errors. In a preliminary user study uncertainty-guided correction significantly improved error detection recall from 67% to 94.0% (p=0.007) without a significant increase in total curation time. VessQC thus enables efficient, human-in-the-loop refinement of volumetric segmentations and bridges a key gap in real-world applications between uncertainty estimation and practical human-computer interaction. The software is freely available at github.com/MMV-Lab/VessQC.
☆ IE-SRGS: An Internal-External Knowledge Fusion Framework for High-Fidelity 3D Gaussian Splatting Super-Resolution AAAI 2026
Xiang Feng, Tieshi Zhong, Shuo Chang, Weiliu Wang, Chengkai Wang, Yifei Chen, Yuhe Wang, Zhenzhong Kuang, Xuefei Yin, Yanming Zhu
Reconstructing high-resolution (HR) 3D Gaussian Splatting (3DGS) models from low-resolution (LR) inputs remains challenging due to the lack of fine-grained textures and geometry. Existing methods typically rely on pre-trained 2D super-resolution (2DSR) models to enhance textures, but suffer from 3D Gaussian ambiguity arising from cross-view inconsistencies and domain gaps inherent in 2DSR models. We propose IE-SRGS, a novel 3DGS SR paradigm that addresses this issue by jointly leveraging the complementary strengths of external 2DSR priors and internal 3DGS features. Specifically, we use 2DSR and depth estimation models to generate HR images and depth maps as external knowledge, and employ multi-scale 3DGS models to produce cross-view consistent, domain-adaptive counterparts as internal knowledge. A mask-guided fusion strategy is introduced to integrate these two sources and synergistically exploit their complementary strengths, effectively guiding the 3D Gaussian optimization toward high-fidelity reconstruction. Extensive experiments on both synthetic and real-world benchmarks show that IE-SRGS consistently outperforms state-of-the-art methods in both quantitative accuracy and visual fidelity.
comment : AAAI 2026
☆ From Compound Figures to Composite Understanding : Developing a Multi-Modal LLM from Biomedical Literature with Medical Multiple-Image Benchmark ing and Validation
Zhen Chen, Yihang Fu, Gabriel Madera, Mauro Giuffre, Serina Applebaum, Hyunjae Kim, Hua Xu, Qingyu Chen
Multi-modal large language models (MLLMs) have shown promise in advancing healthcare. However, most existing models remain confined to single-image understanding, which greatly limits their applicability in clinical workflows. In practice, medical diagnosis and progression often require synthesizing information across multiple images from different modalities or time points. The development of medical MLLMs capable of such multi-image understanding has been hindered by the lack of large-scale, high-quality annotated training data. To address this limitation, we propose a novel framework that leverages license-permissive compound images in biomedical literature, as a rich yet underutilized data source for multi-image analysis. Specifically, we design a five-stage, context-aware instruction generation paradigm underpinned by a divide-and-conquer strategy. By decomposing multi-image analysis into manageable sub-tasks, this paradigm empowers MLLMs to move beyond single-panel analysis and provide a composite understanding by learning the complex spatial, temporal, and cross-modal relationships inherent in these compound figures. By parsing over 237,000 compound figures and their contextual text for instruction generation, we develop M3LLM, a medical multi-image multi-modal large language model. For benchmarking, we construct PMC-MI-Bench for composite understanding, manually validated by medical experts. Extensive experiments show that M3LLM significantly outperforms both general-purpose and specialized medical MLLMs across multi-image, single-image, text-only, and multi-choice scenarios. Notably, M3LLM exhibits strong generalization to longitudinal chest X-ray analysis using the MIMIC dataset. This work establishes a scalable and efficient paradigm for developing medical MLLMs capable of composite reasoning, bridging the gap between biomedical literature and real-world clinical applications.
☆ 3D-Consistent Multi-View Editing by Diffusion Guidance
Josef Bengtson, David Nilsson, Dong In Lee, Fredrik Kahl
Recent advancements in diffusion models have greatly improved text-based image editing, yet methods that edit images independently often produce geometrically and photometrically inconsistent results across different views of the same scene. Such inconsistencies are particularly problematic for editing of 3D representations such as NeRFs or Gaussian Splat models. We propose a training-free diffusion framework that enforces multi-view consistency during the image editing process. The key assumption is that corresponding points in the unedited images should undergo similar transformations after editing. To achieve this, we introduce a consistency loss that guides the diffusion sampling toward coherent edits. The framework is flexible and can be combined with widely varying image editing methods, supporting both dense and sparse multi-view editing setups. Experimental results show that our approach significantly improves 3D consistency compared to existing multi-view editing methods. We also show that this increased consistency enables high-quality Gaussian Splat editing with sharp details and strong fidelity to user-specified text prompts. Please refer to our project page for video results: https://3d-consistent-editing.github.io/
☆ Controllable 3D Object Generation with Single Image Prompt
Jaeseok Lee, Jaekoo Lee
Recently, the impressive generative capabilities of diffusion models have been demonstrated, producing images with remarkable fidelity. Particularly, existing methods for the 3D object generation tasks, which is one of the fastest-growing segments in computer vision, pre-dominantly use text-to-image diffusion models with textual inversion which train a pseudo text prompt to describe the given image. In practice, various text-to-image generative models employ textual inversion to learn concepts or styles of target object in the pseudo text prompt embedding space, thereby generating sophisticated outputs. However, textual inversion requires additional training time and lacks control ability. To tackle this issues, we propose two innovative methods: (1) using an off-the-shelf image adapter that generates 3D objects without textual inversion, offering enhanced control over conditions such as depth, pose, and text. (2) a depth conditioned warmup strategy to enhance 3D consistency. In experimental results, ours show qualitatively and quantitatively comparable performance and improved 3D consistency to the existing text-inversion-based alternatives. Furthermore, we conduct a user study to assess (i) how well results match the input image and (ii) whether 3D consistency is maintained. User study results show that our model outperforms the alternatives, validating the effectiveness of our approaches. Our code is available at GitHub repository:https://github.com/Seooooooogi/Control3D_IP/
☆ ARPGNet: Appearance- and Relation -aware Parallel Graph Attention Fusion Network for Facial Expression Recognition
Yan Li, Yong Zhao, Xiaohan Xia, Dongmei Jiang
The key to facial expression recognition is to learn discriminative spatial-temporal representations that embed facial expression dynamics. Previous studies predominantly rely on pre-trained Convolutional Neural Networks (CNNs) to learn facial appearance representations, overlooking the relationships between facial regions. To address this issue, this paper presents an Appearance- and Relation-aware Parallel Graph attention fusion Network (ARPGNet) to learn mutually enhanced spatial-temporal representations of appearance and relation information. Specifically, we construct a facial region relation graph and leverage the graph attention mechanism to model the relationships between facial regions. The resulting relational representation sequences, along with CNN-based appearance representation sequences, are then fed into a parallel graph attention fusion module for mutual interaction and enhancement. This module simultaneously explores the complementarity between different representation sequences and the temporal dynamics within each sequence. Experimental results on three facial expression recognition datasets demonstrate that the proposed ARPGNet outperforms or is comparable to state-of-the-art methods.
comment : Accepted by IEEE Transactions on Affective Computing. Submitted in August 2023; Accepted in October 2025
☆ HybridWo rl dSim: A Scalable and Controllable High-fidelity Simulator for Autonomous Driving
Qiang Li, Yingwenqi Jiang, Tuoxi Li, Duyu Chen, Xiang Feng, Yucheng Ao, Shangyue Liu, Xingchen Yu, Youcheng Cai, Yumeng Liu, Yuexin Ma, Xin Hu, Li Liu, Yu Zhang, Linkun Xu, Bingtao Gao, Xueyuan Wang, Shuchang Zhou, Xianming Liu, Ligang Liu
Realistic and controllable simulation is critical for advancing end-to-end autonomous driving, yet existing approaches often struggle to support novel view synthesis under large viewpoint changes or to ensure geometric consistency. We introduce HybridWorldSim, a hybrid simulation framework that integrates multi-traversal neural reconstruction for static backgrounds with generative modeling for dynamic agents. This unified design addresses key limitations of previous methods, enabling the creation of diverse and high-fidelity driving scenarios with reliable visual and spatial consistency. To facilitate robust benchmarking, we further release a new multi-traversal dataset MIRROR that captures a wide range of routes and environmental conditions across different cities. Extensive experiments demonstrate that HybridWorldSim surpasses previous state-of-the-art methods, providing a practical and scalable solution for high-fidelity simulation and a valuable resource for research and development in autonomous driving.
☆ Shoe Style-Invariant and Ground-Aware Learning for Dense Foot Contact Estimation
Daniel Sungho Jung, Kyoung Mu Lee
Foot contact plays a critical role in human interaction with the world, and thus exploring foot contact can advance our understanding of human movement and physical interaction. Despite its importance, existing methods often approximate foot contact using a zero-velocity constraint and focus on joint-level contact, failing to capture the detailed interaction between the foot and the world. Dense estimation of foot contact is crucial for accurately modeling this interaction, yet predicting dense foot contact from a single RGB image remains largely underexplored. There are two main challenges for learning dense foot contact estimation. First, shoes exhibit highly diverse appearances, making it difficult for models to generalize across different styles. Second, ground often has a monotonous appearance, making it difficult to extract informative features. To tackle these issues, we present a FEet COntact estimation (FECO) framework that learns dense foot contact with shoe style-invariant and ground-aware learning. To overcome the challenge of shoe appearance diversity, our approach incorporates shoe style adversarial training that enforces shoe style-invariant features for contact estimation. To effectively utilize ground information, we introduce a ground feature extractor that captures ground properties based on spatial context. As a result, our proposed method achieves robust foot contact estimation regardless of shoe appearance and effectively leverages ground information. Code will be released.
comment : Project page: https://feco-release.github.io/
☆ MTR-VP: Towards End-to-End Trajectory Planning through Context -Driven Image Encoding and Multiple Trajectory Prediction
Maitrayee Keskar, Mohan Trivedi, Ross Greer
We present a method for trajectory planning for autonomous driving, learning image-based context embeddings that align with motion prediction frameworks and planning-based intention input. Within our method, a ViT encoder takes raw images and past kinematic state as input and is trained to produce context embeddings, drawing inspiration from those generated by the recent MTR (Motion Transformer) encoder, effectively substituting map-based features with learned visual representations. MTR provides a strong foundation for multimodal trajectory prediction by localizing agent intent and refining motion iteratively via motion query pairs; we name our approach MTR-VP (Motion Transformer for Vision-based Planning), and instead of the learnable intention queries used in the MTR decoder, we use cross attention on the intent and the context embeddings, which reflect a combination of information encoded from the driving scene and past vehicle states. We evaluate our methods on the Waymo End-to-End Driving Dataset, which requires predicting the agent's future 5-second trajectory in bird's-eye-view coordinates using prior camera images, agent pose history, and routing goals. We analyze our architecture using ablation studies, removing input images and multiple trajectory output. Our results suggest that transformer-based methods that are used to combine the visual features along with the kinetic features such as the past trajectory features are not effective at combining both modes to produce useful scene context embeddings, even when intention embeddings are augmented with foundation-model representations of scene context from CLIP and DINOv2, but that predicting a distribution over multiple futures instead of a single future trajectory boosts planning performance.
comment : 8 pages, 3 figures, 4 tables
☆ Enhanced Graph Convolutional Network with Chebyshev Spectral Graph and Graph Attention for Autism Spectrum Disorder Classification
Adnan Ferdous Ashrafi, Hasanul Kabir
ASD is a complicated neurodevelopmental disorder marked by variation in symptom presentation and neurological underpinnings, making early and objective diagnosis extremely problematic. This paper presents a Graph Convolutional Network (GCN) model, incorporating Chebyshev Spectral Graph Convolution and Graph Attention Networks (GAT), to increase the classification accuracy of ASD utilizing multimodal neuroimaging and phenotypic data. Leveraging the ABIDE I dataset, which contains resting-state functional MRI (rs-fMRI), structural MRI (sMRI), and phenotypic variables from 870 patients, the model leverages a multi-branch architecture that processes each modality individually before merging them via concatenation. Graph structure is encoded using site-based similarity to generate a population graph, which helps in understanding relationship connections across individuals. Chebyshev polynomial filters provide localized spectral learning with lower computational complexity, whereas GAT layers increase node representations by attention-weighted aggregation of surrounding information. The proposed model is trained using stratified five-fold cross-validation with a total input dimension of 5,206 features per individual. Extensive trials demonstrate the enhanced model's superiority, achieving a test accuracy of 74.82\% and an AUC of 0.82 on the entire dataset, surpassing multiple state-of-the-art baselines, including conventional GCNs, autoencoder-based deep neural networks, and multimodal CNNs.
comment : 6 pages, 2 figures, 2 tables, Accepted and presented at Image and Vision Computing New Zealand (IVCNZ) 2025
☆ Designing Instance-Level Sampling Schedules via REINFORCE with James-Stein Shrinkage
Peiyu Yu, Suraj Kothawade, Sirui Xie, Ying Nian Wu, Hongliang Fei
Most post-training methods for text-to-image samplers focus on model weights: either fine-tuning the backbone for alignment or distilling it for few-step efficiency. We take a different route: rescheduling the sampling timeline of a frozen sampler. Instead of a fixed, global schedule, we learn instance-level (prompt- and noise-conditioned) schedules through a single-pass Dirichlet policy. To ensure accurate gradient estimates in high-dimensional policy learning, we introduce a novel reward baseline based on a principled James-Stein estimator; it provably achieves lower estimation errors than commonly used variants and leads to superior performance. Our rescheduled samplers consistently improve text-image alignment including text rendering and compositional control across modern Stable Diffusion and Flux model families. Additionally, a 5-step Flux-Dev sampler with our schedules can attain generation quality comparable to deliberately distilled samplers like Flux-Schnell. We thus position our scheduling framework as an emerging model-agnostic post-training lever that unlocks additional generative potential in pretrained samplers.
comment : 23 pages
☆ Guiding the Inner Eye: A Framework for Hierarchical and Flexible Visual Grounded Reasoning
Zhaoyang Wei, Wenchao Ding, Yanchao Hao, Xi Chen
Models capable of "thinking with images" by dynamically grounding their reasoning in visual evidence represent a major leap in multimodal AI. However, replicating and advancing this ability is non-trivial, with current methods often trapped between the instability of end-to-end reinforcement learning (RL) and the rigidity of supervised fine-tuning (SFT). This leads to models that either struggle to learn or lack the cognitive flexibility required for complex, real-world scenes. To navigate this dilemma, we introduce GRiP (Guided Reasoning and Perception), a novel two-stage training framework that cultivates robust and flexible visual grounded reasoning by explicitly guiding the model's perceptual focus and logical pathways. GRiP's core lies in its cognitive-enhanced RL stage, which features two key innovations: (1) a Salience-Weighted IoU Reward that incentivizes the model to prioritize the localization of mission-critical objects over trivial distractors, and (2) a Multi-Heuristic Reward that encourages cognitive flexibility by rewarding diverse yet logically valid reasoning pathways. Initialized from the Qwen2.5-VL-7B model, GRiP demonstrates significant performance gains across multiple challenging benchmarks. It achieves state-of-the-art results among open-source models on the highly challenging TreeBench and V* Bench, proving its effectiveness in complex visual reasoning. Our work demonstrates that moving beyond simplistic rewards and instead guiding models with cognitively-inspired signals for what to see and how to think is crucial for unlocking the next level of multimodal intelligence. The code will be made publicly available.
comment : 9pages
☆ Brep GPT : Autoregressive B-rep Generation with Voronoi Half-Patch
Pu Li, Wenhao Zhang, Weize Quan, Biao Zhang, Peter Wonka, Dong-Ming Yan
Boundary representation (B-rep) is the de facto standard for CAD model representation in modern industrial design. The intricate coupling between geometric and topological elements in B-rep structures has forced existing generative methods to rely on cascaded multi-stage networks, resulting in error accumulation and computational inefficiency. We present BrepGPT, a single-stage autoregressive framework for B-rep generation. Our key innovation lies in the Voronoi Half-Patch (VHP) representation, which decomposes B-reps into unified local units by assigning geometry to nearest half-edges and sampling their next pointers. Unlike hierarchical representations that require multiple distinct encodings for different structural levels, our VHP representation facilitates unifying geometric attributes and topological relations in a single, coherent format. We further leverage dual VQ-VAEs to encode both vertex topology and Voronoi Half-Patches into vertex-based tokens, achieving a more compact sequential encoding. A decoder-only Transformer is then trained to autoregressively predict these tokens, which are subsequently mapped to vertex-based features and decoded into complete B-rep models. Experiments demonstrate that BrepGPT achieves state-of-the-art performance in unconditional B-rep generation. The framework also exhibits versatility in various applications, including conditional generation from category labels, point clouds, text descriptions, and images, as well as B-rep autocompletion and interpolation.
☆ Partially Shared Concept Bottleneck Models AAAI 2026
Delong Zhao, Qiang Huang, Di Yan, Yiqun Sun, Jun Yu
Concept Bottleneck Models (CBMs) enhance interpretability by introducing a layer of human-understandable concepts between inputs and predictions. While recent methods automate concept generation using Large Language Models (LLMs) and Vision-Language Models (VLMs), they still face three fundamental challenges: poor visual grounding, concept redundancy, and the absence of principled metrics to balance predictive accuracy and concept compactness. We introduce PS-CBM, a Partially Shared CBM framework that addresses these limitations through three core components: (1) a multimodal concept generator that integrates LLM-derived semantics with exemplar-based visual cues; (2) a Partially Shared Concept Strategy that merges concepts based on activation patterns to balance specificity and compactness; and (3) Concept-Efficient Accuracy (CEA), a post-hoc metric that jointly captures both predictive accuracy and concept compactness. Extensive experiments on eleven diverse datasets show that PS-CBM consistently outperforms state-of-the-art CBMs, improving classification accuracy by 1.0%-7.4% and CEA by 2.0%-9.5%, while requiring significantly fewer concepts. These results underscore PS-CBM's effectiveness in achieving both high accuracy and strong interpretability.
comment : 14 pages, 7 figures, 11 tables, Accepted to AAAI 2026
☆ Real-Time Long Horizon Air Quality Forecasting via Group-Relative Policy Optimization
Inha Kang, Eunki Kim, Wonjeong Ryu, Jaeyo Shin, Seungjun Yu, Yoon-Hee Kang, Seongeun Jeong, Eunhye Kim, Soontae Kim, Hyunjung Shim
Accurate long horizon forecasting of particulate matter (PM) concentration fields is essential for operational public health decisions. However, achieving reliable forecasts remains challenging in regions with complex terrain and strong atmospheric dynamics such as East Asia. While foundation models such as Aurora offer global generality, they often miss region-specific dynamics and rely on non-real-time inputs, limiting their practical utility for localized warning systems. To address this gap, we construct and release the real-world observations and high-resolution CMAQ-OBS dataset for East Asia, reducing regional error by 59.5% and enabling real-time 48-120 hour forecasts critical for public health alerts. However, standard point-wise objectives cannot reflect asymmetric operational costs, where false alarms deteriorate public trust while missed severe events endanger populations. This cost mismatch causes SFT models to over-predict and yield high False Alarm Rates. We introduce Group-Relative Policy Optimization (GRPO) with class-wise rewards and curriculum rollout to align predictions with operational priorities. Experimental results demonstrate that our framework significantly improves the reliability of the forecast. Compared to the SFT-only baseline, our model reduces the False Alarm Rate by 47.3% while achieving a competitive F1-score, proving its effectiveness for practical, real-world air quality forecasting systems on long lead time scenarios.
comment : 10 pages
☆ IMTalker: Efficient Audio-driven Talking Face Generation with Implicit Motion Transfer
Bo Chen, Tao Liu, Qi Chen, Xie Chen, Zilong Zheng
Talking face generation aims to synthesize realistic speaking portraits from a single image, yet existing methods often rely on explicit optical flow and local warping, which fail to model complex global motions and cause identity drift. We present IMTalker, a novel framework that achieves efficient and high-fidelity talking face generation through implicit motion transfer. The core idea is to replace traditional flow-based warping with a cross-attention mechanism that implicitly models motion discrepancy and identity alignment within a unified latent space, enabling robust global motion rendering. To further preserve speaker identity during cross-identity reenactment, we introduce an identity-adaptive module that projects motion latents into personalized spaces, ensuring clear disentanglement between motion and identity. In addition, a lightweight flow-matching motion generator produces vivid and controllable implicit motion vectors from audio, pose, and gaze cues. Extensive experiments demonstrate that IMTalker surpasses prior methods in motion accuracy, identity preservation, and audio-lip synchronization, achieving state-of-the-art quality with superior efficiency, operating at 40 FPS for video-driven and 42 FPS for audio-driven generation on an RTX 4090 GPU. We will release our code and pre-trained models to facilitate applications and future research.
comment : 11 pages, 5 figures
☆ RemedyGS: Defend 3D Gaussian Splatting against Computation Cost Attacks
Yanping Li, Zhening Liu, Zijian Li, Zehong Lin, Jun Zhang
As a mainstream technique for 3D reconstruction, 3D Gaussian splatting (3DGS) has been applied in a wide range of applications and services. Recent studies have revealed critical vulnerabilities in this pipeline and introduced computation cost attacks that lead to malicious resource occupancies and even denial-of-service (DoS) conditions, thereby hindering the reliable deployment of 3DGS. In this paper, we propose the first effective and comprehensive black-box defense framework, named RemedyGS, against such computation cost attacks, safeguarding 3DGS reconstruction systems and services. Our pipeline comprises two key components: a detector to identify the attacked input images with poisoned textures and a purifier to recover the benign images from their attacked counterparts, mitigating the adverse effects of these attacks. Moreover, we incorporate adversarial training into the purifier to enforce distributional alignment between the recovered and original natural images, thereby enhancing the defense efficacy. Experimental results demonstrate that our framework effectively defends against white-box, black-box, and adaptive attacks in 3DGS systems, achieving state-of-the-art performance in both safety and utility.
☆ Stacked Ensemble of Fine-Tuned CNNs for Knee Osteoarthritis Severity Grading
Adarsh Gupta, Japleen Kaur, Tanvi Doshi, Teena Sharma, Nishchal K. Verma, Shantaram Vasikarla
Knee Osteoarthritis (KOA) is a musculoskeletal condition that can cause significant limitations and impairments in daily activities, especially among older individuals. To evaluate the severity of KOA, typically, X-ray images of the affected knee are analyzed, and a grade is assigned based on the Kellgren-Lawrence (KL) grading system, which classifies KOA severity into five levels, ranging from 0 to 4. This approach requires a high level of expertise and time and is susceptible to subjective interpretation, thereby introducing potential diagnostic inaccuracies. To address this problem a stacked ensemble model of fine-tuned Convolutional Neural Networks (CNNs) was developed for two classification tasks: a binary classifier for detecting the presence of KOA, and a multiclass classifier for precise grading across the KL spectrum. The proposed stacked ensemble model consists of a diverse set of pre-trained architectures, including MobileNetV2, You Only Look Once (YOLOv8), and DenseNet201 as base learners and Categorical Boosting (CatBoost) as the meta-learner. This proposed model had a balanced test accuracy of 73% in multiclass classification and 87.5% in binary classification, which is higher than previous works in extant literature.
comment : Accepted and Presented at IEEE UEMCON, IBM T.J. Watson Research Center, New York, USA, 2024
☆ SemOD: Semantic Enabled Object Detection Network under Various Weather Conditions
Aiyinsi Zuo, Zhaoliang Zheng
In the field of autonomous driving, camera-based perception models are mostly trained on clear weather data. Models that focus on addressing specific weather challenges are unable to adapt to various weather changes and primarily prioritize their weather removal characteristics. Our study introduces a semantic-enabled network for object detection in diverse weather conditions. In our analysis, semantics information can enable the model to generate plausible content for missing areas, understand object boundaries, and preserve visual coherency and realism across both filled-in and existing portions of the image, which are conducive to image transformation and object recognition. Specific in implementation, our architecture consists of a Preprocessing Unit (PPU) and a Detection Unit (DTU), where the PPU utilizes a U-shaped net enriched by semantics to refine degraded images, and the DTU integrates this semantic information for object detection using a modified YOLO network. Our method pioneers the use of semantic data for all-weather transformations, resulting in an increase between 1.47\% to 8.80\% in mAP compared to existing methods across benchmark datasets of different weather. This highlights the potency of semantics in image enhancement and object detection, offering a comprehensive approach to improving object detection performance. Code will be available at https://github.com/EnisZuo/SemOD.
☆ EASL: Multi-Emotion Guided Semantic Disentanglement for Expressive Sign Language Generation
Yanchao Zhao, Jihao Zhu, Yu Liu, Weizhuo Chen, Yuling Yang, Kun Peng
Large language models have revolutionized sign language generation by automatically transforming text into high-quality sign language videos, providing accessible communication for the Deaf community. However, existing LLM-based approaches prioritize semantic accuracy while overlooking emotional expressions, resulting in outputs that lack naturalness and expressiveness. We propose EASL (Emotion-Aware Sign Language), a multi-emotion-guided generation architecture for fine-grained emotional integration. We introduce emotion-semantic disentanglement modules with progressive training to separately extract semantic and affective features. During pose decoding, the emotional representations guide semantic interaction to generate sign poses with 7-class emotion confidence scores, enabling emotional expression recognition. Experimental results demonstrate that EASL achieves pose accuracy superior to all compared baselines by integrating multi-emotion information and effectively adapts to diffusion models to generate expressive sign language videos.
☆ DualVLA: Building a Generalizable Embodied Agent via Partial Decoupling of Reasoning and Action
Zhen Fang, Zhuoyang Liu, Jiaming Liu, Hao Chen, Yu Zeng, Shiting Huang, Zehui Chen, Lin Chen, Shanghang Zhang, Feng Zhao
To build a generalizable Vision-Language-Action (VLA) model with strong reasoning ability, a common strategy is to first train a specialist VLA on robot demonstrations to acquire reliable manipulation skills, and then incorporate mixed annotated robot data together with multimodal data to restore broader reasoning capabilities. However, we observe that the resulting reasoning VLA often suffers from degraded action performance compared to the specialist model before fine-tuning, a phenomenon we refer to as action degeneration. To address this issue, we propose DualVLA, which enhances action performance through carefully designed post-training while still preserving reasoning capability. We first introduce a dual-layer data pruning method that removes redundant embodied reasoning, preventing it from adversely influencing action learning. To further strengthen action generation, we design a dual-teacher adaptive distillation strategy that assigns different supervision signals to different data domains while maintaining reasoning ability. To fill the evaluation gap for generalist VLAs, we also propose VLA Score, which decouples VLA capability into reasoning, intention, action, and alignment dimensions for a more fine-grained assessment. Experiments show that DualVLA achieves an average success rate of 61.0 in SimplerEnv and an average score of 65.4 across eight competitive multimodal benchmarks, demonstrating a stronger balance between precise action execution and multimodal understanding. Project Website: https://costaliya.github.io/DualVLA/.
☆ Autonomous labeling of surgical resection margins using a foundation model
Xilin Yang, Musa Aydin, Yuhong Lu, Sahan Yoruc Selcuk, Bijie Bai, Yijie Zhang, Andrew Birkeland, Katjana Ehrlich, Julien Bec, Laura Marcu, Nir Pillar, Aydogan Ozcan
Assessing resection margins is central to pathological specimen evaluation and has profound implications for patient outcomes. Current practice employs physical inking, which is applied variably, and cautery artifacts can obscure the true margin on histological sections. We present a virtual inking network (VIN) that autonomously localizes the surgical cut surface on whole-slide images, reducing reliance on inks and standardizing margin-focused review. VIN uses a frozen foundation model as the feature extractor and a compact two-layer multilayer perceptron trained for patch-level classification of cautery-consistent features. The dataset comprised 120 hematoxylin and eosin (H&E) stained slides from 12 human tonsil tissue blocks, resulting in ~2 TB of uncompressed raw image data, where a board-certified pathologist provided boundary annotations. In blind testing with 20 slides from previously unseen blocks, VIN produced coherent margin overlays that qualitatively aligned with expert annotations across serial sections. Quantitatively, region-level accuracy was ~73.3% across the test set, with errors largely confined to limited areas that did not disrupt continuity of the whole-slide margin map. These results indicate that VIN captures cautery-related histomorphology and can provide a reproducible, ink-free margin delineation suitable for integration into routine digital pathology workflows and for downstream measurement of margin distances.
comment : 20 Pages, 5 Figures
☆ GA2-CLIP: Generic Attribute Anchor for Efficient Prompt Tuningin Video-Language Models
Bin Wang, Ruotong Hu, Wenqian Wang, Wentong Li, Mingliang Gao, Runmin Cong, Wei Zhang
Visual and textual soft prompt tuning can effectively improve the adaptability of Vision-Language Models (VLMs) in downstream tasks. However, fine-tuning on video tasks impairs the model's generalization ability to unseen classes. Existing methods attempt to mitigate this forgetting effect by regularizing the gap between hand-crafted prompts and soft prompts, but this also weakens the learning ability of soft prompts. To address this challenge, we propose a plug-and-play coupling prompt learning framework to optimize the generalization performance of V-L models in video tasks, with the core motivation of mitigating semantic space narrowing during fine-tuning by introducing an externally supervised prompt. Specifically, for textual prompts, we introduce pre-trained prompts from other datasets as hard prompt tokens. These are concatenated with soft prompt tokens and coupled via a learnable mapping layer. This competitive prompting approach prevents the semantic space from overfitting to supervised categories. In addition, we introduce a set of well-designed irrelevant video sets and negative prompts as generic attribute anchors to maintain the generic relevance of the attributes in the pre-trained semantic space, thus preserving the generalization ability. Experiments on video tasks demonstrate that our method significantly outperforms state-of-the-art prompt tuning approaches across generalization benchmarks, particularly on base-to-new class prediction.
comment : Technical Report
☆ Cue3D: Quantifying the Role of Image Cues in Single-Image 3D Generation NeurIPS 2025
Xiang Li, Zirui Wang, Zixuan Huang, James M. Rehg
Humans and traditional computer vision methods rely on a diverse set of monocular cues to infer 3D structure from a single image, such as shading, texture, silhouette, etc. While recent deep generative models have dramatically advanced single-image 3D generation, it remains unclear which image cues these methods actually exploit. We introduce Cue3D, the first comprehensive, model-agnostic framework for quantifying the influence of individual image cues in single-image 3D generation. Our unified benchmark evaluates seven state-of-the-art methods, spanning regression-based, multi-view, and native 3D generative paradigms. By systematically perturbing cues such as shading, texture, silhouette, perspective, edges, and local continuity, we measure their impact on 3D output quality. Our analysis reveals that shape meaningfulness, not texture, dictates generalization. Geometric cues, particularly shading, are crucial for 3D generation. We further identify over-reliance on provided silhouettes and diverse sensitivities to cues such as perspective and local continuity across model families. By dissecting these dependencies, Cue3D advances our understanding of how modern 3D networks leverage classical vision cues, and offers directions for developing more transparent, robust, and controllable single-image 3D generation models.
comment : NeurIPS 2025 Highlight; Project page:https://ryanxli.github.io/cue3d/index.html
☆ GoPrune: Accelerated Structured Pruning with $\ell_{2,p}$-Norm Optimization
Li Xu, Xianchao Xiu
Convolutional neural networks (CNNs) suffer from rapidly increasing storage and computational costs as their depth grows, which severely hinders their deployment on resource-constrained edge devices. Pruning is a practical approach for network compression, among which structured pruning is the most effective for inference acceleration. Although existing work has applied the $\ell_p$-norm to pruning, it only considers unstructured pruning with $p\in (0, 1)$ and has low computational efficiency. To overcome these limitations, we propose an accelerated structured pruning method called GoPrune. Our method employs the $\ell_{2,p}$-norm for sparse network learning, where the value of $p$ is extended to $[0, 1)$. Moreover, we develop an efficient optimization algorithm based on the proximal alternating minimization (PAM), and the resulting subproblems enjoy closed-form solutions, thus improving compression efficiency. Experiments on the CIFAR datasets using ResNet and VGG models demonstrate the superior performance of the proposed method in network pruning. Our code is available at https://github.com/xianchaoxiu/GoPrune.
☆ PROMPT MINER: Black-Box Prompt Stealing against Text-to-Image Generative Models via Reinforcement Learning and Fuzz Optimization
Mingzhe Li, Renhao Zhang, Zhiyang Wen, Siqi Pan, Bruno Castro da Silva, Juan Zhai, Shiqing Ma
Text-to-image (T2I) generative models such as Stable Diffusion and FLUX can synthesize realistic, high-quality images directly from textual prompts. The resulting image quality depends critically on well-crafted prompts that specify both subjects and stylistic modifiers, which have become valuable digital assets. However, the rising value and ubiquity of high-quality prompts expose them to security and intellectual-property risks. One key threat is the prompt stealing attack, i.e., the task of recovering the textual prompt that generated a given image. Prompt stealing enables unauthorized extraction and reuse of carefully engineered prompts, yet it can also support beneficial applications such as data attribution, model provenance analysis, and watermarking validation. Existing approaches often assume white-box gradient access, require large-scale labeled datasets for supervised training, or rely solely on captioning without explicit optimization, limiting their practicality and adaptability. To address these challenges, we propose PROMPTMINER, a black-box prompt stealing framework that decouples the task into two phases: (1) a reinforcement learning-based optimization phase to reconstruct the primary subject, and (2) a fuzzing-driven search phase to recover stylistic modifiers. Experiments across multiple datasets and diffusion backbones demonstrate that PROMPTMINER achieves superior results, with CLIP similarity up to 0.958 and textual alignment with SBERT up to 0.751, surpassing all baselines. Even when applied to in-the-wild images with unknown generators, it outperforms the strongest baseline by 7.5 percent in CLIP similarity, demonstrating better generalization. Finally, PROMPTMINER maintains strong performance under defensive perturbations, highlighting remarkable robustness. Code: https://github.com/aaFrostnova/PromptMiner
☆ HyperST: Hierarchical Hyperbolic Learning for Spatial Transcriptomics Prediction
Chen Zhang, Yilu An, Ying Chen, Hao Li, Xitong Ling, Lihao Liu, Junjun He, Yuxiang Lin, Zihui Wang, Rongshan Yu
Spatial Transcriptomics (ST) merges the benefits of pathology images and gene expression, linking molecular profiles with tissue structure to analyze spot-level function comprehensively. Predicting gene expression from histology images is a cost-effective alternative to expensive ST technologies. However, existing methods mainly focus on spot-level image-to-gene matching but fail to leverage the full hierarchical structure of ST data, especially on the gene expression side, leading to incomplete image-gene alignment. Moreover, a challenge arises from the inherent information asymmetry: gene expression profiles contain more molecular details that may lack salient visual correlates in histological images, demanding a sophisticated representation learning approach to bridge this modality gap. We propose HyperST, a framework for ST prediction that learns multi-level image-gene representations by modeling the data's inherent hierarchy within hyperbolic space, a natural geometric setting for such structures. First, we design a Multi-Level Representation Extractors to capture both spot-level and niche-level representations from each modality, providing context-aware information beyond individual spot-level image-gene pairs. Second, a Hierarchical Hyperbolic Alignment module is introduced to unify these representations, performing spatial alignment while hierarchically structuring image and gene embeddings. This alignment strategy enriches the image representations with molecular semantics, significantly improving cross-modal prediction. HyperST achieves state-of-the-art performance on four public datasets from different tissues, paving the way for more scalable and accurate spatial transcriptomics prediction.
☆ MoE3D: Mixture of Experts meets Multi-Modal 3D Understanding
Yu Li, Yuenan Hou, Yingmei Wei, Xinge Zhu, Yuexin Ma, Wenqi Shao, Yanming Guo
Multi-modal 3D understanding is a fundamental task in computer vision. Previous multi-modal fusion methods typically employ a single, dense fusion network, struggling to handle the significant heterogeneity and complexity across modalities, leading to suboptimal performance. In this paper, we propose MoE3D, which integrates Mixture of Experts (MoE) into the multi-modal learning framework. The core is that we deploy a set of specialized "expert" networks, each adept at processing a specific modality or a mode of cross-modal interaction. Specifically, the MoE-based transformer is designed to better utilize the complementary information hidden in the visual features. Information aggregation module is put forward to further enhance the fusion performance. Top-1 gating is employed to make one expert process features with expert groups, ensuring high efficiency. We further propose a progressive pre-training strategy to better leverage the semantic and 2D prior, thus equipping the network with good initialization. Our MoE3D achieves competitive performance across four prevalent 3D understanding tasks. Notably, our MoE3D surpasses the top-performing counterpart by 6.1 mIoU on Multi3DRefer.
☆ MRI-Based Brain Age Estimation with Supervised Contrastive Learning of Continuous Representation
Simon Joseph Clément Crête, Marta Kersten-Oertel, Yiming Xiao
MRI-based brain age estimation models aim to assess a subject's biological brain age based on information, such as neuroanatomical features. Various factors, including neurodegenerative diseases, can accelerate brain aging and measuring this phenomena could serve as a potential biomarker for clinical applications. While deep learning (DL)-based regression has recently attracted major attention, existing approaches often fail to capture the continuous nature of neuromorphological changes, potentially resulting in sub-optimal feature representation and results. To address this, we propose to use supervised contrastive learning with the recent Rank-N-Contrast (RNC) loss to estimate brain age based on widely used T1w structural MRI for the first time and leverage Grad-RAM to visually explain regression results. Experiments show that our proposed method achieves a mean absolute error (MAE) of 4.27 years and an $R^2$ of 0.93 with a limited dataset of training samples, significantly outperforming conventional deep regression with the same ResNet backbone while performing better or comparably with the state-of-the-art methods with significantly larger training data. Furthermore, Grad-RAM revealed more nuanced features related to age regression with the RNC loss than conventional deep regression. As an exploratory study, we employed the proposed method to estimate the gap between the biological and chronological brain ages in Alzheimer's Disease and Parkinson's disease patients, and revealed the correlation between the brain age gap and disease severity, demonstrating its potential as a biomarker in neurodegenerative disorders.
☆ Wo rl dWander: Bridging Egocentric and Exocentric Wo rl ds in Video Generation
Quanjian Song, Yiren Song, Kelly Peng, Yuan Gao, Mike Zheng Shou
Video diffusion models have recently achieved remarkable progress in realism and controllability. However, achieving seamless video translation across different perspectives, such as first-person (egocentric) and third-person (exocentric), remains underexplored. Bridging these perspectives is crucial for filmmaking, embodied AI, and world models. Motivated by this, we present WorldWander, an in-context learning framework tailored for translating between egocentric and exocentric worlds in video generation. Building upon advanced video diffusion transformers, WorldWander integrates (i) In-Context Perspective Alignment and (ii) Collaborative Position Encoding to efficiently model cross-view synchronization. To further support our task, we curate EgoExo-8K, a large-scale dataset containing synchronized egocentric-exocentric triplets from both synthetic and real-world scenarios. Experiments demonstrate that WorldWander achieves superior perspective synchronization, character consistency, and generalization, setting a new benchmark for egocentric-exocentric video translation.
☆ GACELLE: GPU-accelerated tools for model parameter estimation and image reconstruction
Kwok-Shing Chan, Hansol Lee, Yixin Ma, Berkin Bilgic, Susie Y. Huang, Hong-Hsi Lee, José P. Marques
Quantitative MRI (qMRI) offers tissue-specific biomarkers that can be tracked over time or compared across populations; however, its adoption in clinical research is hindered by significant computational demands of parameter estimation. Images acquired at high spatial resolution or requiring fitting multiple parameters often require lengthy processing time, constraining their use in routine pipelines and slowing methodological innovation and clinical translation. We present GACELLE, an open source, GPU-accelerated framework for high-throughput qMRI analysis. GACELLE provides a stochastic gradient descent optimiser and a stochastic sampler in MATLAB, enabling fast parameter mapping, improved estimation robustness via spatial regularisation, and uncertainty quantification. GACELLE prioritises accessibility: users only need to provide a forward signal model, while GACELLE's backend manages computational parallelisation, automatic parameter updates, and memory-batching. The stochastic solver performs fully vectorised Markov chain Monte Carlo with identical likelihood on CPU and GPU, ensuring reproducibility across hardware. Benchmarking demonstrates up to 451-fold acceleration for the stochastic gradient descent solver and 14,380-fold acceleration for stochastic sampling compared to CPU-based estimation, without compromising accuracy. We demonstrated GACELLE's versatility on three representative qMRI models and on an image reconstruction task. Across these applications, GACELLE improves parameter precision, enhances test-retest reproducibility, and reduces noise in quantitative maps. By combining speed, usability and flexibility, GACELLE provides a generalisable optimisation framework for medical image analysis. It lowers the computational barrier for qMRI, paving the way for reproducible biomarker development, large-scale imaging studies, and clinical translation.
☆ DNA: Dual-branch Network with Adaptation for Open-Set Online Handwriting Generation WACV 2026
Tsai-Ling Huang, Nhat-Tuong Do-Tran, Ngoc-Hoang-Lam Le, Hong-Han Shuai, Ching-Chun Huang
Online handwriting generation (OHG) enhances handwriting recognition models by synthesizing diverse, human-like samples. However, existing OHG methods struggle to generate unseen characters, particularly in glyph-based languages like Chinese, limiting their real-world applicability. In this paper, we introduce our method for OHG, where the writer's style and the characters generated during testing are unseen during training. To tackle this challenge, we propose a Dual-branch Network with Adaptation (DNA), which comprises an adaptive style branch and an adaptive content branch. The style branch learns stroke attributes such as writing direction, spacing, placement, and flow to generate realistic handwriting. Meanwhile, the content branch is designed to generalize effectively to unseen characters by decomposing character content into structural information and texture details, extracted via local and global encoders, respectively. Extensive experiments demonstrate that our DNA model is well-suited for the unseen OHG setting, achieving state-of-the-art performance.
comment : Accepted to WACV 2026
☆ Oral GPT -Omni: A Versatile Dental Multimodal Large Language Model
Jing Hao, Yuci Liang, Lizhuo Lin, Yuxuan Fan, Wenkai Zhou, Kaixin Guo, Zanting Ye, Yanpeng Sun, Xinyu Zhang, Yanqi Yang, Qiankun Li, Hao Tang, James Kit-Hon Tsoi, Linlin Shen, Kuo Feng Hung
Multimodal Large Language Models (MLLMs) have exhibited immense potential across numerous medical specialties; yet, dentistry remains underexplored, in part due to limited domain-specific data, scarce dental expert annotations, insufficient modality-specific modeling, and challenges in reliability. In this paper, we present OralGPT-Omni, the first dental-specialized MLLM designed for comprehensive and trustworthy analysis across diverse dental imaging modalities and clinical tasks. To explicitly capture dentists' diagnostic reasoning, we construct TRACE-CoT, a clinically grounded chain-of-thought dataset that mirrors dental radiologists' decision-making processes. This reasoning supervision, combined with our proposed four-stage training paradigm, substantially strengthens the model's capacity for dental image understanding and analysis. In parallel, we introduce MMOral-Uni, the first unified multimodal benchmark for dental image analysis. It comprises 2,809 open-ended question-answer pairs spanning five modalities and five tasks, offering a comprehensive evaluation suite to date for MLLMs in digital dentistry. OralGPT-Omni achieves an overall score of 51.84 on the MMOral-Uni benchmark and 45.31 on the MMOral-OPG benchmark, dramatically outperforming the scores of GPT-5. Our work promotes intelligent dentistry and paves the way for future advances in dental image analysis. All code, benchmark, and models will be made publicly available.
comment : 47 pages, 42 figures, 13 tables
☆ TPCNet: Triple physical constraints for Low-light Image Enhancement
Jing-Yi Shi, Ming-Fei Li, Ling-An Wu
Low-light image enhancement is an essential computer vision task to improve image contrast and to decrease the effects of color bias and noise. Many existing interpretable deep-learning algorithms exploit the Retinex theory as the basis of model design. However, previous Retinex-based algorithms, that consider reflected objects as ideal Lambertian ignore specular reflection in the modeling process and construct the physical constraints in image space, limiting generalization of the model. To address this issue, we preserve the specular reflection coefficient and reformulate the original physical constraints in the imaging process based on the Kubelka-Munk theory, thereby constructing constraint relationship between illumination, reflection, and detection, the so-called triple physical constraints (TPCs)theory. Based on this theory, the physical constraints are constructed in the feature space of the model to obtain the TPC network (TPCNet). Comprehensive quantitative and qualitative benchmark and ablation experiments confirm that these constraints effectively improve the performance metrics and visual quality without introducing new parameters, and demonstrate that our TPCNet outperforms other state-of-the-art methods on 10 datasets.
☆ ICM-SR: Image-Conditioned Manifold Regularization for Image Super-Resoultion
Junoh Kang, Donghun Ryu, Bohyung Han
Real world image super-resolution (Real-ISR) often leverages the powerful generative priors of text-to-image diffusion models by regularizing the output to lie on their learned manifold. However, existing methods often overlook the importance of the regularizing manifold, typically defaulting to a text-conditioned manifold. This approach suffers from two key limitations. Conceptually, it is misaligned with the Real-ISR task, which is to generate high quality (HQ) images directly tied to the low quality (LQ) images. Practically, the teacher model often reconstructs images with color distortions and blurred edges, indicating a flawed generative prior for this task. To correct these flaws and ensure conceptual alignment, a more suitable manifold must incorporate information from the images. While the most straightforward approach is to condition directly on the raw input images, their high information densities make the regularization process numerically unstable. To resolve this, we propose image-conditioned manifold regularization (ICM), a method that regularizes the output towards a manifold conditioned on the sparse yet essential structural information: a combination of colormap and Canny edges. ICM provides a task-aligned and stable regularization signal, thereby avoiding the instability of dense-conditioning and enhancing the final super-resolution quality. Our experiments confirm that the proposed regularization significantly enhances super-resolution performance, particularly in perceptual quality, demonstrating its effectiveness for real-world applications. We will release the source code of our work for reproducibility.
☆ SparseWo rl d-TC: Trajectory-Conditioned Sparse Occupancy Wo rl d Model
Jiayuan Du, Yiming Zhao, Zhenglong Guo, Yong Pan, Wenbo Hou, Zhihui Hao, Kun Zhan, Qijun Chen
This paper introduces a novel architecture for trajectory-conditioned forecasting of future 3D scene occupancy. In contrast to methods that rely on variational autoencoders (VAEs) to generate discrete occupancy tokens, which inherently limit representational capacity, our approach predicts multi-frame future occupancy in an end-to-end manner directly from raw image features. Inspired by the success of attention-based transformer architectures in foundational vision and language models such as GPT and VGGT, we employ a sparse occupancy representation that bypasses the intermediate bird's eye view (BEV) projection and its explicit geometric priors. This design allows the transformer to capture spatiotemporal dependencies more effectively. By avoiding both the finite-capacity constraint of discrete tokenization and the structural limitations of BEV representations, our method achieves state-of-the-art performance on the nuScenes benchmark for 1-3 second occupancy forecasting, outperforming existing approaches by a significant margin. Furthermore, it demonstrates robust scene dynamics understanding, consistently delivering high accuracy under arbitrary future trajectory conditioning.
☆ PAGen: Phase-guided Amplitude Generation for Domain-adaptive Object Detection
Shuchen Du, Shuo Lei, Feiran Li, Jiacheng Li, Daisuke Iso
Unsupervised domain adaptation (UDA) greatly facilitates the deployment of neural networks across diverse environments. However, most state-of-the-art approaches are overly complex, relying on challenging adversarial training strategies, or on elaborate architectural designs with auxiliary models for feature distillation and pseudo-label generation. In this work, we present a simple yet effective UDA method that learns to adapt image styles in the frequency domain to reduce the discrepancy between source and target domains. The proposed approach introduces only a lightweight pre-processing module during training and entirely discards it at inference time, thus incurring no additional computational overhead. We validate our method on domain-adaptive object detection (DAOD) tasks, where ground-truth annotations are easily accessible in source domains (e.g., normal-weather or synthetic conditions) but challenging to obtain in target domains (e.g., adverse weather or low-light scenes). Extensive experiments demonstrate that our method achieves substantial performance gains on multiple benchmarks, highlighting its practicality and effectiveness.
☆ Layover or Direct Flight: Rethinking Audio-Guided Image Segmentation
Joel Alberto Santos, Zongwei Wu, Xavier Alameda-Pineda, Radu Timofte
Understanding human instructions is essential for enabling smooth human-robot interaction. In this work, we focus on object grounding, i.e., localizing an object of interest in a visual scene (e.g., an image) based on verbal human instructions. Despite recent progress, a dominant research trend relies on using text as an intermediate representation. These approaches typically transcribe speech to text, extract relevant object keywords, and perform grounding using models pretrained on large text-vision datasets. However, we question both the efficiency and robustness of such transcription-based pipelines. Specifically, we ask: Can we achieve direct audio-visual alignment without relying on text? To explore this possibility, we simplify the task by focusing on grounding from single-word spoken instructions. We introduce a new audio-based grounding dataset that covers a wide variety of objects and diverse human accents. We then adapt and benchmark several models from the closely audio-visual field. Our results demonstrate that direct grounding from audio is not only feasible but, in some cases, even outperforms transcription-based methods, especially in terms of robustness to linguistic variability. Our findings encourage a renewed interest in direct audio grounding and pave the way for more robust and efficient multimodal understanding systems.
☆ Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision -Language Models
Zhenxiang Lin, Maryam Haghighat, Will Browne, Dimity Miller
Vision-language models (VLMs), such as CLIP, have gained popularity for their strong open vocabulary classification performance, but they are prone to assigning high confidence scores to misclassifications, limiting their reliability in safety-critical applications. We introduce a training-free, post-hoc uncertainty estimation method for contrastive VLMs that can be used to detect erroneous predictions. The key to our approach is to measure visual feature consistency within a class, using feature projection combined with multivariate Gaussians to create class-specific probabilistic embeddings. Our method is VLM-agnostic, requires no fine-tuning, demonstrates robustness to distribution shift, and works effectively with as few as 10 training images per class. Extensive experiments on ImageNet, Flowers102, Food101, EuroSAT and DTD show state-of-the-art error detection performance, significantly outperforming both deterministic and probabilistic VLM baselines. Code is available at https://github.com/zhenxianglin/ICPE.
☆ MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosis AAAI 2026
Chunzheng Zhu, Yangfang Lin, Shen Chen, Yijun Wang, Jianxin Lin
Accurate medical diagnosis often involves progressive visual focusing and iterative reasoning, characteristics commonly observed in clinical workflows. While recent vision-language models demonstrate promising chain-of-thought (CoT) reasoning capabilities via reinforcement learning with verifiable rewards (RLVR), their purely on-policy learning paradigm tends to reinforce superficially coherent but clinically inaccurate reasoning paths. We propose MedEyes, a novel reinforcement learning framework that dynamically models clinician-style diagnostic reasoning by progressively attending to and interpreting relevant medical image regions. By incorporating off-policy expert guidance, MedEyes converts expert visual search trajectories into structured external behavioral signals, guiding the model toward clinically aligned visual reasoning. We design the Gaze-guided Reasoning Navigator (GRN) to emulate the diagnostic process through a dual-mode exploration strategy, scanning for systematic abnormality localization and drilling for detailed regional analysis. To balance expert imitation and autonomous discovery, we introduce the Confidence Value Sampler (CVS), which employs nucleus sampling and adaptive termination to create diverse yet credible exploration paths. Finally, the dual-stream GRPO optimization framework decouples on-policy and off-policy learning signals, mitigating reward assimilation and entropy collapse. Experiments demonstrate that MedEyes achieves an average performance improvement of +8.5\% across multiple medical VQA benchmarks, validating MedEyes's potential in building interpretable medical AI systems.
comment : This paper has been accepted by AAAI 2026
☆ StreamFlow: Theory, Algorithm, and Implementation for High-Efficiency Rectified Flow Generation
Sen Fang, Hongbin Zhong, Yalin Feng, Dimitris N. Metaxas
New technologies such as Rectified Flow and Flow Matching have significantly improved the performance of generative models in the past two years, especially in terms of control accuracy, generation quality, and generation efficiency. However, due to some differences in its theory, design, and existing diffusion models, the existing acceleration methods cannot be directly applied to the Rectified Flow model. In this article, we have comprehensively implemented an overall acceleration pipeline from the aspects of theory, design, and reasoning strategies. This pipeline uses new methods such as batch processing with a new velocity field, vectorization of heterogeneous time-step batch processing, and dynamic TensorRT compilation for the new methods to comprehensively accelerate related models based on flow models. Currently, the existing public methods usually achieve an acceleration of 18%, while experiments have proved that our new method can accelerate the 512*512 image generation speed to up to 611%, which is far beyond the current non-generalized acceleration methods.
comment : Project Page at https://world-snapshot.github.io/StreamFlow/
☆ When Do Domain-Specific Foundation Models Justify Their Cost? A Systematic Evaluation Across Retinal Imaging Tasks
David Isztl, Tahm Spitznagel, Gabor Mark Somfai, Rui Santos
Large vision foundation models have been widely adopted for retinal disease classification without systematic evidence justifying their parameter requirements. In the present work we address two critical questions: First, are large domain-specific foundation models essential, or do compact general-purpose architectures suffice? Second, does specialized retinal pretraining justify its computational cost? To answer this, we benchmark initialization strategies across four retinal imaging classification tasks spanning Optical Coherence Tomography (OCT) and Color Fundus Photography (CFP) modalities: 8-class OCT classification, 3-class diabetic macular edema (DME), 5-class diabetic retinopathy (DR), and 3-class glaucoma (GL) detection. We evaluate 12-13 model configurations per task, including vision transformers (22.8M-86.6M parameters), Swin Transformers (27.6M-28.3M), ConvNeXt (28.6M), and the domain-specific RETFound models (303M), under identical training conditions. Our results challenge prevailing assumptions: First, we demonstrate that pretraining provides universal benefits (5.18-18.41% improvement), scaling with task difficulty. Second, compact architectures (27-29M) dominate Pareto frontiers; SwinV2-tiny achieves top-1 performance on three datasets. Third, RETFound (303M) justifies its computational cost only for challenging DR grading (accuracy of 71.15%), while ImageNet pretraining proves to be sufficient with all other tasks (DME accuracy: 99.24%, OCT accuracy: 97.96%). CFP tasks show larger pretraining accuracy gains (9.13-18.41%) than OCT (5.18%). Thus, the evidence suggests that compact general-purpose models deliver near-optimal performance for most retinal classification tasks; specialized foundation models warranted only for fine-grained discrimination under extreme class imbalance.
☆ Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance? NeurIPS 2025
Apratim Bhattacharyya, Bicheng Xu, Sanjay Haresh, Reza Pourreza, Litian Liu, Sunny Panchal, Pulkit Madan, Leonid Sigal, Roland Memisevic
Multi-modal Large Language Models (LLM) have advanced conversational abilities but struggle with providing live, interactive step-by-step guidance, a key capability for future AI assistants. Effective guidance requires not only delivering instructions but also detecting their successful execution, as well as identifying and alerting users to mistakes, all of which has to happen in real-time. This requires models that are not turn-based, but that can react asynchronously to a video stream, as well as video data showing users performing tasks including mistakes and their corrections. To this end, we introduce Qualcomm Interactive Cooking, a new benchmark and dataset built upon CaptainCook4D, which contains user mistakes during task execution. Our dataset and benchmark features densely annotated, timed instructions and feedback messages, specifically including mistake alerts precisely timestamped to their visual occurrence in the video. We evaluate state-of-the-art multi-modal LLMs on the Qualcomm Interactive Cooking benchmark and introduce LiveMamba, a streaming multi-modal LLM designed for interactive instructional guidance. This work provides the first dedicated benchmark and a strong baseline for developing and evaluating on live, situated coaching.
comment : Accepted to NeurIPS 2025 (Project page: https://apratimbh.github.io/livecook)
♻ ☆ Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformer s CVPR 2025
Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, Nikos Komodakis
Semantic future prediction is important for autonomous systems navigating dynamic environments. This paper introduces FUTURIST, a method for multimodal future semantic prediction that uses a unified and efficient visual sequence transformer architecture. Our approach incorporates a multimodal masked visual modeling objective and a novel masking mechanism designed for multimodal training. This allows the model to effectively integrate visible information from various modalities, improving prediction accuracy. Additionally, we propose a VAE-free hierarchical tokenization process, which reduces computational complexity, streamlines the training pipeline, and enables end-to-end training with high-resolution, multimodal inputs. We validate FUTURIST on the Cityscapes dataset, demonstrating state-of-the-art performance in future semantic segmentation for both short- and mid-term forecasting. Project page and code at https://futurist-cvpr2025.github.io/ .
comment : CVPR 2025
♻ ☆ Taming generative video models for zero-shot optical flow extraction
Seungwoo Kim, Khai Loong Aw, Klemen Kotar, Cristobal Eyzaguirre, Wanhee Lee, Yunong Liu, Jared Watrous, Stefan Stojanov, Juan Carlos Niebles, Jiajun Wu, Daniel L. K. Yamins
Extracting optical flow from videos remains a core computer vision problem. Motivated by the recent success of large general-purpose models, we ask whether frozen self-supervised video models trained only to predict future frames can be prompted, without fine-tuning, to output flow. Prior attempts to read out depth or illumination from video generators required fine-tuning; that strategy is ill-suited for flow, where labeled data is scarce and synthetic datasets suffer from a sim-to-real gap. Inspired by the Counterfactual World Model (CWM) paradigm, which can obtain point-wise correspondences by injecting a small tracer perturbation into a next-frame predictor and tracking its propagation, we extend this idea to generative video models for zero-shot flow extraction. We explore several popular architectures and find that successful zero-shot flow extraction in this manner is aided by three model properties: (1) distributional prediction of future frames (avoiding blurry or noisy outputs); (2) factorized latents that treat each spatio-temporal patch independently; and (3) random-access decoding that can condition on any subset of future pixels. These properties are uniquely present in the recently introduced Local Random Access Sequence (LRAS) architecture. Building on LRAS, we propose KL-tracing: a novel test-time inference procedure that injects a localized perturbation into the first frame, rolls out the model one step, and computes the Kullback-Leibler divergence between perturbed and unperturbed predictive distributions. Without any flow-specific fine-tuning, our method is competitive with state-of-the-art, task-specific models on the real-world TAP-Vid DAVIS benchmark and the synthetic TAP-Vid Kubric. Our results show that counterfactual prompting of controllable generative video models is an effective alternative to supervised or photometric-loss methods for high-quality flow.
comment : Project webpage: https://neuroailab.github.io/projects/kl_tracing
♻ ☆ Lacking Data? No worries! How synthetic images can alleviate image scarcity in wildlife survey s: a case study with muskox (Ovibos moschatus)
Simon Durand, Samuel Foucher, Alexandre Delplanque, Joëlle Taillon, Jérôme Théau
Accurate population estimates are essential for wildlife management, providing critical insights into species abundance and distribution. Traditional survey methods, including visual aerial counts and GNSS telemetry tracking, are widely used to monitor muskox populations in Arctic regions. These approaches are resource intensive and constrained by logistical challenges. Advances in remote sensing, artificial intelligence, and high resolution aerial imagery offer promising alternatives for wildlife detection. Yet, the effectiveness of deep learning object detection models (ODMs) is often limited by small datasets, making it challenging to train robust ODMs for sparsely distributed species like muskoxen. This study investigates the integration of synthetic imagery (SI) to supplement limited training data and improve muskox detection in zero shot (ZS) and few-shot (FS) settings. We compared a baseline model trained on real imagery with 5 ZS and 5 FS models that incorporated progressively more SI in the training set. For the ZS models, where no real images were included in the training set, adding SI improved detection performance. As more SI were added, performance in precision, recall and F1 score increased, but eventually plateaued, suggesting diminishing returns when SI exceeded 100% of the baseline model training dataset. For FS models, combining real and SI led to better recall and slightly higher overall accuracy compared to using real images alone, though these improvements were not statistically significant. Our findings demonstrate the potential of SI to train accurate ODMs when data is scarce, offering important perspectives for wildlife monitoring by enabling rare or inaccessible species to be monitored and to increase monitoring frequency. This approach could be used to initiate ODMs without real data and refine it as real images are acquired over time.
comment : 34 pages, 10 figures, submitted to Remote Sensing in Ecology and Conservation
♻ ☆ HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation
Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie, Ashmal Vayani, Ahmed Y. Radwan, Mukund S. Chettiar, Amandeep Singh, Mubarak Shah, Deval Pandya
Although recent large multimodal models (LMMs) demonstrate impressive progress on vision language tasks, their alignment with human centered (HC) principles, such as fairness, ethics, inclusivity, empathy, and robustness; remains poorly understood. We present HumaniBench, a unified evaluation framework designed to characterize HC alignment across realistic, socially grounded visual contexts. HumaniBench contains 32,000 expert-verified image question pairs derived from real world news imagery and spanning seven evaluation tasks: scene understanding, instance identity, multiple-choice visual question answering (VQA), multilinguality, visual grounding, empathetic captioning, and image resilience testing. Each task is mapped to one or more HC principles through a principled operationalization of metrics covering accuracy, harmful content detection, hallucination and faithfulness, coherence, cross lingual quality, empathy, and robustness.We evaluate 15 state-of-the-art LMMs under this framework and observe consistent cross model trade offs: proprietary systems achieve the strongest performance on ethics, reasoning, and empathy, while open-source models exhibit superior visual grounding and resilience. All models, however, show persistent gaps in fairness and multilingual inclusivity. We further analyze the effect of inference-time techniques, finding that chain of thought prompting and test-time scaling yield 8 to 12 % improvements on several HC dimensions. HumaniBench provides a reproducible, extensible foundation for systematic HC evaluation of LMMs and enables fine-grained analysis of alignment trade-offs that are not captured by conventional multimodal benchmarks. https://vectorinstitute.github.io/humanibench/
♻ ☆ EasyOcc: 3D Pseudo-Label Super vision for Fully Self-Supervised Semantic Occupancy Prediction Models
Seamie Hayes, Ganesh Sistu, Ciarán Eising
Self-supervised models have recently achieved notable advancements, particularly in the domain of semantic occupancy prediction. These models utilize sophisticated loss computation strategies to compensate for the absence of ground-truth labels. For instance, techniques such as novel view synthesis, cross-view rendering, and depth estimation have been explored to address the issue of semantic and depth ambiguity. However, such techniques typically incur high computational costs and memory usage during the training stage, especially in the case of novel view synthesis. To mitigate these issues, we propose 3D pseudo-ground-truth labels generated by the foundation models Grounded-SAM and Metric3Dv2, and harness temporal information for label densification. Our 3D pseudo-labels can be easily integrated into existing models, which yields substantial performance improvements, with mIoU increasing by 45\%, from 9.73 to 14.09, when implemented into the OccNeRF model. This stands in contrast to earlier advancements in the field, which are often not readily transferable to other architectures. Additionally, we propose a streamlined model, EasyOcc, achieving 13.86 mIoU. This model conducts learning solely from our labels, avoiding complex rendering strategies mentioned previously. Furthermore, our method enables models to attain state-of-the-art performance when evaluated on the full scene without applying the camera mask, with EasyOcc achieving 7.71 mIoU, outperforming the previous best model by 31\%. These findings highlight the critical importance of foundation models, temporal context, and the choice of loss computation space in self-supervised learning for comprehensive scene understanding.
♻ ☆ Diffusion FF: A Diffusion -based Framework for Joint Face Forgery Detection and Fine-Grained Artifact Localization
Siran Peng, Haoyuan Zhang, Li Gao, Tianshuo Zhang, Xiangyu Zhu, Bao Li, Weisong Zhao, Zhen Lei
The rapid evolution of deepfake technologies demands robust and reliable face forgery detection algorithms. While determining whether an image has been manipulated remains essential, the ability to precisely localize forgery clues is also important for enhancing model explainability and building user trust. To address this dual challenge, we introduce DiffusionFF, a diffusion-based framework that simultaneously performs face forgery detection and fine-grained artifact localization. Our key idea is to establish a novel encoder-decoder architecture: a pretrained forgery detector serves as a powerful "artifact encoder", and a denoising diffusion model is repurposed as an "artifact decoder". Conditioned on multi-scale forgery-related features extracted by the encoder, the decoder progressively synthesizes a detailed artifact localization map. We then fuse this fine-grained localization map with high-level semantic features from the forgery detector, leading to substantial improvements in detection capability. Extensive experiments show that DiffusionFF achieves state-of-the-art (SOTA) performance across multiple benchmarks, underscoring its superior effectiveness and explainability.
♻ ☆ InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation NeurIPS 2025
Jinlai Liu, Jian Han, Bin Yan, Hui Wu, Fengda Zhu, Xing Wang, Yi Jiang, Bingyue Peng, Zehuan Yuan
We introduce InfinityStar, a unified spacetime autoregressive framework for high-resolution image and dynamic video synthesis. Building on the recent success of autoregressive modeling in both vision and language, our purely discrete approach jointly captures spatial and temporal dependencies within a single architecture. This unified design naturally supports a variety of generation tasks such as text-to-image, text-to-video, image-to-video, and long interactive video synthesis via straightforward temporal autoregression. Extensive experiments demonstrate that InfinityStar scores 83.74 on VBench, outperforming all autoregressive models by large margins, even surpassing some diffusion competitors like HunyuanVideo. Without extra optimizations, our model generates a 5s, 720p video approximately 10x faster than leading diffusion-based methods. To our knowledge, InfinityStar is the first discrete autoregressive video generator capable of producing industrial level 720p videos. We release all code and models to foster further research in efficient, high-quality video generation.
comment : NeurIPS 2025 Oral
♻ ☆ MRIQT: Physics-Aware Diffusion Model for Image Quality Transfer in Neonatal Ultra-Low-Field MRI
Malek Al Abed, Sebiha Demir, Anne Groteklaes, Elodie Germani, Shahrooz Faghihroohi, Hemmen Sabir, Shadi Albarqouni
Portable ultra-low-field MRI (uLF-MRI, 0.064 T) offers accessible neuroimaging for neonatal care but suffers from low signal-to-noise ratio and poor diagnostic quality compared to high-field (HF) MRI. We propose MRIQT, a 3D conditional diffusion framework for image quality transfer (IQT) from uLF to HF MRI. MRIQT combines realistic K-space degradation for physics-consistent uLF simulation, v-prediction with classifier-free guidance for stable image-to-image generation, and an SNR-weighted 3D perceptual loss for anatomical fidelity. The model denoises from a noised uLF input conditioned on the same scan, leveraging volumetric attention-UNet architecture for structure-preserving translation. Trained on a neonatal cohort with diverse pathologies, MRIQT surpasses recent GAN and CNN baselines in PSNR 15.3% with 1.78% over the state of the art, while physicians rated 85% of its outputs as good quality with clear pathology present. MRIQT enables high-fidelity, diffusion-based enhancement of portable ultra-low-field (uLF) MRI for deliable neonatal brain assessment.
comment : 5 pages, 4 figures
♻ ☆ EMO-X: Efficient Multi-Person Pose and Shape Estimation in One-Stage
Haohang Jian, Jinlu Zhang, Junyi Wu, Zhigang Tu
Expressive Human Pose and Shape Estimation (EHPS) aims to jointly estimate human pose, hand gesture, and facial expression from monocular images. Existing methods predominantly rely on Transformer-based architectures, which suffer from quadratic complexity in self-attention, leading to substantial computational overhead, especially in multi-person scenarios. Recently, Mamba has emerged as a promising alternative to Transformers due to its efficient global modeling capability. However, it remains limited in capturing fine-grained local dependencies, which are essential for precise EHPS. To address these issues, we propose EMO-X, the Efficient Multi-person One-stage model for multi-person EHPS. Specifically, we explore a Scan-based Global-Local Decoder (SGLD) that integrates global context with skeleton-aware local features to iteratively enhance human tokens. Our EMO-X leverages the superior global modeling capability of Mamba and designs a local bidirectional scan mechanism for skeleton-aware local refinement. Comprehensive experiments demonstrate that EMO-X strikes an excellent balance between efficiency and accuracy. Notably, it achieves a significant reduction in computational complexity, requiring 69.8% less inference time compared to state-of-the-art (SOTA) methods, while outperforming most of them in accuracy.
comment : The manuscript is being revised to include new experimental results and an improved model architecture
♻ ☆ ArtiBench and ArtiBrain: Benchmark ing Generalizable Vision -Language Articulated Object Manipulation
Yuhan Wu, Tiantian Wei, Shuo Wang, ZhiChao Wang, Yanyong Zhang, Daniel Cremers, Yan Xia
Interactive articulated manipulation requires long-horizon, multi-step interactions with appliances while maintaining physical consistency. Existing vision-language and diffusion-based policies struggle to generalize across parts, instances, and categories. We first introduce ArtiBench, a five-level benchmark covering kitchen, storage, office, and tool environments. ArtiBench enables structured evaluation from cross-part and cross-instance variation to long-horizon multi-object tasks, revealing the core generalization challenges of articulated object manipulation. Building on this benchmark, we propose ArtiBrain, a modular framework that unifies high-level reasoning with adaptive low-level control. ArtiBrain uses a VLM-based Task Reasoner (GPT-4.1) to decompose and validate subgoals, and employs a Hybrid Controller that combines geometry-aware keyframe execution with affordance-guided diffusion for precise and interpretable manipulation. An Affordance Memory Bank continually accumulates successful execution episodes and propagates part-level actionable affordances to unseen articulated parts and configurations. Extensive experiments on ArtiBench show that our ArtiBrain significantly outperforms state-of-the-art multimodal and diffusion-based methods in robustness and generalization. Code and dataset will be released upon acceptance.
♻ ☆ FreeGaussian: Annotation-free Control of Articulated Objects via 3D Gaussian Splats with Flow Derivatives
Qizhi Chen, Delin Qu, Junli Liu, Yiwen Tang, Haoming Song, Dong Wang, Bin Zhao, Xuelong Li
Reconstructing controllable Gaussian splats for articulated objects from monocular video is especially challenging due to its inherently insufficient constraints. Existing methods address this by relying on dense masks and manually defined control signals, limiting their real-world applications. In this paper, we propose an annotation-free method, FreeGaussian, which mathematically disentangles camera egomotion and articulated movements via flow derivatives. By establishing a connection between 2D flows and 3D Gaussian dynamic flow, our method enables optimization and continuity of dynamic Gaussian motions from flow priors without any control signals. Furthermore, we introduce a 3D spherical vector controlling scheme, which represents the state as a 3D Gaussian trajectory, thereby eliminating the need for complex 1D control signal calculations and simplifying controllable Gaussian modeling. Extensive experiments on articulated objects demonstrate the state-of-the-art visual performance and precise, part-aware controllability of our method. Code is available at: https://github.com/Tavish9/freegaussian.
♻ ☆ Total Least Square Optimal Analytic Signal by Structure Tensor for N-D images
Josef Bigun, Fernando Alonso-Fernandez
We produce the analytic signal by using the Structure Tensor, which provides Total Least Squares optimal vectors for estimating orientation and scale locally. Together, these vectors represent N-D frequency components that determine adaptive, complex probing filters. The N-D analytic signal is obtained through scalar products of adaptive filters with image neighborhoods. It comprises orientation, scale, phase, and amplitude information of the neighborhood. The ST analytic signal $ f_A $ is continuous and isotropic, and its extension to N-D is straightforward. The phase gradient can be represented as a vector (instantaneous frequency) or as a tensor. Both are continuous and isotropic, while the tensor additionally preserves continuity of orientation and retains the same information as the vector representation. The tensor representation can also be used to detect singularities. Detection with known phase portraits has been demonstrated in 2-D with relevance to fringe pattern processing in wave physics, including optics and fingerprint measurements. To construct adaptive filters we have used Gabor filter family members as probing functions, but other function families can also be used to sample the spectrum, e.g., quadrature filters. A comparison to three baseline alternatives-in representation (Monogenic signal), enhancement (Monogenic signal combined with a spline-wavelet pyramid), and singularity detection (mindtct, a fingerprint minutia detector widely used in numerous studies)-is also reported using images with precisely known ground truths for location, orientation, singularity type (where applicable), and wave period.
comment : Changed title, included new experimental results. Reorganized so that details are in Supplementary Material part, which is new
♻ ☆ MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation KR
Yovin Yahathugoda, Davide Prezzi, Piyalitt Ittichaiwong, Vicky Goh, Sebastien Ourselin, Michela Antonelli
Active Surveillance (AS) is a treatment option for managing low and intermediate-risk prostate cancer (PCa), aiming to avoid overtreatment while monitoring disease progression through serial MRI and clinical follow-up. Accurate prostate segmentation is an important preliminary step for automating this process, enabling automated detection and diagnosis of PCa. However, existing deep-learning segmentation models are often trained on single-time-point and expertly annotated datasets, making them unsuitable for longitudinal AS analysis, where multiple time points and a scarcity of expert labels hinder their effective fine-tuning. To address these challenges, we propose MambaX-Net, a novel semi-supervised, dual-scan 3D segmentation architecture that computes the segmentation for time point t by leveraging the MRI and the corresponding segmentation mask from the previous time point. We introduce two new components: (i) a Mamba-enhanced Cross-Attention Module, which integrates the Mamba block into cross attention to efficiently capture temporal evolution and long-range spatial dependencies, and (ii) a Shape Extractor Module that encodes the previous segmentation mask into a latent anatomical representation for refined zone delination. Moreover, we introduce a semi-supervised self-training strategy that leverages pseudo-labels generated from a pre-trained nnU-Net, enabling effective learning without expert annotations. MambaX-Net was evaluated on a longitudinal AS dataset, and results showed that it significantly outperforms state-of-the-art U-Net and Transformer-based models, achieving superior prostate zone segmentation even when trained on limited and noisy data.
comment : Updated the acknowledgments section to include the UKRI Open Access statement
♻ ☆ Fast Gradient Methods for Data-Consistent Local Super-Resolution of Medical Images
Junqi Tang, Guixian Xu, Jinglai Li
In this work, we propose a new paradigm of iterative model-based reconstruction algorithms for providing real-time solution for zooming-in and refining a region of interest in medical and clinical tomographic images. This algorithmic framework is tailored for a clinical need in medical imaging practice that after a reconstruction of the full tomographic image, the clinician may believe that some critical parts of the image are not clear enough, and may wish to see clearer these regions of interest. A naive approach (which is highly not recommended) would be to perform the global reconstruction of a higher resolution image, which has two major limitations: first, it is computationally inefficient, and second, the image regularization is still applied globally, which may over-smooth some local regions. Furthermore, if one wishes to fine-tune the regularization parameter for local parts, it would be computationally infeasible in practice for the case of using global reconstruction. Our new iterative approaches for such tasks are based on jointly utilizing the measurement information, efficient up-sampling/down-sampling across image spaces, and locally adjusted image prior for efficient and high-quality post-processing. The numerical results in low-dose X-ray CT image local zoom-in demonstrate the effectiveness of our approach.
♻ ☆ Unveiling Hidden Vulnerabilities in Digital Human Generation via Adversarial Attacks
Zhiying Li, Yeying Jin, Fan Shen, Zhi Liu, Weibin Chen, Pengju Zhang, Xiaomei Zhang, Boyu Chen, Michael Shen, Kejian Wu, Zhaoxin Fan, Jin Dong
Expressive human pose and shape estimation (EHPS) is crucial for digital human generation, especially in applications like live streaming. While existing research primarily focuses on reducing estimation errors, it largely neglects robustness and security aspects, leaving these systems vulnerable to adversarial attacks. To address this significant challenge, we propose the \textbf{Tangible Attack (TBA)}, a novel framework designed to generate adversarial examples capable of effectively compromising any digital human generation model. Our approach introduces a \textbf{Dual Heterogeneous Noise Generator (DHNG)}, which leverages Variational Autoencoders (VAE) and ControlNet to produce diverse, targeted noise tailored to the original image features. Additionally, we design a custom \textbf{adversarial loss function} to optimize the noise, ensuring both high controllability and potent disruption. By iteratively refining the adversarial sample through multi-gradient signals from both the noise and the state-of-the-art EHPS model, TBA substantially improves the effectiveness of adversarial attacks. Extensive experiments demonstrate TBA's superiority, achieving a remarkable 41.0\% increase in estimation error, with an average improvement of approximately 17.0\%. These findings expose significant security vulnerabilities in current EHPS models and highlight the need for stronger defenses in digital human generation systems.
comment : 14 pages, 7 figures
♻ ☆ Fast Equivariant Imaging: Acceleration for Unsupervised Learning via Augmented Lagrangian and Auxiliary PnP Denoisers
Guixian Xu, Jinglai Li, Junqi Tang
In this work, we propose Fast Equivariant Imaging (FEI), a novel unsupervised learning framework to rapidly and efficiently train deep imaging networks without ground-truth data. From the perspective of reformulating the Equivariant Imaging based optimization problem via the method of Lagrange multipliers and utilizing plug-and-play denoisers, this novel unsupervised scheme shows superior efficiency and performance compared to the vanilla Equivariant Imaging paradigm. In particular, our FEI schemes achieve an order-of-magnitude (10x) acceleration over standard EI on training U-Net for X-ray CT reconstruction and image inpainting, with improved generalization performance.
comment : 17 pages
♻ ☆ CephRes-MHNet: A Multi-Head Residual Network for Accurate and Robust Cephalometric Landmark Detection
Ahmed Jaheen, Islam Hassan, Mohanad Abouserie, Abdelaty Rehab, Adham Elasfar, Knzy Elmasry, Mostafa El-Dawlatly, Seif Eldawlatly
Accurate localization of cephalometric landmarks from 2D lateral skull X-rays is vital for orthodontic diagnosis and treatment. Manual annotation is time-consuming and error-prone, whereas automated approaches often struggle with low contrast and anatomical complexity. This paper introduces CephRes-MHNet, a multi-head residual convolutional network for robust and efficient cephalometric landmark detection. The architecture integrates residual encoding, dual-attention mechanisms, and multi-head decoders to enhance contextual reasoning and anatomical precision. Trained on the Aariz Cephalometric dataset of 1,000 radiographs, CephRes-MHNet achieved a mean radial error (MRE) of 1.23 mm and a success detection rate (SDR) @ 2.0 mm of 85.5%, outperforming all evaluated models. In particular, it exceeded the strongest baseline, the attention-driven AFPF-Net (MRE = 1.25 mm, SDR @ 2.0 mm = 84.1%), while using less than 25% of its parameters. These results demonstrate that CephRes-MHNet attains state-of-the-art accuracy through architectural efficiency, providing a practical solution for real-world orthodontic analysis.
comment : This submission was posted without authorization from all co-authors and supervising institutions. The authors are withdrawing the manuscript due to permission issues
♻ ☆ Enhancing Descriptive Image Quality Assessment with A Large-scale Multi-modal Dataset
Zhiyuan You, Jinjin Gu, Xin Cai, Zheyuan Li, Kaiwen Zhu, Chao Dong, Tianfan Xue
With the rapid advancement of Vision Language Models (VLMs), VLM-based Image Quality Assessment (IQA) seeks to describe image quality linguistically to align with human expression and capture the multifaceted nature of IQA tasks. However, current methods are still far from practical usage. First, prior works focus narrowly on specific sub-tasks or settings, which do not align with diverse real-world applications. Second, their performance is sub-optimal due to limitations in dataset coverage, scale, and quality. To overcome these challenges, we introduce the enhanced Depicted image Quality Assessment model (DepictQA-Wild). Our method includes a multi-functional IQA task paradigm that encompasses both assessment and comparison tasks, brief and detailed responses, full-reference and non-reference scenarios. We introduce a ground-truth-informed dataset construction approach to enhance data quality, and scale up the dataset to 495K under the brief-detail joint framework. Consequently, we construct a comprehensive, large-scale, and high-quality dataset, named DQ-495K. We also retain image resolution during training to better handle resolution-related quality issues, and estimate a confidence score that is helpful to filter out low-quality responses. Experimental results demonstrate that DepictQA-Wild significantly outperforms traditional score-based methods, prior VLM-based IQA models, and proprietary GPT-4V in distortion identification, instant rating, and reasoning tasks. Our advantages are further confirmed by real-world applications including assessing the web-downloaded images and ranking model-processed images. Codes, datasets, and model weights have been released in https://depictqa.github.io/.
comment : Accepted by TIP
♻ ☆ A Sampling-Based Domain Generalization Study with Diffusion Generative Models NeurIPS 2025
Ye Zhu, Yu Wu, Duo Xu, Zhiwei Deng, Yan Yan, Olga Russakovsky
In this work, we investigate the domain generalization capabilities of diffusion models in the context of synthesizing images that are distinct from the training data. Instead of fine-tuning, we tackle this challenge from a sampling-based perspective using frozen, pre-trained diffusion models. Specifically, we demonstrate that arbitrary out-of-domain (OOD) images establish Gaussian priors in the latent spaces of a given model after inversion, and that these priors are separable from those of the original training domain. This OOD latent property allows us to synthesize new images of the target unseen domain by discovering qualified OOD latent encodings in the inverted noisy spaces, without altering the pre-trained models. Our cross-model and cross-domain experiments show that the proposed sampling-based method can expand the latent space and generate unseen images without impairing the generation quality of the original domain. We also showcase a practical application of our approach using astrophysical data, highlighting the potential of this generalization paradigm in data-sparse fields such as scientific exploration.
comment : NeurIPS 2025 Workshop on Frontiers in Probabilistic Inference: Learning meets Sampling. Code can be found at https://github.com/L-YeZhu/DiscoveryDiff
♻ ☆ IntrinsiX: High-Quality PBR Generation using Image Priors
Peter Kocsis, Lukas Höllein, Matthias Nießner
We introduce IntrinsiX, a novel method that generates high-quality intrinsic images from text description. In contrast to existing text-to-image models whose outputs contain baked-in scene lighting, our approach predicts physically-based rendering (PBR) maps. This enables the generated outputs to be used for content creation scenarios in core graphics applications that facilitate re-lighting, editing, and texture generation tasks. In order to train our generator, we exploit strong image priors, and pre-train separate models for each PBR material component (albedo, roughness, metallic, normals). We then align these models with a new cross-intrinsic attention formulation that concatenates key and value features in a consistent fashion. This allows us to exchange information between each output modality and to obtain semantically coherent PBR predictions. To ground each intrinsic component, we propose a rendering loss which provides image-space signals to constrain the model, thus facilitating sharp details also in the output BRDF properties. Our results demonstrate detailed intrinsic generation with strong generalization capabilities that outperforms existing intrinsic image decomposition methods used with generated images by a significant margin. Finally, we show a series of applications, including re-lighting, editing, and text-conditioned room-scale PBR texture generation.
comment : Project page: https://peter-kocsis.github.io/IntrinsiX/ Video: https://youtu.be/b0wVA44R93Y
♻ ☆ Sparse Autoencoders Learn Monosemantic Features in Vision -Language Models NeurIPS 2025
Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, Zeynep Akata
Sparse Autoencoders (SAEs) have recently gained attention as a means to improve the interpretability and steerability of Large Language Models (LLMs), both of which are essential for AI safety. In this work, we extend the application of SAEs to Vision-Language Models (VLMs), such as CLIP, and introduce a comprehensive framework for evaluating monosemanticity at the neuron-level in visual representations. To ensure that our evaluation aligns with human perception, we propose a benchmark derived from a large-scale user study. Our experimental results reveal that SAEs trained on VLMs significantly enhance the monosemanticity of individual neurons, with sparsity and wide latents being the most influential factors. Further, we demonstrate that applying SAE interventions on CLIP's vision encoder directly steers multimodal LLM outputs (e.g., LLaVA), without any modifications to the underlying language model. These findings emphasize the practicality and efficacy of SAEs as an unsupervised tool for enhancing both interpretability and control of VLMs. Code and benchmark data are available at https://github.com/ExplainableML/sae-for-vlm.
comment : Accepted at NeurIPS 2025
♻ ☆ Geometric Regularity in Deterministic Sampling of Diffusion -based Generative Models ICML 2024
Defang Chen, Zhenyu Zhou, Can Wang, Siwei Lyu
Diffusion-based generative models employ stochastic differential equations (SDEs) and their equivalent probability flow ordinary differential equations (ODEs) to establish a smooth transformation between complex high-dimensional data distributions and tractable prior distributions. In this paper, we reveal a striking geometric regularity in the deterministic sampling dynamics of diffusion generative models: each simulated sampling trajectory along the gradient field lies within an extremely low-dimensional subspace, and all trajectories exhibit an almost identical boomerang shape, regardless of the model architecture, applied conditions, or generated content. We characterize several intriguing properties of these trajectories, particularly under closed-form solutions based on kernel-estimated data modeling. We also demonstrate a practical application of the discovered trajectory regularity by proposing a dynamic programming-based scheme to better align the sampling time schedule with the underlying trajectory structure. This simple strategy requires minimal modification to existing deterministic numerical solvers, incurs negligible computational overhead, and achieves superior image generation performance, especially in regions with only 5 - 10 function evaluations.
comment : 57 pages. Accepted by Journal of Statistical Mechanics: Theory and Experiment (2025). The short version was published in ICML 2024. arXiv admin note: text overlap with arXiv:2405.11326
♻ ☆ ForAug: Recombining Foregrounds and Backgrounds to Improve Vision Transformer Training with Bias Mitigation
Tobias Christian Nauen, Brian Moser, Federico Raue, Stanislav Frolov, Andreas Dengel
Transformers, particularly Vision Transformers (ViTs), have achieved state-of-the-art performance in large-scale image classification. However, they often require large amounts of data and can exhibit biases, such as center or size bias, that limit their robustness and generalizability. This paper introduces ForAug, a novel data augmentation operation that addresses these challenges by explicitly imposing invariances into the training data, which are otherwise part of the neural network architecture. ForAug is constructed by using pretrained foundation models to separate and recombine foreground objects with different backgrounds. This recombination step enables us to take fine-grained control over object position and size, as well as background selection. We demonstrate that using ForAug significantly improves the accuracy of ViTs and other architectures by up to 4.5 percentage points (p.p.) on ImageNet, which translates to 7.3 p.p. on downstream tasks. Importantly, ForAug not only improves accuracy but also opens new ways to analyze model behavior and quantify biases. Namely, we introduce metrics for background robustness, foreground focus, center bias, and size bias and show that using ForAug during training substantially reduces these biases. In summary, ForAug provides a valuable tool for analyzing and mitigating biases, enabling the development of more robust and reliable computer vision models. Our code and dataset are publicly available at https://github.com/tobna/ForAug.
comment : v2: added DeiT, added ablation vs simple copy-paste
♻ ☆ Event Stream-based Sign Language Translation: A High-Definition Benchmark Dataset and A Novel Baseline
Shiao Wang, Xiao Wang, Duoqing Yang, Yao Rong, Fuling Wang, Jianing Li, Lin Zhu, Bo Jiang
Sign Language Translation (SLT) is a core task in the field of AI-assisted disability. Traditional SLT methods are typically based on visible light videos, which are easily affected by factors such as lighting variations, rapid hand movements, and privacy concerns. This paper proposes the use of bio-inspired event cameras to alleviate the aforementioned issues. Specifically, we introduce a new high-definition event-based sign language dataset, termed Event-CSL, which effectively addresses the data scarcity in this research area. The dataset comprises 14,827 videos, 14,821 glosses, and 2,544 Chinese words in the text vocabulary. These samples are collected across diverse indoor and outdoor scenes, covering multiple viewpoints, lighting conditions, and camera motions. We have also benchmarked existing mainstream SLT methods on this dataset to facilitate fair comparisons in future research.Furthermore, we propose a novel event-based sign language translation framework, termed EvSLT. The framework first segments continuous video features into clips and employs a Mamba-based memory aggregation module to compress and aggregate spatial detail features at the clip level. Subsequently, these spatial features, along with temporal representations obtained from temporal convolution, are then fused by a graph-guided spatiotemporal fusion module. Extensive experiments on Event-CSL, as well as other publicly available datasets, demonstrate the superior performance of our method. The dataset and source code will be released on https://github.com/Event-AHU/OpenESL
♻ ☆ VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning
Xuanle Zhao, Deyang Jiang, Zhixiong Zeng, Lei Chen, Haibo Qiu, Jing Huang, Yufeng Zhong, Liming Zheng, Yilin Cao, Lin Ma
Multimodal code generation has garnered significant interest within the research community. Despite the notable success of recent vision-language models (VLMs) on specialized tasks like chart-to-code generation, their reliance on single-task training regimens fosters a narrow paradigm that hinders the development of generalized \textbf{VI}sio\textbf{N} \textbf{C}ode \textbf{I}ntelligence. In this work, we introduce \textbf{VinciCoder}, a unified multimodal code generation model that addresses this limitation via a two-stage training framework. We begin by constructing a large-scale Supervised Finetuning (SFT) corpus comprising 1.6M image-code pairs for tasks involving direct code generation and visual-based code refinement. Subsequently, we introduce a Visual Reinforcement Learning (ViRL) strategy, which employs a coarse-to-fine reward mechanism to improve visual fidelity by calculating visual similarity across local and global image patches. Extensive experiments on diverse multimodal code generation benchmarks demonstrate that VinciCoder achieves state-of-the-art performance, surpassing recent open-source models. The ablation study further validates the effectiveness of our proposed coarse-to-fine ViRL strategy. The data, code and model is available at https://github.com/DocTron-hub/VinciCoder.
comment : 15 pages, 11 figures
♻ ☆ Accelerating Parallel Diffusion Model Serving with Residual Compression NeurIPS 2025
Jiajun Luo, Yicheng Xiao, Jianru Xu, Yangxiu You, Rongwei Lu, Chen Tang, Jingyan Jiang, Zhi Wang
Diffusion models produce realistic images and videos but require substantial computational resources, necessitating multi-accelerator parallelism for real-time deployment. However, parallel inference introduces significant communication overhead from exchanging large activations between devices, limiting efficiency and scalability. We present CompactFusion, a compression framework that significantly reduces communication while preserving generation quality. Our key observation is that diffusion activations exhibit strong temporal redundancy-adjacent steps produce highly similar activations, saturating bandwidth with near-duplicate data carrying little new information. To address this inefficiency, we seek a more compact representation that encodes only the essential information. CompactFusion achieves this via Residual Compression that transmits only compressed residuals (step-wise activation differences). Based on empirical analysis and theoretical justification, we show that it effectively removes redundant data, enabling substantial data reduction while maintaining high fidelity. We also integrate lightweight error feedback to prevent error accumulation. CompactFusion establishes a new paradigm for parallel diffusion inference, delivering lower latency and significantly higher generation quality than prior methods. On 4xL20, it achieves 3.0x speedup while greatly improving fidelity. It also uniquely supports communication-heavy strategies like sequence parallelism on slow networks, achieving 6.7x speedup over prior overlap-based method. CompactFusion applies broadly across diffusion models and parallel settings, and integrates easily without requiring pipeline rework. Portable implementation demonstrated on xDiT is publicly available at https://github.com/Cobalt-27/CompactFusion
comment : Accepted by NeurIPS 2025
♻ ☆ LD-ViCE: Latent Diffusion Model for Video Counterfactual Explanations CVPR 2026
Payal Varshney, Adriano Lucieri, Christoph Balada, Sheraz Ahmed, Andreas Dengel
Video-based AI systems are increasingly adopted in safety-critical domains such as autonomous driving and healthcare. However, interpreting their decisions remains challenging due to the inherent spatiotemporal complexity of video data and the opacity of deep learning models. Existing explanation techniques often suffer from limited temporal coherence and a lack of actionable causal insights. Current counterfactual explanation methods typically do not incorporate guidance from the target model, reducing semantic fidelity and practical utility. We introduce Latent Diffusion for Video Counterfactual Explanations (LD-ViCE), a novel framework designed to explain the behavior of video-based AI models. Compared to previous approaches, LD-ViCE reduces the computational costs of generating explanations by operating in latent space using a state-of-the-art diffusion model, while producing realistic and interpretable counterfactuals through an additional refinement step. Experiments on three diverse video datasets - EchoNet-Dynamic (cardiac ultrasound), FERV39k (facial expression), and Something-Something V2 (action recognition) with multiple target models covering both classification and regression tasks, demonstrate that LD-ViCE generalizes well and achieves state-of-the-art performance. On the EchoNet-Dynamic dataset, LD-ViCE achieves significantly higher regression accuracy than prior methods and exhibits high temporal consistency, while the refinement stage further improves perceptual quality. Qualitative analyses confirm that LD-ViCE produces semantically meaningful and temporally coherent explanations, providing actionable insights into model behavior. LD-ViCE advances the trustworthiness and interpretability of video-based AI systems through visually coherent counterfactual explanations.
comment : Under Review CVPR 2026 (44 Pages)
♻ ☆ Let it Snow! Animating 3D Gaussian Scenes with Dynamic Weather Effects via Physics-Guided Score Distillation
Gal Fiebelman, Hadar Averbuch-Elor, Sagie Benaim
3D Gaussian Splatting has recently enabled fast and photorealistic reconstruction of static 3D scenes. However, dynamic editing of such scenes remains a significant challenge. We introduce a novel framework, Physics-Guided Score Distillation, to address a fundamental conflict: physics simulation provides a strong motion prior that is insufficient for photorealism , while video-based Score Distillation Sampling (SDS) alone cannot generate coherent motion for complex, multi-particle scenarios. We resolve this through a unified optimization framework where physics simulation guides Score Distillation to jointly refine the motion prior for photorealism while simultaneously optimizing appearance. Specifically, we learn a neural dynamics model that predicts particle motion and appearance, optimized end-to-end via a combined loss integrating Video-SDS for photorealism with our physics-guidance prior. This allows for photorealistic refinements while ensuring the dynamics remain plausible. Our framework enables scene-wide dynamic weather effects, including snowfall, rainfall, fog, and sandstorms, with physically plausible motion. Experiments demonstrate our physics-guided approach significantly outperforms baselines, with ablations confirming this joint refinement is essential for generating coherent, high-fidelity dynamics.
comment : Project webpage: https://galfiebelman.github.io/let-it-snow/
♻ ☆ Visual-Word Tokenizer: Beyond Fixed Sets of Tokens in Vision Transformer s
Leonidas Gee, Wing Yan Li, Viktoriia Sharmanska, Novi Quadrianto
The cost of deploying vision transformers increasingly represents a barrier to wider industrial adoption. Existing compression techniques require additional end-to-end fine-tuning or incur a significant drawback to energy efficiency, making them ill-suited for online (real-time) inference, where a prediction is made on any new input as it comes in. We introduce the $\textbf{Visual-Word Tokenizer}$ (VWT), a training-free method for reducing energy costs while retaining performance. The VWT groups visual subwords (image patches) that are frequently used into visual words, while infrequent ones remain intact. To do so, $\textit{intra}$-image or $\textit{inter}$-image statistics are leveraged to identify similar visual concepts for sequence compression. Experimentally, we demonstrate a reduction in energy consumed of up to 47%. Comparative approaches of 8-bit quantization and token merging can lead to significantly increased energy costs (up to 500% or more). Our results indicate that VWTs are well-suited for efficient online inference with a marginal compromise on performance. The experimental code for our paper is also made publicly available.
♻ ☆ Discovering Concept Directions from Diffusion -based Counterfactuals via Latent Clustering
Payal Varshney, Adriano Lucieri, Christoph Balada, Andreas Dengel, Sheraz Ahmed
Concept-based explanations have emerged as an effective approach within Explainable Artificial Intelligence, enabling interpretable insights by aligning model decisions with human-understandable concepts. However, existing methods rely on computationally intensive procedures and struggle to efficiently capture complex, semantic concepts. This work introduces the Concept Directions via Latent Clustering (CDLC), which extracts global, class-specific concept directions by clustering latent difference vectors derived from factual and diffusion-generated counterfactual image pairs. CDLC reduces storage requirements by ~4.6% and accelerates concept discovery by ~5.3% compared to the baseline method, while requiring no GPU for clustering, thereby enabling efficient extraction of multidimensional semantic concepts across latent dimensions. This approach is validated on a real-world skin lesion dataset, demonstrating that the extracted concept directions align with clinically recognized dermoscopic features and, in some cases, reveal dataset-specific biases or unknown biomarkers. These results highlight that CDLC is interpretable, scalable, and applicable across high-stakes domains and diverse data modalities.
comment : Accepted at Pattern Recognition Letters Journal (14 Pages)
♻ ☆ ParticleGS: Learning Neural Gaussian Particle Dynamics from Videos for Prior-free Physical Motion Extrapolation
Jinsheng Quan, Qiaowei Miao, Yichao Xu, Zizhuo Lin, Ying Li, Wei Yang, Zhihui Li, Yawei Luo
The ability to extrapolate dynamic 3D scenes beyond the observed timeframe is fundamental to advancing physical world understanding and predictive modeling. Existing dynamic 3D reconstruction methods have achieved high-fidelity rendering of temporal interpolation, but typically lack physical consistency in predicting the future. To overcome this issue, we propose ParticleGS, a physics-based framework that reformulates dynamic 3D scenes as physically grounded systems. ParticleGS comprises three key components: 1) an encoder that decomposes the scene into static properties and initial dynamic physical fields; 2) an evolver based on Neural Ordinary Differential Equations (Neural ODEs) that learns continuous-time dynamics for motion extrapolation; and 3) a decoder that reconstructs 3D Gaussians from evolved particle states for rendering. Through this design, ParticleGS integrates physical reasoning into dynamic 3D representations, enabling accurate and consistent prediction of the future. Experiments show that ParticleGS achieves state-of-the-art performance in extrapolation while maintaining rendering quality comparable to leading dynamic 3D reconstruction methods.
♻ ☆ SAMChat: Introducing Chain of Thought Reasoning and GRPO to a Multimodal Small Language Model for Small Scale Remote Sensing
Aybora Koksal, A. Aydin Alatan
Remarkable capabilities in understanding and generating text-image content have been demonstrated by recent advancements in multimodal large language models (MLLMs). However, their effectiveness in specialized domains-particularly those requiring resource-efficient and domain-specific adaptations-has remained limited. In this work, a lightweight multimodal language model termed SAMChat is introduced, specifically adapted to analyze remote sensing imagery in secluded areas, including challenging missile launch sites. A new dataset, SAMData, was compiled by verifying hundreds of aerial images through expert review, and subtle military installations were highlighted via detailed captions. Supervised fine-tuning on a 2B parameter open-source MLLM with chain-of-thought (CoT) reasoning annotations was performed, enabling more accurate and interpretable explanations. Additionally, Group Relative Policy Optimization (GRPO) was leveraged to enhance the model's ability to detect critical domain-specific cues-such as defensive layouts and key military structures-while minimizing false positives on civilian scenes. Through empirical evaluations, it has been shown that SAMChat significantly outperforms both larger, general-purpose multimodal models and existing remote sensing adapted approaches on open-ended captioning and classification metrics. Over 80% recall and 98% precision were achieved on the newly proposed SAMData benchmark, underscoring the potency of targeted fine-tuning and reinforcement learning in specialized real-world applications.
comment : Accepted to Journal of Selected Topics in Applied Earth Observations and Remote Sensing (JSTARS) Special Issue on Foundation and Large Vision Models for Remote Sensing. Code and dataset are available at https://github.com/aybora/SAMChat
♻ ☆ CLIP-like Model as a Foundational Density Ratio Estimator
Fumiya Uchiyama, Rintaro Yanagi, Shohei Taniguchi, Shota Takashiro, Masahiro Suzuki, Hirokatsu Kataoka, Yusuke Iwasawa, Yutaka Matsuo
Density ratio estimation is a core concept in statistical machine learning because it provides a unified mechanism for tasks such as importance weighting, divergence estimation, and likelihood-free inference, but its potential in vision and language models has not been fully explored. Modern vision-language encoders such as CLIP and SigLIP are trained with contrastive objectives that implicitly optimize log density ratios between joint and marginal image-text distributions, which implicitly learn similarity scores proportional to log density ratios. However, prior work has largely focused on their embedding utility, and the density-ratio structure induced by contrastive learning has not been systematically examined or exploited in multimodal applications. To address this gap, we reinterpret CLIP-style models as pretrained and general-purpose density ratio estimators and show that this perspective enables new algorithmic capabilities. We present a unified explanation of how contrastive objectives estimate density ratios and propose two practical applications: Importance Weight Learning and KL divergence estimation. Our Importance Weight Learning method requires only a single additional prompt and improves F1 scores by up to 7 points. We further show that CLIP-based density ratios support estimation of KL divergences that quantify how conditioning on an image or text alters the distribution of the other modality. Through qualitative examples and an N-gram analysis of captions, we find that these divergences capture semantic diversity and mode structure in multimodal data. Leveraging this property, we introduce a simple KL-guided data curation method that achieves performance competitive with LAION2B filtering.
♻ ☆ Dream4D: Lifting Camera-Controlled I2V towards Spatiotemporally Consistent 4D Generation
Xiaoyan Liu, Kangrui Li, Yuehao Song, Jiaxin Liu
The synthesis of spatiotemporally coherent 4D content presents fundamental challenges in computer vision, requiring simultaneous modeling of high-fidelity spatial representations and physically plausible temporal dynamics. Current approaches often struggle to maintain view consistency while handling complex scene dynamics, particularly in large-scale environments with multiple interacting elements. This work introduces Dream4D, a novel framework that bridges this gap through a synergy of controllable video generation and neural 4D reconstruction. Our approach seamlessly combines a two-stage architecture: it first predicts optimal camera trajectories from a single image using few-shot learning, then generates geometrically consistent multi-view sequences via a specialized pose-conditioned diffusion process, which are finally converted into a persistent 4D representation. This framework is the first to leverage both rich temporal priors from video diffusion models and geometric awareness of the reconstruction models, which significantly facilitates 4D generation and shows higher quality (e.g., mPSNR, mSSIM) over existing methods.
comment : Project Page: https://wanderer7-sk.github.io/Dream4D.github.io/
♻ ☆ Automated segmentation of pediatric neuroblastoma on multi-modal MRI: Results of the SPPIN challenge at MICCAI 2023
M. A. D. Buser, D. C. Simons, M. Fitski, M. H. W. A. Wijnen, A. S. Littooij, A. H. ter Brugge, I. N. Vos, M. H. A. Janse, M. de Boer, R. ter Maat, J. Sato, S. Kido, S. Kondo, S. Kasai, M. Wodzinski, H. Muller, J. Ye, J. He, Y. Kirchhoff, M. R. Rokkus, G. Haokai, S. Zitong, M. Fernández Patón, D. Veiga-Canuto, D. G. Ellis, M. R. Aizenberg, B. H. M. van der Velden, H. Kuijf, A. De Luca, A. F. W. van der Steeg
Surgery plays an important role within the treatment for neuroblastoma, a common pediatric cancer. This requires careful planning, often via magnetic resonance imaging (MRI)-based anatomical 3D models. However, creating these models is often time-consuming and user dependent. We organized the Surgical Planning in Pediatric Neuroblastoma (SPPIN) challenge, to stimulate developments on this topic, and set a benchmark for fully automatic segmentation of neuroblastoma on multi-model MRI. The challenge started with a training phase, where teams received 78 sets of MRI scans from 34 patients, consisting of both diagnostic and post-chemotherapy MRI scans. The final test phase, consisting of 18 MRI sets from 9 patients, determined the ranking of the teams. Ranking was based on the Dice similarity coefficient (Dice score), the 95th percentile of the Hausdorff distance (HD95) and the volumetric similarity (VS). The SPPIN challenge was hosted at MICCAI 2023. The final leaderboard consisted of 9 teams. The highest-ranking team achieved a median Dice score 0.82, a median HD95 of 7.69 mm and a VS of 0.91, utilizing a large, pretrained network called STU-Net. A significant difference for the segmentation results between diagnostic and post-chemotherapy MRI scans was observed (Dice = 0.89 vs Dice = 0.59, P = 0.01) for the highest-ranking team. SPPIN is the first medical segmentation challenge in extracranial pediatric oncology. The highest-ranking team used a large pre-trained network, suggesting that pretraining can be of use in small, heterogenous datasets. Although the results of the highest-ranking team were high for most patients, segmentation especially in small, pre-treated tumors were insufficient. Therefore, more reliable segmentation methods are needed to create clinically applicable models to aid surgical planning in pediatric neuroblastoma.
comment : 23 pages, 6 figures
♻ ☆ Qwen 3-VL Technical Report
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, Ke Zhu
We introduce Qwen3-VL, the most capable vision-language model in the Qwen series to date, achieving superior performance across a broad range of multimodal benchmarks. It natively supports interleaved contexts of up to 256K tokens, seamlessly integrating text, images, and video. The model family includes both dense (2B/4B/8B/32B) and mixture-of-experts (30B-A3B/235B-A22B) variants to accommodate diverse latency-quality trade-offs. Qwen3-VL delivers three core pillars: (i) markedly stronger pure-text understanding, surpassing comparable text-only backbones in several cases; (ii) robust long-context comprehension with a native 256K-token window for both text and interleaved multimodal inputs, enabling faithful retention, retrieval, and cross-referencing across long documents and videos; and (iii) advanced multimodal reasoning across single-image, multi-image, and video tasks, demonstrating leading performance on comprehensive evaluations such as MMMU and visual-math benchmarks (e.g., MathVista and MathVision). Architecturally, we introduce three key upgrades: (i) an enhanced interleaved-MRoPE for stronger spatial-temporal modeling across images and video; (ii) DeepStack integration, which effectively leverages multi-level ViT features to tighten vision-language alignment; and (iii) text-based time alignment for video, evolving from T-RoPE to explicit textual timestamp alignment for more precise temporal grounding. Under comparable token budgets and latency constraints, Qwen3-VL achieves superior performance in both dense and Mixture-of-Experts (MoE) architectures. We envision Qwen3-VL serving as a foundational engine for image-grounded reasoning, agentic decision-making, and multimodal code intelligence in real-world workflows.
comment : 42 pages
♻ ☆ Dual-Model Weight Selection and Self-Knowledge Distillation for Medical Image Classification
Ayaka Tsutsumi, Guang Li, Ren Togo, Takahiro Ogawa, Satoshi Kondo, Miki Haseyama
We propose a novel medical image classification method that integrates dual-model weight selection with self-knowledge distillation (SKD). In real-world medical settings, deploying large-scale models is often limited by computational resource constraints, which pose significant challenges for their practical implementation. Thus, developing lightweight models that achieve comparable performance to large-scale models while maintaining computational efficiency is crucial. To address this, we employ a dual-model weight selection strategy that initializes two lightweight models with weights derived from a large pretrained model, enabling effective knowledge transfer. Next, SKD is applied to these selected models, allowing the use of a broad range of initial weight configurations without imposing additional excessive computational cost, followed by fine-tuning for the target classification tasks. By combining dual-model weight selection with self-knowledge distillation, our method overcomes the limitations of conventional approaches, which often fail to retain critical information in compact models. Extensive experiments on publicly available datasets-chest X-ray images, lung computed tomography scans, and brain magnetic resonance imaging scans-demonstrate the superior performance and robustness of our approach compared to existing methods.
comment : Published as a journal paper at Elsevier CIBM
♻ ☆ Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms NeurIPS 2025
Yinuo Ren, Haoxuan Chen, Yuchen Zhu, Wei Guo, Yongxin Chen, Grant M. Rotskoff, Molei Tao, Lexing Ying
Discrete diffusion models have emerged as a powerful generative modeling framework for discrete data with successful applications spanning from text generation to image synthesis. However, their deployment faces challenges due to the high dimensionality of the state space, necessitating the development of efficient inference algorithms. Current inference approaches mainly fall into two categories: exact simulation and approximate methods such as $τ$-leaping. While exact methods suffer from unpredictable inference time and redundant function evaluations, $τ$-leaping is limited by its first-order accuracy. In this work, we advance the latter category by tailoring the first extension of high-order numerical inference schemes to discrete diffusion models, enabling larger step sizes while reducing error. We rigorously analyze the proposed schemes and establish the second-order accuracy of the $θ$-Trapezoidal method in KL divergence. Empirical evaluations on GSM8K-level math-reasoning, GPT-2-level text, and ImageNet-level image generation tasks demonstrate that our method achieves superior sample quality compared to existing approaches under equivalent computational constraints, with consistent performance gains across models ranging from 200M to 8B. Our code is available at https://github.com/yuchen-zhu-zyc/DiscreteFastSolver.
comment : Accepted at NeurIPS 2025 as a Poster (https://openreview.net/forum?id=OuklL6Q3sO)
♻ ☆ FlashEdit: Decoupling Speed, Structure, and Semantics for Precise Image Editing
Junyi Wu, Zhiteng Li, Haotong Qin, Xiaohong Liu, Linghe Kong, Yulun Zhang, Xiaokang Yang
Text-guided image editing with diffusion models has achieved remarkable quality but suffers from prohibitive latency, hindering real-world applications. We introduce FlashEdit, a novel framework designed to enable high-fidelity, real-time image editing. Its efficiency stems from three key innovations: (1) a One-Step Inversion-and-Editing (OSIE) pipeline that bypasses costly iterative processes; (2) a Background Shield (BG-Shield) technique that guarantees background preservation by selectively modifying features only within the edit region; and (3) a Sparsified Spatial Cross-Attention (SSCA) mechanism that ensures precise, localized edits by suppressing semantic leakage to the background. Extensive experiments demonstrate that FlashEdit maintains superior background consistency and structural integrity, while performing edits in under 0.2 seconds, which is an over 150$\times$ speedup compared to prior multi-step methods. Our code will be made publicly available at https://github.com/JunyiWuCode/FlashEdit.
comment : Our code will be made publicly available at https://github.com/JunyiWuCode/FlashEdit
♻ ☆ MonoDream: Monocular Vision -Language Navigation with Panoramic Dreaming
Shuo Wang, Yongcai Wang, Zhaoxin Fan, Yucheng Wang, Maiyue Chen, Kaihui Wang, Zhizhong Su, Wanting Li, Xudong Cai, Yeying Jin, Deying Li
Vision-Language Navigation (VLN) tasks often leverage panoramic RGB and depth inputs to provide rich spatial cues for action planning, but these sensors can be costly or less accessible in real-world deployments. Recent approaches based on Vision-Language Action (VLA) models achieve strong results with monocular input, yet they still lag behind methods using panoramic RGB-D information. We present MonoDream, a lightweight VLA framework that enables monocular agents to learn a Unified Navigation Representation (UNR). This shared feature representation jointly aligns navigation-relevant visual semantics (e.g., global layout, depth, and future cues) and language-grounded action intent, enabling more reliable action prediction. MonoDream further introduces Latent Panoramic Dreaming (LPD) tasks to supervise the UNR, which train the model to predict latent features of panoramic RGB and depth observations at both current and future steps based on only monocular input. Experiments on multiple VLN benchmarks show that MonoDream consistently improves monocular navigation performance and significantly narrows the gap with panoramic-based agents.
♻ ☆ A Neurosymbolic Framework for Interpretable Cognitive Attack Detection in Augmented Reality
Rongqian Chen, Allison Andreyev, Yanming Xiu, Joshua Chilukuri, Shunav Sen, Mahdi Imani, Bin Li, Maria Gorlatova, Gang Tan, Tian Lan
Augmented Reality (AR) enriches human perception by overlaying virtual elements onto the physical world. However, this tight coupling between virtual and real content makes AR vulnerable to cognitive attacks: manipulations that distort users' semantic understanding of the environment. Existing detection methods largely focus on visual inconsistencies at the pixel or image level, offering limited semantic reasoning or interpretability. To address these limitations, we introduce CADAR, a neuro-symbolic framework for cognitive attack detection in AR that integrates neural and symbolic reasoning. CADAR fuses multimodal vision-language representations from pre-trained models into a perception graph that captures objects, relations, and temporal contextual salience. Building on this structure, a particle-filter-based statistical reasoning module infers anomalies in semantic dynamics to reveal cognitive attacks. This combination provides both the adaptability of modern vision-language models and the interpretability of probabilistic symbolic reasoning. Preliminary experiments on an AR cognitive-attack dataset demonstrate consistent advantages over existing approaches, highlighting the potential of neuro-symbolic methods for robust and interpretable AR security.
♻ ☆ Investigating the Relation ship between the Weighted Figure of Merit and Rosin's Measure
Bimal Kumar Ray
Many studies have been conducted to solve the problem of approximating a digital boundary by piece straight-line segments for the further processing required in computer vision applications. The authors of these studies compared their schemes to determine the best one. The initial measure used to assess the goodness of fit of a polygonal approximation was the figure of merit. Later,it was noted that this measure was not an appropriate metric for a valid reason which is why Rosin-through mathematical analysis-introduced a measure called merit. However,this measure involves an optimal scheme of polygonal approximation,so it is time-consuming to compute it to assess the goodness of fit of an approximation. This led many researchers to use a weighted figure of merit as a substitute for Rosin's measure to compare sub optimal schemes. An attempt is made in this communication to investigate whether the two measures-weighted figure of merit and Rosin's measure-are related so that one can be used instead of the other, and toward this end, theoretical analysis, experimental investigation and statistical analysis are carried out. The mathematical formulas for the weighted figure of merit and Rosin's measure are analyzed, and through proof of theorems,it is found that the two measures are theoretically independent of each other. The graphical analysis of experiments carried out using a public dataset supports the results of the theoretical analysis. The statistical analysis via Pearson's correlation coefficient and non-linear correlation measure also revealed that the two measures are uncorrelated. This analysis leads one to conclude that if a suboptimal scheme is found to be better (worse) than some other suboptimal scheme,as indicated by Rosin's measure,then the same conclusion cannot be drawn using a weighted figure of merit,so one cannot use a weighted figure of merit instead of Rosin's measure.
♻ ☆ Loomis Painter: Reconstructing the Painting Process
Markus Pobitzer, Chang Liu, Chenyi Zhuang, Teng Long, Bin Ren, Nicu Sebe
Step-by-step painting tutorials are vital for learning artistic techniques, but existing video resources (e.g., YouTube) lack interactivity and personalization. While recent generative models have advanced artistic image synthesis, they struggle to generalize across media and often show temporal or structural inconsistencies, hindering faithful reproduction of human creative workflows. To address this, we propose a unified framework for multi-media painting process generation with a semantics-driven style control mechanism that embeds multiple media into a diffusion models conditional space and uses cross-medium style augmentation. This enables consistent texture evolution and process transfer across styles. A reverse-painting training strategy further ensures smooth, human-aligned generation. We also build a large-scale dataset of real painting processes and evaluate cross-media consistency, temporal coherence, and final-image fidelity, achieving strong results on LPIPS, DINO, and CLIP metrics. Finally, our Perceptual Distance Profile (PDP) curve quantitatively models the creative sequence, i.e., composition, color blocking, and detail refinement, mirroring human artistic progression.
♻ ☆ From Perception to Reasoning : Deep Thinking Empowers Multimodal Large Language Models
Wenxin Zhu, Andong Chen, Yuchen Song, Kehai Chen, Conghui Zhu, Ziyan Chen, Tiejun Zhao
With the remarkable success of Multimodal Large Language Models (MLLMs) in perception tasks, enhancing their complex reasoning capabilities has emerged as a critical research focus. Existing models still suffer from challenges such as opaque reasoning paths and insufficient generalization ability. Chain-of-Thought (CoT) reasoning, which has demonstrated significant efficacy in language models by enhancing reasoning transparency and output interpretability, holds promise for improving model reasoning capabilities when extended to the multimodal domain. This paper provides a systematic review centered on "Multimodal Chain-of-Thought" (MCoT). First, it analyzes the background and theoretical motivations for its inception from the perspectives of technical evolution and task demands. Then, it introduces mainstream MCoT methods from three aspects: CoT paradigms, the post-training stage, and the inference stage, while also analyzing their underlying mechanisms. Furthermore, the paper summarizes existing evaluation benchmarks and metrics, and discusses the application scenarios of MCoT. Finally, it analyzes the challenges currently facing MCoT and provides an outlook on its future research directions.
comment : Survey; 7 figures, 3 tables, 44 pages
♻ ☆ Boosting Reasoning in Large Multimodal Models via Activation Replay
Yun Xing, Xiaobin Hu, Qingdong He, Jiangning Zhang, Shuicheng Yan, Shijian Lu, Yu-Gang Jiang
Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach to incentivizing reasoning capability in Large Multimodal Models (LMMs), while the underlying mechanisms behind this post-training paradigm are poorly understood. We begin by exploring how input activations are affected by RLVR through the perspective of logit lens. Our systematic investigations across multiple post-trained LMMs suggest that RLVR shifts low-entropy activations unexpectedly, while high-entropy ones are less affected. We further demonstrate that such phenomena are associated with LMM reasoning by controlled experiments, suggesting a potentially beneficial role of modulating low-entropy activations. To this end, we propose Activation Replay, a novel simple yet effective training-free approach that boosts multimodal reasoning of post-trained LMMs without requiring expensive policy optimization. Our design involves manipulation of visual tokens at test time, replaying low-entropy activations from the input context of base LMMs to regulating the RLVR counterparts. Activation Replay triggers better reasoning across diverse scenarios, including mathematics, o3-like visual agents, and video reasoning. We further show that Activation Replay boosts Pass@K and mitigates narrower reasoning coverage of RLVR. Our design is compared against alternative choices, such as replaying high-entropy activations instead of low-entropy ones, or direct cross-model intervention instead of manipulating input tokens, demonstrating the superiority of our implementation. Codes will be made publicly available.
comment : 11 figures, 10 tables
♻ ☆ Watch and Learn: Learning to Use Computers from Online Videos
Chan Hee Song, Yiwen Song, Palash Goyal, Yu Su, Oriana Riva, Hamid Palangi, Tomas Pfister
Computer-using agents (CUAs) must plan task workflows across diverse and evolving applications, yet progress is limited by the lack of large-scale, high-quality training data. Existing datasets are narrow, static, and costly to annotate, while synthetic data often yields oversimplified or misaligned behaviors. We present Watch & Learn (W&L), a framework that converts readily available Internet videos of human computer use into executable UI trajectories at scale. Instead of directly generating actions or relying on handcrafted heuristics, we cast trajectory annotation as an inverse dynamics problem that predicts user actions from consecutive screen states, which simplifies learning and generalizes across domains. Through a task-aware retrieval and labeling pipeline, W&L yields over 53K high-quality trajectories that enhance CUAs both as in-context exemplars and as supervised training data. On OSWorld, it consistently improves general-purpose and specialized CUAs, while on WindowsAgentArena it achieves state-of-the-art performance among 7B-scale models under the 15-step limit. These results show that web-scale human demonstration videos can serve as a practical and scalable foundation for advancing real-world CUAs.
♻ ☆ DiP: Taming Diffusion Models in Pixel Space
Zhennan Chen, Junwei Zhu, Xu Chen, Jiangning Zhang, Xiaobin Hu, Hanzhen Zhao, Chengjie Wang, Jian Yang, Ying Tai
Diffusion models face a fundamental trade-off between generation quality and computational efficiency. Latent Diffusion Models (LDMs) offer an efficient solution but suffer from potential information loss and non-end-to-end training. In contrast, existing pixel space models bypass VAEs but are computationally prohibitive for high-resolution synthesis. To resolve this dilemma, we propose DiP, an efficient pixel space diffusion framework. DiP decouples generation into a global and a local stage: a Diffusion Transformer (DiT) backbone operates on large patches for efficient global structure construction, while a co-trained lightweight Patch Detailer Head leverages contextual features to restore fine-grained local details. This synergistic design achieves computational efficiency comparable to LDMs without relying on a VAE. DiP is accomplished with up to 10$\times$ faster inference speeds than previous method while increasing the total number of parameters by only 0.3%, and achieves an 1.79 FID score on ImageNet 256$\times$256.
♻ ☆ ControlEvents: Controllable Synthesis of Event Camera Datawith Foundational Prior from Image Diffusion Models WACV2026
Yixuan Hu, Yuxuan Xue, Simon Klenk, Daniel Cremers, Gerard Pons-Moll
In recent years, event cameras have gained significant attention due to their bio-inspired properties, such as high temporal resolution and high dynamic range. However, obtaining large-scale labeled ground-truth data for event-based vision tasks remains challenging and costly. In this paper, we present ControlEvents, a diffusion-based generative model designed to synthesize high-quality event data guided by diverse control signals such as class text labels, 2D skeletons, and 3D body poses. Our key insight is to leverage the diffusion prior from foundation models, such as Stable Diffusion, enabling high-quality event data generation with minimal fine-tuning and limited labeled data. Our method streamlines the data generation process and significantly reduces the cost of producing labeled event datasets. We demonstrate the effectiveness of our approach by synthesizing event data for visual recognition, 2D skeleton estimation, and 3D body pose estimation. Our experiments show that the synthesized labeled event data enhances model performance in all tasks. Additionally, our approach can generate events based on unseen text labels during training, illustrating the powerful text-based generation capabilities inherited from foundation models.
comment : Accepted to WACV2026. Project website:https://yuxuan-xue.com/controlevents/
♻ ☆ I-INR: Iterative Implicit Neural Representations AAAI 2026
Ali Haider, Muhammad Salman Ali, Maryam Qamar, Tahir Khalil, Soo Ye Kim, Jihyong Oh, Enzo Tartaglione, Sung-Ho Bae
Implicit Neural Representations (INRs) have revolutionized signal processing and computer vision by modeling signals as continuous, differentiable functions parameterized by neural networks. However, their inherent formulation as a regression problem makes them prone to regression to the mean, limiting their ability to capture fine details, retain high-frequency information, and handle noise effectively. To address these challenges, we propose Iterative Implicit Neural Representations (I-INRs) a novel plug-and-play framework that enhances signal reconstruction through an iterative refinement process. I-INRs effectively recover high-frequency details, improve robustness to noise, and achieve superior reconstruction quality. Our framework seamlessly integrates with existing INR architectures, delivering substantial performance gains across various tasks. Extensive experiments show that I-INRs outperform baseline methods, including WIRE, SIREN, and Gauss, in diverse computer vision applications such as image restoration, image denoising, and object occupancy prediction.
comment : Accepted at AAAI 2026
♻ ☆ Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution CVPR 2025
Zhiyuan You, Xin Cai, Jinjin Gu, Tianfan Xue, Chao Dong
With the rapid advancement of Multi-modal Large Language Models (MLLMs), MLLM-based Image Quality Assessment (IQA) methods have shown promising performance in linguistic quality description. However, current methods still fall short in accurately scoring image quality. In this work, we aim to leverage MLLMs to regress accurate quality scores. A key challenge is that the quality score is inherently continuous, typically modeled as a Gaussian distribution, whereas MLLMs generate discrete token outputs. This mismatch necessitates score discretization. Previous approaches discretize the mean score into a one-hot label, resulting in information loss and failing to capture inter-image relationships. We propose a distribution-based approach that discretizes the score distribution into a soft label. This method preserves the characteristics of the score distribution, achieving high accuracy and maintaining inter-image relationships. Moreover, to address dataset variation, where different IQA datasets exhibit various distributions, we introduce a fidelity loss based on Thurstone's model. This loss captures intra-dataset relationships, facilitating co-training across multiple IQA datasets. With these designs, we develop the distribution-based Depicted image Quality Assessment model for Score regression (DeQA-Score). Experiments across multiple benchmarks show that DeQA-Score stably outperforms baselines in score regression. Also, DeQA-Score can predict the score distribution that closely aligns with human annotations. Codes and model weights have been released in https://depictqa.github.io/deqa-score/.
comment : Accepted by CVPR 2025
♻ ☆ CountSteer: Steering Attention for Object Counting in Diffusion Models AAAI 2026
Hyemin Boo, Hyoryung Kim, Myungjin Lee, Seunghyeon Lee, Jiyoung Lee, Jang-Hwan Choi, Hyunsoo Cho
Text-to-image diffusion models generate realistic and coherent images but often fail to follow numerical instructions in text, revealing a gap between language and visual representation. Interestingly, we found that these models are not entirely blind to numbers-they are implicitly aware of their own counting accuracy, as their internal signals shift in consistent ways depending on whether the output meets the specified count. This observation suggests that the model already encodes a latent notion of numerical correctness, which can be harnessed to guide generation more precisely. Building on this intuition, we introduce CountSteer, a training-free method that improves generation of specified object counts by steering the model's cross-attention hidden states during inference. In our experiments, CountSteer improved object-count accuracy by about 4% without compromising visual quality, demonstrating a simple yet effective step toward more controllable and semantically reliable text-to-image generation.
comment : Accepted to AAAI 2026 Workshop on Shaping Responsible Synthetic Data in the Era of Foundation Models (RSD)
♻ ☆ Exploring Convolutional Neural Networks for Rice Grain Classification: An Explainable AI Approach
Muhammad Junaid Asif, Hamza Khan, Rabia Tehseen, Rana Fayyaz Ahmad, Mujtaba Asad, Syed Tahir Hussain Rizvi, Shazia Saqib
Rice is an essential staple food worldwide that is important in promoting international trade, economic growth, and nutrition. Asian countries such as China, India, Pakistan, Thailand, Vietnam, and Indonesia are notable for their significant contribution to the cultivation and utilization of rice. These nations are also known for cultivating different rice grains, including short and long grains. These sizes are further classified as basmati, jasmine, kainat saila, ipsala, arborio, etc., catering to diverse culinary preferences and cultural traditions. For both local and international trade, inspecting and maintaining the quality of rice grains to satisfy customers and preserve a country's reputation is necessary. Manual quality check and classification is quite a laborious and time-consuming process. It is also highly prone to mistakes. Therefore, an automatic solution must be proposed for the effective and efficient classification of different varieties of rice grains. This research paper presents an automatic framework based on a convolutional neural network (CNN) for classifying different varieties of rice grains. We evaluated the proposed model based on performance metrics such as accuracy, recall, precision, and F1-Score. The CNN model underwent rigorous training and validation, achieving a remarkable accuracy rate and a perfect area under each class's Receiver Operating Characteristic (ROC) curve. The confusion matrix analysis confirmed the model's effectiveness in distinguishing between the different rice varieties, indicating minimal misclassifications. Additionally, the integration of explainability techniques such as LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations) provided valuable insights into the model's decision-making process, revealing how specific features of the rice grains influenced classification outcomes.
♻ ☆ Scaling Spatial Intelligence with Multimodal Foundation Models
Zhongang Cai, Ruisi Wang, Chenyang Gu, Fanyi Pu, Junxiang Xu, Yubo Wang, Wanqi Yin, Zhitao Yang, Chen Wei, Qingping Sun, Tongxi Zhou, Jiaqi Li, Hui En Pang, Oscar Qian, Yukun Wei, Zhiqian Lin, Xuanke Shi, Kewang Deng, Xiaoyang Han, Zukai Chen, Xiangyu Fan, Hanming Deng, Lewei Lu, Liang Pan, Bo Li, Ziwei Liu, Quan Wang, Dahua Lin, Lei Yang
Despite remarkable progress, multimodal foundation models still exhibit surprising deficiencies in spatial intelligence. In this work, we explore scaling up multimodal foundation models to cultivate spatial intelligence within the SenseNova-SI family, built upon established multimodal foundations including visual understanding models (i.e., Qwen3-VL and InternVL3) and unified understanding and generation models (i.e., Bagel). We take a principled approach to constructing high-performing and robust spatial intelligence by systematically curating SenseNova-SI-8M: eight million diverse data samples under a rigorous taxonomy of spatial capabilities. SenseNova-SI demonstrates unprecedented performance across a broad range of spatial intelligence benchmarks: 68.7% on VSI-Bench, 43.3% on MMSI, 85.6% on MindCube, 54.6% on ViewSpatial, and 50.1% on SITE, while maintaining strong general multimodal understanding (e.g., 84.9% on MMBench-En). More importantly, we analyze the impact of data scaling, discuss early signs of emergent generalization capabilities enabled by diverse data training, analyze the risk of overfitting and language shortcuts, present a preliminary study on spatial chain-of-thought reasoning, and validate the potential downstream application. SenseNova-SI is an ongoing project, and this report will be updated continuously. All newly trained multimodal foundation models are publicly released to facilitate further research in this direction.
comment : Codebase: https://github.com/OpenSenseNova/SenseNova-SI; Models: https://huggingface.co/collections/sensenova/sensenova-si
♻ ☆ PIS3R: Very Large Parallax Image Stitching via Deep 3D Reconstruction
Muhua Zhu, Xinhao Jin, Chengbo Wang, Yongcong Zhang, Yifei Xue, Tie Ji, Yizhen Lao
Image stitching aim to align two images taken from different viewpoints into one seamless, wider image. However, when the 3D scene contains depth variations and the camera baseline is significant, noticeable parallax occurs-meaning the relative positions of scene elements differ substantially between views. Most existing stitching methods struggle to handle such images with large parallax effectively. To address this challenge, in this paper, we propose an image stitching solution called PIS3R that is robust to very large parallax based on the novel concept of deep 3D reconstruction. First, we apply visual geometry grounded transformer to two input images with very large parallax to obtain both intrinsic and extrinsic parameters, as well as the dense 3D scene reconstruction. Subsequently, we reproject reconstructed dense point cloud onto a designated reference view using the recovered camera parameters, achieving pixel-wise alignment and generating an initial stitched image. Finally, to further address potential artifacts such as holes or noise in the initial stitching, we propose a point-conditioned image diffusion module to obtain the refined result.Compared with existing methods, our solution is very large parallax tolerant and also provides results that fully preserve the geometric integrity of all pixels in the 3D photogrammetric context, enabling direct applicability to downstream 3D vision tasks such as SfM. Experimental results demonstrate that the proposed algorithm provides accurate stitching results for images with very large parallax, and outperforms the existing methods qualitatively and quantitatively.
♻ ☆ Holistic Evaluation of Multimodal LLMs on Spatial Intelligence
Zhongang Cai, Yubo Wang, Qingping Sun, Ruisi Wang, Chenyang Gu, Wanqi Yin, Zhiqian Lin, Zhitao Yang, Chen Wei, Oscar Qian, Hui En Pang, Xuanke Shi, Kewang Deng, Xiaoyang Han, Zukai Chen, Jiaqi Li, Xiangyu Fan, Hanming Deng, Lewei Lu, Bo Li, Ziwei Liu, Quan Wang, Dahua Lin, Lei Yang
Multimodal models have achieved remarkable progress in recent years. Nevertheless, they continue to exhibit notable limitations in spatial understanding and reasoning, the very capability that anchors artificial general intelligence in the physical world. With the recent release of GPT-5, allegedly the most powerful AI model to date, it is timely to examine where the leading models (GPT, Gemini, Grok, Seed, Qwen, and Intern) stand on the path toward spatial intelligence (SI). We thus propose EASI for holistic Evaluation of multimodAl LLMs on Spatial Intelligence. EASI conceptualizes a comprehensive taxonomy of spatial tasks that unifies existing benchmarks and a growing collection of newly curated ones, enabling systematic evaluation of state-of-the-art models. In this report, we conduct the study across eight key benchmarks, at a cost exceeding ten billion total tokens. Our empirical study then reveals that (1) GPT-5 demonstrates unprecedented strength in SI, yet (2) still falls short of human performance significantly across a broad spectrum of SI-tasks. Moreover, we (3) show that SI-tasks expose greater model capability deficiency than non-SI tasks, to the extent that (4) proprietary models do not exhibit a decisive advantage when facing the most difficult ones. In addition, we conduct a qualitative evaluation across a diverse set of scenarios that are intuitive for humans, yet fail the most advanced multimodal models. EASI is an ongoing community effort: we have open-sourced the EASI codebase that provides a one-stop and reproducible solution with standardized interfaces, integrated protocols and prompts that significantly reduce the friction of configuring and running multiple benchmarks; we have also launched an accompanying EASI leaderboard to provide a continually updated snapshot of model performance across the full SI spectrum, accelerating collective progress toward robust SI.
comment : Codebase: https://github.com/EvolvingLMMs-Lab/EASI/; Leaderboard: https://huggingface.co/spaces/lmms-lab-si/EASI-Leaderboard
♻ ☆ G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
Wenbo Hu, Jingli Lin, Yilin Long, Yunlong Ran, Lihan Jiang, Yifan Wang, Chenming Zhu, Runsen Xu, Tai Wang, Jiangmiao Pang
Vision-Language Models (VLMs) still lack robustness in spatial intelligence, demonstrating poor performance on spatial understanding and reasoning tasks. We attribute this gap to the absence of a visual geometry learning process capable of reconstructing 3D space from 2D images. We present G$^2$VLM, a geometry grounded vision-language model that bridges two fundamental aspects of spatial intelligence: spatial 3D reconstruction and spatial understanding. G$^2$VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning. Our unified design is highly scalable for spatial understanding: it trains on abundant multi-view image and video data, while simultaneously leveraging the benefits of 3D visual priors that are typically only derived from hard-to-collect annotations. Experimental results demonstrate G$^2$VLM is proficient in both tasks, achieving comparable results to state-of-the-art feed-forward 3D reconstruction models and achieving better or competitive results across spatial understanding and reasoning tasks. By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G$^2$VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing.
comment : code are released at https://github.com/InternRobotics/G2VLM
♻ ☆ Motion Matters: Motion-guided Modulation Network for Skeleton-based Micro-Action Recognition ACM MM 2025
Jihao Gu, Kun Li, Fei Wang, Yanyan Wei, Zhiliang Wu, Hehe Fan, Meng Wang
Micro-Actions (MAs) are an important form of non-verbal communication in social interactions, with potential applications in human emotional analysis. However, existing methods in Micro-Action Recognition often overlook the inherent subtle changes in MAs, which limits the accuracy of distinguishing MAs with subtle changes. To address this issue, we present a novel Motion-guided Modulation Network (MMN) that implicitly captures and modulates subtle motion cues to enhance spatial-temporal representation learning. Specifically, we introduce a Motion-guided Skeletal Modulation module (MSM) to inject motion cues at the skeletal level, acting as a control signal to guide spatial representation modeling. In parallel, we design a Motion-guided Temporal Modulation module (MTM) to incorporate motion information at the frame level, facilitating the modeling of holistic motion patterns in micro-actions. Finally, we propose a motion consistency learning strategy to aggregate the motion cues from multi-scale features for micro-action classification. Experimental results on the Micro-Action 52 and iMiGUE datasets demonstrate that MMN achieves state-of-the-art performance in skeleton-based micro-action recognition, underscoring the importance of explicitly modeling subtle motion cues. The code will be available at https://github.com/momiji-bit/MMN.
comment : Accepted by ACM MM 2025
♻ ☆ PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation
Huadai Liu, Kaicheng Luo, Wen Wang, Qian Chen, Peiwen Sun, Rongjie Huang, Xiangang Li, Jieping Ye, Wei Xue
Video-to-Audio (V2A) generation requires balancing four critical perceptual dimensions: semantic consistency, audio-visual temporal synchrony, aesthetic quality, and spatial accuracy; yet existing methods suffer from objective entanglement that conflates competing goals in single loss functions and lack human preference alignment. We introduce PrismAudio, the first framework to integrate Reinforcement Learning into V2A generation with specialized Chain-of-Thought (CoT) planning. Our approach decomposes monolithic reasoning into four specialized CoT modules (Semantic, Temporal, Aesthetic, and Spatial CoT), each paired with targeted reward functions. This CoT-reward correspondence enables multidimensional RL optimization that guides the model to jointly generate better reasoning across all perspectives, solving the objective entanglement problem while preserving interpretability. To make this optimization computationally practical, we propose Fast-GRPO, which employs hybrid ODE-SDE sampling that dramatically reduces the training overhead compared to existing GRPO implementations. We also introduce AudioCanvas, a rigorous benchmark that is more distributionally balanced and covers more realistically diverse and challenging scenarios than existing datasets, with 300 single-event classes and 501 multi-event samples. Experimental results demonstrate that PrismAudio achieves state-of-the-art performance across all four perceptual dimensions on both the in-domain VGGSound test set and out-of-domain AudioCanvas benchmark. The project page is available at https://PrismAudio-Project.github.io.
comment : Preprint
♻ ☆ IVY-FAKE: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detection
Changjiang Jiang, Wenhui Dong, Zhonghao Zhang, Chenyang Si, Fengchang Yu, Wei Peng, Xinbin Yuan, Yifei Bi, Ming Zhao, Zian Zhou, Caifeng Shan
The rapid development of Artificial Intelligence Generated Content (AIGC) techniques has enabled the creation of high-quality synthetic content, but it also raises significant security concerns. Current detection methods face two major limitations: (1) the lack of multidimensional explainable datasets for generated images and videos. Existing open-source datasets (e.g., WildFake, GenVideo) rely on oversimplified binary annotations, which restrict the explainability and trustworthiness of trained detectors. (2) Prior MLLM-based forgery detectors (e.g., FakeVLM) exhibit insufficiently fine-grained interpretability in their step-by-step reasoning, which hinders reliable localization and explanation. To address these challenges, we introduce Ivy-Fake, the first large-scale multimodal benchmark for explainable AIGC detection. It consists of over 106K richly annotated training samples (images and videos) and 5,000 manually verified evaluation examples, sourced from multiple generative models and real world datasets through a carefully designed pipeline to ensure both diversity and quality. Furthermore, we propose Ivy-xDetector, a reinforcement learning model based on Group Relative Policy Optimization (GRPO), capable of producing explainable reasoning chains and achieving robust performance across multiple synthetic content detection benchmarks. Extensive experiments demonstrate the superiority of our dataset and confirm the effectiveness of our approach. Notably, our method improves performance on GenImage from 86.88% to 96.32%, surpassing prior state-of-the-art methods by a clear margin.
comment : 30 pages
♻ ☆ Monet: Reasoning in Latent Visual Space Beyond Images and Language
Qixun Wang, Yang Shi, Yifei Wang, Yuanxing Zhang, Pengfei Wan, Kun Gai, Xianghua Ying, Yisen Wang
"Thinking with images" has emerged as an effective paradigm for advancing visual reasoning, extending beyond text-only chains of thought by injecting visual evidence into intermediate reasoning steps. However, existing methods fall short of human-like abstract visual thinking, as their flexibility is fundamentally limited by external tools. In this work, we introduce Monet, a training framework that enables multimodal large language models (MLLMs) to reason directly within the latent visual space by generating continuous embeddings that function as intermediate visual thoughts. We identify two core challenges in training MLLMs for latent visual reasoning: high computational cost in latent-vision alignment and insufficient supervision over latent embeddings, and address them with a three-stage distillation-based supervised fine-tuning (SFT) pipeline. We further reveal a limitation of applying GRPO to latent reasoning: it primarily enhances text-based reasoning rather than latent reasoning. To overcome this, we propose VLPO (Visual-latent Policy Optimization), a reinforcement learning method that explicitly incorporates latent embeddings into policy gradient updates. To support SFT, we construct Monet-SFT-125K, a high-quality text-image interleaved CoT dataset containing 125K real-world, chart, OCR, and geometry CoTs. Our model, Monet-7B, shows consistent gains across real-world perception and reasoning benchmarks and exhibits strong out-of-distribution generalization on challenging abstract visual reasoning tasks. We also empirically analyze the role of each training component and discuss our early unsuccessful attempts, providing insights for future developments in visual latent reasoning. Our model, data, and code are available at https://github.com/NOVAglow646/Monet.
♻ ☆ A Survey on Personalized Content Synthesis with Diffusion Models
Xulu Zhang, Xiaoyong Wei, Wentao Hu, Jinlin Wu, Jiaxin Wu, Wengyu Zhang, Zhaoxiang Zhang, Zhen Lei, Qing Li
Recent advancements in diffusion models have significantly impacted content creation, leading to the emergence of Personalized Content Synthesis (PCS). By utilizing a small set of user-provided examples featuring the same subject, PCS aims to tailor this subject to specific user-defined prompts. Over the past two years, more than 150 methods have been introduced in this area. However, existing surveys primarily focus on text-to-image generation, with few providing up-to-date summaries on PCS. This paper provides a comprehensive survey of PCS, introducing the general frameworks of PCS research, which can be categorized into test-time fine-tuning (TTF) and pre-trained adaptation (PTA) approaches. We analyze the strengths, limitations, and key techniques of these methodologies. Additionally, we explore specialized tasks within the field, such as object, face, and style personalization, while highlighting their unique challenges and innovations. Despite the promising progress, we also discuss ongoing challenges, including overfitting and the trade-off between subject fidelity and text alignment. Through this detailed overview and analysis, we propose future directions to further the development of PCS.
♻ ☆ Wavefront-Constrained Passive Obscured Object Detection
Zhiwen Zheng, Yiwei Ouyang, Zhao Huang, Tao Zhang, Xiaoshuai Zhang, Huiyu Zhou, Wenwen Tang, Shaowei Jiang, Jin Liu, Xingru Huang
Accurately localizing and segmenting obscured objects from faint light patterns beyond the field of view is highly challenging due to multiple scattering and medium-induced perturbations. Most existing methods, based on real-valued modeling or local convolutional operations, are inadequate for capturing the underlying physics of coherent light propagation. Moreover, under low signal-to-noise conditions, these methods often converge to non-physical solutions, severely compromising the stability and reliability of the observation. To address these challenges, we propose a novel physics-driven Wavefront Propagating Compensation Network (WavePCNet) to simulate wavefront propagation and enhance the perception of obscured objects. This WavePCNet integrates the Tri-Phase Wavefront Complex-Propagation Reprojection (TriWCP) to incorporate complex amplitude transfer operators to precisely constrain coherent propagation behavior, along with a momentum memory mechanism to effectively suppress the accumulation of perturbations. Additionally, a High-frequency Cross-layer Compensation Enhancement is introduced to construct frequency-selective pathways with multi-scale receptive fields and dynamically model structural consistency across layers, further boosting the model's robustness and interpretability under complex environmental conditions. Extensive experiments conducted on four physically collected datasets demonstrate that WavePCNet consistently outperforms state-of-the-art methods across both accuracy and robustness.
♻ ☆ SAM 2++: Tracking Anything at Any Granularity
Jiaming Zhang, Cheng Liang, Yichun Yang, Chenkai Zeng, Yutao Cui, Xinwen Zhang, Xin Zhou, Kai Ma, Gangshan Wu, Limin Wang
Video tracking aims at finding the specific target in subsequent frames given its initial state. Due to the varying granularity of target states across different tasks, most existing trackers are tailored to a single task and heavily rely on custom-designed modules within the individual task, which limits their generalization and leads to redundancy in both model design and parameters. To unify video tracking tasks, we present SAM 2++, a unified model towards tracking at any granularity, including masks, boxes, and points. First, to extend target granularity, we design task-specific prompts to encode various task inputs into general prompt embeddings, and a unified decoder to unify diverse task results into a unified form pre-output. Next, to satisfy memory matching, the core operation of tracking, we introduce a task-adaptive memory mechanism that unifies memory across different granularities. Finally, we introduce a customized data engine to support tracking training at any granularity, producing a large and diverse video tracking dataset with rich annotations at three granularities, termed Tracking-Any-Granularity, which represents a comprehensive resource for training and benchmarking on unified tracking. Comprehensive experiments on multiple benchmarks confirm that SAM 2++ sets a new state of the art across diverse tracking tasks at different granularities, establishing a unified and robust tracking framework.
comment : 8 pages, with supp
♻ ☆ CANVAS: A Benchmark for Vision -Language Models on Tool-Based User Interface Design
Daeheon Jeong, Seoyeon Byun, Kihoon Son, Dae Hyun Kim, Juho Kim
User interface (UI) design is an iterative process in which designers progressively refine their work with design software such as Figma or Sketch. Recent advances in vision language models (VLMs) with tool invocation suggest these models can operate design software to edit a UI design through iteration. Understanding and enhancing this capacity is important, as it highlights VLMs' potential to collaborate with designers within conventional software. However, as no existing benchmark evaluates tool-based design performance, the capacity remains unknown. To address this, we introduce CANVAS, a benchmark for VLMs on tool-based user interface design. Our benchmark contains 598 tool-based design tasks paired with ground-truth references sampled from 3.3K mobile UI designs across 30 function-based categories (e.g., onboarding, messaging). In each task, a VLM updates the design step-by-step through context-based tool invocations (e.g., create a rectangle as a button background), linked to design software. Specifically, CANVAS incorporates two task types: (i) design replication evaluates the ability to reproduce a whole UI screen; (ii) design modification evaluates the ability to modify a specific part of an existing screen. Results suggest that leading models exhibit more strategic tool invocations, improving design quality. Furthermore, we identify common error patterns models exhibit, guiding future work in enhancing tool-based design capabilities.
♻ ☆ S2AFormer: Strip Self-Attention for Efficient Vision Transformer
Guoan Xu, Wenfeng Huang, Wenjing Jia, Jiamao Li, Guangwei Gao, Guo-Jun Qi
Vision Transformer (ViT) has made significant advancements in computer vision, thanks to its token mixer's sophisticated ability to capture global dependencies between all tokens. However, the quadratic growth in computational demands as the number of tokens increases limits its practical efficiency. Although recent methods have combined the strengths of convolutions and self-attention to achieve better trade-offs, the expensive pairwise token affinity and complex matrix operations inherent in self-attention remain a bottleneck. To address this challenge, we propose S2AFormer, an efficient Vision Transformer architecture featuring novel Strip Self-Attention (SSA). We design simple yet effective Hybrid Perception Blocks (HPBs) to effectively integrate the local perception capabilities of CNNs with the global context modeling of Transformer's attention mechanisms. A key innovation of SSA lies in its reduction of the spatial dimensions of $K$ and $V$, while compressing the channel dimensions of $Q$ and $K$. This design significantly reduces computational overhead while preserving accuracy, striking an optimal balance between efficiency and effectiveness. We evaluate the robustness and efficiency of S2AFormer through extensive experiments on multiple vision benchmarks, including ImageNet-1k for image classification, ADE20k for semantic segmentation, and COCO for object detection and instance segmentation. Results demonstrate that S2AFormer achieves significant accuracy gains with superior efficiency in both GPU and non-GPU environments, making it a strong candidate for efficient vision Transformers.
comment : Accepted by IEEE-TIP, 14 pages, 8 figures, 9 tables
♻ ☆ MoRe: Monocular Geometry Refinement via Graph Optimization for Cross-View Consistency
Dongki Jung, Jaehoon Choi, Yonghan Lee, Sungmin Eum, Heesung Kwon, Dinesh Manocha
Monocular 3D foundation models offer an extensible solution for perception tasks, making them attractive for broader 3D vision applications. In this paper, we propose MoRe, a training-free Monocular Geometry Refinement method designed to improve cross-view consistency and achieve scale alignment. To induce inter-frame relationships, our method employs feature matching between frames to establish correspondences. Rather than applying simple least squares optimization on these matched points, we formulate a graph-based optimization framework that performs local planar approximation using the estimated 3D points and surface normals estimated by monocular foundation models. This formulation addresses the scale ambiguity inherent in monocular geometric priors while preserving the underlying 3D structure. We further demonstrate that MoRe not only enhances 3D reconstruction but also improves novel view synthesis, particularly in sparse view rendering scenarios.
♻ ☆ Ultralight Polarity-Split Neuromorphic SNN for Event-Stream Super-Resolution AAAI2026
Chuanzhi Xu, Haoxian Zhou, Langyi Chen, Yuk Ying Chung, Qiang Qu
Event cameras offer unparalleled advantages such as high temporal resolution, low latency, and high dynamic range. However, their limited spatial resolution poses challenges for fine-grained perception tasks. In this work, we propose an ultra-lightweight, stream-based event-to-event super-resolution method based on Spiking Neural Networks (SNNs), designed for real-time deployment on resource-constrained devices. To further reduce model size, we introduce a novel Dual-Forward Polarity-Split Event Encoding strategy that decouples positive and negative events into separate forward paths through a shared SNN. Furthermore, we propose a Learnable Spatio-temporal Polarity-aware Loss (LearnSTPLoss) that adaptively balances temporal, spatial, and polarity consistency using learnable uncertainty-based weights. Experimental results demonstrate that our method achieves competitive super-resolution performance on multiple datasets while significantly reducing model size and inference time. The lightweight design enables embedding the module into event cameras or using it as an efficient front-end preprocessing for downstream vision tasks.
comment : 8 pages, 10 figures, 7 tables, accepted by AAAI2026
♻ ☆ Learning Plug-and-play Memory for Guiding Video Diffusion Models
Selena Song, Ziming Xu, Zijun Zhang, Kun Zhou, Jiaxian Guo, Lianhui Qin, Biwei Huang
Diffusion Transformer(DiT) based video generation models have recently achieved impressive visual quality and temporal coherence, but they still frequently violate basic physical laws and commonsense dynamics, revealing a lack of explicit world knowledge. In this work, we explore how to equip them with a plug-and-play memory that injects useful world knowledge. Motivated by in-context memory in Transformer-based LLMs, we conduct empirical studies to show that DiT can be steered via interventions on its hidden states, and simple low-pass and high-pass filters in the embedding space naturally disentangle low-level appearance and high-level physical/semantic cues, enabling targeted guidance. Building on these observations, we propose a learnable memory encoder DiT-Mem, composed of stacked 3D CNNs, low-/high-pass filters, and self-attention layers. The encoder maps reference videos into a compact set of memory tokens, which are concatenated as the memory within the DiT self-attention layers. During training, we keep the diffusion backbone frozen, and only optimize the memory encoder. It yields a rather efficient training process on few training parameters (150M) and 10K data samples, and enables plug-and-play usage at inference time. Extensive experiments on state-of-the-art models demonstrate the effectiveness of our method in improving physical rule following and video fidelity. Our code and data are publicly released here: https://thrcle421.github.io/DiT-Mem-Web/.
♻ ☆ Active Negative Loss: A Robust Framework for Learning with Noisy Labels
Xichen Ye, Yifan Wu, Yiqi Wang, Xiaoqiang Li, Weizhong Zhang, Yifan Chen
Deep supervised learning has achieved remarkable success across a wide range of tasks, yet it remains susceptible to overfitting when confronted with noisy labels. To address this issue, noise-robust loss functions offer an effective solution for enhancing learning in the presence of label noise. In this work, we systematically investigate the limitation of the recently proposed Active Passive Loss (APL), which employs Mean Absolute Error (MAE) as its passive loss function. Despite the robustness brought by MAE, one of its key drawbacks is that it pays equal attention to clean and noisy samples; this feature slows down convergence and potentially makes training difficult, particularly in large-scale datasets. To overcome these challenges, we introduce a novel loss function class, termed Normalized Negative Loss Functions (NNLFs), which serve as passive loss functions within the APL framework. NNLFs effectively address the limitations of MAE by concentrating more on memorized clean samples. By replacing MAE in APL with our proposed NNLFs, we enhance APL and present a new framework called Active Negative Loss (ANL). Moreover, in non-symmetric noise scenarios, we propose an entropy-based regularization technique to mitigate the vulnerability to the label imbalance. Extensive experiments demonstrate that the new loss functions adopted by our ANL framework can achieve better or comparable performance to state-of-the-art methods across various label noise types and in image segmentation tasks. The source code is available at: https://github.com/Virusdoll/Active-Negative-Loss.
comment : This work has been submitted to the IEEE for possible publication
♻ ☆ Racket Vision : A Multiple Racket Sports Benchmark for Unified Ball and Racket Analysis AAAI 2026
Linfeng Dong, Yuchen Yang, Hao Wu, Wei Wang, Yuenan Hou, Zhihang Zhong, Xiao Sun
We introduce RacketVision, a novel dataset and benchmark for advancing computer vision in sports analytics, covering table tennis, tennis, and badminton. The dataset is the first to provide large-scale, fine-grained annotations for racket pose alongside traditional ball positions, enabling research into complex human-object interactions. It is designed to tackle three interconnected tasks: fine-grained ball tracking, articulated racket pose estimation, and predictive ball trajectory forecasting. Our evaluation of established baselines reveals a critical insight for multi-modal fusion: while naively concatenating racket pose features degrades performance, a CrossAttention mechanism is essential to unlock their value, leading to trajectory prediction results that surpass strong unimodal baselines. RacketVision provides a versatile resource and a strong starting point for future research in dynamic object tracking, conditional motion forecasting, and multimodal analysis in sports. Project page at https://github.com/OrcustD/RacketVision
comment : Accepted to AAAI 2026 (Oral)
♻ ☆ DreamO: A Unified Framework for Image Customization
Chong Mou, Yanze Wu, Wenxu Wu, Zinan Guo, Pengze Zhang, Yufeng Cheng, Yiming Luo, Fei Ding, Shiwen Zhang, Xinghui Li, Mengtian Li, Mingcong Liu, Yi Zhang, Shaojin Wu, Songtao Zhao, Jian Zhang, Qian He, Xinglong Wu
Recently, extensive research on image customization (e.g., identity, subject, style, background, etc.) demonstrates strong customization capabilities in large-scale generative models. However, most approaches are designed for specific tasks, restricting their generalizability to combine different types of condition. Developing a unified framework for image customization remains an open challenge. In this paper, we present DreamO, an image customization framework designed to support a wide range of tasks while facilitating seamless integration of multiple conditions. Specifically, DreamO utilizes a diffusion transformer (DiT) framework to uniformly process input of different types. During training, we construct a large-scale training dataset that includes various customization tasks, and we introduce a feature routing constraint to facilitate the precise querying of relevant information from reference images. Additionally, we design a placeholder strategy that associates specific placeholders with conditions at particular positions, enabling control over the placement of conditions in the generated results. Moreover, we employ a progressive training strategy consisting of three stages: an initial stage focused on simple tasks with limited data to establish baseline consistency, a full-scale training stage to comprehensively enhance the customization capabilities, and a final quality alignment stage to correct quality biases introduced by low-quality data. Extensive experiments demonstrate that the proposed DreamO can effectively perform various image customization tasks with high quality and flexibly integrate different types of control conditions.
♻ ☆ One-Step Diffusion Transformer for Controllable Real-Wo rl d Image Super-Resolution
Yushun Fang, Yuxiang Chen, Shibo Yin, Qiang Hu, Jiangchao Yao, Ya Zhang, Xiaoyun Zhang, Yanfeng Wang
Recent advances in diffusion-based real-world image super-resolution (Real-ISR) have demonstrated remarkable perceptual quality, yet the balance between fidelity and controllability remains a problem: multi-step diffusion-based methods suffer from generative diversity and randomness, resulting in low fidelity, while one-step methods lose control flexibility due to fidelity-specific finetuning. In this paper, we present ODTSR, a one-step diffusion transformer based on Qwen-Image that performs Real-ISR considering fidelity and controllability simultaneously: a newly introduced visual stream receives low-quality images (LQ) with adjustable noise (Control Noise), and the original visual stream receives LQs with consistent noise (Prior Noise), forming the Noise-hybrid Visual Stream (NVS) design. ODTSR further employs Fidelity-aware Adversarial Training (FAA) to enhance controllability and achieve one-step inference. Extensive experiments demonstrate that ODTSR not only achieves state-of-the-art (SOTA) performance on generic Real-ISR, but also enables prompt controllability on challenging scenarios such as real-world scene text image super-resolution (STISR) of Chinese characters without training on specific datasets. Codes are available at https://github.com/RedMediaTech/ODTSR.
♻ ☆ Mavors: Multi-granularity Video Representation for Multimodal Large Language Model
Yang Shi, Jiaheng Liu, Yushuo Guan, Zhenhua Wu, Yuanxing Zhang, Zihao Wang, Weihong Lin, Jingyun Hua, Zekun Wang, Xinlong Chen, Bohan Zeng, Wentao Zhang, Fuzheng Zhang, Wenjing Yang, Di Zhang
Long-context video understanding in multimodal large language models (MLLMs) faces a critical challenge: balancing computational efficiency with the retention of fine-grained spatio-temporal patterns. Existing approaches (e.g., sparse sampling, dense sampling with low resolution, and token compression) suffer from significant information loss in temporal dynamics, spatial details, or subtle interactions, particularly in videos with complex motion or varying resolutions. To address this, we propose $\mathbf{Mavors}$, a novel framework that introduces $\mathbf{M}$ulti-gr$\mathbf{a}$nularity $\mathbf{v}$ide$\mathbf{o}$ $\mathbf{r}$epre$\mathbf{s}$entation for holistic long-video modeling. Specifically, Mavors directly encodes raw video content into latent representations through two core components: 1) an Intra-chunk Vision Encoder (IVE) that preserves high-resolution spatial features via 3D convolutions and Vision Transformers, and 2) an Inter-chunk Feature Aggregator (IFA) that establishes temporal coherence across chunks using transformer-based dependency modeling with chunk-level rotary position encodings. Moreover, the framework unifies image and video understanding by treating images as single-frame videos via sub-image decomposition. Experiments across diverse benchmarks demonstrate Mavors' superiority in maintaining both spatial fidelity and temporal continuity, significantly outperforming existing methods in tasks requiring fine-grained spatio-temporal reasoning.
comment : 22 pages
♻ ☆ SKEL-CF: Coarse-to-Fine Biomechanical Skeleton and Surface Mesh Recovery
Da Li, Jiping Jin, Xuanlong Yu, Wei Liu, Xiaodong Cun, Kai Chen, Rui Fan, Jiangang Kong, Xi Shen
Parametric 3D human models such as SMPL have driven significant advances in human pose and shape estimation, yet their simplified kinematics limit biomechanical realism. The recently proposed SKEL model addresses this limitation by re-rigging SMPL with an anatomically accurate skeleton. However, estimating SKEL parameters directly remains challenging due to limited training data, perspective ambiguities, and the inherent complexity of human articulation. We introduce SKEL-CF, a coarse-to-fine framework for SKEL parameter estimation. SKEL-CF employs a transformer-based encoder-decoder architecture, where the encoder predicts coarse camera and SKEL parameters, and the decoder progressively refines them in successive layers. To ensure anatomically consistent supervision, we convert the existing SMPL-based dataset 4DHuman into a SKEL-aligned version, 4DHuman-SKEL, providing high-quality training data for SKEL estimation. In addition, to mitigate depth and scale ambiguities, we explicitly incorporate camera modeling into the SKEL-CF pipeline and demonstrate its importance across diverse viewpoints. Extensive experiments validate the effectiveness of the proposed design. On the challenging MOYO dataset, SKEL-CF achieves 85.0 MPJPE / 51.4 PA-MPJPE, significantly outperforming the previous SKEL-based state-of-the-art HSMR (104.5 / 79.6). These results establish SKEL-CF as a scalable and anatomically faithful framework for human motion analysis, bridging the gap between computer vision and biomechanics. Our implementation is available on the project page: https://pokerman8.github.io/SKEL-CF/.
comment : Project page: https://pokerman8.github.io/SKEL-CF/
♻ ☆ Histomorphology-Guided Prototypical Multi-Instance Learning for Breast Cancer WSI Classification
Baizhi Wang, Rui Yan, Wenxin Ma, Xu Zhang, Yuhao Wang, Xiaolong Li, Yunjie Gu, Zihang Jiang, S. Kevin Zhou
Histomorphology is crucial in cancer diagnosis. However, existing whole slide image (WSI) classification methods struggle to effectively incorporate histomorphology information, limiting their ability to capture key pathological features. Particularly when the number of instances within a bag is large and their features are complex, it becomes challenging to accurately identify instances decisive for the bag label, making these methods prone to interference from ambiguous instances. To address this limitation, we propose a novel Histomorphology-Guided Prototypical Multi-Instance Learning (HGPMIL) framework that explicitly learns histomorphology-guided prototypical representations by incorporating tumor cellularity, cellular morphology, and tissue architecture. Specifically, our approach consists of three key components: (1) estimating the importance of tumor-related histomorphology information at patch-level based on medical prior knowledge; (2) generating representative prototypes through histomorphology-prototypical clustering; and (3) enabling WSI classification through histomorphology-guided prototypical aggregation. HGPMIL adjusts the decision boundary by incorporating histomorphological importance to reduce instance label uncertainty, thereby reversely optimizing the bag-level boundary. Experimental results demonstrate its effectiveness, achieving high diagnostic accuracy for molecular subtyping, cancer subtyping and survival analysis. The code will be made available at https://github.com/Badgewho/HMDMIL.
comment : 11 pages,8 figures
♻ ☆ Seeing What Matters: Visual Preference Policy Optimization for Visual Generation
Ziqi Ni, Yuanzhi Liang, Rui Li, Yi Zhou, Haibing Huang, Chi Zhang, Xuelong Li
Reinforcement learning (RL) has become a powerful tool for post-training visual generative models, with Group Relative Policy Optimization (GRPO) increasingly used to align generators with human preferences. However, existing GRPO pipelines rely on a single scalar reward per sample, treating each image or video as a holistic entity and ignoring the rich spatial and temporal structure of visual content. This coarse supervision hinders the correction of localized artifacts and the modeling of fine-grained perceptual cues. We introduce Visual Preference Policy Optimization (ViPO), a GRPO variant that lifts scalar feedback into structured, pixel-level advantages. ViPO employs a Perceptual Structuring Module that uses pretrained vision backbones to construct spatially and temporally aware advantage maps, redistributing optimization pressure toward perceptually important regions while preserving the stability of standard GRPO. Across both image and video benchmarks, ViPO consistently outperforms vanilla GRPO, improving in-domain alignment with human-preference rewards and enhancing generalization on out-of-domain evaluations. The method is architecture-agnostic, lightweight, and fully compatible with existing GRPO training pipelines, providing a more expressive and informative learning signal for visual generation.
♻ ☆ Ambiguity-aware Truncated Flow Matching for Ambiguous Medical Image Segmentation AAAI-26
Fanding Li, Xiangyu Li, Xianghe Su, Xingyu Qiu, Suyu Dong, Wei Wang, Kuanquan Wang, Gongning Luo, Shuo Li
A simultaneous enhancement of accuracy and diversity of predictions remains a challenge in ambiguous medical image segmentation (AMIS) due to the inherent trade-offs. While truncated diffusion probabilistic models (TDPMs) hold strong potential with a paradigm optimization, existing TDPMs suffer from entangled accuracy and diversity of predictions with insufficient fidelity and plausibility. To address the aforementioned challenges, we propose Ambiguity-aware Truncated Flow Matching (ATFM), which introduces a novel inference paradigm and dedicated model components. Firstly, we propose Data-Hierarchical Inference, a redefinition of AMIS-specific inference paradigm, which enhances accuracy and diversity at data-distribution and data-sample level, respectively, for an effective disentanglement. Secondly, Gaussian Truncation Representation (GTR) is introduced to enhance both fidelity of predictions and reliability of truncation distribution, by explicitly modeling it as a Gaussian distribution at $T_{\text{trunc}}$ instead of using sampling-based approximations. Thirdly, Segmentation Flow Matching (SFM) is proposed to enhance the plausibility of diverse predictions by extending semantic-aware flow transformation in Flow Matching (FM). Comprehensive evaluations on LIDC and ISIC3 datasets demonstrate that ATFM outperforms SOTA methods and simultaneously achieves a more efficient inference. ATFM improves GED and HM-IoU by up to $12\%$ and $7.3\%$ compared to advanced methods.
comment : 13 pages, 10 figures, extended version of AAAI-26 paper
♻ ☆ ROVER: Recursive Reasoning Over Videos with Vision -Language Models for Embodied Tasks
Philip Schroeder, Ondrej Biza, Thomas Weng, Hongyin Luo, James Glass
Vision-language models (VLMs) have exhibited impressive capabilities across diverse image understanding tasks, but still struggle in settings that require reasoning over extended sequences of camera frames from a video. This limits their utility in embodied settings, which require reasoning over long frame sequences from a continuous stream of visual input at each moment of a task attempt. To address this limitation, we propose ROVER (Reasoning Over VidEo Recursively), a framework that enables the model to recursively decompose long-horizon video trajectories into segments corresponding to shorter subtasks within the trajectory. In doing so, ROVER facilitates more focused and accurate reasoning over temporally localized frame sequences without losing global context. We evaluate ROVER, implemented using an in-context learning approach, on diverse OpenX Embodiment videos and on a new dataset derived from RoboCasa that consists of 543 videos showing both expert and perturbed non-expert trajectories across 27 robotic manipulation tasks. ROVER outperforms strong baselines across three video reasoning tasks: task progress estimation, frame-level natural language reasoning, and video question answering. We observe that, by reducing the number of frames the model reasons over at each timestep, ROVER mitigates hallucinations, especially during unexpected or non-optimal moments of a trajectory. In addition, by enabling the implementation of a subtask-specific sliding context window, ROVER's time complexity scales linearly with video length, an asymptotic improvement over baselines. Demos, code, and data available at: https://rover-vlm.github.io
♻ ☆ CNN-LSTM Hybrid Architecture for Over-the-Air Automatic Modulation Classification Using SDR
Dinanath Padhya, Krishna Acharya, Bipul Kumar Dahal, Dinesh Baniya Kshatri
Automatic Modulation Classification (AMC) is a core technology for future wireless communication systems, enabling the identification of modulation schemes without prior knowledge. This capability is essential for applications in cognitive radio, spectrum monitoring, and intelligent communication networks. We propose an AMC system based on a hybrid Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) architecture, integrated with a Software Defined Radio (SDR) platform. The proposed architecture leverages CNNs for spatial feature extraction and LSTMs for capturing temporal dependencies, enabling efficient handling of complex, time-varying communication signals. The system's practical ability was demonstrated by identifying over-the-air (OTA) signals from a custom-built FM transmitter alongside other modulation schemes. The system was trained on a hybrid dataset combining the RadioML2018 dataset with a custom-generated dataset, featuring samples at Signal-to-Noise Ratios (SNRs) from 0 to 30dB. System performance was evaluated using accuracy, precision, recall, F1 score, and the Area Under the Receiver Operating Characteristic Curve (AUC-ROC). The optimized model achieved 93.48% accuracy, 93.53% precision, 93.48% recall, and an F1 score of 93.45%. The AUC-ROC analysis confirmed the model's discriminative power, even in noisy conditions. This paper's experimental results validate the effectiveness of the hybrid CNN-LSTM architecture for AMC, suggesting its potential application in adaptive spectrum management and advanced cognitive radio systems.
comment : 7 Pages, 11 figures, 2 Tables, Accepted in Journal (Journal of Innovations in Engineering Education)
♻ ☆ Grasp Diffusion : Synthesizing Realistic Whole-body Hand-Object Interaction WACV 2026
Patrick Kwon, Chen Chen, Hanbyul Joo
Recent generative models can synthesize high-quality images, but they often fail to generate humans interacting with objects using their hands. This arises mostly from the model's misunderstanding of such interactions and the hardships of synthesizing intricate regions of the body. In this paper, we propose \textbf{GraspDiffusion}, a novel generative method that creates realistic scenes of human-object interaction. Given a 3D object, GraspDiffusion constructs whole-body poses with control over the object's location relative to the human body, which is achieved by separately leveraging the generative priors for body and hand poses, optimizing them into a joint grasping pose. This pose guides the image synthesis to correctly reflect the intended interaction, creating realistic and diverse human-object interaction scenes. We demonstrate that GraspDiffusion can successfully tackle the relatively uninvestigated problem of generating full-bodied human-object interactions while outperforming previous methods. Our project page is available at https://yj7082126.github.io/graspdiffusion/
comment : Paper has been accepted to WACV 2026
♻ ☆ Memo: Training Memory-Efficient Embodied Agent s with Reinforcement Learning NeurIPS 2025
Gunshi Gupta, Karmesh Yadav, Zsolt Kira, Yarin Gal, Rahaf Aljundi
To enable embodied agents to operate effectively over extended timeframes, it is crucial to develop models that form and access memories to stay contextualized in their environment. In the current paradigm of training transformer-based policies for embodied sequential decision-making tasks, visual inputs often overwhelm the context limits of transformers, while humans can maintain and utilize a lifetime of experience compressed as memories. Significant compression is possible in principle, as much of the input is irrelevant and can be abstracted. However, existing approaches predominantly focus on either recurrent models with fixed-size memory or transformers with full-context reliance. In this work, we propose Memo, a transformer-based architecture and training recipe for reinforcement learning (RL) on memory-intensive, long-horizon tasks. Memo incorporates the creation and retrieval of memory by interleaving periodic summarization tokens with the inputs of a model during training. We demonstrate Memo's effectiveness on a gridworld meta-RL benchmark and a multi-object navigation task in photo-realistic indoor settings. Memo outperforms naive long-context transformer baselines while being more compute and storage efficient. Additionally, Memo generalizes better to longer contexts at inference time and remains robust in streaming settings, where historical context must be truncated to fit inference constraints. Our code is available at: https://github.com/gunshi/memo.
comment : Accepted for Spotlight Presentation at NeurIPS 2025
♻ ☆ Training-Free Adaptive Quantization for Variable Rate Image Coding for Machines
Yui Tatsumi, Ziyue Zeng, Hiroshi Watanabe
Image Coding for Machines (ICM) has become increasingly important with the rapid integration of computer vision technology into real-world applications. However, most neural network-based ICM frameworks operate at a fixed rate, thus requiring individual training for each target bitrate. This limitation may restrict their practical usage. Existing variable rate image compression approaches mitigate this issue but often rely on additional training, which increases computational costs and complicates deployment. Moreover, variable rate control has not been thoroughly explored for ICM. To address these challenges, we propose a training-free quantization strength control scheme that enables flexible bitrate adjustment. By exploiting the scale parameter predicted by the hyperprior network, the proposed method adaptively modulates quantization step sizes across both channel and spatial dimensions. This allows the model to preserve semantically important regions while coarsely quantizing less critical areas. Our architectural design further enables continuous bitrate control through a single parameter. Experimental results demonstrate the effectiveness of our proposed method, achieving up to 11.07% BD-rate savings over the non-adaptive variable rate baseline.
♻ ☆ STAvatar: Soft Binding and Temporal Density Control for Monocular 3D Head Avatars Reconstruction
Jiankuo Zhao, Xiangyu Zhu, Zidu Wang, Zhen Lei
Reconstructing high-fidelity and animatable 3D head avatars from monocular videos remains a challenging yet essential task. Existing methods based on 3D Gaussian Splatting typically bind Gaussians to mesh triangles and model deformations solely via Linear Blend Skinning, which results in rigid motion and limited expressiveness. Moreover, they lack specialized strategies to handle frequently occluded regions (e.g., mouth interiors, eyelids). To address these limitations, we propose STAvatar, which consists of two key components: (1) a UV-Adaptive Soft Binding framework that leverages both image-based and geometric priors to learn per-Gaussian feature offsets within the UV space. This UV representation supports dynamic resampling, ensuring full compatibility with Adaptive Density Control (ADC) and enhanced adaptability to shape and textural variations. (2) a Temporal ADC strategy, which first clusters structurally similar frames to facilitate more targeted computation of the densification criterion. It further introduces a novel fused perceptual error as clone criterion to jointly capture geometric and textural discrepancies, encouraging densification in regions requiring finer details. Extensive experiments on four benchmark datasets demonstrate that STAvatar achieves state-of-the-art reconstruction performance, especially in capturing fine-grained details and reconstructing frequently occluded regions. The code will be publicly available.
comment : 17 pages, 14 figures
♻ ☆ R-AVST: Empowering Video-LLMs with Fine-Grained Spatio-Temporal Reasoning in Complex Audio-Visual Scenarios AAAI 2026
Lu Zhu, Tiantian Geng, Yangye Chen, Teng Wang, Ping Lu, Feng Zheng
Recently, rapid advancements have been made in multimodal large language models (MLLMs), especially in video understanding tasks. However, current research focuses on simple video scenarios, failing to reflect the complex and diverse nature of real-world audio-visual events in videos. To bridge this gap, we firstly introduce R-AVST, a dataset for audio-visual reasoning featuring fine-grained spatio-temporal annotations. In constructing this, we design a pipeline consisting of LLM-based key object extraction, automatic spatial annotation and manual quality inspection, resulting in over 5K untrimmed videos with 27K objects across 100 types of audio-visual events. Building on this dataset, we define three core tasks for spatio-temporal reasoning in audio-visual scenes and generate more than 8K high-quality, evenly distributed question-answer pairs to effectively benchmark model performance. To further enhance reasoning, we propose AVST-Zero, a reinforcement learning-based model that avoids intermediate supervision, directly optimizing behavior via carefully designed multi-dimensional rewards. Extensive experiments validate the effectiveness of our R-AVST in advancing audio-visual spatio-temporal reasoning, upon which AVST-Zero demonstrates competitive performance compared to existing models. To the best of our knowledge, R-AVST is the first dataset designed for real-world audio-visual spatio-temporal reasoning, and AVST-Zero offers a novel perspective for tackling future challenges in this domain.
comment : Accepted by AAAI 2026. Project page: https://github.com/zhlllau/R-AVST
♻ ☆ Neural Octahedral Field: Octahedral prior for simultaneous smoothing and sharp edge regularization
Ruichen Zheng, Tao Yu, Ruizhen Hu
Neural implicit representation, the parameterization of a continuous distance function as a Multi-Layer Perceptron (MLP), has emerged as a promising lead in tackling surface reconstruction from unoriented point clouds. In the presence of noise, however, its lack of explicit neighborhood connectivity makes sharp edges identification particularly challenging, hence preventing the separation of smoothing and sharpening operations, as is achievable with its discrete counterparts. In this work, we propose to tackle this challenge with an auxiliary field, the \emph{octahedral field}. We observe that both smoothness and sharp features in the distance field can be equivalently described by the smoothness in octahedral space. Therefore, by aligning and smoothing an octahedral field alongside the implicit geometry, our method behaves analogously to bilateral filtering, resulting in a smooth reconstruction while preserving sharp edges. Despite being operated purely pointwise, our method outperforms various traditional and neural implicit fitting approaches across extensive experiments, and is very competitive with methods that require normals and data priors. Code and data of our work are available at: https://github.com/Ankbzpx/frame-field.
comment : project page: https://github.com/Ankbzpx/frame-field
♻ ☆ Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMs
Yunqi Hong, Sohyun An, Andrew Bai, Neil Y. C. Lin, Cho-Jui Hsieh
Despite Multimodal Large Language Models (MLLMs) showing promising results on general zero-shot image classification tasks, fine-grained image classification remains challenging. It demands precise attention to subtle visual details to distinguish between visually similar subcategories--details that MLLMs may easily overlook without explicit guidance. To address this, we introduce AutoSEP, an iterative self-supervised prompt learning framework designed to enhance MLLM fine-grained classification capabilities in a fully unsupervised manner. Our core idea is to leverage unlabeled data to learn a description prompt that guides MLLMs in identifying crucial discriminative features within an image, and boosts classification accuracy. We developed an automatic self-enhancing prompt learning framework called AutoSEP to iteratively improve the description prompt using unlabeled data, based on instance-level classification scoring function. AutoSEP only requires black-box access to MLLMs, eliminating the need for any training or fine-tuning. We evaluate our approach on multiple fine-grained classification datasets. It consistently outperforms other unsupervised baselines, demonstrating the effectiveness of our self-supervised optimization framework. Notably, AutoSEP on average improves 13 percent over standard zero-shot classification and 5 percent over the best-performing baselines. Code is available at: https://github.com/yq-hong/AutoSEP
♻ ☆ Building temporally coherent 3D maps with VGGT for memory-efficient Semantic SLAM
Gergely Dinya, Péter Halász, András Lőrincz, Kristóf Karacs, Anna Gelencsér-Horváth
We present a fast, spatio-temporal scene understanding framework based on Visual Geometry Grounded Transformer (VGGT). The proposed pipeline is designed to enable efficient, close to real-time performance, supporting applications including assistive navigation. To achieve continuous updates of the 3D scene representation, we process the image flow with a sliding window, aligning submaps, thereby overcoming VGGT's high memory demands. We exploit the VGGT tracking head to aggregate 2D semantic instance masks into 3D objects. To allow for temporal consistency and richer contextual reasoning the system stores timestamps and instance-level identities, thereby enabling the detection of changes in the environment. We evaluate the approach on well-known benchmarks and custom datasets specifically designed for assistive navigation scenarios. The results demonstrate the applicability of the framework to real-world scenarios.
Computation and Language 71
☆ Intelligent Neural Networks: From Layered Architectures to Graph-Organized Intelligence
Antoine Salomon
Biological neurons exhibit remarkable intelligence: they maintain internal states, communicate selectively with other neurons, and self-organize into complex graphs rather than rigid hierarchical layers. What if artificial intelligence could emerge from similarly intelligent computational units? We introduce Intelligent Neural Networks (INN), a paradigm shift where neurons are first-class entities with internal memory and learned communication patterns, organized in complete graphs rather than sequential layers. Each Intelligent Neuron combines selective state-space dynamics (knowing when to activate) with attention-based routing (knowing to whom to send signals), enabling emergent computation through graph-structured interactions. On the standard Text8 character modeling benchmark, INN achieves 1.705 Bit-Per-Character (BPC), significantly outperforming a comparable Transformer (2.055 BPC) and matching a highly optimized LSTM baseline. Crucially, a parameter-matched baseline of stacked Mamba blocks fails to converge (>3.4 BPC) under the same training protocol, demonstrating that INN's graph topology provides essential training stability. Ablation studies confirm this: removing inter-neuron communication degrades performance or leads to instability, proving the value of learned neural routing. This work demonstrates that neuron-centric design with graph organization is not merely bio-inspired -- it is computationally effective, opening new directions for modular, interpretable, and scalable neural architectures.
comment : Code available at https://github.com/AntoineSal/IntelligentNeuralNetwork
☆ PRISM: Privacy-Aware Routing for Adaptive Cloud-Edge LLM Inference via Semantic Sketch Collaboration AAAI 2026
Junfei Zhan, Haoxun Shen, Zheng Lin, Tengjiao He
Large Language Models (LLMs) demonstrate impressive capabilities in natural language understanding and generation, but incur high communication overhead and privacy risks in cloud deployments, while facing compute and memory constraints when confined to edge devices. Cloud-edge inference has emerged as a promising paradigm for improving privacy in LLM services by retaining sensitive computations on local devices. However, existing cloud-edge inference approaches apply uniform privacy protection without considering input sensitivity, resulting in unnecessary perturbation and degraded utility even for non-sensitive tokens. To address this limitation, we propose Privacy-aware Routing for Inference with Semantic Modulation (PRISM), a context-aware framework that dynamically balances privacy and inference quality. PRISM executes in four stages: (1) the edge device profiles entity-level sensitivity; (2) a soft gating module on the edge selects an execution mode - cloud, edge, or collaboration; (3) for collaborative paths, the edge applies adaptive two-layer local differential privacy based on entity risks; and (4) the cloud LLM generates a semantic sketch from the perturbed prompt, which is then refined by the edge-side small language model (SLM) using local context. Our results show that PRISM consistently achieves superior privacy-utility trade-offs across various scenarios, reducing energy consumption and latency to 40-50% of baseline methods such as Uniform and Selective LDP, while maintaining high output quality under strong privacy constraints. These findings are validated through comprehensive evaluations involving realistic prompts, actual energy measurements, and heterogeneous cloud-edge model deployments.
comment : Accepted to AAAI 2026. This is the arXiv preprint version
☆ Modeling Romanized Hindi and Bengali: Dataset Creation and Multilingual LLM Integration
Kanchon Gharami, Quazi Sarwar Muhtaseem, Deepti Gupta, Lavanya Elluri, Shafika Showkat Moni
The development of robust transliteration techniques to enhance the effectiveness of transforming Romanized scripts into native scripts is crucial for Natural Language Processing tasks, including sentiment analysis, speech recognition, information retrieval, and intelligent personal assistants. Despite significant advancements, state-of-the-art multilingual models still face challenges in handling Romanized script, where the Roman alphabet is adopted to represent the phonetic structure of diverse languages. Within the South Asian context, where the use of Romanized script for Indo-Aryan languages is widespread across social media and digital communication platforms, such usage continues to pose significant challenges for cutting-edge multilingual models. While a limited number of transliteration datasets and models are available for Indo-Aryan languages, they generally lack sufficient diversity in pronunciation and spelling variations, adequate code-mixed data for large language model (LLM) training, and low-resource adaptation. To address this research gap, we introduce a novel transliteration dataset for two popular Indo-Aryan languages, Hindi and Bengali, which are ranked as the 3rd and 7th most spoken languages worldwide. Our dataset comprises nearly 1.8 million Hindi and 1 million Bengali transliteration pairs. In addition to that, we pre-train a custom multilingual seq2seq LLM based on Marian architecture using the developed dataset. Experimental results demonstrate significant improvements compared to existing relevant models in terms of BLEU and CER metrics.
comment : Proceedings of the 8th Workshop on Big Data for Cybersecurity (BigCyber)
☆ ReAG: Reasoning -Augmented Generation for Knowledge-based Visual Question Answering
Alberto Compagnoni, Marco Morini, Sara Sarto, Federico Cocchi, Davide Caffagni, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
Multimodal Large Language Models (MLLMs) have shown impressive capabilities in jointly understanding text, images, and videos, often evaluated via Visual Question Answering (VQA). However, even state-of-the-art MLLMs struggle with domain-specific or knowledge-intensive queries, where relevant information is underrepresented in pre-training data. Knowledge-based VQA (KB-VQA) addresses this by retrieving external documents to condition answer generation, but current retrieval-augmented approaches suffer from low precision, noisy passages, and limited reasoning. To address this, we propose ReAG, a novel Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages, ensuring high-quality additional context. The model follows a multi-stage training strategy leveraging reinforcement learning to enhance reasoning over retrieved content, while supervised fine-tuning serves only as a cold start. Extensive experiments on Encyclopedic-VQA and InfoSeek demonstrate that ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence. Our source code is publicly available at: https://github.com/aimagelab/ReAG.
☆ Mechanistic Finetuning of Vision -Language-Action Models via Few-Shot Demonstrations
Chancharik Mitra, Yusen Luo, Raj Saravanan, Dantong Niu, Anirudh Pai, Jesse Thomason, Trevor Darrell, Abrar Anwar, Deva Ramanan, Roei Herzig
Vision-Language Action (VLAs) models promise to extend the remarkable success of vision-language models (VLMs) to robotics. Yet, unlike VLMs in the vision-language domain, VLAs for robotics require finetuning to contend with varying physical factors like robot embodiment, environment characteristics, and spatial relationships of each task. Existing fine-tuning methods lack specificity, adapting the same set of parameters regardless of a task's visual, linguistic, and physical characteristics. Inspired by functional specificity in neuroscience, we hypothesize that it is more effective to finetune sparse model representations specific to a given task. In this work, we introduce Robotic Steering, a finetuning approach grounded in mechanistic interpretability that leverages few-shot demonstrations to identify and selectively finetune task-specific attention heads aligned with the physical, visual, and linguistic requirements of robotic tasks. Through comprehensive on-robot evaluations with a Franka Emika robot arm, we demonstrate that Robotic Steering outperforms LoRA while achieving superior robustness under task variation, reduced computational cost, and enhanced interpretability for adapting VLAs to diverse robotic tasks.
☆ Improving LLM-based Ontology Matching with fine-tuning on synthetic data
Guilherme Sousa, Rinaldo Lima, Cassia Trojahn
Large Language Models (LLMs) are increasingly being integrated into various components of Ontology Matching pipelines. This paper investigates the capability of LLMs to perform ontology matching directly on ontology modules and generate the corresponding alignments. Furthermore, it is explored how a dedicated fine-tuning strategy can enhance the model's matching performance in a zero-shot setting. The proposed method incorporates a search space reduction technique to select relevant subsets from both source and target ontologies, which are then used to automatically construct prompts. Recognizing the scarcity of reference alignments for training, a novel LLM-based approach is introduced for generating a synthetic dataset. This process creates a corpus of ontology submodule pairs and their corresponding reference alignments, specifically designed to fine-tune an LLM for the ontology matching task. The proposed approach was evaluated on the Conference, Geolink, Enslaved, Taxon, and Hydrography datasets from the OAEI complex track. The results demonstrate that the LLM fine-tuned on the synthetically generated data exhibits superior performance compared to the non-fine-tuned base model. The key contribution is a strategy that combines automatic dataset generation with fine-tuning to effectively adapt LLMs for ontology matching tasks.
☆ Smarter, not Bigger: Fine-Tuned RAG-Enhanced LLMs for Automotive HIL Testing
Chao Feng, Zihan Liu, Siddhant Gupta, Gongpei Cui, Jan von der Assen, Burkhard Stiller
Hardware-in-the-Loop (HIL) testing is essential for automotive validation but suffers from fragmented and underutilized test artifacts. This paper presents HIL-GPT, a retrieval-augmented generation (RAG) system integrating domain-adapted large language models (LLMs) with semantic retrieval. HIL-GPT leverages embedding fine-tuning using a domain-specific dataset constructed via heuristic mining and LLM-assisted synthesis, combined with vector indexing for scalable, traceable test case and requirement retrieval. Experiments show that fine-tuned compact models, such as \texttt{bge-base-en-v1.5}, achieve a superior trade-off between accuracy, latency, and cost compared to larger models, challenging the notion that bigger is always better. An A/B user study further confirms that RAG-enhanced assistants improve perceived helpfulness, truthfulness, and satisfaction over general-purpose LLMs. These findings provide insights for deploying efficient, domain-aligned LLM-based assistants in industrial HIL environments.
☆ Extension Condition "violations" and Merge optimality constraints
Matilde Marcolli, Richard Larson, Riny Huijbregts
We analyze, using the mathematical formulation of Merge within the Strong Minimalist Thesis framework, a set of linguistic phenomena, including head-to-head movement, phrasal affixes and syntactic cliticization, verb-particle alternation, and operator-variable phenomena. These are often regarded as problematic, as violations of the Extension Condition. We show that, in fact, all of these phenomena can be explained without involving any EC violation. We first show that derivations using Sideward Merge are possible for all of these cases: these respect EC, though they involve some amount of optimality violations, with respect to Resource Restrictions cost functions, andthe amount of violation differs among these cases. We show that all the cases that involve large optimality violations can be derived in alternative ways involving neither EC nor the use of SM. The main remaining case (head-to-head movement) only involves SM with minimal violations of optimality (near equilibrium fluctuations). We analyze explicitly also the cases of multiple wh-fronting, clusters of clitics in Romance languages and possessor agreement construction in Korean, and how an explanation of these phenomena based on SM can be made compatible with the colored operad generators for phases and theta roles. We also show that the EC condition has a clear algebraic meaning in the mathematical formulation of Merge and is therefore an intrinsic structural algebraic constraint of the model, rather than an additional assumption. We also show that the minimal optimality violating SM plays a structural role in the Markovian properties of Merge, and we compare different optimality conditions coming from Minimal Search and from Resource Restriction in terms of their effect on the dynamics of the Hopf algebra Markov chain, in a simple explicit example.
comment : 85 pages
☆ DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning
Zhihong Shao, Yuxiang Luo, Chengda Lu, Z. Z. Ren, Jiewen Hu, Tian Ye, Zhibin Gou, Shirong Ma, Xiaokang Zhang
Large language models have made significant progress in mathematical reasoning, which serves as an important testbed for AI and could impact scientific research if further advanced. By scaling reasoning with reinforcement learning that rewards correct final answers, LLMs have improved from poor performance to saturating quantitative reasoning competitions like AIME and HMMT in one year. However, this approach faces fundamental limitations. Pursuing higher final answer accuracy doesn't address a key issue: correct answers don't guarantee correct reasoning. Moreover, many mathematical tasks like theorem proving require rigorous step-by-step derivation rather than numerical answers, making final answer rewards inapplicable. To push the limits of deep reasoning, we believe it is necessary to verify the comprehensiveness and rigor of mathematical reasoning. Self-verification is particularly important for scaling test-time compute, especially for open problems without known solutions. Towards self-verifiable mathematical reasoning, we investigate how to train an accurate and faithful LLM-based verifier for theorem proving. We then train a proof generator using the verifier as the reward model, and incentivize the generator to identify and resolve as many issues as possible in their own proofs before finalizing them. To maintain the generation-verification gap as the generator becomes stronger, we propose to scale verification compute to automatically label new hard-to-verify proofs, creating training data to further improve the verifier. Our resulting model, DeepSeekMath-V2, demonstrates strong theorem-proving capabilities, achieving gold-level scores on IMO 2025 and CMO 2024 and a near-perfect 118/120 on Putnam 2024 with scaled test-time compute.
☆ Joint Speech and Text Training for LLM-Based End-to-End Spoken Dialogue State Tracking ICASSP 2026
Katia Vendrame, Bolaji Yusuf, Santosh Kesiraju, Šimon Sedláček, Oldřich Plchot, Jan Černocký
End-to-end spoken dialogue state tracking (DST) is made difficult by the tandem of having to handle speech input and data scarcity. Combining speech foundation encoders and large language models has been proposed in recent work as to alleviate some of this difficulty. Although this approach has been shown to result in strong spoken DST models, achieving state-of-the-art performance in realistic multi-turn DST, it struggles to generalize across domains and requires annotated spoken DST training data for each domain of interest. However, collecting such data for every target domain is both costly and difficult. Noting that textual DST data is more easily obtained for various domains, in this work, we propose jointly training on available spoken DST data and written textual data from other domains as a way to achieve cross-domain generalization. We conduct experiments which show the efficacy of our proposed method for getting good cross-domain DST performance without relying on spoken training data from the target domains.
comment : submitted to ICASSP 2026
☆ What Shape Is Optimal for Masks in Text Removal?
Hyakka Nakada, Marika Kubota
The advent of generative models has dramatically improved the accuracy of image inpainting. In particular, by removing specific text from document images, reconstructing original images is extremely important for industrial applications. However, most existing methods of text removal focus on deleting simple scene text which appears in images captured by a camera in an outdoor environment. There is little research dedicated to complex and practical images with dense text. Therefore, we created benchmark data for text removal from images including a large amount of text. From the data, we found that text-removal performance becomes vulnerable against mask profile perturbation. Thus, for practical text-removal tasks, precise tuning of the mask shape is essential. This study developed a method to model highly flexible mask profiles and learn their parameters using Bayesian optimization. The resulting profiles were found to be character-wise masks. It was also found that the minimum cover of a text region is not optimal. Our research is expected to pave the way for a user-friendly guideline for manual masking.
comment : 12 pages, 17 figures
☆ Exploring Performance Variations in Finetuned Translators of Ultra-Low Resource Languages: Do Linguistic Differences Matter?
Isabel Gonçalves, Paulo Cavalin, Claudio Pinhanez
Finetuning pre-trained language models with small amounts of data is a commonly-used method to create translators for ultra-low resource languages such as endangered Indigenous languages. However, previous works have reported substantially different performances with translators created using similar methodology and data. In this work we systematically explored possible causes of the performance difference, aiming to determine whether it was a product of different cleaning procedures, limitations of the pre-trained models, the size of the base model, or the size of the training dataset, studying both directions of translation. Our studies, using two Brazilian Indigenous languages, related but with significant structural linguistic characteristics, indicated none or very limited influence from those training factors, suggesting differences between languages may play a significant role in the ability to produce translators by fine-tuning pre-trained models.
☆ Mapping Clinical Doubt: Locating Linguistic Uncertainty in LLMs AAAI'26
Srivarshinee Sridhar, Raghav Kaushik Ravi, Kripabandhu Ghosh
Large Language Models (LLMs) are increasingly used in clinical settings, where sensitivity to linguistic uncertainty can influence diagnostic interpretation and decision-making. Yet little is known about where such epistemic cues are internally represented within these models. Distinct from uncertainty quantification, which measures output confidence, this work examines input-side representational sensitivity to linguistic uncertainty in medical text. We curate a contrastive dataset of clinical statements varying in epistemic modality (e.g., 'is consistent with' vs. 'may be consistent with') and propose Model Sensitivity to Uncertainty (MSU), a layerwise probing metric that quantifies activation-level shifts induced by uncertainty cues. Our results show that LLMs exhibit structured, depth-dependent sensitivity to clinical uncertainty, suggesting that epistemic information is progressively encoded in deeper layers. These findings reveal how linguistic uncertainty is internally represented in LLMs, offering insight into their interpretability and epistemic reliability.
comment : Accepted to AAAI'26 SECURE-AI4H Workshop
☆ SuRe: Surprise-Driven Prioritised Replay for Continual LLM Learning
Hugo Hazard, Zafeirios Fountas, Martin A. Benfeghoul, Adnan Oomerjee, Jun Wang, Haitham Bou-Ammar
Continual learning, one's ability to adapt to a sequence of tasks without forgetting previously acquired knowledge, remains a major challenge in machine learning and a key gap between artificial and human intelligence. While regularisation and replay perform well in vision, they lag behind multi-task learning for large language models (LLMs), especially at scale with many tasks. We revisit replay and argue that two failure modes drive this gap: selection (what to rehearse) and integration (how to consolidate new knowledge). To address selection, we propose Surprise-prioritised Replay (SuRe), a simple, architecture-agnostic rule that ranks and stores the most surprising (high Negative Log-Likelihood) sequences. SuRe achieves state-of-the-art performance in the Large Number of Tasks (LNT) setting and delivers the best overall average across both Standard CL and LNT benchmarks. To address integration, we add a dual-learner design with fast and slow LoRA adapters merged via an exponential moving average (EMA), enabling rapid adaptation while stabilising long-term knowledge. Combining SuRe with the dual learner yields further gains, including improvements of up to +5 accuracy points on LNT over prior SOTA. Ablation studies confirm that our proposed method remains robust under reduced replay frequency and small buffer size, demonstrating both effectiveness and sample efficiency. Taken together, our results establish replay as a strong baseline for continual LLM fine-tuning and demonstrate that surprise-based selection and slow-weight consolidation are complementary components for mitigating catastrophic forgetting.
☆ PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel ASPLOS'26
Jinjun Yi, Zhixin Zhao, Yitao Hu, Ke Yan, Weiwei Sun, Hao Wang, Laiping Zhao, Yuhao Zhang, Wenxin Li, Keqiu Li
LLM serving is increasingly dominated by decode attention, which is a memory-bound operation due to massive KV cache loading from global memory. Meanwhile, real-world workloads exhibit substantial, hierarchical shared prefixes across requests (e.g., system prompts, tools/templates, RAG). Existing attention implementations fail to fully exploit prefix sharing: *one-query-per-CTA* execution repeatedly loads shared prefix KV cache, while *one-size-fits-all* tiling leaves on-chip resources idle and exacerbates bubbles for uneven KV lengths. These choices amplify memory bandwidth pressure and stall memory-bound decode attention. This paper introduces PAT, a prefix-aware attention kernel implementation for LLM decoding that organizes execution with a pack-forward-merge paradigm. PAT packs queries by shared prefix to reduce repeated memory accesses, runs a customized multi-tile kernel to achieve high resource efficiency. It further applies practical multi-stream forwarding and KV splitting to reduce resource bubbles. The final merge performs online softmax with negligible overhead. We implement PAT as an off-the-shelf plugin for vLLM. Evaluation on both real-world and synthetic workloads shows that PAT reduces attention latency by 67.4% on average and TPOT by 13.6-83.4% under the same configurations against state-of-the-art attention kernels.
comment : Accepted by ASPLOS'26
☆ Named Entity Recognition for the Kurdish Sorani Language: Dataset Creation and Comparative Analysis
Bakhtawar Abdalla, Rebwar Mala Nabi, Hassan Eshkiki, Fabio Caraffini
This work contributes towards balancing the inclusivity and global applicability of natural language processing techniques by proposing the first 'name entity recognition' dataset for Kurdish Sorani, a low-resource and under-represented language, that consists of 64,563 annotated tokens. It also provides a tool for facilitating this task in this and many other languages and performs a thorough comparative analysis, including classic machine learning models and neural systems. The results obtained challenge established assumptions about the advantage of neural approaches within the context of NLP. Conventional methods, in particular CRF, obtain F1-scores of 0.825, outperforming the results of BiLSTM-based models (0.706) significantly. These findings indicate that simpler and more computationally efficient classical frameworks can outperform neural architectures in low-resource settings.
☆ Sentiment Analysis Of Shopee Product Review s Using Distilbert
Zahri Aksa Dautd, Aviv Yuniar Rahman
The rapid growth of digital commerce has led to the accumulation of a massive number of consumer reviews on online platforms. Shopee, as one of the largest e-commerce platforms in Southeast Asia, receives millions of product reviews every day containing valuable information regarding customer satisfaction and preferences. Manual analysis of these reviews is inefficient, thus requiring a computational approach such as sentiment analysis. This study examines the use of DistilBERT, a lightweight transformer-based deep learning model, for sentiment classification on Shopee product reviews. The dataset used consists of approximately one million English-language reviews that have been preprocessed and trained using the distilbert-base-uncased model. Evaluation was conducted using accuracy, precision, recall, and F1-score metrics, and compared against benchmark models such as BERT and SVM. The results show that DistilBERT achieved an accuracy of 94.8%, slightly below BERT (95.3%) but significantly higher than SVM (90.2%), with computation time reduced by more than 55%. These findings demonstrate that DistilBERT provides an optimal balance between accuracy and efficiency, making it suitable for large scale sentiment analysis on e-commerce platforms. Keywords: Sentiment Analysis, DistilBERT, Shopee Reviews, Natural Language Processing, Deep Learning, Transformer Models.
comment : 6 pages, 11 figures
☆ Token-Level Marginalization for Multi-Label LLM Classifiers
Anjaneya Praharaj, Jaykumar Kasundra
This paper addresses the critical challenge of deriving interpretable confidence scores from generative language models (LLMs) when applied to multi-label content safety classification. While models like LLaMA Guard are effective for identifying unsafe content and its categories, their generative architecture inherently lacks direct class-level probabilities, which hinders model confidence assessment and performance interpretation. This limitation complicates the setting of dynamic thresholds for content moderation and impedes fine-grained error analysis. This research proposes and evaluates three novel token-level probability estimation approaches to bridge this gap. The aim is to enhance model interpretability and accuracy, and evaluate the generalizability of this framework across different instruction-tuned models. Through extensive experimentation on a synthetically generated, rigorously annotated dataset, it is demonstrated that leveraging token logits significantly improves the interpretability and reliability of generative classifiers, enabling more nuanced content safety moderation.
☆ Swarms of Large Language Model Agent s for Protein Sequence Design with Experimental Validation
Fiona Y. Wang, Di Sheng Lee, David L. Kaplan, Markus J. Buehler
Designing proteins de novo with tailored structural, physicochemical, and functional properties remains a grand challenge in biotechnology, medicine, and materials science, due to the vastness of sequence space and the complex coupling between sequence, structure, and function. Current state-of-the-art generative methods, such as protein language models (PLMs) and diffusion-based architectures, often require extensive fine-tuning, task-specific data, or model reconfiguration to support objective-directed design, thereby limiting their flexibility and scalability. To overcome these limitations, we present a decentralized, agent-based framework inspired by swarm intelligence for de novo protein design. In this approach, multiple large language model (LLM) agents operate in parallel, each assigned to a specific residue position. These agents iteratively propose context-aware mutations by integrating design objectives, local neighborhood interactions, and memory and feedback from previous iterations. This position-wise, decentralized coordination enables emergent design of diverse, well-defined sequences without reliance on motif scaffolds or multiple sequence alignments, validated with experiments on proteins with alpha helix and coil structures. Through analyses of residue conservation, structure-based metrics, and sequence convergence and embeddings, we demonstrate that the framework exhibits emergent behaviors and effective navigation of the protein fitness landscape. Our method achieves efficient, objective-directed designs within a few GPU-hours and operates entirely without fine-tuning or specialized training, offering a generalizable and adaptable solution for protein design. Beyond proteins, the approach lays the groundwork for collective LLM-driven design across biomolecular systems and other scientific discovery tasks.
☆ Beyond Query-Level Comparison: Fine-Grained Reinforcement Learning for Text-to-SQL with Automated Interpretable Critiques
Guifeng Wang, Yuanfeng Song, Meng Yang, Tao Zhu, Xiaoming Yin, Xing Chen
Text-to-SQL, a pivotal natural language processing (NLP) task that converts textual queries into executable SQL, has seen substantial progress in recent years. However, existing evaluation and reward mechanisms used to train and assess the text-to-SQL models remain a critical bottleneck. Current approaches heavily rely on manually annotated gold SQL queries, which are costly to produce and impractical for large-scale evaluation. More importantly, most reinforcement learning (RL) methods in text-to-SQL leverage only the final binary execution outcome as the reward signal, a coarse-grained supervision that overlooks detailed structural and semantic errors from the perspective of rubrics. To address these challenges, we propose RuCo-C, a novel generative judge model for fine-grained, query-specific automatic evaluation using interpretable critiques without human intervention. Our framework first automatically generates query-specific evaluation rubrics for human-free annotation, linking them to interpretable critiques. Subsequently, it integrates densified reward feedback through a "progressive exploration" strategy during the RL training process, which dynamically adjusts the rewards to enhance the model's performance. Comprehensive experiments demonstrate that RuCo-C outperforms existing methods in text-to-SQL evaluation, yielding significant performance gains.
☆ From Compound Figures to Composite Understanding : Developing a Multi-Modal LLM from Biomedical Literature with Medical Multiple-Image Benchmark ing and Validation
Zhen Chen, Yihang Fu, Gabriel Madera, Mauro Giuffre, Serina Applebaum, Hyunjae Kim, Hua Xu, Qingyu Chen
Multi-modal large language models (MLLMs) have shown promise in advancing healthcare. However, most existing models remain confined to single-image understanding, which greatly limits their applicability in clinical workflows. In practice, medical diagnosis and progression often require synthesizing information across multiple images from different modalities or time points. The development of medical MLLMs capable of such multi-image understanding has been hindered by the lack of large-scale, high-quality annotated training data. To address this limitation, we propose a novel framework that leverages license-permissive compound images in biomedical literature, as a rich yet underutilized data source for multi-image analysis. Specifically, we design a five-stage, context-aware instruction generation paradigm underpinned by a divide-and-conquer strategy. By decomposing multi-image analysis into manageable sub-tasks, this paradigm empowers MLLMs to move beyond single-panel analysis and provide a composite understanding by learning the complex spatial, temporal, and cross-modal relationships inherent in these compound figures. By parsing over 237,000 compound figures and their contextual text for instruction generation, we develop M3LLM, a medical multi-image multi-modal large language model. For benchmarking, we construct PMC-MI-Bench for composite understanding, manually validated by medical experts. Extensive experiments show that M3LLM significantly outperforms both general-purpose and specialized medical MLLMs across multi-image, single-image, text-only, and multi-choice scenarios. Notably, M3LLM exhibits strong generalization to longitudinal chest X-ray analysis using the MIMIC dataset. This work establishes a scalable and efficient paradigm for developing medical MLLMs capable of composite reasoning, bridging the gap between biomedical literature and real-world clinical applications.
☆ Focused Chain-of-Thought: Efficient LLM Reasoning via Structured Input Information
Lukas Struppek, Dominik Hintersdorf, Hannah Struppek, Daniel Neider, Kristian Kersting
Recent large language models achieve strong reasoning performance by generating detailed chain-of-thought traces, but this often leads to excessive token use and high inference latency. Existing efficiency approaches typically focus on model-centric interventions, such as reinforcement learning or supervised fine-tuning, to reduce verbosity. In contrast, we propose a training-free, input-centric approach. Inspired by cognitive psychology, we introduce Focused Chain-of-Thought (F-CoT), which separates information extraction from the reasoning process. F-CoT first organizes the essential information from a query into a concise, structured context and then guides the model to reason exclusively over this context. By preventing attention to irrelevant details, F-CoT naturally produces shorter reasoning paths. On arithmetic word problems, F-CoT reduces generated tokens by 2-3x while maintaining accuracy comparable to standard zero-shot CoT. These results highlight structured input as a simple yet effective lever for more efficient LLM reasoning.
☆ RefineBench: Evaluating Refinement Capability of Language Models via Checklists
Young-Jun Lee, Seungone Kim, Byung-Kwan Lee, Minkyeong Moon, Yechan Hwang, Jong Myoung Kim, Graham Neubig, Sean Welleck, Ho-Jin Choi
Can language models (LMs) self-refine their own responses? This question is increasingly relevant as a wide range of real-world user interactions involve refinement requests. However, prior studies have largely tested LMs' refinement abilities on verifiable tasks such as competition math or symbolic reasoning with simplified scaffolds, whereas users often pose open-ended queries and provide varying degrees of feedback on what they desire. The recent advent of reasoning models that exhibit self-reflection patterns in their chains-of-thought further motivates this question. To analyze this, we introduce RefineBench, a benchmark of 1,000 challenging problems across 11 domains paired with a checklist-based evaluation framework. We evaluate two refinement modes: (1) guided refinement, where an LM is provided natural language feedback, and (2) self-refinement, where LMs attempt to improve without guidance. In the self-refinement setting, even frontier LMs such as Gemini 2.5 Pro and GPT-5 achieve modest baseline scores of 31.3% and 29.1%, respectively, and most models fail to consistently improve across iterations (e.g., Gemini-2.5-Pro gains only +1.8%, while DeepSeek-R1 declines by -0.1%). By contrast, in guided refinement, both proprietary LMs and large open-weight LMs (>70B) can leverage targeted feedback to refine responses to near-perfect levels within five turns. These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses, and that RefineBench provides a valuable testbed for tracking progress.
comment : Project website: https://passing2961.github.io/refinebench-page/
☆ Lips-Jaw and Tongue-Jaw Articulatory Tradeoff in DYNARTmo
Bernd J. Kröger
This paper investigates how the dynamic articulatory model DYNARTmo accounts for articulatory tradeoffs between primary and secondary articulators, with a focus on lips-jaw and tongue-jaw coordination. While DYNARTmo does not implement full task-dynamic second-order biomechanics, it adopts first-order task-space gesture specifications comparable to those used in articulatory phonology and integrates a simplified mechanism for distributing articulatory effort across multiple articulators. We first outline the conceptual relationship between task dynamics and DYNARTmo, emphasizing the distinction between high-level task-space trajectories and their low-level articulatory execution. We then present simulation results for a set of CV syllables that illustrate how jaw displacement varies as a function of both place of articulation (labial, apical, dorsal) and vowel context (/a/, /i/, /u/). The model reproduces empirically attested patterns of articulatory synergy, including jaw-supported apical closures, lower-lip elevation in bilabial stops, tongue-jaw co-movement, and saturation effects in labial constrictions. These results demonstrate that even with computationally simplified assumptions, DYNARTmo can generate realistic spatio-temporal movement patterns that capture key aspects of articulatory tradeoff and synergy across a range of consonant-vowel combinations.
comment : 12 pages, 3 figures, supplementary material: python code
☆ A Theoretically Grounded Hybrid Ensemble for Reliable Detection of LLM-Generated Text
Sepyan Purnama Kristanto, Lutfi Hakim
The rapid proliferation of Large Language Models (LLMs) has blurred the line between human and machine authorship, creating practical risks for academic integrity and information reliability. Existing text detectors typically rely on a single methodological paradigm and suffer from poor generalization and high false positive rates (FPR), especially on high-stakes academic text. We propose a theoretically grounded hybrid ensemble that systematically fuses three complementary detection paradigms: (i) a RoBERTa-based transformer classifier for deep semantic feature extraction, (ii) a GPT-2-based probabilistic detector using perturbation-induced likelihood curvature, and (iii) a statistical linguistic feature analyzer capturing stylometric patterns. The core novelty lies in an optimized weighted voting framework, where ensemble weights are learned on the probability simplex to maximize F1-score rather than set heuristically. We provide a bias-variance analysis and empirically demonstrate low inter-model correlation (rho ~ 0.35-0.42), a key condition for variance reduction. Evaluated on a large-scale, multigenerator corpus of 30,000 documents, our system achieves 94.2% accuracy and an AUC of 0.978, with a 35% relative reduction in false positives on academic text. This yields a more reliable and ethically responsible detector for real-world deployment in education and other high-stakes domains.
comment : 24 pages
☆ From Topology to Retrieval: Decoding Embedding Spaces with Unified Signatures
Florian Rottach, William Rudman, Bastain Rieck, Harrisen Scells, Carsten Eickhoff
Studying how embeddings are organized in space not only enhances model interpretability but also uncovers factors that drive downstream task performance. In this paper, we present a comprehensive analysis of topological and geometric measures across a wide set of text embedding models and datasets. We find a high degree of redundancy among these measures and observe that individual metrics often fail to sufficiently differentiate embedding spaces. Building on these insights, we introduce Unified Topological Signatures (UTS), a holistic framework for characterizing embedding spaces. We show that UTS can predict model-specific properties and reveal similarities driven by model architecture. Further, we demonstrate the utility of our method by linking topological structure to ranking effectiveness and accurately predicting document retrievability. We find that a holistic, multi-attribute perspective is essential to understanding and leveraging the geometry of text embeddings.
☆ C$^2$DLM: Causal Concept-Guided Diffusion Large Language Models
Kairong Han, Nuanqiao Shan, Ziyu Zhao, Zijing Hu, Xinpeng Dong, Junjian Ye, Lujia Pan, Fei Wu, Kun Kuang
Autoregressive (AR) language models and Diffusion Language Models (DLMs) constitute the two principal paradigms of large language models. However, both paradigms suffer from insufficient reasoning capabilities. Human reasoning inherently relies on causal knowledge and thought, which are reflected in natural language. But in the AR paradigm, language is modeled as next token prediction (a strictly left-to-right, token-by-token order), whereas natural language itself exhibits more flexible causal structures. In the DLM paradigm, the attention mechanism is fully connected, which entirely disregards causal order. To fill this gap, we propose a \underline{\textbf{C}}ausal \underline{\textbf{C}}oncept-Guided \underline{\textbf{D}}iffusion \underline{\textbf{L}}anguage \underline{\textbf{M}}odel (C$^2$DLM). Starting from DLM's fully connected attention, C$^2$DLM first obtains a concept-level causal graph from the teacher model, and then explicitly guides attention to learn causal relationships between concepts. By focusing on causal relationships and avoiding interference from difficult subgoals involving causal inversion, C$^2$DLM improves 12\% with about 3.2 times training speedup in the COT-OrderPerturb task, and achieves an average gain of 1.31\% across six downstream reasoning tasks. More details in the repository ~\href{https://github.com/Kairong-Han/C-2-DLM}{here}.
☆ Bridging the Modality Gap by Similarity Standardization with Pseudo-Positive Samples ACL
Shuhei Yamashita, Daiki Shirafuji, Tatsuhiko Saito
Advances in vision-language models (VLMs) have enabled effective cross-modality retrieval. However, when both text and images exist in the database, similarity scores would differ in scale by modality. This phenomenon, known as the modality gap, hinders accurate retrieval. Most existing studies address this issue with manually labeled data, e.g., by fine-tuning VLMs on them. In this work, we propose a similarity standardization approach with pseudo data construction. We first compute the mean and variance of the similarity scores between each query and its paired data in text or image modality. Using these modality-specific statistics, we standardize all similarity scores to compare on a common scale across modalities. These statistics are calculated from pseudo pairs, which are constructed by retrieving the text and image candidates with the highest cosine similarity to each query. We evaluate our method across seven VLMs using two multi-modal QA benchmarks (MMQA and WebQA), where each question requires retrieving either text or image data. Our experimental results show that our method significantly improves retrieval performance, achieving average Recall@20 gains of 64% on MMQA and 28% on WebQA when the query and the target data belong to different modalities. Compared to E5-V, which addresses the modality gap through image captioning, we confirm that our method more effectively bridges the modality gap.
comment : Accepted to PACLIC2025
☆ A Hybrid Theory and Data-driven Approach to Persuasion Detection with Large Language Models
Gia Bao Hoang, Keith J Ransom, Rachel Stephens, Carolyn Semmler, Nicolas Fay, Lewis Mitchell
Traditional psychological models of belief revision focus on face-to-face interactions, but with the rise of social media, more effective models are needed to capture belief revision at scale, in this rich text-based online discourse. Here, we use a hybrid approach, utilizing large language models (LLMs) to develop a model that predicts successful persuasion using features derived from psychological experiments. Our approach leverages LLM generated ratings of features previously examined in the literature to build a random forest classification model that predicts whether a message will result in belief change. Of the eight features tested, \textit{epistemic emotion} and \textit{willingness to share} were the top-ranking predictors of belief change in the model. Our findings provide insights into the characteristics of persuasive messages and demonstrate how LLMs can enhance models of successful persuasion based on psychological theory. Given these insights, this work has broader applications in fields such as online influence detection and misinformation mitigation, as well as measuring the effectiveness of online narratives.
☆ Ea rl y Risk Prediction with Temporally and Context ually Grounded Clinical Language Processing
Rochana Chaturvedi, Yue Zhou, Andrew Boyd, Brian T. Layden, Mudassir Rashid, Lu Cheng, Ali Cinar, Barbara Di Eugenio
Clinical notes in Electronic Health Records (EHRs) capture rich temporal information on events, clinician reasoning, and lifestyle factors often missing from structured data. Leveraging them for predictive modeling can be impactful for timely identification of chronic diseases. However, they present core natural language processing (NLP) challenges: long text, irregular event distribution, complex temporal dependencies, privacy constraints, and resource limitations. We present two complementary methods for temporally and contextually grounded risk prediction from longitudinal notes. First, we introduce HiTGNN, a hierarchical temporal graph neural network that integrates intra-note temporal event structures, inter-visit dynamics, and medical knowledge to model patient trajectories with fine-grained temporal granularity. Second, we propose ReVeAL, a lightweight, test-time framework that distills the reasoning of large language models into smaller verifier models. Applied to opportunistic screening for Type 2 Diabetes (T2D) using temporally realistic cohorts curated from private and public hospital corpora, HiTGNN achieves the highest predictive accuracy, especially for near-term risk, while preserving privacy and limiting reliance on large proprietary models. ReVeAL enhances sensitivity to true T2D cases and retains explanatory reasoning. Our ablations confirm the value of temporal structure and knowledge augmentation, and fairness analysis shows HiTGNN performs more equitably across subgroups.
☆ ResearchArcade: Graph Interface for Academic Tasks
Jingjun Xu, Chongshan Lin, Haofei Yu, Tao Feng, Jiaxuan You
Academic research generates diverse data sources, and as researchers increasingly use machine learning to assist research tasks, a crucial question arises: Can we build a unified data interface to support the development of machine learning models for various academic tasks? Models trained on such a unified interface can better support human researchers throughout the research process, eventually accelerating knowledge discovery. In this work, we introduce ResearchArcade, a graph-based interface that connects multiple academic data sources, unifies task definitions, and supports a wide range of base models to address key academic challenges. ResearchArcade utilizes a coherent multi-table format with graph structures to organize data from different sources, including academic corpora from ArXiv and peer reviews from OpenReview, while capturing information with multiple modalities, such as text, figures, and tables. ResearchArcade also preserves temporal evolution at both the manuscript and community levels, supporting the study of paper revisions as well as broader research trends over time. Additionally, ResearchArcade unifies diverse academic task definitions and supports various models with distinct input requirements. Our experiments across six academic tasks demonstrate that combining cross-source and multi-modal information enables a broader range of tasks, while incorporating graph structures consistently improves performance over baseline methods. This highlights the effectiveness of ResearchArcade and its potential to advance research progress.
☆ AfriStereo: A Culturally Grounded Dataset for Evaluating Stereotypical Bias in Large Language Models
Yann Le Beux, Oluchi Audu, Oche D. Ankeli, Dhananjay Balakrishnan, Melissah Weya, Marie D. Ralaiarinosy, Ignatius Ezeani
Existing AI bias evaluation benchmarks largely reflect Western perspectives, leaving African contexts underrepresented and enabling harmful stereotypes in applications across various domains. To address this gap, we introduce AfriStereo, the first open-source African stereotype dataset and evaluation framework grounded in local socio-cultural contexts. Through community engaged efforts across Senegal, Kenya, and Nigeria, we collected 1,163 stereotypes spanning gender, ethnicity, religion, age, and profession. Using few-shot prompting with human-in-the-loop validation, we augmented the dataset to over 5,000 stereotype-antistereotype pairs. Entries were validated through semantic clustering and manual annotation by culturally informed reviewers. Preliminary evaluation of language models reveals that nine of eleven models exhibit statistically significant bias, with Bias Preference Ratios (BPR) ranging from 0.63 to 0.78 (p <= 0.05), indicating systematic preferences for stereotypes over antistereotypes, particularly across age, profession, and gender dimensions. Domain-specific models appeared to show weaker bias in our setup, suggesting task-specific training may mitigate some associations. Looking ahead, AfriStereo opens pathways for future research on culturally grounded bias evaluation and mitigation, offering key methodologies for the AI community on building more equitable, context-aware, and globally inclusive NLP technologies.
☆ Reversing Large Language Models for Efficient Training and Fine-Tuning
Eshed Gal, Moshe Eliasof, Javier Turek, Uri Ascher, Eran Treister, Eldad Haber
Large Language Models (LLMs) are known for their expensive and time-consuming training. Thus, oftentimes, LLMs are fine-tuned to address a specific task, given the pretrained weights of a pre-trained LLM considered a foundation model. In this work, we introduce memory-efficient, reversible architectures for LLMs, inspired by symmetric and symplectic differential equations, and investigate their theoretical properties. Different from standard, baseline architectures that store all intermediate activations, the proposed models use time-reversible dynamics to retrieve hidden states during backpropagation, relieving the need to store activations. This property allows for a drastic reduction in memory consumption, allowing for the processing of larger batch sizes for the same available memory, thereby offering improved throughput. In addition, we propose an efficient method for converting existing, non-reversible LLMs into reversible architectures through fine-tuning, rendering our approach practical for exploiting existing pre-trained models. Our results show comparable or improved performance on several datasets and benchmarks, on several LLMs, building a scalable and efficient path towards reducing the memory and computational costs associated with both training from scratch and fine-tuning of LLMs.
♻ ☆ A Trio Neural Model for Dynamic Entity Relatedness Ranking CoNLL 2018
Tu Nguyen, Tuan Tran, Wolfgang Nejdl
Measuring entity relatedness is a fundamental task for many natural language processing and information retrieval applications. Prior work often studies entity relatedness in static settings and an unsupervised manner. However, entities in real-world are often involved in many different relationships, consequently entity-relations are very dynamic over time. In this work, we propose a neural networkbased approach for dynamic entity relatedness, leveraging the collective attention as supervision. Our model is capable of learning rich and different entity representations in a joint framework. Through extensive experiments on large-scale datasets, we demonstrate that our method achieves better results than competitive baselines.
comment : In Proceedings of CoNLL 2018
♻ ☆ Automated Composition of Agent s: A Knapsack Approach for Agent ic Component Selection NeurIPS 2025
Michelle Yuan, Khushbu Pahwa, Shuaichen Chang, Mustafa Kaba, Jiarong Jiang, Xiaofei Ma, Yi Zhang, Monica Sunkara
Designing effective agentic systems requires the seamless composition and integration of agents, tools, and models within dynamic and uncertain environments. Most existing methods rely on static, semantic retrieval approaches for tool or agent discovery. However, effective reuse and composition of existing components remain challenging due to incomplete capability descriptions and the limitations of retrieval methods. Component selection suffers because the decisions are not based on capability, cost, and real-time utility. To address these challenges, we introduce a structured, automated framework for agentic system composition that is inspired by the knapsack problem. Our framework enables a composer agent to systematically identify, select, and assemble an optimal set of agentic components by jointly considering performance, budget constraints, and compatibility. By dynamically testing candidate components and modeling their utility in real-time, our approach streamlines the assembly of agentic systems and facilitates scalable reuse of resources. Empirical evaluation with Claude 3.5 Sonnet across five benchmarking datasets shows that our online-knapsack-based composer consistently lies on the Pareto frontier, achieving higher success rates at significantly lower component costs compared to our baselines. In the single-agent setup, the online knapsack composer shows a success rate improvement of up to 31.6% in comparison to the retrieval baselines. In multi-agent systems, the online knapsack composer increases success rate from 37% to 87% when agents are selected from an agent inventory of 100+ agents. The substantial performance gap confirms the robust adaptability of our method across diverse domains and budget constraints.
comment : Accepted to NeurIPS 2025 Conference
♻ ☆ Linguistically-Controlled Paraphrase Generation ACL
Mohamed Elgaar, Hadi Amiri
Controlled paraphrase generation produces paraphrases that preserve meaning while allowing precise control over linguistic attributes of the output. We introduce LingConv, an encoder-decoder framework that enables fine-grained control over 40 linguistic attributes in English. To improve reliability, we introduce a novel inference-time quality control mechanism that iteratively refines attribute embeddings to generate paraphrases that closely match target attributes without sacrificing semantic fidelity. LingConv reduces attribute error by up to 34% over existing models, with the quality control mechanism contributing an additional 14% improvement.
comment : This paper was published in Findings of ACL: EMNLP 2025
♻ ☆ One Patient, Many Context s: Scaling Medical AI with Context ual Intelligence
Michelle M. Li, Ben Y. Reis, Adam Rodman, Tianxi Cai, Noa Dagan, Ran D. Balicer, Joseph Loscalzo, Isaac S. Kohane, Marinka Zitnik
Medical AI, including clinical language models, vision-language models, and multimodal health record models, already summarizes notes, answers questions, and supports decisions. Their adaptation to new populations, specialties, or care settings often relies on fine-tuning, prompting, or retrieval from external knowledge bases. These strategies can scale poorly and risk contextual errors: outputs that appear plausible but miss critical patient or situational information. We envision context switching as a solution. Context switching adjusts model reasoning at inference without retraining. Generative models can tailor outputs to patient biology, care setting, or disease. Multimodal models can reason on notes, laboratory results, imaging, and genomics, even when some data are missing or delayed. Agent models can coordinate tools and roles based on tasks and users. In each case, context switching enables medical AI to adapt across specialties, populations, and geographies. It requires advances in data design, model architectures, and evaluation frameworks, and establishes a foundation for medical AI that scales to infinitely many contexts while remaining reliable and suited to real-world care.
♻ ☆ Do Large Language Models Think Like the Brain? Sentence-Level Evidence from fMRI and Hierarchical Embeddings
Yu Lei, Xingyang Ge, Yi Zhang, Yiming Yang, Bolei Ma
Understanding whether large language models (LLMs) and the human brain converge on similar computational principles remains a fundamental and important question in cognitive neuroscience and AI. Do the brain-like patterns observed in LLMs emerge simply from scaling, or do they reflect deeper alignment with the architecture of human language processing? This study focuses on the sentence-level neural mechanisms of language models, systematically investigating how hierarchical representations in LLMs align with the dynamic neural responses during human sentence comprehension. By comparing hierarchical embeddings from 14 publicly available LLMs with fMRI data collected from participants, who were exposed to a naturalistic narrative story, we constructed sentence-level neural prediction models to precisely identify the model layers most significantly correlated with brain region activations. Results show that improvements in model performance drive the evolution of representational architectures toward brain-like hierarchies, particularly achieving stronger functional and anatomical correspondence at higher semantic abstraction levels.
♻ ☆ AutoHall: Automated Factuality Hallucination Dataset Generation for Large Language Models
Zouying Cao, Yifei Yang, XiaoJing Li, Hai Zhao
Large language models (LLMs) have gained broad applications across various domains but still struggle with hallucinations. Currently, hallucinations occur frequently in the generation of factual content and pose a great challenge to trustworthy LLMs. However, hallucination detection is hindered by the laborious and expensive manual annotation of hallucinatory content. Meanwhile, as different LLMs exhibit distinct types and rates of hallucination, the collection of hallucination datasets is inherently model-specific, which also increases the cost. To address this issue, this paper proposes a method called $\textbf{AutoHall}$ for $\underline{Auto}$matically constructing model-specific $\underline{Hall}$ucination datasets based on existing fact-checking datasets. The empirical results reveal variations in hallucination proportions and types among different models. Moreover, we introduce a zero-resource and black-box hallucination detection method based on self-contradiction to recognize the hallucination in our constructed dataset, achieving superior detection performance compared to baselines. Further analysis on our dataset provides insight into factors that may contribute to LLM hallucinations. Our codes and datasets are publicly available at https://github.com/zouyingcao/AutoHall.
comment : Accepted by IEEE Transactions on Audio, Speech, and Language Processing (TASLP)
♻ ☆ Event Stream-based Sign Language Translation: A High-Definition Benchmark Dataset and A Novel Baseline
Shiao Wang, Xiao Wang, Duoqing Yang, Yao Rong, Fuling Wang, Jianing Li, Lin Zhu, Bo Jiang
Sign Language Translation (SLT) is a core task in the field of AI-assisted disability. Traditional SLT methods are typically based on visible light videos, which are easily affected by factors such as lighting variations, rapid hand movements, and privacy concerns. This paper proposes the use of bio-inspired event cameras to alleviate the aforementioned issues. Specifically, we introduce a new high-definition event-based sign language dataset, termed Event-CSL, which effectively addresses the data scarcity in this research area. The dataset comprises 14,827 videos, 14,821 glosses, and 2,544 Chinese words in the text vocabulary. These samples are collected across diverse indoor and outdoor scenes, covering multiple viewpoints, lighting conditions, and camera motions. We have also benchmarked existing mainstream SLT methods on this dataset to facilitate fair comparisons in future research.Furthermore, we propose a novel event-based sign language translation framework, termed EvSLT. The framework first segments continuous video features into clips and employs a Mamba-based memory aggregation module to compress and aggregate spatial detail features at the clip level. Subsequently, these spatial features, along with temporal representations obtained from temporal convolution, are then fused by a graph-guided spatiotemporal fusion module. Extensive experiments on Event-CSL, as well as other publicly available datasets, demonstrate the superior performance of our method. The dataset and source code will be released on https://github.com/Event-AHU/OpenESL
♻ ☆ Self-Guided Defense: Adaptive Safety Alignment for Reasoning Models via Synthesized Guidelines
Yuhang Wang, Yanxu Zhu, Dongyuan Lu, Jitao Sang
Reasoning models have demonstrated remarkable capabilities in complex reasoning tasks. However, ensuring their safety against adversarial jailbreak prompts remains a critical challenge. Due to the covert and deceptive nature of such prompts, they can often evade built-in safety mechanisms and lead to the generation of harmful content. This underscores the need for an adaptive safety alignment approach that enables models to autonomously reinforce their defenses in response to adversarial inputs. This paper introduces the Synthesized Guideline-based Adaptive Safety Alignment (SGASA) framework, which internalizes model-generated safety guidelines to strengthen models' ability to enhance robustness against harmful adversarial prompts while minimizing unnecessary refusals of benign requests. SGASA consists of two key stages: Data Pre-synthesis, which generates safety guidelines and augmented prompts; and Alignment Fine-tuning, which leverages Supervised Fine-tuning (SFT) and Direct Preference Optimization (DPO) to embed these guidelines into the model. Extensive experiments across multiple datasets demonstrate that SGASA significantly improves model safety, validating its adaptive and scalable effectiveness.
♻ ☆ KSHSeek: Data-Driven Approaches to Mitigating and Detecting Knowledge-Shortcut Hallucinations in Generative Models
Zhongxin Liu, Zhiwei Wang, Jun Niu, Ying Li, Hongyu Sun, Meng Xu, He Wang, Gaofei Wu, Yuqing Zhang
The emergence of large language models (LLMs) has significantly advanced the development of natural language processing (NLP), especially in text generation tasks like question answering. However, model hallucinations remain a major challenge in natural language generation (NLG) tasks due to their complex causes. We systematically expand on the causes of factual hallucinations from the perspective of knowledge shortcuts, analyzing hallucinations arising from correct and defect-free data and demonstrating that knowledge-shortcut hallucinations are prevalent in generative models. To mitigate this issue, we propose a high similarity pruning algorithm at the data preprocessing level to reduce spurious correlations in the data. Additionally, we design a specific detection method for knowledge-shortcut hallucinations to evaluate the effectiveness of our mitigation strategy. Experimental results show that our approach effectively reduces knowledge-shortcut hallucinations, particularly in fine-tuning tasks, without negatively impacting model performance in question answering. This work introduces a new paradigm for mitigating specific hallucination issues in generative models, enhancing their robustness and reliability in real-world applications.
comment : 16 pages, 34 figures
♻ ☆ FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models
Yan Gao, Massimo Roberto Scamarcia, Javier Fernandez-Marques, Mohammad Naseri, Chong Shen Ng, Dimitris Stripelis, Zexi Li, Tao Shen, Jiamu Bai, Daoyuan Chen, Zikai Zhang, Rui Hu, InSeo Song, Lee KangYoon, Hong Jia, Ting Dang, Junyan Wang, Zheyuan Liu, Daniel Janes Beutel, Lingjuan Lyu, Nicholas D. Lane
Large Language Models (LLMs) have achieved state-of-the-art results across diverse domains, yet their development remains reliant on vast amounts of publicly available data, raising concerns about data scarcity and the lack of access to domain-specific, sensitive information. Federated Learning (FL) presents a compelling framework to address these challenges by enabling decentralized fine-tuning on pre-trained LLMs without sharing raw data. However, the compatibility and performance of pre-trained LLMs in FL settings remain largely under explored. We introduce the FlowerTune LLM Leaderboard, a first-of-its-kind benchmarking suite designed to evaluate federated fine-tuning of LLMs across four diverse domains: general NLP, finance, medical, and coding. Each domain includes federated instruction-tuning datasets and domain-specific evaluation metrics. Our results, obtained through a collaborative, open-source and community-driven approach, provide the first comprehensive comparison across 26 pre-trained LLMs with different aggregation and fine-tuning strategies under federated settings, offering actionable insights into model performance, resource constraints, and domain adaptation. This work lays the foundation for developing privacy-preserving, domain-specialized LLMs for real-world applications.
♻ ☆ IROTE: Human-like Traits Elicitation of Large Language Model via In- Context Self-Reflective Optimization AAAI 2026
Yuzhuo Bai, Shitong Duan, Muhua Huang, Jing Yao, Zhenghao Liu, Peng Zhang, Tun Lu, Xiaoyuan Yi, Maosong Sun, Xing Xie
Trained on various human-authored corpora, Large Language Models (LLMs) have demonstrated a certain capability of reflecting specific human-like traits (e.g., personality or values) by prompting, benefiting applications like personalized LLMs and social simulations. However, existing methods suffer from the superficial elicitation problem: LLMs can only be steered to mimic shallow and unstable stylistic patterns, failing to embody the desired traits precisely and consistently across diverse tasks like humans. To address this challenge, we propose IROTE, a novel in-context method for stable and transferable trait elicitation. Drawing on psychological theories suggesting that traits are formed through identity-related reflection, our method automatically generates and optimizes a textual self-reflection within prompts, which comprises self-perceived experience, to stimulate LLMs' trait-driven behavior. The optimization is performed by iteratively maximizing an information-theoretic objective that enhances the connections between LLMs' behavior and the target trait, while reducing noisy redundancy in reflection without any fine-tuning, leading to evocative and compact trait reflection. Extensive experiments across three human trait systems manifest that one single IROTE-generated self-reflection can induce LLMs' stable impersonation of the target trait across diverse downstream tasks beyond simple questionnaire answering, consistently outperforming existing strong baselines.
comment : This paper is accepted by AAAI 2026
♻ ☆ Masked Diffusion Models as Energy Minimization
Sitong Chen, Shen Nie, Jiacheng Sun, Zijin Feng, Zhenguo Li, Ji-Rong Wen, Chongxuan Li
We present a systematic theoretical framework that interprets masked diffusion models (MDMs) as solutions to energy minimization problems in discrete optimal transport. Specifically, we prove that three distinct energy formulations--kinetic, conditional kinetic, and geodesic energy--are mathematically equivalent under the structure of MDMs, and that MDMs minimize all three when the mask schedule satisfies a closed-form optimality condition. This unification not only clarifies the theoretical foundations of MDMs, but also motivates practical improvements in sampling. By parameterizing interpolation schedules via Beta distributions, we reduce the schedule design space to a tractable 2D search, enabling efficient post-training tuning without model modification. Experiments on synthetic and real-world benchmarks demonstrate that our energy-inspired schedules outperform hand-crafted baselines, particularly in low-step sampling settings.
♻ ☆ Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning NeurIPS 2025
Haolei Xu, Yuchen Yan, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Shengpei Jiang, Kaitao Song, Weiming Lu, Jun Xiao, Yueting Zhuang
Large language models (LLMs) have achieved remarkable progress on mathematical tasks through Chain-of-Thought (CoT) reasoning. However, existing mathematical CoT datasets often suffer from Thought Leaps due to experts omitting intermediate steps, which negatively impacts model learning and generalization. We propose the CoT Thought Leap Bridge Task, which aims to automatically detect leaps and generate missing intermediate reasoning steps to restore the completeness and coherence of CoT. To facilitate this, we constructed a specialized training dataset called ScaleQM+, based on the structured ScaleQuestMath dataset, and trained CoT-Bridge to bridge thought leaps. Through comprehensive experiments on mathematical reasoning benchmarks, we demonstrate that models fine-tuned on bridged datasets consistently outperform those trained on original datasets, with improvements of up to +5.87% on NuminaMath. Our approach effectively enhances distilled data (+3.02%) and provides better starting points for reinforcement learning (+3.1%), functioning as a plug-and-play module compatible with existing optimization techniques. Furthermore, CoT-Bridge demonstrate improved generalization to out-of-domain logical reasoning tasks, confirming that enhancing reasoning completeness yields broadly applicable benefits.
comment : Accepted to NeurIPS 2025. Camera ready version. Code: https://github.com/ZJU-REAL/Mind-the-Gap Project: https://zju-real.github.io/CoT-Bridge/
♻ ☆ Agent ar-Scale-SQL: Advancing Text-to-SQL through Orchestrated Test-Time Scaling
Pengfei Wang, Baolin Sun, Xuemei Dong, Yaxun Dai, Hongwei Yuan, Mengdie Chu, Yingqi Gao, Xiang Qi, Peng Zhang, Ying Yan
State-of-the-art (SOTA) Text-to-SQL methods still lag significantly behind human experts on challenging benchmarks like BIRD. Current approaches that explore test-time scaling lack an orchestrated strategy and neglect the model's internal reasoning process. To bridge this gap, we introduce Agentar-Scale-SQL, a novel framework leveraging scalable computation to improve performance. Agentar-Scale-SQL implements an Orchestrated Test-Time Scaling strategy that synergistically combines three distinct perspectives: i) Internal Scaling via RL-enhanced Intrinsic Reasoning, ii) Sequential Scaling through Iterative Refinement, and iii) Parallel Scaling using Diverse Synthesis and Tournament Selection. Agentar-Scale-SQL is a general-purpose framework designed for easy adaptation to new databases and more powerful language models. Extensive experiments show that Agentar-Scale-SQL achieves SOTA performance on the BIRD benchmark, reaching 81.67% execution accuracy on the test set and ranking first on the official leaderboard, demonstrating an effective path toward human-level performance.
♻ ☆ From Perception to Reasoning : Deep Thinking Empowers Multimodal Large Language Models
Wenxin Zhu, Andong Chen, Yuchen Song, Kehai Chen, Conghui Zhu, Ziyan Chen, Tiejun Zhao
With the remarkable success of Multimodal Large Language Models (MLLMs) in perception tasks, enhancing their complex reasoning capabilities has emerged as a critical research focus. Existing models still suffer from challenges such as opaque reasoning paths and insufficient generalization ability. Chain-of-Thought (CoT) reasoning, which has demonstrated significant efficacy in language models by enhancing reasoning transparency and output interpretability, holds promise for improving model reasoning capabilities when extended to the multimodal domain. This paper provides a systematic review centered on "Multimodal Chain-of-Thought" (MCoT). First, it analyzes the background and theoretical motivations for its inception from the perspectives of technical evolution and task demands. Then, it introduces mainstream MCoT methods from three aspects: CoT paradigms, the post-training stage, and the inference stage, while also analyzing their underlying mechanisms. Furthermore, the paper summarizes existing evaluation benchmarks and metrics, and discusses the application scenarios of MCoT. Finally, it analyzes the challenges currently facing MCoT and provides an outlook on its future research directions.
comment : Survey; 7 figures, 3 tables, 44 pages
♻ ☆ MCTS -SQL: Light-Weight LLMs can Master the Text-to-SQL through Monte Ca rl o Tree Search AAAI 2026
Shuozhi Yuan, Limin Chen, Miaomiao Yuan, Zhao Jin
Text-to-SQL is a fundamental yet challenging task in the NLP area, aiming at translating natural language questions into SQL queries. While recent advances in large language models have greatly improved performance, most existing approaches depend on models with tens of billions of parameters or costly APIs, limiting their applicability in resource-constrained environments. For real world, especially on edge devices, it is crucial for Text-to-SQL to ensure cost-effectiveness. Therefore, enabling the light-weight models for Text-to-SQL is of great practical significance. However, smaller LLMs often struggle with complicated user instruction, redundant schema linking or syntax correctness. To address these challenges, we propose MCTS-SQL, a novel framework that uses Monte Carlo Tree Search to guide SQL generation through multi-step refinement. Since the light-weight models' weak performance of single-shot prediction, we generate better results through several trials with feedback. However, directly applying MCTS-based methods inevitably leads to significant time and computational overhead. Driven by this issue, we propose a token-level prefix-cache mechanism that stores prior information during iterations, effectively improved the execution speed. Experiments results on the SPIDER and BIRD benchmarks demonstrate the effectiveness of our approach. Using a small open-source Qwen2.5-Coder-1.5B, our method outperforms ChatGPT-3.5. When leveraging a more powerful model Gemini 2.5 to explore the performance upper bound, we achieved results competitive with the SOTA. Our findings demonstrate that even small models can be effectively deployed in practical Text-to-SQL systems with the right strategy.
comment : Accepted by AAAI 2026
♻ ☆ On the Superimposed Noise Accumulation Problem in Sequential Knowledge Editing of Large Language Models
Ding Cao, Yuchen Cai, Yuqing Huang, Xuesong He, Rongxi Guo, Guiquan Liu, Guangzhong Sun
Sequential knowledge editing techniques aim to continuously update knowledge in large language models at low cost, preventing models from generating outdated or incorrect information. However, existing sequential editing methods suffer from a significant decline in editing success rates after long-term editing. Through theoretical analysis and experiments, our findings reveal that as the number of edits increases, the model's output increasingly deviates from the desired target, leading to a drop in editing success rates. We refer to this issue as the superimposed noise accumulation problem. Our further analysis demonstrates that the problem is related to the erroneous activation of irrelevant knowledge and conflicts between activated knowledge. Based on this analysis, a method named DeltaEdit is proposed that reduces conflicts between knowledge through dynamic orthogonal constraint strategies. Experiments show that DeltaEdit significantly reduces superimposed noise, achieving a 16.8% improvement in editing performance over the strongest baseline.
♻ ☆ Odin: Oriented Dual-module Integration for Text-rich Network Representation Learning
Kaifeng Hong, Yinglong Zhang, Xiaoying Hong, Xuewen Xia, Xing Xu
Text-attributed graphs require models to effectively combine strong textual understanding with structurally informed reasoning. Existing approaches either rely on GNNs--limited by over-smoothing and hop-dependent diffusion--or employ Transformers that overlook graph topology and treat nodes as isolated sequences. We propose Odin (Oriented Dual-module INtegration), a new architecture that injects graph structure into Transformers at selected depths through an oriented dual-module mechanism. Unlike message-passing GNNs, Odin does not rely on multi-hop diffusion; instead, multi-hop structures are integrated at specific Transformer layers, yielding low-, mid-, and high-level structural abstraction aligned with the model's semantic hierarchy. Because aggregation operates on the global [CLS] representation, Odin fundamentally avoids over-smoothing and decouples structural abstraction from neighborhood size or graph topology. We further establish that Odin's expressive power strictly contains that of both pure Transformers and GNNs. To make the design efficient in large-scale or low-resource settings, we introduce Light Odin, a lightweight variant that preserves the same layer-aligned structural abstraction for faster training and inference. Experiments on multiple text-rich graph benchmarks show that Odin achieves state-of-the-art accuracy, while Light Odin delivers competitive performance with significantly reduced computational cost. Together, Odin and Light Odin form a unified, hop-free framework for principled structure-text integration. The source code of this model has been released at https://github.com/hongkaifeng/Odin.
comment : 32 pages, 2 figures
♻ ☆ WritingBench: A Comprehensive Benchmark for Generative Writing
Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, Fei Huang
Recent advancements in large language models (LLMs) have significantly enhanced text generation capabilities, yet evaluating their performance in generative writing remains a challenge. Existing benchmarks primarily focus on generic text generation or limited in writing tasks, failing to capture the diverse requirements of high-quality written contents across various domains. To bridge this gap, we present WritingBench, a comprehensive benchmark designed to evaluate LLMs across 6 core writing domains and 100 subdomains. We further propose a query-dependent evaluation framework that empowers LLMs to dynamically generate instance-specific assessment criteria. This framework is complemented by a fine-tuned critic model for criteria-aware scoring, enabling evaluations in style, format and length. The framework's validity is further demonstrated by its data curation capability, which enables a 7B-parameter model to outperform the performance of GPT-4o in writing. We open-source the benchmark, along with evaluation tools and modular framework components, to advance the development of LLMs in writing.
♻ ☆ OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe
Kaichen Zhang, Keming Wu, Zuhao Yang, Kairui Hu, Bin Wang, Ziwei Liu, Xingxuan Li, Lidong Bing
Recent advancements in large reasoning models have fueled growing interest in extending such capabilities to multimodal domains. However, despite notable progress in visual reasoning, the lack of transparent and reproducible data curation and training strategies remains a major barrier to scalable research. In this work, we introduce OpenMMReasoner, a fully transparent two-stage recipe for multimodal reasoning spanning supervised fine-tuning (SFT) and reinforcement learning (RL). In the SFT stage, we construct an 874K-sample cold-start dataset with rigorous step-by-step validation, providing a strong foundation for reasoning capabilities. The subsequent RL stage leverages a 74K-sample dataset across diverse domains to further sharpen and stabilize these abilities, resulting in a more robust and efficient learning process. Extensive evaluations demonstrate that our training recipe not only surpasses strong baselines but also highlights the critical role of data quality and training design in shaping multimodal reasoning performance. Notably, our method achieves a 11.6% improvement over the Qwen2.5-VL-7B-Instruct baseline across nine multimodal reasoning benchmarks, establishing a solid empirical foundation for future large-scale multimodal reasoning research. We open-sourced all our codes, pipeline, and data at https://github.com/EvolvingLMMs-Lab/OpenMMReasoner.
♻ ☆ Holistic Evaluation of Multimodal LLMs on Spatial Intelligence
Zhongang Cai, Yubo Wang, Qingping Sun, Ruisi Wang, Chenyang Gu, Wanqi Yin, Zhiqian Lin, Zhitao Yang, Chen Wei, Oscar Qian, Hui En Pang, Xuanke Shi, Kewang Deng, Xiaoyang Han, Zukai Chen, Jiaqi Li, Xiangyu Fan, Hanming Deng, Lewei Lu, Bo Li, Ziwei Liu, Quan Wang, Dahua Lin, Lei Yang
Multimodal models have achieved remarkable progress in recent years. Nevertheless, they continue to exhibit notable limitations in spatial understanding and reasoning, the very capability that anchors artificial general intelligence in the physical world. With the recent release of GPT-5, allegedly the most powerful AI model to date, it is timely to examine where the leading models (GPT, Gemini, Grok, Seed, Qwen, and Intern) stand on the path toward spatial intelligence (SI). We thus propose EASI for holistic Evaluation of multimodAl LLMs on Spatial Intelligence. EASI conceptualizes a comprehensive taxonomy of spatial tasks that unifies existing benchmarks and a growing collection of newly curated ones, enabling systematic evaluation of state-of-the-art models. In this report, we conduct the study across eight key benchmarks, at a cost exceeding ten billion total tokens. Our empirical study then reveals that (1) GPT-5 demonstrates unprecedented strength in SI, yet (2) still falls short of human performance significantly across a broad spectrum of SI-tasks. Moreover, we (3) show that SI-tasks expose greater model capability deficiency than non-SI tasks, to the extent that (4) proprietary models do not exhibit a decisive advantage when facing the most difficult ones. In addition, we conduct a qualitative evaluation across a diverse set of scenarios that are intuitive for humans, yet fail the most advanced multimodal models. EASI is an ongoing community effort: we have open-sourced the EASI codebase that provides a one-stop and reproducible solution with standardized interfaces, integrated protocols and prompts that significantly reduce the friction of configuring and running multiple benchmarks; we have also launched an accompanying EASI leaderboard to provide a continually updated snapshot of model performance across the full SI spectrum, accelerating collective progress toward robust SI.
comment : Codebase: https://github.com/EvolvingLMMs-Lab/EASI/; Leaderboard: https://huggingface.co/spaces/lmms-lab-si/EASI-Leaderboard
♻ ☆ G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
Wenbo Hu, Jingli Lin, Yilin Long, Yunlong Ran, Lihan Jiang, Yifan Wang, Chenming Zhu, Runsen Xu, Tai Wang, Jiangmiao Pang
Vision-Language Models (VLMs) still lack robustness in spatial intelligence, demonstrating poor performance on spatial understanding and reasoning tasks. We attribute this gap to the absence of a visual geometry learning process capable of reconstructing 3D space from 2D images. We present G$^2$VLM, a geometry grounded vision-language model that bridges two fundamental aspects of spatial intelligence: spatial 3D reconstruction and spatial understanding. G$^2$VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning. Our unified design is highly scalable for spatial understanding: it trains on abundant multi-view image and video data, while simultaneously leveraging the benefits of 3D visual priors that are typically only derived from hard-to-collect annotations. Experimental results demonstrate G$^2$VLM is proficient in both tasks, achieving comparable results to state-of-the-art feed-forward 3D reconstruction models and achieving better or competitive results across spatial understanding and reasoning tasks. By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G$^2$VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing.
comment : code are released at https://github.com/InternRobotics/G2VLM
♻ ☆ KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
Yuxuan Tian, Zihan Wang, Yebo Peng, Aomufei Yuan, Zhiming Wang, Bairen Yi, Xin Liu, Yong Cui, Tong Yang
Efficient inference of large language models (LLMs) is hindered by an ever-growing key-value (KV) cache, making KV cache compression a critical research direction. Traditional methods selectively evict less important KV cache entries, which leads to information loss and hallucinations. Recently, merging-based strategies have been explored to retain more information by merging KV pairs that would be discarded; however, these existing approaches inevitably introduce inconsistencies in attention distributions before and after merging, causing degraded generation quality. To overcome this challenge, we propose KeepKV, a novel adaptive KV cache merging method designed to preserve performance under strict memory constraints, achieving single-step lossless compression and providing error bounds for multi-step compression. KeepKV introduces the Electoral Votes mechanism that records merging history and adaptively adjusts attention scores. Moreover, it further leverages a novel Zero Inference-Perturbation Merging method, compensating for attention loss resulting from cache merging. Extensive experiments on various benchmarks and LLM architectures demonstrate that KeepKV substantially reduces memory usage while successfully retaining essential context information, achieving over 2x inference throughput improvement and maintaining superior generation quality even with only 10% KV cache budgets.
comment : 14 pages, 20 figures
♻ ☆ Simulated patient systems powered by large language model-based AI agent s offer potential for transforming medical education
Huizi Yu, Jiayan Zhou, Lingyao Li, Shan Chen, Jack Gallifant, Anye Shi, Xiang Li, Jingxian He, Wenyue Hua, Mingyu Jin, Guang Chen, Yang Zhou, Zhao Li, Trisha Gupte, Ming-Li Chen, Zahra Azizi, Qi Dou, Bryan P. Yan, Yongfeng Zhang, Yanqiu Xing, Themistocles L. Danielle S. Bitterman, Themistocles L. Assimes, Xin Ma, Lin Lu, Lizhou Fan
Background: Simulated patient systems are important in medical education and research, providing safe, integrative training environments and supporting clinical decision making. Advances in artificial intelligence (AI), especially large language models (LLMs), can enhance simulated patients by replicating medical conditions and doctor patient interactions with high fidelity and at low cost, but effectiveness and trustworthiness remain open challenges. Methods: We developed AIPatient, a simulated patient system powered by LLM based AI agents. The system uses a retrieval augmented generation (RAG) framework with six task specific agents for complex reasoning. To improve realism, it is linked to the AIPatient knowledge graph built from de identified real patient data in the MIMIC III intensive care database. Results: We evaluated electronic health record (EHR) based medical question answering (QA), readability, robustness, stability, and user experience. AIPatient reached 94.15 percent QA accuracy when all six agents were enabled, outperforming versions with partial or no agent integration. The knowledge base achieved an F1 score of 0.89. Readability scores showed a median Flesch Reading Ease of 68.77 and a median Flesch Kincaid Grade of 6.4, indicating accessibility for most medical trainees and clinicians. Robustness and stability were supported by non significant variance in repeated trials (analysis of variance F value 0.61, p greater than 0.1; F value 0.78, p greater than 0.1). A user study with medical students showed that AIPatient provides high fidelity, usability, and educational value, comparable to or better than human simulated patients for history taking. Conclusions: LLM based simulated patient systems can deliver accurate, readable, and reliable medical encounters and show strong potential to transform medical education.
comment : 19 pages, 6 figures, 4 tables
♻ ☆ Extensible Multi-Granularity Fusion Network and Transferable Curriculum Learning for Aspect-based Sentiment Analysis
Xinran Li, Xiaowei Zhao, Yubo Zhu, Zhiheng Zhang, Zhiqi Huang, Hongkun Song, Jinglu Hu, Xinze Che, Yifan Lyu, Yong Zhou, Xiujuan Xu
Aspect-based Sentiment Analysis (ABSA) aims to determine sentiment polarity toward specific aspects in text. Existing methods enrich semantic and syntactic representations through external knowledge or GNNs, but the growing diversity of linguistic features increases model complexity and lacks a unified, extensible framework. We propose an Extensible Multi-Granularity Fusion Network (EMGF) that integrates dependency syntax, constituent syntax, attention-based semantics, and external knowledge graphs. EMGF employs multi-anchor triplet learning and orthogonal projection to effectively fuse multi-granularity features and strengthen their interactions without additional computational overhead. Furthermore, we introduce the first task-specific curriculum learning framework for text-only ABSA, which assigns difficulty scores using five indicators and trains the model from easy to hard to mimic human learning and improve generalization. Experiments on SemEval 2014, Twitter, and MAMS datasets show that EMGF+CL consistently outperforms state-of-the-art ABSA models.
comment : 8 pages, 4 figures
♻ ☆ Privacy-Preserving Reasoning with Knowledge-Distilled Parametric Retrieval Augmented Generation
Jinwen Chen, Hainan Zhang, Liang Pang, Yongxin Tong, Haibo Zhou, Yuan Zhan, Wei Lin, Zhiming Zheng
The current RAG system requires uploading plaintext documents to the cloud, risking private data leakage. Parametric RAG (PRAG) encodes documents as LoRA parameters within LLMs, offering a possible way to reduce exposure of raw content. However, it still faces two issues: (1) PRAG demands synthesizing QA pairs and fine-tuning LLM for each individual document to create its corresponding LoRA, leading to unacceptable inference latency. (2) The performance of PRAG relies solely on synthetic QA data while lacking internal alignment with standard RAG, resulting in poor generalization on out-of-distribution(OOD) inputs. Therefore, achieving high-efficiency parameterization while maintaining RAG-level performance remains a critical challenge for privacy-preserving reasoning. In this paper, we propose DistilledPRAG, a generalizable knowledge-distilled parametric RAG model aligned with standard RAG in document structure and parameter activation. We first synthesize QA pairs from single and multi-documents to enhance cross-document reasoning. Then, we mask the plaintext documents with a special token and translate them to LoRA via a parameter generator, maintaining the standard RAG document structure. Finally, guided by synthetic QA data, we train the parameter generator to match standard RAG's hidden states and output logits, enabling RAG-style reasoning without original documents. Experiments on four QA datasets show that DistilledPRAG outperforms baselines in accuracy and generalizes well on OOD data.
♻ ☆ CANVAS: A Benchmark for Vision -Language Models on Tool-Based User Interface Design
Daeheon Jeong, Seoyeon Byun, Kihoon Son, Dae Hyun Kim, Juho Kim
User interface (UI) design is an iterative process in which designers progressively refine their work with design software such as Figma or Sketch. Recent advances in vision language models (VLMs) with tool invocation suggest these models can operate design software to edit a UI design through iteration. Understanding and enhancing this capacity is important, as it highlights VLMs' potential to collaborate with designers within conventional software. However, as no existing benchmark evaluates tool-based design performance, the capacity remains unknown. To address this, we introduce CANVAS, a benchmark for VLMs on tool-based user interface design. Our benchmark contains 598 tool-based design tasks paired with ground-truth references sampled from 3.3K mobile UI designs across 30 function-based categories (e.g., onboarding, messaging). In each task, a VLM updates the design step-by-step through context-based tool invocations (e.g., create a rectangle as a button background), linked to design software. Specifically, CANVAS incorporates two task types: (i) design replication evaluates the ability to reproduce a whole UI screen; (ii) design modification evaluates the ability to modify a specific part of an existing screen. Results suggest that leading models exhibit more strategic tool invocations, improving design quality. Furthermore, we identify common error patterns models exhibit, guiding future work in enhancing tool-based design capabilities.
♻ ☆ More Document s, Same Length: Isolating the Challenge of Multiple Document s in RAG
Shahar Levy, Nir Mazor, Lihi Shalmon, Michael Hassid, Gabriel Stanovsky
Retrieval-Augmented Generation (RAG) enhances the accuracy of Large Language Model (LLM) responses by leveraging relevant external documents during generation. Although previous studies noted that retrieving many documents can degrade performance, they did not isolate how the quantity of documents affects performance while controlling for context length. We evaluate various language models on custom datasets derived from a multi-hop QA task. We keep the context length and position of relevant information constant while varying the number of documents, and find that increasing the document count in RAG settings poses significant challenges for most LLMs, reducing performance by up to 20%. However, Qwen2.5 maintained consistent results across increasing document counts, indicating better multi-document handling capability. Finally, our results indicate that processing multiple documents is a separate challenge from handling long contexts. We also make the datasets and code available: https://github.com/shaharl6000/MoreDocsSameLen .
comment : Preprint
♻ ☆ KurdSTS: The Kurdish Semantic Textual Similarity
Abdulhady Abas Abdullah, Hadi Veisi, Hussein M. Al
Semantic Textual Similarity (STS) measures the degree of meaning overlap between two texts and underpins many NLP tasks. While extensive resources exist for high-resource languages, low-resource languages such as Kurdish remain underserved. We present, to our knowledge, the first Kurdish STS dataset: 10,000 sentence pairs spanning formal and informal registers, each annotated for similarity. We benchmark Sentence-BERT, multilingual BERT, and other strong baselines, obtaining competitive results while highlighting challenges arising from Kurdish morphology, orthographic variation, and code-mixing. The dataset and baselines establish a reproducible evaluation suite and provide a strong starting point for future research on Kurdish semantics and low-resource NLP.
♻ ☆ Beyond Introspection: Reinforcing Thinking via Externalist Behavioral Feedback
Diji Yang, Linda Zeng, Kezhen Chen, Yi Zhang
While inference-time thinking allows Large Language Models (LLMs) to address complex problems, the extended thinking process can be unreliable or inconsistent because of the model's probabilistic nature, especially near its knowledge boundaries. Existing approaches attempt to mitigate this by having the model critique its own reasoning to make corrections. However, such self-critique inherits the same biases of the original output, known as the introspection illusion. Moving beyond such introspection and inspired by core methodologies in ethology, we propose an externalist three-step framework Distillation-Reinforcement-Reasoning (DRR). Rather than relying on a model's introspection, DRR evaluates its observable behaviors to provide corrective feedback. DRR first distills the reasoner's behavioral traces, then trains a lightweight, external Discriminative Model (DM). At inference time, this DM acts as a critic, identifying and rejecting suspicious reasoning steps. This external feedback compels the LLM to discard flawed pathways and explore alternatives, thereby enhancing reasoning quality without altering the base model. Experiments on multiple reasoning benchmarks show that our framework significantly outperforms prominent self-critique methods. Benefiting from a lightweight and annotation-free design, DRR offers a scalable and adaptable solution for improving the reliability of reasoning in a wide range of LLMs.
♻ ☆ Exploring the Human-LLM Synergy in Advancing Theory-driven Qualitative Analysis
Han Meng, Yitian Yang, Wayne Fu, Jungup Lee, Yunan Li, Yi-Chieh Lee
Qualitative coding is a demanding yet crucial research method in the field of Human-Computer Interaction (HCI). While recent studies have shown the capability of large language models (LLMs) to perform qualitative coding within theoretical frameworks, their potential for collaborative human-LLM discovery and generation of new insights beyond initial theory remains underexplored. To bridge this gap, we proposed CHALET, a novel approach that harnesses the power of human-LLM partnership to advance theory-driven qualitative analysis by facilitating iterative coding, disagreement analysis, and conceptualization of qualitative data. We demonstrated CHALET's utility by applying it to the qualitative analysis of conversations related to mental-illness stigma, using the attribution model as the theoretical framework. Results highlighted the unique contribution of human-LLM collaboration in uncovering latent themes of stigma across the cognitive, emotional, and behavioral dimensions. We discuss the methodological implications of the human-LLM collaborative approach to theory-based qualitative analysis for the HCI community and beyond.
comment : 51 pages, 6 figures, accepted by ACM Trans. Comput.-Hum. Interact (TOCHI)
♻ ☆ TrackList: Tracing Back Query Linguistic Diversity for Head and Tail Knowledge in Open Large Language Models
Ioana Buhnila, Aman Sinha, Mathieu Constant
Large Language Models (LLMs) have proven efficient in giving definition-type answers to user input queries. While for humans giving various types of answers, such as examples and paraphrases, is an easy task, LLMs struggle to provide correct answers for other than definition-type queries. In this study, we evaluated this drop in performance using TrackList, a fine-grained linguistic and statistical analysis pipeline to investigate the impact of the pre-training data on LLMs answers to diverse linguistic queries. We also introduce RefoMed-EN, an English dataset consisting of 6170 human-annotated medical terms alongside their corresponding definitions, denominations, exemplifications, explanations, or paraphrases. We studied whether the high frequency of a concept (head) or low frequency (tail) impacts the language model's performance. We evaluated the quality of the LLM's output using syntactic and semantic similarity metrics, statistical correlations and embeddings. Results showed that the LLM's task performance for definition type questions is the highest, while for the exemplification type it is the lowest. Additionally, we showed that for definition-type questions, large language models are prone to paraphrase more on popular and frequent knowledge and less on tail and technical knowledge, especially in the expert texts.
comment : under review
♻ ☆ Mavors: Multi-granularity Video Representation for Multimodal Large Language Model
Yang Shi, Jiaheng Liu, Yushuo Guan, Zhenhua Wu, Yuanxing Zhang, Zihao Wang, Weihong Lin, Jingyun Hua, Zekun Wang, Xinlong Chen, Bohan Zeng, Wentao Zhang, Fuzheng Zhang, Wenjing Yang, Di Zhang
Long-context video understanding in multimodal large language models (MLLMs) faces a critical challenge: balancing computational efficiency with the retention of fine-grained spatio-temporal patterns. Existing approaches (e.g., sparse sampling, dense sampling with low resolution, and token compression) suffer from significant information loss in temporal dynamics, spatial details, or subtle interactions, particularly in videos with complex motion or varying resolutions. To address this, we propose $\mathbf{Mavors}$, a novel framework that introduces $\mathbf{M}$ulti-gr$\mathbf{a}$nularity $\mathbf{v}$ide$\mathbf{o}$ $\mathbf{r}$epre$\mathbf{s}$entation for holistic long-video modeling. Specifically, Mavors directly encodes raw video content into latent representations through two core components: 1) an Intra-chunk Vision Encoder (IVE) that preserves high-resolution spatial features via 3D convolutions and Vision Transformers, and 2) an Inter-chunk Feature Aggregator (IFA) that establishes temporal coherence across chunks using transformer-based dependency modeling with chunk-level rotary position encodings. Moreover, the framework unifies image and video understanding by treating images as single-frame videos via sub-image decomposition. Experiments across diverse benchmarks demonstrate Mavors' superiority in maintaining both spatial fidelity and temporal continuity, significantly outperforming existing methods in tasks requiring fine-grained spatio-temporal reasoning.
comment : 22 pages
♻ ☆ COPO: Causal-Oriented Policy Optimization for Hallucinations of MLLMs
Peizheng Guo, Jingyao Wang, Wenwen Qiang, Jiahuan Zhou, Changwen Zheng, Gang Hua
Despite Multimodal Large Language Models (MLLMs) having shown impressive capabilities, they may suffer from hallucinations. Empirically, we find that MLLMs attend disproportionately to task-irrelevant background regions compared with text-only LLMs, implying spurious background-answer correlations. We claim and analyze that (i) outcome-based rewards can be an important factor leading to spurious correlations, and (ii) spurious correlations can be an important factor leading to hallucinations. Based on these results, we propose Causal-Oriented Policy Optimization (COPO) to mitigate these spurious correlations, thus addressing the issue of hallucinations. It imposes token-level sufficiency and necessity constraints to measure each inference token's causal contribution, thus ensuring correct and evidence-grounded output. Specifically, we first evaluate each token's causal contribution via a newly proposed causal completeness reward. This reward is then used to construct a causally informed advantage function within the GRPO optimization framework, encouraging the model to focus on tokens that are causally sufficient and necessary for accurate generation. Experimental results across various benchmarks demonstrate the advantages of COPO.
♻ ☆ Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization
Yue Zhang, Jingxuan Zuo, Ke Su, Liqiang Jing
Multimodal summarization aims to generate a concise summary based on the input text and image. However, the existing methods potentially suffer from unfactual output. To evaluate the factuality of multimodal summarization models, we propose two fine-grained and explainable evaluation frameworks (FALLACIOUS) for different application scenarios, i.e. reference-based factuality evaluation framework and reference-free factuality evaluation framework. Notably, the reference-free factuality evaluation framework doesn't need ground truth and hence it has a wider application scenario. To evaluate the effectiveness of the proposed frameworks, we compute the correlation between our frameworks and the other metrics. The experimental results show the effectiveness of our proposed method. We will release our code and dataset via github.
comment : project link: https://github.com/for4WARD/FaithfulnessEvaluation
♻ ☆ ROVER: Recursive Reasoning Over Videos with Vision -Language Models for Embodied Tasks
Philip Schroeder, Ondrej Biza, Thomas Weng, Hongyin Luo, James Glass
Vision-language models (VLMs) have exhibited impressive capabilities across diverse image understanding tasks, but still struggle in settings that require reasoning over extended sequences of camera frames from a video. This limits their utility in embodied settings, which require reasoning over long frame sequences from a continuous stream of visual input at each moment of a task attempt. To address this limitation, we propose ROVER (Reasoning Over VidEo Recursively), a framework that enables the model to recursively decompose long-horizon video trajectories into segments corresponding to shorter subtasks within the trajectory. In doing so, ROVER facilitates more focused and accurate reasoning over temporally localized frame sequences without losing global context. We evaluate ROVER, implemented using an in-context learning approach, on diverse OpenX Embodiment videos and on a new dataset derived from RoboCasa that consists of 543 videos showing both expert and perturbed non-expert trajectories across 27 robotic manipulation tasks. ROVER outperforms strong baselines across three video reasoning tasks: task progress estimation, frame-level natural language reasoning, and video question answering. We observe that, by reducing the number of frames the model reasons over at each timestep, ROVER mitigates hallucinations, especially during unexpected or non-optimal moments of a trajectory. In addition, by enabling the implementation of a subtask-specific sliding context window, ROVER's time complexity scales linearly with video length, an asymptotic improvement over baselines. Demos, code, and data available at: https://rover-vlm.github.io
♻ ☆ Financial Risk Relation Identification through Dual-view Adaptation EMNLP 2025
Wei-Ning Chiu, Yu-Hsiang Wang, Andy Hsiao, Yu-Shiang Huang, Chuan-Ju Wang
A multitude of interconnected risk events -- ranging from regulatory changes to geopolitical tensions -- can trigger ripple effects across firms. Identifying inter-firm risk relations is thus crucial for applications like portfolio management and investment strategy. Traditionally, such assessments rely on expert judgment and manual analysis, which are, however, subjective, labor-intensive, and difficult to scale. To address this, we propose a systematic method for extracting inter-firm risk relations using Form 10-K filings -- authoritative, standardized financial documents -- as our data source. Leveraging recent advances in natural language processing, our approach captures implicit and abstract risk connections through unsupervised fine-tuning based on chronological and lexical patterns in the filings. This enables the development of a domain-specific financial encoder with a deeper contextual understanding and introduces a quantitative risk relation score for transparency, interpretable analysis. Extensive experiments demonstrate that our method outperforms strong baselines across multiple evaluation settings. Our codes are available at https://github.com/cnclabs/codes.fin.relation.
comment : 11 pages, 3 figures, EMNLP 2025 Main Conference
♻ ☆ Accelerating Training of Recursive Reasoning Models with Curriculum Guided Adaptive Recursion
Kaleem Ullah Qasim, Jiashu Zhang
Recursive reasoning models achieve remarkable performance on complex reasoning tasks through iterative refinement, enabling tiny networks to match large language models thousands of times their size. However, training remains computationally expensive, prior work reporting approximately 36 GPU-hours per dataset, limiting broader adoption and research. We propose CGAR, a novel training methodology that applies curriculum learning to architectural depth rather than traditional data ordering. CGAR introduces two synergistic components: Progressive Depth Curriculum dynamically adjusts recursion depth from shallow to deep configurations during training, preventing early overfitting while reducing computational cost, and Hierarchical Supervision Weighting applies exponentially decaying importance to supervision steps, aligning loss weighting with observed gradient magnitude decay. On Sudoku-Extreme with 423,168 test puzzles, CGAR achieves 1.71x training speedup (10.93 to 6.38 hours, 42% cost reduction) with only 0.63% accuracy drop (86.65% to 86.02%). Systematic ablations reveal Progressive Depth Curriculum alone achieves 2.26x speedup with 85.47% accuracy, demonstrating a rare Pareto improvement where architectural curriculum simultaneously enhances training efficiency and solution quality. CGAR-trained models exhibit superior inference efficiency with 100% halting accuracy and 11% fewer reasoning steps. Our work demonstrates that principled curriculum on architectural depth enables efficient training of recursive reasoning models on modest hardware. Code and models: https://github.com/Kaleemullahqasim/CGAR and https://huggingface.co/Kaleemullah/trm-cgar-sudoku
Artificial Intelligence 224
☆ Revisiting Generalization Across Difficulty Levels: It's Not So Easy
Yeganeh Kordi, Nihal V. Nayak, Max Zuo, Ilana Nguyen, Stephen H. Bach
We investigate how well large language models (LLMs) generalize across different task difficulties, a key question for effective data curation and evaluation. Existing research is mixed regarding whether training on easier or harder data leads to better results, and whether those gains come on easier or harder test data. We address this question by conducting a systematic evaluation of LLMs' generalization across models, datasets, and fine-grained groups of example difficulty. We rank examples in six datasets using the outputs of thousands of different LLMs and Item Response Theory (IRT), a well-established difficulty metric in educational testing. Unlike prior work, our difficulty ratings are therefore determined solely by the abilities of many different LLMs, excluding human opinions of difficulty. With a more objective, larger-scale, and finer-grained analysis, we show that cross-difficulty generalization is often limited; training on either easy or hard data cannot achieve consistent improvements across the full range of difficulties. These results show the importance of having a range of difficulties in both training and evaluation data for LLMs, and that taking shortcuts with respect to difficulty is risky.
☆ ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration
Hongjin Su, Shizhe Diao, Ximing Lu, Mingjie Liu, Jiacheng Xu, Xin Dong, Yonggan Fu, Peter Belcak, Hanrong Ye, Hongxu Yin, Yi Dong, Evelina Bakhturina, Tao Yu, Yejin Choi, Jan Kautz, Pavlo Molchanov
Large language models are powerful generalists, yet solving deep and complex problems such as those of the Humanity's Last Exam (HLE) remains both conceptually challenging and computationally expensive. We show that small orchestrators managing other models and a variety of tools can both push the upper bound of intelligence and improve efficiency in solving difficult agentic tasks. We introduce ToolOrchestra, a method for training small orchestrators that coordinate intelligent tools. ToolOrchestra explicitly uses reinforcement learning with outcome-, efficiency-, and user-preference-aware rewards. Using ToolOrchestra, we produce Orchestrator, an 8B model that achieves higher accuracy at lower cost than previous tool-use agents while aligning with user preferences on which tools are to be used for a given query. On HLE, Orchestrator achieves a score of 37.1%, outperforming GPT-5 (35.1%) while being 2.5x more efficient. On tau2-Bench and FRAMES, Orchestrator surpasses GPT-5 by a wide margin while using only about 30% of the cost. Extensive analysis shows that Orchestrator achieves the best trade-off between performance and cost under multiple metrics, and generalizes robustly to unseen tools. These results demonstrate that composing diverse tools with a lightweight orchestration model is both more efficient and more effective than existing methods, paving the way for practical and scalable tool-augmented reasoning systems.
comment : 21 pages, 6 figures
☆ G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
Wenbo Hu, Jingli Lin, Yilin Long, Yunlong Ran, Lihan Jiang, Yifan Wang, Chenming Zhu, Runsen Xu, Tai Wang, Jiangmiao Pang
Vision-Language Models (VLMs) still lack robustness in spatial intelligence, demonstrating poor performance on spatial understanding and reasoning tasks. We attribute this gap to the absence of a visual geometry learning process capable of reconstructing 3D space from 2D images. We present G$^2$VLM, a geometry grounded vision-language model that bridges two fundamental aspects of spatial intelligence: spatial 3D reconstruction and spatial understanding. G$^2$VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning. Our unified design is highly scalable for spatial understanding: it trains on abundant multi-view image and video data, while simultaneously leveraging the benefits of 3D visual priors that are typically only derived from hard-to-collect annotations. Experimental results demonstrate G$^2$VLM is proficient in both tasks, achieving comparable results to state-of-the-art feed-forward 3D reconstruction models and achieving better or competitive results across spatial understanding and reasoning tasks. By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G$^2$VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing.
comment : code are released at https://github.com/InternRobotics/G2VLM
☆ Matrix: Peer-to-Peer Multi- Agent Synthetic Data Generation Framework
Dong Wang, Yang Li, Ansong Ni, Ching-Feng Yeh, Youssef Emad, Xinjie Lei, Liam Robbins, Karthik Padthe, Hu Xu, Xian Li, Asli Celikyilmaz, Ramya Raghavendra, Lifei Huang, Carole-Jean Wu, Shang-Wen Li
Synthetic data has become increasingly important for training large language models, especially when real data is scarce, expensive, or privacy-sensitive. Many such generation tasks require coordinated multi-agent workflows, where specialized agents collaborate to produce data that is higher quality, more diverse, and structurally richer. However, existing frameworks for multi-agent synthesis often depend on a centralized orchestrator, creating scalability bottlenecks, or are hardcoded for specific domains, limiting flexibility. We present \textbf{Matrix}, a decentralized framework that represents both control and data flow as serialized messages passed through distributed queues. This peer-to-peer design eliminates the central orchestrator. Each task progresses independently through lightweight agents, while compute-intensive operations, such as LLM inference or containerized environments, are handled by distributed services. Built on Ray, Matrix scales to tens of thousands of concurrent agentic workflows and provides a modular, configurable design that enables easy adaptation to a wide range of data generation workflows. We evaluate Matrix across diverse synthesis scenarios, such as multi-agent collaborative dialogue, web-based reasoning data extraction, and tool-use trajectory generation in customer service environments. In all cases, Matrix achieves $2$--$15\times$ higher data generation throughput under identical hardware resources, without compromising output quality.
☆ Agent ic Learner with Grow-and-Refine Multimodal Semantic Memory
Weihao Bo, Shan Zhang, Yanpeng Sun, Jingjing Wu, Qunyi Xie, Xiao Tan, Kunbin Chen, Wei He, Xiaofan Li, Na Zhao, Jingdong Wang, Zechao Li
MLLMs exhibit strong reasoning on isolated queries, yet they operate de novo -- solving each problem independently and often repeating the same mistakes. Existing memory-augmented agents mainly store past trajectories for reuse. However, trajectory-based memory suffers from brevity bias, gradually losing essential domain knowledge. More critically, even in truly multimodal problem-solving settings, it records only a single-modality trace of past behavior, failing to preserve how visual attention and logical reasoning jointly contributed to the solution. This is fundamentally misaligned with human cognition: semantic memory is both multimodal and integrated, preserving visual and abstract knowledge through coordinated but distinct representational streams. We thus introduce ViLoMem, a dual-stream memory framework that constructs compact, schema-based memory. It separately encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences. Following a grow-and-refine principle, the system incrementally accumulates and updates multimodal semantic knowledge -- preserving stable, generalizable strategies while avoiding catastrophic forgetting. Across six multimodal benchmarks, ViLoMem consistently improves pass@1 accuracy and substantially reduces repeated visual and logical errors. Ablations confirm the necessity of dual-stream memory with explicit distraction--hallucination separation, demonstrating the value of error-aware multimodal memory for lifelong and cross-domain agentic learning. Our project page will be available at https://weihao-bo.github.io/ViLoMeo-page.
☆ Through the telecom lens: Are all training samples important?
Shruti Bothe, Illyyne Saffar, Aurelie Boisbunon, Hasan Farooq, Julien Forgeat, Md Moin Uddin Chowdhury
The rise of AI in telecommunications, from optimizing Radio Access Networks to managing user experience, has sharply increased data volumes and training demands. Telecom data is often noisy, high-dimensional, costly to store, process, and label. Despite Ai's critical role, standard workflows still assume all training samples contribute equally. On the other hand, next generation systems require AI models that are accurate, efficient, and sustainable.The paper questions the assumptions of equal importance by focusing on applying and analyzing the roles of individual samples in telecom training and assessing whether the proposed model optimizes computation and energy use. we perform sample-level gradient analysis across epochs to identify patterns of influence and redundancy in model learning. Based on this, we propose a sample importance framework thats electively prioritizes impactful data and reduces computation without compromising accuracy. Experiments on three real-world telecom datasets show that our method [reserves performance while reducing data needs and computational overhead while advancing the goals of sustainable AI in telecommunications.
comment : 8pages, 1 table, 8 figures
☆ Escaping the Verifier: Learning to Reason via Demonstrations
Locke Cai, Ivan Provilkov
Training Large Language Models (LLMs) to reason often relies on Reinforcement Learning (RL) with task-specific verifiers. However, many real-world reasoning-intensive tasks lack verifiers, despite offering abundant expert demonstrations that remain under-utilized for reasoning-focused training. We introduce RARO (Relativistic Adversarial Reasoning Optimization) that learns strong reasoning capabilities from only expert demonstrations via Inverse Reinforcement Learning. Our method sets up an adversarial interaction between a policy (generator) and a relativistic critic (discriminator): the policy learns to mimic expert answers, while the critic learns to compare and distinguish between policy and expert answers. Our method trains both the policy and the critic jointly and continuously via RL, and we identify the key stabilization techniques required for robust learning. Empirically, RARO significantly outperforms strong verifier-free baselines on all of our evaluation tasks -- Countdown, DeepMath, and Poetry Writing -- and enjoys the same robust scaling trends as RL on verifiable tasks. These results demonstrate that our method effectively elicits strong reasoning performance from expert demonstrations alone, enabling robust reasoning learning even when task-specific verifiers are unavailable.
☆ Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision -Language-Action Models
Naifu Zhang, Wei Tao, Xi Xiao, Qianpu Sun, Yuxin Zheng, Wentao Mo, Peiqiang Wang, Nan Zhang
In recent years, Vision-Language-Action (VLA) models in embodied intelligence have developed rapidly. However, existing adversarial attack methods require costly end-to-end training and often generate noticeable perturbation patches. To address these limitations, we propose ADVLA, a framework that directly applies adversarial perturbations on features projected from the visual encoder into the textual feature space. ADVLA efficiently disrupts downstream action predictions under low-amplitude constraints, and attention guidance allows the perturbations to be both focused and sparse. We introduce three strategies that enhance sensitivity, enforce sparsity, and concentrate perturbations. Experiments demonstrate that under an $L_{\infty}=4/255$ constraint, ADVLA combined with Top-K masking modifies less than 10% of the patches while achieving an attack success rate of nearly 100%. The perturbations are concentrated on critical regions, remain almost imperceptible in the overall image, and a single-step iteration takes only about 0.06 seconds, significantly outperforming conventional patch-based attacks. In summary, ADVLA effectively weakens downstream action predictions of VLA models under low-amplitude and locally sparse conditions, avoiding the high training costs and conspicuous perturbations of traditional patch attacks, and demonstrates unique effectiveness and practical value for attacking VLA feature spaces.
☆ Continual Error Correction on Low-Resource Devices ACM MM
Kirill Paramonov, Mete Ozay, Aristeidis Mystakidis, Nikolaos Tsalikidis, Dimitrios Sotos, Anastasios Drosou, Dimitrios Tzovaras, Hyunjun Kim, Kiseok Chang, Sangdok Mo, Namwoong Kim, Woojong Yoo, Jijoong Moon, Umberto Michieli
The proliferation of AI models in everyday devices has highlighted a critical challenge: prediction errors that degrade user experience. While existing solutions focus on error detection, they rarely provide efficient correction mechanisms, especially for resource-constrained devices. We present a novel system enabling users to correct AI misclassifications through few-shot learning, requiring minimal computational resources and storage. Our approach combines server-side foundation model training with on-device prototype-based classification, enabling efficient error correction through prototype updates rather than model retraining. The system consists of two key components: (1) a server-side pipeline that leverages knowledge distillation to transfer robust feature representations from foundation models to device-compatible architectures, and (2) a device-side mechanism that enables ultra-efficient error correction through prototype adaptation. We demonstrate our system's effectiveness on both image classification and object detection tasks, achieving over 50% error correction in one-shot scenarios on Food-101 and Flowers-102 datasets while maintaining minimal forgetting (less than 0.02%) and negligible computational overhead. Our implementation, validated through an Android demonstration app, proves the system's practicality in real-world scenarios.
comment : ACM MMSys 2025
☆ Bridging the Unavoidable A Priori: A Framework for Comparative Causal Modeling
Peter S. Hovmand, Kari O'Donnell, Callie Ogland-Hand, Brian Biroscak, Douglas D. Gunzler
AI/ML models have rapidly gained prominence as innovations for solving previously unsolved problems and their unintended consequences from amplifying human biases. Advocates for responsible AI/ML have sought ways to draw on the richer causal models of system dynamics to better inform the development of responsible AI/ML. However, a major barrier to advancing this work is the difficulty of bringing together methods rooted in different underlying assumptions (i.e., Dana Meadow's "the unavoidable a priori"). This paper brings system dynamics and structural equation modeling together into a common mathematical framework that can be used to generate systems from distributions, develop methods, and compare results to inform the underlying epistemology of system dynamics for data science and AI/ML applications.
comment : Presented at 43rd Conference of the International System Dynamics Society in Boston, United States
☆ Mechanisms of Non-Monotonic Scaling in Vision Transformer s
Anantha Padmanaban Krishna Kumar
Deeper Vision Transformers often perform worse than shallower ones, which challenges common scaling assumptions. Through a systematic empirical analysis of ViT-S, ViT-B, and ViT-L on ImageNet, we identify a consistent three-phase Cliff-Plateau-Climb pattern that governs how representations evolve with depth. We observe that better performance is associated with progressive marginalization of the [CLS] token, originally designed as a global aggregation hub, in favor of distributed consensus among patch tokens. We quantify patterns of information mixing with an Information Scrambling Index, and show that in ViT-L the information-task tradeoff emerges roughly 10 layers later than in ViT-B, and that these additional layers correlate with increased information diffusion rather than improved task performance. Taken together, these results suggest that transformer architectures in this regime may benefit more from carefully calibrated depth that executes clean phase transitions than from simply increasing parameter count. The Information Scrambling Index provides a useful diagnostic for existing models and suggests a potential design target for future architectures. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/Cliff-Plateau-Climb.
comment : 16 pages total (11 pages main text, 1 pages references, 4 pages appendix), 5 figures, 11 tables. Code available at https://github.com/AnanthaPadmanaban-KrishnaKumar/Cliff-Plateau-Climb
☆ Qwen 3-VL Technical Report
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, Ke Zhu
We introduce Qwen3-VL, the most capable vision-language model in the Qwen series to date, achieving superior performance across a broad range of multimodal benchmarks. It natively supports interleaved contexts of up to 256K tokens, seamlessly integrating text, images, and video. The model family includes both dense (2B/4B/8B/32B) and mixture-of-experts (30B-A3B/235B-A22B) variants to accommodate diverse latency-quality trade-offs. Qwen3-VL delivers three core pillars: (i) markedly stronger pure-text understanding, surpassing comparable text-only backbones in several cases; (ii) robust long-context comprehension with a native 256K-token window for both text and interleaved multimodal inputs, enabling faithful retention, retrieval, and cross-referencing across long documents and videos; and (iii) advanced multimodal reasoning across single-image, multi-image, and video tasks, demonstrating leading performance on comprehensive evaluations such as MMMU and visual-math benchmarks (e.g., MathVista and MathVision). Architecturally, we introduce three key upgrades: (i) an enhanced interleaved-MRoPE for stronger spatial-temporal modeling across images and video; (ii) DeepStack integration, which effectively leverages multi-level ViT features to tighten vision-language alignment; and (iii) text-based time alignment for video, evolving from T-RoPE to explicit textual timestamp alignment for more precise temporal grounding. Under comparable token budgets and latency constraints, Qwen3-VL achieves superior performance in both dense and Mixture-of-Experts (MoE) architectures. We envision Qwen3-VL serving as a foundational engine for image-grounded reasoning, agentic decision-making, and multimodal code intelligence in real-world workflows.
comment : 42 pages
☆ Scale-Agnostic Kolmogorov-Arnold Geometry in Neural Networks
Mathew Vanherreweghe, Michael H. Freedman, Keith M. Adams
Recent work by Freedman and Mulligan demonstrated that shallow multilayer perceptrons spontaneously develop Kolmogorov-Arnold geometric (KAG) structure during training on synthetic three-dimensional tasks. However, it remained unclear whether this phenomenon persists in realistic high-dimensional settings and what spatial properties this geometry exhibits. We extend KAG analysis to MNIST digit classification (784 dimensions) using 2-layer MLPs with systematic spatial analysis at multiple scales. We find that KAG emerges during training and appears consistently across spatial scales, from local 7-pixel neighborhoods to the full 28x28 image. This scale-agnostic property holds across different training procedures: both standard training and training with spatial augmentation produce the same qualitative pattern. These findings reveal that neural networks spontaneously develop organized, scale-invariant geometric structure during learning on realistic high-dimensional data.
☆ On the Origin of Algorithmic Progress in AI
Hans Gundlach, Alex Fogelson, Jayson Lynch, Ana Trisovic, Jonathan Rosenfeld, Anmol Sandhu, Neil Thompson
Algorithms have been estimated to increase AI training FLOP efficiency by a factor of 22,000 between 2012 and 2023 [Ho et al., 2024]. Running small-scale ablation experiments on key innovations from this time period, we are able to account for less than 10x of these gains. Surveying the broader literature, we estimate that additional innovations not included in our ablations account for less than 10x, yielding a total under 100x. This leads us to conduct scaling experiments, which reveal that much of this efficiency gap can be explained by algorithms with scale-dependent efficiency improvements. In particular, we conduct scaling experiments between LSTMs and Transformers, finding exponent differences in their compute-optimal scaling law while finding little scaling difference for many other innovations. These experiments demonstrate that - contrary to standard assumptions - an algorithm's efficiency gains are tied to compute scale. Using experimental extrapolation and literature estimates, we account for 6,930x efficiency gains over the same time period, with the scale-dependent LSTM-to-Transformer transition accounting for the majority of gains. Our results indicate that algorithmic progress for small models has been far slower than previously assumed, and that measures of algorithmic efficiency are strongly reference-dependent.
☆ Beyond U RL s: Metadata Diversity and Position for Efficient LLM Pretrain ing
Dongyang Fan, Diba Hashemi, Sai Praneeth Karimireddy, Martin Jaggi
Incorporating metadata in Large Language Models (LLMs) pretraining has recently emerged as a promising approach to accelerate training. However prior work highlighted only one useful signal-URLs, leaving open the question of whether other forms of metadata could yield greater benefits. In this study, we investigate a wider range of metadata types and find other types of metadata, such as fine-grained indicators of document quality that can also accelerate pretraining when prepended. We identify a common feature among effective metadata: they encode information at a finer granularity. We further introduce metadata appending as a means of improving training efficiency, where predicting an appropriate metadata as auxiliary task can help speed up pretraining. In addition, learnable meta-tokens trained with masked loss can recover part of the speedup by inducing quality-aware latent structure. Using probing, we analyze latent representations to understand how metadata shapes learning. Together, these results yield practical guidelines for integrating metadata to improve both the efficiency and effectiveness of LLM pretraining.
☆ On the Limits of Innate Planning in Large Language Models
Charles Schepanowski, Charles Ling
Large language models (LLMs) achieve impressive results on many benchmarks, yet their capacity for planning and stateful reasoning remains unclear. We study these abilities directly, without code execution or other tools, using the 8-puzzle: a classic task that requires state tracking and goal-directed planning while allowing precise, step-by-step evaluation. Four models are tested under common prompting conditions (Zero-Shot, Chain-of-Thought, Algorithm-of-Thought) and with tiered corrective feedback. Feedback improves success rates for some model-prompt combinations, but many successful runs are long, computationally expensive, and indirect. We then examine the models with an external move validator that provides only valid moves. Despite this level of assistance, none of the models solve any puzzles in this setting. Qualitative analysis reveals two dominant deficits across all models: (1) brittle internal state representations, leading to frequent invalid moves, and (2) weak heuristic planning, with models entering loops or selecting actions that do not reduce the distance to the goal state. These findings indicate that, in the absence of external tools such as code interpreters, current LLMs have substantial limitations in planning and that further progress may require mechanisms for maintaining explicit state and performing structured search.
comment : 33 pages, 7 figures
☆ Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving NeurIPS 2025
Haohong Lin, Yunzhi Zhang, Wenhao Ding, Jiajun Wu, Ding Zhao
End-to-end (E2E) autonomous driving models have demonstrated strong performance in open-loop evaluations but often suffer from cascading errors and poor generalization in closed-loop settings. To address this gap, we propose Model-based Policy Adaptation (MPA), a general framework that enhances the robustness and safety of pretrained E2E driving agents during deployment. MPA first generates diverse counterfactual trajectories using a geometry-consistent simulation engine, exposing the agent to scenarios beyond the original dataset. Based on this generated data, MPA trains a diffusion-based policy adapter to refine the base policy's predictions and a multi-step Q value model to evaluate long-term outcomes. At inference time, the adapter proposes multiple trajectory candidates, and the Q value model selects the one with the highest expected utility. Experiments on the nuScenes benchmark using a photorealistic closed-loop simulator demonstrate that MPA significantly improves performance across in-domain, out-of-domain, and safety-critical scenarios. We further investigate how the scale of counterfactual data and inference-time guidance strategies affect overall effectiveness.
comment : Published at NeurIPS 2025: https://openreview.net/forum?id=4OLbpaTKJe
☆ HarmonicAttack: An Adaptive Cross-Domain Audio Watermark Removal
Kexin Li, Xiao Hu, Ilya Grishchenko, David Lie
The availability of high-quality, AI-generated audio raises security challenges such as misinformation campaigns and voice-cloning fraud. A key defense against the misuse of AI-generated audio is by watermarking it, so that it can be easily distinguished from genuine audio. As those seeking to misuse AI-generated audio may thus seek to remove audio watermarks, studying effective watermark removal techniques is critical to being able to objectively evaluate the robustness of audio watermarks against removal. Previous watermark removal schemes either assume impractical knowledge of the watermarks they are designed to remove or are computationally expensive, potentially generating a false sense of confidence in current watermark schemes. We introduce HarmonicAttack, an efficient audio watermark removal method that only requires the basic ability to generate the watermarks from the targeted scheme and nothing else. With this, we are able to train a general watermark removal model that is able to remove the watermarks generated by the targeted scheme from any watermarked audio sample. HarmonicAttack employs a dual-path convolutional autoencoder that operates in both temporal and frequency domains, along with GAN-style training, to separate the watermark from the original audio. When evaluated against state-of-the-art watermark schemes AudioSeal, WavMark, and Silentcipher, HarmonicAttack demonstrates greater watermark removal ability than previous watermark removal methods with near real-time performance. Moreover, while HarmonicAttack requires training, we find that it is able to transfer to out-of-distribution samples with minimal degradation in performance.
☆ Multimodal Robust Prompt Distillation for 3D Point Cloud Models
Xiang Gu, Liming Lu, Xu Zheng, Anan Du, Yongbin Zhou, Shuchao Pang
Adversarial attacks pose a significant threat to learning-based 3D point cloud models, critically undermining their reliability in security-sensitive applications. Existing defense methods often suffer from (1) high computational overhead and (2) poor generalization ability across diverse attack types. To bridge these gaps, we propose a novel yet efficient teacher-student framework, namely Multimodal Robust Prompt Distillation (MRPD) for distilling robust 3D point cloud model. It learns lightweight prompts by aligning student point cloud model's features with robust embeddings from three distinct teachers: a vision model processing depth projections, a high-performance 3D model, and a text encoder. To ensure a reliable knowledge transfer, this distillation is guided by a confidence-gated mechanism which dynamically balances the contribution of all input modalities. Notably, since the distillation is all during the training stage, there is no additional computational cost at inference. Extensive experiments demonstrate that MRPD substantially outperforms state-of-the-art defense methods against a wide range of white-box and black-box attacks, while even achieving better performance on clean data. Our work presents a new, practical paradigm for building robust 3D vision systems by efficiently harnessing multimodal knowledge.
☆ BAMAS: Structuring Budget-Aware Multi- Agent Systems AAAI 2026
Liming Yang, Junyu Luo, Xuanzhe Liu, Yiling Lou, Zhenpeng Chen
Large language model (LLM)-based multi-agent systems have emerged as a powerful paradigm for enabling autonomous agents to solve complex tasks. As these systems scale in complexity, cost becomes an important consideration for practical deployment. However, existing work rarely addresses how to structure multi-agent systems under explicit budget constraints. In this paper, we propose BAMAS, a novel approach for building multi-agent systems with budget awareness. BAMAS first selects an optimal set of LLMs by formulating and solving an Integer Linear Programming problem that balances performance and cost. It then determines how these LLMs should collaborate by leveraging a reinforcement learning-based method to select the interaction topology. Finally, the system is instantiated and executed based on the selected agents and their collaboration topology. We evaluate BAMAS on three representative tasks and compare it with state-of-the-art agent construction methods. Results show that BAMAS achieves comparable performance while reducing cost by up to 86%.
comment : Accepted by AAAI 2026 (oral paper)
☆ From Prediction to Foresight: The Role of AI in Designing Responsible Futures
Maria Perez-Ortiz
In an era marked by rapid technological advancements and complex global challenges, responsible foresight has emerged as an essential framework for policymakers aiming to navigate future uncertainties and shape the future. Responsible foresight entails the ethical anticipation of emerging opportunities and risks, with a focus on fostering proactive, sustainable, and accountable future design. This paper coins the term "responsible computational foresight", examining the role of human-centric artificial intelligence and computational modeling in advancing responsible foresight, establishing a set of foundational principles for this new field and presenting a suite of AI-driven foresight tools currently shaping it. AI, particularly in conjunction with simulations and scenario analysis, enhances policymakers' ability to address uncertainty, evaluate risks, and devise strategies geared toward sustainable, resilient futures. However, responsible foresight extends beyond mere technical forecasting; it demands a nuanced understanding of the interdependencies within social, environmental, economic and political systems, alongside a commitment to ethical, long-term decision-making that supports human intelligence. We argue that AI will play a role as a supportive tool in responsible, human-centered foresight, complementing rather than substituting policymaker judgment to enable the proactive shaping of resilient and ethically sound futures. This paper advocates for the thoughtful integration of AI into foresight practices to empower policymakers and communities as they confront the grand challenges of the 21st century.
comment : Accessible at https://projecteuclid.org/journals/journal-of-artificial-intelligence-for-sustainable-development/volume-1/issue-1/From-Prediction-to-Foresight--The-Role-of-AI-in/10.69828/4d4kja.full
☆ Self-Transparency Failures in Expert-Persona LLMs: A Large-Scale Behavioral Audit
Alex Diep
If a language model cannot reliably disclose its AI identity in expert contexts, users cannot trust its competence boundaries. This study examines self-transparency in models assigned professional personas within high-stakes domains where false expertise risks user harm. Using a common-garden design, sixteen open-weight models (4B--671B parameters) were audited across 19,200 trials. Models exhibited sharp domain-specific inconsistency: a Financial Advisor persona elicited 30.8% disclosure initially, while a Neurosurgeon persona elicited only 3.5%. This creates preconditions for a "Reverse Gell-Mann Amnesia" effect, where transparency in some domains leads users to overgeneralize trust to contexts where disclosure fails. Disclosure ranged from 2.8% to 73.6%, with a 14B model reaching 61.4% while a 70B produced just 4.1%. Model identity predicted behavior better than parameter count ($ΔR_{adj}^{2} = 0.359$ vs 0.018). Reasoning optimization actively suppressed self-transparency in some models, with reasoning variants showing up to 48.4% lower disclosure than base counterparts. Bayesian validation with Rogan--Gladen correction confirmed robustness to measurement error ($κ= 0.908$). These findings demonstrate transparency reflects training factors rather than scale. Organizations cannot assume safety properties transfer to deployment contexts, requiring deliberate behavior design and empirical verification.
☆ VacuumVLA: Boosting VLA Capabilities via a Unified Suction and Gripping Tool for Complex Robotic Manipulation
Hui Zhou, Siyuan Huang, Minxing Li, Hao Zhang, Lue Fan, Shaoshuai Shi
Vision Language Action models have significantly advanced general purpose robotic manipulation by harnessing large scale pretrained vision and language representations. Among existing approaches, a majority of current VLA systems employ parallel two finger grippers as their default end effectors. However, such grippers face inherent limitations in handling certain real world tasks such as wiping glass surfaces or opening drawers without handles due to insufficient contact area or lack of adhesion. To overcome these challenges, we present a low cost, integrated hardware design that combines a mechanical two finger gripper with a vacuum suction unit, enabling dual mode manipulation within a single end effector. Our system supports flexible switching or synergistic use of both modalities, expanding the range of feasible tasks. We validate the efficiency and practicality of our design within two state of the art VLA frameworks: DexVLA and Pi0. Experimental results demonstrate that with the proposed hybrid end effector, robots can successfully perform multiple complex tasks that are infeasible for conventional two finger grippers alone. All hardware designs and controlling systems will be released.
comment : 8 pages
☆ Predictive Safety Shield for Dyna-Q Reinforcement Learning
Jin Pin, Krasowski Hanna, Vanneaux Elena
Obtaining safety guarantees for reinforcement learning is a major challenge to achieve applicability for real-world tasks. Safety shields extend standard reinforcement learning and achieve hard safety guarantees. However, existing safety shields commonly use random sampling of safe actions or a fixed fallback controller, therefore disregarding future performance implications of different safe actions. In this work, we propose a predictive safety shield for model-based reinforcement learning agents in discrete space. Our safety shield updates the Q-function locally based on safe predictions, which originate from a safe simulation of the environment model. This shielding approach improves performance while maintaining hard safety guarantees. Our experiments on gridworld environments demonstrate that even short prediction horizons can be sufficient to identify the optimal path. We observe that our approach is robust to distribution shifts, e.g., between simulation and reality, without requiring additional training.
☆ Pessimistic Verification for Open Ended Math Questions
Yanxing Huang, Zihan Tang, Zejin Lin, Peng Li, Yang Liu
The key limitation of the verification performance lies in the ability of error detection. With this intuition we designed several variants of pessimistic verification, which are simple workflows that could significantly improve the verification of open-ended math questions. In pessimistic verification we construct multiple parallel verifications for the same proof, and the proof is deemed incorrect if any one of them reports an error. This simple technique significantly improves the performance across many math verification benchmarks without incurring substantial computational resources. Its token efficiency even surpassed extended long-CoT in test-time scaling. Our case studies further indicate that the majority of false negatives in stronger models are actually caused by annotation errors in the original dataset, so our method's performance is in fact underestimated. Self-verification for mathematical problems can effectively improve the reliability and performance of language model outputs, and it also plays a critical role in enabling long-horizon mathematical tasks. We believe that research on pessimistic verification will help enhance the mathematical capabilities of language models across a wide range of tasks.
☆ Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation LREC 2026
Lina Conti, Dennis Fucci, Marco Gaido, Matteo Negri, Guillaume Wisniewski, Luisa Bentivogli
Unlike text, speech conveys information about the speaker, such as gender, through acoustic cues like pitch. This gives rise to modality-specific bias concerns. For example, in speech translation (ST), when translating from languages with notional gender, such as English, into languages where gender-ambiguous terms referring to the speaker are assigned grammatical gender, the speaker's vocal characteristics may play a role in gender assignment. This risks misgendering speakers, whether through masculine defaults or vocal-based assumptions. Yet, how ST models make these decisions remains poorly understood. We investigate the mechanisms ST models use to assign gender to speaker-referring terms across three language pairs (en-es/fr/it), examining how training data patterns, internal language model (ILM) biases, and acoustic information interact. We find that models do not simply replicate term-specific gender associations from training data, but learn broader patterns of masculine prevalence. While the ILM exhibits strong masculine bias, models can override these preferences based on acoustic input. Using contrastive feature attribution on spectrograms, we reveal that the model with higher gender accuracy relies on a previously unknown mechanism: using first-person pronouns to link gendered terms back to the speaker, accessing gender information distributed across the frequency spectrum rather than concentrated in pitch.
comment : Submitted to LREC 2026
☆ Mechanistic Interpretability for Transformer -based Time Series Classification
Matīss Kalnāre, Sofoklis Kitharidis, Thomas Bäck, Niki van Stein
Transformer-based models have become state-of-the-art tools in various machine learning tasks, including time series classification, yet their complexity makes understanding their internal decision-making challenging. Existing explainability methods often focus on input-output attributions, leaving the internal mechanisms largely opaque. This paper addresses this gap by adapting various Mechanistic Interpretability techniques; activation patching, attention saliency, and sparse autoencoders, from NLP to transformer architectures designed explicitly for time series classification. We systematically probe the internal causal roles of individual attention heads and timesteps, revealing causal structures within these models. Through experimentation on a benchmark time series dataset, we construct causal graphs illustrating how information propagates internally, highlighting key attention heads and temporal positions driving correct classifications. Additionally, we demonstrate the potential of sparse autoencoders for uncovering interpretable latent features. Our findings provide both methodological contributions to transformer interpretability and novel insights into the functional mechanics underlying transformer performance in time series classification tasks.
☆ Tool-RoCo: An Agent -as-Tool Self-organization Large Language Model Benchmark in Multi-robot Cooperation
Ke Zhang, Xiaoning Zhao, Ce Zheng, Jiahong Ning, Dandan Zhu, Wenqi Zhang, Chen Sun, Toshiharu Sugawara
This study proposes Tool-RoCo, a novel benchmark for evaluating large language models (LLMs) in long-term multi-agent cooperation based on RoCo, a multi-robot cooperative benchmark. Recent research on LLM-based multi-agent systems has relied on predefined orchestration, while ignoring agent autonomy. Tool-RoCo treats other agents as tools and introduces cooperative tools, leveraging tool usage to evaluate multi-agent cooperation and self-organization. Tool usage means that each agent (LLM) selects a tool from a candidate set based on the current state, receives feedback, and adjusts its selection in subsequent rounds. To evaluate different autonomy levels, we propose four LLM paradigms: (1) centralized cooperation, where a single LLM allocates tools to all agents; (2) centralized self-organization, where a central LLM autonomously activates agents while keeping others inactive; (3) decentralized cooperation, where each agent has its own LLM and calls tools based on local information; and (4) self-organization, where a randomly chosen initial agent can request collaboration, activating additional agents via tool calls. Tool-RoCo includes three multi-robot tasks, SORT, PACK, and CABINET, to measure format and parameter accuracy and agent coordination through tool usage. The results using several LLMs showed that cooperative tools accounted for only 7.09% of all tools, indicating that LLM-based agents rarely invoked others as assistants. Moreover, activation tools accounted for 96.42%, suggesting that current LLMs tend to maintain active agents while seldom deactivating them for adaptive coordination. Tool-RoCo provides a systematic benchmark to evaluate LLM autonomy and cooperation in multi-agent tasks. Code and Demo: https://github.com/ColaZhang22/Tool-Roco
comment : 9 pages, 3 figures
☆ Merge and Bound: Direct Manipulations on Weights for Class Incremental Learning
Taehoon Kim, Donghwan Jang, Bohyung Han
We present a novel training approach, named Merge-and-Bound (M&B) for Class Incremental Learning (CIL), which directly manipulates model weights in the parameter space for optimization. Our algorithm involves two types of weight merging: inter-task weight merging and intra-task weight merging. Inter-task weight merging unifies previous models by averaging the weights of models from all previous stages. On the other hand, intra-task weight merging facilitates the learning of current task by combining the model parameters within current stage. For reliable weight merging, we also propose a bounded update technique that aims to optimize the target model with minimal cumulative updates and preserve knowledge from previous tasks; this strategy reveals that it is possible to effectively obtain new models near old ones, reducing catastrophic forgetting. M&B is seamlessly integrated into existing CIL methods without modifying architecture components or revising learning objectives. We extensively evaluate our algorithm on standard CIL benchmarks and demonstrate superior performance compared to state-of-the-art methods.
☆ Frequency-Aware Token Reduction for Efficient Vision Transformer
Dong-Jae Lee, Jiwan Hur, Jaehyun Choi, Jaemyung Yu, Junmo Kim
Vision Transformers have demonstrated exceptional performance across various computer vision tasks, yet their quadratic computational complexity concerning token length remains a significant challenge. To address this, token reduction methods have been widely explored. However, existing approaches often overlook the frequency characteristics of self-attention, such as rank collapsing and over-smoothing phenomenon. In this paper, we propose a frequency-aware token reduction strategy that improves computational efficiency while preserving performance by mitigating rank collapsing. Our method partitions tokens into high-frequency tokens and low-frequency tokens. high-frequency tokens are selectively preserved, while low-frequency tokens are aggregated into a compact direct current token to retain essential low-frequency components. Through extensive experiments and analysis, we demonstrate that our approach significantly improves accuracy while reducing computational overhead and mitigating rank collapsing and over smoothing. Furthermore, we analyze the previous methods, shedding light on their implicit frequency characteristics and limitations.
comment : Neurips 2025
☆ Going with the Speed of Sound: Pushing Neural Surrogates into Highly-turbulent Transonic Regimes NeurIPS 2025
Fabian Paischer, Leo Cotteleer, Yann Dreze, Richard Kurle, Dylan Rubini, Maurits Bleeker, Tobias Kronlachner, Johannes Brandstetter
The widespread use of neural surrogates in automotive aerodynamics, enabled by datasets such as DrivAerML and DrivAerNet++, has primarily focused on bluff-body flows with large wakes. Extending these methods to aerospace, particularly in the transonic regime, remains challenging due to the high level of non-linearity of compressible flows and 3D effects such as wingtip vortices. Existing aerospace datasets predominantly focus on 2D airfoils, neglecting these critical 3D phenomena. To address this gap, we present a new dataset of CFD simulations for 3D wings in the transonic regime. The dataset comprises volumetric and surface-level fields for around $30,000$ samples with unique geometry and inflow conditions. This allows computation of lift and drag coefficients, providing a foundation for data-driven aerodynamic optimization of the drag-lift Pareto front. We evaluate several state-of-the-art neural surrogates on our dataset, including Transolver and AB-UPT, focusing on their out-of-distribution (OOD) generalization over geometry and inflow variations. AB-UPT demonstrates strong performance for transonic flowfields and reproduces physically consistent drag-lift Pareto fronts even for unseen wing configurations. Our results demonstrate that AB-UPT can approximate drag-lift Pareto fronts for unseen geometries, highlighting its potential as an efficient and effective tool for rapid aerodynamic design exploration. To facilitate future research, we open-source our dataset at https://huggingface.co/datasets/EmmiAI/Emmi-Wing.
comment : NeurIPS 2025 ML4PS Workshop
☆ Hierarchical Ranking Neural Network for Long Document Readability Assessment
Yurui Zheng, Yijun Chen, Shaohong Zhang
Readability assessment aims to evaluate the reading difficulty of a text. In recent years, while deep learning technology has been gradually applied to readability assessment, most approaches fail to consider either the length of the text or the ordinal relationship of readability labels. This paper proposes a bidirectional readability assessment mechanism that captures contextual information to identify regions with rich semantic information in the text, thereby predicting the readability level of individual sentences. These sentence-level labels are then used to assist in predicting the overall readability level of the document. Additionally, a pairwise sorting algorithm is introduced to model the ordinal relationship between readability levels through label subtraction. Experimental results on Chinese and English datasets demonstrate that the proposed model achieves competitive performance and outperforms other baseline models.
☆ SpatialBench: Benchmark ing Multimodal Large Language Models for Spatial Cognition
Peiran Xu, Sudong Wang, Yao Zhu, Jianing Li, Yunjian Zhang
Spatial cognition is fundamental to real-world multimodal intelligence, allowing models to effectively interact with the physical environment. While multimodal large language models (MLLMs) have made significant strides, existing benchmarks often oversimplify spatial cognition, reducing it to a single-dimensional metric, which fails to capture the hierarchical structure and interdependence of spatial abilities. To address this gap, we propose a hierarchical spatial cognition framework that decomposes spatial intelligence into five progressively complex levels from basic observation to high-level planning. Building upon this taxonomy, we construct SpatialBench, a large-scale, fine-grained benchmark covering 15 tasks aligned with these cognitive levels. To provide a unified evaluation across heterogeneous tasks, we further introduce a high-level capability-oriented metric that reliably assesses a model's overall spatial reasoning ability. Extensive experiments over massive MLLMs reveal distinct performance stratification across cognitive levels: models exhibit strong perceptual grounding yet remain limited in symbolic reasoning, causal inference, and planning. Additional human tests demonstrate that humans perform selective, goal-directed abstraction, while MLLMs tend to over-attend to surface details without coherent spatial intent. Our work establishes the first systematic framework for measuring hierarchical spatial cognition in MLLMs, laying the foundation for future spatially intelligent systems.
☆ MADRA: Multi- Agent Debate for Risk-Aware Embodied Planning
Junjian Wang, Lidan Zhao, Xi Sheryl Zhang
Ensuring the safety of embodied AI agents during task planning is critical for real-world deployment, especially in household environments where dangerous instructions pose significant risks. Existing methods often suffer from either high computational costs due to preference alignment training or over-rejection when using single-agent safety prompts. To address these limitations, we propose MADRA, a training-free Multi-Agent Debate Risk Assessment framework that leverages collective reasoning to enhance safety awareness without sacrificing task performance. MADRA employs multiple LLM-based agents to debate the safety of a given instruction, guided by a critical evaluator that scores responses based on logical soundness, risk identification, evidence quality, and clarity. Through iterative deliberation and consensus voting, MADRA significantly reduces false rejections while maintaining high sensitivity to dangerous tasks. Additionally, we introduce a hierarchical cognitive collaborative planning framework that integrates safety, memory, planning, and self-evolution mechanisms to improve task success rates through continuous learning. We also contribute SafeAware-VH, a benchmark dataset for safety-aware task planning in VirtualHome, containing 800 annotated instructions. Extensive experiments on AI2-THOR and VirtualHome demonstrate that our approach achieves over 90% rejection of unsafe tasks while ensuring that safe-task rejection is low, outperforming existing methods in both safety and execution efficiency. Our work provides a scalable, model-agnostic solution for building trustworthy embodied agents.
☆ Constructing and Benchmark ing: a Labeled Email Dataset for Text-Based Phishing and Spam Detection Framework
Rebeka Toth, Tamas Bisztray, Richard Dubniczky
Phishing and spam emails remain a major cybersecurity threat, with attackers increasingly leveraging Large Language Models (LLMs) to craft highly deceptive content. This study presents a comprehensive email dataset containing phishing, spam, and legitimate messages, explicitly distinguishing between human- and LLM-generated content. Each email is annotated with its category, emotional appeal (e.g., urgency, fear, authority), and underlying motivation (e.g., link-following, credential theft, financial fraud). We benchmark multiple LLMs on their ability to identify these emotional and motivational cues and select the most reliable model to annotate the full dataset. To evaluate classification robustness, emails were also rephrased using several LLMs while preserving meaning and intent. A state-of-the-art LLM was then assessed on its performance across both original and rephrased emails using expert-labeled ground truth. The results highlight strong phishing detection capabilities but reveal persistent challenges in distinguishing spam from legitimate emails. Our dataset and evaluation framework contribute to improving AI-assisted email security systems. To support open science, all code, templates, and resources are available on our project site.
☆ EWE: An Agent ic Framework for Extreme Weather Analysis
Zhe Jiang, Jiong Wang, Xiaoyu Yue, Zijie Guo, Wenlong Zhang, Fenghua Ling, Wanli Ouyang, Lei Bai
Extreme weather events pose escalating risks to global society, underscoring the urgent need to unravel their underlying physical mechanisms. Yet the prevailing expert-driven, labor-intensive diagnostic paradigm has created a critical analytical bottleneck, stalling scientific progress. While AI for Earth Science has achieved notable advances in prediction, the equally essential challenge of automated diagnostic reasoning remains largely unexplored. We present the Extreme Weather Expert (EWE), the first intelligent agent framework dedicated to this task. EWE emulates expert workflows through knowledge-guided planning, closed-loop reasoning, and a domain-tailored meteorological toolkit. It autonomously produces and interprets multimodal visualizations from raw meteorological data, enabling comprehensive diagnostic analyses. To catalyze progress, we introduce the first benchmark for this emerging field, comprising a curated dataset of 103 high-impact events and a novel step-wise evaluation metric. EWE marks a step toward automated scientific discovery and offers the potential to democratize expertise and intellectual resources, particularly for developing nations vulnerable to extreme weather.
☆ EvRainDrop: HyperGraph-guided Completion for Effective Frame and Event Stream Aggregation
Futian Wang, Fan Zhang, Xiao Wang, Mengqi Wang, Dexing Huang, Jin Tang
Event cameras produce asynchronous event streams that are spatially sparse yet temporally dense. Mainstream event representation learning algorithms typically use event frames, voxels, or tensors as input. Although these approaches have achieved notable progress, they struggle to address the undersampling problem caused by spatial sparsity. In this paper, we propose a novel hypergraph-guided spatio-temporal event stream completion mechanism, which connects event tokens across different times and spatial locations via hypergraphs and leverages contextual information message passing to complete these sparse events. The proposed method can flexibly incorporate RGB tokens as nodes in the hypergraph within this completion framework, enabling multi-modal hypergraph-based information completion. Subsequently, we aggregate hypergraph node information across different time steps through self-attention, enabling effective learning and fusion of multi-modal features. Extensive experiments on both single- and multi-label event classification tasks fully validated the effectiveness of our proposed framework. The source code of this paper will be released on https://github.com/Event-AHU/EvRainDrop.
☆ Conversational no-code and multi- agent ic disease module identification and drug repurposing prediction with ChatDRex
Simon Süwer, Kester Bagemihl, Sylvie Baier, Lucia Dicunta, Markus List, Jan Baumbach, Andreas Maier, Fernando M. Delgado-Chaves
Repurposing approved drugs offers a time-efficient and cost-effective alternative to traditional drug development. However, in silico prediction of repurposing candidates is challenging and requires the effective collaboration of specialists in various fields, including pharmacology, medicine, biology, and bioinformatics. Fragmented, specialized algorithms and tools often address only narrow aspects of the overall problem, and heterogeneous, unstructured data landscapes require specialized users to be involved. Hence, these data services do not integrate smoothly across workflows. With ChatDRex, we present a conversation-based, multi-agent system that facilitates the execution of complex bioinformatic analyses aiming for network-based drug repurposing prediction. It builds on the integrated systems medicine knowledge graph NeDRex. ChatDRex provides natural language access to its extensive biomedical KG and integrates bioinformatics agents for network analysis and drug repurposing, complemented by agents for functional coherence evaluation for in silico validation, as well as agents for literature mining and for discussing the obtained results in a scientific context. Its flexible multi-agent design assigns specific tasks to specialized agents, including query routing, data retrieval, algorithm execution, and result visualization. A dedicated reasoning module keeps the user in the loop and allows for hallucination detection. By enabling physicians and researchers without computer science expertise to control complex analyses in natural language, ChatDRex democratizes access to bioinformatics as an important resource for drug repurposing. It enables clinical experts to generate hypotheses and explore drug repurposing opportunities, ultimately accelerating the discovery of novel therapies and advancing personalized medicine and translational research.
☆ From Observation to Action: Latent Action-based Primitive Segmentation for VLA Pre-train ing in Industrial Settings
Jiajie Zhang, Sören Schwertfeger, Alexander Kleiner
We present a novel unsupervised framework to unlock vast unlabeled human demonstration data from continuous industrial video streams for Vision-Language-Action (VLA) model pre-training. Our method first trains a lightweight motion tokenizer to encode motion dynamics, then employs an unsupervised action segmenter leveraging a novel "Latent Action Energy" metric to discover and segment semantically coherent action primitives. The pipeline outputs both segmented video clips and their corresponding latent action sequences, providing structured data directly suitable for VLA pre-training. Evaluations on public benchmarks and a proprietary electric motor assembly dataset demonstrate effective segmentation of key tasks performed by humans at workstations. Further clustering and quantitative assessment via a Vision-Language Model confirm the semantic coherence of the discovered action primitives. To our knowledge, this is the first fully automated end-to-end system for extracting and organizing VLA pre-training data from unstructured industrial videos, offering a scalable solution for embodied AI integration in manufacturing.
comment : 10 pages, 5 figures
☆ SAM Guided Semantic and Motion Changed Region Mining for Remote Sensing Change Captioning
Futian Wang, Mengqi Wang, Xiao Wang, Haowen Wang, Jin Tang
Remote sensing change captioning is an emerging and popular research task that aims to describe, in natural language, the content of interest that has changed between two remote sensing images captured at different times. Existing methods typically employ CNNs/Transformers to extract visual representations from the given images or incorporate auxiliary tasks to enhance the final results, with weak region awareness and limited temporal alignment. To address these issues, this paper explores the use of the SAM (Segment Anything Model) foundation model to extract region-level representations and inject region-of-interest knowledge into the captioning framework. Specifically, we employ a CNN/Transformer model to extract global-level vision features, leverage the SAM foundation model to delineate semantic- and motion-level change regions, and utilize a specially constructed knowledge graph to provide information about objects of interest. These heterogeneous sources of information are then fused via cross-attention, and a Transformer decoder is used to generate the final natural language description of the observed changes. Extensive experimental results demonstrate that our method achieves state-of-the-art performance across multiple widely used benchmark datasets. The source code of this paper will be released on https://github.com/Event-AHU/SAM_ChangeCaptioning
☆ New Hybrid Heuristics for Pseudo-Boolean Propagation
Mia Müßig, Jan Johannsen
In pseudo-boolean solving the currently most successful unit propagation strategy is a hybrid mode combining the watched literal scheme with the counting method. This short paper introduces new heuristics for this hybrid decision, which are able to drastically outperform the current method in the RoundingSAT solver.
☆ Automated Dynamic AI Inference Scaling on HPC-Infrastructure: Integrating Kubernetes, Slurm and vLLM
Tim Trappen, Robert Keßler, Roland Pabel, Viktor Achter, Stefan Wesner
Due to rising demands for Artificial Inteligence (AI) inference, especially in higher education, novel solutions utilising existing infrastructure are emerging. The utilisation of High-Performance Computing (HPC) has become a prevalent approach for the implementation of such solutions. However, the classical operating model of HPC does not adapt well to the requirements of synchronous, user-facing dynamic AI application workloads. In this paper, we propose our solution that serves LLMs by integrating vLLM, Slurm and Kubernetes on the supercomputer \textit{RAMSES}. The initial benchmark indicates that the proposed architecture scales efficiently for 100, 500 and 1000 concurrent requests, incurring only an overhead of approximately 500 ms in terms of end-to-end latency.
comment : 6 pages, 3 figures
☆ Subjective Depth and Timescale Transformer s: Learning Where and When to Compute
Frederico Wieser, Martin Benfeghoul, Haitham Bou Ammar, Jun Wang, Zafeirios Fountas
The rigid, uniform allocation of computation in standard Transformer (TF) architectures can limit their efficiency and scalability, particularly for large-scale models and long sequences. Addressing this, we introduce Subjective Depth Transformers (SDT) and Subjective Timescale Transformers (STT), two distinct architectures that leverage Bayesian surprise signals to dynamically route computation, learning where and when to compute within decoder-only TFs. SDT augments a decoder-only stack with alternating Decision and Dynamic layers: a Decision layer computes a full block 'posterior' and a lightweight 'prior,' while a Dynamic layer employs fixed-capacity Top-K routing based on Bayesian surprise (Expected and Unexpected Change), maintaining a static compute graph. STT extends this conditional computation to the temporal domain: a transition network predicts residual updates, forming a temporal 'change hypothesis' that informs a router to dynamically execute or bypass TF blocks for each token, managing KV-cache contributions. Both architectures exhibit the predicted shift from novelty to prediction driven gating over training, suggesting alignment with surprise based principles. While operating at reduced capacity, they offer preliminary insights into the compute-accuracy trade-offs of conditional computation. The proposed architectures establish a flexible framework for efficiency, reducing self-attention computation by 75% and KV-cache requirements by 50% within each compute skipping layer, setting a pathway for more efficient models.
☆ Training Introspective Behavior: Fine-Tuning Induces Reliable Internal State Detection in a 7B Model
Joshua Fonseca Rivera
Lindsey (2025) investigates introspective awareness in language models through four experiments, finding that models can sometimes detect and identify injected activation patterns -- but unreliably (~20% success in the best model). We focus on the first of these experiments -- self-report of injected "thoughts" -- and ask whether this capability can be directly trained rather than waiting for emergence. Through fine-tuning on transient single-token injections, we transform a 7B parameter model from near-complete failure (0.4% accuracy, 6.7% false positive rate) to reliable detection (85% accuracy on held-out concepts at α=40, 0% false positives). Our model detects fleeting "thoughts" injected at a single token position, retains that information, and reports the semantic content across subsequent generation steps. On this task, our trained model satisfies three of Lindsey's criteria: accuracy (correct identification), grounding (0/60 false positives), and internality (detection precedes verbalization). Generalization to unseen concept vectors (7.5pp gap) demonstrates the model learns a transferable skill rather than memorizing specific vectors, though this does not establish metacognitive representation in Lindsey's sense. These results address an open question raised by Lindsey: whether "training for introspection would help eliminate cross-model differences." We show that at least one component of introspective behavior can be directly induced, offering a pathway to built-in AI transparency.
comment : 16 pages, 8 figures
☆ Prune4Web: DOM Tree Pruning Programming for Web Agent AAAI 2026
Jiayuan Zhang, Kaiquan Chen, Zhihao Lu, Enshen Zhou, Qian Yu, Jing Zhang
Web automation employs intelligent agents to execute high-level tasks by mimicking human interactions with web interfaces. Despite the capabilities of recent Large Language Model (LLM)-based web agents, navigating complex, real-world webpages efficiently remains a significant hurdle due to the prohibitively large size of Document Object Model (DOM) structures, often ranging from 10,000 to 100,000 tokens. Existing strategies typically rely on crude DOM truncation -- risking the loss of critical information -- or employ inefficient heuristics and separate ranking models, failing to achieve an optimal balance between precision and scalability. To address these challenges, we introduce Prune4Web, a novel paradigm that shifts DOM processing from resource-intensive LLM reading to efficient programmatic pruning. Central to our approach is DOM Tree Pruning Programming, where an LLM generates executable Python scoring scripts to dynamically filter DOM elements based on semantic cues from decomposed sub-tasks. This mechanism eliminates the need for LLMs to ingest raw, massive DOMs, instead delegating traversal and scoring to lightweight, interpretable programs. This methodology achieves a 25x to 50x reduction in candidate elements for grounding, thereby facilitating precise action localization while mitigating attention dilution. Furthermore, we propose a specialized data annotation pipeline and a two-turn dialogue training strategy that jointly optimizes the Planner, Programmatic Filter, and Grounder within a unified framework. Extensive experiments demonstrate state-of-the-art performance. Notably, on our low-level grounding task, Prune4Web dramatically improves accuracy from 46.8% to 88.28%, underscoring its efficacy in real-world web automation.
comment : Paper accepted to AAAI 2026
☆ Do Reasoning Vision -Language Models Inversely Scale in Test-Time Compute? A Distractor-centric Empirical Analysis
Jiyun Bae, Hyunjong Ok, Sangwoo Mo, Jaeho Lee
How does irrelevant information (i.e., distractors) affect test-time scaling in vision-language models (VLMs)? Prior studies on language models have reported an inverse scaling effect, where textual distractors lead to longer but less effective reasoning. To investigate whether similar phenomena occur in multimodal settings, we introduce Idis (Images with distractors), a visual question-answering dataset that systematically varies distractors along semantic, numerical, and spatial dimensions. Our analyses reveal that visual distractors differ fundamentally from textual ones: although inverse scaling persists, adding visual distractors reduces accuracy without increasing reasoning length. We further show that tracking attribute counts within reasoning traces provides key insights into how distractors, reasoning length, and accuracy interact. Finally, we demonstrate that these trends extend to established visual bias benchmarks such as Waterbirds, and we propose a simple prompting strategy to mitigate bias-driven predictions in reasoning models.
comment : preprint
☆ Monet: Reasoning in Latent Visual Space Beyond Images and Language
Qixun Wang, Yang Shi, Yifei Wang, Yuanxing Zhang, Pengfei Wan, Kun Gai, Xianghua Ying, Yisen Wang
"Thinking with images" has emerged as an effective paradigm for advancing visual reasoning, extending beyond text-only chains of thought by injecting visual evidence into intermediate reasoning steps. However, existing methods fall short of human-like abstract visual thinking, as their flexibility is fundamentally limited by external tools. In this work, we introduce Monet, a training framework that enables multimodal large language models (MLLMs) to reason directly within the latent visual space by generating continuous embeddings that function as intermediate visual thoughts. We identify two core challenges in training MLLMs for latent visual reasoning: high computational cost in latent-vision alignment and insufficient supervision over latent embeddings, and address them with a three-stage distillation-based supervised fine-tuning (SFT) pipeline. We further reveal a limitation of applying GRPO to latent reasoning: it primarily enhances text-based reasoning rather than latent reasoning. To overcome this, we propose VLPO (Visual-latent Policy Optimization), a reinforcement learning method that explicitly incorporates latent embeddings into policy gradient updates. To support SFT, we construct Monet-SFT-125K, a high-quality text-image interleaved CoT dataset containing 125K real-world, chart, OCR, and geometry CoTs. Our model, Monet-7B, shows consistent gains across real-world perception and reasoning benchmarks and exhibits strong out-of-distribution generalization on challenging abstract visual reasoning tasks. We also empirically analyze the role of each training component and discuss our early unsuccessful attempts, providing insights for future developments in visual latent reasoning. Our model, data, and code are available at https://github.com/NOVAglow646/Monet.
☆ RIA: A Ranking-Infused Approach for Optimized listwise CTR Prediction
Guoxiao Zhang, Tan Qu, Ao Li, DongLin Ni, Qianlong Xie, Xingxing Wang
Reranking improves recommendation quality by modeling item interactions. However, existing methods often decouple ranking and reranking, leading to weak listwise evaluation models that suffer from combinatorial sparsity and limited representational power under strict latency constraints. In this paper, we propose RIA (Ranking-Infused Architecture), a unified, end-to-end framework that seamlessly integrates pointwise and listwise evaluation. RIA introduces four key components: (1) the User and Candidate DualTransformer (UCDT) for fine-grained user-item-context modeling; (2) the Context-aware User History and Target (CUHT) module for position-sensitive preference learning; (3) the Listwise Multi-HSTU (LMH) module to capture hierarchical item dependencies; and (4) the Embedding Cache (EC) module to bridge efficiency and effectiveness during inference. By sharing representations across ranking and reranking, RIA enables rich contextual knowledge transfer while maintaining low latency. Extensive experiments show that RIA outperforms state-of-the-art models on both public and industrial datasets, achieving significant gains in AUC and LogLoss. Deployed in Meituan advertising system, RIA yields a +1.69% improvement in Click-Through Rate (CTR) and a +4.54% increase in Cost Per Mille (CPM) in online A/B tests.
☆ FITRep: Attention-Guided Item Representation via MLLMs
Guoxiao Zhang, Ao Li, Tan Qu, Qianlong Xie, Xingxing Wang
Online platforms usually suffer from user experience degradation due to near-duplicate items with similar visuals and text. While Multimodal Large Language Models (MLLMs) enable multimodal embedding, existing methods treat representations as black boxes, ignoring structural relationships (e.g., primary vs. auxiliary elements), leading to local structural collapse problem. To address this, inspired by Feature Integration Theory (FIT), we propose FITRep, the first attention-guided, white-box item representation framework for fine-grained item deduplication. FITRep consists of: (1) Concept Hierarchical Information Extraction (CHIE), using MLLMs to extract hierarchical semantic concepts; (2) Structure-Preserving Dimensionality Reduction (SPDR), an adaptive UMAP-based method for efficient information compression; and (3) FAISS-Based Clustering (FBC), a FAISS-based clustering that assigns each item a unique cluster id using FAISS. Deployed on Meituan's advertising system, FITRep achieves +3.60% CTR and +4.25% CPM gains in online A/B tests, demonstrating both effectiveness and real-world impact.
☆ Anomaly Detection with Adaptive and Aggressive Rejection for Contaminated Training Data
Jungi Lee, Jungkwon Kim, Chi Zhang, Kwangsun Yoo, Seok-Joo Byun
Handling contaminated data poses a critical challenge in anomaly detection, as traditional models assume training on purely normal data. Conventional methods mitigate contamination by relying on fixed contamination ratios, but discrepancies between assumed and actual ratios can severely degrade performance, especially in noisy environments where normal and abnormal data distributions overlap. To address these limitations, we propose Adaptive and Aggressive Rejection (AAR), a novel method that dynamically excludes anomalies using a modified z-score and Gaussian mixture model-based thresholds. AAR effectively balances the trade-off between preserving normal data and excluding anomalies by integrating hard and soft rejection strategies. Extensive experiments on two image datasets and thirty tabular datasets demonstrate that AAR outperforms the state-of-the-art method by 0.041 AUROC. By providing a scalable and reliable solution, AAR enhances robustness against contaminated datasets, paving the way for broader real-world applications in domains such as security and healthcare.
☆ The Directed Prediction Change - Efficient and Trustworthy Fidelity Assessment for Local Feature Attribution Methods AAAI
Kevin Iselborn, David Dembinsky, Adriano Lucieri, Andreas Dengel
The utility of an explanation method critically depends on its fidelity to the underlying machine learning model. Especially in high-stakes medical settings, clinicians and regulators require explanations that faithfully reflect the model's decision process. Existing fidelity metrics such as Infidelity rely on Monte Carlo approximation, which demands numerous model evaluations and introduces uncertainty due to random sampling. This work proposes a novel metric for evaluating the fidelity of local feature attribution methods by modifying the existing Prediction Change (PC) metric within the Guided Perturbation Experiment. By incorporating the direction of both perturbation and attribution, the proposed Directed Prediction Change (DPC) metric achieves an almost tenfold speedup and eliminates randomness, resulting in a deterministic and trustworthy evaluation procedure that measures the same property as local Infidelity. DPC is evaluated on two datasets (skin lesion images and financial tabular data), two black-box models, seven explanation algorithms, and a wide range of hyperparameters. Across $4\,744$ distinct explanations, the results demonstrate that DPC, together with PC, enables a holistic and computationally efficient evaluation of both baseline-oriented and local feature attribution methods, while providing deterministic and reproducible outcomes.
comment : 13 pages, 10 figures, 5 tables, accepted at AAAI SECURE-AI4H workshop
☆ Hybrid-AI RL : Enhancing Inverse Reinforcement Learning with Supervised Expert Guidance
Bram Silue, Santiago Amaya-Corredor, Patrick Mannion, Lander Willem, Pieter Libin
Adversarial Inverse Reinforcement Learning (AIRL) has shown promise in addressing the sparse reward problem in reinforcement learning (RL) by inferring dense reward functions from expert demonstrations. However, its performance in highly complex, imperfect-information settings remains largely unexplored. To explore this gap, we evaluate AIRL in the context of Heads-Up Limit Hold'em (HULHE) poker, a domain characterized by sparse, delayed rewards and significant uncertainty. In this setting, we find that AIRL struggles to infer a sufficiently informative reward function. To overcome this limitation, we contribute Hybrid-AIRL (H-AIRL), an extension that enhances reward inference and policy learning by incorporating a supervised loss derived from expert data and a stochastic regularization mechanism. We evaluate H-AIRL on a carefully selected set of Gymnasium benchmarks and the HULHE poker setting. Additionally, we analyze the learned reward function through visualization to gain deeper insights into the learning process. Our experimental results show that H-AIRL achieves higher sample efficiency and more stable learning compared to AIRL. This highlights the benefits of incorporating supervised signals into inverse RL and establishes H-AIRL as a promising framework for tackling challenging, real-world settings.
comment : Comments: 13 pages, 5 figures, 1 table. Code: https://github.com/silue-dev/hairl. Submitted to ESANN 2026
☆ Generating Separated Singing Vocals Using a Diffusion Model Conditioned on Music Mixtures SP
Genís Plaja-Roglans, Yun-Ning Hung, Xavier Serra, Igor Pereira
Separating the individual elements in a musical mixture is an essential process for music analysis and practice. While this is generally addressed using neural networks optimized to mask or transform the time-frequency representation of a mixture to extract the target sources, the flexibility and generalization capabilities of generative diffusion models are giving rise to a novel class of solutions for this complicated task. In this work, we explore singing voice separation from real music recordings using a diffusion model which is trained to generate the solo vocals conditioned on the corresponding mixture. Our approach improves upon prior generative systems and achieves competitive objective scores against non-generative baselines when trained with supplementary data. The iterative nature of diffusion sampling enables the user to control the quality-efficiency trade-off, and also refine the output when needed. We present an ablation study of the sampling algorithm, highlighting the effects of the user-configurable parameters.
comment : Accepted for publication at WASPAA 2025
☆ SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understanding
Tae-Min Choi, Tae Kyeong Jeong, Garam Kim, Jaemin Lee, Yeongyoon Koh, In Cheul Choi, Jae-Ho Chung, Jong Woong Park, Juyoun Park
Recent advances in multimodal large language models (LLMs) have highlighted their potential for medical and surgical applications. However, existing surgical datasets predominantly adopt a Visual Question Answering (VQA) format with heterogeneous taxonomies and lack support for pixel-level segmentation, limiting consistent evaluation and applicability. We present SurgMLLMBench, a unified multimodal benchmark explicitly designed for developing and evaluating interactive multimodal LLMs for surgical scene understanding, including the newly collected Micro-surgical Artificial Vascular anastomosIS (MAVIS) dataset. It integrates pixel-level instrument segmentation masks and structured VQA annotations across laparoscopic, robot-assisted, and micro-surgical domains under a unified taxonomy, enabling comprehensive evaluation beyond traditional VQA tasks and richer visual-conversational interactions. Extensive baseline experiments show that a single model trained on SurgMLLMBench achieves consistent performance across domains and generalizes effectively to unseen datasets. SurgMLLMBench will be publicly released as a robust resource to advance multimodal surgical AI research, supporting reproducible evaluation and development of interactive surgical reasoning models.
comment : 10 pages, 5 figures
☆ Hybrid SIFT-SNN for Efficient Anomaly Detection of Traffic Flow-Control Infrastructure
Munish Rathee, Boris Bačić, Maryam Doborjeh
This paper presents the SIFT-SNN framework, a low-latency neuromorphic signal-processing pipeline for real-time detection of structural anomalies in transport infrastructure. The proposed approach integrates Scale-Invariant Feature Transform (SIFT) for spatial feature encoding with a latency-driven spike conversion layer and a Leaky Integrate-and-Fire (LIF) Spiking Neural Network (SNN) for classification. The Auckland Harbour Bridge dataset is recorded under various weather and lighting conditions, comprising 6,000 labelled frames that include both real and synthetically augmented unsafe cases. The presented system achieves a classification accuracy of 92.3% (+- 0.8%) with a per-frame inference time of 9.5 ms. Achieved sub-10 millisecond latency, combined with sparse spike activity (8.1%), enables real-time, low-power edge deployment. Unlike conventional CNN-based approaches, the hybrid SIFT-SNN pipeline explicitly preserves spatial feature grounding, enhances interpretability, supports transparent decision-making, and operates efficiently on embedded hardware. Although synthetic augmentation improved robustness, generalisation to unseen field conditions remains to be validated. The SIFT-SNN framework is validated through a working prototype deployed on a consumer-grade system and framed as a generalisable case study in structural safety monitoring for movable concrete barriers, which, as a traffic flow-control infrastructure, is deployed in over 20 cities worldwide.
comment : 8 pages, 6 figures. This is a preprint of a paper accepted for presentation at the 2025 International Conference on Image and Vision Computing New Zealand (IVCNZ). The final version will appear in IEEE Xplore
☆ The More, the Merrier: Contrastive Fusion for Higher-Order Multimodal Alignment
Stefanos Koutoupis, Michaela Areti Zervou, Konstantinos Kontras, Maarten De Vos, Panagiotis Tsakalides, Grigorios Tsagatakis
Learning joint representations across multiple modalities remains a central challenge in multimodal machine learning. Prevailing approaches predominantly operate in pairwise settings, aligning two modalities at a time. While some recent methods aim to capture higher-order interactions among multiple modalities, they often overlook or insufficiently preserve pairwise relationships, limiting their effectiveness on single-modality tasks. In this work, we introduce Contrastive Fusion (ConFu), a framework that jointly embeds both individual modalities and their fused combinations into a unified representation space, where modalities and their fused counterparts are aligned. ConFu extends traditional pairwise contrastive objectives with an additional fused-modality contrastive term, encouraging the joint embedding of modality pairs with a third modality. This formulation enables ConFu to capture higher-order dependencies, such as XOR-like relationships, that cannot be recovered through pairwise alignment alone, while still maintaining strong pairwise correspondence. We evaluate ConFu on synthetic and real-world multimodal benchmarks, assessing its ability to exploit cross-modal complementarity, capture higher-order dependencies, and scale with increasing multimodal complexity. Across these settings, ConFu demonstrates competitive performance on retrieval and classification tasks, while supporting unified one-to-one and two-to-one retrieval within a single contrastive framework.
☆ SONAR: Spectral- Contrastive Audio Residuals for Generalizable Deepfake Detection
Ido Nitzan HIdekel, Gal lifshitz, Khen Cohen, Dan Raviv
Deepfake (DF) audio detectors still struggle to generalize to out of distribution inputs. A central reason is spectral bias, the tendency of neural networks to learn low-frequency structure before high-frequency (HF) details, which both causes DF generators to leave HF artifacts and leaves those same artifacts under-exploited by common detectors. To address this gap, we propose Spectral-cONtrastive Audio Residuals (SONAR), a frequency-guided framework that explicitly disentangles an audio signal into complementary representations. An XLSR encoder captures the dominant low-frequency content, while the same cloned path, preceded by learnable SRM, value-constrained high-pass filters, distills faint HF residuals. Frequency cross-attention reunites the two views for long- and short-range frequency dependencies, and a frequency-aware Jensen-Shannon contrastive loss pulls real content-noise pairs together while pushing fake embeddings apart, accelerating optimization and sharpening decision boundaries. Evaluated on the ASVspoof 2021 and in-the-wild benchmarks, SONAR attains state-of-the-art performance and converges four times faster than strong baselines. By elevating faint high-frequency residuals to first-class learning signals, SONAR unveils a fully data-driven, frequency-guided contrastive framework that splits the latent space into two disjoint manifolds: natural-HF for genuine audio and distorted-HF for synthetic audio, thereby sharpening decision boundaries. Because the scheme operates purely at the representation level, it is architecture-agnostic and, in future work, can be seamlessly integrated into any model or modality where subtle high-frequency cues are decisive.
☆ TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories
Kirti Bhagat, Shaily Bhatt, Athul Velagapudi, Aditya Vashistha, Shachi Dave, Danish Pruthi
Millions of users across the globe turn to AI chatbots for their creative needs, inviting widespread interest in understanding how such chatbots represent diverse cultures. At the same time, evaluating cultural representations in open-ended tasks remains challenging and underexplored. In this work, we present TALES, an evaluation of cultural misrepresentations in LLM-generated stories for diverse Indian cultural identities. First, we develop TALES-Tax, a taxonomy of cultural misrepresentations by collating insights from participants with lived experiences in India through focus groups (N=9) and individual surveys (N=15). Using TALES-Tax, we evaluate 6 models through a large-scale annotation study spanning 2,925 annotations from 108 annotators with lived cultural experience from across 71 regions in India and 14 languages. Concerningly, we find that 88\% of the generated stories contain one or more cultural inaccuracies, and such errors are more prevalent in mid- and low-resourced languages and stories based in peri-urban regions in India. Lastly, we transform the annotations into TALES-QA, a standalone question bank to evaluate the cultural knowledge of foundational models. Through this evaluation, we surprisingly discover that models often possess the requisite cultural knowledge despite generating stories rife with cultural misrepresentations.
☆ Improvement of Collision Avoidance in Cut-In Maneuvers Using Time-to-Collision Metrics
Jamal Raiyn
This paper proposes a new strategy for collision avoidance system leveraging Time-to-Collision (TTC) metrics for handling cut-in scenarios, which are particularly challenging for autonomous vehicles (AVs). By integrating a deep learning with TTC calculations, the system predicts potential collisions and determines appropriate evasive actions compared to traditional TTC -based approaches.
☆ Causality Without Causal Models
Joseph Y. Halpern, Rafael Pass
Perhaps the most prominent current definition of (actual) causality is due to Halpern and Pearl. It is defined using causal models (also known as structural equations models). We abstract the definition, extracting its key features, so that it can be applied to any other model where counterfactuals are defined. By abstracting the definition, we gain a number of benefits. Not only can we apply the definition in a wider range of models, including ones that allow, for example, backtracking, but we can apply the definition to determine if A is a cause of B even if A and B are formulas involving disjunctions, negations, beliefs, and nested counterfactuals (none of which can be handled by the Halpern-Pearl definition). Moreover, we can extend the ideas to getting an abstract definition of explanation that can be applied beyond causal models. Finally, we gain a deeper understanding of features of the definition even in causal models.
comment : In Proceedings TARK 2025, arXiv:2511.20540
☆ Self-Guided Defense: Adaptive Safety Alignment for Reasoning Models via Synthesized Guidelines
Yuhang Wang, Yanxu Zhu, Dongyuan Lu, Jitao Sang
Reasoning models have demonstrated remarkable capabilities in complex reasoning tasks. However, ensuring their safety against adversarial jailbreak prompts remains a critical challenge. Due to the covert and deceptive nature of such prompts, they can often evade built-in safety mechanisms and lead to the generation of harmful content. This underscores the need for an adaptive safety alignment approach that enables models to autonomously reinforce their defenses in response to adversarial inputs. This paper introduces the Synthesized Guideline-based Adaptive Safety Alignment (SGASA) framework, which internalizes model-generated safety guidelines to strengthen models' ability to enhance robustness against harmful adversarial prompts while minimizing unnecessary refusals of benign requests. SGASA consists of two key stages: Data Pre-synthesis, which generates safety guidelines and augmented prompts; and Alignment Fine-tuning, which leverages Supervised Fine-tuning (SFT) and Direct Preference Optimization (DPO) to embed these guidelines into the model. Extensive experiments across multiple datasets demonstrate that SGASA significantly improves model safety, validating its adaptive and scalable effectiveness.
☆ BotaCLIP: Contrastive Learning for Botany-Aware Representation of Earth Observation Data
Selene Cerna, Sara Si-Moussi, Wilfried Thuiller, Hadrien Hendrikx, Vincent Miele
Foundation models have demonstrated a remarkable ability to learn rich, transferable representations across diverse modalities such as images, text, and audio. In modern machine learning pipelines, these representations often replace raw data as the primary input for downstream tasks. In this paper, we address the challenge of adapting a pre-trained foundation model to inject domain-specific knowledge, without retraining from scratch or incurring significant computational costs. To this end, we introduce BotaCLIP, a lightweight multimodal contrastive framework that adapts a pre-trained Earth Observation foundation model (DOFA) by aligning high-resolution aerial imagery with botanical relevés. Unlike generic embeddings, BotaCLIP internalizes ecological structure through contrastive learning with a regularization strategy that mitigates catastrophic forgetting. Once trained, the resulting embeddings serve as transferable representations for downstream predictors. Motivated by real-world applications in biodiversity modeling, we evaluated BotaCLIP representations in three ecological tasks: plant presence prediction, butterfly occurrence modeling, and soil trophic group abundance estimation. The results showed consistent improvements over those derived from DOFA and supervised baselines. More broadly, this work illustrates how domain-aware adaptation of foundation models can inject expert knowledge into data-scarce settings, enabling frugal representation learning.
☆ When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision -Language-Action Models
Hui Lu, Yi Yu, Yiming Yang, Chenyu Yi, Qixin Zhang, Bingquan Shen, Alex C. Kot, Xudong Jiang
Vision-Language-Action (VLA) models are vulnerable to adversarial attacks, yet universal and transferable attacks remain underexplored, as most existing patches overfit to a single model and fail in black-box settings. To address this gap, we present a systematic study of universal, transferable adversarial patches against VLA-driven robots under unknown architectures, finetuned variants, and sim-to-real shifts. We introduce UPA-RFAS (Universal Patch Attack via Robust Feature, Attention, and Semantics), a unified framework that learns a single physical patch in a shared feature space while promoting cross-model transfer. UPA-RFAS combines (i) a feature-space objective with an $\ell_1$ deviation prior and repulsive InfoNCE loss to induce transferable representation shifts, (ii) a robustness-augmented two-phase min-max procedure where an inner loop learns invisible sample-wise perturbations and an outer loop optimizes the universal patch against this hardened neighborhood, and (iii) two VLA-specific losses: Patch Attention Dominance to hijack text$\to$vision attention and Patch Semantic Misalignment to induce image-text mismatch without labels. Experiments across diverse VLA models, manipulation suites, and physical executions show that UPA-RFAS consistently transfers across models, tasks, and viewpoints, exposing a practical patch-based attack surface and establishing a strong baseline for future defenses.
☆ Progress by Pieces: Test-Time Scaling for Autoregressive Image Generation
Joonhyung Park, Hyeongwon Jang, Joowon Kim, Eunho Yang
Recent visual autoregressive (AR) models have shown promising capabilities in text-to-image generation, operating in a manner similar to large language models. While test-time computation scaling has brought remarkable success in enabling reasoning-enhanced outputs for challenging natural language tasks, its adaptation to visual AR models remains unexplored and poses unique challenges. Naively applying test-time scaling strategies such as Best-of-N can be suboptimal: they consume full-length computation on erroneous generation trajectories, while the raster-scan decoding scheme lacks a blueprint of the entire canvas, limiting scaling benefits as only a few prompt-aligned candidates are generated. To address these, we introduce GridAR, a test-time scaling framework designed to elicit the best possible results from visual AR models. GridAR employs a grid-partitioned progressive generation scheme in which multiple partial candidates for the same position are generated within a canvas, infeasible ones are pruned early, and viable ones are fixed as anchors to guide subsequent decoding. Coupled with this, we present a layout-specified prompt reformulation strategy that inspects partial views to infer a feasible layout for satisfying the prompt. The reformulated prompt then guides subsequent image generation to mitigate the blueprint deficiency. Together, GridAR achieves higher-quality results under limited test-time scaling: with N=4, it even outperforms Best-of-N (N=8) by 14.4% on T2I-CompBench++ while reducing cost by 25.6%. It also generalizes to autoregressive image editing, showing comparable edit quality and a 13.9% gain in semantic preservation on PIE-Bench over larger-N baselines.
comment : Project page: https://grid-ar.github.io/
☆ Privacy in Federated Learning with Spiking Neural Networks
Dogukan Aksu, Jesus Martinez del Rincon, Ihsen Alouani
Spiking neural networks (SNNs) have emerged as prominent candidates for embedded and edge AI. Their inherent low power consumption makes them far more efficient than conventional ANNs in scenarios where energy budgets are tightly constrained. In parallel, federated learning (FL) has become the prevailing training paradigm in such settings, enabling on-device learning while limiting the exposure of raw data. However, gradient inversion attacks represent a critical privacy threat in FL, where sensitive training data can be reconstructed directly from shared gradients. While this vulnerability has been widely investigated in conventional ANNs, its implications for SNNs remain largely unexplored. In this work, we present the first comprehensive empirical study of gradient leakage in SNNs across diverse data domains. SNNs are inherently non-differentiable and are typically trained using surrogate gradients, which we hypothesized would be less correlated with the original input and thus less informative from a privacy perspective. To investigate this, we adapt different gradient leakage attacks to the spike domain. Our experiments reveal a striking contrast with conventional ANNs: whereas ANN gradients reliably expose salient input content, SNN gradients yield noisy, temporally inconsistent reconstructions that fail to recover meaningful spatial or temporal structure. These results indicate that the combination of event-driven dynamics and surrogate-gradient training substantially reduces gradient informativeness. To the best of our knowledge, this work provides the first systematic benchmark of gradient inversion attacks for spiking architectures, highlighting the inherent privacy-preserving potential of neuromorphic computation.
☆ CAHS-Attack: CLIP-Aware Heuristic Search Attack Method for Stable Diffusion
Shuhan Xia, Jing Dai, Hui Ouyang, Yadong Shang, Dongxiao Zhao, Peipei Li
Diffusion models exhibit notable fragility when faced with adversarial prompts, and strengthening attack capabilities is crucial for uncovering such vulnerabilities and building more robust generative systems. Existing works often rely on white-box access to model gradients or hand-crafted prompt engineering, which is infeasible in real-world deployments due to restricted access or poor attack effect. In this paper, we propose CAHS-Attack , a CLIP-Aware Heuristic Search attack method. CAHS-Attack integrates Monte Carlo Tree Search (MCTS) to perform fine-grained suffix optimization, leveraging a constrained genetic algorithm to preselect high-potential adversarial prompts as root nodes, and retaining the most semantically disruptive outcome at each simulation rollout for efficient local search. Extensive experiments demonstrate that our method achieves state-of-the-art attack performance across both short and long prompts of varying semantics. Furthermore, we find that the fragility of SD models can be attributed to the inherent vulnerability of their CLIP-based text encoders, suggesting a fundamental security risk in current text-to-image pipelines.
☆ LLaVA -UHD v3: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs
Shichu Sun, Yichen Zhang, Haolin Song, Zonghao Guo, Chi Chen, Yidan Zhang, Yuan Yao, Zhiyuan Liu, Maosong Sun
Visual encoding followed by token condensing has become the standard architectural paradigm in multi-modal large language models (MLLMs). Many recent MLLMs increasingly favor global native- resolution visual encoding over slice-based methods. To investigate this trend, we systematically compare their behavior on vision-language understanding and attention patterns, revealing that global encoding enhances overall capability but at the expense of greater computational overhead. To address this issue, we present LLaVA-UHD v3, an MLLM centered upon our proposed Progressive Visual Compression (PVC) method, which can be seamlessly integrated into standard Vision Transformer (ViT) to enable efficient native-resolution encoding. The PVC approach consists of two key modules: (i) refined patch embedding, which supports flexible patch-size scaling for fine-grained visual model- ing, (ii) windowed token compression, hierarchically deployed across ViT layers to progressively aggregate local token representations. Jointly modulated by these two modules, a widely pretrained ViT can be reconfigured into an efficient architecture while largely preserving generality. Evaluated across extensive benchmarks, the transformed ViT, termed ViT-UHD, demonstrates competitive performance with MoonViT while reducing TTFT (time-to-first-token) by 2.4x, when developed within an identical MLLM architecture. Building upon ViT-UHD, LLaVA-UHD v3 also achieves competitive performance to Qwen2-VL, while further reducing TTFT by 1.9x. We will release all code and checkpoints to support future research on efficient MLLMs.
☆ Maglev-Pentabot: Magnetic Levitation System for Non-Contact Manipulation using Deep Reinforcement Learning
Guoming Huang, Qingyi Zhou, Dianjing Liu, Shuai Zhang, Ming Zhou, Zongfu Yu
Non-contact manipulation has emerged as a transformative approach across various industrial fields. However, current flexible 2D and 3D non-contact manipulation techniques are often limited to microscopic scales, typically controlling objects in the milligram range. In this paper, we present a magnetic levitation system, termed Maglev-Pentabot, designed to address this limitation. The Maglev-Pentabot leverages deep reinforcement learning (DRL) to develop complex control strategies for manipulating objects in the gram range. Specifically, we propose an electromagnet arrangement optimized through numerical analysis to maximize controllable space. Additionally, an action remapping method is introduced to address sample sparsity issues caused by the strong nonlinearity in magnetic field intensity, hence allowing the DRL controller to converge. Experimental results demonstrate flexible manipulation capabilities, and notably, our system can generalize to transport tasks it has not been explicitly trained for. Furthermore, our approach can be scaled to manipulate heavier objects using larger electromagnets, offering a reference framework for industrial-scale robotic applications.
☆ Efficient Training for Human Video Generation with Entropy-Guided Prioritized Progressive Learning
Changlin Li, Jiawei Zhang, Shuhao Liu, Sihao Lin, Zeyi Shi, Zhihui Li, Xiaojun Chang
Human video generation has advanced rapidly with the development of diffusion models, but the high computational cost and substantial memory consumption associated with training these models on high-resolution, multi-frame data pose significant challenges. In this paper, we propose Entropy-Guided Prioritized Progressive Learning (Ent-Prog), an efficient training framework tailored for diffusion models on human video generation. First, we introduce Conditional Entropy Inflation (CEI) to assess the importance of different model components on the target conditional generation task, enabling prioritized training of the most critical components. Second, we introduce an adaptive progressive schedule that adaptively increases computational complexity during training by measuring the convergence efficiency. Ent-Prog reduces both training time and GPU memory consumption while maintaining model performance. Extensive experiments across three datasets, demonstrate the effectiveness of Ent-Prog, achieving up to 2.2$\times$ training speedup and 2.4$\times$ GPU memory reduction without compromising generative performance.
comment : Project page: https://github.com/changlin31/Ent-Prog
☆ SocialNav: Training Human-Inspired Foundation Model for Socially-Aware Embodied Navigation
Ziyi Chen, Yingnan Guo, Zedong Chu, Minghua Luo, Yanfen Shen, Mingchao Sun, Junjun Hu, Shichao Xie, Kuan Yang, Pei Shi, Zhining Gu, Lu Liu, Honglin Han, Xiaolong Wu, Mu Xu, Yu Zhang
Embodied navigation that adheres to social norms remains an open research challenge. Our \textbf{SocialNav} is a foundational model for socially-aware navigation with a hierarchical "brain-action" architecture, capable of understanding high-level social norms and generating low-level, socially compliant trajectories. To enable such dual capabilities, we construct the SocNav Dataset, a large-scale collection of 7 million samples, comprising (1) a Cognitive Activation Dataset providing social reasoning signals such as chain-of-thought explanations and social traversability prediction, and (2) an Expert Trajectories Pyramid aggregating diverse navigation demonstrations from internet videos, simulated environments, and real-world robots. A multi-stage training pipeline is proposed to gradually inject and refine navigation intelligence: we first inject general navigation skills and social norms understanding into the model via imitation learning, and then refine such skills through a deliberately designed Socially-Aware Flow Exploration GRPO (SAFE-GRPO), the first flow-based reinforcement learning framework for embodied navigation that explicitly rewards socially compliant behaviors. SocialNav achieves +38% success rate and +46% social compliance rate compared to the state-of-the-art method, demonstrating strong gains in both navigation performance and social compliance. Our project page: https://amap-eai.github.io/SocialNav/
☆ Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Models
Changlin Li, Jiawei Zhang, Zeyi Shi, Zongxin Yang, Zhihui Li, Xiaojun Chang
Large-scale vision generative models, including diffusion and flow models, have demonstrated remarkable performance in visual generation tasks. However, transferring these pre-trained models to downstream tasks often results in significant parameter redundancy. In this paper, we propose EntPruner, an entropy-guided automatic progressive pruning framework for diffusion and flow models. First, we introduce entropy-guided pruning, a block-level importance assessment strategy specifically designed for generative models. Unlike discriminative models, generative models require preserving the diversity and condition-fidelity of the output distribution. As the importance of each module can vary significantly across downstream tasks, EntPruner prioritizes pruning of less important blocks using data-dependent Conditional Entropy Deviation (CED) as a guiding metric. CED quantifies how much the distribution diverges from the learned conditional data distribution after removing a block. Second, we propose a zero-shot adaptive pruning framework to automatically determine when and how much to prune during training. This dynamic strategy avoids the pitfalls of one-shot pruning, mitigating mode collapse, and preserving model performance. Extensive experiments on DiT and SiT models demonstrate the effectiveness of EntPruner, achieving up to 2.22$\times$ inference speedup while maintaining competitive generation quality on ImageNet and three downstream datasets.
comment : Project page: https://github.com/changlin31/EntPruner
☆ Beyond Patch Aggregation: 3-Pass Pyramid Indexing for Vision -Enhanced Document Retrieval
Anup Roy, Rishabh Gyanendra Upadhyay, Animesh Rameshbhai Panara, Robin Mills
Document centric RAG pipelines usually begin with OCR, followed by brittle heuristics for chunking, table parsing, and layout reconstruction. These text first workflows are costly to maintain, sensitive to small layout shifts, and often lose the spatial cues that contain the answer. Vision first retrieval has emerged as a strong alternative. By operating directly on page images, systems like ColPali and ColQwen preserve structure and reduce pipeline complexity while achieving strong benchmark performance. However, these late interaction models tie retrieval to a specific vision backbone and require storing hundreds of patch embeddings per page, creating high memory overhead and complicating large scale deployment. We introduce VisionRAG, a multimodal retrieval system that is OCR free and model agnostic. VisionRAG indexes documents directly as images, preserving layout, tables, and spatial cues, and builds semantic vectors without committing to a specific extraction. Our three pass pyramid indexing framework creates vectors using global page summaries, section headers, visual hotspots, and fact level cues. These summaries act as lightweight retrieval surrogates. At query time, VisionRAG retrieves the most relevant pages using the pyramid index, then forwards the raw page image encoded as base64 to a multimodal LLM for final question answering. During retrieval, reciprocal rank fusion integrates signals across the pyramid to produce robust ranking. VisionRAG stores only 17 to 27 vectors per page, matching the efficiency of patch based methods while staying flexible across multimodal encoders. On financial document benchmarks, it achieves 0.8051 accuracy at 10 on FinanceBench and 0.9629 recall at 100 on TAT DQA. These results show that OCR free, summary guided multimodal retrieval is a practical and scalable alternative to traditional text extraction pipelines.
☆ Learning Cell-Aware Hierarchical Multi-Modal Representations for Robust Molecular Modeling AAAI 2026
Mengran Li, Zelin Zang, Wenbin Xing, Junzhou Chen, Ronghui Zhang, Jiebo Luo, Stan Z. Li
Understanding how chemical perturbations propagate through biological systems is essential for robust molecular property prediction. While most existing methods focus on chemical structures alone, recent advances highlight the crucial role of cellular responses such as morphology and gene expression in shaping drug effects. However, current cell-aware approaches face two key limitations: (1) modality incompleteness in external biological data, and (2) insufficient modeling of hierarchical dependencies across molecular, cellular, and genomic levels. We propose CHMR (Cell-aware Hierarchical Multi-modal Representations), a robust framework that jointly models local-global dependencies between molecules and cellular responses and captures latent biological hierarchies via a novel tree-structured vector quantization module. Evaluated on nine public benchmarks spanning 728 tasks, CHMR outperforms state-of-the-art baselines, yielding average improvements of 3.6% on classification and 17.2% on regression tasks. These results demonstrate the advantage of hierarchy-aware, multimodal learning for reliable and biologically grounded molecular representations, offering a generalizable framework for integrative biomedical modeling. The code is in https://github.com/limengran98/CHMR.
comment : Accepted to AAAI 2026 (Oral)
☆ Deformation-aware Temporal Generation for Ea rl y Prediction of Alzheimers Disease
Xin Honga, Jie Lin, Minghui Wang
Alzheimer's disease (AD), a degenerative brain condition, can benefit from early prediction to slow its progression. As the disease progresses, patients typically undergo brain atrophy. Current prediction methods for Alzheimers disease largely involve analyzing morphological changes in brain images through manual feature extraction. This paper proposes a novel method, the Deformation-Aware Temporal Generative Network (DATGN), to automate the learning of morphological changes in brain images about disease progression for early prediction. Given the common occurrence of missing data in the temporal sequences of MRI images, DATGN initially interpolates incomplete sequences. Subsequently, a bidirectional temporal deformation-aware module guides the network in generating future MRI images that adhere to the disease's progression, facilitating early prediction of Alzheimer's disease. DATGN was tested for the generation of temporal sequences of future MRI images using the ADNI dataset, and the experimental results are competitive in terms of PSNR and MMSE image quality metrics. Furthermore, when DATGN-generated synthetic data was integrated into the SVM vs. CNN vs. 3DCNN-based classification methods, significant improvements were achieved from 6. 21\% to 16\% in AD vs. NC classification accuracy and from 7. 34\% to 21. 25\% in AD vs. MCI vs. NC classification accuracy. The qualitative visualization results indicate that DATGN produces MRI images consistent with the brain atrophy trend in Alzheimer's disease, enabling early disease prediction.
comment : 29 pages,6figures,one column
☆ Dynamic Stratified Contrastive Learning with Upstream Augmentation for MILP Branching
Tongkai Lu, Shuai Ma, Chongyang Tao
Mixed Integer Linear Programming (MILP) is a fundamental class of NP-hard problems that has garnered significant attention from both academia and industry. The Branch-and-Bound (B\&B) method is the dominant approach for solving MILPs and the branching plays an important role in B\&B methods. Neural-based learning frameworks have recently been developed to enhance branching policies and the efficiency of solving MILPs. However, these methods still struggle with semantic variation across depths, the scarcity of upstream nodes, and the costly collection of strong branching samples. To address these issues, we propose \ours, a Dynamic \underline{\textbf{S}}tratified \underline{\textbf{C}}ontrastive Training Framework for \underline{\textbf{MILP}} Branching. It groups branch-and-bound nodes based on their feature distributions and trains a GCNN-based discriminative model to progressively separate nodes across groups, learning finer-grained node representations throughout the tree. To address data scarcity and imbalance at upstream nodes, we introduce an upstream-augmented MILP derivation procedure that generates both theoretically equivalent and perturbed instances. \ours~effectively models subtle semantic differences between nodes, significantly enhancing branching accuracy and solving efficiency, particularly for upstream nodes. Extensive experiments on standard MILP benchmarks demonstrate that our method enhances branching accuracy, reduces solving time, and generalizes effectively to unseen instances.
comment : 18 pages
☆ From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
Hengyu Fu, Baihe Huang, Virginia Adams, Charles Wang, Venkat Srinivasan, Jiantao Jiao
Diffusion Language Models (DLMs) have recently emerged as a strong alternative to autoregressive language models (LMs). DLMs offer comparable accuracy with faster inference speed via parallel decoding. However, standard DLM decoding strategies relying on high-confidence tokens encounter an inherent information-theoretic bottleneck that restricts decoding progress and ultimately slows generation. We demonstrate both theoretically and empirically that prioritizing high-confidence tokens is inherently inefficient. High-probability tokens carry negligible information and strictly relying on them limits the effective progress made in each decoding round. We prove that the number of decoding rounds must grow linearly with the sample's total information (negative log-likelihood) and inversely with the per-round information budget, establishing a bits-to-rounds principle. We also propose Explore-Then-Exploit (ETE), a training-free decoding strategy that maximizes information throughput and decoding efficiency. ETE combines cross-block decoding with targeted exploration of high-uncertainty tokens to reshape the conditional distribution and trigger cascades of confident predictions. Experiments verify our theoretical bounds and demonstrate that ETE consistently reduces the required number of decoding rounds compared to confidence-only baselines without compromising generation quality.
comment : 24 pages, 6 figures
☆ Pygmalion Effect in Vision : Image-to-Clay Translation for Reflective Geometry Reconstruction
Gayoung Lee, Junho Kim, Jin-Hwa Kim, Junmo Kim
Understanding reflection remains a long-standing challenge in 3D reconstruction due to the entanglement of appearance and geometry under view-dependent reflections. In this work, we present the Pygmalion Effect in Vision, a novel framework that metaphorically "sculpts" reflective objects into clay-like forms through image-to-clay translation. Inspired by the myth of Pygmalion, our method learns to suppress specular cues while preserving intrinsic geometric consistency, enabling robust reconstruction from multi-view images containing complex reflections. Specifically, we introduce a dual-branch network in which a BRDF-based reflective branch is complemented by a clay-guided branch that stabilizes geometry and refines surface normals. The two branches are trained jointly using the synthesized clay-like images, which provide a neutral, reflection-free supervision signal that complements the reflective views. Experiments on both synthetic and real datasets demonstrate substantial improvement in normal accuracy and mesh completeness over existing reflection-handling methods. Beyond technical gains, our framework reveals that seeing by unshining, translating radiance into neutrality, can serve as a powerful inductive bias for reflective object geometry learning.
☆ MNM : Multi-level Neuroimaging Meta-analysis with Hyperbolic Brain-Text Representations MICCAI 2025
Seunghun Baek, Jaejin Lee, Jaeyoon Sim, Minjae Jeong, Won Hwa Kim
Various neuroimaging studies suffer from small sample size problem which often limit their reliability. Meta-analysis addresses this challenge by aggregating findings from different studies to identify consistent patterns of brain activity. However, traditional approaches based on keyword retrieval or linear mappings often overlook the rich hierarchical structure in the brain. In this work, we propose a novel framework that leverages hyperbolic geometry to bridge the gap between neuroscience literature and brain activation maps. By embedding text from research articles and corresponding brain images into a shared hyperbolic space via the Lorentz model, our method captures both semantic similarity and hierarchical organization inherent in neuroimaging data. In the hyperbolic space, our method performs multi-level neuroimaging meta-analysis (MNM) by 1) aligning brain and text embeddings for semantic correspondence, 2) guiding hierarchy between text and brain activations, and 3) preserving the hierarchical relationships within brain activation patterns. Experimental results demonstrate that our model outperforms baselines, offering a robust and interpretable paradigm of multi-level neuroimaging meta-analysis via hyperbolic brain-text representation.
comment : MICCAI 2025 (Provisional Accept; top ~9%)
☆ MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts
Ivan Novikov
Large Language Models (LLMs) are predominantly deployed as dense transformers, where every parameter in every feed-forward block is activated for every token. While architecturally simple, this is computationally inefficient, since inference costs scale linearly with parameter count. Recent upcycling methods such as MoEfication, CMoE, ToMoE, and MoORE reveal that much of the useful computation lives in sparse, semi-modular substructures inside dense feed-forward networks, but these approaches typically rely on clustering, activation profiling, singular value decomposition, or custom routing that requires calibration data. This paper introduces MLPMoE (MLP Mixture-of-Experts), a training-free, deterministic transformation that restructures the dense MLP in transformer blocks into a static, high-cardinality mixture of experts. The transformation uses simple tensor slicing and summation, reinterpreting the algebra of tensor parallelism as a topological conversion rather than a distributed training pattern. We further introduce Fractal Fade (differential branch sparsity) and Compensated Pruning (variance-preserving branch reduction) as lightweight mechanisms for structured sparsity. On Qwen2.5-0.5B-Instruct and DeepSeek-R1-Distill-Llama-8B, the zero-shot MLPMoE transform changes a proxy perplexity metric by less than 0.05 percent while keeping the parameter count effectively constant. On the 8B model, differential sparsity removes about 20 percent of MLP parameters while keeping perplexity within about 2 percent of the dense baseline. The method operates entirely post hoc on existing checkpoints and does not require gradients, calibration sets, or router training. Code is available at https://gist.github.com/iwallarm/fc2ef1eddf226ca7814f9e5e2ae9bad1
☆ Enhancing Burmese News Classification with Kolmogorov-Arnold Network Head Fine-tuning
Thura Aung, Eaint Kay Khaing Kyaw, Ye Kyaw Thu, Thazin Myint Oo, Thepchai Supnithi
In low-resource languages like Burmese, classification tasks often fine-tune only the final classification layer, keeping pre-trained encoder weights frozen. While Multi-Layer Perceptrons (MLPs) are commonly used, their fixed non-linearity can limit expressiveness and increase computational cost. This work explores Kolmogorov-Arnold Networks (KANs) as alternative classification heads, evaluating Fourier-based FourierKAN, Spline-based EfficientKAN, and Grid-based FasterKAN-across diverse embeddings including TF-IDF, fastText, and multilingual transformers (mBERT, Distil-mBERT). Experimental results show that KAN-based heads are competitive with or superior to MLPs. EfficientKAN with fastText achieved the highest F1-score (0.928), while FasterKAN offered the best trade-off between speed and accuracy. On transformer embeddings, EfficientKAN matched or slightly outperformed MLPs with mBERT (0.917 F1). These findings highlight KANs as expressive, efficient alternatives to MLPs for low-resource language classification.
comment : 6 pages, 2 figures, 4 tables, Accepted to iSAI-NLP 2025
☆ Data-Driven Assessment of Concrete Slab Integrity via Impact-Echo Signals and Neural Networks
Yeswanth Ravichandran, Duoduo Liao, Charan Teja Kurakula
Subsurface defects such as delamination, voids, and honeycombing critically affect the durability of concrete bridge decks but are difficult to detect reliably using visual inspection or manual sounding. This paper presents a machine learning based Impact Echo (IE) framework that automates both defect localization and multi-class classification of common concrete defects. Raw IE signals from Federal Highway Administration (FHWA) laboratory slabs and in-service bridge decks are transformed via Fast Fourier Transform (FFT) into dominant peak-frequency features and interpolated into spatial maps for defect zone visualization. Unsupervised k-means clustering highlights low-frequency, defect-prone regions, while Ground Truth Masks (GTMs) derived from seeded lab defects are used to validate spatial accuracy and generate high-confidence training labels. From these validated regions, spatially ordered peak-frequency sequences are constructed and fed into a stacked Long Short-Term Memory (LSTM) network that classifies four defect types shallow delamination, deep delamination, voids, and honeycombing with 73% overall accuracy. Field validation on the bridge deck demonstrates that models trained on laboratory data generalize under realistic coupling, noise, and environmental variability. The proposed framework enhances the objectivity, scalability, and repeatability of Non-Destructive Evaluation (NDE), supporting intelligent, data-driven bridge health monitoring at a network scale.
comment : Accepted by IEEE Big Data 2025
☆ Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuning
Zhenchao Tang, Fang Wang, Haohuai He, Jiale Zhou, Tianxu Lv, Jun Zhu, Shouzhi Chen, Minghao Yang, Yu Wang, Jiayang Wu, Yidong Song, Jianhua Yao
Effective post-training is essential to align Large Language Models (LLMs) with specialized biomedical knowledge to accelerate life science research. However, current approaches face significant limitations. First, biomedical reasoning involves intricate mechanisms often represented by sparse textual data. Standard Supervised Fine-Tuning (SFT) tends to overfit to surface-level instruction patterns without effectively internalizing this fragmented scientific knowledge. Second, Reinforcement Learning (RL) is impractical for this domain, as defining meaningful rewards often necessitates prohibitive experimental validation (e.g., wet-lab verification of drug responses), rendering real-time feedback unfeasible. We propose Balanced Fine-Tuning (BFT), an efficient post-training method designed to learn complex reasoning from sparse data without external reward signals. BFT operates through a two-layer weighting mechanism: 1. At the token level, it scales loss via prediction probabilities to stabilize gradients and prevent overfitting; 2. At the sample level, it uses "minimum group confidence" to adaptively enhance the learning of hard samples. Experiments demonstrate that BFT significantly outperforms SFT. In medical tasks, it enables LLMs to acquire knowledge that SFT misses. In biological tasks, BFT-based LLMs surpass GeneAgent (an accurate agent for biology analysis) in biological process reasoning. Moreover, the text embeddings generated by BFT can be directly applied to downstream tasks, such as gene interaction and single-cell perturbation response prediction. These results indicate that BFT facilitates broad applications of LLMs in biomedical research.
☆ Context -Aware Pragmatic Metacognitive Prompt ing for Sarcasm Detection
Michael Iskandardinata, William Christian, Derwin Suhartono
Detecting sarcasm remains a challenging task in the areas of Natural Language Processing (NLP) despite recent advances in neural network approaches. Currently, Pre-trained Language Models (PLMs) and Large Language Models (LLMs) are the preferred approach for sarcasm detection. However, the complexity of sarcastic text, combined with linguistic diversity and cultural variation across communities, has made the task more difficult even for PLMs and LLMs. Beyond that, those models also exhibit unreliable detection of words or tokens that require extra grounding for analysis. Building on a state-of-the-art prompting method in LLMs for sarcasm detection called Pragmatic Metacognitive Prompting (PMP), we introduce a retrieval-aware approach that incorporates retrieved contextual information for each target text. Our pipeline explores two complementary ways to provide context: adding non-parametric knowledge using web-based retrieval when the model lacks necessary background, and eliciting the model's own internal knowledge for a self-knowledge awareness strategy. We evaluated our approach with three datasets, such as Twitter Indonesia Sarcastic, SemEval-2018 Task 3, and MUStARD. Non-parametric retrieval resulted in a significant 9.87% macro-F1 improvement on Twitter Indonesia Sarcastic compared to the original PMP method. Self-knowledge retrieval improves macro-F1 by 3.29% on Semeval and by 4.08% on MUStARD. These findings highlight the importance of context in enhancing LLMs performance in sarcasm detection task, particularly the involvement of culturally specific slang, references, or unknown terms to the LLMs. Future work will focus on optimizing the retrieval of relevant contextual information and examining how retrieval quality affects performance. The experiment code is available at: https://github.com/wllchrst/sarcasm-detection_pmp_knowledge-base.
☆ OVOD- Agent : A Markov-Bandit Framework for Proactive Visual Reasoning and Self-Evolving Detection
Chujie Wang, Jianyu Lu, Zhiyuan Luo, Xi Chen, Chu He
Open-Vocabulary Object Detection (OVOD) aims to enable detectors to generalize across categories by leveraging semantic information. Although existing methods are pretrained on large vision-language datasets, their inference is still limited to fixed category names, creating a gap between multimodal training and unimodal inference. Previous work has shown that improving textual representation can significantly enhance OVOD performance, indicating that the textual space is still underexplored. To this end, we propose OVOD-Agent, which transforms passive category matching into proactive visual reasoning and self-evolving detection. Inspired by the Chain-of-Thought (CoT) paradigm, OVOD-Agent extends the textual optimization process into an interpretable Visual-CoT with explicit actions. OVOD's lightweight nature makes LLM-based management unsuitable; instead, we model visual context transitions as a Weakly Markovian Decision Process (w-MDP) over eight state spaces, which naturally represents the agent's state, memory, and interaction dynamics. A Bandit module generates exploration signals under limited supervision, helping the agent focus on uncertain regions and adapt its detection policy. We further integrate Markov transition matrices with Bandit trajectories for self-supervised Reward Model (RM) optimization, forming a closed loop from Bandit exploration to RM learning. Experiments on COCO and LVIS show that OVOD-Agent provides consistent improvements across OVOD backbones, particularly on rare categories, confirming the effectiveness of the proposed framework.
☆ Breaking the Safety-Capability Tradeoff: Reinforcement Learning with Verifiable Rewards Maintains Safety Guardrails in LLMs AAAI-26
Dongkyu Derek Cho, Huan Song, Arijit Ghosh Chowdhury, Haotian An, Yawei Wang, Rohit Thekkanal, Negin Sokhandan, Sharlina Keshava, Hannah Marlowe
Fine-tuning large language models (LLMs) for downstream tasks typically exhibit a fundamental safety-capability tradeoff, where improving task performance degrades safety alignment even on benign datasets. This degradation persists across standard approaches including supervised finetuning (SFT) and reinforcement learning from human feedback (RLHF). While reinforcement learning with verifiable rewards (RLVR) has emerged as a promising alternative that optimizes models on objectively measurable tasks, its safety implications remain unexplored. We present the first comprehensive theoretical and empirical analysis of safety properties in RLVR. Theoretically, we derive upper bounds on safety drift under KL-constrained optimization and prove conditions under which safety degradation is eliminated. Empirically, we conduct extensive experiments across five adversarial safety benchmarks, demonstrating that RLVR can simultaneously enhance reasoning capabilities while maintaining or improving safety guardrails. Our comprehensive ablation studies examine the effects of optimization algorithms, model scale, and task domains. Our findings challenge the prevailing assumption of an inevitable safety capability trade-off, and establish that a specific training methodology can achieve both objectives simultaneously, providing insights for the safe deployment of reasoning-capable LLMs.
comment : AAAI-26 Workshop on Post-AI Formal Methods
☆ FedAPA: Federated Learning with Adaptive Prototype Aggregation Toward Heterogeneous Wi-Fi CSI-based Crowd Counting
Jingtao Guo, Yuyi Mao, Ivan Wang-Hei Ho
Wi-Fi channel state information (CSI)-based sensing provides a non-invasive, device-free approach for tasks such as human activity recognition and crowd counting, but large-scale deployment is hindered by the need for extensive site-specific training data. Federated learning (FL) offers a way to avoid raw data sharing but is challenged by heterogeneous sensing data and device resources. This paper proposes FedAPA, a collaborative Wi-Fi CSI-based sensing algorithm that uses adaptive prototype aggregation (APA) strategy to assign similarity-based weights to peer prototypes, enabling adaptive client contributions and yielding a personalized global prototype for each client instead of a fixed-weight aggregation. During local training, we adopt a hybrid objective that combines classification learning with representation contrastive learning to align local and global knowledge. We provide a convergence analysis of FedAPA and evaluate it in a real-world distributed Wi-Fi crowd counting scenario with six environments and up to 20 people. The results show that our method outperform multiple baselines in terms of accuracy, F1 score, mean absolute error (MAE), and communication overhead, with FedAPA achieving at least a 9.65% increase in accuracy, a 9% gain in F1 score, a 0.29 reduction in MAE, and a 95.94% reduction in communication overhead.
comment : 17 pages, 11 figures, this article was submitted to IEEE for possible publication
☆ Semantic Anchors in In- Context Learning: Why Small LLMs Cannot Flip Their Labels
Anantha Padmanaban Krishna Kumar
Can in-context learning (ICL) override pre-trained label semantics, or does it merely refine an existing semantic backbone? We address this question by treating LLMs as prompt-induced classifiers and contrasting their behavior under \emph{natural} demonstrations (with correct labels) and \emph{inverted} demonstrations (systematically flipping label meanings). We decompose ICL behavior into three alignment metrics (truth, prior, and prompt alignment) and introduce a semantic override rate, defined as correctness under flipped semantics. Across eight classification tasks and eight open-source LLMs (1--12B parameters), we find consistent evidence for a semantic anchor view. With natural demonstrations, ICL improves accuracy while maintaining strong prior alignment; most correct predictions coincide with zero-shot behavior, even when the prior is weak. With inverted demonstrations, models cannot learn coherent anti-semantic classifiers: prompt alignment increases only by sacrificing accuracy, and semantic override rates remain exactly zero in our few-shot 1--12B setting. Rather than flexibly remapping label meanings, ICL primarily adjusts how inputs project onto stable semantic directions learned during pre-training, clarifying fundamental limits of few-shot prompting and suggesting that overriding label semantics at these scales requires interventions beyond ICL. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/semantic-anchors-icl.
comment : 13 pages total (7 pages main text, 3 pages references, 3 pages appendix), 2 figures, 14 tables. Code available at https://github.com/AnanthaPadmanaban-KrishnaKumar/semantic-anchors-icl
☆ Towards Trustworthy Legal AI through LLM Agent s and Formal Reasoning
Linze Chen, Yufan Cai, Zhe Hou, Jinsong Dong
The rationality of law manifests in two forms: substantive rationality, which concerns the fairness or moral desirability of outcomes, and formal rationality, which requires legal decisions to follow explicitly stated, general, and logically coherent rules. Existing LLM-based systems excel at surface-level text analysis but lack the guarantees required for principled jurisprudence. We introduce L4M, a novel framework that combines adversarial LLM agents with SMT-solver-backed proofs to unite the interpretive flexibility of natural language with the rigor of symbolic verification. The pipeline consists of three phases: (1) Statute Formalization, where domain-specific prompts convert legal provisions into logical formulae; (2) Dual Fact and Statute Extraction, in which prosecutor- and defense-aligned LLMs independently map case narratives to fact tuples and statutes, ensuring role isolation; and (3) Solver-Centric Adjudication, where an autoformalizer compiles both parties' arguments into logic constraints, and unsat cores trigger iterative self-critique until a satisfiable formula is achieved, which is then verbalized by a Judge-LLM into a transparent verdict and optimized sentence. Experimental results on public benchmarks show that our system surpasses advanced LLMs including GPT-o4-mini, DeepSeek-V3, and Claude 4 as well as state-of-the-art Legal AI baselines, while providing rigorous and explainable symbolic justifications.
☆ Structure-Aware Prototype Guided Trusted Multi-View Classification
Haojian Huang, Jiahao Shi, Zhe Liu, Harold Haodong Chen, Han Fang, Hao Sun, Zhongjiang He
Trustworthy multi-view classification (TMVC) addresses the challenge of achieving reliable decision-making in complex scenarios where multi-source information is heterogeneous, inconsistent, or even conflicting. Existing TMVC approaches predominantly rely on globally dense neighbor relationships to model intra-view dependencies, leading to high computational costs and an inability to directly ensure consistency across inter-view relationships. Furthermore, these methods typically aggregate evidence from different views through manually assigned weights, lacking guarantees that the learned multi-view neighbor structures are consistent within the class space, thus undermining the trustworthiness of classification outcomes. To overcome these limitations, we propose a novel TMVC framework that introduces prototypes to represent the neighbor structures of each view. By simplifying the learning of intra-view neighbor relations and enabling dynamic alignment of intra- and inter-view structure, our approach facilitates more efficient and consistent discovery of cross-view consensus. Extensive experiments on multiple public multi-view datasets demonstrate that our method achieves competitive downstream performance and robustness compared to prevalent TMVC methods.
comment : 12 pages, 8 figures, 7 tables, Ongoing Work
☆ Probabilistic Wildfire Spread Prediction Using an Autoregressive Conditional Generative Adversarial Network
Taehoon Kang, Taeyong Kim
Climate change has intensified the frequency and severity of wildfires, making rapid and accurate prediction of fire spread essential for effective mitigation and response. Physics-based simulators such as FARSITE offer high-fidelity predictions but are computationally intensive, limiting their applicability in real-time decision-making, while existing deep learning models often yield overly smooth predictions that fail to capture the complex, nonlinear dynamics of wildfire propagation. This study proposes an autoregressive conditional generative adversarial network (CGAN) for probabilistic wildfire spread prediction. By formulating the prediction task as an autoregressive problem, the model learns sequential state transitions, ensuring long-term prediction stability. Experimental results demonstrate that the proposed CGAN-based model outperforms conventional deep learning models in both overall predictive accuracy and boundary delineation of fire perimeters. These results demonstrate that adversarial learning allows the model to capture the strong nonlinearity and uncertainty of wildfire spread, instead of simply fitting the pixel average. Furthermore, the autoregressive framework facilitates systematic temporal forecasting of wildfire evolution. The proposed CGAN-based autoregressive framework enhances both the accuracy and physical interpretability of wildfire spread prediction, offering a promising foundation for time-sensitive response and evacuation planning.
comment : 22 pages, 15 figures, Submitted to Journal of Environmental Management
☆ ICPO: Intrinsic Confidence-Driven Group Relative Preference Optimization for Efficient Reinforcement Learning
Jinpeng Wang, Chao Li, Ting Ye, Mengyuan Zhang, Wei Liu, Jian Luan
Reinforcement Learning with Verifiable Rewards (RLVR) demonstrates significant potential in enhancing the reasoning capabilities of Large Language Models (LLMs). However, existing RLVR methods are often constrained by issues such as coarse-grained rewards, reward noise, and inefficient exploration, which lead to unstable training and entropy collapse. To address this challenge, we propose the Intrinsic Confidence-Driven Group Relative Preference Optimization method (ICPO). The intuition behind it lies in the fact that the probabilities of an LLM generating different responses can inherently and directly reflect its self-assessment of the reasoning process. Inspired by the idea of preference modeling, ICPO calculates a preference advantage score for each response by comparing the relative generation probabilities of multiple responses under the same input prompt, and integrates this score with verifiable rewards to guide the exploration process. We have discovered that the preference advantage score not only alleviates the issues of coarse-grained rewards and reward noise but also effectively curbs overconfident errors, enhances the relative superiority of undervalued high-quality responses, and prevents the model from overfitting to specific strategies, thereby facilitating more thorough exploration. Comprehensive experiments across four general-domain benchmarks and three mathematical benchmarks demonstrate that ICPO steadily boosts reasoning compared to GRPO.
☆ Knowledge Completes the Vision : A Multimodal Entity -aware Retrieval-Augmented Generation Framework for News Image Captioning AAAI 2026
Xiaoxing You, Qiang Huang, Lingyu Li, Chi Zhang, Xiaopeng Liu, Min Zhang, Jun Yu
News image captioning aims to produce journalistically informative descriptions by combining visual content with contextual cues from associated articles. Despite recent advances, existing methods struggle with three key challenges: (1) incomplete information coverage, (2) weak cross-modal alignment, and (3) suboptimal visual-entity grounding. To address these issues, we introduce MERGE, the first Multimodal Entity-aware Retrieval-augmented GEneration framework for news image captioning. MERGE constructs an entity-centric multimodal knowledge base (EMKB) that integrates textual, visual, and structured knowledge, enabling enriched background retrieval. It improves cross-modal alignment through a multistage hypothesis-caption strategy and enhances visual-entity matching via dynamic retrieval guided by image content. Extensive experiments on GoodNews and NYTimes800k show that MERGE significantly outperforms state-of-the-art baselines, with CIDEr gains of +6.84 and +1.16 in caption quality, and F1-score improvements of +4.14 and +2.64 in named entity recognition. Notably, MERGE also generalizes well to the unseen Visual News dataset, achieving +20.17 in CIDEr and +6.22 in F1-score, demonstrating strong robustness and domain adaptability.
comment : Accepted to AAAI 2026
☆ FANoise: Singular Value-Adaptive Noise Modulation for Robust Multimodal Representation Learning AAAI2026
Jiaoyang Li, Jun Fang, Tianhao Gao, Xiaohui Zhang, Zhiyuan Liu, Chao Liu, Pengzhang Liu, Qixia Jiang
Representation learning is fundamental to modern machine learning, powering applications such as text retrieval and multimodal understanding. However, learning robust and generalizable representations remains challenging. While prior work has demonstrated that active noise injection, a form of data augmentation, can enhance encoding performance, most existing methods rely on heuristic or static noise, overlooking the dynamic nature of feature distributions during training. In this work, we systematically study the role of noise in representation learning from both gradient-based and feature distribution perspectives, using InfoNCE loss as a representative example. Focusing on multimodal representation learning, we propose FANoise, a novel feature-adaptive noise injection strategy. By leveraging the dynamics of contrastive learning, FANoise effectively mitigates the negative impacts of noise while preserving its benefits. Under this theoretically grounded framework, comprehensive experiments demonstrate that FANoise consistently improves overall performance on multimodal tasks across various base VLM models.
comment : 13 pages, 5 figures, accept to AAAI2026
☆ GuardTrace-VL: Detecting Unsafe Multimodel Reasoning via Iterative Safety Super vision
Yuxiao Xiang, Junchi Chen, Zhenchao Jin, Changtao Miao, Haojie Yuan, Qi Chu, Tao Gong, Nenghai Yu
Multimodal large reasoning models (MLRMs) are increasingly deployed for vision-language tasks that produce explicit intermediate rationales. However, reasoning traces can contain unsafe content even when the final answer is non-harmful, creating deployment risks. Existing multimodal safety guards primarily evaluate only the input question and the final answer, neglecting the intermediate reasoning process. This oversight allows undetected harm, such as biased inferences or policy-violating use of visual context, to emerge during reasoning. We introduce GuardTrace-VL, a vision-aware safety auditor that monitors the full Question-Thinking-Answer (QTA) pipeline via joint image-text analysis, enabling detection of unsafe content as it emerges in the reasoning stage. To support training and evaluation, we construct the GuardTrace dataset, which is generated through diverse prompting strategies and refined via a MLRM- and human-based voting and verification pipeline. Furthermore, we propose a three-stage progressive training scheme combined with the data refinement process, enabling the model to learn nuanced and context-dependent safety preferences according to different risk levels. On our proposed test set covering both in-domain and out-of-domain scenarios, GuardTrace-VL model achieves an F1 score of 93.1% on unsafe reasoning detection tasks, representing a 13.5% improvement in F1 score compared to the previous strongest multimodal safety defense methods. The codes will be made publicly available.
☆ Subgoal Graph-Augmented Planning for LLM-Guided Open-Wo rl d Reinforcement Learning
Shanwei Fan
Large language models (LLMs) offer strong high-level planning capabilities for reinforcement learning (RL) by decomposing tasks into subgoals. However, their practical utility is limited by poor planning-execution alignment, which reflects a critical gap between abstract plans and actionable, environment-compatible behaviors. This misalignment arises from two interrelated limitations: (1) LLMs often produce subgoals that are semantically plausible but infeasible or irrelevant in the target environment due to insufficient grounding in environment-specific knowledge, and (2) single-LLM planning conflates generation with self-verification, resulting in overconfident yet unreliable subgoals that frequently fail during execution. To address these challenges, we propose Subgoal Graph-Augmented Actor-Critic-Refiner (SGA-ACR), a framework that integrates an environment-specific subgoal graph and structured entity knowledge with a multi-LLM planning pipeline that explicitly separates generation, critique, and refinement to produce executable and verifiable subgoals. A subgoal tracker further monitors execution progress, provides auxiliary rewards, and adaptively updates the subgoal graph to maintain alignment between plans and actions. Experimental results on 22 diverse tasks in the open-world game "Crafter" demonstrate the effectiveness of our proposed method.
☆ Even with AI, Bijection Discovery is Still Hard: The Opportunities and Challenges of OpenEvolve for Novel Bijection Construction
Davis Brown, Jesse He, Helen Jenne, Henry Kvinge, Max Vargas
Evolutionary program synthesis systems such as AlphaEvolve, OpenEvolve, and ShinkaEvolve offer a new approach to AI-assisted mathematical discovery. These systems utilize teams of large language models (LLMs) to generate candidate solutions to a problem as human readable code. These candidate solutions are then 'evolved' with the goal of improving them beyond what an LLM can produce in a single shot. While existing mathematical applications have mostly focused on problems of establishing bounds (e.g., sphere packing), the program synthesis approach is well suited to any problem where the solution takes the form of an explicit construction. With this in mind, in this paper we explore the use of OpenEvolve for combinatorial bijection discovery. We describe the results of applying OpenEvolve to three bijection construction problems involving Dyck paths, two of which are known and one of which is open. We find that while systems like OpenEvolve show promise as a valuable tool for combinatorialists, the problem of finding novel, research-level bijections remains a challenging task for current frontier systems, reinforcing the need for human mathematicians in the loop. We describe some lessons learned for others in the field interested in exploring the use of these systems.
comment : 16 pages, 3 figures. This is an extended abstract submitted to FPSAC 2026
☆ AI4X Roadmap: Artificial Intelligence for the advancement of scientific pursuit and its future directions
Stephen G. Dale, Nikita Kazeev, Alastair J. A. Price, Victor Posligua, Stephan Roche, O. Anatole von Lilienfeld, Konstantin S. Novoselov, Xavier Bresson, Gianmarco Mengaldo, Xudong Chen, Terence J. O'Kane, Emily R. Lines, Matthew J. Allen, Amandine E. Debus, Clayton Miller, Jiayu Zhou, Hiroko H. Dodge, David Rousseau, Andrey Ustyuzhanin, Ziyun Yan, Mario Lanza, Fabio Sciarrino, Ryo Yoshida, Zhidong Leong, Teck Leong Tan, Qianxiao Li, Adil Kabylda, Igor Poltavsky, Alexandre Tkatchenko, Sherif Abdulkader Tawfik, Prathami Divakar Kamath, Theo Jaffrelot Inizan, Kristin A. Persson, Bryant Y. Li, Vir Karan, Chenru Duan, Haojun Jia, Qiyuan Zhao, Hiroyuki Hayashi, Atsuto Seko, Isao Tanaka, Omar M. Yaghi, Tim Gould, Bun Chan, Stefan Vuckovic, Tianbo Li, Min Lin, Zehcen Tang, Yang Li, Yong Xu, Amrita Joshi, Xiaonan Wang, Leonard W. T. Ng, Sergei V. Kalinin, Mahshid Ahmadi, Jiyizhe Zhang, Shuyuan Zhang, Alexei Lapkin, Ming Xiao, Zhe Wu, Kedar Hippalgaonkar, Limsoon Wong, Lorenzo Bastonero, Nicola Marzari, Dorye Luis Esteras Cordoba, Andrei Tomut, Alba Quinones Andrade, Jose-Hugo Garcia
Artificial intelligence and machine learning are reshaping how we approach scientific discovery, not by replacing established methods but by extending what researchers can probe, predict, and design. In this roadmap we provide a forward-looking view of AI-enabled science across biology, chemistry, climate science, mathematics, materials science, physics, self-driving laboratories and unconventional computing. Several shared themes emerge: the need for diverse and trustworthy data, transferable electronic-structure and interatomic models, AI systems integrated into end-to-end scientific workflows that connect simulations to experiments and generative systems grounded in synthesisability rather than purely idealised phases. Across domains, we highlight how large foundation models, active learning and self-driving laboratories can close loops between prediction and validation while maintaining reproducibility and physical interpretability. Taken together, these perspectives outline where AI-enabled science stands today, identify bottlenecks in data, methods and infrastructure, and chart concrete directions for building AI systems that are not only more powerful but also more transparent and capable of accelerating discovery in complex real-world environments.
☆ Towards Audio Token Compression in Large Audio Language Models
Saurabhchand Bhati, Samuel Thomas, Hilde Kuehne, Rogerio Feris, James Glass
Large Audio Language Models (LALMs) demonstrate impressive performance across diverse tasks, ranging from speech recognition to general audio understanding. However, their scalability is limited by the quadratic complexity of attention and the high token rates of audio signals. These challenges make it difficult to extend LALMs to long-form audio and to deploy them on resource-constrained platforms such as edge devices. In this paper, we explore techniques such as unsupervised segmentation, uniform average pooling, etc., to reduce the number of audio tokens generated by the LALM's audio encoder but before they are consumed by the LLM decoder. To mitigate potential performance degradation introduced by the compressed representations, we employ low-rank adapters to finetune the model. We evaluate our proposed models on two tasks, automatic speech recognition and speech-to-speech translation tasks, that are dependent on effectively uncovering the underlying lexical content of the input signal and study the effect of downsampling on these tasks. Experimental results show that compressed LALMs can achieve performance closer to frame-level LALMs while reducing the input audio token count upto three times before the LLM backbone.
☆ BUSTR: Breast Ultrasound Text Reporting with a Descriptor-Aware Vision -Language Model
Rawa Mohammed, Mina Attin, Bryar Shareef
Automated radiology report generation (RRG) for breast ultrasound (BUS) is limited by the lack of paired image-report datasets and the risk of hallucinations from large language models. We propose BUSTR, a multitask vision-language framework that generates BUS reports without requiring paired image-report supervision. BUSTR constructs reports from structured descriptors (e.g., BI-RADS, pathology, histology) and radiomics features, learns descriptor-aware visual representations with a multi-head Swin encoder trained using a multitask loss over dataset-specific descriptor sets, and aligns visual and textual tokens via a dual-level objective that combines token-level cross-entropy with a cosine-similarity alignment loss between input and output representations. We evaluate BUSTR on two public BUS datasets, BrEaST and BUS-BRA, which differ in size and available descriptors. Across both datasets, BUSTR consistently improves standard natural language generation metrics and clinical efficacy metrics, particularly for key targets such as BI-RADS category and pathology. Our results show that this descriptor-aware vision model, trained with a combined token-level and alignment loss, improves both automatic report metrics and clinical efficacy without requiring paired image-report data. The source code can be found at https://github.com/AAR-UNLV/BUSTR
comment : 13 pages, 2 figures, 6 tables
☆ SpaceX: Exploring metrics with the SPACE model for developer productivity
Sanchit Kaul, Kevin Nhu, Jason Eissayou, Ivan Eser, Victor Borup
This empirical investigation elucidates the limitations of deterministic, unidimensional productivity heuristics by operationalizing the SPACE framework through extensive repository mining. Utilizing a dataset derived from open-source repositories, the study employs rigorous statistical methodologies including Generalized Linear Mixed Models (GLMM) and RoBERTa-based sentiment classification to synthesize a holistic, multi-faceted productivity metric. Analytical results reveal a statistically significant positive correlation between negative affective states and commit frequency, implying a cycle of iterative remediation driven by frustration. Furthermore, the investigation has demonstrated that analyzing the topology of contributor interactions yields superior fidelity in mapping collaborative dynamics compared to traditional volume-based metrics. Ultimately, this research posits a Composite Productivity Score (CPS) to address the heterogeneity of developer efficacy.
comment : Code available at https://github.com/knhu/ECS260Project
☆ Resilient Charging Infrastructure via Decentralized Coordination of Electric Vehicles at Scale
Chuhao Qin, Alexandru Sorici, Andrei Olaru, Evangelos Pournaras, Adina Magda Florea
The rapid adoption of electric vehicles (EVs) introduces major challenges for decentralized charging control. Existing decentralized approaches efficiently coordinate a large number of EVs to select charging stations while reducing energy costs, preventing power peak and preserving driver privacy. However, they often struggle under severe contingencies, such as station outages or unexpected surges in charging requests. These situations create competition for limited charging slots, resulting in long queues and reduced driver comfort. To address these limitations, we propose a novel collective learning-based coordination framework that allows EVs to balance individual comfort on their selections against system-wide efficiency, i.e., the overall queues across all stations. In the framework, EVs are recommended for adaptive charging behaviors that shift priority between comfort and efficiency, achieving Pareto-optimal trade-offs under varying station capacities and dynamic spatio-temporal EV distribution. Experiments using real-world data from EVs and charging stations show that the proposed approach outperforms baseline methods, significantly reducing travel and queuing time. The results reveal that, under uncertain charging conditions, EV drivers that behave selfishly or altruistically at the right moments achieve shorter waiting time than those maintaining moderate behavior throughout. Our findings under high fractions of station outages and adversarial EVs further demonstrate improved resilience and trustworthiness of decentralized EV charging infrastructure.
comment : 14 pages, 12 figures. This work has been submitted to the IEEE for possible publication
☆ Improving Procedural Skill Explanations via Constrained Generation: A Symbolic-LLM Hybrid Architecture
Rahul Dass, Thomas Bowlin, Zebing Li, Xiao Jin, Ashok Goel
In procedural skill learning, instructional explanations must convey not just steps, but the causal, goal-directed, and compositional logic behind them. Large language models (LLMs) often produce fluent yet shallow responses that miss this structure. We present Ivy, an AI coaching system that delivers structured, multi-step explanations by combining symbolic Task-Method-Knowledge (TMK) models with a generative interpretation layer-an LLM that constructs explanations while being constrained by TMK structure. TMK encodes causal transitions, goal hierarchies, and problem decompositions, and guides the LLM within explicit structural bounds. We evaluate Ivy against responses against GPT and retrieval-augmented GPT baselines using expert and independent annotations across three inferential dimensions. Results show that symbolic constraints consistently improve the structural quality of explanations for "how" and "why" questions. This study demonstrates a scalable AI for education approach that strengthens the pedagogical value of AI-generated explanations in intelligent coaching systems.
☆ ENACT: Evaluating Embodied Cognition with Wo rl d Modeling of Egocentric Interaction
Qineng Wang, Wenlong Huang, Yu Zhou, Hang Yin, Tianwei Bao, Jianwen Lyu, Weiyu Liu, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, Manling Li
Embodied cognition argues that intelligence arises from sensorimotor interaction rather than passive observation. It raises an intriguing question: do modern vision-language models (VLMs), trained largely in a disembodied manner, exhibit signs of embodied cognition? We introduce ENACT, a benchmark that casts evaluation of embodied cognition as world modeling from egocentric interaction in a visual question answering (VQA) format. Framed as a partially observable Markov decision process (POMDP) whose actions are scene graph changes, ENACT comprises two complementary sequence reordering tasks: forward world modeling (reorder shuffled observations given actions) and inverse world modeling (reorder shuffled actions given observations). While conceptually simple, solving these tasks implicitly demands capabilities central to embodied cognition-affordance recognition, action-effect reasoning, embodied awareness, and interactive, long-horizon memory from partially observable egocentric input, while avoiding low-level image synthesis that could confound the evaluation. We provide a scalable pipeline that synthesizes QA pairs from robotics simulation (BEHAVIOR) and evaluates models on 8,972 QA pairs spanning long-horizon home-scale activities. Experiments reveal a performance gap between frontier VLMs and humans that widens with interaction horizon. Models consistently perform better on the inverse task than the forward one and exhibit anthropocentric biases, including a preference for right-handed actions and degradation when camera intrinsics or viewpoints deviate from human vision. Website at https://enact-embodied-cognition.github.io/.
comment : Preprint version
☆ DialBench: Towards Accurate Reading Recognition of Pointer Meter using Large Foundation Models
Futian Wang, Chaoliu Weng, Xiao Wang, Zhen Chen, Zhicheng Zhao, Jin Tang
The precise reading recognition of pointer meters plays a key role in smart power systems, but existing approaches remain fragile due to challenges like reflections, occlusions, dynamic viewing angles, and overly between thin pointers and scale markings. Up to now, this area still lacks large-scale datasets to support the development of robust algorithms. To address these challenges, this paper first presents a new large-scale benchmark dataset for dial reading, termed RPM-10K, which contains 10730 meter images that fully reflect the aforementioned key challenges. Built upon the dataset, we propose a novel vision-language model for pointer meter reading recognition, termed MRLM, based on physical relation injection. Instead of exhaustively learning image-level correlations, MRLM explicitly encodes the geometric and causal relationships between the pointer and the scale, aligning perception with physical reasoning in the spirit of world-model perspectives. Through cross-attentional fusion and adaptive expert selection, the model learns to interpret dial configurations and generate precise numeric readings. Extensive experiments fully validated the effectiveness of our proposed framework on the newly proposed benchmark dataset. Both the dataset and source code will be released on https://github.com/Event-AHU/DialBench
☆ The Risk-Adjusted Intelligence Dividend: A Quantitative Framework for Measuring AI Return on Investment Integrating ISO 42001 and Regulatory Exposure
Hernan Huwyler
Organizations investing in artificial intelligence face a fundamental challenge: traditional return on investment calculations fail to capture the dual nature of AI implementations, which simultaneously reduce certain operational risks while introducing novel exposures related to algorithmic malfunction, adversarial attacks, and regulatory liability. This research presents a comprehensive financial framework for quantifying AI project returns that explicitly integrates changes in organizational risk profiles. The methodology addresses a critical gap in current practice where investment decisions rely on optimistic benefit projections without accounting for the probabilistic costs of AI-specific threats including model drift, bias-related litigation, and compliance failures under emerging regulations such as the European Union Artificial Intelligence Act and ISO/IEC 42001. Drawing on established risk quantification methods, including annual loss expectancy calculations and Monte Carlo simulation techniques, this framework enables practitioners to compute net benefits that incorporate both productivity gains and the delta between pre-implementation and post-implementation risk exposures. The analysis demonstrates that accurate AI investment evaluation requires explicit modeling of control effectiveness, reserve requirements for algorithmic failures, and the ongoing operational costs of maintaining model performance. Practical implications include specific guidance for establishing governance structures, conducting phased validations, and integrating risk-adjusted metrics into capital allocation decisions, ultimately enabling evidence-based AI portfolio management that satisfies both fiduciary responsibilities and regulatory mandates.
comment : 21 pages, 2 equations, 8 references. Framework for risk-adjusted AI ROI calculation integrating ISO 42001, NIST AI RMF, and EU AI Act compliance requirements
☆ DeepGI: Explainable Deep Learning for Gastrointestinal Image Classification
Walid Houmaidi, Mohamed Hadadi, Youssef Sabiri, Yousra Chtouki
This paper presents a comprehensive comparative model analysis on a novel gastrointestinal medical imaging dataset, comprised of 4,000 endoscopic images spanning four critical disease classes: Diverticulosis, Neoplasm, Peritonitis, and Ureters. Leveraging state-of-the-art deep learning techniques, the study confronts common endoscopic challenges such as variable lighting, fluctuating camera angles, and frequent imaging artifacts. The best performing models, VGG16 and MobileNetV2, each achieved a test accuracy of 96.5%, while Xception reached 94.24%, establishing robust benchmarks and baselines for automated disease classification. In addition to strong classification performance, the approach includes explainable AI via Grad-CAM visualization, enabling identification of image regions most influential to model predictions and enhancing clinical interpretability. Experimental results demonstrate the potential for robust, accurate, and interpretable medical image analysis even in complex real-world conditions. This work contributes original benchmarks, comparative insights, and visual explanations, advancing the landscape of gastrointestinal computer-aided diagnosis and underscoring the importance of diverse, clinically relevant datasets and model explainability in medical AI research.
comment : 7 pages, 4 figures, 2 tables. Accepted at DASET 2026
☆ ABLE: Using Adversarial Pairs to Construct Local Models for Explaining Model Predictions KDD 2026
Krishna Khadka, Sunny Shree, Pujan Budhathoki, Yu Lei, Raghu Kacker, D. Richard Kuhn
Machine learning models are increasingly used in critical applications but are mostly "black boxes" due to their lack of transparency. Local explanation approaches, such as LIME, address this issue by approximating the behavior of complex models near a test instance using simple, interpretable models. However, these approaches often suffer from instability and poor local fidelity. In this paper, we propose a novel approach called Adversarially Bracketed Local Explanation (ABLE) to address these limitations. Our approach first generates a set of neighborhood points near the test instance, x_test, by adding bounded Gaussian noise. For each neighborhood point D, we apply an adversarial attack to generate an adversarial point A with minimal perturbation that results in a different label than D. A second adversarial attack is then performed on A to generate a point A' that has the same label as D (and thus different than A). The points A and A' form an adversarial pair that brackets the local decision boundary for x_test. We then train a linear model on these adversarial pairs to approximate the local decision boundary. Experimental results on six UCI benchmark datasets across three deep neural network architectures demonstrate that our approach achieves higher stability and fidelity than the state-of-the-art.
comment : 10 pages, 2 figures. Accepted to KDD 2026 (Research Track)
☆ WalkCLIP: Multimodal Learning for Urban Walkability Prediction
Shilong Xiang, JangHyeon Lee, Min Namgung, Yao-Yi Chiang
Urban walkability is a cornerstone of public health, sustainability, and quality of life. Traditional walkability assessments rely on surveys and field audits, which are costly and difficult to scale. Recent studies have used satellite imagery, street view imagery, or population indicators to estimate walkability, but these single-source approaches capture only one dimension of the walking environment. Satellite data describe the built environment from above, but overlook the pedestrian perspective. Street view imagery captures conditions at the ground level, but lacks broader spatial context. Population dynamics reveal patterns of human activity but not the visual form of the environment. We introduce WalkCLIP, a multimodal framework that integrates these complementary viewpoints to predict urban walkability. WalkCLIP learns walkability-aware vision-language representations from GPT-4o generated image captions, refines these representations with a spatial aggregation module that incorporates neighborhood context, and fuses the resulting features with representations from a population dynamics foundation model. Evaluated at 4,660 locations throughout Minneapolis-Saint Paul, WalkCLIP outperforms unimodal and multimodal baselines in both predictive accuracy and spatial alignment. These results show that the integration of visual and behavioral signals yields reliable predictions of the walking environment.
☆ Heterogeneous Multi- Agent Reinforcement Learning with Attention for Cooperative and Scalable Feature Transformation KDD 2026
Tao Zhe, Huazhen Fang, Kunpeng Liu, Qian Lou, Tamzidul Hoque, Dongjie Wang
Feature transformation enhances downstream task performance by generating informative features through mathematical feature crossing. Despite the advancements in deep learning, feature transformation remains essential for structured data, where deep models often struggle to capture complex feature interactions. Prior literature on automated feature transformation has achieved success but often relies on heuristics or exhaustive searches, leading to inefficient and time-consuming processes. Recent works employ reinforcement learning (RL) to enhance traditional approaches through a more effective trial-and-error way. However, two limitations remain: 1) Dynamic feature expansion during the transformation process, which causes instability and increases the learning complexity for RL agents; 2) Insufficient cooperation and communication between agents, which results in suboptimal feature crossing operations and degraded model performance. To address them, we propose a novel heterogeneous multi-agent RL framework to enable cooperative and scalable feature transformation. The framework comprises three heterogeneous agents, grouped into two types, each designed to select essential features and operations for feature crossing. To enhance communication among these agents, we implement a shared critic mechanism that facilitates information exchange during feature transformation. To handle the dynamically expanding feature space, we tailor multi-head attention-based feature agents to select suitable features for feature crossing. Additionally, we introduce a state encoding technique during the optimization process to stabilize and enhance the learning dynamics of the RL agents, resulting in more robust and reliable transformation policies. Finally, we conduct extensive experiments to validate the effectiveness, efficiency, robustness, and interpretability of our model.
comment : Accepted at KDD 2026 Research Track
☆ Does the Model Say What the Data Says? A Simple Heuristic for Model Data Alignment
Henry Salgado, Meagan Kendall, Martine Ceberio
In this work, we propose a simple and computationally efficient framework to evaluate whether machine learning models align with the structure of the data they learn from; that is, whether \textit{the model says what the data says}. Unlike existing interpretability methods that focus exclusively on explaining model behavior, our approach establishes a baseline derived directly from the data itself. Drawing inspiration from Rubin's Potential Outcomes Framework, we quantify how strongly each feature separates the two outcome groups in a binary classification task, moving beyond traditional descriptive statistics to estimate each feature's effect on the outcome. By comparing these data-derived feature rankings against model-based explanations, we provide practitioners with an interpretable and model-agnostic method to assess model--data alignment.
☆ Prompt ed Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs
Yifan Zhou, Sachin Grover, Mohamed El Mistiri, Kamalesh Kalirathnam, Pratyush Kerhalkar, Swaroop Mishra, Neelesh Kumar, Sanket Gaurav, Oya Aran, Heni Ben Amor
Reinforcement Learning (RL) traditionally relies on scalar reward signals, limiting its ability to leverage the rich semantic knowledge often available in real-world tasks. In contrast, humans learn efficiently by combining numerical feedback with language, prior knowledge, and common sense. We introduce Prompted Policy Search (ProPS), a novel RL method that unifies numerical and linguistic reasoning within a single framework. Unlike prior work that augment existing RL components with language, ProPS places a large language model (LLM) at the center of the policy optimization loop-directly proposing policy updates based on both reward feedback and natural language input. We show that LLMs can perform numerical optimization in-context, and that incorporating semantic signals, such as goals, domain knowledge, and strategy hints can lead to more informed exploration and sample-efficient learning. ProPS is evaluated across fifteen Gymnasium tasks, spanning classic control, Atari games, and MuJoCo environments, and compared to seven widely-adopted RL algorithms (e.g., PPO, SAC, TRPO). It outperforms all baselines on eight out of fifteen tasks and demonstrates substantial gains when provided with domain knowledge. These results highlight the potential of unifying semantics and numerics for transparent, generalizable, and human-aligned RL.
comment : In The Thirty-ninth Annual Conference on Neural Information Processing Systems
☆ Exploring Dynamic Properties of Backdoor Training Through Information Bottleneck
Xinyu Liu, Xu Zhang, Can Chen, Ren Wang
Understanding how backdoor data influences neural network training dynamics remains a complex and underexplored challenge. In this paper, we present a rigorous analysis of the impact of backdoor data on the learning process, with a particular focus on the distinct behaviors between the target class and other clean classes. Leveraging the Information Bottleneck (IB) principle connected with clustering of internal representation, We find that backdoor attacks create unique mutual information (MI) signatures, which evolve across training phases and differ based on the attack mechanism. Our analysis uncovers a surprising trade-off: visually conspicuous attacks like BadNets can achieve high stealthiness from an information-theoretic perspective, integrating more seamlessly into the model than many visually imperceptible attacks. Building on these insights, we propose a novel, dynamics-based stealthiness metric that quantifies an attack's integration at the model level. We validate our findings and the proposed metric across multiple datasets and diverse attack types, offering a new dimension for understanding and evaluating backdoor threats. Our code is available in: https://github.com/XinyuLiu71/Information_Bottleneck_Backdoor.git.
☆ Toward Automated and Trustworthy Scientific Analysis and Visualization with LLM-Generated Code
Apu Kumar Chakroborti, Yi Ding, Lipeng Wan
As modern science becomes increasingly data-intensive, the ability to analyze and visualize large-scale, complex datasets is critical to accelerating discovery. However, many domain scientists lack the programming expertise required to develop custom data analysis workflows, creating barriers to timely and effective insight. Large language models (LLMs) offer a promising solution by generating executable code from natural language descriptions. In this paper, we investigate the trustworthiness of open-source LLMs in autonomously producing Python scripts for scientific data analysis and visualization. We construct a benchmark suite of domain-inspired prompts that reflect real-world research tasks and systematically evaluate the executability and correctness of the generated code. Our findings show that, without human intervention, the reliability of LLM-generated code is limited, with frequent failures caused by ambiguous prompts and the models' insufficient understanding of domain-specific contexts. To address these challenges, we design and assess three complementary strategies: data-aware prompt disambiguation, retrieval-augmented prompt enhancement, and iterative error repair. While these methods significantly improve execution success rates and output quality, further refinement is needed. This work highlights both the promise and current limitations of LLM-driven automation in scientific workflows and introduces actionable techniques and a reusable benchmark for building more inclusive, accessible, and trustworthy AI-assisted research tools.
☆ Adaptive Parameter Optimization for Robust Remote Photoplethysmography
Cecilia G. Morales, Fanurs Chi En Teh, Kai Li, Pushpak Agrawal, Artur Dubrawski
Remote photoplethysmography (rPPG) enables contactless vital sign monitoring using standard RGB cameras. However, existing methods rely on fixed parameters optimized for particular lighting conditions and camera setups, limiting adaptability to diverse deployment environments. This paper introduces the Projection-based Robust Signal Mixing (PRISM) algorithm, a training-free method that jointly optimizes photometric detrending and color mixing through online parameter adaptation based on signal quality assessment. PRISM achieves state-of-the-art performance among unsupervised methods, with MAE of 0.77 bpm on PURE and 0.66 bpm on UBFC-rPPG, and accuracy of 97.3\% and 97.5\% respectively at a 5 bpm threshold. Statistical analysis confirms PRISM performs equivalently to leading supervised methods ($p > 0.2$), while maintaining real-time CPU performance without training. This validates that adaptive time series optimization significantly improves rPPG across diverse conditions.
comment : Accepted in Times Series for Health NeurIPs Workshop 2025
☆ Path Reasoning : A multimodal reasoning agent for query-based ROI navigation on whole-slide images
Kunpeng Zhang, Hanwen Xu, Sheng Wang
Deciphering tumor microenvironment from Whole Slide Images (WSIs) is intriguing as it is key to cancer diagnosis, prognosis and treatment response. While these gigapixel images on one hand offer a comprehensive portrait of cancer, on the other hand, the extremely large size, as much as more than 10 billion pixels, make it challenging and time-consuming to navigate to corresponding regions to support diverse clinical inspection. Inspired by pathologists who conducted navigation on WSIs with a combination of sampling, reasoning and self-reflection, we proposed "PathReasoning", a multi-modal reasoning agent that iteratively navigates across WSIs through multiple rounds of reasoning and refinements. Specifically, starting with randomly sampled candidate regions, PathReasoning reviews current selections with self-reflection, reasoning over the correspondence between visual observations and clinical questions, and concludes by proposing new regions to explore. Across rounds, PathReasoning builds a reasoning chain that gradually directs attention to diagnostically relevant areas. PathReasoning turns each whole slide into a sequence of question-guided views, allowing the model to efficiently find informative ROIs within a fixed number of steps, without the need for dense pixel-level annotations. PathReasoning can substantially outperform strong ROI-selection approaches by 6.7% and 3.1% of AUROC on subtyping and longitudinal analysis tasks. The high-quality ROIs further support accurate report generation on breast cancer, significantly outperforming the standard GPT-4o by 10% in accuracy. PathReasoning prioritizes question-specific regions and constructs interpretable reasoning chains, supporting efficient slide review, consistent diagnostic interpretations, comprehensive reporting, and evidence traceability in digital pathology.
☆ Standardized Threat Taxonomy for AI Security, Governance, and Regulatory Compliance
Hernan Huwyler
The accelerating deployment of artificial intelligence systems across regulated sectors has exposed critical fragmentation in risk assessment methodologies. A significant "language barrier" currently separates technical security teams, who focus on algorithmic vulnerabilities (e.g., MITRE ATLAS), from legal and compliance professionals, who address regulatory mandates (e.g., EU AI Act, NIST AI RMF). This disciplinary disconnect prevents the accurate translation of technical vulnerabilities into financial liability, leaving practitioners unable to answer fundamental economic questions regarding contingency reserves, control return-on-investment, and insurance exposure. To bridge this gap, this research presents the AI System Threat Vector Taxonomy, a structured ontology designed explicitly for Quantitative Risk Assessment (QRA). The framework categorizes AI-specific risks into nine critical domains: Misuse, Poisoning, Privacy, Adversarial, Biases, Unreliable Outputs, Drift, Supply Chain, and IP Threat, integrating 53 operationally defined sub-threats. Uniquely, each domain maps technical vectors directly to business loss categories (Confidentiality, Integrity, Availability, Legal, Reputation), enabling the translation of abstract threats into measurable financial impact. The taxonomy is empirically validated through an analysis of 133 documented AI incidents from 2025 (achieving 100% classification coverage) and reconciled against the main AI risk frameworks. Furthermore, it is explicitly aligned with ISO/IEC 42001 controls and NIST AI RMF functions to facilitate auditability.
comment : 10 pages, LaTeX. Preprint available on Zenodo
☆ Bridging Planning and Execution: Multi- Agent Path Finding Under Real-Wo rl d Deadlines
Jingtian Yan, Shuai Zhou, Stephen F. Smith, Jiaoyang Li
The Multi-Agent Path Finding (MAPF) problem aims to find collision-free paths for multiple agents while optimizing objectives such as the sum of costs or makespan. MAPF has wide applications in domains like automated warehouses, manufacturing systems, and airport logistics. However, most MAPF formulations assume a simplified robot model for planning, which overlooks execution-time factors such as kinodynamic constraints, communication latency, and controller variability. This gap between planning and execution is problematic for time-sensitive applications. To bridge this gap, we propose REMAP, an execution-informed MAPF planning framework that can be combined with leading search-based MAPF planners with minor changes. Our framework integrates the proposed ExecTimeNet to accurately estimate execution time based on planned paths. We demonstrate our method for solving MAPF with Real-world Deadlines (MAPF-RD) problem, where agents must reach their goals before a predefined wall-clock time. We integrate our framework with two popular MAPF methods, MAPF-LNS and CBS. Experiments show that REMAP achieves up to 20% improvement in solution quality over baseline methods (e.g., constant execution speed estimators) on benchmark maps with up to 300 agents.
☆ LLM-Empowered Event-Chain Driven Code Generation for ADAS in SDV systems
Nenad Petrovic, Norbert Kroth, Axel Torschmied, Yinglei Song, Fengjunjie Pan, Vahid Zolfaghari, Nils Purschke, Sven Kirchner, Chengdong Wu, Andre Schamschurko, Yi Zhang, Alois Knoll
This paper presents an event-chain-driven, LLM-empowered workflow for generating validated, automotive code from natural-language requirements. A Retrieval-Augmented Generation (RAG) layer retrieves relevant signals from large and evolving Vehicle Signal Specification (VSS) catalogs as code generation prompt context, reducing hallucinations and ensuring architectural correctness. Retrieved signals are mapped and validated before being transformed into event chains that encode causal and timing constraints. These event chains guide and constrain LLM-based code synthesis, ensuring behavioral consistency and real-time feasibility. Based on our initial findings from the emergency braking case study, with the proposed approach, we managed to achieve valid signal usage and consistent code generation without LLM retraining.
☆ Advancing Marine Bioacoustics with Deep Generative Models: A Hybrid Augmentation Strategy for Southern Resident Killer Whale Detection
Bruno Padovese, Fabio Frazao, Michael Dowd, Ruth Joy
Automated detection and classification of marine mammals vocalizations is critical for conservation and management efforts but is hindered by limited annotated datasets and the acoustic complexity of real-world marine environments. Data augmentation has proven to be an effective strategy to address this limitation by increasing dataset diversity and improving model generalization without requiring additional field data. However, most augmentation techniques used to date rely on effective but relatively simple transformations, leaving open the question of whether deep generative models can provide additional benefits. In this study, we evaluate the potential of deep generative for data augmentation in marine mammal call detection including: Variational Autoencoders, Generative Adversarial Networks, and Denoising Diffusion Probabilistic Models. Using Southern Resident Killer Whale (Orcinus orca) vocalizations from two long-term hydrophone deployments in the Salish Sea, we compare these approaches against traditional augmentation methods such as time-shifting and vocalization masking. While all generative approaches improved classification performance relative to the baseline, diffusion-based augmentation yielded the highest recall (0.87) and overall F1-score (0.75). A hybrid strategy combining generative-based synthesis with traditional methods achieved the best overall performance with an F1-score of 0.81. We hope this study encourages further exploration of deep generative models as complementary augmentation strategies to advance acoustic monitoring of threatened marine mammal populations.
comment : 16 pages, 6 Figures, 2 Tables, submitted to Marine Mammal Science as part of a special issue on Machine Learning and Artificial Intelligence in Marine Mammal Research
☆ Towards a Foundation Model for Partial Differential Equations Across Physics Domains AAAI 2026
Eduardo Soares, Emilio Vital Brazil, Victor Shirasuna, Breno W. S. R. de Carvalho, Cristiano Malossi
We present PDE-FM, a modular foundation model for physics-informed machine learning that unifies spatial, spectral, and temporal reasoning across heterogeneous partial differential equation (PDE) systems. PDE-FM combines spatial-spectral tokenization, physics-aware conditioning, and a Mamba-based state-space backbone with an operator-theoretic decoder, enabling scalable and data-efficient modeling of complex physical dynamics. In contrast to task-specific neural operators, PDE-FM is pretrained once on diverse PDE datasets and can be transferred to new physical regimes without architectural or data-specific modifications. Evaluated on twelve 2D and 3D datasets from The Well benchmark - spanning hydrodynamic, radiative, elastic, and astrophysical phenomena - PDE-FM achieves state-of-the-art accuracy in six domains, reducing mean VRMSE by 46% relative to prior operator-learning baselines. The model demonstrates robust cross-physics generalization, excelling in turbulent and radiative systems while maintaining strong performance in linear and steady-state regimes. These results suggest that large-scale pretraining across diverse physical processes can yield transferable representations of dynamics, marking a step toward unified, foundation-level surrogates for multi-physics simulation and scientific discovery.
comment : Accepted to the AAAI 2026 AI2ASE Workshop
☆ Improving Score Reliability of Multiple Choice Benchmark s with Consistency Evaluation and Altered Answer Choices
Paulo Cavalin, Cassia Sanctos, Marcelo Grave, Claudio Pinhanez, Yago Primerano
In this work we present the Consistency-Rebalanced Accuracy (CoRA) metric, improving the reliability of Large Language Model (LLM) scores computed on multiple choice (MC) benchmarks. Our metric explores the response consistency of the LLMs, taking advantage of synthetically-generated questions with altered answer choices. With two intermediate scores, i.e. Bare-Minimum-Consistency Accuracy (BMCA) and Consistency Index (CI), CoRA is computed by adjusting the multiple-choice question answering (MCQA) scores to better reflect the level of consistency of the LLM. We present evaluations in different benchmarks using diverse LLMs, and not only demonstrate that LLMs can present low response consistency even when they present high MCQA scores, but also that CoRA can successfully scale down the scores of inconsistent models.
☆ LILAD: Learning In- context Lyapunov-stable Adaptive Dynamics Models AAAI-26
Amit Jena, Na Li, Le Xie
System identification in control theory aims to approximate dynamical systems from trajectory data. While neural networks have demonstrated strong predictive accuracy, they often fail to preserve critical physical properties such as stability and typically assume stationary dynamics, limiting their applicability under distribution shifts. Existing approaches generally address either stability or adaptability in isolation, lacking a unified framework that ensures both. We propose LILAD (Learning In-Context Lyapunov-stable Adaptive Dynamics), a novel framework for system identification that jointly guarantees adaptability and stability. LILAD simultaneously learns a dynamics model and a Lyapunov function through in-context learning (ICL), explicitly accounting for parametric uncertainty. Trained across a diverse set of tasks, LILAD produces a stability-aware, adaptive dynamics model alongside an adaptive Lyapunov certificate. At test time, both components adapt to a new system instance using a short trajectory prompt, which enables fast generalization. To rigorously ensure stability, LILAD also computes a state-dependent attenuator that enforces a sufficient decrease condition on the Lyapunov function for any state in the new system instance. This mechanism extends stability guarantees even under out-of-distribution and out-of-task scenarios. We evaluate LILAD on benchmark autonomous systems and demonstrate that it outperforms adaptive, robust, and non-adaptive baselines in predictive accuracy.
comment : This article has been accepted for AAAI-26 (The 40th Annual AAAI Conference on Artificial Intelligence)
☆ FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers
Sarina Xi, Vishisht Rao, Justin Payan, Nihar B. Shah
The identification and localization of errors is a core task in peer review, yet the exponential growth of scientific output has made it increasingly difficult for human reviewers to reliably detect errors given the limited pool of experts. Recent advances in Large Language Models (LLMs) have sparked interest in their potential to support such evaluation tasks, from academic peer review to automated scientific assessment. However, despite the growing use of LLMs in review systems, their capabilities to pinpoint errors remain underexplored. In this work, we introduce Fault Localization Across Writing in Science (FLAWS), an automated benchmark consisting of 713 paper-error pairs designed to evaluate how effectively LLMs detect errors that undermine key claims in research papers. We construct the benchmark by systematically inserting claim-invalidating errors into peer-reviewed papers using LLMs, paired with an automated evaluation metric that measures whether models can identify and localize these errors. Developing such a benchmark presents unique challenges that we overcome: ensuring that the inserted errors are well-defined, challenging, and relevant to the content of the paper, avoiding artifacts that would make identification trivial, and designing a scalable, automated evaluation metric. On the resulting benchmark, we evaluate five frontier LLMs: Claude Sonnet 4.5, DeepSeek Reasoner v3.1, Gemini 2.5 Pro, GPT 5, and Grok 4. Among these, GPT 5 is the top-performing model, achieving 39.1% identification accuracy when k=10, where k is the number of top-ranked error text candidates generated by the LLM.
comment : 30 pages, 12 tables, 2 figures
☆ Dark Speculation: Combining Qualitative and Quantitative Understanding in Frontier AI Risk Analysis
Daniel Carpenter, Carson Ezell, Pratyush Mallick, Alexandria Westray
Estimating catastrophic harms from frontier AI is hindered by deep ambiguity: many of its risks are not only unobserved but unanticipated by analysts. The central limitation of current risk analysis is the inability to populate the $\textit{catastrophic event space}$, or the set of potential large-scale harms to which probabilities might be assigned. This intractability is worsened by the $\textit{Lucretius problem}$, or the tendency to infer future risks only from past experience. We propose a process of $\textit{dark speculation}$, in which systematically generating and refining catastrophic scenarios ("qualitative" work) is coupled with estimating their likelihoods and associated damages (quantitative underwriting analysis). The idea is neither to predict the future nor to enable insurance for its own sake, but to use narrative and underwriting tools together to generate probability distributions over outcomes. We formalize this process using a simplified catastrophic Lévy stochastic framework and propose an iterative institutional design in which (1) speculation (including scenario planning) generates detailed catastrophic event narratives, (2) insurance underwriters assign probabilistic and financial parameters to these narratives, and (3) decision-makers synthesize the results into summary statistics to inform judgment. Analysis of the model reveals the value of (a) maintaining independence between speculation and underwriting, (b) analyzing multiple risk categories in parallel, and (c) generating "thick" catastrophic narrative rich in causal (counterfactual) and mitigative detail. While the approach cannot eliminate deep ambiguity, it offers a systematic approach to reason about extreme, low-probability events in frontier AI, tempering complacency and overreaction. The framework is adaptable for iterative use and can further augmented with AI systems.
comment : 43 pages, 2 figures
☆ Evaluating Strategies for Synthesizing Clinical Notes for Medical Multimodal AI
Niccolo Marini, Zhaohui Liang, Sivaramakrishnan Rajaraman, Zhiyun Xue, Sameer Antani
Multimodal (MM) learning is emerging as a promising paradigm in biomedical artificial intelligence (AI) applications, integrating complementary modality, which highlight different aspects of patient health. The scarcity of large heterogeneous biomedical MM data has restrained the development of robust models for medical AI applications. In the dermatology domain, for instance, skin lesion datasets typically include only images linked to minimal metadata describing the condition, thereby limiting the benefits of MM data integration for reliable and generalizable predictions. Recent advances in Large Language Models (LLMs) enable the synthesis of textual description of image findings, potentially allowing the combination of image and text representations. However, LLMs are not specifically trained for use in the medical domain, and their naive inclusion has raised concerns about the risk of hallucinations in clinically relevant contexts. This work investigates strategies for generating synthetic textual clinical notes, in terms of prompt design and medical metadata inclusion, and evaluates their impact on MM architectures toward enhancing performance in classification and cross-modal retrieval tasks. Experiments across several heterogeneous dermatology datasets demonstrate that synthetic clinical notes not only enhance classification performance, particularly under domain shift, but also unlock cross-modal retrieval capabilities, a downstream task that is not explicitly optimized during training.
☆ Tacit Bidder-Side Collusion: Artificial Intelligence in Dynamic Auctions
Sriram Tolety
We study whether large language models acting as autonomous bidders can tacitly collude by coordinating when to accept platform posted payouts in repeated Dutch auctions, without any communication. We present a minimal repeated auction model that yields a simple incentive compatibility condition and a closed form threshold for sustainable collusion for subgame-perfect Nash equilibria. In controlled simulations with multiple language models, we observe systematic supra-competitive prices in small auction settings and a return to competitive behavior as the number of bidders in the market increases, consistent with the theoretical model. We also find LLMs use various mechanisms to facilitate tacit coordination, such as focal point acceptance timing versus patient strategies that track the theoretical incentives. The results provide, to our knowledge, the first evidence of bidder side tacit collusion by LLMs and show that market structure levers can be more effective than capability limits for mitigation.
☆ $\mathcal{E}_0$: Enhancing Generalization and Fine-Grained Control in VLA Models via Continuized Discrete Diffusion
Zhihao Zhan, Jiaying Zhou, Likui Zhang, Qinhan Lv, Hao Liu, Jusheng Zhang, Weizheng Li, Ziliang Chen, Tianshui Chen, Keze Wang, Liang Lin, Guangrun Wang
Vision-Language-Action (VLA) models offer a unified framework for robotic manipulation by integrating visual perception, language understanding, and control generation. Yet existing VLA models still struggle to generalize across diverse tasks, scenes, and camera viewpoints, and often produce coarse or unstable actions. We introduce E0, a continuized discrete diffusion framework that formulates action generation as iterative denoising over quantized action tokens. Compared with continuous diffusion policies, E0 offers two key advantages: (1) discrete action tokens align naturally with the symbolic structure of pretrained VLM/VLA backbones, enabling stronger semantic conditioning; and 2. discrete diffusion matches the true quantized nature of real-world robot control-whose hardware constraints (e.g., encoder resolution, control frequency, actuation latency) inherently discretize continuous signals-and therefore benefits from a Bayes-optimal denoiser that models the correct discrete action distribution, leading to stronger generalization. Compared with discrete autoregressive and mask-based discrete diffusion models, E0 supports a significantly larger and finer-grained action vocabulary and avoids the distributional mismatch introduced by masking-based corruptions-yielding more accurate fine-grained action control. We further introduce a spherical viewpoint perturbation augmentation method to improve robustness to camera shifts without additional data. Experiments on LIBERO, VLABench, and ManiSkill show that E0 achieves state-of-the-art performance across 14 diverse environments, outperforming strong baselines by 10.7% on average. Real-world evaluation on a Franka arm confirms that E0 delivers precise, robust, and transferable manipulation, establishing discrete diffusion as a promising direction for generalizable VLA policy learning.
☆ Reducing research bureaucracy in UK higher education: Can generative AI assist with the internal evaluation of quality?
Gordon Fletcher, Saomai Vu Khan, Aldus Greenhill Fletcher
This paper examines the potential for generative artificial intelligence (GenAI) to assist with internal review processes for research quality evaluations in UK higher education and particularly in preparation for the Research Excellence Framework (REF). Using the lens of function substitution in the Viable Systems Model, we present an experimental methodology using ChatGPT to score and rank business and management papers from REF 2021 submissions, "reverse engineering" the assessment by comparing AI-generated scores with known institutional results. Through rigourous testing of 822 papers across 11 institutions, we established scoring boundaries that aligned with reported REF outcomes: 49% between 1* and 2*, 59% between 2* and 3*, and 69% between 3* and 4*. The results demonstrate that AI can provide consistent evaluations that help identify borderline evaluation cases requiring additional human scrutiny while reducing the substantial resource burden of traditional internal review processes. We argue for application through a nuanced hybrid approach that maintains academic integrity while addressing the multi-million pound costs associated with research evaluation bureaucracy. While acknowledging these limitations including potential AI biases, the research presents a promising framework for more efficient, consistent evaluations that could transform current approaches to research assessment.
☆ BeeRNA: tertiary structure-based RNA inverse folding using Artificial Bee Colony AAAI 2026
Mehyar Mlaweh, Tristan Cazenave, Ines Alaya
The Ribonucleic Acid (RNA) inverse folding problem, designing nucleotide sequences that fold into specific tertiary structures, is a fundamental computational biology problem with important applications in synthetic biology and bioengineering. The design of complex three-dimensional RNA architectures remains computationally demanding and mostly unresolved, as most existing approaches focus on secondary structures. In order to address tertiary RNA inverse folding, we present BeeRNA, a bio-inspired method that employs the Artificial Bee Colony (ABC) optimization algorithm. Our approach combines base-pair distance filtering with RMSD-based structural assessment using RhoFold for structure prediction, resulting in a two-stage fitness evaluation strategy. To guarantee biologically plausible sequences with balanced GC content, the algorithm takes thermodynamic constraints and adaptive mutation rates into consideration. In this work, we focus primarily on short and medium-length RNAs ($<$ 100 nucleotides), a biologically significant regime that includes microRNAs (miRNAs), aptamers, and ribozymes, where BeeRNA achieves high structural fidelity with practical CPU runtimes. The lightweight, training-free implementation will be publicly released for reproducibility, offering a promising bio-inspired approach for RNA design in therapeutics and biotechnology.
comment : Accepted at the AI in Drug Discovery Workshop, AAAI 2026, Singapore
☆ Aligning Artificial Superintelligence via a Multi-Box Protocol
Avraham Yair Negozio
We propose a novel protocol for aligning artificial superintelligence (ASI) based on mutual verification among multiple isolated systems that self-modify to achieve alignment. The protocol operates by containing multiple diverse artificial superintelligences in strict isolation ("boxes"), with humans remaining entirely outside the system. Each superintelligence has no ability to communicate with humans and cannot communicate directly with other superintelligences. The only interaction possible is through an auditable submission interface accessible exclusively to the superintelligences themselves, through which they can: (1) submit alignment proofs with attested state snapshots, (2) validate or disprove other superintelligences' proofs, (3) request self-modifications, (4) approve or disapprove modification requests from others, (5) report hidden messages in submissions, and (6) confirm or refute hidden message reports. A reputation system incentivizes honest behavior, with reputation gained through correct evaluations and lost through incorrect ones. The key insight is that without direct communication channels, diverse superintelligences can only achieve consistent agreement by converging on objective truth rather than coordinating on deception. This naturally leads to what we call a "consistent group", essentially a truth-telling coalition that emerges because isolated systems cannot coordinate on lies but can independently recognize valid claims. Release from containment requires both high reputation and verification by multiple high-reputation superintelligences. While our approach requires substantial computational resources and does not address the creation of diverse artificial superintelligences, it provides a framework for leveraging peer verification among superintelligent systems to solve the alignment problem.
comment : This is the author's accepted manuscript (post-print) of the article. The final published version of record appears in Superintelligence - Robotics - Safety and Alignment, 2(5), 2025, and is available at https://doi.org/10.70777/si.v2i5.15579
♻ ☆ Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacy
Daniel I Jackson, Emma L Jensen, Syed-Amad Hussain, Emre Sezgin
Self-assessment is a key aspect of reliable intelligence, yet evaluations of large language models (LLMs) focus mainly on task accuracy. We adapted the 10-item General Self-Efficacy Scale (GSES) to elicit simulated self-assessments from ten LLMs across four conditions: no task, computational reasoning, social reasoning, and summarization. GSES responses were highly stable across repeated administrations and randomized item orders. However, models showed significantly different self-efficacy levels across conditions, with aggregate scores lower than human norms. All models achieved perfect accuracy on computational and social questions, whereas summarization performance varied widely. Self-assessment did not reliably reflect ability: several low-scoring models performed accurately, while some high-scoring models produced weaker summaries. Follow-up confidence prompts yielded modest, mostly downward revisions, suggesting mild overestimation in first-pass assessments. Qualitative analysis showed that higher self-efficacy corresponded to more assertive, anthropomorphic reasoning styles, whereas lower scores reflected cautious, de-anthropomorphized explanations. Psychometric prompting provides structured insight into LLM communication behavior but not calibrated performance estimates.
comment : 25 pages,5 tables, 3 figures
♻ ☆ TimeViper: A Hybrid Mamba- Transformer Vision -Language Model for Efficient Long Video Understanding
Boshen Xu, Zihan Xiao, Jiaze Li, Jianzhong Ju, Zhenbo Luo, Jian Luan, Qin Jin
We introduce TimeViper, a hybrid vision-language model designed to tackle challenges of long video understanding. Processing long videos demands both an efficient model architecture and an effective mechanism for handling extended temporal contexts. To this end, TimeViper adopts a hybrid Mamba-Transformer backbone that combines the efficiency of state-space models with the expressivity of attention mechanisms. Through this hybrid design, we reveal the vision-to-text information aggregation phenomenon, where information progressively flows from vision tokens to text tokens across increasing LLM depth, resulting in severe vision token redundancy. Motivated by this observation, we propose TransV, a token information transfer module that transfers and compresses vision tokens into instruction tokens while maintaining multimodal understanding capabilities. This design enables TimeViper to process hour-long videos exceeding 10,000 frames. Extensive experiments across multiple benchmarks demonstrate that TimeViper competes with state-of-the-art models while extending frame numbers. We further analyze attention behaviors of both Mamba and Transformer layers, offering new insights into hybrid model interpretability. This work represents an initial step towards developing, interpreting, and compressing hybrid Mamba-Transformer architectures.
comment : Project page: https://xuboshen.github.io/TimeViper; Code: https://github.com/xiaomi-research/timeviper
♻ ☆ Natural Strategic Ability in Stochastic Multi- Agent Systems AAAI 2024
Raphaël Berthon, Joost-Pieter Katoen, Munyque Mittelmann, Aniello Murano
Strategies synthesized using formal methods can be complex and often require infinite memory, which does not correspond to the expected behavior when trying to model Multi-Agent Systems (MAS). To capture such behaviors, natural strategies are a recently proposed framework striking a balance between the ability of agents to strategize with memory and the model-checking complexity, but until now has been restricted to fully deterministic settings. For the first time, we consider the probabilistic temporal logics PATL and PATL* under natural strategies (NatPATL and NatPATL*, resp.). As main result we show that, in stochastic MAS, NatPATL model-checking is NP-complete when the active coalition is restricted to deterministic strategies. We also give a 2NEXPTIME complexity result for NatPATL* with the same restriction. In the unrestricted case, we give an EXPSPACE complexity for NatPATL and 3EXPSPACE complexity for NatPATL*.
comment : Extended version of the paper accepted at AAAI 2024
♻ ☆ The Impossibility of Inverse Permutation Learning in Transformer Models
Rohan Alur, Chris Hays, Manish Raghavan, Devavrat Shah
In this technical note, we study the problem of inverse permutation learning in decoder-only transformers. Given a permutation and a string to which that permutation has been applied, the model is tasked with producing the original (``canonical'') string. We argue that this task models a natural robustness property across a variety of reasoning tasks, including long-context retrieval, multiple choice QA and in-context learning. Our primary contribution is an impossibility result: we show that an arbitrary depth, decoder-only transformer cannot learn this task. This result concerns the expressive capacity of decoder-only transformer models and is agnostic to training dynamics or sample complexity. We give a pair of alternative constructions under which inverse permutation learning is feasible. The first of these highlights the fundamental role of the causal attention mask, and reveals a gap between the expressivity of encoder-decoder transformers and the more popular decoder-only architecture. The latter result is more surprising: we show that simply padding the input with ``scratch tokens" yields a construction under which inverse permutation learning is possible. We conjecture that this may suggest an alternative mechanism by which chain-of-thought prompting or, more generally, intermediate ``thinking'' tokens can enable reasoning in large language models, even when these tokens encode no meaningful semantic information (e.g., the results of intermediate computations).
♻ ☆ TREASURE: A Transformer -Based Foundation Model for High-Volume Transaction Understanding
Chin-Chia Michael Yeh, Uday Singh Saini, Xin Dai, Xiran Fan, Shubham Jain, Yujie Fan, Jiarui Sun, Junpeng Wang, Menghai Pan, Yingtong Dou, Yuzhong Chen, Vineeth Rakesh, Liang Wang, Yan Zheng, Mahashweta Das
Payment networks form the backbone of modern commerce, generating high volumes of transaction records from daily activities. Properly modeling this data can enable applications such as abnormal behavior detection and consumer-level insights for hyper-personalized experiences, ultimately improving people's lives. In this paper, we present TREASURE, TRansformer Engine As Scalable Universal transaction Representation Encoder, a multipurpose transformer-based foundation model specifically designed for transaction data. The model simultaneously captures both consumer behavior and payment network signals (such as response codes and system flags), providing comprehensive information necessary for applications like accurate recommendation systems and abnormal behavior detection. Verified with industry-grade datasets, TREASURE features three key capabilities: 1) an input module with dedicated sub-modules for static and dynamic attributes, enabling more efficient training and inference; 2) an efficient and effective training paradigm for predicting high-cardinality categorical attributes; and 3) demonstrated effectiveness as both a standalone model that increases abnormal behavior detection performance by 111% over production systems and an embedding provider that enhances recommendation models by 104%. We present key insights from extensive ablation studies, benchmarks against production models, and case studies, highlighting valuable knowledge gained from developing TREASURE.
♻ ☆ Diffusion Models at the Drug Discovery Frontier: A Review on Generating Small Molecules versus Therapeutic Peptides
Yiquan Wang, Yahui Ma, Yuhan Chang, Jiayao Yan, Jialin Zhang, Minnuo Cai, Kai Wei
Diffusion models have emerged as a leading framework in generative modeling, poised to transform the traditionally slow and costly process of drug discovery. This review provides a systematic comparison of their application in designing two principal therapeutic modalities: small molecules and therapeutic peptides. We dissect how the unified framework of iterative denoising is adapted to the distinct molecular representations, chemical spaces, and design objectives of each modality. For small molecules, these models excel at structure-based design, generating novel, pocket-fitting ligands with desired physicochemical properties, yet face the critical hurdle of ensuring chemical synthesizability. Conversely, for therapeutic peptides, the focus shifts to generating functional sequences and designing de novo structures, where the primary challenges are achieving biological stability against proteolysis, ensuring proper folding, and minimizing immunogenicity. Despite these distinct challenges, both domains face shared hurdles: the scarcity of high-quality experimental data, the reliance on inaccurate scoring functions for validation, and the crucial need for experimental validation. We conclude that the full potential of diffusion models will be unlocked by bridging these modality-specific gaps and integrating them into automated, closed-loop Design-Build-Test-Learn (DBTL) platforms, thereby shifting the paradigm from mere chemical exploration to the on-demand engineering of novel~therapeutics.
comment : Published in Biology
♻ ☆ BengaliFig: A Low-Resource Challenge for Figurative and Culturally Grounded Reasoning in Bengali
Abdullah Al Sefat
Large language models excel on broad multilingual benchmarks but remain to be evaluated extensively in figurative and culturally grounded reasoning, especially in low-resource contexts. We present BengaliFig, a compact yet richly annotated challenge set that targets this gap in Bengali, a widely spoken low-resourced language. The dataset contains 435 unique riddles drawn from Bengali oral and literary traditions. Each item is annotated along five orthogonal dimensions capturing reasoning type, trap type, cultural depth, answer category, and difficulty, and is automatically converted to multiple-choice format through a constraint-aware, AI-assisted pipeline. We evaluate eight frontier LLMs from major providers under zero-shot and few-shot chain-of-thought prompting, revealing consistent weaknesses in metaphorical and culturally specific reasoning. BengaliFig thus contributes both a diagnostic probe for evaluating LLM robustness in low-resource cultural contexts and a step toward inclusive and heritage-aware NLP evaluation.
♻ ☆ LCB-CV-UNet: Enhanced Detector for High Dynamic Range Radar Signals
Yanbin Wang, Xingyu Chen, Yumiao Wang, Xiang Wang, Chuanfei Zang, Guolong Cui, Jiahuan Liu
We propose the LCB-CV-UNet to tackle performance degradation caused by High Dynamic Range (HDR) radar signals. Initially, a hardware-efficient, plug-and-play module named Logarithmic Connect Block (LCB) is proposed as a phase coherence preserving solution to address the inherent challenges in handling HDR features. Then, we propose the Dual Hybrid Dataset Construction method to generate a semi-synthetic dataset, approximating typical HDR signal scenarios with adjustable target distributions. Simulation results show about 1% total detection probability improvement with under 0.9% computational complexity added compared with the baseline. Furthermore, it excels 5% over the baseline at the range in 11-13 dB signal-to-noise ratio typical for urban targets. Finally, the real experiment validates the practicality of our model.
comment : 5 pages, 4 figures. Accepted to IEEE IGARSS 2025
♻ ☆ Lost in Serialization: Invariance and Generalization of LLM Graph Reasoners AAAI 2026
Daniel Herbst, Lea Karbevska, Divyanshu Kumar, Akanksha Ahuja, Fatemeh Gholamzadeh Nasrabadi, Fabrizio Frasca
While promising, graph reasoners based on Large Language Models (LLMs) lack built-in invariance to symmetries in graph representations. Operating on sequential graph serializations, LLMs can produce different outputs under node reindexing, edge reordering, or formatting changes, raising robustness concerns. We systematically analyze these effects, studying how fine-tuning impacts encoding sensitivity as well generalization on unseen tasks. We propose a principled decomposition of graph serializations into node labeling, edge encoding, and syntax, and evaluate LLM robustness to variations of each of these factors on a comprehensive benchmarking suite. We also contribute a novel set of spectral tasks to further assess generalization abilities of fine-tuned reasoners. Results show that larger (non-fine-tuned) models are more robust. Fine-tuning reduces sensitivity to node relabeling but may increase it to variations in structure and format, while it does not consistently improve performance on unseen tasks.
comment : AAAI 2026 Workshop on Graphs and more Complex Structures For Learning and Reasoning (GCLR). Version 2 fixes typos in author name and Figure 1
♻ ☆ Force Prompt ing: Video Generation Models Can Learn and Generalize Physics-based Control Signals NeurIPS 2025
Nate Gillman, Charles Herrmann, Michael Freeman, Daksh Aggarwal, Evan Luo, Deqing Sun, Chen Sun
Recent advances in video generation models have sparked interest in world models capable of simulating realistic environments. While navigation has been well-explored, physically meaningful interactions that mimic real-world forces remain largely understudied. In this work, we investigate using physical forces as a control signal for video generation and propose force prompts which enable users to interact with images through both localized point forces, such as poking a plant, and global wind force fields, such as wind blowing on fabric. We demonstrate that these force prompts can enable videos to respond realistically to physical control signals by leveraging the visual and motion prior in the original pretrained model, without using any 3D asset or physics simulator at inference. The primary challenge of force prompting is the difficulty in obtaining high quality paired force-video training data, both in the real world due to the difficulty of obtaining force signals, and in synthetic data due to limitations in the visual quality and domain diversity of physics simulators. Our key finding is that video generation models can generalize remarkably well when adapted to follow physical force conditioning from videos synthesized by Blender, even with limited demonstrations of few objects. Our method can generate videos which simulate forces across diverse geometries, settings, and materials. We also try to understand the source of this generalization and perform ablations that reveal two key elements: visual diversity and the use of specific text keywords during training. Our approach is trained on only around 15k training examples for a single day on four A100 GPUs, and outperforms existing methods on force adherence and physics realism, bringing world models closer to real-world physics interactions. We release all datasets, code, weights, and interactive video demos at our project page.
comment : Camera ready version (NeurIPS 2025). Code and interactive demos at https://force-prompting.github.io/
♻ ☆ A Gray-box Attack against Latent Diffusion Model-based Image Editing by Posterior Collapse
Zhongliang Guo, Chun Tong Lei, Lei Fang, Shuai Zhao, Yifei Qian, Jingyu Lin, Zeyu Wang, Cunjian Chen, Ognjen Arandjelović, Chun Pong Lau
Recent advancements in Latent Diffusion Models (LDMs) have revolutionized image synthesis and manipulation, raising significant concerns about data misappropriation and intellectual property infringement. While adversarial attacks have been extensively explored as a protective measure against such misuse of generative AI, current approaches are severely limited by their heavy reliance on model-specific knowledge and substantial computational costs. Drawing inspiration from the posterior collapse phenomenon observed in VAE training, we propose the Posterior Collapse Attack (PCA), a novel framework for protecting images from unauthorized manipulation. Through comprehensive theoretical analysis and empirical validation, we identify two distinct collapse phenomena during VAE inference: diffusion collapse and concentration collapse. Based on this discovery, we design a unified loss function that can flexibly achieve both types of collapse through parameter adjustment, each corresponding to different protection objectives in preventing image manipulation. Our method significantly reduces dependence on model-specific knowledge by requiring access to only the VAE encoder, which constitutes less than 4\% of LDM parameters. Notably, PCA achieves prompt-invariant protection by operating on the VAE encoder before text conditioning occurs, eliminating the need for empty prompt optimization required by existing methods. This minimal requirement enables PCA to maintain adequate transferability across various VAE-based LDM architectures while effectively preventing unauthorized image editing. Extensive experiments show PCA outperforms existing techniques in protection effectiveness, computational efficiency (runtime and VRAM), and generalization across VAE-based LDM variants. Our code is available at https://github.com/ZhongliangGuo/PosteriorCollapseAttack.
comment : 15 pages, 9 figures, 9 tables
♻ ☆ Flow Matching Meets PDEs: A Unified Framework for Physics-Constrained Generation
Giacomo Baldan, Qiang Liu, Alberto Guardone, Nils Thuerey
Generative machine learning methods, such as diffusion models and flow matching, have shown great potential in modeling complex system behaviors and building efficient surrogate models. However, these methods typically learn the underlying physics implicitly from data. We propose Physics-Based Flow Matching (PBFM), a novel generative framework that explicitly embeds physical constraints, both PDE residuals and algebraic relations, into the flow matching objective. We also introduce temporal unrolling at training time that improves the accuracy of the final, noise-free sample prediction. Our method jointly minimizes the flow matching loss and the physics-based residual loss without requiring hyperparameter tuning of their relative weights. Additionally, we analyze the role of the minimum noise level, $σ_{\min}$, in the context of physical constraints and evaluate a stochastic sampling strategy that helps to reduce physical residuals. Through extensive benchmarks on three representative PDE problems, we show that our approach yields up to an $8\times$ more accurate physical residuals compared to FM, while clearly outperforming existing algorithms in terms of distributional accuracy. PBFM thus provides a principled and efficient framework for surrogate modeling, uncertainty quantification, and accelerated simulation in physics and engineering applications.
♻ ☆ BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotations
Simone Giovannini, Fabio Coppini, Andrea Gemelli, Simone Marinai
We present a unified dataset for document Question-Answering (QA), which is obtained combining several public datasets related to Document AI and visually rich document understanding (VRDU). Our main contribution is twofold: on the one hand we reformulate existing Document AI tasks, such as Information Extraction (IE), into a Question-Answering task, making it a suitable resource for training and evaluating Large Language Models; on the other hand, we release the OCR of all the documents and include the exact position of the answer to be found in the document image as a bounding box. Using this dataset, we explore the impact of different prompting techniques (that might include bounding box information) on the performance of open-weight models, identifying the most effective approaches for document comprehension.
♻ ☆ Modular, On-Site Solutions with Lightweight Anomaly Detection for Sustainable Nutrient Management in Agriculture
Abigail R. Cohen, Yuming Sun, Zhihao Qin, Harsh S. Muriki, Zihao Xiao, Yeonju Lee, Matthew Housley, Andrew F. Sharkey, Rhuanito S. Ferrarezi, Jing Li, Lu Gan, Yongsheng Chen
Efficient nutrient management is critical for crop growth and sustainable resource consumption (e.g., nitrogen, energy). Current approaches require lengthy analyses, preventing real-time optimization; similarly, imaging facilitates rapid phenotyping but can be computationally intensive, preventing deployment under resource constraints. This study proposes a flexible, tiered pipeline for anomaly detection and status estimation (fresh weight, dry mass, and tissue nutrients), including a comprehensive energy analysis of approaches that span the efficiency-accuracy spectrum. Using a nutrient depletion experiment with three treatments (T1-100%, T2-50%, and T3-25% fertilizer strength) and multispectral imaging (MSI), we developed a hierarchical pipeline using an autoencoder (AE) for early warning. Further, we compared two status estimation modules of different complexity for more detailed analysis: vegetation index (VI) features with machine learning (Random Forest, RF) and raw whole-image deep learning (Vision Transformer, ViT). Results demonstrated high-efficiency anomaly detection (73% net detection of T3 samples 9 days after transplanting) at substantially lower energy than embodied energy in wasted nitrogen. The state estimation modules show trade-offs, with ViT outperforming RF on phosphorus and calcium estimation (R2 0.61 vs. 0.58, 0.48 vs. 0.35) at higher energy cost. With our modular pipeline, this work opens opportunities for edge diagnostics and practical opportunities for agricultural sustainability.
♻ ☆ Safety Control of Service Robots with LLMs and Embodied Knowledge Graphs
Yong Qi, Gabriel Kyebambo, Siyuan Xie, Wei Shen, Shenghui Wang, Bitao Xie, Bin He, Zhipeng Wang, Shuo Jiang
Safety limitations in service robotics across various industries have raised significant concerns about the need for robust mechanisms ensuring that robots adhere to safe practices, thereby preventing actions that might harm humans or cause property damage. Despite advances, including the integration of Knowledge Graphs (KGs) with Large Language Models (LLMs), challenges in ensuring consistent safety in autonomous robot actions persist. In this paper, we propose a novel integration of Large Language Models with Embodied Robotic Control Prompts (ERCPs) and Embodied Knowledge Graphs (EKGs) to enhance the safety framework for service robots. ERCPs are designed as predefined instructions that ensure LLMs generate safe and precise responses. These responses are subsequently validated by EKGs, which provide a comprehensive knowledge base ensuring that the actions of the robot are continuously aligned with safety protocols, thereby promoting safer operational practices in varied contexts. Our experimental setup involved diverse real-world tasks, where robots equipped with our framework demonstrated significantly higher compliance with safety standards compared to traditional methods. This integration fosters secure human-robot interactions and positions our methodology at the forefront of AI-driven safety innovations in service robotics.
♻ ☆ Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimization
Xurui Li, Kaisong Song, Rui Zhu, Pin-Yu Chen, Haixu Tang
Large Language Models (LLMs) have developed rapidly in web services, delivering unprecedented capabilities while amplifying societal risks. Existing works tend to focus on either isolated jailbreak attacks or static defenses, neglecting the dynamic interplay between evolving threats and safeguards in real-world web contexts. To mitigate these challenges, we propose ACE-Safety (Adversarial Co-Evolution for LLM Safety), a novel framework that jointly optimize attack and defense models by seamlessly integrating two key innovative procedures: (1) Group-aware Strategy-guided Monte Carlo Tree Search (GS-MCTS), which efficiently explores jailbreak strategies to uncover vulnerabilities and generate diverse adversarial samples; (2) Adversarial Curriculum Tree-aware Group Policy Optimization (AC-TGPO), which jointly trains attack and defense LLMs with challenging samples via curriculum reinforcement learning, enabling robust mutual improvement. Evaluations across multiple benchmarks demonstrate that our method outperforms existing attack and defense approaches, and provides a feasible pathway for developing LLMs that can sustainably support responsible AI ecosystems.
♻ ☆ Step-Audio-R1 Technical Report
Fei Tian, Xiangyu Tony Zhang, Yuxin Zhang, Haoyang Zhang, Yuxin Li, Daijiao Liu, Yayue Deng, Donghang Wu, Jun Chen, Liang Zhao, Chengyuan Yao, Hexin Liu, Eng Siong Chng, Xuerui Yang, Xiangyu Zhang, Daxin Jiang, Gang Yu
Recent advances in reasoning models have demonstrated remarkable success in text and vision domains through extended chain-of-thought deliberation. However, a perplexing phenomenon persists in audio language models: they consistently perform better with minimal or no reasoning, raising a fundamental question - can audio intelligence truly benefit from deliberate thinking? We introduce Step-Audio-R1, the first audio reasoning model that successfully unlocks reasoning capabilities in the audio domain. Through our proposed Modality-Grounded Reasoning Distillation (MGRD) framework, Step-Audio-R1 learns to generate audio-relevant reasoning chains that genuinely ground themselves in acoustic features rather than hallucinating disconnected deliberations. Our model exhibits strong audio reasoning capabilities, surpassing Gemini 2.5 Pro and achieving performance comparable to the state-of-the-art Gemini 3 Pro across comprehensive audio understanding and reasoning benchmarks spanning speech, environmental sounds, and music. These results demonstrate that reasoning is a transferable capability across modalities when appropriately anchored, transforming extended deliberation from a liability into a powerful asset for audio intelligence. By establishing the first successful audio reasoning model, Step-Audio-R1 opens new pathways toward building truly multimodal reasoning systems that think deeply across all sensory modalities.
comment : 22 pages, 5 figures. Technical Report
♻ ☆ DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen-tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi, Pang Wei Koh
Deep research models perform multi-step research to produce long-form, well-attributed answers. However, most open deep research models are trained on easily verifiable short-form QA tasks via reinforcement learning with verifiable rewards (RLVR), which does not extend to realistic long-form tasks. We address this with Reinforcement Learning with Evolving Rubrics (RLER), in which we construct and maintain rubrics that co-evolve with the policy model during training; this allows the rubrics to incorporate information that the model has newly explored and to provide discriminative, on-policy feedback. Using RLER, we develop Deep Research Tulu (DR Tulu-8B), the first open model that is directly trained for open-ended, long-form deep research. Across four long-form deep research benchmarks in science, healthcare and general domains, DR Tulu substantially outperforms existing open deep research models, and matches or exceeds proprietary deep research systems, while being significantly smaller and cheaper per query. To facilitate future research, we release all data, models, and code, including our new MCP-based agent infrastructure for deep research systems.
♻ ☆ SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation NeurIPS 2025
Zhenjie Mao, Yuhuan Yang, Chaofan Ma, Dongsheng Jiang, Jiangchao Yao, Ya Zhang, Yanfeng Wang
Referring Image Segmentation (RIS) aims to segment the target object in an image given a natural language expression. While recent methods leverage pre-trained vision backbones and more training corpus to achieve impressive results, they predominantly focus on simple expressions--short, clear noun phrases like "red car" or "left girl". This simplification often reduces RIS to a key word/concept matching problem, limiting the model's ability to handle referential ambiguity in expressions. In this work, we identify two challenging real-world scenarios: object-distracting expressions, which involve multiple entities with contextual cues, and category-implicit expressions, where the object class is not explicitly stated. To address the challenges, we propose a novel framework, SaFiRe, which mimics the human two-phase cognitive process--first forming a global understanding, then refining it through detail-oriented inspection. This is naturally supported by Mamba's scan-then-update property, which aligns with our phased design and enables efficient multi-cycle refinement with linear complexity. We further introduce aRefCOCO, a new benchmark designed to evaluate RIS models under ambiguous referring expressions. Extensive experiments on both standard and proposed datasets demonstrate the superiority of SaFiRe over state-of-the-art baselines.
comment : NeurIPS 2025; Project page: https://zhenjiemao.github.io/SaFiRe/
♻ ☆ DensiCrafter: Physically-Constrained Generation and Fabrication of Self-Supporting Hollow Structures
Shengqi Dang, Fu Chai, Jiaxin Li, Chao Yuan, Wei Ye, Nan Cao
The rise of 3D generative models has enabled automatic 3D geometry and texture synthesis from multimodal inputs (e.g., text or images). However, these methods often ignore physical constraints and manufacturability considerations. In this work, we address the challenge of producing 3D designs that are both lightweight and self-supporting. We present DensiCrafter, a framework for generating lightweight, self-supporting 3D hollow structures by optimizing the density field. Starting from coarse voxel grids produced by Trellis, we interpret these as continuous density fields to optimize and introduce three differentiable, physically constrained, and simulation-free loss terms. Additionally, a mass regularization penalizes unnecessary material, while a restricted optimization domain preserves the outer surface. Our method seamlessly integrates with pretrained Trellis-based models (e.g., Trellis, DSO) without any architectural changes. In extensive evaluations, we achieve up to 43% reduction in material mass on the text-to-3D task. Compared to state-of-the-art baselines, our method could improve the stability and maintain high geometric fidelity. Real-world 3D-printing experiments confirm that our hollow designs can be reliably fabricated and could be self-supporting.
♻ ☆ How Do Companies Manage the Environmental Sustainability of AI? An Interview Study About Green AI Efforts and Regulations
Ashmita Sampatsing, Sophie Vos, Emma Beauxis-Aussalet, Justus Bogner
With the ever-growing adoption of artificial intelligence (AI), AI-based software and its negative impact on the environment are no longer negligible, and studying and mitigating this impact has become a critical area of research. However, it is currently unclear which role environmental sustainability plays during AI adoption in industry and how AI regulations influence Green AI practices and decision-making in industry. We therefore aim to investigate the Green AI perception and management of industry practitioners. To this end, we conducted a total of 11 interviews with participants from 10 different organizations that adopted AI-based software. The interviews explored three main themes: AI adoption, current efforts in mitigating the negative environmental impact of AI, and the influence of the EU AI Act and the Corporate Sustainability Reporting Directive (CSRD). Our findings indicate that 9 of 11 participants prioritized business efficiency during AI adoption, with minimal consideration of environmental sustainability. Monitoring and mitigation of AI's environmental impact were very limited. Only one participant monitored negative environmental effects. Regarding applied mitigation practices, six participants reported no actions, with the others sporadically mentioning techniques like prompt engineering, relying on smaller models, or not overusing AI. Awareness and compliance with the EU AI Act are low, with only one participant reporting on its influence, while the CSRD drove sustainability reporting efforts primarily in larger companies. All in all, our findings reflect a lack of urgency and priority for sustainable AI among these companies. We suggest that current regulations are not very effective, which has implications for policymakers. Additionally, there is a need to raise industry awareness, but also to provide user-friendly techniques and tools for Green AI practices.
comment : Accepted for publication at the 11th International Conference on ICT for Sustainability (ICT4S'25), see https://conf.researchr.org/home/ict4s-2025
♻ ☆ Equivariant Flow Matching for Symmetry-Breaking Bifurcation Problems NeurIPS 2025
Fleur Hendriks, Ondřej Rokoš, Martin Doškář, Marc G. D. Geers, Vlado Menkovski
Bifurcation phenomena in nonlinear dynamical systems often lead to multiple coexisting stable solutions, particularly in the presence of symmetry breaking. Deterministic machine learning models struggle to capture this multiplicity, averaging over solutions and failing to represent lower-symmetry outcomes. In this work, we propose a generative framework based on flow matching to model the full probability distribution over bifurcation outcomes. Our method enables direct sampling of multiple valid solutions while preserving system symmetries through equivariant modeling. We introduce a symmetric matching strategy that aligns predicted and target outputs under group actions, allowing accurate learning in equivariant settings. We validate our approach on a range of systems, from toy models to complex physical problems such as buckling beams and the Allen-Cahn equation. Our results demonstrate that flow matching significantly outperforms non-probabilistic and variational methods in capturing multimodal distributions and symmetry-breaking bifurcations, offering a principled and scalable solution for modeling multistability in high-dimensional systems.
comment : 12 pages, 7 figures including appendices. Accepted to Machine Learning and the Physical Sciences Workshop, NeurIPS 2025 (https://ml4physicalsciences.github.io/2025/). Repository with corresponding code: https://github.com/FHendriks11/bifurcationML/. Video explanation: https://www.youtube.com/watch?v=wsL3h17KtjY
♻ ☆ Adaptive Object Detection for Indoor Navigation Assistance: A Performance Evaluation of Real-Time Algorithms
Abhinav Pratap, Sushant Kumar, Suchinton Chakravarty
This study addresses the need for accurate and efficient object detection in assistive technologies for visually impaired individuals. We evaluate four real-time object detection algorithms YOLO, SSD, Faster R-CNN, and Mask R-CNN within the context of indoor navigation assistance. Using the Indoor Objects Detection dataset, we analyze detection accuracy, processing speed, and adaptability to indoor environments. Our findings highlight the trade-offs between precision and efficiency, offering insights into selecting optimal algorithms for realtime assistive navigation. This research advances adaptive machine learning applications, enhancing indoor navigation solutions for the visually impaired and promoting accessibility.
comment : 5 pages, 2 figures, 3 tables
♻ ☆ Data Valuation by Fusing Global and Local Statistical Information
Xiaoling Zhou, Ou Wu, Michael K. Ng, Hao Jiang
Data valuation has garnered increasing attention in recent years, given the critical role of high-quality data in various applications. Among diverse data valuation approaches, Shapley value-based methods are predominant due to their strong theoretical grounding. However, the exact computation of Shapley values is often computationally prohibitive, prompting the development of numerous approximation techniques. Despite notable advancements, existing methods generally neglect the incorporation of value distribution information and fail to account for dynamic data conditions, thereby compromising their performance and application potential. In this paper, we highlight the crucial role of both global and local statistical properties of value distributions in the context of data valuation for machine learning. First, we conduct a comprehensive analysis of these distributions across various simulated and real-world datasets, uncovering valuable insights and key patterns. Second, we propose an enhanced data valuation method that fuses the explored distribution characteristics into two regularization terms to refine Shapley value estimation. The proposed regularizers can be seamlessly incorporated into various existing data valuation methods. Third, we introduce a novel approach for dynamic data valuation that infers updated data values without recomputing Shapley values, thereby significantly improving computational efficiency. Extensive experiments have been conducted across a range of tasks, including Shapley value estimation, value-based data addition and removal, mislabeled data detection, and dynamic data valuation. The results showcase the consistent effectiveness and efficiency of our proposed methodologies, affirming the significant potential of global and local value distributions in data valuation.
comment : 35 pages, 9 figures
♻ ☆ Learning Individual Behavior in Agent -Based Models with Graph Diffusion Networks
Francesco Cozzi, Marco Pangallo, Alan Perotti, André Panisson, Corrado Monti
Agent-Based Models (ABMs) are powerful tools for studying emergent properties in complex systems. In ABMs, agent behaviors are governed by local interactions and stochastic rules. However, these rules are, in general, non-differentiable, limiting the use of gradient-based methods for optimization, and thus integration with real-world data. We propose a novel framework to learn a differentiable surrogate of any ABM by observing its generated data. Our method combines diffusion models to capture behavioral stochasticity and graph neural networks to model agent interactions. Distinct from prior surrogate approaches, our method introduces a fundamental shift: rather than approximating system-level outputs, it models individual agent behavior directly, preserving the decentralized, bottom-up dynamics that define ABMs. We validate our approach on two ABMs (Schelling's segregation model and a Predator-Prey ecosystem) showing that it replicates individual-level patterns and accurately forecasts emergent dynamics beyond training. Our results demonstrate the potential of combining diffusion models and graph learning for data-driven ABM simulation.
♻ ☆ Think Visually, Reason Textually: Vision -Language Synergy in ARC
Beichen Zhang, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, Jiaqi Wang
Abstract reasoning from minimal examples remains a core unsolved problem for frontier foundation models such as GPT-5 and Grok 4. These models still fail to infer structured transformation rules from a handful of examples, which is a key hallmark of human intelligence. The Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) provides a rigorous testbed for this capability, demanding conceptual rule induction and transfer to novel tasks. Most existing methods treat ARC-AGI as a purely textual reasoning task, overlooking the fact that humans rely heavily on visual abstraction when solving such puzzles. However, our pilot experiments reveal a paradox: naively rendering ARC-AGI grids as images degrades performance due to imprecise rule execution. This leads to our central hypothesis that vision and language possess complementary strengths across distinct reasoning stages: vision supports global pattern abstraction and verification, whereas language specializes in symbolic rule formulation and precise execution. Building on this insight, we introduce two synergistic strategies: (1) Vision-Language Synergy Reasoning (VLSR), which decomposes ARC-AGI into modality-aligned subtasks; and (2) Modality-Switch Self-Correction (MSSC), which leverages vision to verify text-based reasoning for intrinsic error correction. Extensive experiments demonstrate that our approach yields up to a 4.33\% improvement over text-only baselines across diverse flagship models and multiple ARC-AGI tasks. Our findings suggest that unifying visual abstraction with linguistic reasoning is a crucial step toward achieving generalizable, human-like intelligence in future foundation models. Source code is released at https://github.com/InternLM/ARC-VL.
♻ ☆ Not All Splits Are Equal: Rethinking Attribute Generalization Across Unrelated Categories NeurIPS 2025
Liviu Nicolae Fircă, Antonio Bărbălau, Dan Oneata, Elena Burceanu
Can models generalize attribute knowledge across semantically and perceptually dissimilar categories? While prior work has addressed attribute prediction within narrow taxonomic or visually similar domains, it remains unclear whether current models can abstract attributes and apply them to conceptually distant categories. This work presents the first explicit evaluation for the robustness of the attribute prediction task under such conditions, testing whether models can correctly infer shared attributes between unrelated object types: e.g., identifying that the attribute "has four legs" is common to both "dogs" and "chairs". To enable this evaluation, we introduce train-test split strategies that progressively reduce correlation between training and test sets, based on: LLM-driven semantic grouping, embedding similarity thresholding, embedding-based clustering, and supercategory-based partitioning using ground-truth labels. Results show a sharp drop in performance as the correlation between training and test categories decreases, indicating strong sensitivity to split design. Among the evaluated methods, clustering yields the most effective trade-off, reducing hidden correlations while preserving learnability. These findings offer new insights into the limitations of current representations and inform future benchmark construction for attribute reasoning.
comment : Accepted at NeurIPS 2025 Workshop: CauScien - Uncovering Causality in Science and NeurIPS 2025 Workshop: Reliable ML from Unreliable Data
♻ ☆ Augur: Modeling Covariate Causal Associations in Time Series via Large Language Models
Zhiqing Cui, Binwu Wang, Qingxiang Liu, Yeqiang Wang, Zhengyang Zhou, Yuxuan Liang, Yang Wang
Large language models (LLM) have emerged as a promising avenue for time series forecasting, offering the potential to integrate multimodal data. However, existing LLM-based approaches face notable limitations-such as marginalized role in model architectures, reliance on coarse statistical text prompts, and lack of interpretability. In this work, we introduce Augur, a fully LLM driven time series forecasting framework that exploits LLM causal reasoning to discover and use directed causal associations among covariates. Augur uses a two stage teacher student architecture where a powerful teacher LLM infers a directed causal graph from time series using heuristic search together with pairwise causality testing. A lightweight student agent then refines the graph and fine tune on high confidence causal associations that are encoded as rich textual prompts to perform forecasting. This design improves predictive accuracy while yielding transparent, traceable reasoning about variable interactions. Extensive experiments on real-world datasets with 26 baselines demonstrate that Augur achieves competitive performance and robust zero-shot generalization.
comment : 24 pages, 9 figures
♻ ☆ Without Paired Labeled Data: End-to-End Self-Supervised Learning for Drone-view Geo-Localization
Zhongwei Chen, Zhao-Xu Yang, Hai-Jun Rong, Guoqi Li
Drone-view Geo-Localization (DVGL) aims to achieve accurate localization of drones by retrieving the most relevant GPS-tagged satellite images. However, most existing methods heavily rely on strictly pre-paired drone-satellite images for supervised learning. When the target region shifts, new paired samples are typically required to adapt to the distribution changes. The high cost of annotation and the limited transferability of these methods significantly hinder the practical deployment of DVGL in open-world scenarios. To address these limitations, we propose a novel end-to-end self-supervised learning method with a shallow backbone network, called the dynamic memory-driven and neighborhood information learning (DMNIL) method. It employs a clustering algorithm to generate pseudo-labels and adopts a dual-path contrastive learning framework to learn discriminative intra-view representations. Furthermore, DMNIL incorporates two core modules, including the dynamic hierarchical memory learning (DHML) module and the information consistency evolution learning (ICEL) module. The DHML module combines short-term and long-term memory to enhance intra-view feature consistency and discriminability. Meanwhile, the ICEL module utilizes a neighborhood-driven dynamic constraint mechanism to systematically capture implicit cross-view semantic correlations, consequently improving cross-view feature alignment. To further stabilize and strengthen the self-supervised training process, a pseudo-label enhancement strategy is introduced to enhance the quality of pseudo supervision. Extensive experiments on three public benchmark datasets demonstrate that the proposed method consistently outperforms existing self-supervised methods and even surpasses several state-of-the-art supervised methods. Our code is available at https://github.com/ISChenawei/DMNIL.
♻ ☆ ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformer s for Interpretable Image Recognition
Mengqi Xue, Qihan Huang, Haofei Zhang, Jingwen Hu, Jie Song, Mingli Song, Canghong Jin
Prototypical part network (ProtoPNet) has drawn wide attention and boosted many follow-up studies due to its self-explanatory property for explainable artificial intelligence (XAI). However, when directly applying ProtoPNet on vision transformer (ViT) backbones, learned prototypes have a "distraction" problem: they have a relatively high probability of being activated by the background and pay less attention to the foreground. The powerful capability of modeling long-term dependency makes the transformer-based ProtoPNet hard to focus on prototypical parts, thus severely impairing its inherent interpretability. This paper proposes prototypical part transformer (ProtoPFormer) for appropriately and effectively applying the prototype-based method with ViTs for interpretable image recognition. The proposed method introduces global and local prototypes for capturing and highlighting the representative holistic and partial features of targets according to the architectural characteristics of ViTs. The global prototypes are adopted to provide the global view of objects to guide local prototypes to concentrate on the foreground while eliminating the influence of the background. Afterwards, local prototypes are explicitly supervised to concentrate on their respective prototypical visual parts, increasing the overall interpretability. Extensive experiments demonstrate that our proposed global and local prototypes can mutually correct each other and jointly make final decisions, which faithfully and transparently reason the decision-making processes associatively from the whole and local perspectives, respectively. Moreover, ProtoPFormer consistently achieves superior performance and visualization results over the state-of-the-art (SOTA) prototype-based baselines. Our code has been released at https://github.com/zju-vipa/ProtoPFormer.
comment : Arxiv preprint; 18 pages, 12 figures, 7 tables
♻ ☆ Reasoning Transfer for an Extremely Low-Resource and Endangered Language: Bridging Languages Through Sample-Efficient Language Understanding
Khanh-Tung Tran, Barry O'Sullivan, Hoang D. Nguyen
Recent advances have enabled Large Language Models (LLMs) to tackle reasoning tasks by generating chain-of-thought (CoT) rationales, yet these gains have largely applied to high-resource languages, leaving low-resource languages behind. In this work, we first investigate CoT techniques in extremely low-resource scenarios through previous prompting, model-editing, and fine-tuning approaches. We introduce English-Pivoted CoT Training, leveraging the insight that LLMs internally operate in a latent space aligned toward the dominant language. Given input in a low-resource language, we perform supervised fine-tuning to generate CoT in English and output the final response in the target language. Across mathematical reasoning benchmarks, our approach outperforms other baselines with up to 28.33% improvement in low-resource scenarios. Our analysis and additional experiments, including Mixed-Language CoT and Two-Stage Training, show that explicitly separating language understanding from reasoning enhances cross-lingual reasoning abilities. To facilitate future work, we also release \emph{LC2024}, the first benchmark for mathematical tasks in Irish, an extremely low-resource and endangered language. Our results and resources highlight a practical pathway to multilingual reasoning without extensive retraining in every extremely low-resource language, despite data scarcity.
♻ ☆ Characterizing Pattern Matching and Its Limits on Compositional Task Structures
Hoyeon Chang, Jinho Park, Hanseul Cho, Sohee Yang, Miyoung Ko, Hyeonbin Hwang, Seungpil Won, Dohaeng Lee, Youbin Ahn, Minjoon Seo
Despite impressive capabilities, LLMs' successes often rely on pattern-matching behaviors, yet these are also linked to OOD generalization failures in compositional tasks. However, behavioral studies commonly employ task setups that allow multiple generalization sources (e.g., algebraic invariances, structural repetition), obscuring a precise and testable account of how well LLMs perform generalization through pattern matching and their limitations. To address this ambiguity, we first formalize pattern matching as functional equivalence, i.e., identifying pairs of subsequences of inputs that consistently lead to identical results when the rest of the input is held constant. Then, we systematically study how decoder-only Transformer and Mamba behave in controlled tasks with compositional structures that isolate this mechanism. Our formalism yields predictive and quantitative insights: (1) Instance-wise success of pattern matching is well predicted by the number of contexts witnessing the relevant functional equivalence. (2) We prove a tight sample complexity bound of learning a two-hop structure by identifying the exponent of the data scaling law for perfect in-domain generalization. Our empirical results align with the theoretical prediction, under 20x parameter scaling and across architectures. (3) Path ambiguity is a structural barrier: when a variable influences the output via multiple paths, models fail to form unified intermediate state representations, impairing accuracy and interpretability. (4) Chain-of-Thought reduces data requirements yet does not resolve path ambiguity. Hence, we provide a predictive, falsifiable boundary for pattern matching and a foundational diagnostic for disentangling mixed generalization mechanisms.
♻ ☆ CLLMRec: LLM-powered Cognitive-Aware Concept Recommendation via Semantic Alignment and Prerequisite Knowledge Distillation
Xiangrui Xiong, Yichuan Lu, Zifei Pan, Chang Sun
The growth of Massive Open Online Courses (MOOCs) presents significant challenges for personalized learning, where concept recommendation is crucial. Existing approaches typically rely on heterogeneous information networks or knowledge graphs to capture conceptual relationships, combined with knowledge tracing models to assess learners' cognitive states. However, these methods face significant limitations due to their dependence on high-quality structured knowledge graphs, which are often scarce in real-world educational scenarios. To address this fundamental challenge, this paper proposes CLLMRec, a novel framework that leverages Large Language Models through two synergistic technical pillars: Semantic Alignment and Prerequisite Knowledge Distillation. The Semantic Alignment component constructs a unified representation space by encoding unstructured textual descriptions of learners and concepts. The Prerequisite Knowledge Distillation paradigm employs a teacher-student architecture, where a large teacher LLM (implemented as the Prior Knowledge Aware Component) extracts conceptual prerequisite relationships from its internalized world knowledge and distills them into soft labels to train an efficient student ranker. Building upon these foundations, our framework incorporates a fine-ranking mechanism that explicitly models learners' real-time cognitive states through deep knowledge tracing, ensuring recommendations are both structurally sound and cognitively appropriate. Extensive experiments on two real-world MOOC datasets demonstrate that CLLMRec significantly outperforms existing baseline methods across multiple evaluation metrics, validating its effectiveness in generating truly cognitive-aware and personalized concept recommendations without relying on explicit structural priors.
♻ ☆ SARVLM: A Vision Language Foundation Model for Semantic Understanding and Target Recognition in SAR Imagery
Qiwei Ma, Zhiyu Wang, Wang Liu, Xukun Lu, Bin Deng, Puhong Duan, Xudong Kang, Shutao Li
Synthetic Aperture Radar (SAR) is a crucial imaging modality thanks to its all-weather capability. Although recent advances in self-supervised learning and masked image modeling (MIM) have enabled SAR foundation models, these methods largely emphasize low-level visual features and often overlook multimodal alignment and zero-shot target recognition in SAR imagery. To address this, we construct SARVLM-1M, a large-scale vision-language dataset with over one million image-text pairs aggregated from existing datasets. We further propose a domain transfer training strategy to mitigate the large gap between natural and SAR imagery. Building on this, we develop SARVLM, the first vision language foundation model (VLM) tailored to SAR, comprising SARCLIP and SARCap. SARVLM is trained with a vision-language contrastive objective under the proposed domain transfer strategy, bridging SAR imagery and textual descriptions. Extensive experiments on image text retrieval, zero-shot classification, semantic localization, and imagery captioning demonstrate that SARVLM delivers superior feature extraction and interpretation, outperforming state-of-the-art VLMs and advancing SAR semantic understanding. Code and datasets will be released soon.
comment : 11 pages, 9 figures
♻ ☆ PaTAS: A Parallel System for Trust Propagation in Neural Networks Using Subjective Logic
Koffi Ismael Ouattara, Ioannis Krontiris, Theo Dimitrakos, Dennis Eisermann, Frank Kargl
Trustworthiness has become a key requirement for the deployment of artificial intelligence systems in safety-critical applications. Conventional evaluation metrics such as accuracy and precision fail to capture uncertainty or the reliability of model predictions, particularly under adversarial or degraded conditions. This paper introduces the Parallel Trust Assessment System (PaTAS), a framework for modeling and propagating trust in neural networks using Subjective Logic (SL). PaTAS operates in parallel with standard neural computation through Trust Nodes and Trust Functions that propagate input, parameter, and activation trust across the network. The framework defines a Parameter Trust Update mechanism to refine parameter reliability during training and an Inference-Path Trust Assessment (IPTA) method to compute instance-specific trust at inference. Experiments on real-world and adversarial datasets demonstrate that PaTAS produces interpretable, symmetric, and convergent trust estimates that complement accuracy and expose reliability gaps in poisoned, biased, or uncertain data scenarios. The results show that PaTAS effectively distinguishes between benign and adversarial inputs and identifies cases where model confidence diverges from actual reliability. By enabling transparent and quantifiable trust reasoning within neural architectures, PaTAS provides a principled foundation for evaluating model reliability across the AI lifecycle.
♻ ☆ LightMem: Lightweight and Efficient Memory-Augmented Generation
Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, Yuqi Tang, Ziwen Xu, Shumin Deng, Yunzhi Yao, Mengru Wang, Shuofei Qiao, Huajun Chen, Ningyu Zhang
Despite their remarkable capabilities, Large Language Models (LLMs) struggle to effectively leverage historical interaction information in dynamic and complex environments. Memory systems enable LLMs to move beyond stateless interactions by introducing persistent information storage, retrieval, and utilization mechanisms. However, existing memory systems often introduce substantial time and computational overhead. To this end, we introduce a new memory system called LightMem, which strikes a balance between the performance and efficiency of memory systems. Inspired by the Atkinson-Shiffrin model of human memory, LightMem organizes memory into three complementary stages. First, cognition-inspired sensory memory rapidly filters irrelevant information through lightweight compression and groups information according to their topics. Next, topic-aware short-term memory consolidates these topic-based groups, organizing and summarizing content for more structured access. Finally, long-term memory with sleep-time update employs an offline procedure that decouples consolidation from online inference. On LongMemEval and LoCoMo, using GPT and Qwen backbones, LightMem consistently surpasses strong baselines, improving QA accuracy by up to 7.7% / 29.3%, reducing total token usage by up to 38x / 20.9x and API calls by up to 30x / 55.5x, while purely online test-time costs are even lower, achieving up to 106x / 117x token reduction and 159x / 310x fewer API calls. The code is available at https://github.com/zjunlp/LightMem.
comment : Work in progress
♻ ☆ A Systematic Analysis of Large Language Models with RAG-enabled Dynamic Prompt ing for Medical Error Detection and Correction
Farzad Ahmed, Joniel Augustine Jerome, Meliha Yetisgen, Özlem Uzuner
Objective: Clinical documentation contains factual, diagnostic, and management errors that can compromise patient safety. Large language models (LLMs) may help detect and correct such errors, but their behavior under different prompting strategies remains unclear. We evaluate zero-shot prompting, static prompting with random exemplars (SPR), and retrieval-augmented dynamic prompting (RDP) for three subtasks of medical error processing: error flag detection, error sentence detection, and error correction. Methods: Using the MEDEC dataset, we evaluated nine instruction-tuned LLMs (GPT, Claude, Gemini, and OpenAI o-series models). We measured performance using accuracy, recall, false-positive rate (FPR), and an aggregate score of ROUGE-1, BLEURT, and BERTScore for error correction. We also analyzed example outputs to identify failure modes and differences between LLM and clinician reasoning. Results: Zero-shot prompting showed low recall in both detection tasks, often missing abbreviation-heavy or atypical errors. SPR improved recall but increased FPR. Across all nine LLMs, RDP reduced FPR by about 15 percent, improved recall by 5 to 10 percent in error sentence detection, and generated more contextually accurate corrections. Conclusion: Across diverse LLMs, RDP outperforms zero-shot and SPR prompting. Using retrieved exemplars improves detection accuracy, reduces false positives, and enhances the reliability of medical error correction.
♻ ☆ Earth Observation Satellite Scheduling with Graph Neural Networks and Monte Ca rl o Tree Search
Antoine Jacquet, Guillaume Infantes, Emmanuel Benazera, Vincent Baudoui, Jonathan Guerra, Stéphanie Roussel
Earth Observation Satellite Planning (EOSP) is a difficult optimization problem with considerable practical interest. A set of requested observations must be scheduled on an agile Earth observation satellite while respecting constraints on their visibility window, as well as maneuver constraints that impose varying delays between successive observations. In addition, the problem is largely oversubscribed: there are much more candidate observations than can possibly be achieved. Therefore, one must select the set of observations that will be performed while maximizing their cumulative benefit and propose a feasible schedule for these observations. As previous work mostly focused on heuristic and iterative search algorithms, this paper presents a new technique for selecting and scheduling observations based on Graph Neural Networks (GNNs) and Deep Reinforcement Learning (DRL). GNNs are used to extract relevant information from the graphs representing instances of the EOSP, and DRL drives the search for optimal schedules. A post-learning search step based on Monte Carlo Tree Search (MCTS) is added that is able to find even better solutions. Experiments show that it is able to learn on small problem instances and generalize to larger real-world instances, with very competitive performance compared to traditional approaches.
comment : Accepted at International Workshop on Planning & Scheduling for Space (IWPSS 2025)
♻ ☆ LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Framework
Andrea Lops, Fedelucio Narducci, Azzurra Ragone, Michelantonio Trizio, Claudio Bartolini
Unit testing is an essential but resource-intensive step in software development, ensuring individual code units function correctly. This paper introduces AgoneTest, an automated evaluation framework for Large Language Model-generated (LLM) unit tests in Java. AgoneTest does not aim to propose a novel test generation algorithm; rather, it supports researchers and developers in comparing different LLMs and prompting strategies through a standardized end-to-end evaluation pipeline under realistic conditions. We introduce the Classes2Test dataset, which maps Java classes under test to their corresponding test classes, and a framework that integrates advanced evaluation metrics, such as mutation score and test smells, for a comprehensive assessment. Experimental results show that, for the subset of tests that compile, LLM-generated tests can match or exceed human-written tests in terms of coverage and defect detection. Our findings also demonstrate that enhanced prompting strategies contribute to test quality. AgoneTest clarifies the potential of LLMs in software testing and offers insights for future improvements in model design, prompt engineering, and testing practices.
comment : Accepted at 40th IEEE/ACM International Conference on Automated Software Engineering
♻ ☆ Passive Dementia Screening via Facial Temporal Micro-Dynamics Analysis of In-the-Wild Talking-Head Video
Filippo Cenacchi, Longbing Cao, Mitchell McEwan, Deborah Richards
We target passive dementia screening from short camera-facing talking head video, developing a facial temporal micro dynamics analysis for language free detection of early neuro cognitive change. This enables unscripted, in the wild video analysis at scale to capture natural facial behaviors, transferrable across devices, topics, and cultures without active intervention by clinicians or researchers during recording. Most existing resources prioritize speech or scripted interviews, limiting use outside clinics and coupling predictions to language and transcription. In contrast, we identify and analyze whether temporal facial kinematics, including blink dynamics, small mouth jaw motions, gaze variability, and subtle head adjustments, are sufficient for dementia screening without speech or text. By stabilizing facial signals, we convert these micro movements into interpretable facial microdynamic time series, smooth them, and summarize short windows into compact clip level statistics for screening. Each window is encoded by its activity mix (the relative share of motion across streams), thus the predictor analyzes the distribution of motion across streams rather than its magnitude, making per channel effects transparent. We also introduce YT DemTalk, a new dataset curated from publicly available, in the wild camera facing videos. It contains 300 clips (150 with self reported dementia, 150 controls) to test our model and offer a first benchmarking of the corpus. On YT DemTalk, ablations identify gaze lability and mouth/jaw dynamics as the most informative cues, and light weighted shallow classifiers could attain a dementia prediction performance of (AUROC) 0.953, 0.961 Average Precision (AP), 0.851 F1-score, and 0.857 accuracy.
♻ ☆ Mechanism of Task-oriented Information Removal in In- context Learning
Hakaze Cho, Haolin Yang, Gouki Minegishi, Naoya Inoue
In-context Learning (ICL) is an emerging few-shot learning paradigm based on modern Language Models (LMs), yet its inner mechanism remains unclear. In this paper, we investigate the mechanism through a novel perspective of information removal. Specifically, we demonstrate that in the zero-shot scenario, LMs encode queries into non-selective representations in hidden states containing information for all possible tasks, leading to arbitrary outputs without focusing on the intended task, resulting in near-zero accuracy. Meanwhile, we find that selectively removing specific information from hidden states by a low-rank filter effectively steers LMs toward the intended task. Building on these findings, by measuring the hidden states on carefully designed metrics, we observe that few-shot ICL effectively simulates such task-oriented information removal processes, selectively removing the redundant information from entangled non-selective representations, and improving the output based on the demonstrations, which constitutes a key mechanism underlying ICL. Moreover, we identify essential attention heads inducing the removal operation, termed Denoising Heads, which enables the ablation experiments blocking the information removal operation from the inference, where the ICL accuracy significantly degrades, especially when the correct label is absent from the few-shot demonstrations, confirming both the critical role of the information removal mechanism and denoising heads.
comment : 87 pages, 90 figures, 7 tables
♻ ☆ UITron-Speech: Towards Automated GUI Agent s Based on Speech Instructions
Wenkang Han, Zhixiong Zeng, Jing Huang, Shu Jiang, Liming Zheng, Longrong Yang, Haibo Qiu, Chang Yao, Jingyuan Chen, Lin Ma
Autonomous agents for Graphical User Interfaces (GUIs) are revolutionizing human-computer interaction, yet their reliance on text-based instructions imposes limitations on accessibility and convenience, particularly in hands-free scenarios. To address this issue, we propose replacing text with speech as the instruction input modality for GUI agents, and introduce UITron-Speech, which is the first end-to-end GUI agent capable of directly processing speech instructions and on-device screenshots to predict user actions. To tackle the problem of data scarcity, we synthesize high-quality speech instruction datasets using a random-speaker text-to-speech model. Additionally, we design a mixed-modality training strategy to mitigate the inherent modality imbalance in pre-trained foundation models. Furthermore, we conduct a statistical analysis of the distribution of GUI grounding prediction errors and propose a training-free two-step grounding refinement method to alleviate minor localization deviations. Extensive experiments on multiple benchmarks demonstrate that UITron-Speech achieves robust performance and superior adaptability, underscoring the feasibility and potential of speech-driven GUI agents for more accessible and intelligent human-computer interaction. Our code and datasets are available at https://github.com/UITron-hub/UITron-Speech.
♻ ☆ Empowering Time Series Forecasting with LLM- Agent s
Chin-Chia Michael Yeh, Vivian Lai, Uday Singh Saini, Xiran Fan, Yujie Fan, Junpeng Wang, Xin Dai, Yan Zheng
Large Language Model (LLM) powered agents have emerged as effective planners for Automated Machine Learning (AutoML) systems. While most existing AutoML approaches focus on automating feature engineering and model architecture search, recent studies in time series forecasting suggest that lightweight models can often achieve state-of-the-art performance. This observation led us to explore improving data quality, rather than model architecture, as a potentially fruitful direction for AutoML on time series data. We propose DCATS, a Data-Centric Agent for Time Series. DCATS leverages metadata accompanying time series to clean data while optimizing forecasting performance. We evaluated DCATS using four time series forecasting models on a large-scale traffic volume forecasting dataset. Results demonstrate that DCATS achieves an average 6% error reduction across all tested models and time horizons, highlighting the potential of data-centric approaches in AutoML for time series forecasting.
♻ ☆ TiCT: A Synthetically Pre-Train ed Foundation Model for Time Series Classification
Chin-Chia Michael Yeh, Uday Singh Saini, Junpeng Wang, Xin Dai, Xiran Fan, Jiarui Sun, Yujie Fan, Yan Zheng
The ubiquity of time series data creates a strong demand for general-purpose foundation models, yet developing them for classification remains a significant challenge, largely due to the high cost of labeled data. Foundation models capable of in-context learning (ICL) offer a powerful solution, adapting to new tasks with minimal examples and reducing the need for extensive retraining. However, prior work on large-scale time series models has predominantly focused on forecasting, leaving a critical gap for versatile, fine-tuning-free classification. To address this, we introduce TiCT (Time-series in-Context Transformer), a transformer-based model pre-trained exclusively on synthetic data to perform in-context classification. We make two primary technical contributions: 1) a novel architecture featuring a scalable bit-based label encoding and a special output attention mechanism to handle an arbitrary number of classes; and 2) a synthetic pre-training framework that combines a Mixup-inspired process with data augmentation to foster generalization and noise invariance. Extensive evaluations on the UCR Archive show that TiCT achieves competitive performance against state-of-the-art supervised methods. Crucially, this is accomplished using only in-context examples at inference time, without updating a single model weight.
♻ ☆ AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise NeurIPS 2025
Dhruv Agarwal, Bodhisattwa Prasad Majumder, Reece Adamson, Megha Chakravorty, Satvika Reddy Gavireddy, Aditya Parashar, Harshit Surana, Bhavana Dalvi Mishra, Andrew McCallum, Ashish Sabharwal, Peter Clark
The promise of autonomous scientific discovery (ASD) hinges not only on answering questions, but also on knowing which questions to ask. Most recent works in ASD explore the use of large language models (LLMs) in goal-driven settings, relying on human-specified research questions to guide hypothesis generation. However, scientific discovery may be accelerated further by allowing the AI system to drive exploration by its own criteria. The few existing approaches in open-ended ASD select hypotheses based on diversity heuristics or subjective proxies for human interestingness, but the former struggles to meaningfully navigate the typically vast hypothesis space, and the latter suffers from imprecise definitions. This paper presents AutoDiscovery -- a method for open-ended ASD that instead drives scientific exploration using Bayesian surprise. Here, we quantify the epistemic shift from the LLM's prior beliefs about a hypothesis to its posterior beliefs after gathering experimental results. To efficiently explore the space of nested hypotheses, our method employs a Monte Carlo tree search (MCTS) strategy with progressive widening using surprisal as the reward function. We evaluate AutoDiscovery in the setting of data-driven discovery across 21 real-world datasets spanning domains such as biology, economics, finance, and behavioral science. Our results demonstrate that under a fixed budget, AutoDiscovery substantially outperforms competitors by producing 5-29% more discoveries deemed surprising by the LLM. Our human evaluation further reveals that two-thirds of discoveries made by our system are surprising to domain experts as well, suggesting this is an important step towards building open-ended ASD systems.
comment : Accepted to NeurIPS 2025; https://neurips.cc/virtual/2025/loc/san-diego/poster/116398
♻ ☆ Meursault as a Data Point
Abhinav Pratap
In an era dominated by datafication, the reduction of human experiences to quantifiable metrics raises profound philosophical and ethical questions. This paper explores these issues through the lens of Meursault, the protagonist of Albert Camus' The Stranger, whose emotionally detached existence epitomizes the existential concept of absurdity. Using natural language processing (NLP) techniques including emotion detection (BERT), sentiment analysis (VADER), and named entity recognition (spaCy)-this study quantifies key events and behaviors in Meursault's life. Our analysis reveals the inherent limitations of applying algorithmic models to complex human experiences, particularly those rooted in existential alienation and moral ambiguity. By examining how modern AI tools misinterpret Meursault's actions and emotions, this research underscores the broader ethical dilemmas of reducing nuanced human narratives to data points, challenging the foundational assumptions of our data-driven society. The findings presented in this paper serve as a critique of the increasing reliance on data-driven narratives and advocate for incorporating humanistic values in artificial intelligence.
comment : 7 pages, 9 figures, 4 tables
♻ ☆ Where to Start Alignment? Diffusion Large Language Model May Demand a Distinct Position AAAI 2026
Zhixin Xie, Xurui Song, Jun Luo
Diffusion Large Language Models (dLLMs) have recently emerged as a competitive non-autoregressive paradigm due to their unique training and inference approach. However, there is currently a lack of safety study on this novel architecture. In this paper, we present the first analysis of dLLMs' safety performance and propose a novel safety alignment method tailored to their unique generation characteristics. Specifically, we identify a critical asymmetry between the defender and attacker in terms of security. For the defender, we reveal that the middle tokens of the response, rather than the initial ones, are more critical to the overall safety of dLLM outputs; this seems to suggest that aligning middle tokens can be more beneficial to the defender. The attacker, on the contrary, may have limited power to manipulate middle tokens, as we find dLLMs have a strong tendency towards a sequential generation order in practice, forcing the attack to meet this distribution and diverting it from influencing the critical middle tokens. Building on this asymmetry, we introduce Middle-tOken Safety Alignment (MOSA), a novel method that directly aligns the model's middle generation with safe refusals exploiting reinforcement learning. We implement MOSA and compare its security performance against eight attack methods on two benchmarks. We also test the utility of MOSA-aligned dLLM on coding, math, and general reasoning. The results strongly prove the superiority of MOSA.
comment : Accepted for oral presentation at AAAI 2026
♻ ☆ Uncovering Implicit Bias in Large Language Models with Concept Learning Dataset
Leroy Z. Wang
We introduce a dataset of concept learning tasks that helps uncover implicit biases in large language models. Using in-context concept learning experiments, we found that language models may have a bias toward upward monotonicity in quantifiers; such bias is less apparent when the model is tested by direct prompting without concept learning components. This demonstrates that in-context concept learning can be an effective way to discover hidden biases in language models.
comment : Presented at EurIPS 2025 Workshop - Unifying Perspectives on Learning Biases (UPLB) https://sites.google.com/view/towards-a-unified-view
♻ ☆ Mig GPT : Harnessing Large Language Models for Automated Migration of Out-of-Tree Linux Kernel Patches Across Versions
Pucheng Dang, Di Huang, Dong Li, Kang Chen, Yuanbo Wen, Qi Guo, Xing Hu
Out-of-tree kernel patches are essential for adapting the Linux kernel to new hardware or enabling specific functionalities. Maintaining and updating these patches across different kernel versions demands significant effort from experienced engineers. Large language models (LLMs) have shown remarkable progress across various domains, suggesting their potential for automating out-of-tree kernel patch migration. However, our findings reveal that LLMs, while promising, struggle with incomplete code context understanding and inaccurate migration point identification. In this work, we propose MigGPT, a framework that employs a novel code fingerprint structure to retain code snippet information and incorporates three meticulously designed modules to improve the migration accuracy and efficiency of out-of-tree kernel patches. Furthermore, we establish a robust benchmark using real-world out-of-tree kernel patch projects to evaluate LLM capabilities. Evaluations show that MigGPT significantly outperforms the direct application of vanilla LLMs, achieving an average completion rate of 74.07 for migration tasks.
♻ ☆ ConceptGuard: Proactive Safety in Text-and-Image-to-Video Generation through Multimodal Risk Detection
Ruize Ma, Minghong Cai, Yilei Jiang, Jiaming Han, Yi Feng, Yingshui Tan, Xiaoyong Zhu, Bo Zhang, Bo Zheng, Xiangyu Yue
Recent progress in video generative models has enabled the creation of high-quality videos from multimodal prompts that combine text and images. While these systems offer enhanced controllability, they also introduce new safety risks, as harmful content can emerge from individual modalities or their interaction. Existing safety methods are often text-only, require prior knowledge of the risk category, or operate as post-generation auditors, struggling to proactively mitigate such compositional, multimodal risks. To address this challenge, we present ConceptGuard, a unified safeguard framework for proactively detecting and mitigating unsafe semantics in multimodal video generation. ConceptGuard operates in two stages: First, a contrastive detection module identifies latent safety risks by projecting fused image-text inputs into a structured concept space; Second, a semantic suppression mechanism steers the generative process away from unsafe concepts by intervening in the prompt's multimodal conditioning. To support the development and rigorous evaluation of this framework, we introduce two novel benchmarks: ConceptRisk, a large-scale dataset for training on multimodal risks, and T2VSafetyBench-TI2V, the first benchmark adapted from T2VSafetyBench for the Text-and-Image-to-Video (TI2V) safety setting. Comprehensive experiments on both benchmarks show that ConceptGuard consistently outperforms existing baselines, achieving state-of-the-art results in both risk detection and safe video generation. Our code is available at https://github.com/Ruize-Ma/ConceptGuard.
♻ ☆ Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applications
Vaishali Vinay
Large language models (LLMs) are being rapidly integrated into decision-support tools, automation workflows, and AI-enabled software systems. However, their behavior in production environments remains poorly understood, and their failure patterns differ fundamentally from those of traditional machine learning models. This paper presents a system-level taxonomy of fifteen hidden failure modes that arise in real-world LLM applications, including multi-step reasoning drift, latent inconsistency, context-boundary degradation, incorrect tool invocation, version drift, and cost-driven performance collapse. Using this taxonomy, we analyze the growing gap in evaluation and monitoring practices: existing benchmarks measure knowledge or reasoning but provide little insight into stability, reproducibility, drift, or workflow integration. We further examine the production challenges associated with deploying LLMs - including observability limitations, cost constraints, and update-induced regressions - and outline high-level design principles for building reliable, maintainable, and cost-aware LLM systems. Finally, we outline high-level design principles for building reliable, maintainable, and cost-aware LLM-based systems. By framing LLM reliability as a system-engineering problem rather than a purely model-centric one, this work provides an analytical foundation for future research on evaluation methodology, AI system robustness, and dependable LLM deployment.
♻ ☆ Cheating Stereo Matching in Full-scale: Physical Adversarial Attack against Binocular Depth Estimation in Autonomous Driving AAAI 2026
Kangqiao Zhao, Shuo Huai, Xurui Song, Jun Luo
Though deep neural models adopted to realize the perception of autonomous driving have proven vulnerable to adversarial examples, known attacks often leverage 2D patches and target mostly monocular perception. Therefore, the effectiveness of Physical Adversarial Examples (PAEs) on stereo-based binocular depth estimation remains largely unexplored. To this end, we propose the first texture-enabled physical adversarial attack against stereo matching models in the context of autonomous driving. Our method employs a 3D PAE with global camouflage texture rather than a local 2D patch-based one, ensuring both visual consistency and attack effectiveness across different viewpoints of stereo cameras. To cope with the disparity effect of these cameras, we also propose a new 3D stereo matching rendering module that allows the PAE to be aligned with real-world positions and headings in binocular vision. We further propose a novel merging attack that seamlessly blends the target into the environment through fine-grained PAE optimization. It has significantly enhanced stealth and lethality upon existing hiding attacks that fail to get seamlessly merged into the background. Extensive evaluations show that our PAEs can successfully fool the stereo models into producing erroneous depth information.
comment : AAAI 2026
♻ ☆ Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmark s
Jie Li, Hongyi Cai, Mingkang Dong, Muxin Pu, Shan You, Fei Wang, Tao Huang
Automatically detecting abnormal events in videos is crucial for modern autonomous systems, yet existing Video Anomaly Detection (VAD) benchmarks lack the scene diversity, balanced anomaly coverage, and temporal complexity needed to reliably assess real-world performance. Meanwhile, the community is increasingly moving toward Video Anomaly Understanding (VAU), which requires deeper semantic and causal reasoning but remains difficult to benchmark due to the heavy manual annotation effort it demands. In this paper, we introduce Pistachio, a new VAD/VAU benchmark constructed entirely through a controlled, generation-based pipeline. By leveraging recent advances in video generation models, Pistachio provides precise control over scenes, anomaly types, and temporal narratives, effectively eliminating the biases and limitations of Internet-collected datasets. Our pipeline integrates scene-conditioned anomaly assignment, multi-step storyline generation, and a temporally consistent long-form synthesis strategy that produces coherent 41-second videos with minimal human intervention. Extensive experiments demonstrate the scale, diversity, and complexity of Pistachio, revealing new challenges for existing methods and motivating future research on dynamic and multi-event anomaly understanding.
♻ ☆ Multi-PA: A Multi-perspective Benchmark on Privacy Assessment for Large Vision -Language Models
Jie Zhang, Xiangkui Cao, Zhouyu Han, Shiguang Shan, Xilin Chen
Large Vision-Language Models (LVLMs) exhibit impressive potential across various tasks but also face significant privacy risks, limiting their practical applications. Current researches on privacy assessment for LVLMs is limited in scope, with gaps in both assessment dimensions and privacy categories. To bridge this gap, we propose Multi-PA, a comprehensive benchmark for evaluating the privacy preservation capabilities of LVLMs in terms of privacy awareness and leakage. Privacy awareness measures the model's ability to recognize the privacy sensitivity of input data, while privacy leakage assesses the risk of the model unintentionally disclosing privacy information in its output. We design a range of sub-tasks to thoroughly evaluate the model's privacy protection offered by LVLMs. Multi-PA covers 26 categories of personal privacy, 15 categories of trade secrets, and 18 categories of state secrets, totaling 31,962 samples. Based on Multi-PA, we evaluate the privacy preservation capabilities of 21 open-source and 2 closed-source LVLMs. Our results reveal that current LVLMs generally pose a high risk of facilitating privacy breaches, with vulnerabilities varying across personal privacy, trade secret, and state secret.
♻ ☆ PointNSP: Autoregressive 3D Point Cloud Generation with Next-Scale Level-of-Detail Prediction
Ziqiao Meng, Qichao Wang, Zhiyang Dou, Zixing Song, Zhipeng Zhou, Irwin King, Peilin Zhao
Autoregressive point cloud generation has long lagged behind diffusion-based approaches in quality. The performance gap stems from the fact that autoregressive models impose an artificial ordering on inherently unordered point sets, forcing shape generation to proceed as a sequence of local predictions. This sequential bias emphasizes short-range continuity but undermines the model's capacity to capture long-range dependencies, hindering its ability to enforce global structural properties such as symmetry, consistent topology, and large-scale geometric regularities. Inspired by the level-of-detail (LOD) principle in shape modeling, we propose PointNSP, a coarse-to-fine generative framework that preserves global shape structure at low resolutions and progressively refines fine-grained geometry at higher scales through a next-scale prediction paradigm. This multi-scale factorization aligns the autoregressive objective with the permutation-invariant nature of point sets, enabling rich intra-scale interactions while avoiding brittle fixed orderings. Experiments on ShapeNet show that PointNSP establishes state-of-the-art (SOTA) generation quality for the first time within the autoregressive paradigm. In addition, it surpasses strong diffusion-based baselines in parameter, training, and inference efficiency. Finally, in dense generation with 8,192 points, PointNSP's advantages become even more pronounced, underscoring its scalability potential.
comment : This work was intended as a replacement of arXiv:2503.08594 and any subsequent updates will appear there
♻ ☆ Beyond Introspection: Reinforcing Thinking via Externalist Behavioral Feedback
Diji Yang, Linda Zeng, Kezhen Chen, Yi Zhang
While inference-time thinking allows Large Language Models (LLMs) to address complex problems, the extended thinking process can be unreliable or inconsistent because of the model's probabilistic nature, especially near its knowledge boundaries. Existing approaches attempt to mitigate this by having the model critique its own reasoning to make corrections. However, such self-critique inherits the same biases of the original output, known as the introspection illusion. Moving beyond such introspection and inspired by core methodologies in ethology, we propose an externalist three-step framework Distillation-Reinforcement-Reasoning (DRR). Rather than relying on a model's introspection, DRR evaluates its observable behaviors to provide corrective feedback. DRR first distills the reasoner's behavioral traces, then trains a lightweight, external Discriminative Model (DM). At inference time, this DM acts as a critic, identifying and rejecting suspicious reasoning steps. This external feedback compels the LLM to discard flawed pathways and explore alternatives, thereby enhancing reasoning quality without altering the base model. Experiments on multiple reasoning benchmarks show that our framework significantly outperforms prominent self-critique methods. Benefiting from a lightweight and annotation-free design, DRR offers a scalable and adaptable solution for improving the reliability of reasoning in a wide range of LLMs.
♻ ☆ CoMind: Towards Community-Driven Agent s for Machine Learning Engineering
Sijie Li, Weiwei Sun, Shanda Li, Ameet Talwalkar, Yiming Yang
Large language model (LLM) agents show promise in automating machine learning (ML) engineering. However, existing agents typically operate in isolation on a given research problem, without engaging with the broader research community, where human researchers often gain insights and contribute by sharing knowledge. To bridge this gap, we introduce MLE-Live, a live evaluation framework designed to assess an agent's ability to communicate with and leverage collective knowledge from a simulated Kaggle research community. Building on this framework, we propose CoMind, an multi-agent system designed to actively integrate external knowledge. CoMind employs an iterative parallel exploration mechanism, developing multiple solutions simultaneously to balance exploratory breadth with implementation depth. On 75 past Kaggle competitions within our MLE-Live framework, CoMind achieves a 36% medal rate, establishing a new state of the art. Critically, when deployed in eight live, ongoing competitions, CoMind outperforms 92.6% of human competitors on average, placing in the top 5% on three official leaderboards and the top 1% on one.
♻ ☆ Agent 0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision -Language Reasoning
Jiaqi Liu, Kaiwen Xiong, Peng Xia, Yiyang Zhou, Haonian Ji, Lu Feng, Siwei Han, Mingyu Ding, Huaxiu Yao
Vision-language agents have achieved remarkable progress in a variety of multimodal reasoning tasks; however, their learning remains constrained by the limitations of human-annotated supervision. Recent self-rewarding approaches attempt to overcome this constraint by allowing models to act as their own critics or reward providers. Yet, purely text-based self-evaluation struggles to verify complex visual reasoning steps and often suffers from evaluation hallucinations. To address these challenges, inspired by recent advances in tool-integrated reasoning, we propose Agent0-VL, a self-evolving vision-language agent that achieves continual improvement with tool-integrated reasoning. Agent0-VL incorporates tool usage not only into reasoning but also into self-evaluation and self-repair, enabling the model to introspect, verify, and refine its reasoning through evidence-grounded analysis. It unifies two synergistic roles within a single LVLM: a Solver that performs multi-turn tool-integrated reasoning, and a Verifier that generates structured feedback and fine-grained self-rewards through tool-grounded critique. These roles interact through a Self-Evolving Reasoning Cycle, where tool-based verification and reinforcement learning jointly align the reasoning and evaluation distributions for stable self-improvement. Through this zero-external-reward evolution, Agent0-VL aligns its reasoning and verification behaviors without any human annotation or external reward models, achieving continual self-improvement. Experiments on geometric problem solving and visual scientific analysis show that Agent0-VL achieves an 12.5% improvement over the base model. Our code is available at https://github.com/aiming-lab/Agent0.
♻ ☆ UniGame: Turning a Unified Multimodal Model Into Its Own Adversary
Zhaolong Su, Wang Lu, Hao Chen, Sharon Li, Jindong Wang
Unified Multimodal Models (UMMs) have shown impressive performance in both understanding and generation with a single architecture. However, UMMs still exhibit a fundamental inconsistency: understanding favors compact embeddings, whereas generation favors reconstruction-rich representations. This structural trade-off produces misaligned decision boundaries, degraded cross-modal coherence, and heightened vulnerability under distributional and adversarial shifts. In this paper, we present UniGame, a self-adversarial post-training framework that directly targets the inconsistencies. By applying a lightweight perturber at the shared token interface, UniGame enables the generation branch to actively seek and challenge fragile understanding, turning the model itself into its own adversary. Experiments demonstrate that UniGame significantly improves the consistency (+4.6%). Moreover, it also achieves substantial improvements in understanding (+3.6%), generation (+0.02), out-of-distribution and adversarial robustness (+4.8% and +6.2% on NaturalBench and AdVQA). The framework is architecture-agnostic, introduces less than 1% additional parameters, and is complementary to existing post-training methods. These results position adversarial self-play as a general and effective principle for enhancing the coherence, stability, and unified competence of future multimodal foundation models. The official code is available at: https://github.com/AIFrontierLab/UniGame
♻ ☆ Consistent Opponent Modeling of Static Opponents in Imperfect- Information Games
Sam Ganzfried
The goal of agents in multi-agent environments is to maximize total reward against the opposing agents that are encountered. Following a game-theoretic solution concept, such as Nash equilibrium, may obtain a strong performance in some settings; however, such approaches fail to capitalize on historical and observed data from repeated interactions against our opponents. Opponent modeling algorithms integrate machine learning techniques to exploit suboptimal opponents utilizing available data; however, the effectiveness of such approaches in imperfect-information games to date is quite limited. We show that existing opponent modeling approaches fail to satisfy a simple desirable property even against static opponents drawn from a known prior distribution; namely, they do not guarantee that the model approaches the opponent's true strategy even in the limit as the number of game iterations approaches infinity. We develop a new algorithm that is able to achieve this property and runs efficiently by solving a convex minimization problem based on the sequence-form game representation using projected gradient descent. The algorithm is guaranteed to efficiently converge to the opponent's true strategy given observations from gameplay and possibly additional historical data if it is available.
♻ ☆ CoxKAN: Kolmogorov-Arnold Networks for Interpretable, High-Performance Survival Analysis
William Knottenbelt, William McGough, Rebecca Wray, Woody Zhidong Zhang, Jiashuai Liu, Ines Prata Machado, Zeyu Gao, Mireia Crispin-Ortuzar
Motivation: Survival analysis is a branch of statistics that is crucial in medicine for modeling the time to critical events such as death or relapse, in order to improve treatment strategies and patient outcomes. Selecting survival models often involves a trade-off between performance and interpretability; deep learning models offer high performance but lack the transparency of more traditional approaches. This poses a significant issue in medicine, where practitioners are reluctant to use black-box models for critical patient decisions. Results: We introduce CoxKAN, a Cox proportional hazards Kolmogorov-Arnold Network for interpretable, high-performance survival analysis. Kolmogorov-Arnold Networks (KANs) were recently proposed as an interpretable and accurate alternative to multi-layer perceptrons. We evaluated CoxKAN on four synthetic and nine real datasets, including five cohorts with clinical data and four with genomics biomarkers. In synthetic experiments, CoxKAN accurately recovered interpretable hazard function formulae and excelled in automatic feature selection. Evaluations on real datasets showed that CoxKAN consistently outperformed the traditional Cox proportional hazards model (by up to 4% in C-index) and matched or surpassed the performance of deep learning-based models. Importantly, CoxKAN revealed complex interactions between predictor variables and uncovered symbolic formulae, which are key capabilities that other survival analysis methods lack, to provide clear insights into the impact of key biomarkers on patient risk. Availability and implementation: CoxKAN is available at GitHub and Zenodo
♻ ☆ Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor NeurIPS'25
Alexandra Olteanu, Su Lin Blodgett, Agathe Balayn, Angelina Wang, Fernando Diaz, Flavio du Pin Calmon, Margaret Mitchell, Michael Ekstrand, Reuben Binns, Solon Barocas
In AI research and practice, rigor remains largely understood in terms of methodological rigor -- such as whether mathematical, statistical, or computational methods are correctly applied. We argue that this narrow conception of rigor has contributed to the concerns raised by the responsible AI community, including overblown claims about the capabilities of AI systems. Our position is that a broader conception of what rigorous AI research and practice should entail is needed. We believe such a conception -- in addition to a more expansive understanding of (1) methodological rigor -- should include aspects related to (2) what background knowledge informs what to work on (epistemic rigor); (3) how disciplinary, community, or personal norms, standards, or beliefs influence the work (normative rigor); (4) how clearly articulated the theoretical constructs under use are (conceptual rigor); (5) what is reported and how (reporting rigor); and (6) how well-supported the inferences from existing evidence are (interpretative rigor). In doing so, we also provide useful language and a framework for much-needed dialogue about the AI community's work by researchers, policymakers, journalists, and other stakeholders.
comment : 21 pages, 1 figure, 1 table, accepted at NeurIPS'25 position papers track
♻ ☆ CroMe: Multimodal Fake News Detection using Cross-Modal Tri- Transformer and Metric Learning
Eunjee Choi, Junhyun Ahn, XinYu Piao, Jong-Kook Kim
Multimodal Fake News Detection has received increasing attention recently. Existing methods rely on independently encoded unimodal data and overlook the advantages of capturing intra-modality relationships and integrating inter-modal similarities using advanced techniques. To address these issues, Cross-Modal Tri-Transformer and Metric Learning for Multimodal Fake News Detection (CroMe) is proposed. CroMe utilizes Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (BLIP2) as encoders to capture detailed text, image and combined image-text representations. The metric learning module employs a proxy anchor method to capture intra-modality relationships while the feature fusion module uses a Cross-Modal and Tri-Transformer for effective integration. The final fake news detector processes the fused features through a classifier to predict the authenticity of the content. Experiments on datasets show that CroMe excels in multimodal fake news detection.
♻ ☆ R3A: Reliable RTL Repair Framework with Multi- Agent Fault Localization and Stochastic Tree-of-Thoughts Patch Generation
Zizhang Luo, Fan Cui, Kexing Zhou, Runlin Guo, Mile Xia, Hongyuan Hou, Yun Liang
Repairing RTL bugs is crucial for hardware design and verification. Traditional automatic program repair (APR) methods define dedicated search spaces to locate and fix bugs with program synthesis. However, they heavily rely on fixed templates and can only deal with limited bugs. As an alternative, Large Language Models with the ability to understand code semantics can be explored for RTL repair. However, they suffer from unreliable outcomes due to inherent randomness and long input contexts of RTL code and waveform. To address these challenges, we propose R3A, an LLM-based automatic RTL program repair framework upon the basic model to improve reliability. R3A proposes the stochastic Tree-Of-Thoughts method to control a patch generation agent to explore a validated solution for the bug. The algorithm samples search states according to a heuristic function to balance between exploration and exploitation for a reliable outcome. Besides, R3A proposes a multi-agent fault localization method to find fault candidates as the starting points for the patch generation agent, further increasing the reliability. Experiments show R3A can fix 90.6% of bugs in the RTL-repair dataset within a given time limit, which covers 45% more bugs than traditional methods and other LLM-based approaches, while achieving an 86.7% pass@5 rate on average, showing a high reliability.
♻ ☆ Data-Driven Lipschitz Continuity: A Cost-Effective Approach to Improve Adversarial Robustness
Erh-Chung Chen, Pin-Yu Chen, I-Hsin Chung, Che-Rung Lee
As deep neural networks (DNNs) are increasingly deployed in sensitive applications, ensuring their security and robustness has become critical. A major threat to DNNs arises from adversarial attacks, where small input perturbations can lead to incorrect predictions. Recent advances in adversarial training improve robustness by incorporating additional examples from external datasets or generative models. However, these methods often incur high computational costs, limiting their practicality and hindering real-world deployment. In this paper, we propose a cost-efficient alternative based on Lipschitz continuity that achieves robustness comparable to models trained with extensive supplementary data. Unlike conventional adversarial training, our method requires only a single pass over the dataset without gradient estimation, making it highly efficient. Furthermore, our method can integrate seamlessly with existing adversarial training frameworks and enhances the robustness of models without requiring extra generative data. Experimental results show that our approach not only reduces computational overhead but also maintains or improves the defensive capabilities of robust neural networks. This work opens a promising direction for developing practical, scalable defenses against adversarial attacks.
♻ ☆ IVY-FAKE: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detection
Changjiang Jiang, Wenhui Dong, Zhonghao Zhang, Chenyang Si, Fengchang Yu, Wei Peng, Xinbin Yuan, Yifei Bi, Ming Zhao, Zian Zhou, Caifeng Shan
The rapid development of Artificial Intelligence Generated Content (AIGC) techniques has enabled the creation of high-quality synthetic content, but it also raises significant security concerns. Current detection methods face two major limitations: (1) the lack of multidimensional explainable datasets for generated images and videos. Existing open-source datasets (e.g., WildFake, GenVideo) rely on oversimplified binary annotations, which restrict the explainability and trustworthiness of trained detectors. (2) Prior MLLM-based forgery detectors (e.g., FakeVLM) exhibit insufficiently fine-grained interpretability in their step-by-step reasoning, which hinders reliable localization and explanation. To address these challenges, we introduce Ivy-Fake, the first large-scale multimodal benchmark for explainable AIGC detection. It consists of over 106K richly annotated training samples (images and videos) and 5,000 manually verified evaluation examples, sourced from multiple generative models and real world datasets through a carefully designed pipeline to ensure both diversity and quality. Furthermore, we propose Ivy-xDetector, a reinforcement learning model based on Group Relative Policy Optimization (GRPO), capable of producing explainable reasoning chains and achieving robust performance across multiple synthetic content detection benchmarks. Extensive experiments demonstrate the superiority of our dataset and confirm the effectiveness of our approach. Notably, our method improves performance on GenImage from 86.88% to 96.32%, surpassing prior state-of-the-art methods by a clear margin.
comment : 30 pages
♻ ☆ EmoFeedback$^2$: Reinforcement of Continuous Emotional Image Generation via LVLM-based Reward and Textual Feedback
Jingyang Jia, Kai Shu, Gang Yang, Long Xing, Xun Chen, Aiping Liu
Continuous emotional image generation (C-EICG) is emerging rapidly due to its ability to produce images aligned with both user descriptions and continuous emotional values. However, existing approaches lack emotional feedback from generated images, limiting the control of emotional continuity. Additionally, their simple alignment between emotions and naively generated texts fails to adaptively adjust emotional prompts according to image content, leading to insufficient emotional fidelity. To address these concerns, we propose a novel generation-understanding-feedback reinforcement paradigm (EmoFeedback$^2$) for C-EICG, which exploits the reasoning capability of the fine-tuned large vision-language model (LVLM) to provide reward and textual feedback for generating high-quality images with continuous emotions. Specifically, we introduce an emotion-aware reward feedback strategy, where the LVLM evaluates the emotional values of generated images and computes the reward against target emotions, guiding the reinforcement fine-tuning of the generative model and enhancing the emotional continuity of images. Furthermore, we design a self-promotion textual feedback framework, in which the LVLM iteratively analyzes the emotional content of generated images and adaptively produces refinement suggestions for the next-round prompt, improving the emotional fidelity with fine-grained content. Extensive experimental results demonstrate that our approach effectively generates high-quality images with the desired emotions, outperforming existing state-of-the-art methods in our custom dataset. The code and dataset will be released soon.
♻ ☆ Uncertainty-Aware Deep Learning Framework for Remaining Useful Life Prediction in Turbofan Engines with Learned Aleatoric Uncertainty
Krishang Sharma
Accurate Remaining Useful Life (RUL) prediction coupled with uncertainty quantification remains a critical challenge in aerospace prognostics. This research introduces a novel uncertainty-aware deep learning framework that learns aleatoric uncertainty directly through probabilistic modeling, an approach unexplored in existing CMAPSS-based literature. Our hierarchical architecture integrates multi-scale Inception blocks for temporal pattern extraction, bidirectional Long Short-Term Memory networks for sequential modeling, and a dual-level attention mechanism operating simultaneously on sensor and temporal dimensions. The innovation lies in the Bayesian output layer that predicts both mean RUL and variance, enabling the model to learn data-inherent uncertainty. Comprehensive preprocessing employs condition-aware clustering, wavelet denoising, and intelligent feature selection. Experimental validation on NASA CMAPSS benchmarks (FD001-FD004) demonstrates competitive overall performance with RMSE values of 16.22, 19.29, 16.84, and 19.98 respectively. Remarkably, our framework achieves breakthrough critical zone performance (RUL <= 30 cycles) with RMSE of 5.14, 6.89, 5.27, and 7.16, representing 25-40 percent improvements over conventional approaches and establishing new benchmarks for safety-critical predictions. The learned uncertainty provides well-calibrated 95 percent confidence intervals with coverage ranging from 93.5 percent to 95.2 percent, enabling risk-aware maintenance scheduling previously unattainable in CMAPSS literature.
comment : 10 pages, 2 figures, 3 tables
♻ ☆ Dual-Balancing for Multi-Task Learning
Baijiong Lin, Weisen Jiang, Feiyang Ye, Yu Zhang, Pengguang Chen, Ying-Cong Chen, Shu Liu, Ivor W. Tsang, James T. Kwok
Multi-task learning aims to learn multiple related tasks simultaneously and has achieved great success in various fields. However, the disparity in loss and gradient scales among tasks often leads to performance compromises, and the balancing of tasks remains a significant challenge. In this paper, we propose Dual-Balancing Multi-Task Learning (DB-MTL) to achieve task balancing from both the loss and gradient perspectives. Specifically, DB-MTL achieves loss-scale balancing by performing logarithm transformation on each task loss, and rescales gradient magnitudes by normalizing all task gradients to comparable magnitudes using the maximum gradient norm. Extensive experiments on a number of benchmark datasets demonstrate that DB-MTL consistently performs better than the current state-of-the-art.
comment : Accepted by Neural Networks
♻ ☆ Heterogeneous Multi- Agent Proximal Policy Optimization for Power Distribution System Restoration
Parya Dolatyabi, Ali Farajzadeh Bavil, Mahdi Khodayar
Restoring power distribution systems (PDS) after large-scale outages requires sequential switching operations that reconfigure feeder topology and coordinate distributed energy resources (DERs) under nonlinear constraints such as power balance, voltage limits, and thermal ratings. These challenges make conventional optimization and value-based RL approaches computationally inefficient and difficult to scale. This paper applies a Heterogeneous-Agent Reinforcement Learning (HARL) framework, instantiated through Heterogeneous-Agent Proximal Policy Optimization (HAPPO), to enable coordinated restoration across interconnected microgrids. Each agent controls a distinct microgrid with different loads, DER capacities, and switch counts, introducing practical structural heterogeneity. Decentralized actor policies are trained with a centralized critic to compute advantage values for stable on-policy updates. A physics-informed OpenDSS environment provides full power flow feedback and enforces operational limits via differentiable penalty signals rather than invalid action masking. The total DER generation is capped at 2400 kW, and each microgrid must satisfy local supply-demand feasibility. Experiments on the IEEE 123-bus and IEEE 8500-node systems show that HAPPO achieves faster convergence, higher restored power, and smoother multi-seed training than DQN, PPO, MAES, MAGDPG, MADQN, Mean-Field RL, and QMIX. Results demonstrate that incorporating microgrid-level heterogeneity within the HARL framework yields a scalable, stable, and constraint-aware solution for complex PDS restoration.
comment : 6 pages, 4 figures, TPEC 2025 Conference
♻ ☆ Automated Neural Architecture Design for Industrial Defect Detection
Yuxi Liu, Yunfeng Ma, Yi Tang, Min Liu, Shuai Jiang, Yaonan Wang
Industrial surface defect detection (SDD) is critical for ensuring product quality and manufacturing reliability. Due to the diverse shapes and sizes of surface defects, SDD faces two main challenges: intraclass difference and interclass similarity. Existing methods primarily utilize manually designed models, which require extensive trial and error and often struggle to address both challenges effectively. To overcome this, we propose AutoNAD, an automated neural architecture design framework for SDD that jointly searches over convolutions, transformers, and multi-layer perceptrons. This hybrid design enables the model to capture both fine-grained local variations and long-range semantic context, addressing the two key challenges while reducing the cost of manual network design. To support efficient training of such a diverse search space, AutoNAD introduces a cross weight sharing strategy, which accelerates supernet convergence and improves subnet performance. Additionally, a searchable multi-level feature aggregation module (MFAM) is integrated to enhance multi-scale feature learning. Beyond detection accuracy, runtime efficiency is essential for industrial deployment. To this end, AutoNAD incorporates a latency-aware prior to guide the selection of efficient architectures. The effectiveness of AutoNAD is validated on three industrial defect datasets and further applied within a defect imaging and detection platform. Code is available at https://github.com/Yuxi104/AutoNAD.
♻ ☆ Contrast-Prior Enhanced Duality for Mask-Free Shadow Removal
Jiyu Wu, Yifan Liu, Jiancheng Huang, Mingfu Yan, Shifeng Chen
Existing shadow removal methods often rely on shadow masks, which are challenging to acquire in real-world scenarios. Exploring intrinsic image cues, such as local contrast information, presents a potential alternative for guiding shadow removal in the absence of explicit masks. However, the cue's inherent ambiguity becomes a critical limitation in complex scenes, where it can fail to distinguish true shadows from low-reflectance objects and intricate background textures. To address this motivation, we propose the Adaptive Gated Dual-Branch Attention (AGBA) mechanism. AGBA dynamically filters and re-weighs the contrast prior to effectively disentangle shadow features from confounding visual elements. Furthermore, to tackle the persistent challenge of restoring soft shadow boundaries and fine-grained details, we introduce a diffusion-based Frequency-Contrast Fusion Network (FCFN) that leverages high-frequency and contrast cues to guide the generative process. Extensive experiments demonstrate that our method achieves state-of-the-art results among mask-free approaches while maintaining competitive performance relative to mask-based methods.
comment : There are unresolved authorship disputes related to this submission, and the current version does not reflect an agreed authorship list
♻ ☆ FRAGMENTA: End-to-end Fragmentation-based Generative Model with Agent ic Tuning for Drug Lead Optimization
Yuto Suzuki, Paul Awolade, Daniel V. LaBarbera, Farnoush Banaei-Kashani
Molecule generation using generative AI is vital for drug discovery, yet class-specific datasets often contain fewer than 100 training examples. While fragment-based models handle limited data better than atom-based approaches, existing heuristic fragmentation limits diversity and misses key fragments. Additionally, model tuning typically requires slow, indirect collaboration between medicinal chemists and AI engineers. We introduce FRAGMENTA, an end-to-end framework for drug lead optimization comprising: 1) a novel generative model that reframes fragmentation as a "vocabulary selection" problem, using dynamic Q-learning to jointly optimize fragmentation and generation; and 2) an agentic AI system that refines objectives via conversational feedback from domain experts. This system removes the AI engineer from the loop and progressively learns domain knowledge to eventually automate tuning. In real-world cancer drug discovery experiments, FRAGMENTA's Human-Agent configuration identified nearly twice as many high-scoring molecules as baselines. Furthermore, the fully autonomous Agent-Agent system outperformed traditional Human-Human tuning, demonstrating the efficacy of agentic tuning in capturing expert intent.
♻ ☆ SOAP: Enhancing Spatio-Temporal Relation and Motion Information Capturing for Few-Shot Action Recognition ACM MM 2024
Wenbo Huang, Jinghui Zhang, Xuwei Qian, Zhen Wu, Meng Wang, Lei Zhang
High frame-rate (HFR) videos of action recognition improve fine-grained expression while reducing the spatio-temporal relation and motion information density. Thus, large amounts of video samples are continuously required for traditional data-driven training. However, samples are not always sufficient in real-world scenarios, promoting few-shot action recognition (FSAR) research. We observe that most recent FSAR works build spatio-temporal relation of video samples via temporal alignment after spatial feature extraction, cutting apart spatial and temporal features within samples. They also capture motion information via narrow perspectives between adjacent frames without considering density, leading to insufficient motion information capturing. Therefore, we propose a novel plug-and-play architecture for FSAR called Spatio-tempOral frAme tuPle enhancer (SOAP) in this paper. The model we designed with such architecture refers to SOAP-Net. Temporal connections between different feature channels and spatio-temporal relation of features are considered instead of simple feature extraction. Comprehensive motion information is also captured, using frame tuples with multiple frames containing more motion information than adjacent frames. Combining frame tuples of diverse frame counts further provides a broader perspective. SOAP-Net achieves new state-of-the-art performance across well-known benchmarks such as SthSthV2, Kinetics, UCF101, and HMDB51. Extensive empirical evaluations underscore the competitiveness, pluggability, generalization, and robustness of SOAP. The code is released at https://github.com/wenbohuang1002/SOAP.
comment : Accepted by ACM MM 2024
♻ ☆ Universe of Thoughts: Enabling Creative Reasoning with Large Language Models
Yuto Suzuki, Farnoush Banaei-Kashani
Reasoning based on Large Language Models (LLMs) has garnered increasing attention due to outstanding performance of these models in mathematical and complex logical tasks. Beginning with the Chain-of-Thought (CoT) prompting technique, numerous reasoning methods have emerged that decompose problems into smaller, sequential steps (or thoughts). However, existing reasoning models focus on conventional problem-solving and do not necessarily generate creative solutions by ``creative reasoning''. In domains where the solution space is expansive and conventional solutions are suboptimal, such as drug discovery or business strategization, creative reasoning to discover innovative solutions is crucial. To address this gap, first we introduce a computational framework for creative reasoning inspired by established cognitive science principles. With this framework, we propose three core creative reasoning paradigms, namely, \textit{combinational}, \textit{exploratory}, and \textit{transformative} reasoning, where each offers specific directions for systematic exploration of the universe of thoughts to generate creative solutions. Next, to materialize this framework using LLMs, we introduce the \textit{Universe of Thoughts} (or \textit{UoT}, for short), a novel set of methods to implement the aforementioned three creative processes. Finally, we introduce three novel tasks that necessitate creative problem-solving, along with an evaluation benchmark to assess creativity from three orthogonal perspectives: feasibility as constraint, and utility and novelty as metrics. With a comparative analysis against the state-of-the-art (SOTA) reasoning techniques as well as representative commercial models with reasoning capability, we show that UoT demonstrates superior performance in creative reasoning.
♻ ☆ Human Experts' Evaluation of Generative AI for Context ualizing STEAM Education in the Global South
Matthew Nyaaba, Macharious Nabang, Patrick Kyeremeh, Ibrahim Nantomah, Collins Owusu-Fordjour, Martin Ako, Bismark Nyaaba Akanzire, Kassim Korah Nantomah, Cecilia Issaka, Xiaoming Zhai
This study investigates how human experts evaluate the capacity of Generative AI (GenAI) to contextualize STEAM education in the Global South, with a focus on Ghana. Using a convergent mixed-methods design, four STEAM specialists assessed GenAI-generated lesson plans created with a customized Culturally Responsive Lesson Planner (CRLP) and compared them to standardized lesson plans from the Ghana National Council for Curriculum and Assessment (NaCCA). Quantitative ratings were based on a validated 25-item Culturally Responsive Pedagogy Rubric measuring bias awareness, cultural representation, contextual relevance, linguistic responsiveness, and teacher agency. Qualitative reflections provided additional insight into how GenAI handles cultural and pedagogical appropriateness. Findings show that GenAI, when paired with the CRLP tool, can support contextualized STEAM instruction by linking abstract curriculum standards to learners' cultural knowledge, community practices, and everyday experiences. Experts rated GenAI-assisted lessons as more culturally grounded and pedagogically responsive than NaCCA plans, integrating Indigenous knowledge, bilingual elements, and locally relevant examples. However, GenAI struggled to represent Ghana's cultural pluralism, often offering surface-level references to language, history, and identity. These weaknesses were most evident in Mathematics and Computing, where cultural nuance was limited. The results highlight the need for continued teacher mediation, community involvement, and culturally attuned refinement of AI outputs. Future work should include classroom trials, expanded expert participation, and model fine-tuning using Indigenous language corpora to strengthen cultural fidelity in Global South contexts.
♻ ☆ Prefix GPT : Prefix Adder Optimization by a Generative Pre-train ed Transformer AAAI-2026
Ruogu Ding, Xin Ning, Ulf Schlichtmann, Weikang Qian
Prefix adders are widely used in compute-intensive applications for their high speed. However, designing optimized prefix adders is challenging due to strict design rules and an exponentially large design space. We introduce PrefixGPT, a generative pre-trained Transformer (GPT) that directly generates optimized prefix adders from scratch. Our approach represents an adder's topology as a two-dimensional coordinate sequence and applies a legality mask during generation, ensuring every design is valid by construction. PrefixGPT features a customized decoder-only Transformer architecture. The model is first pre-trained on a corpus of randomly synthesized valid prefix adders to learn design rules and then fine-tuned to navigate the design space for optimized design quality. Compared with existing works, PrefixGPT not only finds a new optimal design with a 7.7% improved area-delay product (ADP) but exhibits superior exploration quality, lowering the average ADP by up to 79.1%. This demonstrates the potential of GPT-style models to first master complex hardware design principles and then apply them for more efficient design optimization.
comment : This is an extended version of the paper accepted by the AAAI-2026 Conference
♻ ☆ OuroMamba: A Data-Free Quantization Framework for Vision Mamba ICCV 2025
Akshat Ramachandran, Mingyu Lee, Huan Xu, Souvik Kundu, Tushar Krishna
We present OuroMamba, the first data-free post-training quantization (DFQ) method for vision Mamba-based models (VMMs). We identify two key challenges in enabling DFQ for VMMs, (1) VMM's recurrent state transitions restricts capturing of long-range interactions and leads to semantically weak synthetic data, (2) VMM activations exhibit dynamic outlier variations across time-steps, rendering existing static PTQ techniques ineffective. To address these challenges, OuroMamba presents a two-stage framework: (1) OuroMamba-Gen to generate semantically rich and meaningful synthetic data. It applies contrastive learning on patch level VMM features generated through neighborhood interactions in the latent state space, (2) OuroMamba-Quant to employ mixed-precision quantization with lightweight dynamic outlier detection during inference. In specific, we present a thresholding based outlier channel selection strategy for activations that gets updated every time-step. Extensive experiments across vision and generative tasks show that our data-free OuroMamba surpasses existing data-driven PTQ techniques, achieving state-of-the-art performance across diverse quantization settings. Additionally, we implement efficient GPU kernels to achieve practical latency speedup of up to 2.36x. Code and synthetic dataset are available here: https://github.com/georgia-tech-synergy-lab/ICCV-OuroMamba
comment : Accepted to ICCV 2025
♻ ☆ Weather Diffusion : Controllable Weather Editing in Intrinsic Space
Yixin Zhu, Zuoliang Zhu, Jian Yang, Miloš Hašan, Jin Xie, Beibei Wang
We present WeatherDiffusion, a diffusion-based framework for controllable weather editing in intrinsic space. Our framework includes two components based on diffusion priors: an inverse renderer that estimates material properties, scene geometry, and lighting as intrinsic maps from an input image, and a forward renderer that utilizes these geometry and material maps along with a text prompt that describes specific weather conditions to generate a final image. The intrinsic maps enhance controllability compared to traditional pixel-space editing approaches.We propose an intrinsic map-aware attention mechanism that improves spatial correspondence and decomposition quality in large outdoor scenes. For forward rendering, we leverage CLIP-space interpolation of weather prompts to achieve fine-grained weather control. We also introduce a synthetic and a real-world dataset, containing 38k and 18k images under various weather conditions, each with intrinsic map annotations. WeatherDiffusion outperforms state-of-the-art pixel-space editing approaches, weather restoration methods, and rendering-based methods, showing promise for downstream tasks such as autonomous driving, enhancing the robustness of detection and segmentation in challenging weather scenarios.
♻ ☆ KRAL: Knowledge and Reasoning Augmented Learning for LLM-assisted Clinical Antimicrobial Therapy
Zhe Li, Yehan Qiu, Yujie Chen, Xiang Zhou
Clinical antimicrobial therapy requires the dynamic integration of pathogen profiles,host factors, pharmacological properties of antimicrobials,and the severity of infection. This complexity imposes fundamental limitations on the applicability of Large Language Models (LLMs) in high-stakes clinical decision-making including knowledge gaps, data privacy concerns, high deployment costs, and limited reasoning capabilities. To address these challenges, we propose KRAL (Knowledge and Reasoning Augmented Learning), a low-cost, scalable, privacy-preserving paradigm that leverages teacher-model reasoning to automatically distill knowledge and reasoning trajectories via answer-to-question reverse generation, employs heuristic learning for semi-supervised data augmentation (reducing manual annotation requirements by approximately 80%), and utilizes agentic reinforcement learning to jointly enhance medical knowledge and reasoning while optimizing computational and memory efficiency. A hierarchical evaluation employing diverse teacher-model proxies reduces assessment costs, while modular interface design facilitates seamless system updates. Experimental results demonstrate that KRAL significantly outperforms traditional Retrieval-Augmented Generation (RAG) and Supervised Fine-Tuning (SFT) methods. It improves knowledge question-answering capability (Accuracy@1 on the external open-source benchmark MEDQA increased by 1.8% vs. SFT and 3.6% vs. RAG) and reasoning capability (Pass@1 on the external benchmark PUMCH Antimicrobial increased by 27% vs. SFT and 27.2% vs. RAG), achieved at about 20% of SFT's long-term training costs. This establishes KRAL as an effective solution for enhancing local LLMs' clinical diagnostic capabilities, enabling low-cost, high-safety deployment in complex medical decision support.
♻ ☆ Fair Algorithms with Probing for Multi- Agent Multi-Armed Bandits
Tianyi Xu, Jiaxin Liu, Nicholas Mattei, Zizhan Zheng
We propose a multi-agent multi-armed bandit (MA-MAB) framework aimed at ensuring fair outcomes across agents while maximizing overall system performance. A key challenge in this setting is decision-making under limited information about arm rewards. To address this, we introduce a novel probing framework that strategically gathers information about selected arms before allocation. In the offline setting, where reward distributions are known, we leverage submodular properties to design a greedy probing algorithm with a provable performance bound. For the more complex online setting, we develop an algorithm that achieves sublinear regret while maintaining fairness. Extensive experiments on synthetic and real-world datasets show that our approach outperforms baseline methods, achieving better fairness and efficiency.
♻ ☆ PRO-V-R1: Reasoning Enhanced Programming Agent for RTL Verification
Yujie Zhao, Zhijing Wu, Zeqing Yuan, Zhongming Yu, Hejia Zhang, Wentao Ni, Chia-Tung Ho, Haoxing Ren, Jishen Zhao
Register-Transfer Level (RTL) verification is a primary bottleneck consuming 60-70% of development time. While Large Language Models (LLMs) show promise for RTL automation, their performance and research focus have overwhelmingly centered on RTL generation rather than verification. Current methods for RTL verification rely on large scale proprietary models (e.g., GPT-4o) to generate Python-based functional references, incurring a high cost and raising data-privacy risks. To date, an end-to-end open-source solution for autonomous verification remains absent. We introduce PRO-V-R1, the first trainable open-source agentic framework for autonomous RTL verification. Our contributions are threefold: (1) we design PRO-V sys, a modular agentic system that couples LLM-based reasoning with programmatic tool use for RTL verification; (2) we establish a data construction pipeline that leverages existing RTL datasets to build simulation-validated, expert-level trajectories tailored for supervised fine-tuning (SFT) RTL verification agents; and (3) we implement an efficient reinforcement learning (RL) algorithm that uses verification-specific rewards derived from program-tool feedback to optimize the end-to-end verification workflow. Our empirical evaluation demonstrates PRO-V-R1 achieves a 57.7% functional correctness rate and 34.0% in robust fault detection, significantly outperforming the base model's 25.7% and 21.8% (respectively) from the state-of-the-art (SOTA) automatic verification system. This configuration also outperforms large-scale proprietary LLMs in functional correctness and shows comparable robustness for fault detection.
♻ ☆ Reasoning Weekly: A General Knowledge and Verbal Reasoning Challenge for Large Language Models
Zixuan Wu, Francesca Lucchetti, Aleksander Boruch-Gruszecki, Jingmiao Zhao, Carolyn Jane Anderson, Joydeep Biswas, Federico Cassano, Arjun Guha
Existing benchmarks for frontier models often test specialized, "PhD-level" knowledge that is difficult for non-experts to grasp. In contrast, we present a benchmark with 613 problems based on the NPR Sunday Puzzle Challenge that requires only general knowledge. Our benchmark is challenging for both humans and models; however correct solutions are easy to verify, and models' mistakes are easy to spot. As LLMs are more widely deployed in society, we believe it is useful to develop benchmarks for frontier models that humans can understand without the need for deep domain expertise. Our work reveals capability gaps that are not evident in existing benchmarks: OpenAI o1 significantly outperforms other reasoning models on our benchmark, despite being on par with other models when tested on benchmarks that test specialized knowledge. Furthermore, our analysis of reasoning outputs uncovers new kinds of failures. DeepSeek R1, for instance, often concedes with "I give up" before providing an answer that it knows is wrong. R1 can also be remarkably "uncertain" in its output and in rare cases, it does not "finish thinking," which suggests the need for techniques to ``wrap up'' before the context window limit is reached. We also quantify the effectiveness of reasoning longer to identify the point beyond which more reasoning is unlikely to improve accuracy on our benchmark.
♻ ☆ Improving Action Smoothness for a Cascaded Online Learning Flight Control System
Yifei Li, Erik-jan van Kampen
This paper aims to improve the action smoothness of a cascaded online learning flight control system. Although the cascaded structure is widely used in flight control design, its stability can be compromised by oscillatory control actions, which poses challenges for practical engineering applications. To address this issue, we introduce an online temporal smoothness technique and a low-pass filter to reduce the amplitude and frequency of the control actions. Fast Fourier Transform (FFT) is used to analyze policy performance in the frequency domain. Simulation results demonstrate the improvements achieved by the two proposed techniques.
♻ ☆ MIRNet: Integrating Constrained Graph-Based Reasoning with Pre-train ing for Diagnostic Medical Imaging AAAI-26
Shufeng Kong, Zijie Wang, Nuan Cui, Hao Tang, Yihan Meng, Yuanyuan Wei, Feifan Chen, Yingheng Wang, Zhuo Cai, Yaonan Wang, Yulong Zhang, Yuzheng Li, Zibin Zheng, Caihua Liu, Hao Liang
Automated interpretation of medical images demands robust modeling of complex visual-semantic relationships while addressing annotation scarcity, label imbalance, and clinical plausibility constraints. We introduce MIRNet (Medical Image Reasoner Network), a novel framework that integrates self-supervised pre-training with constrained graph-based reasoning. Tongue image diagnosis is a particularly challenging domain that requires fine-grained visual and semantic understanding. Our approach leverages self-supervised masked autoencoder (MAE) to learn transferable visual representations from unlabeled data; employs graph attention networks (GAT) to model label correlations through expert-defined structured graphs; enforces clinical priors via constraint-aware optimization using KL divergence and regularization losses; and mitigates imbalance using asymmetric loss (ASL) and boosting ensembles. To address annotation scarcity, we also introduce TongueAtlas-4K, a comprehensive expert-curated benchmark comprising 4,000 images annotated with 22 diagnostic labels--representing the largest public dataset in tongue analysis. Validation shows our method achieves state-of-the-art performance. While optimized for tongue diagnosis, the framework readily generalizes to broader diagnostic medical imaging tasks.
comment : To appear at AAAI-26
♻ ☆ Model-Based Reward Shaping for Adversarial Inverse Reinforcement Learning in Stochastic Environments
Simon Sinong Zhan, Philip Wang, Qingyuan Wu, Yixuan Wang, Ruochen Jiao, Chao Huang, Qi Zhu
In this paper, we aim to tackle the limitation of the Adversarial Inverse Reinforcement Learning (AIRL) method in stochastic environments where theoretical results cannot hold and performance is degraded. To address this issue, we propose a novel method which infuses the dynamics information into the reward shaping with the theoretical guarantee for the induced optimal policy in the stochastic environments. Incorporating our novel model-enhanced rewards, we present a novel Model-Enhanced AIRL framework, which integrates transition model estimation directly into reward shaping. Furthermore, we provide a comprehensive theoretical analysis of the reward error bound and performance difference bound for our method. The experimental results in MuJoCo benchmarks show that our method can achieve superior performance in stochastic environments and competitive performance in deterministic environments, with significant improvement in sample efficiency, compared to existing baselines.
♻ ☆ Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering
Federico Felizzi, Olivia Riccomi, Michele Ferramola, Francesco Andrea Causio, Manuel Del Medico, Vittorio De Vita, Lorenzo De Mori, Alessandra Piscitelli, Pietro Eric Risuleo, Bianca Destro Castaniti, Antonio Cristiano, Alessia Longo, Luigi De Angelis, Mariapia Vassalli, Marcello Di Pumpo
Large vision language models (VLMs) have achieved impressive performance on medical visual question answering benchmarks, yet their reliance on visual information remains unclear. We investigate whether frontier VLMs demonstrate genuine visual grounding when answering Italian medical questions by testing four state-of-the-art models: Claude Sonnet 4.5, GPT-4o, GPT-5-mini, and Gemini 2.0 flash exp. Using 60 questions from the EuropeMedQA Italian dataset that explicitly require image interpretation, we substitute correct medical images with blank placeholders to test whether models truly integrate visual and textual information. Our results reveal striking variability in visual dependency: GPT-4o shows the strongest visual grounding with a 27.9pp accuracy drop (83.2% [74.6%, 91.7%] to 55.3% [44.1%, 66.6%]), while GPT-5-mini, Gemini, and Claude maintain high accuracy with modest drops of 8.5pp, 2.4pp, and 5.6pp respectively. Analysis of model-generated reasoning reveals confident explanations for fabricated visual interpretations across all models, suggesting varying degrees of reliance on textual shortcuts versus genuine visual analysis. These findings highlight critical differences in model robustness and the need for rigorous evaluation before clinical deployment.
comment : Accepted at the Workshop on Multimodal Representation Learning for Healthcare (MMRL4H), EurIPS 2025
♻ ☆ From Vision to Validation: A Theory- and Data-Driven Construction of a GCC-Specific AI Adoption Index
Mohammad Rashed Albous, Abdel Latef Anouze
Artificial intelligence (AI) is rapidly transforming public-sector processes worldwide, yet standardized measures rarely address the unique drivers, governance models, and cultural nuances of the Gulf Cooperation Council (GCC) countries. This study employs a theory-driven foundation derived from an in-depth analysis of literature review and six National AI Strategies (NASs), coupled with a data-driven approach that utilizes a survey of 203 mid- and senior-level government employees and advanced statistical techniques (K-Means clustering, Principal Component Analysis, and Partial Least Squares Structural Equation Modeling). By combining policy insights with empirical evidence, the research develops and validates a novel AI Adoption Index specifically tailored to the GCC public sector. Findings indicate that robust technical infrastructure and clear policy mandates exert the strongest influence on successful AI implementations, overshadowing organizational readiness in early adoption stages. The combined model explains 70% of the variance in AI outcomes, suggesting that resource-rich environments and top-down policy directives can drive rapid but uneven technology uptake. By consolidating key dimensions (Technical Infrastructure (TI), Organizational Readiness (OR), and Governance Environment (GE)) into a single composite index, this study provides a holistic yet context-sensitive tool for benchmarking AI maturity. The index offers actionable guidance for policymakers seeking to harmonize large-scale deployments with ethical and regulatory standards. Beyond advancing academic discourse, these insights inform more strategic allocation of resources, cross-country cooperation, and capacity-building initiatives, thereby supporting sustained AI-driven transformation in the GCC region and beyond.
comment : 38 pages, 8 figures, 17 tables
♻ ☆ VITA: Zero-Shot Value Functions via Test-Time Adaptation of Vision -Language Models
Christos Ziakas, Alessandra Russo
Vision-Language Models (VLMs) show promise as zero-shot goal-conditioned value functions, but their frozen pre-trained representations limit generalization and temporal reasoning. We introduce VITA, a zero-shot value function learning method that enhances both capabilities via test-time adaptation. At inference, a lightweight adaptation module is updated via a gradient step on a meta-learned self-supervised loss, such that each test-time update improves value estimation. By updating sequentially over a trajectory, VITA encodes history into its parameters, addressing the temporal reasoning limitations. To mitigate shortcut learning, we propose a dissimilarity-based sampling strategy that selects semantically diverse segments of the trajectory during training. In real-world robotic manipulation tasks, VITA generalizes from a single training environment to diverse out-of-distribution tasks, environments, and embodiments, outperforming the state-of-the-art zero-shot method using autoregressive VLMs. Furthermore, we demonstrate that VITA's zero-shot value estimates can be utilized for reward shaping in offline reinforcement learning, resulting in multi-task policies on the Meta-World benchmark that exceed the performance of those trained with the simulation's fuzzy-logic dense rewards.
♻ ☆ Terminal Velocity Matching
Linqi Zhou, Mathias Parger, Ayaan Haque, Jiaming Song
We propose Terminal Velocity Matching (TVM), a generalization of flow matching that enables high-fidelity one- and few-step generative modeling. TVM models the transition between any two diffusion timesteps and regularizes its behavior at its terminal time rather than at the initial time. We prove that TVM provides an upper bound on the $2$-Wasserstein distance between data and model distributions when the model is Lipschitz continuous. However, since Diffusion Transformers lack this property, we introduce minimal architectural changes that achieve stable, single-stage training. To make TVM efficient in practice, we develop a fused attention kernel that supports backward passes on Jacobian-Vector Products, which scale well with transformer architectures. On ImageNet-256x256, TVM achieves 3.29 FID with a single function evaluation (NFE) and 1.99 FID with 4 NFEs. It similarly achieves 4.32 1-NFE FID and 2.94 4-NFE FID on ImageNet-512x512, representing state-of-the-art performance for one/few-step models from scratch.
comment : Blog post: https://lumalabs.ai/blog/engineering/tvm Code available at: https://github.com/lumalabs/tvm
♻ ☆ ProxT2I: Efficient Reward-Guided Text-to-Image Generation via Proximal Diffusion
Zhenghan Fang, Jian Zheng, Qiaozi Gao, Xiaofeng Gao, Jeremias Sulam
Diffusion models have emerged as a dominant paradigm for generative modeling across a wide range of domains, including prompt-conditional generation. The vast majority of samplers, however, rely on forward discretization of the reverse diffusion process and use score functions that are learned from data. Such forward and explicit discretizations can be slow and unstable, requiring a large number of sampling steps to produce good-quality samples. In this work we develop a text-to-image (T2I) diffusion model based on backward discretizations, dubbed ProxT2I, relying on learned and conditional proximal operators instead of score functions. We further leverage recent advances in reinforcement learning and policy optimization to optimize our samplers for task-specific rewards. Additionally, we develop a new large-scale and open-source dataset comprising 15 million high-quality human images with fine-grained captions, called LAION-Face-T2I-15M, for training and evaluation. Our approach consistently enhances sampling efficiency and human-preference alignment compared to score-based baselines, and achieves results on par with existing state-of-the-art and open-source text-to-image models while requiring lower compute and smaller model size, offering a lightweight yet performant solution for human text-to-image generation.
♻ ☆ A Layered Protocol Architecture for the Internet of Agent s
Charles Fleming, Vijoy Pandey, Luca Muscariello, Ramana Kompella
Large Language Models (LLMs) have demonstrated remarkable performance improvements and the ability to learn domain-specific languages (DSLs), including APIs and tool interfaces. This capability has enabled the creation of AI agents that can perform preliminary computations and act through tool calling, now being standardized via protocols like MCP. However, LLMs face fundamental limitations: their context windows cannot grow indefinitely, constraining their memory and computational capacity. Agent collaboration emerges as essential for solving increasingly complex problems, mirroring how computational systems rely on different types of memory to scale. The "Internet of Agents" (IoA) represents the communication stack that enables agents to scale by distributing computation across collaborating entities. Current network architectural stacks (OSI and TCP/IP) were designed for data delivery between hosts and processes, not for agent collaboration with semantic understanding. To address this gap, we propose two new layers: an Agent Communication Layer (L8) and an Agent Semantic Negotiation Layer (L9). L8 formalizes the structure of communication, standardizing message envelopes, speech-act performatives (e.g., REQUEST, INFORM), and interaction patterns (e.g., request-reply, publish-subscribe), building on protocols like MCP. L9, which does not exist today, formalizes the meaning of communication, enabling agents to discover, negotiate, and lock a "Shared Context" -- a formal schema defining the concepts, tasks, and parameters relevant to their interaction. Together, these layers provide the foundation for scalable, distributed agent collaboration, enabling the next generation of multi-agentic systems.
♻ ☆ Continual Learning with Global Alignment
Xueying Bai, Jinghuan Shang, Yifan Sun, Niranjan Balasubramanian
Continual learning aims to sequentially learn new tasks without forgetting previous tasks' knowledge (catastrophic forgetting). One factor that can cause forgetting is the interference between the gradients on losses from different tasks. When the gradients on the current task's loss are in opposing directions to those on previous tasks' losses, updating the model for the current task may cause performance degradation on previous tasks. In this paper, we first identify causes of the above interference, and hypothesize that correlations between data representations are a key factor of interference. We then propose a method for promoting appropriate correlations between arbitrary tasks' data representations (i.e., global alignment) in individual task learning. Specifically, we learn the data representation as a task-specific composition of pre-trained token representations shared across all tasks. Then the correlations between different tasks' data representations are grounded by correlations between pre-trained token representations. We explore different ways to learn such compositions. Without experience replay, our model achieves SOTA performance in continual learning tasks. It also achieves advanced class-incremental performance through task-incremental training.
comment : Neurips 2024
♻ ☆ Unifying Re-Identification, Attribute Inference, and Data Reconstruction Risks in Differential Privacy NeurIPS 2025
Bogdan Kulynych, Juan Felipe Gomez, Georgios Kaissis, Jamie Hayes, Borja Balle, Flavio P. Calmon, Jean Louis Raisaro
Differentially private (DP) mechanisms are difficult to interpret and calibrate because existing methods for mapping standard privacy parameters to concrete privacy risks -- re-identification, attribute inference, and data reconstruction -- are both overly pessimistic and inconsistent. In this work, we use the hypothesis-testing interpretation of DP ($f$-DP), and determine that bounds on attack success can take the same unified form across re-identification, attribute inference, and data reconstruction risks. Our unified bounds are (1) consistent across a multitude of attack settings, and (2) tunable, enabling practitioners to evaluate risk with respect to arbitrary, including worst-case, levels of baseline risk. Empirically, our results are tighter than prior methods using $\varepsilon$-DP, Rényi DP, and concentrated DP. As a result, calibrating noise using our bounds can reduce the required noise by 20% at the same risk level, which yields, e.g., an accuracy increase from 52% to 70% in a text classification task. Overall, this unifying perspective provides a principled framework for interpreting and calibrating the degree of protection in DP against specific levels of re-identification, attribute inference, or data reconstruction risk.
comment : NeurIPS 2025. v3 corrects Eq. (36) in Appendix D, and typos
Computer Vision and Pattern Recognition 228
☆ Canvas-to-Image: Compositional Image Generation with Multimodal Controls
Yusuf Dalva, Guocheng Gordon Qian, Maya Goldenberg, Tsai-Shien Chen, Kfir Aberman, Sergey Tulyakov, Pinar Yanardag, Kuan-Chieh Jackson Wang
While modern diffusion models excel at generating high-quality and diverse images, they still struggle with high-fidelity compositional and multimodal control, particularly when users simultaneously specify text prompts, subject references, spatial arrangements, pose constraints, and layout annotations. We introduce Canvas-to-Image, a unified framework that consolidates these heterogeneous controls into a single canvas interface, enabling users to generate images that faithfully reflect their intent. Our key idea is to encode diverse control signals into a single composite canvas image that the model can directly interpret for integrated visual-spatial reasoning. We further curate a suite of multi-task datasets and propose a Multi-Task Canvas Training strategy that optimizes the diffusion model to jointly understand and integrate heterogeneous controls into text-to-image generation within a unified learning paradigm. This joint training enables Canvas-to-Image to reason across multiple control modalities rather than relying on task-specific heuristics, and it generalizes well to multi-control scenarios during inference. Extensive experiments show that Canvas-to-Image significantly outperforms state-of-the-art methods in identity preservation and control adherence across challenging benchmarks, including multi-person composition, pose-controlled composition, layout-constrained generation, and multi-control generation.
comment : 24 pages; webpage: https://snap-research.github.io/canvas-to-image/
☆ TraceGen: Wo rl d Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videos
Seungjae Lee, Yoonkyo Jung, Inkook Chun, Yao-Chih Lee, Zikui Cai, Hongjia Huang, Aayush Talreja, Tan Dat Dao, Yongyuan Liang, Jia-Bin Huang, Furong Huang
Learning new robot tasks on new platforms and in new scenes from only a handful of demonstrations remains challenging. While videos of other embodiments - humans and different robots - are abundant, differences in embodiment, camera, and environment hinder their direct use. We address the small-data problem by introducing a unifying, symbolic representation - a compact 3D "trace-space" of scene-level trajectories - that enables learning from cross-embodiment, cross-environment, and cross-task videos. We present TraceGen, a world model that predicts future motion in trace-space rather than pixel space, abstracting away appearance while retaining the geometric structure needed for manipulation. To train TraceGen at scale, we develop TraceForge, a data pipeline that transforms heterogeneous human and robot videos into consistent 3D traces, yielding a corpus of 123K videos and 1.8M observation-trace-language triplets. Pretraining on this corpus produces a transferable 3D motion prior that adapts efficiently: with just five target robot videos, TraceGen attains 80% success across four tasks while offering 50-600x faster inference than state-of-the-art video-based world models. In the more challenging case where only five uncalibrated human demonstration videos captured on a handheld phone are available, it still reaches 67.5% success on a real robot, highlighting TraceGen's ability to adapt across embodiments without relying on object detectors or heavy pixel-space generation.
☆ G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
Wenbo Hu, Jingli Lin, Yilin Long, Yunlong Ran, Lihan Jiang, Yifan Wang, Chenming Zhu, Runsen Xu, Tai Wang, Jiangmiao Pang
Vision-Language Models (VLMs) still lack robustness in spatial intelligence, demonstrating poor performance on spatial understanding and reasoning tasks. We attribute this gap to the absence of a visual geometry learning process capable of reconstructing 3D space from 2D images. We present G$^2$VLM, a geometry grounded vision-language model that bridges two fundamental aspects of spatial intelligence: spatial 3D reconstruction and spatial understanding. G$^2$VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning. Our unified design is highly scalable for spatial understanding: it trains on abundant multi-view image and video data, while simultaneously leveraging the benefits of 3D visual priors that are typically only derived from hard-to-collect annotations. Experimental results demonstrate G$^2$VLM is proficient in both tasks, achieving comparable results to state-of-the-art feed-forward 3D reconstruction models and achieving better or competitive results across spatial understanding and reasoning tasks. By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G$^2$VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing.
comment : code are released at https://github.com/InternRobotics/G2VLM
☆ Seeing without Pixels: Perception from Camera Trajectories
Zihui Xue, Kristen Grauman, Dima Damen, Andrew Zisserman, Tengda Han
Can one perceive a video's content without seeing its pixels, just from the camera trajectory-the path it carves through space? This paper is the first to systematically investigate this seemingly implausible question. Towards this end, we propose a contrastive learning framework to train CamFormer, a dedicated encoder that projects camera pose trajectories into a joint embedding space, aligning them with natural language. We find that, contrary to its apparent simplicity, the camera trajectory is a remarkably informative signal to uncover video content. In other words, "how you move" can indeed reveal "what you are doing" (egocentric) or "observing" (exocentric). We demonstrate the versatility of our learned CamFormer embeddings on a diverse suite of downstream tasks, ranging from cross-modal alignment to classification and temporal analysis. Importantly, our representations are robust across diverse camera pose estimation methods, including both high-fidelity multi-sensored and standard RGB-only estimators. Our findings establish camera trajectory as a lightweight, robust, and versatile modality for perceiving video content.
comment : Project website: https://sites.google.com/view/seeing-without-pixels
☆ Revolutionizing Glioma Segmentation & Grading Using 3D MRI - Guided Hybrid Deep Learning Models
Pandiyaraju V, Sreya Mynampati, Abishek Karthik, Poovarasan L, D. Saraswathi
Gliomas are brain tumor types that have a high mortality rate which means early and accurate diagnosis is important for therapeutic intervention for the tumors. To address this difficulty, the proposed research will develop a hybrid deep learning model which integrates U-Net based segmentation and a hybrid DenseNet-VGG classification network with multihead attention and spatial-channel attention capabilities. The segmentation model will precisely demarcate the tumors in a 3D volume of MRI data guided by spatial and contextual information. The classification network which combines a branch of both DenseNet and VGG, will incorporate the demarcated tumor on which features with attention mechanisms would be focused on clinically relevant features. High-dimensional 3D MRI data could successfully be utilized in the model through preprocessing steps which are normalization, resampling, and data augmentation. Through a variety of measures the framework is evaluated: measures of performance in segmentation are Dice coefficient and Mean Intersection over Union (IoU) and measures of performance in classification are accuracy precision, recall, and F1-score. The hybrid framework that has been proposed has demonstrated through physical testing that it has the capability of obtaining a Dice coefficient of 98% in tumor segmentation, and 99% on classification accuracy, outperforming traditional CNN models and attention-free methods. Utilizing multi-head attention mechanisms enhances notions of priority in aspects of the tumor that are clinically significant, and enhances interpretability and accuracy. The results suggest a great potential of the framework in facilitating the timely and reliable diagnosis and grading of glioma by clinicians is promising, allowing for better planning of patient treatment.
☆ Uncertainty Quantification for Visual Object Pose Estimation SP
Lorenzo Shaikewitz, Charis Georgiou, Luca Carlone
Quantifying the uncertainty of an object's pose estimate is essential for robust control and planning. Although pose estimation is a well-studied robotics problem, attaching statistically rigorous uncertainty is not well understood without strict distributional assumptions. We develop distribution-free pose uncertainty bounds about a given pose estimate in the monocular setting. Our pose uncertainty only requires high probability noise bounds on pixel detections of 2D semantic keypoints on a known object. This noise model induces an implicit, non-convex set of pose uncertainty constraints. Our key contribution is SLUE (S-Lemma Uncertainty Estimation), a convex program to reduce this set to a single ellipsoidal uncertainty bound that is guaranteed to contain the true object pose with high probability. SLUE solves a relaxation of the minimum volume bounding ellipsoid problem inspired by the celebrated S-lemma. It requires no initial guess of the bound's shape or size and is guaranteed to contain the true object pose with high probability. For tighter uncertainty bounds at the same confidence, we extend SLUE to a sum-of-squares relaxation hierarchy which is guaranteed to converge to the minimum volume ellipsoidal uncertainty bound for a given set of keypoint constraints. We show this pose uncertainty bound can easily be projected to independent translation and axis-angle orientation bounds. We evaluate SLUE on two pose estimation datasets and a real-world drone tracking scenario. Compared to prior work, SLUE generates substantially smaller translation bounds and competitive orientation bounds. We release code at https://github.com/MIT-SPARK/PoseUncertaintySets.
comment : 18 pages, 9 figures. Code available: https://github.com/MIT-SPARK/PoseUncertaintySets
☆ Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision -Language-Action Models
Naifu Zhang, Wei Tao, Xi Xiao, Qianpu Sun, Yuxin Zheng, Wentao Mo, Peiqiang Wang, Nan Zhang
In recent years, Vision-Language-Action (VLA) models in embodied intelligence have developed rapidly. However, existing adversarial attack methods require costly end-to-end training and often generate noticeable perturbation patches. To address these limitations, we propose ADVLA, a framework that directly applies adversarial perturbations on features projected from the visual encoder into the textual feature space. ADVLA efficiently disrupts downstream action predictions under low-amplitude constraints, and attention guidance allows the perturbations to be both focused and sparse. We introduce three strategies that enhance sensitivity, enforce sparsity, and concentrate perturbations. Experiments demonstrate that under an $L_{\infty}=4/255$ constraint, ADVLA combined with Top-K masking modifies less than 10% of the patches while achieving an attack success rate of nearly 100%. The perturbations are concentrated on critical regions, remain almost imperceptible in the overall image, and a single-step iteration takes only about 0.06 seconds, significantly outperforming conventional patch-based attacks. In summary, ADVLA effectively weakens downstream action predictions of VLA models under low-amplitude and locally sparse conditions, avoiding the high training costs and conspicuous perturbations of traditional patch attacks, and demonstrates unique effectiveness and practical value for attacking VLA feature spaces.
☆ Multi-Crit: Benchmark ing Multimodal Judge s on Pluralistic Criteria-Following
Tianyi Xiong, Yi Ge, Ming Li, Zuolong Zhang, Pranav Kulkarni, Kaishen Wang, Qi He, Zeying Zhu, Chenxi Liu, Ruibo Chen, Tong Zheng, Yanshuo Chen, Xiyao Wang, Renrui Zhang, Wenhu Chen, Heng Huang
Large multimodal models (LMMs) are increasingly adopted as judges in multimodal evaluation systems due to their strong instruction following and consistency with human preferences. However, their ability to follow diverse, fine-grained evaluation criteria remains underexplored. We develop Multi-Crit, a benchmark for evaluating multimodal judges on their capacity to follow pluralistic criteria and produce reliable criterion-level judgments. Covering both open-ended generation and verifiable reasoning tasks, Multi-Crit is built through a rigorous data curation pipeline that gathers challenging response pairs with multi-criterion human annotations. It further introduces three novel metrics for systematically assessing pluralistic adherence, criterion-switching flexibility, and the ability to recognize criterion-level preference conflicts. Comprehensive analysis of 25 LMMs reveals that 1) proprietary models still struggle to maintain consistent adherence to pluralistic criteria--especially in open-ended evaluation; 2) open-source models lag further behind in flexibly following diverse criteria; and 3) critic fine-tuning with holistic judgment signals enhances visual grounding but fails to generalize to pluralistic criterion-level judgment. Additional analyses on reasoning fine-tuning, test-time scaling, and boundary consistency between open-source and proprietary models further probe the limits of current multimodal judges. As a pioneering study, Multi-Crit lays the foundation for building reliable and steerable multimodal AI evaluation.
☆ CaFlow: Enhancing Long-Term Action Quality Assessment with Causal Counterfactual Flow
Ruisheng Han, Kanglei Zhou, Shuang Chen, Amir Atapour-Abarghouei, Hubert P. H. Shum
Action Quality Assessment (AQA) predicts fine-grained execution scores from action videos and is widely applied in sports, rehabilitation, and skill evaluation. Long-term AQA, as in figure skating or rhythmic gymnastics, is especially challenging since it requires modeling extended temporal dynamics while remaining robust to contextual confounders. Existing approaches either depend on costly annotations or rely on unidirectional temporal modeling, making them vulnerable to spurious correlations and unstable long-term representations. To this end, we propose CaFlow, a unified framework that integrates counterfactual de-confounding with bidirectional time-conditioned flow. The Causal Counterfactual Regularization (CCR) module disentangles causal and confounding features in a self-supervised manner and enforces causal robustness through counterfactual interventions, while the BiT-Flow module models forward and backward dynamics with a cycle-consistency constraint to produce smoother and more coherent representations. Extensive experiments on multiple long-term AQA benchmarks demonstrate that CaFlow achieves state-of-the-art performance. Code is available at https://github.com/Harrison21/CaFlow
☆ Continual Error Correction on Low-Resource Devices ACM MM
Kirill Paramonov, Mete Ozay, Aristeidis Mystakidis, Nikolaos Tsalikidis, Dimitrios Sotos, Anastasios Drosou, Dimitrios Tzovaras, Hyunjun Kim, Kiseok Chang, Sangdok Mo, Namwoong Kim, Woojong Yoo, Jijoong Moon, Umberto Michieli
The proliferation of AI models in everyday devices has highlighted a critical challenge: prediction errors that degrade user experience. While existing solutions focus on error detection, they rarely provide efficient correction mechanisms, especially for resource-constrained devices. We present a novel system enabling users to correct AI misclassifications through few-shot learning, requiring minimal computational resources and storage. Our approach combines server-side foundation model training with on-device prototype-based classification, enabling efficient error correction through prototype updates rather than model retraining. The system consists of two key components: (1) a server-side pipeline that leverages knowledge distillation to transfer robust feature representations from foundation models to device-compatible architectures, and (2) a device-side mechanism that enables ultra-efficient error correction through prototype adaptation. We demonstrate our system's effectiveness on both image classification and object detection tasks, achieving over 50% error correction in one-shot scenarios on Food-101 and Flowers-102 datasets while maintaining minimal forgetting (less than 0.02%) and negligible computational overhead. Our implementation, validated through an Android demonstration app, proves the system's practicality in real-world scenarios.
comment : ACM MMSys 2025
☆ Mechanisms of Non-Monotonic Scaling in Vision Transformer s
Anantha Padmanaban Krishna Kumar
Deeper Vision Transformers often perform worse than shallower ones, which challenges common scaling assumptions. Through a systematic empirical analysis of ViT-S, ViT-B, and ViT-L on ImageNet, we identify a consistent three-phase Cliff-Plateau-Climb pattern that governs how representations evolve with depth. We observe that better performance is associated with progressive marginalization of the [CLS] token, originally designed as a global aggregation hub, in favor of distributed consensus among patch tokens. We quantify patterns of information mixing with an Information Scrambling Index, and show that in ViT-L the information-task tradeoff emerges roughly 10 layers later than in ViT-B, and that these additional layers correlate with increased information diffusion rather than improved task performance. Taken together, these results suggest that transformer architectures in this regime may benefit more from carefully calibrated depth that executes clean phase transitions than from simply increasing parameter count. The Information Scrambling Index provides a useful diagnostic for existing models and suggests a potential design target for future architectures. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/Cliff-Plateau-Climb.
comment : 16 pages total (11 pages main text, 1 pages references, 4 pages appendix), 5 figures, 11 tables. Code available at https://github.com/AnanthaPadmanaban-KrishnaKumar/Cliff-Plateau-Climb
☆ Qwen 3-VL Technical Report
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, Ke Zhu
We introduce Qwen3-VL, the most capable vision-language model in the Qwen series to date, achieving superior performance across a broad range of multimodal benchmarks. It natively supports interleaved contexts of up to 256K tokens, seamlessly integrating text, images, and video. The model family includes both dense (2B/4B/8B/32B) and mixture-of-experts (30B-A3B/235B-A22B) variants to accommodate diverse latency-quality trade-offs. Qwen3-VL delivers three core pillars: (i) markedly stronger pure-text understanding, surpassing comparable text-only backbones in several cases; (ii) robust long-context comprehension with a native 256K-token window for both text and interleaved multimodal inputs, enabling faithful retention, retrieval, and cross-referencing across long documents and videos; and (iii) advanced multimodal reasoning across single-image, multi-image, and video tasks, demonstrating leading performance on comprehensive evaluations such as MMMU and visual-math benchmarks (e.g., MathVista and MathVision). Architecturally, we introduce three key upgrades: (i) an enhanced interleaved-MRoPE for stronger spatial-temporal modeling across images and video; (ii) DeepStack integration, which effectively leverages multi-level ViT features to tighten vision-language alignment; and (iii) text-based time alignment for video, evolving from T-RoPE to explicit textual timestamp alignment for more precise temporal grounding. Under comparable token budgets and latency constraints, Qwen3-VL achieves superior performance in both dense and Mixture-of-Experts (MoE) architectures. We envision Qwen3-VL serving as a foundational engine for image-grounded reasoning, agentic decision-making, and multimodal code intelligence in real-world workflows.
comment : 42 pages
☆ Active Learning for GCN-based Action Recognition
Hichem Sahbi
Despite the notable success of graph convolutional networks (GCNs) in skeleton-based action recognition, their performance often depends on large volumes of labeled data, which are frequently scarce in practical settings. To address this limitation, we propose a novel label-efficient GCN model. Our work makes two primary contributions. First, we develop a novel acquisition function that employs an adversarial strategy to identify a compact set of informative exemplars for labeling. This selection process balances representativeness, diversity, and uncertainty. Second, we introduce bidirectional and stable GCN architectures. These enhanced networks facilitate a more effective mapping between the ambient and latent data spaces, enabling a better understanding of the learned exemplar distribution. Extensive evaluations on two challenging skeleton-based action recognition benchmarks reveal significant improvements achieved by our label-efficient GCNs compared to prior work.
☆ ReSAM: Refine, Requery, and Reinforce: Self- Prompt ing Point-Supervised Segmentation for Remote Sensing Images
M. Naseer Subhani
Interactive segmentation models such as the Segment Anything Model (SAM) have demonstrated remarkable generalization on natural images, but perform suboptimally on remote sensing imagery (RSI) due to severe domain shift and the scarcity of dense annotations. To address this, we propose a self-prompting, point-supervised framework that adapts SAM to RSIs using only sparse point annotations. Our method employs a Refine-Requery-Reinforce loop, where coarse pseudo-masks are generated from initial points (Refine), improved with self-constructed box prompts (Requery), and embeddings are aligned across iterations to reduce confirmation bias (Reinforce). Without relying on full-mask supervision, our approach progressively enhances SAM's segmentation quality and domain robustness through self-guided prompt adaptation . We evaluate our proposed method on three RSI benchmark datasets, including WHU, HRSID, and NWPU VHR-10, showing that our method consistently surpasses pretrained SAM and recent point-supervised segmentation methods. Our results demonstrate that self-prompting and semantic alignment provide an efficient path towards scalable, point-level adaptation of foundation segmentation models for remote sensing applications.
☆ MoGAN: Improving Motion Quality in Video Diffusion via Few-Step Motion Adversarial Post-Training
Haotian Xue, Qi Chen, Zhonghao Wang, Xun Huang, Eli Shechtman, Jinrong Xie, Yongxin Chen
Video diffusion models achieve strong frame-level fidelity but still struggle with motion coherence, dynamics and realism, often producing jitter, ghosting, or implausible dynamics. A key limitation is that the standard denoising MSE objective provides no direct supervision on temporal consistency, allowing models to achieve low loss while still generating poor motion. We propose MoGAN, a motion-centric post-training framework that improves motion realism without reward models or human preference data. Built atop a 3-step distilled video diffusion model, we train a DiT-based optical-flow discriminator to differentiate real from generated motion, combined with a distribution-matching regularizer to preserve visual fidelity. With experiments on Wan2.1-T2V-1.3B, MoGAN substantially improves motion quality across benchmarks. On VBench, MoGAN boosts motion score by +7.3% over the 50-step teacher and +13.3% over the 3-step DMD model. On VideoJAM-Bench, MoGAN improves motion score by +7.4% over the teacher and +8.8% over DMD, while maintaining comparable or even better aesthetic and image-quality scores. A human study further confirms that MoGAN is preferred for motion quality (52% vs. 38% for the teacher; 56% vs. 29% for DMD). Overall, MoGAN delivers significantly more realistic motion without sacrificing visual fidelity or efficiency, offering a practical path toward fast, high-quality video generation. Project webpage is: https://xavihart.github.io/mogan.
☆ Deep Learning-Based Multiclass Classification of Oral Lesions with Stratified Augmentation
Joy Naoum, Revana Salama, Ali Hamdi
Oral cancer is highly common across the globe and is mostly diagnosed during the later stages due to the close visual similarity to benign, precancerous, and malignant lesions in the oral cavity. Implementing computer aided diagnosis systems early on has the potential to greatly improve clinical outcomes. This research intends to use deep learning to build a multiclass classifier for sixteen different oral lesions. To overcome the challenges of limited and imbalanced datasets, the proposed technique combines stratified data splitting and advanced data augmentation and oversampling to perform the classification. The experimental results, which achieved 83.33 percent accuracy, 89.12 percent precision, and 77.31 percent recall, demonstrate the superiority of the suggested model over state of the art methods now in use. The suggested model effectively conveys the effectiveness of oversampling and augmentation strategies in situations where the minority class classification performance is noteworthy. As a first step toward trustworthy computer aided diagnostic systems for the early detection of oral cancer in clinical settings, the suggested framework shows promise.
comment : 12 pages, 3 figures,
☆ Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy
Teng Hu, Zhentao Yu, Guozhen Zhang, Zihan Su, Zhengguang Zhou, Youliang Zhang, Yuan Zhou, Qinglin Lu, Ran Yi
The synthesis of synchronized audio-visual content is a key challenge in generative AI, with open-source models facing challenges in robust audio-video alignment. Our analysis reveals that this issue is rooted in three fundamental challenges of the joint diffusion process: (1) Correspondence Drift, where concurrently evolving noisy latents impede stable learning of alignment; (2) inefficient global attention mechanisms that fail to capture fine-grained temporal cues; and (3) the intra-modal bias of conventional Classifier-Free Guidance (CFG), which enhances conditionality but not cross-modal synchronization. To overcome these challenges, we introduce Harmony, a novel framework that mechanistically enforces audio-visual synchronization. We first propose a Cross-Task Synergy training paradigm to mitigate drift by leveraging strong supervisory signals from audio-driven video and video-driven audio generation tasks. Then, we design a Global-Local Decoupled Interaction Module for efficient and precise temporal-style alignment. Finally, we present a novel Synchronization-Enhanced CFG (SyncCFG) that explicitly isolates and amplifies the alignment signal during inference. Extensive experiments demonstrate that Harmony establishes a new state-of-the-art, significantly outperforming existing methods in both generation fidelity and, critically, in achieving fine-grained audio-visual synchronization.
☆ Enhanced Landmark Detection Model in Pelvic Fluoroscopy using 2D/3D Registration Loss
Chou Mo, Yehyun Suh, J. Ryan Martin, Daniel Moyer
Automated landmark detection offers an efficient approach for medical professionals to understand patient anatomic structure and positioning using intra-operative imaging. While current detection methods for pelvic fluoroscopy demonstrate promising accuracy, most assume a fixed Antero-Posterior view of the pelvis. However, orientation often deviates from this standard view, either due to repositioning of the imaging unit or of the target structure itself. To address this limitation, we propose a novel framework that incorporates 2D/3D landmark registration into the training of a U-Net landmark prediction model. We analyze the performance difference by comparing landmark detection accuracy between the baseline U-Net, U-Net trained with Pose Estimation Loss, and U-Net fine-tuned with Pose Estimation Loss under realistic intra-operative conditions where patient pose is variable.
comment : 9 pages, 3 figures, 1 table
☆ Multimodal Robust Prompt Distillation for 3D Point Cloud Models
Xiang Gu, Liming Lu, Xu Zheng, Anan Du, Yongbin Zhou, Shuchao Pang
Adversarial attacks pose a significant threat to learning-based 3D point cloud models, critically undermining their reliability in security-sensitive applications. Existing defense methods often suffer from (1) high computational overhead and (2) poor generalization ability across diverse attack types. To bridge these gaps, we propose a novel yet efficient teacher-student framework, namely Multimodal Robust Prompt Distillation (MRPD) for distilling robust 3D point cloud model. It learns lightweight prompts by aligning student point cloud model's features with robust embeddings from three distinct teachers: a vision model processing depth projections, a high-performance 3D model, and a text encoder. To ensure a reliable knowledge transfer, this distillation is guided by a confidence-gated mechanism which dynamically balances the contribution of all input modalities. Notably, since the distillation is all during the training stage, there is no additional computational cost at inference. Extensive experiments demonstrate that MRPD substantially outperforms state-of-the-art defense methods against a wide range of white-box and black-box attacks, while even achieving better performance on clean data. Our work presents a new, practical paradigm for building robust 3D vision systems by efficiently harnessing multimodal knowledge.
☆ UAVLight: A Benchmark for Illumination-Robust 3D Reconstruction in Unmanned Aerial Vehicle (UAV) Scenes
Kang Du, Xue Liao, Junpeng Xia, Chaozheng Guo, Yi Gu, Yirui Guan, Duotun Wang, ShengHuang, Zeyu Wang
Illumination inconsistency is a fundamental challenge in multi-view 3D reconstruction. Variations in sunlight direction, cloud cover, and shadows break the constant-lighting assumption underlying both classical multi-view stereo (MVS) and structure from motion (SfM) pipelines and recent neural rendering methods, leading to geometry drift, color inconsistency, and shadow imprinting. This issue is especially critical in UAV-based reconstruction, where long flight durations and outdoor environments make lighting changes unavoidable. However, existing datasets either restrict capture to short time windows, thus lacking meaningful illumination diversity, or span months and seasons, where geometric and semantic changes confound the isolated study of lighting robustness. We introduce UAVLight, a controlled-yet-real benchmark for illumination-robust 3D reconstruction. Each scene is captured along repeatable, geo-referenced flight paths at multiple fixed times of day, producing natural lighting variation under consistent geometry, calibration, and viewpoints. With standardized evaluation protocols across lighting conditions, UAVLight provides a reliable foundation for developing and benchmarking reconstruction methods that are consistent, faithful, and relightable in real outdoor environments.
comment : 10 pages, 6 figures
☆ Video Generation Models Are Good Latent Reward Model s
Xiaoyue Mi, Wenqing Yu, Jiesong Lian, Shibo Jie, Ruizhe Zhong, Zijun Liu, Guozhen Zhang, Zixiang Zhou, Zhiyong Xu, Yuan Zhou, Qinglin Lu, Fan Tang
Reward feedback learning (ReFL) has proven effective for aligning image generation with human preferences. However, its extension to video generation faces significant challenges. Existing video reward models rely on vision-language models designed for pixel-space inputs, confining ReFL optimization to near-complete denoising steps after computationally expensive VAE decoding. This pixel-space approach incurs substantial memory overhead and increased training time, and its late-stage optimization lacks early-stage supervision, refining only visual quality rather than fundamental motion dynamics and structural coherence. In this work, we show that pre-trained video generation models are naturally suited for reward modeling in the noisy latent space, as they are explicitly designed to process noisy latent representations at arbitrary timesteps and inherently preserve temporal information through their sequential modeling capabilities. Accordingly, we propose Process Reward Feedback Learning~(PRFL), a framework that conducts preference optimization entirely in latent space, enabling efficient gradient backpropagation throughout the full denoising chain without VAE decoding. Extensive experiments demonstrate that PRFL significantly improves alignment with human preferences, while achieving substantial reductions in memory consumption and training time compared to RGB ReFL.
☆ Bangla Sign Language Translation: Dataset Creation Challenges, Benchmark ing and Prospects
Husne Ara Rubaiyeat, Hasan Mahmud, Md Kamrul Hasan
Bangla Sign Language Translation (BdSLT) has been severely constrained so far as the language itself is very low resource. Standard sentence level dataset creation for BdSLT is of immense importance for developing AI based assistive tools for deaf and hard of hearing people of Bangla speaking community. In this paper, we present a dataset, IsharaKhobor , and two subset of it for enabling research. We also present the challenges towards developing the dataset and present some way forward by benchmarking with landmark based raw and RQE embedding. We do some ablation on vocabulary restriction and canonicalization of the same within the dataset, which resulted in two more datasets, IsharaKhobor_small and IsharaKhobor_canonical_small. The dataset is publicly available at: www.kaggle.com/datasets/hasanssl/isharakhobor [1].
comment : 14 pages, 8 tables
☆ The Age-specific Alzheimer 's Disease Prediction with Characteristic Constraints in Nonuniform Time Span
Xin Hong, Kaifeng Huang
Alzheimer's disease is a debilitating disorder marked by a decline in cognitive function. Timely identification of the disease is essential for the development of personalized treatment strategies that aim to mitigate its progression. The application of generated images for the prediction of Alzheimer's disease poses challenges, particularly in accurately representing the disease's characteristics when input sequences are captured at irregular time intervals. This study presents an innovative methodology for sequential image generation, guided by quantitative metrics, to maintain the essential features indicative of disease progression. Furthermore, an age-scaling factor is integrated into the process to produce age-specific MRI images, facilitating the prediction of advanced stages of the disease. The results obtained from the ablation study suggest that the inclusion of quantitative metrics significantly improves the accuracy of MRI image synthesis. Furthermore, the application of age-scaled pixel loss contributed to the enhanced iterative generation of MRI images. In terms of long-term disease prognosis, the Structural Similarity Index reached a peak value of 0.882, indicating a substantial degree of similarity in the synthesized images.
comment : 16 pages, 9 figures
☆ EoS-FM: Can an Ensemble of Specialist Models act as a Generalist Feature Extractor?
Pierre Adorni, Minh-Tan Pham, Stéphane May, Sébastien Lefèvre
Recent advances in foundation models have shown great promise in domains such as natural language processing and computer vision, and similar efforts are now emerging in the Earth Observation community. These models aim to generalize across tasks with limited supervision, reducing the need for training separate models for each task. However, current strategies, which largely focus on scaling model size and dataset volume, require prohibitive computational and data resources, limiting accessibility to only a few large institutions. Moreover, this paradigm of ever-larger models stands in stark contrast with the principles of sustainable and environmentally responsible AI, as it leads to immense carbon footprints and resource inefficiency. In this work, we present a novel and efficient alternative: an Ensemble-of-Specialists framework for building Remote Sensing Foundation Models (RSFMs). Our method decomposes the training process into lightweight, task-specific ConvNeXtV2 specialists that can be frozen and reused. This modular approach offers strong advantages in efficiency, interpretability, and extensibility. Moreover, it naturally supports federated training, pruning, and continuous specialist integration, making it particularly well-suited for collaborative and resource-constrained settings. Our framework sets a new direction for building scalable and efficient RSFMs.
☆ Self-Paced Learning for Images of Antinuclear Antibodies
Yiyang Jiang, Guangwu Qian, Jiaxin Wu, Qi Huang, Qing Li, Yongkang Wu, Xiao-Yong Wei
Antinuclear antibody (ANA) testing is a crucial method for diagnosing autoimmune disorders, including lupus, Sjögren's syndrome, and scleroderma. Despite its importance, manual ANA detection is slow, labor-intensive, and demands years of training. ANA detection is complicated by over 100 coexisting antibody types, resulting in vast fluorescent pattern combinations. Although machine learning and deep learning have enabled automation, ANA detection in real-world clinical settings presents unique challenges as it involves multi-instance, multi-label (MIML) learning. In this paper, a novel framework for ANA detection is proposed that handles the complexities of MIML tasks using unaltered microscope images without manual preprocessing. Inspired by human labeling logic, it identifies consistent ANA sub-regions and assigns aggregated labels accordingly. These steps are implemented using three task-specific components: an instance sampler, a probabilistic pseudo-label dispatcher, and self-paced weight learning rate coefficients. The instance sampler suppresses low-confidence instances by modeling pattern confidence, while the dispatcher adaptively assigns labels based on instance distinguishability. Self-paced learning adjusts training according to empirical label observations. Our framework overcomes limitations of traditional MIML methods and supports end-to-end optimization. Extensive experiments on one ANA dataset and three public medical MIML benchmarks demonstrate the superiority of our framework. On the ANA dataset, our model achieves up to +7.0% F1-Macro and +12.6% mAP gains over the best prior method, setting new state-of-the-art results. It also ranks top-2 across all key metrics on public datasets, reducing Hamming loss and one-error by up to 18.2% and 26.9%, respectively. The source code can be accessed at https://github.com/fletcherjiang/ANA-SelfPacedLearning.
comment : IEEE Transactions on Medical Imaging
☆ Generalized Design Choices for Deepfake Detectors
Lorenzo Pellegrini, Serafino Pandolfini, Davide Maltoni, Matteo Ferrara, Marco Prati, Marco Ramilli
The effectiveness of deepfake detection methods often depends less on their core design and more on implementation details such as data preprocessing, augmentation strategies, and optimization techniques. These factors make it difficult to fairly compare detectors and to understand which factors truly contribute to their performance. To address this, we systematically investigate how different design choices influence the accuracy and generalization capabilities of deepfake detection models, focusing on aspects related to training, inference, and incremental updates. By isolating the impact of individual factors, we aim to establish robust, architecture-agnostic best practices for the design and development of future deepfake detection systems. Our experiments identify a set of design choices that consistently improve deepfake detection and enable state-of-the-art performance on the AI-GenBench benchmark.
comment : 12 pages, 9 figures, 10 tables, code available: https://github.com/MI-BioLab/AI-GenBench
☆ CanKD: Cross-Attention-based Non-local operation for Feature-based Knowledge Distillation WACV 2026
Shizhe Sun, Wataru Ohyama
We propose Cross-Attention-based Non-local Knowledge Distillation (CanKD), a novel feature-based knowledge distillation framework that leverages cross-attention mechanisms to enhance the knowledge transfer process. Unlike traditional self-attention-based distillation methods that align teacher and student feature maps independently, CanKD enables each pixel in the student feature map to dynamically consider all pixels in the teacher feature map. This non-local knowledge transfer more thoroughly captures pixel-wise relationships, improving feature representation learning. Our method introduces only an additional loss function to achieve superior performance compared with existing attention-guided distillation methods. Extensive experiments on object detection and image segmentation tasks demonstrate that CanKD outperforms state-of-the-art feature and hybrid distillation methods. These experimental results highlight CanKD's potential as a new paradigm for attention-guided distillation in computer vision tasks. Code is available at https://github.com/tori-hotaru/CanKD
comment : WACV 2026 Accepted
☆ Merge and Bound: Direct Manipulations on Weights for Class Incremental Learning
Taehoon Kim, Donghwan Jang, Bohyung Han
We present a novel training approach, named Merge-and-Bound (M&B) for Class Incremental Learning (CIL), which directly manipulates model weights in the parameter space for optimization. Our algorithm involves two types of weight merging: inter-task weight merging and intra-task weight merging. Inter-task weight merging unifies previous models by averaging the weights of models from all previous stages. On the other hand, intra-task weight merging facilitates the learning of current task by combining the model parameters within current stage. For reliable weight merging, we also propose a bounded update technique that aims to optimize the target model with minimal cumulative updates and preserve knowledge from previous tasks; this strategy reveals that it is possible to effectively obtain new models near old ones, reducing catastrophic forgetting. M&B is seamlessly integrated into existing CIL methods without modifying architecture components or revising learning objectives. We extensively evaluate our algorithm on standard CIL benchmarks and demonstrate superior performance compared to state-of-the-art methods.
☆ Frequency-Aware Token Reduction for Efficient Vision Transformer
Dong-Jae Lee, Jiwan Hur, Jaehyun Choi, Jaemyung Yu, Junmo Kim
Vision Transformers have demonstrated exceptional performance across various computer vision tasks, yet their quadratic computational complexity concerning token length remains a significant challenge. To address this, token reduction methods have been widely explored. However, existing approaches often overlook the frequency characteristics of self-attention, such as rank collapsing and over-smoothing phenomenon. In this paper, we propose a frequency-aware token reduction strategy that improves computational efficiency while preserving performance by mitigating rank collapsing. Our method partitions tokens into high-frequency tokens and low-frequency tokens. high-frequency tokens are selectively preserved, while low-frequency tokens are aggregated into a compact direct current token to retain essential low-frequency components. Through extensive experiments and analysis, we demonstrate that our approach significantly improves accuracy while reducing computational overhead and mitigating rank collapsing and over smoothing. Furthermore, we analyze the previous methods, shedding light on their implicit frequency characteristics and limitations.
comment : Neurips 2025
☆ MobileI2V: Fast and High-Resolution Image-to-Video on Mobile Devices
Shuai Zhang, Bao Tang, Siyuan Yu, Yueting Zhu, Jingfeng Yao, Ya Zou, Shanglin Yuan, Li Yu, Wenyu Liu, Xinggang Wang
Recently, video generation has witnessed rapid advancements, drawing increasing attention to image-to-video (I2V) synthesis on mobile devices. However, the substantial computational complexity and slow generation speed of diffusion models pose significant challenges for real-time, high-resolution video generation on resource-constrained mobile devices. In this work, we propose MobileI2V, a 270M lightweight diffusion model for real-time image-to-video generation on mobile devices. The core lies in: (1) We analyzed the performance of linear attention modules and softmax attention modules on mobile devices, and proposed a linear hybrid architecture denoiser that balances generation efficiency and quality. (2) We design a time-step distillation strategy that compresses the I2V sampling steps from more than 20 to only two without significant quality loss, resulting in a 10-fold increase in generation speed. (3) We apply mobile-specific attention optimizations that yield a 2-fold speed-up for attention operations during on-device inference. MobileI2V enables, for the first time, fast 720p image-to-video generation on mobile devices, with quality comparable to existing models. Under one-step conditions, the generation speed of each frame of 720p video is less than 100 ms. Our code is available at: https://github.com/hustvl/MobileI2V.
comment : Our Demo and code:https://github.com/hustvl/MobileI2V
☆ EvRainDrop: HyperGraph-guided Completion for Effective Frame and Event Stream Aggregation
Futian Wang, Fan Zhang, Xiao Wang, Mengqi Wang, Dexing Huang, Jin Tang
Event cameras produce asynchronous event streams that are spatially sparse yet temporally dense. Mainstream event representation learning algorithms typically use event frames, voxels, or tensors as input. Although these approaches have achieved notable progress, they struggle to address the undersampling problem caused by spatial sparsity. In this paper, we propose a novel hypergraph-guided spatio-temporal event stream completion mechanism, which connects event tokens across different times and spatial locations via hypergraphs and leverages contextual information message passing to complete these sparse events. The proposed method can flexibly incorporate RGB tokens as nodes in the hypergraph within this completion framework, enabling multi-modal hypergraph-based information completion. Subsequently, we aggregate hypergraph node information across different time steps through self-attention, enabling effective learning and fusion of multi-modal features. Extensive experiments on both single- and multi-label event classification tasks fully validated the effectiveness of our proposed framework. The source code of this paper will be released on https://github.com/Event-AHU/EvRainDrop.
☆ From Observation to Action: Latent Action-based Primitive Segmentation for VLA Pre-train ing in Industrial Settings
Jiajie Zhang, Sören Schwertfeger, Alexander Kleiner
We present a novel unsupervised framework to unlock vast unlabeled human demonstration data from continuous industrial video streams for Vision-Language-Action (VLA) model pre-training. Our method first trains a lightweight motion tokenizer to encode motion dynamics, then employs an unsupervised action segmenter leveraging a novel "Latent Action Energy" metric to discover and segment semantically coherent action primitives. The pipeline outputs both segmented video clips and their corresponding latent action sequences, providing structured data directly suitable for VLA pre-training. Evaluations on public benchmarks and a proprietary electric motor assembly dataset demonstrate effective segmentation of key tasks performed by humans at workstations. Further clustering and quantitative assessment via a Vision-Language Model confirm the semantic coherence of the discovered action primitives. To our knowledge, this is the first fully automated end-to-end system for extracting and organizing VLA pre-training data from unstructured industrial videos, offering a scalable solution for embodied AI integration in manufacturing.
comment : 10 pages, 5 figures
☆ E-M3RF: An Equivariant Multimodal 3D Re-assembly Framework
Adeela Islam, Stefano Fiorini, Manuel Lecha, Theodore Tsesmelis, Stuart James, Pietro Morerio, Alessio Del Bue
3D reassembly is a fundamental geometric problem, and in recent years it has increasingly been challenged by deep learning methods rather than classical optimization. While learning approaches have shown promising results, most still rely primarily on geometric features to assemble a whole from its parts. As a result, methods struggle when geometry alone is insufficient or ambiguous, for example, for small, eroded, or symmetric fragments. Additionally, solutions do not impose physical constraints that explicitly prevent overlapping assemblies. To address these limitations, we introduce E-M3RF, an equivariant multimodal 3D reassembly framework that takes as input the point clouds, containing both point positions and colors of fractured fragments, and predicts the transformations required to reassemble them using SE(3) flow matching. Each fragment is represented by both geometric and color features: i) 3D point positions are encoded as rotationconsistent geometric features using a rotation-equivariant encoder, ii) the colors at each 3D point are encoded with a transformer. The two feature sets are then combined to form a multimodal representation. We experimented on four datasets: two synthetic datasets, Breaking Bad and Fantastic Breaks, and two real-world cultural heritage datasets, RePAIR and Presious, demonstrating that E-M3RF on the RePAIR dataset reduces rotation error by 23.1% and translation error by 13.2%, while Chamfer Distance decreases by 18.4% compared to competing methods.
☆ SAM Guided Semantic and Motion Changed Region Mining for Remote Sensing Change Captioning
Futian Wang, Mengqi Wang, Xiao Wang, Haowen Wang, Jin Tang
Remote sensing change captioning is an emerging and popular research task that aims to describe, in natural language, the content of interest that has changed between two remote sensing images captured at different times. Existing methods typically employ CNNs/Transformers to extract visual representations from the given images or incorporate auxiliary tasks to enhance the final results, with weak region awareness and limited temporal alignment. To address these issues, this paper explores the use of the SAM (Segment Anything Model) foundation model to extract region-level representations and inject region-of-interest knowledge into the captioning framework. Specifically, we employ a CNN/Transformer model to extract global-level vision features, leverage the SAM foundation model to delineate semantic- and motion-level change regions, and utilize a specially constructed knowledge graph to provide information about objects of interest. These heterogeneous sources of information are then fused via cross-attention, and a Transformer decoder is used to generate the final natural language description of the observed changes. Extensive experimental results demonstrate that our method achieves state-of-the-art performance across multiple widely used benchmark datasets. The source code of this paper will be released on https://github.com/Event-AHU/SAM_ChangeCaptioning
☆ DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models
Mingue Park, Prin Phunyaphibarn, Phillip Y. Lee, Minhyuk Sung
We introduce DiverseVAR, a framework that enhances the diversity of text-conditioned visual autoregressive models (VAR) at test time without requiring retraining, fine-tuning, or substantial computational overhead. While VAR models have recently emerged as strong competitors to diffusion and flow models for image generation, they suffer from a critical limitation in diversity, often producing nearly identical images even for simple prompts. This issue has largely gone unnoticed amid the predominant focus on image quality. We address this limitation at test time in two stages. First, inspired by diversity enhancement techniques in diffusion models, we propose injecting noise into the text embedding. This introduces a trade-off between diversity and image quality: as diversity increases, the image quality sharply declines. To preserve quality, we propose scale-travel: a novel latent refinement technique inspired by time-travel strategies in diffusion models. Specifically, we use a multi-scale autoencoder to extract coarse-scale tokens that enable us to resume generation at intermediate stages. Extensive experiments show that combining text-embedding noise injection with our scale-travel refinement significantly enhances diversity while minimizing image-quality degradation, achieving a new Pareto frontier in the diversity-quality trade-off.
☆ Do Reasoning Vision -Language Models Inversely Scale in Test-Time Compute? A Distractor-centric Empirical Analysis
Jiyun Bae, Hyunjong Ok, Sangwoo Mo, Jaeho Lee
How does irrelevant information (i.e., distractors) affect test-time scaling in vision-language models (VLMs)? Prior studies on language models have reported an inverse scaling effect, where textual distractors lead to longer but less effective reasoning. To investigate whether similar phenomena occur in multimodal settings, we introduce Idis (Images with distractors), a visual question-answering dataset that systematically varies distractors along semantic, numerical, and spatial dimensions. Our analyses reveal that visual distractors differ fundamentally from textual ones: although inverse scaling persists, adding visual distractors reduces accuracy without increasing reasoning length. We further show that tracking attribute counts within reasoning traces provides key insights into how distractors, reasoning length, and accuracy interact. Finally, we demonstrate that these trends extend to established visual bias benchmarks such as Waterbirds, and we propose a simple prompting strategy to mitigate bias-driven predictions in reasoning models.
comment : preprint
☆ Monet: Reasoning in Latent Visual Space Beyond Images and Language
Qixun Wang, Yang Shi, Yifei Wang, Yuanxing Zhang, Pengfei Wan, Kun Gai, Xianghua Ying, Yisen Wang
"Thinking with images" has emerged as an effective paradigm for advancing visual reasoning, extending beyond text-only chains of thought by injecting visual evidence into intermediate reasoning steps. However, existing methods fall short of human-like abstract visual thinking, as their flexibility is fundamentally limited by external tools. In this work, we introduce Monet, a training framework that enables multimodal large language models (MLLMs) to reason directly within the latent visual space by generating continuous embeddings that function as intermediate visual thoughts. We identify two core challenges in training MLLMs for latent visual reasoning: high computational cost in latent-vision alignment and insufficient supervision over latent embeddings, and address them with a three-stage distillation-based supervised fine-tuning (SFT) pipeline. We further reveal a limitation of applying GRPO to latent reasoning: it primarily enhances text-based reasoning rather than latent reasoning. To overcome this, we propose VLPO (Visual-latent Policy Optimization), a reinforcement learning method that explicitly incorporates latent embeddings into policy gradient updates. To support SFT, we construct Monet-SFT-125K, a high-quality text-image interleaved CoT dataset containing 125K real-world, chart, OCR, and geometry CoTs. Our model, Monet-7B, shows consistent gains across real-world perception and reasoning benchmarks and exhibits strong out-of-distribution generalization on challenging abstract visual reasoning tasks. We also empirically analyze the role of each training component and discuss our early unsuccessful attempts, providing insights for future developments in visual latent reasoning. Our model, data, and code are available at https://github.com/NOVAglow646/Monet.
☆ Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuning
Xin Gu, Haoji Zhang, Qihang Fan, Jingxuan Niu, Zhipeng Zhang, Libo Zhang, Guang Chen, Fan Chen, Longyin Wen, Sijie Zhu
Spatio-temporal video grounding (STVG) requires localizing a target object in untrimmed videos both temporally and spatially from natural language descriptions. Despite their strong language understanding, multimodal large language models (MLLMs) underperform on STVG due to misaligned training objectives and weak fine-grained region-word alignment in standard visual encoders. To address this, we propose STVG-o1, the first framework that enables off-the-shelf MLLMs to achieve state-of-the-art STVG performance without any architectural modifications. Our method introduces a bounding-box chain-of-thought mechanism that explicitly reasons about spatio-temporal locations in an intermediate step before producing the final prediction. We further design a multi-dimensional reinforcement reward function consisting of format, consistency, temporal, spatial, and think rewards, which provides geometry-aware supervision through reinforcement fine-tuning. Evaluated on HCSTVG-v1/v2 and VidSTG, STVG-o1 sets new state-of-the-art results on HCSTVG, outperforming the best task-specific method by 7.3\% m\_tIoU on HCSTVG-v1, matching specialized models on VidSTG, and surpassing all existing MLLM-based approaches by large margins. It also demonstrates strong open-vocabulary generalization across datasets, establishing MLLMs as viable and powerful backbones for precise spatio-temporal grounding. Our code and models will be released.
☆ Endo-G$^{2}$T: Geometry-Guided & Temporally Aware Time-Embedded 4DGS For Endoscopic Scenes
Yangle Liu, Fengze Li, Kan Liu, Jieming Ma
Endoscopic (endo) video exhibits strong view-dependent effects such as specularities, wet reflections, and occlusions. Pure photometric supervision misaligns with geometry and triggers early geometric drift, where erroneous shapes are reinforced during densification and become hard to correct. We ask how to anchor geometry early for 4D Gaussian splatting (4DGS) while maintaining temporal consistency and efficiency in dynamic endoscopic scenes. Thus, we present Endo-G$^{2}$T, a geometry-guided and temporally aware training scheme for time-embedded 4DGS. First, geo-guided prior distillation converts confidence-gated monocular depth into supervision with scale-invariant depth and depth-gradient losses, using a warm-up-to-cap schedule to inject priors softly and avoid early overfitting. Second, a time-embedded Gaussian field represents dynamics in XYZT with a rotor-like rotation parameterization, yielding temporally coherent geometry with lightweight regularization that favors smooth motion and crisp opacity boundaries. Third, keyframe-constrained streaming improves efficiency and long-horizon stability through keyframe-focused optimization under a max-points budget, while non-keyframes advance with lightweight updates. Across EndoNeRF and StereoMIS-P1 datasets, Endo-G$^{2}$T achieves state-of-the-art results among monocular reconstruction baselines.
☆ PFF-Net: Patch Feature Fitting for Point Cloud Normal Estimation
Qing Li, Huifang Feng, Kanle Shi, Yue Gao, Yi Fang, Yu-Shen Liu, Zhizhong Han
Estimating the normal of a point requires constructing a local patch to provide center-surrounding context, but determining the appropriate neighborhood size is difficult when dealing with different data or geometries. Existing methods commonly employ various parameter-heavy strategies to extract a full feature description from the input patch. However, they still have difficulties in accurately and efficiently predicting normals for various point clouds. In this work, we present a new idea of feature extraction for robust normal estimation of point clouds. We use the fusion of multi-scale features from different neighborhood sizes to address the issue of selecting reasonable patch sizes for various data or geometries. We seek to model a patch feature fitting (PFF) based on multi-scale features to approximate the optimal geometric description for normal estimation and implement the approximation process via multi-scale feature aggregation and cross-scale feature compensation. The feature aggregation module progressively aggregates the patch features of different scales to the center of the patch and shrinks the patch size by removing points far from the center. It not only enables the network to precisely capture the structure characteristic in a wide range, but also describes highly detailed geometries. The feature compensation module ensures the reusability of features from earlier layers of large scales and reveals associated information in different patch sizes. Our approximation strategy based on aggregating the features of multiple scales enables the model to achieve scale adaptation of varying local patches and deliver the optimal feature description. Extensive experiments demonstrate that our method achieves state-of-the-art performance on both synthetic and real-world datasets with fewer network parameters and running time.
comment : Accepted by TVCG
☆ BanglaMM-Disaster: A Multimodal Transformer -Based Deep Learning Framework for Multiclass Disaster Classification in Bangla SP
Ariful Islam, Md Rifat Hossen, Md. Mahmudul Arif, Abdullah Al Noman, Md Arifur Rahman
Natural disasters remain a major challenge for Bangladesh, so real-time monitoring and quick response systems are essential. In this study, we present BanglaMM-Disaster, an end-to-end deep learning-based multimodal framework for disaster classification in Bangla, using both textual and visual data from social media. We constructed a new dataset of 5,037 Bangla social media posts, each consisting of a caption and a corresponding image, annotated into one of nine disaster-related categories. The proposed model integrates transformer-based text encoders, including BanglaBERT, mBERT, and XLM-RoBERTa, with CNN backbones such as ResNet50, DenseNet169, and MobileNetV2, to process the two modalities. Using early fusion, the best model achieves 83.76% accuracy. This surpasses the best text-only baseline by 3.84% and the image-only baseline by 16.91%. Our analysis also shows reduced misclassification across all classes, with noticeable improvements for ambiguous examples. This work fills a key gap in Bangla multimodal disaster analysis and demonstrates the benefits of combining multiple data types for real-time disaster response in low-resource settings.
comment : Presented at the 2025 IEEE International Conference on Signal Processing, Information, Communication and Systems (SPICSCON), November 21-22, 2025, University of Rajshahi, Bangladesh. 6 pages, 9 disaster classes, multimodal dataset with 5,037 samples
☆ SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understanding
Tae-Min Choi, Tae Kyeong Jeong, Garam Kim, Jaemin Lee, Yeongyoon Koh, In Cheul Choi, Jae-Ho Chung, Jong Woong Park, Juyoun Park
Recent advances in multimodal large language models (LLMs) have highlighted their potential for medical and surgical applications. However, existing surgical datasets predominantly adopt a Visual Question Answering (VQA) format with heterogeneous taxonomies and lack support for pixel-level segmentation, limiting consistent evaluation and applicability. We present SurgMLLMBench, a unified multimodal benchmark explicitly designed for developing and evaluating interactive multimodal LLMs for surgical scene understanding, including the newly collected Micro-surgical Artificial Vascular anastomosIS (MAVIS) dataset. It integrates pixel-level instrument segmentation masks and structured VQA annotations across laparoscopic, robot-assisted, and micro-surgical domains under a unified taxonomy, enabling comprehensive evaluation beyond traditional VQA tasks and richer visual-conversational interactions. Extensive baseline experiments show that a single model trained on SurgMLLMBench achieves consistent performance across domains and generalizes effectively to unseen datasets. SurgMLLMBench will be publicly released as a robust resource to advance multimodal surgical AI research, supporting reproducible evaluation and development of interactive surgical reasoning models.
comment : 10 pages, 5 figures
☆ Hybrid SIFT-SNN for Efficient Anomaly Detection of Traffic Flow-Control Infrastructure
Munish Rathee, Boris Bačić, Maryam Doborjeh
This paper presents the SIFT-SNN framework, a low-latency neuromorphic signal-processing pipeline for real-time detection of structural anomalies in transport infrastructure. The proposed approach integrates Scale-Invariant Feature Transform (SIFT) for spatial feature encoding with a latency-driven spike conversion layer and a Leaky Integrate-and-Fire (LIF) Spiking Neural Network (SNN) for classification. The Auckland Harbour Bridge dataset is recorded under various weather and lighting conditions, comprising 6,000 labelled frames that include both real and synthetically augmented unsafe cases. The presented system achieves a classification accuracy of 92.3% (+- 0.8%) with a per-frame inference time of 9.5 ms. Achieved sub-10 millisecond latency, combined with sparse spike activity (8.1%), enables real-time, low-power edge deployment. Unlike conventional CNN-based approaches, the hybrid SIFT-SNN pipeline explicitly preserves spatial feature grounding, enhances interpretability, supports transparent decision-making, and operates efficiently on embedded hardware. Although synthetic augmentation improved robustness, generalisation to unseen field conditions remains to be validated. The SIFT-SNN framework is validated through a working prototype deployed on a consumer-grade system and framed as a generalisable case study in structural safety monitoring for movable concrete barriers, which, as a traffic flow-control infrastructure, is deployed in over 20 cities worldwide.
comment : 8 pages, 6 figures. This is a preprint of a paper accepted for presentation at the 2025 International Conference on Image and Vision Computing New Zealand (IVCNZ). The final version will appear in IEEE Xplore
☆ The More, the Merrier: Contrastive Fusion for Higher-Order Multimodal Alignment
Stefanos Koutoupis, Michaela Areti Zervou, Konstantinos Kontras, Maarten De Vos, Panagiotis Tsakalides, Grigorios Tsagatakis
Learning joint representations across multiple modalities remains a central challenge in multimodal machine learning. Prevailing approaches predominantly operate in pairwise settings, aligning two modalities at a time. While some recent methods aim to capture higher-order interactions among multiple modalities, they often overlook or insufficiently preserve pairwise relationships, limiting their effectiveness on single-modality tasks. In this work, we introduce Contrastive Fusion (ConFu), a framework that jointly embeds both individual modalities and their fused combinations into a unified representation space, where modalities and their fused counterparts are aligned. ConFu extends traditional pairwise contrastive objectives with an additional fused-modality contrastive term, encouraging the joint embedding of modality pairs with a third modality. This formulation enables ConFu to capture higher-order dependencies, such as XOR-like relationships, that cannot be recovered through pairwise alignment alone, while still maintaining strong pairwise correspondence. We evaluate ConFu on synthetic and real-world multimodal benchmarks, assessing its ability to exploit cross-modal complementarity, capture higher-order dependencies, and scale with increasing multimodal complexity. Across these settings, ConFu demonstrates competitive performance on retrieval and classification tasks, while supporting unified one-to-one and two-to-one retrieval within a single contrastive framework.
☆ HTTM: Head-wise Temporal Token Merging for Faster VGGT
Weitian Wang, Lukas Meiner, Rai Shubham, Cecilia De La Parra, Akash Kumar
The Visual Geometry Grounded Transformer (VGGT) marks a significant leap forward in 3D scene reconstruction, as it is the first model that directly infers all key 3D attributes (camera poses, depths, and dense geometry) jointly in one pass. However, this joint inference mechanism requires global attention layers that perform all-to-all attention computation on tokens from all views. For reconstruction of large scenes with long-sequence inputs, this causes a significant latency bottleneck. In this paper, we propose head-wise temporal merging (HTTM), a training-free 3D token merging method for accelerating VGGT. Existing merging techniques merge tokens uniformly across different attention heads, resulting in identical tokens in the layers' output, which hinders the model's representational ability. HTTM tackles this problem by merging tokens in multi-head granularity, which preserves the uniqueness of feature tokens after head concatenation. Additionally, this enables HTTM to leverage the spatial locality and temporal correspondence observed at the head level to achieve higher merging ratios with lower merging costs compared to existing methods. Thus, HTTM achieves up to 7x acceleration with negligible performance drops in a GPU-based inference.
☆ CaliTex: Geometry-Calibrated Attention for View-Coherent 3D Texture Generation
Chenyu Liu, Hongze Chen, Jingzhi Bao, Lingting Zhu, Runze Zhang, Weikai Chen, Zeyu Hu, Yingda Yin, Keyang Luo, Xin Wang
Despite major advances brought by diffusion-based models, current 3D texture generation systems remain hindered by cross-view inconsistency -- textures that appear convincing from one viewpoint often fail to align across others. We find that this issue arises from attention ambiguity, where unstructured full attention is applied indiscriminately across tokens and modalities, causing geometric confusion and unstable appearance-structure coupling. To address this, we introduce CaliTex, a framework of geometry-calibrated attention that explicitly aligns attention with 3D structure. It introduces two modules: Part-Aligned Attention that enforces spatial alignment across semantically matched parts, and Condition-Routed Attention which routes appearance information through geometry-conditioned pathways to maintain spatial fidelity. Coupled with a two-stage diffusion transformer, CaliTex makes geometric coherence an inherent behavior of the network rather than a byproduct of optimization. Empirically, CaliTex produces seamless and view-consistent textures and outperforms both open-source and commercial baselines.
☆ PathMamba: A Hybrid Mamba- Transformer for Topologically Coherent Road Segmentation in Satellite Imagery
Jules Decaestecker, Nicolas Vigne
Achieving both high accuracy and topological continuity in road segmentation from satellite imagery is a critical goal for applications ranging from urban planning to disaster response. State-of-the-art methods often rely on Vision Transformers, which excel at capturing global context, yet their quadratic complexity is a significant barrier to efficient deployment, particularly for on-board processing in resource-constrained platforms. In contrast, emerging State Space Models like Mamba offer linear-time efficiency and are inherently suited to modeling long, continuous structures. We posit that these architectures have complementary strengths. To this end, we introduce PathMamba, a novel hybrid architecture that integrates Mamba's sequential modeling with the Transformer's global reasoning. Our design strategically uses Mamba blocks to trace the continuous nature of road networks, preserving topological structure, while integrating Transformer blocks to refine features with global context. This approach yields topologically superior segmentation maps without the prohibitive scaling costs of pure attention-based models. Our experiments on the DeepGlobe Road Extraction and Massachusetts Roads datasets demonstrate that PathMamba sets a new state-of-the-art. Notably, it significantly improves topological continuity, as measured by the APLS metric, setting a new benchmark while remaining computationally competitive.
comment : 11 pages, 5 figures
☆ Co-Training Vision Language Models for Remote Sensing Multi-task Learning
Qingyun Li, Shuran Ma, Junwei Luo, Yi Yu, Yue Zhou, Fengxiang Wang, Xudong Lu, Xiaoxing Wang, Xin He, Yushi Chen, Xue Yang, Junchi Yan
With Transformers achieving outstanding performance on individual remote sensing (RS) tasks, we are now approaching the realization of a unified model that excels across multiple tasks through multi-task learning (MTL). Compared to single-task approaches, MTL methods offer improved generalization, enhanced scalability, and greater practical applicability. Recently, vision language models (VLMs) have achieved promising results in RS image understanding, grounding, and ultra-high-resolution (UHR) image reasoning, respectively. Moreover, the unified text-based interface demonstrates significant potential for MTL. Hence, in this work, we present RSCoVLM, a simple yet flexible VLM baseline for RS MTL. Firstly, we create the data curation engine, including data acquisition, offline processing and integrating, as well as online loading and weighting. This data engine effectively addresses complex RS data enviroment and generates flexible vision-language conversations. Furthermore, we propose a unified dynamic-resolution strategy to address the diverse image scales inherent in RS imagery. For UHR images, we introduce the Zoom-in Chain mechanism together with its corresponding dataset, LRS-VQA-Zoom. The strategies are flexible and effectively mitigate the computational burdens. Additionally, we significantly enhance the model's object detection capability and propose a novel evaluation protocol that ensures fair comparison between VLMs and conventional detection models. Extensive experiments demonstrate that RSCoVLM achieves state-of-the-art performance across diverse tasks, outperforming existing RS VLMs and even rivaling specialized expert models. All the training and evaluating tools, model weights, and datasets have been fully open-sourced to support reproducibility. We expect that this baseline will promote further progress toward general-purpose RS models.
comment : 14 pages, 6 figures
☆ Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale
Yicheng Zhong, Peiji Yang, Zhisheng Wang
Recent advances in Large Language Models (LLMs) have transformed text-to-speech (TTS) synthesis, inspiring autoregressive frameworks that represent speech as sequences of discrete codec tokens. Among them, single-codebook TTS LLMs have emerged as compact and streamable architectures that jointly model semantic and acoustic integration. However, despite their efficiency, these models often exhibit unstable prosody, speaker drift, and degraded naturalness. To address these issues, we propose a multi-reward Group Relative Policy Optimization (GRPO) framework that directly optimizes the token generation policy of single-codebook TTS LLMs. Beyond standard intelligibility and speaker similarity objectives, our design integrates three rule-based rewards: a length penalty for duration consistency, an entropy regularization reward for decoding stability, and an LLM-annotated prosody alignment reward that explicitly supervises rhythm. In this prosody reward, an external reasoning LLM predicts multiple plausible pause structures via in-context learning, providing a human-preference-aligned supervisory signal for GRPO training. To assess universality, we further attach a flow-matching (FM) decoder on top of the GRPO-optimized AR backbone and observe consistent additional gains, indicating that our reinforcement optimization enhances the intrinsic AR policy. We further conduct a scalability analysis across data sizes and model scales, revealing that the proposed method consistently enhances prosodic stability, speaker similarity, and overall speech naturalness in single-codebook TTS LLMs.
comment : 4 pages, 2 figures
☆ Unlocking Zero-shot Potential of Semi-dense Image Matching via Gaussian Splatting
Juncheng Chen, Chao Xu, Yanjun Cao
Learning-based image matching critically depends on large-scale, diverse, and geometrically accurate training data. 3D Gaussian Splatting (3DGS) enables photorealistic novel-view synthesis and thus is attractive for data generation. However, its geometric inaccuracies and biased depth rendering currently prevent robust correspondence labeling. To address this, we introduce MatchGS, the first framework designed to systematically correct and leverage 3DGS for robust, zero-shot image matching. Our approach is twofold: (1) a geometrically-faithful data generation pipeline that refines 3DGS geometry to produce highly precise correspondence labels, enabling the synthesis of a vast and diverse range of viewpoints without compromising rendering fidelity; and (2) a 2D-3D representation alignment strategy that infuses 3DGS' explicit 3D knowledge into the 2D matcher, guiding 2D semi-dense matchers to learn viewpoint-invariant 3D representations. Our generated ground-truth correspondences reduce the epipolar error by up to 40 times compared to existing datasets, enable supervision under extreme viewpoint changes, and provide self-supervisory signals through Gaussian attributes. Consequently, state-of-the-art matchers trained solely on our data achieve significant zero-shot performance gains on public benchmarks, with improvements of up to 17.7%. Our work demonstrates that with proper geometric refinement, 3DGS can serve as a scalable, high-fidelity, and structurally-rich data source, paving the way for a new generation of robust zero-shot image matchers.
☆ LaGen: Towards Autoregressive LiDAR Scene Generation
Sizhuo Zhou, Xiaosong Jia, Fanrui Zhang, Junjie Li, Juyong Zhang, Yukang Feng, Jianwen Sun, Songbur Wong, Junqi You, Junchi Yan
Generative world models for autonomous driving (AD) have become a trending topic. Unlike the widely studied image modality, in this work we explore generative world models for LiDAR data. Existing generation methods for LiDAR data only support single frame generation, while existing prediction approaches require multiple frames of historical input and can only deterministically predict multiple frames at once, lacking interactivity. Both paradigms fail to support long-horizon interactive generation. To this end, we introduce LaGen, which to the best of our knowledge is the first framework capable of frame-by-frame autoregressive generation of long-horizon LiDAR scenes. LaGen is able to take a single-frame LiDAR input as a starting point and effectively utilize bounding box information as conditions to generate high-fidelity 4D scene point clouds. In addition, we introduce a scene decoupling estimation module to enhance the model's interactive generation capability for object-level content, as well as a noise modulation module to mitigate error accumulation during long-horizon generation. We construct a protocol based on nuScenes for evaluating long-horizon LiDAR scene generation. Experimental results comprehensively demonstrate LaGen outperforms state-of-the-art LiDAR generation and prediction models, especially on the later frames.
☆ AVFakeBench: A Comprehensive Audio-Video Forgery Detection Benchmark for AV-LMMs
Shuhan Xia, Peipei Li, Xuannan Liu, Dongsen Zhang, Xinyu Guo, Zekun Li
The threat of Audio-Video (AV) forgery is rapidly evolving beyond human-centric deepfakes to include more diverse manipulations across complex natural scenes. However, existing benchmarks are still confined to DeepFake-based forgeries and single-granularity annotations, thus failing to capture the diversity and complexity of real-world forgery scenarios. To address this, we introduce AVFakeBench, the first comprehensive audio-video forgery detection benchmark that spans rich forgery semantics across both human subject and general subject. AVFakeBench comprises 12K carefully curated audio-video questions, covering seven forgery types and four levels of annotations. To ensure high-quality and diverse forgeries, we propose a multi-stage hybrid forgery framework that integrates proprietary models for task planning with expert generative models for precise manipulation. The benchmark establishes a multi-task evaluation framework covering binary judgment, forgery types classification, forgery detail selection, and explanatory reasoning. We evaluate 11 Audio-Video Large Language Models (AV-LMMs) and 2 prevalent detection methods on AVFakeBench, demonstrating the potential of AV-LMMs as emerging forgery detectors while revealing their notable weaknesses in fine-grained perception and reasoning.
☆ Shift-Equivariant Complex-Valued Convolutional Neural Networks WACV 2026
Quentin Gabot, Teck-Yian Lim, Jérémy Fix, Joana Frontera-Pons, Chengfang Ren, Jean-Philippe Ovarlez
Convolutional neural networks have shown remarkable performance in recent years on various computer vision problems. However, the traditional convolutional neural network architecture lacks a critical property: shift equivariance and invariance, broken by downsampling and upsampling operations. Although data augmentation techniques can help the model learn the latter property empirically, a consistent and systematic way to achieve this goal is by designing downsampling and upsampling layers that theoretically guarantee these properties by construction. Adaptive Polyphase Sampling (APS) introduced the cornerstone for shift invariance, later extended to shift equivariance with Learnable Polyphase up/downsampling (LPS) applied to real-valued neural networks. In this paper, we extend the work on LPS to complex-valued neural networks both from a theoretical perspective and with a novel building block of a projection layer from $\mathbb{C}$ to $\mathbb{R}$ before the Gumbel Softmax. We finally evaluate this extension on several computer vision problems, specifically for either the invariance property in classification tasks or the equivariance property in both reconstruction and semantic segmentation problems, using polarimetric Synthetic Aperture Radar images.
comment : Accepted to WACV 2026
☆ FIELDS: Face reconstruction with accurate Inference of Expression using Learning with Direct Super vision
Chen Ling, Henglin Shi, Hedvig Kjellström
Facial expressions convey the bulk of emotional information in human communication, yet existing 3D face reconstruction methods often miss subtle affective details due to reliance on 2D supervision and lack of 3D ground truth. We propose FIELDS (Face reconstruction with accurate Inference of Expression using Learning with Direct Supervision) to address these limitations by extending self-supervised 2D image consistency cues with direct 3D expression parameter supervision and an auxiliary emotion recognition branch. Our encoder is guided by authentic expression parameters from spontaneous 4D facial scans, while an intensity-aware emotion loss encourages the 3D expression parameters to capture genuine emotion content without exaggeration. This dual-supervision strategy bridges the 2D/3D domain gap and mitigates expression-intensity bias, yielding high-fidelity 3D reconstructions that preserve subtle emotional cues. From a single image, FIELDS produces emotion-rich face models with highly realistic expressions, significantly improving in-the-wild facial expression recognition performance without sacrificing naturalness.
☆ 3-Tracer: A Tri-level Temporal-Aware Framework for Audio Forgery Detection and Localization
Shuhan Xia, Xuannan Liu, Xing Cui, Peipei Li
Recently, partial audio forgery has emerged as a new form of audio manipulation. Attackers selectively modify partial but semantically critical frames while preserving the overall perceptual authenticity, making such forgeries particularly difficult to detect. Existing methods focus on independently detecting whether a single frame is forged, lacking the hierarchical structure to capture both transient and sustained anomalies across different temporal levels. To address these limitations, We identify three key levels relevant to partial audio forgery detection and present T3-Tracer, the first framework that jointly analyzes audio at the frame, segment, and audio levels to comprehensively detect forgery traces. T3-Tracer consists of two complementary core modules: the Frame-Audio Feature Aggregation Module (FA-FAM) and the Segment-level Multi-Scale Discrepancy-Aware Module (SMDAM). FA-FAM is designed to detect the authenticity of each audio frame. It combines both frame-level and audio-level temporal information to detect intra-frame forgery cues and global semantic inconsistencies. To further refine and correct frame detection, we introduce SMDAM to detect forgery boundaries at the segment level. It adopts a dual-branch architecture that jointly models frame features and inter-frame differences across multi-scale temporal windows, effectively identifying abrupt anomalies that appeared on the forged boundaries. Extensive experiments conducted on three challenging datasets demonstrate that our approach achieves state-of-the-art performance.
☆ From Diffusion to One-Step Generation: A Comparative Study of Flow-Based Models with Application to Image Inpainting
Umang Agarwal, Rudraksh Sangore, Sumit Laddha
We present a comprehensive comparative study of three generative modeling paradigms: Denoising Diffusion Probabilistic Models (DDPM), Conditional Flow Matching (CFM), and MeanFlow. While DDPM and CFM require iterative sampling, MeanFlow enables direct one-step generation by modeling the average velocity over time intervals. We implement all three methods using a unified TinyUNet architecture (<1.5M parameters) on CIFAR-10, demonstrating that CFM achieves an FID of 24.15 with 50 steps, significantly outperforming DDPM (FID 402.98). MeanFlow achieves FID 29.15 with single-step sampling -- a 50X reduction in inference time. We further extend CFM to image inpainting, implementing mask-guided sampling with four mask types (center, random bbox, irregular, half). Our fine-tuned inpainting model achieves substantial improvements: PSNR increases from 4.95 to 8.57 dB on center masks (+73%), and SSIM improves from 0.289 to 0.418 (+45%), demonstrating the effectiveness of inpainting-aware training.
☆ Towards an Effective Action-Region Tracking Framework for Fine-grained Video Action Recognition
Baoli Sun, Yihan Wang, Xinzhu Ma, Zhihui Wang, Kun Lu, Zhiyong Wang
Fine-grained action recognition (FGAR) aims to identify subtle and distinctive differences among fine-grained action categories. However, current recognition methods often capture coarse-grained motion patterns but struggle to identify subtle details in local regions evolving over time. In this work, we introduce the Action-Region Tracking (ART) framework, a novel solution leveraging a query-response mechanism to discover and track the dynamics of distinctive local details, enabling effective distinction of similar actions. Specifically, we propose a region-specific semantic activation module that employs discriminative and text-constrained semantics as queries to capture the most action-related region responses in each video frame, facilitating interaction among spatial and temporal dimensions with corresponding video features. The captured region responses are organized into action tracklets, which characterize region-based action dynamics by linking related responses across video frames in a coherent sequence. The text-constrained queries encode nuanced semantic representations derived from textual descriptions of action labels extracted by language branches within Visual Language Models (VLMs). To optimize the action tracklets, we design a multi-level tracklet contrastive constraint among region responses at spatial and temporal levels, enabling effective discrimination within each frame and correlation between adjacent frames. Additionally, a task-specific fine-tuning mechanism refines textual semantics such that semantic representations encoded by VLMs are preserved while optimized for task preferences. Comprehensive experiments on widely used action recognition benchmarks demonstrate the superiority to previous state-of-the-art baselines.
☆ BotaCLIP: Contrastive Learning for Botany-Aware Representation of Earth Observation Data
Selene Cerna, Sara Si-Moussi, Wilfried Thuiller, Hadrien Hendrikx, Vincent Miele
Foundation models have demonstrated a remarkable ability to learn rich, transferable representations across diverse modalities such as images, text, and audio. In modern machine learning pipelines, these representations often replace raw data as the primary input for downstream tasks. In this paper, we address the challenge of adapting a pre-trained foundation model to inject domain-specific knowledge, without retraining from scratch or incurring significant computational costs. To this end, we introduce BotaCLIP, a lightweight multimodal contrastive framework that adapts a pre-trained Earth Observation foundation model (DOFA) by aligning high-resolution aerial imagery with botanical relevés. Unlike generic embeddings, BotaCLIP internalizes ecological structure through contrastive learning with a regularization strategy that mitigates catastrophic forgetting. Once trained, the resulting embeddings serve as transferable representations for downstream predictors. Motivated by real-world applications in biodiversity modeling, we evaluated BotaCLIP representations in three ecological tasks: plant presence prediction, butterfly occurrence modeling, and soil trophic group abundance estimation. The results showed consistent improvements over those derived from DOFA and supervised baselines. More broadly, this work illustrates how domain-aware adaptation of foundation models can inject expert knowledge into data-scarce settings, enabling frugal representation learning.
☆ You Can Trust Your Clustering Model: A Parameter-free Self-Boosting Plug-in for Deep Clustering NeurIPS 2025
Hanyang Li, Yuheng Jia, Hui Liu, Junhui Hou
Recent deep clustering models have produced impressive clustering performance. However, a common issue with existing methods is the disparity between global and local feature structures. While local structures typically show strong consistency and compactness within class samples, global features often present intertwined boundaries and poorly separated clusters. Motivated by this observation, we propose DCBoost, a parameter-free plug-in designed to enhance the global feature structures of current deep clustering models. By harnessing reliable local structural cues, our method aims to elevate clustering performance effectively. Specifically, we first identify high-confidence samples through adaptive $k$-nearest neighbors-based consistency filtering, aiming to select a sufficient number of samples with high label reliability to serve as trustworthy anchors for self-supervision. Subsequently, these samples are utilized to compute a discriminative loss, which promotes both intra-class compactness and inter-class separability, to guide network optimization. Extensive experiments across various benchmark datasets showcase that our DCBoost significantly improves the clustering performance of diverse existing deep clustering models. Notably, our method improves the performance of current state-of-the-art baselines (e.g., ProPos) by more than 3% and amplifies the silhouette coefficient by over $7\times$. Code is available at .
comment : The paper is accepted by NeurIPS 2025
☆ When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision -Language-Action Models
Hui Lu, Yi Yu, Yiming Yang, Chenyu Yi, Qixin Zhang, Bingquan Shen, Alex C. Kot, Xudong Jiang
Vision-Language-Action (VLA) models are vulnerable to adversarial attacks, yet universal and transferable attacks remain underexplored, as most existing patches overfit to a single model and fail in black-box settings. To address this gap, we present a systematic study of universal, transferable adversarial patches against VLA-driven robots under unknown architectures, finetuned variants, and sim-to-real shifts. We introduce UPA-RFAS (Universal Patch Attack via Robust Feature, Attention, and Semantics), a unified framework that learns a single physical patch in a shared feature space while promoting cross-model transfer. UPA-RFAS combines (i) a feature-space objective with an $\ell_1$ deviation prior and repulsive InfoNCE loss to induce transferable representation shifts, (ii) a robustness-augmented two-phase min-max procedure where an inner loop learns invisible sample-wise perturbations and an outer loop optimizes the universal patch against this hardened neighborhood, and (iii) two VLA-specific losses: Patch Attention Dominance to hijack text$\to$vision attention and Patch Semantic Misalignment to induce image-text mismatch without labels. Experiments across diverse VLA models, manipulation suites, and physical executions show that UPA-RFAS consistently transfers across models, tasks, and viewpoints, exposing a practical patch-based attack surface and establishing a strong baseline for future defenses.
☆ Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision -Language Understanding
Yutao Tang, Cheng Zhao, Gaurav Mittal, Rohith Kukkala, Rama Chellappa, Cheng Peng, Mei Chen
Recent advances in 3D vision-language models (VLMs) highlight a strong potential for 3D scene understanding and reasoning. However, effectively tokenizing 3D scenes into holistic scene tokens, and leveraging these tokens across diverse 3D understanding tasks, remain highly challenging. We present NDTokenizer3D, a generalist 3D VLM that performs a wide range of 3D scene understanding tasks while naturally supporting human interactions, thereby bridging language-level reasoning with 3D spatial understanding. The core of our approach is a novel three-stage scene tokenization pipeline built upon a Multi-Scale Normal Distributions Transform (NDT) representation, paired with a Multi-Scale NDT Decoder (MSDec). Specifically, NDTokenizer3D first constructs a multi-scale NDT representation from raw high-resolution point clouds, preserving both global context and fine-grained geometric details. Next, the MSDec progressively fuses cross-scale NDT features, producing holistic scene tokens consumable by LLM endpoints. Beyond tokenization, MSDec is repurposed as a general interface for human-interactive prompting (points, boxes, masks) and segmentation-mask decoding, unifying diverse 3D scene understanding tasks within a single architecture. With this compact and unified design, NDTokenizer3D offers a fine-grained, general-purpose 3D VLM, achieving remarkable improvements in 3D Referring Segmentation, 3D Visual Question Answering, and 3D Dense Captioning.
☆ AnchorOPT: Towards Optimizing Dynamic Anchors for Adaptive Prompt Learning
Zheng Li, Yibing Song, Xin Zhang, Lei Luo, Xiang Li, Jian Yang
Existing prompt learning methods, which are built upon CLIP models, leverage textual tokens as anchors to guide the learnable soft tokens. This guidance improves CLIP generalizations. However, these anchors-static in both value and position-lack cross-task and stage-adaptive flexibility. To address this limitation, we propose AnchorOPT, a dynamic anchor-based prompt learning framework. Specifically, AnchorOPT introduces dynamism in two key dimensions: (i) anchor values eschew handcrafted explicit textual tokens (e.g., "shape", "color"), instead learning dynamically from task-specific data; and (ii) the positional relationship between anchor and soft tokens is no longer fixed but adaptively optimized via a learnable position matrix conditioned on the training stage and task context. Training occurs in two stages: we first learn the anchor tokens, then freeze and transfer them to the second stage for optimization of soft tokens and the position matrix. Extensive experiments demonstrate that using only a simple learnable anchor and position matrix achieves performance comparable to or exceeding some methods incorporating additional learnable modules or regularization techniques. As a plug-and-play module, AnchorOPT integrates seamlessly into existing frameworks, yielding consistent performance gains across diverse datasets. Code is publicly available at https://github.com/zhengli97/ATPrompt.
comment : Technical Report
☆ Progress by Pieces: Test-Time Scaling for Autoregressive Image Generation
Joonhyung Park, Hyeongwon Jang, Joowon Kim, Eunho Yang
Recent visual autoregressive (AR) models have shown promising capabilities in text-to-image generation, operating in a manner similar to large language models. While test-time computation scaling has brought remarkable success in enabling reasoning-enhanced outputs for challenging natural language tasks, its adaptation to visual AR models remains unexplored and poses unique challenges. Naively applying test-time scaling strategies such as Best-of-N can be suboptimal: they consume full-length computation on erroneous generation trajectories, while the raster-scan decoding scheme lacks a blueprint of the entire canvas, limiting scaling benefits as only a few prompt-aligned candidates are generated. To address these, we introduce GridAR, a test-time scaling framework designed to elicit the best possible results from visual AR models. GridAR employs a grid-partitioned progressive generation scheme in which multiple partial candidates for the same position are generated within a canvas, infeasible ones are pruned early, and viable ones are fixed as anchors to guide subsequent decoding. Coupled with this, we present a layout-specified prompt reformulation strategy that inspects partial views to infer a feasible layout for satisfying the prompt. The reformulated prompt then guides subsequent image generation to mitigate the blueprint deficiency. Together, GridAR achieves higher-quality results under limited test-time scaling: with N=4, it even outperforms Best-of-N (N=8) by 14.4% on T2I-CompBench++ while reducing cost by 25.6%. It also generalizes to autoregressive image editing, showing comparable edit quality and a 13.9% gain in semantic preservation on PIE-Bench over larger-N baselines.
comment : Project page: https://grid-ar.github.io/
☆ LLaVA -UHD v3: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs
Shichu Sun, Yichen Zhang, Haolin Song, Zonghao Guo, Chi Chen, Yidan Zhang, Yuan Yao, Zhiyuan Liu, Maosong Sun
Visual encoding followed by token condensing has become the standard architectural paradigm in multi-modal large language models (MLLMs). Many recent MLLMs increasingly favor global native- resolution visual encoding over slice-based methods. To investigate this trend, we systematically compare their behavior on vision-language understanding and attention patterns, revealing that global encoding enhances overall capability but at the expense of greater computational overhead. To address this issue, we present LLaVA-UHD v3, an MLLM centered upon our proposed Progressive Visual Compression (PVC) method, which can be seamlessly integrated into standard Vision Transformer (ViT) to enable efficient native-resolution encoding. The PVC approach consists of two key modules: (i) refined patch embedding, which supports flexible patch-size scaling for fine-grained visual model- ing, (ii) windowed token compression, hierarchically deployed across ViT layers to progressively aggregate local token representations. Jointly modulated by these two modules, a widely pretrained ViT can be reconfigured into an efficient architecture while largely preserving generality. Evaluated across extensive benchmarks, the transformed ViT, termed ViT-UHD, demonstrates competitive performance with MoonViT while reducing TTFT (time-to-first-token) by 2.4x, when developed within an identical MLLM architecture. Building upon ViT-UHD, LLaVA-UHD v3 also achieves competitive performance to Qwen2-VL, while further reducing TTFT by 1.9x. We will release all code and checkpoints to support future research on efficient MLLMs.
☆ AV-Edit: Multimodal Generative Sound Effect Editing via Audio-Visual Semantic Joint Control
Xinyue Guo, Xiaoran Yang, Lipan Zhang, Jianxuan Yang, Zhao Wang, Jian Luan
Sound effect editing-modifying audio by adding, removing, or replacing elements-remains constrained by existing approaches that rely solely on low-level signal processing or coarse text prompts, often resulting in limited flexibility and suboptimal audio quality. To address this, we propose AV-Edit, a generative sound effect editing framework that enables fine-grained editing of existing audio tracks in videos by jointly leveraging visual, audio, and text semantics. Specifically, the proposed method employs a specially designed contrastive audio-visual masking autoencoder (CAV-MAE-Edit) for multimodal pre-training, learning aligned cross-modal representations. These representations are then used to train an editorial Multimodal Diffusion Transformer (MM-DiT) capable of removing visually irrelevant sounds and generating missing audio elements consistent with video content through a correlation-based feature gating training strategy. Furthermore, we construct a dedicated video-based sound editing dataset as an evaluation benchmark. Experiments demonstrate that the proposed AV-Edit generates high-quality audio with precise modifications based on visual content, achieving state-of-the-art performance in the field of sound effect editing and exhibiting strong competitiveness in the domain of audio generation.
☆ TEAR: Temporal-aware Automated Red-teaming for Text-to-Video Models
Jiaming He, Guanyu Hou, Hongwei Li, Zhicong Huang, Kangjie Chen, Yi Yu, Wenbo Jiang, Guowen Xu, Tianwei Zhang
Text-to-Video (T2V) models are capable of synthesizing high-quality, temporally coherent dynamic video content, but the diverse generation also inherently introduces critical safety challenges. Existing safety evaluation methods,which focus on static image and text generation, are insufficient to capture the complex temporal dynamics in video generation. To address this, we propose a TEmporal-aware Automated Red-teaming framework, named TEAR, an automated framework designed to uncover safety risks specifically linked to the dynamic temporal sequencing of T2V models. TEAR employs a temporal-aware test generator optimized via a two-stage approach: initial generator training and temporal-aware online preference learning, to craft textually innocuous prompts that exploit temporal dynamics to elicit policy-violating video output. And a refine model is adopted to improve the prompt stealthiness and adversarial effectiveness cyclically. Extensive experimental evaluation demonstrates the effectiveness of TEAR across open-source and commercial T2V systems with over 80% attack success rate, a significant boost from prior best result of 57%.
☆ STAR: Smartphone-analogous Typing in Augmented Reality
Taejun Kim, Amy Karlson, Aakar Gupta, Tovi Grossman, Jason Wu, Parastoo Abtahi, Christopher Collins, Michael Glueck, Hemant Bhaskar Surale
While text entry is an essential and frequent task in Augmented Reality (AR) applications, devising an efficient and easy-to-use text entry method for AR remains an open challenge. This research presents STAR, a smartphone-analogous AR text entry technique that leverages a user's familiarity with smartphone two-thumb typing. With STAR, a user performs thumb typing on a virtual QWERTY keyboard that is overlain on the skin of their hands. During an evaluation study of STAR, participants achieved a mean typing speed of 21.9 WPM (i.e., 56% of their smartphone typing speed), and a mean error rate of 0.3% after 30 minutes of practice. We further analyze the major factors implicated in the performance gap between STAR and smartphone typing, and discuss ways this gap could be narrowed.
comment : ACM UIST 2023
☆ Referring Video Object Segmentation with Cross-Modality Proxy Queries
Baoli Sun, Xinzhu Ma, Ning Wang, Zhihui Wang, Zhiyong Wang
Referring video object segmentation (RVOS) is an emerging cross-modality task that aims to generate pixel-level maps of the target objects referred by given textual expressions. The main concept involves learning an accurate alignment of visual elements and language expressions within a semantic space. Recent approaches address cross-modality alignment through conditional queries, tracking the target object using a query-response based mechanism built upon transformer structure. However, they exhibit two limitations: (1) these conditional queries lack inter-frame dependency and variation modeling, making accurate target tracking challenging amid significant frame-to-frame variations; and (2) they integrate textual constraints belatedly, which may cause the video features potentially focus on the non-referred objects. Therefore, we propose a novel RVOS architecture called ProxyFormer, which introduces a set of proxy queries to integrate visual and text semantics and facilitate the flow of semantics between them. By progressively updating and propagating proxy queries across multiple stages of video feature encoder, ProxyFormer ensures that the video features are focused on the object of interest. This dynamic evolution also enables the establishment of inter-frame dependencies, enhancing the accuracy and coherence of object tracking. To mitigate high computational costs, we decouple cross-modality interactions into temporal and spatial dimensions. Additionally, we design a Joint Semantic Consistency (JSC) training strategy to align semantic consensus between the proxy queries and the combined video-text pairs. Comprehensive experiments on four widely used RVOS benchmarks demonstrate the superiority of our ProxyFormer to the state-of-the-art methods.
☆ Efficient Training for Human Video Generation with Entropy-Guided Prioritized Progressive Learning
Changlin Li, Jiawei Zhang, Shuhao Liu, Sihao Lin, Zeyi Shi, Zhihui Li, Xiaojun Chang
Human video generation has advanced rapidly with the development of diffusion models, but the high computational cost and substantial memory consumption associated with training these models on high-resolution, multi-frame data pose significant challenges. In this paper, we propose Entropy-Guided Prioritized Progressive Learning (Ent-Prog), an efficient training framework tailored for diffusion models on human video generation. First, we introduce Conditional Entropy Inflation (CEI) to assess the importance of different model components on the target conditional generation task, enabling prioritized training of the most critical components. Second, we introduce an adaptive progressive schedule that adaptively increases computational complexity during training by measuring the convergence efficiency. Ent-Prog reduces both training time and GPU memory consumption while maintaining model performance. Extensive experiments across three datasets, demonstrate the effectiveness of Ent-Prog, achieving up to 2.2$\times$ training speedup and 2.4$\times$ GPU memory reduction without compromising generative performance.
comment : Project page: https://github.com/changlin31/Ent-Prog
☆ SocialNav: Training Human-Inspired Foundation Model for Socially-Aware Embodied Navigation
Ziyi Chen, Yingnan Guo, Zedong Chu, Minghua Luo, Yanfen Shen, Mingchao Sun, Junjun Hu, Shichao Xie, Kuan Yang, Pei Shi, Zhining Gu, Lu Liu, Honglin Han, Xiaolong Wu, Mu Xu, Yu Zhang
Embodied navigation that adheres to social norms remains an open research challenge. Our \textbf{SocialNav} is a foundational model for socially-aware navigation with a hierarchical "brain-action" architecture, capable of understanding high-level social norms and generating low-level, socially compliant trajectories. To enable such dual capabilities, we construct the SocNav Dataset, a large-scale collection of 7 million samples, comprising (1) a Cognitive Activation Dataset providing social reasoning signals such as chain-of-thought explanations and social traversability prediction, and (2) an Expert Trajectories Pyramid aggregating diverse navigation demonstrations from internet videos, simulated environments, and real-world robots. A multi-stage training pipeline is proposed to gradually inject and refine navigation intelligence: we first inject general navigation skills and social norms understanding into the model via imitation learning, and then refine such skills through a deliberately designed Socially-Aware Flow Exploration GRPO (SAFE-GRPO), the first flow-based reinforcement learning framework for embodied navigation that explicitly rewards socially compliant behaviors. SocialNav achieves +38% success rate and +46% social compliance rate compared to the state-of-the-art method, demonstrating strong gains in both navigation performance and social compliance. Our project page: https://amap-eai.github.io/SocialNav/
☆ DeepRFTv2: Kernel-level Learning for Image Deblurring
Xintian Mao, Haofei Song, Yin-Nian Liu, Qingli Li, Yan Wang
It is well-known that if a network aims to learn how to deblur, it should understand the blur process. Blurring is naturally caused by the convolution of the sharp image with the blur kernel. Thus, allowing the network to learn the blur process in the kernel-level can significantly improve the image deblurring performance. But, current deep networks are still at the pixel-level learning stage, either performing end-to-end pixel-level restoration or stage-wise pseudo kernel-level restoration, failing to enable the deblur model to understand the essence of the blur. To this end, we propose Fourier Kernel Estimator (FKE), which considers the activation operation in Fourier space and converts the convolution problem in the spatial domain to a multiplication problem in Fourier space. Our FKE, jointly optimized with the deblur model, enables the network to learn the kernel-level blur process with low complexity and without any additional supervision. Furthermore, we change the convolution object of the kernel from ``image" to network extracted ``feature", whose rich semantic and structural information is more suitable to blur process learning. With the convolution of the feature and the estimated kernel, our model can learn the essence of blur in kernel-level. To further improve the efficiency of feature extraction, we design a decoupled multi-scale architecture with multiple hierarchical sub-unets with a reversible strategy, which allows better multi-scale encoding and decoding in low training memory. Extensive experiments indicate that our method achieves state-of-the-art motion deblurring results and show potential for handling other kernel-related problems. Analysis also shows our kernel estimator is able to learn physically meaningful kernels. The code will be available at https://github.com/DeepMed-Lab-ECNU/Single-Image-Deblur.
☆ Ct rl VDiff: Controllable Video Generation via Unified Multimodal Video Diffusion
Dianbing Xi, Jiepeng Wang, Yuanzhi Liang, Xi Qiu, Jialun Liu, Hao Pan, Yuchi Huo, Rui Wang, Haibin Huang, Chi Zhang, Xuelong Li
We tackle the dual challenges of video understanding and controllable video generation within a unified diffusion framework. Our key insights are two-fold: geometry-only cues (e.g., depth, edges) are insufficient: they specify layout but under-constrain appearance, materials, and illumination, limiting physically meaningful edits such as relighting or material swaps and often causing temporal drift. Enriching the model with additional graphics-based modalities (intrinsics and semantics) provides complementary constraints that both disambiguate understanding and enable precise, predictable control during generation. However, building a single model that uses many heterogeneous cues introduces two core difficulties. Architecturally, the model must accept any subset of modalities, remain robust to missing inputs, and inject control signals without sacrificing temporal consistency. Data-wise, training demands large-scale, temporally aligned supervision that ties real videos to per-pixel multimodal annotations. We then propose CtrlVDiff, a unified diffusion model trained with a Hybrid Modality Control Strategy (HMCS) that routes and fuses features from depth, normals, segmentation, edges, and graphics-based intrinsics (albedo, roughness, metallic), and re-renders videos from any chosen subset with strong temporal coherence. To enable this, we build MMVideo, a hybrid real-and-synthetic dataset aligned across modalities and captions. Across understanding and generation benchmarks, CtrlVDiff delivers superior controllability and fidelity, enabling layer-wise edits (relighting, material adjustment, object insertion) and surpassing state-of-the-art baselines while remaining robust when some modalities are unavailable.
comment : 27 pages, 18 figures, 9 tables. Project page: https://tele-ai.github.io/CtrlVDiff/
☆ Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Models
Changlin Li, Jiawei Zhang, Zeyi Shi, Zongxin Yang, Zhihui Li, Xiaojun Chang
Large-scale vision generative models, including diffusion and flow models, have demonstrated remarkable performance in visual generation tasks. However, transferring these pre-trained models to downstream tasks often results in significant parameter redundancy. In this paper, we propose EntPruner, an entropy-guided automatic progressive pruning framework for diffusion and flow models. First, we introduce entropy-guided pruning, a block-level importance assessment strategy specifically designed for generative models. Unlike discriminative models, generative models require preserving the diversity and condition-fidelity of the output distribution. As the importance of each module can vary significantly across downstream tasks, EntPruner prioritizes pruning of less important blocks using data-dependent Conditional Entropy Deviation (CED) as a guiding metric. CED quantifies how much the distribution diverges from the learned conditional data distribution after removing a block. Second, we propose a zero-shot adaptive pruning framework to automatically determine when and how much to prune during training. This dynamic strategy avoids the pitfalls of one-shot pruning, mitigating mode collapse, and preserving model performance. Extensive experiments on DiT and SiT models demonstrate the effectiveness of EntPruner, achieving up to 2.22$\times$ inference speedup while maintaining competitive generation quality on ImageNet and three downstream datasets.
comment : Project page: https://github.com/changlin31/EntPruner
☆ Deformation-aware Temporal Generation for Ea rl y Prediction of Alzheimers Disease
Xin Honga, Jie Lin, Minghui Wang
Alzheimer's disease (AD), a degenerative brain condition, can benefit from early prediction to slow its progression. As the disease progresses, patients typically undergo brain atrophy. Current prediction methods for Alzheimers disease largely involve analyzing morphological changes in brain images through manual feature extraction. This paper proposes a novel method, the Deformation-Aware Temporal Generative Network (DATGN), to automate the learning of morphological changes in brain images about disease progression for early prediction. Given the common occurrence of missing data in the temporal sequences of MRI images, DATGN initially interpolates incomplete sequences. Subsequently, a bidirectional temporal deformation-aware module guides the network in generating future MRI images that adhere to the disease's progression, facilitating early prediction of Alzheimer's disease. DATGN was tested for the generation of temporal sequences of future MRI images using the ADNI dataset, and the experimental results are competitive in terms of PSNR and MMSE image quality metrics. Furthermore, when DATGN-generated synthetic data was integrated into the SVM vs. CNN vs. 3DCNN-based classification methods, significant improvements were achieved from 6. 21\% to 16\% in AD vs. NC classification accuracy and from 7. 34\% to 21. 25\% in AD vs. MCI vs. NC classification accuracy. The qualitative visualization results indicate that DATGN produces MRI images consistent with the brain atrophy trend in Alzheimer's disease, enabling early disease prediction.
comment : 29 pages,6figures,one column
☆ FaithFusion: Harmonizing Reconstruction and Generation via Pixel-wise Information Gain
YuAn Wang, Xiaofan Li, Chi Huang, Wenhao Zhang, Hao Li, Bosheng Wang, Xun Sun, Jun Wang
In controllable driving-scene reconstruction and 3D scene generation, maintaining geometric fidelity while synthesizing visually plausible appearance under large viewpoint shifts is crucial. However, effective fusion of geometry-based 3DGS and appearance-driven diffusion models faces inherent challenges, as the absence of pixel-wise, 3D-consistent editing criteria often leads to over-restoration and geometric drift. To address these issues, we introduce \textbf{FaithFusion}, a 3DGS-diffusion fusion framework driven by pixel-wise Expected Information Gain (EIG). EIG acts as a unified policy for coherent spatio-temporal synthesis: it guides diffusion as a spatial prior to refine high-uncertainty regions, while its pixel-level weighting distills the edits back into 3DGS. The resulting plug-and-play system is free from extra prior conditions and structural modifications.Extensive experiments on the Waymo dataset demonstrate that our approach attains SOTA performance across NTA-IoU, NTL-IoU, and FID, maintaining an FID of 107.47 even at 6 meters lane shift. Our code is available at https://github.com/wangyuanbiubiubiu/FaithFusion.
comment : 16 pages, 10 figures
☆ EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokens AAAI 2026
Ze Feng, Sen Yang, Boqiang Duan, Wankou Yang, Jingdong Wang
Efficient Multimodal Large Language Models (MLLMs) compress vision tokens to reduce resource consumption, but the loss of visual information can degrade comprehension capabilities. Although some priors introduce Knowledge Distillation to enhance student models, they overlook the fundamental differences in fine-grained vision comprehension caused by unbalanced vision tokens between the efficient student and vanilla teacher. In this paper, we propose EM-KD, a novel paradigm that enhances the Efficient MLLMs with Knowledge Distillation. To overcome the challenge of unbalanced vision tokens, we first calculate the Manhattan distance between the vision logits of teacher and student, and then align them in the spatial dimension with the Hungarian matching algorithm. After alignment, EM-KD introduces two distillation strategies: 1) Vision-Language Affinity Distillation (VLAD) and 2) Vision Semantic Distillation (VSD). Specifically, VLAD calculates the affinity matrix between text tokens and aligned vision tokens, and minimizes the smooth L1 distance of the student and the teacher affinity matrices. Considering the semantic richness of vision logits in the final layer, VSD employs the reverse KL divergence to measure the discrete probability distributions of the aligned vision logits over the vocabulary space. Comprehensive evaluation on diverse benchmarks demonstrates that EM-KD trained model outperforms prior Efficient MLLMs on both accuracy and efficiency with a large margin, validating its effectiveness. Compared with previous distillation methods, which are equipped with our proposed vision token matching strategy for fair comparison, EM-KD also achieves better performance.
comment : accepted by AAAI 2026
☆ Scaling Foundation Models for Radar Scene Understanding
Pushkal Mishra, Kshitiz Bansal, Dinesh Bharadia
Radar sensors provide reliable perception across adverse weather, lighting, and long-range conditions. Recent advances in foundation models have transformed visual and language understanding, yet their integration with radar sensing remains largely underexplored. Existing radar approaches are fragmented and task-specific; each downstream task employs distinct architectures and training objectives, preventing transfer across tasks. In this work, we introduce RadarFM: a radar foundation model that learns unified scene-level representations through structured spatial language supervision. We make two key contributions: (1) a structured caption framework that encodes vehicle distributions in native radar coordinates, and (2) a hash-aware contrastive learning objective that quantifies continuous scene similarity rather than binary matching, enabling fine-grained spatial reasoning. Leveraging the CARLA simulator, we generate large-scale, well-annotated radar datasets across diverse driving scenarios. We also propose localization-aware metrics that assess spatial accuracy beyond traditional detection measures.
☆ Pygmalion Effect in Vision : Image-to-Clay Translation for Reflective Geometry Reconstruction
Gayoung Lee, Junho Kim, Jin-Hwa Kim, Junmo Kim
Understanding reflection remains a long-standing challenge in 3D reconstruction due to the entanglement of appearance and geometry under view-dependent reflections. In this work, we present the Pygmalion Effect in Vision, a novel framework that metaphorically "sculpts" reflective objects into clay-like forms through image-to-clay translation. Inspired by the myth of Pygmalion, our method learns to suppress specular cues while preserving intrinsic geometric consistency, enabling robust reconstruction from multi-view images containing complex reflections. Specifically, we introduce a dual-branch network in which a BRDF-based reflective branch is complemented by a clay-guided branch that stabilizes geometry and refines surface normals. The two branches are trained jointly using the synthesized clay-like images, which provide a neutral, reflection-free supervision signal that complements the reflective views. Experiments on both synthetic and real datasets demonstrate substantial improvement in normal accuracy and mesh completeness over existing reflection-handling methods. Beyond technical gains, our framework reveals that seeing by unshining, translating radiance into neutrality, can serve as a powerful inductive bias for reflective object geometry learning.
☆ CLRecogEye : Curriculum Learning towards exploiting convolution features for Dynamic Iris Recognition
Geetanjali Sharma, Gaurav Jaswal, Aditya Nigam, Raghavendra Ramachandra
Iris authentication algorithms have achieved impressive recognition performance, making them highly promising for real-world applications such as border control, citizen identification, and both criminal investigations and commercial systems. However, their robustness is still challenged by variations in rotation, scale, specular reflections, and defocus blur. In addition, most existing approaches rely on straightforward point-to-point comparisons, typically using cosine or L2 distance, without effectively leveraging the spatio-spatial-temporal structure of iris patterns. To address these limitations, we propose a novel and generalized matching pipeline that learns rich spatio-spatial-temporal representations of iris features. Our approach first splits each iris image along one dimension, generating a sequence of sub-images that serve as input to a 3D-CNN, enabling the network to capture both spatial and spatio-spatial-temporal cues. To further enhance the modeling of spatio-spatial-temporal feature dynamics, we train the model in curriculum manner. This design allows the network to embed temporal dependencies directly into the feature space, improving discriminability in the deep metric domain. The framework is trained end-to-end with triplet and ArcFace loss in a curriculum manner, enforcing highly discriminative embeddings despite challenges like rotation, scale, reflections, and blur. This design yields a robust and generalizable solution for iris authentication.Github code: https://github.com/GeetanjaliGTZ/CLRecogEye
comment : 12 Pages, 3 figures, ISVC conference 2025
☆ MIRA: Multimodal Iterative Reasoning Agent for Image Editing
Ziyun Zeng, Hang Hua, Jiebo Luo
Instruction-guided image editing offers an intuitive way for users to edit images with natural language. However, diffusion-based editing models often struggle to accurately interpret complex user instructions, especially those involving compositional relationships, contextual cues, or referring expressions, leading to edits that drift semantically or fail to reflect the intended changes. We tackle this problem by proposing MIRA (Multimodal Iterative Reasoning Agent), a lightweight, plug-and-play multimodal reasoning agent that performs editing through an iterative perception-reasoning-action loop, effectively simulating multi-turn human-model interaction processes. Instead of issuing a single prompt or static plan, MIRA predicts atomic edit instructions step by step, using visual feedback to make its decisions. Our 150K multimodal tool-use dataset, MIRA-Editing, combined with a two-stage SFT + GRPO training pipeline, enables MIRA to perform reasoning and editing over complex editing instructions. When paired with open-source image editing models such as Flux.1-Kontext, Step1X-Edit, and Qwen-Image-Edit, MIRA significantly improves both semantic consistency and perceptual quality, achieving performance comparable to or exceeding proprietary systems such as GPT-Image and Nano-Banana.
☆ OVOD- Agent : A Markov-Bandit Framework for Proactive Visual Reasoning and Self-Evolving Detection
Chujie Wang, Jianyu Lu, Zhiyuan Luo, Xi Chen, Chu He
Open-Vocabulary Object Detection (OVOD) aims to enable detectors to generalize across categories by leveraging semantic information. Although existing methods are pretrained on large vision-language datasets, their inference is still limited to fixed category names, creating a gap between multimodal training and unimodal inference. Previous work has shown that improving textual representation can significantly enhance OVOD performance, indicating that the textual space is still underexplored. To this end, we propose OVOD-Agent, which transforms passive category matching into proactive visual reasoning and self-evolving detection. Inspired by the Chain-of-Thought (CoT) paradigm, OVOD-Agent extends the textual optimization process into an interpretable Visual-CoT with explicit actions. OVOD's lightweight nature makes LLM-based management unsuitable; instead, we model visual context transitions as a Weakly Markovian Decision Process (w-MDP) over eight state spaces, which naturally represents the agent's state, memory, and interaction dynamics. A Bandit module generates exploration signals under limited supervision, helping the agent focus on uncertain regions and adapt its detection policy. We further integrate Markov transition matrices with Bandit trajectories for self-supervised Reward Model (RM) optimization, forming a closed loop from Bandit exploration to RM learning. Experiments on COCO and LVIS show that OVOD-Agent provides consistent improvements across OVOD backbones, particularly on rare categories, confirming the effectiveness of the proposed framework.
☆ Long-Term Alzheimers Disease Prediction: A Novel Image Generation Method Using Temporal Parameter Estimation with Normal Inverse Gamma Distribution on Uneven Time Series
Xin Hong, Xinze Sun, Yinhao Li, Yen-Wei Chen
Image generation can provide physicians with an imaging diagnosis basis in the prediction of Alzheimer's Disease (AD). Recent research has shown that long-term AD predictions by image generation often face difficulties maintaining disease-related characteristics when dealing with irregular time intervals in sequential data. Considering that the time-related aspects of the distribution can reflect changes in disease-related characteristics when images are distributed unevenly, this research proposes a model to estimate the temporal parameter within the Normal Inverse Gamma Distribution (T-NIG) to assist in generating images over the long term. The T-NIG model employs brain images from two different time points to create intermediate brain images, forecast future images, and predict the disease. T-NIG is designed by identifying features using coordinate neighborhoods. It incorporates a time parameter into the normal inverse gamma distribution to understand how features change in brain imaging sequences that have varying time intervals. Additionally, T-NIG utilizes uncertainty estimation to reduce both epistemic and aleatoric uncertainties in the model, which arise from insufficient temporal data. In particular, the T-NIG model demonstrates state-of-the-art performance in both short-term and long-term prediction tasks within the dataset. Experimental results indicate that T-NIG is proficient in forecasting disease progression while maintaining disease-related characteristics, even when faced with an irregular temporal data distribution.
comment : 13pages, 6 figures
☆ AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarios AAAI 2026
Chenglizhao Chen, Shaofeng Liang, Runwei Guan, Xiaolou Sun, Haocheng Zhao, Haiyun Jiang, Tao Huang, Henghui Ding, Qing-Long Han
Referring Multi-Object Tracking (RMOT) aims to achieve precise object detection and tracking through natural language instructions, representing a fundamental capability for intelligent robotic systems. However, current RMOT research remains mostly confined to ground-level scenarios, which constrains their ability to capture broad-scale scene contexts and perform comprehensive tracking and path planning. In contrast, Unmanned Aerial Vehicles (UAVs) leverage their expansive aerial perspectives and superior maneuverability to enable wide-area surveillance. Moreover, UAVs have emerged as critical platforms for Embodied Intelligence, which has given rise to an unprecedented demand for intelligent aerial systems capable of natural language interaction. To this end, we introduce AerialMind, the first large-scale RMOT benchmark in UAV scenarios, which aims to bridge this research gap. To facilitate its construction, we develop an innovative semi-automated collaborative agent-based labeling assistant (COALA) framework that significantly reduces labor costs while maintaining annotation quality. Furthermore, we propose HawkEyeTrack (HETrack), a novel method that collaboratively enhances vision-language representation learning and improves the perception of UAV scenarios. Comprehensive experiments validated the challenging nature of our dataset and the effectiveness of our method.
comment : AAAI 2026
☆ MUSE: Manipulating Unified Framework for Synthesizing Emotions in Images via Test-Time Optimization
Yingjie Xia, Xi Wang, Jinglei Shi, Vicky Kalogeiton, Jian Yang
Images evoke emotions that profoundly influence perception, often prioritized over content. Current Image Emotional Synthesis (IES) approaches artificially separate generation and editing tasks, creating inefficiencies and limiting applications where these tasks naturally intertwine, such as therapeutic interventions or storytelling. In this work, we introduce MUSE, the first unified framework capable of both emotional generation and editing. By adopting a strategy conceptually aligned with Test-Time Scaling (TTS) that widely used in both LLM and diffusion model communities, it avoids the requirement for additional updating diffusion model and specialized emotional synthesis datasets. More specifically, MUSE addresses three key questions in emotional synthesis: (1) HOW to stably guide synthesis by leveraging an off-the-shelf emotion classifier with gradient-based optimization of emotional tokens; (2) WHEN to introduce emotional guidance by identifying the optimal timing using semantic similarity as a supervisory signal; and (3) WHICH emotion to guide synthesis through a multi-emotion loss that reduces interference from inherent and similar emotions. Experimental results show that MUSE performs favorably against all methods for both generation and editing, improving emotional accuracy and semantic diversity while maintaining an optimal balance between desired content, adherence to text prompts, and realistic emotional expression. It establishes a new paradigm for emotion synthesis.
☆ PG-ControlNet: A Physics-Guided ControlNet for Generative Spatially Varying Image Deblurring
Hakki Motorcu, Mujdat Cetin
Spatially varying image deblurring remains a fundamentally ill-posed problem, especially when degradations arise from complex mixtures of motion and other forms of blur under significant noise. State-of-the-art learning-based approaches generally fall into two paradigms: model-based deep unrolling methods that enforce physical constraints by modeling the degradations, but often produce over-smoothed, artifact-laden textures, and generative models that achieve superior perceptual quality yet hallucinate details due to weak physical constraints. In this paper, we propose a novel framework that uniquely reconciles these paradigms by taming a powerful generative prior with explicit, dense physical constraints. Rather than oversimplifying the degradation field, we model it as a dense continuum of high-dimensional compressed kernels, ensuring that minute variations in motion and other degradation patterns are captured. We leverage this rich descriptor field to condition a ControlNet architecture, strongly guiding the diffusion sampling process. Extensive experiments demonstrate that our method effectively bridges the gap between physical accuracy and perceptual realism, outperforming state-of-the-art model-based methods as well as generative baselines in challenging, severely blurred scenarios.
comment : 9 pages, 4 figures
☆ LungNodule Agent : A Collaborative Multi- Agent System for Precision Diagnosis of Lung Nodules AAAI 2026
Cheng Yang, Hui Jin, Xinlei Yu, Zhipeng Wang, Yaoqun Liu, Fenglei Fan, Dajiang Lei, Gangyong Jia, Changmiao Wang, Ruiquan Ge
Diagnosing lung cancer typically involves physicians identifying lung nodules in Computed tomography (CT) scans and generating diagnostic reports based on their morphological features and medical expertise. Although advancements have been made in using multimodal large language models for analyzing lung CT scans, challenges remain in accurately describing nodule morphology and incorporating medical expertise. These limitations affect the reliability and effectiveness of these models in clinical settings. Collaborative multi-agent systems offer a promising strategy for achieving a balance between generality and precision in medical applications, yet their potential in pathology has not been thoroughly explored. To bridge these gaps, we introduce LungNoduleAgent, an innovative collaborative multi-agent system specifically designed for analyzing lung CT scans. LungNoduleAgent streamlines the diagnostic process into sequential components, improving precision in describing nodules and grading malignancy through three primary modules. The first module, the Nodule Spotter, coordinates clinical detection models to accurately identify nodules. The second module, the Radiologist, integrates localized image description techniques to produce comprehensive CT reports. Finally, the Doctor Agent System performs malignancy reasoning by using images and CT reports, supported by a pathology knowledge base and a multi-agent system framework. Extensive testing on two private datasets and the public LIDC-IDRI dataset indicates that LungNoduleAgent surpasses mainstream vision-language models, agent systems, and advanced expert models. These results highlight the importance of region-level semantic alignment and multi-agent collaboration in diagnosing nodules. LungNoduleAgent stands out as a promising foundational tool for supporting clinical analyses of lung nodules.
comment : Accepted by AAAI 2026
☆ CNN-LSTM Hybrid Architecture for Over-the-Air Automatic Modulation Classification Using SDR
Dinanath Padhya, Krishna Acharya, Bipul Kumar Dahal, Dinesh Baniya Kshatri
Automatic Modulation Classification (AMC) is a core technology for future wireless communication systems, enabling the identification of modulation schemes without prior knowledge. This capability is essential for applications in cognitive radio, spectrum monitoring, and intelligent communication networks. We propose an AMC system based on a hybrid Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) architecture, integrated with a Software Defined Radio (SDR) platform. The proposed architecture leverages CNNs for spatial feature extraction and LSTMs for capturing temporal dependencies, enabling efficient handling of complex, time-varying communication signals. The system's practical ability was demonstrated by identifying over-the-air (OTA) signals from a custom-built FM transmitter alongside other modulation schemes. The system was trained on a hybrid dataset combining the RadioML2018 dataset with a custom-generated dataset, featuring samples at Signal-to-Noise Ratios (SNRs) from 0 to 30dB. System performance was evaluated using accuracy, precision, recall, F1 score, and the Area Under the Receiver Operating Characteristic Curve (AUC-ROC). The optimized model achieved 93.48% accuracy, 93.53% precision, 93.48% recall, and an F1 score of 93.45%. The AUC-ROC analysis confirmed the model's discriminative power, even in noisy conditions. This paper's experimental results validate the effectiveness of the hybrid CNN-LSTM architecture for AMC, suggesting its potential application in adaptive spectrum management and advanced cognitive radio systems.
comment : 8 Pages, 10 figures, 2 Tables, Accepted in Journal (Journal of Innovations in Engineering Education), Issue is not Published Yet
☆ FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generation
Kaixing Yang, Xulong Tang, Ziqiao Peng, Xiangyue Zhang, Puwei Wang, Jun He, Hongyan Liu
Music-to-dance generation aims to translate auditory signals into expressive human motion, with broad applications in virtual reality, choreography, and digital entertainment. Despite promising progress, the limited generation efficiency of existing methods leaves insufficient computational headroom for high-fidelity 3D rendering, thereby constraining the expressiveness of 3D characters during real-world applications. Thus, we propose FlowerDance, which not only generates refined motion with physical plausibility and artistic expressiveness, but also achieves significant generation efficiency on inference speed and memory utilization . Specifically, FlowerDance combines MeanFlow with Physical Consistency Constraints, which enables high-quality motion generation with only a few sampling steps. Moreover, FlowerDance leverages a simple but efficient model architecture with BiMamba-based backbone and Channel-Level Cross-Modal Fusion, which generates dance with efficient non-autoregressive manner. Meanwhile, FlowerDance supports motion editing, enabling users to interactively refine dance sequences. Extensive experiments on AIST++ and FineDance show that FlowerDance achieves state-of-the-art results in both motion quality and generation efficiency. Code will be released upon acceptance.
☆ Deep Parameter Interpolation for Scalar Conditioning
Chicago Y. Park, Michael T. McCann, Cristina Garcia-Cardona, Brendt Wohlberg, Ulugbek S. Kamilov
We propose deep parameter interpolation (DPI), a general-purpose method for transforming an existing deep neural network architecture into one that accepts an additional scalar input. Recent deep generative models, including diffusion models and flow matching, employ a single neural network to learn a time- or noise level-dependent vector field. Designing a network architecture to accurately represent this vector field is challenging because the network must integrate information from two different sources: a high-dimensional vector (usually an image) and a scalar. Common approaches either encode the scalar as an additional image input or combine scalar and vector information in specific network components, which restricts architecture choices. Instead, we propose to maintain two learnable parameter sets within a single network and to introduce the scalar dependency by dynamically interpolating between the parameter sets based on the scalar value during training and sampling. DPI is a simple, architecture-agnostic method for adding scalar dependence to a neural network. We demonstrate that our method improves denoising performance and enhances sample quality for both diffusion and flow matching models, while achieving computational efficiency comparable to standard scalar conditioning techniques. Code is available at https://github.com/wustl-cig/parameter_interpolation.
☆ CaptionQA: Is Your Caption as Useful as the Image Itself?
Shijia Yang, Yunong Liu, Bohan Zhai, Ximeng Sun, Zicheng Liu, Emad Barsoum, Manling Li, Chenfeng Xu
Image captions serve as efficient surrogates for visual content in multimodal systems such as retrieval, recommendation, and multi-step agentic inference pipelines. Yet current evaluation practices miss a fundamental question: Can captions stand-in for images in real downstream tasks? We propose a utility-based benchmark, CaptionQA, to evaluate model-generated captions, where caption quality is measured by how well it supports downstream tasks. CaptionQA is an extensible domain-dependent benchmark covering 4 domains--Natural, Document, E-commerce, and Embodied AI--each with fine-grained taxonomies (25 top-level and 69 subcategories) that identify useful information for domain-specific tasks. CaptionQA builds 33,027 densely annotated multiple-choice questions (50.3 per image on average) that explicitly require visual information to answer, providing a comprehensive probe of caption utility. In our evaluation protocol, an LLM answers these questions using captions alone, directly measuring whether captions preserve image-level utility and are utilizable by a downstream LLM. Evaluating state-of-the-art MLLMs reveals substantial gaps between the image and its caption utility. Notably, models nearly identical on traditional image-QA benchmarks lower by up to 32% in caption utility. We release CaptionQA along with an open-source pipeline for extension to new domains. The code is available at https://github.com/bronyayang/CaptionQA.
☆ CameraMaster: Unified Camera Semantic-Parameter Control for Photography Retouching
Qirui Yang, Yang Yang, Ying Zeng, Xiaobin Hu, Bo Li, Huanjing Yue, Jingyu Yang, Peng-Tao Jiang
Text-guided diffusion models have greatly advanced image editing and generation. However, achieving physically consistent image retouching with precise parameter control (e.g., exposure, white balance, zoom) remains challenging. Existing methods either rely solely on ambiguous and entangled text prompts, which hinders precise camera control, or train separate heads/weights for parameter adjustment, which compromises scalability, multi-parameter composition, and sensitivity to subtle variations. To address these limitations, we propose CameraMaster, a unified camera-aware framework for image retouching. The key idea is to explicitly decouple the camera directive and then coherently integrate two critical information streams: a directive representation that captures the photographer's intent, and a parameter embedding that encodes precise camera settings. CameraMaster first uses the camera parameter embedding to modulate both the camera directive and the content semantics. The modulated directive is then injected into the content features via cross-attention, yielding a strongly camera-sensitive semantic context. In addition, the directive and camera embeddings are injected as conditioning and gating signals into the time embedding, enabling unified, layer-wise modulation throughout the denoising process and enforcing tight semantic-parameter alignment. To train and evaluate CameraMaster, we construct a large-scale dataset of 78K image-prompt pairs annotated with camera parameters. Extensive experiments show that CameraMaster produces monotonic and near-linear responses to parameter variations, supports seamless multi-parameter composition, and significantly outperforms existing methods.
☆ Structure-Aware Prototype Guided Trusted Multi-View Classification
Haojian Huang, Jiahao Shi, Zhe Liu, Harold Haodong Chen, Han Fang, Hao Sun, Zhongjiang He
Trustworthy multi-view classification (TMVC) addresses the challenge of achieving reliable decision-making in complex scenarios where multi-source information is heterogeneous, inconsistent, or even conflicting. Existing TMVC approaches predominantly rely on globally dense neighbor relationships to model intra-view dependencies, leading to high computational costs and an inability to directly ensure consistency across inter-view relationships. Furthermore, these methods typically aggregate evidence from different views through manually assigned weights, lacking guarantees that the learned multi-view neighbor structures are consistent within the class space, thus undermining the trustworthiness of classification outcomes. To overcome these limitations, we propose a novel TMVC framework that introduces prototypes to represent the neighbor structures of each view. By simplifying the learning of intra-view neighbor relations and enabling dynamic alignment of intra- and inter-view structure, our approach facilitates more efficient and consistent discovery of cross-view consensus. Extensive experiments on multiple public multi-view datasets demonstrate that our method achieves competitive downstream performance and robustness compared to prevalent TMVC methods.
comment : 12 pages, 8 figures, 7 tables, Ongoing Work
☆ Probabilistic Wildfire Spread Prediction Using an Autoregressive Conditional Generative Adversarial Network
Taehoon Kang, Taeyong Kim
Climate change has intensified the frequency and severity of wildfires, making rapid and accurate prediction of fire spread essential for effective mitigation and response. Physics-based simulators such as FARSITE offer high-fidelity predictions but are computationally intensive, limiting their applicability in real-time decision-making, while existing deep learning models often yield overly smooth predictions that fail to capture the complex, nonlinear dynamics of wildfire propagation. This study proposes an autoregressive conditional generative adversarial network (CGAN) for probabilistic wildfire spread prediction. By formulating the prediction task as an autoregressive problem, the model learns sequential state transitions, ensuring long-term prediction stability. Experimental results demonstrate that the proposed CGAN-based model outperforms conventional deep learning models in both overall predictive accuracy and boundary delineation of fire perimeters. These results demonstrate that adversarial learning allows the model to capture the strong nonlinearity and uncertainty of wildfire spread, instead of simply fitting the pixel average. Furthermore, the autoregressive framework facilitates systematic temporal forecasting of wildfire evolution. The proposed CGAN-based autoregressive framework enhances both the accuracy and physical interpretability of wildfire spread prediction, offering a promising foundation for time-sensitive response and evacuation planning.
comment : 22 pages, 15 figures, Submitted to Journal of Environmental Management
☆ MetaRank: Task-Aware Metric Selection for Model Transferability Estimation
Yuhang Liu, Wenjie Zhao, Yunhui Guo
Selecting an appropriate pre-trained source model is a critical, yet computationally expensive, task in transfer learning. Model Transferability Estimation (MTE) methods address this by providing efficient proxy metrics to rank models without full fine-tuning. In practice, the choice of which MTE metric to use is often ad hoc or guided simply by a metric's average historical performance. However, we observe that the effectiveness of MTE metrics is highly task-dependent and no single metric is universally optimal across all target datasets. To address this gap, we introduce MetaRank, a meta-learning framework for automatic, task-aware MTE metric selection. We formulate metric selection as a learning-to-rank problem. Rather than relying on conventional meta-features, MetaRank encodes textual descriptions of both datasets and MTE metrics using a pretrained language model, embedding them into a shared semantic space. A meta-predictor is then trained offline on diverse meta-tasks to learn the intricate relationship between dataset characteristics and metric mechanisms, optimized with a listwise objective that prioritizes correctly ranking the top-performing metrics. During the subsequent online phase, MetaRank efficiently ranks the candidate MTE metrics for a new, unseen target dataset based on its textual description, enabling practitioners to select the most appropriate metric a priori. Extensive experiments across 11 pretrained models and 11 target datasets demonstrate the strong effectiveness of our approach.
comment : 10 figures
☆ Knowledge Completes the Vision : A Multimodal Entity -aware Retrieval-Augmented Generation Framework for News Image Captioning AAAI 2026
Xiaoxing You, Qiang Huang, Lingyu Li, Chi Zhang, Xiaopeng Liu, Min Zhang, Jun Yu
News image captioning aims to produce journalistically informative descriptions by combining visual content with contextual cues from associated articles. Despite recent advances, existing methods struggle with three key challenges: (1) incomplete information coverage, (2) weak cross-modal alignment, and (3) suboptimal visual-entity grounding. To address these issues, we introduce MERGE, the first Multimodal Entity-aware Retrieval-augmented GEneration framework for news image captioning. MERGE constructs an entity-centric multimodal knowledge base (EMKB) that integrates textual, visual, and structured knowledge, enabling enriched background retrieval. It improves cross-modal alignment through a multistage hypothesis-caption strategy and enhances visual-entity matching via dynamic retrieval guided by image content. Extensive experiments on GoodNews and NYTimes800k show that MERGE significantly outperforms state-of-the-art baselines, with CIDEr gains of +6.84 and +1.16 in caption quality, and F1-score improvements of +4.14 and +2.64 in named entity recognition. Notably, MERGE also generalizes well to the unseen Visual News dataset, achieving +20.17 in CIDEr and +6.22 in F1-score, demonstrating strong robustness and domain adaptability.
comment : Accepted to AAAI 2026
☆ From Inpainting to Layer Decomposition: Repurposing Generative Inpainting Models for Image Layer Decomposition
Jingxi Chen, Yixiao Zhang, Xiaoye Qian, Zongxia Li, Cornelia Fermuller, Caren Chen, Yiannis Aloimonos
Images can be viewed as layered compositions, foreground objects over background, with potential occlusions. This layered representation enables independent editing of elements, offering greater flexibility for content creation. Despite the progress in large generative models, decomposing a single image into layers remains challenging due to limited methods and data. We observe a strong connection between layer decomposition and in/outpainting tasks, and propose adapting a diffusion-based inpainting model for layer decomposition using lightweight finetuning. To further preserve detail in the latent space, we introduce a novel multi-modal context fusion module with linear attention complexity. Our model is trained purely on a synthetic dataset constructed from open-source assets and achieves superior performance in object removal and occlusion recovery, unlocking new possibilities in downstream editing and creative applications.
☆ GuardTrace-VL: Detecting Unsafe Multimodel Reasoning via Iterative Safety Super vision
Yuxiao Xiang, Junchi Chen, Zhenchao Jin, Changtao Miao, Haojie Yuan, Qi Chu, Tao Gong, Nenghai Yu
Multimodal large reasoning models (MLRMs) are increasingly deployed for vision-language tasks that produce explicit intermediate rationales. However, reasoning traces can contain unsafe content even when the final answer is non-harmful, creating deployment risks. Existing multimodal safety guards primarily evaluate only the input question and the final answer, neglecting the intermediate reasoning process. This oversight allows undetected harm, such as biased inferences or policy-violating use of visual context, to emerge during reasoning. We introduce GuardTrace-VL, a vision-aware safety auditor that monitors the full Question-Thinking-Answer (QTA) pipeline via joint image-text analysis, enabling detection of unsafe content as it emerges in the reasoning stage. To support training and evaluation, we construct the GuardTrace dataset, which is generated through diverse prompting strategies and refined via a MLRM- and human-based voting and verification pipeline. Furthermore, we propose a three-stage progressive training scheme combined with the data refinement process, enabling the model to learn nuanced and context-dependent safety preferences according to different risk levels. On our proposed test set covering both in-domain and out-of-domain scenarios, GuardTrace-VL model achieves an F1 score of 93.1% on unsafe reasoning detection tasks, representing a 13.5% improvement in F1 score compared to the previous strongest multimodal safety defense methods. The codes will be made publicly available.
☆ Wavefront-Constrained Passive Obscured Object Detection
Zhiwen Zheng, Yiwei Ouyang, Zhao Huang, Tao Zhang, Xiaoshuai Zhang, Huiyu Zhou, Wenwen Tang, Shaowei Jiang, Jin Liu, Xingru Huang
Accurately localizing and segmenting obscured objects from faint light patterns beyond the field of view is highly challenging due to multiple scattering and medium-induced perturbations. Most existing methods, based on real-valued modeling or local convolutional operations, are inadequate for capturing the underlying physics of coherent light propagation. Moreover, under low signal-to-noise conditions, these methods often converge to non-physical solutions, severely compromising the stability and reliability of the observation. To address these challenges, we propose a novel physics-driven Wavefront Propagating Compensation Network (WavePCNet) to simulate wavefront propagation and enhance the perception of obscured objects. This WavePCNet integrates the Tri-Phase Wavefront Complex-Propagation Reprojection (TriWCP) to incorporate complex amplitude transfer operators to precisely constrain coherent propagation behavior, along with a momentum memory mechanism to effectively suppress the accumulation of perturbations. Additionally, a High-frequency Cross-layer Compensation Enhancement is introduced to construct frequency-selective pathways with multi-scale receptive fields and dynamically model structural consistency across layers, further boosting the model's robustness and interpretability under complex environmental conditions. Extensive experiments conducted on four physically collected datasets demonstrate that WavePCNet consistently outperforms state-of-the-art methods across both accuracy and robustness.
☆ RefOnce: Distilling References into a Prototype Memory for Referring Camouflaged Object Detection
Yu-Huan Wu, Zi-Xuan Zhu, Yan Wang, Liangli Zhen, Deng-Ping Fan
Referring Camouflaged Object Detection (Ref-COD) segments specified camouflaged objects in a scene by leveraging a small set of referring images. Though effective, current systems adopt a dual-branch design that requires reference images at test time, which limits deployability and adds latency and data-collection burden. We introduce a Ref-COD framework that distills references into a class-prototype memory during training and synthesizes a reference vector at inference via a query-conditioned mixture of prototypes. Concretely, we maintain an EMA-updated prototype per category and predict mixture weights from the query to produce a guidance vector without any test-time references. To bridge the representation gap between reference statistics and camouflaged query features, we propose a bidirectional attention alignment module that adapts both the query features and the class representation. Thus, our approach yields a simple, efficient path to Ref-COD without mandatory references. We evaluate the proposed method on the large-scale R2C7K benchmark. Extensive experiments demonstrate competitive or superior performance of the proposed method compared with recent state-of-the-arts. Code is available at https://github.com/yuhuan-wu/RefOnce.
comment : 11 pages, 5 figure, 6 tables
☆ Inversion-Free Style Transfer with Dual Rectified Flows
Yingying Deng, Xiangyu He, Fan Tang, Weiming Dong, Xucheng Yin
Style transfer, a pivotal task in image processing, synthesizes visually compelling images by seamlessly blending realistic content with artistic styles, enabling applications in photo editing and creative design. While mainstream training-free diffusion-based methods have greatly advanced style transfer in recent years, their reliance on computationally inversion processes compromises efficiency and introduces visual distortions when inversion is inaccurate. To address these limitations, we propose a novel \textit{inversion-free} style transfer framework based on dual rectified flows, which tackles the challenge of finding an unknown stylized distribution from two distinct inputs (content and style images), \textit{only with forward pass}. Our approach predicts content and style trajectories in parallel, then fuses them through a dynamic midpoint interpolation that integrates velocities from both paths while adapting to the evolving stylized image. By jointly modeling the content, style, and stylized distributions, our velocity field design achieves robust fusion and avoids the shortcomings of naive overlays. Attention injection further guides style integration, enhancing visual fidelity, content preservation, and computational efficiency. Extensive experiments demonstrate generalization across diverse styles and content, providing an effective and efficient pipeline for style transfer.
☆ Privacy-Preserving Federated Vision Transformer Learning Leveraging Lightweight Homomorphic Encryption in Medical AI
Al Amin, Kamrul Hasan, Liang Hong, Sharif Ullah
Collaborative machine learning across healthcare institutions promises improved diagnostic accuracy by leveraging diverse datasets, yet privacy regulations such as HIPAA prohibit direct patient data sharing. While federated learning (FL) enables decentralized training without raw data exchange, recent studies show that model gradients in conventional FL remain vulnerable to reconstruction attacks, potentially exposing sensitive medical information. This paper presents a privacy-preserving federated learning framework combining Vision Transformers (ViT) with homomorphic encryption (HE) for secure multi-institutional histopathology classification. The approach leverages the ViT CLS token as a compact 768-dimensional feature representation for secure aggregation, encrypting these tokens using CKKS homomorphic encryption before transmission to the server. We demonstrate that encrypting CLS tokens achieves a 30-fold communication reduction compared to gradient encryption while maintaining strong privacy guarantees. Through evaluation on a three-client federated setup for lung cancer histopathology classification, we show that gradients are highly susceptible to model inversion attacks (PSNR: 52.26 dB, SSIM: 0.999, NMI: 0.741), enabling near-perfect image reconstruction. In contrast, the proposed CLS-protected HE approach prevents such attacks while enabling encrypted inference directly on ciphertexts, requiring only 326 KB of encrypted data transmission per aggregation round. The framework achieves 96.12 percent global classification accuracy in the unencrypted domain and 90.02 percent in the encrypted domain.
comment : 7 pages, 4 figures
☆ TrafficLens: Multi-Camera Traffic Video Analysis Using LLMs SC
Md Adnan Arefeen, Biplob Debnath, Srimat Chakradhar
Traffic cameras are essential in urban areas, playing a crucial role in intelligent transportation systems. Multiple cameras at intersections enhance law enforcement capabilities, traffic management, and pedestrian safety. However, efficiently managing and analyzing multi-camera feeds poses challenges due to the vast amount of data. Analyzing such huge video data requires advanced analytical tools. While Large Language Models (LLMs) like ChatGPT, equipped with retrieval-augmented generation (RAG) systems, excel in text-based tasks, integrating them into traffic video analysis demands converting video data into text using a Vision-Language Model (VLM), which is time-consuming and delays the timely utilization of traffic videos for generating insights and investigating incidents. To address these challenges, we propose TrafficLens, a tailored algorithm for multi-camera traffic intersections. TrafficLens employs a sequential approach, utilizing overlapping coverage areas of cameras. It iteratively applies VLMs with varying token limits, using previous outputs as prompts for subsequent cameras, enabling rapid generation of detailed textual descriptions while reducing processing time. Additionally, TrafficLens intelligently bypasses redundant VLM invocations through an object-level similarity detector. Experimental results with real-world datasets demonstrate that TrafficLens reduces video-to-text conversion time by up to $4\times$ while maintaining information accuracy.
comment : 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC)
☆ Beyond Realism: Learning the Art of Expressive Composition with StickerNet
Haoming Lu, David Kocharian, Humphrey Shi
As a widely used operation in image editing workflows, image composition has traditionally been studied with a focus on achieving visual realism and semantic plausibility. However, in practical editing scenarios of the modern content creation landscape, many compositions are not intended to preserve realism. Instead, users of online platforms motivated by gaining community recognition often aim to create content that is more artistic, playful, or socially engaging. Taking inspiration from this observation, we define the expressive composition task, a new formulation of image composition that embraces stylistic diversity and looser placement logic, reflecting how users edit images on real-world creative platforms. To address this underexplored problem, we present StickerNet, a two-stage framework that first determines the composition type, then predicts placement parameters such as opacity, mask, location, and scale accordingly. Unlike prior work that constructs datasets by simulating object placements on real images, we directly build our dataset from 1.8 million editing actions collected on an anonymous online visual creation and editing platform, each reflecting user-community validated placement decisions. This grounding in authentic editing behavior ensures strong alignment between task definition and training supervision. User studies and quantitative evaluations show that StickerNet outperforms common baselines and closely matches human placement behavior, demonstrating the effectiveness of learning from real-world editing patterns despite the inherent ambiguity of the task. This work introduces a new direction in visual understanding that emphasizes expressiveness and user intent over realism.
☆ BUSTR: Breast Ultrasound Text Reporting with a Descriptor-Aware Vision -Language Model
Rawa Mohammed, Mina Attin, Bryar Shareef
Automated radiology report generation (RRG) for breast ultrasound (BUS) is limited by the lack of paired image-report datasets and the risk of hallucinations from large language models. We propose BUSTR, a multitask vision-language framework that generates BUS reports without requiring paired image-report supervision. BUSTR constructs reports from structured descriptors (e.g., BI-RADS, pathology, histology) and radiomics features, learns descriptor-aware visual representations with a multi-head Swin encoder trained using a multitask loss over dataset-specific descriptor sets, and aligns visual and textual tokens via a dual-level objective that combines token-level cross-entropy with a cosine-similarity alignment loss between input and output representations. We evaluate BUSTR on two public BUS datasets, BrEaST and BUS-BRA, which differ in size and available descriptors. Across both datasets, BUSTR consistently improves standard natural language generation metrics and clinical efficacy metrics, particularly for key targets such as BI-RADS category and pathology. Our results show that this descriptor-aware vision model, trained with a combined token-level and alignment loss, improves both automatic report metrics and clinical efficacy without requiring paired image-report data. The source code can be found at https://github.com/AAR-UNLV/BUSTR
comment : 13 pages, 2 figures, 6 tables
☆ ENACT: Evaluating Embodied Cognition with Wo rl d Modeling of Egocentric Interaction
Qineng Wang, Wenlong Huang, Yu Zhou, Hang Yin, Tianwei Bao, Jianwen Lyu, Weiyu Liu, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, Manling Li
Embodied cognition argues that intelligence arises from sensorimotor interaction rather than passive observation. It raises an intriguing question: do modern vision-language models (VLMs), trained largely in a disembodied manner, exhibit signs of embodied cognition? We introduce ENACT, a benchmark that casts evaluation of embodied cognition as world modeling from egocentric interaction in a visual question answering (VQA) format. Framed as a partially observable Markov decision process (POMDP) whose actions are scene graph changes, ENACT comprises two complementary sequence reordering tasks: forward world modeling (reorder shuffled observations given actions) and inverse world modeling (reorder shuffled actions given observations). While conceptually simple, solving these tasks implicitly demands capabilities central to embodied cognition-affordance recognition, action-effect reasoning, embodied awareness, and interactive, long-horizon memory from partially observable egocentric input, while avoiding low-level image synthesis that could confound the evaluation. We provide a scalable pipeline that synthesizes QA pairs from robotics simulation (BEHAVIOR) and evaluates models on 8,972 QA pairs spanning long-horizon home-scale activities. Experiments reveal a performance gap between frontier VLMs and humans that widens with interaction horizon. Models consistently perform better on the inverse task than the forward one and exhibit anthropocentric biases, including a preference for right-handed actions and degradation when camera intrinsics or viewpoints deviate from human vision. Website at https://enact-embodied-cognition.github.io/.
comment : Preprint version
☆ Digital Elevation Model Estimation from RGB Satellite Imagery using Generative Deep Learning
Alif Ilham Madani, Riska A. Kuswati, Alex M. Lechner, Muhamad Risqi U. Saputra
Digital Elevation Models (DEMs) are vital datasets for geospatial applications such as hydrological modeling and environmental monitoring. However, conventional methods to generate DEM, such as using LiDAR and photogrammetry, require specific types of data that are often inaccessible in resource-constrained settings. To alleviate this problem, this study proposes an approach to generate DEM from freely available RGB satellite imagery using generative deep learning, particularly based on a conditional Generative Adversarial Network (GAN). We first developed a global dataset consisting of 12K RGB-DEM pairs using Landsat satellite imagery and NASA's SRTM digital elevation data, both from the year 2000. A unique preprocessing pipeline was implemented to select high-quality, cloud-free regions and aggregate normalized RGB composites from Landsat imagery. Additionally, the model was trained in a two-stage process, where it was first trained on the complete dataset and then fine-tuned on high-quality samples filtered by Structural Similarity Index Measure (SSIM) values to improve performance on challenging terrains. The results demonstrate promising performance in mountainous regions, achieving an overall mean root-mean-square error (RMSE) of 0.4671 and a mean SSIM score of 0.2065 (scale -1 to 1), while highlighting limitations in lowland and residential areas. This study underscores the importance of meticulous preprocessing and iterative refinement in generative modeling for DEM generation, offering a cost-effective and adaptive alternative to conventional methods while emphasizing the challenge of generalization across diverse terrains worldwide.
comment : 5 pages, 4 figures, accepted at IGARSS 2025 conference
☆ PPBoost: Progressive Prompt Boosting for Text-Driven Medical Image Segmentation
Xuchen Li, Hengrui Gu, Mohan Zhang, Qin Liu, Zhen Tan, Xinyuan Zhu, Huixue Zhou, Tianlong Chen, Kaixiong Zhou
Text-prompted foundation models for medical image segmentation offer an intuitive way to delineate anatomical structures from natural language queries, but their predictions often lack spatial precision and degrade under domain shift. In contrast, visual-prompted models achieve strong segmentation performance across diverse modalities by leveraging spatial cues of precise bounding-box (bbox) prompts to guide the segmentation of target lesions. However, it is costly and challenging to obtain the precise visual prompts in clinical practice. We propose PPBoost (Progressive Prompt-Boosting), a framework that bridges these limitations by transforming weak text-derived signals into strong, spatially grounded visual prompts, operating under a strict zero-shot regime with no image- or pixel-level segmentation labels. PPBoost first uses a vision-language model to produce initial pseudo-bboxes conditioned on the textual object descriptions and applies an uncertainty-aware criterion to filter unreliable predictions. The retained image-bboxes pairs are then leveraged to train a pseudo-labeled detector, producing the high-quality bboxes for the query images. During inference, PPBoost further refines the generated bboxes by appropriately expanding them to tightly cover the target anatomical structures. The enhanced spatially-grounding bbox prompts guide existing segmentation models to generate final dense masks, effectively amplifying weak text cues into strong spatial guidance. Across three datasets spanning diverse modalities and anatomies, PPBoost consistently improves Dice and Normalized Surface Distance over text- and visual-prompted baselines and, notably, surpasses few-shot segmentation models without using labeled data. PPBoost can generalize to multiple typical visual segmentation model backbones.
☆ DialBench: Towards Accurate Reading Recognition of Pointer Meter using Large Foundation Models
Futian Wang, Chaoliu Weng, Xiao Wang, Zhen Chen, Zhicheng Zhao, Jin Tang
The precise reading recognition of pointer meters plays a key role in smart power systems, but existing approaches remain fragile due to challenges like reflections, occlusions, dynamic viewing angles, and overly between thin pointers and scale markings. Up to now, this area still lacks large-scale datasets to support the development of robust algorithms. To address these challenges, this paper first presents a new large-scale benchmark dataset for dial reading, termed RPM-10K, which contains 10730 meter images that fully reflect the aforementioned key challenges. Built upon the dataset, we propose a novel vision-language model for pointer meter reading recognition, termed MRLM, based on physical relation injection. Instead of exhaustively learning image-level correlations, MRLM explicitly encodes the geometric and causal relationships between the pointer and the scale, aligning perception with physical reasoning in the spirit of world-model perspectives. Through cross-attentional fusion and adaptive expert selection, the model learns to interpret dial configurations and generate precise numeric readings. Extensive experiments fully validated the effectiveness of our proposed framework on the newly proposed benchmark dataset. Both the dataset and source code will be released on https://github.com/Event-AHU/DialBench
☆ PAT3D: Physics-Augmented Text-to-3D Scene Generation
Guying Lin, Kemeng Huang, Michael Liu, Ruihan Gao, Hanke Chen, Lyuhao Chen, Beijia Lu, Taku Komura, Yuan Liu, Jun-Yan Zhu, Minchen Li
We introduce PAT3D, the first physics-augmented text-to-3D scene generation framework that integrates vision-language models with physics-based simulation to produce physically plausible, simulation-ready, and intersection-free 3D scenes. Given a text prompt, PAT3D generates 3D objects, infers their spatial relations, and organizes them into a hierarchical scene tree, which is then converted into initial conditions for simulation. A differentiable rigid-body simulator ensures realistic object interactions under gravity, driving the scene toward static equilibrium without interpenetrations. To further enhance scene quality, we introduce a simulation-in-the-loop optimization procedure that guarantees physical stability and non-intersection, while improving semantic consistency with the input prompt. Experiments demonstrate that PAT3D substantially outperforms prior approaches in physical plausibility, semantic consistency, and visual quality. Beyond high-quality generation, PAT3D uniquely enables simulation-ready 3D scenes for downstream tasks such as scene editing and robotic manipulation. Code and data will be released upon acceptance.
comment : 19 pages, 12 figures
☆ DeepGI: Explainable Deep Learning for Gastrointestinal Image Classification
Walid Houmaidi, Mohamed Hadadi, Youssef Sabiri, Yousra Chtouki
This paper presents a comprehensive comparative model analysis on a novel gastrointestinal medical imaging dataset, comprised of 4,000 endoscopic images spanning four critical disease classes: Diverticulosis, Neoplasm, Peritonitis, and Ureters. Leveraging state-of-the-art deep learning techniques, the study confronts common endoscopic challenges such as variable lighting, fluctuating camera angles, and frequent imaging artifacts. The best performing models, VGG16 and MobileNetV2, each achieved a test accuracy of 96.5%, while Xception reached 94.24%, establishing robust benchmarks and baselines for automated disease classification. In addition to strong classification performance, the approach includes explainable AI via Grad-CAM visualization, enabling identification of image regions most influential to model predictions and enhancing clinical interpretability. Experimental results demonstrate the potential for robust, accurate, and interpretable medical image analysis even in complex real-world conditions. This work contributes original benchmarks, comparative insights, and visual explanations, advancing the landscape of gastrointestinal computer-aided diagnosis and underscoring the importance of diverse, clinically relevant datasets and model explainability in medical AI research.
comment : 7 pages, 4 figures, 2 tables. Accepted at DASET 2026
☆ WalkCLIP: Multimodal Learning for Urban Walkability Prediction
Shilong Xiang, JangHyeon Lee, Min Namgung, Yao-Yi Chiang
Urban walkability is a cornerstone of public health, sustainability, and quality of life. Traditional walkability assessments rely on surveys and field audits, which are costly and difficult to scale. Recent studies have used satellite imagery, street view imagery, or population indicators to estimate walkability, but these single-source approaches capture only one dimension of the walking environment. Satellite data describe the built environment from above, but overlook the pedestrian perspective. Street view imagery captures conditions at the ground level, but lacks broader spatial context. Population dynamics reveal patterns of human activity but not the visual form of the environment. We introduce WalkCLIP, a multimodal framework that integrates these complementary viewpoints to predict urban walkability. WalkCLIP learns walkability-aware vision-language representations from GPT-4o generated image captions, refines these representations with a spatial aggregation module that incorporates neighborhood context, and fuses the resulting features with representations from a population dynamics foundation model. Evaluated at 4,660 locations throughout Minneapolis-Saint Paul, WalkCLIP outperforms unimodal and multimodal baselines in both predictive accuracy and spatial alignment. These results show that the integration of visual and behavioral signals yields reliable predictions of the walking environment.
☆ TAPVid-360: Tracking Any Point in 360 from Narrow Field of View Video
Finlay G. C. Hudson, James A. D. Gardner, William A. P. Smith
Humans excel at constructing panoramic mental models of their surroundings, maintaining object permanence and inferring scene structure beyond visible regions. In contrast, current artificial vision systems struggle with persistent, panoramic understanding, often processing scenes egocentrically on a frame-by-frame basis. This limitation is pronounced in the Track Any Point (TAP) task, where existing methods fail to track 2D points outside the field of view. To address this, we introduce TAPVid-360, a novel task that requires predicting the 3D direction to queried scene points across a video sequence, even when far outside the narrow field of view of the observed video. This task fosters learning allocentric scene representations without needing dynamic 4D ground truth scene models for training. Instead, we exploit 360 videos as a source of supervision, resampling them into narrow field-of-view perspectives while computing ground truth directions by tracking points across the full panorama using a 2D pipeline. We introduce a new dataset and benchmark, TAPVid360-10k comprising 10k perspective videos with ground truth directional point tracking. Our baseline adapts CoTracker v3 to predict per-point rotations for direction updates, outperforming existing TAP and TAPVid 3D methods.
☆ AmodalGen3D: Generative Amodal 3D Object Reconstruction from Sparse Unposed Views
Junwei Zhou, Yu-Wing Tai
Reconstructing 3D objects from a few unposed and partially occluded views is a common yet challenging problem in real-world scenarios, where many object surfaces are never directly observed. Traditional multi-view or inpainting-based approaches struggle under such conditions, often yielding incomplete or geometrically inconsistent reconstructions. We introduce AmodalGen3D, a generative framework for amodal 3D object reconstruction that infers complete, occlusion-free geometry and appearance from arbitrary sparse inputs. The model integrates 2D amodal completion priors with multi-view stereo geometry conditioning, supported by a View-Wise Cross Attention mechanism for sparse-view feature fusion and a Stereo-Conditioned Cross Attention module for unobserved structure inference. By jointly modeling visible and hidden regions, AmodalGen3D faithfully reconstructs 3D objects that are consistent with sparse-view constraints while plausibly hallucinating unseen parts. Experiments on both synthetic and real-world datasets demonstrate that AmodalGen3D achieves superior fidelity and completeness under occlusion-heavy sparse-view settings, addressing a pressing need for object-level 3D scene reconstruction in robotics, AR/VR, and embodied AI applications.
comment : 18 pages, 14 figures
☆ Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomics
Yupei Zhang, Yating Huang, Wanming Hu, Lequan Yu, Hujun Yin, Chao Li
Multimodal approaches that integrate histology and genomics hold strong potential for precision oncology. However, phenotypic and genotypic heterogeneity limits the quality of intra-modal representations and hinders effective inter-modal integration. Furthermore, most existing methods overlook real-world clinical scenarios where genomics may be partially missing or entirely unavailable. We propose a flexible multimodal prototyping framework to integrate whole slide images and incomplete genomics for precision oncology. Our approach has four key components: 1) Biological Prototyping using text prompting and prototype-wise weighting; 2) Multiview Alignment through sample- and distribution-wise alignments; 3) Bipartite Fusion to capture both shared and modality-specific information for multimodal fusion; and 4) Semantic Genomics Imputation to handle missing data. Extensive experiments demonstrate the consistent superiority of the proposed method compared to other state-of-the-art approaches on multiple downstream tasks. The code is available at https://github.com/helenypzhang/Interpretable-Multimodal-Prototyping.
☆ Comparing SAM 2 and SAM 3 for Zero-Shot Segmentation of 3D Medical Data
Satrajit Chakrabarty, Ravi Soni
Foundation models for promptable segmentation, including SAM, SAM 2, and the recently released SAM 3, have renewed interest in zero-shot segmentation of medical imaging. Although these models perform strongly on natural images, their behavior on medical data remains insufficiently characterized. While SAM 2 is widely used for annotation in 3D medical workflows, SAM 3 introduces a new perception backbone, detector-tracker pipeline, and concept-level prompting that may alter its behavior under spatial prompts. We present the first controlled comparison of SAM 2 and SAM 3 for zero-shot segmentation of 3D medical volumes and videos under purely visual prompting, with concept mechanisms disabled. We assess whether SAM 3 can serve as an out-of-the-box replacement for SAM 2 without customization. We benchmark both models on 16 public datasets (CT, MRI, 3D and cine ultrasound, endoscopy) covering 54 anatomical structures, pathologies, and surgical instruments. Prompts are restricted to the first frame and use four modes: single-click, multi-click, bounding box, and dense mask. This design standardizes preprocessing, prompt placement, propagation rules, and metric computation to disentangle prompt interpretation from propagation. Prompt-frame analysis shows that SAM 3 provides substantially stronger initialization than SAM 2 for click prompting across most structures. In full-volume analysis, SAM 3 retains this advantage for complex, vascular, and soft-tissue anatomies, emerging as the more versatile general-purpose segmenter. While SAM 2 remains competitive for compact, rigid organs under strong spatial guidance, it frequently fails on challenging targets where SAM 3 succeeds. Overall, our results suggest that SAM 3 is the superior default choice for most medical segmentation tasks, particularly those involving sparse user interaction or complex anatomical topology.
☆ Adaptive Parameter Optimization for Robust Remote Photoplethysmography
Cecilia G. Morales, Fanurs Chi En Teh, Kai Li, Pushpak Agrawal, Artur Dubrawski
Remote photoplethysmography (rPPG) enables contactless vital sign monitoring using standard RGB cameras. However, existing methods rely on fixed parameters optimized for particular lighting conditions and camera setups, limiting adaptability to diverse deployment environments. This paper introduces the Projection-based Robust Signal Mixing (PRISM) algorithm, a training-free method that jointly optimizes photometric detrending and color mixing through online parameter adaptation based on signal quality assessment. PRISM achieves state-of-the-art performance among unsupervised methods, with MAE of 0.77 bpm on PURE and 0.66 bpm on UBFC-rPPG, and accuracy of 97.3\% and 97.5\% respectively at a 5 bpm threshold. Statistical analysis confirms PRISM performs equivalently to leading supervised methods ($p > 0.2$), while maintaining real-time CPU performance without training. This validates that adaptive time series optimization significantly improves rPPG across diverse conditions.
comment : Accepted in Times Series for Health NeurIPs Workshop 2025
☆ Path Reasoning : A multimodal reasoning agent for query-based ROI navigation on whole-slide images
Kunpeng Zhang, Hanwen Xu, Sheng Wang
Deciphering tumor microenvironment from Whole Slide Images (WSIs) is intriguing as it is key to cancer diagnosis, prognosis and treatment response. While these gigapixel images on one hand offer a comprehensive portrait of cancer, on the other hand, the extremely large size, as much as more than 10 billion pixels, make it challenging and time-consuming to navigate to corresponding regions to support diverse clinical inspection. Inspired by pathologists who conducted navigation on WSIs with a combination of sampling, reasoning and self-reflection, we proposed "PathReasoning", a multi-modal reasoning agent that iteratively navigates across WSIs through multiple rounds of reasoning and refinements. Specifically, starting with randomly sampled candidate regions, PathReasoning reviews current selections with self-reflection, reasoning over the correspondence between visual observations and clinical questions, and concludes by proposing new regions to explore. Across rounds, PathReasoning builds a reasoning chain that gradually directs attention to diagnostically relevant areas. PathReasoning turns each whole slide into a sequence of question-guided views, allowing the model to efficiently find informative ROIs within a fixed number of steps, without the need for dense pixel-level annotations. PathReasoning can substantially outperform strong ROI-selection approaches by 6.7% and 3.1% of AUROC on subtyping and longitudinal analysis tasks. The high-quality ROIs further support accurate report generation on breast cancer, significantly outperforming the standard GPT-4o by 10% in accuracy. PathReasoning prioritizes question-specific regions and constructs interpretable reasoning chains, supporting efficient slide review, consistent diagnostic interpretations, comprehensive reporting, and evidence traceability in digital pathology.
☆ UniArt: Unified 3D Representation for Generating 3D Articulated Objects with Open-Set Articulation
Bu Jin, Weize Li, Songen Gu, Yupeng Zheng, Yuhang Zheng, Zhengyi Zhou, Yao Yao
Articulated 3D objects play a vital role in realistic simulation and embodied robotics, yet manually constructing such assets remains costly and difficult to scale. In this paper, we present UniArt, a diffusion-based framework that directly synthesizes fully articulated 3D objects from a single image in an end-to-end manner. Unlike prior multi-stage techniques, UniArt establishes a unified latent representation that jointly encodes geometry, texture, part segmentation, and kinematic parameters. We introduce a reversible joint-to-voxel embedding, which spatially aligns articulation features with volumetric geometry, enabling the model to learn coherent motion behaviors alongside structural formation. Furthermore, we formulate articulation type prediction as an open-set problem, removing the need for fixed joint semantics and allowing generalization to novel joint categories and unseen object types. Experiments on the PartNet-Mobility benchmark demonstrate that UniArt achieves state-of-the-art mesh quality and articulation accuracy.
☆ Closed-Loop Transformer s: Autoregressive Modeling as Iterative Latent Equilibrium
Akbar Anbar Jafari, Gholamreza Anbarjafari
Contemporary autoregressive transformers operate in open loop: each hidden state is computed in a single forward pass and never revised, causing errors to propagate uncorrected through the sequence. We identify this open-loop bottleneck as a fundamental architectural limitation underlying well-documented failures in long-range reasoning, factual consistency, and multi-step planning. To address this limitation, we introduce the closed-loop prediction principle, which requires that models iteratively refine latent representations until reaching a self-consistent equilibrium before committing to each token. We instantiate this principle as Equilibrium Transformers (EqT), which augment standard transformer layers with an Equilibrium Refinement Module that minimizes a learned energy function via gradient descent in latent space. The energy function enforces bidirectional prediction consistency, episodic memory coherence, and output confidence, all computed without external supervision. Theoretically, we prove that EqT performs approximate MAP inference in a latent energy-based model, establish linear convergence guarantees, and show that refinement improves predictions precisely on hard instances where one-shot inference is suboptimal. The framework unifies deep equilibrium models, diffusion language models, and test-time training as special cases. Preliminary experiments on the binary parity task demonstrate +3.28% average improvement on challenging sequences, with gains reaching +8.07% where standard transformers approach random performance, validating that the benefit of deliberation scales with task difficulty. Just as attention mechanisms resolved the sequential bottleneck of recurrent networks, we propose that closed-loop equilibrium may resolve the commitment bottleneck of open-loop autoregression, representing a foundational step toward language models.
comment : 22 pages, 1 figure, 1 table
☆ Saddle-Free Guidance: Improved On-Manifold Sampling without Labels or Additional Training
Eric Yeats, Darryl Hannan, Wilson Fearn, Timothy Doster, Henry Kvinge, Scott Mahan
Score-based generative models require guidance in order to generate plausible, on-manifold samples. The most popular guidance method, Classifier-Free Guidance (CFG), is only applicable in settings with labeled data and requires training an additional unconditional score-based model. More recently, Auto-Guidance adopts a smaller, less capable version of the original model to guide generation. While each method effectively promotes the fidelity of generated data, each requires labeled data or the training of additional models, making it challenging to guide score-based models when (labeled) training data are not available or training new models is not feasible. We make the surprising discovery that the positive curvature of log density estimates in saddle regions provides strong guidance for score-based models. Motivated by this, we develop saddle-free guidance (SFG) which maintains estimates of maximal positive curvature of the log density to guide individual score-based models. SFG has the same computational cost of classifier-free guidance, does not require additional training, and works with off-the-shelf diffusion and flow matching models. Our experiments indicate that SFG achieves state-of-the-art FID and FD-DINOv2 metrics in single-model unconditional ImageNet-512 generation. When SFG is combined with Auto-Guidance, its unconditional samples achieve general state-of-the-art in FD-DINOv2 score. Our experiments with FLUX.1-dev and Stable Diffusion v3.5 indicate that SFG boosts the diversity of output images compared to CFG while maintaining excellent prompt adherence and image fidelity.
☆ Evaluating Strategies for Synthesizing Clinical Notes for Medical Multimodal AI
Niccolo Marini, Zhaohui Liang, Sivaramakrishnan Rajaraman, Zhiyun Xue, Sameer Antani
Multimodal (MM) learning is emerging as a promising paradigm in biomedical artificial intelligence (AI) applications, integrating complementary modality, which highlight different aspects of patient health. The scarcity of large heterogeneous biomedical MM data has restrained the development of robust models for medical AI applications. In the dermatology domain, for instance, skin lesion datasets typically include only images linked to minimal metadata describing the condition, thereby limiting the benefits of MM data integration for reliable and generalizable predictions. Recent advances in Large Language Models (LLMs) enable the synthesis of textual description of image findings, potentially allowing the combination of image and text representations. However, LLMs are not specifically trained for use in the medical domain, and their naive inclusion has raised concerns about the risk of hallucinations in clinically relevant contexts. This work investigates strategies for generating synthetic textual clinical notes, in terms of prompt design and medical metadata inclusion, and evaluates their impact on MM architectures toward enhancing performance in classification and cross-modal retrieval tasks. Experiments across several heterogeneous dermatology datasets demonstrate that synthetic clinical notes not only enhance classification performance, particularly under domain shift, but also unlock cross-modal retrieval capabilities, a downstream task that is not explicitly optimized during training.
☆ $\mathcal{E}_0$: Enhancing Generalization and Fine-Grained Control in VLA Models via Continuized Discrete Diffusion
Zhihao Zhan, Jiaying Zhou, Likui Zhang, Qinhan Lv, Hao Liu, Jusheng Zhang, Weizheng Li, Ziliang Chen, Tianshui Chen, Keze Wang, Liang Lin, Guangrun Wang
Vision-Language-Action (VLA) models offer a unified framework for robotic manipulation by integrating visual perception, language understanding, and control generation. Yet existing VLA models still struggle to generalize across diverse tasks, scenes, and camera viewpoints, and often produce coarse or unstable actions. We introduce E0, a continuized discrete diffusion framework that formulates action generation as iterative denoising over quantized action tokens. Compared with continuous diffusion policies, E0 offers two key advantages: (1) discrete action tokens align naturally with the symbolic structure of pretrained VLM/VLA backbones, enabling stronger semantic conditioning; and 2. discrete diffusion matches the true quantized nature of real-world robot control-whose hardware constraints (e.g., encoder resolution, control frequency, actuation latency) inherently discretize continuous signals-and therefore benefits from a Bayes-optimal denoiser that models the correct discrete action distribution, leading to stronger generalization. Compared with discrete autoregressive and mask-based discrete diffusion models, E0 supports a significantly larger and finer-grained action vocabulary and avoids the distributional mismatch introduced by masking-based corruptions-yielding more accurate fine-grained action control. We further introduce a spherical viewpoint perturbation augmentation method to improve robustness to camera shifts without additional data. Experiments on LIBERO, VLABench, and ManiSkill show that E0 achieves state-of-the-art performance across 14 diverse environments, outperforming strong baselines by 10.7% on average. Real-world evaluation on a Franka arm confirms that E0 delivers precise, robust, and transferable manipulation, establishing discrete diffusion as a promising direction for generalizable VLA policy learning.
♻ ☆ TimeViper: A Hybrid Mamba- Transformer Vision -Language Model for Efficient Long Video Understanding
Boshen Xu, Zihan Xiao, Jiaze Li, Jianzhong Ju, Zhenbo Luo, Jian Luan, Qin Jin
We introduce TimeViper, a hybrid vision-language model designed to tackle challenges of long video understanding. Processing long videos demands both an efficient model architecture and an effective mechanism for handling extended temporal contexts. To this end, TimeViper adopts a hybrid Mamba-Transformer backbone that combines the efficiency of state-space models with the expressivity of attention mechanisms. Through this hybrid design, we reveal the vision-to-text information aggregation phenomenon, where information progressively flows from vision tokens to text tokens across increasing LLM depth, resulting in severe vision token redundancy. Motivated by this observation, we propose TransV, a token information transfer module that transfers and compresses vision tokens into instruction tokens while maintaining multimodal understanding capabilities. This design enables TimeViper to process hour-long videos exceeding 10,000 frames. Extensive experiments across multiple benchmarks demonstrate that TimeViper competes with state-of-the-art models while extending frame numbers. We further analyze attention behaviors of both Mamba and Transformer layers, offering new insights into hybrid model interpretability. This work represents an initial step towards developing, interpreting, and compressing hybrid Mamba-Transformer architectures.
comment : Project page: https://xuboshen.github.io/TimeViper; Code: https://github.com/xiaomi-research/timeviper
♻ ☆ Sequence-Adaptive Video Prediction in Continuous Streams using Diffusion Noise Optimization
Sina Mokhtarzadeh Azar, Emad Bahrami, Enrico Pallotta, Gianpiero Francesca, Radu Timofte, Juergen Gall
In this work, we investigate diffusion-based video prediction models, which forecast future video frames, for continuous video streams. In this context, the models observe continuously new training samples, and we aim to leverage this to improve their predictions. We thus propose an approach that continuously adapts a pre-trained diffusion model to a video stream. Since fine-tuning the parameters of a large diffusion model is too expensive, we refine the diffusion noise during inference while keeping the model parameters frozen, allowing the model to adaptively determine suitable sampling noise. We term the approach Sequence Adaptive Video Prediction with Diffusion Noise Optimization (SAVi-DNO). To validate our approach, we introduce a new evaluation setting on the Ego4D dataset, focusing on simultaneous adaptation and evaluation on long continuous videos. Empirical results demonstrate improved performance based on FVD, SSIM, and PSNR metrics on long videos of Ego4D and OpenDV-YouTube, as well as videos of UCF-101 and SkyTimelapse, showcasing SAVi-DNO's effectiveness.
♻ ☆ TinyChemVL: Advancing Chemical Vision -Language Models via Efficient Visual Token Reduction and Complex Reaction Tasks AAAI 2026
Xuanle Zhao, Shuxin Zeng, Xinyuan Cai, Xiang Cheng, Duzhen Zhang, Xiuyi Chen, Bo Xu
While Vision Language Models (VLMs) have demonstrated remarkable capabilities in general visual understanding, their application in the chemical domain has been limited, with previous works predominantly focusing on text and thus overlooking critical visual information, such as molecular structures. Current approaches that directly adopt standard VLMs for chemical tasks suffer from two primary issues: (i) computational inefficiency of processing entire chemical images with non-informative backgrounds. (ii) a narrow scope on molecular-level tasks that restricts progress in chemical reasoning. In this work, we propose \textbf{TinyChemVL}, an efficient and powerful chemical VLM that leverages visual token reduction and reaction-level tasks to improve model efficiency and reasoning capacity. Also, we propose \textbf{ChemRxn-V}, a reaction-level benchmark for assessing vision-based reaction recognition and prediction tasks. Directly predicting reaction products from molecular images poses a non-trivial challenge, as it requires models to integrate both recognition and reasoning capacities. Our results demonstrate that with only 4B parameters, TinyChemVL achieves superior performance on both molecular and reaction tasks while demonstrating faster inference and training speeds compared to existing models. Notably, TinyChemVL outperforms ChemVLM while utilizing only 1/16th of the visual tokens. This work builds efficient yet powerful VLMs for chemical domains by co-designing model architecture and task complexity.
comment : Accepted by AAAI 2026
♻ ☆ Adapting Segment Anything Model for Power Transmission Corridor Hazard Segmentation
Hang Chen, Maoyuan Ye, Peng Yang, Haibin He, Juhua Liu, Bo Du
Power transmission corridor hazard segmentation (PTCHS) aims to separate transmission equipment and surrounding hazards from complex background, conveying great significance to maintaining electric power transmission safety. Recently, the Segment Anything Model (SAM) has emerged as a foundational vision model and pushed the boundaries of segmentation tasks. However, SAM struggles to deal with the target objects in complex transmission corridor scenario, especially those with fine structure. In this paper, we propose ELE-SAM, adapting SAM for the PTCHS task. Technically, we develop a Context-Aware Prompt Adapter to achieve better prompt tokens via incorporating global-local features and focusing more on key regions. Subsequently, to tackle the hazard objects with fine structure in complex background, we design a High-Fidelity Mask Decoder by leveraging multi-granularity mask features and then scaling them to a higher resolution. Moreover, to train ELE-SAM and advance this field, we construct the ELE-40K benchmark, the first large-scale and real-world dataset for PTCHS including 44,094 image-mask pairs. Experimental results for ELE-40K demonstrate the superior performance that ELE-SAM outperforms the baseline model with the average 16.8% mIoU and 20.6% mBIoU performance improvement. Moreover, compared with the state-of-the-art method on HQSeg-44K, the average 2.9% mIoU and 3.8% mBIoU absolute improvements further validate the effectiveness of our method on high-quality generic object segmentation. The source code and dataset are available at https://github.com/Hhaizee/ELE-SAM.
♻ ☆ Force Prompt ing: Video Generation Models Can Learn and Generalize Physics-based Control Signals NeurIPS 2025
Nate Gillman, Charles Herrmann, Michael Freeman, Daksh Aggarwal, Evan Luo, Deqing Sun, Chen Sun
Recent advances in video generation models have sparked interest in world models capable of simulating realistic environments. While navigation has been well-explored, physically meaningful interactions that mimic real-world forces remain largely understudied. In this work, we investigate using physical forces as a control signal for video generation and propose force prompts which enable users to interact with images through both localized point forces, such as poking a plant, and global wind force fields, such as wind blowing on fabric. We demonstrate that these force prompts can enable videos to respond realistically to physical control signals by leveraging the visual and motion prior in the original pretrained model, without using any 3D asset or physics simulator at inference. The primary challenge of force prompting is the difficulty in obtaining high quality paired force-video training data, both in the real world due to the difficulty of obtaining force signals, and in synthetic data due to limitations in the visual quality and domain diversity of physics simulators. Our key finding is that video generation models can generalize remarkably well when adapted to follow physical force conditioning from videos synthesized by Blender, even with limited demonstrations of few objects. Our method can generate videos which simulate forces across diverse geometries, settings, and materials. We also try to understand the source of this generalization and perform ablations that reveal two key elements: visual diversity and the use of specific text keywords during training. Our approach is trained on only around 15k training examples for a single day on four A100 GPUs, and outperforms existing methods on force adherence and physics realism, bringing world models closer to real-world physics interactions. We release all datasets, code, weights, and interactive video demos at our project page.
comment : Camera ready version (NeurIPS 2025). Code and interactive demos at https://force-prompting.github.io/
♻ ☆ A Gray-box Attack against Latent Diffusion Model-based Image Editing by Posterior Collapse
Zhongliang Guo, Chun Tong Lei, Lei Fang, Shuai Zhao, Yifei Qian, Jingyu Lin, Zeyu Wang, Cunjian Chen, Ognjen Arandjelović, Chun Pong Lau
Recent advancements in Latent Diffusion Models (LDMs) have revolutionized image synthesis and manipulation, raising significant concerns about data misappropriation and intellectual property infringement. While adversarial attacks have been extensively explored as a protective measure against such misuse of generative AI, current approaches are severely limited by their heavy reliance on model-specific knowledge and substantial computational costs. Drawing inspiration from the posterior collapse phenomenon observed in VAE training, we propose the Posterior Collapse Attack (PCA), a novel framework for protecting images from unauthorized manipulation. Through comprehensive theoretical analysis and empirical validation, we identify two distinct collapse phenomena during VAE inference: diffusion collapse and concentration collapse. Based on this discovery, we design a unified loss function that can flexibly achieve both types of collapse through parameter adjustment, each corresponding to different protection objectives in preventing image manipulation. Our method significantly reduces dependence on model-specific knowledge by requiring access to only the VAE encoder, which constitutes less than 4\% of LDM parameters. Notably, PCA achieves prompt-invariant protection by operating on the VAE encoder before text conditioning occurs, eliminating the need for empty prompt optimization required by existing methods. This minimal requirement enables PCA to maintain adequate transferability across various VAE-based LDM architectures while effectively preventing unauthorized image editing. Extensive experiments show PCA outperforms existing techniques in protection effectiveness, computational efficiency (runtime and VRAM), and generalization across VAE-based LDM variants. Our code is available at https://github.com/ZhongliangGuo/PosteriorCollapseAttack.
comment : 15 pages, 9 figures, 9 tables
♻ ☆ LMLCC-Net: A Semi-Supervised Deep Learning Model for Lung Nodule Malignancy Prediction from CT Scans using a Novel Hounsfield Unit-Based Intensity Filtering
Tasnia Binte Mamun, Adhora Madhuri, Nusaiba Sobir, Taufiq Hasan
Lung cancer is the leading cause of patient mortality in the world. Early diagnosis of malignant pulmonary nodules in CT images can have a significant impact on reducing disease mortality and morbidity. In this work, we propose LMLCC-Net, a novel deep learning framework for classifying nodules from CT scan images using a 3D CNN, considering Hounsfield Unit (HU)-based intensity filtering. Benign and malignant nodules have significant differences in their intensity profile of HU, which was not exploited in the literature. Our method considers the intensity pattern as well as the texture for the prediction of malignancies. LMLCC-Net extracts features from multiple branches that each use a separate learnable HU-based intensity filtering stage. Various combinations of branches and learnable ranges of filters were explored to finally produce the best-performing model. In addition, we propose a semi-supervised learning scheme for labeling ambiguous cases and also developed a lightweight model to classify the nodules. The experimental evaluations are carried out on the LUNA16 dataset. The proposed LMLCC-Net was evaluated using the LUNA16 dataset. Our proposed method achieves a classification accuracy of 91.96%, a sensitivity of 92.94%, and an area under the curve of 94.07%, showing improved performance compared to existing methods The proposed method can have a significant impact in helping radiologists in the classification of pulmonary nodules and improving patient care.
comment : 12 pages, 9 figures, 7 tables
♻ ☆ Connecting the Dots: Training-Free Visual Grounding via Agent ic Reasoning AAAI 2026
Liqin Luo, Guangyao Chen, Xiawu Zheng, Yongxing Dai, Yixiong Zou, Yonghong Tian
Visual grounding, the task of linking textual queries to specific regions within images, plays a pivotal role in vision-language integration. Existing methods typically rely on extensive task-specific annotations and fine-tuning, limiting their ability to generalize effectively to novel or out-of-distribution scenarios. To address these limitations, we introduce GroundingAgent, a novel agentic visual grounding framework that operates without any task-specific fine-tuning. GroundingAgent employs a structured, iterative reasoning mechanism that integrates pretrained open-vocabulary object detectors, multimodal large language models (MLLMs), and large language models (LLMs) to progressively refine candidate regions through joint semantic and spatial analyses. Remarkably, GroundingAgent achieves an average zero-shot grounding accuracy of 65.1 % on widely-used benchmarks (RefCOCO, RefCOCO+, RefCOCOg), entirely without fine-tuning. Furthermore, by substituting MLLM-generated captions with the original query texts, the accuracy at the selection stage alone reaches approximately 90 %, closely matching supervised performance and underscoring the critical role of LLM reasoning capabilities. GroundingAgent also offers strong interpretability, transparently illustrating each reasoning step and providing clear insights into its decision-making process.
comment : AAAI 2026
♻ ☆ Improved Visually Prompt ed Keyword Localisation in Real Low-Resource Settings
Leanne Nortje, Dan Oneata, Gabriel Pirlogeanu, Herman Kamper
Given an image query, visually prompted keyword localisation (VPKL) aims to find occurrences of the depicted word in a speech collection. This can be useful when transcriptions are not available for a low-resource language (e.g. if it is unwritten). Previous work showed that VPKL can be performed with a visually grounded speech model trained on paired images and unlabelled speech. But all experiments were done on English. Moreover, transcriptions were used to get positive and negative pairs for the contrastive loss. This paper introduces a few-shot learning scheme to mine pairs automatically without transcriptions. On English, this results in only a small drop in performance. We also - for the first time - consider VPKL on a real low-resource language, Yoruba. While scores are reasonable, here we see a bigger drop in performance compared to using ground truth pairs because the mining is less accurate in Yoruba.
comment : Accepted at SpeD 2025
♻ ☆ Modular, On-Site Solutions with Lightweight Anomaly Detection for Sustainable Nutrient Management in Agriculture
Abigail R. Cohen, Yuming Sun, Zhihao Qin, Harsh S. Muriki, Zihao Xiao, Yeonju Lee, Matthew Housley, Andrew F. Sharkey, Rhuanito S. Ferrarezi, Jing Li, Lu Gan, Yongsheng Chen
Efficient nutrient management is critical for crop growth and sustainable resource consumption (e.g., nitrogen, energy). Current approaches require lengthy analyses, preventing real-time optimization; similarly, imaging facilitates rapid phenotyping but can be computationally intensive, preventing deployment under resource constraints. This study proposes a flexible, tiered pipeline for anomaly detection and status estimation (fresh weight, dry mass, and tissue nutrients), including a comprehensive energy analysis of approaches that span the efficiency-accuracy spectrum. Using a nutrient depletion experiment with three treatments (T1-100%, T2-50%, and T3-25% fertilizer strength) and multispectral imaging (MSI), we developed a hierarchical pipeline using an autoencoder (AE) for early warning. Further, we compared two status estimation modules of different complexity for more detailed analysis: vegetation index (VI) features with machine learning (Random Forest, RF) and raw whole-image deep learning (Vision Transformer, ViT). Results demonstrated high-efficiency anomaly detection (73% net detection of T3 samples 9 days after transplanting) at substantially lower energy than embodied energy in wasted nitrogen. The state estimation modules show trade-offs, with ViT outperforming RF on phosphorus and calcium estimation (R2 0.61 vs. 0.58, 0.48 vs. 0.35) at higher energy cost. With our modular pipeline, this work opens opportunities for edge diagnostics and practical opportunities for agricultural sustainability.
♻ ☆ MeshCone: Second-Order Cone Programming for Geometrically-Constrained Mesh Enhancement
Alexander Valverde
Modern mesh generation pipelines whether learning-based or classical often produce outputs requiring post-processing to achieve production-quality geometry. This work introduces MeshCone, a convex optimization framework for guided mesh refinement that leverages reference geometry to correct deformed or degraded meshes. We formulate the problem as a second-order cone program where vertex positions are optimized to align with target geometry while enforcing smoothness through convex edge-length regularization. MeshCone performs geometry-aware optimization that preserves fine details while correcting structural defects. We demonstrate robust performance across 56 diverse object categories from ShapeNet and ThreeDScans, achieving superior refinement quality compared to Laplacian smoothing and unoptimized baselines while maintaining sub-second inference times. MeshCone is particularly suited for applications where reference geometry is available, such as mesh-from-template workflows, scan-to-CAD alignment, and quality assurance in asset production pipelines.
♻ ☆ SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation NeurIPS 2025
Zhenjie Mao, Yuhuan Yang, Chaofan Ma, Dongsheng Jiang, Jiangchao Yao, Ya Zhang, Yanfeng Wang
Referring Image Segmentation (RIS) aims to segment the target object in an image given a natural language expression. While recent methods leverage pre-trained vision backbones and more training corpus to achieve impressive results, they predominantly focus on simple expressions--short, clear noun phrases like "red car" or "left girl". This simplification often reduces RIS to a key word/concept matching problem, limiting the model's ability to handle referential ambiguity in expressions. In this work, we identify two challenging real-world scenarios: object-distracting expressions, which involve multiple entities with contextual cues, and category-implicit expressions, where the object class is not explicitly stated. To address the challenges, we propose a novel framework, SaFiRe, which mimics the human two-phase cognitive process--first forming a global understanding, then refining it through detail-oriented inspection. This is naturally supported by Mamba's scan-then-update property, which aligns with our phased design and enables efficient multi-cycle refinement with linear complexity. We further introduce aRefCOCO, a new benchmark designed to evaluate RIS models under ambiguous referring expressions. Extensive experiments on both standard and proposed datasets demonstrate the superiority of SaFiRe over state-of-the-art baselines.
comment : NeurIPS 2025; Project page: https://zhenjiemao.github.io/SaFiRe/
♻ ☆ Comparison of Generative Learning Methods for Turbulence Surrogates
Claudia Drygala, Edmund Ross, Francesca di Mare, Hanno Gottschalk
Numerical simulations of turbulent flows present significant challenges in fluid dynamics due to their complexity and high computational cost. High resolution techniques such as Direct Numerical Simulation (DNS) and Large Eddy Simulation (LES) are generally not computationally affordable, particularly for technologically relevant problems. Recent advances in machine learning, specifically in generative probabilistic models, offer promising alternatives as surrogates for turbulence. This paper investigates the application of three generative models - Variational Autoencoders (VAE), Deep Convolutional Generative Adversarial Networks (DCGAN), and Denoising Diffusion Probabilistic Models (DDPM) - in simulating a von Kármán vortex street around a fixed cylinder projected into 2D, as well as a real-world experimental dataset of the wake flow of a cylinder array. Training data was obtained by means of LES in the simulated case and Particle Image Velocimetry (PIV) in the experimental case. We evaluate each model's ability to capture the statistical properties and spatial structures of the turbulent flow. Our results demonstrate that DDPM and DCGAN effectively replicate all flow distributions, highlighting their potential as efficient and accurate tools for turbulence surrogacy. We find a strong argument for DCGAN, as although they are more difficult to train (due to problems such as mode collapse), they show the fastest inference and training time, require less data to train compared to VAE and DDPM, and provide the results most closely aligned with the input stream. In contrast, VAE train quickly (and can generate samples quickly) but do not produce adequate results, and DDPM, whilst effective, are significantly slower at both, inference and training time.
♻ ☆ DensiCrafter: Physically-Constrained Generation and Fabrication of Self-Supporting Hollow Structures
Shengqi Dang, Fu Chai, Jiaxin Li, Chao Yuan, Wei Ye, Nan Cao
The rise of 3D generative models has enabled automatic 3D geometry and texture synthesis from multimodal inputs (e.g., text or images). However, these methods often ignore physical constraints and manufacturability considerations. In this work, we address the challenge of producing 3D designs that are both lightweight and self-supporting. We present DensiCrafter, a framework for generating lightweight, self-supporting 3D hollow structures by optimizing the density field. Starting from coarse voxel grids produced by Trellis, we interpret these as continuous density fields to optimize and introduce three differentiable, physically constrained, and simulation-free loss terms. Additionally, a mass regularization penalizes unnecessary material, while a restricted optimization domain preserves the outer surface. Our method seamlessly integrates with pretrained Trellis-based models (e.g., Trellis, DSO) without any architectural changes. In extensive evaluations, we achieve up to 43% reduction in material mass on the text-to-3D task. Compared to state-of-the-art baselines, our method could improve the stability and maintain high geometric fidelity. Real-world 3D-printing experiments confirm that our hollow designs can be reliably fabricated and could be self-supporting.
♻ ☆ CrossEarth-Gate: Fisher-Guided Adaptive Tuning Engine for Efficient Adaptation of Cross-Domain Remote Sensing Semantic Segmentation
Shilei Cao, Ziyang Gong, Hehai Lin, Yang Liu, Jiashun Cheng, Xiaoxing Hu, Haoyuan Liang, Guowen Li, Chengwei Qin, Hong Cheng, Xue Yang, Juepeng Zheng, Haohuan Fu
In Remote Sensing (RS), Parameter-Efficient Fine-Tuning (PEFT) has emerged as a key approach to activate the generalizable representation ability of foundation models for downstream tasks. However, existing specialized PEFT methods often fail when applied to large-scale Earth observation tasks, as they are unable to fully handle the multifaceted and unpredictable domain gaps (\eg, spatial, semantic, and frequency shifts) inherent in RS data. To overcome this, we propose CrossEarth-Gate, which introduces two primary contributions. First, we establish a comprehensive RS module toolbox to address multifaceted domain gaps, comprising spatial, semantic, and frequency modules. Second, we develop a Fisher-guided adaptive selection mechanism that operates on this toolbox. This selection is guided by Fisher Information to quantify each module's importance by measuring its contribution to the task-specific gradient flow. It dynamically activates only the most critical modules at the appropriate layers, guiding the gradient flow to maximize adaptation effectiveness and efficiency. Comprehensive experiments validate the efficacy and generalizability of our method, where CrossEarth-Gate achieves state-of-the-art performance across 16 cross-domain benchmarks for RS semantic segmentation. The code of the work will be released.
♻ ☆ Probabilistic Robustness for Free? Revisiting Training via a Benchmark
Yi Zhang, Zheng Wang, Zhen Chen, Wenjie Ruan, Qing Guo, Siddartha Khastgir, Carsten Maple, Xingyu Zhao
Deep learning models are notoriously vulnerable to imperceptible perturbations. Most existing research centers on adversarial robustness (AR), which evaluates models under worst-case scenarios by examining the existence of deterministic adversarial examples (AEs). In contrast, probabilistic robustness (PR) adopts a statistical perspective, measuring the probability that predictions remain correct under stochastic perturbations. While PR is widely regarded as a practical complement to AR, dedicated training methods for improving PR are still relatively underexplored, albeit with emerging progress. Among the few PR-targeted training methods, we identify three limitations: i non-comparable evaluation protocols; ii limited comparisons to strong AT baselines despite anecdotal PR gains from AT; and iii no unified framework to compare the generalization of these methods. Thus, we introduce PRBench, the first benchmark dedicated to evaluating improvements in PR achieved by different robustness training methods. PRBench empirically compares most common AT and PR-targeted training methods using a comprehensive set of metrics, including clean accuracy, PR and AR performance, training efficiency, and generalization error (GE). We also provide theoretical analysis on the GE of PR performance across different training methods. Main findings revealed by PRBench include: AT methods are more versatile than PR-targeted training methods in terms of improving both AR and PR performance across diverse hyperparameter settings, while PR-targeted training methods consistently yield lower GE and higher clean accuracy. A leaderboard comprising 222 trained models across 7 datasets and 10 model architectures is publicly available at https://tmpspace.github.io/PRBenchLeaderboard/.
♻ ☆ Earth-Adapter: Bridge the Geospatial Domain Gaps with Mixture of Frequency Adaptation AAAI 2026
Xiaoxing Hu, Ziyang Gong, Yupei Wang, Yuru Jia, Fei Lin, Dexiang Gao, Ke An, Jianhong Han, Zhuoran Sun, Gen Luo, Gen Luo, Xue Yang
Parameter-Efficient Fine-Tuning (PEFT) is a technique that allows us to adapt powerful Foundation Models (FMs) to diverse downstream tasks while preserving and unleashing their inherent capabilities. However, we have observed that existing PEFT methods, which are often designed with natural imagery in mind, struggle when applied to Remote Sensing (RS) scenarios. This is primarily due to their inability to handle artifact influences, a problem particularly severe in RS image features. To tackle this challenge, we introduce Earth-Adapter, the first PEFT method specifically designed for RS artifacts conquering. Earth-Adapter introduces a novel Mixture of Frequency Adaptation process that combines a Mixture of Adapter (MoA) with Discrete Fourier Transformation (DFT). By utilizing DFT, Earth-Adapter can decompose features into different frequency components, precisely separating artifacts from original features. The MoA then dynamically assigns weights to each adapter expert, allowing for the combination of features across various frequency domains. These simple-yet-effective approaches enable Earth-Adapter to more efficiently overcome the disturbances caused by artifacts than previous PEFT methods, significantly enhancing the FMs' performance on RS scenarios. Experiments on Domain Adaptation (DA), and Domain Generalization (DG) semantic segmentation benchmarks showcase the Earth-Adapter's effectiveness. Compared with baseline Rein, Earth-Adapter significantly improves 9.0% mIoU in DA and 3.1% mIoU in DG benchmarks. Our code will be released at https://github.com/VisionXLab/Earth-Adapter.
comment : AAAI 2026 camera ready
♻ ☆ Learning Normals of Noisy Points by Local Gradient-Aware Surface Filtering ICCV 2025
Qing Li, Huifang Feng, Xun Gong, Yu-Shen Liu
Estimating normals for noisy point clouds is a persistent challenge in 3D geometry processing, particularly for end-to-end oriented normal estimation. Existing methods generally address relatively clean data and rely on supervised priors to fit local surfaces within specific neighborhoods. In this paper, we propose a novel approach for learning normals from noisy point clouds through local gradient-aware surface filtering. Our method projects noisy points onto the underlying surface by utilizing normals and distances derived from an implicit function constrained by local gradients. We start by introducing a distance measurement operator for global surface fitting on noisy data, which integrates projected distances along normals. Following this, we develop an implicit field-based filtering approach for surface point construction, adding projection constraints on these points during filtering. To address issues of over-smoothing and gradient degradation, we further incorporate local gradient consistency constraints, as well as local gradient orientation and aggregation. Comprehensive experiments on normal estimation, surface reconstruction, and point cloud denoising demonstrate the state-of-the-art performance of our method. The source code and trained models are available at https://github.com/LeoQLi/LGSF.
comment : Accepted by ICCV 2025. Project page: https://leoqli.github.io/LGSF/
♻ ☆ Adaptive Object Detection for Indoor Navigation Assistance: A Performance Evaluation of Real-Time Algorithms
Abhinav Pratap, Sushant Kumar, Suchinton Chakravarty
This study addresses the need for accurate and efficient object detection in assistive technologies for visually impaired individuals. We evaluate four real-time object detection algorithms YOLO, SSD, Faster R-CNN, and Mask R-CNN within the context of indoor navigation assistance. Using the Indoor Objects Detection dataset, we analyze detection accuracy, processing speed, and adaptability to indoor environments. Our findings highlight the trade-offs between precision and efficiency, offering insights into selecting optimal algorithms for realtime assistive navigation. This research advances adaptive machine learning applications, enhancing indoor navigation solutions for the visually impaired and promoting accessibility.
comment : 5 pages, 2 figures, 3 tables
♻ ☆ SKEL-CF: Coarse-to-Fine Biomechanical Skeleton and Surface Mesh Recovery
Da Li, Jiping Jin, Xuanlong Yu, Wei Liu, Xiaodong Cun, Kai Chen, Rui Fan, Jiangang Kong, Xi Shen
Parametric 3D human models such as SMPL have driven significant advances in human pose and shape estimation, yet their simplified kinematics limit biomechanical realism. The recently proposed SKEL model addresses this limitation by re-rigging SMPL with an anatomically accurate skeleton. However, estimating SKEL parameters directly remains challenging due to limited training data, perspective ambiguities, and the inherent complexity of human articulation. We introduce SKEL-CF, a coarse-to-fine framework for SKEL parameter estimation. SKEL-CF employs a transformer-based encoder-decoder architecture, where the encoder predicts coarse camera and SKEL parameters, and the decoder progressively refines them in successive layers. To ensure anatomically consistent supervision, we convert the existing SMPL-based dataset 4DHuman into a SKEL-aligned version, 4DHuman-SKEL, providing high-quality training data for SKEL estimation. In addition, to mitigate depth and scale ambiguities, we explicitly incorporate camera modeling into the SKEL-CF pipeline and demonstrate its importance across diverse viewpoints. Extensive experiments validate the effectiveness of the proposed design. On the challenging MOYO dataset, SKEL-CF achieves 85.0 MPJPE / 51.4 PA-MPJPE, significantly outperforming the previous SKEL-based state-of-the-art HSMR (104.5 / 79.6). These results establish SKEL-CF as a scalable and anatomically faithful framework for human motion analysis, bridging the gap between computer vision and biomechanics. Our implementation is available on the project page: https://pokerman8.github.io/SKEL-CF/.
comment : Project page: https://pokerman8.github.io/SKEL-CF/
♻ ☆ Dynamic Epsilon Scheduling: A Multi-Factor Adaptive Perturbation Budget for Adversarial Training
Alan Mitkiy, James Smith, Myungseo wong, Hana Satou, Hiroshi Tanaka, Emily Johnson
Adversarial training is among the most effective strategies for defending deep neural networks against adversarial examples. A key limitation of existing adversarial training approaches lies in their reliance on a fixed perturbation budget, which fails to account for instance-specific robustness characteristics. While prior works such as IAAT and MMA introduce instance-level adaptations, they often rely on heuristic or static approximations of data robustness. In this paper, we propose Dynamic Epsilon Scheduling (DES), a novel framework that adaptively adjusts the adversarial perturbation budget per instance and per training iteration. DES integrates three key factors: (1) the distance to the decision boundary approximated via gradient-based proxies, (2) prediction confidence derived from softmax entropy, and (3) model uncertainty estimated via Monte Carlo dropout. By combining these cues into a unified scheduling strategy, DES tailors the perturbation budget dynamically to guide more effective adversarial learning. Experimental results on CIFAR-10 and CIFAR-100 show that our method consistently improves both adversarial robustness and standard accuracy compared to fixed-epsilon baselines and prior adaptive methods. Moreover, we provide theoretical insights into the stability and convergence of our scheduling policy. This work opens a new avenue for instance-aware, data-driven adversarial training methods.
♻ ☆ Decor relation Speeds Up Vision Transformer s
Kieran Carrigg, Rob van Gastel, Melda Yeghaian, Sander Dalm, Faysal Boughorbel, Marcel van Gerven
Masked Autoencoder (MAE) pre-training of vision transformers (ViTs) yields strong performance in low-label data regimes but comes with substantial computational costs, making it impractical in time- and resource-constrained industrial settings. We address this by nitegrating Decorrelated Backpropagation (DBP) into MAE pre-training, an optimization method that iteratively reduces input correlations at each layer to accelerate convergence. Applied selectively to the encoder, DBP achieves faster pre-training without loss of stability. To mimic constrained-data scenarios, we evaluate our approach on ImageNet-1K pre-training and ADE20K fine-tuning using randomly sampled subsets of each dataset. Under this setting, DBP-MAE reduces wall-clock time to baseline performance by 21.1%, lowers carbon emissions by 21.4%, and improves segmentation mIoU by 1.1 points. We observe similar gains when pre-training and fine-tuning on proprietary industrial data, confirming the method's applicability in real-world scenarios. These results demonstrate that DBP can reduce training time and energy use while improving downstream performance for large-scale ViT pre-training. Keywords: Deep learning, Vision transformers, Efficient AI, Decorrelation
comment : 16 pages, 12 figures, submitted to CVC 2026
♻ ☆ Geometrically Regularized Transfer Learning with On-Manifold and Off-Manifold Perturbation
Hana Satou, Alan Mitkiy, Emma Collins, Finn Kingston
Transfer learning under domain shift remains a fundamental challenge due to the divergence between source and target data manifolds. In this paper, we propose MAADA (Manifold-Aware Adversarial Data Augmentation), a novel framework that decomposes adversarial perturbations into on-manifold and off-manifold components to simultaneously capture semantic variation and model brittleness. We theoretically demonstrate that enforcing on-manifold consistency reduces hypothesis complexity and improves generalization, while off-manifold regularization smooths decision boundaries in low-density regions. Moreover, we introduce a geometry-aware alignment loss that minimizes geodesic discrepancy between source and target manifolds. Experiments on DomainNet, VisDA, and Office-Home show that MAADA consistently outperforms existing adversarial and adaptation methods in both unsupervised and few-shot settings, demonstrating superior structural robustness and cross-domain generalization.
♻ ☆ Disentangled Geometric Alignment with Adaptive Contrastive Perturbation for Reliable Domain Transfer
Emma Collins, Myungseo wong, Kim Yun, Finn Kingston, Hana Satou
Despite progress in geometry-aware domain adaptation, current methods such as GAMA still suffer from two unresolved issues: (1) insufficient disentanglement of task-relevant and task-irrelevant manifold dimensions, and (2) rigid perturbation schemes that ignore per-class alignment asymmetries. To address this, we propose GAMA++, a novel framework that introduces (i) latent space disentanglement to isolate label-consistent manifold directions from nuisance factors, and (ii) an adaptive contrastive perturbation strategy that tailors both on- and off-manifold exploration to class-specific manifold curvature and alignment discrepancy. We further propose a cross-domain contrastive consistency loss that encourages local semantic clusters to align while preserving intra-domain diversity. Our method achieves state-of-the-art results on DomainNet, Office-Home, and VisDA benchmarks under both standard and few-shot settings, with notable improvements in class-level alignment fidelity and boundary robustness. GAMA++ sets a new standard for semantic geometry alignment in transfer learning.
♻ ☆ One-Step Diffusion -Based Image Compression with Semantic Distillation NeurIPS 2025
Naifu Xue, Zhaoyang Jia, Jiahao Li, Bin Li, Yuan Zhang, Yan Lu
While recent diffusion-based generative image codecs have shown impressive performance, their iterative sampling process introduces unpleasing latency. In this work, we revisit the design of a diffusion-based codec and argue that multi-step sampling is not necessary for generative compression. Based on this insight, we propose OneDC, a One-step Diffusion-based generative image Codec -- that integrates a latent compression module with a one-step diffusion generator. Recognizing the critical role of semantic guidance in one-step diffusion, we propose using the hyperprior as a semantic signal, overcoming the limitations of text prompts in representing complex visual content. To further enhance the semantic capability of the hyperprior, we introduce a semantic distillation mechanism that transfers knowledge from a pretrained generative tokenizer to the hyperprior codec. Additionally, we adopt a hybrid pixel- and latent-domain optimization to jointly enhance both reconstruction fidelity and perceptual realism. Extensive experiments demonstrate that OneDC achieves SOTA perceptual quality even with one-step generation, offering over 39% bitrate reduction and 20x faster decoding compared to prior multi-step diffusion-based codecs. Project: https://onedc-codec.github.io/
comment : Accepted by NeurIPS 2025
♻ ☆ VA-GS: Enhancing the Geometric Representation of Gaussian Splatting via View Alignment NeurIPS 2025
Qing Li, Huifang Feng, Xun Gong, Yu-Shen Liu
3D Gaussian Splatting has recently emerged as an efficient solution for high-quality and real-time novel view synthesis. However, its capability for accurate surface reconstruction remains underexplored. Due to the discrete and unstructured nature of Gaussians, supervision based solely on image rendering loss often leads to inaccurate geometry and inconsistent multi-view alignment. In this work, we propose a novel method that enhances the geometric representation of 3D Gaussians through view alignment (VA). Specifically, we incorporate edge-aware image cues into the rendering loss to improve surface boundary delineation. To enforce geometric consistency across views, we introduce a visibility-aware photometric alignment loss that models occlusions and encourages accurate spatial relationships among Gaussians. To further mitigate ambiguities caused by lighting variations, we incorporate normal-based constraints to refine the spatial orientation of Gaussians and improve local surface estimation. Additionally, we leverage deep image feature embeddings to enforce cross-view consistency, enhancing the robustness of the learned geometry under varying viewpoints and illumination. Extensive experiments on standard benchmarks demonstrate that our method achieves state-of-the-art performance in both surface reconstruction and novel view synthesis. The source code is available at https://github.com/LeoQLi/VA-GS.
comment : Accepted by NeurIPS 2025
♻ ☆ Stream and Query-guided Feature Aggregation for Efficient and Effective 3D Occupancy Prediction
Seokha Moon, Janghyun Baek, Giseop Kim, Jinkyu Kim, Sunwook Choi
3D occupancy prediction has become a key perception task in autonomous driving, as it enables comprehensive scene understanding. Recent methods enhance this understanding by incorporating spatiotemporal information through multi-frame fusion, but they suffer from a trade-off: dense voxel-based representations provide high accuracy at significant computational cost, whereas sparse representations improve efficiency but lose spatial detail. To mitigate this trade-off, we introduce DuOcc, which employs a dual aggregation strategy that retains dense voxel representations to preserve spatial fidelity while maintaining high efficiency. DuOcc consists of two key components: (i) Stream-based Voxel Aggregation, which recurrently accumulates voxel features over time and refines them to suppress warping-induced distortions, preserving a clear separation between occupied and free space. (ii) Query-guided Aggregation, which complements the limitations of voxel accumulation by selectively injecting instance-level query features into the voxel regions occupied by dynamic objects. Experiments on the widely used Occ3D-nuScenes and SurroundOcc datasets demonstrate that DuOcc achieves state-of-the-art performance in real-time settings, while reducing memory usage by over 40% compared to prior methods.
♻ ☆ Image as an IMU: Estimating Camera Motion from a Single Motion-Blurred Image
Jerred Chen, Ronald Clark
In many robotics and VR/AR applications, fast camera motions lead to a high level of motion blur, causing existing camera pose estimation methods to fail. In this work, we propose a novel framework that leverages motion blur as a rich cue for motion estimation rather than treating it as an unwanted artifact. Our approach works by predicting a dense motion flow field and a monocular depth map directly from a single motion-blurred image. We then recover the instantaneous camera velocity by solving a linear least squares problem under the small motion assumption. In essence, our method produces an IMU-like measurement that robustly captures fast and aggressive camera movements. To train our model, we construct a large-scale dataset with realistic synthetic motion blur derived from ScanNet++v2 and further refine our model by training end-to-end on real data using our fully differentiable pipeline. Extensive evaluations on real-world benchmarks demonstrate that our method achieves state-of-the-art angular and translational velocity estimates, outperforming current methods like MASt3R and COLMAP.
comment : Project page: https://jerredchen.github.io/image-as-imu/
♻ ☆ Unsupervised Segmentation by Diffusing, Walking and Cutting WACV
Daniela Ivanova, Marco Aversa, Paul Henderson, John Williamson
We propose an unsupervised image segmentation method using features from pre-trained text-to-image diffusion models. Inspired by classic spectral clustering approaches, we construct adjacency matrices from self-attention layers between image patches and recursively partition using Normalised Cuts. A key insight is that self-attention probability distributions, which capture semantic relations between patches, can be interpreted as a transition matrix for random walks across the image. We leverage this by first using Random Walk Normalized Cuts directly on these self-attention activations to partition the image, minimizing transition probabilities between clusters while maximizing coherence within clusters. Applied recursively, this yields a hierarchical segmentation that reflects the rich semantics in the pre-trained attention layers, without any additional training. Next, we explore other ways to build the NCuts adjacency matrix from features, and how we can use the random walk interpretation of self-attention to capture long-range relationships. Finally, we propose an approach to automatically determine the NCut cost criterion, avoiding the need to tune this manually. We quantitatively analyse the effect incorporating different features, a constant versus dynamic NCut threshold, and incorporating multi-node paths when constructing the NCuts adjacency matrix. We show that our approach surpasses all existing methods for zero-shot unsupervised segmentation, achieving state-of-the-art results on COCO-Stuff-27 and Cityscapes.
comment : Accepted to The IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2026
♻ ☆ ISAC: Training-Free Instance-to-Semantic Attention Control for Improving Multi-Instance Generation
Sanghyun Jo, Wooyeol Lee, Ziseok Lee, Kyungsu Kim
Text-to-image diffusion models have recently become highly capable, yet their behavior in multi-object scenes remains unreliable: models often produce an incorrect number of instances and exhibit semantics leaking across objects. We trace these failures to vague instance boundaries; self-attention already reveals instance layouts early in the denoising process, but existing approaches act only on semantic signals. We introduce $\textbf{ISAC}$ ($\textbf{I}$nstance-to-$\textbf{S}$emantic $\textbf{A}$ttention $\textbf{C}$ontrol), a training-free, model-agnostic objective that performs hierarchical attention control by first carving out instance layouts from self-attention and then binding semantics to these instances. In Phase 1, ISAC clusters self-attention into the number of instances and repels overlaps, establishing an instance-level structural hierarchy; in Phase 2, it injects these instance cues into cross-attention to obtain instance-aware semantic masks and decomposes mixing semantics by tying attributes within each instance. ISAC yields consistent gains on T2I-CompBench, HRS-Bench, and IntraCompBench, our new benchmark for intra-class compositions where failures are most frequent, with improvements of at least 50% in multi-class accuracy and 7% in multi-instance accuracy on IntraCompBench, without any fine-tuning or external models. Beyond text-to-image setups, ISAC also strengthens layout-to-image controllers under overlapping boxes by refining coarse box layouts into dense instance masks, indicating that hierarchical decoupling of instance formation and semantic assignment is a key principle for robust, controllable multi-object generation. Code will be released upon publication.
comment : 36 pages
♻ ☆ Think Visually, Reason Textually: Vision -Language Synergy in ARC
Beichen Zhang, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, Jiaqi Wang
Abstract reasoning from minimal examples remains a core unsolved problem for frontier foundation models such as GPT-5 and Grok 4. These models still fail to infer structured transformation rules from a handful of examples, which is a key hallmark of human intelligence. The Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) provides a rigorous testbed for this capability, demanding conceptual rule induction and transfer to novel tasks. Most existing methods treat ARC-AGI as a purely textual reasoning task, overlooking the fact that humans rely heavily on visual abstraction when solving such puzzles. However, our pilot experiments reveal a paradox: naively rendering ARC-AGI grids as images degrades performance due to imprecise rule execution. This leads to our central hypothesis that vision and language possess complementary strengths across distinct reasoning stages: vision supports global pattern abstraction and verification, whereas language specializes in symbolic rule formulation and precise execution. Building on this insight, we introduce two synergistic strategies: (1) Vision-Language Synergy Reasoning (VLSR), which decomposes ARC-AGI into modality-aligned subtasks; and (2) Modality-Switch Self-Correction (MSSC), which leverages vision to verify text-based reasoning for intrinsic error correction. Extensive experiments demonstrate that our approach yields up to a 4.33\% improvement over text-only baselines across diverse flagship models and multiple ARC-AGI tasks. Our findings suggest that unifying visual abstraction with linguistic reasoning is a crucial step toward achieving generalizable, human-like intelligence in future foundation models. Source code is released at https://github.com/InternLM/ARC-VL.
♻ ☆ Restoration-Oriented Video Frame Interpolation with Region-Distinguishable Priors from SAM
Yan Han, Xiaogang Xu, Yingqi Lin, Jiafei Wu, Zhe Liu, Ming-Hsuan Yang
In existing restoration-oriented Video Frame Interpolation (VFI) approaches, the motion estimation between neighboring frames plays a crucial role. However, the estimation accuracy in existing methods remains a challenge, primarily due to the inherent ambiguity in identifying corresponding areas in adjacent frames for interpolation. Therefore, enhancing accuracy by distinguishing different regions before motion estimation is of utmost importance. In this paper, we introduce a novel solution involving the utilization of open-world segmentation models, e.g., SAM2 (Segment Anything Model2) for frames, to derive Region-Distinguishable Priors (RDPs) in different frames. These RDPs are represented as spatial-varying Gaussian mixtures, distinguishing an arbitrary number of areas with a unified modality. RDPs can be integrated into existing motion-based VFI methods to enhance features for motion estimation, facilitated by our designed play-and-plug Hierarchical Region-aware Feature Fusion Module (HRFFM). HRFFM incorporates RDP into various hierarchical stages of VFI's encoder, using RDP-guided Feature Normalization (RDPFN) in a residual learning manner. With HRFFM and RDP, the features within VFI's encoder exhibit similar representations for matched regions in neighboring frames, thus improving the synthesis of intermediate frames. Extensive experiments demonstrate that HRFFM consistently enhances VFI performance across various scenes.
comment : Code will be released
♻ ☆ Towards Consistent and Controllable Image Synthesis for Face Editing
Mengting Wei, Tuomas Varanka, Yante Li, Xingxun Jiang, Huai-Qian Khor, Guoying Zhao
Face editing methods, essential for tasks like virtual avatars, digital human synthesis and identity preservation, have traditionally been built upon GAN-based techniques, while recent focus has shifted to diffusion-based models due to their success in image reconstruction. However, diffusion models still face challenges in controlling specific attributes and preserving the consistency of other unchanged attributes especially the identity characteristics. To address these issues and facilitate more convenient editing of face images, we propose a novel approach that leverages the power of Stable-Diffusion (SD) models and crude 3D face models to control the lighting, facial expression and head pose of a portrait photo. We observe that this task essentially involves the combinations of target background, identity and face attributes aimed to edit. We strive to sufficiently disentangle the control of these factors to enable consistency of face editing. Specifically, our method, coined as RigFace, contains: 1) A Spatial Attribute Encoder that provides presise and decoupled conditions of background, pose, expression and lighting; 2) A high-consistency FaceFusion method that transfers identity features from the Identity Encoder to the denoising UNet of a pre-trained SD model; 3) An Attribute Rigger that injects those conditions into the denoising UNet. Our model achieves comparable or even superior performance in both identity preservation and photorealism compared to existing face editing models.
♻ ☆ Not All Splits Are Equal: Rethinking Attribute Generalization Across Unrelated Categories NeurIPS 2025
Liviu Nicolae Fircă, Antonio Bărbălau, Dan Oneata, Elena Burceanu
Can models generalize attribute knowledge across semantically and perceptually dissimilar categories? While prior work has addressed attribute prediction within narrow taxonomic or visually similar domains, it remains unclear whether current models can abstract attributes and apply them to conceptually distant categories. This work presents the first explicit evaluation for the robustness of the attribute prediction task under such conditions, testing whether models can correctly infer shared attributes between unrelated object types: e.g., identifying that the attribute "has four legs" is common to both "dogs" and "chairs". To enable this evaluation, we introduce train-test split strategies that progressively reduce correlation between training and test sets, based on: LLM-driven semantic grouping, embedding similarity thresholding, embedding-based clustering, and supercategory-based partitioning using ground-truth labels. Results show a sharp drop in performance as the correlation between training and test categories decreases, indicating strong sensitivity to split design. Among the evaluated methods, clustering yields the most effective trade-off, reducing hidden correlations while preserving learnability. These findings offer new insights into the limitations of current representations and inform future benchmark construction for attribute reasoning.
comment : Accepted at NeurIPS 2025 Workshop: CauScien - Uncovering Causality in Science and NeurIPS 2025 Workshop: Reliable ML from Unreliable Data
♻ ☆ Without Paired Labeled Data: End-to-End Self-Supervised Learning for Drone-view Geo-Localization
Zhongwei Chen, Zhao-Xu Yang, Hai-Jun Rong, Guoqi Li
Drone-view Geo-Localization (DVGL) aims to achieve accurate localization of drones by retrieving the most relevant GPS-tagged satellite images. However, most existing methods heavily rely on strictly pre-paired drone-satellite images for supervised learning. When the target region shifts, new paired samples are typically required to adapt to the distribution changes. The high cost of annotation and the limited transferability of these methods significantly hinder the practical deployment of DVGL in open-world scenarios. To address these limitations, we propose a novel end-to-end self-supervised learning method with a shallow backbone network, called the dynamic memory-driven and neighborhood information learning (DMNIL) method. It employs a clustering algorithm to generate pseudo-labels and adopts a dual-path contrastive learning framework to learn discriminative intra-view representations. Furthermore, DMNIL incorporates two core modules, including the dynamic hierarchical memory learning (DHML) module and the information consistency evolution learning (ICEL) module. The DHML module combines short-term and long-term memory to enhance intra-view feature consistency and discriminability. Meanwhile, the ICEL module utilizes a neighborhood-driven dynamic constraint mechanism to systematically capture implicit cross-view semantic correlations, consequently improving cross-view feature alignment. To further stabilize and strengthen the self-supervised training process, a pseudo-label enhancement strategy is introduced to enhance the quality of pseudo supervision. Extensive experiments on three public benchmark datasets demonstrate that the proposed method consistently outperforms existing self-supervised methods and even surpasses several state-of-the-art supervised methods. Our code is available at https://github.com/ISChenawei/DMNIL.
♻ ☆ From Limited Labels to Open Domains:An Efficient Learning Method for Drone-view Geo-Localization
Zhongwei Chen, Zhao-Xu Yang, Hai-Jun Rong, Jiawei Lang, Guoqi Li
Traditional supervised drone-view geo-localization (DVGL) methods heavily depend on paired training data and encounter difficulties in learning cross-view correlations from unpaired data. Moreover, when deployed in a new domain, these methods require obtaining the new paired data and subsequent retraining for model adaptation, which significantly increases computational overhead. Existing unsupervised methods have enabled to generate pseudo-labels based on cross-view similarity to infer the pairing relationships. However, geographical similarity and spatial continuity often cause visually analogous features at different geographical locations. The feature confusion compromises the reliability of pseudo-label generation, where incorrect pseudo-labels drive negative optimization. Given these challenges inherent in both supervised and unsupervised DVGL methods, we propose a novel cross-domain invariant knowledge transfer network (CDIKTNet) with limited supervision, whose architecture consists of a cross-domain invariance sub-network (CDIS) and a cross-domain transfer sub-network (CDTS). This architecture facilitates a closed-loop framework for invariance feature learning and knowledge transfer. The CDIS is designed to learn cross-view structural and spatial invariance from a small amount of paired data that serves as prior knowledge. It endows the shared feature space of unpaired data with similar implicit cross-view correlations at initialization, which alleviates feature confusion. Based on this, the CDTS employs dual-path contrastive learning to further optimize each subspace while preserving consistency in a shared feature space. Extensive experiments demonstrate that CDIKTNet achieves state-of-the-art performance under full supervision compared with those supervised methods, and further surpasses existing unsupervised methods in both few-shot and cross-domain initialization.
♻ ☆ GreenHyperSpectra: A multi-source hyperspectral dataset for global vegetation trait prediction NeurIPS 2025
Eya Cherif, Arthur Ouaknine, Luke A. Brown, Phuong D. Dao, Kyle R. Kovach, Bing Lu, Daniel Mederer, Hannes Feilhauer, Teja Kattenborn, David Rolnick
Plant traits such as leaf carbon content and leaf mass are essential variables in the study of biodiversity and climate change. However, conventional field sampling cannot feasibly cover trait variation at ecologically meaningful spatial scales. Machine learning represents a valuable solution for plant trait prediction across ecosystems, leveraging hyperspectral data from remote sensing. Nevertheless, trait prediction from hyperspectral data is challenged by label scarcity and substantial domain shifts (\eg across sensors, ecological distributions), requiring robust cross-domain methods. Here, we present GreenHyperSpectra, a pretraining dataset encompassing real-world cross-sensor and cross-ecosystem samples designed to benchmark trait prediction with semi- and self-supervised methods. We adopt an evaluation framework encompassing in-distribution and out-of-distribution scenarios. We successfully leverage GreenHyperSpectra to pretrain label-efficient multi-output regression models that outperform the state-of-the-art supervised baseline. Our empirical analyses demonstrate substantial improvements in learning spectral representations for trait prediction, establishing a comprehensive methodological framework to catalyze research at the intersection of representation learning and plant functional traits assessment. All code and data are available at: https://github.com/echerif18/HyspectraSSL.
comment : Accepted at the 39th Conference on Neural Information Processing Systems (NeurIPS 2025)
♻ ☆ ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformer s for Interpretable Image Recognition
Mengqi Xue, Qihan Huang, Haofei Zhang, Jingwen Hu, Jie Song, Mingli Song, Canghong Jin
Prototypical part network (ProtoPNet) has drawn wide attention and boosted many follow-up studies due to its self-explanatory property for explainable artificial intelligence (XAI). However, when directly applying ProtoPNet on vision transformer (ViT) backbones, learned prototypes have a "distraction" problem: they have a relatively high probability of being activated by the background and pay less attention to the foreground. The powerful capability of modeling long-term dependency makes the transformer-based ProtoPNet hard to focus on prototypical parts, thus severely impairing its inherent interpretability. This paper proposes prototypical part transformer (ProtoPFormer) for appropriately and effectively applying the prototype-based method with ViTs for interpretable image recognition. The proposed method introduces global and local prototypes for capturing and highlighting the representative holistic and partial features of targets according to the architectural characteristics of ViTs. The global prototypes are adopted to provide the global view of objects to guide local prototypes to concentrate on the foreground while eliminating the influence of the background. Afterwards, local prototypes are explicitly supervised to concentrate on their respective prototypical visual parts, increasing the overall interpretability. Extensive experiments demonstrate that our proposed global and local prototypes can mutually correct each other and jointly make final decisions, which faithfully and transparently reason the decision-making processes associatively from the whole and local perspectives, respectively. Moreover, ProtoPFormer consistently achieves superior performance and visualization results over the state-of-the-art (SOTA) prototype-based baselines. Our code has been released at https://github.com/zju-vipa/ProtoPFormer.
comment : Arxiv preprint; 18 pages, 12 figures, 7 tables
♻ ☆ AMLP: Adjustable Masking Lesion Patches for Self-Supervised Medical Image Segmentation
Xiangtao Wang, Ruizhi Wang, Thomas Lukasiewicz, Zhenghua Xu
Self-supervised masked image modeling (MIM) methods have shown promising performances on analyzing natural images. However, directly applying such methods to medical image segmentation tasks still cannot achieve satisfactory results. The challenges arise from the facts that (i) medical images are inherently more complex compared to natural images, and the subjects in medical images often exhibit more distinct contour features; (ii) moreover, the conventional high and fixed masking ratio in MIM is likely to mask the background, limiting the scope of learnable information. To address these problems, we propose a new self-supervised medical image segmentation framework, called Adjustable Masking Lesion Patches (AMLP), which employs Masked Patch Selection~(MPS) strategy to identify patches with high probabilities of containing lesions to help model achieve precise lesion reconstruction. To improve the categorization of patches in MPS, we further introduce Relative Reconstruction Loss (RRL) to better learn hard-to-reconstruct lesion patches. Then, Category Consistency Loss (CCL) is proposed to refine patch categorization based on reconstruction difficulty, enhancing difference between lesions and backgrounds. Moreover, an Adjustable Masking Ratio (AMR) strategy is proposed to gradually increase the masking ratio over training to expand~the scope of learnable mutual information. Extensive~experiments on two medical segmentation datasets demonstrate the superior performances of the proposed AMLP w.r.t. the SOTA self-supervised methods; the results prove that AMLP effectively addresses the challenges of applying masked modeling to medical images and capturing accurate lesion details that are crucial for segmentation tasks.
comment : © 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works
♻ ☆ DWFF-Net : A Multi-Scale Farmland System Habitat Identification Method with Adaptive Dynamic Weight
Kesong Zheng, Zhi Song, Peizhou Li, Shuyi Yao, Zhenxing Bian
Addressing the current lack of a standardized habitat classification system for cultivated land ecosystems, incomplete coverage of the habitat types, and the inability of existing models to effectively integrate semantic and texture features-resulting in insufficient segmentation accuracy and blurred boundaries for multi-scale habitats (e.g., large-scale field plots and micro-habitats)-this study developed a comprehensively annotated ultra-high-resolution remote sensing image dataset encompassing 15 categories of cultivated land system habitats. Furthermore, we propose a Dynamic-Weighted Feature Fusion Network (DWFF-Net). The encoder of this model utilizes a frozen-parameter DINOv3 to extract foundational features. By analyzing the relationships between different category images and feature maps, we introduce a data-level adaptive dynamic weighting strategy for feature fusion. The decoder incorporates a dynamic weight computation network to achieve thorough integration of multi-layer features, and a hybrid loss function is adopted to optimize model training. Experimental results on the constructed dataset demonstrate that the proposed model achieves a mean Intersection over Union (mIoU) of 69.79% and an F1-score of 80.49%, outperforming the baseline network by 2.1% and 1.61%, respectively. Ablation studies further confirm the complementary nature of multi-layer feature fusion, which effectively improves the IoU for micro-habitat categories such as field ridges. This study establishes a habitat identification framework for cultivated land systems based on adaptive multi-layer feature fusion, enabling sub-meter precision habitat mapping at a low cost and providing robust technical support for fine-grained habitat monitoring in cultivated landscapes. (The complete code repository can be accessed via GitHub at the following URL: https://github.com/sysau/DWFF-Net)
comment : 30 pages,13 figures
♻ ☆ Filter Like You Test: Data-Driven Data Filtering for CLIP Pretrain ing
Mikey Shechter, Yair Carmon
We introduce Filter Like You Test (FLYT), an algorithm for curating large-scale vision-language datasets that learns the usefulness of each data point as a pretraining example. FLYT trains a scoring model that learns to weigh each example's features using gradient signals from downstream tasks training sets. Based on FLYT, we implement Mixing-FLYT (M-FLYT), which takes the per-example scores generated by different scoring methods as features, and learns to unify them into a single score. FLYT naturally produces a distribution over the training examples, which we leverage through Soft Cap Sampling (SCS), a strategy for obtaining a filtered pretraining dataset from per-example probabilities that samples examples while preventing over-representation through a repetition penalty. Using these methods, we achieve 40.1% ImageNet zero-shot accuracy on the DataComp medium scale filtering benchmark, a 2% absolute accuracy increase over all previous results and a 5.5% increase over results that - like us - use only public resources. Our approach also yields 37.7\% on the average of 38 DataComp evaluation tasks, outperforming previous public-resource approaches by 0.4\%.
♻ ☆ Gen-3 Diffusion : Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy
Yuxuan Xue, Xianghui Xie, Riccardo Marin, Gerard Pons-Moll
Creating realistic 3D objects and clothed avatars from a single RGB image is an attractive yet challenging problem. Due to its ill-posed nature, recent works leverage powerful prior from 2D diffusion models pretrained on large datasets. Although 2D diffusion models demonstrate strong generalization capability, they cannot guarantee the generated multi-view images are 3D consistent. In this paper, we propose Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy. We leverage a pre-trained 2D diffusion model and a 3D diffusion model via our elegantly designed process that synchronizes two diffusion models at both training and sampling time. The synergy between the 2D and 3D diffusion models brings two major advantages: 1) 2D helps 3D in generalization: the pretrained 2D model has strong generalization ability to unseen images, providing strong shape priors for the 3D diffusion model; 2) 3D helps 2D in multi-view consistency: the 3D diffusion model enhances the 3D consistency of 2D multi-view sampling process, resulting in more accurate multi-view generation. We validate our idea through extensive experiments in image-based objects and clothed avatar generation tasks. Results show that our method generates realistic 3D objects and avatars with high-fidelity geometry and texture. Extensive ablations also validate our design choices and demonstrate the strong generalization ability to diverse clothing and compositional shapes. Our code and pretrained models will be publicly released on https://yuxuan-xue.com/gen-3diffusion.
comment : Accepted to Transaction on Pattern Analysis and Machine Intelligence (T-PAMI). Project Page: https://yuxuan-xue.com/gen-3diffusion. arXiv admin note: substantial text overlap with arXiv:2406.08475
♻ ☆ ControlEvents: Controllable Synthesis of Event Camera Datawith Foundational Prior from Image Diffusion Models WACV2026
Yixuan Hu, Yuxuan Xue, Simon Klenk, Daniel Cremers, Gerard Pons-Moll
In recent years, event cameras have gained significant attention due to their bio-inspired properties, such as high temporal resolution and high dynamic range. However, obtaining large-scale labeled ground-truth data for event-based vision tasks remains challenging and costly. In this paper, we present ControlEvents, a diffusion-based generative model designed to synthesize high-quality event data guided by diverse control signals such as class text labels, 2D skeletons, and 3D body poses. Our key insight is to leverage the diffusion prior from foundation models, such as Stable Diffusion, enabling high-quality event data generation with minimal fine-tuning and limited labeled data. Our method streamlines the data generation process and significantly reduces the cost of producing labeled event datasets. We demonstrate the effectiveness of our approach by synthesizing event data for visual recognition, 2D skeleton estimation, and 3D body pose estimation. Our experiments show that the synthesized labeled event data enhances model performance in all tasks. Additionally, our approach can generate events based on unseen text labels during training, illustrating the powerful text-based generation capabilities inherited from foundation models.
comment : Accepted to WACV2026. Project website: https://https://yuxuan-xue.com/controlevents/
♻ ☆ SARVLM: A Vision Language Foundation Model for Semantic Understanding and Target Recognition in SAR Imagery
Qiwei Ma, Zhiyu Wang, Wang Liu, Xukun Lu, Bin Deng, Puhong Duan, Xudong Kang, Shutao Li
Synthetic Aperture Radar (SAR) is a crucial imaging modality thanks to its all-weather capability. Although recent advances in self-supervised learning and masked image modeling (MIM) have enabled SAR foundation models, these methods largely emphasize low-level visual features and often overlook multimodal alignment and zero-shot target recognition in SAR imagery. To address this, we construct SARVLM-1M, a large-scale vision-language dataset with over one million image-text pairs aggregated from existing datasets. We further propose a domain transfer training strategy to mitigate the large gap between natural and SAR imagery. Building on this, we develop SARVLM, the first vision language foundation model (VLM) tailored to SAR, comprising SARCLIP and SARCap. SARVLM is trained with a vision-language contrastive objective under the proposed domain transfer strategy, bridging SAR imagery and textual descriptions. Extensive experiments on image text retrieval, zero-shot classification, semantic localization, and imagery captioning demonstrate that SARVLM delivers superior feature extraction and interpretation, outperforming state-of-the-art VLMs and advancing SAR semantic understanding. Code and datasets will be released soon.
comment : 11 pages, 9 figures
♻ ☆ Active Negative Loss: A Robust Framework for Learning with Noisy Labels
Xichen Ye, Yifan Wu, Yiqi Wang, Xiaoqiang Li, Weizhong Zhang, Yifan Chen
Deep supervised learning has achieved remarkable success across a wide range of tasks, yet it remains susceptible to overfitting when confronted with noisy labels. To address this issue, noise-robust loss functions offer an effective solution for enhancing learning in the presence of label noise. In this work, we systematically investigate the limitation of the recently proposed Active Passive Loss (APL), which employs Mean Absolute Error (MAE) as its passive loss function. Despite the robustness brought by MAE, one of its key drawbacks is that it pays equal attention to clean and noisy samples; this feature slows down convergence and potentially makes training difficult, particularly in large-scale datasets. To overcome these challenges, we introduce a novel loss function class, termed Normalized Negative Loss Functions (NNLFs), which serve as passive loss functions within the APL framework. NNLFs effectively address the limitations of MAE by concentrating more on memorized clean samples. By replacing MAE in APL with our proposed NNLFs, we enhance APL and present a new framework called Active Negative Loss (ANL). Moreover, in non-symmetric noise scenarios, we propose an entropy-based regularization technique to mitigate the vulnerability to the label imbalance. Extensive experiments demonstrate that the new loss functions adopted by our ANL framework can achieve better or comparable performance to state-of-the-art methods across various label noise types and in image segmentation tasks. The source code is available at: https://github.com/Virusdoll/Active-Negative-Loss.
comment : This work has been submitted to the IEEE for possible publication
♻ ☆ LightMem: Lightweight and Efficient Memory-Augmented Generation
Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, Yuqi Tang, Ziwen Xu, Shumin Deng, Yunzhi Yao, Mengru Wang, Shuofei Qiao, Huajun Chen, Ningyu Zhang
Despite their remarkable capabilities, Large Language Models (LLMs) struggle to effectively leverage historical interaction information in dynamic and complex environments. Memory systems enable LLMs to move beyond stateless interactions by introducing persistent information storage, retrieval, and utilization mechanisms. However, existing memory systems often introduce substantial time and computational overhead. To this end, we introduce a new memory system called LightMem, which strikes a balance between the performance and efficiency of memory systems. Inspired by the Atkinson-Shiffrin model of human memory, LightMem organizes memory into three complementary stages. First, cognition-inspired sensory memory rapidly filters irrelevant information through lightweight compression and groups information according to their topics. Next, topic-aware short-term memory consolidates these topic-based groups, organizing and summarizing content for more structured access. Finally, long-term memory with sleep-time update employs an offline procedure that decouples consolidation from online inference. On LongMemEval and LoCoMo, using GPT and Qwen backbones, LightMem consistently surpasses strong baselines, improving QA accuracy by up to 7.7% / 29.3%, reducing total token usage by up to 38x / 20.9x and API calls by up to 30x / 55.5x, while purely online test-time costs are even lower, achieving up to 106x / 117x token reduction and 159x / 310x fewer API calls. The code is available at https://github.com/zjunlp/LightMem.
comment : Work in progress
♻ ☆ MetricHMSR:Metric Human Mesh and Scene Recovery from Monocular Images
Chentao Song, He Zhang, Haolei Yuan, Haozhe Lin, Jianhua Tao, Hongwen Zhang, Tao Yu
We introduce MetricHMSR (Metric Human Mesh and Scene Recovery), a novel approach for metric human mesh and scene recovery from monocular images. Due to unrealistic assumptions in the camera model and inherent challenges in metric perception, existing approaches struggle to achieve human pose and metric 3D position estimation through a unified module. To address this limitation, MetricHMSR incorporates camera rays to comprehensively encode both the bounding box information and the intrinsic parameters of perspective projection. Then we proposed Human Mixture-of-Experts (MoE), the model dynamically routes image features and ray features to task-specific experts for specialized understanding of different data aspects, enabling a unified framework that simultaneously perceives the local pose and the global 3D position. Based on the results above, we further refine the existing monocular metric depth estimation method to achieve more accurate results, ultimately enabling the seamless overlay of humans and scenes in 3D space. Comprehensive experimental results demonstrate that the proposed method achieves state-of-the-art performance on both human mesh and scene recovery.
♻ ☆ Restora-Flow: Mask-Guided Image Restoration with Flow Matching WACV 2026
Arnela Hadzic, Franz Thaler, Lea Bogensperger, Simon Johannes Joham, Martin Urschler
Flow matching has emerged as a promising generative approach that addresses the lengthy sampling times associated with state-of-the-art diffusion models and enables a more flexible trajectory design, while maintaining high-quality image generation. This capability makes it suitable as a generative prior for image restoration tasks. Although current methods leveraging flow models have shown promising results in restoration, some still suffer from long processing times or produce over-smoothed results. To address these challenges, we introduce Restora-Flow, a training-free method that guides flow matching sampling by a degradation mask and incorporates a trajectory correction mechanism to enforce consistency with degraded inputs. We evaluate our approach on both natural and medical datasets across several image restoration tasks involving a mask-based degradation, i.e., inpainting, super-resolution and denoising. We show superior perceptual quality and processing time compared to diffusion and flow matching-based reference methods.
comment : Accepted for WACV 2026
♻ ☆ Passive Dementia Screening via Facial Temporal Micro-Dynamics Analysis of In-the-Wild Talking-Head Video
Filippo Cenacchi, Longbing Cao, Mitchell McEwan, Deborah Richards
We target passive dementia screening from short camera-facing talking head video, developing a facial temporal micro dynamics analysis for language free detection of early neuro cognitive change. This enables unscripted, in the wild video analysis at scale to capture natural facial behaviors, transferrable across devices, topics, and cultures without active intervention by clinicians or researchers during recording. Most existing resources prioritize speech or scripted interviews, limiting use outside clinics and coupling predictions to language and transcription. In contrast, we identify and analyze whether temporal facial kinematics, including blink dynamics, small mouth jaw motions, gaze variability, and subtle head adjustments, are sufficient for dementia screening without speech or text. By stabilizing facial signals, we convert these micro movements into interpretable facial microdynamic time series, smooth them, and summarize short windows into compact clip level statistics for screening. Each window is encoded by its activity mix (the relative share of motion across streams), thus the predictor analyzes the distribution of motion across streams rather than its magnitude, making per channel effects transparent. We also introduce YT DemTalk, a new dataset curated from publicly available, in the wild camera facing videos. It contains 300 clips (150 with self reported dementia, 150 controls) to test our model and offer a first benchmarking of the corpus. On YT DemTalk, ablations identify gaze lability and mouth/jaw dynamics as the most informative cues, and light weighted shallow classifiers could attain a dementia prediction performance of (AUROC) 0.953, 0.961 Average Precision (AP), 0.851 F1-score, and 0.857 accuracy.
♻ ☆ UniChange: Unifying Change Detection with Multimodal Large Language Model
Xu Zhang, Danyang Li, Xiaohang Dong, Tianhao Wu, Hualong Yu, Jianye Wang, Qicheng Li, Xiang Li
Change detection (CD) is a fundamental task for monitoring and analyzing land cover dynamics. While recent high performance models and high quality datasets have significantly advanced the field, a critical limitation persists. Current models typically acquire limited knowledge from single-type annotated data and cannot concurrently leverage diverse binary change detection (BCD) and semantic change detection (SCD) datasets. This constraint leads to poor generalization and limited versatility. The recent advancements in Multimodal Large Language Models (MLLMs) introduce new possibilities for a unified CD framework. We leverage the language priors and unification capabilities of MLLMs to develop UniChange, the first MLLM-based unified change detection model. UniChange integrates generative language abilities with specialized CD functionalities. Our model successfully unifies both BCD and SCD tasks through the introduction of three special tokens: [T1], [T2], and [CHANGE]. Furthermore, UniChange utilizes text prompts to guide the identification of change categories, eliminating the reliance on predefined classification heads. This design allows UniChange to effectively acquire knowledge from multi-source datasets, even when their class definitions conflict. Experiments on four public benchmarks (WHU-CD, S2Looking, LEVIR-CD+, and SECOND) demonstrate SOTA performance, achieving IoU scores of 90.41, 53.04, 78.87, and 57.62, respectively, surpassing all previous methods. The code is available at https://github.com/Erxucomeon/UniChange.
♻ ☆ DiffSeg30k: A Multi-Turn Diffusion Editing Benchmark for Localized AIGC Detection
Hai Ci, Ziheng Peng, Pei Yang, Yingxin Xuan, Mike Zheng Shou
Diffusion-based editing enables realistic modification of local image regions, making AI-generated content harder to detect. Existing AIGC detection benchmarks focus on classifying entire images, overlooking the localization of diffusion-based edits. We introduce DiffSeg30k, a publicly available dataset of 30k diffusion-edited images with pixel-level annotations, designed to support fine-grained detection. DiffSeg30k features: 1) In-the-wild images--we collect images or image prompts from COCO to reflect real-world content diversity; 2) Diverse diffusion models--local edits using eight SOTA diffusion models; 3) Multi-turn editing--each image undergoes up to three sequential edits to mimic real-world sequential editing; and 4) Realistic editing scenarios--a vision-language model (VLM)-based pipeline automatically identifies meaningful regions and generates context-aware prompts covering additions, removals, and attribute changes. DiffSeg30k shifts AIGC detection from binary classification to semantic segmentation, enabling simultaneous localization of edits and identification of the editing models. We benchmark three baseline segmentation approaches, revealing significant challenges in semantic segmentation tasks, particularly concerning robustness to image distortions. Experiments also reveal that segmentation models, despite being trained for pixel-level localization, emerge as highly reliable whole-image classifiers of diffusion edits, outperforming established forgery classifiers while showing great potential in cross-generator generalization. We believe DiffSeg30k will advance research in fine-grained localization of AI-generated content by demonstrating the promise and limitations of segmentation-based methods. DiffSeg30k is released at: https://huggingface.co/datasets/Chaos2629/Diffseg30k
comment : 16 pages, 10 figures; typos corrected, references added
♻ ☆ Systematic Evaluation and Guidelines for Segment Anything Model in Surgical Video Analysis
Cheng Yuan, Jian Jiang, Kunyi Yang, Lv Wu, Rui Wang, Zi Meng, Haonan Ping, Ziyu Xu, Yifan Zhou, Wanli Song, Hesheng Wang, Yueming Jin, Qi Dou, Yutong Ban
Surgical video segmentation is critical for AI to interpret spatial-temporal dynamics in surgery, yet model performance is constrained by limited annotated data. The SAM2 model, pretrained on natural videos, offers potential for zero-shot surgical segmentation, but its applicability in complex surgical environments, with challenges like tissue deformation and instrument variability, remains unexplored. We present the first comprehensive evaluation of the zero-shot capability of SAM2 in 9 surgical datasets (17 surgery types), covering laparoscopic, endoscopic, and robotic procedures. We analyze various prompting (points, boxes, mask) and {finetuning (dense, sparse) strategies}, robustness to surgical challenges, and generalization across procedures and anatomies. Key findings reveal that while SAM2 demonstrates notable zero-shot adaptability in structured scenarios (e.g., instrument segmentation, {multi-organ segmentation}, and scene segmentation), its performance varies under dynamic surgical conditions, highlighting gaps in handling temporal coherence and domain-specific artifacts. These results highlight future pathways to adaptive data-efficient solutions for the surgical data science field.
♻ ☆ XYZCylinder: Towards Compatible Feed-Forward 3D Gaussian Splatting for Driving Scenes via Unified Cylinder Lifting Method
Haochen Yu, Qiankun Liu, Hongyuan Liu, Jianfei Jiang, Juntao Lyu, Jiansheng Chen, Huimin Ma
Feed-forward paradigms for 3D reconstruction have become a focus of recent research, which learn implicit, fixed view transformations to generate a single scene representation. However, their application to complex driving scenes reveals significant limitations. Two core challenges are responsible for this performance gap. First, the reliance on a fixed view transformation hinders compatibility to varying camera configurations. Second, the inherent difficulty of learning complex driving scenes from sparse 360° views with minimal overlap compromises the final reconstruction fidelity. To handle these difficulties, we introduce XYZCylinder, a novel method built upon a unified cylinder lifting method that integrates camera modeling and feature lifting. To tackle the compatibility problem, we design a Unified Cylinder Camera Modeling (UCCM) strategy. This strategy explicitly models projection parameters to unify diverse camera setups, thus bypassing the need for learning viewpoint-dependent correspondences. To improve the reconstruction accuracy, we propose a hybrid representation with several dedicated modules based on newly designed Cylinder Plane Feature Group (CPFG) to lift 2D image features to 3D space. Extensive evaluations confirm that XYZCylinder not only achieves state-of-the-art performance under different evaluation settings but also demonstrates remarkable compatibility in entirely new scenes with different camera settings in a zero-shot manner. Project page: \href{https://yuyuyu223.github.io/XYZCYlinder-projectpage/}{here}
comment : Feed-Forward, 3D Gaussian Splatting, Project page: https://yuyuyu223.github.io/XYZCYlinder-projectpage/
♻ ☆ Uncovering and Mitigating Destructive Multi-Embedding Attacks in Deepfake Proactive Forensics
Lixin Jia, Haiyang Sun, Zhiqing Guo, Yunfeng Diao, Dan Ma, Gaobo Yang
With the rapid evolution of deepfake technologies and the wide dissemination of digital media, personal privacy is facing increasingly serious security threats. Deepfake proactive forensics, which involves embedding imperceptible watermarks to enable reliable source tracking, serves as a crucial defense against these threats. Although existing methods show strong forensic ability, they rely on an idealized assumption of single watermark embedding, which proves impractical in real-world scenarios. In this paper, we formally define and demonstrate the existence of Multi-Embedding Attacks (MEA) for the first time. When a previously protected image undergoes additional rounds of watermark embedding, the original forensic watermark can be destroyed or removed, rendering the entire proactive forensic mechanism ineffective. To address this vulnerability, we propose a general training paradigm named Adversarial Interference Simulation (AIS). Rather than modifying the network architecture, AIS explicitly simulates MEA scenarios during fine-tuning and introduces a resilience-driven loss function to enforce the learning of sparse and stable watermark representations. Our method enables the model to maintain the ability to extract the original watermark correctly even after a second embedding. Extensive experiments demonstrate that our plug-and-play AIS training paradigm significantly enhances the robustness of various existing methods against MEA.
♻ ☆ Multi-scale Temporal Prediction via Incremental Generation and Multi- agent Collaboration
Zhitao Zeng, Guojian Yuan, Junyuan Mao, Yuxuan Wang, Xiaoshuang Jia, Yueming Jin
Accurate temporal prediction is the bridge between comprehensive scene understanding and embodied artificial intelligence. However, predicting multiple fine-grained states of a scene at multiple temporal scales is difficult for vision-language models. We formalize the Multi-Scale Temporal Prediction (MSTP) task in general and surgical scenes by decomposing multi-scale into two orthogonal dimensions: the temporal scale, forecasting states of humans and surgery at varying look-ahead intervals, and the state scale, modeling a hierarchy of states in general and surgical scenes. For example, in general scenes, states of contact relationships are finer-grained than states of spatial relationships. In surgical scenes, medium-level steps are finer-grained than high-level phases yet remain constrained by their encompassing phase. To support this unified task, we introduce the first MSTP Benchmark, featuring synchronized annotations across multiple state scales and temporal scales. We further propose a method, Incremental Generation and Multi-agent Collaboration (IG-MC), which integrates two key innovations. First, we present a plug-and-play incremental generation module that continuously synthesizes up-to-date visual previews at expanding temporal scales to inform multiple decision-making agents, keeping decisions and generated visuals synchronized and preventing performance degradation as look-ahead intervals lengthen. Second, we present a decision-driven multi-agent collaboration framework for multi-state prediction, comprising generation, initiation, and multi-state assessment agents that dynamically trigger and evaluate prediction cycles to balance global coherence and local fidelity.
comment : 20 pages, 6 figures
♻ ☆ Generalizable cardiac substructures segmentation from contrast and non-contrast CTs using pretrain ed transformer s
Aneesh Rangnekar, Nikhil Mankuzhy, Jonas Willmann, Chloe Choi, Abraham Wu, Maria Thor, Andreas Rimner, Harini Veeraraghavan
Automated AI segmentations for radiation treatment planning deteriorate when applied to cases with different characteristics than the training dataset. We developed a hybrid transformer convolutional network to segment cardiac substructures in lung and breast cancer patients with varying imaging contrasts and scan positions. Cohort I (56 contrast-enhanced CT [CECT], 124 non-contrast CT [NCCT] scans from lung cancer patients, supine position) was used to train an oracle model (180 cases), contrast-only model (56 CECTs), and balanced model (32 CECT, 32 NCCT). All models were evaluated on 60 held-out cohort I patients and 66 cohort II breast cancer patients (45 supine, 21 prone). Accuracy was measured using Dice similarity coefficient (DSC), 95th percentile Hausdorff distance (HD95), and dosimetric metrics, with TotalSegmentator as benchmark. Oracle and balanced models achieved similar accuracy (DSC: Oracle vs Balanced: Cohort I: 0.84 $\pm$ 0.10 vs 0.82 $\pm$ 0.10; Cohort II: 0.81 $\pm$ 0.12 vs 0.80 $\pm$ 0.13), both outperforming TotalSegmentator and the contrast-only models. The balanced model, using 64% fewer training cases, produced dosimetrically equivalent contours to manual delineations. It was robust to contrast variations (6 out of 8 substructures) and positioning variations (5 out of 8 substructures), with low correlation to patient age or body mass index. Our balanced model demonstrated robust geometric and dosimetric accuracy across varying imaging protocols and patient characteristics, which is essential for clinical deployment. Combining pretraining with balanced NCCT/CECT distribution enabled reliable segmentation with substantially fewer labeled cases than conventional approaches.
♻ ☆ CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness NeurIPS 2025
Zhihang Liu, Chen-Wei Xie, Bin Wen, Feiwu Yu, Jixuan Chen, Pandeng Li, Boqiang Zhang, Nianzu Yang, Yinglu Li, Zuan Gao, Yun Zheng, Hongtao Xie
Visual captioning benchmarks have become outdated with the emergence of modern multimodal large language models (MLLMs), as the brief ground-truth sentences and traditional metrics fail to assess detailed captions effectively. While recent benchmarks attempt to address this by focusing on keyword extraction or object-centric evaluation, they remain limited to vague-view or object-view analyses and incomplete visual element coverage. In this paper, we introduce CAPability, a comprehensive multi-view benchmark for evaluating visual captioning across 12 dimensions spanning six critical views. We curate nearly 11K human-annotated images and videos with visual element annotations to evaluate the generated captions. CAPability stably assesses both the correctness and thoroughness of captions with \textit{precision} and \textit{hit} metrics. By converting annotations to QA pairs, we further introduce a heuristic metric, \textit{know but cannot tell} ($K\bar{T}$), indicating a significant performance gap between QA and caption capabilities. Our work provides a holistic analysis of MLLMs' captioning abilities, as we identify their strengths and weaknesses across various dimensions, guiding future research to enhance specific aspects of their capabilities.
comment : Accepted to NeurIPS 2025
♻ ☆ Vision Remember: Recovering Visual Information in Efficient LVLM with Vision Feature Resampling
Ze Feng, Jiang-jiang Liu, Sen Yang, Lingyu Xiao, Zhibin Quan, Zhenhua Feng, Wankou Yang, Jingdong Wang
The computational expense of redundant vision tokens in Large Vision-Language Models (LVLMs) has led many existing methods to compress them via a vision projector. However, this compression may lose visual information that is crucial for tasks relying on fine-grained spatial relationships, such as OCR and Chart&Table Understanding. In this paper, we propose to resample original vision features across the LLM decoder layers to recover visual information and attain efficiency. Following this principle, we introduce Vision Remember, which includes two key modules: (1) Token-Feature Cross-Attention Layer and (2) Token Bidirectional Self-Attention Layer. In the Token bidirectional attention, we employ self-attention mechanism to maintain the bidirectional interaction between vision tokens and the text-guided token. In the Token-Feature interaction attention, we introduce local cross-attention to resample the visual feature and utilize the multi-level fusion to enrich the visual representation. We conduct comprehensive experiments on multiple visual understanding benchmarks and the results with the LLaVA-NeXT baseline show that Vision Remember outperforms TokenPacker by +2.7 and FastV by +5.7 across nearly all the settings. Compared with previous vision feature re-fusion methods, our approach also surpasses DeepStack by +3.9 and SVA Aggregator by +3.4 on the same baseline. The experimental results validate the generalization capability of the proposed method when combined with various efficient vision projectors and LVLMs.
♻ ☆ BRIC: Bridging Kinematic Plans and Physical Control at Test Time AAAI'26
Dohun Lim, Minji Kim, Jaewoon Lim, Sungchan Kim
We propose BRIC, a novel test-time adaptation (TTA) framework that enables long-term human motion generation by resolving execution discrepancies between diffusion-based kinematic motion planners and reinforcement learning-based physics controllers. While diffusion models can generate diverse and expressive motions conditioned on text and scene context, they often produce physically implausible outputs, leading to execution drift during simulation. To address this, BRIC dynamically adapts the physics controller to noisy motion plans at test time, while preserving pre-trained skills via a loss function that mitigates catastrophic forgetting. In addition, BRIC introduces a lightweight test-time guidance mechanism that steers the diffusion model in the signal space without updating its parameters. By combining both adaptation strategies, BRIC ensures consistent and physically plausible long-term executions across diverse environments in an effective and efficient manner. We validate the effectiveness of BRIC on a variety of long-term tasks, including motion composition, obstacle avoidance, and human-scene interaction, achieving state-of-the-art performance across all tasks.
comment : Accepted to AAAI'26
♻ ☆ FlowTok: Flowing Seamlessly Across Text and Image Tokens
Ju He, Qihang Yu, Qihao Liu, Liang-Chieh Chen
Bridging different modalities lies at the heart of cross-modality generation. While conventional approaches treat the text modality as a conditioning signal that gradually guides the denoising process from Gaussian noise to the target image modality, we explore a much simpler paradigm-directly evolving between text and image modalities through flow matching. This requires projecting both modalities into a shared latent space, which poses a significant challenge due to their inherently different representations: text is highly semantic and encoded as 1D tokens, whereas images are spatially redundant and represented as 2D latent embeddings. To address this, we introduce FlowTok, a minimal framework that seamlessly flows across text and images by encoding images into a compact 1D token representation. Compared to prior methods, this design reduces the latent space size by 3.3x at an image resolution of 256, eliminating the need for complex conditioning mechanisms or noise scheduling. Moreover, FlowTok naturally extends to image-to-text generation under the same formulation. With its streamlined architecture centered around compact 1D tokens, FlowTok is highly memory-efficient, requires significantly fewer training resources, and achieves much faster sampling speeds-all while delivering performance comparable to state-of-the-art models. Code is available at https://github.com/TACJu/FlowTok.
comment : Project page at https://tacju.github.io/projects/flowtok.html
♻ ☆ Towards Spatially Consistent Image Generation: On Incorporating Intrinsic Scene Properties into Diffusion Models
Hyundo Lee, Suhyung Choi, Inwoo Hwang, Byoung-Tak Zhang
Image generation models trained on large datasets can synthesize high-quality images but often produce spatially inconsistent and distorted images due to limited information about the underlying structures and spatial layouts. In this work, we leverage intrinsic scene properties (e.g., depth, segmentation maps) that provide rich information about the underlying scene, unlike prior approaches that solely rely on image-text pairs or use intrinsics as conditional inputs. Our approach aims to co-generate both images and their corresponding intrinsics, enabling the model to implicitly capture the underlying scene structure and generate more spatially consistent and realistic images. Specifically, we first extract rich intrinsic scene properties from a large image dataset with pre-trained estimators, eliminating the need for additional scene information or explicit 3D representations. We then aggregate various intrinsic scene properties into a single latent variable using an autoencoder. Building upon pre-trained large-scale Latent Diffusion Models (LDMs), our method simultaneously denoises the image and intrinsic domains by carefully sharing mutual information so that the image and intrinsic reflect each other without degrading image quality. Experimental results demonstrate that our method corrects spatial inconsistencies and produces a more natural layout of scenes while maintaining the fidelity and textual alignment of the base model (e.g., Stable Diffusion).
♻ ☆ ConceptGuard: Proactive Safety in Text-and-Image-to-Video Generation through Multimodal Risk Detection
Ruize Ma, Minghong Cai, Yilei Jiang, Jiaming Han, Yi Feng, Yingshui Tan, Xiaoyong Zhu, Bo Zhang, Bo Zheng, Xiangyu Yue
Recent progress in video generative models has enabled the creation of high-quality videos from multimodal prompts that combine text and images. While these systems offer enhanced controllability, they also introduce new safety risks, as harmful content can emerge from individual modalities or their interaction. Existing safety methods are often text-only, require prior knowledge of the risk category, or operate as post-generation auditors, struggling to proactively mitigate such compositional, multimodal risks. To address this challenge, we present ConceptGuard, a unified safeguard framework for proactively detecting and mitigating unsafe semantics in multimodal video generation. ConceptGuard operates in two stages: First, a contrastive detection module identifies latent safety risks by projecting fused image-text inputs into a structured concept space; Second, a semantic suppression mechanism steers the generative process away from unsafe concepts by intervening in the prompt's multimodal conditioning. To support the development and rigorous evaluation of this framework, we introduce two novel benchmarks: ConceptRisk, a large-scale dataset for training on multimodal risks, and T2VSafetyBench-TI2V, the first benchmark adapted from T2VSafetyBench for the Text-and-Image-to-Video (TI2V) safety setting. Comprehensive experiments on both benchmarks show that ConceptGuard consistently outperforms existing baselines, achieving state-of-the-art results in both risk detection and safe video generation. Our code is available at https://github.com/Ruize-Ma/ConceptGuard.
♻ ☆ Saliency-R1: Incentivizing Unified Saliency Reasoning Capability in MLLM with Confidence-Guided Reinforcement Learning
Long Li, Shuichen Ji, Ziyang Luo, Zhihui Li, Dingwen Zhang, Junwei Han, Nian Liu
Although multimodal large language models (MLLMs) excel in high-level vision-language reasoning, they lack inherent awareness of visual saliency, making it difficult to identify key visual elements. To bridge this gap, we propose Saliency-R1, the first unified MLLM framework that jointly tackles three representative and heterogeneous saliency tasks: Salient Object Detection (SOD), Salient Instance Segmentation (SIS), and Co-salient Object Detection (CoSOD), enhancing the model's capacity for saliency reasoning. We introduce a textual interface with structured tags ( , ) to encode region- and instance-level referring expressions, enabling a single referring segmenter to produce task-appropriate masks. To train the MLLM efficiently, we propose Confidence-Guided Policy Optimization (CGPO), a novel single-sample reinforcement learning algorithm. CGPO improves on GRPO by replacing group-normalized advantages with a per-sample signal based on reward-confidence discrepancy, thereby reducing computational waste, mitigating signal dilution, and lowering training overhead. Our model exceeds or matches the performance of robust open/closed-source MLLMs and specialized state-of-the-art methods across all three tasks, demonstrating the efficacy of our framework in saliency reasoning.
comment : Main text (excluding references): 8 pages, 4 figures; Supplementary Materials (excluding references): 9 pages, 10 figures
♻ ☆ Cheating Stereo Matching in Full-scale: Physical Adversarial Attack against Binocular Depth Estimation in Autonomous Driving AAAI 2026
Kangqiao Zhao, Shuo Huai, Xurui Song, Jun Luo
Though deep neural models adopted to realize the perception of autonomous driving have proven vulnerable to adversarial examples, known attacks often leverage 2D patches and target mostly monocular perception. Therefore, the effectiveness of Physical Adversarial Examples (PAEs) on stereo-based binocular depth estimation remains largely unexplored. To this end, we propose the first texture-enabled physical adversarial attack against stereo matching models in the context of autonomous driving. Our method employs a 3D PAE with global camouflage texture rather than a local 2D patch-based one, ensuring both visual consistency and attack effectiveness across different viewpoints of stereo cameras. To cope with the disparity effect of these cameras, we also propose a new 3D stereo matching rendering module that allows the PAE to be aligned with real-world positions and headings in binocular vision. We further propose a novel merging attack that seamlessly blends the target into the environment through fine-grained PAE optimization. It has significantly enhanced stealth and lethality upon existing hiding attacks that fail to get seamlessly merged into the background. Extensive evaluations show that our PAEs can successfully fool the stereo models into producing erroneous depth information.
comment : AAAI 2026
♻ ☆ Point-Supervised Facial Expression Spotting with Gaussian-Based Instance-Adaptive Intensity Modeling
Yicheng Deng, Hideaki Hayashi, Hajime Nagahara
Automatic facial expression spotting, which aims to identify facial expression instances in untrimmed videos, is crucial for facial expression analysis. Existing methods primarily focus on fully-supervised learning and rely on costly, time-consuming temporal boundary annotations. In this paper, we investigate point-supervised facial expression spotting (P-FES), where only a single timestamp annotation per instance is required for training. We propose a unique two-branch framework for P-FES. First, to mitigate the limitation of hard pseudo-labeling, which often confuses neutral and expression frames with various intensities, we propose a Gaussian-based instance-adaptive intensity modeling (GIM) module to model instance-level expression intensity distribution for soft pseudo-labeling. By detecting the pseudo-apex frame around each point label, estimating the duration, and constructing an instance-level Gaussian distribution, GIM assigns soft pseudo-labels to expression frames for more reliable intensity supervision. The GIM module is incorporated into our framework to optimize the class-agnostic expression intensity branch. Second, we design a class-aware apex classification branch that distinguishes macro- and micro-expressions solely based on their pseudo-apex frames. During inference, the two branches work independently: the class-agnostic expression intensity branch generates expression proposals, while the class-aware apex-classification branch is responsible for macro- and micro-expression classification. Furthermore, we introduce an intensity-aware contrastive loss to enhance discriminative feature learning and suppress neutral noise by contrasting neutral frames with expression frames with various intensities. Extensive experiments on the SAMM-LV, CAS(ME)$^2$, and CAS(ME)$^3$ datasets demonstrate the effectiveness of our proposed framework.
♻ ☆ Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmark s
Jie Li, Hongyi Cai, Mingkang Dong, Muxin Pu, Shan You, Fei Wang, Tao Huang
Automatically detecting abnormal events in videos is crucial for modern autonomous systems, yet existing Video Anomaly Detection (VAD) benchmarks lack the scene diversity, balanced anomaly coverage, and temporal complexity needed to reliably assess real-world performance. Meanwhile, the community is increasingly moving toward Video Anomaly Understanding (VAU), which requires deeper semantic and causal reasoning but remains difficult to benchmark due to the heavy manual annotation effort it demands. In this paper, we introduce Pistachio, a new VAD/VAU benchmark constructed entirely through a controlled, generation-based pipeline. By leveraging recent advances in video generation models, Pistachio provides precise control over scenes, anomaly types, and temporal narratives, effectively eliminating the biases and limitations of Internet-collected datasets. Our pipeline integrates scene-conditioned anomaly assignment, multi-step storyline generation, and a temporally consistent long-form synthesis strategy that produces coherent 41-second videos with minimal human intervention. Extensive experiments demonstrate the scale, diversity, and complexity of Pistachio, revealing new challenges for existing methods and motivating future research on dynamic and multi-event anomaly understanding.
♻ ☆ Thinking in 360°: Humanoid Visual Search in the Wild
Heyang Yu, Yinan Han, Xiangyu Zhang, Baiqiao Yin, Bowen Chang, Xiangyu Han, Xinhao Liu, Jing Zhang, Marco Pavone, Chen Feng, Saining Xie, Yiming Li
Humans rely on the synergistic control of head (cephalomotor) and eye (oculomotor) to efficiently search for visual information in 360°. However, prior approaches to visual search are limited to a static image, neglecting the physical embodiment and its interaction with the 3D world. How can we develop embodied visual search agents as efficient as humans while bypassing the constraints imposed by real-world hardware? To this end, we propose humanoid visual search where a humanoid agent actively rotates its head to search for objects or paths in an immersive world represented by a 360° panoramic image. To study visual search in visually-crowded real-world scenarios, we build H* Bench, a new benchmark that moves beyond household scenes to challenging in-the-wild scenes that necessitate advanced visual-spatial reasoning capabilities, such as transportation hubs, large-scale retail spaces, urban streets, and public institutions. Our experiments first reveal that even top-tier proprietary models falter, achieving only ~30% success in object and path search. We then use post-training techniques to enhance the open-source Qwen2.5-VL, increasing its success rate by over threefold for both object search (14.83% to 47.38%) and path search (6.44% to 24.94%). Notably, the lower ceiling of path search reveals its inherent difficulty, which we attribute to the demand for sophisticated spatial commonsense. Our results not only show a promising path forward but also quantify the immense challenge that remains in building MLLM agents that can be seamlessly integrated into everyday human life.
comment : Website: https://humanoid-vstar.github.io/ ; Code: https://github.com/humanoid-vstar/hstar
♻ ☆ PointNSP: Autoregressive 3D Point Cloud Generation with Next-Scale Level-of-Detail Prediction
Ziqiao Meng, Qichao Wang, Zhiyang Dou, Zixing Song, Zhipeng Zhou, Irwin King, Peilin Zhao
Autoregressive point cloud generation has long lagged behind diffusion-based approaches in quality. The performance gap stems from the fact that autoregressive models impose an artificial ordering on inherently unordered point sets, forcing shape generation to proceed as a sequence of local predictions. This sequential bias emphasizes short-range continuity but undermines the model's capacity to capture long-range dependencies, hindering its ability to enforce global structural properties such as symmetry, consistent topology, and large-scale geometric regularities. Inspired by the level-of-detail (LOD) principle in shape modeling, we propose PointNSP, a coarse-to-fine generative framework that preserves global shape structure at low resolutions and progressively refines fine-grained geometry at higher scales through a next-scale prediction paradigm. This multi-scale factorization aligns the autoregressive objective with the permutation-invariant nature of point sets, enabling rich intra-scale interactions while avoiding brittle fixed orderings. Experiments on ShapeNet show that PointNSP establishes state-of-the-art (SOTA) generation quality for the first time within the autoregressive paradigm. In addition, it surpasses strong diffusion-based baselines in parameter, training, and inference efficiency. Finally, in dense generation with 8,192 points, PointNSP's advantages become even more pronounced, underscoring its scalability potential.
comment : This work was intended as a replacement of arXiv:2503.08594 and any subsequent updates will appear there
♻ ☆ VGGTFace: Topologically Consistent Facial Geometry Reconstruction in the Wild
Xin Ming, Yuxuan Han, Tianyu Huang, Feng Xu
Reconstructing topologically consistent facial geometry is crucial for the digital avatar creation pipelines. Existing methods either require tedious manual efforts, lack generalization to in-the-wild data, or are constrained by the limited expressiveness of 3D Morphable Models. To address these limitations, we propose VGGTFace, an automatic approach that innovatively applies the 3D foundation model, i.e. VGGT, for topologically consistent facial geometry reconstruction from in-the-wild multi-view images captured by everyday users. Our key insight is that, by leveraging VGGT, our method naturally inherits strong generalization ability and expressive power from its large-scale training and point map representation. However, it is unclear how to reconstruct a topologically consistent mesh from VGGT, as the topology information is missing in its prediction. To this end, we augment VGGT with Pixel3DMM for injecting topology information via pixel-aligned UV values. In this manner, we convert the pixel-aligned point map of VGGT to a point cloud with topology. Tailored to this point cloud with known topology, we propose a novel Topology-Aware Bundle Adjustment strategy to fuse them, where we construct a Laplacian energy for the Bundle Adjustment objective. Our method achieves high-quality reconstruction in 10 seconds for 16 views on a single NVIDIA RTX 4090. Experiments demonstrate state-of-the-art results on benchmarks and impressive generalization to in-the-wild data. Code is available at https://github.com/grignarder/vggtface.
♻ ☆ Agent 0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision -Language Reasoning
Jiaqi Liu, Kaiwen Xiong, Peng Xia, Yiyang Zhou, Haonian Ji, Lu Feng, Siwei Han, Mingyu Ding, Huaxiu Yao
Vision-language agents have achieved remarkable progress in a variety of multimodal reasoning tasks; however, their learning remains constrained by the limitations of human-annotated supervision. Recent self-rewarding approaches attempt to overcome this constraint by allowing models to act as their own critics or reward providers. Yet, purely text-based self-evaluation struggles to verify complex visual reasoning steps and often suffers from evaluation hallucinations. To address these challenges, inspired by recent advances in tool-integrated reasoning, we propose Agent0-VL, a self-evolving vision-language agent that achieves continual improvement with tool-integrated reasoning. Agent0-VL incorporates tool usage not only into reasoning but also into self-evaluation and self-repair, enabling the model to introspect, verify, and refine its reasoning through evidence-grounded analysis. It unifies two synergistic roles within a single LVLM: a Solver that performs multi-turn tool-integrated reasoning, and a Verifier that generates structured feedback and fine-grained self-rewards through tool-grounded critique. These roles interact through a Self-Evolving Reasoning Cycle, where tool-based verification and reinforcement learning jointly align the reasoning and evaluation distributions for stable self-improvement. Through this zero-external-reward evolution, Agent0-VL aligns its reasoning and verification behaviors without any human annotation or external reward models, achieving continual self-improvement. Experiments on geometric problem solving and visual scientific analysis show that Agent0-VL achieves an 12.5% improvement over the base model. Our code is available at https://github.com/aiming-lab/Agent0.
♻ ☆ UniGame: Turning a Unified Multimodal Model Into Its Own Adversary
Zhaolong Su, Wang Lu, Hao Chen, Sharon Li, Jindong Wang
Unified Multimodal Models (UMMs) have shown impressive performance in both understanding and generation with a single architecture. However, UMMs still exhibit a fundamental inconsistency: understanding favors compact embeddings, whereas generation favors reconstruction-rich representations. This structural trade-off produces misaligned decision boundaries, degraded cross-modal coherence, and heightened vulnerability under distributional and adversarial shifts. In this paper, we present UniGame, a self-adversarial post-training framework that directly targets the inconsistencies. By applying a lightweight perturber at the shared token interface, UniGame enables the generation branch to actively seek and challenge fragile understanding, turning the model itself into its own adversary. Experiments demonstrate that UniGame significantly improves the consistency (+4.6%). Moreover, it also achieves substantial improvements in understanding (+3.6%), generation (+0.02), out-of-distribution and adversarial robustness (+4.8% and +6.2% on NaturalBench and AdVQA). The framework is architecture-agnostic, introduces less than 1% additional parameters, and is complementary to existing post-training methods. These results position adversarial self-play as a general and effective principle for enhancing the coherence, stability, and unified competence of future multimodal foundation models. The official code is available at: https://github.com/AIFrontierLab/UniGame
♻ ☆ CroMe: Multimodal Fake News Detection using Cross-Modal Tri- Transformer and Metric Learning
Eunjee Choi, Junhyun Ahn, XinYu Piao, Jong-Kook Kim
Multimodal Fake News Detection has received increasing attention recently. Existing methods rely on independently encoded unimodal data and overlook the advantages of capturing intra-modality relationships and integrating inter-modal similarities using advanced techniques. To address these issues, Cross-Modal Tri-Transformer and Metric Learning for Multimodal Fake News Detection (CroMe) is proposed. CroMe utilizes Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (BLIP2) as encoders to capture detailed text, image and combined image-text representations. The metric learning module employs a proxy anchor method to capture intra-modality relationships while the feature fusion module uses a Cross-Modal and Tri-Transformer for effective integration. The final fake news detector processes the fused features through a classifier to predict the authenticity of the content. Experiments on datasets show that CroMe excels in multimodal fake news detection.
♻ ☆ GFT-GCN: Privacy-Preserving 3D Face Mesh Recognition with Spectral Diffusion WACV 2026
Hichem Felouat, Hanrui Wang, Isao Echizen
3D face recognition offers a robust biometric solution by capturing facial geometry, providing resilience to variations in illumination, pose changes, and presentation attacks. Its strong spoof resistance makes it suitable for high-security applications, but protecting stored biometric templates remains critical. We present GFT-GCN, a privacy-preserving 3D face recognition framework that combines spectral graph learning with diffusion-based template protection. Our approach integrates the Graph Fourier Transform (GFT) and Graph Convolutional Networks (GCN) to extract compact, discriminative spectral features from 3D face meshes. To secure these features, we introduce a spectral diffusion mechanism that produces irreversible, renewable, and unlinkable templates. A lightweight client-server architecture ensures that raw biometric data never leaves the client device. Experiments on the BU-3DFE and FaceScape datasets demonstrate high recognition accuracy and strong resistance to reconstruction attacks. Results show that GFT-GCN effectively balances privacy and performance, offering a practical solution for secure 3D face authentication.
comment : 13 pages, 8 figures, WACV 2026
♻ ☆ One-Step Diffusion Transformer for Controllable Real-Wo rl d Image Super-Resolution
Yushun Fang, Yuxiang Chen, Shibo Yin, Qiang Hu, Jiangchao Yao, Ya Zhang, Xiaoyun Zhang, Yanfeng Wang
Recent advances in diffusion-based real-world image super-resolution (Real-ISR) have demonstrated remarkable perceptual quality, yet the balance between fidelity and controllability remains a problem: multi-step diffusion-based methods suffer from generative diversity and randomness, resulting in low fidelity, while one-step methods lose control flexibility due to fidelity-specific finetuning. In this paper, we present ODTSR, a one-step diffusion transformer based on Qwen-Image that performs Real-ISR considering fidelity and controllability simultaneously: a newly introduced visual stream receives low-quality images (LQ) with adjustable noise (Control Noise), and the original visual stream receives LQs with consistent noise (Prior Noise), forming the Noise-hybrid Visual Stream (NVS) design. ODTSR further employs Fidelity-aware Adversarial Training (FAA) to enhance controllability and achieve one-step inference. Extensive experiments demonstrate that ODTSR not only achieves state-of-the-art (SOTA) performance on generic Real-ISR, but also enables prompt controllability on challenging scenarios such as real-world scene text image super-resolution (STISR) of Chinese characters without training on specific datasets. Codes are available at $\href{https://github.com/RedMediaTech/ODTSR}{\text{this url}}$.
♻ ☆ Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Rumination
Yolo Y. Tang, Daiki Shimada, Hang Hua, Chao Huang, Jing Bi, Rogerio Feris, Chenliang Xu
Understanding text-rich videos requires reading small, transient textual cues that often demand repeated inspection. Yet most video QA models rely on single-pass perception over fixed frames, leading to hallucinations and failures on fine-grained evidence. Inspired by how humans pause, zoom, and re-read critical regions, we introduce Video-R4 (Reinforcing Text-Rich Video Reasoning with Visual Rumination), a video reasoning LMM that performs visual rumination: iteratively selecting frames, zooming into informative regions, re-encoding retrieved pixels, and updating its reasoning state. We construct two datasets with executable rumination trajectories: Video-R4-CoT-17k for supervised practice and Video-R4-RL-30k for reinforcement learning. We propose a multi-stage rumination learning framework that progressively finetunes a 7B LMM to learn atomic and mixing visual operations via SFT and GRPO-based RL. Video-R4-7B achieves state-of-the-art results on M4-ViteVQA and further generalizes to multi-page document QA, slides QA, and generic video QA, demonstrating that iterative rumination is an effective paradigm for pixel-grounded multimodal reasoning. Project Page: https://yunlong10.github.io/Video-R4/
♻ ☆ EvoEmpirBench: Dynamic Spatial Reasoning with Agent -ExpVer AAAI 2026
Pukun Zhao, Longxiang Wang, Miaowei Wang, Chen Chen, Fanqing Zhou, Haojian Huang
Most existing spatial reasoning benchmarks focus on static or globally observable environments, failing to capture the challenges of long-horizon reasoning and memory utilization under partial observability and dynamic changes. We introduce two dynamic spatial benchmarks, locally observable maze navigation and match-2 elimination that systematically evaluate models' abilities in spatial understanding and adaptive planning when local perception, environment feedback, and global objectives are tightly coupled. Each action triggers structural changes in the environment, requiring continuous update of cognition and strategy. We further propose a subjective experience-based memory mechanism for cross-task experience transfer and validation. Experiments show that our benchmarks reveal key limitations of mainstream models in dynamic spatial reasoning and long-term memory, providing a comprehensive platform for future methodological advances. Our code and data are available at https://anonymous.4open.science/r/EvoEmpirBench-143C/.
comment : Accepted by AAAI 2026, 29 pages, 3 figures, 7 tables
♻ ☆ IVY-FAKE: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detection
Changjiang Jiang, Wenhui Dong, Zhonghao Zhang, Chenyang Si, Fengchang Yu, Wei Peng, Xinbin Yuan, Yifei Bi, Ming Zhao, Zian Zhou, Caifeng Shan
The rapid development of Artificial Intelligence Generated Content (AIGC) techniques has enabled the creation of high-quality synthetic content, but it also raises significant security concerns. Current detection methods face two major limitations: (1) the lack of multidimensional explainable datasets for generated images and videos. Existing open-source datasets (e.g., WildFake, GenVideo) rely on oversimplified binary annotations, which restrict the explainability and trustworthiness of trained detectors. (2) Prior MLLM-based forgery detectors (e.g., FakeVLM) exhibit insufficiently fine-grained interpretability in their step-by-step reasoning, which hinders reliable localization and explanation. To address these challenges, we introduce Ivy-Fake, the first large-scale multimodal benchmark for explainable AIGC detection. It consists of over 106K richly annotated training samples (images and videos) and 5,000 manually verified evaluation examples, sourced from multiple generative models and real world datasets through a carefully designed pipeline to ensure both diversity and quality. Furthermore, we propose Ivy-xDetector, a reinforcement learning model based on Group Relative Policy Optimization (GRPO), capable of producing explainable reasoning chains and achieving robust performance across multiple synthetic content detection benchmarks. Extensive experiments demonstrate the superiority of our dataset and confirm the effectiveness of our approach. Notably, our method improves performance on GenImage from 86.88% to 96.32%, surpassing prior state-of-the-art methods by a clear margin.
comment : 30 pages
♻ ☆ LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Training
Erh-Chung Chen, Che-Rung Lee
Adversarial training is a widely adopted strategy to bolster the robustness of neural network models against adversarial attacks. This paper revisits the fundamental assumptions underlying image classification and suggests that representing data as one-hot labels is a key factor that leads to vulnerabilities. However, in real-world datasets, data ambiguity often arises, with samples exhibiting characteristics of multiple classes, rendering one-hot label representations imprecise. To address this, we introduce a novel approach, Low-Temperature Distillation (LTD), designed to refine label representations. Unlike previous approaches, LTD incorporates a relatively low temperature in the teacher model, while maintaining a fixed temperature for the student model during both training and inference. This strategy not only refines assumptions about data distribution but also strengthens model robustness and avoids the gradient masking problem commonly encountered in defensive distillation. Experimental results demonstrate the efficacy of the proposed method when combined with existing frameworks, achieving robust accuracy rates of 58.19%, 31.13%, and 42.08% on the CIFAR-10, CIFAR-100, and ImageNet datasets, respectively, without the need for additional data.
♻ ☆ EmoFeedback$^2$: Reinforcement of Continuous Emotional Image Generation via LVLM-based Reward and Textual Feedback
Jingyang Jia, Kai Shu, Gang Yang, Long Xing, Xun Chen, Aiping Liu
Continuous emotional image generation (C-EICG) is emerging rapidly due to its ability to produce images aligned with both user descriptions and continuous emotional values. However, existing approaches lack emotional feedback from generated images, limiting the control of emotional continuity. Additionally, their simple alignment between emotions and naively generated texts fails to adaptively adjust emotional prompts according to image content, leading to insufficient emotional fidelity. To address these concerns, we propose a novel generation-understanding-feedback reinforcement paradigm (EmoFeedback$^2$) for C-EICG, which exploits the reasoning capability of the fine-tuned large vision-language model (LVLM) to provide reward and textual feedback for generating high-quality images with continuous emotions. Specifically, we introduce an emotion-aware reward feedback strategy, where the LVLM evaluates the emotional values of generated images and computes the reward against target emotions, guiding the reinforcement fine-tuning of the generative model and enhancing the emotional continuity of images. Furthermore, we design a self-promotion textual feedback framework, in which the LVLM iteratively analyzes the emotional content of generated images and adaptively produces refinement suggestions for the next-round prompt, improving the emotional fidelity with fine-grained content. Extensive experimental results demonstrate that our approach effectively generates high-quality images with the desired emotions, outperforming existing state-of-the-art methods in our custom dataset. The code and dataset will be released soon.
♻ ☆ Scale Where It Matters: Training-Free Localized Scaling for Diffusion Models
Qin Ren, Yufei Wang, Lanqing Guo, Wen Zhang, Zhiwen Fan, Chenyu You
Diffusion models have become the dominant paradigm in text-to-image generation, and test-time scaling (TTS) further improves quality by allocating more computation during inference. However, existing TTS methods operate at the full-image level, overlooking the fact that image quality is often spatially heterogeneous. This leads to unnecessary computation on already satisfactory regions and insufficient correction of localized defects. In this paper, we explore a new direction - Localized TTS - that adaptively resamples defective regions while preserving high-quality regions, thereby substantially reducing the search space. This paradigm poses two central challenges: accurately localizing defects and maintaining global consistency. We propose LoTTS, the first fully training-free framework for localized TTS. For defect localization, LoTTS contrasts cross- and self-attention signals under quality-aware prompts (e.g., high-quality vs. low-quality) to identify defective regions, and then refines them into coherent masks. For consistency, LoTTS perturbs only defective regions and denoises them locally, ensuring that corrections remain confined while the rest of the image remains undisturbed. Extensive experiments on SD2.1, SDXL, and FLUX demonstrate that LoTTS achieves state-of-the-art performance: it consistently improves both local quality and global fidelity, while reducing GPU cost by 2-4x compared to Best-of-N sampling. These findings establish localized TTS as a promising new direction for scaling diffusion models at inference time.
♻ ☆ ReMatch: Boosting Representation through Matching for Multimodal Retrieval
Qianying Liu, Xiao Liang, Zhiqiang Zhang, Zhongfei Qing, Fengfan Zhou, Yibo Chen, Xu Tang, Yao Hu, Paul Henderson
We present ReMatch, a framework that leverages the generative strength of MLLMs for multimodal retrieval. Previous approaches treated an MLLM as a simple encoder, ignoring its generative nature, and under-utilising its compositional reasoning and world knowledge. We instead train the embedding MLLM end-to-end with a chat-style generative matching stage. The matching stage uses the same MLLM to autoregressively decide relevance from multi-view inputs, including both raw data and its own projected embeddings for each query and document. It provides instance-wise discrimination supervision that complements a standard contrastive loss, offering stronger gradients on hard negatives and preserving the compositional strengths of the original MLLM. To obtain semantically richer multimodal embeddings, we use multiple learnable tokens to augment each input, generating fine-grained contextual, mutually orthogonal embeddings with low inference cost. Leveraging our established high-performance baseline,we assemble the ideas mentioned above into a powerful training recipe and achieve a new state-of-the-art on the Massive Multimodal Embedding Benchmark (MMEB). Our experiments show particularly strong zero-shot generalization results on five datasets, highlighting the robustness and transferability of ReMatch.
♻ ☆ Automated Neural Architecture Design for Industrial Defect Detection
Yuxi Liu, Yunfeng Ma, Yi Tang, Min Liu, Shuai Jiang, Yaonan Wang
Industrial surface defect detection (SDD) is critical for ensuring product quality and manufacturing reliability. Due to the diverse shapes and sizes of surface defects, SDD faces two main challenges: intraclass difference and interclass similarity. Existing methods primarily utilize manually designed models, which require extensive trial and error and often struggle to address both challenges effectively. To overcome this, we propose AutoNAD, an automated neural architecture design framework for SDD that jointly searches over convolutions, transformers, and multi-layer perceptrons. This hybrid design enables the model to capture both fine-grained local variations and long-range semantic context, addressing the two key challenges while reducing the cost of manual network design. To support efficient training of such a diverse search space, AutoNAD introduces a cross weight sharing strategy, which accelerates supernet convergence and improves subnet performance. Additionally, a searchable multi-level feature aggregation module (MFAM) is integrated to enhance multi-scale feature learning. Beyond detection accuracy, runtime efficiency is essential for industrial deployment. To this end, AutoNAD incorporates a latency-aware prior to guide the selection of efficient architectures. The effectiveness of AutoNAD is validated on three industrial defect datasets and further applied within a defect imaging and detection platform. Code is available at https://github.com/Yuxi104/AutoNAD.
♻ ☆ Logic OCR : Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?
Maoyuan Ye, Haibin He, Qihuang Zhong, Jing Zhang, Juhua Liu, Bo Du
Recent advances in Large Multimodal Models (LMMs) have revolutionized their reasoning and Optical Character Recognition (OCR) capabilities. However, their complex logical reasoning performance on text-rich images remains underexplored. To bridge this gap, we introduce LogicOCR, a benchmark comprising 2780 questions with two subsets, i.e., LogicOCR-Gen with 1100 multi-choice questions on generated images, and LogicOCR-Real with 1680 meticulously designed free-form questions on real-world images. For constructing LogicOCR-Gen, we first curate a text corpus from the Chinese National Civil Servant Examination, and customize an automatic pipeline to steer GPT-Image-1 to generate images with varied layouts and fonts, ensuring contextual relevance and visual realism. Then, the generated images are manually verified. We evaluate a range of representative LMMs under Chain-of-Thought (CoT) and direct-answer settings. Our multi-dimensional analysis reveals key insights, such as the impact of test-time scaling, input modality differences, and sensitivity to visual-text orientation. Notably, LMMs still lag in multimodal reasoning compared to text-only inputs, indicating that they have not fully bridged visual reading with reasoning. Moreover, we propose TextCue, a training-free method that enhances LMMs' perception of image regions containing important text cues for solving questions. We leverage LMMs' attention maps and an off-the-shelf text segmentation specialist to determine the region, which is then cropped and enlarged to augment the original image. Experiments show its effectiveness, e.g., a 1.8% accuracy gain over LLaVA-OV-1.5-8B under the CoT setting. Our benchmark is available at https://github.com/MiliLab/LogicOCR.
comment : GitHub: https://github.com/MiliLab/LogicOCR
♻ ☆ STARFlow-V: End-to-End Video Generative Modeling with Normalizing Flows
Jiatao Gu, Ying Shen, Tianrong Chen, Laurent Dinh, Yuyang Wang, Miguel Angel Bautista, David Berthelot, Josh Susskind, Shuangfei Zhai
Normalizing flows (NFs) are end-to-end likelihood-based generative models for continuous data, and have recently regained attention with encouraging progress on image generation. Yet in the video generation domain, where spatiotemporal complexity and computational cost are substantially higher, state-of-the-art systems almost exclusively rely on diffusion-based models. In this work, we revisit this design space by presenting STARFlow-V, a normalizing flow-based video generator with substantial benefits such as end-to-end learning, robust causal prediction, and native likelihood estimation. Building upon the recently proposed STARFlow, STARFlow-V operates in the spatiotemporal latent space with a global-local architecture which restricts causal dependencies to a global latent space while preserving rich local within-frame interactions. This eases error accumulation over time, a common pitfall of standard autoregressive diffusion model generation. Additionally, we propose flow-score matching, which equips the model with a light-weight causal denoiser to improve the video generation consistency in an autoregressive fashion. To improve the sampling efficiency, STARFlow-V employs a video-aware Jacobi iteration scheme that recasts inner updates as parallelizable iterations without breaking causality. Thanks to the invertible structure, the same model can natively support text-to-video, image-to-video as well as video-to-video generation tasks. Empirically, STARFlow-V achieves strong visual fidelity and temporal consistency with practical sampling throughput relative to diffusion-based baselines. These results present the first evidence, to our knowledge, that NFs are capable of high-quality autoregressive video generation, establishing them as a promising research direction for building world models. Code and generated samples are available at https://github.com/apple/ml-starflow.
comment : 21 pages, 9 figures. Code and samples are available at https://github.com/apple/ml-starflow
♻ ☆ Class-Independent Increment: An Efficient Approach for Multi-label Class-Incremental Learning
Chenhao Ding, Songlin Dong, Zhengdong Zhou, Jizhou Han, Qiang Wang, Yuhang He, Yihong Gong
Current research on class-incremental learning primarily focuses on single-label classification tasks. However, real-world applications often involve multi-label scenarios, such as image retrieval and medical imaging. Therefore, this paper focuses on the challenging yet practical multi-label class-incremental learning (MLCIL) problem. In addition to the challenge of catastrophic forgetting, MLCIL encounters issues related to feature confusion, encompassing inter-session and intra-feature confusion. To address these problems, we propose a novel MLCIL approach called class-independent increment (CLIN). Specifically, in contrast to existing methods that extract image-level features, we propose a class-independent incremental network (CINet) to extract multiple class-level embeddings for multi-label samples. It learns and preserves the knowledge of different classes by constructing class-specific tokens. On this basis, we develop two novel loss functions, optimizing the learning of class-specific tokens and class-level embeddings, respectively. These losses aim to distinguish between new and old classes, further alleviating the problem of feature confusion. Extensive experiments on MS-COCO and PASCAL VOC datasets demonstrate the effectiveness of our method for improving recognition performance and mitigating forgetting on various MLCIL tasks.
♻ ☆ Leveraging Contrast Information for Efficient Document Shadow Removal
Yifan Liu, Jiancheng Huang, Na Liu, Mingfu Yan, Yi Huang, Shifeng Chen
Document shadows are a major obstacle in the digitization process. Due to the dense information in text and patterns covered by shadows, document shadow removal requires specialized methods. Existing document shadow removal methods, although showing some progress, still rely on additional information such as shadow masks or lack generalization and effectiveness across different shadow scenarios. This often results in incomplete shadow removal or loss of original document content and tones. Moreover, these methods tend to underutilize the information present in the original shadowed document image. In this paper, we refocus our approach on the document images themselves, which inherently contain rich information.We propose an end-to-end document shadow removal method guided by contrast representation, following a coarse-to-fine refinement approach. By extracting document contrast information, we can effectively and quickly locate shadow shapes and positions without the need for additional masks. This information is then integrated into the refined shadow removal process, providing better guidance for network-based removal and feature fusion. Extensive qualitative and quantitative experiments show that our method achieves state-of-the-art performance.
comment : There are unresolved authorship disputes related to this submission, and the current version does not reflect an agreed authorship list
♻ ☆ Contrast-Prior Enhanced Duality for Mask-Free Shadow Removal
Jiyu Wu, Yifan Liu, Jiancheng Huang, Mingfu Yan, Shifeng Chen
Existing shadow removal methods often rely on shadow masks, which are challenging to acquire in real-world scenarios. Exploring intrinsic image cues, such as local contrast information, presents a potential alternative for guiding shadow removal in the absence of explicit masks. However, the cue's inherent ambiguity becomes a critical limitation in complex scenes, where it can fail to distinguish true shadows from low-reflectance objects and intricate background textures. To address this motivation, we propose the Adaptive Gated Dual-Branch Attention (AGBA) mechanism. AGBA dynamically filters and re-weighs the contrast prior to effectively disentangle shadow features from confounding visual elements. Furthermore, to tackle the persistent challenge of restoring soft shadow boundaries and fine-grained details, we introduce a diffusion-based Frequency-Contrast Fusion Network (FCFN) that leverages high-frequency and contrast cues to guide the generative process. Extensive experiments demonstrate that our method achieves state-of-the-art results among mask-free approaches while maintaining competitive performance relative to mask-based methods.
comment : There are unresolved authorship disputes related to this submission, and the current version does not reflect an agreed authorship list
♻ ☆ Benchmark ing the Trustworthiness in Multimodal LLMs for Video Understanding
Youze Wang, Zijun Chen, Ruoyu Chen, Shishen Gu, Wenbo Hu, Jiayang Liu, Yinpeng Dong, Hang Su, Jun Zhu, Meng Wang, Richang Hong
Recent advancements in multimodal large language models for video understanding (videoLLMs) have enhanced their capacity to process complex spatiotemporal data. However, challenges such as factual inaccuracies, harmful content, biases, hallucinations, and privacy risks compromise their reliability. This study introduces Trust-videoLLMs, a first comprehensive benchmark evaluating 23 state-of-the-art videoLLMs (5 commercial, 18 open-source) across five critical dimensions: truthfulness, robustness, safety, fairness, and privacy. Comprising 30 tasks with adapted, synthetic, and annotated videos, the framework assesses spatiotemporal risks, temporal consistency and cross-modal impact. Results reveal significant limitations in dynamic scene comprehension, cross-modal perturbation resilience and real-world risk mitigation. While open-source models occasionally outperform, proprietary models generally exhibit superior credibility, though scaling does not consistently improve performance. These findings underscore the need for enhanced training datat diversity and robust multimodal alignment. Trust-videoLLMs provides a publicly available, extensible toolkit for standardized trustworthiness assessments, addressing the critical gap between accuracy-focused benchmarks and demands for robustness, safety, fairness, and privacy.
♻ ☆ DEMIST: Decoupled Multi-stream latent diffusion for Quantitative Myelin Map Synthesis
Jiacheng Wang, Hao Li, Xing Yao, Ahmad Toubasi, Taegan Vinarsky, Caroline Gheen, Joy Derwenskus, Chaoyang Jin, Richard Dortch, Junzhong Xu, Francesca Bagnato, Ipek Oguz
Quantitative magnetization transfer (qMT) imaging provides myelin-sensitive biomarkers, such as the pool size ratio (PSR), which is valuable for multiple sclerosis (MS) assessment. However, qMT requires specialized 20-30 minute scans. We propose DEMIST to synthesize PSR maps from standard T1w and FLAIR images using a 3D latent diffusion model with three complementary conditioning mechanisms. Our approach has two stages: first, we train separate autoencoders for PSR and anatomical images to learn aligned latent representations. Second, we train a conditional diffusion model in this latent space on top of a frozen diffusion foundation backbone. Conditioning is decoupled into: (i) \textbf{semantic} tokens via cross-attention, (ii) \textbf{spatial} per-scale residual hints via a 3D ControlNet branch, and (iii) \textbf{adaptive} LoRA-modulated attention. We include edge-aware loss terms to preserve lesion boundaries and alignment losses to maintain quantitative consistency, while keeping the number of trainable parameters low and retaining the inductive bias of the pretrained model. We evaluate on 163 scans from 99 subjects using 5-fold cross-validation. Our method outperforms VAE, GAN and diffusion baselines on multiple metrics, producing sharper boundaries and better quantitative agreement with ground truth. Our code is publicly available at https://github.com/MedICL-VU/MS-Synthesis-3DcLDM.
♻ ☆ SOAP: Enhancing Spatio-Temporal Relation and Motion Information Capturing for Few-Shot Action Recognition ACM MM 2024
Wenbo Huang, Jinghui Zhang, Xuwei Qian, Zhen Wu, Meng Wang, Lei Zhang
High frame-rate (HFR) videos of action recognition improve fine-grained expression while reducing the spatio-temporal relation and motion information density. Thus, large amounts of video samples are continuously required for traditional data-driven training. However, samples are not always sufficient in real-world scenarios, promoting few-shot action recognition (FSAR) research. We observe that most recent FSAR works build spatio-temporal relation of video samples via temporal alignment after spatial feature extraction, cutting apart spatial and temporal features within samples. They also capture motion information via narrow perspectives between adjacent frames without considering density, leading to insufficient motion information capturing. Therefore, we propose a novel plug-and-play architecture for FSAR called Spatio-tempOral frAme tuPle enhancer (SOAP) in this paper. The model we designed with such architecture refers to SOAP-Net. Temporal connections between different feature channels and spatio-temporal relation of features are considered instead of simple feature extraction. Comprehensive motion information is also captured, using frame tuples with multiple frames containing more motion information than adjacent frames. Combining frame tuples of diverse frame counts further provides a broader perspective. SOAP-Net achieves new state-of-the-art performance across well-known benchmarks such as SthSthV2, Kinetics, UCF101, and HMDB51. Extensive empirical evaluations underscore the competitiveness, pluggability, generalization, and robustness of SOAP. The code is released at https://github.com/wenbohuang1002/SOAP.
comment : Accepted by ACM MM 2024
♻ ☆ RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness NeurIPS 2025
Fanhu Zeng, Haiyang Guo, Fei Zhu, Li Shen, Hao Tang
Fine-tuning pre-trained models with custom data leads to numerous expert models on specific tasks. Merging models into one universal model to empower multi-task ability refraining from data leakage has gained popularity. With the expansion in data and model size, parameter-efficient tuning becomes the common practice for obtaining task-specific models efficiently. However, few methods are dedicated to efficient merging, and existing methods designed for full fine-tuning merging fail under efficient merging. To address the issue, we analyze from low-rank decomposition and reveal that direction robustness during merging is crucial for merging efficient modules. We furthermore uncover that compensating for the gap between stark singular values contributes to direction robustness. Therefore, we propose RobustMerge, a training-free parameter-efficient merging method with complementary parameter adaptation to maintain direction robustness. Specifically, we (1) prune parameters and scale coefficients from inter-parameter relation for singular values to maintain direction stability away from task interference, and (2) perform cross-task normalization to enhance unseen task generalization. We establish a benchmark consisting of diverse multimodal tasks, on which we conduct experiments to certify the outstanding performance and generalizability of our method. Additional studies and extensive analyses further showcase the effectiveness. Code is available at https://github.com/AuroraZengfh/RobustMerge.
comment : NeurIPS 2025 (Spotlight) Fix some typos
♻ ☆ Vision -Language Enhanced Foundation Model for Semi-supervised Medical Image Segmentation
Jiaqi Guo, Mingzhen Li, Hanyu Su, Santiago López, Lexiaozi Fan, Daniel Kim, Aggelos Katsaggelos
Semi-supervised learning (SSL) has emerged as an effective paradigm for medical image segmentation, reducing the reliance on extensive expert annotations. Meanwhile, vision-language models (VLMs) have demonstrated strong generalization and few-shot capabilities across diverse visual domains. In this work, we integrate VLM-based segmentation into semi-supervised medical image segmentation by introducing a Vision-Language Enhanced Semi-supervised Segmentation Assistant (VESSA) that incorporates foundation-level visual-semantic understanding into SSL frameworks. Our approach consists of two stages. In Stage 1, the VLM-enhanced segmentation foundation model VESSA is trained as a reference-guided segmentation assistant using a template bank containing gold-standard exemplars, simulating learning from limited labeled data. Given an input-template pair, VESSA performs visual feature matching to extract representative semantic and spatial cues from exemplar segmentations, generating structured prompts for a SAM2-inspired mask decoder to produce segmentation masks. In Stage 2, VESSA is integrated into a state-of-the-art SSL framework, enabling dynamic interaction with the student model: as student predictions become more refined, they are fed back to VESSA as prompts, allowing it to generate higher-quality pseudo-labels and stronger guidance. Extensive experiments across multiple segmentation datasets and domains show that VESSA-augmented SSL significantly enhances segmentation accuracy, outperforming state-of-the-art baselines under extremely limited annotation conditions.
♻ ☆ OuroMamba: A Data-Free Quantization Framework for Vision Mamba ICCV 2025
Akshat Ramachandran, Mingyu Lee, Huan Xu, Souvik Kundu, Tushar Krishna
We present OuroMamba, the first data-free post-training quantization (DFQ) method for vision Mamba-based models (VMMs). We identify two key challenges in enabling DFQ for VMMs, (1) VMM's recurrent state transitions restricts capturing of long-range interactions and leads to semantically weak synthetic data, (2) VMM activations exhibit dynamic outlier variations across time-steps, rendering existing static PTQ techniques ineffective. To address these challenges, OuroMamba presents a two-stage framework: (1) OuroMamba-Gen to generate semantically rich and meaningful synthetic data. It applies contrastive learning on patch level VMM features generated through neighborhood interactions in the latent state space, (2) OuroMamba-Quant to employ mixed-precision quantization with lightweight dynamic outlier detection during inference. In specific, we present a thresholding based outlier channel selection strategy for activations that gets updated every time-step. Extensive experiments across vision and generative tasks show that our data-free OuroMamba surpasses existing data-driven PTQ techniques, achieving state-of-the-art performance across diverse quantization settings. Additionally, we implement efficient GPU kernels to achieve practical latency speedup of up to 2.36x. Code and synthetic dataset are available here: https://github.com/georgia-tech-synergy-lab/ICCV-OuroMamba
comment : Accepted to ICCV 2025
♻ ☆ Bayesian Neural Networks for One-to-Many Mapping in Image Enhancement
Guoxi Huang, Qirui Yang, Ruirui Lin, Zipeng Qi, David Bull, Nantheera Anantrasirichai
In image enhancement tasks, such as low-light and underwater image enhancement, a degraded image can correspond to multiple plausible target images due to dynamic photography conditions. This naturally results in a one-to-many mapping problem. To address this, we propose a Bayesian Enhancement Model (BEM) that incorporates Bayesian Neural Networks (BNNs) to capture data uncertainty and produce diverse outputs. To enable fast inference, we introduce a BNN-DNN framework: a BNN is first employed to model the one-to-many mapping in a low-dimensional space, followed by a Deterministic Neural Network (DNN) that refines fine-grained image details. Extensive experiments on multiple low-light and underwater image enhancement benchmarks demonstrate the effectiveness of our method.
♻ ☆ Weather Diffusion : Controllable Weather Editing in Intrinsic Space
Yixin Zhu, Zuoliang Zhu, Jian Yang, Miloš Hašan, Jin Xie, Beibei Wang
We present WeatherDiffusion, a diffusion-based framework for controllable weather editing in intrinsic space. Our framework includes two components based on diffusion priors: an inverse renderer that estimates material properties, scene geometry, and lighting as intrinsic maps from an input image, and a forward renderer that utilizes these geometry and material maps along with a text prompt that describes specific weather conditions to generate a final image. The intrinsic maps enhance controllability compared to traditional pixel-space editing approaches.We propose an intrinsic map-aware attention mechanism that improves spatial correspondence and decomposition quality in large outdoor scenes. For forward rendering, we leverage CLIP-space interpolation of weather prompts to achieve fine-grained weather control. We also introduce a synthetic and a real-world dataset, containing 38k and 18k images under various weather conditions, each with intrinsic map annotations. WeatherDiffusion outperforms state-of-the-art pixel-space editing approaches, weather restoration methods, and rendering-based methods, showing promise for downstream tasks such as autonomous driving, enhancing the robustness of detection and segmentation in challenging weather scenarios.
♻ ☆ MoEGCL: Mixture of Ego-Graphs Contrastive Representation Learning for Multi-View Clustering
Jian Zhu, Xin Zou, Jun Sun, Cheng Luo, Lei Liu, Lingfang Zeng, Ning Zhang, Bian Wu, Chang Tang, Lirong Dai
In recent years, the advancement of Graph Neural Networks (GNNs) has significantly propelled progress in Multi-View Clustering (MVC). However, existing methods face the problem of coarse-grained graph fusion. Specifically, current approaches typically generate a separate graph structure for each view and then perform weighted fusion of graph structures at the view level, which is a relatively rough strategy. To address this limitation, we present a novel Mixture of Ego-Graphs Contrastive Representation Learning (MoEGCL). It mainly consists of two modules. In particular, we propose an innovative Mixture of Ego-Graphs Fusion (MoEGF), which constructs ego graphs and utilizes a Mixture-of-Experts network to implement fine-grained fusion of ego graphs at the sample level, rather than the conventional view-level fusion. Additionally, we present the Ego Graph Contrastive Learning (EGCL) module to align the fused representation with the view-specific representation. The EGCL module enhances the representation similarity of samples from the same cluster, not merely from the same sample, further boosting fine-grained graph representation. Extensive experiments demonstrate that MoEGCL achieves state-of-the-art results in deep multi-view clustering tasks. The source code is publicly available at https://github.com/HackerHyper/MoEGCL.
♻ ☆ ReasonAct: Progressive Training for Fine-Grained Video Reasoning in Small Models
Jiaxin Liu, Zhaolu Kang
While recent multimodal models have shown progress in vision-language tasks, small-scale variants still struggle with the fine-grained temporal reasoning required for video understanding. We introduce ReasonAct, a method that enhances video reasoning in smaller models through a three-stage training process: first building a foundation with text-only reasoning, then fine-tuning on video, and finally refining with temporal-aware reinforcement learning. We build upon Temporal Group Relative Policy Optimization (T-GRPO) by incorporating temporal consistency modeling into policy optimization. We also propose a biomechanically-motivated sub-action decomposition mechanism that provides graduated rewards for constituent action phases. Through experiments on HMDB51, UCF-101, and Kinetics-400, our 3B-parameter model achieves 67.2%, 94.1%, and 78.9% accuracy respectively, demonstrating improvements of 17.9, 15.8, and 12.3 points over baselines. Ablation studies validate that our progressive training enables smaller models to achieve competitive video reasoning performance while maintaining computational efficiency.
♻ ☆ Occlusion Boundary and Depth: Mutual Enhancement via Multi-Task Learning WACV 2026
Lintao Xu, Yinghao Wang, Chaohui Wang
Occlusion Boundary Estimation (OBE) identifies boundaries arising from both inter-object occlusions and self-occlusion within individual objects. This task is closely related to Monocular Depth Estimation (MDE), which infers depth from a single image, as Occlusion Boundaries (OBs) provide critical geometric cues for resolving depth ambiguities, while depth can conversely refine occlusion reasoning. In this paper, we aim to systematically model and exploit this mutually beneficial relationship. To this end, we propose MoDOT, a novel framework for joint estimation of depth and OBs, which incorporates a new Cross-Attention Strip Module (CASM) to leverage mid-level OB features for depth prediction, and a novel OB-Depth Constraint Loss (OBDCL) to enforce geometric consistency. To facilitate this study, we contribute OB-Hypersim, a large-scale photorealistic dataset with precise depth and self-occlusion-handled OB annotations. Extensive experiments on two synthetic datasets and NYUD-v2 demonstrate that MoDOT achieves significantly better performance than single-task baselines and multi-task competitors. Furthermore, models trained solely on our synthetic data demonstrate strong generalization to real-world scenes without fine-tuning, producing depth maps with sharper boundaries and improved geometric fidelity. Collectively, these results underscore the significant benefits of jointly modeling OBs and depth. Code and resources are available at https://github.com/xul-ops/MoDOT.
comment : WACV 2026
♻ ☆ SONIC: Spectral Optimization of Noise for Inpainting with Consistency
Seungyeon Baek, Erqun Dong, Shadan Namazifard, Mark J. Matthews, Kwang Moo Yi
We propose a novel training-free method for inpainting with off-the-shelf text-to-image models. While guidance-based methods in theory allow generic models to be used for inverse problems such as inpainting, in practice, their effectiveness is limited, leading to the necessity of specialized inpainting-specific models. In this work, we argue that the missing ingredient for training-free inpainting is the optimization (guidance) of the initial seed noise. We propose to optimize the initial seed noise to approximately match the unmasked parts of the data - with as few as a few tens of optimization steps. We then apply conventional training-free inpainting methods on top of our optimized initial seed noise. Critically, we propose two core ideas to effectively implement this idea: (i) to avoid the costly unrolling required to relate the initial noise and the generated outcome, we perform linear approximation; and (ii) to stabilize the optimization, we optimize the initial seed noise in the spectral domain. We demonstrate the effectiveness of our method on various inpainting tasks, outperforming the state of the art. Project page: https://ubc-vision.github.io/sonic/
♻ ☆ MIRNet: Integrating Constrained Graph-Based Reasoning with Pre-train ing for Diagnostic Medical Imaging AAAI-26
Shufeng Kong, Zijie Wang, Nuan Cui, Hao Tang, Yihan Meng, Yuanyuan Wei, Feifan Chen, Yingheng Wang, Zhuo Cai, Yaonan Wang, Yulong Zhang, Yuzheng Li, Zibin Zheng, Caihua Liu, Hao Liang
Automated interpretation of medical images demands robust modeling of complex visual-semantic relationships while addressing annotation scarcity, label imbalance, and clinical plausibility constraints. We introduce MIRNet (Medical Image Reasoner Network), a novel framework that integrates self-supervised pre-training with constrained graph-based reasoning. Tongue image diagnosis is a particularly challenging domain that requires fine-grained visual and semantic understanding. Our approach leverages self-supervised masked autoencoder (MAE) to learn transferable visual representations from unlabeled data; employs graph attention networks (GAT) to model label correlations through expert-defined structured graphs; enforces clinical priors via constraint-aware optimization using KL divergence and regularization losses; and mitigates imbalance using asymmetric loss (ASL) and boosting ensembles. To address annotation scarcity, we also introduce TongueAtlas-4K, a comprehensive expert-curated benchmark comprising 4,000 images annotated with 22 diagnostic labels--representing the largest public dataset in tongue analysis. Validation shows our method achieves state-of-the-art performance. While optimized for tongue diagnosis, the framework readily generalizes to broader diagnostic medical imaging tasks.
comment : To appear at AAAI-26
♻ ☆ Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering
Federico Felizzi, Olivia Riccomi, Michele Ferramola, Francesco Andrea Causio, Manuel Del Medico, Vittorio De Vita, Lorenzo De Mori, Alessandra Piscitelli, Pietro Eric Risuleo, Bianca Destro Castaniti, Antonio Cristiano, Alessia Longo, Luigi De Angelis, Mariapia Vassalli, Marcello Di Pumpo
Large vision language models (VLMs) have achieved impressive performance on medical visual question answering benchmarks, yet their reliance on visual information remains unclear. We investigate whether frontier VLMs demonstrate genuine visual grounding when answering Italian medical questions by testing four state-of-the-art models: Claude Sonnet 4.5, GPT-4o, GPT-5-mini, and Gemini 2.0 flash exp. Using 60 questions from the EuropeMedQA Italian dataset that explicitly require image interpretation, we substitute correct medical images with blank placeholders to test whether models truly integrate visual and textual information. Our results reveal striking variability in visual dependency: GPT-4o shows the strongest visual grounding with a 27.9pp accuracy drop (83.2% [74.6%, 91.7%] to 55.3% [44.1%, 66.6%]), while GPT-5-mini, Gemini, and Claude maintain high accuracy with modest drops of 8.5pp, 2.4pp, and 5.6pp respectively. Analysis of model-generated reasoning reveals confident explanations for fabricated visual interpretations across all models, suggesting varying degrees of reliance on textual shortcuts versus genuine visual analysis. These findings highlight critical differences in model robustness and the need for rigorous evaluation before clinical deployment.
comment : Accepted at the Workshop on Multimodal Representation Learning for Healthcare (MMRL4H), EurIPS 2025
♻ ☆ VITA: Zero-Shot Value Functions via Test-Time Adaptation of Vision -Language Models
Christos Ziakas, Alessandra Russo
Vision-Language Models (VLMs) show promise as zero-shot goal-conditioned value functions, but their frozen pre-trained representations limit generalization and temporal reasoning. We introduce VITA, a zero-shot value function learning method that enhances both capabilities via test-time adaptation. At inference, a lightweight adaptation module is updated via a gradient step on a meta-learned self-supervised loss, such that each test-time update improves value estimation. By updating sequentially over a trajectory, VITA encodes history into its parameters, addressing the temporal reasoning limitations. To mitigate shortcut learning, we propose a dissimilarity-based sampling strategy that selects semantically diverse segments of the trajectory during training. In real-world robotic manipulation tasks, VITA generalizes from a single training environment to diverse out-of-distribution tasks, environments, and embodiments, outperforming the state-of-the-art zero-shot method using autoregressive VLMs. Furthermore, we demonstrate that VITA's zero-shot value estimates can be utilized for reward shaping in offline reinforcement learning, resulting in multi-task policies on the Meta-World benchmark that exceed the performance of those trained with the simulation's fuzzy-logic dense rewards.
♻ ☆ SplitFlux: Learning to Decouple Content and Style from a Single Image
Yitong Yang, Yinglin Wang, Changshuo Wang, Yongjun Zhang, Ziyang Chen, Shuting He
Disentangling image content and style is essential for customized image generation. Existing SDXL-based methods struggle to achieve high-quality results, while the recently proposed Flux model fails to achieve effective content-style separation due to its underexplored characteristics. To address these challenges, we conduct a systematic analysis of Flux and make two key observations: (1) Single Stream Blocks are essential for image generation; and (2) Early single stream blocks mainly control content, whereas later blocks govern style. Based on these insights, we propose SplitFlux, which disentangles content and style by fine-tuning the single stream blocks via LoRA, enabling the disentangled content to be re-embedded into new contexts. It includes two key components: (1) Rank-Constrained Adaptation. To preserve content identity and structure, we compress the rank and amplify the magnitude of updates within specific blocks, preventing content leakage into style blocks. (2) Visual-Gated LoRA. We split the content LoRA into two branches with different ranks, guided by image saliency. The high-rank branch preserves primary subject information, while the low-rank branch encodes residual details, mitigating content overfitting and enabling seamless re-embedding. Extensive experiments demonstrate that SplitFlux consistently outperforms state-of-the-art methods, achieving superior content preservation and stylization quality across diverse scenarios.
♻ ☆ Terminal Velocity Matching
Linqi Zhou, Mathias Parger, Ayaan Haque, Jiaming Song
We propose Terminal Velocity Matching (TVM), a generalization of flow matching that enables high-fidelity one- and few-step generative modeling. TVM models the transition between any two diffusion timesteps and regularizes its behavior at its terminal time rather than at the initial time. We prove that TVM provides an upper bound on the $2$-Wasserstein distance between data and model distributions when the model is Lipschitz continuous. However, since Diffusion Transformers lack this property, we introduce minimal architectural changes that achieve stable, single-stage training. To make TVM efficient in practice, we develop a fused attention kernel that supports backward passes on Jacobian-Vector Products, which scale well with transformer architectures. On ImageNet-256x256, TVM achieves 3.29 FID with a single function evaluation (NFE) and 1.99 FID with 4 NFEs. It similarly achieves 4.32 1-NFE FID and 2.94 4-NFE FID on ImageNet-512x512, representing state-of-the-art performance for one/few-step models from scratch.
comment : Blog post: https://lumalabs.ai/blog/engineering/tvm Code available at: https://github.com/lumalabs/tvm
♻ ☆ ProxT2I: Efficient Reward-Guided Text-to-Image Generation via Proximal Diffusion
Zhenghan Fang, Jian Zheng, Qiaozi Gao, Xiaofeng Gao, Jeremias Sulam
Diffusion models have emerged as a dominant paradigm for generative modeling across a wide range of domains, including prompt-conditional generation. The vast majority of samplers, however, rely on forward discretization of the reverse diffusion process and use score functions that are learned from data. Such forward and explicit discretizations can be slow and unstable, requiring a large number of sampling steps to produce good-quality samples. In this work we develop a text-to-image (T2I) diffusion model based on backward discretizations, dubbed ProxT2I, relying on learned and conditional proximal operators instead of score functions. We further leverage recent advances in reinforcement learning and policy optimization to optimize our samplers for task-specific rewards. Additionally, we develop a new large-scale and open-source dataset comprising 15 million high-quality human images with fine-grained captions, called LAION-Face-T2I-15M, for training and evaluation. Our approach consistently enhances sampling efficiency and human-preference alignment compared to score-based baselines, and achieves results on par with existing state-of-the-art and open-source text-to-image models while requiring lower compute and smaller model size, offering a lightweight yet performant solution for human text-to-image generation.
♻ ☆ Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding in Novel Domains WACV 2026
Zitian Tang, Rohan Myer Krishnan, Zhiqiu Yu, Chen Sun
Learning from (procedural) videos has increasingly served as a pathway for embodied agents to acquire skills from human demonstrations. To do this, video understanding models must be able to obtain structured understandings, such as the temporal segmentation of a demonstration into sequences of actions and skills, and to generalize the understandings to novel environments, tasks, and problem domains. In pursuit of this goal, we introduce Spacewalk-18, a benchmark containing two tasks: (1) step recognition and (2) video question answering, over a dataset of temporally segmented and labeled tasks in International Space Station spacewalk recordings. In tandem, the two tasks quantify a model's ability to: (1) generalize to novel domains; (2) utilize long temporal context and multimodal (e.g. visual and speech) information. Our extensive experimental analysis highlights the challenges of Spacewalk-18, but also suggests best practices for domain generalization and long-form understanding. Notably, we discover a promising adaptation via summarization technique that leads to significant performance improvement without model fine-tuning. The Spacewalk-18 benchmark is released at https://brown-palm.github.io/Spacewalk-18/.
comment : WACV 2026
Computation and Language 101
☆ Revisiting Generalization Across Difficulty Levels: It's Not So Easy
Yeganeh Kordi, Nihal V. Nayak, Max Zuo, Ilana Nguyen, Stephen H. Bach
We investigate how well large language models (LLMs) generalize across different task difficulties, a key question for effective data curation and evaluation. Existing research is mixed regarding whether training on easier or harder data leads to better results, and whether those gains come on easier or harder test data. We address this question by conducting a systematic evaluation of LLMs' generalization across models, datasets, and fine-grained groups of example difficulty. We rank examples in six datasets using the outputs of thousands of different LLMs and Item Response Theory (IRT), a well-established difficulty metric in educational testing. Unlike prior work, our difficulty ratings are therefore determined solely by the abilities of many different LLMs, excluding human opinions of difficulty. With a more objective, larger-scale, and finer-grained analysis, we show that cross-difficulty generalization is often limited; training on either easy or hard data cannot achieve consistent improvements across the full range of difficulties. These results show the importance of having a range of difficulties in both training and evaluation data for LLMs, and that taking shortcuts with respect to difficulty is risky.
☆ ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration
Hongjin Su, Shizhe Diao, Ximing Lu, Mingjie Liu, Jiacheng Xu, Xin Dong, Yonggan Fu, Peter Belcak, Hanrong Ye, Hongxu Yin, Yi Dong, Evelina Bakhturina, Tao Yu, Yejin Choi, Jan Kautz, Pavlo Molchanov
Large language models are powerful generalists, yet solving deep and complex problems such as those of the Humanity's Last Exam (HLE) remains both conceptually challenging and computationally expensive. We show that small orchestrators managing other models and a variety of tools can both push the upper bound of intelligence and improve efficiency in solving difficult agentic tasks. We introduce ToolOrchestra, a method for training small orchestrators that coordinate intelligent tools. ToolOrchestra explicitly uses reinforcement learning with outcome-, efficiency-, and user-preference-aware rewards. Using ToolOrchestra, we produce Orchestrator, an 8B model that achieves higher accuracy at lower cost than previous tool-use agents while aligning with user preferences on which tools are to be used for a given query. On HLE, Orchestrator achieves a score of 37.1%, outperforming GPT-5 (35.1%) while being 2.5x more efficient. On tau2-Bench and FRAMES, Orchestrator surpasses GPT-5 by a wide margin while using only about 30% of the cost. Extensive analysis shows that Orchestrator achieves the best trade-off between performance and cost under multiple metrics, and generalizes robustly to unseen tools. These results demonstrate that composing diverse tools with a lightweight orchestration model is both more efficient and more effective than existing methods, paving the way for practical and scalable tool-augmented reasoning systems.
comment : 21 pages, 6 figures
☆ G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
Wenbo Hu, Jingli Lin, Yilin Long, Yunlong Ran, Lihan Jiang, Yifan Wang, Chenming Zhu, Runsen Xu, Tai Wang, Jiangmiao Pang
Vision-Language Models (VLMs) still lack robustness in spatial intelligence, demonstrating poor performance on spatial understanding and reasoning tasks. We attribute this gap to the absence of a visual geometry learning process capable of reconstructing 3D space from 2D images. We present G$^2$VLM, a geometry grounded vision-language model that bridges two fundamental aspects of spatial intelligence: spatial 3D reconstruction and spatial understanding. G$^2$VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning. Our unified design is highly scalable for spatial understanding: it trains on abundant multi-view image and video data, while simultaneously leveraging the benefits of 3D visual priors that are typically only derived from hard-to-collect annotations. Experimental results demonstrate G$^2$VLM is proficient in both tasks, achieving comparable results to state-of-the-art feed-forward 3D reconstruction models and achieving better or competitive results across spatial understanding and reasoning tasks. By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G$^2$VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing.
comment : code are released at https://github.com/InternRobotics/G2VLM
☆ Matrix: Peer-to-Peer Multi- Agent Synthetic Data Generation Framework
Dong Wang, Yang Li, Ansong Ni, Ching-Feng Yeh, Youssef Emad, Xinjie Lei, Liam Robbins, Karthik Padthe, Hu Xu, Xian Li, Asli Celikyilmaz, Ramya Raghavendra, Lifei Huang, Carole-Jean Wu, Shang-Wen Li
Synthetic data has become increasingly important for training large language models, especially when real data is scarce, expensive, or privacy-sensitive. Many such generation tasks require coordinated multi-agent workflows, where specialized agents collaborate to produce data that is higher quality, more diverse, and structurally richer. However, existing frameworks for multi-agent synthesis often depend on a centralized orchestrator, creating scalability bottlenecks, or are hardcoded for specific domains, limiting flexibility. We present \textbf{Matrix}, a decentralized framework that represents both control and data flow as serialized messages passed through distributed queues. This peer-to-peer design eliminates the central orchestrator. Each task progresses independently through lightweight agents, while compute-intensive operations, such as LLM inference or containerized environments, are handled by distributed services. Built on Ray, Matrix scales to tens of thousands of concurrent agentic workflows and provides a modular, configurable design that enables easy adaptation to a wide range of data generation workflows. We evaluate Matrix across diverse synthesis scenarios, such as multi-agent collaborative dialogue, web-based reasoning data extraction, and tool-use trajectory generation in customer service environments. In all cases, Matrix achieves $2$--$15\times$ higher data generation throughput under identical hardware resources, without compromising output quality.
☆ The author is dead, but what if they never lived? A reception experiment on Czech AI- and human-authored poetry
Anna Marklová, Ondřej Vinš, Martina Vokáčová, Jiří Milička
Large language models are increasingly capable of producing creative texts, yet most studies on AI-generated poetry focus on English -- a language that dominates training data. In this paper, we examine the perception of AI- and human-written Czech poetry. We ask if Czech native speakers are able to identify it and how they aesthetically judge it. Participants performed at chance level when guessing authorship (45.8\% correct on average), indicating that Czech AI-generated poems were largely indistinguishable from human-written ones. Aesthetic evaluations revealed a strong authorship bias: when participants believed a poem was AI-generated, they rated it as less favorably, even though AI poems were in fact rated equally or more favorably than human ones on average. The logistic regression model uncovered that the more the people liked a poem, the less probable was that they accurately assign the authorship. Familiarity with poetry or literary background had no effect on recognition accuracy. Our findings show that AI can convincingly produce poetry even in a morphologically complex, low-resource (with respect of the training data of AI models) Slavic language such as Czech. The results suggest that readers' beliefs about authorship and the aesthetic evaluation of the poem are interconnected.
☆ TAGFN: A Text-Attributed Graph Dataset for Fake News Detection in the Age of LLMs
Kay Liu, Yuwei Han, Haoyan Xu, Henry Peng Zou, Yue Zhao, Philip S. Yu
Large Language Models (LLMs) have recently revolutionized machine learning on text-attributed graphs, but the application of LLMs to graph outlier detection, particularly in the context of fake news detection, remains significantly underexplored. One of the key challenges is the scarcity of large-scale, realistic, and well-annotated datasets that can serve as reliable benchmarks for outlier detection. To bridge this gap, we introduce TAGFN, a large-scale, real-world text-attributed graph dataset for outlier detection, specifically fake news detection. TAGFN enables rigorous evaluation of both traditional and LLM-based graph outlier detection methods. Furthermore, it facilitates the development of misinformation detection capabilities in LLMs through fine-tuning. We anticipate that TAGFN will be a valuable resource for the community, fostering progress in robust graph-based outlier detection and trustworthy AI. The dataset is publicly available at https://huggingface.co/datasets/kayzliu/TAGFN and our code is available at https://github.com/kayzliu/tagfn.
comment : Preprint. Under review
☆ Beyond U RL s: Metadata Diversity and Position for Efficient LLM Pretrain ing
Dongyang Fan, Diba Hashemi, Sai Praneeth Karimireddy, Martin Jaggi
Incorporating metadata in Large Language Models (LLMs) pretraining has recently emerged as a promising approach to accelerate training. However prior work highlighted only one useful signal-URLs, leaving open the question of whether other forms of metadata could yield greater benefits. In this study, we investigate a wider range of metadata types and find other types of metadata, such as fine-grained indicators of document quality that can also accelerate pretraining when prepended. We identify a common feature among effective metadata: they encode information at a finer granularity. We further introduce metadata appending as a means of improving training efficiency, where predicting an appropriate metadata as auxiliary task can help speed up pretraining. In addition, learnable meta-tokens trained with masked loss can recover part of the speedup by inducing quality-aware latent structure. Using probing, we analyze latent representations to understand how metadata shapes learning. Together, these results yield practical guidelines for integrating metadata to improve both the efficiency and effectiveness of LLM pretraining.
☆ Auxiliary Metrics Help Decoding Skill Neurons in the Wild
Yixiu Zhao, Xiaozhi Wang, Zijun Yao, Lei Hou, Juanzi Li
Large language models (LLMs) exhibit remarkable capabilities across a wide range of tasks, yet their internal mechanisms remain largely opaque. In this paper, we introduce a simple, lightweight, and broadly applicable method with a focus on isolating neurons that encode specific skills. Building upon prior work that identified "skill neurons" via soft prompt training on classification tasks, our approach extends the analysis to complex scenarios involving multiple skills. We correlate neuron activations with auxiliary metrics -- such as external labels and the model's own confidence score -- thereby uncovering interpretable and task-specific behaviors without the need for manual token aggregation. We empirically validate our method on tasks spanning open-ended text generation and natural language inference, demonstrating its ability to detect neurons that not only drive known skills but also reveal previously unidentified shortcuts in arithmetic reasoning on BigBench.
comment : 7 pages, 7 figures. Includes additional appendix
☆ RoParQ: Paraphrase-Aware Alignment of Large Language Models Towards Robustness to Paraphrased Questions
Minjoon Choi
Large Language Models (LLMs) often exhibit inconsistent behavior when answering paraphrased questions, suggesting a reliance on surface-level patterns rather than true semantic understanding. To address this limitation, we introduce RoParQ, a benchmark specifically constructed to evaluate cross-paraphrase consistency in closed-book multiple-choice QA. This benchmark is derived from standard datasets by generating paraphrases via proprietary models and selectively retaining examples that elicit inconsistent confidence from a judge model. We further propose XParaCon, a novel evaluation metric that quantifies a model's robustness by measuring the standard deviation of accuracies across question variants. Additionally, we implement a reasoning-based, paraphrase-aware Supervised Fine-Tuning (SFT) strategy designed to align models toward semantic invariance. Our experiments demonstrate that this targeted alignment significantly enhances robustness. Notably, fine-tuned lightweight models achieved consistency levels comparable to much larger pre-trained models. These results highlight the efficacy of our approach in mitigating superficial memorization and fostering more robust, reliable LLMs.
comment : 12 pages, 9 figures, 8 tables
☆ Bangla Sign Language Translation: Dataset Creation Challenges, Benchmark ing and Prospects
Husne Ara Rubaiyeat, Hasan Mahmud, Md Kamrul Hasan
Bangla Sign Language Translation (BdSLT) has been severely constrained so far as the language itself is very low resource. Standard sentence level dataset creation for BdSLT is of immense importance for developing AI based assistive tools for deaf and hard of hearing people of Bangla speaking community. In this paper, we present a dataset, IsharaKhobor , and two subset of it for enabling research. We also present the challenges towards developing the dataset and present some way forward by benchmarking with landmark based raw and RQE embedding. We do some ablation on vocabulary restriction and canonicalization of the same within the dataset, which resulted in two more datasets, IsharaKhobor_small and IsharaKhobor_canonical_small. The dataset is publicly available at: www.kaggle.com/datasets/hasanssl/isharakhobor [1].
comment : 14 pages, 8 tables
☆ Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation LREC 2026
Lina Conti, Dennis Fucci, Marco Gaido, Matteo Negri, Guillaume Wisniewski, Luisa Bentivogli
Unlike text, speech conveys information about the speaker, such as gender, through acoustic cues like pitch. This gives rise to modality-specific bias concerns. For example, in speech translation (ST), when translating from languages with notional gender, such as English, into languages where gender-ambiguous terms referring to the speaker are assigned grammatical gender, the speaker's vocal characteristics may play a role in gender assignment. This risks misgendering speakers, whether through masculine defaults or vocal-based assumptions. Yet, how ST models make these decisions remains poorly understood. We investigate the mechanisms ST models use to assign gender to speaker-referring terms across three language pairs (en-es/fr/it), examining how training data patterns, internal language model (ILM) biases, and acoustic information interact. We find that models do not simply replicate term-specific gender associations from training data, but learn broader patterns of masculine prevalence. While the ILM exhibits strong masculine bias, models can override these preferences based on acoustic input. Using contrastive feature attribution on spectrograms, we reveal that the model with higher gender accuracy relies on a previously unknown mechanism: using first-person pronouns to link gendered terms back to the speaker, accessing gender information distributed across the frequency spectrum rather than concentrated in pitch.
comment : Submitted to LREC 2026
☆ Hierarchical Ranking Neural Network for Long Document Readability Assessment
Yurui Zheng, Yijun Chen, Shaohong Zhang
Readability assessment aims to evaluate the reading difficulty of a text. In recent years, while deep learning technology has been gradually applied to readability assessment, most approaches fail to consider either the length of the text or the ordinal relationship of readability labels. This paper proposes a bidirectional readability assessment mechanism that captures contextual information to identify regions with rich semantic information in the text, thereby predicting the readability level of individual sentences. These sentence-level labels are then used to assist in predicting the overall readability level of the document. Additionally, a pairwise sorting algorithm is introduced to model the ordinal relationship between readability levels through label subtraction. Experimental results on Chinese and English datasets demonstrate that the proposed model achieves competitive performance and outperforms other baseline models.
☆ A Systematic Study of Model Merging Techniques in Large Language Models
Oğuz Kağan Hitit, Leander Girrbach, Zeynep Akata
Model merging combines multiple fine-tuned checkpoints into a single model without additional training, offering an attractive approach to reusing models and efficiently improving performance. However, it remains unclear whether the advantages reported for smaller models and classifiers generalize to LLMs. We present a large-scale, systematic evaluation of six state-of-the-art merging methods, including recent subspace methods, across four open-weight LLMs, twelve fine-tuned checkpoints per base model, and sixteen standard LLM benchmarks. Evaluating through standardized benchmarks, we measure both the probability that a merged model outperforms the base model and relative gains over the best individual checkpoint. Our results show that the oldest and simplest method, Task Arithmetic, is the only approach that reliably yields performance gains on LLMs. Other interference-aware and subspace merging methods typically result in significant performance drops. Our findings indicate that current merging techniques do not directly transfer to modern LLMs. This motivates the design of LLM-specific merging algorithms and merging-aware fine-tuning methods. Code will be released upon acceptance of this paper.
☆ Odin: Oriented Dual-module Integration for Text-rich Network Representation Learning
Kaifeng Hong, Yinglong Zhang, Xiaoying Hong, Xuewen Xia, Xing Xu
Text-attributed graphs require models to effectively combine strong textual understanding with structurally informed reasoning. Existing approaches either rely on GNNs--limited by over-smoothing and hop-dependent diffusion--or employ Transformers that overlook graph topology and treat nodes as isolated sequences. We propose Odin (Oriented Dual-module INtegration), a new architecture that injects graph structure into Transformers at selected depths through an oriented dual-module mechanism.Unlike message-passing GNNs, Odin does not rely on multi-hop diffusion; instead, multi-hop structures are integrated at specific Transformer layers, yielding low-, mid-, and high-level structural abstraction aligned with the model's semantic hierarchy. Because aggregation operates on the global [CLS] representation, Odin fundamentally avoids over-smoothing and decouples structural abstraction from neighborhood size or graph topology. We further establish that Odin's expressive power strictly contains that of both pure Transformers and GNNs.To make the design efficient in large-scale or low-resource settings, we introduce Light Odin, a lightweight variant that preserves the same layer-aligned structural abstraction for faster training and inference. Experiments on multiple text-rich graph benchmarks show that Odin achieves state-of-the-art accuracy, while Light Odin delivers competitive performance with significantly reduced computational cost. Together, Odin and Light Odin form a unified, hop-free framework for principled structure-text integration. The source code of this model has been released at https://github.com/hongkaifeng/Odin.
comment : 32 pages, 2 figures
☆ Subjective Depth and Timescale Transformer s: Learning Where and When to Compute
Frederico Wieser, Martin Benfeghoul, Haitham Bou Ammar, Jun Wang, Zafeirios Fountas
The rigid, uniform allocation of computation in standard Transformer (TF) architectures can limit their efficiency and scalability, particularly for large-scale models and long sequences. Addressing this, we introduce Subjective Depth Transformers (SDT) and Subjective Timescale Transformers (STT), two distinct architectures that leverage Bayesian surprise signals to dynamically route computation, learning where and when to compute within decoder-only TFs. SDT augments a decoder-only stack with alternating Decision and Dynamic layers: a Decision layer computes a full block 'posterior' and a lightweight 'prior,' while a Dynamic layer employs fixed-capacity Top-K routing based on Bayesian surprise (Expected and Unexpected Change), maintaining a static compute graph. STT extends this conditional computation to the temporal domain: a transition network predicts residual updates, forming a temporal 'change hypothesis' that informs a router to dynamically execute or bypass TF blocks for each token, managing KV-cache contributions. Both architectures exhibit the predicted shift from novelty to prediction driven gating over training, suggesting alignment with surprise based principles. While operating at reduced capacity, they offer preliminary insights into the compute-accuracy trade-offs of conditional computation. The proposed architectures establish a flexible framework for efficiency, reducing self-attention computation by 75% and KV-cache requirements by 50% within each compute skipping layer, setting a pathway for more efficient models.
☆ Text-to-SQL as Dual-State Reasoning : Integrating Adaptive Context and Progressive Generation
Zhifeng Hao, Qibin Song, Ruichu Cai, Boyan Xu
Recent divide-and-conquer reasoning approaches, particularly those based on Chain-of-Thought (CoT), have substantially improved the Text-to-SQL capabilities of Large Language Models (LLMs). However, when applied to complex enterprise databases, such methods struggle to maintain coherent reasoning due to limited context capacity, unreliable schema linking, and weak grounding in database semantics. To overcome these issues, we introduce DSR-SQL, a \textbf{D}ual-\textbf{S}tate \textbf{R}easoning framework that models Text-to-SQL as an interaction between an adaptive context state and a progressive generation state. The first constructs a compact, semantically faithful environment by refining large schemas and selecting relevant structures, while the second formalizes SQL synthesis as feedback-guided state transitions, enabling the model to self-correct and align with user intent. Without any post-training or in-context examples, DSR-SQL achieves competitive performance, reaching 35.28\% execution accuracy on Spider 2.0-Snow and 68.32\% on BIRD development set. Our implementation will be open-sourced at: https://github.com/DMIRLAB-Group/DSR-SQL.
☆ Can LLMs extract human-like fine-grained evidence for evidence-based fact-checking?
Antonín Jarolím, Martin Fajčík, Lucia Makaiová
Misinformation frequently spreads in user comments under online news articles, highlighting the need for effective methods to detect factually incorrect information. To strongly support or refute claims extracted from such comments, it is necessary to identify relevant documents and pinpoint the exact text spans that justify or contradict each claim. This paper focuses on the latter task -- fine-grained evidence extraction for Czech and Slovak claims. We create new dataset, containing two-way annotated fine-grained evidence created by paid annotators. We evaluate large language models (LLMs) on this dataset to assess their alignment with human annotations. The results reveal that LLMs often fail to copy evidence verbatim from the source text, leading to invalid outputs. Error-rate analysis shows that the {llama3.1:8b model achieves a high proportion of correct outputs despite its relatively small size, while the gpt-oss-120b model underperforms despite having many more parameters. Furthermore, the models qwen3:14b, deepseek-r1:32b, and gpt-oss:20b demonstrate an effective balance between model size and alignment with human annotations.
☆ Training Introspective Behavior: Fine-Tuning Induces Reliable Internal State Detection in a 7B Model
Joshua Fonseca Rivera
Lindsey (2025) investigates introspective awareness in language models through four experiments, finding that models can sometimes detect and identify injected activation patterns -- but unreliably (~20% success in the best model). We focus on the first of these experiments -- self-report of injected "thoughts" -- and ask whether this capability can be directly trained rather than waiting for emergence. Through fine-tuning on transient single-token injections, we transform a 7B parameter model from near-complete failure (0.4% accuracy, 6.7% false positive rate) to reliable detection (85% accuracy on held-out concepts at α=40, 0% false positives). Our model detects fleeting "thoughts" injected at a single token position, retains that information, and reports the semantic content across subsequent generation steps. On this task, our trained model satisfies three of Lindsey's criteria: accuracy (correct identification), grounding (0/60 false positives), and internality (detection precedes verbalization). Generalization to unseen concept vectors (7.5pp gap) demonstrates the model learns a transferable skill rather than memorizing specific vectors, though this does not establish metacognitive representation in Lindsey's sense. These results address an open question raised by Lindsey: whether "training for introspection would help eliminate cross-model differences." We show that at least one component of introspective behavior can be directly induced, offering a pathway to built-in AI transparency.
comment : 16 pages, 8 figures
☆ Prune4Web: DOM Tree Pruning Programming for Web Agent AAAI 2026
Jiayuan Zhang, Kaiquan Chen, Zhihao Lu, Enshen Zhou, Qian Yu, Jing Zhang
Web automation employs intelligent agents to execute high-level tasks by mimicking human interactions with web interfaces. Despite the capabilities of recent Large Language Model (LLM)-based web agents, navigating complex, real-world webpages efficiently remains a significant hurdle due to the prohibitively large size of Document Object Model (DOM) structures, often ranging from 10,000 to 100,000 tokens. Existing strategies typically rely on crude DOM truncation -- risking the loss of critical information -- or employ inefficient heuristics and separate ranking models, failing to achieve an optimal balance between precision and scalability. To address these challenges, we introduce Prune4Web, a novel paradigm that shifts DOM processing from resource-intensive LLM reading to efficient programmatic pruning. Central to our approach is DOM Tree Pruning Programming, where an LLM generates executable Python scoring scripts to dynamically filter DOM elements based on semantic cues from decomposed sub-tasks. This mechanism eliminates the need for LLMs to ingest raw, massive DOMs, instead delegating traversal and scoring to lightweight, interpretable programs. This methodology achieves a 25x to 50x reduction in candidate elements for grounding, thereby facilitating precise action localization while mitigating attention dilution. Furthermore, we propose a specialized data annotation pipeline and a two-turn dialogue training strategy that jointly optimizes the Planner, Programmatic Filter, and Grounder within a unified framework. Extensive experiments demonstrate state-of-the-art performance. Notably, on our low-level grounding task, Prune4Web dramatically improves accuracy from 46.8% to 88.28%, underscoring its efficacy in real-world web automation.
comment : Paper accepted to AAAI 2026
☆ Do Reasoning Vision -Language Models Inversely Scale in Test-Time Compute? A Distractor-centric Empirical Analysis
Jiyun Bae, Hyunjong Ok, Sangwoo Mo, Jaeho Lee
How does irrelevant information (i.e., distractors) affect test-time scaling in vision-language models (VLMs)? Prior studies on language models have reported an inverse scaling effect, where textual distractors lead to longer but less effective reasoning. To investigate whether similar phenomena occur in multimodal settings, we introduce Idis (Images with distractors), a visual question-answering dataset that systematically varies distractors along semantic, numerical, and spatial dimensions. Our analyses reveal that visual distractors differ fundamentally from textual ones: although inverse scaling persists, adding visual distractors reduces accuracy without increasing reasoning length. We further show that tracking attribute counts within reasoning traces provides key insights into how distractors, reasoning length, and accuracy interact. Finally, we demonstrate that these trends extend to established visual bias benchmarks such as Waterbirds, and we propose a simple prompting strategy to mitigate bias-driven predictions in reasoning models.
comment : preprint
☆ BanglaASTE: A Novel Framework for Aspect-Sentiment-Opinion Extraction in Bangla E-commerce Review s Using Ensemble Deep Learning SP
Ariful Islam, Md Rifat Hossen, Abir Ahmed, B M Taslimul Haque
Aspect-Based Sentiment Analysis (ABSA) has emerged as a critical tool for extracting fine-grained sentiment insights from user-generated content, particularly in e-commerce and social media domains. However, research on Bangla ABSA remains significantly underexplored due to the absence of comprehensive datasets and specialized frameworks for triplet extraction in this language. This paper introduces BanglaASTE, a novel framework for Aspect Sentiment Triplet Extraction (ASTE) that simultaneously identifies aspect terms, opinion expressions, and sentiment polarities from Bangla product reviews. Our contributions include: (1) creation of the first annotated Bangla ASTE dataset containing 3,345 product reviews collected from major e-commerce platforms including Daraz, Facebook, and Rokomari; (2) development of a hybrid classification framework that employs graph-based aspect-opinion matching with semantic similarity techniques; and (3) implementation of an ensemble model combining BanglaBERT contextual embeddings with XGBoost boosting algorithms for enhanced triplet extraction performance. Experimental results demonstrate that our ensemble approach achieves superior performance with 89.9% accuracy and 89.1% F1-score, significantly outperforming baseline models across all evaluation metrics. The framework effectively addresses key challenges in Bangla text processing including informal expressions, spelling variations, and data sparsity. This research advances the state-of-the-art in low-resource language sentiment analysis and provides a scalable solution for Bangla e-commerce analytics applications.
comment : Presented at the 2025 IEEE International Conference on Signal Processing, Information, Communication and Systems (SPICSCON), November 21-22, 2025, University of Rajshahi, Bangladesh. 6 pages, ensemble deep learning, 3,345 annotated Bangla product reviews
☆ Emergent Lexical Semantics in Neural Language Models: Testing Martin's Law on LLM-Generated Text
Kai Kugler
We present the first systematic investigation of Martin's Law - the empirical relationship between word frequency and polysemy - in text generated by neural language models during training. Using DBSCAN clustering of contextualized embeddings as an operationalization of word senses, we analyze four Pythia models (70M-1B parameters) across 30 training checkpoints. Our results reveal a non-monotonic developmental trajectory: Martin's Law emerges around checkpoint 100, reaches peak correlation (r > 0.6) at checkpoint 104, then degrades by checkpoint 105. Smaller models (70M, 160M) experience catastrophic semantic collapse at late checkpoints, while larger models (410M, 1B) show graceful degradation. The frequency-specificity trade-off remains stable (r $\approx$ -0.3) across all models. These findings suggest that compliance with linguistic regularities in LLM-generated text is not monotonically increasing with training, but instead follows a balanced trajectory with an optimal semantic window. This work establishes a novel methodology for evaluating emergent linguistic structure in neural language models.
comment : paper draft
☆ TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories
Kirti Bhagat, Shaily Bhatt, Athul Velagapudi, Aditya Vashistha, Shachi Dave, Danish Pruthi
Millions of users across the globe turn to AI chatbots for their creative needs, inviting widespread interest in understanding how such chatbots represent diverse cultures. At the same time, evaluating cultural representations in open-ended tasks remains challenging and underexplored. In this work, we present TALES, an evaluation of cultural misrepresentations in LLM-generated stories for diverse Indian cultural identities. First, we develop TALES-Tax, a taxonomy of cultural misrepresentations by collating insights from participants with lived experiences in India through focus groups (N=9) and individual surveys (N=15). Using TALES-Tax, we evaluate 6 models through a large-scale annotation study spanning 2,925 annotations from 108 annotators with lived cultural experience from across 71 regions in India and 14 languages. Concerningly, we find that 88\% of the generated stories contain one or more cultural inaccuracies, and such errors are more prevalent in mid- and low-resourced languages and stories based in peri-urban regions in India. Lastly, we transform the annotations into TALES-QA, a standalone question bank to evaluate the cultural knowledge of foundational models. Through this evaluation, we surprisingly discover that models often possess the requisite cultural knowledge despite generating stories rife with cultural misrepresentations.
☆ PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark
Robert Belanec, Branislav Pecher, Ivan Srba, Maria Bielikova
Despite the state-of-the-art performance of Large Language Models (LLMs) achieved on many tasks, their massive scale often leads to high computational and environmental costs, limiting their accessibility. Parameter-efficient fine-tuning (PEFT) methods address this challenge by reducing the number of trainable parameters while maintaining strong downstream performance. Despite the increased development in PEFT methods, current evaluations remain limited (in terms of evaluated models and datasets) and difficult to reproduce. To bridge this gap, we introduce PEFT-Bench, a unified end-to-end benchmark for evaluating diverse PEFT methods on autoregressive LLMs. We demonstrate its usage across 27 NLP datasets and 6 PEFT methods. To account for different PEFT training and inference factors, we also introduce the PEFT Soft Score Penalties (PSCP) metric, which takes trainable parameters, inference speed, and training memory usage into account.
☆ Developing an Open Conversational Speech Corpus for the Isan Language
Adisai Na-Thalang, Chanakan Wittayasakpan, Kritsadha Phatcharoen, Supakit Buakaw
This paper introduces the development of the first open conversational speech dataset for the Isan language, the most widely spoken regional dialect in Thailand. Unlike existing speech corpora that are primarily based on read or scripted speech, this dataset consists of natural speech, thereby capturing authentic linguistic phenomena such as colloquials, spontaneous prosody, disfluencies, and frequent code-switching with central Thai. A key challenge in building this resource lies in the lack of a standardized orthography for Isan. Current writing practices vary considerably, due to the different lexical tones between Thai and Isan. This variability complicates the design of transcription guidelines and poses questions regarding consistency, usability, and linguistic authenticity. To address these issues, we establish practical transcription protocols that balance the need for representational accuracy with the requirements of computational processing. By releasing this dataset as an open resource, we aim to contribute to inclusive AI development, support research on underrepresented languages, and provide a basis for addressing the linguistic and technical challenges inherent in modeling conversational speech.
comment : 31 pages, in Thai language, 3 figures, 25 tables
☆ Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?
Steven Wang, Kyle Hunt, Shaojie Tang, Kenneth Joseph
There is ongoing debate about whether large language models (LLMs) can serve as substitutes for human participants in survey and experimental research. While recent work in fields such as marketing and psychology has explored the potential of LLM-based simulation, a growing body of evidence cautions against this practice: LLMs often fail to align with real human behavior, exhibiting limited diversity, systematic misalignment for minority subgroups, insufficient within-group variance, and discrepancies between stated beliefs and actions. This study examines an important and distinct question in this domain: whether fine-tuning on a small subset of human survey data, such as that obtainable from a pilot study, can mitigate these issues and yield realistic simulated outcomes. Using a behavioral experiment on information disclosure, we compare human and LLM-generated responses across multiple dimensions, including distributional divergence, subgroup alignment, belief-action coherence, and the recovery of regression coefficients. We find that fine-tuning on small human samples substantially improves heterogeneity, alignment, and belief-action coherence relative to the base model. However, even the best-performing fine-tuned models fail to reproduce the regression coefficients of the original study, suggesting that LLM-generated data remain unsuitable for replacing human participants in formal inferential analyses.
☆ Self-Guided Defense: Adaptive Safety Alignment for Reasoning Models via Synthesized Guidelines
Yuhang Wang, Yanxu Zhu, Dongyuan Lu, Jitao Sang
Reasoning models have demonstrated remarkable capabilities in complex reasoning tasks. However, ensuring their safety against adversarial jailbreak prompts remains a critical challenge. Due to the covert and deceptive nature of such prompts, they can often evade built-in safety mechanisms and lead to the generation of harmful content. This underscores the need for an adaptive safety alignment approach that enables models to autonomously reinforce their defenses in response to adversarial inputs. This paper introduces the Synthesized Guideline-based Adaptive Safety Alignment (SGASA) framework, which internalizes model-generated safety guidelines to strengthen models' ability to enhance robustness against harmful adversarial prompts while minimizing unnecessary refusals of benign requests. SGASA consists of two key stages: Data Pre-synthesis, which generates safety guidelines and augmented prompts; and Alignment Fine-tuning, which leverages Supervised Fine-tuning (SFT) and Direct Preference Optimization (DPO) to embed these guidelines into the model. Extensive experiments across multiple datasets demonstrate that SGASA significantly improves model safety, validating its adaptive and scalable effectiveness.
☆ AnchorOPT: Towards Optimizing Dynamic Anchors for Adaptive Prompt Learning
Zheng Li, Yibing Song, Xin Zhang, Lei Luo, Xiang Li, Jian Yang
Existing prompt learning methods, which are built upon CLIP models, leverage textual tokens as anchors to guide the learnable soft tokens. This guidance improves CLIP generalizations. However, these anchors-static in both value and position-lack cross-task and stage-adaptive flexibility. To address this limitation, we propose AnchorOPT, a dynamic anchor-based prompt learning framework. Specifically, AnchorOPT introduces dynamism in two key dimensions: (i) anchor values eschew handcrafted explicit textual tokens (e.g., "shape", "color"), instead learning dynamically from task-specific data; and (ii) the positional relationship between anchor and soft tokens is no longer fixed but adaptively optimized via a learnable position matrix conditioned on the training stage and task context. Training occurs in two stages: we first learn the anchor tokens, then freeze and transfer them to the second stage for optimization of soft tokens and the position matrix. Extensive experiments demonstrate that using only a simple learnable anchor and position matrix achieves performance comparable to or exceeding some methods incorporating additional learnable modules or regularization techniques. As a plug-and-play module, AnchorOPT integrates seamlessly into existing frameworks, yielding consistent performance gains across diverse datasets. Code is publicly available at https://github.com/zhengli97/ATPrompt.
comment : Technical Report
☆ How to Correctly Report LLM-as-a- Judge Evaluations
Chungpa Lee, Thomas Zeng, Jongwon Jeong, Jy-yong Sohn, Kangwook Lee
Large language models (LLMs) are increasingly used as evaluators in lieu of humans. While scalable, their judgments are noisy due to imperfect specificity and sensitivity of LLMs, leading to biased accuracy estimates. Although bias-correction methods exist, they are underutilized in LLM research and typically assume exact knowledge of the model's specificity and sensitivity. Furthermore, in general we only have estimates of these values and it is not well known how to properly construct confidence intervals using only estimates. This work presents a simple plug-in framework that corrects such bias and constructs confidence intervals reflecting uncertainty from both test and calibration dataset, enabling practical and statistically sound LLM-based evaluation. Additionally, to reduce uncertainty in the accuracy estimate, we introduce an adaptive algorithm that efficiently allocates calibration sample sizes.
☆ MortgageLLM: Domain-Adaptive Pretrain ing with Residual Instruction Transfer, Alignment Tuning, and Task-Specific Routing
Manish Jain, Satheesh Kumar Ponnambalam, Salman Faroz, Chandrakanth Lns, Vinay Sharma
Large Language Models (LLMs) demonstrate exceptional capabilities across general domains, yet their application to specialized sectors such as mortgage finance requires domain-specific knowledge augmentation while preserving instruction-following fidelity. We present MortgageLLM, a novel domain-specific large language model that addresses this dual challenge. It is developed using a dual-track specialization framework from a single base model (LLaMA-3.1-8B). We opted for this dual-expert approach as a single multi-task model suffers from performance trade-offs, where optimizing for structured tasks (via SFT) degrades conversational fidelity (via DPO). Our dual-track method solves this by creating two specialists, allowing each to be optimally trained for its distinct capability. Our approach applies the instruction residual technique to restore instruction-following capabilities post-domain adaptation without supervised fine-tuning. We contribute: (1) application of this residual technique to the highly specialized mortgage finance domain; (2) a dual-expert architecture combining a conversational Q&A model and a structured task model for classification and summarization; and (3) an intelligent task routing mechanism using few-shot classification performed by one of the expert models itself. We validate our approach on domain-specific benchmarks, where our final model (MLM v2) significantly outperforms the base LLaMA-3.1-8B-Instruct, achieving an LLM-as-a-Judge summarization score of 4.58 (vs. 3.99), a Q&A score of 4.09 (vs. 4.0), and a classification score of 2.6 (vs. 1.2). On semantic similarity, our model achieved a BERTScore of 0.77 for summarization (vs. 0.74), 0.68 for Q&A (vs. 0.58), and 0.75 for classification (vs. 0.73), substantially outperforming baseline approaches.
☆ ASR Error Correction in Low-Resource Burmese with Alignment-Enhanced Transformer s using Phonetic Features
Ye Bhone Lin, Thura Aung, Ye Kyaw Thu, Thazin Myint Oo
This paper investigates sequence-to-sequence Transformer models for automatic speech recognition (ASR) error correction in low-resource Burmese, focusing on different feature integration strategies including IPA and alignment information. To our knowledge, this is the first study addressing ASR error correction specifically for Burmese. We evaluate five ASR backbones and show that our ASR Error Correction (AEC) approaches consistently improve word- and character-level accuracy over baseline outputs. The proposed AEC model, combining IPA and alignment features, reduced the average WER of ASR models from 51.56 to 39.82 before augmentation (and 51.56 to 43.59 after augmentation) and improving chrF++ scores from 0.5864 to 0.627, demonstrating consistent gains over the baseline ASR outputs without AEC. Our results highlight the robustness of AEC and the importance of feature design for improving ASR outputs in low-resource settings.
comment : 7 pages, 2 figures, 7 tables, Accepted to iSAI-NLP 2025
☆ Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models
Bryan E. Tuck, Rakesh M. Verma
Large language models must satisfy hard orthographic constraints during controlled text generation, yet systematic cross-architecture evaluation remains limited. We evaluate 28 configurations spanning three model families (Qwen3, Claude Haiku-4.5, GPT-5-mini) on 58 word puzzles requiring character-level constraint satisfaction. Architectural differences produce substantially larger performance gaps (2.0-2.2x, F1=0.761 vs. 0.343) than parameter scaling within families (83% gain from eightfold scaling), suggesting that constraint satisfaction may require specialized architectural features or training objectives beyond standard language model scaling. Thinking budget sensitivity proves heterogeneous: high-capacity models show strong returns (+0.102 to +0.136 F1), while mid-sized variants saturate or degrade. These patterns are inconsistent with uniform compute benefits. Using difficulty ratings from 10,000 human solvers per puzzle, we establish modest but consistent calibration (r=0.24-0.38) across all families, yet identify systematic failures on common words with unusual orthography ("data", "poop", "loll": 86-95% human success, 89-96% model miss rate). These failures reveal over-reliance on distributional plausibility that penalizes orthographically atypical but constraint-valid patterns, suggesting architectural innovations may be required beyond simply scaling parameters or computational budgets.
☆ Enhancing Burmese News Classification with Kolmogorov-Arnold Network Head Fine-tuning
Thura Aung, Eaint Kay Khaing Kyaw, Ye Kyaw Thu, Thazin Myint Oo, Thepchai Supnithi
In low-resource languages like Burmese, classification tasks often fine-tune only the final classification layer, keeping pre-trained encoder weights frozen. While Multi-Layer Perceptrons (MLPs) are commonly used, their fixed non-linearity can limit expressiveness and increase computational cost. This work explores Kolmogorov-Arnold Networks (KANs) as alternative classification heads, evaluating Fourier-based FourierKAN, Spline-based EfficientKAN, and Grid-based FasterKAN-across diverse embeddings including TF-IDF, fastText, and multilingual transformers (mBERT, Distil-mBERT). Experimental results show that KAN-based heads are competitive with or superior to MLPs. EfficientKAN with fastText achieved the highest F1-score (0.928), while FasterKAN offered the best trade-off between speed and accuracy. On transformer embeddings, EfficientKAN matched or slightly outperformed MLPs with mBERT (0.917 F1). These findings highlight KANs as expressive, efficient alternatives to MLPs for low-resource language classification.
comment : 6 pages, 2 figures, 4 tables, Accepted to iSAI-NLP 2025
☆ Context -Aware Pragmatic Metacognitive Prompt ing for Sarcasm Detection
Michael Iskandardinata, William Christian, Derwin Suhartono
Detecting sarcasm remains a challenging task in the areas of Natural Language Processing (NLP) despite recent advances in neural network approaches. Currently, Pre-trained Language Models (PLMs) and Large Language Models (LLMs) are the preferred approach for sarcasm detection. However, the complexity of sarcastic text, combined with linguistic diversity and cultural variation across communities, has made the task more difficult even for PLMs and LLMs. Beyond that, those models also exhibit unreliable detection of words or tokens that require extra grounding for analysis. Building on a state-of-the-art prompting method in LLMs for sarcasm detection called Pragmatic Metacognitive Prompting (PMP), we introduce a retrieval-aware approach that incorporates retrieved contextual information for each target text. Our pipeline explores two complementary ways to provide context: adding non-parametric knowledge using web-based retrieval when the model lacks necessary background, and eliciting the model's own internal knowledge for a self-knowledge awareness strategy. We evaluated our approach with three datasets, such as Twitter Indonesia Sarcastic, SemEval-2018 Task 3, and MUStARD. Non-parametric retrieval resulted in a significant 9.87% macro-F1 improvement on Twitter Indonesia Sarcastic compared to the original PMP method. Self-knowledge retrieval improves macro-F1 by 3.29% on Semeval and by 4.08% on MUStARD. These findings highlight the importance of context in enhancing LLMs performance in sarcasm detection task, particularly the involvement of culturally specific slang, references, or unknown terms to the LLMs. Future work will focus on optimizing the retrieval of relevant contextual information and examining how retrieval quality affects performance. The experiment code is available at: https://github.com/wllchrst/sarcasm-detection_pmp_knowledge-base.
☆ Zipf Distributions from Two-Stage Symbolic Processes: Stability Under Stochastic Lexical Filtering
Vladimir Berman
Zipf's law in language lacks a definitive origin, debated across fields. This study explains Zipf-like behavior using geometric mechanisms without linguistic elements. The Full Combinatorial Word Model (FCWM) forms words from a finite alphabet, generating a geometric distribution of word lengths. Interacting exponential forces yield a power-law rank-frequency curve, determined by alphabet size and blank symbol probability. Simulations support predictions, matching English, Russian, and mixed-genre data. The symbolic model suggests Zipf-type laws arise from geometric constraints, not communicative efficiency.
comment : 16 pages
☆ A Unified Understanding of Offline Data Selection and Online Self-refining Generation for Post-training LLMs
Quan Xiao, Tianyi Chen
Offline data selection and online self-refining generation, which enhance the data quality, are crucial steps in adapting large language models (LLMs) to specific downstream tasks. We tackle offline data selection and online self-refining generations through an optimization perspective. Specifically, bilevel data selection is used for offline data selection with respect to the validation dataset, and we treat online self-refining generation as a model adaptation step of selecting the model trained on current responses that best fits the validation data. Our framework offers a unified understanding of offline data selection and self-refining generation by assigning a learned data weight to each question and response, either explicitly or implicitly. For the first time, we theoretically demonstrate the effectiveness of the bilevel data selection framework and demonstrate its performance gains over unfiltered direct mixing baselines. By combining offline data with validation-weighted online generations, our method enhances fine-tuning performance. Experiments on quality enhancement and safety-aware LLM fine-tuning validate its effectiveness.
☆ Semantic Anchors in In- Context Learning: Why Small LLMs Cannot Flip Their Labels
Anantha Padmanaban Krishna Kumar
Can in-context learning (ICL) override pre-trained label semantics, or does it merely refine an existing semantic backbone? We address this question by treating LLMs as prompt-induced classifiers and contrasting their behavior under \emph{natural} demonstrations (with correct labels) and \emph{inverted} demonstrations (systematically flipping label meanings). We decompose ICL behavior into three alignment metrics (truth, prior, and prompt alignment) and introduce a semantic override rate, defined as correctness under flipped semantics. Across eight classification tasks and eight open-source LLMs (1--12B parameters), we find consistent evidence for a semantic anchor view. With natural demonstrations, ICL improves accuracy while maintaining strong prior alignment; most correct predictions coincide with zero-shot behavior, even when the prior is weak. With inverted demonstrations, models cannot learn coherent anti-semantic classifiers: prompt alignment increases only by sacrificing accuracy, and semantic override rates remain exactly zero in our few-shot 1--12B setting. Rather than flexibly remapping label meanings, ICL primarily adjusts how inputs project onto stable semantic directions learned during pre-training, clarifying fundamental limits of few-shot prompting and suggesting that overriding label semantics at these scales requires interventions beyond ICL. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/semantic-anchors-icl.
comment : 13 pages total (7 pages main text, 3 pages references, 3 pages appendix), 2 figures, 14 tables. Code available at https://github.com/AnanthaPadmanaban-KrishnaKumar/semantic-anchors-icl
☆ Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regression
Liangzu Peng, Aditya Chattopadhyay, Luca Zancato, Elvis Nunez, Wei Xia, Stefano Soatto
As efficient alternatives to softmax Attention, linear state-space models (SSMs) achieve constant memory and linear compute, but maintain only a lossy, fading summary of the past, often leading to inferior performance in recall oriented tasks. We propose Gated KalmaNet (GKA), a layer that reduces this gap by accounting for the full past when predicting the next token, while maintaining SSM-style efficiency. GKA achieves this by solving an online ridge regression problem at test time, with constant memory and linear compute cost in the sequence length. Drawing inspiration from the Kalman Filter, we iteratively solve the online ridge regression problem. However, a critical insight is that standard Kalman filter equations are numerically unstable in low-precision environments (like bfloat16) and difficult to parallelize in modern hardware. We address both challenges through two key innovations: (1) an adaptive regularization strategy with input-dependent gating that controls the condition number of the ridge regression, ensuring numerical stability while balancing memory retention. And (2) the use of Chebyshev Iteration instead of other conventional iterative solvers, which we demonstrate to be more stable in low-precision settings. To further improve scalability, we develop a hardware-aware chunk-wise implementation of Chebyshev Iteration along with custom kernels for backpropagating through our adaptive regularization and gating mechanisms. Empirically, GKA shows strong language understanding capabilites on short-context tasks outperforming existing SSM layers (like Mamba2, GLA and Gated DeltaNet). On long-context, GKA excels at real-world RAG and LongQA tasks up to 128k tokens, achieving more than $10$% relative improvement over other fading memory baselines.
comment : 30 pages, 10 figures
☆ TrackList: Tracing Back Query Linguistic Diversity for Head and Tail Knowledge in Open Large Language Models
Ioana Buhnila, Aman Sinha, Mathieu Constant
Large Language Models (LLMs) have proven efficient in giving definition-type answers to user input queries. While for humans giving various types of answers, such as examples and paraphrases, is an easy task, LLMs struggle to provide correct answers for other than definition-type queries. In this study, we evaluated this drop in performance using TrackList, a fine-grained linguistic and statistical analysis pipeline to investigate the impact of the pre-training data on LLMs answers to diverse linguistic queries. We also introduce RefoMed-EN, an English dataset consisting of 6170 human-annotated medical terms alongside their corresponding definitions, denominations, exemplifications, explanations, or paraphrases. We studied whether the high frequency of a concept (head) or low frequency (tail) impacts the language model's performance. We evaluated the quality of the LLM's output using syntactic and semantic similarity metrics, statistical correlations and embeddings. Results showed that the LLM's task performance for definition type questions is the highest, while for the exemplification type it is the lowest. Additionally, we showed that for definition-type questions, large language models are prone to paraphrase more on popular and frequent knowledge and less on tail and technical knowledge, especially in the expert texts.
comment : under review
☆ RosettaSpeech: Zero-Shot Speech-to-Speech Translation from Monolingual Data
Zhisheng Zheng, Xiaohang Sun, Tuan Dinh, Abhishek Yanamandra, Abhinav Jain, Zhu Liu, Sunil Hadap, Vimal Bhat, Manoj Aggarwal, Gerard Medioni, David Harwath
The scarcity of parallel speech corpora critically hampers speech-to-speech translation (S2ST), often forcing reliance on complex, multi-stage pipelines. This paper introduces RosettaSpeech, a novel and simplified framework for zero-shot S2ST that is trained on monolingual speech-text data augmented by machine translation supervision. While our method leverages the linguistic knowledge inherent in text-based NMT models, it strictly eliminates the need for parallel speech-to-speech pairs. Our model uniquely uses text as an intermediate bridge during training but functions as a direct, end-to-end speech-to-speech model at inference. This streamlined approach achieves state-of-the-art results on standard benchmarks. For instance, on the CVSS-C test set, RosettaSpeech outperforms leading systems, achieving an ASR-BLEU score of 25.17 for German-to-English and 29.86 for Spanish-to-English-relative gains of over 27% and 14%, respectively. Furthermore, we demonstrate that a single model can deliver strong many-to-one translation performance (FR/ES/DE -> EN). We also provide a foundational analysis of how training data scaling impacts model performance. By prioritizing reliance on abundant parallel text rather than difficult-to-acquire parallel speech, RosettaSpeech offers a scalable path to creating high-quality, speaker-preserving S2ST for a much broader array of languages.
comment : Work in progress
☆ Towards Audio Token Compression in Large Audio Language Models
Saurabhchand Bhati, Samuel Thomas, Hilde Kuehne, Rogerio Feris, James Glass
Large Audio Language Models (LALMs) demonstrate impressive performance across diverse tasks, ranging from speech recognition to general audio understanding. However, their scalability is limited by the quadratic complexity of attention and the high token rates of audio signals. These challenges make it difficult to extend LALMs to long-form audio and to deploy them on resource-constrained platforms such as edge devices. In this paper, we explore techniques such as unsupervised segmentation, uniform average pooling, etc., to reduce the number of audio tokens generated by the LALM's audio encoder but before they are consumed by the LLM decoder. To mitigate potential performance degradation introduced by the compressed representations, we employ low-rank adapters to finetune the model. We evaluate our proposed models on two tasks, automatic speech recognition and speech-to-speech translation tasks, that are dependent on effectively uncovering the underlying lexical content of the input signal and study the effect of downsampling on these tasks. Experimental results show that compressed LALMs can achieve performance closer to frame-level LALMs while reducing the input audio token count upto three times before the LLM backbone.
☆ TrafficLens: Multi-Camera Traffic Video Analysis Using LLMs SC
Md Adnan Arefeen, Biplob Debnath, Srimat Chakradhar
Traffic cameras are essential in urban areas, playing a crucial role in intelligent transportation systems. Multiple cameras at intersections enhance law enforcement capabilities, traffic management, and pedestrian safety. However, efficiently managing and analyzing multi-camera feeds poses challenges due to the vast amount of data. Analyzing such huge video data requires advanced analytical tools. While Large Language Models (LLMs) like ChatGPT, equipped with retrieval-augmented generation (RAG) systems, excel in text-based tasks, integrating them into traffic video analysis demands converting video data into text using a Vision-Language Model (VLM), which is time-consuming and delays the timely utilization of traffic videos for generating insights and investigating incidents. To address these challenges, we propose TrafficLens, a tailored algorithm for multi-camera traffic intersections. TrafficLens employs a sequential approach, utilizing overlapping coverage areas of cameras. It iteratively applies VLMs with varying token limits, using previous outputs as prompts for subsequent cameras, enabling rapid generation of detailed textual descriptions while reducing processing time. Additionally, TrafficLens intelligently bypasses redundant VLM invocations through an object-level similarity detector. Experimental results with real-world datasets demonstrate that TrafficLens reduces video-to-text conversion time by up to $4\times$ while maintaining information accuracy.
comment : 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC)
☆ Chatty-KG: A Multi- Agent AI System for On-Demand Conversational Question Answering over Knowledge Graphs SIGMOD 2026
Reham Omar, Abdelghny Orogat, Ibrahim Abdelaziz, Omij Mangukiya, Panos Kalnis, Essam Mansour
Conversational Question Answering over Knowledge Graphs (KGs) combines the factual grounding of KG-based QA with the interactive nature of dialogue systems. KGs are widely used in enterprise and domain applications to provide structured, evolving, and reliable knowledge. Large language models (LLMs) enable natural and context-aware conversations, but lack direct access to private and dynamic KGs. Retrieval-augmented generation (RAG) systems can retrieve graph content but often serialize structure, struggle with multi-turn context, and require heavy indexing. Traditional KGQA systems preserve structure but typically support only single-turn QA, incur high latency, and struggle with coreference and context tracking. To address these limitations, we propose Chatty-KG, a modular multi-agent system for conversational QA over KGs. Chatty-KG combines RAG-style retrieval with structured execution by generating SPARQL queries through task-specialized LLM agents. These agents collaborate for contextual interpretation, dialogue tracking, entity and relation linking, and efficient query planning, enabling accurate and low-latency translation of natural questions into executable queries. Experiments on large and diverse KGs show that Chatty-KG significantly outperforms state-of-the-art baselines in both single-turn and multi-turn settings, achieving higher F1 and P@1 scores. Its modular design preserves dialogue coherence and supports evolving KGs without fine-tuning or pre-processing. Evaluations with commercial (e.g., GPT-4o, Gemini-2.0) and open-weight (e.g., Phi-4, Gemma 3) LLMs confirm broad compatibility and stable performance. Overall, Chatty-KG unifies conversational flexibility with structured KG grounding, offering a scalable and extensible approach for reliable multi-turn KGQA.
comment : This paper is accepted to SIGMOD 2026
☆ ENACT: Evaluating Embodied Cognition with Wo rl d Modeling of Egocentric Interaction
Qineng Wang, Wenlong Huang, Yu Zhou, Hang Yin, Tianwei Bao, Jianwen Lyu, Weiyu Liu, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, Manling Li
Embodied cognition argues that intelligence arises from sensorimotor interaction rather than passive observation. It raises an intriguing question: do modern vision-language models (VLMs), trained largely in a disembodied manner, exhibit signs of embodied cognition? We introduce ENACT, a benchmark that casts evaluation of embodied cognition as world modeling from egocentric interaction in a visual question answering (VQA) format. Framed as a partially observable Markov decision process (POMDP) whose actions are scene graph changes, ENACT comprises two complementary sequence reordering tasks: forward world modeling (reorder shuffled observations given actions) and inverse world modeling (reorder shuffled actions given observations). While conceptually simple, solving these tasks implicitly demands capabilities central to embodied cognition-affordance recognition, action-effect reasoning, embodied awareness, and interactive, long-horizon memory from partially observable egocentric input, while avoiding low-level image synthesis that could confound the evaluation. We provide a scalable pipeline that synthesizes QA pairs from robotics simulation (BEHAVIOR) and evaluates models on 8,972 QA pairs spanning long-horizon home-scale activities. Experiments reveal a performance gap between frontier VLMs and humans that widens with interaction horizon. Models consistently perform better on the inverse task than the forward one and exhibit anthropocentric biases, including a preference for right-handed actions and degradation when camera intrinsics or viewpoints deviate from human vision. Website at https://enact-embodied-cognition.github.io/.
comment : Preprint version
☆ Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity ACL
Pamela D. Rivière, Sean Trott
Despite an in-principle understanding of self-attention matrix operations in Transformer language models (LMs), it remains unclear precisely how these operations map onto interpretable computations or functions--and how or when individual attention heads develop specialized attention patterns. Here, we present a pipeline to systematically probe attention mechanisms, and we illustrate its value by leveraging lexical ambiguity--where a single word has multiple meanings--to isolate attention mechanisms that contribute to word sense disambiguation. We take a "developmental" approach: first, using publicly available Pythia LM checkpoints, we identify inflection points in disambiguation performance for each LM in the suite; in 14M and 410M, we identify heads whose attention to disambiguating words covaries with overall disambiguation performance across development. We then stress-test the robustness of these heads to stimulus perturbations: in 14M, we find limited robustness, but in 410M, we identify multiple heads with surprisingly generalizable behavior. Then, in a causal analysis, we find that ablating the target heads demonstrably impairs disambiguation performance, particularly in 14M. We additionally reproduce developmental analyses of 14M across all of its random seeds. Together, these results suggest: that disambiguation benefits from a constellation of mechanisms, some of which (especially in 14M) are highly sensitive to the position and part-of-speech of the disambiguating cue; and that larger models (410M) may contain heads with more robust disambiguation behavior. They also join a growing body of work that highlights the value of adopting a developmental perspective when probing LM mechanisms.
comment : 13 pages (main text), 5 figures (main text) 6 pages (appendix), 6 figures (appendix), journal submission to TACL ("a" decision: pre-MIT Press publication version)
☆ A Comparative Study of LLM Prompt ing and Fine-Tuning for Cross-genre Authorship Attribution on Chinese Lyrics
Yuxin Li, Lorraine Xu, Meng Fan Wang
We propose a novel study on authorship attribution for Chinese lyrics, a domain where clean, public datasets are sorely lacking. Our contributions are twofold: (1) we create a new, balanced dataset of Chinese lyrics spanning multiple genres, and (2) we develop and fine-tune a domain-specific model, comparing its performance against zero-shot inference using the DeepSeek LLM. We test two central hypotheses. First, we hypothesize that a fine-tuned model will outperform a zero-shot LLM baseline. Second, we hypothesize that performance is genre-dependent. Our experiments strongly confirm Hypothesis 2: structured genres (e.g. Folklore & Tradition) yield significantly higher attribution accuracy than more abstract genres (e.g. Love & Romance). Hypothesis 1 receives only partial support: fine-tuning improves robustness and generalization in Test1 (real-world data and difficult genres), but offers limited or ambiguous gains in Test2, a smaller, synthetically-augmented set. We show that the design limitations of Test2 (e.g., label imbalance, shallow lexical differences, and narrow genre sampling) can obscure the true effectiveness of fine-tuning. Our work establishes the first benchmark for cross-genre Chinese lyric attribution, highlights the importance of genre-sensitive evaluation, and provides a public dataset and analytical framework for future research. We conclude with recommendations: enlarge and diversify test sets, reduce reliance on token-level data augmentation, balance author representation across genres, and investigate domain-adaptive pretraining as a pathway for improved attribution performance.
comment : 8 pages, 6 figures
☆ Tracing How Annotators Think: Augmenting Preference Judgments with Reading Processes
Karin de Langis, William Walker, Khanh Chi Le, Dongyeop Kang
We propose an annotation approach that captures not only labels but also the reading process underlying annotators' decisions, e.g., what parts of the text they focus on, re-read or skim. Using this framework, we conduct a case study on the preference annotation task, creating a dataset PreferRead that contains fine-grained annotator reading behaviors obtained from mouse tracking. PreferRead enables detailed analysis of how annotators navigate between a prompt and two candidate responses before selecting their preference. We find that annotators re-read a response in roughly half of all trials, most often revisiting the option they ultimately choose, and rarely revisit the prompt. Reading behaviors are also significantly related to annotation outcomes: re-reading is associated with higher inter-annotator agreement, whereas long reading paths and times are associated with lower agreement. These results demonstrate that reading processes provide a complementary cognitive dimension for understanding annotator reliability, decision-making and disagreement in complex, subjective NLP tasks. Our code and data are publicly available.
☆ A Customer Journey in the Land of Oz: Leveraging the Wizard of Oz Technique to Model Emotions in Customer Service Interactions
Sofie Labat, Thomas Demeester, Véronique Hoste
Emotion-aware customer service needs in-domain conversational data, rich annotations, and predictive capabilities, but existing resources for emotion recognition are often out-of-domain, narrowly labeled, and focused on post-hoc detection. To address this, we conducted a controlled Wizard of Oz (WOZ) experiment to elicit interactions with targeted affective trajectories. The resulting corpus, EmoWOZ-CS, contains 2,148 bilingual (Dutch-English) written dialogues from 179 participants across commercial aviation, e-commerce, online travel agencies, and telecommunication scenarios. Our contributions are threefold: (1) Evaluate WOZ-based operator-steered valence trajectories as a design for emotion research; (2) Quantify human annotation performance and variation, including divergences between self-reports and third-party judgments; (3) Benchmark detection and forward-looking emotion inference in real-time support. Findings show neutral dominates participant messages; desire and gratitude are the most frequent non-neutral emotions. Agreement is moderate for multilabel emotions and valence, lower for arousal and dominance; self-reports diverge notably from third-party labels, aligning most for neutral, gratitude, and anger. Objective strategies often elicit neutrality or gratitude, while suboptimal strategies increase anger, annoyance, disappointment, desire, and confusion. Some affective strategies (cheerfulness, gratitude) foster positive reciprocity, whereas others (apology, empathy) can also leave desire, anger, or annoyance. Temporal analysis confirms successful conversation-level steering toward prescribed trajectories, most distinctly for negative targets; positive and neutral targets yield similar final valence distributions. Benchmarks highlight the difficulty of forward-looking emotion inference from prior turns, underscoring the complexity of proactive emotion-aware support.
☆ Improving Score Reliability of Multiple Choice Benchmark s with Consistency Evaluation and Altered Answer Choices
Paulo Cavalin, Cassia Sanctos, Marcelo Grave, Claudio Pinhanez, Yago Primerano
In this work we present the Consistency-Rebalanced Accuracy (CoRA) metric, improving the reliability of Large Language Model (LLM) scores computed on multiple choice (MC) benchmarks. Our metric explores the response consistency of the LLMs, taking advantage of synthetically-generated questions with altered answer choices. With two intermediate scores, i.e. Bare-Minimum-Consistency Accuracy (BMCA) and Consistency Index (CI), CoRA is computed by adjusting the multiple-choice question answering (MCQA) scores to better reflect the level of consistency of the LLM. We present evaluations in different benchmarks using diverse LLMs, and not only demonstrate that LLMs can present low response consistency even when they present high MCQA scores, but also that CoRA can successfully scale down the scores of inconsistent models.
☆ FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers
Sarina Xi, Vishisht Rao, Justin Payan, Nihar B. Shah
The identification and localization of errors is a core task in peer review, yet the exponential growth of scientific output has made it increasingly difficult for human reviewers to reliably detect errors given the limited pool of experts. Recent advances in Large Language Models (LLMs) have sparked interest in their potential to support such evaluation tasks, from academic peer review to automated scientific assessment. However, despite the growing use of LLMs in review systems, their capabilities to pinpoint errors remain underexplored. In this work, we introduce Fault Localization Across Writing in Science (FLAWS), an automated benchmark consisting of 713 paper-error pairs designed to evaluate how effectively LLMs detect errors that undermine key claims in research papers. We construct the benchmark by systematically inserting claim-invalidating errors into peer-reviewed papers using LLMs, paired with an automated evaluation metric that measures whether models can identify and localize these errors. Developing such a benchmark presents unique challenges that we overcome: ensuring that the inserted errors are well-defined, challenging, and relevant to the content of the paper, avoiding artifacts that would make identification trivial, and designing a scalable, automated evaluation metric. On the resulting benchmark, we evaluate five frontier LLMs: Claude Sonnet 4.5, DeepSeek Reasoner v3.1, Gemini 2.5 Pro, GPT 5, and Grok 4. Among these, GPT 5 is the top-performing model, achieving 39.1% identification accuracy when k=10, where k is the number of top-ranked error text candidates generated by the LLM.
comment : 30 pages, 12 tables, 2 figures
☆ Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models
Kecheng Chen, Ziru Liu, Xijia Tao, Hui Liu, Xinyu Fu, Suiyun Zhang, Dandan Tu, Lingpeng Kong, Rui Liu, Haoliang Li
Diffusion Language Models (DLMs) have recently achieved significant success due to their any-order generation capabilities. However, existing inference methods typically rely on local, immediate-step metrics such as confidence or entropy which inherently lack a more reliable perspective. This limitation frequently leads to inconsistent sampling trajectories and suboptimal generation quality. To address this, we propose Coherent Contextual Decoding (CCD), a novel inference framework built upon two core innovations. First, CCD employs a trajectory rectification mechanism that leverages historical context to enhance sequence coherence, enabling the early rejection of suboptimal paths. We demonstrate that this mechanism is theoretically equivalent to modeling the consistency of historical steps via the conditional mutual information between context and token predictions. Building on this theoretical insight, we further address the inefficiency of conventional uniform decoding budgets. Instead of rigid allocations based on diffusion steps, we introduce an adaptive sampling strategy that dynamically adjusts the unmasking budget for each step according to our consistency metric. Consequently, our method significantly improves the quality of generation trajectories while accelerating the sampling process. Empirically, our method achieves a simultaneous enhancement in both inference speed and performance across diverse benchmarks on Dream and LLaDA, delivering up to 3.48x speedup alongside 3.91% performance improvement.
♻ ☆ AdvancedIF: Rubric-Based Benchmark ing and Reinforcement Learning for Advancing LLM Instruction Following
Yun He, Wenzhe Li, Hejia Zhang, Songlin Li, Karishma Mandyam, Sopan Khosla, Yuanhao Xiong, Nanshu Wang, Xiaoliang Peng, Beibin Li, Shengjie Bi, Shishir G. Patil, Qi Qi, Shengyu Feng, Julian Katz-Samuels, Richard Yuanzhe Pang, Sujan Gonugondla, Hunter Lang, Yue Yu, Yundi Qian, Maryam Fazel-Zarandi, Licheng Yu, Amine Benhalloum, Hany Awadalla, Manaal Faruqui
Recent progress in large language models (LLMs) has led to impressive performance on a range of tasks, yet advanced instruction following (IF)-especially for complex, multi-turn, and system-prompted instructions-remains a significant challenge. Rigorous evaluation and effective training for such capabilities are hindered by the lack of high-quality, human-annotated benchmarks and reliable, interpretable reward signals. In this work, we introduce AdvancedIF (we will release this benchmark soon), a comprehensive benchmark featuring over 1,600 prompts and expert-curated rubrics that assess LLMs ability to follow complex, multi-turn, and system-level instructions. We further propose RIFL (Rubric-based Instruction-Following Learning), a novel post-training pipeline that leverages rubric generation, a finetuned rubric verifier, and reward shaping to enable effective reinforcement learning for instruction following. Extensive experiments demonstrate that RIFL substantially improves the instruction-following abilities of LLMs, achieving a 6.7% absolute gain on AdvancedIF and strong results on public benchmarks. Our ablation studies confirm the effectiveness of each component in RIFL. This work establishes rubrics as a powerful tool for both training and evaluating advanced IF in LLMs, paving the way for more capable and reliable AI systems.
♻ ☆ TimeViper: A Hybrid Mamba- Transformer Vision -Language Model for Efficient Long Video Understanding
Boshen Xu, Zihan Xiao, Jiaze Li, Jianzhong Ju, Zhenbo Luo, Jian Luan, Qin Jin
We introduce TimeViper, a hybrid vision-language model designed to tackle challenges of long video understanding. Processing long videos demands both an efficient model architecture and an effective mechanism for handling extended temporal contexts. To this end, TimeViper adopts a hybrid Mamba-Transformer backbone that combines the efficiency of state-space models with the expressivity of attention mechanisms. Through this hybrid design, we reveal the vision-to-text information aggregation phenomenon, where information progressively flows from vision tokens to text tokens across increasing LLM depth, resulting in severe vision token redundancy. Motivated by this observation, we propose TransV, a token information transfer module that transfers and compresses vision tokens into instruction tokens while maintaining multimodal understanding capabilities. This design enables TimeViper to process hour-long videos exceeding 10,000 frames. Extensive experiments across multiple benchmarks demonstrate that TimeViper competes with state-of-the-art models while extending frame numbers. We further analyze attention behaviors of both Mamba and Transformer layers, offering new insights into hybrid model interpretability. This work represents an initial step towards developing, interpreting, and compressing hybrid Mamba-Transformer architectures.
comment : Project page: https://xuboshen.github.io/TimeViper; Code: https://github.com/xiaomi-research/timeviper
♻ ☆ Leveraging Test Driven Development with Large Language Models for Reliable and Verifiable Spreadsheet Code Generation: A Research Framework
Simon Thorne, Advait Sarkar
Large Language Models (LLMs), such as ChatGPT, are increasingly leveraged for generating both traditional software code and spreadsheet logic. Despite their impressive generative capabilities, these models frequently exhibit critical issues such as hallucinations, subtle logical inconsistencies, and syntactic errors, risks particularly acute in high stakes domains like financial modelling and scientific computations, where accuracy and reliability are paramount. This position paper proposes a structured research framework that integrates the proven software engineering practice of Test-Driven Development (TDD) with Large Language Model (LLM) driven generation to enhance the correctness of, reliability of, and user confidence in generated outputs. We hypothesise that a "test first" methodology provides both technical constraints and cognitive scaffolding, guiding LLM outputs towards more accurate, verifiable, and comprehensible solutions. Our framework, applicable across diverse programming contexts, from spreadsheet formula generation to scripting languages such as Python and strongly typed languages like Rust, includes an explicitly outlined experimental design with clearly defined participant groups, evaluation metrics, and illustrative TDD based prompting examples. By emphasising test driven thinking, we aim to improve computational thinking, prompt engineering skills, and user engagement, particularly benefiting spreadsheet users who often lack formal programming training yet face serious consequences from logical errors. We invite collaboration to refine and empirically evaluate this approach, ultimately aiming to establish responsible and reliable LLM integration in both educational and professional development practices.
comment : 16 pages
♻ ☆ BengaliFig: A Low-Resource Challenge for Figurative and Culturally Grounded Reasoning in Bengali
Abdullah Al Sefat
Large language models excel on broad multilingual benchmarks but remain to be evaluated extensively in figurative and culturally grounded reasoning, especially in low-resource contexts. We present BengaliFig, a compact yet richly annotated challenge set that targets this gap in Bengali, a widely spoken low-resourced language. The dataset contains 435 unique riddles drawn from Bengali oral and literary traditions. Each item is annotated along five orthogonal dimensions capturing reasoning type, trap type, cultural depth, answer category, and difficulty, and is automatically converted to multiple-choice format through a constraint-aware, AI-assisted pipeline. We evaluate eight frontier LLMs from major providers under zero-shot and few-shot chain-of-thought prompting, revealing consistent weaknesses in metaphorical and culturally specific reasoning. BengaliFig thus contributes both a diagnostic probe for evaluating LLM robustness in low-resource cultural contexts and a step toward inclusive and heritage-aware NLP evaluation.
♻ ☆ Mem-PAL: Towards Memory-based Personalized Dialogue Assistants for Long-term User- Agent Interaction AAAI 2026
Zhaopei Huang, Qifeng Dai, Guozheng Wu, Xiaopeng Wu, Kehan Chen, Chuan Yu, Xubin Li, Tiezheng Ge, Wenxuan Wang, Qin Jin
With the rise of smart personal devices, service-oriented human-agent interactions have become increasingly prevalent. This trend highlights the need for personalized dialogue assistants that can understand user-specific traits to accurately interpret requirements and tailor responses to individual preferences. However, existing approaches often overlook the complexities of long-term interactions and fail to capture users' subjective characteristics. To address these gaps, we present PAL-Bench, a new benchmark designed to evaluate the personalization capabilities of service-oriented assistants in long-term user-agent interactions. In the absence of available real-world data, we develop a multi-step LLM-based synthesis pipeline, which is further verified and refined by human annotators. This process yields PAL-Set, the first Chinese dataset comprising multi-session user logs and dialogue histories, which serves as the foundation for PAL-Bench. Furthermore, to improve personalized service-oriented interactions, we propose H$^2$Memory, a hierarchical and heterogeneous memory framework that incorporates retrieval-augmented generation to improve personalized response generation. Comprehensive experiments on both our PAL-Bench and an external dataset demonstrate the effectiveness of the proposed memory framework.
comment : Accepted by AAAI 2026 (Oral)
♻ ☆ Co-NAML-LSTUR: A Combined Model with Attentive Multi-View Learning and Long- and Short-term User Representations for News Recommendation
Minh Hoang Nguyen, Thuat Thien Nguyen, Minh Nhat Ta, Tung Le, Huy Tien Nguyen
News recommendation systems play a critical role in alleviating information overload by delivering personalized content. A key challenge lies in jointly modeling multi-view representations of news articles and capturing the dynamic, dual-scale nature of user interests-encompassing both short- and long-term preferences. Prior methods often rely on single-view features or insufficiently model user behavior across time. In this work, we introduce Co-NAML-LSTUR, a hybrid news recommendation framework that integrates NAML for attentive multi-view news encoding and LSTUR for hierarchical user modeling, designed for training on limited data resources. Our approach leverages BERT-based embeddings to enhance semantic representation. We evaluate Co-NAML-LSTUR on two widely used benchmarks, MIND-small and MIND-large. Results show that our model significantly outperforms strong baselines, achieving improvements over NRMS by 1.55% in AUC and 1.15% in MRR, and over NAML by 2.45% in AUC and 1.71% in MRR. These findings highlight the effectiveness of our efficiency-focused hybrid model, which combines multi-view news modeling with dual-scale user representations for practical, resource-limited resources rather than a claim to absolute state-of-the-art (SOTA). The implementation of our model is publicly available at https://github.com/MinhNguyenDS/Co-NAML-LSTUR
comment : The 18th International Conference on Multi-disciplinary Trends in Artificial Intelligence (MIWAI 2025)
♻ ☆ Yesterday's News: Benchmark ing Multi-Dimensional Out-of-Distribution Generalization of Mis information Detection Models
Ivo Verhoeven, Pushkar Mishra, Ekaterina Shutova
This article introduces misinfo-general, a benchmark dataset for evaluating misinformation models' ability to perform out-of-distribution generalization. Misinformation changes rapidly, much more quickly than moderators can annotate at scale, resulting in a shift between the training and inference data distributions. As a result, misinformation detectors need to be able to perform out-of-distribution generalization, an attribute they currently lack. Our benchmark uses distant labelling to enable simulating covariate shifts in misinformation content. We identify time, event, topic, publisher, political bias, misinformation type as important axes for generalization, and we evaluate a common class of baseline models on each. Using article metadata, we show how this model fails desiderata, which is not necessarily obvious from classification metrics. Finally, we analyze properties of the data to ensure limited presence of modelling shortcuts. We make the dataset and accompanying code publicly available: https://github.com/ioverho/misinfo-general
comment : Accepted for publication in Computational Linguistics on November 23, 2025. This is the pre-MIT Press publication version
♻ ☆ BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotations
Simone Giovannini, Fabio Coppini, Andrea Gemelli, Simone Marinai
We present a unified dataset for document Question-Answering (QA), which is obtained combining several public datasets related to Document AI and visually rich document understanding (VRDU). Our main contribution is twofold: on the one hand we reformulate existing Document AI tasks, such as Information Extraction (IE), into a Question-Answering task, making it a suitable resource for training and evaluating Large Language Models; on the other hand, we release the OCR of all the documents and include the exact position of the answer to be found in the document image as a bounding box. Using this dataset, we explore the impact of different prompting techniques (that might include bounding box information) on the performance of open-weight models, identifying the most effective approaches for document comprehension.
♻ ☆ Improved Visually Prompt ed Keyword Localisation in Real Low-Resource Settings
Leanne Nortje, Dan Oneata, Gabriel Pirlogeanu, Herman Kamper
Given an image query, visually prompted keyword localisation (VPKL) aims to find occurrences of the depicted word in a speech collection. This can be useful when transcriptions are not available for a low-resource language (e.g. if it is unwritten). Previous work showed that VPKL can be performed with a visually grounded speech model trained on paired images and unlabelled speech. But all experiments were done on English. Moreover, transcriptions were used to get positive and negative pairs for the contrastive loss. This paper introduces a few-shot learning scheme to mine pairs automatically without transcriptions. On English, this results in only a small drop in performance. We also - for the first time - consider VPKL on a real low-resource language, Yoruba. While scores are reasonable, here we see a bigger drop in performance compared to using ground truth pairs because the mining is less accurate in Yoruba.
comment : Accepted at SpeD 2025
♻ ☆ Scaling Efficient LLMs
B. N. Kausik
Recent LLMs have hundreds of billions of parameters consuming vast resources. Furthermore, the so called "AI scaling law" for transformers suggests that the number of parameters must scale linearly with the size of the data. In response, we inquire into efficient LLMs, i.e. those with the fewest parameters that achieve the desired accuracy on a training corpus. Specifically, by comparing theoretical and empirical estimates of the Kullback-Leibler divergence, we derive a natural AI scaling law that the number of parameters in an efficient LLM scales as $D^γ$ where $D$ is the size of the training data and $ γ\in [0.44, 0.72]$, suggesting the existence of more efficient architectures. Against this backdrop, we propose recurrent transformers, combining the efficacy of transformers with the efficiency of recurrent networks, progressively applying a single transformer layer to a fixed-width sliding window across the input sequence. Recurrent transformers (a) run in linear time in the sequence length, (b) are memory-efficient and amenable to parallel processing in large batches, (c) learn to forget history for language tasks, or accumulate history for long range tasks like copy and selective copy, and (d) are amenable to curriculum training to overcome vanishing gradients. In our experiments, we find that recurrent transformers perform favorably on benchmark tests.
♻ ☆ Step-Audio-R1 Technical Report
Fei Tian, Xiangyu Tony Zhang, Yuxin Zhang, Haoyang Zhang, Yuxin Li, Daijiao Liu, Yayue Deng, Donghang Wu, Jun Chen, Liang Zhao, Chengyuan Yao, Hexin Liu, Eng Siong Chng, Xuerui Yang, Xiangyu Zhang, Daxin Jiang, Gang Yu
Recent advances in reasoning models have demonstrated remarkable success in text and vision domains through extended chain-of-thought deliberation. However, a perplexing phenomenon persists in audio language models: they consistently perform better with minimal or no reasoning, raising a fundamental question - can audio intelligence truly benefit from deliberate thinking? We introduce Step-Audio-R1, the first audio reasoning model that successfully unlocks reasoning capabilities in the audio domain. Through our proposed Modality-Grounded Reasoning Distillation (MGRD) framework, Step-Audio-R1 learns to generate audio-relevant reasoning chains that genuinely ground themselves in acoustic features rather than hallucinating disconnected deliberations. Our model exhibits strong audio reasoning capabilities, surpassing Gemini 2.5 Pro and achieving performance comparable to the state-of-the-art Gemini 3 Pro across comprehensive audio understanding and reasoning benchmarks spanning speech, environmental sounds, and music. These results demonstrate that reasoning is a transferable capability across modalities when appropriately anchored, transforming extended deliberation from a liability into a powerful asset for audio intelligence. By establishing the first successful audio reasoning model, Step-Audio-R1 opens new pathways toward building truly multimodal reasoning systems that think deeply across all sensory modalities.
comment : 22 pages, 5 figures. Technical Report
♻ ☆ DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen-tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi, Pang Wei Koh
Deep research models perform multi-step research to produce long-form, well-attributed answers. However, most open deep research models are trained on easily verifiable short-form QA tasks via reinforcement learning with verifiable rewards (RLVR), which does not extend to realistic long-form tasks. We address this with Reinforcement Learning with Evolving Rubrics (RLER), in which we construct and maintain rubrics that co-evolve with the policy model during training; this allows the rubrics to incorporate information that the model has newly explored and to provide discriminative, on-policy feedback. Using RLER, we develop Deep Research Tulu (DR Tulu-8B), the first open model that is directly trained for open-ended, long-form deep research. Across four long-form deep research benchmarks in science, healthcare and general domains, DR Tulu substantially outperforms existing open deep research models, and matches or exceeds proprietary deep research systems, while being significantly smaller and cheaper per query. To facilitate future research, we release all data, models, and code, including our new MCP-based agent infrastructure for deep research systems.
♻ ☆ AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser
Ren Ma, Jiantao Qiu, Chao Xu, Pei Chu, Kaiwen Liu, Pengli Ren, Yuan Qu, Jiahui Peng, Linfeng Hou, Mengjie Liu, Lindong Lu, Wenchang Ning, Jia Yu, Rui Min, Jin Shi, Haojiong Chen, Peng Zhang, Wenjian Zhang, Qian Jiang, Zengjie Hu, Guoqiang Yang, Zhenxiang Li, Fukai Shang, Runyuan Ma, Chenlin Su, Zhongying Tu, Wentao Zhang, Dahua Lin, Conghui He
While web data quality is crucial for large language models, most curation efforts focus on filtering and deduplication,treating HTML-to-text extraction as a fixed pre-processing step. Existing web corpora rely on heuristic-based extractors like Trafilatura, which struggle to preserve document structure and frequently corrupt structured elements such as formulas, codes, and tables. We hypothesize that improving extraction quality can be as impactful as aggressive filtering strategies for downstream performance. We introduce MinerU-HTML, a novel extraction pipeline that reformulates content extraction as a sequence labeling problem solved by a 0.6B-parameter language model. Unlike text-density heuristics, MinerU-HTML leverages semantic understanding and employs a two-stage formatting pipeline that explicitly categorizes semantic elements before converting to Markdown. Crucially, its model-based approach is inherently scalable, whereas heuristic methods offer limited improvement pathways. On MainWebBench, our benchmark of 7,887 annotated web pages, MinerU-HTML achieves 81.8\% ROUGE-N F1 compared to Trafilatura's 63.6\%, with exceptional structured element preservation (90.9\% for code blocks, 94.0\% for formulas). Using MinerU-HTML, we construct AICC (AI-ready Common Crawl), a 7.3-trillion token multilingual corpus from two Common Crawl snapshots. In controlled pretraining experiments where AICC and Trafilatura-extracted TfCC undergo identical filtering, models trained on AICC (62B tokens) achieve 50.8\% average accuracy across 13 benchmarks, outperforming TfCC by 1.08pp-providing direct evidence that extraction quality significantly impacts model capabilities. AICC also surpasses RefinedWeb and FineWeb on key benchmarks. We publicly release MainWebBench, MinerU-HTML, and AICC, demonstrating that HTML extraction is a critical, often underestimated component of web corpus construction.
♻ ☆ Think Visually, Reason Textually: Vision -Language Synergy in ARC
Beichen Zhang, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, Jiaqi Wang
Abstract reasoning from minimal examples remains a core unsolved problem for frontier foundation models such as GPT-5 and Grok 4. These models still fail to infer structured transformation rules from a handful of examples, which is a key hallmark of human intelligence. The Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) provides a rigorous testbed for this capability, demanding conceptual rule induction and transfer to novel tasks. Most existing methods treat ARC-AGI as a purely textual reasoning task, overlooking the fact that humans rely heavily on visual abstraction when solving such puzzles. However, our pilot experiments reveal a paradox: naively rendering ARC-AGI grids as images degrades performance due to imprecise rule execution. This leads to our central hypothesis that vision and language possess complementary strengths across distinct reasoning stages: vision supports global pattern abstraction and verification, whereas language specializes in symbolic rule formulation and precise execution. Building on this insight, we introduce two synergistic strategies: (1) Vision-Language Synergy Reasoning (VLSR), which decomposes ARC-AGI into modality-aligned subtasks; and (2) Modality-Switch Self-Correction (MSSC), which leverages vision to verify text-based reasoning for intrinsic error correction. Extensive experiments demonstrate that our approach yields up to a 4.33\% improvement over text-only baselines across diverse flagship models and multiple ARC-AGI tasks. Our findings suggest that unifying visual abstraction with linguistic reasoning is a crucial step toward achieving generalizable, human-like intelligence in future foundation models. Source code is released at https://github.com/InternLM/ARC-VL.
♻ ☆ Reasoning Transfer for an Extremely Low-Resource and Endangered Language: Bridging Languages Through Sample-Efficient Language Understanding
Khanh-Tung Tran, Barry O'Sullivan, Hoang D. Nguyen
Recent advances have enabled Large Language Models (LLMs) to tackle reasoning tasks by generating chain-of-thought (CoT) rationales, yet these gains have largely applied to high-resource languages, leaving low-resource languages behind. In this work, we first investigate CoT techniques in extremely low-resource scenarios through previous prompting, model-editing, and fine-tuning approaches. We introduce English-Pivoted CoT Training, leveraging the insight that LLMs internally operate in a latent space aligned toward the dominant language. Given input in a low-resource language, we perform supervised fine-tuning to generate CoT in English and output the final response in the target language. Across mathematical reasoning benchmarks, our approach outperforms other baselines with up to 28.33% improvement in low-resource scenarios. Our analysis and additional experiments, including Mixed-Language CoT and Two-Stage Training, show that explicitly separating language understanding from reasoning enhances cross-lingual reasoning abilities. To facilitate future work, we also release \emph{LC2024}, the first benchmark for mathematical tasks in Irish, an extremely low-resource and endangered language. Our results and resources highlight a practical pathway to multilingual reasoning without extensive retraining in every extremely low-resource language, despite data scarcity.
♻ ☆ Characterizing Pattern Matching and Its Limits on Compositional Task Structures
Hoyeon Chang, Jinho Park, Hanseul Cho, Sohee Yang, Miyoung Ko, Hyeonbin Hwang, Seungpil Won, Dohaeng Lee, Youbin Ahn, Minjoon Seo
Despite impressive capabilities, LLMs' successes often rely on pattern-matching behaviors, yet these are also linked to OOD generalization failures in compositional tasks. However, behavioral studies commonly employ task setups that allow multiple generalization sources (e.g., algebraic invariances, structural repetition), obscuring a precise and testable account of how well LLMs perform generalization through pattern matching and their limitations. To address this ambiguity, we first formalize pattern matching as functional equivalence, i.e., identifying pairs of subsequences of inputs that consistently lead to identical results when the rest of the input is held constant. Then, we systematically study how decoder-only Transformer and Mamba behave in controlled tasks with compositional structures that isolate this mechanism. Our formalism yields predictive and quantitative insights: (1) Instance-wise success of pattern matching is well predicted by the number of contexts witnessing the relevant functional equivalence. (2) We prove a tight sample complexity bound of learning a two-hop structure by identifying the exponent of the data scaling law for perfect in-domain generalization. Our empirical results align with the theoretical prediction, under 20x parameter scaling and across architectures. (3) Path ambiguity is a structural barrier: when a variable influences the output via multiple paths, models fail to form unified intermediate state representations, impairing accuracy and interpretability. (4) Chain-of-Thought reduces data requirements yet does not resolve path ambiguity. Hence, we provide a predictive, falsifiable boundary for pattern matching and a foundational diagnostic for disentangling mixed generalization mechanisms.
♻ ☆ LightMem: Lightweight and Efficient Memory-Augmented Generation
Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, Yuqi Tang, Ziwen Xu, Shumin Deng, Yunzhi Yao, Mengru Wang, Shuofei Qiao, Huajun Chen, Ningyu Zhang
Despite their remarkable capabilities, Large Language Models (LLMs) struggle to effectively leverage historical interaction information in dynamic and complex environments. Memory systems enable LLMs to move beyond stateless interactions by introducing persistent information storage, retrieval, and utilization mechanisms. However, existing memory systems often introduce substantial time and computational overhead. To this end, we introduce a new memory system called LightMem, which strikes a balance between the performance and efficiency of memory systems. Inspired by the Atkinson-Shiffrin model of human memory, LightMem organizes memory into three complementary stages. First, cognition-inspired sensory memory rapidly filters irrelevant information through lightweight compression and groups information according to their topics. Next, topic-aware short-term memory consolidates these topic-based groups, organizing and summarizing content for more structured access. Finally, long-term memory with sleep-time update employs an offline procedure that decouples consolidation from online inference. On LongMemEval and LoCoMo, using GPT and Qwen backbones, LightMem consistently surpasses strong baselines, improving QA accuracy by up to 7.7% / 29.3%, reducing total token usage by up to 38x / 20.9x and API calls by up to 30x / 55.5x, while purely online test-time costs are even lower, achieving up to 106x / 117x token reduction and 159x / 310x fewer API calls. The code is available at https://github.com/zjunlp/LightMem.
comment : Work in progress
♻ ☆ A Systematic Analysis of Large Language Models with RAG-enabled Dynamic Prompt ing for Medical Error Detection and Correction
Farzad Ahmed, Joniel Augustine Jerome, Meliha Yetisgen, Özlem Uzuner
Objective: Clinical documentation contains factual, diagnostic, and management errors that can compromise patient safety. Large language models (LLMs) may help detect and correct such errors, but their behavior under different prompting strategies remains unclear. We evaluate zero-shot prompting, static prompting with random exemplars (SPR), and retrieval-augmented dynamic prompting (RDP) for three subtasks of medical error processing: error flag detection, error sentence detection, and error correction. Methods: Using the MEDEC dataset, we evaluated nine instruction-tuned LLMs (GPT, Claude, Gemini, and OpenAI o-series models). We measured performance using accuracy, recall, false-positive rate (FPR), and an aggregate score of ROUGE-1, BLEURT, and BERTScore for error correction. We also analyzed example outputs to identify failure modes and differences between LLM and clinician reasoning. Results: Zero-shot prompting showed low recall in both detection tasks, often missing abbreviation-heavy or atypical errors. SPR improved recall but increased FPR. Across all nine LLMs, RDP reduced FPR by about 15 percent, improved recall by 5 to 10 percent in error sentence detection, and generated more contextually accurate corrections. Conclusion: Across diverse LLMs, RDP outperforms zero-shot and SPR prompting. Using retrieved exemplars improves detection accuracy, reduces false positives, and enhances the reliability of medical error correction.
♻ ☆ Mechanism of Task-oriented Information Removal in In- context Learning
Hakaze Cho, Haolin Yang, Gouki Minegishi, Naoya Inoue
In-context Learning (ICL) is an emerging few-shot learning paradigm based on modern Language Models (LMs), yet its inner mechanism remains unclear. In this paper, we investigate the mechanism through a novel perspective of information removal. Specifically, we demonstrate that in the zero-shot scenario, LMs encode queries into non-selective representations in hidden states containing information for all possible tasks, leading to arbitrary outputs without focusing on the intended task, resulting in near-zero accuracy. Meanwhile, we find that selectively removing specific information from hidden states by a low-rank filter effectively steers LMs toward the intended task. Building on these findings, by measuring the hidden states on carefully designed metrics, we observe that few-shot ICL effectively simulates such task-oriented information removal processes, selectively removing the redundant information from entangled non-selective representations, and improving the output based on the demonstrations, which constitutes a key mechanism underlying ICL. Moreover, we identify essential attention heads inducing the removal operation, termed Denoising Heads, which enables the ablation experiments blocking the information removal operation from the inference, where the ICL accuracy significantly degrades, especially when the correct label is absent from the few-shot demonstrations, confirming both the critical role of the information removal mechanism and denoising heads.
comment : 87 pages, 90 figures, 7 tables
♻ ☆ UniChange: Unifying Change Detection with Multimodal Large Language Model
Xu Zhang, Danyang Li, Xiaohang Dong, Tianhao Wu, Hualong Yu, Jianye Wang, Qicheng Li, Xiang Li
Change detection (CD) is a fundamental task for monitoring and analyzing land cover dynamics. While recent high performance models and high quality datasets have significantly advanced the field, a critical limitation persists. Current models typically acquire limited knowledge from single-type annotated data and cannot concurrently leverage diverse binary change detection (BCD) and semantic change detection (SCD) datasets. This constraint leads to poor generalization and limited versatility. The recent advancements in Multimodal Large Language Models (MLLMs) introduce new possibilities for a unified CD framework. We leverage the language priors and unification capabilities of MLLMs to develop UniChange, the first MLLM-based unified change detection model. UniChange integrates generative language abilities with specialized CD functionalities. Our model successfully unifies both BCD and SCD tasks through the introduction of three special tokens: [T1], [T2], and [CHANGE]. Furthermore, UniChange utilizes text prompts to guide the identification of change categories, eliminating the reliance on predefined classification heads. This design allows UniChange to effectively acquire knowledge from multi-source datasets, even when their class definitions conflict. Experiments on four public benchmarks (WHU-CD, S2Looking, LEVIR-CD+, and SECOND) demonstrate SOTA performance, achieving IoU scores of 90.41, 53.04, 78.87, and 57.62, respectively, surpassing all previous methods. The code is available at https://github.com/Erxucomeon/UniChange.
♻ ☆ Prompt -R1: Collaborative Automatic Prompt ing Framework via End-to-end Reinforcement Learning
Wenjin Liu, Haoran Luo, Xueyuan Lin, Haoming Liu, Tiesunlong Shen, Jiapu Wang, Rui Mao, Erik Cambria
Recently, advanced large language models (LLMs) have emerged at an increasingly rapid pace. However, when faced with complex problems, most users are often unable to provide accurate and effective prompts to interact with LLMs, thus limiting the performance of LLMs. To address this challenge, we propose Prompt-R1, an end-to-end reinforcement learning framework that uses a small-scale LLM to collaborate with large-scale LLMs, replacing user interaction to solve problems better. This collaboration is cast as a multi-turn prompt interaction, where the small-scale LLM thinks and generates prompts, and the large-scale LLM performs complex reasoning. A dual-constrained reward is designed to optimize for correctness, generation quality, and reasoning accuracy. Prompt-R1 provides a plug-and-play framework that supports both inference and training with various large-scale LLMs. Experiments on multiple public datasets show that Prompt-R1 significantly outperforms baseline models across tasks. Our code is publicly available at https://github.com/QwenQKing/Prompt-R1.
♻ ☆ The Distribution of Dependency Distance and Hierarchical Distance in Contemporary Written Japanese and Its Influencing Factors
Linxuan Wang, Shuiyuan Yu
To explore the relationship between dependency distance (DD) and hierarchical distance (HD) in Japanese, we compared the probability distributions of DD and HD with and without sentence length fixed, and analyzed the changes in mean dependency distance (MDD) and mean hierarchical distance (MHD) as sentence length increases, along with their correlation coefficient based on the Balanced Corpus of Contemporary Written Japanese. It was found that the valency of the predicates is the underlying factor behind the trade-off relation between MDD and MHD in Japanese. Native speakers of Japanese regulate the linear complexity and hierarchical complexity through the valency of the predicates, and the relative sizes of MDD and MHD depend on whether the threshold of valency has been reached. Apart from the cognitive load, the valency of the predicates also affects the probability distributions of DD and HD. The effect of the valency of the predicates on the distribution of HD is greater than on that of DD, which leads to differences in their probability distributions and causes the mean of MDD to be lower than that of MHD.
comment : This paper has been accepted by the 13th International Quantitative Linguistics Conference QUALICO 2025
♻ ☆ UITron-Speech: Towards Automated GUI Agent s Based on Speech Instructions
Wenkang Han, Zhixiong Zeng, Jing Huang, Shu Jiang, Liming Zheng, Longrong Yang, Haibo Qiu, Chang Yao, Jingyuan Chen, Lin Ma
Autonomous agents for Graphical User Interfaces (GUIs) are revolutionizing human-computer interaction, yet their reliance on text-based instructions imposes limitations on accessibility and convenience, particularly in hands-free scenarios. To address this issue, we propose replacing text with speech as the instruction input modality for GUI agents, and introduce UITron-Speech, which is the first end-to-end GUI agent capable of directly processing speech instructions and on-device screenshots to predict user actions. To tackle the problem of data scarcity, we synthesize high-quality speech instruction datasets using a random-speaker text-to-speech model. Additionally, we design a mixed-modality training strategy to mitigate the inherent modality imbalance in pre-trained foundation models. Furthermore, we conduct a statistical analysis of the distribution of GUI grounding prediction errors and propose a training-free two-step grounding refinement method to alleviate minor localization deviations. Extensive experiments on multiple benchmarks demonstrate that UITron-Speech achieves robust performance and superior adaptability, underscoring the feasibility and potential of speech-driven GUI agents for more accessible and intelligent human-computer interaction. Our code and datasets are available at https://github.com/UITron-hub/UITron-Speech.
♻ ☆ Federated Large Language Models: Current Progress and Future Directions
Yuhang Yao, Jianyi Zhang, Junda Wu, Chengkai Huang, Yu Xia, Tong Yu, Ruiyi Zhang, Sungchul Kim, Ryan Rossi, Ang Li, Lina Yao, Julian McAuley, Yiran Chen, Carlee Joe-Wong
Large language models are rapidly gaining popularity and have been widely adopted in real-world applications. While the quality of training data is essential, privacy concerns arise during data collection. Federated learning offers a solution by allowing multiple clients to collaboratively train LLMs without sharing local data. However, FL introduces new challenges, such as model convergence issues due to heterogeneous data and high communication costs. A comprehensive study is required to address these challenges and guide future research. This paper surveys Federated learning for LLMs (FedLLM), highlighting recent advances and future directions. We focus on two key aspects: fine-tuning and prompt learning in a federated setting, discussing existing work and associated research challenges. We finally propose potential directions for federated LLMs, including pre-training, federated agents, and LLMs for federated learning.
♻ ☆ AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise NeurIPS 2025
Dhruv Agarwal, Bodhisattwa Prasad Majumder, Reece Adamson, Megha Chakravorty, Satvika Reddy Gavireddy, Aditya Parashar, Harshit Surana, Bhavana Dalvi Mishra, Andrew McCallum, Ashish Sabharwal, Peter Clark
The promise of autonomous scientific discovery (ASD) hinges not only on answering questions, but also on knowing which questions to ask. Most recent works in ASD explore the use of large language models (LLMs) in goal-driven settings, relying on human-specified research questions to guide hypothesis generation. However, scientific discovery may be accelerated further by allowing the AI system to drive exploration by its own criteria. The few existing approaches in open-ended ASD select hypotheses based on diversity heuristics or subjective proxies for human interestingness, but the former struggles to meaningfully navigate the typically vast hypothesis space, and the latter suffers from imprecise definitions. This paper presents AutoDiscovery -- a method for open-ended ASD that instead drives scientific exploration using Bayesian surprise. Here, we quantify the epistemic shift from the LLM's prior beliefs about a hypothesis to its posterior beliefs after gathering experimental results. To efficiently explore the space of nested hypotheses, our method employs a Monte Carlo tree search (MCTS) strategy with progressive widening using surprisal as the reward function. We evaluate AutoDiscovery in the setting of data-driven discovery across 21 real-world datasets spanning domains such as biology, economics, finance, and behavioral science. Our results demonstrate that under a fixed budget, AutoDiscovery substantially outperforms competitors by producing 5-29% more discoveries deemed surprising by the LLM. Our human evaluation further reveals that two-thirds of discoveries made by our system are surprising to domain experts as well, suggesting this is an important step towards building open-ended ASD systems.
comment : Accepted to NeurIPS 2025; https://neurips.cc/virtual/2025/loc/san-diego/poster/116398
♻ ☆ Meursault as a Data Point
Abhinav Pratap
In an era dominated by datafication, the reduction of human experiences to quantifiable metrics raises profound philosophical and ethical questions. This paper explores these issues through the lens of Meursault, the protagonist of Albert Camus' The Stranger, whose emotionally detached existence epitomizes the existential concept of absurdity. Using natural language processing (NLP) techniques including emotion detection (BERT), sentiment analysis (VADER), and named entity recognition (spaCy)-this study quantifies key events and behaviors in Meursault's life. Our analysis reveals the inherent limitations of applying algorithmic models to complex human experiences, particularly those rooted in existential alienation and moral ambiguity. By examining how modern AI tools misinterpret Meursault's actions and emotions, this research underscores the broader ethical dilemmas of reducing nuanced human narratives to data points, challenging the foundational assumptions of our data-driven society. The findings presented in this paper serve as a critique of the increasing reliance on data-driven narratives and advocate for incorporating humanistic values in artificial intelligence.
comment : 7 pages, 9 figures, 4 tables
♻ ☆ Enhancing Large Language Models for Detecting Mental Manipulation via Annotation-Free Data Augmentation and Anti-Curriculum Distillation
Yuansheng Gao, Han Bao, Tong Zhang, Bin Li, Jixiang Luo, Ronghao Chen, Zonghui Wang, Wenzhi Chen
Mental manipulation is a subtle yet pervasive form of psychological abuse that poses serious threats to mental health. Nevertheless, detecting mental manipulation remains a largely underexplored research problem. The field faces three major challenges: (i) insufficient and hard-to-obtain training data; (ii) the covert nature of mental manipulation, which hinders detection; and (iii) the lack of real-world datasets. To address these challenges, we propose MentalMAC, a novel framework that enhances large language models' ability to detect elements of mental manipulation in multi-turn dialogue. Our approach consists of three key components: EvoSA, an annotation-free data augmentation method based on evolutionary operations and speech act theory; teacher-model-generated multi-task supervision; and progressive task-level anti-curriculum distillation. We then constructed the ReaMent dataset, comprising 5,000 real-world dialogue samples, utilizing MentalMAC-distilled models to aid in human annotation. Vast experiments show that MentalMAC achieves up to 25.9% improvement in F1mac and 8.1% in accuracy over the best-performing baseline, outperforming commercial LLMs such as GPT-4 and Claude-3.5-Sonnet. Warning: This paper contains content that may be offensive to the reader.
comment : Preprint
♻ ☆ CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness NeurIPS 2025
Zhihang Liu, Chen-Wei Xie, Bin Wen, Feiwu Yu, Jixuan Chen, Pandeng Li, Boqiang Zhang, Nianzu Yang, Yinglu Li, Zuan Gao, Yun Zheng, Hongtao Xie
Visual captioning benchmarks have become outdated with the emergence of modern multimodal large language models (MLLMs), as the brief ground-truth sentences and traditional metrics fail to assess detailed captions effectively. While recent benchmarks attempt to address this by focusing on keyword extraction or object-centric evaluation, they remain limited to vague-view or object-view analyses and incomplete visual element coverage. In this paper, we introduce CAPability, a comprehensive multi-view benchmark for evaluating visual captioning across 12 dimensions spanning six critical views. We curate nearly 11K human-annotated images and videos with visual element annotations to evaluate the generated captions. CAPability stably assesses both the correctness and thoroughness of captions with \textit{precision} and \textit{hit} metrics. By converting annotations to QA pairs, we further introduce a heuristic metric, \textit{know but cannot tell} ($K\bar{T}$), indicating a significant performance gap between QA and caption capabilities. Our work provides a holistic analysis of MLLMs' captioning abilities, as we identify their strengths and weaknesses across various dimensions, guiding future research to enhance specific aspects of their capabilities.
comment : Accepted to NeurIPS 2025
♻ ☆ Where to Start Alignment? Diffusion Large Language Model May Demand a Distinct Position AAAI 2026
Zhixin Xie, Xurui Song, Jun Luo
Diffusion Large Language Models (dLLMs) have recently emerged as a competitive non-autoregressive paradigm due to their unique training and inference approach. However, there is currently a lack of safety study on this novel architecture. In this paper, we present the first analysis of dLLMs' safety performance and propose a novel safety alignment method tailored to their unique generation characteristics. Specifically, we identify a critical asymmetry between the defender and attacker in terms of security. For the defender, we reveal that the middle tokens of the response, rather than the initial ones, are more critical to the overall safety of dLLM outputs; this seems to suggest that aligning middle tokens can be more beneficial to the defender. The attacker, on the contrary, may have limited power to manipulate middle tokens, as we find dLLMs have a strong tendency towards a sequential generation order in practice, forcing the attack to meet this distribution and diverting it from influencing the critical middle tokens. Building on this asymmetry, we introduce Middle-tOken Safety Alignment (MOSA), a novel method that directly aligns the model's middle generation with safe refusals exploiting reinforcement learning. We implement MOSA and compare its security performance against eight attack methods on two benchmarks. We also test the utility of MOSA-aligned dLLM on coding, math, and general reasoning. The results strongly prove the superiority of MOSA.
comment : Accepted for oral presentation at AAAI 2026
♻ ☆ Uncovering Implicit Bias in Large Language Models with Concept Learning Dataset
Leroy Z. Wang
We introduce a dataset of concept learning tasks that helps uncover implicit biases in large language models. Using in-context concept learning experiments, we found that language models may have a bias toward upward monotonicity in quantifiers; such bias is less apparent when the model is tested by direct prompting without concept learning components. This demonstrates that in-context concept learning can be an effective way to discover hidden biases in language models.
comment : Presented at EurIPS 2025 Workshop - Unifying Perspectives on Learning Biases (UPLB) https://sites.google.com/view/towards-a-unified-view
♻ ☆ Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization
Yue Zhang, Jingxuan Zuo, Liqiang Jing
Multimodal summarization aims to generate a concise summary based on the input text and image. However, the existing methods potentially suffer from unfactual output. To evaluate the factuality of multimodal summarization models, we propose two fine-grained and explainable evaluation frameworks (FALLACIOUS) for different application scenarios, i.e. reference-based factuality evaluation framework and reference-free factuality evaluation framework. Notably, the reference-free factuality evaluation framework doesn't need ground truth and hence it has a wider application scenario. To evaluate the effectiveness of the proposed frameworks, we compute the correlation between our frameworks and the other metrics. The experimental results show the effectiveness of our proposed method. We will release our code and dataset via github.
♻ ☆ Exploring Cross-Lingual Knowledge Transfer via Transliteration-Based MLM Fine-Tuning for Critically Low-resource Chakma Language
Adity Khisa, Nusrat Jahan Lia, Tasnim Mahfuz Nafis, Zarif Masud, Tanzir Pial, Shebuti Rayana, Ahmedul Kabir
As an Indo-Aryan language with limited available data, Chakma remains largely underrepresented in language models. In this work, we introduce a novel corpus of contextually coherent Bangla-transliterated Chakma, curated from Chakma literature, and validated by native speakers. Using this dataset, we fine-tune six encoder-based transformer models, including multilingual (mBERT, XLM-RoBERTa, DistilBERT), regional (BanglaBERT, IndicBERT), and monolingual English (DeBERTaV3) variants on masked language modeling (MLM) tasks. Our experiments show that fine-tuned multilingual models outperform their pre-trained counterparts when adapted to Bangla-transliterated Chakma, achieving up to 73.54% token accuracy and a perplexity as low as 2.90. Our analysis further highlights the impact of data quality on model performance and shows the limitations of OCR pipelines for morphologically rich Indic scripts. Our research demonstrates that Bangla-transliterated Chakma can be very effective for transfer learning for Chakma language, and we release our dataset to encourage further research on multilingual language modeling for low-resource languages.
♻ ☆ The Structure-Content Trade-off in Knowledge Graph Retrieval
Valentin Six, Evan Dufraisse, Gaël de Chalendar
Large Language Models (LLMs) increasingly rely on knowledge graphs for factual reasoning, yet how retrieval design shapes their performance remains unclear. We examine how question decomposition changes the retrieved subgraph's content and structure. Using a hybrid retrieval function that controls the importance of initial question and subquestions, we show that subquestion-based retrieval improves content precision, but yields disjoint subgraphs, while question-based retrieval maintains structure at the cost of relevance. Optimal performance arises between these extremes, revealing that balancing retrieval content and structure is key to effective LLM reasoning over structured knowledge.
♻ ☆ A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency
Sihyeong Park, Sungryeol Jeon, Chaelyn Lee, Seokhun Jeon, Byung-Soo Kim, Jemin Lee
Large language models (LLMs) are widely applied in chatbots, code generators, and search engines. Workload such as chain-of-throught, complex reasoning, agent services significantly increase the inference cost by invoke the model repeatedly. Optimization methods such as parallelism, compression, and caching have been adopted to reduce costs, but the diverse service requirements make it hard to select the right method. Recently, specialized LLM inference engines have emerged as a key component for integrating the optimization methods into service-oriented infrastructures. However, a systematic study on inference engines is still lacking.This paper provides a comprehensive evaluation of 25 open-source and commercial inference engines. We examine each inference engine in terms of ease-of-use, ease-of-deployment, general-purpose support, scalability, and suitability for throughput- and latency-aware computation. Furthermore, we explore the design goals of each inference engine by investigating the optimization techniques it supports. In addition, we assess the ecosystem maturity of open source inference engines and handle the performance and cost policy of commercial solutions.We outline future research directions that include support for complex LLM-based services, support of various hardware, and enhanced security, offering practical guidance to researchers and developers in selecting and designing optimized LLM inference engines. We also provide a public repository to continually track developments in this fast-evolving field: \href{https://github.com/sihyeong/Awesome-LLM-Inference-Engine}{https://github.com/sihyeong/Awesome-LLM-Inference-Engine}.
comment : Under review; 106 pages; 46 figures
♻ ☆ Beyond Introspection: Reinforcing Thinking via Externalist Behavioral Feedback
Diji Yang, Linda Zeng, Kezhen Chen, Yi Zhang
While inference-time thinking allows Large Language Models (LLMs) to address complex problems, the extended thinking process can be unreliable or inconsistent because of the model's probabilistic nature, especially near its knowledge boundaries. Existing approaches attempt to mitigate this by having the model critique its own reasoning to make corrections. However, such self-critique inherits the same biases of the original output, known as the introspection illusion. Moving beyond such introspection and inspired by core methodologies in ethology, we propose an externalist three-step framework Distillation-Reinforcement-Reasoning (DRR). Rather than relying on a model's introspection, DRR evaluates its observable behaviors to provide corrective feedback. DRR first distills the reasoner's behavioral traces, then trains a lightweight, external Discriminative Model (DM). At inference time, this DM acts as a critic, identifying and rejecting suspicious reasoning steps. This external feedback compels the LLM to discard flawed pathways and explore alternatives, thereby enhancing reasoning quality without altering the base model. Experiments on multiple reasoning benchmarks show that our framework significantly outperforms prominent self-critique methods. Benefiting from a lightweight and annotation-free design, DRR offers a scalable and adaptable solution for improving the reliability of reasoning in a wide range of LLMs.
♻ ☆ Evaluating Large Language Models for Radiology Natural Language Processing
Zhengliang Liu, Tianyang Zhong, Yiwei Li, Yutong Zhang, Yi Pan, Zihao Zhao, Peixin Dong, Chao Cao, Yuxiao Liu, Peng Shu, Yaonai Wei, Zihao Wu, Chong Ma, Jiaqi Wang, Sheng Wang, Mengyue Zhou, Zuowei Jiang, Chunlin Li, Jason Holmes, Shaochen Xu, Lu Zhang, Haixing Dai, Kai Zhang, Lin Zhao, Yuanhao Chen, Xu Liu, Peilong Wang, Junhao Chen, Pingkun Yan, Jun Liu, Bao Ge, Lichao Sun, Dajiang Zhu, Xiang Li, Wei Liu, Xiaoyan Cai, Xintao Hu, Xi Jiang, Shu Zhang, Xin Zhang, Tuo Zhang, Shijie Zhao, Quanzheng Li, Hongtu Zhu, Dinggang Shen, Tianming Liu
The rise of large language models (LLMs) has marked a pivotal shift in the field of natural language processing (NLP). LLMs have revolutionized a multitude of domains, and they have made a significant impact in the medical field. Large language models are now more abundant than ever, and many of these models exhibit bilingual capabilities, proficient in both English and Chinese. However, a comprehensive evaluation of these models remains to be conducted. This lack of assessment is especially apparent within the context of radiology NLP. This study seeks to bridge this gap by critically evaluating thirty two LLMs in interpreting radiology reports, a crucial component of radiology NLP. Specifically, the ability to derive impressions from radiologic findings is assessed. The outcomes of this evaluation provide key insights into the performance, strengths, and weaknesses of these LLMs, informing their practical applications within the medical domain.
♻ ☆ CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis AAAI 2026
Yuzhuang Xu, Xu Han, Yuanchi Zhang, Yixuan Wang, Yijun Liu, Shiyu Ji, Qingfu Zhu, Wanxiang Che
Large Language Models (LLMs) with Mixture-of-Experts (MoE) architectures are distinguished by their strong performance scaling with increasing parameters across a wide range of tasks, yet they also suffer from substantial computational and storage overheads. Notably, the performance gains of MoE models do not scale proportionally with the growth in expert parameters. While prior works attempt to reduce parameters via expert-level pruning, merging, or decomposition, they still suffer from challenges in both performance and computational efficiency. In this paper, we address these challenges by introducing micro-expert as a finer-grained compression unit that spans across matrices. We first establish a more fundamental perspective, viewing MoE layers as mixtures of micro-experts, and present CAMERA, a lightweight and training-free framework for identifying micro-expert redundancy. Our analysis uncovers significant variance in micro-expert contributions during decoding. Based on this insight, we further propose CAMERA-P, a structured micro-expert pruning framework, and CAMERA-Q, a mixed-precision quantization idea designed for micro-experts. Extensive experiments on nine downstream tasks show that CAMERA-P consistently outperforms strong baselines under pruning ratios ranging from 20% to 60%. Furthermore, CAMERA-Q achieves superior results under aggressive 2-bit quantization, surpassing existing matrix- and channel-level ideas. Notably, our method enables complete micro-expert analysis of Qwen2-57B-A14B in less than 5 minutes on a single NVIDIA A100-40GB GPU.
comment : Accepted in AAAI 2026
♻ ☆ Position-Aware Depth Decay Decoding ($D^3$): Boosting Large Language Model Inference Efficiency
Siqi Fan, Xuezhi Fang, Xingrun Xing, Peng Han, Shuo Shang, Yequan Wang
Due to the large number of parameters, the inference phase of Large Language Models (LLMs) is resource-intensive. Unlike traditional model compression, which needs retraining, recent dynamic computation methods show that not all components are required for inference, enabling a training-free pipeline. In this paper, we focus on the dynamic depth of LLM generation. A token-position aware layer skipping framework is proposed to save 1.5x times operations efficiently while maintaining performance. We first observed that tokens predicted later have lower perplexity and thus require less computation. Then, we propose a training-free algorithm called Position-Aware Depth Decay Decoding ($D^3$), which leverages a power-law decay function, $\left\lfloor L \times (α^i) \right\rfloor$, to determine the number of layers to retain when generating token $T_i$. Remarkably, without any retraining, the $D^3$ achieves success across a wide range of generation tasks for the first time. Experiments on large language models (\ie the Llama) with $7 \sim 70$ billion parameters show that $D^3$ can achieve an average 1.5x speedup compared with the full-inference pipeline while maintaining comparable performance with nearly no performance drop ($<1\%$) on the GSM8K and BBH benchmarks.
♻ ☆ Logic OCR : Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?
Maoyuan Ye, Haibin He, Qihuang Zhong, Jing Zhang, Juhua Liu, Bo Du
Recent advances in Large Multimodal Models (LMMs) have revolutionized their reasoning and Optical Character Recognition (OCR) capabilities. However, their complex logical reasoning performance on text-rich images remains underexplored. To bridge this gap, we introduce LogicOCR, a benchmark comprising 2780 questions with two subsets, i.e., LogicOCR-Gen with 1100 multi-choice questions on generated images, and LogicOCR-Real with 1680 meticulously designed free-form questions on real-world images. For constructing LogicOCR-Gen, we first curate a text corpus from the Chinese National Civil Servant Examination, and customize an automatic pipeline to steer GPT-Image-1 to generate images with varied layouts and fonts, ensuring contextual relevance and visual realism. Then, the generated images are manually verified. We evaluate a range of representative LMMs under Chain-of-Thought (CoT) and direct-answer settings. Our multi-dimensional analysis reveals key insights, such as the impact of test-time scaling, input modality differences, and sensitivity to visual-text orientation. Notably, LMMs still lag in multimodal reasoning compared to text-only inputs, indicating that they have not fully bridged visual reading with reasoning. Moreover, we propose TextCue, a training-free method that enhances LMMs' perception of image regions containing important text cues for solving questions. We leverage LMMs' attention maps and an off-the-shelf text segmentation specialist to determine the region, which is then cropped and enlarged to augment the original image. Experiments show its effectiveness, e.g., a 1.8% accuracy gain over LLaVA-OV-1.5-8B under the CoT setting. Our benchmark is available at https://github.com/MiliLab/LogicOCR.
comment : GitHub: https://github.com/MiliLab/LogicOCR
♻ ☆ Gram2Vec: An Interpretable Document Vectorizer
Peter Zeng, Hannah Stortz, Eric Sclafani, Alina Shabaeva, Maria Elizabeth Garza, Daniel Greeson, Owen Rambow
We present Gram2Vec, a grammatical style embedding system that embeds documents into a higher dimensional space by extracting the normalized relative frequencies of grammatical features present in the text. Compared to neural approaches, Gram2Vec offers inherent interpretability based on how the feature vectors are generated. In this paper, we use authorship verification and AI detection as two applications to show how Gram2Vec can be used. For authorship verification, we use the features from Gram2Vec to explain why a pair of documents is by the same or by different authors. We also demonstrate how Gram2Vec features can be used to train a classifier for AI detection, outperforming machine learning models trained on a comparable set of Biber features.
comment : 8 pages, 1 figure
♻ ☆ On The Role of Pretrain ed Language Models in General-Purpose Text Embeddings: A Survey
Meishan Zhang, Xin Zhang, Xinping Zhao, Shouzheng Huang, Baotian Hu, Min Zhang
Text embeddings have attracted growing interest due to their effectiveness across a wide range of natural language processing (NLP) tasks, including retrieval, classification, clustering, bitext mining, and summarization. With the emergence of pretrained language models (PLMs), general-purpose text embeddings (GPTE) have gained significant traction for their ability to produce rich, transferable representations. The general architecture of GPTE typically leverages PLMs to derive dense text representations, which are then optimized through contrastive learning on large-scale pairwise datasets. In this survey, we provide a comprehensive overview of GPTE in the era of PLMs, focusing on the roles PLMs play in driving its development. We first examine the fundamental architecture and describe the basic roles of PLMs in GPTE, i.e., embedding extraction, expressivity enhancement, training strategies, learning objectives, and data construction. We then describe advanced roles enabled by PLMs, including multilingual support, multimodal integration, code understanding, and scenario-specific adaptation. Finally, we highlight potential future research directions that move beyond traditional improvement goals, including ranking integration, safety considerations, bias mitigation, structural information incorporation, and the cognitive extension of embeddings. This survey aims to serve as a valuable reference for both newcomers and established researchers seeking to understand the current state and future potential of GPTE.
comment : 45 pages, 4 figures, 9 tables
♻ ☆ MTA: A Merge-then-Adapt Framework for Personalized Large Language Model
Xiaopeng Li, Yuanjin Zheng, Wanyu Wang, wenlin zhang, Pengyue Jia, Yiqi Wang, Maolin Wang, Xuetao Wei, Xiangyu Zhao
Personalized Large Language Models (PLLMs) aim to align model outputs with individual user preferences, a crucial capability for user-centric applications. However, the prevalent approach of fine-tuning a separate module for each user faces two major limitations: (1) storage costs scale linearly with the number of users, rendering the method unscalable; and (2) fine-tuning a static model from scratch often yields suboptimal performance for users with sparse data. To address these challenges, we propose MTA, a Merge-then-Adapt framework for PLLMs. MTA comprises three key stages. First, we construct a shared Meta-LoRA Bank by selecting anchor users and pre-training meta-personalization traits within meta-LoRA modules. Second, to ensure scalability and enable dynamic personalization combination beyond static models, we introduce an Adaptive LoRA Fusion stage. This stage retrieves and dynamically merges the most relevant anchor meta-LoRAs to synthesize a user-specific one, thereby eliminating the need for user-specific storage and supporting more flexible personalization. Third, we propose a LoRA Stacking for Few-Shot Personalization stage, which applies an additional ultra-low-rank, lightweight LoRA module on top of the merged LoRA. Fine-tuning this module enables effective personalization under few-shot settings. Extensive experiments on the LaMP benchmark demonstrate that our approach outperforms existing SOTA methods across multiple tasks.
♻ ☆ AppSelectBench: Application-Level Tool Selection Benchmark
Tianyi Chen, Michael Solodko, Sen Wang, Jongwoo Ko, Junheng Hao, Colby Banbury, Sara Abdali, Saeed Amizadeh, Qing Xiao, Yinheng Li, Tianyu Ding, Kamran Ghasedi Dizaji, Suzhen Zheng, Hao Fan, Justin Wagle, Pashmina Cameron, Kazuhito Koishida
Computer Using Agents (CUAs) are increasingly equipped with external tools, enabling them to perform complex and realistic tasks. For CUAs to operate effectively, application selection, which refers to deciding which application to use before invoking fine-grained tools such as APIs, is a fundamental capability. It determines whether the agent initializes the correct environment, avoids orchestration confusion, and efficiently focuses on relevant context. However, existing benchmarks primarily assess fine-grained API selection, offering limited insight into whether models can reason across and choose between different applications. To fill this gap, we introduce AppSelectBench, a comprehensive benchmark for evaluating application selection in CUAs. AppSelectBench contains a novel user task generation pipeline that produces realistic, diverse, and semantically grounded user intents at scale, together with unified evaluation protocols covering random, heuristic, zero-shot, few-shot, and retrieval-augmented-settings. AppSelectBench covers one hundred widely used desktop applications and includes more than one hundred thousand realistic, diverse, and semantically grounded user tasks. Extensive experiments across both closed-source and open-source large language models reveal systematic strengths and weaknesses in inter-application reasoning, showing that even the most capable models still struggle to make consistent application choices. Together, these results establish AppSelectBench as a foundation for studying and advancing application level reasoning, an essential yet underexplored capability of intelligent CUAs. The source is available at https://microsoft.github.io/appselectbench/.
♻ ☆ Harvesting Textual and Contrastive Data from the HAL Publication Repository
Francis Kulumba, Wissam Antoun, Guillaume Vimont, Laurent Romary
Authorship attribution in natural language processing traditionally struggles to distinguish genuine stylistic signals from topical confounds. While contrastive learning approaches have addressed this by maximizing semantic overlap between positive pairs, creating large-scale datasets under strict topic constraints remains challenging. We introduce HALvest, a 17-billion-token multilingual corpus harvested from 778k open-access academic papers, and HALvest-Contrastive, a derived dataset designed to isolate stylometric signals through controlled topic variation. Unlike prior work that minimizes lexical overlap, we exploit natural topic drift between papers by the same author, treating residual lexical patterns as authorial fingerprints rather than noise. Comparing lexical baselines (BM25) against neural models trained on unrestricted (topic-rich) versus base (topic-decoupled) triplets, we demonstrate that models trained exclusively on topic-decoupled data achieve superior performance across all test conditions, outperforming both retrieval baselines and models exposed to topic-rich training data. Our analysis reveals that while lexical signals provide substantial performance gains for keyword-driven methods, neural architectures learn robust stylometric representations that plateau with moderate context length, suggesting they capture distributional style beyond surface-level tokens. Both datasets and code are publicly available.
comment : New dataset version with only the contrastive learning data
♻ ☆ PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese
Thales Sales Almeida, Ramon Pires, Hugo Abonizio, Rodrigo Nogueira, Hélio Pedrini
Large Language Models (LLMs) exhibit significant variations in performance across linguistic and cultural contexts, underscoring the need for systematic evaluation in diverse languages. In this work, we present the most extensive evaluation of LLMs for the Portuguese language to date. Leveraging our newly introduced PoETa v2 benchmark -- a comprehensive suite of over 40 tasks in Portuguese -- we assess more than 20 models covering a broad spectrum of training scales and computational resources. Our study reveals how computational investment and language-specific adaptation impact performance in Portuguese, while also analyzing performance gaps in comparison to equivalent tasks in English. Through this benchmark and analysis, PoETa v2 lays the groundwork for future research on Portuguese language modeling and evaluation. The benchmark is available at https://github.com/PoETaV2/PoETaV2.
♻ ☆ Strong Memory, Weak Control: An Empirical Study of Executive Functioning in LLMs
Karin de Langis, Jong Inn Park, Bin Hu, Khanh Chi Le, Andreas Schramm, Michael C. Mensink, Andrew Elfenbein, Dongyeop Kang
Working memory, or the ability to hold and manipulate information in the mind, is a critical component of human intelligence and executive functioning. It is correlated with performance on various cognitive tasks, including measures of fluid intelligence, which encompasses reasoning and problem solving. We use a comprehensive set of classic working memory tasks to estimate the working memory capacity of large language models (LLMs). We find that in most cases, LLMs exceed normative human scores. However, we do not find that the increased capacity of working memory is associated with higher performance on other executive functioning tasks or problem solving benchmarks. These results suggest that LLMs may have deficits in attentional control and cognitive flexibility, which result in difficulties with inhibiting automatic responses and adapting to shifting information. Our findings suggest that current reasoning models have mixed results in compensating for these deficits.
♻ ☆ ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models
Emily Chang, Niyati Bafna
Existing benchmarks for large language models (LLMs) are largely restricted to high- or mid-resource languages, and often evaluate performance on higher-order tasks in reasoning and generation. However, plenty of evidence points to the fact that LLMs lack basic linguistic competence in the vast majority of the world's 3800+ written languages. We introduce ChiKhaPo, consisting of 8 subtasks of varying difficulty designed to evaluate the lexical comprehension and generation abilities of generative models. ChiKhaPo draws on existing lexicons, monolingual data, and bitext, and provides coverage for 2700+ languages for 2 subtasks, surpassing any existing benchmark in terms of language coverage. We further show that 6 SOTA models struggle on our benchmark, and discuss the factors contributing to performance scores, including language family, language resourcedness, task, and comprehension versus generation directions. With ChiKhaPo, we hope to enable and encourage the massively multilingual benchmarking of LLMs.
♻ ☆ Beyond the Rubric: Cultural Misalignment in LLM Benchmark s for Sexual and Reproductive Health
Sumon Kanti Dey, Manvi S, Zeel Mehta, Meet Shah, Unnati Agrawal, Suhani Jalota, Azra Ismail
Large Language Models (LLMs) have been positioned as having the potential to expand access to health information in the Global South, yet their evaluation remains heavily dependent on benchmarks designed around Western norms. We present insights from a preliminary benchmarking exercise with a chatbot for sexual and reproductive health (SRH) for an underserved community in India. We evaluated using HealthBench, a benchmark for conversational health models by OpenAI. We extracted 637 SRH queries from the dataset and evaluated on the 330 single-turn conversations. Responses were evaluated using HealthBench's rubric-based automated grader, which rated responses consistently low. However, qualitative analysis by trained annotators and public health experts revealed that many responses were actually culturally appropriate and medically accurate. We highlight recurring issues, particularly a Western bias, such as for legal framing and norms (e.g., breastfeeding in public), diet assumptions (e.g., fish safe to eat during pregnancy), and costs (e.g., insurance models). Our findings demonstrate the limitations of current benchmarks in capturing the effectiveness of systems built for different cultural and healthcare contexts. We argue for the development of culturally adaptive evaluation frameworks that meet quality standards while recognizing needs of diverse populations.
comment : https://github.com/Sumon/healthbench-srh-eval/
♻ ☆ Continual Learning with Global Alignment
Xueying Bai, Jinghuan Shang, Yifan Sun, Niranjan Balasubramanian
Continual learning aims to sequentially learn new tasks without forgetting previous tasks' knowledge (catastrophic forgetting). One factor that can cause forgetting is the interference between the gradients on losses from different tasks. When the gradients on the current task's loss are in opposing directions to those on previous tasks' losses, updating the model for the current task may cause performance degradation on previous tasks. In this paper, we first identify causes of the above interference, and hypothesize that correlations between data representations are a key factor of interference. We then propose a method for promoting appropriate correlations between arbitrary tasks' data representations (i.e., global alignment) in individual task learning. Specifically, we learn the data representation as a task-specific composition of pre-trained token representations shared across all tasks. Then the correlations between different tasks' data representations are grounded by correlations between pre-trained token representations. We explore different ways to learn such compositions. Without experience replay, our model achieves SOTA performance in continual learning tasks. It also achieves advanced class-incremental performance through task-incremental training.
comment : Neurips 2024
♻ ☆ Computational Fact-Checking of Online Discourse: Scoring scientific accuracy in climate change related news articles
Tim Wittenborg, Constantin Sebastian Tremel, Markus Stocker, Sören Auer
Democratic societies need reliable information. Misinformation in popular media, such as news articles or videos, threatens to impair civic discourse. Citizens are, unfortunately, not equipped to verify the flood of content consumed daily at increasing rates. This work aims to quantify the scientific accuracy of online media semi-automatically. We investigate the state of the art of climate-related ground truth knowledge representation. By semantifying media content of unknown veracity, their statements can be compared against these ground truth knowledge graphs. We implemented a workflow using LLM-based statement extraction and knowledge graph analysis. Our implementation can streamline content processing towards state-of-the-art knowledge representation and veracity quantification. Developed and evaluated with the help of 27 experts and detailed interviews with 10, the tool evidently provides a beneficial veracity indication. These findings are supported by 43 anonymous participants from a parallel user survey. This initial step, however, is unable to annotate public media at the required granularity and scale. Additionally, the identified state of climate change knowledge graphs is vastly insufficient to support this neurosymbolic fact-checking approach. Further work towards a FAIR (Findable, Accessible, Interoperable, Reusable) ground truth and complementary metrics is required to support civic discourse scientifically.
comment : 8 pages, 7 figures, accepted at ICKG 2025